2024-03-24T00:25:14.109 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-24T00:25:14.109 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/forward-scrub} --owner mchangir --archive /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616442 --name mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi --verbose -- /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616442/orig.config.yaml 2024-03-24T00:25:14.159 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-24T00:25:14.251 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616442 branch: wip-mchangir-testing-main-20240318.032620 description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/forward-scrub} email: mchangir@redhat.com first_in_suite: false job_id: '7616442' kernel: client: sha1: distro flavor: default kdb: true sha1: 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-mchangir-testing-main-20240318.032620 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - inode wrongly marked free - bad backtrace on inode - inode table repaired for inode - Scrub error on inode - Scrub error on dir - Metadata damage detected sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 kclient: syntax: v1 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 subvolume_version: 1 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-mchangir-testing-main-20240318.032620 sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 owner: mchangir priority: 100 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 3550 sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 sleep_before_teardown: 0 subset: 613/1024 suite: fs suite_branch: wip-mchangir-testing-main-20240318.032620 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi067.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB8CD0DPVKDGSdsXcNniFZ/nMWBblB6BCthjTFboTuuuZGK72PZsmuWPKA4ddb/TZ5BhSV6teWCVIHkd5gVxOLw= smithi072.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHKZ2OBRMHPdFxGrk+8dJ5aBUfN0YoKZJ/yHulR5ozeKvvv8ePCbz4UDpFjr7jPvwgWT4STFERjN8cT9GeVy2Ps= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - kclient: null - cephfs_test_runner: modules: - tasks.cephfs.test_forward_scrub teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-22_09:49:57 tube: smithi user: mchangir verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-24T00:25:14.251 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa; will attempt to use it 2024-03-24T00:25:14.252 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks 2024-03-24T00:25:14.252 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-24T00:25:14.253 INFO:teuthology.task.internal:Checking packages... 2024-03-24T00:25:14.280 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3be9a5c9bc793e11e2800a8c0c696e8b46742033' 2024-03-24T00:25:14.280 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-24T00:25:14.280 INFO:teuthology.packaging:ref: None 2024-03-24T00:25:14.280 INFO:teuthology.packaging:tag: None 2024-03-24T00:25:14.280 INFO:teuthology.packaging:branch: wip-mchangir-testing-main-20240318.032620 2024-03-24T00:25:14.280 INFO:teuthology.packaging:sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T00:25:14.280 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-mchangir-testing-main-20240318.032620 2024-03-24T00:25:14.495 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2145.g3be9a5c9 2024-03-24T00:25:14.497 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-24T00:25:14.503 INFO:teuthology.task.internal:no buildpackages task found 2024-03-24T00:25:14.503 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-24T00:25:14.524 INFO:teuthology.task.internal:Saving configuration 2024-03-24T00:25:14.541 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-24T00:25:14.549 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T00:25:14.573 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi067.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616442', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 00:17:14.605264', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB8CD0DPVKDGSdsXcNniFZ/nMWBblB6BCthjTFboTuuuZGK72PZsmuWPKA4ddb/TZ5BhSV6teWCVIHkd5gVxOLw='} 2024-03-24T00:25:14.595 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi072.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616442', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 00:17:14.603777', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHKZ2OBRMHPdFxGrk+8dJ5aBUfN0YoKZJ/yHulR5ozeKvvv8ePCbz4UDpFjr7jPvwgWT4STFERjN8cT9GeVy2Ps='} 2024-03-24T00:25:14.595 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-24T00:25:14.601 INFO:teuthology.task.internal:roles: ubuntu@smithi067.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-03-24T00:25:14.601 INFO:teuthology.task.internal:roles: ubuntu@smithi072.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-03-24T00:25:14.601 INFO:teuthology.run_tasks:Running task console_log... 2024-03-24T00:25:14.776 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f173cc1df70>, signals=[15]) 2024-03-24T00:25:14.776 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-24T00:25:14.782 INFO:teuthology.task.internal:Opening connections... 2024-03-24T00:25:14.782 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi067.front.sepia.ceph.com 2024-03-24T00:25:14.784 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T00:25:14.893 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi072.front.sepia.ceph.com 2024-03-24T00:25:14.894 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T00:25:14.982 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-24T00:25:14.988 DEBUG:teuthology.orchestra.run.smithi067:> uname -m 2024-03-24T00:25:15.006 INFO:teuthology.orchestra.run.smithi067.stdout:x86_64 2024-03-24T00:25:15.006 DEBUG:teuthology.orchestra.run.smithi067:> cat /etc/os-release 2024-03-24T00:25:15.062 INFO:teuthology.orchestra.run.smithi067.stdout:NAME="CentOS Stream" 2024-03-24T00:25:15.063 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION="9" 2024-03-24T00:25:15.063 INFO:teuthology.orchestra.run.smithi067.stdout:ID="centos" 2024-03-24T00:25:15.063 INFO:teuthology.orchestra.run.smithi067.stdout:ID_LIKE="rhel fedora" 2024-03-24T00:25:15.063 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION_ID="9" 2024-03-24T00:25:15.063 INFO:teuthology.orchestra.run.smithi067.stdout:PLATFORM_ID="platform:el9" 2024-03-24T00:25:15.063 INFO:teuthology.orchestra.run.smithi067.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-24T00:25:15.063 INFO:teuthology.orchestra.run.smithi067.stdout:ANSI_COLOR="0;31" 2024-03-24T00:25:15.063 INFO:teuthology.orchestra.run.smithi067.stdout:LOGO="fedora-logo-icon" 2024-03-24T00:25:15.063 INFO:teuthology.orchestra.run.smithi067.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-24T00:25:15.063 INFO:teuthology.orchestra.run.smithi067.stdout:HOME_URL="https://centos.org/" 2024-03-24T00:25:15.063 INFO:teuthology.orchestra.run.smithi067.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-24T00:25:15.063 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-24T00:25:15.063 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-24T00:25:15.064 INFO:teuthology.lock.ops:Updating smithi067.front.sepia.ceph.com on lock server 2024-03-24T00:25:15.088 DEBUG:teuthology.orchestra.run.smithi072:> uname -m 2024-03-24T00:25:15.105 INFO:teuthology.orchestra.run.smithi072.stdout:x86_64 2024-03-24T00:25:15.105 DEBUG:teuthology.orchestra.run.smithi072:> cat /etc/os-release 2024-03-24T00:25:15.162 INFO:teuthology.orchestra.run.smithi072.stdout:NAME="CentOS Stream" 2024-03-24T00:25:15.162 INFO:teuthology.orchestra.run.smithi072.stdout:VERSION="9" 2024-03-24T00:25:15.162 INFO:teuthology.orchestra.run.smithi072.stdout:ID="centos" 2024-03-24T00:25:15.162 INFO:teuthology.orchestra.run.smithi072.stdout:ID_LIKE="rhel fedora" 2024-03-24T00:25:15.162 INFO:teuthology.orchestra.run.smithi072.stdout:VERSION_ID="9" 2024-03-24T00:25:15.162 INFO:teuthology.orchestra.run.smithi072.stdout:PLATFORM_ID="platform:el9" 2024-03-24T00:25:15.162 INFO:teuthology.orchestra.run.smithi072.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-24T00:25:15.162 INFO:teuthology.orchestra.run.smithi072.stdout:ANSI_COLOR="0;31" 2024-03-24T00:25:15.162 INFO:teuthology.orchestra.run.smithi072.stdout:LOGO="fedora-logo-icon" 2024-03-24T00:25:15.162 INFO:teuthology.orchestra.run.smithi072.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-24T00:25:15.163 INFO:teuthology.orchestra.run.smithi072.stdout:HOME_URL="https://centos.org/" 2024-03-24T00:25:15.163 INFO:teuthology.orchestra.run.smithi072.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-24T00:25:15.163 INFO:teuthology.orchestra.run.smithi072.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-24T00:25:15.163 INFO:teuthology.orchestra.run.smithi072.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-24T00:25:15.163 INFO:teuthology.lock.ops:Updating smithi072.front.sepia.ceph.com on lock server 2024-03-24T00:25:15.179 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-24T00:25:15.189 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-24T00:25:15.194 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-24T00:25:15.194 DEBUG:teuthology.orchestra.run.smithi067:> test '!' -e /home/ubuntu/cephtest 2024-03-24T00:25:15.196 DEBUG:teuthology.orchestra.run.smithi072:> test '!' -e /home/ubuntu/cephtest 2024-03-24T00:25:15.217 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-24T00:25:15.222 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-24T00:25:15.223 DEBUG:teuthology.orchestra.run.smithi067:> test -z $(ls -A /var/lib/ceph) 2024-03-24T00:25:15.254 DEBUG:teuthology.orchestra.run.smithi072:> test -z $(ls -A /var/lib/ceph) 2024-03-24T00:25:15.283 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-24T00:25:15.335 INFO:teuthology.run_tasks:Running task kernel... 2024-03-24T00:25:15.350 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': '5ec3e6e88b2e2783e09498588bb1f433ae9c8986'} 2024-03-24T00:25:15.350 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}, 'client.2': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}, 'client.3': {'sha1': 'distro'}} 2024-03-24T00:25:15.350 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}}, timeout 300 2024-03-24T00:25:15.351 DEBUG:teuthology.orchestra.run.smithi067:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T00:25:15.351 DEBUG:teuthology.orchestra.run.smithi072:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T00:25:15.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:25:15.367 DEBUG:teuthology.orchestra.run.smithi072:> uname -r 2024-03-24T00:25:15.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:25:15.368 DEBUG:teuthology.orchestra.run.smithi067:> uname -r 2024-03-24T00:25:15.423 INFO:teuthology.orchestra.run.smithi072.stdout:5.14.0-361.el9.x86_64 2024-03-24T00:25:15.423 INFO:teuthology.task.kernel:Running kernel on smithi072: 5.14.0-361.el9.x86_64 2024-03-24T00:25:15.423 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum install -y kernel 2024-03-24T00:25:15.426 INFO:teuthology.orchestra.run.smithi067.stdout:5.14.0-361.el9.x86_64 2024-03-24T00:25:15.426 INFO:teuthology.task.kernel:Running kernel on smithi067: 5.14.0-361.el9.x86_64 2024-03-24T00:25:15.426 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum install -y kernel 2024-03-24T00:25:17.480 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-24T00:25:18.533 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-24T00:25:18.956 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS-9 - BaseOS 6.4 MB/s | 2.0 MB 00:00 2024-03-24T00:25:19.926 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS-9 - BaseOS 7.2 MB/s | 2.0 MB 00:00 2024-03-24T00:25:20.945 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-24T00:25:22.030 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - BaseOS 3.0 MB/s | 8.0 MB 00:02 2024-03-24T00:25:23.166 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - AppStream 16 MB/s | 19 MB 00:01 2024-03-24T00:25:23.848 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - AppStream 25 MB/s | 19 MB 00:00 2024-03-24T00:25:27.234 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - CRB 9.6 MB/s | 6.1 MB 00:00 2024-03-24T00:25:28.558 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - Extras packages 78 kB/s | 16 kB 00:00 2024-03-24T00:25:30.122 INFO:teuthology.orchestra.run.smithi072.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-03-24T00:25:35.013 INFO:teuthology.orchestra.run.smithi072.stdout:lab-extras 38 kB/s | 1.7 kB 00:00 2024-03-24T00:25:35.181 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - CRB 822 kB/s | 6.1 MB 00:07 2024-03-24T00:25:36.865 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - Extras packages 31 kB/s | 16 kB 00:00 2024-03-24T00:25:37.071 INFO:teuthology.orchestra.run.smithi072.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-24T00:25:37.071 INFO:teuthology.orchestra.run.smithi072.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-24T00:25:37.171 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-24T00:25:37.186 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-24T00:25:37.186 INFO:teuthology.orchestra.run.smithi072.stdout: Package Architecture Version Repository Size 2024-03-24T00:25:37.186 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-24T00:25:37.186 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-03-24T00:25:37.187 INFO:teuthology.orchestra.run.smithi072.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-24T00:25:37.187 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2024-03-24T00:25:37.187 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-24T00:25:37.187 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-24T00:25:37.187 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-24T00:25:37.187 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:25:37.187 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-03-24T00:25:37.187 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-24T00:25:37.187 INFO:teuthology.orchestra.run.smithi072.stdout:Install 4 Packages 2024-03-24T00:25:37.187 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:25:37.188 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 76 M 2024-03-24T00:25:37.188 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 123 M 2024-03-24T00:25:37.188 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-03-24T00:25:37.325 INFO:teuthology.orchestra.run.smithi072.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 480 kB/s | 44 kB 00:00 2024-03-24T00:25:37.800 INFO:teuthology.orchestra.run.smithi072.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 26 MB/s | 15 MB 00:00 2024-03-24T00:25:38.186 INFO:teuthology.orchestra.run.smithi067.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-03-24T00:25:38.359 INFO:teuthology.orchestra.run.smithi072.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 27 MB/s | 27 MB 00:01 2024-03-24T00:25:38.893 INFO:teuthology.orchestra.run.smithi072.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 20 MB/s | 33 MB 00:01 2024-03-24T00:25:38.893 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-03-24T00:25:38.893 INFO:teuthology.orchestra.run.smithi072.stdout:Total 44 MB/s | 76 MB 00:01 2024-03-24T00:25:39.560 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-03-24T00:25:39.939 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-03-24T00:25:39.939 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-03-24T00:25:40.935 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-03-24T00:25:40.936 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-03-24T00:25:43.342 INFO:teuthology.orchestra.run.smithi067.stdout:lab-extras 41 kB/s | 1.7 kB 00:00 2024-03-24T00:25:43.373 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-03-24T00:25:44.040 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-24T00:25:44.140 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T00:25:45.424 INFO:teuthology.orchestra.run.smithi067.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-24T00:25:45.424 INFO:teuthology.orchestra.run.smithi067.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-24T00:25:45.519 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-24T00:25:45.528 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-24T00:25:45.529 INFO:teuthology.orchestra.run.smithi067.stdout: Package Architecture Version Repository Size 2024-03-24T00:25:45.529 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-24T00:25:45.529 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-03-24T00:25:45.529 INFO:teuthology.orchestra.run.smithi067.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-24T00:25:45.529 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-03-24T00:25:45.529 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-24T00:25:45.529 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-24T00:25:45.529 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-24T00:25:45.529 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:25:45.529 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-03-24T00:25:45.529 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-24T00:25:45.529 INFO:teuthology.orchestra.run.smithi067.stdout:Install 4 Packages 2024-03-24T00:25:45.529 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:25:45.530 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 76 M 2024-03-24T00:25:45.530 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 123 M 2024-03-24T00:25:45.530 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-03-24T00:25:45.685 INFO:teuthology.orchestra.run.smithi067.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 413 kB/s | 44 kB 00:00 2024-03-24T00:25:46.904 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T00:25:47.083 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T00:25:48.143 INFO:teuthology.orchestra.run.smithi067.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 5.8 MB/s | 15 MB 00:02 2024-03-24T00:25:49.802 INFO:teuthology.orchestra.run.smithi067.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 6.7 MB/s | 27 MB 00:04 2024-03-24T00:25:50.110 INFO:teuthology.orchestra.run.smithi067.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 7.4 MB/s | 33 MB 00:04 2024-03-24T00:25:50.111 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-03-24T00:25:50.111 INFO:teuthology.orchestra.run.smithi067.stdout:Total 17 MB/s | 76 MB 00:04 2024-03-24T00:25:50.800 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-03-24T00:25:51.207 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-03-24T00:25:51.207 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-03-24T00:25:51.785 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T00:25:51.822 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T00:25:52.255 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-03-24T00:25:52.255 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-03-24T00:25:54.862 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-03-24T00:25:55.531 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-24T00:25:55.635 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T00:25:56.755 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T00:25:58.549 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T00:25:58.687 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T00:26:03.789 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T00:26:03.820 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T00:26:08.950 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T00:26:28.109 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T00:26:28.109 INFO:teuthology.orchestra.run.smithi072.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-24T00:26:28.109 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:26:28.153 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-24T00:26:28.598 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T00:26:28.598 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-24T00:26:28.598 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T00:26:28.598 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T00:26:29.022 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T00:26:29.022 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:26:29.022 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-03-24T00:26:29.022 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-24T00:26:29.022 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-24T00:26:29.022 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-24T00:26:29.022 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-24T00:26:29.022 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:26:29.022 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-24T00:26:29.454 DEBUG:teuthology.orchestra.run.smithi072:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T00:26:29.563 INFO:teuthology.orchestra.run.smithi072.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T00:26:29.564 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T00:26:29.564 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-24T00:26:29.564 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-24T00:26:29.564 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-24T00:26:29.564 INFO:teuthology.task.kernel:Installing distro kernel on client.1... 2024-03-24T00:26:29.564 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi072.front.sepia.ceph.com, path=None, version=distro) 2024-03-24T00:26:29.564 DEBUG:teuthology.orchestra.run.smithi072:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T00:26:29.643 INFO:teuthology.orchestra.run.smithi072.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T00:26:29.644 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T00:26:29.644 DEBUG:teuthology.orchestra.run.smithi072:> sudo rpm -qi grub2-tools 2024-03-24T00:26:29.676 INFO:teuthology.orchestra.run.smithi072.stdout:Name : grub2-tools 2024-03-24T00:26:29.677 INFO:teuthology.orchestra.run.smithi072.stdout:Epoch : 1 2024-03-24T00:26:29.677 INFO:teuthology.orchestra.run.smithi072.stdout:Version : 2.06 2024-03-24T00:26:29.677 INFO:teuthology.orchestra.run.smithi072.stdout:Release : 68.el9 2024-03-24T00:26:29.677 INFO:teuthology.orchestra.run.smithi072.stdout:Architecture: x86_64 2024-03-24T00:26:29.677 INFO:teuthology.orchestra.run.smithi072.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-24T00:26:29.677 INFO:teuthology.orchestra.run.smithi072.stdout:Group : Unspecified 2024-03-24T00:26:29.677 INFO:teuthology.orchestra.run.smithi072.stdout:Size : 8264254 2024-03-24T00:26:29.677 INFO:teuthology.orchestra.run.smithi072.stdout:License : GPLv3+ 2024-03-24T00:26:29.677 INFO:teuthology.orchestra.run.smithi072.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-24T00:26:29.677 INFO:teuthology.orchestra.run.smithi072.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-24T00:26:29.677 INFO:teuthology.orchestra.run.smithi072.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-24T00:26:29.677 INFO:teuthology.orchestra.run.smithi072.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-24T00:26:29.677 INFO:teuthology.orchestra.run.smithi072.stdout:Packager : builder@centos.org 2024-03-24T00:26:29.678 INFO:teuthology.orchestra.run.smithi072.stdout:Vendor : CentOS 2024-03-24T00:26:29.678 INFO:teuthology.orchestra.run.smithi072.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-24T00:26:29.678 INFO:teuthology.orchestra.run.smithi072.stdout:Summary : Support tools for GRUB. 2024-03-24T00:26:29.678 INFO:teuthology.orchestra.run.smithi072.stdout:Description : 2024-03-24T00:26:29.678 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:26:29.678 INFO:teuthology.orchestra.run.smithi072.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-24T00:26:29.678 INFO:teuthology.orchestra.run.smithi072.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-24T00:26:29.678 INFO:teuthology.orchestra.run.smithi072.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-24T00:26:29.678 INFO:teuthology.orchestra.run.smithi072.stdout:hardware devices. 2024-03-24T00:26:29.678 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:26:29.678 INFO:teuthology.orchestra.run.smithi072.stdout:This subpackage provides tools for support of all platforms. 2024-03-24T00:26:29.679 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-24T00:26:29.679 INFO:teuthology.task.kernel:Updating grub on smithi072 to boot 5.14.0-430.el9.x86_64 2024-03-24T00:26:29.679 DEBUG:teuthology.orchestra.run.smithi072:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-24T00:26:30.294 INFO:teuthology.orchestra.run.smithi072.stderr:Generating grub configuration file ... 2024-03-24T00:26:31.896 INFO:teuthology.orchestra.run.smithi072.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-24T00:26:31.921 INFO:teuthology.orchestra.run.smithi072.stderr:done 2024-03-24T00:26:31.924 DEBUG:teuthology.orchestra.run.smithi072:> mktemp 2024-03-24T00:26:31.975 INFO:teuthology.orchestra.run.smithi072.stdout:/tmp/tmp.D4fPGMWIle 2024-03-24T00:26:31.976 DEBUG:teuthology.orchestra.run.smithi072:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.D4fPGMWIle 2024-03-24T00:26:32.011 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 0666 /tmp/tmp.D4fPGMWIle 2024-03-24T00:26:32.152 DEBUG:teuthology.orchestra.remote:smithi072:/tmp/tmp.D4fPGMWIle is 6KB 2024-03-24T00:26:32.201 DEBUG:teuthology.orchestra.run.smithi072:> rm -fr /tmp/tmp.D4fPGMWIle 2024-03-24T00:26:32.215 DEBUG:teuthology.orchestra.run.smithi072:> sudo /bin/ls /boot/loader/entries || true 2024-03-24T00:26:32.286 INFO:teuthology.orchestra.run.smithi072.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-24T00:26:32.287 INFO:teuthology.orchestra.run.smithi072.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-24T00:26:32.287 INFO:teuthology.orchestra.run.smithi072.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-24T00:26:32.287 INFO:teuthology.orchestra.run.smithi072.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-24T00:26:32.288 DEBUG:teuthology.orchestra.run.smithi072:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-24T00:26:32.437 DEBUG:teuthology.orchestra.run.smithi072:> sudo shutdown -r now 2024-03-24T00:26:39.820 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T00:26:39.821 INFO:teuthology.orchestra.run.smithi067.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-24T00:26:39.821 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:26:39.866 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-24T00:26:40.239 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T00:26:40.239 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-24T00:26:40.240 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T00:26:40.240 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T00:26:40.618 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T00:26:40.618 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:26:40.618 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-03-24T00:26:40.618 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-24T00:26:40.618 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-24T00:26:40.618 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-24T00:26:40.618 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-24T00:26:40.618 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:26:40.618 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-24T00:26:40.958 DEBUG:teuthology.orchestra.run.smithi067:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T00:26:41.062 INFO:teuthology.orchestra.run.smithi067.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T00:26:41.062 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T00:26:41.063 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-24T00:26:41.063 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-24T00:26:41.063 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-24T00:26:41.063 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-03-24T00:26:41.063 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi067.front.sepia.ceph.com, path=None, version=distro) 2024-03-24T00:26:41.063 DEBUG:teuthology.orchestra.run.smithi067:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T00:26:41.146 INFO:teuthology.orchestra.run.smithi067.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T00:26:41.146 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T00:26:41.147 DEBUG:teuthology.orchestra.run.smithi067:> sudo rpm -qi grub2-tools 2024-03-24T00:26:41.195 INFO:teuthology.orchestra.run.smithi067.stdout:Name : grub2-tools 2024-03-24T00:26:41.195 INFO:teuthology.orchestra.run.smithi067.stdout:Epoch : 1 2024-03-24T00:26:41.195 INFO:teuthology.orchestra.run.smithi067.stdout:Version : 2.06 2024-03-24T00:26:41.195 INFO:teuthology.orchestra.run.smithi067.stdout:Release : 68.el9 2024-03-24T00:26:41.196 INFO:teuthology.orchestra.run.smithi067.stdout:Architecture: x86_64 2024-03-24T00:26:41.196 INFO:teuthology.orchestra.run.smithi067.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-24T00:26:41.196 INFO:teuthology.orchestra.run.smithi067.stdout:Group : Unspecified 2024-03-24T00:26:41.196 INFO:teuthology.orchestra.run.smithi067.stdout:Size : 8264254 2024-03-24T00:26:41.196 INFO:teuthology.orchestra.run.smithi067.stdout:License : GPLv3+ 2024-03-24T00:26:41.196 INFO:teuthology.orchestra.run.smithi067.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-24T00:26:41.196 INFO:teuthology.orchestra.run.smithi067.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-24T00:26:41.196 INFO:teuthology.orchestra.run.smithi067.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-24T00:26:41.196 INFO:teuthology.orchestra.run.smithi067.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-24T00:26:41.196 INFO:teuthology.orchestra.run.smithi067.stdout:Packager : builder@centos.org 2024-03-24T00:26:41.196 INFO:teuthology.orchestra.run.smithi067.stdout:Vendor : CentOS 2024-03-24T00:26:41.196 INFO:teuthology.orchestra.run.smithi067.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-24T00:26:41.196 INFO:teuthology.orchestra.run.smithi067.stdout:Summary : Support tools for GRUB. 2024-03-24T00:26:41.196 INFO:teuthology.orchestra.run.smithi067.stdout:Description : 2024-03-24T00:26:41.196 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:26:41.197 INFO:teuthology.orchestra.run.smithi067.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-24T00:26:41.197 INFO:teuthology.orchestra.run.smithi067.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-24T00:26:41.197 INFO:teuthology.orchestra.run.smithi067.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-24T00:26:41.197 INFO:teuthology.orchestra.run.smithi067.stdout:hardware devices. 2024-03-24T00:26:41.197 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:26:41.197 INFO:teuthology.orchestra.run.smithi067.stdout:This subpackage provides tools for support of all platforms. 2024-03-24T00:26:41.198 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-24T00:26:41.198 INFO:teuthology.task.kernel:Updating grub on smithi067 to boot 5.14.0-430.el9.x86_64 2024-03-24T00:26:41.198 DEBUG:teuthology.orchestra.run.smithi067:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-24T00:26:41.756 INFO:teuthology.orchestra.run.smithi067.stderr:Generating grub configuration file ... 2024-03-24T00:26:43.182 INFO:teuthology.orchestra.run.smithi067.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-24T00:26:43.210 INFO:teuthology.orchestra.run.smithi067.stderr:done 2024-03-24T00:26:43.213 DEBUG:teuthology.orchestra.run.smithi067:> mktemp 2024-03-24T00:26:43.229 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.iO3ux2BjKI 2024-03-24T00:26:43.230 DEBUG:teuthology.orchestra.run.smithi067:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.iO3ux2BjKI 2024-03-24T00:26:43.302 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.iO3ux2BjKI 2024-03-24T00:26:43.454 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.iO3ux2BjKI is 6KB 2024-03-24T00:26:43.503 DEBUG:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.iO3ux2BjKI 2024-03-24T00:26:43.518 DEBUG:teuthology.orchestra.run.smithi067:> sudo /bin/ls /boot/loader/entries || true 2024-03-24T00:26:43.588 INFO:teuthology.orchestra.run.smithi067.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-24T00:26:43.588 INFO:teuthology.orchestra.run.smithi067.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-24T00:26:43.588 INFO:teuthology.orchestra.run.smithi067.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-24T00:26:43.589 INFO:teuthology.orchestra.run.smithi067.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-24T00:26:43.590 DEBUG:teuthology.orchestra.run.smithi067:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-24T00:26:43.746 DEBUG:teuthology.orchestra.run.smithi067:> sudo shutdown -r now 2024-03-24T00:27:02.459 INFO:teuthology.task.kernel:Checking client client.1 for new kernel version... 2024-03-24T00:27:02.460 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2024-03-24T00:27:02.460 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T00:27:13.759 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-24T00:27:13.759 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-03-24T00:27:13.760 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T00:28:02.495 DEBUG:teuthology.orchestra.remote:timed out 2024-03-24T00:28:11.500 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2024-03-24T00:28:11.501 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T00:28:13.765 DEBUG:teuthology.orchestra.remote:timed out 2024-03-24T00:28:22.774 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-03-24T00:28:22.775 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T00:28:27.412 DEBUG:teuthology.orchestra.run.smithi072:> true 2024-03-24T00:28:27.990 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi072.front.sepia.ceph.com' 2024-03-24T00:28:27.990 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "5.14.0-430.el9.x86_64"... 2024-03-24T00:28:27.991 DEBUG:teuthology.orchestra.run.smithi072:> uname -r 2024-03-24T00:28:28.004 INFO:teuthology.orchestra.run.smithi072.stdout:5.14.0-430.el9.x86_64 2024-03-24T00:28:28.005 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-24T00:28:28.005 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-24T00:28:28.005 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-24T00:28:29.005 DEBUG:teuthology.parallel:result is None 2024-03-24T00:28:38.346 DEBUG:teuthology.orchestra.run.smithi067:> true 2024-03-24T00:28:38.908 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-03-24T00:28:38.909 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-430.el9.x86_64"... 2024-03-24T00:28:38.909 DEBUG:teuthology.orchestra.run.smithi067:> uname -r 2024-03-24T00:28:38.923 INFO:teuthology.orchestra.run.smithi067.stdout:5.14.0-430.el9.x86_64 2024-03-24T00:28:38.923 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-24T00:28:38.923 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-24T00:28:38.923 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-24T00:28:39.924 DEBUG:teuthology.parallel:result is None 2024-03-24T00:28:39.924 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-24T00:28:39.932 INFO:teuthology.task.internal:Creating test directory... 2024-03-24T00:28:39.932 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T00:28:39.934 DEBUG:teuthology.orchestra.run.smithi072:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T00:28:39.953 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-24T00:28:39.960 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-24T00:28:39.967 INFO:teuthology.task.internal:Creating archive directory... 2024-03-24T00:28:39.968 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T00:28:39.993 DEBUG:teuthology.orchestra.run.smithi072:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T00:28:40.024 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-24T00:28:40.031 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-24T00:28:40.031 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T00:28:40.056 DEBUG:teuthology.orchestra.run.smithi072:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T00:28:40.156 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T00:28:40.170 INFO:teuthology.orchestra.run.smithi072.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T00:28:40.188 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T00:28:40.223 INFO:teuthology.orchestra.run.smithi072.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T00:28:40.224 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-24T00:28:40.232 INFO:teuthology.task.internal:Configuring sudo... 2024-03-24T00:28:40.232 DEBUG:teuthology.orchestra.run.smithi067:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T00:28:40.234 DEBUG:teuthology.orchestra.run.smithi072:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T00:28:40.264 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-24T00:28:40.273 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-24T00:28:40.273 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T00:28:40.305 DEBUG:teuthology.orchestra.run.smithi072:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T00:28:40.320 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T00:28:40.400 DEBUG:teuthology.orchestra.run.smithi067:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T00:28:40.522 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T00:28:40.561 DEBUG:teuthology.orchestra.run.smithi067:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T00:28:40.609 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:28:40.610 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T00:28:40.682 DEBUG:teuthology.orchestra.run.smithi072:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T00:28:40.728 DEBUG:teuthology.orchestra.run.smithi072:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T00:28:40.789 DEBUG:teuthology.orchestra.run.smithi072:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T00:28:40.832 DEBUG:teuthology.orchestra.run.smithi072:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T00:28:40.876 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:28:40.877 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T00:28:40.945 DEBUG:teuthology.orchestra.run.smithi067:> sudo service rsyslog restart 2024-03-24T00:28:40.948 DEBUG:teuthology.orchestra.run.smithi072:> sudo service rsyslog restart 2024-03-24T00:28:40.998 INFO:teuthology.orchestra.run.smithi067.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T00:28:41.025 INFO:teuthology.orchestra.run.smithi072.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T00:28:41.465 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-24T00:28:41.473 INFO:teuthology.task.internal:Starting timer... 2024-03-24T00:28:41.473 INFO:teuthology.run_tasks:Running task pcp... 2024-03-24T00:28:41.483 INFO:teuthology.run_tasks:Running task selinux... 2024-03-24T00:28:41.492 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-24T00:28:41.493 DEBUG:teuthology.orchestra.run.smithi067:> sudo service auditd rotate 2024-03-24T00:28:41.558 INFO:teuthology.orchestra.run.smithi067.stdout:Rotating logs: 2024-03-24T00:28:41.559 DEBUG:teuthology.orchestra.run.smithi072:> sudo service auditd rotate 2024-03-24T00:28:41.660 INFO:teuthology.orchestra.run.smithi072.stdout:Rotating logs: 2024-03-24T00:28:41.662 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-24T00:28:41.662 DEBUG:teuthology.orchestra.run.smithi067:> /usr/sbin/getenforce 2024-03-24T00:28:41.678 INFO:teuthology.orchestra.run.smithi067.stdout:Permissive 2024-03-24T00:28:41.679 DEBUG:teuthology.orchestra.run.smithi072:> /usr/sbin/getenforce 2024-03-24T00:28:41.704 INFO:teuthology.orchestra.run.smithi072.stdout:Permissive 2024-03-24T00:28:41.704 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi067.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi072.front.sepia.ceph.com': 'permissive'} 2024-03-24T00:28:41.705 DEBUG:teuthology.orchestra.run.smithi067:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T00:28:41.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:28:41.749 DEBUG:teuthology.orchestra.run.smithi072:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T00:28:41.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:28:41.778 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-24T00:28:41.779 DEBUG:teuthology.orchestra.run.smithi067:> sudo /usr/sbin/setenforce permissive 2024-03-24T00:28:41.822 DEBUG:teuthology.orchestra.run.smithi072:> sudo /usr/sbin/setenforce permissive 2024-03-24T00:28:41.866 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-24T00:28:41.878 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-24T00:28:41.885 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-24T00:28:42.032 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-24T00:28:42.048 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-24T00:28:42.050 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi067.front.sepia.ceph.com,smithi072.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-24T00:35:11.425 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi067.front.sepia.ceph.com'), Remote(name='ubuntu@smithi072.front.sepia.ceph.com')] 2024-03-24T00:35:11.426 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-03-24T00:35:11.427 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T00:35:11.517 DEBUG:teuthology.orchestra.run.smithi067:> true 2024-03-24T00:35:11.595 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-03-24T00:35:11.596 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2024-03-24T00:35:11.596 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T00:35:11.686 DEBUG:teuthology.orchestra.run.smithi072:> true 2024-03-24T00:35:11.761 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi072.front.sepia.ceph.com' 2024-03-24T00:35:11.761 INFO:teuthology.run_tasks:Running task clock... 2024-03-24T00:35:11.773 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-24T00:35:11.773 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T00:35:11.774 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T00:35:11.777 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T00:35:11.778 DEBUG:teuthology.orchestra.run.smithi072:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T00:35:11.804 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T00:35:11.821 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T00:35:11.831 INFO:teuthology.orchestra.run.smithi072.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T00:35:11.847 INFO:teuthology.orchestra.run.smithi072.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T00:35:11.850 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: ntpd: command not found 2024-03-24T00:35:11.864 INFO:teuthology.orchestra.run.smithi067.stdout:506 Cannot talk to daemon 2024-03-24T00:35:11.876 INFO:teuthology.orchestra.run.smithi072.stderr:sudo: ntpd: command not found 2024-03-24T00:35:11.880 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T00:35:11.889 INFO:teuthology.orchestra.run.smithi072.stdout:506 Cannot talk to daemon 2024-03-24T00:35:11.896 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T00:35:11.905 INFO:teuthology.orchestra.run.smithi072.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T00:35:11.922 INFO:teuthology.orchestra.run.smithi072.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T00:35:11.951 INFO:teuthology.orchestra.run.smithi067.stderr:bash: line 1: ntpq: command not found 2024-03-24T00:35:11.955 INFO:teuthology.orchestra.run.smithi067.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T00:35:11.955 INFO:teuthology.orchestra.run.smithi067.stdout:=============================================================================== 2024-03-24T00:35:11.955 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T00:35:11.955 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T00:35:11.955 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T00:35:11.955 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T00:35:11.977 INFO:teuthology.orchestra.run.smithi072.stderr:bash: line 1: ntpq: command not found 2024-03-24T00:35:11.980 INFO:teuthology.orchestra.run.smithi072.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T00:35:11.980 INFO:teuthology.orchestra.run.smithi072.stdout:=============================================================================== 2024-03-24T00:35:11.980 INFO:teuthology.orchestra.run.smithi072.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T00:35:11.980 INFO:teuthology.orchestra.run.smithi072.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T00:35:11.980 INFO:teuthology.orchestra.run.smithi072.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T00:35:11.980 INFO:teuthology.orchestra.run.smithi072.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T00:35:11.981 INFO:teuthology.run_tasks:Running task install... 2024-03-24T00:35:11.994 DEBUG:teuthology.task.install:project ceph 2024-03-24T00:35:11.995 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'}} 2024-03-24T00:35:11.995 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'} 2024-03-24T00:35:11.995 INFO:teuthology.task.install:Using flavor: default 2024-03-24T00:35:12.001 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-24T00:35:12.001 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-24T00:35:12.001 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'tag': None, 'wait_for_package': False} 2024-03-24T00:35:12.002 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T00:35:12.004 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'tag': None, 'wait_for_package': False} 2024-03-24T00:35:12.005 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T00:35:12.223 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/ 2024-03-24T00:35:12.223 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2145.g3be9a5c9 2024-03-24T00:35:12.227 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/ 2024-03-24T00:35:12.227 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2145.g3be9a5c9 2024-03-24T00:35:12.402 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-24T00:35:12.403 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:35:12.403 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-24T00:35:12.408 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-24T00:35:12.408 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:35:12.408 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-24T00:35:12.433 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-24T00:35:12.434 DEBUG:teuthology.orchestra.run.smithi067:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3be9a5c9bc793e11e2800a8c0c696e8b46742033/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-24T00:35:12.437 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-24T00:35:12.437 DEBUG:teuthology.orchestra.run.smithi072:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3be9a5c9bc793e11e2800a8c0c696e8b46742033/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-24T00:35:12.503 DEBUG:teuthology.orchestra.run.smithi067:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-24T00:35:12.505 DEBUG:teuthology.orchestra.run.smithi072:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-24T00:35:12.583 DEBUG:teuthology.orchestra.run.smithi067:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-24T00:35:12.584 DEBUG:teuthology.orchestra.run.smithi072:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-24T00:35:12.650 INFO:teuthology.orchestra.run.smithi067.stdout:check_obsoletes = 1 2024-03-24T00:35:12.652 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum clean all 2024-03-24T00:35:12.653 INFO:teuthology.orchestra.run.smithi072.stdout:check_obsoletes = 1 2024-03-24T00:35:12.653 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum clean all 2024-03-24T00:35:12.893 INFO:teuthology.orchestra.run.smithi072.stdout:74 files removed 2024-03-24T00:35:12.895 INFO:teuthology.orchestra.run.smithi067.stdout:74 files removed 2024-03-24T00:35:12.920 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-24T00:35:12.922 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-24T00:35:13.634 INFO:teuthology.orchestra.run.smithi067.stdout:ceph packages for x86_64 205 kB/s | 83 kB 00:00 2024-03-24T00:35:13.634 INFO:teuthology.orchestra.run.smithi072.stdout:ceph packages for x86_64 199 kB/s | 83 kB 00:00 2024-03-24T00:35:14.002 INFO:teuthology.orchestra.run.smithi072.stdout:ceph noarch packages 32 kB/s | 10 kB 00:00 2024-03-24T00:35:14.018 INFO:teuthology.orchestra.run.smithi067.stdout:ceph noarch packages 30 kB/s | 10 kB 00:00 2024-03-24T00:35:14.420 INFO:teuthology.orchestra.run.smithi072.stdout:ceph source packages 5.0 kB/s | 1.9 kB 00:00 2024-03-24T00:35:14.443 INFO:teuthology.orchestra.run.smithi067.stdout:ceph source packages 4.9 kB/s | 1.9 kB 00:00 2024-03-24T00:35:15.020 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-24T00:35:15.114 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS-9 - AppStream 10 MB/s | 6.7 MB 00:00 2024-03-24T00:35:16.525 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS-9 - BaseOS 7.0 MB/s | 2.0 MB 00:00 2024-03-24T00:35:16.529 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS-9 - BaseOS 5.6 MB/s | 2.0 MB 00:00 2024-03-24T00:35:17.476 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-03-24T00:35:17.731 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - BaseOS 9.3 MB/s | 8.0 MB 00:00 2024-03-24T00:35:19.320 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-03-24T00:35:20.746 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - AppStream 9.7 MB/s | 19 MB 00:01 2024-03-24T00:35:25.040 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - CRB 2.7 MB/s | 6.1 MB 00:02 2024-03-24T00:35:25.047 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - CRB 9.9 MB/s | 6.1 MB 00:00 2024-03-24T00:35:26.434 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - Extras packages 76 kB/s | 16 kB 00:00 2024-03-24T00:35:27.447 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 16 kB 00:01 2024-03-24T00:35:29.353 INFO:teuthology.orchestra.run.smithi072.stdout:Extra Packages for Enterprise Linux 7.4 MB/s | 21 MB 00:02 2024-03-24T00:35:34.196 INFO:teuthology.orchestra.run.smithi072.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-24T00:35:36.611 INFO:teuthology.orchestra.run.smithi072.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T00:35:36.612 INFO:teuthology.orchestra.run.smithi072.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T00:35:36.626 INFO:teuthology.orchestra.run.smithi072.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-24T00:35:36.627 INFO:teuthology.orchestra.run.smithi072.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-24T00:35:36.628 INFO:teuthology.orchestra.run.smithi072.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-24T00:35:36.629 INFO:teuthology.orchestra.run.smithi072.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-24T00:35:36.632 INFO:teuthology.orchestra.run.smithi072.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-24T00:35:36.634 INFO:teuthology.orchestra.run.smithi072.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-24T00:35:36.636 INFO:teuthology.orchestra.run.smithi072.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-24T00:35:36.715 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-24T00:35:36.742 INFO:teuthology.orchestra.run.smithi072.stdout:============================================================================================ 2024-03-24T00:35:36.742 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2024-03-24T00:35:36.742 INFO:teuthology.orchestra.run.smithi072.stdout:============================================================================================ 2024-03-24T00:35:36.742 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-03-24T00:35:36.742 INFO:teuthology.orchestra.run.smithi072.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-24T00:35:36.742 INFO:teuthology.orchestra.run.smithi072.stdout: ceph x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 6.6 k 2024-03-24T00:35:36.742 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-base x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.4 M 2024-03-24T00:35:36.742 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 890 k 2024-03-24T00:35:36.743 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 145 k 2024-03-24T00:35:36.743 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.5 M 2024-03-24T00:35:36.743 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 139 k 2024-03-24T00:35:36.743 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 1.8 M 2024-03-24T00:35:36.743 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 7.4 M 2024-03-24T00:35:36.743 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-rook noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 49 k 2024-03-24T00:35:36.743 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-radosgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 9.9 M 2024-03-24T00:35:36.743 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-test x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 48 M 2024-03-24T00:35:36.743 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-volume noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 264 k 2024-03-24T00:35:36.743 INFO:teuthology.orchestra.run.smithi072.stdout: cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 748 k 2024-03-24T00:35:36.743 INFO:teuthology.orchestra.run.smithi072.stdout: cephfs-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 220 k 2024-03-24T00:35:36.743 INFO:teuthology.orchestra.run.smithi072.stdout: cephfs-shell x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 27 k 2024-03-24T00:35:36.743 INFO:teuthology.orchestra.run.smithi072.stdout: cephfs-top noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 18 k 2024-03-24T00:35:36.743 INFO:teuthology.orchestra.run.smithi072.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-24T00:35:36.743 INFO:teuthology.orchestra.run.smithi072.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-24T00:35:36.744 INFO:teuthology.orchestra.run.smithi072.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-24T00:35:36.744 INFO:teuthology.orchestra.run.smithi072.stdout: libcephfs-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 31 k 2024-03-24T00:35:36.744 INFO:teuthology.orchestra.run.smithi072.stdout: libcephfs2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 746 k 2024-03-24T00:35:36.744 INFO:teuthology.orchestra.run.smithi072.stdout: librados-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 127 k 2024-03-24T00:35:36.744 INFO:teuthology.orchestra.run.smithi072.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-24T00:35:36.744 INFO:teuthology.orchestra.run.smithi072.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-24T00:35:36.744 INFO:teuthology.orchestra.run.smithi072.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-24T00:35:36.744 INFO:teuthology.orchestra.run.smithi072.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-24T00:35:36.744 INFO:teuthology.orchestra.run.smithi072.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-24T00:35:36.744 INFO:teuthology.orchestra.run.smithi072.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-24T00:35:36.744 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cephfs x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 162 k 2024-03-24T00:35:36.744 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rados x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 321 k 2024-03-24T00:35:36.744 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 298 k 2024-03-24T00:35:36.744 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 99 k 2024-03-24T00:35:36.744 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 85 k 2024-03-24T00:35:36.745 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-24T00:35:36.745 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-nbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 170 k 2024-03-24T00:35:36.745 INFO:teuthology.orchestra.run.smithi072.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-24T00:35:36.745 INFO:teuthology.orchestra.run.smithi072.stdout:Upgrading: 2024-03-24T00:35:36.745 INFO:teuthology.orchestra.run.smithi072.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-24T00:35:36.745 INFO:teuthology.orchestra.run.smithi072.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-24T00:35:36.745 INFO:teuthology.orchestra.run.smithi072.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-24T00:35:36.745 INFO:teuthology.orchestra.run.smithi072.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-24T00:35:36.745 INFO:teuthology.orchestra.run.smithi072.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-24T00:35:36.745 INFO:teuthology.orchestra.run.smithi072.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-24T00:35:36.745 INFO:teuthology.orchestra.run.smithi072.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-24T00:35:36.745 INFO:teuthology.orchestra.run.smithi072.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-24T00:35:36.745 INFO:teuthology.orchestra.run.smithi072.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-24T00:35:36.745 INFO:teuthology.orchestra.run.smithi072.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-24T00:35:36.746 INFO:teuthology.orchestra.run.smithi072.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-24T00:35:36.746 INFO:teuthology.orchestra.run.smithi072.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-24T00:35:36.746 INFO:teuthology.orchestra.run.smithi072.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-24T00:35:36.746 INFO:teuthology.orchestra.run.smithi072.stdout: librados2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.4 M 2024-03-24T00:35:36.746 INFO:teuthology.orchestra.run.smithi072.stdout: librbd1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-24T00:35:36.746 INFO:teuthology.orchestra.run.smithi072.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-24T00:35:36.746 INFO:teuthology.orchestra.run.smithi072.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-24T00:35:36.746 INFO:teuthology.orchestra.run.smithi072.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-24T00:35:36.746 INFO:teuthology.orchestra.run.smithi072.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-24T00:35:36.746 INFO:teuthology.orchestra.run.smithi072.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-24T00:35:36.746 INFO:teuthology.orchestra.run.smithi072.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-24T00:35:36.746 INFO:teuthology.orchestra.run.smithi072.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-24T00:35:36.746 INFO:teuthology.orchestra.run.smithi072.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-24T00:35:36.746 INFO:teuthology.orchestra.run.smithi072.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-24T00:35:36.747 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2024-03-24T00:35:36.747 INFO:teuthology.orchestra.run.smithi072.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-24T00:35:36.747 INFO:teuthology.orchestra.run.smithi072.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-24T00:35:36.747 INFO:teuthology.orchestra.run.smithi072.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-24T00:35:36.747 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 20 M 2024-03-24T00:35:36.747 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 34 k 2024-03-24T00:35:36.747 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mds x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 2.3 M 2024-03-24T00:35:36.747 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 248 k 2024-03-24T00:35:36.747 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mon x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 4.7 M 2024-03-24T00:35:36.747 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-osd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 17 M 2024-03-24T00:35:36.747 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 16 k 2024-03-24T00:35:36.747 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-selinux x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 25 k 2024-03-24T00:35:36.747 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-24T00:35:36.747 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-24T00:35:36.747 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-24T00:35:36.748 INFO:teuthology.orchestra.run.smithi072.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-24T00:35:36.748 INFO:teuthology.orchestra.run.smithi072.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-24T00:35:36.748 INFO:teuthology.orchestra.run.smithi072.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-24T00:35:36.748 INFO:teuthology.orchestra.run.smithi072.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-24T00:35:36.748 INFO:teuthology.orchestra.run.smithi072.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-24T00:35:36.748 INFO:teuthology.orchestra.run.smithi072.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-24T00:35:36.748 INFO:teuthology.orchestra.run.smithi072.stdout: libcephsqlite x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 163 k 2024-03-24T00:35:36.748 INFO:teuthology.orchestra.run.smithi072.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-24T00:35:36.748 INFO:teuthology.orchestra.run.smithi072.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-24T00:35:36.748 INFO:teuthology.orchestra.run.smithi072.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-24T00:35:36.748 INFO:teuthology.orchestra.run.smithi072.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-24T00:35:36.748 INFO:teuthology.orchestra.run.smithi072.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-24T00:35:36.748 INFO:teuthology.orchestra.run.smithi072.stdout: libradosstriper1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 501 k 2024-03-24T00:35:36.748 INFO:teuthology.orchestra.run.smithi072.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-24T00:35:36.749 INFO:teuthology.orchestra.run.smithi072.stdout: librgw2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.1 M 2024-03-24T00:35:36.749 INFO:teuthology.orchestra.run.smithi072.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-24T00:35:36.749 INFO:teuthology.orchestra.run.smithi072.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-24T00:35:36.749 INFO:teuthology.orchestra.run.smithi072.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-24T00:35:36.749 INFO:teuthology.orchestra.run.smithi072.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-24T00:35:36.749 INFO:teuthology.orchestra.run.smithi072.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-24T00:35:36.749 INFO:teuthology.orchestra.run.smithi072.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-24T00:35:36.749 INFO:teuthology.orchestra.run.smithi072.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-24T00:35:36.749 INFO:teuthology.orchestra.run.smithi072.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-24T00:35:36.749 INFO:teuthology.orchestra.run.smithi072.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-24T00:35:36.749 INFO:teuthology.orchestra.run.smithi072.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-24T00:35:36.749 INFO:teuthology.orchestra.run.smithi072.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-24T00:35:36.749 INFO:teuthology.orchestra.run.smithi072.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-24T00:35:36.749 INFO:teuthology.orchestra.run.smithi072.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-24T00:35:36.749 INFO:teuthology.orchestra.run.smithi072.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-24T00:35:36.750 INFO:teuthology.orchestra.run.smithi072.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-24T00:35:36.750 INFO:teuthology.orchestra.run.smithi072.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-24T00:35:36.750 INFO:teuthology.orchestra.run.smithi072.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-24T00:35:36.750 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-24T00:35:36.750 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ceph-argparse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 45 k 2024-03-24T00:35:36.750 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 131 k 2024-03-24T00:35:36.750 INFO:teuthology.orchestra.run.smithi072.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-24T00:35:36.750 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-24T00:35:36.750 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-24T00:35:36.750 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-24T00:35:36.750 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-24T00:35:36.754 INFO:teuthology.orchestra.run.smithi072.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-24T00:35:36.754 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-24T00:35:36.754 INFO:teuthology.orchestra.run.smithi072.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-24T00:35:36.754 INFO:teuthology.orchestra.run.smithi072.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-24T00:35:36.754 INFO:teuthology.orchestra.run.smithi072.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-24T00:35:36.755 INFO:teuthology.orchestra.run.smithi072.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-24T00:35:36.755 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-24T00:35:36.755 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-24T00:35:36.755 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-24T00:35:36.755 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-24T00:35:36.755 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-24T00:35:36.755 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-24T00:35:36.755 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-24T00:35:36.755 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-24T00:35:36.755 INFO:teuthology.orchestra.run.smithi072.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-24T00:35:36.755 INFO:teuthology.orchestra.run.smithi072.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-24T00:35:36.755 INFO:teuthology.orchestra.run.smithi072.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-24T00:35:36.755 INFO:teuthology.orchestra.run.smithi072.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-24T00:35:36.755 INFO:teuthology.orchestra.run.smithi072.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-24T00:35:36.756 INFO:teuthology.orchestra.run.smithi072.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-24T00:35:36.756 INFO:teuthology.orchestra.run.smithi072.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-24T00:35:36.756 INFO:teuthology.orchestra.run.smithi072.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-24T00:35:36.756 INFO:teuthology.orchestra.run.smithi072.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-24T00:35:36.756 INFO:teuthology.orchestra.run.smithi072.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-24T00:35:36.756 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-24T00:35:36.756 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-24T00:35:36.756 INFO:teuthology.orchestra.run.smithi072.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-24T00:35:36.756 INFO:teuthology.orchestra.run.smithi072.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-24T00:35:36.756 INFO:teuthology.orchestra.run.smithi072.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-24T00:35:36.756 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-24T00:35:36.756 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-24T00:35:36.756 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-24T00:35:36.756 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-24T00:35:36.756 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-24T00:35:36.757 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-24T00:35:36.757 INFO:teuthology.orchestra.run.smithi072.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-24T00:35:36.757 INFO:teuthology.orchestra.run.smithi072.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-24T00:35:36.757 INFO:teuthology.orchestra.run.smithi072.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-24T00:35:36.757 INFO:teuthology.orchestra.run.smithi072.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-24T00:35:36.757 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-24T00:35:36.757 INFO:teuthology.orchestra.run.smithi072.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-24T00:35:36.757 INFO:teuthology.orchestra.run.smithi072.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-24T00:35:36.757 INFO:teuthology.orchestra.run.smithi072.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-24T00:35:36.757 INFO:teuthology.orchestra.run.smithi072.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-24T00:35:36.757 INFO:teuthology.orchestra.run.smithi072.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-24T00:35:36.757 INFO:teuthology.orchestra.run.smithi072.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-24T00:35:36.757 INFO:teuthology.orchestra.run.smithi072.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-24T00:35:36.757 INFO:teuthology.orchestra.run.smithi072.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-24T00:35:36.757 INFO:teuthology.orchestra.run.smithi072.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-24T00:35:36.758 INFO:teuthology.orchestra.run.smithi072.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-24T00:35:36.758 INFO:teuthology.orchestra.run.smithi072.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-24T00:35:36.758 INFO:teuthology.orchestra.run.smithi072.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-24T00:35:36.758 INFO:teuthology.orchestra.run.smithi072.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-24T00:35:36.758 INFO:teuthology.orchestra.run.smithi072.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-24T00:35:36.758 INFO:teuthology.orchestra.run.smithi072.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-24T00:35:36.758 INFO:teuthology.orchestra.run.smithi072.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-24T00:35:36.758 INFO:teuthology.orchestra.run.smithi072.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-24T00:35:36.758 INFO:teuthology.orchestra.run.smithi072.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-24T00:35:36.758 INFO:teuthology.orchestra.run.smithi072.stdout:Installing weak dependencies: 2024-03-24T00:35:36.758 INFO:teuthology.orchestra.run.smithi072.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-24T00:35:36.758 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:35:36.758 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-03-24T00:35:36.758 INFO:teuthology.orchestra.run.smithi072.stdout:============================================================================================ 2024-03-24T00:35:36.758 INFO:teuthology.orchestra.run.smithi072.stdout:Install 150 Packages 2024-03-24T00:35:36.759 INFO:teuthology.orchestra.run.smithi072.stdout:Upgrade 24 Packages 2024-03-24T00:35:36.759 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:35:36.759 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 224 M 2024-03-24T00:35:36.759 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-03-24T00:35:37.242 INFO:teuthology.orchestra.run.smithi072.stdout:(1/174): ceph-19.0.0-2145.g3be9a5c9.el9.x86_64. 41 kB/s | 6.6 kB 00:00 2024-03-24T00:35:37.442 INFO:teuthology.orchestra.run.smithi072.stdout:(2/174): ceph-fuse-19.0.0-2145.g3be9a5c9.el9.x8 4.3 MB/s | 890 kB 00:00 2024-03-24T00:35:37.493 INFO:teuthology.orchestra.run.smithi072.stdout:(3/174): ceph-immutable-object-cache-19.0.0-214 2.8 MB/s | 145 kB 00:00 2024-03-24T00:35:37.810 INFO:teuthology.orchestra.run.smithi072.stdout:(4/174): ceph-mds-19.0.0-2145.g3be9a5c9.el9.x86 7.3 MB/s | 2.3 MB 00:00 2024-03-24T00:35:37.977 INFO:teuthology.orchestra.run.smithi072.stdout:(5/174): ceph-base-19.0.0-2145.g3be9a5c9.el9.x8 6.0 MB/s | 5.4 MB 00:00 2024-03-24T00:35:38.028 INFO:teuthology.orchestra.run.smithi072.stdout:(6/174): ceph-mgr-19.0.0-2145.g3be9a5c9.el9.x86 6.9 MB/s | 1.5 MB 00:00 2024-03-24T00:35:38.487 INFO:teuthology.orchestra.run.smithi072.stdout:(7/174): ceph-mon-19.0.0-2145.g3be9a5c9.el9.x86 9.2 MB/s | 4.7 MB 00:00 2024-03-24T00:35:39.653 INFO:teuthology.orchestra.run.smithi072.stdout:(8/174): ceph-common-19.0.0-2145.g3be9a5c9.el9. 8.0 MB/s | 20 MB 00:02 2024-03-24T00:35:39.829 INFO:teuthology.orchestra.run.smithi072.stdout:(9/174): ceph-radosgw-19.0.0-2145.g3be9a5c9.el9 7.3 MB/s | 9.9 MB 00:01 2024-03-24T00:35:39.855 INFO:teuthology.orchestra.run.smithi072.stdout:(10/174): ceph-selinux-19.0.0-2145.g3be9a5c9.el 124 kB/s | 25 kB 00:00 2024-03-24T00:35:39.930 INFO:teuthology.orchestra.run.smithi072.stdout:(11/174): cephfs-mirror-19.0.0-2145.g3be9a5c9.e 2.9 MB/s | 220 kB 00:00 2024-03-24T00:35:40.241 INFO:teuthology.orchestra.run.smithi072.stdout:(12/174): ceph-osd-19.0.0-2145.g3be9a5c9.el9.x8 7.6 MB/s | 17 MB 00:02 2024-03-24T00:35:40.266 INFO:teuthology.orchestra.run.smithi072.stdout:(13/174): cephfs-shell-19.0.0-2145.g3be9a5c9.el 81 kB/s | 27 kB 00:00 2024-03-24T00:35:40.292 INFO:teuthology.orchestra.run.smithi072.stdout:(14/174): libcephfs-devel-19.0.0-2145.g3be9a5c9 620 kB/s | 31 kB 00:00 2024-03-24T00:35:40.342 INFO:teuthology.orchestra.run.smithi072.stdout:(15/174): libcephsqlite-19.0.0-2145.g3be9a5c9.e 3.2 MB/s | 163 kB 00:00 2024-03-24T00:35:40.401 INFO:teuthology.orchestra.run.smithi072.stdout:(16/174): librados-devel-19.0.0-2145.g3be9a5c9. 2.1 MB/s | 127 kB 00:00 2024-03-24T00:35:40.443 INFO:teuthology.orchestra.run.smithi072.stdout:(17/174): libcephfs2-19.0.0-2145.g3be9a5c9.el9. 4.1 MB/s | 746 kB 00:00 2024-03-24T00:35:40.511 INFO:teuthology.orchestra.run.smithi072.stdout:(18/174): libradosstriper1-19.0.0-2145.g3be9a5c 4.5 MB/s | 501 kB 00:00 2024-03-24T00:35:40.586 INFO:teuthology.orchestra.run.smithi072.stdout:(19/174): python3-ceph-argparse-19.0.0-2145.g3b 601 kB/s | 45 kB 00:00 2024-03-24T00:35:40.670 INFO:teuthology.orchestra.run.smithi072.stdout:(20/174): python3-ceph-common-19.0.0-2145.g3be9 1.5 MB/s | 131 kB 00:00 2024-03-24T00:35:40.754 INFO:teuthology.orchestra.run.smithi072.stdout:(21/174): python3-cephfs-19.0.0-2145.g3be9a5c9. 1.9 MB/s | 162 kB 00:00 2024-03-24T00:35:40.838 INFO:teuthology.orchestra.run.smithi072.stdout:(22/174): python3-rados-19.0.0-2145.g3be9a5c9.e 3.8 MB/s | 321 kB 00:00 2024-03-24T00:35:40.905 INFO:teuthology.orchestra.run.smithi072.stdout:(23/174): python3-rbd-19.0.0-2145.g3be9a5c9.el9 4.4 MB/s | 298 kB 00:00 2024-03-24T00:35:41.005 INFO:teuthology.orchestra.run.smithi072.stdout:(24/174): librgw2-19.0.0-2145.g3be9a5c9.el9.x86 9.2 MB/s | 5.1 MB 00:00 2024-03-24T00:35:41.031 INFO:teuthology.orchestra.run.smithi072.stdout:(25/174): python3-rgw-19.0.0-2145.g3be9a5c9.el9 790 kB/s | 99 kB 00:00 2024-03-24T00:35:41.073 INFO:teuthology.orchestra.run.smithi072.stdout:(26/174): rbd-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.2 MB/s | 85 kB 00:00 2024-03-24T00:35:41.167 INFO:teuthology.orchestra.run.smithi072.stdout:(27/174): rbd-nbd-19.0.0-2145.g3be9a5c9.el9.x86 1.8 MB/s | 170 kB 00:00 2024-03-24T00:35:41.251 INFO:teuthology.orchestra.run.smithi072.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2145.g 405 kB/s | 34 kB 00:00 2024-03-24T00:35:41.335 INFO:teuthology.orchestra.run.smithi072.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2145.g3be9a5c 1.6 MB/s | 139 kB 00:00 2024-03-24T00:35:41.419 INFO:teuthology.orchestra.run.smithi072.stdout:(30/174): rbd-mirror-19.0.0-2145.g3be9a5c9.el9. 8.0 MB/s | 3.1 MB 00:00 2024-03-24T00:35:41.519 INFO:teuthology.orchestra.run.smithi072.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2145.g3be9a 9.8 MB/s | 1.8 MB 00:00 2024-03-24T00:35:41.603 INFO:teuthology.orchestra.run.smithi072.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2145.g3b 2.9 MB/s | 248 kB 00:00 2024-03-24T00:35:41.662 INFO:teuthology.orchestra.run.smithi072.stdout:(33/174): ceph-mgr-rook-19.0.0-2145.g3be9a5c9.e 843 kB/s | 49 kB 00:00 2024-03-24T00:35:41.737 INFO:teuthology.orchestra.run.smithi072.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2145.g3 219 kB/s | 16 kB 00:00 2024-03-24T00:35:41.830 INFO:teuthology.orchestra.run.smithi072.stdout:(35/174): ceph-volume-19.0.0-2145.g3be9a5c9.el9 2.8 MB/s | 264 kB 00:00 2024-03-24T00:35:41.947 INFO:teuthology.orchestra.run.smithi072.stdout:(36/174): cephadm-19.0.0-2145.g3be9a5c9.el9.noa 6.2 MB/s | 748 kB 00:00 2024-03-24T00:35:42.031 INFO:teuthology.orchestra.run.smithi072.stdout:(37/174): cephfs-top-19.0.0-2145.g3be9a5c9.el9. 218 kB/s | 18 kB 00:00 2024-03-24T00:35:42.240 INFO:teuthology.orchestra.run.smithi072.stdout:(38/174): ceph-mgr-diskprediction-local-19.0.0- 9.0 MB/s | 7.4 MB 00:00 2024-03-24T00:35:42.299 INFO:teuthology.orchestra.run.smithi072.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 2.4 MB/s | 662 kB 00:00 2024-03-24T00:35:42.366 INFO:teuthology.orchestra.run.smithi072.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 7.2 MB/s | 920 kB 00:00 2024-03-24T00:35:42.391 INFO:teuthology.orchestra.run.smithi072.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.1 MB/s | 106 kB 00:00 2024-03-24T00:35:42.418 INFO:teuthology.orchestra.run.smithi072.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 5.8 MB/s | 307 kB 00:00 2024-03-24T00:35:42.444 INFO:teuthology.orchestra.run.smithi072.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 607 kB/s | 32 kB 00:00 2024-03-24T00:35:42.611 INFO:teuthology.orchestra.run.smithi072.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 16 MB/s | 3.0 MB 00:00 2024-03-24T00:35:42.637 INFO:teuthology.orchestra.run.smithi072.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 89 kB/s | 17 kB 00:00 2024-03-24T00:35:42.662 INFO:teuthology.orchestra.run.smithi072.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 199 kB/s | 10 kB 00:00 2024-03-24T00:35:42.688 INFO:teuthology.orchestra.run.smithi072.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-24T00:35:42.714 INFO:teuthology.orchestra.run.smithi072.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 870 kB/s | 44 kB 00:00 2024-03-24T00:35:42.747 INFO:teuthology.orchestra.run.smithi072.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-24T00:35:42.781 INFO:teuthology.orchestra.run.smithi072.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 8.5 MB/s | 578 kB 00:00 2024-03-24T00:35:42.807 INFO:teuthology.orchestra.run.smithi072.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 792 kB/s | 46 kB 00:00 2024-03-24T00:35:42.841 INFO:teuthology.orchestra.run.smithi072.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.7 MB/s | 282 kB 00:00 2024-03-24T00:35:42.877 INFO:teuthology.orchestra.run.smithi072.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 2.7 MB/s | 187 kB 00:00 2024-03-24T00:35:42.908 INFO:teuthology.orchestra.run.smithi072.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.2 MB/s | 217 kB 00:00 2024-03-24T00:35:42.934 INFO:teuthology.orchestra.run.smithi072.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 579 kB/s | 34 kB 00:00 2024-03-24T00:35:43.118 INFO:teuthology.orchestra.run.smithi072.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 23 MB/s | 4.7 MB 00:00 2024-03-24T00:35:43.170 INFO:teuthology.orchestra.run.smithi072.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 6.5 MB/s | 1.5 MB 00:00 2024-03-24T00:35:43.204 INFO:teuthology.orchestra.run.smithi072.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 9.5 MB/s | 811 kB 00:00 2024-03-24T00:35:43.230 INFO:teuthology.orchestra.run.smithi072.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 2.3 MB/s | 134 kB 00:00 2024-03-24T00:35:43.981 INFO:teuthology.orchestra.run.smithi072.stdout:(60/174): ceph-test-19.0.0-2145.g3be9a5c9.el9.x 11 MB/s | 48 MB 00:04 2024-03-24T00:35:44.106 INFO:teuthology.orchestra.run.smithi072.stdout:(61/174): postgresql-server-13.11-1.el9.x86_64. 6.3 MB/s | 5.7 MB 00:00 2024-03-24T00:35:44.148 INFO:teuthology.orchestra.run.smithi072.stdout:(62/174): protobuf-3.14.0-13.el9.x86_64.rpm 1.1 MB/s | 1.0 MB 00:00 2024-03-24T00:35:44.174 INFO:teuthology.orchestra.run.smithi072.stdout:(63/174): python3-attrs-20.3.0-7.el9.noarch.rpm 409 kB/s | 78 kB 00:00 2024-03-24T00:35:44.316 INFO:teuthology.orchestra.run.smithi072.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 28 MB/s | 5.8 MB 00:00 2024-03-24T00:35:44.341 INFO:teuthology.orchestra.run.smithi072.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 110 kB/s | 18 kB 00:00 2024-03-24T00:35:44.367 INFO:teuthology.orchestra.run.smithi072.stdout:(66/174): python3-jsonpatch-1.21-16.el9.noarch. 129 kB/s | 24 kB 00:00 2024-03-24T00:35:44.400 INFO:teuthology.orchestra.run.smithi072.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.7 MB/s | 144 kB 00:00 2024-03-24T00:35:44.426 INFO:teuthology.orchestra.run.smithi072.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 375 kB/s | 32 kB 00:00 2024-03-24T00:35:44.460 INFO:teuthology.orchestra.run.smithi072.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 3.6 MB/s | 217 kB 00:00 2024-03-24T00:35:44.569 INFO:teuthology.orchestra.run.smithi072.stdout:(70/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 28 MB/s | 4.8 MB 00:00 2024-03-24T00:35:44.594 INFO:teuthology.orchestra.run.smithi072.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.0 MB/s | 174 kB 00:00 2024-03-24T00:35:44.619 INFO:teuthology.orchestra.run.smithi072.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 438 kB/s | 69 kB 00:00 2024-03-24T00:35:44.646 INFO:teuthology.orchestra.run.smithi072.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 535 kB/s | 41 kB 00:00 2024-03-24T00:35:44.672 INFO:teuthology.orchestra.run.smithi072.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.5 MB/s | 132 kB 00:00 2024-03-24T00:35:44.759 INFO:teuthology.orchestra.run.smithi072.stdout:(75/174): python3-protobuf-3.14.0-13.el9.noarch 1.7 MB/s | 238 kB 00:00 2024-03-24T00:35:44.781 INFO:teuthology.orchestra.run.smithi072.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 1.5 MB/s | 211 kB 00:00 2024-03-24T00:35:44.798 INFO:teuthology.orchestra.run.smithi072.stdout:(77/174): python3-pytz-2021.1-5.el9.noarch.rpm 378 kB/s | 47 kB 00:00 2024-03-24T00:35:44.815 INFO:teuthology.orchestra.run.smithi072.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 1.2 MB/s | 43 kB 00:00 2024-03-24T00:35:44.941 INFO:teuthology.orchestra.run.smithi072.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 274 kB/s | 39 kB 00:00 2024-03-24T00:35:44.975 INFO:teuthology.orchestra.run.smithi072.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 262 kB/s | 41 kB 00:00 2024-03-24T00:35:45.226 INFO:teuthology.orchestra.run.smithi072.stdout:(81/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 35 MB/s | 16 MB 00:00 2024-03-24T00:35:45.259 INFO:teuthology.orchestra.run.smithi072.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 951 kB/s | 300 kB 00:00 2024-03-24T00:35:45.276 INFO:teuthology.orchestra.run.smithi072.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 188 kB/s | 56 kB 00:00 2024-03-24T00:35:45.294 INFO:teuthology.orchestra.run.smithi072.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-24T00:35:45.311 INFO:teuthology.orchestra.run.smithi072.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 866 kB/s | 44 kB 00:00 2024-03-24T00:35:45.336 INFO:teuthology.orchestra.run.smithi072.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.3 MB/s | 79 kB 00:00 2024-03-24T00:35:45.361 INFO:teuthology.orchestra.run.smithi072.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-24T00:35:45.403 INFO:teuthology.orchestra.run.smithi072.stdout:(88/174): python3-cryptography-36.0.1-4.el9.x86 12 MB/s | 1.1 MB 00:00 2024-03-24T00:35:45.429 INFO:teuthology.orchestra.run.smithi072.stdout:(89/174): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2024-03-24T00:35:45.456 INFO:teuthology.orchestra.run.smithi072.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 2.3 MB/s | 124 kB 00:00 2024-03-24T00:35:45.482 INFO:teuthology.orchestra.run.smithi072.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.4 MB/s | 187 kB 00:00 2024-03-24T00:35:45.607 INFO:teuthology.orchestra.run.smithi072.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2024-03-24T00:35:45.632 INFO:teuthology.orchestra.run.smithi072.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-03-24T00:35:45.666 INFO:teuthology.orchestra.run.smithi072.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-03-24T00:35:45.709 INFO:teuthology.orchestra.run.smithi072.stdout:(95/174): libgfortran-11.4.1-3.el9.x86_64.rpm 3.5 MB/s | 803 kB 00:00 2024-03-24T00:35:45.768 INFO:teuthology.orchestra.run.smithi072.stdout:(96/174): elfutils-libelf-devel-0.190-2.el9.x86 159 kB/s | 21 kB 00:00 2024-03-24T00:35:45.793 INFO:teuthology.orchestra.run.smithi072.stdout:(97/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 201 kB/s | 25 kB 00:00 2024-03-24T00:35:45.910 INFO:teuthology.orchestra.run.smithi072.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 1.7 MB/s | 245 kB 00:00 2024-03-24T00:35:45.944 INFO:teuthology.orchestra.run.smithi072.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 1.6 MB/s | 248 kB 00:00 2024-03-24T00:35:45.969 INFO:teuthology.orchestra.run.smithi072.stdout:(100/174): qatlib-23.11.0-1.el9.x86_64.rpm 3.6 MB/s | 216 kB 00:00 2024-03-24T00:35:45.995 INFO:teuthology.orchestra.run.smithi072.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 689 kB/s | 35 kB 00:00 2024-03-24T00:35:46.021 INFO:teuthology.orchestra.run.smithi072.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 875 kB/s | 44 kB 00:00 2024-03-24T00:35:46.055 INFO:teuthology.orchestra.run.smithi072.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.5 MB/s | 94 kB 00:00 2024-03-24T00:35:46.113 INFO:teuthology.orchestra.run.smithi072.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 242 kB/s | 22 kB 00:00 2024-03-24T00:35:46.231 INFO:teuthology.orchestra.run.smithi072.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 4.8 MB/s | 863 kB 00:00 2024-03-24T00:35:46.400 INFO:teuthology.orchestra.run.smithi072.stdout:(106/174): openssl-devel-3.0.7-27.el9.x86_64.rp 6.0 MB/s | 4.1 MB 00:00 2024-03-24T00:35:46.500 INFO:teuthology.orchestra.run.smithi072.stdout:(107/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.4 MB/s | 548 kB 00:00 2024-03-24T00:35:46.525 INFO:teuthology.orchestra.run.smithi072.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 155 kB/s | 19 kB 00:00 2024-03-24T00:35:46.576 INFO:teuthology.orchestra.run.smithi072.stdout:(109/174): gperftools-libs-2.9.1-2.el9.x86_64.r 896 kB/s | 309 kB 00:00 2024-03-24T00:35:46.601 INFO:teuthology.orchestra.run.smithi072.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 343 kB/s | 26 kB 00:00 2024-03-24T00:35:46.643 INFO:teuthology.orchestra.run.smithi072.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 722 kB/s | 49 kB 00:00 2024-03-24T00:35:46.694 INFO:teuthology.orchestra.run.smithi072.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 729 kB/s | 67 kB 00:00 2024-03-24T00:35:46.745 INFO:teuthology.orchestra.run.smithi072.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.5 MB/s | 151 kB 00:00 2024-03-24T00:35:46.895 INFO:teuthology.orchestra.run.smithi072.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.1 MB/s | 838 kB 00:00 2024-03-24T00:35:46.937 INFO:teuthology.orchestra.run.smithi072.stdout:(115/174): python3-asyncssh-2.13.2-5.el9.noarch 2.8 MB/s | 548 kB 00:00 2024-03-24T00:35:46.963 INFO:teuthology.orchestra.run.smithi072.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 644 kB/s | 43 kB 00:00 2024-03-24T00:35:46.997 INFO:teuthology.orchestra.run.smithi072.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 544 kB/s | 32 kB 00:00 2024-03-24T00:35:47.022 INFO:teuthology.orchestra.run.smithi072.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 255 kB/s | 15 kB 00:00 2024-03-24T00:35:47.106 INFO:teuthology.orchestra.run.smithi072.stdout:(119/174): python3-cherrypy-18.6.1-2.el9.noarch 4.2 MB/s | 358 kB 00:00 2024-03-24T00:35:47.132 INFO:teuthology.orchestra.run.smithi072.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.2 MB/s | 172 kB 00:00 2024-03-24T00:35:47.232 INFO:teuthology.orchestra.run.smithi072.stdout:(121/174): libarrow-9.0.0-10.el9.x86_64.rpm 6.0 MB/s | 4.4 MB 00:00 2024-03-24T00:35:47.259 INFO:teuthology.orchestra.run.smithi072.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 277 kB/s | 35 kB 00:00 2024-03-24T00:35:47.293 INFO:teuthology.orchestra.run.smithi072.stdout:(123/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.6 MB/s | 303 kB 00:00 2024-03-24T00:35:47.327 INFO:teuthology.orchestra.run.smithi072.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 2.1 MB/s | 203 kB 00:00 2024-03-24T00:35:47.352 INFO:teuthology.orchestra.run.smithi072.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 2.4 MB/s | 144 kB 00:00 2024-03-24T00:35:47.378 INFO:teuthology.orchestra.run.smithi072.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 211 kB/s | 11 kB 00:00 2024-03-24T00:35:47.403 INFO:teuthology.orchestra.run.smithi072.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 349 kB/s | 18 kB 00:00 2024-03-24T00:35:47.429 INFO:teuthology.orchestra.run.smithi072.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 456 kB/s | 23 kB 00:00 2024-03-24T00:35:47.454 INFO:teuthology.orchestra.run.smithi072.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 384 kB/s | 19 kB 00:00 2024-03-24T00:35:47.480 INFO:teuthology.orchestra.run.smithi072.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 388 kB/s | 20 kB 00:00 2024-03-24T00:35:47.556 INFO:teuthology.orchestra.run.smithi072.stdout:(131/174): python3-kubernetes-26.1.0-2.el9.noar 10 MB/s | 1.0 MB 00:00 2024-03-24T00:35:47.582 INFO:teuthology.orchestra.run.smithi072.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 459 kB/s | 46 kB 00:00 2024-03-24T00:35:47.615 INFO:teuthology.orchestra.run.smithi072.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 1.3 MB/s | 79 kB 00:00 2024-03-24T00:35:47.641 INFO:teuthology.orchestra.run.smithi072.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 977 kB/s | 58 kB 00:00 2024-03-24T00:35:47.708 INFO:teuthology.orchestra.run.smithi072.stdout:(135/174): python3-grpcio-1.46.7-10.el9.x86_64. 4.5 MB/s | 2.0 MB 00:00 2024-03-24T00:35:47.733 INFO:teuthology.orchestra.run.smithi072.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 178 kB/s | 16 kB 00:00 2024-03-24T00:35:47.768 INFO:teuthology.orchestra.run.smithi072.stdout:(137/174): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-03-24T00:35:47.794 INFO:teuthology.orchestra.run.smithi072.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 465 kB/s | 28 kB 00:00 2024-03-24T00:35:47.828 INFO:teuthology.orchestra.run.smithi072.stdout:(139/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 749 kB/s | 90 kB 00:00 2024-03-24T00:35:47.854 INFO:teuthology.orchestra.run.smithi072.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 357 kB/s | 31 kB 00:00 2024-03-24T00:35:47.888 INFO:teuthology.orchestra.run.smithi072.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 2.0 MB/s | 188 kB 00:00 2024-03-24T00:35:47.913 INFO:teuthology.orchestra.run.smithi072.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 699 kB/s | 59 kB 00:00 2024-03-24T00:35:47.938 INFO:teuthology.orchestra.run.smithi072.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 423 kB/s | 36 kB 00:00 2024-03-24T00:35:47.964 INFO:teuthology.orchestra.run.smithi072.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 671 kB/s | 51 kB 00:00 2024-03-24T00:35:47.998 INFO:teuthology.orchestra.run.smithi072.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 1.5 MB/s | 90 kB 00:00 2024-03-24T00:35:48.032 INFO:teuthology.orchestra.run.smithi072.stdout:(146/174): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-03-24T00:35:48.066 INFO:teuthology.orchestra.run.smithi072.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 4.1 MB/s | 427 kB 00:00 2024-03-24T00:35:48.091 INFO:teuthology.orchestra.run.smithi072.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 215 kB/s | 20 kB 00:00 2024-03-24T00:35:48.125 INFO:teuthology.orchestra.run.smithi072.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2024-03-24T00:35:48.201 INFO:teuthology.orchestra.run.smithi072.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-03-24T00:35:48.426 INFO:teuthology.orchestra.run.smithi072.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 1.6 MB/s | 321 kB 00:00 2024-03-24T00:35:48.543 INFO:teuthology.orchestra.run.smithi072.stdout:(152/174): NetworkManager-1.47.2-1.el9.x86_64.r 20 MB/s | 2.3 MB 00:00 2024-03-24T00:35:48.629 INFO:teuthology.orchestra.run.smithi072.stdout:(153/174): NetworkManager-libnm-1.47.2-1.el9.x8 22 MB/s | 1.8 MB 00:00 2024-03-24T00:35:48.671 INFO:teuthology.orchestra.run.smithi072.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 1.0 MB/s | 42 kB 00:00 2024-03-24T00:35:48.755 INFO:teuthology.orchestra.run.smithi072.stdout:(155/174): librados2-19.0.0-2145.g3be9a5c9.el9. 5.1 MB/s | 3.4 MB 00:00 2024-03-24T00:35:48.788 INFO:teuthology.orchestra.run.smithi072.stdout:(156/174): NetworkManager-tui-1.47.2-1.el9.x86_ 2.1 MB/s | 248 kB 00:00 2024-03-24T00:35:48.831 INFO:teuthology.orchestra.run.smithi072.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 872 kB/s | 36 kB 00:00 2024-03-24T00:35:48.881 INFO:teuthology.orchestra.run.smithi072.stdout:(158/174): elfutils-libelf-0.190-2.el9.x86_64.r 3.7 MB/s | 192 kB 00:00 2024-03-24T00:35:48.907 INFO:teuthology.orchestra.run.smithi072.stdout:(159/174): acl-2.3.1-4.el9.x86_64.rpm 470 kB/s | 71 kB 00:00 2024-03-24T00:35:48.982 INFO:teuthology.orchestra.run.smithi072.stdout:(160/174): librbd1-19.0.0-2145.g3be9a5c9.el9.x8 3.7 MB/s | 3.1 MB 00:00 2024-03-24T00:35:49.007 INFO:teuthology.orchestra.run.smithi072.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 229 kB/s | 23 kB 00:00 2024-03-24T00:35:49.033 INFO:teuthology.orchestra.run.smithi072.stdout:(162/174): elfutils-libs-0.190-2.el9.x86_64.rpm 1.6 MB/s | 256 kB 00:00 2024-03-24T00:35:49.075 INFO:teuthology.orchestra.run.smithi072.stdout:(163/174): libmount-2.37.4-18.el9.x86_64.rpm 3.2 MB/s | 136 kB 00:00 2024-03-24T00:35:49.101 INFO:teuthology.orchestra.run.smithi072.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.6 MB/s | 155 kB 00:00 2024-03-24T00:35:49.126 INFO:teuthology.orchestra.run.smithi072.stdout:(165/174): libblkid-2.37.4-18.el9.x86_64.rpm 753 kB/s | 108 kB 00:00 2024-03-24T00:35:49.152 INFO:teuthology.orchestra.run.smithi072.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 837 kB/s | 64 kB 00:00 2024-03-24T00:35:49.177 INFO:teuthology.orchestra.run.smithi072.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 376 kB/s | 29 kB 00:00 2024-03-24T00:35:49.220 INFO:teuthology.orchestra.run.smithi072.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 13 MB/s | 1.2 MB 00:00 2024-03-24T00:35:49.280 INFO:teuthology.orchestra.run.smithi072.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 7.6 MB/s | 465 kB 00:00 2024-03-24T00:35:49.573 INFO:teuthology.orchestra.run.smithi072.stdout:(170/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 5.1 MB/s | 2.2 MB 00:00 2024-03-24T00:35:49.598 INFO:teuthology.orchestra.run.smithi072.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 98 kB/s | 31 kB 00:00 2024-03-24T00:35:49.665 INFO:teuthology.orchestra.run.smithi072.stdout:(172/174): util-linux-2.37.4-18.el9.x86_64.rpm 4.7 MB/s | 2.3 MB 00:00 2024-03-24T00:35:49.707 INFO:teuthology.orchestra.run.smithi072.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 8.0 MB/s | 1.1 MB 00:00 2024-03-24T00:35:49.733 INFO:teuthology.orchestra.run.smithi072.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 554 kB/s | 74 kB 00:00 2024-03-24T00:35:49.737 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-03-24T00:35:49.737 INFO:teuthology.orchestra.run.smithi072.stdout:Total 17 MB/s | 224 MB 00:12 2024-03-24T00:35:50.212 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-03-24T00:35:50.485 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-03-24T00:35:50.485 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-03-24T00:35:52.537 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-03-24T00:35:52.538 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-03-24T00:35:59.908 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-03-24T00:36:00.137 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-24T00:36:00.404 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-24T00:36:00.448 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-24T00:36:00.460 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-24T00:36:00.529 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-24T00:36:00.597 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-24T00:36:00.637 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-24T00:36:00.679 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-24T00:36:00.734 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-24T00:36:01.069 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-24T00:36:01.223 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-24T00:36:01.328 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-24T00:36:01.359 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-24T00:36:01.434 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-24T00:36:01.472 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-24T00:36:01.521 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-24T00:36:01.712 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-24T00:36:01.775 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-24T00:36:01.804 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-24T00:36:01.907 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-24T00:36:01.939 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-24T00:36:01.997 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-24T00:36:02.036 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-24T00:36:02.060 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-24T00:36:02.075 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-24T00:36:02.236 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-24T00:36:02.269 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-24T00:36:02.316 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-24T00:36:02.736 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-24T00:36:02.821 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-24T00:36:02.874 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-24T00:36:03.023 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T00:36:03.071 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T00:36:03.313 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T00:36:03.422 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-24T00:36:03.514 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-24T00:36:03.552 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-24T00:36:03.588 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-24T00:36:03.639 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-24T00:36:03.716 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-24T00:36:03.771 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-24T00:36:03.810 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-24T00:36:03.874 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-24T00:36:03.919 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-24T00:36:03.972 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-24T00:36:04.006 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-24T00:36:04.035 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-24T00:36:04.104 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-24T00:36:04.141 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 39/198 2024-03-24T00:36:04.188 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 40/198 2024-03-24T00:36:04.222 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 41/198 2024-03-24T00:36:04.290 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-24T00:36:04.337 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-24T00:36:04.412 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-24T00:36:04.545 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-24T00:36:04.602 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-24T00:36:05.115 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-24T00:36:05.201 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-24T00:36:06.003 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-24T00:36:06.177 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-24T00:36:06.226 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-24T00:36:06.274 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-24T00:36:06.318 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-24T00:36:06.356 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 53/198 2024-03-24T00:36:06.396 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-24T00:36:06.700 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-24T00:36:06.905 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-24T00:36:06.991 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-24T00:36:07.035 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-24T00:36:07.166 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-24T00:36:07.218 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-24T00:36:07.272 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-24T00:36:07.346 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-24T00:36:07.378 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-24T00:36:07.417 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-24T00:36:07.464 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-24T00:36:07.501 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-24T00:36:07.544 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-24T00:36:07.587 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-24T00:36:07.727 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-24T00:36:07.780 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-24T00:36:07.819 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-24T00:36:07.861 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-24T00:36:07.955 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-24T00:36:07.996 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-24T00:36:08.038 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-24T00:36:08.065 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-24T00:36:08.090 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-24T00:36:08.178 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-24T00:36:08.287 INFO:teuthology.orchestra.run.smithi067.stdout:Extra Packages for Enterprise Linux 521 kB/s | 21 MB 00:40 2024-03-24T00:36:08.303 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-24T00:36:08.404 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-24T00:36:08.456 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-24T00:36:08.504 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-24T00:36:08.562 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-24T00:36:08.753 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-24T00:36:08.806 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-24T00:36:08.941 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-24T00:36:08.989 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-24T00:36:09.032 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-24T00:36:09.063 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-24T00:36:09.108 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-24T00:36:09.143 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-24T00:36:09.183 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-24T00:36:09.214 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-24T00:36:09.255 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-24T00:36:09.323 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-24T00:36:09.359 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-24T00:36:09.393 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-24T00:36:09.434 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-24T00:36:09.470 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-24T00:36:09.557 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-24T00:36:09.642 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-24T00:36:09.696 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-24T00:36:09.736 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-24T00:36:09.783 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-24T00:36:09.880 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-24T00:36:09.941 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-24T00:36:10.659 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-24T00:36:10.717 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-24T00:36:10.753 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-24T00:36:10.801 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-24T00:36:10.888 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-24T00:36:10.933 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-24T00:36:11.112 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-24T00:36:11.139 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-24T00:36:11.218 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-24T00:36:11.862 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-24T00:36:11.985 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-24T00:36:13.205 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-24T00:36:13.305 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-24T00:36:13.337 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-24T00:36:13.370 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-24T00:36:13.398 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-24T00:36:13.471 INFO:teuthology.orchestra.run.smithi067.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-03-24T00:36:13.704 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-24T00:36:13.738 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-24T00:36:13.821 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-24T00:36:13.853 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 125/198 2024-03-24T00:36:13.898 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-24T00:36:14.360 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-24T00:36:14.387 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-24T00:36:14.450 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-24T00:36:14.482 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 129/198 2024-03-24T00:36:15.924 INFO:teuthology.orchestra.run.smithi067.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T00:36:15.925 INFO:teuthology.orchestra.run.smithi067.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T00:36:15.937 INFO:teuthology.orchestra.run.smithi067.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-24T00:36:15.938 INFO:teuthology.orchestra.run.smithi067.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-24T00:36:15.939 INFO:teuthology.orchestra.run.smithi067.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-24T00:36:15.940 INFO:teuthology.orchestra.run.smithi067.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-24T00:36:15.942 INFO:teuthology.orchestra.run.smithi067.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-24T00:36:15.945 INFO:teuthology.orchestra.run.smithi067.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-24T00:36:15.945 INFO:teuthology.orchestra.run.smithi067.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-24T00:36:16.030 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-24T00:36:16.049 INFO:teuthology.orchestra.run.smithi067.stdout:============================================================================================ 2024-03-24T00:36:16.049 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-03-24T00:36:16.049 INFO:teuthology.orchestra.run.smithi067.stdout:============================================================================================ 2024-03-24T00:36:16.049 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-03-24T00:36:16.049 INFO:teuthology.orchestra.run.smithi067.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-24T00:36:16.049 INFO:teuthology.orchestra.run.smithi067.stdout: ceph x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 6.6 k 2024-03-24T00:36:16.049 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-base x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.4 M 2024-03-24T00:36:16.049 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 890 k 2024-03-24T00:36:16.049 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 145 k 2024-03-24T00:36:16.050 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.5 M 2024-03-24T00:36:16.050 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 139 k 2024-03-24T00:36:16.050 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 1.8 M 2024-03-24T00:36:16.050 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 7.4 M 2024-03-24T00:36:16.050 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-rook noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 49 k 2024-03-24T00:36:16.050 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-radosgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 9.9 M 2024-03-24T00:36:16.050 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-test x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 48 M 2024-03-24T00:36:16.050 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-volume noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 264 k 2024-03-24T00:36:16.050 INFO:teuthology.orchestra.run.smithi067.stdout: cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 748 k 2024-03-24T00:36:16.050 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 220 k 2024-03-24T00:36:16.050 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-shell x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 27 k 2024-03-24T00:36:16.050 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-top noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 18 k 2024-03-24T00:36:16.051 INFO:teuthology.orchestra.run.smithi067.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-24T00:36:16.051 INFO:teuthology.orchestra.run.smithi067.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-24T00:36:16.051 INFO:teuthology.orchestra.run.smithi067.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-24T00:36:16.051 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 31 k 2024-03-24T00:36:16.051 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 746 k 2024-03-24T00:36:16.051 INFO:teuthology.orchestra.run.smithi067.stdout: librados-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 127 k 2024-03-24T00:36:16.051 INFO:teuthology.orchestra.run.smithi067.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-24T00:36:16.051 INFO:teuthology.orchestra.run.smithi067.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-24T00:36:16.051 INFO:teuthology.orchestra.run.smithi067.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-24T00:36:16.051 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-24T00:36:16.051 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-24T00:36:16.051 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-24T00:36:16.052 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cephfs x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 162 k 2024-03-24T00:36:16.052 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rados x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 321 k 2024-03-24T00:36:16.052 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 298 k 2024-03-24T00:36:16.052 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 99 k 2024-03-24T00:36:16.052 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 85 k 2024-03-24T00:36:16.052 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-24T00:36:16.052 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-nbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 170 k 2024-03-24T00:36:16.052 INFO:teuthology.orchestra.run.smithi067.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-24T00:36:16.052 INFO:teuthology.orchestra.run.smithi067.stdout:Upgrading: 2024-03-24T00:36:16.052 INFO:teuthology.orchestra.run.smithi067.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-24T00:36:16.052 INFO:teuthology.orchestra.run.smithi067.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-24T00:36:16.053 INFO:teuthology.orchestra.run.smithi067.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-24T00:36:16.053 INFO:teuthology.orchestra.run.smithi067.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-24T00:36:16.053 INFO:teuthology.orchestra.run.smithi067.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-24T00:36:16.053 INFO:teuthology.orchestra.run.smithi067.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-24T00:36:16.053 INFO:teuthology.orchestra.run.smithi067.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-24T00:36:16.053 INFO:teuthology.orchestra.run.smithi067.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-24T00:36:16.053 INFO:teuthology.orchestra.run.smithi067.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-24T00:36:16.053 INFO:teuthology.orchestra.run.smithi067.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-24T00:36:16.053 INFO:teuthology.orchestra.run.smithi067.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-24T00:36:16.053 INFO:teuthology.orchestra.run.smithi067.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-24T00:36:16.053 INFO:teuthology.orchestra.run.smithi067.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-24T00:36:16.053 INFO:teuthology.orchestra.run.smithi067.stdout: librados2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.4 M 2024-03-24T00:36:16.053 INFO:teuthology.orchestra.run.smithi067.stdout: librbd1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-24T00:36:16.054 INFO:teuthology.orchestra.run.smithi067.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-24T00:36:16.054 INFO:teuthology.orchestra.run.smithi067.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-24T00:36:16.054 INFO:teuthology.orchestra.run.smithi067.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-24T00:36:16.054 INFO:teuthology.orchestra.run.smithi067.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-24T00:36:16.054 INFO:teuthology.orchestra.run.smithi067.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-24T00:36:16.054 INFO:teuthology.orchestra.run.smithi067.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-24T00:36:16.054 INFO:teuthology.orchestra.run.smithi067.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-24T00:36:16.054 INFO:teuthology.orchestra.run.smithi067.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-24T00:36:16.054 INFO:teuthology.orchestra.run.smithi067.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-24T00:36:16.054 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-03-24T00:36:16.054 INFO:teuthology.orchestra.run.smithi067.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-24T00:36:16.055 INFO:teuthology.orchestra.run.smithi067.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-24T00:36:16.055 INFO:teuthology.orchestra.run.smithi067.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-24T00:36:16.055 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 20 M 2024-03-24T00:36:16.055 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 34 k 2024-03-24T00:36:16.055 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mds x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 2.3 M 2024-03-24T00:36:16.055 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 248 k 2024-03-24T00:36:16.055 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mon x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 4.7 M 2024-03-24T00:36:16.055 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-osd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 17 M 2024-03-24T00:36:16.055 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 16 k 2024-03-24T00:36:16.055 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-selinux x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 25 k 2024-03-24T00:36:16.055 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-24T00:36:16.055 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-24T00:36:16.055 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-24T00:36:16.056 INFO:teuthology.orchestra.run.smithi067.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-24T00:36:16.056 INFO:teuthology.orchestra.run.smithi067.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-24T00:36:16.056 INFO:teuthology.orchestra.run.smithi067.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-24T00:36:16.056 INFO:teuthology.orchestra.run.smithi067.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-24T00:36:16.056 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-24T00:36:16.056 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-24T00:36:16.056 INFO:teuthology.orchestra.run.smithi067.stdout: libcephsqlite x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 163 k 2024-03-24T00:36:16.056 INFO:teuthology.orchestra.run.smithi067.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-24T00:36:16.056 INFO:teuthology.orchestra.run.smithi067.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-24T00:36:16.056 INFO:teuthology.orchestra.run.smithi067.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-24T00:36:16.056 INFO:teuthology.orchestra.run.smithi067.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-24T00:36:16.056 INFO:teuthology.orchestra.run.smithi067.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-24T00:36:16.057 INFO:teuthology.orchestra.run.smithi067.stdout: libradosstriper1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 501 k 2024-03-24T00:36:16.057 INFO:teuthology.orchestra.run.smithi067.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-24T00:36:16.057 INFO:teuthology.orchestra.run.smithi067.stdout: librgw2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.1 M 2024-03-24T00:36:16.057 INFO:teuthology.orchestra.run.smithi067.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-24T00:36:16.057 INFO:teuthology.orchestra.run.smithi067.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-24T00:36:16.057 INFO:teuthology.orchestra.run.smithi067.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-24T00:36:16.057 INFO:teuthology.orchestra.run.smithi067.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-24T00:36:16.057 INFO:teuthology.orchestra.run.smithi067.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-24T00:36:16.057 INFO:teuthology.orchestra.run.smithi067.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-24T00:36:16.057 INFO:teuthology.orchestra.run.smithi067.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-24T00:36:16.057 INFO:teuthology.orchestra.run.smithi067.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-24T00:36:16.057 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-24T00:36:16.058 INFO:teuthology.orchestra.run.smithi067.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-24T00:36:16.058 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-24T00:36:16.058 INFO:teuthology.orchestra.run.smithi067.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-24T00:36:16.058 INFO:teuthology.orchestra.run.smithi067.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-24T00:36:16.058 INFO:teuthology.orchestra.run.smithi067.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-24T00:36:16.058 INFO:teuthology.orchestra.run.smithi067.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-24T00:36:16.058 INFO:teuthology.orchestra.run.smithi067.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-24T00:36:16.058 INFO:teuthology.orchestra.run.smithi067.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-24T00:36:16.058 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-24T00:36:16.058 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-argparse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 45 k 2024-03-24T00:36:16.058 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 131 k 2024-03-24T00:36:16.058 INFO:teuthology.orchestra.run.smithi067.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-24T00:36:16.059 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-24T00:36:16.059 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-24T00:36:16.059 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-24T00:36:16.059 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-24T00:36:16.059 INFO:teuthology.orchestra.run.smithi067.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-24T00:36:16.059 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-24T00:36:16.059 INFO:teuthology.orchestra.run.smithi067.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-24T00:36:16.059 INFO:teuthology.orchestra.run.smithi067.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-24T00:36:16.059 INFO:teuthology.orchestra.run.smithi067.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-24T00:36:16.059 INFO:teuthology.orchestra.run.smithi067.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-24T00:36:16.059 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-24T00:36:16.059 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-24T00:36:16.059 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-24T00:36:16.060 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-24T00:36:16.060 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-24T00:36:16.060 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-24T00:36:16.060 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-24T00:36:16.060 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-24T00:36:16.060 INFO:teuthology.orchestra.run.smithi067.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-24T00:36:16.060 INFO:teuthology.orchestra.run.smithi067.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-24T00:36:16.060 INFO:teuthology.orchestra.run.smithi067.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-24T00:36:16.060 INFO:teuthology.orchestra.run.smithi067.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-24T00:36:16.060 INFO:teuthology.orchestra.run.smithi067.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-24T00:36:16.060 INFO:teuthology.orchestra.run.smithi067.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-24T00:36:16.060 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-24T00:36:16.060 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-24T00:36:16.061 INFO:teuthology.orchestra.run.smithi067.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-24T00:36:16.061 INFO:teuthology.orchestra.run.smithi067.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-24T00:36:16.061 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-24T00:36:16.061 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-24T00:36:16.061 INFO:teuthology.orchestra.run.smithi067.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-24T00:36:16.061 INFO:teuthology.orchestra.run.smithi067.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-24T00:36:16.061 INFO:teuthology.orchestra.run.smithi067.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-24T00:36:16.061 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-24T00:36:16.061 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-24T00:36:16.061 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-24T00:36:16.061 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-24T00:36:16.061 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-24T00:36:16.061 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-24T00:36:16.062 INFO:teuthology.orchestra.run.smithi067.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-24T00:36:16.062 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-24T00:36:16.062 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-24T00:36:16.062 INFO:teuthology.orchestra.run.smithi067.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-24T00:36:16.062 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-24T00:36:16.062 INFO:teuthology.orchestra.run.smithi067.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-24T00:36:16.062 INFO:teuthology.orchestra.run.smithi067.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-24T00:36:16.062 INFO:teuthology.orchestra.run.smithi067.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-24T00:36:16.062 INFO:teuthology.orchestra.run.smithi067.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-24T00:36:16.062 INFO:teuthology.orchestra.run.smithi067.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-24T00:36:16.062 INFO:teuthology.orchestra.run.smithi067.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-24T00:36:16.062 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-24T00:36:16.063 INFO:teuthology.orchestra.run.smithi067.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-24T00:36:16.063 INFO:teuthology.orchestra.run.smithi067.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-24T00:36:16.063 INFO:teuthology.orchestra.run.smithi067.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-24T00:36:16.063 INFO:teuthology.orchestra.run.smithi067.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-24T00:36:16.063 INFO:teuthology.orchestra.run.smithi067.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-24T00:36:16.063 INFO:teuthology.orchestra.run.smithi067.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-24T00:36:16.063 INFO:teuthology.orchestra.run.smithi067.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-24T00:36:16.063 INFO:teuthology.orchestra.run.smithi067.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-24T00:36:16.063 INFO:teuthology.orchestra.run.smithi067.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-24T00:36:16.063 INFO:teuthology.orchestra.run.smithi067.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-24T00:36:16.063 INFO:teuthology.orchestra.run.smithi067.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-24T00:36:16.063 INFO:teuthology.orchestra.run.smithi067.stdout:Installing weak dependencies: 2024-03-24T00:36:16.063 INFO:teuthology.orchestra.run.smithi067.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-24T00:36:16.064 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:36:16.064 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-03-24T00:36:16.064 INFO:teuthology.orchestra.run.smithi067.stdout:============================================================================================ 2024-03-24T00:36:16.064 INFO:teuthology.orchestra.run.smithi067.stdout:Install 150 Packages 2024-03-24T00:36:16.064 INFO:teuthology.orchestra.run.smithi067.stdout:Upgrade 24 Packages 2024-03-24T00:36:16.064 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:36:16.064 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 224 M 2024-03-24T00:36:16.065 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-03-24T00:36:16.548 INFO:teuthology.orchestra.run.smithi067.stdout:(1/174): ceph-19.0.0-2145.g3be9a5c9.el9.x86_64. 30 kB/s | 6.6 kB 00:00 2024-03-24T00:36:16.690 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-24T00:36:16.977 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-24T00:36:16.989 INFO:teuthology.orchestra.run.smithi067.stdout:(2/174): ceph-fuse-19.0.0-2145.g3be9a5c9.el9.x8 2.0 MB/s | 890 kB 00:00 2024-03-24T00:36:17.081 INFO:teuthology.orchestra.run.smithi067.stdout:(3/174): ceph-immutable-object-cache-19.0.0-214 1.5 MB/s | 145 kB 00:00 2024-03-24T00:36:17.157 INFO:teuthology.orchestra.run.smithi067.stdout:(4/174): ceph-base-19.0.0-2145.g3be9a5c9.el9.x8 6.5 MB/s | 5.4 MB 00:00 2024-03-24T00:36:17.415 INFO:teuthology.orchestra.run.smithi067.stdout:(5/174): ceph-mds-19.0.0-2145.g3be9a5c9.el9.x86 6.9 MB/s | 2.3 MB 00:00 2024-03-24T00:36:17.457 INFO:teuthology.orchestra.run.smithi067.stdout:(6/174): ceph-mgr-19.0.0-2145.g3be9a5c9.el9.x86 5.0 MB/s | 1.5 MB 00:00 2024-03-24T00:36:17.528 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-24T00:36:17.576 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-24T00:36:17.669 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-24T00:36:17.670 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-24T00:36:17.670 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-24T00:36:17.670 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:36:17.704 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-24T00:36:17.908 INFO:teuthology.orchestra.run.smithi067.stdout:(7/174): ceph-mon-19.0.0-2145.g3be9a5c9.el9.x86 9.5 MB/s | 4.7 MB 00:00 2024-03-24T00:36:18.624 INFO:teuthology.orchestra.run.smithi067.stdout:(8/174): ceph-common-19.0.0-2145.g3be9a5c9.el9. 8.9 MB/s | 20 MB 00:02 2024-03-24T00:36:18.750 INFO:teuthology.orchestra.run.smithi067.stdout:(9/174): ceph-radosgw-19.0.0-2145.g3be9a5c9.el9 12 MB/s | 9.9 MB 00:00 2024-03-24T00:36:18.775 INFO:teuthology.orchestra.run.smithi067.stdout:(10/174): ceph-selinux-19.0.0-2145.g3be9a5c9.el 167 kB/s | 25 kB 00:00 2024-03-24T00:36:18.875 INFO:teuthology.orchestra.run.smithi067.stdout:(11/174): cephfs-mirror-19.0.0-2145.g3be9a5c9.e 2.1 MB/s | 220 kB 00:00 2024-03-24T00:36:18.992 INFO:teuthology.orchestra.run.smithi067.stdout:(12/174): cephfs-shell-19.0.0-2145.g3be9a5c9.el 232 kB/s | 27 kB 00:00 2024-03-24T00:36:19.101 INFO:teuthology.orchestra.run.smithi067.stdout:(13/174): libcephfs-devel-19.0.0-2145.g3be9a5c9 288 kB/s | 31 kB 00:00 2024-03-24T00:36:19.226 INFO:teuthology.orchestra.run.smithi067.stdout:(14/174): libcephfs2-19.0.0-2145.g3be9a5c9.el9. 5.8 MB/s | 746 kB 00:00 2024-03-24T00:36:19.326 INFO:teuthology.orchestra.run.smithi067.stdout:(15/174): libcephsqlite-19.0.0-2145.g3be9a5c9.e 1.6 MB/s | 163 kB 00:00 2024-03-24T00:36:19.418 INFO:teuthology.orchestra.run.smithi067.stdout:(16/174): librados-devel-19.0.0-2145.g3be9a5c9. 1.4 MB/s | 127 kB 00:00 2024-03-24T00:36:19.535 INFO:teuthology.orchestra.run.smithi067.stdout:(17/174): libradosstriper1-19.0.0-2145.g3be9a5c 4.2 MB/s | 501 kB 00:00 2024-03-24T00:36:19.919 INFO:teuthology.orchestra.run.smithi067.stdout:(18/174): ceph-osd-19.0.0-2145.g3be9a5c9.el9.x8 6.8 MB/s | 17 MB 00:02 2024-03-24T00:36:19.994 INFO:teuthology.orchestra.run.smithi067.stdout:(19/174): python3-ceph-argparse-19.0.0-2145.g3b 602 kB/s | 45 kB 00:00 2024-03-24T00:36:20.078 INFO:teuthology.orchestra.run.smithi067.stdout:(20/174): python3-ceph-common-19.0.0-2145.g3be9 1.5 MB/s | 131 kB 00:00 2024-03-24T00:36:20.169 INFO:teuthology.orchestra.run.smithi067.stdout:(21/174): librgw2-19.0.0-2145.g3be9a5c9.el9.x86 8.1 MB/s | 5.1 MB 00:00 2024-03-24T00:36:20.194 INFO:teuthology.orchestra.run.smithi067.stdout:(22/174): python3-cephfs-19.0.0-2145.g3be9a5c9. 1.4 MB/s | 162 kB 00:00 2024-03-24T00:36:20.245 INFO:teuthology.orchestra.run.smithi067.stdout:(23/174): python3-rados-19.0.0-2145.g3be9a5c9.e 4.2 MB/s | 321 kB 00:00 2024-03-24T00:36:20.270 INFO:teuthology.orchestra.run.smithi067.stdout:(24/174): python3-rbd-19.0.0-2145.g3be9a5c9.el9 3.9 MB/s | 298 kB 00:00 2024-03-24T00:36:20.312 INFO:teuthology.orchestra.run.smithi067.stdout:(25/174): python3-rgw-19.0.0-2145.g3be9a5c9.el9 1.5 MB/s | 99 kB 00:00 2024-03-24T00:36:20.337 INFO:teuthology.orchestra.run.smithi067.stdout:(26/174): rbd-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.2 MB/s | 85 kB 00:00 2024-03-24T00:36:20.421 INFO:teuthology.orchestra.run.smithi067.stdout:(27/174): rbd-nbd-19.0.0-2145.g3be9a5c9.el9.x86 2.0 MB/s | 170 kB 00:00 2024-03-24T00:36:20.505 INFO:teuthology.orchestra.run.smithi067.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2145.g 408 kB/s | 34 kB 00:00 2024-03-24T00:36:20.588 INFO:teuthology.orchestra.run.smithi067.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2145.g3be9a5c 1.6 MB/s | 139 kB 00:00 2024-03-24T00:36:20.673 INFO:teuthology.orchestra.run.smithi067.stdout:(30/174): rbd-mirror-19.0.0-2145.g3be9a5c9.el9. 8.6 MB/s | 3.1 MB 00:00 2024-03-24T00:36:20.922 INFO:teuthology.orchestra.run.smithi067.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2145.g3be9a 5.4 MB/s | 1.8 MB 00:00 2024-03-24T00:36:21.031 INFO:teuthology.orchestra.run.smithi067.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2145.g3b 2.2 MB/s | 248 kB 00:00 2024-03-24T00:36:21.123 INFO:teuthology.orchestra.run.smithi067.stdout:(33/174): ceph-mgr-rook-19.0.0-2145.g3be9a5c9.e 539 kB/s | 49 kB 00:00 2024-03-24T00:36:21.207 INFO:teuthology.orchestra.run.smithi067.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2145.g3 197 kB/s | 16 kB 00:00 2024-03-24T00:36:21.348 INFO:teuthology.orchestra.run.smithi067.stdout:(35/174): ceph-volume-19.0.0-2145.g3be9a5c9.el9 1.8 MB/s | 264 kB 00:00 2024-03-24T00:36:21.532 INFO:teuthology.orchestra.run.smithi067.stdout:(36/174): ceph-mgr-diskprediction-local-19.0.0- 8.6 MB/s | 7.4 MB 00:00 2024-03-24T00:36:21.566 INFO:teuthology.orchestra.run.smithi067.stdout:(37/174): cephadm-19.0.0-2145.g3be9a5c9.el9.noa 3.4 MB/s | 748 kB 00:00 2024-03-24T00:36:21.591 INFO:teuthology.orchestra.run.smithi067.stdout:(38/174): cephfs-top-19.0.0-2145.g3be9a5c9.el9. 312 kB/s | 18 kB 00:00 2024-03-24T00:36:22.050 INFO:teuthology.orchestra.run.smithi067.stdout:(39/174): ceph-test-19.0.0-2145.g3be9a5c9.el9.x 14 MB/s | 48 MB 00:03 2024-03-24T00:36:22.183 INFO:teuthology.orchestra.run.smithi067.stdout:(40/174): automake-1.16.2-8.el9.noarch.rpm 1.0 MB/s | 662 kB 00:00 2024-03-24T00:36:22.225 INFO:teuthology.orchestra.run.smithi067.stdout:(41/174): bison-3.7.4-5.el9.x86_64.rpm 1.5 MB/s | 920 kB 00:00 2024-03-24T00:36:22.250 INFO:teuthology.orchestra.run.smithi067.stdout:(42/174): boost-program-options-1.75.0-8.el9.x8 617 kB/s | 106 kB 00:00 2024-03-24T00:36:22.275 INFO:teuthology.orchestra.run.smithi067.stdout:(43/174): flex-2.6.4-9.el9.x86_64.rpm 3.3 MB/s | 307 kB 00:00 2024-03-24T00:36:22.301 INFO:teuthology.orchestra.run.smithi067.stdout:(44/174): flexiblas-3.0.4-8.el9.x86_64.rpm 639 kB/s | 32 kB 00:00 2024-03-24T00:36:22.359 INFO:teuthology.orchestra.run.smithi067.stdout:(45/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 28 MB/s | 3.0 MB 00:00 2024-03-24T00:36:22.385 INFO:teuthology.orchestra.run.smithi067.stdout:(46/174): flexiblas-openblas-openmp-3.0.4-8.el9 160 kB/s | 17 kB 00:00 2024-03-24T00:36:22.410 INFO:teuthology.orchestra.run.smithi067.stdout:(47/174): libaio-devel-0.3.111-13.el9.x86_64.rp 95 kB/s | 10 kB 00:00 2024-03-24T00:36:22.435 INFO:teuthology.orchestra.run.smithi067.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 885 kB/s | 44 kB 00:00 2024-03-24T00:36:22.452 INFO:teuthology.orchestra.run.smithi067.stdout:(49/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.3 MB/s | 159 kB 00:00 2024-03-24T00:36:22.477 INFO:teuthology.orchestra.run.smithi067.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-03-24T00:36:22.511 INFO:teuthology.orchestra.run.smithi067.stdout:(51/174): libtool-2.4.6-45.el9.x86_64.rpm 7.7 MB/s | 578 kB 00:00 2024-03-24T00:36:22.536 INFO:teuthology.orchestra.run.smithi067.stdout:(52/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 807 kB/s | 46 kB 00:00 2024-03-24T00:36:22.561 INFO:teuthology.orchestra.run.smithi067.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.3 MB/s | 282 kB 00:00 2024-03-24T00:36:22.586 INFO:teuthology.orchestra.run.smithi067.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 2.4 MB/s | 187 kB 00:00 2024-03-24T00:36:22.612 INFO:teuthology.orchestra.run.smithi067.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 690 kB/s | 34 kB 00:00 2024-03-24T00:36:22.645 INFO:teuthology.orchestra.run.smithi067.stdout:(56/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.5 MB/s | 217 kB 00:00 2024-03-24T00:36:22.713 INFO:teuthology.orchestra.run.smithi067.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 38 MB/s | 4.7 MB 00:00 2024-03-24T00:36:22.754 INFO:teuthology.orchestra.run.smithi067.stdout:(58/174): postgresql-13.11-1.el9.x86_64.rpm 11 MB/s | 1.5 MB 00:00 2024-03-24T00:36:22.779 INFO:teuthology.orchestra.run.smithi067.stdout:(59/174): postgresql-contrib-13.11-1.el9.x86_64 6.0 MB/s | 811 kB 00:00 2024-03-24T00:36:22.805 INFO:teuthology.orchestra.run.smithi067.stdout:(60/174): postgresql-private-libs-13.11-1.el9.x 1.5 MB/s | 134 kB 00:00 2024-03-24T00:36:22.847 INFO:teuthology.orchestra.run.smithi067.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 15 MB/s | 1.0 MB 00:00 2024-03-24T00:36:22.930 INFO:teuthology.orchestra.run.smithi067.stdout:(62/174): postgresql-server-13.11-1.el9.x86_64. 38 MB/s | 5.7 MB 00:00 2024-03-24T00:36:22.956 INFO:teuthology.orchestra.run.smithi067.stdout:(63/174): python3-attrs-20.3.0-7.el9.noarch.rpm 515 kB/s | 78 kB 00:00 2024-03-24T00:36:23.039 INFO:teuthology.orchestra.run.smithi067.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 30 MB/s | 5.8 MB 00:00 2024-03-24T00:36:23.065 INFO:teuthology.orchestra.run.smithi067.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 187 kB/s | 24 kB 00:00 2024-03-24T00:36:23.090 INFO:teuthology.orchestra.run.smithi067.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 138 kB/s | 18 kB 00:00 2024-03-24T00:36:23.115 INFO:teuthology.orchestra.run.smithi067.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2024-03-24T00:36:23.140 INFO:teuthology.orchestra.run.smithi067.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 633 kB/s | 32 kB 00:00 2024-03-24T00:36:23.216 INFO:teuthology.orchestra.run.smithi067.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 38 MB/s | 4.8 MB 00:00 2024-03-24T00:36:23.241 INFO:teuthology.orchestra.run.smithi067.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.7 MB/s | 217 kB 00:00 2024-03-24T00:36:23.266 INFO:teuthology.orchestra.run.smithi067.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.4 MB/s | 174 kB 00:00 2024-03-24T00:36:23.283 INFO:teuthology.orchestra.run.smithi067.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 1.0 MB/s | 69 kB 00:00 2024-03-24T00:36:23.308 INFO:teuthology.orchestra.run.smithi067.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 986 kB/s | 41 kB 00:00 2024-03-24T00:36:23.334 INFO:teuthology.orchestra.run.smithi067.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 3.5 MB/s | 238 kB 00:00 2024-03-24T00:36:23.358 INFO:teuthology.orchestra.run.smithi067.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-24T00:36:23.384 INFO:teuthology.orchestra.run.smithi067.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 947 kB/s | 47 kB 00:00 2024-03-24T00:36:23.409 INFO:teuthology.orchestra.run.smithi067.stdout:(77/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.7 MB/s | 211 kB 00:00 2024-03-24T00:36:23.434 INFO:teuthology.orchestra.run.smithi067.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 568 kB/s | 43 kB 00:00 2024-03-24T00:36:23.710 INFO:teuthology.orchestra.run.smithi067.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 48 MB/s | 16 MB 00:00 2024-03-24T00:36:23.735 INFO:teuthology.orchestra.run.smithi067.stdout:(80/174): python3-toml-0.10.2-6.el9.noarch.rpm 130 kB/s | 39 kB 00:00 2024-03-24T00:36:23.760 INFO:teuthology.orchestra.run.smithi067.stdout:(81/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 127 kB/s | 41 kB 00:00 2024-03-24T00:36:23.785 INFO:teuthology.orchestra.run.smithi067.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 3.9 MB/s | 300 kB 00:00 2024-03-24T00:36:23.810 INFO:teuthology.orchestra.run.smithi067.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-24T00:36:23.835 INFO:teuthology.orchestra.run.smithi067.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 839 kB/s | 63 kB 00:00 2024-03-24T00:36:23.861 INFO:teuthology.orchestra.run.smithi067.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 585 kB/s | 44 kB 00:00 2024-03-24T00:36:23.886 INFO:teuthology.orchestra.run.smithi067.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-24T00:36:23.911 INFO:teuthology.orchestra.run.smithi067.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-03-24T00:36:23.945 INFO:teuthology.orchestra.run.smithi067.stdout:(88/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-24T00:36:23.970 INFO:teuthology.orchestra.run.smithi067.stdout:(89/174): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 103 kB 00:00 2024-03-24T00:36:23.995 INFO:teuthology.orchestra.run.smithi067.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 1.5 MB/s | 124 kB 00:00 2024-03-24T00:36:24.020 INFO:teuthology.orchestra.run.smithi067.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.4 MB/s | 187 kB 00:00 2024-03-24T00:36:25.162 INFO:teuthology.orchestra.run.smithi067.stdout:(92/174): libquadmath-11.4.1-3.el9.x86_64.rpm 171 kB/s | 195 kB 00:01 2024-03-24T00:36:25.296 INFO:teuthology.orchestra.run.smithi067.stdout:(93/174): jq-1.6-16.el9.x86_64.rpm 144 kB/s | 187 kB 00:01 2024-03-24T00:36:25.438 INFO:teuthology.orchestra.run.smithi067.stdout:(94/174): elfutils-libelf-devel-0.190-2.el9.x86 152 kB/s | 21 kB 00:00 2024-03-24T00:36:25.496 INFO:teuthology.orchestra.run.smithi067.stdout:(95/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 438 kB/s | 25 kB 00:00 2024-03-24T00:36:25.582 INFO:teuthology.orchestra.run.smithi067.stdout:(96/174): python3-requests-2.25.1-8.el9.noarch. 299 kB/s | 125 kB 00:00 2024-03-24T00:36:25.848 INFO:teuthology.orchestra.run.smithi067.stdout:(97/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 12 MB/s | 4.1 MB 00:00 2024-03-24T00:36:25.898 INFO:teuthology.orchestra.run.smithi067.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 773 kB/s | 245 kB 00:00 2024-03-24T00:36:25.914 INFO:teuthology.orchestra.run.smithi067.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 3.6 MB/s | 248 kB 00:00 2024-03-24T00:36:25.973 INFO:teuthology.orchestra.run.smithi067.stdout:(100/174): qatlib-23.11.0-1.el9.x86_64.rpm 2.8 MB/s | 216 kB 00:00 2024-03-24T00:36:25.998 INFO:teuthology.orchestra.run.smithi067.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 417 kB/s | 35 kB 00:00 2024-03-24T00:36:26.032 INFO:teuthology.orchestra.run.smithi067.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 764 kB/s | 44 kB 00:00 2024-03-24T00:36:26.074 INFO:teuthology.orchestra.run.smithi067.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.2 MB/s | 94 kB 00:00 2024-03-24T00:36:26.291 INFO:teuthology.orchestra.run.smithi067.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 86 kB/s | 22 kB 00:00 2024-03-24T00:36:26.766 INFO:teuthology.orchestra.run.smithi067.stdout:(105/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.1 MB/s | 548 kB 00:00 2024-03-24T00:36:26.825 INFO:teuthology.orchestra.run.smithi067.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 1.1 MB/s | 863 kB 00:00 2024-03-24T00:36:26.858 INFO:teuthology.orchestra.run.smithi067.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 3.3 MB/s | 309 kB 00:00 2024-03-24T00:36:27.010 INFO:teuthology.orchestra.run.smithi067.stdout:(108/174): libgfortran-11.4.1-3.el9.x86_64.rpm 267 kB/s | 803 kB 00:03 2024-03-24T00:36:27.076 INFO:teuthology.orchestra.run.smithi067.stdout:(109/174): grpc-data-1.46.7-10.el9.noarch.rpm 78 kB/s | 19 kB 00:00 2024-03-24T00:36:27.167 INFO:teuthology.orchestra.run.smithi067.stdout:(110/174): libarrow-9.0.0-10.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-03-24T00:36:27.234 INFO:teuthology.orchestra.run.smithi067.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 371 kB/s | 49 kB 00:00 2024-03-24T00:36:27.259 INFO:teuthology.orchestra.run.smithi067.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 742 kB/s | 67 kB 00:00 2024-03-24T00:36:27.301 INFO:teuthology.orchestra.run.smithi067.stdout:(113/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 89 kB/s | 26 kB 00:00 2024-03-24T00:36:27.343 INFO:teuthology.orchestra.run.smithi067.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 9.8 MB/s | 838 kB 00:00 2024-03-24T00:36:27.403 INFO:teuthology.orchestra.run.smithi067.stdout:(115/174): luarocks-3.9.2-1.el9.noarch.rpm 900 kB/s | 151 kB 00:00 2024-03-24T00:36:27.427 INFO:teuthology.orchestra.run.smithi067.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 520 kB/s | 43 kB 00:00 2024-03-24T00:36:27.478 INFO:teuthology.orchestra.run.smithi067.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 432 kB/s | 32 kB 00:00 2024-03-24T00:36:27.503 INFO:teuthology.orchestra.run.smithi067.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 199 kB/s | 15 kB 00:00 2024-03-24T00:36:27.553 INFO:teuthology.orchestra.run.smithi067.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-03-24T00:36:27.611 INFO:teuthology.orchestra.run.smithi067.stdout:(120/174): python3-asyncssh-2.13.2-5.el9.noarch 1.7 MB/s | 548 kB 00:00 2024-03-24T00:36:27.645 INFO:teuthology.orchestra.run.smithi067.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.2 MB/s | 303 kB 00:00 2024-03-24T00:36:27.670 INFO:teuthology.orchestra.run.smithi067.stdout:(122/174): python3-cherrypy-18.6.1-2.el9.noarch 2.1 MB/s | 358 kB 00:00 2024-03-24T00:36:27.696 INFO:teuthology.orchestra.run.smithi067.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 421 kB/s | 35 kB 00:00 2024-03-24T00:36:27.721 INFO:teuthology.orchestra.run.smithi067.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 2.6 MB/s | 203 kB 00:00 2024-03-24T00:36:27.779 INFO:teuthology.orchestra.run.smithi067.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 1.7 MB/s | 144 kB 00:00 2024-03-24T00:36:27.805 INFO:teuthology.orchestra.run.smithi067.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 128 kB/s | 11 kB 00:00 2024-03-24T00:36:27.855 INFO:teuthology.orchestra.run.smithi067.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 235 kB/s | 18 kB 00:00 2024-03-24T00:36:27.889 INFO:teuthology.orchestra.run.smithi067.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 275 kB/s | 23 kB 00:00 2024-03-24T00:36:27.948 INFO:teuthology.orchestra.run.smithi067.stdout:(129/174): python3-grpcio-1.46.7-10.el9.x86_64. 7.4 MB/s | 2.0 MB 00:00 2024-03-24T00:36:27.973 INFO:teuthology.orchestra.run.smithi067.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 165 kB/s | 19 kB 00:00 2024-03-24T00:36:27.998 INFO:teuthology.orchestra.run.smithi067.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 181 kB/s | 20 kB 00:00 2024-03-24T00:36:28.057 INFO:teuthology.orchestra.run.smithi067.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 9.4 MB/s | 1.0 MB 00:00 2024-03-24T00:36:28.082 INFO:teuthology.orchestra.run.smithi067.stdout:(133/174): python3-logutils-0.3.5-21.el9.noarch 554 kB/s | 46 kB 00:00 2024-03-24T00:36:28.107 INFO:teuthology.orchestra.run.smithi067.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 725 kB/s | 79 kB 00:00 2024-03-24T00:36:28.132 INFO:teuthology.orchestra.run.smithi067.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 766 kB/s | 58 kB 00:00 2024-03-24T00:36:28.158 INFO:teuthology.orchestra.run.smithi067.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-03-24T00:36:28.183 INFO:teuthology.orchestra.run.smithi067.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 218 kB/s | 16 kB 00:00 2024-03-24T00:36:28.208 INFO:teuthology.orchestra.run.smithi067.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-03-24T00:36:28.234 INFO:teuthology.orchestra.run.smithi067.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 374 kB/s | 28 kB 00:00 2024-03-24T00:36:28.259 INFO:teuthology.orchestra.run.smithi067.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 407 kB/s | 31 kB 00:00 2024-03-24T00:36:28.284 INFO:teuthology.orchestra.run.smithi067.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-03-24T00:36:28.309 INFO:teuthology.orchestra.run.smithi067.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 785 kB/s | 59 kB 00:00 2024-03-24T00:36:28.334 INFO:teuthology.orchestra.run.smithi067.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 473 kB/s | 36 kB 00:00 2024-03-24T00:36:28.360 INFO:teuthology.orchestra.run.smithi067.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 675 kB/s | 51 kB 00:00 2024-03-24T00:36:28.386 INFO:teuthology.orchestra.run.smithi067.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-24T00:36:28.410 INFO:teuthology.orchestra.run.smithi067.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-03-24T00:36:28.444 INFO:teuthology.orchestra.run.smithi067.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 5.0 MB/s | 427 kB 00:00 2024-03-24T00:36:28.461 INFO:teuthology.orchestra.run.smithi067.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 264 kB/s | 20 kB 00:00 2024-03-24T00:36:28.494 INFO:teuthology.orchestra.run.smithi067.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-03-24T00:36:28.595 INFO:teuthology.orchestra.run.smithi067.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-03-24T00:36:28.720 INFO:teuthology.orchestra.run.smithi067.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 2.5 MB/s | 321 kB 00:00 2024-03-24T00:36:29.120 INFO:teuthology.orchestra.run.smithi067.stdout:(152/174): librbd1-19.0.0-2145.g3be9a5c9.el9.x8 5.0 MB/s | 3.1 MB 00:00 2024-03-24T00:36:29.170 INFO:teuthology.orchestra.run.smithi067.stdout:(153/174): librados2-19.0.0-2145.g3be9a5c9.el9. 4.8 MB/s | 3.4 MB 00:00 2024-03-24T00:36:29.363 INFO:teuthology.orchestra.run.smithi067.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 219 kB/s | 42 kB 00:00 2024-03-24T00:36:30.238 INFO:teuthology.orchestra.run.smithi067.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 284 kB/s | 248 kB 00:00 2024-03-24T00:36:30.355 INFO:teuthology.orchestra.run.smithi067.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 612 kB/s | 71 kB 00:00 2024-03-24T00:36:30.446 INFO:teuthology.orchestra.run.smithi067.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 399 kB/s | 36 kB 00:00 2024-03-24T00:36:30.780 INFO:teuthology.orchestra.run.smithi067.stdout:(158/174): elfutils-libelf-0.190-2.el9.x86_64.r 578 kB/s | 192 kB 00:00 2024-03-24T00:36:31.964 INFO:teuthology.orchestra.run.smithi067.stdout:(159/174): elfutils-libs-0.190-2.el9.x86_64.rpm 216 kB/s | 256 kB 00:01 2024-03-24T00:36:32.048 INFO:teuthology.orchestra.run.smithi067.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 283 kB/s | 23 kB 00:00 2024-03-24T00:36:32.339 INFO:teuthology.orchestra.run.smithi067.stdout:(161/174): libblkid-2.37.4-18.el9.x86_64.rpm 372 kB/s | 108 kB 00:00 2024-03-24T00:36:32.765 INFO:teuthology.orchestra.run.smithi067.stdout:(162/174): libfdisk-2.37.4-18.el9.x86_64.rpm 364 kB/s | 155 kB 00:00 2024-03-24T00:36:32.931 INFO:teuthology.orchestra.run.smithi067.stdout:(163/174): libmount-2.37.4-18.el9.x86_64.rpm 815 kB/s | 136 kB 00:00 2024-03-24T00:36:33.023 INFO:teuthology.orchestra.run.smithi067.stdout:(164/174): libsmartcols-2.37.4-18.el9.x86_64.rp 698 kB/s | 64 kB 00:00 2024-03-24T00:36:33.266 INFO:teuthology.orchestra.run.smithi067.stdout:(165/174): libuuid-2.37.4-18.el9.x86_64.rpm 119 kB/s | 29 kB 00:00 2024-03-24T00:36:33.731 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-24T00:36:33.731 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /sys 2024-03-24T00:36:33.731 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /proc 2024-03-24T00:36:33.731 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /mnt 2024-03-24T00:36:33.731 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /var/tmp 2024-03-24T00:36:33.731 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /home 2024-03-24T00:36:33.731 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /root 2024-03-24T00:36:33.731 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /tmp 2024-03-24T00:36:33.731 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:36:33.903 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-24T00:36:33.918 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-24T00:36:33.918 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:36:33.918 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-24T00:36:33.919 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:36:34.574 INFO:teuthology.orchestra.run.smithi067.stdout:(166/174): NetworkManager-libnm-1.47.2-1.el9.x8 344 kB/s | 1.8 MB 00:05 2024-03-24T00:36:34.674 INFO:teuthology.orchestra.run.smithi067.stdout:(167/174): NetworkManager-1.47.2-1.el9.x86_64.r 396 kB/s | 2.3 MB 00:05 2024-03-24T00:36:35.384 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-24T00:36:35.410 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-24T00:36:35.410 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:36:35.410 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-24T00:36:35.410 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T00:36:35.410 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T00:36:35.410 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:36:35.637 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-24T00:36:35.660 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-24T00:36:35.660 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:36:35.660 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-24T00:36:35.660 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T00:36:35.660 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T00:36:35.660 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:36:36.097 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-24T00:36:36.119 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-24T00:36:36.120 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:36:36.120 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-24T00:36:36.120 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T00:36:36.120 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T00:36:36.120 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:36:36.216 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-24T00:36:36.654 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-24T00:36:36.686 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-24T00:36:36.720 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 139/198 2024-03-24T00:36:36.779 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 140/198 2024-03-24T00:36:36.813 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-24T00:36:37.044 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-24T00:36:37.080 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-24T00:36:37.857 INFO:teuthology.orchestra.run.smithi067.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 270 kB/s | 1.2 MB 00:04 2024-03-24T00:36:38.038 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-24T00:36:38.065 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-24T00:36:38.165 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-24T00:36:38.328 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 144/198 2024-03-24T00:36:38.361 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-24T00:36:38.384 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-24T00:36:38.384 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:36:38.384 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-24T00:36:38.384 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T00:36:38.384 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T00:36:38.384 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:36:38.429 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-24T00:36:38.440 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-24T00:36:38.517 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 147/198 2024-03-24T00:36:38.824 INFO:teuthology.orchestra.run.smithi067.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 481 kB/s | 465 kB 00:00 2024-03-24T00:36:38.891 INFO:teuthology.orchestra.run.smithi067.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 468 kB/s | 31 kB 00:00 2024-03-24T00:36:39.390 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-24T00:36:39.426 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-24T00:36:39.449 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-24T00:36:39.449 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:36:39.449 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-24T00:36:39.449 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T00:36:39.449 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T00:36:39.449 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:36:39.494 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-24T00:36:39.516 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-24T00:36:39.517 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:36:39.517 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-24T00:36:39.517 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:36:39.571 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-24T00:36:39.593 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-24T00:36:39.593 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:36:39.593 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-24T00:36:39.593 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T00:36:39.594 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T00:36:39.594 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:36:39.897 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-24T00:36:39.919 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-24T00:36:39.920 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:36:39.920 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-24T00:36:39.920 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T00:36:39.920 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T00:36:39.920 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:36:40.466 INFO:teuthology.orchestra.run.smithi067.stdout:(171/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 375 kB/s | 2.2 MB 00:05 2024-03-24T00:36:40.558 INFO:teuthology.orchestra.run.smithi067.stdout:(172/174): util-linux-2.37.4-18.el9.x86_64.rpm 397 kB/s | 2.3 MB 00:05 2024-03-24T00:36:40.650 INFO:teuthology.orchestra.run.smithi067.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 405 kB/s | 74 kB 00:00 2024-03-24T00:36:42.342 INFO:teuthology.orchestra.run.smithi067.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 318 kB/s | 1.1 MB 00:03 2024-03-24T00:36:42.344 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-03-24T00:36:42.345 INFO:teuthology.orchestra.run.smithi067.stdout:Total 8.5 MB/s | 224 MB 00:26 2024-03-24T00:36:42.796 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-03-24T00:36:43.093 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-03-24T00:36:43.093 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-03-24T00:36:44.383 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-24T00:36:44.425 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 154/198 2024-03-24T00:36:44.456 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 155/198 2024-03-24T00:36:44.484 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-24T00:36:44.666 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 157/198 2024-03-24T00:36:44.791 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-24T00:36:44.839 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 159/198 2024-03-24T00:36:44.900 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-24T00:36:44.967 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-24T00:36:45.004 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-24T00:36:45.052 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 163/198 2024-03-24T00:36:45.070 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-24T00:36:45.108 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-24T00:36:45.173 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-24T00:36:45.239 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-24T00:36:45.278 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-24T00:36:45.363 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-03-24T00:36:45.364 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-03-24T00:36:45.866 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T00:36:45.918 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T00:36:45.939 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T00:36:46.761 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 170/198 2024-03-24T00:36:46.859 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-24T00:36:46.900 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-24T00:36:46.982 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-24T00:36:47.026 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-24T00:36:47.051 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-24T00:36:47.086 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-24T00:36:47.121 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-24T00:36:47.121 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-24T00:36:47.251 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-24T00:36:47.290 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-24T00:36:47.290 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-24T00:36:47.385 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-24T00:36:47.427 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-24T00:36:47.463 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-24T00:36:47.497 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-24T00:36:47.522 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-24T00:36:47.556 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-24T00:36:47.607 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-24T00:36:47.649 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-24T00:36:47.683 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-24T00:36:47.709 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-24T00:36:47.734 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-24T00:36:47.761 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-24T00:36:47.768 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T00:36:47.768 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T00:36:47.948 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T00:36:47.948 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-24T00:36:48.024 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-24T00:36:48.057 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-24T00:36:48.098 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-24T00:36:48.125 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-24T00:36:48.159 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-24T00:36:48.234 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-24T00:36:52.548 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-03-24T00:36:52.638 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-24T00:36:52.681 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-24T00:36:52.719 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-24T00:36:52.727 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-24T00:36:52.772 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-24T00:36:52.824 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-24T00:36:52.856 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-24T00:36:52.886 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-24T00:36:52.939 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-24T00:36:53.283 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-24T00:36:53.370 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-24T00:36:53.494 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-24T00:36:53.521 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-24T00:36:53.605 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-24T00:36:53.637 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-24T00:36:53.692 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-24T00:36:53.868 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-24T00:36:53.975 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-24T00:36:54.002 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-24T00:36:54.107 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-24T00:36:54.136 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-24T00:36:54.184 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-24T00:36:54.223 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-24T00:36:54.246 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-24T00:36:54.261 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-24T00:36:54.449 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-24T00:36:54.486 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-24T00:36:54.526 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-24T00:36:54.700 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-24T00:36:54.812 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-24T00:36:54.873 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-24T00:36:54.956 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T00:36:55.015 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T00:36:55.230 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T00:36:55.342 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-24T00:36:55.435 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-24T00:36:55.587 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-24T00:36:55.614 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-24T00:36:55.650 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-24T00:36:55.725 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-24T00:36:55.778 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-24T00:36:55.816 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-24T00:36:55.882 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-24T00:36:55.915 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-24T00:36:55.959 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-24T00:36:55.991 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-24T00:36:56.018 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-24T00:36:56.089 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-24T00:36:56.124 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 39/198 2024-03-24T00:36:56.187 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 40/198 2024-03-24T00:36:56.221 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 41/198 2024-03-24T00:36:56.292 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-24T00:36:56.335 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-24T00:36:56.414 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-24T00:36:56.549 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-24T00:36:56.597 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-24T00:36:57.088 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-24T00:36:57.088 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 1/198 2024-03-24T00:36:57.088 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2/198 2024-03-24T00:36:57.088 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 3/198 2024-03-24T00:36:57.088 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 4/198 2024-03-24T00:36:57.088 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 5/198 2024-03-24T00:36:57.088 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 6/198 2024-03-24T00:36:57.088 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 7/198 2024-03-24T00:36:57.088 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 8/198 2024-03-24T00:36:57.088 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 9/198 2024-03-24T00:36:57.088 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-24T00:36:57.088 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-24T00:36:57.089 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-24T00:36:57.089 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 13/198 2024-03-24T00:36:57.089 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 14/198 2024-03-24T00:36:57.089 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 15/198 2024-03-24T00:36:57.089 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-24T00:36:57.089 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 17/198 2024-03-24T00:36:57.089 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 18/198 2024-03-24T00:36:57.089 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 19/198 2024-03-24T00:36:57.089 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 20/198 2024-03-24T00:36:57.089 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 21/198 2024-03-24T00:36:57.089 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 22/198 2024-03-24T00:36:57.089 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 23/198 2024-03-24T00:36:57.089 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 24/198 2024-03-24T00:36:57.089 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 25/198 2024-03-24T00:36:57.089 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 26/198 2024-03-24T00:36:57.090 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 27/198 2024-03-24T00:36:57.090 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 28/198 2024-03-24T00:36:57.090 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 29/198 2024-03-24T00:36:57.105 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 30/198 2024-03-24T00:36:57.105 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 31/198 2024-03-24T00:36:57.106 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 32/198 2024-03-24T00:36:57.106 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 33/198 2024-03-24T00:36:57.106 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 34/198 2024-03-24T00:36:57.106 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 35/198 2024-03-24T00:36:57.106 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 36/198 2024-03-24T00:36:57.106 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 37/198 2024-03-24T00:36:57.106 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 38/198 2024-03-24T00:36:57.106 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 39/198 2024-03-24T00:36:57.106 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-24T00:36:57.106 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-24T00:36:57.106 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-24T00:36:57.106 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-24T00:36:57.106 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-24T00:36:57.107 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-24T00:36:57.107 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-24T00:36:57.107 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-24T00:36:57.107 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-24T00:36:57.107 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-24T00:36:57.107 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-24T00:36:57.107 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-24T00:36:57.107 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-24T00:36:57.107 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-24T00:36:57.107 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-24T00:36:57.107 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-24T00:36:57.107 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-24T00:36:57.107 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-24T00:36:57.107 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-24T00:36:57.108 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-24T00:36:57.108 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-24T00:36:57.108 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-24T00:36:57.108 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-24T00:36:57.108 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-24T00:36:57.108 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-24T00:36:57.108 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-24T00:36:57.108 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-24T00:36:57.108 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-24T00:36:57.108 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-24T00:36:57.108 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-24T00:36:57.108 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-24T00:36:57.108 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-24T00:36:57.108 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-24T00:36:57.108 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-24T00:36:57.109 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-24T00:36:57.109 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-24T00:36:57.109 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-24T00:36:57.109 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-24T00:36:57.109 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-24T00:36:57.109 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-24T00:36:57.109 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-24T00:36:57.109 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-24T00:36:57.109 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-24T00:36:57.109 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-24T00:36:57.109 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-24T00:36:57.109 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-24T00:36:57.109 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-24T00:36:57.109 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-24T00:36:57.109 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-24T00:36:57.110 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-24T00:36:57.110 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-24T00:36:57.110 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-24T00:36:57.110 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-24T00:36:57.110 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-24T00:36:57.110 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-24T00:36:57.110 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-24T00:36:57.110 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-24T00:36:57.110 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-24T00:36:57.110 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-24T00:36:57.110 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-24T00:36:57.110 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-24T00:36:57.110 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-24T00:36:57.110 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-24T00:36:57.110 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-24T00:36:57.111 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-24T00:36:57.111 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-24T00:36:57.111 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-24T00:36:57.111 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-24T00:36:57.111 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-24T00:36:57.111 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-24T00:36:57.111 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-24T00:36:57.111 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-24T00:36:57.111 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-24T00:36:57.111 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-24T00:36:57.111 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-24T00:36:57.111 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-24T00:36:57.111 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-24T00:36:57.111 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-24T00:36:57.112 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-24T00:36:57.112 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-24T00:36:57.112 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-24T00:36:57.112 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-24T00:36:57.112 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-24T00:36:57.112 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-24T00:36:57.112 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-24T00:36:57.112 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-24T00:36:57.112 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-24T00:36:57.112 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-24T00:36:57.112 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-24T00:36:57.112 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-24T00:36:57.112 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-24T00:36:57.112 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-24T00:36:57.112 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-24T00:36:57.112 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-24T00:36:57.113 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-24T00:36:57.113 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-24T00:36:57.113 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-24T00:36:57.113 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-24T00:36:57.113 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-24T00:36:57.113 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-24T00:36:57.113 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-24T00:36:57.113 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-24T00:36:57.113 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-24T00:36:57.113 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-24T00:36:57.113 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-24T00:36:57.113 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-24T00:36:57.113 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-24T00:36:57.113 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-24T00:36:57.113 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-24T00:36:57.114 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-24T00:36:57.114 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-24T00:36:57.114 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-24T00:36:57.114 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-24T00:36:57.114 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-24T00:36:57.114 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-24T00:36:57.114 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-24T00:36:57.114 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-24T00:36:57.114 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-24T00:36:57.114 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-24T00:36:57.114 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-24T00:36:57.114 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-24T00:36:57.114 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-24T00:36:57.114 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-24T00:36:57.114 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-24T00:36:57.115 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-24T00:36:57.115 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-24T00:36:57.115 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-24T00:36:57.115 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-24T00:36:57.115 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-24T00:36:57.115 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-24T00:36:57.115 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-24T00:36:57.115 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-24T00:36:57.115 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-24T00:36:57.115 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-24T00:36:57.115 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-24T00:36:57.115 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-24T00:36:57.115 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-24T00:36:57.115 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-24T00:36:57.116 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-24T00:36:57.116 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-24T00:36:57.116 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-24T00:36:57.116 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-24T00:36:57.116 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-24T00:36:57.116 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-24T00:36:57.116 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-24T00:36:57.116 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-24T00:36:57.116 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-24T00:36:57.116 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-24T00:36:57.116 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-24T00:36:57.116 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-24T00:36:57.116 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-24T00:36:57.116 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-24T00:36:57.116 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-24T00:36:57.117 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-24T00:36:57.117 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-24T00:36:57.117 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-24T00:36:57.117 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-24T00:36:57.117 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-24T00:36:57.151 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-24T00:36:57.227 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-24T00:36:57.301 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-24T00:36:57.353 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-24T00:36:57.361 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-24T00:36:57.393 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-24T00:36:57.443 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-24T00:36:57.480 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 53/198 2024-03-24T00:36:57.519 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-24T00:36:57.795 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-24T00:36:57.977 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-24T00:36:58.056 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-24T00:36:58.100 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-24T00:36:58.240 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-24T00:36:58.286 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-24T00:36:58.342 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-24T00:36:58.415 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-24T00:36:58.445 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-24T00:36:58.482 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-24T00:36:58.529 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-24T00:36:58.564 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-24T00:36:58.609 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-24T00:36:58.649 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-24T00:36:58.791 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-24T00:36:58.830 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-24T00:36:58.860 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-24T00:36:58.903 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-24T00:36:59.000 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-24T00:36:59.034 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-24T00:36:59.075 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-24T00:36:59.100 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-24T00:36:59.124 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-24T00:36:59.221 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-24T00:36:59.356 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-24T00:36:59.464 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-24T00:36:59.517 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-24T00:36:59.575 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-24T00:36:59.640 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-24T00:36:59.841 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-24T00:36:59.887 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-24T00:36:59.948 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-24T00:36:59.948 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:36:59.948 INFO:teuthology.orchestra.run.smithi072.stdout:Upgraded: 2024-03-24T00:36:59.948 INFO:teuthology.orchestra.run.smithi072.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-24T00:36:59.948 INFO:teuthology.orchestra.run.smithi072.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-24T00:36:59.949 INFO:teuthology.orchestra.run.smithi072.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-24T00:36:59.949 INFO:teuthology.orchestra.run.smithi072.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-24T00:36:59.949 INFO:teuthology.orchestra.run.smithi072.stdout: acl-2.3.1-4.el9.x86_64 2024-03-24T00:36:59.950 INFO:teuthology.orchestra.run.smithi072.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-24T00:36:59.950 INFO:teuthology.orchestra.run.smithi072.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-24T00:36:59.950 INFO:teuthology.orchestra.run.smithi072.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-24T00:36:59.950 INFO:teuthology.orchestra.run.smithi072.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-24T00:36:59.950 INFO:teuthology.orchestra.run.smithi072.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-24T00:36:59.951 INFO:teuthology.orchestra.run.smithi072.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-24T00:36:59.951 INFO:teuthology.orchestra.run.smithi072.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-24T00:36:59.951 INFO:teuthology.orchestra.run.smithi072.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-24T00:36:59.951 INFO:teuthology.orchestra.run.smithi072.stdout: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:36:59.951 INFO:teuthology.orchestra.run.smithi072.stdout: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:36:59.951 INFO:teuthology.orchestra.run.smithi072.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-24T00:36:59.952 INFO:teuthology.orchestra.run.smithi072.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-24T00:36:59.952 INFO:teuthology.orchestra.run.smithi072.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-24T00:36:59.952 INFO:teuthology.orchestra.run.smithi072.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-24T00:36:59.952 INFO:teuthology.orchestra.run.smithi072.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-24T00:36:59.952 INFO:teuthology.orchestra.run.smithi072.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-24T00:36:59.952 INFO:teuthology.orchestra.run.smithi072.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-24T00:36:59.952 INFO:teuthology.orchestra.run.smithi072.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-24T00:36:59.952 INFO:teuthology.orchestra.run.smithi072.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-24T00:36:59.952 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-03-24T00:36:59.952 INFO:teuthology.orchestra.run.smithi072.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-24T00:36:59.952 INFO:teuthology.orchestra.run.smithi072.stdout: automake-1.16.2-8.el9.noarch 2024-03-24T00:36:59.952 INFO:teuthology.orchestra.run.smithi072.stdout: bison-3.7.4-5.el9.x86_64 2024-03-24T00:36:59.952 INFO:teuthology.orchestra.run.smithi072.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-24T00:36:59.952 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:36:59.952 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:36:59.953 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:36:59.953 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:36:59.953 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:36:59.953 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:36:59.953 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:36:59.953 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:36:59.953 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:36:59.953 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:36:59.953 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:36:59.953 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:36:59.953 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:36:59.953 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:36:59.953 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:36:59.954 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:36:59.954 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:36:59.954 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:36:59.954 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:36:59.954 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:36:59.954 INFO:teuthology.orchestra.run.smithi072.stdout: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:36:59.954 INFO:teuthology.orchestra.run.smithi072.stdout: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:36:59.954 INFO:teuthology.orchestra.run.smithi072.stdout: cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:36:59.954 INFO:teuthology.orchestra.run.smithi072.stdout: cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:36:59.954 INFO:teuthology.orchestra.run.smithi072.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-24T00:36:59.954 INFO:teuthology.orchestra.run.smithi072.stdout: flex-2.6.4-9.el9.x86_64 2024-03-24T00:36:59.954 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-24T00:36:59.954 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-24T00:36:59.955 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-24T00:36:59.955 INFO:teuthology.orchestra.run.smithi072.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-24T00:36:59.955 INFO:teuthology.orchestra.run.smithi072.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-24T00:36:59.955 INFO:teuthology.orchestra.run.smithi072.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-24T00:36:59.955 INFO:teuthology.orchestra.run.smithi072.stdout: jq-1.6-16.el9.x86_64 2024-03-24T00:36:59.955 INFO:teuthology.orchestra.run.smithi072.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-24T00:36:59.955 INFO:teuthology.orchestra.run.smithi072.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-24T00:36:59.955 INFO:teuthology.orchestra.run.smithi072.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-24T00:36:59.955 INFO:teuthology.orchestra.run.smithi072.stdout: libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:36:59.955 INFO:teuthology.orchestra.run.smithi072.stdout: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:36:59.955 INFO:teuthology.orchestra.run.smithi072.stdout: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:36:59.955 INFO:teuthology.orchestra.run.smithi072.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-24T00:36:59.955 INFO:teuthology.orchestra.run.smithi072.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-24T00:36:59.955 INFO:teuthology.orchestra.run.smithi072.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-24T00:36:59.955 INFO:teuthology.orchestra.run.smithi072.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-24T00:36:59.956 INFO:teuthology.orchestra.run.smithi072.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-24T00:36:59.956 INFO:teuthology.orchestra.run.smithi072.stdout: librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:36:59.956 INFO:teuthology.orchestra.run.smithi072.stdout: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:36:59.956 INFO:teuthology.orchestra.run.smithi072.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-24T00:36:59.956 INFO:teuthology.orchestra.run.smithi072.stdout: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:36:59.956 INFO:teuthology.orchestra.run.smithi072.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-24T00:36:59.956 INFO:teuthology.orchestra.run.smithi072.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-24T00:36:59.956 INFO:teuthology.orchestra.run.smithi072.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-24T00:36:59.956 INFO:teuthology.orchestra.run.smithi072.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-24T00:36:59.956 INFO:teuthology.orchestra.run.smithi072.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-24T00:36:59.956 INFO:teuthology.orchestra.run.smithi072.stdout: lua-5.4.4-4.el9.x86_64 2024-03-24T00:36:59.956 INFO:teuthology.orchestra.run.smithi072.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-24T00:36:59.956 INFO:teuthology.orchestra.run.smithi072.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-24T00:36:59.956 INFO:teuthology.orchestra.run.smithi072.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-24T00:36:59.956 INFO:teuthology.orchestra.run.smithi072.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-24T00:36:59.956 INFO:teuthology.orchestra.run.smithi072.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-24T00:36:59.957 INFO:teuthology.orchestra.run.smithi072.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-24T00:36:59.957 INFO:teuthology.orchestra.run.smithi072.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-24T00:36:59.957 INFO:teuthology.orchestra.run.smithi072.stdout: postgresql-13.11-1.el9.x86_64 2024-03-24T00:36:59.957 INFO:teuthology.orchestra.run.smithi072.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-24T00:36:59.957 INFO:teuthology.orchestra.run.smithi072.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-24T00:36:59.957 INFO:teuthology.orchestra.run.smithi072.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-24T00:36:59.957 INFO:teuthology.orchestra.run.smithi072.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-24T00:36:59.957 INFO:teuthology.orchestra.run.smithi072.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-24T00:36:59.957 INFO:teuthology.orchestra.run.smithi072.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-24T00:36:59.957 INFO:teuthology.orchestra.run.smithi072.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-24T00:36:59.957 INFO:teuthology.orchestra.run.smithi072.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-24T00:36:59.957 INFO:teuthology.orchestra.run.smithi072.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-24T00:36:59.957 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-24T00:36:59.957 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:36:59.957 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:36:59.957 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:36:59.958 INFO:teuthology.orchestra.run.smithi072.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-24T00:36:59.958 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-24T00:36:59.958 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-24T00:36:59.958 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-24T00:36:59.958 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-24T00:36:59.958 INFO:teuthology.orchestra.run.smithi072.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-24T00:36:59.958 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-24T00:36:59.958 INFO:teuthology.orchestra.run.smithi072.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-24T00:36:59.958 INFO:teuthology.orchestra.run.smithi072.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-24T00:36:59.958 INFO:teuthology.orchestra.run.smithi072.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-24T00:36:59.958 INFO:teuthology.orchestra.run.smithi072.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-24T00:36:59.958 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-24T00:36:59.958 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-24T00:36:59.958 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-24T00:36:59.958 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-24T00:36:59.959 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-24T00:36:59.959 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-24T00:36:59.959 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-24T00:36:59.959 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-24T00:36:59.959 INFO:teuthology.orchestra.run.smithi072.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-24T00:36:59.959 INFO:teuthology.orchestra.run.smithi072.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-24T00:36:59.959 INFO:teuthology.orchestra.run.smithi072.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-24T00:36:59.959 INFO:teuthology.orchestra.run.smithi072.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-24T00:36:59.959 INFO:teuthology.orchestra.run.smithi072.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-24T00:36:59.959 INFO:teuthology.orchestra.run.smithi072.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-24T00:36:59.959 INFO:teuthology.orchestra.run.smithi072.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-24T00:36:59.959 INFO:teuthology.orchestra.run.smithi072.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-24T00:36:59.959 INFO:teuthology.orchestra.run.smithi072.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-24T00:36:59.959 INFO:teuthology.orchestra.run.smithi072.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-24T00:36:59.960 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-24T00:36:59.960 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ply-3.11-14.el9.noarch 2024-03-24T00:36:59.960 INFO:teuthology.orchestra.run.smithi072.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-24T00:36:59.960 INFO:teuthology.orchestra.run.smithi072.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-24T00:36:59.960 INFO:teuthology.orchestra.run.smithi072.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-24T00:36:59.960 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-24T00:36:59.960 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-24T00:36:59.960 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-24T00:36:59.960 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-24T00:36:59.960 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-24T00:36:59.960 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-24T00:36:59.960 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:36:59.960 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:36:59.960 INFO:teuthology.orchestra.run.smithi072.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-24T00:36:59.961 INFO:teuthology.orchestra.run.smithi072.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-24T00:36:59.961 INFO:teuthology.orchestra.run.smithi072.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-24T00:36:59.961 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:36:59.961 INFO:teuthology.orchestra.run.smithi072.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-24T00:36:59.961 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-24T00:36:59.961 INFO:teuthology.orchestra.run.smithi072.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-24T00:36:59.961 INFO:teuthology.orchestra.run.smithi072.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-24T00:36:59.961 INFO:teuthology.orchestra.run.smithi072.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-24T00:36:59.961 INFO:teuthology.orchestra.run.smithi072.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-24T00:36:59.961 INFO:teuthology.orchestra.run.smithi072.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-24T00:36:59.961 INFO:teuthology.orchestra.run.smithi072.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-24T00:36:59.961 INFO:teuthology.orchestra.run.smithi072.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-24T00:36:59.961 INFO:teuthology.orchestra.run.smithi072.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-24T00:36:59.961 INFO:teuthology.orchestra.run.smithi072.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-24T00:36:59.961 INFO:teuthology.orchestra.run.smithi072.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-24T00:36:59.962 INFO:teuthology.orchestra.run.smithi072.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-24T00:36:59.962 INFO:teuthology.orchestra.run.smithi072.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-24T00:36:59.962 INFO:teuthology.orchestra.run.smithi072.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-24T00:36:59.962 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:36:59.962 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:36:59.962 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:36:59.962 INFO:teuthology.orchestra.run.smithi072.stdout: re2-1:20211101-3.el9.x86_64 2024-03-24T00:36:59.962 INFO:teuthology.orchestra.run.smithi072.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-24T00:36:59.962 INFO:teuthology.orchestra.run.smithi072.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-24T00:36:59.962 INFO:teuthology.orchestra.run.smithi072.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-24T00:36:59.962 INFO:teuthology.orchestra.run.smithi072.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-24T00:36:59.962 INFO:teuthology.orchestra.run.smithi072.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-24T00:36:59.962 INFO:teuthology.orchestra.run.smithi072.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-24T00:36:59.962 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:36:59.963 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-24T00:37:00.032 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-24T00:37:00.077 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-24T00:37:00.108 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-24T00:37:00.139 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-24T00:37:00.167 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-24T00:37:00.202 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-24T00:37:00.241 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-24T00:37:00.272 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-24T00:37:00.302 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-24T00:37:00.382 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-24T00:37:00.421 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-24T00:37:00.454 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-24T00:37:00.495 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-24T00:37:00.529 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-24T00:37:00.607 DEBUG:teuthology.parallel:result is None 2024-03-24T00:37:00.624 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-24T00:37:00.713 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-24T00:37:00.770 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-24T00:37:00.800 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-24T00:37:00.849 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-24T00:37:00.960 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-24T00:37:01.012 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-24T00:37:01.778 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-24T00:37:01.832 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-24T00:37:01.866 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-24T00:37:01.915 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-24T00:37:02.014 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-24T00:37:02.048 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-24T00:37:02.234 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-24T00:37:02.262 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-24T00:37:02.343 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-24T00:37:03.039 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-24T00:37:03.155 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-24T00:37:04.457 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-24T00:37:04.537 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-24T00:37:04.568 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-24T00:37:04.599 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-24T00:37:04.626 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-24T00:37:04.957 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-24T00:37:04.990 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-24T00:37:05.064 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-24T00:37:05.094 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 125/198 2024-03-24T00:37:05.131 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-24T00:37:05.620 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-24T00:37:05.642 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-24T00:37:05.697 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-24T00:37:05.719 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 129/198 2024-03-24T00:37:08.027 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-24T00:37:08.057 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-24T00:37:08.629 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-24T00:37:08.668 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-24T00:37:08.753 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-24T00:37:08.753 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-24T00:37:08.753 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-24T00:37:08.754 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:37:08.786 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-24T00:37:21.059 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-24T00:37:21.059 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /sys 2024-03-24T00:37:21.059 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /proc 2024-03-24T00:37:21.059 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /mnt 2024-03-24T00:37:21.059 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /var/tmp 2024-03-24T00:37:21.059 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /home 2024-03-24T00:37:21.059 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /root 2024-03-24T00:37:21.059 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /tmp 2024-03-24T00:37:21.060 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:37:21.223 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-24T00:37:21.239 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-24T00:37:21.239 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:37:21.239 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-24T00:37:21.239 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:37:22.785 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-24T00:37:22.810 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-24T00:37:22.810 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:37:22.810 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-24T00:37:22.810 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T00:37:22.810 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T00:37:22.810 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:37:23.043 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-24T00:37:23.065 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-24T00:37:23.065 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:37:23.065 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-24T00:37:23.065 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T00:37:23.065 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T00:37:23.065 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:37:23.527 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-24T00:37:23.549 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-24T00:37:23.549 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:37:23.549 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-24T00:37:23.549 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T00:37:23.549 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T00:37:23.549 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:37:23.646 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-24T00:37:24.063 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-24T00:37:24.094 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-24T00:37:24.127 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 139/198 2024-03-24T00:37:24.188 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 140/198 2024-03-24T00:37:24.208 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-24T00:37:24.456 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-24T00:37:24.501 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-24T00:37:25.516 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-24T00:37:25.543 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-24T00:37:25.652 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-24T00:37:25.813 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 144/198 2024-03-24T00:37:25.837 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-24T00:37:25.859 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-24T00:37:25.859 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:37:25.859 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-24T00:37:25.859 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T00:37:25.859 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T00:37:25.859 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:37:25.913 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-24T00:37:25.923 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-24T00:37:25.977 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 147/198 2024-03-24T00:37:26.900 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-24T00:37:26.931 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-24T00:37:26.953 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-24T00:37:26.953 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:37:26.953 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-24T00:37:26.953 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T00:37:26.953 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T00:37:26.953 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:37:26.998 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-24T00:37:27.020 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-24T00:37:27.020 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:37:27.020 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-24T00:37:27.020 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:37:27.082 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-24T00:37:27.104 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-24T00:37:27.105 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:37:27.105 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-24T00:37:27.105 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T00:37:27.105 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T00:37:27.105 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:37:27.416 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-24T00:37:27.438 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-24T00:37:27.438 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:37:27.438 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-24T00:37:27.438 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T00:37:27.438 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T00:37:27.439 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:37:32.173 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-24T00:37:32.221 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 154/198 2024-03-24T00:37:32.251 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 155/198 2024-03-24T00:37:32.278 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-24T00:37:32.493 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 157/198 2024-03-24T00:37:32.615 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-24T00:37:32.660 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 159/198 2024-03-24T00:37:32.710 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-24T00:37:32.770 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-24T00:37:32.811 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-24T00:37:32.843 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 163/198 2024-03-24T00:37:32.884 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-24T00:37:32.913 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-24T00:37:32.972 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-24T00:37:33.039 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-24T00:37:33.078 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-24T00:37:33.678 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T00:37:33.717 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T00:37:33.736 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T00:37:34.628 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 170/198 2024-03-24T00:37:34.712 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-24T00:37:34.754 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-24T00:37:34.839 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-24T00:37:34.866 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-24T00:37:34.890 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-24T00:37:34.918 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-24T00:37:34.942 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-24T00:37:34.942 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-24T00:37:35.014 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-24T00:37:35.043 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-24T00:37:35.043 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-24T00:37:35.119 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-24T00:37:35.146 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-24T00:37:35.179 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-24T00:37:35.204 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-24T00:37:35.229 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-24T00:37:35.264 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-24T00:37:35.314 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-24T00:37:35.350 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-24T00:37:35.375 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-24T00:37:35.400 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-24T00:37:35.426 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-24T00:37:35.451 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-24T00:37:35.457 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T00:37:35.458 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T00:37:35.638 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T00:37:35.638 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-24T00:37:35.720 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-24T00:37:35.745 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-24T00:37:35.777 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-24T00:37:35.804 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-24T00:37:35.830 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-24T00:37:35.898 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-24T00:37:40.548 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-24T00:37:40.548 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 1/198 2024-03-24T00:37:40.548 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2/198 2024-03-24T00:37:40.548 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 3/198 2024-03-24T00:37:40.548 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 4/198 2024-03-24T00:37:40.548 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 5/198 2024-03-24T00:37:40.548 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 6/198 2024-03-24T00:37:40.548 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 7/198 2024-03-24T00:37:40.548 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 8/198 2024-03-24T00:37:40.548 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 9/198 2024-03-24T00:37:40.549 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-24T00:37:40.549 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-24T00:37:40.549 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-24T00:37:40.549 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 13/198 2024-03-24T00:37:40.549 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 14/198 2024-03-24T00:37:40.549 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 15/198 2024-03-24T00:37:40.549 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-24T00:37:40.549 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 17/198 2024-03-24T00:37:40.549 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 18/198 2024-03-24T00:37:40.549 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 19/198 2024-03-24T00:37:40.549 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 20/198 2024-03-24T00:37:40.549 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 21/198 2024-03-24T00:37:40.549 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 22/198 2024-03-24T00:37:40.549 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 23/198 2024-03-24T00:37:40.550 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 24/198 2024-03-24T00:37:40.550 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 25/198 2024-03-24T00:37:40.550 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 26/198 2024-03-24T00:37:40.550 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 27/198 2024-03-24T00:37:40.550 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 28/198 2024-03-24T00:37:40.550 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 29/198 2024-03-24T00:37:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 30/198 2024-03-24T00:37:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 31/198 2024-03-24T00:37:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 32/198 2024-03-24T00:37:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 33/198 2024-03-24T00:37:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 34/198 2024-03-24T00:37:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 35/198 2024-03-24T00:37:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 36/198 2024-03-24T00:37:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 37/198 2024-03-24T00:37:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 38/198 2024-03-24T00:37:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 39/198 2024-03-24T00:37:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-24T00:37:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-24T00:37:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-24T00:37:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-24T00:37:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-24T00:37:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-24T00:37:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-24T00:37:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-24T00:37:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-24T00:37:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-24T00:37:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-24T00:37:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-24T00:37:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-24T00:37:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-24T00:37:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-24T00:37:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-24T00:37:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-24T00:37:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-24T00:37:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-24T00:37:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-24T00:37:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-24T00:37:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-24T00:37:40.570 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-24T00:37:40.570 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-24T00:37:40.570 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-24T00:37:40.570 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-24T00:37:40.570 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-24T00:37:40.570 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-24T00:37:40.570 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-24T00:37:40.570 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-24T00:37:40.570 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-24T00:37:40.570 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-24T00:37:40.570 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-24T00:37:40.570 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-24T00:37:40.570 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-24T00:37:40.571 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-24T00:37:40.571 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-24T00:37:40.571 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-24T00:37:40.571 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-24T00:37:40.571 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-24T00:37:40.571 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-24T00:37:40.571 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-24T00:37:40.571 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-24T00:37:40.571 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-24T00:37:40.571 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-24T00:37:40.571 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-24T00:37:40.571 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-24T00:37:40.571 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-24T00:37:40.571 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-24T00:37:40.571 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-24T00:37:40.572 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-24T00:37:40.572 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-24T00:37:40.572 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-24T00:37:40.572 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-24T00:37:40.572 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-24T00:37:40.572 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-24T00:37:40.572 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-24T00:37:40.572 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-24T00:37:40.572 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-24T00:37:40.572 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-24T00:37:40.572 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-24T00:37:40.572 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-24T00:37:40.572 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-24T00:37:40.572 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-24T00:37:40.573 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-24T00:37:40.573 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-24T00:37:40.573 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-24T00:37:40.573 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-24T00:37:40.573 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-24T00:37:40.573 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-24T00:37:40.573 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-24T00:37:40.573 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-24T00:37:40.573 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-24T00:37:40.573 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-24T00:37:40.573 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-24T00:37:40.573 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-24T00:37:40.573 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-24T00:37:40.573 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-24T00:37:40.573 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-24T00:37:40.574 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-24T00:37:40.574 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-24T00:37:40.574 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-24T00:37:40.574 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-24T00:37:40.574 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-24T00:37:40.574 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-24T00:37:40.574 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-24T00:37:40.574 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-24T00:37:40.574 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-24T00:37:40.574 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-24T00:37:40.574 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-24T00:37:40.574 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-24T00:37:40.574 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-24T00:37:40.574 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-24T00:37:40.575 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-24T00:37:40.575 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-24T00:37:40.575 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-24T00:37:40.575 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-24T00:37:40.575 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-24T00:37:40.575 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-24T00:37:40.575 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-24T00:37:40.575 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-24T00:37:40.575 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-24T00:37:40.575 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-24T00:37:40.575 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-24T00:37:40.575 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-24T00:37:40.575 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-24T00:37:40.575 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-24T00:37:40.575 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-24T00:37:40.576 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-24T00:37:40.576 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-24T00:37:40.576 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-24T00:37:40.576 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-24T00:37:40.576 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-24T00:37:40.576 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-24T00:37:40.576 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-24T00:37:40.576 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-24T00:37:40.576 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-24T00:37:40.576 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-24T00:37:40.576 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-24T00:37:40.576 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-24T00:37:40.576 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-24T00:37:40.576 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-24T00:37:40.576 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-24T00:37:40.577 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-24T00:37:40.577 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-24T00:37:40.577 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-24T00:37:40.577 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-24T00:37:40.577 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-24T00:37:40.577 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-24T00:37:40.577 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-24T00:37:40.577 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-24T00:37:40.577 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-24T00:37:40.577 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-24T00:37:40.577 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-24T00:37:40.577 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-24T00:37:40.577 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-24T00:37:40.577 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-24T00:37:40.577 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-24T00:37:40.578 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-24T00:37:40.578 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-24T00:37:40.578 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-24T00:37:40.578 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-24T00:37:40.578 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-24T00:37:40.578 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-24T00:37:40.578 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-24T00:37:40.578 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-24T00:37:40.578 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-24T00:37:40.578 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-24T00:37:40.578 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-24T00:37:40.578 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-24T00:37:40.578 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-24T00:37:40.578 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-24T00:37:40.579 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-24T00:37:40.579 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-24T00:37:40.579 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-24T00:37:40.579 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-24T00:37:40.579 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-24T00:37:40.579 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-24T00:37:43.639 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-24T00:37:43.639 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:37:43.639 INFO:teuthology.orchestra.run.smithi067.stdout:Upgraded: 2024-03-24T00:37:43.639 INFO:teuthology.orchestra.run.smithi067.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-24T00:37:43.639 INFO:teuthology.orchestra.run.smithi067.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-24T00:37:43.640 INFO:teuthology.orchestra.run.smithi067.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-24T00:37:43.640 INFO:teuthology.orchestra.run.smithi067.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-24T00:37:43.640 INFO:teuthology.orchestra.run.smithi067.stdout: acl-2.3.1-4.el9.x86_64 2024-03-24T00:37:43.640 INFO:teuthology.orchestra.run.smithi067.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-24T00:37:43.640 INFO:teuthology.orchestra.run.smithi067.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-24T00:37:43.640 INFO:teuthology.orchestra.run.smithi067.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-24T00:37:43.640 INFO:teuthology.orchestra.run.smithi067.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-24T00:37:43.640 INFO:teuthology.orchestra.run.smithi067.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-24T00:37:43.640 INFO:teuthology.orchestra.run.smithi067.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-24T00:37:43.640 INFO:teuthology.orchestra.run.smithi067.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-24T00:37:43.640 INFO:teuthology.orchestra.run.smithi067.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-24T00:37:43.640 INFO:teuthology.orchestra.run.smithi067.stdout: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:37:43.640 INFO:teuthology.orchestra.run.smithi067.stdout: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:37:43.640 INFO:teuthology.orchestra.run.smithi067.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-24T00:37:43.641 INFO:teuthology.orchestra.run.smithi067.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-24T00:37:43.641 INFO:teuthology.orchestra.run.smithi067.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-24T00:37:43.641 INFO:teuthology.orchestra.run.smithi067.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-24T00:37:43.641 INFO:teuthology.orchestra.run.smithi067.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-24T00:37:43.641 INFO:teuthology.orchestra.run.smithi067.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-24T00:37:43.641 INFO:teuthology.orchestra.run.smithi067.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-24T00:37:43.641 INFO:teuthology.orchestra.run.smithi067.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-24T00:37:43.641 INFO:teuthology.orchestra.run.smithi067.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-24T00:37:43.641 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-03-24T00:37:43.641 INFO:teuthology.orchestra.run.smithi067.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-24T00:37:43.641 INFO:teuthology.orchestra.run.smithi067.stdout: automake-1.16.2-8.el9.noarch 2024-03-24T00:37:43.641 INFO:teuthology.orchestra.run.smithi067.stdout: bison-3.7.4-5.el9.x86_64 2024-03-24T00:37:43.641 INFO:teuthology.orchestra.run.smithi067.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-24T00:37:43.642 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:37:43.642 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:37:43.642 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:37:43.642 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:37:43.642 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:37:43.642 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:37:43.642 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:37:43.642 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:37:43.642 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:37:43.642 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:37:43.642 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:37:43.642 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:37:43.642 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:37:43.643 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:37:43.643 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:37:43.643 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:37:43.643 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:37:43.643 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:37:43.643 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:37:43.643 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:37:43.643 INFO:teuthology.orchestra.run.smithi067.stdout: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:37:43.643 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:37:43.643 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:37:43.643 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:37:43.643 INFO:teuthology.orchestra.run.smithi067.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-24T00:37:43.643 INFO:teuthology.orchestra.run.smithi067.stdout: flex-2.6.4-9.el9.x86_64 2024-03-24T00:37:43.643 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-24T00:37:43.644 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-24T00:37:43.644 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-24T00:37:43.644 INFO:teuthology.orchestra.run.smithi067.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-24T00:37:43.644 INFO:teuthology.orchestra.run.smithi067.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-24T00:37:43.644 INFO:teuthology.orchestra.run.smithi067.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-24T00:37:43.644 INFO:teuthology.orchestra.run.smithi067.stdout: jq-1.6-16.el9.x86_64 2024-03-24T00:37:43.644 INFO:teuthology.orchestra.run.smithi067.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-24T00:37:43.644 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-24T00:37:43.644 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-24T00:37:43.644 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:37:43.644 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:37:43.644 INFO:teuthology.orchestra.run.smithi067.stdout: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:37:43.644 INFO:teuthology.orchestra.run.smithi067.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-24T00:37:43.644 INFO:teuthology.orchestra.run.smithi067.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-24T00:37:43.645 INFO:teuthology.orchestra.run.smithi067.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-24T00:37:43.645 INFO:teuthology.orchestra.run.smithi067.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-24T00:37:43.645 INFO:teuthology.orchestra.run.smithi067.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-24T00:37:43.645 INFO:teuthology.orchestra.run.smithi067.stdout: librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:37:43.645 INFO:teuthology.orchestra.run.smithi067.stdout: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:37:43.645 INFO:teuthology.orchestra.run.smithi067.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-24T00:37:43.645 INFO:teuthology.orchestra.run.smithi067.stdout: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:37:43.645 INFO:teuthology.orchestra.run.smithi067.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-24T00:37:43.645 INFO:teuthology.orchestra.run.smithi067.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-24T00:37:43.645 INFO:teuthology.orchestra.run.smithi067.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-24T00:37:43.645 INFO:teuthology.orchestra.run.smithi067.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-24T00:37:43.645 INFO:teuthology.orchestra.run.smithi067.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-24T00:37:43.645 INFO:teuthology.orchestra.run.smithi067.stdout: lua-5.4.4-4.el9.x86_64 2024-03-24T00:37:43.645 INFO:teuthology.orchestra.run.smithi067.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-24T00:37:43.646 INFO:teuthology.orchestra.run.smithi067.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-24T00:37:43.646 INFO:teuthology.orchestra.run.smithi067.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-24T00:37:43.646 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-24T00:37:43.646 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-24T00:37:43.646 INFO:teuthology.orchestra.run.smithi067.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-24T00:37:43.646 INFO:teuthology.orchestra.run.smithi067.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-24T00:37:43.646 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-13.11-1.el9.x86_64 2024-03-24T00:37:43.646 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-24T00:37:43.646 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-24T00:37:43.646 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-24T00:37:43.646 INFO:teuthology.orchestra.run.smithi067.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-24T00:37:43.646 INFO:teuthology.orchestra.run.smithi067.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-24T00:37:43.646 INFO:teuthology.orchestra.run.smithi067.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-24T00:37:43.646 INFO:teuthology.orchestra.run.smithi067.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-24T00:37:43.647 INFO:teuthology.orchestra.run.smithi067.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-24T00:37:43.647 INFO:teuthology.orchestra.run.smithi067.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-24T00:37:43.647 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-24T00:37:43.647 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:37:43.647 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:37:43.647 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:37:43.647 INFO:teuthology.orchestra.run.smithi067.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-24T00:37:43.647 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-24T00:37:43.647 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-24T00:37:43.647 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-24T00:37:43.647 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-24T00:37:43.647 INFO:teuthology.orchestra.run.smithi067.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-24T00:37:43.647 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-24T00:37:43.647 INFO:teuthology.orchestra.run.smithi067.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-24T00:37:43.648 INFO:teuthology.orchestra.run.smithi067.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-24T00:37:43.648 INFO:teuthology.orchestra.run.smithi067.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-24T00:37:43.648 INFO:teuthology.orchestra.run.smithi067.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-24T00:37:43.648 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-24T00:37:43.648 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-24T00:37:43.648 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-24T00:37:43.648 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-24T00:37:43.648 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-24T00:37:43.648 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-24T00:37:43.648 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-24T00:37:43.648 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-24T00:37:43.648 INFO:teuthology.orchestra.run.smithi067.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-24T00:37:43.648 INFO:teuthology.orchestra.run.smithi067.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-24T00:37:43.649 INFO:teuthology.orchestra.run.smithi067.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-24T00:37:43.649 INFO:teuthology.orchestra.run.smithi067.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-24T00:37:43.649 INFO:teuthology.orchestra.run.smithi067.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-24T00:37:43.649 INFO:teuthology.orchestra.run.smithi067.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-24T00:37:43.649 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-24T00:37:43.649 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-24T00:37:43.649 INFO:teuthology.orchestra.run.smithi067.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-24T00:37:43.649 INFO:teuthology.orchestra.run.smithi067.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-24T00:37:43.649 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-24T00:37:43.649 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ply-3.11-14.el9.noarch 2024-03-24T00:37:43.649 INFO:teuthology.orchestra.run.smithi067.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-24T00:37:43.649 INFO:teuthology.orchestra.run.smithi067.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-24T00:37:43.649 INFO:teuthology.orchestra.run.smithi067.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-24T00:37:43.649 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-24T00:37:43.649 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-24T00:37:43.649 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-24T00:37:43.650 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-24T00:37:43.650 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-24T00:37:43.650 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-24T00:37:43.650 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:37:43.650 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:37:43.650 INFO:teuthology.orchestra.run.smithi067.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-24T00:37:43.650 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-24T00:37:43.650 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-24T00:37:43.650 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:37:43.650 INFO:teuthology.orchestra.run.smithi067.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-24T00:37:43.650 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-24T00:37:43.650 INFO:teuthology.orchestra.run.smithi067.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-24T00:37:43.650 INFO:teuthology.orchestra.run.smithi067.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-24T00:37:43.650 INFO:teuthology.orchestra.run.smithi067.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-24T00:37:43.651 INFO:teuthology.orchestra.run.smithi067.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-24T00:37:43.651 INFO:teuthology.orchestra.run.smithi067.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-24T00:37:43.651 INFO:teuthology.orchestra.run.smithi067.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-24T00:37:43.651 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-24T00:37:43.651 INFO:teuthology.orchestra.run.smithi067.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-24T00:37:43.651 INFO:teuthology.orchestra.run.smithi067.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-24T00:37:43.651 INFO:teuthology.orchestra.run.smithi067.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-24T00:37:43.651 INFO:teuthology.orchestra.run.smithi067.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-24T00:37:43.651 INFO:teuthology.orchestra.run.smithi067.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-24T00:37:43.651 INFO:teuthology.orchestra.run.smithi067.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-24T00:37:43.651 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:37:43.651 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:37:43.651 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:37:43.651 INFO:teuthology.orchestra.run.smithi067.stdout: re2-1:20211101-3.el9.x86_64 2024-03-24T00:37:43.652 INFO:teuthology.orchestra.run.smithi067.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-24T00:37:43.652 INFO:teuthology.orchestra.run.smithi067.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-24T00:37:43.652 INFO:teuthology.orchestra.run.smithi067.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-24T00:37:43.652 INFO:teuthology.orchestra.run.smithi067.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-24T00:37:43.652 INFO:teuthology.orchestra.run.smithi067.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-24T00:37:43.652 INFO:teuthology.orchestra.run.smithi067.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-24T00:37:43.652 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:37:43.652 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-24T00:37:44.094 DEBUG:teuthology.parallel:result is None 2024-03-24T00:37:44.094 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T00:37:44.243 DEBUG:teuthology.orchestra.run.smithi067:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-24T00:37:44.266 INFO:teuthology.orchestra.run.smithi067.stdout:19.0.0-2145.g3be9a5c9.el9 2024-03-24T00:37:44.266 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2145.g3be9a5c9.el9 2024-03-24T00:37:44.266 INFO:teuthology.task.install:The correct ceph version 19.0.0-2145.g3be9a5c9 is installed. 2024-03-24T00:37:44.268 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T00:37:44.407 DEBUG:teuthology.orchestra.run.smithi072:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-24T00:37:44.432 INFO:teuthology.orchestra.run.smithi072.stdout:19.0.0-2145.g3be9a5c9.el9 2024-03-24T00:37:44.432 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2145.g3be9a5c9.el9 2024-03-24T00:37:44.432 INFO:teuthology.task.install:The correct ceph version 19.0.0-2145.g3be9a5c9 is installed. 2024-03-24T00:37:44.434 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-24T00:37:44.434 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:37:44.434 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-24T00:37:44.465 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:37:44.465 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-24T00:37:44.505 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-24T00:37:44.505 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:37:44.505 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/daemon-helper 2024-03-24T00:37:44.533 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-24T00:37:44.601 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:37:44.601 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/usr/bin/daemon-helper 2024-03-24T00:37:44.630 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-24T00:37:44.699 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-24T00:37:44.700 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:37:44.700 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-24T00:37:44.727 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-24T00:37:44.794 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:37:44.794 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-24T00:37:44.822 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-24T00:37:44.891 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-24T00:37:44.891 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:37:44.891 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/stdin-killer 2024-03-24T00:37:44.917 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-24T00:37:44.984 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:37:44.984 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/usr/bin/stdin-killer 2024-03-24T00:37:45.012 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-24T00:37:45.079 INFO:teuthology.run_tasks:Running task ceph... 2024-03-24T00:37:45.168 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-24T00:37:45.169 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 777 /var/log/ceph 2024-03-24T00:37:45.171 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 777 /var/log/ceph 2024-03-24T00:37:45.198 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-24T00:37:45.199 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-24T00:37:45.238 DEBUG:teuthology.orchestra.run.smithi072:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-24T00:37:45.267 INFO:tasks.ceph:Creating extra log directories... 2024-03-24T00:37:45.267 DEBUG:teuthology.orchestra.run.smithi067:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-24T00:37:45.305 DEBUG:teuthology.orchestra.run.smithi072:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-24T00:37:45.342 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-03-24T00:37:45.343 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-03-24T00:37:45.343 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-03-24T00:37:45.343 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:37:45.343 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-24T00:37:45.384 DEBUG:teuthology.orchestra.run.smithi067:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-24T00:37:45.495 DEBUG:teuthology.orchestra.run.smithi067:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-24T00:37:45.521 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:37:45.521 DEBUG:teuthology.orchestra.run.smithi072:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-24T00:37:45.535 DEBUG:teuthology.orchestra.run.smithi072:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-24T00:37:45.647 DEBUG:teuthology.orchestra.run.smithi072:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-24T00:37:45.680 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-24T00:37:45.680 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'inode wrongly marked free', 'bad backtrace on inode', 'inode table repaired for inode', 'Scrub error on inode', 'Scrub error on dir', 'Metadata damage detected'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-24T00:37:45.680 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616442', 'branch': 'wip-mchangir-testing-main-20240318.032620', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/forward-scrub}', 'email': 'mchangir@redhat.com', 'first_in_suite': False, 'job_id': '7616442', 'kernel': {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': '5ec3e6e88b2e2783e09498588bb1f433ae9c8986'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-mchangir-testing-main-20240318.032620'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'inode wrongly marked free', 'bad backtrace on inode', 'inode table repaired for inode', 'Scrub error on inode', 'Scrub error on dir', 'Metadata damage detected'], 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'}}, 'kclient': {'syntax': 'v1'}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']}, 'subvolume_version': 1, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-mchangir-testing-main-20240318.032620', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'}}, 'owner': 'mchangir', 'priority': 100, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 3550, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'sleep_before_teardown': 0, 'subset': '613/1024', 'suite': 'fs', 'suite_branch': 'wip-mchangir-testing-main-20240318.032620', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi067.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB8CD0DPVKDGSdsXcNniFZ/nMWBblB6BCthjTFboTuuuZGK72PZsmuWPKA4ddb/TZ5BhSV6teWCVIHkd5gVxOLw=', 'smithi072.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHKZ2OBRMHPdFxGrk+8dJ5aBUfN0YoKZJ/yHulR5ozeKvvv8ePCbz4UDpFjr7jPvwgWT4STFERjN8cT9GeVy2Ps='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': '5ec3e6e88b2e2783e09498588bb1f433ae9c8986'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'inode wrongly marked free', 'bad backtrace on inode', 'inode table repaired for inode', 'Scrub error on inode', 'Scrub error on dir', 'Metadata damage detected'], 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'cluster': 'ceph'}}, {'kclient': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_forward_scrub']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-22_09:49:57', 'tube': 'smithi', 'user': 'mchangir', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-24T00:37:45.681 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-24T00:37:45.683 DEBUG:teuthology.orchestra.run.smithi072:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-24T00:37:45.730 DEBUG:teuthology.orchestra.run.smithi067:> sudo install -d -m0777 -- /var/run/ceph 2024-03-24T00:37:45.737 DEBUG:teuthology.orchestra.run.smithi072:> sudo install -d -m0777 -- /var/run/ceph 2024-03-24T00:37:45.796 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:37:45.797 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T00:37:45.816 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T00:37:45.816 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_1 2024-03-24T00:37:45.870 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T00:37:45.871 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:37:45.871 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-24T00:37:45.871 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:37:45.871 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T00:37:45.871 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-03-24 00:37:36.248608218 +0000 2024-03-24T00:37:45.871 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-03-24 00:34:49.600784180 +0000 2024-03-24T00:37:45.871 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-03-24 00:34:49.600784180 +0000 2024-03-24T00:37:45.871 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-03-24 00:34:49.600784180 +0000 2024-03-24T00:37:45.871 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T00:37:45.935 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-03-24T00:37:45.935 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-03-24T00:37:45.936 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000265062 s, 1.9 MB/s 2024-03-24T00:37:45.937 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T00:37:45.993 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_2 2024-03-24T00:37:46.048 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T00:37:46.048 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:37:46.048 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-24T00:37:46.048 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:37:46.048 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T00:37:46.048 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-03-24 00:37:36.248608218 +0000 2024-03-24T00:37:46.048 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-03-24 00:34:49.950776725 +0000 2024-03-24T00:37:46.048 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-03-24 00:34:49.950776725 +0000 2024-03-24T00:37:46.048 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-03-24 00:34:49.950776725 +0000 2024-03-24T00:37:46.049 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T00:37:46.113 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-03-24T00:37:46.113 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-03-24T00:37:46.113 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000198279 s, 2.6 MB/s 2024-03-24T00:37:46.114 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T00:37:46.169 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_3 2024-03-24T00:37:46.224 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T00:37:46.224 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:37:46.224 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-24T00:37:46.224 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:37:46.224 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T00:37:46.224 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-03-24 00:37:36.248608218 +0000 2024-03-24T00:37:46.224 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-03-24 00:34:50.302769225 +0000 2024-03-24T00:37:46.224 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-03-24 00:34:50.302769225 +0000 2024-03-24T00:37:46.224 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-03-24 00:34:50.302769225 +0000 2024-03-24T00:37:46.225 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T00:37:46.290 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-03-24T00:37:46.290 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-03-24T00:37:46.291 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000185598 s, 2.8 MB/s 2024-03-24T00:37:46.291 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T00:37:46.345 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_4 2024-03-24T00:37:46.399 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T00:37:46.400 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:37:46.400 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-24T00:37:46.400 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:37:46.400 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T00:37:46.400 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-03-24 00:37:36.249608197 +0000 2024-03-24T00:37:46.400 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-03-24 00:34:50.634762152 +0000 2024-03-24T00:37:46.400 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-03-24 00:34:50.634762152 +0000 2024-03-24T00:37:46.400 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-03-24 00:34:50.634762152 +0000 2024-03-24T00:37:46.400 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T00:37:46.464 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-03-24T00:37:46.464 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-03-24T00:37:46.464 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000242598 s, 2.1 MB/s 2024-03-24T00:37:46.465 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T00:37:46.520 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-24T00:37:46.520 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:37:46.520 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T00:37:46.537 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T00:37:46.537 DEBUG:teuthology.orchestra.run.smithi072:> stat /dev/vg_nvme/lv_1 2024-03-24T00:37:46.592 INFO:teuthology.orchestra.run.smithi072.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T00:37:46.592 INFO:teuthology.orchestra.run.smithi072.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:37:46.592 INFO:teuthology.orchestra.run.smithi072.stdout:Device: 5h/5d Inode: 658 Links: 1 2024-03-24T00:37:46.592 INFO:teuthology.orchestra.run.smithi072.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:37:46.592 INFO:teuthology.orchestra.run.smithi072.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T00:37:46.592 INFO:teuthology.orchestra.run.smithi072.stdout:Access: 2024-03-24 00:36:48.576376127 +0000 2024-03-24T00:37:46.592 INFO:teuthology.orchestra.run.smithi072.stdout:Modify: 2024-03-24 00:32:45.763573793 +0000 2024-03-24T00:37:46.593 INFO:teuthology.orchestra.run.smithi072.stdout:Change: 2024-03-24 00:32:45.763573793 +0000 2024-03-24T00:37:46.593 INFO:teuthology.orchestra.run.smithi072.stdout: Birth: 2024-03-24 00:32:45.763573793 +0000 2024-03-24T00:37:46.593 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T00:37:46.657 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records in 2024-03-24T00:37:46.657 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records out 2024-03-24T00:37:46.658 INFO:teuthology.orchestra.run.smithi072.stderr:512 bytes copied, 0.000325518 s, 1.6 MB/s 2024-03-24T00:37:46.659 DEBUG:teuthology.orchestra.run.smithi072:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T00:37:46.715 DEBUG:teuthology.orchestra.run.smithi072:> stat /dev/vg_nvme/lv_2 2024-03-24T00:37:46.770 INFO:teuthology.orchestra.run.smithi072.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T00:37:46.770 INFO:teuthology.orchestra.run.smithi072.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:37:46.770 INFO:teuthology.orchestra.run.smithi072.stdout:Device: 5h/5d Inode: 680 Links: 1 2024-03-24T00:37:46.770 INFO:teuthology.orchestra.run.smithi072.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:37:46.770 INFO:teuthology.orchestra.run.smithi072.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T00:37:46.770 INFO:teuthology.orchestra.run.smithi072.stdout:Access: 2024-03-24 00:36:48.576376127 +0000 2024-03-24T00:37:46.770 INFO:teuthology.orchestra.run.smithi072.stdout:Modify: 2024-03-24 00:32:46.098566952 +0000 2024-03-24T00:37:46.770 INFO:teuthology.orchestra.run.smithi072.stdout:Change: 2024-03-24 00:32:46.098566952 +0000 2024-03-24T00:37:46.770 INFO:teuthology.orchestra.run.smithi072.stdout: Birth: 2024-03-24 00:32:46.098566952 +0000 2024-03-24T00:37:46.770 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T00:37:46.835 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records in 2024-03-24T00:37:46.836 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records out 2024-03-24T00:37:46.836 INFO:teuthology.orchestra.run.smithi072.stderr:512 bytes copied, 0.000228337 s, 2.2 MB/s 2024-03-24T00:37:46.836 DEBUG:teuthology.orchestra.run.smithi072:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T00:37:46.892 DEBUG:teuthology.orchestra.run.smithi072:> stat /dev/vg_nvme/lv_3 2024-03-24T00:37:46.948 INFO:teuthology.orchestra.run.smithi072.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T00:37:46.948 INFO:teuthology.orchestra.run.smithi072.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:37:46.948 INFO:teuthology.orchestra.run.smithi072.stdout:Device: 5h/5d Inode: 696 Links: 1 2024-03-24T00:37:46.948 INFO:teuthology.orchestra.run.smithi072.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:37:46.948 INFO:teuthology.orchestra.run.smithi072.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T00:37:46.948 INFO:teuthology.orchestra.run.smithi072.stdout:Access: 2024-03-24 00:36:48.576376127 +0000 2024-03-24T00:37:46.948 INFO:teuthology.orchestra.run.smithi072.stdout:Modify: 2024-03-24 00:32:46.426560254 +0000 2024-03-24T00:37:46.948 INFO:teuthology.orchestra.run.smithi072.stdout:Change: 2024-03-24 00:32:46.426560254 +0000 2024-03-24T00:37:46.948 INFO:teuthology.orchestra.run.smithi072.stdout: Birth: 2024-03-24 00:32:46.426560254 +0000 2024-03-24T00:37:46.948 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T00:37:47.013 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records in 2024-03-24T00:37:47.013 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records out 2024-03-24T00:37:47.014 INFO:teuthology.orchestra.run.smithi072.stderr:512 bytes copied, 0.000208302 s, 2.5 MB/s 2024-03-24T00:37:47.014 DEBUG:teuthology.orchestra.run.smithi072:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T00:37:47.070 DEBUG:teuthology.orchestra.run.smithi072:> stat /dev/vg_nvme/lv_4 2024-03-24T00:37:47.125 INFO:teuthology.orchestra.run.smithi072.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T00:37:47.125 INFO:teuthology.orchestra.run.smithi072.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:37:47.125 INFO:teuthology.orchestra.run.smithi072.stdout:Device: 5h/5d Inode: 718 Links: 1 2024-03-24T00:37:47.125 INFO:teuthology.orchestra.run.smithi072.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:37:47.125 INFO:teuthology.orchestra.run.smithi072.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T00:37:47.125 INFO:teuthology.orchestra.run.smithi072.stdout:Access: 2024-03-24 00:36:48.577376107 +0000 2024-03-24T00:37:47.125 INFO:teuthology.orchestra.run.smithi072.stdout:Modify: 2024-03-24 00:32:46.760553433 +0000 2024-03-24T00:37:47.125 INFO:teuthology.orchestra.run.smithi072.stdout:Change: 2024-03-24 00:32:46.760553433 +0000 2024-03-24T00:37:47.126 INFO:teuthology.orchestra.run.smithi072.stdout: Birth: 2024-03-24 00:32:46.760553433 +0000 2024-03-24T00:37:47.126 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T00:37:47.191 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records in 2024-03-24T00:37:47.191 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records out 2024-03-24T00:37:47.191 INFO:teuthology.orchestra.run.smithi072.stderr:512 bytes copied, 0.000202756 s, 2.5 MB/s 2024-03-24T00:37:47.192 DEBUG:teuthology.orchestra.run.smithi072:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T00:37:47.247 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-24T00:37:47.248 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi067.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi072.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-24T00:37:47.248 INFO:tasks.ceph:Generating config... 2024-03-24T00:37:47.249 INFO:tasks.ceph:[client] client mount timeout = 600 2024-03-24T00:37:47.249 INFO:tasks.ceph:[client] debug client = 20 2024-03-24T00:37:47.249 INFO:tasks.ceph:[client] debug ms = 1 2024-03-24T00:37:47.249 INFO:tasks.ceph:[client] pid file = 2024-03-24T00:37:47.249 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-03-24T00:37:47.249 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-03-24T00:37:47.249 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-03-24T00:37:47.249 INFO:tasks.ceph:[mds] debug mds = 20 2024-03-24T00:37:47.250 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-03-24T00:37:47.250 INFO:tasks.ceph:[mds] debug ms = 1 2024-03-24T00:37:47.250 INFO:tasks.ceph:[mds] mds debug frag = True 2024-03-24T00:37:47.250 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-03-24T00:37:47.250 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-03-24T00:37:47.250 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-03-24T00:37:47.250 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-03-24T00:37:47.250 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-03-24T00:37:47.250 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-03-24T00:37:47.250 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-24T00:37:47.250 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-24T00:37:47.250 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-24T00:37:47.250 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-24T00:37:47.250 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-24T00:37:47.250 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-03-24T00:37:47.251 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-24T00:37:47.251 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-24T00:37:47.251 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-24T00:37:47.251 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-24T00:37:47.251 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-24T00:37:47.251 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-24T00:37:47.251 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-24T00:37:47.251 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-24T00:37:47.251 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-24T00:37:47.251 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-24T00:37:47.251 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-24T00:37:47.251 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-24T00:37:47.251 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-03-24T00:37:47.252 INFO:tasks.ceph:Setting up mon.a... 2024-03-24T00:37:47.252 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-24T00:37:47.320 INFO:teuthology.orchestra.run.smithi067.stdout:creating /etc/ceph/ceph.keyring 2024-03-24T00:37:47.322 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-24T00:37:47.368 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-24T00:37:47.395 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.67'), ('mon.b', '172.21.15.72'), ('mon.c', '[v2:172.21.15.72:3301,v1:172.21.15.72:6790]')] 2024-03-24T00:37:47.395 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.67,172.21.15.72,[v2:172.21.15.72:3301,v1:172.21.15.72:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-03-24T00:37:47.396 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:37:47.396 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-24T00:37:47.450 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.67 --add b 172.21.15.72 --addv c '[v2:172.21.15.72:3301,v1:172.21.15.72:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-24T00:37:47.535 INFO:teuthology.orchestra.run.smithi067.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-24T00:37:47.535 INFO:teuthology.orchestra.run.smithi067.stdout:monmaptool: generated fsid de1d0178-808e-49ca-a986-e297e523b976 2024-03-24T00:37:47.535 INFO:teuthology.orchestra.run.smithi067.stdout:setting min_mon_release = quincy 2024-03-24T00:37:47.535 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 0 2024-03-24T00:37:47.535 INFO:teuthology.orchestra.run.smithi067.stdout:fsid de1d0178-808e-49ca-a986-e297e523b976 2024-03-24T00:37:47.535 INFO:teuthology.orchestra.run.smithi067.stdout:last_changed 2024-03-24T00:37:47.535879+0000 2024-03-24T00:37:47.535 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-03-24T00:37:47.535879+0000 2024-03-24T00:37:47.535 INFO:teuthology.orchestra.run.smithi067.stdout:min_mon_release 17 (quincy) 2024-03-24T00:37:47.535 INFO:teuthology.orchestra.run.smithi067.stdout:election_strategy: 1 2024-03-24T00:37:47.535 INFO:teuthology.orchestra.run.smithi067.stdout:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2024-03-24T00:37:47.536 INFO:teuthology.orchestra.run.smithi067.stdout:1: [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] mon.b 2024-03-24T00:37:47.536 INFO:teuthology.orchestra.run.smithi067.stdout:2: [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] mon.c 2024-03-24T00:37:47.536 INFO:teuthology.orchestra.run.smithi067.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-24T00:37:47.537 DEBUG:teuthology.orchestra.run.smithi067:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-24T00:37:47.551 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID de1d0178-808e-49ca-a986-e297e523b976... 2024-03-24T00:37:47.552 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-24T00:37:47.593 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-24T00:37:47.641 INFO:teuthology.orchestra.run.smithi067.stdout:[global] 2024-03-24T00:37:47.641 INFO:teuthology.orchestra.run.smithi067.stdout: chdir = "" 2024-03-24T00:37:47.641 INFO:teuthology.orchestra.run.smithi067.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-24T00:37:47.641 INFO:teuthology.orchestra.run.smithi067.stdout: auth supported = cephx 2024-03-24T00:37:47.641 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:37:47.641 INFO:teuthology.orchestra.run.smithi067.stdout: filestore xattr use omap = true 2024-03-24T00:37:47.641 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:37:47.641 INFO:teuthology.orchestra.run.smithi067.stdout: mon clock drift allowed = 1.000 2024-03-24T00:37:47.641 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:37:47.641 INFO:teuthology.orchestra.run.smithi067.stdout: osd crush chooseleaf type = 0 2024-03-24T00:37:47.641 INFO:teuthology.orchestra.run.smithi067.stdout: auth debug = true 2024-03-24T00:37:47.641 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:37:47.642 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on old message = true 2024-03-24T00:37:47.642 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on bug = true 2024-03-24T00:37:47.642 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:37:47.642 INFO:teuthology.orchestra.run.smithi067.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-24T00:37:47.642 INFO:teuthology.orchestra.run.smithi067.stdout: mon pg warn max object skew = 0 2024-03-24T00:37:47.642 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:37:47.642 INFO:teuthology.orchestra.run.smithi067.stdout: # disable pg_autoscaler by default for new pools 2024-03-24T00:37:47.642 INFO:teuthology.orchestra.run.smithi067.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-24T00:37:47.642 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:37:47.642 INFO:teuthology.orchestra.run.smithi067.stdout: osd pool default size = 2 2024-03-24T00:37:47.642 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:37:47.642 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd allow primary affinity = true 2024-03-24T00:37:47.642 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd allow pg remap = true 2024-03-24T00:37:47.642 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on legacy crush tunables = false 2024-03-24T00:37:47.643 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on crush straw calc version zero = false 2024-03-24T00:37:47.643 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on no sortbitwise = false 2024-03-24T00:37:47.643 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on osd down out interval zero = false 2024-03-24T00:37:47.643 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on too few osds = false 2024-03-24T00:37:47.643 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-24T00:37:47.643 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-24T00:37:47.643 INFO:teuthology.orchestra.run.smithi067.stdout: mon_allow_pool_size_one = true 2024-03-24T00:37:47.643 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:37:47.643 INFO:teuthology.orchestra.run.smithi067.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-24T00:37:47.643 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:37:47.643 INFO:teuthology.orchestra.run.smithi067.stdout: osd default data pool replay window = 5 2024-03-24T00:37:47.643 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:37:47.643 INFO:teuthology.orchestra.run.smithi067.stdout: mon allow pool delete = true 2024-03-24T00:37:47.643 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:37:47.643 INFO:teuthology.orchestra.run.smithi067.stdout: mon cluster log file level = debug 2024-03-24T00:37:47.644 INFO:teuthology.orchestra.run.smithi067.stdout: debug asserts on shutdown = true 2024-03-24T00:37:47.644 INFO:teuthology.orchestra.run.smithi067.stdout: mon health detail to clog = false 2024-03-24T00:37:47.644 INFO:teuthology.orchestra.run.smithi067.stdout: mon host = "172.21.15.67,172.21.15.72,[v2:172.21.15.72:3301,v1:172.21.15.72:6790]" 2024-03-24T00:37:47.644 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on skipped message = False 2024-03-24T00:37:47.644 INFO:teuthology.orchestra.run.smithi067.stdout: fsid = de1d0178-808e-49ca-a986-e297e523b976 2024-03-24T00:37:47.644 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:37:47.644 INFO:teuthology.orchestra.run.smithi067.stdout:[osd] 2024-03-24T00:37:47.644 INFO:teuthology.orchestra.run.smithi067.stdout: osd journal size = 100 2024-03-24T00:37:47.644 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:37:47.644 INFO:teuthology.orchestra.run.smithi067.stdout: osd scrub load threshold = 5.0 2024-03-24T00:37:47.644 INFO:teuthology.orchestra.run.smithi067.stdout: osd scrub max interval = 600 2024-03-24T00:37:47.644 INFO:teuthology.orchestra.run.smithi067.stdout: osd mclock profile = high_recovery_ops 2024-03-24T00:37:47.644 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:37:47.644 INFO:teuthology.orchestra.run.smithi067.stdout: osd recover clone overlap = true 2024-03-24T00:37:47.644 INFO:teuthology.orchestra.run.smithi067.stdout: osd recovery max chunk = 1048576 2024-03-24T00:37:47.644 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:37:47.644 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug shutdown = true 2024-03-24T00:37:47.645 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug op order = true 2024-03-24T00:37:47.645 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug verify stray on activate = true 2024-03-24T00:37:47.645 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug trim objects = true 2024-03-24T00:37:47.645 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:37:47.645 INFO:teuthology.orchestra.run.smithi067.stdout: osd open classes on start = true 2024-03-24T00:37:47.645 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug pg log writeout = true 2024-03-24T00:37:47.645 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:37:47.645 INFO:teuthology.orchestra.run.smithi067.stdout: osd deep scrub update digest min age = 30 2024-03-24T00:37:47.645 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:37:47.645 INFO:teuthology.orchestra.run.smithi067.stdout: osd map max advance = 10 2024-03-24T00:37:47.645 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:37:47.645 INFO:teuthology.orchestra.run.smithi067.stdout: journal zero on create = true 2024-03-24T00:37:47.645 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:37:47.645 INFO:teuthology.orchestra.run.smithi067.stdout: filestore ondisk finisher threads = 3 2024-03-24T00:37:47.645 INFO:teuthology.orchestra.run.smithi067.stdout: filestore apply finisher threads = 3 2024-03-24T00:37:47.646 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:37:47.646 INFO:teuthology.orchestra.run.smithi067.stdout: bdev debug aio = true 2024-03-24T00:37:47.646 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug misdirected ops = true 2024-03-24T00:37:47.646 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore block size = 96636764160 2024-03-24T00:37:47.646 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore fsck on mount = True 2024-03-24T00:37:47.646 INFO:teuthology.orchestra.run.smithi067.stdout: debug bluefs = 20 2024-03-24T00:37:47.646 INFO:teuthology.orchestra.run.smithi067.stdout: debug bluestore = 20 2024-03-24T00:37:47.646 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-03-24T00:37:47.646 INFO:teuthology.orchestra.run.smithi067.stdout: debug osd = 20 2024-03-24T00:37:47.646 INFO:teuthology.orchestra.run.smithi067.stdout: debug rocksdb = 10 2024-03-24T00:37:47.646 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd backfillfull_ratio = 0.85 2024-03-24T00:37:47.646 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd full ratio = 0.9 2024-03-24T00:37:47.646 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd nearfull ratio = 0.8 2024-03-24T00:37:47.646 INFO:teuthology.orchestra.run.smithi067.stdout: osd failsafe full ratio = 0.95 2024-03-24T00:37:47.647 INFO:teuthology.orchestra.run.smithi067.stdout: osd objectstore = bluestore 2024-03-24T00:37:47.647 INFO:teuthology.orchestra.run.smithi067.stdout: osd op complaint time = 180 2024-03-24T00:37:47.647 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:37:47.647 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr] 2024-03-24T00:37:47.647 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-03-24T00:37:47.647 INFO:teuthology.orchestra.run.smithi067.stdout: debug mgr = 20 2024-03-24T00:37:47.647 INFO:teuthology.orchestra.run.smithi067.stdout: debug mon = 20 2024-03-24T00:37:47.647 INFO:teuthology.orchestra.run.smithi067.stdout: debug auth = 20 2024-03-24T00:37:47.647 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min pgs per osd = 4 2024-03-24T00:37:47.647 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min bytes per osd = 10 2024-03-24T00:37:47.648 INFO:teuthology.orchestra.run.smithi067.stdout: mgr/telemetry/nag = false 2024-03-24T00:37:47.648 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:37:47.648 INFO:teuthology.orchestra.run.smithi067.stdout:[mon] 2024-03-24T00:37:47.648 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-03-24T00:37:47.648 INFO:teuthology.orchestra.run.smithi067.stdout: debug mon = 20 2024-03-24T00:37:47.648 INFO:teuthology.orchestra.run.smithi067.stdout: debug paxos = 20 2024-03-24T00:37:47.648 INFO:teuthology.orchestra.run.smithi067.stdout: debug auth = 20 2024-03-24T00:37:47.648 INFO:teuthology.orchestra.run.smithi067.stdout: mon data avail warn = 5 2024-03-24T00:37:47.648 INFO:teuthology.orchestra.run.smithi067.stdout: mon mgr mkfs grace = 240 2024-03-24T00:37:47.648 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min pgs per osd = 4 2024-03-24T00:37:47.648 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd reporter subtree level = osd 2024-03-24T00:37:47.648 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd prime pg temp = true 2024-03-24T00:37:47.648 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min bytes per osd = 10 2024-03-24T00:37:47.648 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:37:47.648 INFO:teuthology.orchestra.run.smithi067.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-24T00:37:47.649 INFO:teuthology.orchestra.run.smithi067.stdout: auth mon ticket ttl = 660 # 11m 2024-03-24T00:37:47.649 INFO:teuthology.orchestra.run.smithi067.stdout: auth service ticket ttl = 240 # 4m 2024-03-24T00:37:47.649 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:37:47.649 INFO:teuthology.orchestra.run.smithi067.stdout: # don't complain about insecure global_id in the test suite 2024-03-24T00:37:47.649 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-24T00:37:47.649 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-24T00:37:47.649 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:37:47.649 INFO:teuthology.orchestra.run.smithi067.stdout: # 1m isn't quite enough 2024-03-24T00:37:47.649 INFO:teuthology.orchestra.run.smithi067.stdout: mon_down_mkfs_grace = 2m 2024-03-24T00:37:47.649 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:37:47.649 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_filestore_osds = false 2024-03-24T00:37:47.649 INFO:teuthology.orchestra.run.smithi067.stdout: mon op complaint time = 120 2024-03-24T00:37:47.649 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:37:47.649 INFO:teuthology.orchestra.run.smithi067.stdout:[client] 2024-03-24T00:37:47.649 INFO:teuthology.orchestra.run.smithi067.stdout: rgw cache enabled = true 2024-03-24T00:37:47.650 INFO:teuthology.orchestra.run.smithi067.stdout: rgw enable ops log = true 2024-03-24T00:37:47.650 INFO:teuthology.orchestra.run.smithi067.stdout: rgw enable usage log = true 2024-03-24T00:37:47.650 INFO:teuthology.orchestra.run.smithi067.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-24T00:37:47.650 INFO:teuthology.orchestra.run.smithi067.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-24T00:37:47.650 INFO:teuthology.orchestra.run.smithi067.stdout: client mount timeout = 600 2024-03-24T00:37:47.650 INFO:teuthology.orchestra.run.smithi067.stdout: debug client = 20 2024-03-24T00:37:47.650 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-03-24T00:37:47.650 INFO:teuthology.orchestra.run.smithi067.stdout: pid file = "" 2024-03-24T00:37:47.650 INFO:teuthology.orchestra.run.smithi067.stdout: rados mon op timeout = 900 2024-03-24T00:37:47.650 INFO:teuthology.orchestra.run.smithi067.stdout: rados osd op timeout = 900 2024-03-24T00:37:47.650 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.a] 2024-03-24T00:37:47.650 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.b] 2024-03-24T00:37:47.650 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.c] 2024-03-24T00:37:47.650 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.a] 2024-03-24T00:37:47.650 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.c] 2024-03-24T00:37:47.650 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.b] 2024-03-24T00:37:47.651 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.d] 2024-03-24T00:37:47.651 INFO:teuthology.orchestra.run.smithi067.stdout:[mds] 2024-03-24T00:37:47.651 INFO:teuthology.orchestra.run.smithi067.stdout: debug mds = 20 2024-03-24T00:37:47.651 INFO:teuthology.orchestra.run.smithi067.stdout: debug mds balancer = 20 2024-03-24T00:37:47.651 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-03-24T00:37:47.651 INFO:teuthology.orchestra.run.smithi067.stdout: mds debug frag = True 2024-03-24T00:37:47.651 INFO:teuthology.orchestra.run.smithi067.stdout: mds debug scatterstat = True 2024-03-24T00:37:47.651 INFO:teuthology.orchestra.run.smithi067.stdout: mds op complaint time = 180 2024-03-24T00:37:47.651 INFO:teuthology.orchestra.run.smithi067.stdout: mds verify scatter = True 2024-03-24T00:37:47.651 INFO:teuthology.orchestra.run.smithi067.stdout: osd op complaint time = 180 2024-03-24T00:37:47.651 INFO:teuthology.orchestra.run.smithi067.stdout: rados mon op timeout = 900 2024-03-24T00:37:47.651 INFO:teuthology.orchestra.run.smithi067.stdout: rados osd op timeout = 900 2024-03-24T00:37:47.652 INFO:teuthology.orchestra.run.smithi072.stdout:[global] 2024-03-24T00:37:47.652 INFO:teuthology.orchestra.run.smithi072.stdout: chdir = "" 2024-03-24T00:37:47.652 INFO:teuthology.orchestra.run.smithi072.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-24T00:37:47.652 INFO:teuthology.orchestra.run.smithi072.stdout: auth supported = cephx 2024-03-24T00:37:47.652 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:37:47.652 INFO:teuthology.orchestra.run.smithi072.stdout: filestore xattr use omap = true 2024-03-24T00:37:47.652 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:37:47.652 INFO:teuthology.orchestra.run.smithi072.stdout: mon clock drift allowed = 1.000 2024-03-24T00:37:47.652 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:37:47.652 INFO:teuthology.orchestra.run.smithi072.stdout: osd crush chooseleaf type = 0 2024-03-24T00:37:47.652 INFO:teuthology.orchestra.run.smithi072.stdout: auth debug = true 2024-03-24T00:37:47.653 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:37:47.653 INFO:teuthology.orchestra.run.smithi072.stdout: ms die on old message = true 2024-03-24T00:37:47.653 INFO:teuthology.orchestra.run.smithi072.stdout: ms die on bug = true 2024-03-24T00:37:47.653 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:37:47.653 INFO:teuthology.orchestra.run.smithi072.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-24T00:37:47.653 INFO:teuthology.orchestra.run.smithi072.stdout: mon pg warn max object skew = 0 2024-03-24T00:37:47.653 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:37:47.653 INFO:teuthology.orchestra.run.smithi072.stdout: # disable pg_autoscaler by default for new pools 2024-03-24T00:37:47.653 INFO:teuthology.orchestra.run.smithi072.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-24T00:37:47.653 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:37:47.653 INFO:teuthology.orchestra.run.smithi072.stdout: osd pool default size = 2 2024-03-24T00:37:47.653 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:37:47.653 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd allow primary affinity = true 2024-03-24T00:37:47.654 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd allow pg remap = true 2024-03-24T00:37:47.654 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on legacy crush tunables = false 2024-03-24T00:37:47.654 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on crush straw calc version zero = false 2024-03-24T00:37:47.654 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on no sortbitwise = false 2024-03-24T00:37:47.654 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on osd down out interval zero = false 2024-03-24T00:37:47.654 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on too few osds = false 2024-03-24T00:37:47.654 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-24T00:37:47.654 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-24T00:37:47.654 INFO:teuthology.orchestra.run.smithi072.stdout: mon_allow_pool_size_one = true 2024-03-24T00:37:47.654 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:37:47.654 INFO:teuthology.orchestra.run.smithi072.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-24T00:37:47.654 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:37:47.654 INFO:teuthology.orchestra.run.smithi072.stdout: osd default data pool replay window = 5 2024-03-24T00:37:47.654 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:37:47.654 INFO:teuthology.orchestra.run.smithi072.stdout: mon allow pool delete = true 2024-03-24T00:37:47.655 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:37:47.655 INFO:teuthology.orchestra.run.smithi072.stdout: mon cluster log file level = debug 2024-03-24T00:37:47.655 INFO:teuthology.orchestra.run.smithi072.stdout: debug asserts on shutdown = true 2024-03-24T00:37:47.655 INFO:teuthology.orchestra.run.smithi072.stdout: mon health detail to clog = false 2024-03-24T00:37:47.655 INFO:teuthology.orchestra.run.smithi072.stdout: mon host = "172.21.15.67,172.21.15.72,[v2:172.21.15.72:3301,v1:172.21.15.72:6790]" 2024-03-24T00:37:47.655 INFO:teuthology.orchestra.run.smithi072.stdout: ms die on skipped message = False 2024-03-24T00:37:47.655 INFO:teuthology.orchestra.run.smithi072.stdout: fsid = de1d0178-808e-49ca-a986-e297e523b976 2024-03-24T00:37:47.655 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:37:47.655 INFO:teuthology.orchestra.run.smithi072.stdout:[osd] 2024-03-24T00:37:47.655 INFO:teuthology.orchestra.run.smithi072.stdout: osd journal size = 100 2024-03-24T00:37:47.655 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:37:47.655 INFO:teuthology.orchestra.run.smithi072.stdout: osd scrub load threshold = 5.0 2024-03-24T00:37:47.655 INFO:teuthology.orchestra.run.smithi072.stdout: osd scrub max interval = 600 2024-03-24T00:37:47.655 INFO:teuthology.orchestra.run.smithi072.stdout: osd mclock profile = high_recovery_ops 2024-03-24T00:37:47.655 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:37:47.656 INFO:teuthology.orchestra.run.smithi072.stdout: osd recover clone overlap = true 2024-03-24T00:37:47.656 INFO:teuthology.orchestra.run.smithi072.stdout: osd recovery max chunk = 1048576 2024-03-24T00:37:47.656 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:37:47.656 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug shutdown = true 2024-03-24T00:37:47.656 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug op order = true 2024-03-24T00:37:47.656 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug verify stray on activate = true 2024-03-24T00:37:47.656 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug trim objects = true 2024-03-24T00:37:47.656 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:37:47.656 INFO:teuthology.orchestra.run.smithi072.stdout: osd open classes on start = true 2024-03-24T00:37:47.656 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug pg log writeout = true 2024-03-24T00:37:47.656 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:37:47.656 INFO:teuthology.orchestra.run.smithi072.stdout: osd deep scrub update digest min age = 30 2024-03-24T00:37:47.656 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:37:47.656 INFO:teuthology.orchestra.run.smithi072.stdout: osd map max advance = 10 2024-03-24T00:37:47.656 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:37:47.657 INFO:teuthology.orchestra.run.smithi072.stdout: journal zero on create = true 2024-03-24T00:37:47.657 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:37:47.657 INFO:teuthology.orchestra.run.smithi072.stdout: filestore ondisk finisher threads = 3 2024-03-24T00:37:47.657 INFO:teuthology.orchestra.run.smithi072.stdout: filestore apply finisher threads = 3 2024-03-24T00:37:47.657 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:37:47.657 INFO:teuthology.orchestra.run.smithi072.stdout: bdev debug aio = true 2024-03-24T00:37:47.657 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug misdirected ops = true 2024-03-24T00:37:47.657 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore block size = 96636764160 2024-03-24T00:37:47.657 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore fsck on mount = True 2024-03-24T00:37:47.657 INFO:teuthology.orchestra.run.smithi072.stdout: debug bluefs = 20 2024-03-24T00:37:47.657 INFO:teuthology.orchestra.run.smithi072.stdout: debug bluestore = 20 2024-03-24T00:37:47.658 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-24T00:37:47.658 INFO:teuthology.orchestra.run.smithi072.stdout: debug osd = 20 2024-03-24T00:37:47.658 INFO:teuthology.orchestra.run.smithi072.stdout: debug rocksdb = 10 2024-03-24T00:37:47.658 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd backfillfull_ratio = 0.85 2024-03-24T00:37:47.658 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd full ratio = 0.9 2024-03-24T00:37:47.658 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd nearfull ratio = 0.8 2024-03-24T00:37:47.658 INFO:teuthology.orchestra.run.smithi072.stdout: osd failsafe full ratio = 0.95 2024-03-24T00:37:47.658 INFO:teuthology.orchestra.run.smithi072.stdout: osd objectstore = bluestore 2024-03-24T00:37:47.658 INFO:teuthology.orchestra.run.smithi072.stdout: osd op complaint time = 180 2024-03-24T00:37:47.658 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:37:47.658 INFO:teuthology.orchestra.run.smithi072.stdout:[mgr] 2024-03-24T00:37:47.658 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-24T00:37:47.659 INFO:teuthology.orchestra.run.smithi072.stdout: debug mgr = 20 2024-03-24T00:37:47.659 INFO:teuthology.orchestra.run.smithi072.stdout: debug mon = 20 2024-03-24T00:37:47.659 INFO:teuthology.orchestra.run.smithi072.stdout: debug auth = 20 2024-03-24T00:37:47.659 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min pgs per osd = 4 2024-03-24T00:37:47.659 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min bytes per osd = 10 2024-03-24T00:37:47.659 INFO:teuthology.orchestra.run.smithi072.stdout: mgr/telemetry/nag = false 2024-03-24T00:37:47.659 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:37:47.659 INFO:teuthology.orchestra.run.smithi072.stdout:[mon] 2024-03-24T00:37:47.659 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-24T00:37:47.659 INFO:teuthology.orchestra.run.smithi072.stdout: debug mon = 20 2024-03-24T00:37:47.659 INFO:teuthology.orchestra.run.smithi072.stdout: debug paxos = 20 2024-03-24T00:37:47.659 INFO:teuthology.orchestra.run.smithi072.stdout: debug auth = 20 2024-03-24T00:37:47.659 INFO:teuthology.orchestra.run.smithi072.stdout: mon data avail warn = 5 2024-03-24T00:37:47.659 INFO:teuthology.orchestra.run.smithi072.stdout: mon mgr mkfs grace = 240 2024-03-24T00:37:47.660 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min pgs per osd = 4 2024-03-24T00:37:47.660 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd reporter subtree level = osd 2024-03-24T00:37:47.660 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd prime pg temp = true 2024-03-24T00:37:47.660 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min bytes per osd = 10 2024-03-24T00:37:47.660 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:37:47.660 INFO:teuthology.orchestra.run.smithi072.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-24T00:37:47.660 INFO:teuthology.orchestra.run.smithi072.stdout: auth mon ticket ttl = 660 # 11m 2024-03-24T00:37:47.660 INFO:teuthology.orchestra.run.smithi072.stdout: auth service ticket ttl = 240 # 4m 2024-03-24T00:37:47.660 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:37:47.660 INFO:teuthology.orchestra.run.smithi072.stdout: # don't complain about insecure global_id in the test suite 2024-03-24T00:37:47.660 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-24T00:37:47.660 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-24T00:37:47.660 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:37:47.660 INFO:teuthology.orchestra.run.smithi072.stdout: # 1m isn't quite enough 2024-03-24T00:37:47.660 INFO:teuthology.orchestra.run.smithi072.stdout: mon_down_mkfs_grace = 2m 2024-03-24T00:37:47.661 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:37:47.661 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_filestore_osds = false 2024-03-24T00:37:47.661 INFO:teuthology.orchestra.run.smithi072.stdout: mon op complaint time = 120 2024-03-24T00:37:47.661 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:37:47.661 INFO:teuthology.orchestra.run.smithi072.stdout:[client] 2024-03-24T00:37:47.661 INFO:teuthology.orchestra.run.smithi072.stdout: rgw cache enabled = true 2024-03-24T00:37:47.661 INFO:teuthology.orchestra.run.smithi072.stdout: rgw enable ops log = true 2024-03-24T00:37:47.661 INFO:teuthology.orchestra.run.smithi072.stdout: rgw enable usage log = true 2024-03-24T00:37:47.661 INFO:teuthology.orchestra.run.smithi072.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-24T00:37:47.661 INFO:teuthology.orchestra.run.smithi072.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-24T00:37:47.661 INFO:teuthology.orchestra.run.smithi072.stdout: client mount timeout = 600 2024-03-24T00:37:47.661 INFO:teuthology.orchestra.run.smithi072.stdout: debug client = 20 2024-03-24T00:37:47.661 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-24T00:37:47.661 INFO:teuthology.orchestra.run.smithi072.stdout: pid file = "" 2024-03-24T00:37:47.661 INFO:teuthology.orchestra.run.smithi072.stdout: rados mon op timeout = 900 2024-03-24T00:37:47.662 INFO:teuthology.orchestra.run.smithi072.stdout: rados osd op timeout = 900 2024-03-24T00:37:47.662 INFO:teuthology.orchestra.run.smithi072.stdout:[mon.a] 2024-03-24T00:37:47.662 INFO:teuthology.orchestra.run.smithi072.stdout:[mon.b] 2024-03-24T00:37:47.662 INFO:teuthology.orchestra.run.smithi072.stdout:[mon.c] 2024-03-24T00:37:47.662 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.a] 2024-03-24T00:37:47.662 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.c] 2024-03-24T00:37:47.662 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.b] 2024-03-24T00:37:47.662 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.d] 2024-03-24T00:37:47.662 INFO:teuthology.orchestra.run.smithi072.stdout:[mds] 2024-03-24T00:37:47.662 INFO:teuthology.orchestra.run.smithi072.stdout: debug mds = 20 2024-03-24T00:37:47.662 INFO:teuthology.orchestra.run.smithi072.stdout: debug mds balancer = 20 2024-03-24T00:37:47.662 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-24T00:37:47.662 INFO:teuthology.orchestra.run.smithi072.stdout: mds debug frag = True 2024-03-24T00:37:47.662 INFO:teuthology.orchestra.run.smithi072.stdout: mds debug scatterstat = True 2024-03-24T00:37:47.663 INFO:teuthology.orchestra.run.smithi072.stdout: mds op complaint time = 180 2024-03-24T00:37:47.663 INFO:teuthology.orchestra.run.smithi072.stdout: mds verify scatter = True 2024-03-24T00:37:47.663 INFO:teuthology.orchestra.run.smithi072.stdout: osd op complaint time = 180 2024-03-24T00:37:47.663 INFO:teuthology.orchestra.run.smithi072.stdout: rados mon op timeout = 900 2024-03-24T00:37:47.663 INFO:teuthology.orchestra.run.smithi072.stdout: rados osd op timeout = 900 2024-03-24T00:37:47.664 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-24T00:37:47.664 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-24T00:37:47.760 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-24T00:37:47.760 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:37:47.760 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-24T00:37:47.774 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:37:47.774 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-24T00:37:47.828 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi067.front.sepia.ceph.com 2024-03-24T00:37:47.828 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:37:47.829 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-24T00:37:47.829 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-24T00:37:47.907 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:37:47.908 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-24T00:37:47.962 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi072.front.sepia.ceph.com 2024-03-24T00:37:47.962 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:37:47.962 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-24T00:37:47.962 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-24T00:37:48.001 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:37:48.001 DEBUG:teuthology.orchestra.run.smithi072:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-24T00:37:48.056 INFO:tasks.ceph:Setting up mon nodes... 2024-03-24T00:37:48.056 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-24T00:37:48.056 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-24T00:37:48.110 INFO:teuthology.orchestra.run.smithi067.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-24T00:37:48.113 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-24T00:37:48.178 INFO:teuthology.orchestra.run.smithi072.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-24T00:37:48.180 INFO:tasks.ceph:Setting up mds nodes... 2024-03-24T00:37:48.181 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-03-24T00:37:48.234 INFO:teuthology.orchestra.run.smithi067.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-03-24T00:37:48.237 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-03-24T00:37:48.263 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-03-24T00:37:48.357 INFO:teuthology.orchestra.run.smithi067.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-03-24T00:37:48.360 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-03-24T00:37:48.386 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-03-24T00:37:48.441 INFO:teuthology.orchestra.run.smithi072.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-03-24T00:37:48.443 DEBUG:teuthology.orchestra.run.smithi072:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-03-24T00:37:48.469 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-03-24T00:37:48.564 INFO:teuthology.orchestra.run.smithi072.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-03-24T00:37:48.567 DEBUG:teuthology.orchestra.run.smithi072:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-03-24T00:37:48.592 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-24T00:37:48.593 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-24T00:37:48.635 INFO:teuthology.orchestra.run.smithi067.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-24T00:37:48.650 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-03-24T00:37:48.732 INFO:teuthology.orchestra.run.smithi067.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-03-24T00:37:48.747 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-24T00:37:48.789 INFO:teuthology.orchestra.run.smithi072.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-24T00:37:48.804 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-03-24T00:37:48.885 INFO:teuthology.orchestra.run.smithi072.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-03-24T00:37:48.901 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-24T00:37:48.901 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi067.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi072.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-24T00:37:48.901 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-24T00:37:48.928 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-24T00:37:48.928 INFO:tasks.ceph:role: osd.0 2024-03-24T00:37:48.928 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi067.front.sepia.ceph.com 2024-03-24T00:37:48.929 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-24T00:37:49.002 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-24T00:37:49.002 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-24T00:37:49.002 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-24T00:37:49.002 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-24T00:37:49.002 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-24T00:37:49.002 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2024-03-24T00:37:49.002 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-24T00:37:49.002 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-24T00:37:49.003 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-24T00:37:49.003 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-24T00:37:49.498 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2024-03-24T00:37:49.500 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2024-03-24T00:37:49.500 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-24T00:37:49.610 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-24T00:37:49.645 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-24T00:37:49.715 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-24T00:37:49.716 INFO:tasks.ceph:role: osd.1 2024-03-24T00:37:49.716 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi067.front.sepia.ceph.com 2024-03-24T00:37:49.716 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-24T00:37:49.789 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-24T00:37:49.789 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-24T00:37:49.789 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-24T00:37:49.789 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-24T00:37:49.790 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-24T00:37:49.790 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2024-03-24T00:37:49.790 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-24T00:37:49.790 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-24T00:37:49.790 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-24T00:37:49.790 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-24T00:37:50.256 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2024-03-24T00:37:50.258 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2024-03-24T00:37:50.258 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-24T00:37:50.349 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-24T00:37:50.381 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-24T00:37:50.448 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-24T00:37:50.449 INFO:tasks.ceph:role: osd.2 2024-03-24T00:37:50.449 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi067.front.sepia.ceph.com 2024-03-24T00:37:50.449 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-24T00:37:50.522 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-24T00:37:50.522 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-24T00:37:50.523 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-24T00:37:50.523 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-24T00:37:50.523 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-24T00:37:50.523 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2024-03-24T00:37:50.523 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-24T00:37:50.523 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-24T00:37:50.523 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-24T00:37:50.523 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-24T00:37:51.002 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2024-03-24T00:37:51.004 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2024-03-24T00:37:51.004 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-24T00:37:51.129 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-24T00:37:51.162 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-24T00:37:51.228 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-24T00:37:51.228 INFO:tasks.ceph:role: osd.3 2024-03-24T00:37:51.228 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi067.front.sepia.ceph.com 2024-03-24T00:37:51.228 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-24T00:37:51.301 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-24T00:37:51.301 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-24T00:37:51.302 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-24T00:37:51.302 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-24T00:37:51.302 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-24T00:37:51.302 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2024-03-24T00:37:51.302 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-24T00:37:51.302 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-24T00:37:51.302 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-24T00:37:51.302 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-24T00:37:51.798 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2024-03-24T00:37:51.843 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2024-03-24T00:37:51.844 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-24T00:37:51.959 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-24T00:37:51.992 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-24T00:37:52.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:37:52.089+0000 7fee39c0c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-24T00:37:52.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:37:52.089+0000 7fee39c0c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-24T00:37:52.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:37:52.089+0000 7fee39c0c640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-24T00:37:52.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:37:52.089+0000 7fee39c0c640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-24T00:37:52.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:37:52.089+0000 7fee39c0c640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-24T00:37:52.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:37:52.089+0000 7fee39c0c640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-24T00:37:54.937 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-24T00:37:54.968 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-24T00:37:55.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:37:55.060+0000 7f2df1a22640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-24T00:37:55.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:37:55.061+0000 7f2df1a22640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-24T00:37:55.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:37:55.061+0000 7f2df1a22640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-24T00:37:55.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:37:55.061+0000 7f2df1a22640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-24T00:37:55.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:37:55.061+0000 7f2df1a22640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-24T00:37:55.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:37:55.061+0000 7f2df1a22640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-24T00:37:57.906 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-24T00:37:57.937 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-24T00:37:58.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:37:58.029+0000 7ff263a1e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-24T00:37:58.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:37:58.029+0000 7ff263a1e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-24T00:37:58.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:37:58.029+0000 7ff263a1e640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-24T00:37:58.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:37:58.029+0000 7ff263a1e640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-24T00:37:58.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:37:58.029+0000 7ff263a1e640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-24T00:37:58.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:37:58.030+0000 7ff263a1e640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-24T00:38:00.883 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-24T00:38:00.914 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-24T00:38:01.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:01.005+0000 7fbfad61c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-24T00:38:01.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:01.005+0000 7fbfad61c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-24T00:38:01.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:01.005+0000 7fbfad61c640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-24T00:38:01.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:01.006+0000 7fbfad61c640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-24T00:38:01.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:01.006+0000 7fbfad61c640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-24T00:38:01.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:01.006+0000 7fbfad61c640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-24T00:38:03.862 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-24T00:38:03.893 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-24T00:38:03.925 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-24T00:38:03.925 INFO:tasks.ceph:role: osd.4 2024-03-24T00:38:03.925 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi072.front.sepia.ceph.com 2024-03-24T00:38:03.925 DEBUG:teuthology.orchestra.run.smithi072:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-24T00:38:04.003 INFO:teuthology.orchestra.run.smithi072.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-24T00:38:04.003 INFO:teuthology.orchestra.run.smithi072.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-24T00:38:04.003 INFO:teuthology.orchestra.run.smithi072.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-24T00:38:04.003 INFO:teuthology.orchestra.run.smithi072.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-24T00:38:04.003 INFO:teuthology.orchestra.run.smithi072.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-24T00:38:04.003 INFO:teuthology.orchestra.run.smithi072.stdout: = sunit=0 swidth=0 blks 2024-03-24T00:38:04.003 INFO:teuthology.orchestra.run.smithi072.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-24T00:38:04.003 INFO:teuthology.orchestra.run.smithi072.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-24T00:38:04.003 INFO:teuthology.orchestra.run.smithi072.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-24T00:38:04.003 INFO:teuthology.orchestra.run.smithi072.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-24T00:38:04.510 INFO:teuthology.orchestra.run.smithi072.stdout:Discarding blocks...Done. 2024-03-24T00:38:04.513 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi072.front.sepia.ceph.com -o noatime 2024-03-24T00:38:04.513 DEBUG:teuthology.orchestra.run.smithi072:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-24T00:38:04.634 DEBUG:teuthology.orchestra.run.smithi072:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-24T00:38:04.672 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-24T00:38:04.742 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-24T00:38:04.742 INFO:tasks.ceph:role: osd.5 2024-03-24T00:38:04.742 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi072.front.sepia.ceph.com 2024-03-24T00:38:04.742 DEBUG:teuthology.orchestra.run.smithi072:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-24T00:38:04.818 INFO:teuthology.orchestra.run.smithi072.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-24T00:38:04.819 INFO:teuthology.orchestra.run.smithi072.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-24T00:38:04.819 INFO:teuthology.orchestra.run.smithi072.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-24T00:38:04.819 INFO:teuthology.orchestra.run.smithi072.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-24T00:38:04.819 INFO:teuthology.orchestra.run.smithi072.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-24T00:38:04.819 INFO:teuthology.orchestra.run.smithi072.stdout: = sunit=0 swidth=0 blks 2024-03-24T00:38:04.819 INFO:teuthology.orchestra.run.smithi072.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-24T00:38:04.819 INFO:teuthology.orchestra.run.smithi072.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-24T00:38:04.819 INFO:teuthology.orchestra.run.smithi072.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-24T00:38:04.819 INFO:teuthology.orchestra.run.smithi072.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-24T00:38:05.278 INFO:teuthology.orchestra.run.smithi072.stdout:Discarding blocks...Done. 2024-03-24T00:38:05.281 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi072.front.sepia.ceph.com -o noatime 2024-03-24T00:38:05.281 DEBUG:teuthology.orchestra.run.smithi072:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-24T00:38:05.374 DEBUG:teuthology.orchestra.run.smithi072:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-24T00:38:05.407 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-24T00:38:05.476 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-24T00:38:05.476 INFO:tasks.ceph:role: osd.6 2024-03-24T00:38:05.476 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi072.front.sepia.ceph.com 2024-03-24T00:38:05.476 DEBUG:teuthology.orchestra.run.smithi072:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-24T00:38:05.551 INFO:teuthology.orchestra.run.smithi072.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-24T00:38:05.552 INFO:teuthology.orchestra.run.smithi072.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-24T00:38:05.552 INFO:teuthology.orchestra.run.smithi072.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-24T00:38:05.552 INFO:teuthology.orchestra.run.smithi072.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-24T00:38:05.552 INFO:teuthology.orchestra.run.smithi072.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-24T00:38:05.552 INFO:teuthology.orchestra.run.smithi072.stdout: = sunit=0 swidth=0 blks 2024-03-24T00:38:05.552 INFO:teuthology.orchestra.run.smithi072.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-24T00:38:05.552 INFO:teuthology.orchestra.run.smithi072.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-24T00:38:05.552 INFO:teuthology.orchestra.run.smithi072.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-24T00:38:05.552 INFO:teuthology.orchestra.run.smithi072.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-24T00:38:06.059 INFO:teuthology.orchestra.run.smithi072.stdout:Discarding blocks...Done. 2024-03-24T00:38:06.061 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi072.front.sepia.ceph.com -o noatime 2024-03-24T00:38:06.061 DEBUG:teuthology.orchestra.run.smithi072:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-24T00:38:06.203 DEBUG:teuthology.orchestra.run.smithi072:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-24T00:38:06.236 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-24T00:38:06.304 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-24T00:38:06.304 INFO:tasks.ceph:role: osd.7 2024-03-24T00:38:06.304 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi072.front.sepia.ceph.com 2024-03-24T00:38:06.304 DEBUG:teuthology.orchestra.run.smithi072:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-24T00:38:06.379 INFO:teuthology.orchestra.run.smithi072.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-24T00:38:06.379 INFO:teuthology.orchestra.run.smithi072.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-24T00:38:06.379 INFO:teuthology.orchestra.run.smithi072.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-24T00:38:06.379 INFO:teuthology.orchestra.run.smithi072.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-24T00:38:06.379 INFO:teuthology.orchestra.run.smithi072.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-24T00:38:06.379 INFO:teuthology.orchestra.run.smithi072.stdout: = sunit=0 swidth=0 blks 2024-03-24T00:38:06.379 INFO:teuthology.orchestra.run.smithi072.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-24T00:38:06.379 INFO:teuthology.orchestra.run.smithi072.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-24T00:38:06.379 INFO:teuthology.orchestra.run.smithi072.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-24T00:38:06.379 INFO:teuthology.orchestra.run.smithi072.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-24T00:38:06.890 INFO:teuthology.orchestra.run.smithi072.stdout:Discarding blocks...Done. 2024-03-24T00:38:06.892 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi072.front.sepia.ceph.com -o noatime 2024-03-24T00:38:06.892 DEBUG:teuthology.orchestra.run.smithi072:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-24T00:38:07.024 DEBUG:teuthology.orchestra.run.smithi072:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-24T00:38:07.060 DEBUG:teuthology.orchestra.run.smithi072:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-24T00:38:07.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:07.161+0000 7f1e7a824640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-24T00:38:07.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:07.161+0000 7f1e7a824640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-24T00:38:07.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:07.161+0000 7f1e7a824640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-24T00:38:07.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:07.161+0000 7f1e7a824640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-24T00:38:07.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:07.161+0000 7f1e7a824640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-24T00:38:07.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:07.161+0000 7f1e7a824640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-24T00:38:10.015 DEBUG:teuthology.orchestra.run.smithi072:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-24T00:38:10.045 DEBUG:teuthology.orchestra.run.smithi072:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-24T00:38:10.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:10.135+0000 7ff4dd9bd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-24T00:38:10.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:10.135+0000 7ff4dd9bd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-24T00:38:10.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:10.135+0000 7ff4dd9bd640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-24T00:38:10.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:10.135+0000 7ff4dd9bd640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-24T00:38:10.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:10.135+0000 7ff4dd9bd640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-24T00:38:10.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:10.135+0000 7ff4dd9bd640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-24T00:38:12.985 DEBUG:teuthology.orchestra.run.smithi072:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-24T00:38:13.015 DEBUG:teuthology.orchestra.run.smithi072:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-24T00:38:13.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:13.106+0000 7f33da20f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-24T00:38:13.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:13.106+0000 7f33da20f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-24T00:38:13.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:13.106+0000 7f33da20f640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-24T00:38:13.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:13.106+0000 7f33da20f640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-24T00:38:13.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:13.106+0000 7f33da20f640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-24T00:38:13.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:13.107+0000 7f33da20f640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-24T00:38:15.681 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:38:15.684 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:38:15.967 DEBUG:teuthology.orchestra.run.smithi072:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-24T00:38:15.998 DEBUG:teuthology.orchestra.run.smithi072:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-24T00:38:16.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:16.089+0000 7f112698a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-24T00:38:16.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:16.089+0000 7f112698a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-24T00:38:16.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:16.089+0000 7f112698a640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-24T00:38:16.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:16.089+0000 7f112698a640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-24T00:38:16.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:16.089+0000 7f112698a640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-24T00:38:16.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:16.089+0000 7f112698a640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-24T00:38:18.931 DEBUG:teuthology.orchestra.run.smithi072:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-24T00:38:18.962 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-24T00:38:18.963 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:38:18.963 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-24T00:38:18.994 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:38:18.994 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-03-24T00:38:19.065 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:38:19.065 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-03-24T00:38:19.135 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:38:19.135 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-24T00:38:19.204 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:38:19.205 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-24T00:38:19.272 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:38:19.272 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-24T00:38:19.339 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:38:19.339 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-24T00:38:19.406 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:38:19.406 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-24T00:38:19.438 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:38:19.438 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-03-24T00:38:19.508 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:38:19.508 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-03-24T00:38:19.579 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:38:19.579 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-24T00:38:19.648 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:38:19.648 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-24T00:38:19.717 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:38:19.717 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-24T00:38:19.786 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:38:19.786 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-24T00:38:19.854 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:38:19.855 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-24T00:38:19.871 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:38:19.871 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-03-24T00:38:19.927 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:38:19.927 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-24T00:38:19.943 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:38:19.943 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-03-24T00:38:19.999 INFO:tasks.ceph:Adding keys to all mons... 2024-03-24T00:38:19.999 DEBUG:teuthology.orchestra.run.smithi067:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-24T00:38:20.002 DEBUG:teuthology.orchestra.run.smithi072:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-24T00:38:20.042 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr.y] 2024-03-24T00:38:20.042 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQDcdf9lSvufBhAAoQ9L1GjLkstjrAw2lqlryA== 2024-03-24T00:38:20.043 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.a] 2024-03-24T00:38:20.043 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQDcdf9l4UENDhAASKUDtqcbSQ7wADj7uo4eyQ== 2024-03-24T00:38:20.043 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.c] 2024-03-24T00:38:20.043 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQDcdf9l8mRjFRAAQ73yexNddpcyp8ayTcRKCg== 2024-03-24T00:38:20.043 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.0] 2024-03-24T00:38:20.043 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQDgdf9lV/NUBRAAyJNajFAnNABJErnBlB8TEA== 2024-03-24T00:38:20.043 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.1] 2024-03-24T00:38:20.043 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQDjdf9l0J6lAxAA9RInt/X22jqzthE0vOiiFg== 2024-03-24T00:38:20.043 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.2] 2024-03-24T00:38:20.043 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQDmdf9lSmrEARAAwmtSQRNePsn4WIZODnjhfQ== 2024-03-24T00:38:20.043 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.3] 2024-03-24T00:38:20.043 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQDpdf9lm59bABAAgJ5AV774lw2BgkSCgaweyA== 2024-03-24T00:38:20.043 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr.x] 2024-03-24T00:38:20.043 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQDcdf9lGwKvChAAxl+fRnjbAQ7dYJ727QFUgw== 2024-03-24T00:38:20.043 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.b] 2024-03-24T00:38:20.044 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQDcdf9luhJWGhAAsS6PFHV2XdF9htjIjzdrdA== 2024-03-24T00:38:20.044 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.d] 2024-03-24T00:38:20.044 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQDcdf9lcJyvIRAA2tpDb6qxn1MtljjEno4qPA== 2024-03-24T00:38:20.044 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.4] 2024-03-24T00:38:20.044 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQDvdf9lCo+pCRAANHkL0pPxxok6IsBCLAFCbg== 2024-03-24T00:38:20.044 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.5] 2024-03-24T00:38:20.044 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQDydf9lyGEdCBAAcuszPsthGlLvVryE7+D9iQ== 2024-03-24T00:38:20.044 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.6] 2024-03-24T00:38:20.044 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQD1df9ljuZqBhAACt2OPNkjKOD3rK7g2mVBBA== 2024-03-24T00:38:20.044 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.7] 2024-03-24T00:38:20.044 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQD4df9lYKBfBRAAy/cWFF6pvGDjXZHAuBYw7g== 2024-03-24T00:38:20.044 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-03-24T00:38:20.044 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQDcdf9lY4nvJRAA6HWOZDWA+SSvJVmyjG+51g== 2024-03-24T00:38:20.044 INFO:teuthology.orchestra.run.smithi067.stdout:[client.2] 2024-03-24T00:38:20.044 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQDcdf9lLEm2KxAAkT65mhLrTirlDnOUyvBY7g== 2024-03-24T00:38:20.045 INFO:teuthology.orchestra.run.smithi067.stdout:[client.1] 2024-03-24T00:38:20.045 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQDcdf9lonsXLxAAaltVMW3n0qHX8u9X8EkSmg== 2024-03-24T00:38:20.045 INFO:teuthology.orchestra.run.smithi067.stdout:[client.3] 2024-03-24T00:38:20.045 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQDcdf9l8njZNBAAaAktZpX2SPcJR9ymPm9ebw== 2024-03-24T00:38:20.066 INFO:teuthology.orchestra.run.smithi072.stdout:[mgr.y] 2024-03-24T00:38:20.066 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQDcdf9lSvufBhAAoQ9L1GjLkstjrAw2lqlryA== 2024-03-24T00:38:20.066 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.a] 2024-03-24T00:38:20.066 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQDcdf9l4UENDhAASKUDtqcbSQ7wADj7uo4eyQ== 2024-03-24T00:38:20.066 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.c] 2024-03-24T00:38:20.066 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQDcdf9l8mRjFRAAQ73yexNddpcyp8ayTcRKCg== 2024-03-24T00:38:20.066 INFO:teuthology.orchestra.run.smithi072.stdout:[osd.0] 2024-03-24T00:38:20.066 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQDgdf9lV/NUBRAAyJNajFAnNABJErnBlB8TEA== 2024-03-24T00:38:20.066 INFO:teuthology.orchestra.run.smithi072.stdout:[osd.1] 2024-03-24T00:38:20.067 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQDjdf9l0J6lAxAA9RInt/X22jqzthE0vOiiFg== 2024-03-24T00:38:20.067 INFO:teuthology.orchestra.run.smithi072.stdout:[osd.2] 2024-03-24T00:38:20.067 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQDmdf9lSmrEARAAwmtSQRNePsn4WIZODnjhfQ== 2024-03-24T00:38:20.067 INFO:teuthology.orchestra.run.smithi072.stdout:[osd.3] 2024-03-24T00:38:20.067 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQDpdf9lm59bABAAgJ5AV774lw2BgkSCgaweyA== 2024-03-24T00:38:20.067 INFO:teuthology.orchestra.run.smithi072.stdout:[mgr.x] 2024-03-24T00:38:20.067 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQDcdf9lGwKvChAAxl+fRnjbAQ7dYJ727QFUgw== 2024-03-24T00:38:20.067 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.b] 2024-03-24T00:38:20.067 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQDcdf9luhJWGhAAsS6PFHV2XdF9htjIjzdrdA== 2024-03-24T00:38:20.067 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.d] 2024-03-24T00:38:20.067 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQDcdf9lcJyvIRAA2tpDb6qxn1MtljjEno4qPA== 2024-03-24T00:38:20.067 INFO:teuthology.orchestra.run.smithi072.stdout:[osd.4] 2024-03-24T00:38:20.067 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQDvdf9lCo+pCRAANHkL0pPxxok6IsBCLAFCbg== 2024-03-24T00:38:20.067 INFO:teuthology.orchestra.run.smithi072.stdout:[osd.5] 2024-03-24T00:38:20.068 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQDydf9lyGEdCBAAcuszPsthGlLvVryE7+D9iQ== 2024-03-24T00:38:20.068 INFO:teuthology.orchestra.run.smithi072.stdout:[osd.6] 2024-03-24T00:38:20.068 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQD1df9ljuZqBhAACt2OPNkjKOD3rK7g2mVBBA== 2024-03-24T00:38:20.068 INFO:teuthology.orchestra.run.smithi072.stdout:[osd.7] 2024-03-24T00:38:20.068 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQD4df9lYKBfBRAAy/cWFF6pvGDjXZHAuBYw7g== 2024-03-24T00:38:20.068 INFO:teuthology.orchestra.run.smithi072.stdout:[client.0] 2024-03-24T00:38:20.068 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQDcdf9lY4nvJRAA6HWOZDWA+SSvJVmyjG+51g== 2024-03-24T00:38:20.068 INFO:teuthology.orchestra.run.smithi072.stdout:[client.2] 2024-03-24T00:38:20.068 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQDcdf9lLEm2KxAAkT65mhLrTirlDnOUyvBY7g== 2024-03-24T00:38:20.068 INFO:teuthology.orchestra.run.smithi072.stdout:[client.1] 2024-03-24T00:38:20.068 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQDcdf9lonsXLxAAaltVMW3n0qHX8u9X8EkSmg== 2024-03-24T00:38:20.068 INFO:teuthology.orchestra.run.smithi072.stdout:[client.3] 2024-03-24T00:38:20.068 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQDcdf9l8njZNBAAaAktZpX2SPcJR9ymPm9ebw== 2024-03-24T00:38:20.069 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-24T00:38:20.088 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-24T00:38:20.163 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-24T00:38:20.166 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-24T00:38:20.221 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-24T00:38:20.223 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-24T00:38:20.277 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T00:38:20.280 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T00:38:20.332 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T00:38:20.334 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T00:38:20.385 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T00:38:20.387 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T00:38:20.440 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T00:38:20.442 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T00:38:20.495 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-24T00:38:20.497 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-24T00:38:20.548 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-24T00:38:20.551 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-24T00:38:20.606 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-24T00:38:20.608 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-24T00:38:20.662 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T00:38:20.664 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T00:38:20.715 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T00:38:20.717 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T00:38:20.769 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T00:38:20.771 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T00:38:20.823 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T00:38:20.825 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T00:38:20.877 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-24T00:38:20.879 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-24T00:38:20.934 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-24T00:38:20.936 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-24T00:38:20.992 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-24T00:38:20.994 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-24T00:38:21.048 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-24T00:38:21.050 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-24T00:38:21.105 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-24T00:38:21.105 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-24T00:38:21.131 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-24T00:38:21.229 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-24T00:38:21.256 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-24T00:38:21.282 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-24T00:38:21.379 DEBUG:teuthology.orchestra.run.smithi072:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-24T00:38:21.405 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-24T00:38:21.471 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-24T00:38:21.568 DEBUG:teuthology.orchestra.run.smithi072:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-24T00:38:21.594 DEBUG:teuthology.orchestra.run.smithi067:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-24T00:38:21.596 DEBUG:teuthology.orchestra.run.smithi072:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-24T00:38:21.649 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-24T00:38:21.650 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-24T00:38:21.650 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-24T00:38:21.652 INFO:tasks.ceph.mon.a:Started 2024-03-24T00:38:21.652 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-24T00:38:21.652 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-24T00:38:21.692 INFO:tasks.ceph.mon.b:Started 2024-03-24T00:38:21.692 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-24T00:38:21.692 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-24T00:38:21.695 INFO:tasks.ceph.mon.c:Started 2024-03-24T00:38:21.695 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-24T00:38:21.695 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-24T00:38:21.695 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-24T00:38:21.698 INFO:tasks.ceph.mgr.y:Started 2024-03-24T00:38:21.698 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-24T00:38:21.698 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-24T00:38:21.700 INFO:tasks.ceph.mgr.x:Started 2024-03-24T00:38:21.700 INFO:tasks.ceph:Setting crush tunables to default 2024-03-24T00:38:21.700 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph osd crush tunables default 2024-03-24T00:38:21.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:21.801+0000 7ff619481640 1 Processor -- start 2024-03-24T00:38:21.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:21.801+0000 7ff619481640 1 -- start start 2024-03-24T00:38:21.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:21.802+0000 7ff619481640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff6140d0e10 0x7ff6140cb560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:21.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:21.802+0000 7ff619481640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6140cc290 con 0x7ff61415e350 2024-03-24T00:38:21.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:21.802+0000 7ff619481640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6140cc570 con 0x7ff6140cbaa0 2024-03-24T00:38:21.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:21.802+0000 7ff619481640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6140ce950 con 0x7ff6140d0e10 2024-03-24T00:38:21.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:21.802+0000 7ff612ffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff61415e350 0x7ff6140d0700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53976/0 (socket says 172.21.15.67:53976) 2024-03-24T00:38:21.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:21.802+0000 7ff612ffd640 1 -- 172.21.15.67:0/1376095656 learned_addr learned my addr 172.21.15.67:0/1376095656 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:38:21.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:21.802+0000 7ff6127fc640 1 --2- 172.21.15.67:0/1376095656 >> v2:172.21.15.72:3300/0 conn(0x7ff6140d0e10 0x7ff6140cb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:21.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:21.802+0000 7ff6127fc640 1 -- 172.21.15.67:0/1376095656 >> v2:172.21.15.72:3300/0 conn(0x7ff6140d0e10 msgr2=0x7ff6140cb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-24T00:38:21.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:21.802+0000 7ff6127fc640 1 -- 172.21.15.67:0/1376095656 >> v2:172.21.15.72:3300/0 conn(0x7ff6140d0e10 msgr2=0x7ff6140cb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T00:38:21.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:21.802+0000 7ff6127fc640 1 --2- 172.21.15.67:0/1376095656 >> v2:172.21.15.72:3300/0 conn(0x7ff6140d0e10 0x7ff6140cb560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T00:38:21.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:21.803+0000 7ff6127fc640 1 --2- 172.21.15.67:0/1376095656 >> v2:172.21.15.72:3300/0 conn(0x7ff6140d0e10 0x7ff6140cb560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:38:22.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:22.004+0000 7ff6127fc640 1 --2- 172.21.15.67:0/1376095656 >> v2:172.21.15.72:3300/0 conn(0x7ff6140d0e10 0x7ff6140cb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:22.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:22.004+0000 7ff6127fc640 1 -- 172.21.15.67:0/1376095656 >> v2:172.21.15.72:3300/0 conn(0x7ff6140d0e10 msgr2=0x7ff6140cb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-24T00:38:22.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:22.004+0000 7ff6127fc640 1 -- 172.21.15.67:0/1376095656 >> v2:172.21.15.72:3300/0 conn(0x7ff6140d0e10 msgr2=0x7ff6140cb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T00:38:22.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:22.004+0000 7ff6127fc640 1 --2- 172.21.15.67:0/1376095656 >> v2:172.21.15.72:3300/0 conn(0x7ff6140d0e10 0x7ff6140cb560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T00:38:22.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:22.004+0000 7ff6127fc640 1 --2- 172.21.15.67:0/1376095656 >> v2:172.21.15.72:3300/0 conn(0x7ff6140d0e10 0x7ff6140cb560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:38:22.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:22.405+0000 7ff6127fc640 1 --2- 172.21.15.67:0/1376095656 >> v2:172.21.15.72:3300/0 conn(0x7ff6140d0e10 0x7ff6140cb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:22.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:22.405+0000 7ff6127fc640 1 -- 172.21.15.67:0/1376095656 >> v2:172.21.15.72:3300/0 conn(0x7ff6140d0e10 msgr2=0x7ff6140cb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-24T00:38:22.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:22.405+0000 7ff6127fc640 1 -- 172.21.15.67:0/1376095656 >> v2:172.21.15.72:3300/0 conn(0x7ff6140d0e10 msgr2=0x7ff6140cb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T00:38:22.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:22.405+0000 7ff6127fc640 1 --2- 172.21.15.67:0/1376095656 >> v2:172.21.15.72:3300/0 conn(0x7ff6140d0e10 0x7ff6140cb560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T00:38:22.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:22.405+0000 7ff6127fc640 1 --2- 172.21.15.67:0/1376095656 >> v2:172.21.15.72:3300/0 conn(0x7ff6140d0e10 0x7ff6140cb560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:38:23.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:23.206+0000 7ff6127fc640 1 --2- 172.21.15.67:0/1376095656 >> v2:172.21.15.72:3300/0 conn(0x7ff6140d0e10 0x7ff6140cb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:23.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:23.206+0000 7ff6127fc640 1 -- 172.21.15.67:0/1376095656 >> v2:172.21.15.72:3300/0 conn(0x7ff6140d0e10 msgr2=0x7ff6140cb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-24T00:38:23.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:23.206+0000 7ff6127fc640 1 -- 172.21.15.67:0/1376095656 >> v2:172.21.15.72:3300/0 conn(0x7ff6140d0e10 msgr2=0x7ff6140cb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T00:38:23.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:23.206+0000 7ff6127fc640 1 --2- 172.21.15.67:0/1376095656 >> v2:172.21.15.72:3300/0 conn(0x7ff6140d0e10 0x7ff6140cb560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T00:38:23.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:23.206+0000 7ff6127fc640 1 --2- 172.21.15.67:0/1376095656 >> v2:172.21.15.72:3300/0 conn(0x7ff6140d0e10 0x7ff6140cb560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:38:24.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:24.802+0000 7ff610ff9640 1 -- 172.21.15.67:0/1376095656 >> v2:172.21.15.72:3300/0 conn(0x7ff6140d0e10 msgr2=0x7ff6140cb560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:38:24.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:24.802+0000 7ff610ff9640 1 --2- 172.21.15.67:0/1376095656 >> v2:172.21.15.72:3300/0 conn(0x7ff6140d0e10 0x7ff6140cb560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:24.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:24.802+0000 7ff610ff9640 1 -- 172.21.15.67:0/1376095656 >> v1:172.21.15.72:6789/0 conn(0x7ff6140cbaa0 legacy=0x7ff6140c4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:24.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:24.802+0000 7ff610ff9640 1 -- 172.21.15.67:0/1376095656 >> v1:172.21.15.67:6789/0 conn(0x7ff61415e350 legacy=0x7ff6140d0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:24.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:24.802+0000 7ff610ff9640 1 --2- 172.21.15.67:0/1376095656 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff600003710 0x7ff600005b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:24.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:24.802+0000 7ff610ff9640 1 --2- 172.21.15.67:0/1376095656 >> v2:172.21.15.67:3300/0 conn(0x7ff6000061f0 0x7ff6000086b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:24.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:24.802+0000 7ff610ff9640 1 -- 172.21.15.67:0/1376095656 --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6140ce950 con 0x7ff6000061f0 2024-03-24T00:38:24.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:24.802+0000 7ff610ff9640 1 -- 172.21.15.67:0/1376095656 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff600008d70 con 0x7ff600003710 2024-03-24T00:38:24.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:24.802+0000 7ff6127fc640 1 --2- 172.21.15.67:0/1376095656 >> v2:172.21.15.67:3300/0 conn(0x7ff6000061f0 0x7ff6000086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:24.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:24.802+0000 7ff6127fc640 1 -- 172.21.15.67:0/1376095656 >> v2:172.21.15.67:3300/0 conn(0x7ff6000061f0 msgr2=0x7ff6000086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-24T00:38:24.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:24.802+0000 7ff6127fc640 1 -- 172.21.15.67:0/1376095656 >> v2:172.21.15.67:3300/0 conn(0x7ff6000061f0 msgr2=0x7ff6000086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T00:38:24.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:24.802+0000 7ff6127fc640 1 --2- 172.21.15.67:0/1376095656 >> v2:172.21.15.67:3300/0 conn(0x7ff6000061f0 0x7ff6000086b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T00:38:24.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:24.802+0000 7ff612ffd640 1 --2- 172.21.15.67:0/1376095656 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff600003710 0x7ff600005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:24.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:24.802+0000 7ff6127fc640 1 --2- 172.21.15.67:0/1376095656 >> v2:172.21.15.67:3300/0 conn(0x7ff6000061f0 0x7ff6000086b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:38:24.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:24.803+0000 7ff612ffd640 1 -- 172.21.15.67:0/1376095656 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff600003710 msgr2=0x7ff600005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-24T00:38:24.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:24.803+0000 7ff612ffd640 1 -- 172.21.15.67:0/1376095656 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff600003710 msgr2=0x7ff600005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T00:38:24.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:24.803+0000 7ff612ffd640 1 --2- 172.21.15.67:0/1376095656 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff600003710 0x7ff600005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T00:38:24.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:24.803+0000 7ff612ffd640 1 --2- 172.21.15.67:0/1376095656 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff600003710 0x7ff600005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:38:25.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:25.003+0000 7ff6127fc640 1 --2- 172.21.15.67:0/1376095656 >> v2:172.21.15.67:3300/0 conn(0x7ff6000061f0 0x7ff6000086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:25.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:25.003+0000 7ff6127fc640 1 -- 172.21.15.67:0/1376095656 >> v2:172.21.15.67:3300/0 conn(0x7ff6000061f0 msgr2=0x7ff6000086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-24T00:38:25.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:25.003+0000 7ff6127fc640 1 -- 172.21.15.67:0/1376095656 >> v2:172.21.15.67:3300/0 conn(0x7ff6000061f0 msgr2=0x7ff6000086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T00:38:25.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:25.003+0000 7ff6127fc640 1 --2- 172.21.15.67:0/1376095656 >> v2:172.21.15.67:3300/0 conn(0x7ff6000061f0 0x7ff6000086b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T00:38:25.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:25.003+0000 7ff6127fc640 1 --2- 172.21.15.67:0/1376095656 >> v2:172.21.15.67:3300/0 conn(0x7ff6000061f0 0x7ff6000086b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:38:25.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:25.004+0000 7ff612ffd640 1 --2- 172.21.15.67:0/1376095656 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff600003710 0x7ff600005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:25.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:25.004+0000 7ff612ffd640 1 -- 172.21.15.67:0/1376095656 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff600003710 msgr2=0x7ff600005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-24T00:38:25.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:25.004+0000 7ff612ffd640 1 -- 172.21.15.67:0/1376095656 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff600003710 msgr2=0x7ff600005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T00:38:25.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:25.004+0000 7ff612ffd640 1 --2- 172.21.15.67:0/1376095656 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff600003710 0x7ff600005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T00:38:25.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:25.004+0000 7ff612ffd640 1 --2- 172.21.15.67:0/1376095656 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff600003710 0x7ff600005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:38:25.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:25.404+0000 7ff6127fc640 1 --2- 172.21.15.67:0/1376095656 >> v2:172.21.15.67:3300/0 conn(0x7ff6000061f0 0x7ff6000086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:25.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:25.404+0000 7ff6127fc640 1 -- 172.21.15.67:0/1376095656 >> v2:172.21.15.67:3300/0 conn(0x7ff6000061f0 msgr2=0x7ff6000086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-24T00:38:25.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:25.404+0000 7ff6127fc640 1 -- 172.21.15.67:0/1376095656 >> v2:172.21.15.67:3300/0 conn(0x7ff6000061f0 msgr2=0x7ff6000086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T00:38:25.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:25.404+0000 7ff6127fc640 1 --2- 172.21.15.67:0/1376095656 >> v2:172.21.15.67:3300/0 conn(0x7ff6000061f0 0x7ff6000086b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T00:38:25.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:25.404+0000 7ff6127fc640 1 --2- 172.21.15.67:0/1376095656 >> v2:172.21.15.67:3300/0 conn(0x7ff6000061f0 0x7ff6000086b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:38:25.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:25.405+0000 7ff612ffd640 1 --2- 172.21.15.67:0/1376095656 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff600003710 0x7ff600005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:25.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:25.405+0000 7ff612ffd640 1 -- 172.21.15.67:0/1376095656 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff600003710 msgr2=0x7ff600005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-24T00:38:25.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:25.405+0000 7ff612ffd640 1 -- 172.21.15.67:0/1376095656 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff600003710 msgr2=0x7ff600005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T00:38:25.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:25.405+0000 7ff612ffd640 1 --2- 172.21.15.67:0/1376095656 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff600003710 0x7ff600005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T00:38:25.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:25.405+0000 7ff612ffd640 1 --2- 172.21.15.67:0/1376095656 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff600003710 0x7ff600005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:38:26.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:26.205+0000 7ff6127fc640 1 --2- 172.21.15.67:0/1376095656 >> v2:172.21.15.67:3300/0 conn(0x7ff6000061f0 0x7ff6000086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:26.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:26.205+0000 7ff6127fc640 1 -- 172.21.15.67:0/1376095656 >> v2:172.21.15.67:3300/0 conn(0x7ff6000061f0 msgr2=0x7ff6000086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-24T00:38:26.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:26.205+0000 7ff6127fc640 1 -- 172.21.15.67:0/1376095656 >> v2:172.21.15.67:3300/0 conn(0x7ff6000061f0 msgr2=0x7ff6000086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T00:38:26.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:26.205+0000 7ff6127fc640 1 --2- 172.21.15.67:0/1376095656 >> v2:172.21.15.67:3300/0 conn(0x7ff6000061f0 0x7ff6000086b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T00:38:26.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:26.205+0000 7ff6127fc640 1 --2- 172.21.15.67:0/1376095656 >> v2:172.21.15.67:3300/0 conn(0x7ff6000061f0 0x7ff6000086b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:38:26.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:26.206+0000 7ff612ffd640 1 --2- 172.21.15.67:0/1376095656 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff600003710 0x7ff600005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:26.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:26.206+0000 7ff612ffd640 1 -- 172.21.15.67:0/1376095656 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff600003710 msgr2=0x7ff600005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-24T00:38:26.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:26.206+0000 7ff612ffd640 1 -- 172.21.15.67:0/1376095656 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff600003710 msgr2=0x7ff600005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T00:38:26.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:26.206+0000 7ff612ffd640 1 --2- 172.21.15.67:0/1376095656 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff600003710 0x7ff600005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T00:38:26.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:26.206+0000 7ff612ffd640 1 --2- 172.21.15.67:0/1376095656 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff600003710 0x7ff600005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:38:27.529 INFO:tasks.ceph.mgr.x.smithi072.stderr:2024-03-24T00:38:27.528+0000 7f19dd22b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T00:38:27.542 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-24T00:38:27.541+0000 7fa361c66040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T00:38:27.582 INFO:tasks.ceph.mgr.x.smithi072.stderr:2024-03-24T00:38:27.582+0000 7f19dd22b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T00:38:27.597 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-24T00:38:27.597+0000 7fa361c66040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T00:38:27.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.802+0000 7ff610ff9640 1 -- 172.21.15.67:0/1376095656 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff600003710 msgr2=0x7ff600005b60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:38:27.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.802+0000 7ff610ff9640 1 --2- 172.21.15.67:0/1376095656 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff600003710 0x7ff600005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:27.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.802+0000 7ff610ff9640 1 -- 172.21.15.67:0/1376095656 >> v2:172.21.15.67:3300/0 conn(0x7ff6000061f0 msgr2=0x7ff6000086b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:38:27.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.802+0000 7ff610ff9640 1 --2- 172.21.15.67:0/1376095656 >> v2:172.21.15.67:3300/0 conn(0x7ff6000061f0 0x7ff6000086b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:27.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.802+0000 7ff610ff9640 1 --2- 172.21.15.67:0/1376095656 >> v2:172.21.15.67:3300/0 conn(0x7ff600009270 0x7ff60000b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:27.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.802+0000 7ff6137fe640 1 -- 172.21.15.67:0/1376095656 reap_dead start 2024-03-24T00:38:27.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.802+0000 7ff610ff9640 1 -- 172.21.15.67:0/1376095656 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6140cc850 con 0x7ff61415e350 2024-03-24T00:38:27.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.802+0000 7ff610ff9640 1 -- 172.21.15.67:0/1376095656 --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff600008840 con 0x7ff6140cbaa0 2024-03-24T00:38:27.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.802+0000 7ff610ff9640 1 -- 172.21.15.67:0/1376095656 --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6140ce950 con 0x7ff600009270 2024-03-24T00:38:27.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.802+0000 7ff612ffd640 1 --2- 172.21.15.67:0/1376095656 >> v2:172.21.15.67:3300/0 conn(0x7ff600009270 0x7ff60000b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:27.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.803+0000 7ff612ffd640 1 -- 172.21.15.67:0/1376095656 >> v1:172.21.15.72:6789/0 conn(0x7ff6140cbaa0 legacy=0x7ff60000f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:27.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.803+0000 7ff612ffd640 1 -- 172.21.15.67:0/1376095656 >> v1:172.21.15.67:6789/0 conn(0x7ff61415e350 legacy=0x7ff60000bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:27.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.803+0000 7ff612ffd640 1 -- 172.21.15.67:0/1376095656 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff600008a10 con 0x7ff600009270 2024-03-24T00:38:27.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.803+0000 7ff611ffb640 1 -- 172.21.15.67:0/1376095656 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1122397340 0 0) 0x7ff6140cc850 con 0x7ff61415e350 2024-03-24T00:38:27.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.803+0000 7ff612ffd640 1 --2- 172.21.15.67:0/1376095656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff600009270 0x7ff60000b660 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff608007d20 tx=0x7ff60802c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f77baea7d454bd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:27.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.804+0000 7ff611ffb640 1 -- 172.21.15.67:0/1376095656 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff60803a070 con 0x7ff600009270 2024-03-24T00:38:27.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.804+0000 7ff611ffb640 1 -- 172.21.15.67:0/1376095656 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff608035410 con 0x7ff600009270 2024-03-24T00:38:27.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.804+0000 7ff611ffb640 1 -- 172.21.15.67:0/1376095656 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6080357e0 con 0x7ff600009270 2024-03-24T00:38:27.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.804+0000 7ff619481640 1 -- 172.21.15.67:0/1376095656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff600009270 msgr2=0x7ff60000b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:27.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.804+0000 7ff619481640 1 --2- 172.21.15.67:0/1376095656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff600009270 0x7ff60000b660 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff608007d20 tx=0x7ff60802c4e0 comp rx=0 tx=0).stop 2024-03-24T00:38:27.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.804+0000 7ff619481640 1 -- 172.21.15.67:0/1376095656 shutdown_connections 2024-03-24T00:38:27.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.804+0000 7ff619481640 1 --2- 172.21.15.67:0/1376095656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff600009270 0x7ff60000b660 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:27.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.804+0000 7ff619481640 1 -- 172.21.15.67:0/1376095656 >> 172.21.15.67:0/1376095656 conn(0x7ff614099a30 msgr2=0x7ff614099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:27.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.804+0000 7ff619481640 1 -- 172.21.15.67:0/1376095656 shutdown_connections 2024-03-24T00:38:27.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.804+0000 7ff619481640 1 -- 172.21.15.67:0/1376095656 wait complete. 2024-03-24T00:38:27.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.806+0000 7ff619481640 1 Processor -- start 2024-03-24T00:38:27.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.806+0000 7ff619481640 1 -- start start 2024-03-24T00:38:27.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.806+0000 7ff619481640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff600009270 0x7ff614150e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:27.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.806+0000 7ff619481640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff6140cbaa0 0x7ff6140b4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:27.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.807+0000 7ff619481640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff61415e350 0x7ff6140c9f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:27.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.807+0000 7ff619481640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6140cbf30 con 0x7ff600009270 2024-03-24T00:38:27.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.807+0000 7ff619481640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6140cce10 con 0x7ff6140cbaa0 2024-03-24T00:38:27.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.807+0000 7ff619481640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6140cb560 con 0x7ff61415e350 2024-03-24T00:38:27.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.807+0000 7ff612ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff600009270 0x7ff614150e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:27.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.807+0000 7ff612ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff600009270 0x7ff614150e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55830/0 (socket says 172.21.15.67:55830) 2024-03-24T00:38:27.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.807+0000 7ff612ffd640 1 -- 172.21.15.67:0/453993216 learned_addr learned my addr 172.21.15.67:0/453993216 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:38:27.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.807+0000 7ff6127fc640 1 --2- 172.21.15.67:0/453993216 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff6140cbaa0 0x7ff6140b4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:27.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.807+0000 7ff6137fe640 1 --2- 172.21.15.67:0/453993216 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff61415e350 0x7ff6140c9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:27.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.807+0000 7ff612ffd640 1 -- 172.21.15.67:0/453993216 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff61415e350 msgr2=0x7ff6140c9f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:27.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.807+0000 7ff612ffd640 1 --2- 172.21.15.67:0/453993216 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff61415e350 0x7ff6140c9f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:27.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.807+0000 7ff612ffd640 1 -- 172.21.15.67:0/453993216 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff6140cbaa0 msgr2=0x7ff6140b4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:27.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.807+0000 7ff612ffd640 1 --2- 172.21.15.67:0/453993216 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff6140cbaa0 0x7ff6140b4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:27.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.807+0000 7ff612ffd640 1 -- 172.21.15.67:0/453993216 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6141523e0 con 0x7ff600009270 2024-03-24T00:38:27.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.807+0000 7ff612ffd640 1 --2- 172.21.15.67:0/453993216 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff600009270 0x7ff614150e60 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff60802c520 tx=0x7ff608001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:27.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.807+0000 7ff5fb7fe640 1 -- 172.21.15.67:0/453993216 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff608044070 con 0x7ff600009270 2024-03-24T00:38:27.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.807+0000 7ff5fb7fe640 1 -- 172.21.15.67:0/453993216 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff608005630 con 0x7ff600009270 2024-03-24T00:38:27.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.807+0000 7ff619481640 1 -- 172.21.15.67:0/453993216 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6141526c0 con 0x7ff600009270 2024-03-24T00:38:27.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.807+0000 7ff5fb7fe640 1 -- 172.21.15.67:0/453993216 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff60803a040 con 0x7ff600009270 2024-03-24T00:38:27.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.807+0000 7ff619481640 1 -- 172.21.15.67:0/453993216 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff614152c80 con 0x7ff600009270 2024-03-24T00:38:27.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.807+0000 7ff5fb7fe640 1 -- 172.21.15.67:0/453993216 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7ff608047070 con 0x7ff600009270 2024-03-24T00:38:27.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.807+0000 7ff5fb7fe640 1 -- 172.21.15.67:0/453993216 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff60803f610 con 0x7ff600009270 2024-03-24T00:38:27.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.808+0000 7ff619481640 1 -- 172.21.15.67:0/453993216 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff61404f400 con 0x7ff600009270 2024-03-24T00:38:27.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.810+0000 7ff5fb7fe640 1 -- 172.21.15.67:0/453993216 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7ff600008a10 con 0x7ff600009270 2024-03-24T00:38:27.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.914+0000 7ff619481640 1 -- 172.21.15.67:0/453993216 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7ff6140c3cc0 con 0x7ff600009270 2024-03-24T00:38:27.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.915+0000 7ff5fb7fe640 1 -- 172.21.15.67:0/453993216 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7ff608035800 con 0x7ff600009270 2024-03-24T00:38:27.916 INFO:teuthology.orchestra.run.smithi067.stderr:adjusted tunables profile to default 2024-03-24T00:38:27.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.916+0000 7ff619481640 1 -- 172.21.15.67:0/453993216 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff600009270 msgr2=0x7ff614150e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:27.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.916+0000 7ff619481640 1 --2- 172.21.15.67:0/453993216 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff600009270 0x7ff614150e60 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff60802c520 tx=0x7ff608001120 comp rx=0 tx=0).stop 2024-03-24T00:38:27.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.916+0000 7ff619481640 1 -- 172.21.15.67:0/453993216 shutdown_connections 2024-03-24T00:38:27.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.916+0000 7ff619481640 1 --2- 172.21.15.67:0/453993216 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff61415e350 0x7ff6140c9f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:27.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.916+0000 7ff619481640 1 --2- 172.21.15.67:0/453993216 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff6140cbaa0 0x7ff6140b4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:27.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.916+0000 7ff619481640 1 --2- 172.21.15.67:0/453993216 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff600009270 0x7ff614150e60 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:27.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.916+0000 7ff619481640 1 -- 172.21.15.67:0/453993216 >> 172.21.15.67:0/453993216 conn(0x7ff614099a30 msgr2=0x7ff61406b040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:27.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.916+0000 7ff619481640 1 -- 172.21.15.67:0/453993216 shutdown_connections 2024-03-24T00:38:27.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:27.916+0000 7ff619481640 1 -- 172.21.15.67:0/453993216 wait complete. 2024-03-24T00:38:27.928 INFO:tasks.ceph:check_enable_crimson: False 2024-03-24T00:38:27.928 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-24T00:38:27.928 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:38:27.928 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-24T00:38:27.955 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:38:27.955 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-24T00:38:28.022 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:38:28.022 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-24T00:38:28.089 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:38:28.089 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-24T00:38:28.117 INFO:tasks.ceph.mgr.x.smithi072.stderr:2024-03-24T00:38:28.117+0000 7f19dd22b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T00:38:28.155 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:38:28.155 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-24T00:38:28.172 INFO:tasks.ceph.mgr.x.smithi072.stderr:2024-03-24T00:38:28.172+0000 7f19dd22b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T00:38:28.182 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:38:28.182 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-24T00:38:28.183 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-24T00:38:28.181+0000 7fa361c66040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T00:38:28.218 INFO:tasks.ceph.mgr.x.smithi072.stderr:2024-03-24T00:38:28.218+0000 7f19dd22b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T00:38:28.237 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-24T00:38:28.237+0000 7fa361c66040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T00:38:28.244 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:38:28.244 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-24T00:38:28.274 INFO:tasks.ceph.mgr.x.smithi072.stderr:2024-03-24T00:38:28.274+0000 7f19dd22b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T00:38:28.283 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-24T00:38:28.283+0000 7fa361c66040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T00:38:28.300 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:38:28.300 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-24T00:38:28.319 INFO:tasks.ceph.mgr.x.smithi072.stderr:2024-03-24T00:38:28.319+0000 7f19dd22b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T00:38:28.339 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-24T00:38:28.339+0000 7fa361c66040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T00:38:28.345 DEBUG:teuthology.orchestra.run.smithi072:> sudo ceph --cluster ceph osd new edd550ab-622a-4dfb-9caf-c261811d80a9 0 2024-03-24T00:38:28.383 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-24T00:38:28.382+0000 7fa361c66040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T00:38:28.440 INFO:tasks.ceph.mgr.x.smithi072.stderr:2024-03-24T00:38:28.440+0000 7f19dd22b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T00:38:28.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.484+0000 7f9d7c2ae640 1 Processor -- start 2024-03-24T00:38:28.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.484+0000 7f9d7c2ae640 1 -- start start 2024-03-24T00:38:28.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.485+0000 7f9d7c2ae640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f9d7415bee0 0x7f9d74099a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:28.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.485+0000 7f9d7c2ae640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d740af730 con 0x7f9d74160430 2024-03-24T00:38:28.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.485+0000 7f9d7c2ae640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d740b2880 con 0x7f9d740ccc50 2024-03-24T00:38:28.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.485+0000 7f9d7c2ae640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d740cf180 con 0x7f9d7415bee0 2024-03-24T00:38:28.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.485+0000 7f9d7a04a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f9d7415bee0 0x7f9d74099a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:28.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.485+0000 7f9d7a84b640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f9d740ccc50 0x7f9d740cf610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:37572/0 (socket says 172.21.15.72:37572) 2024-03-24T00:38:28.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.485+0000 7f9d7a84b640 1 -- 172.21.15.72:0/3218139102 learned_addr learned my addr 172.21.15.72:0/3218139102 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-24T00:38:28.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.486+0000 7f9d7a04a640 1 -- 172.21.15.72:0/3218139102 >> v1:172.21.15.72:6789/0 conn(0x7f9d740ccc50 legacy=0x7f9d740cf610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:28.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.486+0000 7f9d7a04a640 1 -- 172.21.15.72:0/3218139102 >> v1:172.21.15.67:6789/0 conn(0x7f9d74160430 legacy=0x7f9d7406b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:28.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.486+0000 7f9d7a04a640 1 -- 172.21.15.72:0/3218139102 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d740b2b60 con 0x7f9d7415bee0 2024-03-24T00:38:28.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.486+0000 7f9d7a04a640 1 --2- 172.21.15.72:0/3218139102 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d7415bee0 0x7f9d74099a30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9d700097a0 tx=0x7f9d7002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cff58ca643f7416f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:28.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.486+0000 7f9d79048640 1 -- 172.21.15.72:0/3218139102 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d70034070 con 0x7f9d7415bee0 2024-03-24T00:38:28.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.486+0000 7f9d79048640 1 -- 172.21.15.72:0/3218139102 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d70030610 con 0x7f9d7415bee0 2024-03-24T00:38:28.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.486+0000 7f9d79048640 1 -- 172.21.15.72:0/3218139102 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d700309e0 con 0x7f9d7415bee0 2024-03-24T00:38:28.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.487+0000 7f9d7c2ae640 1 -- 172.21.15.72:0/3218139102 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d7415bee0 msgr2=0x7f9d74099a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:28.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.487+0000 7f9d7c2ae640 1 --2- 172.21.15.72:0/3218139102 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d7415bee0 0x7f9d74099a30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9d700097a0 tx=0x7f9d7002b960 comp rx=0 tx=0).stop 2024-03-24T00:38:28.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.487+0000 7f9d7c2ae640 1 -- 172.21.15.72:0/3218139102 shutdown_connections 2024-03-24T00:38:28.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.487+0000 7f9d7c2ae640 1 --2- 172.21.15.72:0/3218139102 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d7415bee0 0x7f9d74099a30 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:28.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.487+0000 7f9d7c2ae640 1 -- 172.21.15.72:0/3218139102 >> 172.21.15.72:0/3218139102 conn(0x7f9d740b3c40 msgr2=0x7f9d740b1dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:28.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.487+0000 7f9d7c2ae640 1 -- 172.21.15.72:0/3218139102 shutdown_connections 2024-03-24T00:38:28.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.487+0000 7f9d7c2ae640 1 -- 172.21.15.72:0/3218139102 wait complete. 2024-03-24T00:38:28.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.488+0000 7f9d7c2ae640 1 Processor -- start 2024-03-24T00:38:28.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.488+0000 7f9d7c2ae640 1 -- start start 2024-03-24T00:38:28.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.489+0000 7f9d7c2ae640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d740ccc50 0x7f9d74155310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:28.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.489+0000 7f9d7c2ae640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d7415bee0 0x7f9d7415a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:28.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.489+0000 7f9d7c2ae640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9d74160430 0x7f9d7415ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:28.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.489+0000 7f9d7c2ae640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d740cfd20 con 0x7f9d740ccc50 2024-03-24T00:38:28.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.489+0000 7f9d7c2ae640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d740ceda0 con 0x7f9d7415bee0 2024-03-24T00:38:28.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.489+0000 7f9d7c2ae640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d740cfea0 con 0x7f9d74160430 2024-03-24T00:38:28.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.489+0000 7f9d7a84b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9d74160430 0x7f9d7415ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:28.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.489+0000 7f9d79849640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d7415bee0 0x7f9d7415a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:28.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.489+0000 7f9d7a84b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9d74160430 0x7f9d7415ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.72:45260/0 (socket says 172.21.15.72:45260) 2024-03-24T00:38:28.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.489+0000 7f9d7a84b640 1 -- 172.21.15.72:0/1601824469 learned_addr learned my addr 172.21.15.72:0/1601824469 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-24T00:38:28.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.489+0000 7f9d7a04a640 1 --2- 172.21.15.72:0/1601824469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d740ccc50 0x7f9d74155310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:28.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.489+0000 7f9d79849640 1 -- 172.21.15.72:0/1601824469 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9d74160430 msgr2=0x7f9d7415ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:28.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.489+0000 7f9d79849640 1 --2- 172.21.15.72:0/1601824469 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9d74160430 0x7f9d7415ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:28.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.489+0000 7f9d79849640 1 -- 172.21.15.72:0/1601824469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d740ccc50 msgr2=0x7f9d74155310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:28.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.489+0000 7f9d79849640 1 --2- 172.21.15.72:0/1601824469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d740ccc50 0x7f9d74155310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:28.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.489+0000 7f9d79849640 1 -- 172.21.15.72:0/1601824469 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d741f64d0 con 0x7f9d7415bee0 2024-03-24T00:38:28.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.489+0000 7f9d7a84b640 1 --2- 172.21.15.72:0/1601824469 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9d74160430 0x7f9d7415ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:28.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.490+0000 7f9d79849640 1 --2- 172.21.15.72:0/1601824469 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d7415bee0 0x7f9d7415a5f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9d6400c310 tx=0x7f9d6400c7d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:28.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.490+0000 7f9d6affd640 1 -- 172.21.15.72:0/1601824469 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d640052d0 con 0x7f9d7415bee0 2024-03-24T00:38:28.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.490+0000 7f9d6affd640 1 -- 172.21.15.72:0/1601824469 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d64005490 con 0x7f9d7415bee0 2024-03-24T00:38:28.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.490+0000 7f9d7c2ae640 1 -- 172.21.15.72:0/1601824469 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d741f61f0 con 0x7f9d7415bee0 2024-03-24T00:38:28.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.490+0000 7f9d6affd640 1 -- 172.21.15.72:0/1601824469 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d64007260 con 0x7f9d7415bee0 2024-03-24T00:38:28.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.490+0000 7f9d7c2ae640 1 -- 172.21.15.72:0/1601824469 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d741e6d90 con 0x7f9d7415bee0 2024-03-24T00:38:28.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.490+0000 7f9d6affd640 1 -- 172.21.15.72:0/1601824469 <== mon.1 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f9d64007400 con 0x7f9d7415bee0 2024-03-24T00:38:28.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.490+0000 7f9d6affd640 1 -- 172.21.15.72:0/1601824469 <== mon.1 v2:172.21.15.72:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f9d64003b20 con 0x7f9d7415bee0 2024-03-24T00:38:28.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.491+0000 7f9d7c2ae640 1 -- 172.21.15.72:0/1601824469 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d7404f400 con 0x7f9d7415bee0 2024-03-24T00:38:28.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.493+0000 7f9d6affd640 1 -- 172.21.15.72:0/1601824469 <== mon.1 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f9d640035f0 con 0x7f9d7415bee0 2024-03-24T00:38:28.503 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-24T00:38:28.502+0000 7fa361c66040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T00:38:28.513 INFO:tasks.ceph.mgr.x.smithi072.stderr:2024-03-24T00:38:28.512+0000 7f19dd22b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T00:38:28.560 INFO:tasks.ceph.mgr.x.smithi072.stderr:2024-03-24T00:38:28.560+0000 7f19dd22b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T00:38:28.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.566+0000 7f9d7c2ae640 1 -- 172.21.15.72:0/1601824469 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd new", "uuid": "edd550ab-622a-4dfb-9caf-c261811d80a9", "id": 0} v 0) v1 -- 0x7f9d7415bcc0 con 0x7f9d7415bee0 2024-03-24T00:38:28.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.570+0000 7f9d6affd640 1 -- 172.21.15.72:0/1601824469 <== mon.1 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "edd550ab-622a-4dfb-9caf-c261811d80a9", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9d64003dd0 con 0x7f9d7415bee0 2024-03-24T00:38:28.570 INFO:teuthology.orchestra.run.smithi072.stdout:0 2024-03-24T00:38:28.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.570+0000 7f9d7c2ae640 1 -- 172.21.15.72:0/1601824469 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d7415bee0 msgr2=0x7f9d7415a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:28.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.570+0000 7f9d7c2ae640 1 --2- 172.21.15.72:0/1601824469 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d7415bee0 0x7f9d7415a5f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9d6400c310 tx=0x7f9d6400c7d0 comp rx=0 tx=0).stop 2024-03-24T00:38:28.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.570+0000 7f9d7c2ae640 1 -- 172.21.15.72:0/1601824469 shutdown_connections 2024-03-24T00:38:28.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.570+0000 7f9d7c2ae640 1 --2- 172.21.15.72:0/1601824469 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9d74160430 0x7f9d7415ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:28.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.570+0000 7f9d7c2ae640 1 --2- 172.21.15.72:0/1601824469 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d7415bee0 0x7f9d7415a5f0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:28.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.570+0000 7f9d7c2ae640 1 --2- 172.21.15.72:0/1601824469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d740ccc50 0x7f9d74155310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:28.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.570+0000 7f9d7c2ae640 1 -- 172.21.15.72:0/1601824469 >> 172.21.15.72:0/1601824469 conn(0x7f9d740b3c40 msgr2=0x7f9d7406b040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:28.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.570+0000 7f9d7c2ae640 1 -- 172.21.15.72:0/1601824469 shutdown_connections 2024-03-24T00:38:28.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.571+0000 7f9d7c2ae640 1 -- 172.21.15.72:0/1601824469 wait complete. 2024-03-24T00:38:28.576 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-24T00:38:28.575+0000 7fa361c66040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T00:38:28.581 DEBUG:teuthology.orchestra.run.smithi072:> sudo ceph --cluster ceph osd new 5ec859de-6637-4591-872a-1694d640bc01 1 2024-03-24T00:38:28.617 INFO:tasks.ceph.mgr.x.smithi072.stderr:2024-03-24T00:38:28.617+0000 7f19dd22b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T00:38:28.623 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-24T00:38:28.623+0000 7fa361c66040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T00:38:28.666 INFO:tasks.ceph.mgr.x.smithi072.stderr:2024-03-24T00:38:28.666+0000 7f19dd22b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T00:38:28.678 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-24T00:38:28.677+0000 7fa361c66040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T00:38:28.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.681+0000 7f28ed57e640 1 Processor -- start 2024-03-24T00:38:28.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.681+0000 7f28ed57e640 1 -- start start 2024-03-24T00:38:28.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.682+0000 7f28ed57e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f28e80d0d50 0x7f28e80ce610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:28.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.682+0000 7f28ed57e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f28e80d22b0 0x7f28e80d2700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:28.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.682+0000 7f28ed57e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28e80cf570 con 0x7f28e815e130 2024-03-24T00:38:28.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.682+0000 7f28ed57e640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28e80ceb50 con 0x7f28e80d22b0 2024-03-24T00:38:28.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.682+0000 7f28ed57e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28e80cecf0 con 0x7f28e80d0d50 2024-03-24T00:38:28.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.682+0000 7f28e67fc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f28e80d0d50 0x7f28e80ce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:28.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.682+0000 7f28e6ffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f28e815e130 0x7f28e80d0640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:37582/0 (socket says 172.21.15.72:37582) 2024-03-24T00:38:28.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.682+0000 7f28e6ffd640 1 -- 172.21.15.72:0/3559758313 learned_addr learned my addr 172.21.15.72:0/3559758313 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-24T00:38:28.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.682+0000 7f28e77fe640 1 --2- 172.21.15.72:0/3559758313 >> v2:172.21.15.67:3300/0 conn(0x7f28e80d22b0 0x7f28e80d2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:28.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.682+0000 7f28e67fc640 1 -- 172.21.15.72:0/3559758313 >> v2:172.21.15.67:3300/0 conn(0x7f28e80d22b0 msgr2=0x7f28e80d2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:28.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.682+0000 7f28e67fc640 1 --2- 172.21.15.72:0/3559758313 >> v2:172.21.15.67:3300/0 conn(0x7f28e80d22b0 0x7f28e80d2700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:28.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.682+0000 7f28e67fc640 1 -- 172.21.15.72:0/3559758313 >> v1:172.21.15.72:6789/0 conn(0x7f28e815e130 legacy=0x7f28e80d0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:28.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.682+0000 7f28e67fc640 1 -- 172.21.15.72:0/3559758313 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28e8162a60 con 0x7f28e80d0d50 2024-03-24T00:38:28.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.682+0000 7f28e5ffb640 1 -- 172.21.15.72:0/3559758313 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1890667596 0 0) 0x7f28e80cf570 con 0x7f28e815e130 2024-03-24T00:38:28.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.683+0000 7f28e67fc640 1 --2- 172.21.15.72:0/3559758313 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f28e80d0d50 0x7f28e80ce610 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f28e8161bd0 tx=0x7f28d802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b94524f7eff297b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:28.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.683+0000 7f28e5ffb640 1 -- 172.21.15.72:0/3559758313 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d8035020 con 0x7f28e80d0d50 2024-03-24T00:38:28.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.683+0000 7f28e5ffb640 1 -- 172.21.15.72:0/3559758313 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28d8031830 con 0x7f28e80d0d50 2024-03-24T00:38:28.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.683+0000 7f28e5ffb640 1 -- 172.21.15.72:0/3559758313 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d8031c00 con 0x7f28e80d0d50 2024-03-24T00:38:28.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.684+0000 7f28ed57e640 1 -- 172.21.15.72:0/3559758313 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f28e80d0d50 msgr2=0x7f28e80ce610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:28.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.684+0000 7f28ed57e640 1 --2- 172.21.15.72:0/3559758313 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f28e80d0d50 0x7f28e80ce610 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f28e8161bd0 tx=0x7f28d802b8d0 comp rx=0 tx=0).stop 2024-03-24T00:38:28.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.684+0000 7f28ed57e640 1 -- 172.21.15.72:0/3559758313 shutdown_connections 2024-03-24T00:38:28.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.684+0000 7f28ed57e640 1 --2- 172.21.15.72:0/3559758313 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f28e80d0d50 0x7f28e80ce610 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:28.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.684+0000 7f28ed57e640 1 --2- 172.21.15.72:0/3559758313 >> v2:172.21.15.67:3300/0 conn(0x7f28e80d22b0 0x7f28e80d2700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:28.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.684+0000 7f28ed57e640 1 -- 172.21.15.72:0/3559758313 >> 172.21.15.72:0/3559758313 conn(0x7f28e806c8a0 msgr2=0x7f28e806ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:28.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.684+0000 7f28ed57e640 1 -- 172.21.15.72:0/3559758313 shutdown_connections 2024-03-24T00:38:28.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.684+0000 7f28ed57e640 1 -- 172.21.15.72:0/3559758313 wait complete. 2024-03-24T00:38:28.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.685+0000 7f28ed57e640 1 Processor -- start 2024-03-24T00:38:28.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.685+0000 7f28ed57e640 1 -- start start 2024-03-24T00:38:28.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.686+0000 7f28ed57e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f28e80d0d50 0x7f28e8160420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:28.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.686+0000 7f28ed57e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f28e80d22b0 0x7f28e81f5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:28.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.686+0000 7f28ed57e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28e815e130 0x7f28e815be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:28.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.686+0000 7f28ed57e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28e815e510 con 0x7f28e815e130 2024-03-24T00:38:28.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.686+0000 7f28ed57e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28e80cf290 con 0x7f28e80d22b0 2024-03-24T00:38:28.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.686+0000 7f28ed57e640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28e80b2e10 con 0x7f28e80d0d50 2024-03-24T00:38:28.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.686+0000 7f28e6ffd640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f28e80d0d50 0x7f28e8160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:28.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.686+0000 7f28e67fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f28e80d22b0 0x7f28e81f5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:28.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.686+0000 7f28e6ffd640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f28e80d0d50 0x7f28e8160420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.72:45262/0 (socket says 172.21.15.72:45262) 2024-03-24T00:38:28.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.686+0000 7f28e67fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f28e80d22b0 0x7f28e81f5080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43434/0 (socket says 172.21.15.72:43434) 2024-03-24T00:38:28.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.686+0000 7f28e6ffd640 1 -- 172.21.15.72:0/2030966213 learned_addr learned my addr 172.21.15.72:0/2030966213 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-24T00:38:28.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.686+0000 7f28e77fe640 1 --2- 172.21.15.72:0/2030966213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28e815e130 0x7f28e815be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:28.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.686+0000 7f28e6ffd640 1 -- 172.21.15.72:0/2030966213 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f28e80d22b0 msgr2=0x7f28e81f5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:28.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.686+0000 7f28e6ffd640 1 --2- 172.21.15.72:0/2030966213 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f28e80d22b0 0x7f28e81f5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:28.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.686+0000 7f28e6ffd640 1 -- 172.21.15.72:0/2030966213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28e815e130 msgr2=0x7f28e815be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:28.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.686+0000 7f28e6ffd640 1 --2- 172.21.15.72:0/2030966213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28e815e130 0x7f28e815be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:28.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.686+0000 7f28e6ffd640 1 -- 172.21.15.72:0/2030966213 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28e81f3c10 con 0x7f28e80d0d50 2024-03-24T00:38:28.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.686+0000 7f28e67fc640 1 --2- 172.21.15.72:0/2030966213 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f28e80d22b0 0x7f28e81f5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:28.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.686+0000 7f28e77fe640 1 --2- 172.21.15.72:0/2030966213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28e815e130 0x7f28e815be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:28.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.686+0000 7f28e6ffd640 1 --2- 172.21.15.72:0/2030966213 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f28e80d0d50 0x7f28e8160420 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f28dc00c4a0 tx=0x7f28dc00c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:28.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.687+0000 7f28cf7fe640 1 -- 172.21.15.72:0/2030966213 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28dc017070 con 0x7f28e80d0d50 2024-03-24T00:38:28.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.687+0000 7f28cf7fe640 1 -- 172.21.15.72:0/2030966213 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28dc005ce0 con 0x7f28e80d0d50 2024-03-24T00:38:28.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.687+0000 7f28ed57e640 1 -- 172.21.15.72:0/2030966213 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28e8160a20 con 0x7f28e80d0d50 2024-03-24T00:38:28.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.687+0000 7f28cf7fe640 1 -- 172.21.15.72:0/2030966213 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28dc0072b0 con 0x7f28e80d0d50 2024-03-24T00:38:28.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.687+0000 7f28ed57e640 1 -- 172.21.15.72:0/2030966213 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28e8160fe0 con 0x7f28e80d0d50 2024-03-24T00:38:28.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.687+0000 7f28cf7fe640 1 -- 172.21.15.72:0/2030966213 <== mon.2 v2:172.21.15.72:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f28dc01a070 con 0x7f28e80d0d50 2024-03-24T00:38:28.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.687+0000 7f28cf7fe640 1 -- 172.21.15.72:0/2030966213 <== mon.2 v2:172.21.15.72:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f28dc0127b0 con 0x7f28e80d0d50 2024-03-24T00:38:28.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.688+0000 7f28ed57e640 1 -- 172.21.15.72:0/2030966213 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28e804f400 con 0x7f28e80d0d50 2024-03-24T00:38:28.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.690+0000 7f28cf7fe640 1 -- 172.21.15.72:0/2030966213 <== mon.2 v2:172.21.15.72:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f28dc021070 con 0x7f28e80d0d50 2024-03-24T00:38:28.726 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-24T00:38:28.726+0000 7fa361c66040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T00:38:28.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.762+0000 7f28ed57e640 1 -- 172.21.15.72:0/2030966213 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "osd new", "uuid": "5ec859de-6637-4591-872a-1694d640bc01", "id": 1} v 0) v1 -- 0x7f28e80b8c70 con 0x7f28e80d0d50 2024-03-24T00:38:28.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.768+0000 7f28cf7fe640 1 -- 172.21.15.72:0/2030966213 <== mon.2 v2:172.21.15.72:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5ec859de-6637-4591-872a-1694d640bc01", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f28dc0076f0 con 0x7f28e80d0d50 2024-03-24T00:38:28.768 INFO:teuthology.orchestra.run.smithi072.stdout:1 2024-03-24T00:38:28.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.769+0000 7f28ed57e640 1 -- 172.21.15.72:0/2030966213 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f28e80d0d50 msgr2=0x7f28e8160420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:28.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.769+0000 7f28ed57e640 1 --2- 172.21.15.72:0/2030966213 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f28e80d0d50 0x7f28e8160420 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f28dc00c4a0 tx=0x7f28dc00c960 comp rx=0 tx=0).stop 2024-03-24T00:38:28.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.769+0000 7f28ed57e640 1 -- 172.21.15.72:0/2030966213 shutdown_connections 2024-03-24T00:38:28.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.769+0000 7f28ed57e640 1 --2- 172.21.15.72:0/2030966213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28e815e130 0x7f28e815be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:28.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.769+0000 7f28ed57e640 1 --2- 172.21.15.72:0/2030966213 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f28e80d22b0 0x7f28e81f5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:28.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.769+0000 7f28ed57e640 1 --2- 172.21.15.72:0/2030966213 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f28e80d0d50 0x7f28e8160420 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:28.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.769+0000 7f28ed57e640 1 -- 172.21.15.72:0/2030966213 >> 172.21.15.72:0/2030966213 conn(0x7f28e806c8a0 msgr2=0x7f28e80c7b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:28.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.769+0000 7f28ed57e640 1 -- 172.21.15.72:0/2030966213 shutdown_connections 2024-03-24T00:38:28.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.769+0000 7f28ed57e640 1 -- 172.21.15.72:0/2030966213 wait complete. 2024-03-24T00:38:28.779 DEBUG:teuthology.orchestra.run.smithi072:> sudo ceph --cluster ceph osd new 5984a001-8fe2-4555-a6f9-8dbfee8267e7 2 2024-03-24T00:38:28.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.920+0000 7f88a473b640 1 Processor -- start 2024-03-24T00:38:28.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.920+0000 7f88a473b640 1 -- start start 2024-03-24T00:38:28.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.921+0000 7f88a473b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f889c0cc1e0 0x7f889c06b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:28.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.921+0000 7f88a473b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f889c06b580 0x7f889c167c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:28.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.921+0000 7f88a473b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f889c0d0ae0 con 0x7f889c15e2f0 2024-03-24T00:38:28.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.921+0000 7f88a473b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f889c0d0dc0 con 0x7f889c0cc1e0 2024-03-24T00:38:28.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.921+0000 7f88a473b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f889c0d10a0 con 0x7f889c06b580 2024-03-24T00:38:28.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.921+0000 7f88a2cd8640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f889c06b580 0x7f889c167c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:28.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.921+0000 7f88a2cd8640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f889c06b580 0x7f889c167c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43448/0 (socket says 172.21.15.72:43448) 2024-03-24T00:38:28.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.921+0000 7f88a2cd8640 1 -- 172.21.15.72:0/166605587 learned_addr learned my addr 172.21.15.72:0/166605587 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-24T00:38:28.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.921+0000 7f88a1cd6640 1 --2- 172.21.15.72:0/166605587 >> v2:172.21.15.67:3300/0 conn(0x7f889c0cc1e0 0x7f889c06b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:28.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.922+0000 7f88a2cd8640 1 -- 172.21.15.72:0/166605587 >> v2:172.21.15.67:3300/0 conn(0x7f889c0cc1e0 msgr2=0x7f889c06b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:28.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.922+0000 7f88a2cd8640 1 --2- 172.21.15.72:0/166605587 >> v2:172.21.15.67:3300/0 conn(0x7f889c0cc1e0 0x7f889c06b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:28.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.922+0000 7f88a2cd8640 1 -- 172.21.15.72:0/166605587 >> v1:172.21.15.67:6789/0 conn(0x7f889c15e2f0 legacy=0x7f889c0cff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:28.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.922+0000 7f88a2cd8640 1 -- 172.21.15.72:0/166605587 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f889c0d1380 con 0x7f889c06b580 2024-03-24T00:38:28.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.922+0000 7f88a2cd8640 1 --2- 172.21.15.72:0/166605587 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f889c06b580 0x7f889c167c80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f889c0b1bf0 tx=0x7f889402bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6226ce9bbaf2f85f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:28.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.923+0000 7f88a14d5640 1 -- 172.21.15.72:0/166605587 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f889402ca80 con 0x7f889c06b580 2024-03-24T00:38:28.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.923+0000 7f88a14d5640 1 -- 172.21.15.72:0/166605587 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f889402cc20 con 0x7f889c06b580 2024-03-24T00:38:28.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.923+0000 7f88a14d5640 1 -- 172.21.15.72:0/166605587 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88940343f0 con 0x7f889c06b580 2024-03-24T00:38:28.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.923+0000 7f88a473b640 1 -- 172.21.15.72:0/166605587 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f889c06b580 msgr2=0x7f889c167c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:28.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.923+0000 7f88a473b640 1 --2- 172.21.15.72:0/166605587 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f889c06b580 0x7f889c167c80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f889c0b1bf0 tx=0x7f889402bee0 comp rx=0 tx=0).stop 2024-03-24T00:38:28.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.923+0000 7f88a473b640 1 -- 172.21.15.72:0/166605587 shutdown_connections 2024-03-24T00:38:28.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.923+0000 7f88a473b640 1 --2- 172.21.15.72:0/166605587 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f889c06b580 0x7f889c167c80 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:28.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.923+0000 7f88a473b640 1 --2- 172.21.15.72:0/166605587 >> v2:172.21.15.67:3300/0 conn(0x7f889c0cc1e0 0x7f889c06b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:28.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.923+0000 7f88a473b640 1 -- 172.21.15.72:0/166605587 >> 172.21.15.72:0/166605587 conn(0x7f889c099a30 msgr2=0x7f889c099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:28.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.923+0000 7f88a473b640 1 -- 172.21.15.72:0/166605587 shutdown_connections 2024-03-24T00:38:28.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.923+0000 7f88a473b640 1 -- 172.21.15.72:0/166605587 wait complete. 2024-03-24T00:38:28.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.925+0000 7f88a473b640 1 Processor -- start 2024-03-24T00:38:28.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.925+0000 7f88a473b640 1 -- start start 2024-03-24T00:38:28.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.925+0000 7f88a473b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f889c06b580 0x7f889c0b4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:28.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.925+0000 7f88a473b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f889c0cc1e0 0x7f889c15e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:28.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.925+0000 7f88a473b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f889c0b25e0 0x7f889c0b2a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:28.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.925+0000 7f88a473b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f889c0d0820 con 0x7f889c0b25e0 2024-03-24T00:38:28.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.925+0000 7f88a473b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f889c0d06a0 con 0x7f889c06b580 2024-03-24T00:38:28.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.925+0000 7f88a473b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f889c1681c0 con 0x7f889c0cc1e0 2024-03-24T00:38:28.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.925+0000 7f88a1cd6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f889c0cc1e0 0x7f889c15e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:28.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.925+0000 7f88a24d7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f889c06b580 0x7f889c0b4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:28.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.925+0000 7f88a24d7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f889c06b580 0x7f889c0b4cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43454/0 (socket says 172.21.15.72:43454) 2024-03-24T00:38:28.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.925+0000 7f88a1cd6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f889c0cc1e0 0x7f889c15e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.72:45278/0 (socket says 172.21.15.72:45278) 2024-03-24T00:38:28.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.925+0000 7f88a24d7640 1 -- 172.21.15.72:0/4285841441 learned_addr learned my addr 172.21.15.72:0/4285841441 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-24T00:38:28.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.925+0000 7f88a2cd8640 1 --2- 172.21.15.72:0/4285841441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f889c0b25e0 0x7f889c0b2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:28.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.926+0000 7f88a24d7640 1 -- 172.21.15.72:0/4285841441 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f889c0cc1e0 msgr2=0x7f889c15e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:28.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.926+0000 7f88a24d7640 1 --2- 172.21.15.72:0/4285841441 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f889c0cc1e0 0x7f889c15e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:28.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.926+0000 7f88a24d7640 1 -- 172.21.15.72:0/4285841441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f889c0b25e0 msgr2=0x7f889c0b2a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:28.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.926+0000 7f88a24d7640 1 --2- 172.21.15.72:0/4285841441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f889c0b25e0 0x7f889c0b2a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:28.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.926+0000 7f88a24d7640 1 -- 172.21.15.72:0/4285841441 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f889c0b3c30 con 0x7f889c06b580 2024-03-24T00:38:28.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.926+0000 7f88a1cd6640 1 --2- 172.21.15.72:0/4285841441 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f889c0cc1e0 0x7f889c15e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:28.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.926+0000 7f88a24d7640 1 --2- 172.21.15.72:0/4285841441 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f889c06b580 0x7f889c0b4cd0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f889800c430 tx=0x7f889800c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:28.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.926+0000 7f8892ffd640 1 -- 172.21.15.72:0/4285841441 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8898017070 con 0x7f889c06b580 2024-03-24T00:38:28.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.926+0000 7f8892ffd640 1 -- 172.21.15.72:0/4285841441 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88980052d0 con 0x7f889c06b580 2024-03-24T00:38:28.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.926+0000 7f8892ffd640 1 -- 172.21.15.72:0/4285841441 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8898003830 con 0x7f889c06b580 2024-03-24T00:38:28.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.926+0000 7f88a473b640 1 -- 172.21.15.72:0/4285841441 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f889c0b3f10 con 0x7f889c06b580 2024-03-24T00:38:28.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.926+0000 7f88a473b640 1 -- 172.21.15.72:0/4285841441 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f889c0b44d0 con 0x7f889c06b580 2024-03-24T00:38:28.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.926+0000 7f8892ffd640 1 -- 172.21.15.72:0/4285841441 <== mon.1 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f889801a070 con 0x7f889c06b580 2024-03-24T00:38:28.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.926+0000 7f8892ffd640 1 -- 172.21.15.72:0/4285841441 <== mon.1 v2:172.21.15.72:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f8898012420 con 0x7f889c06b580 2024-03-24T00:38:28.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.927+0000 7f88a473b640 1 -- 172.21.15.72:0/4285841441 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f889c04f400 con 0x7f889c06b580 2024-03-24T00:38:28.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:28.929+0000 7f8892ffd640 1 -- 172.21.15.72:0/4285841441 <== mon.1 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f8898021070 con 0x7f889c06b580 2024-03-24T00:38:29.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.002+0000 7f88a473b640 1 -- 172.21.15.72:0/4285841441 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd new", "uuid": "5984a001-8fe2-4555-a6f9-8dbfee8267e7", "id": 2} v 0) v1 -- 0x7f889c06b9f0 con 0x7f889c06b580 2024-03-24T00:38:29.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.008+0000 7f8892ffd640 1 -- 172.21.15.72:0/4285841441 <== mon.1 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5984a001-8fe2-4555-a6f9-8dbfee8267e7", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f88980115f0 con 0x7f889c06b580 2024-03-24T00:38:29.008 INFO:teuthology.orchestra.run.smithi072.stdout:2 2024-03-24T00:38:29.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.009+0000 7f88a473b640 1 -- 172.21.15.72:0/4285841441 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f889c06b580 msgr2=0x7f889c0b4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:29.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.009+0000 7f88a473b640 1 --2- 172.21.15.72:0/4285841441 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f889c06b580 0x7f889c0b4cd0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f889800c430 tx=0x7f889800c8f0 comp rx=0 tx=0).stop 2024-03-24T00:38:29.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.009+0000 7f88a473b640 1 -- 172.21.15.72:0/4285841441 shutdown_connections 2024-03-24T00:38:29.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.009+0000 7f88a473b640 1 --2- 172.21.15.72:0/4285841441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f889c0b25e0 0x7f889c0b2a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:29.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.009+0000 7f88a473b640 1 --2- 172.21.15.72:0/4285841441 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f889c0cc1e0 0x7f889c15e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:29.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.009+0000 7f88a473b640 1 --2- 172.21.15.72:0/4285841441 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f889c06b580 0x7f889c0b4cd0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:29.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.009+0000 7f88a473b640 1 -- 172.21.15.72:0/4285841441 >> 172.21.15.72:0/4285841441 conn(0x7f889c099a30 msgr2=0x7f889c099e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:29.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.009+0000 7f88a473b640 1 -- 172.21.15.72:0/4285841441 shutdown_connections 2024-03-24T00:38:29.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.009+0000 7f88a473b640 1 -- 172.21.15.72:0/4285841441 wait complete. 2024-03-24T00:38:29.019 DEBUG:teuthology.orchestra.run.smithi072:> sudo ceph --cluster ceph osd new 29a7b5b0-5027-4578-a0c8-06025a48ef5c 3 2024-03-24T00:38:29.069 INFO:tasks.ceph.mgr.x.smithi072.stderr:2024-03-24T00:38:29.069+0000 7f19dd22b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T00:38:29.118 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-24T00:38:29.118+0000 7fa361c66040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T00:38:29.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.120+0000 7f2886412640 1 Processor -- start 2024-03-24T00:38:29.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.120+0000 7f2886412640 1 -- start start 2024-03-24T00:38:29.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.121+0000 7f2886412640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2880165ff0 0x7f28800cdcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:29.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.121+0000 7f2886412640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f28800ce210 0x7f28800cf380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:29.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.121+0000 7f2886412640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28800cd4f0 con 0x7f28800c3ac0 2024-03-24T00:38:29.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.121+0000 7f2886412640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2880162950 con 0x7f28800ce210 2024-03-24T00:38:29.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.121+0000 7f2886412640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28800b0310 con 0x7f2880165ff0 2024-03-24T00:38:29.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.121+0000 7f287f7fe640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2880165ff0 0x7f28800cdcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:29.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.121+0000 7f28849af640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f28800ce210 0x7f28800cf380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:29.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.121+0000 7f28849af640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f28800ce210 0x7f28800cf380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43458/0 (socket says 172.21.15.72:43458) 2024-03-24T00:38:29.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.121+0000 7f287f7fe640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2880165ff0 0x7f28800cdcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.72:45294/0 (socket says 172.21.15.72:45294) 2024-03-24T00:38:29.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.121+0000 7f28849af640 1 -- 172.21.15.72:0/1757112673 learned_addr learned my addr 172.21.15.72:0/1757112673 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-24T00:38:29.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.121+0000 7f28849af640 1 -- 172.21.15.72:0/1757112673 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2880165ff0 msgr2=0x7f28800cdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:29.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.121+0000 7f28849af640 1 --2- 172.21.15.72:0/1757112673 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2880165ff0 0x7f28800cdcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:29.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.121+0000 7f28849af640 1 -- 172.21.15.72:0/1757112673 >> v1:172.21.15.67:6789/0 conn(0x7f28800c3ac0 legacy=0x7f28800c3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:29.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.121+0000 7f28849af640 1 -- 172.21.15.72:0/1757112673 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28801669b0 con 0x7f28800ce210 2024-03-24T00:38:29.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.122+0000 7f28849af640 1 --2- 172.21.15.72:0/1757112673 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f28800ce210 0x7f28800cf380 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2878009e50 tx=0x7f287802bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ba2ac81fb4d240c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:29.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.122+0000 7f287effd640 1 -- 172.21.15.72:0/1757112673 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287802cb00 con 0x7f28800ce210 2024-03-24T00:38:29.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.122+0000 7f287effd640 1 -- 172.21.15.72:0/1757112673 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f287802cca0 con 0x7f28800ce210 2024-03-24T00:38:29.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.122+0000 7f287effd640 1 -- 172.21.15.72:0/1757112673 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2878034610 con 0x7f28800ce210 2024-03-24T00:38:29.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.123+0000 7f2886412640 1 -- 172.21.15.72:0/1757112673 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f28800ce210 msgr2=0x7f28800cf380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:29.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.123+0000 7f2886412640 1 --2- 172.21.15.72:0/1757112673 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f28800ce210 0x7f28800cf380 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2878009e50 tx=0x7f287802bf60 comp rx=0 tx=0).stop 2024-03-24T00:38:29.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.123+0000 7f2886412640 1 -- 172.21.15.72:0/1757112673 shutdown_connections 2024-03-24T00:38:29.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.123+0000 7f2886412640 1 --2- 172.21.15.72:0/1757112673 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f28800ce210 0x7f28800cf380 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:29.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.123+0000 7f2886412640 1 --2- 172.21.15.72:0/1757112673 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2880165ff0 0x7f28800cdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:29.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.123+0000 7f2886412640 1 -- 172.21.15.72:0/1757112673 >> 172.21.15.72:0/1757112673 conn(0x7f2880099a30 msgr2=0x7f2880099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:29.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.123+0000 7f2886412640 1 -- 172.21.15.72:0/1757112673 shutdown_connections 2024-03-24T00:38:29.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.123+0000 7f2886412640 1 -- 172.21.15.72:0/1757112673 wait complete. 2024-03-24T00:38:29.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.124+0000 7f2886412640 1 Processor -- start 2024-03-24T00:38:29.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.124+0000 7f2886412640 1 -- start start 2024-03-24T00:38:29.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.125+0000 7f2886412640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f28800c3ac0 0x7f288015c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:29.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.125+0000 7f2886412640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f28800ce210 0x7f28800c4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:29.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.125+0000 7f2886412640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2880165ff0 0x7f28800c7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:29.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.125+0000 7f2886412640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28800b0e00 con 0x7f2880165ff0 2024-03-24T00:38:29.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.125+0000 7f2886412640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28800cd7b0 con 0x7f28800c3ac0 2024-03-24T00:38:29.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.125+0000 7f2886412640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28800b0f80 con 0x7f28800ce210 2024-03-24T00:38:29.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.125+0000 7f287ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f28800c3ac0 0x7f288015c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:29.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.125+0000 7f287f7fe640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f28800ce210 0x7f28800c4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:29.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.125+0000 7f287f7fe640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f28800ce210 0x7f28800c4740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.72:45310/0 (socket says 172.21.15.72:45310) 2024-03-24T00:38:29.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.125+0000 7f287ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f28800c3ac0 0x7f288015c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43470/0 (socket says 172.21.15.72:43470) 2024-03-24T00:38:29.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.125+0000 7f287f7fe640 1 -- 172.21.15.72:0/1941175798 learned_addr learned my addr 172.21.15.72:0/1941175798 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-24T00:38:29.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.125+0000 7f28849af640 1 --2- 172.21.15.72:0/1941175798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2880165ff0 0x7f28800c7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:29.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.125+0000 7f287f7fe640 1 -- 172.21.15.72:0/1941175798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f28800c3ac0 msgr2=0x7f288015c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:29.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.125+0000 7f287f7fe640 1 --2- 172.21.15.72:0/1941175798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f28800c3ac0 0x7f288015c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:29.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.125+0000 7f287f7fe640 1 -- 172.21.15.72:0/1941175798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2880165ff0 msgr2=0x7f28800c7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:29.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.125+0000 7f287f7fe640 1 --2- 172.21.15.72:0/1941175798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2880165ff0 0x7f28800c7bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:29.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.125+0000 7f287f7fe640 1 -- 172.21.15.72:0/1941175798 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28800c8bd0 con 0x7f28800ce210 2024-03-24T00:38:29.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.125+0000 7f287ffff640 1 --2- 172.21.15.72:0/1941175798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f28800c3ac0 0x7f288015c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:29.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.125+0000 7f287f7fe640 1 --2- 172.21.15.72:0/1941175798 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f28800ce210 0x7f28800c4740 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f287000e9b0 tx=0x7f287000ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:29.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.125+0000 7f287cff9640 1 -- 172.21.15.72:0/1941175798 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2870017070 con 0x7f28800ce210 2024-03-24T00:38:29.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.125+0000 7f287cff9640 1 -- 172.21.15.72:0/1941175798 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f287000c9a0 con 0x7f28800ce210 2024-03-24T00:38:29.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.125+0000 7f2886412640 1 -- 172.21.15.72:0/1941175798 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28800c8610 con 0x7f28800ce210 2024-03-24T00:38:29.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.125+0000 7f287cff9640 1 -- 172.21.15.72:0/1941175798 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287000cd90 con 0x7f28800ce210 2024-03-24T00:38:29.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.125+0000 7f2886412640 1 -- 172.21.15.72:0/1941175798 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28800c3f40 con 0x7f28800ce210 2024-03-24T00:38:29.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.126+0000 7f287cff9640 1 -- 172.21.15.72:0/1941175798 <== mon.2 v2:172.21.15.72:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f287001a070 con 0x7f28800ce210 2024-03-24T00:38:29.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.126+0000 7f287cff9640 1 -- 172.21.15.72:0/1941175798 <== mon.2 v2:172.21.15.72:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f2870006660 con 0x7f28800ce210 2024-03-24T00:38:29.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.126+0000 7f2886412640 1 -- 172.21.15.72:0/1941175798 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f288004f400 con 0x7f28800ce210 2024-03-24T00:38:29.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.128+0000 7f287cff9640 1 -- 172.21.15.72:0/1941175798 <== mon.2 v2:172.21.15.72:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f2870006020 con 0x7f28800ce210 2024-03-24T00:38:29.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.201+0000 7f2886412640 1 -- 172.21.15.72:0/1941175798 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "osd new", "uuid": "29a7b5b0-5027-4578-a0c8-06025a48ef5c", "id": 3} v 0) v1 -- 0x7f28800c4340 con 0x7f28800ce210 2024-03-24T00:38:29.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.207+0000 7f287cff9640 1 -- 172.21.15.72:0/1941175798 <== mon.2 v2:172.21.15.72:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "29a7b5b0-5027-4578-a0c8-06025a48ef5c", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f2870005990 con 0x7f28800ce210 2024-03-24T00:38:29.207 INFO:teuthology.orchestra.run.smithi072.stdout:3 2024-03-24T00:38:29.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.208+0000 7f2886412640 1 -- 172.21.15.72:0/1941175798 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f28800ce210 msgr2=0x7f28800c4740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:29.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.208+0000 7f2886412640 1 --2- 172.21.15.72:0/1941175798 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f28800ce210 0x7f28800c4740 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f287000e9b0 tx=0x7f287000ee70 comp rx=0 tx=0).stop 2024-03-24T00:38:29.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.208+0000 7f2886412640 1 -- 172.21.15.72:0/1941175798 shutdown_connections 2024-03-24T00:38:29.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.208+0000 7f2886412640 1 --2- 172.21.15.72:0/1941175798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2880165ff0 0x7f28800c7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:29.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.208+0000 7f2886412640 1 --2- 172.21.15.72:0/1941175798 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f28800ce210 0x7f28800c4740 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:29.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.208+0000 7f2886412640 1 --2- 172.21.15.72:0/1941175798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f28800c3ac0 0x7f288015c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:29.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.208+0000 7f2886412640 1 -- 172.21.15.72:0/1941175798 >> 172.21.15.72:0/1941175798 conn(0x7f2880099a30 msgr2=0x7f28800b0550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:29.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.208+0000 7f2886412640 1 -- 172.21.15.72:0/1941175798 shutdown_connections 2024-03-24T00:38:29.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.208+0000 7f2886412640 1 -- 172.21.15.72:0/1941175798 wait complete. 2024-03-24T00:38:29.209 INFO:tasks.ceph.mgr.x.smithi072.stderr:2024-03-24T00:38:29.209+0000 7f19dd22b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T00:38:29.218 DEBUG:teuthology.orchestra.run.smithi072:> sudo ceph --cluster ceph osd new 4d47499f-402f-43e1-a57c-25209f3c763b 4 2024-03-24T00:38:29.259 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-24T00:38:29.258+0000 7fa361c66040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T00:38:29.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.318+0000 7f1b5cc12640 1 Processor -- start 2024-03-24T00:38:29.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.318+0000 7f1b5cc12640 1 -- start start 2024-03-24T00:38:29.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.319+0000 7f1b5cc12640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1b5806b040 0x7f1b580afe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:29.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.319+0000 7f1b5cc12640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1b5815e480 0x7f1b580cd860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:29.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.319+0000 7f1b5cc12640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1b5806b420 0x7f1b58161d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:29.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.319+0000 7f1b5cc12640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b580ceaf0 con 0x7f1b5806b420 2024-03-24T00:38:29.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.319+0000 7f1b5cc12640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b580cdda0 con 0x7f1b5815e480 2024-03-24T00:38:29.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.319+0000 7f1b5cc12640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b580d0b30 con 0x7f1b5806b040 2024-03-24T00:38:29.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.319+0000 7f1b4ffff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1b5815e480 0x7f1b580cd860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:29.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.319+0000 7f1b5659c640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1b5806b040 0x7f1b580afe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:29.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.319+0000 7f1b4ffff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1b5815e480 0x7f1b580cd860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43484/0 (socket says 172.21.15.72:43484) 2024-03-24T00:38:29.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.319+0000 7f1b5659c640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1b5806b040 0x7f1b580afe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.72:45318/0 (socket says 172.21.15.72:45318) 2024-03-24T00:38:29.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.319+0000 7f1b4ffff640 1 -- 172.21.15.72:0/3631920187 learned_addr learned my addr 172.21.15.72:0/3631920187 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-24T00:38:29.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.319+0000 7f1b56d9d640 1 --2- 172.21.15.72:0/3631920187 >> v2:172.21.15.67:3300/0 conn(0x7f1b5806b420 0x7f1b58161d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:29.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.319+0000 7f1b4ffff640 1 -- 172.21.15.72:0/3631920187 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1b5806b040 msgr2=0x7f1b580afe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:29.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.319+0000 7f1b4ffff640 1 --2- 172.21.15.72:0/3631920187 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1b5806b040 0x7f1b580afe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:29.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.319+0000 7f1b4ffff640 1 -- 172.21.15.72:0/3631920187 >> v2:172.21.15.67:3300/0 conn(0x7f1b5806b420 msgr2=0x7f1b58161d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:29.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.319+0000 7f1b4ffff640 1 --2- 172.21.15.72:0/3631920187 >> v2:172.21.15.67:3300/0 conn(0x7f1b5806b420 0x7f1b58161d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:29.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.319+0000 7f1b4ffff640 1 -- 172.21.15.72:0/3631920187 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b580d0e10 con 0x7f1b5815e480 2024-03-24T00:38:29.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.319+0000 7f1b5659c640 1 --2- 172.21.15.72:0/3631920187 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1b5806b040 0x7f1b580afe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:29.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.319+0000 7f1b56d9d640 1 --2- 172.21.15.72:0/3631920187 >> v2:172.21.15.67:3300/0 conn(0x7f1b5806b420 0x7f1b58161d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:38:29.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.320+0000 7f1b4ffff640 1 --2- 172.21.15.72:0/3631920187 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b5815e480 0x7f1b580cd860 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1b580d1b30 tx=0x7f1b4002b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ebfc350c465019 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:29.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.320+0000 7f1b55d9b640 1 -- 172.21.15.72:0/3631920187 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b4002e070 con 0x7f1b5815e480 2024-03-24T00:38:29.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.320+0000 7f1b55d9b640 1 -- 172.21.15.72:0/3631920187 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b40037070 con 0x7f1b5815e480 2024-03-24T00:38:29.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.320+0000 7f1b55d9b640 1 -- 172.21.15.72:0/3631920187 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b40031a00 con 0x7f1b5815e480 2024-03-24T00:38:29.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.321+0000 7f1b5cc12640 1 -- 172.21.15.72:0/3631920187 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b5815e480 msgr2=0x7f1b580cd860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:29.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.321+0000 7f1b5cc12640 1 --2- 172.21.15.72:0/3631920187 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b5815e480 0x7f1b580cd860 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1b580d1b30 tx=0x7f1b4002b880 comp rx=0 tx=0).stop 2024-03-24T00:38:29.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.321+0000 7f1b5cc12640 1 -- 172.21.15.72:0/3631920187 shutdown_connections 2024-03-24T00:38:29.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.321+0000 7f1b5cc12640 1 --2- 172.21.15.72:0/3631920187 >> v2:172.21.15.67:3300/0 conn(0x7f1b5806b420 0x7f1b58161d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:29.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.321+0000 7f1b5cc12640 1 --2- 172.21.15.72:0/3631920187 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b5815e480 0x7f1b580cd860 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:29.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.321+0000 7f1b5cc12640 1 --2- 172.21.15.72:0/3631920187 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1b5806b040 0x7f1b580afe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:29.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.321+0000 7f1b5cc12640 1 -- 172.21.15.72:0/3631920187 >> 172.21.15.72:0/3631920187 conn(0x7f1b58099a30 msgr2=0x7f1b58099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:29.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.321+0000 7f1b5cc12640 1 -- 172.21.15.72:0/3631920187 shutdown_connections 2024-03-24T00:38:29.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.321+0000 7f1b5cc12640 1 -- 172.21.15.72:0/3631920187 wait complete. 2024-03-24T00:38:29.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.322+0000 7f1b5cc12640 1 Processor -- start 2024-03-24T00:38:29.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.322+0000 7f1b5cc12640 1 -- start start 2024-03-24T00:38:29.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.323+0000 7f1b5cc12640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1b5806b040 0x7f1b5814e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:29.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.323+0000 7f1b5cc12640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b5806b420 0x7f1b580c60c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:29.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.323+0000 7f1b5cc12640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b5815e480 0x7f1b58160c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:29.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.323+0000 7f1b5cc12640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b580d16b0 con 0x7f1b5815e480 2024-03-24T00:38:29.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.323+0000 7f1b5cc12640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b580d13d0 con 0x7f1b5806b420 2024-03-24T00:38:29.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.323+0000 7f1b5cc12640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5806b850 con 0x7f1b5806b040 2024-03-24T00:38:29.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.323+0000 7f1b5659c640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1b5806b040 0x7f1b5814e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:29.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.323+0000 7f1b4ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b5806b420 0x7f1b580c60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:29.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.323+0000 7f1b5659c640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1b5806b040 0x7f1b5814e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.72:45330/0 (socket says 172.21.15.72:45330) 2024-03-24T00:38:29.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.323+0000 7f1b5659c640 1 -- 172.21.15.72:0/1708415748 learned_addr learned my addr 172.21.15.72:0/1708415748 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-24T00:38:29.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.323+0000 7f1b56d9d640 1 --2- 172.21.15.72:0/1708415748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b5815e480 0x7f1b58160c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:29.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.323+0000 7f1b4ffff640 1 -- 172.21.15.72:0/1708415748 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1b5806b040 msgr2=0x7f1b5814e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:29.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.323+0000 7f1b4ffff640 1 --2- 172.21.15.72:0/1708415748 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1b5806b040 0x7f1b5814e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:29.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.323+0000 7f1b4ffff640 1 -- 172.21.15.72:0/1708415748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b5815e480 msgr2=0x7f1b58160c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:29.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.323+0000 7f1b4ffff640 1 --2- 172.21.15.72:0/1708415748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b5815e480 0x7f1b58160c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:29.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.324+0000 7f1b4ffff640 1 -- 172.21.15.72:0/1708415748 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b580c6bb0 con 0x7f1b5806b420 2024-03-24T00:38:29.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.324+0000 7f1b4ffff640 1 --2- 172.21.15.72:0/1708415748 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b5806b420 0x7f1b580c60c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f1b40033000 tx=0x7f1b400046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:29.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.324+0000 7f1b4effd640 1 -- 172.21.15.72:0/1708415748 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b40032670 con 0x7f1b5806b420 2024-03-24T00:38:29.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.324+0000 7f1b4effd640 1 -- 172.21.15.72:0/1708415748 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b40032850 con 0x7f1b5806b420 2024-03-24T00:38:29.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.324+0000 7f1b4effd640 1 -- 172.21.15.72:0/1708415748 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b40032c40 con 0x7f1b5806b420 2024-03-24T00:38:29.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.324+0000 7f1b5cc12640 1 -- 172.21.15.72:0/1708415748 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b580c6e90 con 0x7f1b5806b420 2024-03-24T00:38:29.329 INFO:tasks.ceph.mgr.x.smithi072.stderr:2024-03-24T00:38:29.324+0000 7f19dd22b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T00:38:29.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.324+0000 7f1b5cc12640 1 -- 172.21.15.72:0/1708415748 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b580c7450 con 0x7f1b5806b420 2024-03-24T00:38:29.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.324+0000 7f1b4effd640 1 -- 172.21.15.72:0/1708415748 <== mon.1 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f1b400453f0 con 0x7f1b5806b420 2024-03-24T00:38:29.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.324+0000 7f1b4effd640 1 -- 172.21.15.72:0/1708415748 <== mon.1 v2:172.21.15.72:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f1b4004e900 con 0x7f1b5806b420 2024-03-24T00:38:29.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.325+0000 7f1b5cc12640 1 -- 172.21.15.72:0/1708415748 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b5804f400 con 0x7f1b5806b420 2024-03-24T00:38:29.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.327+0000 7f1b4effd640 1 -- 172.21.15.72:0/1708415748 <== mon.1 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f1b40040070 con 0x7f1b5806b420 2024-03-24T00:38:29.371 INFO:tasks.ceph.mgr.x.smithi072.stderr:2024-03-24T00:38:29.371+0000 7f19dd22b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T00:38:29.383 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-24T00:38:29.383+0000 7fa361c66040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T00:38:29.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.399+0000 7f1b5cc12640 1 -- 172.21.15.72:0/1708415748 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd new", "uuid": "4d47499f-402f-43e1-a57c-25209f3c763b", "id": 4} v 0) v1 -- 0x7f1b5815a6b0 con 0x7f1b5806b420 2024-03-24T00:38:29.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.402+0000 7f1b4effd640 1 -- 172.21.15.72:0/1708415748 <== mon.1 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4d47499f-402f-43e1-a57c-25209f3c763b", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f1b4004eb90 con 0x7f1b5806b420 2024-03-24T00:38:29.403 INFO:teuthology.orchestra.run.smithi072.stdout:4 2024-03-24T00:38:29.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.403+0000 7f1b5cc12640 1 -- 172.21.15.72:0/1708415748 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b5806b420 msgr2=0x7f1b580c60c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:29.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.403+0000 7f1b5cc12640 1 --2- 172.21.15.72:0/1708415748 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b5806b420 0x7f1b580c60c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f1b40033000 tx=0x7f1b400046c0 comp rx=0 tx=0).stop 2024-03-24T00:38:29.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.403+0000 7f1b5cc12640 1 -- 172.21.15.72:0/1708415748 shutdown_connections 2024-03-24T00:38:29.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.403+0000 7f1b5cc12640 1 --2- 172.21.15.72:0/1708415748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b5815e480 0x7f1b58160c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:29.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.403+0000 7f1b5cc12640 1 --2- 172.21.15.72:0/1708415748 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b5806b420 0x7f1b580c60c0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:29.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.403+0000 7f1b5cc12640 1 --2- 172.21.15.72:0/1708415748 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1b5806b040 0x7f1b5814e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:29.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.403+0000 7f1b5cc12640 1 -- 172.21.15.72:0/1708415748 >> 172.21.15.72:0/1708415748 conn(0x7f1b58099a30 msgr2=0x7f1b58070100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:29.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.403+0000 7f1b5cc12640 1 -- 172.21.15.72:0/1708415748 shutdown_connections 2024-03-24T00:38:29.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.403+0000 7f1b5cc12640 1 -- 172.21.15.72:0/1708415748 wait complete. 2024-03-24T00:38:29.414 DEBUG:teuthology.orchestra.run.smithi072:> sudo ceph --cluster ceph osd new b946d742-1ace-4dd5-a08d-79767c863893 5 2024-03-24T00:38:29.431 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-24T00:38:29.431+0000 7fa361c66040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T00:38:29.490 INFO:tasks.ceph.mgr.x.smithi072.stderr:2024-03-24T00:38:29.490+0000 7f19dd22b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T00:38:29.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.513+0000 7f046b57e640 1 Processor -- start 2024-03-24T00:38:29.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.514+0000 7f046b57e640 1 -- start start 2024-03-24T00:38:29.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.514+0000 7f046b57e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f046415bff0 0x7f04640b1530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:29.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.514+0000 7f046b57e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04640d22a0 0x7f04640ccb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:29.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.514+0000 7f046b57e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04640b1fd0 con 0x7f04640b1a70 2024-03-24T00:38:29.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.514+0000 7f046b57e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04640b22b0 con 0x7f046415bff0 2024-03-24T00:38:29.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.514+0000 7f046b57e640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04640b2590 con 0x7f04640d22a0 2024-03-24T00:38:29.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.515+0000 7f0469b1b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04640d22a0 0x7f04640ccb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:29.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.515+0000 7f046931a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f046415bff0 0x7f04640b1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:29.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.515+0000 7f0469b1b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04640d22a0 0x7f04640ccb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.72:45344/0 (socket says 172.21.15.72:45344) 2024-03-24T00:38:29.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.515+0000 7f046931a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f046415bff0 0x7f04640b1530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43502/0 (socket says 172.21.15.72:43502) 2024-03-24T00:38:29.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.515+0000 7f0469b1b640 1 -- 172.21.15.72:0/2413166857 learned_addr learned my addr 172.21.15.72:0/2413166857 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-24T00:38:29.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.515+0000 7f046931a640 1 -- 172.21.15.72:0/2413166857 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04640d22a0 msgr2=0x7f04640ccb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:29.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.515+0000 7f046931a640 1 --2- 172.21.15.72:0/2413166857 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04640d22a0 0x7f04640ccb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:29.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.515+0000 7f046931a640 1 -- 172.21.15.72:0/2413166857 >> v1:172.21.15.72:6789/0 conn(0x7f04640b1a70 legacy=0x7f04640d1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:29.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.515+0000 7f046931a640 1 -- 172.21.15.72:0/2413166857 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04640b2870 con 0x7f046415bff0 2024-03-24T00:38:29.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.515+0000 7f0469b1b640 1 --2- 172.21.15.72:0/2413166857 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04640d22a0 0x7f04640ccb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:29.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.516+0000 7f046931a640 1 --2- 172.21.15.72:0/2413166857 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f046415bff0 0x7f04640b1530 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f04540097a0 tx=0x7f045402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c76587e197e508b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:29.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.516+0000 7f045bfff640 1 -- 172.21.15.72:0/2413166857 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0454035020 con 0x7f046415bff0 2024-03-24T00:38:29.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.516+0000 7f045bfff640 1 -- 172.21.15.72:0/2413166857 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0454031830 con 0x7f046415bff0 2024-03-24T00:38:29.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.516+0000 7f045bfff640 1 -- 172.21.15.72:0/2413166857 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0454031c00 con 0x7f046415bff0 2024-03-24T00:38:29.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.516+0000 7f046b57e640 1 -- 172.21.15.72:0/2413166857 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f046415bff0 msgr2=0x7f04640b1530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:29.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.516+0000 7f046b57e640 1 --2- 172.21.15.72:0/2413166857 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f046415bff0 0x7f04640b1530 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f04540097a0 tx=0x7f045402b8d0 comp rx=0 tx=0).stop 2024-03-24T00:38:29.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.516+0000 7f046b57e640 1 -- 172.21.15.72:0/2413166857 shutdown_connections 2024-03-24T00:38:29.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.516+0000 7f046b57e640 1 --2- 172.21.15.72:0/2413166857 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04640d22a0 0x7f04640ccb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:29.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.516+0000 7f046b57e640 1 --2- 172.21.15.72:0/2413166857 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f046415bff0 0x7f04640b1530 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:29.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.516+0000 7f046b57e640 1 -- 172.21.15.72:0/2413166857 >> 172.21.15.72:0/2413166857 conn(0x7f046408b1b0 msgr2=0x7f046408b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:29.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.516+0000 7f046b57e640 1 -- 172.21.15.72:0/2413166857 shutdown_connections 2024-03-24T00:38:29.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.517+0000 7f046b57e640 1 -- 172.21.15.72:0/2413166857 wait complete. 2024-03-24T00:38:29.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.518+0000 7f046b57e640 1 Processor -- start 2024-03-24T00:38:29.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.518+0000 7f046b57e640 1 -- start start 2024-03-24T00:38:29.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.518+0000 7f046b57e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04640b1a70 0x7f0464159f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:29.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.518+0000 7f046b57e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04640d22a0 0x7f046409b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:29.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.518+0000 7f046b57e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f046415bff0 0x7f046409d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:29.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.518+0000 7f046b57e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04640cd240 con 0x7f04640d22a0 2024-03-24T00:38:29.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.518+0000 7f046b57e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04640cd0c0 con 0x7f046415bff0 2024-03-24T00:38:29.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.518+0000 7f046b57e640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04640cd3c0 con 0x7f04640b1a70 2024-03-24T00:38:29.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.518+0000 7f046931a640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04640b1a70 0x7f0464159f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:29.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.518+0000 7f046931a640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04640b1a70 0x7f0464159f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.72:45350/0 (socket says 172.21.15.72:45350) 2024-03-24T00:38:29.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.518+0000 7f046931a640 1 -- 172.21.15.72:0/2013926182 learned_addr learned my addr 172.21.15.72:0/2013926182 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-24T00:38:29.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.518+0000 7f0469b1b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f046415bff0 0x7f046409d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:29.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.519+0000 7f0468b19640 1 --2- 172.21.15.72:0/2013926182 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04640d22a0 0x7f046409b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:29.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.519+0000 7f0469b1b640 1 -- 172.21.15.72:0/2013926182 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04640b1a70 msgr2=0x7f0464159f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:29.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.519+0000 7f0469b1b640 1 --2- 172.21.15.72:0/2013926182 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04640b1a70 0x7f0464159f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:29.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.519+0000 7f0469b1b640 1 -- 172.21.15.72:0/2013926182 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04640d22a0 msgr2=0x7f046409b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:29.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.519+0000 7f0469b1b640 1 --2- 172.21.15.72:0/2013926182 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04640d22a0 0x7f046409b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:29.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.519+0000 7f0469b1b640 1 -- 172.21.15.72:0/2013926182 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f046409e240 con 0x7f046415bff0 2024-03-24T00:38:29.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.519+0000 7f046931a640 1 --2- 172.21.15.72:0/2013926182 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04640b1a70 0x7f0464159f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:29.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.519+0000 7f0468b19640 1 --2- 172.21.15.72:0/2013926182 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04640d22a0 0x7f046409b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:38:29.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.519+0000 7f0469b1b640 1 --2- 172.21.15.72:0/2013926182 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f046415bff0 0x7f046409d8e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f045c00ef90 tx=0x7f045c00c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:29.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.519+0000 7f0459ffb640 1 -- 172.21.15.72:0/2013926182 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f045c00cd20 con 0x7f046415bff0 2024-03-24T00:38:29.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.519+0000 7f0459ffb640 1 -- 172.21.15.72:0/2013926182 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f045c019070 con 0x7f046415bff0 2024-03-24T00:38:29.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.519+0000 7f0459ffb640 1 -- 172.21.15.72:0/2013926182 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f045c0146c0 con 0x7f046415bff0 2024-03-24T00:38:29.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.519+0000 7f046b57e640 1 -- 172.21.15.72:0/2013926182 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f046409e520 con 0x7f046415bff0 2024-03-24T00:38:29.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.519+0000 7f0459ffb640 1 -- 172.21.15.72:0/2013926182 <== mon.1 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f045c005d10 con 0x7f046415bff0 2024-03-24T00:38:29.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.519+0000 7f046b57e640 1 -- 172.21.15.72:0/2013926182 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f046409eae0 con 0x7f046415bff0 2024-03-24T00:38:29.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.520+0000 7f0459ffb640 1 -- 172.21.15.72:0/2013926182 <== mon.1 v2:172.21.15.72:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f045c014dd0 con 0x7f046415bff0 2024-03-24T00:38:29.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.520+0000 7f046b57e640 1 -- 172.21.15.72:0/2013926182 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f046415a790 con 0x7f046415bff0 2024-03-24T00:38:29.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.524+0000 7f0459ffb640 1 -- 172.21.15.72:0/2013926182 <== mon.1 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f045c022020 con 0x7f046415bff0 2024-03-24T00:38:29.548 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-24T00:38:29.547+0000 7fa361c66040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T00:38:29.592 INFO:tasks.ceph.mgr.x.smithi072.stderr:2024-03-24T00:38:29.591+0000 7f19dd22b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T00:38:29.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.596+0000 7f046b57e640 1 -- 172.21.15.72:0/2013926182 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b946d742-1ace-4dd5-a08d-79767c863893", "id": 5} v 0) v1 -- 0x7f046415b6f0 con 0x7f046415bff0 2024-03-24T00:38:29.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.600+0000 7f0459ffb640 1 -- 172.21.15.72:0/2013926182 <== mon.1 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b946d742-1ace-4dd5-a08d-79767c863893", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f045c010040 con 0x7f046415bff0 2024-03-24T00:38:29.601 INFO:teuthology.orchestra.run.smithi072.stdout:5 2024-03-24T00:38:29.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.601+0000 7f046b57e640 1 -- 172.21.15.72:0/2013926182 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f046415bff0 msgr2=0x7f046409d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:29.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.601+0000 7f046b57e640 1 --2- 172.21.15.72:0/2013926182 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f046415bff0 0x7f046409d8e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f045c00ef90 tx=0x7f045c00c490 comp rx=0 tx=0).stop 2024-03-24T00:38:29.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.601+0000 7f046b57e640 1 -- 172.21.15.72:0/2013926182 shutdown_connections 2024-03-24T00:38:29.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.601+0000 7f046b57e640 1 --2- 172.21.15.72:0/2013926182 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f046415bff0 0x7f046409d8e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:29.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.601+0000 7f046b57e640 1 --2- 172.21.15.72:0/2013926182 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04640d22a0 0x7f046409b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:29.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.601+0000 7f046b57e640 1 --2- 172.21.15.72:0/2013926182 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04640b1a70 0x7f0464159f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:29.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.601+0000 7f046b57e640 1 -- 172.21.15.72:0/2013926182 >> 172.21.15.72:0/2013926182 conn(0x7f046408b1b0 msgr2=0x7f04640af0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:29.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.601+0000 7f046b57e640 1 -- 172.21.15.72:0/2013926182 shutdown_connections 2024-03-24T00:38:29.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.601+0000 7f046b57e640 1 -- 172.21.15.72:0/2013926182 wait complete. 2024-03-24T00:38:29.612 DEBUG:teuthology.orchestra.run.smithi072:> sudo ceph --cluster ceph osd new 93a33f2e-4084-4745-9938-3bb16b67ba37 6 2024-03-24T00:38:29.646 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-24T00:38:29.646+0000 7fa361c66040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T00:38:29.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.713+0000 7f1490834640 1 Processor -- start 2024-03-24T00:38:29.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.713+0000 7f1490834640 1 -- start start 2024-03-24T00:38:29.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.713+0000 7f1490834640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f148c15e2f0 0x7f148c0cff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:29.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.713+0000 7f1490834640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f148c0d04d0 0x7f148c06b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:29.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.713+0000 7f1490834640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f148c0d0ae0 con 0x7f148c06b580 2024-03-24T00:38:29.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.713+0000 7f1490834640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f148c0d0dc0 con 0x7f148c0d04d0 2024-03-24T00:38:29.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.713+0000 7f1490834640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f148c0d10a0 con 0x7f148c15e2f0 2024-03-24T00:38:29.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.714+0000 7f148a59c640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f148c15e2f0 0x7f148c0cff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:29.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.714+0000 7f1489d9b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f148c0d04d0 0x7f148c06b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:29.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.714+0000 7f148ad9d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f148c06b580 0x7f148c0cb1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:37592/0 (socket says 172.21.15.72:37592) 2024-03-24T00:38:29.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.714+0000 7f148a59c640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f148c15e2f0 0x7f148c0cff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.72:45364/0 (socket says 172.21.15.72:45364) 2024-03-24T00:38:29.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.714+0000 7f148ad9d640 1 -- 172.21.15.72:0/3710380880 learned_addr learned my addr 172.21.15.72:0/3710380880 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-24T00:38:29.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.714+0000 7f1489d9b640 1 -- 172.21.15.72:0/3710380880 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f148c15e2f0 msgr2=0x7f148c0cff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:29.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.714+0000 7f1489d9b640 1 --2- 172.21.15.72:0/3710380880 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f148c15e2f0 0x7f148c0cff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:29.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.714+0000 7f1489d9b640 1 -- 172.21.15.72:0/3710380880 >> v1:172.21.15.72:6789/0 conn(0x7f148c06b580 legacy=0x7f148c0cb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:29.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.714+0000 7f1489d9b640 1 -- 172.21.15.72:0/3710380880 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f148c0d1380 con 0x7f148c0d04d0 2024-03-24T00:38:29.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.714+0000 7f148959a640 1 -- 172.21.15.72:0/3710380880 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2461255406 0 0) 0x7f148c0d0ae0 con 0x7f148c06b580 2024-03-24T00:38:29.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.714+0000 7f148a59c640 1 --2- 172.21.15.72:0/3710380880 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f148c15e2f0 0x7f148c0cff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:29.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.715+0000 7f1489d9b640 1 --2- 172.21.15.72:0/3710380880 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f148c0d04d0 0x7f148c06b040 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f14800097a0 tx=0x7f148002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63809b0512820cc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:29.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.715+0000 7f148959a640 1 -- 172.21.15.72:0/3710380880 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14800315a0 con 0x7f148c0d04d0 2024-03-24T00:38:29.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.715+0000 7f148959a640 1 -- 172.21.15.72:0/3710380880 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1480037070 con 0x7f148c0d04d0 2024-03-24T00:38:29.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.715+0000 7f148959a640 1 -- 172.21.15.72:0/3710380880 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1480031960 con 0x7f148c0d04d0 2024-03-24T00:38:29.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.715+0000 7f1490834640 1 -- 172.21.15.72:0/3710380880 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f148c0d04d0 msgr2=0x7f148c06b040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:29.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.715+0000 7f1490834640 1 --2- 172.21.15.72:0/3710380880 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f148c0d04d0 0x7f148c06b040 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f14800097a0 tx=0x7f148002b840 comp rx=0 tx=0).stop 2024-03-24T00:38:29.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.715+0000 7f1490834640 1 -- 172.21.15.72:0/3710380880 shutdown_connections 2024-03-24T00:38:29.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.715+0000 7f1490834640 1 --2- 172.21.15.72:0/3710380880 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f148c0d04d0 0x7f148c06b040 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:29.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.715+0000 7f1490834640 1 --2- 172.21.15.72:0/3710380880 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f148c15e2f0 0x7f148c0cff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:29.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.715+0000 7f1490834640 1 -- 172.21.15.72:0/3710380880 >> 172.21.15.72:0/3710380880 conn(0x7f148c099a30 msgr2=0x7f148c099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:29.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.716+0000 7f1490834640 1 -- 172.21.15.72:0/3710380880 shutdown_connections 2024-03-24T00:38:29.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.716+0000 7f1490834640 1 -- 172.21.15.72:0/3710380880 wait complete. 2024-03-24T00:38:29.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.717+0000 7f1490834640 1 Processor -- start 2024-03-24T00:38:29.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.717+0000 7f1490834640 1 -- start start 2024-03-24T00:38:29.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.717+0000 7f1490834640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f148c06b580 0x7f148c157440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:29.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.718+0000 7f1490834640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f148c0d04d0 0x7f148c0c85d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:29.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.718+0000 7f1490834640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f148c0b6400 0x7f148c0c9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:29.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.718+0000 7f1490834640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f148c0cba60 con 0x7f148c06b580 2024-03-24T00:38:29.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.718+0000 7f1490834640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f148c0cb8e0 con 0x7f148c0b6400 2024-03-24T00:38:29.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.718+0000 7f1490834640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f148c0cbbe0 con 0x7f148c0d04d0 2024-03-24T00:38:29.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.718+0000 7f1489d9b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f148c0d04d0 0x7f148c0c85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:29.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.718+0000 7f1489d9b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f148c0d04d0 0x7f148c0c85d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.72:45366/0 (socket says 172.21.15.72:45366) 2024-03-24T00:38:29.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.718+0000 7f1489d9b640 1 -- 172.21.15.72:0/2562001246 learned_addr learned my addr 172.21.15.72:0/2562001246 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-24T00:38:29.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.718+0000 7f148ad9d640 1 --2- 172.21.15.72:0/2562001246 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f148c0b6400 0x7f148c0c9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:29.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.718+0000 7f148a59c640 1 --2- 172.21.15.72:0/2562001246 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f148c06b580 0x7f148c157440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:29.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.718+0000 7f1489d9b640 1 -- 172.21.15.72:0/2562001246 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f148c0b6400 msgr2=0x7f148c0c9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:29.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.718+0000 7f1489d9b640 1 --2- 172.21.15.72:0/2562001246 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f148c0b6400 0x7f148c0c9aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:29.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.718+0000 7f1489d9b640 1 -- 172.21.15.72:0/2562001246 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f148c06b580 msgr2=0x7f148c157440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:29.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.718+0000 7f1489d9b640 1 --2- 172.21.15.72:0/2562001246 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f148c06b580 0x7f148c157440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:29.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.718+0000 7f1489d9b640 1 -- 172.21.15.72:0/2562001246 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f148c157980 con 0x7f148c0d04d0 2024-03-24T00:38:29.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.718+0000 7f148ad9d640 1 --2- 172.21.15.72:0/2562001246 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f148c0b6400 0x7f148c0c9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:29.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.718+0000 7f148a59c640 1 --2- 172.21.15.72:0/2562001246 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f148c06b580 0x7f148c157440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:38:29.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.718+0000 7f1489d9b640 1 --2- 172.21.15.72:0/2562001246 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f148c0d04d0 0x7f148c0c85d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1480034040 tx=0x7f14800046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:29.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.718+0000 7f147affd640 1 -- 172.21.15.72:0/2562001246 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1480032630 con 0x7f148c0d04d0 2024-03-24T00:38:29.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.718+0000 7f147affd640 1 -- 172.21.15.72:0/2562001246 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14800327f0 con 0x7f148c0d04d0 2024-03-24T00:38:29.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.718+0000 7f147affd640 1 -- 172.21.15.72:0/2562001246 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1480032be0 con 0x7f148c0d04d0 2024-03-24T00:38:29.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.718+0000 7f1490834640 1 -- 172.21.15.72:0/2562001246 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f148c0c8f20 con 0x7f148c0d04d0 2024-03-24T00:38:29.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.718+0000 7f1490834640 1 -- 172.21.15.72:0/2562001246 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f148c0b3c10 con 0x7f148c0d04d0 2024-03-24T00:38:29.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.719+0000 7f147affd640 1 -- 172.21.15.72:0/2562001246 <== mon.2 v2:172.21.15.72:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f1480006020 con 0x7f148c0d04d0 2024-03-24T00:38:29.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.719+0000 7f147affd640 1 -- 172.21.15.72:0/2562001246 <== mon.2 v2:172.21.15.72:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f148002f030 con 0x7f148c0d04d0 2024-03-24T00:38:29.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.719+0000 7f1490834640 1 -- 172.21.15.72:0/2562001246 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f148c0b3740 con 0x7f148c0d04d0 2024-03-24T00:38:29.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.721+0000 7f147affd640 1 -- 172.21.15.72:0/2562001246 <== mon.2 v2:172.21.15.72:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f1480040070 con 0x7f148c0d04d0 2024-03-24T00:38:29.757 INFO:tasks.ceph.mgr.x.smithi072.stderr:2024-03-24T00:38:29.757+0000 7f19dd22b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T00:38:29.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.794+0000 7f1490834640 1 -- 172.21.15.72:0/2562001246 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "osd new", "uuid": "93a33f2e-4084-4745-9938-3bb16b67ba37", "id": 6} v 0) v1 -- 0x7f148c04f400 con 0x7f148c0d04d0 2024-03-24T00:38:29.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.797+0000 7f147affd640 1 -- 172.21.15.72:0/2562001246 <== mon.2 v2:172.21.15.72:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "93a33f2e-4084-4745-9938-3bb16b67ba37", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f14800064f0 con 0x7f148c0d04d0 2024-03-24T00:38:29.798 INFO:teuthology.orchestra.run.smithi072.stdout:6 2024-03-24T00:38:29.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.798+0000 7f1490834640 1 -- 172.21.15.72:0/2562001246 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f148c0d04d0 msgr2=0x7f148c0c85d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:29.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.798+0000 7f1490834640 1 --2- 172.21.15.72:0/2562001246 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f148c0d04d0 0x7f148c0c85d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1480034040 tx=0x7f14800046c0 comp rx=0 tx=0).stop 2024-03-24T00:38:29.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.798+0000 7f1490834640 1 -- 172.21.15.72:0/2562001246 shutdown_connections 2024-03-24T00:38:29.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.798+0000 7f1490834640 1 --2- 172.21.15.72:0/2562001246 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f148c0b6400 0x7f148c0c9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:29.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.798+0000 7f1490834640 1 --2- 172.21.15.72:0/2562001246 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f148c0d04d0 0x7f148c0c85d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:29.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.798+0000 7f1490834640 1 --2- 172.21.15.72:0/2562001246 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f148c06b580 0x7f148c157440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:29.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.798+0000 7f1490834640 1 -- 172.21.15.72:0/2562001246 >> 172.21.15.72:0/2562001246 conn(0x7f148c099a30 msgr2=0x7f148c099e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:29.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.798+0000 7f1490834640 1 -- 172.21.15.72:0/2562001246 shutdown_connections 2024-03-24T00:38:29.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.798+0000 7f1490834640 1 -- 172.21.15.72:0/2562001246 wait complete. 2024-03-24T00:38:29.801 INFO:tasks.ceph.mgr.x.smithi072.stderr:2024-03-24T00:38:29.801+0000 7f19dd22b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T00:38:29.807 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-24T00:38:29.806+0000 7fa361c66040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T00:38:29.809 DEBUG:teuthology.orchestra.run.smithi072:> sudo ceph --cluster ceph osd new 7f138bad-98b0-4632-b528-e517933b1d47 7 2024-03-24T00:38:29.851 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-24T00:38:29.850+0000 7fa361c66040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T00:38:29.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.908+0000 7fe27a4d1640 1 Processor -- start 2024-03-24T00:38:29.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.908+0000 7fe27a4d1640 1 -- start start 2024-03-24T00:38:29.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.909+0000 7fe27a4d1640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe2740d1030 0x7fe2740cb940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:29.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.909+0000 7fe27a4d1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe2740cbe80 0x7fe2740c4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:29.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.909+0000 7fe27a4d1640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2740cc700 con 0x7fe27415afc0 2024-03-24T00:38:29.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.909+0000 7fe27a4d1640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe27415b3a0 con 0x7fe2740d1030 2024-03-24T00:38:29.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.909+0000 7fe27a4d1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2740cc9e0 con 0x7fe2740cbe80 2024-03-24T00:38:29.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.909+0000 7fe273fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fe27415afc0 0x7fe2740d0920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:37606/0 (socket says 172.21.15.72:37606) 2024-03-24T00:38:29.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.909+0000 7fe273fff640 1 -- 172.21.15.72:0/1860060942 learned_addr learned my addr 172.21.15.72:0/1860060942 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-24T00:38:29.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.909+0000 7fe278a6e640 1 --2- 172.21.15.72:0/1860060942 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe2740cbe80 0x7fe2740c4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:29.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.910+0000 7fe2737fe640 1 --2- 172.21.15.72:0/1860060942 >> v2:172.21.15.67:3300/0 conn(0x7fe2740d1030 0x7fe2740cb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:29.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.910+0000 7fe272ffd640 1 -- 172.21.15.72:0/1860060942 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3330813324 0 0) 0x7fe2740cc700 con 0x7fe27415afc0 2024-03-24T00:38:29.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.910+0000 7fe278a6e640 1 -- 172.21.15.72:0/1860060942 >> v2:172.21.15.67:3300/0 conn(0x7fe2740d1030 msgr2=0x7fe2740cb940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:29.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.910+0000 7fe278a6e640 1 --2- 172.21.15.72:0/1860060942 >> v2:172.21.15.67:3300/0 conn(0x7fe2740d1030 0x7fe2740cb940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:29.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.910+0000 7fe278a6e640 1 -- 172.21.15.72:0/1860060942 >> v1:172.21.15.72:6789/0 conn(0x7fe27415afc0 legacy=0x7fe2740d0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:29.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.910+0000 7fe278a6e640 1 -- 172.21.15.72:0/1860060942 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2740cccc0 con 0x7fe2740cbe80 2024-03-24T00:38:29.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.910+0000 7fe278a6e640 1 --2- 172.21.15.72:0/1860060942 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe2740cbe80 0x7fe2740c4a20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe26c009d90 tx=0x7fe26c02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cfd249e9ae9f3fa8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:29.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.911+0000 7fe272ffd640 1 -- 172.21.15.72:0/1860060942 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe26c039070 con 0x7fe2740cbe80 2024-03-24T00:38:29.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.911+0000 7fe272ffd640 1 -- 172.21.15.72:0/1860060942 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe26c02cc50 con 0x7fe2740cbe80 2024-03-24T00:38:29.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.911+0000 7fe272ffd640 1 -- 172.21.15.72:0/1860060942 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe26c034440 con 0x7fe2740cbe80 2024-03-24T00:38:29.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.911+0000 7fe27a4d1640 1 -- 172.21.15.72:0/1860060942 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe2740cbe80 msgr2=0x7fe2740c4a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:29.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.911+0000 7fe27a4d1640 1 --2- 172.21.15.72:0/1860060942 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe2740cbe80 0x7fe2740c4a20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe26c009d90 tx=0x7fe26c02bea0 comp rx=0 tx=0).stop 2024-03-24T00:38:29.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.911+0000 7fe27a4d1640 1 -- 172.21.15.72:0/1860060942 shutdown_connections 2024-03-24T00:38:29.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.911+0000 7fe27a4d1640 1 --2- 172.21.15.72:0/1860060942 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe2740cbe80 0x7fe2740c4a20 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:29.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.911+0000 7fe27a4d1640 1 --2- 172.21.15.72:0/1860060942 >> v2:172.21.15.67:3300/0 conn(0x7fe2740d1030 0x7fe2740cb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:29.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.911+0000 7fe27a4d1640 1 -- 172.21.15.72:0/1860060942 >> 172.21.15.72:0/1860060942 conn(0x7fe274099a30 msgr2=0x7fe274099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:29.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.911+0000 7fe27a4d1640 1 -- 172.21.15.72:0/1860060942 shutdown_connections 2024-03-24T00:38:29.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.911+0000 7fe27a4d1640 1 -- 172.21.15.72:0/1860060942 wait complete. 2024-03-24T00:38:29.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.912+0000 7fe27a4d1640 1 Processor -- start 2024-03-24T00:38:29.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.913+0000 7fe27a4d1640 1 -- start start 2024-03-24T00:38:29.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.913+0000 7fe27a4d1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2740cbe80 0x7fe274158100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:29.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.913+0000 7fe27a4d1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2740d1030 0x7fe2741f4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:29.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.913+0000 7fe27a4d1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe27415afc0 0x7fe2741f6f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:29.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.913+0000 7fe27a4d1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2740cc490 con 0x7fe2740d1030 2024-03-24T00:38:29.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.913+0000 7fe27a4d1640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2740cc310 con 0x7fe2740cbe80 2024-03-24T00:38:29.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.913+0000 7fe27a4d1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2740c4f60 con 0x7fe27415afc0 2024-03-24T00:38:29.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.913+0000 7fe273fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2740cbe80 0x7fe274158100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:29.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.913+0000 7fe278a6e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe27415afc0 0x7fe2741f6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:29.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.913+0000 7fe278a6e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe27415afc0 0x7fe2741f6f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.72:45378/0 (socket says 172.21.15.72:45378) 2024-03-24T00:38:29.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.913+0000 7fe273fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2740cbe80 0x7fe274158100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43544/0 (socket says 172.21.15.72:43544) 2024-03-24T00:38:29.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.913+0000 7fe278a6e640 1 -- 172.21.15.72:0/3212868686 learned_addr learned my addr 172.21.15.72:0/3212868686 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-24T00:38:29.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.913+0000 7fe2737fe640 1 --2- 172.21.15.72:0/3212868686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2740d1030 0x7fe2741f4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:29.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.914+0000 7fe278a6e640 1 -- 172.21.15.72:0/3212868686 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2740cbe80 msgr2=0x7fe274158100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:29.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.914+0000 7fe278a6e640 1 --2- 172.21.15.72:0/3212868686 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2740cbe80 0x7fe274158100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:29.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.914+0000 7fe278a6e640 1 -- 172.21.15.72:0/3212868686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2740d1030 msgr2=0x7fe2741f4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:29.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.914+0000 7fe278a6e640 1 --2- 172.21.15.72:0/3212868686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2740d1030 0x7fe2741f4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:29.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.914+0000 7fe278a6e640 1 -- 172.21.15.72:0/3212868686 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2741f78c0 con 0x7fe27415afc0 2024-03-24T00:38:29.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.914+0000 7fe278a6e640 1 --2- 172.21.15.72:0/3212868686 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe27415afc0 0x7fe2741f6f50 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe26c009eb0 tx=0x7fe26c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:29.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.914+0000 7fe270ff9640 1 -- 172.21.15.72:0/3212868686 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe26c039050 con 0x7fe27415afc0 2024-03-24T00:38:29.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.914+0000 7fe270ff9640 1 -- 172.21.15.72:0/3212868686 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe26c045070 con 0x7fe27415afc0 2024-03-24T00:38:29.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.914+0000 7fe27a4d1640 1 -- 172.21.15.72:0/3212868686 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2741f7ba0 con 0x7fe27415afc0 2024-03-24T00:38:29.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.914+0000 7fe270ff9640 1 -- 172.21.15.72:0/3212868686 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe26c009450 con 0x7fe27415afc0 2024-03-24T00:38:29.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.914+0000 7fe27a4d1640 1 -- 172.21.15.72:0/3212868686 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2741f48b0 con 0x7fe27415afc0 2024-03-24T00:38:29.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.914+0000 7fe270ff9640 1 -- 172.21.15.72:0/3212868686 <== mon.2 v2:172.21.15.72:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fe26c03e610 con 0x7fe27415afc0 2024-03-24T00:38:29.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.914+0000 7fe270ff9640 1 -- 172.21.15.72:0/3212868686 <== mon.2 v2:172.21.15.72:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7fe26c040450 con 0x7fe27415afc0 2024-03-24T00:38:29.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.915+0000 7fe27a4d1640 1 -- 172.21.15.72:0/3212868686 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2741f3960 con 0x7fe27415afc0 2024-03-24T00:38:29.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.917+0000 7fe270ff9640 1 -- 172.21.15.72:0/3212868686 <== mon.2 v2:172.21.15.72:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fe26c03ea40 con 0x7fe27415afc0 2024-03-24T00:38:29.965 INFO:tasks.ceph.mgr.x.smithi072.stderr:2024-03-24T00:38:29.965+0000 7f19dd22b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T00:38:29.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.990+0000 7fe27a4d1640 1 -- 172.21.15.72:0/3212868686 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "osd new", "uuid": "7f138bad-98b0-4632-b528-e517933b1d47", "id": 7} v 0) v1 -- 0x7fe27404f400 con 0x7fe27415afc0 2024-03-24T00:38:29.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.998+0000 7fe270ff9640 1 -- 172.21.15.72:0/3212868686 <== mon.2 v2:172.21.15.72:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7f138bad-98b0-4632-b528-e517933b1d47", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe26c03ec20 con 0x7fe27415afc0 2024-03-24T00:38:29.999 INFO:teuthology.orchestra.run.smithi072.stdout:7 2024-03-24T00:38:30.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.999+0000 7fe27a4d1640 1 -- 172.21.15.72:0/3212868686 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe27415afc0 msgr2=0x7fe2741f6f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:30.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.999+0000 7fe27a4d1640 1 --2- 172.21.15.72:0/3212868686 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe27415afc0 0x7fe2741f6f50 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe26c009eb0 tx=0x7fe26c004a40 comp rx=0 tx=0).stop 2024-03-24T00:38:30.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.999+0000 7fe27a4d1640 1 -- 172.21.15.72:0/3212868686 shutdown_connections 2024-03-24T00:38:30.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.999+0000 7fe27a4d1640 1 --2- 172.21.15.72:0/3212868686 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe27415afc0 0x7fe2741f6f50 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:30.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.999+0000 7fe27a4d1640 1 --2- 172.21.15.72:0/3212868686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2740d1030 0x7fe2741f4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:30.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.999+0000 7fe27a4d1640 1 --2- 172.21.15.72:0/3212868686 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2740cbe80 0x7fe274158100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:30.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.999+0000 7fe27a4d1640 1 -- 172.21.15.72:0/3212868686 >> 172.21.15.72:0/3212868686 conn(0x7fe274099a30 msgr2=0x7fe2740cb770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:30.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.999+0000 7fe27a4d1640 1 -- 172.21.15.72:0/3212868686 shutdown_connections 2024-03-24T00:38:30.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-24T00:38:29.999+0000 7fe27a4d1640 1 -- 172.21.15.72:0/3212868686 wait complete. 2024-03-24T00:38:30.010 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-24T00:38:30.010 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-24T00:38:30.012 INFO:tasks.ceph.osd.0:Started 2024-03-24T00:38:30.012 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-24T00:38:30.012 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-24T00:38:30.013 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-24T00:38:30.012+0000 7fa361c66040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T00:38:30.017 INFO:tasks.ceph.osd.1:Started 2024-03-24T00:38:30.017 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-24T00:38:30.017 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-24T00:38:30.019 INFO:tasks.ceph.osd.2:Started 2024-03-24T00:38:30.019 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-24T00:38:30.019 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-24T00:38:30.021 INFO:tasks.ceph.osd.3:Started 2024-03-24T00:38:30.021 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-24T00:38:30.021 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-24T00:38:30.023 INFO:tasks.ceph.osd.4:Started 2024-03-24T00:38:30.023 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-24T00:38:30.024 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-24T00:38:30.025 INFO:tasks.ceph.osd.5:Started 2024-03-24T00:38:30.026 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-24T00:38:30.026 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-24T00:38:30.027 INFO:tasks.ceph.osd.6:Started 2024-03-24T00:38:30.028 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-24T00:38:30.028 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-24T00:38:30.030 INFO:tasks.ceph.osd.7:Started 2024-03-24T00:38:30.030 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:38:30.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.160+0000 7fc188627640 1 Processor -- start 2024-03-24T00:38:30.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.160+0000 7fc188627640 1 -- start start 2024-03-24T00:38:30.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.161+0000 7fc188627640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc1800cf7b0 0x7fc1800cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:30.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.161+0000 7fc188627640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1800ccca0 con 0x7fc18015bcd0 2024-03-24T00:38:30.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.161+0000 7fc188627640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1800ced20 con 0x7fc1800d0180 2024-03-24T00:38:30.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.161+0000 7fc188627640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1800b1b00 con 0x7fc1800cf7b0 2024-03-24T00:38:30.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.162+0000 7fc1863c3640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc18015bcd0 0x7fc1800ce540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33182/0 (socket says 172.21.15.67:33182) 2024-03-24T00:38:30.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.162+0000 7fc1863c3640 1 -- 172.21.15.67:0/2881948366 learned_addr learned my addr 172.21.15.67:0/2881948366 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:38:30.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.162+0000 7fc185bc2640 1 --2- 172.21.15.67:0/2881948366 >> v2:172.21.15.72:3300/0 conn(0x7fc1800cf7b0 0x7fc1800cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:30.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.162+0000 7fc1853c1640 1 -- 172.21.15.67:0/2881948366 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3206590051 0 0) 0x7fc1800ccca0 con 0x7fc18015bcd0 2024-03-24T00:38:30.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.162+0000 7fc1853c1640 1 -- 172.21.15.67:0/2881948366 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc16c003520 con 0x7fc18015bcd0 2024-03-24T00:38:30.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.162+0000 7fc1853c1640 1 -- 172.21.15.67:0/2881948366 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1616943272 0 0) 0x7fc1800ced20 con 0x7fc1800d0180 2024-03-24T00:38:30.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.162+0000 7fc1853c1640 1 -- 172.21.15.67:0/2881948366 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1800ccca0 con 0x7fc1800d0180 2024-03-24T00:38:30.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.162+0000 7fc1853c1640 1 -- 172.21.15.67:0/2881948366 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1909275453 0 0) 0x7fc16c003520 con 0x7fc18015bcd0 2024-03-24T00:38:30.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.162+0000 7fc1853c1640 1 -- 172.21.15.67:0/2881948366 >> v2:172.21.15.72:3300/0 conn(0x7fc1800cf7b0 msgr2=0x7fc1800cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:30.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.162+0000 7fc1853c1640 1 --2- 172.21.15.67:0/2881948366 >> v2:172.21.15.72:3300/0 conn(0x7fc1800cf7b0 0x7fc1800cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:30.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.162+0000 7fc1853c1640 1 -- 172.21.15.67:0/2881948366 >> v1:172.21.15.72:6789/0 conn(0x7fc1800d0180 legacy=0x7fc1800b1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:30.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.162+0000 7fc1853c1640 1 -- 172.21.15.67:0/2881948366 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1800cf000 con 0x7fc18015bcd0 2024-03-24T00:38:30.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.162+0000 7fc1853c1640 1 -- 172.21.15.67:0/2881948366 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7fc17c001690 con 0x7fc18015bcd0 2024-03-24T00:38:30.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.162+0000 7fc1853c1640 1 -- 172.21.15.67:0/2881948366 >> v1:172.21.15.67:6789/0 conn(0x7fc18015bcd0 legacy=0x7fc1800ce540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:30.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.162+0000 7fc1853c1640 1 --2- 172.21.15.67:0/2881948366 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc16c003b70 0x7fc16c003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:30.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.162+0000 7fc1853c1640 1 --2- 172.21.15.67:0/2881948366 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc16c004800 0x7fc16c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:30.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.162+0000 7fc1853c1640 1 --2- 172.21.15.67:0/2881948366 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc16c0051c0 0x7fc16c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:30.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.162+0000 7fc1853c1640 1 -- 172.21.15.67:0/2881948366 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc16c00c010 con 0x7fc16c0051c0 2024-03-24T00:38:30.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.162+0000 7fc1853c1640 1 -- 172.21.15.67:0/2881948366 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc16c00c1c0 con 0x7fc16c004800 2024-03-24T00:38:30.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.162+0000 7fc1853c1640 1 -- 172.21.15.67:0/2881948366 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc16c00c3c0 con 0x7fc16c003b70 2024-03-24T00:38:30.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.162+0000 7fc185bc2640 1 --2- 172.21.15.67:0/2881948366 >> v2:172.21.15.72:3300/0 conn(0x7fc1800cf7b0 0x7fc1800cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:30.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.162+0000 7fc186bc4640 1 --2- 172.21.15.67:0/2881948366 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc16c0051c0 0x7fc16c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:30.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.162+0000 7fc1863c3640 1 --2- 172.21.15.67:0/2881948366 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc16c003b70 0x7fc16c003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:30.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.162+0000 7fc185bc2640 1 --2- 172.21.15.67:0/2881948366 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc16c004800 0x7fc16c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:30.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.163+0000 7fc185bc2640 1 -- 172.21.15.67:0/2881948366 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc16c003b70 msgr2=0x7fc16c003f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:30.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.163+0000 7fc185bc2640 1 --2- 172.21.15.67:0/2881948366 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc16c003b70 0x7fc16c003f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:30.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.163+0000 7fc185bc2640 1 -- 172.21.15.67:0/2881948366 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc16c0051c0 msgr2=0x7fc16c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:30.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.163+0000 7fc185bc2640 1 --2- 172.21.15.67:0/2881948366 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc16c0051c0 0x7fc16c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:30.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.163+0000 7fc185bc2640 1 -- 172.21.15.67:0/2881948366 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc16c003520 con 0x7fc16c004800 2024-03-24T00:38:30.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.163+0000 7fc186bc4640 1 --2- 172.21.15.67:0/2881948366 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc16c0051c0 0x7fc16c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:30.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.163+0000 7fc186bc4640 1 -- 172.21.15.67:0/2881948366 reap_dead start 2024-03-24T00:38:30.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.163+0000 7fc185bc2640 1 --2- 172.21.15.67:0/2881948366 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc16c004800 0x7fc16c004c80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc170009c90 tx=0x7fc16c005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f77116be6557de84 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:30.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.164+0000 7fc1853c1640 1 -- 172.21.15.67:0/2881948366 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc170039070 con 0x7fc16c004800 2024-03-24T00:38:30.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.164+0000 7fc1853c1640 1 -- 172.21.15.67:0/2881948366 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc17002c8f0 con 0x7fc16c004800 2024-03-24T00:38:30.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.164+0000 7fc1853c1640 1 -- 172.21.15.67:0/2881948366 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc170009860 con 0x7fc16c004800 2024-03-24T00:38:30.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.164+0000 7fc188627640 1 -- 172.21.15.67:0/2881948366 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc16c004800 msgr2=0x7fc16c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:30.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.164+0000 7fc188627640 1 --2- 172.21.15.67:0/2881948366 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc16c004800 0x7fc16c004c80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc170009c90 tx=0x7fc16c005830 comp rx=0 tx=0).stop 2024-03-24T00:38:30.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.164+0000 7fc188627640 1 -- 172.21.15.67:0/2881948366 shutdown_connections 2024-03-24T00:38:30.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.164+0000 7fc188627640 1 --2- 172.21.15.67:0/2881948366 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc16c004800 0x7fc16c004c80 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:30.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.164+0000 7fc188627640 1 -- 172.21.15.67:0/2881948366 >> 172.21.15.67:0/2881948366 conn(0x7fc18008b1b0 msgr2=0x7fc18008b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:30.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.164+0000 7fc188627640 1 -- 172.21.15.67:0/2881948366 shutdown_connections 2024-03-24T00:38:30.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.164+0000 7fc188627640 1 -- 172.21.15.67:0/2881948366 wait complete. 2024-03-24T00:38:30.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.166+0000 7fc188627640 1 Processor -- start 2024-03-24T00:38:30.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.166+0000 7fc188627640 1 -- start start 2024-03-24T00:38:30.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.166+0000 7fc188627640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc16c004800 0x7fc1800ca730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:30.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.166+0000 7fc188627640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc18015eb50 0x7fc1800b0a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:30.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.166+0000 7fc188627640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc18015efd0 0x7fc18014ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:30.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.166+0000 7fc188627640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc18015c230 con 0x7fc18015eb50 2024-03-24T00:38:30.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.166+0000 7fc188627640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc18015c0b0 con 0x7fc16c004800 2024-03-24T00:38:30.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.166+0000 7fc188627640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1800cc760 con 0x7fc18015efd0 2024-03-24T00:38:30.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.167+0000 7fc1863c3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc16c004800 0x7fc1800ca730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:30.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.167+0000 7fc185bc2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc18015eb50 0x7fc1800b0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:30.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.167+0000 7fc1863c3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc16c004800 0x7fc1800ca730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:43946/0 (socket says 172.21.15.67:43946) 2024-03-24T00:38:30.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.167+0000 7fc1863c3640 1 -- 172.21.15.67:0/182163145 learned_addr learned my addr 172.21.15.67:0/182163145 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:38:30.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.167+0000 7fc185bc2640 1 --2- 172.21.15.67:0/182163145 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc18015eb50 0x7fc1800b0a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55864/0 (socket says 172.21.15.67:55864) 2024-03-24T00:38:30.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.167+0000 7fc186bc4640 1 --2- 172.21.15.67:0/182163145 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc18015efd0 0x7fc18014ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:30.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.167+0000 7fc1863c3640 1 -- 172.21.15.67:0/182163145 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc18015efd0 msgr2=0x7fc18014ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:30.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.167+0000 7fc1863c3640 1 --2- 172.21.15.67:0/182163145 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc18015efd0 0x7fc18014ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:30.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.167+0000 7fc1863c3640 1 -- 172.21.15.67:0/182163145 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc18015eb50 msgr2=0x7fc1800b0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:30.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.167+0000 7fc1863c3640 1 --2- 172.21.15.67:0/182163145 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc18015eb50 0x7fc1800b0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:30.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.167+0000 7fc1863c3640 1 -- 172.21.15.67:0/182163145 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1800cbdb0 con 0x7fc16c004800 2024-03-24T00:38:30.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.167+0000 7fc185bc2640 1 --2- 172.21.15.67:0/182163145 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc18015eb50 0x7fc1800b0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:30.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.167+0000 7fc1863c3640 1 --2- 172.21.15.67:0/182163145 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc16c004800 0x7fc1800ca730 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc17c007290 tx=0x7fc17c007690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:30.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.167+0000 7fc176ffd640 1 -- 172.21.15.67:0/182163145 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc17c001660 con 0x7fc16c004800 2024-03-24T00:38:30.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.167+0000 7fc176ffd640 1 -- 172.21.15.67:0/182163145 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc17c018070 con 0x7fc16c004800 2024-03-24T00:38:30.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.167+0000 7fc188627640 1 -- 172.21.15.67:0/182163145 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1800cad10 con 0x7fc16c004800 2024-03-24T00:38:30.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.167+0000 7fc176ffd640 1 -- 172.21.15.67:0/182163145 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc17c00fbc0 con 0x7fc16c004800 2024-03-24T00:38:30.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.167+0000 7fc188627640 1 -- 172.21.15.67:0/182163145 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1800cb2d0 con 0x7fc16c004800 2024-03-24T00:38:30.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.167+0000 7fc176ffd640 1 -- 172.21.15.67:0/182163145 <== mon.1 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fc17c01d460 con 0x7fc16c004800 2024-03-24T00:38:30.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.167+0000 7fc176ffd640 1 -- 172.21.15.67:0/182163145 <== mon.1 v2:172.21.15.72:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fc17c00fd60 con 0x7fc16c004800 2024-03-24T00:38:30.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.169+0000 7fc188627640 1 -- 172.21.15.67:0/182163145 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc18004f400 con 0x7fc16c004800 2024-03-24T00:38:30.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.171+0000 7fc176ffd640 1 -- 172.21.15.67:0/182163145 <== mon.1 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fc17c0135d0 con 0x7fc16c004800 2024-03-24T00:38:30.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.233+0000 7fc188627640 1 -- 172.21.15.67:0/182163145 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc1800c8540 con 0x7fc16c004800 2024-03-24T00:38:30.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.233+0000 7fc176ffd640 1 -- 172.21.15.67:0/182163145 <== mon.1 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7fc17c0137b0 con 0x7fc16c004800 2024-03-24T00:38:30.234 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:38:30.234 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":10,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:38:29.992263+0000","last_up_change":"0.000000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:38:30.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.234+0000 7fc188627640 1 -- 172.21.15.67:0/182163145 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc16c004800 msgr2=0x7fc1800ca730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:30.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.234+0000 7fc188627640 1 --2- 172.21.15.67:0/182163145 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc16c004800 0x7fc1800ca730 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc17c007290 tx=0x7fc17c007690 comp rx=0 tx=0).stop 2024-03-24T00:38:30.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.234+0000 7fc188627640 1 -- 172.21.15.67:0/182163145 shutdown_connections 2024-03-24T00:38:30.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.234+0000 7fc188627640 1 --2- 172.21.15.67:0/182163145 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc18015efd0 0x7fc18014ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:30.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.234+0000 7fc188627640 1 --2- 172.21.15.67:0/182163145 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc18015eb50 0x7fc1800b0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:30.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.234+0000 7fc188627640 1 --2- 172.21.15.67:0/182163145 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc16c004800 0x7fc1800ca730 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:30.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.234+0000 7fc188627640 1 -- 172.21.15.67:0/182163145 >> 172.21.15.67:0/182163145 conn(0x7fc18008b1b0 msgr2=0x7fc1800ccea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:30.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.234+0000 7fc188627640 1 -- 172.21.15.67:0/182163145 shutdown_connections 2024-03-24T00:38:30.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:30.234+0000 7fc188627640 1 -- 172.21.15.67:0/182163145 wait complete. 2024-03-24T00:38:30.245 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-24T00:38:30.245 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-24T00:38:30.621 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-24T00:38:30.620+0000 7f532081a640 -1 Falling back to public interface 2024-03-24T00:38:30.628 INFO:tasks.ceph.osd.5.smithi072.stderr:2024-03-24T00:38:30.628+0000 7febe2815640 -1 Falling back to public interface 2024-03-24T00:38:30.632 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-03-24T00:38:30.631+0000 7f21fda11640 -1 Falling back to public interface 2024-03-24T00:38:30.632 INFO:tasks.ceph.osd.4.smithi072.stderr:2024-03-24T00:38:30.632+0000 7f9b9e106640 -1 Falling back to public interface 2024-03-24T00:38:30.650 INFO:tasks.ceph.osd.6.smithi072.stderr:2024-03-24T00:38:30.649+0000 7f142860d640 -1 Falling back to public interface 2024-03-24T00:38:30.651 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-03-24T00:38:30.650+0000 7fc83a219640 -1 Falling back to public interface 2024-03-24T00:38:30.651 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-03-24T00:38:30.650+0000 7f1340a24640 -1 Falling back to public interface 2024-03-24T00:38:30.654 INFO:tasks.ceph.osd.7.smithi072.stderr:2024-03-24T00:38:30.654+0000 7f32b730f640 -1 Falling back to public interface 2024-03-24T00:38:30.853 INFO:tasks.ceph.mgr.x.smithi072.stderr:2024-03-24T00:38:30.853+0000 7f19dd22b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T00:38:30.997 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-24T00:38:30.997+0000 7fa361c66040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T00:38:31.052 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-24T00:38:31.131 INFO:tasks.ceph.mgr.x.smithi072.stderr:2024-03-24T00:38:31.131+0000 7f19dd22b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T00:38:31.141 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.141+0000 7fdf51e7e640 1 Processor -- start 2024-03-24T00:38:31.142 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.141+0000 7fdf51e7e640 1 -- start start 2024-03-24T00:38:31.143 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.142+0000 7fdf51e7e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdf4c06b090 0x7fdf4c06b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:31.143 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.142+0000 7fdf51e7e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf4c0b02f0 con 0x7fdf4c0ccc40 2024-03-24T00:38:31.143 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.142+0000 7fdf51e7e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf4c0cdb60 con 0x7fdf4c0b07d0 2024-03-24T00:38:31.143 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.142+0000 7fdf51e7e640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf4c0cde40 con 0x7fdf4c06b090 2024-03-24T00:38:31.143 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.142+0000 7fdf4affd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdf4c06b090 0x7fdf4c06b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:31.143 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.142+0000 7fdf4b7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fdf4c0ccc40 0x7fdf4c0cd020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33198/0 (socket says 172.21.15.67:33198) 2024-03-24T00:38:31.143 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.142+0000 7fdf4b7fe640 1 -- 172.21.15.67:0/116061908 learned_addr learned my addr 172.21.15.67:0/116061908 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:38:31.143 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.142+0000 7fdf4affd640 1 -- 172.21.15.67:0/116061908 >> v1:172.21.15.72:6789/0 conn(0x7fdf4c0b07d0 legacy=0x7fdf4c0b0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:31.144 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.142+0000 7fdf4affd640 1 -- 172.21.15.67:0/116061908 >> v1:172.21.15.67:6789/0 conn(0x7fdf4c0ccc40 legacy=0x7fdf4c0cd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:31.144 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.142+0000 7fdf4affd640 1 -- 172.21.15.67:0/116061908 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf4c0ce120 con 0x7fdf4c06b090 2024-03-24T00:38:31.144 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.142+0000 7fdf4a7fc640 1 -- 172.21.15.67:0/116061908 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1725381236 0 0) 0x7fdf4c0b02f0 con 0x7fdf4c0ccc40 2024-03-24T00:38:31.145 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.143+0000 7fdf4affd640 1 --2- 172.21.15.67:0/116061908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf4c06b090 0x7fdf4c06b520 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fdf340097a0 tx=0x7fdf3402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e21b804b6eb8a12 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:31.145 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.143+0000 7fdf4a7fc640 1 -- 172.21.15.67:0/116061908 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf34034070 con 0x7fdf4c06b090 2024-03-24T00:38:31.145 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.143+0000 7fdf4a7fc640 1 -- 172.21.15.67:0/116061908 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf34030610 con 0x7fdf4c06b090 2024-03-24T00:38:31.145 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.143+0000 7fdf4a7fc640 1 -- 172.21.15.67:0/116061908 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf340309e0 con 0x7fdf4c06b090 2024-03-24T00:38:31.145 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.144+0000 7fdf51e7e640 1 -- 172.21.15.67:0/116061908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf4c06b090 msgr2=0x7fdf4c06b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:31.145 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.144+0000 7fdf51e7e640 1 --2- 172.21.15.67:0/116061908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf4c06b090 0x7fdf4c06b520 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fdf340097a0 tx=0x7fdf3402b960 comp rx=0 tx=0).stop 2024-03-24T00:38:31.145 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.144+0000 7fdf51e7e640 1 -- 172.21.15.67:0/116061908 shutdown_connections 2024-03-24T00:38:31.145 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.144+0000 7fdf51e7e640 1 --2- 172.21.15.67:0/116061908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf4c06b090 0x7fdf4c06b520 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:31.145 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.144+0000 7fdf51e7e640 1 -- 172.21.15.67:0/116061908 >> 172.21.15.67:0/116061908 conn(0x7fdf4c099a30 msgr2=0x7fdf4c099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:31.146 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.144+0000 7fdf51e7e640 1 -- 172.21.15.67:0/116061908 shutdown_connections 2024-03-24T00:38:31.146 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.144+0000 7fdf51e7e640 1 -- 172.21.15.67:0/116061908 wait complete. 2024-03-24T00:38:31.148 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.145+0000 7fdf51e7e640 1 Processor -- start 2024-03-24T00:38:31.148 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.145+0000 7fdf51e7e640 1 -- start start 2024-03-24T00:38:31.148 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.146+0000 7fdf51e7e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdf4c06b090 0x7fdf4c1f1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:31.148 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.146+0000 7fdf51e7e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf4c0b07d0 0x7fdf4c157d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:31.148 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.146+0000 7fdf51e7e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf4c0ccc40 0x7fdf4c0c5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:31.149 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.146+0000 7fdf51e7e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4c0cd8b0 con 0x7fdf4c0b07d0 2024-03-24T00:38:31.149 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.146+0000 7fdf51e7e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4c0cd730 con 0x7fdf4c0ccc40 2024-03-24T00:38:31.149 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.146+0000 7fdf51e7e640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4c0d16f0 con 0x7fdf4c06b090 2024-03-24T00:38:31.149 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.146+0000 7fdf4affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf4c0b07d0 0x7fdf4c157d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:31.149 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.146+0000 7fdf4affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf4c0b07d0 0x7fdf4c157d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55872/0 (socket says 172.21.15.67:55872) 2024-03-24T00:38:31.149 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.146+0000 7fdf4affd640 1 -- 172.21.15.67:0/3507495746 learned_addr learned my addr 172.21.15.67:0/3507495746 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:38:31.149 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.146+0000 7fdf4b7fe640 1 --2- 172.21.15.67:0/3507495746 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdf4c06b090 0x7fdf4c1f1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:31.149 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.146+0000 7fdf4bfff640 1 --2- 172.21.15.67:0/3507495746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf4c0ccc40 0x7fdf4c0c5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:31.149 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.146+0000 7fdf4affd640 1 -- 172.21.15.67:0/3507495746 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdf4c06b090 msgr2=0x7fdf4c1f1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:31.149 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.146+0000 7fdf4affd640 1 --2- 172.21.15.67:0/3507495746 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdf4c06b090 0x7fdf4c1f1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:31.149 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.146+0000 7fdf4affd640 1 -- 172.21.15.67:0/3507495746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf4c0ccc40 msgr2=0x7fdf4c0c5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:31.149 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.146+0000 7fdf4affd640 1 --2- 172.21.15.67:0/3507495746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf4c0ccc40 0x7fdf4c0c5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:31.149 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.146+0000 7fdf4affd640 1 -- 172.21.15.67:0/3507495746 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf4c0c7db0 con 0x7fdf4c0b07d0 2024-03-24T00:38:31.149 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.146+0000 7fdf4affd640 1 --2- 172.21.15.67:0/3507495746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf4c0b07d0 0x7fdf4c157d10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fdf3402bf70 tx=0x7fdf340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:31.149 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.147+0000 7fdf43fff640 1 -- 172.21.15.67:0/3507495746 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf34034040 con 0x7fdf4c0b07d0 2024-03-24T00:38:31.150 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.147+0000 7fdf43fff640 1 -- 172.21.15.67:0/3507495746 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf3403acc0 con 0x7fdf4c0b07d0 2024-03-24T00:38:31.150 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.147+0000 7fdf51e7e640 1 -- 172.21.15.67:0/3507495746 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf4c0c8090 con 0x7fdf4c0b07d0 2024-03-24T00:38:31.150 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.147+0000 7fdf43fff640 1 -- 172.21.15.67:0/3507495746 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf34044630 con 0x7fdf4c0b07d0 2024-03-24T00:38:31.150 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.147+0000 7fdf51e7e640 1 -- 172.21.15.67:0/3507495746 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf4c158250 con 0x7fdf4c0b07d0 2024-03-24T00:38:31.150 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.147+0000 7fdf43fff640 1 -- 172.21.15.67:0/3507495746 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fdf34037070 con 0x7fdf4c0b07d0 2024-03-24T00:38:31.150 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.147+0000 7fdf43fff640 1 -- 172.21.15.67:0/3507495746 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fdf3404dac0 con 0x7fdf4c0b07d0 2024-03-24T00:38:31.150 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.148+0000 7fdf51e7e640 1 -- 172.21.15.67:0/3507495746 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf4c04f470 con 0x7fdf4c0b07d0 2024-03-24T00:38:31.150 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.150+0000 7fdf43fff640 1 -- 172.21.15.67:0/3507495746 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fdf3403f070 con 0x7fdf4c0b07d0 2024-03-24T00:38:31.184 INFO:tasks.ceph.mgr.x.smithi072.stderr:2024-03-24T00:38:31.183+0000 7f19dd22b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T00:38:31.213 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.212+0000 7fdf51e7e640 1 -- 172.21.15.67:0/3507495746 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdf4c0d0ca0 con 0x7fdf4c0b07d0 2024-03-24T00:38:31.213 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.213+0000 7fdf43fff640 1 -- 172.21.15.67:0/3507495746 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7fdf340305a0 con 0x7fdf4c0b07d0 2024-03-24T00:38:31.214 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-24T00:38:31.214 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:38:29.992263+0000","last_up_change":"0.000000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:38:31.215 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.214+0000 7fdf51e7e640 1 -- 172.21.15.67:0/3507495746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf4c0b07d0 msgr2=0x7fdf4c157d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:31.215 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.214+0000 7fdf51e7e640 1 --2- 172.21.15.67:0/3507495746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf4c0b07d0 0x7fdf4c157d10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fdf3402bf70 tx=0x7fdf340046c0 comp rx=0 tx=0).stop 2024-03-24T00:38:31.215 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.214+0000 7fdf51e7e640 1 -- 172.21.15.67:0/3507495746 shutdown_connections 2024-03-24T00:38:31.215 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.214+0000 7fdf51e7e640 1 --2- 172.21.15.67:0/3507495746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf4c0ccc40 0x7fdf4c0c5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:31.215 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.214+0000 7fdf51e7e640 1 --2- 172.21.15.67:0/3507495746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf4c0b07d0 0x7fdf4c157d10 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:31.215 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.214+0000 7fdf51e7e640 1 --2- 172.21.15.67:0/3507495746 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdf4c06b090 0x7fdf4c1f1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:31.215 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.214+0000 7fdf51e7e640 1 -- 172.21.15.67:0/3507495746 >> 172.21.15.67:0/3507495746 conn(0x7fdf4c099a30 msgr2=0x7fdf4c0ae180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:31.215 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.214+0000 7fdf51e7e640 1 -- 172.21.15.67:0/3507495746 shutdown_connections 2024-03-24T00:38:31.215 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:31.214+0000 7fdf51e7e640 1 -- 172.21.15.67:0/3507495746 wait complete. 2024-03-24T00:38:31.223 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-24T00:38:31.288 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-24T00:38:31.288+0000 7fa361c66040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T00:38:31.299 INFO:tasks.ceph.mgr.x.smithi072.stderr:2024-03-24T00:38:31.299+0000 7f19dd22b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T00:38:31.343 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-24T00:38:31.343+0000 7fa361c66040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T00:38:31.464 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-24T00:38:31.464+0000 7fa361c66040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T00:38:32.326 INFO:tasks.ceph.mgr.x.smithi072.stderr:2024-03-24T00:38:32.325+0000 7f197d791640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-24T00:38:32.326 INFO:tasks.ceph.mgr.x.smithi072.stderr:2024-03-24T00:38:32.326+0000 7f197d791640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-24T00:38:34.068 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-24T00:38:34.067+0000 7f532081a640 -1 osd.1 0 log_to_monitors true 2024-03-24T00:38:34.092 INFO:tasks.ceph.osd.5.smithi072.stderr:2024-03-24T00:38:34.092+0000 7febe2815640 -1 osd.5 0 log_to_monitors true 2024-03-24T00:38:34.110 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-03-24T00:38:34.109+0000 7f21fda11640 -1 osd.0 0 log_to_monitors true 2024-03-24T00:38:34.129 INFO:tasks.ceph.osd.4.smithi072.stderr:2024-03-24T00:38:34.128+0000 7f9b9e106640 -1 osd.4 0 log_to_monitors true 2024-03-24T00:38:34.152 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-03-24T00:38:34.152+0000 7fc83a219640 -1 osd.2 0 log_to_monitors true 2024-03-24T00:38:34.157 INFO:tasks.ceph.osd.6.smithi072.stderr:2024-03-24T00:38:34.156+0000 7f142860d640 -1 osd.6 0 log_to_monitors true 2024-03-24T00:38:34.168 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-03-24T00:38:34.168+0000 7f1340a24640 -1 osd.3 0 log_to_monitors true 2024-03-24T00:38:34.170 INFO:tasks.ceph.osd.7.smithi072.stderr:2024-03-24T00:38:34.170+0000 7f32b730f640 -1 osd.7 0 log_to_monitors true 2024-03-24T00:38:37.369 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-03-24T00:38:37.370+0000 7f133c2fb640 -1 osd.3 0 waiting for initial osdmap 2024-03-24T00:38:37.388 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-03-24T00:38:37.389+0000 7f133411b640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T00:38:37.421 INFO:tasks.ceph.osd.6.smithi072.stderr:2024-03-24T00:38:37.421+0000 7f1423ed4640 -1 osd.6 0 waiting for initial osdmap 2024-03-24T00:38:37.427 INFO:tasks.ceph.osd.7.smithi072.stderr:2024-03-24T00:38:37.426+0000 7f32b32a5640 -1 osd.7 0 waiting for initial osdmap 2024-03-24T00:38:37.437 INFO:tasks.ceph.osd.6.smithi072.stderr:2024-03-24T00:38:37.437+0000 7f141bcf4640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T00:38:37.441 INFO:tasks.ceph.osd.7.smithi072.stderr:2024-03-24T00:38:37.441+0000 7f32ab0c5640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T00:38:37.452 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-03-24T00:38:37.452+0000 7fc835c0c640 -1 osd.2 0 waiting for initial osdmap 2024-03-24T00:38:37.456 INFO:tasks.ceph.osd.4.smithi072.stderr:2024-03-24T00:38:37.456+0000 7f9b9a09c640 -1 osd.4 0 waiting for initial osdmap 2024-03-24T00:38:37.457 INFO:tasks.ceph.osd.5.smithi072.stderr:2024-03-24T00:38:37.456+0000 7febde1c8640 -1 osd.5 0 waiting for initial osdmap 2024-03-24T00:38:37.466 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-03-24T00:38:37.467+0000 7fc82da2c640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T00:38:37.478 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-24T00:38:37.479+0000 7f531c20c640 -1 osd.1 0 waiting for initial osdmap 2024-03-24T00:38:37.478 INFO:tasks.ceph.osd.4.smithi072.stderr:2024-03-24T00:38:37.478+0000 7f9b91ebc640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T00:38:37.479 INFO:tasks.ceph.osd.5.smithi072.stderr:2024-03-24T00:38:37.478+0000 7febd5fe8640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T00:38:37.491 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-24T00:38:37.492+0000 7f531402c640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T00:38:37.503 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-03-24T00:38:37.504+0000 7f21f9404640 -1 osd.0 0 waiting for initial osdmap 2024-03-24T00:38:37.520 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-03-24T00:38:37.521+0000 7f21f1224640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T00:38:38.029 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-24T00:38:38.121 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.121+0000 7f275492f640 1 Processor -- start 2024-03-24T00:38:38.121 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.121+0000 7f275492f640 1 -- start start 2024-03-24T00:38:38.122 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.122+0000 7f275492f640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f275015e0c0 0x7f27500d1e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:38.123 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.122+0000 7f275492f640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f27500d1400 0x7f27500b6f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:38.123 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.122+0000 7f275492f640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27500d2920 con 0x7f27500d23a0 2024-03-24T00:38:38.123 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.122+0000 7f275492f640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27500d3530 con 0x7f27500d1400 2024-03-24T00:38:38.123 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.122+0000 7f275492f640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27500d2c00 con 0x7f275015e0c0 2024-03-24T00:38:38.123 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.122+0000 7f274ed9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f27500d1400 0x7f27500b6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:38.123 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.122+0000 7f274e59c640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f275015e0c0 0x7f27500d1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:38.123 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.122+0000 7f274dd9b640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f27500d23a0 0x7f27500d0cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:55412/0 (socket says 172.21.15.67:55412) 2024-03-24T00:38:38.123 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.122+0000 7f274dd9b640 1 -- 172.21.15.67:0/3877466622 learned_addr learned my addr 172.21.15.67:0/3877466622 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:38:38.123 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.123+0000 7f274d59a640 1 -- 172.21.15.67:0/3877466622 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2956253258 0 0) 0x7f27500d2920 con 0x7f27500d23a0 2024-03-24T00:38:38.123 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.123+0000 7f274d59a640 1 -- 172.21.15.67:0/3877466622 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f273c003520 con 0x7f27500d23a0 2024-03-24T00:38:38.123 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.123+0000 7f274ed9d640 1 -- 172.21.15.67:0/3877466622 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f275015e0c0 msgr2=0x7f27500d1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:38.123 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.123+0000 7f274ed9d640 1 --2- 172.21.15.67:0/3877466622 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f275015e0c0 0x7f27500d1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:38.124 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.123+0000 7f274ed9d640 1 -- 172.21.15.67:0/3877466622 >> v1:172.21.15.72:6789/0 conn(0x7f27500d23a0 legacy=0x7f27500d0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:38.124 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.123+0000 7f274ed9d640 1 -- 172.21.15.67:0/3877466622 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27500d2ee0 con 0x7f27500d1400 2024-03-24T00:38:38.124 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.123+0000 7f274e59c640 1 --2- 172.21.15.67:0/3877466622 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f275015e0c0 0x7f27500d1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:38.125 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.124+0000 7f274ed9d640 1 --2- 172.21.15.67:0/3877466622 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27500d1400 0x7f27500b6f50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2740009d90 tx=0x7f274002bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c523402206054a88 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:38.125 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.124+0000 7f274d59a640 1 -- 172.21.15.67:0/3877466622 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2740039070 con 0x7f27500d1400 2024-03-24T00:38:38.125 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.124+0000 7f274d59a640 1 -- 172.21.15.67:0/3877466622 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f274002cc50 con 0x7f27500d1400 2024-03-24T00:38:38.125 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.124+0000 7f274d59a640 1 -- 172.21.15.67:0/3877466622 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2740034440 con 0x7f27500d1400 2024-03-24T00:38:38.125 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.124+0000 7f275492f640 1 -- 172.21.15.67:0/3877466622 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27500d1400 msgr2=0x7f27500b6f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:38.125 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.124+0000 7f275492f640 1 --2- 172.21.15.67:0/3877466622 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27500d1400 0x7f27500b6f50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2740009d90 tx=0x7f274002bea0 comp rx=0 tx=0).stop 2024-03-24T00:38:38.125 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.124+0000 7f275492f640 1 -- 172.21.15.67:0/3877466622 shutdown_connections 2024-03-24T00:38:38.126 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.124+0000 7f275492f640 1 --2- 172.21.15.67:0/3877466622 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27500d1400 0x7f27500b6f50 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:38.126 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.124+0000 7f275492f640 1 --2- 172.21.15.67:0/3877466622 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f275015e0c0 0x7f27500d1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:38.126 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.124+0000 7f275492f640 1 -- 172.21.15.67:0/3877466622 >> 172.21.15.67:0/3877466622 conn(0x7f2750072bb0 msgr2=0x7f2750072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:38.126 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.124+0000 7f275492f640 1 -- 172.21.15.67:0/3877466622 shutdown_connections 2024-03-24T00:38:38.126 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.124+0000 7f275492f640 1 -- 172.21.15.67:0/3877466622 wait complete. 2024-03-24T00:38:38.126 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.126+0000 7f275492f640 1 Processor -- start 2024-03-24T00:38:38.126 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.126+0000 7f275492f640 1 -- start start 2024-03-24T00:38:38.128 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.126+0000 7f275492f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27500d1400 0x7f275015ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:38.128 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.126+0000 7f275492f640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f27500d23a0 0x7f275015a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:38.128 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.126+0000 7f275492f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f275015e0c0 0x7f2750153a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:38.128 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.126+0000 7f275492f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27500ce7a0 con 0x7f27500d1400 2024-03-24T00:38:38.128 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.126+0000 7f275492f640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f275015e4a0 con 0x7f275015e0c0 2024-03-24T00:38:38.128 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.126+0000 7f275492f640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27500d49d0 con 0x7f27500d23a0 2024-03-24T00:38:38.128 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.126+0000 7f274e59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27500d1400 0x7f275015ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:38.129 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.126+0000 7f274ed9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f275015e0c0 0x7f2750153a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:38.129 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.126+0000 7f274e59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27500d1400 0x7f275015ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44090/0 (socket says 172.21.15.67:44090) 2024-03-24T00:38:38.129 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.126+0000 7f274e59c640 1 -- 172.21.15.67:0/2468603854 learned_addr learned my addr 172.21.15.67:0/2468603854 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:38:38.129 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.126+0000 7f274dd9b640 1 --2- 172.21.15.67:0/2468603854 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f27500d23a0 0x7f275015a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:38.129 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.127+0000 7f274ed9d640 1 -- 172.21.15.67:0/2468603854 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f27500d23a0 msgr2=0x7f275015a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:38.129 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.127+0000 7f274ed9d640 1 --2- 172.21.15.67:0/2468603854 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f27500d23a0 0x7f275015a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:38.129 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.127+0000 7f274ed9d640 1 -- 172.21.15.67:0/2468603854 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27500d1400 msgr2=0x7f275015ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:38.129 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.127+0000 7f274ed9d640 1 --2- 172.21.15.67:0/2468603854 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27500d1400 0x7f275015ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:38.129 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.127+0000 7f274ed9d640 1 -- 172.21.15.67:0/2468603854 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2750158b30 con 0x7f275015e0c0 2024-03-24T00:38:38.129 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.127+0000 7f274ed9d640 1 --2- 172.21.15.67:0/2468603854 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f275015e0c0 0x7f2750153a00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2740033000 tx=0x7f2740004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:38.129 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.127+0000 7f2746ffd640 1 -- 172.21.15.67:0/2468603854 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2740039050 con 0x7f275015e0c0 2024-03-24T00:38:38.129 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.127+0000 7f2746ffd640 1 -- 172.21.15.67:0/2468603854 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f274003a040 con 0x7f275015e0c0 2024-03-24T00:38:38.130 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.127+0000 7f275492f640 1 -- 172.21.15.67:0/2468603854 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2750158e10 con 0x7f275015e0c0 2024-03-24T00:38:38.130 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.127+0000 7f2746ffd640 1 -- 172.21.15.67:0/2468603854 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2740009450 con 0x7f275015e0c0 2024-03-24T00:38:38.130 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.127+0000 7f275492f640 1 -- 172.21.15.67:0/2468603854 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27501593d0 con 0x7f275015e0c0 2024-03-24T00:38:38.130 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.128+0000 7f275492f640 1 -- 172.21.15.67:0/2468603854 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2750068550 con 0x7f275015e0c0 2024-03-24T00:38:38.130 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.129+0000 7f2746ffd640 1 -- 172.21.15.67:0/2468603854 <== mon.1 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 4) v1 ==== 90080+0+0 (secure 0 0 0) 0x7f2740047020 con 0x7f275015e0c0 2024-03-24T00:38:38.131 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.129+0000 7f2746ffd640 1 --2- 172.21.15.67:0/2468603854 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f273006bb40 0x7f273006e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:38.131 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.129+0000 7f2746ffd640 1 -- 172.21.15.67:0/2468603854 <== mon.1 v2:172.21.15.72:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7f27400ab910 con 0x7f275015e0c0 2024-03-24T00:38:38.131 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.129+0000 7f274e59c640 1 --2- 172.21.15.67:0/2468603854 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f273006bb40 0x7f273006e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:38.131 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.130+0000 7f274e59c640 1 --2- 172.21.15.67:0/2468603854 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f273006bb40 0x7f273006e000 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2748009560 tx=0x7f2748009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:38.133 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.134+0000 7f2746ffd640 1 -- 172.21.15.67:0/2468603854 <== mon.1 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f274004e020 con 0x7f275015e0c0 2024-03-24T00:38:38.315 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.316+0000 7f275492f640 1 -- 172.21.15.67:0/2468603854 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f275004f470 con 0x7f275015e0c0 2024-03-24T00:38:38.316 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.317+0000 7f2746ffd640 1 -- 172.21.15.67:0/2468603854 <== mon.1 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f274007acb0 con 0x7f275015e0c0 2024-03-24T00:38:38.316 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-24T00:38:38.317 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":12,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:38:35.325497+0000","last_up_change":"0.000000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:38:38.319 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.318+0000 7f275492f640 1 -- 172.21.15.67:0/2468603854 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f273006bb40 msgr2=0x7f273006e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:38.319 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.319+0000 7f275492f640 1 --2- 172.21.15.67:0/2468603854 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f273006bb40 0x7f273006e000 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2748009560 tx=0x7f2748009190 comp rx=0 tx=0).stop 2024-03-24T00:38:38.319 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.319+0000 7f275492f640 1 -- 172.21.15.67:0/2468603854 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f275015e0c0 msgr2=0x7f2750153a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:38.319 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.319+0000 7f275492f640 1 --2- 172.21.15.67:0/2468603854 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f275015e0c0 0x7f2750153a00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2740033000 tx=0x7f2740004a40 comp rx=0 tx=0).stop 2024-03-24T00:38:38.319 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.319+0000 7f275492f640 1 -- 172.21.15.67:0/2468603854 shutdown_connections 2024-03-24T00:38:38.319 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.319+0000 7f275492f640 1 --2- 172.21.15.67:0/2468603854 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f273006bb40 0x7f273006e000 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:38.319 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.319+0000 7f275492f640 1 --2- 172.21.15.67:0/2468603854 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f275015e0c0 0x7f2750153a00 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:38.319 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.319+0000 7f275492f640 1 --2- 172.21.15.67:0/2468603854 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f27500d23a0 0x7f275015a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:38.319 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.319+0000 7f275492f640 1 --2- 172.21.15.67:0/2468603854 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27500d1400 0x7f275015ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:38.319 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.319+0000 7f275492f640 1 -- 172.21.15.67:0/2468603854 >> 172.21.15.67:0/2468603854 conn(0x7f2750072bb0 msgr2=0x7f27500d0cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:38.319 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.319+0000 7f275492f640 1 -- 172.21.15.67:0/2468603854 shutdown_connections 2024-03-24T00:38:38.320 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:38.319+0000 7f275492f640 1 -- 172.21.15.67:0/2468603854 wait complete. 2024-03-24T00:38:38.328 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-24T00:38:45.134 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-24T00:38:45.236 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.236+0000 7fd6fa590640 1 Processor -- start 2024-03-24T00:38:45.236 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.236+0000 7fd6fa590640 1 -- start start 2024-03-24T00:38:45.239 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.237+0000 7fd6fa590640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd6f40d0cf0 0x7fd6f40d1180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:45.239 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.237+0000 7fd6fa590640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6f40d2920 con 0x7fd6f415e0c0 2024-03-24T00:38:45.239 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.237+0000 7fd6fa590640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6f40d2c00 con 0x7fd6f40b6f50 2024-03-24T00:38:45.239 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.237+0000 7fd6fa590640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6f40d3530 con 0x7fd6f40d0cf0 2024-03-24T00:38:45.239 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.237+0000 7fd6f3fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd6f415e0c0 0x7fd6f40d1e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60582/0 (socket says 172.21.15.67:60582) 2024-03-24T00:38:45.240 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.237+0000 7fd6f3fff640 1 -- 172.21.15.67:0/2432634137 learned_addr learned my addr 172.21.15.67:0/2432634137 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:38:45.240 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.237+0000 7fd6f37fe640 1 --2- 172.21.15.67:0/2432634137 >> v2:172.21.15.72:3300/0 conn(0x7fd6f40d0cf0 0x7fd6f40d1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:45.240 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.237+0000 7fd6f2ffd640 1 -- 172.21.15.67:0/2432634137 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2829208071 0 0) 0x7fd6f40d2920 con 0x7fd6f415e0c0 2024-03-24T00:38:45.240 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.237+0000 7fd6f2ffd640 1 -- 172.21.15.67:0/2432634137 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6e0003520 con 0x7fd6f415e0c0 2024-03-24T00:38:45.240 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.238+0000 7fd6f2ffd640 1 -- 172.21.15.67:0/2432634137 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1274330918 0 0) 0x7fd6e0003520 con 0x7fd6f415e0c0 2024-03-24T00:38:45.240 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.238+0000 7fd6f2ffd640 1 -- 172.21.15.67:0/2432634137 >> v2:172.21.15.72:3300/0 conn(0x7fd6f40d0cf0 msgr2=0x7fd6f40d1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:45.240 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.238+0000 7fd6f2ffd640 1 --2- 172.21.15.67:0/2432634137 >> v2:172.21.15.72:3300/0 conn(0x7fd6f40d0cf0 0x7fd6f40d1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:45.240 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.238+0000 7fd6f2ffd640 1 -- 172.21.15.67:0/2432634137 >> v1:172.21.15.72:6789/0 conn(0x7fd6f40b6f50 legacy=0x7fd6f40b7360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:45.240 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.238+0000 7fd6f2ffd640 1 -- 172.21.15.67:0/2432634137 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6f40d2ee0 con 0x7fd6f415e0c0 2024-03-24T00:38:45.240 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.238+0000 7fd6f2ffd640 1 -- 172.21.15.67:0/2432634137 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7fd6e4001670 con 0x7fd6f415e0c0 2024-03-24T00:38:45.240 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.238+0000 7fd6f37fe640 1 --2- 172.21.15.67:0/2432634137 >> v2:172.21.15.72:3300/0 conn(0x7fd6f40d0cf0 0x7fd6f40d1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:45.240 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.238+0000 7fd6f2ffd640 1 -- 172.21.15.67:0/2432634137 >> v1:172.21.15.67:6789/0 conn(0x7fd6f415e0c0 legacy=0x7fd6f40d1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:45.240 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.238+0000 7fd6f2ffd640 1 --2- 172.21.15.67:0/2432634137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6e00039b0 0x7fd6e0003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:45.240 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.238+0000 7fd6f2ffd640 1 --2- 172.21.15.67:0/2432634137 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd6e0004800 0x7fd6e0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:45.240 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.238+0000 7fd6f2ffd640 1 --2- 172.21.15.67:0/2432634137 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd6e00051c0 0x7fd6e000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:45.240 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.238+0000 7fd6f2ffd640 1 -- 172.21.15.67:0/2432634137 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f40d3530 con 0x7fd6e00039b0 2024-03-24T00:38:45.241 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.238+0000 7fd6f2ffd640 1 -- 172.21.15.67:0/2432634137 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e000c090 con 0x7fd6e00051c0 2024-03-24T00:38:45.241 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.238+0000 7fd6f2ffd640 1 -- 172.21.15.67:0/2432634137 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e000c290 con 0x7fd6e0004800 2024-03-24T00:38:45.241 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.238+0000 7fd6f3fff640 1 --2- 172.21.15.67:0/2432634137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6e00039b0 0x7fd6e0003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:45.241 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.238+0000 7fd6f37fe640 1 --2- 172.21.15.67:0/2432634137 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd6e0004800 0x7fd6e0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:45.241 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.238+0000 7fd6f8b2d640 1 --2- 172.21.15.67:0/2432634137 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd6e00051c0 0x7fd6e000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:45.241 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.238+0000 7fd6f3fff640 1 -- 172.21.15.67:0/2432634137 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd6e0004800 msgr2=0x7fd6e0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:45.241 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.238+0000 7fd6f3fff640 1 --2- 172.21.15.67:0/2432634137 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd6e0004800 0x7fd6e0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:45.241 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.238+0000 7fd6f3fff640 1 -- 172.21.15.67:0/2432634137 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd6e00051c0 msgr2=0x7fd6e000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:45.241 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.238+0000 7fd6f3fff640 1 --2- 172.21.15.67:0/2432634137 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd6e00051c0 0x7fd6e000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:45.241 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.238+0000 7fd6f3fff640 1 -- 172.21.15.67:0/2432634137 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6e0003520 con 0x7fd6e00039b0 2024-03-24T00:38:45.241 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.238+0000 7fd6f8b2d640 1 -- 172.21.15.67:0/2432634137 reap_dead start 2024-03-24T00:38:45.241 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.239+0000 7fd6f3fff640 1 --2- 172.21.15.67:0/2432634137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6e00039b0 0x7fd6e0003df0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd6e400c240 tx=0x7fd6e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e636947e79e10b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:45.241 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.239+0000 7fd6f2ffd640 1 -- 172.21.15.67:0/2432634137 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e4007cf0 con 0x7fd6e00039b0 2024-03-24T00:38:45.242 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.239+0000 7fd6f2ffd640 1 -- 172.21.15.67:0/2432634137 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6e4030070 con 0x7fd6e00039b0 2024-03-24T00:38:45.242 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.239+0000 7fd6f2ffd640 1 -- 172.21.15.67:0/2432634137 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e4005d30 con 0x7fd6e00039b0 2024-03-24T00:38:45.242 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.240+0000 7fd6fa590640 1 -- 172.21.15.67:0/2432634137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6e00039b0 msgr2=0x7fd6e0003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:45.242 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.240+0000 7fd6fa590640 1 --2- 172.21.15.67:0/2432634137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6e00039b0 0x7fd6e0003df0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd6e400c240 tx=0x7fd6e0005830 comp rx=0 tx=0).stop 2024-03-24T00:38:45.242 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.240+0000 7fd6fa590640 1 -- 172.21.15.67:0/2432634137 shutdown_connections 2024-03-24T00:38:45.242 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.240+0000 7fd6fa590640 1 --2- 172.21.15.67:0/2432634137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6e00039b0 0x7fd6e0003df0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:45.242 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.240+0000 7fd6fa590640 1 -- 172.21.15.67:0/2432634137 >> 172.21.15.67:0/2432634137 conn(0x7fd6f4072bb0 msgr2=0x7fd6f4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:45.242 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.240+0000 7fd6fa590640 1 -- 172.21.15.67:0/2432634137 shutdown_connections 2024-03-24T00:38:45.242 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.240+0000 7fd6fa590640 1 -- 172.21.15.67:0/2432634137 wait complete. 2024-03-24T00:38:45.245 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.241+0000 7fd6fa590640 1 Processor -- start 2024-03-24T00:38:45.246 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.241+0000 7fd6fa590640 1 -- start start 2024-03-24T00:38:45.246 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.242+0000 7fd6fa590640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd6e00039b0 0x7fd6f4152af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:45.246 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.242+0000 7fd6fa590640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd6f40c8570 0x7fd6f415ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:45.246 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.242+0000 7fd6fa590640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6f40cc690 0x7fd6f40ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:45.246 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.242+0000 7fd6fa590640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f40ce7a0 con 0x7fd6f40cc690 2024-03-24T00:38:45.246 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.242+0000 7fd6fa590640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f415e4a0 con 0x7fd6e00039b0 2024-03-24T00:38:45.246 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.242+0000 7fd6fa590640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f40d1180 con 0x7fd6f40c8570 2024-03-24T00:38:45.246 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.242+0000 7fd6f8b2d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6f40cc690 0x7fd6f40ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:45.246 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.242+0000 7fd6f8b2d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6f40cc690 0x7fd6f40ccb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35844/0 (socket says 172.21.15.67:35844) 2024-03-24T00:38:45.246 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.242+0000 7fd6f8b2d640 1 -- 172.21.15.67:0/3443931639 learned_addr learned my addr 172.21.15.67:0/3443931639 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:38:45.246 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.242+0000 7fd6f37fe640 1 --2- 172.21.15.67:0/3443931639 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd6f40c8570 0x7fd6f415ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:45.247 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.242+0000 7fd6f3fff640 1 --2- 172.21.15.67:0/3443931639 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd6e00039b0 0x7fd6f4152af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:45.247 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.242+0000 7fd6f8b2d640 1 -- 172.21.15.67:0/3443931639 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd6f40c8570 msgr2=0x7fd6f415ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:45.247 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.242+0000 7fd6f8b2d640 1 --2- 172.21.15.67:0/3443931639 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd6f40c8570 0x7fd6f415ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:45.247 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.242+0000 7fd6f8b2d640 1 -- 172.21.15.67:0/3443931639 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd6e00039b0 msgr2=0x7fd6f4152af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:45.247 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.242+0000 7fd6f8b2d640 1 --2- 172.21.15.67:0/3443931639 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd6e00039b0 0x7fd6f4152af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:45.247 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.242+0000 7fd6f8b2d640 1 -- 172.21.15.67:0/3443931639 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6f4153030 con 0x7fd6f40cc690 2024-03-24T00:38:45.247 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.242+0000 7fd6f37fe640 1 --2- 172.21.15.67:0/3443931639 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd6f40c8570 0x7fd6f415ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:38:45.247 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.242+0000 7fd6f8b2d640 1 --2- 172.21.15.67:0/3443931639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6f40cc690 0x7fd6f40ccb10 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd6f4161b60 tx=0x7fd6ec00fbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:45.247 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.243+0000 7fd6f0ff9640 1 -- 172.21.15.67:0/3443931639 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6ec01a070 con 0x7fd6f40cc690 2024-03-24T00:38:45.247 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.243+0000 7fd6f0ff9640 1 -- 172.21.15.67:0/3443931639 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6ec006d60 con 0x7fd6f40cc690 2024-03-24T00:38:45.247 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.243+0000 7fd6fa590640 1 -- 172.21.15.67:0/3443931639 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6f415f640 con 0x7fd6f40cc690 2024-03-24T00:38:45.247 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.243+0000 7fd6f0ff9640 1 -- 172.21.15.67:0/3443931639 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6ec003bb0 con 0x7fd6f40cc690 2024-03-24T00:38:45.247 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.243+0000 7fd6fa590640 1 -- 172.21.15.67:0/3443931639 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6f415fc00 con 0x7fd6f40cc690 2024-03-24T00:38:45.247 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.244+0000 7fd6fa590640 1 -- 172.21.15.67:0/3443931639 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6f4068550 con 0x7fd6f40cc690 2024-03-24T00:38:45.247 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.244+0000 7fd6f0ff9640 1 -- 172.21.15.67:0/3443931639 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd6ec01d070 con 0x7fd6f40cc690 2024-03-24T00:38:45.248 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.245+0000 7fd6f0ff9640 1 --2- 172.21.15.67:0/3443931639 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd6d406bc00 0x7fd6d406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:45.248 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.245+0000 7fd6f0ff9640 1 -- 172.21.15.67:0/3443931639 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd6ec089c50 con 0x7fd6f40cc690 2024-03-24T00:38:45.248 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.245+0000 7fd6f3fff640 1 --2- 172.21.15.67:0/3443931639 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd6d406bc00 0x7fd6d406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:45.248 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.245+0000 7fd6f3fff640 1 --2- 172.21.15.67:0/3443931639 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd6d406bc00 0x7fd6d406e0c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd6e400c240 tx=0x7fd6e4039040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:45.249 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.250+0000 7fd6f0ff9640 1 -- 172.21.15.67:0/3443931639 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd6ec058a80 con 0x7fd6f40cc690 2024-03-24T00:38:45.431 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.432+0000 7fd6fa590640 1 -- 172.21.15.67:0/3443931639 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd6f404f470 con 0x7fd6f40cc690 2024-03-24T00:38:45.432 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.432+0000 7fd6f0ff9640 1 -- 172.21.15.67:0/3443931639 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12824 (secure 0 0 0) 0x7fd6ec081e20 con 0x7fd6f40cc690 2024-03-24T00:38:45.432 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-24T00:38:45.432 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":16,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:38:41.725986+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:38:45.434 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.434+0000 7fd6fa590640 1 -- 172.21.15.67:0/3443931639 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd6d406bc00 msgr2=0x7fd6d406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:45.434 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.434+0000 7fd6fa590640 1 --2- 172.21.15.67:0/3443931639 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd6d406bc00 0x7fd6d406e0c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd6e400c240 tx=0x7fd6e4039040 comp rx=0 tx=0).stop 2024-03-24T00:38:45.434 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.434+0000 7fd6fa590640 1 -- 172.21.15.67:0/3443931639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6f40cc690 msgr2=0x7fd6f40ccb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:45.434 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.434+0000 7fd6fa590640 1 --2- 172.21.15.67:0/3443931639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6f40cc690 0x7fd6f40ccb10 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd6f4161b60 tx=0x7fd6ec00fbe0 comp rx=0 tx=0).stop 2024-03-24T00:38:45.435 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.434+0000 7fd6fa590640 1 -- 172.21.15.67:0/3443931639 shutdown_connections 2024-03-24T00:38:45.435 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.434+0000 7fd6fa590640 1 --2- 172.21.15.67:0/3443931639 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd6d406bc00 0x7fd6d406e0c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:45.435 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.434+0000 7fd6fa590640 1 --2- 172.21.15.67:0/3443931639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6f40cc690 0x7fd6f40ccb10 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:45.435 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.434+0000 7fd6fa590640 1 --2- 172.21.15.67:0/3443931639 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd6f40c8570 0x7fd6f415ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:45.435 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.434+0000 7fd6fa590640 1 --2- 172.21.15.67:0/3443931639 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd6e00039b0 0x7fd6f4152af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:45.435 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.434+0000 7fd6fa590640 1 -- 172.21.15.67:0/3443931639 >> 172.21.15.67:0/3443931639 conn(0x7fd6f4072bb0 msgr2=0x7fd6f40b8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:45.435 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.434+0000 7fd6fa590640 1 -- 172.21.15.67:0/3443931639 shutdown_connections 2024-03-24T00:38:45.435 INFO:teuthology.misc.health.smithi067.stderr:2024-03-24T00:38:45.434+0000 7fd6fa590640 1 -- 172.21.15.67:0/3443931639 wait complete. 2024-03-24T00:38:45.444 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-24T00:38:45.444 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-24T00:38:45.444 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-24T00:38:45.444 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-24T00:38:45.447 INFO:tasks.ceph.mds.a:Started 2024-03-24T00:38:45.447 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-24T00:38:45.447 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-24T00:38:45.449 INFO:tasks.ceph.mds.c:Started 2024-03-24T00:38:45.449 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-24T00:38:45.449 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-24T00:38:45.452 INFO:tasks.ceph.mds.b:Started 2024-03-24T00:38:45.452 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-24T00:38:45.452 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-24T00:38:45.454 INFO:tasks.ceph.mds.d:Started 2024-03-24T00:38:45.454 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-03-24T00:38:45.455 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:38:45.522 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-03-24T00:38:45.523 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-03-24T00:38:45.536 INFO:tasks.ceph.mds.b.smithi072.stdout:starting mds.b at 2024-03-24T00:38:45.537 INFO:tasks.ceph.mds.d.smithi072.stdout:starting mds.d at 2024-03-24T00:38:45.537 INFO:tasks.ceph.mds.b.smithi072.stderr:2024-03-24T00:38:45.537+0000 7f1f273bb640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2024-03-24T00:38:45.538 INFO:tasks.ceph.mds.d.smithi072.stderr:2024-03-24T00:38:45.538+0000 7f4b40827640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6836/0: (98) Address already in use 2024-03-24T00:38:45.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.560+0000 7f7b0d79e640 1 Processor -- start 2024-03-24T00:38:45.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.560+0000 7f7b0d79e640 1 -- start start 2024-03-24T00:38:45.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.561+0000 7f7b0d79e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7b0815bcd0 0x7f7b080ce540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:45.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.561+0000 7f7b0d79e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b080cea80 con 0x7f7b080d0180 2024-03-24T00:38:45.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.561+0000 7f7b0d79e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b080ced20 con 0x7f7b080cf7b0 2024-03-24T00:38:45.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.561+0000 7f7b0d79e640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b080b1b00 con 0x7f7b0815bcd0 2024-03-24T00:38:45.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.561+0000 7f7b077fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7b080d0180 0x7f7b080b1f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60602/0 (socket says 172.21.15.67:60602) 2024-03-24T00:38:45.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.561+0000 7f7b077fe640 1 -- 172.21.15.67:0/2252449542 learned_addr learned my addr 172.21.15.67:0/2252449542 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:38:45.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.561+0000 7f7b06ffd640 1 --2- 172.21.15.67:0/2252449542 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7b0815bcd0 0x7f7b080ce540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:45.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.562+0000 7f7b067fc640 1 -- 172.21.15.67:0/2252449542 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1541180996 0 0) 0x7f7b080cea80 con 0x7f7b080d0180 2024-03-24T00:38:45.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.562+0000 7f7b067fc640 1 -- 172.21.15.67:0/2252449542 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7af4003520 con 0x7f7b080d0180 2024-03-24T00:38:45.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.562+0000 7f7b067fc640 1 -- 172.21.15.67:0/2252449542 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 122139672 0 0) 0x7f7b080ced20 con 0x7f7b080cf7b0 2024-03-24T00:38:45.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.562+0000 7f7b06ffd640 1 -- 172.21.15.67:0/2252449542 >> v1:172.21.15.72:6789/0 conn(0x7f7b080cf7b0 legacy=0x7f7b080cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:45.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.562+0000 7f7b06ffd640 1 -- 172.21.15.67:0/2252449542 >> v1:172.21.15.67:6789/0 conn(0x7f7b080d0180 legacy=0x7f7b080b1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:45.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.562+0000 7f7b06ffd640 1 -- 172.21.15.67:0/2252449542 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b080cf000 con 0x7f7b0815bcd0 2024-03-24T00:38:45.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.563+0000 7f7b06ffd640 1 --2- 172.21.15.67:0/2252449542 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7b0815bcd0 0x7f7b080ce540 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7af80097d0 tx=0x7f7af802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a08eebcf8c10c2c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:45.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.563+0000 7f7b067fc640 1 -- 172.21.15.67:0/2252449542 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af8035020 con 0x7f7b0815bcd0 2024-03-24T00:38:45.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.563+0000 7f7b067fc640 1 -- 172.21.15.67:0/2252449542 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7af8031890 con 0x7f7b0815bcd0 2024-03-24T00:38:45.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.563+0000 7f7b067fc640 1 -- 172.21.15.67:0/2252449542 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af8031c40 con 0x7f7b0815bcd0 2024-03-24T00:38:45.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.563+0000 7f7b0d79e640 1 -- 172.21.15.67:0/2252449542 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7b0815bcd0 msgr2=0x7f7b080ce540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:45.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.563+0000 7f7b0d79e640 1 --2- 172.21.15.67:0/2252449542 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7b0815bcd0 0x7f7b080ce540 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7af80097d0 tx=0x7f7af802b930 comp rx=0 tx=0).stop 2024-03-24T00:38:45.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.563+0000 7f7b0d79e640 1 -- 172.21.15.67:0/2252449542 shutdown_connections 2024-03-24T00:38:45.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.563+0000 7f7b0d79e640 1 --2- 172.21.15.67:0/2252449542 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7b0815bcd0 0x7f7b080ce540 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:45.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.563+0000 7f7b0d79e640 1 -- 172.21.15.67:0/2252449542 >> 172.21.15.67:0/2252449542 conn(0x7f7b0808b1b0 msgr2=0x7f7b0808b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:45.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.563+0000 7f7b0d79e640 1 -- 172.21.15.67:0/2252449542 shutdown_connections 2024-03-24T00:38:45.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.563+0000 7f7b0d79e640 1 -- 172.21.15.67:0/2252449542 wait complete. 2024-03-24T00:38:45.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.565+0000 7f7b0d79e640 1 Processor -- start 2024-03-24T00:38:45.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.565+0000 7f7b0d79e640 1 -- start start 2024-03-24T00:38:45.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.565+0000 7f7b0d79e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b080cf7b0 0x7f7b08148b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:45.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.565+0000 7f7b0d79e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b080d0180 0x7f7b08149090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:45.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.565+0000 7f7b0d79e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7b0815bcd0 0x7f7b080ae7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:45.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.565+0000 7f7b0d79e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0815c230 con 0x7f7b080cf7b0 2024-03-24T00:38:45.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.565+0000 7f7b0d79e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0815c0b0 con 0x7f7b080d0180 2024-03-24T00:38:45.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.565+0000 7f7b0d79e640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b080d0630 con 0x7f7b0815bcd0 2024-03-24T00:38:45.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.565+0000 7f7b06ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b080cf7b0 0x7f7b08148b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:45.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.565+0000 7f7b06ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b080cf7b0 0x7f7b08148b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35892/0 (socket says 172.21.15.67:35892) 2024-03-24T00:38:45.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.565+0000 7f7b06ffd640 1 -- 172.21.15.67:0/1261132157 learned_addr learned my addr 172.21.15.67:0/1261132157 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:38:45.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.565+0000 7f7b077fe640 1 --2- 172.21.15.67:0/1261132157 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7b0815bcd0 0x7f7b080ae7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:45.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.565+0000 7f7afffff640 1 --2- 172.21.15.67:0/1261132157 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b080d0180 0x7f7b08149090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:45.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.566+0000 7f7b06ffd640 1 -- 172.21.15.67:0/1261132157 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7b0815bcd0 msgr2=0x7f7b080ae7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:45.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.566+0000 7f7b06ffd640 1 --2- 172.21.15.67:0/1261132157 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7b0815bcd0 0x7f7b080ae7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:45.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.566+0000 7f7b06ffd640 1 -- 172.21.15.67:0/1261132157 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b080d0180 msgr2=0x7f7b08149090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:45.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.566+0000 7f7b06ffd640 1 --2- 172.21.15.67:0/1261132157 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b080d0180 0x7f7b08149090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:45.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.566+0000 7f7b06ffd640 1 -- 172.21.15.67:0/1261132157 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b080afa00 con 0x7f7b080cf7b0 2024-03-24T00:38:45.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.566+0000 7f7b06ffd640 1 --2- 172.21.15.67:0/1261132157 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b080cf7b0 0x7f7b08148b50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7af8000c00 tx=0x7f7af80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:45.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.566+0000 7f7aff7fe640 1 -- 172.21.15.67:0/1261132157 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af8040070 con 0x7f7b080cf7b0 2024-03-24T00:38:45.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.566+0000 7f7aff7fe640 1 -- 172.21.15.67:0/1261132157 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7af8030d20 con 0x7f7b080cf7b0 2024-03-24T00:38:45.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.566+0000 7f7b0d79e640 1 -- 172.21.15.67:0/1261132157 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b080afce0 con 0x7f7b080cf7b0 2024-03-24T00:38:45.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.566+0000 7f7aff7fe640 1 -- 172.21.15.67:0/1261132157 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af8035040 con 0x7f7b080cf7b0 2024-03-24T00:38:45.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.566+0000 7f7b0d79e640 1 -- 172.21.15.67:0/1261132157 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b080b02a0 con 0x7f7b080cf7b0 2024-03-24T00:38:45.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.567+0000 7f7b0d79e640 1 -- 172.21.15.67:0/1261132157 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b0804f400 con 0x7f7b080cf7b0 2024-03-24T00:38:45.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.568+0000 7f7aff7fe640 1 -- 172.21.15.67:0/1261132157 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7af804b060 con 0x7f7b080cf7b0 2024-03-24T00:38:45.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.568+0000 7f7aff7fe640 1 --2- 172.21.15.67:0/1261132157 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7ae806bbd0 0x7f7ae806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:45.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.568+0000 7f7aff7fe640 1 -- 172.21.15.67:0/1261132157 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7af80ac3e0 con 0x7f7b080cf7b0 2024-03-24T00:38:45.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.568+0000 7f7afffff640 1 --2- 172.21.15.67:0/1261132157 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7ae806bbd0 0x7f7ae806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:45.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.569+0000 7f7afffff640 1 --2- 172.21.15.67:0/1261132157 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7ae806bbd0 0x7f7ae806e090 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7b08156c50 tx=0x7f7af0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:45.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.572+0000 7f7aff7fe640 1 -- 172.21.15.67:0/1261132157 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7af802d040 con 0x7f7b080cf7b0 2024-03-24T00:38:45.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.757+0000 7f7b0d79e640 1 -- 172.21.15.67:0/1261132157 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b080cbf80 con 0x7f7b080cf7b0 2024-03-24T00:38:45.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.758+0000 7f7aff7fe640 1 -- 172.21.15.67:0/1261132157 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12824 (secure 0 0 0) 0x7f7af807ac60 con 0x7f7b080cf7b0 2024-03-24T00:38:45.758 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:38:45.758 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":16,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:38:41.725986+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:38:45.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.760+0000 7f7b0d79e640 1 -- 172.21.15.67:0/1261132157 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7ae806bbd0 msgr2=0x7f7ae806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:45.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.760+0000 7f7b0d79e640 1 --2- 172.21.15.67:0/1261132157 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7ae806bbd0 0x7f7ae806e090 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7b08156c50 tx=0x7f7af0009040 comp rx=0 tx=0).stop 2024-03-24T00:38:45.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.760+0000 7f7b0d79e640 1 -- 172.21.15.67:0/1261132157 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b080cf7b0 msgr2=0x7f7b08148b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:45.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.760+0000 7f7b0d79e640 1 --2- 172.21.15.67:0/1261132157 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b080cf7b0 0x7f7b08148b50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7af8000c00 tx=0x7f7af80046c0 comp rx=0 tx=0).stop 2024-03-24T00:38:45.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.760+0000 7f7b0d79e640 1 -- 172.21.15.67:0/1261132157 shutdown_connections 2024-03-24T00:38:45.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.760+0000 7f7b0d79e640 1 --2- 172.21.15.67:0/1261132157 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7ae806bbd0 0x7f7ae806e090 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:45.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.760+0000 7f7b0d79e640 1 --2- 172.21.15.67:0/1261132157 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7b0815bcd0 0x7f7b080ae7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:45.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.760+0000 7f7b0d79e640 1 --2- 172.21.15.67:0/1261132157 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b080d0180 0x7f7b08149090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:45.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.760+0000 7f7b0d79e640 1 --2- 172.21.15.67:0/1261132157 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b080cf7b0 0x7f7b08148b50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:45.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.760+0000 7f7b0d79e640 1 -- 172.21.15.67:0/1261132157 >> 172.21.15.67:0/1261132157 conn(0x7f7b0808b1b0 msgr2=0x7f7b080d1d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:45.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.760+0000 7f7b0d79e640 1 -- 172.21.15.67:0/1261132157 shutdown_connections 2024-03-24T00:38:45.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.760+0000 7f7b0d79e640 1 -- 172.21.15.67:0/1261132157 wait complete. 2024-03-24T00:38:45.770 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:38:39.364950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:38:45.771 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:38:45.780 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:38:45.816 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:38:45.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.875+0000 7f1070f7e640 1 Processor -- start 2024-03-24T00:38:45.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.875+0000 7f1070f7e640 1 -- start start 2024-03-24T00:38:45.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.876+0000 7f1070f7e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f106c15bc30 0x7f106c0b1a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:45.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.876+0000 7f1070f7e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f106c15c010 0x7f106c0b1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:45.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.876+0000 7f1070f7e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f106c0b2d90 0x7f106c0b3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:45.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.876+0000 7f1070f7e640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f106c0cca60 con 0x7f106c15c010 2024-03-24T00:38:45.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.876+0000 7f1070f7e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f106c0b3ae0 con 0x7f106c0b2d90 2024-03-24T00:38:45.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.876+0000 7f1070f7e640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f106c0b3dc0 con 0x7f106c15bc30 2024-03-24T00:38:45.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.876+0000 7f1069d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f106c15c010 0x7f106c0b1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:45.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.876+0000 7f1069d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f106c15c010 0x7f106c0b1f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35906/0 (socket says 172.21.15.67:35906) 2024-03-24T00:38:45.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.876+0000 7f1069d9b640 1 -- 172.21.15.67:0/3638627634 learned_addr learned my addr 172.21.15.67:0/3638627634 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:38:45.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.876+0000 7f106ad9d640 1 --2- 172.21.15.67:0/3638627634 >> v2:172.21.15.72:3300/0 conn(0x7f106c0b2d90 0x7f106c0b3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:45.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.876+0000 7f106a59c640 1 --2- 172.21.15.67:0/3638627634 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f106c15bc30 0x7f106c0b1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:45.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.876+0000 7f1069d9b640 1 -- 172.21.15.67:0/3638627634 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f106c15bc30 msgr2=0x7f106c0b1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:45.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.876+0000 7f1069d9b640 1 --2- 172.21.15.67:0/3638627634 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f106c15bc30 0x7f106c0b1a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:45.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.876+0000 7f1069d9b640 1 -- 172.21.15.67:0/3638627634 >> v2:172.21.15.72:3300/0 conn(0x7f106c0b2d90 msgr2=0x7f106c0b3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:45.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.876+0000 7f1069d9b640 1 --2- 172.21.15.67:0/3638627634 >> v2:172.21.15.72:3300/0 conn(0x7f106c0b2d90 0x7f106c0b3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:45.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.876+0000 7f1069d9b640 1 -- 172.21.15.67:0/3638627634 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f106c0b40a0 con 0x7f106c15c010 2024-03-24T00:38:45.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.877+0000 7f1069d9b640 1 --2- 172.21.15.67:0/3638627634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f106c15c010 0x7f106c0b1f40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f106c1609c0 tx=0x7f105402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68e3556e266fafa1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:45.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.877+0000 7f106959a640 1 -- 172.21.15.67:0/3638627634 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1054035020 con 0x7f106c15c010 2024-03-24T00:38:45.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.877+0000 7f106959a640 1 -- 172.21.15.67:0/3638627634 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1054031870 con 0x7f106c15c010 2024-03-24T00:38:45.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.877+0000 7f106959a640 1 -- 172.21.15.67:0/3638627634 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1054031c40 con 0x7f106c15c010 2024-03-24T00:38:45.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.878+0000 7f1070f7e640 1 -- 172.21.15.67:0/3638627634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f106c15c010 msgr2=0x7f106c0b1f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:45.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.878+0000 7f1070f7e640 1 --2- 172.21.15.67:0/3638627634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f106c15c010 0x7f106c0b1f40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f106c1609c0 tx=0x7f105402b910 comp rx=0 tx=0).stop 2024-03-24T00:38:45.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.878+0000 7f1070f7e640 1 -- 172.21.15.67:0/3638627634 shutdown_connections 2024-03-24T00:38:45.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.878+0000 7f1070f7e640 1 --2- 172.21.15.67:0/3638627634 >> v2:172.21.15.72:3300/0 conn(0x7f106c0b2d90 0x7f106c0b3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:45.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.878+0000 7f1070f7e640 1 --2- 172.21.15.67:0/3638627634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f106c15c010 0x7f106c0b1f40 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:45.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.878+0000 7f1070f7e640 1 --2- 172.21.15.67:0/3638627634 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f106c15bc30 0x7f106c0b1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:45.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.878+0000 7f1070f7e640 1 -- 172.21.15.67:0/3638627634 >> 172.21.15.67:0/3638627634 conn(0x7f106c08b180 msgr2=0x7f106c08b590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:45.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.878+0000 7f1070f7e640 1 -- 172.21.15.67:0/3638627634 shutdown_connections 2024-03-24T00:38:45.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.878+0000 7f1070f7e640 1 -- 172.21.15.67:0/3638627634 wait complete. 2024-03-24T00:38:45.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.879+0000 7f1070f7e640 1 Processor -- start 2024-03-24T00:38:45.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.879+0000 7f1070f7e640 1 -- start start 2024-03-24T00:38:45.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.880+0000 7f1070f7e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f106c0b2d90 0x7f106c154420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:45.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.880+0000 7f1070f7e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f106c15bc30 0x7f106c15c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:45.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.880+0000 7f1070f7e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f106c15c010 0x7f106c0b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:45.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.880+0000 7f1070f7e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106c0cfe70 con 0x7f106c0b2d90 2024-03-24T00:38:45.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.880+0000 7f1070f7e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106c0b3760 con 0x7f106c15bc30 2024-03-24T00:38:45.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.880+0000 7f1070f7e640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f106c0cfff0 con 0x7f106c15c010 2024-03-24T00:38:45.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.880+0000 7f106a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f106c0b2d90 0x7f106c154420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:45.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.880+0000 7f106a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f106c0b2d90 0x7f106c154420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35916/0 (socket says 172.21.15.67:35916) 2024-03-24T00:38:45.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.880+0000 7f106a59c640 1 -- 172.21.15.67:0/3602012898 learned_addr learned my addr 172.21.15.67:0/3602012898 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:38:45.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.880+0000 7f106ad9d640 1 --2- 172.21.15.67:0/3602012898 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f106c15c010 0x7f106c0b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:45.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.880+0000 7f1069d9b640 1 --2- 172.21.15.67:0/3602012898 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f106c15bc30 0x7f106c15c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:45.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.880+0000 7f106a59c640 1 -- 172.21.15.67:0/3602012898 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f106c15c010 msgr2=0x7f106c0b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:45.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.880+0000 7f106a59c640 1 --2- 172.21.15.67:0/3602012898 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f106c15c010 0x7f106c0b01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:45.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.880+0000 7f106a59c640 1 -- 172.21.15.67:0/3602012898 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f106c15bc30 msgr2=0x7f106c15c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:45.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.880+0000 7f106a59c640 1 --2- 172.21.15.67:0/3602012898 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f106c15bc30 0x7f106c15c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:45.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.880+0000 7f106a59c640 1 -- 172.21.15.67:0/3602012898 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f106c154960 con 0x7f106c0b2d90 2024-03-24T00:38:45.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.880+0000 7f106a59c640 1 --2- 172.21.15.67:0/3602012898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f106c0b2d90 0x7f106c154420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1064007ef0 tx=0x7f106400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:45.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.881+0000 7f1062ffd640 1 -- 172.21.15.67:0/3602012898 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1064014620 con 0x7f106c0b2d90 2024-03-24T00:38:45.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.881+0000 7f1062ffd640 1 -- 172.21.15.67:0/3602012898 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1064019070 con 0x7f106c0b2d90 2024-03-24T00:38:45.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.881+0000 7f1070f7e640 1 -- 172.21.15.67:0/3602012898 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f106c0afc80 con 0x7f106c0b2d90 2024-03-24T00:38:45.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.881+0000 7f1062ffd640 1 -- 172.21.15.67:0/3602012898 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1064014a00 con 0x7f106c0b2d90 2024-03-24T00:38:45.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.881+0000 7f1070f7e640 1 -- 172.21.15.67:0/3602012898 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f106c0ca8e0 con 0x7f106c0b2d90 2024-03-24T00:38:45.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.882+0000 7f1070f7e640 1 -- 172.21.15.67:0/3602012898 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f106c04f400 con 0x7f106c0b2d90 2024-03-24T00:38:45.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.882+0000 7f1062ffd640 1 -- 172.21.15.67:0/3602012898 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1064014ba0 con 0x7f106c0b2d90 2024-03-24T00:38:45.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.883+0000 7f1062ffd640 1 --2- 172.21.15.67:0/3602012898 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f104c06bbd0 0x7f104c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:45.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.883+0000 7f1062ffd640 1 -- 172.21.15.67:0/3602012898 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f106408adf0 con 0x7f106c0b2d90 2024-03-24T00:38:45.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.883+0000 7f1069d9b640 1 --2- 172.21.15.67:0/3602012898 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f104c06bbd0 0x7f104c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:45.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.883+0000 7f1069d9b640 1 --2- 172.21.15.67:0/3602012898 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f104c06bbd0 0x7f104c06e090 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1054036000 tx=0x7f105403c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:45.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:45.887+0000 7f1062ffd640 1 -- 172.21.15.67:0/3602012898 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1064059670 con 0x7f106c0b2d90 2024-03-24T00:38:46.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.068+0000 7f1070f7e640 1 -- 172.21.15.67:0/3602012898 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f106c15fb90 con 0x7f106c0b2d90 2024-03-24T00:38:46.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.068+0000 7f1062ffd640 1 -- 172.21.15.67:0/3602012898 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f106405d710 con 0x7f106c0b2d90 2024-03-24T00:38:46.068 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-24T00:38:46.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.070+0000 7f1070f7e640 1 -- 172.21.15.67:0/3602012898 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f104c06bbd0 msgr2=0x7f104c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:46.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.070+0000 7f1070f7e640 1 --2- 172.21.15.67:0/3602012898 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f104c06bbd0 0x7f104c06e090 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1054036000 tx=0x7f105403c040 comp rx=0 tx=0).stop 2024-03-24T00:38:46.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.070+0000 7f1070f7e640 1 -- 172.21.15.67:0/3602012898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f106c0b2d90 msgr2=0x7f106c154420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:46.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.070+0000 7f1070f7e640 1 --2- 172.21.15.67:0/3602012898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f106c0b2d90 0x7f106c154420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1064007ef0 tx=0x7f106400e4b0 comp rx=0 tx=0).stop 2024-03-24T00:38:46.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.070+0000 7f1070f7e640 1 -- 172.21.15.67:0/3602012898 shutdown_connections 2024-03-24T00:38:46.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.070+0000 7f1070f7e640 1 --2- 172.21.15.67:0/3602012898 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f104c06bbd0 0x7f104c06e090 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:46.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.070+0000 7f1070f7e640 1 --2- 172.21.15.67:0/3602012898 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f106c15c010 0x7f106c0b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:46.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.070+0000 7f1070f7e640 1 --2- 172.21.15.67:0/3602012898 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f106c15bc30 0x7f106c15c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:46.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.070+0000 7f1070f7e640 1 --2- 172.21.15.67:0/3602012898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f106c0b2d90 0x7f106c154420 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:46.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.070+0000 7f1070f7e640 1 -- 172.21.15.67:0/3602012898 >> 172.21.15.67:0/3602012898 conn(0x7f106c08b180 msgr2=0x7f106c0b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:46.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.070+0000 7f1070f7e640 1 -- 172.21.15.67:0/3602012898 shutdown_connections 2024-03-24T00:38:46.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.070+0000 7f1070f7e640 1 -- 172.21.15.67:0/3602012898 wait complete. 2024-03-24T00:38:46.080 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:38:46.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.222+0000 7fa9a15b3640 1 Processor -- start 2024-03-24T00:38:46.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.222+0000 7fa9a15b3640 1 -- start start 2024-03-24T00:38:46.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.223+0000 7fa9a15b3640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa99c1445a0 0x7fa99c0cee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:46.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.223+0000 7fa9a15b3640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa99c0b4d30 con 0x7fa99c06f350 2024-03-24T00:38:46.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.223+0000 7fa9a15b3640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa99c0cddc0 con 0x7fa99c0b48d0 2024-03-24T00:38:46.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.223+0000 7fa9a15b3640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa99c0ce990 con 0x7fa99c1445a0 2024-03-24T00:38:46.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.223+0000 7fa99affd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa99c1445a0 0x7fa99c0cee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:46.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.223+0000 7fa99b7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa99c06f350 0x7fa99c0b2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60610/0 (socket says 172.21.15.67:60610) 2024-03-24T00:38:46.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.223+0000 7fa99b7fe640 1 -- 172.21.15.67:0/3624208427 learned_addr learned my addr 172.21.15.67:0/3624208427 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:38:46.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.223+0000 7fa999ffb640 1 -- 172.21.15.67:0/3624208427 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 124457824 0 0) 0x7fa994001490 con 0x7fa99c06f350 2024-03-24T00:38:46.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.223+0000 7fa999ffb640 1 -- 172.21.15.67:0/3624208427 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa984005730 con 0x7fa99c06f350 2024-03-24T00:38:46.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.223+0000 7fa99affd640 1 -- 172.21.15.67:0/3624208427 >> v1:172.21.15.72:6789/0 conn(0x7fa99c0b48d0 legacy=0x7fa99c06ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:46.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.223+0000 7fa99affd640 1 -- 172.21.15.67:0/3624208427 >> v1:172.21.15.67:6789/0 conn(0x7fa99c06f350 legacy=0x7fa99c0b2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:46.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.223+0000 7fa999ffb640 1 -- 172.21.15.67:0/3624208427 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3191994658 0 0) 0x7fa99c0cddc0 con 0x7fa99c0b48d0 2024-03-24T00:38:46.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.223+0000 7fa99affd640 1 -- 172.21.15.67:0/3624208427 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa99c0ce060 con 0x7fa99c1445a0 2024-03-24T00:38:46.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.224+0000 7fa99affd640 1 --2- 172.21.15.67:0/3624208427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa99c1445a0 0x7fa99c0cee20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa98c0097a0 tx=0x7fa98c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10e0aa9ed84101ed server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:46.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.224+0000 7fa999ffb640 1 -- 172.21.15.67:0/3624208427 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98c034070 con 0x7fa99c1445a0 2024-03-24T00:38:46.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.224+0000 7fa999ffb640 1 -- 172.21.15.67:0/3624208427 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa98c030610 con 0x7fa99c1445a0 2024-03-24T00:38:46.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.224+0000 7fa999ffb640 1 -- 172.21.15.67:0/3624208427 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98c0309e0 con 0x7fa99c1445a0 2024-03-24T00:38:46.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.225+0000 7fa9a15b3640 1 -- 172.21.15.67:0/3624208427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa99c1445a0 msgr2=0x7fa99c0cee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:46.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.225+0000 7fa9a15b3640 1 --2- 172.21.15.67:0/3624208427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa99c1445a0 0x7fa99c0cee20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa98c0097a0 tx=0x7fa98c02b960 comp rx=0 tx=0).stop 2024-03-24T00:38:46.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.225+0000 7fa9a15b3640 1 -- 172.21.15.67:0/3624208427 shutdown_connections 2024-03-24T00:38:46.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.225+0000 7fa9a15b3640 1 --2- 172.21.15.67:0/3624208427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa99c1445a0 0x7fa99c0cee20 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:46.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.225+0000 7fa9a15b3640 1 -- 172.21.15.67:0/3624208427 >> 172.21.15.67:0/3624208427 conn(0x7fa99c0cf9e0 msgr2=0x7fa99c0cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:46.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.225+0000 7fa9a15b3640 1 -- 172.21.15.67:0/3624208427 shutdown_connections 2024-03-24T00:38:46.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.225+0000 7fa9a15b3640 1 -- 172.21.15.67:0/3624208427 wait complete. 2024-03-24T00:38:46.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.226+0000 7fa9a15b3640 1 Processor -- start 2024-03-24T00:38:46.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.226+0000 7fa9a15b3640 1 -- start start 2024-03-24T00:38:46.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.227+0000 7fa9a15b3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa99c06f350 0x7fa99c135eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:46.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.227+0000 7fa9a15b3640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa99c0b48d0 0x7fa99c138ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:46.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.227+0000 7fa9a15b3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa99c1445a0 0x7fa99c14dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:46.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.227+0000 7fa9a15b3640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa99c072550 con 0x7fa99c1445a0 2024-03-24T00:38:46.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.227+0000 7fa9a15b3640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa99c0ce620 con 0x7fa99c06f350 2024-03-24T00:38:46.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.227+0000 7fa9a15b3640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa99c0726d0 con 0x7fa99c0b48d0 2024-03-24T00:38:46.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.227+0000 7fa99b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa99c1445a0 0x7fa99c14dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:46.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.227+0000 7fa99b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa99c1445a0 0x7fa99c14dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35934/0 (socket says 172.21.15.67:35934) 2024-03-24T00:38:46.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.227+0000 7fa99b7fe640 1 -- 172.21.15.67:0/1351274751 learned_addr learned my addr 172.21.15.67:0/1351274751 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:38:46.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.227+0000 7fa99b7fe640 1 -- 172.21.15.67:0/1351274751 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa99c0b48d0 msgr2=0x7fa99c138ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:46.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.227+0000 7fa99b7fe640 1 --2- 172.21.15.67:0/1351274751 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa99c0b48d0 0x7fa99c138ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:46.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.227+0000 7fa99b7fe640 1 -- 172.21.15.67:0/1351274751 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa99c06f350 msgr2=0x7fa99c135eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:46.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.227+0000 7fa99affd640 1 --2- 172.21.15.67:0/1351274751 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa99c06f350 0x7fa99c135eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:46.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.227+0000 7fa99b7fe640 1 --2- 172.21.15.67:0/1351274751 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa99c06f350 0x7fa99c135eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:46.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.227+0000 7fa99b7fe640 1 -- 172.21.15.67:0/1351274751 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa99c136540 con 0x7fa99c1445a0 2024-03-24T00:38:46.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.227+0000 7fa99b7fe640 1 --2- 172.21.15.67:0/1351274751 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa99c1445a0 0x7fa99c14dce0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa99400cb20 tx=0x7fa99400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:46.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.228+0000 7fa9837fe640 1 -- 172.21.15.67:0/1351274751 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa994017070 con 0x7fa99c1445a0 2024-03-24T00:38:46.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.228+0000 7fa9837fe640 1 -- 172.21.15.67:0/1351274751 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa99400aca0 con 0x7fa99c1445a0 2024-03-24T00:38:46.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.228+0000 7fa9a15b3640 1 -- 172.21.15.67:0/1351274751 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa99c141c90 con 0x7fa99c1445a0 2024-03-24T00:38:46.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.228+0000 7fa9837fe640 1 -- 172.21.15.67:0/1351274751 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa994012600 con 0x7fa99c1445a0 2024-03-24T00:38:46.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.228+0000 7fa9a15b3640 1 -- 172.21.15.67:0/1351274751 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa99c142250 con 0x7fa99c1445a0 2024-03-24T00:38:46.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.229+0000 7fa9a15b3640 1 -- 172.21.15.67:0/1351274751 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa99c143d50 con 0x7fa99c1445a0 2024-03-24T00:38:46.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.229+0000 7fa9837fe640 1 -- 172.21.15.67:0/1351274751 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa99401a070 con 0x7fa99c1445a0 2024-03-24T00:38:46.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.229+0000 7fa9837fe640 1 --2- 172.21.15.67:0/1351274751 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa98806bbd0 0x7fa98806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:46.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.229+0000 7fa9837fe640 1 -- 172.21.15.67:0/1351274751 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa99408a520 con 0x7fa99c1445a0 2024-03-24T00:38:46.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.230+0000 7fa99affd640 1 --2- 172.21.15.67:0/1351274751 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa98806bbd0 0x7fa98806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:46.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.230+0000 7fa99affd640 1 --2- 172.21.15.67:0/1351274751 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa98806bbd0 0x7fa98806e090 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa98c036000 tx=0x7fa98c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:46.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.234+0000 7fa9837fe640 1 -- 172.21.15.67:0/1351274751 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa99c143d50 con 0x7fa99c1445a0 2024-03-24T00:38:46.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.452+0000 7fa9a15b3640 1 -- 172.21.15.67:0/1351274751 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa99c04f400 con 0x7fa99c1445a0 2024-03-24T00:38:46.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.452+0000 7fa9837fe640 1 -- 172.21.15.67:0/1351274751 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1321 (secure 0 0 0) 0x7fa99c0b4d30 con 0x7fa99c1445a0 2024-03-24T00:38:46.452 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:38:46.452 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4263,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/837528243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":837528243},{"type":"v1","addr":"172.21.15.67:6834","nonce":837528243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-24T00:38:46.452 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 2 2024-03-24T00:38:46.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.454+0000 7fa9a15b3640 1 -- 172.21.15.67:0/1351274751 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa98806bbd0 msgr2=0x7fa98806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:46.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.454+0000 7fa9a15b3640 1 --2- 172.21.15.67:0/1351274751 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa98806bbd0 0x7fa98806e090 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa98c036000 tx=0x7fa98c008030 comp rx=0 tx=0).stop 2024-03-24T00:38:46.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.454+0000 7fa9a15b3640 1 -- 172.21.15.67:0/1351274751 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa99c1445a0 msgr2=0x7fa99c14dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:46.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.454+0000 7fa9a15b3640 1 --2- 172.21.15.67:0/1351274751 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa99c1445a0 0x7fa99c14dce0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa99400cb20 tx=0x7fa99400a060 comp rx=0 tx=0).stop 2024-03-24T00:38:46.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.454+0000 7fa9a15b3640 1 -- 172.21.15.67:0/1351274751 shutdown_connections 2024-03-24T00:38:46.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.454+0000 7fa9a15b3640 1 --2- 172.21.15.67:0/1351274751 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa98806bbd0 0x7fa98806e090 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:46.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.454+0000 7fa9a15b3640 1 --2- 172.21.15.67:0/1351274751 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa99c1445a0 0x7fa99c14dce0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:46.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.454+0000 7fa9a15b3640 1 --2- 172.21.15.67:0/1351274751 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa99c0b48d0 0x7fa99c138ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:46.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.454+0000 7fa9a15b3640 1 --2- 172.21.15.67:0/1351274751 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa99c06f350 0x7fa99c135eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:46.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.454+0000 7fa9a15b3640 1 -- 172.21.15.67:0/1351274751 >> 172.21.15.67:0/1351274751 conn(0x7fa99c0cf9e0 msgr2=0x7fa99c0b0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:46.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.454+0000 7fa9a15b3640 1 -- 172.21.15.67:0/1351274751 shutdown_connections 2024-03-24T00:38:46.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:46.454+0000 7fa9a15b3640 1 -- 172.21.15.67:0/1351274751 wait complete. 2024-03-24T00:38:48.465 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:38:48.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.579+0000 7f7236c4b640 1 Processor -- start 2024-03-24T00:38:48.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.579+0000 7f7236c4b640 1 -- start start 2024-03-24T00:38:48.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.580+0000 7f7236c4b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f723006cee0 0x7f7230099d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:48.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.580+0000 7f7236c4b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72300d1930 con 0x7f72300d1550 2024-03-24T00:38:48.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.580+0000 7f7236c4b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f723006d390 con 0x7f723009a2b0 2024-03-24T00:38:48.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.580+0000 7f7236c4b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f723009a700 con 0x7f723006cee0 2024-03-24T00:38:48.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.580+0000 7f722ffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f723006cee0 0x7f7230099d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:48.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.580+0000 7f72349e7640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f72300d1550 0x7f72300b2780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60618/0 (socket says 172.21.15.67:60618) 2024-03-24T00:38:48.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.580+0000 7f72349e7640 1 -- 172.21.15.67:0/4207355992 learned_addr learned my addr 172.21.15.67:0/4207355992 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:38:48.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.580+0000 7f722ffff640 1 -- 172.21.15.67:0/4207355992 >> v1:172.21.15.72:6789/0 conn(0x7f723009a2b0 legacy=0x7f72300cfd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:48.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.580+0000 7f722ffff640 1 -- 172.21.15.67:0/4207355992 >> v1:172.21.15.67:6789/0 conn(0x7f72300d1550 legacy=0x7f72300b2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:48.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.580+0000 7f722ffff640 1 -- 172.21.15.67:0/4207355992 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72300b1f60 con 0x7f723006cee0 2024-03-24T00:38:48.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.581+0000 7f722ffff640 1 --2- 172.21.15.67:0/4207355992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f723006cee0 0x7f7230099d70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f72180097d0 tx=0x7f721802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23e55ccabec91a27 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:48.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.581+0000 7f722f7fe640 1 -- 172.21.15.67:0/4207355992 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7218034070 con 0x7f723006cee0 2024-03-24T00:38:48.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.581+0000 7f722f7fe640 1 -- 172.21.15.67:0/4207355992 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7218030610 con 0x7f723006cee0 2024-03-24T00:38:48.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.581+0000 7f722f7fe640 1 -- 172.21.15.67:0/4207355992 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72180309e0 con 0x7f723006cee0 2024-03-24T00:38:48.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.582+0000 7f7236c4b640 1 -- 172.21.15.67:0/4207355992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f723006cee0 msgr2=0x7f7230099d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:48.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.582+0000 7f7236c4b640 1 --2- 172.21.15.67:0/4207355992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f723006cee0 0x7f7230099d70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f72180097d0 tx=0x7f721802b960 comp rx=0 tx=0).stop 2024-03-24T00:38:48.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.582+0000 7f7236c4b640 1 -- 172.21.15.67:0/4207355992 shutdown_connections 2024-03-24T00:38:48.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.582+0000 7f7236c4b640 1 --2- 172.21.15.67:0/4207355992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f723006cee0 0x7f7230099d70 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:48.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.582+0000 7f7236c4b640 1 -- 172.21.15.67:0/4207355992 >> 172.21.15.67:0/4207355992 conn(0x7f72300b36e0 msgr2=0x7f72300b3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:48.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.582+0000 7f7236c4b640 1 -- 172.21.15.67:0/4207355992 shutdown_connections 2024-03-24T00:38:48.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.582+0000 7f7236c4b640 1 -- 172.21.15.67:0/4207355992 wait complete. 2024-03-24T00:38:48.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.583+0000 7f7236c4b640 1 Processor -- start 2024-03-24T00:38:48.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.583+0000 7f7236c4b640 1 -- start start 2024-03-24T00:38:48.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.584+0000 7f7236c4b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f723006cee0 0x7f723015b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:48.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.584+0000 7f7236c4b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f723009a2b0 0x7f7230160a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:48.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.584+0000 7f7236c4b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f72300d1550 0x7f7230160f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:48.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.584+0000 7f7236c4b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72300b2520 con 0x7f723006cee0 2024-03-24T00:38:48.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.584+0000 7f7236c4b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f723006ca50 con 0x7f723009a2b0 2024-03-24T00:38:48.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.584+0000 7f7236c4b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f723006c400 con 0x7f72300d1550 2024-03-24T00:38:48.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.584+0000 7f72349e7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f723006cee0 0x7f723015b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:48.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.584+0000 7f72349e7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f723006cee0 0x7f723015b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35950/0 (socket says 172.21.15.67:35950) 2024-03-24T00:38:48.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.584+0000 7f72349e7640 1 -- 172.21.15.67:0/3463257830 learned_addr learned my addr 172.21.15.67:0/3463257830 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:38:48.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.584+0000 7f722ffff640 1 --2- 172.21.15.67:0/3463257830 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f723009a2b0 0x7f7230160a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:48.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.584+0000 7f72351e8640 1 --2- 172.21.15.67:0/3463257830 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f72300d1550 0x7f7230160f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:48.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.584+0000 7f72349e7640 1 -- 172.21.15.67:0/3463257830 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f72300d1550 msgr2=0x7f7230160f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:48.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.584+0000 7f72349e7640 1 --2- 172.21.15.67:0/3463257830 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f72300d1550 0x7f7230160f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:48.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.584+0000 7f72349e7640 1 -- 172.21.15.67:0/3463257830 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f723009a2b0 msgr2=0x7f7230160a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:48.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.584+0000 7f72349e7640 1 --2- 172.21.15.67:0/3463257830 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f723009a2b0 0x7f7230160a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:48.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.584+0000 7f72349e7640 1 -- 172.21.15.67:0/3463257830 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72300b4990 con 0x7f723006cee0 2024-03-24T00:38:48.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.584+0000 7f72349e7640 1 --2- 172.21.15.67:0/3463257830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f723006cee0 0x7f723015b600 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f722000c480 tx=0x7f722000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:48.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.585+0000 7f722d7fa640 1 -- 172.21.15.67:0/3463257830 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7220017070 con 0x7f723006cee0 2024-03-24T00:38:48.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.585+0000 7f722d7fa640 1 -- 172.21.15.67:0/3463257830 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7220005ce0 con 0x7f723006cee0 2024-03-24T00:38:48.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.585+0000 7f7236c4b640 1 -- 172.21.15.67:0/3463257830 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72300b4c70 con 0x7f723006cee0 2024-03-24T00:38:48.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.585+0000 7f722d7fa640 1 -- 172.21.15.67:0/3463257830 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72200072f0 con 0x7f723006cee0 2024-03-24T00:38:48.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.585+0000 7f7236c4b640 1 -- 172.21.15.67:0/3463257830 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72300b5230 con 0x7f723006cee0 2024-03-24T00:38:48.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.586+0000 7f7236c4b640 1 -- 172.21.15.67:0/3463257830 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f723004f400 con 0x7f723006cee0 2024-03-24T00:38:48.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.586+0000 7f722d7fa640 1 -- 172.21.15.67:0/3463257830 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f722001a070 con 0x7f723006cee0 2024-03-24T00:38:48.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.586+0000 7f722d7fa640 1 --2- 172.21.15.67:0/3463257830 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f721006bbd0 0x7f721006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:48.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.587+0000 7f722d7fa640 1 -- 172.21.15.67:0/3463257830 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7220089940 con 0x7f723006cee0 2024-03-24T00:38:48.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.587+0000 7f722ffff640 1 --2- 172.21.15.67:0/3463257830 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f721006bbd0 0x7f721006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:48.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.587+0000 7f722ffff640 1 --2- 172.21.15.67:0/3463257830 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f721006bbd0 0x7f721006e090 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7218036000 tx=0x7f721803b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:48.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.591+0000 7f722d7fa640 1 -- 172.21.15.67:0/3463257830 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f72200581c0 con 0x7f723006cee0 2024-03-24T00:38:48.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.815+0000 7f7236c4b640 1 -- 172.21.15.67:0/3463257830 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72300cc000 con 0x7f723006cee0 2024-03-24T00:38:48.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.816+0000 7f722d7fa640 1 -- 172.21.15.67:0/3463257830 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3691 (secure 0 0 0) 0x7f722005c260 con 0x7f723006cee0 2024-03-24T00:38:48.816 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:38:48.816 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4263,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/837528243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":837528243},{"type":"v1","addr":"172.21.15.67:6834","nonce":837528243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3987071456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":3987071456},{"type":"v1","addr":"172.21.15.67:6835","nonce":3987071456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4306,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/3953492281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3953492281},{"type":"v1","addr":"172.21.15.72:6837","nonce":3953492281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4309,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/3737698654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3737698654},{"type":"v1","addr":"172.21.15.72:6836","nonce":3737698654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-24T00:38:48.816 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 3 2024-03-24T00:38:48.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.817+0000 7f7236c4b640 1 -- 172.21.15.67:0/3463257830 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f721006bbd0 msgr2=0x7f721006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:48.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.817+0000 7f7236c4b640 1 --2- 172.21.15.67:0/3463257830 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f721006bbd0 0x7f721006e090 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7218036000 tx=0x7f721803b040 comp rx=0 tx=0).stop 2024-03-24T00:38:48.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.817+0000 7f7236c4b640 1 -- 172.21.15.67:0/3463257830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f723006cee0 msgr2=0x7f723015b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:48.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.817+0000 7f7236c4b640 1 --2- 172.21.15.67:0/3463257830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f723006cee0 0x7f723015b600 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f722000c480 tx=0x7f722000c940 comp rx=0 tx=0).stop 2024-03-24T00:38:48.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.818+0000 7f7236c4b640 1 -- 172.21.15.67:0/3463257830 shutdown_connections 2024-03-24T00:38:48.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.818+0000 7f7236c4b640 1 --2- 172.21.15.67:0/3463257830 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f721006bbd0 0x7f721006e090 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:48.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.818+0000 7f7236c4b640 1 --2- 172.21.15.67:0/3463257830 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f72300d1550 0x7f7230160f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:48.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.818+0000 7f7236c4b640 1 --2- 172.21.15.67:0/3463257830 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f723009a2b0 0x7f7230160a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:48.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.818+0000 7f7236c4b640 1 --2- 172.21.15.67:0/3463257830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f723006cee0 0x7f723015b600 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:48.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.818+0000 7f7236c4b640 1 -- 172.21.15.67:0/3463257830 >> 172.21.15.67:0/3463257830 conn(0x7f72300b36e0 msgr2=0x7f7230070070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:48.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.818+0000 7f7236c4b640 1 -- 172.21.15.67:0/3463257830 shutdown_connections 2024-03-24T00:38:48.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.818+0000 7f7236c4b640 1 -- 172.21.15.67:0/3463257830 wait complete. 2024-03-24T00:38:48.828 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:38:48.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.929+0000 7f1a54d1b640 1 Processor -- start 2024-03-24T00:38:48.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.929+0000 7f1a54d1b640 1 -- start start 2024-03-24T00:38:48.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.930+0000 7f1a54d1b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1a5015e310 0x7f1a50162610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:48.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.930+0000 7f1a54d1b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a500ce920 con 0x7f1a500d0d50 2024-03-24T00:38:48.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.930+0000 7f1a54d1b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a500cef20 con 0x7f1a5015df30 2024-03-24T00:38:48.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.930+0000 7f1a54d1b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a50162b50 con 0x7f1a5015e310 2024-03-24T00:38:48.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.930+0000 7f1a4ed9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1a500d0d50 0x7f1a500d2490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60632/0 (socket says 172.21.15.67:60632) 2024-03-24T00:38:48.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.930+0000 7f1a4ed9d640 1 -- 172.21.15.67:0/1010729892 learned_addr learned my addr 172.21.15.67:0/1010729892 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:38:48.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.930+0000 7f1a45d9b640 1 --2- 172.21.15.67:0/1010729892 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1a5015e310 0x7f1a50162610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:48.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.930+0000 7f1a4dd9b640 1 -- 172.21.15.67:0/1010729892 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3791232059 0 0) 0x7f1a500ce920 con 0x7f1a500d0d50 2024-03-24T00:38:48.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.930+0000 7f1a4dd9b640 1 -- 172.21.15.67:0/1010729892 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a3c003520 con 0x7f1a500d0d50 2024-03-24T00:38:48.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.930+0000 7f1a4dd9b640 1 -- 172.21.15.67:0/1010729892 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 401347100 0 0) 0x7f1a500cef20 con 0x7f1a5015df30 2024-03-24T00:38:48.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.930+0000 7f1a4dd9b640 1 -- 172.21.15.67:0/1010729892 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a500ce920 con 0x7f1a5015df30 2024-03-24T00:38:48.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.930+0000 7f1a4dd9b640 1 -- 172.21.15.67:0/1010729892 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1566605164 0 0) 0x7f1a3c003520 con 0x7f1a500d0d50 2024-03-24T00:38:48.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.930+0000 7f1a4dd9b640 1 -- 172.21.15.67:0/1010729892 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1a5015e310 msgr2=0x7f1a50162610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:48.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.930+0000 7f1a4dd9b640 1 --2- 172.21.15.67:0/1010729892 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1a5015e310 0x7f1a50162610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:48.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.930+0000 7f1a4dd9b640 1 -- 172.21.15.67:0/1010729892 >> v1:172.21.15.72:6789/0 conn(0x7f1a5015df30 legacy=0x7f1a500d0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:48.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.930+0000 7f1a4dd9b640 1 -- 172.21.15.67:0/1010729892 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a500cf200 con 0x7f1a500d0d50 2024-03-24T00:38:48.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.930+0000 7f1a4dd9b640 1 -- 172.21.15.67:0/1010729892 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f1a480019f0 con 0x7f1a500d0d50 2024-03-24T00:38:48.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.930+0000 7f1a4dd9b640 1 -- 172.21.15.67:0/1010729892 >> v1:172.21.15.67:6789/0 conn(0x7f1a500d0d50 legacy=0x7f1a500d2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:48.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.931+0000 7f1a4dd9b640 1 --2- 172.21.15.67:0/1010729892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a3c003b70 0x7f1a3c003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:48.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.931+0000 7f1a4dd9b640 1 --2- 172.21.15.67:0/1010729892 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1a3c004800 0x7f1a3c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:48.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.931+0000 7f1a4dd9b640 1 --2- 172.21.15.67:0/1010729892 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1a3c005180 0x7f1a3c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:48.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.931+0000 7f1a4dd9b640 1 -- 172.21.15.67:0/1010729892 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a50162b50 con 0x7f1a3c003b70 2024-03-24T00:38:48.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.931+0000 7f1a4dd9b640 1 -- 172.21.15.67:0/1010729892 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a3c00bf00 con 0x7f1a3c005180 2024-03-24T00:38:48.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.931+0000 7f1a4dd9b640 1 -- 172.21.15.67:0/1010729892 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a3c00c100 con 0x7f1a3c004800 2024-03-24T00:38:48.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.931+0000 7f1a4e59c640 1 --2- 172.21.15.67:0/1010729892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a3c003b70 0x7f1a3c003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:48.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.931+0000 7f1a45d9b640 1 --2- 172.21.15.67:0/1010729892 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1a3c004800 0x7f1a3c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:48.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.931+0000 7f1a4ed9d640 1 --2- 172.21.15.67:0/1010729892 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1a3c005180 0x7f1a3c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:48.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.931+0000 7f1a4e59c640 1 -- 172.21.15.67:0/1010729892 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1a3c004800 msgr2=0x7f1a3c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:48.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.931+0000 7f1a4e59c640 1 --2- 172.21.15.67:0/1010729892 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1a3c004800 0x7f1a3c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:48.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.931+0000 7f1a4e59c640 1 -- 172.21.15.67:0/1010729892 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1a3c005180 msgr2=0x7f1a3c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:48.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.931+0000 7f1a4e59c640 1 --2- 172.21.15.67:0/1010729892 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1a3c005180 0x7f1a3c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:48.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.931+0000 7f1a4e59c640 1 -- 172.21.15.67:0/1010729892 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a3c003520 con 0x7f1a3c003b70 2024-03-24T00:38:48.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.932+0000 7f1a4e59c640 1 --2- 172.21.15.67:0/1010729892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a3c003b70 0x7f1a3c003f70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1a400018a0 tx=0x7f1a3c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=918708522a4e06d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:48.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.932+0000 7f1a4dd9b640 1 -- 172.21.15.67:0/1010729892 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a40037070 con 0x7f1a3c003b70 2024-03-24T00:38:48.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.932+0000 7f1a4dd9b640 1 -- 172.21.15.67:0/1010729892 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a400052d0 con 0x7f1a3c003b70 2024-03-24T00:38:48.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.932+0000 7f1a4dd9b640 1 -- 172.21.15.67:0/1010729892 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a400014f0 con 0x7f1a3c003b70 2024-03-24T00:38:48.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.932+0000 7f1a54d1b640 1 -- 172.21.15.67:0/1010729892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a3c003b70 msgr2=0x7f1a3c003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:48.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.932+0000 7f1a54d1b640 1 --2- 172.21.15.67:0/1010729892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a3c003b70 0x7f1a3c003f70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1a400018a0 tx=0x7f1a3c005830 comp rx=0 tx=0).stop 2024-03-24T00:38:48.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.932+0000 7f1a4ed9d640 1 -- 172.21.15.67:0/1010729892 reap_dead start 2024-03-24T00:38:48.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.932+0000 7f1a54d1b640 1 -- 172.21.15.67:0/1010729892 shutdown_connections 2024-03-24T00:38:48.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.932+0000 7f1a54d1b640 1 -- 172.21.15.67:0/1010729892 >> 172.21.15.67:0/1010729892 conn(0x7f1a5006c880 msgr2=0x7f1a5006cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:48.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.932+0000 7f1a54d1b640 1 -- 172.21.15.67:0/1010729892 shutdown_connections 2024-03-24T00:38:48.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.932+0000 7f1a54d1b640 1 -- 172.21.15.67:0/1010729892 wait complete. 2024-03-24T00:38:48.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.934+0000 7f1a54d1b640 1 Processor -- start 2024-03-24T00:38:48.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.934+0000 7f1a54d1b640 1 -- start start 2024-03-24T00:38:48.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.934+0000 7f1a54d1b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1a3c003b70 0x7f1a501f32f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:48.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.934+0000 7f1a54d1b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1a50147e80 0x7f1a501f6f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:48.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.934+0000 7f1a54d1b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a501f7460 0x7f1a500b85b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:48.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.934+0000 7f1a54d1b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a500ce580 con 0x7f1a501f7460 2024-03-24T00:38:48.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.934+0000 7f1a54d1b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a500b32f0 con 0x7f1a50147e80 2024-03-24T00:38:48.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.934+0000 7f1a54d1b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a500ce700 con 0x7f1a3c003b70 2024-03-24T00:38:48.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.934+0000 7f1a4ed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a501f7460 0x7f1a500b85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:48.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.934+0000 7f1a4ed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a501f7460 0x7f1a500b85b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35982/0 (socket says 172.21.15.67:35982) 2024-03-24T00:38:48.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.934+0000 7f1a4ed9d640 1 -- 172.21.15.67:0/2449758977 learned_addr learned my addr 172.21.15.67:0/2449758977 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:38:48.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.935+0000 7f1a45d9b640 1 --2- 172.21.15.67:0/2449758977 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1a50147e80 0x7f1a501f6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:48.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.935+0000 7f1a4e59c640 1 --2- 172.21.15.67:0/2449758977 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1a3c003b70 0x7f1a501f32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:48.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.935+0000 7f1a4ed9d640 1 -- 172.21.15.67:0/2449758977 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1a3c003b70 msgr2=0x7f1a501f32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:48.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.935+0000 7f1a4ed9d640 1 --2- 172.21.15.67:0/2449758977 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1a3c003b70 0x7f1a501f32f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:48.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.935+0000 7f1a4ed9d640 1 -- 172.21.15.67:0/2449758977 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1a50147e80 msgr2=0x7f1a501f6f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:48.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.935+0000 7f1a4ed9d640 1 --2- 172.21.15.67:0/2449758977 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1a50147e80 0x7f1a501f6f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:48.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.935+0000 7f1a4ed9d640 1 -- 172.21.15.67:0/2449758977 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a501f3830 con 0x7f1a501f7460 2024-03-24T00:38:48.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.935+0000 7f1a4ed9d640 1 --2- 172.21.15.67:0/2449758977 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a501f7460 0x7f1a500b85b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1a5008d1f0 tx=0x7f1a48007b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:48.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.935+0000 7f1a477fe640 1 -- 172.21.15.67:0/2449758977 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4800ea40 con 0x7f1a501f7460 2024-03-24T00:38:48.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.935+0000 7f1a477fe640 1 -- 172.21.15.67:0/2449758977 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a4800ebe0 con 0x7f1a501f7460 2024-03-24T00:38:48.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.935+0000 7f1a54d1b640 1 -- 172.21.15.67:0/2449758977 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a501f7950 con 0x7f1a501f7460 2024-03-24T00:38:48.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.935+0000 7f1a477fe640 1 -- 172.21.15.67:0/2449758977 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a48014430 con 0x7f1a501f7460 2024-03-24T00:38:48.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.935+0000 7f1a54d1b640 1 -- 172.21.15.67:0/2449758977 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a500ccc50 con 0x7f1a501f7460 2024-03-24T00:38:48.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.936+0000 7f1a54d1b640 1 -- 172.21.15.67:0/2449758977 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a501f7e70 con 0x7f1a501f7460 2024-03-24T00:38:48.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.936+0000 7f1a477fe640 1 -- 172.21.15.67:0/2449758977 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1a480145d0 con 0x7f1a501f7460 2024-03-24T00:38:48.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.937+0000 7f1a477fe640 1 --2- 172.21.15.67:0/2449758977 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1a3006bbd0 0x7f1a3006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:48.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.937+0000 7f1a477fe640 1 -- 172.21.15.67:0/2449758977 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1a4808a300 con 0x7f1a501f7460 2024-03-24T00:38:48.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.937+0000 7f1a4e59c640 1 --2- 172.21.15.67:0/2449758977 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1a3006bbd0 0x7f1a3006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:48.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.937+0000 7f1a4e59c640 1 --2- 172.21.15.67:0/2449758977 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1a3006bbd0 0x7f1a3006e090 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1a40034000 tx=0x7f1a40031000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:48.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:48.941+0000 7f1a477fe640 1 -- 172.21.15.67:0/2449758977 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1a501f7e70 con 0x7f1a501f7460 2024-03-24T00:38:49.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.124+0000 7f1a54d1b640 1 -- 172.21.15.67:0/2449758977 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1a5004f400 con 0x7f1a501f7460 2024-03-24T00:38:49.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.124+0000 7f1a477fe640 1 -- 172.21.15.67:0/2449758977 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12824 (secure 0 0 0) 0x7f1a480591d0 con 0x7f1a501f7460 2024-03-24T00:38:49.124 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:38:49.124 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":16,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:38:41.725986+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:38:49.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.127+0000 7f1a54d1b640 1 -- 172.21.15.67:0/2449758977 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1a3006bbd0 msgr2=0x7f1a3006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:49.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.127+0000 7f1a54d1b640 1 --2- 172.21.15.67:0/2449758977 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1a3006bbd0 0x7f1a3006e090 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1a40034000 tx=0x7f1a40031000 comp rx=0 tx=0).stop 2024-03-24T00:38:49.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.127+0000 7f1a54d1b640 1 -- 172.21.15.67:0/2449758977 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a501f7460 msgr2=0x7f1a500b85b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:49.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.127+0000 7f1a54d1b640 1 --2- 172.21.15.67:0/2449758977 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a501f7460 0x7f1a500b85b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1a5008d1f0 tx=0x7f1a48007b70 comp rx=0 tx=0).stop 2024-03-24T00:38:49.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.127+0000 7f1a54d1b640 1 -- 172.21.15.67:0/2449758977 shutdown_connections 2024-03-24T00:38:49.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.127+0000 7f1a54d1b640 1 --2- 172.21.15.67:0/2449758977 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1a3006bbd0 0x7f1a3006e090 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.127+0000 7f1a54d1b640 1 --2- 172.21.15.67:0/2449758977 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a501f7460 0x7f1a500b85b0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.127+0000 7f1a54d1b640 1 --2- 172.21.15.67:0/2449758977 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1a50147e80 0x7f1a501f6f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.127+0000 7f1a54d1b640 1 --2- 172.21.15.67:0/2449758977 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1a3c003b70 0x7f1a501f32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.127+0000 7f1a54d1b640 1 -- 172.21.15.67:0/2449758977 >> 172.21.15.67:0/2449758977 conn(0x7f1a5006c880 msgr2=0x7f1a500b3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:49.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.127+0000 7f1a54d1b640 1 -- 172.21.15.67:0/2449758977 shutdown_connections 2024-03-24T00:38:49.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.127+0000 7f1a54d1b640 1 -- 172.21.15.67:0/2449758977 wait complete. 2024-03-24T00:38:49.137 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:38:39.364950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:38:49.138 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:38:49.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.241+0000 7f8db009a640 1 Processor -- start 2024-03-24T00:38:49.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.241+0000 7f8db009a640 1 -- start start 2024-03-24T00:38:49.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.242+0000 7f8db009a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8da815afc0 0x7f8da80d0920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:49.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.242+0000 7f8db009a640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8da80d12c0 con 0x7f8da80d0e60 2024-03-24T00:38:49.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.242+0000 7f8db009a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8da80cc700 con 0x7f8da80cc050 2024-03-24T00:38:49.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.242+0000 7f8db009a640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8da80cc4e0 con 0x7f8da815afc0 2024-03-24T00:38:49.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.242+0000 7f8dade36640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8da815afc0 0x7f8da80d0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:49.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.242+0000 7f8dad635640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8da80d0e60 0x7f8da80cb940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60648/0 (socket says 172.21.15.67:60648) 2024-03-24T00:38:49.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.242+0000 7f8dad635640 1 -- 172.21.15.67:0/1242707670 learned_addr learned my addr 172.21.15.67:0/1242707670 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:38:49.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.243+0000 7f8dade36640 1 -- 172.21.15.67:0/1242707670 >> v1:172.21.15.72:6789/0 conn(0x7f8da80cc050 legacy=0x7f8da80c4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:49.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.243+0000 7f8dade36640 1 -- 172.21.15.67:0/1242707670 >> v1:172.21.15.67:6789/0 conn(0x7f8da80d0e60 legacy=0x7f8da80cb940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:49.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.243+0000 7f8dade36640 1 -- 172.21.15.67:0/1242707670 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8da80cc9e0 con 0x7f8da815afc0 2024-03-24T00:38:49.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.243+0000 7f8dace34640 1 -- 172.21.15.67:0/1242707670 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1497626055 0 0) 0x7f8da80d12c0 con 0x7f8da80d0e60 2024-03-24T00:38:49.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.243+0000 7f8dade36640 1 --2- 172.21.15.67:0/1242707670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8da815afc0 0x7f8da80d0920 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8da00097a0 tx=0x7f8da002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a749dc0aa6f5b352 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:49.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.243+0000 7f8dace34640 1 -- 172.21.15.67:0/1242707670 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8da0035020 con 0x7f8da815afc0 2024-03-24T00:38:49.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.244+0000 7f8dace34640 1 -- 172.21.15.67:0/1242707670 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8da0031830 con 0x7f8da815afc0 2024-03-24T00:38:49.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.244+0000 7f8dace34640 1 -- 172.21.15.67:0/1242707670 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8da0031c00 con 0x7f8da815afc0 2024-03-24T00:38:49.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.244+0000 7f8db009a640 1 -- 172.21.15.67:0/1242707670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8da815afc0 msgr2=0x7f8da80d0920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:49.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.244+0000 7f8db009a640 1 --2- 172.21.15.67:0/1242707670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8da815afc0 0x7f8da80d0920 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8da00097a0 tx=0x7f8da002b8d0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.244+0000 7f8db009a640 1 -- 172.21.15.67:0/1242707670 shutdown_connections 2024-03-24T00:38:49.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.244+0000 7f8db009a640 1 --2- 172.21.15.67:0/1242707670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8da815afc0 0x7f8da80d0920 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.244+0000 7f8db009a640 1 -- 172.21.15.67:0/1242707670 >> 172.21.15.67:0/1242707670 conn(0x7f8da8099a30 msgr2=0x7f8da8099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:49.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.244+0000 7f8db009a640 1 -- 172.21.15.67:0/1242707670 shutdown_connections 2024-03-24T00:38:49.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.244+0000 7f8db009a640 1 -- 172.21.15.67:0/1242707670 wait complete. 2024-03-24T00:38:49.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.245+0000 7f8db009a640 1 Processor -- start 2024-03-24T00:38:49.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.245+0000 7f8db009a640 1 -- start start 2024-03-24T00:38:49.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.246+0000 7f8db009a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8da80cc050 0x7f8da80b36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:49.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.246+0000 7f8db009a640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8da80d0e60 0x7f8da8156dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:49.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.246+0000 7f8db009a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8da815afc0 0x7f8da80b5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:49.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.246+0000 7f8db009a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da80ccfa0 con 0x7f8da80cc050 2024-03-24T00:38:49.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.246+0000 7f8db009a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da815b3a0 con 0x7f8da815afc0 2024-03-24T00:38:49.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.246+0000 7f8db009a640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8da80c5130 con 0x7f8da80d0e60 2024-03-24T00:38:49.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.246+0000 7f8dade36640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8da80cc050 0x7f8da80b36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:49.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.246+0000 7f8dade36640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8da80cc050 0x7f8da80b36e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35990/0 (socket says 172.21.15.67:35990) 2024-03-24T00:38:49.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.246+0000 7f8dade36640 1 -- 172.21.15.67:0/3280158837 learned_addr learned my addr 172.21.15.67:0/3280158837 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:38:49.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.246+0000 7f8dad635640 1 --2- 172.21.15.67:0/3280158837 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8da80d0e60 0x7f8da8156dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:49.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.246+0000 7f8dae637640 1 --2- 172.21.15.67:0/3280158837 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8da815afc0 0x7f8da80b5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:49.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.246+0000 7f8dade36640 1 -- 172.21.15.67:0/3280158837 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8da80d0e60 msgr2=0x7f8da8156dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:49.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.246+0000 7f8dade36640 1 --2- 172.21.15.67:0/3280158837 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8da80d0e60 0x7f8da8156dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.246+0000 7f8dade36640 1 -- 172.21.15.67:0/3280158837 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8da815afc0 msgr2=0x7f8da80b5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:49.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.246+0000 7f8dade36640 1 --2- 172.21.15.67:0/3280158837 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8da815afc0 0x7f8da80b5ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.246+0000 7f8dade36640 1 -- 172.21.15.67:0/3280158837 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8da80b1250 con 0x7f8da80cc050 2024-03-24T00:38:49.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.246+0000 7f8dad635640 1 --2- 172.21.15.67:0/3280158837 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8da80d0e60 0x7f8da8156dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:38:49.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.246+0000 7f8dae637640 1 --2- 172.21.15.67:0/3280158837 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8da815afc0 0x7f8da80b5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:38:49.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.247+0000 7f8dade36640 1 --2- 172.21.15.67:0/3280158837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8da80cc050 0x7f8da80b36e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8da00097a0 tx=0x7f8da00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:49.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.247+0000 7f8da67fc640 1 -- 172.21.15.67:0/3280158837 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8da003aa70 con 0x7f8da80cc050 2024-03-24T00:38:49.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.247+0000 7f8da67fc640 1 -- 172.21.15.67:0/3280158837 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8da003ac10 con 0x7f8da80cc050 2024-03-24T00:38:49.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.247+0000 7f8da67fc640 1 -- 172.21.15.67:0/3280158837 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8da00443f0 con 0x7f8da80cc050 2024-03-24T00:38:49.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.247+0000 7f8db009a640 1 -- 172.21.15.67:0/3280158837 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8da80b1530 con 0x7f8da80cc050 2024-03-24T00:38:49.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.247+0000 7f8db009a640 1 -- 172.21.15.67:0/3280158837 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8da80b6920 con 0x7f8da80cc050 2024-03-24T00:38:49.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.248+0000 7f8db009a640 1 -- 172.21.15.67:0/3280158837 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8da804f400 con 0x7f8da80cc050 2024-03-24T00:38:49.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.248+0000 7f8da67fc640 1 -- 172.21.15.67:0/3280158837 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8da004b060 con 0x7f8da80cc050 2024-03-24T00:38:49.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.248+0000 7f8da67fc640 1 --2- 172.21.15.67:0/3280158837 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8d9406bbd0 0x7f8d9406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:49.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.249+0000 7f8da67fc640 1 -- 172.21.15.67:0/3280158837 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8da00ac4b0 con 0x7f8da80cc050 2024-03-24T00:38:49.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.249+0000 7f8dad635640 1 --2- 172.21.15.67:0/3280158837 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8d9406bbd0 0x7f8d9406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:49.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.249+0000 7f8dad635640 1 --2- 172.21.15.67:0/3280158837 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8d9406bbd0 0x7f8d9406e090 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8da806eea0 tx=0x7f8d9c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:49.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.253+0000 7f8da67fc640 1 -- 172.21.15.67:0/3280158837 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8da003f070 con 0x7f8da80cc050 2024-03-24T00:38:49.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.433+0000 7f8db009a640 1 -- 172.21.15.67:0/3280158837 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8da80b2160 con 0x7f8da80cc050 2024-03-24T00:38:49.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.433+0000 7f8da67fc640 1 -- 172.21.15.67:0/3280158837 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8da002f2e0 con 0x7f8da80cc050 2024-03-24T00:38:49.434 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-24T00:38:49.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.435+0000 7f8db009a640 1 -- 172.21.15.67:0/3280158837 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8d9406bbd0 msgr2=0x7f8d9406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:49.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.435+0000 7f8db009a640 1 --2- 172.21.15.67:0/3280158837 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8d9406bbd0 0x7f8d9406e090 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8da806eea0 tx=0x7f8d9c009040 comp rx=0 tx=0).stop 2024-03-24T00:38:49.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.435+0000 7f8db009a640 1 -- 172.21.15.67:0/3280158837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8da80cc050 msgr2=0x7f8da80b36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:49.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.435+0000 7f8db009a640 1 --2- 172.21.15.67:0/3280158837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8da80cc050 0x7f8da80b36e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8da00097a0 tx=0x7f8da00046c0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.435+0000 7f8db009a640 1 -- 172.21.15.67:0/3280158837 shutdown_connections 2024-03-24T00:38:49.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.435+0000 7f8db009a640 1 --2- 172.21.15.67:0/3280158837 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8d9406bbd0 0x7f8d9406e090 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.435+0000 7f8db009a640 1 --2- 172.21.15.67:0/3280158837 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8da815afc0 0x7f8da80b5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.435+0000 7f8db009a640 1 --2- 172.21.15.67:0/3280158837 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8da80d0e60 0x7f8da8156dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.435+0000 7f8db009a640 1 --2- 172.21.15.67:0/3280158837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8da80cc050 0x7f8da80b36e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.435+0000 7f8db009a640 1 -- 172.21.15.67:0/3280158837 >> 172.21.15.67:0/3280158837 conn(0x7f8da8099a30 msgr2=0x7f8da8159170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:49.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.435+0000 7f8db009a640 1 -- 172.21.15.67:0/3280158837 shutdown_connections 2024-03-24T00:38:49.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.435+0000 7f8db009a640 1 -- 172.21.15.67:0/3280158837 wait complete. 2024-03-24T00:38:49.446 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-24T00:38:49.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.548+0000 7fceb6885640 1 Processor -- start 2024-03-24T00:38:49.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.548+0000 7fceb6885640 1 -- start start 2024-03-24T00:38:49.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.549+0000 7fceb6885640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fceb00cf220 0x7fceb00d2660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:49.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.549+0000 7fceb6885640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fceb0162120 con 0x7fceb0161bb0 2024-03-24T00:38:49.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.549+0000 7fceb6885640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fceb0162400 con 0x7fceb015dfe0 2024-03-24T00:38:49.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.549+0000 7fceb6885640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fceb00d21d0 con 0x7fceb00cf220 2024-03-24T00:38:49.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.549+0000 7fceb4e22640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fceb0161bb0 0x7fceb00d2ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60654/0 (socket says 172.21.15.67:60654) 2024-03-24T00:38:49.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.549+0000 7fceb4e22640 1 -- 172.21.15.67:0/593298102 learned_addr learned my addr 172.21.15.67:0/593298102 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:38:49.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.549+0000 7fceaeffd640 1 -- 172.21.15.67:0/593298102 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2226306628 0 0) 0x7fceb0162120 con 0x7fceb0161bb0 2024-03-24T00:38:49.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.549+0000 7fceaf7fe640 1 --2- 172.21.15.67:0/593298102 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fceb00cf220 0x7fceb00d2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:49.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.549+0000 7fceaeffd640 1 -- 172.21.15.67:0/593298102 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce98003520 con 0x7fceb0161bb0 2024-03-24T00:38:49.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.549+0000 7fceaeffd640 1 -- 172.21.15.67:0/593298102 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 744662943 0 0) 0x7fce98003520 con 0x7fceb0161bb0 2024-03-24T00:38:49.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.549+0000 7fceaeffd640 1 -- 172.21.15.67:0/593298102 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fceb00cf220 msgr2=0x7fceb00d2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:49.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.549+0000 7fceaeffd640 1 --2- 172.21.15.67:0/593298102 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fceb00cf220 0x7fceb00d2660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.549+0000 7fceaeffd640 1 -- 172.21.15.67:0/593298102 >> v1:172.21.15.72:6789/0 conn(0x7fceb015dfe0 legacy=0x7fceb00ceb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:49.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.549+0000 7fceaeffd640 1 -- 172.21.15.67:0/593298102 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceb01626e0 con 0x7fceb0161bb0 2024-03-24T00:38:49.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.549+0000 7fceaeffd640 1 -- 172.21.15.67:0/593298102 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7fcea80019f0 con 0x7fceb0161bb0 2024-03-24T00:38:49.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.549+0000 7fceaeffd640 1 -- 172.21.15.67:0/593298102 >> v1:172.21.15.67:6789/0 conn(0x7fceb0161bb0 legacy=0x7fceb00d2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:49.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.549+0000 7fceaeffd640 1 --2- 172.21.15.67:0/593298102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce980039e0 0x7fce98003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:49.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.549+0000 7fceaeffd640 1 --2- 172.21.15.67:0/593298102 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fce98004800 0x7fce98004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:49.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.549+0000 7fceaeffd640 1 --2- 172.21.15.67:0/593298102 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fceb00cf220 0x7fce98005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:49.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.549+0000 7fceaeffd640 1 -- 172.21.15.67:0/593298102 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceb00d21d0 con 0x7fce980039e0 2024-03-24T00:38:49.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.549+0000 7fceaeffd640 1 -- 172.21.15.67:0/593298102 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce9800baf0 con 0x7fce98004800 2024-03-24T00:38:49.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.549+0000 7fceaeffd640 1 -- 172.21.15.67:0/593298102 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce9800bcf0 con 0x7fceb00cf220 2024-03-24T00:38:49.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.550+0000 7fceaffff640 1 --2- 172.21.15.67:0/593298102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce980039e0 0x7fce98003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:49.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.550+0000 7fceaf7fe640 1 --2- 172.21.15.67:0/593298102 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fce98004800 0x7fce98004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:49.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.550+0000 7fceb4e22640 1 --2- 172.21.15.67:0/593298102 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fceb00cf220 0x7fce98005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:49.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.550+0000 7fceaffff640 1 -- 172.21.15.67:0/593298102 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fceb00cf220 msgr2=0x7fce98005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:49.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.550+0000 7fceaffff640 1 --2- 172.21.15.67:0/593298102 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fceb00cf220 0x7fce98005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.550+0000 7fceaffff640 1 -- 172.21.15.67:0/593298102 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fce98004800 msgr2=0x7fce98004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:49.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.550+0000 7fceaffff640 1 --2- 172.21.15.67:0/593298102 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fce98004800 0x7fce98004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.550+0000 7fceaffff640 1 -- 172.21.15.67:0/593298102 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce98003520 con 0x7fce980039e0 2024-03-24T00:38:49.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.551+0000 7fceaffff640 1 --2- 172.21.15.67:0/593298102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce980039e0 0x7fce98003e00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcea4001750 tx=0x7fce98005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb482898afa66ee server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:49.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.551+0000 7fceaeffd640 1 -- 172.21.15.67:0/593298102 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea4001370 con 0x7fce980039e0 2024-03-24T00:38:49.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.551+0000 7fceaeffd640 1 -- 172.21.15.67:0/593298102 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcea40052d0 con 0x7fce980039e0 2024-03-24T00:38:49.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.551+0000 7fceaeffd640 1 -- 172.21.15.67:0/593298102 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea4003830 con 0x7fce980039e0 2024-03-24T00:38:49.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.551+0000 7fceb6885640 1 -- 172.21.15.67:0/593298102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce980039e0 msgr2=0x7fce98003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:49.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.551+0000 7fceb6885640 1 --2- 172.21.15.67:0/593298102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce980039e0 0x7fce98003e00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcea4001750 tx=0x7fce98005830 comp rx=0 tx=0).stop 2024-03-24T00:38:49.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.551+0000 7fceb4e22640 1 -- 172.21.15.67:0/593298102 reap_dead start 2024-03-24T00:38:49.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.551+0000 7fceb6885640 1 -- 172.21.15.67:0/593298102 shutdown_connections 2024-03-24T00:38:49.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.551+0000 7fceb6885640 1 -- 172.21.15.67:0/593298102 >> 172.21.15.67:0/593298102 conn(0x7fceb006c890 msgr2=0x7fceb006cc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:49.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.551+0000 7fceb6885640 1 -- 172.21.15.67:0/593298102 shutdown_connections 2024-03-24T00:38:49.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.551+0000 7fceb6885640 1 -- 172.21.15.67:0/593298102 wait complete. 2024-03-24T00:38:49.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.552+0000 7fceb6885640 1 Processor -- start 2024-03-24T00:38:49.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.553+0000 7fceb6885640 1 -- start start 2024-03-24T00:38:49.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.553+0000 7fceb6885640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce980039e0 0x7fceb0160120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:49.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.553+0000 7fceb6885640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fceb00b36a0 0x7fceb0151160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:49.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.553+0000 7fceb6885640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fceb00b3b00 0x7fceb01516a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:49.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.553+0000 7fceb6885640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceb00d1120 con 0x7fce980039e0 2024-03-24T00:38:49.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.553+0000 7fceb6885640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceb015e3c0 con 0x7fceb00b36a0 2024-03-24T00:38:49.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.553+0000 7fceb6885640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fceb00d12a0 con 0x7fceb00b3b00 2024-03-24T00:38:49.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.553+0000 7fceaffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce980039e0 0x7fceb0160120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:49.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.553+0000 7fceaffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce980039e0 0x7fceb0160120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35998/0 (socket says 172.21.15.67:35998) 2024-03-24T00:38:49.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.553+0000 7fceaffff640 1 -- 172.21.15.67:0/992842744 learned_addr learned my addr 172.21.15.67:0/992842744 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:38:49.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.554+0000 7fceb4e22640 1 --2- 172.21.15.67:0/992842744 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fceb00b3b00 0x7fceb01516a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:49.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.554+0000 7fceaf7fe640 1 --2- 172.21.15.67:0/992842744 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fceb00b36a0 0x7fceb0151160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:49.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.554+0000 7fceaffff640 1 -- 172.21.15.67:0/992842744 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fceb00b3b00 msgr2=0x7fceb01516a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:49.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.554+0000 7fceaffff640 1 --2- 172.21.15.67:0/992842744 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fceb00b3b00 0x7fceb01516a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.554+0000 7fceaffff640 1 -- 172.21.15.67:0/992842744 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fceb00b36a0 msgr2=0x7fceb0151160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:49.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.554+0000 7fceaffff640 1 --2- 172.21.15.67:0/992842744 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fceb00b36a0 0x7fceb0151160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.554+0000 7fceaffff640 1 -- 172.21.15.67:0/992842744 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceb00b40b0 con 0x7fce980039e0 2024-03-24T00:38:49.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.554+0000 7fceaffff640 1 --2- 172.21.15.67:0/992842744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce980039e0 0x7fceb0160120 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fcea4005040 tx=0x7fcea4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:49.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.554+0000 7fceacff9640 1 -- 172.21.15.67:0/992842744 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea4041070 con 0x7fce980039e0 2024-03-24T00:38:49.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.554+0000 7fceacff9640 1 -- 172.21.15.67:0/992842744 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcea4005480 con 0x7fce980039e0 2024-03-24T00:38:49.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.554+0000 7fceb6885640 1 -- 172.21.15.67:0/992842744 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fceb00b4390 con 0x7fce980039e0 2024-03-24T00:38:49.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.554+0000 7fceacff9640 1 -- 172.21.15.67:0/992842744 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea4001370 con 0x7fce980039e0 2024-03-24T00:38:49.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.554+0000 7fceb6885640 1 -- 172.21.15.67:0/992842744 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fceb00b4bb0 con 0x7fce980039e0 2024-03-24T00:38:49.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.555+0000 7fceb6885640 1 -- 172.21.15.67:0/992842744 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fceb006b180 con 0x7fce980039e0 2024-03-24T00:38:49.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.556+0000 7fceacff9640 1 -- 172.21.15.67:0/992842744 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fcea4044070 con 0x7fce980039e0 2024-03-24T00:38:49.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.556+0000 7fceacff9640 1 --2- 172.21.15.67:0/992842744 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fce9c06bbd0 0x7fce9c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:49.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.556+0000 7fceacff9640 1 -- 172.21.15.67:0/992842744 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fcea40ac140 con 0x7fce980039e0 2024-03-24T00:38:49.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.556+0000 7fceaf7fe640 1 --2- 172.21.15.67:0/992842744 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fce9c06bbd0 0x7fce9c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:49.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.557+0000 7fceaf7fe640 1 --2- 172.21.15.67:0/992842744 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fce9c06bbd0 0x7fce9c06e090 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fceb0160bf0 tx=0x7fcea0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:49.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.560+0000 7fceacff9640 1 -- 172.21.15.67:0/992842744 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fcea407b010 con 0x7fce980039e0 2024-03-24T00:38:49.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.746+0000 7fceb6885640 1 -- 172.21.15.67:0/992842744 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fceb004f400 con 0x7fce980039e0 2024-03-24T00:38:49.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.746+0000 7fceacff9640 1 -- 172.21.15.67:0/992842744 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+65 (secure 0 0 0) 0x7fcea407a910 con 0x7fce980039e0 2024-03-24T00:38:49.746 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:38:49.747 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-03-24T00:38:49.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:38:49.747 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2024-03-24T00:38:49.747 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2024-03-24T00:38:49.747 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:38:49.747 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-03-24T00:38:49.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.748+0000 7fceb6885640 1 -- 172.21.15.67:0/992842744 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fce9c06bbd0 msgr2=0x7fce9c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:49.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.748+0000 7fceb6885640 1 --2- 172.21.15.67:0/992842744 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fce9c06bbd0 0x7fce9c06e090 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fceb0160bf0 tx=0x7fcea0008040 comp rx=0 tx=0).stop 2024-03-24T00:38:49.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.748+0000 7fceb6885640 1 -- 172.21.15.67:0/992842744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce980039e0 msgr2=0x7fceb0160120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:49.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.748+0000 7fceb6885640 1 --2- 172.21.15.67:0/992842744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce980039e0 0x7fceb0160120 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fcea4005040 tx=0x7fcea4001120 comp rx=0 tx=0).stop 2024-03-24T00:38:49.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.748+0000 7fceb6885640 1 -- 172.21.15.67:0/992842744 shutdown_connections 2024-03-24T00:38:49.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.748+0000 7fceb6885640 1 --2- 172.21.15.67:0/992842744 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fce9c06bbd0 0x7fce9c06e090 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.748+0000 7fceb6885640 1 --2- 172.21.15.67:0/992842744 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fceb00b3b00 0x7fceb01516a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.748+0000 7fceb6885640 1 --2- 172.21.15.67:0/992842744 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fceb00b36a0 0x7fceb0151160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.748+0000 7fceb6885640 1 --2- 172.21.15.67:0/992842744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce980039e0 0x7fceb0160120 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.748+0000 7fceb6885640 1 -- 172.21.15.67:0/992842744 >> 172.21.15.67:0/992842744 conn(0x7fceb006c890 msgr2=0x7fceb015d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:49.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.749+0000 7fceb6885640 1 -- 172.21.15.67:0/992842744 shutdown_connections 2024-03-24T00:38:49.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.749+0000 7fceb6885640 1 -- 172.21.15.67:0/992842744 wait complete. 2024-03-24T00:38:49.759 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-24T00:38:49.759 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-24T00:38:49.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.864+0000 7fc56bb77640 1 Processor -- start 2024-03-24T00:38:49.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.864+0000 7fc56bb77640 1 -- start start 2024-03-24T00:38:49.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.865+0000 7fc56bb77640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc56415f630 0x7fc564099d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:49.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.865+0000 7fc56bb77640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc5640b1070 0x7fc56409a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:49.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.865+0000 7fc56bb77640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5640cdb50 con 0x7fc5640b0ab0 2024-03-24T00:38:49.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.865+0000 7fc56bb77640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc56409b090 con 0x7fc5640b1070 2024-03-24T00:38:49.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.865+0000 7fc56bb77640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc56409ad20 con 0x7fc56415f630 2024-03-24T00:38:49.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.865+0000 7fc569112640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc5640b0ab0 0x7fc5640cc860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60664/0 (socket says 172.21.15.67:60664) 2024-03-24T00:38:49.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.865+0000 7fc569112640 1 -- 172.21.15.67:0/1848296597 learned_addr learned my addr 172.21.15.67:0/1848296597 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:38:49.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.866+0000 7fc568911640 1 -- 172.21.15.67:0/1848296597 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 735026078 0 0) 0x7fc5640cdb50 con 0x7fc5640b0ab0 2024-03-24T00:38:49.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.866+0000 7fc569913640 1 --2- 172.21.15.67:0/1848296597 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc56415f630 0x7fc564099d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:49.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.866+0000 7fc568911640 1 -- 172.21.15.67:0/1848296597 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc54c003520 con 0x7fc5640b0ab0 2024-03-24T00:38:49.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.866+0000 7fc56a114640 1 --2- 172.21.15.67:0/1848296597 >> v2:172.21.15.72:3300/0 conn(0x7fc5640b1070 0x7fc56409a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:49.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.866+0000 7fc568911640 1 -- 172.21.15.67:0/1848296597 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4077438381 0 0) 0x7fc54c003520 con 0x7fc5640b0ab0 2024-03-24T00:38:49.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.866+0000 7fc568911640 1 -- 172.21.15.67:0/1848296597 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc56415f630 msgr2=0x7fc564099d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:49.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.866+0000 7fc568911640 1 --2- 172.21.15.67:0/1848296597 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc56415f630 0x7fc564099d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.866+0000 7fc568911640 1 -- 172.21.15.67:0/1848296597 >> v2:172.21.15.72:3300/0 conn(0x7fc5640b1070 msgr2=0x7fc56409a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:49.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.866+0000 7fc568911640 1 --2- 172.21.15.67:0/1848296597 >> v2:172.21.15.72:3300/0 conn(0x7fc5640b1070 0x7fc56409a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.866+0000 7fc568911640 1 -- 172.21.15.67:0/1848296597 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc56415fa10 con 0x7fc5640b0ab0 2024-03-24T00:38:49.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.866+0000 7fc568911640 1 -- 172.21.15.67:0/1848296597 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7fc558001670 con 0x7fc5640b0ab0 2024-03-24T00:38:49.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.866+0000 7fc568911640 1 -- 172.21.15.67:0/1848296597 >> v1:172.21.15.67:6789/0 conn(0x7fc5640b0ab0 legacy=0x7fc5640cc860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:49.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.866+0000 7fc568911640 1 --2- 172.21.15.67:0/1848296597 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc56415f630 0x7fc54c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:49.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.866+0000 7fc568911640 1 --2- 172.21.15.67:0/1848296597 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc54c003f30 0x7fc54c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:49.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.866+0000 7fc568911640 1 --2- 172.21.15.67:0/1848296597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc54c004d40 0x7fc54c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:49.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.866+0000 7fc568911640 1 -- 172.21.15.67:0/1848296597 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc56409b090 con 0x7fc54c004d40 2024-03-24T00:38:49.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.866+0000 7fc568911640 1 -- 172.21.15.67:0/1848296597 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc56409ad20 con 0x7fc54c003f30 2024-03-24T00:38:49.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.866+0000 7fc568911640 1 -- 172.21.15.67:0/1848296597 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc54c00b9f0 con 0x7fc56415f630 2024-03-24T00:38:49.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.866+0000 7fc56a114640 1 --2- 172.21.15.67:0/1848296597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc54c004d40 0x7fc54c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:49.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.866+0000 7fc569913640 1 --2- 172.21.15.67:0/1848296597 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc56415f630 0x7fc54c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:49.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.866+0000 7fc569112640 1 --2- 172.21.15.67:0/1848296597 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc54c003f30 0x7fc54c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:49.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.866+0000 7fc56a114640 1 -- 172.21.15.67:0/1848296597 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc56415f630 msgr2=0x7fc54c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:49.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.866+0000 7fc56a114640 1 --2- 172.21.15.67:0/1848296597 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc56415f630 0x7fc54c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.866+0000 7fc56a114640 1 -- 172.21.15.67:0/1848296597 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc54c003f30 msgr2=0x7fc54c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:49.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.866+0000 7fc56a114640 1 --2- 172.21.15.67:0/1848296597 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc54c003f30 0x7fc54c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.866+0000 7fc56a114640 1 -- 172.21.15.67:0/1848296597 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc54c003520 con 0x7fc54c004d40 2024-03-24T00:38:49.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.866+0000 7fc569913640 1 --2- 172.21.15.67:0/1848296597 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc56415f630 0x7fc54c0039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:38:49.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.867+0000 7fc56a114640 1 --2- 172.21.15.67:0/1848296597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc54c004d40 0x7fc54c0051c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc564073380 tx=0x7fc54c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc74c4907a5a35e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:49.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.867+0000 7fc568911640 1 -- 172.21.15.67:0/1848296597 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc554039070 con 0x7fc54c004d40 2024-03-24T00:38:49.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.867+0000 7fc568911640 1 -- 172.21.15.67:0/1848296597 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc55402cbd0 con 0x7fc54c004d40 2024-03-24T00:38:49.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.867+0000 7fc568911640 1 -- 172.21.15.67:0/1848296597 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc554009c10 con 0x7fc54c004d40 2024-03-24T00:38:49.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.868+0000 7fc56bb77640 1 -- 172.21.15.67:0/1848296597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc54c004d40 msgr2=0x7fc54c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:49.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.868+0000 7fc56bb77640 1 --2- 172.21.15.67:0/1848296597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc54c004d40 0x7fc54c0051c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc564073380 tx=0x7fc54c005830 comp rx=0 tx=0).stop 2024-03-24T00:38:49.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.868+0000 7fc56a114640 1 -- 172.21.15.67:0/1848296597 reap_dead start 2024-03-24T00:38:49.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.868+0000 7fc56bb77640 1 -- 172.21.15.67:0/1848296597 shutdown_connections 2024-03-24T00:38:49.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.868+0000 7fc56bb77640 1 -- 172.21.15.67:0/1848296597 >> 172.21.15.67:0/1848296597 conn(0x7fc5640b36d0 msgr2=0x7fc5640b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:49.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.868+0000 7fc56bb77640 1 -- 172.21.15.67:0/1848296597 shutdown_connections 2024-03-24T00:38:49.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.868+0000 7fc56bb77640 1 -- 172.21.15.67:0/1848296597 wait complete. 2024-03-24T00:38:49.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.869+0000 7fc56bb77640 1 Processor -- start 2024-03-24T00:38:49.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.869+0000 7fc56bb77640 1 -- start start 2024-03-24T00:38:49.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.870+0000 7fc56bb77640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc5641f1000 0x7fc5641f13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:49.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.870+0000 7fc56bb77640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5641f5a00 0x7fc5640b6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:49.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.870+0000 7fc56bb77640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc5640b45d0 0x7fc5640b4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:49.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.870+0000 7fc56bb77640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5640cf9c0 con 0x7fc5641f5a00 2024-03-24T00:38:49.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.870+0000 7fc56bb77640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5640b2910 con 0x7fc5641f1000 2024-03-24T00:38:49.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.870+0000 7fc56bb77640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5640cfb40 con 0x7fc5640b45d0 2024-03-24T00:38:49.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.870+0000 7fc569112640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5641f5a00 0x7fc5640b6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:49.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.870+0000 7fc569112640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5641f5a00 0x7fc5640b6550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36030/0 (socket says 172.21.15.67:36030) 2024-03-24T00:38:49.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.870+0000 7fc569112640 1 -- 172.21.15.67:0/951898192 learned_addr learned my addr 172.21.15.67:0/951898192 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:38:49.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.870+0000 7fc56a114640 1 --2- 172.21.15.67:0/951898192 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc5640b45d0 0x7fc5640b4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:49.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.870+0000 7fc569913640 1 --2- 172.21.15.67:0/951898192 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc5641f1000 0x7fc5641f13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:49.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.870+0000 7fc569112640 1 -- 172.21.15.67:0/951898192 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc5640b45d0 msgr2=0x7fc5640b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:49.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.870+0000 7fc569112640 1 --2- 172.21.15.67:0/951898192 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc5640b45d0 0x7fc5640b4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.870+0000 7fc569112640 1 -- 172.21.15.67:0/951898192 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc5641f1000 msgr2=0x7fc5641f13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:49.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.870+0000 7fc569112640 1 --2- 172.21.15.67:0/951898192 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc5641f1000 0x7fc5641f13e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.870+0000 7fc569112640 1 -- 172.21.15.67:0/951898192 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5640b6e60 con 0x7fc5641f5a00 2024-03-24T00:38:49.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.870+0000 7fc56a114640 1 --2- 172.21.15.67:0/951898192 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc5640b45d0 0x7fc5640b4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:38:49.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.870+0000 7fc569913640 1 --2- 172.21.15.67:0/951898192 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc5641f1000 0x7fc5641f13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:38:49.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.870+0000 7fc569112640 1 --2- 172.21.15.67:0/951898192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5641f5a00 0x7fc5640b6550 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc5580017e0 tx=0x7fc558007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:49.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.870+0000 7fc5627fc640 1 -- 172.21.15.67:0/951898192 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55800f940 con 0x7fc5641f5a00 2024-03-24T00:38:49.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.870+0000 7fc5627fc640 1 -- 172.21.15.67:0/951898192 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc558018070 con 0x7fc5641f5a00 2024-03-24T00:38:49.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.870+0000 7fc5627fc640 1 -- 172.21.15.67:0/951898192 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55800fd60 con 0x7fc5641f5a00 2024-03-24T00:38:49.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.870+0000 7fc56bb77640 1 -- 172.21.15.67:0/951898192 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5640b7140 con 0x7fc5641f5a00 2024-03-24T00:38:49.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.870+0000 7fc56bb77640 1 -- 172.21.15.67:0/951898192 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc56415c2e0 con 0x7fc5641f5a00 2024-03-24T00:38:49.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.871+0000 7fc56bb77640 1 -- 172.21.15.67:0/951898192 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5640b7660 con 0x7fc5641f5a00 2024-03-24T00:38:49.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.872+0000 7fc5627fc640 1 -- 172.21.15.67:0/951898192 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc55801d460 con 0x7fc5641f5a00 2024-03-24T00:38:49.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.872+0000 7fc5627fc640 1 --2- 172.21.15.67:0/951898192 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc55006bbd0 0x7fc55006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:49.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.872+0000 7fc5627fc640 1 -- 172.21.15.67:0/951898192 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc55808a980 con 0x7fc5641f5a00 2024-03-24T00:38:49.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.872+0000 7fc569913640 1 --2- 172.21.15.67:0/951898192 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc55006bbd0 0x7fc55006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:49.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.873+0000 7fc569913640 1 --2- 172.21.15.67:0/951898192 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc55006bbd0 0x7fc55006e090 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc55c009b50 tx=0x7fc55c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:49.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:49.876+0000 7fc5627fc640 1 -- 172.21.15.67:0/951898192 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc5640b7660 con 0x7fc5641f5a00 2024-03-24T00:38:50.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.058+0000 7fc56bb77640 1 -- 172.21.15.67:0/951898192 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fc5640cdb50 con 0x7fc5641f5a00 2024-03-24T00:38:50.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.753+0000 7fc5627fc640 1 -- 172.21.15.67:0/951898192 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7fc5640cdb50 con 0x7fc5641f5a00 2024-03-24T00:38:50.752 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' created 2024-03-24T00:38:50.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.755+0000 7fc56bb77640 1 -- 172.21.15.67:0/951898192 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc55006bbd0 msgr2=0x7fc55006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:50.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.755+0000 7fc56bb77640 1 --2- 172.21.15.67:0/951898192 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc55006bbd0 0x7fc55006e090 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc55c009b50 tx=0x7fc55c008040 comp rx=0 tx=0).stop 2024-03-24T00:38:50.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.755+0000 7fc56bb77640 1 -- 172.21.15.67:0/951898192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5641f5a00 msgr2=0x7fc5640b6550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:50.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.755+0000 7fc56bb77640 1 --2- 172.21.15.67:0/951898192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5641f5a00 0x7fc5640b6550 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc5580017e0 tx=0x7fc558007720 comp rx=0 tx=0).stop 2024-03-24T00:38:50.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.755+0000 7fc56bb77640 1 -- 172.21.15.67:0/951898192 shutdown_connections 2024-03-24T00:38:50.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.755+0000 7fc56bb77640 1 --2- 172.21.15.67:0/951898192 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc55006bbd0 0x7fc55006e090 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:50.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.755+0000 7fc56bb77640 1 --2- 172.21.15.67:0/951898192 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc5640b45d0 0x7fc5640b4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:50.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.755+0000 7fc56bb77640 1 --2- 172.21.15.67:0/951898192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5641f5a00 0x7fc5640b6550 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:50.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.755+0000 7fc56bb77640 1 --2- 172.21.15.67:0/951898192 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc5641f1000 0x7fc5641f13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:50.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.755+0000 7fc56bb77640 1 -- 172.21.15.67:0/951898192 >> 172.21.15.67:0/951898192 conn(0x7fc5640b36d0 msgr2=0x7fc564070340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:50.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.755+0000 7fc56bb77640 1 -- 172.21.15.67:0/951898192 shutdown_connections 2024-03-24T00:38:50.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.755+0000 7fc56bb77640 1 -- 172.21.15.67:0/951898192 wait complete. 2024-03-24T00:38:50.770 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-03-24T00:38:50.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.880+0000 7f20520e8640 1 Processor -- start 2024-03-24T00:38:50.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.880+0000 7f20520e8640 1 -- start start 2024-03-24T00:38:50.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.881+0000 7f20520e8640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f204c06ef60 0x7f204c0b4960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:50.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.881+0000 7f20520e8640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f204c15c0b0 0x7f204c158400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:50.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.881+0000 7f20520e8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f204c071cb0 con 0x7f204c06f340 2024-03-24T00:38:50.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.881+0000 7f20520e8640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f204c071f90 con 0x7f204c15c0b0 2024-03-24T00:38:50.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.881+0000 7f20520e8640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f204c072270 con 0x7f204c06ef60 2024-03-24T00:38:50.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.881+0000 7f204bfff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f204c06f340 0x7f204c15e9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60674/0 (socket says 172.21.15.67:60674) 2024-03-24T00:38:50.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.881+0000 7f204bfff640 1 -- 172.21.15.67:0/4222567634 learned_addr learned my addr 172.21.15.67:0/4222567634 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:38:50.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.881+0000 7f204affd640 1 --2- 172.21.15.67:0/4222567634 >> v2:172.21.15.72:3300/0 conn(0x7f204c15c0b0 0x7f204c158400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:50.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.881+0000 7f204b7fe640 1 --2- 172.21.15.67:0/4222567634 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f204c06ef60 0x7f204c0b4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:50.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.881+0000 7f204a7fc640 1 -- 172.21.15.67:0/4222567634 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2079881742 0 0) 0x7f204c071cb0 con 0x7f204c06f340 2024-03-24T00:38:50.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.881+0000 7f204a7fc640 1 -- 172.21.15.67:0/4222567634 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2034003520 con 0x7f204c06f340 2024-03-24T00:38:50.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.881+0000 7f204a7fc640 1 -- 172.21.15.67:0/4222567634 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 418933527 0 0) 0x7f2034003520 con 0x7f204c06f340 2024-03-24T00:38:50.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.881+0000 7f204a7fc640 1 -- 172.21.15.67:0/4222567634 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f204c06ef60 msgr2=0x7f204c0b4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:50.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.881+0000 7f204a7fc640 1 --2- 172.21.15.67:0/4222567634 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f204c06ef60 0x7f204c0b4960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:50.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.881+0000 7f204a7fc640 1 -- 172.21.15.67:0/4222567634 >> v2:172.21.15.72:3300/0 conn(0x7f204c15c0b0 msgr2=0x7f204c158400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:50.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.881+0000 7f204a7fc640 1 --2- 172.21.15.67:0/4222567634 >> v2:172.21.15.72:3300/0 conn(0x7f204c15c0b0 0x7f204c158400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:50.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.881+0000 7f204a7fc640 1 -- 172.21.15.67:0/4222567634 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f204c072550 con 0x7f204c06f340 2024-03-24T00:38:50.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.881+0000 7f204a7fc640 1 -- 172.21.15.67:0/4222567634 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f20440019f0 con 0x7f204c06f340 2024-03-24T00:38:50.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.881+0000 7f204a7fc640 1 -- 172.21.15.67:0/4222567634 >> v1:172.21.15.67:6789/0 conn(0x7f204c06f340 legacy=0x7f204c15e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:50.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.881+0000 7f204a7fc640 1 --2- 172.21.15.67:0/4222567634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20340039b0 0x7f2034003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:50.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.881+0000 7f204a7fc640 1 --2- 172.21.15.67:0/4222567634 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2034004800 0x7f2034004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:50.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.881+0000 7f204a7fc640 1 --2- 172.21.15.67:0/4222567634 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f204c06ef60 0x7f20340051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:50.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.881+0000 7f204a7fc640 1 -- 172.21.15.67:0/4222567634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204c071f90 con 0x7f20340039b0 2024-03-24T00:38:50.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.881+0000 7f204a7fc640 1 -- 172.21.15.67:0/4222567634 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204c072270 con 0x7f2034004800 2024-03-24T00:38:50.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.881+0000 7f204a7fc640 1 -- 172.21.15.67:0/4222567634 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f203400b9c0 con 0x7f204c06ef60 2024-03-24T00:38:50.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.881+0000 7f204b7fe640 1 --2- 172.21.15.67:0/4222567634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20340039b0 0x7f2034003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:50.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.882+0000 7f204affd640 1 --2- 172.21.15.67:0/4222567634 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2034004800 0x7f2034004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:50.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.882+0000 7f204bfff640 1 --2- 172.21.15.67:0/4222567634 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f204c06ef60 0x7f20340051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:50.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.882+0000 7f204b7fe640 1 -- 172.21.15.67:0/4222567634 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f204c06ef60 msgr2=0x7f20340051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:50.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.882+0000 7f204b7fe640 1 --2- 172.21.15.67:0/4222567634 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f204c06ef60 0x7f20340051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:50.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.882+0000 7f204b7fe640 1 -- 172.21.15.67:0/4222567634 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2034004800 msgr2=0x7f2034004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:50.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.882+0000 7f204b7fe640 1 --2- 172.21.15.67:0/4222567634 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2034004800 0x7f2034004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:50.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.882+0000 7f204b7fe640 1 -- 172.21.15.67:0/4222567634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2034003520 con 0x7f20340039b0 2024-03-24T00:38:50.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.882+0000 7f204b7fe640 1 --2- 172.21.15.67:0/4222567634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20340039b0 0x7f2034003dd0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f203c009650 tx=0x7f2034005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbdf6b69228fdd7e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:50.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.883+0000 7f204a7fc640 1 -- 172.21.15.67:0/4222567634 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f203c03b070 con 0x7f20340039b0 2024-03-24T00:38:50.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.883+0000 7f204a7fc640 1 -- 172.21.15.67:0/4222567634 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f203c02e8c0 con 0x7f20340039b0 2024-03-24T00:38:50.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.883+0000 7f204a7fc640 1 -- 172.21.15.67:0/4222567634 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f203c009690 con 0x7f20340039b0 2024-03-24T00:38:50.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.883+0000 7f20520e8640 1 -- 172.21.15.67:0/4222567634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20340039b0 msgr2=0x7f2034003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:50.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.883+0000 7f20520e8640 1 --2- 172.21.15.67:0/4222567634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20340039b0 0x7f2034003dd0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f203c009650 tx=0x7f2034005830 comp rx=0 tx=0).stop 2024-03-24T00:38:50.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.883+0000 7f204bfff640 1 -- 172.21.15.67:0/4222567634 reap_dead start 2024-03-24T00:38:50.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.883+0000 7f20520e8640 1 -- 172.21.15.67:0/4222567634 shutdown_connections 2024-03-24T00:38:50.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.883+0000 7f20520e8640 1 -- 172.21.15.67:0/4222567634 >> 172.21.15.67:0/4222567634 conn(0x7f204c0cfa70 msgr2=0x7f204c0cfe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:50.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.883+0000 7f20520e8640 1 -- 172.21.15.67:0/4222567634 shutdown_connections 2024-03-24T00:38:50.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.883+0000 7f20520e8640 1 -- 172.21.15.67:0/4222567634 wait complete. 2024-03-24T00:38:50.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.884+0000 7f20520e8640 1 Processor -- start 2024-03-24T00:38:50.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.884+0000 7f20520e8640 1 -- start start 2024-03-24T00:38:50.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.885+0000 7f20520e8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20340039b0 0x7f204c141e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:50.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.885+0000 7f20520e8640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f204c06b040 0x7f204c06b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:50.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.885+0000 7f20520e8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f204c0c5ad0 0x7f204c0c5f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:50.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.885+0000 7f20520e8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204c0b3c10 con 0x7f204c0c5ad0 2024-03-24T00:38:50.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.885+0000 7f20520e8640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204c0b3a90 con 0x7f20340039b0 2024-03-24T00:38:50.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.885+0000 7f20520e8640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f204c0b3d90 con 0x7f204c06b040 2024-03-24T00:38:50.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.885+0000 7f204bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f204c0c5ad0 0x7f204c0c5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:50.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.885+0000 7f204bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f204c0c5ad0 0x7f204c0c5f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36038/0 (socket says 172.21.15.67:36038) 2024-03-24T00:38:50.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.885+0000 7f204bfff640 1 -- 172.21.15.67:0/2769506345 learned_addr learned my addr 172.21.15.67:0/2769506345 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:38:50.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.885+0000 7f204affd640 1 --2- 172.21.15.67:0/2769506345 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f204c06b040 0x7f204c06b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:50.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.885+0000 7f204b7fe640 1 --2- 172.21.15.67:0/2769506345 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20340039b0 0x7f204c141e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:50.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.886+0000 7f204bfff640 1 -- 172.21.15.67:0/2769506345 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f204c06b040 msgr2=0x7f204c06b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:50.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.886+0000 7f204bfff640 1 --2- 172.21.15.67:0/2769506345 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f204c06b040 0x7f204c06b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:50.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.886+0000 7f204bfff640 1 -- 172.21.15.67:0/2769506345 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20340039b0 msgr2=0x7f204c141e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:50.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.886+0000 7f204bfff640 1 --2- 172.21.15.67:0/2769506345 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20340039b0 0x7f204c141e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:50.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.886+0000 7f204bfff640 1 -- 172.21.15.67:0/2769506345 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f204c0c6800 con 0x7f204c0c5ad0 2024-03-24T00:38:50.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.886+0000 7f204bfff640 1 --2- 172.21.15.67:0/2769506345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f204c0c5ad0 0x7f204c0c5f50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f204c06f790 tx=0x7f204400a750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:50.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.886+0000 7f2033fff640 1 -- 172.21.15.67:0/2769506345 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f204400ad50 con 0x7f204c0c5ad0 2024-03-24T00:38:50.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.886+0000 7f2033fff640 1 -- 172.21.15.67:0/2769506345 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2044018070 con 0x7f204c0c5ad0 2024-03-24T00:38:50.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.886+0000 7f2033fff640 1 -- 172.21.15.67:0/2769506345 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2044013630 con 0x7f204c0c5ad0 2024-03-24T00:38:50.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.886+0000 7f20520e8640 1 -- 172.21.15.67:0/2769506345 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f204c1efd20 con 0x7f204c0c5ad0 2024-03-24T00:38:50.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.886+0000 7f20520e8640 1 -- 172.21.15.67:0/2769506345 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f204c1f02e0 con 0x7f204c0c5ad0 2024-03-24T00:38:50.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.887+0000 7f20520e8640 1 -- 172.21.15.67:0/2769506345 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f204c04f400 con 0x7f204c0c5ad0 2024-03-24T00:38:50.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.887+0000 7f2033fff640 1 -- 172.21.15.67:0/2769506345 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f20440137d0 con 0x7f204c0c5ad0 2024-03-24T00:38:50.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.888+0000 7f2033fff640 1 --2- 172.21.15.67:0/2769506345 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f203806bc00 0x7f203806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:50.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.888+0000 7f2033fff640 1 -- 172.21.15.67:0/2769506345 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f204408afa0 con 0x7f204c0c5ad0 2024-03-24T00:38:50.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.888+0000 7f204b7fe640 1 --2- 172.21.15.67:0/2769506345 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f203806bc00 0x7f203806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:50.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.888+0000 7f204b7fe640 1 --2- 172.21.15.67:0/2769506345 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f203806bc00 0x7f203806e0c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f203c038000 tx=0x7f203c032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:50.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:50.892+0000 7f2033fff640 1 -- 172.21.15.67:0/2769506345 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2044059c70 con 0x7f204c0c5ad0 2024-03-24T00:38:51.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.076+0000 7f20520e8640 1 -- 172.21.15.67:0/2769506345 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f204c142ac0 con 0x7f204c0c5ad0 2024-03-24T00:38:51.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.759+0000 7f2033fff640 1 -- 172.21.15.67:0/2769506345 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v18) v1 ==== 171+0+0 (secure 0 0 0) 0x7f2044059570 con 0x7f204c0c5ad0 2024-03-24T00:38:51.760 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' created 2024-03-24T00:38:51.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.761+0000 7f20520e8640 1 -- 172.21.15.67:0/2769506345 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f203806bc00 msgr2=0x7f203806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:51.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.761+0000 7f20520e8640 1 --2- 172.21.15.67:0/2769506345 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f203806bc00 0x7f203806e0c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f203c038000 tx=0x7f203c032000 comp rx=0 tx=0).stop 2024-03-24T00:38:51.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.761+0000 7f20520e8640 1 -- 172.21.15.67:0/2769506345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f204c0c5ad0 msgr2=0x7f204c0c5f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:51.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.761+0000 7f20520e8640 1 --2- 172.21.15.67:0/2769506345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f204c0c5ad0 0x7f204c0c5f50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f204c06f790 tx=0x7f204400a750 comp rx=0 tx=0).stop 2024-03-24T00:38:51.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.761+0000 7f20520e8640 1 -- 172.21.15.67:0/2769506345 shutdown_connections 2024-03-24T00:38:51.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.761+0000 7f20520e8640 1 --2- 172.21.15.67:0/2769506345 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f203806bc00 0x7f203806e0c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:51.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.761+0000 7f20520e8640 1 --2- 172.21.15.67:0/2769506345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f204c0c5ad0 0x7f204c0c5f50 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:51.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.761+0000 7f20520e8640 1 --2- 172.21.15.67:0/2769506345 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f204c06b040 0x7f204c06b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:51.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.761+0000 7f20520e8640 1 --2- 172.21.15.67:0/2769506345 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20340039b0 0x7f204c141e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:51.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.761+0000 7f20520e8640 1 -- 172.21.15.67:0/2769506345 >> 172.21.15.67:0/2769506345 conn(0x7f204c0cfa70 msgr2=0x7f204c0ceff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:51.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.761+0000 7f20520e8640 1 -- 172.21.15.67:0/2769506345 shutdown_connections 2024-03-24T00:38:51.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.762+0000 7f20520e8640 1 -- 172.21.15.67:0/2769506345 wait complete. 2024-03-24T00:38:51.783 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-24T00:38:51.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.890+0000 7f3bff7ba640 1 Processor -- start 2024-03-24T00:38:51.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.890+0000 7f3bff7ba640 1 -- start start 2024-03-24T00:38:51.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.891+0000 7f3bff7ba640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bf80b0c10 0x7f3bf80b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:51.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.891+0000 7f3bff7ba640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3bf806b040 0x7f3bf806b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:51.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.891+0000 7f3bff7ba640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bf815c220 con 0x7f3bf80afae0 2024-03-24T00:38:51.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.891+0000 7f3bff7ba640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bf80adfd0 con 0x7f3bf806b040 2024-03-24T00:38:51.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.891+0000 7f3bff7ba640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf80cc950 con 0x7f3bf80b0c10 2024-03-24T00:38:51.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.891+0000 7f3bfd556640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3bf80afae0 0x7f3bf80ad8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60690/0 (socket says 172.21.15.67:60690) 2024-03-24T00:38:51.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.891+0000 7f3bfd556640 1 -- 172.21.15.67:0/1879544171 learned_addr learned my addr 172.21.15.67:0/1879544171 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:38:51.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.891+0000 7f3bf7fff640 1 -- 172.21.15.67:0/1879544171 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3730940245 0 0) 0x7f3bf815c220 con 0x7f3bf80afae0 2024-03-24T00:38:51.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.891+0000 7f3bfcd55640 1 --2- 172.21.15.67:0/1879544171 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bf80b0c10 0x7f3bf80b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:51.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.891+0000 7f3bfdd57640 1 --2- 172.21.15.67:0/1879544171 >> v2:172.21.15.72:3300/0 conn(0x7f3bf806b040 0x7f3bf806b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:51.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.891+0000 7f3bf7fff640 1 -- 172.21.15.67:0/1879544171 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3be0003520 con 0x7f3bf80afae0 2024-03-24T00:38:51.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.892+0000 7f3bf7fff640 1 -- 172.21.15.67:0/1879544171 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2224169207 0 0) 0x7f3be0003520 con 0x7f3bf80afae0 2024-03-24T00:38:51.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.892+0000 7f3bf7fff640 1 -- 172.21.15.67:0/1879544171 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bf80b0c10 msgr2=0x7f3bf80b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:51.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.892+0000 7f3bf7fff640 1 --2- 172.21.15.67:0/1879544171 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bf80b0c10 0x7f3bf80b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:51.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.892+0000 7f3bf7fff640 1 -- 172.21.15.67:0/1879544171 >> v2:172.21.15.72:3300/0 conn(0x7f3bf806b040 msgr2=0x7f3bf806b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:51.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.892+0000 7f3bf7fff640 1 --2- 172.21.15.67:0/1879544171 >> v2:172.21.15.72:3300/0 conn(0x7f3bf806b040 0x7f3bf806b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:51.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.892+0000 7f3bf7fff640 1 -- 172.21.15.67:0/1879544171 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf80ccc30 con 0x7f3bf80afae0 2024-03-24T00:38:51.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.892+0000 7f3bf7fff640 1 -- 172.21.15.67:0/1879544171 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f3bf0001670 con 0x7f3bf80afae0 2024-03-24T00:38:51.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.892+0000 7f3bf7fff640 1 -- 172.21.15.67:0/1879544171 >> v1:172.21.15.67:6789/0 conn(0x7f3bf80afae0 legacy=0x7f3bf80ad8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:51.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.892+0000 7f3bf7fff640 1 --2- 172.21.15.67:0/1879544171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3be00039b0 0x7f3be0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:51.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.892+0000 7f3bf7fff640 1 --2- 172.21.15.67:0/1879544171 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bf80b0c10 0x7f3be0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:51.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.892+0000 7f3bf7fff640 1 --2- 172.21.15.67:0/1879544171 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3be0004d40 0x7f3be00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:51.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.892+0000 7f3bf7fff640 1 -- 172.21.15.67:0/1879544171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf80adfd0 con 0x7f3be00039b0 2024-03-24T00:38:51.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.892+0000 7f3bf7fff640 1 -- 172.21.15.67:0/1879544171 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf80cc950 con 0x7f3be0004d40 2024-03-24T00:38:51.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.892+0000 7f3bf7fff640 1 -- 172.21.15.67:0/1879544171 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3be000b9c0 con 0x7f3bf80b0c10 2024-03-24T00:38:51.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.892+0000 7f3bfd556640 1 --2- 172.21.15.67:0/1879544171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3be00039b0 0x7f3be0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:51.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.892+0000 7f3bfcd55640 1 --2- 172.21.15.67:0/1879544171 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bf80b0c10 0x7f3be0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:51.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.892+0000 7f3bfdd57640 1 --2- 172.21.15.67:0/1879544171 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3be0004d40 0x7f3be00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:51.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.892+0000 7f3bfd556640 1 -- 172.21.15.67:0/1879544171 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bf80b0c10 msgr2=0x7f3be0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:51.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.892+0000 7f3bfd556640 1 --2- 172.21.15.67:0/1879544171 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bf80b0c10 0x7f3be0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:51.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.892+0000 7f3bfd556640 1 -- 172.21.15.67:0/1879544171 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3be0004d40 msgr2=0x7f3be00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:51.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.892+0000 7f3bfd556640 1 --2- 172.21.15.67:0/1879544171 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3be0004d40 0x7f3be00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:51.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.892+0000 7f3bfd556640 1 -- 172.21.15.67:0/1879544171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3be0003520 con 0x7f3be00039b0 2024-03-24T00:38:51.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.893+0000 7f3bfd556640 1 --2- 172.21.15.67:0/1879544171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3be00039b0 0x7f3be0003dd0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3bf000c210 tx=0x7f3be0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8011788cf5bf0d05 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:51.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.893+0000 7f3bf7fff640 1 -- 172.21.15.67:0/1879544171 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf0007d50 con 0x7f3be00039b0 2024-03-24T00:38:51.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.893+0000 7f3bf7fff640 1 -- 172.21.15.67:0/1879544171 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bf0030070 con 0x7f3be00039b0 2024-03-24T00:38:51.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.893+0000 7f3bf7fff640 1 -- 172.21.15.67:0/1879544171 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf002cd80 con 0x7f3be00039b0 2024-03-24T00:38:51.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.893+0000 7f3bff7ba640 1 -- 172.21.15.67:0/1879544171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3be00039b0 msgr2=0x7f3be0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:51.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.893+0000 7f3bff7ba640 1 --2- 172.21.15.67:0/1879544171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3be00039b0 0x7f3be0003dd0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3bf000c210 tx=0x7f3be0005830 comp rx=0 tx=0).stop 2024-03-24T00:38:51.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.893+0000 7f3bfdd57640 1 -- 172.21.15.67:0/1879544171 reap_dead start 2024-03-24T00:38:51.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.893+0000 7f3bff7ba640 1 -- 172.21.15.67:0/1879544171 shutdown_connections 2024-03-24T00:38:51.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.893+0000 7f3bff7ba640 1 -- 172.21.15.67:0/1879544171 >> 172.21.15.67:0/1879544171 conn(0x7f3bf8099a30 msgr2=0x7f3bf8099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:51.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.894+0000 7f3bff7ba640 1 -- 172.21.15.67:0/1879544171 shutdown_connections 2024-03-24T00:38:51.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.894+0000 7f3bff7ba640 1 -- 172.21.15.67:0/1879544171 wait complete. 2024-03-24T00:38:51.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.895+0000 7f3bff7ba640 1 Processor -- start 2024-03-24T00:38:51.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.895+0000 7f3bff7ba640 1 -- start start 2024-03-24T00:38:51.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.895+0000 7f3bff7ba640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3be00039b0 0x7f3bf80c6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:51.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.895+0000 7f3bff7ba640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bf80b5370 0x7f3bf80b5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:51.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.895+0000 7f3bff7ba640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bf806ec70 0x7f3bf806f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:51.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.895+0000 7f3bff7ba640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf80cd4d0 con 0x7f3bf806ec70 2024-03-24T00:38:51.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.895+0000 7f3bff7ba640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf80cd1f0 con 0x7f3bf80b5370 2024-03-24T00:38:51.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.895+0000 7f3bff7ba640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf80cc600 con 0x7f3be00039b0 2024-03-24T00:38:51.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.895+0000 7f3bfdd57640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bf806ec70 0x7f3bf806f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:51.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.895+0000 7f3bfdd57640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bf806ec70 0x7f3bf806f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36066/0 (socket says 172.21.15.67:36066) 2024-03-24T00:38:51.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.895+0000 7f3bfdd57640 1 -- 172.21.15.67:0/3380508196 learned_addr learned my addr 172.21.15.67:0/3380508196 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:38:51.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.896+0000 7f3bfd556640 1 --2- 172.21.15.67:0/3380508196 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3be00039b0 0x7f3bf80c6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:51.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.896+0000 7f3bfcd55640 1 --2- 172.21.15.67:0/3380508196 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bf80b5370 0x7f3bf80b5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:51.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.896+0000 7f3bfdd57640 1 -- 172.21.15.67:0/3380508196 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3be00039b0 msgr2=0x7f3bf80c6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:51.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.896+0000 7f3bfdd57640 1 --2- 172.21.15.67:0/3380508196 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3be00039b0 0x7f3bf80c6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:51.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.896+0000 7f3bfdd57640 1 -- 172.21.15.67:0/3380508196 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bf80b5370 msgr2=0x7f3bf80b5fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:51.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.896+0000 7f3bfdd57640 1 --2- 172.21.15.67:0/3380508196 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bf80b5370 0x7f3bf80b5fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:51.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.896+0000 7f3bfdd57640 1 -- 172.21.15.67:0/3380508196 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf80b42d0 con 0x7f3bf806ec70 2024-03-24T00:38:51.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.896+0000 7f3bfdd57640 1 --2- 172.21.15.67:0/3380508196 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bf806ec70 0x7f3bf806f0f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3bf80b25a0 tx=0x7f3be800f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:51.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.896+0000 7f3bf5ffb640 1 -- 172.21.15.67:0/3380508196 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be800fde0 con 0x7f3bf806ec70 2024-03-24T00:38:51.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.896+0000 7f3bf5ffb640 1 -- 172.21.15.67:0/3380508196 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3be801d070 con 0x7f3bf806ec70 2024-03-24T00:38:51.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.896+0000 7f3bff7ba640 1 -- 172.21.15.67:0/3380508196 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bf80b45b0 con 0x7f3bf806ec70 2024-03-24T00:38:51.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.896+0000 7f3bf5ffb640 1 -- 172.21.15.67:0/3380508196 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be8015670 con 0x7f3bf806ec70 2024-03-24T00:38:51.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.896+0000 7f3bff7ba640 1 -- 172.21.15.67:0/3380508196 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bf80b4b70 con 0x7f3bf806ec70 2024-03-24T00:38:51.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.897+0000 7f3bff7ba640 1 -- 172.21.15.67:0/3380508196 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bf804f400 con 0x7f3bf806ec70 2024-03-24T00:38:51.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.898+0000 7f3bf5ffb640 1 -- 172.21.15.67:0/3380508196 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3be8015810 con 0x7f3bf806ec70 2024-03-24T00:38:51.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.898+0000 7f3bf5ffb640 1 --2- 172.21.15.67:0/3380508196 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3be406bc00 0x7f3be406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:51.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.898+0000 7f3bf5ffb640 1 -- 172.21.15.67:0/3380508196 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f3be808b260 con 0x7f3bf806ec70 2024-03-24T00:38:51.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.898+0000 7f3bfd556640 1 --2- 172.21.15.67:0/3380508196 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3be406bc00 0x7f3be406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:51.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.899+0000 7f3bfd556640 1 --2- 172.21.15.67:0/3380508196 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3be406bc00 0x7f3be406e0c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3bf0034000 tx=0x7f3bf0007000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:51.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:51.903+0000 7f3bf5ffb640 1 -- 172.21.15.67:0/3380508196 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3be8010040 con 0x7f3bf806ec70 2024-03-24T00:38:52.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.125+0000 7f3bff7ba640 1 -- 172.21.15.67:0/3380508196 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f3bf8099fc0 con 0x7f3bf806ec70 2024-03-24T00:38:52.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.760+0000 7f3bf5ffb640 1 -- 172.21.15.67:0/3380508196 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f3be8059db0 con 0x7f3bf806ec70 2024-03-24T00:38:52.761 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 2 and data pool 3 2024-03-24T00:38:52.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.763+0000 7f3bff7ba640 1 -- 172.21.15.67:0/3380508196 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3be406bc00 msgr2=0x7f3be406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:52.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.763+0000 7f3bff7ba640 1 --2- 172.21.15.67:0/3380508196 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3be406bc00 0x7f3be406e0c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3bf0034000 tx=0x7f3bf0007000 comp rx=0 tx=0).stop 2024-03-24T00:38:52.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.763+0000 7f3bff7ba640 1 -- 172.21.15.67:0/3380508196 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bf806ec70 msgr2=0x7f3bf806f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:52.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.763+0000 7f3bff7ba640 1 --2- 172.21.15.67:0/3380508196 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bf806ec70 0x7f3bf806f0f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3bf80b25a0 tx=0x7f3be800f580 comp rx=0 tx=0).stop 2024-03-24T00:38:52.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.763+0000 7f3bff7ba640 1 -- 172.21.15.67:0/3380508196 shutdown_connections 2024-03-24T00:38:52.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.763+0000 7f3bff7ba640 1 --2- 172.21.15.67:0/3380508196 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3be406bc00 0x7f3be406e0c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.763+0000 7f3bff7ba640 1 --2- 172.21.15.67:0/3380508196 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bf806ec70 0x7f3bf806f0f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.763+0000 7f3bff7ba640 1 --2- 172.21.15.67:0/3380508196 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bf80b5370 0x7f3bf80b5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.763+0000 7f3bff7ba640 1 --2- 172.21.15.67:0/3380508196 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3be00039b0 0x7f3bf80c6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.763+0000 7f3bff7ba640 1 -- 172.21.15.67:0/3380508196 >> 172.21.15.67:0/3380508196 conn(0x7f3bf8099a30 msgr2=0x7f3bf8070020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:52.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.764+0000 7f3bff7ba640 1 -- 172.21.15.67:0/3380508196 shutdown_connections 2024-03-24T00:38:52.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.764+0000 7f3bff7ba640 1 -- 172.21.15.67:0/3380508196 wait complete. 2024-03-24T00:38:52.788 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-03-24T00:38:52.788 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-03-24T00:38:52.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.897+0000 7f811b8ac640 1 Processor -- start 2024-03-24T00:38:52.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.897+0000 7f811b8ac640 1 -- start start 2024-03-24T00:38:52.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.898+0000 7f811b8ac640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f811415dfc0 0x7f81141624c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:52.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.898+0000 7f811b8ac640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f81140ced60 0x7f81140d0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:52.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.898+0000 7f811b8ac640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81140d0fa0 con 0x7f81140d0b80 2024-03-24T00:38:52.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.898+0000 7f811b8ac640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f811415e3a0 con 0x7f81140ced60 2024-03-24T00:38:52.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.898+0000 7f811b8ac640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81140cf4d0 con 0x7f811415dfc0 2024-03-24T00:38:52.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.898+0000 7f8119648640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f811415dfc0 0x7f81141624c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:52.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.898+0000 7f8118e47640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f81140ced60 0x7f81140d0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:52.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.898+0000 7f8119e49640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f81140d0b80 0x7f81140b5830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:50490/0 (socket says 172.21.15.67:50490) 2024-03-24T00:38:52.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.898+0000 7f8119e49640 1 -- 172.21.15.67:0/249688889 learned_addr learned my addr 172.21.15.67:0/249688889 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:38:52.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.898+0000 7f8118e47640 1 -- 172.21.15.67:0/249688889 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f811415dfc0 msgr2=0x7f81141624c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:52.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.898+0000 7f8118e47640 1 --2- 172.21.15.67:0/249688889 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f811415dfc0 0x7f81141624c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.898+0000 7f8118e47640 1 -- 172.21.15.67:0/249688889 >> v1:172.21.15.72:6789/0 conn(0x7f81140d0b80 legacy=0x7f81140b5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:52.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.898+0000 7f810bfff640 1 -- 172.21.15.67:0/249688889 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 661020980 0 0) 0x7f81140d0fa0 con 0x7f81140d0b80 2024-03-24T00:38:52.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.898+0000 7f8118e47640 1 -- 172.21.15.67:0/249688889 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8114162a00 con 0x7f81140ced60 2024-03-24T00:38:52.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.899+0000 7f8118e47640 1 --2- 172.21.15.67:0/249688889 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f81140ced60 0x7f81140d0640 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f81040097a0 tx=0x7f810402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd3fb064ebb00f38 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:52.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.899+0000 7f810bfff640 1 -- 172.21.15.67:0/249688889 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8104035020 con 0x7f81140ced60 2024-03-24T00:38:52.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.899+0000 7f810bfff640 1 -- 172.21.15.67:0/249688889 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8104031830 con 0x7f81140ced60 2024-03-24T00:38:52.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.899+0000 7f810bfff640 1 -- 172.21.15.67:0/249688889 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8104031c00 con 0x7f81140ced60 2024-03-24T00:38:52.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.900+0000 7f811b8ac640 1 -- 172.21.15.67:0/249688889 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f81140ced60 msgr2=0x7f81140d0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:52.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.900+0000 7f811b8ac640 1 --2- 172.21.15.67:0/249688889 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f81140ced60 0x7f81140d0640 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f81040097a0 tx=0x7f810402b8d0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.900+0000 7f811b8ac640 1 -- 172.21.15.67:0/249688889 shutdown_connections 2024-03-24T00:38:52.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.900+0000 7f811b8ac640 1 --2- 172.21.15.67:0/249688889 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f81140ced60 0x7f81140d0640 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.900+0000 7f811b8ac640 1 --2- 172.21.15.67:0/249688889 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f811415dfc0 0x7f81141624c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.900+0000 7f811b8ac640 1 -- 172.21.15.67:0/249688889 >> 172.21.15.67:0/249688889 conn(0x7f811406c880 msgr2=0x7f811406cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:52.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.900+0000 7f811b8ac640 1 -- 172.21.15.67:0/249688889 shutdown_connections 2024-03-24T00:38:52.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.900+0000 7f811b8ac640 1 -- 172.21.15.67:0/249688889 wait complete. 2024-03-24T00:38:52.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.901+0000 7f811b8ac640 1 Processor -- start 2024-03-24T00:38:52.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.901+0000 7f811b8ac640 1 -- start start 2024-03-24T00:38:52.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.902+0000 7f811b8ac640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f81140d0b80 0x7f81141f41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:52.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.902+0000 7f811b8ac640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f811415dfc0 0x7f81141532f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:52.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.902+0000 7f811b8ac640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f811415fb00 0x7f811415e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:52.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.902+0000 7f811b8ac640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81140d2300 con 0x7f811415dfc0 2024-03-24T00:38:52.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.902+0000 7f811b8ac640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81140d2020 con 0x7f811415fb00 2024-03-24T00:38:52.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.902+0000 7f811b8ac640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81140d25e0 con 0x7f81140d0b80 2024-03-24T00:38:52.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.902+0000 7f8118e47640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f811415dfc0 0x7f81141532f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:52.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.902+0000 7f8118e47640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f811415dfc0 0x7f81141532f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36080/0 (socket says 172.21.15.67:36080) 2024-03-24T00:38:52.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.902+0000 7f8118e47640 1 -- 172.21.15.67:0/3577191442 learned_addr learned my addr 172.21.15.67:0/3577191442 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:38:52.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.902+0000 7f8119648640 1 --2- 172.21.15.67:0/3577191442 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f81140d0b80 0x7f81141f41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:52.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.902+0000 7f8119e49640 1 --2- 172.21.15.67:0/3577191442 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f811415fb00 0x7f811415e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:52.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.902+0000 7f8118e47640 1 -- 172.21.15.67:0/3577191442 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f81140d0b80 msgr2=0x7f81141f41a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:52.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.902+0000 7f8118e47640 1 --2- 172.21.15.67:0/3577191442 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f81140d0b80 0x7f81141f41a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.902+0000 7f8118e47640 1 -- 172.21.15.67:0/3577191442 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f811415fb00 msgr2=0x7f811415e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:52.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.902+0000 7f8118e47640 1 --2- 172.21.15.67:0/3577191442 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f811415fb00 0x7f811415e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.902+0000 7f8118e47640 1 -- 172.21.15.67:0/3577191442 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81141f46e0 con 0x7f811415dfc0 2024-03-24T00:38:52.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.902+0000 7f8118e47640 1 --2- 172.21.15.67:0/3577191442 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f811415dfc0 0x7f81141532f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f81040098c0 tx=0x7f81040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:52.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.903+0000 7f8109ffb640 1 -- 172.21.15.67:0/3577191442 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8104035070 con 0x7f811415dfc0 2024-03-24T00:38:52.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.903+0000 7f8109ffb640 1 -- 172.21.15.67:0/3577191442 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8104004230 con 0x7f811415dfc0 2024-03-24T00:38:52.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.903+0000 7f811b8ac640 1 -- 172.21.15.67:0/3577191442 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8114153830 con 0x7f811415dfc0 2024-03-24T00:38:52.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.903+0000 7f8109ffb640 1 -- 172.21.15.67:0/3577191442 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f810403a630 con 0x7f811415dfc0 2024-03-24T00:38:52.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.903+0000 7f811b8ac640 1 -- 172.21.15.67:0/3577191442 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8114153de0 con 0x7f811415dfc0 2024-03-24T00:38:52.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.904+0000 7f811b8ac640 1 -- 172.21.15.67:0/3577191442 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81141530f0 con 0x7f811415dfc0 2024-03-24T00:38:52.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.904+0000 7f8109ffb640 1 -- 172.21.15.67:0/3577191442 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8104038020 con 0x7f811415dfc0 2024-03-24T00:38:52.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.904+0000 7f8109ffb640 1 --2- 172.21.15.67:0/3577191442 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f80f806bc00 0x7f80f806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:52.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.905+0000 7f8109ffb640 1 -- 172.21.15.67:0/3577191442 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f81040ac560 con 0x7f811415dfc0 2024-03-24T00:38:52.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.905+0000 7f8119648640 1 --2- 172.21.15.67:0/3577191442 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f80f806bc00 0x7f80f806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:52.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.905+0000 7f8119648640 1 --2- 172.21.15.67:0/3577191442 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f80f806bc00 0x7f80f806e0c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8110009aa0 tx=0x7f8110009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:52.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:52.909+0000 7f8109ffb640 1 -- 172.21.15.67:0/3577191442 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f81141530f0 con 0x7f811415dfc0 2024-03-24T00:38:53.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.099+0000 7f811b8ac640 1 -- 172.21.15.67:0/3577191442 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7f811404f400 con 0x7f811415dfc0 2024-03-24T00:38:53.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.774+0000 7f8109ffb640 1 -- 172.21.15.67:0/3577191442 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v20) v1 ==== 157+0+0 (secure 0 0 0) 0x7f8104040070 con 0x7f811415dfc0 2024-03-24T00:38:53.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.777+0000 7f811b8ac640 1 -- 172.21.15.67:0/3577191442 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f80f806bc00 msgr2=0x7f80f806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:53.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.777+0000 7f811b8ac640 1 --2- 172.21.15.67:0/3577191442 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f80f806bc00 0x7f80f806e0c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8110009aa0 tx=0x7f8110009190 comp rx=0 tx=0).stop 2024-03-24T00:38:53.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.777+0000 7f811b8ac640 1 -- 172.21.15.67:0/3577191442 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f811415dfc0 msgr2=0x7f81141532f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:53.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.777+0000 7f811b8ac640 1 --2- 172.21.15.67:0/3577191442 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f811415dfc0 0x7f81141532f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f81040098c0 tx=0x7f81040046c0 comp rx=0 tx=0).stop 2024-03-24T00:38:53.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.777+0000 7f811b8ac640 1 -- 172.21.15.67:0/3577191442 shutdown_connections 2024-03-24T00:38:53.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.777+0000 7f811b8ac640 1 --2- 172.21.15.67:0/3577191442 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f80f806bc00 0x7f80f806e0c0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:53.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.777+0000 7f811b8ac640 1 --2- 172.21.15.67:0/3577191442 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f811415fb00 0x7f811415e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:53.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.777+0000 7f811b8ac640 1 --2- 172.21.15.67:0/3577191442 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f811415dfc0 0x7f81141532f0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:53.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.777+0000 7f811b8ac640 1 --2- 172.21.15.67:0/3577191442 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f81140d0b80 0x7f81141f41a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:53.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.777+0000 7f811b8ac640 1 -- 172.21.15.67:0/3577191442 >> 172.21.15.67:0/3577191442 conn(0x7f811406c880 msgr2=0x7f81141529e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:53.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.778+0000 7f811b8ac640 1 -- 172.21.15.67:0/3577191442 shutdown_connections 2024-03-24T00:38:53.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.778+0000 7f811b8ac640 1 -- 172.21.15.67:0/3577191442 wait complete. 2024-03-24T00:38:53.792 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-03-24T00:38:53.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.907+0000 7f81a5f05640 1 Processor -- start 2024-03-24T00:38:53.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.907+0000 7f81a5f05640 1 -- start start 2024-03-24T00:38:53.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.908+0000 7f81a5f05640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f81a015e090 0x7f81a00b7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:53.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.908+0000 7f81a5f05640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f81a00d06e0 0x7f81a00d0af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:53.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.908+0000 7f81a5f05640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81a00d24b0 con 0x7f81a0161c60 2024-03-24T00:38:53.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.908+0000 7f81a5f05640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81a00cf740 con 0x7f81a00d06e0 2024-03-24T00:38:53.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.908+0000 7f81a5f05640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81a00ceea0 con 0x7f81a015e090 2024-03-24T00:38:53.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.908+0000 7f819ffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f81a00d06e0 0x7f81a00d0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:53.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.908+0000 7f819effd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f81a0161c60 0x7f81a00ce650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56820/0 (socket says 172.21.15.67:56820) 2024-03-24T00:38:53.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.908+0000 7f819effd640 1 -- 172.21.15.67:0/1258153819 learned_addr learned my addr 172.21.15.67:0/1258153819 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:38:53.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.908+0000 7f819f7fe640 1 --2- 172.21.15.67:0/1258153819 >> v2:172.21.15.72:3300/0 conn(0x7f81a015e090 0x7f81a00b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:53.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.908+0000 7f819ffff640 1 -- 172.21.15.67:0/1258153819 >> v2:172.21.15.72:3300/0 conn(0x7f81a015e090 msgr2=0x7f81a00b7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:53.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.908+0000 7f819ffff640 1 --2- 172.21.15.67:0/1258153819 >> v2:172.21.15.72:3300/0 conn(0x7f81a015e090 0x7f81a00b7120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:53.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.908+0000 7f819ffff640 1 -- 172.21.15.67:0/1258153819 >> v1:172.21.15.67:6789/0 conn(0x7f81a0161c60 legacy=0x7f81a00ce650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:53.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.908+0000 7f819e7fc640 1 -- 172.21.15.67:0/1258153819 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 839946016 0 0) 0x7f81a00d24b0 con 0x7f81a0161c60 2024-03-24T00:38:53.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.908+0000 7f819ffff640 1 -- 172.21.15.67:0/1258153819 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81a00cf180 con 0x7f81a00d06e0 2024-03-24T00:38:53.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.909+0000 7f819ffff640 1 --2- 172.21.15.67:0/1258153819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81a00d06e0 0x7f81a00d0af0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f81a008c3d0 tx=0x7f819802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b64074dfaa96112 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:53.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.909+0000 7f819e7fc640 1 -- 172.21.15.67:0/1258153819 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8198039070 con 0x7f81a00d06e0 2024-03-24T00:38:53.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.909+0000 7f819e7fc640 1 -- 172.21.15.67:0/1258153819 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f819802cc80 con 0x7f81a00d06e0 2024-03-24T00:38:53.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.909+0000 7f819e7fc640 1 -- 172.21.15.67:0/1258153819 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8198034660 con 0x7f81a00d06e0 2024-03-24T00:38:53.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.910+0000 7f81a5f05640 1 -- 172.21.15.67:0/1258153819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81a00d06e0 msgr2=0x7f81a00d0af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:53.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.910+0000 7f81a5f05640 1 --2- 172.21.15.67:0/1258153819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81a00d06e0 0x7f81a00d0af0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f81a008c3d0 tx=0x7f819802bed0 comp rx=0 tx=0).stop 2024-03-24T00:38:53.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.910+0000 7f81a5f05640 1 -- 172.21.15.67:0/1258153819 shutdown_connections 2024-03-24T00:38:53.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.910+0000 7f81a5f05640 1 --2- 172.21.15.67:0/1258153819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81a00d06e0 0x7f81a00d0af0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:53.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.910+0000 7f81a5f05640 1 --2- 172.21.15.67:0/1258153819 >> v2:172.21.15.72:3300/0 conn(0x7f81a015e090 0x7f81a00b7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:53.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.910+0000 7f81a5f05640 1 -- 172.21.15.67:0/1258153819 >> 172.21.15.67:0/1258153819 conn(0x7f81a006c830 msgr2=0x7f81a006cc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:53.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.910+0000 7f81a5f05640 1 -- 172.21.15.67:0/1258153819 shutdown_connections 2024-03-24T00:38:53.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.910+0000 7f81a5f05640 1 -- 172.21.15.67:0/1258153819 wait complete. 2024-03-24T00:38:53.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.911+0000 7f81a5f05640 1 Processor -- start 2024-03-24T00:38:53.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.911+0000 7f81a5f05640 1 -- start start 2024-03-24T00:38:53.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.912+0000 7f81a5f05640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81a00d06e0 0x7f81a01592c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:53.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.912+0000 7f81a5f05640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f81a015e090 0x7f81a015efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:53.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.912+0000 7f81a5f05640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f81a0161c60 0x7f81a00ccc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:53.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.912+0000 7f81a5f05640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81a015e470 con 0x7f81a00d06e0 2024-03-24T00:38:53.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.912+0000 7f81a5f05640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81a00d1030 con 0x7f81a0161c60 2024-03-24T00:38:53.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.912+0000 7f81a5f05640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81a00d2990 con 0x7f81a015e090 2024-03-24T00:38:53.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.912+0000 7f819f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81a00d06e0 0x7f81a01592c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:53.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.912+0000 7f819f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81a00d06e0 0x7f81a01592c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59546/0 (socket says 172.21.15.67:59546) 2024-03-24T00:38:53.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.912+0000 7f819f7fe640 1 -- 172.21.15.67:0/2716987451 learned_addr learned my addr 172.21.15.67:0/2716987451 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:38:53.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.912+0000 7f819ffff640 1 --2- 172.21.15.67:0/2716987451 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f81a0161c60 0x7f81a00ccc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:53.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.912+0000 7f819effd640 1 --2- 172.21.15.67:0/2716987451 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f81a015e090 0x7f81a015efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:53.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.912+0000 7f819f7fe640 1 -- 172.21.15.67:0/2716987451 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f81a015e090 msgr2=0x7f81a015efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:53.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.912+0000 7f819f7fe640 1 --2- 172.21.15.67:0/2716987451 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f81a015e090 0x7f81a015efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:53.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.912+0000 7f819f7fe640 1 -- 172.21.15.67:0/2716987451 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f81a0161c60 msgr2=0x7f81a00ccc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:53.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.912+0000 7f819f7fe640 1 --2- 172.21.15.67:0/2716987451 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f81a0161c60 0x7f81a00ccc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:53.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.912+0000 7f819f7fe640 1 -- 172.21.15.67:0/2716987451 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81a015f930 con 0x7f81a00d06e0 2024-03-24T00:38:53.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.912+0000 7f819f7fe640 1 --2- 172.21.15.67:0/2716987451 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81a00d06e0 0x7f81a01592c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f819000e9b0 tx=0x7f819000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:53.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.912+0000 7f8187fff640 1 -- 172.21.15.67:0/2716987451 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8190017070 con 0x7f81a00d06e0 2024-03-24T00:38:53.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.912+0000 7f8187fff640 1 -- 172.21.15.67:0/2716987451 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f819000c9a0 con 0x7f81a00d06e0 2024-03-24T00:38:53.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.912+0000 7f81a5f05640 1 -- 172.21.15.67:0/2716987451 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81a015fc10 con 0x7f81a00d06e0 2024-03-24T00:38:53.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.912+0000 7f8187fff640 1 -- 172.21.15.67:0/2716987451 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f819000cd90 con 0x7f81a00d06e0 2024-03-24T00:38:53.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.912+0000 7f81a5f05640 1 -- 172.21.15.67:0/2716987451 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81a015a8f0 con 0x7f81a00d06e0 2024-03-24T00:38:53.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.913+0000 7f81a5f05640 1 -- 172.21.15.67:0/2716987451 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81a0160000 con 0x7f81a00d06e0 2024-03-24T00:38:53.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.914+0000 7f8187fff640 1 -- 172.21.15.67:0/2716987451 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f819001a070 con 0x7f81a00d06e0 2024-03-24T00:38:53.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.914+0000 7f8187fff640 1 --2- 172.21.15.67:0/2716987451 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f818c06bbd0 0x7f818c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:53.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.915+0000 7f8187fff640 1 -- 172.21.15.67:0/2716987451 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6007+0+0 (secure 0 0 0) 0x7f819008aed0 con 0x7f81a00d06e0 2024-03-24T00:38:53.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.915+0000 7f819effd640 1 --2- 172.21.15.67:0/2716987451 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f818c06bbd0 0x7f818c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:53.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.915+0000 7f819effd640 1 --2- 172.21.15.67:0/2716987451 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f818c06bbd0 0x7f818c06e090 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f81a00b6890 tx=0x7f8194009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:53.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:53.919+0000 7f8187fff640 1 -- 172.21.15.67:0/2716987451 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8190059240 con 0x7f81a00d06e0 2024-03-24T00:38:54.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:54.100+0000 7f81a5f05640 1 -- 172.21.15.67:0/2716987451 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f81a004f400 con 0x7f81a00d06e0 2024-03-24T00:38:55.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.790+0000 7f8187fff640 1 -- 172.21.15.67:0/2716987451 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v22) v1 ==== 228+0+0 (secure 0 0 0) 0x7f819005d2e0 con 0x7f81a00d06e0 2024-03-24T00:38:55.791 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data_ec' created 2024-03-24T00:38:55.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.794+0000 7f81a5f05640 1 -- 172.21.15.67:0/2716987451 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f818c06bbd0 msgr2=0x7f818c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:55.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.794+0000 7f81a5f05640 1 --2- 172.21.15.67:0/2716987451 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f818c06bbd0 0x7f818c06e090 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f81a00b6890 tx=0x7f8194009040 comp rx=0 tx=0).stop 2024-03-24T00:38:55.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.794+0000 7f81a5f05640 1 -- 172.21.15.67:0/2716987451 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81a00d06e0 msgr2=0x7f81a01592c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:55.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.794+0000 7f81a5f05640 1 --2- 172.21.15.67:0/2716987451 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81a00d06e0 0x7f81a01592c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f819000e9b0 tx=0x7f819000ee70 comp rx=0 tx=0).stop 2024-03-24T00:38:55.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.794+0000 7f81a5f05640 1 -- 172.21.15.67:0/2716987451 shutdown_connections 2024-03-24T00:38:55.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.794+0000 7f81a5f05640 1 --2- 172.21.15.67:0/2716987451 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f818c06bbd0 0x7f818c06e090 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:55.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.794+0000 7f81a5f05640 1 --2- 172.21.15.67:0/2716987451 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f81a0161c60 0x7f81a00ccc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:55.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.794+0000 7f81a5f05640 1 --2- 172.21.15.67:0/2716987451 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f81a015e090 0x7f81a015efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:55.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.794+0000 7f81a5f05640 1 --2- 172.21.15.67:0/2716987451 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81a00d06e0 0x7f81a01592c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:55.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.794+0000 7f81a5f05640 1 -- 172.21.15.67:0/2716987451 >> 172.21.15.67:0/2716987451 conn(0x7f81a006c830 msgr2=0x7f81a00b4360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:55.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.795+0000 7f81a5f05640 1 -- 172.21.15.67:0/2716987451 shutdown_connections 2024-03-24T00:38:55.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.795+0000 7f81a5f05640 1 -- 172.21.15.67:0/2716987451 wait complete. 2024-03-24T00:38:55.813 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-03-24T00:38:55.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.923+0000 7fa4c09cd640 1 Processor -- start 2024-03-24T00:38:55.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.923+0000 7fa4c09cd640 1 -- start start 2024-03-24T00:38:55.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.924+0000 7fa4c09cd640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa4bc0add20 0x7fa4bc06b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:55.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.924+0000 7fa4c09cd640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4bc162b50 con 0x7fa4bc0ad8a0 2024-03-24T00:38:55.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.924+0000 7fa4c09cd640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4bc0cfc00 con 0x7fa4bc15e050 2024-03-24T00:38:55.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.924+0000 7fa4c09cd640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4bc0cfee0 con 0x7fa4bc0add20 2024-03-24T00:38:55.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.924+0000 7fa4b9d9b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa4bc0ad8a0 0x7fa4bc0d0f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56834/0 (socket says 172.21.15.67:56834) 2024-03-24T00:38:55.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.924+0000 7fa4b9d9b640 1 -- 172.21.15.67:0/1295229333 learned_addr learned my addr 172.21.15.67:0/1295229333 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:38:55.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.924+0000 7fa4bad9d640 1 --2- 172.21.15.67:0/1295229333 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa4bc0add20 0x7fa4bc06b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:55.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.925+0000 7fa4b959a640 1 -- 172.21.15.67:0/1295229333 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3990807696 0 0) 0x7fa4bc162b50 con 0x7fa4bc0ad8a0 2024-03-24T00:38:55.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.925+0000 7fa4b959a640 1 -- 172.21.15.67:0/1295229333 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4a8003520 con 0x7fa4bc0ad8a0 2024-03-24T00:38:55.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.925+0000 7fa4b959a640 1 -- 172.21.15.67:0/1295229333 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2058758280 0 0) 0x7fa4bc0cfc00 con 0x7fa4bc15e050 2024-03-24T00:38:55.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.925+0000 7fa4b959a640 1 -- 172.21.15.67:0/1295229333 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4bc162b50 con 0x7fa4bc15e050 2024-03-24T00:38:55.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.925+0000 7fa4bad9d640 1 -- 172.21.15.67:0/1295229333 >> v1:172.21.15.72:6789/0 conn(0x7fa4bc15e050 legacy=0x7fa4bc0afb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:55.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.925+0000 7fa4bad9d640 1 -- 172.21.15.67:0/1295229333 >> v1:172.21.15.67:6789/0 conn(0x7fa4bc0ad8a0 legacy=0x7fa4bc0d0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:55.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.925+0000 7fa4bad9d640 1 -- 172.21.15.67:0/1295229333 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4bc0d01c0 con 0x7fa4bc0add20 2024-03-24T00:38:55.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.926+0000 7fa4bad9d640 1 --2- 172.21.15.67:0/1295229333 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa4bc0add20 0x7fa4bc06b040 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac009dc0 tx=0x7fa4ac02bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1dbe5da582f32280 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:55.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.926+0000 7fa4b959a640 1 -- 172.21.15.67:0/1295229333 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ac039070 con 0x7fa4bc0add20 2024-03-24T00:38:55.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.926+0000 7fa4b959a640 1 -- 172.21.15.67:0/1295229333 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4ac02ccd0 con 0x7fa4bc0add20 2024-03-24T00:38:55.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.926+0000 7fa4b959a640 1 -- 172.21.15.67:0/1295229333 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ac034640 con 0x7fa4bc0add20 2024-03-24T00:38:55.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.926+0000 7fa4c09cd640 1 -- 172.21.15.67:0/1295229333 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa4bc0add20 msgr2=0x7fa4bc06b040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:55.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.926+0000 7fa4c09cd640 1 --2- 172.21.15.67:0/1295229333 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa4bc0add20 0x7fa4bc06b040 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac009dc0 tx=0x7fa4ac02bf20 comp rx=0 tx=0).stop 2024-03-24T00:38:55.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.926+0000 7fa4c09cd640 1 -- 172.21.15.67:0/1295229333 shutdown_connections 2024-03-24T00:38:55.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.926+0000 7fa4c09cd640 1 --2- 172.21.15.67:0/1295229333 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa4bc0add20 0x7fa4bc06b040 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:55.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.926+0000 7fa4c09cd640 1 -- 172.21.15.67:0/1295229333 >> 172.21.15.67:0/1295229333 conn(0x7fa4bc099a30 msgr2=0x7fa4bc099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:55.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.926+0000 7fa4c09cd640 1 -- 172.21.15.67:0/1295229333 shutdown_connections 2024-03-24T00:38:55.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.926+0000 7fa4c09cd640 1 -- 172.21.15.67:0/1295229333 wait complete. 2024-03-24T00:38:55.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.928+0000 7fa4c09cd640 1 Processor -- start 2024-03-24T00:38:55.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.928+0000 7fa4c09cd640 1 -- start start 2024-03-24T00:38:55.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.928+0000 7fa4c09cd640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa4bc0ad8a0 0x7fa4bc0c8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:55.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.928+0000 7fa4c09cd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa4bc0add20 0x7fa4bc0ca560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:55.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.928+0000 7fa4c09cd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4bc15e050 0x7fa4bc15a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:55.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.928+0000 7fa4c09cd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4bc15f440 con 0x7fa4bc15e050 2024-03-24T00:38:55.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.928+0000 7fa4c09cd640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4bc0d0780 con 0x7fa4bc0add20 2024-03-24T00:38:55.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.928+0000 7fa4c09cd640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4bc0b0270 con 0x7fa4bc0ad8a0 2024-03-24T00:38:55.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.928+0000 7fa4bad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4bc15e050 0x7fa4bc15a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:55.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.928+0000 7fa4b9d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa4bc0add20 0x7fa4bc0ca560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:55.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.928+0000 7fa4ba59c640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa4bc0ad8a0 0x7fa4bc0c8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:55.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.928+0000 7fa4bad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4bc15e050 0x7fa4bc15a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59550/0 (socket says 172.21.15.67:59550) 2024-03-24T00:38:55.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.928+0000 7fa4bad9d640 1 -- 172.21.15.67:0/4175333249 learned_addr learned my addr 172.21.15.67:0/4175333249 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:38:55.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.929+0000 7fa4ba59c640 1 -- 172.21.15.67:0/4175333249 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa4bc0add20 msgr2=0x7fa4bc0ca560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:55.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.929+0000 7fa4ba59c640 1 --2- 172.21.15.67:0/4175333249 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa4bc0add20 0x7fa4bc0ca560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:55.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.929+0000 7fa4ba59c640 1 -- 172.21.15.67:0/4175333249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4bc15e050 msgr2=0x7fa4bc15a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:55.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.929+0000 7fa4ba59c640 1 --2- 172.21.15.67:0/4175333249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4bc15e050 0x7fa4bc15a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:55.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.929+0000 7fa4ba59c640 1 -- 172.21.15.67:0/4175333249 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4bc1f4080 con 0x7fa4bc0ad8a0 2024-03-24T00:38:55.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.929+0000 7fa4bad9d640 1 --2- 172.21.15.67:0/4175333249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4bc15e050 0x7fa4bc15a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:55.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.929+0000 7fa4ba59c640 1 --2- 172.21.15.67:0/4175333249 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa4bc0ad8a0 0x7fa4bc0c8bf0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa4b400c430 tx=0x7fa4b400c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:55.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.929+0000 7fa4b2ffd640 1 -- 172.21.15.67:0/4175333249 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b4017070 con 0x7fa4bc0ad8a0 2024-03-24T00:38:55.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.929+0000 7fa4b2ffd640 1 -- 172.21.15.67:0/4175333249 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4b40052d0 con 0x7fa4bc0ad8a0 2024-03-24T00:38:55.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.929+0000 7fa4c09cd640 1 -- 172.21.15.67:0/4175333249 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4bc1f4360 con 0x7fa4bc0ad8a0 2024-03-24T00:38:55.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.929+0000 7fa4b2ffd640 1 -- 172.21.15.67:0/4175333249 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b4003830 con 0x7fa4bc0ad8a0 2024-03-24T00:38:55.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.929+0000 7fa4c09cd640 1 -- 172.21.15.67:0/4175333249 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4bc1f4920 con 0x7fa4bc0ad8a0 2024-03-24T00:38:55.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.930+0000 7fa4c09cd640 1 -- 172.21.15.67:0/4175333249 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4bc04f400 con 0x7fa4bc0ad8a0 2024-03-24T00:38:55.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.931+0000 7fa4b2ffd640 1 -- 172.21.15.67:0/4175333249 <== mon.2 v2:172.21.15.72:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa4b401a070 con 0x7fa4bc0ad8a0 2024-03-24T00:38:55.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.931+0000 7fa4b2ffd640 1 --2- 172.21.15.67:0/4175333249 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa49c06bbd0 0x7fa49c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:55.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.931+0000 7fa4b2ffd640 1 -- 172.21.15.67:0/4175333249 <== mon.2 v2:172.21.15.72:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa4b408a050 con 0x7fa4bc0ad8a0 2024-03-24T00:38:55.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.931+0000 7fa4b9d9b640 1 --2- 172.21.15.67:0/4175333249 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa49c06bbd0 0x7fa49c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:55.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.932+0000 7fa4b9d9b640 1 --2- 172.21.15.67:0/4175333249 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa49c06bbd0 0x7fa49c06e090 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa4a4000f60 tx=0x7fa4a4007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:55.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:55.936+0000 7fa4b2ffd640 1 -- 172.21.15.67:0/4175333249 <== mon.2 v2:172.21.15.72:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa4b4058260 con 0x7fa4bc0ad8a0 2024-03-24T00:38:56.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.115+0000 7fa4c09cd640 1 -- 172.21.15.67:0/4175333249 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7fa4bc0cd680 con 0x7fa4bc0ad8a0 2024-03-24T00:38:56.547 INFO:tasks.ceph.mds.b.smithi072.stderr:2024-03-24T00:38:56.547+0000 7f1f213af640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T00:38:56.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.797+0000 7fa4b2ffd640 1 -- 172.21.15.67:0/4175333249 <== mon.2 v2:172.21.15.72:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v23) v1 ==== 169+0+0 (secure 0 0 0) 0x7fa4b405c300 con 0x7fa4bc0ad8a0 2024-03-24T00:38:56.798 INFO:teuthology.orchestra.run.smithi067.stderr:set pool 4 allow_ec_overwrites to true 2024-03-24T00:38:56.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.803+0000 7fa4c09cd640 1 -- 172.21.15.67:0/4175333249 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa49c06bbd0 msgr2=0x7fa49c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:56.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.803+0000 7fa4c09cd640 1 --2- 172.21.15.67:0/4175333249 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa49c06bbd0 0x7fa49c06e090 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa4a4000f60 tx=0x7fa4a4007d50 comp rx=0 tx=0).stop 2024-03-24T00:38:56.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.803+0000 7fa4c09cd640 1 -- 172.21.15.67:0/4175333249 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa4bc0ad8a0 msgr2=0x7fa4bc0c8bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:56.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.803+0000 7fa4c09cd640 1 --2- 172.21.15.67:0/4175333249 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa4bc0ad8a0 0x7fa4bc0c8bf0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa4b400c430 tx=0x7fa4b400c8f0 comp rx=0 tx=0).stop 2024-03-24T00:38:56.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.803+0000 7fa4c09cd640 1 -- 172.21.15.67:0/4175333249 shutdown_connections 2024-03-24T00:38:56.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.803+0000 7fa4c09cd640 1 --2- 172.21.15.67:0/4175333249 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa49c06bbd0 0x7fa49c06e090 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:56.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.803+0000 7fa4c09cd640 1 --2- 172.21.15.67:0/4175333249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4bc15e050 0x7fa4bc15a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:56.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.803+0000 7fa4c09cd640 1 --2- 172.21.15.67:0/4175333249 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa4bc0add20 0x7fa4bc0ca560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:56.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.803+0000 7fa4c09cd640 1 --2- 172.21.15.67:0/4175333249 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa4bc0ad8a0 0x7fa4bc0c8bf0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:56.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.803+0000 7fa4c09cd640 1 -- 172.21.15.67:0/4175333249 >> 172.21.15.67:0/4175333249 conn(0x7fa4bc099a30 msgr2=0x7fa4bc15ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:56.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.812+0000 7fa4c09cd640 1 -- 172.21.15.67:0/4175333249 shutdown_connections 2024-03-24T00:38:56.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.812+0000 7fa4c09cd640 1 -- 172.21.15.67:0/4175333249 wait complete. 2024-03-24T00:38:56.833 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-03-24T00:38:56.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.939+0000 7f2b921e6640 1 Processor -- start 2024-03-24T00:38:56.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.939+0000 7f2b921e6640 1 -- start start 2024-03-24T00:38:56.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.939+0000 7f2b921e6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2b8c0b9690 0x7f2b8c0ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:56.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.939+0000 7f2b921e6640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2b8c0cef30 0x7f2b8c0d2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:56.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.939+0000 7f2b921e6640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b8c162290 con 0x7f2b8c161370 2024-03-24T00:38:56.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.939+0000 7f2b921e6640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b8c0b6e10 con 0x7f2b8c0cef30 2024-03-24T00:38:56.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.939+0000 7f2b921e6640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8c0d2190 con 0x7f2b8c0b9690 2024-03-24T00:38:56.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.940+0000 7f2b8bfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2b8c0cef30 0x7f2b8c0d2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:56.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.940+0000 7f2b8b7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2b8c161370 0x7f2b8c161ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56846/0 (socket says 172.21.15.67:56846) 2024-03-24T00:38:56.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.940+0000 7f2b8b7fe640 1 -- 172.21.15.67:0/448005299 learned_addr learned my addr 172.21.15.67:0/448005299 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:38:56.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.940+0000 7f2b8affd640 1 --2- 172.21.15.67:0/448005299 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2b8c0b9690 0x7f2b8c0ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:56.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.940+0000 7f2b8bfff640 1 -- 172.21.15.67:0/448005299 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2b8c0b9690 msgr2=0x7f2b8c0ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:56.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.940+0000 7f2b8bfff640 1 --2- 172.21.15.67:0/448005299 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2b8c0b9690 0x7f2b8c0ce9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:56.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.940+0000 7f2b8bfff640 1 -- 172.21.15.67:0/448005299 >> v1:172.21.15.67:6789/0 conn(0x7f2b8c161370 legacy=0x7f2b8c161ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:56.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.940+0000 7f2b8bfff640 1 -- 172.21.15.67:0/448005299 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b8c162570 con 0x7f2b8c0cef30 2024-03-24T00:38:56.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.940+0000 7f2b8a7fc640 1 -- 172.21.15.67:0/448005299 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2650779799 0 0) 0x7f2b8c162290 con 0x7f2b8c161370 2024-03-24T00:38:56.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.941+0000 7f2b8bfff640 1 --2- 172.21.15.67:0/448005299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b8c0cef30 0x7f2b8c0d2620 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2b84009dc0 tx=0x7f2b8402bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb4e086c12e1c8c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:56.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.941+0000 7f2b8a7fc640 1 -- 172.21.15.67:0/448005299 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b84039070 con 0x7f2b8c0cef30 2024-03-24T00:38:56.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.941+0000 7f2b8a7fc640 1 -- 172.21.15.67:0/448005299 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b8402ccd0 con 0x7f2b8c0cef30 2024-03-24T00:38:56.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.941+0000 7f2b8a7fc640 1 -- 172.21.15.67:0/448005299 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b84034660 con 0x7f2b8c0cef30 2024-03-24T00:38:56.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.941+0000 7f2b921e6640 1 -- 172.21.15.67:0/448005299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b8c0cef30 msgr2=0x7f2b8c0d2620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:56.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.941+0000 7f2b921e6640 1 --2- 172.21.15.67:0/448005299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b8c0cef30 0x7f2b8c0d2620 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2b84009dc0 tx=0x7f2b8402bf20 comp rx=0 tx=0).stop 2024-03-24T00:38:56.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.941+0000 7f2b921e6640 1 -- 172.21.15.67:0/448005299 shutdown_connections 2024-03-24T00:38:56.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.941+0000 7f2b921e6640 1 --2- 172.21.15.67:0/448005299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b8c0cef30 0x7f2b8c0d2620 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:56.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.941+0000 7f2b921e6640 1 --2- 172.21.15.67:0/448005299 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2b8c0b9690 0x7f2b8c0ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:56.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.941+0000 7f2b921e6640 1 -- 172.21.15.67:0/448005299 >> 172.21.15.67:0/448005299 conn(0x7f2b8c06c880 msgr2=0x7f2b8c06cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:56.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.941+0000 7f2b921e6640 1 -- 172.21.15.67:0/448005299 shutdown_connections 2024-03-24T00:38:56.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.941+0000 7f2b921e6640 1 -- 172.21.15.67:0/448005299 wait complete. 2024-03-24T00:38:56.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.942+0000 7f2b921e6640 1 Processor -- start 2024-03-24T00:38:56.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.943+0000 7f2b921e6640 1 -- start start 2024-03-24T00:38:56.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.943+0000 7f2b921e6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b8c0b9690 0x7f2b8c155ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:56.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.943+0000 7f2b921e6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2b8c0cef30 0x7f2b8c158040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:56.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.943+0000 7f2b921e6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b8c161370 0x7f2b8c143c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:56.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.943+0000 7f2b921e6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8c0d2b60 con 0x7f2b8c0b9690 2024-03-24T00:38:56.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.943+0000 7f2b921e6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8c161750 con 0x7f2b8c161370 2024-03-24T00:38:56.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.943+0000 7f2b921e6640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8c0d2ce0 con 0x7f2b8c0cef30 2024-03-24T00:38:56.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.943+0000 7f2b8b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b8c0b9690 0x7f2b8c155ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:56.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.943+0000 7f2b8b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b8c0b9690 0x7f2b8c155ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59572/0 (socket says 172.21.15.67:59572) 2024-03-24T00:38:56.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.943+0000 7f2b8b7fe640 1 -- 172.21.15.67:0/1112290424 learned_addr learned my addr 172.21.15.67:0/1112290424 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:38:56.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.943+0000 7f2b8affd640 1 --2- 172.21.15.67:0/1112290424 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2b8c0cef30 0x7f2b8c158040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:56.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.943+0000 7f2b8bfff640 1 --2- 172.21.15.67:0/1112290424 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b8c161370 0x7f2b8c143c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:56.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.944+0000 7f2b8b7fe640 1 -- 172.21.15.67:0/1112290424 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2b8c0cef30 msgr2=0x7f2b8c158040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:56.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.944+0000 7f2b8b7fe640 1 --2- 172.21.15.67:0/1112290424 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2b8c0cef30 0x7f2b8c158040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:56.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.944+0000 7f2b8b7fe640 1 -- 172.21.15.67:0/1112290424 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b8c161370 msgr2=0x7f2b8c143c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:56.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.944+0000 7f2b8b7fe640 1 --2- 172.21.15.67:0/1112290424 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b8c161370 0x7f2b8c143c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:56.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.944+0000 7f2b8b7fe640 1 -- 172.21.15.67:0/1112290424 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b8c158880 con 0x7f2b8c0b9690 2024-03-24T00:38:56.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.944+0000 7f2b8b7fe640 1 --2- 172.21.15.67:0/1112290424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b8c0b9690 0x7f2b8c155ff0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c00c430 tx=0x7f2b7c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:56.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.944+0000 7f2b83fff640 1 -- 172.21.15.67:0/1112290424 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c0074e0 con 0x7f2b8c0b9690 2024-03-24T00:38:56.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.944+0000 7f2b83fff640 1 -- 172.21.15.67:0/1112290424 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b7c0047e0 con 0x7f2b8c0b9690 2024-03-24T00:38:56.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.944+0000 7f2b921e6640 1 -- 172.21.15.67:0/1112290424 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b8c158b60 con 0x7f2b8c0b9690 2024-03-24T00:38:56.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.944+0000 7f2b83fff640 1 -- 172.21.15.67:0/1112290424 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c004bd0 con 0x7f2b8c0b9690 2024-03-24T00:38:56.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.944+0000 7f2b921e6640 1 -- 172.21.15.67:0/1112290424 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b8c158580 con 0x7f2b8c0b9690 2024-03-24T00:38:56.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.945+0000 7f2b921e6640 1 -- 172.21.15.67:0/1112290424 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b8c068a00 con 0x7f2b8c0b9690 2024-03-24T00:38:56.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.946+0000 7f2b83fff640 1 -- 172.21.15.67:0/1112290424 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2b7c011070 con 0x7f2b8c0b9690 2024-03-24T00:38:56.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.946+0000 7f2b83fff640 1 --2- 172.21.15.67:0/1112290424 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2b6c06bbd0 0x7f2b6c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:56.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.946+0000 7f2b83fff640 1 -- 172.21.15.67:0/1112290424 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2b7c08a8b0 con 0x7f2b8c0b9690 2024-03-24T00:38:56.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.946+0000 7f2b8affd640 1 --2- 172.21.15.67:0/1112290424 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2b6c06bbd0 0x7f2b6c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:56.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.947+0000 7f2b8affd640 1 --2- 172.21.15.67:0/1112290424 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2b6c06bbd0 0x7f2b6c06e090 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c152a20 tx=0x7f2b74009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:56.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:56.951+0000 7f2b83fff640 1 -- 172.21.15.67:0/1112290424 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2b8c068a00 con 0x7f2b8c0b9690 2024-03-24T00:38:57.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.172+0000 7f2b921e6640 1 -- 172.21.15.67:0/1112290424 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7f2b8c04f400 con 0x7f2b8c0b9690 2024-03-24T00:38:57.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.800+0000 7f2b83fff640 1 -- 172.21.15.67:0/1112290424 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v7) v1 ==== 137+0+0 (secure 0 0 0) 0x7f2b7c058a30 con 0x7f2b8c0b9690 2024-03-24T00:38:57.801 INFO:teuthology.orchestra.run.smithi067.stderr:added data pool 4 to fsmap 2024-03-24T00:38:57.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.804+0000 7f2b921e6640 1 -- 172.21.15.67:0/1112290424 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2b6c06bbd0 msgr2=0x7f2b6c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:57.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.804+0000 7f2b921e6640 1 --2- 172.21.15.67:0/1112290424 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2b6c06bbd0 0x7f2b6c06e090 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c152a20 tx=0x7f2b74009190 comp rx=0 tx=0).stop 2024-03-24T00:38:57.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.804+0000 7f2b921e6640 1 -- 172.21.15.67:0/1112290424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b8c0b9690 msgr2=0x7f2b8c155ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:57.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.804+0000 7f2b921e6640 1 --2- 172.21.15.67:0/1112290424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b8c0b9690 0x7f2b8c155ff0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c00c430 tx=0x7f2b7c00c8f0 comp rx=0 tx=0).stop 2024-03-24T00:38:57.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.804+0000 7f2b921e6640 1 -- 172.21.15.67:0/1112290424 shutdown_connections 2024-03-24T00:38:57.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.804+0000 7f2b921e6640 1 --2- 172.21.15.67:0/1112290424 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2b6c06bbd0 0x7f2b6c06e090 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:57.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.804+0000 7f2b921e6640 1 --2- 172.21.15.67:0/1112290424 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b8c161370 0x7f2b8c143c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:57.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.804+0000 7f2b921e6640 1 --2- 172.21.15.67:0/1112290424 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2b8c0cef30 0x7f2b8c158040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:57.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.804+0000 7f2b921e6640 1 --2- 172.21.15.67:0/1112290424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b8c0b9690 0x7f2b8c155ff0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:57.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.804+0000 7f2b921e6640 1 -- 172.21.15.67:0/1112290424 >> 172.21.15.67:0/1112290424 conn(0x7f2b8c06c880 msgr2=0x7f2b8c15c840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:57.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.804+0000 7f2b921e6640 1 -- 172.21.15.67:0/1112290424 shutdown_connections 2024-03-24T00:38:57.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.804+0000 7f2b921e6640 1 -- 172.21.15.67:0/1112290424 wait complete. 2024-03-24T00:38:57.817 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:38:57.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.924+0000 7f4bf62a1640 1 Processor -- start 2024-03-24T00:38:57.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.924+0000 7f4bf62a1640 1 -- start start 2024-03-24T00:38:57.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.925+0000 7f4bf62a1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4bf015df30 0x7f4bf0162610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:57.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.925+0000 7f4bf62a1640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4bf00d0d50 0x7f4bf00d2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:57.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.925+0000 7f4bf62a1640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bf00ce920 con 0x7f4bf015e310 2024-03-24T00:38:57.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.925+0000 7f4bf62a1640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bf0162b50 con 0x7f4bf00d0d50 2024-03-24T00:38:57.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.925+0000 7f4bf62a1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf00cef20 con 0x7f4bf015df30 2024-03-24T00:38:57.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.925+0000 7f4bf483e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4bf00d0d50 0x7f4bf00d2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:57.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.925+0000 7f4bef7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4bf015e310 0x7f4bf00d0640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56860/0 (socket says 172.21.15.67:56860) 2024-03-24T00:38:57.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.925+0000 7f4bef7fe640 1 -- 172.21.15.67:0/239162393 learned_addr learned my addr 172.21.15.67:0/239162393 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:38:57.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.925+0000 7f4beffff640 1 --2- 172.21.15.67:0/239162393 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4bf015df30 0x7f4bf0162610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:57.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.925+0000 7f4bf483e640 1 -- 172.21.15.67:0/239162393 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4bf015df30 msgr2=0x7f4bf0162610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:57.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.925+0000 7f4bf483e640 1 --2- 172.21.15.67:0/239162393 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4bf015df30 0x7f4bf0162610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:57.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.925+0000 7f4bf483e640 1 -- 172.21.15.67:0/239162393 >> v1:172.21.15.67:6789/0 conn(0x7f4bf015e310 legacy=0x7f4bf00d0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:57.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.925+0000 7f4bf483e640 1 -- 172.21.15.67:0/239162393 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bf00cf200 con 0x7f4bf00d0d50 2024-03-24T00:38:57.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.925+0000 7f4beeffd640 1 -- 172.21.15.67:0/239162393 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3610929828 0 0) 0x7f4bf00ce920 con 0x7f4bf015e310 2024-03-24T00:38:57.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.926+0000 7f4bf483e640 1 --2- 172.21.15.67:0/239162393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bf00d0d50 0x7f4bf00d2490 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4be8009d90 tx=0x7f4be802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b84fe1062742fbd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:57.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.926+0000 7f4beeffd640 1 -- 172.21.15.67:0/239162393 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be8039070 con 0x7f4bf00d0d50 2024-03-24T00:38:57.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.926+0000 7f4beeffd640 1 -- 172.21.15.67:0/239162393 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4be802cbc0 con 0x7f4bf00d0d50 2024-03-24T00:38:57.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.926+0000 7f4beeffd640 1 -- 172.21.15.67:0/239162393 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be8034440 con 0x7f4bf00d0d50 2024-03-24T00:38:57.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.927+0000 7f4bf62a1640 1 -- 172.21.15.67:0/239162393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bf00d0d50 msgr2=0x7f4bf00d2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:57.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.927+0000 7f4bf62a1640 1 --2- 172.21.15.67:0/239162393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bf00d0d50 0x7f4bf00d2490 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4be8009d90 tx=0x7f4be802be10 comp rx=0 tx=0).stop 2024-03-24T00:38:57.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.927+0000 7f4bf62a1640 1 -- 172.21.15.67:0/239162393 shutdown_connections 2024-03-24T00:38:57.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.927+0000 7f4bf62a1640 1 --2- 172.21.15.67:0/239162393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bf00d0d50 0x7f4bf00d2490 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:57.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.927+0000 7f4bf62a1640 1 --2- 172.21.15.67:0/239162393 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4bf015df30 0x7f4bf0162610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:57.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.927+0000 7f4bf62a1640 1 -- 172.21.15.67:0/239162393 >> 172.21.15.67:0/239162393 conn(0x7f4bf006c880 msgr2=0x7f4bf006cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:57.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.927+0000 7f4bf62a1640 1 -- 172.21.15.67:0/239162393 shutdown_connections 2024-03-24T00:38:57.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.927+0000 7f4bf62a1640 1 -- 172.21.15.67:0/239162393 wait complete. 2024-03-24T00:38:57.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.928+0000 7f4bf62a1640 1 Processor -- start 2024-03-24T00:38:57.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.928+0000 7f4bf62a1640 1 -- start start 2024-03-24T00:38:57.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.929+0000 7f4bf62a1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bf00d0d50 0x7f4bf0149c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:57.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.929+0000 7f4bf62a1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4bf015df30 0x7f4bf00c9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:57.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.929+0000 7f4bf62a1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4bf01f74e0 0x7f4bf01f7930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:57.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.929+0000 7f4bf62a1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf00ce580 con 0x7f4bf00d0d50 2024-03-24T00:38:57.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.929+0000 7f4bf62a1640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf00b32f0 con 0x7f4bf015df30 2024-03-24T00:38:57.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.929+0000 7f4bf62a1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf00ce700 con 0x7f4bf01f74e0 2024-03-24T00:38:57.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.929+0000 7f4beffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bf00d0d50 0x7f4bf0149c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:57.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.929+0000 7f4beffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bf00d0d50 0x7f4bf0149c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59598/0 (socket says 172.21.15.67:59598) 2024-03-24T00:38:57.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.929+0000 7f4beffff640 1 -- 172.21.15.67:0/422691293 learned_addr learned my addr 172.21.15.67:0/422691293 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:38:57.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.929+0000 7f4bef7fe640 1 --2- 172.21.15.67:0/422691293 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4bf015df30 0x7f4bf00c9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:57.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.929+0000 7f4bf483e640 1 --2- 172.21.15.67:0/422691293 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4bf01f74e0 0x7f4bf01f7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:57.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.929+0000 7f4beffff640 1 -- 172.21.15.67:0/422691293 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4bf01f74e0 msgr2=0x7f4bf01f7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:57.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.929+0000 7f4beffff640 1 --2- 172.21.15.67:0/422691293 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4bf01f74e0 0x7f4bf01f7930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:57.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.929+0000 7f4beffff640 1 -- 172.21.15.67:0/422691293 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4bf015df30 msgr2=0x7f4bf00c9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:57.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.929+0000 7f4beffff640 1 --2- 172.21.15.67:0/422691293 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4bf015df30 0x7f4bf00c9fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:57.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.929+0000 7f4beffff640 1 -- 172.21.15.67:0/422691293 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bf00cba00 con 0x7f4bf00d0d50 2024-03-24T00:38:57.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.929+0000 7f4beffff640 1 --2- 172.21.15.67:0/422691293 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bf00d0d50 0x7f4bf0149c20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4be400e9b0 tx=0x7f4be400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:57.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.930+0000 7f4becff9640 1 -- 172.21.15.67:0/422691293 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be4017070 con 0x7f4bf00d0d50 2024-03-24T00:38:57.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.930+0000 7f4becff9640 1 -- 172.21.15.67:0/422691293 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4be400c9a0 con 0x7f4bf00d0d50 2024-03-24T00:38:57.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.930+0000 7f4bf62a1640 1 -- 172.21.15.67:0/422691293 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bf01f8080 con 0x7f4bf00d0d50 2024-03-24T00:38:57.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.930+0000 7f4becff9640 1 -- 172.21.15.67:0/422691293 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be400cd90 con 0x7f4bf00d0d50 2024-03-24T00:38:57.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.930+0000 7f4bf62a1640 1 -- 172.21.15.67:0/422691293 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bf0149140 con 0x7f4bf00d0d50 2024-03-24T00:38:57.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.931+0000 7f4bf62a1640 1 -- 172.21.15.67:0/422691293 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bf004f400 con 0x7f4bf00d0d50 2024-03-24T00:38:57.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.931+0000 7f4becff9640 1 -- 172.21.15.67:0/422691293 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4be401a070 con 0x7f4bf00d0d50 2024-03-24T00:38:57.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.931+0000 7f4becff9640 1 --2- 172.21.15.67:0/422691293 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4bdc06bbd0 0x7f4bdc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:57.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.931+0000 7f4becff9640 1 -- 172.21.15.67:0/422691293 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f4be408ace0 con 0x7f4bf00d0d50 2024-03-24T00:38:57.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.931+0000 7f4bef7fe640 1 --2- 172.21.15.67:0/422691293 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4bdc06bbd0 0x7f4bdc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:57.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.932+0000 7f4bef7fe640 1 --2- 172.21.15.67:0/422691293 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4bdc06bbd0 0x7f4bdc06e090 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4bf01579c0 tx=0x7f4be0007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:57.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:57.936+0000 7f4becff9640 1 -- 172.21.15.67:0/422691293 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4be4058ed0 con 0x7f4bf00d0d50 2024-03-24T00:38:58.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.161+0000 7f4bf62a1640 1 -- 172.21.15.67:0/422691293 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4bf00b2d10 con 0x7f4bf00d0d50 2024-03-24T00:38:58.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.162+0000 7f4becff9640 1 -- 172.21.15.67:0/422691293 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4970 (secure 0 0 0) 0x7f4be405cf70 con 0x7f4bf00d0d50 2024-03-24T00:38:58.163 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:38:58.163 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4263,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/837528243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":837528243},{"type":"v1","addr":"172.21.15.67:6834","nonce":837528243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3987071456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":3987071456},{"type":"v1","addr":"172.21.15.67:6835","nonce":3987071456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4306,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/3953492281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3953492281},{"type":"v1","addr":"172.21.15.72:6837","nonce":3953492281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:38:52.127871+0000","modified":"2024-03-24T00:38:57.174252+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4309},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4309":{"gid":4309,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6836/3737698654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3737698654},{"type":"v1","addr":"172.21.15.72:6836","nonce":3737698654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4309,"qdb_cluster":[4309]},"id":1}]} 2024-03-24T00:38:58.163 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 7 2024-03-24T00:38:58.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.164+0000 7f4bf62a1640 1 -- 172.21.15.67:0/422691293 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4bdc06bbd0 msgr2=0x7f4bdc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:58.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.164+0000 7f4bf62a1640 1 --2- 172.21.15.67:0/422691293 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4bdc06bbd0 0x7f4bdc06e090 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4bf01579c0 tx=0x7f4be0007de0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.164+0000 7f4bf62a1640 1 -- 172.21.15.67:0/422691293 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bf00d0d50 msgr2=0x7f4bf0149c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:58.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.164+0000 7f4bf62a1640 1 --2- 172.21.15.67:0/422691293 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bf00d0d50 0x7f4bf0149c20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4be400e9b0 tx=0x7f4be400ee70 comp rx=0 tx=0).stop 2024-03-24T00:38:58.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.164+0000 7f4bf62a1640 1 -- 172.21.15.67:0/422691293 shutdown_connections 2024-03-24T00:38:58.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.164+0000 7f4bf62a1640 1 --2- 172.21.15.67:0/422691293 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4bdc06bbd0 0x7f4bdc06e090 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.164+0000 7f4bf62a1640 1 --2- 172.21.15.67:0/422691293 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4bf01f74e0 0x7f4bf01f7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.164+0000 7f4bf62a1640 1 --2- 172.21.15.67:0/422691293 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4bf015df30 0x7f4bf00c9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.164+0000 7f4bf62a1640 1 --2- 172.21.15.67:0/422691293 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bf00d0d50 0x7f4bf0149c20 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.164+0000 7f4bf62a1640 1 -- 172.21.15.67:0/422691293 >> 172.21.15.67:0/422691293 conn(0x7f4bf006c880 msgr2=0x7f4bf00b3980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:58.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.164+0000 7f4bf62a1640 1 -- 172.21.15.67:0/422691293 shutdown_connections 2024-03-24T00:38:58.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.164+0000 7f4bf62a1640 1 -- 172.21.15.67:0/422691293 wait complete. 2024-03-24T00:38:58.176 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:38:58.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.276+0000 7fb8fe777640 1 Processor -- start 2024-03-24T00:38:58.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.276+0000 7fb8fe777640 1 -- start start 2024-03-24T00:38:58.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.277+0000 7fb8fe777640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb8f815dfe0 0x7fb8f80ceb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:58.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.277+0000 7fb8fe777640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb8f8161bb0 0x7fb8f80cf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:58.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.277+0000 7fb8fe777640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb8f80b96d0 0x7fb8f80b57c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:58.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.277+0000 7fb8fe777640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8f80d21d0 con 0x7fb8f8161bb0 2024-03-24T00:38:58.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.277+0000 7fb8fe777640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8f8162120 con 0x7fb8f815dfe0 2024-03-24T00:38:58.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.277+0000 7fb8fe777640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f8162400 con 0x7fb8f80b96d0 2024-03-24T00:38:58.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.277+0000 7fb8f77fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb8f8161bb0 0x7fb8f80cf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:58.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.277+0000 7fb8f77fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb8f8161bb0 0x7fb8f80cf050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59610/0 (socket says 172.21.15.67:59610) 2024-03-24T00:38:58.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.277+0000 7fb8f77fe640 1 -- 172.21.15.67:0/1437418052 learned_addr learned my addr 172.21.15.67:0/1437418052 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:38:58.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.277+0000 7fb8fcd14640 1 --2- 172.21.15.67:0/1437418052 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb8f80b96d0 0x7fb8f80b57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:58.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.277+0000 7fb8f7fff640 1 --2- 172.21.15.67:0/1437418052 >> v2:172.21.15.72:3300/0 conn(0x7fb8f815dfe0 0x7fb8f80ceb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:58.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.277+0000 7fb8f77fe640 1 -- 172.21.15.67:0/1437418052 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb8f80b96d0 msgr2=0x7fb8f80b57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:58.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.277+0000 7fb8f77fe640 1 --2- 172.21.15.67:0/1437418052 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb8f80b96d0 0x7fb8f80b57c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.277+0000 7fb8f77fe640 1 -- 172.21.15.67:0/1437418052 >> v2:172.21.15.72:3300/0 conn(0x7fb8f815dfe0 msgr2=0x7fb8f80ceb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:58.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.277+0000 7fb8f77fe640 1 --2- 172.21.15.67:0/1437418052 >> v2:172.21.15.72:3300/0 conn(0x7fb8f815dfe0 0x7fb8f80ceb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.277+0000 7fb8f77fe640 1 -- 172.21.15.67:0/1437418052 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8f81626e0 con 0x7fb8f8161bb0 2024-03-24T00:38:58.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.278+0000 7fb8f77fe640 1 --2- 172.21.15.67:0/1437418052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8f8161bb0 0x7fb8f80cf050 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb8f806d630 tx=0x7fb8ec02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b250c2de5f12f3ea server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:58.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.278+0000 7fb8f6ffd640 1 -- 172.21.15.67:0/1437418052 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ec035020 con 0x7fb8f8161bb0 2024-03-24T00:38:58.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.278+0000 7fb8f6ffd640 1 -- 172.21.15.67:0/1437418052 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8ec031870 con 0x7fb8f8161bb0 2024-03-24T00:38:58.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.278+0000 7fb8f6ffd640 1 -- 172.21.15.67:0/1437418052 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ec031c40 con 0x7fb8f8161bb0 2024-03-24T00:38:58.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.278+0000 7fb8fe777640 1 -- 172.21.15.67:0/1437418052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8f8161bb0 msgr2=0x7fb8f80cf050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:58.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.278+0000 7fb8fe777640 1 --2- 172.21.15.67:0/1437418052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8f8161bb0 0x7fb8f80cf050 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb8f806d630 tx=0x7fb8ec02b910 comp rx=0 tx=0).stop 2024-03-24T00:38:58.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.278+0000 7fb8fe777640 1 -- 172.21.15.67:0/1437418052 shutdown_connections 2024-03-24T00:38:58.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.278+0000 7fb8fe777640 1 --2- 172.21.15.67:0/1437418052 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb8f80b96d0 0x7fb8f80b57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.278+0000 7fb8fe777640 1 --2- 172.21.15.67:0/1437418052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8f8161bb0 0x7fb8f80cf050 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.278+0000 7fb8fe777640 1 --2- 172.21.15.67:0/1437418052 >> v2:172.21.15.72:3300/0 conn(0x7fb8f815dfe0 0x7fb8f80ceb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.278+0000 7fb8fe777640 1 -- 172.21.15.67:0/1437418052 >> 172.21.15.67:0/1437418052 conn(0x7fb8f806c890 msgr2=0x7fb8f806cc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:58.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.278+0000 7fb8fe777640 1 -- 172.21.15.67:0/1437418052 shutdown_connections 2024-03-24T00:38:58.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.278+0000 7fb8fe777640 1 -- 172.21.15.67:0/1437418052 wait complete. 2024-03-24T00:38:58.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.279+0000 7fb8fe777640 1 Processor -- start 2024-03-24T00:38:58.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.280+0000 7fb8fe777640 1 -- start start 2024-03-24T00:38:58.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.280+0000 7fb8fe777640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8f80b96d0 0x7fb8f80cd230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:58.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.280+0000 7fb8fe777640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb8f815dfe0 0x7fb8f81e79b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:58.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.280+0000 7fb8fe777640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb8f8161bb0 0x7fb8f81f3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:58.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.280+0000 7fb8fe777640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f80d1120 con 0x7fb8f80b96d0 2024-03-24T00:38:58.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.280+0000 7fb8fe777640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f815e3c0 con 0x7fb8f815dfe0 2024-03-24T00:38:58.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.280+0000 7fb8fe777640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f80d12a0 con 0x7fb8f8161bb0 2024-03-24T00:38:58.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.280+0000 7fb8f7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8f80b96d0 0x7fb8f80cd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:58.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.280+0000 7fb8f7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8f80b96d0 0x7fb8f80cd230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59620/0 (socket says 172.21.15.67:59620) 2024-03-24T00:38:58.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.280+0000 7fb8f7fff640 1 -- 172.21.15.67:0/2096668157 learned_addr learned my addr 172.21.15.67:0/2096668157 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:38:58.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.281+0000 7fb8f77fe640 1 --2- 172.21.15.67:0/2096668157 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb8f815dfe0 0x7fb8f81e79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:58.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.281+0000 7fb8fcd14640 1 --2- 172.21.15.67:0/2096668157 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb8f8161bb0 0x7fb8f81f3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:58.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.281+0000 7fb8f7fff640 1 -- 172.21.15.67:0/2096668157 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb8f8161bb0 msgr2=0x7fb8f81f3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:58.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.281+0000 7fb8f7fff640 1 --2- 172.21.15.67:0/2096668157 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb8f8161bb0 0x7fb8f81f3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.281+0000 7fb8f7fff640 1 -- 172.21.15.67:0/2096668157 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb8f815dfe0 msgr2=0x7fb8f81e79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:58.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.281+0000 7fb8f7fff640 1 --2- 172.21.15.67:0/2096668157 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb8f815dfe0 0x7fb8f81e79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.281+0000 7fb8f7fff640 1 -- 172.21.15.67:0/2096668157 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8f81e89c0 con 0x7fb8f80b96d0 2024-03-24T00:38:58.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.281+0000 7fb8f77fe640 1 --2- 172.21.15.67:0/2096668157 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb8f815dfe0 0x7fb8f81e79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:58.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.281+0000 7fb8f7fff640 1 --2- 172.21.15.67:0/2096668157 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8f80b96d0 0x7fb8f80cd230 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb8e800e9b0 tx=0x7fb8e800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:58.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.281+0000 7fb8f4ff9640 1 -- 172.21.15.67:0/2096668157 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e8017070 con 0x7fb8f80b96d0 2024-03-24T00:38:58.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.281+0000 7fb8f4ff9640 1 -- 172.21.15.67:0/2096668157 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8e800c9a0 con 0x7fb8f80b96d0 2024-03-24T00:38:58.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.281+0000 7fb8fe777640 1 -- 172.21.15.67:0/2096668157 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8f81e8120 con 0x7fb8f80b96d0 2024-03-24T00:38:58.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.281+0000 7fb8f4ff9640 1 -- 172.21.15.67:0/2096668157 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e800cd90 con 0x7fb8f80b96d0 2024-03-24T00:38:58.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.281+0000 7fb8fe777640 1 -- 172.21.15.67:0/2096668157 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8f81e86e0 con 0x7fb8f80b96d0 2024-03-24T00:38:58.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.282+0000 7fb8fe777640 1 -- 172.21.15.67:0/2096668157 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8f806b180 con 0x7fb8f80b96d0 2024-03-24T00:38:58.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.283+0000 7fb8f4ff9640 1 -- 172.21.15.67:0/2096668157 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb8e801a070 con 0x7fb8f80b96d0 2024-03-24T00:38:58.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.283+0000 7fb8f4ff9640 1 --2- 172.21.15.67:0/2096668157 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb8e406bbd0 0x7fb8e406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:58.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.283+0000 7fb8f4ff9640 1 -- 172.21.15.67:0/2096668157 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fb8e808ad70 con 0x7fb8f80b96d0 2024-03-24T00:38:58.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.283+0000 7fb8f77fe640 1 --2- 172.21.15.67:0/2096668157 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb8e406bbd0 0x7fb8e406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:58.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.284+0000 7fb8f77fe640 1 --2- 172.21.15.67:0/2096668157 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb8e406bbd0 0x7fb8e406e090 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec036000 tx=0x7fb8ec008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:58.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.288+0000 7fb8f4ff9640 1 -- 172.21.15.67:0/2096668157 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb8e8058ed0 con 0x7fb8f80b96d0 2024-03-24T00:38:58.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.470+0000 7fb8fe777640 1 -- 172.21.15.67:0/2096668157 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb8f8068a40 con 0x7fb8f80b96d0 2024-03-24T00:38:58.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.471+0000 7fb8f4ff9640 1 -- 172.21.15.67:0/2096668157 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18467 (secure 0 0 0) 0x7fb8f8068a40 con 0x7fb8f80b96d0 2024-03-24T00:38:58.472 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:38:58.472 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":24,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:38:57.174062+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:38:50.077085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:38:51.095454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T00:38:54.807117+0000","flags":32773,"flags_names":"hashpspool,ec_overwrites,creating","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:38:58.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.473+0000 7fb8fe777640 1 -- 172.21.15.67:0/2096668157 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb8e406bbd0 msgr2=0x7fb8e406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:58.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.473+0000 7fb8fe777640 1 --2- 172.21.15.67:0/2096668157 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb8e406bbd0 0x7fb8e406e090 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec036000 tx=0x7fb8ec008030 comp rx=0 tx=0).stop 2024-03-24T00:38:58.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.473+0000 7fb8fe777640 1 -- 172.21.15.67:0/2096668157 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8f80b96d0 msgr2=0x7fb8f80cd230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:58.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.473+0000 7fb8fe777640 1 --2- 172.21.15.67:0/2096668157 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8f80b96d0 0x7fb8f80cd230 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb8e800e9b0 tx=0x7fb8e800ee70 comp rx=0 tx=0).stop 2024-03-24T00:38:58.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.473+0000 7fb8fe777640 1 -- 172.21.15.67:0/2096668157 shutdown_connections 2024-03-24T00:38:58.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.473+0000 7fb8fe777640 1 --2- 172.21.15.67:0/2096668157 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb8e406bbd0 0x7fb8e406e090 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.473+0000 7fb8fe777640 1 --2- 172.21.15.67:0/2096668157 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb8f8161bb0 0x7fb8f81f3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.473+0000 7fb8fe777640 1 --2- 172.21.15.67:0/2096668157 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb8f815dfe0 0x7fb8f81e79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.473+0000 7fb8fe777640 1 --2- 172.21.15.67:0/2096668157 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8f80b96d0 0x7fb8f80cd230 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.473+0000 7fb8fe777640 1 -- 172.21.15.67:0/2096668157 >> 172.21.15.67:0/2096668157 conn(0x7fb8f806c890 msgr2=0x7fb8f80d0580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:58.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.473+0000 7fb8fe777640 1 -- 172.21.15.67:0/2096668157 shutdown_connections 2024-03-24T00:38:58.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.473+0000 7fb8fe777640 1 -- 172.21.15.67:0/2096668157 wait complete. 2024-03-24T00:38:58.485 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:38:58.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.586+0000 7fecfbd62640 1 Processor -- start 2024-03-24T00:38:58.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.586+0000 7fecfbd62640 1 -- start start 2024-03-24T00:38:58.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.587+0000 7fecfbd62640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fecf415e2a0 0x7fecf40ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:58.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.587+0000 7fecfbd62640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fecf40d0ef0 0x7fecf4162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:58.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.587+0000 7fecfbd62640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecf40cf640 con 0x7fecf415dec0 2024-03-24T00:38:58.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.587+0000 7fecfbd62640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecf40cee30 con 0x7fecf415e2a0 2024-03-24T00:38:58.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.587+0000 7fecfbd62640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecf40d2120 con 0x7fecf40d0ef0 2024-03-24T00:38:58.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.587+0000 7fecf92fd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fecf415e2a0 0x7fecf40ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:58.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.587+0000 7fecf92fd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fecf415e2a0 0x7fecf40ce630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59624/0 (socket says 172.21.15.67:59624) 2024-03-24T00:38:58.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.587+0000 7fecf92fd640 1 -- 172.21.15.67:0/2695664026 learned_addr learned my addr 172.21.15.67:0/2695664026 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:38:58.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.587+0000 7fecfa2ff640 1 --2- 172.21.15.67:0/2695664026 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fecf40d0ef0 0x7fecf4162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:58.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.587+0000 7fecf92fd640 1 -- 172.21.15.67:0/2695664026 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fecf40d0ef0 msgr2=0x7fecf4162630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:58.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.587+0000 7fecf92fd640 1 --2- 172.21.15.67:0/2695664026 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fecf40d0ef0 0x7fecf4162630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.587+0000 7fecf92fd640 1 -- 172.21.15.67:0/2695664026 >> v1:172.21.15.72:6789/0 conn(0x7fecf415dec0 legacy=0x7fecf40d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:58.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.587+0000 7fecf92fd640 1 -- 172.21.15.67:0/2695664026 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf40cf080 con 0x7fecf415e2a0 2024-03-24T00:38:58.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.588+0000 7fecf92fd640 1 --2- 172.21.15.67:0/2695664026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecf415e2a0 0x7fecf40ce630 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fece8009870 tx=0x7fece802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e94d0b95dd36dcce server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:58.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.588+0000 7fecf8afc640 1 -- 172.21.15.67:0/2695664026 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece8035020 con 0x7fecf415e2a0 2024-03-24T00:38:58.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.588+0000 7fecf8afc640 1 -- 172.21.15.67:0/2695664026 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fece8031640 con 0x7fecf415e2a0 2024-03-24T00:38:58.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.588+0000 7fecf8afc640 1 -- 172.21.15.67:0/2695664026 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece802d040 con 0x7fecf415e2a0 2024-03-24T00:38:58.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.589+0000 7fecfbd62640 1 -- 172.21.15.67:0/2695664026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecf415e2a0 msgr2=0x7fecf40ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:58.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.589+0000 7fecfbd62640 1 --2- 172.21.15.67:0/2695664026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecf415e2a0 0x7fecf40ce630 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fece8009870 tx=0x7fece802b8f0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.589+0000 7fecfbd62640 1 -- 172.21.15.67:0/2695664026 shutdown_connections 2024-03-24T00:38:58.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.589+0000 7fecfbd62640 1 --2- 172.21.15.67:0/2695664026 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fecf40d0ef0 0x7fecf4162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.589+0000 7fecfbd62640 1 --2- 172.21.15.67:0/2695664026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecf415e2a0 0x7fecf40ce630 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.589+0000 7fecfbd62640 1 -- 172.21.15.67:0/2695664026 >> 172.21.15.67:0/2695664026 conn(0x7fecf406c810 msgr2=0x7fecf406cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:58.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.589+0000 7fecfbd62640 1 -- 172.21.15.67:0/2695664026 shutdown_connections 2024-03-24T00:38:58.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.589+0000 7fecfbd62640 1 -- 172.21.15.67:0/2695664026 wait complete. 2024-03-24T00:38:58.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.590+0000 7fecfbd62640 1 Processor -- start 2024-03-24T00:38:58.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.590+0000 7fecfbd62640 1 -- start start 2024-03-24T00:38:58.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.591+0000 7fecfbd62640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fecf40d0ef0 0x7fecf4152350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:58.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.591+0000 7fecfbd62640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecf415dec0 0x7fecf415e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:58.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.591+0000 7fecfbd62640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fecf415e2a0 0x7fecf415ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:58.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.591+0000 7fecfbd62640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf40d2730 con 0x7fecf415dec0 2024-03-24T00:38:58.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.591+0000 7fecfbd62640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf40d25b0 con 0x7fecf415e2a0 2024-03-24T00:38:58.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.591+0000 7fecfbd62640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecf40d28b0 con 0x7fecf40d0ef0 2024-03-24T00:38:58.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.591+0000 7fecf92fd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecf415dec0 0x7fecf415e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:58.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.591+0000 7fecf92fd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecf415dec0 0x7fecf415e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59626/0 (socket says 172.21.15.67:59626) 2024-03-24T00:38:58.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.591+0000 7fecf92fd640 1 -- 172.21.15.67:0/1127609725 learned_addr learned my addr 172.21.15.67:0/1127609725 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:38:58.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.591+0000 7fecfa2ff640 1 --2- 172.21.15.67:0/1127609725 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fecf415e2a0 0x7fecf415ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:58.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.591+0000 7fecf9afe640 1 --2- 172.21.15.67:0/1127609725 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fecf40d0ef0 0x7fecf4152350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:58.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.591+0000 7fecf92fd640 1 -- 172.21.15.67:0/1127609725 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fecf40d0ef0 msgr2=0x7fecf4152350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:58.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.591+0000 7fecf92fd640 1 --2- 172.21.15.67:0/1127609725 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fecf40d0ef0 0x7fecf4152350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.591+0000 7fecf92fd640 1 -- 172.21.15.67:0/1127609725 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fecf415e2a0 msgr2=0x7fecf415ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:58.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.591+0000 7fecf92fd640 1 --2- 172.21.15.67:0/1127609725 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fecf415e2a0 0x7fecf415ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.591+0000 7fecf92fd640 1 -- 172.21.15.67:0/1127609725 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf40cbc60 con 0x7fecf415dec0 2024-03-24T00:38:58.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.591+0000 7fecf92fd640 1 --2- 172.21.15.67:0/1127609725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecf415dec0 0x7fecf415e6f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fece8000c00 tx=0x7fece80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:58.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.591+0000 7fecf27fc640 1 -- 172.21.15.67:0/1127609725 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece8040070 con 0x7fecf415dec0 2024-03-24T00:38:58.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.591+0000 7fecf27fc640 1 -- 172.21.15.67:0/1127609725 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fece80399f0 con 0x7fecf415dec0 2024-03-24T00:38:58.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.591+0000 7fecfbd62640 1 -- 172.21.15.67:0/1127609725 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecf40cced0 con 0x7fecf415dec0 2024-03-24T00:38:58.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.591+0000 7fecf27fc640 1 -- 172.21.15.67:0/1127609725 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece8035040 con 0x7fecf415dec0 2024-03-24T00:38:58.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.591+0000 7fecfbd62640 1 -- 172.21.15.67:0/1127609725 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecf40cd490 con 0x7fecf415dec0 2024-03-24T00:38:58.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.592+0000 7fecfbd62640 1 -- 172.21.15.67:0/1127609725 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecf404f400 con 0x7fecf415dec0 2024-03-24T00:38:58.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.593+0000 7fecf27fc640 1 -- 172.21.15.67:0/1127609725 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fece804b060 con 0x7fecf415dec0 2024-03-24T00:38:58.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.593+0000 7fecf27fc640 1 --2- 172.21.15.67:0/1127609725 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fece006bbd0 0x7fece006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:58.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.593+0000 7fecf27fc640 1 -- 172.21.15.67:0/1127609725 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fece80ac400 con 0x7fecf415dec0 2024-03-24T00:38:58.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.593+0000 7fecf9afe640 1 --2- 172.21.15.67:0/1127609725 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fece006bbd0 0x7fece006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:58.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.593+0000 7fecf9afe640 1 --2- 172.21.15.67:0/1127609725 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fece006bbd0 0x7fece006e090 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fecec001780 tx=0x7fecec009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:58.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.597+0000 7fecf27fc640 1 -- 172.21.15.67:0/1127609725 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fece807a5f0 con 0x7fecf415dec0 2024-03-24T00:38:58.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.778+0000 7fecfbd62640 1 -- 172.21.15.67:0/1127609725 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fecf4162010 con 0x7fecf415dec0 2024-03-24T00:38:58.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.779+0000 7fecf27fc640 1 -- 172.21.15.67:0/1127609725 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18467 (secure 0 0 0) 0x7fece807e690 con 0x7fecf415dec0 2024-03-24T00:38:58.781 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:38:58.781 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":24,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:38:57.174062+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:38:50.077085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:38:51.095454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T00:38:54.807117+0000","flags":32773,"flags_names":"hashpspool,ec_overwrites,creating","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:38:58.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.782+0000 7fecfbd62640 1 -- 172.21.15.67:0/1127609725 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fece006bbd0 msgr2=0x7fece006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:58.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.782+0000 7fecfbd62640 1 --2- 172.21.15.67:0/1127609725 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fece006bbd0 0x7fece006e090 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fecec001780 tx=0x7fecec009040 comp rx=0 tx=0).stop 2024-03-24T00:38:58.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.782+0000 7fecfbd62640 1 -- 172.21.15.67:0/1127609725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecf415dec0 msgr2=0x7fecf415e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:58.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.782+0000 7fecfbd62640 1 --2- 172.21.15.67:0/1127609725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecf415dec0 0x7fecf415e6f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fece8000c00 tx=0x7fece80046c0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.782+0000 7fecfbd62640 1 -- 172.21.15.67:0/1127609725 shutdown_connections 2024-03-24T00:38:58.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.782+0000 7fecfbd62640 1 --2- 172.21.15.67:0/1127609725 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fece006bbd0 0x7fece006e090 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.782+0000 7fecfbd62640 1 --2- 172.21.15.67:0/1127609725 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fecf415e2a0 0x7fecf415ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.782+0000 7fecfbd62640 1 --2- 172.21.15.67:0/1127609725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecf415dec0 0x7fecf415e6f0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.782+0000 7fecfbd62640 1 --2- 172.21.15.67:0/1127609725 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fecf40d0ef0 0x7fecf4152350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.782+0000 7fecfbd62640 1 -- 172.21.15.67:0/1127609725 >> 172.21.15.67:0/1127609725 conn(0x7fecf406c810 msgr2=0x7fecf415d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:58.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.782+0000 7fecfbd62640 1 -- 172.21.15.67:0/1127609725 shutdown_connections 2024-03-24T00:38:58.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.782+0000 7fecfbd62640 1 -- 172.21.15.67:0/1127609725 wait complete. 2024-03-24T00:38:58.808 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:38:58.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.909+0000 7f5699a63640 1 Processor -- start 2024-03-24T00:38:58.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.909+0000 7f5699a63640 1 -- start start 2024-03-24T00:38:58.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.910+0000 7f5699a63640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f569415dfe0 0x7f56940ceb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:58.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.910+0000 7f5699a63640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5694161bb0 0x7f56940cf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:58.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.910+0000 7f5699a63640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5694162120 con 0x7f56940b96d0 2024-03-24T00:38:58.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.910+0000 7f5699a63640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56940d21d0 con 0x7f5694161bb0 2024-03-24T00:38:58.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.910+0000 7f5699a63640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5694162400 con 0x7f569415dfe0 2024-03-24T00:38:58.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.910+0000 7f56937fe640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f569415dfe0 0x7f56940ceb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:58.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.910+0000 7f5692ffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5694161bb0 0x7f56940cf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:58.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.910+0000 7f5693fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f56940b96d0 0x7f56940b57c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:51216/0 (socket says 172.21.15.67:51216) 2024-03-24T00:38:58.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.910+0000 7f5693fff640 1 -- 172.21.15.67:0/3964488901 learned_addr learned my addr 172.21.15.67:0/3964488901 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:38:58.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.910+0000 7f56937fe640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f569415dfe0 0x7f56940ceb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.67:35398/0 (socket says 172.21.15.67:35398) 2024-03-24T00:38:58.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.910+0000 7f5692ffd640 1 -- 172.21.15.67:0/3964488901 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f569415dfe0 msgr2=0x7f56940ceb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:58.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.910+0000 7f5692ffd640 1 --2- 172.21.15.67:0/3964488901 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f569415dfe0 0x7f56940ceb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.910+0000 7f5692ffd640 1 -- 172.21.15.67:0/3964488901 >> v1:172.21.15.72:6789/0 conn(0x7f56940b96d0 legacy=0x7f56940b57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:58.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.910+0000 7f5692ffd640 1 -- 172.21.15.67:0/3964488901 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56941626e0 con 0x7f5694161bb0 2024-03-24T00:38:58.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.910+0000 7f56927fc640 1 -- 172.21.15.67:0/3964488901 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2109648151 0 0) 0x7f5694162120 con 0x7f56940b96d0 2024-03-24T00:38:58.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.910+0000 7f56937fe640 1 --2- 172.21.15.67:0/3964488901 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f569415dfe0 0x7f56940ceb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:58.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.911+0000 7f5692ffd640 1 --2- 172.21.15.67:0/3964488901 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5694161bb0 0x7f56940cf050 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f56840097a0 tx=0x7f568402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5f8211132a89d78 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:58.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.911+0000 7f56927fc640 1 -- 172.21.15.67:0/3964488901 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5684035020 con 0x7f5694161bb0 2024-03-24T00:38:58.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.911+0000 7f56927fc640 1 -- 172.21.15.67:0/3964488901 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5684031830 con 0x7f5694161bb0 2024-03-24T00:38:58.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.911+0000 7f56927fc640 1 -- 172.21.15.67:0/3964488901 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5684031c00 con 0x7f5694161bb0 2024-03-24T00:38:58.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.912+0000 7f5699a63640 1 -- 172.21.15.67:0/3964488901 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5694161bb0 msgr2=0x7f56940cf050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:58.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.912+0000 7f5699a63640 1 --2- 172.21.15.67:0/3964488901 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5694161bb0 0x7f56940cf050 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f56840097a0 tx=0x7f568402b8d0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.912+0000 7f5699a63640 1 -- 172.21.15.67:0/3964488901 shutdown_connections 2024-03-24T00:38:58.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.912+0000 7f5699a63640 1 --2- 172.21.15.67:0/3964488901 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5694161bb0 0x7f56940cf050 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.912+0000 7f5699a63640 1 --2- 172.21.15.67:0/3964488901 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f569415dfe0 0x7f56940ceb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.912+0000 7f5699a63640 1 -- 172.21.15.67:0/3964488901 >> 172.21.15.67:0/3964488901 conn(0x7f569406c890 msgr2=0x7f569406cc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:58.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.912+0000 7f5699a63640 1 -- 172.21.15.67:0/3964488901 shutdown_connections 2024-03-24T00:38:58.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.912+0000 7f5699a63640 1 -- 172.21.15.67:0/3964488901 wait complete. 2024-03-24T00:38:58.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.913+0000 7f5699a63640 1 Processor -- start 2024-03-24T00:38:58.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.913+0000 7f5699a63640 1 -- start start 2024-03-24T00:38:58.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.914+0000 7f5699a63640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f56940b96d0 0x7f5694161170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:58.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.914+0000 7f5699a63640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f569415dfe0 0x7f56941f5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:58.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.914+0000 7f5699a63640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5694161bb0 0x7f56940b7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:58.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.914+0000 7f5699a63640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56940d1120 con 0x7f569415dfe0 2024-03-24T00:38:58.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.914+0000 7f5699a63640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f569415e3c0 con 0x7f56940b96d0 2024-03-24T00:38:58.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.914+0000 7f5699a63640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56940d12a0 con 0x7f5694161bb0 2024-03-24T00:38:58.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.914+0000 7f5692ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f569415dfe0 0x7f56941f5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:58.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.914+0000 7f5692ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f569415dfe0 0x7f56941f5ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59634/0 (socket says 172.21.15.67:59634) 2024-03-24T00:38:58.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.914+0000 7f5692ffd640 1 -- 172.21.15.67:0/1387180852 learned_addr learned my addr 172.21.15.67:0/1387180852 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:38:58.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.914+0000 7f56937fe640 1 --2- 172.21.15.67:0/1387180852 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f56940b96d0 0x7f5694161170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:58.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.914+0000 7f5693fff640 1 --2- 172.21.15.67:0/1387180852 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5694161bb0 0x7f56940b7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:58.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.914+0000 7f56937fe640 1 -- 172.21.15.67:0/1387180852 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5694161bb0 msgr2=0x7f56940b7ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:58.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.914+0000 7f56937fe640 1 --2- 172.21.15.67:0/1387180852 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5694161bb0 0x7f56940b7ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.914+0000 7f56937fe640 1 -- 172.21.15.67:0/1387180852 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f569415dfe0 msgr2=0x7f56941f5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:58.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.914+0000 7f56937fe640 1 --2- 172.21.15.67:0/1387180852 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f569415dfe0 0x7f56941f5ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.914+0000 7f56937fe640 1 -- 172.21.15.67:0/1387180852 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56941f6580 con 0x7f56940b96d0 2024-03-24T00:38:58.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.914+0000 7f56937fe640 1 --2- 172.21.15.67:0/1387180852 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f56940b96d0 0x7f5694161170 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f568800e930 tx=0x7f568800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:58.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.914+0000 7f567bfff640 1 -- 172.21.15.67:0/1387180852 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5688017070 con 0x7f56940b96d0 2024-03-24T00:38:58.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.914+0000 7f567bfff640 1 -- 172.21.15.67:0/1387180852 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f568800ca00 con 0x7f56940b96d0 2024-03-24T00:38:58.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.914+0000 7f5699a63640 1 -- 172.21.15.67:0/1387180852 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56941f6860 con 0x7f56940b96d0 2024-03-24T00:38:58.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.915+0000 7f567bfff640 1 -- 172.21.15.67:0/1387180852 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f568800cdd0 con 0x7f56940b96d0 2024-03-24T00:38:58.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.915+0000 7f5699a63640 1 -- 172.21.15.67:0/1387180852 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56941f6e20 con 0x7f56940b96d0 2024-03-24T00:38:58.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.915+0000 7f5699a63640 1 -- 172.21.15.67:0/1387180852 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f569406b180 con 0x7f56940b96d0 2024-03-24T00:38:58.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.916+0000 7f567bfff640 1 -- 172.21.15.67:0/1387180852 <== mon.1 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f568801a070 con 0x7f56940b96d0 2024-03-24T00:38:58.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.916+0000 7f567bfff640 1 --2- 172.21.15.67:0/1387180852 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f568006bbd0 0x7f568006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:58.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.916+0000 7f567bfff640 1 -- 172.21.15.67:0/1387180852 <== mon.1 v2:172.21.15.72:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f568808af80 con 0x7f56940b96d0 2024-03-24T00:38:58.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.916+0000 7f5692ffd640 1 --2- 172.21.15.67:0/1387180852 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f568006bbd0 0x7f568006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:58.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.917+0000 7f5692ffd640 1 --2- 172.21.15.67:0/1387180852 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f568006bbd0 0x7f568006e090 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f568402b910 tx=0x7f568403c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:58.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:58.921+0000 7f567bfff640 1 -- 172.21.15.67:0/1387180852 <== mon.1 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f56880590c0 con 0x7f56940b96d0 2024-03-24T00:38:59.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.142+0000 7f5699a63640 1 -- 172.21.15.67:0/1387180852 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f569404f400 con 0x7f56940b96d0 2024-03-24T00:38:59.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.142+0000 7f567bfff640 1 -- 172.21.15.67:0/1387180852 <== mon.1 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4970 (secure 0 0 0) 0x7f568805d160 con 0x7f56940b96d0 2024-03-24T00:38:59.143 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:38:59.143 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4263,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/837528243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":837528243},{"type":"v1","addr":"172.21.15.67:6834","nonce":837528243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3987071456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":3987071456},{"type":"v1","addr":"172.21.15.67:6835","nonce":3987071456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4306,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/3953492281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3953492281},{"type":"v1","addr":"172.21.15.72:6837","nonce":3953492281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:38:52.127871+0000","modified":"2024-03-24T00:38:57.174252+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4309},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4309":{"gid":4309,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6836/3737698654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3737698654},{"type":"v1","addr":"172.21.15.72:6836","nonce":3737698654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4309,"qdb_cluster":[4309]},"id":1}]} 2024-03-24T00:38:59.144 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 7 2024-03-24T00:38:59.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.144+0000 7f5699a63640 1 -- 172.21.15.67:0/1387180852 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f568006bbd0 msgr2=0x7f568006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:59.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.144+0000 7f5699a63640 1 --2- 172.21.15.67:0/1387180852 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f568006bbd0 0x7f568006e090 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f568402b910 tx=0x7f568403c040 comp rx=0 tx=0).stop 2024-03-24T00:38:59.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.144+0000 7f5699a63640 1 -- 172.21.15.67:0/1387180852 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f56940b96d0 msgr2=0x7f5694161170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:59.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.144+0000 7f5699a63640 1 --2- 172.21.15.67:0/1387180852 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f56940b96d0 0x7f5694161170 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f568800e930 tx=0x7f568800edf0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.144+0000 7f5699a63640 1 -- 172.21.15.67:0/1387180852 shutdown_connections 2024-03-24T00:38:59.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.144+0000 7f5699a63640 1 --2- 172.21.15.67:0/1387180852 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f568006bbd0 0x7f568006e090 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.144+0000 7f5699a63640 1 --2- 172.21.15.67:0/1387180852 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5694161bb0 0x7f56940b7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.144+0000 7f5699a63640 1 --2- 172.21.15.67:0/1387180852 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f569415dfe0 0x7f56941f5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.144+0000 7f5699a63640 1 --2- 172.21.15.67:0/1387180852 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f56940b96d0 0x7f5694161170 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.144+0000 7f5699a63640 1 -- 172.21.15.67:0/1387180852 >> 172.21.15.67:0/1387180852 conn(0x7f569406c890 msgr2=0x7f56940b36a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:59.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.144+0000 7f5699a63640 1 -- 172.21.15.67:0/1387180852 shutdown_connections 2024-03-24T00:38:59.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.144+0000 7f5699a63640 1 -- 172.21.15.67:0/1387180852 wait complete. 2024-03-24T00:38:59.156 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:38:59.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.256+0000 7f945f426640 1 Processor -- start 2024-03-24T00:38:59.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.256+0000 7f945f426640 1 -- start start 2024-03-24T00:38:59.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.256+0000 7f945f426640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f945815df30 0x7f9458162610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:59.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.257+0000 7f945f426640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f94580d0d50 0x7f94580d2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:59.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.257+0000 7f945f426640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94580ce920 con 0x7f945815e310 2024-03-24T00:38:59.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.257+0000 7f945f426640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9458162b50 con 0x7f945815df30 2024-03-24T00:38:59.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.257+0000 7f945f426640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94580cef20 con 0x7f94580d0d50 2024-03-24T00:38:59.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.257+0000 7f945d1c2640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f945815df30 0x7f9458162610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:59.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.257+0000 7f945d1c2640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f945815df30 0x7f9458162610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59648/0 (socket says 172.21.15.67:59648) 2024-03-24T00:38:59.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.257+0000 7f945d1c2640 1 -- 172.21.15.67:0/4247263209 learned_addr learned my addr 172.21.15.67:0/4247263209 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:38:59.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.257+0000 7f945d9c3640 1 --2- 172.21.15.67:0/4247263209 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f94580d0d50 0x7f94580d2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:59.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.257+0000 7f945d1c2640 1 -- 172.21.15.67:0/4247263209 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f94580d0d50 msgr2=0x7f94580d2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:59.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.257+0000 7f945d1c2640 1 --2- 172.21.15.67:0/4247263209 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f94580d0d50 0x7f94580d2490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.257+0000 7f945d1c2640 1 -- 172.21.15.67:0/4247263209 >> v1:172.21.15.72:6789/0 conn(0x7f945815e310 legacy=0x7f94580d0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:59.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.257+0000 7f945d1c2640 1 -- 172.21.15.67:0/4247263209 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94580cf200 con 0x7f945815df30 2024-03-24T00:38:59.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.257+0000 7f9457fff640 1 -- 172.21.15.67:0/4247263209 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1494740684 0 0) 0x7f94580ce920 con 0x7f945815e310 2024-03-24T00:38:59.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.258+0000 7f945d1c2640 1 --2- 172.21.15.67:0/4247263209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f945815df30 0x7f9458162610 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9448009e50 tx=0x7f944802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36f1bd0d959f4a0f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:59.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.258+0000 7f9457fff640 1 -- 172.21.15.67:0/4247263209 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f944802ca70 con 0x7f945815df30 2024-03-24T00:38:59.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.258+0000 7f9457fff640 1 -- 172.21.15.67:0/4247263209 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f944802cc10 con 0x7f945815df30 2024-03-24T00:38:59.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.258+0000 7f9457fff640 1 -- 172.21.15.67:0/4247263209 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94480343f0 con 0x7f945815df30 2024-03-24T00:38:59.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.258+0000 7f945f426640 1 -- 172.21.15.67:0/4247263209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f945815df30 msgr2=0x7f9458162610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:59.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.258+0000 7f945f426640 1 --2- 172.21.15.67:0/4247263209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f945815df30 0x7f9458162610 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9448009e50 tx=0x7f944802bed0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.258+0000 7f945f426640 1 -- 172.21.15.67:0/4247263209 shutdown_connections 2024-03-24T00:38:59.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.258+0000 7f945f426640 1 --2- 172.21.15.67:0/4247263209 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f94580d0d50 0x7f94580d2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.258+0000 7f945f426640 1 --2- 172.21.15.67:0/4247263209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f945815df30 0x7f9458162610 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.258+0000 7f945f426640 1 -- 172.21.15.67:0/4247263209 >> 172.21.15.67:0/4247263209 conn(0x7f945806c880 msgr2=0x7f945806cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:59.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.258+0000 7f945f426640 1 -- 172.21.15.67:0/4247263209 shutdown_connections 2024-03-24T00:38:59.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.258+0000 7f945f426640 1 -- 172.21.15.67:0/4247263209 wait complete. 2024-03-24T00:38:59.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.260+0000 7f945f426640 1 Processor -- start 2024-03-24T00:38:59.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.260+0000 7f945f426640 1 -- start start 2024-03-24T00:38:59.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.260+0000 7f945f426640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f94580d0d50 0x7f94581e9120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:59.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.260+0000 7f945f426640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f945815df30 0x7f9458156cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:59.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.260+0000 7f945f426640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9458157200 0x7f94580b4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:59.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.260+0000 7f945f426640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94580ce580 con 0x7f945815df30 2024-03-24T00:38:59.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.260+0000 7f945f426640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94580b32f0 con 0x7f94580d0d50 2024-03-24T00:38:59.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.260+0000 7f945f426640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94580ce700 con 0x7f9458157200 2024-03-24T00:38:59.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.260+0000 7f945c9c1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f945815df30 0x7f9458156cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:59.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.260+0000 7f945c9c1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f945815df30 0x7f9458156cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59650/0 (socket says 172.21.15.67:59650) 2024-03-24T00:38:59.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.260+0000 7f945d1c2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f94580d0d50 0x7f94581e9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:59.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.260+0000 7f945c9c1640 1 -- 172.21.15.67:0/1983507241 learned_addr learned my addr 172.21.15.67:0/1983507241 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:38:59.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.260+0000 7f945d9c3640 1 --2- 172.21.15.67:0/1983507241 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9458157200 0x7f94580b4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:59.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.261+0000 7f945c9c1640 1 -- 172.21.15.67:0/1983507241 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9458157200 msgr2=0x7f94580b4050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:59.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.261+0000 7f945c9c1640 1 --2- 172.21.15.67:0/1983507241 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9458157200 0x7f94580b4050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.261+0000 7f945c9c1640 1 -- 172.21.15.67:0/1983507241 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f94580d0d50 msgr2=0x7f94581e9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:59.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.261+0000 7f945c9c1640 1 --2- 172.21.15.67:0/1983507241 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f94580d0d50 0x7f94581e9120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.261+0000 7f945c9c1640 1 -- 172.21.15.67:0/1983507241 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94580cbf50 con 0x7f945815df30 2024-03-24T00:38:59.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.261+0000 7f945c9c1640 1 --2- 172.21.15.67:0/1983507241 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f945815df30 0x7f9458156cc0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f944c00c410 tx=0x7f944c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:59.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.261+0000 7f9455ffb640 1 -- 172.21.15.67:0/1983507241 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f944c017070 con 0x7f945815df30 2024-03-24T00:38:59.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.261+0000 7f9455ffb640 1 -- 172.21.15.67:0/1983507241 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f944c0035f0 con 0x7f945815df30 2024-03-24T00:38:59.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.261+0000 7f945f426640 1 -- 172.21.15.67:0/1983507241 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94580cc230 con 0x7f945815df30 2024-03-24T00:38:59.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.261+0000 7f9455ffb640 1 -- 172.21.15.67:0/1983507241 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f944c0039e0 con 0x7f945815df30 2024-03-24T00:38:59.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.261+0000 7f945f426640 1 -- 172.21.15.67:0/1983507241 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94580cc7f0 con 0x7f945815df30 2024-03-24T00:38:59.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.262+0000 7f945f426640 1 -- 172.21.15.67:0/1983507241 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94580cc9f0 con 0x7f945815df30 2024-03-24T00:38:59.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.263+0000 7f9455ffb640 1 -- 172.21.15.67:0/1983507241 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f944c01a070 con 0x7f945815df30 2024-03-24T00:38:59.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.263+0000 7f9455ffb640 1 --2- 172.21.15.67:0/1983507241 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f944406bbd0 0x7f944406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:59.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.263+0000 7f9455ffb640 1 -- 172.21.15.67:0/1983507241 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f944c08a100 con 0x7f945815df30 2024-03-24T00:38:59.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.263+0000 7f945d1c2640 1 --2- 172.21.15.67:0/1983507241 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f944406bbd0 0x7f944406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:59.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.264+0000 7f945d1c2640 1 --2- 172.21.15.67:0/1983507241 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f944406bbd0 0x7f944406e090 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9448035000 tx=0x7f9448030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:59.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.267+0000 7f9455ffb640 1 -- 172.21.15.67:0/1983507241 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f94580cc9f0 con 0x7f945815df30 2024-03-24T00:38:59.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.453+0000 7f945f426640 1 -- 172.21.15.67:0/1983507241 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94580d1a30 con 0x7f945815df30 2024-03-24T00:38:59.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.454+0000 7f9455ffb640 1 -- 172.21.15.67:0/1983507241 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18454 (secure 0 0 0) 0x7f944c058320 con 0x7f945815df30 2024-03-24T00:38:59.455 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:38:59.456 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":25,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:38:58.797179+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:38:50.077085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:38:51.095454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T00:38:54.807117+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:38:59.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.456+0000 7f945f426640 1 -- 172.21.15.67:0/1983507241 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f944406bbd0 msgr2=0x7f944406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:59.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.456+0000 7f945f426640 1 --2- 172.21.15.67:0/1983507241 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f944406bbd0 0x7f944406e090 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9448035000 tx=0x7f9448030000 comp rx=0 tx=0).stop 2024-03-24T00:38:59.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.456+0000 7f945f426640 1 -- 172.21.15.67:0/1983507241 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f945815df30 msgr2=0x7f9458156cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:59.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.456+0000 7f945f426640 1 --2- 172.21.15.67:0/1983507241 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f945815df30 0x7f9458156cc0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f944c00c410 tx=0x7f944c00c8d0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.456+0000 7f945f426640 1 -- 172.21.15.67:0/1983507241 shutdown_connections 2024-03-24T00:38:59.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.456+0000 7f945f426640 1 --2- 172.21.15.67:0/1983507241 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f944406bbd0 0x7f944406e090 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.456+0000 7f945f426640 1 --2- 172.21.15.67:0/1983507241 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9458157200 0x7f94580b4050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.456+0000 7f945f426640 1 --2- 172.21.15.67:0/1983507241 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f945815df30 0x7f9458156cc0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.456+0000 7f945f426640 1 --2- 172.21.15.67:0/1983507241 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f94580d0d50 0x7f94581e9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.456+0000 7f945f426640 1 -- 172.21.15.67:0/1983507241 >> 172.21.15.67:0/1983507241 conn(0x7f945806c880 msgr2=0x7f94580d2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:59.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.456+0000 7f945f426640 1 -- 172.21.15.67:0/1983507241 shutdown_connections 2024-03-24T00:38:59.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.456+0000 7f945f426640 1 -- 172.21.15.67:0/1983507241 wait complete. 2024-03-24T00:38:59.870 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:38:52.127871+0000', 'modified': '2024-03-24T00:38:57.174252+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4309}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4309': {'gid': 4309, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6836/3737698654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 3737698654}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 3737698654}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 4309, 'qdb_cluster': [4309]} 2024-03-24T00:38:59.870 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-24T00:38:59.871 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-24T00:38:59.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.973+0000 7f90fbbdf640 1 Processor -- start 2024-03-24T00:38:59.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.974+0000 7f90fbbdf640 1 -- start start 2024-03-24T00:38:59.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.974+0000 7f90fbbdf640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f90f406b750 0x7f90f40d01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:59.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.974+0000 7f90fbbdf640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f90f40aeba0 0x7f90f40d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:59.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.974+0000 7f90fbbdf640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90f40ce970 con 0x7f90f415e200 2024-03-24T00:38:59.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.974+0000 7f90fbbdf640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90f40ad700 con 0x7f90f406b750 2024-03-24T00:38:59.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.974+0000 7f90fbbdf640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90f4162120 con 0x7f90f40aeba0 2024-03-24T00:38:59.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.974+0000 7f90f917a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f90f406b750 0x7f90f40d01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:59.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.974+0000 7f90f917a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f90f406b750 0x7f90f40d01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59662/0 (socket says 172.21.15.67:59662) 2024-03-24T00:38:59.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.974+0000 7f90f917a640 1 -- 172.21.15.67:0/2692115581 learned_addr learned my addr 172.21.15.67:0/2692115581 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:38:59.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.975+0000 7f90fa17c640 1 --2- 172.21.15.67:0/2692115581 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f90f40aeba0 0x7f90f40d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:59.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.975+0000 7f90f917a640 1 -- 172.21.15.67:0/2692115581 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f90f40aeba0 msgr2=0x7f90f40d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:59.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.975+0000 7f90f917a640 1 --2- 172.21.15.67:0/2692115581 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f90f40aeba0 0x7f90f40d06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.975+0000 7f90f917a640 1 -- 172.21.15.67:0/2692115581 >> v1:172.21.15.72:6789/0 conn(0x7f90f415e200 legacy=0x7f90f406b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:59.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.975+0000 7f90f917a640 1 -- 172.21.15.67:0/2692115581 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90f4162400 con 0x7f90f406b750 2024-03-24T00:38:59.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.975+0000 7f90f8979640 1 -- 172.21.15.67:0/2692115581 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2983778958 0 0) 0x7f90f40ce970 con 0x7f90f415e200 2024-03-24T00:38:59.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.975+0000 7f90f917a640 1 --2- 172.21.15.67:0/2692115581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90f406b750 0x7f90f40d01a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f90f0009870 tx=0x7f90f002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29854cf5b6ea8d5a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:59.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.976+0000 7f90f8979640 1 -- 172.21.15.67:0/2692115581 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f0035020 con 0x7f90f406b750 2024-03-24T00:38:59.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.976+0000 7f90f8979640 1 -- 172.21.15.67:0/2692115581 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90f0031640 con 0x7f90f406b750 2024-03-24T00:38:59.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.976+0000 7f90f8979640 1 -- 172.21.15.67:0/2692115581 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f002d040 con 0x7f90f406b750 2024-03-24T00:38:59.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.976+0000 7f90fbbdf640 1 -- 172.21.15.67:0/2692115581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90f406b750 msgr2=0x7f90f40d01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:59.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.976+0000 7f90fbbdf640 1 --2- 172.21.15.67:0/2692115581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90f406b750 0x7f90f40d01a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f90f0009870 tx=0x7f90f002b8f0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.976+0000 7f90fbbdf640 1 -- 172.21.15.67:0/2692115581 shutdown_connections 2024-03-24T00:38:59.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.976+0000 7f90fbbdf640 1 --2- 172.21.15.67:0/2692115581 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f90f40aeba0 0x7f90f40d06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.976+0000 7f90fbbdf640 1 --2- 172.21.15.67:0/2692115581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90f406b750 0x7f90f40d01a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.976+0000 7f90fbbdf640 1 -- 172.21.15.67:0/2692115581 >> 172.21.15.67:0/2692115581 conn(0x7f90f4099a30 msgr2=0x7f90f4099e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:59.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.976+0000 7f90fbbdf640 1 -- 172.21.15.67:0/2692115581 shutdown_connections 2024-03-24T00:38:59.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.976+0000 7f90fbbdf640 1 -- 172.21.15.67:0/2692115581 wait complete. 2024-03-24T00:38:59.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.977+0000 7f90fbbdf640 1 Processor -- start 2024-03-24T00:38:59.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.977+0000 7f90fbbdf640 1 -- start start 2024-03-24T00:38:59.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.978+0000 7f90fbbdf640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f90f406b750 0x7f90f40c7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:59.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.978+0000 7f90fbbdf640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90f40aeba0 0x7f90f414e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:59.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.978+0000 7f90fbbdf640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90f415e200 0x7f90f414eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:59.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.978+0000 7f90fbbdf640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f4161bb0 con 0x7f90f40aeba0 2024-03-24T00:38:59.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.978+0000 7f90fbbdf640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f41629c0 con 0x7f90f415e200 2024-03-24T00:38:59.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.978+0000 7f90fbbdf640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90f4161d30 con 0x7f90f406b750 2024-03-24T00:38:59.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.978+0000 7f90f917a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90f40aeba0 0x7f90f414e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:59.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.978+0000 7f90f917a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90f40aeba0 0x7f90f414e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59668/0 (socket says 172.21.15.67:59668) 2024-03-24T00:38:59.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.978+0000 7f90f917a640 1 -- 172.21.15.67:0/1475371634 learned_addr learned my addr 172.21.15.67:0/1475371634 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:38:59.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.978+0000 7f90fa17c640 1 --2- 172.21.15.67:0/1475371634 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90f415e200 0x7f90f414eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:59.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.978+0000 7f90f997b640 1 --2- 172.21.15.67:0/1475371634 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f90f406b750 0x7f90f40c7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:59.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.978+0000 7f90f917a640 1 -- 172.21.15.67:0/1475371634 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f90f406b750 msgr2=0x7f90f40c7d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:59.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.978+0000 7f90f917a640 1 --2- 172.21.15.67:0/1475371634 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f90f406b750 0x7f90f40c7d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.978+0000 7f90f917a640 1 -- 172.21.15.67:0/1475371634 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90f415e200 msgr2=0x7f90f414eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:59.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.978+0000 7f90f917a640 1 --2- 172.21.15.67:0/1475371634 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90f415e200 0x7f90f414eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.978+0000 7f90f917a640 1 -- 172.21.15.67:0/1475371634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90f4152e20 con 0x7f90f40aeba0 2024-03-24T00:38:59.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.979+0000 7f90f917a640 1 --2- 172.21.15.67:0/1475371634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90f40aeba0 0x7f90f414e570 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f90f0000c00 tx=0x7f90f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:59.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.979+0000 7f90e67fc640 1 -- 172.21.15.67:0/1475371634 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f0040070 con 0x7f90f40aeba0 2024-03-24T00:38:59.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.979+0000 7f90e67fc640 1 -- 172.21.15.67:0/1475371634 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90f00399f0 con 0x7f90f40aeba0 2024-03-24T00:38:59.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.979+0000 7f90e67fc640 1 -- 172.21.15.67:0/1475371634 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f0035040 con 0x7f90f40aeba0 2024-03-24T00:38:59.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.979+0000 7f90fbbdf640 1 -- 172.21.15.67:0/1475371634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90f4153100 con 0x7f90f40aeba0 2024-03-24T00:38:59.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.979+0000 7f90fbbdf640 1 -- 172.21.15.67:0/1475371634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90f41536c0 con 0x7f90f40aeba0 2024-03-24T00:38:59.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.980+0000 7f90e67fc640 1 -- 172.21.15.67:0/1475371634 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f90f004b060 con 0x7f90f40aeba0 2024-03-24T00:38:59.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.980+0000 7f90e67fc640 1 --2- 172.21.15.67:0/1475371634 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f90d806bc00 0x7f90d806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:59.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.981+0000 7f90e67fc640 1 -- 172.21.15.67:0/1475371634 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f90f00ac4e0 con 0x7f90f40aeba0 2024-03-24T00:38:59.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.981+0000 7f90f997b640 1 --2- 172.21.15.67:0/1475371634 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f90d806bc00 0x7f90d806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:59.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.981+0000 7f90f997b640 1 --2- 172.21.15.67:0/1475371634 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f90d806bc00 0x7f90d806e0c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f90e8000f60 tx=0x7f90e8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:59.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.982+0000 7f90fe7c8740 1 Processor -- start 2024-03-24T00:38:59.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.982+0000 7f90fe7c8740 1 -- start start 2024-03-24T00:38:59.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.982+0000 7f90fe7c8740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5599a543dc40 0x5599a5464770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:59.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.982+0000 7f90fe7c8740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5599a5464cb0 0x5599a543cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:59.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.982+0000 7f90fe7c8740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5599a543e020 0x5599a543d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:59.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.982+0000 7f90fe7c8740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599a53a1c20 con 0x5599a543dc40 2024-03-24T00:38:59.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.982+0000 7f90fe7c8740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599a534aa40 con 0x5599a5464cb0 2024-03-24T00:38:59.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.982+0000 7f90fe7c8740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5599a539dcc0 con 0x5599a543e020 2024-03-24T00:38:59.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.982+0000 7f90f997b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5599a543dc40 0x5599a5464770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:59.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.982+0000 7f90f997b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5599a543dc40 0x5599a5464770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59676/0 (socket says 172.21.15.67:59676) 2024-03-24T00:38:59.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.982+0000 7f90f997b640 1 -- 172.21.15.67:0/925486244 learned_addr learned my addr 172.21.15.67:0/925486244 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:38:59.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.982+0000 7f90fa17c640 1 --2- 172.21.15.67:0/925486244 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5599a543e020 0x5599a543d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:59.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.982+0000 7f90f917a640 1 --2- 172.21.15.67:0/925486244 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5599a5464cb0 0x5599a543cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:59.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.983+0000 7f90f997b640 1 -- 172.21.15.67:0/925486244 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5599a543e020 msgr2=0x5599a543d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:59.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.983+0000 7f90f997b640 1 --2- 172.21.15.67:0/925486244 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5599a543e020 0x5599a543d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.983+0000 7f90f997b640 1 -- 172.21.15.67:0/925486244 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5599a5464cb0 msgr2=0x5599a543cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:59.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.983+0000 7f90f997b640 1 --2- 172.21.15.67:0/925486244 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5599a5464cb0 0x5599a543cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.983+0000 7f90f997b640 1 -- 172.21.15.67:0/925486244 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5599a536cf90 con 0x5599a543dc40 2024-03-24T00:38:59.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.983+0000 7f90f917a640 1 --2- 172.21.15.67:0/925486244 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5599a5464cb0 0x5599a543cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:59.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.983+0000 7f90f997b640 1 --2- 172.21.15.67:0/925486244 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5599a543dc40 0x5599a5464770 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f90e800fb70 tx=0x7f90e8012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c8654ef5c4d776b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:59.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.983+0000 7f90d77fe640 1 -- 172.21.15.67:0/925486244 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e8025070 con 0x5599a543dc40 2024-03-24T00:38:59.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.983+0000 7f90d77fe640 1 -- 172.21.15.67:0/925486244 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90e801b050 con 0x5599a543dc40 2024-03-24T00:38:59.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.983+0000 7f90d77fe640 1 -- 172.21.15.67:0/925486244 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e8014de0 con 0x5599a543dc40 2024-03-24T00:38:59.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.984+0000 7f90fe7c8740 1 -- 172.21.15.67:0/925486244 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5599a543dc40 msgr2=0x5599a5464770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:59.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.984+0000 7f90fe7c8740 1 --2- 172.21.15.67:0/925486244 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5599a543dc40 0x5599a5464770 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f90e800fb70 tx=0x7f90e8012000 comp rx=0 tx=0).stop 2024-03-24T00:38:59.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.984+0000 7f90fe7c8740 1 -- 172.21.15.67:0/925486244 shutdown_connections 2024-03-24T00:38:59.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.984+0000 7f90fe7c8740 1 --2- 172.21.15.67:0/925486244 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5599a543e020 0x5599a543d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.984+0000 7f90fe7c8740 1 --2- 172.21.15.67:0/925486244 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5599a5464cb0 0x5599a543cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.984+0000 7f90fe7c8740 1 --2- 172.21.15.67:0/925486244 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5599a543dc40 0x5599a5464770 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.984+0000 7f90fe7c8740 1 -- 172.21.15.67:0/925486244 >> 172.21.15.67:0/925486244 conn(0x5599a5404a60 msgr2=0x5599a5482960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:59.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.984+0000 7f90fe7c8740 1 -- 172.21.15.67:0/925486244 shutdown_connections 2024-03-24T00:38:59.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.984+0000 7f90fe7c8740 1 -- 172.21.15.67:0/925486244 wait complete. 2024-03-24T00:38:59.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.984+0000 7f90fe7c8740 1 -- start start 2024-03-24T00:38:59.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.984+0000 7f90fe7c8740 1 Processor -- start 2024-03-24T00:38:59.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.985+0000 7f90fe7c8740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5599a543dc40 0x5599a545bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:59.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.985+0000 7f90fe7c8740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5599a543e020 0x5599a54819b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:59.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.985+0000 7f90fe7c8740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5599a5483510 0x5599a5468ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:59.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.985+0000 7f90fe7c8740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599a54642e0 con 0x5599a543dc40 2024-03-24T00:38:59.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.985+0000 7f90fe7c8740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599a535de90 con 0x5599a543e020 2024-03-24T00:38:59.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.985+0000 7f90fe7c8740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5599a545d060 con 0x5599a5483510 2024-03-24T00:38:59.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.985+0000 7f90fe7c8740 10 client.0 _ll_register_callbacks cb 0x5599a50b5ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:38:59.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.985+0000 7f90f997b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5599a543dc40 0x5599a545bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:59.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.985+0000 7f90f997b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5599a543dc40 0x5599a545bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59692/0 (socket says 172.21.15.67:59692) 2024-03-24T00:38:59.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.985+0000 7f90f997b640 1 -- 172.21.15.67:0/13350116 learned_addr learned my addr 172.21.15.67:0/13350116 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:38:59.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.985+0000 7f90f917a640 1 --2- 172.21.15.67:0/13350116 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5599a543e020 0x5599a54819b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:59.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.985+0000 7f90fa17c640 1 --2- 172.21.15.67:0/13350116 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5599a5483510 0x5599a5468ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:59.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.985+0000 7f90f997b640 1 -- 172.21.15.67:0/13350116 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5599a5483510 msgr2=0x5599a5468ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:59.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.985+0000 7f90f997b640 1 --2- 172.21.15.67:0/13350116 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5599a5483510 0x5599a5468ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.985+0000 7f90f997b640 1 -- 172.21.15.67:0/13350116 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5599a543e020 msgr2=0x5599a54819b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:59.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.985+0000 7f90f997b640 1 --2- 172.21.15.67:0/13350116 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5599a543e020 0x5599a54819b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.985+0000 7f90f997b640 1 -- 172.21.15.67:0/13350116 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5599a536a9d0 con 0x5599a543dc40 2024-03-24T00:38:59.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.985+0000 7f90f917a640 1 --2- 172.21.15.67:0/13350116 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5599a543e020 0x5599a54819b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:59.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.986+0000 7f90f997b640 1 --2- 172.21.15.67:0/13350116 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5599a543dc40 0x5599a545bfc0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f90e8014820 tx=0x7f90e8001080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41c04ce568f0b33c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:59.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.986+0000 7f90d4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:38:59.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.986+0000 7f90d4ff9640 1 -- 172.21.15.67:0/13350116 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e802f020 con 0x5599a543dc40 2024-03-24T00:38:59.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.986+0000 7f90d4ff9640 1 -- 172.21.15.67:0/13350116 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f90e8020a70 con 0x5599a543dc40 2024-03-24T00:38:59.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.986+0000 7f90fe7c8740 1 -- 172.21.15.67:0/13350116 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f90e801b050 con 0x5599a543dc40 2024-03-24T00:38:59.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.986+0000 7f90d4ff9640 1 -- 172.21.15.67:0/13350116 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7f90e8039050 con 0x5599a543dc40 2024-03-24T00:38:59.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.986+0000 7f90fe7c8740 10 client.4416 fetch_fsmap learned FSMap version 7 2024-03-24T00:38:59.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.986+0000 7f90fe7c8740 1 -- 172.21.15.67:0/13350116 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x5599a50b62d0 con 0x5599a543dc40 2024-03-24T00:38:59.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.986+0000 7f90d4ff9640 1 -- 172.21.15.67:0/13350116 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90e801b050 con 0x5599a543dc40 2024-03-24T00:38:59.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.986+0000 7f90d4ff9640 1 -- 172.21.15.67:0/13350116 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 7) v1 ==== 3322+0+0 (secure 0 0 0) 0x7f90e803c020 con 0x5599a543dc40 2024-03-24T00:38:59.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.986+0000 7f90fe7c8740 10 client.4416 fetch_fsmap finished waiting for FSMap version 7 2024-03-24T00:38:59.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.986+0000 7f90fe7c8740 10 client.4416 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T00:38:59.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.986+0000 7f90fe7c8740 20 client.4416 populate_metadata read hostname 'smithi067' 2024-03-24T00:38:59.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.986+0000 7f90fe7c8740 1 --2- 172.21.15.67:0/13350116 >> [v2:172.21.15.72:6835/3737698654,v1:172.21.15.72:6836/3737698654] conn(0x5599a549de50 0x5599a54a0240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:59.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.986+0000 7f90fe7c8740 4 client.4416 mds_command: new command op to 4309 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:38:59.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.986+0000 7f90fe7c8740 1 -- 172.21.15.67:0/13350116 --> [v2:172.21.15.72:6835/3737698654,v1:172.21.15.72:6836/3737698654] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5599a53af160 con 0x5599a549de50 2024-03-24T00:38:59.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.986+0000 7f90f917a640 1 --2- 172.21.15.67:0/13350116 >> [v2:172.21.15.72:6835/3737698654,v1:172.21.15.72:6836/3737698654] conn(0x5599a549de50 0x5599a54a0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:59.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.987+0000 7f90f917a640 1 --2- 172.21.15.67:0/13350116 >> [v2:172.21.15.72:6835/3737698654,v1:172.21.15.72:6836/3737698654] conn(0x5599a549de50 0x5599a54a0240 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f1435c3535ab8fee server_cookie=66b22f43ddb2e598 in_seq=0 out_seq=0 2024-03-24T00:38:59.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.987+0000 7f90d4ff9640 10 client.4416 ms_handle_connect on v2:172.21.15.72:6835/3737698654 2024-03-24T00:38:59.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.987+0000 7f90d4ff9640 1 -- 172.21.15.67:0/13350116 <== mds.0 v2:172.21.15.72:6835/3737698654 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5599a53af160 con 0x5599a549de50 2024-03-24T00:38:59.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.987+0000 7f90d4ff9640 10 client.4416 handle_command_reply: tid=0 2024-03-24T00:38:59.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.987+0000 7f90fe7c8740 1 client.4416 shutdown 2024-03-24T00:38:59.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.988+0000 7f90fe7c8740 1 -- 172.21.15.67:0/13350116 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5599a543dc40 msgr2=0x5599a545bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:59.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.988+0000 7f90fe7c8740 1 --2- 172.21.15.67:0/13350116 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5599a543dc40 0x5599a545bfc0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f90e8014820 tx=0x7f90e8001080 comp rx=0 tx=0).stop 2024-03-24T00:38:59.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.988+0000 7f90fe7c8740 1 -- 172.21.15.67:0/13350116 shutdown_connections 2024-03-24T00:38:59.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.988+0000 7f90fe7c8740 1 --2- 172.21.15.67:0/13350116 >> [v2:172.21.15.72:6835/3737698654,v1:172.21.15.72:6836/3737698654] conn(0x5599a549de50 0x5599a54a0240 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.988+0000 7f90fe7c8740 1 --2- 172.21.15.67:0/13350116 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5599a5483510 0x5599a5468ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.988+0000 7f90d4ff9640 0 client.4416 ms_handle_reset on v2:172.21.15.72:6835/3737698654 2024-03-24T00:38:59.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.988+0000 7f90fe7c8740 1 --2- 172.21.15.67:0/13350116 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5599a543e020 0x5599a54819b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.988+0000 7f90fe7c8740 1 --2- 172.21.15.67:0/13350116 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5599a543dc40 0x5599a545bfc0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.988+0000 7f90fe7c8740 1 -- 172.21.15.67:0/13350116 >> 172.21.15.67:0/13350116 conn(0x5599a5404a60 msgr2=0x5599a5463b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:59.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.988+0000 7f90fe7c8740 1 -- 172.21.15.67:0/13350116 shutdown_connections 2024-03-24T00:38:59.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.988+0000 7f90fe7c8740 1 -- 172.21.15.67:0/13350116 wait complete. 2024-03-24T00:38:59.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:38:59.988+0000 7f90fe7c8740 20 client.4416 trim_cache size 0 max 16384 2024-03-24T00:39:00.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.005+0000 7f90fe7c8740 1 Processor -- start 2024-03-24T00:39:00.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.005+0000 7f90fe7c8740 1 -- start start 2024-03-24T00:39:00.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.005+0000 7f90fe7c8740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5599a543dc40 0x5599a5463d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:00.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.005+0000 7f90fe7c8740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5599a543e020 0x5599a5485ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:00.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.005+0000 7f90fe7c8740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5599a5483510 0x5599a547eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:00.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.005+0000 7f90fe7c8740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f42007d0 con 0x5599a5483510 2024-03-24T00:39:00.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.005+0000 7f90fe7c8740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f41550a0 con 0x5599a543e020 2024-03-24T00:39:00.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.005+0000 7f90fe7c8740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90f40c6a60 con 0x5599a543dc40 2024-03-24T00:39:00.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.005+0000 7f90fa17c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5599a5483510 0x5599a547eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:00.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.005+0000 7f90fa17c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5599a5483510 0x5599a547eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59700/0 (socket says 172.21.15.67:59700) 2024-03-24T00:39:00.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.005+0000 7f90f917a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5599a543e020 0x5599a5485ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:00.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.005+0000 7f90fa17c640 1 -- 172.21.15.67:0/1886890545 learned_addr learned my addr 172.21.15.67:0/1886890545 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:00.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.005+0000 7f90f997b640 1 --2- 172.21.15.67:0/1886890545 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5599a543dc40 0x5599a5463d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:00.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.006+0000 7f90fa17c640 1 -- 172.21.15.67:0/1886890545 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5599a543dc40 msgr2=0x5599a5463d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:00.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.006+0000 7f90fa17c640 1 --2- 172.21.15.67:0/1886890545 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5599a543dc40 0x5599a5463d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.006+0000 7f90fa17c640 1 -- 172.21.15.67:0/1886890545 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5599a543e020 msgr2=0x5599a5485ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:00.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.006+0000 7f90fa17c640 1 --2- 172.21.15.67:0/1886890545 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5599a543e020 0x5599a5485ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.006+0000 7f90fa17c640 1 -- 172.21.15.67:0/1886890545 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5599a51806b0 con 0x5599a5483510 2024-03-24T00:39:00.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.006+0000 7f90f917a640 1 --2- 172.21.15.67:0/1886890545 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5599a543e020 0x5599a5485ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:00.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.006+0000 7f90fa17c640 1 --2- 172.21.15.67:0/1886890545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5599a5483510 0x5599a547eba0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f90ec017760 tx=0x7f90ec009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca404569da1cae97 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:00.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.006+0000 7f90d6ffd640 1 -- 172.21.15.67:0/1886890545 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ec024070 con 0x5599a5483510 2024-03-24T00:39:00.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.006+0000 7f90d6ffd640 1 -- 172.21.15.67:0/1886890545 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90ec015cc0 con 0x5599a5483510 2024-03-24T00:39:00.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.006+0000 7f90d6ffd640 1 -- 172.21.15.67:0/1886890545 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ec01f670 con 0x5599a5483510 2024-03-24T00:39:00.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.006+0000 7f90fe7c8740 1 -- 172.21.15.67:0/1886890545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5599a5483510 msgr2=0x5599a547eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:00.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.006+0000 7f90fe7c8740 1 --2- 172.21.15.67:0/1886890545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5599a5483510 0x5599a547eba0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f90ec017760 tx=0x7f90ec009750 comp rx=0 tx=0).stop 2024-03-24T00:39:00.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.006+0000 7f90fe7c8740 1 -- 172.21.15.67:0/1886890545 shutdown_connections 2024-03-24T00:39:00.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.006+0000 7f90fe7c8740 1 --2- 172.21.15.67:0/1886890545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5599a5483510 0x5599a547eba0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.006+0000 7f90fe7c8740 1 --2- 172.21.15.67:0/1886890545 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5599a543e020 0x5599a5485ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.006+0000 7f90fe7c8740 1 --2- 172.21.15.67:0/1886890545 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5599a543dc40 0x5599a5463d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.006+0000 7f90fe7c8740 1 -- 172.21.15.67:0/1886890545 >> 172.21.15.67:0/1886890545 conn(0x5599a5404a60 msgr2=0x5599a5480230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:00.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.006+0000 7f90fe7c8740 1 -- 172.21.15.67:0/1886890545 shutdown_connections 2024-03-24T00:39:00.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.006+0000 7f90fe7c8740 1 -- 172.21.15.67:0/1886890545 wait complete. 2024-03-24T00:39:00.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.007+0000 7f90fe7c8740 1 -- start start 2024-03-24T00:39:00.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.007+0000 7f90fe7c8740 1 Processor -- start 2024-03-24T00:39:00.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.008+0000 7f90fe7c8740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5599a543dc40 0x5599a5469880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:00.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.008+0000 7f90fe7c8740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5599a543e020 0x5599a54f13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:00.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.008+0000 7f90fe7c8740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5599a5483510 0x5599a547bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:00.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.008+0000 7f90fe7c8740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599a54695f0 con 0x5599a5483510 2024-03-24T00:39:00.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.008+0000 7f90fe7c8740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599a5482ff0 con 0x5599a543dc40 2024-03-24T00:39:00.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.008+0000 7f90fe7c8740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5599a5484c10 con 0x5599a543e020 2024-03-24T00:39:00.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.008+0000 7f90fe7c8740 10 client.0 _ll_register_callbacks cb 0x7f90e8023070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:39:00.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.008+0000 7f90fa17c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5599a5483510 0x5599a547bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:00.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.008+0000 7f90f997b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5599a543dc40 0x5599a5469880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:00.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.008+0000 7f90fa17c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5599a5483510 0x5599a547bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59710/0 (socket says 172.21.15.67:59710) 2024-03-24T00:39:00.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.008+0000 7f90f997b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5599a543dc40 0x5599a5469880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:53808/0 (socket says 172.21.15.67:53808) 2024-03-24T00:39:00.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.008+0000 7f90f917a640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5599a543e020 0x5599a54f13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:00.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.008+0000 7f90fa17c640 1 -- 172.21.15.67:0/1482356857 learned_addr learned my addr 172.21.15.67:0/1482356857 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:00.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.008+0000 7f90f997b640 1 -- 172.21.15.67:0/1482356857 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5599a543e020 msgr2=0x5599a54f13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:00.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.008+0000 7f90f997b640 1 --2- 172.21.15.67:0/1482356857 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5599a543e020 0x5599a54f13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.008+0000 7f90f997b640 1 -- 172.21.15.67:0/1482356857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5599a5483510 msgr2=0x5599a547bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:00.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.008+0000 7f90f997b640 1 --2- 172.21.15.67:0/1482356857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5599a5483510 0x5599a547bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.008+0000 7f90f997b640 1 -- 172.21.15.67:0/1482356857 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5599a54f5810 con 0x5599a543dc40 2024-03-24T00:39:00.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.008+0000 7f90fa17c640 1 --2- 172.21.15.67:0/1482356857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5599a5483510 0x5599a547bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:00.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.009+0000 7f90f997b640 1 --2- 172.21.15.67:0/1482356857 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5599a543dc40 0x5599a5469880 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f90e800f610 tx=0x7f90e800a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3e80a011e352333 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:00.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.009+0000 7f90d4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-24T00:39:00.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.009+0000 7f90d4ff9640 1 -- 172.21.15.67:0/1482356857 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e802f070 con 0x5599a543dc40 2024-03-24T00:39:00.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.009+0000 7f90d4ff9640 1 -- 172.21.15.67:0/1482356857 <== mon.1 v2:172.21.15.72:3300/0 2 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f90e803ede0 con 0x5599a543dc40 2024-03-24T00:39:00.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.009+0000 7f90fe7c8740 1 -- 172.21.15.67:0/1482356857 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f90ec015cc0 con 0x5599a543dc40 2024-03-24T00:39:00.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.009+0000 7f90d4ff9640 1 -- 172.21.15.67:0/1482356857 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7f90e8045050 con 0x5599a543dc40 2024-03-24T00:39:00.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.009+0000 7f90fe7c8740 10 client.4432 fetch_fsmap learned FSMap version 7 2024-03-24T00:39:00.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.009+0000 7f90fe7c8740 1 -- 172.21.15.67:0/1482356857 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x5599a54a53a0 con 0x5599a543dc40 2024-03-24T00:39:00.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.009+0000 7f90d4ff9640 1 -- 172.21.15.67:0/1482356857 <== mon.1 v2:172.21.15.72:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90ec015cc0 con 0x5599a543dc40 2024-03-24T00:39:00.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.009+0000 7f90d4ff9640 1 -- 172.21.15.67:0/1482356857 <== mon.1 v2:172.21.15.72:3300/0 5 ==== fsmap(e 7) v1 ==== 3322+0+0 (secure 0 0 0) 0x7f90e8048020 con 0x5599a543dc40 2024-03-24T00:39:00.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.009+0000 7f90fe7c8740 10 client.4432 fetch_fsmap finished waiting for FSMap version 7 2024-03-24T00:39:00.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.009+0000 7f90fe7c8740 10 client.4432 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T00:39:00.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.009+0000 7f90fe7c8740 20 client.4432 populate_metadata read hostname 'smithi067' 2024-03-24T00:39:00.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.009+0000 7f90fe7c8740 1 --2- 172.21.15.67:0/1482356857 >> [v2:172.21.15.72:6835/3737698654,v1:172.21.15.72:6836/3737698654] conn(0x5599a54a46b0 0x5599a54782f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:00.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.009+0000 7f90fe7c8740 4 client.4432 mds_command: new command op to 4309 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:39:00.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.009+0000 7f90fe7c8740 1 -- 172.21.15.67:0/1482356857 --> [v2:172.21.15.72:6835/3737698654,v1:172.21.15.72:6836/3737698654] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5599a53a94b0 con 0x5599a54a46b0 2024-03-24T00:39:00.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.009+0000 7f90f917a640 1 --2- 172.21.15.67:0/1482356857 >> [v2:172.21.15.72:6835/3737698654,v1:172.21.15.72:6836/3737698654] conn(0x5599a54a46b0 0x5599a54782f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:00.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.010+0000 7f90d4ff9640 10 client.4432 ms_handle_connect on v2:172.21.15.72:6835/3737698654 2024-03-24T00:39:00.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.010+0000 7f90f917a640 1 --2- 172.21.15.67:0/1482356857 >> [v2:172.21.15.72:6835/3737698654,v1:172.21.15.72:6836/3737698654] conn(0x5599a54a46b0 0x5599a54782f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18fb01bd2a72d48e server_cookie=e16398139d9fa977 in_seq=0 out_seq=0 2024-03-24T00:39:00.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.010+0000 7f90d4ff9640 1 -- 172.21.15.67:0/1482356857 <== mds.0 v2:172.21.15.72:6835/3737698654 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x5599a53a94b0 con 0x5599a54a46b0 2024-03-24T00:39:00.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.010+0000 7f90d4ff9640 10 client.4432 handle_command_reply: tid=0 2024-03-24T00:39:00.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.010+0000 7f90fe7c8740 1 client.4432 shutdown 2024-03-24T00:39:00.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.010+0000 7f90fe7c8740 1 -- 172.21.15.67:0/1482356857 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5599a543dc40 msgr2=0x5599a5469880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:00.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.010+0000 7f90fe7c8740 1 --2- 172.21.15.67:0/1482356857 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5599a543dc40 0x5599a5469880 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f90e800f610 tx=0x7f90e800a000 comp rx=0 tx=0).stop 2024-03-24T00:39:00.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.010+0000 7f90fe7c8740 1 -- 172.21.15.67:0/1482356857 shutdown_connections 2024-03-24T00:39:00.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.010+0000 7f90fe7c8740 1 --2- 172.21.15.67:0/1482356857 >> [v2:172.21.15.72:6835/3737698654,v1:172.21.15.72:6836/3737698654] conn(0x5599a54a46b0 0x5599a54782f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.010+0000 7f90fe7c8740 1 --2- 172.21.15.67:0/1482356857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5599a5483510 0x5599a547bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.010+0000 7f90fe7c8740 1 --2- 172.21.15.67:0/1482356857 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5599a543e020 0x5599a54f13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.010+0000 7f90fe7c8740 1 --2- 172.21.15.67:0/1482356857 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5599a543dc40 0x5599a5469880 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.010+0000 7f90d4ff9640 0 client.4432 ms_handle_reset on v2:172.21.15.72:6835/3737698654 2024-03-24T00:39:00.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.010+0000 7f90fe7c8740 1 -- 172.21.15.67:0/1482356857 >> 172.21.15.67:0/1482356857 conn(0x5599a5404a60 msgr2=0x5599a5468540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:00.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.010+0000 7f90fe7c8740 1 -- 172.21.15.67:0/1482356857 shutdown_connections 2024-03-24T00:39:00.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.010+0000 7f90fe7c8740 1 -- 172.21.15.67:0/1482356857 wait complete. 2024-03-24T00:39:00.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.010+0000 7f90fe7c8740 20 client.4432 trim_cache size 0 max 16384 2024-03-24T00:39:00.017 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:39:00.018 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "de1d0178-808e-49ca-a986-e297e523b976", 2024-03-24T00:39:00.018 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-03-24T00:39:00.018 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 4309, 2024-03-24T00:39:00.018 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-03-24T00:39:00.018 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-03-24T00:39:00.018 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-03-24T00:39:00.018 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 7.2488534360000001, 2024-03-24T00:39:00.018 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 7, 2024-03-24T00:39:00.018 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 25, 2024-03-24T00:39:00.018 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 24, 2024-03-24T00:39:00.018 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 14.472291783999999 2024-03-24T00:39:00.018 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:39:00.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.011+0000 7f90fbbdf640 1 -- 172.21.15.67:0/1475371634 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f90d806bc00 msgr2=0x7f90d806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:00.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.011+0000 7f90fbbdf640 1 --2- 172.21.15.67:0/1475371634 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f90d806bc00 0x7f90d806e0c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f90e8000f60 tx=0x7f90e8009040 comp rx=0 tx=0).stop 2024-03-24T00:39:00.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.011+0000 7f90fbbdf640 1 -- 172.21.15.67:0/1475371634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90f40aeba0 msgr2=0x7f90f414e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:00.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.011+0000 7f90fbbdf640 1 --2- 172.21.15.67:0/1475371634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90f40aeba0 0x7f90f414e570 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f90f0000c00 tx=0x7f90f00046c0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.011+0000 7f90fbbdf640 1 -- 172.21.15.67:0/1475371634 shutdown_connections 2024-03-24T00:39:00.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.011+0000 7f90fbbdf640 1 --2- 172.21.15.67:0/1475371634 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f90d806bc00 0x7f90d806e0c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.011+0000 7f90fbbdf640 1 --2- 172.21.15.67:0/1475371634 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90f415e200 0x7f90f414eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.011+0000 7f90fbbdf640 1 --2- 172.21.15.67:0/1475371634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90f40aeba0 0x7f90f414e570 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.011+0000 7f90fbbdf640 1 --2- 172.21.15.67:0/1475371634 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f90f406b750 0x7f90f40c7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.011+0000 7f90fbbdf640 1 -- 172.21.15.67:0/1475371634 >> 172.21.15.67:0/1475371634 conn(0x7f90f4099a30 msgr2=0x7f90f40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:00.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.011+0000 7f90fbbdf640 1 -- 172.21.15.67:0/1475371634 shutdown_connections 2024-03-24T00:39:00.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.011+0000 7f90fbbdf640 1 -- 172.21.15.67:0/1475371634 wait complete. 2024-03-24T00:39:00.022 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-24T00:39:00.022 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-24T00:39:00.022 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T00:39:00.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.122+0000 7f55e1c4b640 1 Processor -- start 2024-03-24T00:39:00.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.122+0000 7f55e1c4b640 1 -- start start 2024-03-24T00:39:00.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.123+0000 7f55e1c4b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f55dc15dec0 0x7f55dc0ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:00.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.123+0000 7f55e1c4b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f55dc0d0ef0 0x7f55dc162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:00.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.123+0000 7f55e1c4b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55dc0cf640 con 0x7f55dc15e2a0 2024-03-24T00:39:00.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.123+0000 7f55e1c4b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55dc0cee30 con 0x7f55dc0d0ef0 2024-03-24T00:39:00.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.123+0000 7f55e1c4b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55dc0d2120 con 0x7f55dc15dec0 2024-03-24T00:39:00.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.123+0000 7f55daffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f55dc15e2a0 0x7f55dc0d07e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56872/0 (socket says 172.21.15.67:56872) 2024-03-24T00:39:00.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.123+0000 7f55daffd640 1 -- 172.21.15.67:0/1424467733 learned_addr learned my addr 172.21.15.67:0/1424467733 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:00.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.123+0000 7f55db7fe640 1 --2- 172.21.15.67:0/1424467733 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f55dc15dec0 0x7f55dc0ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:00.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.123+0000 7f55dbfff640 1 --2- 172.21.15.67:0/1424467733 >> v2:172.21.15.72:3300/0 conn(0x7f55dc0d0ef0 0x7f55dc162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:00.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.124+0000 7f55da7fc640 1 -- 172.21.15.67:0/1424467733 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 184267391 0 0) 0x7f55d0001110 con 0x7f55dc15e2a0 2024-03-24T00:39:00.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.124+0000 7f55da7fc640 1 -- 172.21.15.67:0/1424467733 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55c4003520 con 0x7f55dc15e2a0 2024-03-24T00:39:00.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.124+0000 7f55dbfff640 1 -- 172.21.15.67:0/1424467733 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f55dc15dec0 msgr2=0x7f55dc0ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:00.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.124+0000 7f55dbfff640 1 --2- 172.21.15.67:0/1424467733 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f55dc15dec0 0x7f55dc0ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.124+0000 7f55dbfff640 1 -- 172.21.15.67:0/1424467733 >> v1:172.21.15.67:6789/0 conn(0x7f55dc15e2a0 legacy=0x7f55dc0d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:00.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.124+0000 7f55dbfff640 1 -- 172.21.15.67:0/1424467733 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55dc0cf080 con 0x7f55dc0d0ef0 2024-03-24T00:39:00.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.124+0000 7f55db7fe640 1 --2- 172.21.15.67:0/1424467733 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f55dc15dec0 0x7f55dc0ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:00.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.125+0000 7f55dbfff640 1 --2- 172.21.15.67:0/1424467733 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f55dc0d0ef0 0x7f55dc162630 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f55d4009d90 tx=0x7f55d402be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0b13902436984a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:00.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.125+0000 7f55da7fc640 1 -- 172.21.15.67:0/1424467733 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55d4039070 con 0x7f55dc0d0ef0 2024-03-24T00:39:00.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.125+0000 7f55da7fc640 1 -- 172.21.15.67:0/1424467733 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55d402cbc0 con 0x7f55dc0d0ef0 2024-03-24T00:39:00.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.125+0000 7f55da7fc640 1 -- 172.21.15.67:0/1424467733 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55d4034440 con 0x7f55dc0d0ef0 2024-03-24T00:39:00.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.125+0000 7f55e1c4b640 1 -- 172.21.15.67:0/1424467733 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f55dc0d0ef0 msgr2=0x7f55dc162630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:00.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.125+0000 7f55e1c4b640 1 --2- 172.21.15.67:0/1424467733 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f55dc0d0ef0 0x7f55dc162630 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f55d4009d90 tx=0x7f55d402be10 comp rx=0 tx=0).stop 2024-03-24T00:39:00.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.125+0000 7f55e1c4b640 1 -- 172.21.15.67:0/1424467733 shutdown_connections 2024-03-24T00:39:00.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.125+0000 7f55e1c4b640 1 --2- 172.21.15.67:0/1424467733 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f55dc0d0ef0 0x7f55dc162630 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.125+0000 7f55e1c4b640 1 --2- 172.21.15.67:0/1424467733 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f55dc15dec0 0x7f55dc0ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.125+0000 7f55e1c4b640 1 -- 172.21.15.67:0/1424467733 >> 172.21.15.67:0/1424467733 conn(0x7f55dc06c810 msgr2=0x7f55dc06cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:00.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.125+0000 7f55e1c4b640 1 -- 172.21.15.67:0/1424467733 shutdown_connections 2024-03-24T00:39:00.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.125+0000 7f55e1c4b640 1 -- 172.21.15.67:0/1424467733 wait complete. 2024-03-24T00:39:00.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.126+0000 7f55e1c4b640 1 Processor -- start 2024-03-24T00:39:00.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.127+0000 7f55e1c4b640 1 -- start start 2024-03-24T00:39:00.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.127+0000 7f55e1c4b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f55dc0d0ef0 0x7f55dc152670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:00.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.127+0000 7f55e1c4b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f55dc15dec0 0x7f55dc160800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:00.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.127+0000 7f55e1c4b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55dc15e2a0 0x7f55dc160d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:00.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.127+0000 7f55e1c4b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55dc0d2730 con 0x7f55dc15e2a0 2024-03-24T00:39:00.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.127+0000 7f55e1c4b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55dc0d25b0 con 0x7f55dc0d0ef0 2024-03-24T00:39:00.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.127+0000 7f55e1c4b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55dc0d28b0 con 0x7f55dc15dec0 2024-03-24T00:39:00.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.127+0000 7f55dbfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55dc15e2a0 0x7f55dc160d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:00.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.127+0000 7f55dbfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55dc15e2a0 0x7f55dc160d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59724/0 (socket says 172.21.15.67:59724) 2024-03-24T00:39:00.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.127+0000 7f55db7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f55dc0d0ef0 0x7f55dc152670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:00.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.127+0000 7f55dbfff640 1 -- 172.21.15.67:0/130258446 learned_addr learned my addr 172.21.15.67:0/130258446 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:00.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.127+0000 7f55daffd640 1 --2- 172.21.15.67:0/130258446 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f55dc15dec0 0x7f55dc160800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:00.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.128+0000 7f55db7fe640 1 -- 172.21.15.67:0/130258446 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f55dc15dec0 msgr2=0x7f55dc160800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:00.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.128+0000 7f55db7fe640 1 --2- 172.21.15.67:0/130258446 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f55dc15dec0 0x7f55dc160800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.128+0000 7f55db7fe640 1 -- 172.21.15.67:0/130258446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55dc15e2a0 msgr2=0x7f55dc160d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:00.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.128+0000 7f55db7fe640 1 --2- 172.21.15.67:0/130258446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55dc15e2a0 0x7f55dc160d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.128+0000 7f55db7fe640 1 -- 172.21.15.67:0/130258446 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55dc0b3990 con 0x7f55dc0d0ef0 2024-03-24T00:39:00.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.128+0000 7f55dbfff640 1 --2- 172.21.15.67:0/130258446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55dc15e2a0 0x7f55dc160d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:00.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.128+0000 7f55db7fe640 1 --2- 172.21.15.67:0/130258446 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f55dc0d0ef0 0x7f55dc152670 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f55cc00e900 tx=0x7f55cc00edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:00.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.128+0000 7f55c3fff640 1 -- 172.21.15.67:0/130258446 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55cc017070 con 0x7f55dc0d0ef0 2024-03-24T00:39:00.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.128+0000 7f55c3fff640 1 -- 172.21.15.67:0/130258446 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55cc00ca00 con 0x7f55dc0d0ef0 2024-03-24T00:39:00.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.128+0000 7f55c3fff640 1 -- 172.21.15.67:0/130258446 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55cc00cdf0 con 0x7f55dc0d0ef0 2024-03-24T00:39:00.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.128+0000 7f55e1c4b640 1 -- 172.21.15.67:0/130258446 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55dc1f48b0 con 0x7f55dc0d0ef0 2024-03-24T00:39:00.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.128+0000 7f55e1c4b640 1 -- 172.21.15.67:0/130258446 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55dc152bb0 con 0x7f55dc0d0ef0 2024-03-24T00:39:00.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.129+0000 7f55e1c4b640 1 -- 172.21.15.67:0/130258446 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55dc04f400 con 0x7f55dc0d0ef0 2024-03-24T00:39:00.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.129+0000 7f55c3fff640 1 -- 172.21.15.67:0/130258446 <== mon.1 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f55cc01a070 con 0x7f55dc0d0ef0 2024-03-24T00:39:00.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.130+0000 7f55c3fff640 1 --2- 172.21.15.67:0/130258446 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f55c806bbd0 0x7f55c806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:00.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.130+0000 7f55c3fff640 1 -- 172.21.15.67:0/130258446 <== mon.1 v2:172.21.15.72:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f55cc08abb0 con 0x7f55dc0d0ef0 2024-03-24T00:39:00.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.130+0000 7f55daffd640 1 --2- 172.21.15.67:0/130258446 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f55c806bbd0 0x7f55c806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:00.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.130+0000 7f55daffd640 1 --2- 172.21.15.67:0/130258446 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f55c806bbd0 0x7f55c806e090 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f55dc159010 tx=0x7f55d0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:00.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.134+0000 7f55c3fff640 1 -- 172.21.15.67:0/130258446 <== mon.1 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f55cc058d20 con 0x7f55dc0d0ef0 2024-03-24T00:39:00.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.354+0000 7f55e1c4b640 1 -- 172.21.15.67:0/130258446 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f55dc06ce80 con 0x7f55dc0d0ef0 2024-03-24T00:39:00.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.354+0000 7f55c3fff640 1 -- 172.21.15.67:0/130258446 <== mon.1 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+88 (secure 0 0 0) 0x7f55cc05cdc0 con 0x7f55dc0d0ef0 2024-03-24T00:39:00.355 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-24T00:39:00.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.356+0000 7f55e1c4b640 1 -- 172.21.15.67:0/130258446 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f55c806bbd0 msgr2=0x7f55c806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:00.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.356+0000 7f55e1c4b640 1 --2- 172.21.15.67:0/130258446 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f55c806bbd0 0x7f55c806e090 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f55dc159010 tx=0x7f55d0009040 comp rx=0 tx=0).stop 2024-03-24T00:39:00.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.356+0000 7f55e1c4b640 1 -- 172.21.15.67:0/130258446 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f55dc0d0ef0 msgr2=0x7f55dc152670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:00.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.356+0000 7f55e1c4b640 1 --2- 172.21.15.67:0/130258446 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f55dc0d0ef0 0x7f55dc152670 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f55cc00e900 tx=0x7f55cc00edc0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.356+0000 7f55e1c4b640 1 -- 172.21.15.67:0/130258446 shutdown_connections 2024-03-24T00:39:00.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.356+0000 7f55e1c4b640 1 --2- 172.21.15.67:0/130258446 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f55c806bbd0 0x7f55c806e090 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.356+0000 7f55e1c4b640 1 --2- 172.21.15.67:0/130258446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55dc15e2a0 0x7f55dc160d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.356+0000 7f55e1c4b640 1 --2- 172.21.15.67:0/130258446 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f55dc15dec0 0x7f55dc160800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.356+0000 7f55e1c4b640 1 --2- 172.21.15.67:0/130258446 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f55dc0d0ef0 0x7f55dc152670 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.356+0000 7f55e1c4b640 1 -- 172.21.15.67:0/130258446 >> 172.21.15.67:0/130258446 conn(0x7f55dc06c810 msgr2=0x7f55dc15d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:00.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.356+0000 7f55e1c4b640 1 -- 172.21.15.67:0/130258446 shutdown_connections 2024-03-24T00:39:00.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:00.356+0000 7f55e1c4b640 1 -- 172.21.15.67:0/130258446 wait complete. 2024-03-24T00:39:00.367 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-24T00:39:00.367 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T00:39:00.367 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-03-24T00:39:00.367 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-03-24T00:39:00.368 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-24T00:39:00.368 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-24T00:39:00.368 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T00:39:00.368 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T00:39:00.368 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-03-24T00:39:00.368 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:00.368 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-03-24T00:39:00.383 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T00:39:00.383 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T00:39:00.383 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T00:39:00.383 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:39:00.383 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-03-24T00:39:00.383 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:39:00.383 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:39:00.383 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-03-24T00:39:00.383 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-03-24T00:39:00.383 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:39:00.383 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-03-24T00:39:00.383 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-03-24T00:39:00.383 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:39:00.384 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-03-24T00:39:00.384 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T00:39:00.384 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-03-24T00:39:00.384 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-24T00:39:00.384 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 42573sec preferred_lft 42573sec 2024-03-24T00:39:00.384 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-03-24T00:39:00.384 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:39:00.384 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-24T00:39:00.385 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:00.385 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-03-24T00:39:00.385 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-03-24T00:39:00.385 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link add name ceph-brx type bridge 2024-03-24T00:39:00.385 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip addr flush dev ceph-brx 2024-03-24T00:39:00.385 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set ceph-brx up 2024-03-24T00:39:00.385 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-24T00:39:00.385 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-03-24T00:39:00.544 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:00.544 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T00:39:00.570 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-03-24T00:39:00.572 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:00.572 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-03-24T00:39:00.648 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-03-24T00:39:00.648 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T00:39:00.648 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-24T00:39:00.648 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-24T00:39:00.648 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T00:39:00.649 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:00.649 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-03-24T00:39:00.649 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-03-24T00:39:00.649 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-24T00:39:00.649 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-24T00:39:00.649 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-24T00:39:00.649 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-03-24T00:39:00.900 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:00.901 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-03-24T00:39:00.915 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:00.916 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list-id 2024-03-24T00:39:00.969 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:00.970 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-03-24T00:39:00.970 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-03-24T00:39:00.970 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-24T00:39:00.970 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-03-24T00:39:00.970 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-03-24T00:39:01.052 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-03-24T00:39:01.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:01.052 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-03-24T00:39:01.052 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-03-24T00:39:01.052 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-03-24T00:39:01.052 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-24T00:39:01.052 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-03-24T00:39:01.053 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-03-24T00:39:01.053 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-03-24T00:39:01.053 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-03-24T00:39:01.196 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:01.196 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-03-24T00:39:01.196 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-03-24T00:39:01.196 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-03-24T00:39:01.196 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set dev brx.0 master ceph-brx 2024-03-24T00:39:01.196 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-03-24T00:39:01.237 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-03-24T00:39:01.237 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:39:01.237 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-03-24T00:39:01.292 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-03-24T00:39:01.292 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:39:01.292 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-03-24T00:39:01.346 DEBUG:teuthology.orchestra.run.smithi067:> sudo modprobe fuse 2024-03-24T00:39:01.413 DEBUG:teuthology.orchestra.run.smithi067:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T00:39:01.468 INFO:teuthology.orchestra.run.smithi067.stdout:/proc 2024-03-24T00:39:01.469 INFO:teuthology.orchestra.run.smithi067.stdout:/sys 2024-03-24T00:39:01.469 INFO:teuthology.orchestra.run.smithi067.stdout:/dev 2024-03-24T00:39:01.469 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/security 2024-03-24T00:39:01.469 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/shm 2024-03-24T00:39:01.469 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/pts 2024-03-24T00:39:01.469 INFO:teuthology.orchestra.run.smithi067.stdout:/run 2024-03-24T00:39:01.469 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/cgroup 2024-03-24T00:39:01.469 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/pstore 2024-03-24T00:39:01.469 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/bpf 2024-03-24T00:39:01.469 INFO:teuthology.orchestra.run.smithi067.stdout:/ 2024-03-24T00:39:01.469 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/selinux 2024-03-24T00:39:01.469 INFO:teuthology.orchestra.run.smithi067.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T00:39:01.469 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/mqueue 2024-03-24T00:39:01.470 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/hugepages 2024-03-24T00:39:01.470 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/debug 2024-03-24T00:39:01.470 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/tracing 2024-03-24T00:39:01.470 INFO:teuthology.orchestra.run.smithi067.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T00:39:01.470 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/fuse/connections 2024-03-24T00:39:01.470 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/config 2024-03-24T00:39:01.470 INFO:teuthology.orchestra.run.smithi067.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T00:39:01.470 INFO:teuthology.orchestra.run.smithi067.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T00:39:01.470 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T00:39:01.470 INFO:teuthology.orchestra.run.smithi067.stdout:/run/user/1000 2024-03-24T00:39:01.470 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/ceph 2024-03-24T00:39:01.470 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T00:39:01.470 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T00:39:01.470 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T00:39:01.471 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T00:39:01.471 INFO:teuthology.orchestra.run.smithi067.stdout:/run/netns 2024-03-24T00:39:01.471 INFO:teuthology.orchestra.run.smithi067.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-24T00:39:01.471 INFO:teuthology.orchestra.run.smithi067.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-24T00:39:01.471 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:01.471 DEBUG:teuthology.orchestra.run.smithi067:> ls /sys/fs/fuse/connections 2024-03-24T00:39:01.523 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-24T00:39:01.523 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-03-24T00:39:01.566 DEBUG:teuthology.orchestra.run.smithi067:> sudo modprobe fuse 2024-03-24T00:39:01.592 DEBUG:teuthology.orchestra.run.smithi067:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T00:39:01.647 INFO:teuthology.orchestra.run.smithi067.stdout:/proc 2024-03-24T00:39:01.647 INFO:teuthology.orchestra.run.smithi067.stdout:/sys 2024-03-24T00:39:01.648 INFO:teuthology.orchestra.run.smithi067.stdout:/dev 2024-03-24T00:39:01.648 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/security 2024-03-24T00:39:01.648 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/shm 2024-03-24T00:39:01.648 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/pts 2024-03-24T00:39:01.648 INFO:teuthology.orchestra.run.smithi067.stdout:/run 2024-03-24T00:39:01.648 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/cgroup 2024-03-24T00:39:01.648 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/pstore 2024-03-24T00:39:01.648 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/bpf 2024-03-24T00:39:01.648 INFO:teuthology.orchestra.run.smithi067.stdout:/ 2024-03-24T00:39:01.648 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/selinux 2024-03-24T00:39:01.648 INFO:teuthology.orchestra.run.smithi067.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T00:39:01.648 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/mqueue 2024-03-24T00:39:01.648 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/hugepages 2024-03-24T00:39:01.648 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/debug 2024-03-24T00:39:01.648 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/tracing 2024-03-24T00:39:01.648 INFO:teuthology.orchestra.run.smithi067.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T00:39:01.649 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/fuse/connections 2024-03-24T00:39:01.649 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/config 2024-03-24T00:39:01.649 INFO:teuthology.orchestra.run.smithi067.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T00:39:01.649 INFO:teuthology.orchestra.run.smithi067.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T00:39:01.649 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T00:39:01.649 INFO:teuthology.orchestra.run.smithi067.stdout:/run/user/1000 2024-03-24T00:39:01.649 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/ceph 2024-03-24T00:39:01.649 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T00:39:01.649 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T00:39:01.649 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T00:39:01.649 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T00:39:01.649 INFO:teuthology.orchestra.run.smithi067.stdout:/run/netns 2024-03-24T00:39:01.649 INFO:teuthology.orchestra.run.smithi067.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-24T00:39:01.649 INFO:teuthology.orchestra.run.smithi067.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-24T00:39:01.650 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:01.650 DEBUG:teuthology.orchestra.run.smithi067:> ls /sys/fs/fuse/connections 2024-03-24T00:39:01.651 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi067.stderr:2024-03-24T00:39:01.651+0000 7f383843b480 -1 init, newargv = 0x555880fb8be0 newargc=15 2024-03-24T00:39:01.651 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi067.stderr:ceph-fuse[44841]: starting ceph client 2024-03-24T00:39:01.657 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi067.stderr:ceph-fuse[44841]: starting fuse 2024-03-24T00:39:01.666 INFO:teuthology.orchestra.run.smithi067.stdout:41 2024-03-24T00:39:01.666 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-24T00:39:01.666 DEBUG:teuthology.orchestra.run.smithi067:> sudo stdin-killer -- python3 -c ' 2024-03-24T00:39:01.666 DEBUG:teuthology.orchestra.run.smithi067:> import glob 2024-03-24T00:39:01.666 DEBUG:teuthology.orchestra.run.smithi067:> import re 2024-03-24T00:39:01.666 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-03-24T00:39:01.666 DEBUG:teuthology.orchestra.run.smithi067:> import subprocess 2024-03-24T00:39:01.666 DEBUG:teuthology.orchestra.run.smithi067:> 2024-03-24T00:39:01.666 DEBUG:teuthology.orchestra.run.smithi067:> def _find_admin_socket(client_name): 2024-03-24T00:39:01.666 DEBUG:teuthology.orchestra.run.smithi067:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-03-24T00:39:01.666 DEBUG:teuthology.orchestra.run.smithi067:> files = glob.glob(asok_path) 2024-03-24T00:39:01.667 DEBUG:teuthology.orchestra.run.smithi067:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-03-24T00:39:01.667 DEBUG:teuthology.orchestra.run.smithi067:> 2024-03-24T00:39:01.667 DEBUG:teuthology.orchestra.run.smithi067:> # Given a non-glob path, it better be there 2024-03-24T00:39:01.667 DEBUG:teuthology.orchestra.run.smithi067:> if "*" not in asok_path: 2024-03-24T00:39:01.667 DEBUG:teuthology.orchestra.run.smithi067:> assert(len(files) == 1) 2024-03-24T00:39:01.667 DEBUG:teuthology.orchestra.run.smithi067:> return files[0] 2024-03-24T00:39:01.667 DEBUG:teuthology.orchestra.run.smithi067:> 2024-03-24T00:39:01.667 DEBUG:teuthology.orchestra.run.smithi067:> for f in files: 2024-03-24T00:39:01.667 DEBUG:teuthology.orchestra.run.smithi067:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-24T00:39:01.667 DEBUG:teuthology.orchestra.run.smithi067:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-24T00:39:01.667 DEBUG:teuthology.orchestra.run.smithi067:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-24T00:39:01.667 DEBUG:teuthology.orchestra.run.smithi067:> contents = proc_f.read() 2024-03-24T00:39:01.667 DEBUG:teuthology.orchestra.run.smithi067:> if mountpoint in contents: 2024-03-24T00:39:01.667 DEBUG:teuthology.orchestra.run.smithi067:> return f 2024-03-24T00:39:01.668 DEBUG:teuthology.orchestra.run.smithi067:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-24T00:39:01.668 DEBUG:teuthology.orchestra.run.smithi067:> 2024-03-24T00:39:01.668 DEBUG:teuthology.orchestra.run.smithi067:> print(_find_admin_socket("client.admin")) 2024-03-24T00:39:01.668 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-03-24T00:39:01.786 INFO:teuthology.orchestra.run.smithi067.stdout:/var/run/ceph/ceph-client.admin.44841.asok 2024-03-24T00:39:01.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:39:01.796 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.44841.asok 2024-03-24T00:39:01.796 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:01.797 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.44841.asok status 2024-03-24T00:39:01.933 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:39:01.933 INFO:teuthology.orchestra.run.smithi067.stdout: "metadata": { 2024-03-24T00:39:01.933 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_sha1": "3be9a5c9bc793e11e2800a8c0c696e8b46742033", 2024-03-24T00:39:01.933 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-24T00:39:01.933 INFO:teuthology.orchestra.run.smithi067.stdout: "entity_id": "admin", 2024-03-24T00:39:01.933 INFO:teuthology.orchestra.run.smithi067.stdout: "hostname": "smithi067", 2024-03-24T00:39:01.933 INFO:teuthology.orchestra.run.smithi067.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-03-24T00:39:01.933 INFO:teuthology.orchestra.run.smithi067.stdout: "pid": "44841", 2024-03-24T00:39:01.934 INFO:teuthology.orchestra.run.smithi067.stdout: "root": "/" 2024-03-24T00:39:01.934 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:39:01.934 INFO:teuthology.orchestra.run.smithi067.stdout: "dentry_count": 0, 2024-03-24T00:39:01.934 INFO:teuthology.orchestra.run.smithi067.stdout: "dentry_pinned_count": 0, 2024-03-24T00:39:01.934 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 4434, 2024-03-24T00:39:01.934 INFO:teuthology.orchestra.run.smithi067.stdout: "inst": { 2024-03-24T00:39:01.934 INFO:teuthology.orchestra.run.smithi067.stdout: "name": { 2024-03-24T00:39:01.934 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "client", 2024-03-24T00:39:01.934 INFO:teuthology.orchestra.run.smithi067.stdout: "num": 4434 2024-03-24T00:39:01.934 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:39:01.934 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": { 2024-03-24T00:39:01.934 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-03-24T00:39:01.935 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "192.168.0.1:0", 2024-03-24T00:39:01.935 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 906127515 2024-03-24T00:39:01.935 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:39:01.935 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:39:01.935 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": { 2024-03-24T00:39:01.935 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-03-24T00:39:01.935 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "192.168.0.1:0", 2024-03-24T00:39:01.935 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 906127515 2024-03-24T00:39:01.935 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:39:01.935 INFO:teuthology.orchestra.run.smithi067.stdout: "inst_str": "client.4434 192.168.0.1:0/906127515", 2024-03-24T00:39:01.935 INFO:teuthology.orchestra.run.smithi067.stdout: "addr_str": "192.168.0.1:0/906127515", 2024-03-24T00:39:01.935 INFO:teuthology.orchestra.run.smithi067.stdout: "inode_count": 1, 2024-03-24T00:39:01.935 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_epoch": 7, 2024-03-24T00:39:01.935 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_epoch": 25, 2024-03-24T00:39:01.936 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_epoch_barrier": 0, 2024-03-24T00:39:01.936 INFO:teuthology.orchestra.run.smithi067.stdout: "blocklisted": false, 2024-03-24T00:39:01.936 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs" 2024-03-24T00:39:01.936 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:39:01.941 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:01.941 DEBUG:teuthology.orchestra.run.smithi067:> stat --file-system '--printf=%T 2024-03-24T00:39:01.941 DEBUG:teuthology.orchestra.run.smithi067:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-03-24T00:39:01.956 INFO:teuthology.orchestra.run.smithi067.stdout:fuseblk 2024-03-24T00:39:01.956 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-03-24T00:39:01.956 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:01.956 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-03-24T00:39:02.024 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:02.024 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-03-24T00:39:02.091 INFO:teuthology.orchestra.run.smithi067.stdout:# file: . 2024-03-24T00:39:02.091 INFO:teuthology.orchestra.run.smithi067.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-03-24T00:39:02.091 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:39:02.092 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:39:02.092 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:39:02.147 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:39:02.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.287+0000 7f3c02348640 1 Processor -- start 2024-03-24T00:39:02.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.287+0000 7f3c02348640 1 -- start start 2024-03-24T00:39:02.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.287+0000 7f3c02348640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3bfc0d1550 0x7f3bfc06cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:02.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.288+0000 7f3c02348640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3bfc0b2780 0x7f3bfc0b2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:02.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.288+0000 7f3c02348640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bfc0d1930 con 0x7f3bfc099d70 2024-03-24T00:39:02.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.288+0000 7f3c02348640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bfc06d420 con 0x7f3bfc0b2780 2024-03-24T00:39:02.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.288+0000 7f3c02348640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bfc06ca50 con 0x7f3bfc0d1550 2024-03-24T00:39:02.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.288+0000 7f3bfb7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3bfc0b2780 0x7f3bfc0b2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:02.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.288+0000 7f3bfb7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3bfc0b2780 0x7f3bfc0b2c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59734/0 (socket says 172.21.15.67:59734) 2024-03-24T00:39:02.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.288+0000 7f3c008e5640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3bfc099d70 0x7f3bfc0cfd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56878/0 (socket says 172.21.15.67:56878) 2024-03-24T00:39:02.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.288+0000 7f3bfb7fe640 1 -- 172.21.15.67:0/3482771253 learned_addr learned my addr 172.21.15.67:0/3482771253 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:02.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.288+0000 7f3bfbfff640 1 --2- 172.21.15.67:0/3482771253 >> v2:172.21.15.72:3300/0 conn(0x7f3bfc0d1550 0x7f3bfc06cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:02.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.288+0000 7f3bfb7fe640 1 -- 172.21.15.67:0/3482771253 >> v2:172.21.15.72:3300/0 conn(0x7f3bfc0d1550 msgr2=0x7f3bfc06cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:02.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.288+0000 7f3bfb7fe640 1 --2- 172.21.15.67:0/3482771253 >> v2:172.21.15.72:3300/0 conn(0x7f3bfc0d1550 0x7f3bfc06cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:02.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.288+0000 7f3bfb7fe640 1 -- 172.21.15.67:0/3482771253 >> v1:172.21.15.67:6789/0 conn(0x7f3bfc099d70 legacy=0x7f3bfc0cfd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:02.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.288+0000 7f3bfb7fe640 1 -- 172.21.15.67:0/3482771253 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bfc0b1f60 con 0x7f3bfc0b2780 2024-03-24T00:39:02.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.288+0000 7f3bfaffd640 1 -- 172.21.15.67:0/3482771253 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3842661633 0 0) 0x7f3bfc0d1930 con 0x7f3bfc099d70 2024-03-24T00:39:02.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.289+0000 7f3bfb7fe640 1 --2- 172.21.15.67:0/3482771253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bfc0b2780 0x7f3bfc0b2c30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3bec009870 tx=0x7f3bec02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fe72b6cc40d1ff3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:02.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.289+0000 7f3bfaffd640 1 -- 172.21.15.67:0/3482771253 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bec035020 con 0x7f3bfc0b2780 2024-03-24T00:39:02.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.289+0000 7f3bfaffd640 1 -- 172.21.15.67:0/3482771253 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bec031870 con 0x7f3bfc0b2780 2024-03-24T00:39:02.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.289+0000 7f3bfaffd640 1 -- 172.21.15.67:0/3482771253 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bec031c40 con 0x7f3bfc0b2780 2024-03-24T00:39:02.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.289+0000 7f3c02348640 1 -- 172.21.15.67:0/3482771253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bfc0b2780 msgr2=0x7f3bfc0b2c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:02.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.289+0000 7f3c02348640 1 --2- 172.21.15.67:0/3482771253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bfc0b2780 0x7f3bfc0b2c30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3bec009870 tx=0x7f3bec02b910 comp rx=0 tx=0).stop 2024-03-24T00:39:02.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.289+0000 7f3c02348640 1 -- 172.21.15.67:0/3482771253 shutdown_connections 2024-03-24T00:39:02.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.289+0000 7f3c02348640 1 --2- 172.21.15.67:0/3482771253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bfc0b2780 0x7f3bfc0b2c30 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:02.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.289+0000 7f3c02348640 1 --2- 172.21.15.67:0/3482771253 >> v2:172.21.15.72:3300/0 conn(0x7f3bfc0d1550 0x7f3bfc06cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:02.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.289+0000 7f3c02348640 1 -- 172.21.15.67:0/3482771253 >> 172.21.15.67:0/3482771253 conn(0x7f3bfc0b36e0 msgr2=0x7f3bfc0b3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:02.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.289+0000 7f3c02348640 1 -- 172.21.15.67:0/3482771253 shutdown_connections 2024-03-24T00:39:02.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.289+0000 7f3c02348640 1 -- 172.21.15.67:0/3482771253 wait complete. 2024-03-24T00:39:02.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.291+0000 7f3c02348640 1 Processor -- start 2024-03-24T00:39:02.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.291+0000 7f3c02348640 1 -- start start 2024-03-24T00:39:02.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.291+0000 7f3c02348640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bfc099d70 0x7f3bfc0cba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:02.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.291+0000 7f3c02348640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bfc0b2780 0x7f3bfc0b4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:02.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.291+0000 7f3c02348640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bfc0d1550 0x7f3bfc150ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:02.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.291+0000 7f3c02348640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bfc06c400 con 0x7f3bfc0b2780 2024-03-24T00:39:02.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.291+0000 7f3c02348640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bfc0b2520 con 0x7f3bfc0d1550 2024-03-24T00:39:02.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.291+0000 7f3c02348640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bfc06c6e0 con 0x7f3bfc099d70 2024-03-24T00:39:02.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.292+0000 7f3bfb7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bfc0b2780 0x7f3bfc0b4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:02.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.292+0000 7f3bfb7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bfc0b2780 0x7f3bfc0b4810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59742/0 (socket says 172.21.15.67:59742) 2024-03-24T00:39:02.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.292+0000 7f3bfb7fe640 1 -- 172.21.15.67:0/1864574425 learned_addr learned my addr 172.21.15.67:0/1864574425 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:02.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.292+0000 7f3bfbfff640 1 --2- 172.21.15.67:0/1864574425 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bfc099d70 0x7f3bfc0cba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:02.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.292+0000 7f3c008e5640 1 --2- 172.21.15.67:0/1864574425 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bfc0d1550 0x7f3bfc150ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:02.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.292+0000 7f3bfb7fe640 1 -- 172.21.15.67:0/1864574425 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bfc099d70 msgr2=0x7f3bfc0cba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:02.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.292+0000 7f3bfb7fe640 1 --2- 172.21.15.67:0/1864574425 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bfc099d70 0x7f3bfc0cba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:02.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.292+0000 7f3bfb7fe640 1 -- 172.21.15.67:0/1864574425 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bfc0d1550 msgr2=0x7f3bfc150ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:02.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.292+0000 7f3bfb7fe640 1 --2- 172.21.15.67:0/1864574425 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bfc0d1550 0x7f3bfc150ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:02.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.292+0000 7f3bfb7fe640 1 -- 172.21.15.67:0/1864574425 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bfc0ca9a0 con 0x7f3bfc0b2780 2024-03-24T00:39:02.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.292+0000 7f3bfb7fe640 1 --2- 172.21.15.67:0/1864574425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bfc0b2780 0x7f3bfc0b4810 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3bec000c00 tx=0x7f3bec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:02.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.292+0000 7f3bf8ff9640 1 -- 172.21.15.67:0/1864574425 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bec040070 con 0x7f3bfc0b2780 2024-03-24T00:39:02.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.292+0000 7f3bf8ff9640 1 -- 172.21.15.67:0/1864574425 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bec030d00 con 0x7f3bfc0b2780 2024-03-24T00:39:02.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.292+0000 7f3c02348640 1 -- 172.21.15.67:0/1864574425 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bfc0cac80 con 0x7f3bfc0b2780 2024-03-24T00:39:02.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.292+0000 7f3bf8ff9640 1 -- 172.21.15.67:0/1864574425 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bec035040 con 0x7f3bfc0b2780 2024-03-24T00:39:02.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.292+0000 7f3c02348640 1 -- 172.21.15.67:0/1864574425 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bfc0cb240 con 0x7f3bfc0b2780 2024-03-24T00:39:02.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.293+0000 7f3c02348640 1 -- 172.21.15.67:0/1864574425 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bfc04f400 con 0x7f3bfc0b2780 2024-03-24T00:39:02.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.294+0000 7f3bf8ff9640 1 -- 172.21.15.67:0/1864574425 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3bec04b060 con 0x7f3bfc0b2780 2024-03-24T00:39:02.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.294+0000 7f3bf8ff9640 1 --2- 172.21.15.67:0/1864574425 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3bdc06bc00 0x7f3bdc06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:02.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.294+0000 7f3bf8ff9640 1 -- 172.21.15.67:0/1864574425 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f3bec0ac9c0 con 0x7f3bfc0b2780 2024-03-24T00:39:02.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.294+0000 7f3bfbfff640 1 --2- 172.21.15.67:0/1864574425 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3bdc06bc00 0x7f3bdc06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:02.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.295+0000 7f3bfbfff640 1 --2- 172.21.15.67:0/1864574425 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3bdc06bc00 0x7f3bdc06e0c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3be4009b40 tx=0x7f3be4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:02.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.299+0000 7f3bf8ff9640 1 -- 172.21.15.67:0/1864574425 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3bec0aee20 con 0x7f3bfc0b2780 2024-03-24T00:39:02.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.482+0000 7f3c02348640 1 -- 172.21.15.67:0/1864574425 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3bfc0b1310 con 0x7f3bfc0b2780 2024-03-24T00:39:02.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.483+0000 7f3bf8ff9640 1 -- 172.21.15.67:0/1864574425 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18454 (secure 0 0 0) 0x7f3bec07aa90 con 0x7f3bfc0b2780 2024-03-24T00:39:02.484 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:39:02.484 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":25,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:38:58.797179+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:38:50.077085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:38:51.095454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T00:38:54.807117+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:39:02.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.485+0000 7f3c02348640 1 -- 172.21.15.67:0/1864574425 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3bdc06bc00 msgr2=0x7f3bdc06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:02.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.485+0000 7f3c02348640 1 --2- 172.21.15.67:0/1864574425 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3bdc06bc00 0x7f3bdc06e0c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3be4009b40 tx=0x7f3be4009190 comp rx=0 tx=0).stop 2024-03-24T00:39:02.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.485+0000 7f3c02348640 1 -- 172.21.15.67:0/1864574425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bfc0b2780 msgr2=0x7f3bfc0b4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:02.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.485+0000 7f3c02348640 1 --2- 172.21.15.67:0/1864574425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bfc0b2780 0x7f3bfc0b4810 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3bec000c00 tx=0x7f3bec0046c0 comp rx=0 tx=0).stop 2024-03-24T00:39:02.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.485+0000 7f3c02348640 1 -- 172.21.15.67:0/1864574425 shutdown_connections 2024-03-24T00:39:02.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.485+0000 7f3c02348640 1 --2- 172.21.15.67:0/1864574425 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3bdc06bc00 0x7f3bdc06e0c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:02.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.485+0000 7f3c02348640 1 --2- 172.21.15.67:0/1864574425 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bfc0d1550 0x7f3bfc150ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:02.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.485+0000 7f3c02348640 1 --2- 172.21.15.67:0/1864574425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bfc0b2780 0x7f3bfc0b4810 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:02.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.485+0000 7f3c02348640 1 --2- 172.21.15.67:0/1864574425 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bfc099d70 0x7f3bfc0cba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:02.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.485+0000 7f3c02348640 1 -- 172.21.15.67:0/1864574425 >> 172.21.15.67:0/1864574425 conn(0x7f3bfc0b36e0 msgr2=0x7f3bfc0b3ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:02.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.485+0000 7f3c02348640 1 -- 172.21.15.67:0/1864574425 shutdown_connections 2024-03-24T00:39:02.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.485+0000 7f3c02348640 1 -- 172.21.15.67:0/1864574425 wait complete. 2024-03-24T00:39:02.496 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:38:39.364950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:38:50.077085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:38:51.095454+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T00:38:54.807117+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T00:39:02.497 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:39:02.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.601+0000 7fa180ac0640 1 Processor -- start 2024-03-24T00:39:02.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.601+0000 7fa180ac0640 1 -- start start 2024-03-24T00:39:02.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.602+0000 7fa180ac0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa17c099d00 0x7fa17c09a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:02.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.602+0000 7fa180ac0640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa17c0d1530 0x7fa17c15d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:02.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.602+0000 7fa180ac0640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa17c09a620 con 0x7fa17c0b49b0 2024-03-24T00:39:02.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.602+0000 7fa180ac0640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa17c0cc1b0 con 0x7fa17c099d00 2024-03-24T00:39:02.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.602+0000 7fa180ac0640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa17c0cc400 con 0x7fa17c0d1530 2024-03-24T00:39:02.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.602+0000 7fa17a59c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa17c099d00 0x7fa17c09a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:02.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.602+0000 7fa179d9b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa17c0b49b0 0x7fa17c0d0e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56892/0 (socket says 172.21.15.67:56892) 2024-03-24T00:39:02.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.602+0000 7fa179d9b640 1 -- 172.21.15.67:0/3967050424 learned_addr learned my addr 172.21.15.67:0/3967050424 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:02.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.602+0000 7fa17a59c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa17c099d00 0x7fa17c09a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59754/0 (socket says 172.21.15.67:59754) 2024-03-24T00:39:02.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.602+0000 7fa17ad9d640 1 --2- 172.21.15.67:0/3967050424 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa17c0d1530 0x7fa17c15d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:02.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.602+0000 7fa17ad9d640 1 -- 172.21.15.67:0/3967050424 >> v2:172.21.15.67:3300/0 conn(0x7fa17c099d00 msgr2=0x7fa17c09a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:02.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.602+0000 7fa17ad9d640 1 --2- 172.21.15.67:0/3967050424 >> v2:172.21.15.67:3300/0 conn(0x7fa17c099d00 0x7fa17c09a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:02.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.602+0000 7fa17ad9d640 1 -- 172.21.15.67:0/3967050424 >> v1:172.21.15.67:6789/0 conn(0x7fa17c0b49b0 legacy=0x7fa17c0d0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:02.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.602+0000 7fa17959a640 1 -- 172.21.15.67:0/3967050424 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3366929459 0 0) 0x7fa17c09a620 con 0x7fa17c0b49b0 2024-03-24T00:39:02.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.602+0000 7fa17ad9d640 1 -- 172.21.15.67:0/3967050424 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa17c0cc6e0 con 0x7fa17c0d1530 2024-03-24T00:39:02.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.602+0000 7fa17a59c640 1 --2- 172.21.15.67:0/3967050424 >> v2:172.21.15.67:3300/0 conn(0x7fa17c099d00 0x7fa17c09a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:02.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.603+0000 7fa17ad9d640 1 --2- 172.21.15.67:0/3967050424 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa17c0d1530 0x7fa17c15d7d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa1740097a0 tx=0x7fa17402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=acab3035832a0e45 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:02.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.603+0000 7fa17959a640 1 -- 172.21.15.67:0/3967050424 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa174035020 con 0x7fa17c0d1530 2024-03-24T00:39:02.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.603+0000 7fa17959a640 1 -- 172.21.15.67:0/3967050424 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa174031810 con 0x7fa17c0d1530 2024-03-24T00:39:02.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.603+0000 7fa17959a640 1 -- 172.21.15.67:0/3967050424 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa174031be0 con 0x7fa17c0d1530 2024-03-24T00:39:02.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.604+0000 7fa180ac0640 1 -- 172.21.15.67:0/3967050424 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa17c0d1530 msgr2=0x7fa17c15d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:02.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.604+0000 7fa180ac0640 1 --2- 172.21.15.67:0/3967050424 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa17c0d1530 0x7fa17c15d7d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa1740097a0 tx=0x7fa17402b8b0 comp rx=0 tx=0).stop 2024-03-24T00:39:02.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.604+0000 7fa180ac0640 1 -- 172.21.15.67:0/3967050424 shutdown_connections 2024-03-24T00:39:02.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.604+0000 7fa180ac0640 1 --2- 172.21.15.67:0/3967050424 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa17c0d1530 0x7fa17c15d7d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:02.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.604+0000 7fa180ac0640 1 --2- 172.21.15.67:0/3967050424 >> v2:172.21.15.67:3300/0 conn(0x7fa17c099d00 0x7fa17c09a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:02.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.604+0000 7fa180ac0640 1 -- 172.21.15.67:0/3967050424 >> 172.21.15.67:0/3967050424 conn(0x7fa17c0b3650 msgr2=0x7fa17c0b3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:02.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.604+0000 7fa180ac0640 1 -- 172.21.15.67:0/3967050424 shutdown_connections 2024-03-24T00:39:02.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.604+0000 7fa180ac0640 1 -- 172.21.15.67:0/3967050424 wait complete. 2024-03-24T00:39:02.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.605+0000 7fa180ac0640 1 Processor -- start 2024-03-24T00:39:02.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.605+0000 7fa180ac0640 1 -- start start 2024-03-24T00:39:02.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.606+0000 7fa180ac0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa17c099d00 0x7fa17c14f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:02.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.606+0000 7fa180ac0640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa17c0b49b0 0x7fa17c14bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:02.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.606+0000 7fa180ac0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa17c0c7570 0x7fa17c0b5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:02.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.606+0000 7fa180ac0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa17c0d01c0 con 0x7fa17c099d00 2024-03-24T00:39:02.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.606+0000 7fa180ac0640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa17c0ccca0 con 0x7fa17c0c7570 2024-03-24T00:39:02.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.606+0000 7fa180ac0640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa17c0b32a0 con 0x7fa17c0b49b0 2024-03-24T00:39:02.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.606+0000 7fa17a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa17c099d00 0x7fa17c14f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:02.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.606+0000 7fa17a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa17c099d00 0x7fa17c14f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59762/0 (socket says 172.21.15.67:59762) 2024-03-24T00:39:02.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.606+0000 7fa17a59c640 1 -- 172.21.15.67:0/1601208053 learned_addr learned my addr 172.21.15.67:0/1601208053 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:02.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.606+0000 7fa179d9b640 1 --2- 172.21.15.67:0/1601208053 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa17c0b49b0 0x7fa17c14bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:02.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.606+0000 7fa17ad9d640 1 --2- 172.21.15.67:0/1601208053 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa17c0c7570 0x7fa17c0b5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:02.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.606+0000 7fa17a59c640 1 -- 172.21.15.67:0/1601208053 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa17c0b49b0 msgr2=0x7fa17c14bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:02.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.606+0000 7fa17a59c640 1 --2- 172.21.15.67:0/1601208053 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa17c0b49b0 0x7fa17c14bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:02.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.606+0000 7fa17a59c640 1 -- 172.21.15.67:0/1601208053 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa17c0c7570 msgr2=0x7fa17c0b5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:02.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.606+0000 7fa17a59c640 1 --2- 172.21.15.67:0/1601208053 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa17c0c7570 0x7fa17c0b5ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:02.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.606+0000 7fa17a59c640 1 -- 172.21.15.67:0/1601208053 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa17c0c7a50 con 0x7fa17c099d00 2024-03-24T00:39:02.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.607+0000 7fa17a59c640 1 --2- 172.21.15.67:0/1601208053 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa17c099d00 0x7fa17c14f050 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa16c00efc0 tx=0x7fa16c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:02.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.607+0000 7fa172ffd640 1 -- 172.21.15.67:0/1601208053 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16c018020 con 0x7fa17c099d00 2024-03-24T00:39:02.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.607+0000 7fa172ffd640 1 -- 172.21.15.67:0/1601208053 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa16c014680 con 0x7fa17c099d00 2024-03-24T00:39:02.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.607+0000 7fa172ffd640 1 -- 172.21.15.67:0/1601208053 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16c014a50 con 0x7fa17c099d00 2024-03-24T00:39:02.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.607+0000 7fa180ac0640 1 -- 172.21.15.67:0/1601208053 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa17c0c7d30 con 0x7fa17c099d00 2024-03-24T00:39:02.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.607+0000 7fa180ac0640 1 -- 172.21.15.67:0/1601208053 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa17c0c82f0 con 0x7fa17c099d00 2024-03-24T00:39:02.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.608+0000 7fa180ac0640 1 -- 172.21.15.67:0/1601208053 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa17c04f400 con 0x7fa17c099d00 2024-03-24T00:39:02.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.608+0000 7fa172ffd640 1 -- 172.21.15.67:0/1601208053 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa16c005d10 con 0x7fa17c099d00 2024-03-24T00:39:02.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.609+0000 7fa172ffd640 1 --2- 172.21.15.67:0/1601208053 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa15c06bbd0 0x7fa15c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:02.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.609+0000 7fa172ffd640 1 -- 172.21.15.67:0/1601208053 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fa16c08bb70 con 0x7fa17c099d00 2024-03-24T00:39:02.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.609+0000 7fa179d9b640 1 --2- 172.21.15.67:0/1601208053 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa15c06bbd0 0x7fa15c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:02.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.609+0000 7fa179d9b640 1 --2- 172.21.15.67:0/1601208053 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa15c06bbd0 0x7fa15c06e090 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa17c15b960 tx=0x7fa164007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:02.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.613+0000 7fa172ffd640 1 -- 172.21.15.67:0/1601208053 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa16c022030 con 0x7fa17c099d00 2024-03-24T00:39:02.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.793+0000 7fa180ac0640 1 -- 172.21.15.67:0/1601208053 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa17c070c10 con 0x7fa17c099d00 2024-03-24T00:39:02.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.793+0000 7fa172ffd640 1 -- 172.21.15.67:0/1601208053 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa16c010040 con 0x7fa17c099d00 2024-03-24T00:39:02.794 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-24T00:39:02.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.795+0000 7fa180ac0640 1 -- 172.21.15.67:0/1601208053 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa15c06bbd0 msgr2=0x7fa15c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:02.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.795+0000 7fa180ac0640 1 --2- 172.21.15.67:0/1601208053 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa15c06bbd0 0x7fa15c06e090 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa17c15b960 tx=0x7fa164007d50 comp rx=0 tx=0).stop 2024-03-24T00:39:02.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.795+0000 7fa180ac0640 1 -- 172.21.15.67:0/1601208053 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa17c099d00 msgr2=0x7fa17c14f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:02.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.795+0000 7fa180ac0640 1 --2- 172.21.15.67:0/1601208053 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa17c099d00 0x7fa17c14f050 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa16c00efc0 tx=0x7fa16c00c490 comp rx=0 tx=0).stop 2024-03-24T00:39:02.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.795+0000 7fa180ac0640 1 -- 172.21.15.67:0/1601208053 shutdown_connections 2024-03-24T00:39:02.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.795+0000 7fa180ac0640 1 --2- 172.21.15.67:0/1601208053 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa15c06bbd0 0x7fa15c06e090 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:02.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.795+0000 7fa180ac0640 1 --2- 172.21.15.67:0/1601208053 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa17c0c7570 0x7fa17c0b5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:02.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.795+0000 7fa180ac0640 1 --2- 172.21.15.67:0/1601208053 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa17c0b49b0 0x7fa17c14bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:02.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.795+0000 7fa180ac0640 1 --2- 172.21.15.67:0/1601208053 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa17c099d00 0x7fa17c14f050 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:02.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.795+0000 7fa180ac0640 1 -- 172.21.15.67:0/1601208053 >> 172.21.15.67:0/1601208053 conn(0x7fa17c0b3650 msgr2=0x7fa17c0702d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:02.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.795+0000 7fa180ac0640 1 -- 172.21.15.67:0/1601208053 shutdown_connections 2024-03-24T00:39:02.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.795+0000 7fa180ac0640 1 -- 172.21.15.67:0/1601208053 wait complete. 2024-03-24T00:39:02.806 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:39:02.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.910+0000 7f46c9c36640 1 Processor -- start 2024-03-24T00:39:02.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.910+0000 7f46c9c36640 1 -- start start 2024-03-24T00:39:02.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.911+0000 7f46c9c36640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f46c415de50 0x7f46c40b96d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:02.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.911+0000 7f46c9c36640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f46c40cf120 0x7f46c40b6e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:02.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.911+0000 7f46c9c36640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46c40d23d0 con 0x7f46c415e230 2024-03-24T00:39:02.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.911+0000 7f46c9c36640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46c40b9c10 con 0x7f46c415de50 2024-03-24T00:39:02.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.911+0000 7f46c9c36640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46c40d2180 con 0x7f46c40cf120 2024-03-24T00:39:02.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.911+0000 7f46c37fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f46c415de50 0x7f46c40b96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:02.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.911+0000 7f46c2ffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f46c415e230 0x7f46c40cea10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56906/0 (socket says 172.21.15.67:56906) 2024-03-24T00:39:02.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.911+0000 7f46c2ffd640 1 -- 172.21.15.67:0/3966238803 learned_addr learned my addr 172.21.15.67:0/3966238803 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:02.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.911+0000 7f46c3fff640 1 --2- 172.21.15.67:0/3966238803 >> v2:172.21.15.72:3300/0 conn(0x7f46c40cf120 0x7f46c40b6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:02.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.911+0000 7f46c27fc640 1 -- 172.21.15.67:0/3966238803 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3740869888 0 0) 0x7f46c40d23d0 con 0x7f46c415e230 2024-03-24T00:39:02.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.912+0000 7f46c27fc640 1 -- 172.21.15.67:0/3966238803 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46ac003520 con 0x7f46c415e230 2024-03-24T00:39:02.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.912+0000 7f46c3fff640 1 -- 172.21.15.67:0/3966238803 >> v2:172.21.15.67:3300/0 conn(0x7f46c415de50 msgr2=0x7f46c40b96d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:02.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.912+0000 7f46c3fff640 1 --2- 172.21.15.67:0/3966238803 >> v2:172.21.15.67:3300/0 conn(0x7f46c415de50 0x7f46c40b96d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:02.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.912+0000 7f46c3fff640 1 -- 172.21.15.67:0/3966238803 >> v1:172.21.15.67:6789/0 conn(0x7f46c415e230 legacy=0x7f46c40cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:02.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.912+0000 7f46c3fff640 1 -- 172.21.15.67:0/3966238803 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46c40d26b0 con 0x7f46c40cf120 2024-03-24T00:39:02.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.912+0000 7f46c37fe640 1 --2- 172.21.15.67:0/3966238803 >> v2:172.21.15.67:3300/0 conn(0x7f46c415de50 0x7f46c40b96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:02.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.912+0000 7f46c3fff640 1 --2- 172.21.15.67:0/3966238803 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f46c40cf120 0x7f46c40b6e50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f46c40ce280 tx=0x7f46bc02bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=422fb7fc8cf2aefa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:02.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.912+0000 7f46c27fc640 1 -- 172.21.15.67:0/3966238803 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46bc039070 con 0x7f46c40cf120 2024-03-24T00:39:02.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.913+0000 7f46c27fc640 1 -- 172.21.15.67:0/3966238803 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46bc02cd00 con 0x7f46c40cf120 2024-03-24T00:39:02.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.913+0000 7f46c27fc640 1 -- 172.21.15.67:0/3966238803 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46bc034660 con 0x7f46c40cf120 2024-03-24T00:39:02.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.913+0000 7f46c9c36640 1 -- 172.21.15.67:0/3966238803 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f46c40cf120 msgr2=0x7f46c40b6e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:02.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.913+0000 7f46c9c36640 1 --2- 172.21.15.67:0/3966238803 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f46c40cf120 0x7f46c40b6e50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f46c40ce280 tx=0x7f46bc02bf50 comp rx=0 tx=0).stop 2024-03-24T00:39:02.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.913+0000 7f46c9c36640 1 -- 172.21.15.67:0/3966238803 shutdown_connections 2024-03-24T00:39:02.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.913+0000 7f46c9c36640 1 --2- 172.21.15.67:0/3966238803 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f46c40cf120 0x7f46c40b6e50 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:02.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.913+0000 7f46c9c36640 1 --2- 172.21.15.67:0/3966238803 >> v2:172.21.15.67:3300/0 conn(0x7f46c415de50 0x7f46c40b96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:02.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.913+0000 7f46c9c36640 1 -- 172.21.15.67:0/3966238803 >> 172.21.15.67:0/3966238803 conn(0x7f46c4072b60 msgr2=0x7f46c4072f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:02.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.913+0000 7f46c9c36640 1 -- 172.21.15.67:0/3966238803 shutdown_connections 2024-03-24T00:39:02.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.913+0000 7f46c9c36640 1 -- 172.21.15.67:0/3966238803 wait complete. 2024-03-24T00:39:02.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.914+0000 7f46c9c36640 1 Processor -- start 2024-03-24T00:39:02.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.914+0000 7f46c9c36640 1 -- start start 2024-03-24T00:39:02.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.915+0000 7f46c9c36640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f46c40cf120 0x7f46c41584d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:02.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.915+0000 7f46c9c36640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f46c415de50 0x7f46c4158b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:02.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.915+0000 7f46c9c36640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46c415e230 0x7f46c415a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:02.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.915+0000 7f46c9c36640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c40d37b0 con 0x7f46c415e230 2024-03-24T00:39:02.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.915+0000 7f46c9c36640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c40d2cb0 con 0x7f46c40cf120 2024-03-24T00:39:02.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.915+0000 7f46c9c36640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46c40d3930 con 0x7f46c415de50 2024-03-24T00:39:02.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.915+0000 7f46c3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46c415e230 0x7f46c415a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:02.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.915+0000 7f46c3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46c415e230 0x7f46c415a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59774/0 (socket says 172.21.15.67:59774) 2024-03-24T00:39:02.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.915+0000 7f46c3fff640 1 -- 172.21.15.67:0/1333236525 learned_addr learned my addr 172.21.15.67:0/1333236525 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:02.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.915+0000 7f46c2ffd640 1 --2- 172.21.15.67:0/1333236525 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f46c415de50 0x7f46c4158b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:02.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.915+0000 7f46c37fe640 1 --2- 172.21.15.67:0/1333236525 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f46c40cf120 0x7f46c41584d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:02.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.915+0000 7f46c3fff640 1 -- 172.21.15.67:0/1333236525 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f46c415de50 msgr2=0x7f46c4158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:02.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.915+0000 7f46c3fff640 1 --2- 172.21.15.67:0/1333236525 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f46c415de50 0x7f46c4158b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:02.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.915+0000 7f46c3fff640 1 -- 172.21.15.67:0/1333236525 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f46c40cf120 msgr2=0x7f46c41584d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:02.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.915+0000 7f46c3fff640 1 --2- 172.21.15.67:0/1333236525 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f46c40cf120 0x7f46c41584d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:02.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.916+0000 7f46c3fff640 1 -- 172.21.15.67:0/1333236525 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46c41590a0 con 0x7f46c415e230 2024-03-24T00:39:02.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.916+0000 7f46c3fff640 1 --2- 172.21.15.67:0/1333236525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46c415e230 0x7f46c415a990 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f46bc02c560 tx=0x7f46bc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:02.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.916+0000 7f46abfff640 1 -- 172.21.15.67:0/1333236525 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46bc039050 con 0x7f46c415e230 2024-03-24T00:39:02.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.916+0000 7f46abfff640 1 -- 172.21.15.67:0/1333236525 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46bc045070 con 0x7f46c415e230 2024-03-24T00:39:02.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.916+0000 7f46c9c36640 1 -- 172.21.15.67:0/1333236525 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46c41f3a40 con 0x7f46c415e230 2024-03-24T00:39:02.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.916+0000 7f46abfff640 1 -- 172.21.15.67:0/1333236525 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46bc009450 con 0x7f46c415e230 2024-03-24T00:39:02.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.916+0000 7f46c9c36640 1 -- 172.21.15.67:0/1333236525 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46c41f4000 con 0x7f46c415e230 2024-03-24T00:39:02.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.917+0000 7f46c9c36640 1 -- 172.21.15.67:0/1333236525 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46c404f400 con 0x7f46c415e230 2024-03-24T00:39:02.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.917+0000 7f46abfff640 1 -- 172.21.15.67:0/1333236525 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f46bc03e850 con 0x7f46c415e230 2024-03-24T00:39:02.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.918+0000 7f46abfff640 1 --2- 172.21.15.67:0/1333236525 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f46b006bbd0 0x7f46b006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:02.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.918+0000 7f46abfff640 1 -- 172.21.15.67:0/1333236525 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f46bc0ada40 con 0x7f46c415e230 2024-03-24T00:39:02.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.918+0000 7f46c37fe640 1 --2- 172.21.15.67:0/1333236525 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f46b006bbd0 0x7f46b006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:02.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.918+0000 7f46c37fe640 1 --2- 172.21.15.67:0/1333236525 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f46b006bbd0 0x7f46b006e090 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f46b4009560 tx=0x7f46b4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:02.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:02.922+0000 7f46abfff640 1 -- 172.21.15.67:0/1333236525 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f46bc07bcc0 con 0x7f46c415e230 2024-03-24T00:39:03.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.102+0000 7f46c9c36640 1 -- 172.21.15.67:0/1333236525 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f46c406b800 con 0x7f46c415e230 2024-03-24T00:39:03.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.102+0000 7f46abfff640 1 -- 172.21.15.67:0/1333236525 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7f46bc07fd60 con 0x7f46c415e230 2024-03-24T00:39:03.103 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-24T00:39:03.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.104+0000 7f46c9c36640 1 -- 172.21.15.67:0/1333236525 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f46b006bbd0 msgr2=0x7f46b006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:03.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.104+0000 7f46c9c36640 1 --2- 172.21.15.67:0/1333236525 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f46b006bbd0 0x7f46b006e090 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f46b4009560 tx=0x7f46b4009190 comp rx=0 tx=0).stop 2024-03-24T00:39:03.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.104+0000 7f46c9c36640 1 -- 172.21.15.67:0/1333236525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46c415e230 msgr2=0x7f46c415a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:03.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.104+0000 7f46c9c36640 1 --2- 172.21.15.67:0/1333236525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46c415e230 0x7f46c415a990 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f46bc02c560 tx=0x7f46bc004a40 comp rx=0 tx=0).stop 2024-03-24T00:39:03.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.104+0000 7f46c9c36640 1 -- 172.21.15.67:0/1333236525 shutdown_connections 2024-03-24T00:39:03.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.104+0000 7f46c9c36640 1 --2- 172.21.15.67:0/1333236525 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f46b006bbd0 0x7f46b006e090 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.104+0000 7f46c9c36640 1 --2- 172.21.15.67:0/1333236525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46c415e230 0x7f46c415a990 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.104+0000 7f46c9c36640 1 --2- 172.21.15.67:0/1333236525 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f46c415de50 0x7f46c4158b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.104+0000 7f46c9c36640 1 --2- 172.21.15.67:0/1333236525 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f46c40cf120 0x7f46c41584d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.104+0000 7f46c9c36640 1 -- 172.21.15.67:0/1333236525 >> 172.21.15.67:0/1333236525 conn(0x7f46c4072b60 msgr2=0x7f46c415e640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:03.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.104+0000 7f46c9c36640 1 -- 172.21.15.67:0/1333236525 shutdown_connections 2024-03-24T00:39:03.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.104+0000 7f46c9c36640 1 -- 172.21.15.67:0/1333236525 wait complete. 2024-03-24T00:39:03.115 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:39:03.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.219+0000 7f557d2e8640 1 Processor -- start 2024-03-24T00:39:03.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.219+0000 7f557d2e8640 1 -- start start 2024-03-24T00:39:03.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.219+0000 7f557d2e8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f557815e3a0 0x7f55780b2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.219+0000 7f557d2e8640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f55781626d0 0x7f55780ccaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.219+0000 7f557d2e8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f557815e860 con 0x7f55780cd030 2024-03-24T00:39:03.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.219+0000 7f557d2e8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55780afb80 con 0x7f557815e3a0 2024-03-24T00:39:03.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.219+0000 7f557d2e8640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f557815f470 con 0x7f55781626d0 2024-03-24T00:39:03.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.219+0000 7f5576ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f557815e3a0 0x7f55780b2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.220+0000 7f55777fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f55780cd030 0x7f55780b4f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56908/0 (socket says 172.21.15.67:56908) 2024-03-24T00:39:03.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.220+0000 7f5576ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f557815e3a0 0x7f55780b2890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59776/0 (socket says 172.21.15.67:59776) 2024-03-24T00:39:03.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.220+0000 7f55777fe640 1 -- 172.21.15.67:0/3260455757 learned_addr learned my addr 172.21.15.67:0/3260455757 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:03.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.220+0000 7f55767fc640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f55781626d0 0x7f55780ccaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.220+0000 7f5576ffd640 1 -- 172.21.15.67:0/3260455757 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f55781626d0 msgr2=0x7f55780ccaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:03.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.220+0000 7f5576ffd640 1 --2- 172.21.15.67:0/3260455757 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f55781626d0 0x7f55780ccaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.220+0000 7f5576ffd640 1 -- 172.21.15.67:0/3260455757 >> v1:172.21.15.67:6789/0 conn(0x7f55780cd030 legacy=0x7f55780b4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:03.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.220+0000 7f5576ffd640 1 -- 172.21.15.67:0/3260455757 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f557815eb40 con 0x7f557815e3a0 2024-03-24T00:39:03.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.220+0000 7f5575ffb640 1 -- 172.21.15.67:0/3260455757 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2873714497 0 0) 0x7f557815e860 con 0x7f55780cd030 2024-03-24T00:39:03.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.220+0000 7f55767fc640 1 --2- 172.21.15.67:0/3260455757 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f55781626d0 0x7f55780ccaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:03.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.221+0000 7f5576ffd640 1 --2- 172.21.15.67:0/3260455757 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f557815e3a0 0x7f55780b2890 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f556c0097a0 tx=0x7f556c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd79c94237345655 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:03.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.221+0000 7f5575ffb640 1 -- 172.21.15.67:0/3260455757 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f556c035020 con 0x7f557815e3a0 2024-03-24T00:39:03.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.221+0000 7f5575ffb640 1 -- 172.21.15.67:0/3260455757 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f556c031830 con 0x7f557815e3a0 2024-03-24T00:39:03.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.221+0000 7f5575ffb640 1 -- 172.21.15.67:0/3260455757 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f556c031c00 con 0x7f557815e3a0 2024-03-24T00:39:03.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.221+0000 7f557d2e8640 1 -- 172.21.15.67:0/3260455757 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f557815e3a0 msgr2=0x7f55780b2890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:03.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.221+0000 7f557d2e8640 1 --2- 172.21.15.67:0/3260455757 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f557815e3a0 0x7f55780b2890 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f556c0097a0 tx=0x7f556c02b8d0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.221+0000 7f557d2e8640 1 -- 172.21.15.67:0/3260455757 shutdown_connections 2024-03-24T00:39:03.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.221+0000 7f557d2e8640 1 --2- 172.21.15.67:0/3260455757 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f55781626d0 0x7f55780ccaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.221+0000 7f557d2e8640 1 --2- 172.21.15.67:0/3260455757 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f557815e3a0 0x7f55780b2890 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.221+0000 7f557d2e8640 1 -- 172.21.15.67:0/3260455757 >> 172.21.15.67:0/3260455757 conn(0x7f55780b3c40 msgr2=0x7f55780b1dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:03.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.221+0000 7f557d2e8640 1 -- 172.21.15.67:0/3260455757 shutdown_connections 2024-03-24T00:39:03.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.221+0000 7f557d2e8640 1 -- 172.21.15.67:0/3260455757 wait complete. 2024-03-24T00:39:03.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.223+0000 7f557d2e8640 1 Processor -- start 2024-03-24T00:39:03.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.223+0000 7f557d2e8640 1 -- start start 2024-03-24T00:39:03.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.223+0000 7f557d2e8640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f55780cd030 0x7f557814f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.223+0000 7f557d2e8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f557815e3a0 0x7f5578160850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.223+0000 7f557d2e8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f55781626d0 0x7f55780c6500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.223+0000 7f557d2e8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5578162b10 con 0x7f557815e3a0 2024-03-24T00:39:03.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.223+0000 7f557d2e8640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f557815f100 con 0x7f55781626d0 2024-03-24T00:39:03.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.223+0000 7f557d2e8640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f557806b040 con 0x7f55780cd030 2024-03-24T00:39:03.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.223+0000 7f55767fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f557815e3a0 0x7f5578160850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.223+0000 7f55777fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f55781626d0 0x7f55780c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.223+0000 7f55767fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f557815e3a0 0x7f5578160850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59786/0 (socket says 172.21.15.67:59786) 2024-03-24T00:39:03.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.223+0000 7f55767fc640 1 -- 172.21.15.67:0/3706113076 learned_addr learned my addr 172.21.15.67:0/3706113076 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:03.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.223+0000 7f5576ffd640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f55780cd030 0x7f557814f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.224+0000 7f55767fc640 1 -- 172.21.15.67:0/3706113076 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f55780cd030 msgr2=0x7f557814f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:03.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.224+0000 7f55767fc640 1 --2- 172.21.15.67:0/3706113076 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f55780cd030 0x7f557814f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.224+0000 7f55767fc640 1 -- 172.21.15.67:0/3706113076 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f55781626d0 msgr2=0x7f55780c6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:03.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.224+0000 7f55767fc640 1 --2- 172.21.15.67:0/3706113076 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f55781626d0 0x7f55780c6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.224+0000 7f55767fc640 1 -- 172.21.15.67:0/3706113076 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5578161d40 con 0x7f557815e3a0 2024-03-24T00:39:03.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.224+0000 7f55777fe640 1 --2- 172.21.15.67:0/3706113076 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f55781626d0 0x7f55780c6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:03.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.224+0000 7f5576ffd640 1 --2- 172.21.15.67:0/3706113076 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f55780cd030 0x7f557814f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:03.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.224+0000 7f55767fc640 1 --2- 172.21.15.67:0/3706113076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f557815e3a0 0x7f5578160850 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f556800e960 tx=0x7f556800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:03.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.224+0000 7f555f7fe640 1 -- 172.21.15.67:0/3706113076 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5568017070 con 0x7f557815e3a0 2024-03-24T00:39:03.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.224+0000 7f555f7fe640 1 -- 172.21.15.67:0/3706113076 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f556800ca00 con 0x7f557815e3a0 2024-03-24T00:39:03.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.224+0000 7f555f7fe640 1 -- 172.21.15.67:0/3706113076 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f556800cdd0 con 0x7f557815e3a0 2024-03-24T00:39:03.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.224+0000 7f557d2e8640 1 -- 172.21.15.67:0/3706113076 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55781611c0 con 0x7f557815e3a0 2024-03-24T00:39:03.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.224+0000 7f557d2e8640 1 -- 172.21.15.67:0/3706113076 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5578161780 con 0x7f557815e3a0 2024-03-24T00:39:03.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.225+0000 7f557d2e8640 1 -- 172.21.15.67:0/3706113076 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f557804f400 con 0x7f557815e3a0 2024-03-24T00:39:03.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.226+0000 7f555f7fe640 1 -- 172.21.15.67:0/3706113076 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f556801a070 con 0x7f557815e3a0 2024-03-24T00:39:03.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.226+0000 7f555f7fe640 1 --2- 172.21.15.67:0/3706113076 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f556406bbd0 0x7f556406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.226+0000 7f555f7fe640 1 -- 172.21.15.67:0/3706113076 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f556808aed0 con 0x7f557815e3a0 2024-03-24T00:39:03.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.226+0000 7f5576ffd640 1 --2- 172.21.15.67:0/3706113076 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f556406bbd0 0x7f556406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.227+0000 7f5576ffd640 1 --2- 172.21.15.67:0/3706113076 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f556406bbd0 0x7f556406e090 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f556c036000 tx=0x7f556c03d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:03.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.230+0000 7f555f7fe640 1 -- 172.21.15.67:0/3706113076 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f55680590c0 con 0x7f557815e3a0 2024-03-24T00:39:03.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.412+0000 7f557d2e8640 1 -- 172.21.15.67:0/3706113076 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f55780b13f0 con 0x7f557815e3a0 2024-03-24T00:39:03.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.412+0000 7f555f7fe640 1 -- 172.21.15.67:0/3706113076 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7f556805d160 con 0x7f557815e3a0 2024-03-24T00:39:03.413 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-03-24T00:39:03.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.414+0000 7f557d2e8640 1 -- 172.21.15.67:0/3706113076 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f556406bbd0 msgr2=0x7f556406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:03.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.414+0000 7f557d2e8640 1 --2- 172.21.15.67:0/3706113076 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f556406bbd0 0x7f556406e090 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f556c036000 tx=0x7f556c03d040 comp rx=0 tx=0).stop 2024-03-24T00:39:03.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.414+0000 7f557d2e8640 1 -- 172.21.15.67:0/3706113076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f557815e3a0 msgr2=0x7f5578160850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:03.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.414+0000 7f557d2e8640 1 --2- 172.21.15.67:0/3706113076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f557815e3a0 0x7f5578160850 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f556800e960 tx=0x7f556800ee20 comp rx=0 tx=0).stop 2024-03-24T00:39:03.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.414+0000 7f557d2e8640 1 -- 172.21.15.67:0/3706113076 shutdown_connections 2024-03-24T00:39:03.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.414+0000 7f557d2e8640 1 --2- 172.21.15.67:0/3706113076 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f556406bbd0 0x7f556406e090 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.414+0000 7f557d2e8640 1 --2- 172.21.15.67:0/3706113076 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f55781626d0 0x7f55780c6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.414+0000 7f557d2e8640 1 --2- 172.21.15.67:0/3706113076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f557815e3a0 0x7f5578160850 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.414+0000 7f557d2e8640 1 --2- 172.21.15.67:0/3706113076 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f55780cd030 0x7f557814f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.414+0000 7f557d2e8640 1 -- 172.21.15.67:0/3706113076 >> 172.21.15.67:0/3706113076 conn(0x7f55780b3c40 msgr2=0x7f557806beb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:03.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.414+0000 7f557d2e8640 1 -- 172.21.15.67:0/3706113076 shutdown_connections 2024-03-24T00:39:03.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.414+0000 7f557d2e8640 1 -- 172.21.15.67:0/3706113076 wait complete. 2024-03-24T00:39:03.425 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T00:39:03.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.529+0000 7fdcc78dc640 1 Processor -- start 2024-03-24T00:39:03.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.529+0000 7fdcc78dc640 1 -- start start 2024-03-24T00:39:03.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.530+0000 7fdcc78dc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fdcc015e1e0 0x7fdcc00cb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.530+0000 7fdcc78dc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdcc00ccca0 0x7fdcc006b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.530+0000 7fdcc78dc640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc015e5c0 con 0x7fdcc006b580 2024-03-24T00:39:03.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.530+0000 7fdcc78dc640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcc00b1200 con 0x7fdcc00ccca0 2024-03-24T00:39:03.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.530+0000 7fdcc78dc640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcc00cb6c0 con 0x7fdcc015e1e0 2024-03-24T00:39:03.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.530+0000 7fdcc4e77640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdcc00ccca0 0x7fdcc006b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.530+0000 7fdcc4e77640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdcc00ccca0 0x7fdcc006b040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60482/0 (socket says 172.21.15.67:60482) 2024-03-24T00:39:03.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.530+0000 7fdcc4e77640 1 -- 172.21.15.67:0/136592139 learned_addr learned my addr 172.21.15.67:0/136592139 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:03.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.530+0000 7fdcc5678640 1 --2- 172.21.15.67:0/136592139 >> v2:172.21.15.72:3300/0 conn(0x7fdcc015e1e0 0x7fdcc00cb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.531+0000 7fdcc4e77640 1 -- 172.21.15.67:0/136592139 >> v2:172.21.15.72:3300/0 conn(0x7fdcc015e1e0 msgr2=0x7fdcc00cb0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:03.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.531+0000 7fdcc4e77640 1 --2- 172.21.15.67:0/136592139 >> v2:172.21.15.72:3300/0 conn(0x7fdcc015e1e0 0x7fdcc00cb0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.531+0000 7fdcc4e77640 1 -- 172.21.15.67:0/136592139 >> v1:172.21.15.72:6789/0 conn(0x7fdcc006b580 legacy=0x7fdcc00d0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:03.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.531+0000 7fdcc4e77640 1 -- 172.21.15.67:0/136592139 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc00cb9a0 con 0x7fdcc00ccca0 2024-03-24T00:39:03.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.531+0000 7fdcc4e77640 1 --2- 172.21.15.67:0/136592139 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcc00ccca0 0x7fdcc006b040 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fdcb4009870 tx=0x7fdcb402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b4d142ed59d46d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:03.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.531+0000 7fdcbffff640 1 -- 172.21.15.67:0/136592139 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb4035020 con 0x7fdcc00ccca0 2024-03-24T00:39:03.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.531+0000 7fdcbffff640 1 -- 172.21.15.67:0/136592139 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcb4031870 con 0x7fdcc00ccca0 2024-03-24T00:39:03.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.532+0000 7fdcbffff640 1 -- 172.21.15.67:0/136592139 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb4031c40 con 0x7fdcc00ccca0 2024-03-24T00:39:03.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.532+0000 7fdcc78dc640 1 -- 172.21.15.67:0/136592139 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcc00ccca0 msgr2=0x7fdcc006b040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:03.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.532+0000 7fdcc78dc640 1 --2- 172.21.15.67:0/136592139 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcc00ccca0 0x7fdcc006b040 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fdcb4009870 tx=0x7fdcb402b910 comp rx=0 tx=0).stop 2024-03-24T00:39:03.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.532+0000 7fdcc78dc640 1 -- 172.21.15.67:0/136592139 shutdown_connections 2024-03-24T00:39:03.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.532+0000 7fdcc78dc640 1 --2- 172.21.15.67:0/136592139 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcc00ccca0 0x7fdcc006b040 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.532+0000 7fdcc78dc640 1 --2- 172.21.15.67:0/136592139 >> v2:172.21.15.72:3300/0 conn(0x7fdcc015e1e0 0x7fdcc00cb0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.532+0000 7fdcc78dc640 1 -- 172.21.15.67:0/136592139 >> 172.21.15.67:0/136592139 conn(0x7fdcc0099a30 msgr2=0x7fdcc0099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:03.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.532+0000 7fdcc78dc640 1 -- 172.21.15.67:0/136592139 shutdown_connections 2024-03-24T00:39:03.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.532+0000 7fdcc78dc640 1 -- 172.21.15.67:0/136592139 wait complete. 2024-03-24T00:39:03.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.533+0000 7fdcc78dc640 1 Processor -- start 2024-03-24T00:39:03.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.533+0000 7fdcc78dc640 1 -- start start 2024-03-24T00:39:03.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.534+0000 7fdcc78dc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcc006b580 0x7fdcc01540b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.534+0000 7fdcc78dc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcc00ccca0 0x7fdcc015bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.534+0000 7fdcc78dc640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdcc015e1e0 0x7fdcc015c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.534+0000 7fdcc78dc640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc00ce800 con 0x7fdcc006b580 2024-03-24T00:39:03.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.534+0000 7fdcc78dc640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc00cbf60 con 0x7fdcc00ccca0 2024-03-24T00:39:03.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.534+0000 7fdcc78dc640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc015e9a0 con 0x7fdcc015e1e0 2024-03-24T00:39:03.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.534+0000 7fdcc5678640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcc006b580 0x7fdcc01540b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.534+0000 7fdcc5678640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcc006b580 0x7fdcc01540b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60484/0 (socket says 172.21.15.67:60484) 2024-03-24T00:39:03.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.534+0000 7fdcc5678640 1 -- 172.21.15.67:0/1331287947 learned_addr learned my addr 172.21.15.67:0/1331287947 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:03.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.534+0000 7fdcc4e77640 1 --2- 172.21.15.67:0/1331287947 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcc00ccca0 0x7fdcc015bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.534+0000 7fdcc5e79640 1 --2- 172.21.15.67:0/1331287947 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdcc015e1e0 0x7fdcc015c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.534+0000 7fdcc5678640 1 -- 172.21.15.67:0/1331287947 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdcc015e1e0 msgr2=0x7fdcc015c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:03.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.534+0000 7fdcc5678640 1 --2- 172.21.15.67:0/1331287947 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdcc015e1e0 0x7fdcc015c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.534+0000 7fdcc5678640 1 -- 172.21.15.67:0/1331287947 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcc00ccca0 msgr2=0x7fdcc015bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:03.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.534+0000 7fdcc5678640 1 --2- 172.21.15.67:0/1331287947 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcc00ccca0 0x7fdcc015bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.534+0000 7fdcc5678640 1 -- 172.21.15.67:0/1331287947 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc015ec80 con 0x7fdcc006b580 2024-03-24T00:39:03.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.535+0000 7fdcc5678640 1 --2- 172.21.15.67:0/1331287947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcc006b580 0x7fdcc01540b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdcb000efc0 tx=0x7fdcb000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:03.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.535+0000 7fdcbdffb640 1 -- 172.21.15.67:0/1331287947 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb0018020 con 0x7fdcc006b580 2024-03-24T00:39:03.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.535+0000 7fdcbdffb640 1 -- 172.21.15.67:0/1331287947 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcb0014480 con 0x7fdcc006b580 2024-03-24T00:39:03.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.535+0000 7fdcc78dc640 1 -- 172.21.15.67:0/1331287947 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcc015ef60 con 0x7fdcc006b580 2024-03-24T00:39:03.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.535+0000 7fdcbdffb640 1 -- 172.21.15.67:0/1331287947 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb0014850 con 0x7fdcc006b580 2024-03-24T00:39:03.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.535+0000 7fdcc78dc640 1 -- 172.21.15.67:0/1331287947 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcc015f760 con 0x7fdcc006b580 2024-03-24T00:39:03.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.536+0000 7fdcc78dc640 1 -- 172.21.15.67:0/1331287947 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcc004f400 con 0x7fdcc006b580 2024-03-24T00:39:03.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.536+0000 7fdcbdffb640 1 -- 172.21.15.67:0/1331287947 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fdcb0005d10 con 0x7fdcc006b580 2024-03-24T00:39:03.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.537+0000 7fdcbdffb640 1 --2- 172.21.15.67:0/1331287947 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdcac06bbd0 0x7fdcac06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.537+0000 7fdcbdffb640 1 -- 172.21.15.67:0/1331287947 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fdcb008b640 con 0x7fdcc006b580 2024-03-24T00:39:03.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.537+0000 7fdcc4e77640 1 --2- 172.21.15.67:0/1331287947 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdcac06bbd0 0x7fdcac06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.537+0000 7fdcc4e77640 1 --2- 172.21.15.67:0/1331287947 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdcac06bbd0 0x7fdcac06e090 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fdcb4036000 tx=0x7fdcb403c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:03.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.541+0000 7fdcbdffb640 1 -- 172.21.15.67:0/1331287947 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fdcb0022070 con 0x7fdcc006b580 2024-03-24T00:39:03.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.722+0000 7fdcc78dc640 1 -- 172.21.15.67:0/1331287947 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fdcc00700f0 con 0x7fdcc006b580 2024-03-24T00:39:03.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.723+0000 7fdcbdffb640 1 -- 172.21.15.67:0/1331287947 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7fdcb0010040 con 0x7fdcc006b580 2024-03-24T00:39:03.724 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-24T00:39:03.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.725+0000 7fdcc78dc640 1 -- 172.21.15.67:0/1331287947 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdcac06bbd0 msgr2=0x7fdcac06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:03.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.725+0000 7fdcc78dc640 1 --2- 172.21.15.67:0/1331287947 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdcac06bbd0 0x7fdcac06e090 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fdcb4036000 tx=0x7fdcb403c040 comp rx=0 tx=0).stop 2024-03-24T00:39:03.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.725+0000 7fdcc78dc640 1 -- 172.21.15.67:0/1331287947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcc006b580 msgr2=0x7fdcc01540b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:03.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.725+0000 7fdcc78dc640 1 --2- 172.21.15.67:0/1331287947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcc006b580 0x7fdcc01540b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdcb000efc0 tx=0x7fdcb000c490 comp rx=0 tx=0).stop 2024-03-24T00:39:03.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.725+0000 7fdcc78dc640 1 -- 172.21.15.67:0/1331287947 shutdown_connections 2024-03-24T00:39:03.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.725+0000 7fdcc78dc640 1 --2- 172.21.15.67:0/1331287947 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdcac06bbd0 0x7fdcac06e090 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.725+0000 7fdcc78dc640 1 --2- 172.21.15.67:0/1331287947 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdcc015e1e0 0x7fdcc015c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.725+0000 7fdcc78dc640 1 --2- 172.21.15.67:0/1331287947 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcc00ccca0 0x7fdcc015bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.725+0000 7fdcc78dc640 1 --2- 172.21.15.67:0/1331287947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcc006b580 0x7fdcc01540b0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.725+0000 7fdcc78dc640 1 -- 172.21.15.67:0/1331287947 >> 172.21.15.67:0/1331287947 conn(0x7fdcc0099a30 msgr2=0x7fdcc006f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:03.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.725+0000 7fdcc78dc640 1 -- 172.21.15.67:0/1331287947 shutdown_connections 2024-03-24T00:39:03.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.725+0000 7fdcc78dc640 1 -- 172.21.15.67:0/1331287947 wait complete. 2024-03-24T00:39:03.737 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T00:39:03.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.839+0000 7fdd612bd640 1 Processor -- start 2024-03-24T00:39:03.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.839+0000 7fdd612bd640 1 -- start start 2024-03-24T00:39:03.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.840+0000 7fdd612bd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdd5c0ceb10 0x7fdd5c06c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.840+0000 7fdd612bd640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdd5c0d26d0 0x7fdd5c06cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.840+0000 7fdd612bd640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd5c0d0270 con 0x7fdd5c06d060 2024-03-24T00:39:03.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.840+0000 7fdd612bd640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd5c06d4d0 con 0x7fdd5c0ceb10 2024-03-24T00:39:03.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.840+0000 7fdd612bd640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5c0d0e80 con 0x7fdd5c0d26d0 2024-03-24T00:39:03.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.840+0000 7fdd5affd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdd5c0ceb10 0x7fdd5c06c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.840+0000 7fdd5affd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdd5c0ceb10 0x7fdd5c06c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60496/0 (socket says 172.21.15.67:60496) 2024-03-24T00:39:03.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.840+0000 7fdd5affd640 1 -- 172.21.15.67:0/180242640 learned_addr learned my addr 172.21.15.67:0/180242640 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:03.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.840+0000 7fdd5a7fc640 1 --2- 172.21.15.67:0/180242640 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdd5c0d26d0 0x7fdd5c06cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.840+0000 7fdd5affd640 1 -- 172.21.15.67:0/180242640 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdd5c0d26d0 msgr2=0x7fdd5c06cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:03.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.840+0000 7fdd5affd640 1 --2- 172.21.15.67:0/180242640 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdd5c0d26d0 0x7fdd5c06cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.840+0000 7fdd59ffb640 1 -- 172.21.15.67:0/180242640 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 488582418 0 0) 0x7fdd5c0d0270 con 0x7fdd5c06d060 2024-03-24T00:39:03.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.841+0000 7fdd5affd640 1 -- 172.21.15.67:0/180242640 >> v1:172.21.15.72:6789/0 conn(0x7fdd5c06d060 legacy=0x7fdd5c161c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:03.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.841+0000 7fdd5affd640 1 -- 172.21.15.67:0/180242640 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd5c0d0550 con 0x7fdd5c0ceb10 2024-03-24T00:39:03.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.841+0000 7fdd5affd640 1 --2- 172.21.15.67:0/180242640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd5c0ceb10 0x7fdd5c06c4f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fdd54009870 tx=0x7fdd5402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1927339554bac52b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:03.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.841+0000 7fdd59ffb640 1 -- 172.21.15.67:0/180242640 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd54034070 con 0x7fdd5c0ceb10 2024-03-24T00:39:03.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.841+0000 7fdd59ffb640 1 -- 172.21.15.67:0/180242640 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd54030840 con 0x7fdd5c0ceb10 2024-03-24T00:39:03.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.841+0000 7fdd59ffb640 1 -- 172.21.15.67:0/180242640 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd54030c10 con 0x7fdd5c0ceb10 2024-03-24T00:39:03.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.842+0000 7fdd612bd640 1 -- 172.21.15.67:0/180242640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd5c0ceb10 msgr2=0x7fdd5c06c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:03.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.842+0000 7fdd612bd640 1 --2- 172.21.15.67:0/180242640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd5c0ceb10 0x7fdd5c06c4f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fdd54009870 tx=0x7fdd5402b980 comp rx=0 tx=0).stop 2024-03-24T00:39:03.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.842+0000 7fdd612bd640 1 -- 172.21.15.67:0/180242640 shutdown_connections 2024-03-24T00:39:03.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.842+0000 7fdd612bd640 1 --2- 172.21.15.67:0/180242640 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdd5c0d26d0 0x7fdd5c06cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.842+0000 7fdd612bd640 1 --2- 172.21.15.67:0/180242640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd5c0ceb10 0x7fdd5c06c4f0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.842+0000 7fdd612bd640 1 -- 172.21.15.67:0/180242640 >> 172.21.15.67:0/180242640 conn(0x7fdd5c0d1ad0 msgr2=0x7fdd5c0d1ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:03.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.842+0000 7fdd612bd640 1 -- 172.21.15.67:0/180242640 shutdown_connections 2024-03-24T00:39:03.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.842+0000 7fdd612bd640 1 -- 172.21.15.67:0/180242640 wait complete. 2024-03-24T00:39:03.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.843+0000 7fdd612bd640 1 Processor -- start 2024-03-24T00:39:03.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.843+0000 7fdd612bd640 1 -- start start 2024-03-24T00:39:03.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.844+0000 7fdd612bd640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdd5c06d060 0x7fdd5c0afb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.844+0000 7fdd612bd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd5c0ceb10 0x7fdd5c1566d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.844+0000 7fdd612bd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd5c0d26d0 0x7fdd5c0c5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.844+0000 7fdd612bd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5c0ceef0 con 0x7fdd5c0ceb10 2024-03-24T00:39:03.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.844+0000 7fdd612bd640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5c0d0b10 con 0x7fdd5c0d26d0 2024-03-24T00:39:03.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.844+0000 7fdd612bd640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5c0cf070 con 0x7fdd5c06d060 2024-03-24T00:39:03.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.844+0000 7fdd5a7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd5c0ceb10 0x7fdd5c1566d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.844+0000 7fdd5a7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd5c0ceb10 0x7fdd5c1566d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60508/0 (socket says 172.21.15.67:60508) 2024-03-24T00:39:03.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.844+0000 7fdd5a7fc640 1 -- 172.21.15.67:0/3678295811 learned_addr learned my addr 172.21.15.67:0/3678295811 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:03.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.844+0000 7fdd5b7fe640 1 --2- 172.21.15.67:0/3678295811 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd5c0d26d0 0x7fdd5c0c5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.844+0000 7fdd5affd640 1 --2- 172.21.15.67:0/3678295811 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdd5c06d060 0x7fdd5c0afb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.844+0000 7fdd5a7fc640 1 -- 172.21.15.67:0/3678295811 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdd5c06d060 msgr2=0x7fdd5c0afb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:03.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.844+0000 7fdd5a7fc640 1 --2- 172.21.15.67:0/3678295811 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdd5c06d060 0x7fdd5c0afb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.845+0000 7fdd5a7fc640 1 -- 172.21.15.67:0/3678295811 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd5c0d26d0 msgr2=0x7fdd5c0c5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:03.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.845+0000 7fdd5a7fc640 1 --2- 172.21.15.67:0/3678295811 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd5c0d26d0 0x7fdd5c0c5910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.845+0000 7fdd5a7fc640 1 -- 172.21.15.67:0/3678295811 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd5c0b0040 con 0x7fdd5c0ceb10 2024-03-24T00:39:03.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.845+0000 7fdd5a7fc640 1 --2- 172.21.15.67:0/3678295811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd5c0ceb10 0x7fdd5c1566d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fdd4400e9b0 tx=0x7fdd4400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:03.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.845+0000 7fdd537fe640 1 -- 172.21.15.67:0/3678295811 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd44017070 con 0x7fdd5c0ceb10 2024-03-24T00:39:03.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.845+0000 7fdd537fe640 1 -- 172.21.15.67:0/3678295811 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd4400c9a0 con 0x7fdd5c0ceb10 2024-03-24T00:39:03.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.845+0000 7fdd537fe640 1 -- 172.21.15.67:0/3678295811 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd4400cd90 con 0x7fdd5c0ceb10 2024-03-24T00:39:03.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.845+0000 7fdd612bd640 1 -- 172.21.15.67:0/3678295811 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd5c0aea60 con 0x7fdd5c0ceb10 2024-03-24T00:39:03.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.845+0000 7fdd612bd640 1 -- 172.21.15.67:0/3678295811 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd5c0af020 con 0x7fdd5c0ceb10 2024-03-24T00:39:03.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.846+0000 7fdd612bd640 1 -- 172.21.15.67:0/3678295811 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd5c04f400 con 0x7fdd5c0ceb10 2024-03-24T00:39:03.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.846+0000 7fdd537fe640 1 -- 172.21.15.67:0/3678295811 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fdd4401a070 con 0x7fdd5c0ceb10 2024-03-24T00:39:03.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.847+0000 7fdd537fe640 1 --2- 172.21.15.67:0/3678295811 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdd3c06bbd0 0x7fdd3c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.847+0000 7fdd537fe640 1 -- 172.21.15.67:0/3678295811 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fdd4408ad70 con 0x7fdd5c0ceb10 2024-03-24T00:39:03.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.847+0000 7fdd5affd640 1 --2- 172.21.15.67:0/3678295811 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdd3c06bbd0 0x7fdd3c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.847+0000 7fdd5affd640 1 --2- 172.21.15.67:0/3678295811 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdd3c06bbd0 0x7fdd3c06e090 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fdd54036000 tx=0x7fdd54008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:03.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:03.851+0000 7fdd537fe640 1 -- 172.21.15.67:0/3678295811 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fdd44058ed0 con 0x7fdd5c0ceb10 2024-03-24T00:39:04.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.045+0000 7fdd612bd640 1 -- 172.21.15.67:0/3678295811 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fdd5c0cf460 con 0x7fdd5c0ceb10 2024-03-24T00:39:04.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.046+0000 7fdd537fe640 1 -- 172.21.15.67:0/3678295811 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v25) v1 ==== 80+0+0 (secure 0 0 0) 0x7fdd4405cf70 con 0x7fdd5c0ceb10 2024-03-24T00:39:04.047 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-03-24T00:39:04.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.048+0000 7fdd612bd640 1 -- 172.21.15.67:0/3678295811 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdd3c06bbd0 msgr2=0x7fdd3c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:04.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.048+0000 7fdd612bd640 1 --2- 172.21.15.67:0/3678295811 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdd3c06bbd0 0x7fdd3c06e090 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fdd54036000 tx=0x7fdd54008030 comp rx=0 tx=0).stop 2024-03-24T00:39:04.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.048+0000 7fdd612bd640 1 -- 172.21.15.67:0/3678295811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd5c0ceb10 msgr2=0x7fdd5c1566d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:04.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.048+0000 7fdd612bd640 1 --2- 172.21.15.67:0/3678295811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd5c0ceb10 0x7fdd5c1566d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fdd4400e9b0 tx=0x7fdd4400ee70 comp rx=0 tx=0).stop 2024-03-24T00:39:04.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.048+0000 7fdd612bd640 1 -- 172.21.15.67:0/3678295811 shutdown_connections 2024-03-24T00:39:04.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.048+0000 7fdd612bd640 1 --2- 172.21.15.67:0/3678295811 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdd3c06bbd0 0x7fdd3c06e090 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.048+0000 7fdd612bd640 1 --2- 172.21.15.67:0/3678295811 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd5c0d26d0 0x7fdd5c0c5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.048+0000 7fdd612bd640 1 --2- 172.21.15.67:0/3678295811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd5c0ceb10 0x7fdd5c1566d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.048+0000 7fdd612bd640 1 --2- 172.21.15.67:0/3678295811 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdd5c06d060 0x7fdd5c0afb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.048+0000 7fdd612bd640 1 -- 172.21.15.67:0/3678295811 >> 172.21.15.67:0/3678295811 conn(0x7fdd5c0d1ad0 msgr2=0x7fdd5c0b2ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:04.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.048+0000 7fdd612bd640 1 -- 172.21.15.67:0/3678295811 shutdown_connections 2024-03-24T00:39:04.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.048+0000 7fdd612bd640 1 -- 172.21.15.67:0/3678295811 wait complete. 2024-03-24T00:39:04.059 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:39:04.059 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:39:04.074 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:39:04.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.215+0000 7f7b0aeb2640 1 Processor -- start 2024-03-24T00:39:04.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.215+0000 7f7b0aeb2640 1 -- start start 2024-03-24T00:39:04.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.216+0000 7f7b0aeb2640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7b0415df30 0x7f7b04162610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:04.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.216+0000 7f7b0aeb2640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7b0415e310 0x7f7b040d0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:04.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.216+0000 7f7b0aeb2640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b040ce920 con 0x7f7b040d0b80 2024-03-24T00:39:04.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.216+0000 7f7b0aeb2640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b04162b50 con 0x7f7b0415e310 2024-03-24T00:39:04.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.216+0000 7f7b0aeb2640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b040d0ff0 con 0x7f7b0415df30 2024-03-24T00:39:04.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.216+0000 7f7b03fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7b0415e310 0x7f7b040d0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:04.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.216+0000 7f7b03fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7b0415e310 0x7f7b040d0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60524/0 (socket says 172.21.15.67:60524) 2024-03-24T00:39:04.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.216+0000 7f7b03fff640 1 -- 172.21.15.67:0/515896044 learned_addr learned my addr 172.21.15.67:0/515896044 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:04.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.216+0000 7f7b0944f640 1 --1- 172.21.15.67:0/515896044 >> v1:172.21.15.67:6789/0 conn(0x7f7b040d0b80 0x7f7b040b57a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45298/0 (socket says 172.21.15.67:45298) 2024-03-24T00:39:04.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.216+0000 7f7b08c4e640 1 --2- 172.21.15.67:0/515896044 >> v2:172.21.15.72:3300/0 conn(0x7f7b0415df30 0x7f7b04162610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:04.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.216+0000 7f7b08c4e640 1 -- 172.21.15.67:0/515896044 >> v2:172.21.15.67:3300/0 conn(0x7f7b0415e310 msgr2=0x7f7b040d0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:04.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.216+0000 7f7b08c4e640 1 --2- 172.21.15.67:0/515896044 >> v2:172.21.15.67:3300/0 conn(0x7f7b0415e310 0x7f7b040d0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.216+0000 7f7b08c4e640 1 -- 172.21.15.67:0/515896044 >> v1:172.21.15.67:6789/0 conn(0x7f7b040d0b80 legacy=0x7f7b040b57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:04.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.216+0000 7f7b08c4e640 1 -- 172.21.15.67:0/515896044 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b040cef20 con 0x7f7b0415df30 2024-03-24T00:39:04.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.216+0000 7f7b037fe640 1 -- 172.21.15.67:0/515896044 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 322355838 0 0) 0x7f7b040ce920 con 0x7f7b040d0b80 2024-03-24T00:39:04.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.216+0000 7f7b03fff640 1 --2- 172.21.15.67:0/515896044 >> v2:172.21.15.67:3300/0 conn(0x7f7b0415e310 0x7f7b040d0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:04.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.217+0000 7f7b08c4e640 1 --2- 172.21.15.67:0/515896044 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b0415df30 0x7f7b04162610 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7b0408c970 tx=0x7f7afc02be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49e2d6c1ef78ac77 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:04.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.217+0000 7f7b037fe640 1 -- 172.21.15.67:0/515896044 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7afc039070 con 0x7f7b0415df30 2024-03-24T00:39:04.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.217+0000 7f7b037fe640 1 -- 172.21.15.67:0/515896044 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7afc02cbf0 con 0x7f7b0415df30 2024-03-24T00:39:04.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.217+0000 7f7b037fe640 1 -- 172.21.15.67:0/515896044 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7afc034440 con 0x7f7b0415df30 2024-03-24T00:39:04.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.218+0000 7f7b0aeb2640 1 -- 172.21.15.67:0/515896044 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b0415df30 msgr2=0x7f7b04162610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:04.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.218+0000 7f7b0aeb2640 1 --2- 172.21.15.67:0/515896044 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b0415df30 0x7f7b04162610 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7b0408c970 tx=0x7f7afc02be40 comp rx=0 tx=0).stop 2024-03-24T00:39:04.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.218+0000 7f7b0aeb2640 1 -- 172.21.15.67:0/515896044 shutdown_connections 2024-03-24T00:39:04.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.218+0000 7f7b0aeb2640 1 --2- 172.21.15.67:0/515896044 >> v2:172.21.15.67:3300/0 conn(0x7f7b0415e310 0x7f7b040d0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.218+0000 7f7b0aeb2640 1 --2- 172.21.15.67:0/515896044 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b0415df30 0x7f7b04162610 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.218+0000 7f7b0aeb2640 1 -- 172.21.15.67:0/515896044 >> 172.21.15.67:0/515896044 conn(0x7f7b0406c880 msgr2=0x7f7b0406cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:04.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.218+0000 7f7b0aeb2640 1 -- 172.21.15.67:0/515896044 shutdown_connections 2024-03-24T00:39:04.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.218+0000 7f7b0aeb2640 1 -- 172.21.15.67:0/515896044 wait complete. 2024-03-24T00:39:04.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.219+0000 7f7b0aeb2640 1 Processor -- start 2024-03-24T00:39:04.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.219+0000 7f7b0aeb2640 1 -- start start 2024-03-24T00:39:04.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.220+0000 7f7b0aeb2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b040d0b80 0x7f7b040b3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:04.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.220+0000 7f7b0aeb2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b0415df30 0x7f7b04149ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:04.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.220+0000 7f7b0aeb2640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7b041f3bc0 0x7f7b040c9220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:04.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.220+0000 7f7b0aeb2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b040b32f0 con 0x7f7b040d0b80 2024-03-24T00:39:04.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.220+0000 7f7b0aeb2640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b040cf4e0 con 0x7f7b0415df30 2024-03-24T00:39:04.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.220+0000 7f7b0aeb2640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b040ce580 con 0x7f7b041f3bc0 2024-03-24T00:39:04.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.220+0000 7f7b08c4e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b040d0b80 0x7f7b040b3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:04.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.220+0000 7f7b08c4e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b040d0b80 0x7f7b040b3700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60526/0 (socket says 172.21.15.67:60526) 2024-03-24T00:39:04.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.220+0000 7f7b08c4e640 1 -- 172.21.15.67:0/2797016744 learned_addr learned my addr 172.21.15.67:0/2797016744 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:04.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.220+0000 7f7b0944f640 1 --2- 172.21.15.67:0/2797016744 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7b041f3bc0 0x7f7b040c9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:04.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.220+0000 7f7b03fff640 1 --2- 172.21.15.67:0/2797016744 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b0415df30 0x7f7b04149ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:04.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.220+0000 7f7b08c4e640 1 -- 172.21.15.67:0/2797016744 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7b041f3bc0 msgr2=0x7f7b040c9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:04.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.220+0000 7f7b08c4e640 1 --2- 172.21.15.67:0/2797016744 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7b041f3bc0 0x7f7b040c9220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.220+0000 7f7b08c4e640 1 -- 172.21.15.67:0/2797016744 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b0415df30 msgr2=0x7f7b04149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:04.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.220+0000 7f7b08c4e640 1 --2- 172.21.15.67:0/2797016744 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b0415df30 0x7f7b04149ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.220+0000 7f7b08c4e640 1 -- 172.21.15.67:0/2797016744 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b041e9e60 con 0x7f7b040d0b80 2024-03-24T00:39:04.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.220+0000 7f7b03fff640 1 --2- 172.21.15.67:0/2797016744 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b0415df30 0x7f7b04149ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:04.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.221+0000 7f7b08c4e640 1 --2- 172.21.15.67:0/2797016744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b040d0b80 0x7f7b040b3700 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7afc037040 tx=0x7f7afc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:04.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.221+0000 7f7b017fa640 1 -- 172.21.15.67:0/2797016744 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7afc039050 con 0x7f7b040d0b80 2024-03-24T00:39:04.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.221+0000 7f7b017fa640 1 -- 172.21.15.67:0/2797016744 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7afc03a040 con 0x7f7b040d0b80 2024-03-24T00:39:04.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.221+0000 7f7b0aeb2640 1 -- 172.21.15.67:0/2797016744 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b0414a530 con 0x7f7b040d0b80 2024-03-24T00:39:04.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.221+0000 7f7b017fa640 1 -- 172.21.15.67:0/2797016744 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7afc003990 con 0x7f7b040d0b80 2024-03-24T00:39:04.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.221+0000 7f7b0aeb2640 1 -- 172.21.15.67:0/2797016744 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b040b40e0 con 0x7f7b040d0b80 2024-03-24T00:39:04.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.222+0000 7f7b0aeb2640 1 -- 172.21.15.67:0/2797016744 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b0404f400 con 0x7f7b040d0b80 2024-03-24T00:39:04.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.222+0000 7f7b017fa640 1 -- 172.21.15.67:0/2797016744 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7afc047030 con 0x7f7b040d0b80 2024-03-24T00:39:04.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.223+0000 7f7b017fa640 1 --2- 172.21.15.67:0/2797016744 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7ae406bbd0 0x7f7ae406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:04.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.223+0000 7f7b017fa640 1 -- 172.21.15.67:0/2797016744 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f7afc0acf30 con 0x7f7b040d0b80 2024-03-24T00:39:04.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.223+0000 7f7b03fff640 1 --2- 172.21.15.67:0/2797016744 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7ae406bbd0 0x7f7ae406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:04.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.223+0000 7f7b03fff640 1 --2- 172.21.15.67:0/2797016744 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7ae406bbd0 0x7f7ae406e090 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7b041ea1d0 tx=0x7f7af4009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:04.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.228+0000 7f7b017fa640 1 -- 172.21.15.67:0/2797016744 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7afc04e060 con 0x7f7b040d0b80 2024-03-24T00:39:04.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.414+0000 7f7b0aeb2640 1 -- 172.21.15.67:0/2797016744 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b040b9310 con 0x7f7b040d0b80 2024-03-24T00:39:04.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.415+0000 7f7b017fa640 1 -- 172.21.15.67:0/2797016744 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18454 (secure 0 0 0) 0x7f7afc07b120 con 0x7f7b040d0b80 2024-03-24T00:39:04.416 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:39:04.416 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":25,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:38:58.797179+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:38:50.077085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:38:51.095454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T00:38:54.807117+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:39:04.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.417+0000 7f7b0aeb2640 1 -- 172.21.15.67:0/2797016744 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7ae406bbd0 msgr2=0x7f7ae406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:04.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.417+0000 7f7b0aeb2640 1 --2- 172.21.15.67:0/2797016744 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7ae406bbd0 0x7f7ae406e090 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7b041ea1d0 tx=0x7f7af4009240 comp rx=0 tx=0).stop 2024-03-24T00:39:04.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.417+0000 7f7b0aeb2640 1 -- 172.21.15.67:0/2797016744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b040d0b80 msgr2=0x7f7b040b3700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:04.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.417+0000 7f7b0aeb2640 1 --2- 172.21.15.67:0/2797016744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b040d0b80 0x7f7b040b3700 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7afc037040 tx=0x7f7afc004a40 comp rx=0 tx=0).stop 2024-03-24T00:39:04.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.417+0000 7f7b0aeb2640 1 -- 172.21.15.67:0/2797016744 shutdown_connections 2024-03-24T00:39:04.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.417+0000 7f7b0aeb2640 1 --2- 172.21.15.67:0/2797016744 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7ae406bbd0 0x7f7ae406e090 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.417+0000 7f7b0aeb2640 1 --2- 172.21.15.67:0/2797016744 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7b041f3bc0 0x7f7b040c9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.417+0000 7f7b0aeb2640 1 --2- 172.21.15.67:0/2797016744 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b0415df30 0x7f7b04149ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.417+0000 7f7b0aeb2640 1 --2- 172.21.15.67:0/2797016744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b040d0b80 0x7f7b040b3700 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.417+0000 7f7b0aeb2640 1 -- 172.21.15.67:0/2797016744 >> 172.21.15.67:0/2797016744 conn(0x7f7b0406c880 msgr2=0x7f7b04162020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:04.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.417+0000 7f7b0aeb2640 1 -- 172.21.15.67:0/2797016744 shutdown_connections 2024-03-24T00:39:04.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.417+0000 7f7b0aeb2640 1 -- 172.21.15.67:0/2797016744 wait complete. 2024-03-24T00:39:04.428 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:38:39.364950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:38:50.077085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:38:51.095454+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T00:38:54.807117+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T00:39:04.429 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:39:04.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.531+0000 7fc6fb6c6640 1 Processor -- start 2024-03-24T00:39:04.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.532+0000 7fc6fb6c6640 1 -- start start 2024-03-24T00:39:04.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.532+0000 7fc6fb6c6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc6f40d18b0 0x7fc6f415c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:04.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.532+0000 7fc6fb6c6640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc6f40d1c90 0x7fc6f415fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:04.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.532+0000 7fc6fb6c6640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6f415c7c0 con 0x7fc6f4160260 2024-03-24T00:39:04.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.532+0000 7fc6fb6c6640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6f415caa0 con 0x7fc6f40d1c90 2024-03-24T00:39:04.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.532+0000 7fc6fb6c6640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f415cd80 con 0x7fc6f40d18b0 2024-03-24T00:39:04.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.532+0000 7fc6f8c61640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc6f40d1c90 0x7fc6f415fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:04.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.532+0000 7fc6f8c61640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc6f40d1c90 0x7fc6f415fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60536/0 (socket says 172.21.15.67:60536) 2024-03-24T00:39:04.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.532+0000 7fc6f9c63640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc6f4160260 0x7fc6f40b1d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45300/0 (socket says 172.21.15.67:45300) 2024-03-24T00:39:04.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.533+0000 7fc6f8c61640 1 -- 172.21.15.67:0/2278250576 learned_addr learned my addr 172.21.15.67:0/2278250576 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:04.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.533+0000 7fc6f9462640 1 --2- 172.21.15.67:0/2278250576 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc6f40d18b0 0x7fc6f415c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:04.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.533+0000 7fc6f8c61640 1 -- 172.21.15.67:0/2278250576 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc6f40d18b0 msgr2=0x7fc6f415c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:04.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.533+0000 7fc6f8c61640 1 --2- 172.21.15.67:0/2278250576 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc6f40d18b0 0x7fc6f415c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.533+0000 7fc6f8c61640 1 -- 172.21.15.67:0/2278250576 >> v1:172.21.15.67:6789/0 conn(0x7fc6f4160260 legacy=0x7fc6f40b1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:04.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.533+0000 7fc6f8c61640 1 -- 172.21.15.67:0/2278250576 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6f415d060 con 0x7fc6f40d1c90 2024-03-24T00:39:04.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.533+0000 7fc6f3fff640 1 -- 172.21.15.67:0/2278250576 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4179964574 0 0) 0x7fc6f415c7c0 con 0x7fc6f4160260 2024-03-24T00:39:04.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.534+0000 7fc6f8c61640 1 --2- 172.21.15.67:0/2278250576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6f40d1c90 0x7fc6f415fd20 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc6e8009870 tx=0x7fc6e802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48965a2eeb9d6fa4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:04.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.534+0000 7fc6f3fff640 1 -- 172.21.15.67:0/2278250576 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e8035020 con 0x7fc6f40d1c90 2024-03-24T00:39:04.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.534+0000 7fc6f3fff640 1 -- 172.21.15.67:0/2278250576 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6e8031640 con 0x7fc6f40d1c90 2024-03-24T00:39:04.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.534+0000 7fc6f3fff640 1 -- 172.21.15.67:0/2278250576 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e802d040 con 0x7fc6f40d1c90 2024-03-24T00:39:04.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.534+0000 7fc6fb6c6640 1 -- 172.21.15.67:0/2278250576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6f40d1c90 msgr2=0x7fc6f415fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:04.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.534+0000 7fc6fb6c6640 1 --2- 172.21.15.67:0/2278250576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6f40d1c90 0x7fc6f415fd20 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc6e8009870 tx=0x7fc6e802b8f0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.534+0000 7fc6fb6c6640 1 -- 172.21.15.67:0/2278250576 shutdown_connections 2024-03-24T00:39:04.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.534+0000 7fc6fb6c6640 1 --2- 172.21.15.67:0/2278250576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6f40d1c90 0x7fc6f415fd20 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.534+0000 7fc6fb6c6640 1 --2- 172.21.15.67:0/2278250576 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc6f40d18b0 0x7fc6f415c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.534+0000 7fc6fb6c6640 1 -- 172.21.15.67:0/2278250576 >> 172.21.15.67:0/2278250576 conn(0x7fc6f408b1b0 msgr2=0x7fc6f408b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:04.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.534+0000 7fc6fb6c6640 1 -- 172.21.15.67:0/2278250576 shutdown_connections 2024-03-24T00:39:04.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.534+0000 7fc6fb6c6640 1 -- 172.21.15.67:0/2278250576 wait complete. 2024-03-24T00:39:04.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.535+0000 7fc6fb6c6640 1 Processor -- start 2024-03-24T00:39:04.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.536+0000 7fc6fb6c6640 1 -- start start 2024-03-24T00:39:04.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.536+0000 7fc6fb6c6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc6f40d18b0 0x7fc6f4151de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:04.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.536+0000 7fc6fb6c6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6f40d1c90 0x7fc6f40cac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:04.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.536+0000 7fc6fb6c6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc6f4160260 0x7fc6f415f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:04.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.536+0000 7fc6fb6c6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f40d2270 con 0x7fc6f40d1c90 2024-03-24T00:39:04.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.536+0000 7fc6fb6c6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f40d20f0 con 0x7fc6f40d18b0 2024-03-24T00:39:04.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.536+0000 7fc6fb6c6640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f40d07e0 con 0x7fc6f4160260 2024-03-24T00:39:04.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.536+0000 7fc6f8c61640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6f40d1c90 0x7fc6f40cac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:04.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.536+0000 7fc6f8c61640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6f40d1c90 0x7fc6f40cac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60538/0 (socket says 172.21.15.67:60538) 2024-03-24T00:39:04.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.536+0000 7fc6f8c61640 1 -- 172.21.15.67:0/542634912 learned_addr learned my addr 172.21.15.67:0/542634912 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:04.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.536+0000 7fc6f9462640 1 --2- 172.21.15.67:0/542634912 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc6f40d18b0 0x7fc6f4151de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:04.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.536+0000 7fc6f9c63640 1 --2- 172.21.15.67:0/542634912 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc6f4160260 0x7fc6f415f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:04.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.537+0000 7fc6f8c61640 1 -- 172.21.15.67:0/542634912 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc6f4160260 msgr2=0x7fc6f415f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:04.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.537+0000 7fc6f8c61640 1 --2- 172.21.15.67:0/542634912 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc6f4160260 0x7fc6f415f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.537+0000 7fc6f8c61640 1 -- 172.21.15.67:0/542634912 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc6f40d18b0 msgr2=0x7fc6f4151de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:04.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.537+0000 7fc6f8c61640 1 --2- 172.21.15.67:0/542634912 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc6f40d18b0 0x7fc6f4151de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.537+0000 7fc6f8c61640 1 -- 172.21.15.67:0/542634912 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6f4156c00 con 0x7fc6f40d1c90 2024-03-24T00:39:04.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.537+0000 7fc6f8c61640 1 --2- 172.21.15.67:0/542634912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6f40d1c90 0x7fc6f40cac90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc6e8000c00 tx=0x7fc6e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:04.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.537+0000 7fc6f1ffb640 1 -- 172.21.15.67:0/542634912 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e8040070 con 0x7fc6f40d1c90 2024-03-24T00:39:04.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.537+0000 7fc6f1ffb640 1 -- 172.21.15.67:0/542634912 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6e80399f0 con 0x7fc6f40d1c90 2024-03-24T00:39:04.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.537+0000 7fc6fb6c6640 1 -- 172.21.15.67:0/542634912 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6f40cb1d0 con 0x7fc6f40d1c90 2024-03-24T00:39:04.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.537+0000 7fc6f1ffb640 1 -- 172.21.15.67:0/542634912 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e8035040 con 0x7fc6f40d1c90 2024-03-24T00:39:04.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.537+0000 7fc6fb6c6640 1 -- 172.21.15.67:0/542634912 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6f40c92e0 con 0x7fc6f40d1c90 2024-03-24T00:39:04.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.538+0000 7fc6fb6c6640 1 -- 172.21.15.67:0/542634912 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6f404f400 con 0x7fc6f40d1c90 2024-03-24T00:39:04.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.538+0000 7fc6f1ffb640 1 -- 172.21.15.67:0/542634912 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc6e8038020 con 0x7fc6f40d1c90 2024-03-24T00:39:04.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.539+0000 7fc6f1ffb640 1 --2- 172.21.15.67:0/542634912 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc6e006bc00 0x7fc6e006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:04.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.539+0000 7fc6f1ffb640 1 -- 172.21.15.67:0/542634912 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fc6e80ac5d0 con 0x7fc6f40d1c90 2024-03-24T00:39:04.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.539+0000 7fc6f9462640 1 --2- 172.21.15.67:0/542634912 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc6e006bc00 0x7fc6e006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:04.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.539+0000 7fc6f9462640 1 --2- 172.21.15.67:0/542634912 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc6e006bc00 0x7fc6e006e0c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc6f40b7c50 tx=0x7fc6ec009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:04.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.543+0000 7fc6f1ffb640 1 -- 172.21.15.67:0/542634912 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc6f4156c00 con 0x7fc6f40d1c90 2024-03-24T00:39:04.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.723+0000 7fc6fb6c6640 1 -- 172.21.15.67:0/542634912 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc6f40b1840 con 0x7fc6f40d1c90 2024-03-24T00:39:04.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.724+0000 7fc6f1ffb640 1 -- 172.21.15.67:0/542634912 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc6e807a730 con 0x7fc6f40d1c90 2024-03-24T00:39:04.724 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-24T00:39:04.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.726+0000 7fc6fb6c6640 1 -- 172.21.15.67:0/542634912 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc6e006bc00 msgr2=0x7fc6e006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:04.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.726+0000 7fc6fb6c6640 1 --2- 172.21.15.67:0/542634912 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc6e006bc00 0x7fc6e006e0c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc6f40b7c50 tx=0x7fc6ec009190 comp rx=0 tx=0).stop 2024-03-24T00:39:04.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.726+0000 7fc6fb6c6640 1 -- 172.21.15.67:0/542634912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6f40d1c90 msgr2=0x7fc6f40cac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:04.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.726+0000 7fc6fb6c6640 1 --2- 172.21.15.67:0/542634912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6f40d1c90 0x7fc6f40cac90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc6e8000c00 tx=0x7fc6e80046c0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.726+0000 7fc6fb6c6640 1 -- 172.21.15.67:0/542634912 shutdown_connections 2024-03-24T00:39:04.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.726+0000 7fc6fb6c6640 1 --2- 172.21.15.67:0/542634912 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc6e006bc00 0x7fc6e006e0c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.726+0000 7fc6fb6c6640 1 --2- 172.21.15.67:0/542634912 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc6f4160260 0x7fc6f415f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.726+0000 7fc6fb6c6640 1 --2- 172.21.15.67:0/542634912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6f40d1c90 0x7fc6f40cac90 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.726+0000 7fc6fb6c6640 1 --2- 172.21.15.67:0/542634912 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc6f40d18b0 0x7fc6f4151de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.726+0000 7fc6fb6c6640 1 -- 172.21.15.67:0/542634912 >> 172.21.15.67:0/542634912 conn(0x7fc6f408b1b0 msgr2=0x7fc6f4070810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:04.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.726+0000 7fc6fb6c6640 1 -- 172.21.15.67:0/542634912 shutdown_connections 2024-03-24T00:39:04.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.726+0000 7fc6fb6c6640 1 -- 172.21.15.67:0/542634912 wait complete. 2024-03-24T00:39:04.737 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:39:04.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.843+0000 7fee926ff640 1 Processor -- start 2024-03-24T00:39:04.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.843+0000 7fee926ff640 1 -- start start 2024-03-24T00:39:04.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.844+0000 7fee926ff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fee8c161370 0x7fee8c161ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:04.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.844+0000 7fee926ff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fee8c0b9690 0x7fee8c0ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:04.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.844+0000 7fee926ff640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee8c161ff0 con 0x7fee8c0cef30 2024-03-24T00:39:04.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.844+0000 7fee926ff640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee8c0b6e10 con 0x7fee8c0b9690 2024-03-24T00:39:04.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.844+0000 7fee926ff640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee8c0cf380 con 0x7fee8c161370 2024-03-24T00:39:04.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.844+0000 7fee8b7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fee8c0b9690 0x7fee8c0ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:04.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.844+0000 7fee90c9c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fee8c0cef30 0x7fee8c0b5830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45314/0 (socket says 172.21.15.67:45314) 2024-03-24T00:39:04.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.844+0000 7fee90c9c640 1 -- 172.21.15.67:0/2530523857 learned_addr learned my addr 172.21.15.67:0/2530523857 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:04.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.844+0000 7fee8bfff640 1 --2- 172.21.15.67:0/2530523857 >> v2:172.21.15.72:3300/0 conn(0x7fee8c161370 0x7fee8c161ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:04.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.844+0000 7fee8b7fe640 1 -- 172.21.15.67:0/2530523857 >> v2:172.21.15.72:3300/0 conn(0x7fee8c161370 msgr2=0x7fee8c161ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:04.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.844+0000 7fee8b7fe640 1 --2- 172.21.15.67:0/2530523857 >> v2:172.21.15.72:3300/0 conn(0x7fee8c161370 0x7fee8c161ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.844+0000 7fee8b7fe640 1 -- 172.21.15.67:0/2530523857 >> v1:172.21.15.67:6789/0 conn(0x7fee8c0cef30 legacy=0x7fee8c0b5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:04.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.844+0000 7fee8b7fe640 1 -- 172.21.15.67:0/2530523857 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee8c162290 con 0x7fee8c0b9690 2024-03-24T00:39:04.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.845+0000 7fee8b7fe640 1 --2- 172.21.15.67:0/2530523857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee8c0b9690 0x7fee8c0ce9f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fee8c06b4a0 tx=0x7fee7c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a4fa7681db19cd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:04.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.845+0000 7fee8affd640 1 -- 172.21.15.67:0/2530523857 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee7c034070 con 0x7fee8c0b9690 2024-03-24T00:39:04.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.845+0000 7fee8affd640 1 -- 172.21.15.67:0/2530523857 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee7c030610 con 0x7fee8c0b9690 2024-03-24T00:39:04.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.845+0000 7fee8affd640 1 -- 172.21.15.67:0/2530523857 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee7c0309e0 con 0x7fee8c0b9690 2024-03-24T00:39:04.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.845+0000 7fee926ff640 1 -- 172.21.15.67:0/2530523857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee8c0b9690 msgr2=0x7fee8c0ce9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:04.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.845+0000 7fee926ff640 1 --2- 172.21.15.67:0/2530523857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee8c0b9690 0x7fee8c0ce9f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fee8c06b4a0 tx=0x7fee7c02b960 comp rx=0 tx=0).stop 2024-03-24T00:39:04.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.846+0000 7fee926ff640 1 -- 172.21.15.67:0/2530523857 shutdown_connections 2024-03-24T00:39:04.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.846+0000 7fee926ff640 1 --2- 172.21.15.67:0/2530523857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee8c0b9690 0x7fee8c0ce9f0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.846+0000 7fee926ff640 1 --2- 172.21.15.67:0/2530523857 >> v2:172.21.15.72:3300/0 conn(0x7fee8c161370 0x7fee8c161ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.846+0000 7fee926ff640 1 -- 172.21.15.67:0/2530523857 >> 172.21.15.67:0/2530523857 conn(0x7fee8c06c880 msgr2=0x7fee8c06cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:04.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.846+0000 7fee926ff640 1 -- 172.21.15.67:0/2530523857 shutdown_connections 2024-03-24T00:39:04.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.846+0000 7fee926ff640 1 -- 172.21.15.67:0/2530523857 wait complete. 2024-03-24T00:39:04.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.847+0000 7fee926ff640 1 Processor -- start 2024-03-24T00:39:04.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.847+0000 7fee926ff640 1 -- start start 2024-03-24T00:39:04.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.848+0000 7fee926ff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee8c0b9690 0x7fee8c14fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:04.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.848+0000 7fee926ff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fee8c0cef30 0x7fee8c0cd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:04.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.848+0000 7fee926ff640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fee8c161370 0x7fee8c0b6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:04.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.848+0000 7fee926ff640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee8c162850 con 0x7fee8c0b9690 2024-03-24T00:39:04.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.848+0000 7fee926ff640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee8c0d2190 con 0x7fee8c0cef30 2024-03-24T00:39:04.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.848+0000 7fee926ff640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee8c161750 con 0x7fee8c161370 2024-03-24T00:39:04.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.848+0000 7fee8bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee8c0b9690 0x7fee8c14fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:04.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.848+0000 7fee8bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee8c0b9690 0x7fee8c14fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60558/0 (socket says 172.21.15.67:60558) 2024-03-24T00:39:04.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.848+0000 7fee8bfff640 1 -- 172.21.15.67:0/3725132443 learned_addr learned my addr 172.21.15.67:0/3725132443 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:04.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.848+0000 7fee8b7fe640 1 --2- 172.21.15.67:0/3725132443 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fee8c0cef30 0x7fee8c0cd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:04.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.848+0000 7fee90c9c640 1 --2- 172.21.15.67:0/3725132443 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fee8c161370 0x7fee8c0b6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:04.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.848+0000 7fee8bfff640 1 -- 172.21.15.67:0/3725132443 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fee8c161370 msgr2=0x7fee8c0b6fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:04.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.848+0000 7fee8bfff640 1 --2- 172.21.15.67:0/3725132443 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fee8c161370 0x7fee8c0b6fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.848+0000 7fee8bfff640 1 -- 172.21.15.67:0/3725132443 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fee8c0cef30 msgr2=0x7fee8c0cd1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:04.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.848+0000 7fee8bfff640 1 --2- 172.21.15.67:0/3725132443 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fee8c0cef30 0x7fee8c0cd1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.848+0000 7fee8bfff640 1 -- 172.21.15.67:0/3725132443 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee8c0cda20 con 0x7fee8c0b9690 2024-03-24T00:39:04.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.848+0000 7fee8bfff640 1 --2- 172.21.15.67:0/3725132443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee8c0b9690 0x7fee8c14fb90 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fee8000e8a0 tx=0x7fee8000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:04.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.848+0000 7fee88ff9640 1 -- 172.21.15.67:0/3725132443 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee80017070 con 0x7fee8c0b9690 2024-03-24T00:39:04.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.848+0000 7fee88ff9640 1 -- 172.21.15.67:0/3725132443 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee8000c8e0 con 0x7fee8c0b9690 2024-03-24T00:39:04.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.848+0000 7fee926ff640 1 -- 172.21.15.67:0/3725132443 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee8c0b3a60 con 0x7fee8c0b9690 2024-03-24T00:39:04.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.848+0000 7fee88ff9640 1 -- 172.21.15.67:0/3725132443 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee8000ccd0 con 0x7fee8c0b9690 2024-03-24T00:39:04.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.848+0000 7fee926ff640 1 -- 172.21.15.67:0/3725132443 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee8c0b4020 con 0x7fee8c0b9690 2024-03-24T00:39:04.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.849+0000 7fee926ff640 1 -- 172.21.15.67:0/3725132443 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee8c04f400 con 0x7fee8c0b9690 2024-03-24T00:39:04.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.850+0000 7fee88ff9640 1 -- 172.21.15.67:0/3725132443 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fee8001a070 con 0x7fee8c0b9690 2024-03-24T00:39:04.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.850+0000 7fee88ff9640 1 --2- 172.21.15.67:0/3725132443 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fee7806bbd0 0x7fee7806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:04.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.850+0000 7fee88ff9640 1 -- 172.21.15.67:0/3725132443 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fee8008ae70 con 0x7fee8c0b9690 2024-03-24T00:39:04.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.850+0000 7fee8b7fe640 1 --2- 172.21.15.67:0/3725132443 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fee7806bbd0 0x7fee7806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:04.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.851+0000 7fee8b7fe640 1 --2- 172.21.15.67:0/3725132443 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fee7806bbd0 0x7fee7806e090 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fee7c036000 tx=0x7fee7c03b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:04.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:04.854+0000 7fee88ff9640 1 -- 172.21.15.67:0/3725132443 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fee80059060 con 0x7fee8c0b9690 2024-03-24T00:39:05.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.035+0000 7fee926ff640 1 -- 172.21.15.67:0/3725132443 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fee8c0b9c90 con 0x7fee8c0b9690 2024-03-24T00:39:05.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.035+0000 7fee88ff9640 1 -- 172.21.15.67:0/3725132443 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7fee8005d100 con 0x7fee8c0b9690 2024-03-24T00:39:05.036 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-24T00:39:05.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.037+0000 7fee926ff640 1 -- 172.21.15.67:0/3725132443 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fee7806bbd0 msgr2=0x7fee7806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.037+0000 7fee926ff640 1 --2- 172.21.15.67:0/3725132443 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fee7806bbd0 0x7fee7806e090 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fee7c036000 tx=0x7fee7c03b040 comp rx=0 tx=0).stop 2024-03-24T00:39:05.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.037+0000 7fee926ff640 1 -- 172.21.15.67:0/3725132443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee8c0b9690 msgr2=0x7fee8c14fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.037+0000 7fee926ff640 1 --2- 172.21.15.67:0/3725132443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee8c0b9690 0x7fee8c14fb90 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fee8000e8a0 tx=0x7fee8000ed60 comp rx=0 tx=0).stop 2024-03-24T00:39:05.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.037+0000 7fee926ff640 1 -- 172.21.15.67:0/3725132443 shutdown_connections 2024-03-24T00:39:05.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.037+0000 7fee926ff640 1 --2- 172.21.15.67:0/3725132443 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fee7806bbd0 0x7fee7806e090 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.037+0000 7fee926ff640 1 --2- 172.21.15.67:0/3725132443 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fee8c161370 0x7fee8c0b6fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.037+0000 7fee926ff640 1 --2- 172.21.15.67:0/3725132443 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fee8c0cef30 0x7fee8c0cd1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.037+0000 7fee926ff640 1 --2- 172.21.15.67:0/3725132443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee8c0b9690 0x7fee8c14fb90 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.037+0000 7fee926ff640 1 -- 172.21.15.67:0/3725132443 >> 172.21.15.67:0/3725132443 conn(0x7fee8c06c880 msgr2=0x7fee8c0c8820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:05.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.037+0000 7fee926ff640 1 -- 172.21.15.67:0/3725132443 shutdown_connections 2024-03-24T00:39:05.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.037+0000 7fee926ff640 1 -- 172.21.15.67:0/3725132443 wait complete. 2024-03-24T00:39:05.049 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:39:05.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.152+0000 7f81d426c640 1 Processor -- start 2024-03-24T00:39:05.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.152+0000 7f81d426c640 1 -- start start 2024-03-24T00:39:05.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.153+0000 7f81d426c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f81cc0b9690 0x7f81cc0ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.153+0000 7f81d426c640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81cc162290 con 0x7f81cc0cef30 2024-03-24T00:39:05.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.153+0000 7f81d426c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81cc162570 con 0x7f81cc161370 2024-03-24T00:39:05.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.153+0000 7f81d426c640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81cc0b6e10 con 0x7f81cc0b9690 2024-03-24T00:39:05.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.153+0000 7f81d1807640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f81cc0b9690 0x7f81cc0ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.153+0000 7f81d2809640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f81cc0cef30 0x7f81cc0d2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45326/0 (socket says 172.21.15.67:45326) 2024-03-24T00:39:05.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.153+0000 7f81d2809640 1 -- 172.21.15.67:0/1226519010 learned_addr learned my addr 172.21.15.67:0/1226519010 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:05.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.153+0000 7f81d1807640 1 -- 172.21.15.67:0/1226519010 >> v1:172.21.15.72:6789/0 conn(0x7f81cc161370 legacy=0x7f81cc161ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.153+0000 7f81d1807640 1 -- 172.21.15.67:0/1226519010 >> v1:172.21.15.67:6789/0 conn(0x7f81cc0cef30 legacy=0x7f81cc0d2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.153+0000 7f81d1807640 1 -- 172.21.15.67:0/1226519010 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81cc162850 con 0x7f81cc0b9690 2024-03-24T00:39:05.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.153+0000 7f81d1006640 1 -- 172.21.15.67:0/1226519010 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 44434256 0 0) 0x7f81cc162570 con 0x7f81cc161370 2024-03-24T00:39:05.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.154+0000 7f81d1807640 1 --2- 172.21.15.67:0/1226519010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81cc0b9690 0x7f81cc0ce9f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f81bc0097a0 tx=0x7f81bc02b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49140930607fcb14 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:05.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.154+0000 7f81d1006640 1 -- 172.21.15.67:0/1226519010 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81bc038070 con 0x7f81cc0b9690 2024-03-24T00:39:05.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.154+0000 7f81d1006640 1 -- 172.21.15.67:0/1226519010 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81bc031700 con 0x7f81cc0b9690 2024-03-24T00:39:05.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.154+0000 7f81d1006640 1 -- 172.21.15.67:0/1226519010 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81bc031ad0 con 0x7f81cc0b9690 2024-03-24T00:39:05.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.154+0000 7f81d426c640 1 -- 172.21.15.67:0/1226519010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81cc0b9690 msgr2=0x7f81cc0ce9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.154+0000 7f81d426c640 1 --2- 172.21.15.67:0/1226519010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81cc0b9690 0x7f81cc0ce9f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f81bc0097a0 tx=0x7f81bc02b820 comp rx=0 tx=0).stop 2024-03-24T00:39:05.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.155+0000 7f81d426c640 1 -- 172.21.15.67:0/1226519010 shutdown_connections 2024-03-24T00:39:05.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.155+0000 7f81d426c640 1 --2- 172.21.15.67:0/1226519010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81cc0b9690 0x7f81cc0ce9f0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.155+0000 7f81d426c640 1 -- 172.21.15.67:0/1226519010 >> 172.21.15.67:0/1226519010 conn(0x7f81cc06c880 msgr2=0x7f81cc06cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:05.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.155+0000 7f81d426c640 1 -- 172.21.15.67:0/1226519010 shutdown_connections 2024-03-24T00:39:05.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.155+0000 7f81d426c640 1 -- 172.21.15.67:0/1226519010 wait complete. 2024-03-24T00:39:05.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.156+0000 7f81d426c640 1 Processor -- start 2024-03-24T00:39:05.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.156+0000 7f81d426c640 1 -- start start 2024-03-24T00:39:05.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.156+0000 7f81d426c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f81cc0b9690 0x7f81cc148350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.156+0000 7f81d426c640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f81cc0cef30 0x7f81cc158160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.156+0000 7f81d426c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81cc161370 0x7f81cc15bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.156+0000 7f81d426c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81cc0d2190 con 0x7f81cc161370 2024-03-24T00:39:05.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.156+0000 7f81d426c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81cc0cf380 con 0x7f81cc0b9690 2024-03-24T00:39:05.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.156+0000 7f81d426c640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81cc0d2d30 con 0x7f81cc0cef30 2024-03-24T00:39:05.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.157+0000 7f81d2809640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81cc161370 0x7f81cc15bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.157+0000 7f81d1807640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f81cc0cef30 0x7f81cc158160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.157+0000 7f81d2008640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f81cc0b9690 0x7f81cc148350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.157+0000 7f81d2809640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81cc161370 0x7f81cc15bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60580/0 (socket says 172.21.15.67:60580) 2024-03-24T00:39:05.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.157+0000 7f81d1807640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f81cc0cef30 0x7f81cc158160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.67:48764/0 (socket says 172.21.15.67:48764) 2024-03-24T00:39:05.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.157+0000 7f81d2809640 1 -- 172.21.15.67:0/1946723975 learned_addr learned my addr 172.21.15.67:0/1946723975 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:05.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.157+0000 7f81d2008640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f81cc0b9690 0x7f81cc148350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:33834/0 (socket says 172.21.15.67:33834) 2024-03-24T00:39:05.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.157+0000 7f81d2008640 1 -- 172.21.15.67:0/1946723975 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f81cc0cef30 msgr2=0x7f81cc158160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.157+0000 7f81d2008640 1 --2- 172.21.15.67:0/1946723975 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f81cc0cef30 0x7f81cc158160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.157+0000 7f81d2008640 1 -- 172.21.15.67:0/1946723975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81cc161370 msgr2=0x7f81cc15bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.157+0000 7f81d2008640 1 --2- 172.21.15.67:0/1946723975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81cc161370 0x7f81cc15bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.157+0000 7f81d2008640 1 -- 172.21.15.67:0/1946723975 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81cc148890 con 0x7f81cc0b9690 2024-03-24T00:39:05.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.157+0000 7f81d2008640 1 --2- 172.21.15.67:0/1946723975 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f81cc0b9690 0x7f81cc148350 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f81c800c410 tx=0x7f81c800c810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:05.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.157+0000 7f81c2ffd640 1 -- 172.21.15.67:0/1946723975 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c8017070 con 0x7f81cc0b9690 2024-03-24T00:39:05.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.157+0000 7f81c2ffd640 1 -- 172.21.15.67:0/1946723975 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81c80035f0 con 0x7f81cc0b9690 2024-03-24T00:39:05.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.157+0000 7f81d426c640 1 -- 172.21.15.67:0/1946723975 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81cc158c60 con 0x7f81cc0b9690 2024-03-24T00:39:05.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.157+0000 7f81c2ffd640 1 -- 172.21.15.67:0/1946723975 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c8003a20 con 0x7f81cc0b9690 2024-03-24T00:39:05.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.157+0000 7f81d426c640 1 -- 172.21.15.67:0/1946723975 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81cc146150 con 0x7f81cc0b9690 2024-03-24T00:39:05.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.158+0000 7f81d426c640 1 -- 172.21.15.67:0/1946723975 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81cc04f400 con 0x7f81cc0b9690 2024-03-24T00:39:05.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.159+0000 7f81c2ffd640 1 -- 172.21.15.67:0/1946723975 <== mon.1 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f81c801a070 con 0x7f81cc0b9690 2024-03-24T00:39:05.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.159+0000 7f81c2ffd640 1 --2- 172.21.15.67:0/1946723975 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f81b006bbd0 0x7f81b006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.159+0000 7f81c2ffd640 1 -- 172.21.15.67:0/1946723975 <== mon.1 v2:172.21.15.72:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f81c808a050 con 0x7f81cc0b9690 2024-03-24T00:39:05.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.159+0000 7f81d1807640 1 --2- 172.21.15.67:0/1946723975 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f81b006bbd0 0x7f81b006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.160+0000 7f81d1807640 1 --2- 172.21.15.67:0/1946723975 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f81b006bbd0 0x7f81b006e090 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f81bc035000 tx=0x7f81bc008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:05.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.164+0000 7f81c2ffd640 1 -- 172.21.15.67:0/1946723975 <== mon.1 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f81c8058280 con 0x7f81cc0b9690 2024-03-24T00:39:05.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.346+0000 7f81d426c640 1 -- 172.21.15.67:0/1946723975 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f81cc0b9c90 con 0x7f81cc0b9690 2024-03-24T00:39:05.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.346+0000 7f81c2ffd640 1 -- 172.21.15.67:0/1946723975 <== mon.1 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7f81c805c320 con 0x7f81cc0b9690 2024-03-24T00:39:05.347 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-03-24T00:39:05.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.349+0000 7f81d426c640 1 -- 172.21.15.67:0/1946723975 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f81b006bbd0 msgr2=0x7f81b006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.349+0000 7f81d426c640 1 --2- 172.21.15.67:0/1946723975 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f81b006bbd0 0x7f81b006e090 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f81bc035000 tx=0x7f81bc008030 comp rx=0 tx=0).stop 2024-03-24T00:39:05.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.349+0000 7f81d426c640 1 -- 172.21.15.67:0/1946723975 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f81cc0b9690 msgr2=0x7f81cc148350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.349+0000 7f81d426c640 1 --2- 172.21.15.67:0/1946723975 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f81cc0b9690 0x7f81cc148350 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f81c800c410 tx=0x7f81c800c810 comp rx=0 tx=0).stop 2024-03-24T00:39:05.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.349+0000 7f81d426c640 1 -- 172.21.15.67:0/1946723975 shutdown_connections 2024-03-24T00:39:05.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.349+0000 7f81d426c640 1 --2- 172.21.15.67:0/1946723975 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f81b006bbd0 0x7f81b006e090 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.349+0000 7f81d426c640 1 --2- 172.21.15.67:0/1946723975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81cc161370 0x7f81cc15bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.349+0000 7f81d426c640 1 --2- 172.21.15.67:0/1946723975 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f81cc0cef30 0x7f81cc158160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.349+0000 7f81d426c640 1 --2- 172.21.15.67:0/1946723975 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f81cc0b9690 0x7f81cc148350 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.349+0000 7f81d426c640 1 -- 172.21.15.67:0/1946723975 >> 172.21.15.67:0/1946723975 conn(0x7f81cc06c880 msgr2=0x7f81cc0d3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:05.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.349+0000 7f81d426c640 1 -- 172.21.15.67:0/1946723975 shutdown_connections 2024-03-24T00:39:05.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.349+0000 7f81d426c640 1 -- 172.21.15.67:0/1946723975 wait complete. 2024-03-24T00:39:05.360 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T00:39:05.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.464+0000 7fc12160b640 1 Processor -- start 2024-03-24T00:39:05.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.464+0000 7fc12160b640 1 -- start start 2024-03-24T00:39:05.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.465+0000 7fc12160b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc11c0d22a0 0x7fc11c0b1f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.465+0000 7fc12160b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc11c0b12f0 con 0x7fc11c0b0f10 2024-03-24T00:39:05.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.465+0000 7fc12160b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc11c0b2c50 con 0x7fc11c0b2440 2024-03-24T00:39:05.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.465+0000 7fc12160b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc11c0d09f0 con 0x7fc11c0d22a0 2024-03-24T00:39:05.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.465+0000 7fc11affd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc11c0b0f10 0x7fc11c0d1b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45332/0 (socket says 172.21.15.67:45332) 2024-03-24T00:39:05.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.465+0000 7fc11affd640 1 -- 172.21.15.67:0/3369791344 learned_addr learned my addr 172.21.15.67:0/3369791344 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:05.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.465+0000 7fc11a7fc640 1 --2- 172.21.15.67:0/3369791344 >> v2:172.21.15.72:3300/0 conn(0x7fc11c0d22a0 0x7fc11c0b1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.466+0000 7fc119ffb640 1 -- 172.21.15.67:0/3369791344 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2419882825 0 0) 0x7fc11c0b12f0 con 0x7fc11c0b0f10 2024-03-24T00:39:05.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.466+0000 7fc119ffb640 1 -- 172.21.15.67:0/3369791344 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc108003520 con 0x7fc11c0b0f10 2024-03-24T00:39:05.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.466+0000 7fc119ffb640 1 -- 172.21.15.67:0/3369791344 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1134950096 0 0) 0x7fc11c0b2c50 con 0x7fc11c0b2440 2024-03-24T00:39:05.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.466+0000 7fc119ffb640 1 -- 172.21.15.67:0/3369791344 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc11c0b12f0 con 0x7fc11c0b2440 2024-03-24T00:39:05.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.466+0000 7fc11a7fc640 1 -- 172.21.15.67:0/3369791344 >> v1:172.21.15.72:6789/0 conn(0x7fc11c0b2440 legacy=0x7fc11c15d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.466+0000 7fc11a7fc640 1 -- 172.21.15.67:0/3369791344 >> v1:172.21.15.67:6789/0 conn(0x7fc11c0b0f10 legacy=0x7fc11c0d1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.466+0000 7fc11a7fc640 1 -- 172.21.15.67:0/3369791344 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc11c0ce880 con 0x7fc11c0d22a0 2024-03-24T00:39:05.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.466+0000 7fc11a7fc640 1 --2- 172.21.15.67:0/3369791344 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc11c0d22a0 0x7fc11c0b1f00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc1040097a0 tx=0x7fc10402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b18d27508dee5215 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:05.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.467+0000 7fc119ffb640 1 -- 172.21.15.67:0/3369791344 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc104034070 con 0x7fc11c0d22a0 2024-03-24T00:39:05.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.467+0000 7fc119ffb640 1 -- 172.21.15.67:0/3369791344 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc104030610 con 0x7fc11c0d22a0 2024-03-24T00:39:05.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.467+0000 7fc119ffb640 1 -- 172.21.15.67:0/3369791344 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1040309c0 con 0x7fc11c0d22a0 2024-03-24T00:39:05.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.467+0000 7fc12160b640 1 -- 172.21.15.67:0/3369791344 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc11c0d22a0 msgr2=0x7fc11c0b1f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.467+0000 7fc12160b640 1 --2- 172.21.15.67:0/3369791344 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc11c0d22a0 0x7fc11c0b1f00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc1040097a0 tx=0x7fc10402b960 comp rx=0 tx=0).stop 2024-03-24T00:39:05.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.467+0000 7fc12160b640 1 -- 172.21.15.67:0/3369791344 shutdown_connections 2024-03-24T00:39:05.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.467+0000 7fc12160b640 1 --2- 172.21.15.67:0/3369791344 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc11c0d22a0 0x7fc11c0b1f00 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.467+0000 7fc12160b640 1 -- 172.21.15.67:0/3369791344 >> 172.21.15.67:0/3369791344 conn(0x7fc11c08b1b0 msgr2=0x7fc11c08b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:05.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.467+0000 7fc12160b640 1 -- 172.21.15.67:0/3369791344 shutdown_connections 2024-03-24T00:39:05.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.467+0000 7fc12160b640 1 -- 172.21.15.67:0/3369791344 wait complete. 2024-03-24T00:39:05.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.468+0000 7fc12160b640 1 Processor -- start 2024-03-24T00:39:05.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.468+0000 7fc12160b640 1 -- start start 2024-03-24T00:39:05.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.469+0000 7fc12160b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc11c0b0f10 0x7fc11c0c72d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.469+0000 7fc12160b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc11c0b2440 0x7fc11c0c6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.469+0000 7fc12160b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc11c0d22a0 0x7fc11c14f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.469+0000 7fc12160b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc11c0ce560 con 0x7fc11c0b0f10 2024-03-24T00:39:05.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.469+0000 7fc12160b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc11c0cee40 con 0x7fc11c0d22a0 2024-03-24T00:39:05.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.469+0000 7fc12160b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc11c0cf360 con 0x7fc11c0b2440 2024-03-24T00:39:05.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.469+0000 7fc11affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc11c0b0f10 0x7fc11c0c72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.469+0000 7fc11affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc11c0b0f10 0x7fc11c0c72d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60586/0 (socket says 172.21.15.67:60586) 2024-03-24T00:39:05.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.469+0000 7fc11affd640 1 -- 172.21.15.67:0/2508929786 learned_addr learned my addr 172.21.15.67:0/2508929786 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:05.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.469+0000 7fc11b7fe640 1 --2- 172.21.15.67:0/2508929786 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc11c0d22a0 0x7fc11c14f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.469+0000 7fc11a7fc640 1 --2- 172.21.15.67:0/2508929786 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc11c0b2440 0x7fc11c0c6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.470+0000 7fc11affd640 1 -- 172.21.15.67:0/2508929786 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc11c0b2440 msgr2=0x7fc11c0c6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.470+0000 7fc11affd640 1 --2- 172.21.15.67:0/2508929786 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc11c0b2440 0x7fc11c0c6290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.470+0000 7fc11affd640 1 -- 172.21.15.67:0/2508929786 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc11c0d22a0 msgr2=0x7fc11c14f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.470+0000 7fc11affd640 1 --2- 172.21.15.67:0/2508929786 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc11c0d22a0 0x7fc11c14f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.470+0000 7fc11affd640 1 -- 172.21.15.67:0/2508929786 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc11c0c7960 con 0x7fc11c0b0f10 2024-03-24T00:39:05.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.470+0000 7fc11affd640 1 --2- 172.21.15.67:0/2508929786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc11c0b0f10 0x7fc11c0c72d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc11400c410 tx=0x7fc11400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:05.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.470+0000 7fc1137fe640 1 -- 172.21.15.67:0/2508929786 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc114017070 con 0x7fc11c0b0f10 2024-03-24T00:39:05.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.470+0000 7fc1137fe640 1 -- 172.21.15.67:0/2508929786 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc114005ce0 con 0x7fc11c0b0f10 2024-03-24T00:39:05.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.470+0000 7fc12160b640 1 -- 172.21.15.67:0/2508929786 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc11c0c8cd0 con 0x7fc11c0b0f10 2024-03-24T00:39:05.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.470+0000 7fc1137fe640 1 -- 172.21.15.67:0/2508929786 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1140072d0 con 0x7fc11c0b0f10 2024-03-24T00:39:05.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.470+0000 7fc12160b640 1 -- 172.21.15.67:0/2508929786 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc11c0c9290 con 0x7fc11c0b0f10 2024-03-24T00:39:05.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.471+0000 7fc12160b640 1 -- 172.21.15.67:0/2508929786 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc11c04f400 con 0x7fc11c0b0f10 2024-03-24T00:39:05.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.471+0000 7fc1137fe640 1 -- 172.21.15.67:0/2508929786 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc11401a070 con 0x7fc11c0b0f10 2024-03-24T00:39:05.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.472+0000 7fc1137fe640 1 --2- 172.21.15.67:0/2508929786 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc0fc06bbd0 0x7fc0fc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.472+0000 7fc1137fe640 1 -- 172.21.15.67:0/2508929786 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fc11408a0a0 con 0x7fc11c0b0f10 2024-03-24T00:39:05.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.472+0000 7fc11a7fc640 1 --2- 172.21.15.67:0/2508929786 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc0fc06bbd0 0x7fc0fc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.472+0000 7fc11a7fc640 1 --2- 172.21.15.67:0/2508929786 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc0fc06bbd0 0x7fc0fc06e090 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc104036000 tx=0x7fc10403b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:05.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.476+0000 7fc1137fe640 1 -- 172.21.15.67:0/2508929786 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc114058200 con 0x7fc11c0b0f10 2024-03-24T00:39:05.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.658+0000 7fc12160b640 1 -- 172.21.15.67:0/2508929786 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fc11c0b21f0 con 0x7fc11c0b0f10 2024-03-24T00:39:05.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.658+0000 7fc1137fe640 1 -- 172.21.15.67:0/2508929786 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7fc11405c2a0 con 0x7fc11c0b0f10 2024-03-24T00:39:05.659 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-24T00:39:05.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.660+0000 7fc12160b640 1 -- 172.21.15.67:0/2508929786 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc0fc06bbd0 msgr2=0x7fc0fc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.660+0000 7fc12160b640 1 --2- 172.21.15.67:0/2508929786 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc0fc06bbd0 0x7fc0fc06e090 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc104036000 tx=0x7fc10403b040 comp rx=0 tx=0).stop 2024-03-24T00:39:05.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.660+0000 7fc12160b640 1 -- 172.21.15.67:0/2508929786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc11c0b0f10 msgr2=0x7fc11c0c72d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.660+0000 7fc12160b640 1 --2- 172.21.15.67:0/2508929786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc11c0b0f10 0x7fc11c0c72d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc11400c410 tx=0x7fc11400c8d0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.660+0000 7fc12160b640 1 -- 172.21.15.67:0/2508929786 shutdown_connections 2024-03-24T00:39:05.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.660+0000 7fc12160b640 1 --2- 172.21.15.67:0/2508929786 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc0fc06bbd0 0x7fc0fc06e090 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.660+0000 7fc12160b640 1 --2- 172.21.15.67:0/2508929786 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc11c0d22a0 0x7fc11c14f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.660+0000 7fc12160b640 1 --2- 172.21.15.67:0/2508929786 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc11c0b2440 0x7fc11c0c6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.660+0000 7fc12160b640 1 --2- 172.21.15.67:0/2508929786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc11c0b0f10 0x7fc11c0c72d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.660+0000 7fc12160b640 1 -- 172.21.15.67:0/2508929786 >> 172.21.15.67:0/2508929786 conn(0x7fc11c08b1b0 msgr2=0x7fc11c0ae7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:05.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.660+0000 7fc12160b640 1 -- 172.21.15.67:0/2508929786 shutdown_connections 2024-03-24T00:39:05.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.660+0000 7fc12160b640 1 -- 172.21.15.67:0/2508929786 wait complete. 2024-03-24T00:39:05.672 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T00:39:05.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.773+0000 7f046ec9a640 1 Processor -- start 2024-03-24T00:39:05.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.773+0000 7f046ec9a640 1 -- start start 2024-03-24T00:39:05.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.774+0000 7f046ec9a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f046815e0a0 0x7f04680cd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.774+0000 7f046ec9a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f04680cdc30 0x7f046806b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.774+0000 7f046ec9a640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04680cb0f0 con 0x7f046806b580 2024-03-24T00:39:05.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.774+0000 7f046ec9a640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f046806b9d0 con 0x7f046815e0a0 2024-03-24T00:39:05.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.774+0000 7f046ec9a640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04680afd80 con 0x7f04680cdc30 2024-03-24T00:39:05.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.774+0000 7f046ca36640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f046815e0a0 0x7f04680cd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.774+0000 7f046d237640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f046806b580 0x7f04680d0a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45346/0 (socket says 172.21.15.67:45346) 2024-03-24T00:39:05.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.774+0000 7f046d237640 1 -- 172.21.15.67:0/2701391413 learned_addr learned my addr 172.21.15.67:0/2701391413 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:05.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.774+0000 7f0467fff640 1 --2- 172.21.15.67:0/2701391413 >> v2:172.21.15.72:3300/0 conn(0x7f04680cdc30 0x7f046806b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.775+0000 7f046ca36640 1 -- 172.21.15.67:0/2701391413 >> v2:172.21.15.72:3300/0 conn(0x7f04680cdc30 msgr2=0x7f046806b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:05.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.775+0000 7f046ca36640 1 --2- 172.21.15.67:0/2701391413 >> v2:172.21.15.72:3300/0 conn(0x7f04680cdc30 0x7f046806b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.775+0000 7f046ca36640 1 -- 172.21.15.67:0/2701391413 >> v1:172.21.15.67:6789/0 conn(0x7f046806b580 legacy=0x7f04680d0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.775+0000 7f046ca36640 1 -- 172.21.15.67:0/2701391413 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04680d0080 con 0x7f046815e0a0 2024-03-24T00:39:05.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.775+0000 7f04677fe640 1 -- 172.21.15.67:0/2701391413 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3075563005 0 0) 0x7f04680cb0f0 con 0x7f046806b580 2024-03-24T00:39:05.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.775+0000 7f0467fff640 1 --2- 172.21.15.67:0/2701391413 >> v2:172.21.15.72:3300/0 conn(0x7f04680cdc30 0x7f046806b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:05.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.775+0000 7f046ca36640 1 --2- 172.21.15.67:0/2701391413 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f046815e0a0 0x7f04680cd6f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f046806e480 tx=0x7f045802b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1676c7154635e569 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:05.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.775+0000 7f04677fe640 1 -- 172.21.15.67:0/2701391413 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0458038070 con 0x7f046815e0a0 2024-03-24T00:39:05.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.775+0000 7f04677fe640 1 -- 172.21.15.67:0/2701391413 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0458031710 con 0x7f046815e0a0 2024-03-24T00:39:05.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.775+0000 7f04677fe640 1 -- 172.21.15.67:0/2701391413 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0458031ae0 con 0x7f046815e0a0 2024-03-24T00:39:05.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.776+0000 7f046ec9a640 1 -- 172.21.15.67:0/2701391413 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f046815e0a0 msgr2=0x7f04680cd6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.776+0000 7f046ec9a640 1 --2- 172.21.15.67:0/2701391413 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f046815e0a0 0x7f04680cd6f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f046806e480 tx=0x7f045802b7b0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.776+0000 7f046ec9a640 1 -- 172.21.15.67:0/2701391413 shutdown_connections 2024-03-24T00:39:05.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.776+0000 7f046ec9a640 1 --2- 172.21.15.67:0/2701391413 >> v2:172.21.15.72:3300/0 conn(0x7f04680cdc30 0x7f046806b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.776+0000 7f046ec9a640 1 --2- 172.21.15.67:0/2701391413 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f046815e0a0 0x7f04680cd6f0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.776+0000 7f046ec9a640 1 -- 172.21.15.67:0/2701391413 >> 172.21.15.67:0/2701391413 conn(0x7f0468099a30 msgr2=0x7f0468099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:05.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.776+0000 7f046ec9a640 1 -- 172.21.15.67:0/2701391413 shutdown_connections 2024-03-24T00:39:05.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.776+0000 7f046ec9a640 1 -- 172.21.15.67:0/2701391413 wait complete. 2024-03-24T00:39:05.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.777+0000 7f046ec9a640 1 Processor -- start 2024-03-24T00:39:05.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.777+0000 7f046ec9a640 1 -- start start 2024-03-24T00:39:05.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.778+0000 7f046ec9a640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f046806b580 0x7f04681f81d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.778+0000 7f046ec9a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04680cdc30 0x7f046815b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.778+0000 7f046ec9a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f046815e0a0 0x7f046815bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.778+0000 7f046ec9a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04680b1220 con 0x7f04680cdc30 2024-03-24T00:39:05.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.778+0000 7f046ec9a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04680d0640 con 0x7f046815e0a0 2024-03-24T00:39:05.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.778+0000 7f046ec9a640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04680ce350 con 0x7f046806b580 2024-03-24T00:39:05.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.778+0000 7f0467fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04680cdc30 0x7f046815b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.778+0000 7f0467fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04680cdc30 0x7f046815b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60594/0 (socket says 172.21.15.67:60594) 2024-03-24T00:39:05.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.778+0000 7f0467fff640 1 -- 172.21.15.67:0/2039934438 learned_addr learned my addr 172.21.15.67:0/2039934438 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:05.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.778+0000 7f046ca36640 1 --2- 172.21.15.67:0/2039934438 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f046806b580 0x7f04681f81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.778+0000 7f046d237640 1 --2- 172.21.15.67:0/2039934438 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f046815e0a0 0x7f046815bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.778+0000 7f0467fff640 1 -- 172.21.15.67:0/2039934438 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f046806b580 msgr2=0x7f04681f81d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.778+0000 7f0467fff640 1 --2- 172.21.15.67:0/2039934438 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f046806b580 0x7f04681f81d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.778+0000 7f0467fff640 1 -- 172.21.15.67:0/2039934438 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f046815e0a0 msgr2=0x7f046815bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.778+0000 7f0467fff640 1 --2- 172.21.15.67:0/2039934438 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f046815e0a0 0x7f046815bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.778+0000 7f0467fff640 1 -- 172.21.15.67:0/2039934438 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04681f4050 con 0x7f04680cdc30 2024-03-24T00:39:05.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.779+0000 7f0467fff640 1 --2- 172.21.15.67:0/2039934438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04680cdc30 0x7f046815b9f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f045400e900 tx=0x7f045400edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:05.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.779+0000 7f04657fa640 1 -- 172.21.15.67:0/2039934438 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0454017070 con 0x7f04680cdc30 2024-03-24T00:39:05.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.779+0000 7f04657fa640 1 -- 172.21.15.67:0/2039934438 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f045400ca00 con 0x7f04680cdc30 2024-03-24T00:39:05.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.779+0000 7f046ec9a640 1 -- 172.21.15.67:0/2039934438 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04680ca460 con 0x7f04680cdc30 2024-03-24T00:39:05.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.779+0000 7f04657fa640 1 -- 172.21.15.67:0/2039934438 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f045400cdf0 con 0x7f04680cdc30 2024-03-24T00:39:05.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.779+0000 7f046ec9a640 1 -- 172.21.15.67:0/2039934438 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04681f79d0 con 0x7f04680cdc30 2024-03-24T00:39:05.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.780+0000 7f046ec9a640 1 -- 172.21.15.67:0/2039934438 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f046804f400 con 0x7f04680cdc30 2024-03-24T00:39:05.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.780+0000 7f04657fa640 1 -- 172.21.15.67:0/2039934438 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f045401a070 con 0x7f04680cdc30 2024-03-24T00:39:05.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.781+0000 7f04657fa640 1 --2- 172.21.15.67:0/2039934438 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f045006bbd0 0x7f045006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.781+0000 7f04657fa640 1 -- 172.21.15.67:0/2039934438 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f045408a850 con 0x7f04680cdc30 2024-03-24T00:39:05.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.781+0000 7f046ca36640 1 --2- 172.21.15.67:0/2039934438 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f045006bbd0 0x7f045006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.781+0000 7f046ca36640 1 --2- 172.21.15.67:0/2039934438 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f045006bbd0 0x7f045006e090 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0458035000 tx=0x7f0458008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:05.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.785+0000 7f04657fa640 1 -- 172.21.15.67:0/2039934438 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f04540589b0 con 0x7f04680cdc30 2024-03-24T00:39:05.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.979+0000 7f046ec9a640 1 -- 172.21.15.67:0/2039934438 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f04680afb60 con 0x7f04680cdc30 2024-03-24T00:39:05.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.979+0000 7f04657fa640 1 -- 172.21.15.67:0/2039934438 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v25) v1 ==== 80+0+0 (secure 0 0 0) 0x7f045405ca50 con 0x7f04680cdc30 2024-03-24T00:39:05.980 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-03-24T00:39:05.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.981+0000 7f046ec9a640 1 -- 172.21.15.67:0/2039934438 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f045006bbd0 msgr2=0x7f045006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.981+0000 7f046ec9a640 1 --2- 172.21.15.67:0/2039934438 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f045006bbd0 0x7f045006e090 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0458035000 tx=0x7f0458008030 comp rx=0 tx=0).stop 2024-03-24T00:39:05.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.981+0000 7f046ec9a640 1 -- 172.21.15.67:0/2039934438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04680cdc30 msgr2=0x7f046815b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.981+0000 7f046ec9a640 1 --2- 172.21.15.67:0/2039934438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04680cdc30 0x7f046815b9f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f045400e900 tx=0x7f045400edc0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.981+0000 7f046ec9a640 1 -- 172.21.15.67:0/2039934438 shutdown_connections 2024-03-24T00:39:05.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.981+0000 7f046ec9a640 1 --2- 172.21.15.67:0/2039934438 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f045006bbd0 0x7f045006e090 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.981+0000 7f046ec9a640 1 --2- 172.21.15.67:0/2039934438 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f046815e0a0 0x7f046815bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.981+0000 7f046ec9a640 1 --2- 172.21.15.67:0/2039934438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04680cdc30 0x7f046815b9f0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.981+0000 7f046ec9a640 1 --2- 172.21.15.67:0/2039934438 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f046806b580 0x7f04681f81d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.981+0000 7f046ec9a640 1 -- 172.21.15.67:0/2039934438 >> 172.21.15.67:0/2039934438 conn(0x7f0468099a30 msgr2=0x7f0468099e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:05.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.981+0000 7f046ec9a640 1 -- 172.21.15.67:0/2039934438 shutdown_connections 2024-03-24T00:39:05.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:05.981+0000 7f046ec9a640 1 -- 172.21.15.67:0/2039934438 wait complete. 2024-03-24T00:39:05.992 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi067.front.sepia.ceph.com... 2024-03-24T00:39:05.992 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:05.992 DEBUG:teuthology.orchestra.run.smithi067:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-03-24T00:39:06.017 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi067.stderr:ceph-fuse[44841]: fuse finished with error 0 and tester_r 0 2024-03-24T00:39:06.019 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:39:06.043 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-03-24T00:39:06.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:06.043 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-03-24T00:39:06.110 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:39:06.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.250+0000 7f838da4b640 1 Processor -- start 2024-03-24T00:39:06.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.250+0000 7f838da4b640 1 -- start start 2024-03-24T00:39:06.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.251+0000 7f838da4b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f838815ae70 0x7f83880b01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:06.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.251+0000 7f838da4b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83880b24f0 con 0x7f83880cf240 2024-03-24T00:39:06.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.251+0000 7f838da4b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83880b1730 con 0x7f83880d0b90 2024-03-24T00:39:06.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.251+0000 7f838da4b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83880b1a10 con 0x7f838815ae70 2024-03-24T00:39:06.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.251+0000 7f83877fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f83880cf240 0x7f83880ad8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45360/0 (socket says 172.21.15.67:45360) 2024-03-24T00:39:06.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.251+0000 7f83877fe640 1 -- 172.21.15.67:0/2815094424 learned_addr learned my addr 172.21.15.67:0/2815094424 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:06.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.251+0000 7f8386ffd640 1 --1- 172.21.15.67:0/2815094424 >> v1:172.21.15.72:6789/0 conn(0x7f83880d0b90 0x7f83880d0f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:51234/0 (socket says 172.21.15.67:51234) 2024-03-24T00:39:06.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.251+0000 7f83867fc640 1 --2- 172.21.15.67:0/2815094424 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f838815ae70 0x7f83880b01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:06.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.252+0000 7f8385ffb640 1 -- 172.21.15.67:0/2815094424 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 594932437 0 0) 0x7f83880b1730 con 0x7f83880d0b90 2024-03-24T00:39:06.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.252+0000 7f8385ffb640 1 -- 172.21.15.67:0/2815094424 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8374003520 con 0x7f83880d0b90 2024-03-24T00:39:06.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.252+0000 7f8385ffb640 1 -- 172.21.15.67:0/2815094424 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2374322595 0 0) 0x7f83880b24f0 con 0x7f83880cf240 2024-03-24T00:39:06.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.252+0000 7f8385ffb640 1 -- 172.21.15.67:0/2815094424 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83880b1730 con 0x7f83880cf240 2024-03-24T00:39:06.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.252+0000 7f83867fc640 1 -- 172.21.15.67:0/2815094424 >> v1:172.21.15.72:6789/0 conn(0x7f83880d0b90 legacy=0x7f83880d0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:06.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.252+0000 7f83867fc640 1 -- 172.21.15.67:0/2815094424 >> v1:172.21.15.67:6789/0 conn(0x7f83880cf240 legacy=0x7f83880ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:06.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.252+0000 7f83867fc640 1 -- 172.21.15.67:0/2815094424 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83880b1cf0 con 0x7f838815ae70 2024-03-24T00:39:06.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.253+0000 7f83867fc640 1 --2- 172.21.15.67:0/2815094424 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f838815ae70 0x7f83880b01d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f83700097d0 tx=0x7f837002b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10eeba890e76599f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:06.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.253+0000 7f8385ffb640 1 -- 172.21.15.67:0/2815094424 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8370035020 con 0x7f838815ae70 2024-03-24T00:39:06.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.253+0000 7f8385ffb640 1 -- 172.21.15.67:0/2815094424 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8370031890 con 0x7f838815ae70 2024-03-24T00:39:06.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.253+0000 7f8385ffb640 1 -- 172.21.15.67:0/2815094424 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8370031c40 con 0x7f838815ae70 2024-03-24T00:39:06.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.253+0000 7f838da4b640 1 -- 172.21.15.67:0/2815094424 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f838815ae70 msgr2=0x7f83880b01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:06.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.253+0000 7f838da4b640 1 --2- 172.21.15.67:0/2815094424 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f838815ae70 0x7f83880b01d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f83700097d0 tx=0x7f837002b930 comp rx=0 tx=0).stop 2024-03-24T00:39:06.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.253+0000 7f838da4b640 1 -- 172.21.15.67:0/2815094424 shutdown_connections 2024-03-24T00:39:06.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.253+0000 7f838da4b640 1 --2- 172.21.15.67:0/2815094424 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f838815ae70 0x7f83880b01d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:06.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.253+0000 7f838da4b640 1 -- 172.21.15.67:0/2815094424 >> 172.21.15.67:0/2815094424 conn(0x7f8388099a30 msgr2=0x7f8388099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:06.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.253+0000 7f838da4b640 1 -- 172.21.15.67:0/2815094424 shutdown_connections 2024-03-24T00:39:06.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.253+0000 7f838da4b640 1 -- 172.21.15.67:0/2815094424 wait complete. 2024-03-24T00:39:06.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.254+0000 7f838da4b640 1 Processor -- start 2024-03-24T00:39:06.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.255+0000 7f838da4b640 1 -- start start 2024-03-24T00:39:06.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.255+0000 7f838da4b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f83880cf240 0x7f8388159a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:06.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.255+0000 7f838da4b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83880d0b90 0x7f83880b3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:06.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.255+0000 7f838da4b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f838815ae70 0x7f83881e6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:06.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.255+0000 7f838da4b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83880cf9e0 con 0x7f83880d0b90 2024-03-24T00:39:06.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.255+0000 7f838da4b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83880cf6d0 con 0x7f838815ae70 2024-03-24T00:39:06.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.255+0000 7f838da4b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83880adfe0 con 0x7f83880cf240 2024-03-24T00:39:06.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.255+0000 7f83867fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83880d0b90 0x7f83880b3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:06.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.255+0000 7f83877fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f838815ae70 0x7f83881e6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:06.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.255+0000 7f8386ffd640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f83880cf240 0x7f8388159a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:06.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.256+0000 7f83877fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f838815ae70 0x7f83881e6220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:33866/0 (socket says 172.21.15.67:33866) 2024-03-24T00:39:06.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.256+0000 7f83877fe640 1 -- 172.21.15.67:0/4271657465 learned_addr learned my addr 172.21.15.67:0/4271657465 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:06.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.256+0000 7f8386ffd640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f83880cf240 0x7f8388159a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.67:48798/0 (socket says 172.21.15.67:48798) 2024-03-24T00:39:06.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.256+0000 7f83867fc640 1 -- 172.21.15.67:0/4271657465 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f83880cf240 msgr2=0x7f8388159a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:06.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.256+0000 7f83867fc640 1 --2- 172.21.15.67:0/4271657465 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f83880cf240 0x7f8388159a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:06.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.256+0000 7f83867fc640 1 -- 172.21.15.67:0/4271657465 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f838815ae70 msgr2=0x7f83881e6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:06.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.256+0000 7f83867fc640 1 --2- 172.21.15.67:0/4271657465 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f838815ae70 0x7f83881e6220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:06.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.256+0000 7f83867fc640 1 -- 172.21.15.67:0/4271657465 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83880b4260 con 0x7f83880d0b90 2024-03-24T00:39:06.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.256+0000 7f8386ffd640 1 --2- 172.21.15.67:0/4271657465 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f83880cf240 0x7f8388159a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:06.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.256+0000 7f83877fe640 1 --2- 172.21.15.67:0/4271657465 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f838815ae70 0x7f83881e6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:06.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.256+0000 7f83867fc640 1 --2- 172.21.15.67:0/4271657465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83880d0b90 0x7f83880b3880 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f83880727b0 tx=0x7f83700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:06.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.256+0000 7f837ffff640 1 -- 172.21.15.67:0/4271657465 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8370035040 con 0x7f83880d0b90 2024-03-24T00:39:06.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.256+0000 7f837ffff640 1 -- 172.21.15.67:0/4271657465 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8370030d20 con 0x7f83880d0b90 2024-03-24T00:39:06.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.256+0000 7f838da4b640 1 -- 172.21.15.67:0/4271657465 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83880b4540 con 0x7f83880d0b90 2024-03-24T00:39:06.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.256+0000 7f837ffff640 1 -- 172.21.15.67:0/4271657465 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8370044670 con 0x7f83880d0b90 2024-03-24T00:39:06.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.256+0000 7f838da4b640 1 -- 172.21.15.67:0/4271657465 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83880b4b00 con 0x7f83880d0b90 2024-03-24T00:39:06.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.257+0000 7f838da4b640 1 -- 172.21.15.67:0/4271657465 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f838804f400 con 0x7f83880d0b90 2024-03-24T00:39:06.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.258+0000 7f837ffff640 1 -- 172.21.15.67:0/4271657465 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8370038020 con 0x7f83880d0b90 2024-03-24T00:39:06.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.258+0000 7f837ffff640 1 --2- 172.21.15.67:0/4271657465 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f836806bbd0 0x7f836806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:06.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.258+0000 7f837ffff640 1 -- 172.21.15.67:0/4271657465 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f83700aca40 con 0x7f83880d0b90 2024-03-24T00:39:06.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.258+0000 7f8386ffd640 1 --2- 172.21.15.67:0/4271657465 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f836806bbd0 0x7f836806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:06.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.259+0000 7f8386ffd640 1 --2- 172.21.15.67:0/4271657465 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f836806bbd0 0x7f836806e090 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8380007fd0 tx=0x7f8380007cd0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:06.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.263+0000 7f837ffff640 1 -- 172.21.15.67:0/4271657465 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f837002d040 con 0x7f83880d0b90 2024-03-24T00:39:06.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.445+0000 7f838da4b640 1 -- 172.21.15.67:0/4271657465 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8388072f30 con 0x7f83880d0b90 2024-03-24T00:39:06.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.446+0000 7f837ffff640 1 -- 172.21.15.67:0/4271657465 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18454 (secure 0 0 0) 0x7f83700aae20 con 0x7f83880d0b90 2024-03-24T00:39:06.447 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:39:06.447 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":25,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:38:58.797179+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:38:50.077085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:38:51.095454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T00:38:54.807117+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:39:06.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.448+0000 7f838da4b640 1 -- 172.21.15.67:0/4271657465 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f836806bbd0 msgr2=0x7f836806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:06.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.448+0000 7f838da4b640 1 --2- 172.21.15.67:0/4271657465 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f836806bbd0 0x7f836806e090 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8380007fd0 tx=0x7f8380007cd0 comp rx=0 tx=0).stop 2024-03-24T00:39:06.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.448+0000 7f838da4b640 1 -- 172.21.15.67:0/4271657465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83880d0b90 msgr2=0x7f83880b3880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:06.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.448+0000 7f838da4b640 1 --2- 172.21.15.67:0/4271657465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83880d0b90 0x7f83880b3880 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f83880727b0 tx=0x7f83700046c0 comp rx=0 tx=0).stop 2024-03-24T00:39:06.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.448+0000 7f838da4b640 1 -- 172.21.15.67:0/4271657465 shutdown_connections 2024-03-24T00:39:06.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.448+0000 7f838da4b640 1 --2- 172.21.15.67:0/4271657465 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f836806bbd0 0x7f836806e090 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:06.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.448+0000 7f838da4b640 1 --2- 172.21.15.67:0/4271657465 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f838815ae70 0x7f83881e6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:06.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.448+0000 7f838da4b640 1 --2- 172.21.15.67:0/4271657465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83880d0b90 0x7f83880b3880 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:06.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.448+0000 7f838da4b640 1 --2- 172.21.15.67:0/4271657465 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f83880cf240 0x7f8388159a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:06.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.448+0000 7f838da4b640 1 -- 172.21.15.67:0/4271657465 >> 172.21.15.67:0/4271657465 conn(0x7f8388099a30 msgr2=0x7f838806ed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:06.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.448+0000 7f838da4b640 1 -- 172.21.15.67:0/4271657465 shutdown_connections 2024-03-24T00:39:06.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.448+0000 7f838da4b640 1 -- 172.21.15.67:0/4271657465 wait complete. 2024-03-24T00:39:06.460 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:39:06.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.561+0000 7fd7c1ca1640 1 Processor -- start 2024-03-24T00:39:06.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.561+0000 7fd7c1ca1640 1 -- start start 2024-03-24T00:39:06.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.562+0000 7fd7c1ca1640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd7bc0cf220 0x7fd7bc0d2660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:06.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.562+0000 7fd7c1ca1640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd7bc161bb0 0x7fd7bc0d2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:06.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.562+0000 7fd7c1ca1640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7bc162120 con 0x7fd7bc15dfe0 2024-03-24T00:39:06.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.562+0000 7fd7c1ca1640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7bc0d21d0 con 0x7fd7bc0cf220 2024-03-24T00:39:06.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.562+0000 7fd7c1ca1640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7bc162400 con 0x7fd7bc161bb0 2024-03-24T00:39:06.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.562+0000 7fd7baffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd7bc0cf220 0x7fd7bc0d2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:06.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.562+0000 7fd7baffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd7bc0cf220 0x7fd7bc0d2660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60622/0 (socket says 172.21.15.67:60622) 2024-03-24T00:39:06.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.562+0000 7fd7baffd640 1 -- 172.21.15.67:0/921599733 learned_addr learned my addr 172.21.15.67:0/921599733 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:06.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.562+0000 7fd7bbfff640 1 --2- 172.21.15.67:0/921599733 >> v2:172.21.15.72:3300/0 conn(0x7fd7bc161bb0 0x7fd7bc0d2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:06.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.562+0000 7fd7ba7fc640 1 -- 172.21.15.67:0/921599733 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 685608707 0 0) 0x7fd7bc162120 con 0x7fd7bc15dfe0 2024-03-24T00:39:06.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.562+0000 7fd7baffd640 1 -- 172.21.15.67:0/921599733 >> v2:172.21.15.72:3300/0 conn(0x7fd7bc161bb0 msgr2=0x7fd7bc0d2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:06.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.562+0000 7fd7baffd640 1 --2- 172.21.15.67:0/921599733 >> v2:172.21.15.72:3300/0 conn(0x7fd7bc161bb0 0x7fd7bc0d2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:06.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.562+0000 7fd7baffd640 1 -- 172.21.15.67:0/921599733 >> v1:172.21.15.72:6789/0 conn(0x7fd7bc15dfe0 legacy=0x7fd7bc0ceb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:06.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.562+0000 7fd7baffd640 1 -- 172.21.15.67:0/921599733 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7bc1626e0 con 0x7fd7bc0cf220 2024-03-24T00:39:06.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.563+0000 7fd7baffd640 1 --2- 172.21.15.67:0/921599733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7bc0cf220 0x7fd7bc0d2660 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc0b9e90 tx=0x7fd7b002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8c184106e89349 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:06.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.563+0000 7fd7ba7fc640 1 -- 172.21.15.67:0/921599733 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b0034070 con 0x7fd7bc0cf220 2024-03-24T00:39:06.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.563+0000 7fd7ba7fc640 1 -- 172.21.15.67:0/921599733 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7b0030860 con 0x7fd7bc0cf220 2024-03-24T00:39:06.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.563+0000 7fd7ba7fc640 1 -- 172.21.15.67:0/921599733 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b0030c30 con 0x7fd7bc0cf220 2024-03-24T00:39:06.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.564+0000 7fd7c1ca1640 1 -- 172.21.15.67:0/921599733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7bc0cf220 msgr2=0x7fd7bc0d2660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:06.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.564+0000 7fd7c1ca1640 1 --2- 172.21.15.67:0/921599733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7bc0cf220 0x7fd7bc0d2660 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc0b9e90 tx=0x7fd7b002b9a0 comp rx=0 tx=0).stop 2024-03-24T00:39:06.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.564+0000 7fd7c1ca1640 1 -- 172.21.15.67:0/921599733 shutdown_connections 2024-03-24T00:39:06.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.564+0000 7fd7c1ca1640 1 --2- 172.21.15.67:0/921599733 >> v2:172.21.15.72:3300/0 conn(0x7fd7bc161bb0 0x7fd7bc0d2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:06.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.564+0000 7fd7c1ca1640 1 --2- 172.21.15.67:0/921599733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7bc0cf220 0x7fd7bc0d2660 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:06.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.564+0000 7fd7c1ca1640 1 -- 172.21.15.67:0/921599733 >> 172.21.15.67:0/921599733 conn(0x7fd7bc06c890 msgr2=0x7fd7bc06cc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:06.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.564+0000 7fd7c1ca1640 1 -- 172.21.15.67:0/921599733 shutdown_connections 2024-03-24T00:39:06.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.564+0000 7fd7c1ca1640 1 -- 172.21.15.67:0/921599733 wait complete. 2024-03-24T00:39:06.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.565+0000 7fd7c1ca1640 1 Processor -- start 2024-03-24T00:39:06.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.565+0000 7fd7c1ca1640 1 -- start start 2024-03-24T00:39:06.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.566+0000 7fd7c1ca1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7bc0cf220 0x7fd7bc0b2230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:06.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.566+0000 7fd7c1ca1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd7bc15dfe0 0x7fd7bc1f4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:06.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.566+0000 7fd7c1ca1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd7bc161bb0 0x7fd7bc0cb930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:06.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.566+0000 7fd7c1ca1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7bc0d1120 con 0x7fd7bc0cf220 2024-03-24T00:39:06.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.566+0000 7fd7c1ca1640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7bc15e3c0 con 0x7fd7bc161bb0 2024-03-24T00:39:06.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.566+0000 7fd7c1ca1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7bc0d12a0 con 0x7fd7bc15dfe0 2024-03-24T00:39:06.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.566+0000 7fd7bb7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7bc0cf220 0x7fd7bc0b2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:06.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.566+0000 7fd7bb7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7bc0cf220 0x7fd7bc0b2230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60632/0 (socket says 172.21.15.67:60632) 2024-03-24T00:39:06.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.566+0000 7fd7bb7fe640 1 -- 172.21.15.67:0/1454084703 learned_addr learned my addr 172.21.15.67:0/1454084703 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:06.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.566+0000 7fd7bbfff640 1 --2- 172.21.15.67:0/1454084703 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd7bc161bb0 0x7fd7bc0cb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:06.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.566+0000 7fd7baffd640 1 --2- 172.21.15.67:0/1454084703 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd7bc15dfe0 0x7fd7bc1f4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:06.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.566+0000 7fd7bb7fe640 1 -- 172.21.15.67:0/1454084703 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd7bc15dfe0 msgr2=0x7fd7bc1f4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:06.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.566+0000 7fd7bb7fe640 1 --2- 172.21.15.67:0/1454084703 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd7bc15dfe0 0x7fd7bc1f4560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:06.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.566+0000 7fd7bb7fe640 1 -- 172.21.15.67:0/1454084703 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd7bc161bb0 msgr2=0x7fd7bc0cb930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:06.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.566+0000 7fd7bb7fe640 1 --2- 172.21.15.67:0/1454084703 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd7bc161bb0 0x7fd7bc0cb930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:06.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.566+0000 7fd7bb7fe640 1 -- 172.21.15.67:0/1454084703 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7bc1f5010 con 0x7fd7bc0cf220 2024-03-24T00:39:06.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.566+0000 7fd7bb7fe640 1 --2- 172.21.15.67:0/1454084703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7bc0cf220 0x7fd7bc0b2230 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac00c4e0 tx=0x7fd7ac00c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:06.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.567+0000 7fd7a3fff640 1 -- 172.21.15.67:0/1454084703 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7ac017070 con 0x7fd7bc0cf220 2024-03-24T00:39:06.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.567+0000 7fd7a3fff640 1 -- 172.21.15.67:0/1454084703 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7ac0035f0 con 0x7fd7bc0cf220 2024-03-24T00:39:06.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.567+0000 7fd7a3fff640 1 -- 172.21.15.67:0/1454084703 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7ac0039e0 con 0x7fd7bc0cf220 2024-03-24T00:39:06.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.567+0000 7fd7c1ca1640 1 -- 172.21.15.67:0/1454084703 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7bc1f52d0 con 0x7fd7bc0cf220 2024-03-24T00:39:06.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.567+0000 7fd7c1ca1640 1 -- 172.21.15.67:0/1454084703 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7bc1f5890 con 0x7fd7bc0cf220 2024-03-24T00:39:06.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.568+0000 7fd7c1ca1640 1 -- 172.21.15.67:0/1454084703 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7bc06b180 con 0x7fd7bc0cf220 2024-03-24T00:39:06.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.568+0000 7fd7a3fff640 1 -- 172.21.15.67:0/1454084703 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd7ac01a070 con 0x7fd7bc0cf220 2024-03-24T00:39:06.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.569+0000 7fd7a3fff640 1 --2- 172.21.15.67:0/1454084703 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd7a806bbd0 0x7fd7a806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:06.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.569+0000 7fd7a3fff640 1 -- 172.21.15.67:0/1454084703 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fd7ac08a070 con 0x7fd7bc0cf220 2024-03-24T00:39:06.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.569+0000 7fd7baffd640 1 --2- 172.21.15.67:0/1454084703 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd7a806bbd0 0x7fd7a806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:06.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.569+0000 7fd7baffd640 1 --2- 172.21.15.67:0/1454084703 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd7a806bbd0 0x7fd7a806e090 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd7b0036000 tx=0x7fd7b0008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:06.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.573+0000 7fd7a3fff640 1 -- 172.21.15.67:0/1454084703 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd7ac0581d0 con 0x7fd7bc0cf220 2024-03-24T00:39:06.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.755+0000 7fd7c1ca1640 1 -- 172.21.15.67:0/1454084703 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd7bc04f400 con 0x7fd7bc0cf220 2024-03-24T00:39:06.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.756+0000 7fd7a3fff640 1 -- 172.21.15.67:0/1454084703 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18454 (secure 0 0 0) 0x7fd7ac05c270 con 0x7fd7bc0cf220 2024-03-24T00:39:06.757 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:39:06.757 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":25,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:38:58.797179+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:38:50.077085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:38:51.095454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T00:38:54.807117+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:39:06.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.758+0000 7fd7c1ca1640 1 -- 172.21.15.67:0/1454084703 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd7a806bbd0 msgr2=0x7fd7a806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:06.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.758+0000 7fd7c1ca1640 1 --2- 172.21.15.67:0/1454084703 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd7a806bbd0 0x7fd7a806e090 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd7b0036000 tx=0x7fd7b0008030 comp rx=0 tx=0).stop 2024-03-24T00:39:06.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.758+0000 7fd7c1ca1640 1 -- 172.21.15.67:0/1454084703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7bc0cf220 msgr2=0x7fd7bc0b2230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:06.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.758+0000 7fd7c1ca1640 1 --2- 172.21.15.67:0/1454084703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7bc0cf220 0x7fd7bc0b2230 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac00c4e0 tx=0x7fd7ac00c9a0 comp rx=0 tx=0).stop 2024-03-24T00:39:06.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.758+0000 7fd7c1ca1640 1 -- 172.21.15.67:0/1454084703 shutdown_connections 2024-03-24T00:39:06.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.758+0000 7fd7c1ca1640 1 --2- 172.21.15.67:0/1454084703 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd7a806bbd0 0x7fd7a806e090 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:06.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.758+0000 7fd7c1ca1640 1 --2- 172.21.15.67:0/1454084703 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd7bc161bb0 0x7fd7bc0cb930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:06.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.758+0000 7fd7c1ca1640 1 --2- 172.21.15.67:0/1454084703 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd7bc15dfe0 0x7fd7bc1f4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:06.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.758+0000 7fd7c1ca1640 1 --2- 172.21.15.67:0/1454084703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7bc0cf220 0x7fd7bc0b2230 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:06.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.758+0000 7fd7c1ca1640 1 -- 172.21.15.67:0/1454084703 >> 172.21.15.67:0/1454084703 conn(0x7fd7bc06c890 msgr2=0x7fd7bc0d2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:06.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.758+0000 7fd7c1ca1640 1 -- 172.21.15.67:0/1454084703 shutdown_connections 2024-03-24T00:39:06.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.758+0000 7fd7c1ca1640 1 -- 172.21.15.67:0/1454084703 wait complete. 2024-03-24T00:39:06.770 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-24T00:39:06.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.873+0000 7f7d54b48640 1 Processor -- start 2024-03-24T00:39:06.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.873+0000 7f7d54b48640 1 -- start start 2024-03-24T00:39:06.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.874+0000 7f7d54b48640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7d5015bc30 0x7f7d500b1a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:06.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.874+0000 7f7d54b48640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7d5015c010 0x7f7d500b1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:06.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.874+0000 7f7d54b48640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7d500b2d90 0x7f7d500b3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:06.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.874+0000 7f7d54b48640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d500cca60 con 0x7f7d5015bc30 2024-03-24T00:39:06.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.874+0000 7f7d54b48640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d500b3ae0 con 0x7f7d5015c010 2024-03-24T00:39:06.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.874+0000 7f7d54b48640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d500b3dc0 con 0x7f7d500b2d90 2024-03-24T00:39:06.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.874+0000 7f7d4e59c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7d5015bc30 0x7f7d500b1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:06.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.874+0000 7f7d4e59c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7d5015bc30 0x7f7d500b1a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60640/0 (socket says 172.21.15.67:60640) 2024-03-24T00:39:06.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.874+0000 7f7d4e59c640 1 -- 172.21.15.67:0/2431173520 learned_addr learned my addr 172.21.15.67:0/2431173520 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:06.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.874+0000 7f7d4dd9b640 1 --2- 172.21.15.67:0/2431173520 >> v2:172.21.15.72:3300/0 conn(0x7f7d5015c010 0x7f7d500b1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:06.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.874+0000 7f7d4ed9d640 1 --2- 172.21.15.67:0/2431173520 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7d500b2d90 0x7f7d500b3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:06.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.875+0000 7f7d4e59c640 1 -- 172.21.15.67:0/2431173520 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7d500b2d90 msgr2=0x7f7d500b3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:06.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.875+0000 7f7d4e59c640 1 --2- 172.21.15.67:0/2431173520 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7d500b2d90 0x7f7d500b3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:06.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.875+0000 7f7d4e59c640 1 -- 172.21.15.67:0/2431173520 >> v2:172.21.15.72:3300/0 conn(0x7f7d5015c010 msgr2=0x7f7d500b1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:06.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.875+0000 7f7d4e59c640 1 --2- 172.21.15.67:0/2431173520 >> v2:172.21.15.72:3300/0 conn(0x7f7d5015c010 0x7f7d500b1f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:06.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.875+0000 7f7d4e59c640 1 -- 172.21.15.67:0/2431173520 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d500b40a0 con 0x7f7d5015bc30 2024-03-24T00:39:06.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.875+0000 7f7d4dd9b640 1 --2- 172.21.15.67:0/2431173520 >> v2:172.21.15.72:3300/0 conn(0x7f7d5015c010 0x7f7d500b1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:06.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.875+0000 7f7d4e59c640 1 --2- 172.21.15.67:0/2431173520 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d5015bc30 0x7f7d500b1a00 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7d500b46d0 tx=0x7f7d4802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbe0145a7932ff90 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:06.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.876+0000 7f7d4d59a640 1 -- 172.21.15.67:0/2431173520 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d48035020 con 0x7f7d5015bc30 2024-03-24T00:39:06.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.876+0000 7f7d4d59a640 1 -- 172.21.15.67:0/2431173520 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d48031870 con 0x7f7d5015bc30 2024-03-24T00:39:06.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.876+0000 7f7d4d59a640 1 -- 172.21.15.67:0/2431173520 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d48031c40 con 0x7f7d5015bc30 2024-03-24T00:39:06.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.876+0000 7f7d54b48640 1 -- 172.21.15.67:0/2431173520 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d5015bc30 msgr2=0x7f7d500b1a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:06.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.876+0000 7f7d54b48640 1 --2- 172.21.15.67:0/2431173520 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d5015bc30 0x7f7d500b1a00 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7d500b46d0 tx=0x7f7d4802b910 comp rx=0 tx=0).stop 2024-03-24T00:39:06.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.876+0000 7f7d54b48640 1 -- 172.21.15.67:0/2431173520 shutdown_connections 2024-03-24T00:39:06.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.876+0000 7f7d54b48640 1 --2- 172.21.15.67:0/2431173520 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7d500b2d90 0x7f7d500b3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:06.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.876+0000 7f7d54b48640 1 --2- 172.21.15.67:0/2431173520 >> v2:172.21.15.72:3300/0 conn(0x7f7d5015c010 0x7f7d500b1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:06.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.876+0000 7f7d54b48640 1 --2- 172.21.15.67:0/2431173520 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d5015bc30 0x7f7d500b1a00 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:06.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.876+0000 7f7d54b48640 1 -- 172.21.15.67:0/2431173520 >> 172.21.15.67:0/2431173520 conn(0x7f7d5008b180 msgr2=0x7f7d5008b590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:06.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.876+0000 7f7d54b48640 1 -- 172.21.15.67:0/2431173520 shutdown_connections 2024-03-24T00:39:06.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.876+0000 7f7d54b48640 1 -- 172.21.15.67:0/2431173520 wait complete. 2024-03-24T00:39:06.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.877+0000 7f7d54b48640 1 Processor -- start 2024-03-24T00:39:06.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.878+0000 7f7d54b48640 1 -- start start 2024-03-24T00:39:06.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.878+0000 7f7d54b48640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d500b2d90 0x7f7d5014fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:06.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.878+0000 7f7d54b48640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7d5015bc30 0x7f7d50070e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:06.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.878+0000 7f7d54b48640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d5015c010 0x7f7d50071390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:06.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.878+0000 7f7d54b48640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d500cfe70 con 0x7f7d5015c010 2024-03-24T00:39:06.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.878+0000 7f7d54b48640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d500b3760 con 0x7f7d500b2d90 2024-03-24T00:39:06.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.878+0000 7f7d54b48640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d500cfff0 con 0x7f7d5015bc30 2024-03-24T00:39:06.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.878+0000 7f7d4ed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d5015c010 0x7f7d50071390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:06.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.878+0000 7f7d4ed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d5015c010 0x7f7d50071390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60654/0 (socket says 172.21.15.67:60654) 2024-03-24T00:39:06.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.878+0000 7f7d4ed9d640 1 -- 172.21.15.67:0/1341784487 learned_addr learned my addr 172.21.15.67:0/1341784487 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:06.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.878+0000 7f7d4dd9b640 1 --2- 172.21.15.67:0/1341784487 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7d5015bc30 0x7f7d50070e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:06.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.878+0000 7f7d4e59c640 1 --2- 172.21.15.67:0/1341784487 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d500b2d90 0x7f7d5014fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:06.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.879+0000 7f7d4ed9d640 1 -- 172.21.15.67:0/1341784487 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7d5015bc30 msgr2=0x7f7d50070e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:06.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.879+0000 7f7d4ed9d640 1 --2- 172.21.15.67:0/1341784487 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7d5015bc30 0x7f7d50070e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:06.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.879+0000 7f7d4ed9d640 1 -- 172.21.15.67:0/1341784487 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d500b2d90 msgr2=0x7f7d5014fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:06.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.879+0000 7f7d4ed9d640 1 --2- 172.21.15.67:0/1341784487 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d500b2d90 0x7f7d5014fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:06.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.879+0000 7f7d4ed9d640 1 -- 172.21.15.67:0/1341784487 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d50150320 con 0x7f7d5015c010 2024-03-24T00:39:06.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.879+0000 7f7d4dd9b640 1 --2- 172.21.15.67:0/1341784487 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7d5015bc30 0x7f7d50070e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:06.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.879+0000 7f7d4ed9d640 1 --2- 172.21.15.67:0/1341784487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d5015c010 0x7f7d50071390 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f7d40009dd0 tx=0x7f7d4000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:06.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.879+0000 7f7d3effd640 1 -- 172.21.15.67:0/1341784487 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d40018020 con 0x7f7d5015c010 2024-03-24T00:39:06.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.879+0000 7f7d3effd640 1 -- 172.21.15.67:0/1341784487 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d40014480 con 0x7f7d5015c010 2024-03-24T00:39:06.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.879+0000 7f7d54b48640 1 -- 172.21.15.67:0/1341784487 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d500ae7a0 con 0x7f7d5015c010 2024-03-24T00:39:06.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.879+0000 7f7d3effd640 1 -- 172.21.15.67:0/1341784487 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d40014870 con 0x7f7d5015c010 2024-03-24T00:39:06.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.879+0000 7f7d54b48640 1 -- 172.21.15.67:0/1341784487 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d500aed60 con 0x7f7d5015c010 2024-03-24T00:39:06.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.880+0000 7f7d54b48640 1 -- 172.21.15.67:0/1341784487 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d5004f400 con 0x7f7d5015c010 2024-03-24T00:39:06.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.881+0000 7f7d3effd640 1 -- 172.21.15.67:0/1341784487 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7d40005d10 con 0x7f7d5015c010 2024-03-24T00:39:06.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.881+0000 7f7d3effd640 1 --2- 172.21.15.67:0/1341784487 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7d3006bbd0 0x7f7d3006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:06.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.881+0000 7f7d3effd640 1 -- 172.21.15.67:0/1341784487 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f7d4008b780 con 0x7f7d5015c010 2024-03-24T00:39:06.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.881+0000 7f7d4e59c640 1 --2- 172.21.15.67:0/1341784487 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7d3006bbd0 0x7f7d3006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:06.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.882+0000 7f7d4e59c640 1 --2- 172.21.15.67:0/1341784487 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7d3006bbd0 0x7f7d3006e090 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7d48036000 tx=0x7f7d4803c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:06.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:06.886+0000 7f7d3effd640 1 -- 172.21.15.67:0/1341784487 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7d40022070 con 0x7f7d5015c010 2024-03-24T00:39:07.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.107+0000 7f7d54b48640 1 -- 172.21.15.67:0/1341784487 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f7d500d07e0 con 0x7f7d5015c010 2024-03-24T00:39:07.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.829+0000 7f7d3effd640 1 -- 172.21.15.67:0/1341784487 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v8) v1 ==== 118+0+0 (secure 0 0 0) 0x7f7d50150320 con 0x7f7d5015c010 2024-03-24T00:39:07.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.831+0000 7f7d54b48640 1 -- 172.21.15.67:0/1341784487 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7d3006bbd0 msgr2=0x7f7d3006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:07.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.831+0000 7f7d54b48640 1 --2- 172.21.15.67:0/1341784487 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7d3006bbd0 0x7f7d3006e090 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7d48036000 tx=0x7f7d4803c040 comp rx=0 tx=0).stop 2024-03-24T00:39:07.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.831+0000 7f7d54b48640 1 -- 172.21.15.67:0/1341784487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d5015c010 msgr2=0x7f7d50071390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:07.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.831+0000 7f7d54b48640 1 --2- 172.21.15.67:0/1341784487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d5015c010 0x7f7d50071390 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f7d40009dd0 tx=0x7f7d4000c490 comp rx=0 tx=0).stop 2024-03-24T00:39:07.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.831+0000 7f7d54b48640 1 -- 172.21.15.67:0/1341784487 shutdown_connections 2024-03-24T00:39:07.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.831+0000 7f7d54b48640 1 --2- 172.21.15.67:0/1341784487 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7d3006bbd0 0x7f7d3006e090 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:07.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.831+0000 7f7d54b48640 1 --2- 172.21.15.67:0/1341784487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d5015c010 0x7f7d50071390 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:07.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.831+0000 7f7d54b48640 1 --2- 172.21.15.67:0/1341784487 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7d5015bc30 0x7f7d50070e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:07.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.831+0000 7f7d54b48640 1 --2- 172.21.15.67:0/1341784487 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d500b2d90 0x7f7d5014fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:07.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.831+0000 7f7d54b48640 1 -- 172.21.15.67:0/1341784487 >> 172.21.15.67:0/1341784487 conn(0x7f7d5008b180 msgr2=0x7f7d500ccfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:07.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.831+0000 7f7d54b48640 1 -- 172.21.15.67:0/1341784487 shutdown_connections 2024-03-24T00:39:07.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.831+0000 7f7d54b48640 1 -- 172.21.15.67:0/1341784487 wait complete. 2024-03-24T00:39:07.847 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']} 2024-03-24T00:39:07.847 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-24T00:39:07.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.953+0000 7f2ff32c6640 1 Processor -- start 2024-03-24T00:39:07.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.954+0000 7f2ff32c6640 1 -- start start 2024-03-24T00:39:07.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.954+0000 7f2ff32c6640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2fec15c1b0 0x7f2fec15fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:07.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.954+0000 7f2ff32c6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2fec160260 0x7f2fec0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:07.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.954+0000 7f2ff32c6640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fec15c7c0 con 0x7f2fec0d18b0 2024-03-24T00:39:07.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.954+0000 7f2ff32c6640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fec15caa0 con 0x7f2fec15c1b0 2024-03-24T00:39:07.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.954+0000 7f2ff32c6640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fec15cd80 con 0x7f2fec160260 2024-03-24T00:39:07.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.954+0000 7f2ff0861640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2fec15c1b0 0x7f2fec15fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:07.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.955+0000 7f2ff0861640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2fec15c1b0 0x7f2fec15fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60670/0 (socket says 172.21.15.67:60670) 2024-03-24T00:39:07.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.955+0000 7f2ff0861640 1 -- 172.21.15.67:0/3385128817 learned_addr learned my addr 172.21.15.67:0/3385128817 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:07.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.955+0000 7f2ff1062640 1 --1- 172.21.15.67:0/3385128817 >> v1:172.21.15.72:6789/0 conn(0x7f2fec0d18b0 0x7f2fec0d1c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:51260/0 (socket says 172.21.15.67:51260) 2024-03-24T00:39:07.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.955+0000 7f2ff1863640 1 --2- 172.21.15.67:0/3385128817 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2fec160260 0x7f2fec0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:07.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.955+0000 7f2fe3fff640 1 -- 172.21.15.67:0/3385128817 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 734074621 0 0) 0x7f2fec15c7c0 con 0x7f2fec0d18b0 2024-03-24T00:39:07.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.955+0000 7f2fe3fff640 1 -- 172.21.15.67:0/3385128817 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fd8003520 con 0x7f2fec0d18b0 2024-03-24T00:39:07.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.955+0000 7f2ff0861640 1 -- 172.21.15.67:0/3385128817 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2fec160260 msgr2=0x7f2fec0cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:07.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.955+0000 7f2ff0861640 1 --2- 172.21.15.67:0/3385128817 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2fec160260 0x7f2fec0cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:07.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.955+0000 7f2ff0861640 1 -- 172.21.15.67:0/3385128817 >> v1:172.21.15.72:6789/0 conn(0x7f2fec0d18b0 legacy=0x7f2fec0d1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:07.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.955+0000 7f2ff0861640 1 -- 172.21.15.67:0/3385128817 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fec15d060 con 0x7f2fec15c1b0 2024-03-24T00:39:07.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.955+0000 7f2ff1863640 1 --2- 172.21.15.67:0/3385128817 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2fec160260 0x7f2fec0cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:07.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.956+0000 7f2ff0861640 1 --2- 172.21.15.67:0/3385128817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fec15c1b0 0x7f2fec15fd20 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc009870 tx=0x7f2fdc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=433931e33310e02b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:07.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.956+0000 7f2fe3fff640 1 -- 172.21.15.67:0/3385128817 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc035020 con 0x7f2fec15c1b0 2024-03-24T00:39:07.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.956+0000 7f2fe3fff640 1 -- 172.21.15.67:0/3385128817 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fdc031640 con 0x7f2fec15c1b0 2024-03-24T00:39:07.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.956+0000 7f2fe3fff640 1 -- 172.21.15.67:0/3385128817 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc02d040 con 0x7f2fec15c1b0 2024-03-24T00:39:07.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.956+0000 7f2ff32c6640 1 -- 172.21.15.67:0/3385128817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fec15c1b0 msgr2=0x7f2fec15fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:07.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.956+0000 7f2ff32c6640 1 --2- 172.21.15.67:0/3385128817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fec15c1b0 0x7f2fec15fd20 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc009870 tx=0x7f2fdc02b8f0 comp rx=0 tx=0).stop 2024-03-24T00:39:07.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.956+0000 7f2ff32c6640 1 -- 172.21.15.67:0/3385128817 shutdown_connections 2024-03-24T00:39:07.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.956+0000 7f2ff32c6640 1 --2- 172.21.15.67:0/3385128817 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2fec160260 0x7f2fec0cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:07.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.956+0000 7f2ff32c6640 1 --2- 172.21.15.67:0/3385128817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fec15c1b0 0x7f2fec15fd20 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:07.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.956+0000 7f2ff32c6640 1 -- 172.21.15.67:0/3385128817 >> 172.21.15.67:0/3385128817 conn(0x7f2fec08b1b0 msgr2=0x7f2fec08b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:07.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.956+0000 7f2ff32c6640 1 -- 172.21.15.67:0/3385128817 shutdown_connections 2024-03-24T00:39:07.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.956+0000 7f2ff32c6640 1 -- 172.21.15.67:0/3385128817 wait complete. 2024-03-24T00:39:07.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.957+0000 7f2ff32c6640 1 Processor -- start 2024-03-24T00:39:07.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.958+0000 7f2ff32c6640 1 -- start start 2024-03-24T00:39:07.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.958+0000 7f2ff32c6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2fec0d18b0 0x7f2fec1519d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:07.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.958+0000 7f2ff32c6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2fec15c1b0 0x7f2fec14c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:07.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.958+0000 7f2ff32c6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fec160260 0x7f2fec1566b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:07.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.958+0000 7f2ff32c6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fec0cfe90 con 0x7f2fec160260 2024-03-24T00:39:07.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.958+0000 7f2ff32c6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fec0cfd10 con 0x7f2fec15c1b0 2024-03-24T00:39:07.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.958+0000 7f2ff32c6640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fec0d0010 con 0x7f2fec0d18b0 2024-03-24T00:39:07.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.958+0000 7f2ff1863640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fec160260 0x7f2fec1566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:07.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.958+0000 7f2ff1863640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fec160260 0x7f2fec1566b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60686/0 (socket says 172.21.15.67:60686) 2024-03-24T00:39:07.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.958+0000 7f2ff1863640 1 -- 172.21.15.67:0/1900358651 learned_addr learned my addr 172.21.15.67:0/1900358651 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:07.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.958+0000 7f2ff1062640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2fec0d18b0 0x7f2fec1519d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:07.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.958+0000 7f2ff0861640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2fec15c1b0 0x7f2fec14c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:07.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.959+0000 7f2ff1863640 1 -- 172.21.15.67:0/1900358651 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2fec0d18b0 msgr2=0x7f2fec1519d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:07.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.959+0000 7f2ff1863640 1 --2- 172.21.15.67:0/1900358651 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2fec0d18b0 0x7f2fec1519d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:07.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.959+0000 7f2ff1863640 1 -- 172.21.15.67:0/1900358651 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2fec15c1b0 msgr2=0x7f2fec14c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:07.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.959+0000 7f2ff1863640 1 --2- 172.21.15.67:0/1900358651 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2fec15c1b0 0x7f2fec14c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:07.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.959+0000 7f2ff1863640 1 -- 172.21.15.67:0/1900358651 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fec0cab10 con 0x7f2fec160260 2024-03-24T00:39:07.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.959+0000 7f2ff0861640 1 --2- 172.21.15.67:0/1900358651 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2fec15c1b0 0x7f2fec14c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:07.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.959+0000 7f2ff1863640 1 --2- 172.21.15.67:0/1900358651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fec160260 0x7f2fec1566b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2fe400efc0 tx=0x7f2fe400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:07.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.959+0000 7f2fe1ffb640 1 -- 172.21.15.67:0/1900358651 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe4018030 con 0x7f2fec160260 2024-03-24T00:39:07.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.959+0000 7f2fe1ffb640 1 -- 172.21.15.67:0/1900358651 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fe400cdf0 con 0x7f2fec160260 2024-03-24T00:39:07.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.959+0000 7f2ff32c6640 1 -- 172.21.15.67:0/1900358651 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fec0cadf0 con 0x7f2fec160260 2024-03-24T00:39:07.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.959+0000 7f2fe1ffb640 1 -- 172.21.15.67:0/1900358651 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe4003bb0 con 0x7f2fec160260 2024-03-24T00:39:07.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.959+0000 7f2ff32c6640 1 -- 172.21.15.67:0/1900358651 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fec0cb3b0 con 0x7f2fec160260 2024-03-24T00:39:07.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.960+0000 7f2ff32c6640 1 -- 172.21.15.67:0/1900358651 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fec151f70 con 0x7f2fec160260 2024-03-24T00:39:07.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.961+0000 7f2fe1ffb640 1 -- 172.21.15.67:0/1900358651 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2fe401b070 con 0x7f2fec160260 2024-03-24T00:39:07.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.961+0000 7f2fe1ffb640 1 --2- 172.21.15.67:0/1900358651 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2fd006bbd0 0x7f2fd006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:07.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.961+0000 7f2fe1ffb640 1 -- 172.21.15.67:0/1900358651 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f2fe408b550 con 0x7f2fec160260 2024-03-24T00:39:07.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.961+0000 7f2ff1062640 1 --2- 172.21.15.67:0/1900358651 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2fd006bbd0 0x7f2fd006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:07.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.962+0000 7f2ff1062640 1 --2- 172.21.15.67:0/1900358651 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2fd006bbd0 0x7f2fd006e090 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2fe8000f60 tx=0x7f2fe8007cf0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:07.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:07.965+0000 7f2fe1ffb640 1 -- 172.21.15.67:0/1900358651 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2fe4010040 con 0x7f2fec160260 2024-03-24T00:39:08.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.185+0000 7f2ff32c6640 1 -- 172.21.15.67:0/1900358651 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f2fec0b1840 con 0x7f2fec160260 2024-03-24T00:39:08.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.830+0000 7f2fe1ffb640 1 -- 172.21.15.67:0/1900358651 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v9) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2fe4059740 con 0x7f2fec160260 2024-03-24T00:39:08.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.833+0000 7f2ff32c6640 1 -- 172.21.15.67:0/1900358651 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2fd006bbd0 msgr2=0x7f2fd006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:08.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.833+0000 7f2ff32c6640 1 --2- 172.21.15.67:0/1900358651 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2fd006bbd0 0x7f2fd006e090 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2fe8000f60 tx=0x7f2fe8007cf0 comp rx=0 tx=0).stop 2024-03-24T00:39:08.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.833+0000 7f2ff32c6640 1 -- 172.21.15.67:0/1900358651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fec160260 msgr2=0x7f2fec1566b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:08.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.833+0000 7f2ff32c6640 1 --2- 172.21.15.67:0/1900358651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fec160260 0x7f2fec1566b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2fe400efc0 tx=0x7f2fe400c490 comp rx=0 tx=0).stop 2024-03-24T00:39:08.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.833+0000 7f2ff32c6640 1 -- 172.21.15.67:0/1900358651 shutdown_connections 2024-03-24T00:39:08.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.833+0000 7f2ff32c6640 1 --2- 172.21.15.67:0/1900358651 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2fd006bbd0 0x7f2fd006e090 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:08.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.833+0000 7f2ff32c6640 1 --2- 172.21.15.67:0/1900358651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fec160260 0x7f2fec1566b0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:08.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.833+0000 7f2ff32c6640 1 --2- 172.21.15.67:0/1900358651 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2fec15c1b0 0x7f2fec14c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:08.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.833+0000 7f2ff32c6640 1 --2- 172.21.15.67:0/1900358651 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2fec0d18b0 0x7f2fec1519d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:08.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.833+0000 7f2ff32c6640 1 -- 172.21.15.67:0/1900358651 >> 172.21.15.67:0/1900358651 conn(0x7f2fec08b1b0 msgr2=0x7f2fec0712e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:08.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.833+0000 7f2ff32c6640 1 -- 172.21.15.67:0/1900358651 shutdown_connections 2024-03-24T00:39:08.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.833+0000 7f2ff32c6640 1 -- 172.21.15.67:0/1900358651 wait complete. 2024-03-24T00:39:08.846 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-24T00:39:08.846 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:39:08.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.951+0000 7fa57fff4640 1 Processor -- start 2024-03-24T00:39:08.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.951+0000 7fa57fff4640 1 -- start start 2024-03-24T00:39:08.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.952+0000 7fa57fff4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa57815df30 0x7fa578162610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:08.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.952+0000 7fa57fff4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa57815e310 0x7fa5780d0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:08.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.952+0000 7fa57fff4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa5780d0b80 0x7fa5780b57a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:08.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.952+0000 7fa57fff4640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa578162b50 con 0x7fa57815e310 2024-03-24T00:39:08.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.952+0000 7fa57fff4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5780d0fd0 con 0x7fa5780d0b80 2024-03-24T00:39:08.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.952+0000 7fa57fff4640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5780ce920 con 0x7fa57815df30 2024-03-24T00:39:08.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.952+0000 7fa57d58f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa57815e310 0x7fa5780d0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:08.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.952+0000 7fa57e591640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa5780d0b80 0x7fa5780b57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:08.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.952+0000 7fa57dd90640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa57815df30 0x7fa578162610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:08.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.952+0000 7fa57d58f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa57815e310 0x7fa5780d0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60690/0 (socket says 172.21.15.67:60690) 2024-03-24T00:39:08.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.952+0000 7fa57e591640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa5780d0b80 0x7fa5780b57a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:33930/0 (socket says 172.21.15.67:33930) 2024-03-24T00:39:08.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.952+0000 7fa57d58f640 1 -- 172.21.15.67:0/4257285121 learned_addr learned my addr 172.21.15.67:0/4257285121 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:08.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.952+0000 7fa57d58f640 1 -- 172.21.15.67:0/4257285121 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa57815df30 msgr2=0x7fa578162610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:08.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.952+0000 7fa57d58f640 1 --2- 172.21.15.67:0/4257285121 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa57815df30 0x7fa578162610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:08.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.952+0000 7fa57d58f640 1 -- 172.21.15.67:0/4257285121 >> v2:172.21.15.72:3300/0 conn(0x7fa5780d0b80 msgr2=0x7fa5780b57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:08.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.952+0000 7fa57d58f640 1 --2- 172.21.15.67:0/4257285121 >> v2:172.21.15.72:3300/0 conn(0x7fa5780d0b80 0x7fa5780b57a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:08.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.952+0000 7fa57d58f640 1 -- 172.21.15.67:0/4257285121 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5780cef20 con 0x7fa57815e310 2024-03-24T00:39:08.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.952+0000 7fa57dd90640 1 --2- 172.21.15.67:0/4257285121 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa57815df30 0x7fa578162610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:08.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.952+0000 7fa57e591640 1 --2- 172.21.15.67:0/4257285121 >> v2:172.21.15.72:3300/0 conn(0x7fa5780d0b80 0x7fa5780b57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:08.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.953+0000 7fa57d58f640 1 --2- 172.21.15.67:0/4257285121 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa57815e310 0x7fa5780d0640 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa57806d7b0 tx=0x7fa56802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15c807a127b9ef88 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:08.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.953+0000 7fa57cd8e640 1 -- 172.21.15.67:0/4257285121 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5680317c0 con 0x7fa57815e310 2024-03-24T00:39:08.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.953+0000 7fa57cd8e640 1 -- 172.21.15.67:0/4257285121 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa568037070 con 0x7fa57815e310 2024-03-24T00:39:08.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.953+0000 7fa57cd8e640 1 -- 172.21.15.67:0/4257285121 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa568031b80 con 0x7fa57815e310 2024-03-24T00:39:08.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.954+0000 7fa57fff4640 1 -- 172.21.15.67:0/4257285121 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa57815e310 msgr2=0x7fa5780d0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:08.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.954+0000 7fa57fff4640 1 --2- 172.21.15.67:0/4257285121 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa57815e310 0x7fa5780d0640 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa57806d7b0 tx=0x7fa56802b860 comp rx=0 tx=0).stop 2024-03-24T00:39:08.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.954+0000 7fa57fff4640 1 -- 172.21.15.67:0/4257285121 shutdown_connections 2024-03-24T00:39:08.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.954+0000 7fa57fff4640 1 --2- 172.21.15.67:0/4257285121 >> v2:172.21.15.72:3300/0 conn(0x7fa5780d0b80 0x7fa5780b57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:08.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.954+0000 7fa57fff4640 1 --2- 172.21.15.67:0/4257285121 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa57815e310 0x7fa5780d0640 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:08.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.954+0000 7fa57fff4640 1 --2- 172.21.15.67:0/4257285121 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa57815df30 0x7fa578162610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:08.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.954+0000 7fa57fff4640 1 -- 172.21.15.67:0/4257285121 >> 172.21.15.67:0/4257285121 conn(0x7fa57806c880 msgr2=0x7fa57806cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:08.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.954+0000 7fa57fff4640 1 -- 172.21.15.67:0/4257285121 shutdown_connections 2024-03-24T00:39:08.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.954+0000 7fa57fff4640 1 -- 172.21.15.67:0/4257285121 wait complete. 2024-03-24T00:39:08.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.955+0000 7fa57fff4640 1 Processor -- start 2024-03-24T00:39:08.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.955+0000 7fa57fff4640 1 -- start start 2024-03-24T00:39:08.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.956+0000 7fa57fff4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa5780d0b80 0x7fa578159bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:08.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.956+0000 7fa57fff4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa57815df30 0x7fa5780b77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:08.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.956+0000 7fa57fff4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5781f38a0 0x7fa5780b7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:08.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.956+0000 7fa57fff4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5780b32f0 con 0x7fa5781f38a0 2024-03-24T00:39:08.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.956+0000 7fa57fff4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5780cf4e0 con 0x7fa57815df30 2024-03-24T00:39:08.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.956+0000 7fa57fff4640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5780ce580 con 0x7fa5780d0b80 2024-03-24T00:39:08.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.956+0000 7fa57e591640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5781f38a0 0x7fa5780b7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:08.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.956+0000 7fa57e591640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5781f38a0 0x7fa5780b7ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60704/0 (socket says 172.21.15.67:60704) 2024-03-24T00:39:08.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.956+0000 7fa57e591640 1 -- 172.21.15.67:0/992270921 learned_addr learned my addr 172.21.15.67:0/992270921 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:08.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.956+0000 7fa57dd90640 1 --2- 172.21.15.67:0/992270921 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa5780d0b80 0x7fa578159bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:08.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.956+0000 7fa57d58f640 1 --2- 172.21.15.67:0/992270921 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa57815df30 0x7fa5780b77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:08.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.956+0000 7fa57dd90640 1 -- 172.21.15.67:0/992270921 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa57815df30 msgr2=0x7fa5780b77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:08.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.956+0000 7fa57dd90640 1 --2- 172.21.15.67:0/992270921 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa57815df30 0x7fa5780b77a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:08.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.956+0000 7fa57dd90640 1 -- 172.21.15.67:0/992270921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5781f38a0 msgr2=0x7fa5780b7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:08.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.956+0000 7fa57dd90640 1 --2- 172.21.15.67:0/992270921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5781f38a0 0x7fa5780b7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:08.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.956+0000 7fa57dd90640 1 -- 172.21.15.67:0/992270921 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa57815a260 con 0x7fa5780d0b80 2024-03-24T00:39:08.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.956+0000 7fa57e591640 1 --2- 172.21.15.67:0/992270921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5781f38a0 0x7fa5780b7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:08.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.956+0000 7fa57dd90640 1 --2- 172.21.15.67:0/992270921 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa5780d0b80 0x7fa578159bb0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa57400e930 tx=0x7fa57400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:08.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.956+0000 7fa56e7fc640 1 -- 172.21.15.67:0/992270921 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa574017070 con 0x7fa5780d0b80 2024-03-24T00:39:08.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.956+0000 7fa56e7fc640 1 -- 172.21.15.67:0/992270921 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa57400ca00 con 0x7fa5780d0b80 2024-03-24T00:39:08.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.956+0000 7fa57fff4640 1 -- 172.21.15.67:0/992270921 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa57815a540 con 0x7fa5780d0b80 2024-03-24T00:39:08.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.956+0000 7fa56e7fc640 1 -- 172.21.15.67:0/992270921 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa57400cdd0 con 0x7fa5780d0b80 2024-03-24T00:39:08.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.956+0000 7fa57fff4640 1 -- 172.21.15.67:0/992270921 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa57815ab00 con 0x7fa5780d0b80 2024-03-24T00:39:08.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.957+0000 7fa57fff4640 1 -- 172.21.15.67:0/992270921 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa57804f400 con 0x7fa5780d0b80 2024-03-24T00:39:08.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.958+0000 7fa56e7fc640 1 -- 172.21.15.67:0/992270921 <== mon.2 v2:172.21.15.72:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa57401a070 con 0x7fa5780d0b80 2024-03-24T00:39:08.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.958+0000 7fa56e7fc640 1 --2- 172.21.15.67:0/992270921 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa55c06bbd0 0x7fa55c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:08.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.958+0000 7fa56e7fc640 1 -- 172.21.15.67:0/992270921 <== mon.2 v2:172.21.15.72:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fa57408af80 con 0x7fa5780d0b80 2024-03-24T00:39:08.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.958+0000 7fa57d58f640 1 --2- 172.21.15.67:0/992270921 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa55c06bbd0 0x7fa55c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:08.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.959+0000 7fa57d58f640 1 --2- 172.21.15.67:0/992270921 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa55c06bbd0 0x7fa55c06e090 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa568036000 tx=0x7fa56803d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:08.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:08.963+0000 7fa56e7fc640 1 -- 172.21.15.67:0/992270921 <== mon.2 v2:172.21.15.72:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa5740590c0 con 0x7fa5780d0b80 2024-03-24T00:39:09.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.183+0000 7fa57fff4640 1 -- 172.21.15.67:0/992270921 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa5780d1a30 con 0x7fa5780d0b80 2024-03-24T00:39:09.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.183+0000 7fa56e7fc640 1 -- 172.21.15.67:0/992270921 <== mon.2 v2:172.21.15.72:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4970 (secure 0 0 0) 0x7fa57405d160 con 0x7fa5780d0b80 2024-03-24T00:39:09.184 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:39:09.184 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4263,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/837528243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":837528243},{"type":"v1","addr":"172.21.15.67:6834","nonce":837528243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3987071456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":3987071456},{"type":"v1","addr":"172.21.15.67:6835","nonce":3987071456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4306,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/3953492281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3953492281},{"type":"v1","addr":"172.21.15.72:6837","nonce":3953492281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:38:52.127871+0000","modified":"2024-03-24T00:39:08.187262+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4309},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4309":{"gid":4309,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6836/3737698654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3737698654},{"type":"v1","addr":"172.21.15.72:6836","nonce":3737698654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4309,"qdb_cluster":[4309]},"id":1}]} 2024-03-24T00:39:09.185 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 9 2024-03-24T00:39:09.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.186+0000 7fa57fff4640 1 -- 172.21.15.67:0/992270921 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa55c06bbd0 msgr2=0x7fa55c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:09.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.186+0000 7fa57fff4640 1 --2- 172.21.15.67:0/992270921 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa55c06bbd0 0x7fa55c06e090 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa568036000 tx=0x7fa56803d040 comp rx=0 tx=0).stop 2024-03-24T00:39:09.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.186+0000 7fa57fff4640 1 -- 172.21.15.67:0/992270921 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa5780d0b80 msgr2=0x7fa578159bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:09.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.186+0000 7fa57fff4640 1 --2- 172.21.15.67:0/992270921 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa5780d0b80 0x7fa578159bb0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa57400e930 tx=0x7fa57400edf0 comp rx=0 tx=0).stop 2024-03-24T00:39:09.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.186+0000 7fa57fff4640 1 -- 172.21.15.67:0/992270921 shutdown_connections 2024-03-24T00:39:09.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.186+0000 7fa57fff4640 1 --2- 172.21.15.67:0/992270921 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa55c06bbd0 0x7fa55c06e090 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:09.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.186+0000 7fa57fff4640 1 --2- 172.21.15.67:0/992270921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5781f38a0 0x7fa5780b7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:09.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.186+0000 7fa57fff4640 1 --2- 172.21.15.67:0/992270921 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa57815df30 0x7fa5780b77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:09.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.186+0000 7fa57fff4640 1 --2- 172.21.15.67:0/992270921 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa5780d0b80 0x7fa578159bb0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:09.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.186+0000 7fa57fff4640 1 -- 172.21.15.67:0/992270921 >> 172.21.15.67:0/992270921 conn(0x7fa57806c880 msgr2=0x7fa57806cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:09.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.186+0000 7fa57fff4640 1 -- 172.21.15.67:0/992270921 shutdown_connections 2024-03-24T00:39:09.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.186+0000 7fa57fff4640 1 -- 172.21.15.67:0/992270921 wait complete. 2024-03-24T00:39:09.197 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:39:09.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.297+0000 7f80a5977640 1 Processor -- start 2024-03-24T00:39:09.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.298+0000 7f80a5977640 1 -- start start 2024-03-24T00:39:09.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.298+0000 7f80a5977640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f80a015e390 0x7f80a006b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:09.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.298+0000 7f80a5977640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f80a00cded0 0x7f80a006b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:09.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.298+0000 7f80a5977640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80a00d07d0 con 0x7f80a015dfb0 2024-03-24T00:39:09.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.298+0000 7f80a5977640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80a00d0590 con 0x7f80a015e390 2024-03-24T00:39:09.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.298+0000 7f80a5977640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80a00d0ab0 con 0x7f80a00cded0 2024-03-24T00:39:09.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.298+0000 7f809effd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f80a015dfb0 0x7f80a00cfe80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45374/0 (socket says 172.21.15.67:45374) 2024-03-24T00:39:09.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.298+0000 7f809effd640 1 -- 172.21.15.67:0/1031366133 learned_addr learned my addr 172.21.15.67:0/1031366133 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:09.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.298+0000 7f809e7fc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f80a015e390 0x7f80a006b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:09.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.298+0000 7f809f7fe640 1 --2- 172.21.15.67:0/1031366133 >> v2:172.21.15.72:3300/0 conn(0x7f80a00cded0 0x7f80a006b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:09.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.299+0000 7f809dffb640 1 -- 172.21.15.67:0/1031366133 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1831715312 0 0) 0x7f80a00d07d0 con 0x7f80a015dfb0 2024-03-24T00:39:09.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.299+0000 7f809dffb640 1 -- 172.21.15.67:0/1031366133 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f808c003520 con 0x7f80a015dfb0 2024-03-24T00:39:09.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.299+0000 7f809e7fc640 1 -- 172.21.15.67:0/1031366133 >> v2:172.21.15.72:3300/0 conn(0x7f80a00cded0 msgr2=0x7f80a006b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:09.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.299+0000 7f809e7fc640 1 --2- 172.21.15.67:0/1031366133 >> v2:172.21.15.72:3300/0 conn(0x7f80a00cded0 0x7f80a006b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:09.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.299+0000 7f809e7fc640 1 -- 172.21.15.67:0/1031366133 >> v1:172.21.15.67:6789/0 conn(0x7f80a015dfb0 legacy=0x7f80a00cfe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:09.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.299+0000 7f809e7fc640 1 -- 172.21.15.67:0/1031366133 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a00d0d90 con 0x7f80a015e390 2024-03-24T00:39:09.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.299+0000 7f809f7fe640 1 --2- 172.21.15.67:0/1031366133 >> v2:172.21.15.72:3300/0 conn(0x7f80a00cded0 0x7f80a006b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:09.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.299+0000 7f809e7fc640 1 --2- 172.21.15.67:0/1031366133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80a015e390 0x7f80a006b040 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f80a0072f90 tx=0x7f808802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cfa8e47f73d4e5f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:09.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.300+0000 7f809dffb640 1 -- 172.21.15.67:0/1031366133 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8088035020 con 0x7f80a015e390 2024-03-24T00:39:09.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.300+0000 7f809dffb640 1 -- 172.21.15.67:0/1031366133 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8088031830 con 0x7f80a015e390 2024-03-24T00:39:09.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.300+0000 7f809dffb640 1 -- 172.21.15.67:0/1031366133 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8088031c00 con 0x7f80a015e390 2024-03-24T00:39:09.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.300+0000 7f80a5977640 1 -- 172.21.15.67:0/1031366133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80a015e390 msgr2=0x7f80a006b040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:09.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.300+0000 7f80a5977640 1 --2- 172.21.15.67:0/1031366133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80a015e390 0x7f80a006b040 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f80a0072f90 tx=0x7f808802b8d0 comp rx=0 tx=0).stop 2024-03-24T00:39:09.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.300+0000 7f80a5977640 1 -- 172.21.15.67:0/1031366133 shutdown_connections 2024-03-24T00:39:09.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.300+0000 7f80a5977640 1 --2- 172.21.15.67:0/1031366133 >> v2:172.21.15.72:3300/0 conn(0x7f80a00cded0 0x7f80a006b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:09.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.300+0000 7f80a5977640 1 --2- 172.21.15.67:0/1031366133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80a015e390 0x7f80a006b040 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:09.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.300+0000 7f80a5977640 1 -- 172.21.15.67:0/1031366133 >> 172.21.15.67:0/1031366133 conn(0x7f80a0099a30 msgr2=0x7f80a0099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:09.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.300+0000 7f80a5977640 1 -- 172.21.15.67:0/1031366133 shutdown_connections 2024-03-24T00:39:09.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.300+0000 7f80a5977640 1 -- 172.21.15.67:0/1031366133 wait complete. 2024-03-24T00:39:09.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.301+0000 7f80a5977640 1 Processor -- start 2024-03-24T00:39:09.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.302+0000 7f80a5977640 1 -- start start 2024-03-24T00:39:09.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.302+0000 7f80a5977640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80a00cded0 0x7f80a00b5c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:09.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.302+0000 7f80a5977640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f80a015dfb0 0x7f80a0149ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:09.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.302+0000 7f80a5977640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f80a015e390 0x7f80a014a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:09.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.302+0000 7f80a5977640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a00d1870 con 0x7f80a00cded0 2024-03-24T00:39:09.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.302+0000 7f80a5977640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a00d1350 con 0x7f80a015dfb0 2024-03-24T00:39:09.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.302+0000 7f80a5977640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80a00cb120 con 0x7f80a015e390 2024-03-24T00:39:09.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.302+0000 7f809effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80a00cded0 0x7f80a00b5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:09.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.302+0000 7f809effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80a00cded0 0x7f80a00b5c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60722/0 (socket says 172.21.15.67:60722) 2024-03-24T00:39:09.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.302+0000 7f809effd640 1 -- 172.21.15.67:0/3062446106 learned_addr learned my addr 172.21.15.67:0/3062446106 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:09.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.302+0000 7f809e7fc640 1 --2- 172.21.15.67:0/3062446106 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f80a015dfb0 0x7f80a0149ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:09.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.302+0000 7f809f7fe640 1 --2- 172.21.15.67:0/3062446106 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f80a015e390 0x7f80a014a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:09.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.303+0000 7f809effd640 1 -- 172.21.15.67:0/3062446106 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f80a015e390 msgr2=0x7f80a014a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:09.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.303+0000 7f809effd640 1 --2- 172.21.15.67:0/3062446106 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f80a015e390 0x7f80a014a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:09.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.303+0000 7f809effd640 1 -- 172.21.15.67:0/3062446106 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f80a015dfb0 msgr2=0x7f80a0149ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:09.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.303+0000 7f809effd640 1 --2- 172.21.15.67:0/3062446106 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f80a015dfb0 0x7f80a0149ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:09.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.303+0000 7f809effd640 1 -- 172.21.15.67:0/3062446106 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a00b5190 con 0x7f80a00cded0 2024-03-24T00:39:09.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.303+0000 7f809effd640 1 --2- 172.21.15.67:0/3062446106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80a00cded0 0x7f80a00b5c00 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f809800c430 tx=0x7f809800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:09.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.303+0000 7f8097fff640 1 -- 172.21.15.67:0/3062446106 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8098017070 con 0x7f80a00cded0 2024-03-24T00:39:09.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.303+0000 7f8097fff640 1 -- 172.21.15.67:0/3062446106 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8098005ce0 con 0x7f80a00cded0 2024-03-24T00:39:09.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.303+0000 7f80a5977640 1 -- 172.21.15.67:0/3062446106 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80a0157270 con 0x7f80a00cded0 2024-03-24T00:39:09.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.303+0000 7f8097fff640 1 -- 172.21.15.67:0/3062446106 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80980072f0 con 0x7f80a00cded0 2024-03-24T00:39:09.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.303+0000 7f80a5977640 1 -- 172.21.15.67:0/3062446106 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80a00b5400 con 0x7f80a00cded0 2024-03-24T00:39:09.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.304+0000 7f80a5977640 1 -- 172.21.15.67:0/3062446106 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80a004f400 con 0x7f80a00cded0 2024-03-24T00:39:09.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.304+0000 7f8097fff640 1 -- 172.21.15.67:0/3062446106 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f809801a070 con 0x7f80a00cded0 2024-03-24T00:39:09.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.305+0000 7f8097fff640 1 --2- 172.21.15.67:0/3062446106 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f808006bbd0 0x7f808006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:09.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.305+0000 7f8097fff640 1 -- 172.21.15.67:0/3062446106 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f809808a150 con 0x7f80a00cded0 2024-03-24T00:39:09.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.305+0000 7f809e7fc640 1 --2- 172.21.15.67:0/3062446106 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f808006bbd0 0x7f808006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:09.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.305+0000 7f809e7fc640 1 --2- 172.21.15.67:0/3062446106 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f808006bbd0 0x7f808006e090 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8088036000 tx=0x7f808803c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:09.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.309+0000 7f8097fff640 1 -- 172.21.15.67:0/3062446106 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f80980582b0 con 0x7f80a00cded0 2024-03-24T00:39:09.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.491+0000 7f80a5977640 1 -- 172.21.15.67:0/3062446106 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f80a00b6a50 con 0x7f80a00cded0 2024-03-24T00:39:09.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.492+0000 7f8097fff640 1 -- 172.21.15.67:0/3062446106 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18454 (secure 0 0 0) 0x7f809805c350 con 0x7f80a00cded0 2024-03-24T00:39:09.493 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:39:09.493 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":25,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:38:58.797179+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:38:50.077085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:38:51.095454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T00:38:54.807117+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:39:09.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.494+0000 7f80a5977640 1 -- 172.21.15.67:0/3062446106 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f808006bbd0 msgr2=0x7f808006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:09.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.494+0000 7f80a5977640 1 --2- 172.21.15.67:0/3062446106 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f808006bbd0 0x7f808006e090 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8088036000 tx=0x7f808803c040 comp rx=0 tx=0).stop 2024-03-24T00:39:09.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.494+0000 7f80a5977640 1 -- 172.21.15.67:0/3062446106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80a00cded0 msgr2=0x7f80a00b5c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:09.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.494+0000 7f80a5977640 1 --2- 172.21.15.67:0/3062446106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80a00cded0 0x7f80a00b5c00 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f809800c430 tx=0x7f809800c830 comp rx=0 tx=0).stop 2024-03-24T00:39:09.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.494+0000 7f80a5977640 1 -- 172.21.15.67:0/3062446106 shutdown_connections 2024-03-24T00:39:09.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.494+0000 7f80a5977640 1 --2- 172.21.15.67:0/3062446106 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f808006bbd0 0x7f808006e090 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:09.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.494+0000 7f80a5977640 1 --2- 172.21.15.67:0/3062446106 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f80a015e390 0x7f80a014a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:09.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.494+0000 7f80a5977640 1 --2- 172.21.15.67:0/3062446106 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f80a015dfb0 0x7f80a0149ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:09.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.494+0000 7f80a5977640 1 --2- 172.21.15.67:0/3062446106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80a00cded0 0x7f80a00b5c00 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:09.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.494+0000 7f80a5977640 1 -- 172.21.15.67:0/3062446106 >> 172.21.15.67:0/3062446106 conn(0x7f80a0099a30 msgr2=0x7f80a015d1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:09.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.494+0000 7f80a5977640 1 -- 172.21.15.67:0/3062446106 shutdown_connections 2024-03-24T00:39:09.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.494+0000 7f80a5977640 1 -- 172.21.15.67:0/3062446106 wait complete. 2024-03-24T00:39:09.506 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-24T00:39:09.506 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:39:09.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.606+0000 7f6f0d2f1640 1 Processor -- start 2024-03-24T00:39:09.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.607+0000 7f6f0d2f1640 1 -- start start 2024-03-24T00:39:09.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.607+0000 7f6f0d2f1640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f6f0815dfe0 0x7f6f080ceb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:09.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.607+0000 7f6f0d2f1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6f08161bb0 0x7f6f080cf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:09.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.607+0000 7f6f0d2f1640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f08162120 con 0x7f6f080b96d0 2024-03-24T00:39:09.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.607+0000 7f6f0d2f1640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f080d21d0 con 0x7f6f0815dfe0 2024-03-24T00:39:09.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.607+0000 7f6f0d2f1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f08162400 con 0x7f6f08161bb0 2024-03-24T00:39:09.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.607+0000 7f6f067fc640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6f08161bb0 0x7f6f080cf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:09.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.607+0000 7f6f06ffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f6f0815dfe0 0x7f6f080ceb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:09.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.607+0000 7f6f077fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f6f080b96d0 0x7f6f080b57c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:51262/0 (socket says 172.21.15.67:51262) 2024-03-24T00:39:09.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.607+0000 7f6f077fe640 1 -- 172.21.15.67:0/3725983438 learned_addr learned my addr 172.21.15.67:0/3725983438 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:09.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.607+0000 7f6f067fc640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6f08161bb0 0x7f6f080cf050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.67:48876/0 (socket says 172.21.15.67:48876) 2024-03-24T00:39:09.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.608+0000 7f6f05ffb640 1 -- 172.21.15.67:0/3725983438 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1770582045 0 0) 0x7f6f08162120 con 0x7f6f080b96d0 2024-03-24T00:39:09.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.608+0000 7f6f067fc640 1 -- 172.21.15.67:0/3725983438 >> v2:172.21.15.72:3300/0 conn(0x7f6f0815dfe0 msgr2=0x7f6f080ceb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:09.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.608+0000 7f6f067fc640 1 --2- 172.21.15.67:0/3725983438 >> v2:172.21.15.72:3300/0 conn(0x7f6f0815dfe0 0x7f6f080ceb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:09.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.608+0000 7f6f067fc640 1 -- 172.21.15.67:0/3725983438 >> v1:172.21.15.72:6789/0 conn(0x7f6f080b96d0 legacy=0x7f6f080b57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:09.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.608+0000 7f6f067fc640 1 -- 172.21.15.67:0/3725983438 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f081626e0 con 0x7f6f08161bb0 2024-03-24T00:39:09.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.608+0000 7f6f06ffd640 1 --2- 172.21.15.67:0/3725983438 >> v2:172.21.15.72:3300/0 conn(0x7f6f0815dfe0 0x7f6f080ceb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:09.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.609+0000 7f6f067fc640 1 --2- 172.21.15.67:0/3725983438 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6f08161bb0 0x7f6f080cf050 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6ef00097a0 tx=0x7f6ef002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5995e8e71b852be9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:09.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.609+0000 7f6f05ffb640 1 -- 172.21.15.67:0/3725983438 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef0035020 con 0x7f6f08161bb0 2024-03-24T00:39:09.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.609+0000 7f6f05ffb640 1 -- 172.21.15.67:0/3725983438 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ef0031830 con 0x7f6f08161bb0 2024-03-24T00:39:09.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.609+0000 7f6f05ffb640 1 -- 172.21.15.67:0/3725983438 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef0031c00 con 0x7f6f08161bb0 2024-03-24T00:39:09.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.609+0000 7f6f0d2f1640 1 -- 172.21.15.67:0/3725983438 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6f08161bb0 msgr2=0x7f6f080cf050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:09.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.609+0000 7f6f0d2f1640 1 --2- 172.21.15.67:0/3725983438 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6f08161bb0 0x7f6f080cf050 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6ef00097a0 tx=0x7f6ef002b8d0 comp rx=0 tx=0).stop 2024-03-24T00:39:09.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.609+0000 7f6f0d2f1640 1 -- 172.21.15.67:0/3725983438 shutdown_connections 2024-03-24T00:39:09.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.609+0000 7f6f0d2f1640 1 --2- 172.21.15.67:0/3725983438 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6f08161bb0 0x7f6f080cf050 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:09.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.609+0000 7f6f0d2f1640 1 --2- 172.21.15.67:0/3725983438 >> v2:172.21.15.72:3300/0 conn(0x7f6f0815dfe0 0x7f6f080ceb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:09.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.609+0000 7f6f0d2f1640 1 -- 172.21.15.67:0/3725983438 >> 172.21.15.67:0/3725983438 conn(0x7f6f0806c890 msgr2=0x7f6f0806cc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:09.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.609+0000 7f6f0d2f1640 1 -- 172.21.15.67:0/3725983438 shutdown_connections 2024-03-24T00:39:09.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.609+0000 7f6f0d2f1640 1 -- 172.21.15.67:0/3725983438 wait complete. 2024-03-24T00:39:09.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.611+0000 7f6f0d2f1640 1 Processor -- start 2024-03-24T00:39:09.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.611+0000 7f6f0d2f1640 1 -- start start 2024-03-24T00:39:09.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.611+0000 7f6f0d2f1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f080b96d0 0x7f6f08151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:09.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.611+0000 7f6f0d2f1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f0815dfe0 0x7f6f08151a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:09.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.611+0000 7f6f0d2f1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6f08161bb0 0x7f6f08161150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:09.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.611+0000 7f6f0d2f1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f080d1120 con 0x7f6f0815dfe0 2024-03-24T00:39:09.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.611+0000 7f6f0d2f1640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f0815e3c0 con 0x7f6f080b96d0 2024-03-24T00:39:09.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.611+0000 7f6f0d2f1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f080d12a0 con 0x7f6f08161bb0 2024-03-24T00:39:09.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.611+0000 7f6f06ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f080b96d0 0x7f6f08151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:09.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.611+0000 7f6f067fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f0815dfe0 0x7f6f08151a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:09.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.611+0000 7f6f077fe640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6f08161bb0 0x7f6f08161150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:09.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.611+0000 7f6f06ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f080b96d0 0x7f6f08151fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:33982/0 (socket says 172.21.15.67:33982) 2024-03-24T00:39:09.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.611+0000 7f6f06ffd640 1 -- 172.21.15.67:0/3813259970 learned_addr learned my addr 172.21.15.67:0/3813259970 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:09.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.612+0000 7f6f077fe640 1 -- 172.21.15.67:0/3813259970 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f080b96d0 msgr2=0x7f6f08151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:09.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.612+0000 7f6f077fe640 1 --2- 172.21.15.67:0/3813259970 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f080b96d0 0x7f6f08151fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:09.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.612+0000 7f6f077fe640 1 -- 172.21.15.67:0/3813259970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f0815dfe0 msgr2=0x7f6f08151a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:09.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.612+0000 7f6f077fe640 1 --2- 172.21.15.67:0/3813259970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f0815dfe0 0x7f6f08151a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:09.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.612+0000 7f6f077fe640 1 -- 172.21.15.67:0/3813259970 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f08152d10 con 0x7f6f08161bb0 2024-03-24T00:39:09.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.612+0000 7f6f06ffd640 1 --2- 172.21.15.67:0/3813259970 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f080b96d0 0x7f6f08151fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:09.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.612+0000 7f6f077fe640 1 --2- 172.21.15.67:0/3813259970 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6f08161bb0 0x7f6f08161150 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6f00007fd0 tx=0x7f6f0000c870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:09.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.612+0000 7f6eff7fe640 1 -- 172.21.15.67:0/3813259970 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f000074e0 con 0x7f6f08161bb0 2024-03-24T00:39:09.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.612+0000 7f6eff7fe640 1 -- 172.21.15.67:0/3813259970 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f00007680 con 0x7f6f08161bb0 2024-03-24T00:39:09.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.612+0000 7f6eff7fe640 1 -- 172.21.15.67:0/3813259970 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f00004a20 con 0x7f6f08161bb0 2024-03-24T00:39:09.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.612+0000 7f6f0d2f1640 1 -- 172.21.15.67:0/3813259970 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f08152fb0 con 0x7f6f08161bb0 2024-03-24T00:39:09.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.612+0000 7f6f0d2f1640 1 -- 172.21.15.67:0/3813259970 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f08153570 con 0x7f6f08161bb0 2024-03-24T00:39:09.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.613+0000 7f6f0d2f1640 1 -- 172.21.15.67:0/3813259970 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f0806b180 con 0x7f6f08161bb0 2024-03-24T00:39:09.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.614+0000 7f6eff7fe640 1 -- 172.21.15.67:0/3813259970 <== mon.2 v2:172.21.15.72:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6f00011070 con 0x7f6f08161bb0 2024-03-24T00:39:09.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.614+0000 7f6eff7fe640 1 --2- 172.21.15.67:0/3813259970 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6ee806bbd0 0x7f6ee806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:09.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.614+0000 7f6eff7fe640 1 -- 172.21.15.67:0/3813259970 <== mon.2 v2:172.21.15.72:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f6f0008a780 con 0x7f6f08161bb0 2024-03-24T00:39:09.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.614+0000 7f6f06ffd640 1 --2- 172.21.15.67:0/3813259970 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6ee806bbd0 0x7f6ee806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:09.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.614+0000 7f6f06ffd640 1 --2- 172.21.15.67:0/3813259970 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6ee806bbd0 0x7f6ee806e090 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8009bd0 tx=0x7f6ef800b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:09.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.618+0000 7f6eff7fe640 1 -- 172.21.15.67:0/3813259970 <== mon.2 v2:172.21.15.72:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6f000588e0 con 0x7f6f08161bb0 2024-03-24T00:39:09.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.799+0000 7f6f0d2f1640 1 -- 172.21.15.67:0/3813259970 --> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6f0804f400 con 0x7f6ee806bbd0 2024-03-24T00:39:09.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.808+0000 7f6eff7fe640 1 -- 172.21.15.67:0/3813259970 <== mgr.4105 v2:172.21.15.72:6832/2768292274 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+332534 (secure 0 0 0) 0x7f6f0804f400 con 0x7f6ee806bbd0 2024-03-24T00:39:09.812 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:39:09.814 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-24T00:39:09.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.811+0000 7f6f0d2f1640 1 -- 172.21.15.67:0/3813259970 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6ee806bbd0 msgr2=0x7f6ee806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:09.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.811+0000 7f6f0d2f1640 1 --2- 172.21.15.67:0/3813259970 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6ee806bbd0 0x7f6ee806e090 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8009bd0 tx=0x7f6ef800b040 comp rx=0 tx=0).stop 2024-03-24T00:39:09.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.811+0000 7f6f0d2f1640 1 -- 172.21.15.67:0/3813259970 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6f08161bb0 msgr2=0x7f6f08161150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:09.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.811+0000 7f6f0d2f1640 1 --2- 172.21.15.67:0/3813259970 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6f08161bb0 0x7f6f08161150 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6f00007fd0 tx=0x7f6f0000c870 comp rx=0 tx=0).stop 2024-03-24T00:39:09.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.811+0000 7f6f0d2f1640 1 -- 172.21.15.67:0/3813259970 shutdown_connections 2024-03-24T00:39:09.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.811+0000 7f6f0d2f1640 1 --2- 172.21.15.67:0/3813259970 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6ee806bbd0 0x7f6ee806e090 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:09.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.811+0000 7f6f0d2f1640 1 --2- 172.21.15.67:0/3813259970 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6f08161bb0 0x7f6f08161150 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:09.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.811+0000 7f6f0d2f1640 1 --2- 172.21.15.67:0/3813259970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f0815dfe0 0x7f6f08151a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:09.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.811+0000 7f6f0d2f1640 1 --2- 172.21.15.67:0/3813259970 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f080b96d0 0x7f6f08151fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:09.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.811+0000 7f6f0d2f1640 1 -- 172.21.15.67:0/3813259970 >> 172.21.15.67:0/3813259970 conn(0x7f6f0806c890 msgr2=0x7f6f080b8ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:09.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.811+0000 7f6f0d2f1640 1 -- 172.21.15.67:0/3813259970 shutdown_connections 2024-03-24T00:39:09.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.811+0000 7f6f0d2f1640 1 -- 172.21.15.67:0/3813259970 wait complete. 2024-03-24T00:39:09.823 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2024-03-24T00:39:09.327243+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594867,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":206,"num_write_kb":2433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158,"ondisk_log_size":158,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219476,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755244,"statfs":{"total":773094113280,"available":772869369856,"internally_reserved":0,"allocated":3383296,"data_stored":2048878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2089,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.527575"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":28,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676794+0000","last_change":"2024-03-24T00:38:51.779360+0000","last_active":"2024-03-24T00:38:59.676794+0000","last_peered":"2024-03-24T00:38:59.676794+0000","last_clean":"2024-03-24T00:38:59.676794+0000","last_became_active":"2024-03-24T00:38:51.779066+0000","last_became_peered":"2024-03-24T00:38:51.779066+0000","last_unstale":"2024-03-24T00:38:59.676794+0000","last_undegraded":"2024-03-24T00:38:59.676794+0000","last_fullsized":"2024-03-24T00:38:59.676794+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:44:08.508305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800440+0000","last_change":"2024-03-24T00:38:52.781668+0000","last_active":"2024-03-24T00:38:58.800440+0000","last_peered":"2024-03-24T00:38:58.800440+0000","last_clean":"2024-03-24T00:38:58.800440+0000","last_became_active":"2024-03-24T00:38:52.781401+0000","last_became_peered":"2024-03-24T00:38:52.781401+0000","last_unstale":"2024-03-24T00:38:58.800440+0000","last_undegraded":"2024-03-24T00:38:58.800440+0000","last_fullsized":"2024-03-24T00:38:58.800440+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:01:21.675152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800050+0000","last_change":"2024-03-24T00:38:57.197819+0000","last_active":"2024-03-24T00:38:58.800050+0000","last_peered":"2024-03-24T00:38:58.800050+0000","last_clean":"2024-03-24T00:38:58.800050+0000","last_became_active":"2024-03-24T00:38:57.196369+0000","last_became_peered":"2024-03-24T00:38:57.196369+0000","last_unstale":"2024-03-24T00:38:58.800050+0000","last_undegraded":"2024-03-24T00:38:58.800050+0000","last_fullsized":"2024-03-24T00:38:58.800050+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:04:26.722612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800456+0000","last_change":"2024-03-24T00:38:51.772822+0000","last_active":"2024-03-24T00:38:58.800456+0000","last_peered":"2024-03-24T00:38:58.800456+0000","last_clean":"2024-03-24T00:38:58.800456+0000","last_became_active":"2024-03-24T00:38:51.772506+0000","last_became_peered":"2024-03-24T00:38:51.772506+0000","last_unstale":"2024-03-24T00:38:58.800456+0000","last_undegraded":"2024-03-24T00:38:58.800456+0000","last_fullsized":"2024-03-24T00:38:58.800456+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:09:30.121404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676632+0000","last_change":"2024-03-24T00:38:52.778282+0000","last_active":"2024-03-24T00:38:59.676632+0000","last_peered":"2024-03-24T00:38:59.676632+0000","last_clean":"2024-03-24T00:38:59.676632+0000","last_became_active":"2024-03-24T00:38:52.778003+0000","last_became_peered":"2024-03-24T00:38:52.778003+0000","last_unstale":"2024-03-24T00:38:59.676632+0000","last_undegraded":"2024-03-24T00:38:59.676632+0000","last_fullsized":"2024-03-24T00:38:59.676632+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:01:21.675152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801128+0000","last_change":"2024-03-24T00:38:56.822022+0000","last_active":"2024-03-24T00:38:58.801128+0000","last_peered":"2024-03-24T00:38:58.801128+0000","last_clean":"2024-03-24T00:38:58.801128+0000","last_became_active":"2024-03-24T00:38:56.821498+0000","last_became_peered":"2024-03-24T00:38:56.821498+0000","last_unstale":"2024-03-24T00:38:58.801128+0000","last_undegraded":"2024-03-24T00:38:58.801128+0000","last_fullsized":"2024-03-24T00:38:58.801128+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:02:47.160631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.677000+0000","last_change":"2024-03-24T00:38:51.781748+0000","last_active":"2024-03-24T00:38:59.677000+0000","last_peered":"2024-03-24T00:38:59.677000+0000","last_clean":"2024-03-24T00:38:59.677000+0000","last_became_active":"2024-03-24T00:38:51.781399+0000","last_became_peered":"2024-03-24T00:38:51.781399+0000","last_unstale":"2024-03-24T00:38:59.677000+0000","last_undegraded":"2024-03-24T00:38:59.677000+0000","last_fullsized":"2024-03-24T00:38:59.677000+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:51:21.173485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802757+0000","last_change":"2024-03-24T00:38:52.794918+0000","last_active":"2024-03-24T00:38:58.802757+0000","last_peered":"2024-03-24T00:38:58.802757+0000","last_clean":"2024-03-24T00:38:58.802757+0000","last_became_active":"2024-03-24T00:38:52.794638+0000","last_became_peered":"2024-03-24T00:38:52.794638+0000","last_unstale":"2024-03-24T00:38:58.802757+0000","last_undegraded":"2024-03-24T00:38:58.802757+0000","last_fullsized":"2024-03-24T00:38:58.802757+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:03:33.725080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802076+0000","last_change":"2024-03-24T00:38:56.817596+0000","last_active":"2024-03-24T00:38:58.802076+0000","last_peered":"2024-03-24T00:38:58.802076+0000","last_clean":"2024-03-24T00:38:58.802076+0000","last_became_active":"2024-03-24T00:38:56.817300+0000","last_became_peered":"2024-03-24T00:38:56.817300+0000","last_unstale":"2024-03-24T00:38:58.802076+0000","last_undegraded":"2024-03-24T00:38:58.802076+0000","last_fullsized":"2024-03-24T00:38:58.802076+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:53:16.442548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676343+0000","last_change":"2024-03-24T00:38:51.780697+0000","last_active":"2024-03-24T00:38:59.676343+0000","last_peered":"2024-03-24T00:38:59.676343+0000","last_clean":"2024-03-24T00:38:59.676343+0000","last_became_active":"2024-03-24T00:38:51.780118+0000","last_became_peered":"2024-03-24T00:38:51.780118+0000","last_unstale":"2024-03-24T00:38:59.676343+0000","last_undegraded":"2024-03-24T00:38:59.676343+0000","last_fullsized":"2024-03-24T00:38:59.676343+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:53:39.260829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802424+0000","last_change":"2024-03-24T00:38:52.795487+0000","last_active":"2024-03-24T00:38:58.802424+0000","last_peered":"2024-03-24T00:38:58.802424+0000","last_clean":"2024-03-24T00:38:58.802424+0000","last_became_active":"2024-03-24T00:38:52.794564+0000","last_became_peered":"2024-03-24T00:38:52.794564+0000","last_unstale":"2024-03-24T00:38:58.802424+0000","last_undegraded":"2024-03-24T00:38:58.802424+0000","last_fullsized":"2024-03-24T00:38:58.802424+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:48:02.197473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801403+0000","last_change":"2024-03-24T00:38:56.815283+0000","last_active":"2024-03-24T00:38:58.801403+0000","last_peered":"2024-03-24T00:38:58.801403+0000","last_clean":"2024-03-24T00:38:58.801403+0000","last_became_active":"2024-03-24T00:38:56.814490+0000","last_became_peered":"2024-03-24T00:38:56.814490+0000","last_unstale":"2024-03-24T00:38:58.801403+0000","last_undegraded":"2024-03-24T00:38:58.801403+0000","last_fullsized":"2024-03-24T00:38:58.801403+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:08:04.920818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676589+0000","last_change":"2024-03-24T00:38:52.785835+0000","last_active":"2024-03-24T00:38:59.676589+0000","last_peered":"2024-03-24T00:38:59.676589+0000","last_clean":"2024-03-24T00:38:59.676589+0000","last_became_active":"2024-03-24T00:38:52.785539+0000","last_became_peered":"2024-03-24T00:38:52.785539+0000","last_unstale":"2024-03-24T00:38:59.676589+0000","last_undegraded":"2024-03-24T00:38:59.676589+0000","last_fullsized":"2024-03-24T00:38:59.676589+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:01:21.675152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676805+0000","last_change":"2024-03-24T00:38:51.778378+0000","last_active":"2024-03-24T00:38:59.676805+0000","last_peered":"2024-03-24T00:38:59.676805+0000","last_clean":"2024-03-24T00:38:59.676805+0000","last_became_active":"2024-03-24T00:38:51.778093+0000","last_became_peered":"2024-03-24T00:38:51.778093+0000","last_unstale":"2024-03-24T00:38:59.676805+0000","last_undegraded":"2024-03-24T00:38:59.676805+0000","last_fullsized":"2024-03-24T00:38:59.676805+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:05.055760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801870+0000","last_change":"2024-03-24T00:38:56.821541+0000","last_active":"2024-03-24T00:38:58.801870+0000","last_peered":"2024-03-24T00:38:58.801870+0000","last_clean":"2024-03-24T00:38:58.801870+0000","last_became_active":"2024-03-24T00:38:56.821097+0000","last_became_peered":"2024-03-24T00:38:56.821097+0000","last_unstale":"2024-03-24T00:38:58.801870+0000","last_undegraded":"2024-03-24T00:38:58.801870+0000","last_fullsized":"2024-03-24T00:38:58.801870+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:22:37.069261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800616+0000","last_change":"2024-03-24T00:38:52.782691+0000","last_active":"2024-03-24T00:38:58.800616+0000","last_peered":"2024-03-24T00:38:58.800616+0000","last_clean":"2024-03-24T00:38:58.800616+0000","last_became_active":"2024-03-24T00:38:52.782439+0000","last_became_peered":"2024-03-24T00:38:52.782439+0000","last_unstale":"2024-03-24T00:38:58.800616+0000","last_undegraded":"2024-03-24T00:38:58.800616+0000","last_fullsized":"2024-03-24T00:38:58.800616+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:21:00.861116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975579+0000","last_change":"2024-03-24T00:38:51.788163+0000","last_active":"2024-03-24T00:38:58.975579+0000","last_peered":"2024-03-24T00:38:58.975579+0000","last_clean":"2024-03-24T00:38:58.975579+0000","last_became_active":"2024-03-24T00:38:51.787884+0000","last_became_peered":"2024-03-24T00:38:51.787884+0000","last_unstale":"2024-03-24T00:38:58.975579+0000","last_undegraded":"2024-03-24T00:38:58.975579+0000","last_fullsized":"2024-03-24T00:38:58.975579+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:51:21.173485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802074+0000","last_change":"2024-03-24T00:38:56.822013+0000","last_active":"2024-03-24T00:38:58.802074+0000","last_peered":"2024-03-24T00:38:58.802074+0000","last_clean":"2024-03-24T00:38:58.802074+0000","last_became_active":"2024-03-24T00:38:56.821572+0000","last_became_peered":"2024-03-24T00:38:56.821572+0000","last_unstale":"2024-03-24T00:38:58.802074+0000","last_undegraded":"2024-03-24T00:38:58.802074+0000","last_fullsized":"2024-03-24T00:38:58.802074+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:28:30.107102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800068+0000","last_change":"2024-03-24T00:38:52.782995+0000","last_active":"2024-03-24T00:38:58.800068+0000","last_peered":"2024-03-24T00:38:58.800068+0000","last_clean":"2024-03-24T00:38:58.800068+0000","last_became_active":"2024-03-24T00:38:52.782426+0000","last_became_peered":"2024-03-24T00:38:52.782426+0000","last_unstale":"2024-03-24T00:38:58.800068+0000","last_undegraded":"2024-03-24T00:38:58.800068+0000","last_fullsized":"2024-03-24T00:38:58.800068+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:46:05.527509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.974999+0000","last_change":"2024-03-24T00:38:51.787097+0000","last_active":"2024-03-24T00:38:58.974999+0000","last_peered":"2024-03-24T00:38:58.974999+0000","last_clean":"2024-03-24T00:38:58.974999+0000","last_became_active":"2024-03-24T00:38:51.786714+0000","last_became_peered":"2024-03-24T00:38:51.786714+0000","last_unstale":"2024-03-24T00:38:58.974999+0000","last_undegraded":"2024-03-24T00:38:58.974999+0000","last_fullsized":"2024-03-24T00:38:58.974999+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:05.055760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975812+0000","last_change":"2024-03-24T00:38:57.196574+0000","last_active":"2024-03-24T00:38:58.975812+0000","last_peered":"2024-03-24T00:38:58.975812+0000","last_clean":"2024-03-24T00:38:58.975812+0000","last_became_active":"2024-03-24T00:38:57.196154+0000","last_became_peered":"2024-03-24T00:38:57.196154+0000","last_unstale":"2024-03-24T00:38:58.975812+0000","last_undegraded":"2024-03-24T00:38:58.975812+0000","last_fullsized":"2024-03-24T00:38:58.975812+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:01:41.498040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801624+0000","last_change":"2024-03-24T00:38:52.784816+0000","last_active":"2024-03-24T00:38:58.801624+0000","last_peered":"2024-03-24T00:38:58.801624+0000","last_clean":"2024-03-24T00:38:58.801624+0000","last_became_active":"2024-03-24T00:38:52.784492+0000","last_became_peered":"2024-03-24T00:38:52.784492+0000","last_unstale":"2024-03-24T00:38:58.801624+0000","last_undegraded":"2024-03-24T00:38:58.801624+0000","last_fullsized":"2024-03-24T00:38:58.801624+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:23:26.012087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676478+0000","last_change":"2024-03-24T00:38:51.779160+0000","last_active":"2024-03-24T00:38:59.676478+0000","last_peered":"2024-03-24T00:38:59.676478+0000","last_clean":"2024-03-24T00:38:59.676478+0000","last_became_active":"2024-03-24T00:38:51.778859+0000","last_became_peered":"2024-03-24T00:38:51.778859+0000","last_unstale":"2024-03-24T00:38:59.676478+0000","last_undegraded":"2024-03-24T00:38:59.676478+0000","last_fullsized":"2024-03-24T00:38:59.676478+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:09:30.121404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802037+0000","last_change":"2024-03-24T00:38:56.822074+0000","last_active":"2024-03-24T00:38:58.802037+0000","last_peered":"2024-03-24T00:38:58.802037+0000","last_clean":"2024-03-24T00:38:58.802037+0000","last_became_active":"2024-03-24T00:38:56.821310+0000","last_became_peered":"2024-03-24T00:38:56.821310+0000","last_unstale":"2024-03-24T00:38:58.802037+0000","last_undegraded":"2024-03-24T00:38:58.802037+0000","last_fullsized":"2024-03-24T00:38:58.802037+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:17:07.239714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676939+0000","last_change":"2024-03-24T00:38:52.773192+0000","last_active":"2024-03-24T00:38:59.676939+0000","last_peered":"2024-03-24T00:38:59.676939+0000","last_clean":"2024-03-24T00:38:59.676939+0000","last_became_active":"2024-03-24T00:38:52.772910+0000","last_became_peered":"2024-03-24T00:38:52.772910+0000","last_unstale":"2024-03-24T00:38:59.676939+0000","last_undegraded":"2024-03-24T00:38:59.676939+0000","last_fullsized":"2024-03-24T00:38:59.676939+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:46:05.527509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.677252+0000","last_change":"2024-03-24T00:38:51.778294+0000","last_active":"2024-03-24T00:38:59.677252+0000","last_peered":"2024-03-24T00:38:59.677252+0000","last_clean":"2024-03-24T00:38:59.677252+0000","last_became_active":"2024-03-24T00:38:51.777905+0000","last_became_peered":"2024-03-24T00:38:51.777905+0000","last_unstale":"2024-03-24T00:38:59.677252+0000","last_undegraded":"2024-03-24T00:38:59.677252+0000","last_fullsized":"2024-03-24T00:38:59.677252+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:09:30.121404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801340+0000","last_change":"2024-03-24T00:38:57.196338+0000","last_active":"2024-03-24T00:38:58.801340+0000","last_peered":"2024-03-24T00:38:58.801340+0000","last_clean":"2024-03-24T00:38:58.801340+0000","last_became_active":"2024-03-24T00:38:57.195957+0000","last_became_peered":"2024-03-24T00:38:57.195957+0000","last_unstale":"2024-03-24T00:38:58.801340+0000","last_undegraded":"2024-03-24T00:38:58.801340+0000","last_fullsized":"2024-03-24T00:38:58.801340+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:57:34.164907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800713+0000","last_change":"2024-03-24T00:38:52.798013+0000","last_active":"2024-03-24T00:38:58.800713+0000","last_peered":"2024-03-24T00:38:58.800713+0000","last_clean":"2024-03-24T00:38:58.800713+0000","last_became_active":"2024-03-24T00:38:52.797709+0000","last_became_peered":"2024-03-24T00:38:52.797709+0000","last_unstale":"2024-03-24T00:38:58.800713+0000","last_undegraded":"2024-03-24T00:38:58.800713+0000","last_fullsized":"2024-03-24T00:38:58.800713+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:21:12.524662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975704+0000","last_change":"2024-03-24T00:38:51.787052+0000","last_active":"2024-03-24T00:38:58.975704+0000","last_peered":"2024-03-24T00:38:58.975704+0000","last_clean":"2024-03-24T00:38:58.975704+0000","last_became_active":"2024-03-24T00:38:51.786750+0000","last_became_peered":"2024-03-24T00:38:51.786750+0000","last_unstale":"2024-03-24T00:38:58.975704+0000","last_undegraded":"2024-03-24T00:38:58.975704+0000","last_fullsized":"2024-03-24T00:38:58.975704+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:36:18.921609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800214+0000","last_change":"2024-03-24T00:38:57.198329+0000","last_active":"2024-03-24T00:38:58.800214+0000","last_peered":"2024-03-24T00:38:58.800214+0000","last_clean":"2024-03-24T00:38:58.800214+0000","last_became_active":"2024-03-24T00:38:57.196158+0000","last_became_peered":"2024-03-24T00:38:57.196158+0000","last_unstale":"2024-03-24T00:38:58.800214+0000","last_undegraded":"2024-03-24T00:38:58.800214+0000","last_fullsized":"2024-03-24T00:38:58.800214+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:52:58.271700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975518+0000","last_change":"2024-03-24T00:38:52.778532+0000","last_active":"2024-03-24T00:38:58.975518+0000","last_peered":"2024-03-24T00:38:58.975518+0000","last_clean":"2024-03-24T00:38:58.975518+0000","last_became_active":"2024-03-24T00:38:52.778181+0000","last_became_peered":"2024-03-24T00:38:52.778181+0000","last_unstale":"2024-03-24T00:38:58.975518+0000","last_undegraded":"2024-03-24T00:38:58.975518+0000","last_fullsized":"2024-03-24T00:38:58.975518+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:39:26.969331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.677108+0000","last_change":"2024-03-24T00:38:51.779037+0000","last_active":"2024-03-24T00:38:59.677108+0000","last_peered":"2024-03-24T00:38:59.677108+0000","last_clean":"2024-03-24T00:38:59.677108+0000","last_became_active":"2024-03-24T00:38:51.778651+0000","last_became_peered":"2024-03-24T00:38:51.778651+0000","last_unstale":"2024-03-24T00:38:59.677108+0000","last_undegraded":"2024-03-24T00:38:59.677108+0000","last_fullsized":"2024-03-24T00:38:59.677108+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:36:18.921609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676465+0000","last_change":"2024-03-24T00:38:57.196547+0000","last_active":"2024-03-24T00:38:59.676465+0000","last_peered":"2024-03-24T00:38:59.676465+0000","last_clean":"2024-03-24T00:38:59.676465+0000","last_became_active":"2024-03-24T00:38:57.196148+0000","last_became_peered":"2024-03-24T00:38:57.196148+0000","last_unstale":"2024-03-24T00:38:59.676465+0000","last_undegraded":"2024-03-24T00:38:59.676465+0000","last_fullsized":"2024-03-24T00:38:59.676465+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:28:30.107102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802493+0000","last_change":"2024-03-24T00:38:52.789024+0000","last_active":"2024-03-24T00:38:58.802493+0000","last_peered":"2024-03-24T00:38:58.802493+0000","last_clean":"2024-03-24T00:38:58.802493+0000","last_became_active":"2024-03-24T00:38:52.788652+0000","last_became_peered":"2024-03-24T00:38:52.788652+0000","last_unstale":"2024-03-24T00:38:58.802493+0000","last_undegraded":"2024-03-24T00:38:58.802493+0000","last_fullsized":"2024-03-24T00:38:58.802493+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:21:00.861116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975533+0000","last_change":"2024-03-24T00:38:51.787599+0000","last_active":"2024-03-24T00:38:58.975533+0000","last_peered":"2024-03-24T00:38:58.975533+0000","last_clean":"2024-03-24T00:38:58.975533+0000","last_became_active":"2024-03-24T00:38:51.786958+0000","last_became_peered":"2024-03-24T00:38:51.786958+0000","last_unstale":"2024-03-24T00:38:58.975533+0000","last_undegraded":"2024-03-24T00:38:58.975533+0000","last_fullsized":"2024-03-24T00:38:58.975533+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:23:25.006444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975106+0000","last_change":"2024-03-24T00:38:56.822670+0000","last_active":"2024-03-24T00:38:58.975106+0000","last_peered":"2024-03-24T00:38:58.975106+0000","last_clean":"2024-03-24T00:38:58.975106+0000","last_became_active":"2024-03-24T00:38:56.822257+0000","last_became_peered":"2024-03-24T00:38:56.822257+0000","last_unstale":"2024-03-24T00:38:58.975106+0000","last_undegraded":"2024-03-24T00:38:58.975106+0000","last_fullsized":"2024-03-24T00:38:58.975106+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:17:07.239714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802301+0000","last_change":"2024-03-24T00:38:52.786986+0000","last_active":"2024-03-24T00:38:58.802301+0000","last_peered":"2024-03-24T00:38:58.802301+0000","last_clean":"2024-03-24T00:38:58.802301+0000","last_became_active":"2024-03-24T00:38:52.786700+0000","last_became_peered":"2024-03-24T00:38:52.786700+0000","last_unstale":"2024-03-24T00:38:58.802301+0000","last_undegraded":"2024-03-24T00:38:58.802301+0000","last_fullsized":"2024-03-24T00:38:58.802301+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:06.061403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802704+0000","last_change":"2024-03-24T00:38:51.789060+0000","last_active":"2024-03-24T00:38:58.802704+0000","last_peered":"2024-03-24T00:38:58.802704+0000","last_clean":"2024-03-24T00:38:58.802704+0000","last_became_active":"2024-03-24T00:38:51.788751+0000","last_became_peered":"2024-03-24T00:38:51.788751+0000","last_unstale":"2024-03-24T00:38:58.802704+0000","last_undegraded":"2024-03-24T00:38:58.802704+0000","last_fullsized":"2024-03-24T00:38:58.802704+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:09:30.121404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975477+0000","last_change":"2024-03-24T00:38:56.818475+0000","last_active":"2024-03-24T00:38:58.975477+0000","last_peered":"2024-03-24T00:38:58.975477+0000","last_clean":"2024-03-24T00:38:58.975477+0000","last_became_active":"2024-03-24T00:38:56.817655+0000","last_became_peered":"2024-03-24T00:38:56.817655+0000","last_unstale":"2024-03-24T00:38:58.975477+0000","last_undegraded":"2024-03-24T00:38:58.975477+0000","last_fullsized":"2024-03-24T00:38:58.975477+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:53:16.442548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.974995+0000","last_change":"2024-03-24T00:38:52.780020+0000","last_active":"2024-03-24T00:38:58.974995+0000","last_peered":"2024-03-24T00:38:58.974995+0000","last_clean":"2024-03-24T00:38:58.974995+0000","last_became_active":"2024-03-24T00:38:52.779730+0000","last_became_peered":"2024-03-24T00:38:52.779730+0000","last_unstale":"2024-03-24T00:38:58.974995+0000","last_undegraded":"2024-03-24T00:38:58.974995+0000","last_fullsized":"2024-03-24T00:38:58.974995+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:01:21.675152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.675893+0000","last_change":"2024-03-24T00:38:51.771263+0000","last_active":"2024-03-24T00:38:59.675893+0000","last_peered":"2024-03-24T00:38:59.675893+0000","last_clean":"2024-03-24T00:38:59.675893+0000","last_became_active":"2024-03-24T00:38:51.770920+0000","last_became_peered":"2024-03-24T00:38:51.770920+0000","last_unstale":"2024-03-24T00:38:59.675893+0000","last_undegraded":"2024-03-24T00:38:59.675893+0000","last_fullsized":"2024-03-24T00:38:59.675893+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:09:30.121404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975825+0000","last_change":"2024-03-24T00:38:56.822793+0000","last_active":"2024-03-24T00:38:58.975825+0000","last_peered":"2024-03-24T00:38:58.975825+0000","last_clean":"2024-03-24T00:38:58.975825+0000","last_became_active":"2024-03-24T00:38:56.822282+0000","last_became_peered":"2024-03-24T00:38:56.822282+0000","last_unstale":"2024-03-24T00:38:58.975825+0000","last_undegraded":"2024-03-24T00:38:58.975825+0000","last_fullsized":"2024-03-24T00:38:58.975825+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:26:44.414745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676155+0000","last_change":"2024-03-24T00:38:52.782329+0000","last_active":"2024-03-24T00:38:59.676155+0000","last_peered":"2024-03-24T00:38:59.676155+0000","last_clean":"2024-03-24T00:38:59.676155+0000","last_became_active":"2024-03-24T00:38:52.782006+0000","last_became_peered":"2024-03-24T00:38:52.782006+0000","last_unstale":"2024-03-24T00:38:59.676155+0000","last_undegraded":"2024-03-24T00:38:59.676155+0000","last_fullsized":"2024-03-24T00:38:59.676155+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:41:26.388475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676562+0000","last_change":"2024-03-24T00:38:51.787246+0000","last_active":"2024-03-24T00:38:59.676562+0000","last_peered":"2024-03-24T00:38:59.676562+0000","last_clean":"2024-03-24T00:38:59.676562+0000","last_became_active":"2024-03-24T00:38:51.787019+0000","last_became_peered":"2024-03-24T00:38:51.787019+0000","last_unstale":"2024-03-24T00:38:59.676562+0000","last_undegraded":"2024-03-24T00:38:59.676562+0000","last_fullsized":"2024-03-24T00:38:59.676562+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:23:25.006444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676812+0000","last_change":"2024-03-24T00:38:57.196855+0000","last_active":"2024-03-24T00:38:59.676812+0000","last_peered":"2024-03-24T00:38:59.676812+0000","last_clean":"2024-03-24T00:38:59.676812+0000","last_became_active":"2024-03-24T00:38:57.196127+0000","last_became_peered":"2024-03-24T00:38:57.196127+0000","last_unstale":"2024-03-24T00:38:59.676812+0000","last_undegraded":"2024-03-24T00:38:59.676812+0000","last_fullsized":"2024-03-24T00:38:59.676812+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:05:46.867916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800028+0000","last_change":"2024-03-24T00:38:52.792357+0000","last_active":"2024-03-24T00:38:58.800028+0000","last_peered":"2024-03-24T00:38:58.800028+0000","last_clean":"2024-03-24T00:38:58.800028+0000","last_became_active":"2024-03-24T00:38:52.792077+0000","last_became_peered":"2024-03-24T00:38:52.792077+0000","last_unstale":"2024-03-24T00:38:58.800028+0000","last_undegraded":"2024-03-24T00:38:58.800028+0000","last_fullsized":"2024-03-24T00:38:58.800028+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:53:36.522410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802065+0000","last_change":"2024-03-24T00:38:51.777903+0000","last_active":"2024-03-24T00:38:58.802065+0000","last_peered":"2024-03-24T00:38:58.802065+0000","last_clean":"2024-03-24T00:38:58.802065+0000","last_became_active":"2024-03-24T00:38:51.777178+0000","last_became_peered":"2024-03-24T00:38:51.777178+0000","last_unstale":"2024-03-24T00:38:58.802065+0000","last_undegraded":"2024-03-24T00:38:58.802065+0000","last_fullsized":"2024-03-24T00:38:58.802065+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:58:23.822944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":180,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676606+0000","last_change":"2024-03-24T00:38:40.953871+0000","last_active":"2024-03-24T00:38:59.676606+0000","last_peered":"2024-03-24T00:38:59.676606+0000","last_clean":"2024-03-24T00:38:59.676606+0000","last_became_active":"2024-03-24T00:38:40.952926+0000","last_became_peered":"2024-03-24T00:38:40.952926+0000","last_unstale":"2024-03-24T00:38:59.676606+0000","last_undegraded":"2024-03-24T00:38:59.676606+0000","last_fullsized":"2024-03-24T00:38:59.676606+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_clean_scrub_stamp":"2024-03-24T00:38:39.722194+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:05.397257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676410+0000","last_change":"2024-03-24T00:38:57.196309+0000","last_active":"2024-03-24T00:38:59.676410+0000","last_peered":"2024-03-24T00:38:59.676410+0000","last_clean":"2024-03-24T00:38:59.676410+0000","last_became_active":"2024-03-24T00:38:57.195975+0000","last_became_peered":"2024-03-24T00:38:57.195975+0000","last_unstale":"2024-03-24T00:38:59.676410+0000","last_undegraded":"2024-03-24T00:38:59.676410+0000","last_fullsized":"2024-03-24T00:38:59.676410+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:02:47.160631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676898+0000","last_change":"2024-03-24T00:38:52.790380+0000","last_active":"2024-03-24T00:38:59.676898+0000","last_peered":"2024-03-24T00:38:59.676898+0000","last_clean":"2024-03-24T00:38:59.676898+0000","last_became_active":"2024-03-24T00:38:52.788665+0000","last_became_peered":"2024-03-24T00:38:52.788665+0000","last_unstale":"2024-03-24T00:38:59.676898+0000","last_undegraded":"2024-03-24T00:38:59.676898+0000","last_fullsized":"2024-03-24T00:38:59.676898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:03:33.725080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676181+0000","last_change":"2024-03-24T00:38:51.781949+0000","last_active":"2024-03-24T00:38:59.676181+0000","last_peered":"2024-03-24T00:38:59.676181+0000","last_clean":"2024-03-24T00:38:59.676181+0000","last_became_active":"2024-03-24T00:38:51.781662+0000","last_became_peered":"2024-03-24T00:38:51.781662+0000","last_unstale":"2024-03-24T00:38:59.676181+0000","last_undegraded":"2024-03-24T00:38:59.676181+0000","last_fullsized":"2024-03-24T00:38:59.676181+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:59:15.934422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676501+0000","last_change":"2024-03-24T00:38:57.196217+0000","last_active":"2024-03-24T00:38:59.676501+0000","last_peered":"2024-03-24T00:38:59.676501+0000","last_clean":"2024-03-24T00:38:59.676501+0000","last_became_active":"2024-03-24T00:38:57.195860+0000","last_became_peered":"2024-03-24T00:38:57.195860+0000","last_unstale":"2024-03-24T00:38:59.676501+0000","last_undegraded":"2024-03-24T00:38:59.676501+0000","last_fullsized":"2024-03-24T00:38:59.676501+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:17:46.736676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802773+0000","last_change":"2024-03-24T00:38:52.780048+0000","last_active":"2024-03-24T00:38:58.802773+0000","last_peered":"2024-03-24T00:38:58.802773+0000","last_clean":"2024-03-24T00:38:58.802773+0000","last_became_active":"2024-03-24T00:38:52.779706+0000","last_became_peered":"2024-03-24T00:38:52.779706+0000","last_unstale":"2024-03-24T00:38:58.802773+0000","last_undegraded":"2024-03-24T00:38:58.802773+0000","last_fullsized":"2024-03-24T00:38:58.802773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:34:43.724190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975641+0000","last_change":"2024-03-24T00:38:51.789047+0000","last_active":"2024-03-24T00:38:58.975641+0000","last_peered":"2024-03-24T00:38:58.975641+0000","last_clean":"2024-03-24T00:38:58.975641+0000","last_became_active":"2024-03-24T00:38:51.788769+0000","last_became_peered":"2024-03-24T00:38:51.788769+0000","last_unstale":"2024-03-24T00:38:58.975641+0000","last_undegraded":"2024-03-24T00:38:58.975641+0000","last_fullsized":"2024-03-24T00:38:58.975641+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:09:30.121404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676730+0000","last_change":"2024-03-24T00:38:56.822795+0000","last_active":"2024-03-24T00:38:59.676730+0000","last_peered":"2024-03-24T00:38:59.676730+0000","last_clean":"2024-03-24T00:38:59.676730+0000","last_became_active":"2024-03-24T00:38:56.822410+0000","last_became_peered":"2024-03-24T00:38:56.822410+0000","last_unstale":"2024-03-24T00:38:59.676730+0000","last_undegraded":"2024-03-24T00:38:59.676730+0000","last_fullsized":"2024-03-24T00:38:59.676730+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:26:44.414745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.677245+0000","last_change":"2024-03-24T00:38:52.788023+0000","last_active":"2024-03-24T00:38:59.677245+0000","last_peered":"2024-03-24T00:38:59.677245+0000","last_clean":"2024-03-24T00:38:59.677245+0000","last_became_active":"2024-03-24T00:38:52.787528+0000","last_became_peered":"2024-03-24T00:38:52.787528+0000","last_unstale":"2024-03-24T00:38:59.677245+0000","last_undegraded":"2024-03-24T00:38:59.677245+0000","last_fullsized":"2024-03-24T00:38:59.677245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:57:30.131729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676813+0000","last_change":"2024-03-24T00:38:51.771300+0000","last_active":"2024-03-24T00:38:59.676813+0000","last_peered":"2024-03-24T00:38:59.676813+0000","last_clean":"2024-03-24T00:38:59.676813+0000","last_became_active":"2024-03-24T00:38:51.770924+0000","last_became_peered":"2024-03-24T00:38:51.770924+0000","last_unstale":"2024-03-24T00:38:59.676813+0000","last_undegraded":"2024-03-24T00:38:59.676813+0000","last_fullsized":"2024-03-24T00:38:59.676813+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:58:23.822944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801216+0000","last_change":"2024-03-24T00:38:57.196385+0000","last_active":"2024-03-24T00:38:58.801216+0000","last_peered":"2024-03-24T00:38:58.801216+0000","last_clean":"2024-03-24T00:38:58.801216+0000","last_became_active":"2024-03-24T00:38:57.196000+0000","last_became_peered":"2024-03-24T00:38:57.196000+0000","last_unstale":"2024-03-24T00:38:58.801216+0000","last_undegraded":"2024-03-24T00:38:58.801216+0000","last_fullsized":"2024-03-24T00:38:58.801216+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:26:44.414745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.677147+0000","last_change":"2024-03-24T00:38:52.791416+0000","last_active":"2024-03-24T00:38:59.677147+0000","last_peered":"2024-03-24T00:38:59.677147+0000","last_clean":"2024-03-24T00:38:59.677147+0000","last_became_active":"2024-03-24T00:38:52.790334+0000","last_became_peered":"2024-03-24T00:38:52.790334+0000","last_unstale":"2024-03-24T00:38:59.677147+0000","last_undegraded":"2024-03-24T00:38:59.677147+0000","last_fullsized":"2024-03-24T00:38:59.677147+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:34:43.724190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801647+0000","last_change":"2024-03-24T00:38:51.780196+0000","last_active":"2024-03-24T00:38:58.801647+0000","last_peered":"2024-03-24T00:38:58.801647+0000","last_clean":"2024-03-24T00:38:58.801647+0000","last_became_active":"2024-03-24T00:38:51.779921+0000","last_became_peered":"2024-03-24T00:38:51.779921+0000","last_unstale":"2024-03-24T00:38:58.801647+0000","last_undegraded":"2024-03-24T00:38:58.801647+0000","last_fullsized":"2024-03-24T00:38:58.801647+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:52:59.726516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801216+0000","last_change":"2024-03-24T00:38:56.821775+0000","last_active":"2024-03-24T00:38:58.801216+0000","last_peered":"2024-03-24T00:38:58.801216+0000","last_clean":"2024-03-24T00:38:58.801216+0000","last_became_active":"2024-03-24T00:38:56.821188+0000","last_became_peered":"2024-03-24T00:38:56.821188+0000","last_unstale":"2024-03-24T00:38:58.801216+0000","last_undegraded":"2024-03-24T00:38:58.801216+0000","last_fullsized":"2024-03-24T00:38:58.801216+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:17:46.736676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676771+0000","last_change":"2024-03-24T00:38:52.791631+0000","last_active":"2024-03-24T00:38:59.676771+0000","last_peered":"2024-03-24T00:38:59.676771+0000","last_clean":"2024-03-24T00:38:59.676771+0000","last_became_active":"2024-03-24T00:38:52.790109+0000","last_became_peered":"2024-03-24T00:38:52.790109+0000","last_unstale":"2024-03-24T00:38:59.676771+0000","last_undegraded":"2024-03-24T00:38:59.676771+0000","last_fullsized":"2024-03-24T00:38:59.676771+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:53:36.522410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":32,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.675790+0000","last_change":"2024-03-24T00:38:51.789009+0000","last_active":"2024-03-24T00:38:59.675790+0000","last_peered":"2024-03-24T00:38:59.675790+0000","last_clean":"2024-03-24T00:38:59.675790+0000","last_became_active":"2024-03-24T00:38:51.788721+0000","last_became_peered":"2024-03-24T00:38:51.788721+0000","last_unstale":"2024-03-24T00:38:59.675790+0000","last_undegraded":"2024-03-24T00:38:59.675790+0000","last_fullsized":"2024-03-24T00:38:59.675790+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:43:41.080723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676661+0000","last_change":"2024-03-24T00:38:56.815041+0000","last_active":"2024-03-24T00:38:59.676661+0000","last_peered":"2024-03-24T00:38:59.676661+0000","last_clean":"2024-03-24T00:38:59.676661+0000","last_became_active":"2024-03-24T00:38:56.814651+0000","last_became_peered":"2024-03-24T00:38:56.814651+0000","last_unstale":"2024-03-24T00:38:59.676661+0000","last_undegraded":"2024-03-24T00:38:59.676661+0000","last_fullsized":"2024-03-24T00:38:59.676661+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:01:41.498040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676169+0000","last_change":"2024-03-24T00:38:52.779914+0000","last_active":"2024-03-24T00:38:59.676169+0000","last_peered":"2024-03-24T00:38:59.676169+0000","last_clean":"2024-03-24T00:38:59.676169+0000","last_became_active":"2024-03-24T00:38:52.779330+0000","last_became_peered":"2024-03-24T00:38:52.779330+0000","last_unstale":"2024-03-24T00:38:59.676169+0000","last_undegraded":"2024-03-24T00:38:59.676169+0000","last_fullsized":"2024-03-24T00:38:59.676169+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:42:38.046862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676645+0000","last_change":"2024-03-24T00:38:51.781129+0000","last_active":"2024-03-24T00:38:59.676645+0000","last_peered":"2024-03-24T00:38:59.676645+0000","last_clean":"2024-03-24T00:38:59.676645+0000","last_became_active":"2024-03-24T00:38:51.780828+0000","last_became_peered":"2024-03-24T00:38:51.780828+0000","last_unstale":"2024-03-24T00:38:59.676645+0000","last_undegraded":"2024-03-24T00:38:59.676645+0000","last_fullsized":"2024-03-24T00:38:59.676645+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:59:15.934422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800545+0000","last_change":"2024-03-24T00:38:57.197537+0000","last_active":"2024-03-24T00:38:58.800545+0000","last_peered":"2024-03-24T00:38:58.800545+0000","last_clean":"2024-03-24T00:38:58.800545+0000","last_became_active":"2024-03-24T00:38:57.197082+0000","last_became_peered":"2024-03-24T00:38:57.197082+0000","last_unstale":"2024-03-24T00:38:58.800545+0000","last_undegraded":"2024-03-24T00:38:58.800545+0000","last_fullsized":"2024-03-24T00:38:58.800545+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:34:30.798923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676725+0000","last_change":"2024-03-24T00:38:52.789686+0000","last_active":"2024-03-24T00:38:59.676725+0000","last_peered":"2024-03-24T00:38:59.676725+0000","last_clean":"2024-03-24T00:38:59.676725+0000","last_became_active":"2024-03-24T00:38:52.789416+0000","last_became_peered":"2024-03-24T00:38:52.789416+0000","last_unstale":"2024-03-24T00:38:59.676725+0000","last_undegraded":"2024-03-24T00:38:59.676725+0000","last_fullsized":"2024-03-24T00:38:59.676725+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:04:24.260803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802696+0000","last_change":"2024-03-24T00:38:51.777070+0000","last_active":"2024-03-24T00:38:58.802696+0000","last_peered":"2024-03-24T00:38:58.802696+0000","last_clean":"2024-03-24T00:38:58.802696+0000","last_became_active":"2024-03-24T00:38:51.776786+0000","last_became_peered":"2024-03-24T00:38:51.776786+0000","last_unstale":"2024-03-24T00:38:58.802696+0000","last_undegraded":"2024-03-24T00:38:58.802696+0000","last_fullsized":"2024-03-24T00:38:58.802696+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:05.055760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800552+0000","last_change":"2024-03-24T00:38:57.196551+0000","last_active":"2024-03-24T00:38:58.800552+0000","last_peered":"2024-03-24T00:38:58.800552+0000","last_clean":"2024-03-24T00:38:58.800552+0000","last_became_active":"2024-03-24T00:38:57.196138+0000","last_became_peered":"2024-03-24T00:38:57.196138+0000","last_unstale":"2024-03-24T00:38:58.800552+0000","last_undegraded":"2024-03-24T00:38:58.800552+0000","last_fullsized":"2024-03-24T00:38:58.800552+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:47:53.855251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676423+0000","last_change":"2024-03-24T00:38:52.782561+0000","last_active":"2024-03-24T00:38:59.676423+0000","last_peered":"2024-03-24T00:38:59.676423+0000","last_clean":"2024-03-24T00:38:59.676423+0000","last_became_active":"2024-03-24T00:38:52.781961+0000","last_became_peered":"2024-03-24T00:38:52.781961+0000","last_unstale":"2024-03-24T00:38:59.676423+0000","last_undegraded":"2024-03-24T00:38:59.676423+0000","last_fullsized":"2024-03-24T00:38:59.676423+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:42:38.046862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676934+0000","last_change":"2024-03-24T00:38:51.778517+0000","last_active":"2024-03-24T00:38:59.676934+0000","last_peered":"2024-03-24T00:38:59.676934+0000","last_clean":"2024-03-24T00:38:59.676934+0000","last_became_active":"2024-03-24T00:38:51.778218+0000","last_became_peered":"2024-03-24T00:38:51.778218+0000","last_unstale":"2024-03-24T00:38:59.676934+0000","last_undegraded":"2024-03-24T00:38:59.676934+0000","last_fullsized":"2024-03-24T00:38:59.676934+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:01:37.155677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802393+0000","last_change":"2024-03-24T00:38:57.196343+0000","last_active":"2024-03-24T00:38:58.802393+0000","last_peered":"2024-03-24T00:38:58.802393+0000","last_clean":"2024-03-24T00:38:58.802393+0000","last_became_active":"2024-03-24T00:38:57.195953+0000","last_became_peered":"2024-03-24T00:38:57.195953+0000","last_unstale":"2024-03-24T00:38:58.802393+0000","last_undegraded":"2024-03-24T00:38:58.802393+0000","last_fullsized":"2024-03-24T00:38:58.802393+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:05:46.867916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676892+0000","last_change":"2024-03-24T00:38:52.792925+0000","last_active":"2024-03-24T00:38:59.676892+0000","last_peered":"2024-03-24T00:38:59.676892+0000","last_clean":"2024-03-24T00:38:59.676892+0000","last_became_active":"2024-03-24T00:38:52.792661+0000","last_became_peered":"2024-03-24T00:38:52.792661+0000","last_unstale":"2024-03-24T00:38:59.676892+0000","last_undegraded":"2024-03-24T00:38:59.676892+0000","last_fullsized":"2024-03-24T00:38:59.676892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:48:02.197473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'8","reported_seq":32,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:39:01.657516+0000","last_change":"2024-03-24T00:38:51.775463+0000","last_active":"2024-03-24T00:39:01.657516+0000","last_peered":"2024-03-24T00:39:01.657516+0000","last_clean":"2024-03-24T00:39:01.657516+0000","last_became_active":"2024-03-24T00:38:51.775167+0000","last_became_peered":"2024-03-24T00:38:51.775167+0000","last_unstale":"2024-03-24T00:39:01.657516+0000","last_undegraded":"2024-03-24T00:39:01.657516+0000","last_fullsized":"2024-03-24T00:39:01.657516+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:23:25.006444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3701,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975203+0000","last_change":"2024-03-24T00:38:56.819965+0000","last_active":"2024-03-24T00:38:58.975203+0000","last_peered":"2024-03-24T00:38:58.975203+0000","last_clean":"2024-03-24T00:38:58.975203+0000","last_became_active":"2024-03-24T00:38:56.819580+0000","last_became_peered":"2024-03-24T00:38:56.819580+0000","last_unstale":"2024-03-24T00:38:58.975203+0000","last_undegraded":"2024-03-24T00:38:58.975203+0000","last_fullsized":"2024-03-24T00:38:58.975203+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:28:30.107102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802246+0000","last_change":"2024-03-24T00:38:52.790290+0000","last_active":"2024-03-24T00:38:58.802246+0000","last_peered":"2024-03-24T00:38:58.802246+0000","last_clean":"2024-03-24T00:38:58.802246+0000","last_became_active":"2024-03-24T00:38:52.790010+0000","last_became_peered":"2024-03-24T00:38:52.790010+0000","last_unstale":"2024-03-24T00:38:58.802246+0000","last_undegraded":"2024-03-24T00:38:58.802246+0000","last_fullsized":"2024-03-24T00:38:58.802246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:36:19.927252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975485+0000","last_change":"2024-03-24T00:38:51.788684+0000","last_active":"2024-03-24T00:38:58.975485+0000","last_peered":"2024-03-24T00:38:58.975485+0000","last_clean":"2024-03-24T00:38:58.975485+0000","last_became_active":"2024-03-24T00:38:51.788414+0000","last_became_peered":"2024-03-24T00:38:51.788414+0000","last_unstale":"2024-03-24T00:38:58.975485+0000","last_undegraded":"2024-03-24T00:38:58.975485+0000","last_fullsized":"2024-03-24T00:38:58.975485+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:59:15.934422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802225+0000","last_change":"2024-03-24T00:38:56.822362+0000","last_active":"2024-03-24T00:38:58.802225+0000","last_peered":"2024-03-24T00:38:58.802225+0000","last_clean":"2024-03-24T00:38:58.802225+0000","last_became_active":"2024-03-24T00:38:56.821516+0000","last_became_peered":"2024-03-24T00:38:56.821516+0000","last_unstale":"2024-03-24T00:38:58.802225+0000","last_undegraded":"2024-03-24T00:38:58.802225+0000","last_fullsized":"2024-03-24T00:38:58.802225+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:01:41.498040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676720+0000","last_change":"2024-03-24T00:38:52.778191+0000","last_active":"2024-03-24T00:38:59.676720+0000","last_peered":"2024-03-24T00:38:59.676720+0000","last_clean":"2024-03-24T00:38:59.676720+0000","last_became_active":"2024-03-24T00:38:52.777911+0000","last_became_peered":"2024-03-24T00:38:52.777911+0000","last_unstale":"2024-03-24T00:38:59.676720+0000","last_undegraded":"2024-03-24T00:38:59.676720+0000","last_fullsized":"2024-03-24T00:38:59.676720+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:41:59.361242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801321+0000","last_change":"2024-03-24T00:38:51.778447+0000","last_active":"2024-03-24T00:38:58.801321+0000","last_peered":"2024-03-24T00:38:58.801321+0000","last_clean":"2024-03-24T00:38:58.801321+0000","last_became_active":"2024-03-24T00:38:51.778151+0000","last_became_peered":"2024-03-24T00:38:51.778151+0000","last_unstale":"2024-03-24T00:38:58.801321+0000","last_undegraded":"2024-03-24T00:38:58.801321+0000","last_fullsized":"2024-03-24T00:38:58.801321+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:23:25.006444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800051+0000","last_change":"2024-03-24T00:38:56.818959+0000","last_active":"2024-03-24T00:38:58.800051+0000","last_peered":"2024-03-24T00:38:58.800051+0000","last_clean":"2024-03-24T00:38:58.800051+0000","last_became_active":"2024-03-24T00:38:56.818582+0000","last_became_peered":"2024-03-24T00:38:56.818582+0000","last_unstale":"2024-03-24T00:38:58.800051+0000","last_undegraded":"2024-03-24T00:38:58.800051+0000","last_fullsized":"2024-03-24T00:38:58.800051+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:19:21.320451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801240+0000","last_change":"2024-03-24T00:38:52.787525+0000","last_active":"2024-03-24T00:38:58.801240+0000","last_peered":"2024-03-24T00:38:58.801240+0000","last_clean":"2024-03-24T00:38:58.801240+0000","last_became_active":"2024-03-24T00:38:52.787167+0000","last_became_peered":"2024-03-24T00:38:52.787167+0000","last_unstale":"2024-03-24T00:38:58.801240+0000","last_undegraded":"2024-03-24T00:38:58.801240+0000","last_fullsized":"2024-03-24T00:38:58.801240+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:53:36.522410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676940+0000","last_change":"2024-03-24T00:38:51.780424+0000","last_active":"2024-03-24T00:38:59.676940+0000","last_peered":"2024-03-24T00:38:59.676940+0000","last_clean":"2024-03-24T00:38:59.676940+0000","last_became_active":"2024-03-24T00:38:51.780120+0000","last_became_peered":"2024-03-24T00:38:51.780120+0000","last_unstale":"2024-03-24T00:38:59.676940+0000","last_undegraded":"2024-03-24T00:38:59.676940+0000","last_fullsized":"2024-03-24T00:38:59.676940+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:58:23.822944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800129+0000","last_change":"2024-03-24T00:38:57.197505+0000","last_active":"2024-03-24T00:38:58.800129+0000","last_peered":"2024-03-24T00:38:58.800129+0000","last_clean":"2024-03-24T00:38:58.800129+0000","last_became_active":"2024-03-24T00:38:57.197104+0000","last_became_peered":"2024-03-24T00:38:57.197104+0000","last_unstale":"2024-03-24T00:38:58.800129+0000","last_undegraded":"2024-03-24T00:38:58.800129+0000","last_fullsized":"2024-03-24T00:38:58.800129+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:44:07.562150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676497+0000","last_change":"2024-03-24T00:38:57.197065+0000","last_active":"2024-03-24T00:38:59.676497+0000","last_peered":"2024-03-24T00:38:59.676497+0000","last_clean":"2024-03-24T00:38:59.676497+0000","last_became_active":"2024-03-24T00:38:57.196687+0000","last_became_peered":"2024-03-24T00:38:57.196687+0000","last_unstale":"2024-03-24T00:38:59.676497+0000","last_undegraded":"2024-03-24T00:38:59.676497+0000","last_fullsized":"2024-03-24T00:38:59.676497+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:53:16.442548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.974900+0000","last_change":"2024-03-24T00:38:52.790233+0000","last_active":"2024-03-24T00:38:58.974900+0000","last_peered":"2024-03-24T00:38:58.974900+0000","last_clean":"2024-03-24T00:38:58.974900+0000","last_became_active":"2024-03-24T00:38:52.789951+0000","last_became_peered":"2024-03-24T00:38:52.789951+0000","last_unstale":"2024-03-24T00:38:58.974900+0000","last_undegraded":"2024-03-24T00:38:58.974900+0000","last_fullsized":"2024-03-24T00:38:58.974900+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:42:38.046862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801116+0000","last_change":"2024-03-24T00:38:51.775723+0000","last_active":"2024-03-24T00:38:58.801116+0000","last_peered":"2024-03-24T00:38:58.801116+0000","last_clean":"2024-03-24T00:38:58.801116+0000","last_became_active":"2024-03-24T00:38:51.773333+0000","last_became_peered":"2024-03-24T00:38:51.773333+0000","last_unstale":"2024-03-24T00:38:58.801116+0000","last_undegraded":"2024-03-24T00:38:58.801116+0000","last_fullsized":"2024-03-24T00:38:58.801116+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:59:15.934422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676945+0000","last_change":"2024-03-24T00:38:56.819973+0000","last_active":"2024-03-24T00:38:59.676945+0000","last_peered":"2024-03-24T00:38:59.676945+0000","last_clean":"2024-03-24T00:38:59.676945+0000","last_became_active":"2024-03-24T00:38:56.819659+0000","last_became_peered":"2024-03-24T00:38:56.819659+0000","last_unstale":"2024-03-24T00:38:59.676945+0000","last_undegraded":"2024-03-24T00:38:59.676945+0000","last_fullsized":"2024-03-24T00:38:59.676945+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:57:34.164907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.799976+0000","last_change":"2024-03-24T00:38:52.785779+0000","last_active":"2024-03-24T00:38:58.799976+0000","last_peered":"2024-03-24T00:38:58.799976+0000","last_clean":"2024-03-24T00:38:58.799976+0000","last_became_active":"2024-03-24T00:38:52.785184+0000","last_became_peered":"2024-03-24T00:38:52.785184+0000","last_unstale":"2024-03-24T00:38:58.799976+0000","last_undegraded":"2024-03-24T00:38:58.799976+0000","last_fullsized":"2024-03-24T00:38:58.799976+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:39:26.969331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800170+0000","last_change":"2024-03-24T00:38:51.784065+0000","last_active":"2024-03-24T00:38:58.800170+0000","last_peered":"2024-03-24T00:38:58.800170+0000","last_clean":"2024-03-24T00:38:58.800170+0000","last_became_active":"2024-03-24T00:38:51.783789+0000","last_became_peered":"2024-03-24T00:38:51.783789+0000","last_unstale":"2024-03-24T00:38:58.800170+0000","last_undegraded":"2024-03-24T00:38:58.800170+0000","last_fullsized":"2024-03-24T00:38:58.800170+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:23:25.006444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676753+0000","last_change":"2024-03-24T00:38:51.777458+0000","last_active":"2024-03-24T00:38:59.676753+0000","last_peered":"2024-03-24T00:38:59.676753+0000","last_clean":"2024-03-24T00:38:59.676753+0000","last_became_active":"2024-03-24T00:38:51.777189+0000","last_became_peered":"2024-03-24T00:38:51.777189+0000","last_unstale":"2024-03-24T00:38:59.676753+0000","last_undegraded":"2024-03-24T00:38:59.676753+0000","last_fullsized":"2024-03-24T00:38:59.676753+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:05.055760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676860+0000","last_change":"2024-03-24T00:38:52.786644+0000","last_active":"2024-03-24T00:38:59.676860+0000","last_peered":"2024-03-24T00:38:59.676860+0000","last_clean":"2024-03-24T00:38:59.676860+0000","last_became_active":"2024-03-24T00:38:52.786348+0000","last_became_peered":"2024-03-24T00:38:52.786348+0000","last_unstale":"2024-03-24T00:38:59.676860+0000","last_undegraded":"2024-03-24T00:38:59.676860+0000","last_fullsized":"2024-03-24T00:38:59.676860+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:36:54.790308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676080+0000","last_change":"2024-03-24T00:38:56.816034+0000","last_active":"2024-03-24T00:38:59.676080+0000","last_peered":"2024-03-24T00:38:59.676080+0000","last_clean":"2024-03-24T00:38:59.676080+0000","last_became_active":"2024-03-24T00:38:56.815608+0000","last_became_peered":"2024-03-24T00:38:56.815608+0000","last_unstale":"2024-03-24T00:38:59.676080+0000","last_undegraded":"2024-03-24T00:38:59.676080+0000","last_fullsized":"2024-03-24T00:38:59.676080+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:27:27.336047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800713+0000","last_change":"2024-03-24T00:38:51.782948+0000","last_active":"2024-03-24T00:38:58.800713+0000","last_peered":"2024-03-24T00:38:58.800713+0000","last_clean":"2024-03-24T00:38:58.800713+0000","last_became_active":"2024-03-24T00:38:51.780063+0000","last_became_peered":"2024-03-24T00:38:51.780063+0000","last_unstale":"2024-03-24T00:38:58.800713+0000","last_undegraded":"2024-03-24T00:38:58.800713+0000","last_fullsized":"2024-03-24T00:38:58.800713+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:05.055760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801390+0000","last_change":"2024-03-24T00:38:52.778373+0000","last_active":"2024-03-24T00:38:58.801390+0000","last_peered":"2024-03-24T00:38:58.801390+0000","last_clean":"2024-03-24T00:38:58.801390+0000","last_became_active":"2024-03-24T00:38:52.778019+0000","last_became_peered":"2024-03-24T00:38:52.778019+0000","last_unstale":"2024-03-24T00:38:58.801390+0000","last_undegraded":"2024-03-24T00:38:58.801390+0000","last_fullsized":"2024-03-24T00:38:58.801390+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:09:31.127048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800819+0000","last_change":"2024-03-24T00:38:56.818404+0000","last_active":"2024-03-24T00:38:58.800819+0000","last_peered":"2024-03-24T00:38:58.800819+0000","last_clean":"2024-03-24T00:38:58.800819+0000","last_became_active":"2024-03-24T00:38:56.817801+0000","last_became_peered":"2024-03-24T00:38:56.817801+0000","last_unstale":"2024-03-24T00:38:58.800819+0000","last_undegraded":"2024-03-24T00:38:58.800819+0000","last_fullsized":"2024-03-24T00:38:58.800819+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:15:28.193092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676081+0000","last_change":"2024-03-24T00:38:52.780120+0000","last_active":"2024-03-24T00:38:59.676081+0000","last_peered":"2024-03-24T00:38:59.676081+0000","last_clean":"2024-03-24T00:38:59.676081+0000","last_became_active":"2024-03-24T00:38:52.779624+0000","last_became_peered":"2024-03-24T00:38:52.779624+0000","last_unstale":"2024-03-24T00:38:59.676081+0000","last_undegraded":"2024-03-24T00:38:59.676081+0000","last_fullsized":"2024-03-24T00:38:59.676081+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:54:01.232233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676019+0000","last_change":"2024-03-24T00:38:52.783615+0000","last_active":"2024-03-24T00:38:59.676019+0000","last_peered":"2024-03-24T00:38:59.676019+0000","last_clean":"2024-03-24T00:38:59.676019+0000","last_became_active":"2024-03-24T00:38:52.783339+0000","last_became_peered":"2024-03-24T00:38:52.783339+0000","last_unstale":"2024-03-24T00:38:59.676019+0000","last_undegraded":"2024-03-24T00:38:59.676019+0000","last_fullsized":"2024-03-24T00:38:59.676019+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:51:22.179128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802656+0000","last_change":"2024-03-24T00:38:52.793626+0000","last_active":"2024-03-24T00:38:58.802656+0000","last_peered":"2024-03-24T00:38:58.802656+0000","last_clean":"2024-03-24T00:38:58.802656+0000","last_became_active":"2024-03-24T00:38:52.793322+0000","last_became_peered":"2024-03-24T00:38:52.793322+0000","last_unstale":"2024-03-24T00:38:58.802656+0000","last_undegraded":"2024-03-24T00:38:58.802656+0000","last_fullsized":"2024-03-24T00:38:58.802656+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:57:30.131729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676831+0000","last_change":"2024-03-24T00:38:52.791203+0000","last_active":"2024-03-24T00:38:59.676831+0000","last_peered":"2024-03-24T00:38:59.676831+0000","last_clean":"2024-03-24T00:38:59.676831+0000","last_became_active":"2024-03-24T00:38:52.790914+0000","last_became_peered":"2024-03-24T00:38:52.790914+0000","last_unstale":"2024-03-24T00:38:59.676831+0000","last_undegraded":"2024-03-24T00:38:59.676831+0000","last_fullsized":"2024-03-24T00:38:59.676831+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:21:12.524662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800715+0000","last_change":"2024-03-24T00:38:52.788614+0000","last_active":"2024-03-24T00:38:58.800715+0000","last_peered":"2024-03-24T00:38:58.800715+0000","last_clean":"2024-03-24T00:38:58.800715+0000","last_became_active":"2024-03-24T00:38:52.788155+0000","last_became_peered":"2024-03-24T00:38:52.788155+0000","last_unstale":"2024-03-24T00:38:58.800715+0000","last_undegraded":"2024-03-24T00:38:58.800715+0000","last_fullsized":"2024-03-24T00:38:58.800715+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:54:01.232233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975795+0000","last_change":"2024-03-24T00:38:52.790018+0000","last_active":"2024-03-24T00:38:58.975795+0000","last_peered":"2024-03-24T00:38:58.975795+0000","last_clean":"2024-03-24T00:38:58.975795+0000","last_became_active":"2024-03-24T00:38:52.789744+0000","last_became_peered":"2024-03-24T00:38:52.789744+0000","last_unstale":"2024-03-24T00:38:58.975795+0000","last_undegraded":"2024-03-24T00:38:58.975795+0000","last_fullsized":"2024-03-24T00:38:58.975795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:46:05.527509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676904+0000","last_change":"2024-03-24T00:38:52.791896+0000","last_active":"2024-03-24T00:38:59.676904+0000","last_peered":"2024-03-24T00:38:59.676904+0000","last_clean":"2024-03-24T00:38:59.676904+0000","last_became_active":"2024-03-24T00:38:52.789711+0000","last_became_peered":"2024-03-24T00:38:52.789711+0000","last_unstale":"2024-03-24T00:38:59.676904+0000","last_undegraded":"2024-03-24T00:38:59.676904+0000","last_fullsized":"2024-03-24T00:38:59.676904+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:21:12.524662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800598+0000","last_change":"2024-03-24T00:38:52.780901+0000","last_active":"2024-03-24T00:38:58.800598+0000","last_peered":"2024-03-24T00:38:58.800598+0000","last_clean":"2024-03-24T00:38:58.800598+0000","last_became_active":"2024-03-24T00:38:52.780625+0000","last_became_peered":"2024-03-24T00:38:52.780625+0000","last_unstale":"2024-03-24T00:38:58.800598+0000","last_undegraded":"2024-03-24T00:38:58.800598+0000","last_fullsized":"2024-03-24T00:38:58.800598+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:42:38.046862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801027+0000","last_change":"2024-03-24T00:38:52.784339+0000","last_active":"2024-03-24T00:38:58.801027+0000","last_peered":"2024-03-24T00:38:58.801027+0000","last_clean":"2024-03-24T00:38:58.801027+0000","last_became_active":"2024-03-24T00:38:52.783135+0000","last_became_peered":"2024-03-24T00:38:52.783135+0000","last_unstale":"2024-03-24T00:38:58.801027+0000","last_undegraded":"2024-03-24T00:38:58.801027+0000","last_fullsized":"2024-03-24T00:38:58.801027+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:21:12.524662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800355+0000","last_change":"2024-03-24T00:38:52.788394+0000","last_active":"2024-03-24T00:38:58.800355+0000","last_peered":"2024-03-24T00:38:58.800355+0000","last_clean":"2024-03-24T00:38:58.800355+0000","last_became_active":"2024-03-24T00:38:52.788105+0000","last_became_peered":"2024-03-24T00:38:52.788105+0000","last_unstale":"2024-03-24T00:38:58.800355+0000","last_undegraded":"2024-03-24T00:38:58.800355+0000","last_fullsized":"2024-03-24T00:38:58.800355+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:04:24.260803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802590+0000","last_change":"2024-03-24T00:38:52.787942+0000","last_active":"2024-03-24T00:38:58.802590+0000","last_peered":"2024-03-24T00:38:58.802590+0000","last_clean":"2024-03-24T00:38:58.802590+0000","last_became_active":"2024-03-24T00:38:52.787691+0000","last_became_peered":"2024-03-24T00:38:52.787691+0000","last_unstale":"2024-03-24T00:38:58.802590+0000","last_undegraded":"2024-03-24T00:38:58.802590+0000","last_fullsized":"2024-03-24T00:38:58.802590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:17:42.703499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676562+0000","last_change":"2024-03-24T00:38:52.780488+0000","last_active":"2024-03-24T00:38:59.676562+0000","last_peered":"2024-03-24T00:38:59.676562+0000","last_clean":"2024-03-24T00:38:59.676562+0000","last_became_active":"2024-03-24T00:38:52.780227+0000","last_became_peered":"2024-03-24T00:38:52.780227+0000","last_unstale":"2024-03-24T00:38:59.676562+0000","last_undegraded":"2024-03-24T00:38:59.676562+0000","last_fullsized":"2024-03-24T00:38:59.676562+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:39:26.969331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676482+0000","last_change":"2024-03-24T00:38:52.785757+0000","last_active":"2024-03-24T00:38:59.676482+0000","last_peered":"2024-03-24T00:38:59.676482+0000","last_clean":"2024-03-24T00:38:59.676482+0000","last_became_active":"2024-03-24T00:38:52.785476+0000","last_became_peered":"2024-03-24T00:38:52.785476+0000","last_unstale":"2024-03-24T00:38:59.676482+0000","last_undegraded":"2024-03-24T00:38:59.676482+0000","last_fullsized":"2024-03-24T00:38:59.676482+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:39:26.969331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.677056+0000","last_change":"2024-03-24T00:38:52.786263+0000","last_active":"2024-03-24T00:38:59.677056+0000","last_peered":"2024-03-24T00:38:59.677056+0000","last_clean":"2024-03-24T00:38:59.677056+0000","last_became_active":"2024-03-24T00:38:52.785957+0000","last_became_peered":"2024-03-24T00:38:52.785957+0000","last_unstale":"2024-03-24T00:38:59.677056+0000","last_undegraded":"2024-03-24T00:38:59.677056+0000","last_fullsized":"2024-03-24T00:38:59.677056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:02:43.127453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676749+0000","last_change":"2024-03-24T00:38:52.786178+0000","last_active":"2024-03-24T00:38:59.676749+0000","last_peered":"2024-03-24T00:38:59.676749+0000","last_clean":"2024-03-24T00:38:59.676749+0000","last_became_active":"2024-03-24T00:38:52.785894+0000","last_became_peered":"2024-03-24T00:38:52.785894+0000","last_unstale":"2024-03-24T00:38:59.676749+0000","last_undegraded":"2024-03-24T00:38:59.676749+0000","last_fullsized":"2024-03-24T00:38:59.676749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:36:19.927252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801779+0000","last_change":"2024-03-24T00:38:52.787876+0000","last_active":"2024-03-24T00:38:58.801779+0000","last_peered":"2024-03-24T00:38:58.801779+0000","last_clean":"2024-03-24T00:38:58.801779+0000","last_became_active":"2024-03-24T00:38:52.787243+0000","last_became_peered":"2024-03-24T00:38:52.787243+0000","last_unstale":"2024-03-24T00:38:58.801779+0000","last_undegraded":"2024-03-24T00:38:58.801779+0000","last_fullsized":"2024-03-24T00:38:58.801779+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:36:54.790308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800934+0000","last_change":"2024-03-24T00:38:52.776125+0000","last_active":"2024-03-24T00:38:58.800934+0000","last_peered":"2024-03-24T00:38:58.800934+0000","last_clean":"2024-03-24T00:38:58.800934+0000","last_became_active":"2024-03-24T00:38:52.775255+0000","last_became_peered":"2024-03-24T00:38:52.775255+0000","last_unstale":"2024-03-24T00:38:58.800934+0000","last_undegraded":"2024-03-24T00:38:58.800934+0000","last_fullsized":"2024-03-24T00:38:58.800934+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:46:05.527509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.974980+0000","last_change":"2024-03-24T00:38:52.781546+0000","last_active":"2024-03-24T00:38:58.974980+0000","last_peered":"2024-03-24T00:38:58.974980+0000","last_clean":"2024-03-24T00:38:58.974980+0000","last_became_active":"2024-03-24T00:38:52.781220+0000","last_became_peered":"2024-03-24T00:38:52.781220+0000","last_unstale":"2024-03-24T00:38:58.974980+0000","last_undegraded":"2024-03-24T00:38:58.974980+0000","last_fullsized":"2024-03-24T00:38:58.974980+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:41:59.361242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676394+0000","last_change":"2024-03-24T00:38:52.773210+0000","last_active":"2024-03-24T00:38:59.676394+0000","last_peered":"2024-03-24T00:38:59.676394+0000","last_clean":"2024-03-24T00:38:59.676394+0000","last_became_active":"2024-03-24T00:38:52.772887+0000","last_became_peered":"2024-03-24T00:38:52.772887+0000","last_unstale":"2024-03-24T00:38:59.676394+0000","last_undegraded":"2024-03-24T00:38:59.676394+0000","last_fullsized":"2024-03-24T00:38:59.676394+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:27:23.302869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800436+0000","last_change":"2024-03-24T00:38:52.784496+0000","last_active":"2024-03-24T00:38:58.800436+0000","last_peered":"2024-03-24T00:38:58.800436+0000","last_clean":"2024-03-24T00:38:58.800436+0000","last_became_active":"2024-03-24T00:38:52.784234+0000","last_became_peered":"2024-03-24T00:38:52.784234+0000","last_unstale":"2024-03-24T00:38:58.800436+0000","last_undegraded":"2024-03-24T00:38:58.800436+0000","last_fullsized":"2024-03-24T00:38:58.800436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:36:54.790308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802210+0000","last_change":"2024-03-24T00:38:52.786080+0000","last_active":"2024-03-24T00:38:58.802210+0000","last_peered":"2024-03-24T00:38:58.802210+0000","last_clean":"2024-03-24T00:38:58.802210+0000","last_became_active":"2024-03-24T00:38:52.785803+0000","last_became_peered":"2024-03-24T00:38:52.785803+0000","last_unstale":"2024-03-24T00:38:58.802210+0000","last_undegraded":"2024-03-24T00:38:58.802210+0000","last_fullsized":"2024-03-24T00:38:58.802210+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:51:22.179128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676587+0000","last_change":"2024-03-24T00:38:52.788391+0000","last_active":"2024-03-24T00:38:59.676587+0000","last_peered":"2024-03-24T00:38:59.676587+0000","last_clean":"2024-03-24T00:38:59.676587+0000","last_became_active":"2024-03-24T00:38:52.788122+0000","last_became_peered":"2024-03-24T00:38:52.788122+0000","last_unstale":"2024-03-24T00:38:59.676587+0000","last_undegraded":"2024-03-24T00:38:59.676587+0000","last_fullsized":"2024-03-24T00:38:59.676587+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:41:59.361242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676867+0000","last_change":"2024-03-24T00:38:52.787696+0000","last_active":"2024-03-24T00:38:59.676867+0000","last_peered":"2024-03-24T00:38:59.676867+0000","last_clean":"2024-03-24T00:38:59.676867+0000","last_became_active":"2024-03-24T00:38:52.787431+0000","last_became_peered":"2024-03-24T00:38:52.787431+0000","last_unstale":"2024-03-24T00:38:59.676867+0000","last_undegraded":"2024-03-24T00:38:59.676867+0000","last_fullsized":"2024-03-24T00:38:59.676867+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:46:05.527509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.677244+0000","last_change":"2024-03-24T00:38:52.791775+0000","last_active":"2024-03-24T00:38:59.677244+0000","last_peered":"2024-03-24T00:38:59.677244+0000","last_clean":"2024-03-24T00:38:59.677244+0000","last_became_active":"2024-03-24T00:38:52.788742+0000","last_became_peered":"2024-03-24T00:38:52.788742+0000","last_unstale":"2024-03-24T00:38:59.677244+0000","last_undegraded":"2024-03-24T00:38:59.677244+0000","last_fullsized":"2024-03-24T00:38:59.677244+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:08:00.887641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.799866+0000","last_change":"2024-03-24T00:38:52.790297+0000","last_active":"2024-03-24T00:38:58.799866+0000","last_peered":"2024-03-24T00:38:58.799866+0000","last_clean":"2024-03-24T00:38:58.799866+0000","last_became_active":"2024-03-24T00:38:52.789879+0000","last_became_peered":"2024-03-24T00:38:52.789879+0000","last_unstale":"2024-03-24T00:38:58.799866+0000","last_undegraded":"2024-03-24T00:38:58.799866+0000","last_fullsized":"2024-03-24T00:38:58.799866+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:41:59.361242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676732+0000","last_change":"2024-03-24T00:38:52.780267+0000","last_active":"2024-03-24T00:38:59.676732+0000","last_peered":"2024-03-24T00:38:59.676732+0000","last_clean":"2024-03-24T00:38:59.676732+0000","last_became_active":"2024-03-24T00:38:52.779973+0000","last_became_peered":"2024-03-24T00:38:52.779973+0000","last_unstale":"2024-03-24T00:38:59.676732+0000","last_undegraded":"2024-03-24T00:38:59.676732+0000","last_fullsized":"2024-03-24T00:38:59.676732+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:36:54.790308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802578+0000","last_change":"2024-03-24T00:38:52.776890+0000","last_active":"2024-03-24T00:38:58.802578+0000","last_peered":"2024-03-24T00:38:58.802578+0000","last_clean":"2024-03-24T00:38:58.802578+0000","last_became_active":"2024-03-24T00:38:52.776573+0000","last_became_peered":"2024-03-24T00:38:52.776573+0000","last_unstale":"2024-03-24T00:38:58.802578+0000","last_undegraded":"2024-03-24T00:38:58.802578+0000","last_fullsized":"2024-03-24T00:38:58.802578+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:02:43.127453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676561+0000","last_change":"2024-03-24T00:38:52.783960+0000","last_active":"2024-03-24T00:38:59.676561+0000","last_peered":"2024-03-24T00:38:59.676561+0000","last_clean":"2024-03-24T00:38:59.676561+0000","last_became_active":"2024-03-24T00:38:52.783606+0000","last_became_peered":"2024-03-24T00:38:52.783606+0000","last_unstale":"2024-03-24T00:38:59.676561+0000","last_undegraded":"2024-03-24T00:38:59.676561+0000","last_fullsized":"2024-03-24T00:38:59.676561+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:21:00.861116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802672+0000","last_change":"2024-03-24T00:38:52.784222+0000","last_active":"2024-03-24T00:38:58.802672+0000","last_peered":"2024-03-24T00:38:58.802672+0000","last_clean":"2024-03-24T00:38:58.802672+0000","last_became_active":"2024-03-24T00:38:52.783951+0000","last_became_peered":"2024-03-24T00:38:52.783951+0000","last_unstale":"2024-03-24T00:38:58.802672+0000","last_undegraded":"2024-03-24T00:38:58.802672+0000","last_fullsized":"2024-03-24T00:38:58.802672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:04:24.260803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676177+0000","last_change":"2024-03-24T00:38:52.774199+0000","last_active":"2024-03-24T00:38:59.676177+0000","last_peered":"2024-03-24T00:38:59.676177+0000","last_clean":"2024-03-24T00:38:59.676177+0000","last_became_active":"2024-03-24T00:38:52.773914+0000","last_became_peered":"2024-03-24T00:38:52.773914+0000","last_unstale":"2024-03-24T00:38:59.676177+0000","last_undegraded":"2024-03-24T00:38:59.676177+0000","last_fullsized":"2024-03-24T00:38:59.676177+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:54:01.232233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975533+0000","last_change":"2024-03-24T00:38:52.778927+0000","last_active":"2024-03-24T00:38:58.975533+0000","last_peered":"2024-03-24T00:38:58.975533+0000","last_clean":"2024-03-24T00:38:58.975533+0000","last_became_active":"2024-03-24T00:38:52.778655+0000","last_became_peered":"2024-03-24T00:38:52.778655+0000","last_unstale":"2024-03-24T00:38:58.975533+0000","last_undegraded":"2024-03-24T00:38:58.975533+0000","last_fullsized":"2024-03-24T00:38:58.975533+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:21:12.524662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.677156+0000","last_change":"2024-03-24T00:38:52.785456+0000","last_active":"2024-03-24T00:38:59.677156+0000","last_peered":"2024-03-24T00:38:59.677156+0000","last_clean":"2024-03-24T00:38:59.677156+0000","last_became_active":"2024-03-24T00:38:52.785150+0000","last_became_peered":"2024-03-24T00:38:52.785150+0000","last_unstale":"2024-03-24T00:38:59.677156+0000","last_undegraded":"2024-03-24T00:38:59.677156+0000","last_fullsized":"2024-03-24T00:38:59.677156+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:17:42.703499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4499,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":8998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27868,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343972,"statfs":{"total":96636764160,"available":96608227328,"internally_reserved":0,"allocated":864256,"data_stored":698343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":111094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":107483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":278528,"data_stored":107519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":111094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":107393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":274432,"data_stored":107553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27872,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343968,"statfs":{"total":96636764160,"available":96608223232,"internally_reserved":0,"allocated":872448,"data_stored":698399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:39:09.828 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:39:09.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.963+0000 7f74630e8640 1 Processor -- start 2024-03-24T00:39:09.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.963+0000 7f74630e8640 1 -- start start 2024-03-24T00:39:09.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.964+0000 7f74630e8640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f745c161bb0 0x7f745c0d2d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:09.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.964+0000 7f74630e8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f745c162120 con 0x7f745c15dfe0 2024-03-24T00:39:09.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.964+0000 7f74630e8640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f745c162400 con 0x7f745c0cf220 2024-03-24T00:39:09.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.964+0000 7f74630e8640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f745c0d21d0 con 0x7f745c161bb0 2024-03-24T00:39:09.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.964+0000 7f7460e84640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f745c15dfe0 0x7f745c0ceb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45384/0 (socket says 172.21.15.67:45384) 2024-03-24T00:39:09.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.964+0000 7f7460e84640 1 -- 172.21.15.67:0/941049703 learned_addr learned my addr 172.21.15.67:0/941049703 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:09.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.964+0000 7f7461685640 1 --2- 172.21.15.67:0/941049703 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f745c161bb0 0x7f745c0d2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:09.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.964+0000 7f745b7fe640 1 -- 172.21.15.67:0/941049703 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 612933458 0 0) 0x7f745c162400 con 0x7f745c0cf220 2024-03-24T00:39:09.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.964+0000 7f745b7fe640 1 -- 172.21.15.67:0/941049703 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f744c003520 con 0x7f745c0cf220 2024-03-24T00:39:09.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.964+0000 7f745b7fe640 1 -- 172.21.15.67:0/941049703 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2336945230 0 0) 0x7f745c162120 con 0x7f745c15dfe0 2024-03-24T00:39:09.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.964+0000 7f745b7fe640 1 -- 172.21.15.67:0/941049703 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f745c162400 con 0x7f745c15dfe0 2024-03-24T00:39:09.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.964+0000 7f7461685640 1 -- 172.21.15.67:0/941049703 >> v1:172.21.15.72:6789/0 conn(0x7f745c0cf220 legacy=0x7f745c0d2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:09.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.964+0000 7f7461685640 1 -- 172.21.15.67:0/941049703 >> v1:172.21.15.67:6789/0 conn(0x7f745c15dfe0 legacy=0x7f745c0ceb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:09.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.964+0000 7f7461685640 1 -- 172.21.15.67:0/941049703 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f745c1626e0 con 0x7f745c161bb0 2024-03-24T00:39:09.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.965+0000 7f7461685640 1 --2- 172.21.15.67:0/941049703 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f745c161bb0 0x7f745c0d2d70 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7454009dc0 tx=0x7f745402bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62a84aafa2df12ac server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:09.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.965+0000 7f745b7fe640 1 -- 172.21.15.67:0/941049703 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7454039070 con 0x7f745c161bb0 2024-03-24T00:39:09.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.965+0000 7f745b7fe640 1 -- 172.21.15.67:0/941049703 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f745402ccd0 con 0x7f745c161bb0 2024-03-24T00:39:09.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.965+0000 7f745b7fe640 1 -- 172.21.15.67:0/941049703 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7454034660 con 0x7f745c161bb0 2024-03-24T00:39:09.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.966+0000 7f74630e8640 1 -- 172.21.15.67:0/941049703 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f745c161bb0 msgr2=0x7f745c0d2d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:09.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.966+0000 7f74630e8640 1 --2- 172.21.15.67:0/941049703 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f745c161bb0 0x7f745c0d2d70 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7454009dc0 tx=0x7f745402bf20 comp rx=0 tx=0).stop 2024-03-24T00:39:09.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.966+0000 7f74630e8640 1 -- 172.21.15.67:0/941049703 shutdown_connections 2024-03-24T00:39:09.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.966+0000 7f74630e8640 1 --2- 172.21.15.67:0/941049703 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f745c161bb0 0x7f745c0d2d70 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:09.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.966+0000 7f74630e8640 1 -- 172.21.15.67:0/941049703 >> 172.21.15.67:0/941049703 conn(0x7f745c06c890 msgr2=0x7f745c06cc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:09.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.966+0000 7f74630e8640 1 -- 172.21.15.67:0/941049703 shutdown_connections 2024-03-24T00:39:09.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.966+0000 7f74630e8640 1 -- 172.21.15.67:0/941049703 wait complete. 2024-03-24T00:39:09.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.967+0000 7f74630e8640 1 Processor -- start 2024-03-24T00:39:09.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.967+0000 7f74630e8640 1 -- start start 2024-03-24T00:39:09.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.968+0000 7f74630e8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f745c0cf220 0x7f745c15afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:09.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.968+0000 7f74630e8640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f745c15dfe0 0x7f745c0b2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:09.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.968+0000 7f74630e8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f745c161bb0 0x7f745c160740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:09.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.968+0000 7f74630e8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f745c0d1120 con 0x7f745c161bb0 2024-03-24T00:39:09.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.968+0000 7f74630e8640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f745c15e3c0 con 0x7f745c0cf220 2024-03-24T00:39:09.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.968+0000 7f74630e8640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f745c0d12a0 con 0x7f745c15dfe0 2024-03-24T00:39:09.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.968+0000 7f7461685640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f745c161bb0 0x7f745c160740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:09.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.968+0000 7f7461685640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f745c161bb0 0x7f745c160740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60742/0 (socket says 172.21.15.67:60742) 2024-03-24T00:39:09.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.968+0000 7f7461685640 1 -- 172.21.15.67:0/984365312 learned_addr learned my addr 172.21.15.67:0/984365312 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:09.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.968+0000 7f745bfff640 1 --2- 172.21.15.67:0/984365312 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f745c15dfe0 0x7f745c0b2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:09.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.968+0000 7f7460e84640 1 --2- 172.21.15.67:0/984365312 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f745c0cf220 0x7f745c15afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:09.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.968+0000 7f7461685640 1 -- 172.21.15.67:0/984365312 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f745c15dfe0 msgr2=0x7f745c0b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:09.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.968+0000 7f7461685640 1 --2- 172.21.15.67:0/984365312 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f745c15dfe0 0x7f745c0b2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:09.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.968+0000 7f7461685640 1 -- 172.21.15.67:0/984365312 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f745c0cf220 msgr2=0x7f745c15afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:09.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.968+0000 7f7461685640 1 --2- 172.21.15.67:0/984365312 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f745c0cf220 0x7f745c15afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:09.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.968+0000 7f7461685640 1 -- 172.21.15.67:0/984365312 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f745c0b3180 con 0x7f745c161bb0 2024-03-24T00:39:09.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.969+0000 7f7461685640 1 --2- 172.21.15.67:0/984365312 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f745c161bb0 0x7f745c160740 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7454037040 tx=0x7f7454004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:09.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.969+0000 7f74597fa640 1 -- 172.21.15.67:0/984365312 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7454039050 con 0x7f745c161bb0 2024-03-24T00:39:09.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.969+0000 7f74597fa640 1 -- 172.21.15.67:0/984365312 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7454045070 con 0x7f745c161bb0 2024-03-24T00:39:09.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.969+0000 7f74630e8640 1 -- 172.21.15.67:0/984365312 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f745c0b52f0 con 0x7f745c161bb0 2024-03-24T00:39:09.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.969+0000 7f74597fa640 1 -- 172.21.15.67:0/984365312 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7454009450 con 0x7f745c161bb0 2024-03-24T00:39:09.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.969+0000 7f74630e8640 1 -- 172.21.15.67:0/984365312 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f745c0b58b0 con 0x7f745c161bb0 2024-03-24T00:39:09.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.970+0000 7f74630e8640 1 -- 172.21.15.67:0/984365312 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f745c06b180 con 0x7f745c161bb0 2024-03-24T00:39:09.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.970+0000 7f74597fa640 1 -- 172.21.15.67:0/984365312 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f745403e850 con 0x7f745c161bb0 2024-03-24T00:39:09.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.971+0000 7f74597fa640 1 --2- 172.21.15.67:0/984365312 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f744006bbd0 0x7f744006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:09.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.971+0000 7f74597fa640 1 -- 172.21.15.67:0/984365312 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f74540ad990 con 0x7f745c161bb0 2024-03-24T00:39:09.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.971+0000 7f7460e84640 1 --2- 172.21.15.67:0/984365312 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f744006bbd0 0x7f744006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:09.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.971+0000 7f7460e84640 1 --2- 172.21.15.67:0/984365312 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f744006bbd0 0x7f744006e090 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f74500011e0 tx=0x7f7450007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:09.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:09.975+0000 7f74597fa640 1 -- 172.21.15.67:0/984365312 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f745407bc10 con 0x7f745c161bb0 2024-03-24T00:39:10.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.154+0000 7f74630e8640 1 -- 172.21.15.67:0/984365312 --> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f745c068a40 con 0x7f744006bbd0 2024-03-24T00:39:10.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.162+0000 7f74597fa640 1 -- 172.21.15.67:0/984365312 <== mgr.4105 v2:172.21.15.72:6832/2768292274 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+332534 (secure 0 0 0) 0x7f745c068a40 con 0x7f744006bbd0 2024-03-24T00:39:10.166 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:39:10.168 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-24T00:39:10.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.165+0000 7f74630e8640 1 -- 172.21.15.67:0/984365312 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f744006bbd0 msgr2=0x7f744006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:10.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.165+0000 7f74630e8640 1 --2- 172.21.15.67:0/984365312 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f744006bbd0 0x7f744006e090 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f74500011e0 tx=0x7f7450007d30 comp rx=0 tx=0).stop 2024-03-24T00:39:10.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.165+0000 7f74630e8640 1 -- 172.21.15.67:0/984365312 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f745c161bb0 msgr2=0x7f745c160740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:10.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.165+0000 7f74630e8640 1 --2- 172.21.15.67:0/984365312 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f745c161bb0 0x7f745c160740 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7454037040 tx=0x7f7454004a40 comp rx=0 tx=0).stop 2024-03-24T00:39:10.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.165+0000 7f74630e8640 1 -- 172.21.15.67:0/984365312 shutdown_connections 2024-03-24T00:39:10.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.165+0000 7f74630e8640 1 --2- 172.21.15.67:0/984365312 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f744006bbd0 0x7f744006e090 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:10.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.165+0000 7f74630e8640 1 --2- 172.21.15.67:0/984365312 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f745c161bb0 0x7f745c160740 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:10.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.165+0000 7f74630e8640 1 --2- 172.21.15.67:0/984365312 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f745c15dfe0 0x7f745c0b2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:10.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.165+0000 7f74630e8640 1 --2- 172.21.15.67:0/984365312 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f745c0cf220 0x7f745c15afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:10.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.165+0000 7f74630e8640 1 -- 172.21.15.67:0/984365312 >> 172.21.15.67:0/984365312 conn(0x7f745c06c890 msgr2=0x7f745c15bcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:10.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.166+0000 7f74630e8640 1 -- 172.21.15.67:0/984365312 shutdown_connections 2024-03-24T00:39:10.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.166+0000 7f74630e8640 1 -- 172.21.15.67:0/984365312 wait complete. 2024-03-24T00:39:10.177 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2024-03-24T00:39:09.327243+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594867,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":206,"num_write_kb":2433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158,"ondisk_log_size":158,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219476,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755244,"statfs":{"total":773094113280,"available":772869369856,"internally_reserved":0,"allocated":3383296,"data_stored":2048878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2089,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.527575"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":28,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676794+0000","last_change":"2024-03-24T00:38:51.779360+0000","last_active":"2024-03-24T00:38:59.676794+0000","last_peered":"2024-03-24T00:38:59.676794+0000","last_clean":"2024-03-24T00:38:59.676794+0000","last_became_active":"2024-03-24T00:38:51.779066+0000","last_became_peered":"2024-03-24T00:38:51.779066+0000","last_unstale":"2024-03-24T00:38:59.676794+0000","last_undegraded":"2024-03-24T00:38:59.676794+0000","last_fullsized":"2024-03-24T00:38:59.676794+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:44:08.508305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800440+0000","last_change":"2024-03-24T00:38:52.781668+0000","last_active":"2024-03-24T00:38:58.800440+0000","last_peered":"2024-03-24T00:38:58.800440+0000","last_clean":"2024-03-24T00:38:58.800440+0000","last_became_active":"2024-03-24T00:38:52.781401+0000","last_became_peered":"2024-03-24T00:38:52.781401+0000","last_unstale":"2024-03-24T00:38:58.800440+0000","last_undegraded":"2024-03-24T00:38:58.800440+0000","last_fullsized":"2024-03-24T00:38:58.800440+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:01:21.675152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800050+0000","last_change":"2024-03-24T00:38:57.197819+0000","last_active":"2024-03-24T00:38:58.800050+0000","last_peered":"2024-03-24T00:38:58.800050+0000","last_clean":"2024-03-24T00:38:58.800050+0000","last_became_active":"2024-03-24T00:38:57.196369+0000","last_became_peered":"2024-03-24T00:38:57.196369+0000","last_unstale":"2024-03-24T00:38:58.800050+0000","last_undegraded":"2024-03-24T00:38:58.800050+0000","last_fullsized":"2024-03-24T00:38:58.800050+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:04:26.722612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800456+0000","last_change":"2024-03-24T00:38:51.772822+0000","last_active":"2024-03-24T00:38:58.800456+0000","last_peered":"2024-03-24T00:38:58.800456+0000","last_clean":"2024-03-24T00:38:58.800456+0000","last_became_active":"2024-03-24T00:38:51.772506+0000","last_became_peered":"2024-03-24T00:38:51.772506+0000","last_unstale":"2024-03-24T00:38:58.800456+0000","last_undegraded":"2024-03-24T00:38:58.800456+0000","last_fullsized":"2024-03-24T00:38:58.800456+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:09:30.121404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676632+0000","last_change":"2024-03-24T00:38:52.778282+0000","last_active":"2024-03-24T00:38:59.676632+0000","last_peered":"2024-03-24T00:38:59.676632+0000","last_clean":"2024-03-24T00:38:59.676632+0000","last_became_active":"2024-03-24T00:38:52.778003+0000","last_became_peered":"2024-03-24T00:38:52.778003+0000","last_unstale":"2024-03-24T00:38:59.676632+0000","last_undegraded":"2024-03-24T00:38:59.676632+0000","last_fullsized":"2024-03-24T00:38:59.676632+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:01:21.675152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801128+0000","last_change":"2024-03-24T00:38:56.822022+0000","last_active":"2024-03-24T00:38:58.801128+0000","last_peered":"2024-03-24T00:38:58.801128+0000","last_clean":"2024-03-24T00:38:58.801128+0000","last_became_active":"2024-03-24T00:38:56.821498+0000","last_became_peered":"2024-03-24T00:38:56.821498+0000","last_unstale":"2024-03-24T00:38:58.801128+0000","last_undegraded":"2024-03-24T00:38:58.801128+0000","last_fullsized":"2024-03-24T00:38:58.801128+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:02:47.160631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.677000+0000","last_change":"2024-03-24T00:38:51.781748+0000","last_active":"2024-03-24T00:38:59.677000+0000","last_peered":"2024-03-24T00:38:59.677000+0000","last_clean":"2024-03-24T00:38:59.677000+0000","last_became_active":"2024-03-24T00:38:51.781399+0000","last_became_peered":"2024-03-24T00:38:51.781399+0000","last_unstale":"2024-03-24T00:38:59.677000+0000","last_undegraded":"2024-03-24T00:38:59.677000+0000","last_fullsized":"2024-03-24T00:38:59.677000+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:51:21.173485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802757+0000","last_change":"2024-03-24T00:38:52.794918+0000","last_active":"2024-03-24T00:38:58.802757+0000","last_peered":"2024-03-24T00:38:58.802757+0000","last_clean":"2024-03-24T00:38:58.802757+0000","last_became_active":"2024-03-24T00:38:52.794638+0000","last_became_peered":"2024-03-24T00:38:52.794638+0000","last_unstale":"2024-03-24T00:38:58.802757+0000","last_undegraded":"2024-03-24T00:38:58.802757+0000","last_fullsized":"2024-03-24T00:38:58.802757+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:03:33.725080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802076+0000","last_change":"2024-03-24T00:38:56.817596+0000","last_active":"2024-03-24T00:38:58.802076+0000","last_peered":"2024-03-24T00:38:58.802076+0000","last_clean":"2024-03-24T00:38:58.802076+0000","last_became_active":"2024-03-24T00:38:56.817300+0000","last_became_peered":"2024-03-24T00:38:56.817300+0000","last_unstale":"2024-03-24T00:38:58.802076+0000","last_undegraded":"2024-03-24T00:38:58.802076+0000","last_fullsized":"2024-03-24T00:38:58.802076+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:53:16.442548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676343+0000","last_change":"2024-03-24T00:38:51.780697+0000","last_active":"2024-03-24T00:38:59.676343+0000","last_peered":"2024-03-24T00:38:59.676343+0000","last_clean":"2024-03-24T00:38:59.676343+0000","last_became_active":"2024-03-24T00:38:51.780118+0000","last_became_peered":"2024-03-24T00:38:51.780118+0000","last_unstale":"2024-03-24T00:38:59.676343+0000","last_undegraded":"2024-03-24T00:38:59.676343+0000","last_fullsized":"2024-03-24T00:38:59.676343+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:53:39.260829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802424+0000","last_change":"2024-03-24T00:38:52.795487+0000","last_active":"2024-03-24T00:38:58.802424+0000","last_peered":"2024-03-24T00:38:58.802424+0000","last_clean":"2024-03-24T00:38:58.802424+0000","last_became_active":"2024-03-24T00:38:52.794564+0000","last_became_peered":"2024-03-24T00:38:52.794564+0000","last_unstale":"2024-03-24T00:38:58.802424+0000","last_undegraded":"2024-03-24T00:38:58.802424+0000","last_fullsized":"2024-03-24T00:38:58.802424+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:48:02.197473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801403+0000","last_change":"2024-03-24T00:38:56.815283+0000","last_active":"2024-03-24T00:38:58.801403+0000","last_peered":"2024-03-24T00:38:58.801403+0000","last_clean":"2024-03-24T00:38:58.801403+0000","last_became_active":"2024-03-24T00:38:56.814490+0000","last_became_peered":"2024-03-24T00:38:56.814490+0000","last_unstale":"2024-03-24T00:38:58.801403+0000","last_undegraded":"2024-03-24T00:38:58.801403+0000","last_fullsized":"2024-03-24T00:38:58.801403+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:08:04.920818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676589+0000","last_change":"2024-03-24T00:38:52.785835+0000","last_active":"2024-03-24T00:38:59.676589+0000","last_peered":"2024-03-24T00:38:59.676589+0000","last_clean":"2024-03-24T00:38:59.676589+0000","last_became_active":"2024-03-24T00:38:52.785539+0000","last_became_peered":"2024-03-24T00:38:52.785539+0000","last_unstale":"2024-03-24T00:38:59.676589+0000","last_undegraded":"2024-03-24T00:38:59.676589+0000","last_fullsized":"2024-03-24T00:38:59.676589+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:01:21.675152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676805+0000","last_change":"2024-03-24T00:38:51.778378+0000","last_active":"2024-03-24T00:38:59.676805+0000","last_peered":"2024-03-24T00:38:59.676805+0000","last_clean":"2024-03-24T00:38:59.676805+0000","last_became_active":"2024-03-24T00:38:51.778093+0000","last_became_peered":"2024-03-24T00:38:51.778093+0000","last_unstale":"2024-03-24T00:38:59.676805+0000","last_undegraded":"2024-03-24T00:38:59.676805+0000","last_fullsized":"2024-03-24T00:38:59.676805+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:05.055760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801870+0000","last_change":"2024-03-24T00:38:56.821541+0000","last_active":"2024-03-24T00:38:58.801870+0000","last_peered":"2024-03-24T00:38:58.801870+0000","last_clean":"2024-03-24T00:38:58.801870+0000","last_became_active":"2024-03-24T00:38:56.821097+0000","last_became_peered":"2024-03-24T00:38:56.821097+0000","last_unstale":"2024-03-24T00:38:58.801870+0000","last_undegraded":"2024-03-24T00:38:58.801870+0000","last_fullsized":"2024-03-24T00:38:58.801870+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:22:37.069261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800616+0000","last_change":"2024-03-24T00:38:52.782691+0000","last_active":"2024-03-24T00:38:58.800616+0000","last_peered":"2024-03-24T00:38:58.800616+0000","last_clean":"2024-03-24T00:38:58.800616+0000","last_became_active":"2024-03-24T00:38:52.782439+0000","last_became_peered":"2024-03-24T00:38:52.782439+0000","last_unstale":"2024-03-24T00:38:58.800616+0000","last_undegraded":"2024-03-24T00:38:58.800616+0000","last_fullsized":"2024-03-24T00:38:58.800616+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:21:00.861116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975579+0000","last_change":"2024-03-24T00:38:51.788163+0000","last_active":"2024-03-24T00:38:58.975579+0000","last_peered":"2024-03-24T00:38:58.975579+0000","last_clean":"2024-03-24T00:38:58.975579+0000","last_became_active":"2024-03-24T00:38:51.787884+0000","last_became_peered":"2024-03-24T00:38:51.787884+0000","last_unstale":"2024-03-24T00:38:58.975579+0000","last_undegraded":"2024-03-24T00:38:58.975579+0000","last_fullsized":"2024-03-24T00:38:58.975579+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:51:21.173485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802074+0000","last_change":"2024-03-24T00:38:56.822013+0000","last_active":"2024-03-24T00:38:58.802074+0000","last_peered":"2024-03-24T00:38:58.802074+0000","last_clean":"2024-03-24T00:38:58.802074+0000","last_became_active":"2024-03-24T00:38:56.821572+0000","last_became_peered":"2024-03-24T00:38:56.821572+0000","last_unstale":"2024-03-24T00:38:58.802074+0000","last_undegraded":"2024-03-24T00:38:58.802074+0000","last_fullsized":"2024-03-24T00:38:58.802074+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:28:30.107102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800068+0000","last_change":"2024-03-24T00:38:52.782995+0000","last_active":"2024-03-24T00:38:58.800068+0000","last_peered":"2024-03-24T00:38:58.800068+0000","last_clean":"2024-03-24T00:38:58.800068+0000","last_became_active":"2024-03-24T00:38:52.782426+0000","last_became_peered":"2024-03-24T00:38:52.782426+0000","last_unstale":"2024-03-24T00:38:58.800068+0000","last_undegraded":"2024-03-24T00:38:58.800068+0000","last_fullsized":"2024-03-24T00:38:58.800068+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:46:05.527509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.974999+0000","last_change":"2024-03-24T00:38:51.787097+0000","last_active":"2024-03-24T00:38:58.974999+0000","last_peered":"2024-03-24T00:38:58.974999+0000","last_clean":"2024-03-24T00:38:58.974999+0000","last_became_active":"2024-03-24T00:38:51.786714+0000","last_became_peered":"2024-03-24T00:38:51.786714+0000","last_unstale":"2024-03-24T00:38:58.974999+0000","last_undegraded":"2024-03-24T00:38:58.974999+0000","last_fullsized":"2024-03-24T00:38:58.974999+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:05.055760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975812+0000","last_change":"2024-03-24T00:38:57.196574+0000","last_active":"2024-03-24T00:38:58.975812+0000","last_peered":"2024-03-24T00:38:58.975812+0000","last_clean":"2024-03-24T00:38:58.975812+0000","last_became_active":"2024-03-24T00:38:57.196154+0000","last_became_peered":"2024-03-24T00:38:57.196154+0000","last_unstale":"2024-03-24T00:38:58.975812+0000","last_undegraded":"2024-03-24T00:38:58.975812+0000","last_fullsized":"2024-03-24T00:38:58.975812+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:01:41.498040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801624+0000","last_change":"2024-03-24T00:38:52.784816+0000","last_active":"2024-03-24T00:38:58.801624+0000","last_peered":"2024-03-24T00:38:58.801624+0000","last_clean":"2024-03-24T00:38:58.801624+0000","last_became_active":"2024-03-24T00:38:52.784492+0000","last_became_peered":"2024-03-24T00:38:52.784492+0000","last_unstale":"2024-03-24T00:38:58.801624+0000","last_undegraded":"2024-03-24T00:38:58.801624+0000","last_fullsized":"2024-03-24T00:38:58.801624+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:23:26.012087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676478+0000","last_change":"2024-03-24T00:38:51.779160+0000","last_active":"2024-03-24T00:38:59.676478+0000","last_peered":"2024-03-24T00:38:59.676478+0000","last_clean":"2024-03-24T00:38:59.676478+0000","last_became_active":"2024-03-24T00:38:51.778859+0000","last_became_peered":"2024-03-24T00:38:51.778859+0000","last_unstale":"2024-03-24T00:38:59.676478+0000","last_undegraded":"2024-03-24T00:38:59.676478+0000","last_fullsized":"2024-03-24T00:38:59.676478+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:09:30.121404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802037+0000","last_change":"2024-03-24T00:38:56.822074+0000","last_active":"2024-03-24T00:38:58.802037+0000","last_peered":"2024-03-24T00:38:58.802037+0000","last_clean":"2024-03-24T00:38:58.802037+0000","last_became_active":"2024-03-24T00:38:56.821310+0000","last_became_peered":"2024-03-24T00:38:56.821310+0000","last_unstale":"2024-03-24T00:38:58.802037+0000","last_undegraded":"2024-03-24T00:38:58.802037+0000","last_fullsized":"2024-03-24T00:38:58.802037+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:17:07.239714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676939+0000","last_change":"2024-03-24T00:38:52.773192+0000","last_active":"2024-03-24T00:38:59.676939+0000","last_peered":"2024-03-24T00:38:59.676939+0000","last_clean":"2024-03-24T00:38:59.676939+0000","last_became_active":"2024-03-24T00:38:52.772910+0000","last_became_peered":"2024-03-24T00:38:52.772910+0000","last_unstale":"2024-03-24T00:38:59.676939+0000","last_undegraded":"2024-03-24T00:38:59.676939+0000","last_fullsized":"2024-03-24T00:38:59.676939+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:46:05.527509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.677252+0000","last_change":"2024-03-24T00:38:51.778294+0000","last_active":"2024-03-24T00:38:59.677252+0000","last_peered":"2024-03-24T00:38:59.677252+0000","last_clean":"2024-03-24T00:38:59.677252+0000","last_became_active":"2024-03-24T00:38:51.777905+0000","last_became_peered":"2024-03-24T00:38:51.777905+0000","last_unstale":"2024-03-24T00:38:59.677252+0000","last_undegraded":"2024-03-24T00:38:59.677252+0000","last_fullsized":"2024-03-24T00:38:59.677252+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:09:30.121404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801340+0000","last_change":"2024-03-24T00:38:57.196338+0000","last_active":"2024-03-24T00:38:58.801340+0000","last_peered":"2024-03-24T00:38:58.801340+0000","last_clean":"2024-03-24T00:38:58.801340+0000","last_became_active":"2024-03-24T00:38:57.195957+0000","last_became_peered":"2024-03-24T00:38:57.195957+0000","last_unstale":"2024-03-24T00:38:58.801340+0000","last_undegraded":"2024-03-24T00:38:58.801340+0000","last_fullsized":"2024-03-24T00:38:58.801340+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:57:34.164907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800713+0000","last_change":"2024-03-24T00:38:52.798013+0000","last_active":"2024-03-24T00:38:58.800713+0000","last_peered":"2024-03-24T00:38:58.800713+0000","last_clean":"2024-03-24T00:38:58.800713+0000","last_became_active":"2024-03-24T00:38:52.797709+0000","last_became_peered":"2024-03-24T00:38:52.797709+0000","last_unstale":"2024-03-24T00:38:58.800713+0000","last_undegraded":"2024-03-24T00:38:58.800713+0000","last_fullsized":"2024-03-24T00:38:58.800713+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:21:12.524662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975704+0000","last_change":"2024-03-24T00:38:51.787052+0000","last_active":"2024-03-24T00:38:58.975704+0000","last_peered":"2024-03-24T00:38:58.975704+0000","last_clean":"2024-03-24T00:38:58.975704+0000","last_became_active":"2024-03-24T00:38:51.786750+0000","last_became_peered":"2024-03-24T00:38:51.786750+0000","last_unstale":"2024-03-24T00:38:58.975704+0000","last_undegraded":"2024-03-24T00:38:58.975704+0000","last_fullsized":"2024-03-24T00:38:58.975704+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:36:18.921609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800214+0000","last_change":"2024-03-24T00:38:57.198329+0000","last_active":"2024-03-24T00:38:58.800214+0000","last_peered":"2024-03-24T00:38:58.800214+0000","last_clean":"2024-03-24T00:38:58.800214+0000","last_became_active":"2024-03-24T00:38:57.196158+0000","last_became_peered":"2024-03-24T00:38:57.196158+0000","last_unstale":"2024-03-24T00:38:58.800214+0000","last_undegraded":"2024-03-24T00:38:58.800214+0000","last_fullsized":"2024-03-24T00:38:58.800214+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:52:58.271700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975518+0000","last_change":"2024-03-24T00:38:52.778532+0000","last_active":"2024-03-24T00:38:58.975518+0000","last_peered":"2024-03-24T00:38:58.975518+0000","last_clean":"2024-03-24T00:38:58.975518+0000","last_became_active":"2024-03-24T00:38:52.778181+0000","last_became_peered":"2024-03-24T00:38:52.778181+0000","last_unstale":"2024-03-24T00:38:58.975518+0000","last_undegraded":"2024-03-24T00:38:58.975518+0000","last_fullsized":"2024-03-24T00:38:58.975518+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:39:26.969331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.677108+0000","last_change":"2024-03-24T00:38:51.779037+0000","last_active":"2024-03-24T00:38:59.677108+0000","last_peered":"2024-03-24T00:38:59.677108+0000","last_clean":"2024-03-24T00:38:59.677108+0000","last_became_active":"2024-03-24T00:38:51.778651+0000","last_became_peered":"2024-03-24T00:38:51.778651+0000","last_unstale":"2024-03-24T00:38:59.677108+0000","last_undegraded":"2024-03-24T00:38:59.677108+0000","last_fullsized":"2024-03-24T00:38:59.677108+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:36:18.921609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676465+0000","last_change":"2024-03-24T00:38:57.196547+0000","last_active":"2024-03-24T00:38:59.676465+0000","last_peered":"2024-03-24T00:38:59.676465+0000","last_clean":"2024-03-24T00:38:59.676465+0000","last_became_active":"2024-03-24T00:38:57.196148+0000","last_became_peered":"2024-03-24T00:38:57.196148+0000","last_unstale":"2024-03-24T00:38:59.676465+0000","last_undegraded":"2024-03-24T00:38:59.676465+0000","last_fullsized":"2024-03-24T00:38:59.676465+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:28:30.107102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802493+0000","last_change":"2024-03-24T00:38:52.789024+0000","last_active":"2024-03-24T00:38:58.802493+0000","last_peered":"2024-03-24T00:38:58.802493+0000","last_clean":"2024-03-24T00:38:58.802493+0000","last_became_active":"2024-03-24T00:38:52.788652+0000","last_became_peered":"2024-03-24T00:38:52.788652+0000","last_unstale":"2024-03-24T00:38:58.802493+0000","last_undegraded":"2024-03-24T00:38:58.802493+0000","last_fullsized":"2024-03-24T00:38:58.802493+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:21:00.861116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975533+0000","last_change":"2024-03-24T00:38:51.787599+0000","last_active":"2024-03-24T00:38:58.975533+0000","last_peered":"2024-03-24T00:38:58.975533+0000","last_clean":"2024-03-24T00:38:58.975533+0000","last_became_active":"2024-03-24T00:38:51.786958+0000","last_became_peered":"2024-03-24T00:38:51.786958+0000","last_unstale":"2024-03-24T00:38:58.975533+0000","last_undegraded":"2024-03-24T00:38:58.975533+0000","last_fullsized":"2024-03-24T00:38:58.975533+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:23:25.006444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975106+0000","last_change":"2024-03-24T00:38:56.822670+0000","last_active":"2024-03-24T00:38:58.975106+0000","last_peered":"2024-03-24T00:38:58.975106+0000","last_clean":"2024-03-24T00:38:58.975106+0000","last_became_active":"2024-03-24T00:38:56.822257+0000","last_became_peered":"2024-03-24T00:38:56.822257+0000","last_unstale":"2024-03-24T00:38:58.975106+0000","last_undegraded":"2024-03-24T00:38:58.975106+0000","last_fullsized":"2024-03-24T00:38:58.975106+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:17:07.239714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802301+0000","last_change":"2024-03-24T00:38:52.786986+0000","last_active":"2024-03-24T00:38:58.802301+0000","last_peered":"2024-03-24T00:38:58.802301+0000","last_clean":"2024-03-24T00:38:58.802301+0000","last_became_active":"2024-03-24T00:38:52.786700+0000","last_became_peered":"2024-03-24T00:38:52.786700+0000","last_unstale":"2024-03-24T00:38:58.802301+0000","last_undegraded":"2024-03-24T00:38:58.802301+0000","last_fullsized":"2024-03-24T00:38:58.802301+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:06.061403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802704+0000","last_change":"2024-03-24T00:38:51.789060+0000","last_active":"2024-03-24T00:38:58.802704+0000","last_peered":"2024-03-24T00:38:58.802704+0000","last_clean":"2024-03-24T00:38:58.802704+0000","last_became_active":"2024-03-24T00:38:51.788751+0000","last_became_peered":"2024-03-24T00:38:51.788751+0000","last_unstale":"2024-03-24T00:38:58.802704+0000","last_undegraded":"2024-03-24T00:38:58.802704+0000","last_fullsized":"2024-03-24T00:38:58.802704+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:09:30.121404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975477+0000","last_change":"2024-03-24T00:38:56.818475+0000","last_active":"2024-03-24T00:38:58.975477+0000","last_peered":"2024-03-24T00:38:58.975477+0000","last_clean":"2024-03-24T00:38:58.975477+0000","last_became_active":"2024-03-24T00:38:56.817655+0000","last_became_peered":"2024-03-24T00:38:56.817655+0000","last_unstale":"2024-03-24T00:38:58.975477+0000","last_undegraded":"2024-03-24T00:38:58.975477+0000","last_fullsized":"2024-03-24T00:38:58.975477+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:53:16.442548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.974995+0000","last_change":"2024-03-24T00:38:52.780020+0000","last_active":"2024-03-24T00:38:58.974995+0000","last_peered":"2024-03-24T00:38:58.974995+0000","last_clean":"2024-03-24T00:38:58.974995+0000","last_became_active":"2024-03-24T00:38:52.779730+0000","last_became_peered":"2024-03-24T00:38:52.779730+0000","last_unstale":"2024-03-24T00:38:58.974995+0000","last_undegraded":"2024-03-24T00:38:58.974995+0000","last_fullsized":"2024-03-24T00:38:58.974995+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:01:21.675152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.675893+0000","last_change":"2024-03-24T00:38:51.771263+0000","last_active":"2024-03-24T00:38:59.675893+0000","last_peered":"2024-03-24T00:38:59.675893+0000","last_clean":"2024-03-24T00:38:59.675893+0000","last_became_active":"2024-03-24T00:38:51.770920+0000","last_became_peered":"2024-03-24T00:38:51.770920+0000","last_unstale":"2024-03-24T00:38:59.675893+0000","last_undegraded":"2024-03-24T00:38:59.675893+0000","last_fullsized":"2024-03-24T00:38:59.675893+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:09:30.121404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975825+0000","last_change":"2024-03-24T00:38:56.822793+0000","last_active":"2024-03-24T00:38:58.975825+0000","last_peered":"2024-03-24T00:38:58.975825+0000","last_clean":"2024-03-24T00:38:58.975825+0000","last_became_active":"2024-03-24T00:38:56.822282+0000","last_became_peered":"2024-03-24T00:38:56.822282+0000","last_unstale":"2024-03-24T00:38:58.975825+0000","last_undegraded":"2024-03-24T00:38:58.975825+0000","last_fullsized":"2024-03-24T00:38:58.975825+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:26:44.414745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676155+0000","last_change":"2024-03-24T00:38:52.782329+0000","last_active":"2024-03-24T00:38:59.676155+0000","last_peered":"2024-03-24T00:38:59.676155+0000","last_clean":"2024-03-24T00:38:59.676155+0000","last_became_active":"2024-03-24T00:38:52.782006+0000","last_became_peered":"2024-03-24T00:38:52.782006+0000","last_unstale":"2024-03-24T00:38:59.676155+0000","last_undegraded":"2024-03-24T00:38:59.676155+0000","last_fullsized":"2024-03-24T00:38:59.676155+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:41:26.388475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676562+0000","last_change":"2024-03-24T00:38:51.787246+0000","last_active":"2024-03-24T00:38:59.676562+0000","last_peered":"2024-03-24T00:38:59.676562+0000","last_clean":"2024-03-24T00:38:59.676562+0000","last_became_active":"2024-03-24T00:38:51.787019+0000","last_became_peered":"2024-03-24T00:38:51.787019+0000","last_unstale":"2024-03-24T00:38:59.676562+0000","last_undegraded":"2024-03-24T00:38:59.676562+0000","last_fullsized":"2024-03-24T00:38:59.676562+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:23:25.006444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676812+0000","last_change":"2024-03-24T00:38:57.196855+0000","last_active":"2024-03-24T00:38:59.676812+0000","last_peered":"2024-03-24T00:38:59.676812+0000","last_clean":"2024-03-24T00:38:59.676812+0000","last_became_active":"2024-03-24T00:38:57.196127+0000","last_became_peered":"2024-03-24T00:38:57.196127+0000","last_unstale":"2024-03-24T00:38:59.676812+0000","last_undegraded":"2024-03-24T00:38:59.676812+0000","last_fullsized":"2024-03-24T00:38:59.676812+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:05:46.867916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800028+0000","last_change":"2024-03-24T00:38:52.792357+0000","last_active":"2024-03-24T00:38:58.800028+0000","last_peered":"2024-03-24T00:38:58.800028+0000","last_clean":"2024-03-24T00:38:58.800028+0000","last_became_active":"2024-03-24T00:38:52.792077+0000","last_became_peered":"2024-03-24T00:38:52.792077+0000","last_unstale":"2024-03-24T00:38:58.800028+0000","last_undegraded":"2024-03-24T00:38:58.800028+0000","last_fullsized":"2024-03-24T00:38:58.800028+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:53:36.522410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802065+0000","last_change":"2024-03-24T00:38:51.777903+0000","last_active":"2024-03-24T00:38:58.802065+0000","last_peered":"2024-03-24T00:38:58.802065+0000","last_clean":"2024-03-24T00:38:58.802065+0000","last_became_active":"2024-03-24T00:38:51.777178+0000","last_became_peered":"2024-03-24T00:38:51.777178+0000","last_unstale":"2024-03-24T00:38:58.802065+0000","last_undegraded":"2024-03-24T00:38:58.802065+0000","last_fullsized":"2024-03-24T00:38:58.802065+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:58:23.822944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":180,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676606+0000","last_change":"2024-03-24T00:38:40.953871+0000","last_active":"2024-03-24T00:38:59.676606+0000","last_peered":"2024-03-24T00:38:59.676606+0000","last_clean":"2024-03-24T00:38:59.676606+0000","last_became_active":"2024-03-24T00:38:40.952926+0000","last_became_peered":"2024-03-24T00:38:40.952926+0000","last_unstale":"2024-03-24T00:38:59.676606+0000","last_undegraded":"2024-03-24T00:38:59.676606+0000","last_fullsized":"2024-03-24T00:38:59.676606+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_clean_scrub_stamp":"2024-03-24T00:38:39.722194+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:05.397257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676410+0000","last_change":"2024-03-24T00:38:57.196309+0000","last_active":"2024-03-24T00:38:59.676410+0000","last_peered":"2024-03-24T00:38:59.676410+0000","last_clean":"2024-03-24T00:38:59.676410+0000","last_became_active":"2024-03-24T00:38:57.195975+0000","last_became_peered":"2024-03-24T00:38:57.195975+0000","last_unstale":"2024-03-24T00:38:59.676410+0000","last_undegraded":"2024-03-24T00:38:59.676410+0000","last_fullsized":"2024-03-24T00:38:59.676410+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:02:47.160631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676898+0000","last_change":"2024-03-24T00:38:52.790380+0000","last_active":"2024-03-24T00:38:59.676898+0000","last_peered":"2024-03-24T00:38:59.676898+0000","last_clean":"2024-03-24T00:38:59.676898+0000","last_became_active":"2024-03-24T00:38:52.788665+0000","last_became_peered":"2024-03-24T00:38:52.788665+0000","last_unstale":"2024-03-24T00:38:59.676898+0000","last_undegraded":"2024-03-24T00:38:59.676898+0000","last_fullsized":"2024-03-24T00:38:59.676898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:03:33.725080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676181+0000","last_change":"2024-03-24T00:38:51.781949+0000","last_active":"2024-03-24T00:38:59.676181+0000","last_peered":"2024-03-24T00:38:59.676181+0000","last_clean":"2024-03-24T00:38:59.676181+0000","last_became_active":"2024-03-24T00:38:51.781662+0000","last_became_peered":"2024-03-24T00:38:51.781662+0000","last_unstale":"2024-03-24T00:38:59.676181+0000","last_undegraded":"2024-03-24T00:38:59.676181+0000","last_fullsized":"2024-03-24T00:38:59.676181+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:59:15.934422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676501+0000","last_change":"2024-03-24T00:38:57.196217+0000","last_active":"2024-03-24T00:38:59.676501+0000","last_peered":"2024-03-24T00:38:59.676501+0000","last_clean":"2024-03-24T00:38:59.676501+0000","last_became_active":"2024-03-24T00:38:57.195860+0000","last_became_peered":"2024-03-24T00:38:57.195860+0000","last_unstale":"2024-03-24T00:38:59.676501+0000","last_undegraded":"2024-03-24T00:38:59.676501+0000","last_fullsized":"2024-03-24T00:38:59.676501+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:17:46.736676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802773+0000","last_change":"2024-03-24T00:38:52.780048+0000","last_active":"2024-03-24T00:38:58.802773+0000","last_peered":"2024-03-24T00:38:58.802773+0000","last_clean":"2024-03-24T00:38:58.802773+0000","last_became_active":"2024-03-24T00:38:52.779706+0000","last_became_peered":"2024-03-24T00:38:52.779706+0000","last_unstale":"2024-03-24T00:38:58.802773+0000","last_undegraded":"2024-03-24T00:38:58.802773+0000","last_fullsized":"2024-03-24T00:38:58.802773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:34:43.724190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975641+0000","last_change":"2024-03-24T00:38:51.789047+0000","last_active":"2024-03-24T00:38:58.975641+0000","last_peered":"2024-03-24T00:38:58.975641+0000","last_clean":"2024-03-24T00:38:58.975641+0000","last_became_active":"2024-03-24T00:38:51.788769+0000","last_became_peered":"2024-03-24T00:38:51.788769+0000","last_unstale":"2024-03-24T00:38:58.975641+0000","last_undegraded":"2024-03-24T00:38:58.975641+0000","last_fullsized":"2024-03-24T00:38:58.975641+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:09:30.121404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676730+0000","last_change":"2024-03-24T00:38:56.822795+0000","last_active":"2024-03-24T00:38:59.676730+0000","last_peered":"2024-03-24T00:38:59.676730+0000","last_clean":"2024-03-24T00:38:59.676730+0000","last_became_active":"2024-03-24T00:38:56.822410+0000","last_became_peered":"2024-03-24T00:38:56.822410+0000","last_unstale":"2024-03-24T00:38:59.676730+0000","last_undegraded":"2024-03-24T00:38:59.676730+0000","last_fullsized":"2024-03-24T00:38:59.676730+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:26:44.414745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.677245+0000","last_change":"2024-03-24T00:38:52.788023+0000","last_active":"2024-03-24T00:38:59.677245+0000","last_peered":"2024-03-24T00:38:59.677245+0000","last_clean":"2024-03-24T00:38:59.677245+0000","last_became_active":"2024-03-24T00:38:52.787528+0000","last_became_peered":"2024-03-24T00:38:52.787528+0000","last_unstale":"2024-03-24T00:38:59.677245+0000","last_undegraded":"2024-03-24T00:38:59.677245+0000","last_fullsized":"2024-03-24T00:38:59.677245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:57:30.131729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676813+0000","last_change":"2024-03-24T00:38:51.771300+0000","last_active":"2024-03-24T00:38:59.676813+0000","last_peered":"2024-03-24T00:38:59.676813+0000","last_clean":"2024-03-24T00:38:59.676813+0000","last_became_active":"2024-03-24T00:38:51.770924+0000","last_became_peered":"2024-03-24T00:38:51.770924+0000","last_unstale":"2024-03-24T00:38:59.676813+0000","last_undegraded":"2024-03-24T00:38:59.676813+0000","last_fullsized":"2024-03-24T00:38:59.676813+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:58:23.822944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801216+0000","last_change":"2024-03-24T00:38:57.196385+0000","last_active":"2024-03-24T00:38:58.801216+0000","last_peered":"2024-03-24T00:38:58.801216+0000","last_clean":"2024-03-24T00:38:58.801216+0000","last_became_active":"2024-03-24T00:38:57.196000+0000","last_became_peered":"2024-03-24T00:38:57.196000+0000","last_unstale":"2024-03-24T00:38:58.801216+0000","last_undegraded":"2024-03-24T00:38:58.801216+0000","last_fullsized":"2024-03-24T00:38:58.801216+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:26:44.414745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.677147+0000","last_change":"2024-03-24T00:38:52.791416+0000","last_active":"2024-03-24T00:38:59.677147+0000","last_peered":"2024-03-24T00:38:59.677147+0000","last_clean":"2024-03-24T00:38:59.677147+0000","last_became_active":"2024-03-24T00:38:52.790334+0000","last_became_peered":"2024-03-24T00:38:52.790334+0000","last_unstale":"2024-03-24T00:38:59.677147+0000","last_undegraded":"2024-03-24T00:38:59.677147+0000","last_fullsized":"2024-03-24T00:38:59.677147+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:34:43.724190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801647+0000","last_change":"2024-03-24T00:38:51.780196+0000","last_active":"2024-03-24T00:38:58.801647+0000","last_peered":"2024-03-24T00:38:58.801647+0000","last_clean":"2024-03-24T00:38:58.801647+0000","last_became_active":"2024-03-24T00:38:51.779921+0000","last_became_peered":"2024-03-24T00:38:51.779921+0000","last_unstale":"2024-03-24T00:38:58.801647+0000","last_undegraded":"2024-03-24T00:38:58.801647+0000","last_fullsized":"2024-03-24T00:38:58.801647+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:52:59.726516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801216+0000","last_change":"2024-03-24T00:38:56.821775+0000","last_active":"2024-03-24T00:38:58.801216+0000","last_peered":"2024-03-24T00:38:58.801216+0000","last_clean":"2024-03-24T00:38:58.801216+0000","last_became_active":"2024-03-24T00:38:56.821188+0000","last_became_peered":"2024-03-24T00:38:56.821188+0000","last_unstale":"2024-03-24T00:38:58.801216+0000","last_undegraded":"2024-03-24T00:38:58.801216+0000","last_fullsized":"2024-03-24T00:38:58.801216+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:17:46.736676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676771+0000","last_change":"2024-03-24T00:38:52.791631+0000","last_active":"2024-03-24T00:38:59.676771+0000","last_peered":"2024-03-24T00:38:59.676771+0000","last_clean":"2024-03-24T00:38:59.676771+0000","last_became_active":"2024-03-24T00:38:52.790109+0000","last_became_peered":"2024-03-24T00:38:52.790109+0000","last_unstale":"2024-03-24T00:38:59.676771+0000","last_undegraded":"2024-03-24T00:38:59.676771+0000","last_fullsized":"2024-03-24T00:38:59.676771+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:53:36.522410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":32,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.675790+0000","last_change":"2024-03-24T00:38:51.789009+0000","last_active":"2024-03-24T00:38:59.675790+0000","last_peered":"2024-03-24T00:38:59.675790+0000","last_clean":"2024-03-24T00:38:59.675790+0000","last_became_active":"2024-03-24T00:38:51.788721+0000","last_became_peered":"2024-03-24T00:38:51.788721+0000","last_unstale":"2024-03-24T00:38:59.675790+0000","last_undegraded":"2024-03-24T00:38:59.675790+0000","last_fullsized":"2024-03-24T00:38:59.675790+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:43:41.080723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676661+0000","last_change":"2024-03-24T00:38:56.815041+0000","last_active":"2024-03-24T00:38:59.676661+0000","last_peered":"2024-03-24T00:38:59.676661+0000","last_clean":"2024-03-24T00:38:59.676661+0000","last_became_active":"2024-03-24T00:38:56.814651+0000","last_became_peered":"2024-03-24T00:38:56.814651+0000","last_unstale":"2024-03-24T00:38:59.676661+0000","last_undegraded":"2024-03-24T00:38:59.676661+0000","last_fullsized":"2024-03-24T00:38:59.676661+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:01:41.498040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676169+0000","last_change":"2024-03-24T00:38:52.779914+0000","last_active":"2024-03-24T00:38:59.676169+0000","last_peered":"2024-03-24T00:38:59.676169+0000","last_clean":"2024-03-24T00:38:59.676169+0000","last_became_active":"2024-03-24T00:38:52.779330+0000","last_became_peered":"2024-03-24T00:38:52.779330+0000","last_unstale":"2024-03-24T00:38:59.676169+0000","last_undegraded":"2024-03-24T00:38:59.676169+0000","last_fullsized":"2024-03-24T00:38:59.676169+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:42:38.046862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676645+0000","last_change":"2024-03-24T00:38:51.781129+0000","last_active":"2024-03-24T00:38:59.676645+0000","last_peered":"2024-03-24T00:38:59.676645+0000","last_clean":"2024-03-24T00:38:59.676645+0000","last_became_active":"2024-03-24T00:38:51.780828+0000","last_became_peered":"2024-03-24T00:38:51.780828+0000","last_unstale":"2024-03-24T00:38:59.676645+0000","last_undegraded":"2024-03-24T00:38:59.676645+0000","last_fullsized":"2024-03-24T00:38:59.676645+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:59:15.934422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800545+0000","last_change":"2024-03-24T00:38:57.197537+0000","last_active":"2024-03-24T00:38:58.800545+0000","last_peered":"2024-03-24T00:38:58.800545+0000","last_clean":"2024-03-24T00:38:58.800545+0000","last_became_active":"2024-03-24T00:38:57.197082+0000","last_became_peered":"2024-03-24T00:38:57.197082+0000","last_unstale":"2024-03-24T00:38:58.800545+0000","last_undegraded":"2024-03-24T00:38:58.800545+0000","last_fullsized":"2024-03-24T00:38:58.800545+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:34:30.798923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676725+0000","last_change":"2024-03-24T00:38:52.789686+0000","last_active":"2024-03-24T00:38:59.676725+0000","last_peered":"2024-03-24T00:38:59.676725+0000","last_clean":"2024-03-24T00:38:59.676725+0000","last_became_active":"2024-03-24T00:38:52.789416+0000","last_became_peered":"2024-03-24T00:38:52.789416+0000","last_unstale":"2024-03-24T00:38:59.676725+0000","last_undegraded":"2024-03-24T00:38:59.676725+0000","last_fullsized":"2024-03-24T00:38:59.676725+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:04:24.260803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802696+0000","last_change":"2024-03-24T00:38:51.777070+0000","last_active":"2024-03-24T00:38:58.802696+0000","last_peered":"2024-03-24T00:38:58.802696+0000","last_clean":"2024-03-24T00:38:58.802696+0000","last_became_active":"2024-03-24T00:38:51.776786+0000","last_became_peered":"2024-03-24T00:38:51.776786+0000","last_unstale":"2024-03-24T00:38:58.802696+0000","last_undegraded":"2024-03-24T00:38:58.802696+0000","last_fullsized":"2024-03-24T00:38:58.802696+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:05.055760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800552+0000","last_change":"2024-03-24T00:38:57.196551+0000","last_active":"2024-03-24T00:38:58.800552+0000","last_peered":"2024-03-24T00:38:58.800552+0000","last_clean":"2024-03-24T00:38:58.800552+0000","last_became_active":"2024-03-24T00:38:57.196138+0000","last_became_peered":"2024-03-24T00:38:57.196138+0000","last_unstale":"2024-03-24T00:38:58.800552+0000","last_undegraded":"2024-03-24T00:38:58.800552+0000","last_fullsized":"2024-03-24T00:38:58.800552+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:47:53.855251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676423+0000","last_change":"2024-03-24T00:38:52.782561+0000","last_active":"2024-03-24T00:38:59.676423+0000","last_peered":"2024-03-24T00:38:59.676423+0000","last_clean":"2024-03-24T00:38:59.676423+0000","last_became_active":"2024-03-24T00:38:52.781961+0000","last_became_peered":"2024-03-24T00:38:52.781961+0000","last_unstale":"2024-03-24T00:38:59.676423+0000","last_undegraded":"2024-03-24T00:38:59.676423+0000","last_fullsized":"2024-03-24T00:38:59.676423+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:42:38.046862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676934+0000","last_change":"2024-03-24T00:38:51.778517+0000","last_active":"2024-03-24T00:38:59.676934+0000","last_peered":"2024-03-24T00:38:59.676934+0000","last_clean":"2024-03-24T00:38:59.676934+0000","last_became_active":"2024-03-24T00:38:51.778218+0000","last_became_peered":"2024-03-24T00:38:51.778218+0000","last_unstale":"2024-03-24T00:38:59.676934+0000","last_undegraded":"2024-03-24T00:38:59.676934+0000","last_fullsized":"2024-03-24T00:38:59.676934+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:01:37.155677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802393+0000","last_change":"2024-03-24T00:38:57.196343+0000","last_active":"2024-03-24T00:38:58.802393+0000","last_peered":"2024-03-24T00:38:58.802393+0000","last_clean":"2024-03-24T00:38:58.802393+0000","last_became_active":"2024-03-24T00:38:57.195953+0000","last_became_peered":"2024-03-24T00:38:57.195953+0000","last_unstale":"2024-03-24T00:38:58.802393+0000","last_undegraded":"2024-03-24T00:38:58.802393+0000","last_fullsized":"2024-03-24T00:38:58.802393+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:05:46.867916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676892+0000","last_change":"2024-03-24T00:38:52.792925+0000","last_active":"2024-03-24T00:38:59.676892+0000","last_peered":"2024-03-24T00:38:59.676892+0000","last_clean":"2024-03-24T00:38:59.676892+0000","last_became_active":"2024-03-24T00:38:52.792661+0000","last_became_peered":"2024-03-24T00:38:52.792661+0000","last_unstale":"2024-03-24T00:38:59.676892+0000","last_undegraded":"2024-03-24T00:38:59.676892+0000","last_fullsized":"2024-03-24T00:38:59.676892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:48:02.197473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'8","reported_seq":32,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:39:01.657516+0000","last_change":"2024-03-24T00:38:51.775463+0000","last_active":"2024-03-24T00:39:01.657516+0000","last_peered":"2024-03-24T00:39:01.657516+0000","last_clean":"2024-03-24T00:39:01.657516+0000","last_became_active":"2024-03-24T00:38:51.775167+0000","last_became_peered":"2024-03-24T00:38:51.775167+0000","last_unstale":"2024-03-24T00:39:01.657516+0000","last_undegraded":"2024-03-24T00:39:01.657516+0000","last_fullsized":"2024-03-24T00:39:01.657516+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:23:25.006444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3701,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975203+0000","last_change":"2024-03-24T00:38:56.819965+0000","last_active":"2024-03-24T00:38:58.975203+0000","last_peered":"2024-03-24T00:38:58.975203+0000","last_clean":"2024-03-24T00:38:58.975203+0000","last_became_active":"2024-03-24T00:38:56.819580+0000","last_became_peered":"2024-03-24T00:38:56.819580+0000","last_unstale":"2024-03-24T00:38:58.975203+0000","last_undegraded":"2024-03-24T00:38:58.975203+0000","last_fullsized":"2024-03-24T00:38:58.975203+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:28:30.107102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802246+0000","last_change":"2024-03-24T00:38:52.790290+0000","last_active":"2024-03-24T00:38:58.802246+0000","last_peered":"2024-03-24T00:38:58.802246+0000","last_clean":"2024-03-24T00:38:58.802246+0000","last_became_active":"2024-03-24T00:38:52.790010+0000","last_became_peered":"2024-03-24T00:38:52.790010+0000","last_unstale":"2024-03-24T00:38:58.802246+0000","last_undegraded":"2024-03-24T00:38:58.802246+0000","last_fullsized":"2024-03-24T00:38:58.802246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:36:19.927252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975485+0000","last_change":"2024-03-24T00:38:51.788684+0000","last_active":"2024-03-24T00:38:58.975485+0000","last_peered":"2024-03-24T00:38:58.975485+0000","last_clean":"2024-03-24T00:38:58.975485+0000","last_became_active":"2024-03-24T00:38:51.788414+0000","last_became_peered":"2024-03-24T00:38:51.788414+0000","last_unstale":"2024-03-24T00:38:58.975485+0000","last_undegraded":"2024-03-24T00:38:58.975485+0000","last_fullsized":"2024-03-24T00:38:58.975485+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:59:15.934422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802225+0000","last_change":"2024-03-24T00:38:56.822362+0000","last_active":"2024-03-24T00:38:58.802225+0000","last_peered":"2024-03-24T00:38:58.802225+0000","last_clean":"2024-03-24T00:38:58.802225+0000","last_became_active":"2024-03-24T00:38:56.821516+0000","last_became_peered":"2024-03-24T00:38:56.821516+0000","last_unstale":"2024-03-24T00:38:58.802225+0000","last_undegraded":"2024-03-24T00:38:58.802225+0000","last_fullsized":"2024-03-24T00:38:58.802225+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:01:41.498040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676720+0000","last_change":"2024-03-24T00:38:52.778191+0000","last_active":"2024-03-24T00:38:59.676720+0000","last_peered":"2024-03-24T00:38:59.676720+0000","last_clean":"2024-03-24T00:38:59.676720+0000","last_became_active":"2024-03-24T00:38:52.777911+0000","last_became_peered":"2024-03-24T00:38:52.777911+0000","last_unstale":"2024-03-24T00:38:59.676720+0000","last_undegraded":"2024-03-24T00:38:59.676720+0000","last_fullsized":"2024-03-24T00:38:59.676720+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:41:59.361242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801321+0000","last_change":"2024-03-24T00:38:51.778447+0000","last_active":"2024-03-24T00:38:58.801321+0000","last_peered":"2024-03-24T00:38:58.801321+0000","last_clean":"2024-03-24T00:38:58.801321+0000","last_became_active":"2024-03-24T00:38:51.778151+0000","last_became_peered":"2024-03-24T00:38:51.778151+0000","last_unstale":"2024-03-24T00:38:58.801321+0000","last_undegraded":"2024-03-24T00:38:58.801321+0000","last_fullsized":"2024-03-24T00:38:58.801321+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:23:25.006444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800051+0000","last_change":"2024-03-24T00:38:56.818959+0000","last_active":"2024-03-24T00:38:58.800051+0000","last_peered":"2024-03-24T00:38:58.800051+0000","last_clean":"2024-03-24T00:38:58.800051+0000","last_became_active":"2024-03-24T00:38:56.818582+0000","last_became_peered":"2024-03-24T00:38:56.818582+0000","last_unstale":"2024-03-24T00:38:58.800051+0000","last_undegraded":"2024-03-24T00:38:58.800051+0000","last_fullsized":"2024-03-24T00:38:58.800051+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:19:21.320451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801240+0000","last_change":"2024-03-24T00:38:52.787525+0000","last_active":"2024-03-24T00:38:58.801240+0000","last_peered":"2024-03-24T00:38:58.801240+0000","last_clean":"2024-03-24T00:38:58.801240+0000","last_became_active":"2024-03-24T00:38:52.787167+0000","last_became_peered":"2024-03-24T00:38:52.787167+0000","last_unstale":"2024-03-24T00:38:58.801240+0000","last_undegraded":"2024-03-24T00:38:58.801240+0000","last_fullsized":"2024-03-24T00:38:58.801240+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:53:36.522410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676940+0000","last_change":"2024-03-24T00:38:51.780424+0000","last_active":"2024-03-24T00:38:59.676940+0000","last_peered":"2024-03-24T00:38:59.676940+0000","last_clean":"2024-03-24T00:38:59.676940+0000","last_became_active":"2024-03-24T00:38:51.780120+0000","last_became_peered":"2024-03-24T00:38:51.780120+0000","last_unstale":"2024-03-24T00:38:59.676940+0000","last_undegraded":"2024-03-24T00:38:59.676940+0000","last_fullsized":"2024-03-24T00:38:59.676940+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:58:23.822944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800129+0000","last_change":"2024-03-24T00:38:57.197505+0000","last_active":"2024-03-24T00:38:58.800129+0000","last_peered":"2024-03-24T00:38:58.800129+0000","last_clean":"2024-03-24T00:38:58.800129+0000","last_became_active":"2024-03-24T00:38:57.197104+0000","last_became_peered":"2024-03-24T00:38:57.197104+0000","last_unstale":"2024-03-24T00:38:58.800129+0000","last_undegraded":"2024-03-24T00:38:58.800129+0000","last_fullsized":"2024-03-24T00:38:58.800129+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:44:07.562150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676497+0000","last_change":"2024-03-24T00:38:57.197065+0000","last_active":"2024-03-24T00:38:59.676497+0000","last_peered":"2024-03-24T00:38:59.676497+0000","last_clean":"2024-03-24T00:38:59.676497+0000","last_became_active":"2024-03-24T00:38:57.196687+0000","last_became_peered":"2024-03-24T00:38:57.196687+0000","last_unstale":"2024-03-24T00:38:59.676497+0000","last_undegraded":"2024-03-24T00:38:59.676497+0000","last_fullsized":"2024-03-24T00:38:59.676497+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:53:16.442548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.974900+0000","last_change":"2024-03-24T00:38:52.790233+0000","last_active":"2024-03-24T00:38:58.974900+0000","last_peered":"2024-03-24T00:38:58.974900+0000","last_clean":"2024-03-24T00:38:58.974900+0000","last_became_active":"2024-03-24T00:38:52.789951+0000","last_became_peered":"2024-03-24T00:38:52.789951+0000","last_unstale":"2024-03-24T00:38:58.974900+0000","last_undegraded":"2024-03-24T00:38:58.974900+0000","last_fullsized":"2024-03-24T00:38:58.974900+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:42:38.046862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801116+0000","last_change":"2024-03-24T00:38:51.775723+0000","last_active":"2024-03-24T00:38:58.801116+0000","last_peered":"2024-03-24T00:38:58.801116+0000","last_clean":"2024-03-24T00:38:58.801116+0000","last_became_active":"2024-03-24T00:38:51.773333+0000","last_became_peered":"2024-03-24T00:38:51.773333+0000","last_unstale":"2024-03-24T00:38:58.801116+0000","last_undegraded":"2024-03-24T00:38:58.801116+0000","last_fullsized":"2024-03-24T00:38:58.801116+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:59:15.934422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676945+0000","last_change":"2024-03-24T00:38:56.819973+0000","last_active":"2024-03-24T00:38:59.676945+0000","last_peered":"2024-03-24T00:38:59.676945+0000","last_clean":"2024-03-24T00:38:59.676945+0000","last_became_active":"2024-03-24T00:38:56.819659+0000","last_became_peered":"2024-03-24T00:38:56.819659+0000","last_unstale":"2024-03-24T00:38:59.676945+0000","last_undegraded":"2024-03-24T00:38:59.676945+0000","last_fullsized":"2024-03-24T00:38:59.676945+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:57:34.164907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.799976+0000","last_change":"2024-03-24T00:38:52.785779+0000","last_active":"2024-03-24T00:38:58.799976+0000","last_peered":"2024-03-24T00:38:58.799976+0000","last_clean":"2024-03-24T00:38:58.799976+0000","last_became_active":"2024-03-24T00:38:52.785184+0000","last_became_peered":"2024-03-24T00:38:52.785184+0000","last_unstale":"2024-03-24T00:38:58.799976+0000","last_undegraded":"2024-03-24T00:38:58.799976+0000","last_fullsized":"2024-03-24T00:38:58.799976+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:39:26.969331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800170+0000","last_change":"2024-03-24T00:38:51.784065+0000","last_active":"2024-03-24T00:38:58.800170+0000","last_peered":"2024-03-24T00:38:58.800170+0000","last_clean":"2024-03-24T00:38:58.800170+0000","last_became_active":"2024-03-24T00:38:51.783789+0000","last_became_peered":"2024-03-24T00:38:51.783789+0000","last_unstale":"2024-03-24T00:38:58.800170+0000","last_undegraded":"2024-03-24T00:38:58.800170+0000","last_fullsized":"2024-03-24T00:38:58.800170+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:23:25.006444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676753+0000","last_change":"2024-03-24T00:38:51.777458+0000","last_active":"2024-03-24T00:38:59.676753+0000","last_peered":"2024-03-24T00:38:59.676753+0000","last_clean":"2024-03-24T00:38:59.676753+0000","last_became_active":"2024-03-24T00:38:51.777189+0000","last_became_peered":"2024-03-24T00:38:51.777189+0000","last_unstale":"2024-03-24T00:38:59.676753+0000","last_undegraded":"2024-03-24T00:38:59.676753+0000","last_fullsized":"2024-03-24T00:38:59.676753+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:05.055760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676860+0000","last_change":"2024-03-24T00:38:52.786644+0000","last_active":"2024-03-24T00:38:59.676860+0000","last_peered":"2024-03-24T00:38:59.676860+0000","last_clean":"2024-03-24T00:38:59.676860+0000","last_became_active":"2024-03-24T00:38:52.786348+0000","last_became_peered":"2024-03-24T00:38:52.786348+0000","last_unstale":"2024-03-24T00:38:59.676860+0000","last_undegraded":"2024-03-24T00:38:59.676860+0000","last_fullsized":"2024-03-24T00:38:59.676860+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:36:54.790308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676080+0000","last_change":"2024-03-24T00:38:56.816034+0000","last_active":"2024-03-24T00:38:59.676080+0000","last_peered":"2024-03-24T00:38:59.676080+0000","last_clean":"2024-03-24T00:38:59.676080+0000","last_became_active":"2024-03-24T00:38:56.815608+0000","last_became_peered":"2024-03-24T00:38:56.815608+0000","last_unstale":"2024-03-24T00:38:59.676080+0000","last_undegraded":"2024-03-24T00:38:59.676080+0000","last_fullsized":"2024-03-24T00:38:59.676080+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:27:27.336047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800713+0000","last_change":"2024-03-24T00:38:51.782948+0000","last_active":"2024-03-24T00:38:58.800713+0000","last_peered":"2024-03-24T00:38:58.800713+0000","last_clean":"2024-03-24T00:38:58.800713+0000","last_became_active":"2024-03-24T00:38:51.780063+0000","last_became_peered":"2024-03-24T00:38:51.780063+0000","last_unstale":"2024-03-24T00:38:58.800713+0000","last_undegraded":"2024-03-24T00:38:58.800713+0000","last_fullsized":"2024-03-24T00:38:58.800713+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:05.055760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801390+0000","last_change":"2024-03-24T00:38:52.778373+0000","last_active":"2024-03-24T00:38:58.801390+0000","last_peered":"2024-03-24T00:38:58.801390+0000","last_clean":"2024-03-24T00:38:58.801390+0000","last_became_active":"2024-03-24T00:38:52.778019+0000","last_became_peered":"2024-03-24T00:38:52.778019+0000","last_unstale":"2024-03-24T00:38:58.801390+0000","last_undegraded":"2024-03-24T00:38:58.801390+0000","last_fullsized":"2024-03-24T00:38:58.801390+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:09:31.127048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800819+0000","last_change":"2024-03-24T00:38:56.818404+0000","last_active":"2024-03-24T00:38:58.800819+0000","last_peered":"2024-03-24T00:38:58.800819+0000","last_clean":"2024-03-24T00:38:58.800819+0000","last_became_active":"2024-03-24T00:38:56.817801+0000","last_became_peered":"2024-03-24T00:38:56.817801+0000","last_unstale":"2024-03-24T00:38:58.800819+0000","last_undegraded":"2024-03-24T00:38:58.800819+0000","last_fullsized":"2024-03-24T00:38:58.800819+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:15:28.193092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676081+0000","last_change":"2024-03-24T00:38:52.780120+0000","last_active":"2024-03-24T00:38:59.676081+0000","last_peered":"2024-03-24T00:38:59.676081+0000","last_clean":"2024-03-24T00:38:59.676081+0000","last_became_active":"2024-03-24T00:38:52.779624+0000","last_became_peered":"2024-03-24T00:38:52.779624+0000","last_unstale":"2024-03-24T00:38:59.676081+0000","last_undegraded":"2024-03-24T00:38:59.676081+0000","last_fullsized":"2024-03-24T00:38:59.676081+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:54:01.232233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676019+0000","last_change":"2024-03-24T00:38:52.783615+0000","last_active":"2024-03-24T00:38:59.676019+0000","last_peered":"2024-03-24T00:38:59.676019+0000","last_clean":"2024-03-24T00:38:59.676019+0000","last_became_active":"2024-03-24T00:38:52.783339+0000","last_became_peered":"2024-03-24T00:38:52.783339+0000","last_unstale":"2024-03-24T00:38:59.676019+0000","last_undegraded":"2024-03-24T00:38:59.676019+0000","last_fullsized":"2024-03-24T00:38:59.676019+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:51:22.179128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802656+0000","last_change":"2024-03-24T00:38:52.793626+0000","last_active":"2024-03-24T00:38:58.802656+0000","last_peered":"2024-03-24T00:38:58.802656+0000","last_clean":"2024-03-24T00:38:58.802656+0000","last_became_active":"2024-03-24T00:38:52.793322+0000","last_became_peered":"2024-03-24T00:38:52.793322+0000","last_unstale":"2024-03-24T00:38:58.802656+0000","last_undegraded":"2024-03-24T00:38:58.802656+0000","last_fullsized":"2024-03-24T00:38:58.802656+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:57:30.131729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676831+0000","last_change":"2024-03-24T00:38:52.791203+0000","last_active":"2024-03-24T00:38:59.676831+0000","last_peered":"2024-03-24T00:38:59.676831+0000","last_clean":"2024-03-24T00:38:59.676831+0000","last_became_active":"2024-03-24T00:38:52.790914+0000","last_became_peered":"2024-03-24T00:38:52.790914+0000","last_unstale":"2024-03-24T00:38:59.676831+0000","last_undegraded":"2024-03-24T00:38:59.676831+0000","last_fullsized":"2024-03-24T00:38:59.676831+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:21:12.524662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800715+0000","last_change":"2024-03-24T00:38:52.788614+0000","last_active":"2024-03-24T00:38:58.800715+0000","last_peered":"2024-03-24T00:38:58.800715+0000","last_clean":"2024-03-24T00:38:58.800715+0000","last_became_active":"2024-03-24T00:38:52.788155+0000","last_became_peered":"2024-03-24T00:38:52.788155+0000","last_unstale":"2024-03-24T00:38:58.800715+0000","last_undegraded":"2024-03-24T00:38:58.800715+0000","last_fullsized":"2024-03-24T00:38:58.800715+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:54:01.232233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975795+0000","last_change":"2024-03-24T00:38:52.790018+0000","last_active":"2024-03-24T00:38:58.975795+0000","last_peered":"2024-03-24T00:38:58.975795+0000","last_clean":"2024-03-24T00:38:58.975795+0000","last_became_active":"2024-03-24T00:38:52.789744+0000","last_became_peered":"2024-03-24T00:38:52.789744+0000","last_unstale":"2024-03-24T00:38:58.975795+0000","last_undegraded":"2024-03-24T00:38:58.975795+0000","last_fullsized":"2024-03-24T00:38:58.975795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:46:05.527509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676904+0000","last_change":"2024-03-24T00:38:52.791896+0000","last_active":"2024-03-24T00:38:59.676904+0000","last_peered":"2024-03-24T00:38:59.676904+0000","last_clean":"2024-03-24T00:38:59.676904+0000","last_became_active":"2024-03-24T00:38:52.789711+0000","last_became_peered":"2024-03-24T00:38:52.789711+0000","last_unstale":"2024-03-24T00:38:59.676904+0000","last_undegraded":"2024-03-24T00:38:59.676904+0000","last_fullsized":"2024-03-24T00:38:59.676904+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:21:12.524662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800598+0000","last_change":"2024-03-24T00:38:52.780901+0000","last_active":"2024-03-24T00:38:58.800598+0000","last_peered":"2024-03-24T00:38:58.800598+0000","last_clean":"2024-03-24T00:38:58.800598+0000","last_became_active":"2024-03-24T00:38:52.780625+0000","last_became_peered":"2024-03-24T00:38:52.780625+0000","last_unstale":"2024-03-24T00:38:58.800598+0000","last_undegraded":"2024-03-24T00:38:58.800598+0000","last_fullsized":"2024-03-24T00:38:58.800598+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:42:38.046862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801027+0000","last_change":"2024-03-24T00:38:52.784339+0000","last_active":"2024-03-24T00:38:58.801027+0000","last_peered":"2024-03-24T00:38:58.801027+0000","last_clean":"2024-03-24T00:38:58.801027+0000","last_became_active":"2024-03-24T00:38:52.783135+0000","last_became_peered":"2024-03-24T00:38:52.783135+0000","last_unstale":"2024-03-24T00:38:58.801027+0000","last_undegraded":"2024-03-24T00:38:58.801027+0000","last_fullsized":"2024-03-24T00:38:58.801027+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:21:12.524662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800355+0000","last_change":"2024-03-24T00:38:52.788394+0000","last_active":"2024-03-24T00:38:58.800355+0000","last_peered":"2024-03-24T00:38:58.800355+0000","last_clean":"2024-03-24T00:38:58.800355+0000","last_became_active":"2024-03-24T00:38:52.788105+0000","last_became_peered":"2024-03-24T00:38:52.788105+0000","last_unstale":"2024-03-24T00:38:58.800355+0000","last_undegraded":"2024-03-24T00:38:58.800355+0000","last_fullsized":"2024-03-24T00:38:58.800355+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:04:24.260803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802590+0000","last_change":"2024-03-24T00:38:52.787942+0000","last_active":"2024-03-24T00:38:58.802590+0000","last_peered":"2024-03-24T00:38:58.802590+0000","last_clean":"2024-03-24T00:38:58.802590+0000","last_became_active":"2024-03-24T00:38:52.787691+0000","last_became_peered":"2024-03-24T00:38:52.787691+0000","last_unstale":"2024-03-24T00:38:58.802590+0000","last_undegraded":"2024-03-24T00:38:58.802590+0000","last_fullsized":"2024-03-24T00:38:58.802590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:17:42.703499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676562+0000","last_change":"2024-03-24T00:38:52.780488+0000","last_active":"2024-03-24T00:38:59.676562+0000","last_peered":"2024-03-24T00:38:59.676562+0000","last_clean":"2024-03-24T00:38:59.676562+0000","last_became_active":"2024-03-24T00:38:52.780227+0000","last_became_peered":"2024-03-24T00:38:52.780227+0000","last_unstale":"2024-03-24T00:38:59.676562+0000","last_undegraded":"2024-03-24T00:38:59.676562+0000","last_fullsized":"2024-03-24T00:38:59.676562+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:39:26.969331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676482+0000","last_change":"2024-03-24T00:38:52.785757+0000","last_active":"2024-03-24T00:38:59.676482+0000","last_peered":"2024-03-24T00:38:59.676482+0000","last_clean":"2024-03-24T00:38:59.676482+0000","last_became_active":"2024-03-24T00:38:52.785476+0000","last_became_peered":"2024-03-24T00:38:52.785476+0000","last_unstale":"2024-03-24T00:38:59.676482+0000","last_undegraded":"2024-03-24T00:38:59.676482+0000","last_fullsized":"2024-03-24T00:38:59.676482+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:39:26.969331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.677056+0000","last_change":"2024-03-24T00:38:52.786263+0000","last_active":"2024-03-24T00:38:59.677056+0000","last_peered":"2024-03-24T00:38:59.677056+0000","last_clean":"2024-03-24T00:38:59.677056+0000","last_became_active":"2024-03-24T00:38:52.785957+0000","last_became_peered":"2024-03-24T00:38:52.785957+0000","last_unstale":"2024-03-24T00:38:59.677056+0000","last_undegraded":"2024-03-24T00:38:59.677056+0000","last_fullsized":"2024-03-24T00:38:59.677056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:02:43.127453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676749+0000","last_change":"2024-03-24T00:38:52.786178+0000","last_active":"2024-03-24T00:38:59.676749+0000","last_peered":"2024-03-24T00:38:59.676749+0000","last_clean":"2024-03-24T00:38:59.676749+0000","last_became_active":"2024-03-24T00:38:52.785894+0000","last_became_peered":"2024-03-24T00:38:52.785894+0000","last_unstale":"2024-03-24T00:38:59.676749+0000","last_undegraded":"2024-03-24T00:38:59.676749+0000","last_fullsized":"2024-03-24T00:38:59.676749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:36:19.927252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801779+0000","last_change":"2024-03-24T00:38:52.787876+0000","last_active":"2024-03-24T00:38:58.801779+0000","last_peered":"2024-03-24T00:38:58.801779+0000","last_clean":"2024-03-24T00:38:58.801779+0000","last_became_active":"2024-03-24T00:38:52.787243+0000","last_became_peered":"2024-03-24T00:38:52.787243+0000","last_unstale":"2024-03-24T00:38:58.801779+0000","last_undegraded":"2024-03-24T00:38:58.801779+0000","last_fullsized":"2024-03-24T00:38:58.801779+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:36:54.790308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800934+0000","last_change":"2024-03-24T00:38:52.776125+0000","last_active":"2024-03-24T00:38:58.800934+0000","last_peered":"2024-03-24T00:38:58.800934+0000","last_clean":"2024-03-24T00:38:58.800934+0000","last_became_active":"2024-03-24T00:38:52.775255+0000","last_became_peered":"2024-03-24T00:38:52.775255+0000","last_unstale":"2024-03-24T00:38:58.800934+0000","last_undegraded":"2024-03-24T00:38:58.800934+0000","last_fullsized":"2024-03-24T00:38:58.800934+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:46:05.527509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.974980+0000","last_change":"2024-03-24T00:38:52.781546+0000","last_active":"2024-03-24T00:38:58.974980+0000","last_peered":"2024-03-24T00:38:58.974980+0000","last_clean":"2024-03-24T00:38:58.974980+0000","last_became_active":"2024-03-24T00:38:52.781220+0000","last_became_peered":"2024-03-24T00:38:52.781220+0000","last_unstale":"2024-03-24T00:38:58.974980+0000","last_undegraded":"2024-03-24T00:38:58.974980+0000","last_fullsized":"2024-03-24T00:38:58.974980+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:41:59.361242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676394+0000","last_change":"2024-03-24T00:38:52.773210+0000","last_active":"2024-03-24T00:38:59.676394+0000","last_peered":"2024-03-24T00:38:59.676394+0000","last_clean":"2024-03-24T00:38:59.676394+0000","last_became_active":"2024-03-24T00:38:52.772887+0000","last_became_peered":"2024-03-24T00:38:52.772887+0000","last_unstale":"2024-03-24T00:38:59.676394+0000","last_undegraded":"2024-03-24T00:38:59.676394+0000","last_fullsized":"2024-03-24T00:38:59.676394+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:27:23.302869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800436+0000","last_change":"2024-03-24T00:38:52.784496+0000","last_active":"2024-03-24T00:38:58.800436+0000","last_peered":"2024-03-24T00:38:58.800436+0000","last_clean":"2024-03-24T00:38:58.800436+0000","last_became_active":"2024-03-24T00:38:52.784234+0000","last_became_peered":"2024-03-24T00:38:52.784234+0000","last_unstale":"2024-03-24T00:38:58.800436+0000","last_undegraded":"2024-03-24T00:38:58.800436+0000","last_fullsized":"2024-03-24T00:38:58.800436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:36:54.790308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802210+0000","last_change":"2024-03-24T00:38:52.786080+0000","last_active":"2024-03-24T00:38:58.802210+0000","last_peered":"2024-03-24T00:38:58.802210+0000","last_clean":"2024-03-24T00:38:58.802210+0000","last_became_active":"2024-03-24T00:38:52.785803+0000","last_became_peered":"2024-03-24T00:38:52.785803+0000","last_unstale":"2024-03-24T00:38:58.802210+0000","last_undegraded":"2024-03-24T00:38:58.802210+0000","last_fullsized":"2024-03-24T00:38:58.802210+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:51:22.179128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676587+0000","last_change":"2024-03-24T00:38:52.788391+0000","last_active":"2024-03-24T00:38:59.676587+0000","last_peered":"2024-03-24T00:38:59.676587+0000","last_clean":"2024-03-24T00:38:59.676587+0000","last_became_active":"2024-03-24T00:38:52.788122+0000","last_became_peered":"2024-03-24T00:38:52.788122+0000","last_unstale":"2024-03-24T00:38:59.676587+0000","last_undegraded":"2024-03-24T00:38:59.676587+0000","last_fullsized":"2024-03-24T00:38:59.676587+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:41:59.361242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676867+0000","last_change":"2024-03-24T00:38:52.787696+0000","last_active":"2024-03-24T00:38:59.676867+0000","last_peered":"2024-03-24T00:38:59.676867+0000","last_clean":"2024-03-24T00:38:59.676867+0000","last_became_active":"2024-03-24T00:38:52.787431+0000","last_became_peered":"2024-03-24T00:38:52.787431+0000","last_unstale":"2024-03-24T00:38:59.676867+0000","last_undegraded":"2024-03-24T00:38:59.676867+0000","last_fullsized":"2024-03-24T00:38:59.676867+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:46:05.527509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.677244+0000","last_change":"2024-03-24T00:38:52.791775+0000","last_active":"2024-03-24T00:38:59.677244+0000","last_peered":"2024-03-24T00:38:59.677244+0000","last_clean":"2024-03-24T00:38:59.677244+0000","last_became_active":"2024-03-24T00:38:52.788742+0000","last_became_peered":"2024-03-24T00:38:52.788742+0000","last_unstale":"2024-03-24T00:38:59.677244+0000","last_undegraded":"2024-03-24T00:38:59.677244+0000","last_fullsized":"2024-03-24T00:38:59.677244+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:08:00.887641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.799866+0000","last_change":"2024-03-24T00:38:52.790297+0000","last_active":"2024-03-24T00:38:58.799866+0000","last_peered":"2024-03-24T00:38:58.799866+0000","last_clean":"2024-03-24T00:38:58.799866+0000","last_became_active":"2024-03-24T00:38:52.789879+0000","last_became_peered":"2024-03-24T00:38:52.789879+0000","last_unstale":"2024-03-24T00:38:58.799866+0000","last_undegraded":"2024-03-24T00:38:58.799866+0000","last_fullsized":"2024-03-24T00:38:58.799866+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:41:59.361242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676732+0000","last_change":"2024-03-24T00:38:52.780267+0000","last_active":"2024-03-24T00:38:59.676732+0000","last_peered":"2024-03-24T00:38:59.676732+0000","last_clean":"2024-03-24T00:38:59.676732+0000","last_became_active":"2024-03-24T00:38:52.779973+0000","last_became_peered":"2024-03-24T00:38:52.779973+0000","last_unstale":"2024-03-24T00:38:59.676732+0000","last_undegraded":"2024-03-24T00:38:59.676732+0000","last_fullsized":"2024-03-24T00:38:59.676732+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:36:54.790308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802578+0000","last_change":"2024-03-24T00:38:52.776890+0000","last_active":"2024-03-24T00:38:58.802578+0000","last_peered":"2024-03-24T00:38:58.802578+0000","last_clean":"2024-03-24T00:38:58.802578+0000","last_became_active":"2024-03-24T00:38:52.776573+0000","last_became_peered":"2024-03-24T00:38:52.776573+0000","last_unstale":"2024-03-24T00:38:58.802578+0000","last_undegraded":"2024-03-24T00:38:58.802578+0000","last_fullsized":"2024-03-24T00:38:58.802578+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:02:43.127453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676561+0000","last_change":"2024-03-24T00:38:52.783960+0000","last_active":"2024-03-24T00:38:59.676561+0000","last_peered":"2024-03-24T00:38:59.676561+0000","last_clean":"2024-03-24T00:38:59.676561+0000","last_became_active":"2024-03-24T00:38:52.783606+0000","last_became_peered":"2024-03-24T00:38:52.783606+0000","last_unstale":"2024-03-24T00:38:59.676561+0000","last_undegraded":"2024-03-24T00:38:59.676561+0000","last_fullsized":"2024-03-24T00:38:59.676561+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:21:00.861116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802672+0000","last_change":"2024-03-24T00:38:52.784222+0000","last_active":"2024-03-24T00:38:58.802672+0000","last_peered":"2024-03-24T00:38:58.802672+0000","last_clean":"2024-03-24T00:38:58.802672+0000","last_became_active":"2024-03-24T00:38:52.783951+0000","last_became_peered":"2024-03-24T00:38:52.783951+0000","last_unstale":"2024-03-24T00:38:58.802672+0000","last_undegraded":"2024-03-24T00:38:58.802672+0000","last_fullsized":"2024-03-24T00:38:58.802672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:04:24.260803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676177+0000","last_change":"2024-03-24T00:38:52.774199+0000","last_active":"2024-03-24T00:38:59.676177+0000","last_peered":"2024-03-24T00:38:59.676177+0000","last_clean":"2024-03-24T00:38:59.676177+0000","last_became_active":"2024-03-24T00:38:52.773914+0000","last_became_peered":"2024-03-24T00:38:52.773914+0000","last_unstale":"2024-03-24T00:38:59.676177+0000","last_undegraded":"2024-03-24T00:38:59.676177+0000","last_fullsized":"2024-03-24T00:38:59.676177+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:54:01.232233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975533+0000","last_change":"2024-03-24T00:38:52.778927+0000","last_active":"2024-03-24T00:38:58.975533+0000","last_peered":"2024-03-24T00:38:58.975533+0000","last_clean":"2024-03-24T00:38:58.975533+0000","last_became_active":"2024-03-24T00:38:52.778655+0000","last_became_peered":"2024-03-24T00:38:52.778655+0000","last_unstale":"2024-03-24T00:38:58.975533+0000","last_undegraded":"2024-03-24T00:38:58.975533+0000","last_fullsized":"2024-03-24T00:38:58.975533+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:21:12.524662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.677156+0000","last_change":"2024-03-24T00:38:52.785456+0000","last_active":"2024-03-24T00:38:59.677156+0000","last_peered":"2024-03-24T00:38:59.677156+0000","last_clean":"2024-03-24T00:38:59.677156+0000","last_became_active":"2024-03-24T00:38:52.785150+0000","last_became_peered":"2024-03-24T00:38:52.785150+0000","last_unstale":"2024-03-24T00:38:59.677156+0000","last_undegraded":"2024-03-24T00:38:59.677156+0000","last_fullsized":"2024-03-24T00:38:59.677156+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:17:42.703499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4499,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":8998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27868,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343972,"statfs":{"total":96636764160,"available":96608227328,"internally_reserved":0,"allocated":864256,"data_stored":698343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":111094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":107483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":278528,"data_stored":107519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":111094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":107393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":274432,"data_stored":107553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27872,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343968,"statfs":{"total":96636764160,"available":96608223232,"internally_reserved":0,"allocated":872448,"data_stored":698399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:39:10.182 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-24T00:39:10.182 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-24T00:39:10.182 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-24T00:39:10.182 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-24T00:39:10.183 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-24T00:39:10.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.318+0000 7fdcafa0b640 1 Processor -- start 2024-03-24T00:39:10.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.318+0000 7fdcafa0b640 1 -- start start 2024-03-24T00:39:10.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.319+0000 7fdcafa0b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdca815e2a0 0x7fdca80ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:10.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.319+0000 7fdcafa0b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fdca80d0ef0 0x7fdca8162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:10.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.319+0000 7fdcafa0b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdca80cf640 con 0x7fdca815dec0 2024-03-24T00:39:10.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.319+0000 7fdcafa0b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdca80cee30 con 0x7fdca80d0ef0 2024-03-24T00:39:10.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.319+0000 7fdcafa0b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdca80d2120 con 0x7fdca815e2a0 2024-03-24T00:39:10.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.319+0000 7fdcad7a7640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fdca815dec0 0x7fdca80d07e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45400/0 (socket says 172.21.15.67:45400) 2024-03-24T00:39:10.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.319+0000 7fdcad7a7640 1 -- 172.21.15.67:0/1463847048 learned_addr learned my addr 172.21.15.67:0/1463847048 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:10.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.319+0000 7fdcacfa6640 1 --2- 172.21.15.67:0/1463847048 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdca815e2a0 0x7fdca80ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:10.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.319+0000 7fdcadfa8640 1 --2- 172.21.15.67:0/1463847048 >> v2:172.21.15.72:3300/0 conn(0x7fdca80d0ef0 0x7fdca8162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:10.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.320+0000 7fdc9ffff640 1 -- 172.21.15.67:0/1463847048 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1369538697 0 0) 0x7fdca4001110 con 0x7fdca815dec0 2024-03-24T00:39:10.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.320+0000 7fdc9ffff640 1 -- 172.21.15.67:0/1463847048 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc94003520 con 0x7fdca815dec0 2024-03-24T00:39:10.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.320+0000 7fdcadfa8640 1 -- 172.21.15.67:0/1463847048 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdca815e2a0 msgr2=0x7fdca80ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:10.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.320+0000 7fdcadfa8640 1 --2- 172.21.15.67:0/1463847048 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdca815e2a0 0x7fdca80ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:10.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.320+0000 7fdcadfa8640 1 -- 172.21.15.67:0/1463847048 >> v1:172.21.15.67:6789/0 conn(0x7fdca815dec0 legacy=0x7fdca80d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:10.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.320+0000 7fdcadfa8640 1 -- 172.21.15.67:0/1463847048 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca80cf080 con 0x7fdca80d0ef0 2024-03-24T00:39:10.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.320+0000 7fdcacfa6640 1 --2- 172.21.15.67:0/1463847048 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdca815e2a0 0x7fdca80ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:10.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.321+0000 7fdcadfa8640 1 --2- 172.21.15.67:0/1463847048 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdca80d0ef0 0x7fdca8162630 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fdca0009d90 tx=0x7fdca002be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=963b4e35462c619c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:10.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.321+0000 7fdc9ffff640 1 -- 172.21.15.67:0/1463847048 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca0039070 con 0x7fdca80d0ef0 2024-03-24T00:39:10.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.321+0000 7fdc9ffff640 1 -- 172.21.15.67:0/1463847048 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdca002cbc0 con 0x7fdca80d0ef0 2024-03-24T00:39:10.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.321+0000 7fdc9ffff640 1 -- 172.21.15.67:0/1463847048 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca0034440 con 0x7fdca80d0ef0 2024-03-24T00:39:10.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.321+0000 7fdcafa0b640 1 -- 172.21.15.67:0/1463847048 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdca80d0ef0 msgr2=0x7fdca8162630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:10.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.321+0000 7fdcafa0b640 1 --2- 172.21.15.67:0/1463847048 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdca80d0ef0 0x7fdca8162630 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fdca0009d90 tx=0x7fdca002be10 comp rx=0 tx=0).stop 2024-03-24T00:39:10.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.321+0000 7fdcafa0b640 1 -- 172.21.15.67:0/1463847048 shutdown_connections 2024-03-24T00:39:10.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.321+0000 7fdcafa0b640 1 --2- 172.21.15.67:0/1463847048 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdca80d0ef0 0x7fdca8162630 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:10.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.321+0000 7fdcafa0b640 1 --2- 172.21.15.67:0/1463847048 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdca815e2a0 0x7fdca80ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:10.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.321+0000 7fdcafa0b640 1 -- 172.21.15.67:0/1463847048 >> 172.21.15.67:0/1463847048 conn(0x7fdca806c810 msgr2=0x7fdca806cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:10.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.321+0000 7fdcafa0b640 1 -- 172.21.15.67:0/1463847048 shutdown_connections 2024-03-24T00:39:10.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.321+0000 7fdcafa0b640 1 -- 172.21.15.67:0/1463847048 wait complete. 2024-03-24T00:39:10.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.322+0000 7fdcafa0b640 1 Processor -- start 2024-03-24T00:39:10.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.322+0000 7fdcafa0b640 1 -- start start 2024-03-24T00:39:10.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.323+0000 7fdcafa0b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdca80d0ef0 0x7fdca815ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:10.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.323+0000 7fdcafa0b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdca815dec0 0x7fdca80b42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:10.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.323+0000 7fdcafa0b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdca815e2a0 0x7fdca80b47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:10.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.323+0000 7fdcafa0b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdca80d2730 con 0x7fdca815e2a0 2024-03-24T00:39:10.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.323+0000 7fdcafa0b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdca80d25b0 con 0x7fdca815dec0 2024-03-24T00:39:10.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.323+0000 7fdcafa0b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdca80d28b0 con 0x7fdca80d0ef0 2024-03-24T00:39:10.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.323+0000 7fdcadfa8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdca815e2a0 0x7fdca80b47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:10.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.323+0000 7fdcadfa8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdca815e2a0 0x7fdca80b47f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60758/0 (socket says 172.21.15.67:60758) 2024-03-24T00:39:10.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.323+0000 7fdcadfa8640 1 -- 172.21.15.67:0/3945094166 learned_addr learned my addr 172.21.15.67:0/3945094166 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:10.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.323+0000 7fdcacfa6640 1 --2- 172.21.15.67:0/3945094166 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdca815dec0 0x7fdca80b42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:10.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.323+0000 7fdcad7a7640 1 --2- 172.21.15.67:0/3945094166 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdca80d0ef0 0x7fdca815ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:10.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.324+0000 7fdcadfa8640 1 -- 172.21.15.67:0/3945094166 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdca80d0ef0 msgr2=0x7fdca815ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:10.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.324+0000 7fdcadfa8640 1 --2- 172.21.15.67:0/3945094166 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdca80d0ef0 0x7fdca815ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:10.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.324+0000 7fdcadfa8640 1 -- 172.21.15.67:0/3945094166 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdca815dec0 msgr2=0x7fdca80b42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:10.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.324+0000 7fdcadfa8640 1 --2- 172.21.15.67:0/3945094166 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdca815dec0 0x7fdca80b42b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:10.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.324+0000 7fdcadfa8640 1 -- 172.21.15.67:0/3945094166 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca80b5260 con 0x7fdca815e2a0 2024-03-24T00:39:10.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.324+0000 7fdcadfa8640 1 --2- 172.21.15.67:0/3945094166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdca815e2a0 0x7fdca80b47f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fdca0009d90 tx=0x7fdca0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:10.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.324+0000 7fdc9dffb640 1 -- 172.21.15.67:0/3945094166 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca0039050 con 0x7fdca815e2a0 2024-03-24T00:39:10.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.324+0000 7fdc9dffb640 1 -- 172.21.15.67:0/3945094166 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdca0045070 con 0x7fdca815e2a0 2024-03-24T00:39:10.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.324+0000 7fdcafa0b640 1 -- 172.21.15.67:0/3945094166 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdca80b5500 con 0x7fdca815e2a0 2024-03-24T00:39:10.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.324+0000 7fdc9dffb640 1 -- 172.21.15.67:0/3945094166 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca0009450 con 0x7fdca815e2a0 2024-03-24T00:39:10.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.324+0000 7fdcafa0b640 1 -- 172.21.15.67:0/3945094166 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdca80b5ac0 con 0x7fdca815e2a0 2024-03-24T00:39:10.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.325+0000 7fdcafa0b640 1 -- 172.21.15.67:0/3945094166 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdca804f400 con 0x7fdca815e2a0 2024-03-24T00:39:10.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.326+0000 7fdc9dffb640 1 -- 172.21.15.67:0/3945094166 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fdca003e830 con 0x7fdca815e2a0 2024-03-24T00:39:10.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.326+0000 7fdc9dffb640 1 --2- 172.21.15.67:0/3945094166 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdc8c06bbd0 0x7fdc8c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:10.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.326+0000 7fdc9dffb640 1 -- 172.21.15.67:0/3945094166 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fdca00ada30 con 0x7fdca815e2a0 2024-03-24T00:39:10.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.326+0000 7fdcad7a7640 1 --2- 172.21.15.67:0/3945094166 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdc8c06bbd0 0x7fdc8c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:10.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.327+0000 7fdcad7a7640 1 --2- 172.21.15.67:0/3945094166 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdc8c06bbd0 0x7fdc8c06e090 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdca4005880 tx=0x7fdca4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:10.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.330+0000 7fdc9dffb640 1 -- 172.21.15.67:0/3945094166 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fdca007bcf0 con 0x7fdca815e2a0 2024-03-24T00:39:10.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.565+0000 7fdcafa0b640 1 -- 172.21.15.67:0/3945094166 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fdca80ce5c0 con 0x7fdca815e2a0 2024-03-24T00:39:10.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.568+0000 7fdc9dffb640 1 -- 172.21.15.67:0/3945094166 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+172577 (secure 0 0 0) 0x7fdca007fd90 con 0x7fdca815e2a0 2024-03-24T00:39:10.570 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:39:10.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.571+0000 7fdcafa0b640 1 -- 172.21.15.67:0/3945094166 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdc8c06bbd0 msgr2=0x7fdc8c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:10.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.571+0000 7fdcafa0b640 1 --2- 172.21.15.67:0/3945094166 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdc8c06bbd0 0x7fdc8c06e090 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdca4005880 tx=0x7fdca4009040 comp rx=0 tx=0).stop 2024-03-24T00:39:10.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.571+0000 7fdcafa0b640 1 -- 172.21.15.67:0/3945094166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdca815e2a0 msgr2=0x7fdca80b47f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:10.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.571+0000 7fdcafa0b640 1 --2- 172.21.15.67:0/3945094166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdca815e2a0 0x7fdca80b47f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fdca0009d90 tx=0x7fdca0004a40 comp rx=0 tx=0).stop 2024-03-24T00:39:10.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.571+0000 7fdcafa0b640 1 -- 172.21.15.67:0/3945094166 shutdown_connections 2024-03-24T00:39:10.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.571+0000 7fdcafa0b640 1 --2- 172.21.15.67:0/3945094166 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdc8c06bbd0 0x7fdc8c06e090 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:10.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.571+0000 7fdcafa0b640 1 --2- 172.21.15.67:0/3945094166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdca815e2a0 0x7fdca80b47f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:10.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.571+0000 7fdcafa0b640 1 --2- 172.21.15.67:0/3945094166 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdca815dec0 0x7fdca80b42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:10.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.571+0000 7fdcafa0b640 1 --2- 172.21.15.67:0/3945094166 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdca80d0ef0 0x7fdca815ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:10.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.571+0000 7fdcafa0b640 1 -- 172.21.15.67:0/3945094166 >> 172.21.15.67:0/3945094166 conn(0x7fdca806c810 msgr2=0x7fdca815f990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:10.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.571+0000 7fdcafa0b640 1 -- 172.21.15.67:0/3945094166 shutdown_connections 2024-03-24T00:39:10.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.571+0000 7fdcafa0b640 1 -- 172.21.15.67:0/3945094166 wait complete. 2024-03-24T00:39:10.582 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":5,"flags":0,"active_gid":4105,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6832","nonce":2768292274},{"type":"v1","addr":"172.21.15.72:6833","nonce":2768292274}]},"active_addr":"172.21.15.72:6833/2768292274","active_change":"2024-03-24T00:38:31.313546+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4098,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.72:0","nonce":558724348}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.72:0","nonce":1178949522}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.72:0","nonce":4281185702}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.72:0","nonce":637072451}]}]} 2024-03-24T00:39:10.585 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-24T00:39:10.586 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-24T00:39:10.586 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:39:10.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.687+0000 7f6c2ef97640 1 Processor -- start 2024-03-24T00:39:10.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.687+0000 7f6c2ef97640 1 -- start start 2024-03-24T00:39:10.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.688+0000 7f6c2ef97640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6c2815dfe0 0x7f6c280ceb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:10.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.688+0000 7f6c2ef97640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6c28161bb0 0x7f6c280cf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:10.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.688+0000 7f6c2ef97640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c28162120 con 0x7f6c280b96d0 2024-03-24T00:39:10.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.688+0000 7f6c2ef97640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c280d21d0 con 0x7f6c2815dfe0 2024-03-24T00:39:10.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.688+0000 7f6c2ef97640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c28162400 con 0x7f6c28161bb0 2024-03-24T00:39:10.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.688+0000 7f6c2cd33640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6c2815dfe0 0x7f6c280ceb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:10.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.688+0000 7f6c2d534640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6c280b96d0 0x7f6c280b57c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45408/0 (socket says 172.21.15.67:45408) 2024-03-24T00:39:10.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.688+0000 7f6c2d534640 1 -- 172.21.15.67:0/3042415594 learned_addr learned my addr 172.21.15.67:0/3042415594 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:10.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.688+0000 7f6c27fff640 1 --2- 172.21.15.67:0/3042415594 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6c28161bb0 0x7f6c280cf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:10.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.688+0000 7f6c2cd33640 1 -- 172.21.15.67:0/3042415594 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6c28161bb0 msgr2=0x7f6c280cf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:10.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.688+0000 7f6c2cd33640 1 --2- 172.21.15.67:0/3042415594 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6c28161bb0 0x7f6c280cf050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:10.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.688+0000 7f6c2cd33640 1 -- 172.21.15.67:0/3042415594 >> v1:172.21.15.67:6789/0 conn(0x7f6c280b96d0 legacy=0x7f6c280b57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:10.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.688+0000 7f6c2cd33640 1 -- 172.21.15.67:0/3042415594 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c281626e0 con 0x7f6c2815dfe0 2024-03-24T00:39:10.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.688+0000 7f6c277fe640 1 -- 172.21.15.67:0/3042415594 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1977064732 0 0) 0x7f6c28162120 con 0x7f6c280b96d0 2024-03-24T00:39:10.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.689+0000 7f6c27fff640 1 --2- 172.21.15.67:0/3042415594 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6c28161bb0 0x7f6c280cf050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:10.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.689+0000 7f6c2cd33640 1 --2- 172.21.15.67:0/3042415594 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c2815dfe0 0x7f6c280ceb10 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6c20009d90 tx=0x7f6c2002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=692c1746f6a9c3a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:10.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.689+0000 7f6c277fe640 1 -- 172.21.15.67:0/3042415594 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c20039070 con 0x7f6c2815dfe0 2024-03-24T00:39:10.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.689+0000 7f6c277fe640 1 -- 172.21.15.67:0/3042415594 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c2002cc50 con 0x7f6c2815dfe0 2024-03-24T00:39:10.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.689+0000 7f6c277fe640 1 -- 172.21.15.67:0/3042415594 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c20034440 con 0x7f6c2815dfe0 2024-03-24T00:39:10.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.690+0000 7f6c2ef97640 1 -- 172.21.15.67:0/3042415594 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c2815dfe0 msgr2=0x7f6c280ceb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:10.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.690+0000 7f6c2ef97640 1 --2- 172.21.15.67:0/3042415594 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c2815dfe0 0x7f6c280ceb10 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6c20009d90 tx=0x7f6c2002bea0 comp rx=0 tx=0).stop 2024-03-24T00:39:10.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.690+0000 7f6c2ef97640 1 -- 172.21.15.67:0/3042415594 shutdown_connections 2024-03-24T00:39:10.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.690+0000 7f6c2ef97640 1 --2- 172.21.15.67:0/3042415594 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6c28161bb0 0x7f6c280cf050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:10.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.690+0000 7f6c2ef97640 1 --2- 172.21.15.67:0/3042415594 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c2815dfe0 0x7f6c280ceb10 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:10.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.690+0000 7f6c2ef97640 1 -- 172.21.15.67:0/3042415594 >> 172.21.15.67:0/3042415594 conn(0x7f6c2806c890 msgr2=0x7f6c2806cc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:10.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.690+0000 7f6c2ef97640 1 -- 172.21.15.67:0/3042415594 shutdown_connections 2024-03-24T00:39:10.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.690+0000 7f6c2ef97640 1 -- 172.21.15.67:0/3042415594 wait complete. 2024-03-24T00:39:10.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.691+0000 7f6c2ef97640 1 Processor -- start 2024-03-24T00:39:10.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.691+0000 7f6c2ef97640 1 -- start start 2024-03-24T00:39:10.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.692+0000 7f6c2ef97640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6c280b96d0 0x7f6c2814eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:10.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.692+0000 7f6c2ef97640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6c2815dfe0 0x7f6c28152ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:10.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.692+0000 7f6c2ef97640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c28161bb0 0x7f6c2815a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:10.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.692+0000 7f6c2ef97640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c280d1120 con 0x7f6c28161bb0 2024-03-24T00:39:10.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.692+0000 7f6c2ef97640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c2815e3c0 con 0x7f6c280b96d0 2024-03-24T00:39:10.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.692+0000 7f6c2ef97640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c280d12a0 con 0x7f6c2815dfe0 2024-03-24T00:39:10.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.692+0000 7f6c2d534640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c28161bb0 0x7f6c2815a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:10.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.692+0000 7f6c2cd33640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6c280b96d0 0x7f6c2814eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:10.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.692+0000 7f6c2d534640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c28161bb0 0x7f6c2815a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60780/0 (socket says 172.21.15.67:60780) 2024-03-24T00:39:10.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.692+0000 7f6c27fff640 1 --2- 172.21.15.67:0/189621813 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6c2815dfe0 0x7f6c28152ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:10.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.692+0000 7f6c2d534640 1 -- 172.21.15.67:0/189621813 learned_addr learned my addr 172.21.15.67:0/189621813 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:10.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.692+0000 7f6c2cd33640 1 -- 172.21.15.67:0/189621813 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6c2815dfe0 msgr2=0x7f6c28152ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:10.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.692+0000 7f6c2cd33640 1 --2- 172.21.15.67:0/189621813 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6c2815dfe0 0x7f6c28152ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:10.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.692+0000 7f6c2cd33640 1 -- 172.21.15.67:0/189621813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c28161bb0 msgr2=0x7f6c2815a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:10.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.692+0000 7f6c2cd33640 1 --2- 172.21.15.67:0/189621813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c28161bb0 0x7f6c2815a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:10.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.692+0000 7f6c2cd33640 1 -- 172.21.15.67:0/189621813 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c2815b070 con 0x7f6c280b96d0 2024-03-24T00:39:10.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.692+0000 7f6c27fff640 1 --2- 172.21.15.67:0/189621813 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6c2815dfe0 0x7f6c28152ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:10.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.692+0000 7f6c2d534640 1 --2- 172.21.15.67:0/189621813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c28161bb0 0x7f6c2815a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:10.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.693+0000 7f6c2cd33640 1 --2- 172.21.15.67:0/189621813 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6c280b96d0 0x7f6c2814eb10 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6c20033000 tx=0x7f6c20004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:10.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.693+0000 7f6c257fa640 1 -- 172.21.15.67:0/189621813 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c20039050 con 0x7f6c280b96d0 2024-03-24T00:39:10.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.693+0000 7f6c257fa640 1 -- 172.21.15.67:0/189621813 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c20045070 con 0x7f6c280b96d0 2024-03-24T00:39:10.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.693+0000 7f6c257fa640 1 -- 172.21.15.67:0/189621813 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c2003e610 con 0x7f6c280b96d0 2024-03-24T00:39:10.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.693+0000 7f6c2ef97640 1 -- 172.21.15.67:0/189621813 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c2815b350 con 0x7f6c280b96d0 2024-03-24T00:39:10.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.693+0000 7f6c2ef97640 1 -- 172.21.15.67:0/189621813 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c2815b910 con 0x7f6c280b96d0 2024-03-24T00:39:10.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.694+0000 7f6c2ef97640 1 -- 172.21.15.67:0/189621813 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c2806b180 con 0x7f6c280b96d0 2024-03-24T00:39:10.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.694+0000 7f6c257fa640 1 -- 172.21.15.67:0/189621813 <== mon.1 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6c2003e7b0 con 0x7f6c280b96d0 2024-03-24T00:39:10.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.694+0000 7f6c257fa640 1 --2- 172.21.15.67:0/189621813 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6c1406bbd0 0x7f6c1406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:10.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.695+0000 7f6c257fa640 1 -- 172.21.15.67:0/189621813 <== mon.1 v2:172.21.15.72:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f6c200ad6e0 con 0x7f6c280b96d0 2024-03-24T00:39:10.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.695+0000 7f6c27fff640 1 --2- 172.21.15.67:0/189621813 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6c1406bbd0 0x7f6c1406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:10.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.695+0000 7f6c27fff640 1 --2- 172.21.15.67:0/189621813 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6c1406bbd0 0x7f6c1406e090 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6c281601e0 tx=0x7f6c1c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:10.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.699+0000 7f6c257fa640 1 -- 172.21.15.67:0/189621813 <== mon.1 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6c2007b840 con 0x7f6c280b96d0 2024-03-24T00:39:10.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.883+0000 7f6c2ef97640 1 -- 172.21.15.67:0/189621813 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6c280d1a10 con 0x7f6c280b96d0 2024-03-24T00:39:10.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.884+0000 7f6c257fa640 1 -- 172.21.15.67:0/189621813 <== mon.1 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18454 (secure 0 0 0) 0x7f6c2007f8e0 con 0x7f6c280b96d0 2024-03-24T00:39:10.885 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:39:10.886 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":25,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:38:58.797179+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:38:50.077085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:38:51.095454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T00:38:54.807117+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:39:10.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.887+0000 7f6c2ef97640 1 -- 172.21.15.67:0/189621813 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6c1406bbd0 msgr2=0x7f6c1406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:10.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.887+0000 7f6c2ef97640 1 --2- 172.21.15.67:0/189621813 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6c1406bbd0 0x7f6c1406e090 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6c281601e0 tx=0x7f6c1c009190 comp rx=0 tx=0).stop 2024-03-24T00:39:10.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.887+0000 7f6c2ef97640 1 -- 172.21.15.67:0/189621813 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6c280b96d0 msgr2=0x7f6c2814eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:10.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.887+0000 7f6c2ef97640 1 --2- 172.21.15.67:0/189621813 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6c280b96d0 0x7f6c2814eb10 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6c20033000 tx=0x7f6c20004a40 comp rx=0 tx=0).stop 2024-03-24T00:39:10.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.887+0000 7f6c2ef97640 1 -- 172.21.15.67:0/189621813 shutdown_connections 2024-03-24T00:39:10.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.887+0000 7f6c2ef97640 1 --2- 172.21.15.67:0/189621813 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6c1406bbd0 0x7f6c1406e090 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:10.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.887+0000 7f6c2ef97640 1 --2- 172.21.15.67:0/189621813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c28161bb0 0x7f6c2815a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:10.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.887+0000 7f6c2ef97640 1 --2- 172.21.15.67:0/189621813 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6c2815dfe0 0x7f6c28152ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:10.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.887+0000 7f6c2ef97640 1 --2- 172.21.15.67:0/189621813 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6c280b96d0 0x7f6c2814eb10 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:10.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.887+0000 7f6c2ef97640 1 -- 172.21.15.67:0/189621813 >> 172.21.15.67:0/189621813 conn(0x7f6c2806c890 msgr2=0x7f6c280b8840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:10.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.887+0000 7f6c2ef97640 1 -- 172.21.15.67:0/189621813 shutdown_connections 2024-03-24T00:39:10.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:10.887+0000 7f6c2ef97640 1 -- 172.21.15.67:0/189621813 wait complete. 2024-03-24T00:39:10.899 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-24T00:39:10.899 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:39:11.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.000+0000 7f673ae97640 1 Processor -- start 2024-03-24T00:39:11.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.000+0000 7f673ae97640 1 -- start start 2024-03-24T00:39:11.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.000+0000 7f673ae97640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f673409a480 0x7f67340cfd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.000+0000 7f673ae97640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67340d1930 con 0x7f673406cee0 2024-03-24T00:39:11.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.000+0000 7f673ae97640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f673406d390 con 0x7f67340d1550 2024-03-24T00:39:11.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.000+0000 7f673ae97640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f673406ca50 con 0x7f673409a480 2024-03-24T00:39:11.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.001+0000 7f6733fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f673406cee0 0x7f6734099d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45422/0 (socket says 172.21.15.67:45422) 2024-03-24T00:39:11.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.001+0000 7f6733fff640 1 -- 172.21.15.67:0/2452877537 learned_addr learned my addr 172.21.15.67:0/2452877537 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:11.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.001+0000 7f6739434640 1 --2- 172.21.15.67:0/2452877537 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f673409a480 0x7f67340cfd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.001+0000 7f67337fe640 1 -- 172.21.15.67:0/2452877537 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3061641919 0 0) 0x7f67340d1930 con 0x7f673406cee0 2024-03-24T00:39:11.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.001+0000 7f67337fe640 1 -- 172.21.15.67:0/2452877537 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6724003520 con 0x7f673406cee0 2024-03-24T00:39:11.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.001+0000 7f67337fe640 1 -- 172.21.15.67:0/2452877537 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1444848238 0 0) 0x7f673406d390 con 0x7f67340d1550 2024-03-24T00:39:11.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.001+0000 7f67337fe640 1 -- 172.21.15.67:0/2452877537 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67340d1930 con 0x7f67340d1550 2024-03-24T00:39:11.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.001+0000 7f6739434640 1 -- 172.21.15.67:0/2452877537 >> v1:172.21.15.72:6789/0 conn(0x7f67340d1550 legacy=0x7f67340b2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.001+0000 7f6739434640 1 -- 172.21.15.67:0/2452877537 >> v1:172.21.15.67:6789/0 conn(0x7f673406cee0 legacy=0x7f6734099d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.001+0000 7f6739434640 1 -- 172.21.15.67:0/2452877537 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67340b1f60 con 0x7f673409a480 2024-03-24T00:39:11.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.002+0000 7f6739434640 1 --2- 172.21.15.67:0/2452877537 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f673409a480 0x7f67340cfd60 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f672c009df0 tx=0x7f672c02bf00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dca916aa4f7d815 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.002+0000 7f67337fe640 1 -- 172.21.15.67:0/2452877537 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f672c039070 con 0x7f673409a480 2024-03-24T00:39:11.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.002+0000 7f67337fe640 1 -- 172.21.15.67:0/2452877537 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f672c02ccb0 con 0x7f673409a480 2024-03-24T00:39:11.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.002+0000 7f67337fe640 1 -- 172.21.15.67:0/2452877537 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f672c02ce60 con 0x7f673409a480 2024-03-24T00:39:11.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.002+0000 7f673ae97640 1 -- 172.21.15.67:0/2452877537 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f673409a480 msgr2=0x7f67340cfd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.002+0000 7f673ae97640 1 --2- 172.21.15.67:0/2452877537 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f673409a480 0x7f67340cfd60 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f672c009df0 tx=0x7f672c02bf00 comp rx=0 tx=0).stop 2024-03-24T00:39:11.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.002+0000 7f673ae97640 1 -- 172.21.15.67:0/2452877537 shutdown_connections 2024-03-24T00:39:11.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.002+0000 7f673ae97640 1 --2- 172.21.15.67:0/2452877537 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f673409a480 0x7f67340cfd60 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.002+0000 7f673ae97640 1 -- 172.21.15.67:0/2452877537 >> 172.21.15.67:0/2452877537 conn(0x7f67340b36e0 msgr2=0x7f67340b3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:11.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.003+0000 7f673ae97640 1 -- 172.21.15.67:0/2452877537 shutdown_connections 2024-03-24T00:39:11.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.003+0000 7f673ae97640 1 -- 172.21.15.67:0/2452877537 wait complete. 2024-03-24T00:39:11.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.004+0000 7f673ae97640 1 Processor -- start 2024-03-24T00:39:11.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.004+0000 7f673ae97640 1 -- start start 2024-03-24T00:39:11.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.004+0000 7f673ae97640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f673406cee0 0x7f67340b6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.004+0000 7f673ae97640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f673409a480 0x7f67340c9890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.004+0000 7f673ae97640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67340d1550 0x7f67341f47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.004+0000 7f673ae97640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f673406c400 con 0x7f67340d1550 2024-03-24T00:39:11.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.004+0000 7f673ae97640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67340b2520 con 0x7f673406cee0 2024-03-24T00:39:11.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.004+0000 7f673ae97640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f673406c6e0 con 0x7f673409a480 2024-03-24T00:39:11.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.005+0000 7f6739434640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67340d1550 0x7f67341f47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.005+0000 7f6738c33640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f673406cee0 0x7f67340b6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.005+0000 7f6739434640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67340d1550 0x7f67341f47b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60790/0 (socket says 172.21.15.67:60790) 2024-03-24T00:39:11.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.005+0000 7f6739434640 1 -- 172.21.15.67:0/1887322084 learned_addr learned my addr 172.21.15.67:0/1887322084 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:11.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.005+0000 7f6738c33640 1 --2- 172.21.15.67:0/1887322084 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f673406cee0 0x7f67340b6050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:34022/0 (socket says 172.21.15.67:34022) 2024-03-24T00:39:11.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.005+0000 7f6733fff640 1 --2- 172.21.15.67:0/1887322084 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f673409a480 0x7f67340c9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.005+0000 7f6739434640 1 -- 172.21.15.67:0/1887322084 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f673409a480 msgr2=0x7f67340c9890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.005+0000 7f6739434640 1 --2- 172.21.15.67:0/1887322084 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f673409a480 0x7f67340c9890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.005+0000 7f6739434640 1 -- 172.21.15.67:0/1887322084 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f673406cee0 msgr2=0x7f67340b6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.005+0000 7f6739434640 1 --2- 172.21.15.67:0/1887322084 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f673406cee0 0x7f67340b6050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.005+0000 7f6739434640 1 -- 172.21.15.67:0/1887322084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67340c56e0 con 0x7f67340d1550 2024-03-24T00:39:11.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.005+0000 7f6738c33640 1 --2- 172.21.15.67:0/1887322084 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f673406cee0 0x7f67340b6050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:11.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.005+0000 7f6739434640 1 --2- 172.21.15.67:0/1887322084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67340d1550 0x7f67341f47b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f672c009d90 tx=0x7f672c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.005+0000 7f67317fa640 1 -- 172.21.15.67:0/1887322084 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f672c039050 con 0x7f67340d1550 2024-03-24T00:39:11.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.005+0000 7f67317fa640 1 -- 172.21.15.67:0/1887322084 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f672c045070 con 0x7f67340d1550 2024-03-24T00:39:11.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.005+0000 7f673ae97640 1 -- 172.21.15.67:0/1887322084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67340c59c0 con 0x7f67340d1550 2024-03-24T00:39:11.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.005+0000 7f67317fa640 1 -- 172.21.15.67:0/1887322084 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f672c02ce60 con 0x7f67340d1550 2024-03-24T00:39:11.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.005+0000 7f673ae97640 1 -- 172.21.15.67:0/1887322084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67340c5f80 con 0x7f67340d1550 2024-03-24T00:39:11.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.006+0000 7f673ae97640 1 -- 172.21.15.67:0/1887322084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f673404f400 con 0x7f67340d1550 2024-03-24T00:39:11.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.007+0000 7f67317fa640 1 -- 172.21.15.67:0/1887322084 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f672c03e850 con 0x7f67340d1550 2024-03-24T00:39:11.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.007+0000 7f67317fa640 1 --2- 172.21.15.67:0/1887322084 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f671806bbd0 0x7f671806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.007+0000 7f67317fa640 1 -- 172.21.15.67:0/1887322084 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f672c0ad940 con 0x7f67340d1550 2024-03-24T00:39:11.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.007+0000 7f6738c33640 1 --2- 172.21.15.67:0/1887322084 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f671806bbd0 0x7f671806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.008+0000 7f6738c33640 1 --2- 172.21.15.67:0/1887322084 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f671806bbd0 0x7f671806e090 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6728007fd0 tx=0x7f6728009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.011+0000 7f67317fa640 1 -- 172.21.15.67:0/1887322084 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f672c07bbd0 con 0x7f67340d1550 2024-03-24T00:39:11.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.195+0000 7f673ae97640 1 -- 172.21.15.67:0/1887322084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f67340b1310 con 0x7f67340d1550 2024-03-24T00:39:11.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.196+0000 7f67317fa640 1 -- 172.21.15.67:0/1887322084 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18454 (secure 0 0 0) 0x7f672c07fc70 con 0x7f67340d1550 2024-03-24T00:39:11.197 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:39:11.197 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":25,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:38:58.797179+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:38:50.077085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:38:51.095454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T00:38:54.807117+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:39:11.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.198+0000 7f673ae97640 1 -- 172.21.15.67:0/1887322084 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f671806bbd0 msgr2=0x7f671806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.198+0000 7f673ae97640 1 --2- 172.21.15.67:0/1887322084 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f671806bbd0 0x7f671806e090 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6728007fd0 tx=0x7f6728009040 comp rx=0 tx=0).stop 2024-03-24T00:39:11.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.198+0000 7f673ae97640 1 -- 172.21.15.67:0/1887322084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67340d1550 msgr2=0x7f67341f47b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.198+0000 7f673ae97640 1 --2- 172.21.15.67:0/1887322084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67340d1550 0x7f67341f47b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f672c009d90 tx=0x7f672c004a40 comp rx=0 tx=0).stop 2024-03-24T00:39:11.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.198+0000 7f673ae97640 1 -- 172.21.15.67:0/1887322084 shutdown_connections 2024-03-24T00:39:11.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.198+0000 7f673ae97640 1 --2- 172.21.15.67:0/1887322084 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f671806bbd0 0x7f671806e090 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.198+0000 7f673ae97640 1 --2- 172.21.15.67:0/1887322084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67340d1550 0x7f67341f47b0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.198+0000 7f673ae97640 1 --2- 172.21.15.67:0/1887322084 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f673409a480 0x7f67340c9890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.198+0000 7f673ae97640 1 --2- 172.21.15.67:0/1887322084 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f673406cee0 0x7f67340b6050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.198+0000 7f673ae97640 1 -- 172.21.15.67:0/1887322084 >> 172.21.15.67:0/1887322084 conn(0x7f67340b36e0 msgr2=0x7f67340b0860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:11.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.198+0000 7f673ae97640 1 -- 172.21.15.67:0/1887322084 shutdown_connections 2024-03-24T00:39:11.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.198+0000 7f673ae97640 1 -- 172.21.15.67:0/1887322084 wait complete. 2024-03-24T00:39:11.210 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-24T00:39:11.211 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-24T00:39:11.211 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-24T00:39:11.212 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-24T00:39:11.212 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-24T00:39:11.213 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-24T00:39:11.213 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-24T00:39:11.213 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-24T00:39:11.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.382+0000 7fec7cb6b640 1 Processor -- start 2024-03-24T00:39:11.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.382+0000 7fec7cb6b640 1 -- start start 2024-03-24T00:39:11.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.384+0000 7fec7cb6b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fec78159c70 0x7fec780b3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.384+0000 7fec7cb6b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fec780aee10 0x7fec7815d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.384+0000 7fec7cb6b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec780b3f10 con 0x7fec7815a050 2024-03-24T00:39:11.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.384+0000 7fec7cb6b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec780b41f0 con 0x7fec78159c70 2024-03-24T00:39:11.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.384+0000 7fec7cb6b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec7815a4e0 con 0x7fec780aee10 2024-03-24T00:39:11.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.384+0000 7fec777fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fec78159c70 0x7fec780b3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.384+0000 7fec777fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fec78159c70 0x7fec780b3450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60806/0 (socket says 172.21.15.67:60806) 2024-03-24T00:39:11.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.384+0000 7fec777fe640 1 -- 172.21.15.67:0/1382905452 learned_addr learned my addr 172.21.15.67:0/1382905452 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:11.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.385+0000 7f39d6e9a640 1 Processor -- start 2024-03-24T00:39:11.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.385+0000 7f9e62464640 1 Processor -- start 2024-03-24T00:39:11.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.385+0000 7f39d6e9a640 1 -- start start 2024-03-24T00:39:11.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.384+0000 7fec77fff640 1 --2- 172.21.15.67:0/1382905452 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fec780aee10 0x7fec7815d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.385+0000 7fec777fe640 1 -- 172.21.15.67:0/1382905452 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fec780aee10 msgr2=0x7fec7815d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:11.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.385+0000 7fec777fe640 1 --2- 172.21.15.67:0/1382905452 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fec780aee10 0x7fec7815d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.385+0000 7fec777fe640 1 -- 172.21.15.67:0/1382905452 >> v1:172.21.15.67:6789/0 conn(0x7fec7815a050 legacy=0x7fec780ae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.386+0000 7f04b32c0640 1 Processor -- start 2024-03-24T00:39:11.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.385+0000 7fec777fe640 1 -- 172.21.15.67:0/1382905452 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec7815a7c0 con 0x7fec78159c70 2024-03-24T00:39:11.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.386+0000 7fec777fe640 1 --2- 172.21.15.67:0/1382905452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec78159c70 0x7fec780b3450 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fec6800b1e0 tx=0x7fec6802c210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a510b179c1a73020 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.386+0000 7fec767fc640 1 -- 172.21.15.67:0/1382905452 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec68039070 con 0x7fec78159c70 2024-03-24T00:39:11.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.386+0000 7fec767fc640 1 -- 172.21.15.67:0/1382905452 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec68034440 con 0x7fec78159c70 2024-03-24T00:39:11.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.386+0000 7f3f1aec6640 1 Processor -- start 2024-03-24T00:39:11.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.386+0000 7f9e62464640 1 -- start start 2024-03-24T00:39:11.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.387+0000 7f9e62464640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f9e540ead00 0x7f9e5404fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.387+0000 7f9e62464640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9e5404ffd0 0x7f9e54037bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.387+0000 7f9e62464640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e54053720 con 0x7f9e54038110 2024-03-24T00:39:11.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.387+0000 7f9e62464640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e54054330 con 0x7f9e5404ffd0 2024-03-24T00:39:11.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.387+0000 7f9e62464640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e54053a00 con 0x7f9e540ead00 2024-03-24T00:39:11.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.387+0000 7f9e61462640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f9e540ead00 0x7f9e5404fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.387+0000 7f9e61462640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f9e540ead00 0x7f9e5404fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:34026/0 (socket says 172.21.15.67:34026) 2024-03-24T00:39:11.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.387+0000 7f9e61462640 1 -- 172.21.15.67:0/1129510977 learned_addr learned my addr 172.21.15.67:0/1129510977 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:11.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.387+0000 7f39d6e9a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f39c803acd0 0x7f39c8053460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.387+0000 7f39d6e9a640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f39c8054c50 0x7f39c80df380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.387+0000 7f39d6e9a640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39c8053bb0 con 0x7f39c8050010 2024-03-24T00:39:11.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.387+0000 7f39d6e9a640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39c80547c0 con 0x7f39c803acd0 2024-03-24T00:39:11.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.387+0000 7f39d6e9a640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39c8053e90 con 0x7f39c8054c50 2024-03-24T00:39:11.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.386+0000 7f3f1aec6640 1 -- start start 2024-03-24T00:39:11.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.386+0000 7f04b32c0640 1 -- start start 2024-03-24T00:39:11.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.387+0000 7f04b32c0640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04ac15ac20 0x7f04ac15e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.387+0000 7f04b32c0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f04ac0d0ac0 0x7f04ac0d0f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.387+0000 7f04b32c0640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04ac15f1c0 con 0x7f04ac0cec50 2024-03-24T00:39:11.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.388+0000 7f04b32c0640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04ac0b0bc0 con 0x7f04ac0d0ac0 2024-03-24T00:39:11.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.388+0000 7f04b32c0640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac15f4a0 con 0x7f04ac15ac20 2024-03-24T00:39:11.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.387+0000 7fec7cb6b640 1 -- 172.21.15.67:0/1382905452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec78159c70 msgr2=0x7fec780b3450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.387+0000 7fec7cb6b640 1 --2- 172.21.15.67:0/1382905452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec78159c70 0x7fec780b3450 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fec6800b1e0 tx=0x7fec6802c210 comp rx=0 tx=0).stop 2024-03-24T00:39:11.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.387+0000 7fec7cb6b640 1 -- 172.21.15.67:0/1382905452 shutdown_connections 2024-03-24T00:39:11.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.387+0000 7fec7cb6b640 1 --2- 172.21.15.67:0/1382905452 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fec780aee10 0x7fec7815d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.387+0000 7fec7cb6b640 1 --2- 172.21.15.67:0/1382905452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec78159c70 0x7fec780b3450 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.387+0000 7fec7cb6b640 1 -- 172.21.15.67:0/1382905452 >> 172.21.15.67:0/1382905452 conn(0x7fec780b13e0 msgr2=0x7fec780b17f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:11.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.388+0000 7f3f1aec6640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3f0c053e50 0x7f3f0c0542e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.388+0000 7f3f1aec6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3f0c04f930 0x7f3f0c04fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.388+0000 7f3f1aec6640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f0c005f10 con 0x7f3f0c048100 2024-03-24T00:39:11.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.388+0000 7f3f1aec6640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f0c0e7b90 con 0x7f3f0c053e50 2024-03-24T00:39:11.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.388+0000 7f3f1aec6640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0c006ae0 con 0x7f3f0c04f930 2024-03-24T00:39:11.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.388+0000 7f04b085b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f04ac0cec50 0x7f04ac0cb6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45462/0 (socket says 172.21.15.67:45462) 2024-03-24T00:39:11.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.388+0000 7f3f19463640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3f0c04f930 0x7f3f0c04fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.388+0000 7f04b085b640 1 -- 172.21.15.67:0/4221134434 learned_addr learned my addr 172.21.15.67:0/4221134434 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:11.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.388+0000 7f3f19463640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3f0c04f930 0x7f3f0c04fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.67:48974/0 (socket says 172.21.15.67:48974) 2024-03-24T00:39:11.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.388+0000 7f39d4c36640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f39c803acd0 0x7f39c8053460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.388+0000 7f39d4c36640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f39c803acd0 0x7f39c8053460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60830/0 (socket says 172.21.15.67:60830) 2024-03-24T00:39:11.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.388+0000 7f39d4c36640 1 -- 172.21.15.67:0/4159041165 learned_addr learned my addr 172.21.15.67:0/4159041165 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:11.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.388+0000 7f3f19463640 1 -- 172.21.15.67:0/1096473855 learned_addr learned my addr 172.21.15.67:0/1096473855 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:11.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.388+0000 7f39cffff640 1 --2- 172.21.15.67:0/4159041165 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f39c8054c50 0x7f39c80df380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.388+0000 7f3f13fff640 1 --2- 172.21.15.67:0/1096473855 >> v2:172.21.15.72:3300/0 conn(0x7f3f0c053e50 0x7f3f0c0542e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.388+0000 7f04b105c640 1 --2- 172.21.15.67:0/4221134434 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04ac15ac20 0x7f04ac15e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.388+0000 7f39cffff640 1 -- 172.21.15.67:0/4159041165 >> v2:172.21.15.67:3300/0 conn(0x7f39c803acd0 msgr2=0x7f39c8053460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:11.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.388+0000 7f39cffff640 1 --2- 172.21.15.67:0/4159041165 >> v2:172.21.15.67:3300/0 conn(0x7f39c803acd0 0x7f39c8053460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.388+0000 7f39cffff640 1 -- 172.21.15.67:0/4159041165 >> v1:172.21.15.67:6789/0 conn(0x7f39c8050010 legacy=0x7f39c80384e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.388+0000 7f04b185d640 1 --2- 172.21.15.67:0/4221134434 >> v2:172.21.15.67:3300/0 conn(0x7f04ac0d0ac0 0x7f04ac0d0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.388+0000 7f3f19463640 1 -- 172.21.15.67:0/1096473855 >> v2:172.21.15.72:3300/0 conn(0x7f3f0c053e50 msgr2=0x7f3f0c0542e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:11.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.387+0000 7f9e61462640 1 -- 172.21.15.67:0/1129510977 >> v2:172.21.15.67:3300/0 conn(0x7f9e5404ffd0 msgr2=0x7f9e54037bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:11.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.387+0000 7f9e61462640 1 --2- 172.21.15.67:0/1129510977 >> v2:172.21.15.67:3300/0 conn(0x7f9e5404ffd0 0x7f9e54037bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.387+0000 7f9e61462640 1 -- 172.21.15.67:0/1129510977 >> v1:172.21.15.72:6789/0 conn(0x7f9e54038110 legacy=0x7f9e54038580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:11.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.388+0000 7f9e61462640 1 -- 172.21.15.67:0/1129510977 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e54053ce0 con 0x7f9e540ead00 2024-03-24T00:39:11.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.388+0000 7f3f19463640 1 --2- 172.21.15.67:0/1096473855 >> v2:172.21.15.72:3300/0 conn(0x7f3f0c053e50 0x7f3f0c0542e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.388+0000 7f3f19463640 1 -- 172.21.15.67:0/1096473855 >> v1:172.21.15.72:6789/0 conn(0x7f3f0c048100 legacy=0x7f3f0c006f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.388+0000 7f3f19463640 1 -- 172.21.15.67:0/1096473855 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f0c053280 con 0x7f3f0c04f930 2024-03-24T00:39:11.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.389+0000 7f9e61462640 1 --2- 172.21.15.67:0/1129510977 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9e540ead00 0x7f9e5404fa90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c04ff90 tx=0x7f9e5c06a6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb4b9b7df46fcd71 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.389+0000 7f9e53fff640 1 -- 172.21.15.67:0/1129510977 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e5c0b4070 con 0x7f9e540ead00 2024-03-24T00:39:11.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.389+0000 7f9e53fff640 1 -- 172.21.15.67:0/1129510977 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e5c0a7de0 con 0x7f9e540ead00 2024-03-24T00:39:11.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.389+0000 7f9e53fff640 1 -- 172.21.15.67:0/1129510977 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e5c0af6a0 con 0x7f9e540ead00 2024-03-24T00:39:11.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.388+0000 7f39cffff640 1 -- 172.21.15.67:0/4159041165 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39c8054170 con 0x7f39c8054c50 2024-03-24T00:39:11.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.389+0000 7f04b105c640 1 -- 172.21.15.67:0/4221134434 >> v2:172.21.15.67:3300/0 conn(0x7f04ac0d0ac0 msgr2=0x7f04ac0d0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:11.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.389+0000 7f04b105c640 1 --2- 172.21.15.67:0/4221134434 >> v2:172.21.15.67:3300/0 conn(0x7f04ac0d0ac0 0x7f04ac0d0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.389+0000 7f04b105c640 1 -- 172.21.15.67:0/4221134434 >> v1:172.21.15.67:6789/0 conn(0x7f04ac0cec50 legacy=0x7f04ac0cb6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.389+0000 7fec7cb6b640 1 -- 172.21.15.67:0/1382905452 shutdown_connections 2024-03-24T00:39:11.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.389+0000 7f04b105c640 1 -- 172.21.15.67:0/4221134434 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04ac15f780 con 0x7f04ac15ac20 2024-03-24T00:39:11.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.389+0000 7fec7cb6b640 1 -- 172.21.15.67:0/1382905452 wait complete. 2024-03-24T00:39:11.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.389+0000 7f345520b640 1 Processor -- start 2024-03-24T00:39:11.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.389+0000 7fab6896f640 1 Processor -- start 2024-03-24T00:39:11.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.389+0000 7fab6896f640 1 -- start start 2024-03-24T00:39:11.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.390+0000 7f0c0ac13640 1 Processor -- start 2024-03-24T00:39:11.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.390+0000 7f3f19463640 1 --2- 172.21.15.67:0/1096473855 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3f0c04f930 0x7f3f0c04fd60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c006c60 tx=0x7f3f1406a6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb23aad4d8be3450 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.390+0000 7f0c0ac13640 1 -- start start 2024-03-24T00:39:11.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.390+0000 7f3f137fe640 1 -- 172.21.15.67:0/1096473855 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f140b4070 con 0x7f3f0c04f930 2024-03-24T00:39:11.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.390+0000 7f3f137fe640 1 -- 172.21.15.67:0/1096473855 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f140af440 con 0x7f3f0c04f930 2024-03-24T00:39:11.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.390+0000 7f3f137fe640 1 -- 172.21.15.67:0/1096473855 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f140af8d0 con 0x7f3f0c04f930 2024-03-24T00:39:11.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.390+0000 7f9e62464640 1 -- 172.21.15.67:0/1129510977 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9e540ead00 msgr2=0x7f9e5404fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.390+0000 7f9e62464640 1 --2- 172.21.15.67:0/1129510977 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9e540ead00 0x7f9e5404fa90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c04ff90 tx=0x7f9e5c06a6a0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.389+0000 7f345520b640 1 -- start start 2024-03-24T00:39:11.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.390+0000 7f9e62464640 1 -- 172.21.15.67:0/1129510977 shutdown_connections 2024-03-24T00:39:11.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.390+0000 7f9e62464640 1 --2- 172.21.15.67:0/1129510977 >> v2:172.21.15.67:3300/0 conn(0x7f9e5404ffd0 0x7f9e54037bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.390+0000 7f9e62464640 1 --2- 172.21.15.67:0/1129510977 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9e540ead00 0x7f9e5404fa90 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.390+0000 7f9e62464640 1 -- 172.21.15.67:0/1129510977 >> 172.21.15.67:0/1129510977 conn(0x7f9e54009fb0 msgr2=0x7f9e5400a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:11.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.390+0000 7f04b105c640 1 --2- 172.21.15.67:0/4221134434 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04ac15ac20 0x7f04ac15e8c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f04a000ba40 tx=0x7f04a0030280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce46fcdf73eedb2f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.390+0000 7f049ffff640 1 -- 172.21.15.67:0/4221134434 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a003d070 con 0x7f04ac15ac20 2024-03-24T00:39:11.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.390+0000 7f9e62464640 1 -- 172.21.15.67:0/1129510977 shutdown_connections 2024-03-24T00:39:11.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.390+0000 7f9e62464640 1 -- 172.21.15.67:0/1129510977 wait complete. 2024-03-24T00:39:11.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.390+0000 7f049ffff640 1 -- 172.21.15.67:0/4221134434 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04a0038640 con 0x7f04ac15ac20 2024-03-24T00:39:11.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.390+0000 7f049ffff640 1 -- 172.21.15.67:0/4221134434 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a0038a10 con 0x7f04ac15ac20 2024-03-24T00:39:11.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.391+0000 7f39cffff640 1 --2- 172.21.15.67:0/4159041165 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f39c8054c50 0x7f39c80df380 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f39bc0097a0 tx=0x7f39bc02b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=279815d54522fa7b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.391+0000 7f3f1aec6640 1 -- 172.21.15.67:0/1096473855 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3f0c04f930 msgr2=0x7f3f0c04fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.391+0000 7f3f1aec6640 1 --2- 172.21.15.67:0/1096473855 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3f0c04f930 0x7f3f0c04fd60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c006c60 tx=0x7f3f1406a6a0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.391+0000 7fab6896f640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fab64162370 0x7fab6406f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.391+0000 7fab6896f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab6406e490 con 0x7fab640724f0 2024-03-24T00:39:11.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.391+0000 7fab6896f640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab6406e770 con 0x7fab6406df20 2024-03-24T00:39:11.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.391+0000 7fab6896f640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab6406ea50 con 0x7fab64162370 2024-03-24T00:39:11.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.391+0000 7f345520b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f34500ced80 0x7f34500cf1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.391+0000 7f345520b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34500cc770 con 0x7f345015a870 2024-03-24T00:39:11.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.391+0000 7f345520b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f345006e2c0 con 0x7f345015a490 2024-03-24T00:39:11.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.391+0000 7f345520b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f345006e5a0 con 0x7f34500ced80 2024-03-24T00:39:11.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.391+0000 7f3f1aec6640 1 -- 172.21.15.67:0/1096473855 shutdown_connections 2024-03-24T00:39:11.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.391+0000 7f3f1aec6640 1 --2- 172.21.15.67:0/1096473855 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3f0c04f930 0x7f3f0c04fd60 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.391+0000 7f3f1aec6640 1 --2- 172.21.15.67:0/1096473855 >> v2:172.21.15.72:3300/0 conn(0x7f3f0c053e50 0x7f3f0c0542e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.391+0000 7f3f1aec6640 1 -- 172.21.15.67:0/1096473855 >> 172.21.15.67:0/1096473855 conn(0x7f3f0c009fe0 msgr2=0x7f3f0c00a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:11.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.391+0000 7f3f1aec6640 1 -- 172.21.15.67:0/1096473855 shutdown_connections 2024-03-24T00:39:11.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.391+0000 7f344f59e640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f345015a870 0x7f345015ad00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45472/0 (socket says 172.21.15.67:45472) 2024-03-24T00:39:11.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.391+0000 7f344f59e640 1 -- 172.21.15.67:0/1999473146 learned_addr learned my addr 172.21.15.67:0/1999473146 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:11.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.391+0000 7f3f1aec6640 1 -- 172.21.15.67:0/1096473855 wait complete. 2024-03-24T00:39:11.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.391+0000 7f04b32c0640 1 -- 172.21.15.67:0/4221134434 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04ac15ac20 msgr2=0x7f04ac15e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.391+0000 7f04b32c0640 1 --2- 172.21.15.67:0/4221134434 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04ac15ac20 0x7f04ac15e8c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f04a000ba40 tx=0x7f04a0030280 comp rx=0 tx=0).stop 2024-03-24T00:39:11.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.391+0000 7f0c0ac13640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f0c04159f30 0x7f0c040ccd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.392+0000 7f0c0ac13640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0c040b2c10 0x7f0c04071850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.392+0000 7f04b32c0640 1 -- 172.21.15.67:0/4221134434 shutdown_connections 2024-03-24T00:39:11.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.392+0000 7f04b32c0640 1 --2- 172.21.15.67:0/4221134434 >> v2:172.21.15.67:3300/0 conn(0x7f04ac0d0ac0 0x7f04ac0d0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.392+0000 7f04b32c0640 1 --2- 172.21.15.67:0/4221134434 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04ac15ac20 0x7f04ac15e8c0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.392+0000 7f04b32c0640 1 -- 172.21.15.67:0/4221134434 >> 172.21.15.67:0/4221134434 conn(0x7f04ac099a30 msgr2=0x7f04ac099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:11.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.392+0000 7f0c0ac13640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c0406da30 con 0x7f0c040b3070 2024-03-24T00:39:11.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.392+0000 7f0c09c11640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f0c04159f30 0x7f0c040ccd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.392+0000 7f0c09410640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0c040b2c10 0x7f0c04071850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.391+0000 7fec7cb6b640 1 Processor -- start 2024-03-24T00:39:11.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.391+0000 7fec7cb6b640 1 -- start start 2024-03-24T00:39:11.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.392+0000 7f0c09c11640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f0c04159f30 0x7f0c040ccd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:34056/0 (socket says 172.21.15.67:34056) 2024-03-24T00:39:11.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.392+0000 7f0c09c11640 1 -- 172.21.15.67:0/2536990068 learned_addr learned my addr 172.21.15.67:0/2536990068 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:11.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.392+0000 7f0c09410640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0c040b2c10 0x7f0c04071850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.67:48994/0 (socket says 172.21.15.67:48994) 2024-03-24T00:39:11.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.392+0000 7f04b32c0640 1 -- 172.21.15.67:0/4221134434 shutdown_connections 2024-03-24T00:39:11.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.392+0000 7f0c0ac13640 1 -- 172.21.15.67:0/2536990068 --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c0406dd10 con 0x7f0c04159f30 2024-03-24T00:39:11.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.392+0000 7f0c0ac13640 1 -- 172.21.15.67:0/2536990068 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0406dff0 con 0x7f0c040b2c10 2024-03-24T00:39:11.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.391+0000 7fab62d9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fab640724f0 0x7fab640ccfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45474/0 (socket says 172.21.15.67:45474) 2024-03-24T00:39:11.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.391+0000 7fab62d9d640 1 -- 172.21.15.67:0/2135899448 learned_addr learned my addr 172.21.15.67:0/2135899448 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:11.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.392+0000 7fab6159a640 1 -- 172.21.15.67:0/2135899448 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3061196730 0 0) 0x7fab6406e490 con 0x7fab640724f0 2024-03-24T00:39:11.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.392+0000 7fab6159a640 1 -- 172.21.15.67:0/2135899448 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab4c003520 con 0x7fab640724f0 2024-03-24T00:39:11.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.392+0000 7fab6259c640 1 --2- 172.21.15.67:0/2135899448 >> v2:172.21.15.72:3300/0 conn(0x7fab64162370 0x7fab6406f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.392+0000 7fab6159a640 1 -- 172.21.15.67:0/2135899448 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 55902957 0 0) 0x7fab4c003520 con 0x7fab640724f0 2024-03-24T00:39:11.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.392+0000 7fab6159a640 1 -- 172.21.15.67:0/2135899448 >> v2:172.21.15.72:3300/0 conn(0x7fab64162370 msgr2=0x7fab6406f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:11.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.392+0000 7fab6159a640 1 --2- 172.21.15.67:0/2135899448 >> v2:172.21.15.72:3300/0 conn(0x7fab64162370 0x7fab6406f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.392+0000 7fab6159a640 1 -- 172.21.15.67:0/2135899448 >> v1:172.21.15.72:6789/0 conn(0x7fab6406df20 legacy=0x7fab64071de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:11.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.392+0000 7fab6159a640 1 -- 172.21.15.67:0/2135899448 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab6406ed30 con 0x7fab640724f0 2024-03-24T00:39:11.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.392+0000 7fab6159a640 1 -- 172.21.15.67:0/2135899448 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7fab58001650 con 0x7fab640724f0 2024-03-24T00:39:11.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.392+0000 7fab6159a640 1 -- 172.21.15.67:0/2135899448 >> v1:172.21.15.67:6789/0 conn(0x7fab640724f0 legacy=0x7fab640ccfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.392+0000 7fab6159a640 1 --2- 172.21.15.67:0/2135899448 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fab4c0039d0 0x7fab4c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.392+0000 7f9e62464640 1 Processor -- start 2024-03-24T00:39:11.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.392+0000 7f9e62464640 1 -- start start 2024-03-24T00:39:11.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.392+0000 7fab6159a640 1 --2- 172.21.15.67:0/2135899448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab4c004800 0x7fab4c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.392+0000 7fab6159a640 1 --2- 172.21.15.67:0/2135899448 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fab4c0051c0 0x7fab4c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.392+0000 7fab6159a640 1 -- 172.21.15.67:0/2135899448 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab6406ea50 con 0x7fab4c004800 2024-03-24T00:39:11.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.393+0000 7fab6159a640 1 -- 172.21.15.67:0/2135899448 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab4c00c1f0 con 0x7fab4c0051c0 2024-03-24T00:39:11.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.392+0000 7f39cf7fe640 1 -- 172.21.15.67:0/4159041165 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39bc034070 con 0x7f39c8054c50 2024-03-24T00:39:11.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.392+0000 7f39cf7fe640 1 -- 172.21.15.67:0/4159041165 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39bc030800 con 0x7f39c8054c50 2024-03-24T00:39:11.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.392+0000 7f39cf7fe640 1 -- 172.21.15.67:0/4159041165 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39bc034070 con 0x7f39c8054c50 2024-03-24T00:39:11.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.392+0000 7f0c08c0f640 1 -- 172.21.15.67:0/2536990068 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3823771970 0 0) 0x7f0c0406da30 con 0x7f0c040b3070 2024-03-24T00:39:11.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.392+0000 7f0c08c0f640 1 -- 172.21.15.67:0/2536990068 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bf0003520 con 0x7f0c040b3070 2024-03-24T00:39:11.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.393+0000 7fab6159a640 1 -- 172.21.15.67:0/2135899448 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab4c00c3f0 con 0x7fab4c0039d0 2024-03-24T00:39:11.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.392+0000 7f344e59c640 1 --2- 172.21.15.67:0/1999473146 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f34500ced80 0x7f34500cf1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.392+0000 7f344dd9b640 1 -- 172.21.15.67:0/1999473146 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1843629588 0 0) 0x7f34500cc770 con 0x7f345015a870 2024-03-24T00:39:11.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.392+0000 7f344dd9b640 1 -- 172.21.15.67:0/1999473146 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3438003520 con 0x7f345015a870 2024-03-24T00:39:11.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.393+0000 7fab61d9b640 1 --2- 172.21.15.67:0/2135899448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab4c004800 0x7fab4c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.393+0000 7f0c08c0f640 1 -- 172.21.15.67:0/2536990068 <== mon.1 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 308750083 0 0) 0x7f0bf0003520 con 0x7f0c040b3070 2024-03-24T00:39:11.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.393+0000 7f0c08c0f640 1 -- 172.21.15.67:0/2536990068 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0c040b2c10 msgr2=0x7f0c04071850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:11.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.393+0000 7f0c08c0f640 1 --2- 172.21.15.67:0/2536990068 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0c040b2c10 0x7f0c04071850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.393+0000 7f0c08c0f640 1 -- 172.21.15.67:0/2536990068 >> v2:172.21.15.72:3300/0 conn(0x7f0c04159f30 msgr2=0x7f0c040ccd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:11.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.393+0000 7f0c08c0f640 1 --2- 172.21.15.67:0/2536990068 >> v2:172.21.15.72:3300/0 conn(0x7f0c04159f30 0x7f0c040ccd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.393+0000 7f0c08c0f640 1 -- 172.21.15.67:0/2536990068 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c0406e2d0 con 0x7f0c040b3070 2024-03-24T00:39:11.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.393+0000 7fec7cb6b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec780aee10 0x7fec78148a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.393+0000 7f0c08c0f640 1 -- 172.21.15.67:0/2536990068 <== mon.1 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f0bfc0019d0 con 0x7f0c040b3070 2024-03-24T00:39:11.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.393+0000 7f0c08c0f640 1 -- 172.21.15.67:0/2536990068 >> v1:172.21.15.72:6789/0 conn(0x7f0c040b3070 legacy=0x7f0c0406ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.393+0000 7fec7cb6b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fec78159c70 0x7fec780aa3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.393+0000 7fec7cb6b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec78072030 0x7fec780724c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.393+0000 7fec7cb6b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec780b2a10 con 0x7fec780aee10 2024-03-24T00:39:11.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.393+0000 7fec77fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec78072030 0x7fec780724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.393+0000 7fec77fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec78072030 0x7fec780724c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:34072/0 (socket says 172.21.15.67:34072) 2024-03-24T00:39:11.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.393+0000 7fec77fff640 1 -- 172.21.15.67:0/3867314983 learned_addr learned my addr 172.21.15.67:0/3867314983 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:11.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.393+0000 7fec7cb6b640 1 -- 172.21.15.67:0/3867314983 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec780b2890 con 0x7fec78072030 2024-03-24T00:39:11.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.393+0000 7f344dd9b640 1 -- 172.21.15.67:0/1999473146 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1302883099 0 0) 0x7f3438003520 con 0x7f345015a870 2024-03-24T00:39:11.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.393+0000 7f344dd9b640 1 -- 172.21.15.67:0/1999473146 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f34500ced80 msgr2=0x7f34500cf1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:11.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.393+0000 7f344dd9b640 1 --2- 172.21.15.67:0/1999473146 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f34500ced80 0x7f34500cf1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.393+0000 7f344dd9b640 1 -- 172.21.15.67:0/1999473146 >> v1:172.21.15.72:6789/0 conn(0x7f345015a490 legacy=0x7f34500b2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.393+0000 7f0c08c0f640 1 --2- 172.21.15.67:0/2536990068 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0bf0003a10 0x7f0bf0003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.393+0000 7f0c08c0f640 1 --2- 172.21.15.67:0/2536990068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bf0004390 0x7f0bf0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.393+0000 7f0c08c0f640 1 --2- 172.21.15.67:0/2536990068 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0bf0004e60 0x7f0bf000ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.393+0000 7f0c08c0f640 1 -- 172.21.15.67:0/2536990068 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0406dd10 con 0x7f0bf0004390 2024-03-24T00:39:11.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.393+0000 7f0c08c0f640 1 -- 172.21.15.67:0/2536990068 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0406dff0 con 0x7f0bf0004e60 2024-03-24T00:39:11.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.393+0000 7f0c08c0f640 1 -- 172.21.15.67:0/2536990068 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf0010510 con 0x7f0bf0003a10 2024-03-24T00:39:11.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.393+0000 7f0c0a412640 1 --2- 172.21.15.67:0/2536990068 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0bf0004e60 0x7f0bf000ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.393+0000 7f344dd9b640 1 -- 172.21.15.67:0/1999473146 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f345006e880 con 0x7f345015a870 2024-03-24T00:39:11.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.392+0000 7f04b32c0640 1 -- 172.21.15.67:0/4221134434 wait complete. 2024-03-24T00:39:11.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.393+0000 7f344dd9b640 1 -- 172.21.15.67:0/1999473146 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f34480019f0 con 0x7f345015a870 2024-03-24T00:39:11.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.393+0000 7fec7cb6b640 1 -- 172.21.15.67:0/3867314983 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec7815afc0 con 0x7fec78159c70 2024-03-24T00:39:11.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.393+0000 7fec777fe640 1 --2- 172.21.15.67:0/3867314983 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec780aee10 0x7fec78148a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.393+0000 7f39d6e9a640 1 -- 172.21.15.67:0/4159041165 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f39c8054c50 msgr2=0x7f39c80df380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.393+0000 7f39d6e9a640 1 --2- 172.21.15.67:0/4159041165 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f39c8054c50 0x7f39c80df380 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f39bc0097a0 tx=0x7f39bc02b940 comp rx=0 tx=0).stop 2024-03-24T00:39:11.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.393+0000 7f3f1aec6640 1 Processor -- start 2024-03-24T00:39:11.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.393+0000 7f3f1aec6640 1 -- start start 2024-03-24T00:39:11.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7f344dd9b640 1 -- 172.21.15.67:0/1999473146 >> v1:172.21.15.67:6789/0 conn(0x7f345015a870 legacy=0x7f345015ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7fec76ffd640 1 --2- 172.21.15.67:0/3867314983 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fec78159c70 0x7fec780aa3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7fec77fff640 1 -- 172.21.15.67:0/3867314983 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fec78159c70 msgr2=0x7fec780aa3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7fec77fff640 1 --2- 172.21.15.67:0/3867314983 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fec78159c70 0x7fec780aa3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7fec77fff640 1 -- 172.21.15.67:0/3867314983 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec780aee10 msgr2=0x7fec78148a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7fec77fff640 1 --2- 172.21.15.67:0/3867314983 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec780aee10 0x7fec78148a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7fec77fff640 1 -- 172.21.15.67:0/3867314983 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec780a8800 con 0x7fec78072030 2024-03-24T00:39:11.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7f0c09c11640 1 --2- 172.21.15.67:0/2536990068 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0bf0003a10 0x7f0bf0003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7f344dd9b640 1 --2- 172.21.15.67:0/1999473146 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34380039e0 0x7f3438003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7f9e62464640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e54038110 0x7f9e540dc040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7f9e62464640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9e5404ffd0 0x7f9e540355f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7f9e62464640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9e540ead00 0x7f9e540dc7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7f9e62464640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e540547c0 con 0x7f9e54038110 2024-03-24T00:39:11.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7f9e62464640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e54038c90 con 0x7f9e540ead00 2024-03-24T00:39:11.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7f9e62464640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e54054940 con 0x7f9e5404ffd0 2024-03-24T00:39:11.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7fab62d9d640 1 --2- 172.21.15.67:0/2135899448 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fab4c0051c0 0x7fab4c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7fab6259c640 1 --2- 172.21.15.67:0/2135899448 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fab4c0039d0 0x7fab4c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7fab6259c640 1 -- 172.21.15.67:0/2135899448 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fab4c0051c0 msgr2=0x7fab4c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:11.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7fab6259c640 1 --2- 172.21.15.67:0/2135899448 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fab4c0051c0 0x7fab4c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7fab6259c640 1 -- 172.21.15.67:0/2135899448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab4c004800 msgr2=0x7fab4c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:11.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7fab6259c640 1 --2- 172.21.15.67:0/2135899448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab4c004800 0x7fab4c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7fab6259c640 1 -- 172.21.15.67:0/2135899448 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab6406ed30 con 0x7fab4c0039d0 2024-03-24T00:39:11.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7f39d6e9a640 1 -- 172.21.15.67:0/4159041165 shutdown_connections 2024-03-24T00:39:11.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7f39d6e9a640 1 --2- 172.21.15.67:0/4159041165 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f39c8054c50 0x7f39c80df380 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7f39d6e9a640 1 --2- 172.21.15.67:0/4159041165 >> v2:172.21.15.67:3300/0 conn(0x7f39c803acd0 0x7f39c8053460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7f344dd9b640 1 --2- 172.21.15.67:0/1999473146 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f34500ced80 0x7f3438004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7f39d6e9a640 1 -- 172.21.15.67:0/4159041165 >> 172.21.15.67:0/4159041165 conn(0x7f39c8055340 msgr2=0x7f39c8055750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:11.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7f344dd9b640 1 --2- 172.21.15.67:0/1999473146 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3438004d40 0x7f34380051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7f9e60c61640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9e5404ffd0 0x7f9e540355f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7f9e61c63640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9e540ead00 0x7f9e540dc7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7f9e61c63640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9e540ead00 0x7f9e540dc7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:34080/0 (socket says 172.21.15.67:34080) 2024-03-24T00:39:11.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7f9e61c63640 1 -- 172.21.15.67:0/1881287113 learned_addr learned my addr 172.21.15.67:0/1881287113 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:11.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7f344ed9d640 1 --2- 172.21.15.67:0/1999473146 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34380039e0 0x7f3438003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7f3f1aec6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3f0c048100 0x7f3f0c180400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7f3f1aec6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f0c04f930 0x7f3f0c048570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7f3f1aec6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f0c053e50 0x7f3f0c0e92e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7f3f1aec6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0c006490 con 0x7f3f0c04f930 2024-03-24T00:39:11.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7f3f1aec6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0c0502a0 con 0x7f3f0c053e50 2024-03-24T00:39:11.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7f3f1aec6640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0c006770 con 0x7f3f0c048100 2024-03-24T00:39:11.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7f3f19463640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f0c053e50 0x7f3f0c0e92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7f3f19463640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f0c053e50 0x7f3f0c0e92e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:34088/0 (socket says 172.21.15.67:34088) 2024-03-24T00:39:11.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7f3f19463640 1 -- 172.21.15.67:0/3619062385 learned_addr learned my addr 172.21.15.67:0/3619062385 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:11.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.395+0000 7f3f13fff640 1 --2- 172.21.15.67:0/3619062385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f0c04f930 0x7f3f0c048570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7f9e61462640 1 --2- 172.21.15.67:0/1881287113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e54038110 0x7f9e540dc040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.395+0000 7f344dd9b640 1 -- 172.21.15.67:0/1999473146 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f345006e5a0 con 0x7f34380039e0 2024-03-24T00:39:11.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7f04b32c0640 1 Processor -- start 2024-03-24T00:39:11.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7f04b32c0640 1 -- start start 2024-03-24T00:39:11.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.395+0000 7f344dd9b640 1 -- 172.21.15.67:0/1999473146 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f343800cd50 con 0x7f3438004d40 2024-03-24T00:39:11.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.395+0000 7f344dd9b640 1 -- 172.21.15.67:0/1999473146 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f343800bc90 con 0x7f34500ced80 2024-03-24T00:39:11.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.395+0000 7f3f18c62640 1 --2- 172.21.15.67:0/3619062385 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3f0c048100 0x7f3f0c180400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.395+0000 7f3f19463640 1 -- 172.21.15.67:0/3619062385 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3f0c048100 msgr2=0x7f3f0c180400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.395+0000 7f3f19463640 1 --2- 172.21.15.67:0/3619062385 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3f0c048100 0x7f3f0c180400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.395+0000 7f3f19463640 1 -- 172.21.15.67:0/3619062385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f0c04f930 msgr2=0x7f3f0c048570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.395+0000 7f3f19463640 1 --2- 172.21.15.67:0/3619062385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f0c04f930 0x7f3f0c048570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.395+0000 7f344e59c640 1 --2- 172.21.15.67:0/1999473146 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f34500ced80 0x7f3438004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.395+0000 7f3f19463640 1 -- 172.21.15.67:0/3619062385 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f0c0d8cd0 con 0x7f3f0c053e50 2024-03-24T00:39:11.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.395+0000 7f3f13fff640 1 --2- 172.21.15.67:0/3619062385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f0c04f930 0x7f3f0c048570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:11.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.395+0000 7f3f19463640 1 --2- 172.21.15.67:0/3619062385 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f0c053e50 0x7f3f0c0e92e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3f14068730 tx=0x7f3f14069920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.395+0000 7f9e61c63640 1 -- 172.21.15.67:0/1881287113 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9e5404ffd0 msgr2=0x7f9e540355f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.395+0000 7f9e61c63640 1 --2- 172.21.15.67:0/1881287113 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9e5404ffd0 0x7f9e540355f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.395+0000 7f9e61c63640 1 -- 172.21.15.67:0/1881287113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e54038110 msgr2=0x7f9e540dc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.395+0000 7f9e61c63640 1 --2- 172.21.15.67:0/1881287113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e54038110 0x7f9e540dc040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.395+0000 7f9e61c63640 1 -- 172.21.15.67:0/1881287113 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e540e8660 con 0x7f9e540ead00 2024-03-24T00:39:11.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.395+0000 7f04b32c0640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04ac0cec50 0x7f04ac0ca9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.395+0000 7f39d6e9a640 1 -- 172.21.15.67:0/4159041165 shutdown_connections 2024-03-24T00:39:11.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.395+0000 7f344e59c640 1 -- 172.21.15.67:0/1999473146 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3438004d40 msgr2=0x7f34380051c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T00:39:11.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.395+0000 7f344e59c640 1 --2- 172.21.15.67:0/1999473146 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3438004d40 0x7f34380051c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.395+0000 7f344e59c640 1 -- 172.21.15.67:0/1999473146 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34380039e0 msgr2=0x7f3438003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:11.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.395+0000 7f344e59c640 1 --2- 172.21.15.67:0/1999473146 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34380039e0 0x7f3438003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.395+0000 7f3f117fa640 1 -- 172.21.15.67:0/3619062385 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f140b4050 con 0x7f3f0c053e50 2024-03-24T00:39:11.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.395+0000 7fab62d9d640 1 -- 172.21.15.67:0/2135899448 reap_dead start 2024-03-24T00:39:11.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.395+0000 7f344e59c640 1 -- 172.21.15.67:0/1999473146 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3438003520 con 0x7f34500ced80 2024-03-24T00:39:11.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.395+0000 7f04b32c0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04ac0d0ac0 0x7f04ac0c8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.395+0000 7f39d6e9a640 1 -- 172.21.15.67:0/4159041165 wait complete. 2024-03-24T00:39:11.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7f0c09410640 1 --2- 172.21.15.67:0/2536990068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bf0004390 0x7f0bf0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7f0c0a412640 1 -- 172.21.15.67:0/2536990068 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0bf0003a10 msgr2=0x7f0bf0003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:11.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7f0c0a412640 1 --2- 172.21.15.67:0/2536990068 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0bf0003a10 0x7f0bf0003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7f0c0a412640 1 -- 172.21.15.67:0/2536990068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bf0004390 msgr2=0x7f0bf0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:11.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7f0c0a412640 1 --2- 172.21.15.67:0/2536990068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bf0004390 0x7f0bf0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.395+0000 7fab6259c640 1 --2- 172.21.15.67:0/2135899448 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fab4c0039d0 0x7fab4c003e10 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fab5c007360 tx=0x7fab4c005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42b0bd8bb81677d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7f0c0a412640 1 -- 172.21.15.67:0/2536990068 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c0406e2d0 con 0x7f0bf0004e60 2024-03-24T00:39:11.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7f0c09c11640 1 --2- 172.21.15.67:0/2536990068 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0bf0003a10 0x7f0bf0003e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:11.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.394+0000 7f0c09410640 1 --2- 172.21.15.67:0/2536990068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bf0004390 0x7f0bf0004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:39:11.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.395+0000 7f04b105c640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04ac0cec50 0x7f04ac0ca9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.395+0000 7f04b105c640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04ac0cec50 0x7f04ac0ca9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.67:49042/0 (socket says 172.21.15.67:49042) 2024-03-24T00:39:11.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.396+0000 7f04b32c0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04ac15ac20 0x7f04ac0c6200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.396+0000 7f04b32c0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac15ee00 con 0x7f04ac0d0ac0 2024-03-24T00:39:11.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.396+0000 7f04b32c0640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac0cbdc0 con 0x7f04ac15ac20 2024-03-24T00:39:11.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.396+0000 7f04b32c0640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac15ef80 con 0x7f04ac0cec50 2024-03-24T00:39:11.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.396+0000 7f04b105c640 1 -- 172.21.15.67:0/4217347271 learned_addr learned my addr 172.21.15.67:0/4217347271 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:11.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.396+0000 7fab6159a640 1 -- 172.21.15.67:0/2135899448 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab5c039070 con 0x7fab4c0039d0 2024-03-24T00:39:11.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.396+0000 7fab6159a640 1 -- 172.21.15.67:0/2135899448 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab5c02cd10 con 0x7fab4c0039d0 2024-03-24T00:39:11.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.396+0000 7fab6159a640 1 -- 172.21.15.67:0/2135899448 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab5c009be0 con 0x7fab4c0039d0 2024-03-24T00:39:11.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.396+0000 7f04b185d640 1 --2- 172.21.15.67:0/4217347271 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04ac15ac20 0x7f04ac0c6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.396+0000 7f04b085b640 1 --2- 172.21.15.67:0/4217347271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04ac0d0ac0 0x7f04ac0c8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.396+0000 7f04b105c640 1 -- 172.21.15.67:0/4217347271 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04ac15ac20 msgr2=0x7f04ac0c6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.396+0000 7f04b105c640 1 --2- 172.21.15.67:0/4217347271 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04ac15ac20 0x7f04ac0c6200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.396+0000 7f04b105c640 1 -- 172.21.15.67:0/4217347271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04ac0d0ac0 msgr2=0x7f04ac0c8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.396+0000 7f9e61c63640 1 --2- 172.21.15.67:0/1881287113 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9e540ead00 0x7f9e540dc7f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c00c390 tx=0x7f9e4c00c790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.396+0000 7fec77fff640 1 --2- 172.21.15.67:0/3867314983 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec78072030 0x7fec780724c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fec6c00d5a0 tx=0x7fec6c00da60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.396+0000 7f04b105c640 1 --2- 172.21.15.67:0/4217347271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04ac0d0ac0 0x7f04ac0c8930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.395+0000 7f3f1aec6640 1 -- 172.21.15.67:0/3619062385 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f0c0d8fb0 con 0x7f3f0c053e50 2024-03-24T00:39:11.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.395+0000 7f3f1aec6640 1 -- 172.21.15.67:0/3619062385 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f0c0d9570 con 0x7f3f0c053e50 2024-03-24T00:39:11.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.396+0000 7f3f117fa640 1 -- 172.21.15.67:0/3619062385 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f140b5040 con 0x7f3f0c053e50 2024-03-24T00:39:11.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.396+0000 7f3f117fa640 1 -- 172.21.15.67:0/3619062385 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f14062240 con 0x7f3f0c053e50 2024-03-24T00:39:11.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.396+0000 7f04b105c640 1 -- 172.21.15.67:0/4217347271 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04ac201b20 con 0x7f04ac0cec50 2024-03-24T00:39:11.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.396+0000 7fec5ffff640 1 -- 172.21.15.67:0/3867314983 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec6c0125f0 con 0x7fec78072030 2024-03-24T00:39:11.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.396+0000 7fec5ffff640 1 -- 172.21.15.67:0/3867314983 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec6c019070 con 0x7fec78072030 2024-03-24T00:39:11.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.396+0000 7f9e51ffb640 1 -- 172.21.15.67:0/1881287113 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e4c017070 con 0x7f9e540ead00 2024-03-24T00:39:11.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.396+0000 7fab6896f640 1 -- 172.21.15.67:0/2135899448 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fab4c0039d0 msgr2=0x7fab4c003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.396+0000 7fab6896f640 1 --2- 172.21.15.67:0/2135899448 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fab4c0039d0 0x7fab4c003e10 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fab5c007360 tx=0x7fab4c005830 comp rx=0 tx=0).stop 2024-03-24T00:39:11.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.396+0000 7fec5ffff640 1 -- 172.21.15.67:0/3867314983 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec6c0129d0 con 0x7fec78072030 2024-03-24T00:39:11.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.396+0000 7f04b085b640 1 --2- 172.21.15.67:0/4217347271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04ac0d0ac0 0x7f04ac0c8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:39:11.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.397+0000 7fab6896f640 1 -- 172.21.15.67:0/2135899448 shutdown_connections 2024-03-24T00:39:11.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.397+0000 7fab6896f640 1 --2- 172.21.15.67:0/2135899448 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fab4c0039d0 0x7fab4c003e10 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.397+0000 7fab6896f640 1 -- 172.21.15.67:0/2135899448 >> 172.21.15.67:0/2135899448 conn(0x7fab640b1ae0 msgr2=0x7fab640b1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:11.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.397+0000 7f344e59c640 1 --2- 172.21.15.67:0/1999473146 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f34500ced80 0x7f3438004800 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f344400ba30 tx=0x7f3438005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81eca082248b3c64 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.397+0000 7fab6896f640 1 -- 172.21.15.67:0/2135899448 shutdown_connections 2024-03-24T00:39:11.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.397+0000 7f9e51ffb640 1 -- 172.21.15.67:0/1881287113 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e4c0035f0 con 0x7f9e540ead00 2024-03-24T00:39:11.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.397+0000 7f344dd9b640 1 -- 172.21.15.67:0/1999473146 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3444035070 con 0x7f34500ced80 2024-03-24T00:39:11.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.397+0000 7f04b105c640 1 --2- 172.21.15.67:0/4217347271 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04ac0cec50 0x7f04ac0ca9c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f04a000bb30 tx=0x7f04a00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.397+0000 7f344dd9b640 1 -- 172.21.15.67:0/1999473146 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3444030660 con 0x7f34500ced80 2024-03-24T00:39:11.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.397+0000 7f049dffb640 1 -- 172.21.15.67:0/4217347271 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a003d070 con 0x7f04ac0cec50 2024-03-24T00:39:11.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.397+0000 7fab6896f640 1 -- 172.21.15.67:0/2135899448 wait complete. 2024-03-24T00:39:11.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.397+0000 7f04b32c0640 1 -- 172.21.15.67:0/4217347271 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04ac0c7e50 con 0x7f04ac0cec50 2024-03-24T00:39:11.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.398+0000 7f39d6e9a640 1 Processor -- start 2024-03-24T00:39:11.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.398+0000 7f049dffb640 1 -- 172.21.15.67:0/4217347271 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04a0012450 con 0x7f04ac0cec50 2024-03-24T00:39:11.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.398+0000 7f39d6e9a640 1 -- start start 2024-03-24T00:39:11.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.398+0000 7f049dffb640 1 -- 172.21.15.67:0/4217347271 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a0010a00 con 0x7f04ac0cec50 2024-03-24T00:39:11.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.398+0000 7f3f117fa640 1 -- 172.21.15.67:0/3619062385 <== mon.1 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3f140c2020 con 0x7f3f0c053e50 2024-03-24T00:39:11.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.398+0000 7f04b32c0640 1 -- 172.21.15.67:0/4217347271 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04ac0c8e70 con 0x7f04ac0cec50 2024-03-24T00:39:11.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.397+0000 7f9e51ffb640 1 -- 172.21.15.67:0/1881287113 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e4c0039e0 con 0x7f9e540ead00 2024-03-24T00:39:11.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.397+0000 7f9e62464640 1 -- 172.21.15.67:0/1881287113 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e54034550 con 0x7f9e540ead00 2024-03-24T00:39:11.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.397+0000 7f9e62464640 1 -- 172.21.15.67:0/1881287113 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e54034b10 con 0x7f9e540ead00 2024-03-24T00:39:11.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.398+0000 7f9e3f7fe640 1 -- 172.21.15.67:0/1881287113 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9e38001030 con 0x7f9e540ead00 2024-03-24T00:39:11.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.398+0000 7f0c0a412640 1 --2- 172.21.15.67:0/2536990068 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0bf0004e60 0x7f0bf000ffd0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc00c570 tx=0x7f0bf0005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ef9f188badcd4c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.398+0000 7f344dd9b640 1 -- 172.21.15.67:0/1999473146 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f344400b600 con 0x7f34500ced80 2024-03-24T00:39:11.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.398+0000 7f345520b640 1 -- 172.21.15.67:0/1999473146 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f34500ced80 msgr2=0x7f3438004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.398+0000 7f345520b640 1 --2- 172.21.15.67:0/1999473146 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f34500ced80 0x7f3438004800 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f344400ba30 tx=0x7f3438005830 comp rx=0 tx=0).stop 2024-03-24T00:39:11.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.398+0000 7fec7cb6b640 1 -- 172.21.15.67:0/3867314983 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec780a8ae0 con 0x7fec78072030 2024-03-24T00:39:11.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.398+0000 7f344f59e640 1 -- 172.21.15.67:0/1999473146 reap_dead start 2024-03-24T00:39:11.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.398+0000 7f345520b640 1 -- 172.21.15.67:0/1999473146 shutdown_connections 2024-03-24T00:39:11.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.398+0000 7f345520b640 1 -- 172.21.15.67:0/1999473146 >> 172.21.15.67:0/1999473146 conn(0x7f345006b670 msgr2=0x7f34500b17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:11.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.398+0000 7fec7cb6b640 1 -- 172.21.15.67:0/3867314983 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec780a90a0 con 0x7fec78072030 2024-03-24T00:39:11.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.399+0000 7f3efeffd640 1 -- 172.21.15.67:0/3619062385 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3eec001030 con 0x7f3f0c053e50 2024-03-24T00:39:11.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.399+0000 7f3f117fa640 1 --2- 172.21.15.67:0/3619062385 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3ef806bbd0 0x7f3ef806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.399+0000 7fab6896f640 1 Processor -- start 2024-03-24T00:39:11.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.399+0000 7f345520b640 1 -- 172.21.15.67:0/1999473146 shutdown_connections 2024-03-24T00:39:11.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.399+0000 7fab6896f640 1 -- start start 2024-03-24T00:39:11.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.399+0000 7f345520b640 1 -- 172.21.15.67:0/1999473146 wait complete. 2024-03-24T00:39:11.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.399+0000 7f3f18c62640 1 --2- 172.21.15.67:0/3619062385 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3ef806bbd0 0x7f3ef806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.399+0000 7f04b32c0640 1 -- 172.21.15.67:0/4217347271 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f04ac04f400 con 0x7f04ac0cec50 2024-03-24T00:39:11.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.399+0000 7f3f117fa640 1 -- 172.21.15.67:0/3619062385 <== mon.1 v2:172.21.15.72:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f3f14127e70 con 0x7f3f0c053e50 2024-03-24T00:39:11.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.399+0000 7f3f18c62640 1 --2- 172.21.15.67:0/3619062385 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3ef806bbd0 0x7f3ef806e090 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3f08000f60 tx=0x7f3f08009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.400+0000 7f3f117fa640 1 --2- 172.21.15.67:0/3619062385 >> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] conn(0x7f3ef80752d0 0x7f3ef8077710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.400+0000 7f3f117fa640 1 -- 172.21.15.67:0/3619062385 --> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3f140af440 con 0x7f3ef80752d0 2024-03-24T00:39:11.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.400+0000 7f3f117fa640 1 -- 172.21.15.67:0/3619062385 <== mon.1 v2:172.21.15.72:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3f140f1fe0 con 0x7f3f0c053e50 2024-03-24T00:39:11.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.399+0000 7f39d6e9a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39c803acd0 0x7f39c80070d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.400+0000 7f39d4c36640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39c803acd0 0x7f39c80070d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.400+0000 7f39d4c36640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39c803acd0 0x7f39c80070d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60916/0 (socket says 172.21.15.67:60916) 2024-03-24T00:39:11.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.400+0000 7f39d4c36640 1 -- 172.21.15.67:0/3177867923 learned_addr learned my addr 172.21.15.67:0/3177867923 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:11.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.400+0000 7f39d6e9a640 1 --2- 172.21.15.67:0/3177867923 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39c8050010 0x7f39c8039e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.399+0000 7f0c08c0f640 1 -- 172.21.15.67:0/2536990068 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bfc039070 con 0x7f0bf0004e60 2024-03-24T00:39:11.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.399+0000 7f0c08c0f640 1 -- 172.21.15.67:0/2536990068 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bfc02ce50 con 0x7f0bf0004e60 2024-03-24T00:39:11.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.400+0000 7f39d6e9a640 1 --2- 172.21.15.67:0/3177867923 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f39c8054c50 0x7f39c80cdc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.400+0000 7f39cffff640 1 --2- 172.21.15.67:0/3177867923 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39c8050010 0x7f39c8039e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.400+0000 7fab6896f640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fab4c0039d0 0x7fab640cf170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.400+0000 7f39d5437640 1 --2- 172.21.15.67:0/3177867923 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f39c8054c50 0x7f39c80cdc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.400+0000 7f9e51ffb640 1 -- 172.21.15.67:0/1881287113 <== mon.1 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9e4c01a070 con 0x7f9e540ead00 2024-03-24T00:39:11.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.400+0000 7f39d6e9a640 1 -- 172.21.15.67:0/3177867923 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39c80df8c0 con 0x7f39c803acd0 2024-03-24T00:39:11.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.400+0000 7f39d6e9a640 1 -- 172.21.15.67:0/3177867923 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39c80550b0 con 0x7f39c8050010 2024-03-24T00:39:11.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.400+0000 7fab6896f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fab640c5f20 0x7fab64151eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.400+0000 7fab6896f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab640c64c0 0x7fab640c6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.400+0000 7fab6896f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab640729f0 con 0x7fab640c64c0 2024-03-24T00:39:11.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.400+0000 7f39d6e9a640 1 -- 172.21.15.67:0/3177867923 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39c80e70e0 con 0x7f39c8054c50 2024-03-24T00:39:11.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.400+0000 7fab6896f640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab64162750 con 0x7fab640c5f20 2024-03-24T00:39:11.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.400+0000 7fab6896f640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab64072b70 con 0x7fab4c0039d0 2024-03-24T00:39:11.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.400+0000 7f0c08c0f640 1 -- 172.21.15.67:0/2536990068 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bfc039070 con 0x7f0bf0004e60 2024-03-24T00:39:11.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.400+0000 7f0c0ac13640 1 -- 172.21.15.67:0/2536990068 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0bf0004e60 msgr2=0x7f0bf000ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.400+0000 7f0c0ac13640 1 --2- 172.21.15.67:0/2536990068 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0bf0004e60 0x7f0bf000ffd0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc00c570 tx=0x7f0bf0005830 comp rx=0 tx=0).stop 2024-03-24T00:39:11.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.401+0000 7fab62d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab640c64c0 0x7fab640c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.401+0000 7fab62d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab640c64c0 0x7fab640c6950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60932/0 (socket says 172.21.15.67:60932) 2024-03-24T00:39:11.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.401+0000 7fab62d9d640 1 -- 172.21.15.67:0/2824837902 learned_addr learned my addr 172.21.15.67:0/2824837902 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:11.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.400+0000 7f0c0a412640 1 -- 172.21.15.67:0/2536990068 reap_dead start 2024-03-24T00:39:11.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.401+0000 7f0c0ac13640 1 -- 172.21.15.67:0/2536990068 shutdown_connections 2024-03-24T00:39:11.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.401+0000 7f0c0ac13640 1 -- 172.21.15.67:0/2536990068 >> 172.21.15.67:0/2536990068 conn(0x7f0c040cf7b0 msgr2=0x7f0c040cfbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:11.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.401+0000 7f9e51ffb640 1 --2- 172.21.15.67:0/1881287113 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9e4006bcd0 0x7f9e4006e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.400+0000 7f3f13fff640 1 --2- 172.21.15.67:0/3619062385 >> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] conn(0x7f3ef80752d0 0x7f3ef8077710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.401+0000 7f39cffff640 1 -- 172.21.15.67:0/3177867923 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f39c8054c50 msgr2=0x7f39c80cdc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.401+0000 7f39cffff640 1 --2- 172.21.15.67:0/3177867923 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f39c8054c50 0x7f39c80cdc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.401+0000 7f39cffff640 1 -- 172.21.15.67:0/3177867923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39c803acd0 msgr2=0x7f39c80070d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.401+0000 7f39cffff640 1 --2- 172.21.15.67:0/3177867923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39c803acd0 0x7f39c80070d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.401+0000 7f39cffff640 1 -- 172.21.15.67:0/3177867923 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39c8049e40 con 0x7f39c8050010 2024-03-24T00:39:11.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.401+0000 7fab61d9b640 1 --2- 172.21.15.67:0/2824837902 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fab640c5f20 0x7fab64151eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.401+0000 7f9e61462640 1 --2- 172.21.15.67:0/1881287113 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9e4006bcd0 0x7f9e4006e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.401+0000 7f39d4c36640 1 --2- 172.21.15.67:0/3177867923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39c803acd0 0x7f39c80070d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:11.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.401+0000 7f39d5437640 1 --2- 172.21.15.67:0/3177867923 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f39c8054c50 0x7f39c80cdc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:11.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.401+0000 7f0c0ac13640 1 -- 172.21.15.67:0/2536990068 shutdown_connections 2024-03-24T00:39:11.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.401+0000 7f0c0ac13640 1 -- 172.21.15.67:0/2536990068 wait complete. 2024-03-24T00:39:11.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.401+0000 7fab6259c640 1 --2- 172.21.15.67:0/2824837902 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fab4c0039d0 0x7fab640cf170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.401+0000 7fab62d9d640 1 -- 172.21.15.67:0/2824837902 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fab4c0039d0 msgr2=0x7fab640cf170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.401+0000 7fab62d9d640 1 --2- 172.21.15.67:0/2824837902 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fab4c0039d0 0x7fab640cf170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.401+0000 7fab62d9d640 1 -- 172.21.15.67:0/2824837902 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fab640c5f20 msgr2=0x7fab64151eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.401+0000 7fab62d9d640 1 --2- 172.21.15.67:0/2824837902 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fab640c5f20 0x7fab64151eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.401+0000 7fab62d9d640 1 -- 172.21.15.67:0/2824837902 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab640cdb60 con 0x7fab640c64c0 2024-03-24T00:39:11.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.401+0000 7f345520b640 1 Processor -- start 2024-03-24T00:39:11.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.401+0000 7f9e51ffb640 1 -- 172.21.15.67:0/1881287113 <== mon.1 v2:172.21.15.72:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f9e4c08a740 con 0x7f9e540ead00 2024-03-24T00:39:11.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.401+0000 7f39cffff640 1 --2- 172.21.15.67:0/3177867923 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39c8050010 0x7f39c8039e20 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f39bc033000 tx=0x7f39bc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.401+0000 7f3f13fff640 1 --2- 172.21.15.67:0/3619062385 >> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] conn(0x7f3ef80752d0 0x7f3ef8077710 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.401+0000 7f345520b640 1 -- start start 2024-03-24T00:39:11.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.401+0000 7fec5ffff640 1 -- 172.21.15.67:0/3867314983 <== mon.1 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fec6c012b70 con 0x7fec78072030 2024-03-24T00:39:11.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.401+0000 7fab6259c640 1 --2- 172.21.15.67:0/2824837902 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fab4c0039d0 0x7fab640cf170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:39:11.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.402+0000 7fab62d9d640 1 --2- 172.21.15.67:0/2824837902 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab640c64c0 0x7fab640c6950 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fab58001eb0 tx=0x7fab58007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.401+0000 7f39cd7fa640 1 -- 172.21.15.67:0/3177867923 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39bc034070 con 0x7f39c8050010 2024-03-24T00:39:11.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.402+0000 7f39cd7fa640 1 -- 172.21.15.67:0/3177867923 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39bc039d10 con 0x7f39c8050010 2024-03-24T00:39:11.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.402+0000 7f049dffb640 1 -- 172.21.15.67:0/4217347271 <== mon.2 v2:172.21.15.72:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f04a0010ba0 con 0x7f04ac0cec50 2024-03-24T00:39:11.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.402+0000 7fec7cb6b640 1 -- 172.21.15.67:0/3867314983 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fec780751c0 con 0x7fec78072030 2024-03-24T00:39:11.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.402+0000 7fab52ffd640 1 -- 172.21.15.67:0/2824837902 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab5801a070 con 0x7fab640c64c0 2024-03-24T00:39:11.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.402+0000 7f39cd7fa640 1 -- 172.21.15.67:0/3177867923 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39bc0423f0 con 0x7f39c8050010 2024-03-24T00:39:11.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.402+0000 7f39d6e9a640 1 -- 172.21.15.67:0/3177867923 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39c804a120 con 0x7f39c8050010 2024-03-24T00:39:11.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.402+0000 7f39d6e9a640 1 -- 172.21.15.67:0/3177867923 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39c804a6e0 con 0x7f39c8050010 2024-03-24T00:39:11.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.402+0000 7fab6896f640 1 -- 172.21.15.67:0/2824837902 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab640cde40 con 0x7fab640c64c0 2024-03-24T00:39:11.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.402+0000 7fec5ffff640 1 --2- 172.21.15.67:0/3867314983 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fec6406bca0 0x7fec6406e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.402+0000 7fab6896f640 1 -- 172.21.15.67:0/2824837902 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab640ce400 con 0x7fab640c64c0 2024-03-24T00:39:11.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.402+0000 7fec5ffff640 1 -- 172.21.15.67:0/3867314983 <== mon.1 v2:172.21.15.72:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fec6c08b1b0 con 0x7fec78072030 2024-03-24T00:39:11.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.402+0000 7fec5ffff640 1 --2- 172.21.15.67:0/3867314983 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x7fec640753a0 0x7fec640777e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.402+0000 7f049dffb640 1 --2- 172.21.15.67:0/4217347271 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f049006bcd0 0x7f049006e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.402+0000 7fec5ffff640 1 -- 172.21.15.67:0/3867314983 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fec64077eb0 con 0x7fec640753a0 2024-03-24T00:39:11.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.402+0000 7fec777fe640 1 --2- 172.21.15.67:0/3867314983 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fec6406bca0 0x7fec6406e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.402+0000 7fec5ffff640 1 -- 172.21.15.67:0/3867314983 <== mon.1 v2:172.21.15.72:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fec6c092050 con 0x7fec78072030 2024-03-24T00:39:11.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.403+0000 7fab52ffd640 1 -- 172.21.15.67:0/2824837902 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab5800f8b0 con 0x7fab640c64c0 2024-03-24T00:39:11.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.403+0000 7f049dffb640 1 -- 172.21.15.67:0/4217347271 <== mon.2 v2:172.21.15.72:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f04a00add60 con 0x7f04ac0cec50 2024-03-24T00:39:11.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.403+0000 7f049dffb640 1 --2- 172.21.15.67:0/4217347271 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x7f04900753d0 0x7f0490077810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.403+0000 7f345520b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34500b0b30 0x7f34500cb500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.403+0000 7f345520b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f34500b0f10 0x7f34501f8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.403+0000 7f04b185d640 1 --2- 172.21.15.67:0/4217347271 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x7f04900753d0 0x7f0490077810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.404+0000 7f3f117fa640 1 -- 172.21.15.67:0/3619062385 <== osd.2 v2:172.21.15.67:6817/1268432561 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3f140af440 con 0x7f3ef80752d0 2024-03-24T00:39:11.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.403+0000 7fab52ffd640 1 -- 172.21.15.67:0/2824837902 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab5800fca0 con 0x7fab640c64c0 2024-03-24T00:39:11.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.403+0000 7fab6896f640 1 -- 172.21.15.67:0/2824837902 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fab6404f400 con 0x7fab640c64c0 2024-03-24T00:39:11.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.403+0000 7f345520b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34501f93c0 0x7f34501f9850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.403+0000 7f345520b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3450072c90 con 0x7f34501f93c0 2024-03-24T00:39:11.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.403+0000 7f345520b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3450072b10 con 0x7f34500b0b30 2024-03-24T00:39:11.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.403+0000 7f345520b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34500cf1e0 con 0x7f34500b0f10 2024-03-24T00:39:11.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.403+0000 7f344f59e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34501f93c0 0x7f34501f9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.403+0000 7f344f59e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34501f93c0 0x7f34501f9850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60942/0 (socket says 172.21.15.67:60942) 2024-03-24T00:39:11.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.403+0000 7f344f59e640 1 -- 172.21.15.67:0/448189333 learned_addr learned my addr 172.21.15.67:0/448189333 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:11.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.403+0000 7f0c0ac13640 1 Processor -- start 2024-03-24T00:39:11.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.404+0000 7f0c0ac13640 1 -- start start 2024-03-24T00:39:11.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.403+0000 7f39d6e9a640 1 -- 172.21.15.67:0/3177867923 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f39c8005700 con 0x7f39c8050010 2024-03-24T00:39:11.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.404+0000 7f04b185d640 1 --2- 172.21.15.67:0/4217347271 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x7f04900753d0 0x7f0490077810 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.404+0000 7f049dffb640 1 -- 172.21.15.67:0/4217347271 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f04a000d140 con 0x7f04900753d0 2024-03-24T00:39:11.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.404+0000 7f9e51ffb640 1 --2- 172.21.15.67:0/1881287113 >> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] conn(0x7f9e40075450 0x7f9e40077890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.404+0000 7f9e61462640 1 --2- 172.21.15.67:0/1881287113 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9e4006bcd0 0x7f9e4006e190 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9e5400b0a0 tx=0x7f9e5c0755a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.404+0000 7f344f59e640 1 -- 172.21.15.67:0/448189333 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f34500b0f10 msgr2=0x7f34501f8270 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-24T00:39:11.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.404+0000 7f344f59e640 1 --2- 172.21.15.67:0/448189333 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f34500b0f10 0x7f34501f8270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.404+0000 7f344f59e640 1 -- 172.21.15.67:0/448189333 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34500b0b30 msgr2=0x7f34500cb500 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-24T00:39:11.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.404+0000 7f344f59e640 1 --2- 172.21.15.67:0/448189333 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34500b0b30 0x7f34500cb500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.404+0000 7f344f59e640 1 -- 172.21.15.67:0/448189333 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34501f8900 con 0x7f34501f93c0 2024-03-24T00:39:11.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.404+0000 7f9e51ffb640 1 -- 172.21.15.67:0/1881287113 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9e40077f60 con 0x7f9e40075450 2024-03-24T00:39:11.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.404+0000 7f9e51ffb640 1 -- 172.21.15.67:0/1881287113 <== mon.1 v2:172.21.15.72:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9e4c08ab30 con 0x7f9e540ead00 2024-03-24T00:39:11.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.404+0000 7f9e60c61640 1 --2- 172.21.15.67:0/1881287113 >> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] conn(0x7f9e40075450 0x7f9e40077890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.404+0000 7f344f59e640 1 --2- 172.21.15.67:0/448189333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34501f93c0 0x7f34501f9850 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f34500af6b0 tx=0x7f3448008c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.404+0000 7fec76ffd640 1 --2- 172.21.15.67:0/3867314983 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x7fec640753a0 0x7fec640777e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.405+0000 7fab52ffd640 1 -- 172.21.15.67:0/2824837902 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fab5801d070 con 0x7fab640c64c0 2024-03-24T00:39:11.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.405+0000 7f049dffb640 1 -- 172.21.15.67:0/4217347271 <== osd.0 v2:172.21.15.67:6808/2628130519 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f04a000d140 con 0x7f04900753d0 2024-03-24T00:39:11.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.405+0000 7fec76ffd640 1 --2- 172.21.15.67:0/3867314983 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x7fec640753a0 0x7fec640777e0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.405+0000 7f9e60c61640 1 --2- 172.21.15.67:0/1881287113 >> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] conn(0x7f9e40075450 0x7f9e40077890 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.405+0000 7fab52ffd640 1 --2- 172.21.15.67:0/2824837902 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fab4406bcd0 0x7fab4406e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.405+0000 7fab52ffd640 1 -- 172.21.15.67:0/2824837902 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fab5808a620 con 0x7fab640c64c0 2024-03-24T00:39:11.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.405+0000 7fab52ffd640 1 --2- 172.21.15.67:0/2824837902 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x7fab44075450 0x7fab44077890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.406+0000 7fab52ffd640 1 -- 172.21.15.67:0/2824837902 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fab44077f60 con 0x7fab44075450 2024-03-24T00:39:11.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.406+0000 7fab52ffd640 1 -- 172.21.15.67:0/2824837902 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fab5808aa10 con 0x7fab640c64c0 2024-03-24T00:39:11.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.405+0000 7f39cd7fa640 1 -- 172.21.15.67:0/3177867923 <== mon.1 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f39bc04a060 con 0x7f39c8050010 2024-03-24T00:39:11.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.405+0000 7f34377fe640 1 -- 172.21.15.67:0/448189333 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f344800d690 con 0x7f34501f93c0 2024-03-24T00:39:11.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.405+0000 7f34377fe640 1 -- 172.21.15.67:0/448189333 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f344800d870 con 0x7f34501f93c0 2024-03-24T00:39:11.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.405+0000 7f345520b640 1 -- 172.21.15.67:0/448189333 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34501f8be0 con 0x7f34501f93c0 2024-03-24T00:39:11.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.405+0000 7f345520b640 1 -- 172.21.15.67:0/448189333 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34500c4b70 con 0x7f34501f93c0 2024-03-24T00:39:11.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.405+0000 7f39cd7fa640 1 --2- 172.21.15.67:0/3177867923 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f39b406bcd0 0x7f39b406e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.406+0000 7f39cd7fa640 1 -- 172.21.15.67:0/3177867923 <== mon.1 v2:172.21.15.72:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f39bc068050 con 0x7f39c8050010 2024-03-24T00:39:11.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.406+0000 7f39cd7fa640 1 --2- 172.21.15.67:0/3177867923 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x7f39b4075440 0x7f39b4077880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.406+0000 7f39cd7fa640 1 -- 172.21.15.67:0/3177867923 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f39b4077f50 con 0x7f39b4075440 2024-03-24T00:39:11.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.406+0000 7f39cd7fa640 1 -- 172.21.15.67:0/3177867923 <== mon.1 v2:172.21.15.72:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f39bc0ace50 con 0x7f39c8050010 2024-03-24T00:39:11.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.406+0000 7f34377fe640 1 -- 172.21.15.67:0/448189333 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f344800dc60 con 0x7f34501f93c0 2024-03-24T00:39:11.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.406+0000 7f9e51ffb640 1 -- 172.21.15.67:0/1881287113 <== osd.3 v2:172.21.15.67:6816/3303489481 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9e40077f60 con 0x7f9e40075450 2024-03-24T00:39:11.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.406+0000 7f04b085b640 1 --2- 172.21.15.67:0/4217347271 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f049006bcd0 0x7f049006e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.406+0000 7f049dffb640 1 -- 172.21.15.67:0/4217347271 <== mon.2 v2:172.21.15.72:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f04a0077ec0 con 0x7f04ac0cec50 2024-03-24T00:39:11.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.406+0000 7fec777fe640 1 --2- 172.21.15.67:0/3867314983 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fec6406bca0 0x7fec6406e160 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fec68036000 tx=0x7fec68030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.406+0000 7f39d5437640 1 --2- 172.21.15.67:0/3177867923 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x7f39b4075440 0x7f39b4077880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.407+0000 7f345520b640 1 -- 172.21.15.67:0/448189333 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3430001030 con 0x7f34501f93c0 2024-03-24T00:39:11.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.407+0000 7fec5ffff640 1 -- 172.21.15.67:0/3867314983 <== osd.1 v2:172.21.15.67:6800/4062510404 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fec64077eb0 con 0x7fec640753a0 2024-03-24T00:39:11.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.407+0000 7f39d4c36640 1 --2- 172.21.15.67:0/3177867923 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f39b406bcd0 0x7f39b406e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.406+0000 7fab6259c640 1 --2- 172.21.15.67:0/2824837902 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fab4406bcd0 0x7fab4406e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.406+0000 7f0c0ac13640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0c04147110 0x7f0c0415b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.406+0000 7f0c0ac13640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c04153c60 0x7f0c041540e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.406+0000 7f0c0ac13640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0c04144c70 0x7f0c041450f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.406+0000 7f0c0ac13640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c040b40e0 con 0x7f0c04153c60 2024-03-24T00:39:11.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.406+0000 7f0c0ac13640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c040cd280 con 0x7f0c04147110 2024-03-24T00:39:11.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.406+0000 7f0c0ac13640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c040b4260 con 0x7f0c04144c70 2024-03-24T00:39:11.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.407+0000 7f0c0a412640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0c04144c70 0x7f0c041450f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.407+0000 7f0c09c11640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0c04147110 0x7f0c0415b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.407+0000 7f0c09c11640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0c04147110 0x7f0c0415b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:34134/0 (socket says 172.21.15.67:34134) 2024-03-24T00:39:11.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.407+0000 7f0c09c11640 1 -- 172.21.15.67:0/1485836817 learned_addr learned my addr 172.21.15.67:0/1485836817 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:11.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.407+0000 7f0c09410640 1 --2- 172.21.15.67:0/1485836817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c04153c60 0x7f0c041540e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.407+0000 7f39d4c36640 1 --2- 172.21.15.67:0/3177867923 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f39b406bcd0 0x7f39b406e190 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f39d004f6d0 tx=0x7f39d006aef0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.407+0000 7f39d5437640 1 --2- 172.21.15.67:0/3177867923 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x7f39b4075440 0x7f39b4077880 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.408+0000 7f0c09c11640 1 -- 172.21.15.67:0/1485836817 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0c04144c70 msgr2=0x7f0c041450f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.408+0000 7f0c09c11640 1 --2- 172.21.15.67:0/1485836817 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0c04144c70 0x7f0c041450f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.408+0000 7f0c09c11640 1 -- 172.21.15.67:0/1485836817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c04153c60 msgr2=0x7f0c041540e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.408+0000 7f0c09c11640 1 --2- 172.21.15.67:0/1485836817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c04153c60 0x7f0c041540e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.408+0000 7f0c09c11640 1 -- 172.21.15.67:0/1485836817 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c0415acf0 con 0x7f0c04147110 2024-03-24T00:39:11.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.408+0000 7f0c09410640 1 --2- 172.21.15.67:0/1485836817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c04153c60 0x7f0c041540e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:39:11.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.408+0000 7f04b085b640 1 --2- 172.21.15.67:0/4217347271 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f049006bcd0 0x7f049006e190 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f04ac0cebd0 tx=0x7f04a8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.408+0000 7fab6259c640 1 --2- 172.21.15.67:0/2824837902 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fab4406bcd0 0x7fab4406e190 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fab5c036000 tx=0x7fab5c030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.408+0000 7f0c09c11640 1 --2- 172.21.15.67:0/1485836817 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0c04147110 0x7f0c0415b770 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0bf4010960 tx=0x7f0bf4010e20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.408+0000 7f34377fe640 1 -- 172.21.15.67:0/448189333 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f34480054b0 con 0x7f34501f93c0 2024-03-24T00:39:11.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.408+0000 7fab61d9b640 1 --2- 172.21.15.67:0/2824837902 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x7fab44075450 0x7fab44077890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.408+0000 7f0bfa7fc640 1 -- 172.21.15.67:0/1485836817 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf401a070 con 0x7f0c04147110 2024-03-24T00:39:11.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.409+0000 7f39cd7fa640 1 -- 172.21.15.67:0/3177867923 <== osd.5 v2:172.21.15.72:6800/288486206 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f39b4077f50 con 0x7f39b4075440 2024-03-24T00:39:11.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.409+0000 7f34377fe640 1 --2- 172.21.15.67:0/448189333 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f343c06bc20 0x7f343c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.408+0000 7f0bfa7fc640 1 -- 172.21.15.67:0/1485836817 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bf400ea80 con 0x7f0c04147110 2024-03-24T00:39:11.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.408+0000 7f0bfa7fc640 1 -- 172.21.15.67:0/1485836817 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf400ee30 con 0x7f0c04147110 2024-03-24T00:39:11.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.408+0000 7f0c0ac13640 1 -- 172.21.15.67:0/1485836817 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c0415ca00 con 0x7f0c04147110 2024-03-24T00:39:11.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.409+0000 7f0c0ac13640 1 -- 172.21.15.67:0/1485836817 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c0415cfc0 con 0x7f0c04147110 2024-03-24T00:39:11.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.409+0000 7f344ed9d640 1 --2- 172.21.15.67:0/448189333 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f343c06bc20 0x7f343c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.409+0000 7f34377fe640 1 -- 172.21.15.67:0/448189333 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f344808b2d0 con 0x7f34501f93c0 2024-03-24T00:39:11.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.409+0000 7f344ed9d640 1 --2- 172.21.15.67:0/448189333 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f343c06bc20 0x7f343c06e0e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3440001750 tx=0x7f3440009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.409+0000 7f34377fe640 1 --2- 172.21.15.67:0/448189333 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x7f343c0753a0 0x7f343c0777e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.410+0000 7f34377fe640 1 -- 172.21.15.67:0/448189333 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f344800ae50 con 0x7f343c0753a0 2024-03-24T00:39:11.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.410+0000 7f344e59c640 1 --2- 172.21.15.67:0/448189333 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x7f343c0753a0 0x7f343c0777e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.410+0000 7f34377fe640 1 -- 172.21.15.67:0/448189333 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f344808b680 con 0x7f34501f93c0 2024-03-24T00:39:11.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.410+0000 7fab61d9b640 1 --2- 172.21.15.67:0/2824837902 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x7fab44075450 0x7fab44077890 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.410+0000 7f0c0ac13640 1 -- 172.21.15.67:0/1485836817 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0c040cf550 con 0x7f0c04147110 2024-03-24T00:39:11.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.410+0000 7f0bfa7fc640 1 -- 172.21.15.67:0/1485836817 <== mon.1 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0bf4012040 con 0x7f0c04147110 2024-03-24T00:39:11.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.410+0000 7f344e59c640 1 --2- 172.21.15.67:0/448189333 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x7f343c0753a0 0x7f343c0777e0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.411+0000 7fab52ffd640 1 -- 172.21.15.67:0/2824837902 <== osd.4 v2:172.21.15.72:6808/477220793 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fab44077f60 con 0x7fab44075450 2024-03-24T00:39:11.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.411+0000 7f0bfa7fc640 1 --2- 172.21.15.67:0/1485836817 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0be806bc50 0x7f0be806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.411+0000 7f0bfa7fc640 1 -- 172.21.15.67:0/1485836817 <== mon.1 v2:172.21.15.72:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f0bf408ad40 con 0x7f0c04147110 2024-03-24T00:39:11.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.411+0000 7f34377fe640 1 -- 172.21.15.67:0/448189333 <== osd.6 v2:172.21.15.72:6816/1725206562 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f344800ae50 con 0x7f343c0753a0 2024-03-24T00:39:11.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.411+0000 7f0bfa7fc640 1 --2- 172.21.15.67:0/1485836817 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x7f0be80753d0 0x7f0be8077810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.412+0000 7f0c09410640 1 --2- 172.21.15.67:0/1485836817 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0be806bc50 0x7f0be806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.412+0000 7f0bfa7fc640 1 -- 172.21.15.67:0/1485836817 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0bf4016e50 con 0x7f0be80753d0 2024-03-24T00:39:11.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.412+0000 7f0bfa7fc640 1 -- 172.21.15.67:0/1485836817 <== mon.1 v2:172.21.15.72:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0bf408d360 con 0x7f0c04147110 2024-03-24T00:39:11.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.412+0000 7f0c0a412640 1 --2- 172.21.15.67:0/1485836817 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x7f0be80753d0 0x7f0be8077810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.412+0000 7f0c0a412640 1 --2- 172.21.15.67:0/1485836817 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x7f0be80753d0 0x7f0be8077810 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.413+0000 7f0bfa7fc640 1 -- 172.21.15.67:0/1485836817 <== osd.7 v2:172.21.15.72:6824/2893038746 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0bf4016e50 con 0x7f0be80753d0 2024-03-24T00:39:11.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.432+0000 7f0c09410640 1 --2- 172.21.15.67:0/1485836817 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0be806bc50 0x7f0be806e110 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0c040b0ed0 tx=0x7f0c00008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.437+0000 7f3efeffd640 1 -- 172.21.15.67:0/3619062385 --> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3eec002d50 con 0x7f3ef80752d0 2024-03-24T00:39:11.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.437+0000 7f3f117fa640 1 -- 172.21.15.67:0/3619062385 <== osd.2 v2:172.21.15.67:6817/1268432561 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3eec002d50 con 0x7f3ef80752d0 2024-03-24T00:39:11.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.438+0000 7f3f1aec6640 1 -- 172.21.15.67:0/3619062385 >> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] conn(0x7f3ef80752d0 msgr2=0x7f3ef8077710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.438+0000 7f3f1aec6640 1 --2- 172.21.15.67:0/3619062385 >> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] conn(0x7f3ef80752d0 0x7f3ef8077710 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.438+0000 7f3f1aec6640 1 -- 172.21.15.67:0/3619062385 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3ef806bbd0 msgr2=0x7f3ef806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.438+0000 7f3f1aec6640 1 --2- 172.21.15.67:0/3619062385 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3ef806bbd0 0x7f3ef806e090 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3f08000f60 tx=0x7f3f08009040 comp rx=0 tx=0).stop 2024-03-24T00:39:11.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.438+0000 7f3f1aec6640 1 -- 172.21.15.67:0/3619062385 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f0c053e50 msgr2=0x7f3f0c0e92e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.438+0000 7f3f1aec6640 1 --2- 172.21.15.67:0/3619062385 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f0c053e50 0x7f3f0c0e92e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3f14068730 tx=0x7f3f14069920 comp rx=0 tx=0).stop 2024-03-24T00:39:11.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.438+0000 7f3f19463640 1 -- 172.21.15.67:0/3619062385 reap_dead start 2024-03-24T00:39:11.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.438+0000 7f3f1aec6640 1 -- 172.21.15.67:0/3619062385 shutdown_connections 2024-03-24T00:39:11.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.438+0000 7f3f1aec6640 1 -- 172.21.15.67:0/3619062385 >> 172.21.15.67:0/3619062385 conn(0x7f3f0c009fe0 msgr2=0x7f3f0c005710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:11.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.439+0000 7f3f1aec6640 1 -- 172.21.15.67:0/3619062385 shutdown_connections 2024-03-24T00:39:11.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.439+0000 7f3f1aec6640 1 -- 172.21.15.67:0/3619062385 wait complete. 2024-03-24T00:39:11.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.439+0000 7f04b32c0640 1 -- 172.21.15.67:0/4217347271 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f04ac0b0590 con 0x7f04900753d0 2024-03-24T00:39:11.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.439+0000 7f049dffb640 1 -- 172.21.15.67:0/4217347271 <== osd.0 v2:172.21.15.67:6808/2628130519 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f04ac0b0590 con 0x7f04900753d0 2024-03-24T00:39:11.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.440+0000 7f04b32c0640 1 -- 172.21.15.67:0/4217347271 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x7f04900753d0 msgr2=0x7f0490077810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.440+0000 7f04b32c0640 1 --2- 172.21.15.67:0/4217347271 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x7f04900753d0 0x7f0490077810 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.440+0000 7f9e3f7fe640 1 -- 172.21.15.67:0/1881287113 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9e38002dd0 con 0x7f9e40075450 2024-03-24T00:39:11.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.440+0000 7f04b32c0640 1 -- 172.21.15.67:0/4217347271 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f049006bcd0 msgr2=0x7f049006e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.441+0000 7f04b32c0640 1 --2- 172.21.15.67:0/4217347271 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f049006bcd0 0x7f049006e190 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f04ac0cebd0 tx=0x7f04a8009040 comp rx=0 tx=0).stop 2024-03-24T00:39:11.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.441+0000 7f04b32c0640 1 -- 172.21.15.67:0/4217347271 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04ac0cec50 msgr2=0x7f04ac0ca9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.441+0000 7f04b32c0640 1 --2- 172.21.15.67:0/4217347271 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04ac0cec50 0x7f04ac0ca9c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f04a000bb30 tx=0x7f04a00046c0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.441+0000 7f04b185d640 1 -- 172.21.15.67:0/4217347271 reap_dead start 2024-03-24T00:39:11.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.441+0000 7f9e51ffb640 1 -- 172.21.15.67:0/1881287113 <== osd.3 v2:172.21.15.67:6816/3303489481 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9e38002dd0 con 0x7f9e40075450 2024-03-24T00:39:11.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.441+0000 7f04b32c0640 1 -- 172.21.15.67:0/4217347271 shutdown_connections 2024-03-24T00:39:11.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.441+0000 7f04b32c0640 1 -- 172.21.15.67:0/4217347271 >> 172.21.15.67:0/4217347271 conn(0x7f04ac099a30 msgr2=0x7f04ac06f2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:11.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.441+0000 7fec7cb6b640 1 -- 172.21.15.67:0/3867314983 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fec780cbfd0 con 0x7fec640753a0 2024-03-24T00:39:11.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.441+0000 7f9e3f7fe640 1 -- 172.21.15.67:0/1881287113 >> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] conn(0x7f9e40075450 msgr2=0x7f9e40077890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.441+0000 7f9e3f7fe640 1 --2- 172.21.15.67:0/1881287113 >> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] conn(0x7f9e40075450 0x7f9e40077890 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.442+0000 7fec5ffff640 1 -- 172.21.15.67:0/3867314983 <== osd.1 v2:172.21.15.67:6800/4062510404 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fec780cbfd0 con 0x7fec640753a0 2024-03-24T00:39:11.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.442+0000 7f04b32c0640 1 -- 172.21.15.67:0/4217347271 shutdown_connections 2024-03-24T00:39:11.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.442+0000 7f04b32c0640 1 -- 172.21.15.67:0/4217347271 wait complete. 2024-03-24T00:39:11.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.442+0000 7f9e3f7fe640 1 -- 172.21.15.67:0/1881287113 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9e4006bcd0 msgr2=0x7f9e4006e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.442+0000 7f9e3f7fe640 1 --2- 172.21.15.67:0/1881287113 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9e4006bcd0 0x7f9e4006e190 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9e5400b0a0 tx=0x7f9e5c0755a0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.442+0000 7f9e3f7fe640 1 -- 172.21.15.67:0/1881287113 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9e540ead00 msgr2=0x7f9e540dc7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.442+0000 7f9e3f7fe640 1 --2- 172.21.15.67:0/1881287113 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9e540ead00 0x7f9e540dc7f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c00c390 tx=0x7f9e4c00c790 comp rx=0 tx=0).stop 2024-03-24T00:39:11.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.442+0000 7f39d6e9a640 1 -- 172.21.15.67:0/3177867923 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f39c80ddca0 con 0x7f39b4075440 2024-03-24T00:39:11.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.442+0000 7f9e61c63640 1 -- 172.21.15.67:0/1881287113 reap_dead start 2024-03-24T00:39:11.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.442+0000 7f9e3f7fe640 1 -- 172.21.15.67:0/1881287113 shutdown_connections 2024-03-24T00:39:11.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.442+0000 7f9e3f7fe640 1 -- 172.21.15.67:0/1881287113 >> 172.21.15.67:0/1881287113 conn(0x7f9e54009fb0 msgr2=0x7f9e5404f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:11.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.442+0000 7fec7cb6b640 1 -- 172.21.15.67:0/3867314983 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x7fec640753a0 msgr2=0x7fec640777e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.442+0000 7fec7cb6b640 1 --2- 172.21.15.67:0/3867314983 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x7fec640753a0 0x7fec640777e0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.442+0000 7f9e3f7fe640 1 -- 172.21.15.67:0/1881287113 shutdown_connections 2024-03-24T00:39:11.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.442+0000 7f9e3f7fe640 1 -- 172.21.15.67:0/1881287113 wait complete. 2024-03-24T00:39:11.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.442+0000 7f39cd7fa640 1 -- 172.21.15.67:0/3177867923 <== osd.5 v2:172.21.15.72:6800/288486206 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f39c80ddca0 con 0x7f39b4075440 2024-03-24T00:39:11.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.442+0000 7fec7cb6b640 1 -- 172.21.15.67:0/3867314983 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fec6406bca0 msgr2=0x7fec6406e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.442+0000 7fec7cb6b640 1 --2- 172.21.15.67:0/3867314983 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fec6406bca0 0x7fec6406e160 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fec68036000 tx=0x7fec68030000 comp rx=0 tx=0).stop 2024-03-24T00:39:11.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.443+0000 7fec7cb6b640 1 -- 172.21.15.67:0/3867314983 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec78072030 msgr2=0x7fec780724c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.443+0000 7fec7cb6b640 1 --2- 172.21.15.67:0/3867314983 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec78072030 0x7fec780724c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fec6c00d5a0 tx=0x7fec6c00da60 comp rx=0 tx=0).stop 2024-03-24T00:39:11.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.443+0000 7fec77fff640 1 -- 172.21.15.67:0/3867314983 reap_dead start 2024-03-24T00:39:11.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.443+0000 7fec7cb6b640 1 -- 172.21.15.67:0/3867314983 shutdown_connections 2024-03-24T00:39:11.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.443+0000 7fec7cb6b640 1 -- 172.21.15.67:0/3867314983 >> 172.21.15.67:0/3867314983 conn(0x7fec780b13e0 msgr2=0x7fec780ccf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:11.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.443+0000 7f39d6e9a640 1 -- 172.21.15.67:0/3177867923 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x7f39b4075440 msgr2=0x7f39b4077880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.443+0000 7f39d6e9a640 1 --2- 172.21.15.67:0/3177867923 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x7f39b4075440 0x7f39b4077880 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.443+0000 7f39d6e9a640 1 -- 172.21.15.67:0/3177867923 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f39b406bcd0 msgr2=0x7f39b406e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.443+0000 7f39d6e9a640 1 --2- 172.21.15.67:0/3177867923 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f39b406bcd0 0x7f39b406e190 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f39d004f6d0 tx=0x7f39d006aef0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.443+0000 7f39d6e9a640 1 -- 172.21.15.67:0/3177867923 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39c8050010 msgr2=0x7f39c8039e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.443+0000 7f39d6e9a640 1 --2- 172.21.15.67:0/3177867923 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39c8050010 0x7f39c8039e20 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f39bc033000 tx=0x7f39bc0046c0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.443+0000 7f39d5437640 1 -- 172.21.15.67:0/3177867923 reap_dead start 2024-03-24T00:39:11.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.443+0000 7f39d6e9a640 1 -- 172.21.15.67:0/3177867923 shutdown_connections 2024-03-24T00:39:11.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.443+0000 7f39d6e9a640 1 -- 172.21.15.67:0/3177867923 >> 172.21.15.67:0/3177867923 conn(0x7f39c8055340 msgr2=0x7f39c8006830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:11.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.444+0000 7fec7cb6b640 1 -- 172.21.15.67:0/3867314983 shutdown_connections 2024-03-24T00:39:11.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.444+0000 7fec7cb6b640 1 -- 172.21.15.67:0/3867314983 wait complete. 2024-03-24T00:39:11.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.444+0000 7fab6896f640 1 -- 172.21.15.67:0/2824837902 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fab640b2800 con 0x7fab44075450 2024-03-24T00:39:11.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.444+0000 7f39d6e9a640 1 -- 172.21.15.67:0/3177867923 shutdown_connections 2024-03-24T00:39:11.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.444+0000 7fab52ffd640 1 -- 172.21.15.67:0/2824837902 <== osd.4 v2:172.21.15.72:6808/477220793 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fab640b2800 con 0x7fab44075450 2024-03-24T00:39:11.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.444+0000 7f39d6e9a640 1 -- 172.21.15.67:0/3177867923 wait complete. 2024-03-24T00:39:11.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.444+0000 7f345520b640 1 -- 172.21.15.67:0/448189333 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3430002d50 con 0x7f343c0753a0 2024-03-24T00:39:11.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.445+0000 7fab50ff9640 1 -- 172.21.15.67:0/2824837902 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x7fab44075450 msgr2=0x7fab44077890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.445+0000 7fab50ff9640 1 --2- 172.21.15.67:0/2824837902 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x7fab44075450 0x7fab44077890 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.445+0000 7f34377fe640 1 -- 172.21.15.67:0/448189333 <== osd.6 v2:172.21.15.72:6816/1725206562 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3430002d50 con 0x7f343c0753a0 2024-03-24T00:39:11.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.445+0000 7fab50ff9640 1 -- 172.21.15.67:0/2824837902 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fab4406bcd0 msgr2=0x7fab4406e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.445+0000 7fab50ff9640 1 --2- 172.21.15.67:0/2824837902 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fab4406bcd0 0x7fab4406e190 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fab5c036000 tx=0x7fab5c030000 comp rx=0 tx=0).stop 2024-03-24T00:39:11.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.445+0000 7fab50ff9640 1 -- 172.21.15.67:0/2824837902 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab640c64c0 msgr2=0x7fab640c6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.445+0000 7fab50ff9640 1 --2- 172.21.15.67:0/2824837902 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab640c64c0 0x7fab640c6950 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fab58001eb0 tx=0x7fab58007720 comp rx=0 tx=0).stop 2024-03-24T00:39:11.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.445+0000 7fab62d9d640 1 -- 172.21.15.67:0/2824837902 reap_dead start 2024-03-24T00:39:11.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.445+0000 7fab50ff9640 1 -- 172.21.15.67:0/2824837902 shutdown_connections 2024-03-24T00:39:11.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.445+0000 7fab50ff9640 1 -- 172.21.15.67:0/2824837902 >> 172.21.15.67:0/2824837902 conn(0x7fab640b1ae0 msgr2=0x7fab64160050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:11.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.445+0000 7f345520b640 1 -- 172.21.15.67:0/448189333 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x7f343c0753a0 msgr2=0x7f343c0777e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.445+0000 7f345520b640 1 --2- 172.21.15.67:0/448189333 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x7f343c0753a0 0x7f343c0777e0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.446+0000 7f345520b640 1 -- 172.21.15.67:0/448189333 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f343c06bc20 msgr2=0x7f343c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.446+0000 7f345520b640 1 --2- 172.21.15.67:0/448189333 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f343c06bc20 0x7f343c06e0e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3440001750 tx=0x7f3440009040 comp rx=0 tx=0).stop 2024-03-24T00:39:11.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.446+0000 7f345520b640 1 -- 172.21.15.67:0/448189333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34501f93c0 msgr2=0x7f34501f9850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.446+0000 7f345520b640 1 --2- 172.21.15.67:0/448189333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34501f93c0 0x7f34501f9850 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f34500af6b0 tx=0x7f3448008c90 comp rx=0 tx=0).stop 2024-03-24T00:39:11.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.446+0000 7f344f59e640 1 -- 172.21.15.67:0/448189333 reap_dead start 2024-03-24T00:39:11.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.446+0000 7f345520b640 1 -- 172.21.15.67:0/448189333 shutdown_connections 2024-03-24T00:39:11.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.446+0000 7fab50ff9640 1 -- 172.21.15.67:0/2824837902 shutdown_connections 2024-03-24T00:39:11.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.446+0000 7f345520b640 1 -- 172.21.15.67:0/448189333 >> 172.21.15.67:0/448189333 conn(0x7f345006b670 msgr2=0x7f34501595c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:11.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.446+0000 7f345520b640 1 -- 172.21.15.67:0/448189333 shutdown_connections 2024-03-24T00:39:11.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.446+0000 7f345520b640 1 -- 172.21.15.67:0/448189333 wait complete. 2024-03-24T00:39:11.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.446+0000 7fab50ff9640 1 -- 172.21.15.67:0/2824837902 wait complete. 2024-03-24T00:39:11.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.447+0000 7f0c0ac13640 1 -- 172.21.15.67:0/1485836817 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0c040cc320 con 0x7f0be80753d0 2024-03-24T00:39:11.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.447+0000 7f0bfa7fc640 1 -- 172.21.15.67:0/1485836817 <== osd.7 v2:172.21.15.72:6824/2893038746 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0c040cc320 con 0x7f0be80753d0 2024-03-24T00:39:11.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.448+0000 7f0c0ac13640 1 -- 172.21.15.67:0/1485836817 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x7f0be80753d0 msgr2=0x7f0be8077810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.448+0000 7f0c0ac13640 1 --2- 172.21.15.67:0/1485836817 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x7f0be80753d0 0x7f0be8077810 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.448+0000 7f0c0ac13640 1 -- 172.21.15.67:0/1485836817 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0be806bc50 msgr2=0x7f0be806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.448+0000 7f0c0ac13640 1 --2- 172.21.15.67:0/1485836817 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0be806bc50 0x7f0be806e110 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0c040b0ed0 tx=0x7f0c00008040 comp rx=0 tx=0).stop 2024-03-24T00:39:11.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.448+0000 7f0c0ac13640 1 -- 172.21.15.67:0/1485836817 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0c04147110 msgr2=0x7f0c0415b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.448+0000 7f0c0ac13640 1 --2- 172.21.15.67:0/1485836817 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0c04147110 0x7f0c0415b770 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0bf4010960 tx=0x7f0bf4010e20 comp rx=0 tx=0).stop 2024-03-24T00:39:11.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.451+0000 7f0c0a412640 1 -- 172.21.15.67:0/1485836817 reap_dead start 2024-03-24T00:39:11.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.451+0000 7f0c0ac13640 1 -- 172.21.15.67:0/1485836817 shutdown_connections 2024-03-24T00:39:11.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.451+0000 7f0c0ac13640 1 -- 172.21.15.67:0/1485836817 >> 172.21.15.67:0/1485836817 conn(0x7f0c040cf7b0 msgr2=0x7f0c0415d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:11.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.451+0000 7f0c0ac13640 1 -- 172.21.15.67:0/1485836817 shutdown_connections 2024-03-24T00:39:11.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.452+0000 7f0c0ac13640 1 -- 172.21.15.67:0/1485836817 wait complete. 2024-03-24T00:39:11.506 INFO:teuthology.orchestra.run.smithi067.stdout:55834574858 2024-03-24T00:39:11.507 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-24T00:39:11.507 INFO:teuthology.orchestra.run.smithi067.stdout:55834574857 2024-03-24T00:39:11.508 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-24T00:39:11.508 INFO:teuthology.orchestra.run.smithi067.stdout:55834574857 2024-03-24T00:39:11.509 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-24T00:39:11.509 INFO:teuthology.orchestra.run.smithi067.stdout:55834574858 2024-03-24T00:39:11.510 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-24T00:39:11.510 INFO:teuthology.orchestra.run.smithi067.stdout:55834574858 2024-03-24T00:39:11.511 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-24T00:39:11.511 INFO:teuthology.orchestra.run.smithi067.stdout:55834574857 2024-03-24T00:39:11.512 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-24T00:39:11.512 INFO:teuthology.orchestra.run.smithi067.stdout:55834574858 2024-03-24T00:39:11.513 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-24T00:39:11.513 INFO:teuthology.orchestra.run.smithi067.stdout:55834574859 2024-03-24T00:39:11.514 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-24T00:39:11.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.715+0000 7f02534c0640 1 Processor -- start 2024-03-24T00:39:11.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.716+0000 7f02534c0640 1 -- start start 2024-03-24T00:39:11.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.716+0000 7f02534c0640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f024c0cc5c0 0x7f024c0ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.716+0000 7f02534c0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f024c06b040 0x7f024c06b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.716+0000 7f02534c0640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f024c0cc9a0 con 0x7f024c161cc0 2024-03-24T00:39:11.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.716+0000 7f02534c0640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f024c0b1f90 con 0x7f024c06b040 2024-03-24T00:39:11.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.716+0000 7f02534c0640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f024c0b0e60 con 0x7f024c0cc5c0 2024-03-24T00:39:11.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.717+0000 7f025125c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f024c0cc5c0 0x7f024c0ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.717+0000 7f0250a5b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f024c06b040 0x7f024c06b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.717+0000 7f0251a5d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f024c161cc0 0x7f024c1620d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45490/0 (socket says 172.21.15.67:45490) 2024-03-24T00:39:11.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.717+0000 7f0251a5d640 1 -- 172.21.15.67:0/1800703410 learned_addr learned my addr 172.21.15.67:0/1800703410 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:11.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.718+0000 7f0250a5b640 1 -- 172.21.15.67:0/1800703410 >> v2:172.21.15.72:3300/0 conn(0x7f024c0cc5c0 msgr2=0x7f024c0ccde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:11.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.718+0000 7f0250a5b640 1 --2- 172.21.15.67:0/1800703410 >> v2:172.21.15.72:3300/0 conn(0x7f024c0cc5c0 0x7f024c0ccde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.718+0000 7f0250a5b640 1 -- 172.21.15.67:0/1800703410 >> v1:172.21.15.67:6789/0 conn(0x7f024c161cc0 legacy=0x7f024c1620d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:11.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.718+0000 7f0250a5b640 1 -- 172.21.15.67:0/1800703410 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f024c0b1140 con 0x7f024c06b040 2024-03-24T00:39:11.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.718+0000 7f3ee8159640 1 Processor -- start 2024-03-24T00:39:11.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.718+0000 7fdc2196b640 1 Processor -- start 2024-03-24T00:39:11.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.719+0000 7f3ee8159640 1 -- start start 2024-03-24T00:39:11.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.718+0000 7fdc2196b640 1 -- start start 2024-03-24T00:39:11.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.719+0000 7f0250a5b640 1 --2- 172.21.15.67:0/1800703410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f024c06b040 0x7f024c06b4d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f024c06e340 tx=0x7f024002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5920b4437ff1d55 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.720+0000 7f023ffff640 1 -- 172.21.15.67:0/1800703410 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02400315a0 con 0x7f024c06b040 2024-03-24T00:39:11.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.720+0000 7f023ffff640 1 -- 172.21.15.67:0/1800703410 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0240037070 con 0x7f024c06b040 2024-03-24T00:39:11.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.720+0000 7f023ffff640 1 -- 172.21.15.67:0/1800703410 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0240031960 con 0x7f024c06b040 2024-03-24T00:39:11.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.720+0000 7fdc2196b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fdc1c162360 0x7fdc1c071850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.720+0000 7fdc2196b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdc1c071d90 0x7fdc1c0b36f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.720+0000 7fdc2196b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc1c06e7a0 con 0x7fdc1c06e2a0 2024-03-24T00:39:11.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.720+0000 7f3ee8159640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3ee00cf020 0x7f3ee0165720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.720+0000 7f3ee8159640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ee00cf490 con 0x7f3ee015e050 2024-03-24T00:39:11.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.720+0000 7f3ee8159640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ee015e760 con 0x7f3ee006f280 2024-03-24T00:39:11.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.720+0000 7f3ee8159640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee015ea40 con 0x7f3ee00cf020 2024-03-24T00:39:11.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.720+0000 7fdc2196b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc1c06f3b0 con 0x7fdc1c071d90 2024-03-24T00:39:11.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.720+0000 7fdc2196b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc1c0721c0 con 0x7fdc1c162360 2024-03-24T00:39:11.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.720+0000 7fdc20969640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fdc1c162360 0x7fdc1c071850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.720+0000 7fdc20969640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fdc1c162360 0x7fdc1c071850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:34146/0 (socket says 172.21.15.67:34146) 2024-03-24T00:39:11.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.720+0000 7fdc20969640 1 -- 172.21.15.67:0/1431484291 learned_addr learned my addr 172.21.15.67:0/1431484291 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:11.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.720+0000 7fdc2116a640 1 --2- 172.21.15.67:0/1431484291 >> v2:172.21.15.67:3300/0 conn(0x7fdc1c071d90 0x7fdc1c0b36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.720+0000 7fdc20969640 1 -- 172.21.15.67:0/1431484291 >> v2:172.21.15.67:3300/0 conn(0x7fdc1c071d90 msgr2=0x7fdc1c0b36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:11.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.720+0000 7fdc20969640 1 --2- 172.21.15.67:0/1431484291 >> v2:172.21.15.67:3300/0 conn(0x7fdc1c071d90 0x7fdc1c0b36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.720+0000 7f3ee5ef5640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3ee015e050 0x7f3ee006eb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45514/0 (socket says 172.21.15.67:45514) 2024-03-24T00:39:11.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.720+0000 7f3ee5ef5640 1 -- 172.21.15.67:0/236111473 learned_addr learned my addr 172.21.15.67:0/236111473 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:11.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.720+0000 7f3ee66f6640 1 --2- 172.21.15.67:0/236111473 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3ee00cf020 0x7f3ee0165720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.720+0000 7fdc20969640 1 -- 172.21.15.67:0/1431484291 >> v1:172.21.15.67:6789/0 conn(0x7fdc1c06e2a0 legacy=0x7fdc1c0cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:11.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.720+0000 7fdc20969640 1 -- 172.21.15.67:0/1431484291 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc1c06ea80 con 0x7fdc1c162360 2024-03-24T00:39:11.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.721+0000 7f45a65b0640 1 Processor -- start 2024-03-24T00:39:11.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.721+0000 7f02534c0640 1 -- 172.21.15.67:0/1800703410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f024c06b040 msgr2=0x7f024c06b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.721+0000 7f3ee4ef3640 1 -- 172.21.15.67:0/236111473 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1919608060 0 0) 0x7f3ee00cf490 con 0x7f3ee015e050 2024-03-24T00:39:11.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.721+0000 7f3ee4ef3640 1 -- 172.21.15.67:0/236111473 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ecc003520 con 0x7f3ee015e050 2024-03-24T00:39:11.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.721+0000 7f02534c0640 1 --2- 172.21.15.67:0/1800703410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f024c06b040 0x7f024c06b4d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f024c06e340 tx=0x7f024002b840 comp rx=0 tx=0).stop 2024-03-24T00:39:11.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.721+0000 7f02534c0640 1 -- 172.21.15.67:0/1800703410 shutdown_connections 2024-03-24T00:39:11.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.721+0000 7f02534c0640 1 --2- 172.21.15.67:0/1800703410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f024c06b040 0x7f024c06b4d0 secure :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f024c06e340 tx=0x7f024002b840 comp rx=0 tx=0).stop 2024-03-24T00:39:11.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.721+0000 7f02534c0640 1 --2- 172.21.15.67:0/1800703410 >> v2:172.21.15.72:3300/0 conn(0x7f024c0cc5c0 0x7f024c0ccde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.721+0000 7f02534c0640 1 -- 172.21.15.67:0/1800703410 >> 172.21.15.67:0/1800703410 conn(0x7f024c099a30 msgr2=0x7f024c099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:11.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.721+0000 7f45a65b0640 1 -- start start 2024-03-24T00:39:11.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.721+0000 7f3ee66f6640 1 -- 172.21.15.67:0/236111473 >> v1:172.21.15.72:6789/0 conn(0x7f3ee006f280 legacy=0x7f3ee00ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.721+0000 7f3ee66f6640 1 -- 172.21.15.67:0/236111473 >> v1:172.21.15.67:6789/0 conn(0x7f3ee015e050 legacy=0x7f3ee006eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.721+0000 7f3ee4ef3640 1 -- 172.21.15.67:0/236111473 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2379131867 0 0) 0x7f3ecc003520 con 0x7f3ee015e050 2024-03-24T00:39:11.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.721+0000 7f3ee66f6640 1 -- 172.21.15.67:0/236111473 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ee015ed20 con 0x7f3ee00cf020 2024-03-24T00:39:11.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.722+0000 7f02534c0640 1 -- 172.21.15.67:0/1800703410 shutdown_connections 2024-03-24T00:39:11.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.722+0000 7f997f59e640 1 Processor -- start 2024-03-24T00:39:11.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.722+0000 7f997f59e640 1 -- start start 2024-03-24T00:39:11.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.722+0000 7f02534c0640 1 -- 172.21.15.67:0/1800703410 wait complete. 2024-03-24T00:39:11.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.722+0000 7fdc20969640 1 --2- 172.21.15.67:0/1431484291 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc1c162360 0x7fdc1c071850 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c073150 tx=0x7fdc1402be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2cea7538f5c5d2b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.722+0000 7f45a65b0640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f45a015df10 0x7f45a0161c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.722+0000 7f45a65b0640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f45a00d0cc0 0x7f45a00d27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.722+0000 7f45a65b0640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45a00b6f00 con 0x7f45a00b5780 2024-03-24T00:39:11.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.722+0000 7f45a65b0640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45a0162b80 con 0x7f45a015df10 2024-03-24T00:39:11.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.722+0000 7f45a65b0640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45a01625c0 con 0x7f45a00d0cc0 2024-03-24T00:39:11.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.723+0000 7f3ee66f6640 1 --2- 172.21.15.67:0/236111473 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3ee00cf020 0x7f3ee0165720 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3edc0097d0 tx=0x7f3edc02b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d126d160d2609011 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.723+0000 7fdc1b7fe640 1 -- 172.21.15.67:0/1431484291 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc1402c9f0 con 0x7fdc1c162360 2024-03-24T00:39:11.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.723+0000 7fdc1b7fe640 1 -- 172.21.15.67:0/1431484291 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc1402cb90 con 0x7fdc1c162360 2024-03-24T00:39:11.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.723+0000 7f45a4dad640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f45a00b5780 0x7f45a00d05b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45518/0 (socket says 172.21.15.67:45518) 2024-03-24T00:39:11.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.723+0000 7f45a4dad640 1 -- 172.21.15.67:0/718511418 learned_addr learned my addr 172.21.15.67:0/718511418 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:11.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.723+0000 7f45a55ae640 1 --2- 172.21.15.67:0/718511418 >> v2:172.21.15.72:3300/0 conn(0x7f45a015df10 0x7f45a0161c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.723+0000 7f45a5daf640 1 --2- 172.21.15.67:0/718511418 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f45a00d0cc0 0x7f45a00d27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.723+0000 7f1bb3337640 1 Processor -- start 2024-03-24T00:39:11.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.723+0000 7f4597fff640 1 -- 172.21.15.67:0/718511418 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1441766795 0 0) 0x7f4598001490 con 0x7f45a00b5780 2024-03-24T00:39:11.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.723+0000 7f4597fff640 1 -- 172.21.15.67:0/718511418 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f458c003520 con 0x7f45a00b5780 2024-03-24T00:39:11.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.723+0000 7f45a55ae640 1 -- 172.21.15.67:0/718511418 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f45a00d0cc0 msgr2=0x7f45a00d27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:11.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.723+0000 7f45a55ae640 1 --2- 172.21.15.67:0/718511418 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f45a00d0cc0 0x7f45a00d27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.723+0000 7f45a55ae640 1 -- 172.21.15.67:0/718511418 >> v1:172.21.15.67:6789/0 conn(0x7f45a00b5780 legacy=0x7f45a00d05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.723+0000 7f45a55ae640 1 -- 172.21.15.67:0/718511418 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a01628a0 con 0x7f45a015df10 2024-03-24T00:39:11.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.723+0000 7f45a5daf640 1 --2- 172.21.15.67:0/718511418 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f45a00d0cc0 0x7f45a00d27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:39:11.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.723+0000 7f3ee4ef3640 1 -- 172.21.15.67:0/236111473 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3edc037070 con 0x7f3ee00cf020 2024-03-24T00:39:11.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.723+0000 7f3ee4ef3640 1 -- 172.21.15.67:0/236111473 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3edc030510 con 0x7f3ee00cf020 2024-03-24T00:39:11.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.723+0000 7f3ee4ef3640 1 -- 172.21.15.67:0/236111473 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed80019f0 con 0x7f3ee00cf020 2024-03-24T00:39:11.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.724+0000 7fdc2196b640 1 -- 172.21.15.67:0/1431484291 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc1c162360 msgr2=0x7fdc1c071850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.724+0000 7fdc2196b640 1 --2- 172.21.15.67:0/1431484291 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc1c162360 0x7fdc1c071850 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c073150 tx=0x7fdc1402be50 comp rx=0 tx=0).stop 2024-03-24T00:39:11.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.724+0000 7f997f59e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f99800b4b30 0x7f998006f380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.724+0000 7f997f59e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99800b3060 con 0x7f9980140990 2024-03-24T00:39:11.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.724+0000 7f997f59e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99800cf9b0 con 0x7f998006ef20 2024-03-24T00:39:11.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.724+0000 7f997f59e640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99800cfc90 con 0x7f99800b4b30 2024-03-24T00:39:11.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.723+0000 7f02534c0640 1 Processor -- start 2024-03-24T00:39:11.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.723+0000 7f02534c0640 1 -- start start 2024-03-24T00:39:11.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.724+0000 7f997dd9b640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f998006ef20 0x7f99800b4420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:51342/0 (socket says 172.21.15.67:51342) 2024-03-24T00:39:11.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.724+0000 7f997dd9b640 1 -- 172.21.15.67:0/4166009737 learned_addr learned my addr 172.21.15.67:0/4166009737 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:11.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.724+0000 7fdfb317d640 1 Processor -- start 2024-03-24T00:39:11.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.724+0000 7fdfb317d640 1 -- start start 2024-03-24T00:39:11.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.724+0000 7f1bb3337640 1 -- start start 2024-03-24T00:39:11.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.724+0000 7f997d59a640 1 -- 172.21.15.67:0/4166009737 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1555124639 0 0) 0x7f99800b3060 con 0x7f9980140990 2024-03-24T00:39:11.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.724+0000 7f997d59a640 1 -- 172.21.15.67:0/4166009737 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9964003520 con 0x7f9980140990 2024-03-24T00:39:11.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.724+0000 7f997d59a640 1 -- 172.21.15.67:0/4166009737 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2851684367 0 0) 0x7f99800cf9b0 con 0x7f998006ef20 2024-03-24T00:39:11.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.724+0000 7f997d59a640 1 -- 172.21.15.67:0/4166009737 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99640048e0 con 0x7f998006ef20 2024-03-24T00:39:11.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.724+0000 7f3ee8159640 1 -- 172.21.15.67:0/236111473 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3ee00cf020 msgr2=0x7f3ee0165720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.724+0000 7f3ee8159640 1 --2- 172.21.15.67:0/236111473 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3ee00cf020 0x7f3ee0165720 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3edc0097d0 tx=0x7f3edc02b800 comp rx=0 tx=0).stop 2024-03-24T00:39:11.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.725+0000 7f02534c0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f024c0cc5c0 0x7f024c1f7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.725+0000 7fdc2196b640 1 -- 172.21.15.67:0/1431484291 shutdown_connections 2024-03-24T00:39:11.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.725+0000 7fdc2196b640 1 --2- 172.21.15.67:0/1431484291 >> v2:172.21.15.67:3300/0 conn(0x7fdc1c071d90 0x7fdc1c0b36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.725+0000 7fdc2196b640 1 --2- 172.21.15.67:0/1431484291 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc1c162360 0x7fdc1c071850 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.724+0000 7f997ed9d640 1 --2- 172.21.15.67:0/4166009737 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f99800b4b30 0x7f998006f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.725+0000 7fdc2196b640 1 -- 172.21.15.67:0/1431484291 >> 172.21.15.67:0/1431484291 conn(0x7fdc1c0cf7b0 msgr2=0x7fdc1c0cfbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:11.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.725+0000 7fdc2196b640 1 -- 172.21.15.67:0/1431484291 shutdown_connections 2024-03-24T00:39:11.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.725+0000 7f45a55ae640 1 --2- 172.21.15.67:0/718511418 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f45a015df10 0x7f45a0161c50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f45900097a0 tx=0x7f459002b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=50d7de36d51dba6c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.725+0000 7f997d59a640 1 -- 172.21.15.67:0/4166009737 <== mon.1 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3447640217 0 0) 0x7f99640048e0 con 0x7f998006ef20 2024-03-24T00:39:11.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.725+0000 7f997d59a640 1 -- 172.21.15.67:0/4166009737 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f99800b4b30 msgr2=0x7f998006f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:11.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.725+0000 7f997d59a640 1 --2- 172.21.15.67:0/4166009737 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f99800b4b30 0x7f998006f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.725+0000 7f997d59a640 1 -- 172.21.15.67:0/4166009737 >> v1:172.21.15.67:6789/0 conn(0x7f9980140990 legacy=0x7f99800b3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.725+0000 7f997d59a640 1 -- 172.21.15.67:0/4166009737 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99800b3300 con 0x7f998006ef20 2024-03-24T00:39:11.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.725+0000 7f02534c0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f024c161cc0 0x7f024c0b4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.725+0000 7f3ee8159640 1 -- 172.21.15.67:0/236111473 shutdown_connections 2024-03-24T00:39:11.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.725+0000 7f3ee8159640 1 --2- 172.21.15.67:0/236111473 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3ee00cf020 0x7f3ee0165720 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.725+0000 7f3ee8159640 1 -- 172.21.15.67:0/236111473 >> 172.21.15.67:0/236111473 conn(0x7f3ee00b1450 msgr2=0x7f3ee00b1830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:11.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.725+0000 7f997d59a640 1 -- 172.21.15.67:0/4166009737 <== mon.1 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f9970001670 con 0x7f998006ef20 2024-03-24T00:39:11.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.725+0000 7f997d59a640 1 -- 172.21.15.67:0/4166009737 >> v1:172.21.15.72:6789/0 conn(0x7f998006ef20 legacy=0x7f99800b4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.725+0000 7f997d59a640 1 --2- 172.21.15.67:0/4166009737 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9964004dc0 0x7f99640051a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.725+0000 7f997d59a640 1 --2- 172.21.15.67:0/4166009737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99640037a0 0x7f9964003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.725+0000 7f997d59a640 1 --2- 172.21.15.67:0/4166009737 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9964004280 0x7f99640079a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.725+0000 7f997d59a640 1 -- 172.21.15.67:0/4166009737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99800cfc90 con 0x7f99640037a0 2024-03-24T00:39:11.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.725+0000 7f997d59a640 1 -- 172.21.15.67:0/4166009737 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9964008170 con 0x7f9964004dc0 2024-03-24T00:39:11.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.725+0000 7f997d59a640 1 -- 172.21.15.67:0/4166009737 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9964008370 con 0x7f9964004280 2024-03-24T00:39:11.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.725+0000 7fdc2196b640 1 -- 172.21.15.67:0/1431484291 wait complete. 2024-03-24T00:39:11.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.725+0000 7f025125c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f024c0cc5c0 0x7f024c1f7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.725+0000 7f3ee8159640 1 -- 172.21.15.67:0/236111473 shutdown_connections 2024-03-24T00:39:11.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.725+0000 7f4597fff640 1 -- 172.21.15.67:0/718511418 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4590035020 con 0x7f45a015df10 2024-03-24T00:39:11.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.725+0000 7f4597fff640 1 -- 172.21.15.67:0/718511418 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4590031810 con 0x7f45a015df10 2024-03-24T00:39:11.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.725+0000 7f4597fff640 1 -- 172.21.15.67:0/718511418 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4590031be0 con 0x7f45a015df10 2024-03-24T00:39:11.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.725+0000 7f3ee8159640 1 -- 172.21.15.67:0/236111473 wait complete. 2024-03-24T00:39:11.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.725+0000 7f997e59c640 1 --2- 172.21.15.67:0/4166009737 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9964004dc0 0x7f99640051a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.725+0000 7f997dd9b640 1 --2- 172.21.15.67:0/4166009737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99640037a0 0x7f9964003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.725+0000 7f025125c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f024c0cc5c0 0x7f024c1f7170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:34154/0 (socket says 172.21.15.67:34154) 2024-03-24T00:39:11.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.725+0000 7f1bb3337640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1ba40e3210 0x7f1ba4053f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.725+0000 7f1bb3337640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ba4056c10 con 0x7f1ba4006220 2024-03-24T00:39:11.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.725+0000 7f1bb3337640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ba40545f0 con 0x7f1ba4006640 2024-03-24T00:39:11.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.725+0000 7f1bb3337640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ba4057400 con 0x7f1ba40e3210 2024-03-24T00:39:11.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.726+0000 7f025125c640 1 -- 172.21.15.67:0/3773371621 learned_addr learned my addr 172.21.15.67:0/3773371621 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:11.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.726+0000 7f02534c0640 1 --2- 172.21.15.67:0/3773371621 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f024c0c8530 0x7f024c0c9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.726+0000 7f02534c0640 1 -- 172.21.15.67:0/3773371621 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f024c0cd320 con 0x7f024c161cc0 2024-03-24T00:39:11.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.726+0000 7f997ed9d640 1 --2- 172.21.15.67:0/4166009737 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9964004280 0x7f99640079a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.726+0000 7f0251a5d640 1 --2- 172.21.15.67:0/3773371621 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f024c0c8530 0x7f024c0c9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.726+0000 7f02534c0640 1 -- 172.21.15.67:0/3773371621 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f024c0ce970 con 0x7f024c0cc5c0 2024-03-24T00:39:11.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.726+0000 7f997e59c640 1 -- 172.21.15.67:0/4166009737 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9964004280 msgr2=0x7f99640079a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:11.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.726+0000 7f02534c0640 1 -- 172.21.15.67:0/3773371621 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f024c0b2270 con 0x7f024c0c8530 2024-03-24T00:39:11.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.726+0000 7f997e59c640 1 --2- 172.21.15.67:0/4166009737 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9964004280 0x7f99640079a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.726+0000 7f997e59c640 1 -- 172.21.15.67:0/4166009737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99640037a0 msgr2=0x7f9964003c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:11.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.726+0000 7f0250a5b640 1 --2- 172.21.15.67:0/3773371621 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f024c161cc0 0x7f024c0b4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.726+0000 7f997e59c640 1 --2- 172.21.15.67:0/4166009737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99640037a0 0x7f9964003c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.726+0000 7f1bb10d3640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1ba40e3210 0x7f1ba4053f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.726+0000 7f1bb08d2640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1ba4006220 0x7f1ba40581c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45534/0 (socket says 172.21.15.67:45534) 2024-03-24T00:39:11.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.726+0000 7f997e59c640 1 -- 172.21.15.67:0/4166009737 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99800b3300 con 0x7f9964004dc0 2024-03-24T00:39:11.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.726+0000 7f1bb08d2640 1 -- 172.21.15.67:0/2515221167 learned_addr learned my addr 172.21.15.67:0/2515221167 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:11.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.725+0000 7fdfb317d640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdfa4054200 0x7fdfa4050740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.725+0000 7fdfb317d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfa40526a0 con 0x7fdfa40df800 2024-03-24T00:39:11.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.725+0000 7fdfb317d640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfa4053850 con 0x7fdfa4006770 2024-03-24T00:39:11.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.725+0000 7fdfb317d640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa40dfbe0 con 0x7fdfa4054200 2024-03-24T00:39:11.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.726+0000 7f1bb18d4640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f1ba4006640 0x7f1ba40e35f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:51356/0 (socket says 172.21.15.67:51356) 2024-03-24T00:39:11.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.726+0000 7fdfb217b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fdfa40df800 0x7fdfa4053140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45524/0 (socket says 172.21.15.67:45524) 2024-03-24T00:39:11.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.726+0000 7fdfb217b640 1 -- 172.21.15.67:0/1717811189 learned_addr learned my addr 172.21.15.67:0/1717811189 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:11.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.726+0000 7fdfb197a640 1 --2- 172.21.15.67:0/1717811189 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdfa4054200 0x7fdfa4050740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.726+0000 7f976f5c4640 1 Processor -- start 2024-03-24T00:39:11.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.727+0000 7f997dd9b640 1 --2- 172.21.15.67:0/4166009737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99640037a0 0x7f9964003c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:11.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.726+0000 7f0251a5d640 1 -- 172.21.15.67:0/3773371621 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f024c0cc5c0 msgr2=0x7f024c1f7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.726+0000 7f0251a5d640 1 --2- 172.21.15.67:0/3773371621 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f024c0cc5c0 0x7f024c1f7170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.726+0000 7f0251a5d640 1 -- 172.21.15.67:0/3773371621 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f024c161cc0 msgr2=0x7f024c0b4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.726+0000 7f0251a5d640 1 --2- 172.21.15.67:0/3773371621 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f024c161cc0 0x7f024c0b4e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.726+0000 7f0251a5d640 1 -- 172.21.15.67:0/3773371621 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f024c0c89c0 con 0x7f024c0c8530 2024-03-24T00:39:11.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.727+0000 7f0251a5d640 1 --2- 172.21.15.67:0/3773371621 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f024c0c8530 0x7f024c0c9fb0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f024400be10 tx=0x7f024400bf00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.727+0000 7f1bb10d3640 1 -- 172.21.15.67:0/2515221167 >> v1:172.21.15.72:6789/0 conn(0x7f1ba4006640 legacy=0x7f1ba40e35f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.727+0000 7f1bb10d3640 1 -- 172.21.15.67:0/2515221167 >> v1:172.21.15.67:6789/0 conn(0x7f1ba4006220 legacy=0x7f1ba40581c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.727+0000 7f1bb10d3640 1 -- 172.21.15.67:0/2515221167 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ba40576e0 con 0x7f1ba40e3210 2024-03-24T00:39:11.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.727+0000 7fdfb197a640 1 -- 172.21.15.67:0/1717811189 >> v1:172.21.15.72:6789/0 conn(0x7fdfa4006770 legacy=0x7fdfa4038520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:11.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.727+0000 7fdfb197a640 1 -- 172.21.15.67:0/1717811189 >> v1:172.21.15.67:6789/0 conn(0x7fdfa40df800 legacy=0x7fdfa4053140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.727+0000 7fdfb197a640 1 -- 172.21.15.67:0/1717811189 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa4053c80 con 0x7fdfa4054200 2024-03-24T00:39:11.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.727+0000 7fdc2196b640 1 Processor -- start 2024-03-24T00:39:11.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.727+0000 7fdc2196b640 1 -- start start 2024-03-24T00:39:11.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.727+0000 7f997e59c640 1 --2- 172.21.15.67:0/4166009737 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9964004dc0 0x7f99640051a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f996c007ee0 tx=0x7f9964005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62aac9fc6abc044b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.727+0000 7f976f5c4640 1 -- start start 2024-03-24T00:39:11.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.727+0000 7f3ee8159640 1 Processor -- start 2024-03-24T00:39:11.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.727+0000 7f45a65b0640 1 -- 172.21.15.67:0/718511418 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f45a015df10 msgr2=0x7f45a0161c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.727+0000 7f45a65b0640 1 --2- 172.21.15.67:0/718511418 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f45a015df10 0x7f45a0161c50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f45900097a0 tx=0x7f459002b8b0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.728+0000 7f3ee8159640 1 -- start start 2024-03-24T00:39:11.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.728+0000 7f023dffb640 1 -- 172.21.15.67:0/3773371621 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0244011710 con 0x7f024c0c8530 2024-03-24T00:39:11.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.728+0000 7f997d59a640 1 -- 172.21.15.67:0/4166009737 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996c039070 con 0x7f9964004dc0 2024-03-24T00:39:11.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.728+0000 7f45a65b0640 1 -- 172.21.15.67:0/718511418 shutdown_connections 2024-03-24T00:39:11.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.728+0000 7f45a65b0640 1 --2- 172.21.15.67:0/718511418 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f45a00d0cc0 0x7f45a00d27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.728+0000 7f45a65b0640 1 --2- 172.21.15.67:0/718511418 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f45a015df10 0x7f45a0161c50 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.728+0000 7f45a65b0640 1 -- 172.21.15.67:0/718511418 >> 172.21.15.67:0/718511418 conn(0x7f45a006c6b0 msgr2=0x7f45a006cac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:11.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.728+0000 7f023dffb640 1 -- 172.21.15.67:0/3773371621 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02440118b0 con 0x7f024c0c8530 2024-03-24T00:39:11.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.728+0000 7f997d59a640 1 -- 172.21.15.67:0/4166009737 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f996c02c890 con 0x7f9964004dc0 2024-03-24T00:39:11.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.728+0000 7f997d59a640 1 -- 172.21.15.67:0/4166009737 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996c039070 con 0x7f9964004dc0 2024-03-24T00:39:11.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.728+0000 7f1bb10d3640 1 --2- 172.21.15.67:0/2515221167 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1ba40e3210 0x7f1ba4053f50 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1bac064e00 tx=0x7f1bac06a6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e028830d86cc130 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.728+0000 7f976f5c4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f976814bb50 0x7f9768162930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.728+0000 7f976f5c4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9768072210 0x7f97680cddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.728+0000 7f976f5c4640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9768163030 con 0x7f9768166930 2024-03-24T00:39:11.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.728+0000 7f976f5c4640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97680ce330 con 0x7f9768072210 2024-03-24T00:39:11.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.728+0000 7f976f5c4640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9768163310 con 0x7f976814bb50 2024-03-24T00:39:11.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.728+0000 7f1b9ffff640 1 -- 172.21.15.67:0/2515221167 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bac0b4070 con 0x7f1ba40e3210 2024-03-24T00:39:11.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.728+0000 7f45a65b0640 1 -- 172.21.15.67:0/718511418 shutdown_connections 2024-03-24T00:39:11.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.728+0000 7f976d360640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f976814bb50 0x7f9768162930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.728+0000 7f976d360640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f976814bb50 0x7f9768162930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.67:49122/0 (socket says 172.21.15.67:49122) 2024-03-24T00:39:11.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.728+0000 7f976d360640 1 -- 172.21.15.67:0/2681551876 learned_addr learned my addr 172.21.15.67:0/2681551876 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:11.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.728+0000 7f976db61640 1 --2- 172.21.15.67:0/2681551876 >> v2:172.21.15.67:3300/0 conn(0x7f9768072210 0x7f97680cddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.728+0000 7f1b9ffff640 1 -- 172.21.15.67:0/2515221167 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bac0a7a30 con 0x7f1ba40e3210 2024-03-24T00:39:11.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.728+0000 7fdc2196b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc1c06e2a0 0x7fdc1c163710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.728+0000 7f45a65b0640 1 -- 172.21.15.67:0/718511418 wait complete. 2024-03-24T00:39:11.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.728+0000 7fdc2196b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc1c071d90 0x7fdc1c15dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.728+0000 7fdc20969640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc1c06e2a0 0x7fdc1c163710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.728+0000 7fdc20969640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc1c06e2a0 0x7fdc1c163710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:34178/0 (socket says 172.21.15.67:34178) 2024-03-24T00:39:11.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.729+0000 7fdfb197a640 1 --2- 172.21.15.67:0/1717811189 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdfa4054200 0x7fdfa4050740 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdfa80097d0 tx=0x7fdfa802b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=955d180750372d6c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.728+0000 7f1b9ffff640 1 -- 172.21.15.67:0/2515221167 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bac0af440 con 0x7f1ba40e3210 2024-03-24T00:39:11.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.729+0000 7f997f59e640 1 -- 172.21.15.67:0/4166009737 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9964004dc0 msgr2=0x7f99640051a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.729+0000 7f997f59e640 1 --2- 172.21.15.67:0/4166009737 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9964004dc0 0x7f99640051a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f996c007ee0 tx=0x7f9964005830 comp rx=0 tx=0).stop 2024-03-24T00:39:11.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.729+0000 7f997ed9d640 1 -- 172.21.15.67:0/4166009737 reap_dead start 2024-03-24T00:39:11.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.729+0000 7f02534c0640 1 -- 172.21.15.67:0/3773371621 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f024c0c8ca0 con 0x7f024c0c8530 2024-03-24T00:39:11.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.729+0000 7f997f59e640 1 -- 172.21.15.67:0/4166009737 shutdown_connections 2024-03-24T00:39:11.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.729+0000 7f997f59e640 1 -- 172.21.15.67:0/4166009737 >> 172.21.15.67:0/4166009737 conn(0x7f9980071d10 msgr2=0x7f9980072120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:11.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.729+0000 7fdc20969640 1 -- 172.21.15.67:0/777669300 learned_addr learned my addr 172.21.15.67:0/777669300 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:11.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.729+0000 7fdc2196b640 1 --2- 172.21.15.67:0/777669300 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdc1c162360 0x7fdc1c158650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.729+0000 7fdfb1179640 1 -- 172.21.15.67:0/1717811189 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa8035020 con 0x7fdfa4054200 2024-03-24T00:39:11.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.729+0000 7fdc2196b640 1 -- 172.21.15.67:0/777669300 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1c0b3c30 con 0x7fdc1c071d90 2024-03-24T00:39:11.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.729+0000 7f02534c0640 1 -- 172.21.15.67:0/3773371621 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f024c0c9260 con 0x7f024c0c8530 2024-03-24T00:39:11.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.729+0000 7fdfb1179640 1 -- 172.21.15.67:0/1717811189 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfa8031800 con 0x7fdfa4054200 2024-03-24T00:39:11.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.729+0000 7f3ee8159640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ee006f280 0x7f3ee0159730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.729+0000 7fdc2196b640 1 -- 172.21.15.67:0/777669300 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1c06f040 con 0x7fdc1c06e2a0 2024-03-24T00:39:11.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.729+0000 7fdc2196b640 1 -- 172.21.15.67:0/777669300 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1c0b3db0 con 0x7fdc1c162360 2024-03-24T00:39:11.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.729+0000 7f1bb3337640 1 -- 172.21.15.67:0/2515221167 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1ba40e3210 msgr2=0x7f1ba4053f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.729+0000 7f1bb3337640 1 --2- 172.21.15.67:0/2515221167 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1ba40e3210 0x7f1ba4053f50 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1bac064e00 tx=0x7f1bac06a6a0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.729+0000 7f3ee5ef5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ee006f280 0x7f3ee0159730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.730+0000 7f3ee5ef5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ee006f280 0x7f3ee0159730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32784/0 (socket says 172.21.15.67:32784) 2024-03-24T00:39:11.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.730+0000 7f3ee5ef5640 1 -- 172.21.15.67:0/3300752746 learned_addr learned my addr 172.21.15.67:0/3300752746 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:11.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.730+0000 7f3ee8159640 1 --2- 172.21.15.67:0/3300752746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3ee00cf020 0x7f3ee015c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.730+0000 7f997f59e640 1 -- 172.21.15.67:0/4166009737 shutdown_connections 2024-03-24T00:39:11.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.729+0000 7f976d360640 1 -- 172.21.15.67:0/2681551876 >> v2:172.21.15.67:3300/0 conn(0x7f9768072210 msgr2=0x7f97680cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:11.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.730+0000 7f976d360640 1 --2- 172.21.15.67:0/2681551876 >> v2:172.21.15.67:3300/0 conn(0x7f9768072210 0x7f97680cddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.730+0000 7f997f59e640 1 -- 172.21.15.67:0/4166009737 wait complete. 2024-03-24T00:39:11.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.730+0000 7f976d360640 1 -- 172.21.15.67:0/2681551876 >> v1:172.21.15.67:6789/0 conn(0x7f9768166930 legacy=0x7f9768071b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:11.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.730+0000 7f3ee8159640 1 --2- 172.21.15.67:0/3300752746 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3ee015e050 0x7f3ee00cb360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.730+0000 7f023dffb640 1 -- 172.21.15.67:0/3773371621 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0244011ca0 con 0x7f024c0c8530 2024-03-24T00:39:11.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.730+0000 7f976d360640 1 -- 172.21.15.67:0/2681551876 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97681635f0 con 0x7f976814bb50 2024-03-24T00:39:11.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.729+0000 7fdc1bfff640 1 --2- 172.21.15.67:0/777669300 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc1c071d90 0x7fdc1c15dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.730+0000 7f3ee56f4640 1 --2- 172.21.15.67:0/3300752746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3ee00cf020 0x7f3ee015c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.729+0000 7fdc2116a640 1 --2- 172.21.15.67:0/777669300 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdc1c162360 0x7fdc1c158650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.730+0000 7fdc2116a640 1 -- 172.21.15.67:0/777669300 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc1c06e2a0 msgr2=0x7fdc1c163710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.730+0000 7fdc2116a640 1 --2- 172.21.15.67:0/777669300 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc1c06e2a0 0x7fdc1c163710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.730+0000 7fdc2116a640 1 -- 172.21.15.67:0/777669300 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc1c071d90 msgr2=0x7fdc1c15dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.730+0000 7fdc2116a640 1 --2- 172.21.15.67:0/777669300 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc1c071d90 0x7fdc1c15dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.730+0000 7fdc2116a640 1 -- 172.21.15.67:0/777669300 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc1c162950 con 0x7fdc1c162360 2024-03-24T00:39:11.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.730+0000 7fdc1bfff640 1 --2- 172.21.15.67:0/777669300 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc1c071d90 0x7fdc1c15dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:11.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.730+0000 7f976db61640 1 --2- 172.21.15.67:0/2681551876 >> v2:172.21.15.67:3300/0 conn(0x7f9768072210 0x7f97680cddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:11.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.730+0000 7fdc20969640 1 --2- 172.21.15.67:0/777669300 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc1c06e2a0 0x7fdc1c163710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:11.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.730+0000 7fdc2116a640 1 --2- 172.21.15.67:0/777669300 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdc1c162360 0x7fdc1c158650 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdc1000d580 tx=0x7fdc1000da40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.730+0000 7f3ee8159640 1 -- 172.21.15.67:0/3300752746 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee0071780 con 0x7f3ee006f280 2024-03-24T00:39:11.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.730+0000 7f3ee8159640 1 -- 172.21.15.67:0/3300752746 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee00c68f0 con 0x7f3ee00cf020 2024-03-24T00:39:11.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.730+0000 7f3ee8159640 1 -- 172.21.15.67:0/3300752746 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee0071900 con 0x7f3ee015e050 2024-03-24T00:39:11.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.730+0000 7f45a65b0640 1 Processor -- start 2024-03-24T00:39:11.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.730+0000 7fdfb317d640 1 -- 172.21.15.67:0/1717811189 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdfa4054200 msgr2=0x7fdfa4050740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.730+0000 7fdfb317d640 1 --2- 172.21.15.67:0/1717811189 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdfa4054200 0x7fdfa4050740 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdfa80097d0 tx=0x7fdfa802b8a0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.730+0000 7f3ee66f6640 1 --2- 172.21.15.67:0/3300752746 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3ee015e050 0x7f3ee00cb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.731+0000 7f1bb3337640 1 -- 172.21.15.67:0/2515221167 shutdown_connections 2024-03-24T00:39:11.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.731+0000 7f1bb3337640 1 --2- 172.21.15.67:0/2515221167 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1ba40e3210 0x7f1ba4053f50 secure :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1bac064e00 tx=0x7f1bac06a6a0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.731+0000 7f1bb3337640 1 -- 172.21.15.67:0/2515221167 >> 172.21.15.67:0/2515221167 conn(0x7f1ba4058f20 msgr2=0x7f1ba4059330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:11.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.730+0000 7fdc197fa640 1 -- 172.21.15.67:0/777669300 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc10018020 con 0x7fdc1c162360 2024-03-24T00:39:11.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.731+0000 7fdc197fa640 1 -- 172.21.15.67:0/777669300 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc10014620 con 0x7fdc1c162360 2024-03-24T00:39:11.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.731+0000 7fdc197fa640 1 -- 172.21.15.67:0/777669300 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc10014a10 con 0x7fdc1c162360 2024-03-24T00:39:11.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.731+0000 7f3ee66f6640 1 -- 172.21.15.67:0/3300752746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3ee00cf020 msgr2=0x7f3ee015c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.731+0000 7fdc2196b640 1 -- 172.21.15.67:0/777669300 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc1c162c30 con 0x7fdc1c162360 2024-03-24T00:39:11.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.731+0000 7fdc2196b640 1 -- 172.21.15.67:0/777669300 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc1c15e2e0 con 0x7fdc1c162360 2024-03-24T00:39:11.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.731+0000 7f3ee66f6640 1 --2- 172.21.15.67:0/3300752746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3ee00cf020 0x7f3ee015c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.731+0000 7f3ee66f6640 1 -- 172.21.15.67:0/3300752746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ee006f280 msgr2=0x7f3ee0159730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.730+0000 7f45a65b0640 1 -- start start 2024-03-24T00:39:11.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.731+0000 7f976d360640 1 --2- 172.21.15.67:0/2681551876 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f976814bb50 0x7f9768162930 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9764009870 tx=0x7f976402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=268afaf9c29ae374 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.731+0000 7f975ffff640 1 -- 172.21.15.67:0/2681551876 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9764035020 con 0x7f976814bb50 2024-03-24T00:39:11.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.731+0000 7f975ffff640 1 -- 172.21.15.67:0/2681551876 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9764031640 con 0x7f976814bb50 2024-03-24T00:39:11.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.731+0000 7f975ffff640 1 -- 172.21.15.67:0/2681551876 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f976402d040 con 0x7f976814bb50 2024-03-24T00:39:11.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.731+0000 7f02534c0640 1 -- 172.21.15.67:0/3773371621 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f024c04f400 con 0x7f024c0c8530 2024-03-24T00:39:11.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.732+0000 7f3ee66f6640 1 --2- 172.21.15.67:0/3300752746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ee006f280 0x7f3ee0159730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.732+0000 7f3ee66f6640 1 -- 172.21.15.67:0/3300752746 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ee00ca2e0 con 0x7f3ee015e050 2024-03-24T00:39:11.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.732+0000 7f3ee5ef5640 1 --2- 172.21.15.67:0/3300752746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ee006f280 0x7f3ee0159730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:11.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.732+0000 7f023dffb640 1 -- 172.21.15.67:0/3773371621 <== mon.2 v2:172.21.15.72:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f024401c400 con 0x7f024c0c8530 2024-03-24T00:39:11.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.733+0000 7f023dffb640 1 --2- 172.21.15.67:0/3773371621 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f023006bc50 0x7f023006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.732+0000 7f45a65b0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45a00b5780 0x7f45a00cd150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.732+0000 7f45a65b0640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f45a00d0cc0 0x7f45a015cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.732+0000 7f45a65b0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f45a015df10 0x7f45a0160890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.732+0000 7f45a65b0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45a0162190 con 0x7f45a00b5780 2024-03-24T00:39:11.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.732+0000 7f45a65b0640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45a00d2290 con 0x7f45a015df10 2024-03-24T00:39:11.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.732+0000 7f45a65b0640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45a0162310 con 0x7f45a00d0cc0 2024-03-24T00:39:11.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.732+0000 7f45a4dad640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f45a00d0cc0 0x7f45a015cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.732+0000 7f45a4dad640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f45a00d0cc0 0x7f45a015cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.67:49156/0 (socket says 172.21.15.67:49156) 2024-03-24T00:39:11.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.732+0000 7f45a4dad640 1 -- 172.21.15.67:0/2024318973 learned_addr learned my addr 172.21.15.67:0/2024318973 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:11.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.732+0000 7f45a5daf640 1 --2- 172.21.15.67:0/2024318973 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f45a015df10 0x7f45a0160890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.733+0000 7f45a55ae640 1 --2- 172.21.15.67:0/2024318973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45a00b5780 0x7f45a00cd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.732+0000 7f997f59e640 1 Processor -- start 2024-03-24T00:39:11.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.732+0000 7f997f59e640 1 -- start start 2024-03-24T00:39:11.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.733+0000 7f997f59e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f998013c8c0 0x7f9980200ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.732+0000 7fdfb317d640 1 -- 172.21.15.67:0/1717811189 shutdown_connections 2024-03-24T00:39:11.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.732+0000 7fdfb317d640 1 --2- 172.21.15.67:0/1717811189 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdfa4054200 0x7fdfa4050740 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.732+0000 7fdfb317d640 1 -- 172.21.15.67:0/1717811189 >> 172.21.15.67:0/1717811189 conn(0x7fdfa4009fb0 msgr2=0x7fdfa400a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:11.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.732+0000 7fdfb317d640 1 -- 172.21.15.67:0/1717811189 shutdown_connections 2024-03-24T00:39:11.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.732+0000 7f1bb3337640 1 -- 172.21.15.67:0/2515221167 shutdown_connections 2024-03-24T00:39:11.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.733+0000 7f997f59e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f998013cca0 0x7f9980159960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.733+0000 7f997f59e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9980155910 0x7f9980159ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.733+0000 7f023dffb640 1 -- 172.21.15.67:0/3773371621 <== mon.2 v2:172.21.15.72:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f024408ba10 con 0x7f024c0c8530 2024-03-24T00:39:11.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.733+0000 7f997f59e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f998006f500 con 0x7f9980155910 2024-03-24T00:39:11.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.733+0000 7f997f59e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f998006f380 con 0x7f998013cca0 2024-03-24T00:39:11.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.733+0000 7f997f59e640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f998006f680 con 0x7f998013c8c0 2024-03-24T00:39:11.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.733+0000 7f997dd9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f998013cca0 0x7f9980159960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.732+0000 7fdfb317d640 1 -- 172.21.15.67:0/1717811189 wait complete. 2024-03-24T00:39:11.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.733+0000 7f997dd9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f998013cca0 0x7f9980159960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:34188/0 (socket says 172.21.15.67:34188) 2024-03-24T00:39:11.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.733+0000 7f997dd9b640 1 -- 172.21.15.67:0/2324264500 learned_addr learned my addr 172.21.15.67:0/2324264500 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:11.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.733+0000 7f1bb3337640 1 -- 172.21.15.67:0/2515221167 wait complete. 2024-03-24T00:39:11.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.732+0000 7f3ee66f6640 1 --2- 172.21.15.67:0/3300752746 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3ee015e050 0x7f3ee00cb360 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3edc032000 tx=0x7f3edc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.733+0000 7f997ed9d640 1 --2- 172.21.15.67:0/2324264500 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9980155910 0x7f9980159ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.733+0000 7f997e59c640 1 --2- 172.21.15.67:0/2324264500 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f998013c8c0 0x7f9980200ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.733+0000 7f025125c640 1 --2- 172.21.15.67:0/3773371621 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f023006bc50 0x7f023006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.733+0000 7f997dd9b640 1 -- 172.21.15.67:0/2324264500 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f998013c8c0 msgr2=0x7f9980200ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.732+0000 7fdc2196b640 1 -- 172.21.15.67:0/777669300 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc1c0751c0 con 0x7fdc1c162360 2024-03-24T00:39:11.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.732+0000 7fdc197fa640 1 -- 172.21.15.67:0/777669300 <== mon.2 v2:172.21.15.72:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fdc1001b020 con 0x7fdc1c162360 2024-03-24T00:39:11.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.733+0000 7fdc197fa640 1 --2- 172.21.15.67:0/777669300 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdc0806bca0 0x7fdc0806e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.733+0000 7fdc20969640 1 --2- 172.21.15.67:0/777669300 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdc0806bca0 0x7fdc0806e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.733+0000 7fdc197fa640 1 -- 172.21.15.67:0/777669300 <== mon.2 v2:172.21.15.72:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fdc1008ae10 con 0x7fdc1c162360 2024-03-24T00:39:11.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.733+0000 7f976f5c4640 1 -- 172.21.15.67:0/2681551876 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f976814bb50 msgr2=0x7f9768162930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.733+0000 7f976f5c4640 1 --2- 172.21.15.67:0/2681551876 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f976814bb50 0x7f9768162930 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9764009870 tx=0x7f976402b8f0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.733+0000 7f976f5c4640 1 -- 172.21.15.67:0/2681551876 shutdown_connections 2024-03-24T00:39:11.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.733+0000 7f976f5c4640 1 --2- 172.21.15.67:0/2681551876 >> v2:172.21.15.67:3300/0 conn(0x7f9768072210 0x7f97680cddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.733+0000 7f976f5c4640 1 --2- 172.21.15.67:0/2681551876 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f976814bb50 0x7f9768162930 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.733+0000 7f976f5c4640 1 -- 172.21.15.67:0/2681551876 >> 172.21.15.67:0/2681551876 conn(0x7f9768067470 msgr2=0x7f976806b040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:11.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.733+0000 7f976f5c4640 1 -- 172.21.15.67:0/2681551876 shutdown_connections 2024-03-24T00:39:11.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.733+0000 7f976f5c4640 1 -- 172.21.15.67:0/2681551876 wait complete. 2024-03-24T00:39:11.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.733+0000 7f997dd9b640 1 --2- 172.21.15.67:0/2324264500 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f998013c8c0 0x7f9980200ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.733+0000 7f45a4dad640 1 -- 172.21.15.67:0/2024318973 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f45a015df10 msgr2=0x7f45a0160890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.733+0000 7f45a4dad640 1 --2- 172.21.15.67:0/2024318973 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f45a015df10 0x7f45a0160890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.733+0000 7f45a4dad640 1 -- 172.21.15.67:0/2024318973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45a00b5780 msgr2=0x7f45a00cd150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.734+0000 7fdc20969640 1 --2- 172.21.15.67:0/777669300 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdc0806bca0 0x7fdc0806e160 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c1604e0 tx=0x7fdc14030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.733+0000 7f997dd9b640 1 -- 172.21.15.67:0/2324264500 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9980155910 msgr2=0x7f9980159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.733+0000 7f997dd9b640 1 --2- 172.21.15.67:0/2324264500 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9980155910 0x7f9980159ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.733+0000 7f997dd9b640 1 -- 172.21.15.67:0/2324264500 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9980201820 con 0x7f998013cca0 2024-03-24T00:39:11.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.734+0000 7f997e59c640 1 --2- 172.21.15.67:0/2324264500 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f998013c8c0 0x7f9980200ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:39:11.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.734+0000 7f45a4dad640 1 --2- 172.21.15.67:0/2024318973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45a00b5780 0x7f45a00cd150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.734+0000 7f45a4dad640 1 -- 172.21.15.67:0/2024318973 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a01f8ef0 con 0x7f45a00d0cc0 2024-03-24T00:39:11.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.734+0000 7f45a55ae640 1 --2- 172.21.15.67:0/2024318973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45a00b5780 0x7f45a00cd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:11.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.734+0000 7f997dd9b640 1 --2- 172.21.15.67:0/2324264500 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f998013cca0 0x7f9980159960 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f99700017e0 tx=0x7f997000f5a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.734+0000 7f9976ffd640 1 -- 172.21.15.67:0/2324264500 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f997001a070 con 0x7f998013cca0 2024-03-24T00:39:11.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.734+0000 7f45a4dad640 1 --2- 172.21.15.67:0/2024318973 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f45a00d0cc0 0x7f45a015cad0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f459800cbc0 tx=0x7f459800a430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.734+0000 7f9976ffd640 1 -- 172.21.15.67:0/2324264500 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9970012350 con 0x7f998013cca0 2024-03-24T00:39:11.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.734+0000 7f9976ffd640 1 -- 172.21.15.67:0/2324264500 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9970012540 con 0x7f998013cca0 2024-03-24T00:39:11.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.734+0000 7f997f59e640 1 -- 172.21.15.67:0/2324264500 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9980201b00 con 0x7f998013cca0 2024-03-24T00:39:11.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.734+0000 7f997f59e640 1 -- 172.21.15.67:0/2324264500 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9980201530 con 0x7f998013cca0 2024-03-24T00:39:11.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.734+0000 7fdfb317d640 1 Processor -- start 2024-03-24T00:39:11.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.734+0000 7fdfb317d640 1 -- start start 2024-03-24T00:39:11.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.734+0000 7f025125c640 1 --2- 172.21.15.67:0/3773371621 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f023006bc50 0x7f023006e110 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0248006fd0 tx=0x7f0248008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.734+0000 7f1bb3337640 1 Processor -- start 2024-03-24T00:39:11.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.735+0000 7f976f5c4640 1 Processor -- start 2024-03-24T00:39:11.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.735+0000 7f1bb3337640 1 -- start start 2024-03-24T00:39:11.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.735+0000 7f976f5c4640 1 -- start start 2024-03-24T00:39:11.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.735+0000 7f1bb3337640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1ba4006220 0x7f1ba4181a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.735+0000 7f1bb3337640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ba4006640 0x7f1ba40520c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.735+0000 7f1bb3337640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1ba40e4f50 0x7f1ba40e5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.735+0000 7f997f59e640 1 -- 172.21.15.67:0/2324264500 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99802021b0 con 0x7f998013cca0 2024-03-24T00:39:11.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.735+0000 7f1bb3337640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba40588d0 con 0x7f1ba4006640 2024-03-24T00:39:11.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.735+0000 7f1bb3337640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba4057ca0 con 0x7f1ba4006220 2024-03-24T00:39:11.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.735+0000 7f1bb3337640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba4058a50 con 0x7f1ba40e4f50 2024-03-24T00:39:11.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.736+0000 7fdfb317d640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdfa4006770 0x7fdfa40485d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.736+0000 7fdfb317d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfa4054200 0x7fdfa40d8220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.735+0000 7f1bb10d3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1ba4006220 0x7f1ba4181a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.736+0000 7f1bb08d2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ba4006640 0x7f1ba40520c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.735+0000 7f1bb10d3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1ba4006220 0x7f1ba4181a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:34192/0 (socket says 172.21.15.67:34192) 2024-03-24T00:39:11.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.736+0000 7f1bb10d3640 1 -- 172.21.15.67:0/4281148724 learned_addr learned my addr 172.21.15.67:0/4281148724 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:11.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.736+0000 7f1bb18d4640 1 --2- 172.21.15.67:0/4281148724 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1ba40e4f50 0x7f1ba40e5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.736+0000 7fdfb317d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdfa40df800 0x7fdfa4039530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.736+0000 7fdfb317d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa4054850 con 0x7fdfa4054200 2024-03-24T00:39:11.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.736+0000 7fdfb317d640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa4052c20 con 0x7fdfa40df800 2024-03-24T00:39:11.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.736+0000 7fdfb317d640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa4053f60 con 0x7fdfa4006770 2024-03-24T00:39:11.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.736+0000 7f976f5c4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9768072210 0x7f976815e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.736+0000 7f1bb10d3640 1 -- 172.21.15.67:0/4281148724 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1ba40e4f50 msgr2=0x7f1ba40e5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.736+0000 7f1bb10d3640 1 --2- 172.21.15.67:0/4281148724 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1ba40e4f50 0x7f1ba40e5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.736+0000 7fdfb297c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdfa40df800 0x7fdfa4039530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.736+0000 7fdfb197a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfa4054200 0x7fdfa40d8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.736+0000 7fdfb297c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdfa40df800 0x7fdfa4039530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:34202/0 (socket says 172.21.15.67:34202) 2024-03-24T00:39:11.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.736+0000 7fdfb197a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfa4054200 0x7fdfa40d8220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32802/0 (socket says 172.21.15.67:32802) 2024-03-24T00:39:11.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.736+0000 7fdfb297c640 1 -- 172.21.15.67:0/3008509033 learned_addr learned my addr 172.21.15.67:0/3008509033 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:11.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.736+0000 7f976f5c4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f976814bb50 0x7f976813e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.736+0000 7f1bb10d3640 1 -- 172.21.15.67:0/4281148724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ba4006640 msgr2=0x7f1ba40520c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.736+0000 7f1bb10d3640 1 --2- 172.21.15.67:0/4281148724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ba4006640 0x7f1ba40520c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.736+0000 7f1bb10d3640 1 -- 172.21.15.67:0/4281148724 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ba40dea30 con 0x7f1ba4006220 2024-03-24T00:39:11.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.736+0000 7f1bb18d4640 1 --2- 172.21.15.67:0/4281148724 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1ba40e4f50 0x7f1ba40e5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:11.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.736+0000 7fdfb217b640 1 --2- 172.21.15.67:0/3008509033 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdfa4006770 0x7fdfa40485d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.736+0000 7f976d360640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9768072210 0x7f976815e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.736+0000 7f976f5c4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9768166930 0x7f9768163a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.736+0000 7f976f5c4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f976814bf30 con 0x7f9768166930 2024-03-24T00:39:11.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.736+0000 7f976f5c4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9768166d70 con 0x7f9768072210 2024-03-24T00:39:11.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.736+0000 7f976f5c4640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97680b3190 con 0x7f976814bb50 2024-03-24T00:39:11.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.736+0000 7f976d360640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9768072210 0x7f976815e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:34212/0 (socket says 172.21.15.67:34212) 2024-03-24T00:39:11.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.736+0000 7f976d360640 1 -- 172.21.15.67:0/4205714727 learned_addr learned my addr 172.21.15.67:0/4205714727 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:11.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.736+0000 7f976cb5f640 1 --2- 172.21.15.67:0/4205714727 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f976814bb50 0x7f976813e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.736+0000 7f976db61640 1 --2- 172.21.15.67:0/4205714727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9768166930 0x7f9768163a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.736+0000 7f1bb10d3640 1 --2- 172.21.15.67:0/4281148724 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1ba4006220 0x7f1ba4181a10 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1bac075f70 tx=0x7f1bac069920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.736+0000 7fdfb297c640 1 -- 172.21.15.67:0/3008509033 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdfa4006770 msgr2=0x7fdfa40485d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.736+0000 7fdfb297c640 1 --2- 172.21.15.67:0/3008509033 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdfa4006770 0x7fdfa40485d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.736+0000 7fdfb297c640 1 -- 172.21.15.67:0/3008509033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfa4054200 msgr2=0x7fdfa40d8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.736+0000 7fdfb297c640 1 --2- 172.21.15.67:0/3008509033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfa4054200 0x7fdfa40d8220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.736+0000 7fdfb297c640 1 -- 172.21.15.67:0/3008509033 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa417a320 con 0x7fdfa40df800 2024-03-24T00:39:11.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.736+0000 7fdfb197a640 1 --2- 172.21.15.67:0/3008509033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfa4054200 0x7fdfa40d8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:11.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.737+0000 7f9976ffd640 1 -- 172.21.15.67:0/2324264500 <== mon.1 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f99700126e0 con 0x7f998013cca0 2024-03-24T00:39:11.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.737+0000 7f976d360640 1 -- 172.21.15.67:0/4205714727 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f976814bb50 msgr2=0x7f976813e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.737+0000 7f976d360640 1 --2- 172.21.15.67:0/4205714727 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f976814bb50 0x7f976813e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.737+0000 7fdfb297c640 1 --2- 172.21.15.67:0/3008509033 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdfa40df800 0x7fdfa4039530 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fdfac0691e0 tx=0x7fdfac068b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.737+0000 7f976d360640 1 -- 172.21.15.67:0/4205714727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9768166930 msgr2=0x7f9768163a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.737+0000 7f976d360640 1 --2- 172.21.15.67:0/4205714727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9768166930 0x7f9768163a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.737+0000 7f976d360640 1 -- 172.21.15.67:0/4205714727 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9768164440 con 0x7f9768072210 2024-03-24T00:39:11.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.737+0000 7f976cb5f640 1 --2- 172.21.15.67:0/4205714727 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f976814bb50 0x7f976813e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:11.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.737+0000 7f976db61640 1 --2- 172.21.15.67:0/4205714727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9768166930 0x7f9768163a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:11.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.737+0000 7f1b9dffb640 1 -- 172.21.15.67:0/4281148724 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bac0be070 con 0x7f1ba4006220 2024-03-24T00:39:11.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.737+0000 7f1b9dffb640 1 -- 172.21.15.67:0/4281148724 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bac0b9660 con 0x7f1ba4006220 2024-03-24T00:39:11.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.737+0000 7f1b9dffb640 1 -- 172.21.15.67:0/4281148724 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bac0b4040 con 0x7f1ba4006220 2024-03-24T00:39:11.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.737+0000 7f1bb3337640 1 -- 172.21.15.67:0/4281148724 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ba417de60 con 0x7f1ba4006220 2024-03-24T00:39:11.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.737+0000 7f9976ffd640 1 --2- 172.21.15.67:0/2324264500 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f996806bc00 0x7f996806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.737+0000 7f9976ffd640 1 -- 172.21.15.67:0/2324264500 <== mon.1 v2:172.21.15.72:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f997008aa80 con 0x7f998013cca0 2024-03-24T00:39:11.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.737+0000 7f976d360640 1 --2- 172.21.15.67:0/4205714727 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9768072210 0x7f976815e220 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9764009870 tx=0x7f97640046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.737+0000 7f1bb3337640 1 -- 172.21.15.67:0/4281148724 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ba417e420 con 0x7f1ba4006220 2024-03-24T00:39:11.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.737+0000 7f997e59c640 1 --2- 172.21.15.67:0/2324264500 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f996806bc00 0x7f996806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.737+0000 7f4595ffb640 1 -- 172.21.15.67:0/2024318973 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4598017070 con 0x7f45a00d0cc0 2024-03-24T00:39:11.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.738+0000 7f3ed67fc640 1 -- 172.21.15.67:0/3300752746 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3edc042070 con 0x7f3ee015e050 2024-03-24T00:39:11.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.738+0000 7f3ee8159640 1 -- 172.21.15.67:0/3300752746 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ee00ca8a0 con 0x7f3ee015e050 2024-03-24T00:39:11.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.738+0000 7f45a65b0640 1 -- 172.21.15.67:0/2024318973 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45a00cc390 con 0x7f45a00d0cc0 2024-03-24T00:39:11.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.738+0000 7f023dffb640 1 -- 172.21.15.67:0/3773371621 <== mon.2 v2:172.21.15.72:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0244059b60 con 0x7f024c0c8530 2024-03-24T00:39:11.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.738+0000 7f3ee8159640 1 -- 172.21.15.67:0/3300752746 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ee00cae60 con 0x7f3ee015e050 2024-03-24T00:39:11.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.738+0000 7f45a65b0640 1 -- 172.21.15.67:0/2024318973 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45a00cc950 con 0x7f45a00d0cc0 2024-03-24T00:39:11.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.738+0000 7f997e59c640 1 --2- 172.21.15.67:0/2324264500 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f996806bc00 0x7f996806e0c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f996c007eb0 tx=0x7f996c009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.738+0000 7f1bb3337640 1 -- 172.21.15.67:0/4281148724 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ba4005740 con 0x7f1ba4006220 2024-03-24T00:39:11.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.739+0000 7f3ee8159640 1 -- 172.21.15.67:0/3300752746 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ee00751c0 con 0x7f3ee015e050 2024-03-24T00:39:11.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.739+0000 7f45a65b0640 1 -- 172.21.15.67:0/2024318973 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45a00751c0 con 0x7f45a00d0cc0 2024-03-24T00:39:11.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.741+0000 7fdfa2ffd640 1 -- 172.21.15.67:0/3008509033 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfac04f970 con 0x7fdfa40df800 2024-03-24T00:39:11.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.741+0000 7f975dffb640 1 -- 172.21.15.67:0/4205714727 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9764040070 con 0x7f9768072210 2024-03-24T00:39:11.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.741+0000 7fdfb317d640 1 -- 172.21.15.67:0/3008509033 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfa417a600 con 0x7fdfa40df800 2024-03-24T00:39:11.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.742+0000 7f9976ffd640 1 -- 172.21.15.67:0/2324264500 <== mon.1 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9970007bf0 con 0x7f998013cca0 2024-03-24T00:39:11.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.742+0000 7fdfa2ffd640 1 -- 172.21.15.67:0/3008509033 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfac074110 con 0x7fdfa40df800 2024-03-24T00:39:11.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.742+0000 7fdfb317d640 1 -- 172.21.15.67:0/3008509033 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfa417abc0 con 0x7fdfa40df800 2024-03-24T00:39:11.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.742+0000 7fdfa2ffd640 1 -- 172.21.15.67:0/3008509033 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfac0935f0 con 0x7fdfa40df800 2024-03-24T00:39:11.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.742+0000 7f975dffb640 1 -- 172.21.15.67:0/4205714727 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97640399f0 con 0x7f9768072210 2024-03-24T00:39:11.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.742+0000 7f975dffb640 1 -- 172.21.15.67:0/4205714727 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9764035040 con 0x7f9768072210 2024-03-24T00:39:11.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.742+0000 7f3ed67fc640 1 -- 172.21.15.67:0/3300752746 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3edc030510 con 0x7f3ee015e050 2024-03-24T00:39:11.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.742+0000 7f3ed67fc640 1 -- 172.21.15.67:0/3300752746 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3edc04c060 con 0x7f3ee015e050 2024-03-24T00:39:11.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.742+0000 7fdc197fa640 1 -- 172.21.15.67:0/777669300 <== mon.2 v2:172.21.15.72:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fdc10022030 con 0x7fdc1c162360 2024-03-24T00:39:11.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.742+0000 7f4595ffb640 1 -- 172.21.15.67:0/2024318973 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f459800ad40 con 0x7f45a00d0cc0 2024-03-24T00:39:11.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.742+0000 7f4595ffb640 1 -- 172.21.15.67:0/2024318973 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4598021060 con 0x7f45a00d0cc0 2024-03-24T00:39:11.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.743+0000 7f1b9dffb640 1 -- 172.21.15.67:0/4281148724 <== mon.1 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1bac0c1070 con 0x7f1ba4006220 2024-03-24T00:39:11.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.743+0000 7fdfa0ff9640 1 -- 172.21.15.67:0/3008509033 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfa40052c0 con 0x7fdfa40df800 2024-03-24T00:39:11.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.744+0000 7f976f5c4640 1 -- 172.21.15.67:0/4205714727 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9768164720 con 0x7f9768072210 2024-03-24T00:39:11.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.744+0000 7f1b9dffb640 1 --2- 172.21.15.67:0/4281148724 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1b9006bca0 0x7f1b9006e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.744+0000 7f1b9dffb640 1 -- 172.21.15.67:0/4281148724 <== mon.1 v2:172.21.15.72:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f1bac128350 con 0x7f1ba4006220 2024-03-24T00:39:11.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.744+0000 7f3ed67fc640 1 -- 172.21.15.67:0/3300752746 <== mon.2 v2:172.21.15.72:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3edc045070 con 0x7f3ee015e050 2024-03-24T00:39:11.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.744+0000 7f1bb08d2640 1 --2- 172.21.15.67:0/4281148724 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1b9006bca0 0x7f1b9006e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.744+0000 7f976f5c4640 1 -- 172.21.15.67:0/4205714727 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9768141740 con 0x7f9768072210 2024-03-24T00:39:11.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.744+0000 7f4595ffb640 1 -- 172.21.15.67:0/2024318973 <== mon.2 v2:172.21.15.72:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f459801a070 con 0x7f45a00d0cc0 2024-03-24T00:39:11.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.744+0000 7f3ed67fc640 1 --2- 172.21.15.67:0/3300752746 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3ec406bc00 0x7f3ec406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.744+0000 7f1bb08d2640 1 --2- 172.21.15.67:0/4281148724 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1b9006bca0 0x7f1b9006e160 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1ba416d460 tx=0x7f1ba0007e00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.744+0000 7f3ed67fc640 1 -- 172.21.15.67:0/3300752746 <== mon.2 v2:172.21.15.72:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f3edc0acdc0 con 0x7f3ee015e050 2024-03-24T00:39:11.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.744+0000 7f3ee5ef5640 1 --2- 172.21.15.67:0/3300752746 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3ec406bc00 0x7f3ec406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.745+0000 7f4595ffb640 1 --2- 172.21.15.67:0/2024318973 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f458406bbd0 0x7f458406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.745+0000 7f4595ffb640 1 -- 172.21.15.67:0/2024318973 <== mon.2 v2:172.21.15.72:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f459808b2a0 con 0x7f45a00d0cc0 2024-03-24T00:39:11.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.745+0000 7f45a55ae640 1 --2- 172.21.15.67:0/2024318973 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f458406bbd0 0x7f458406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.745+0000 7f3ee5ef5640 1 --2- 172.21.15.67:0/3300752746 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3ec406bc00 0x7f3ec406e0c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3ed80012e0 tx=0x7f3ed8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.745+0000 7f976f5c4640 1 -- 172.21.15.67:0/4205714727 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f976804f400 con 0x7f9768072210 2024-03-24T00:39:11.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.745+0000 7f45a55ae640 1 --2- 172.21.15.67:0/2024318973 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f458406bbd0 0x7f458406e090 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4590036000 tx=0x7f4590008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.746+0000 7f1b9dffb640 1 -- 172.21.15.67:0/4281148724 <== mon.1 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1bac0c8030 con 0x7f1ba4006220 2024-03-24T00:39:11.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.747+0000 7f3ed67fc640 1 -- 172.21.15.67:0/3300752746 <== mon.2 v2:172.21.15.72:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3edc07ae70 con 0x7f3ee015e050 2024-03-24T00:39:11.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.748+0000 7fdfa2ffd640 1 -- 172.21.15.67:0/3008509033 <== mon.1 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fdfac093790 con 0x7fdfa40df800 2024-03-24T00:39:11.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.748+0000 7f975dffb640 1 -- 172.21.15.67:0/4205714727 <== mon.1 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f976404b060 con 0x7f9768072210 2024-03-24T00:39:11.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.748+0000 7fdfa2ffd640 1 --2- 172.21.15.67:0/3008509033 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdf9006bbd0 0x7fdf9006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.748+0000 7fdfa2ffd640 1 -- 172.21.15.67:0/3008509033 <== mon.1 v2:172.21.15.72:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fdfac106780 con 0x7fdfa40df800 2024-03-24T00:39:11.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.748+0000 7fdfb217b640 1 --2- 172.21.15.67:0/3008509033 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdf9006bbd0 0x7fdf9006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.749+0000 7fdfb217b640 1 --2- 172.21.15.67:0/3008509033 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdf9006bbd0 0x7fdf9006e090 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c005880 tx=0x7fdf9c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.749+0000 7f975dffb640 1 --2- 172.21.15.67:0/4205714727 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f974c06bc00 0x7f974c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.749+0000 7f975dffb640 1 -- 172.21.15.67:0/4205714727 <== mon.1 v2:172.21.15.72:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f97640ac8a0 con 0x7f9768072210 2024-03-24T00:39:11.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.749+0000 7f976cb5f640 1 --2- 172.21.15.67:0/4205714727 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f974c06bc00 0x7f974c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.750+0000 7f976cb5f640 1 --2- 172.21.15.67:0/4205714727 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f974c06bc00 0x7f974c06e0c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f97681622c0 tx=0x7f9758009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.752+0000 7fdfa2ffd640 1 -- 172.21.15.67:0/3008509033 <== mon.1 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fdfac0d4950 con 0x7fdfa40df800 2024-03-24T00:39:11.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.752+0000 7f4595ffb640 1 -- 172.21.15.67:0/2024318973 <== mon.2 v2:172.21.15.72:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f45a00b6f00 con 0x7f45a00d0cc0 2024-03-24T00:39:11.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:11.756+0000 7f975dffb640 1 -- 172.21.15.67:0/4205714727 <== mon.1 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f976407b0c0 con 0x7f9768072210 2024-03-24T00:39:12.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.156+0000 7f02534c0640 1 -- 172.21.15.67:0/3773371621 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f024c0cd610 con 0x7f024c0c8530 2024-03-24T00:39:12.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.157+0000 7f023dffb640 1 -- 172.21.15.67:0/3773371621 <== mon.2 v2:172.21.15.72:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f024405dc00 con 0x7f024c0c8530 2024-03-24T00:39:12.157 INFO:teuthology.orchestra.run.smithi067.stdout:55834574856 2024-03-24T00:39:12.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.160+0000 7f022f7fe640 1 -- 172.21.15.67:0/3773371621 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f023006bc50 msgr2=0x7f023006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:12.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.160+0000 7f022f7fe640 1 --2- 172.21.15.67:0/3773371621 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f023006bc50 0x7f023006e110 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0248006fd0 tx=0x7f0248008040 comp rx=0 tx=0).stop 2024-03-24T00:39:12.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.161+0000 7f022f7fe640 1 -- 172.21.15.67:0/3773371621 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f024c0c8530 msgr2=0x7f024c0c9fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:12.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.161+0000 7f022f7fe640 1 --2- 172.21.15.67:0/3773371621 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f024c0c8530 0x7f024c0c9fb0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f024400be10 tx=0x7f024400bf00 comp rx=0 tx=0).stop 2024-03-24T00:39:12.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.161+0000 7f022f7fe640 1 -- 172.21.15.67:0/3773371621 shutdown_connections 2024-03-24T00:39:12.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.161+0000 7f022f7fe640 1 --2- 172.21.15.67:0/3773371621 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f023006bc50 0x7f023006e110 secure :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0248006fd0 tx=0x7f0248008040 comp rx=0 tx=0).stop 2024-03-24T00:39:12.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.161+0000 7f022f7fe640 1 --2- 172.21.15.67:0/3773371621 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f024c0c8530 0x7f024c0c9fb0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.161+0000 7f022f7fe640 1 --2- 172.21.15.67:0/3773371621 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f024c161cc0 0x7f024c0b4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.161+0000 7f022f7fe640 1 --2- 172.21.15.67:0/3773371621 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f024c0cc5c0 0x7f024c1f7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.161+0000 7f022f7fe640 1 -- 172.21.15.67:0/3773371621 >> 172.21.15.67:0/3773371621 conn(0x7f024c099a30 msgr2=0x7f024c0afa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:12.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.161+0000 7f022f7fe640 1 -- 172.21.15.67:0/3773371621 shutdown_connections 2024-03-24T00:39:12.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.161+0000 7f022f7fe640 1 -- 172.21.15.67:0/3773371621 wait complete. 2024-03-24T00:39:12.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.168+0000 7f45a65b0640 1 -- 172.21.15.67:0/2024318973 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f45a00d19a0 con 0x7f45a00d0cc0 2024-03-24T00:39:12.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.169+0000 7f4595ffb640 1 -- 172.21.15.67:0/2024318973 <== mon.2 v2:172.21.15.72:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4598059570 con 0x7f45a00d0cc0 2024-03-24T00:39:12.169 INFO:teuthology.orchestra.run.smithi067.stdout:55834574856 2024-03-24T00:39:12.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.170+0000 7f997f59e640 1 -- 172.21.15.67:0/2324264500 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f99800b5670 con 0x7f998013cca0 2024-03-24T00:39:12.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.170+0000 7f9976ffd640 1 -- 172.21.15.67:0/2324264500 <== mon.1 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9970058b60 con 0x7f998013cca0 2024-03-24T00:39:12.171 INFO:teuthology.orchestra.run.smithi067.stdout:55834574857 2024-03-24T00:39:12.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.172+0000 7f45a65b0640 1 -- 172.21.15.67:0/2024318973 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f458406bbd0 msgr2=0x7f458406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:12.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.172+0000 7f45a65b0640 1 --2- 172.21.15.67:0/2024318973 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f458406bbd0 0x7f458406e090 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4590036000 tx=0x7f4590008030 comp rx=0 tx=0).stop 2024-03-24T00:39:12.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.172+0000 7f45a65b0640 1 -- 172.21.15.67:0/2024318973 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f45a00d0cc0 msgr2=0x7f45a015cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:12.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.172+0000 7f45a65b0640 1 --2- 172.21.15.67:0/2024318973 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f45a00d0cc0 0x7f45a015cad0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f459800cbc0 tx=0x7f459800a430 comp rx=0 tx=0).stop 2024-03-24T00:39:12.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.173+0000 7f45a65b0640 1 -- 172.21.15.67:0/2024318973 shutdown_connections 2024-03-24T00:39:12.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.173+0000 7f45a65b0640 1 --2- 172.21.15.67:0/2024318973 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f458406bbd0 0x7f458406e090 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.173+0000 7f45a65b0640 1 --2- 172.21.15.67:0/2024318973 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f45a015df10 0x7f45a0160890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.173+0000 7f45a65b0640 1 --2- 172.21.15.67:0/2024318973 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f45a00d0cc0 0x7f45a015cad0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.173+0000 7f45a65b0640 1 --2- 172.21.15.67:0/2024318973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45a00b5780 0x7f45a00cd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.173+0000 7f45a65b0640 1 -- 172.21.15.67:0/2024318973 >> 172.21.15.67:0/2024318973 conn(0x7f45a006c6b0 msgr2=0x7f45a00ce400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:12.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.173+0000 7f45a65b0640 1 -- 172.21.15.67:0/2024318973 shutdown_connections 2024-03-24T00:39:12.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.173+0000 7f45a65b0640 1 -- 172.21.15.67:0/2024318973 wait complete. 2024-03-24T00:39:12.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.174+0000 7f9974ff9640 1 -- 172.21.15.67:0/2324264500 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f996806bc00 msgr2=0x7f996806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:12.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.174+0000 7f9974ff9640 1 --2- 172.21.15.67:0/2324264500 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f996806bc00 0x7f996806e0c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f996c007eb0 tx=0x7f996c009030 comp rx=0 tx=0).stop 2024-03-24T00:39:12.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.174+0000 7f9974ff9640 1 -- 172.21.15.67:0/2324264500 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f998013cca0 msgr2=0x7f9980159960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:12.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.174+0000 7f9974ff9640 1 --2- 172.21.15.67:0/2324264500 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f998013cca0 0x7f9980159960 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f99700017e0 tx=0x7f997000f5a0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.174+0000 7f9974ff9640 1 -- 172.21.15.67:0/2324264500 shutdown_connections 2024-03-24T00:39:12.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.174+0000 7f9974ff9640 1 --2- 172.21.15.67:0/2324264500 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f996806bc00 0x7f996806e0c0 secure :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f996c007eb0 tx=0x7f996c009030 comp rx=0 tx=0).stop 2024-03-24T00:39:12.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.174+0000 7f9974ff9640 1 --2- 172.21.15.67:0/2324264500 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9980155910 0x7f9980159ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.174+0000 7fdc2196b640 1 -- 172.21.15.67:0/777669300 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fdc1c0cbc80 con 0x7fdc1c162360 2024-03-24T00:39:12.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.174+0000 7f9974ff9640 1 --2- 172.21.15.67:0/2324264500 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f998013cca0 0x7f9980159960 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.174+0000 7f9974ff9640 1 --2- 172.21.15.67:0/2324264500 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f998013c8c0 0x7f9980200ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.174+0000 7f9974ff9640 1 -- 172.21.15.67:0/2324264500 >> 172.21.15.67:0/2324264500 conn(0x7f9980071d10 msgr2=0x7f998013fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:12.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.174+0000 7f9974ff9640 1 -- 172.21.15.67:0/2324264500 shutdown_connections 2024-03-24T00:39:12.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.174+0000 7f9974ff9640 1 -- 172.21.15.67:0/2324264500 wait complete. 2024-03-24T00:39:12.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.174+0000 7fdc197fa640 1 -- 172.21.15.67:0/777669300 <== mon.2 v2:172.21.15.72:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fdc10058f60 con 0x7fdc1c162360 2024-03-24T00:39:12.178 INFO:teuthology.orchestra.run.smithi067.stdout:55834574857 2024-03-24T00:39:12.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.175+0000 7f1bb3337640 1 -- 172.21.15.67:0/4281148724 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f1ba403ee10 con 0x7f1ba4006220 2024-03-24T00:39:12.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.175+0000 7f1b9dffb640 1 -- 172.21.15.67:0/4281148724 <== mon.1 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1ba40dea30 con 0x7f1ba4006220 2024-03-24T00:39:12.179 INFO:teuthology.orchestra.run.smithi067.stdout:55834574858 2024-03-24T00:39:12.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.177+0000 7f3ee8159640 1 -- 172.21.15.67:0/3300752746 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f3ee00ca5c0 con 0x7f3ee015e050 2024-03-24T00:39:12.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.178+0000 7fdc02ffd640 1 -- 172.21.15.67:0/777669300 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdc0806bca0 msgr2=0x7fdc0806e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:12.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.178+0000 7fdc02ffd640 1 --2- 172.21.15.67:0/777669300 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdc0806bca0 0x7fdc0806e160 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c1604e0 tx=0x7fdc14030000 comp rx=0 tx=0).stop 2024-03-24T00:39:12.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.178+0000 7fdc02ffd640 1 -- 172.21.15.67:0/777669300 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdc1c162360 msgr2=0x7fdc1c158650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:12.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.178+0000 7fdc02ffd640 1 --2- 172.21.15.67:0/777669300 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdc1c162360 0x7fdc1c158650 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdc1000d580 tx=0x7fdc1000da40 comp rx=0 tx=0).stop 2024-03-24T00:39:12.181 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.0 2024-03-24T00:39:12.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.178+0000 7f3ed67fc640 1 -- 172.21.15.67:0/3300752746 <== mon.2 v2:172.21.15.72:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3edc07ef10 con 0x7f3ee015e050 2024-03-24T00:39:12.181 INFO:teuthology.orchestra.run.smithi067.stdout:55834574857 2024-03-24T00:39:12.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.178+0000 7fdc02ffd640 1 -- 172.21.15.67:0/777669300 shutdown_connections 2024-03-24T00:39:12.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.178+0000 7fdc02ffd640 1 --2- 172.21.15.67:0/777669300 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdc0806bca0 0x7fdc0806e160 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.178+0000 7fdc02ffd640 1 --2- 172.21.15.67:0/777669300 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdc1c162360 0x7fdc1c158650 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.178+0000 7fdc02ffd640 1 --2- 172.21.15.67:0/777669300 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc1c071d90 0x7fdc1c15dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.178+0000 7fdc02ffd640 1 --2- 172.21.15.67:0/777669300 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc1c06e2a0 0x7fdc1c163710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.178+0000 7fdc02ffd640 1 -- 172.21.15.67:0/777669300 >> 172.21.15.67:0/777669300 conn(0x7fdc1c0cf7b0 msgr2=0x7fdc1c15f470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:12.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.178+0000 7fdc02ffd640 1 -- 172.21.15.67:0/777669300 shutdown_connections 2024-03-24T00:39:12.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.178+0000 7fdc02ffd640 1 -- 172.21.15.67:0/777669300 wait complete. 2024-03-24T00:39:12.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.179+0000 7fdfa0ff9640 1 -- 172.21.15.67:0/3008509033 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fdfa40e3130 con 0x7fdfa40df800 2024-03-24T00:39:12.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.179+0000 7fdfa2ffd640 1 -- 172.21.15.67:0/3008509033 <== mon.1 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fdfac0d89f0 con 0x7fdfa40df800 2024-03-24T00:39:12.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.179+0000 7f1bb3337640 1 -- 172.21.15.67:0/4281148724 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1b9006bca0 msgr2=0x7f1b9006e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:12.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.179+0000 7f1bb3337640 1 --2- 172.21.15.67:0/4281148724 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1b9006bca0 0x7f1b9006e160 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1ba416d460 tx=0x7f1ba0007e00 comp rx=0 tx=0).stop 2024-03-24T00:39:12.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.179+0000 7f1bb3337640 1 -- 172.21.15.67:0/4281148724 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1ba4006220 msgr2=0x7f1ba4181a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:12.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.179+0000 7f1bb3337640 1 --2- 172.21.15.67:0/4281148724 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1ba4006220 0x7f1ba4181a10 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1bac075f70 tx=0x7f1bac069920 comp rx=0 tx=0).stop 2024-03-24T00:39:12.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.179+0000 7f1bb3337640 1 -- 172.21.15.67:0/4281148724 shutdown_connections 2024-03-24T00:39:12.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.179+0000 7f1bb3337640 1 --2- 172.21.15.67:0/4281148724 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1b9006bca0 0x7f1b9006e160 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.184 INFO:teuthology.orchestra.run.smithi067.stdout:55834574856 2024-03-24T00:39:12.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.179+0000 7f1bb3337640 1 --2- 172.21.15.67:0/4281148724 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1ba40e4f50 0x7f1ba40e5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.179+0000 7f1bb3337640 1 --2- 172.21.15.67:0/4281148724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ba4006640 0x7f1ba40520c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.179+0000 7f1bb3337640 1 --2- 172.21.15.67:0/4281148724 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1ba4006220 0x7f1ba4181a10 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.179+0000 7f1bb3337640 1 -- 172.21.15.67:0/4281148724 >> 172.21.15.67:0/4281148724 conn(0x7f1ba4058f20 msgr2=0x7f1ba403c080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:12.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.180+0000 7f1bb3337640 1 -- 172.21.15.67:0/4281148724 shutdown_connections 2024-03-24T00:39:12.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.180+0000 7f1bb3337640 1 -- 172.21.15.67:0/4281148724 wait complete. 2024-03-24T00:39:12.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.182+0000 7f3ee8159640 1 -- 172.21.15.67:0/3300752746 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3ec406bc00 msgr2=0x7f3ec406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:12.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.182+0000 7f3ee8159640 1 --2- 172.21.15.67:0/3300752746 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3ec406bc00 0x7f3ec406e0c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3ed80012e0 tx=0x7f3ed8009040 comp rx=0 tx=0).stop 2024-03-24T00:39:12.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.182+0000 7f3ee8159640 1 -- 172.21.15.67:0/3300752746 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3ee015e050 msgr2=0x7f3ee00cb360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:12.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.182+0000 7f3ee8159640 1 --2- 172.21.15.67:0/3300752746 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3ee015e050 0x7f3ee00cb360 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3edc032000 tx=0x7f3edc0046c0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.182+0000 7f3ee8159640 1 -- 172.21.15.67:0/3300752746 shutdown_connections 2024-03-24T00:39:12.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.182+0000 7f3ee8159640 1 --2- 172.21.15.67:0/3300752746 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3ec406bc00 0x7f3ec406e0c0 secure :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3ed80012e0 tx=0x7f3ed8009040 comp rx=0 tx=0).stop 2024-03-24T00:39:12.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.182+0000 7f3ee8159640 1 --2- 172.21.15.67:0/3300752746 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3ee015e050 0x7f3ee00cb360 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.182+0000 7f3ee8159640 1 --2- 172.21.15.67:0/3300752746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3ee00cf020 0x7f3ee015c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.182+0000 7f3ee8159640 1 --2- 172.21.15.67:0/3300752746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ee006f280 0x7f3ee0159730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.182+0000 7f3ee8159640 1 -- 172.21.15.67:0/3300752746 >> 172.21.15.67:0/3300752746 conn(0x7f3ee00b1450 msgr2=0x7f3ee00cfe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:12.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.182+0000 7f3ee8159640 1 -- 172.21.15.67:0/3300752746 shutdown_connections 2024-03-24T00:39:12.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.182+0000 7f3ee8159640 1 -- 172.21.15.67:0/3300752746 wait complete. 2024-03-24T00:39:12.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.183+0000 7fdfa0ff9640 1 -- 172.21.15.67:0/3008509033 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdf9006bbd0 msgr2=0x7fdf9006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:12.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.183+0000 7fdfa0ff9640 1 --2- 172.21.15.67:0/3008509033 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdf9006bbd0 0x7fdf9006e090 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c005880 tx=0x7fdf9c009040 comp rx=0 tx=0).stop 2024-03-24T00:39:12.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.183+0000 7fdfa0ff9640 1 -- 172.21.15.67:0/3008509033 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdfa40df800 msgr2=0x7fdfa4039530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:12.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.183+0000 7fdfa0ff9640 1 --2- 172.21.15.67:0/3008509033 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdfa40df800 0x7fdfa4039530 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fdfac0691e0 tx=0x7fdfac068b60 comp rx=0 tx=0).stop 2024-03-24T00:39:12.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.183+0000 7fdfa0ff9640 1 -- 172.21.15.67:0/3008509033 shutdown_connections 2024-03-24T00:39:12.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.183+0000 7fdfa0ff9640 1 --2- 172.21.15.67:0/3008509033 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdf9006bbd0 0x7fdf9006e090 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.183+0000 7fdfa0ff9640 1 --2- 172.21.15.67:0/3008509033 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdfa40df800 0x7fdfa4039530 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.183+0000 7fdfa0ff9640 1 --2- 172.21.15.67:0/3008509033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfa4054200 0x7fdfa40d8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.183+0000 7fdfa0ff9640 1 --2- 172.21.15.67:0/3008509033 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdfa4006770 0x7fdfa40485d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.183+0000 7fdfa0ff9640 1 -- 172.21.15.67:0/3008509033 >> 172.21.15.67:0/3008509033 conn(0x7fdfa4009fb0 msgr2=0x7fdfa4004990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:12.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.184+0000 7fdfa0ff9640 1 -- 172.21.15.67:0/3008509033 shutdown_connections 2024-03-24T00:39:12.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.184+0000 7fdfa0ff9640 1 -- 172.21.15.67:0/3008509033 wait complete. 2024-03-24T00:39:12.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.186+0000 7f976f5c4640 1 -- 172.21.15.67:0/4205714727 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f976814a610 con 0x7f9768072210 2024-03-24T00:39:12.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.186+0000 7f975dffb640 1 -- 172.21.15.67:0/4205714727 <== mon.1 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f976407a9c0 con 0x7f9768072210 2024-03-24T00:39:12.190 INFO:teuthology.orchestra.run.smithi067.stdout:55834574857 2024-03-24T00:39:12.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.190+0000 7f976f5c4640 1 -- 172.21.15.67:0/4205714727 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f974c06bc00 msgr2=0x7f974c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:12.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.190+0000 7f976f5c4640 1 --2- 172.21.15.67:0/4205714727 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f974c06bc00 0x7f974c06e0c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f97681622c0 tx=0x7f9758009060 comp rx=0 tx=0).stop 2024-03-24T00:39:12.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.190+0000 7f976f5c4640 1 -- 172.21.15.67:0/4205714727 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9768072210 msgr2=0x7f976815e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:12.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.190+0000 7f976f5c4640 1 --2- 172.21.15.67:0/4205714727 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9768072210 0x7f976815e220 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9764009870 tx=0x7f97640046c0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.190+0000 7f976f5c4640 1 -- 172.21.15.67:0/4205714727 shutdown_connections 2024-03-24T00:39:12.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.190+0000 7f976f5c4640 1 --2- 172.21.15.67:0/4205714727 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f974c06bc00 0x7f974c06e0c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.190+0000 7f976f5c4640 1 --2- 172.21.15.67:0/4205714727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9768166930 0x7f9768163a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.190+0000 7f976f5c4640 1 --2- 172.21.15.67:0/4205714727 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f976814bb50 0x7f976813e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.190+0000 7f976f5c4640 1 --2- 172.21.15.67:0/4205714727 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9768072210 0x7f976815e220 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.190+0000 7f976f5c4640 1 -- 172.21.15.67:0/4205714727 >> 172.21.15.67:0/4205714727 conn(0x7f9768067470 msgr2=0x7f9768071b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:12.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.190+0000 7f976f5c4640 1 -- 172.21.15.67:0/4205714727 shutdown_connections 2024-03-24T00:39:12.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:12.191+0000 7f976f5c4640 1 -- 172.21.15.67:0/4205714727 wait complete. 2024-03-24T00:39:12.197 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.5 2024-03-24T00:39:12.197 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.3 2024-03-24T00:39:12.202 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.1 2024-03-24T00:39:12.202 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574858 for osd.7 2024-03-24T00:39:12.203 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.4 2024-03-24T00:39:12.204 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.2 2024-03-24T00:39:12.206 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.6 2024-03-24T00:39:13.182 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-24T00:39:13.198 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-24T00:39:13.198 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-24T00:39:13.203 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-24T00:39:13.203 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-24T00:39:13.203 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-24T00:39:13.205 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-24T00:39:13.206 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-24T00:39:13.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.347+0000 7f74bb4ff640 1 Processor -- start 2024-03-24T00:39:13.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.348+0000 7f74bb4ff640 1 -- start start 2024-03-24T00:39:13.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.349+0000 7f74bb4ff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f74b406dc40 0x7f74b406e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.349+0000 7f74bb4ff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f74b4071b00 0x7f74b4071f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.349+0000 7f74bb4ff640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b40ce290 con 0x7f74b41404a0 2024-03-24T00:39:13.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.349+0000 7f74bb4ff640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74b406ecc0 con 0x7f74b4071b00 2024-03-24T00:39:13.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.349+0000 7f74bb4ff640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74b4140880 con 0x7f74b406dc40 2024-03-24T00:39:13.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.350+0000 7f74b929b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f74b41404a0 0x7f74b40b30a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45550/0 (socket says 172.21.15.67:45550) 2024-03-24T00:39:13.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.350+0000 7f74b929b640 1 -- 172.21.15.67:0/4124345681 learned_addr learned my addr 172.21.15.67:0/4124345681 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:13.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.350+0000 7f74b8a9a640 1 --2- 172.21.15.67:0/4124345681 >> v2:172.21.15.72:3300/0 conn(0x7f74b406dc40 0x7f74b406e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.350+0000 7f74b8a9a640 1 -- 172.21.15.67:0/4124345681 >> v2:172.21.15.67:3300/0 conn(0x7f74b4071b00 msgr2=0x7f74b4071f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:39:13.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.350+0000 7f74b8a9a640 1 --2- 172.21.15.67:0/4124345681 >> v2:172.21.15.67:3300/0 conn(0x7f74b4071b00 0x7f74b4071f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.350+0000 7f74b8a9a640 1 -- 172.21.15.67:0/4124345681 >> v1:172.21.15.67:6789/0 conn(0x7f74b41404a0 legacy=0x7f74b40b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.350+0000 7f74b8a9a640 1 -- 172.21.15.67:0/4124345681 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b4072440 con 0x7f74b406dc40 2024-03-24T00:39:13.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.353+0000 7f74b8a9a640 1 --2- 172.21.15.67:0/4124345681 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f74b406dc40 0x7f74b406e080 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f74b40726a0 tx=0x7f74b002b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a268f9c6aa2b7aa9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:13.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.354+0000 7f74abfff640 1 -- 172.21.15.67:0/4124345681 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b0035020 con 0x7f74b406dc40 2024-03-24T00:39:13.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.354+0000 7f74abfff640 1 -- 172.21.15.67:0/4124345681 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74b0031870 con 0x7f74b406dc40 2024-03-24T00:39:13.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.355+0000 7f74bb4ff640 1 -- 172.21.15.67:0/4124345681 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f74b406dc40 msgr2=0x7f74b406e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.355+0000 7f74bb4ff640 1 --2- 172.21.15.67:0/4124345681 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f74b406dc40 0x7f74b406e080 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f74b40726a0 tx=0x7f74b002b910 comp rx=0 tx=0).stop 2024-03-24T00:39:13.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.355+0000 7f74bb4ff640 1 -- 172.21.15.67:0/4124345681 shutdown_connections 2024-03-24T00:39:13.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.355+0000 7f74bb4ff640 1 --2- 172.21.15.67:0/4124345681 >> v2:172.21.15.67:3300/0 conn(0x7f74b4071b00 0x7f74b4071f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.355+0000 7f74bb4ff640 1 --2- 172.21.15.67:0/4124345681 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f74b406dc40 0x7f74b406e080 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.355+0000 7f74bb4ff640 1 -- 172.21.15.67:0/4124345681 >> 172.21.15.67:0/4124345681 conn(0x7f74b4067470 msgr2=0x7f74b406b040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:13.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.357+0000 7f74bb4ff640 1 -- 172.21.15.67:0/4124345681 shutdown_connections 2024-03-24T00:39:13.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.357+0000 7f74bb4ff640 1 -- 172.21.15.67:0/4124345681 wait complete. 2024-03-24T00:39:13.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.359+0000 7f74bb4ff640 1 Processor -- start 2024-03-24T00:39:13.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.359+0000 7f74bb4ff640 1 -- start start 2024-03-24T00:39:13.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.361+0000 7f74bb4ff640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f74b4071b00 0x7f74b41f7730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.361+0000 7f74bb4ff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74b41404a0 0x7f74b4165f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.361+0000 7f74b8a9a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74b41404a0 0x7f74b4165f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.361+0000 7f74b929b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f74b4071b00 0x7f74b41f7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.361+0000 7f74b929b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f74b4071b00 0x7f74b41f7730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.67:49212/0 (socket says 172.21.15.67:49212) 2024-03-24T00:39:13.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.361+0000 7f74b8a9a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74b41404a0 0x7f74b4165f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32816/0 (socket says 172.21.15.67:32816) 2024-03-24T00:39:13.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.362+0000 7f74bb4ff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f74b4153770 0x7f74b413f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.362+0000 7f74b8a9a640 1 -- 172.21.15.67:0/146962757 learned_addr learned my addr 172.21.15.67:0/146962757 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:13.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.362+0000 7f74bb4ff640 1 -- 172.21.15.67:0/146962757 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b40ce510 con 0x7f74b41404a0 2024-03-24T00:39:13.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.362+0000 7f74bb4ff640 1 -- 172.21.15.67:0/146962757 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b406e9e0 con 0x7f74b4153770 2024-03-24T00:39:13.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.362+0000 7f74bb4ff640 1 -- 172.21.15.67:0/146962757 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74b40ce7f0 con 0x7f74b4071b00 2024-03-24T00:39:13.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.362+0000 7f74b9a9c640 1 --2- 172.21.15.67:0/146962757 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f74b4153770 0x7f74b413f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.363+0000 7f74b8a9a640 1 -- 172.21.15.67:0/146962757 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f74b4071b00 msgr2=0x7f74b41f7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.363+0000 7f74b8a9a640 1 --2- 172.21.15.67:0/146962757 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f74b4071b00 0x7f74b41f7730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.363+0000 7f74b8a9a640 1 -- 172.21.15.67:0/146962757 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f74b4153770 msgr2=0x7f74b413f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.363+0000 7f74b8a9a640 1 --2- 172.21.15.67:0/146962757 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f74b4153770 0x7f74b413f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.363+0000 7f74b8a9a640 1 -- 172.21.15.67:0/146962757 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b41f7c70 con 0x7f74b41404a0 2024-03-24T00:39:13.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.364+0000 7f74b8a9a640 1 --2- 172.21.15.67:0/146962757 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74b41404a0 0x7f74b4165f30 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f74b0009890 tx=0x7f74b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:13.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.364+0000 7f74a9ffb640 1 -- 172.21.15.67:0/146962757 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b0040070 con 0x7f74b41404a0 2024-03-24T00:39:13.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.364+0000 7f74bb4ff640 1 -- 172.21.15.67:0/146962757 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74b414f0f0 con 0x7f74b41404a0 2024-03-24T00:39:13.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.364+0000 7f74bb4ff640 1 -- 172.21.15.67:0/146962757 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74b40b0f80 con 0x7f74b41404a0 2024-03-24T00:39:13.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.368+0000 7f74a9ffb640 1 -- 172.21.15.67:0/146962757 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74b003a620 con 0x7f74b41404a0 2024-03-24T00:39:13.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.368+0000 7f74bb4ff640 1 -- 172.21.15.67:0/146962757 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74b404f400 con 0x7f74b41404a0 2024-03-24T00:39:13.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.370+0000 7f74a9ffb640 1 -- 172.21.15.67:0/146962757 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b0035040 con 0x7f74b41404a0 2024-03-24T00:39:13.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.371+0000 7f74a9ffb640 1 -- 172.21.15.67:0/146962757 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f74b0038020 con 0x7f74b41404a0 2024-03-24T00:39:13.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.371+0000 7f74a9ffb640 1 --2- 172.21.15.67:0/146962757 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f749806bd00 0x7f749806e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.371+0000 7f74a9ffb640 1 -- 172.21.15.67:0/146962757 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f74b00acf20 con 0x7f74b41404a0 2024-03-24T00:39:13.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.378+0000 7f74b929b640 1 --2- 172.21.15.67:0/146962757 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f749806bd00 0x7f749806e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.379+0000 7f74a9ffb640 1 -- 172.21.15.67:0/146962757 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f74b007b120 con 0x7f74b41404a0 2024-03-24T00:39:13.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.379+0000 7f74b929b640 1 --2- 172.21.15.67:0/146962757 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f749806bd00 0x7f749806e1c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f74ac005c00 tx=0x7f74ac009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:13.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.404+0000 7f40791c6640 1 Processor -- start 2024-03-24T00:39:13.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.404+0000 7f40791c6640 1 -- start start 2024-03-24T00:39:13.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.405+0000 7fd78ef1e640 1 Processor -- start 2024-03-24T00:39:13.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.405+0000 7fd78ef1e640 1 -- start start 2024-03-24T00:39:13.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.405+0000 7f40791c6640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f40740d07a0 0x7f40740cc770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.405+0000 7f40791c6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f40740cccb0 0x7f40740b2790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.405+0000 7f40791c6640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40740cd160 con 0x7f407415c010 2024-03-24T00:39:13.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.405+0000 7f40791c6640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40740cfbe0 con 0x7f40740d07a0 2024-03-24T00:39:13.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.405+0000 7f40791c6640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40740b2020 con 0x7f40740cccb0 2024-03-24T00:39:13.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.405+0000 7f4072d9d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f407415c010 0x7f40740d0090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:51370/0 (socket says 172.21.15.67:51370) 2024-03-24T00:39:13.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.405+0000 7f4072d9d640 1 -- 172.21.15.67:0/1245550647 learned_addr learned my addr 172.21.15.67:0/1245550647 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:13.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.405+0000 7f406bfff640 1 --2- 172.21.15.67:0/1245550647 >> v2:172.21.15.72:3300/0 conn(0x7f40740d07a0 0x7f40740cc770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.405+0000 7f407359e640 1 --2- 172.21.15.67:0/1245550647 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f40740cccb0 0x7f40740b2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.406+0000 7fd78ef1e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd7800e1090 0x7fd780038630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.406+0000 7fd78ef1e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd780050dd0 0x7fd780039ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.406+0000 7fd78ef1e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd78004f540 con 0x7fd78003a400 2024-03-24T00:39:13.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.406+0000 7fd78ef1e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd78004f820 con 0x7fd7800e1090 2024-03-24T00:39:13.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.406+0000 7fd78ef1e640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd78004fb00 con 0x7fd780050dd0 2024-03-24T00:39:13.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.406+0000 7f76a0deb640 1 Processor -- start 2024-03-24T00:39:13.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.406+0000 7f76a0deb640 1 -- start start 2024-03-24T00:39:13.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.406+0000 7fd78df1c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd7800e1090 0x7fd780038630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.406+0000 7fd78e71d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fd78003a400 0x7fd7800e4a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:51376/0 (socket says 172.21.15.67:51376) 2024-03-24T00:39:13.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.406+0000 7fd78e71d640 1 -- 172.21.15.67:0/472866799 learned_addr learned my addr 172.21.15.67:0/472866799 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:13.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.406+0000 7fd78d71b640 1 --2- 172.21.15.67:0/472866799 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd780050dd0 0x7fd780039ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.406+0000 7f406bfff640 1 -- 172.21.15.67:0/1245550647 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f40740cccb0 msgr2=0x7f40740b2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:13.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.406+0000 7f406bfff640 1 --2- 172.21.15.67:0/1245550647 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f40740cccb0 0x7f40740b2790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.406+0000 7f406bfff640 1 -- 172.21.15.67:0/1245550647 >> v1:172.21.15.72:6789/0 conn(0x7f407415c010 legacy=0x7f40740d0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.406+0000 7f406bfff640 1 -- 172.21.15.67:0/1245550647 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40740cd430 con 0x7f40740d07a0 2024-03-24T00:39:13.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.406+0000 7f407359e640 1 --2- 172.21.15.67:0/1245550647 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f40740cccb0 0x7f40740b2790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:13.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.407+0000 7fd78df1c640 1 -- 172.21.15.67:0/472866799 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd780050dd0 msgr2=0x7fd780039ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:13.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.407+0000 7fd78df1c640 1 --2- 172.21.15.67:0/472866799 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd780050dd0 0x7fd780039ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.407+0000 7fd78df1c640 1 -- 172.21.15.67:0/472866799 >> v1:172.21.15.72:6789/0 conn(0x7fd78003a400 legacy=0x7fd7800e4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:13.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.407+0000 7fd78df1c640 1 -- 172.21.15.67:0/472866799 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd78004fde0 con 0x7fd7800e1090 2024-03-24T00:39:13.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.407+0000 7f9a6b59e640 1 Processor -- start 2024-03-24T00:39:13.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.407+0000 7f1b30e68640 1 Processor -- start 2024-03-24T00:39:13.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.407+0000 7f1b30e68640 1 -- start start 2024-03-24T00:39:13.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.408+0000 7fd78df1c640 1 --2- 172.21.15.67:0/472866799 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd7800e1090 0x7fd780038630 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd7780097a0 tx=0x7fd77802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f436f07eb518ecf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:13.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.408+0000 7f406bfff640 1 --2- 172.21.15.67:0/1245550647 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f40740d07a0 0x7f40740cc770 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f405c0097a0 tx=0x7f405c02b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e290ef4ba871901 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:13.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.407+0000 7f9a6b59e640 1 -- start start 2024-03-24T00:39:13.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.408+0000 7f76a0deb640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f769c15bfa0 0x7f769c0ce2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.408+0000 7f407259c640 1 -- 172.21.15.67:0/1245550647 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f405c035020 con 0x7f40740d07a0 2024-03-24T00:39:13.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.408+0000 7f407259c640 1 -- 172.21.15.67:0/1245550647 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f405c031810 con 0x7f40740d07a0 2024-03-24T00:39:13.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.408+0000 7f407259c640 1 -- 172.21.15.67:0/1245550647 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f405c031be0 con 0x7f40740d07a0 2024-03-24T00:39:13.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.408+0000 7f76a0deb640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f769c0d25f0 0x7f769c0ce7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.408+0000 7f2866234640 1 Processor -- start 2024-03-24T00:39:13.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.408+0000 7f769a59c640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f769c15bfa0 0x7f769c0ce2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.408+0000 7f769a59c640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f769c15bfa0 0x7f769c0ce2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.67:49242/0 (socket says 172.21.15.67:49242) 2024-03-24T00:39:13.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.408+0000 7fd78cf1a640 1 -- 172.21.15.67:0/472866799 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7780315a0 con 0x7fd7800e1090 2024-03-24T00:39:13.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.408+0000 7fd78cf1a640 1 -- 172.21.15.67:0/472866799 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd778037070 con 0x7fd7800e1090 2024-03-24T00:39:13.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.408+0000 7fd78cf1a640 1 -- 172.21.15.67:0/472866799 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd778031940 con 0x7fd7800e1090 2024-03-24T00:39:13.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.408+0000 7f769a59c640 1 -- 172.21.15.67:0/2474679210 learned_addr learned my addr 172.21.15.67:0/2474679210 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:13.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.409+0000 7f7693fff640 1 --2- 172.21.15.67:0/2474679210 >> v2:172.21.15.72:3300/0 conn(0x7f769c0d25f0 0x7f769c0ce7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.409+0000 7f76a0deb640 1 -- 172.21.15.67:0/2474679210 --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f769c06fb70 con 0x7f769c0b1060 2024-03-24T00:39:13.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.409+0000 7f9a6b59e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9a64007340 0x7f9a6404f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.409+0000 7f9a6b59e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a64007850 con 0x7f9a640e7760 2024-03-24T00:39:13.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.409+0000 7f84b584b640 1 Processor -- start 2024-03-24T00:39:13.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.409+0000 7f9a69d9b640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f9a6400b0b0 0x7f9a64053f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:51390/0 (socket says 172.21.15.67:51390) 2024-03-24T00:39:13.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.409+0000 7f9a69d9b640 1 -- 172.21.15.67:0/2118944058 learned_addr learned my addr 172.21.15.67:0/2118944058 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:13.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.409+0000 7f40791c6640 1 -- 172.21.15.67:0/1245550647 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f40740d07a0 msgr2=0x7f40740cc770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.409+0000 7f40791c6640 1 --2- 172.21.15.67:0/1245550647 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f40740d07a0 0x7f40740cc770 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f405c0097a0 tx=0x7f405c02b8b0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.409+0000 7f76a0deb640 1 -- 172.21.15.67:0/2474679210 --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f769c0cd270 con 0x7f769c0d25f0 2024-03-24T00:39:13.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.409+0000 7f76a0deb640 1 -- 172.21.15.67:0/2474679210 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f769c0d0af0 con 0x7f769c15bfa0 2024-03-24T00:39:13.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.409+0000 7f40791c6640 1 -- 172.21.15.67:0/1245550647 shutdown_connections 2024-03-24T00:39:13.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.409+0000 7f7699d9b640 1 -- 172.21.15.67:0/2474679210 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3658219751 0 0) 0x7f769c06fb70 con 0x7f769c0b1060 2024-03-24T00:39:13.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.409+0000 7f9a6b59e640 1 -- 172.21.15.67:0/2118944058 --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a64007b30 con 0x7f9a6400b0b0 2024-03-24T00:39:13.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.409+0000 7f40791c6640 1 --2- 172.21.15.67:0/1245550647 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f40740cccb0 0x7f40740b2790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.409+0000 7f40791c6640 1 --2- 172.21.15.67:0/1245550647 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f40740d07a0 0x7f40740cc770 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.409+0000 7f9a6b59e640 1 -- 172.21.15.67:0/2118944058 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a640e7b40 con 0x7f9a64007340 2024-03-24T00:39:13.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.409+0000 7f84b584b640 1 -- start start 2024-03-24T00:39:13.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.409+0000 7f1b30e68640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1b2c162910 0x7f1b2c162da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.409+0000 7f40791c6640 1 -- 172.21.15.67:0/1245550647 >> 172.21.15.67:0/1245550647 conn(0x7f407409aea0 msgr2=0x7f407408b180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:13.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.409+0000 7f1b30e68640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1b2c0ce540 0x7f1b2c0ce970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.409+0000 7f1b30e68640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b2c06f3c0 con 0x7f1b2c14bbe0 2024-03-24T00:39:13.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.409+0000 7f1b30e68640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b2c0cddf0 con 0x7f1b2c0ce540 2024-03-24T00:39:13.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.409+0000 7f1b30e68640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b2c1632e0 con 0x7f1b2c162910 2024-03-24T00:39:13.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.409+0000 7f1b2a59c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1b2c14bbe0 0x7f1b2c06e110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45552/0 (socket says 172.21.15.67:45552) 2024-03-24T00:39:13.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.409+0000 7f1b2a59c640 1 -- 172.21.15.67:0/1871304661 learned_addr learned my addr 172.21.15.67:0/1871304661 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:13.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.409+0000 7f1b2ad9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1b2c0ce540 0x7f1b2c0ce970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.409+0000 7f1b29d9b640 1 --2- 172.21.15.67:0/1871304661 >> v2:172.21.15.72:3300/0 conn(0x7f1b2c162910 0x7f1b2c162da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.409+0000 7f9a6959a640 1 -- 172.21.15.67:0/2118944058 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3390766663 0 0) 0x7f9a64007b30 con 0x7f9a6400b0b0 2024-03-24T00:39:13.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.409+0000 7f9a6959a640 1 -- 172.21.15.67:0/2118944058 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a54003520 con 0x7f9a6400b0b0 2024-03-24T00:39:13.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.409+0000 7f40791c6640 1 -- 172.21.15.67:0/1245550647 shutdown_connections 2024-03-24T00:39:13.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.409+0000 7fd78ef1e640 1 -- 172.21.15.67:0/472866799 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd7800e1090 msgr2=0x7fd780038630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.409+0000 7fd78ef1e640 1 --2- 172.21.15.67:0/472866799 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd7800e1090 0x7fd780038630 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd7780097a0 tx=0x7fd77802b840 comp rx=0 tx=0).stop 2024-03-24T00:39:13.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.409+0000 7f40791c6640 1 -- 172.21.15.67:0/1245550647 wait complete. 2024-03-24T00:39:13.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.410+0000 7fd78ef1e640 1 -- 172.21.15.67:0/472866799 shutdown_connections 2024-03-24T00:39:13.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.410+0000 7fd78ef1e640 1 --2- 172.21.15.67:0/472866799 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd780050dd0 0x7fd780039ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.410+0000 7fd78ef1e640 1 --2- 172.21.15.67:0/472866799 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd7800e1090 0x7fd780038630 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.410+0000 7fd78ef1e640 1 -- 172.21.15.67:0/472866799 >> 172.21.15.67:0/472866799 conn(0x7fd780009fb0 msgr2=0x7fd78000a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:13.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.410+0000 7f1b29d9b640 1 -- 172.21.15.67:0/1871304661 >> v2:172.21.15.67:3300/0 conn(0x7f1b2c0ce540 msgr2=0x7f1b2c0ce970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:13.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.410+0000 7f1b29d9b640 1 --2- 172.21.15.67:0/1871304661 >> v2:172.21.15.67:3300/0 conn(0x7f1b2c0ce540 0x7f1b2c0ce970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.410+0000 7f1b29d9b640 1 -- 172.21.15.67:0/1871304661 >> v1:172.21.15.67:6789/0 conn(0x7f1b2c14bbe0 legacy=0x7f1b2c06e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.410+0000 7f1b29d9b640 1 -- 172.21.15.67:0/1871304661 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b2c06eb20 con 0x7f1b2c162910 2024-03-24T00:39:13.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.410+0000 7fd78ef1e640 1 -- 172.21.15.67:0/472866799 shutdown_connections 2024-03-24T00:39:13.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.410+0000 7f84b584b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f84b015e0a0 0x7f84b00d3730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.410+0000 7f769a59c640 1 -- 172.21.15.67:0/2474679210 >> v2:172.21.15.72:3300/0 conn(0x7f769c0d25f0 msgr2=0x7f769c0ce7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:13.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.410+0000 7f769a59c640 1 --2- 172.21.15.67:0/2474679210 >> v2:172.21.15.72:3300/0 conn(0x7f769c0d25f0 0x7f769c0ce7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.410+0000 7f769a59c640 1 -- 172.21.15.67:0/2474679210 >> v1:172.21.15.72:6789/0 conn(0x7f769c0b1060 legacy=0x7f769c0cf8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.410+0000 7f769a59c640 1 -- 172.21.15.67:0/2474679210 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f769c0cd4f0 con 0x7f769c15bfa0 2024-03-24T00:39:13.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.410+0000 7fd78ef1e640 1 -- 172.21.15.67:0/472866799 wait complete. 2024-03-24T00:39:13.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.410+0000 7f2866234640 1 -- start start 2024-03-24T00:39:13.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.411+0000 7f2866234640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f286015e0a0 0x7f28600d3730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.411+0000 7f2866234640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f28600b97b0 0x7f28600d1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.411+0000 7f2866234640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28600d23c0 con 0x7f28600ceb80 2024-03-24T00:39:13.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.411+0000 7f2866234640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28600d2690 con 0x7f28600b97b0 2024-03-24T00:39:13.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.411+0000 7f2866234640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28600d2970 con 0x7f286015e0a0 2024-03-24T00:39:13.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.411+0000 7f769a59c640 1 --2- 172.21.15.67:0/2474679210 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f769c15bfa0 0x7f769c0ce2b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f768c009870 tx=0x7f768c02ba50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57ec0665b1816c45 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:13.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.411+0000 7f9a6ad9d640 1 --2- 172.21.15.67:0/2118944058 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9a64007340 0x7f9a6404f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.410+0000 7f84b584b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f84b00b97b0 0x7f84b00d1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.410+0000 7f84b584b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84b00d23c0 con 0x7f84b00ceb80 2024-03-24T00:39:13.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.410+0000 7f84b584b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84b00d2690 con 0x7f84b00b97b0 2024-03-24T00:39:13.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.410+0000 7f84b584b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84b00d2970 con 0x7f84b015e0a0 2024-03-24T00:39:13.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.411+0000 7f84ae7fc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f84b00b97b0 0x7f84b00d1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.411+0000 7f84af7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f84b00ceb80 0x7f84b00b5830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45572/0 (socket says 172.21.15.67:45572) 2024-03-24T00:39:13.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.411+0000 7f84af7fe640 1 -- 172.21.15.67:0/3192424875 learned_addr learned my addr 172.21.15.67:0/3192424875 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:13.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.411+0000 7f84aeffd640 1 --2- 172.21.15.67:0/3192424875 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f84b015e0a0 0x7f84b00d3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.411+0000 7f1b29d9b640 1 --2- 172.21.15.67:0/1871304661 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b2c162910 0x7f1b2c162da0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c0b25c0 tx=0x7f1b2402bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a15682dfe8003b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:13.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.411+0000 7f9a6959a640 1 -- 172.21.15.67:0/2118944058 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 553214347 0 0) 0x7f9a64007850 con 0x7f9a640e7760 2024-03-24T00:39:13.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.411+0000 7f9a6959a640 1 -- 172.21.15.67:0/2118944058 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a64007b30 con 0x7f9a640e7760 2024-03-24T00:39:13.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.411+0000 7f9a6959a640 1 -- 172.21.15.67:0/2118944058 <== mon.1 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2497426090 0 0) 0x7f9a54003520 con 0x7f9a6400b0b0 2024-03-24T00:39:13.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.411+0000 7f9a6959a640 1 -- 172.21.15.67:0/2118944058 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9a64007340 msgr2=0x7f9a6404f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:13.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.411+0000 7f9a6959a640 1 --2- 172.21.15.67:0/2118944058 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9a64007340 0x7f9a6404f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.411+0000 7f9a6959a640 1 -- 172.21.15.67:0/2118944058 >> v1:172.21.15.67:6789/0 conn(0x7f9a640e7760 legacy=0x7f9a640489d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.411+0000 7f9a6959a640 1 -- 172.21.15.67:0/2118944058 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a64053740 con 0x7f9a6400b0b0 2024-03-24T00:39:13.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.411+0000 7f9a6959a640 1 -- 172.21.15.67:0/2118944058 <== mon.1 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f9a6c04fc50 con 0x7f9a6400b0b0 2024-03-24T00:39:13.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.411+0000 7f9a6959a640 1 -- 172.21.15.67:0/2118944058 >> v1:172.21.15.72:6789/0 conn(0x7f9a6400b0b0 legacy=0x7f9a64053f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.411+0000 7f9a6959a640 1 --2- 172.21.15.67:0/2118944058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a54003b40 0x7f9a54003f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.411+0000 7f9a6959a640 1 --2- 172.21.15.67:0/2118944058 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9a54004800 0x7f9a54004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.411+0000 7f9a6959a640 1 --2- 172.21.15.67:0/2118944058 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a540051c0 0x7f9a5400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.411+0000 7f9a6959a640 1 -- 172.21.15.67:0/2118944058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5400bf00 con 0x7f9a54003b40 2024-03-24T00:39:13.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.411+0000 7f40791c6640 1 Processor -- start 2024-03-24T00:39:13.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.411+0000 7f40791c6640 1 -- start start 2024-03-24T00:39:13.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.412+0000 7f84aeffd640 1 -- 172.21.15.67:0/3192424875 >> v2:172.21.15.72:3300/0 conn(0x7f84b00b97b0 msgr2=0x7f84b00d1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:13.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.412+0000 7f84aeffd640 1 --2- 172.21.15.67:0/3192424875 >> v2:172.21.15.72:3300/0 conn(0x7f84b00b97b0 0x7f84b00d1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.412+0000 7f285ffff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f28600ceb80 0x7f28600b5830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45580/0 (socket says 172.21.15.67:45580) 2024-03-24T00:39:13.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.412+0000 7f285ffff640 1 -- 172.21.15.67:0/2564460527 learned_addr learned my addr 172.21.15.67:0/2564460527 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:13.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.412+0000 7fd78ef1e640 1 Processor -- start 2024-03-24T00:39:13.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.412+0000 7f7699d9b640 1 -- 172.21.15.67:0/2474679210 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768c02f870 con 0x7f769c15bfa0 2024-03-24T00:39:13.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.412+0000 7f7699d9b640 1 -- 172.21.15.67:0/2474679210 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f768c02fa10 con 0x7f769c15bfa0 2024-03-24T00:39:13.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.412+0000 7f285effd640 1 --2- 172.21.15.67:0/2564460527 >> v2:172.21.15.67:3300/0 conn(0x7f28600b97b0 0x7f28600d1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.412+0000 7f7699d9b640 1 -- 172.21.15.67:0/2474679210 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768c02f870 con 0x7f769c15bfa0 2024-03-24T00:39:13.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.411+0000 7f9a6959a640 1 -- 172.21.15.67:0/2118944058 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5400c080 con 0x7f9a540051c0 2024-03-24T00:39:13.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.412+0000 7f9a6959a640 1 -- 172.21.15.67:0/2118944058 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5400c200 con 0x7f9a54004800 2024-03-24T00:39:13.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.412+0000 7f9a69d9b640 1 --2- 172.21.15.67:0/2118944058 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9a54004800 0x7f9a54004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.412+0000 7f9a6a59c640 1 --2- 172.21.15.67:0/2118944058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a54003b40 0x7f9a54003f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.412+0000 7f9a6ad9d640 1 --2- 172.21.15.67:0/2118944058 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a540051c0 0x7f9a5400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.412+0000 7f1b2959a640 1 -- 172.21.15.67:0/1871304661 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b24039070 con 0x7f1b2c162910 2024-03-24T00:39:13.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.412+0000 7f1b2959a640 1 -- 172.21.15.67:0/1871304661 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b2402cc90 con 0x7f1b2c162910 2024-03-24T00:39:13.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.412+0000 7f1b2959a640 1 -- 172.21.15.67:0/1871304661 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b24034660 con 0x7f1b2c162910 2024-03-24T00:39:13.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.413+0000 7f285f7fe640 1 --2- 172.21.15.67:0/2564460527 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f286015e0a0 0x7f28600d3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.413+0000 7f40791c6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f40740cccb0 0x7f40740d21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.412+0000 7f84aeffd640 1 -- 172.21.15.67:0/3192424875 >> v1:172.21.15.67:6789/0 conn(0x7f84b00ceb80 legacy=0x7f84b00b5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.413+0000 7f40791c6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f40740d07a0 0x7f40740b4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.413+0000 7f40791c6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f407415c010 0x7f40740b4c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.412+0000 7f84aeffd640 1 -- 172.21.15.67:0/3192424875 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84b00d2c50 con 0x7f84b015e0a0 2024-03-24T00:39:13.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.412+0000 7f84ae7fc640 1 --2- 172.21.15.67:0/3192424875 >> v2:172.21.15.72:3300/0 conn(0x7f84b00b97b0 0x7f84b00d1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:13.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.413+0000 7f406bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f40740d07a0 0x7f40740b4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.413+0000 7f406bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f40740d07a0 0x7f40740b4700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:34282/0 (socket says 172.21.15.67:34282) 2024-03-24T00:39:13.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.413+0000 7f76a0deb640 1 -- 172.21.15.67:0/2474679210 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f769c15bfa0 msgr2=0x7f769c0ce2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.413+0000 7f76a0deb640 1 --2- 172.21.15.67:0/2474679210 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f769c15bfa0 0x7f769c0ce2b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f768c009870 tx=0x7f768c02ba50 comp rx=0 tx=0).stop 2024-03-24T00:39:13.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.413+0000 7f84aeffd640 1 --2- 172.21.15.67:0/3192424875 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f84b015e0a0 0x7f84b00d3730 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f84a8009d90 tx=0x7f84a802beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=240f769dd62b8a50 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:13.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.412+0000 7fd78ef1e640 1 -- start start 2024-03-24T00:39:13.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.413+0000 7f1b30e68640 1 -- 172.21.15.67:0/1871304661 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b2c162910 msgr2=0x7f1b2c162da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.413+0000 7fd78ef1e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd78003a400 0x7fd7800dcc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.413+0000 7fd78ef1e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd780050dd0 0x7fd7800db200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.413+0000 7fd78ef1e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd7800e1090 0x7fd7800db740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.413+0000 7f1b30e68640 1 --2- 172.21.15.67:0/1871304661 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b2c162910 0x7f1b2c162da0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c0b25c0 tx=0x7f1b2402bee0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.413+0000 7f4072d9d640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f40740cccb0 0x7f40740d21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.413+0000 7f285f7fe640 1 -- 172.21.15.67:0/2564460527 >> v2:172.21.15.67:3300/0 conn(0x7f28600b97b0 msgr2=0x7f28600d1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:13.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.413+0000 7f285f7fe640 1 --2- 172.21.15.67:0/2564460527 >> v2:172.21.15.67:3300/0 conn(0x7f28600b97b0 0x7f28600d1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.413+0000 7f9a6a59c640 1 -- 172.21.15.67:0/2118944058 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9a54004800 msgr2=0x7f9a54004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:13.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.413+0000 7f285f7fe640 1 -- 172.21.15.67:0/2564460527 >> v1:172.21.15.67:6789/0 conn(0x7f28600ceb80 legacy=0x7f28600b5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.413+0000 7f9a6a59c640 1 --2- 172.21.15.67:0/2118944058 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9a54004800 0x7f9a54004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.413+0000 7f9a6a59c640 1 -- 172.21.15.67:0/2118944058 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a540051c0 msgr2=0x7f9a5400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:13.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.413+0000 7f9a6a59c640 1 --2- 172.21.15.67:0/2118944058 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a540051c0 0x7f9a5400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.413+0000 7f9a6a59c640 1 -- 172.21.15.67:0/2118944058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a64053740 con 0x7f9a54003b40 2024-03-24T00:39:13.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.413+0000 7f285f7fe640 1 -- 172.21.15.67:0/2564460527 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28600d2c50 con 0x7f286015e0a0 2024-03-24T00:39:13.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.413+0000 7f4072d9d640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f40740cccb0 0x7f40740d21a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.67:49290/0 (socket says 172.21.15.67:49290) 2024-03-24T00:39:13.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.414+0000 7f406bfff640 1 -- 172.21.15.67:0/2536310860 learned_addr learned my addr 172.21.15.67:0/2536310860 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:13.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.413+0000 7fd78d71b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd780050dd0 0x7fd7800db200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.413+0000 7fd78d71b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd780050dd0 0x7fd7800db200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.67:49300/0 (socket says 172.21.15.67:49300) 2024-03-24T00:39:13.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.413+0000 7fd78d71b640 1 -- 172.21.15.67:0/3082185594 learned_addr learned my addr 172.21.15.67:0/3082185594 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:13.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.413+0000 7fd78e71d640 1 --2- 172.21.15.67:0/3082185594 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd7800e1090 0x7fd7800db740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.414+0000 7fd78df1c640 1 --2- 172.21.15.67:0/3082185594 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd78003a400 0x7fd7800dcc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.414+0000 7fd78ef1e640 1 -- 172.21.15.67:0/3082185594 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd78003a870 con 0x7fd78003a400 2024-03-24T00:39:13.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.414+0000 7f1b30e68640 1 -- 172.21.15.67:0/1871304661 shutdown_connections 2024-03-24T00:39:13.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.414+0000 7f1b30e68640 1 --2- 172.21.15.67:0/1871304661 >> v2:172.21.15.67:3300/0 conn(0x7f1b2c0ce540 0x7f1b2c0ce970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.414+0000 7f1b30e68640 1 --2- 172.21.15.67:0/1871304661 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b2c162910 0x7f1b2c162da0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.414+0000 7f1b30e68640 1 -- 172.21.15.67:0/1871304661 >> 172.21.15.67:0/1871304661 conn(0x7f1b2c06b670 msgr2=0x7f1b2c0b17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:13.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.414+0000 7fd78ef1e640 1 -- 172.21.15.67:0/3082185594 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7800503a0 con 0x7fd7800e1090 2024-03-24T00:39:13.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.414+0000 7fd78ef1e640 1 -- 172.21.15.67:0/3082185594 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd780006ff0 con 0x7fd780050dd0 2024-03-24T00:39:13.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.414+0000 7f76a0deb640 1 -- 172.21.15.67:0/2474679210 shutdown_connections 2024-03-24T00:39:13.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.414+0000 7f76a0deb640 1 --2- 172.21.15.67:0/2474679210 >> v2:172.21.15.72:3300/0 conn(0x7f769c0d25f0 0x7f769c0ce7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.414+0000 7f76a0deb640 1 --2- 172.21.15.67:0/2474679210 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f769c15bfa0 0x7f769c0ce2b0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.414+0000 7f76a0deb640 1 -- 172.21.15.67:0/2474679210 >> 172.21.15.67:0/2474679210 conn(0x7f769c0d1ad0 msgr2=0x7f769c0d1ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:13.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.414+0000 7f1b30e68640 1 -- 172.21.15.67:0/1871304661 shutdown_connections 2024-03-24T00:39:13.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.414+0000 7f407359e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f407415c010 0x7f40740b4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.414+0000 7f40791c6640 1 -- 172.21.15.67:0/2536310860 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40740cd9f0 con 0x7f407415c010 2024-03-24T00:39:13.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.414+0000 7f40791c6640 1 -- 172.21.15.67:0/2536310860 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40740b2300 con 0x7f40740d07a0 2024-03-24T00:39:13.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.414+0000 7f40791c6640 1 -- 172.21.15.67:0/2536310860 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40740b2cd0 con 0x7f40740cccb0 2024-03-24T00:39:13.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.414+0000 7f1b30e68640 1 -- 172.21.15.67:0/1871304661 wait complete. 2024-03-24T00:39:13.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.414+0000 7fd78d71b640 1 -- 172.21.15.67:0/3082185594 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd7800e1090 msgr2=0x7fd7800db740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.414+0000 7fd78d71b640 1 --2- 172.21.15.67:0/3082185594 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd7800e1090 0x7fd7800db740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.414+0000 7f406bfff640 1 -- 172.21.15.67:0/2536310860 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f40740cccb0 msgr2=0x7f40740d21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.414+0000 7f406bfff640 1 --2- 172.21.15.67:0/2536310860 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f40740cccb0 0x7f40740d21a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.414+0000 7f406bfff640 1 -- 172.21.15.67:0/2536310860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f407415c010 msgr2=0x7f40740b4c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.414+0000 7f84adffb640 1 -- 172.21.15.67:0/3192424875 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a8039070 con 0x7f84b015e0a0 2024-03-24T00:39:13.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.414+0000 7f76a0deb640 1 -- 172.21.15.67:0/2474679210 shutdown_connections 2024-03-24T00:39:13.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.414+0000 7f406bfff640 1 --2- 172.21.15.67:0/2536310860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f407415c010 0x7f40740b4c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.415+0000 7f84adffb640 1 -- 172.21.15.67:0/3192424875 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84a802cc60 con 0x7f84b015e0a0 2024-03-24T00:39:13.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.415+0000 7f84adffb640 1 -- 172.21.15.67:0/3192424875 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a8034660 con 0x7f84b015e0a0 2024-03-24T00:39:13.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.414+0000 7f406bfff640 1 -- 172.21.15.67:0/2536310860 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f407415d2b0 con 0x7f40740d07a0 2024-03-24T00:39:13.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.415+0000 7f76a0deb640 1 -- 172.21.15.67:0/2474679210 wait complete. 2024-03-24T00:39:13.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.415+0000 7f406bfff640 1 --2- 172.21.15.67:0/2536310860 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f40740d07a0 0x7f40740b4700 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f405c02bec0 tx=0x7f405c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:13.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.415+0000 7f406b7fe640 1 -- 172.21.15.67:0/2536310860 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f405c041070 con 0x7f40740d07a0 2024-03-24T00:39:13.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.415+0000 7f285f7fe640 1 --2- 172.21.15.67:0/2564460527 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f286015e0a0 0x7f28600d3730 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f28480097a0 tx=0x7f284802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7470c4265cfa2650 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:13.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.415+0000 7f406b7fe640 1 -- 172.21.15.67:0/2536310860 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f405c03ad00 con 0x7f40740d07a0 2024-03-24T00:39:13.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.414+0000 7fd78d71b640 1 -- 172.21.15.67:0/3082185594 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd78003a400 msgr2=0x7fd7800dcc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.414+0000 7fd78d71b640 1 --2- 172.21.15.67:0/3082185594 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd78003a400 0x7fd7800dcc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.414+0000 7fd78d71b640 1 -- 172.21.15.67:0/3082185594 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd78004bb90 con 0x7fd780050dd0 2024-03-24T00:39:13.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.415+0000 7f285e7fc640 1 -- 172.21.15.67:0/2564460527 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2848035020 con 0x7f286015e0a0 2024-03-24T00:39:13.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.414+0000 7fd78df1c640 1 --2- 172.21.15.67:0/3082185594 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd78003a400 0x7fd7800dcc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:13.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.415+0000 7f285e7fc640 1 -- 172.21.15.67:0/2564460527 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2848031810 con 0x7f286015e0a0 2024-03-24T00:39:13.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.415+0000 7f285e7fc640 1 -- 172.21.15.67:0/2564460527 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2848031be0 con 0x7f286015e0a0 2024-03-24T00:39:13.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.415+0000 7fd78d71b640 1 --2- 172.21.15.67:0/3082185594 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd780050dd0 0x7fd7800db200 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd78400d800 tx=0x7fd78400dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:13.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.415+0000 7f40791c6640 1 -- 172.21.15.67:0/2536310860 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f407415dcd0 con 0x7f40740d07a0 2024-03-24T00:39:13.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.416+0000 7f40791c6640 1 -- 172.21.15.67:0/2536310860 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f407415e290 con 0x7f40740d07a0 2024-03-24T00:39:13.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.415+0000 7f9a6a59c640 1 --2- 172.21.15.67:0/2118944058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a54003b40 0x7f9a54003f20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9a60001890 tx=0x7f9a54005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=118b62ff65f3b01 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:13.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.416+0000 7f84b584b640 1 -- 172.21.15.67:0/3192424875 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f84b015e0a0 msgr2=0x7f84b00d3730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.416+0000 7f84b584b640 1 --2- 172.21.15.67:0/3192424875 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f84b015e0a0 0x7f84b00d3730 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f84a8009d90 tx=0x7f84a802beb0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.415+0000 7f9a6959a640 1 -- 172.21.15.67:0/2118944058 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a60001490 con 0x7f9a54003b40 2024-03-24T00:39:13.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.416+0000 7f1b30e68640 1 Processor -- start 2024-03-24T00:39:13.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.416+0000 7f9a6959a640 1 -- 172.21.15.67:0/2118944058 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a60005330 con 0x7f9a54003b40 2024-03-24T00:39:13.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.416+0000 7f84b584b640 1 -- 172.21.15.67:0/3192424875 shutdown_connections 2024-03-24T00:39:13.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.416+0000 7f84b584b640 1 --2- 172.21.15.67:0/3192424875 >> v2:172.21.15.72:3300/0 conn(0x7f84b00b97b0 0x7f84b00d1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.416+0000 7f84b584b640 1 --2- 172.21.15.67:0/3192424875 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f84b015e0a0 0x7f84b00d3730 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.416+0000 7f84b584b640 1 -- 172.21.15.67:0/3192424875 >> 172.21.15.67:0/3192424875 conn(0x7f84b0072da0 msgr2=0x7f84b006c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:13.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.416+0000 7f9a6959a640 1 -- 172.21.15.67:0/2118944058 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a60001490 con 0x7f9a54003b40 2024-03-24T00:39:13.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.417+0000 7f84b584b640 1 -- 172.21.15.67:0/3192424875 shutdown_connections 2024-03-24T00:39:13.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.417+0000 7f76a0deb640 1 Processor -- start 2024-03-24T00:39:13.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.417+0000 7f2866234640 1 -- 172.21.15.67:0/2564460527 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f286015e0a0 msgr2=0x7f28600d3730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.417+0000 7f9a6b59e640 1 -- 172.21.15.67:0/2118944058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a54003b40 msgr2=0x7f9a54003f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.417+0000 7f9a6b59e640 1 --2- 172.21.15.67:0/2118944058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a54003b40 0x7f9a54003f20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9a60001890 tx=0x7f9a54005830 comp rx=0 tx=0).stop 2024-03-24T00:39:13.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.417+0000 7f406b7fe640 1 -- 172.21.15.67:0/2536310860 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f405c035040 con 0x7f40740d07a0 2024-03-24T00:39:13.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.416+0000 7f1b30e68640 1 -- start start 2024-03-24T00:39:13.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.417+0000 7f1b30e68640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b2c0ce540 0x7f1b2c160620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.417+0000 7f2866234640 1 --2- 172.21.15.67:0/2564460527 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f286015e0a0 0x7f28600d3730 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f28480097a0 tx=0x7f284802b8b0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.417+0000 7f9a6ad9d640 1 -- 172.21.15.67:0/2118944058 reap_dead start 2024-03-24T00:39:13.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.417+0000 7f2866234640 1 -- 172.21.15.67:0/2564460527 shutdown_connections 2024-03-24T00:39:13.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.417+0000 7f2866234640 1 --2- 172.21.15.67:0/2564460527 >> v2:172.21.15.67:3300/0 conn(0x7f28600b97b0 0x7f28600d1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.417+0000 7f2866234640 1 --2- 172.21.15.67:0/2564460527 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f286015e0a0 0x7f28600d3730 secure :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f28480097a0 tx=0x7f284802b8b0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.417+0000 7f2866234640 1 -- 172.21.15.67:0/2564460527 >> 172.21.15.67:0/2564460527 conn(0x7f2860072da0 msgr2=0x7f286006c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:13.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.417+0000 7f9a6b59e640 1 -- 172.21.15.67:0/2118944058 shutdown_connections 2024-03-24T00:39:13.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.417+0000 7f76a0deb640 1 -- start start 2024-03-24T00:39:13.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.417+0000 7f9a6b59e640 1 -- 172.21.15.67:0/2118944058 >> 172.21.15.67:0/2118944058 conn(0x7f9a6400a400 msgr2=0x7f9a6400a810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:13.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.415+0000 7fd77e7fc640 1 -- 172.21.15.67:0/3082185594 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78400b6c0 con 0x7fd780050dd0 2024-03-24T00:39:13.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.416+0000 7fd77e7fc640 1 -- 172.21.15.67:0/3082185594 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd784019070 con 0x7fd780050dd0 2024-03-24T00:39:13.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.416+0000 7fd77e7fc640 1 -- 172.21.15.67:0/3082185594 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78400ba80 con 0x7fd780050dd0 2024-03-24T00:39:13.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.416+0000 7fd78ef1e640 1 -- 172.21.15.67:0/3082185594 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd78004be70 con 0x7fd780050dd0 2024-03-24T00:39:13.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.416+0000 7fd78ef1e640 1 -- 172.21.15.67:0/3082185594 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd78004c430 con 0x7fd780050dd0 2024-03-24T00:39:13.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.418+0000 7f1b2a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b2c0ce540 0x7f1b2c160620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.418+0000 7f1b2a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b2c0ce540 0x7f1b2c160620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32888/0 (socket says 172.21.15.67:32888) 2024-03-24T00:39:13.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.418+0000 7f1b2a59c640 1 -- 172.21.15.67:0/23053611 learned_addr learned my addr 172.21.15.67:0/23053611 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:13.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.418+0000 7f1b30e68640 1 --2- 172.21.15.67:0/23053611 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b2c14bbe0 0x7f1b2c142870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.418+0000 7f1b30e68640 1 --2- 172.21.15.67:0/23053611 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1b2c162910 0x7f1b2c145e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.418+0000 7f1b29d9b640 1 --2- 172.21.15.67:0/23053611 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b2c14bbe0 0x7f1b2c142870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.418+0000 7fd77e7fc640 1 -- 172.21.15.67:0/3082185594 <== mon.2 v2:172.21.15.72:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd78400bc20 con 0x7fd780050dd0 2024-03-24T00:39:13.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.418+0000 7fd78ef1e640 1 -- 172.21.15.67:0/3082185594 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7800052c0 con 0x7fd780050dd0 2024-03-24T00:39:13.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.417+0000 7f84b584b640 1 -- 172.21.15.67:0/3192424875 wait complete. 2024-03-24T00:39:13.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.418+0000 7f1b2ad9d640 1 --2- 172.21.15.67:0/23053611 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1b2c162910 0x7f1b2c145e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.418+0000 7f76a0deb640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f769c0b1060 0x7f769c0c9c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.418+0000 7f1b30e68640 1 -- 172.21.15.67:0/23053611 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2c0ce020 con 0x7f1b2c0ce540 2024-03-24T00:39:13.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.418+0000 7f76a0deb640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f769c0d25f0 0x7f769c0ca140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.418+0000 7f1b30e68640 1 -- 172.21.15.67:0/23053611 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2c06f0e0 con 0x7f1b2c14bbe0 2024-03-24T00:39:13.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.418+0000 7f1b30e68640 1 -- 172.21.15.67:0/23053611 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2c06e820 con 0x7f1b2c162910 2024-03-24T00:39:13.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.418+0000 7f76a0deb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f769c15bfa0 0x7f769c1f3b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.418+0000 7f76a0deb640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f769c15c380 con 0x7f769c0b1060 2024-03-24T00:39:13.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.418+0000 7f76a0deb640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f769c06fde0 con 0x7f769c15bfa0 2024-03-24T00:39:13.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.418+0000 7f76a0deb640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f769c0cdab0 con 0x7f769c0d25f0 2024-03-24T00:39:13.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.418+0000 7f769a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f769c0b1060 0x7f769c0c9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.418+0000 7f406b7fe640 1 -- 172.21.15.67:0/2536310860 <== mon.1 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f405c038020 con 0x7f40740d07a0 2024-03-24T00:39:13.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.418+0000 7f769a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f769c0b1060 0x7f769c0c9c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32890/0 (socket says 172.21.15.67:32890) 2024-03-24T00:39:13.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.418+0000 7f769a59c640 1 -- 172.21.15.67:0/3772178982 learned_addr learned my addr 172.21.15.67:0/3772178982 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:13.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.418+0000 7f9a6b59e640 1 -- 172.21.15.67:0/2118944058 shutdown_connections 2024-03-24T00:39:13.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.419+0000 7fd77e7fc640 1 --2- 172.21.15.67:0/3082185594 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd76c06bcd0 0x7fd76c06e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.419+0000 7f9a6b59e640 1 -- 172.21.15.67:0/2118944058 wait complete. 2024-03-24T00:39:13.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.419+0000 7f1b2ad9d640 1 -- 172.21.15.67:0/23053611 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b2c14bbe0 msgr2=0x7f1b2c142870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.419+0000 7fd77e7fc640 1 -- 172.21.15.67:0/3082185594 <== mon.2 v2:172.21.15.72:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fd78408b130 con 0x7fd780050dd0 2024-03-24T00:39:13.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.419+0000 7f1b2ad9d640 1 --2- 172.21.15.67:0/23053611 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b2c14bbe0 0x7f1b2c142870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.419+0000 7f1b2ad9d640 1 -- 172.21.15.67:0/23053611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b2c0ce540 msgr2=0x7f1b2c160620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.419+0000 7f40697fa640 1 -- 172.21.15.67:0/2536310860 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f404c005290 con 0x7f40740d07a0 2024-03-24T00:39:13.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.419+0000 7f769ad9d640 1 --2- 172.21.15.67:0/3772178982 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f769c15bfa0 0x7f769c1f3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.419+0000 7fd78df1c640 1 --2- 172.21.15.67:0/3082185594 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd76c06bcd0 0x7fd76c06e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.419+0000 7f2866234640 1 -- 172.21.15.67:0/2564460527 shutdown_connections 2024-03-24T00:39:13.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.419+0000 7f1b2ad9d640 1 --2- 172.21.15.67:0/23053611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b2c0ce540 0x7f1b2c160620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.419+0000 7f769a59c640 1 -- 172.21.15.67:0/3772178982 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f769c0d25f0 msgr2=0x7f769c0ca140 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-24T00:39:13.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.419+0000 7f406b7fe640 1 --2- 172.21.15.67:0/2536310860 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f405406bbd0 0x7f405406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.419+0000 7f769a59c640 1 --2- 172.21.15.67:0/3772178982 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f769c0d25f0 0x7f769c0ca140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.419+0000 7f1b2ad9d640 1 -- 172.21.15.67:0/23053611 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b2c1613e0 con 0x7f1b2c162910 2024-03-24T00:39:13.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.419+0000 7f769a59c640 1 -- 172.21.15.67:0/3772178982 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f769c15bfa0 msgr2=0x7f769c1f3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.419+0000 7f84b584b640 1 Processor -- start 2024-03-24T00:39:13.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.419+0000 7f769a59c640 1 --2- 172.21.15.67:0/3772178982 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f769c15bfa0 0x7f769c1f3b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.419+0000 7f769a59c640 1 -- 172.21.15.67:0/3772178982 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f769c0b5c10 con 0x7f769c0b1060 2024-03-24T00:39:13.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.420+0000 7f406b7fe640 1 -- 172.21.15.67:0/2536310860 <== mon.1 v2:172.21.15.72:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f405c0accf0 con 0x7f40740d07a0 2024-03-24T00:39:13.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.419+0000 7f769ad9d640 1 --2- 172.21.15.67:0/3772178982 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f769c15bfa0 0x7f769c1f3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:13.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.419+0000 7f769a59c640 1 --2- 172.21.15.67:0/3772178982 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f769c0b1060 0x7f769c0c9c00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f768c000c00 tx=0x7f768c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:13.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.420+0000 7f1b2ad9d640 1 --2- 172.21.15.67:0/23053611 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1b2c162910 0x7f1b2c145e90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c00e7f0 tx=0x7f1b1c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:13.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.420+0000 7f84b584b640 1 -- start start 2024-03-24T00:39:13.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.419+0000 7f2866234640 1 -- 172.21.15.67:0/2564460527 wait complete. 2024-03-24T00:39:13.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.420+0000 7f4072d9d640 1 --2- 172.21.15.67:0/2536310860 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f405406bbd0 0x7f405406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.420+0000 7fd78df1c640 1 --2- 172.21.15.67:0/3082185594 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd76c06bcd0 0x7fd76c06e190 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd778036000 tx=0x7fd778008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:13.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.420+0000 7f7692ffd640 1 -- 172.21.15.67:0/3772178982 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768c003810 con 0x7f769c0b1060 2024-03-24T00:39:13.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.420+0000 7f7692ffd640 1 -- 172.21.15.67:0/3772178982 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f768c0039b0 con 0x7f769c0b1060 2024-03-24T00:39:13.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.420+0000 7f76a0deb640 1 -- 172.21.15.67:0/3772178982 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f769c0b5ef0 con 0x7f769c0b1060 2024-03-24T00:39:13.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.420+0000 7f76a0deb640 1 -- 172.21.15.67:0/3772178982 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f769c0ca680 con 0x7f769c0b1060 2024-03-24T00:39:13.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.421+0000 7f7692ffd640 1 -- 172.21.15.67:0/3772178982 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768c003d60 con 0x7f769c0b1060 2024-03-24T00:39:13.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.421+0000 7f9a6b59e640 1 Processor -- start 2024-03-24T00:39:13.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.421+0000 7f9a6b59e640 1 -- start start 2024-03-24T00:39:13.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.421+0000 7f4072d9d640 1 --2- 172.21.15.67:0/2536310860 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f405406bbd0 0x7f405406e090 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4074071760 tx=0x7f4064007e80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:13.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.421+0000 7f2866234640 1 Processor -- start 2024-03-24T00:39:13.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.421+0000 7f84b584b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f84b00b97b0 0x7f84b00b47c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.421+0000 7f2866234640 1 -- start start 2024-03-24T00:39:13.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.421+0000 7f84b584b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f84b00ceb80 0x7f84b00b5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.421+0000 7f84b584b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84b015e0a0 0x7f84b00b5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.422+0000 7f84aeffd640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f84b00b97b0 0x7f84b00b47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.422+0000 7f84ae7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f84b00ceb80 0x7f84b00b5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.422+0000 7f84aeffd640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f84b00b97b0 0x7f84b00b47c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.67:49316/0 (socket says 172.21.15.67:49316) 2024-03-24T00:39:13.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.422+0000 7f84ae7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f84b00ceb80 0x7f84b00b5500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:34306/0 (socket says 172.21.15.67:34306) 2024-03-24T00:39:13.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.422+0000 7f76a0deb640 1 -- 172.21.15.67:0/3772178982 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7680005290 con 0x7f769c0b1060 2024-03-24T00:39:13.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.422+0000 7f9a6b59e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a640472f0 0x7f9a6404ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.422+0000 7f84b584b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b015e480 con 0x7f84b015e0a0 2024-03-24T00:39:13.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.422+0000 7f84b584b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b00d3210 con 0x7f84b00ceb80 2024-03-24T00:39:13.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.422+0000 7f84b584b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84b00ceff0 con 0x7f84b00b97b0 2024-03-24T00:39:13.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.422+0000 7f84aeffd640 1 -- 172.21.15.67:0/1983703333 learned_addr learned my addr 172.21.15.67:0/1983703333 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:13.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.422+0000 7f84af7fe640 1 --2- 172.21.15.67:0/1983703333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84b015e0a0 0x7f84b00b5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.422+0000 7f9a6b59e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9a640476d0 0x7f9a6404b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.422+0000 7f9a6a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a640472f0 0x7f9a6404ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.422+0000 7f9a6a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a640472f0 0x7f9a6404ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32906/0 (socket says 172.21.15.67:32906) 2024-03-24T00:39:13.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.422+0000 7f9a6a59c640 1 -- 172.21.15.67:0/2367145358 learned_addr learned my addr 172.21.15.67:0/2367145358 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:13.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.422+0000 7f9a6b59e640 1 --2- 172.21.15.67:0/2367145358 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a640dfd50 0x7f9a640e0200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.422+0000 7f9a69d9b640 1 --2- 172.21.15.67:0/2367145358 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9a640476d0 0x7f9a6404b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.422+0000 7f9a6b59e640 1 -- 172.21.15.67:0/2367145358 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a64053430 con 0x7f9a640472f0 2024-03-24T00:39:13.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.423+0000 7f9a6b59e640 1 -- 172.21.15.67:0/2367145358 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a640490e0 con 0x7f9a640dfd50 2024-03-24T00:39:13.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.423+0000 7f9a6b59e640 1 -- 172.21.15.67:0/2367145358 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a64054650 con 0x7f9a640476d0 2024-03-24T00:39:13.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.423+0000 7f9a6ad9d640 1 --2- 172.21.15.67:0/2367145358 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a640dfd50 0x7f9a640e0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.423+0000 7f84aeffd640 1 -- 172.21.15.67:0/1983703333 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f84b00ceb80 msgr2=0x7f84b00b5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.423+0000 7f84aeffd640 1 --2- 172.21.15.67:0/1983703333 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f84b00ceb80 0x7f84b00b5500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.423+0000 7f84aeffd640 1 -- 172.21.15.67:0/1983703333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84b015e0a0 msgr2=0x7f84b00b5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.423+0000 7f84aeffd640 1 --2- 172.21.15.67:0/1983703333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84b015e0a0 0x7f84b00b5a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.423+0000 7f84aeffd640 1 -- 172.21.15.67:0/1983703333 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84b00b60d0 con 0x7f84b00b97b0 2024-03-24T00:39:13.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.423+0000 7f2866234640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f28600b97b0 0x7f2860160f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.423+0000 7f7692ffd640 1 -- 172.21.15.67:0/3772178982 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f768c049060 con 0x7f769c0b1060 2024-03-24T00:39:13.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.423+0000 7f2866234640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f28600ceb80 0x7f28600b71c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.423+0000 7f2866234640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28600c9730 0x7f28600c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.423+0000 7f2866234640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f286015e480 con 0x7f28600c9730 2024-03-24T00:39:13.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.423+0000 7f2866234640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28600d3210 con 0x7f28600ceb80 2024-03-24T00:39:13.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.423+0000 7f2866234640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28600ceff0 con 0x7f28600b97b0 2024-03-24T00:39:13.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.423+0000 7f84aeffd640 1 --2- 172.21.15.67:0/1983703333 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f84b00b97b0 0x7f84b00b47c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f84a8009d90 tx=0x7f84a8004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:13.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.423+0000 7fd77e7fc640 1 -- 172.21.15.67:0/3082185594 <== mon.2 v2:172.21.15.72:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd784059270 con 0x7fd780050dd0 2024-03-24T00:39:13.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.423+0000 7f1b1affd640 1 -- 172.21.15.67:0/23053611 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b1c017070 con 0x7f1b2c162910 2024-03-24T00:39:13.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.423+0000 7f1b1affd640 1 -- 172.21.15.67:0/23053611 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b1c00c730 con 0x7f1b2c162910 2024-03-24T00:39:13.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.423+0000 7f1b1affd640 1 -- 172.21.15.67:0/23053611 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b1c00cb40 con 0x7f1b2c162910 2024-03-24T00:39:13.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.423+0000 7f285effd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f28600ceb80 0x7f28600b71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.423+0000 7f285ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28600c9730 0x7f28600c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.423+0000 7f285f7fe640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f28600b97b0 0x7f2860160f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.423+0000 7f285f7fe640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f28600b97b0 0x7f2860160f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.67:49336/0 (socket says 172.21.15.67:49336) 2024-03-24T00:39:13.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.423+0000 7f285f7fe640 1 -- 172.21.15.67:0/963406612 learned_addr learned my addr 172.21.15.67:0/963406612 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:13.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.423+0000 7f7692ffd640 1 --2- 172.21.15.67:0/3772178982 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f767c06bc20 0x7f767c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.423+0000 7f7692ffd640 1 -- 172.21.15.67:0/3772178982 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f768c0ac960 con 0x7f769c0b1060 2024-03-24T00:39:13.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.423+0000 7f9a6a59c640 1 -- 172.21.15.67:0/2367145358 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9a640476d0 msgr2=0x7f9a6404b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.423+0000 7f9a6a59c640 1 --2- 172.21.15.67:0/2367145358 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9a640476d0 0x7f9a6404b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.423+0000 7f9a6a59c640 1 -- 172.21.15.67:0/2367145358 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a640dfd50 msgr2=0x7f9a640e0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.423+0000 7f9a6a59c640 1 --2- 172.21.15.67:0/2367145358 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a640dfd50 0x7f9a640e0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.423+0000 7f9a6a59c640 1 -- 172.21.15.67:0/2367145358 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a640e7c70 con 0x7f9a640472f0 2024-03-24T00:39:13.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.423+0000 7f8497fff640 1 -- 172.21.15.67:0/1983703333 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a8044020 con 0x7f84b00b97b0 2024-03-24T00:39:13.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.423+0000 7f8497fff640 1 -- 172.21.15.67:0/1983703333 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84a803f470 con 0x7f84b00b97b0 2024-03-24T00:39:13.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.423+0000 7f8497fff640 1 -- 172.21.15.67:0/1983703333 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a8039040 con 0x7f84b00b97b0 2024-03-24T00:39:13.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.424+0000 7f7693fff640 1 --2- 172.21.15.67:0/3772178982 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f767c06bc20 0x7f767c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.424+0000 7f9a6a59c640 1 --2- 172.21.15.67:0/2367145358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a640472f0 0x7f9a6404ac50 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9a60035040 tx=0x7f9a60000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:13.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.423+0000 7f84b584b640 1 -- 172.21.15.67:0/1983703333 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84b00b63b0 con 0x7f84b00b97b0 2024-03-24T00:39:13.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.424+0000 7f1b30e68640 1 -- 172.21.15.67:0/23053611 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b2c1616c0 con 0x7f1b2c162910 2024-03-24T00:39:13.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.424+0000 7f1b30e68640 1 -- 172.21.15.67:0/23053611 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b2c161f60 con 0x7f1b2c162910 2024-03-24T00:39:13.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.424+0000 7f84b584b640 1 -- 172.21.15.67:0/1983703333 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84b00b42a0 con 0x7f84b00b97b0 2024-03-24T00:39:13.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.424+0000 7f285effd640 1 -- 172.21.15.67:0/963406612 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f28600b97b0 msgr2=0x7f2860160f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.424+0000 7f285effd640 1 --2- 172.21.15.67:0/963406612 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f28600b97b0 0x7f2860160f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.424+0000 7f285effd640 1 -- 172.21.15.67:0/963406612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28600c9730 msgr2=0x7f28600c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.424+0000 7f285effd640 1 --2- 172.21.15.67:0/963406612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28600c9730 0x7f28600c9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.424+0000 7f285effd640 1 -- 172.21.15.67:0/963406612 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f286015ff10 con 0x7f28600ceb80 2024-03-24T00:39:13.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.424+0000 7f285ffff640 1 --2- 172.21.15.67:0/963406612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28600c9730 0x7f28600c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:39:13.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.424+0000 7f285effd640 1 --2- 172.21.15.67:0/963406612 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f28600ceb80 0x7f28600b71c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f285000e850 tx=0x7f285000ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:13.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.424+0000 7f2857fff640 1 -- 172.21.15.67:0/963406612 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2850017070 con 0x7f28600ceb80 2024-03-24T00:39:13.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.425+0000 7f2857fff640 1 -- 172.21.15.67:0/963406612 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f285000c940 con 0x7f28600ceb80 2024-03-24T00:39:13.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.425+0000 7f7693fff640 1 --2- 172.21.15.67:0/3772178982 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f767c06bc20 0x7f767c06e0e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f769c15f9e0 tx=0x7f7684009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:13.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.425+0000 7f406b7fe640 1 -- 172.21.15.67:0/2536310860 <== mon.1 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f405c07ae40 con 0x7f40740d07a0 2024-03-24T00:39:13.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.426+0000 7f2866234640 1 -- 172.21.15.67:0/963406612 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28601601b0 con 0x7f28600ceb80 2024-03-24T00:39:13.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.426+0000 7f2866234640 1 -- 172.21.15.67:0/963406612 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2860160770 con 0x7f28600ceb80 2024-03-24T00:39:13.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.426+0000 7f1b1affd640 1 -- 172.21.15.67:0/23053611 <== mon.2 v2:172.21.15.72:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1b1c01a070 con 0x7f1b2c162910 2024-03-24T00:39:13.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.426+0000 7f2857fff640 1 -- 172.21.15.67:0/963406612 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2850017070 con 0x7f28600ceb80 2024-03-24T00:39:13.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.426+0000 7f1b30e68640 1 -- 172.21.15.67:0/23053611 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b2c161890 con 0x7f1b2c162910 2024-03-24T00:39:13.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.427+0000 7f1b1affd640 1 --2- 172.21.15.67:0/23053611 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1b0c06bca0 0x7f1b0c06e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.427+0000 7f1b2a59c640 1 --2- 172.21.15.67:0/23053611 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1b0c06bca0 0x7f1b0c06e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.427+0000 7f1b1affd640 1 -- 172.21.15.67:0/23053611 <== mon.2 v2:172.21.15.72:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f1b1c08ae20 con 0x7f1b2c162910 2024-03-24T00:39:13.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.426+0000 7f84b584b640 1 -- 172.21.15.67:0/1983703333 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84b004f400 con 0x7f84b00b97b0 2024-03-24T00:39:13.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.426+0000 7f8497fff640 1 -- 172.21.15.67:0/1983703333 <== mon.2 v2:172.21.15.72:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f84a8047020 con 0x7f84b00b97b0 2024-03-24T00:39:13.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.427+0000 7f8497fff640 1 --2- 172.21.15.67:0/1983703333 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f849c06bc20 0x7f849c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.427+0000 7f8497fff640 1 -- 172.21.15.67:0/1983703333 <== mon.2 v2:172.21.15.72:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f84a80ad6a0 con 0x7f84b00b97b0 2024-03-24T00:39:13.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.427+0000 7f84ae7fc640 1 --2- 172.21.15.67:0/1983703333 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f849c06bc20 0x7f849c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.427+0000 7f2866234640 1 -- 172.21.15.67:0/963406612 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f286004f400 con 0x7f28600ceb80 2024-03-24T00:39:13.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.428+0000 7f84ae7fc640 1 --2- 172.21.15.67:0/1983703333 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f849c06bc20 0x7f849c06e0e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f84a4009560 tx=0x7f84a4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:13.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.428+0000 7f1b2a59c640 1 --2- 172.21.15.67:0/23053611 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1b0c06bca0 0x7f1b0c06e160 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1b20005880 tx=0x7f1b20009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:13.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.428+0000 7f2857fff640 1 -- 172.21.15.67:0/963406612 <== mon.1 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f285001a070 con 0x7f28600ceb80 2024-03-24T00:39:13.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.429+0000 7f2857fff640 1 --2- 172.21.15.67:0/963406612 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f284006bfd0 0x7f284006e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.429+0000 7f2857fff640 1 -- 172.21.15.67:0/963406612 <== mon.1 v2:172.21.15.72:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f285000f040 con 0x7f28600ceb80 2024-03-24T00:39:13.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.429+0000 7f285f7fe640 1 --2- 172.21.15.67:0/963406612 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f284006bfd0 0x7f284006e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.430+0000 7f285f7fe640 1 --2- 172.21.15.67:0/963406612 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f284006bfd0 0x7f284006e490 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f284802b8f0 tx=0x7f2848008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:13.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.430+0000 7f7692ffd640 1 -- 172.21.15.67:0/3772178982 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f768c033030 con 0x7f769c0b1060 2024-03-24T00:39:13.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.430+0000 7f9a5affd640 1 -- 172.21.15.67:0/2367145358 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a60043020 con 0x7f9a640472f0 2024-03-24T00:39:13.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.430+0000 7f9a5affd640 1 -- 172.21.15.67:0/2367145358 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a6003f690 con 0x7f9a640472f0 2024-03-24T00:39:13.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.430+0000 7f9a6b59e640 1 -- 172.21.15.67:0/2367145358 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a640e7f50 con 0x7f9a640472f0 2024-03-24T00:39:13.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.431+0000 7f9a6b59e640 1 -- 172.21.15.67:0/2367145358 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a640e8510 con 0x7f9a640472f0 2024-03-24T00:39:13.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.431+0000 7f9a5affd640 1 -- 172.21.15.67:0/2367145358 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a60001490 con 0x7f9a640472f0 2024-03-24T00:39:13.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.432+0000 7f9a6b59e640 1 -- 172.21.15.67:0/2367145358 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a64005710 con 0x7f9a640472f0 2024-03-24T00:39:13.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.433+0000 7f9a5affd640 1 -- 172.21.15.67:0/2367145358 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9a60046020 con 0x7f9a640472f0 2024-03-24T00:39:13.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.433+0000 7f2857fff640 1 -- 172.21.15.67:0/963406612 <== mon.1 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2850058760 con 0x7f28600ceb80 2024-03-24T00:39:13.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.433+0000 7f9a5affd640 1 --2- 172.21.15.67:0/2367145358 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9a4c06bbd0 0x7f9a4c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.434+0000 7f9a5affd640 1 -- 172.21.15.67:0/2367145358 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f9a600acd00 con 0x7f9a640472f0 2024-03-24T00:39:13.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.434+0000 7f9a69d9b640 1 --2- 172.21.15.67:0/2367145358 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9a4c06bbd0 0x7f9a4c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.434+0000 7f9a69d9b640 1 --2- 172.21.15.67:0/2367145358 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9a4c06bbd0 0x7f9a4c06e090 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c0690c0 tx=0x7f9a6c08c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:13.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.436+0000 7f1b1affd640 1 -- 172.21.15.67:0/23053611 <== mon.2 v2:172.21.15.72:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1b1c058ff0 con 0x7f1b2c162910 2024-03-24T00:39:13.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.443+0000 7f9a5affd640 1 -- 172.21.15.67:0/2367145358 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9a6007b4c0 con 0x7f9a640472f0 2024-03-24T00:39:13.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.446+0000 7f8497fff640 1 -- 172.21.15.67:0/1983703333 <== mon.2 v2:172.21.15.72:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f84a804e030 con 0x7f84b00b97b0 2024-03-24T00:39:13.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.789+0000 7f74bb4ff640 1 -- 172.21.15.67:0/146962757 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f74b406f280 con 0x7f74b41404a0 2024-03-24T00:39:13.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.790+0000 7f74a9ffb640 1 -- 172.21.15.67:0/146962757 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f74b007f1c0 con 0x7f74b41404a0 2024-03-24T00:39:13.791 INFO:teuthology.orchestra.run.smithi067.stdout:55834574856 2024-03-24T00:39:13.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.793+0000 7f749f7fe640 1 -- 172.21.15.67:0/146962757 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f749806bd00 msgr2=0x7f749806e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.793+0000 7f749f7fe640 1 --2- 172.21.15.67:0/146962757 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f749806bd00 0x7f749806e1c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f74ac005c00 tx=0x7f74ac009040 comp rx=0 tx=0).stop 2024-03-24T00:39:13.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.793+0000 7f749f7fe640 1 -- 172.21.15.67:0/146962757 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74b41404a0 msgr2=0x7f74b4165f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.793+0000 7f749f7fe640 1 --2- 172.21.15.67:0/146962757 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74b41404a0 0x7f74b4165f30 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f74b0009890 tx=0x7f74b00046c0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.794+0000 7f749f7fe640 1 -- 172.21.15.67:0/146962757 shutdown_connections 2024-03-24T00:39:13.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.794+0000 7f749f7fe640 1 --2- 172.21.15.67:0/146962757 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f749806bd00 0x7f749806e1c0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.794+0000 7f749f7fe640 1 --2- 172.21.15.67:0/146962757 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f74b4153770 0x7f74b413f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.794+0000 7f749f7fe640 1 --2- 172.21.15.67:0/146962757 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74b41404a0 0x7f74b4165f30 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.794+0000 7f749f7fe640 1 --2- 172.21.15.67:0/146962757 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f74b4071b00 0x7f74b41f7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.794+0000 7f749f7fe640 1 -- 172.21.15.67:0/146962757 >> 172.21.15.67:0/146962757 conn(0x7f74b4067470 msgr2=0x7f74b406b040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:13.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.794+0000 7f749f7fe640 1 -- 172.21.15.67:0/146962757 shutdown_connections 2024-03-24T00:39:13.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.794+0000 7f749f7fe640 1 -- 172.21.15.67:0/146962757 wait complete. 2024-03-24T00:39:13.811 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.0 2024-03-24T00:39:13.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.830+0000 7f1b30e68640 1 -- 172.21.15.67:0/23053611 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f1b2c1621a0 con 0x7f1b2c162910 2024-03-24T00:39:13.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.830+0000 7f1b1affd640 1 -- 172.21.15.67:0/23053611 <== mon.2 v2:172.21.15.72:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1b1c05d090 con 0x7f1b2c162910 2024-03-24T00:39:13.831 INFO:teuthology.orchestra.run.smithi067.stdout:55834574856 2024-03-24T00:39:13.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.832+0000 7f1b30e68640 1 -- 172.21.15.67:0/23053611 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1b0c06bca0 msgr2=0x7f1b0c06e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.832+0000 7f1b30e68640 1 --2- 172.21.15.67:0/23053611 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1b0c06bca0 0x7f1b0c06e160 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1b20005880 tx=0x7f1b20009040 comp rx=0 tx=0).stop 2024-03-24T00:39:13.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.832+0000 7f1b30e68640 1 -- 172.21.15.67:0/23053611 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1b2c162910 msgr2=0x7f1b2c145e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.833+0000 7f1b30e68640 1 --2- 172.21.15.67:0/23053611 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1b2c162910 0x7f1b2c145e90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c00e7f0 tx=0x7f1b1c00ecb0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.833+0000 7f1b30e68640 1 -- 172.21.15.67:0/23053611 shutdown_connections 2024-03-24T00:39:13.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.833+0000 7f1b30e68640 1 --2- 172.21.15.67:0/23053611 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1b0c06bca0 0x7f1b0c06e160 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.833+0000 7f1b30e68640 1 --2- 172.21.15.67:0/23053611 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1b2c162910 0x7f1b2c145e90 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.833+0000 7f1b30e68640 1 --2- 172.21.15.67:0/23053611 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b2c14bbe0 0x7f1b2c142870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.833+0000 7f1b30e68640 1 --2- 172.21.15.67:0/23053611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b2c0ce540 0x7f1b2c160620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.833+0000 7f1b30e68640 1 -- 172.21.15.67:0/23053611 >> 172.21.15.67:0/23053611 conn(0x7f1b2c06b670 msgr2=0x7f1b2c06e0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:13.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.833+0000 7f1b30e68640 1 -- 172.21.15.67:0/23053611 shutdown_connections 2024-03-24T00:39:13.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.833+0000 7f1b30e68640 1 -- 172.21.15.67:0/23053611 wait complete. 2024-03-24T00:39:13.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.842+0000 7f76a0deb640 1 -- 172.21.15.67:0/3772178982 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f7680005850 con 0x7f769c0b1060 2024-03-24T00:39:13.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.843+0000 7f7692ffd640 1 -- 172.21.15.67:0/3772178982 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f768c07b1f0 con 0x7f769c0b1060 2024-03-24T00:39:13.843 INFO:teuthology.orchestra.run.smithi067.stdout:55834574858 2024-03-24T00:39:13.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.844+0000 7fd78ef1e640 1 -- 172.21.15.67:0/3082185594 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fd780034d20 con 0x7fd780050dd0 2024-03-24T00:39:13.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.844+0000 7fd77e7fc640 1 -- 172.21.15.67:0/3082185594 <== mon.2 v2:172.21.15.72:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd78405d310 con 0x7fd780050dd0 2024-03-24T00:39:13.845 INFO:teuthology.orchestra.run.smithi067.stdout:55834574857 2024-03-24T00:39:13.846 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.3 2024-03-24T00:39:13.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.845+0000 7f76a0deb640 1 -- 172.21.15.67:0/3772178982 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f767c06bc20 msgr2=0x7f767c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.845+0000 7f76a0deb640 1 --2- 172.21.15.67:0/3772178982 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f767c06bc20 0x7f767c06e0e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f769c15f9e0 tx=0x7f7684009190 comp rx=0 tx=0).stop 2024-03-24T00:39:13.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.845+0000 7f76a0deb640 1 -- 172.21.15.67:0/3772178982 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f769c0b1060 msgr2=0x7f769c0c9c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.845+0000 7f76a0deb640 1 --2- 172.21.15.67:0/3772178982 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f769c0b1060 0x7f769c0c9c00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f768c000c00 tx=0x7f768c0046c0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.846+0000 7f76a0deb640 1 -- 172.21.15.67:0/3772178982 shutdown_connections 2024-03-24T00:39:13.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.846+0000 7f76a0deb640 1 --2- 172.21.15.67:0/3772178982 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f767c06bc20 0x7f767c06e0e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.846+0000 7f76a0deb640 1 --2- 172.21.15.67:0/3772178982 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f769c15bfa0 0x7f769c1f3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.846+0000 7f76a0deb640 1 --2- 172.21.15.67:0/3772178982 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f769c0d25f0 0x7f769c0ca140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.846+0000 7f76a0deb640 1 --2- 172.21.15.67:0/3772178982 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f769c0b1060 0x7f769c0c9c00 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.846+0000 7f76a0deb640 1 -- 172.21.15.67:0/3772178982 >> 172.21.15.67:0/3772178982 conn(0x7f769c0d1ad0 msgr2=0x7f769c070200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:13.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.847+0000 7f76a0deb640 1 -- 172.21.15.67:0/3772178982 shutdown_connections 2024-03-24T00:39:13.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.847+0000 7f76a0deb640 1 -- 172.21.15.67:0/3772178982 wait complete. 2024-03-24T00:39:13.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.847+0000 7fd78ef1e640 1 -- 172.21.15.67:0/3082185594 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd76c06bcd0 msgr2=0x7fd76c06e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.847+0000 7fd78ef1e640 1 --2- 172.21.15.67:0/3082185594 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd76c06bcd0 0x7fd76c06e190 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd778036000 tx=0x7fd778008030 comp rx=0 tx=0).stop 2024-03-24T00:39:13.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.847+0000 7fd78ef1e640 1 -- 172.21.15.67:0/3082185594 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd780050dd0 msgr2=0x7fd7800db200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.847+0000 7fd78ef1e640 1 --2- 172.21.15.67:0/3082185594 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd780050dd0 0x7fd7800db200 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd78400d800 tx=0x7fd78400dcc0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.847+0000 7fd78ef1e640 1 -- 172.21.15.67:0/3082185594 shutdown_connections 2024-03-24T00:39:13.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.847+0000 7fd78ef1e640 1 --2- 172.21.15.67:0/3082185594 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd76c06bcd0 0x7fd76c06e190 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.847+0000 7fd78ef1e640 1 --2- 172.21.15.67:0/3082185594 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd7800e1090 0x7fd7800db740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.847+0000 7fd78ef1e640 1 --2- 172.21.15.67:0/3082185594 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd780050dd0 0x7fd7800db200 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.847+0000 7fd78ef1e640 1 --2- 172.21.15.67:0/3082185594 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd78003a400 0x7fd7800dcc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.847+0000 7fd78ef1e640 1 -- 172.21.15.67:0/3082185594 >> 172.21.15.67:0/3082185594 conn(0x7fd780009fb0 msgr2=0x7fd7800e50a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:13.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.847+0000 7fd78ef1e640 1 -- 172.21.15.67:0/3082185594 shutdown_connections 2024-03-24T00:39:13.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.847+0000 7fd78ef1e640 1 -- 172.21.15.67:0/3082185594 wait complete. 2024-03-24T00:39:13.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.853+0000 7f2866234640 1 -- 172.21.15.67:0/963406612 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f286006d1e0 con 0x7f28600ceb80 2024-03-24T00:39:13.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.856+0000 7f2857fff640 1 -- 172.21.15.67:0/963406612 <== mon.1 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f285005c800 con 0x7f28600ceb80 2024-03-24T00:39:13.856 INFO:teuthology.orchestra.run.smithi067.stdout:55834574858 2024-03-24T00:39:13.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.859+0000 7f40697fa640 1 -- 172.21.15.67:0/2536310860 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f404c005580 con 0x7f40740d07a0 2024-03-24T00:39:13.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.859+0000 7f2866234640 1 -- 172.21.15.67:0/963406612 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f284006bfd0 msgr2=0x7f284006e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.859+0000 7f2866234640 1 --2- 172.21.15.67:0/963406612 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f284006bfd0 0x7f284006e490 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f284802b8f0 tx=0x7f2848008030 comp rx=0 tx=0).stop 2024-03-24T00:39:13.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.859+0000 7f2866234640 1 -- 172.21.15.67:0/963406612 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f28600ceb80 msgr2=0x7f28600b71c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.859+0000 7f2866234640 1 --2- 172.21.15.67:0/963406612 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f28600ceb80 0x7f28600b71c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f285000e850 tx=0x7f285000ed10 comp rx=0 tx=0).stop 2024-03-24T00:39:13.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.860+0000 7f2866234640 1 -- 172.21.15.67:0/963406612 shutdown_connections 2024-03-24T00:39:13.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.860+0000 7f2866234640 1 --2- 172.21.15.67:0/963406612 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f284006bfd0 0x7f284006e490 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.860+0000 7f2866234640 1 --2- 172.21.15.67:0/963406612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28600c9730 0x7f28600c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.860+0000 7f2866234640 1 --2- 172.21.15.67:0/963406612 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f28600ceb80 0x7f28600b71c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.860+0000 7f2866234640 1 --2- 172.21.15.67:0/963406612 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f28600b97b0 0x7f2860160f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.860+0000 7f2866234640 1 -- 172.21.15.67:0/963406612 >> 172.21.15.67:0/963406612 conn(0x7f2860072da0 msgr2=0x7f286015d0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:13.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.860+0000 7f2866234640 1 -- 172.21.15.67:0/963406612 shutdown_connections 2024-03-24T00:39:13.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.860+0000 7f2866234640 1 -- 172.21.15.67:0/963406612 wait complete. 2024-03-24T00:39:13.862 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574858 for osd.7 2024-03-24T00:39:13.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.863+0000 7f406b7fe640 1 -- 172.21.15.67:0/2536310860 <== mon.1 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f405c07eee0 con 0x7f40740d07a0 2024-03-24T00:39:13.864 INFO:teuthology.orchestra.run.smithi067.stdout:55834574857 2024-03-24T00:39:13.866 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.5 2024-03-24T00:39:13.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.866+0000 7f9a6b59e640 1 -- 172.21.15.67:0/2367145358 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f9a64047b10 con 0x7f9a640472f0 2024-03-24T00:39:13.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.866+0000 7f9a5affd640 1 -- 172.21.15.67:0/2367145358 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9a6007adc0 con 0x7f9a640472f0 2024-03-24T00:39:13.867 INFO:teuthology.orchestra.run.smithi067.stdout:55834574858 2024-03-24T00:39:13.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.866+0000 7f40697fa640 1 -- 172.21.15.67:0/2536310860 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f405406bbd0 msgr2=0x7f405406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.866+0000 7f40697fa640 1 --2- 172.21.15.67:0/2536310860 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f405406bbd0 0x7f405406e090 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4074071760 tx=0x7f4064007e80 comp rx=0 tx=0).stop 2024-03-24T00:39:13.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.866+0000 7f40697fa640 1 -- 172.21.15.67:0/2536310860 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f40740d07a0 msgr2=0x7f40740b4700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.866+0000 7f40697fa640 1 --2- 172.21.15.67:0/2536310860 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f40740d07a0 0x7f40740b4700 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f405c02bec0 tx=0x7f405c0046c0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.866+0000 7f40697fa640 1 -- 172.21.15.67:0/2536310860 shutdown_connections 2024-03-24T00:39:13.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.866+0000 7f40697fa640 1 --2- 172.21.15.67:0/2536310860 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f405406bbd0 0x7f405406e090 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.866+0000 7f40697fa640 1 --2- 172.21.15.67:0/2536310860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f407415c010 0x7f40740b4c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.866+0000 7f40697fa640 1 --2- 172.21.15.67:0/2536310860 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f40740d07a0 0x7f40740b4700 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.866+0000 7f40697fa640 1 --2- 172.21.15.67:0/2536310860 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f40740cccb0 0x7f40740d21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.866+0000 7f40697fa640 1 -- 172.21.15.67:0/2536310860 >> 172.21.15.67:0/2536310860 conn(0x7f407409aea0 msgr2=0x7f40740ce6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:13.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.866+0000 7f40697fa640 1 -- 172.21.15.67:0/2536310860 shutdown_connections 2024-03-24T00:39:13.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.866+0000 7f40697fa640 1 -- 172.21.15.67:0/2536310860 wait complete. 2024-03-24T00:39:13.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.868+0000 7f9a6b59e640 1 -- 172.21.15.67:0/2367145358 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9a4c06bbd0 msgr2=0x7f9a4c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.868+0000 7f9a6b59e640 1 --2- 172.21.15.67:0/2367145358 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9a4c06bbd0 0x7f9a4c06e090 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c0690c0 tx=0x7f9a6c08c040 comp rx=0 tx=0).stop 2024-03-24T00:39:13.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.868+0000 7f9a6b59e640 1 -- 172.21.15.67:0/2367145358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a640472f0 msgr2=0x7f9a6404ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.868+0000 7f9a6b59e640 1 --2- 172.21.15.67:0/2367145358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a640472f0 0x7f9a6404ac50 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9a60035040 tx=0x7f9a60000bc0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.868+0000 7f9a6b59e640 1 -- 172.21.15.67:0/2367145358 shutdown_connections 2024-03-24T00:39:13.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.868+0000 7f9a6b59e640 1 --2- 172.21.15.67:0/2367145358 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9a4c06bbd0 0x7f9a4c06e090 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.868+0000 7f9a6b59e640 1 --2- 172.21.15.67:0/2367145358 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a640dfd50 0x7f9a640e0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.868+0000 7f9a6b59e640 1 --2- 172.21.15.67:0/2367145358 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9a640476d0 0x7f9a6404b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.868+0000 7f9a6b59e640 1 --2- 172.21.15.67:0/2367145358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a640472f0 0x7f9a6404ac50 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.868+0000 7f9a6b59e640 1 -- 172.21.15.67:0/2367145358 >> 172.21.15.67:0/2367145358 conn(0x7f9a6400a400 msgr2=0x7f9a64005b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:13.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.868+0000 7f84b584b640 1 -- 172.21.15.67:0/1983703333 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f84b00b4470 con 0x7f84b00b97b0 2024-03-24T00:39:13.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.868+0000 7f9a6b59e640 1 -- 172.21.15.67:0/2367145358 shutdown_connections 2024-03-24T00:39:13.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.868+0000 7f9a6b59e640 1 -- 172.21.15.67:0/2367145358 wait complete. 2024-03-24T00:39:13.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.868+0000 7f8497fff640 1 -- 172.21.15.67:0/1983703333 <== mon.2 v2:172.21.15.72:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f84a807b8f0 con 0x7f84b00b97b0 2024-03-24T00:39:13.871 INFO:teuthology.orchestra.run.smithi067.stdout:55834574858 2024-03-24T00:39:13.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.871+0000 7f84b584b640 1 -- 172.21.15.67:0/1983703333 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f849c06bc20 msgr2=0x7f849c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.871+0000 7f84b584b640 1 --2- 172.21.15.67:0/1983703333 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f849c06bc20 0x7f849c06e0e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f84a4009560 tx=0x7f84a4009190 comp rx=0 tx=0).stop 2024-03-24T00:39:13.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.871+0000 7f84b584b640 1 -- 172.21.15.67:0/1983703333 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f84b00b97b0 msgr2=0x7f84b00b47c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.871+0000 7f84b584b640 1 --2- 172.21.15.67:0/1983703333 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f84b00b97b0 0x7f84b00b47c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f84a8009d90 tx=0x7f84a8004a40 comp rx=0 tx=0).stop 2024-03-24T00:39:13.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.871+0000 7f84b584b640 1 -- 172.21.15.67:0/1983703333 shutdown_connections 2024-03-24T00:39:13.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.871+0000 7f84b584b640 1 --2- 172.21.15.67:0/1983703333 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f849c06bc20 0x7f849c06e0e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.871+0000 7f84b584b640 1 --2- 172.21.15.67:0/1983703333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84b015e0a0 0x7f84b00b5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.871+0000 7f84b584b640 1 --2- 172.21.15.67:0/1983703333 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f84b00ceb80 0x7f84b00b5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.871+0000 7f84b584b640 1 --2- 172.21.15.67:0/1983703333 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f84b00b97b0 0x7f84b00b47c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.871+0000 7f84b584b640 1 -- 172.21.15.67:0/1983703333 >> 172.21.15.67:0/1983703333 conn(0x7f84b0072da0 msgr2=0x7f84b00cae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:13.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.871+0000 7f84b584b640 1 -- 172.21.15.67:0/1983703333 shutdown_connections 2024-03-24T00:39:13.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:13.871+0000 7f84b584b640 1 -- 172.21.15.67:0/1983703333 wait complete. 2024-03-24T00:39:13.878 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.1 2024-03-24T00:39:13.878 DEBUG:teuthology.parallel:result is None 2024-03-24T00:39:13.883 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.2 2024-03-24T00:39:13.883 DEBUG:teuthology.parallel:result is None 2024-03-24T00:39:13.884 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.4 2024-03-24T00:39:13.884 DEBUG:teuthology.parallel:result is None 2024-03-24T00:39:13.885 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.6 2024-03-24T00:39:13.885 DEBUG:teuthology.parallel:result is None 2024-03-24T00:39:14.812 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-24T00:39:14.847 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-24T00:39:14.863 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-24T00:39:14.866 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-24T00:39:14.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.930+0000 7f03fbbf4640 1 Processor -- start 2024-03-24T00:39:14.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.930+0000 7f03fbbf4640 1 -- start start 2024-03-24T00:39:14.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.931+0000 7f03fbbf4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f03f40cedc0 0x7f03f4168e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:14.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.931+0000 7f03fbbf4640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03f40acf30 con 0x7f03f4072e20 2024-03-24T00:39:14.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.931+0000 7f03fbbf4640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03f40ad210 con 0x7f03f415e0e0 2024-03-24T00:39:14.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.931+0000 7f03fbbf4640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03f406ca60 con 0x7f03f40cedc0 2024-03-24T00:39:14.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.931+0000 7f03f918f640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f03f4072e20 0x7f03f4072280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47812/0 (socket says 172.21.15.67:47812) 2024-03-24T00:39:14.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.931+0000 7f03f918f640 1 -- 172.21.15.67:0/1588540398 learned_addr learned my addr 172.21.15.67:0/1588540398 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:14.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.931+0000 7f03fa191640 1 --2- 172.21.15.67:0/1588540398 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f03f40cedc0 0x7f03f4168e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:14.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.931+0000 7f03f898e640 1 -- 172.21.15.67:0/1588540398 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2410146462 0 0) 0x7f03f40acf30 con 0x7f03f4072e20 2024-03-24T00:39:14.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.932+0000 7f03fa191640 1 -- 172.21.15.67:0/1588540398 >> v1:172.21.15.72:6789/0 conn(0x7f03f415e0e0 legacy=0x7f03f406f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:14.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.932+0000 7f03fa191640 1 -- 172.21.15.67:0/1588540398 >> v1:172.21.15.67:6789/0 conn(0x7f03f4072e20 legacy=0x7f03f4072280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:14.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.932+0000 7f03fa191640 1 -- 172.21.15.67:0/1588540398 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03f40ad4f0 con 0x7f03f40cedc0 2024-03-24T00:39:14.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.932+0000 7f03fa191640 1 --2- 172.21.15.67:0/1588540398 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f03f40cedc0 0x7f03f4168e20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f03ec009dc0 tx=0x7f03ec02bd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca028522aede832c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:14.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.933+0000 7f03f898e640 1 -- 172.21.15.67:0/1588540398 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ec039070 con 0x7f03f40cedc0 2024-03-24T00:39:14.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.933+0000 7f03f898e640 1 -- 172.21.15.67:0/1588540398 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03ec02caa0 con 0x7f03f40cedc0 2024-03-24T00:39:14.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.933+0000 7f03f898e640 1 -- 172.21.15.67:0/1588540398 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ec034440 con 0x7f03f40cedc0 2024-03-24T00:39:14.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.933+0000 7f03fbbf4640 1 -- 172.21.15.67:0/1588540398 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f03f40cedc0 msgr2=0x7f03f4168e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:14.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.933+0000 7f03fbbf4640 1 --2- 172.21.15.67:0/1588540398 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f03f40cedc0 0x7f03f4168e20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f03ec009dc0 tx=0x7f03ec02bd70 comp rx=0 tx=0).stop 2024-03-24T00:39:14.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.933+0000 7f03fbbf4640 1 -- 172.21.15.67:0/1588540398 shutdown_connections 2024-03-24T00:39:14.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.933+0000 7f03fbbf4640 1 --2- 172.21.15.67:0/1588540398 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f03f40cedc0 0x7f03f4168e20 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:14.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.933+0000 7f03fbbf4640 1 -- 172.21.15.67:0/1588540398 >> 172.21.15.67:0/1588540398 conn(0x7f03f40cd320 msgr2=0x7f03f40d01e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:14.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.933+0000 7f03fbbf4640 1 -- 172.21.15.67:0/1588540398 shutdown_connections 2024-03-24T00:39:14.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.934+0000 7f03fbbf4640 1 -- 172.21.15.67:0/1588540398 wait complete. 2024-03-24T00:39:14.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.935+0000 7f03fbbf4640 1 Processor -- start 2024-03-24T00:39:14.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.935+0000 7f03fbbf4640 1 -- start start 2024-03-24T00:39:14.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.935+0000 7f03fbbf4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f03f4072e20 0x7f03f414e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:14.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.935+0000 7f03fbbf4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03f40cedc0 0x7f03f40c5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:14.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.935+0000 7f03fbbf4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03f415e0e0 0x7f03f40c6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:14.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.936+0000 7f03fbbf4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f406ff80 con 0x7f03f40cedc0 2024-03-24T00:39:14.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.936+0000 7f03fbbf4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f40adb40 con 0x7f03f415e0e0 2024-03-24T00:39:14.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.936+0000 7f03fbbf4640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03f4072990 con 0x7f03f4072e20 2024-03-24T00:39:14.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.936+0000 7f03f918f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03f40cedc0 0x7f03f40c5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:14.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.936+0000 7f03fa191640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03f415e0e0 0x7f03f40c6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:14.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.936+0000 7f03f9990640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f03f4072e20 0x7f03f414e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:14.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.936+0000 7f03f918f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03f40cedc0 0x7f03f40c5ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35908/0 (socket says 172.21.15.67:35908) 2024-03-24T00:39:14.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.936+0000 7f03f918f640 1 -- 172.21.15.67:0/3639046541 learned_addr learned my addr 172.21.15.67:0/3639046541 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:14.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.936+0000 7f03f9990640 1 -- 172.21.15.67:0/3639046541 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03f415e0e0 msgr2=0x7f03f40c6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:14.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.936+0000 7f03f9990640 1 --2- 172.21.15.67:0/3639046541 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03f415e0e0 0x7f03f40c6220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:14.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.936+0000 7f03f9990640 1 -- 172.21.15.67:0/3639046541 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03f40cedc0 msgr2=0x7f03f40c5ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:14.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.936+0000 7f03f9990640 1 --2- 172.21.15.67:0/3639046541 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03f40cedc0 0x7f03f40c5ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:14.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.936+0000 7f03f9990640 1 -- 172.21.15.67:0/3639046541 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03f41f7100 con 0x7f03f4072e20 2024-03-24T00:39:14.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.936+0000 7f03fa191640 1 --2- 172.21.15.67:0/3639046541 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03f415e0e0 0x7f03f40c6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:14.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.936+0000 7f03f9990640 1 --2- 172.21.15.67:0/3639046541 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f03f4072e20 0x7f03f414e660 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f03f000c4e0 tx=0x7f03f000c9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:14.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.936+0000 7f03ea7fc640 1 -- 172.21.15.67:0/3639046541 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f0017070 con 0x7f03f4072e20 2024-03-24T00:39:14.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.937+0000 7f03ea7fc640 1 -- 172.21.15.67:0/3639046541 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03f0005480 con 0x7f03f4072e20 2024-03-24T00:39:14.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.937+0000 7f03fbbf4640 1 -- 172.21.15.67:0/3639046541 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03f41f73e0 con 0x7f03f4072e20 2024-03-24T00:39:14.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.937+0000 7f03ea7fc640 1 -- 172.21.15.67:0/3639046541 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f0003810 con 0x7f03f4072e20 2024-03-24T00:39:14.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.937+0000 7f03fbbf4640 1 -- 172.21.15.67:0/3639046541 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03f41f79a0 con 0x7f03f4072e20 2024-03-24T00:39:14.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.938+0000 7f03ea7fc640 1 -- 172.21.15.67:0/3639046541 <== mon.2 v2:172.21.15.72:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f03f001a070 con 0x7f03f4072e20 2024-03-24T00:39:14.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.938+0000 7f03fbbf4640 1 -- 172.21.15.67:0/3639046541 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03f404f400 con 0x7f03f4072e20 2024-03-24T00:39:14.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.938+0000 7f03ea7fc640 1 --2- 172.21.15.67:0/3639046541 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f03d806bbd0 0x7f03d806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:14.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.938+0000 7f03ea7fc640 1 -- 172.21.15.67:0/3639046541 <== mon.2 v2:172.21.15.72:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f03f008a070 con 0x7f03f4072e20 2024-03-24T00:39:14.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.939+0000 7f03f918f640 1 --2- 172.21.15.67:0/3639046541 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f03d806bbd0 0x7f03d806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:14.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.939+0000 7f03f918f640 1 --2- 172.21.15.67:0/3639046541 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f03d806bbd0 0x7f03d806e090 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f03f41e69d0 tx=0x7f03e4007a60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:14.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.943+0000 7f03ea7fc640 1 -- 172.21.15.67:0/3639046541 <== mon.2 v2:172.21.15.72:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f03f0058320 con 0x7f03f4072e20 2024-03-24T00:39:14.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.968+0000 7f789cd29640 1 Processor -- start 2024-03-24T00:39:14.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.968+0000 7f789cd29640 1 -- start start 2024-03-24T00:39:14.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.969+0000 7f789cd29640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f789815e0f0 0x7f78980b95b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:14.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.969+0000 7f789cd29640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f78980b6dc0 0x7f78980b7230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:14.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.969+0000 7f789cd29640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78980d2230 con 0x7f78980b9af0 2024-03-24T00:39:14.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.969+0000 7f789cd29640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78980b7770 con 0x7f78980b6dc0 2024-03-24T00:39:14.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.969+0000 7f789cd29640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78980d2510 con 0x7f789815e0f0 2024-03-24T00:39:14.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.969+0000 7f788ffff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f78980b9af0 0x7f78980d32d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47822/0 (socket says 172.21.15.67:47822) 2024-03-24T00:39:14.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.969+0000 7f788ffff640 1 -- 172.21.15.67:0/3110792288 learned_addr learned my addr 172.21.15.67:0/3110792288 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:14.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.969+0000 7f7896d9d640 1 --2- 172.21.15.67:0/3110792288 >> v2:172.21.15.67:3300/0 conn(0x7f78980b6dc0 0x7f78980b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:14.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.969+0000 7f789659c640 1 --2- 172.21.15.67:0/3110792288 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f789815e0f0 0x7f78980b95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:14.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.969+0000 7f789659c640 1 -- 172.21.15.67:0/3110792288 >> v2:172.21.15.67:3300/0 conn(0x7f78980b6dc0 msgr2=0x7f78980b7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:14.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.969+0000 7f789659c640 1 --2- 172.21.15.67:0/3110792288 >> v2:172.21.15.67:3300/0 conn(0x7f78980b6dc0 0x7f78980b7230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:14.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.969+0000 7f789659c640 1 -- 172.21.15.67:0/3110792288 >> v1:172.21.15.67:6789/0 conn(0x7f78980b9af0 legacy=0x7f78980d32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:14.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.969+0000 7f789659c640 1 -- 172.21.15.67:0/3110792288 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78980d27f0 con 0x7f789815e0f0 2024-03-24T00:39:14.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.970+0000 7f789659c640 1 --2- 172.21.15.67:0/3110792288 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f789815e0f0 0x7f78980b95b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f78880097a0 tx=0x7f788802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8eb71bf3743e0cd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:14.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.970+0000 7f7895d9b640 1 -- 172.21.15.67:0/3110792288 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7888035020 con 0x7f789815e0f0 2024-03-24T00:39:14.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.971+0000 7f7895d9b640 1 -- 172.21.15.67:0/3110792288 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7888031810 con 0x7f789815e0f0 2024-03-24T00:39:14.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.971+0000 7f7895d9b640 1 -- 172.21.15.67:0/3110792288 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7888031be0 con 0x7f789815e0f0 2024-03-24T00:39:14.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.971+0000 7f789cd29640 1 -- 172.21.15.67:0/3110792288 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f789815e0f0 msgr2=0x7f78980b95b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:14.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.971+0000 7f789cd29640 1 --2- 172.21.15.67:0/3110792288 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f789815e0f0 0x7f78980b95b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f78880097a0 tx=0x7f788802b8b0 comp rx=0 tx=0).stop 2024-03-24T00:39:14.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.971+0000 7f789cd29640 1 -- 172.21.15.67:0/3110792288 shutdown_connections 2024-03-24T00:39:14.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.971+0000 7f789cd29640 1 --2- 172.21.15.67:0/3110792288 >> v2:172.21.15.67:3300/0 conn(0x7f78980b6dc0 0x7f78980b7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:14.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.971+0000 7f789cd29640 1 --2- 172.21.15.67:0/3110792288 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f789815e0f0 0x7f78980b95b0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:14.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.971+0000 7f789cd29640 1 -- 172.21.15.67:0/3110792288 >> 172.21.15.67:0/3110792288 conn(0x7f7898072b60 msgr2=0x7f7898072f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:14.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.971+0000 7f789cd29640 1 -- 172.21.15.67:0/3110792288 shutdown_connections 2024-03-24T00:39:14.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.971+0000 7f789cd29640 1 -- 172.21.15.67:0/3110792288 wait complete. 2024-03-24T00:39:14.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.973+0000 7f789cd29640 1 Processor -- start 2024-03-24T00:39:14.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.973+0000 7f789cd29640 1 -- start start 2024-03-24T00:39:14.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.973+0000 7f789cd29640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78980b6dc0 0x7f78980cde50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:14.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.973+0000 7f789cd29640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78980b9af0 0x7f78981f72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:14.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.973+0000 7f789cd29640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f789815e0f0 0x7f7898154f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:14.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.974+0000 7f789cd29640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78980b31d0 con 0x7f78980b6dc0 2024-03-24T00:39:14.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.974+0000 7f789cd29640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78980d2db0 con 0x7f78980b9af0 2024-03-24T00:39:14.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.974+0000 7f789cd29640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78980b3350 con 0x7f789815e0f0 2024-03-24T00:39:14.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.974+0000 7f789659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78980b6dc0 0x7f78980cde50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:14.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.974+0000 7f789659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78980b6dc0 0x7f78980cde50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35928/0 (socket says 172.21.15.67:35928) 2024-03-24T00:39:14.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.974+0000 7f789659c640 1 -- 172.21.15.67:0/24099109 learned_addr learned my addr 172.21.15.67:0/24099109 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:14.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.974+0000 7f788ffff640 1 --2- 172.21.15.67:0/24099109 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78980b9af0 0x7f78981f72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:14.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.974+0000 7f7896d9d640 1 --2- 172.21.15.67:0/24099109 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f789815e0f0 0x7f7898154f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:14.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.974+0000 7f788ffff640 1 -- 172.21.15.67:0/24099109 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f789815e0f0 msgr2=0x7f7898154f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:14.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.974+0000 7f788ffff640 1 --2- 172.21.15.67:0/24099109 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f789815e0f0 0x7f7898154f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:14.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.974+0000 7f788ffff640 1 -- 172.21.15.67:0/24099109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78980b6dc0 msgr2=0x7f78980cde50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:14.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.974+0000 7f788ffff640 1 --2- 172.21.15.67:0/24099109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78980b6dc0 0x7f78980cde50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:14.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.974+0000 7f788ffff640 1 -- 172.21.15.67:0/24099109 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7898158330 con 0x7f78980b9af0 2024-03-24T00:39:14.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.974+0000 7f789659c640 1 --2- 172.21.15.67:0/24099109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78980b6dc0 0x7f78980cde50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:14.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.974+0000 7f788ffff640 1 --2- 172.21.15.67:0/24099109 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78980b9af0 0x7f78981f72e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f788000c390 tx=0x7f788000c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:14.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.974+0000 7f788effd640 1 -- 172.21.15.67:0/24099109 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7880017070 con 0x7f78980b9af0 2024-03-24T00:39:14.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.974+0000 7f788effd640 1 -- 172.21.15.67:0/24099109 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78800074e0 con 0x7f78980b9af0 2024-03-24T00:39:14.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.974+0000 7f789cd29640 1 -- 172.21.15.67:0/24099109 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f789815f110 con 0x7f78980b9af0 2024-03-24T00:39:14.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.974+0000 7f788effd640 1 -- 172.21.15.67:0/24099109 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78800059a0 con 0x7f78980b9af0 2024-03-24T00:39:14.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.974+0000 7f789cd29640 1 -- 172.21.15.67:0/24099109 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f789815f6d0 con 0x7f78980b9af0 2024-03-24T00:39:14.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.976+0000 7f789cd29640 1 -- 172.21.15.67:0/24099109 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f789815f8d0 con 0x7f78980b9af0 2024-03-24T00:39:14.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.976+0000 7f788effd640 1 -- 172.21.15.67:0/24099109 <== mon.1 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f788001a070 con 0x7f78980b9af0 2024-03-24T00:39:14.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.976+0000 7f788effd640 1 --2- 172.21.15.67:0/24099109 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f787806bbd0 0x7f787806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:14.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.976+0000 7f788effd640 1 -- 172.21.15.67:0/24099109 <== mon.1 v2:172.21.15.72:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f7880089fa0 con 0x7f78980b9af0 2024-03-24T00:39:14.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.976+0000 7f789659c640 1 --2- 172.21.15.67:0/24099109 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f787806bbd0 0x7f787806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:14.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.977+0000 7f789659c640 1 --2- 172.21.15.67:0/24099109 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f787806bbd0 0x7f787806e090 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f788802b8f0 tx=0x7f788803c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:14.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:14.981+0000 7f788effd640 1 -- 172.21.15.67:0/24099109 <== mon.1 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f789815f8d0 con 0x7f78980b9af0 2024-03-24T00:39:15.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.009+0000 7f047c9f1640 1 Processor -- start 2024-03-24T00:39:15.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.009+0000 7f047c9f1640 1 -- start start 2024-03-24T00:39:15.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.010+0000 7f047c9f1640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f047815e0f0 0x7f04780b95b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:15.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.010+0000 7f047c9f1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04780b9af0 0x7f04780d32d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:15.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.010+0000 7f047c9f1640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04780d2230 con 0x7f04780d3810 2024-03-24T00:39:15.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.010+0000 7f047c9f1640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04780d2510 con 0x7f047815e0f0 2024-03-24T00:39:15.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.010+0000 7f047c9f1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04780d27f0 con 0x7f04780b9af0 2024-03-24T00:39:15.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.010+0000 7f047659c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f047815e0f0 0x7f04780b95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:15.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.010+0000 7f0476d9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f04780d3810 0x7f04780b6dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47836/0 (socket says 172.21.15.67:47836) 2024-03-24T00:39:15.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.010+0000 7f0476d9d640 1 -- 172.21.15.67:0/2009739574 learned_addr learned my addr 172.21.15.67:0/2009739574 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:15.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.010+0000 7f0475d9b640 1 --2- 172.21.15.67:0/2009739574 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04780b9af0 0x7f04780d32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:15.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.011+0000 7f0475d9b640 1 -- 172.21.15.67:0/2009739574 >> v2:172.21.15.67:3300/0 conn(0x7f047815e0f0 msgr2=0x7f04780b95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:15.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.011+0000 7f0475d9b640 1 --2- 172.21.15.67:0/2009739574 >> v2:172.21.15.67:3300/0 conn(0x7f047815e0f0 0x7f04780b95b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.011+0000 7f0475d9b640 1 -- 172.21.15.67:0/2009739574 >> v1:172.21.15.67:6789/0 conn(0x7f04780d3810 legacy=0x7f04780b6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:15.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.011+0000 7f0475d9b640 1 -- 172.21.15.67:0/2009739574 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04780d2ad0 con 0x7f04780b9af0 2024-03-24T00:39:15.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.011+0000 7f0475d9b640 1 --2- 172.21.15.67:0/2009739574 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04780b9af0 0x7f04780d32d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f046c009790 tx=0x7f046c02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7e5e0b90a96e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:15.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.012+0000 7f047559a640 1 -- 172.21.15.67:0/2009739574 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f046c035020 con 0x7f04780b9af0 2024-03-24T00:39:15.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.012+0000 7f047559a640 1 -- 172.21.15.67:0/2009739574 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f046c031810 con 0x7f04780b9af0 2024-03-24T00:39:15.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.012+0000 7f047559a640 1 -- 172.21.15.67:0/2009739574 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f046c031be0 con 0x7f04780b9af0 2024-03-24T00:39:15.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.012+0000 7f047c9f1640 1 -- 172.21.15.67:0/2009739574 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04780b9af0 msgr2=0x7f04780d32d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:15.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.012+0000 7f047c9f1640 1 --2- 172.21.15.67:0/2009739574 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04780b9af0 0x7f04780d32d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f046c009790 tx=0x7f046c02b8b0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.012+0000 7f047c9f1640 1 -- 172.21.15.67:0/2009739574 shutdown_connections 2024-03-24T00:39:15.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.012+0000 7f047c9f1640 1 --2- 172.21.15.67:0/2009739574 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04780b9af0 0x7f04780d32d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.012+0000 7f047c9f1640 1 --2- 172.21.15.67:0/2009739574 >> v2:172.21.15.67:3300/0 conn(0x7f047815e0f0 0x7f04780b95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.012+0000 7f047c9f1640 1 -- 172.21.15.67:0/2009739574 >> 172.21.15.67:0/2009739574 conn(0x7f0478072b60 msgr2=0x7f0478072f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:15.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.013+0000 7f047c9f1640 1 -- 172.21.15.67:0/2009739574 shutdown_connections 2024-03-24T00:39:15.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.013+0000 7f047c9f1640 1 -- 172.21.15.67:0/2009739574 wait complete. 2024-03-24T00:39:15.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.014+0000 7f047c9f1640 1 Processor -- start 2024-03-24T00:39:15.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.014+0000 7f047c9f1640 1 -- start start 2024-03-24T00:39:15.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.015+0000 7f047c9f1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04780b9af0 0x7f04780b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:15.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.015+0000 7f047c9f1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04780d3810 0x7f047815f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:15.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.015+0000 7f047c9f1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f047815e0f0 0x7f0478147f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:15.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.015+0000 7f047c9f1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04780b3350 con 0x7f04780d3810 2024-03-24T00:39:15.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.015+0000 7f047c9f1640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04780b31d0 con 0x7f047815e0f0 2024-03-24T00:39:15.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.015+0000 7f047c9f1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04780b74d0 con 0x7f04780b9af0 2024-03-24T00:39:15.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.015+0000 7f0475d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04780d3810 0x7f047815f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:15.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.015+0000 7f0475d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04780d3810 0x7f047815f9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35948/0 (socket says 172.21.15.67:35948) 2024-03-24T00:39:15.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.015+0000 7f0475d9b640 1 -- 172.21.15.67:0/2209159978 learned_addr learned my addr 172.21.15.67:0/2209159978 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:15.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.015+0000 7f047659c640 1 --2- 172.21.15.67:0/2209159978 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04780b9af0 0x7f04780b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:15.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.015+0000 7f0476d9d640 1 --2- 172.21.15.67:0/2209159978 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f047815e0f0 0x7f0478147f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:15.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.015+0000 7f047659c640 1 -- 172.21.15.67:0/2209159978 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f047815e0f0 msgr2=0x7f0478147f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:15.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.015+0000 7f047659c640 1 --2- 172.21.15.67:0/2209159978 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f047815e0f0 0x7f0478147f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.015+0000 7f047659c640 1 -- 172.21.15.67:0/2209159978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04780d3810 msgr2=0x7f047815f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:15.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.015+0000 7f047659c640 1 --2- 172.21.15.67:0/2209159978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04780d3810 0x7f047815f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.015+0000 7f047659c640 1 -- 172.21.15.67:0/2209159978 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04781604a0 con 0x7f04780b9af0 2024-03-24T00:39:15.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.015+0000 7f0475d9b640 1 --2- 172.21.15.67:0/2209159978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04780d3810 0x7f047815f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:15.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.015+0000 7f047659c640 1 --2- 172.21.15.67:0/2209159978 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04780b9af0 0x7f04780b5d50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f046800e870 tx=0x7f046800ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:15.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.015+0000 7f0466ffd640 1 -- 172.21.15.67:0/2209159978 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0468017070 con 0x7f04780b9af0 2024-03-24T00:39:15.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.016+0000 7f0466ffd640 1 -- 172.21.15.67:0/2209159978 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f046800ee60 con 0x7f04780b9af0 2024-03-24T00:39:15.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.016+0000 7f0466ffd640 1 -- 172.21.15.67:0/2209159978 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f046800cb60 con 0x7f04780b9af0 2024-03-24T00:39:15.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.016+0000 7f047c9f1640 1 -- 172.21.15.67:0/2209159978 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0478160780 con 0x7f04780b9af0 2024-03-24T00:39:15.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.016+0000 7f047c9f1640 1 -- 172.21.15.67:0/2209159978 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0478160d40 con 0x7f04780b9af0 2024-03-24T00:39:15.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.017+0000 7f047c9f1640 1 -- 172.21.15.67:0/2209159978 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0478145f00 con 0x7f04780b9af0 2024-03-24T00:39:15.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.017+0000 7f0466ffd640 1 -- 172.21.15.67:0/2209159978 <== mon.2 v2:172.21.15.72:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f046801a070 con 0x7f04780b9af0 2024-03-24T00:39:15.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.017+0000 7f0466ffd640 1 --2- 172.21.15.67:0/2209159978 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f045806bbd0 0x7f045806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:15.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.017+0000 7f0466ffd640 1 -- 172.21.15.67:0/2209159978 <== mon.2 v2:172.21.15.72:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f046808aaf0 con 0x7f04780b9af0 2024-03-24T00:39:15.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.018+0000 7f0475d9b640 1 --2- 172.21.15.67:0/2209159978 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f045806bbd0 0x7f045806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:15.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.018+0000 7f0475d9b640 1 --2- 172.21.15.67:0/2209159978 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f045806bbd0 0x7f045806e090 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0478145dd0 tx=0x7f046c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:15.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.022+0000 7f0466ffd640 1 -- 172.21.15.67:0/2209159978 <== mon.2 v2:172.21.15.72:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0468058c40 con 0x7f04780b9af0 2024-03-24T00:39:15.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.045+0000 7f3884cbf640 1 Processor -- start 2024-03-24T00:39:15.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.045+0000 7f3884cbf640 1 -- start start 2024-03-24T00:39:15.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.046+0000 7f3884cbf640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f388015e0f0 0x7f38800b95b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:15.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.046+0000 7f3884cbf640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f38800b6dc0 0x7f38800b7230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:15.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.046+0000 7f3884cbf640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38800d2230 con 0x7f38800b9af0 2024-03-24T00:39:15.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.046+0000 7f3884cbf640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38800b7770 con 0x7f38800b6dc0 2024-03-24T00:39:15.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.046+0000 7f3884cbf640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38800d2510 con 0x7f388015e0f0 2024-03-24T00:39:15.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.046+0000 7f387e59c640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f388015e0f0 0x7f38800b95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:15.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.046+0000 7f387e59c640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f388015e0f0 0x7f38800b95b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.67:42694/0 (socket says 172.21.15.67:42694) 2024-03-24T00:39:15.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.046+0000 7f387dd9b640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f38800b9af0 0x7f38800d32d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:49952/0 (socket says 172.21.15.67:49952) 2024-03-24T00:39:15.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.046+0000 7f387e59c640 1 -- 172.21.15.67:0/860502138 learned_addr learned my addr 172.21.15.67:0/860502138 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:15.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.046+0000 7f387ed9d640 1 --2- 172.21.15.67:0/860502138 >> v2:172.21.15.67:3300/0 conn(0x7f38800b6dc0 0x7f38800b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:15.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.046+0000 7f387d59a640 1 -- 172.21.15.67:0/860502138 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2162625180 0 0) 0x7f38800d2230 con 0x7f38800b9af0 2024-03-24T00:39:15.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.046+0000 7f387e59c640 1 -- 172.21.15.67:0/860502138 >> v2:172.21.15.67:3300/0 conn(0x7f38800b6dc0 msgr2=0x7f38800b7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:15.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.046+0000 7f387e59c640 1 --2- 172.21.15.67:0/860502138 >> v2:172.21.15.67:3300/0 conn(0x7f38800b6dc0 0x7f38800b7230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.046+0000 7f387e59c640 1 -- 172.21.15.67:0/860502138 >> v1:172.21.15.72:6789/0 conn(0x7f38800b9af0 legacy=0x7f38800d32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:15.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.046+0000 7f387e59c640 1 -- 172.21.15.67:0/860502138 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38800d27f0 con 0x7f388015e0f0 2024-03-24T00:39:15.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.048+0000 7f387e59c640 1 --2- 172.21.15.67:0/860502138 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f388015e0f0 0x7f38800b95b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3870009870 tx=0x7f387002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3840877f482a004 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:15.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.048+0000 7f387d59a640 1 -- 172.21.15.67:0/860502138 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3870034070 con 0x7f388015e0f0 2024-03-24T00:39:15.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.048+0000 7f387d59a640 1 -- 172.21.15.67:0/860502138 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3870030840 con 0x7f388015e0f0 2024-03-24T00:39:15.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.048+0000 7f387d59a640 1 -- 172.21.15.67:0/860502138 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3870030c10 con 0x7f388015e0f0 2024-03-24T00:39:15.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.049+0000 7f3884cbf640 1 -- 172.21.15.67:0/860502138 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f388015e0f0 msgr2=0x7f38800b95b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:15.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.049+0000 7f3884cbf640 1 --2- 172.21.15.67:0/860502138 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f388015e0f0 0x7f38800b95b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3870009870 tx=0x7f387002b980 comp rx=0 tx=0).stop 2024-03-24T00:39:15.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.049+0000 7f3884cbf640 1 -- 172.21.15.67:0/860502138 shutdown_connections 2024-03-24T00:39:15.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.049+0000 7f3884cbf640 1 --2- 172.21.15.67:0/860502138 >> v2:172.21.15.67:3300/0 conn(0x7f38800b6dc0 0x7f38800b7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.049+0000 7f3884cbf640 1 --2- 172.21.15.67:0/860502138 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f388015e0f0 0x7f38800b95b0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.049+0000 7f3884cbf640 1 -- 172.21.15.67:0/860502138 >> 172.21.15.67:0/860502138 conn(0x7f3880072b60 msgr2=0x7f3880072f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:15.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.049+0000 7f3884cbf640 1 -- 172.21.15.67:0/860502138 shutdown_connections 2024-03-24T00:39:15.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.049+0000 7f3884cbf640 1 -- 172.21.15.67:0/860502138 wait complete. 2024-03-24T00:39:15.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.051+0000 7f3884cbf640 1 Processor -- start 2024-03-24T00:39:15.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.051+0000 7f3884cbf640 1 -- start start 2024-03-24T00:39:15.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.052+0000 7f3884cbf640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38800b6dc0 0x7f38801e69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:15.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.052+0000 7f3884cbf640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f38800b9af0 0x7f3880160d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:15.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.052+0000 7f3884cbf640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f388015e0f0 0x7f388015f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:15.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.052+0000 7f3884cbf640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38800b31d0 con 0x7f38800b6dc0 2024-03-24T00:39:15.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.052+0000 7f3884cbf640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38800d2db0 con 0x7f388015e0f0 2024-03-24T00:39:15.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.052+0000 7f3884cbf640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38800b3350 con 0x7f38800b9af0 2024-03-24T00:39:15.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.052+0000 7f387ed9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f388015e0f0 0x7f388015f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:15.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.052+0000 7f387dd9b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f38800b9af0 0x7f3880160d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:15.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.052+0000 7f387ed9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f388015e0f0 0x7f388015f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:47678/0 (socket says 172.21.15.67:47678) 2024-03-24T00:39:15.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.052+0000 7f387ed9d640 1 -- 172.21.15.67:0/1453742760 learned_addr learned my addr 172.21.15.67:0/1453742760 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:15.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.052+0000 7f387e59c640 1 --2- 172.21.15.67:0/1453742760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38800b6dc0 0x7f38801e69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:15.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.053+0000 7f387ed9d640 1 -- 172.21.15.67:0/1453742760 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f38800b9af0 msgr2=0x7f3880160d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:15.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.053+0000 7f387ed9d640 1 --2- 172.21.15.67:0/1453742760 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f38800b9af0 0x7f3880160d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.053+0000 7f387ed9d640 1 -- 172.21.15.67:0/1453742760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38800b6dc0 msgr2=0x7f38801e69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:15.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.053+0000 7f387ed9d640 1 --2- 172.21.15.67:0/1453742760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38800b6dc0 0x7f38801e69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.053+0000 7f387ed9d640 1 -- 172.21.15.67:0/1453742760 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38800b4770 con 0x7f388015e0f0 2024-03-24T00:39:15.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.053+0000 7f387dd9b640 1 --2- 172.21.15.67:0/1453742760 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f38800b9af0 0x7f3880160d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:15.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.053+0000 7f387ed9d640 1 --2- 172.21.15.67:0/1453742760 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f388015e0f0 0x7f388015f310 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f387800efc0 tx=0x7f387800c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:15.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.053+0000 7f386effd640 1 -- 172.21.15.67:0/1453742760 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3878018020 con 0x7f388015e0f0 2024-03-24T00:39:15.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.053+0000 7f386effd640 1 -- 172.21.15.67:0/1453742760 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3878014480 con 0x7f388015e0f0 2024-03-24T00:39:15.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.053+0000 7f3884cbf640 1 -- 172.21.15.67:0/1453742760 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38800b4a50 con 0x7f388015e0f0 2024-03-24T00:39:15.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.053+0000 7f386effd640 1 -- 172.21.15.67:0/1453742760 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3878014870 con 0x7f388015e0f0 2024-03-24T00:39:15.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.053+0000 7f3884cbf640 1 -- 172.21.15.67:0/1453742760 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3880161250 con 0x7f388015e0f0 2024-03-24T00:39:15.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.055+0000 7f386cff9640 1 -- 172.21.15.67:0/1453742760 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3858005290 con 0x7f388015e0f0 2024-03-24T00:39:15.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.055+0000 7f386effd640 1 -- 172.21.15.67:0/1453742760 <== mon.1 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3878005d10 con 0x7f388015e0f0 2024-03-24T00:39:15.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.056+0000 7f386effd640 1 --2- 172.21.15.67:0/1453742760 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f386006bc00 0x7f386006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:15.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.056+0000 7f386effd640 1 -- 172.21.15.67:0/1453742760 <== mon.1 v2:172.21.15.72:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f387808ba10 con 0x7f388015e0f0 2024-03-24T00:39:15.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.056+0000 7f387e59c640 1 --2- 172.21.15.67:0/1453742760 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f386006bc00 0x7f386006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:15.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.056+0000 7f387e59c640 1 --2- 172.21.15.67:0/1453742760 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f386006bc00 0x7f386006e0c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3870036000 tx=0x7f387003b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:15.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.060+0000 7f386effd640 1 -- 172.21.15.67:0/1453742760 <== mon.1 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3878022030 con 0x7f388015e0f0 2024-03-24T00:39:15.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.170+0000 7f789cd29640 1 -- 172.21.15.67:0/24099109 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f789804f400 con 0x7f78980b9af0 2024-03-24T00:39:15.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.170+0000 7f788effd640 1 -- 172.21.15.67:0/24099109 <== mon.1 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7880058100 con 0x7f78980b9af0 2024-03-24T00:39:15.171 INFO:teuthology.orchestra.run.smithi067.stdout:55834574857 2024-03-24T00:39:15.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.172+0000 7f789cd29640 1 -- 172.21.15.67:0/24099109 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f787806bbd0 msgr2=0x7f787806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:15.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.172+0000 7f789cd29640 1 --2- 172.21.15.67:0/24099109 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f787806bbd0 0x7f787806e090 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f788802b8f0 tx=0x7f788803c040 comp rx=0 tx=0).stop 2024-03-24T00:39:15.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.172+0000 7f789cd29640 1 -- 172.21.15.67:0/24099109 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78980b9af0 msgr2=0x7f78981f72e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:15.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.172+0000 7f789cd29640 1 --2- 172.21.15.67:0/24099109 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78980b9af0 0x7f78981f72e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f788000c390 tx=0x7f788000c850 comp rx=0 tx=0).stop 2024-03-24T00:39:15.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.172+0000 7f789cd29640 1 -- 172.21.15.67:0/24099109 shutdown_connections 2024-03-24T00:39:15.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.172+0000 7f789cd29640 1 --2- 172.21.15.67:0/24099109 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f787806bbd0 0x7f787806e090 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.172+0000 7f789cd29640 1 --2- 172.21.15.67:0/24099109 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f789815e0f0 0x7f7898154f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.172+0000 7f789cd29640 1 --2- 172.21.15.67:0/24099109 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78980b9af0 0x7f78981f72e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.172+0000 7f789cd29640 1 --2- 172.21.15.67:0/24099109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78980b6dc0 0x7f78980cde50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.172+0000 7f789cd29640 1 -- 172.21.15.67:0/24099109 >> 172.21.15.67:0/24099109 conn(0x7f7898072b60 msgr2=0x7f78980d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:15.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.173+0000 7f789cd29640 1 -- 172.21.15.67:0/24099109 shutdown_connections 2024-03-24T00:39:15.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.173+0000 7f789cd29640 1 -- 172.21.15.67:0/24099109 wait complete. 2024-03-24T00:39:15.188 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.3 2024-03-24T00:39:15.189 DEBUG:teuthology.parallel:result is None 2024-03-24T00:39:15.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.217+0000 7f047c9f1640 1 -- 172.21.15.67:0/2209159978 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f04780b95f0 con 0x7f04780b9af0 2024-03-24T00:39:15.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.218+0000 7f0466ffd640 1 -- 172.21.15.67:0/2209159978 <== mon.2 v2:172.21.15.72:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f046805cce0 con 0x7f04780b9af0 2024-03-24T00:39:15.218 INFO:teuthology.orchestra.run.smithi067.stdout:55834574859 2024-03-24T00:39:15.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.220+0000 7f047c9f1640 1 -- 172.21.15.67:0/2209159978 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f045806bbd0 msgr2=0x7f045806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:15.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.220+0000 7f047c9f1640 1 --2- 172.21.15.67:0/2209159978 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f045806bbd0 0x7f045806e090 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0478145dd0 tx=0x7f046c008030 comp rx=0 tx=0).stop 2024-03-24T00:39:15.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.220+0000 7f047c9f1640 1 -- 172.21.15.67:0/2209159978 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04780b9af0 msgr2=0x7f04780b5d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:15.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.220+0000 7f047c9f1640 1 --2- 172.21.15.67:0/2209159978 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04780b9af0 0x7f04780b5d50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f046800e870 tx=0x7f046800ed30 comp rx=0 tx=0).stop 2024-03-24T00:39:15.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.220+0000 7f047c9f1640 1 -- 172.21.15.67:0/2209159978 shutdown_connections 2024-03-24T00:39:15.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.220+0000 7f047c9f1640 1 --2- 172.21.15.67:0/2209159978 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f045806bbd0 0x7f045806e090 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.220+0000 7f047c9f1640 1 --2- 172.21.15.67:0/2209159978 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f047815e0f0 0x7f0478147f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.220+0000 7f047c9f1640 1 --2- 172.21.15.67:0/2209159978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04780d3810 0x7f047815f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.220+0000 7f047c9f1640 1 --2- 172.21.15.67:0/2209159978 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04780b9af0 0x7f04780b5d50 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.220+0000 7f047c9f1640 1 -- 172.21.15.67:0/2209159978 >> 172.21.15.67:0/2209159978 conn(0x7f0478072b60 msgr2=0x7f04780b6910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:15.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.220+0000 7f047c9f1640 1 -- 172.21.15.67:0/2209159978 shutdown_connections 2024-03-24T00:39:15.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.220+0000 7f047c9f1640 1 -- 172.21.15.67:0/2209159978 wait complete. 2024-03-24T00:39:15.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.223+0000 7f03fbbf4640 1 -- 172.21.15.67:0/3639046541 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f03f40ae520 con 0x7f03f4072e20 2024-03-24T00:39:15.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.223+0000 7f03ea7fc640 1 -- 172.21.15.67:0/3639046541 <== mon.2 v2:172.21.15.72:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f03f005c3c0 con 0x7f03f4072e20 2024-03-24T00:39:15.224 INFO:teuthology.orchestra.run.smithi067.stdout:55834574857 2024-03-24T00:39:15.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.227+0000 7f03fbbf4640 1 -- 172.21.15.67:0/3639046541 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f03d806bbd0 msgr2=0x7f03d806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:15.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.227+0000 7f03fbbf4640 1 --2- 172.21.15.67:0/3639046541 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f03d806bbd0 0x7f03d806e090 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f03f41e69d0 tx=0x7f03e4007a60 comp rx=0 tx=0).stop 2024-03-24T00:39:15.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.227+0000 7f03fbbf4640 1 -- 172.21.15.67:0/3639046541 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f03f4072e20 msgr2=0x7f03f414e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:15.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.227+0000 7f03fbbf4640 1 --2- 172.21.15.67:0/3639046541 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f03f4072e20 0x7f03f414e660 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f03f000c4e0 tx=0x7f03f000c9a0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.227+0000 7f03fbbf4640 1 -- 172.21.15.67:0/3639046541 shutdown_connections 2024-03-24T00:39:15.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.227+0000 7f03fbbf4640 1 --2- 172.21.15.67:0/3639046541 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f03d806bbd0 0x7f03d806e090 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.227+0000 7f03fbbf4640 1 --2- 172.21.15.67:0/3639046541 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03f415e0e0 0x7f03f40c6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.227+0000 7f03fbbf4640 1 --2- 172.21.15.67:0/3639046541 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03f40cedc0 0x7f03f40c5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.227+0000 7f03fbbf4640 1 --2- 172.21.15.67:0/3639046541 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f03f4072e20 0x7f03f414e660 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.227+0000 7f03fbbf4640 1 -- 172.21.15.67:0/3639046541 >> 172.21.15.67:0/3639046541 conn(0x7f03f40cd320 msgr2=0x7f03f40b19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:15.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.227+0000 7f03fbbf4640 1 -- 172.21.15.67:0/3639046541 shutdown_connections 2024-03-24T00:39:15.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.227+0000 7f03fbbf4640 1 -- 172.21.15.67:0/3639046541 wait complete. 2024-03-24T00:39:15.231 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.7 2024-03-24T00:39:15.232 DEBUG:teuthology.parallel:result is None 2024-03-24T00:39:15.242 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.0 2024-03-24T00:39:15.242 DEBUG:teuthology.parallel:result is None 2024-03-24T00:39:15.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.296+0000 7f386cff9640 1 -- 172.21.15.67:0/1453742760 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f3858005850 con 0x7f388015e0f0 2024-03-24T00:39:15.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.297+0000 7f386effd640 1 -- 172.21.15.67:0/1453742760 <== mon.1 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3878010040 con 0x7f388015e0f0 2024-03-24T00:39:15.298 INFO:teuthology.orchestra.run.smithi067.stdout:55834574858 2024-03-24T00:39:15.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.298+0000 7f386cff9640 1 -- 172.21.15.67:0/1453742760 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f386006bc00 msgr2=0x7f386006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:15.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.298+0000 7f386cff9640 1 --2- 172.21.15.67:0/1453742760 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f386006bc00 0x7f386006e0c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3870036000 tx=0x7f387003b040 comp rx=0 tx=0).stop 2024-03-24T00:39:15.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.298+0000 7f386cff9640 1 -- 172.21.15.67:0/1453742760 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f388015e0f0 msgr2=0x7f388015f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:15.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.298+0000 7f386cff9640 1 --2- 172.21.15.67:0/1453742760 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f388015e0f0 0x7f388015f310 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f387800efc0 tx=0x7f387800c490 comp rx=0 tx=0).stop 2024-03-24T00:39:15.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.299+0000 7f386cff9640 1 -- 172.21.15.67:0/1453742760 shutdown_connections 2024-03-24T00:39:15.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.299+0000 7f386cff9640 1 --2- 172.21.15.67:0/1453742760 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f386006bc00 0x7f386006e0c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.299+0000 7f386cff9640 1 --2- 172.21.15.67:0/1453742760 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f388015e0f0 0x7f388015f310 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.299+0000 7f386cff9640 1 --2- 172.21.15.67:0/1453742760 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f38800b9af0 0x7f3880160d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.299+0000 7f386cff9640 1 --2- 172.21.15.67:0/1453742760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38800b6dc0 0x7f38801e69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.299+0000 7f386cff9640 1 -- 172.21.15.67:0/1453742760 >> 172.21.15.67:0/1453742760 conn(0x7f3880072b60 msgr2=0x7f38800c8260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:15.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.299+0000 7f386cff9640 1 -- 172.21.15.67:0/1453742760 shutdown_connections 2024-03-24T00:39:15.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.299+0000 7f386cff9640 1 -- 172.21.15.67:0/1453742760 wait complete. 2024-03-24T00:39:15.310 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.5 2024-03-24T00:39:15.310 DEBUG:teuthology.parallel:result is None 2024-03-24T00:39:15.310 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-24T00:39:15.310 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:39:15.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.411+0000 7f2ec2475640 1 Processor -- start 2024-03-24T00:39:15.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.411+0000 7f2ec2475640 1 -- start start 2024-03-24T00:39:15.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.411+0000 7f2ec2475640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2ebc0d0940 0x7f2ebc0cfed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:15.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.411+0000 7f2ec2475640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2ebc099d70 0x7f2ebc0b19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:15.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.411+0000 7f2ec2475640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ebc0cdb50 con 0x7f2ebc15f410 2024-03-24T00:39:15.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.411+0000 7f2ec2475640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ebc0cd2b0 con 0x7f2ebc099d70 2024-03-24T00:39:15.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.411+0000 7f2ec2475640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ebc0cd590 con 0x7f2ebc0d0940 2024-03-24T00:39:15.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.412+0000 7f2ec0a12640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2ebc099d70 0x7f2ebc0b19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:15.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.412+0000 7f2ebb7fe640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2ebc0d0940 0x7f2ebc0cfed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:15.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.412+0000 7f2ebb7fe640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2ebc0d0940 0x7f2ebc0cfed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.67:42714/0 (socket says 172.21.15.67:42714) 2024-03-24T00:39:15.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.412+0000 7f2ec0a12640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2ebc099d70 0x7f2ebc0b19f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35972/0 (socket says 172.21.15.67:35972) 2024-03-24T00:39:15.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.412+0000 7f2ebb7fe640 1 -- 172.21.15.67:0/3550751802 learned_addr learned my addr 172.21.15.67:0/3550751802 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:15.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.412+0000 7f2ebbfff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f2ebc15f410 0x7f2ebc0cf7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:49954/0 (socket says 172.21.15.67:49954) 2024-03-24T00:39:15.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.412+0000 7f2ebb7fe640 1 -- 172.21.15.67:0/3550751802 >> v2:172.21.15.67:3300/0 conn(0x7f2ebc099d70 msgr2=0x7f2ebc0b19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:15.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.412+0000 7f2ebb7fe640 1 --2- 172.21.15.67:0/3550751802 >> v2:172.21.15.67:3300/0 conn(0x7f2ebc099d70 0x7f2ebc0b19f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.412+0000 7f2ebb7fe640 1 -- 172.21.15.67:0/3550751802 >> v1:172.21.15.72:6789/0 conn(0x7f2ebc15f410 legacy=0x7f2ebc0cf7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:15.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.412+0000 7f2ebaffd640 1 -- 172.21.15.67:0/3550751802 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 674380797 0 0) 0x7f2ebc0cdb50 con 0x7f2ebc15f410 2024-03-24T00:39:15.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.412+0000 7f2ebb7fe640 1 -- 172.21.15.67:0/3550751802 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ebc0cd870 con 0x7f2ebc0d0940 2024-03-24T00:39:15.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.413+0000 7f2ebb7fe640 1 --2- 172.21.15.67:0/3550751802 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2ebc0d0940 0x7f2ebc0cfed0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2eb0009870 tx=0x7f2eb002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b4a67b93696a83d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:15.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.413+0000 7f2ebaffd640 1 -- 172.21.15.67:0/3550751802 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb0034070 con 0x7f2ebc0d0940 2024-03-24T00:39:15.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.413+0000 7f2ebaffd640 1 -- 172.21.15.67:0/3550751802 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eb0030840 con 0x7f2ebc0d0940 2024-03-24T00:39:15.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.413+0000 7f2ebaffd640 1 -- 172.21.15.67:0/3550751802 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb0030c10 con 0x7f2ebc0d0940 2024-03-24T00:39:15.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.413+0000 7f2ec2475640 1 -- 172.21.15.67:0/3550751802 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2ebc0d0940 msgr2=0x7f2ebc0cfed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:15.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.413+0000 7f2ec2475640 1 --2- 172.21.15.67:0/3550751802 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2ebc0d0940 0x7f2ebc0cfed0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2eb0009870 tx=0x7f2eb002b980 comp rx=0 tx=0).stop 2024-03-24T00:39:15.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.413+0000 7f2ec2475640 1 -- 172.21.15.67:0/3550751802 shutdown_connections 2024-03-24T00:39:15.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.413+0000 7f2ec2475640 1 --2- 172.21.15.67:0/3550751802 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2ebc0d0940 0x7f2ebc0cfed0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.413+0000 7f2ec2475640 1 --2- 172.21.15.67:0/3550751802 >> v2:172.21.15.67:3300/0 conn(0x7f2ebc099d70 0x7f2ebc0b19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.413+0000 7f2ec2475640 1 -- 172.21.15.67:0/3550751802 >> 172.21.15.67:0/3550751802 conn(0x7f2ebc0b36e0 msgr2=0x7f2ebc0b3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:15.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.413+0000 7f2ec2475640 1 -- 172.21.15.67:0/3550751802 shutdown_connections 2024-03-24T00:39:15.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.413+0000 7f2ec2475640 1 -- 172.21.15.67:0/3550751802 wait complete. 2024-03-24T00:39:15.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.415+0000 7f2ec2475640 1 Processor -- start 2024-03-24T00:39:15.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.415+0000 7f2ec2475640 1 -- start start 2024-03-24T00:39:15.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.415+0000 7f2ec2475640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2ebc099d70 0x7f2ebc15dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:15.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.415+0000 7f2ec2475640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ebc15f410 0x7f2ebc0c7830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:15.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.415+0000 7f2ec2475640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2ebc0b06b0 0x7f2ebc0c5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:15.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.415+0000 7f2ec2475640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ebc15f7f0 con 0x7f2ebc15f410 2024-03-24T00:39:15.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.415+0000 7f2ec2475640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ebc0d0410 con 0x7f2ebc099d70 2024-03-24T00:39:15.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.415+0000 7f2ec2475640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ebc0c85f0 con 0x7f2ebc0b06b0 2024-03-24T00:39:15.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.416+0000 7f2ebb7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ebc15f410 0x7f2ebc0c7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:15.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.416+0000 7f2ebbfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2ebc099d70 0x7f2ebc15dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:15.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.416+0000 7f2ebb7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ebc15f410 0x7f2ebc0c7830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35976/0 (socket says 172.21.15.67:35976) 2024-03-24T00:39:15.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.416+0000 7f2ebb7fe640 1 -- 172.21.15.67:0/157080459 learned_addr learned my addr 172.21.15.67:0/157080459 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:15.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.416+0000 7f2ec0a12640 1 --2- 172.21.15.67:0/157080459 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2ebc0b06b0 0x7f2ebc0c5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:15.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.416+0000 7f2ebb7fe640 1 -- 172.21.15.67:0/157080459 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2ebc0b06b0 msgr2=0x7f2ebc0c5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:15.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.416+0000 7f2ebb7fe640 1 --2- 172.21.15.67:0/157080459 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2ebc0b06b0 0x7f2ebc0c5a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.416+0000 7f2ebb7fe640 1 -- 172.21.15.67:0/157080459 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2ebc099d70 msgr2=0x7f2ebc15dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:15.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.416+0000 7f2ebb7fe640 1 --2- 172.21.15.67:0/157080459 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2ebc099d70 0x7f2ebc15dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.416+0000 7f2ebb7fe640 1 -- 172.21.15.67:0/157080459 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ebc1f5980 con 0x7f2ebc15f410 2024-03-24T00:39:15.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.416+0000 7f2ebbfff640 1 --2- 172.21.15.67:0/157080459 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2ebc099d70 0x7f2ebc15dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:15.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.416+0000 7f2ebb7fe640 1 --2- 172.21.15.67:0/157080459 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ebc15f410 0x7f2ebc0c7830 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2eb0000c00 tx=0x7f2eb00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:15.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.416+0000 7f2eb8ff9640 1 -- 172.21.15.67:0/157080459 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb0034040 con 0x7f2ebc15f410 2024-03-24T00:39:15.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.416+0000 7f2eb8ff9640 1 -- 172.21.15.67:0/157080459 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eb0030640 con 0x7f2ebc15f410 2024-03-24T00:39:15.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.416+0000 7f2eb8ff9640 1 -- 172.21.15.67:0/157080459 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb0039d40 con 0x7f2ebc15f410 2024-03-24T00:39:15.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.416+0000 7f2ec2475640 1 -- 172.21.15.67:0/157080459 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ebc0c8310 con 0x7f2ebc15f410 2024-03-24T00:39:15.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.416+0000 7f2ec2475640 1 -- 172.21.15.67:0/157080459 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ebc15e200 con 0x7f2ebc15f410 2024-03-24T00:39:15.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.417+0000 7f2ec2475640 1 -- 172.21.15.67:0/157080459 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ebc04f400 con 0x7f2ebc15f410 2024-03-24T00:39:15.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.418+0000 7f2eb8ff9640 1 -- 172.21.15.67:0/157080459 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2eb004b060 con 0x7f2ebc15f410 2024-03-24T00:39:15.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.418+0000 7f2eb8ff9640 1 --2- 172.21.15.67:0/157080459 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2ea806bc00 0x7f2ea806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:15.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.418+0000 7f2eb8ff9640 1 -- 172.21.15.67:0/157080459 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f2eb00ac990 con 0x7f2ebc15f410 2024-03-24T00:39:15.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.418+0000 7f2ebbfff640 1 --2- 172.21.15.67:0/157080459 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2ea806bc00 0x7f2ea806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:15.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.419+0000 7f2ebbfff640 1 --2- 172.21.15.67:0/157080459 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2ea806bc00 0x7f2ea806e0c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2eac007fd0 tx=0x7f2eac007e00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:15.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.422+0000 7f2eb8ff9640 1 -- 172.21.15.67:0/157080459 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2eb007aca0 con 0x7f2ebc15f410 2024-03-24T00:39:15.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.601+0000 7f2ec2475640 1 -- 172.21.15.67:0/157080459 --> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2ebc15e3d0 con 0x7f2ea806bc00 2024-03-24T00:39:15.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.609+0000 7f2eb8ff9640 1 -- 172.21.15.67:0/157080459 <== mgr.4105 v2:172.21.15.72:6832/2768292274 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+332534 (secure 0 0 0) 0x7f2ebc15e3d0 con 0x7f2ea806bc00 2024-03-24T00:39:15.613 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:39:15.615 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-24T00:39:15.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.612+0000 7f2ec2475640 1 -- 172.21.15.67:0/157080459 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2ea806bc00 msgr2=0x7f2ea806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:15.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.612+0000 7f2ec2475640 1 --2- 172.21.15.67:0/157080459 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2ea806bc00 0x7f2ea806e0c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2eac007fd0 tx=0x7f2eac007e00 comp rx=0 tx=0).stop 2024-03-24T00:39:15.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.612+0000 7f2ec2475640 1 -- 172.21.15.67:0/157080459 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ebc15f410 msgr2=0x7f2ebc0c7830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:15.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.612+0000 7f2ec2475640 1 --2- 172.21.15.67:0/157080459 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ebc15f410 0x7f2ebc0c7830 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2eb0000c00 tx=0x7f2eb00046c0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.612+0000 7f2ec2475640 1 -- 172.21.15.67:0/157080459 shutdown_connections 2024-03-24T00:39:15.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.612+0000 7f2ec2475640 1 --2- 172.21.15.67:0/157080459 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2ea806bc00 0x7f2ea806e0c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.612+0000 7f2ec2475640 1 --2- 172.21.15.67:0/157080459 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2ebc0b06b0 0x7f2ebc0c5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.612+0000 7f2ec2475640 1 --2- 172.21.15.67:0/157080459 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ebc15f410 0x7f2ebc0c7830 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.612+0000 7f2ec2475640 1 --2- 172.21.15.67:0/157080459 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2ebc099d70 0x7f2ebc15dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.612+0000 7f2ec2475640 1 -- 172.21.15.67:0/157080459 >> 172.21.15.67:0/157080459 conn(0x7f2ebc0b36e0 msgr2=0x7f2ebc0ce570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:15.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.612+0000 7f2ec2475640 1 -- 172.21.15.67:0/157080459 shutdown_connections 2024-03-24T00:39:15.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.612+0000 7f2ec2475640 1 -- 172.21.15.67:0/157080459 wait complete. 2024-03-24T00:39:15.624 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2024-03-24T00:39:15.328569+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219476,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755244,"statfs":{"total":773094113280,"available":772869369856,"internally_reserved":0,"allocated":3383296,"data_stored":2049160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002600"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":28,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676794+0000","last_change":"2024-03-24T00:38:51.779360+0000","last_active":"2024-03-24T00:38:59.676794+0000","last_peered":"2024-03-24T00:38:59.676794+0000","last_clean":"2024-03-24T00:38:59.676794+0000","last_became_active":"2024-03-24T00:38:51.779066+0000","last_became_peered":"2024-03-24T00:38:51.779066+0000","last_unstale":"2024-03-24T00:38:59.676794+0000","last_undegraded":"2024-03-24T00:38:59.676794+0000","last_fullsized":"2024-03-24T00:38:59.676794+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:44:08.508305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800440+0000","last_change":"2024-03-24T00:38:52.781668+0000","last_active":"2024-03-24T00:38:58.800440+0000","last_peered":"2024-03-24T00:38:58.800440+0000","last_clean":"2024-03-24T00:38:58.800440+0000","last_became_active":"2024-03-24T00:38:52.781401+0000","last_became_peered":"2024-03-24T00:38:52.781401+0000","last_unstale":"2024-03-24T00:38:58.800440+0000","last_undegraded":"2024-03-24T00:38:58.800440+0000","last_fullsized":"2024-03-24T00:38:58.800440+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:01:21.675152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800050+0000","last_change":"2024-03-24T00:38:57.197819+0000","last_active":"2024-03-24T00:38:58.800050+0000","last_peered":"2024-03-24T00:38:58.800050+0000","last_clean":"2024-03-24T00:38:58.800050+0000","last_became_active":"2024-03-24T00:38:57.196369+0000","last_became_peered":"2024-03-24T00:38:57.196369+0000","last_unstale":"2024-03-24T00:38:58.800050+0000","last_undegraded":"2024-03-24T00:38:58.800050+0000","last_fullsized":"2024-03-24T00:38:58.800050+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:04:26.722612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800456+0000","last_change":"2024-03-24T00:38:51.772822+0000","last_active":"2024-03-24T00:38:58.800456+0000","last_peered":"2024-03-24T00:38:58.800456+0000","last_clean":"2024-03-24T00:38:58.800456+0000","last_became_active":"2024-03-24T00:38:51.772506+0000","last_became_peered":"2024-03-24T00:38:51.772506+0000","last_unstale":"2024-03-24T00:38:58.800456+0000","last_undegraded":"2024-03-24T00:38:58.800456+0000","last_fullsized":"2024-03-24T00:38:58.800456+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:09:30.121404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676632+0000","last_change":"2024-03-24T00:38:52.778282+0000","last_active":"2024-03-24T00:38:59.676632+0000","last_peered":"2024-03-24T00:38:59.676632+0000","last_clean":"2024-03-24T00:38:59.676632+0000","last_became_active":"2024-03-24T00:38:52.778003+0000","last_became_peered":"2024-03-24T00:38:52.778003+0000","last_unstale":"2024-03-24T00:38:59.676632+0000","last_undegraded":"2024-03-24T00:38:59.676632+0000","last_fullsized":"2024-03-24T00:38:59.676632+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:01:21.675152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801128+0000","last_change":"2024-03-24T00:38:56.822022+0000","last_active":"2024-03-24T00:38:58.801128+0000","last_peered":"2024-03-24T00:38:58.801128+0000","last_clean":"2024-03-24T00:38:58.801128+0000","last_became_active":"2024-03-24T00:38:56.821498+0000","last_became_peered":"2024-03-24T00:38:56.821498+0000","last_unstale":"2024-03-24T00:38:58.801128+0000","last_undegraded":"2024-03-24T00:38:58.801128+0000","last_fullsized":"2024-03-24T00:38:58.801128+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:02:47.160631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.677000+0000","last_change":"2024-03-24T00:38:51.781748+0000","last_active":"2024-03-24T00:38:59.677000+0000","last_peered":"2024-03-24T00:38:59.677000+0000","last_clean":"2024-03-24T00:38:59.677000+0000","last_became_active":"2024-03-24T00:38:51.781399+0000","last_became_peered":"2024-03-24T00:38:51.781399+0000","last_unstale":"2024-03-24T00:38:59.677000+0000","last_undegraded":"2024-03-24T00:38:59.677000+0000","last_fullsized":"2024-03-24T00:38:59.677000+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:51:21.173485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802757+0000","last_change":"2024-03-24T00:38:52.794918+0000","last_active":"2024-03-24T00:38:58.802757+0000","last_peered":"2024-03-24T00:38:58.802757+0000","last_clean":"2024-03-24T00:38:58.802757+0000","last_became_active":"2024-03-24T00:38:52.794638+0000","last_became_peered":"2024-03-24T00:38:52.794638+0000","last_unstale":"2024-03-24T00:38:58.802757+0000","last_undegraded":"2024-03-24T00:38:58.802757+0000","last_fullsized":"2024-03-24T00:38:58.802757+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:03:33.725080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802076+0000","last_change":"2024-03-24T00:38:56.817596+0000","last_active":"2024-03-24T00:38:58.802076+0000","last_peered":"2024-03-24T00:38:58.802076+0000","last_clean":"2024-03-24T00:38:58.802076+0000","last_became_active":"2024-03-24T00:38:56.817300+0000","last_became_peered":"2024-03-24T00:38:56.817300+0000","last_unstale":"2024-03-24T00:38:58.802076+0000","last_undegraded":"2024-03-24T00:38:58.802076+0000","last_fullsized":"2024-03-24T00:38:58.802076+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:53:16.442548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676343+0000","last_change":"2024-03-24T00:38:51.780697+0000","last_active":"2024-03-24T00:38:59.676343+0000","last_peered":"2024-03-24T00:38:59.676343+0000","last_clean":"2024-03-24T00:38:59.676343+0000","last_became_active":"2024-03-24T00:38:51.780118+0000","last_became_peered":"2024-03-24T00:38:51.780118+0000","last_unstale":"2024-03-24T00:38:59.676343+0000","last_undegraded":"2024-03-24T00:38:59.676343+0000","last_fullsized":"2024-03-24T00:38:59.676343+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:53:39.260829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802424+0000","last_change":"2024-03-24T00:38:52.795487+0000","last_active":"2024-03-24T00:38:58.802424+0000","last_peered":"2024-03-24T00:38:58.802424+0000","last_clean":"2024-03-24T00:38:58.802424+0000","last_became_active":"2024-03-24T00:38:52.794564+0000","last_became_peered":"2024-03-24T00:38:52.794564+0000","last_unstale":"2024-03-24T00:38:58.802424+0000","last_undegraded":"2024-03-24T00:38:58.802424+0000","last_fullsized":"2024-03-24T00:38:58.802424+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:48:02.197473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801403+0000","last_change":"2024-03-24T00:38:56.815283+0000","last_active":"2024-03-24T00:38:58.801403+0000","last_peered":"2024-03-24T00:38:58.801403+0000","last_clean":"2024-03-24T00:38:58.801403+0000","last_became_active":"2024-03-24T00:38:56.814490+0000","last_became_peered":"2024-03-24T00:38:56.814490+0000","last_unstale":"2024-03-24T00:38:58.801403+0000","last_undegraded":"2024-03-24T00:38:58.801403+0000","last_fullsized":"2024-03-24T00:38:58.801403+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:08:04.920818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676589+0000","last_change":"2024-03-24T00:38:52.785835+0000","last_active":"2024-03-24T00:38:59.676589+0000","last_peered":"2024-03-24T00:38:59.676589+0000","last_clean":"2024-03-24T00:38:59.676589+0000","last_became_active":"2024-03-24T00:38:52.785539+0000","last_became_peered":"2024-03-24T00:38:52.785539+0000","last_unstale":"2024-03-24T00:38:59.676589+0000","last_undegraded":"2024-03-24T00:38:59.676589+0000","last_fullsized":"2024-03-24T00:38:59.676589+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:01:21.675152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676805+0000","last_change":"2024-03-24T00:38:51.778378+0000","last_active":"2024-03-24T00:38:59.676805+0000","last_peered":"2024-03-24T00:38:59.676805+0000","last_clean":"2024-03-24T00:38:59.676805+0000","last_became_active":"2024-03-24T00:38:51.778093+0000","last_became_peered":"2024-03-24T00:38:51.778093+0000","last_unstale":"2024-03-24T00:38:59.676805+0000","last_undegraded":"2024-03-24T00:38:59.676805+0000","last_fullsized":"2024-03-24T00:38:59.676805+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:05.055760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801870+0000","last_change":"2024-03-24T00:38:56.821541+0000","last_active":"2024-03-24T00:38:58.801870+0000","last_peered":"2024-03-24T00:38:58.801870+0000","last_clean":"2024-03-24T00:38:58.801870+0000","last_became_active":"2024-03-24T00:38:56.821097+0000","last_became_peered":"2024-03-24T00:38:56.821097+0000","last_unstale":"2024-03-24T00:38:58.801870+0000","last_undegraded":"2024-03-24T00:38:58.801870+0000","last_fullsized":"2024-03-24T00:38:58.801870+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:22:37.069261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800616+0000","last_change":"2024-03-24T00:38:52.782691+0000","last_active":"2024-03-24T00:38:58.800616+0000","last_peered":"2024-03-24T00:38:58.800616+0000","last_clean":"2024-03-24T00:38:58.800616+0000","last_became_active":"2024-03-24T00:38:52.782439+0000","last_became_peered":"2024-03-24T00:38:52.782439+0000","last_unstale":"2024-03-24T00:38:58.800616+0000","last_undegraded":"2024-03-24T00:38:58.800616+0000","last_fullsized":"2024-03-24T00:38:58.800616+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:21:00.861116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975579+0000","last_change":"2024-03-24T00:38:51.788163+0000","last_active":"2024-03-24T00:38:58.975579+0000","last_peered":"2024-03-24T00:38:58.975579+0000","last_clean":"2024-03-24T00:38:58.975579+0000","last_became_active":"2024-03-24T00:38:51.787884+0000","last_became_peered":"2024-03-24T00:38:51.787884+0000","last_unstale":"2024-03-24T00:38:58.975579+0000","last_undegraded":"2024-03-24T00:38:58.975579+0000","last_fullsized":"2024-03-24T00:38:58.975579+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:51:21.173485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802074+0000","last_change":"2024-03-24T00:38:56.822013+0000","last_active":"2024-03-24T00:38:58.802074+0000","last_peered":"2024-03-24T00:38:58.802074+0000","last_clean":"2024-03-24T00:38:58.802074+0000","last_became_active":"2024-03-24T00:38:56.821572+0000","last_became_peered":"2024-03-24T00:38:56.821572+0000","last_unstale":"2024-03-24T00:38:58.802074+0000","last_undegraded":"2024-03-24T00:38:58.802074+0000","last_fullsized":"2024-03-24T00:38:58.802074+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:28:30.107102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800068+0000","last_change":"2024-03-24T00:38:52.782995+0000","last_active":"2024-03-24T00:38:58.800068+0000","last_peered":"2024-03-24T00:38:58.800068+0000","last_clean":"2024-03-24T00:38:58.800068+0000","last_became_active":"2024-03-24T00:38:52.782426+0000","last_became_peered":"2024-03-24T00:38:52.782426+0000","last_unstale":"2024-03-24T00:38:58.800068+0000","last_undegraded":"2024-03-24T00:38:58.800068+0000","last_fullsized":"2024-03-24T00:38:58.800068+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:46:05.527509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.974999+0000","last_change":"2024-03-24T00:38:51.787097+0000","last_active":"2024-03-24T00:38:58.974999+0000","last_peered":"2024-03-24T00:38:58.974999+0000","last_clean":"2024-03-24T00:38:58.974999+0000","last_became_active":"2024-03-24T00:38:51.786714+0000","last_became_peered":"2024-03-24T00:38:51.786714+0000","last_unstale":"2024-03-24T00:38:58.974999+0000","last_undegraded":"2024-03-24T00:38:58.974999+0000","last_fullsized":"2024-03-24T00:38:58.974999+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:05.055760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975812+0000","last_change":"2024-03-24T00:38:57.196574+0000","last_active":"2024-03-24T00:38:58.975812+0000","last_peered":"2024-03-24T00:38:58.975812+0000","last_clean":"2024-03-24T00:38:58.975812+0000","last_became_active":"2024-03-24T00:38:57.196154+0000","last_became_peered":"2024-03-24T00:38:57.196154+0000","last_unstale":"2024-03-24T00:38:58.975812+0000","last_undegraded":"2024-03-24T00:38:58.975812+0000","last_fullsized":"2024-03-24T00:38:58.975812+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:01:41.498040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801624+0000","last_change":"2024-03-24T00:38:52.784816+0000","last_active":"2024-03-24T00:38:58.801624+0000","last_peered":"2024-03-24T00:38:58.801624+0000","last_clean":"2024-03-24T00:38:58.801624+0000","last_became_active":"2024-03-24T00:38:52.784492+0000","last_became_peered":"2024-03-24T00:38:52.784492+0000","last_unstale":"2024-03-24T00:38:58.801624+0000","last_undegraded":"2024-03-24T00:38:58.801624+0000","last_fullsized":"2024-03-24T00:38:58.801624+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:23:26.012087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676478+0000","last_change":"2024-03-24T00:38:51.779160+0000","last_active":"2024-03-24T00:38:59.676478+0000","last_peered":"2024-03-24T00:38:59.676478+0000","last_clean":"2024-03-24T00:38:59.676478+0000","last_became_active":"2024-03-24T00:38:51.778859+0000","last_became_peered":"2024-03-24T00:38:51.778859+0000","last_unstale":"2024-03-24T00:38:59.676478+0000","last_undegraded":"2024-03-24T00:38:59.676478+0000","last_fullsized":"2024-03-24T00:38:59.676478+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:09:30.121404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802037+0000","last_change":"2024-03-24T00:38:56.822074+0000","last_active":"2024-03-24T00:38:58.802037+0000","last_peered":"2024-03-24T00:38:58.802037+0000","last_clean":"2024-03-24T00:38:58.802037+0000","last_became_active":"2024-03-24T00:38:56.821310+0000","last_became_peered":"2024-03-24T00:38:56.821310+0000","last_unstale":"2024-03-24T00:38:58.802037+0000","last_undegraded":"2024-03-24T00:38:58.802037+0000","last_fullsized":"2024-03-24T00:38:58.802037+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:17:07.239714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676939+0000","last_change":"2024-03-24T00:38:52.773192+0000","last_active":"2024-03-24T00:38:59.676939+0000","last_peered":"2024-03-24T00:38:59.676939+0000","last_clean":"2024-03-24T00:38:59.676939+0000","last_became_active":"2024-03-24T00:38:52.772910+0000","last_became_peered":"2024-03-24T00:38:52.772910+0000","last_unstale":"2024-03-24T00:38:59.676939+0000","last_undegraded":"2024-03-24T00:38:59.676939+0000","last_fullsized":"2024-03-24T00:38:59.676939+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:46:05.527509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.677252+0000","last_change":"2024-03-24T00:38:51.778294+0000","last_active":"2024-03-24T00:38:59.677252+0000","last_peered":"2024-03-24T00:38:59.677252+0000","last_clean":"2024-03-24T00:38:59.677252+0000","last_became_active":"2024-03-24T00:38:51.777905+0000","last_became_peered":"2024-03-24T00:38:51.777905+0000","last_unstale":"2024-03-24T00:38:59.677252+0000","last_undegraded":"2024-03-24T00:38:59.677252+0000","last_fullsized":"2024-03-24T00:38:59.677252+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:09:30.121404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801340+0000","last_change":"2024-03-24T00:38:57.196338+0000","last_active":"2024-03-24T00:38:58.801340+0000","last_peered":"2024-03-24T00:38:58.801340+0000","last_clean":"2024-03-24T00:38:58.801340+0000","last_became_active":"2024-03-24T00:38:57.195957+0000","last_became_peered":"2024-03-24T00:38:57.195957+0000","last_unstale":"2024-03-24T00:38:58.801340+0000","last_undegraded":"2024-03-24T00:38:58.801340+0000","last_fullsized":"2024-03-24T00:38:58.801340+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:57:34.164907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800713+0000","last_change":"2024-03-24T00:38:52.798013+0000","last_active":"2024-03-24T00:38:58.800713+0000","last_peered":"2024-03-24T00:38:58.800713+0000","last_clean":"2024-03-24T00:38:58.800713+0000","last_became_active":"2024-03-24T00:38:52.797709+0000","last_became_peered":"2024-03-24T00:38:52.797709+0000","last_unstale":"2024-03-24T00:38:58.800713+0000","last_undegraded":"2024-03-24T00:38:58.800713+0000","last_fullsized":"2024-03-24T00:38:58.800713+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:21:12.524662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975704+0000","last_change":"2024-03-24T00:38:51.787052+0000","last_active":"2024-03-24T00:38:58.975704+0000","last_peered":"2024-03-24T00:38:58.975704+0000","last_clean":"2024-03-24T00:38:58.975704+0000","last_became_active":"2024-03-24T00:38:51.786750+0000","last_became_peered":"2024-03-24T00:38:51.786750+0000","last_unstale":"2024-03-24T00:38:58.975704+0000","last_undegraded":"2024-03-24T00:38:58.975704+0000","last_fullsized":"2024-03-24T00:38:58.975704+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:36:18.921609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800214+0000","last_change":"2024-03-24T00:38:57.198329+0000","last_active":"2024-03-24T00:38:58.800214+0000","last_peered":"2024-03-24T00:38:58.800214+0000","last_clean":"2024-03-24T00:38:58.800214+0000","last_became_active":"2024-03-24T00:38:57.196158+0000","last_became_peered":"2024-03-24T00:38:57.196158+0000","last_unstale":"2024-03-24T00:38:58.800214+0000","last_undegraded":"2024-03-24T00:38:58.800214+0000","last_fullsized":"2024-03-24T00:38:58.800214+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:52:58.271700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975518+0000","last_change":"2024-03-24T00:38:52.778532+0000","last_active":"2024-03-24T00:38:58.975518+0000","last_peered":"2024-03-24T00:38:58.975518+0000","last_clean":"2024-03-24T00:38:58.975518+0000","last_became_active":"2024-03-24T00:38:52.778181+0000","last_became_peered":"2024-03-24T00:38:52.778181+0000","last_unstale":"2024-03-24T00:38:58.975518+0000","last_undegraded":"2024-03-24T00:38:58.975518+0000","last_fullsized":"2024-03-24T00:38:58.975518+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:39:26.969331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.677108+0000","last_change":"2024-03-24T00:38:51.779037+0000","last_active":"2024-03-24T00:38:59.677108+0000","last_peered":"2024-03-24T00:38:59.677108+0000","last_clean":"2024-03-24T00:38:59.677108+0000","last_became_active":"2024-03-24T00:38:51.778651+0000","last_became_peered":"2024-03-24T00:38:51.778651+0000","last_unstale":"2024-03-24T00:38:59.677108+0000","last_undegraded":"2024-03-24T00:38:59.677108+0000","last_fullsized":"2024-03-24T00:38:59.677108+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:36:18.921609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676465+0000","last_change":"2024-03-24T00:38:57.196547+0000","last_active":"2024-03-24T00:38:59.676465+0000","last_peered":"2024-03-24T00:38:59.676465+0000","last_clean":"2024-03-24T00:38:59.676465+0000","last_became_active":"2024-03-24T00:38:57.196148+0000","last_became_peered":"2024-03-24T00:38:57.196148+0000","last_unstale":"2024-03-24T00:38:59.676465+0000","last_undegraded":"2024-03-24T00:38:59.676465+0000","last_fullsized":"2024-03-24T00:38:59.676465+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:28:30.107102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802493+0000","last_change":"2024-03-24T00:38:52.789024+0000","last_active":"2024-03-24T00:38:58.802493+0000","last_peered":"2024-03-24T00:38:58.802493+0000","last_clean":"2024-03-24T00:38:58.802493+0000","last_became_active":"2024-03-24T00:38:52.788652+0000","last_became_peered":"2024-03-24T00:38:52.788652+0000","last_unstale":"2024-03-24T00:38:58.802493+0000","last_undegraded":"2024-03-24T00:38:58.802493+0000","last_fullsized":"2024-03-24T00:38:58.802493+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:21:00.861116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975533+0000","last_change":"2024-03-24T00:38:51.787599+0000","last_active":"2024-03-24T00:38:58.975533+0000","last_peered":"2024-03-24T00:38:58.975533+0000","last_clean":"2024-03-24T00:38:58.975533+0000","last_became_active":"2024-03-24T00:38:51.786958+0000","last_became_peered":"2024-03-24T00:38:51.786958+0000","last_unstale":"2024-03-24T00:38:58.975533+0000","last_undegraded":"2024-03-24T00:38:58.975533+0000","last_fullsized":"2024-03-24T00:38:58.975533+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:23:25.006444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975106+0000","last_change":"2024-03-24T00:38:56.822670+0000","last_active":"2024-03-24T00:38:58.975106+0000","last_peered":"2024-03-24T00:38:58.975106+0000","last_clean":"2024-03-24T00:38:58.975106+0000","last_became_active":"2024-03-24T00:38:56.822257+0000","last_became_peered":"2024-03-24T00:38:56.822257+0000","last_unstale":"2024-03-24T00:38:58.975106+0000","last_undegraded":"2024-03-24T00:38:58.975106+0000","last_fullsized":"2024-03-24T00:38:58.975106+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:17:07.239714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802301+0000","last_change":"2024-03-24T00:38:52.786986+0000","last_active":"2024-03-24T00:38:58.802301+0000","last_peered":"2024-03-24T00:38:58.802301+0000","last_clean":"2024-03-24T00:38:58.802301+0000","last_became_active":"2024-03-24T00:38:52.786700+0000","last_became_peered":"2024-03-24T00:38:52.786700+0000","last_unstale":"2024-03-24T00:38:58.802301+0000","last_undegraded":"2024-03-24T00:38:58.802301+0000","last_fullsized":"2024-03-24T00:38:58.802301+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:06.061403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802704+0000","last_change":"2024-03-24T00:38:51.789060+0000","last_active":"2024-03-24T00:38:58.802704+0000","last_peered":"2024-03-24T00:38:58.802704+0000","last_clean":"2024-03-24T00:38:58.802704+0000","last_became_active":"2024-03-24T00:38:51.788751+0000","last_became_peered":"2024-03-24T00:38:51.788751+0000","last_unstale":"2024-03-24T00:38:58.802704+0000","last_undegraded":"2024-03-24T00:38:58.802704+0000","last_fullsized":"2024-03-24T00:38:58.802704+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:09:30.121404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975477+0000","last_change":"2024-03-24T00:38:56.818475+0000","last_active":"2024-03-24T00:38:58.975477+0000","last_peered":"2024-03-24T00:38:58.975477+0000","last_clean":"2024-03-24T00:38:58.975477+0000","last_became_active":"2024-03-24T00:38:56.817655+0000","last_became_peered":"2024-03-24T00:38:56.817655+0000","last_unstale":"2024-03-24T00:38:58.975477+0000","last_undegraded":"2024-03-24T00:38:58.975477+0000","last_fullsized":"2024-03-24T00:38:58.975477+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:53:16.442548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.974995+0000","last_change":"2024-03-24T00:38:52.780020+0000","last_active":"2024-03-24T00:38:58.974995+0000","last_peered":"2024-03-24T00:38:58.974995+0000","last_clean":"2024-03-24T00:38:58.974995+0000","last_became_active":"2024-03-24T00:38:52.779730+0000","last_became_peered":"2024-03-24T00:38:52.779730+0000","last_unstale":"2024-03-24T00:38:58.974995+0000","last_undegraded":"2024-03-24T00:38:58.974995+0000","last_fullsized":"2024-03-24T00:38:58.974995+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:01:21.675152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.675893+0000","last_change":"2024-03-24T00:38:51.771263+0000","last_active":"2024-03-24T00:38:59.675893+0000","last_peered":"2024-03-24T00:38:59.675893+0000","last_clean":"2024-03-24T00:38:59.675893+0000","last_became_active":"2024-03-24T00:38:51.770920+0000","last_became_peered":"2024-03-24T00:38:51.770920+0000","last_unstale":"2024-03-24T00:38:59.675893+0000","last_undegraded":"2024-03-24T00:38:59.675893+0000","last_fullsized":"2024-03-24T00:38:59.675893+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:09:30.121404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975825+0000","last_change":"2024-03-24T00:38:56.822793+0000","last_active":"2024-03-24T00:38:58.975825+0000","last_peered":"2024-03-24T00:38:58.975825+0000","last_clean":"2024-03-24T00:38:58.975825+0000","last_became_active":"2024-03-24T00:38:56.822282+0000","last_became_peered":"2024-03-24T00:38:56.822282+0000","last_unstale":"2024-03-24T00:38:58.975825+0000","last_undegraded":"2024-03-24T00:38:58.975825+0000","last_fullsized":"2024-03-24T00:38:58.975825+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:26:44.414745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676155+0000","last_change":"2024-03-24T00:38:52.782329+0000","last_active":"2024-03-24T00:38:59.676155+0000","last_peered":"2024-03-24T00:38:59.676155+0000","last_clean":"2024-03-24T00:38:59.676155+0000","last_became_active":"2024-03-24T00:38:52.782006+0000","last_became_peered":"2024-03-24T00:38:52.782006+0000","last_unstale":"2024-03-24T00:38:59.676155+0000","last_undegraded":"2024-03-24T00:38:59.676155+0000","last_fullsized":"2024-03-24T00:38:59.676155+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:41:26.388475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676562+0000","last_change":"2024-03-24T00:38:51.787246+0000","last_active":"2024-03-24T00:38:59.676562+0000","last_peered":"2024-03-24T00:38:59.676562+0000","last_clean":"2024-03-24T00:38:59.676562+0000","last_became_active":"2024-03-24T00:38:51.787019+0000","last_became_peered":"2024-03-24T00:38:51.787019+0000","last_unstale":"2024-03-24T00:38:59.676562+0000","last_undegraded":"2024-03-24T00:38:59.676562+0000","last_fullsized":"2024-03-24T00:38:59.676562+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:23:25.006444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676812+0000","last_change":"2024-03-24T00:38:57.196855+0000","last_active":"2024-03-24T00:38:59.676812+0000","last_peered":"2024-03-24T00:38:59.676812+0000","last_clean":"2024-03-24T00:38:59.676812+0000","last_became_active":"2024-03-24T00:38:57.196127+0000","last_became_peered":"2024-03-24T00:38:57.196127+0000","last_unstale":"2024-03-24T00:38:59.676812+0000","last_undegraded":"2024-03-24T00:38:59.676812+0000","last_fullsized":"2024-03-24T00:38:59.676812+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:05:46.867916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800028+0000","last_change":"2024-03-24T00:38:52.792357+0000","last_active":"2024-03-24T00:38:58.800028+0000","last_peered":"2024-03-24T00:38:58.800028+0000","last_clean":"2024-03-24T00:38:58.800028+0000","last_became_active":"2024-03-24T00:38:52.792077+0000","last_became_peered":"2024-03-24T00:38:52.792077+0000","last_unstale":"2024-03-24T00:38:58.800028+0000","last_undegraded":"2024-03-24T00:38:58.800028+0000","last_fullsized":"2024-03-24T00:38:58.800028+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:53:36.522410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802065+0000","last_change":"2024-03-24T00:38:51.777903+0000","last_active":"2024-03-24T00:38:58.802065+0000","last_peered":"2024-03-24T00:38:58.802065+0000","last_clean":"2024-03-24T00:38:58.802065+0000","last_became_active":"2024-03-24T00:38:51.777178+0000","last_became_peered":"2024-03-24T00:38:51.777178+0000","last_unstale":"2024-03-24T00:38:58.802065+0000","last_undegraded":"2024-03-24T00:38:58.802065+0000","last_fullsized":"2024-03-24T00:38:58.802065+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:58:23.822944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":180,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676606+0000","last_change":"2024-03-24T00:38:40.953871+0000","last_active":"2024-03-24T00:38:59.676606+0000","last_peered":"2024-03-24T00:38:59.676606+0000","last_clean":"2024-03-24T00:38:59.676606+0000","last_became_active":"2024-03-24T00:38:40.952926+0000","last_became_peered":"2024-03-24T00:38:40.952926+0000","last_unstale":"2024-03-24T00:38:59.676606+0000","last_undegraded":"2024-03-24T00:38:59.676606+0000","last_fullsized":"2024-03-24T00:38:59.676606+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_clean_scrub_stamp":"2024-03-24T00:38:39.722194+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:05.397257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676410+0000","last_change":"2024-03-24T00:38:57.196309+0000","last_active":"2024-03-24T00:38:59.676410+0000","last_peered":"2024-03-24T00:38:59.676410+0000","last_clean":"2024-03-24T00:38:59.676410+0000","last_became_active":"2024-03-24T00:38:57.195975+0000","last_became_peered":"2024-03-24T00:38:57.195975+0000","last_unstale":"2024-03-24T00:38:59.676410+0000","last_undegraded":"2024-03-24T00:38:59.676410+0000","last_fullsized":"2024-03-24T00:38:59.676410+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:02:47.160631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676898+0000","last_change":"2024-03-24T00:38:52.790380+0000","last_active":"2024-03-24T00:38:59.676898+0000","last_peered":"2024-03-24T00:38:59.676898+0000","last_clean":"2024-03-24T00:38:59.676898+0000","last_became_active":"2024-03-24T00:38:52.788665+0000","last_became_peered":"2024-03-24T00:38:52.788665+0000","last_unstale":"2024-03-24T00:38:59.676898+0000","last_undegraded":"2024-03-24T00:38:59.676898+0000","last_fullsized":"2024-03-24T00:38:59.676898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:03:33.725080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676181+0000","last_change":"2024-03-24T00:38:51.781949+0000","last_active":"2024-03-24T00:38:59.676181+0000","last_peered":"2024-03-24T00:38:59.676181+0000","last_clean":"2024-03-24T00:38:59.676181+0000","last_became_active":"2024-03-24T00:38:51.781662+0000","last_became_peered":"2024-03-24T00:38:51.781662+0000","last_unstale":"2024-03-24T00:38:59.676181+0000","last_undegraded":"2024-03-24T00:38:59.676181+0000","last_fullsized":"2024-03-24T00:38:59.676181+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:59:15.934422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676501+0000","last_change":"2024-03-24T00:38:57.196217+0000","last_active":"2024-03-24T00:38:59.676501+0000","last_peered":"2024-03-24T00:38:59.676501+0000","last_clean":"2024-03-24T00:38:59.676501+0000","last_became_active":"2024-03-24T00:38:57.195860+0000","last_became_peered":"2024-03-24T00:38:57.195860+0000","last_unstale":"2024-03-24T00:38:59.676501+0000","last_undegraded":"2024-03-24T00:38:59.676501+0000","last_fullsized":"2024-03-24T00:38:59.676501+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:17:46.736676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802773+0000","last_change":"2024-03-24T00:38:52.780048+0000","last_active":"2024-03-24T00:38:58.802773+0000","last_peered":"2024-03-24T00:38:58.802773+0000","last_clean":"2024-03-24T00:38:58.802773+0000","last_became_active":"2024-03-24T00:38:52.779706+0000","last_became_peered":"2024-03-24T00:38:52.779706+0000","last_unstale":"2024-03-24T00:38:58.802773+0000","last_undegraded":"2024-03-24T00:38:58.802773+0000","last_fullsized":"2024-03-24T00:38:58.802773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:34:43.724190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975641+0000","last_change":"2024-03-24T00:38:51.789047+0000","last_active":"2024-03-24T00:38:58.975641+0000","last_peered":"2024-03-24T00:38:58.975641+0000","last_clean":"2024-03-24T00:38:58.975641+0000","last_became_active":"2024-03-24T00:38:51.788769+0000","last_became_peered":"2024-03-24T00:38:51.788769+0000","last_unstale":"2024-03-24T00:38:58.975641+0000","last_undegraded":"2024-03-24T00:38:58.975641+0000","last_fullsized":"2024-03-24T00:38:58.975641+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:09:30.121404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676730+0000","last_change":"2024-03-24T00:38:56.822795+0000","last_active":"2024-03-24T00:38:59.676730+0000","last_peered":"2024-03-24T00:38:59.676730+0000","last_clean":"2024-03-24T00:38:59.676730+0000","last_became_active":"2024-03-24T00:38:56.822410+0000","last_became_peered":"2024-03-24T00:38:56.822410+0000","last_unstale":"2024-03-24T00:38:59.676730+0000","last_undegraded":"2024-03-24T00:38:59.676730+0000","last_fullsized":"2024-03-24T00:38:59.676730+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:26:44.414745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.677245+0000","last_change":"2024-03-24T00:38:52.788023+0000","last_active":"2024-03-24T00:38:59.677245+0000","last_peered":"2024-03-24T00:38:59.677245+0000","last_clean":"2024-03-24T00:38:59.677245+0000","last_became_active":"2024-03-24T00:38:52.787528+0000","last_became_peered":"2024-03-24T00:38:52.787528+0000","last_unstale":"2024-03-24T00:38:59.677245+0000","last_undegraded":"2024-03-24T00:38:59.677245+0000","last_fullsized":"2024-03-24T00:38:59.677245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:57:30.131729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676813+0000","last_change":"2024-03-24T00:38:51.771300+0000","last_active":"2024-03-24T00:38:59.676813+0000","last_peered":"2024-03-24T00:38:59.676813+0000","last_clean":"2024-03-24T00:38:59.676813+0000","last_became_active":"2024-03-24T00:38:51.770924+0000","last_became_peered":"2024-03-24T00:38:51.770924+0000","last_unstale":"2024-03-24T00:38:59.676813+0000","last_undegraded":"2024-03-24T00:38:59.676813+0000","last_fullsized":"2024-03-24T00:38:59.676813+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:58:23.822944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801216+0000","last_change":"2024-03-24T00:38:57.196385+0000","last_active":"2024-03-24T00:38:58.801216+0000","last_peered":"2024-03-24T00:38:58.801216+0000","last_clean":"2024-03-24T00:38:58.801216+0000","last_became_active":"2024-03-24T00:38:57.196000+0000","last_became_peered":"2024-03-24T00:38:57.196000+0000","last_unstale":"2024-03-24T00:38:58.801216+0000","last_undegraded":"2024-03-24T00:38:58.801216+0000","last_fullsized":"2024-03-24T00:38:58.801216+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:26:44.414745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.677147+0000","last_change":"2024-03-24T00:38:52.791416+0000","last_active":"2024-03-24T00:38:59.677147+0000","last_peered":"2024-03-24T00:38:59.677147+0000","last_clean":"2024-03-24T00:38:59.677147+0000","last_became_active":"2024-03-24T00:38:52.790334+0000","last_became_peered":"2024-03-24T00:38:52.790334+0000","last_unstale":"2024-03-24T00:38:59.677147+0000","last_undegraded":"2024-03-24T00:38:59.677147+0000","last_fullsized":"2024-03-24T00:38:59.677147+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:34:43.724190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801647+0000","last_change":"2024-03-24T00:38:51.780196+0000","last_active":"2024-03-24T00:38:58.801647+0000","last_peered":"2024-03-24T00:38:58.801647+0000","last_clean":"2024-03-24T00:38:58.801647+0000","last_became_active":"2024-03-24T00:38:51.779921+0000","last_became_peered":"2024-03-24T00:38:51.779921+0000","last_unstale":"2024-03-24T00:38:58.801647+0000","last_undegraded":"2024-03-24T00:38:58.801647+0000","last_fullsized":"2024-03-24T00:38:58.801647+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:52:59.726516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801216+0000","last_change":"2024-03-24T00:38:56.821775+0000","last_active":"2024-03-24T00:38:58.801216+0000","last_peered":"2024-03-24T00:38:58.801216+0000","last_clean":"2024-03-24T00:38:58.801216+0000","last_became_active":"2024-03-24T00:38:56.821188+0000","last_became_peered":"2024-03-24T00:38:56.821188+0000","last_unstale":"2024-03-24T00:38:58.801216+0000","last_undegraded":"2024-03-24T00:38:58.801216+0000","last_fullsized":"2024-03-24T00:38:58.801216+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:17:46.736676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676771+0000","last_change":"2024-03-24T00:38:52.791631+0000","last_active":"2024-03-24T00:38:59.676771+0000","last_peered":"2024-03-24T00:38:59.676771+0000","last_clean":"2024-03-24T00:38:59.676771+0000","last_became_active":"2024-03-24T00:38:52.790109+0000","last_became_peered":"2024-03-24T00:38:52.790109+0000","last_unstale":"2024-03-24T00:38:59.676771+0000","last_undegraded":"2024-03-24T00:38:59.676771+0000","last_fullsized":"2024-03-24T00:38:59.676771+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:53:36.522410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":32,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.675790+0000","last_change":"2024-03-24T00:38:51.789009+0000","last_active":"2024-03-24T00:38:59.675790+0000","last_peered":"2024-03-24T00:38:59.675790+0000","last_clean":"2024-03-24T00:38:59.675790+0000","last_became_active":"2024-03-24T00:38:51.788721+0000","last_became_peered":"2024-03-24T00:38:51.788721+0000","last_unstale":"2024-03-24T00:38:59.675790+0000","last_undegraded":"2024-03-24T00:38:59.675790+0000","last_fullsized":"2024-03-24T00:38:59.675790+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:43:41.080723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676661+0000","last_change":"2024-03-24T00:38:56.815041+0000","last_active":"2024-03-24T00:38:59.676661+0000","last_peered":"2024-03-24T00:38:59.676661+0000","last_clean":"2024-03-24T00:38:59.676661+0000","last_became_active":"2024-03-24T00:38:56.814651+0000","last_became_peered":"2024-03-24T00:38:56.814651+0000","last_unstale":"2024-03-24T00:38:59.676661+0000","last_undegraded":"2024-03-24T00:38:59.676661+0000","last_fullsized":"2024-03-24T00:38:59.676661+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:01:41.498040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676169+0000","last_change":"2024-03-24T00:38:52.779914+0000","last_active":"2024-03-24T00:38:59.676169+0000","last_peered":"2024-03-24T00:38:59.676169+0000","last_clean":"2024-03-24T00:38:59.676169+0000","last_became_active":"2024-03-24T00:38:52.779330+0000","last_became_peered":"2024-03-24T00:38:52.779330+0000","last_unstale":"2024-03-24T00:38:59.676169+0000","last_undegraded":"2024-03-24T00:38:59.676169+0000","last_fullsized":"2024-03-24T00:38:59.676169+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:42:38.046862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676645+0000","last_change":"2024-03-24T00:38:51.781129+0000","last_active":"2024-03-24T00:38:59.676645+0000","last_peered":"2024-03-24T00:38:59.676645+0000","last_clean":"2024-03-24T00:38:59.676645+0000","last_became_active":"2024-03-24T00:38:51.780828+0000","last_became_peered":"2024-03-24T00:38:51.780828+0000","last_unstale":"2024-03-24T00:38:59.676645+0000","last_undegraded":"2024-03-24T00:38:59.676645+0000","last_fullsized":"2024-03-24T00:38:59.676645+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:59:15.934422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800545+0000","last_change":"2024-03-24T00:38:57.197537+0000","last_active":"2024-03-24T00:38:58.800545+0000","last_peered":"2024-03-24T00:38:58.800545+0000","last_clean":"2024-03-24T00:38:58.800545+0000","last_became_active":"2024-03-24T00:38:57.197082+0000","last_became_peered":"2024-03-24T00:38:57.197082+0000","last_unstale":"2024-03-24T00:38:58.800545+0000","last_undegraded":"2024-03-24T00:38:58.800545+0000","last_fullsized":"2024-03-24T00:38:58.800545+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:34:30.798923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676725+0000","last_change":"2024-03-24T00:38:52.789686+0000","last_active":"2024-03-24T00:38:59.676725+0000","last_peered":"2024-03-24T00:38:59.676725+0000","last_clean":"2024-03-24T00:38:59.676725+0000","last_became_active":"2024-03-24T00:38:52.789416+0000","last_became_peered":"2024-03-24T00:38:52.789416+0000","last_unstale":"2024-03-24T00:38:59.676725+0000","last_undegraded":"2024-03-24T00:38:59.676725+0000","last_fullsized":"2024-03-24T00:38:59.676725+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:04:24.260803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802696+0000","last_change":"2024-03-24T00:38:51.777070+0000","last_active":"2024-03-24T00:38:58.802696+0000","last_peered":"2024-03-24T00:38:58.802696+0000","last_clean":"2024-03-24T00:38:58.802696+0000","last_became_active":"2024-03-24T00:38:51.776786+0000","last_became_peered":"2024-03-24T00:38:51.776786+0000","last_unstale":"2024-03-24T00:38:58.802696+0000","last_undegraded":"2024-03-24T00:38:58.802696+0000","last_fullsized":"2024-03-24T00:38:58.802696+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:05.055760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800552+0000","last_change":"2024-03-24T00:38:57.196551+0000","last_active":"2024-03-24T00:38:58.800552+0000","last_peered":"2024-03-24T00:38:58.800552+0000","last_clean":"2024-03-24T00:38:58.800552+0000","last_became_active":"2024-03-24T00:38:57.196138+0000","last_became_peered":"2024-03-24T00:38:57.196138+0000","last_unstale":"2024-03-24T00:38:58.800552+0000","last_undegraded":"2024-03-24T00:38:58.800552+0000","last_fullsized":"2024-03-24T00:38:58.800552+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:47:53.855251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676423+0000","last_change":"2024-03-24T00:38:52.782561+0000","last_active":"2024-03-24T00:38:59.676423+0000","last_peered":"2024-03-24T00:38:59.676423+0000","last_clean":"2024-03-24T00:38:59.676423+0000","last_became_active":"2024-03-24T00:38:52.781961+0000","last_became_peered":"2024-03-24T00:38:52.781961+0000","last_unstale":"2024-03-24T00:38:59.676423+0000","last_undegraded":"2024-03-24T00:38:59.676423+0000","last_fullsized":"2024-03-24T00:38:59.676423+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:42:38.046862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676934+0000","last_change":"2024-03-24T00:38:51.778517+0000","last_active":"2024-03-24T00:38:59.676934+0000","last_peered":"2024-03-24T00:38:59.676934+0000","last_clean":"2024-03-24T00:38:59.676934+0000","last_became_active":"2024-03-24T00:38:51.778218+0000","last_became_peered":"2024-03-24T00:38:51.778218+0000","last_unstale":"2024-03-24T00:38:59.676934+0000","last_undegraded":"2024-03-24T00:38:59.676934+0000","last_fullsized":"2024-03-24T00:38:59.676934+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:01:37.155677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802393+0000","last_change":"2024-03-24T00:38:57.196343+0000","last_active":"2024-03-24T00:38:58.802393+0000","last_peered":"2024-03-24T00:38:58.802393+0000","last_clean":"2024-03-24T00:38:58.802393+0000","last_became_active":"2024-03-24T00:38:57.195953+0000","last_became_peered":"2024-03-24T00:38:57.195953+0000","last_unstale":"2024-03-24T00:38:58.802393+0000","last_undegraded":"2024-03-24T00:38:58.802393+0000","last_fullsized":"2024-03-24T00:38:58.802393+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:05:46.867916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676892+0000","last_change":"2024-03-24T00:38:52.792925+0000","last_active":"2024-03-24T00:38:59.676892+0000","last_peered":"2024-03-24T00:38:59.676892+0000","last_clean":"2024-03-24T00:38:59.676892+0000","last_became_active":"2024-03-24T00:38:52.792661+0000","last_became_peered":"2024-03-24T00:38:52.792661+0000","last_unstale":"2024-03-24T00:38:59.676892+0000","last_undegraded":"2024-03-24T00:38:59.676892+0000","last_fullsized":"2024-03-24T00:38:59.676892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:48:02.197473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":33,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:39:02.090137+0000","last_change":"2024-03-24T00:38:51.775463+0000","last_active":"2024-03-24T00:39:02.090137+0000","last_peered":"2024-03-24T00:39:02.090137+0000","last_clean":"2024-03-24T00:39:02.090137+0000","last_became_active":"2024-03-24T00:38:51.775167+0000","last_became_peered":"2024-03-24T00:38:51.775167+0000","last_unstale":"2024-03-24T00:39:02.090137+0000","last_undegraded":"2024-03-24T00:39:02.090137+0000","last_fullsized":"2024-03-24T00:39:02.090137+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:23:25.006444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975203+0000","last_change":"2024-03-24T00:38:56.819965+0000","last_active":"2024-03-24T00:38:58.975203+0000","last_peered":"2024-03-24T00:38:58.975203+0000","last_clean":"2024-03-24T00:38:58.975203+0000","last_became_active":"2024-03-24T00:38:56.819580+0000","last_became_peered":"2024-03-24T00:38:56.819580+0000","last_unstale":"2024-03-24T00:38:58.975203+0000","last_undegraded":"2024-03-24T00:38:58.975203+0000","last_fullsized":"2024-03-24T00:38:58.975203+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:28:30.107102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802246+0000","last_change":"2024-03-24T00:38:52.790290+0000","last_active":"2024-03-24T00:38:58.802246+0000","last_peered":"2024-03-24T00:38:58.802246+0000","last_clean":"2024-03-24T00:38:58.802246+0000","last_became_active":"2024-03-24T00:38:52.790010+0000","last_became_peered":"2024-03-24T00:38:52.790010+0000","last_unstale":"2024-03-24T00:38:58.802246+0000","last_undegraded":"2024-03-24T00:38:58.802246+0000","last_fullsized":"2024-03-24T00:38:58.802246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:36:19.927252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975485+0000","last_change":"2024-03-24T00:38:51.788684+0000","last_active":"2024-03-24T00:38:58.975485+0000","last_peered":"2024-03-24T00:38:58.975485+0000","last_clean":"2024-03-24T00:38:58.975485+0000","last_became_active":"2024-03-24T00:38:51.788414+0000","last_became_peered":"2024-03-24T00:38:51.788414+0000","last_unstale":"2024-03-24T00:38:58.975485+0000","last_undegraded":"2024-03-24T00:38:58.975485+0000","last_fullsized":"2024-03-24T00:38:58.975485+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:59:15.934422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802225+0000","last_change":"2024-03-24T00:38:56.822362+0000","last_active":"2024-03-24T00:38:58.802225+0000","last_peered":"2024-03-24T00:38:58.802225+0000","last_clean":"2024-03-24T00:38:58.802225+0000","last_became_active":"2024-03-24T00:38:56.821516+0000","last_became_peered":"2024-03-24T00:38:56.821516+0000","last_unstale":"2024-03-24T00:38:58.802225+0000","last_undegraded":"2024-03-24T00:38:58.802225+0000","last_fullsized":"2024-03-24T00:38:58.802225+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:01:41.498040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676720+0000","last_change":"2024-03-24T00:38:52.778191+0000","last_active":"2024-03-24T00:38:59.676720+0000","last_peered":"2024-03-24T00:38:59.676720+0000","last_clean":"2024-03-24T00:38:59.676720+0000","last_became_active":"2024-03-24T00:38:52.777911+0000","last_became_peered":"2024-03-24T00:38:52.777911+0000","last_unstale":"2024-03-24T00:38:59.676720+0000","last_undegraded":"2024-03-24T00:38:59.676720+0000","last_fullsized":"2024-03-24T00:38:59.676720+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:41:59.361242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801321+0000","last_change":"2024-03-24T00:38:51.778447+0000","last_active":"2024-03-24T00:38:58.801321+0000","last_peered":"2024-03-24T00:38:58.801321+0000","last_clean":"2024-03-24T00:38:58.801321+0000","last_became_active":"2024-03-24T00:38:51.778151+0000","last_became_peered":"2024-03-24T00:38:51.778151+0000","last_unstale":"2024-03-24T00:38:58.801321+0000","last_undegraded":"2024-03-24T00:38:58.801321+0000","last_fullsized":"2024-03-24T00:38:58.801321+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:23:25.006444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800051+0000","last_change":"2024-03-24T00:38:56.818959+0000","last_active":"2024-03-24T00:38:58.800051+0000","last_peered":"2024-03-24T00:38:58.800051+0000","last_clean":"2024-03-24T00:38:58.800051+0000","last_became_active":"2024-03-24T00:38:56.818582+0000","last_became_peered":"2024-03-24T00:38:56.818582+0000","last_unstale":"2024-03-24T00:38:58.800051+0000","last_undegraded":"2024-03-24T00:38:58.800051+0000","last_fullsized":"2024-03-24T00:38:58.800051+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:19:21.320451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801240+0000","last_change":"2024-03-24T00:38:52.787525+0000","last_active":"2024-03-24T00:38:58.801240+0000","last_peered":"2024-03-24T00:38:58.801240+0000","last_clean":"2024-03-24T00:38:58.801240+0000","last_became_active":"2024-03-24T00:38:52.787167+0000","last_became_peered":"2024-03-24T00:38:52.787167+0000","last_unstale":"2024-03-24T00:38:58.801240+0000","last_undegraded":"2024-03-24T00:38:58.801240+0000","last_fullsized":"2024-03-24T00:38:58.801240+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:53:36.522410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676940+0000","last_change":"2024-03-24T00:38:51.780424+0000","last_active":"2024-03-24T00:38:59.676940+0000","last_peered":"2024-03-24T00:38:59.676940+0000","last_clean":"2024-03-24T00:38:59.676940+0000","last_became_active":"2024-03-24T00:38:51.780120+0000","last_became_peered":"2024-03-24T00:38:51.780120+0000","last_unstale":"2024-03-24T00:38:59.676940+0000","last_undegraded":"2024-03-24T00:38:59.676940+0000","last_fullsized":"2024-03-24T00:38:59.676940+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:58:23.822944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800129+0000","last_change":"2024-03-24T00:38:57.197505+0000","last_active":"2024-03-24T00:38:58.800129+0000","last_peered":"2024-03-24T00:38:58.800129+0000","last_clean":"2024-03-24T00:38:58.800129+0000","last_became_active":"2024-03-24T00:38:57.197104+0000","last_became_peered":"2024-03-24T00:38:57.197104+0000","last_unstale":"2024-03-24T00:38:58.800129+0000","last_undegraded":"2024-03-24T00:38:58.800129+0000","last_fullsized":"2024-03-24T00:38:58.800129+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:44:07.562150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676497+0000","last_change":"2024-03-24T00:38:57.197065+0000","last_active":"2024-03-24T00:38:59.676497+0000","last_peered":"2024-03-24T00:38:59.676497+0000","last_clean":"2024-03-24T00:38:59.676497+0000","last_became_active":"2024-03-24T00:38:57.196687+0000","last_became_peered":"2024-03-24T00:38:57.196687+0000","last_unstale":"2024-03-24T00:38:59.676497+0000","last_undegraded":"2024-03-24T00:38:59.676497+0000","last_fullsized":"2024-03-24T00:38:59.676497+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:53:16.442548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.974900+0000","last_change":"2024-03-24T00:38:52.790233+0000","last_active":"2024-03-24T00:38:58.974900+0000","last_peered":"2024-03-24T00:38:58.974900+0000","last_clean":"2024-03-24T00:38:58.974900+0000","last_became_active":"2024-03-24T00:38:52.789951+0000","last_became_peered":"2024-03-24T00:38:52.789951+0000","last_unstale":"2024-03-24T00:38:58.974900+0000","last_undegraded":"2024-03-24T00:38:58.974900+0000","last_fullsized":"2024-03-24T00:38:58.974900+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:42:38.046862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801116+0000","last_change":"2024-03-24T00:38:51.775723+0000","last_active":"2024-03-24T00:38:58.801116+0000","last_peered":"2024-03-24T00:38:58.801116+0000","last_clean":"2024-03-24T00:38:58.801116+0000","last_became_active":"2024-03-24T00:38:51.773333+0000","last_became_peered":"2024-03-24T00:38:51.773333+0000","last_unstale":"2024-03-24T00:38:58.801116+0000","last_undegraded":"2024-03-24T00:38:58.801116+0000","last_fullsized":"2024-03-24T00:38:58.801116+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:59:15.934422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676945+0000","last_change":"2024-03-24T00:38:56.819973+0000","last_active":"2024-03-24T00:38:59.676945+0000","last_peered":"2024-03-24T00:38:59.676945+0000","last_clean":"2024-03-24T00:38:59.676945+0000","last_became_active":"2024-03-24T00:38:56.819659+0000","last_became_peered":"2024-03-24T00:38:56.819659+0000","last_unstale":"2024-03-24T00:38:59.676945+0000","last_undegraded":"2024-03-24T00:38:59.676945+0000","last_fullsized":"2024-03-24T00:38:59.676945+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:57:34.164907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.799976+0000","last_change":"2024-03-24T00:38:52.785779+0000","last_active":"2024-03-24T00:38:58.799976+0000","last_peered":"2024-03-24T00:38:58.799976+0000","last_clean":"2024-03-24T00:38:58.799976+0000","last_became_active":"2024-03-24T00:38:52.785184+0000","last_became_peered":"2024-03-24T00:38:52.785184+0000","last_unstale":"2024-03-24T00:38:58.799976+0000","last_undegraded":"2024-03-24T00:38:58.799976+0000","last_fullsized":"2024-03-24T00:38:58.799976+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:39:26.969331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800170+0000","last_change":"2024-03-24T00:38:51.784065+0000","last_active":"2024-03-24T00:38:58.800170+0000","last_peered":"2024-03-24T00:38:58.800170+0000","last_clean":"2024-03-24T00:38:58.800170+0000","last_became_active":"2024-03-24T00:38:51.783789+0000","last_became_peered":"2024-03-24T00:38:51.783789+0000","last_unstale":"2024-03-24T00:38:58.800170+0000","last_undegraded":"2024-03-24T00:38:58.800170+0000","last_fullsized":"2024-03-24T00:38:58.800170+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:23:25.006444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676753+0000","last_change":"2024-03-24T00:38:51.777458+0000","last_active":"2024-03-24T00:38:59.676753+0000","last_peered":"2024-03-24T00:38:59.676753+0000","last_clean":"2024-03-24T00:38:59.676753+0000","last_became_active":"2024-03-24T00:38:51.777189+0000","last_became_peered":"2024-03-24T00:38:51.777189+0000","last_unstale":"2024-03-24T00:38:59.676753+0000","last_undegraded":"2024-03-24T00:38:59.676753+0000","last_fullsized":"2024-03-24T00:38:59.676753+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:05.055760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676860+0000","last_change":"2024-03-24T00:38:52.786644+0000","last_active":"2024-03-24T00:38:59.676860+0000","last_peered":"2024-03-24T00:38:59.676860+0000","last_clean":"2024-03-24T00:38:59.676860+0000","last_became_active":"2024-03-24T00:38:52.786348+0000","last_became_peered":"2024-03-24T00:38:52.786348+0000","last_unstale":"2024-03-24T00:38:59.676860+0000","last_undegraded":"2024-03-24T00:38:59.676860+0000","last_fullsized":"2024-03-24T00:38:59.676860+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:36:54.790308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676080+0000","last_change":"2024-03-24T00:38:56.816034+0000","last_active":"2024-03-24T00:38:59.676080+0000","last_peered":"2024-03-24T00:38:59.676080+0000","last_clean":"2024-03-24T00:38:59.676080+0000","last_became_active":"2024-03-24T00:38:56.815608+0000","last_became_peered":"2024-03-24T00:38:56.815608+0000","last_unstale":"2024-03-24T00:38:59.676080+0000","last_undegraded":"2024-03-24T00:38:59.676080+0000","last_fullsized":"2024-03-24T00:38:59.676080+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:27:27.336047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800713+0000","last_change":"2024-03-24T00:38:51.782948+0000","last_active":"2024-03-24T00:38:58.800713+0000","last_peered":"2024-03-24T00:38:58.800713+0000","last_clean":"2024-03-24T00:38:58.800713+0000","last_became_active":"2024-03-24T00:38:51.780063+0000","last_became_peered":"2024-03-24T00:38:51.780063+0000","last_unstale":"2024-03-24T00:38:58.800713+0000","last_undegraded":"2024-03-24T00:38:58.800713+0000","last_fullsized":"2024-03-24T00:38:58.800713+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:05.055760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801390+0000","last_change":"2024-03-24T00:38:52.778373+0000","last_active":"2024-03-24T00:38:58.801390+0000","last_peered":"2024-03-24T00:38:58.801390+0000","last_clean":"2024-03-24T00:38:58.801390+0000","last_became_active":"2024-03-24T00:38:52.778019+0000","last_became_peered":"2024-03-24T00:38:52.778019+0000","last_unstale":"2024-03-24T00:38:58.801390+0000","last_undegraded":"2024-03-24T00:38:58.801390+0000","last_fullsized":"2024-03-24T00:38:58.801390+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:09:31.127048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800819+0000","last_change":"2024-03-24T00:38:56.818404+0000","last_active":"2024-03-24T00:38:58.800819+0000","last_peered":"2024-03-24T00:38:58.800819+0000","last_clean":"2024-03-24T00:38:58.800819+0000","last_became_active":"2024-03-24T00:38:56.817801+0000","last_became_peered":"2024-03-24T00:38:56.817801+0000","last_unstale":"2024-03-24T00:38:58.800819+0000","last_undegraded":"2024-03-24T00:38:58.800819+0000","last_fullsized":"2024-03-24T00:38:58.800819+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:15:28.193092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676081+0000","last_change":"2024-03-24T00:38:52.780120+0000","last_active":"2024-03-24T00:38:59.676081+0000","last_peered":"2024-03-24T00:38:59.676081+0000","last_clean":"2024-03-24T00:38:59.676081+0000","last_became_active":"2024-03-24T00:38:52.779624+0000","last_became_peered":"2024-03-24T00:38:52.779624+0000","last_unstale":"2024-03-24T00:38:59.676081+0000","last_undegraded":"2024-03-24T00:38:59.676081+0000","last_fullsized":"2024-03-24T00:38:59.676081+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:54:01.232233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676019+0000","last_change":"2024-03-24T00:38:52.783615+0000","last_active":"2024-03-24T00:38:59.676019+0000","last_peered":"2024-03-24T00:38:59.676019+0000","last_clean":"2024-03-24T00:38:59.676019+0000","last_became_active":"2024-03-24T00:38:52.783339+0000","last_became_peered":"2024-03-24T00:38:52.783339+0000","last_unstale":"2024-03-24T00:38:59.676019+0000","last_undegraded":"2024-03-24T00:38:59.676019+0000","last_fullsized":"2024-03-24T00:38:59.676019+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:51:22.179128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802656+0000","last_change":"2024-03-24T00:38:52.793626+0000","last_active":"2024-03-24T00:38:58.802656+0000","last_peered":"2024-03-24T00:38:58.802656+0000","last_clean":"2024-03-24T00:38:58.802656+0000","last_became_active":"2024-03-24T00:38:52.793322+0000","last_became_peered":"2024-03-24T00:38:52.793322+0000","last_unstale":"2024-03-24T00:38:58.802656+0000","last_undegraded":"2024-03-24T00:38:58.802656+0000","last_fullsized":"2024-03-24T00:38:58.802656+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:57:30.131729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676831+0000","last_change":"2024-03-24T00:38:52.791203+0000","last_active":"2024-03-24T00:38:59.676831+0000","last_peered":"2024-03-24T00:38:59.676831+0000","last_clean":"2024-03-24T00:38:59.676831+0000","last_became_active":"2024-03-24T00:38:52.790914+0000","last_became_peered":"2024-03-24T00:38:52.790914+0000","last_unstale":"2024-03-24T00:38:59.676831+0000","last_undegraded":"2024-03-24T00:38:59.676831+0000","last_fullsized":"2024-03-24T00:38:59.676831+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:21:12.524662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800715+0000","last_change":"2024-03-24T00:38:52.788614+0000","last_active":"2024-03-24T00:38:58.800715+0000","last_peered":"2024-03-24T00:38:58.800715+0000","last_clean":"2024-03-24T00:38:58.800715+0000","last_became_active":"2024-03-24T00:38:52.788155+0000","last_became_peered":"2024-03-24T00:38:52.788155+0000","last_unstale":"2024-03-24T00:38:58.800715+0000","last_undegraded":"2024-03-24T00:38:58.800715+0000","last_fullsized":"2024-03-24T00:38:58.800715+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:54:01.232233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975795+0000","last_change":"2024-03-24T00:38:52.790018+0000","last_active":"2024-03-24T00:38:58.975795+0000","last_peered":"2024-03-24T00:38:58.975795+0000","last_clean":"2024-03-24T00:38:58.975795+0000","last_became_active":"2024-03-24T00:38:52.789744+0000","last_became_peered":"2024-03-24T00:38:52.789744+0000","last_unstale":"2024-03-24T00:38:58.975795+0000","last_undegraded":"2024-03-24T00:38:58.975795+0000","last_fullsized":"2024-03-24T00:38:58.975795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:46:05.527509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676904+0000","last_change":"2024-03-24T00:38:52.791896+0000","last_active":"2024-03-24T00:38:59.676904+0000","last_peered":"2024-03-24T00:38:59.676904+0000","last_clean":"2024-03-24T00:38:59.676904+0000","last_became_active":"2024-03-24T00:38:52.789711+0000","last_became_peered":"2024-03-24T00:38:52.789711+0000","last_unstale":"2024-03-24T00:38:59.676904+0000","last_undegraded":"2024-03-24T00:38:59.676904+0000","last_fullsized":"2024-03-24T00:38:59.676904+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:21:12.524662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800598+0000","last_change":"2024-03-24T00:38:52.780901+0000","last_active":"2024-03-24T00:38:58.800598+0000","last_peered":"2024-03-24T00:38:58.800598+0000","last_clean":"2024-03-24T00:38:58.800598+0000","last_became_active":"2024-03-24T00:38:52.780625+0000","last_became_peered":"2024-03-24T00:38:52.780625+0000","last_unstale":"2024-03-24T00:38:58.800598+0000","last_undegraded":"2024-03-24T00:38:58.800598+0000","last_fullsized":"2024-03-24T00:38:58.800598+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:42:38.046862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801027+0000","last_change":"2024-03-24T00:38:52.784339+0000","last_active":"2024-03-24T00:38:58.801027+0000","last_peered":"2024-03-24T00:38:58.801027+0000","last_clean":"2024-03-24T00:38:58.801027+0000","last_became_active":"2024-03-24T00:38:52.783135+0000","last_became_peered":"2024-03-24T00:38:52.783135+0000","last_unstale":"2024-03-24T00:38:58.801027+0000","last_undegraded":"2024-03-24T00:38:58.801027+0000","last_fullsized":"2024-03-24T00:38:58.801027+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:21:12.524662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800355+0000","last_change":"2024-03-24T00:38:52.788394+0000","last_active":"2024-03-24T00:38:58.800355+0000","last_peered":"2024-03-24T00:38:58.800355+0000","last_clean":"2024-03-24T00:38:58.800355+0000","last_became_active":"2024-03-24T00:38:52.788105+0000","last_became_peered":"2024-03-24T00:38:52.788105+0000","last_unstale":"2024-03-24T00:38:58.800355+0000","last_undegraded":"2024-03-24T00:38:58.800355+0000","last_fullsized":"2024-03-24T00:38:58.800355+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:04:24.260803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802590+0000","last_change":"2024-03-24T00:38:52.787942+0000","last_active":"2024-03-24T00:38:58.802590+0000","last_peered":"2024-03-24T00:38:58.802590+0000","last_clean":"2024-03-24T00:38:58.802590+0000","last_became_active":"2024-03-24T00:38:52.787691+0000","last_became_peered":"2024-03-24T00:38:52.787691+0000","last_unstale":"2024-03-24T00:38:58.802590+0000","last_undegraded":"2024-03-24T00:38:58.802590+0000","last_fullsized":"2024-03-24T00:38:58.802590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:17:42.703499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676562+0000","last_change":"2024-03-24T00:38:52.780488+0000","last_active":"2024-03-24T00:38:59.676562+0000","last_peered":"2024-03-24T00:38:59.676562+0000","last_clean":"2024-03-24T00:38:59.676562+0000","last_became_active":"2024-03-24T00:38:52.780227+0000","last_became_peered":"2024-03-24T00:38:52.780227+0000","last_unstale":"2024-03-24T00:38:59.676562+0000","last_undegraded":"2024-03-24T00:38:59.676562+0000","last_fullsized":"2024-03-24T00:38:59.676562+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:39:26.969331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676482+0000","last_change":"2024-03-24T00:38:52.785757+0000","last_active":"2024-03-24T00:38:59.676482+0000","last_peered":"2024-03-24T00:38:59.676482+0000","last_clean":"2024-03-24T00:38:59.676482+0000","last_became_active":"2024-03-24T00:38:52.785476+0000","last_became_peered":"2024-03-24T00:38:52.785476+0000","last_unstale":"2024-03-24T00:38:59.676482+0000","last_undegraded":"2024-03-24T00:38:59.676482+0000","last_fullsized":"2024-03-24T00:38:59.676482+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:39:26.969331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.677056+0000","last_change":"2024-03-24T00:38:52.786263+0000","last_active":"2024-03-24T00:38:59.677056+0000","last_peered":"2024-03-24T00:38:59.677056+0000","last_clean":"2024-03-24T00:38:59.677056+0000","last_became_active":"2024-03-24T00:38:52.785957+0000","last_became_peered":"2024-03-24T00:38:52.785957+0000","last_unstale":"2024-03-24T00:38:59.677056+0000","last_undegraded":"2024-03-24T00:38:59.677056+0000","last_fullsized":"2024-03-24T00:38:59.677056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:02:43.127453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676749+0000","last_change":"2024-03-24T00:38:52.786178+0000","last_active":"2024-03-24T00:38:59.676749+0000","last_peered":"2024-03-24T00:38:59.676749+0000","last_clean":"2024-03-24T00:38:59.676749+0000","last_became_active":"2024-03-24T00:38:52.785894+0000","last_became_peered":"2024-03-24T00:38:52.785894+0000","last_unstale":"2024-03-24T00:38:59.676749+0000","last_undegraded":"2024-03-24T00:38:59.676749+0000","last_fullsized":"2024-03-24T00:38:59.676749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:36:19.927252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801779+0000","last_change":"2024-03-24T00:38:52.787876+0000","last_active":"2024-03-24T00:38:58.801779+0000","last_peered":"2024-03-24T00:38:58.801779+0000","last_clean":"2024-03-24T00:38:58.801779+0000","last_became_active":"2024-03-24T00:38:52.787243+0000","last_became_peered":"2024-03-24T00:38:52.787243+0000","last_unstale":"2024-03-24T00:38:58.801779+0000","last_undegraded":"2024-03-24T00:38:58.801779+0000","last_fullsized":"2024-03-24T00:38:58.801779+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:36:54.790308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800934+0000","last_change":"2024-03-24T00:38:52.776125+0000","last_active":"2024-03-24T00:38:58.800934+0000","last_peered":"2024-03-24T00:38:58.800934+0000","last_clean":"2024-03-24T00:38:58.800934+0000","last_became_active":"2024-03-24T00:38:52.775255+0000","last_became_peered":"2024-03-24T00:38:52.775255+0000","last_unstale":"2024-03-24T00:38:58.800934+0000","last_undegraded":"2024-03-24T00:38:58.800934+0000","last_fullsized":"2024-03-24T00:38:58.800934+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:46:05.527509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.974980+0000","last_change":"2024-03-24T00:38:52.781546+0000","last_active":"2024-03-24T00:38:58.974980+0000","last_peered":"2024-03-24T00:38:58.974980+0000","last_clean":"2024-03-24T00:38:58.974980+0000","last_became_active":"2024-03-24T00:38:52.781220+0000","last_became_peered":"2024-03-24T00:38:52.781220+0000","last_unstale":"2024-03-24T00:38:58.974980+0000","last_undegraded":"2024-03-24T00:38:58.974980+0000","last_fullsized":"2024-03-24T00:38:58.974980+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:41:59.361242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676394+0000","last_change":"2024-03-24T00:38:52.773210+0000","last_active":"2024-03-24T00:38:59.676394+0000","last_peered":"2024-03-24T00:38:59.676394+0000","last_clean":"2024-03-24T00:38:59.676394+0000","last_became_active":"2024-03-24T00:38:52.772887+0000","last_became_peered":"2024-03-24T00:38:52.772887+0000","last_unstale":"2024-03-24T00:38:59.676394+0000","last_undegraded":"2024-03-24T00:38:59.676394+0000","last_fullsized":"2024-03-24T00:38:59.676394+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:27:23.302869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800436+0000","last_change":"2024-03-24T00:38:52.784496+0000","last_active":"2024-03-24T00:38:58.800436+0000","last_peered":"2024-03-24T00:38:58.800436+0000","last_clean":"2024-03-24T00:38:58.800436+0000","last_became_active":"2024-03-24T00:38:52.784234+0000","last_became_peered":"2024-03-24T00:38:52.784234+0000","last_unstale":"2024-03-24T00:38:58.800436+0000","last_undegraded":"2024-03-24T00:38:58.800436+0000","last_fullsized":"2024-03-24T00:38:58.800436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:36:54.790308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802210+0000","last_change":"2024-03-24T00:38:52.786080+0000","last_active":"2024-03-24T00:38:58.802210+0000","last_peered":"2024-03-24T00:38:58.802210+0000","last_clean":"2024-03-24T00:38:58.802210+0000","last_became_active":"2024-03-24T00:38:52.785803+0000","last_became_peered":"2024-03-24T00:38:52.785803+0000","last_unstale":"2024-03-24T00:38:58.802210+0000","last_undegraded":"2024-03-24T00:38:58.802210+0000","last_fullsized":"2024-03-24T00:38:58.802210+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:51:22.179128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676587+0000","last_change":"2024-03-24T00:38:52.788391+0000","last_active":"2024-03-24T00:38:59.676587+0000","last_peered":"2024-03-24T00:38:59.676587+0000","last_clean":"2024-03-24T00:38:59.676587+0000","last_became_active":"2024-03-24T00:38:52.788122+0000","last_became_peered":"2024-03-24T00:38:52.788122+0000","last_unstale":"2024-03-24T00:38:59.676587+0000","last_undegraded":"2024-03-24T00:38:59.676587+0000","last_fullsized":"2024-03-24T00:38:59.676587+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:41:59.361242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676867+0000","last_change":"2024-03-24T00:38:52.787696+0000","last_active":"2024-03-24T00:38:59.676867+0000","last_peered":"2024-03-24T00:38:59.676867+0000","last_clean":"2024-03-24T00:38:59.676867+0000","last_became_active":"2024-03-24T00:38:52.787431+0000","last_became_peered":"2024-03-24T00:38:52.787431+0000","last_unstale":"2024-03-24T00:38:59.676867+0000","last_undegraded":"2024-03-24T00:38:59.676867+0000","last_fullsized":"2024-03-24T00:38:59.676867+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:46:05.527509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.677244+0000","last_change":"2024-03-24T00:38:52.791775+0000","last_active":"2024-03-24T00:38:59.677244+0000","last_peered":"2024-03-24T00:38:59.677244+0000","last_clean":"2024-03-24T00:38:59.677244+0000","last_became_active":"2024-03-24T00:38:52.788742+0000","last_became_peered":"2024-03-24T00:38:52.788742+0000","last_unstale":"2024-03-24T00:38:59.677244+0000","last_undegraded":"2024-03-24T00:38:59.677244+0000","last_fullsized":"2024-03-24T00:38:59.677244+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:08:00.887641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.799866+0000","last_change":"2024-03-24T00:38:52.790297+0000","last_active":"2024-03-24T00:38:58.799866+0000","last_peered":"2024-03-24T00:38:58.799866+0000","last_clean":"2024-03-24T00:38:58.799866+0000","last_became_active":"2024-03-24T00:38:52.789879+0000","last_became_peered":"2024-03-24T00:38:52.789879+0000","last_unstale":"2024-03-24T00:38:58.799866+0000","last_undegraded":"2024-03-24T00:38:58.799866+0000","last_fullsized":"2024-03-24T00:38:58.799866+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:41:59.361242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676732+0000","last_change":"2024-03-24T00:38:52.780267+0000","last_active":"2024-03-24T00:38:59.676732+0000","last_peered":"2024-03-24T00:38:59.676732+0000","last_clean":"2024-03-24T00:38:59.676732+0000","last_became_active":"2024-03-24T00:38:52.779973+0000","last_became_peered":"2024-03-24T00:38:52.779973+0000","last_unstale":"2024-03-24T00:38:59.676732+0000","last_undegraded":"2024-03-24T00:38:59.676732+0000","last_fullsized":"2024-03-24T00:38:59.676732+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:36:54.790308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802578+0000","last_change":"2024-03-24T00:38:52.776890+0000","last_active":"2024-03-24T00:38:58.802578+0000","last_peered":"2024-03-24T00:38:58.802578+0000","last_clean":"2024-03-24T00:38:58.802578+0000","last_became_active":"2024-03-24T00:38:52.776573+0000","last_became_peered":"2024-03-24T00:38:52.776573+0000","last_unstale":"2024-03-24T00:38:58.802578+0000","last_undegraded":"2024-03-24T00:38:58.802578+0000","last_fullsized":"2024-03-24T00:38:58.802578+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:02:43.127453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676561+0000","last_change":"2024-03-24T00:38:52.783960+0000","last_active":"2024-03-24T00:38:59.676561+0000","last_peered":"2024-03-24T00:38:59.676561+0000","last_clean":"2024-03-24T00:38:59.676561+0000","last_became_active":"2024-03-24T00:38:52.783606+0000","last_became_peered":"2024-03-24T00:38:52.783606+0000","last_unstale":"2024-03-24T00:38:59.676561+0000","last_undegraded":"2024-03-24T00:38:59.676561+0000","last_fullsized":"2024-03-24T00:38:59.676561+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:21:00.861116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802672+0000","last_change":"2024-03-24T00:38:52.784222+0000","last_active":"2024-03-24T00:38:58.802672+0000","last_peered":"2024-03-24T00:38:58.802672+0000","last_clean":"2024-03-24T00:38:58.802672+0000","last_became_active":"2024-03-24T00:38:52.783951+0000","last_became_peered":"2024-03-24T00:38:52.783951+0000","last_unstale":"2024-03-24T00:38:58.802672+0000","last_undegraded":"2024-03-24T00:38:58.802672+0000","last_fullsized":"2024-03-24T00:38:58.802672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:04:24.260803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676177+0000","last_change":"2024-03-24T00:38:52.774199+0000","last_active":"2024-03-24T00:38:59.676177+0000","last_peered":"2024-03-24T00:38:59.676177+0000","last_clean":"2024-03-24T00:38:59.676177+0000","last_became_active":"2024-03-24T00:38:52.773914+0000","last_became_peered":"2024-03-24T00:38:52.773914+0000","last_unstale":"2024-03-24T00:38:59.676177+0000","last_undegraded":"2024-03-24T00:38:59.676177+0000","last_fullsized":"2024-03-24T00:38:59.676177+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:54:01.232233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975533+0000","last_change":"2024-03-24T00:38:52.778927+0000","last_active":"2024-03-24T00:38:58.975533+0000","last_peered":"2024-03-24T00:38:58.975533+0000","last_clean":"2024-03-24T00:38:58.975533+0000","last_became_active":"2024-03-24T00:38:52.778655+0000","last_became_peered":"2024-03-24T00:38:52.778655+0000","last_unstale":"2024-03-24T00:38:58.975533+0000","last_undegraded":"2024-03-24T00:38:58.975533+0000","last_fullsized":"2024-03-24T00:38:58.975533+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:21:12.524662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.677156+0000","last_change":"2024-03-24T00:38:52.785456+0000","last_active":"2024-03-24T00:38:59.677156+0000","last_peered":"2024-03-24T00:38:59.677156+0000","last_clean":"2024-03-24T00:38:59.677156+0000","last_became_active":"2024-03-24T00:38:52.785150+0000","last_became_peered":"2024-03-24T00:38:52.785150+0000","last_unstale":"2024-03-24T00:38:59.677156+0000","last_undegraded":"2024-03-24T00:38:59.677156+0000","last_fullsized":"2024-03-24T00:38:59.677156+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:17:42.703499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574860,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27868,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343972,"statfs":{"total":96636764160,"available":96608227328,"internally_reserved":0,"allocated":864256,"data_stored":698343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574859,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":111235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574859,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":107483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574858,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":278528,"data_stored":107519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":111235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574859,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":107393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574859,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":274432,"data_stored":107553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27872,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343968,"statfs":{"total":96636764160,"available":96608223232,"internally_reserved":0,"allocated":872448,"data_stored":698399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:39:15.629 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:39:15.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.766+0000 7f6e28369640 1 Processor -- start 2024-03-24T00:39:15.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.766+0000 7f6e28369640 1 -- start start 2024-03-24T00:39:15.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.766+0000 7f6e28369640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6e200d1550 0x7f6e2006cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:15.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.766+0000 7f6e28369640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f6e200b2780 0x7f6e200b2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:15.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.766+0000 7f6e28369640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e200d1930 con 0x7f6e20099d70 2024-03-24T00:39:15.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.766+0000 7f6e28369640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e2006d420 con 0x7f6e200b2780 2024-03-24T00:39:15.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.766+0000 7f6e28369640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e2006ca50 con 0x7f6e200d1550 2024-03-24T00:39:15.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.767+0000 7f6e26105640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6e200d1550 0x7f6e2006cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:15.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.767+0000 7f6e25904640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f6e200b2780 0x7f6e200b2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:15.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.767+0000 7f6e26906640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f6e20099d70 0x7f6e200cfd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:49966/0 (socket says 172.21.15.67:49966) 2024-03-24T00:39:15.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.767+0000 7f6e26906640 1 -- 172.21.15.67:0/767055916 learned_addr learned my addr 172.21.15.67:0/767055916 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:15.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.767+0000 7f6e25904640 1 -- 172.21.15.67:0/767055916 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6e200d1550 msgr2=0x7f6e2006cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:15.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.767+0000 7f6e25904640 1 --2- 172.21.15.67:0/767055916 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6e200d1550 0x7f6e2006cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.767+0000 7f6e25904640 1 -- 172.21.15.67:0/767055916 >> v1:172.21.15.72:6789/0 conn(0x7f6e20099d70 legacy=0x7f6e200cfd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:15.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.767+0000 7f6e25904640 1 -- 172.21.15.67:0/767055916 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e200b1f60 con 0x7f6e200b2780 2024-03-24T00:39:15.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.767+0000 7f6e25103640 1 -- 172.21.15.67:0/767055916 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2544870099 0 0) 0x7f6e200d1930 con 0x7f6e20099d70 2024-03-24T00:39:15.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.768+0000 7f6e25904640 1 --2- 172.21.15.67:0/767055916 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6e200b2780 0x7f6e200b2c00 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6e14009d90 tx=0x7f6e1402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ed9bd17b6b60d5d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:15.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.768+0000 7f6e25103640 1 -- 172.21.15.67:0/767055916 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e14039070 con 0x7f6e200b2780 2024-03-24T00:39:15.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.768+0000 7f6e25103640 1 -- 172.21.15.67:0/767055916 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e1402cc50 con 0x7f6e200b2780 2024-03-24T00:39:15.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.768+0000 7f6e25103640 1 -- 172.21.15.67:0/767055916 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e14034440 con 0x7f6e200b2780 2024-03-24T00:39:15.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.768+0000 7f6e28369640 1 -- 172.21.15.67:0/767055916 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6e200b2780 msgr2=0x7f6e200b2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:15.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.768+0000 7f6e28369640 1 --2- 172.21.15.67:0/767055916 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6e200b2780 0x7f6e200b2c00 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6e14009d90 tx=0x7f6e1402bea0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.768+0000 7f6e28369640 1 -- 172.21.15.67:0/767055916 shutdown_connections 2024-03-24T00:39:15.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.768+0000 7f6e28369640 1 --2- 172.21.15.67:0/767055916 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6e200b2780 0x7f6e200b2c00 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.768+0000 7f6e28369640 1 --2- 172.21.15.67:0/767055916 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6e200d1550 0x7f6e2006cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.768+0000 7f6e28369640 1 -- 172.21.15.67:0/767055916 >> 172.21.15.67:0/767055916 conn(0x7f6e200b36e0 msgr2=0x7f6e200b3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:15.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.768+0000 7f6e28369640 1 -- 172.21.15.67:0/767055916 shutdown_connections 2024-03-24T00:39:15.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.768+0000 7f6e28369640 1 -- 172.21.15.67:0/767055916 wait complete. 2024-03-24T00:39:15.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.770+0000 7f6e28369640 1 Processor -- start 2024-03-24T00:39:15.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.770+0000 7f6e28369640 1 -- start start 2024-03-24T00:39:15.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.770+0000 7f6e28369640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6e20099d70 0x7f6e201f81d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:15.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.770+0000 7f6e28369640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e200b2780 0x7f6e201f4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:15.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.770+0000 7f6e28369640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6e200d1550 0x7f6e2015b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:15.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.770+0000 7f6e28369640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e2006c400 con 0x7f6e200b2780 2024-03-24T00:39:15.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.770+0000 7f6e28369640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e200b2520 con 0x7f6e200d1550 2024-03-24T00:39:15.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.770+0000 7f6e28369640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e2006c6e0 con 0x7f6e20099d70 2024-03-24T00:39:15.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.770+0000 7f6e25904640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e200b2780 0x7f6e201f4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:15.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.770+0000 7f6e26906640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6e200d1550 0x7f6e2015b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:15.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.770+0000 7f6e26105640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6e20099d70 0x7f6e201f81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:15.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.771+0000 7f6e26906640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6e200d1550 0x7f6e2015b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:47694/0 (socket says 172.21.15.67:47694) 2024-03-24T00:39:15.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.771+0000 7f6e26906640 1 -- 172.21.15.67:0/3806972544 learned_addr learned my addr 172.21.15.67:0/3806972544 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:15.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.771+0000 7f6e25904640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e200b2780 0x7f6e201f4240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35988/0 (socket says 172.21.15.67:35988) 2024-03-24T00:39:15.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.771+0000 7f6e26906640 1 -- 172.21.15.67:0/3806972544 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6e20099d70 msgr2=0x7f6e201f81d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:15.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.771+0000 7f6e26906640 1 --2- 172.21.15.67:0/3806972544 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6e20099d70 0x7f6e201f81d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.771+0000 7f6e26906640 1 -- 172.21.15.67:0/3806972544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e200b2780 msgr2=0x7f6e201f4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:15.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.771+0000 7f6e26906640 1 --2- 172.21.15.67:0/3806972544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e200b2780 0x7f6e201f4240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.771+0000 7f6e26906640 1 -- 172.21.15.67:0/3806972544 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e201f8710 con 0x7f6e200d1550 2024-03-24T00:39:15.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.771+0000 7f6e26906640 1 --2- 172.21.15.67:0/3806972544 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6e200d1550 0x7f6e2015b540 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6e1000c450 tx=0x7f6e1000c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:15.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.771+0000 7f6e1effd640 1 -- 172.21.15.67:0/3806972544 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e10018070 con 0x7f6e200d1550 2024-03-24T00:39:15.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.771+0000 7f6e1effd640 1 -- 172.21.15.67:0/3806972544 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e10007500 con 0x7f6e200d1550 2024-03-24T00:39:15.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.771+0000 7f6e28369640 1 -- 172.21.15.67:0/3806972544 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e201f4780 con 0x7f6e200d1550 2024-03-24T00:39:15.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.771+0000 7f6e1effd640 1 -- 172.21.15.67:0/3806972544 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e10004a20 con 0x7f6e200d1550 2024-03-24T00:39:15.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.771+0000 7f6e28369640 1 -- 172.21.15.67:0/3806972544 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e200c93f0 con 0x7f6e200d1550 2024-03-24T00:39:15.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.772+0000 7f6e28369640 1 -- 172.21.15.67:0/3806972544 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e2004f400 con 0x7f6e200d1550 2024-03-24T00:39:15.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.773+0000 7f6e1effd640 1 -- 172.21.15.67:0/3806972544 <== mon.1 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6e1001b070 con 0x7f6e200d1550 2024-03-24T00:39:15.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.773+0000 7f6e1effd640 1 --2- 172.21.15.67:0/3806972544 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6e0c06bbd0 0x7f6e0c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:15.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.773+0000 7f6e1effd640 1 -- 172.21.15.67:0/3806972544 <== mon.1 v2:172.21.15.72:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f6e10089f10 con 0x7f6e200d1550 2024-03-24T00:39:15.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.773+0000 7f6e26105640 1 --2- 172.21.15.67:0/3806972544 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6e0c06bbd0 0x7f6e0c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:15.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.774+0000 7f6e26105640 1 --2- 172.21.15.67:0/3806972544 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6e0c06bbd0 0x7f6e0c06e090 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6e18009aa0 tx=0x7f6e18009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:15.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.779+0000 7f6e1effd640 1 -- 172.21.15.67:0/3806972544 <== mon.1 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6e201f8710 con 0x7f6e200d1550 2024-03-24T00:39:15.867 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:39:15.870 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:39:15.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.961+0000 7f6e28369640 1 -- 172.21.15.67:0/3806972544 --> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6e200cc680 con 0x7f6e0c06bbd0 2024-03-24T00:39:15.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.969+0000 7f6e1effd640 1 -- 172.21.15.67:0/3806972544 <== mgr.4105 v2:172.21.15.72:6832/2768292274 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+332534 (secure 0 0 0) 0x7f6e200cc680 con 0x7f6e0c06bbd0 2024-03-24T00:39:15.973 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:39:15.975 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-24T00:39:15.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.973+0000 7f6e28369640 1 -- 172.21.15.67:0/3806972544 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6e0c06bbd0 msgr2=0x7f6e0c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:15.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.973+0000 7f6e28369640 1 --2- 172.21.15.67:0/3806972544 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6e0c06bbd0 0x7f6e0c06e090 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6e18009aa0 tx=0x7f6e18009190 comp rx=0 tx=0).stop 2024-03-24T00:39:15.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.973+0000 7f6e28369640 1 -- 172.21.15.67:0/3806972544 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6e200d1550 msgr2=0x7f6e2015b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:15.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.973+0000 7f6e28369640 1 --2- 172.21.15.67:0/3806972544 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6e200d1550 0x7f6e2015b540 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6e1000c450 tx=0x7f6e1000c910 comp rx=0 tx=0).stop 2024-03-24T00:39:15.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.973+0000 7f6e28369640 1 -- 172.21.15.67:0/3806972544 shutdown_connections 2024-03-24T00:39:15.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.973+0000 7f6e28369640 1 --2- 172.21.15.67:0/3806972544 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6e0c06bbd0 0x7f6e0c06e090 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.973+0000 7f6e28369640 1 --2- 172.21.15.67:0/3806972544 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6e200d1550 0x7f6e2015b540 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.973+0000 7f6e28369640 1 --2- 172.21.15.67:0/3806972544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e200b2780 0x7f6e201f4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.973+0000 7f6e28369640 1 --2- 172.21.15.67:0/3806972544 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6e20099d70 0x7f6e201f81d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.973+0000 7f6e28369640 1 -- 172.21.15.67:0/3806972544 >> 172.21.15.67:0/3806972544 conn(0x7f6e200b36e0 msgr2=0x7f6e200cd810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:15.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.973+0000 7f6e28369640 1 -- 172.21.15.67:0/3806972544 shutdown_connections 2024-03-24T00:39:15.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:15.973+0000 7f6e28369640 1 -- 172.21.15.67:0/3806972544 wait complete. 2024-03-24T00:39:15.984 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2024-03-24T00:39:15.328569+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219476,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755244,"statfs":{"total":773094113280,"available":772869369856,"internally_reserved":0,"allocated":3383296,"data_stored":2049160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002600"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":28,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676794+0000","last_change":"2024-03-24T00:38:51.779360+0000","last_active":"2024-03-24T00:38:59.676794+0000","last_peered":"2024-03-24T00:38:59.676794+0000","last_clean":"2024-03-24T00:38:59.676794+0000","last_became_active":"2024-03-24T00:38:51.779066+0000","last_became_peered":"2024-03-24T00:38:51.779066+0000","last_unstale":"2024-03-24T00:38:59.676794+0000","last_undegraded":"2024-03-24T00:38:59.676794+0000","last_fullsized":"2024-03-24T00:38:59.676794+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:44:08.508305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800440+0000","last_change":"2024-03-24T00:38:52.781668+0000","last_active":"2024-03-24T00:38:58.800440+0000","last_peered":"2024-03-24T00:38:58.800440+0000","last_clean":"2024-03-24T00:38:58.800440+0000","last_became_active":"2024-03-24T00:38:52.781401+0000","last_became_peered":"2024-03-24T00:38:52.781401+0000","last_unstale":"2024-03-24T00:38:58.800440+0000","last_undegraded":"2024-03-24T00:38:58.800440+0000","last_fullsized":"2024-03-24T00:38:58.800440+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:01:21.675152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800050+0000","last_change":"2024-03-24T00:38:57.197819+0000","last_active":"2024-03-24T00:38:58.800050+0000","last_peered":"2024-03-24T00:38:58.800050+0000","last_clean":"2024-03-24T00:38:58.800050+0000","last_became_active":"2024-03-24T00:38:57.196369+0000","last_became_peered":"2024-03-24T00:38:57.196369+0000","last_unstale":"2024-03-24T00:38:58.800050+0000","last_undegraded":"2024-03-24T00:38:58.800050+0000","last_fullsized":"2024-03-24T00:38:58.800050+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:04:26.722612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800456+0000","last_change":"2024-03-24T00:38:51.772822+0000","last_active":"2024-03-24T00:38:58.800456+0000","last_peered":"2024-03-24T00:38:58.800456+0000","last_clean":"2024-03-24T00:38:58.800456+0000","last_became_active":"2024-03-24T00:38:51.772506+0000","last_became_peered":"2024-03-24T00:38:51.772506+0000","last_unstale":"2024-03-24T00:38:58.800456+0000","last_undegraded":"2024-03-24T00:38:58.800456+0000","last_fullsized":"2024-03-24T00:38:58.800456+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:09:30.121404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676632+0000","last_change":"2024-03-24T00:38:52.778282+0000","last_active":"2024-03-24T00:38:59.676632+0000","last_peered":"2024-03-24T00:38:59.676632+0000","last_clean":"2024-03-24T00:38:59.676632+0000","last_became_active":"2024-03-24T00:38:52.778003+0000","last_became_peered":"2024-03-24T00:38:52.778003+0000","last_unstale":"2024-03-24T00:38:59.676632+0000","last_undegraded":"2024-03-24T00:38:59.676632+0000","last_fullsized":"2024-03-24T00:38:59.676632+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:01:21.675152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801128+0000","last_change":"2024-03-24T00:38:56.822022+0000","last_active":"2024-03-24T00:38:58.801128+0000","last_peered":"2024-03-24T00:38:58.801128+0000","last_clean":"2024-03-24T00:38:58.801128+0000","last_became_active":"2024-03-24T00:38:56.821498+0000","last_became_peered":"2024-03-24T00:38:56.821498+0000","last_unstale":"2024-03-24T00:38:58.801128+0000","last_undegraded":"2024-03-24T00:38:58.801128+0000","last_fullsized":"2024-03-24T00:38:58.801128+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:02:47.160631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.677000+0000","last_change":"2024-03-24T00:38:51.781748+0000","last_active":"2024-03-24T00:38:59.677000+0000","last_peered":"2024-03-24T00:38:59.677000+0000","last_clean":"2024-03-24T00:38:59.677000+0000","last_became_active":"2024-03-24T00:38:51.781399+0000","last_became_peered":"2024-03-24T00:38:51.781399+0000","last_unstale":"2024-03-24T00:38:59.677000+0000","last_undegraded":"2024-03-24T00:38:59.677000+0000","last_fullsized":"2024-03-24T00:38:59.677000+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:51:21.173485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802757+0000","last_change":"2024-03-24T00:38:52.794918+0000","last_active":"2024-03-24T00:38:58.802757+0000","last_peered":"2024-03-24T00:38:58.802757+0000","last_clean":"2024-03-24T00:38:58.802757+0000","last_became_active":"2024-03-24T00:38:52.794638+0000","last_became_peered":"2024-03-24T00:38:52.794638+0000","last_unstale":"2024-03-24T00:38:58.802757+0000","last_undegraded":"2024-03-24T00:38:58.802757+0000","last_fullsized":"2024-03-24T00:38:58.802757+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:03:33.725080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802076+0000","last_change":"2024-03-24T00:38:56.817596+0000","last_active":"2024-03-24T00:38:58.802076+0000","last_peered":"2024-03-24T00:38:58.802076+0000","last_clean":"2024-03-24T00:38:58.802076+0000","last_became_active":"2024-03-24T00:38:56.817300+0000","last_became_peered":"2024-03-24T00:38:56.817300+0000","last_unstale":"2024-03-24T00:38:58.802076+0000","last_undegraded":"2024-03-24T00:38:58.802076+0000","last_fullsized":"2024-03-24T00:38:58.802076+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:53:16.442548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676343+0000","last_change":"2024-03-24T00:38:51.780697+0000","last_active":"2024-03-24T00:38:59.676343+0000","last_peered":"2024-03-24T00:38:59.676343+0000","last_clean":"2024-03-24T00:38:59.676343+0000","last_became_active":"2024-03-24T00:38:51.780118+0000","last_became_peered":"2024-03-24T00:38:51.780118+0000","last_unstale":"2024-03-24T00:38:59.676343+0000","last_undegraded":"2024-03-24T00:38:59.676343+0000","last_fullsized":"2024-03-24T00:38:59.676343+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:53:39.260829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802424+0000","last_change":"2024-03-24T00:38:52.795487+0000","last_active":"2024-03-24T00:38:58.802424+0000","last_peered":"2024-03-24T00:38:58.802424+0000","last_clean":"2024-03-24T00:38:58.802424+0000","last_became_active":"2024-03-24T00:38:52.794564+0000","last_became_peered":"2024-03-24T00:38:52.794564+0000","last_unstale":"2024-03-24T00:38:58.802424+0000","last_undegraded":"2024-03-24T00:38:58.802424+0000","last_fullsized":"2024-03-24T00:38:58.802424+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:48:02.197473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801403+0000","last_change":"2024-03-24T00:38:56.815283+0000","last_active":"2024-03-24T00:38:58.801403+0000","last_peered":"2024-03-24T00:38:58.801403+0000","last_clean":"2024-03-24T00:38:58.801403+0000","last_became_active":"2024-03-24T00:38:56.814490+0000","last_became_peered":"2024-03-24T00:38:56.814490+0000","last_unstale":"2024-03-24T00:38:58.801403+0000","last_undegraded":"2024-03-24T00:38:58.801403+0000","last_fullsized":"2024-03-24T00:38:58.801403+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:08:04.920818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676589+0000","last_change":"2024-03-24T00:38:52.785835+0000","last_active":"2024-03-24T00:38:59.676589+0000","last_peered":"2024-03-24T00:38:59.676589+0000","last_clean":"2024-03-24T00:38:59.676589+0000","last_became_active":"2024-03-24T00:38:52.785539+0000","last_became_peered":"2024-03-24T00:38:52.785539+0000","last_unstale":"2024-03-24T00:38:59.676589+0000","last_undegraded":"2024-03-24T00:38:59.676589+0000","last_fullsized":"2024-03-24T00:38:59.676589+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:01:21.675152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676805+0000","last_change":"2024-03-24T00:38:51.778378+0000","last_active":"2024-03-24T00:38:59.676805+0000","last_peered":"2024-03-24T00:38:59.676805+0000","last_clean":"2024-03-24T00:38:59.676805+0000","last_became_active":"2024-03-24T00:38:51.778093+0000","last_became_peered":"2024-03-24T00:38:51.778093+0000","last_unstale":"2024-03-24T00:38:59.676805+0000","last_undegraded":"2024-03-24T00:38:59.676805+0000","last_fullsized":"2024-03-24T00:38:59.676805+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:05.055760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801870+0000","last_change":"2024-03-24T00:38:56.821541+0000","last_active":"2024-03-24T00:38:58.801870+0000","last_peered":"2024-03-24T00:38:58.801870+0000","last_clean":"2024-03-24T00:38:58.801870+0000","last_became_active":"2024-03-24T00:38:56.821097+0000","last_became_peered":"2024-03-24T00:38:56.821097+0000","last_unstale":"2024-03-24T00:38:58.801870+0000","last_undegraded":"2024-03-24T00:38:58.801870+0000","last_fullsized":"2024-03-24T00:38:58.801870+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:22:37.069261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800616+0000","last_change":"2024-03-24T00:38:52.782691+0000","last_active":"2024-03-24T00:38:58.800616+0000","last_peered":"2024-03-24T00:38:58.800616+0000","last_clean":"2024-03-24T00:38:58.800616+0000","last_became_active":"2024-03-24T00:38:52.782439+0000","last_became_peered":"2024-03-24T00:38:52.782439+0000","last_unstale":"2024-03-24T00:38:58.800616+0000","last_undegraded":"2024-03-24T00:38:58.800616+0000","last_fullsized":"2024-03-24T00:38:58.800616+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:21:00.861116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975579+0000","last_change":"2024-03-24T00:38:51.788163+0000","last_active":"2024-03-24T00:38:58.975579+0000","last_peered":"2024-03-24T00:38:58.975579+0000","last_clean":"2024-03-24T00:38:58.975579+0000","last_became_active":"2024-03-24T00:38:51.787884+0000","last_became_peered":"2024-03-24T00:38:51.787884+0000","last_unstale":"2024-03-24T00:38:58.975579+0000","last_undegraded":"2024-03-24T00:38:58.975579+0000","last_fullsized":"2024-03-24T00:38:58.975579+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:51:21.173485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802074+0000","last_change":"2024-03-24T00:38:56.822013+0000","last_active":"2024-03-24T00:38:58.802074+0000","last_peered":"2024-03-24T00:38:58.802074+0000","last_clean":"2024-03-24T00:38:58.802074+0000","last_became_active":"2024-03-24T00:38:56.821572+0000","last_became_peered":"2024-03-24T00:38:56.821572+0000","last_unstale":"2024-03-24T00:38:58.802074+0000","last_undegraded":"2024-03-24T00:38:58.802074+0000","last_fullsized":"2024-03-24T00:38:58.802074+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:28:30.107102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800068+0000","last_change":"2024-03-24T00:38:52.782995+0000","last_active":"2024-03-24T00:38:58.800068+0000","last_peered":"2024-03-24T00:38:58.800068+0000","last_clean":"2024-03-24T00:38:58.800068+0000","last_became_active":"2024-03-24T00:38:52.782426+0000","last_became_peered":"2024-03-24T00:38:52.782426+0000","last_unstale":"2024-03-24T00:38:58.800068+0000","last_undegraded":"2024-03-24T00:38:58.800068+0000","last_fullsized":"2024-03-24T00:38:58.800068+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:46:05.527509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.974999+0000","last_change":"2024-03-24T00:38:51.787097+0000","last_active":"2024-03-24T00:38:58.974999+0000","last_peered":"2024-03-24T00:38:58.974999+0000","last_clean":"2024-03-24T00:38:58.974999+0000","last_became_active":"2024-03-24T00:38:51.786714+0000","last_became_peered":"2024-03-24T00:38:51.786714+0000","last_unstale":"2024-03-24T00:38:58.974999+0000","last_undegraded":"2024-03-24T00:38:58.974999+0000","last_fullsized":"2024-03-24T00:38:58.974999+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:05.055760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975812+0000","last_change":"2024-03-24T00:38:57.196574+0000","last_active":"2024-03-24T00:38:58.975812+0000","last_peered":"2024-03-24T00:38:58.975812+0000","last_clean":"2024-03-24T00:38:58.975812+0000","last_became_active":"2024-03-24T00:38:57.196154+0000","last_became_peered":"2024-03-24T00:38:57.196154+0000","last_unstale":"2024-03-24T00:38:58.975812+0000","last_undegraded":"2024-03-24T00:38:58.975812+0000","last_fullsized":"2024-03-24T00:38:58.975812+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:01:41.498040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801624+0000","last_change":"2024-03-24T00:38:52.784816+0000","last_active":"2024-03-24T00:38:58.801624+0000","last_peered":"2024-03-24T00:38:58.801624+0000","last_clean":"2024-03-24T00:38:58.801624+0000","last_became_active":"2024-03-24T00:38:52.784492+0000","last_became_peered":"2024-03-24T00:38:52.784492+0000","last_unstale":"2024-03-24T00:38:58.801624+0000","last_undegraded":"2024-03-24T00:38:58.801624+0000","last_fullsized":"2024-03-24T00:38:58.801624+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:23:26.012087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676478+0000","last_change":"2024-03-24T00:38:51.779160+0000","last_active":"2024-03-24T00:38:59.676478+0000","last_peered":"2024-03-24T00:38:59.676478+0000","last_clean":"2024-03-24T00:38:59.676478+0000","last_became_active":"2024-03-24T00:38:51.778859+0000","last_became_peered":"2024-03-24T00:38:51.778859+0000","last_unstale":"2024-03-24T00:38:59.676478+0000","last_undegraded":"2024-03-24T00:38:59.676478+0000","last_fullsized":"2024-03-24T00:38:59.676478+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:09:30.121404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802037+0000","last_change":"2024-03-24T00:38:56.822074+0000","last_active":"2024-03-24T00:38:58.802037+0000","last_peered":"2024-03-24T00:38:58.802037+0000","last_clean":"2024-03-24T00:38:58.802037+0000","last_became_active":"2024-03-24T00:38:56.821310+0000","last_became_peered":"2024-03-24T00:38:56.821310+0000","last_unstale":"2024-03-24T00:38:58.802037+0000","last_undegraded":"2024-03-24T00:38:58.802037+0000","last_fullsized":"2024-03-24T00:38:58.802037+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:17:07.239714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676939+0000","last_change":"2024-03-24T00:38:52.773192+0000","last_active":"2024-03-24T00:38:59.676939+0000","last_peered":"2024-03-24T00:38:59.676939+0000","last_clean":"2024-03-24T00:38:59.676939+0000","last_became_active":"2024-03-24T00:38:52.772910+0000","last_became_peered":"2024-03-24T00:38:52.772910+0000","last_unstale":"2024-03-24T00:38:59.676939+0000","last_undegraded":"2024-03-24T00:38:59.676939+0000","last_fullsized":"2024-03-24T00:38:59.676939+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:46:05.527509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.677252+0000","last_change":"2024-03-24T00:38:51.778294+0000","last_active":"2024-03-24T00:38:59.677252+0000","last_peered":"2024-03-24T00:38:59.677252+0000","last_clean":"2024-03-24T00:38:59.677252+0000","last_became_active":"2024-03-24T00:38:51.777905+0000","last_became_peered":"2024-03-24T00:38:51.777905+0000","last_unstale":"2024-03-24T00:38:59.677252+0000","last_undegraded":"2024-03-24T00:38:59.677252+0000","last_fullsized":"2024-03-24T00:38:59.677252+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:09:30.121404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801340+0000","last_change":"2024-03-24T00:38:57.196338+0000","last_active":"2024-03-24T00:38:58.801340+0000","last_peered":"2024-03-24T00:38:58.801340+0000","last_clean":"2024-03-24T00:38:58.801340+0000","last_became_active":"2024-03-24T00:38:57.195957+0000","last_became_peered":"2024-03-24T00:38:57.195957+0000","last_unstale":"2024-03-24T00:38:58.801340+0000","last_undegraded":"2024-03-24T00:38:58.801340+0000","last_fullsized":"2024-03-24T00:38:58.801340+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:57:34.164907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800713+0000","last_change":"2024-03-24T00:38:52.798013+0000","last_active":"2024-03-24T00:38:58.800713+0000","last_peered":"2024-03-24T00:38:58.800713+0000","last_clean":"2024-03-24T00:38:58.800713+0000","last_became_active":"2024-03-24T00:38:52.797709+0000","last_became_peered":"2024-03-24T00:38:52.797709+0000","last_unstale":"2024-03-24T00:38:58.800713+0000","last_undegraded":"2024-03-24T00:38:58.800713+0000","last_fullsized":"2024-03-24T00:38:58.800713+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:21:12.524662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975704+0000","last_change":"2024-03-24T00:38:51.787052+0000","last_active":"2024-03-24T00:38:58.975704+0000","last_peered":"2024-03-24T00:38:58.975704+0000","last_clean":"2024-03-24T00:38:58.975704+0000","last_became_active":"2024-03-24T00:38:51.786750+0000","last_became_peered":"2024-03-24T00:38:51.786750+0000","last_unstale":"2024-03-24T00:38:58.975704+0000","last_undegraded":"2024-03-24T00:38:58.975704+0000","last_fullsized":"2024-03-24T00:38:58.975704+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:36:18.921609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800214+0000","last_change":"2024-03-24T00:38:57.198329+0000","last_active":"2024-03-24T00:38:58.800214+0000","last_peered":"2024-03-24T00:38:58.800214+0000","last_clean":"2024-03-24T00:38:58.800214+0000","last_became_active":"2024-03-24T00:38:57.196158+0000","last_became_peered":"2024-03-24T00:38:57.196158+0000","last_unstale":"2024-03-24T00:38:58.800214+0000","last_undegraded":"2024-03-24T00:38:58.800214+0000","last_fullsized":"2024-03-24T00:38:58.800214+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:52:58.271700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975518+0000","last_change":"2024-03-24T00:38:52.778532+0000","last_active":"2024-03-24T00:38:58.975518+0000","last_peered":"2024-03-24T00:38:58.975518+0000","last_clean":"2024-03-24T00:38:58.975518+0000","last_became_active":"2024-03-24T00:38:52.778181+0000","last_became_peered":"2024-03-24T00:38:52.778181+0000","last_unstale":"2024-03-24T00:38:58.975518+0000","last_undegraded":"2024-03-24T00:38:58.975518+0000","last_fullsized":"2024-03-24T00:38:58.975518+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:39:26.969331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.677108+0000","last_change":"2024-03-24T00:38:51.779037+0000","last_active":"2024-03-24T00:38:59.677108+0000","last_peered":"2024-03-24T00:38:59.677108+0000","last_clean":"2024-03-24T00:38:59.677108+0000","last_became_active":"2024-03-24T00:38:51.778651+0000","last_became_peered":"2024-03-24T00:38:51.778651+0000","last_unstale":"2024-03-24T00:38:59.677108+0000","last_undegraded":"2024-03-24T00:38:59.677108+0000","last_fullsized":"2024-03-24T00:38:59.677108+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:36:18.921609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676465+0000","last_change":"2024-03-24T00:38:57.196547+0000","last_active":"2024-03-24T00:38:59.676465+0000","last_peered":"2024-03-24T00:38:59.676465+0000","last_clean":"2024-03-24T00:38:59.676465+0000","last_became_active":"2024-03-24T00:38:57.196148+0000","last_became_peered":"2024-03-24T00:38:57.196148+0000","last_unstale":"2024-03-24T00:38:59.676465+0000","last_undegraded":"2024-03-24T00:38:59.676465+0000","last_fullsized":"2024-03-24T00:38:59.676465+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:28:30.107102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802493+0000","last_change":"2024-03-24T00:38:52.789024+0000","last_active":"2024-03-24T00:38:58.802493+0000","last_peered":"2024-03-24T00:38:58.802493+0000","last_clean":"2024-03-24T00:38:58.802493+0000","last_became_active":"2024-03-24T00:38:52.788652+0000","last_became_peered":"2024-03-24T00:38:52.788652+0000","last_unstale":"2024-03-24T00:38:58.802493+0000","last_undegraded":"2024-03-24T00:38:58.802493+0000","last_fullsized":"2024-03-24T00:38:58.802493+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:21:00.861116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975533+0000","last_change":"2024-03-24T00:38:51.787599+0000","last_active":"2024-03-24T00:38:58.975533+0000","last_peered":"2024-03-24T00:38:58.975533+0000","last_clean":"2024-03-24T00:38:58.975533+0000","last_became_active":"2024-03-24T00:38:51.786958+0000","last_became_peered":"2024-03-24T00:38:51.786958+0000","last_unstale":"2024-03-24T00:38:58.975533+0000","last_undegraded":"2024-03-24T00:38:58.975533+0000","last_fullsized":"2024-03-24T00:38:58.975533+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:23:25.006444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975106+0000","last_change":"2024-03-24T00:38:56.822670+0000","last_active":"2024-03-24T00:38:58.975106+0000","last_peered":"2024-03-24T00:38:58.975106+0000","last_clean":"2024-03-24T00:38:58.975106+0000","last_became_active":"2024-03-24T00:38:56.822257+0000","last_became_peered":"2024-03-24T00:38:56.822257+0000","last_unstale":"2024-03-24T00:38:58.975106+0000","last_undegraded":"2024-03-24T00:38:58.975106+0000","last_fullsized":"2024-03-24T00:38:58.975106+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:17:07.239714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802301+0000","last_change":"2024-03-24T00:38:52.786986+0000","last_active":"2024-03-24T00:38:58.802301+0000","last_peered":"2024-03-24T00:38:58.802301+0000","last_clean":"2024-03-24T00:38:58.802301+0000","last_became_active":"2024-03-24T00:38:52.786700+0000","last_became_peered":"2024-03-24T00:38:52.786700+0000","last_unstale":"2024-03-24T00:38:58.802301+0000","last_undegraded":"2024-03-24T00:38:58.802301+0000","last_fullsized":"2024-03-24T00:38:58.802301+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:06.061403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802704+0000","last_change":"2024-03-24T00:38:51.789060+0000","last_active":"2024-03-24T00:38:58.802704+0000","last_peered":"2024-03-24T00:38:58.802704+0000","last_clean":"2024-03-24T00:38:58.802704+0000","last_became_active":"2024-03-24T00:38:51.788751+0000","last_became_peered":"2024-03-24T00:38:51.788751+0000","last_unstale":"2024-03-24T00:38:58.802704+0000","last_undegraded":"2024-03-24T00:38:58.802704+0000","last_fullsized":"2024-03-24T00:38:58.802704+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:09:30.121404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975477+0000","last_change":"2024-03-24T00:38:56.818475+0000","last_active":"2024-03-24T00:38:58.975477+0000","last_peered":"2024-03-24T00:38:58.975477+0000","last_clean":"2024-03-24T00:38:58.975477+0000","last_became_active":"2024-03-24T00:38:56.817655+0000","last_became_peered":"2024-03-24T00:38:56.817655+0000","last_unstale":"2024-03-24T00:38:58.975477+0000","last_undegraded":"2024-03-24T00:38:58.975477+0000","last_fullsized":"2024-03-24T00:38:58.975477+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:53:16.442548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.974995+0000","last_change":"2024-03-24T00:38:52.780020+0000","last_active":"2024-03-24T00:38:58.974995+0000","last_peered":"2024-03-24T00:38:58.974995+0000","last_clean":"2024-03-24T00:38:58.974995+0000","last_became_active":"2024-03-24T00:38:52.779730+0000","last_became_peered":"2024-03-24T00:38:52.779730+0000","last_unstale":"2024-03-24T00:38:58.974995+0000","last_undegraded":"2024-03-24T00:38:58.974995+0000","last_fullsized":"2024-03-24T00:38:58.974995+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:01:21.675152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.675893+0000","last_change":"2024-03-24T00:38:51.771263+0000","last_active":"2024-03-24T00:38:59.675893+0000","last_peered":"2024-03-24T00:38:59.675893+0000","last_clean":"2024-03-24T00:38:59.675893+0000","last_became_active":"2024-03-24T00:38:51.770920+0000","last_became_peered":"2024-03-24T00:38:51.770920+0000","last_unstale":"2024-03-24T00:38:59.675893+0000","last_undegraded":"2024-03-24T00:38:59.675893+0000","last_fullsized":"2024-03-24T00:38:59.675893+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:09:30.121404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975825+0000","last_change":"2024-03-24T00:38:56.822793+0000","last_active":"2024-03-24T00:38:58.975825+0000","last_peered":"2024-03-24T00:38:58.975825+0000","last_clean":"2024-03-24T00:38:58.975825+0000","last_became_active":"2024-03-24T00:38:56.822282+0000","last_became_peered":"2024-03-24T00:38:56.822282+0000","last_unstale":"2024-03-24T00:38:58.975825+0000","last_undegraded":"2024-03-24T00:38:58.975825+0000","last_fullsized":"2024-03-24T00:38:58.975825+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:26:44.414745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676155+0000","last_change":"2024-03-24T00:38:52.782329+0000","last_active":"2024-03-24T00:38:59.676155+0000","last_peered":"2024-03-24T00:38:59.676155+0000","last_clean":"2024-03-24T00:38:59.676155+0000","last_became_active":"2024-03-24T00:38:52.782006+0000","last_became_peered":"2024-03-24T00:38:52.782006+0000","last_unstale":"2024-03-24T00:38:59.676155+0000","last_undegraded":"2024-03-24T00:38:59.676155+0000","last_fullsized":"2024-03-24T00:38:59.676155+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:41:26.388475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676562+0000","last_change":"2024-03-24T00:38:51.787246+0000","last_active":"2024-03-24T00:38:59.676562+0000","last_peered":"2024-03-24T00:38:59.676562+0000","last_clean":"2024-03-24T00:38:59.676562+0000","last_became_active":"2024-03-24T00:38:51.787019+0000","last_became_peered":"2024-03-24T00:38:51.787019+0000","last_unstale":"2024-03-24T00:38:59.676562+0000","last_undegraded":"2024-03-24T00:38:59.676562+0000","last_fullsized":"2024-03-24T00:38:59.676562+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:23:25.006444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676812+0000","last_change":"2024-03-24T00:38:57.196855+0000","last_active":"2024-03-24T00:38:59.676812+0000","last_peered":"2024-03-24T00:38:59.676812+0000","last_clean":"2024-03-24T00:38:59.676812+0000","last_became_active":"2024-03-24T00:38:57.196127+0000","last_became_peered":"2024-03-24T00:38:57.196127+0000","last_unstale":"2024-03-24T00:38:59.676812+0000","last_undegraded":"2024-03-24T00:38:59.676812+0000","last_fullsized":"2024-03-24T00:38:59.676812+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:05:46.867916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800028+0000","last_change":"2024-03-24T00:38:52.792357+0000","last_active":"2024-03-24T00:38:58.800028+0000","last_peered":"2024-03-24T00:38:58.800028+0000","last_clean":"2024-03-24T00:38:58.800028+0000","last_became_active":"2024-03-24T00:38:52.792077+0000","last_became_peered":"2024-03-24T00:38:52.792077+0000","last_unstale":"2024-03-24T00:38:58.800028+0000","last_undegraded":"2024-03-24T00:38:58.800028+0000","last_fullsized":"2024-03-24T00:38:58.800028+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:53:36.522410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802065+0000","last_change":"2024-03-24T00:38:51.777903+0000","last_active":"2024-03-24T00:38:58.802065+0000","last_peered":"2024-03-24T00:38:58.802065+0000","last_clean":"2024-03-24T00:38:58.802065+0000","last_became_active":"2024-03-24T00:38:51.777178+0000","last_became_peered":"2024-03-24T00:38:51.777178+0000","last_unstale":"2024-03-24T00:38:58.802065+0000","last_undegraded":"2024-03-24T00:38:58.802065+0000","last_fullsized":"2024-03-24T00:38:58.802065+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:58:23.822944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":180,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676606+0000","last_change":"2024-03-24T00:38:40.953871+0000","last_active":"2024-03-24T00:38:59.676606+0000","last_peered":"2024-03-24T00:38:59.676606+0000","last_clean":"2024-03-24T00:38:59.676606+0000","last_became_active":"2024-03-24T00:38:40.952926+0000","last_became_peered":"2024-03-24T00:38:40.952926+0000","last_unstale":"2024-03-24T00:38:59.676606+0000","last_undegraded":"2024-03-24T00:38:59.676606+0000","last_fullsized":"2024-03-24T00:38:59.676606+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_clean_scrub_stamp":"2024-03-24T00:38:39.722194+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:05.397257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676410+0000","last_change":"2024-03-24T00:38:57.196309+0000","last_active":"2024-03-24T00:38:59.676410+0000","last_peered":"2024-03-24T00:38:59.676410+0000","last_clean":"2024-03-24T00:38:59.676410+0000","last_became_active":"2024-03-24T00:38:57.195975+0000","last_became_peered":"2024-03-24T00:38:57.195975+0000","last_unstale":"2024-03-24T00:38:59.676410+0000","last_undegraded":"2024-03-24T00:38:59.676410+0000","last_fullsized":"2024-03-24T00:38:59.676410+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:02:47.160631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676898+0000","last_change":"2024-03-24T00:38:52.790380+0000","last_active":"2024-03-24T00:38:59.676898+0000","last_peered":"2024-03-24T00:38:59.676898+0000","last_clean":"2024-03-24T00:38:59.676898+0000","last_became_active":"2024-03-24T00:38:52.788665+0000","last_became_peered":"2024-03-24T00:38:52.788665+0000","last_unstale":"2024-03-24T00:38:59.676898+0000","last_undegraded":"2024-03-24T00:38:59.676898+0000","last_fullsized":"2024-03-24T00:38:59.676898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:03:33.725080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676181+0000","last_change":"2024-03-24T00:38:51.781949+0000","last_active":"2024-03-24T00:38:59.676181+0000","last_peered":"2024-03-24T00:38:59.676181+0000","last_clean":"2024-03-24T00:38:59.676181+0000","last_became_active":"2024-03-24T00:38:51.781662+0000","last_became_peered":"2024-03-24T00:38:51.781662+0000","last_unstale":"2024-03-24T00:38:59.676181+0000","last_undegraded":"2024-03-24T00:38:59.676181+0000","last_fullsized":"2024-03-24T00:38:59.676181+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:59:15.934422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676501+0000","last_change":"2024-03-24T00:38:57.196217+0000","last_active":"2024-03-24T00:38:59.676501+0000","last_peered":"2024-03-24T00:38:59.676501+0000","last_clean":"2024-03-24T00:38:59.676501+0000","last_became_active":"2024-03-24T00:38:57.195860+0000","last_became_peered":"2024-03-24T00:38:57.195860+0000","last_unstale":"2024-03-24T00:38:59.676501+0000","last_undegraded":"2024-03-24T00:38:59.676501+0000","last_fullsized":"2024-03-24T00:38:59.676501+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:17:46.736676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802773+0000","last_change":"2024-03-24T00:38:52.780048+0000","last_active":"2024-03-24T00:38:58.802773+0000","last_peered":"2024-03-24T00:38:58.802773+0000","last_clean":"2024-03-24T00:38:58.802773+0000","last_became_active":"2024-03-24T00:38:52.779706+0000","last_became_peered":"2024-03-24T00:38:52.779706+0000","last_unstale":"2024-03-24T00:38:58.802773+0000","last_undegraded":"2024-03-24T00:38:58.802773+0000","last_fullsized":"2024-03-24T00:38:58.802773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:34:43.724190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975641+0000","last_change":"2024-03-24T00:38:51.789047+0000","last_active":"2024-03-24T00:38:58.975641+0000","last_peered":"2024-03-24T00:38:58.975641+0000","last_clean":"2024-03-24T00:38:58.975641+0000","last_became_active":"2024-03-24T00:38:51.788769+0000","last_became_peered":"2024-03-24T00:38:51.788769+0000","last_unstale":"2024-03-24T00:38:58.975641+0000","last_undegraded":"2024-03-24T00:38:58.975641+0000","last_fullsized":"2024-03-24T00:38:58.975641+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:09:30.121404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676730+0000","last_change":"2024-03-24T00:38:56.822795+0000","last_active":"2024-03-24T00:38:59.676730+0000","last_peered":"2024-03-24T00:38:59.676730+0000","last_clean":"2024-03-24T00:38:59.676730+0000","last_became_active":"2024-03-24T00:38:56.822410+0000","last_became_peered":"2024-03-24T00:38:56.822410+0000","last_unstale":"2024-03-24T00:38:59.676730+0000","last_undegraded":"2024-03-24T00:38:59.676730+0000","last_fullsized":"2024-03-24T00:38:59.676730+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:26:44.414745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.677245+0000","last_change":"2024-03-24T00:38:52.788023+0000","last_active":"2024-03-24T00:38:59.677245+0000","last_peered":"2024-03-24T00:38:59.677245+0000","last_clean":"2024-03-24T00:38:59.677245+0000","last_became_active":"2024-03-24T00:38:52.787528+0000","last_became_peered":"2024-03-24T00:38:52.787528+0000","last_unstale":"2024-03-24T00:38:59.677245+0000","last_undegraded":"2024-03-24T00:38:59.677245+0000","last_fullsized":"2024-03-24T00:38:59.677245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:57:30.131729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676813+0000","last_change":"2024-03-24T00:38:51.771300+0000","last_active":"2024-03-24T00:38:59.676813+0000","last_peered":"2024-03-24T00:38:59.676813+0000","last_clean":"2024-03-24T00:38:59.676813+0000","last_became_active":"2024-03-24T00:38:51.770924+0000","last_became_peered":"2024-03-24T00:38:51.770924+0000","last_unstale":"2024-03-24T00:38:59.676813+0000","last_undegraded":"2024-03-24T00:38:59.676813+0000","last_fullsized":"2024-03-24T00:38:59.676813+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:58:23.822944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801216+0000","last_change":"2024-03-24T00:38:57.196385+0000","last_active":"2024-03-24T00:38:58.801216+0000","last_peered":"2024-03-24T00:38:58.801216+0000","last_clean":"2024-03-24T00:38:58.801216+0000","last_became_active":"2024-03-24T00:38:57.196000+0000","last_became_peered":"2024-03-24T00:38:57.196000+0000","last_unstale":"2024-03-24T00:38:58.801216+0000","last_undegraded":"2024-03-24T00:38:58.801216+0000","last_fullsized":"2024-03-24T00:38:58.801216+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:26:44.414745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.677147+0000","last_change":"2024-03-24T00:38:52.791416+0000","last_active":"2024-03-24T00:38:59.677147+0000","last_peered":"2024-03-24T00:38:59.677147+0000","last_clean":"2024-03-24T00:38:59.677147+0000","last_became_active":"2024-03-24T00:38:52.790334+0000","last_became_peered":"2024-03-24T00:38:52.790334+0000","last_unstale":"2024-03-24T00:38:59.677147+0000","last_undegraded":"2024-03-24T00:38:59.677147+0000","last_fullsized":"2024-03-24T00:38:59.677147+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:34:43.724190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801647+0000","last_change":"2024-03-24T00:38:51.780196+0000","last_active":"2024-03-24T00:38:58.801647+0000","last_peered":"2024-03-24T00:38:58.801647+0000","last_clean":"2024-03-24T00:38:58.801647+0000","last_became_active":"2024-03-24T00:38:51.779921+0000","last_became_peered":"2024-03-24T00:38:51.779921+0000","last_unstale":"2024-03-24T00:38:58.801647+0000","last_undegraded":"2024-03-24T00:38:58.801647+0000","last_fullsized":"2024-03-24T00:38:58.801647+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:52:59.726516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801216+0000","last_change":"2024-03-24T00:38:56.821775+0000","last_active":"2024-03-24T00:38:58.801216+0000","last_peered":"2024-03-24T00:38:58.801216+0000","last_clean":"2024-03-24T00:38:58.801216+0000","last_became_active":"2024-03-24T00:38:56.821188+0000","last_became_peered":"2024-03-24T00:38:56.821188+0000","last_unstale":"2024-03-24T00:38:58.801216+0000","last_undegraded":"2024-03-24T00:38:58.801216+0000","last_fullsized":"2024-03-24T00:38:58.801216+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:17:46.736676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676771+0000","last_change":"2024-03-24T00:38:52.791631+0000","last_active":"2024-03-24T00:38:59.676771+0000","last_peered":"2024-03-24T00:38:59.676771+0000","last_clean":"2024-03-24T00:38:59.676771+0000","last_became_active":"2024-03-24T00:38:52.790109+0000","last_became_peered":"2024-03-24T00:38:52.790109+0000","last_unstale":"2024-03-24T00:38:59.676771+0000","last_undegraded":"2024-03-24T00:38:59.676771+0000","last_fullsized":"2024-03-24T00:38:59.676771+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:53:36.522410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":32,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.675790+0000","last_change":"2024-03-24T00:38:51.789009+0000","last_active":"2024-03-24T00:38:59.675790+0000","last_peered":"2024-03-24T00:38:59.675790+0000","last_clean":"2024-03-24T00:38:59.675790+0000","last_became_active":"2024-03-24T00:38:51.788721+0000","last_became_peered":"2024-03-24T00:38:51.788721+0000","last_unstale":"2024-03-24T00:38:59.675790+0000","last_undegraded":"2024-03-24T00:38:59.675790+0000","last_fullsized":"2024-03-24T00:38:59.675790+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:43:41.080723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676661+0000","last_change":"2024-03-24T00:38:56.815041+0000","last_active":"2024-03-24T00:38:59.676661+0000","last_peered":"2024-03-24T00:38:59.676661+0000","last_clean":"2024-03-24T00:38:59.676661+0000","last_became_active":"2024-03-24T00:38:56.814651+0000","last_became_peered":"2024-03-24T00:38:56.814651+0000","last_unstale":"2024-03-24T00:38:59.676661+0000","last_undegraded":"2024-03-24T00:38:59.676661+0000","last_fullsized":"2024-03-24T00:38:59.676661+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:01:41.498040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676169+0000","last_change":"2024-03-24T00:38:52.779914+0000","last_active":"2024-03-24T00:38:59.676169+0000","last_peered":"2024-03-24T00:38:59.676169+0000","last_clean":"2024-03-24T00:38:59.676169+0000","last_became_active":"2024-03-24T00:38:52.779330+0000","last_became_peered":"2024-03-24T00:38:52.779330+0000","last_unstale":"2024-03-24T00:38:59.676169+0000","last_undegraded":"2024-03-24T00:38:59.676169+0000","last_fullsized":"2024-03-24T00:38:59.676169+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:42:38.046862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676645+0000","last_change":"2024-03-24T00:38:51.781129+0000","last_active":"2024-03-24T00:38:59.676645+0000","last_peered":"2024-03-24T00:38:59.676645+0000","last_clean":"2024-03-24T00:38:59.676645+0000","last_became_active":"2024-03-24T00:38:51.780828+0000","last_became_peered":"2024-03-24T00:38:51.780828+0000","last_unstale":"2024-03-24T00:38:59.676645+0000","last_undegraded":"2024-03-24T00:38:59.676645+0000","last_fullsized":"2024-03-24T00:38:59.676645+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:59:15.934422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800545+0000","last_change":"2024-03-24T00:38:57.197537+0000","last_active":"2024-03-24T00:38:58.800545+0000","last_peered":"2024-03-24T00:38:58.800545+0000","last_clean":"2024-03-24T00:38:58.800545+0000","last_became_active":"2024-03-24T00:38:57.197082+0000","last_became_peered":"2024-03-24T00:38:57.197082+0000","last_unstale":"2024-03-24T00:38:58.800545+0000","last_undegraded":"2024-03-24T00:38:58.800545+0000","last_fullsized":"2024-03-24T00:38:58.800545+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:34:30.798923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676725+0000","last_change":"2024-03-24T00:38:52.789686+0000","last_active":"2024-03-24T00:38:59.676725+0000","last_peered":"2024-03-24T00:38:59.676725+0000","last_clean":"2024-03-24T00:38:59.676725+0000","last_became_active":"2024-03-24T00:38:52.789416+0000","last_became_peered":"2024-03-24T00:38:52.789416+0000","last_unstale":"2024-03-24T00:38:59.676725+0000","last_undegraded":"2024-03-24T00:38:59.676725+0000","last_fullsized":"2024-03-24T00:38:59.676725+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:04:24.260803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802696+0000","last_change":"2024-03-24T00:38:51.777070+0000","last_active":"2024-03-24T00:38:58.802696+0000","last_peered":"2024-03-24T00:38:58.802696+0000","last_clean":"2024-03-24T00:38:58.802696+0000","last_became_active":"2024-03-24T00:38:51.776786+0000","last_became_peered":"2024-03-24T00:38:51.776786+0000","last_unstale":"2024-03-24T00:38:58.802696+0000","last_undegraded":"2024-03-24T00:38:58.802696+0000","last_fullsized":"2024-03-24T00:38:58.802696+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:05.055760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800552+0000","last_change":"2024-03-24T00:38:57.196551+0000","last_active":"2024-03-24T00:38:58.800552+0000","last_peered":"2024-03-24T00:38:58.800552+0000","last_clean":"2024-03-24T00:38:58.800552+0000","last_became_active":"2024-03-24T00:38:57.196138+0000","last_became_peered":"2024-03-24T00:38:57.196138+0000","last_unstale":"2024-03-24T00:38:58.800552+0000","last_undegraded":"2024-03-24T00:38:58.800552+0000","last_fullsized":"2024-03-24T00:38:58.800552+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:47:53.855251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676423+0000","last_change":"2024-03-24T00:38:52.782561+0000","last_active":"2024-03-24T00:38:59.676423+0000","last_peered":"2024-03-24T00:38:59.676423+0000","last_clean":"2024-03-24T00:38:59.676423+0000","last_became_active":"2024-03-24T00:38:52.781961+0000","last_became_peered":"2024-03-24T00:38:52.781961+0000","last_unstale":"2024-03-24T00:38:59.676423+0000","last_undegraded":"2024-03-24T00:38:59.676423+0000","last_fullsized":"2024-03-24T00:38:59.676423+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:42:38.046862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676934+0000","last_change":"2024-03-24T00:38:51.778517+0000","last_active":"2024-03-24T00:38:59.676934+0000","last_peered":"2024-03-24T00:38:59.676934+0000","last_clean":"2024-03-24T00:38:59.676934+0000","last_became_active":"2024-03-24T00:38:51.778218+0000","last_became_peered":"2024-03-24T00:38:51.778218+0000","last_unstale":"2024-03-24T00:38:59.676934+0000","last_undegraded":"2024-03-24T00:38:59.676934+0000","last_fullsized":"2024-03-24T00:38:59.676934+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:01:37.155677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802393+0000","last_change":"2024-03-24T00:38:57.196343+0000","last_active":"2024-03-24T00:38:58.802393+0000","last_peered":"2024-03-24T00:38:58.802393+0000","last_clean":"2024-03-24T00:38:58.802393+0000","last_became_active":"2024-03-24T00:38:57.195953+0000","last_became_peered":"2024-03-24T00:38:57.195953+0000","last_unstale":"2024-03-24T00:38:58.802393+0000","last_undegraded":"2024-03-24T00:38:58.802393+0000","last_fullsized":"2024-03-24T00:38:58.802393+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:05:46.867916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676892+0000","last_change":"2024-03-24T00:38:52.792925+0000","last_active":"2024-03-24T00:38:59.676892+0000","last_peered":"2024-03-24T00:38:59.676892+0000","last_clean":"2024-03-24T00:38:59.676892+0000","last_became_active":"2024-03-24T00:38:52.792661+0000","last_became_peered":"2024-03-24T00:38:52.792661+0000","last_unstale":"2024-03-24T00:38:59.676892+0000","last_undegraded":"2024-03-24T00:38:59.676892+0000","last_fullsized":"2024-03-24T00:38:59.676892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:48:02.197473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":33,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:39:02.090137+0000","last_change":"2024-03-24T00:38:51.775463+0000","last_active":"2024-03-24T00:39:02.090137+0000","last_peered":"2024-03-24T00:39:02.090137+0000","last_clean":"2024-03-24T00:39:02.090137+0000","last_became_active":"2024-03-24T00:38:51.775167+0000","last_became_peered":"2024-03-24T00:38:51.775167+0000","last_unstale":"2024-03-24T00:39:02.090137+0000","last_undegraded":"2024-03-24T00:39:02.090137+0000","last_fullsized":"2024-03-24T00:39:02.090137+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:23:25.006444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975203+0000","last_change":"2024-03-24T00:38:56.819965+0000","last_active":"2024-03-24T00:38:58.975203+0000","last_peered":"2024-03-24T00:38:58.975203+0000","last_clean":"2024-03-24T00:38:58.975203+0000","last_became_active":"2024-03-24T00:38:56.819580+0000","last_became_peered":"2024-03-24T00:38:56.819580+0000","last_unstale":"2024-03-24T00:38:58.975203+0000","last_undegraded":"2024-03-24T00:38:58.975203+0000","last_fullsized":"2024-03-24T00:38:58.975203+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:28:30.107102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802246+0000","last_change":"2024-03-24T00:38:52.790290+0000","last_active":"2024-03-24T00:38:58.802246+0000","last_peered":"2024-03-24T00:38:58.802246+0000","last_clean":"2024-03-24T00:38:58.802246+0000","last_became_active":"2024-03-24T00:38:52.790010+0000","last_became_peered":"2024-03-24T00:38:52.790010+0000","last_unstale":"2024-03-24T00:38:58.802246+0000","last_undegraded":"2024-03-24T00:38:58.802246+0000","last_fullsized":"2024-03-24T00:38:58.802246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:36:19.927252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975485+0000","last_change":"2024-03-24T00:38:51.788684+0000","last_active":"2024-03-24T00:38:58.975485+0000","last_peered":"2024-03-24T00:38:58.975485+0000","last_clean":"2024-03-24T00:38:58.975485+0000","last_became_active":"2024-03-24T00:38:51.788414+0000","last_became_peered":"2024-03-24T00:38:51.788414+0000","last_unstale":"2024-03-24T00:38:58.975485+0000","last_undegraded":"2024-03-24T00:38:58.975485+0000","last_fullsized":"2024-03-24T00:38:58.975485+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:59:15.934422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802225+0000","last_change":"2024-03-24T00:38:56.822362+0000","last_active":"2024-03-24T00:38:58.802225+0000","last_peered":"2024-03-24T00:38:58.802225+0000","last_clean":"2024-03-24T00:38:58.802225+0000","last_became_active":"2024-03-24T00:38:56.821516+0000","last_became_peered":"2024-03-24T00:38:56.821516+0000","last_unstale":"2024-03-24T00:38:58.802225+0000","last_undegraded":"2024-03-24T00:38:58.802225+0000","last_fullsized":"2024-03-24T00:38:58.802225+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:01:41.498040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676720+0000","last_change":"2024-03-24T00:38:52.778191+0000","last_active":"2024-03-24T00:38:59.676720+0000","last_peered":"2024-03-24T00:38:59.676720+0000","last_clean":"2024-03-24T00:38:59.676720+0000","last_became_active":"2024-03-24T00:38:52.777911+0000","last_became_peered":"2024-03-24T00:38:52.777911+0000","last_unstale":"2024-03-24T00:38:59.676720+0000","last_undegraded":"2024-03-24T00:38:59.676720+0000","last_fullsized":"2024-03-24T00:38:59.676720+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:41:59.361242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801321+0000","last_change":"2024-03-24T00:38:51.778447+0000","last_active":"2024-03-24T00:38:58.801321+0000","last_peered":"2024-03-24T00:38:58.801321+0000","last_clean":"2024-03-24T00:38:58.801321+0000","last_became_active":"2024-03-24T00:38:51.778151+0000","last_became_peered":"2024-03-24T00:38:51.778151+0000","last_unstale":"2024-03-24T00:38:58.801321+0000","last_undegraded":"2024-03-24T00:38:58.801321+0000","last_fullsized":"2024-03-24T00:38:58.801321+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:23:25.006444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800051+0000","last_change":"2024-03-24T00:38:56.818959+0000","last_active":"2024-03-24T00:38:58.800051+0000","last_peered":"2024-03-24T00:38:58.800051+0000","last_clean":"2024-03-24T00:38:58.800051+0000","last_became_active":"2024-03-24T00:38:56.818582+0000","last_became_peered":"2024-03-24T00:38:56.818582+0000","last_unstale":"2024-03-24T00:38:58.800051+0000","last_undegraded":"2024-03-24T00:38:58.800051+0000","last_fullsized":"2024-03-24T00:38:58.800051+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:19:21.320451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801240+0000","last_change":"2024-03-24T00:38:52.787525+0000","last_active":"2024-03-24T00:38:58.801240+0000","last_peered":"2024-03-24T00:38:58.801240+0000","last_clean":"2024-03-24T00:38:58.801240+0000","last_became_active":"2024-03-24T00:38:52.787167+0000","last_became_peered":"2024-03-24T00:38:52.787167+0000","last_unstale":"2024-03-24T00:38:58.801240+0000","last_undegraded":"2024-03-24T00:38:58.801240+0000","last_fullsized":"2024-03-24T00:38:58.801240+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:53:36.522410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676940+0000","last_change":"2024-03-24T00:38:51.780424+0000","last_active":"2024-03-24T00:38:59.676940+0000","last_peered":"2024-03-24T00:38:59.676940+0000","last_clean":"2024-03-24T00:38:59.676940+0000","last_became_active":"2024-03-24T00:38:51.780120+0000","last_became_peered":"2024-03-24T00:38:51.780120+0000","last_unstale":"2024-03-24T00:38:59.676940+0000","last_undegraded":"2024-03-24T00:38:59.676940+0000","last_fullsized":"2024-03-24T00:38:59.676940+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:58:23.822944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800129+0000","last_change":"2024-03-24T00:38:57.197505+0000","last_active":"2024-03-24T00:38:58.800129+0000","last_peered":"2024-03-24T00:38:58.800129+0000","last_clean":"2024-03-24T00:38:58.800129+0000","last_became_active":"2024-03-24T00:38:57.197104+0000","last_became_peered":"2024-03-24T00:38:57.197104+0000","last_unstale":"2024-03-24T00:38:58.800129+0000","last_undegraded":"2024-03-24T00:38:58.800129+0000","last_fullsized":"2024-03-24T00:38:58.800129+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:44:07.562150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676497+0000","last_change":"2024-03-24T00:38:57.197065+0000","last_active":"2024-03-24T00:38:59.676497+0000","last_peered":"2024-03-24T00:38:59.676497+0000","last_clean":"2024-03-24T00:38:59.676497+0000","last_became_active":"2024-03-24T00:38:57.196687+0000","last_became_peered":"2024-03-24T00:38:57.196687+0000","last_unstale":"2024-03-24T00:38:59.676497+0000","last_undegraded":"2024-03-24T00:38:59.676497+0000","last_fullsized":"2024-03-24T00:38:59.676497+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:53:16.442548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.974900+0000","last_change":"2024-03-24T00:38:52.790233+0000","last_active":"2024-03-24T00:38:58.974900+0000","last_peered":"2024-03-24T00:38:58.974900+0000","last_clean":"2024-03-24T00:38:58.974900+0000","last_became_active":"2024-03-24T00:38:52.789951+0000","last_became_peered":"2024-03-24T00:38:52.789951+0000","last_unstale":"2024-03-24T00:38:58.974900+0000","last_undegraded":"2024-03-24T00:38:58.974900+0000","last_fullsized":"2024-03-24T00:38:58.974900+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:42:38.046862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801116+0000","last_change":"2024-03-24T00:38:51.775723+0000","last_active":"2024-03-24T00:38:58.801116+0000","last_peered":"2024-03-24T00:38:58.801116+0000","last_clean":"2024-03-24T00:38:58.801116+0000","last_became_active":"2024-03-24T00:38:51.773333+0000","last_became_peered":"2024-03-24T00:38:51.773333+0000","last_unstale":"2024-03-24T00:38:58.801116+0000","last_undegraded":"2024-03-24T00:38:58.801116+0000","last_fullsized":"2024-03-24T00:38:58.801116+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:59:15.934422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676945+0000","last_change":"2024-03-24T00:38:56.819973+0000","last_active":"2024-03-24T00:38:59.676945+0000","last_peered":"2024-03-24T00:38:59.676945+0000","last_clean":"2024-03-24T00:38:59.676945+0000","last_became_active":"2024-03-24T00:38:56.819659+0000","last_became_peered":"2024-03-24T00:38:56.819659+0000","last_unstale":"2024-03-24T00:38:59.676945+0000","last_undegraded":"2024-03-24T00:38:59.676945+0000","last_fullsized":"2024-03-24T00:38:59.676945+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:57:34.164907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.799976+0000","last_change":"2024-03-24T00:38:52.785779+0000","last_active":"2024-03-24T00:38:58.799976+0000","last_peered":"2024-03-24T00:38:58.799976+0000","last_clean":"2024-03-24T00:38:58.799976+0000","last_became_active":"2024-03-24T00:38:52.785184+0000","last_became_peered":"2024-03-24T00:38:52.785184+0000","last_unstale":"2024-03-24T00:38:58.799976+0000","last_undegraded":"2024-03-24T00:38:58.799976+0000","last_fullsized":"2024-03-24T00:38:58.799976+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:39:26.969331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800170+0000","last_change":"2024-03-24T00:38:51.784065+0000","last_active":"2024-03-24T00:38:58.800170+0000","last_peered":"2024-03-24T00:38:58.800170+0000","last_clean":"2024-03-24T00:38:58.800170+0000","last_became_active":"2024-03-24T00:38:51.783789+0000","last_became_peered":"2024-03-24T00:38:51.783789+0000","last_unstale":"2024-03-24T00:38:58.800170+0000","last_undegraded":"2024-03-24T00:38:58.800170+0000","last_fullsized":"2024-03-24T00:38:58.800170+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:23:25.006444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676753+0000","last_change":"2024-03-24T00:38:51.777458+0000","last_active":"2024-03-24T00:38:59.676753+0000","last_peered":"2024-03-24T00:38:59.676753+0000","last_clean":"2024-03-24T00:38:59.676753+0000","last_became_active":"2024-03-24T00:38:51.777189+0000","last_became_peered":"2024-03-24T00:38:51.777189+0000","last_unstale":"2024-03-24T00:38:59.676753+0000","last_undegraded":"2024-03-24T00:38:59.676753+0000","last_fullsized":"2024-03-24T00:38:59.676753+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:05.055760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676860+0000","last_change":"2024-03-24T00:38:52.786644+0000","last_active":"2024-03-24T00:38:59.676860+0000","last_peered":"2024-03-24T00:38:59.676860+0000","last_clean":"2024-03-24T00:38:59.676860+0000","last_became_active":"2024-03-24T00:38:52.786348+0000","last_became_peered":"2024-03-24T00:38:52.786348+0000","last_unstale":"2024-03-24T00:38:59.676860+0000","last_undegraded":"2024-03-24T00:38:59.676860+0000","last_fullsized":"2024-03-24T00:38:59.676860+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:36:54.790308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676080+0000","last_change":"2024-03-24T00:38:56.816034+0000","last_active":"2024-03-24T00:38:59.676080+0000","last_peered":"2024-03-24T00:38:59.676080+0000","last_clean":"2024-03-24T00:38:59.676080+0000","last_became_active":"2024-03-24T00:38:56.815608+0000","last_became_peered":"2024-03-24T00:38:56.815608+0000","last_unstale":"2024-03-24T00:38:59.676080+0000","last_undegraded":"2024-03-24T00:38:59.676080+0000","last_fullsized":"2024-03-24T00:38:59.676080+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:27:27.336047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800713+0000","last_change":"2024-03-24T00:38:51.782948+0000","last_active":"2024-03-24T00:38:58.800713+0000","last_peered":"2024-03-24T00:38:58.800713+0000","last_clean":"2024-03-24T00:38:58.800713+0000","last_became_active":"2024-03-24T00:38:51.780063+0000","last_became_peered":"2024-03-24T00:38:51.780063+0000","last_unstale":"2024-03-24T00:38:58.800713+0000","last_undegraded":"2024-03-24T00:38:58.800713+0000","last_fullsized":"2024-03-24T00:38:58.800713+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:50.748117+0000","last_clean_scrub_stamp":"2024-03-24T00:38:50.748117+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:05.055760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801390+0000","last_change":"2024-03-24T00:38:52.778373+0000","last_active":"2024-03-24T00:38:58.801390+0000","last_peered":"2024-03-24T00:38:58.801390+0000","last_clean":"2024-03-24T00:38:58.801390+0000","last_became_active":"2024-03-24T00:38:52.778019+0000","last_became_peered":"2024-03-24T00:38:52.778019+0000","last_unstale":"2024-03-24T00:38:58.801390+0000","last_undegraded":"2024-03-24T00:38:58.801390+0000","last_fullsized":"2024-03-24T00:38:58.801390+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:09:31.127048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800819+0000","last_change":"2024-03-24T00:38:56.818404+0000","last_active":"2024-03-24T00:38:58.800819+0000","last_peered":"2024-03-24T00:38:58.800819+0000","last_clean":"2024-03-24T00:38:58.800819+0000","last_became_active":"2024-03-24T00:38:56.817801+0000","last_became_peered":"2024-03-24T00:38:56.817801+0000","last_unstale":"2024-03-24T00:38:58.800819+0000","last_undegraded":"2024-03-24T00:38:58.800819+0000","last_fullsized":"2024-03-24T00:38:58.800819+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:55.786938+0000","last_clean_scrub_stamp":"2024-03-24T00:38:55.786938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:15:28.193092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676081+0000","last_change":"2024-03-24T00:38:52.780120+0000","last_active":"2024-03-24T00:38:59.676081+0000","last_peered":"2024-03-24T00:38:59.676081+0000","last_clean":"2024-03-24T00:38:59.676081+0000","last_became_active":"2024-03-24T00:38:52.779624+0000","last_became_peered":"2024-03-24T00:38:52.779624+0000","last_unstale":"2024-03-24T00:38:59.676081+0000","last_undegraded":"2024-03-24T00:38:59.676081+0000","last_fullsized":"2024-03-24T00:38:59.676081+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:54:01.232233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676019+0000","last_change":"2024-03-24T00:38:52.783615+0000","last_active":"2024-03-24T00:38:59.676019+0000","last_peered":"2024-03-24T00:38:59.676019+0000","last_clean":"2024-03-24T00:38:59.676019+0000","last_became_active":"2024-03-24T00:38:52.783339+0000","last_became_peered":"2024-03-24T00:38:52.783339+0000","last_unstale":"2024-03-24T00:38:59.676019+0000","last_undegraded":"2024-03-24T00:38:59.676019+0000","last_fullsized":"2024-03-24T00:38:59.676019+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:51:22.179128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802656+0000","last_change":"2024-03-24T00:38:52.793626+0000","last_active":"2024-03-24T00:38:58.802656+0000","last_peered":"2024-03-24T00:38:58.802656+0000","last_clean":"2024-03-24T00:38:58.802656+0000","last_became_active":"2024-03-24T00:38:52.793322+0000","last_became_peered":"2024-03-24T00:38:52.793322+0000","last_unstale":"2024-03-24T00:38:58.802656+0000","last_undegraded":"2024-03-24T00:38:58.802656+0000","last_fullsized":"2024-03-24T00:38:58.802656+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:57:30.131729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676831+0000","last_change":"2024-03-24T00:38:52.791203+0000","last_active":"2024-03-24T00:38:59.676831+0000","last_peered":"2024-03-24T00:38:59.676831+0000","last_clean":"2024-03-24T00:38:59.676831+0000","last_became_active":"2024-03-24T00:38:52.790914+0000","last_became_peered":"2024-03-24T00:38:52.790914+0000","last_unstale":"2024-03-24T00:38:59.676831+0000","last_undegraded":"2024-03-24T00:38:59.676831+0000","last_fullsized":"2024-03-24T00:38:59.676831+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:21:12.524662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800715+0000","last_change":"2024-03-24T00:38:52.788614+0000","last_active":"2024-03-24T00:38:58.800715+0000","last_peered":"2024-03-24T00:38:58.800715+0000","last_clean":"2024-03-24T00:38:58.800715+0000","last_became_active":"2024-03-24T00:38:52.788155+0000","last_became_peered":"2024-03-24T00:38:52.788155+0000","last_unstale":"2024-03-24T00:38:58.800715+0000","last_undegraded":"2024-03-24T00:38:58.800715+0000","last_fullsized":"2024-03-24T00:38:58.800715+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:54:01.232233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975795+0000","last_change":"2024-03-24T00:38:52.790018+0000","last_active":"2024-03-24T00:38:58.975795+0000","last_peered":"2024-03-24T00:38:58.975795+0000","last_clean":"2024-03-24T00:38:58.975795+0000","last_became_active":"2024-03-24T00:38:52.789744+0000","last_became_peered":"2024-03-24T00:38:52.789744+0000","last_unstale":"2024-03-24T00:38:58.975795+0000","last_undegraded":"2024-03-24T00:38:58.975795+0000","last_fullsized":"2024-03-24T00:38:58.975795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:46:05.527509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676904+0000","last_change":"2024-03-24T00:38:52.791896+0000","last_active":"2024-03-24T00:38:59.676904+0000","last_peered":"2024-03-24T00:38:59.676904+0000","last_clean":"2024-03-24T00:38:59.676904+0000","last_became_active":"2024-03-24T00:38:52.789711+0000","last_became_peered":"2024-03-24T00:38:52.789711+0000","last_unstale":"2024-03-24T00:38:59.676904+0000","last_undegraded":"2024-03-24T00:38:59.676904+0000","last_fullsized":"2024-03-24T00:38:59.676904+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:21:12.524662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800598+0000","last_change":"2024-03-24T00:38:52.780901+0000","last_active":"2024-03-24T00:38:58.800598+0000","last_peered":"2024-03-24T00:38:58.800598+0000","last_clean":"2024-03-24T00:38:58.800598+0000","last_became_active":"2024-03-24T00:38:52.780625+0000","last_became_peered":"2024-03-24T00:38:52.780625+0000","last_unstale":"2024-03-24T00:38:58.800598+0000","last_undegraded":"2024-03-24T00:38:58.800598+0000","last_fullsized":"2024-03-24T00:38:58.800598+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:42:38.046862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801027+0000","last_change":"2024-03-24T00:38:52.784339+0000","last_active":"2024-03-24T00:38:58.801027+0000","last_peered":"2024-03-24T00:38:58.801027+0000","last_clean":"2024-03-24T00:38:58.801027+0000","last_became_active":"2024-03-24T00:38:52.783135+0000","last_became_peered":"2024-03-24T00:38:52.783135+0000","last_unstale":"2024-03-24T00:38:58.801027+0000","last_undegraded":"2024-03-24T00:38:58.801027+0000","last_fullsized":"2024-03-24T00:38:58.801027+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:21:12.524662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800355+0000","last_change":"2024-03-24T00:38:52.788394+0000","last_active":"2024-03-24T00:38:58.800355+0000","last_peered":"2024-03-24T00:38:58.800355+0000","last_clean":"2024-03-24T00:38:58.800355+0000","last_became_active":"2024-03-24T00:38:52.788105+0000","last_became_peered":"2024-03-24T00:38:52.788105+0000","last_unstale":"2024-03-24T00:38:58.800355+0000","last_undegraded":"2024-03-24T00:38:58.800355+0000","last_fullsized":"2024-03-24T00:38:58.800355+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:04:24.260803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802590+0000","last_change":"2024-03-24T00:38:52.787942+0000","last_active":"2024-03-24T00:38:58.802590+0000","last_peered":"2024-03-24T00:38:58.802590+0000","last_clean":"2024-03-24T00:38:58.802590+0000","last_became_active":"2024-03-24T00:38:52.787691+0000","last_became_peered":"2024-03-24T00:38:52.787691+0000","last_unstale":"2024-03-24T00:38:58.802590+0000","last_undegraded":"2024-03-24T00:38:58.802590+0000","last_fullsized":"2024-03-24T00:38:58.802590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:17:42.703499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676562+0000","last_change":"2024-03-24T00:38:52.780488+0000","last_active":"2024-03-24T00:38:59.676562+0000","last_peered":"2024-03-24T00:38:59.676562+0000","last_clean":"2024-03-24T00:38:59.676562+0000","last_became_active":"2024-03-24T00:38:52.780227+0000","last_became_peered":"2024-03-24T00:38:52.780227+0000","last_unstale":"2024-03-24T00:38:59.676562+0000","last_undegraded":"2024-03-24T00:38:59.676562+0000","last_fullsized":"2024-03-24T00:38:59.676562+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:39:26.969331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676482+0000","last_change":"2024-03-24T00:38:52.785757+0000","last_active":"2024-03-24T00:38:59.676482+0000","last_peered":"2024-03-24T00:38:59.676482+0000","last_clean":"2024-03-24T00:38:59.676482+0000","last_became_active":"2024-03-24T00:38:52.785476+0000","last_became_peered":"2024-03-24T00:38:52.785476+0000","last_unstale":"2024-03-24T00:38:59.676482+0000","last_undegraded":"2024-03-24T00:38:59.676482+0000","last_fullsized":"2024-03-24T00:38:59.676482+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:39:26.969331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.677056+0000","last_change":"2024-03-24T00:38:52.786263+0000","last_active":"2024-03-24T00:38:59.677056+0000","last_peered":"2024-03-24T00:38:59.677056+0000","last_clean":"2024-03-24T00:38:59.677056+0000","last_became_active":"2024-03-24T00:38:52.785957+0000","last_became_peered":"2024-03-24T00:38:52.785957+0000","last_unstale":"2024-03-24T00:38:59.677056+0000","last_undegraded":"2024-03-24T00:38:59.677056+0000","last_fullsized":"2024-03-24T00:38:59.677056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:02:43.127453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676749+0000","last_change":"2024-03-24T00:38:52.786178+0000","last_active":"2024-03-24T00:38:59.676749+0000","last_peered":"2024-03-24T00:38:59.676749+0000","last_clean":"2024-03-24T00:38:59.676749+0000","last_became_active":"2024-03-24T00:38:52.785894+0000","last_became_peered":"2024-03-24T00:38:52.785894+0000","last_unstale":"2024-03-24T00:38:59.676749+0000","last_undegraded":"2024-03-24T00:38:59.676749+0000","last_fullsized":"2024-03-24T00:38:59.676749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:36:19.927252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.801779+0000","last_change":"2024-03-24T00:38:52.787876+0000","last_active":"2024-03-24T00:38:58.801779+0000","last_peered":"2024-03-24T00:38:58.801779+0000","last_clean":"2024-03-24T00:38:58.801779+0000","last_became_active":"2024-03-24T00:38:52.787243+0000","last_became_peered":"2024-03-24T00:38:52.787243+0000","last_unstale":"2024-03-24T00:38:58.801779+0000","last_undegraded":"2024-03-24T00:38:58.801779+0000","last_fullsized":"2024-03-24T00:38:58.801779+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:36:54.790308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800934+0000","last_change":"2024-03-24T00:38:52.776125+0000","last_active":"2024-03-24T00:38:58.800934+0000","last_peered":"2024-03-24T00:38:58.800934+0000","last_clean":"2024-03-24T00:38:58.800934+0000","last_became_active":"2024-03-24T00:38:52.775255+0000","last_became_peered":"2024-03-24T00:38:52.775255+0000","last_unstale":"2024-03-24T00:38:58.800934+0000","last_undegraded":"2024-03-24T00:38:58.800934+0000","last_fullsized":"2024-03-24T00:38:58.800934+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:46:05.527509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.974980+0000","last_change":"2024-03-24T00:38:52.781546+0000","last_active":"2024-03-24T00:38:58.974980+0000","last_peered":"2024-03-24T00:38:58.974980+0000","last_clean":"2024-03-24T00:38:58.974980+0000","last_became_active":"2024-03-24T00:38:52.781220+0000","last_became_peered":"2024-03-24T00:38:52.781220+0000","last_unstale":"2024-03-24T00:38:58.974980+0000","last_undegraded":"2024-03-24T00:38:58.974980+0000","last_fullsized":"2024-03-24T00:38:58.974980+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:41:59.361242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676394+0000","last_change":"2024-03-24T00:38:52.773210+0000","last_active":"2024-03-24T00:38:59.676394+0000","last_peered":"2024-03-24T00:38:59.676394+0000","last_clean":"2024-03-24T00:38:59.676394+0000","last_became_active":"2024-03-24T00:38:52.772887+0000","last_became_peered":"2024-03-24T00:38:52.772887+0000","last_unstale":"2024-03-24T00:38:59.676394+0000","last_undegraded":"2024-03-24T00:38:59.676394+0000","last_fullsized":"2024-03-24T00:38:59.676394+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:27:23.302869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.800436+0000","last_change":"2024-03-24T00:38:52.784496+0000","last_active":"2024-03-24T00:38:58.800436+0000","last_peered":"2024-03-24T00:38:58.800436+0000","last_clean":"2024-03-24T00:38:58.800436+0000","last_became_active":"2024-03-24T00:38:52.784234+0000","last_became_peered":"2024-03-24T00:38:52.784234+0000","last_unstale":"2024-03-24T00:38:58.800436+0000","last_undegraded":"2024-03-24T00:38:58.800436+0000","last_fullsized":"2024-03-24T00:38:58.800436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:36:54.790308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802210+0000","last_change":"2024-03-24T00:38:52.786080+0000","last_active":"2024-03-24T00:38:58.802210+0000","last_peered":"2024-03-24T00:38:58.802210+0000","last_clean":"2024-03-24T00:38:58.802210+0000","last_became_active":"2024-03-24T00:38:52.785803+0000","last_became_peered":"2024-03-24T00:38:52.785803+0000","last_unstale":"2024-03-24T00:38:58.802210+0000","last_undegraded":"2024-03-24T00:38:58.802210+0000","last_fullsized":"2024-03-24T00:38:58.802210+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:51:22.179128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676587+0000","last_change":"2024-03-24T00:38:52.788391+0000","last_active":"2024-03-24T00:38:59.676587+0000","last_peered":"2024-03-24T00:38:59.676587+0000","last_clean":"2024-03-24T00:38:59.676587+0000","last_became_active":"2024-03-24T00:38:52.788122+0000","last_became_peered":"2024-03-24T00:38:52.788122+0000","last_unstale":"2024-03-24T00:38:59.676587+0000","last_undegraded":"2024-03-24T00:38:59.676587+0000","last_fullsized":"2024-03-24T00:38:59.676587+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:41:59.361242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676867+0000","last_change":"2024-03-24T00:38:52.787696+0000","last_active":"2024-03-24T00:38:59.676867+0000","last_peered":"2024-03-24T00:38:59.676867+0000","last_clean":"2024-03-24T00:38:59.676867+0000","last_became_active":"2024-03-24T00:38:52.787431+0000","last_became_peered":"2024-03-24T00:38:52.787431+0000","last_unstale":"2024-03-24T00:38:59.676867+0000","last_undegraded":"2024-03-24T00:38:59.676867+0000","last_fullsized":"2024-03-24T00:38:59.676867+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:46:05.527509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.677244+0000","last_change":"2024-03-24T00:38:52.791775+0000","last_active":"2024-03-24T00:38:59.677244+0000","last_peered":"2024-03-24T00:38:59.677244+0000","last_clean":"2024-03-24T00:38:59.677244+0000","last_became_active":"2024-03-24T00:38:52.788742+0000","last_became_peered":"2024-03-24T00:38:52.788742+0000","last_unstale":"2024-03-24T00:38:59.677244+0000","last_undegraded":"2024-03-24T00:38:59.677244+0000","last_fullsized":"2024-03-24T00:38:59.677244+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:08:00.887641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.799866+0000","last_change":"2024-03-24T00:38:52.790297+0000","last_active":"2024-03-24T00:38:58.799866+0000","last_peered":"2024-03-24T00:38:58.799866+0000","last_clean":"2024-03-24T00:38:58.799866+0000","last_became_active":"2024-03-24T00:38:52.789879+0000","last_became_peered":"2024-03-24T00:38:52.789879+0000","last_unstale":"2024-03-24T00:38:58.799866+0000","last_undegraded":"2024-03-24T00:38:58.799866+0000","last_fullsized":"2024-03-24T00:38:58.799866+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:41:59.361242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676732+0000","last_change":"2024-03-24T00:38:52.780267+0000","last_active":"2024-03-24T00:38:59.676732+0000","last_peered":"2024-03-24T00:38:59.676732+0000","last_clean":"2024-03-24T00:38:59.676732+0000","last_became_active":"2024-03-24T00:38:52.779973+0000","last_became_peered":"2024-03-24T00:38:52.779973+0000","last_unstale":"2024-03-24T00:38:59.676732+0000","last_undegraded":"2024-03-24T00:38:59.676732+0000","last_fullsized":"2024-03-24T00:38:59.676732+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:36:54.790308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802578+0000","last_change":"2024-03-24T00:38:52.776890+0000","last_active":"2024-03-24T00:38:58.802578+0000","last_peered":"2024-03-24T00:38:58.802578+0000","last_clean":"2024-03-24T00:38:58.802578+0000","last_became_active":"2024-03-24T00:38:52.776573+0000","last_became_peered":"2024-03-24T00:38:52.776573+0000","last_unstale":"2024-03-24T00:38:58.802578+0000","last_undegraded":"2024-03-24T00:38:58.802578+0000","last_fullsized":"2024-03-24T00:38:58.802578+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:02:43.127453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676561+0000","last_change":"2024-03-24T00:38:52.783960+0000","last_active":"2024-03-24T00:38:59.676561+0000","last_peered":"2024-03-24T00:38:59.676561+0000","last_clean":"2024-03-24T00:38:59.676561+0000","last_became_active":"2024-03-24T00:38:52.783606+0000","last_became_peered":"2024-03-24T00:38:52.783606+0000","last_unstale":"2024-03-24T00:38:59.676561+0000","last_undegraded":"2024-03-24T00:38:59.676561+0000","last_fullsized":"2024-03-24T00:38:59.676561+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:21:00.861116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.802672+0000","last_change":"2024-03-24T00:38:52.784222+0000","last_active":"2024-03-24T00:38:58.802672+0000","last_peered":"2024-03-24T00:38:58.802672+0000","last_clean":"2024-03-24T00:38:58.802672+0000","last_became_active":"2024-03-24T00:38:52.783951+0000","last_became_peered":"2024-03-24T00:38:52.783951+0000","last_unstale":"2024-03-24T00:38:58.802672+0000","last_undegraded":"2024-03-24T00:38:58.802672+0000","last_fullsized":"2024-03-24T00:38:58.802672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:04:24.260803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.676177+0000","last_change":"2024-03-24T00:38:52.774199+0000","last_active":"2024-03-24T00:38:59.676177+0000","last_peered":"2024-03-24T00:38:59.676177+0000","last_clean":"2024-03-24T00:38:59.676177+0000","last_became_active":"2024-03-24T00:38:52.773914+0000","last_became_peered":"2024-03-24T00:38:52.773914+0000","last_unstale":"2024-03-24T00:38:59.676177+0000","last_undegraded":"2024-03-24T00:38:59.676177+0000","last_fullsized":"2024-03-24T00:38:59.676177+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:54:01.232233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:58.975533+0000","last_change":"2024-03-24T00:38:52.778927+0000","last_active":"2024-03-24T00:38:58.975533+0000","last_peered":"2024-03-24T00:38:58.975533+0000","last_clean":"2024-03-24T00:38:58.975533+0000","last_became_active":"2024-03-24T00:38:52.778655+0000","last_became_peered":"2024-03-24T00:38:52.778655+0000","last_unstale":"2024-03-24T00:38:58.975533+0000","last_undegraded":"2024-03-24T00:38:58.975533+0000","last_fullsized":"2024-03-24T00:38:58.975533+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:21:12.524662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T00:38:59.677156+0000","last_change":"2024-03-24T00:38:52.785456+0000","last_active":"2024-03-24T00:38:59.677156+0000","last_peered":"2024-03-24T00:38:59.677156+0000","last_clean":"2024-03-24T00:38:59.677156+0000","last_became_active":"2024-03-24T00:38:52.785150+0000","last_became_peered":"2024-03-24T00:38:52.785150+0000","last_unstale":"2024-03-24T00:38:59.677156+0000","last_undegraded":"2024-03-24T00:38:59.677156+0000","last_fullsized":"2024-03-24T00:38:59.677156+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:51.753761+0000","last_clean_scrub_stamp":"2024-03-24T00:38:51.753761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:17:42.703499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574860,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27868,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343972,"statfs":{"total":96636764160,"available":96608227328,"internally_reserved":0,"allocated":864256,"data_stored":698343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574859,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":111235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574859,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":107483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574858,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":278528,"data_stored":107519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":111235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574859,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":107393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574859,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":274432,"data_stored":107553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27872,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343968,"statfs":{"total":96636764160,"available":96608223232,"internally_reserved":0,"allocated":872448,"data_stored":698399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:39:15.989 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-24T00:39:15.989 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-24T00:39:15.989 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-24T00:39:15.989 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-24T00:39:16.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.124+0000 7fd0cb2f8640 1 Processor -- start 2024-03-24T00:39:16.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.124+0000 7fd0cb2f8640 1 -- start start 2024-03-24T00:39:16.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.125+0000 7fd0cb2f8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd0c415dfb0 0x7fd0c40cfe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:16.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.125+0000 7fd0cb2f8640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd0c415e390 0x7fd0c406b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:16.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.125+0000 7fd0cb2f8640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0c40d07d0 con 0x7fd0c40cded0 2024-03-24T00:39:16.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.125+0000 7fd0cb2f8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0c40d0ab0 con 0x7fd0c415dfb0 2024-03-24T00:39:16.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.125+0000 7fd0cb2f8640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c40d0d90 con 0x7fd0c415e390 2024-03-24T00:39:16.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.125+0000 7fd0c9094640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd0c415dfb0 0x7fd0c40cfe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:16.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.125+0000 7fd0c9094640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd0c415dfb0 0x7fd0c40cfe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35994/0 (socket says 172.21.15.67:35994) 2024-03-24T00:39:16.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.125+0000 7fd0c9094640 1 -- 172.21.15.67:0/2083340400 learned_addr learned my addr 172.21.15.67:0/2083340400 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:16.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.125+0000 7fd0c8893640 1 --2- 172.21.15.67:0/2083340400 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd0c415e390 0x7fd0c406b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:16.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.125+0000 7fd0bbfff640 1 -- 172.21.15.67:0/2083340400 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 938348533 0 0) 0x7fd0c40d07d0 con 0x7fd0c40cded0 2024-03-24T00:39:16.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.125+0000 7fd0c9094640 1 -- 172.21.15.67:0/2083340400 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd0c415e390 msgr2=0x7fd0c406b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:16.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.125+0000 7fd0c9094640 1 --2- 172.21.15.67:0/2083340400 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd0c415e390 0x7fd0c406b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.125+0000 7fd0c9094640 1 -- 172.21.15.67:0/2083340400 >> v1:172.21.15.72:6789/0 conn(0x7fd0c40cded0 legacy=0x7fd0c4161d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:16.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.125+0000 7fd0c9094640 1 -- 172.21.15.67:0/2083340400 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0c40d1070 con 0x7fd0c415dfb0 2024-03-24T00:39:16.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.126+0000 7fd0c9094640 1 --2- 172.21.15.67:0/2083340400 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0c415dfb0 0x7fd0c40cfe80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc009870 tx=0x7fd0bc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81d3db56707cb992 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:16.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.126+0000 7fd0bbfff640 1 -- 172.21.15.67:0/2083340400 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0bc034070 con 0x7fd0c415dfb0 2024-03-24T00:39:16.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.126+0000 7fd0bbfff640 1 -- 172.21.15.67:0/2083340400 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0bc030840 con 0x7fd0c415dfb0 2024-03-24T00:39:16.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.126+0000 7fd0bbfff640 1 -- 172.21.15.67:0/2083340400 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0bc030c10 con 0x7fd0c415dfb0 2024-03-24T00:39:16.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.127+0000 7fd0cb2f8640 1 -- 172.21.15.67:0/2083340400 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0c415dfb0 msgr2=0x7fd0c40cfe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:16.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.127+0000 7fd0cb2f8640 1 --2- 172.21.15.67:0/2083340400 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0c415dfb0 0x7fd0c40cfe80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc009870 tx=0x7fd0bc02b980 comp rx=0 tx=0).stop 2024-03-24T00:39:16.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.127+0000 7fd0cb2f8640 1 -- 172.21.15.67:0/2083340400 shutdown_connections 2024-03-24T00:39:16.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.127+0000 7fd0cb2f8640 1 --2- 172.21.15.67:0/2083340400 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd0c415e390 0x7fd0c406b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.127+0000 7fd0cb2f8640 1 --2- 172.21.15.67:0/2083340400 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0c415dfb0 0x7fd0c40cfe80 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.127+0000 7fd0cb2f8640 1 -- 172.21.15.67:0/2083340400 >> 172.21.15.67:0/2083340400 conn(0x7fd0c4099a30 msgr2=0x7fd0c4099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:16.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.127+0000 7fd0cb2f8640 1 -- 172.21.15.67:0/2083340400 shutdown_connections 2024-03-24T00:39:16.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.127+0000 7fd0cb2f8640 1 -- 172.21.15.67:0/2083340400 wait complete. 2024-03-24T00:39:16.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.128+0000 7fd0cb2f8640 1 Processor -- start 2024-03-24T00:39:16.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.128+0000 7fd0cb2f8640 1 -- start start 2024-03-24T00:39:16.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.129+0000 7fd0cb2f8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0c40cded0 0x7fd0c40c9570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:16.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.129+0000 7fd0cb2f8640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd0c415dfb0 0x7fd0c40c9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:16.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.129+0000 7fd0cb2f8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0c415e390 0x7fd0c4149db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:16.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.129+0000 7fd0cb2f8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c40d1870 con 0x7fd0c415e390 2024-03-24T00:39:16.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.129+0000 7fd0cb2f8640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c40cc130 con 0x7fd0c40cded0 2024-03-24T00:39:16.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.129+0000 7fd0cb2f8640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c40d03c0 con 0x7fd0c415dfb0 2024-03-24T00:39:16.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.129+0000 7fd0c9895640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0c415e390 0x7fd0c4149db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:16.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.129+0000 7fd0c9895640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0c415e390 0x7fd0c4149db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36000/0 (socket says 172.21.15.67:36000) 2024-03-24T00:39:16.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.129+0000 7fd0c9895640 1 -- 172.21.15.67:0/904090974 learned_addr learned my addr 172.21.15.67:0/904090974 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:16.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.129+0000 7fd0c8893640 1 --2- 172.21.15.67:0/904090974 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd0c415dfb0 0x7fd0c40c9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:16.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.129+0000 7fd0c9094640 1 --2- 172.21.15.67:0/904090974 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0c40cded0 0x7fd0c40c9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:16.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.129+0000 7fd0c9895640 1 -- 172.21.15.67:0/904090974 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd0c415dfb0 msgr2=0x7fd0c40c9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:16.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.129+0000 7fd0c9895640 1 --2- 172.21.15.67:0/904090974 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd0c415dfb0 0x7fd0c40c9ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.129+0000 7fd0c9895640 1 -- 172.21.15.67:0/904090974 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0c40cded0 msgr2=0x7fd0c40c9570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:16.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.129+0000 7fd0c9895640 1 --2- 172.21.15.67:0/904090974 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0c40cded0 0x7fd0c40c9570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.129+0000 7fd0c9895640 1 -- 172.21.15.67:0/904090974 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0c40c9ff0 con 0x7fd0c415e390 2024-03-24T00:39:16.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.130+0000 7fd0c9895640 1 --2- 172.21.15.67:0/904090974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0c415e390 0x7fd0c4149db0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd0c000c790 tx=0x7fd0c000cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:16.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.130+0000 7fd0b9ffb640 1 -- 172.21.15.67:0/904090974 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c0003da0 con 0x7fd0c415e390 2024-03-24T00:39:16.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.130+0000 7fd0b9ffb640 1 -- 172.21.15.67:0/904090974 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0c0005070 con 0x7fd0c415e390 2024-03-24T00:39:16.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.130+0000 7fd0cb2f8640 1 -- 172.21.15.67:0/904090974 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0c40ca280 con 0x7fd0c415e390 2024-03-24T00:39:16.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.130+0000 7fd0b9ffb640 1 -- 172.21.15.67:0/904090974 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c0005460 con 0x7fd0c415e390 2024-03-24T00:39:16.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.130+0000 7fd0cb2f8640 1 -- 172.21.15.67:0/904090974 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0c40ca840 con 0x7fd0c415e390 2024-03-24T00:39:16.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.131+0000 7fd0cb2f8640 1 -- 172.21.15.67:0/904090974 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0c404f400 con 0x7fd0c415e390 2024-03-24T00:39:16.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.131+0000 7fd0b9ffb640 1 -- 172.21.15.67:0/904090974 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd0c0005600 con 0x7fd0c415e390 2024-03-24T00:39:16.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.132+0000 7fd0b9ffb640 1 --2- 172.21.15.67:0/904090974 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd0a806bbd0 0x7fd0a806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:16.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.132+0000 7fd0b9ffb640 1 -- 172.21.15.67:0/904090974 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fd0c008ab10 con 0x7fd0c415e390 2024-03-24T00:39:16.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.132+0000 7fd0c9094640 1 --2- 172.21.15.67:0/904090974 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd0a806bbd0 0x7fd0a806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:16.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.132+0000 7fd0c9094640 1 --2- 172.21.15.67:0/904090974 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd0a806bbd0 0x7fd0a806e090 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc036000 tx=0x7fd0bc008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:16.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.136+0000 7fd0b9ffb640 1 -- 172.21.15.67:0/904090974 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd0c0058d00 con 0x7fd0c415e390 2024-03-24T00:39:16.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.360+0000 7fd0cb2f8640 1 -- 172.21.15.67:0/904090974 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd0c4073c70 con 0x7fd0c415e390 2024-03-24T00:39:16.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.360+0000 7fd0b9ffb640 1 -- 172.21.15.67:0/904090974 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fd0c005cda0 con 0x7fd0c415e390 2024-03-24T00:39:16.361 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:39:16.361 INFO:teuthology.orchestra.run.smithi067.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-24T00:39:16.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.362+0000 7fd0cb2f8640 1 -- 172.21.15.67:0/904090974 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd0a806bbd0 msgr2=0x7fd0a806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:16.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.362+0000 7fd0cb2f8640 1 --2- 172.21.15.67:0/904090974 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd0a806bbd0 0x7fd0a806e090 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc036000 tx=0x7fd0bc008030 comp rx=0 tx=0).stop 2024-03-24T00:39:16.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.362+0000 7fd0cb2f8640 1 -- 172.21.15.67:0/904090974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0c415e390 msgr2=0x7fd0c4149db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:16.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.362+0000 7fd0cb2f8640 1 --2- 172.21.15.67:0/904090974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0c415e390 0x7fd0c4149db0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd0c000c790 tx=0x7fd0c000cc50 comp rx=0 tx=0).stop 2024-03-24T00:39:16.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.363+0000 7fd0cb2f8640 1 -- 172.21.15.67:0/904090974 shutdown_connections 2024-03-24T00:39:16.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.363+0000 7fd0cb2f8640 1 --2- 172.21.15.67:0/904090974 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd0a806bbd0 0x7fd0a806e090 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.363+0000 7fd0cb2f8640 1 --2- 172.21.15.67:0/904090974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0c415e390 0x7fd0c4149db0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.363+0000 7fd0cb2f8640 1 --2- 172.21.15.67:0/904090974 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd0c415dfb0 0x7fd0c40c9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.363+0000 7fd0cb2f8640 1 --2- 172.21.15.67:0/904090974 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0c40cded0 0x7fd0c40c9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.363+0000 7fd0cb2f8640 1 -- 172.21.15.67:0/904090974 >> 172.21.15.67:0/904090974 conn(0x7fd0c4099a30 msgr2=0x7fd0c40b1ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:16.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.363+0000 7fd0cb2f8640 1 -- 172.21.15.67:0/904090974 shutdown_connections 2024-03-24T00:39:16.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.363+0000 7fd0cb2f8640 1 -- 172.21.15.67:0/904090974 wait complete. 2024-03-24T00:39:16.374 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-24T00:39:16.374 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:39:16.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.475+0000 7f3416b45640 1 Processor -- start 2024-03-24T00:39:16.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.475+0000 7f3416b45640 1 -- start start 2024-03-24T00:39:16.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.476+0000 7f3416b45640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f341015e2a0 0x7f34100ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:16.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.476+0000 7f3416b45640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f34100d0ef0 0x7f3410162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:16.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.476+0000 7f3416b45640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34100cf640 con 0x7f341015dec0 2024-03-24T00:39:16.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.476+0000 7f3416b45640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34100cee30 con 0x7f341015e2a0 2024-03-24T00:39:16.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.476+0000 7f3416b45640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34100d2120 con 0x7f34100d0ef0 2024-03-24T00:39:16.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.476+0000 7f340ffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f341015e2a0 0x7f34100ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:16.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.476+0000 7f34148e1640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f341015dec0 0x7f34100d07e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47852/0 (socket says 172.21.15.67:47852) 2024-03-24T00:39:16.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.476+0000 7f34148e1640 1 -- 172.21.15.67:0/95211269 learned_addr learned my addr 172.21.15.67:0/95211269 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:16.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.476+0000 7f34150e2640 1 --2- 172.21.15.67:0/95211269 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f34100d0ef0 0x7f3410162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:16.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.477+0000 7f34150e2640 1 -- 172.21.15.67:0/95211269 >> v2:172.21.15.67:3300/0 conn(0x7f341015e2a0 msgr2=0x7f34100ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:16.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.477+0000 7f34150e2640 1 --2- 172.21.15.67:0/95211269 >> v2:172.21.15.67:3300/0 conn(0x7f341015e2a0 0x7f34100ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.477+0000 7f34150e2640 1 -- 172.21.15.67:0/95211269 >> v1:172.21.15.67:6789/0 conn(0x7f341015dec0 legacy=0x7f34100d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:16.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.477+0000 7f34150e2640 1 -- 172.21.15.67:0/95211269 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34100cf080 con 0x7f34100d0ef0 2024-03-24T00:39:16.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.477+0000 7f340f7fe640 1 -- 172.21.15.67:0/95211269 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2344580092 0 0) 0x7f3408001490 con 0x7f341015dec0 2024-03-24T00:39:16.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.477+0000 7f340ffff640 1 --2- 172.21.15.67:0/95211269 >> v2:172.21.15.67:3300/0 conn(0x7f341015e2a0 0x7f34100ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:16.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.477+0000 7f34150e2640 1 --2- 172.21.15.67:0/95211269 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f34100d0ef0 0x7f3410162630 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f34000097a0 tx=0x7f340002b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9bffbedbf69a2b27 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:16.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.478+0000 7f340f7fe640 1 -- 172.21.15.67:0/95211269 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3400031780 con 0x7f34100d0ef0 2024-03-24T00:39:16.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.478+0000 7f340f7fe640 1 -- 172.21.15.67:0/95211269 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3400037070 con 0x7f34100d0ef0 2024-03-24T00:39:16.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.478+0000 7f340f7fe640 1 -- 172.21.15.67:0/95211269 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3400031b40 con 0x7f34100d0ef0 2024-03-24T00:39:16.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.478+0000 7f3416b45640 1 -- 172.21.15.67:0/95211269 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f34100d0ef0 msgr2=0x7f3410162630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:16.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.478+0000 7f3416b45640 1 --2- 172.21.15.67:0/95211269 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f34100d0ef0 0x7f3410162630 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f34000097a0 tx=0x7f340002b820 comp rx=0 tx=0).stop 2024-03-24T00:39:16.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.478+0000 7f3416b45640 1 -- 172.21.15.67:0/95211269 shutdown_connections 2024-03-24T00:39:16.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.478+0000 7f3416b45640 1 --2- 172.21.15.67:0/95211269 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f34100d0ef0 0x7f3410162630 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.478+0000 7f3416b45640 1 --2- 172.21.15.67:0/95211269 >> v2:172.21.15.67:3300/0 conn(0x7f341015e2a0 0x7f34100ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.478+0000 7f3416b45640 1 -- 172.21.15.67:0/95211269 >> 172.21.15.67:0/95211269 conn(0x7f341006c810 msgr2=0x7f341006cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:16.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.478+0000 7f3416b45640 1 -- 172.21.15.67:0/95211269 shutdown_connections 2024-03-24T00:39:16.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.478+0000 7f3416b45640 1 -- 172.21.15.67:0/95211269 wait complete. 2024-03-24T00:39:16.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.479+0000 7f3416b45640 1 Processor -- start 2024-03-24T00:39:16.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.479+0000 7f3416b45640 1 -- start start 2024-03-24T00:39:16.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.480+0000 7f3416b45640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f34100d0ef0 0x7f34101f3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:16.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.480+0000 7f3416b45640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f341015dec0 0x7f34100ab100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:16.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.480+0000 7f3416b45640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f341015e2a0 0x7f3410160a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:16.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.480+0000 7f3416b45640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34100d2730 con 0x7f341015dec0 2024-03-24T00:39:16.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.480+0000 7f3416b45640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34100d25b0 con 0x7f341015e2a0 2024-03-24T00:39:16.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.480+0000 7f3416b45640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34100d28b0 con 0x7f34100d0ef0 2024-03-24T00:39:16.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.480+0000 7f340ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f341015dec0 0x7f34100ab100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:16.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.480+0000 7f340ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f341015dec0 0x7f34100ab100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36032/0 (socket says 172.21.15.67:36032) 2024-03-24T00:39:16.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.480+0000 7f34150e2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f341015e2a0 0x7f3410160a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:16.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.480+0000 7f340ffff640 1 -- 172.21.15.67:0/71489219 learned_addr learned my addr 172.21.15.67:0/71489219 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:16.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.480+0000 7f34148e1640 1 --2- 172.21.15.67:0/71489219 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f34100d0ef0 0x7f34101f3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:16.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.481+0000 7f340ffff640 1 -- 172.21.15.67:0/71489219 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f34100d0ef0 msgr2=0x7f34101f3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:16.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.481+0000 7f340ffff640 1 --2- 172.21.15.67:0/71489219 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f34100d0ef0 0x7f34101f3cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.481+0000 7f340ffff640 1 -- 172.21.15.67:0/71489219 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f341015e2a0 msgr2=0x7f3410160a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:16.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.481+0000 7f340ffff640 1 --2- 172.21.15.67:0/71489219 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f341015e2a0 0x7f3410160a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.481+0000 7f340ffff640 1 -- 172.21.15.67:0/71489219 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34100ab9f0 con 0x7f341015dec0 2024-03-24T00:39:16.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.481+0000 7f34150e2640 1 --2- 172.21.15.67:0/71489219 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f341015e2a0 0x7f3410160a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:16.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.481+0000 7f340ffff640 1 --2- 172.21.15.67:0/71489219 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f341015dec0 0x7f34100ab100 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f33f800e900 tx=0x7f33f800edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:16.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.481+0000 7f340d7fa640 1 -- 172.21.15.67:0/71489219 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f8017070 con 0x7f341015dec0 2024-03-24T00:39:16.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.481+0000 7f340d7fa640 1 -- 172.21.15.67:0/71489219 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33f800ca00 con 0x7f341015dec0 2024-03-24T00:39:16.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.481+0000 7f340d7fa640 1 -- 172.21.15.67:0/71489219 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f800cdf0 con 0x7f341015dec0 2024-03-24T00:39:16.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.481+0000 7f3416b45640 1 -- 172.21.15.67:0/71489219 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34100abcd0 con 0x7f341015dec0 2024-03-24T00:39:16.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.481+0000 7f3416b45640 1 -- 172.21.15.67:0/71489219 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34100ac290 con 0x7f341015dec0 2024-03-24T00:39:16.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.482+0000 7f3416b45640 1 -- 172.21.15.67:0/71489219 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f341004f400 con 0x7f341015dec0 2024-03-24T00:39:16.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.482+0000 7f340d7fa640 1 -- 172.21.15.67:0/71489219 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f33f801a070 con 0x7f341015dec0 2024-03-24T00:39:16.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.483+0000 7f340d7fa640 1 --2- 172.21.15.67:0/71489219 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f33f006bbd0 0x7f33f006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:16.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.483+0000 7f340d7fa640 1 -- 172.21.15.67:0/71489219 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f33f808ac20 con 0x7f341015dec0 2024-03-24T00:39:16.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.483+0000 7f34148e1640 1 --2- 172.21.15.67:0/71489219 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f33f006bbd0 0x7f33f006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:16.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.483+0000 7f34148e1640 1 --2- 172.21.15.67:0/71489219 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f33f006bbd0 0x7f33f006e090 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f34080012e0 tx=0x7f3408007ed0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:16.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.487+0000 7f340d7fa640 1 -- 172.21.15.67:0/71489219 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f33f8058d90 con 0x7f341015dec0 2024-03-24T00:39:16.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.670+0000 7f3416b45640 1 -- 172.21.15.67:0/71489219 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34100d1aa0 con 0x7f341015dec0 2024-03-24T00:39:16.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.671+0000 7f340d7fa640 1 -- 172.21.15.67:0/71489219 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18454 (secure 0 0 0) 0x7f33f805ce30 con 0x7f341015dec0 2024-03-24T00:39:16.672 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:39:16.673 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":25,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:38:58.797179+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:38:50.077085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:38:51.095454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T00:38:54.807117+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:39:16.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.673+0000 7f3416b45640 1 -- 172.21.15.67:0/71489219 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f33f006bbd0 msgr2=0x7f33f006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:16.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.673+0000 7f3416b45640 1 --2- 172.21.15.67:0/71489219 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f33f006bbd0 0x7f33f006e090 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f34080012e0 tx=0x7f3408007ed0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.673+0000 7f3416b45640 1 -- 172.21.15.67:0/71489219 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f341015dec0 msgr2=0x7f34100ab100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:16.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.673+0000 7f3416b45640 1 --2- 172.21.15.67:0/71489219 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f341015dec0 0x7f34100ab100 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f33f800e900 tx=0x7f33f800edc0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.673+0000 7f3416b45640 1 -- 172.21.15.67:0/71489219 shutdown_connections 2024-03-24T00:39:16.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.673+0000 7f3416b45640 1 --2- 172.21.15.67:0/71489219 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f33f006bbd0 0x7f33f006e090 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.673+0000 7f3416b45640 1 --2- 172.21.15.67:0/71489219 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f341015e2a0 0x7f3410160a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.673+0000 7f3416b45640 1 --2- 172.21.15.67:0/71489219 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f341015dec0 0x7f34100ab100 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.673+0000 7f3416b45640 1 --2- 172.21.15.67:0/71489219 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f34100d0ef0 0x7f34101f3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.673+0000 7f3416b45640 1 -- 172.21.15.67:0/71489219 >> 172.21.15.67:0/71489219 conn(0x7f341006c810 msgr2=0x7f341015bde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:16.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.674+0000 7f3416b45640 1 -- 172.21.15.67:0/71489219 shutdown_connections 2024-03-24T00:39:16.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.674+0000 7f3416b45640 1 -- 172.21.15.67:0/71489219 wait complete. 2024-03-24T00:39:16.685 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:38:39.364950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:38:50.077085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:38:51.095454+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T00:38:54.807117+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T00:39:16.686 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:39:16.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.793+0000 7f64e66ea640 1 Processor -- start 2024-03-24T00:39:16.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.793+0000 7f64e66ea640 1 -- start start 2024-03-24T00:39:16.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.794+0000 7f64e66ea640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f64e015afc0 0x7f64e00d0920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:16.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.794+0000 7f64e66ea640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f64e00cc050 0x7f64e00c4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:16.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.794+0000 7f64e66ea640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64e00d12c0 con 0x7f64e00d0e60 2024-03-24T00:39:16.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.794+0000 7f64e66ea640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64e00cc4e0 con 0x7f64e00cc050 2024-03-24T00:39:16.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.794+0000 7f64e66ea640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64e015b3a0 con 0x7f64e015afc0 2024-03-24T00:39:16.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.794+0000 7f64e4c87640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f64e00cc050 0x7f64e00c4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:16.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.794+0000 7f64df7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f64e00d0e60 0x7f64e00cb940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47860/0 (socket says 172.21.15.67:47860) 2024-03-24T00:39:16.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.794+0000 7f64df7fe640 1 -- 172.21.15.67:0/3130909455 learned_addr learned my addr 172.21.15.67:0/3130909455 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:16.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.794+0000 7f64dffff640 1 --2- 172.21.15.67:0/3130909455 >> v2:172.21.15.72:3300/0 conn(0x7f64e015afc0 0x7f64e00d0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:16.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.794+0000 7f64e4c87640 1 -- 172.21.15.67:0/3130909455 >> v2:172.21.15.72:3300/0 conn(0x7f64e015afc0 msgr2=0x7f64e00d0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:16.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.794+0000 7f64e4c87640 1 --2- 172.21.15.67:0/3130909455 >> v2:172.21.15.72:3300/0 conn(0x7f64e015afc0 0x7f64e00d0920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.794+0000 7f64e4c87640 1 -- 172.21.15.67:0/3130909455 >> v1:172.21.15.67:6789/0 conn(0x7f64e00d0e60 legacy=0x7f64e00cb940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:16.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.794+0000 7f64e4c87640 1 -- 172.21.15.67:0/3130909455 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e00cc700 con 0x7f64e00cc050 2024-03-24T00:39:16.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.794+0000 7f64deffd640 1 -- 172.21.15.67:0/3130909455 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2644090727 0 0) 0x7f64e00d12c0 con 0x7f64e00d0e60 2024-03-24T00:39:16.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.794+0000 7f64dffff640 1 --2- 172.21.15.67:0/3130909455 >> v2:172.21.15.72:3300/0 conn(0x7f64e015afc0 0x7f64e00d0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:16.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.795+0000 7f64e4c87640 1 --2- 172.21.15.67:0/3130909455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64e00cc050 0x7f64e00c4a20 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f64e00ce600 tx=0x7f64d802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b61a22a6034fd86 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:16.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.795+0000 7f64deffd640 1 -- 172.21.15.67:0/3130909455 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d8039070 con 0x7f64e00cc050 2024-03-24T00:39:16.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.795+0000 7f64deffd640 1 -- 172.21.15.67:0/3130909455 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64d802cc80 con 0x7f64e00cc050 2024-03-24T00:39:16.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.795+0000 7f64deffd640 1 -- 172.21.15.67:0/3130909455 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d8034660 con 0x7f64e00cc050 2024-03-24T00:39:16.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.796+0000 7f64e66ea640 1 -- 172.21.15.67:0/3130909455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64e00cc050 msgr2=0x7f64e00c4a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:16.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.796+0000 7f64e66ea640 1 --2- 172.21.15.67:0/3130909455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64e00cc050 0x7f64e00c4a20 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f64e00ce600 tx=0x7f64d802bed0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.796+0000 7f64e66ea640 1 -- 172.21.15.67:0/3130909455 shutdown_connections 2024-03-24T00:39:16.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.796+0000 7f64e66ea640 1 --2- 172.21.15.67:0/3130909455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64e00cc050 0x7f64e00c4a20 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.796+0000 7f64e66ea640 1 --2- 172.21.15.67:0/3130909455 >> v2:172.21.15.72:3300/0 conn(0x7f64e015afc0 0x7f64e00d0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.796+0000 7f64e66ea640 1 -- 172.21.15.67:0/3130909455 >> 172.21.15.67:0/3130909455 conn(0x7f64e0099a30 msgr2=0x7f64e0099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:16.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.796+0000 7f64e66ea640 1 -- 172.21.15.67:0/3130909455 shutdown_connections 2024-03-24T00:39:16.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.796+0000 7f64e66ea640 1 -- 172.21.15.67:0/3130909455 wait complete. 2024-03-24T00:39:16.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.797+0000 7f64e66ea640 1 Processor -- start 2024-03-24T00:39:16.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.797+0000 7f64e66ea640 1 -- start start 2024-03-24T00:39:16.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.798+0000 7f64e66ea640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64e00cc050 0x7f64e00c6e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:16.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.798+0000 7f64e66ea640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64e00d0e60 0x7f64e006b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:16.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.798+0000 7f64e66ea640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f64e015afc0 0x7f64e00b4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:16.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.798+0000 7f64e66ea640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e00ccfa0 con 0x7f64e00d0e60 2024-03-24T00:39:16.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.798+0000 7f64e66ea640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e00cccc0 con 0x7f64e00cc050 2024-03-24T00:39:16.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.798+0000 7f64e66ea640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64e00c4f60 con 0x7f64e015afc0 2024-03-24T00:39:16.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.798+0000 7f64df7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64e00d0e60 0x7f64e006b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:16.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.798+0000 7f64df7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64e00d0e60 0x7f64e006b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36060/0 (socket says 172.21.15.67:36060) 2024-03-24T00:39:16.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.798+0000 7f64df7fe640 1 -- 172.21.15.67:0/866970400 learned_addr learned my addr 172.21.15.67:0/866970400 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:16.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.798+0000 7f64e4c87640 1 --2- 172.21.15.67:0/866970400 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f64e015afc0 0x7f64e00b4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:16.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.798+0000 7f64dffff640 1 --2- 172.21.15.67:0/866970400 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64e00cc050 0x7f64e00c6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:16.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.798+0000 7f64dffff640 1 -- 172.21.15.67:0/866970400 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f64e015afc0 msgr2=0x7f64e00b4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:16.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.798+0000 7f64dffff640 1 --2- 172.21.15.67:0/866970400 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f64e015afc0 0x7f64e00b4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.798+0000 7f64dffff640 1 -- 172.21.15.67:0/866970400 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64e00d0e60 msgr2=0x7f64e006b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:16.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.798+0000 7f64dffff640 1 --2- 172.21.15.67:0/866970400 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64e00d0e60 0x7f64e006b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.798+0000 7f64dffff640 1 -- 172.21.15.67:0/866970400 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e00b4c10 con 0x7f64e00cc050 2024-03-24T00:39:16.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.798+0000 7f64df7fe640 1 --2- 172.21.15.67:0/866970400 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64e00d0e60 0x7f64e006b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:16.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.798+0000 7f64e4c87640 1 --2- 172.21.15.67:0/866970400 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f64e015afc0 0x7f64e00b4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:16.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.798+0000 7f64dffff640 1 --2- 172.21.15.67:0/866970400 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64e00cc050 0x7f64e00c6e90 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f64d000e870 tx=0x7f64d000ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:16.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.799+0000 7f64dcff9640 1 -- 172.21.15.67:0/866970400 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d0017070 con 0x7f64e00cc050 2024-03-24T00:39:16.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.799+0000 7f64dcff9640 1 -- 172.21.15.67:0/866970400 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64d000c940 con 0x7f64e00cc050 2024-03-24T00:39:16.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.799+0000 7f64e66ea640 1 -- 172.21.15.67:0/866970400 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64e00c66b0 con 0x7f64e00cc050 2024-03-24T00:39:16.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.799+0000 7f64dcff9640 1 -- 172.21.15.67:0/866970400 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d000cd10 con 0x7f64e00cc050 2024-03-24T00:39:16.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.799+0000 7f64e66ea640 1 -- 172.21.15.67:0/866970400 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64e00b4ef0 con 0x7f64e00cc050 2024-03-24T00:39:16.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.800+0000 7f64e66ea640 1 -- 172.21.15.67:0/866970400 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64e004f400 con 0x7f64e00cc050 2024-03-24T00:39:16.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.800+0000 7f64dcff9640 1 -- 172.21.15.67:0/866970400 <== mon.1 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f64d001a070 con 0x7f64e00cc050 2024-03-24T00:39:16.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.800+0000 7f64dcff9640 1 --2- 172.21.15.67:0/866970400 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f64cc06bc00 0x7f64cc06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:16.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.800+0000 7f64dcff9640 1 -- 172.21.15.67:0/866970400 <== mon.1 v2:172.21.15.72:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f64d008ae50 con 0x7f64e00cc050 2024-03-24T00:39:16.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.801+0000 7f64df7fe640 1 --2- 172.21.15.67:0/866970400 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f64cc06bc00 0x7f64cc06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:16.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.801+0000 7f64df7fe640 1 --2- 172.21.15.67:0/866970400 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f64cc06bc00 0x7f64cc06e0c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f64e006eb50 tx=0x7f64d4007e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:16.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.805+0000 7f64dcff9640 1 -- 172.21.15.67:0/866970400 <== mon.1 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f64d0058fc0 con 0x7f64e00cc050 2024-03-24T00:39:16.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.988+0000 7f64e66ea640 1 -- 172.21.15.67:0/866970400 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f64e00d17a0 con 0x7f64e00cc050 2024-03-24T00:39:16.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.988+0000 7f64dcff9640 1 -- 172.21.15.67:0/866970400 <== mon.1 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7f64d005d060 con 0x7f64e00cc050 2024-03-24T00:39:16.989 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-24T00:39:16.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.990+0000 7f64e66ea640 1 -- 172.21.15.67:0/866970400 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f64cc06bc00 msgr2=0x7f64cc06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:16.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.990+0000 7f64e66ea640 1 --2- 172.21.15.67:0/866970400 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f64cc06bc00 0x7f64cc06e0c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f64e006eb50 tx=0x7f64d4007e10 comp rx=0 tx=0).stop 2024-03-24T00:39:16.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.990+0000 7f64e66ea640 1 -- 172.21.15.67:0/866970400 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64e00cc050 msgr2=0x7f64e00c6e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:16.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.990+0000 7f64e66ea640 1 --2- 172.21.15.67:0/866970400 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64e00cc050 0x7f64e00c6e90 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f64d000e870 tx=0x7f64d000ed30 comp rx=0 tx=0).stop 2024-03-24T00:39:16.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.990+0000 7f64e66ea640 1 -- 172.21.15.67:0/866970400 shutdown_connections 2024-03-24T00:39:16.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.990+0000 7f64e66ea640 1 --2- 172.21.15.67:0/866970400 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f64cc06bc00 0x7f64cc06e0c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.990+0000 7f64e66ea640 1 --2- 172.21.15.67:0/866970400 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f64e015afc0 0x7f64e00b4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.990+0000 7f64e66ea640 1 --2- 172.21.15.67:0/866970400 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64e00d0e60 0x7f64e006b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.990+0000 7f64e66ea640 1 --2- 172.21.15.67:0/866970400 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64e00cc050 0x7f64e00c6e90 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.990+0000 7f64e66ea640 1 -- 172.21.15.67:0/866970400 >> 172.21.15.67:0/866970400 conn(0x7f64e0099a30 msgr2=0x7f64e0070350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:16.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.990+0000 7f64e66ea640 1 -- 172.21.15.67:0/866970400 shutdown_connections 2024-03-24T00:39:16.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:16.990+0000 7f64e66ea640 1 -- 172.21.15.67:0/866970400 wait complete. 2024-03-24T00:39:17.002 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:39:17.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.105+0000 7f3264c0b640 1 Processor -- start 2024-03-24T00:39:17.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.105+0000 7f3264c0b640 1 -- start start 2024-03-24T00:39:17.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.106+0000 7f3264c0b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f326015dea0 0x7f32600d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:17.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.106+0000 7f3264c0b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f326015e280 0x7f32600d3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:17.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.106+0000 7f3264c0b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f32600b5a60 0x7f32600b5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:17.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.106+0000 7f3264c0b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32600b63f0 con 0x7f326015e280 2024-03-24T00:39:17.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.106+0000 7f3264c0b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32600d2250 con 0x7f326015dea0 2024-03-24T00:39:17.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.106+0000 7f3264c0b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32600d2530 con 0x7f32600b5a60 2024-03-24T00:39:17.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.106+0000 7f3255d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f326015e280 0x7f32600d3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:17.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.106+0000 7f3255d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f326015e280 0x7f32600d3910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36070/0 (socket says 172.21.15.67:36070) 2024-03-24T00:39:17.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.106+0000 7f3255d9b640 1 -- 172.21.15.67:0/458663998 learned_addr learned my addr 172.21.15.67:0/458663998 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:17.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.106+0000 7f325e59c640 1 --2- 172.21.15.67:0/458663998 >> v2:172.21.15.72:3300/0 conn(0x7f326015dea0 0x7f32600d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:17.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.106+0000 7f325ed9d640 1 --2- 172.21.15.67:0/458663998 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f32600b5a60 0x7f32600b5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:17.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.106+0000 7f3255d9b640 1 -- 172.21.15.67:0/458663998 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f32600b5a60 msgr2=0x7f32600b5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:17.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.106+0000 7f3255d9b640 1 --2- 172.21.15.67:0/458663998 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f32600b5a60 0x7f32600b5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:17.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.106+0000 7f3255d9b640 1 -- 172.21.15.67:0/458663998 >> v2:172.21.15.72:3300/0 conn(0x7f326015dea0 msgr2=0x7f32600d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:17.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.106+0000 7f3255d9b640 1 --2- 172.21.15.67:0/458663998 >> v2:172.21.15.72:3300/0 conn(0x7f326015dea0 0x7f32600d33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:17.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.107+0000 7f3255d9b640 1 -- 172.21.15.67:0/458663998 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32600d2810 con 0x7f326015e280 2024-03-24T00:39:17.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.107+0000 7f3255d9b640 1 --2- 172.21.15.67:0/458663998 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f326015e280 0x7f32600d3910 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f32600d4c30 tx=0x7f324802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3858aa4928c592d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:17.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.107+0000 7f325dd9b640 1 -- 172.21.15.67:0/458663998 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f324802e070 con 0x7f326015e280 2024-03-24T00:39:17.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.107+0000 7f325dd9b640 1 -- 172.21.15.67:0/458663998 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3248037070 con 0x7f326015e280 2024-03-24T00:39:17.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.107+0000 7f325dd9b640 1 -- 172.21.15.67:0/458663998 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3248031a00 con 0x7f326015e280 2024-03-24T00:39:17.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.108+0000 7f3264c0b640 1 -- 172.21.15.67:0/458663998 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f326015e280 msgr2=0x7f32600d3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:17.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.108+0000 7f3264c0b640 1 --2- 172.21.15.67:0/458663998 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f326015e280 0x7f32600d3910 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f32600d4c30 tx=0x7f324802b880 comp rx=0 tx=0).stop 2024-03-24T00:39:17.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.108+0000 7f3264c0b640 1 -- 172.21.15.67:0/458663998 shutdown_connections 2024-03-24T00:39:17.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.108+0000 7f3264c0b640 1 --2- 172.21.15.67:0/458663998 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f32600b5a60 0x7f32600b5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:17.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.108+0000 7f3264c0b640 1 --2- 172.21.15.67:0/458663998 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f326015e280 0x7f32600d3910 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:17.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.108+0000 7f3264c0b640 1 --2- 172.21.15.67:0/458663998 >> v2:172.21.15.72:3300/0 conn(0x7f326015dea0 0x7f32600d33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:17.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.108+0000 7f3264c0b640 1 -- 172.21.15.67:0/458663998 >> 172.21.15.67:0/458663998 conn(0x7f3260072da0 msgr2=0x7f32600d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:17.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.108+0000 7f3264c0b640 1 -- 172.21.15.67:0/458663998 shutdown_connections 2024-03-24T00:39:17.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.108+0000 7f3264c0b640 1 -- 172.21.15.67:0/458663998 wait complete. 2024-03-24T00:39:17.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.109+0000 7f3264c0b640 1 Processor -- start 2024-03-24T00:39:17.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.109+0000 7f3264c0b640 1 -- start start 2024-03-24T00:39:17.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.110+0000 7f3264c0b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32600b5a60 0x7f32600acad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:17.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.110+0000 7f3264c0b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f326015dea0 0x7f32600ae430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:17.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.110+0000 7f3264c0b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f326015e280 0x7f32600ab3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:17.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.110+0000 7f3264c0b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f326006c7c0 con 0x7f326015dea0 2024-03-24T00:39:17.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.110+0000 7f3264c0b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f326006c640 con 0x7f32600b5a60 2024-03-24T00:39:17.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.110+0000 7f3264c0b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f326006c940 con 0x7f326015e280 2024-03-24T00:39:17.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.110+0000 7f3255d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f326015dea0 0x7f32600ae430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:17.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.110+0000 7f3255d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f326015dea0 0x7f32600ae430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36082/0 (socket says 172.21.15.67:36082) 2024-03-24T00:39:17.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.110+0000 7f3255d9b640 1 -- 172.21.15.67:0/879561129 learned_addr learned my addr 172.21.15.67:0/879561129 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:17.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.110+0000 7f325ed9d640 1 --2- 172.21.15.67:0/879561129 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f326015e280 0x7f32600ab3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:17.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.110+0000 7f325e59c640 1 --2- 172.21.15.67:0/879561129 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32600b5a60 0x7f32600acad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:17.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.111+0000 7f3255d9b640 1 -- 172.21.15.67:0/879561129 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f326015e280 msgr2=0x7f32600ab3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:17.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.111+0000 7f3255d9b640 1 --2- 172.21.15.67:0/879561129 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f326015e280 0x7f32600ab3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:17.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.111+0000 7f3255d9b640 1 -- 172.21.15.67:0/879561129 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32600b5a60 msgr2=0x7f32600acad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:17.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.111+0000 7f3255d9b640 1 --2- 172.21.15.67:0/879561129 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32600b5a60 0x7f32600acad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:17.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.111+0000 7f3255d9b640 1 -- 172.21.15.67:0/879561129 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32600cc1a0 con 0x7f326015dea0 2024-03-24T00:39:17.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.111+0000 7f325e59c640 1 --2- 172.21.15.67:0/879561129 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32600b5a60 0x7f32600acad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:17.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.111+0000 7f3255d9b640 1 --2- 172.21.15.67:0/879561129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f326015dea0 0x7f32600ae430 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f324802b8c0 tx=0x7f32480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:17.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.111+0000 7f32577fe640 1 -- 172.21.15.67:0/879561129 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3248032670 con 0x7f326015dea0 2024-03-24T00:39:17.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.111+0000 7f32577fe640 1 -- 172.21.15.67:0/879561129 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3248032830 con 0x7f326015dea0 2024-03-24T00:39:17.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.111+0000 7f32577fe640 1 -- 172.21.15.67:0/879561129 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3248032c20 con 0x7f326015dea0 2024-03-24T00:39:17.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.111+0000 7f3264c0b640 1 -- 172.21.15.67:0/879561129 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32600cc480 con 0x7f326015dea0 2024-03-24T00:39:17.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.111+0000 7f3264c0b640 1 -- 172.21.15.67:0/879561129 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32600ccd20 con 0x7f326015dea0 2024-03-24T00:39:17.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.112+0000 7f3264c0b640 1 -- 172.21.15.67:0/879561129 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f326004f400 con 0x7f326015dea0 2024-03-24T00:39:17.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.112+0000 7f32577fe640 1 -- 172.21.15.67:0/879561129 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f324804c060 con 0x7f326015dea0 2024-03-24T00:39:17.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.113+0000 7f32577fe640 1 --2- 172.21.15.67:0/879561129 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f324006bbd0 0x7f324006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:17.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.113+0000 7f32577fe640 1 -- 172.21.15.67:0/879561129 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f32480ac550 con 0x7f326015dea0 2024-03-24T00:39:17.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.113+0000 7f325e59c640 1 --2- 172.21.15.67:0/879561129 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f324006bbd0 0x7f324006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:17.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.113+0000 7f325e59c640 1 --2- 172.21.15.67:0/879561129 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f324006bbd0 0x7f324006e090 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f32600cb030 tx=0x7f3250009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:17.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.117+0000 7f32577fe640 1 -- 172.21.15.67:0/879561129 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3248040070 con 0x7f326015dea0 2024-03-24T00:39:17.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.298+0000 7f3264c0b640 1 -- 172.21.15.67:0/879561129 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f32600b9bd0 con 0x7f326015dea0 2024-03-24T00:39:17.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.298+0000 7f32577fe640 1 -- 172.21.15.67:0/879561129 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7f324807a740 con 0x7f326015dea0 2024-03-24T00:39:17.299 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-24T00:39:17.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.300+0000 7f3264c0b640 1 -- 172.21.15.67:0/879561129 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f324006bbd0 msgr2=0x7f324006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:17.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.300+0000 7f3264c0b640 1 --2- 172.21.15.67:0/879561129 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f324006bbd0 0x7f324006e090 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f32600cb030 tx=0x7f3250009190 comp rx=0 tx=0).stop 2024-03-24T00:39:17.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.300+0000 7f3264c0b640 1 -- 172.21.15.67:0/879561129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f326015dea0 msgr2=0x7f32600ae430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:17.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.300+0000 7f3264c0b640 1 --2- 172.21.15.67:0/879561129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f326015dea0 0x7f32600ae430 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f324802b8c0 tx=0x7f32480046c0 comp rx=0 tx=0).stop 2024-03-24T00:39:17.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.300+0000 7f3264c0b640 1 -- 172.21.15.67:0/879561129 shutdown_connections 2024-03-24T00:39:17.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.300+0000 7f3264c0b640 1 --2- 172.21.15.67:0/879561129 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f324006bbd0 0x7f324006e090 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:17.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.300+0000 7f3264c0b640 1 --2- 172.21.15.67:0/879561129 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f326015e280 0x7f32600ab3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:17.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.300+0000 7f3264c0b640 1 --2- 172.21.15.67:0/879561129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f326015dea0 0x7f32600ae430 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:17.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.300+0000 7f3264c0b640 1 --2- 172.21.15.67:0/879561129 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32600b5a60 0x7f32600acad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:17.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.300+0000 7f3264c0b640 1 -- 172.21.15.67:0/879561129 >> 172.21.15.67:0/879561129 conn(0x7f3260072da0 msgr2=0x7f32600d31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:17.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.300+0000 7f3264c0b640 1 -- 172.21.15.67:0/879561129 shutdown_connections 2024-03-24T00:39:17.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.300+0000 7f3264c0b640 1 -- 172.21.15.67:0/879561129 wait complete. 2024-03-24T00:39:17.312 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:39:17.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.415+0000 7f7998b9f640 1 Processor -- start 2024-03-24T00:39:17.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.415+0000 7f7998b9f640 1 -- start start 2024-03-24T00:39:17.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.416+0000 7f7998b9f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f79940d3070 0x7f79940ce8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:17.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.416+0000 7f7998b9f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79940d1cf0 con 0x7f799415de50 2024-03-24T00:39:17.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.416+0000 7f7998b9f640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79940d1fd0 con 0x7f79940cee30 2024-03-24T00:39:17.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.416+0000 7f7998b9f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79940d22b0 con 0x7f79940d3070 2024-03-24T00:39:17.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.416+0000 7f7992ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f79940d3070 0x7f79940ce8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:17.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.416+0000 7f79937fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f799415de50 0x7f79940b9520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47876/0 (socket says 172.21.15.67:47876) 2024-03-24T00:39:17.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.416+0000 7f79937fe640 1 -- 172.21.15.67:0/1644505105 learned_addr learned my addr 172.21.15.67:0/1644505105 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:17.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.416+0000 7f7992ffd640 1 -- 172.21.15.67:0/1644505105 >> v1:172.21.15.72:6789/0 conn(0x7f79940cee30 legacy=0x7f79940b6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:17.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.416+0000 7f7992ffd640 1 -- 172.21.15.67:0/1644505105 >> v1:172.21.15.67:6789/0 conn(0x7f799415de50 legacy=0x7f79940b9520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:17.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.416+0000 7f79927fc640 1 -- 172.21.15.67:0/1644505105 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2117106238 0 0) 0x7f79940d1fd0 con 0x7f79940cee30 2024-03-24T00:39:17.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.416+0000 7f7992ffd640 1 -- 172.21.15.67:0/1644505105 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79940d2590 con 0x7f79940d3070 2024-03-24T00:39:17.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.417+0000 7f7992ffd640 1 --2- 172.21.15.67:0/1644505105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79940d3070 0x7f79940ce8f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f79880097a0 tx=0x7f798802b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=745a7a71eed17a8c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:17.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.417+0000 7f79927fc640 1 -- 172.21.15.67:0/1644505105 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7988038070 con 0x7f79940d3070 2024-03-24T00:39:17.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.417+0000 7f79927fc640 1 -- 172.21.15.67:0/1644505105 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7988031700 con 0x7f79940d3070 2024-03-24T00:39:17.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.417+0000 7f79927fc640 1 -- 172.21.15.67:0/1644505105 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7988031ad0 con 0x7f79940d3070 2024-03-24T00:39:17.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.418+0000 7f7998b9f640 1 -- 172.21.15.67:0/1644505105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79940d3070 msgr2=0x7f79940ce8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:17.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.418+0000 7f7998b9f640 1 --2- 172.21.15.67:0/1644505105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79940d3070 0x7f79940ce8f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f79880097a0 tx=0x7f798802b820 comp rx=0 tx=0).stop 2024-03-24T00:39:17.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.418+0000 7f7998b9f640 1 -- 172.21.15.67:0/1644505105 shutdown_connections 2024-03-24T00:39:17.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.418+0000 7f7998b9f640 1 --2- 172.21.15.67:0/1644505105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79940d3070 0x7f79940ce8f0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:17.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.418+0000 7f7998b9f640 1 -- 172.21.15.67:0/1644505105 >> 172.21.15.67:0/1644505105 conn(0x7f7994072b60 msgr2=0x7f7994072f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:17.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.418+0000 7f7998b9f640 1 -- 172.21.15.67:0/1644505105 shutdown_connections 2024-03-24T00:39:17.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.418+0000 7f7998b9f640 1 -- 172.21.15.67:0/1644505105 wait complete. 2024-03-24T00:39:17.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.419+0000 7f7998b9f640 1 Processor -- start 2024-03-24T00:39:17.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.419+0000 7f7998b9f640 1 -- start start 2024-03-24T00:39:17.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.420+0000 7f7998b9f640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f79940cee30 0x7f79941f40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:17.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.420+0000 7f7998b9f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79940d3070 0x7f79940c9fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:17.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.420+0000 7f7998b9f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f799415de50 0x7f79940ab5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:17.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.420+0000 7f7998b9f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79940d3840 con 0x7f799415de50 2024-03-24T00:39:17.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.420+0000 7f7998b9f640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79940d2b50 con 0x7f79940d3070 2024-03-24T00:39:17.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.420+0000 7f7998b9f640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f799415e230 con 0x7f79940cee30 2024-03-24T00:39:17.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.420+0000 7f7993fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f799415de50 0x7f79940ab5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:17.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.420+0000 7f7993fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f799415de50 0x7f79940ab5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36098/0 (socket says 172.21.15.67:36098) 2024-03-24T00:39:17.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.420+0000 7f7993fff640 1 -- 172.21.15.67:0/2536758054 learned_addr learned my addr 172.21.15.67:0/2536758054 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:17.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.420+0000 7f79937fe640 1 --2- 172.21.15.67:0/2536758054 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f79940cee30 0x7f79941f40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:17.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.420+0000 7f7992ffd640 1 --2- 172.21.15.67:0/2536758054 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79940d3070 0x7f79940c9fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:17.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.420+0000 7f7993fff640 1 -- 172.21.15.67:0/2536758054 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f79940cee30 msgr2=0x7f79941f40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:17.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.420+0000 7f7993fff640 1 --2- 172.21.15.67:0/2536758054 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f79940cee30 0x7f79941f40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:17.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.420+0000 7f7993fff640 1 -- 172.21.15.67:0/2536758054 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79940d3070 msgr2=0x7f79940c9fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:17.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.420+0000 7f7993fff640 1 --2- 172.21.15.67:0/2536758054 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79940d3070 0x7f79940c9fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:17.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.420+0000 7f7993fff640 1 -- 172.21.15.67:0/2536758054 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79940abe20 con 0x7f799415de50 2024-03-24T00:39:17.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.421+0000 7f7993fff640 1 --2- 172.21.15.67:0/2536758054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f799415de50 0x7f79940ab5d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f798c00c790 tx=0x7f798c00cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:17.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.421+0000 7f797bfff640 1 -- 172.21.15.67:0/2536758054 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f798c003da0 con 0x7f799415de50 2024-03-24T00:39:17.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.421+0000 7f797bfff640 1 -- 172.21.15.67:0/2536758054 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f798c005070 con 0x7f799415de50 2024-03-24T00:39:17.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.421+0000 7f7998b9f640 1 -- 172.21.15.67:0/2536758054 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79940ac100 con 0x7f799415de50 2024-03-24T00:39:17.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.421+0000 7f797bfff640 1 -- 172.21.15.67:0/2536758054 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f798c005460 con 0x7f799415de50 2024-03-24T00:39:17.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.421+0000 7f7998b9f640 1 -- 172.21.15.67:0/2536758054 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79940abb10 con 0x7f799415de50 2024-03-24T00:39:17.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.422+0000 7f7998b9f640 1 -- 172.21.15.67:0/2536758054 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f799404f400 con 0x7f799415de50 2024-03-24T00:39:17.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.422+0000 7f797bfff640 1 -- 172.21.15.67:0/2536758054 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f798c005600 con 0x7f799415de50 2024-03-24T00:39:17.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.423+0000 7f797bfff640 1 --2- 172.21.15.67:0/2536758054 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f798006bbd0 0x7f798006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:17.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.423+0000 7f797bfff640 1 -- 172.21.15.67:0/2536758054 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f798c08ab50 con 0x7f799415de50 2024-03-24T00:39:17.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.423+0000 7f79937fe640 1 --2- 172.21.15.67:0/2536758054 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f798006bbd0 0x7f798006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:17.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.423+0000 7f79937fe640 1 --2- 172.21.15.67:0/2536758054 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f798006bbd0 0x7f798006e090 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7984000f60 tx=0x7f7984007e20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:17.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.427+0000 7f797bfff640 1 -- 172.21.15.67:0/2536758054 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f798c058d40 con 0x7f799415de50 2024-03-24T00:39:17.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.609+0000 7f7998b9f640 1 -- 172.21.15.67:0/2536758054 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f79940b2a00 con 0x7f799415de50 2024-03-24T00:39:17.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.609+0000 7f797bfff640 1 -- 172.21.15.67:0/2536758054 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7f798c05cde0 con 0x7f799415de50 2024-03-24T00:39:17.610 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-03-24T00:39:17.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.611+0000 7f7998b9f640 1 -- 172.21.15.67:0/2536758054 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f798006bbd0 msgr2=0x7f798006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:17.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.611+0000 7f7998b9f640 1 --2- 172.21.15.67:0/2536758054 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f798006bbd0 0x7f798006e090 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7984000f60 tx=0x7f7984007e20 comp rx=0 tx=0).stop 2024-03-24T00:39:17.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.611+0000 7f7998b9f640 1 -- 172.21.15.67:0/2536758054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f799415de50 msgr2=0x7f79940ab5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:17.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.611+0000 7f7998b9f640 1 --2- 172.21.15.67:0/2536758054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f799415de50 0x7f79940ab5d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f798c00c790 tx=0x7f798c00cc50 comp rx=0 tx=0).stop 2024-03-24T00:39:17.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.611+0000 7f7998b9f640 1 -- 172.21.15.67:0/2536758054 shutdown_connections 2024-03-24T00:39:17.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.611+0000 7f7998b9f640 1 --2- 172.21.15.67:0/2536758054 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f798006bbd0 0x7f798006e090 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:17.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.611+0000 7f7998b9f640 1 --2- 172.21.15.67:0/2536758054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f799415de50 0x7f79940ab5d0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:17.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.611+0000 7f7998b9f640 1 --2- 172.21.15.67:0/2536758054 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79940d3070 0x7f79940c9fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:17.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.611+0000 7f7998b9f640 1 --2- 172.21.15.67:0/2536758054 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f79940cee30 0x7f79941f40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:17.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.611+0000 7f7998b9f640 1 -- 172.21.15.67:0/2536758054 >> 172.21.15.67:0/2536758054 conn(0x7f7994072b60 msgr2=0x7f799415e7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:17.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.611+0000 7f7998b9f640 1 -- 172.21.15.67:0/2536758054 shutdown_connections 2024-03-24T00:39:17.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.611+0000 7f7998b9f640 1 -- 172.21.15.67:0/2536758054 wait complete. 2024-03-24T00:39:17.623 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T00:39:17.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.726+0000 7f3f7d463640 1 Processor -- start 2024-03-24T00:39:17.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.726+0000 7f3f7d463640 1 -- start start 2024-03-24T00:39:17.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.727+0000 7f3f7d463640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3f78161370 0x7f3f78161ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:17.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.727+0000 7f3f7d463640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3f780cf100 0x7f3f780d2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:17.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.727+0000 7f3f7d463640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f78161ff0 con 0x7f3f780b9690 2024-03-24T00:39:17.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.727+0000 7f3f7d463640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f780b6e10 con 0x7f3f78161370 2024-03-24T00:39:17.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.727+0000 7f3f7d463640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f780d2190 con 0x7f3f780cf100 2024-03-24T00:39:17.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.727+0000 7f3f76ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3f78161370 0x7f3f78161ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:17.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.727+0000 7f3f767fc640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3f780b9690 0x7f3f780ce9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47890/0 (socket says 172.21.15.67:47890) 2024-03-24T00:39:17.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.727+0000 7f3f767fc640 1 -- 172.21.15.67:0/1354967031 learned_addr learned my addr 172.21.15.67:0/1354967031 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:17.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.727+0000 7f3f777fe640 1 --2- 172.21.15.67:0/1354967031 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3f780cf100 0x7f3f780d2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:17.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.727+0000 7f3f76ffd640 1 -- 172.21.15.67:0/1354967031 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3f780cf100 msgr2=0x7f3f780d2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:17.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.727+0000 7f3f76ffd640 1 --2- 172.21.15.67:0/1354967031 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3f780cf100 0x7f3f780d2620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:17.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.727+0000 7f3f76ffd640 1 -- 172.21.15.67:0/1354967031 >> v1:172.21.15.67:6789/0 conn(0x7f3f780b9690 legacy=0x7f3f780ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:17.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.727+0000 7f3f75ffb640 1 -- 172.21.15.67:0/1354967031 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4031657980 0 0) 0x7f3f78161ff0 con 0x7f3f780b9690 2024-03-24T00:39:17.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.727+0000 7f3f76ffd640 1 -- 172.21.15.67:0/1354967031 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f78162290 con 0x7f3f78161370 2024-03-24T00:39:17.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.728+0000 7f3f76ffd640 1 --2- 172.21.15.67:0/1354967031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f78161370 0x7f3f78161ab0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3f68009d90 tx=0x7f3f6802bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4df29b5c2da25325 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:17.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.728+0000 7f3f75ffb640 1 -- 172.21.15.67:0/1354967031 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f68039070 con 0x7f3f78161370 2024-03-24T00:39:17.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.728+0000 7f3f75ffb640 1 -- 172.21.15.67:0/1354967031 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f6802cce0 con 0x7f3f78161370 2024-03-24T00:39:17.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.728+0000 7f3f75ffb640 1 -- 172.21.15.67:0/1354967031 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f68034660 con 0x7f3f78161370 2024-03-24T00:39:17.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.729+0000 7f3f7d463640 1 -- 172.21.15.67:0/1354967031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f78161370 msgr2=0x7f3f78161ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:17.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.729+0000 7f3f7d463640 1 --2- 172.21.15.67:0/1354967031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f78161370 0x7f3f78161ab0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3f68009d90 tx=0x7f3f6802bf30 comp rx=0 tx=0).stop 2024-03-24T00:39:17.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.729+0000 7f3f7d463640 1 -- 172.21.15.67:0/1354967031 shutdown_connections 2024-03-24T00:39:17.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.729+0000 7f3f7d463640 1 --2- 172.21.15.67:0/1354967031 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3f780cf100 0x7f3f780d2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:17.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.729+0000 7f3f7d463640 1 --2- 172.21.15.67:0/1354967031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f78161370 0x7f3f78161ab0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:17.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.729+0000 7f3f7d463640 1 -- 172.21.15.67:0/1354967031 >> 172.21.15.67:0/1354967031 conn(0x7f3f7806c880 msgr2=0x7f3f7806cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:17.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.729+0000 7f3f7d463640 1 -- 172.21.15.67:0/1354967031 shutdown_connections 2024-03-24T00:39:17.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.729+0000 7f3f7d463640 1 -- 172.21.15.67:0/1354967031 wait complete. 2024-03-24T00:39:17.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.730+0000 7f3f7d463640 1 Processor -- start 2024-03-24T00:39:17.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.730+0000 7f3f7d463640 1 -- start start 2024-03-24T00:39:17.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.731+0000 7f3f7d463640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f780b9690 0x7f3f781521b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:17.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.731+0000 7f3f7d463640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3f780cf100 0x7f3f78151990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:17.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.731+0000 7f3f7d463640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f78161370 0x7f3f781e78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:17.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.731+0000 7f3f7d463640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f78161750 con 0x7f3f780b9690 2024-03-24T00:39:17.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.731+0000 7f3f7d463640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f78162850 con 0x7f3f78161370 2024-03-24T00:39:17.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.731+0000 7f3f7d463640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f780d2b60 con 0x7f3f780cf100 2024-03-24T00:39:17.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.731+0000 7f3f76ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f780b9690 0x7f3f781521b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:17.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.731+0000 7f3f76ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f780b9690 0x7f3f781521b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36116/0 (socket says 172.21.15.67:36116) 2024-03-24T00:39:17.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.731+0000 7f3f76ffd640 1 -- 172.21.15.67:0/1049626962 learned_addr learned my addr 172.21.15.67:0/1049626962 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:17.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.731+0000 7f3f767fc640 1 --2- 172.21.15.67:0/1049626962 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3f780cf100 0x7f3f78151990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:17.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.731+0000 7f3f777fe640 1 --2- 172.21.15.67:0/1049626962 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f78161370 0x7f3f781e78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:17.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.731+0000 7f3f76ffd640 1 -- 172.21.15.67:0/1049626962 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3f780cf100 msgr2=0x7f3f78151990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:17.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.731+0000 7f3f76ffd640 1 --2- 172.21.15.67:0/1049626962 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3f780cf100 0x7f3f78151990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:17.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.731+0000 7f3f76ffd640 1 -- 172.21.15.67:0/1049626962 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f78161370 msgr2=0x7f3f781e78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:17.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.731+0000 7f3f76ffd640 1 --2- 172.21.15.67:0/1049626962 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f78161370 0x7f3f781e78f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:17.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.731+0000 7f3f76ffd640 1 -- 172.21.15.67:0/1049626962 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f780cc8f0 con 0x7f3f780b9690 2024-03-24T00:39:17.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.731+0000 7f3f76ffd640 1 --2- 172.21.15.67:0/1049626962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f780b9690 0x7f3f781521b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3f68037040 tx=0x7f3f68004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:17.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.731+0000 7f3f6f7fe640 1 -- 172.21.15.67:0/1049626962 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f68039050 con 0x7f3f780b9690 2024-03-24T00:39:17.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.731+0000 7f3f6f7fe640 1 -- 172.21.15.67:0/1049626962 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f68045070 con 0x7f3f780b9690 2024-03-24T00:39:17.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.732+0000 7f3f7d463640 1 -- 172.21.15.67:0/1049626962 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f780ccb90 con 0x7f3f780b9690 2024-03-24T00:39:17.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.732+0000 7f3f6f7fe640 1 -- 172.21.15.67:0/1049626962 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f6803e830 con 0x7f3f780b9690 2024-03-24T00:39:17.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.732+0000 7f3f7d463640 1 -- 172.21.15.67:0/1049626962 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f780cd150 con 0x7f3f780b9690 2024-03-24T00:39:17.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.733+0000 7f3f7d463640 1 -- 172.21.15.67:0/1049626962 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f7804f400 con 0x7f3f780b9690 2024-03-24T00:39:17.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.733+0000 7f3f6f7fe640 1 -- 172.21.15.67:0/1049626962 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3f6803e9d0 con 0x7f3f780b9690 2024-03-24T00:39:17.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.733+0000 7f3f6f7fe640 1 --2- 172.21.15.67:0/1049626962 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3f5806bc00 0x7f3f5806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:17.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.734+0000 7f3f6f7fe640 1 -- 172.21.15.67:0/1049626962 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f3f680ad350 con 0x7f3f780b9690 2024-03-24T00:39:17.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.734+0000 7f3f767fc640 1 --2- 172.21.15.67:0/1049626962 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3f5806bc00 0x7f3f5806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:17.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.734+0000 7f3f767fc640 1 --2- 172.21.15.67:0/1049626962 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3f5806bc00 0x7f3f5806e0c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f3f78156260 tx=0x7f3f60009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:17.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.738+0000 7f3f6f7fe640 1 -- 172.21.15.67:0/1049626962 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3f6804e020 con 0x7f3f780b9690 2024-03-24T00:39:17.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.918+0000 7f3f7d463640 1 -- 172.21.15.67:0/1049626962 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f3f781528a0 con 0x7f3f780b9690 2024-03-24T00:39:17.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.918+0000 7f3f6f7fe640 1 -- 172.21.15.67:0/1049626962 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7f3f6807b4b0 con 0x7f3f780b9690 2024-03-24T00:39:17.919 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-24T00:39:17.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.920+0000 7f3f7d463640 1 -- 172.21.15.67:0/1049626962 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3f5806bc00 msgr2=0x7f3f5806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:17.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.920+0000 7f3f7d463640 1 --2- 172.21.15.67:0/1049626962 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3f5806bc00 0x7f3f5806e0c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f3f78156260 tx=0x7f3f60009040 comp rx=0 tx=0).stop 2024-03-24T00:39:17.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.920+0000 7f3f7d463640 1 -- 172.21.15.67:0/1049626962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f780b9690 msgr2=0x7f3f781521b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:17.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.920+0000 7f3f7d463640 1 --2- 172.21.15.67:0/1049626962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f780b9690 0x7f3f781521b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3f68037040 tx=0x7f3f68004a40 comp rx=0 tx=0).stop 2024-03-24T00:39:17.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.920+0000 7f3f7d463640 1 -- 172.21.15.67:0/1049626962 shutdown_connections 2024-03-24T00:39:17.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.920+0000 7f3f7d463640 1 --2- 172.21.15.67:0/1049626962 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3f5806bc00 0x7f3f5806e0c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:17.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.920+0000 7f3f7d463640 1 --2- 172.21.15.67:0/1049626962 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f78161370 0x7f3f781e78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:17.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.920+0000 7f3f7d463640 1 --2- 172.21.15.67:0/1049626962 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3f780cf100 0x7f3f78151990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:17.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.920+0000 7f3f7d463640 1 --2- 172.21.15.67:0/1049626962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f780b9690 0x7f3f781521b0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:17.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.920+0000 7f3f7d463640 1 -- 172.21.15.67:0/1049626962 >> 172.21.15.67:0/1049626962 conn(0x7f3f7806c880 msgr2=0x7f3f780c7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:17.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.921+0000 7f3f7d463640 1 -- 172.21.15.67:0/1049626962 shutdown_connections 2024-03-24T00:39:17.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:17.921+0000 7f3f7d463640 1 -- 172.21.15.67:0/1049626962 wait complete. 2024-03-24T00:39:17.932 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:39:18.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.033+0000 7f7ca497e640 1 Processor -- start 2024-03-24T00:39:18.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.033+0000 7f7ca497e640 1 -- start start 2024-03-24T00:39:18.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.033+0000 7f7ca497e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7ca015f410 0x7f7ca00cf7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:18.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.033+0000 7f7ca497e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7ca00d0940 0x7f7ca00cfd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:18.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.033+0000 7f7ca497e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ca00cdb50 con 0x7f7ca0099d70 2024-03-24T00:39:18.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.034+0000 7f7ca497e640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ca00cd2b0 con 0x7f7ca00d0940 2024-03-24T00:39:18.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.034+0000 7f7ca497e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ca00cd590 con 0x7f7ca015f410 2024-03-24T00:39:18.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.034+0000 7f7c95d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7ca00d0940 0x7f7ca00cfd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:18.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.034+0000 7f7c9ed9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7ca0099d70 0x7f7ca015fb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47892/0 (socket says 172.21.15.67:47892) 2024-03-24T00:39:18.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.034+0000 7f7c9ed9d640 1 -- 172.21.15.67:0/1042794670 learned_addr learned my addr 172.21.15.67:0/1042794670 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:18.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.034+0000 7f7c9e59c640 1 --2- 172.21.15.67:0/1042794670 >> v2:172.21.15.72:3300/0 conn(0x7f7ca015f410 0x7f7ca00cf7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:18.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.034+0000 7f7c95d9b640 1 -- 172.21.15.67:0/1042794670 >> v2:172.21.15.72:3300/0 conn(0x7f7ca015f410 msgr2=0x7f7ca00cf7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:18.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.034+0000 7f7c95d9b640 1 --2- 172.21.15.67:0/1042794670 >> v2:172.21.15.72:3300/0 conn(0x7f7ca015f410 0x7f7ca00cf7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:18.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.034+0000 7f7c95d9b640 1 -- 172.21.15.67:0/1042794670 >> v1:172.21.15.67:6789/0 conn(0x7f7ca0099d70 legacy=0x7f7ca015fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:18.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.034+0000 7f7c95d9b640 1 -- 172.21.15.67:0/1042794670 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ca00cd870 con 0x7f7ca00d0940 2024-03-24T00:39:18.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.034+0000 7f7c9e59c640 1 --2- 172.21.15.67:0/1042794670 >> v2:172.21.15.72:3300/0 conn(0x7f7ca015f410 0x7f7ca00cf7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:18.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.034+0000 7f7c9dd9b640 1 -- 172.21.15.67:0/1042794670 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1643815841 0 0) 0x7f7ca00cdb50 con 0x7f7ca0099d70 2024-03-24T00:39:18.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.035+0000 7f7c95d9b640 1 --2- 172.21.15.67:0/1042794670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ca00d0940 0x7f7ca00cfd00 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7ca00733e0 tx=0x7f7c8802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bafe9d0a57f89189 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:18.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.035+0000 7f7c9dd9b640 1 -- 172.21.15.67:0/1042794670 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c88035020 con 0x7f7ca00d0940 2024-03-24T00:39:18.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.035+0000 7f7c9dd9b640 1 -- 172.21.15.67:0/1042794670 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c88031830 con 0x7f7ca00d0940 2024-03-24T00:39:18.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.035+0000 7f7c9dd9b640 1 -- 172.21.15.67:0/1042794670 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c88031c00 con 0x7f7ca00d0940 2024-03-24T00:39:18.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.035+0000 7f7ca497e640 1 -- 172.21.15.67:0/1042794670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ca00d0940 msgr2=0x7f7ca00cfd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:18.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.035+0000 7f7ca497e640 1 --2- 172.21.15.67:0/1042794670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ca00d0940 0x7f7ca00cfd00 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7ca00733e0 tx=0x7f7c8802b8d0 comp rx=0 tx=0).stop 2024-03-24T00:39:18.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.035+0000 7f7ca497e640 1 -- 172.21.15.67:0/1042794670 shutdown_connections 2024-03-24T00:39:18.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.035+0000 7f7ca497e640 1 --2- 172.21.15.67:0/1042794670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ca00d0940 0x7f7ca00cfd00 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:18.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.035+0000 7f7ca497e640 1 --2- 172.21.15.67:0/1042794670 >> v2:172.21.15.72:3300/0 conn(0x7f7ca015f410 0x7f7ca00cf7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:18.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.035+0000 7f7ca497e640 1 -- 172.21.15.67:0/1042794670 >> 172.21.15.67:0/1042794670 conn(0x7f7ca00b36e0 msgr2=0x7f7ca00b3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:18.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.035+0000 7f7ca497e640 1 -- 172.21.15.67:0/1042794670 shutdown_connections 2024-03-24T00:39:18.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.035+0000 7f7ca497e640 1 -- 172.21.15.67:0/1042794670 wait complete. 2024-03-24T00:39:18.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.037+0000 7f7ca497e640 1 Processor -- start 2024-03-24T00:39:18.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.037+0000 7f7ca497e640 1 -- start start 2024-03-24T00:39:18.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.037+0000 7f7ca497e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ca0099d70 0x7f7ca00b0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:18.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.038+0000 7f7ca497e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ca015f410 0x7f7ca014e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:18.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.038+0000 7f7ca497e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7ca006c150 0x7f7ca00c5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:18.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.038+0000 7f7ca497e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca015f7f0 con 0x7f7ca015f410 2024-03-24T00:39:18.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.038+0000 7f7ca497e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca00d0240 con 0x7f7ca0099d70 2024-03-24T00:39:18.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.038+0000 7f7ca497e640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca00b07a0 con 0x7f7ca006c150 2024-03-24T00:39:18.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.038+0000 7f7c95d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ca015f410 0x7f7ca014e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:18.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.038+0000 7f7c95d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ca015f410 0x7f7ca014e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36134/0 (socket says 172.21.15.67:36134) 2024-03-24T00:39:18.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.038+0000 7f7c95d9b640 1 -- 172.21.15.67:0/3931155090 learned_addr learned my addr 172.21.15.67:0/3931155090 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:18.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.038+0000 7f7c9e59c640 1 --2- 172.21.15.67:0/3931155090 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ca0099d70 0x7f7ca00b0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:18.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.038+0000 7f7c9ed9d640 1 --2- 172.21.15.67:0/3931155090 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7ca006c150 0x7f7ca00c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:18.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.038+0000 7f7c95d9b640 1 -- 172.21.15.67:0/3931155090 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7ca006c150 msgr2=0x7f7ca00c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:18.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.038+0000 7f7c95d9b640 1 --2- 172.21.15.67:0/3931155090 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7ca006c150 0x7f7ca00c5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:18.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.038+0000 7f7c95d9b640 1 -- 172.21.15.67:0/3931155090 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ca0099d70 msgr2=0x7f7ca00b0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:18.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.038+0000 7f7c95d9b640 1 --2- 172.21.15.67:0/3931155090 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ca0099d70 0x7f7ca00b0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:18.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.038+0000 7f7c95d9b640 1 -- 172.21.15.67:0/3931155090 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ca006c6f0 con 0x7f7ca015f410 2024-03-24T00:39:18.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.038+0000 7f7c95d9b640 1 --2- 172.21.15.67:0/3931155090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ca015f410 0x7f7ca014e050 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7c88009890 tx=0x7f7c880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:18.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.038+0000 7f7c977fe640 1 -- 172.21.15.67:0/3931155090 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c88041070 con 0x7f7ca015f410 2024-03-24T00:39:18.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.038+0000 7f7c977fe640 1 -- 172.21.15.67:0/3931155090 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c8803b920 con 0x7f7ca015f410 2024-03-24T00:39:18.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.038+0000 7f7ca497e640 1 -- 172.21.15.67:0/3931155090 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ca006c9d0 con 0x7f7ca015f410 2024-03-24T00:39:18.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.038+0000 7f7c977fe640 1 -- 172.21.15.67:0/3931155090 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c88035040 con 0x7f7ca015f410 2024-03-24T00:39:18.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.038+0000 7f7ca497e640 1 -- 172.21.15.67:0/3931155090 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ca006cf90 con 0x7f7ca015f410 2024-03-24T00:39:18.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.039+0000 7f7ca497e640 1 -- 172.21.15.67:0/3931155090 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ca004f400 con 0x7f7ca015f410 2024-03-24T00:39:18.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.040+0000 7f7c977fe640 1 -- 172.21.15.67:0/3931155090 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7c88038020 con 0x7f7ca015f410 2024-03-24T00:39:18.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.040+0000 7f7c977fe640 1 --2- 172.21.15.67:0/3931155090 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7c8006bc00 0x7f7c8006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:18.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.040+0000 7f7c977fe640 1 -- 172.21.15.67:0/3931155090 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f7c880ac4c0 con 0x7f7ca015f410 2024-03-24T00:39:18.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.040+0000 7f7c9e59c640 1 --2- 172.21.15.67:0/3931155090 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7c8006bc00 0x7f7c8006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:18.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.041+0000 7f7c9e59c640 1 --2- 172.21.15.67:0/3931155090 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7c8006bc00 0x7f7c8006e0c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7c90009560 tx=0x7f7c90009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:18.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.045+0000 7f7c977fe640 1 -- 172.21.15.67:0/3931155090 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7c88031800 con 0x7f7ca015f410 2024-03-24T00:39:18.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.266+0000 7f7ca497e640 1 -- 172.21.15.67:0/3931155090 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ca00d19d0 con 0x7f7ca015f410 2024-03-24T00:39:18.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.266+0000 7f7c977fe640 1 -- 172.21.15.67:0/3931155090 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4970 (secure 0 0 0) 0x7f7c8807a6b0 con 0x7f7ca015f410 2024-03-24T00:39:18.267 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:39:18.268 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4263,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/837528243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":837528243},{"type":"v1","addr":"172.21.15.67:6834","nonce":837528243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3987071456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":3987071456},{"type":"v1","addr":"172.21.15.67:6835","nonce":3987071456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4306,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/3953492281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3953492281},{"type":"v1","addr":"172.21.15.72:6837","nonce":3953492281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:38:52.127871+0000","modified":"2024-03-24T00:39:08.187262+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4309},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4309":{"gid":4309,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6836/3737698654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3737698654},{"type":"v1","addr":"172.21.15.72:6836","nonce":3737698654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4309,"qdb_cluster":[4309]},"id":1}]} 2024-03-24T00:39:18.268 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 9 2024-03-24T00:39:18.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.268+0000 7f7ca497e640 1 -- 172.21.15.67:0/3931155090 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7c8006bc00 msgr2=0x7f7c8006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:18.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.268+0000 7f7ca497e640 1 --2- 172.21.15.67:0/3931155090 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7c8006bc00 0x7f7c8006e0c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7c90009560 tx=0x7f7c90009190 comp rx=0 tx=0).stop 2024-03-24T00:39:18.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.268+0000 7f7ca497e640 1 -- 172.21.15.67:0/3931155090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ca015f410 msgr2=0x7f7ca014e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:18.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.268+0000 7f7ca497e640 1 --2- 172.21.15.67:0/3931155090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ca015f410 0x7f7ca014e050 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7c88009890 tx=0x7f7c880046c0 comp rx=0 tx=0).stop 2024-03-24T00:39:18.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.268+0000 7f7ca497e640 1 -- 172.21.15.67:0/3931155090 shutdown_connections 2024-03-24T00:39:18.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.268+0000 7f7ca497e640 1 --2- 172.21.15.67:0/3931155090 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7c8006bc00 0x7f7c8006e0c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:18.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.268+0000 7f7ca497e640 1 --2- 172.21.15.67:0/3931155090 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7ca006c150 0x7f7ca00c5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:18.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.268+0000 7f7ca497e640 1 --2- 172.21.15.67:0/3931155090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ca015f410 0x7f7ca014e050 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:18.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.268+0000 7f7ca497e640 1 --2- 172.21.15.67:0/3931155090 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ca0099d70 0x7f7ca00b0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:18.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.268+0000 7f7ca497e640 1 -- 172.21.15.67:0/3931155090 >> 172.21.15.67:0/3931155090 conn(0x7f7ca00b36e0 msgr2=0x7f7ca009a850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:18.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.268+0000 7f7ca497e640 1 -- 172.21.15.67:0/3931155090 shutdown_connections 2024-03-24T00:39:18.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.268+0000 7f7ca497e640 1 -- 172.21.15.67:0/3931155090 wait complete. 2024-03-24T00:39:18.280 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:39:18.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.381+0000 7f3a088dc640 1 Processor -- start 2024-03-24T00:39:18.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.381+0000 7f3a088dc640 1 -- start start 2024-03-24T00:39:18.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.382+0000 7f3a088dc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3a040d1550 0x7f3a0406cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:18.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.382+0000 7f3a088dc640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3a040b2780 0x7f3a040b2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:18.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.382+0000 7f3a088dc640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a040d1930 con 0x7f3a04099d70 2024-03-24T00:39:18.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.382+0000 7f3a088dc640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a0406d420 con 0x7f3a040d1550 2024-03-24T00:39:18.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.382+0000 7f3a088dc640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a0406ca50 con 0x7f3a040b2780 2024-03-24T00:39:18.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.382+0000 7f3a02d9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3a04099d70 0x7f3a040cfd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47906/0 (socket says 172.21.15.67:47906) 2024-03-24T00:39:18.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.382+0000 7f3a0259c640 1 --2- 172.21.15.67:0/794064270 >> v2:172.21.15.72:3300/0 conn(0x7f3a040d1550 0x7f3a0406cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:18.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.382+0000 7f3a02d9d640 1 -- 172.21.15.67:0/794064270 learned_addr learned my addr 172.21.15.67:0/794064270 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:18.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.382+0000 7f3a01d9b640 1 --2- 172.21.15.67:0/794064270 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3a040b2780 0x7f3a040b2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:18.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.382+0000 7f3a0259c640 1 -- 172.21.15.67:0/794064270 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3a040b2780 msgr2=0x7f3a040b2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:18.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.382+0000 7f3a0259c640 1 --2- 172.21.15.67:0/794064270 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3a040b2780 0x7f3a040b2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:18.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.382+0000 7f3a0159a640 1 -- 172.21.15.67:0/794064270 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2130665651 0 0) 0x7f3a040d1930 con 0x7f3a04099d70 2024-03-24T00:39:18.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.382+0000 7f3a0259c640 1 -- 172.21.15.67:0/794064270 >> v1:172.21.15.67:6789/0 conn(0x7f3a04099d70 legacy=0x7f3a040cfd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:18.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.383+0000 7f3a0259c640 1 -- 172.21.15.67:0/794064270 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a040b1f60 con 0x7f3a040d1550 2024-03-24T00:39:18.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.383+0000 7f3a0259c640 1 --2- 172.21.15.67:0/794064270 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a040d1550 0x7f3a0406cee0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f39fc0097a0 tx=0x7f39fc02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71ef23a64e63d5e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:18.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.383+0000 7f3a0159a640 1 -- 172.21.15.67:0/794064270 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39fc035020 con 0x7f3a040d1550 2024-03-24T00:39:18.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.383+0000 7f3a0159a640 1 -- 172.21.15.67:0/794064270 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39fc031830 con 0x7f3a040d1550 2024-03-24T00:39:18.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.383+0000 7f3a0159a640 1 -- 172.21.15.67:0/794064270 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39fc031c00 con 0x7f3a040d1550 2024-03-24T00:39:18.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.384+0000 7f3a088dc640 1 -- 172.21.15.67:0/794064270 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a040d1550 msgr2=0x7f3a0406cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:18.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.384+0000 7f3a088dc640 1 --2- 172.21.15.67:0/794064270 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a040d1550 0x7f3a0406cee0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f39fc0097a0 tx=0x7f39fc02b8d0 comp rx=0 tx=0).stop 2024-03-24T00:39:18.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.384+0000 7f3a088dc640 1 -- 172.21.15.67:0/794064270 shutdown_connections 2024-03-24T00:39:18.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.384+0000 7f3a088dc640 1 --2- 172.21.15.67:0/794064270 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3a040b2780 0x7f3a040b2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:18.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.384+0000 7f3a088dc640 1 --2- 172.21.15.67:0/794064270 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a040d1550 0x7f3a0406cee0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:18.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.384+0000 7f3a088dc640 1 -- 172.21.15.67:0/794064270 >> 172.21.15.67:0/794064270 conn(0x7f3a040b36e0 msgr2=0x7f3a040b3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:18.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.384+0000 7f3a088dc640 1 -- 172.21.15.67:0/794064270 shutdown_connections 2024-03-24T00:39:18.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.384+0000 7f3a088dc640 1 -- 172.21.15.67:0/794064270 wait complete. 2024-03-24T00:39:18.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.385+0000 7f3a088dc640 1 Processor -- start 2024-03-24T00:39:18.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.385+0000 7f3a088dc640 1 -- start start 2024-03-24T00:39:18.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.386+0000 7f3a088dc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a04099d70 0x7f3a041e8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:18.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.386+0000 7f3a088dc640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3a040b2780 0x7f3a0414d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:18.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.386+0000 7f3a088dc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a040d1550 0x7f3a041504b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:18.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.386+0000 7f3a088dc640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a0406c400 con 0x7f3a04099d70 2024-03-24T00:39:18.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.386+0000 7f3a088dc640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a040b2520 con 0x7f3a040d1550 2024-03-24T00:39:18.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.386+0000 7f3a088dc640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a0406c6e0 con 0x7f3a040b2780 2024-03-24T00:39:18.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.386+0000 7f3a0259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a04099d70 0x7f3a041e8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:18.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.386+0000 7f3a0259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a04099d70 0x7f3a041e8bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36138/0 (socket says 172.21.15.67:36138) 2024-03-24T00:39:18.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.386+0000 7f3a0259c640 1 -- 172.21.15.67:0/371861418 learned_addr learned my addr 172.21.15.67:0/371861418 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:18.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.386+0000 7f3a01d9b640 1 --2- 172.21.15.67:0/371861418 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3a040b2780 0x7f3a0414d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:18.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.386+0000 7f3a02d9d640 1 --2- 172.21.15.67:0/371861418 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a040d1550 0x7f3a041504b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:18.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.386+0000 7f3a0259c640 1 -- 172.21.15.67:0/371861418 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3a040b2780 msgr2=0x7f3a0414d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:18.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.386+0000 7f3a0259c640 1 --2- 172.21.15.67:0/371861418 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3a040b2780 0x7f3a0414d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:18.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.386+0000 7f3a0259c640 1 -- 172.21.15.67:0/371861418 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a040d1550 msgr2=0x7f3a041504b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:18.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.386+0000 7f3a0259c640 1 --2- 172.21.15.67:0/371861418 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a040d1550 0x7f3a041504b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:18.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.386+0000 7f3a0259c640 1 -- 172.21.15.67:0/371861418 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a0414d7a0 con 0x7f3a04099d70 2024-03-24T00:39:18.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.387+0000 7f3a0259c640 1 --2- 172.21.15.67:0/371861418 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a04099d70 0x7f3a041e8bc0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f39fc000c00 tx=0x7f39fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:18.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.387+0000 7f39faffd640 1 -- 172.21.15.67:0/371861418 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39fc03a970 con 0x7f3a04099d70 2024-03-24T00:39:18.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.387+0000 7f39faffd640 1 -- 172.21.15.67:0/371861418 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39fc03ab50 con 0x7f3a04099d70 2024-03-24T00:39:18.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.387+0000 7f3a088dc640 1 -- 172.21.15.67:0/371861418 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a041e91a0 con 0x7f3a04099d70 2024-03-24T00:39:18.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.387+0000 7f39faffd640 1 -- 172.21.15.67:0/371861418 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39fc0443f0 con 0x7f3a04099d70 2024-03-24T00:39:18.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.387+0000 7f3a088dc640 1 -- 172.21.15.67:0/371861418 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a041e9760 con 0x7f3a04099d70 2024-03-24T00:39:18.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.388+0000 7f3a088dc640 1 -- 172.21.15.67:0/371861418 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a040c9ab0 con 0x7f3a04099d70 2024-03-24T00:39:18.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.388+0000 7f39faffd640 1 -- 172.21.15.67:0/371861418 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f39fc04b060 con 0x7f3a04099d70 2024-03-24T00:39:18.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.389+0000 7f39faffd640 1 --2- 172.21.15.67:0/371861418 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f39e406bbd0 0x7f39e406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:18.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.389+0000 7f39faffd640 1 -- 172.21.15.67:0/371861418 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f39fc0ac7b0 con 0x7f3a04099d70 2024-03-24T00:39:18.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.389+0000 7f3a01d9b640 1 --2- 172.21.15.67:0/371861418 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f39e406bbd0 0x7f39e406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:18.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.389+0000 7f3a01d9b640 1 --2- 172.21.15.67:0/371861418 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f39e406bbd0 0x7f39e406e090 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3a040b53c0 tx=0x7f39ec009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:18.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.393+0000 7f39faffd640 1 -- 172.21.15.67:0/371861418 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f39fc03f070 con 0x7f3a04099d70 2024-03-24T00:39:18.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.576+0000 7f3a088dc640 1 -- 172.21.15.67:0/371861418 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3a040b1310 con 0x7f3a04099d70 2024-03-24T00:39:18.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.577+0000 7f39faffd640 1 -- 172.21.15.67:0/371861418 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18454 (secure 0 0 0) 0x7f39fc031800 con 0x7f3a04099d70 2024-03-24T00:39:18.578 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:39:18.578 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":25,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:38:58.797179+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:38:50.077085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:38:51.095454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T00:38:54.807117+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:39:18.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.579+0000 7f3a088dc640 1 -- 172.21.15.67:0/371861418 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f39e406bbd0 msgr2=0x7f39e406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:18.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.579+0000 7f3a088dc640 1 --2- 172.21.15.67:0/371861418 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f39e406bbd0 0x7f39e406e090 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3a040b53c0 tx=0x7f39ec009190 comp rx=0 tx=0).stop 2024-03-24T00:39:18.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.579+0000 7f3a088dc640 1 -- 172.21.15.67:0/371861418 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a04099d70 msgr2=0x7f3a041e8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:18.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.579+0000 7f3a088dc640 1 --2- 172.21.15.67:0/371861418 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a04099d70 0x7f3a041e8bc0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f39fc000c00 tx=0x7f39fc0046c0 comp rx=0 tx=0).stop 2024-03-24T00:39:18.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.579+0000 7f3a088dc640 1 -- 172.21.15.67:0/371861418 shutdown_connections 2024-03-24T00:39:18.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.579+0000 7f3a088dc640 1 --2- 172.21.15.67:0/371861418 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f39e406bbd0 0x7f39e406e090 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:18.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.579+0000 7f3a088dc640 1 --2- 172.21.15.67:0/371861418 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a040d1550 0x7f3a041504b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:18.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.579+0000 7f3a088dc640 1 --2- 172.21.15.67:0/371861418 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3a040b2780 0x7f3a0414d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:18.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.579+0000 7f3a088dc640 1 --2- 172.21.15.67:0/371861418 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a04099d70 0x7f3a041e8bc0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:18.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.579+0000 7f3a088dc640 1 -- 172.21.15.67:0/371861418 >> 172.21.15.67:0/371861418 conn(0x7f3a040b36e0 msgr2=0x7f3a04161d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:18.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.579+0000 7f3a088dc640 1 -- 172.21.15.67:0/371861418 shutdown_connections 2024-03-24T00:39:18.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.579+0000 7f3a088dc640 1 -- 172.21.15.67:0/371861418 wait complete. 2024-03-24T00:39:18.591 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:38:39.364950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:38:50.077085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:38:51.095454+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T00:38:54.807117+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T00:39:18.591 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:39:18.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.695+0000 7f4a16d5b640 1 Processor -- start 2024-03-24T00:39:18.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.695+0000 7f4a16d5b640 1 -- start start 2024-03-24T00:39:18.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.696+0000 7f4a16d5b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4a10099d00 0x7f4a1009a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:18.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.696+0000 7f4a16d5b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4a100d1530 0x7f4a1015d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:18.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.696+0000 7f4a16d5b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a1009a620 con 0x7f4a100b49b0 2024-03-24T00:39:18.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.696+0000 7f4a16d5b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a100cc1b0 con 0x7f4a10099d00 2024-03-24T00:39:18.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.696+0000 7f4a16d5b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a100cc400 con 0x7f4a100d1530 2024-03-24T00:39:18.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.696+0000 7f4a14af7640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4a10099d00 0x7f4a1009a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:18.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.696+0000 7f4a14af7640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4a10099d00 0x7f4a1009a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36142/0 (socket says 172.21.15.67:36142) 2024-03-24T00:39:18.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.696+0000 7f4a14af7640 1 -- 172.21.15.67:0/569879658 learned_addr learned my addr 172.21.15.67:0/569879658 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:18.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.696+0000 7f4a0ffff640 1 --1- 172.21.15.67:0/569879658 >> v1:172.21.15.72:6789/0 conn(0x7f4a100b49b0 0x7f4a100d0e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:49988/0 (socket says 172.21.15.67:49988) 2024-03-24T00:39:18.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.696+0000 7f4a152f8640 1 --2- 172.21.15.67:0/569879658 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4a100d1530 0x7f4a1015d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:18.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.696+0000 7f4a14af7640 1 -- 172.21.15.67:0/569879658 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4a100d1530 msgr2=0x7f4a1015d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:18.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.696+0000 7f4a14af7640 1 --2- 172.21.15.67:0/569879658 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4a100d1530 0x7f4a1015d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:18.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.696+0000 7f4a14af7640 1 -- 172.21.15.67:0/569879658 >> v1:172.21.15.72:6789/0 conn(0x7f4a100b49b0 legacy=0x7f4a100d0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:18.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.696+0000 7f4a14af7640 1 -- 172.21.15.67:0/569879658 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a100cc6e0 con 0x7f4a10099d00 2024-03-24T00:39:18.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.697+0000 7f4a14af7640 1 --2- 172.21.15.67:0/569879658 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a10099d00 0x7f4a1009a0e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4a08009e50 tx=0x7f4a0802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96fafb5d932fe7f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:18.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.697+0000 7f4a0f7fe640 1 -- 172.21.15.67:0/569879658 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a0802cb00 con 0x7f4a10099d00 2024-03-24T00:39:18.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.697+0000 7f4a0f7fe640 1 -- 172.21.15.67:0/569879658 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a0802cca0 con 0x7f4a10099d00 2024-03-24T00:39:18.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.697+0000 7f4a0f7fe640 1 -- 172.21.15.67:0/569879658 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a08034610 con 0x7f4a10099d00 2024-03-24T00:39:18.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.698+0000 7f4a16d5b640 1 -- 172.21.15.67:0/569879658 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a10099d00 msgr2=0x7f4a1009a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:18.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.698+0000 7f4a16d5b640 1 --2- 172.21.15.67:0/569879658 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a10099d00 0x7f4a1009a0e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4a08009e50 tx=0x7f4a0802bf60 comp rx=0 tx=0).stop 2024-03-24T00:39:18.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.698+0000 7f4a16d5b640 1 -- 172.21.15.67:0/569879658 shutdown_connections 2024-03-24T00:39:18.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.698+0000 7f4a16d5b640 1 --2- 172.21.15.67:0/569879658 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4a100d1530 0x7f4a1015d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:18.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.698+0000 7f4a16d5b640 1 --2- 172.21.15.67:0/569879658 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a10099d00 0x7f4a1009a0e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:18.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.698+0000 7f4a16d5b640 1 -- 172.21.15.67:0/569879658 >> 172.21.15.67:0/569879658 conn(0x7f4a100b3650 msgr2=0x7f4a100b3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:18.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.698+0000 7f4a16d5b640 1 -- 172.21.15.67:0/569879658 shutdown_connections 2024-03-24T00:39:18.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.698+0000 7f4a16d5b640 1 -- 172.21.15.67:0/569879658 wait complete. 2024-03-24T00:39:18.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.699+0000 7f4a16d5b640 1 Processor -- start 2024-03-24T00:39:18.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.699+0000 7f4a16d5b640 1 -- start start 2024-03-24T00:39:18.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.700+0000 7f4a16d5b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4a10099d00 0x7f4a10156d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:18.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.700+0000 7f4a16d5b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a100b49b0 0x7f4a100c55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:18.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.700+0000 7f4a16d5b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a100c9640 0x7f4a100cb220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:18.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.700+0000 7f4a16d5b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a100d01c0 con 0x7f4a100c9640 2024-03-24T00:39:18.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.700+0000 7f4a16d5b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a100ccca0 con 0x7f4a100b49b0 2024-03-24T00:39:18.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.700+0000 7f4a16d5b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a100b32a0 con 0x7f4a10099d00 2024-03-24T00:39:18.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.700+0000 7f4a152f8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a100c9640 0x7f4a100cb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:18.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.700+0000 7f4a152f8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a100c9640 0x7f4a100cb220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36158/0 (socket says 172.21.15.67:36158) 2024-03-24T00:39:18.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.700+0000 7f4a152f8640 1 -- 172.21.15.67:0/2328462194 learned_addr learned my addr 172.21.15.67:0/2328462194 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:18.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.700+0000 7f4a14af7640 1 --2- 172.21.15.67:0/2328462194 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4a10099d00 0x7f4a10156d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:18.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.700+0000 7f4a0ffff640 1 --2- 172.21.15.67:0/2328462194 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a100b49b0 0x7f4a100c55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:18.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.700+0000 7f4a152f8640 1 -- 172.21.15.67:0/2328462194 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4a10099d00 msgr2=0x7f4a10156d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:18.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.700+0000 7f4a152f8640 1 --2- 172.21.15.67:0/2328462194 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4a10099d00 0x7f4a10156d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:18.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.700+0000 7f4a152f8640 1 -- 172.21.15.67:0/2328462194 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a100b49b0 msgr2=0x7f4a100c55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:18.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.700+0000 7f4a152f8640 1 --2- 172.21.15.67:0/2328462194 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a100b49b0 0x7f4a100c55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:18.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.700+0000 7f4a152f8640 1 -- 172.21.15.67:0/2328462194 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a100c9b90 con 0x7f4a100c9640 2024-03-24T00:39:18.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.700+0000 7f4a152f8640 1 --2- 172.21.15.67:0/2328462194 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a100c9640 0x7f4a100cb220 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4a0000e7f0 tx=0x7f4a0000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:18.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.701+0000 7f4a0d7fa640 1 -- 172.21.15.67:0/2328462194 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a00017070 con 0x7f4a100c9640 2024-03-24T00:39:18.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.701+0000 7f4a0d7fa640 1 -- 172.21.15.67:0/2328462194 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a0000c6d0 con 0x7f4a100c9640 2024-03-24T00:39:18.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.701+0000 7f4a0d7fa640 1 -- 172.21.15.67:0/2328462194 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a0000caa0 con 0x7f4a100c9640 2024-03-24T00:39:18.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.701+0000 7f4a16d5b640 1 -- 172.21.15.67:0/2328462194 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a100c9e70 con 0x7f4a100c9640 2024-03-24T00:39:18.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.701+0000 7f4a16d5b640 1 -- 172.21.15.67:0/2328462194 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a100caa20 con 0x7f4a100c9640 2024-03-24T00:39:18.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.702+0000 7f4a16d5b640 1 -- 172.21.15.67:0/2328462194 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a1009a0e0 con 0x7f4a100c9640 2024-03-24T00:39:18.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.702+0000 7f4a0d7fa640 1 -- 172.21.15.67:0/2328462194 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4a0001a070 con 0x7f4a100c9640 2024-03-24T00:39:18.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.702+0000 7f4a0d7fa640 1 --2- 172.21.15.67:0/2328462194 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f49fc06bc00 0x7f49fc06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:18.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.703+0000 7f4a0d7fa640 1 -- 172.21.15.67:0/2328462194 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f4a0008ad90 con 0x7f4a100c9640 2024-03-24T00:39:18.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.703+0000 7f4a14af7640 1 --2- 172.21.15.67:0/2328462194 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f49fc06bc00 0x7f49fc06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:18.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.703+0000 7f4a14af7640 1 --2- 172.21.15.67:0/2328462194 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f49fc06bc00 0x7f49fc06e0c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4a08035000 tx=0x7f4a08030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:18.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.707+0000 7f4a0d7fa640 1 -- 172.21.15.67:0/2328462194 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4a1009a0e0 con 0x7f4a100c9640 2024-03-24T00:39:18.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.892+0000 7f4a16d5b640 1 -- 172.21.15.67:0/2328462194 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4a1004f400 con 0x7f4a100c9640 2024-03-24T00:39:18.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.892+0000 7f4a0d7fa640 1 -- 172.21.15.67:0/2328462194 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4a00058f80 con 0x7f4a100c9640 2024-03-24T00:39:18.893 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-24T00:39:18.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.895+0000 7f4a16d5b640 1 -- 172.21.15.67:0/2328462194 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f49fc06bc00 msgr2=0x7f49fc06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:18.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.895+0000 7f4a16d5b640 1 --2- 172.21.15.67:0/2328462194 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f49fc06bc00 0x7f49fc06e0c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4a08035000 tx=0x7f4a08030000 comp rx=0 tx=0).stop 2024-03-24T00:39:18.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.895+0000 7f4a16d5b640 1 -- 172.21.15.67:0/2328462194 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a100c9640 msgr2=0x7f4a100cb220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:18.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.895+0000 7f4a16d5b640 1 --2- 172.21.15.67:0/2328462194 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a100c9640 0x7f4a100cb220 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4a0000e7f0 tx=0x7f4a0000ecb0 comp rx=0 tx=0).stop 2024-03-24T00:39:18.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.895+0000 7f4a16d5b640 1 -- 172.21.15.67:0/2328462194 shutdown_connections 2024-03-24T00:39:18.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.895+0000 7f4a16d5b640 1 --2- 172.21.15.67:0/2328462194 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f49fc06bc00 0x7f49fc06e0c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:18.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.895+0000 7f4a16d5b640 1 --2- 172.21.15.67:0/2328462194 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a100c9640 0x7f4a100cb220 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:18.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.895+0000 7f4a16d5b640 1 --2- 172.21.15.67:0/2328462194 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a100b49b0 0x7f4a100c55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:18.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.895+0000 7f4a16d5b640 1 --2- 172.21.15.67:0/2328462194 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4a10099d00 0x7f4a10156d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:18.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.895+0000 7f4a16d5b640 1 -- 172.21.15.67:0/2328462194 >> 172.21.15.67:0/2328462194 conn(0x7f4a100b3650 msgr2=0x7f4a100b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:18.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.895+0000 7f4a16d5b640 1 -- 172.21.15.67:0/2328462194 shutdown_connections 2024-03-24T00:39:18.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:18.895+0000 7f4a16d5b640 1 -- 172.21.15.67:0/2328462194 wait complete. 2024-03-24T00:39:18.906 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:39:19.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.010+0000 7f5d847f4640 1 Processor -- start 2024-03-24T00:39:19.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.010+0000 7f5d847f4640 1 -- start start 2024-03-24T00:39:19.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.010+0000 7f5d847f4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5d7c15dea0 0x7f5d7c0d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:19.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.011+0000 7f5d847f4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5d7c0b6170 0x7f5d7c0d3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:19.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.011+0000 7f5d847f4640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d7c0d2250 con 0x7f5d7c15e280 2024-03-24T00:39:19.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.011+0000 7f5d847f4640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d7c0d2530 con 0x7f5d7c0b6170 2024-03-24T00:39:19.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.011+0000 7f5d847f4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d7c0d2810 con 0x7f5d7c15dea0 2024-03-24T00:39:19.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.011+0000 7f5d82d91640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5d7c0b6170 0x7f5d7c0d3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:19.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.011+0000 7f5d82d91640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5d7c0b6170 0x7f5d7c0d3910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36168/0 (socket says 172.21.15.67:36168) 2024-03-24T00:39:19.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.011+0000 7f5d82d91640 1 -- 172.21.15.67:0/1847808512 learned_addr learned my addr 172.21.15.67:0/1847808512 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:19.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.011+0000 7f5d82590640 1 --2- 172.21.15.67:0/1847808512 >> v2:172.21.15.72:3300/0 conn(0x7f5d7c15dea0 0x7f5d7c0d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:19.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.011+0000 7f5d82d91640 1 -- 172.21.15.67:0/1847808512 >> v2:172.21.15.72:3300/0 conn(0x7f5d7c15dea0 msgr2=0x7f5d7c0d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:19.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.011+0000 7f5d82d91640 1 --2- 172.21.15.67:0/1847808512 >> v2:172.21.15.72:3300/0 conn(0x7f5d7c15dea0 0x7f5d7c0d33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:19.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.011+0000 7f5d82d91640 1 -- 172.21.15.67:0/1847808512 >> v1:172.21.15.72:6789/0 conn(0x7f5d7c15e280 legacy=0x7f5d7c0b5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:19.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.011+0000 7f5d82d91640 1 -- 172.21.15.67:0/1847808512 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d7c0d2af0 con 0x7f5d7c0b6170 2024-03-24T00:39:19.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.011+0000 7f5d8158e640 1 -- 172.21.15.67:0/1847808512 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1151634275 0 0) 0x7f5d7c0d2250 con 0x7f5d7c15e280 2024-03-24T00:39:19.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.012+0000 7f5d82d91640 1 --2- 172.21.15.67:0/1847808512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d7c0b6170 0x7f5d7c0d3910 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5d78009870 tx=0x7f5d7802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e81365c582b33bc7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:19.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.012+0000 7f5d8158e640 1 -- 172.21.15.67:0/1847808512 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d78035020 con 0x7f5d7c0b6170 2024-03-24T00:39:19.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.012+0000 7f5d8158e640 1 -- 172.21.15.67:0/1847808512 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d78031870 con 0x7f5d7c0b6170 2024-03-24T00:39:19.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.012+0000 7f5d8158e640 1 -- 172.21.15.67:0/1847808512 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d78031c40 con 0x7f5d7c0b6170 2024-03-24T00:39:19.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.012+0000 7f5d847f4640 1 -- 172.21.15.67:0/1847808512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d7c0b6170 msgr2=0x7f5d7c0d3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:19.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.012+0000 7f5d847f4640 1 --2- 172.21.15.67:0/1847808512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d7c0b6170 0x7f5d7c0d3910 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5d78009870 tx=0x7f5d7802b910 comp rx=0 tx=0).stop 2024-03-24T00:39:19.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.012+0000 7f5d847f4640 1 -- 172.21.15.67:0/1847808512 shutdown_connections 2024-03-24T00:39:19.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.012+0000 7f5d847f4640 1 --2- 172.21.15.67:0/1847808512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d7c0b6170 0x7f5d7c0d3910 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:19.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.012+0000 7f5d847f4640 1 --2- 172.21.15.67:0/1847808512 >> v2:172.21.15.72:3300/0 conn(0x7f5d7c15dea0 0x7f5d7c0d33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:19.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.012+0000 7f5d847f4640 1 -- 172.21.15.67:0/1847808512 >> 172.21.15.67:0/1847808512 conn(0x7f5d7c072da0 msgr2=0x7f5d7c0d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:19.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.013+0000 7f5d847f4640 1 -- 172.21.15.67:0/1847808512 shutdown_connections 2024-03-24T00:39:19.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.013+0000 7f5d847f4640 1 -- 172.21.15.67:0/1847808512 wait complete. 2024-03-24T00:39:19.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.014+0000 7f5d847f4640 1 Processor -- start 2024-03-24T00:39:19.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.014+0000 7f5d847f4640 1 -- start start 2024-03-24T00:39:19.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.014+0000 7f5d847f4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5d7c0b6170 0x7f5d7c0cc640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:19.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.014+0000 7f5d847f4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d7c15dea0 0x7f5d7c0cdd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:19.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.014+0000 7f5d847f4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d7c15e280 0x7f5d7c14f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:19.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.014+0000 7f5d847f4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7c0b7480 con 0x7f5d7c15dea0 2024-03-24T00:39:19.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.014+0000 7f5d847f4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7c0b7300 con 0x7f5d7c15e280 2024-03-24T00:39:19.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.014+0000 7f5d847f4640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7c0b7600 con 0x7f5d7c0b6170 2024-03-24T00:39:19.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.014+0000 7f5d81d8f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d7c15dea0 0x7f5d7c0cdd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:19.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.014+0000 7f5d81d8f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d7c15dea0 0x7f5d7c0cdd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36174/0 (socket says 172.21.15.67:36174) 2024-03-24T00:39:19.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.014+0000 7f5d81d8f640 1 -- 172.21.15.67:0/1091848774 learned_addr learned my addr 172.21.15.67:0/1091848774 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:19.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.015+0000 7f5d82590640 1 --2- 172.21.15.67:0/1091848774 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5d7c0b6170 0x7f5d7c0cc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:19.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.015+0000 7f5d82d91640 1 --2- 172.21.15.67:0/1091848774 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d7c15e280 0x7f5d7c14f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:19.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.015+0000 7f5d81d8f640 1 -- 172.21.15.67:0/1091848774 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5d7c0b6170 msgr2=0x7f5d7c0cc640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:19.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.015+0000 7f5d81d8f640 1 --2- 172.21.15.67:0/1091848774 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5d7c0b6170 0x7f5d7c0cc640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:19.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.015+0000 7f5d81d8f640 1 -- 172.21.15.67:0/1091848774 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d7c15e280 msgr2=0x7f5d7c14f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:19.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.015+0000 7f5d81d8f640 1 --2- 172.21.15.67:0/1091848774 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d7c15e280 0x7f5d7c14f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:19.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.015+0000 7f5d81d8f640 1 -- 172.21.15.67:0/1091848774 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d7c1e7cb0 con 0x7f5d7c15dea0 2024-03-24T00:39:19.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.015+0000 7f5d81d8f640 1 --2- 172.21.15.67:0/1091848774 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d7c15dea0 0x7f5d7c0cdd60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c00c410 tx=0x7f5d6c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:19.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.015+0000 7f5d72ffd640 1 -- 172.21.15.67:0/1091848774 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d6c017070 con 0x7f5d7c15dea0 2024-03-24T00:39:19.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.015+0000 7f5d72ffd640 1 -- 172.21.15.67:0/1091848774 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d6c0035f0 con 0x7f5d7c15dea0 2024-03-24T00:39:19.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.015+0000 7f5d847f4640 1 -- 172.21.15.67:0/1091848774 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d7c1e7f90 con 0x7f5d7c15dea0 2024-03-24T00:39:19.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.015+0000 7f5d72ffd640 1 -- 172.21.15.67:0/1091848774 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d6c0039e0 con 0x7f5d7c15dea0 2024-03-24T00:39:19.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.015+0000 7f5d847f4640 1 -- 172.21.15.67:0/1091848774 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d7c0cbe40 con 0x7f5d7c15dea0 2024-03-24T00:39:19.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.016+0000 7f5d847f4640 1 -- 172.21.15.67:0/1091848774 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d7c04f400 con 0x7f5d7c15dea0 2024-03-24T00:39:19.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.016+0000 7f5d72ffd640 1 -- 172.21.15.67:0/1091848774 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5d6c01a070 con 0x7f5d7c15dea0 2024-03-24T00:39:19.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.017+0000 7f5d72ffd640 1 --2- 172.21.15.67:0/1091848774 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5d6006bbd0 0x7f5d6006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:19.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.017+0000 7f5d72ffd640 1 -- 172.21.15.67:0/1091848774 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f5d6c08a180 con 0x7f5d7c15dea0 2024-03-24T00:39:19.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.017+0000 7f5d82590640 1 --2- 172.21.15.67:0/1091848774 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5d6006bbd0 0x7f5d6006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:19.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.017+0000 7f5d82590640 1 --2- 172.21.15.67:0/1091848774 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5d6006bbd0 0x7f5d6006e090 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5d740113e0 tx=0x7f5d7400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:19.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.021+0000 7f5d72ffd640 1 -- 172.21.15.67:0/1091848774 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5d6c0582e0 con 0x7f5d7c15dea0 2024-03-24T00:39:19.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.203+0000 7f5d847f4640 1 -- 172.21.15.67:0/1091848774 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5d7c0b9c40 con 0x7f5d7c15dea0 2024-03-24T00:39:19.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.204+0000 7f5d72ffd640 1 -- 172.21.15.67:0/1091848774 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5d6c05c380 con 0x7f5d7c15dea0 2024-03-24T00:39:19.205 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-24T00:39:19.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.206+0000 7f5d847f4640 1 -- 172.21.15.67:0/1091848774 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5d6006bbd0 msgr2=0x7f5d6006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:19.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.206+0000 7f5d847f4640 1 --2- 172.21.15.67:0/1091848774 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5d6006bbd0 0x7f5d6006e090 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5d740113e0 tx=0x7f5d7400b040 comp rx=0 tx=0).stop 2024-03-24T00:39:19.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.206+0000 7f5d847f4640 1 -- 172.21.15.67:0/1091848774 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d7c15dea0 msgr2=0x7f5d7c0cdd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:19.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.206+0000 7f5d847f4640 1 --2- 172.21.15.67:0/1091848774 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d7c15dea0 0x7f5d7c0cdd60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c00c410 tx=0x7f5d6c00c8d0 comp rx=0 tx=0).stop 2024-03-24T00:39:19.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.206+0000 7f5d847f4640 1 -- 172.21.15.67:0/1091848774 shutdown_connections 2024-03-24T00:39:19.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.206+0000 7f5d847f4640 1 --2- 172.21.15.67:0/1091848774 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5d6006bbd0 0x7f5d6006e090 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:19.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.206+0000 7f5d847f4640 1 --2- 172.21.15.67:0/1091848774 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d7c15e280 0x7f5d7c14f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:19.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.206+0000 7f5d847f4640 1 --2- 172.21.15.67:0/1091848774 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d7c15dea0 0x7f5d7c0cdd60 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:19.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.206+0000 7f5d847f4640 1 --2- 172.21.15.67:0/1091848774 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5d7c0b6170 0x7f5d7c0cc640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:19.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.206+0000 7f5d847f4640 1 -- 172.21.15.67:0/1091848774 >> 172.21.15.67:0/1091848774 conn(0x7f5d7c072da0 msgr2=0x7f5d7c0d31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:19.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.206+0000 7f5d847f4640 1 -- 172.21.15.67:0/1091848774 shutdown_connections 2024-03-24T00:39:19.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.206+0000 7f5d847f4640 1 -- 172.21.15.67:0/1091848774 wait complete. 2024-03-24T00:39:19.217 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:39:19.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.321+0000 7f2d32c44640 1 Processor -- start 2024-03-24T00:39:19.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.321+0000 7f2d32c44640 1 -- start start 2024-03-24T00:39:19.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.322+0000 7f2d32c44640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2d2c0cf100 0x7f2d2c168e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:19.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.322+0000 7f2d32c44640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d2c162290 con 0x7f2d2c0b9690 2024-03-24T00:39:19.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.322+0000 7f2d32c44640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d2c162570 con 0x7f2d2c161370 2024-03-24T00:39:19.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.322+0000 7f2d32c44640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d2c0b6e10 con 0x7f2d2c0cf100 2024-03-24T00:39:19.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.322+0000 7f2d311e1640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2d2c0cf100 0x7f2d2c168e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:19.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.322+0000 7f2d2bfff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2d2c0b9690 0x7f2d2c0ce9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47908/0 (socket says 172.21.15.67:47908) 2024-03-24T00:39:19.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.322+0000 7f2d2bfff640 1 -- 172.21.15.67:0/2674600820 learned_addr learned my addr 172.21.15.67:0/2674600820 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:19.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.322+0000 7f2d309e0640 1 --1- 172.21.15.67:0/2674600820 >> v1:172.21.15.72:6789/0 conn(0x7f2d2c161370 0x7f2d2c161ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:49996/0 (socket says 172.21.15.67:49996) 2024-03-24T00:39:19.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.322+0000 7f2d311e1640 1 -- 172.21.15.67:0/2674600820 >> v1:172.21.15.72:6789/0 conn(0x7f2d2c161370 legacy=0x7f2d2c161ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:19.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.322+0000 7f2d311e1640 1 -- 172.21.15.67:0/2674600820 >> v1:172.21.15.67:6789/0 conn(0x7f2d2c0b9690 legacy=0x7f2d2c0ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:19.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.322+0000 7f2d2b7fe640 1 -- 172.21.15.67:0/2674600820 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 783481922 0 0) 0x7f2d2c162570 con 0x7f2d2c161370 2024-03-24T00:39:19.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.322+0000 7f2d311e1640 1 -- 172.21.15.67:0/2674600820 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d2c162850 con 0x7f2d2c0cf100 2024-03-24T00:39:19.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.323+0000 7f2d311e1640 1 --2- 172.21.15.67:0/2674600820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d2c0cf100 0x7f2d2c168e40 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2d24009d90 tx=0x7f2d2402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36d0a7ae1e1f3dd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:19.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.323+0000 7f2d2b7fe640 1 -- 172.21.15.67:0/2674600820 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d24039070 con 0x7f2d2c0cf100 2024-03-24T00:39:19.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.323+0000 7f2d2b7fe640 1 -- 172.21.15.67:0/2674600820 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d2402cb40 con 0x7f2d2c0cf100 2024-03-24T00:39:19.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.323+0000 7f2d2b7fe640 1 -- 172.21.15.67:0/2674600820 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d24034440 con 0x7f2d2c0cf100 2024-03-24T00:39:19.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.324+0000 7f2d32c44640 1 -- 172.21.15.67:0/2674600820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d2c0cf100 msgr2=0x7f2d2c168e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:19.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.324+0000 7f2d32c44640 1 --2- 172.21.15.67:0/2674600820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d2c0cf100 0x7f2d2c168e40 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2d24009d90 tx=0x7f2d2402be10 comp rx=0 tx=0).stop 2024-03-24T00:39:19.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.324+0000 7f2d32c44640 1 -- 172.21.15.67:0/2674600820 shutdown_connections 2024-03-24T00:39:19.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.324+0000 7f2d32c44640 1 --2- 172.21.15.67:0/2674600820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d2c0cf100 0x7f2d2c168e40 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:19.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.324+0000 7f2d32c44640 1 -- 172.21.15.67:0/2674600820 >> 172.21.15.67:0/2674600820 conn(0x7f2d2c06c880 msgr2=0x7f2d2c06cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:19.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.324+0000 7f2d32c44640 1 -- 172.21.15.67:0/2674600820 shutdown_connections 2024-03-24T00:39:19.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.324+0000 7f2d32c44640 1 -- 172.21.15.67:0/2674600820 wait complete. 2024-03-24T00:39:19.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.325+0000 7f2d32c44640 1 Processor -- start 2024-03-24T00:39:19.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.325+0000 7f2d32c44640 1 -- start start 2024-03-24T00:39:19.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.326+0000 7f2d32c44640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d2c0b9690 0x7f2d2c1f4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:19.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.326+0000 7f2d32c44640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2d2c0cf100 0x7f2d2c1f75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:19.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.326+0000 7f2d32c44640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d2c161370 0x7f2d2c151bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:19.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.326+0000 7f2d32c44640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2c0d3630 con 0x7f2d2c0b9690 2024-03-24T00:39:19.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.326+0000 7f2d32c44640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2c0d2190 con 0x7f2d2c161370 2024-03-24T00:39:19.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.326+0000 7f2d32c44640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2c0d37b0 con 0x7f2d2c0cf100 2024-03-24T00:39:19.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.326+0000 7f2d309e0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d2c0b9690 0x7f2d2c1f4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:19.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.326+0000 7f2d309e0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d2c0b9690 0x7f2d2c1f4ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36188/0 (socket says 172.21.15.67:36188) 2024-03-24T00:39:19.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.326+0000 7f2d309e0640 1 -- 172.21.15.67:0/680277264 learned_addr learned my addr 172.21.15.67:0/680277264 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:19.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.326+0000 7f2d2bfff640 1 --2- 172.21.15.67:0/680277264 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2d2c0cf100 0x7f2d2c1f75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:19.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.326+0000 7f2d311e1640 1 --2- 172.21.15.67:0/680277264 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d2c161370 0x7f2d2c151bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:19.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.326+0000 7f2d309e0640 1 -- 172.21.15.67:0/680277264 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2d2c0cf100 msgr2=0x7f2d2c1f75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:19.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.326+0000 7f2d309e0640 1 --2- 172.21.15.67:0/680277264 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2d2c0cf100 0x7f2d2c1f75c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:19.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.326+0000 7f2d309e0640 1 -- 172.21.15.67:0/680277264 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d2c161370 msgr2=0x7f2d2c151bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:19.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.326+0000 7f2d309e0640 1 --2- 172.21.15.67:0/680277264 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d2c161370 0x7f2d2c151bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:19.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.326+0000 7f2d309e0640 1 -- 172.21.15.67:0/680277264 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d2c1f7d10 con 0x7f2d2c0b9690 2024-03-24T00:39:19.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.326+0000 7f2d2bfff640 1 --2- 172.21.15.67:0/680277264 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2d2c0cf100 0x7f2d2c1f75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:19.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.326+0000 7f2d311e1640 1 --2- 172.21.15.67:0/680277264 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d2c161370 0x7f2d2c151bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:19.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.326+0000 7f2d309e0640 1 --2- 172.21.15.67:0/680277264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d2c0b9690 0x7f2d2c1f4ae0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f2d2000c430 tx=0x7f2d2000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:19.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.326+0000 7f2d297fa640 1 -- 172.21.15.67:0/680277264 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d200074e0 con 0x7f2d2c0b9690 2024-03-24T00:39:19.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.326+0000 7f2d297fa640 1 -- 172.21.15.67:0/680277264 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d200047e0 con 0x7f2d2c0b9690 2024-03-24T00:39:19.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.326+0000 7f2d297fa640 1 -- 172.21.15.67:0/680277264 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d20004bd0 con 0x7f2d2c0b9690 2024-03-24T00:39:19.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.326+0000 7f2d32c44640 1 -- 172.21.15.67:0/680277264 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d2c1f7ff0 con 0x7f2d2c0b9690 2024-03-24T00:39:19.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.327+0000 7f2d32c44640 1 -- 172.21.15.67:0/680277264 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d2c154540 con 0x7f2d2c0b9690 2024-03-24T00:39:19.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.327+0000 7f2d32c44640 1 -- 172.21.15.67:0/680277264 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d2c154740 con 0x7f2d2c0b9690 2024-03-24T00:39:19.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.328+0000 7f2d297fa640 1 -- 172.21.15.67:0/680277264 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2d20011070 con 0x7f2d2c0b9690 2024-03-24T00:39:19.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.328+0000 7f2d297fa640 1 --2- 172.21.15.67:0/680277264 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2d1006bbd0 0x7f2d1006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:19.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.328+0000 7f2d297fa640 1 -- 172.21.15.67:0/680277264 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f2d2008a8d0 con 0x7f2d2c0b9690 2024-03-24T00:39:19.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.328+0000 7f2d2bfff640 1 --2- 172.21.15.67:0/680277264 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2d1006bbd0 0x7f2d1006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:19.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.329+0000 7f2d2bfff640 1 --2- 172.21.15.67:0/680277264 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2d1006bbd0 0x7f2d1006e090 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c160b60 tx=0x7f2d18007e20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:19.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.333+0000 7f2d297fa640 1 -- 172.21.15.67:0/680277264 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2d2c154740 con 0x7f2d2c0b9690 2024-03-24T00:39:19.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.515+0000 7f2d32c44640 1 -- 172.21.15.67:0/680277264 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2d2c04f400 con 0x7f2d2c0b9690 2024-03-24T00:39:19.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.516+0000 7f2d297fa640 1 -- 172.21.15.67:0/680277264 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2d20058a30 con 0x7f2d2c0b9690 2024-03-24T00:39:19.517 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-03-24T00:39:19.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.518+0000 7f2d32c44640 1 -- 172.21.15.67:0/680277264 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2d1006bbd0 msgr2=0x7f2d1006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:19.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.518+0000 7f2d32c44640 1 --2- 172.21.15.67:0/680277264 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2d1006bbd0 0x7f2d1006e090 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c160b60 tx=0x7f2d18007e20 comp rx=0 tx=0).stop 2024-03-24T00:39:19.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.518+0000 7f2d32c44640 1 -- 172.21.15.67:0/680277264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d2c0b9690 msgr2=0x7f2d2c1f4ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:19.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.518+0000 7f2d32c44640 1 --2- 172.21.15.67:0/680277264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d2c0b9690 0x7f2d2c1f4ae0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f2d2000c430 tx=0x7f2d2000c8f0 comp rx=0 tx=0).stop 2024-03-24T00:39:19.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.518+0000 7f2d32c44640 1 -- 172.21.15.67:0/680277264 shutdown_connections 2024-03-24T00:39:19.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.518+0000 7f2d32c44640 1 --2- 172.21.15.67:0/680277264 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2d1006bbd0 0x7f2d1006e090 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:19.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.518+0000 7f2d32c44640 1 --2- 172.21.15.67:0/680277264 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d2c161370 0x7f2d2c151bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:19.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.518+0000 7f2d32c44640 1 --2- 172.21.15.67:0/680277264 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2d2c0cf100 0x7f2d2c1f75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:19.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.518+0000 7f2d32c44640 1 --2- 172.21.15.67:0/680277264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d2c0b9690 0x7f2d2c1f4ae0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:19.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.518+0000 7f2d32c44640 1 -- 172.21.15.67:0/680277264 >> 172.21.15.67:0/680277264 conn(0x7f2d2c06c880 msgr2=0x7f2d2c15c540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:19.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.518+0000 7f2d32c44640 1 -- 172.21.15.67:0/680277264 shutdown_connections 2024-03-24T00:39:19.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.518+0000 7f2d32c44640 1 -- 172.21.15.67:0/680277264 wait complete. 2024-03-24T00:39:19.529 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T00:39:19.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.633+0000 7fa8fbdcd640 1 Processor -- start 2024-03-24T00:39:19.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.633+0000 7fa8fbdcd640 1 -- start start 2024-03-24T00:39:19.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.634+0000 7fa8fbdcd640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa8f415e130 0x7fa8f40ce610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:19.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.634+0000 7fa8fbdcd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa8f40d32c0 0x7fa8f40d0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:19.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.634+0000 7fa8fbdcd640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8f40cf570 con 0x7fa8f40d0640 2024-03-24T00:39:19.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.634+0000 7fa8fbdcd640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8f40ceb50 con 0x7fa8f40d32c0 2024-03-24T00:39:19.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.634+0000 7fa8fbdcd640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f40d0fe0 con 0x7fa8f415e130 2024-03-24T00:39:19.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.634+0000 7fa8f9368640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa8f40d0640 0x7fa8f40b57a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47922/0 (socket says 172.21.15.67:47922) 2024-03-24T00:39:19.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.634+0000 7fa8f9368640 1 -- 172.21.15.67:0/1872517036 learned_addr learned my addr 172.21.15.67:0/1872517036 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:19.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.634+0000 7fa8fa36a640 1 --2- 172.21.15.67:0/1872517036 >> v2:172.21.15.72:3300/0 conn(0x7fa8f40d32c0 0x7fa8f40d0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:19.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.634+0000 7fa8f9b69640 1 --2- 172.21.15.67:0/1872517036 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa8f415e130 0x7fa8f40ce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:19.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.634+0000 7fa8f8b67640 1 -- 172.21.15.67:0/1872517036 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 440648185 0 0) 0x7fa8f40cf570 con 0x7fa8f40d0640 2024-03-24T00:39:19.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.634+0000 7fa8f8b67640 1 -- 172.21.15.67:0/1872517036 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8dc003520 con 0x7fa8f40d0640 2024-03-24T00:39:19.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.635+0000 7fa8f9b69640 1 -- 172.21.15.67:0/1872517036 >> v2:172.21.15.72:3300/0 conn(0x7fa8f40d32c0 msgr2=0x7fa8f40d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:19.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.635+0000 7fa8f9b69640 1 --2- 172.21.15.67:0/1872517036 >> v2:172.21.15.72:3300/0 conn(0x7fa8f40d32c0 0x7fa8f40d0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:19.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.635+0000 7fa8f9b69640 1 -- 172.21.15.67:0/1872517036 >> v1:172.21.15.67:6789/0 conn(0x7fa8f40d0640 legacy=0x7fa8f40b57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:19.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.635+0000 7fa8f9b69640 1 -- 172.21.15.67:0/1872517036 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8f40cecf0 con 0x7fa8f415e130 2024-03-24T00:39:19.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.635+0000 7fa8fa36a640 1 --2- 172.21.15.67:0/1872517036 >> v2:172.21.15.72:3300/0 conn(0x7fa8f40d32c0 0x7fa8f40d0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:19.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.635+0000 7fa8f9b69640 1 --2- 172.21.15.67:0/1872517036 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa8f415e130 0x7fa8f40ce610 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa8e4009d90 tx=0x7fa8e402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=707f5c39b103d0c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:19.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.636+0000 7fa8f8b67640 1 -- 172.21.15.67:0/1872517036 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e4039070 con 0x7fa8f415e130 2024-03-24T00:39:19.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.636+0000 7fa8f8b67640 1 -- 172.21.15.67:0/1872517036 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8e402cc50 con 0x7fa8f415e130 2024-03-24T00:39:19.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.636+0000 7fa8f8b67640 1 -- 172.21.15.67:0/1872517036 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e4034440 con 0x7fa8f415e130 2024-03-24T00:39:19.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.636+0000 7fa8fbdcd640 1 -- 172.21.15.67:0/1872517036 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa8f415e130 msgr2=0x7fa8f40ce610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:19.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.636+0000 7fa8fbdcd640 1 --2- 172.21.15.67:0/1872517036 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa8f415e130 0x7fa8f40ce610 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa8e4009d90 tx=0x7fa8e402bea0 comp rx=0 tx=0).stop 2024-03-24T00:39:19.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.636+0000 7fa8fbdcd640 1 -- 172.21.15.67:0/1872517036 shutdown_connections 2024-03-24T00:39:19.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.636+0000 7fa8fbdcd640 1 --2- 172.21.15.67:0/1872517036 >> v2:172.21.15.72:3300/0 conn(0x7fa8f40d32c0 0x7fa8f40d0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:19.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.636+0000 7fa8fbdcd640 1 --2- 172.21.15.67:0/1872517036 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa8f415e130 0x7fa8f40ce610 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:19.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.636+0000 7fa8fbdcd640 1 -- 172.21.15.67:0/1872517036 >> 172.21.15.67:0/1872517036 conn(0x7fa8f406c8a0 msgr2=0x7fa8f406ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:19.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.636+0000 7fa8fbdcd640 1 -- 172.21.15.67:0/1872517036 shutdown_connections 2024-03-24T00:39:19.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.636+0000 7fa8fbdcd640 1 -- 172.21.15.67:0/1872517036 wait complete. 2024-03-24T00:39:19.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.637+0000 7fa8fbdcd640 1 Processor -- start 2024-03-24T00:39:19.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.637+0000 7fa8fbdcd640 1 -- start start 2024-03-24T00:39:19.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.638+0000 7fa8fbdcd640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa8f40d0640 0x7fa8f4149cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:19.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.638+0000 7fa8fbdcd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8f40d32c0 0x7fa8f40ccc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:19.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.638+0000 7fa8fbdcd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa8f415e130 0x7fa8f40cd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:19.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.638+0000 7fa8fbdcd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f40cf290 con 0x7fa8f40d32c0 2024-03-24T00:39:19.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.638+0000 7fa8fbdcd640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f40cefb0 con 0x7fa8f415e130 2024-03-24T00:39:19.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.638+0000 7fa8fbdcd640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f415e510 con 0x7fa8f40d0640 2024-03-24T00:39:19.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.638+0000 7fa8f9368640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8f40d32c0 0x7fa8f40ccc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:19.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.638+0000 7fa8f9368640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8f40d32c0 0x7fa8f40ccc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36200/0 (socket says 172.21.15.67:36200) 2024-03-24T00:39:19.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.638+0000 7fa8f9368640 1 -- 172.21.15.67:0/1029931965 learned_addr learned my addr 172.21.15.67:0/1029931965 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:19.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.638+0000 7fa8f9b69640 1 --2- 172.21.15.67:0/1029931965 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa8f40d0640 0x7fa8f4149cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:19.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.638+0000 7fa8fa36a640 1 --2- 172.21.15.67:0/1029931965 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa8f415e130 0x7fa8f40cd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:19.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.638+0000 7fa8f9368640 1 -- 172.21.15.67:0/1029931965 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa8f40d0640 msgr2=0x7fa8f4149cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:19.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.638+0000 7fa8f9368640 1 --2- 172.21.15.67:0/1029931965 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa8f40d0640 0x7fa8f4149cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:19.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.638+0000 7fa8f9368640 1 -- 172.21.15.67:0/1029931965 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa8f415e130 msgr2=0x7fa8f40cd1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:19.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.638+0000 7fa8f9368640 1 --2- 172.21.15.67:0/1029931965 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa8f415e130 0x7fa8f40cd1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:19.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.638+0000 7fa8f9368640 1 -- 172.21.15.67:0/1029931965 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8f40ce3a0 con 0x7fa8f40d32c0 2024-03-24T00:39:19.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.639+0000 7fa8f9368640 1 --2- 172.21.15.67:0/1029931965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8f40d32c0 0x7fa8f40ccc60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa8e800c430 tx=0x7fa8e800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:19.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.639+0000 7fa8f27fc640 1 -- 172.21.15.67:0/1029931965 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e8017070 con 0x7fa8f40d32c0 2024-03-24T00:39:19.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.639+0000 7fa8f27fc640 1 -- 172.21.15.67:0/1029931965 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8e8005ce0 con 0x7fa8f40d32c0 2024-03-24T00:39:19.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.639+0000 7fa8fbdcd640 1 -- 172.21.15.67:0/1029931965 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8f414a1f0 con 0x7fa8f40d32c0 2024-03-24T00:39:19.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.639+0000 7fa8f27fc640 1 -- 172.21.15.67:0/1029931965 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e80072f0 con 0x7fa8f40d32c0 2024-03-24T00:39:19.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.639+0000 7fa8fbdcd640 1 -- 172.21.15.67:0/1029931965 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8f40cdb00 con 0x7fa8f40d32c0 2024-03-24T00:39:19.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.640+0000 7fa8fbdcd640 1 -- 172.21.15.67:0/1029931965 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8f404f400 con 0x7fa8f40d32c0 2024-03-24T00:39:19.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.640+0000 7fa8f27fc640 1 -- 172.21.15.67:0/1029931965 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa8e801a070 con 0x7fa8f40d32c0 2024-03-24T00:39:19.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.641+0000 7fa8f27fc640 1 --2- 172.21.15.67:0/1029931965 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa8e006bbd0 0x7fa8e006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:19.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.641+0000 7fa8f27fc640 1 -- 172.21.15.67:0/1029931965 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fa8e808a1c0 con 0x7fa8f40d32c0 2024-03-24T00:39:19.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.641+0000 7fa8f9b69640 1 --2- 172.21.15.67:0/1029931965 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa8e006bbd0 0x7fa8e006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:19.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.641+0000 7fa8f9b69640 1 --2- 172.21.15.67:0/1029931965 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa8e006bbd0 0x7fa8e006e090 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa8e4036000 tx=0x7fa8e4030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:19.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.645+0000 7fa8f27fc640 1 -- 172.21.15.67:0/1029931965 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa8e8058320 con 0x7fa8f40d32c0 2024-03-24T00:39:19.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.829+0000 7fa8fbdcd640 1 -- 172.21.15.67:0/1029931965 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fa8f40c88a0 con 0x7fa8f40d32c0 2024-03-24T00:39:19.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.829+0000 7fa8f27fc640 1 -- 172.21.15.67:0/1029931965 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7fa8e805c3c0 con 0x7fa8f40d32c0 2024-03-24T00:39:19.830 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-24T00:39:19.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.831+0000 7fa8fbdcd640 1 -- 172.21.15.67:0/1029931965 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa8e006bbd0 msgr2=0x7fa8e006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:19.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.831+0000 7fa8fbdcd640 1 --2- 172.21.15.67:0/1029931965 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa8e006bbd0 0x7fa8e006e090 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa8e4036000 tx=0x7fa8e4030000 comp rx=0 tx=0).stop 2024-03-24T00:39:19.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.831+0000 7fa8fbdcd640 1 -- 172.21.15.67:0/1029931965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8f40d32c0 msgr2=0x7fa8f40ccc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:19.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.831+0000 7fa8fbdcd640 1 --2- 172.21.15.67:0/1029931965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8f40d32c0 0x7fa8f40ccc60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa8e800c430 tx=0x7fa8e800c8f0 comp rx=0 tx=0).stop 2024-03-24T00:39:19.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.831+0000 7fa8fbdcd640 1 -- 172.21.15.67:0/1029931965 shutdown_connections 2024-03-24T00:39:19.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.831+0000 7fa8fbdcd640 1 --2- 172.21.15.67:0/1029931965 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa8e006bbd0 0x7fa8e006e090 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:19.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.831+0000 7fa8fbdcd640 1 --2- 172.21.15.67:0/1029931965 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa8f415e130 0x7fa8f40cd1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:19.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.831+0000 7fa8fbdcd640 1 --2- 172.21.15.67:0/1029931965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8f40d32c0 0x7fa8f40ccc60 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:19.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.831+0000 7fa8fbdcd640 1 --2- 172.21.15.67:0/1029931965 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa8f40d0640 0x7fa8f4149cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:19.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.831+0000 7fa8fbdcd640 1 -- 172.21.15.67:0/1029931965 >> 172.21.15.67:0/1029931965 conn(0x7fa8f406c8a0 msgr2=0x7fa8f406cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:19.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.831+0000 7fa8fbdcd640 1 -- 172.21.15.67:0/1029931965 shutdown_connections 2024-03-24T00:39:19.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.831+0000 7fa8fbdcd640 1 -- 172.21.15.67:0/1029931965 wait complete. 2024-03-24T00:39:19.843 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:39:19.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.943+0000 7fb919fc6640 1 Processor -- start 2024-03-24T00:39:19.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.943+0000 7fb919fc6640 1 -- start start 2024-03-24T00:39:19.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.944+0000 7fb919fc6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb91415dec0 0x7fb9140ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:19.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.944+0000 7fb919fc6640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb91415e2a0 0x7fb914162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:19.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.944+0000 7fb919fc6640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9140cf640 con 0x7fb9140d07e0 2024-03-24T00:39:19.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.944+0000 7fb919fc6640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9140cee30 con 0x7fb91415e2a0 2024-03-24T00:39:19.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.944+0000 7fb919fc6640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9140d2120 con 0x7fb91415dec0 2024-03-24T00:39:19.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.944+0000 7fb912ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb91415e2a0 0x7fb914162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:19.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.944+0000 7fb913fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb9140d07e0 0x7fb9140b5850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47932/0 (socket says 172.21.15.67:47932) 2024-03-24T00:39:19.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.944+0000 7fb913fff640 1 -- 172.21.15.67:0/3709563059 learned_addr learned my addr 172.21.15.67:0/3709563059 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:19.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.944+0000 7fb9137fe640 1 --2- 172.21.15.67:0/3709563059 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb91415dec0 0x7fb9140ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:19.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.945+0000 7fb912ffd640 1 -- 172.21.15.67:0/3709563059 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb91415dec0 msgr2=0x7fb9140ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:19.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.945+0000 7fb912ffd640 1 --2- 172.21.15.67:0/3709563059 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb91415dec0 0x7fb9140ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:19.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.945+0000 7fb912ffd640 1 -- 172.21.15.67:0/3709563059 >> v1:172.21.15.67:6789/0 conn(0x7fb9140d07e0 legacy=0x7fb9140b5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:19.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.945+0000 7fb9127fc640 1 -- 172.21.15.67:0/3709563059 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 750275602 0 0) 0x7fb90c001490 con 0x7fb9140d07e0 2024-03-24T00:39:19.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.945+0000 7fb912ffd640 1 -- 172.21.15.67:0/3709563059 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9140cf080 con 0x7fb91415e2a0 2024-03-24T00:39:19.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.945+0000 7fb912ffd640 1 --2- 172.21.15.67:0/3709563059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb91415e2a0 0x7fb914162630 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb9140cefb0 tx=0x7fb90802b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbd8151e7f4817e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:19.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.945+0000 7fb9127fc640 1 -- 172.21.15.67:0/3709563059 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb908031770 con 0x7fb91415e2a0 2024-03-24T00:39:19.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.945+0000 7fb9127fc640 1 -- 172.21.15.67:0/3709563059 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb908037070 con 0x7fb91415e2a0 2024-03-24T00:39:19.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.945+0000 7fb9127fc640 1 -- 172.21.15.67:0/3709563059 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb908031b30 con 0x7fb91415e2a0 2024-03-24T00:39:19.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.946+0000 7fb919fc6640 1 -- 172.21.15.67:0/3709563059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb91415e2a0 msgr2=0x7fb914162630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:19.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.946+0000 7fb919fc6640 1 --2- 172.21.15.67:0/3709563059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb91415e2a0 0x7fb914162630 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb9140cefb0 tx=0x7fb90802b810 comp rx=0 tx=0).stop 2024-03-24T00:39:19.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.946+0000 7fb919fc6640 1 -- 172.21.15.67:0/3709563059 shutdown_connections 2024-03-24T00:39:19.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.946+0000 7fb919fc6640 1 --2- 172.21.15.67:0/3709563059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb91415e2a0 0x7fb914162630 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:19.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.946+0000 7fb919fc6640 1 --2- 172.21.15.67:0/3709563059 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb91415dec0 0x7fb9140ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:19.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.946+0000 7fb919fc6640 1 -- 172.21.15.67:0/3709563059 >> 172.21.15.67:0/3709563059 conn(0x7fb91406c810 msgr2=0x7fb91406cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:19.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.946+0000 7fb919fc6640 1 -- 172.21.15.67:0/3709563059 shutdown_connections 2024-03-24T00:39:19.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.946+0000 7fb919fc6640 1 -- 172.21.15.67:0/3709563059 wait complete. 2024-03-24T00:39:19.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.947+0000 7fb919fc6640 1 Processor -- start 2024-03-24T00:39:19.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.947+0000 7fb919fc6640 1 -- start start 2024-03-24T00:39:19.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.948+0000 7fb919fc6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9140d07e0 0x7fb914154850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:19.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.948+0000 7fb919fc6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb91415dec0 0x7fb9140caea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:19.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.948+0000 7fb919fc6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb91415e2a0 0x7fb91415fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:19.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.948+0000 7fb919fc6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9140d3740 con 0x7fb9140d07e0 2024-03-24T00:39:19.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.948+0000 7fb919fc6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9140d35c0 con 0x7fb91415dec0 2024-03-24T00:39:19.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.948+0000 7fb919fc6640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9140d38c0 con 0x7fb91415e2a0 2024-03-24T00:39:19.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.948+0000 7fb9137fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9140d07e0 0x7fb914154850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:19.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.948+0000 7fb9137fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9140d07e0 0x7fb914154850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36226/0 (socket says 172.21.15.67:36226) 2024-03-24T00:39:19.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.948+0000 7fb9137fe640 1 -- 172.21.15.67:0/2339095211 learned_addr learned my addr 172.21.15.67:0/2339095211 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:19.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.948+0000 7fb913fff640 1 --2- 172.21.15.67:0/2339095211 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb91415e2a0 0x7fb91415fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:19.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.948+0000 7fb912ffd640 1 --2- 172.21.15.67:0/2339095211 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb91415dec0 0x7fb9140caea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:19.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.948+0000 7fb9137fe640 1 -- 172.21.15.67:0/2339095211 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb91415e2a0 msgr2=0x7fb91415fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:19.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.948+0000 7fb9137fe640 1 --2- 172.21.15.67:0/2339095211 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb91415e2a0 0x7fb91415fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:19.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.948+0000 7fb9137fe640 1 -- 172.21.15.67:0/2339095211 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb91415dec0 msgr2=0x7fb9140caea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:19.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.948+0000 7fb9137fe640 1 --2- 172.21.15.67:0/2339095211 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb91415dec0 0x7fb9140caea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:19.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.948+0000 7fb9137fe640 1 -- 172.21.15.67:0/2339095211 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9140cc620 con 0x7fb9140d07e0 2024-03-24T00:39:19.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.949+0000 7fb9137fe640 1 --2- 172.21.15.67:0/2339095211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9140d07e0 0x7fb914154850 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb90400e9b0 tx=0x7fb90400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:19.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.949+0000 7fb8fbfff640 1 -- 172.21.15.67:0/2339095211 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb904017070 con 0x7fb9140d07e0 2024-03-24T00:39:19.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.949+0000 7fb8fbfff640 1 -- 172.21.15.67:0/2339095211 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb90400c9a0 con 0x7fb9140d07e0 2024-03-24T00:39:19.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.949+0000 7fb919fc6640 1 -- 172.21.15.67:0/2339095211 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9140cb880 con 0x7fb9140d07e0 2024-03-24T00:39:19.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.949+0000 7fb8fbfff640 1 -- 172.21.15.67:0/2339095211 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb90400cd90 con 0x7fb9140d07e0 2024-03-24T00:39:19.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.949+0000 7fb919fc6640 1 -- 172.21.15.67:0/2339095211 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9140cc8c0 con 0x7fb9140d07e0 2024-03-24T00:39:19.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.950+0000 7fb919fc6640 1 -- 172.21.15.67:0/2339095211 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb91404f400 con 0x7fb9140d07e0 2024-03-24T00:39:19.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.950+0000 7fb8fbfff640 1 -- 172.21.15.67:0/2339095211 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb90401a070 con 0x7fb9140d07e0 2024-03-24T00:39:19.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.951+0000 7fb8fbfff640 1 --2- 172.21.15.67:0/2339095211 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb90006bbd0 0x7fb90006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:19.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.951+0000 7fb8fbfff640 1 -- 172.21.15.67:0/2339095211 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fb90408ace0 con 0x7fb9140d07e0 2024-03-24T00:39:19.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.951+0000 7fb912ffd640 1 --2- 172.21.15.67:0/2339095211 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb90006bbd0 0x7fb90006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:19.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.951+0000 7fb912ffd640 1 --2- 172.21.15.67:0/2339095211 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb90006bbd0 0x7fb90006e090 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb908036000 tx=0x7fb90803d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:19.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:19.955+0000 7fb8fbfff640 1 -- 172.21.15.67:0/2339095211 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb904058ed0 con 0x7fb9140d07e0 2024-03-24T00:39:20.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.175+0000 7fb919fc6640 1 -- 172.21.15.67:0/2339095211 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb914161530 con 0x7fb9140d07e0 2024-03-24T00:39:20.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.175+0000 7fb8fbfff640 1 -- 172.21.15.67:0/2339095211 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4970 (secure 0 0 0) 0x7fb90405cf70 con 0x7fb9140d07e0 2024-03-24T00:39:20.177 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:39:20.177 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4263,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/837528243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":837528243},{"type":"v1","addr":"172.21.15.67:6834","nonce":837528243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3987071456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":3987071456},{"type":"v1","addr":"172.21.15.67:6835","nonce":3987071456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4306,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/3953492281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3953492281},{"type":"v1","addr":"172.21.15.72:6837","nonce":3953492281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:38:52.127871+0000","modified":"2024-03-24T00:39:08.187262+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4309},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4309":{"gid":4309,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6836/3737698654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3737698654},{"type":"v1","addr":"172.21.15.72:6836","nonce":3737698654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4309,"qdb_cluster":[4309]},"id":1}]} 2024-03-24T00:39:20.178 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 9 2024-03-24T00:39:20.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.177+0000 7fb919fc6640 1 -- 172.21.15.67:0/2339095211 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb90006bbd0 msgr2=0x7fb90006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:20.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.177+0000 7fb919fc6640 1 --2- 172.21.15.67:0/2339095211 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb90006bbd0 0x7fb90006e090 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb908036000 tx=0x7fb90803d040 comp rx=0 tx=0).stop 2024-03-24T00:39:20.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.177+0000 7fb919fc6640 1 -- 172.21.15.67:0/2339095211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9140d07e0 msgr2=0x7fb914154850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:20.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.177+0000 7fb919fc6640 1 --2- 172.21.15.67:0/2339095211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9140d07e0 0x7fb914154850 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb90400e9b0 tx=0x7fb90400ee70 comp rx=0 tx=0).stop 2024-03-24T00:39:20.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.177+0000 7fb919fc6640 1 -- 172.21.15.67:0/2339095211 shutdown_connections 2024-03-24T00:39:20.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.177+0000 7fb919fc6640 1 --2- 172.21.15.67:0/2339095211 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb90006bbd0 0x7fb90006e090 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:20.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.177+0000 7fb919fc6640 1 --2- 172.21.15.67:0/2339095211 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb91415e2a0 0x7fb91415fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:20.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.177+0000 7fb919fc6640 1 --2- 172.21.15.67:0/2339095211 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb91415dec0 0x7fb9140caea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:20.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.177+0000 7fb919fc6640 1 --2- 172.21.15.67:0/2339095211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9140d07e0 0x7fb914154850 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:20.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.177+0000 7fb919fc6640 1 -- 172.21.15.67:0/2339095211 >> 172.21.15.67:0/2339095211 conn(0x7fb91406c810 msgr2=0x7fb9140b8760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:20.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.177+0000 7fb919fc6640 1 -- 172.21.15.67:0/2339095211 shutdown_connections 2024-03-24T00:39:20.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.177+0000 7fb919fc6640 1 -- 172.21.15.67:0/2339095211 wait complete. 2024-03-24T00:39:20.188 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:39:20.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.289+0000 7fb56e5b3640 1 Processor -- start 2024-03-24T00:39:20.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.289+0000 7fb56e5b3640 1 -- start start 2024-03-24T00:39:20.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.290+0000 7fb56e5b3640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb5680cef40 0x7fb5680cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:20.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.290+0000 7fb56e5b3640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb56806dce0 0x7fb56806e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:20.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.290+0000 7fb56e5b3640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb568072890 con 0x7fb5680ccce0 2024-03-24T00:39:20.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.290+0000 7fb56e5b3640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb56815c580 con 0x7fb5680cef40 2024-03-24T00:39:20.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.290+0000 7fb56e5b3640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb56815c860 con 0x7fb56806dce0 2024-03-24T00:39:20.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.290+0000 7fb5677fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb5680cef40 0x7fb5680cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:20.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.290+0000 7fb567fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb5680ccce0 0x7fb5680cd0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47946/0 (socket says 172.21.15.67:47946) 2024-03-24T00:39:20.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.290+0000 7fb567fff640 1 -- 172.21.15.67:0/2747336201 learned_addr learned my addr 172.21.15.67:0/2747336201 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:20.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.290+0000 7fb56cb50640 1 --2- 172.21.15.67:0/2747336201 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb56806dce0 0x7fb56806e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:20.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.290+0000 7fb5677fe640 1 -- 172.21.15.67:0/2747336201 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb56806dce0 msgr2=0x7fb56806e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:20.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.290+0000 7fb5677fe640 1 --2- 172.21.15.67:0/2747336201 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb56806dce0 0x7fb56806e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:20.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.290+0000 7fb5677fe640 1 -- 172.21.15.67:0/2747336201 >> v1:172.21.15.67:6789/0 conn(0x7fb5680ccce0 legacy=0x7fb5680cd0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:20.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.290+0000 7fb5677fe640 1 -- 172.21.15.67:0/2747336201 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb56815cb40 con 0x7fb5680cef40 2024-03-24T00:39:20.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.291+0000 7fb5677fe640 1 --2- 172.21.15.67:0/2747336201 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5680cef40 0x7fb5680cf3c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb5580097a0 tx=0x7fb55802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a70f20e7085cda7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:20.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.291+0000 7fb566ffd640 1 -- 172.21.15.67:0/2747336201 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb558035020 con 0x7fb5680cef40 2024-03-24T00:39:20.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.291+0000 7fb566ffd640 1 -- 172.21.15.67:0/2747336201 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb558031830 con 0x7fb5680cef40 2024-03-24T00:39:20.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.291+0000 7fb566ffd640 1 -- 172.21.15.67:0/2747336201 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb558031c00 con 0x7fb5680cef40 2024-03-24T00:39:20.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.292+0000 7fb56e5b3640 1 -- 172.21.15.67:0/2747336201 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5680cef40 msgr2=0x7fb5680cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:20.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.292+0000 7fb56e5b3640 1 --2- 172.21.15.67:0/2747336201 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5680cef40 0x7fb5680cf3c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb5580097a0 tx=0x7fb55802b8d0 comp rx=0 tx=0).stop 2024-03-24T00:39:20.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.292+0000 7fb56e5b3640 1 -- 172.21.15.67:0/2747336201 shutdown_connections 2024-03-24T00:39:20.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.292+0000 7fb56e5b3640 1 --2- 172.21.15.67:0/2747336201 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb56806dce0 0x7fb56806e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:20.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.292+0000 7fb56e5b3640 1 --2- 172.21.15.67:0/2747336201 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5680cef40 0x7fb5680cf3c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:20.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.292+0000 7fb56e5b3640 1 -- 172.21.15.67:0/2747336201 >> 172.21.15.67:0/2747336201 conn(0x7fb5680cf9e0 msgr2=0x7fb5680cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:20.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.292+0000 7fb56e5b3640 1 -- 172.21.15.67:0/2747336201 shutdown_connections 2024-03-24T00:39:20.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.292+0000 7fb56e5b3640 1 -- 172.21.15.67:0/2747336201 wait complete. 2024-03-24T00:39:20.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.293+0000 7fb56e5b3640 1 Processor -- start 2024-03-24T00:39:20.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.293+0000 7fb56e5b3640 1 -- start start 2024-03-24T00:39:20.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.294+0000 7fb56e5b3640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb56806dce0 0x7fb5680cc100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:20.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.294+0000 7fb56e5b3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5680ccce0 0x7fb5681f1f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:20.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.294+0000 7fb56e5b3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb5680cef40 0x7fb568159150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:20.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.294+0000 7fb56e5b3640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb568145900 con 0x7fb5680ccce0 2024-03-24T00:39:20.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.294+0000 7fb56e5b3640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb56815c290 con 0x7fb5680cef40 2024-03-24T00:39:20.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.294+0000 7fb56e5b3640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb56806e6b0 con 0x7fb56806dce0 2024-03-24T00:39:20.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.294+0000 7fb5677fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5680ccce0 0x7fb5681f1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:20.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.294+0000 7fb5677fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5680ccce0 0x7fb5681f1f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36242/0 (socket says 172.21.15.67:36242) 2024-03-24T00:39:20.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.294+0000 7fb5677fe640 1 -- 172.21.15.67:0/1242494291 learned_addr learned my addr 172.21.15.67:0/1242494291 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:20.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.294+0000 7fb56cb50640 1 --2- 172.21.15.67:0/1242494291 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb5680cef40 0x7fb568159150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:20.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.294+0000 7fb567fff640 1 --2- 172.21.15.67:0/1242494291 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb56806dce0 0x7fb5680cc100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:20.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.294+0000 7fb5677fe640 1 -- 172.21.15.67:0/1242494291 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb56806dce0 msgr2=0x7fb5680cc100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:20.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.294+0000 7fb5677fe640 1 --2- 172.21.15.67:0/1242494291 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb56806dce0 0x7fb5680cc100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:20.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.294+0000 7fb5677fe640 1 -- 172.21.15.67:0/1242494291 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb5680cef40 msgr2=0x7fb568159150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:20.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.294+0000 7fb5677fe640 1 --2- 172.21.15.67:0/1242494291 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb5680cef40 0x7fb568159150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:20.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.294+0000 7fb5677fe640 1 -- 172.21.15.67:0/1242494291 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5681f2470 con 0x7fb5680ccce0 2024-03-24T00:39:20.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.294+0000 7fb5677fe640 1 --2- 172.21.15.67:0/1242494291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5680ccce0 0x7fb5681f1f30 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb558000c00 tx=0x7fb5580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:20.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.294+0000 7fb564ff9640 1 -- 172.21.15.67:0/1242494291 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb558040070 con 0x7fb5680ccce0 2024-03-24T00:39:20.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.294+0000 7fb564ff9640 1 -- 172.21.15.67:0/1242494291 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb55803a910 con 0x7fb5680ccce0 2024-03-24T00:39:20.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.294+0000 7fb56e5b3640 1 -- 172.21.15.67:0/1242494291 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5681f2740 con 0x7fb5680ccce0 2024-03-24T00:39:20.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.294+0000 7fb564ff9640 1 -- 172.21.15.67:0/1242494291 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb558035040 con 0x7fb5680ccce0 2024-03-24T00:39:20.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.294+0000 7fb56e5b3640 1 -- 172.21.15.67:0/1242494291 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5681f2d00 con 0x7fb5680ccce0 2024-03-24T00:39:20.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.295+0000 7fb56e5b3640 1 -- 172.21.15.67:0/1242494291 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb56804f400 con 0x7fb5680ccce0 2024-03-24T00:39:20.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.296+0000 7fb564ff9640 1 -- 172.21.15.67:0/1242494291 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb55804b060 con 0x7fb5680ccce0 2024-03-24T00:39:20.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.296+0000 7fb564ff9640 1 --2- 172.21.15.67:0/1242494291 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb55406bbd0 0x7fb55406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:20.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.296+0000 7fb564ff9640 1 -- 172.21.15.67:0/1242494291 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fb5580ac3f0 con 0x7fb5680ccce0 2024-03-24T00:39:20.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.296+0000 7fb567fff640 1 --2- 172.21.15.67:0/1242494291 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb55406bbd0 0x7fb55406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:20.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.297+0000 7fb567fff640 1 --2- 172.21.15.67:0/1242494291 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb55406bbd0 0x7fb55406e090 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb55c007fd0 tx=0x7fb55c007e80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:20.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.301+0000 7fb564ff9640 1 -- 172.21.15.67:0/1242494291 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb558031800 con 0x7fb5680ccce0 2024-03-24T00:39:20.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.484+0000 7fb56e5b3640 1 -- 172.21.15.67:0/1242494291 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb5680d08d0 con 0x7fb5680ccce0 2024-03-24T00:39:20.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.485+0000 7fb564ff9640 1 -- 172.21.15.67:0/1242494291 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18454 (secure 0 0 0) 0x7fb55807a550 con 0x7fb5680ccce0 2024-03-24T00:39:20.486 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:39:20.486 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":25,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:38:58.797179+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:38:50.077085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:38:51.095454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T00:38:54.807117+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:39:20.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.487+0000 7fb56e5b3640 1 -- 172.21.15.67:0/1242494291 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb55406bbd0 msgr2=0x7fb55406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:20.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.487+0000 7fb56e5b3640 1 --2- 172.21.15.67:0/1242494291 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb55406bbd0 0x7fb55406e090 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb55c007fd0 tx=0x7fb55c007e80 comp rx=0 tx=0).stop 2024-03-24T00:39:20.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.487+0000 7fb56e5b3640 1 -- 172.21.15.67:0/1242494291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5680ccce0 msgr2=0x7fb5681f1f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:20.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.487+0000 7fb56e5b3640 1 --2- 172.21.15.67:0/1242494291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5680ccce0 0x7fb5681f1f30 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb558000c00 tx=0x7fb5580046c0 comp rx=0 tx=0).stop 2024-03-24T00:39:20.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.487+0000 7fb56e5b3640 1 -- 172.21.15.67:0/1242494291 shutdown_connections 2024-03-24T00:39:20.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.487+0000 7fb56e5b3640 1 --2- 172.21.15.67:0/1242494291 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb55406bbd0 0x7fb55406e090 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:20.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.487+0000 7fb56e5b3640 1 --2- 172.21.15.67:0/1242494291 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb5680cef40 0x7fb568159150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:20.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.487+0000 7fb56e5b3640 1 --2- 172.21.15.67:0/1242494291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5680ccce0 0x7fb5681f1f30 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:20.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.487+0000 7fb56e5b3640 1 --2- 172.21.15.67:0/1242494291 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb56806dce0 0x7fb5680cc100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:20.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.487+0000 7fb56e5b3640 1 -- 172.21.15.67:0/1242494291 >> 172.21.15.67:0/1242494291 conn(0x7fb5680cf9e0 msgr2=0x7fb5680cd810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:20.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.487+0000 7fb56e5b3640 1 -- 172.21.15.67:0/1242494291 shutdown_connections 2024-03-24T00:39:20.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.487+0000 7fb56e5b3640 1 -- 172.21.15.67:0/1242494291 wait complete. 2024-03-24T00:39:20.499 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:39:20.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.600+0000 7f5663bb2640 1 Processor -- start 2024-03-24T00:39:20.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.600+0000 7f5663bb2640 1 -- start start 2024-03-24T00:39:20.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.601+0000 7f5663bb2640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f565c15dfe0 0x7f565c0ceb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:20.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.601+0000 7f5663bb2640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f565c161bb0 0x7f565c0cf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:20.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.601+0000 7f5663bb2640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f565c162120 con 0x7f565c0b96d0 2024-03-24T00:39:20.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.601+0000 7f5663bb2640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f565c0d21d0 con 0x7f565c15dfe0 2024-03-24T00:39:20.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.601+0000 7f5663bb2640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f565c162400 con 0x7f565c161bb0 2024-03-24T00:39:20.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.601+0000 7f566194e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f565c15dfe0 0x7f565c0ceb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:20.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.601+0000 7f566214f640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f565c0b96d0 0x7f565c0b57c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47960/0 (socket says 172.21.15.67:47960) 2024-03-24T00:39:20.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.601+0000 7f566214f640 1 -- 172.21.15.67:0/156657576 learned_addr learned my addr 172.21.15.67:0/156657576 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:20.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.601+0000 7f566114d640 1 --2- 172.21.15.67:0/156657576 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f565c161bb0 0x7f565c0cf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:20.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.601+0000 7f566194e640 1 -- 172.21.15.67:0/156657576 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f565c161bb0 msgr2=0x7f565c0cf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:20.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.601+0000 7f566194e640 1 --2- 172.21.15.67:0/156657576 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f565c161bb0 0x7f565c0cf050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:20.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.601+0000 7f566194e640 1 -- 172.21.15.67:0/156657576 >> v1:172.21.15.67:6789/0 conn(0x7f565c0b96d0 legacy=0x7f565c0b57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:20.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.601+0000 7f566194e640 1 -- 172.21.15.67:0/156657576 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f565c1626e0 con 0x7f565c15dfe0 2024-03-24T00:39:20.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.602+0000 7f566194e640 1 --2- 172.21.15.67:0/156657576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f565c15dfe0 0x7f565c0ceb10 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f56580097a0 tx=0x7f565802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b31d04a59fdb92d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:20.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.602+0000 7f566094c640 1 -- 172.21.15.67:0/156657576 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5658035020 con 0x7f565c15dfe0 2024-03-24T00:39:20.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.602+0000 7f566094c640 1 -- 172.21.15.67:0/156657576 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5658031830 con 0x7f565c15dfe0 2024-03-24T00:39:20.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.602+0000 7f566094c640 1 -- 172.21.15.67:0/156657576 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5658031c00 con 0x7f565c15dfe0 2024-03-24T00:39:20.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.602+0000 7f5663bb2640 1 -- 172.21.15.67:0/156657576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f565c15dfe0 msgr2=0x7f565c0ceb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:20.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.602+0000 7f5663bb2640 1 --2- 172.21.15.67:0/156657576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f565c15dfe0 0x7f565c0ceb10 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f56580097a0 tx=0x7f565802b8d0 comp rx=0 tx=0).stop 2024-03-24T00:39:20.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.603+0000 7f5663bb2640 1 -- 172.21.15.67:0/156657576 shutdown_connections 2024-03-24T00:39:20.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.603+0000 7f5663bb2640 1 --2- 172.21.15.67:0/156657576 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f565c161bb0 0x7f565c0cf050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:20.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.603+0000 7f5663bb2640 1 --2- 172.21.15.67:0/156657576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f565c15dfe0 0x7f565c0ceb10 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:20.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.603+0000 7f5663bb2640 1 -- 172.21.15.67:0/156657576 >> 172.21.15.67:0/156657576 conn(0x7f565c06c890 msgr2=0x7f565c06cc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:20.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.603+0000 7f5663bb2640 1 -- 172.21.15.67:0/156657576 shutdown_connections 2024-03-24T00:39:20.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.603+0000 7f5663bb2640 1 -- 172.21.15.67:0/156657576 wait complete. 2024-03-24T00:39:20.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.604+0000 7f5663bb2640 1 Processor -- start 2024-03-24T00:39:20.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.604+0000 7f5663bb2640 1 -- start start 2024-03-24T00:39:20.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.604+0000 7f5663bb2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f565c0b96d0 0x7f565c15aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:20.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.604+0000 7f5663bb2640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f565c15dfe0 0x7f565c15b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:20.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.604+0000 7f5663bb2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f565c161bb0 0x7f565c157330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:20.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.604+0000 7f5663bb2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f565c0d1120 con 0x7f565c0b96d0 2024-03-24T00:39:20.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.604+0000 7f5663bb2640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f565c15e3c0 con 0x7f565c161bb0 2024-03-24T00:39:20.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.604+0000 7f5663bb2640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f565c0d12a0 con 0x7f565c15dfe0 2024-03-24T00:39:20.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.605+0000 7f566194e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f565c0b96d0 0x7f565c15aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:20.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.605+0000 7f566194e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f565c0b96d0 0x7f565c15aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36264/0 (socket says 172.21.15.67:36264) 2024-03-24T00:39:20.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.605+0000 7f566194e640 1 -- 172.21.15.67:0/1693513170 learned_addr learned my addr 172.21.15.67:0/1693513170 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:20.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.605+0000 7f566214f640 1 --2- 172.21.15.67:0/1693513170 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f565c161bb0 0x7f565c157330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:20.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.605+0000 7f566114d640 1 --2- 172.21.15.67:0/1693513170 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f565c15dfe0 0x7f565c15b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:20.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.605+0000 7f566194e640 1 -- 172.21.15.67:0/1693513170 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f565c15dfe0 msgr2=0x7f565c15b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:20.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.605+0000 7f566194e640 1 --2- 172.21.15.67:0/1693513170 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f565c15dfe0 0x7f565c15b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:20.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.605+0000 7f566194e640 1 -- 172.21.15.67:0/1693513170 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f565c161bb0 msgr2=0x7f565c157330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:20.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.605+0000 7f566194e640 1 --2- 172.21.15.67:0/1693513170 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f565c161bb0 0x7f565c157330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:20.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.605+0000 7f566194e640 1 -- 172.21.15.67:0/1693513170 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f565c1606a0 con 0x7f565c0b96d0 2024-03-24T00:39:20.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.605+0000 7f566194e640 1 --2- 172.21.15.67:0/1693513170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f565c0b96d0 0x7f565c15aef0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f565802bee0 tx=0x7f56580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:20.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.605+0000 7f56527fc640 1 -- 172.21.15.67:0/1693513170 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5658035040 con 0x7f565c0b96d0 2024-03-24T00:39:20.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.605+0000 7f56527fc640 1 -- 172.21.15.67:0/1693513170 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f565803b920 con 0x7f565c0b96d0 2024-03-24T00:39:20.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.605+0000 7f56527fc640 1 -- 172.21.15.67:0/1693513170 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f565803bd10 con 0x7f565c0b96d0 2024-03-24T00:39:20.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.605+0000 7f5663bb2640 1 -- 172.21.15.67:0/1693513170 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f565c157f10 con 0x7f565c0b96d0 2024-03-24T00:39:20.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.605+0000 7f5663bb2640 1 -- 172.21.15.67:0/1693513170 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f565c1f7940 con 0x7f565c0b96d0 2024-03-24T00:39:20.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.606+0000 7f5663bb2640 1 -- 172.21.15.67:0/1693513170 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f565c06b180 con 0x7f565c0b96d0 2024-03-24T00:39:20.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.607+0000 7f56527fc640 1 -- 172.21.15.67:0/1693513170 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5658038020 con 0x7f565c0b96d0 2024-03-24T00:39:20.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.607+0000 7f56527fc640 1 --2- 172.21.15.67:0/1693513170 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f564006bbd0 0x7f564006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:20.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.607+0000 7f56527fc640 1 -- 172.21.15.67:0/1693513170 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f56580ace30 con 0x7f565c0b96d0 2024-03-24T00:39:20.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.607+0000 7f566114d640 1 --2- 172.21.15.67:0/1693513170 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f564006bbd0 0x7f564006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:20.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.607+0000 7f566114d640 1 --2- 172.21.15.67:0/1693513170 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f564006bbd0 0x7f564006e090 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f565c15acb0 tx=0x7f564c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:20.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.611+0000 7f56527fc640 1 -- 172.21.15.67:0/1693513170 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f565c1606a0 con 0x7f565c0b96d0 2024-03-24T00:39:20.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.834+0000 7f5663bb2640 1 -- 172.21.15.67:0/1693513170 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f565c04f400 con 0x7f565c0b96d0 2024-03-24T00:39:20.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.835+0000 7f56527fc640 1 -- 172.21.15.67:0/1693513170 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4970 (secure 0 0 0) 0x7f5658031800 con 0x7f565c0b96d0 2024-03-24T00:39:20.835 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:39:20.835 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4263,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/837528243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":837528243},{"type":"v1","addr":"172.21.15.67:6834","nonce":837528243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3987071456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":3987071456},{"type":"v1","addr":"172.21.15.67:6835","nonce":3987071456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4306,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/3953492281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3953492281},{"type":"v1","addr":"172.21.15.72:6837","nonce":3953492281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:38:52.127871+0000","modified":"2024-03-24T00:39:08.187262+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4309},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4309":{"gid":4309,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6836/3737698654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3737698654},{"type":"v1","addr":"172.21.15.72:6836","nonce":3737698654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4309,"qdb_cluster":[4309]},"id":1}]} 2024-03-24T00:39:20.836 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 9 2024-03-24T00:39:20.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.836+0000 7f5663bb2640 1 -- 172.21.15.67:0/1693513170 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f564006bbd0 msgr2=0x7f564006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:20.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.836+0000 7f5663bb2640 1 --2- 172.21.15.67:0/1693513170 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f564006bbd0 0x7f564006e090 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f565c15acb0 tx=0x7f564c009190 comp rx=0 tx=0).stop 2024-03-24T00:39:20.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.837+0000 7f5663bb2640 1 -- 172.21.15.67:0/1693513170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f565c0b96d0 msgr2=0x7f565c15aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:20.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.837+0000 7f5663bb2640 1 --2- 172.21.15.67:0/1693513170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f565c0b96d0 0x7f565c15aef0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f565802bee0 tx=0x7f56580046c0 comp rx=0 tx=0).stop 2024-03-24T00:39:20.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.837+0000 7f5663bb2640 1 -- 172.21.15.67:0/1693513170 shutdown_connections 2024-03-24T00:39:20.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.837+0000 7f5663bb2640 1 --2- 172.21.15.67:0/1693513170 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f564006bbd0 0x7f564006e090 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:20.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.837+0000 7f5663bb2640 1 --2- 172.21.15.67:0/1693513170 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f565c161bb0 0x7f565c157330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:20.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.837+0000 7f5663bb2640 1 --2- 172.21.15.67:0/1693513170 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f565c15dfe0 0x7f565c15b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:20.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.837+0000 7f5663bb2640 1 --2- 172.21.15.67:0/1693513170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f565c0b96d0 0x7f565c15aef0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:20.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.837+0000 7f5663bb2640 1 -- 172.21.15.67:0/1693513170 >> 172.21.15.67:0/1693513170 conn(0x7f565c06c890 msgr2=0x7f565c0b37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:20.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.837+0000 7f5663bb2640 1 -- 172.21.15.67:0/1693513170 shutdown_connections 2024-03-24T00:39:20.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:20.837+0000 7f5663bb2640 1 -- 172.21.15.67:0/1693513170 wait complete. 2024-03-24T00:39:21.250 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 9, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:38:52.127871+0000', 'modified': '2024-03-24T00:39:08.187262+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4309}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4309': {'gid': 4309, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6836/3737698654', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 3737698654}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 3737698654}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4309, 'qdb_cluster': [4309]} 2024-03-24T00:39:21.250 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-24T00:39:21.250 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-24T00:39:21.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.354+0000 7f9f39fc6640 1 Processor -- start 2024-03-24T00:39:21.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.354+0000 7f9f39fc6640 1 -- start start 2024-03-24T00:39:21.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.355+0000 7f9f39fc6640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9f340d32d0 0x7f9f340d3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:21.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.355+0000 7f9f39fc6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9f340b6dc0 0x7f9f340b7250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:21.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.355+0000 7f9f39fc6640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f340b9cc0 con 0x7f9f3415e0f0 2024-03-24T00:39:21.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.355+0000 7f9f39fc6640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f340b7790 con 0x7f9f340d32d0 2024-03-24T00:39:21.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.355+0000 7f9f39fc6640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f340d2230 con 0x7f9f340b6dc0 2024-03-24T00:39:21.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.355+0000 7f9f32ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9f340d32d0 0x7f9f340d3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:21.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.355+0000 7f9f32ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9f340d32d0 0x7f9f340d3740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36270/0 (socket says 172.21.15.67:36270) 2024-03-24T00:39:21.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.355+0000 7f9f337fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f9f3415e0f0 0x7f9f340b95b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:50008/0 (socket says 172.21.15.67:50008) 2024-03-24T00:39:21.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.355+0000 7f9f32ffd640 1 -- 172.21.15.67:0/1289256665 learned_addr learned my addr 172.21.15.67:0/1289256665 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:21.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.355+0000 7f9f33fff640 1 --2- 172.21.15.67:0/1289256665 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9f340b6dc0 0x7f9f340b7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:21.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.356+0000 7f9f327fc640 1 -- 172.21.15.67:0/1289256665 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4020961161 0 0) 0x7f9f340b9cc0 con 0x7f9f3415e0f0 2024-03-24T00:39:21.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.356+0000 7f9f327fc640 1 -- 172.21.15.67:0/1289256665 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f1c003520 con 0x7f9f3415e0f0 2024-03-24T00:39:21.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.356+0000 7f9f32ffd640 1 -- 172.21.15.67:0/1289256665 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9f340b6dc0 msgr2=0x7f9f340b7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:21.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.356+0000 7f9f32ffd640 1 --2- 172.21.15.67:0/1289256665 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9f340b6dc0 0x7f9f340b7250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:21.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.356+0000 7f9f32ffd640 1 -- 172.21.15.67:0/1289256665 >> v1:172.21.15.72:6789/0 conn(0x7f9f3415e0f0 legacy=0x7f9f340b95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:21.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.356+0000 7f9f32ffd640 1 -- 172.21.15.67:0/1289256665 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f340d2510 con 0x7f9f340d32d0 2024-03-24T00:39:21.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.356+0000 7f9f33fff640 1 --2- 172.21.15.67:0/1289256665 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9f340b6dc0 0x7f9f340b7250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:21.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.356+0000 7f9f32ffd640 1 --2- 172.21.15.67:0/1289256665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f340d32d0 0x7f9f340d3740 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f9f28009850 tx=0x7f9f2802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f43fca333eec9a10 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:21.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.357+0000 7f9f327fc640 1 -- 172.21.15.67:0/1289256665 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f28004230 con 0x7f9f340d32d0 2024-03-24T00:39:21.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.357+0000 7f9f327fc640 1 -- 172.21.15.67:0/1289256665 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f280043d0 con 0x7f9f340d32d0 2024-03-24T00:39:21.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.357+0000 7f9f327fc640 1 -- 172.21.15.67:0/1289256665 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f28003810 con 0x7f9f340d32d0 2024-03-24T00:39:21.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.357+0000 7f9f39fc6640 1 -- 172.21.15.67:0/1289256665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f340d32d0 msgr2=0x7f9f340d3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:21.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.357+0000 7f9f39fc6640 1 --2- 172.21.15.67:0/1289256665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f340d32d0 0x7f9f340d3740 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f9f28009850 tx=0x7f9f2802b960 comp rx=0 tx=0).stop 2024-03-24T00:39:21.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.357+0000 7f9f39fc6640 1 -- 172.21.15.67:0/1289256665 shutdown_connections 2024-03-24T00:39:21.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.357+0000 7f9f39fc6640 1 --2- 172.21.15.67:0/1289256665 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9f340b6dc0 0x7f9f340b7250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:21.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.357+0000 7f9f39fc6640 1 --2- 172.21.15.67:0/1289256665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f340d32d0 0x7f9f340d3740 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:21.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.357+0000 7f9f39fc6640 1 -- 172.21.15.67:0/1289256665 >> 172.21.15.67:0/1289256665 conn(0x7f9f34072b60 msgr2=0x7f9f34072f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:21.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.357+0000 7f9f39fc6640 1 -- 172.21.15.67:0/1289256665 shutdown_connections 2024-03-24T00:39:21.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.357+0000 7f9f39fc6640 1 -- 172.21.15.67:0/1289256665 wait complete. 2024-03-24T00:39:21.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.358+0000 7f9f39fc6640 1 Processor -- start 2024-03-24T00:39:21.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.358+0000 7f9f39fc6640 1 -- start start 2024-03-24T00:39:21.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.359+0000 7f9f39fc6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f340b6dc0 0x7f9f340aa720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:21.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.359+0000 7f9f39fc6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f340d32d0 0x7f9f340cad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:21.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.359+0000 7f9f39fc6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9f3415e0f0 0x7f9f3415f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:21.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.359+0000 7f9f39fc6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f340d2db0 con 0x7f9f340d32d0 2024-03-24T00:39:21.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.359+0000 7f9f39fc6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f340d2ad0 con 0x7f9f340b6dc0 2024-03-24T00:39:21.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.359+0000 7f9f39fc6640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f340b31d0 con 0x7f9f3415e0f0 2024-03-24T00:39:21.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.359+0000 7f9f32ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f340d32d0 0x7f9f340cad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:21.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.359+0000 7f9f32ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f340d32d0 0x7f9f340cad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36282/0 (socket says 172.21.15.67:36282) 2024-03-24T00:39:21.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.359+0000 7f9f32ffd640 1 -- 172.21.15.67:0/415249888 learned_addr learned my addr 172.21.15.67:0/415249888 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:21.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.359+0000 7f9f337fe640 1 --2- 172.21.15.67:0/415249888 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f340b6dc0 0x7f9f340aa720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:21.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.359+0000 7f9f33fff640 1 --2- 172.21.15.67:0/415249888 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9f3415e0f0 0x7f9f3415f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:21.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.360+0000 7f9f32ffd640 1 -- 172.21.15.67:0/415249888 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9f3415e0f0 msgr2=0x7f9f3415f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:21.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.360+0000 7f9f32ffd640 1 --2- 172.21.15.67:0/415249888 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9f3415e0f0 0x7f9f3415f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:21.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.360+0000 7f9f32ffd640 1 -- 172.21.15.67:0/415249888 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f340b6dc0 msgr2=0x7f9f340aa720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:21.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.360+0000 7f9f32ffd640 1 --2- 172.21.15.67:0/415249888 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f340b6dc0 0x7f9f340aa720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:21.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.360+0000 7f9f32ffd640 1 -- 172.21.15.67:0/415249888 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f3414cf70 con 0x7f9f340d32d0 2024-03-24T00:39:21.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.360+0000 7f9f32ffd640 1 --2- 172.21.15.67:0/415249888 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f340d32d0 0x7f9f340cad90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9f2802b9a0 tx=0x7f9f280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:21.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.360+0000 7f9f1bfff640 1 -- 172.21.15.67:0/415249888 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f28048070 con 0x7f9f340d32d0 2024-03-24T00:39:21.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.360+0000 7f9f1bfff640 1 -- 172.21.15.67:0/415249888 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f28005e10 con 0x7f9f340d32d0 2024-03-24T00:39:21.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.360+0000 7f9f39fc6640 1 -- 172.21.15.67:0/415249888 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f3414d250 con 0x7f9f340d32d0 2024-03-24T00:39:21.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.360+0000 7f9f1bfff640 1 -- 172.21.15.67:0/415249888 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f28003e10 con 0x7f9f340d32d0 2024-03-24T00:39:21.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.360+0000 7f9f39fc6640 1 -- 172.21.15.67:0/415249888 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f3414d810 con 0x7f9f340d32d0 2024-03-24T00:39:21.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.361+0000 7f9f1bfff640 1 -- 172.21.15.67:0/415249888 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9f2804b070 con 0x7f9f340d32d0 2024-03-24T00:39:21.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.362+0000 7f9f1bfff640 1 --2- 172.21.15.67:0/415249888 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9f2006bbd0 0x7f9f2006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:21.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.362+0000 7f9f1bfff640 1 -- 172.21.15.67:0/415249888 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f9f280ac550 con 0x7f9f340d32d0 2024-03-24T00:39:21.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.362+0000 7f9f337fe640 1 --2- 172.21.15.67:0/415249888 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9f2006bbd0 0x7f9f2006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:21.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.362+0000 7f9f337fe640 1 --2- 172.21.15.67:0/415249888 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9f2006bbd0 0x7f9f2006e090 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9f24000f60 tx=0x7f9f24007cf0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:21.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.363+0000 7f9f3cbb4740 1 Processor -- start 2024-03-24T00:39:21.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.363+0000 7f9f3cbb4740 1 -- start start 2024-03-24T00:39:21.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.364+0000 7f9f3cbb4740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d964eafc40 0x55d964ed6770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:21.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.364+0000 7f9f3cbb4740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d964ed6cb0 0x55d964eaec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:21.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.364+0000 7f9f3cbb4740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d964eb0020 0x55d964eaf1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:21.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.364+0000 7f9f3cbb4740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d964e13c20 con 0x55d964eb0020 2024-03-24T00:39:21.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.364+0000 7f9f3cbb4740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d964dbca40 con 0x55d964eafc40 2024-03-24T00:39:21.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.364+0000 7f9f3cbb4740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d964e0fcc0 con 0x55d964ed6cb0 2024-03-24T00:39:21.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.364+0000 7f9f33fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d964eb0020 0x55d964eaf1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:21.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.364+0000 7f9f32ffd640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d964ed6cb0 0x55d964eaec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:21.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.364+0000 7f9f337fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d964eafc40 0x55d964ed6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:21.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.364+0000 7f9f33fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d964eb0020 0x55d964eaf1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36298/0 (socket says 172.21.15.67:36298) 2024-03-24T00:39:21.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.364+0000 7f9f33fff640 1 -- 172.21.15.67:0/1913784383 learned_addr learned my addr 172.21.15.67:0/1913784383 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:21.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.364+0000 7f9f32ffd640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d964ed6cb0 0x55d964eaec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.67:43004/0 (socket says 172.21.15.67:43004) 2024-03-24T00:39:21.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.364+0000 7f9f33fff640 1 -- 172.21.15.67:0/1913784383 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d964ed6cb0 msgr2=0x55d964eaec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:21.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.364+0000 7f9f33fff640 1 --2- 172.21.15.67:0/1913784383 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d964ed6cb0 0x55d964eaec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:21.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.364+0000 7f9f33fff640 1 -- 172.21.15.67:0/1913784383 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d964eafc40 msgr2=0x55d964ed6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:21.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.364+0000 7f9f33fff640 1 --2- 172.21.15.67:0/1913784383 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d964eafc40 0x55d964ed6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:21.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.364+0000 7f9f33fff640 1 -- 172.21.15.67:0/1913784383 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d964ddef90 con 0x55d964eb0020 2024-03-24T00:39:21.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.364+0000 7f9f32ffd640 1 --2- 172.21.15.67:0/1913784383 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d964ed6cb0 0x55d964eaec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:21.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.364+0000 7f9f337fe640 1 --2- 172.21.15.67:0/1913784383 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d964eafc40 0x55d964ed6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:21.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.364+0000 7f9f33fff640 1 --2- 172.21.15.67:0/1913784383 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d964eb0020 0x55d964eaf1c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c011580 tx=0x7f9f2c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9548aa59562f6f0b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:21.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.364+0000 7f9f197fa640 1 -- 172.21.15.67:0/1913784383 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f2c003b90 con 0x55d964eb0020 2024-03-24T00:39:21.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.364+0000 7f9f197fa640 1 -- 172.21.15.67:0/1913784383 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f2c003d30 con 0x55d964eb0020 2024-03-24T00:39:21.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.364+0000 7f9f197fa640 1 -- 172.21.15.67:0/1913784383 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f2c004220 con 0x55d964eb0020 2024-03-24T00:39:21.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.365+0000 7f9f3cbb4740 1 -- 172.21.15.67:0/1913784383 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d964eb0020 msgr2=0x55d964eaf1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:21.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.365+0000 7f9f3cbb4740 1 --2- 172.21.15.67:0/1913784383 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d964eb0020 0x55d964eaf1c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c011580 tx=0x7f9f2c009750 comp rx=0 tx=0).stop 2024-03-24T00:39:21.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.365+0000 7f9f3cbb4740 1 -- 172.21.15.67:0/1913784383 shutdown_connections 2024-03-24T00:39:21.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.365+0000 7f9f3cbb4740 1 --2- 172.21.15.67:0/1913784383 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d964eb0020 0x55d964eaf1c0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:21.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.365+0000 7f9f3cbb4740 1 --2- 172.21.15.67:0/1913784383 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d964ed6cb0 0x55d964eaec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:21.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.365+0000 7f9f3cbb4740 1 --2- 172.21.15.67:0/1913784383 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d964eafc40 0x55d964ed6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:21.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.365+0000 7f9f3cbb4740 1 -- 172.21.15.67:0/1913784383 >> 172.21.15.67:0/1913784383 conn(0x55d964e76a60 msgr2=0x55d964ef4960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:21.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.365+0000 7f9f3cbb4740 1 -- 172.21.15.67:0/1913784383 shutdown_connections 2024-03-24T00:39:21.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.365+0000 7f9f3cbb4740 1 -- 172.21.15.67:0/1913784383 wait complete. 2024-03-24T00:39:21.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.366+0000 7f9f3cbb4740 1 -- start start 2024-03-24T00:39:21.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.366+0000 7f9f3cbb4740 1 Processor -- start 2024-03-24T00:39:21.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.366+0000 7f9f3cbb4740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d964eafc40 0x55d964ed2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:21.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.367+0000 7f9f3cbb4740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d964eb0020 0x55d964ed26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:21.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.367+0000 7f9f3cbb4740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d964ed4670 0x55d964ed2c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:21.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.367+0000 7f9f3cbb4740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d964ed62e0 con 0x55d964eafc40 2024-03-24T00:39:21.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.367+0000 7f9f3cbb4740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d964dcfe90 con 0x55d964ed4670 2024-03-24T00:39:21.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.367+0000 7f9f3cbb4740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d964ecf060 con 0x55d964eb0020 2024-03-24T00:39:21.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.367+0000 7f9f337fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d964eafc40 0x55d964ed2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:21.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.367+0000 7f9f3cbb4740 10 client.0 _ll_register_callbacks cb 0x55d964b27ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:39:21.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.367+0000 7f9f337fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d964eafc40 0x55d964ed2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36306/0 (socket says 172.21.15.67:36306) 2024-03-24T00:39:21.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.367+0000 7f9f337fe640 1 -- 172.21.15.67:0/2135582384 learned_addr learned my addr 172.21.15.67:0/2135582384 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:21.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.367+0000 7f9f33fff640 1 --2- 172.21.15.67:0/2135582384 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d964ed4670 0x55d964ed2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:21.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.367+0000 7f9f32ffd640 1 --2- 172.21.15.67:0/2135582384 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d964eb0020 0x55d964ed26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:21.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.367+0000 7f9f337fe640 1 -- 172.21.15.67:0/2135582384 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d964eb0020 msgr2=0x55d964ed26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:21.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.367+0000 7f9f337fe640 1 --2- 172.21.15.67:0/2135582384 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d964eb0020 0x55d964ed26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:21.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.367+0000 7f9f337fe640 1 -- 172.21.15.67:0/2135582384 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d964ed4670 msgr2=0x55d964ed2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:21.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.367+0000 7f9f337fe640 1 --2- 172.21.15.67:0/2135582384 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d964ed4670 0x55d964ed2c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:21.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.367+0000 7f9f337fe640 1 -- 172.21.15.67:0/2135582384 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d964ddc9d0 con 0x55d964eafc40 2024-03-24T00:39:21.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.367+0000 7f9f33fff640 1 --2- 172.21.15.67:0/2135582384 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d964ed4670 0x55d964ed2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:21.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.367+0000 7f9f337fe640 1 --2- 172.21.15.67:0/2135582384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d964eafc40 0x55d964ed2020 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9f240136d0 tx=0x7f9f24000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3400850a4c3b8d2d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:21.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.367+0000 7f9f127fc640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:39:21.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.367+0000 7f9f127fc640 1 -- 172.21.15.67:0/2135582384 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f24020580 con 0x55d964eafc40 2024-03-24T00:39:21.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.367+0000 7f9f127fc640 1 -- 172.21.15.67:0/2135582384 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f9f24018af0 con 0x55d964eafc40 2024-03-24T00:39:21.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.367+0000 7f9f3cbb4740 1 -- 172.21.15.67:0/2135582384 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9f2c003d30 con 0x55d964eafc40 2024-03-24T00:39:21.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.367+0000 7f9f127fc640 1 -- 172.21.15.67:0/2135582384 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9f24018cf0 con 0x55d964eafc40 2024-03-24T00:39:21.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.367+0000 7f9f3cbb4740 10 client.4899 fetch_fsmap learned FSMap version 9 2024-03-24T00:39:21.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.367+0000 7f9f3cbb4740 1 -- 172.21.15.67:0/2135582384 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) v3 -- 0x55d964b282d0 con 0x55d964eafc40 2024-03-24T00:39:21.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.368+0000 7f9f127fc640 1 -- 172.21.15.67:0/2135582384 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f2c003d30 con 0x55d964eafc40 2024-03-24T00:39:21.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.368+0000 7f9f127fc640 1 -- 172.21.15.67:0/2135582384 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 9) v1 ==== 3322+0+0 (secure 0 0 0) 0x7f9f24020720 con 0x55d964eafc40 2024-03-24T00:39:21.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.368+0000 7f9f3cbb4740 10 client.4899 fetch_fsmap finished waiting for FSMap version 9 2024-03-24T00:39:21.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.368+0000 7f9f3cbb4740 10 client.4899 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T00:39:21.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.368+0000 7f9f3cbb4740 20 client.4899 populate_metadata read hostname 'smithi067' 2024-03-24T00:39:21.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.368+0000 7f9f3cbb4740 1 --2- 172.21.15.67:0/2135582384 >> [v2:172.21.15.72:6835/3737698654,v1:172.21.15.72:6836/3737698654] conn(0x55d964ecffb0 0x55d964f04d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:21.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.368+0000 7f9f3cbb4740 4 client.4899 mds_command: new command op to 4309 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:39:21.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.368+0000 7f9f3cbb4740 1 -- 172.21.15.67:0/2135582384 --> [v2:172.21.15.72:6835/3737698654,v1:172.21.15.72:6836/3737698654] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d964e21160 con 0x55d964ecffb0 2024-03-24T00:39:21.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.368+0000 7f9f32ffd640 1 --2- 172.21.15.67:0/2135582384 >> [v2:172.21.15.72:6835/3737698654,v1:172.21.15.72:6836/3737698654] conn(0x55d964ecffb0 0x55d964f04d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:21.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.368+0000 7f9f32ffd640 1 --2- 172.21.15.67:0/2135582384 >> [v2:172.21.15.72:6835/3737698654,v1:172.21.15.72:6836/3737698654] conn(0x55d964ecffb0 0x55d964f04d60 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eecd01036ecadc99 server_cookie=a343f8b179eb6c78 in_seq=0 out_seq=0 2024-03-24T00:39:21.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.368+0000 7f9f127fc640 10 client.4899 ms_handle_connect on v2:172.21.15.72:6835/3737698654 2024-03-24T00:39:21.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.369+0000 7f9f127fc640 1 -- 172.21.15.67:0/2135582384 <== mds.0 v2:172.21.15.72:6835/3737698654 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d964e21160 con 0x55d964ecffb0 2024-03-24T00:39:21.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.369+0000 7f9f127fc640 10 client.4899 handle_command_reply: tid=0 2024-03-24T00:39:21.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.369+0000 7f9f3cbb4740 1 client.4899 shutdown 2024-03-24T00:39:21.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.369+0000 7f9f3cbb4740 1 -- 172.21.15.67:0/2135582384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d964eafc40 msgr2=0x55d964ed2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:21.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.369+0000 7f9f3cbb4740 1 --2- 172.21.15.67:0/2135582384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d964eafc40 0x55d964ed2020 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9f240136d0 tx=0x7f9f24000ed0 comp rx=0 tx=0).stop 2024-03-24T00:39:21.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.369+0000 7f9f3cbb4740 1 -- 172.21.15.67:0/2135582384 shutdown_connections 2024-03-24T00:39:21.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.369+0000 7f9f3cbb4740 1 --2- 172.21.15.67:0/2135582384 >> [v2:172.21.15.72:6835/3737698654,v1:172.21.15.72:6836/3737698654] conn(0x55d964ecffb0 0x55d964f04d60 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:21.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.369+0000 7f9f3cbb4740 1 --2- 172.21.15.67:0/2135582384 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d964ed4670 0x55d964ed2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:21.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.369+0000 7f9f3cbb4740 1 --2- 172.21.15.67:0/2135582384 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d964eb0020 0x55d964ed26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:21.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.369+0000 7f9f127fc640 0 client.4899 ms_handle_reset on v2:172.21.15.72:6835/3737698654 2024-03-24T00:39:21.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.369+0000 7f9f3cbb4740 1 --2- 172.21.15.67:0/2135582384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d964eafc40 0x55d964ed2020 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:21.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.369+0000 7f9f3cbb4740 1 -- 172.21.15.67:0/2135582384 >> 172.21.15.67:0/2135582384 conn(0x55d964e76a60 msgr2=0x55d964ed7940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:21.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.369+0000 7f9f3cbb4740 1 -- 172.21.15.67:0/2135582384 shutdown_connections 2024-03-24T00:39:21.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.369+0000 7f9f3cbb4740 1 -- 172.21.15.67:0/2135582384 wait complete. 2024-03-24T00:39:21.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.369+0000 7f9f3cbb4740 20 client.4899 trim_cache size 0 max 16384 2024-03-24T00:39:21.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.386+0000 7f9f3cbb4740 1 Processor -- start 2024-03-24T00:39:21.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.386+0000 7f9f3cbb4740 1 -- start start 2024-03-24T00:39:21.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.387+0000 7f9f3cbb4740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d964eafc40 0x55d964ed74d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:21.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.387+0000 7f9f3cbb4740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d964eb0020 0x55d964ed7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:21.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.387+0000 7f9f3cbb4740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d964ecffb0 0x55d964f04bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:21.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.387+0000 7f9f3cbb4740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f342007d0 con 0x55d964ecffb0 2024-03-24T00:39:21.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.387+0000 7f9f3cbb4740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f3414f8d0 con 0x55d964eb0020 2024-03-24T00:39:21.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.387+0000 7f9f3cbb4740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f342001d0 con 0x55d964eafc40 2024-03-24T00:39:21.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.387+0000 7f9f33fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d964ecffb0 0x55d964f04bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:21.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.387+0000 7f9f33fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d964ecffb0 0x55d964f04bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36318/0 (socket says 172.21.15.67:36318) 2024-03-24T00:39:21.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.387+0000 7f9f33fff640 1 -- 172.21.15.67:0/2431395082 learned_addr learned my addr 172.21.15.67:0/2431395082 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:21.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.387+0000 7f9f337fe640 1 --2- 172.21.15.67:0/2431395082 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d964eafc40 0x55d964ed74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:21.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.387+0000 7f9f32ffd640 1 --2- 172.21.15.67:0/2431395082 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d964eb0020 0x55d964ed7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:21.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.387+0000 7f9f33fff640 1 -- 172.21.15.67:0/2431395082 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d964eafc40 msgr2=0x55d964ed74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:21.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.387+0000 7f9f33fff640 1 --2- 172.21.15.67:0/2431395082 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d964eafc40 0x55d964ed74d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:21.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.387+0000 7f9f33fff640 1 -- 172.21.15.67:0/2431395082 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d964eb0020 msgr2=0x55d964ed7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:21.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.387+0000 7f9f33fff640 1 --2- 172.21.15.67:0/2431395082 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d964eb0020 0x55d964ed7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:21.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.387+0000 7f9f33fff640 1 -- 172.21.15.67:0/2431395082 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d964ecf1e0 con 0x55d964ecffb0 2024-03-24T00:39:21.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.387+0000 7f9f33fff640 1 --2- 172.21.15.67:0/2431395082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d964ecffb0 0x55d964f04bc0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c019000 tx=0x7f9f2c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a18d6e5a125ca6eb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:21.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.388+0000 7f9f18ff9640 1 -- 172.21.15.67:0/2431395082 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f2c02c070 con 0x55d964ecffb0 2024-03-24T00:39:21.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.388+0000 7f9f18ff9640 1 -- 172.21.15.67:0/2431395082 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f2c015ad0 con 0x55d964ecffb0 2024-03-24T00:39:21.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.388+0000 7f9f18ff9640 1 -- 172.21.15.67:0/2431395082 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f2c027400 con 0x55d964ecffb0 2024-03-24T00:39:21.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.388+0000 7f9f3cbb4740 1 -- 172.21.15.67:0/2431395082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d964ecffb0 msgr2=0x55d964f04bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:21.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.388+0000 7f9f3cbb4740 1 --2- 172.21.15.67:0/2431395082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d964ecffb0 0x55d964f04bc0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c019000 tx=0x7f9f2c016000 comp rx=0 tx=0).stop 2024-03-24T00:39:21.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.388+0000 7f9f3cbb4740 1 -- 172.21.15.67:0/2431395082 shutdown_connections 2024-03-24T00:39:21.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.388+0000 7f9f3cbb4740 1 --2- 172.21.15.67:0/2431395082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d964ecffb0 0x55d964f04bc0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:21.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.388+0000 7f9f3cbb4740 1 --2- 172.21.15.67:0/2431395082 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d964eb0020 0x55d964ed7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:21.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.388+0000 7f9f3cbb4740 1 --2- 172.21.15.67:0/2431395082 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d964eafc40 0x55d964ed74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:21.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.388+0000 7f9f3cbb4740 1 -- 172.21.15.67:0/2431395082 >> 172.21.15.67:0/2431395082 conn(0x55d964e76a60 msgr2=0x55d964ed5860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:21.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.388+0000 7f9f3cbb4740 1 -- 172.21.15.67:0/2431395082 shutdown_connections 2024-03-24T00:39:21.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.388+0000 7f9f3cbb4740 1 -- 172.21.15.67:0/2431395082 wait complete. 2024-03-24T00:39:21.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.389+0000 7f9f3cbb4740 1 -- start start 2024-03-24T00:39:21.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.389+0000 7f9f3cbb4740 1 Processor -- start 2024-03-24T00:39:21.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.390+0000 7f9f3cbb4740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d964eafc40 0x55d964f637b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:21.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.390+0000 7f9f3cbb4740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d964eb0020 0x55d964ea8a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:21.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.390+0000 7f9f3cbb4740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d964ecffb0 0x55d964ea8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:21.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.390+0000 7f9f3cbb4740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d964ecfe30 con 0x55d964eb0020 2024-03-24T00:39:21.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.390+0000 7f9f3cbb4740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d964ecf450 con 0x55d964eafc40 2024-03-24T00:39:21.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.390+0000 7f9f3cbb4740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d964ed0400 con 0x55d964ecffb0 2024-03-24T00:39:21.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.390+0000 7f9f3cbb4740 10 client.0 _ll_register_callbacks cb 0x7f9f240073a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:39:21.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.390+0000 7f9f32ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d964eb0020 0x55d964ea8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:21.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.390+0000 7f9f337fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d964eafc40 0x55d964f637b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:21.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.390+0000 7f9f32ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d964eb0020 0x55d964ea8a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36322/0 (socket says 172.21.15.67:36322) 2024-03-24T00:39:21.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.390+0000 7f9f337fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d964eafc40 0x55d964f637b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:47926/0 (socket says 172.21.15.67:47926) 2024-03-24T00:39:21.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.390+0000 7f9f32ffd640 1 -- 172.21.15.67:0/3735665326 learned_addr learned my addr 172.21.15.67:0/3735665326 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:21.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.390+0000 7f9f33fff640 1 --2- 172.21.15.67:0/3735665326 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d964ecffb0 0x55d964ea8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:21.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.390+0000 7f9f337fe640 1 -- 172.21.15.67:0/3735665326 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d964ecffb0 msgr2=0x55d964ea8fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:21.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.390+0000 7f9f337fe640 1 --2- 172.21.15.67:0/3735665326 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d964ecffb0 0x55d964ea8fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:21.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.390+0000 7f9f337fe640 1 -- 172.21.15.67:0/3735665326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d964eb0020 msgr2=0x55d964ea8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:21.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.390+0000 7f9f337fe640 1 --2- 172.21.15.67:0/3735665326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d964eb0020 0x55d964ea8a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:21.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.390+0000 7f9f337fe640 1 -- 172.21.15.67:0/3735665326 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d964edde50 con 0x55d964eafc40 2024-03-24T00:39:21.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.390+0000 7f9f32ffd640 1 --2- 172.21.15.67:0/3735665326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d964eb0020 0x55d964ea8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:21.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.390+0000 7f9f337fe640 1 --2- 172.21.15.67:0/3735665326 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d964eafc40 0x55d964f637b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9f24001620 tx=0x7f9f24014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=becd9410da3450d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:21.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.390+0000 7f9f127fc640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-24T00:39:21.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.390+0000 7f9f127fc640 1 -- 172.21.15.67:0/3735665326 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f2403a060 con 0x55d964eafc40 2024-03-24T00:39:21.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.391+0000 7f9f127fc640 1 -- 172.21.15.67:0/3735665326 <== mon.1 v2:172.21.15.72:3300/0 2 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f9f24018380 con 0x55d964eafc40 2024-03-24T00:39:21.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.391+0000 7f9f3cbb4740 1 -- 172.21.15.67:0/3735665326 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9f2c015ad0 con 0x55d964eafc40 2024-03-24T00:39:21.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.391+0000 7f9f127fc640 1 -- 172.21.15.67:0/3735665326 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9f24040050 con 0x55d964eafc40 2024-03-24T00:39:21.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.391+0000 7f9f3cbb4740 10 client.4846 fetch_fsmap learned FSMap version 9 2024-03-24T00:39:21.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.391+0000 7f9f3cbb4740 1 -- 172.21.15.67:0/3735665326 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) v3 -- 0x55d964bf26b0 con 0x55d964eafc40 2024-03-24T00:39:21.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.391+0000 7f9f127fc640 1 -- 172.21.15.67:0/3735665326 <== mon.1 v2:172.21.15.72:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f2c015ad0 con 0x55d964eafc40 2024-03-24T00:39:21.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.391+0000 7f9f127fc640 1 -- 172.21.15.67:0/3735665326 <== mon.1 v2:172.21.15.72:3300/0 5 ==== fsmap(e 9) v1 ==== 3322+0+0 (secure 0 0 0) 0x7f9f24043020 con 0x55d964eafc40 2024-03-24T00:39:21.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.391+0000 7f9f3cbb4740 10 client.4846 fetch_fsmap finished waiting for FSMap version 9 2024-03-24T00:39:21.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.391+0000 7f9f3cbb4740 10 client.4846 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T00:39:21.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.391+0000 7f9f3cbb4740 20 client.4846 populate_metadata read hostname 'smithi067' 2024-03-24T00:39:21.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.391+0000 7f9f3cbb4740 1 --2- 172.21.15.67:0/3735665326 >> [v2:172.21.15.72:6835/3737698654,v1:172.21.15.72:6836/3737698654] conn(0x55d964ed4670 0x55d964f0e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:21.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.391+0000 7f9f3cbb4740 4 client.4846 mds_command: new command op to 4309 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:39:21.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.391+0000 7f9f3cbb4740 1 -- 172.21.15.67:0/3735665326 --> [v2:172.21.15.72:6835/3737698654,v1:172.21.15.72:6836/3737698654] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55d964e1b4b0 con 0x55d964ed4670 2024-03-24T00:39:21.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.391+0000 7f9f32ffd640 1 --2- 172.21.15.67:0/3735665326 >> [v2:172.21.15.72:6835/3737698654,v1:172.21.15.72:6836/3737698654] conn(0x55d964ed4670 0x55d964f0e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:21.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.391+0000 7f9f32ffd640 1 --2- 172.21.15.67:0/3735665326 >> [v2:172.21.15.72:6835/3737698654,v1:172.21.15.72:6836/3737698654] conn(0x55d964ed4670 0x55d964f0e600 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2e39b508e47cb0b server_cookie=e4d3860162bafc24 in_seq=0 out_seq=0 2024-03-24T00:39:21.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.391+0000 7f9f127fc640 10 client.4846 ms_handle_connect on v2:172.21.15.72:6835/3737698654 2024-03-24T00:39:21.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.392+0000 7f9f127fc640 1 -- 172.21.15.67:0/3735665326 <== mds.0 v2:172.21.15.72:6835/3737698654 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x55d964e1b4b0 con 0x55d964ed4670 2024-03-24T00:39:21.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.392+0000 7f9f127fc640 10 client.4846 handle_command_reply: tid=0 2024-03-24T00:39:21.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.392+0000 7f9f3cbb4740 1 client.4846 shutdown 2024-03-24T00:39:21.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.392+0000 7f9f3cbb4740 1 -- 172.21.15.67:0/3735665326 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d964eafc40 msgr2=0x55d964f637b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:21.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.392+0000 7f9f3cbb4740 1 --2- 172.21.15.67:0/3735665326 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d964eafc40 0x55d964f637b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9f24001620 tx=0x7f9f24014000 comp rx=0 tx=0).stop 2024-03-24T00:39:21.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.392+0000 7f9f3cbb4740 1 -- 172.21.15.67:0/3735665326 shutdown_connections 2024-03-24T00:39:21.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.392+0000 7f9f3cbb4740 1 --2- 172.21.15.67:0/3735665326 >> [v2:172.21.15.72:6835/3737698654,v1:172.21.15.72:6836/3737698654] conn(0x55d964ed4670 0x55d964f0e600 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:21.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.392+0000 7f9f3cbb4740 1 --2- 172.21.15.67:0/3735665326 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d964ecffb0 0x55d964ea8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:21.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.392+0000 7f9f3cbb4740 1 --2- 172.21.15.67:0/3735665326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d964eb0020 0x55d964ea8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:21.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.392+0000 7f9f127fc640 0 client.4846 ms_handle_reset on v2:172.21.15.72:6835/3737698654 2024-03-24T00:39:21.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.392+0000 7f9f3cbb4740 1 --2- 172.21.15.67:0/3735665326 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d964eafc40 0x55d964f637b0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:21.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.392+0000 7f9f3cbb4740 1 -- 172.21.15.67:0/3735665326 >> 172.21.15.67:0/3735665326 conn(0x55d964e76a60 msgr2=0x55d964ed5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:21.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.392+0000 7f9f3cbb4740 1 -- 172.21.15.67:0/3735665326 shutdown_connections 2024-03-24T00:39:21.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.392+0000 7f9f3cbb4740 1 -- 172.21.15.67:0/3735665326 wait complete. 2024-03-24T00:39:21.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.392+0000 7f9f3cbb4740 20 client.4846 trim_cache size 0 max 16384 2024-03-24T00:39:21.399 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:39:21.400 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "de1d0178-808e-49ca-a986-e297e523b976", 2024-03-24T00:39:21.400 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-03-24T00:39:21.400 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 4309, 2024-03-24T00:39:21.400 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-03-24T00:39:21.400 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-03-24T00:39:21.400 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-03-24T00:39:21.400 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 28.630621832999999, 2024-03-24T00:39:21.400 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 9, 2024-03-24T00:39:21.400 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 25, 2024-03-24T00:39:21.400 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 25, 2024-03-24T00:39:21.400 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 35.854058412000001 2024-03-24T00:39:21.400 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:39:21.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.392+0000 7f9f39fc6640 1 -- 172.21.15.67:0/415249888 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9f2006bbd0 msgr2=0x7f9f2006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:21.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.392+0000 7f9f39fc6640 1 --2- 172.21.15.67:0/415249888 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9f2006bbd0 0x7f9f2006e090 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9f24000f60 tx=0x7f9f24007cf0 comp rx=0 tx=0).stop 2024-03-24T00:39:21.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.392+0000 7f9f39fc6640 1 -- 172.21.15.67:0/415249888 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f340d32d0 msgr2=0x7f9f340cad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:21.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.392+0000 7f9f39fc6640 1 --2- 172.21.15.67:0/415249888 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f340d32d0 0x7f9f340cad90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9f2802b9a0 tx=0x7f9f280046c0 comp rx=0 tx=0).stop 2024-03-24T00:39:21.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.392+0000 7f9f39fc6640 1 -- 172.21.15.67:0/415249888 shutdown_connections 2024-03-24T00:39:21.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.392+0000 7f9f39fc6640 1 --2- 172.21.15.67:0/415249888 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9f2006bbd0 0x7f9f2006e090 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:21.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.392+0000 7f9f39fc6640 1 --2- 172.21.15.67:0/415249888 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9f3415e0f0 0x7f9f3415f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:21.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.392+0000 7f9f39fc6640 1 --2- 172.21.15.67:0/415249888 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f340d32d0 0x7f9f340cad90 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:21.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.392+0000 7f9f39fc6640 1 --2- 172.21.15.67:0/415249888 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f340b6dc0 0x7f9f340aa720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:21.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.392+0000 7f9f39fc6640 1 -- 172.21.15.67:0/415249888 >> 172.21.15.67:0/415249888 conn(0x7f9f34072b60 msgr2=0x7f9f340d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:21.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.392+0000 7f9f39fc6640 1 -- 172.21.15.67:0/415249888 shutdown_connections 2024-03-24T00:39:21.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:21.392+0000 7f9f39fc6640 1 -- 172.21.15.67:0/415249888 wait complete. 2024-03-24T00:39:21.404 INFO:teuthology.run_tasks:Running task kclient... 2024-03-24T00:39:21.417 INFO:tasks.kclient:Mounting kernel clients... 2024-03-24T00:39:21.417 INFO:tasks.kclient:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-03-24T00:39:21.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:21.418 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-03-24T00:39:21.432 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-03-24T00:39:21.433 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:21.433 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-24T00:39:21.499 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:21.499 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link delete ceph-brx 2024-03-24T00:39:21.613 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:21.613 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-03-24T00:39:21.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:21.628 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link delete ceph-brx 2024-03-24T00:39:21.697 INFO:teuthology.orchestra.run.smithi067.stderr:Cannot find device "ceph-brx" 2024-03-24T00:39:21.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:39:21.698 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:21.698 DEBUG:teuthology.orchestra.run.smithi072:> ip netns list 2024-03-24T00:39:21.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:21.713 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link delete ceph-brx 2024-03-24T00:39:21.782 INFO:teuthology.orchestra.run.smithi072.stderr:Cannot find device "ceph-brx" 2024-03-24T00:39:21.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:39:21.784 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:21.784 DEBUG:teuthology.orchestra.run.smithi072:> ip netns list 2024-03-24T00:39:21.839 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:21.839 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link delete ceph-brx 2024-03-24T00:39:21.909 INFO:teuthology.orchestra.run.smithi072.stderr:Cannot find device "ceph-brx" 2024-03-24T00:39:21.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:39:21.911 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-03-24T00:39:21.911 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-24T00:39:21.911 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T00:39:21.911 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T00:39:22.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.012+0000 7fcd1f6ff640 1 Processor -- start 2024-03-24T00:39:22.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.013+0000 7fcd1f6ff640 1 -- start start 2024-03-24T00:39:22.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.013+0000 7fcd1f6ff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcd180b49c0 0x7fcd1815b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:22.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.013+0000 7fcd1f6ff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fcd18072a70 0x7fcd180b2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:22.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.013+0000 7fcd1f6ff640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd180b4da0 con 0x7fcd1806f100 2024-03-24T00:39:22.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.013+0000 7fcd1f6ff640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd1806db30 con 0x7fcd180b49c0 2024-03-24T00:39:22.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.013+0000 7fcd1f6ff640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd1806de10 con 0x7fcd18072a70 2024-03-24T00:39:22.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.013+0000 7fcd1d49b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcd180b49c0 0x7fcd1815b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:22.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.014+0000 7fcd1cc9a640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fcd1806f100 0x7fcd18072360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47972/0 (socket says 172.21.15.67:47972) 2024-03-24T00:39:22.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.014+0000 7fcd1cc9a640 1 -- 172.21.15.67:0/3041601648 learned_addr learned my addr 172.21.15.67:0/3041601648 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:22.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.014+0000 7fcd1dc9c640 1 --2- 172.21.15.67:0/3041601648 >> v2:172.21.15.72:3300/0 conn(0x7fcd18072a70 0x7fcd180b2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:22.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.014+0000 7fcd1d49b640 1 -- 172.21.15.67:0/3041601648 >> v2:172.21.15.72:3300/0 conn(0x7fcd18072a70 msgr2=0x7fcd180b2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:22.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.014+0000 7fcd1d49b640 1 --2- 172.21.15.67:0/3041601648 >> v2:172.21.15.72:3300/0 conn(0x7fcd18072a70 0x7fcd180b2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:22.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.014+0000 7fcd1d49b640 1 -- 172.21.15.67:0/3041601648 >> v1:172.21.15.67:6789/0 conn(0x7fcd1806f100 legacy=0x7fcd18072360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:22.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.014+0000 7fcd0ffff640 1 -- 172.21.15.67:0/3041601648 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3879993280 0 0) 0x7fcd180b4da0 con 0x7fcd1806f100 2024-03-24T00:39:22.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.014+0000 7fcd1d49b640 1 -- 172.21.15.67:0/3041601648 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd1806e0f0 con 0x7fcd180b49c0 2024-03-24T00:39:22.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.015+0000 7fcd1d49b640 1 --2- 172.21.15.67:0/3041601648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd180b49c0 0x7fcd1815b430 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fcd18073920 tx=0x7fcd1402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40d5cda6a60aeebb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:22.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.015+0000 7fcd0ffff640 1 -- 172.21.15.67:0/3041601648 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd140315a0 con 0x7fcd180b49c0 2024-03-24T00:39:22.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.015+0000 7fcd0ffff640 1 -- 172.21.15.67:0/3041601648 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd14037070 con 0x7fcd180b49c0 2024-03-24T00:39:22.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.015+0000 7fcd0ffff640 1 -- 172.21.15.67:0/3041601648 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd14031960 con 0x7fcd180b49c0 2024-03-24T00:39:22.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.015+0000 7fcd1f6ff640 1 -- 172.21.15.67:0/3041601648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd180b49c0 msgr2=0x7fcd1815b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:22.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.015+0000 7fcd1f6ff640 1 --2- 172.21.15.67:0/3041601648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd180b49c0 0x7fcd1815b430 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fcd18073920 tx=0x7fcd1402b840 comp rx=0 tx=0).stop 2024-03-24T00:39:22.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.015+0000 7fcd1f6ff640 1 -- 172.21.15.67:0/3041601648 shutdown_connections 2024-03-24T00:39:22.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.015+0000 7fcd1f6ff640 1 --2- 172.21.15.67:0/3041601648 >> v2:172.21.15.72:3300/0 conn(0x7fcd18072a70 0x7fcd180b2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:22.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.015+0000 7fcd1f6ff640 1 --2- 172.21.15.67:0/3041601648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd180b49c0 0x7fcd1815b430 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:22.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.015+0000 7fcd1f6ff640 1 -- 172.21.15.67:0/3041601648 >> 172.21.15.67:0/3041601648 conn(0x7fcd180cf9e0 msgr2=0x7fcd180cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:22.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.015+0000 7fcd1f6ff640 1 -- 172.21.15.67:0/3041601648 shutdown_connections 2024-03-24T00:39:22.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.015+0000 7fcd1f6ff640 1 -- 172.21.15.67:0/3041601648 wait complete. 2024-03-24T00:39:22.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.017+0000 7fcd1f6ff640 1 Processor -- start 2024-03-24T00:39:22.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.017+0000 7fcd1f6ff640 1 -- start start 2024-03-24T00:39:22.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.017+0000 7fcd1f6ff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd1806f100 0x7fcd1813b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:22.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.017+0000 7fcd1f6ff640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcd18072a70 0x7fcd1814f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:22.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.017+0000 7fcd1f6ff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcd180b49c0 0x7fcd1814f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:22.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.017+0000 7fcd1f6ff640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1806f590 con 0x7fcd1806f100 2024-03-24T00:39:22.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.017+0000 7fcd1f6ff640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1806e6b0 con 0x7fcd180b49c0 2024-03-24T00:39:22.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.017+0000 7fcd1f6ff640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd180b3d30 con 0x7fcd18072a70 2024-03-24T00:39:22.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.017+0000 7fcd1d49b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd1806f100 0x7fcd1813b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:22.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.017+0000 7fcd1d49b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd1806f100 0x7fcd1813b270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36338/0 (socket says 172.21.15.67:36338) 2024-03-24T00:39:22.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.017+0000 7fcd1d49b640 1 -- 172.21.15.67:0/4207281417 learned_addr learned my addr 172.21.15.67:0/4207281417 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:22.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.017+0000 7fcd1cc9a640 1 --2- 172.21.15.67:0/4207281417 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcd18072a70 0x7fcd1814f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:22.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.017+0000 7fcd1dc9c640 1 --2- 172.21.15.67:0/4207281417 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcd180b49c0 0x7fcd1814f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:22.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.018+0000 7fcd1d49b640 1 -- 172.21.15.67:0/4207281417 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcd18072a70 msgr2=0x7fcd1814f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:22.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.018+0000 7fcd1d49b640 1 --2- 172.21.15.67:0/4207281417 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcd18072a70 0x7fcd1814f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:22.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.018+0000 7fcd1d49b640 1 -- 172.21.15.67:0/4207281417 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcd180b49c0 msgr2=0x7fcd1814f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:22.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.018+0000 7fcd1d49b640 1 --2- 172.21.15.67:0/4207281417 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcd180b49c0 0x7fcd1814f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:22.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.018+0000 7fcd1d49b640 1 -- 172.21.15.67:0/4207281417 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd1813cb40 con 0x7fcd1806f100 2024-03-24T00:39:22.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.018+0000 7fcd1d49b640 1 --2- 172.21.15.67:0/4207281417 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd1806f100 0x7fcd1813b270 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fcd14000c00 tx=0x7fcd140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:22.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.018+0000 7fcd0dffb640 1 -- 172.21.15.67:0/4207281417 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd14032630 con 0x7fcd1806f100 2024-03-24T00:39:22.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.018+0000 7fcd0dffb640 1 -- 172.21.15.67:0/4207281417 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd140327f0 con 0x7fcd1806f100 2024-03-24T00:39:22.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.018+0000 7fcd0dffb640 1 -- 172.21.15.67:0/4207281417 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd14032be0 con 0x7fcd1806f100 2024-03-24T00:39:22.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.018+0000 7fcd1f6ff640 1 -- 172.21.15.67:0/4207281417 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd1813ce20 con 0x7fcd1806f100 2024-03-24T00:39:22.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.018+0000 7fcd1f6ff640 1 -- 172.21.15.67:0/4207281417 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd1813d3e0 con 0x7fcd1806f100 2024-03-24T00:39:22.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.019+0000 7fcd1f6ff640 1 -- 172.21.15.67:0/4207281417 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd1804f400 con 0x7fcd1806f100 2024-03-24T00:39:22.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.020+0000 7fcd0dffb640 1 -- 172.21.15.67:0/4207281417 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fcd1404d3f0 con 0x7fcd1806f100 2024-03-24T00:39:22.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.020+0000 7fcd0dffb640 1 --2- 172.21.15.67:0/4207281417 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fccfc06bbd0 0x7fccfc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:22.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.020+0000 7fcd0dffb640 1 -- 172.21.15.67:0/4207281417 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fcd1402f030 con 0x7fcd1806f100 2024-03-24T00:39:22.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.020+0000 7fcd1cc9a640 1 --2- 172.21.15.67:0/4207281417 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fccfc06bbd0 0x7fccfc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:22.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.021+0000 7fcd1cc9a640 1 --2- 172.21.15.67:0/4207281417 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fccfc06bbd0 0x7fccfc06e090 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fcd1815d560 tx=0x7fcd08007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:22.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.024+0000 7fcd0dffb640 1 -- 172.21.15.67:0/4207281417 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fcd14040070 con 0x7fcd1806f100 2024-03-24T00:39:22.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.244+0000 7fcd1f6ff640 1 -- 172.21.15.67:0/4207281417 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fcd1806cab0 con 0x7fcd1806f100 2024-03-24T00:39:22.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.244+0000 7fcd0dffb640 1 -- 172.21.15.67:0/4207281417 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7fcd1407a800 con 0x7fcd1806f100 2024-03-24T00:39:22.245 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-24T00:39:22.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.246+0000 7fcd1f6ff640 1 -- 172.21.15.67:0/4207281417 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fccfc06bbd0 msgr2=0x7fccfc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:22.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.246+0000 7fcd1f6ff640 1 --2- 172.21.15.67:0/4207281417 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fccfc06bbd0 0x7fccfc06e090 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fcd1815d560 tx=0x7fcd08007de0 comp rx=0 tx=0).stop 2024-03-24T00:39:22.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.246+0000 7fcd1f6ff640 1 -- 172.21.15.67:0/4207281417 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd1806f100 msgr2=0x7fcd1813b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:22.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.246+0000 7fcd1f6ff640 1 --2- 172.21.15.67:0/4207281417 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd1806f100 0x7fcd1813b270 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fcd14000c00 tx=0x7fcd140046c0 comp rx=0 tx=0).stop 2024-03-24T00:39:22.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.246+0000 7fcd1f6ff640 1 -- 172.21.15.67:0/4207281417 shutdown_connections 2024-03-24T00:39:22.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.246+0000 7fcd1f6ff640 1 --2- 172.21.15.67:0/4207281417 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fccfc06bbd0 0x7fccfc06e090 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:22.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.246+0000 7fcd1f6ff640 1 --2- 172.21.15.67:0/4207281417 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcd180b49c0 0x7fcd1814f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:22.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.246+0000 7fcd1f6ff640 1 --2- 172.21.15.67:0/4207281417 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcd18072a70 0x7fcd1814f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:22.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.246+0000 7fcd1f6ff640 1 --2- 172.21.15.67:0/4207281417 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd1806f100 0x7fcd1813b270 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:22.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.246+0000 7fcd1f6ff640 1 -- 172.21.15.67:0/4207281417 >> 172.21.15.67:0/4207281417 conn(0x7fcd180cf9e0 msgr2=0x7fcd18154290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:22.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.246+0000 7fcd1f6ff640 1 -- 172.21.15.67:0/4207281417 shutdown_connections 2024-03-24T00:39:22.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:22.247+0000 7fcd1f6ff640 1 -- 172.21.15.67:0/4207281417 wait complete. 2024-03-24T00:39:22.258 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-24T00:39:22.258 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T00:39:22.258 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-03-24T00:39:22.258 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-24T00:39:22.258 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T00:39:22.258 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-24T00:39:22.258 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T00:39:22.258 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T00:39:22.258 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:39:22.258 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:22.259 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-03-24T00:39:22.273 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T00:39:22.273 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T00:39:22.273 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T00:39:22.274 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:39:22.274 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-03-24T00:39:22.274 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:39:22.274 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:39:22.274 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-03-24T00:39:22.274 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-03-24T00:39:22.274 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:39:22.274 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-03-24T00:39:22.274 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-03-24T00:39:22.274 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:39:22.274 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-03-24T00:39:22.274 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T00:39:22.274 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-03-24T00:39:22.275 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-24T00:39:22.275 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 42551sec preferred_lft 42551sec 2024-03-24T00:39:22.275 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-03-24T00:39:22.275 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:39:22.275 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-24T00:39:22.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:22.275 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-03-24T00:39:22.275 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-03-24T00:39:22.275 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link add name ceph-brx type bridge 2024-03-24T00:39:22.275 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip addr flush dev ceph-brx 2024-03-24T00:39:22.275 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set ceph-brx up 2024-03-24T00:39:22.275 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-24T00:39:22.275 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-03-24T00:39:22.381 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:22.381 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T00:39:22.407 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-03-24T00:39:22.408 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:22.408 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-03-24T00:39:22.463 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-03-24T00:39:22.463 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T00:39:22.463 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-24T00:39:22.463 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-24T00:39:22.463 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T00:39:22.464 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:22.464 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-03-24T00:39:22.464 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-03-24T00:39:22.464 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-24T00:39:22.464 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-24T00:39:22.464 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-24T00:39:22.464 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-03-24T00:39:22.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:22.558 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-03-24T00:39:22.573 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:22.573 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list-id 2024-03-24T00:39:22.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:22.628 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-03-24T00:39:22.628 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-03-24T00:39:22.628 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:39:22.628 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-24T00:39:22.628 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-03-24T00:39:22.710 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-24T00:39:22.710 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:22.710 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-03-24T00:39:22.710 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-03-24T00:39:22.710 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-24T00:39:22.710 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-24T00:39:22.710 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-24T00:39:22.710 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-24T00:39:22.710 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-24T00:39:22.710 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-03-24T00:39:22.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:22.833 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-03-24T00:39:22.833 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-03-24T00:39:22.833 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-03-24T00:39:22.834 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set dev brx.0 master ceph-brx 2024-03-24T00:39:22.834 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-03-24T00:39:22.874 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:39:22.875 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-24T00:39:22.929 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:39:22.929 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:39:22.929 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:39:22.983 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-24T00:39:22.983 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:22.983 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:39:23.067 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:39:23.067 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T00:39:23.067 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-03-24T00:39:23.191 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:39:23.191 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T00:39:23.191 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-03-24T00:39:23.191 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.72:3300,172.21.15.72:3301" 2024-03-24T00:39:23.191 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-24T00:39:23.201 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:39:23.201 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T00:39:23.201 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-03-24T00:39:23.201 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.72:3300,172.21.15.72:3301" 2024-03-24T00:39:23.201 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-24T00:39:23.201 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:6789,172.21.15.72:6789,172.21.15.72:6790" 2024-03-24T00:39:23.201 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.67:6789,172.21.15.72:6789,172.21.15.72:6790:/ 2024-03-24T00:39:23.201 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=de1d0178-808e-49ca-a986-e297e523b976" will pass to kernel 2024-03-24T00:39:23.202 INFO:teuthology.orchestra.run.smithi067.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-24T00:39:23.202 INFO:teuthology.orchestra.run.smithi067.stdout: You just mounted a file system that supports labels which does not 2024-03-24T00:39:23.202 INFO:teuthology.orchestra.run.smithi067.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-24T00:39:23.202 INFO:teuthology.orchestra.run.smithi067.stdout: applications will generate AVC messages and not be allowed access to 2024-03-24T00:39:23.202 INFO:teuthology.orchestra.run.smithi067.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-24T00:39:23.203 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-24T00:39:23.203 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:23.204 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:39:23.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:23.230 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T00:39:23.286 INFO:teuthology.orchestra.run.smithi067.stdout:client4855 2024-03-24T00:39:23.286 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:39:23.286 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:39:23.341 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:23.341 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T00:39:23.397 INFO:teuthology.orchestra.run.smithi067.stdout:client4855 2024-03-24T00:39:23.397 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:23.397 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T00:39:23.452 INFO:teuthology.orchestra.run.smithi067.stdout:client4855 2024-03-24T00:39:23.453 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:23.453 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/de1d0178-808e-49ca-a986-e297e523b976.client4855/status') 2024-03-24T00:39:23.519 INFO:teuthology.orchestra.run.smithi067.stdout:instance: client.4855 (3)172.21.15.67:0/3122454247 2024-03-24T00:39:23.519 INFO:teuthology.orchestra.run.smithi067.stdout:blocklisted: false 2024-03-24T00:39:23.519 INFO:teuthology.orchestra.run.smithi067.stderr:0+1 records in 2024-03-24T00:39:23.520 INFO:teuthology.orchestra.run.smithi067.stderr:0+1 records out 2024-03-24T00:39:23.520 INFO:teuthology.orchestra.run.smithi067.stderr:70 bytes copied, 2.1084e-05 s, 3.3 MB/s 2024-03-24T00:39:23.521 INFO:tasks.kclient:client.2 config is {'syntax': 'v1'} 2024-03-24T00:39:23.521 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-24T00:39:23.521 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-24T00:39:23.522 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T00:39:23.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.662+0000 7f4af9a61640 1 Processor -- start 2024-03-24T00:39:23.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.662+0000 7f4af9a61640 1 -- start start 2024-03-24T00:39:23.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.663+0000 7f4af9a61640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4af40ce930 0x7f4af4166b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:23.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.663+0000 7f4af9a61640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4af40ccfe0 con 0x7f4af415bf70 2024-03-24T00:39:23.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.663+0000 7f4af9a61640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4af40cd2c0 con 0x7f4af40cfc70 2024-03-24T00:39:23.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.663+0000 7f4af9a61640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4af40ccda0 con 0x7f4af40ce930 2024-03-24T00:39:23.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.663+0000 7f4af37fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4af40ce930 0x7f4af4166b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:23.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.663+0000 7f4af2ffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4af415bf70 0x7f4af40cc690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46900/0 (socket says 172.21.15.67:46900) 2024-03-24T00:39:23.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.663+0000 7f4af2ffd640 1 -- 172.21.15.67:0/2177188643 learned_addr learned my addr 172.21.15.67:0/2177188643 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:23.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.663+0000 7f4af37fe640 1 -- 172.21.15.67:0/2177188643 >> v1:172.21.15.72:6789/0 conn(0x7f4af40cfc70 legacy=0x7f4af40ce220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:23.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.663+0000 7f4af37fe640 1 -- 172.21.15.67:0/2177188643 >> v1:172.21.15.67:6789/0 conn(0x7f4af415bf70 legacy=0x7f4af40cc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:23.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.663+0000 7f4af37fe640 1 -- 172.21.15.67:0/2177188643 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af40cd5a0 con 0x7f4af40ce930 2024-03-24T00:39:23.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.663+0000 7f4af1ffb640 1 -- 172.21.15.67:0/2177188643 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2927078573 0 0) 0x7f4af40ccfe0 con 0x7f4af415bf70 2024-03-24T00:39:23.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.664+0000 7f4af37fe640 1 --2- 172.21.15.67:0/2177188643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4af40ce930 0x7f4af4166b10 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f4ae4009d90 tx=0x7f4ae402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35d8773ed5986fbe server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:23.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.664+0000 7f4af1ffb640 1 -- 172.21.15.67:0/2177188643 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ae4039070 con 0x7f4af40ce930 2024-03-24T00:39:23.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.664+0000 7f4af1ffb640 1 -- 172.21.15.67:0/2177188643 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ae402cd00 con 0x7f4af40ce930 2024-03-24T00:39:23.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.664+0000 7f4af1ffb640 1 -- 172.21.15.67:0/2177188643 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ae4034660 con 0x7f4af40ce930 2024-03-24T00:39:23.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.665+0000 7f4af9a61640 1 -- 172.21.15.67:0/2177188643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4af40ce930 msgr2=0x7f4af4166b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:23.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.665+0000 7f4af9a61640 1 --2- 172.21.15.67:0/2177188643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4af40ce930 0x7f4af4166b10 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f4ae4009d90 tx=0x7f4ae402bf50 comp rx=0 tx=0).stop 2024-03-24T00:39:23.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.665+0000 7f4af9a61640 1 -- 172.21.15.67:0/2177188643 shutdown_connections 2024-03-24T00:39:23.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.665+0000 7f4af9a61640 1 --2- 172.21.15.67:0/2177188643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4af40ce930 0x7f4af4166b10 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:23.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.665+0000 7f4af9a61640 1 -- 172.21.15.67:0/2177188643 >> 172.21.15.67:0/2177188643 conn(0x7f4af40d1bf0 msgr2=0x7f4af40d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:23.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.665+0000 7f4af9a61640 1 -- 172.21.15.67:0/2177188643 shutdown_connections 2024-03-24T00:39:23.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.665+0000 7f4af9a61640 1 -- 172.21.15.67:0/2177188643 wait complete. 2024-03-24T00:39:23.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.666+0000 7f4af9a61640 1 Processor -- start 2024-03-24T00:39:23.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.666+0000 7f4af9a61640 1 -- start start 2024-03-24T00:39:23.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.667+0000 7f4af9a61640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4af40ce930 0x7f4af4151160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:23.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.667+0000 7f4af9a61640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4af40cfc70 0x7f4af41f4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:23.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.667+0000 7f4af9a61640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4af415bf70 0x7f4af41f54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:23.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.667+0000 7f4af9a61640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af40b2ce0 con 0x7f4af40cfc70 2024-03-24T00:39:23.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.667+0000 7f4af9a61640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af415c350 con 0x7f4af40ce930 2024-03-24T00:39:23.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.667+0000 7f4af9a61640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4af40b2e60 con 0x7f4af415bf70 2024-03-24T00:39:23.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.667+0000 7f4af27fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4af40cfc70 0x7f4af41f4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:23.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.667+0000 7f4af27fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4af40cfc70 0x7f4af41f4f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33924/0 (socket says 172.21.15.67:33924) 2024-03-24T00:39:23.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.667+0000 7f4af27fc640 1 -- 172.21.15.67:0/1207254172 learned_addr learned my addr 172.21.15.67:0/1207254172 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:23.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.667+0000 7f4af2ffd640 1 --2- 172.21.15.67:0/1207254172 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4af40ce930 0x7f4af4151160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:23.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.667+0000 7f4af37fe640 1 --2- 172.21.15.67:0/1207254172 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4af415bf70 0x7f4af41f54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:23.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.667+0000 7f4af27fc640 1 -- 172.21.15.67:0/1207254172 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4af415bf70 msgr2=0x7f4af41f54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:23.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.667+0000 7f4af27fc640 1 --2- 172.21.15.67:0/1207254172 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4af415bf70 0x7f4af41f54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:23.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.667+0000 7f4af27fc640 1 -- 172.21.15.67:0/1207254172 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4af40ce930 msgr2=0x7f4af4151160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:23.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.667+0000 7f4af27fc640 1 --2- 172.21.15.67:0/1207254172 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4af40ce930 0x7f4af4151160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:23.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.667+0000 7f4af27fc640 1 -- 172.21.15.67:0/1207254172 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af41516a0 con 0x7f4af40cfc70 2024-03-24T00:39:23.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.667+0000 7f4af27fc640 1 --2- 172.21.15.67:0/1207254172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4af40cfc70 0x7f4af41f4f60 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f4adc00c3f0 tx=0x7f4adc00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:23.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.668+0000 7f4aebfff640 1 -- 172.21.15.67:0/1207254172 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4adc017070 con 0x7f4af40cfc70 2024-03-24T00:39:23.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.668+0000 7f4aebfff640 1 -- 172.21.15.67:0/1207254172 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4adc0052f0 con 0x7f4af40cfc70 2024-03-24T00:39:23.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.668+0000 7f4af9a61640 1 -- 172.21.15.67:0/1207254172 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4af406c150 con 0x7f4af40cfc70 2024-03-24T00:39:23.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.668+0000 7f4aebfff640 1 -- 172.21.15.67:0/1207254172 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4adc003830 con 0x7f4af40cfc70 2024-03-24T00:39:23.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.668+0000 7f4af9a61640 1 -- 172.21.15.67:0/1207254172 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4af406c9f0 con 0x7f4af40cfc70 2024-03-24T00:39:23.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.669+0000 7f4af9a61640 1 -- 172.21.15.67:0/1207254172 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4af404f400 con 0x7f4af40cfc70 2024-03-24T00:39:23.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.669+0000 7f4aebfff640 1 -- 172.21.15.67:0/1207254172 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4adc01a070 con 0x7f4af40cfc70 2024-03-24T00:39:23.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.669+0000 7f4aebfff640 1 --2- 172.21.15.67:0/1207254172 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4ad406bbd0 0x7f4ad406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:23.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.670+0000 7f4aebfff640 1 -- 172.21.15.67:0/1207254172 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f4adc08a070 con 0x7f4af40cfc70 2024-03-24T00:39:23.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.670+0000 7f4af2ffd640 1 --2- 172.21.15.67:0/1207254172 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4ad406bbd0 0x7f4ad406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:23.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.670+0000 7f4af2ffd640 1 --2- 172.21.15.67:0/1207254172 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4ad406bbd0 0x7f4ad406e090 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f4aec007f70 tx=0x7f4aec007f00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:23.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.674+0000 7f4aebfff640 1 -- 172.21.15.67:0/1207254172 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4adc0581d0 con 0x7f4af40cfc70 2024-03-24T00:39:23.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.895+0000 7f4af9a61640 1 -- 172.21.15.67:0/1207254172 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f4af415abd0 con 0x7f4af40cfc70 2024-03-24T00:39:23.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.895+0000 7f4aebfff640 1 -- 172.21.15.67:0/1207254172 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7f4adc05c270 con 0x7f4af40cfc70 2024-03-24T00:39:23.896 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-24T00:39:23.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.897+0000 7f4af9a61640 1 -- 172.21.15.67:0/1207254172 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4ad406bbd0 msgr2=0x7f4ad406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:23.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.897+0000 7f4af9a61640 1 --2- 172.21.15.67:0/1207254172 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4ad406bbd0 0x7f4ad406e090 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f4aec007f70 tx=0x7f4aec007f00 comp rx=0 tx=0).stop 2024-03-24T00:39:23.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.897+0000 7f4af9a61640 1 -- 172.21.15.67:0/1207254172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4af40cfc70 msgr2=0x7f4af41f4f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:23.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.897+0000 7f4af9a61640 1 --2- 172.21.15.67:0/1207254172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4af40cfc70 0x7f4af41f4f60 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f4adc00c3f0 tx=0x7f4adc00c8b0 comp rx=0 tx=0).stop 2024-03-24T00:39:23.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.897+0000 7f4af9a61640 1 -- 172.21.15.67:0/1207254172 shutdown_connections 2024-03-24T00:39:23.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.897+0000 7f4af9a61640 1 --2- 172.21.15.67:0/1207254172 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4ad406bbd0 0x7f4ad406e090 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:23.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.897+0000 7f4af9a61640 1 --2- 172.21.15.67:0/1207254172 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4af415bf70 0x7f4af41f54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:23.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.897+0000 7f4af9a61640 1 --2- 172.21.15.67:0/1207254172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4af40cfc70 0x7f4af41f4f60 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:23.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.897+0000 7f4af9a61640 1 --2- 172.21.15.67:0/1207254172 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4af40ce930 0x7f4af4151160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:23.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.897+0000 7f4af9a61640 1 -- 172.21.15.67:0/1207254172 >> 172.21.15.67:0/1207254172 conn(0x7f4af40d1bf0 msgr2=0x7f4af4070740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:23.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.897+0000 7f4af9a61640 1 -- 172.21.15.67:0/1207254172 shutdown_connections 2024-03-24T00:39:23.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:23.897+0000 7f4af9a61640 1 -- 172.21.15.67:0/1207254172 wait complete. 2024-03-24T00:39:23.908 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-24T00:39:23.908 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T00:39:23.909 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-03-24T00:39:23.909 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-03-24T00:39:23.909 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-24T00:39:23.909 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-24T00:39:23.909 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T00:39:23.909 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T00:39:23.909 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-03-24T00:39:23.909 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:23.909 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-03-24T00:39:23.924 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T00:39:23.924 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T00:39:23.924 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T00:39:23.924 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:39:23.924 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-03-24T00:39:23.924 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:39:23.924 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:39:23.924 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-03-24T00:39:23.924 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-03-24T00:39:23.924 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:39:23.924 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-03-24T00:39:23.925 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-03-24T00:39:23.925 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:39:23.925 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-03-24T00:39:23.925 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T00:39:23.925 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-03-24T00:39:23.925 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-24T00:39:23.925 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 42549sec preferred_lft 42549sec 2024-03-24T00:39:23.925 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-03-24T00:39:23.925 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:39:23.925 INFO:teuthology.orchestra.run.smithi067.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T00:39:23.925 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ea:22:4b:d5:48:3d brd ff:ff:ff:ff:ff:ff 2024-03-24T00:39:23.925 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-24T00:39:23.925 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:39:23.925 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::7c6a:5fff:feaa:d41/64 scope link 2024-03-24T00:39:23.926 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:39:23.926 INFO:teuthology.orchestra.run.smithi067.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T00:39:23.926 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ea:22:4b:d5:48:3d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:39:23.926 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::e822:4bff:fed5:483d/64 scope link tentative 2024-03-24T00:39:23.926 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:39:23.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:23.926 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T00:39:23.989 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-03-24T00:39:23.990 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:23.991 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-03-24T00:39:24.045 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-03-24T00:39:24.045 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T00:39:24.045 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-24T00:39:24.045 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-24T00:39:24.045 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T00:39:24.046 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:24.046 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-03-24T00:39:24.046 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-03-24T00:39:24.046 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-24T00:39:24.046 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-24T00:39:24.046 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-24T00:39:24.046 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-03-24T00:39:24.140 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:24.140 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-03-24T00:39:24.154 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-24T00:39:24.154 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:24.154 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list-id 2024-03-24T00:39:24.208 INFO:teuthology.orchestra.run.smithi067.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-03-24T00:39:24.208 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:24.209 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-03-24T00:39:24.209 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-03-24T00:39:24.209 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:39:24.209 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-03-24T00:39:24.209 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-03-24T00:39:24.289 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:24.289 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-24T00:39:24.355 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T00:39:24.355 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T00:39:24.356 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T00:39:24.356 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:39:24.356 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-03-24T00:39:24.356 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:39:24.356 INFO:teuthology.orchestra.run.smithi067.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T00:39:24.356 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether fa:4b:2b:81:d0:55 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-24T00:39:24.356 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-24T00:39:24.356 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:39:24.356 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::f84b:2bff:fe81:d055/64 scope link 2024-03-24T00:39:24.356 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:39:24.357 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:24.357 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-24T00:39:24.423 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T00:39:24.423 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T00:39:24.424 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T00:39:24.424 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:39:24.424 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-03-24T00:39:24.424 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:39:24.424 INFO:teuthology.orchestra.run.smithi067.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T00:39:24.424 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether fa:4b:2b:81:d0:55 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-24T00:39:24.424 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-24T00:39:24.424 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:39:24.424 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::f84b:2bff:fe81:d055/64 scope link 2024-03-24T00:39:24.424 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:39:24.425 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-03-24T00:39:24.425 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:24.426 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-03-24T00:39:24.426 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-03-24T00:39:24.426 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-03-24T00:39:24.426 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-24T00:39:24.426 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-03-24T00:39:24.426 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-03-24T00:39:24.426 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-03-24T00:39:24.426 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-03-24T00:39:24.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:24.550 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-03-24T00:39:24.550 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-03-24T00:39:24.550 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.1 up 2024-03-24T00:39:24.550 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set dev brx.1 master ceph-brx 2024-03-24T00:39:24.550 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-03-24T00:39:24.591 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:39:24.591 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-03-24T00:39:24.646 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-03-24T00:39:24.646 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:39:24.646 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-03-24T00:39:24.700 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-24T00:39:24.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:24.700 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-03-24T00:39:24.784 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-03-24T00:39:24.784 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2024-03-24T00:39:24.785 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-03-24T00:39:24.794 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-03-24T00:39:24.794 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2024-03-24T00:39:24.794 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-03-24T00:39:24.794 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.72:3300,172.21.15.72:3301" 2024-03-24T00:39:24.794 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-24T00:39:24.801 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-03-24T00:39:24.801 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2024-03-24T00:39:24.801 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-03-24T00:39:24.801 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.72:3300,172.21.15.72:3301" 2024-03-24T00:39:24.801 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-24T00:39:24.801 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:6789,172.21.15.72:6789,172.21.15.72:6790" 2024-03-24T00:39:24.802 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.67:6789,172.21.15.72:6789,172.21.15.72:6790:/ 2024-03-24T00:39:24.802 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs,key=2,fsid=de1d0178-808e-49ca-a986-e297e523b976" will pass to kernel 2024-03-24T00:39:24.802 INFO:teuthology.orchestra.run.smithi067.stdout:mount: /home/ubuntu/cephtest/mnt.2 does not contain SELinux labels. 2024-03-24T00:39:24.802 INFO:teuthology.orchestra.run.smithi067.stdout: You just mounted a file system that supports labels which does not 2024-03-24T00:39:24.802 INFO:teuthology.orchestra.run.smithi067.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-24T00:39:24.802 INFO:teuthology.orchestra.run.smithi067.stdout: applications will generate AVC messages and not be allowed access to 2024-03-24T00:39:24.802 INFO:teuthology.orchestra.run.smithi067.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-24T00:39:24.803 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-24T00:39:24.803 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:24.804 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-03-24T00:39:24.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:24.829 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T00:39:24.885 INFO:teuthology.orchestra.run.smithi067.stdout:client4864 2024-03-24T00:39:24.885 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:39:24.885 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:39:24.940 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:24.940 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T00:39:24.995 INFO:teuthology.orchestra.run.smithi067.stdout:client4864 2024-03-24T00:39:24.996 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:24.996 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T00:39:25.051 INFO:teuthology.orchestra.run.smithi067.stdout:client4864 2024-03-24T00:39:25.051 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:25.051 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/de1d0178-808e-49ca-a986-e297e523b976.client4864/status') 2024-03-24T00:39:25.118 INFO:teuthology.orchestra.run.smithi067.stdout:instance: client.4864 (3)172.21.15.67:0/1382483344 2024-03-24T00:39:25.118 INFO:teuthology.orchestra.run.smithi067.stdout:blocklisted: false 2024-03-24T00:39:25.118 INFO:teuthology.orchestra.run.smithi067.stderr:0+1 records in 2024-03-24T00:39:25.118 INFO:teuthology.orchestra.run.smithi067.stderr:0+1 records out 2024-03-24T00:39:25.118 INFO:teuthology.orchestra.run.smithi067.stderr:70 bytes copied, 2.016e-05 s, 3.5 MB/s 2024-03-24T00:39:25.119 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2024-03-24T00:39:25.119 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-24T00:39:25.119 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-24T00:39:25.120 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T00:39:25.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.261+0000 7ff734026640 1 Processor -- start 2024-03-24T00:39:25.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.261+0000 7ff734026640 1 -- start start 2024-03-24T00:39:25.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.262+0000 7ff734026640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff72c15dec0 0x7ff72c0ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:25.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.262+0000 7ff734026640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff72c15e2a0 0x7ff72c162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:25.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.262+0000 7ff734026640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff72c0cf640 con 0x7ff72c0d07e0 2024-03-24T00:39:25.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.262+0000 7ff734026640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff72c0cee30 con 0x7ff72c15e2a0 2024-03-24T00:39:25.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.262+0000 7ff734026640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff72c0d2120 con 0x7ff72c15dec0 2024-03-24T00:39:25.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.262+0000 7ff7315c1640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff72c15e2a0 0x7ff72c162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:25.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.262+0000 7ff7315c1640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff72c15e2a0 0x7ff72c162630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33940/0 (socket says 172.21.15.67:33940) 2024-03-24T00:39:25.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.262+0000 7ff7315c1640 1 -- 172.21.15.67:0/1827032219 learned_addr learned my addr 172.21.15.67:0/1827032219 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:25.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.262+0000 7ff731dc2640 1 --2- 172.21.15.67:0/1827032219 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff72c15dec0 0x7ff72c0ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:25.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.263+0000 7ff730dc0640 1 -- 172.21.15.67:0/1827032219 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3930626179 0 0) 0x7ff71c001490 con 0x7ff72c0d07e0 2024-03-24T00:39:25.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.263+0000 7ff7315c1640 1 -- 172.21.15.67:0/1827032219 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff72c15dec0 msgr2=0x7ff72c0ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:25.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.263+0000 7ff7315c1640 1 --2- 172.21.15.67:0/1827032219 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff72c15dec0 0x7ff72c0ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.263+0000 7ff7315c1640 1 -- 172.21.15.67:0/1827032219 >> v1:172.21.15.72:6789/0 conn(0x7ff72c0d07e0 legacy=0x7ff72c0b5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:25.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.263+0000 7ff7315c1640 1 -- 172.21.15.67:0/1827032219 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff72c0cf080 con 0x7ff72c15e2a0 2024-03-24T00:39:25.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.263+0000 7ff7315c1640 1 --2- 172.21.15.67:0/1827032219 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff72c15e2a0 0x7ff72c162630 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff72c0cefb0 tx=0x7ff72002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6317540794fb4d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:25.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.264+0000 7ff730dc0640 1 -- 172.21.15.67:0/1827032219 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff720035020 con 0x7ff72c15e2a0 2024-03-24T00:39:25.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.264+0000 7ff730dc0640 1 -- 172.21.15.67:0/1827032219 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff720031640 con 0x7ff72c15e2a0 2024-03-24T00:39:25.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.264+0000 7ff730dc0640 1 -- 172.21.15.67:0/1827032219 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff72002d040 con 0x7ff72c15e2a0 2024-03-24T00:39:25.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.264+0000 7ff734026640 1 -- 172.21.15.67:0/1827032219 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff72c15e2a0 msgr2=0x7ff72c162630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:25.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.264+0000 7ff734026640 1 --2- 172.21.15.67:0/1827032219 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff72c15e2a0 0x7ff72c162630 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff72c0cefb0 tx=0x7ff72002b8f0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.264+0000 7ff734026640 1 -- 172.21.15.67:0/1827032219 shutdown_connections 2024-03-24T00:39:25.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.264+0000 7ff734026640 1 --2- 172.21.15.67:0/1827032219 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff72c15e2a0 0x7ff72c162630 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.264+0000 7ff734026640 1 --2- 172.21.15.67:0/1827032219 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff72c15dec0 0x7ff72c0ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.264+0000 7ff734026640 1 -- 172.21.15.67:0/1827032219 >> 172.21.15.67:0/1827032219 conn(0x7ff72c06c810 msgr2=0x7ff72c06cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:25.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.264+0000 7ff734026640 1 -- 172.21.15.67:0/1827032219 shutdown_connections 2024-03-24T00:39:25.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.264+0000 7ff734026640 1 -- 172.21.15.67:0/1827032219 wait complete. 2024-03-24T00:39:25.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.265+0000 7ff734026640 1 Processor -- start 2024-03-24T00:39:25.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.265+0000 7ff734026640 1 -- start start 2024-03-24T00:39:25.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.266+0000 7ff734026640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff72c0d07e0 0x7ff72c1f4480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:25.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.266+0000 7ff734026640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff72c15dec0 0x7ff72c0c9b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:25.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.266+0000 7ff734026640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff72c15e2a0 0x7ff72c0ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:25.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.266+0000 7ff734026640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff72c0d3740 con 0x7ff72c15e2a0 2024-03-24T00:39:25.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.266+0000 7ff734026640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff72c0d35c0 con 0x7ff72c0d07e0 2024-03-24T00:39:25.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.266+0000 7ff734026640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff72c0d38c0 con 0x7ff72c15dec0 2024-03-24T00:39:25.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.266+0000 7ff7325c3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff72c15e2a0 0x7ff72c0ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:25.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.266+0000 7ff7325c3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff72c15e2a0 0x7ff72c0ccf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33942/0 (socket says 172.21.15.67:33942) 2024-03-24T00:39:25.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.266+0000 7ff7325c3640 1 -- 172.21.15.67:0/2071397395 learned_addr learned my addr 172.21.15.67:0/2071397395 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:25.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.266+0000 7ff731dc2640 1 --2- 172.21.15.67:0/2071397395 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff72c0d07e0 0x7ff72c1f4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:25.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.266+0000 7ff7315c1640 1 --2- 172.21.15.67:0/2071397395 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff72c15dec0 0x7ff72c0c9b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:25.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.266+0000 7ff7325c3640 1 -- 172.21.15.67:0/2071397395 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff72c15dec0 msgr2=0x7ff72c0c9b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:25.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.266+0000 7ff7325c3640 1 --2- 172.21.15.67:0/2071397395 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff72c15dec0 0x7ff72c0c9b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.266+0000 7ff7325c3640 1 -- 172.21.15.67:0/2071397395 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff72c0d07e0 msgr2=0x7ff72c1f4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:25.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.266+0000 7ff7325c3640 1 --2- 172.21.15.67:0/2071397395 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff72c0d07e0 0x7ff72c1f4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.266+0000 7ff7325c3640 1 -- 172.21.15.67:0/2071397395 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff72c158560 con 0x7ff72c15e2a0 2024-03-24T00:39:25.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.267+0000 7ff7325c3640 1 --2- 172.21.15.67:0/2071397395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff72c15e2a0 0x7ff72c0ccf60 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff71c00c7c0 tx=0x7ff71c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:25.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.267+0000 7ff72a7fc640 1 -- 172.21.15.67:0/2071397395 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff71c003da0 con 0x7ff72c15e2a0 2024-03-24T00:39:25.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.267+0000 7ff72a7fc640 1 -- 172.21.15.67:0/2071397395 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff71c005070 con 0x7ff72c15e2a0 2024-03-24T00:39:25.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.267+0000 7ff734026640 1 -- 172.21.15.67:0/2071397395 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff72c158840 con 0x7ff72c15e2a0 2024-03-24T00:39:25.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.267+0000 7ff72a7fc640 1 -- 172.21.15.67:0/2071397395 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff71c005460 con 0x7ff72c15e2a0 2024-03-24T00:39:25.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.267+0000 7ff734026640 1 -- 172.21.15.67:0/2071397395 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff72c1590e0 con 0x7ff72c15e2a0 2024-03-24T00:39:25.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.268+0000 7ff734026640 1 -- 172.21.15.67:0/2071397395 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff72c04f400 con 0x7ff72c15e2a0 2024-03-24T00:39:25.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.268+0000 7ff72a7fc640 1 -- 172.21.15.67:0/2071397395 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff71c005600 con 0x7ff72c15e2a0 2024-03-24T00:39:25.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.268+0000 7ff72a7fc640 1 --2- 172.21.15.67:0/2071397395 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff71806bbd0 0x7ff71806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:25.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.269+0000 7ff72a7fc640 1 -- 172.21.15.67:0/2071397395 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7ff71c08abe0 con 0x7ff72c15e2a0 2024-03-24T00:39:25.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.269+0000 7ff731dc2640 1 --2- 172.21.15.67:0/2071397395 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff71806bbd0 0x7ff71806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:25.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.269+0000 7ff731dc2640 1 --2- 172.21.15.67:0/2071397395 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff71806bbd0 0x7ff71806e090 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff724009b40 tx=0x7ff724009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:25.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.273+0000 7ff72a7fc640 1 -- 172.21.15.67:0/2071397395 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff72c0cf640 con 0x7ff72c15e2a0 2024-03-24T00:39:25.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.495+0000 7ff734026640 1 -- 172.21.15.67:0/2071397395 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff72c0d1aa0 con 0x7ff72c15e2a0 2024-03-24T00:39:25.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.495+0000 7ff72a7fc640 1 -- 172.21.15.67:0/2071397395 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7ff71c058d40 con 0x7ff72c15e2a0 2024-03-24T00:39:25.495 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-24T00:39:25.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.497+0000 7ff734026640 1 -- 172.21.15.67:0/2071397395 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff71806bbd0 msgr2=0x7ff71806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:25.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.497+0000 7ff734026640 1 --2- 172.21.15.67:0/2071397395 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff71806bbd0 0x7ff71806e090 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff724009b40 tx=0x7ff724009190 comp rx=0 tx=0).stop 2024-03-24T00:39:25.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.497+0000 7ff734026640 1 -- 172.21.15.67:0/2071397395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff72c15e2a0 msgr2=0x7ff72c0ccf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:25.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.497+0000 7ff734026640 1 --2- 172.21.15.67:0/2071397395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff72c15e2a0 0x7ff72c0ccf60 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff71c00c7c0 tx=0x7ff71c00cc80 comp rx=0 tx=0).stop 2024-03-24T00:39:25.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.497+0000 7ff734026640 1 -- 172.21.15.67:0/2071397395 shutdown_connections 2024-03-24T00:39:25.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.497+0000 7ff734026640 1 --2- 172.21.15.67:0/2071397395 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff71806bbd0 0x7ff71806e090 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.497+0000 7ff734026640 1 --2- 172.21.15.67:0/2071397395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff72c15e2a0 0x7ff72c0ccf60 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.497+0000 7ff734026640 1 --2- 172.21.15.67:0/2071397395 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff72c15dec0 0x7ff72c0c9b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.497+0000 7ff734026640 1 --2- 172.21.15.67:0/2071397395 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff72c0d07e0 0x7ff72c1f4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.497+0000 7ff734026640 1 -- 172.21.15.67:0/2071397395 >> 172.21.15.67:0/2071397395 conn(0x7ff72c06c810 msgr2=0x7ff72c0b8760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:25.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.497+0000 7ff734026640 1 -- 172.21.15.67:0/2071397395 shutdown_connections 2024-03-24T00:39:25.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:25.497+0000 7ff734026640 1 -- 172.21.15.67:0/2071397395 wait complete. 2024-03-24T00:39:25.507 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-24T00:39:25.507 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T00:39:25.507 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi072.front.sepia.ceph.com 2024-03-24T00:39:25.508 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-24T00:39:25.508 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-24T00:39:25.508 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-24T00:39:25.508 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T00:39:25.508 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T00:39:25.508 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-24T00:39:25.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:25.508 DEBUG:teuthology.orchestra.run.smithi072:> ip addr 2024-03-24T00:39:25.523 INFO:teuthology.orchestra.run.smithi072.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T00:39:25.523 INFO:teuthology.orchestra.run.smithi072.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T00:39:25.523 INFO:teuthology.orchestra.run.smithi072.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T00:39:25.524 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:39:25.524 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 ::1/128 scope host 2024-03-24T00:39:25.524 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:39:25.524 INFO:teuthology.orchestra.run.smithi072.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:39:25.524 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:d8:da:f4 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:39:25.524 INFO:teuthology.orchestra.run.smithi072.stdout: altname enp7s0f0 2024-03-24T00:39:25.524 INFO:teuthology.orchestra.run.smithi072.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:39:25.524 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:d8:da:f5 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:39:25.524 INFO:teuthology.orchestra.run.smithi072.stdout: altname enp7s0f1 2024-03-24T00:39:25.524 INFO:teuthology.orchestra.run.smithi072.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:39:25.524 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:8f:b7:64 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:39:25.525 INFO:teuthology.orchestra.run.smithi072.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T00:39:25.525 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:8f:b7:65 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:39:25.525 INFO:teuthology.orchestra.run.smithi072.stdout: inet 172.21.15.72/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-24T00:39:25.525 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft 42536sec preferred_lft 42536sec 2024-03-24T00:39:25.525 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::ec4:7aff:fe8f:b765/64 scope link 2024-03-24T00:39:25.525 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:39:25.525 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-24T00:39:25.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:25.525 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec bash -c ' 2024-03-24T00:39:25.525 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-03-24T00:39:25.526 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link add name ceph-brx type bridge 2024-03-24T00:39:25.526 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip addr flush dev ceph-brx 2024-03-24T00:39:25.526 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link set ceph-brx up 2024-03-24T00:39:25.526 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-24T00:39:25.526 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-03-24T00:39:25.715 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:25.716 DEBUG:teuthology.orchestra.run.smithi072:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T00:39:25.742 INFO:teuthology.orchestra.run.smithi072.stdout:1 2024-03-24T00:39:25.743 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:25.743 DEBUG:teuthology.orchestra.run.smithi072:> route 2024-03-24T00:39:25.813 INFO:teuthology.orchestra.run.smithi072.stdout:Kernel IP routing table 2024-03-24T00:39:25.814 INFO:teuthology.orchestra.run.smithi072.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T00:39:25.814 INFO:teuthology.orchestra.run.smithi072.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-24T00:39:25.814 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-24T00:39:25.814 INFO:teuthology.orchestra.run.smithi072.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T00:39:25.814 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:25.814 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec bash -c ' 2024-03-24T00:39:25.814 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-03-24T00:39:25.814 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-24T00:39:25.814 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-24T00:39:25.815 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-24T00:39:25.815 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-03-24T00:39:26.107 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:26.107 DEBUG:teuthology.orchestra.run.smithi072:> ip netns list 2024-03-24T00:39:26.122 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:26.122 DEBUG:teuthology.orchestra.run.smithi072:> ip netns list-id 2024-03-24T00:39:26.178 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:26.178 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec bash -c ' 2024-03-24T00:39:26.178 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-03-24T00:39:26.178 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-24T00:39:26.178 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-03-24T00:39:26.178 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-03-24T00:39:26.262 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-03-24T00:39:26.262 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:26.262 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec bash -c ' 2024-03-24T00:39:26.262 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-03-24T00:39:26.262 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-03-24T00:39:26.262 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-24T00:39:26.262 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-03-24T00:39:26.262 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-03-24T00:39:26.263 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-03-24T00:39:26.263 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-03-24T00:39:26.402 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:26.402 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec bash -c ' 2024-03-24T00:39:26.402 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-03-24T00:39:26.402 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link set brx.0 up 2024-03-24T00:39:26.402 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link set dev brx.0 master ceph-brx 2024-03-24T00:39:26.402 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-03-24T00:39:26.445 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:39:26.445 DEBUG:teuthology.orchestra.run.smithi072:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-24T00:39:26.500 INFO:teuthology.orchestra.run.smithi072.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-24T00:39:26.500 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:39:26.500 DEBUG:teuthology.orchestra.run.smithi072:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-24T00:39:26.555 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-24T00:39:26.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:26.555 DEBUG:teuthology.orchestra.run.smithi072:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-24T00:39:26.641 INFO:teuthology.orchestra.run.smithi072.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-24T00:39:26.641 INFO:teuthology.orchestra.run.smithi072.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-03-24T00:39:26.641 INFO:teuthology.orchestra.run.smithi072.stdout:invalid new device string format 2024-03-24T00:39:26.758 INFO:teuthology.orchestra.run.smithi072.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-24T00:39:26.758 INFO:teuthology.orchestra.run.smithi072.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-03-24T00:39:26.758 INFO:teuthology.orchestra.run.smithi072.stdout:invalid new device string format 2024-03-24T00:39:26.758 INFO:teuthology.orchestra.run.smithi072.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.72:3300,172.21.15.72:3301" 2024-03-24T00:39:26.758 INFO:teuthology.orchestra.run.smithi072.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-24T00:39:26.763 INFO:teuthology.orchestra.run.smithi072.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-24T00:39:26.763 INFO:teuthology.orchestra.run.smithi072.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-03-24T00:39:26.763 INFO:teuthology.orchestra.run.smithi072.stdout:invalid new device string format 2024-03-24T00:39:26.764 INFO:teuthology.orchestra.run.smithi072.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.72:3300,172.21.15.72:3301" 2024-03-24T00:39:26.764 INFO:teuthology.orchestra.run.smithi072.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-24T00:39:26.764 INFO:teuthology.orchestra.run.smithi072.stdout:mount.ceph: resolved to: "172.21.15.67:6789,172.21.15.72:6789,172.21.15.72:6790" 2024-03-24T00:39:26.764 INFO:teuthology.orchestra.run.smithi072.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.67:6789,172.21.15.72:6789,172.21.15.72:6790:/ 2024-03-24T00:39:26.764 INFO:teuthology.orchestra.run.smithi072.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=de1d0178-808e-49ca-a986-e297e523b976" will pass to kernel 2024-03-24T00:39:26.765 INFO:teuthology.orchestra.run.smithi072.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-03-24T00:39:26.765 INFO:teuthology.orchestra.run.smithi072.stdout: You just mounted a file system that supports labels which does not 2024-03-24T00:39:26.765 INFO:teuthology.orchestra.run.smithi072.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-24T00:39:26.765 INFO:teuthology.orchestra.run.smithi072.stdout: applications will generate AVC messages and not be allowed access to 2024-03-24T00:39:26.765 INFO:teuthology.orchestra.run.smithi072.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-24T00:39:26.767 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-24T00:39:26.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:26.767 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-24T00:39:26.794 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:26.794 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T00:39:26.850 INFO:teuthology.orchestra.run.smithi072.stdout:client4932 2024-03-24T00:39:26.851 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:39:26.851 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:39:26.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:26.906 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T00:39:26.962 INFO:teuthology.orchestra.run.smithi072.stdout:client4932 2024-03-24T00:39:26.962 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:26.962 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T00:39:27.018 INFO:teuthology.orchestra.run.smithi072.stdout:client4932 2024-03-24T00:39:27.018 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:27.018 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/de1d0178-808e-49ca-a986-e297e523b976.client4932/status') 2024-03-24T00:39:27.085 INFO:teuthology.orchestra.run.smithi072.stdout:instance: client.4932 (3)172.21.15.72:0/236552796 2024-03-24T00:39:27.085 INFO:teuthology.orchestra.run.smithi072.stdout:blocklisted: false 2024-03-24T00:39:27.085 INFO:teuthology.orchestra.run.smithi072.stderr:0+1 records in 2024-03-24T00:39:27.085 INFO:teuthology.orchestra.run.smithi072.stderr:0+1 records out 2024-03-24T00:39:27.085 INFO:teuthology.orchestra.run.smithi072.stderr:69 bytes copied, 2.1073e-05 s, 3.3 MB/s 2024-03-24T00:39:27.087 INFO:tasks.kclient:client.3 config is {'syntax': 'v1'} 2024-03-24T00:39:27.087 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-24T00:39:27.087 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-24T00:39:27.087 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T00:39:27.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.201+0000 7fb758df3640 1 Processor -- start 2024-03-24T00:39:27.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.201+0000 7fb758df3640 1 -- start start 2024-03-24T00:39:27.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.202+0000 7fb758df3640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb7540d1550 0x7fb75406cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.202+0000 7fb758df3640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb7540b2780 0x7fb7540b2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.202+0000 7fb758df3640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7540d1930 con 0x7fb754099d70 2024-03-24T00:39:27.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.202+0000 7fb758df3640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb75406d420 con 0x7fb7540b2780 2024-03-24T00:39:27.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.202+0000 7fb758df3640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb75406ca50 con 0x7fb7540d1550 2024-03-24T00:39:27.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.202+0000 7fb75259c640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb7540d1550 0x7fb75406cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.202+0000 7fb751d9b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb7540b2780 0x7fb7540b2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.202+0000 7fb752d9d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fb754099d70 0x7fb7540cfd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:45738/0 (socket says 172.21.15.67:45738) 2024-03-24T00:39:27.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.202+0000 7fb752d9d640 1 -- 172.21.15.67:0/2035582833 learned_addr learned my addr 172.21.15.67:0/2035582833 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:27.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.203+0000 7fb751d9b640 1 -- 172.21.15.67:0/2035582833 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb7540d1550 msgr2=0x7fb75406cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:27.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.203+0000 7fb751d9b640 1 --2- 172.21.15.67:0/2035582833 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb7540d1550 0x7fb75406cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.203+0000 7fb751d9b640 1 -- 172.21.15.67:0/2035582833 >> v1:172.21.15.72:6789/0 conn(0x7fb754099d70 legacy=0x7fb7540cfd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:27.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.203+0000 7fb751d9b640 1 -- 172.21.15.67:0/2035582833 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7540b1f60 con 0x7fb7540b2780 2024-03-24T00:39:27.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.203+0000 7fb75159a640 1 -- 172.21.15.67:0/2035582833 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 75629731 0 0) 0x7fb7540d1930 con 0x7fb754099d70 2024-03-24T00:39:27.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.203+0000 7fb751d9b640 1 --2- 172.21.15.67:0/2035582833 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7540b2780 0x7fb7540b2c00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb73c0097a0 tx=0x7fb73c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=922ddd95b9a830d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:27.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.204+0000 7fb75159a640 1 -- 172.21.15.67:0/2035582833 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73c035020 con 0x7fb7540b2780 2024-03-24T00:39:27.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.204+0000 7fb75159a640 1 -- 172.21.15.67:0/2035582833 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb73c031830 con 0x7fb7540b2780 2024-03-24T00:39:27.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.204+0000 7fb75159a640 1 -- 172.21.15.67:0/2035582833 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73c031c00 con 0x7fb7540b2780 2024-03-24T00:39:27.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.204+0000 7fb758df3640 1 -- 172.21.15.67:0/2035582833 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7540b2780 msgr2=0x7fb7540b2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:27.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.204+0000 7fb758df3640 1 --2- 172.21.15.67:0/2035582833 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7540b2780 0x7fb7540b2c00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb73c0097a0 tx=0x7fb73c02b8d0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.204+0000 7fb758df3640 1 -- 172.21.15.67:0/2035582833 shutdown_connections 2024-03-24T00:39:27.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.204+0000 7fb758df3640 1 --2- 172.21.15.67:0/2035582833 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7540b2780 0x7fb7540b2c00 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.204+0000 7fb758df3640 1 --2- 172.21.15.67:0/2035582833 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb7540d1550 0x7fb75406cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.204+0000 7fb758df3640 1 -- 172.21.15.67:0/2035582833 >> 172.21.15.67:0/2035582833 conn(0x7fb7540b36e0 msgr2=0x7fb7540b3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:27.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.204+0000 7fb758df3640 1 -- 172.21.15.67:0/2035582833 shutdown_connections 2024-03-24T00:39:27.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.204+0000 7fb758df3640 1 -- 172.21.15.67:0/2035582833 wait complete. 2024-03-24T00:39:27.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.205+0000 7fb758df3640 1 Processor -- start 2024-03-24T00:39:27.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.206+0000 7fb758df3640 1 -- start start 2024-03-24T00:39:27.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.206+0000 7fb758df3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb754099d70 0x7fb7540ca960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.206+0000 7fb758df3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7540b2780 0x7fb7540c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.206+0000 7fb758df3640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb7540d1550 0x7fb754160870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.206+0000 7fb758df3640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb75406c400 con 0x7fb7540b2780 2024-03-24T00:39:27.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.206+0000 7fb758df3640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7540b2520 con 0x7fb754099d70 2024-03-24T00:39:27.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.206+0000 7fb758df3640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb75406c6e0 con 0x7fb7540d1550 2024-03-24T00:39:27.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.206+0000 7fb751d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7540b2780 0x7fb7540c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.206+0000 7fb75259c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb754099d70 0x7fb7540ca960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.206+0000 7fb751d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7540b2780 0x7fb7540c8cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33958/0 (socket says 172.21.15.67:33958) 2024-03-24T00:39:27.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.206+0000 7fb751d9b640 1 -- 172.21.15.67:0/202735901 learned_addr learned my addr 172.21.15.67:0/202735901 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:27.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.206+0000 7fb752d9d640 1 --2- 172.21.15.67:0/202735901 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb7540d1550 0x7fb754160870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.207+0000 7fb751d9b640 1 -- 172.21.15.67:0/202735901 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb7540d1550 msgr2=0x7fb754160870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:27.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.207+0000 7fb751d9b640 1 --2- 172.21.15.67:0/202735901 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb7540d1550 0x7fb754160870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.207+0000 7fb751d9b640 1 -- 172.21.15.67:0/202735901 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb754099d70 msgr2=0x7fb7540ca960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:27.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.207+0000 7fb751d9b640 1 --2- 172.21.15.67:0/202735901 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb754099d70 0x7fb7540ca960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.207+0000 7fb751d9b640 1 -- 172.21.15.67:0/202735901 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7540c9430 con 0x7fb7540b2780 2024-03-24T00:39:27.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.207+0000 7fb752d9d640 1 --2- 172.21.15.67:0/202735901 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb7540d1550 0x7fb754160870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:27.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.207+0000 7fb75259c640 1 --2- 172.21.15.67:0/202735901 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb754099d70 0x7fb7540ca960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:27.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.207+0000 7fb751d9b640 1 --2- 172.21.15.67:0/202735901 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7540b2780 0x7fb7540c8cc0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb73c0098c0 tx=0x7fb73c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:27.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.207+0000 7fb74affd640 1 -- 172.21.15.67:0/202735901 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73c03b980 con 0x7fb7540b2780 2024-03-24T00:39:27.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.207+0000 7fb74affd640 1 -- 172.21.15.67:0/202735901 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb73c03bb40 con 0x7fb7540b2780 2024-03-24T00:39:27.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.207+0000 7fb74affd640 1 -- 172.21.15.67:0/202735901 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73c0453f0 con 0x7fb7540b2780 2024-03-24T00:39:27.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.207+0000 7fb758df3640 1 -- 172.21.15.67:0/202735901 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7540c9710 con 0x7fb7540b2780 2024-03-24T00:39:27.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.207+0000 7fb758df3640 1 -- 172.21.15.67:0/202735901 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb75415b770 con 0x7fb7540b2780 2024-03-24T00:39:27.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.208+0000 7fb758df3640 1 -- 172.21.15.67:0/202735901 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7541f3d30 con 0x7fb7540b2780 2024-03-24T00:39:27.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.209+0000 7fb74affd640 1 -- 172.21.15.67:0/202735901 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb73c04c060 con 0x7fb7540b2780 2024-03-24T00:39:27.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.209+0000 7fb74affd640 1 --2- 172.21.15.67:0/202735901 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb73406bbd0 0x7fb73406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.209+0000 7fb74affd640 1 -- 172.21.15.67:0/202735901 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fb73c0ac9b0 con 0x7fb7540b2780 2024-03-24T00:39:27.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.209+0000 7fb75259c640 1 --2- 172.21.15.67:0/202735901 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb73406bbd0 0x7fb73406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.210+0000 7fb75259c640 1 --2- 172.21.15.67:0/202735901 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb73406bbd0 0x7fb73406e090 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb74c009e90 tx=0x7fb74c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:27.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.214+0000 7fb74affd640 1 -- 172.21.15.67:0/202735901 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb73c040070 con 0x7fb7540b2780 2024-03-24T00:39:27.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.436+0000 7fb758df3640 1 -- 172.21.15.67:0/202735901 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fb75415b9a0 con 0x7fb7540b2780 2024-03-24T00:39:27.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.436+0000 7fb74affd640 1 -- 172.21.15.67:0/202735901 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7fb73c031800 con 0x7fb7540b2780 2024-03-24T00:39:27.436 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-24T00:39:27.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.438+0000 7fb758df3640 1 -- 172.21.15.67:0/202735901 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb73406bbd0 msgr2=0x7fb73406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:27.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.438+0000 7fb758df3640 1 --2- 172.21.15.67:0/202735901 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb73406bbd0 0x7fb73406e090 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb74c009e90 tx=0x7fb74c009190 comp rx=0 tx=0).stop 2024-03-24T00:39:27.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.438+0000 7fb758df3640 1 -- 172.21.15.67:0/202735901 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7540b2780 msgr2=0x7fb7540c8cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:27.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.438+0000 7fb758df3640 1 --2- 172.21.15.67:0/202735901 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7540b2780 0x7fb7540c8cc0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb73c0098c0 tx=0x7fb73c0046c0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.438+0000 7fb758df3640 1 -- 172.21.15.67:0/202735901 shutdown_connections 2024-03-24T00:39:27.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.438+0000 7fb758df3640 1 --2- 172.21.15.67:0/202735901 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb73406bbd0 0x7fb73406e090 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.438+0000 7fb758df3640 1 --2- 172.21.15.67:0/202735901 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb7540d1550 0x7fb754160870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.438+0000 7fb758df3640 1 --2- 172.21.15.67:0/202735901 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7540b2780 0x7fb7540c8cc0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.438+0000 7fb758df3640 1 --2- 172.21.15.67:0/202735901 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb754099d70 0x7fb7540ca960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.438+0000 7fb758df3640 1 -- 172.21.15.67:0/202735901 >> 172.21.15.67:0/202735901 conn(0x7fb7540b36e0 msgr2=0x7fb754161d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:27.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.438+0000 7fb758df3640 1 -- 172.21.15.67:0/202735901 shutdown_connections 2024-03-24T00:39:27.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:27.438+0000 7fb758df3640 1 -- 172.21.15.67:0/202735901 wait complete. 2024-03-24T00:39:27.448 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-24T00:39:27.448 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T00:39:27.448 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi072.front.sepia.ceph.com 2024-03-24T00:39:27.448 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-03-24T00:39:27.448 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-24T00:39:27.448 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-24T00:39:27.448 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T00:39:27.449 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T00:39:27.449 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-03-24T00:39:27.449 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:27.449 DEBUG:teuthology.orchestra.run.smithi072:> ip addr 2024-03-24T00:39:27.464 INFO:teuthology.orchestra.run.smithi072.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T00:39:27.465 INFO:teuthology.orchestra.run.smithi072.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T00:39:27.465 INFO:teuthology.orchestra.run.smithi072.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T00:39:27.465 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:39:27.465 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 ::1/128 scope host 2024-03-24T00:39:27.465 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:39:27.465 INFO:teuthology.orchestra.run.smithi072.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:39:27.465 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:d8:da:f4 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:39:27.465 INFO:teuthology.orchestra.run.smithi072.stdout: altname enp7s0f0 2024-03-24T00:39:27.465 INFO:teuthology.orchestra.run.smithi072.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:39:27.465 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:d8:da:f5 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:39:27.465 INFO:teuthology.orchestra.run.smithi072.stdout: altname enp7s0f1 2024-03-24T00:39:27.465 INFO:teuthology.orchestra.run.smithi072.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:39:27.465 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:8f:b7:64 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:39:27.466 INFO:teuthology.orchestra.run.smithi072.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T00:39:27.466 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:8f:b7:65 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:39:27.466 INFO:teuthology.orchestra.run.smithi072.stdout: inet 172.21.15.72/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-24T00:39:27.466 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft 42534sec preferred_lft 42534sec 2024-03-24T00:39:27.466 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::ec4:7aff:fe8f:b765/64 scope link 2024-03-24T00:39:27.466 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:39:27.466 INFO:teuthology.orchestra.run.smithi072.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T00:39:27.466 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 9a:50:d1:19:88:5a brd ff:ff:ff:ff:ff:ff 2024-03-24T00:39:27.466 INFO:teuthology.orchestra.run.smithi072.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-24T00:39:27.466 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:39:27.466 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::8d4:97ff:fe8d:c94f/64 scope link 2024-03-24T00:39:27.466 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:39:27.466 INFO:teuthology.orchestra.run.smithi072.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T00:39:27.466 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 9a:50:d1:19:88:5a brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-24T00:39:27.467 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::9850:d1ff:fe19:885a/64 scope link tentative 2024-03-24T00:39:27.467 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:39:27.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:27.467 DEBUG:teuthology.orchestra.run.smithi072:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T00:39:27.530 INFO:teuthology.orchestra.run.smithi072.stdout:1 2024-03-24T00:39:27.532 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:27.532 DEBUG:teuthology.orchestra.run.smithi072:> route 2024-03-24T00:39:27.587 INFO:teuthology.orchestra.run.smithi072.stdout:Kernel IP routing table 2024-03-24T00:39:27.587 INFO:teuthology.orchestra.run.smithi072.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T00:39:27.587 INFO:teuthology.orchestra.run.smithi072.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-24T00:39:27.587 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-24T00:39:27.587 INFO:teuthology.orchestra.run.smithi072.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T00:39:27.587 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:27.588 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec bash -c ' 2024-03-24T00:39:27.588 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-03-24T00:39:27.588 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-24T00:39:27.588 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-24T00:39:27.588 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-24T00:39:27.588 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-03-24T00:39:27.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:27.683 DEBUG:teuthology.orchestra.run.smithi072:> ip netns list 2024-03-24T00:39:27.697 INFO:teuthology.orchestra.run.smithi072.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-24T00:39:27.698 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:27.698 DEBUG:teuthology.orchestra.run.smithi072:> ip netns list-id 2024-03-24T00:39:27.752 INFO:teuthology.orchestra.run.smithi072.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-03-24T00:39:27.752 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:27.752 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec bash -c ' 2024-03-24T00:39:27.752 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-03-24T00:39:27.752 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-24T00:39:27.752 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-03-24T00:39:27.753 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-03-24T00:39:27.834 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:27.834 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-24T00:39:27.902 INFO:teuthology.orchestra.run.smithi072.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T00:39:27.902 INFO:teuthology.orchestra.run.smithi072.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T00:39:27.902 INFO:teuthology.orchestra.run.smithi072.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T00:39:27.903 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:39:27.903 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 ::1/128 scope host 2024-03-24T00:39:27.903 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:39:27.903 INFO:teuthology.orchestra.run.smithi072.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T00:39:27.903 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 62:85:80:c0:98:eb brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-24T00:39:27.903 INFO:teuthology.orchestra.run.smithi072.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-24T00:39:27.903 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:39:27.903 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::6085:80ff:fec0:98eb/64 scope link tentative 2024-03-24T00:39:27.903 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:39:27.904 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:27.904 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-24T00:39:27.973 INFO:teuthology.orchestra.run.smithi072.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T00:39:27.973 INFO:teuthology.orchestra.run.smithi072.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T00:39:27.973 INFO:teuthology.orchestra.run.smithi072.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T00:39:27.973 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:39:27.973 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 ::1/128 scope host 2024-03-24T00:39:27.973 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:39:27.973 INFO:teuthology.orchestra.run.smithi072.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T00:39:27.973 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 62:85:80:c0:98:eb brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-24T00:39:27.973 INFO:teuthology.orchestra.run.smithi072.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-24T00:39:27.973 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:39:27.974 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::6085:80ff:fec0:98eb/64 scope link tentative 2024-03-24T00:39:27.974 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:39:27.975 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-03-24T00:39:27.975 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:27.975 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec bash -c ' 2024-03-24T00:39:27.975 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-03-24T00:39:27.975 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-03-24T00:39:27.975 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-24T00:39:27.975 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-03-24T00:39:27.975 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-03-24T00:39:27.975 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-03-24T00:39:27.975 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-03-24T00:39:28.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:28.102 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec bash -c ' 2024-03-24T00:39:28.102 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-03-24T00:39:28.102 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link set brx.1 up 2024-03-24T00:39:28.102 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link set dev brx.1 master ceph-brx 2024-03-24T00:39:28.102 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-03-24T00:39:28.144 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:39:28.144 DEBUG:teuthology.orchestra.run.smithi072:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-03-24T00:39:28.198 INFO:teuthology.orchestra.run.smithi072.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-03-24T00:39:28.198 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:39:28.198 DEBUG:teuthology.orchestra.run.smithi072:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-03-24T00:39:28.252 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-24T00:39:28.252 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:28.252 DEBUG:teuthology.orchestra.run.smithi072:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-03-24T00:39:28.335 INFO:teuthology.orchestra.run.smithi072.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-03-24T00:39:28.335 INFO:teuthology.orchestra.run.smithi072.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2024-03-24T00:39:28.336 INFO:teuthology.orchestra.run.smithi072.stdout:invalid new device string format 2024-03-24T00:39:28.344 INFO:teuthology.orchestra.run.smithi072.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-03-24T00:39:28.344 INFO:teuthology.orchestra.run.smithi072.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2024-03-24T00:39:28.344 INFO:teuthology.orchestra.run.smithi072.stdout:invalid new device string format 2024-03-24T00:39:28.344 INFO:teuthology.orchestra.run.smithi072.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.72:3300,172.21.15.72:3301" 2024-03-24T00:39:28.344 INFO:teuthology.orchestra.run.smithi072.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-24T00:39:28.353 INFO:teuthology.orchestra.run.smithi072.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-03-24T00:39:28.353 INFO:teuthology.orchestra.run.smithi072.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2024-03-24T00:39:28.353 INFO:teuthology.orchestra.run.smithi072.stdout:invalid new device string format 2024-03-24T00:39:28.353 INFO:teuthology.orchestra.run.smithi072.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.72:3300,172.21.15.72:3301" 2024-03-24T00:39:28.353 INFO:teuthology.orchestra.run.smithi072.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-24T00:39:28.353 INFO:teuthology.orchestra.run.smithi072.stdout:mount.ceph: resolved to: "172.21.15.67:6789,172.21.15.72:6789,172.21.15.72:6790" 2024-03-24T00:39:28.353 INFO:teuthology.orchestra.run.smithi072.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.67:6789,172.21.15.72:6789,172.21.15.72:6790:/ 2024-03-24T00:39:28.353 INFO:teuthology.orchestra.run.smithi072.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs,key=3,fsid=de1d0178-808e-49ca-a986-e297e523b976" will pass to kernel 2024-03-24T00:39:28.360 INFO:teuthology.orchestra.run.smithi072.stdout:mount: /home/ubuntu/cephtest/mnt.3 does not contain SELinux labels. 2024-03-24T00:39:28.360 INFO:teuthology.orchestra.run.smithi072.stdout: You just mounted a file system that supports labels which does not 2024-03-24T00:39:28.360 INFO:teuthology.orchestra.run.smithi072.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-24T00:39:28.360 INFO:teuthology.orchestra.run.smithi072.stdout: applications will generate AVC messages and not be allowed access to 2024-03-24T00:39:28.360 INFO:teuthology.orchestra.run.smithi072.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-24T00:39:28.360 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-24T00:39:28.360 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:28.361 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-03-24T00:39:28.387 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:28.387 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T00:39:28.442 INFO:teuthology.orchestra.run.smithi072.stdout:client4938 2024-03-24T00:39:28.443 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:39:28.443 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:39:28.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:28.498 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T00:39:28.554 INFO:teuthology.orchestra.run.smithi072.stdout:client4938 2024-03-24T00:39:28.554 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:28.555 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T00:39:28.609 INFO:teuthology.orchestra.run.smithi072.stdout:client4938 2024-03-24T00:39:28.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:28.609 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/de1d0178-808e-49ca-a986-e297e523b976.client4938/status') 2024-03-24T00:39:28.675 INFO:teuthology.orchestra.run.smithi072.stdout:instance: client.4938 (3)172.21.15.72:0/1712186849 2024-03-24T00:39:28.675 INFO:teuthology.orchestra.run.smithi072.stdout:blocklisted: false 2024-03-24T00:39:28.676 INFO:teuthology.orchestra.run.smithi072.stderr:0+1 records in 2024-03-24T00:39:28.676 INFO:teuthology.orchestra.run.smithi072.stderr:0+1 records out 2024-03-24T00:39:28.676 INFO:teuthology.orchestra.run.smithi072.stderr:70 bytes copied, 2.0105e-05 s, 3.5 MB/s 2024-03-24T00:39:28.677 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-03-24T00:39:28.693 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:39:28.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.807+0000 7f52f0ca1640 1 Processor -- start 2024-03-24T00:39:28.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.807+0000 7f52f0ca1640 1 -- start start 2024-03-24T00:39:28.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.808+0000 7f52f0ca1640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f52ec0ccce0 0x7f52ec0cd0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:28.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.808+0000 7f52f0ca1640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52ec1606d0 con 0x7f52ec0cef40 2024-03-24T00:39:28.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.808+0000 7f52f0ca1640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52ec0cd600 con 0x7f52ec06ecf0 2024-03-24T00:39:28.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.808+0000 7f52f0ca1640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52ec072890 con 0x7f52ec0ccce0 2024-03-24T00:39:28.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.808+0000 7f52ea59c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f52ec0ccce0 0x7f52ec0cd0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:28.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.808+0000 7f52e9d9b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f52ec0cef40 0x7f52ec15f8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46910/0 (socket says 172.21.15.67:46910) 2024-03-24T00:39:28.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.808+0000 7f52e9d9b640 1 -- 172.21.15.67:0/1945581035 learned_addr learned my addr 172.21.15.67:0/1945581035 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:28.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.809+0000 7f52ea59c640 1 -- 172.21.15.67:0/1945581035 >> v1:172.21.15.72:6789/0 conn(0x7f52ec06ecf0 legacy=0x7f52ec15ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:28.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.809+0000 7f52ea59c640 1 -- 172.21.15.67:0/1945581035 >> v1:172.21.15.67:6789/0 conn(0x7f52ec0cef40 legacy=0x7f52ec15f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:28.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.809+0000 7f52ea59c640 1 -- 172.21.15.67:0/1945581035 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52ec15c580 con 0x7f52ec0ccce0 2024-03-24T00:39:28.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.809+0000 7f52ea59c640 1 --2- 172.21.15.67:0/1945581035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52ec0ccce0 0x7f52ec0cd0c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f52e40097a0 tx=0x7f52e402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7a56fd81e119c2f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:28.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.810+0000 7f52e959a640 1 -- 172.21.15.67:0/1945581035 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e4034070 con 0x7f52ec0ccce0 2024-03-24T00:39:28.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.810+0000 7f52e959a640 1 -- 172.21.15.67:0/1945581035 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52e4030610 con 0x7f52ec0ccce0 2024-03-24T00:39:28.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.810+0000 7f52e959a640 1 -- 172.21.15.67:0/1945581035 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e40309e0 con 0x7f52ec0ccce0 2024-03-24T00:39:28.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.810+0000 7f52f0ca1640 1 -- 172.21.15.67:0/1945581035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52ec0ccce0 msgr2=0x7f52ec0cd0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:28.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.810+0000 7f52f0ca1640 1 --2- 172.21.15.67:0/1945581035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52ec0ccce0 0x7f52ec0cd0c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f52e40097a0 tx=0x7f52e402b960 comp rx=0 tx=0).stop 2024-03-24T00:39:28.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.810+0000 7f52f0ca1640 1 -- 172.21.15.67:0/1945581035 shutdown_connections 2024-03-24T00:39:28.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.810+0000 7f52f0ca1640 1 --2- 172.21.15.67:0/1945581035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52ec0ccce0 0x7f52ec0cd0c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:28.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.810+0000 7f52f0ca1640 1 -- 172.21.15.67:0/1945581035 >> 172.21.15.67:0/1945581035 conn(0x7f52ec0cf9e0 msgr2=0x7f52ec0cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:28.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.810+0000 7f52f0ca1640 1 -- 172.21.15.67:0/1945581035 shutdown_connections 2024-03-24T00:39:28.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.810+0000 7f52f0ca1640 1 -- 172.21.15.67:0/1945581035 wait complete. 2024-03-24T00:39:28.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.812+0000 7f52f0ca1640 1 Processor -- start 2024-03-24T00:39:28.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.812+0000 7f52f0ca1640 1 -- start start 2024-03-24T00:39:28.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.812+0000 7f52f0ca1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52ec06ecf0 0x7f52ec15aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:28.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.812+0000 7f52f0ca1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f52ec0ccce0 0x7f52ec13e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:28.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.812+0000 7f52f0ca1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f52ec0cef40 0x7f52ec13ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:28.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.812+0000 7f52f0ca1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52ec15ce20 con 0x7f52ec06ecf0 2024-03-24T00:39:28.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.812+0000 7f52f0ca1640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52ec15cb40 con 0x7f52ec0ccce0 2024-03-24T00:39:28.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.812+0000 7f52f0ca1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52ec15c290 con 0x7f52ec0cef40 2024-03-24T00:39:28.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.813+0000 7f52ea59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52ec06ecf0 0x7f52ec15aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:28.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.813+0000 7f52ea59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52ec06ecf0 0x7f52ec15aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33976/0 (socket says 172.21.15.67:33976) 2024-03-24T00:39:28.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.813+0000 7f52ea59c640 1 -- 172.21.15.67:0/1161585192 learned_addr learned my addr 172.21.15.67:0/1161585192 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:28.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.813+0000 7f52e9d9b640 1 --2- 172.21.15.67:0/1161585192 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f52ec0ccce0 0x7f52ec13e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:28.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.813+0000 7f52ead9d640 1 --2- 172.21.15.67:0/1161585192 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f52ec0cef40 0x7f52ec13ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:28.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.813+0000 7f52ea59c640 1 -- 172.21.15.67:0/1161585192 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f52ec0cef40 msgr2=0x7f52ec13ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:28.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.813+0000 7f52ea59c640 1 --2- 172.21.15.67:0/1161585192 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f52ec0cef40 0x7f52ec13ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:28.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.813+0000 7f52ea59c640 1 -- 172.21.15.67:0/1161585192 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f52ec0ccce0 msgr2=0x7f52ec13e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:28.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.813+0000 7f52ea59c640 1 --2- 172.21.15.67:0/1161585192 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f52ec0ccce0 0x7f52ec13e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:28.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.813+0000 7f52ea59c640 1 -- 172.21.15.67:0/1161585192 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52ec1f2080 con 0x7f52ec06ecf0 2024-03-24T00:39:28.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.813+0000 7f52ea59c640 1 --2- 172.21.15.67:0/1161585192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52ec06ecf0 0x7f52ec15aef0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f52e402bf70 tx=0x7f52e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:28.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.813+0000 7f52e2ffd640 1 -- 172.21.15.67:0/1161585192 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e4034040 con 0x7f52ec06ecf0 2024-03-24T00:39:28.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.813+0000 7f52e2ffd640 1 -- 172.21.15.67:0/1161585192 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52e4039a10 con 0x7f52ec06ecf0 2024-03-24T00:39:28.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.813+0000 7f52f0ca1640 1 -- 172.21.15.67:0/1161585192 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52ec1f2360 con 0x7f52ec06ecf0 2024-03-24T00:39:28.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.813+0000 7f52e2ffd640 1 -- 172.21.15.67:0/1161585192 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e4039e20 con 0x7f52ec06ecf0 2024-03-24T00:39:28.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.813+0000 7f52f0ca1640 1 -- 172.21.15.67:0/1161585192 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52ec15b430 con 0x7f52ec06ecf0 2024-03-24T00:39:28.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.815+0000 7f52f0ca1640 1 -- 172.21.15.67:0/1161585192 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52ec04f400 con 0x7f52ec06ecf0 2024-03-24T00:39:28.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.815+0000 7f52e2ffd640 1 -- 172.21.15.67:0/1161585192 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f52e4037070 con 0x7f52ec06ecf0 2024-03-24T00:39:28.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.815+0000 7f52e2ffd640 1 --2- 172.21.15.67:0/1161585192 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f52cc06bbd0 0x7f52cc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:28.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.816+0000 7f52e2ffd640 1 -- 172.21.15.67:0/1161585192 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f52e40accf0 con 0x7f52ec06ecf0 2024-03-24T00:39:28.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.816+0000 7f52e9d9b640 1 --2- 172.21.15.67:0/1161585192 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f52cc06bbd0 0x7f52cc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:28.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.816+0000 7f52e9d9b640 1 --2- 172.21.15.67:0/1161585192 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f52cc06bbd0 0x7f52cc06e090 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f52ec15ba30 tx=0x7f52d4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:28.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:28.821+0000 7f52e2ffd640 1 -- 172.21.15.67:0/1161585192 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f52e40b1070 con 0x7f52ec06ecf0 2024-03-24T00:39:29.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.006+0000 7f52f0ca1640 1 -- 172.21.15.67:0/1161585192 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52ec0d0600 con 0x7f52ec06ecf0 2024-03-24T00:39:29.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.007+0000 7f52e2ffd640 1 -- 172.21.15.67:0/1161585192 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18454 (secure 0 0 0) 0x7f52e407ae50 con 0x7f52ec06ecf0 2024-03-24T00:39:29.007 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:39:29.007 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":25,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:38:58.797179+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:38:50.077085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:38:51.095454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T00:38:54.807117+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:39:29.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.009+0000 7f52f0ca1640 1 -- 172.21.15.67:0/1161585192 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f52cc06bbd0 msgr2=0x7f52cc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:29.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.009+0000 7f52f0ca1640 1 --2- 172.21.15.67:0/1161585192 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f52cc06bbd0 0x7f52cc06e090 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f52ec15ba30 tx=0x7f52d4009040 comp rx=0 tx=0).stop 2024-03-24T00:39:29.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.009+0000 7f52f0ca1640 1 -- 172.21.15.67:0/1161585192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52ec06ecf0 msgr2=0x7f52ec15aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:29.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.009+0000 7f52f0ca1640 1 --2- 172.21.15.67:0/1161585192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52ec06ecf0 0x7f52ec15aef0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f52e402bf70 tx=0x7f52e40046c0 comp rx=0 tx=0).stop 2024-03-24T00:39:29.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.009+0000 7f52f0ca1640 1 -- 172.21.15.67:0/1161585192 shutdown_connections 2024-03-24T00:39:29.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.009+0000 7f52f0ca1640 1 --2- 172.21.15.67:0/1161585192 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f52cc06bbd0 0x7f52cc06e090 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:29.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.009+0000 7f52f0ca1640 1 --2- 172.21.15.67:0/1161585192 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f52ec0cef40 0x7f52ec13ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:29.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.009+0000 7f52f0ca1640 1 --2- 172.21.15.67:0/1161585192 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f52ec0ccce0 0x7f52ec13e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:29.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.009+0000 7f52f0ca1640 1 --2- 172.21.15.67:0/1161585192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52ec06ecf0 0x7f52ec15aef0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:29.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.009+0000 7f52f0ca1640 1 -- 172.21.15.67:0/1161585192 >> 172.21.15.67:0/1161585192 conn(0x7f52ec0cf9e0 msgr2=0x7f52ec06b040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:29.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.009+0000 7f52f0ca1640 1 -- 172.21.15.67:0/1161585192 shutdown_connections 2024-03-24T00:39:29.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.009+0000 7f52f0ca1640 1 -- 172.21.15.67:0/1161585192 wait complete. 2024-03-24T00:39:29.020 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:38:39.364950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:38:50.077085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:38:51.095454+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T00:38:54.807117+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T00:39:29.021 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:39:29.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.126+0000 7f5217f53640 1 Processor -- start 2024-03-24T00:39:29.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.126+0000 7f5217f53640 1 -- start start 2024-03-24T00:39:29.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.127+0000 7f5217f53640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f52100b0f10 0x7f52100d1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:29.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.127+0000 7f5217f53640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f52100b2610 0x7f521015d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:29.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.127+0000 7f5217f53640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52100b12f0 con 0x7f52100d20d0 2024-03-24T00:39:29.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.127+0000 7f5217f53640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52100d2540 con 0x7f52100b0f10 2024-03-24T00:39:29.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.127+0000 7f5217f53640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52100d09f0 con 0x7f52100b2610 2024-03-24T00:39:29.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.127+0000 7f5215cef640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f52100b0f10 0x7f52100d1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:29.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.127+0000 7f5215cef640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f52100b0f10 0x7f52100d1b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33988/0 (socket says 172.21.15.67:33988) 2024-03-24T00:39:29.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.127+0000 7f5215cef640 1 -- 172.21.15.67:0/4032942179 learned_addr learned my addr 172.21.15.67:0/4032942179 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:29.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.127+0000 7f52164f0640 1 --2- 172.21.15.67:0/4032942179 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f52100b2610 0x7f521015d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:29.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.128+0000 7f52164f0640 1 -- 172.21.15.67:0/4032942179 >> v2:172.21.15.67:3300/0 conn(0x7f52100b0f10 msgr2=0x7f52100d1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:29.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.128+0000 7f52164f0640 1 --2- 172.21.15.67:0/4032942179 >> v2:172.21.15.67:3300/0 conn(0x7f52100b0f10 0x7f52100d1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:29.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.128+0000 7f52164f0640 1 -- 172.21.15.67:0/4032942179 >> v1:172.21.15.72:6789/0 conn(0x7f52100d20d0 legacy=0x7f52100b1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:29.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.128+0000 7f5214ced640 1 -- 172.21.15.67:0/4032942179 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4127083247 0 0) 0x7f52100b12f0 con 0x7f52100d20d0 2024-03-24T00:39:29.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.128+0000 7f52164f0640 1 -- 172.21.15.67:0/4032942179 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52100b2c50 con 0x7f52100b2610 2024-03-24T00:39:29.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.128+0000 7f5215cef640 1 --2- 172.21.15.67:0/4032942179 >> v2:172.21.15.67:3300/0 conn(0x7f52100b0f10 0x7f52100d1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:29.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.128+0000 7f52164f0640 1 --2- 172.21.15.67:0/4032942179 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f52100b2610 0x7f521015d870 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5208009d90 tx=0x7f520802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ca39a00bf1acf62 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:29.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.129+0000 7f5214ced640 1 -- 172.21.15.67:0/4032942179 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5208039070 con 0x7f52100b2610 2024-03-24T00:39:29.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.129+0000 7f5214ced640 1 -- 172.21.15.67:0/4032942179 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f520802cc50 con 0x7f52100b2610 2024-03-24T00:39:29.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.129+0000 7f5214ced640 1 -- 172.21.15.67:0/4032942179 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5208034440 con 0x7f52100b2610 2024-03-24T00:39:29.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.129+0000 7f5217f53640 1 -- 172.21.15.67:0/4032942179 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f52100b2610 msgr2=0x7f521015d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:29.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.129+0000 7f5217f53640 1 --2- 172.21.15.67:0/4032942179 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f52100b2610 0x7f521015d870 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5208009d90 tx=0x7f520802bea0 comp rx=0 tx=0).stop 2024-03-24T00:39:29.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.129+0000 7f5217f53640 1 -- 172.21.15.67:0/4032942179 shutdown_connections 2024-03-24T00:39:29.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.129+0000 7f5217f53640 1 --2- 172.21.15.67:0/4032942179 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f52100b2610 0x7f521015d870 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:29.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.129+0000 7f5217f53640 1 --2- 172.21.15.67:0/4032942179 >> v2:172.21.15.67:3300/0 conn(0x7f52100b0f10 0x7f52100d1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:29.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.129+0000 7f5217f53640 1 -- 172.21.15.67:0/4032942179 >> 172.21.15.67:0/4032942179 conn(0x7f521008b1b0 msgr2=0x7f521008b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:29.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.129+0000 7f5217f53640 1 -- 172.21.15.67:0/4032942179 shutdown_connections 2024-03-24T00:39:29.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.129+0000 7f5217f53640 1 -- 172.21.15.67:0/4032942179 wait complete. 2024-03-24T00:39:29.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.130+0000 7f5217f53640 1 Processor -- start 2024-03-24T00:39:29.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.130+0000 7f5217f53640 1 -- start start 2024-03-24T00:39:29.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.131+0000 7f5217f53640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f52100b0f10 0x7f52100cbb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:29.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.131+0000 7f5217f53640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f52100b2610 0x7f52101f5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:29.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.131+0000 7f5217f53640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52100d20d0 0x7f52101f1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:29.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.131+0000 7f5217f53640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52100cee40 con 0x7f52100d20d0 2024-03-24T00:39:29.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.131+0000 7f5217f53640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52100ceb60 con 0x7f52100b2610 2024-03-24T00:39:29.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.131+0000 7f5217f53640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52100ce560 con 0x7f52100b0f10 2024-03-24T00:39:29.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.131+0000 7f52164f0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52100d20d0 0x7f52101f1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:29.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.131+0000 7f52164f0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52100d20d0 0x7f52101f1ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34002/0 (socket says 172.21.15.67:34002) 2024-03-24T00:39:29.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.131+0000 7f52164f0640 1 -- 172.21.15.67:0/527370064 learned_addr learned my addr 172.21.15.67:0/527370064 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:29.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.131+0000 7f5215cef640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f52100b0f10 0x7f52100cbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:29.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.131+0000 7f52154ee640 1 --2- 172.21.15.67:0/527370064 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f52100b2610 0x7f52101f5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:29.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.132+0000 7f52164f0640 1 -- 172.21.15.67:0/527370064 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f52100b0f10 msgr2=0x7f52100cbb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:29.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.132+0000 7f52164f0640 1 --2- 172.21.15.67:0/527370064 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f52100b0f10 0x7f52100cbb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:29.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.132+0000 7f52164f0640 1 -- 172.21.15.67:0/527370064 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f52100b2610 msgr2=0x7f52101f5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:29.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.132+0000 7f52164f0640 1 --2- 172.21.15.67:0/527370064 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f52100b2610 0x7f52101f5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:29.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.132+0000 7f52164f0640 1 -- 172.21.15.67:0/527370064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52101f5880 con 0x7f52100d20d0 2024-03-24T00:39:29.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.132+0000 7f5215cef640 1 --2- 172.21.15.67:0/527370064 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f52100b0f10 0x7f52100cbb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:29.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.132+0000 7f52164f0640 1 --2- 172.21.15.67:0/527370064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52100d20d0 0x7f52101f1ab0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5208037040 tx=0x7f5208004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:29.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.132+0000 7f52027fc640 1 -- 172.21.15.67:0/527370064 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5208044020 con 0x7f52100d20d0 2024-03-24T00:39:29.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.132+0000 7f52027fc640 1 -- 172.21.15.67:0/527370064 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f520803a040 con 0x7f52100d20d0 2024-03-24T00:39:29.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.132+0000 7f5217f53640 1 -- 172.21.15.67:0/527370064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52101f5b60 con 0x7f52100d20d0 2024-03-24T00:39:29.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.132+0000 7f52027fc640 1 -- 172.21.15.67:0/527370064 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5208039050 con 0x7f52100d20d0 2024-03-24T00:39:29.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.132+0000 7f5217f53640 1 -- 172.21.15.67:0/527370064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52101f6120 con 0x7f52100d20d0 2024-03-24T00:39:29.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.133+0000 7f5217f53640 1 -- 172.21.15.67:0/527370064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f521004f400 con 0x7f52100d20d0 2024-03-24T00:39:29.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.133+0000 7f52027fc640 1 -- 172.21.15.67:0/527370064 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5208003990 con 0x7f52100d20d0 2024-03-24T00:39:29.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.134+0000 7f52027fc640 1 --2- 172.21.15.67:0/527370064 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f51f406bbd0 0x7f51f406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:29.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.134+0000 7f52027fc640 1 -- 172.21.15.67:0/527370064 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f52080ad0a0 con 0x7f52100d20d0 2024-03-24T00:39:29.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.134+0000 7f5215cef640 1 --2- 172.21.15.67:0/527370064 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f51f406bbd0 0x7f51f406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:29.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.134+0000 7f5215cef640 1 --2- 172.21.15.67:0/527370064 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f51f406bbd0 0x7f51f406e090 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5204009630 tx=0x7f5204009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:29.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.138+0000 7f52027fc640 1 -- 172.21.15.67:0/527370064 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f520804e060 con 0x7f52100d20d0 2024-03-24T00:39:29.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.320+0000 7f5217f53640 1 -- 172.21.15.67:0/527370064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f52100d07d0 con 0x7f52100d20d0 2024-03-24T00:39:29.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.321+0000 7f52027fc640 1 -- 172.21.15.67:0/527370064 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7f520807b210 con 0x7f52100d20d0 2024-03-24T00:39:29.321 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-24T00:39:29.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.323+0000 7f5217f53640 1 -- 172.21.15.67:0/527370064 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f51f406bbd0 msgr2=0x7f51f406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:29.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.323+0000 7f5217f53640 1 --2- 172.21.15.67:0/527370064 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f51f406bbd0 0x7f51f406e090 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5204009630 tx=0x7f5204009240 comp rx=0 tx=0).stop 2024-03-24T00:39:29.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.323+0000 7f5217f53640 1 -- 172.21.15.67:0/527370064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52100d20d0 msgr2=0x7f52101f1ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:29.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.323+0000 7f5217f53640 1 --2- 172.21.15.67:0/527370064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52100d20d0 0x7f52101f1ab0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5208037040 tx=0x7f5208004a40 comp rx=0 tx=0).stop 2024-03-24T00:39:29.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.323+0000 7f5217f53640 1 -- 172.21.15.67:0/527370064 shutdown_connections 2024-03-24T00:39:29.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.323+0000 7f5217f53640 1 --2- 172.21.15.67:0/527370064 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f51f406bbd0 0x7f51f406e090 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:29.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.323+0000 7f5217f53640 1 --2- 172.21.15.67:0/527370064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52100d20d0 0x7f52101f1ab0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:29.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.323+0000 7f5217f53640 1 --2- 172.21.15.67:0/527370064 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f52100b2610 0x7f52101f5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:29.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.323+0000 7f5217f53640 1 --2- 172.21.15.67:0/527370064 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f52100b0f10 0x7f52100cbb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:29.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.323+0000 7f5217f53640 1 -- 172.21.15.67:0/527370064 >> 172.21.15.67:0/527370064 conn(0x7f521008b1b0 msgr2=0x7f5210161800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:29.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.323+0000 7f5217f53640 1 -- 172.21.15.67:0/527370064 shutdown_connections 2024-03-24T00:39:29.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.323+0000 7f5217f53640 1 -- 172.21.15.67:0/527370064 wait complete. 2024-03-24T00:39:29.333 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:39:29.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.440+0000 7fbf67759640 1 Processor -- start 2024-03-24T00:39:29.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.441+0000 7fbf67759640 1 -- start start 2024-03-24T00:39:29.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.441+0000 7fbf67759640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbf6015dea0 0x7fbf600d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:29.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.441+0000 7fbf67759640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbf600b6170 0x7fbf600d3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:29.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.441+0000 7fbf67759640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf600d2250 con 0x7fbf6015e280 2024-03-24T00:39:29.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.441+0000 7fbf67759640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf600d2530 con 0x7fbf600b6170 2024-03-24T00:39:29.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.441+0000 7fbf67759640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf600d2810 con 0x7fbf6015dea0 2024-03-24T00:39:29.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.441+0000 7fbf65cf6640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbf600b6170 0x7fbf600d3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:29.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.441+0000 7fbf64cf4640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbf6015e280 0x7fbf600b5a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46926/0 (socket says 172.21.15.67:46926) 2024-03-24T00:39:29.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.441+0000 7fbf65cf6640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbf600b6170 0x7fbf600d3910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34010/0 (socket says 172.21.15.67:34010) 2024-03-24T00:39:29.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.441+0000 7fbf64cf4640 1 -- 172.21.15.67:0/3435579351 learned_addr learned my addr 172.21.15.67:0/3435579351 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:29.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.441+0000 7fbf654f5640 1 --2- 172.21.15.67:0/3435579351 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbf6015dea0 0x7fbf600d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:29.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.442+0000 7fbf65cf6640 1 -- 172.21.15.67:0/3435579351 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbf6015dea0 msgr2=0x7fbf600d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:29.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.442+0000 7fbf65cf6640 1 --2- 172.21.15.67:0/3435579351 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbf6015dea0 0x7fbf600d33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:29.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.442+0000 7fbf65cf6640 1 -- 172.21.15.67:0/3435579351 >> v1:172.21.15.67:6789/0 conn(0x7fbf6015e280 legacy=0x7fbf600b5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:29.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.442+0000 7fbf65cf6640 1 -- 172.21.15.67:0/3435579351 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf600d2af0 con 0x7fbf600b6170 2024-03-24T00:39:29.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.442+0000 7fbf654f5640 1 --2- 172.21.15.67:0/3435579351 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbf6015dea0 0x7fbf600d33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:29.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.443+0000 7fbf65cf6640 1 --2- 172.21.15.67:0/3435579351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf600b6170 0x7fbf600d3910 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c009d90 tx=0x7fbf5c02be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f93b945f882993e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:29.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.443+0000 7fbf57fff640 1 -- 172.21.15.67:0/3435579351 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf5c039070 con 0x7fbf600b6170 2024-03-24T00:39:29.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.443+0000 7fbf57fff640 1 -- 172.21.15.67:0/3435579351 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf5c02cbc0 con 0x7fbf600b6170 2024-03-24T00:39:29.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.443+0000 7fbf57fff640 1 -- 172.21.15.67:0/3435579351 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf5c034440 con 0x7fbf600b6170 2024-03-24T00:39:29.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.443+0000 7fbf67759640 1 -- 172.21.15.67:0/3435579351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf600b6170 msgr2=0x7fbf600d3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:29.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.443+0000 7fbf67759640 1 --2- 172.21.15.67:0/3435579351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf600b6170 0x7fbf600d3910 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c009d90 tx=0x7fbf5c02be10 comp rx=0 tx=0).stop 2024-03-24T00:39:29.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.443+0000 7fbf67759640 1 -- 172.21.15.67:0/3435579351 shutdown_connections 2024-03-24T00:39:29.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.443+0000 7fbf67759640 1 --2- 172.21.15.67:0/3435579351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf600b6170 0x7fbf600d3910 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:29.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.443+0000 7fbf67759640 1 --2- 172.21.15.67:0/3435579351 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbf6015dea0 0x7fbf600d33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:29.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.443+0000 7fbf67759640 1 -- 172.21.15.67:0/3435579351 >> 172.21.15.67:0/3435579351 conn(0x7fbf60072da0 msgr2=0x7fbf600d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:29.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.444+0000 7fbf67759640 1 -- 172.21.15.67:0/3435579351 shutdown_connections 2024-03-24T00:39:29.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.444+0000 7fbf67759640 1 -- 172.21.15.67:0/3435579351 wait complete. 2024-03-24T00:39:29.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.445+0000 7fbf67759640 1 Processor -- start 2024-03-24T00:39:29.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.445+0000 7fbf67759640 1 -- start start 2024-03-24T00:39:29.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.446+0000 7fbf67759640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf600b6170 0x7fbf601533e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:29.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.446+0000 7fbf67759640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbf6015dea0 0x7fbf600cdd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:29.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.446+0000 7fbf67759640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf6015e280 0x7fbf601554f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:29.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.446+0000 7fbf67759640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf600b7480 con 0x7fbf6015e280 2024-03-24T00:39:29.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.446+0000 7fbf67759640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf600b7300 con 0x7fbf600b6170 2024-03-24T00:39:29.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.446+0000 7fbf67759640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf600b7600 con 0x7fbf6015dea0 2024-03-24T00:39:29.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.446+0000 7fbf65cf6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf6015e280 0x7fbf601554f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:29.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.446+0000 7fbf64cf4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbf6015dea0 0x7fbf600cdd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:29.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.446+0000 7fbf65cf6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf6015e280 0x7fbf601554f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34026/0 (socket says 172.21.15.67:34026) 2024-03-24T00:39:29.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.446+0000 7fbf65cf6640 1 -- 172.21.15.67:0/3041989748 learned_addr learned my addr 172.21.15.67:0/3041989748 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:29.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.446+0000 7fbf654f5640 1 --2- 172.21.15.67:0/3041989748 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf600b6170 0x7fbf601533e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:29.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.446+0000 7fbf65cf6640 1 -- 172.21.15.67:0/3041989748 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbf6015dea0 msgr2=0x7fbf600cdd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:29.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.446+0000 7fbf65cf6640 1 --2- 172.21.15.67:0/3041989748 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbf6015dea0 0x7fbf600cdd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:29.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.446+0000 7fbf65cf6640 1 -- 172.21.15.67:0/3041989748 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf600b6170 msgr2=0x7fbf601533e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:29.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.446+0000 7fbf65cf6640 1 --2- 172.21.15.67:0/3041989748 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf600b6170 0x7fbf601533e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:29.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.446+0000 7fbf65cf6640 1 -- 172.21.15.67:0/3041989748 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf60154450 con 0x7fbf6015e280 2024-03-24T00:39:29.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.446+0000 7fbf65cf6640 1 --2- 172.21.15.67:0/3041989748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf6015e280 0x7fbf601554f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c009d90 tx=0x7fbf5c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:29.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.446+0000 7fbf55ffb640 1 -- 172.21.15.67:0/3041989748 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf5c039050 con 0x7fbf6015e280 2024-03-24T00:39:29.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.446+0000 7fbf55ffb640 1 -- 172.21.15.67:0/3041989748 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf5c045070 con 0x7fbf6015e280 2024-03-24T00:39:29.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.446+0000 7fbf55ffb640 1 -- 172.21.15.67:0/3041989748 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf5c009450 con 0x7fbf6015e280 2024-03-24T00:39:29.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.446+0000 7fbf67759640 1 -- 172.21.15.67:0/3041989748 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf60154730 con 0x7fbf6015e280 2024-03-24T00:39:29.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.447+0000 7fbf67759640 1 -- 172.21.15.67:0/3041989748 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf60154cf0 con 0x7fbf6015e280 2024-03-24T00:39:29.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.448+0000 7fbf67759640 1 -- 172.21.15.67:0/3041989748 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf6004f400 con 0x7fbf6015e280 2024-03-24T00:39:29.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.448+0000 7fbf55ffb640 1 -- 172.21.15.67:0/3041989748 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fbf5c03e850 con 0x7fbf6015e280 2024-03-24T00:39:29.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.449+0000 7fbf55ffb640 1 --2- 172.21.15.67:0/3041989748 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbf4406bbd0 0x7fbf4406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:29.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.449+0000 7fbf55ffb640 1 -- 172.21.15.67:0/3041989748 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fbf5c0adb60 con 0x7fbf6015e280 2024-03-24T00:39:29.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.449+0000 7fbf654f5640 1 --2- 172.21.15.67:0/3041989748 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbf4406bbd0 0x7fbf4406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:29.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.449+0000 7fbf654f5640 1 --2- 172.21.15.67:0/3041989748 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbf4406bbd0 0x7fbf4406e090 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbf58009560 tx=0x7fbf58009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:29.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.453+0000 7fbf55ffb640 1 -- 172.21.15.67:0/3041989748 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fbf5c07bde0 con 0x7fbf6015e280 2024-03-24T00:39:29.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.635+0000 7fbf67759640 1 -- 172.21.15.67:0/3041989748 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbf600b50e0 con 0x7fbf6015e280 2024-03-24T00:39:29.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.636+0000 7fbf55ffb640 1 -- 172.21.15.67:0/3041989748 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbf5c07fe80 con 0x7fbf6015e280 2024-03-24T00:39:29.636 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-24T00:39:29.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.638+0000 7fbf67759640 1 -- 172.21.15.67:0/3041989748 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbf4406bbd0 msgr2=0x7fbf4406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:29.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.638+0000 7fbf67759640 1 --2- 172.21.15.67:0/3041989748 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbf4406bbd0 0x7fbf4406e090 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbf58009560 tx=0x7fbf58009190 comp rx=0 tx=0).stop 2024-03-24T00:39:29.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.638+0000 7fbf67759640 1 -- 172.21.15.67:0/3041989748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf6015e280 msgr2=0x7fbf601554f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:29.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.638+0000 7fbf67759640 1 --2- 172.21.15.67:0/3041989748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf6015e280 0x7fbf601554f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c009d90 tx=0x7fbf5c004a40 comp rx=0 tx=0).stop 2024-03-24T00:39:29.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.638+0000 7fbf67759640 1 -- 172.21.15.67:0/3041989748 shutdown_connections 2024-03-24T00:39:29.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.638+0000 7fbf67759640 1 --2- 172.21.15.67:0/3041989748 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbf4406bbd0 0x7fbf4406e090 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:29.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.638+0000 7fbf67759640 1 --2- 172.21.15.67:0/3041989748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf6015e280 0x7fbf601554f0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:29.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.638+0000 7fbf67759640 1 --2- 172.21.15.67:0/3041989748 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbf6015dea0 0x7fbf600cdd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:29.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.638+0000 7fbf67759640 1 --2- 172.21.15.67:0/3041989748 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf600b6170 0x7fbf601533e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:29.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.638+0000 7fbf67759640 1 -- 172.21.15.67:0/3041989748 >> 172.21.15.67:0/3041989748 conn(0x7fbf60072da0 msgr2=0x7fbf600d3170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:29.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.638+0000 7fbf67759640 1 -- 172.21.15.67:0/3041989748 shutdown_connections 2024-03-24T00:39:29.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.638+0000 7fbf67759640 1 -- 172.21.15.67:0/3041989748 wait complete. 2024-03-24T00:39:29.649 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:39:29.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.754+0000 7f5644745640 1 Processor -- start 2024-03-24T00:39:29.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.755+0000 7f5644745640 1 -- start start 2024-03-24T00:39:29.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.755+0000 7f5644745640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f563c15dea0 0x7f563c0d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:29.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.755+0000 7f5644745640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f563c0d2250 con 0x7f563c15e280 2024-03-24T00:39:29.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.755+0000 7f5644745640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f563c0d2530 con 0x7f563c0b6170 2024-03-24T00:39:29.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.755+0000 7f5644745640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f563c0d2810 con 0x7f563c15dea0 2024-03-24T00:39:29.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.756+0000 7f5641ce0640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f563c15e280 0x7f563c0b5a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46942/0 (socket says 172.21.15.67:46942) 2024-03-24T00:39:29.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.756+0000 7f5641ce0640 1 -- 172.21.15.67:0/3420652911 learned_addr learned my addr 172.21.15.67:0/3420652911 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:29.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.756+0000 7f56424e1640 1 --2- 172.21.15.67:0/3420652911 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f563c15dea0 0x7f563c0d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:29.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.756+0000 7f56414df640 1 -- 172.21.15.67:0/3420652911 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 425883407 0 0) 0x7f563c0d2250 con 0x7f563c15e280 2024-03-24T00:39:29.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.756+0000 7f56414df640 1 -- 172.21.15.67:0/3420652911 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5628003520 con 0x7f563c15e280 2024-03-24T00:39:29.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.756+0000 7f56414df640 1 -- 172.21.15.67:0/3420652911 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1923337834 0 0) 0x7f563c0d2530 con 0x7f563c0b6170 2024-03-24T00:39:29.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.756+0000 7f56424e1640 1 -- 172.21.15.67:0/3420652911 >> v1:172.21.15.72:6789/0 conn(0x7f563c0b6170 legacy=0x7f563c0b70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:29.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.756+0000 7f56424e1640 1 -- 172.21.15.67:0/3420652911 >> v1:172.21.15.67:6789/0 conn(0x7f563c15e280 legacy=0x7f563c0b5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:29.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.756+0000 7f56424e1640 1 -- 172.21.15.67:0/3420652911 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f563c0d2af0 con 0x7f563c15dea0 2024-03-24T00:39:29.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.757+0000 7f56424e1640 1 --2- 172.21.15.67:0/3420652911 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f563c15dea0 0x7f563c0d33d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f56380097d0 tx=0x7f563802b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d65250ce1341537 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:29.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.757+0000 7f56414df640 1 -- 172.21.15.67:0/3420652911 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5638035020 con 0x7f563c15dea0 2024-03-24T00:39:29.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.757+0000 7f56414df640 1 -- 172.21.15.67:0/3420652911 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5638031800 con 0x7f563c15dea0 2024-03-24T00:39:29.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.757+0000 7f56414df640 1 -- 172.21.15.67:0/3420652911 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5638031bb0 con 0x7f563c15dea0 2024-03-24T00:39:29.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.757+0000 7f5644745640 1 -- 172.21.15.67:0/3420652911 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f563c15dea0 msgr2=0x7f563c0d33d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:29.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.757+0000 7f5644745640 1 --2- 172.21.15.67:0/3420652911 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f563c15dea0 0x7f563c0d33d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f56380097d0 tx=0x7f563802b8a0 comp rx=0 tx=0).stop 2024-03-24T00:39:29.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.758+0000 7f5644745640 1 -- 172.21.15.67:0/3420652911 shutdown_connections 2024-03-24T00:39:29.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.758+0000 7f5644745640 1 --2- 172.21.15.67:0/3420652911 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f563c15dea0 0x7f563c0d33d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:29.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.758+0000 7f5644745640 1 -- 172.21.15.67:0/3420652911 >> 172.21.15.67:0/3420652911 conn(0x7f563c072da0 msgr2=0x7f563c0d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:29.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.758+0000 7f5644745640 1 -- 172.21.15.67:0/3420652911 shutdown_connections 2024-03-24T00:39:29.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.758+0000 7f5644745640 1 -- 172.21.15.67:0/3420652911 wait complete. 2024-03-24T00:39:29.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.759+0000 7f5644745640 1 Processor -- start 2024-03-24T00:39:29.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.759+0000 7f5644745640 1 -- start start 2024-03-24T00:39:29.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.759+0000 7f5644745640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f563c0b6170 0x7f563c0ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:29.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.759+0000 7f5644745640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f563c15dea0 0x7f563c1f2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:29.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.759+0000 7f5644745640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f563c15e280 0x7f563c0acc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:29.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.759+0000 7f5644745640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f563c0d3d20 con 0x7f563c0b6170 2024-03-24T00:39:29.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.759+0000 7f5644745640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f563c0d3ba0 con 0x7f563c15dea0 2024-03-24T00:39:29.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.759+0000 7f5644745640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f563c0b7800 con 0x7f563c15e280 2024-03-24T00:39:29.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.759+0000 7f56424e1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f563c0b6170 0x7f563c0ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:29.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.760+0000 7f5641ce0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f563c15dea0 0x7f563c1f2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:29.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.760+0000 7f56424e1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f563c0b6170 0x7f563c0ccab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34032/0 (socket says 172.21.15.67:34032) 2024-03-24T00:39:29.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.760+0000 7f56424e1640 1 -- 172.21.15.67:0/1860299492 learned_addr learned my addr 172.21.15.67:0/1860299492 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:29.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.760+0000 7f5642ce2640 1 --2- 172.21.15.67:0/1860299492 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f563c15e280 0x7f563c0acc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:29.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.760+0000 7f56424e1640 1 -- 172.21.15.67:0/1860299492 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f563c15e280 msgr2=0x7f563c0acc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:29.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.760+0000 7f56424e1640 1 --2- 172.21.15.67:0/1860299492 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f563c15e280 0x7f563c0acc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:29.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.760+0000 7f56424e1640 1 -- 172.21.15.67:0/1860299492 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f563c15dea0 msgr2=0x7f563c1f2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:29.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.760+0000 7f56424e1640 1 --2- 172.21.15.67:0/1860299492 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f563c15dea0 0x7f563c1f2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:29.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.760+0000 7f56424e1640 1 -- 172.21.15.67:0/1860299492 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f563c1f3340 con 0x7f563c0b6170 2024-03-24T00:39:29.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.760+0000 7f56424e1640 1 --2- 172.21.15.67:0/1860299492 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f563c0b6170 0x7f563c0ccab0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f5638000c00 tx=0x7f56380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:29.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.760+0000 7f562effd640 1 -- 172.21.15.67:0/1860299492 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5638041070 con 0x7f563c0b6170 2024-03-24T00:39:29.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.760+0000 7f562effd640 1 -- 172.21.15.67:0/1860299492 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f563803ad00 con 0x7f563c0b6170 2024-03-24T00:39:29.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.760+0000 7f5644745640 1 -- 172.21.15.67:0/1860299492 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f563c0cd0f0 con 0x7f563c0b6170 2024-03-24T00:39:29.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.760+0000 7f562effd640 1 -- 172.21.15.67:0/1860299492 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5638035040 con 0x7f563c0b6170 2024-03-24T00:39:29.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.760+0000 7f5644745640 1 -- 172.21.15.67:0/1860299492 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f563c0cd6b0 con 0x7f563c0b6170 2024-03-24T00:39:29.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.761+0000 7f5644745640 1 -- 172.21.15.67:0/1860299492 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f563c04f400 con 0x7f563c0b6170 2024-03-24T00:39:29.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.762+0000 7f562effd640 1 -- 172.21.15.67:0/1860299492 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f563804c060 con 0x7f563c0b6170 2024-03-24T00:39:29.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.762+0000 7f562effd640 1 --2- 172.21.15.67:0/1860299492 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f562006bbd0 0x7f562006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:29.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.762+0000 7f562effd640 1 -- 172.21.15.67:0/1860299492 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f56380ac7c0 con 0x7f563c0b6170 2024-03-24T00:39:29.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.762+0000 7f5641ce0640 1 --2- 172.21.15.67:0/1860299492 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f562006bbd0 0x7f562006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:29.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.763+0000 7f5641ce0640 1 --2- 172.21.15.67:0/1860299492 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f562006bbd0 0x7f562006e090 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f563c0b3c50 tx=0x7f5630009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:29.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.767+0000 7f562effd640 1 -- 172.21.15.67:0/1860299492 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f563807a920 con 0x7f563c0b6170 2024-03-24T00:39:29.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.953+0000 7f5644745640 1 -- 172.21.15.67:0/1860299492 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f563c0d2360 con 0x7f563c0b6170 2024-03-24T00:39:29.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.953+0000 7f562effd640 1 -- 172.21.15.67:0/1860299492 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7f563807e9c0 con 0x7f563c0b6170 2024-03-24T00:39:29.953 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-03-24T00:39:29.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.955+0000 7f5644745640 1 -- 172.21.15.67:0/1860299492 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f562006bbd0 msgr2=0x7f562006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:29.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.955+0000 7f5644745640 1 --2- 172.21.15.67:0/1860299492 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f562006bbd0 0x7f562006e090 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f563c0b3c50 tx=0x7f5630009040 comp rx=0 tx=0).stop 2024-03-24T00:39:29.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.955+0000 7f5644745640 1 -- 172.21.15.67:0/1860299492 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f563c0b6170 msgr2=0x7f563c0ccab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:29.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.955+0000 7f5644745640 1 --2- 172.21.15.67:0/1860299492 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f563c0b6170 0x7f563c0ccab0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f5638000c00 tx=0x7f56380046c0 comp rx=0 tx=0).stop 2024-03-24T00:39:29.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.955+0000 7f5644745640 1 -- 172.21.15.67:0/1860299492 shutdown_connections 2024-03-24T00:39:29.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.955+0000 7f5644745640 1 --2- 172.21.15.67:0/1860299492 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f562006bbd0 0x7f562006e090 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:29.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.955+0000 7f5644745640 1 --2- 172.21.15.67:0/1860299492 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f563c15e280 0x7f563c0acc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:29.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.955+0000 7f5644745640 1 --2- 172.21.15.67:0/1860299492 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f563c15dea0 0x7f563c1f2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:29.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.955+0000 7f5644745640 1 --2- 172.21.15.67:0/1860299492 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f563c0b6170 0x7f563c0ccab0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:29.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.955+0000 7f5644745640 1 -- 172.21.15.67:0/1860299492 >> 172.21.15.67:0/1860299492 conn(0x7f563c072da0 msgr2=0x7f563c0b5760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:29.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.956+0000 7f5644745640 1 -- 172.21.15.67:0/1860299492 shutdown_connections 2024-03-24T00:39:29.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:29.956+0000 7f5644745640 1 -- 172.21.15.67:0/1860299492 wait complete. 2024-03-24T00:39:29.966 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T00:39:30.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.071+0000 7ff21a710640 1 Processor -- start 2024-03-24T00:39:30.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.071+0000 7ff21a710640 1 -- start start 2024-03-24T00:39:30.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.072+0000 7ff21a710640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff2140b0f10 0x7ff2140d1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:30.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.072+0000 7ff21a710640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff2140d20d0 0x7ff2140b1f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:30.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.072+0000 7ff21a710640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2140b12f0 con 0x7ff2140b2440 2024-03-24T00:39:30.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.072+0000 7ff21a710640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2140d2510 con 0x7ff2140b0f10 2024-03-24T00:39:30.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.072+0000 7ff21a710640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2140b2c50 con 0x7ff2140d20d0 2024-03-24T00:39:30.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.072+0000 7ff213fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff2140b0f10 0x7ff2140d1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:30.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.072+0000 7ff213fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff2140b0f10 0x7ff2140d1b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34038/0 (socket says 172.21.15.67:34038) 2024-03-24T00:39:30.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.072+0000 7ff213fff640 1 -- 172.21.15.67:0/100548563 learned_addr learned my addr 172.21.15.67:0/100548563 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:30.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.072+0000 7ff218cad640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff2140b2440 0x7ff2140cf9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46958/0 (socket says 172.21.15.67:46958) 2024-03-24T00:39:30.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.072+0000 7ff2137fe640 1 --2- 172.21.15.67:0/100548563 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff2140d20d0 0x7ff2140b1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:30.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.072+0000 7ff213fff640 1 -- 172.21.15.67:0/100548563 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff2140d20d0 msgr2=0x7ff2140b1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:30.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.072+0000 7ff213fff640 1 --2- 172.21.15.67:0/100548563 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff2140d20d0 0x7ff2140b1f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:30.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.072+0000 7ff213fff640 1 -- 172.21.15.67:0/100548563 >> v1:172.21.15.67:6789/0 conn(0x7ff2140b2440 legacy=0x7ff2140cf9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:30.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.072+0000 7ff212ffd640 1 -- 172.21.15.67:0/100548563 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4153253988 0 0) 0x7ff2140b12f0 con 0x7ff2140b2440 2024-03-24T00:39:30.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.072+0000 7ff213fff640 1 -- 172.21.15.67:0/100548563 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2140ce880 con 0x7ff2140b0f10 2024-03-24T00:39:30.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.073+0000 7ff213fff640 1 --2- 172.21.15.67:0/100548563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2140b0f10 0x7ff2140d1b90 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff204009870 tx=0x7ff20402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f717e21a259ebac1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:30.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.073+0000 7ff212ffd640 1 -- 172.21.15.67:0/100548563 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff204034070 con 0x7ff2140b0f10 2024-03-24T00:39:30.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.073+0000 7ff212ffd640 1 -- 172.21.15.67:0/100548563 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff204030840 con 0x7ff2140b0f10 2024-03-24T00:39:30.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.073+0000 7ff212ffd640 1 -- 172.21.15.67:0/100548563 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff204030c10 con 0x7ff2140b0f10 2024-03-24T00:39:30.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.073+0000 7ff21a710640 1 -- 172.21.15.67:0/100548563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2140b0f10 msgr2=0x7ff2140d1b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:30.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.074+0000 7ff21a710640 1 --2- 172.21.15.67:0/100548563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2140b0f10 0x7ff2140d1b90 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff204009870 tx=0x7ff20402b980 comp rx=0 tx=0).stop 2024-03-24T00:39:30.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.074+0000 7ff21a710640 1 -- 172.21.15.67:0/100548563 shutdown_connections 2024-03-24T00:39:30.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.074+0000 7ff21a710640 1 --2- 172.21.15.67:0/100548563 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff2140d20d0 0x7ff2140b1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:30.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.074+0000 7ff21a710640 1 --2- 172.21.15.67:0/100548563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2140b0f10 0x7ff2140d1b90 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:30.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.074+0000 7ff21a710640 1 -- 172.21.15.67:0/100548563 >> 172.21.15.67:0/100548563 conn(0x7ff21408b1b0 msgr2=0x7ff21408b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:30.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.074+0000 7ff21a710640 1 -- 172.21.15.67:0/100548563 shutdown_connections 2024-03-24T00:39:30.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.074+0000 7ff21a710640 1 -- 172.21.15.67:0/100548563 wait complete. 2024-03-24T00:39:30.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.075+0000 7ff21a710640 1 Processor -- start 2024-03-24T00:39:30.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.075+0000 7ff21a710640 1 -- start start 2024-03-24T00:39:30.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.076+0000 7ff21a710640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2140b0f10 0x7ff214151100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:30.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.076+0000 7ff21a710640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff2140b2440 0x7ff2141519c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:30.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.076+0000 7ff21a710640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2140d20d0 0x7ff2140b54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:30.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.076+0000 7ff21a710640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2140ce560 con 0x7ff2140d20d0 2024-03-24T00:39:30.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.076+0000 7ff21a710640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2140cee40 con 0x7ff2140b0f10 2024-03-24T00:39:30.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.076+0000 7ff21a710640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2140b28b0 con 0x7ff2140b2440 2024-03-24T00:39:30.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.076+0000 7ff218cad640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2140d20d0 0x7ff2140b54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:30.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.076+0000 7ff218cad640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2140d20d0 0x7ff2140b54d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34054/0 (socket says 172.21.15.67:34054) 2024-03-24T00:39:30.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.076+0000 7ff2137fe640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff2140b2440 0x7ff2141519c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:30.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.076+0000 7ff218cad640 1 -- 172.21.15.67:0/3883145792 learned_addr learned my addr 172.21.15.67:0/3883145792 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:30.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.076+0000 7ff213fff640 1 --2- 172.21.15.67:0/3883145792 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2140b0f10 0x7ff214151100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:30.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.076+0000 7ff218cad640 1 -- 172.21.15.67:0/3883145792 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff2140b2440 msgr2=0x7ff2141519c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:30.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.076+0000 7ff218cad640 1 --2- 172.21.15.67:0/3883145792 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff2140b2440 0x7ff2141519c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:30.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.076+0000 7ff218cad640 1 -- 172.21.15.67:0/3883145792 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2140b0f10 msgr2=0x7ff214151100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:30.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.076+0000 7ff218cad640 1 --2- 172.21.15.67:0/3883145792 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2140b0f10 0x7ff214151100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:30.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.076+0000 7ff218cad640 1 -- 172.21.15.67:0/3883145792 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff214151f00 con 0x7ff2140d20d0 2024-03-24T00:39:30.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.076+0000 7ff218cad640 1 --2- 172.21.15.67:0/3883145792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2140d20d0 0x7ff2140b54d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7ff20c00c7a0 tx=0x7ff20c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:30.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.077+0000 7ff210ff9640 1 -- 172.21.15.67:0/3883145792 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20c018070 con 0x7ff2140d20d0 2024-03-24T00:39:30.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.077+0000 7ff210ff9640 1 -- 172.21.15.67:0/3883145792 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff20c00a640 con 0x7ff2140d20d0 2024-03-24T00:39:30.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.077+0000 7ff21a710640 1 -- 172.21.15.67:0/3883145792 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2140c6cd0 con 0x7ff2140d20d0 2024-03-24T00:39:30.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.077+0000 7ff210ff9640 1 -- 172.21.15.67:0/3883145792 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20c00aab0 con 0x7ff2140d20d0 2024-03-24T00:39:30.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.077+0000 7ff21a710640 1 -- 172.21.15.67:0/3883145792 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2140c7290 con 0x7ff2140d20d0 2024-03-24T00:39:30.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.078+0000 7ff21a710640 1 -- 172.21.15.67:0/3883145792 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff21404f400 con 0x7ff2140d20d0 2024-03-24T00:39:30.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.078+0000 7ff210ff9640 1 -- 172.21.15.67:0/3883145792 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff20c01b070 con 0x7ff2140d20d0 2024-03-24T00:39:30.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.078+0000 7ff210ff9640 1 --2- 172.21.15.67:0/3883145792 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff20006bc00 0x7ff20006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:30.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.078+0000 7ff210ff9640 1 -- 172.21.15.67:0/3883145792 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7ff20c08a210 con 0x7ff2140d20d0 2024-03-24T00:39:30.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.079+0000 7ff213fff640 1 --2- 172.21.15.67:0/3883145792 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff20006bc00 0x7ff20006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:30.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.079+0000 7ff213fff640 1 --2- 172.21.15.67:0/3883145792 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff20006bc00 0x7ff20006e0c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff204036000 tx=0x7ff204008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:30.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.083+0000 7ff210ff9640 1 -- 172.21.15.67:0/3883145792 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff20c058400 con 0x7ff2140d20d0 2024-03-24T00:39:30.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.263+0000 7ff21a710640 1 -- 172.21.15.67:0/3883145792 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7ff2140d1d30 con 0x7ff2140d20d0 2024-03-24T00:39:30.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.264+0000 7ff210ff9640 1 -- 172.21.15.67:0/3883145792 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7ff20c05c4a0 con 0x7ff2140d20d0 2024-03-24T00:39:30.264 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-24T00:39:30.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.266+0000 7ff21a710640 1 -- 172.21.15.67:0/3883145792 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff20006bc00 msgr2=0x7ff20006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:30.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.266+0000 7ff21a710640 1 --2- 172.21.15.67:0/3883145792 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff20006bc00 0x7ff20006e0c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff204036000 tx=0x7ff204008030 comp rx=0 tx=0).stop 2024-03-24T00:39:30.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.266+0000 7ff21a710640 1 -- 172.21.15.67:0/3883145792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2140d20d0 msgr2=0x7ff2140b54d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:30.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.266+0000 7ff21a710640 1 --2- 172.21.15.67:0/3883145792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2140d20d0 0x7ff2140b54d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7ff20c00c7a0 tx=0x7ff20c00cc60 comp rx=0 tx=0).stop 2024-03-24T00:39:30.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.266+0000 7ff21a710640 1 -- 172.21.15.67:0/3883145792 shutdown_connections 2024-03-24T00:39:30.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.266+0000 7ff21a710640 1 --2- 172.21.15.67:0/3883145792 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff20006bc00 0x7ff20006e0c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:30.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.266+0000 7ff21a710640 1 --2- 172.21.15.67:0/3883145792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2140d20d0 0x7ff2140b54d0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:30.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.266+0000 7ff21a710640 1 --2- 172.21.15.67:0/3883145792 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff2140b2440 0x7ff2141519c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:30.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.266+0000 7ff21a710640 1 --2- 172.21.15.67:0/3883145792 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2140b0f10 0x7ff214151100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:30.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.266+0000 7ff21a710640 1 -- 172.21.15.67:0/3883145792 >> 172.21.15.67:0/3883145792 conn(0x7ff21408b1b0 msgr2=0x7ff2141621c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:30.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.266+0000 7ff21a710640 1 -- 172.21.15.67:0/3883145792 shutdown_connections 2024-03-24T00:39:30.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.266+0000 7ff21a710640 1 -- 172.21.15.67:0/3883145792 wait complete. 2024-03-24T00:39:30.276 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:39:30.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.377+0000 7f2038dab640 1 Processor -- start 2024-03-24T00:39:30.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.377+0000 7f2038dab640 1 -- start start 2024-03-24T00:39:30.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.378+0000 7f2038dab640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f203415dec0 0x7f20340ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:30.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.378+0000 7f2038dab640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f20340d0ef0 0x7f2034162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:30.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.378+0000 7f2038dab640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20340cf640 con 0x7f203415e2a0 2024-03-24T00:39:30.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.378+0000 7f2038dab640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20340cee30 con 0x7f20340d0ef0 2024-03-24T00:39:30.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.378+0000 7f2038dab640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20340d2120 con 0x7f203415dec0 2024-03-24T00:39:30.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.378+0000 7f2032d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f20340d0ef0 0x7f2034162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:30.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.378+0000 7f203259c640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f203415dec0 0x7f20340ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:30.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.378+0000 7f2032d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f20340d0ef0 0x7f2034162630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34068/0 (socket says 172.21.15.67:34068) 2024-03-24T00:39:30.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.378+0000 7f2032d9d640 1 -- 172.21.15.67:0/2924113414 learned_addr learned my addr 172.21.15.67:0/2924113414 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:30.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.378+0000 7f2031d9b640 1 --1- 172.21.15.67:0/2924113414 >> v1:172.21.15.72:6789/0 conn(0x7f203415e2a0 0x7f20340d07e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:45772/0 (socket says 172.21.15.67:45772) 2024-03-24T00:39:30.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.379+0000 7f203259c640 1 -- 172.21.15.67:0/2924113414 >> v2:172.21.15.67:3300/0 conn(0x7f20340d0ef0 msgr2=0x7f2034162630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:30.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.379+0000 7f203259c640 1 --2- 172.21.15.67:0/2924113414 >> v2:172.21.15.67:3300/0 conn(0x7f20340d0ef0 0x7f2034162630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:30.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.379+0000 7f203259c640 1 -- 172.21.15.67:0/2924113414 >> v1:172.21.15.72:6789/0 conn(0x7f203415e2a0 legacy=0x7f20340d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:30.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.379+0000 7f203259c640 1 -- 172.21.15.67:0/2924113414 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20340cf080 con 0x7f203415dec0 2024-03-24T00:39:30.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.379+0000 7f203159a640 1 -- 172.21.15.67:0/2924113414 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2538789525 0 0) 0x7f201c001110 con 0x7f203415e2a0 2024-03-24T00:39:30.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.379+0000 7f203259c640 1 --2- 172.21.15.67:0/2924113414 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f203415dec0 0x7f20340ce630 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f202c009d90 tx=0x7f202c02be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6fd0f239be5b2c0f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:30.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.380+0000 7f203159a640 1 -- 172.21.15.67:0/2924113414 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202c039070 con 0x7f203415dec0 2024-03-24T00:39:30.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.380+0000 7f203159a640 1 -- 172.21.15.67:0/2924113414 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f202c02cbc0 con 0x7f203415dec0 2024-03-24T00:39:30.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.380+0000 7f203159a640 1 -- 172.21.15.67:0/2924113414 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202c034440 con 0x7f203415dec0 2024-03-24T00:39:30.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.380+0000 7f2038dab640 1 -- 172.21.15.67:0/2924113414 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f203415dec0 msgr2=0x7f20340ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:30.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.380+0000 7f2038dab640 1 --2- 172.21.15.67:0/2924113414 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f203415dec0 0x7f20340ce630 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f202c009d90 tx=0x7f202c02be10 comp rx=0 tx=0).stop 2024-03-24T00:39:30.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.380+0000 7f2038dab640 1 -- 172.21.15.67:0/2924113414 shutdown_connections 2024-03-24T00:39:30.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.380+0000 7f2038dab640 1 --2- 172.21.15.67:0/2924113414 >> v2:172.21.15.67:3300/0 conn(0x7f20340d0ef0 0x7f2034162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:30.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.380+0000 7f2038dab640 1 --2- 172.21.15.67:0/2924113414 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f203415dec0 0x7f20340ce630 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:30.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.380+0000 7f2038dab640 1 -- 172.21.15.67:0/2924113414 >> 172.21.15.67:0/2924113414 conn(0x7f203406c810 msgr2=0x7f203406cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:30.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.380+0000 7f2038dab640 1 -- 172.21.15.67:0/2924113414 shutdown_connections 2024-03-24T00:39:30.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.380+0000 7f2038dab640 1 -- 172.21.15.67:0/2924113414 wait complete. 2024-03-24T00:39:30.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.382+0000 7f2038dab640 1 Processor -- start 2024-03-24T00:39:30.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.382+0000 7f2038dab640 1 -- start start 2024-03-24T00:39:30.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.382+0000 7f2038dab640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20340d0ef0 0x7f2034157ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:30.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.382+0000 7f2038dab640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f203415dec0 0x7f20340caea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:30.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.382+0000 7f2038dab640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f203415e2a0 0x7f20340b8050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:30.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.382+0000 7f2038dab640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20340d2730 con 0x7f20340d0ef0 2024-03-24T00:39:30.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.382+0000 7f2038dab640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20340d25b0 con 0x7f203415e2a0 2024-03-24T00:39:30.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.382+0000 7f2038dab640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20340d28b0 con 0x7f203415dec0 2024-03-24T00:39:30.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.382+0000 7f203259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20340d0ef0 0x7f2034157ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:30.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.382+0000 7f203259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20340d0ef0 0x7f2034157ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34082/0 (socket says 172.21.15.67:34082) 2024-03-24T00:39:30.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.382+0000 7f203259c640 1 -- 172.21.15.67:0/543366457 learned_addr learned my addr 172.21.15.67:0/543366457 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:30.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.382+0000 7f2031d9b640 1 --2- 172.21.15.67:0/543366457 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f203415dec0 0x7f20340caea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:30.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.382+0000 7f2032d9d640 1 --2- 172.21.15.67:0/543366457 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f203415e2a0 0x7f20340b8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:30.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.383+0000 7f203259c640 1 -- 172.21.15.67:0/543366457 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f203415dec0 msgr2=0x7f20340caea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:30.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.383+0000 7f203259c640 1 --2- 172.21.15.67:0/543366457 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f203415dec0 0x7f20340caea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:30.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.383+0000 7f203259c640 1 -- 172.21.15.67:0/543366457 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f203415e2a0 msgr2=0x7f20340b8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:30.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.383+0000 7f203259c640 1 --2- 172.21.15.67:0/543366457 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f203415e2a0 0x7f20340b8050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:30.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.383+0000 7f203259c640 1 -- 172.21.15.67:0/543366457 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2034158020 con 0x7f20340d0ef0 2024-03-24T00:39:30.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.383+0000 7f203259c640 1 --2- 172.21.15.67:0/543366457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20340d0ef0 0x7f2034157ae0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f202c009d90 tx=0x7f202c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:30.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.383+0000 7f202affd640 1 -- 172.21.15.67:0/543366457 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202c039050 con 0x7f20340d0ef0 2024-03-24T00:39:30.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.383+0000 7f202affd640 1 -- 172.21.15.67:0/543366457 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f202c045070 con 0x7f20340d0ef0 2024-03-24T00:39:30.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.383+0000 7f202affd640 1 -- 172.21.15.67:0/543366457 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202c03e850 con 0x7f20340d0ef0 2024-03-24T00:39:30.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.383+0000 7f2038dab640 1 -- 172.21.15.67:0/543366457 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20340cb3e0 con 0x7f20340d0ef0 2024-03-24T00:39:30.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.383+0000 7f2038dab640 1 -- 172.21.15.67:0/543366457 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2034158650 con 0x7f20340d0ef0 2024-03-24T00:39:30.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.384+0000 7f2038dab640 1 -- 172.21.15.67:0/543366457 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f203404f400 con 0x7f20340d0ef0 2024-03-24T00:39:30.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.385+0000 7f202affd640 1 -- 172.21.15.67:0/543366457 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f202c03e9f0 con 0x7f20340d0ef0 2024-03-24T00:39:30.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.385+0000 7f202affd640 1 --2- 172.21.15.67:0/543366457 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f201406bbd0 0x7f201406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:30.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.385+0000 7f202affd640 1 -- 172.21.15.67:0/543366457 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f202c0adb90 con 0x7f20340d0ef0 2024-03-24T00:39:30.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.385+0000 7f2031d9b640 1 --2- 172.21.15.67:0/543366457 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f201406bbd0 0x7f201406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:30.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.386+0000 7f2031d9b640 1 --2- 172.21.15.67:0/543366457 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f201406bbd0 0x7f201406e090 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f20341e7b50 tx=0x7f201c007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:30.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.389+0000 7f202affd640 1 -- 172.21.15.67:0/543366457 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f202c07be10 con 0x7f20340d0ef0 2024-03-24T00:39:30.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.572+0000 7f2038dab640 1 -- 172.21.15.67:0/543366457 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f20341e7b80 con 0x7f20340d0ef0 2024-03-24T00:39:30.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.573+0000 7f202affd640 1 -- 172.21.15.67:0/543366457 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18454 (secure 0 0 0) 0x7f202c07feb0 con 0x7f20340d0ef0 2024-03-24T00:39:30.573 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:39:30.573 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":25,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:38:58.797179+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:38:50.077085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:38:51.095454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T00:38:54.807117+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:39:30.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.575+0000 7f2038dab640 1 -- 172.21.15.67:0/543366457 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f201406bbd0 msgr2=0x7f201406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:30.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.575+0000 7f2038dab640 1 --2- 172.21.15.67:0/543366457 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f201406bbd0 0x7f201406e090 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f20341e7b50 tx=0x7f201c007de0 comp rx=0 tx=0).stop 2024-03-24T00:39:30.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.575+0000 7f2038dab640 1 -- 172.21.15.67:0/543366457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20340d0ef0 msgr2=0x7f2034157ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:30.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.575+0000 7f2038dab640 1 --2- 172.21.15.67:0/543366457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20340d0ef0 0x7f2034157ae0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f202c009d90 tx=0x7f202c004a40 comp rx=0 tx=0).stop 2024-03-24T00:39:30.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.575+0000 7f2038dab640 1 -- 172.21.15.67:0/543366457 shutdown_connections 2024-03-24T00:39:30.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.575+0000 7f2038dab640 1 --2- 172.21.15.67:0/543366457 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f201406bbd0 0x7f201406e090 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:30.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.575+0000 7f2038dab640 1 --2- 172.21.15.67:0/543366457 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f203415e2a0 0x7f20340b8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:30.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.575+0000 7f2038dab640 1 --2- 172.21.15.67:0/543366457 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f203415dec0 0x7f20340caea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:30.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.575+0000 7f2038dab640 1 --2- 172.21.15.67:0/543366457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20340d0ef0 0x7f2034157ae0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:30.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.575+0000 7f2038dab640 1 -- 172.21.15.67:0/543366457 >> 172.21.15.67:0/543366457 conn(0x7f203406c810 msgr2=0x7f20340b2b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:30.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.575+0000 7f2038dab640 1 -- 172.21.15.67:0/543366457 shutdown_connections 2024-03-24T00:39:30.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.575+0000 7f2038dab640 1 -- 172.21.15.67:0/543366457 wait complete. 2024-03-24T00:39:30.586 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:38:39.364950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:38:50.077085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:38:51.095454+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T00:38:54.807117+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T00:39:30.586 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:39:30.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.690+0000 7f4671ac6640 1 Processor -- start 2024-03-24T00:39:30.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.690+0000 7f4671ac6640 1 -- start start 2024-03-24T00:39:30.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.691+0000 7f4671ac6640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f466c15e130 0x7f466c0ce610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:30.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.691+0000 7f4671ac6640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f466c0d32c0 0x7f466c0d0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:30.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.691+0000 7f4671ac6640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f466c0cf570 con 0x7f466c0d0640 2024-03-24T00:39:30.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.691+0000 7f4671ac6640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f466c0ceb50 con 0x7f466c15e130 2024-03-24T00:39:30.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.691+0000 7f4671ac6640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f466c0d0fe0 con 0x7f466c0d32c0 2024-03-24T00:39:30.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.691+0000 7f466b7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f466c15e130 0x7f466c0ce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:30.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.691+0000 7f466b7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f466c15e130 0x7f466c0ce610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34084/0 (socket says 172.21.15.67:34084) 2024-03-24T00:39:30.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.691+0000 7f466b7fe640 1 -- 172.21.15.67:0/161765728 learned_addr learned my addr 172.21.15.67:0/161765728 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:30.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.691+0000 7f466bfff640 1 --2- 172.21.15.67:0/161765728 >> v2:172.21.15.72:3300/0 conn(0x7f466c0d32c0 0x7f466c0d0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:30.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.691+0000 7f466b7fe640 1 -- 172.21.15.67:0/161765728 >> v2:172.21.15.72:3300/0 conn(0x7f466c0d32c0 msgr2=0x7f466c0d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:30.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.691+0000 7f466b7fe640 1 --2- 172.21.15.67:0/161765728 >> v2:172.21.15.72:3300/0 conn(0x7f466c0d32c0 0x7f466c0d0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:30.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.691+0000 7f466a7fc640 1 -- 172.21.15.67:0/161765728 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3465062889 0 0) 0x7f466c0cf570 con 0x7f466c0d0640 2024-03-24T00:39:30.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.691+0000 7f466b7fe640 1 -- 172.21.15.67:0/161765728 >> v1:172.21.15.72:6789/0 conn(0x7f466c0d0640 legacy=0x7f466c0b57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:30.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.691+0000 7f466b7fe640 1 -- 172.21.15.67:0/161765728 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f466c0cecf0 con 0x7f466c15e130 2024-03-24T00:39:30.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.692+0000 7f466b7fe640 1 --2- 172.21.15.67:0/161765728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f466c15e130 0x7f466c0ce610 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f466c0b2bf0 tx=0x7f466002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85af794e6fce90dd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:30.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.692+0000 7f466a7fc640 1 -- 172.21.15.67:0/161765728 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4660035020 con 0x7f466c15e130 2024-03-24T00:39:30.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.692+0000 7f466a7fc640 1 -- 172.21.15.67:0/161765728 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4660031870 con 0x7f466c15e130 2024-03-24T00:39:30.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.692+0000 7f466a7fc640 1 -- 172.21.15.67:0/161765728 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4660031c40 con 0x7f466c15e130 2024-03-24T00:39:30.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.693+0000 7f4671ac6640 1 -- 172.21.15.67:0/161765728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f466c15e130 msgr2=0x7f466c0ce610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:30.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.693+0000 7f4671ac6640 1 --2- 172.21.15.67:0/161765728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f466c15e130 0x7f466c0ce610 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f466c0b2bf0 tx=0x7f466002b910 comp rx=0 tx=0).stop 2024-03-24T00:39:30.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.693+0000 7f4671ac6640 1 -- 172.21.15.67:0/161765728 shutdown_connections 2024-03-24T00:39:30.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.693+0000 7f4671ac6640 1 --2- 172.21.15.67:0/161765728 >> v2:172.21.15.72:3300/0 conn(0x7f466c0d32c0 0x7f466c0d0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:30.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.693+0000 7f4671ac6640 1 --2- 172.21.15.67:0/161765728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f466c15e130 0x7f466c0ce610 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:30.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.693+0000 7f4671ac6640 1 -- 172.21.15.67:0/161765728 >> 172.21.15.67:0/161765728 conn(0x7f466c06c8a0 msgr2=0x7f466c06ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:30.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.693+0000 7f4671ac6640 1 -- 172.21.15.67:0/161765728 shutdown_connections 2024-03-24T00:39:30.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.693+0000 7f4671ac6640 1 -- 172.21.15.67:0/161765728 wait complete. 2024-03-24T00:39:30.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.694+0000 7f4671ac6640 1 Processor -- start 2024-03-24T00:39:30.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.694+0000 7f4671ac6640 1 -- start start 2024-03-24T00:39:30.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.695+0000 7f4671ac6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f466c0d0640 0x7f466c14f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:30.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.695+0000 7f4671ac6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f466c0d32c0 0x7f466c0cc860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:30.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.695+0000 7f4671ac6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f466c15e130 0x7f466c160700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:30.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.695+0000 7f4671ac6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466c0cf290 con 0x7f466c0d0640 2024-03-24T00:39:30.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.695+0000 7f4671ac6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466c0cefb0 con 0x7f466c0d32c0 2024-03-24T00:39:30.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.695+0000 7f4671ac6640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f466c15e510 con 0x7f466c15e130 2024-03-24T00:39:30.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.695+0000 7f466b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f466c0d0640 0x7f466c14f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:30.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.695+0000 7f466b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f466c0d0640 0x7f466c14f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34092/0 (socket says 172.21.15.67:34092) 2024-03-24T00:39:30.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.695+0000 7f466b7fe640 1 -- 172.21.15.67:0/2727194260 learned_addr learned my addr 172.21.15.67:0/2727194260 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:30.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.695+0000 7f466bfff640 1 --2- 172.21.15.67:0/2727194260 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f466c15e130 0x7f466c160700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:30.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.695+0000 7f466affd640 1 --2- 172.21.15.67:0/2727194260 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f466c0d32c0 0x7f466c0cc860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:30.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.695+0000 7f466b7fe640 1 -- 172.21.15.67:0/2727194260 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f466c15e130 msgr2=0x7f466c160700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:30.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.695+0000 7f466b7fe640 1 --2- 172.21.15.67:0/2727194260 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f466c15e130 0x7f466c160700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:30.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.695+0000 7f466b7fe640 1 -- 172.21.15.67:0/2727194260 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f466c0d32c0 msgr2=0x7f466c0cc860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:30.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.695+0000 7f466b7fe640 1 --2- 172.21.15.67:0/2727194260 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f466c0d32c0 0x7f466c0cc860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:30.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.695+0000 7f466b7fe640 1 -- 172.21.15.67:0/2727194260 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f466c1f3fc0 con 0x7f466c0d0640 2024-03-24T00:39:30.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.695+0000 7f466b7fe640 1 --2- 172.21.15.67:0/2727194260 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f466c0d0640 0x7f466c14f0e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f4660000c00 tx=0x7f46600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:30.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.696+0000 7f4653fff640 1 -- 172.21.15.67:0/2727194260 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4660035040 con 0x7f466c0d0640 2024-03-24T00:39:30.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.696+0000 7f4653fff640 1 -- 172.21.15.67:0/2727194260 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4660030d00 con 0x7f466c0d0640 2024-03-24T00:39:30.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.696+0000 7f4671ac6640 1 -- 172.21.15.67:0/2727194260 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f466c1f4260 con 0x7f466c0d0640 2024-03-24T00:39:30.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.696+0000 7f4653fff640 1 -- 172.21.15.67:0/2727194260 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46600456a0 con 0x7f466c0d0640 2024-03-24T00:39:30.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.696+0000 7f4671ac6640 1 -- 172.21.15.67:0/2727194260 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f466c1f4d40 con 0x7f466c0d0640 2024-03-24T00:39:30.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.697+0000 7f4671ac6640 1 -- 172.21.15.67:0/2727194260 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f466c15dc70 con 0x7f466c0d0640 2024-03-24T00:39:30.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.697+0000 7f4653fff640 1 -- 172.21.15.67:0/2727194260 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4660038020 con 0x7f466c0d0640 2024-03-24T00:39:30.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.697+0000 7f4653fff640 1 --2- 172.21.15.67:0/2727194260 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f465806bbd0 0x7f465806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:30.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.697+0000 7f4653fff640 1 -- 172.21.15.67:0/2727194260 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f46600acfc0 con 0x7f466c0d0640 2024-03-24T00:39:30.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.697+0000 7f466affd640 1 --2- 172.21.15.67:0/2727194260 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f465806bbd0 0x7f465806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:30.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.698+0000 7f466affd640 1 --2- 172.21.15.67:0/2727194260 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f465806bbd0 0x7f465806e090 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f466c1f3ef0 tx=0x7f465c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:30.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.702+0000 7f4653fff640 1 -- 172.21.15.67:0/2727194260 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f466007b120 con 0x7f466c0d0640 2024-03-24T00:39:30.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.885+0000 7f4671ac6640 1 -- 172.21.15.67:0/2727194260 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f466c04f400 con 0x7f466c0d0640 2024-03-24T00:39:30.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.885+0000 7f4653fff640 1 -- 172.21.15.67:0/2727194260 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7f466007f1c0 con 0x7f466c0d0640 2024-03-24T00:39:30.885 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-24T00:39:30.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.888+0000 7f4671ac6640 1 -- 172.21.15.67:0/2727194260 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f465806bbd0 msgr2=0x7f465806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:30.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.888+0000 7f4671ac6640 1 --2- 172.21.15.67:0/2727194260 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f465806bbd0 0x7f465806e090 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f466c1f3ef0 tx=0x7f465c009040 comp rx=0 tx=0).stop 2024-03-24T00:39:30.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.888+0000 7f4671ac6640 1 -- 172.21.15.67:0/2727194260 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f466c0d0640 msgr2=0x7f466c14f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:30.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.888+0000 7f4671ac6640 1 --2- 172.21.15.67:0/2727194260 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f466c0d0640 0x7f466c14f0e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f4660000c00 tx=0x7f46600046c0 comp rx=0 tx=0).stop 2024-03-24T00:39:30.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.888+0000 7f4671ac6640 1 -- 172.21.15.67:0/2727194260 shutdown_connections 2024-03-24T00:39:30.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.888+0000 7f4671ac6640 1 --2- 172.21.15.67:0/2727194260 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f465806bbd0 0x7f465806e090 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:30.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.888+0000 7f4671ac6640 1 --2- 172.21.15.67:0/2727194260 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f466c15e130 0x7f466c160700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:30.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.888+0000 7f4671ac6640 1 --2- 172.21.15.67:0/2727194260 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f466c0d32c0 0x7f466c0cc860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:30.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.888+0000 7f4671ac6640 1 --2- 172.21.15.67:0/2727194260 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f466c0d0640 0x7f466c14f0e0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:30.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.888+0000 7f4671ac6640 1 -- 172.21.15.67:0/2727194260 >> 172.21.15.67:0/2727194260 conn(0x7f466c06c8a0 msgr2=0x7f466c0b6f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:30.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.888+0000 7f4671ac6640 1 -- 172.21.15.67:0/2727194260 shutdown_connections 2024-03-24T00:39:30.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:30.888+0000 7f4671ac6640 1 -- 172.21.15.67:0/2727194260 wait complete. 2024-03-24T00:39:30.898 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:39:31.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.002+0000 7f5327019640 1 Processor -- start 2024-03-24T00:39:31.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.002+0000 7f5327019640 1 -- start start 2024-03-24T00:39:31.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.003+0000 7f5327019640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f53200afae0 0x7f53200ad8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:31.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.003+0000 7f5327019640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f532006b750 0x7f53200b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:31.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.003+0000 7f5327019640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f532015c220 con 0x7f53200b0c10 2024-03-24T00:39:31.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.003+0000 7f5327019640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53200cc950 con 0x7f532006b750 2024-03-24T00:39:31.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.003+0000 7f5327019640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53200ccc30 con 0x7f53200afae0 2024-03-24T00:39:31.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.003+0000 7f53255b6640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f532006b750 0x7f53200b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:31.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.003+0000 7f531ffff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f53200b0c10 0x7f532006b040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46964/0 (socket says 172.21.15.67:46964) 2024-03-24T00:39:31.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.003+0000 7f531ffff640 1 -- 172.21.15.67:0/2694230276 learned_addr learned my addr 172.21.15.67:0/2694230276 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:31.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.003+0000 7f5324db5640 1 --2- 172.21.15.67:0/2694230276 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f53200afae0 0x7f53200ad8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:31.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.003+0000 7f53255b6640 1 -- 172.21.15.67:0/2694230276 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f53200afae0 msgr2=0x7f53200ad8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:31.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.003+0000 7f53255b6640 1 --2- 172.21.15.67:0/2694230276 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f53200afae0 0x7f53200ad8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:31.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.003+0000 7f53255b6640 1 -- 172.21.15.67:0/2694230276 >> v1:172.21.15.67:6789/0 conn(0x7f53200b0c10 legacy=0x7f532006b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:31.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.003+0000 7f53255b6640 1 -- 172.21.15.67:0/2694230276 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53200ccf10 con 0x7f532006b750 2024-03-24T00:39:31.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.003+0000 7f531f7fe640 1 -- 172.21.15.67:0/2694230276 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3900817373 0 0) 0x7f532015c220 con 0x7f53200b0c10 2024-03-24T00:39:31.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.004+0000 7f53255b6640 1 --2- 172.21.15.67:0/2694230276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f532006b750 0x7f53200b1070 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f5318009d90 tx=0x7f531802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b75f867ae38186b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:31.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.004+0000 7f531f7fe640 1 -- 172.21.15.67:0/2694230276 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5318039070 con 0x7f532006b750 2024-03-24T00:39:31.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.004+0000 7f531f7fe640 1 -- 172.21.15.67:0/2694230276 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f531802cc50 con 0x7f532006b750 2024-03-24T00:39:31.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.004+0000 7f531f7fe640 1 -- 172.21.15.67:0/2694230276 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5318034440 con 0x7f532006b750 2024-03-24T00:39:31.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.005+0000 7f5327019640 1 -- 172.21.15.67:0/2694230276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f532006b750 msgr2=0x7f53200b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:31.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.005+0000 7f5327019640 1 --2- 172.21.15.67:0/2694230276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f532006b750 0x7f53200b1070 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f5318009d90 tx=0x7f531802bea0 comp rx=0 tx=0).stop 2024-03-24T00:39:31.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.005+0000 7f5327019640 1 -- 172.21.15.67:0/2694230276 shutdown_connections 2024-03-24T00:39:31.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.005+0000 7f5327019640 1 --2- 172.21.15.67:0/2694230276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f532006b750 0x7f53200b1070 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:31.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.005+0000 7f5327019640 1 --2- 172.21.15.67:0/2694230276 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f53200afae0 0x7f53200ad8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:31.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.005+0000 7f5327019640 1 -- 172.21.15.67:0/2694230276 >> 172.21.15.67:0/2694230276 conn(0x7f5320099a30 msgr2=0x7f5320099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:31.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.005+0000 7f5327019640 1 -- 172.21.15.67:0/2694230276 shutdown_connections 2024-03-24T00:39:31.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.005+0000 7f5327019640 1 -- 172.21.15.67:0/2694230276 wait complete. 2024-03-24T00:39:31.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.006+0000 7f5327019640 1 Processor -- start 2024-03-24T00:39:31.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.006+0000 7f5327019640 1 -- start start 2024-03-24T00:39:31.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.007+0000 7f5327019640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f532006b750 0x7f53200c65b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:31.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.007+0000 7f5327019640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53200afae0 0x7f53200b26a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:31.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.007+0000 7f5327019640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53200b0c10 0x7f53200b2be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:31.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.007+0000 7f5327019640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53200cc600 con 0x7f53200b0c10 2024-03-24T00:39:31.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.007+0000 7f5327019640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53200cd4d0 con 0x7f53200afae0 2024-03-24T00:39:31.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.007+0000 7f5327019640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53200ade00 con 0x7f532006b750 2024-03-24T00:39:31.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.007+0000 7f53255b6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53200b0c10 0x7f53200b2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:31.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.007+0000 7f53255b6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53200b0c10 0x7f53200b2be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34118/0 (socket says 172.21.15.67:34118) 2024-03-24T00:39:31.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.007+0000 7f53255b6640 1 -- 172.21.15.67:0/3168302981 learned_addr learned my addr 172.21.15.67:0/3168302981 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:31.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.007+0000 7f5324db5640 1 --2- 172.21.15.67:0/3168302981 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f532006b750 0x7f53200c65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:31.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.007+0000 7f531ffff640 1 --2- 172.21.15.67:0/3168302981 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53200afae0 0x7f53200b26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:31.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.007+0000 7f53255b6640 1 -- 172.21.15.67:0/3168302981 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f532006b750 msgr2=0x7f53200c65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:31.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.007+0000 7f53255b6640 1 --2- 172.21.15.67:0/3168302981 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f532006b750 0x7f53200c65b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:31.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.007+0000 7f53255b6640 1 -- 172.21.15.67:0/3168302981 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53200afae0 msgr2=0x7f53200b26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:31.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.007+0000 7f53255b6640 1 --2- 172.21.15.67:0/3168302981 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53200afae0 0x7f53200b26a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:31.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.007+0000 7f53255b6640 1 -- 172.21.15.67:0/3168302981 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53200b65b0 con 0x7f53200b0c10 2024-03-24T00:39:31.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.008+0000 7f53255b6640 1 --2- 172.21.15.67:0/3168302981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53200b0c10 0x7f53200b2be0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5318009d90 tx=0x7f5318004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:31.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.008+0000 7f531d7fa640 1 -- 172.21.15.67:0/3168302981 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5318039050 con 0x7f53200b0c10 2024-03-24T00:39:31.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.008+0000 7f531d7fa640 1 -- 172.21.15.67:0/3168302981 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5318045070 con 0x7f53200b0c10 2024-03-24T00:39:31.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.008+0000 7f5327019640 1 -- 172.21.15.67:0/3168302981 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53200b37e0 con 0x7f53200b0c10 2024-03-24T00:39:31.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.008+0000 7f531d7fa640 1 -- 172.21.15.67:0/3168302981 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5318009450 con 0x7f53200b0c10 2024-03-24T00:39:31.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.008+0000 7f5327019640 1 -- 172.21.15.67:0/3168302981 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53200b3da0 con 0x7f53200b0c10 2024-03-24T00:39:31.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.009+0000 7f5327019640 1 -- 172.21.15.67:0/3168302981 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f532004f400 con 0x7f53200b0c10 2024-03-24T00:39:31.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.009+0000 7f531d7fa640 1 -- 172.21.15.67:0/3168302981 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f531803e610 con 0x7f53200b0c10 2024-03-24T00:39:31.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.009+0000 7f531d7fa640 1 --2- 172.21.15.67:0/3168302981 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f530406bbd0 0x7f530406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:31.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.010+0000 7f531d7fa640 1 -- 172.21.15.67:0/3168302981 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f53180ad680 con 0x7f53200b0c10 2024-03-24T00:39:31.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.010+0000 7f5324db5640 1 --2- 172.21.15.67:0/3168302981 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f530406bbd0 0x7f530406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:31.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.010+0000 7f5324db5640 1 --2- 172.21.15.67:0/3168302981 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f530406bbd0 0x7f530406e090 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f53200b48c0 tx=0x7f5314009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:31.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.014+0000 7f531d7fa640 1 -- 172.21.15.67:0/3168302981 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f531807b870 con 0x7f53200b0c10 2024-03-24T00:39:31.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.194+0000 7f5327019640 1 -- 172.21.15.67:0/3168302981 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f532015efc0 con 0x7f53200b0c10 2024-03-24T00:39:31.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.194+0000 7f531d7fa640 1 -- 172.21.15.67:0/3168302981 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7f531807f910 con 0x7f53200b0c10 2024-03-24T00:39:31.195 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-24T00:39:31.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.196+0000 7f5327019640 1 -- 172.21.15.67:0/3168302981 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f530406bbd0 msgr2=0x7f530406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:31.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.196+0000 7f5327019640 1 --2- 172.21.15.67:0/3168302981 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f530406bbd0 0x7f530406e090 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f53200b48c0 tx=0x7f5314009190 comp rx=0 tx=0).stop 2024-03-24T00:39:31.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.196+0000 7f5327019640 1 -- 172.21.15.67:0/3168302981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53200b0c10 msgr2=0x7f53200b2be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:31.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.196+0000 7f5327019640 1 --2- 172.21.15.67:0/3168302981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53200b0c10 0x7f53200b2be0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5318009d90 tx=0x7f5318004a40 comp rx=0 tx=0).stop 2024-03-24T00:39:31.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.197+0000 7f5327019640 1 -- 172.21.15.67:0/3168302981 shutdown_connections 2024-03-24T00:39:31.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.197+0000 7f5327019640 1 --2- 172.21.15.67:0/3168302981 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f530406bbd0 0x7f530406e090 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:31.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.197+0000 7f5327019640 1 --2- 172.21.15.67:0/3168302981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53200b0c10 0x7f53200b2be0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:31.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.197+0000 7f5327019640 1 --2- 172.21.15.67:0/3168302981 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53200afae0 0x7f53200b26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:31.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.197+0000 7f5327019640 1 --2- 172.21.15.67:0/3168302981 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f532006b750 0x7f53200c65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:31.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.197+0000 7f5327019640 1 -- 172.21.15.67:0/3168302981 >> 172.21.15.67:0/3168302981 conn(0x7f5320099a30 msgr2=0x7f53200c73e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:31.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.197+0000 7f5327019640 1 -- 172.21.15.67:0/3168302981 shutdown_connections 2024-03-24T00:39:31.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.197+0000 7f5327019640 1 -- 172.21.15.67:0/3168302981 wait complete. 2024-03-24T00:39:31.207 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:39:31.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.311+0000 7f3987dbc640 1 Processor -- start 2024-03-24T00:39:31.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.311+0000 7f3987dbc640 1 -- start start 2024-03-24T00:39:31.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.312+0000 7f3987dbc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f39800d3ae0 0x7f39800b5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:31.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.312+0000 7f3987dbc640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39800d2250 con 0x7f398015e280 2024-03-24T00:39:31.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.312+0000 7f3987dbc640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39800d2530 con 0x7f398015dea0 2024-03-24T00:39:31.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.312+0000 7f3987dbc640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39800d2810 con 0x7f39800d3ae0 2024-03-24T00:39:31.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.312+0000 7f3986359640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f398015e280 0x7f39800b70f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46970/0 (socket says 172.21.15.67:46970) 2024-03-24T00:39:31.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.312+0000 7f3986359640 1 -- 172.21.15.67:0/637037779 learned_addr learned my addr 172.21.15.67:0/637037779 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:31.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.312+0000 7f3985357640 1 --2- 172.21.15.67:0/637037779 >> v2:172.21.15.72:3300/0 conn(0x7f39800d3ae0 0x7f39800b5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:31.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.313+0000 7f3984b56640 1 -- 172.21.15.67:0/637037779 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2579175575 0 0) 0x7f39800d2250 con 0x7f398015e280 2024-03-24T00:39:31.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.313+0000 7f3985357640 1 -- 172.21.15.67:0/637037779 >> v1:172.21.15.72:6789/0 conn(0x7f398015dea0 legacy=0x7f39800d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:31.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.313+0000 7f3985357640 1 -- 172.21.15.67:0/637037779 >> v1:172.21.15.67:6789/0 conn(0x7f398015e280 legacy=0x7f39800b70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:31.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.313+0000 7f3985357640 1 -- 172.21.15.67:0/637037779 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39800d2af0 con 0x7f39800d3ae0 2024-03-24T00:39:31.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.313+0000 7f3985357640 1 --2- 172.21.15.67:0/637037779 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39800d3ae0 0x7f39800b5a60 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3974009750 tx=0x7f397402b690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f05f04d71af533d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:31.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.314+0000 7f3984b56640 1 -- 172.21.15.67:0/637037779 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3974037070 con 0x7f39800d3ae0 2024-03-24T00:39:31.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.314+0000 7f3984b56640 1 -- 172.21.15.67:0/637037779 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3974030710 con 0x7f39800d3ae0 2024-03-24T00:39:31.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.314+0000 7f3984b56640 1 -- 172.21.15.67:0/637037779 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3974030ae0 con 0x7f39800d3ae0 2024-03-24T00:39:31.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.314+0000 7f3987dbc640 1 -- 172.21.15.67:0/637037779 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39800d3ae0 msgr2=0x7f39800b5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:31.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.314+0000 7f3987dbc640 1 --2- 172.21.15.67:0/637037779 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39800d3ae0 0x7f39800b5a60 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3974009750 tx=0x7f397402b690 comp rx=0 tx=0).stop 2024-03-24T00:39:31.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.314+0000 7f3987dbc640 1 -- 172.21.15.67:0/637037779 shutdown_connections 2024-03-24T00:39:31.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.314+0000 7f3987dbc640 1 --2- 172.21.15.67:0/637037779 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39800d3ae0 0x7f39800b5a60 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:31.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.314+0000 7f3987dbc640 1 -- 172.21.15.67:0/637037779 >> 172.21.15.67:0/637037779 conn(0x7f3980072da0 msgr2=0x7f39800d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:31.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.314+0000 7f3987dbc640 1 -- 172.21.15.67:0/637037779 shutdown_connections 2024-03-24T00:39:31.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.314+0000 7f3987dbc640 1 -- 172.21.15.67:0/637037779 wait complete. 2024-03-24T00:39:31.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.315+0000 7f3987dbc640 1 Processor -- start 2024-03-24T00:39:31.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.315+0000 7f3987dbc640 1 -- start start 2024-03-24T00:39:31.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.316+0000 7f3987dbc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39800d3ae0 0x7f39801f33c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:31.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.316+0000 7f3987dbc640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f398015dea0 0x7f39800cbac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:31.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.316+0000 7f3987dbc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f398015e280 0x7f39800ccc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:31.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.316+0000 7f3987dbc640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39800b6370 con 0x7f398015e280 2024-03-24T00:39:31.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.316+0000 7f3987dbc640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39800b61f0 con 0x7f39800d3ae0 2024-03-24T00:39:31.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.316+0000 7f3987dbc640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39800b7800 con 0x7f398015dea0 2024-03-24T00:39:31.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.316+0000 7f3986359640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f398015e280 0x7f39800ccc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:31.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.316+0000 7f3986359640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f398015e280 0x7f39800ccc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34120/0 (socket says 172.21.15.67:34120) 2024-03-24T00:39:31.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.316+0000 7f3986359640 1 -- 172.21.15.67:0/2066880249 learned_addr learned my addr 172.21.15.67:0/2066880249 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:31.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.316+0000 7f3985b58640 1 --2- 172.21.15.67:0/2066880249 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39800d3ae0 0x7f39801f33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:31.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.316+0000 7f3985357640 1 --2- 172.21.15.67:0/2066880249 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f398015dea0 0x7f39800cbac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:31.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.317+0000 7f3986359640 1 -- 172.21.15.67:0/2066880249 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f398015dea0 msgr2=0x7f39800cbac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:31.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.317+0000 7f3986359640 1 --2- 172.21.15.67:0/2066880249 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f398015dea0 0x7f39800cbac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:31.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.317+0000 7f3986359640 1 -- 172.21.15.67:0/2066880249 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39800d3ae0 msgr2=0x7f39801f33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:31.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.317+0000 7f3986359640 1 --2- 172.21.15.67:0/2066880249 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39800d3ae0 0x7f39801f33c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:31.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.317+0000 7f3986359640 1 -- 172.21.15.67:0/2066880249 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f398014b0f0 con 0x7f398015e280 2024-03-24T00:39:31.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.317+0000 7f3986359640 1 --2- 172.21.15.67:0/2066880249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f398015e280 0x7f39800ccc80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f397000c780 tx=0x7f397000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:31.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.317+0000 7f397e7fc640 1 -- 172.21.15.67:0/2066880249 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3970018070 con 0x7f398015e280 2024-03-24T00:39:31.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.317+0000 7f397e7fc640 1 -- 172.21.15.67:0/2066880249 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3970004d60 con 0x7f398015e280 2024-03-24T00:39:31.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.317+0000 7f3987dbc640 1 -- 172.21.15.67:0/2066880249 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39800cc4f0 con 0x7f398015e280 2024-03-24T00:39:31.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.317+0000 7f397e7fc640 1 -- 172.21.15.67:0/2066880249 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f397000a880 con 0x7f398015e280 2024-03-24T00:39:31.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.317+0000 7f3987dbc640 1 -- 172.21.15.67:0/2066880249 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3980158f00 con 0x7f398015e280 2024-03-24T00:39:31.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.318+0000 7f3987dbc640 1 -- 172.21.15.67:0/2066880249 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f398004f400 con 0x7f398015e280 2024-03-24T00:39:31.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.318+0000 7f397e7fc640 1 -- 172.21.15.67:0/2066880249 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f397001b070 con 0x7f398015e280 2024-03-24T00:39:31.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.319+0000 7f397e7fc640 1 --2- 172.21.15.67:0/2066880249 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f396c06bbd0 0x7f396c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:31.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.319+0000 7f397e7fc640 1 -- 172.21.15.67:0/2066880249 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f397008a0e0 con 0x7f398015e280 2024-03-24T00:39:31.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.319+0000 7f3985b58640 1 --2- 172.21.15.67:0/2066880249 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f396c06bbd0 0x7f396c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:31.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.319+0000 7f3985b58640 1 --2- 172.21.15.67:0/2066880249 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f396c06bbd0 0x7f396c06e090 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3978000f60 tx=0x7f3978009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:31.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.323+0000 7f397e7fc640 1 -- 172.21.15.67:0/2066880249 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f39700582d0 con 0x7f398015e280 2024-03-24T00:39:31.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.504+0000 7f3987dbc640 1 -- 172.21.15.67:0/2066880249 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3980160cb0 con 0x7f398015e280 2024-03-24T00:39:31.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.505+0000 7f397e7fc640 1 -- 172.21.15.67:0/2066880249 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7f397005c370 con 0x7f398015e280 2024-03-24T00:39:31.504 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-03-24T00:39:31.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.506+0000 7f3987dbc640 1 -- 172.21.15.67:0/2066880249 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f396c06bbd0 msgr2=0x7f396c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:31.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.506+0000 7f3987dbc640 1 --2- 172.21.15.67:0/2066880249 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f396c06bbd0 0x7f396c06e090 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3978000f60 tx=0x7f3978009040 comp rx=0 tx=0).stop 2024-03-24T00:39:31.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.506+0000 7f3987dbc640 1 -- 172.21.15.67:0/2066880249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f398015e280 msgr2=0x7f39800ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:31.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.506+0000 7f3987dbc640 1 --2- 172.21.15.67:0/2066880249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f398015e280 0x7f39800ccc80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f397000c780 tx=0x7f397000cc40 comp rx=0 tx=0).stop 2024-03-24T00:39:31.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.506+0000 7f3987dbc640 1 -- 172.21.15.67:0/2066880249 shutdown_connections 2024-03-24T00:39:31.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.506+0000 7f3987dbc640 1 --2- 172.21.15.67:0/2066880249 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f396c06bbd0 0x7f396c06e090 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:31.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.506+0000 7f3987dbc640 1 --2- 172.21.15.67:0/2066880249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f398015e280 0x7f39800ccc80 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:31.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.506+0000 7f3987dbc640 1 --2- 172.21.15.67:0/2066880249 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f398015dea0 0x7f39800cbac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:31.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.506+0000 7f3987dbc640 1 --2- 172.21.15.67:0/2066880249 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39800d3ae0 0x7f39801f33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:31.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.506+0000 7f3987dbc640 1 -- 172.21.15.67:0/2066880249 >> 172.21.15.67:0/2066880249 conn(0x7f3980072da0 msgr2=0x7f39800b3ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:31.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.507+0000 7f3987dbc640 1 -- 172.21.15.67:0/2066880249 shutdown_connections 2024-03-24T00:39:31.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.507+0000 7f3987dbc640 1 -- 172.21.15.67:0/2066880249 wait complete. 2024-03-24T00:39:31.517 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T00:39:31.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.621+0000 7efd8c30a640 1 Processor -- start 2024-03-24T00:39:31.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.622+0000 7efd8c30a640 1 -- start start 2024-03-24T00:39:31.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.622+0000 7efd8c30a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efd8415e130 0x7efd840ce610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:31.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.622+0000 7efd8c30a640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd840b65c0 con 0x7efd840d0640 2024-03-24T00:39:31.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.622+0000 7efd8c30a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd840cf570 con 0x7efd840d32c0 2024-03-24T00:39:31.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.622+0000 7efd8c30a640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd840ceb50 con 0x7efd8415e130 2024-03-24T00:39:31.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.623+0000 7efd8a0a6640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efd8415e130 0x7efd840ce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:31.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.623+0000 7efd898a5640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7efd840d0640 0x7efd840b57a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46986/0 (socket says 172.21.15.67:46986) 2024-03-24T00:39:31.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.623+0000 7efd898a5640 1 -- 172.21.15.67:0/1599511187 learned_addr learned my addr 172.21.15.67:0/1599511187 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:31.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.623+0000 7efd8a0a6640 1 -- 172.21.15.67:0/1599511187 >> v1:172.21.15.72:6789/0 conn(0x7efd840d32c0 legacy=0x7efd840b5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:31.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.623+0000 7efd8a0a6640 1 -- 172.21.15.67:0/1599511187 >> v1:172.21.15.67:6789/0 conn(0x7efd840d0640 legacy=0x7efd840b57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:31.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.623+0000 7efd8a0a6640 1 -- 172.21.15.67:0/1599511187 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd840cecf0 con 0x7efd8415e130 2024-03-24T00:39:31.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.623+0000 7efd890a4640 1 -- 172.21.15.67:0/1599511187 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2588417622 0 0) 0x7efd840b65c0 con 0x7efd840d0640 2024-03-24T00:39:31.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.624+0000 7efd8a0a6640 1 --2- 172.21.15.67:0/1599511187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd8415e130 0x7efd840ce610 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7efd800097a0 tx=0x7efd8002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17dc9b4e493e6a52 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:31.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.624+0000 7efd890a4640 1 -- 172.21.15.67:0/1599511187 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd80035020 con 0x7efd8415e130 2024-03-24T00:39:31.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.624+0000 7efd890a4640 1 -- 172.21.15.67:0/1599511187 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd80031830 con 0x7efd8415e130 2024-03-24T00:39:31.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.624+0000 7efd890a4640 1 -- 172.21.15.67:0/1599511187 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd80031c00 con 0x7efd8415e130 2024-03-24T00:39:31.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.624+0000 7efd8c30a640 1 -- 172.21.15.67:0/1599511187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd8415e130 msgr2=0x7efd840ce610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:31.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.624+0000 7efd8c30a640 1 --2- 172.21.15.67:0/1599511187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd8415e130 0x7efd840ce610 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7efd800097a0 tx=0x7efd8002b8d0 comp rx=0 tx=0).stop 2024-03-24T00:39:31.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.624+0000 7efd8c30a640 1 -- 172.21.15.67:0/1599511187 shutdown_connections 2024-03-24T00:39:31.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.624+0000 7efd8c30a640 1 --2- 172.21.15.67:0/1599511187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd8415e130 0x7efd840ce610 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:31.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.624+0000 7efd8c30a640 1 -- 172.21.15.67:0/1599511187 >> 172.21.15.67:0/1599511187 conn(0x7efd8406c8a0 msgr2=0x7efd8406ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:31.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.624+0000 7efd8c30a640 1 -- 172.21.15.67:0/1599511187 shutdown_connections 2024-03-24T00:39:31.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.624+0000 7efd8c30a640 1 -- 172.21.15.67:0/1599511187 wait complete. 2024-03-24T00:39:31.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.626+0000 7efd8c30a640 1 Processor -- start 2024-03-24T00:39:31.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.626+0000 7efd8c30a640 1 -- start start 2024-03-24T00:39:31.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.626+0000 7efd8c30a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd840d0640 0x7efd840c67c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:31.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.626+0000 7efd8c30a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd840d32c0 0x7efd840caf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:31.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.626+0000 7efd8c30a640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efd8415e130 0x7efd841f3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:31.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.626+0000 7efd8c30a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd840cf290 con 0x7efd840d32c0 2024-03-24T00:39:31.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.626+0000 7efd8c30a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd840cefb0 con 0x7efd840d0640 2024-03-24T00:39:31.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.626+0000 7efd8c30a640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd8415e510 con 0x7efd8415e130 2024-03-24T00:39:31.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.626+0000 7efd898a5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd840d32c0 0x7efd840caf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:31.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.626+0000 7efd898a5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd840d32c0 0x7efd840caf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34144/0 (socket says 172.21.15.67:34144) 2024-03-24T00:39:31.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.626+0000 7efd898a5640 1 -- 172.21.15.67:0/2259345473 learned_addr learned my addr 172.21.15.67:0/2259345473 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:31.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.626+0000 7efd8a0a6640 1 --2- 172.21.15.67:0/2259345473 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd840d0640 0x7efd840c67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:31.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.626+0000 7efd8a8a7640 1 --2- 172.21.15.67:0/2259345473 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efd8415e130 0x7efd841f3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:31.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.627+0000 7efd898a5640 1 -- 172.21.15.67:0/2259345473 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efd8415e130 msgr2=0x7efd841f3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:31.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.627+0000 7efd898a5640 1 --2- 172.21.15.67:0/2259345473 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efd8415e130 0x7efd841f3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:31.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.627+0000 7efd898a5640 1 -- 172.21.15.67:0/2259345473 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd840d0640 msgr2=0x7efd840c67c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:31.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.627+0000 7efd898a5640 1 --2- 172.21.15.67:0/2259345473 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd840d0640 0x7efd840c67c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:31.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.627+0000 7efd898a5640 1 -- 172.21.15.67:0/2259345473 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd840ccce0 con 0x7efd840d32c0 2024-03-24T00:39:31.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.627+0000 7efd898a5640 1 --2- 172.21.15.67:0/2259345473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd840d32c0 0x7efd840caf30 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7efd7800c4a0 tx=0x7efd7800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:31.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.627+0000 7efd76ffd640 1 -- 172.21.15.67:0/2259345473 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd78018070 con 0x7efd840d32c0 2024-03-24T00:39:31.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.627+0000 7efd76ffd640 1 -- 172.21.15.67:0/2259345473 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd780074e0 con 0x7efd840d32c0 2024-03-24T00:39:31.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.627+0000 7efd76ffd640 1 -- 172.21.15.67:0/2259345473 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd78004a20 con 0x7efd840d32c0 2024-03-24T00:39:31.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.627+0000 7efd8c30a640 1 -- 172.21.15.67:0/2259345473 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd840ccfc0 con 0x7efd840d32c0 2024-03-24T00:39:31.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.627+0000 7efd8c30a640 1 -- 172.21.15.67:0/2259345473 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd840cd580 con 0x7efd840d32c0 2024-03-24T00:39:31.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.628+0000 7efd8c30a640 1 -- 172.21.15.67:0/2259345473 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd8404f400 con 0x7efd840d32c0 2024-03-24T00:39:31.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.629+0000 7efd76ffd640 1 -- 172.21.15.67:0/2259345473 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7efd7801b070 con 0x7efd840d32c0 2024-03-24T00:39:31.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.629+0000 7efd76ffd640 1 --2- 172.21.15.67:0/2259345473 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7efd6806bbd0 0x7efd6806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:31.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.629+0000 7efd76ffd640 1 -- 172.21.15.67:0/2259345473 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7efd78089fd0 con 0x7efd840d32c0 2024-03-24T00:39:31.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.629+0000 7efd8a0a6640 1 --2- 172.21.15.67:0/2259345473 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7efd6806bbd0 0x7efd6806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:31.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.630+0000 7efd8a0a6640 1 --2- 172.21.15.67:0/2259345473 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7efd6806bbd0 0x7efd6806e090 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7efd80036000 tx=0x7efd8003c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:31.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.634+0000 7efd76ffd640 1 -- 172.21.15.67:0/2259345473 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7efd780581c0 con 0x7efd840d32c0 2024-03-24T00:39:31.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.817+0000 7efd8c30a640 1 -- 172.21.15.67:0/2259345473 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7efd840ca710 con 0x7efd840d32c0 2024-03-24T00:39:31.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.818+0000 7efd76ffd640 1 -- 172.21.15.67:0/2259345473 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7efd7805c260 con 0x7efd840d32c0 2024-03-24T00:39:31.818 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-24T00:39:31.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.820+0000 7efd8c30a640 1 -- 172.21.15.67:0/2259345473 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7efd6806bbd0 msgr2=0x7efd6806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:31.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.820+0000 7efd8c30a640 1 --2- 172.21.15.67:0/2259345473 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7efd6806bbd0 0x7efd6806e090 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7efd80036000 tx=0x7efd8003c040 comp rx=0 tx=0).stop 2024-03-24T00:39:31.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.820+0000 7efd8c30a640 1 -- 172.21.15.67:0/2259345473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd840d32c0 msgr2=0x7efd840caf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:31.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.820+0000 7efd8c30a640 1 --2- 172.21.15.67:0/2259345473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd840d32c0 0x7efd840caf30 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7efd7800c4a0 tx=0x7efd7800c960 comp rx=0 tx=0).stop 2024-03-24T00:39:31.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.820+0000 7efd8c30a640 1 -- 172.21.15.67:0/2259345473 shutdown_connections 2024-03-24T00:39:31.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.820+0000 7efd8c30a640 1 --2- 172.21.15.67:0/2259345473 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7efd6806bbd0 0x7efd6806e090 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:31.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.820+0000 7efd8c30a640 1 --2- 172.21.15.67:0/2259345473 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efd8415e130 0x7efd841f3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:31.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.820+0000 7efd8c30a640 1 --2- 172.21.15.67:0/2259345473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd840d32c0 0x7efd840caf30 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:31.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.820+0000 7efd8c30a640 1 --2- 172.21.15.67:0/2259345473 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd840d0640 0x7efd840c67c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:31.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.820+0000 7efd8c30a640 1 -- 172.21.15.67:0/2259345473 >> 172.21.15.67:0/2259345473 conn(0x7efd8406c8a0 msgr2=0x7efd840b5b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:31.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.820+0000 7efd8c30a640 1 -- 172.21.15.67:0/2259345473 shutdown_connections 2024-03-24T00:39:31.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.820+0000 7efd8c30a640 1 -- 172.21.15.67:0/2259345473 wait complete. 2024-03-24T00:39:31.830 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:39:31.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.937+0000 7fc6da47e640 1 Processor -- start 2024-03-24T00:39:31.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.937+0000 7fc6da47e640 1 -- start start 2024-03-24T00:39:31.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.938+0000 7fc6da47e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc6d415df30 0x7fc6d4162610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:31.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.938+0000 7fc6da47e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6d40ce920 con 0x7fc6d415e310 2024-03-24T00:39:31.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.938+0000 7fc6da47e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6d40cef20 con 0x7fc6d40d0d50 2024-03-24T00:39:31.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.938+0000 7fc6da47e640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6d4162b50 con 0x7fc6d415df30 2024-03-24T00:39:31.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.938+0000 7fc6d3fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc6d415df30 0x7fc6d4162610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:31.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.938+0000 7fc6d3fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc6d415df30 0x7fc6d4162610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34146/0 (socket says 172.21.15.67:34146) 2024-03-24T00:39:31.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.938+0000 7fc6d3fff640 1 -- 172.21.15.67:0/1694958539 learned_addr learned my addr 172.21.15.67:0/1694958539 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:31.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.938+0000 7fc6d37fe640 1 --1- 172.21.15.67:0/1694958539 >> v1:172.21.15.67:6789/0 conn(0x7fc6d415e310 0x7fc6d40d0640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46998/0 (socket says 172.21.15.67:46998) 2024-03-24T00:39:31.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.938+0000 7fc6d3fff640 1 -- 172.21.15.67:0/1694958539 >> v1:172.21.15.72:6789/0 conn(0x7fc6d40d0d50 legacy=0x7fc6d40d2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:31.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.938+0000 7fc6d3fff640 1 -- 172.21.15.67:0/1694958539 >> v1:172.21.15.67:6789/0 conn(0x7fc6d415e310 legacy=0x7fc6d40d0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:31.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.938+0000 7fc6d3fff640 1 -- 172.21.15.67:0/1694958539 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6d40cf200 con 0x7fc6d415df30 2024-03-24T00:39:31.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.939+0000 7fc6d3fff640 1 --2- 172.21.15.67:0/1694958539 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6d415df30 0x7fc6d4162610 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc6c4009870 tx=0x7fc6c402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=588adcf0a73722a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:31.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.939+0000 7fc6d2ffd640 1 -- 172.21.15.67:0/1694958539 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c4034070 con 0x7fc6d415df30 2024-03-24T00:39:31.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.939+0000 7fc6d2ffd640 1 -- 172.21.15.67:0/1694958539 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6c4030860 con 0x7fc6d415df30 2024-03-24T00:39:31.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.939+0000 7fc6d2ffd640 1 -- 172.21.15.67:0/1694958539 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c4030c30 con 0x7fc6d415df30 2024-03-24T00:39:31.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.939+0000 7fc6da47e640 1 -- 172.21.15.67:0/1694958539 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6d415df30 msgr2=0x7fc6d4162610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:31.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.939+0000 7fc6da47e640 1 --2- 172.21.15.67:0/1694958539 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6d415df30 0x7fc6d4162610 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc6c4009870 tx=0x7fc6c402b9a0 comp rx=0 tx=0).stop 2024-03-24T00:39:31.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.939+0000 7fc6da47e640 1 -- 172.21.15.67:0/1694958539 shutdown_connections 2024-03-24T00:39:31.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.939+0000 7fc6da47e640 1 --2- 172.21.15.67:0/1694958539 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6d415df30 0x7fc6d4162610 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:31.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.939+0000 7fc6da47e640 1 -- 172.21.15.67:0/1694958539 >> 172.21.15.67:0/1694958539 conn(0x7fc6d406c880 msgr2=0x7fc6d406cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:31.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.939+0000 7fc6da47e640 1 -- 172.21.15.67:0/1694958539 shutdown_connections 2024-03-24T00:39:31.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.939+0000 7fc6da47e640 1 -- 172.21.15.67:0/1694958539 wait complete. 2024-03-24T00:39:31.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.941+0000 7fc6da47e640 1 Processor -- start 2024-03-24T00:39:31.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.941+0000 7fc6da47e640 1 -- start start 2024-03-24T00:39:31.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.941+0000 7fc6da47e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc6d40d0d50 0x7fc6d415cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:31.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.941+0000 7fc6da47e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc6d415df30 0x7fc6d41e82a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:31.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.941+0000 7fc6da47e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6d40aa720 0x7fc6d41e87e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:31.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.941+0000 7fc6da47e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d40ce580 con 0x7fc6d40aa720 2024-03-24T00:39:31.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.941+0000 7fc6da47e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d40b32f0 con 0x7fc6d415df30 2024-03-24T00:39:31.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.941+0000 7fc6da47e640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d40ce700 con 0x7fc6d40d0d50 2024-03-24T00:39:31.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.942+0000 7fc6d8a1b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6d40aa720 0x7fc6d41e87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:31.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.942+0000 7fc6d8a1b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6d40aa720 0x7fc6d41e87e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34162/0 (socket says 172.21.15.67:34162) 2024-03-24T00:39:31.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.942+0000 7fc6d8a1b640 1 -- 172.21.15.67:0/1410165009 learned_addr learned my addr 172.21.15.67:0/1410165009 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:31.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.942+0000 7fc6d37fe640 1 --2- 172.21.15.67:0/1410165009 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc6d415df30 0x7fc6d41e82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:31.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.942+0000 7fc6d3fff640 1 --2- 172.21.15.67:0/1410165009 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc6d40d0d50 0x7fc6d415cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:31.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.942+0000 7fc6d8a1b640 1 -- 172.21.15.67:0/1410165009 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc6d40d0d50 msgr2=0x7fc6d415cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:31.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.942+0000 7fc6d8a1b640 1 --2- 172.21.15.67:0/1410165009 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc6d40d0d50 0x7fc6d415cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:31.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.942+0000 7fc6d8a1b640 1 -- 172.21.15.67:0/1410165009 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc6d415df30 msgr2=0x7fc6d41e82a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:31.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.942+0000 7fc6d8a1b640 1 --2- 172.21.15.67:0/1410165009 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc6d415df30 0x7fc6d41e82a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:31.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.942+0000 7fc6d8a1b640 1 -- 172.21.15.67:0/1410165009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6d40ac2e0 con 0x7fc6d40aa720 2024-03-24T00:39:31.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.942+0000 7fc6d3fff640 1 --2- 172.21.15.67:0/1410165009 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc6d40d0d50 0x7fc6d415cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:31.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.942+0000 7fc6d8a1b640 1 --2- 172.21.15.67:0/1410165009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6d40aa720 0x7fc6d41e87e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc00c730 tx=0x7fc6cc00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:31.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.942+0000 7fc6d0ff9640 1 -- 172.21.15.67:0/1410165009 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc003b70 con 0x7fc6d40aa720 2024-03-24T00:39:31.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.942+0000 7fc6d0ff9640 1 -- 172.21.15.67:0/1410165009 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6cc003d10 con 0x7fc6d40aa720 2024-03-24T00:39:31.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.942+0000 7fc6d0ff9640 1 -- 172.21.15.67:0/1410165009 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc00aa90 con 0x7fc6d40aa720 2024-03-24T00:39:31.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.942+0000 7fc6da47e640 1 -- 172.21.15.67:0/1410165009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6d40ac5c0 con 0x7fc6d40aa720 2024-03-24T00:39:31.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.942+0000 7fc6da47e640 1 -- 172.21.15.67:0/1410165009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6d40acb80 con 0x7fc6d40aa720 2024-03-24T00:39:31.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.943+0000 7fc6da47e640 1 -- 172.21.15.67:0/1410165009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6d404f400 con 0x7fc6d40aa720 2024-03-24T00:39:31.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.944+0000 7fc6d0ff9640 1 -- 172.21.15.67:0/1410165009 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc6cc00ac30 con 0x7fc6d40aa720 2024-03-24T00:39:31.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.944+0000 7fc6d0ff9640 1 --2- 172.21.15.67:0/1410165009 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc6c006bbd0 0x7fc6c006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:31.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.944+0000 7fc6d0ff9640 1 -- 172.21.15.67:0/1410165009 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fc6cc08ac30 con 0x7fc6d40aa720 2024-03-24T00:39:31.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.944+0000 7fc6d3fff640 1 --2- 172.21.15.67:0/1410165009 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc6c006bbd0 0x7fc6c006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:31.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.945+0000 7fc6d3fff640 1 --2- 172.21.15.67:0/1410165009 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc6c006bbd0 0x7fc6c006e090 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc6c4036000 tx=0x7fc6c403b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:31.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:31.948+0000 7fc6d0ff9640 1 -- 172.21.15.67:0/1410165009 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc6cc0593b0 con 0x7fc6d40aa720 2024-03-24T00:39:32.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.130+0000 7fc6da47e640 1 -- 172.21.15.67:0/1410165009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc6d40d1a30 con 0x7fc6d40aa720 2024-03-24T00:39:32.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.131+0000 7fc6d0ff9640 1 -- 172.21.15.67:0/1410165009 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18843 (secure 0 0 0) 0x7fc6cc058cb0 con 0x7fc6d40aa720 2024-03-24T00:39:32.131 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:39:32.131 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":26,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:39:31.901149+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:38:50.077085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:38:51.095454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T00:38:54.807117+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.14","mappings":[{"from":4,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:39:32.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.133+0000 7fc6da47e640 1 -- 172.21.15.67:0/1410165009 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc6c006bbd0 msgr2=0x7fc6c006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:32.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.133+0000 7fc6da47e640 1 --2- 172.21.15.67:0/1410165009 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc6c006bbd0 0x7fc6c006e090 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc6c4036000 tx=0x7fc6c403b040 comp rx=0 tx=0).stop 2024-03-24T00:39:32.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.133+0000 7fc6da47e640 1 -- 172.21.15.67:0/1410165009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6d40aa720 msgr2=0x7fc6d41e87e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:32.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.133+0000 7fc6da47e640 1 --2- 172.21.15.67:0/1410165009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6d40aa720 0x7fc6d41e87e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc00c730 tx=0x7fc6cc00cbf0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.133+0000 7fc6da47e640 1 -- 172.21.15.67:0/1410165009 shutdown_connections 2024-03-24T00:39:32.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.133+0000 7fc6da47e640 1 --2- 172.21.15.67:0/1410165009 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc6c006bbd0 0x7fc6c006e090 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.133+0000 7fc6da47e640 1 --2- 172.21.15.67:0/1410165009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6d40aa720 0x7fc6d41e87e0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.133+0000 7fc6da47e640 1 --2- 172.21.15.67:0/1410165009 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc6d415df30 0x7fc6d41e82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.133+0000 7fc6da47e640 1 --2- 172.21.15.67:0/1410165009 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc6d40d0d50 0x7fc6d415cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.133+0000 7fc6da47e640 1 -- 172.21.15.67:0/1410165009 >> 172.21.15.67:0/1410165009 conn(0x7fc6d406c880 msgr2=0x7fc6d4161b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:32.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.133+0000 7fc6da47e640 1 -- 172.21.15.67:0/1410165009 shutdown_connections 2024-03-24T00:39:32.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.133+0000 7fc6da47e640 1 -- 172.21.15.67:0/1410165009 wait complete. 2024-03-24T00:39:32.144 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:38:39.364950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:38:50.077085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:38:51.095454+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T00:38:54.807117+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T00:39:32.144 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:39:32.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.248+0000 7f6ed2c7e640 1 Processor -- start 2024-03-24T00:39:32.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.249+0000 7f6ed2c7e640 1 -- start start 2024-03-24T00:39:32.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.249+0000 7f6ed2c7e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f6ecc15dea0 0x7f6ecc0d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:32.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.249+0000 7f6ed2c7e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6ecc0b6170 0x7f6ecc0d3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:32.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.249+0000 7f6ed2c7e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ecc0d2250 con 0x7f6ecc15e280 2024-03-24T00:39:32.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.249+0000 7f6ed2c7e640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ecc0d2530 con 0x7f6ecc0b6170 2024-03-24T00:39:32.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.249+0000 7f6ed2c7e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ecc0d2810 con 0x7f6ecc15dea0 2024-03-24T00:39:32.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.249+0000 7f6ed121b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6ecc0b6170 0x7f6ecc0d3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:32.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.249+0000 7f6ecbfff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6ecc15e280 0x7f6ecc0b5a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47012/0 (socket says 172.21.15.67:47012) 2024-03-24T00:39:32.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.249+0000 7f6ecbfff640 1 -- 172.21.15.67:0/176986067 learned_addr learned my addr 172.21.15.67:0/176986067 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:32.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.249+0000 7f6ed0a1a640 1 --2- 172.21.15.67:0/176986067 >> v2:172.21.15.72:3300/0 conn(0x7f6ecc15dea0 0x7f6ecc0d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:32.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.250+0000 7f6ed121b640 1 -- 172.21.15.67:0/176986067 >> v2:172.21.15.72:3300/0 conn(0x7f6ecc15dea0 msgr2=0x7f6ecc0d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:32.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.250+0000 7f6ed121b640 1 --2- 172.21.15.67:0/176986067 >> v2:172.21.15.72:3300/0 conn(0x7f6ecc15dea0 0x7f6ecc0d33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.250+0000 7f6ed121b640 1 -- 172.21.15.67:0/176986067 >> v1:172.21.15.67:6789/0 conn(0x7f6ecc15e280 legacy=0x7f6ecc0b5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:32.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.250+0000 7f6ed121b640 1 -- 172.21.15.67:0/176986067 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ecc0d2af0 con 0x7f6ecc0b6170 2024-03-24T00:39:32.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.250+0000 7f6ecb7fe640 1 -- 172.21.15.67:0/176986067 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1742270928 0 0) 0x7f6ecc0d2250 con 0x7f6ecc15e280 2024-03-24T00:39:32.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.251+0000 7f6ed121b640 1 --2- 172.21.15.67:0/176986067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ecc0b6170 0x7f6ecc0d3910 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6ec4009d90 tx=0x7f6ec402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ede2abdac17a42ce server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:32.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.251+0000 7f6ecb7fe640 1 -- 172.21.15.67:0/176986067 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec4039070 con 0x7f6ecc0b6170 2024-03-24T00:39:32.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.251+0000 7f6ecb7fe640 1 -- 172.21.15.67:0/176986067 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ec402cbd0 con 0x7f6ecc0b6170 2024-03-24T00:39:32.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.251+0000 7f6ecb7fe640 1 -- 172.21.15.67:0/176986067 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec4034440 con 0x7f6ecc0b6170 2024-03-24T00:39:32.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.251+0000 7f6ed2c7e640 1 -- 172.21.15.67:0/176986067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ecc0b6170 msgr2=0x7f6ecc0d3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:32.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.251+0000 7f6ed2c7e640 1 --2- 172.21.15.67:0/176986067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ecc0b6170 0x7f6ecc0d3910 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6ec4009d90 tx=0x7f6ec402be20 comp rx=0 tx=0).stop 2024-03-24T00:39:32.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.251+0000 7f6ed2c7e640 1 -- 172.21.15.67:0/176986067 shutdown_connections 2024-03-24T00:39:32.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.251+0000 7f6ed2c7e640 1 --2- 172.21.15.67:0/176986067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ecc0b6170 0x7f6ecc0d3910 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.251+0000 7f6ed2c7e640 1 --2- 172.21.15.67:0/176986067 >> v2:172.21.15.72:3300/0 conn(0x7f6ecc15dea0 0x7f6ecc0d33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.251+0000 7f6ed2c7e640 1 -- 172.21.15.67:0/176986067 >> 172.21.15.67:0/176986067 conn(0x7f6ecc072da0 msgr2=0x7f6ecc0d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:32.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.251+0000 7f6ed2c7e640 1 -- 172.21.15.67:0/176986067 shutdown_connections 2024-03-24T00:39:32.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.251+0000 7f6ed2c7e640 1 -- 172.21.15.67:0/176986067 wait complete. 2024-03-24T00:39:32.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.252+0000 7f6ed2c7e640 1 Processor -- start 2024-03-24T00:39:32.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.252+0000 7f6ed2c7e640 1 -- start start 2024-03-24T00:39:32.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.253+0000 7f6ed2c7e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6ecc0b6170 0x7f6ecc15bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:32.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.253+0000 7f6ed2c7e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ecc15dea0 0x7f6ecc0b5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:32.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.253+0000 7f6ed2c7e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ecc15e280 0x7f6ecc0b59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:32.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.253+0000 7f6ed2c7e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ecc0b7480 con 0x7f6ecc15e280 2024-03-24T00:39:32.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.253+0000 7f6ed2c7e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ecc0b7300 con 0x7f6ecc15dea0 2024-03-24T00:39:32.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.253+0000 7f6ed2c7e640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ecc0b7600 con 0x7f6ecc0b6170 2024-03-24T00:39:32.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.253+0000 7f6ed121b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ecc15e280 0x7f6ecc0b59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:32.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.253+0000 7f6ed121b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ecc15e280 0x7f6ecc0b59d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34176/0 (socket says 172.21.15.67:34176) 2024-03-24T00:39:32.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.253+0000 7f6ed121b640 1 -- 172.21.15.67:0/3534618085 learned_addr learned my addr 172.21.15.67:0/3534618085 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:32.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.253+0000 7f6ed0a1a640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6ecc0b6170 0x7f6ecc15bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:32.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.253+0000 7f6ecbfff640 1 --2- 172.21.15.67:0/3534618085 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ecc15dea0 0x7f6ecc0b5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:32.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.254+0000 7f6ed121b640 1 -- 172.21.15.67:0/3534618085 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6ecc0b6170 msgr2=0x7f6ecc15bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:32.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.254+0000 7f6ed121b640 1 --2- 172.21.15.67:0/3534618085 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6ecc0b6170 0x7f6ecc15bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.254+0000 7f6ed121b640 1 -- 172.21.15.67:0/3534618085 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ecc15dea0 msgr2=0x7f6ecc0b5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:32.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.254+0000 7f6ed121b640 1 --2- 172.21.15.67:0/3534618085 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ecc15dea0 0x7f6ecc0b5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.254+0000 7f6ed121b640 1 -- 172.21.15.67:0/3534618085 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ecc1f6fc0 con 0x7f6ecc15e280 2024-03-24T00:39:32.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.254+0000 7f6ed0a1a640 1 --2- 172.21.15.67:0/3534618085 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6ecc0b6170 0x7f6ecc15bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:32.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.254+0000 7f6ed121b640 1 --2- 172.21.15.67:0/3534618085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ecc15e280 0x7f6ecc0b59d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6ec4009d90 tx=0x7f6ec4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:32.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.254+0000 7f6ec97fa640 1 -- 172.21.15.67:0/3534618085 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec4039050 con 0x7f6ecc15e280 2024-03-24T00:39:32.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.254+0000 7f6ec97fa640 1 -- 172.21.15.67:0/3534618085 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ec4045070 con 0x7f6ecc15e280 2024-03-24T00:39:32.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.254+0000 7f6ed2c7e640 1 -- 172.21.15.67:0/3534618085 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ecc1f72a0 con 0x7f6ecc15e280 2024-03-24T00:39:32.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.254+0000 7f6ec97fa640 1 -- 172.21.15.67:0/3534618085 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec4009450 con 0x7f6ecc15e280 2024-03-24T00:39:32.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.254+0000 7f6ed2c7e640 1 -- 172.21.15.67:0/3534618085 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ecc1f7860 con 0x7f6ecc15e280 2024-03-24T00:39:32.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.255+0000 7f6ed2c7e640 1 -- 172.21.15.67:0/3534618085 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ecc0d1bb0 con 0x7f6ecc15e280 2024-03-24T00:39:32.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.255+0000 7f6ec97fa640 1 -- 172.21.15.67:0/3534618085 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6ec403e850 con 0x7f6ecc15e280 2024-03-24T00:39:32.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.256+0000 7f6ec97fa640 1 --2- 172.21.15.67:0/3534618085 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6eb006bbd0 0x7f6eb006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:32.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.256+0000 7f6ec97fa640 1 -- 172.21.15.67:0/3534618085 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f6ec40adc70 con 0x7f6ecc15e280 2024-03-24T00:39:32.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.256+0000 7f6ed0a1a640 1 --2- 172.21.15.67:0/3534618085 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6eb006bbd0 0x7f6eb006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:32.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.256+0000 7f6ed0a1a640 1 --2- 172.21.15.67:0/3534618085 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6eb006bbd0 0x7f6eb006e090 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc0ca4c0 tx=0x7f6ec0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:32.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.260+0000 7f6ec97fa640 1 -- 172.21.15.67:0/3534618085 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6ecc0d1bb0 con 0x7f6ecc15e280 2024-03-24T00:39:32.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.441+0000 7f6ed2c7e640 1 -- 172.21.15.67:0/3534618085 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6ecc04f400 con 0x7f6ecc15e280 2024-03-24T00:39:32.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.441+0000 7f6ec97fa640 1 -- 172.21.15.67:0/3534618085 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6ec407be30 con 0x7f6ecc15e280 2024-03-24T00:39:32.441 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-24T00:39:32.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.443+0000 7f6ed2c7e640 1 -- 172.21.15.67:0/3534618085 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6eb006bbd0 msgr2=0x7f6eb006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:32.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.443+0000 7f6ed2c7e640 1 --2- 172.21.15.67:0/3534618085 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6eb006bbd0 0x7f6eb006e090 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc0ca4c0 tx=0x7f6ec0009190 comp rx=0 tx=0).stop 2024-03-24T00:39:32.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.443+0000 7f6ed2c7e640 1 -- 172.21.15.67:0/3534618085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ecc15e280 msgr2=0x7f6ecc0b59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:32.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.443+0000 7f6ed2c7e640 1 --2- 172.21.15.67:0/3534618085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ecc15e280 0x7f6ecc0b59d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6ec4009d90 tx=0x7f6ec4004a40 comp rx=0 tx=0).stop 2024-03-24T00:39:32.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.443+0000 7f6ed2c7e640 1 -- 172.21.15.67:0/3534618085 shutdown_connections 2024-03-24T00:39:32.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.443+0000 7f6ed2c7e640 1 --2- 172.21.15.67:0/3534618085 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6eb006bbd0 0x7f6eb006e090 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.443+0000 7f6ed2c7e640 1 --2- 172.21.15.67:0/3534618085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ecc15e280 0x7f6ecc0b59d0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.443+0000 7f6ed2c7e640 1 --2- 172.21.15.67:0/3534618085 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ecc15dea0 0x7f6ecc0b5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.443+0000 7f6ed2c7e640 1 --2- 172.21.15.67:0/3534618085 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6ecc0b6170 0x7f6ecc15bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.443+0000 7f6ed2c7e640 1 -- 172.21.15.67:0/3534618085 >> 172.21.15.67:0/3534618085 conn(0x7f6ecc072da0 msgr2=0x7f6ecc0d09b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:32.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.443+0000 7f6ed2c7e640 1 -- 172.21.15.67:0/3534618085 shutdown_connections 2024-03-24T00:39:32.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.443+0000 7f6ed2c7e640 1 -- 172.21.15.67:0/3534618085 wait complete. 2024-03-24T00:39:32.453 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:39:32.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.558+0000 7f8249ac6640 1 Processor -- start 2024-03-24T00:39:32.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.558+0000 7f8249ac6640 1 -- start start 2024-03-24T00:39:32.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.559+0000 7f8249ac6640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f824415bc30 0x7f82440b1a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:32.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.559+0000 7f8249ac6640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f824415c010 0x7f82440b1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:32.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.559+0000 7f8249ac6640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82440b3ae0 con 0x7f82440b2d90 2024-03-24T00:39:32.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.559+0000 7f8249ac6640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82440cca60 con 0x7f824415c010 2024-03-24T00:39:32.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.559+0000 7f8249ac6640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82440b3dc0 con 0x7f824415bc30 2024-03-24T00:39:32.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.559+0000 7f8242ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f824415c010 0x7f82440b1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:32.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.559+0000 7f8242ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f824415c010 0x7f82440b1f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34192/0 (socket says 172.21.15.67:34192) 2024-03-24T00:39:32.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.559+0000 7f8242ffd640 1 -- 172.21.15.67:0/947818483 learned_addr learned my addr 172.21.15.67:0/947818483 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:32.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.559+0000 7f82437fe640 1 --2- 172.21.15.67:0/947818483 >> v2:172.21.15.72:3300/0 conn(0x7f824415bc30 0x7f82440b1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:32.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.559+0000 7f8242ffd640 1 -- 172.21.15.67:0/947818483 >> v2:172.21.15.72:3300/0 conn(0x7f824415bc30 msgr2=0x7f82440b1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:32.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.559+0000 7f8242ffd640 1 --2- 172.21.15.67:0/947818483 >> v2:172.21.15.72:3300/0 conn(0x7f824415bc30 0x7f82440b1a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.559+0000 7f8242ffd640 1 -- 172.21.15.67:0/947818483 >> v1:172.21.15.72:6789/0 conn(0x7f82440b2d90 legacy=0x7f82440b3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:32.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.559+0000 7f82427fc640 1 -- 172.21.15.67:0/947818483 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3691629959 0 0) 0x7f82440b3ae0 con 0x7f82440b2d90 2024-03-24T00:39:32.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.559+0000 7f8242ffd640 1 -- 172.21.15.67:0/947818483 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82440b40a0 con 0x7f824415c010 2024-03-24T00:39:32.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.560+0000 7f8242ffd640 1 --2- 172.21.15.67:0/947818483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f824415c010 0x7f82440b1f40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f82440b4700 tx=0x7f823802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5015cd384e529494 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:32.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.560+0000 7f82427fc640 1 -- 172.21.15.67:0/947818483 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8238034070 con 0x7f824415c010 2024-03-24T00:39:32.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.560+0000 7f82427fc640 1 -- 172.21.15.67:0/947818483 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8238030860 con 0x7f824415c010 2024-03-24T00:39:32.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.560+0000 7f82427fc640 1 -- 172.21.15.67:0/947818483 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8238030c30 con 0x7f824415c010 2024-03-24T00:39:32.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.561+0000 7f8249ac6640 1 -- 172.21.15.67:0/947818483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f824415c010 msgr2=0x7f82440b1f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:32.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.561+0000 7f8249ac6640 1 --2- 172.21.15.67:0/947818483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f824415c010 0x7f82440b1f40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f82440b4700 tx=0x7f823802b9a0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.561+0000 7f8249ac6640 1 -- 172.21.15.67:0/947818483 shutdown_connections 2024-03-24T00:39:32.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.561+0000 7f8249ac6640 1 --2- 172.21.15.67:0/947818483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f824415c010 0x7f82440b1f40 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.561+0000 7f8249ac6640 1 --2- 172.21.15.67:0/947818483 >> v2:172.21.15.72:3300/0 conn(0x7f824415bc30 0x7f82440b1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.561+0000 7f8249ac6640 1 -- 172.21.15.67:0/947818483 >> 172.21.15.67:0/947818483 conn(0x7f824408b180 msgr2=0x7f824408b590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:32.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.561+0000 7f8249ac6640 1 -- 172.21.15.67:0/947818483 shutdown_connections 2024-03-24T00:39:32.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.561+0000 7f8249ac6640 1 -- 172.21.15.67:0/947818483 wait complete. 2024-03-24T00:39:32.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.562+0000 7f8249ac6640 1 Processor -- start 2024-03-24T00:39:32.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.562+0000 7f8249ac6640 1 -- start start 2024-03-24T00:39:32.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.563+0000 7f8249ac6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82440b2d90 0x7f8244155310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:32.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.563+0000 7f8249ac6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f824415bc30 0x7f824415cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:32.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.563+0000 7f8249ac6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f824415c010 0x7f82440c6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:32.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.563+0000 7f8249ac6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82440cd070 con 0x7f82440b2d90 2024-03-24T00:39:32.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.563+0000 7f8249ac6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82440ccef0 con 0x7f824415c010 2024-03-24T00:39:32.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.563+0000 7f8249ac6640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82440cd1f0 con 0x7f824415bc30 2024-03-24T00:39:32.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.563+0000 7f82437fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82440b2d90 0x7f8244155310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:32.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.563+0000 7f82437fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82440b2d90 0x7f8244155310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34208/0 (socket says 172.21.15.67:34208) 2024-03-24T00:39:32.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.563+0000 7f82437fe640 1 -- 172.21.15.67:0/638579507 learned_addr learned my addr 172.21.15.67:0/638579507 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:32.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.563+0000 7f8242ffd640 1 --2- 172.21.15.67:0/638579507 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f824415bc30 0x7f824415cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:32.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.563+0000 7f8243fff640 1 --2- 172.21.15.67:0/638579507 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f824415c010 0x7f82440c6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:32.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.563+0000 7f82437fe640 1 -- 172.21.15.67:0/638579507 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f824415bc30 msgr2=0x7f824415cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:32.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.563+0000 7f82437fe640 1 --2- 172.21.15.67:0/638579507 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f824415bc30 0x7f824415cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.563+0000 7f82437fe640 1 -- 172.21.15.67:0/638579507 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f824415c010 msgr2=0x7f82440c6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:32.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.563+0000 7f82437fe640 1 --2- 172.21.15.67:0/638579507 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f824415c010 0x7f82440c6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.563+0000 7f82437fe640 1 -- 172.21.15.67:0/638579507 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82440c7040 con 0x7f82440b2d90 2024-03-24T00:39:32.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.563+0000 7f82437fe640 1 --2- 172.21.15.67:0/638579507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82440b2d90 0x7f8244155310 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f823400e9b0 tx=0x7f823400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:32.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.564+0000 7f822bfff640 1 -- 172.21.15.67:0/638579507 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8234017070 con 0x7f82440b2d90 2024-03-24T00:39:32.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.564+0000 7f822bfff640 1 -- 172.21.15.67:0/638579507 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f823400c9a0 con 0x7f82440b2d90 2024-03-24T00:39:32.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.564+0000 7f8249ac6640 1 -- 172.21.15.67:0/638579507 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82440c7320 con 0x7f82440b2d90 2024-03-24T00:39:32.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.564+0000 7f822bfff640 1 -- 172.21.15.67:0/638579507 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f823400cd90 con 0x7f82440b2d90 2024-03-24T00:39:32.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.564+0000 7f8249ac6640 1 -- 172.21.15.67:0/638579507 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82440c7b20 con 0x7f82440b2d90 2024-03-24T00:39:32.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.565+0000 7f8249ac6640 1 -- 172.21.15.67:0/638579507 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f824404f400 con 0x7f82440b2d90 2024-03-24T00:39:32.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.565+0000 7f822bfff640 1 -- 172.21.15.67:0/638579507 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f823401a070 con 0x7f82440b2d90 2024-03-24T00:39:32.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.565+0000 7f822bfff640 1 --2- 172.21.15.67:0/638579507 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f823006bc00 0x7f823006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:32.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.565+0000 7f822bfff640 1 -- 172.21.15.67:0/638579507 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f823408b150 con 0x7f82440b2d90 2024-03-24T00:39:32.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.566+0000 7f8242ffd640 1 --2- 172.21.15.67:0/638579507 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f823006bc00 0x7f823006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:32.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.566+0000 7f8242ffd640 1 --2- 172.21.15.67:0/638579507 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f823006bc00 0x7f823006e0c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8238036000 tx=0x7f823803c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:32.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.570+0000 7f822bfff640 1 -- 172.21.15.67:0/638579507 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f82340591f0 con 0x7f82440b2d90 2024-03-24T00:39:32.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.750+0000 7f8249ac6640 1 -- 172.21.15.67:0/638579507 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f82440b17e0 con 0x7f82440b2d90 2024-03-24T00:39:32.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.750+0000 7f822bfff640 1 -- 172.21.15.67:0/638579507 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7f823405f070 con 0x7f82440b2d90 2024-03-24T00:39:32.750 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-24T00:39:32.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.752+0000 7f8249ac6640 1 -- 172.21.15.67:0/638579507 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f823006bc00 msgr2=0x7f823006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:32.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.752+0000 7f8249ac6640 1 --2- 172.21.15.67:0/638579507 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f823006bc00 0x7f823006e0c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8238036000 tx=0x7f823803c040 comp rx=0 tx=0).stop 2024-03-24T00:39:32.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.752+0000 7f8249ac6640 1 -- 172.21.15.67:0/638579507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82440b2d90 msgr2=0x7f8244155310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:32.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.752+0000 7f8249ac6640 1 --2- 172.21.15.67:0/638579507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82440b2d90 0x7f8244155310 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f823400e9b0 tx=0x7f823400ee70 comp rx=0 tx=0).stop 2024-03-24T00:39:32.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.752+0000 7f8249ac6640 1 -- 172.21.15.67:0/638579507 shutdown_connections 2024-03-24T00:39:32.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.753+0000 7f8249ac6640 1 --2- 172.21.15.67:0/638579507 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f823006bc00 0x7f823006e0c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.753+0000 7f8249ac6640 1 --2- 172.21.15.67:0/638579507 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f824415c010 0x7f82440c6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.753+0000 7f8249ac6640 1 --2- 172.21.15.67:0/638579507 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f824415bc30 0x7f824415cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.753+0000 7f8249ac6640 1 --2- 172.21.15.67:0/638579507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82440b2d90 0x7f8244155310 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.753+0000 7f8249ac6640 1 -- 172.21.15.67:0/638579507 >> 172.21.15.67:0/638579507 conn(0x7f824408b180 msgr2=0x7f8244156730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:32.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.753+0000 7f8249ac6640 1 -- 172.21.15.67:0/638579507 shutdown_connections 2024-03-24T00:39:32.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.753+0000 7f8249ac6640 1 -- 172.21.15.67:0/638579507 wait complete. 2024-03-24T00:39:32.763 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:39:32.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.868+0000 7f3327a85640 1 Processor -- start 2024-03-24T00:39:32.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.868+0000 7f3327a85640 1 -- start start 2024-03-24T00:39:32.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.869+0000 7f3327a85640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f33200b0f10 0x7f33200d1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:32.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.869+0000 7f3327a85640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33200b12f0 con 0x7f33200b2610 2024-03-24T00:39:32.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.869+0000 7f3327a85640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33200b2c50 con 0x7f33200d20d0 2024-03-24T00:39:32.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.869+0000 7f3327a85640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33200d2540 con 0x7f33200b0f10 2024-03-24T00:39:32.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.869+0000 7f3326022640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f33200b2610 0x7f332015d870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47026/0 (socket says 172.21.15.67:47026) 2024-03-24T00:39:32.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.869+0000 7f3326022640 1 -- 172.21.15.67:0/1893760431 learned_addr learned my addr 172.21.15.67:0/1893760431 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:32.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.869+0000 7f3325821640 1 --2- 172.21.15.67:0/1893760431 >> v2:172.21.15.72:3300/0 conn(0x7f33200b0f10 0x7f33200d1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:32.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.870+0000 7f332481f640 1 -- 172.21.15.67:0/1893760431 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 716439577 0 0) 0x7f33200b12f0 con 0x7f33200b2610 2024-03-24T00:39:32.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.870+0000 7f332481f640 1 -- 172.21.15.67:0/1893760431 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3308003520 con 0x7f33200b2610 2024-03-24T00:39:32.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.870+0000 7f332481f640 1 -- 172.21.15.67:0/1893760431 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3336106091 0 0) 0x7f33200b2c50 con 0x7f33200d20d0 2024-03-24T00:39:32.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.870+0000 7f332481f640 1 -- 172.21.15.67:0/1893760431 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33200b12f0 con 0x7f33200d20d0 2024-03-24T00:39:32.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.870+0000 7f3325821640 1 -- 172.21.15.67:0/1893760431 >> v1:172.21.15.72:6789/0 conn(0x7f33200d20d0 legacy=0x7f33200b1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:32.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.870+0000 7f3325821640 1 -- 172.21.15.67:0/1893760431 >> v1:172.21.15.67:6789/0 conn(0x7f33200b2610 legacy=0x7f332015d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:32.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.870+0000 7f3325821640 1 -- 172.21.15.67:0/1893760431 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33200ce880 con 0x7f33200b0f10 2024-03-24T00:39:32.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.870+0000 7f3325821640 1 --2- 172.21.15.67:0/1893760431 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f33200b0f10 0x7f33200d1b90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3310009d90 tx=0x7f331002bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef004512396ff47e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:32.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.871+0000 7f332481f640 1 -- 172.21.15.67:0/1893760431 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3310039070 con 0x7f33200b0f10 2024-03-24T00:39:32.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.871+0000 7f332481f640 1 -- 172.21.15.67:0/1893760431 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f331002cd00 con 0x7f33200b0f10 2024-03-24T00:39:32.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.871+0000 7f332481f640 1 -- 172.21.15.67:0/1893760431 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3310034660 con 0x7f33200b0f10 2024-03-24T00:39:32.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.871+0000 7f3327a85640 1 -- 172.21.15.67:0/1893760431 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f33200b0f10 msgr2=0x7f33200d1b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:32.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.871+0000 7f3327a85640 1 --2- 172.21.15.67:0/1893760431 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f33200b0f10 0x7f33200d1b90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3310009d90 tx=0x7f331002bf50 comp rx=0 tx=0).stop 2024-03-24T00:39:32.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.871+0000 7f3327a85640 1 -- 172.21.15.67:0/1893760431 shutdown_connections 2024-03-24T00:39:32.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.871+0000 7f3327a85640 1 --2- 172.21.15.67:0/1893760431 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f33200b0f10 0x7f33200d1b90 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.871+0000 7f3327a85640 1 -- 172.21.15.67:0/1893760431 >> 172.21.15.67:0/1893760431 conn(0x7f332008b1b0 msgr2=0x7f332008b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:32.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.871+0000 7f3327a85640 1 -- 172.21.15.67:0/1893760431 shutdown_connections 2024-03-24T00:39:32.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.871+0000 7f3327a85640 1 -- 172.21.15.67:0/1893760431 wait complete. 2024-03-24T00:39:32.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.872+0000 7f3327a85640 1 Processor -- start 2024-03-24T00:39:32.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.872+0000 7f3327a85640 1 -- start start 2024-03-24T00:39:32.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.873+0000 7f3327a85640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33200b0f10 0x7f33200c8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:32.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.873+0000 7f3327a85640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f33200b2610 0x7f33201560e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:32.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.873+0000 7f3327a85640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f33200d20d0 0x7f33200cb5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:32.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.873+0000 7f3327a85640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33200cee40 con 0x7f33200b0f10 2024-03-24T00:39:32.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.873+0000 7f3327a85640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33200ce560 con 0x7f33200b2610 2024-03-24T00:39:32.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.873+0000 7f3327a85640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33200cbae0 con 0x7f33200d20d0 2024-03-24T00:39:32.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.873+0000 7f3325821640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33200b0f10 0x7f33200c8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:32.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.873+0000 7f3325821640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33200b0f10 0x7f33200c8cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34212/0 (socket says 172.21.15.67:34212) 2024-03-24T00:39:32.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.873+0000 7f3325821640 1 -- 172.21.15.67:0/609690573 learned_addr learned my addr 172.21.15.67:0/609690573 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:32.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.873+0000 7f3325020640 1 --2- 172.21.15.67:0/609690573 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f33200b2610 0x7f33201560e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:32.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.873+0000 7f3326022640 1 --2- 172.21.15.67:0/609690573 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f33200d20d0 0x7f33200cb5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:32.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.873+0000 7f3325821640 1 -- 172.21.15.67:0/609690573 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f33200d20d0 msgr2=0x7f33200cb5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:32.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.873+0000 7f3325821640 1 --2- 172.21.15.67:0/609690573 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f33200d20d0 0x7f33200cb5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.873+0000 7f3325821640 1 -- 172.21.15.67:0/609690573 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f33200b2610 msgr2=0x7f33201560e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:32.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.873+0000 7f3325821640 1 --2- 172.21.15.67:0/609690573 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f33200b2610 0x7f33201560e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:32.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.873+0000 7f3325821640 1 -- 172.21.15.67:0/609690573 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33200c9360 con 0x7f33200b0f10 2024-03-24T00:39:32.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.874+0000 7f3325821640 1 --2- 172.21.15.67:0/609690573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33200b0f10 0x7f33200c8cd0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f3310009d90 tx=0x7f3310004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:32.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.874+0000 7f331e7fc640 1 -- 172.21.15.67:0/609690573 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3310039050 con 0x7f33200b0f10 2024-03-24T00:39:32.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.874+0000 7f331e7fc640 1 -- 172.21.15.67:0/609690573 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3310045070 con 0x7f33200b0f10 2024-03-24T00:39:32.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.874+0000 7f3327a85640 1 -- 172.21.15.67:0/609690573 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33200c9640 con 0x7f33200b0f10 2024-03-24T00:39:32.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.874+0000 7f331e7fc640 1 -- 172.21.15.67:0/609690573 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3310009450 con 0x7f33200b0f10 2024-03-24T00:39:32.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.874+0000 7f3327a85640 1 -- 172.21.15.67:0/609690573 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33200c9c00 con 0x7f33200b0f10 2024-03-24T00:39:32.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.875+0000 7f3327a85640 1 -- 172.21.15.67:0/609690573 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f332004f400 con 0x7f33200b0f10 2024-03-24T00:39:32.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.875+0000 7f331e7fc640 1 -- 172.21.15.67:0/609690573 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f331003e850 con 0x7f33200b0f10 2024-03-24T00:39:32.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.876+0000 7f331e7fc640 1 --2- 172.21.15.67:0/609690573 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f330c06bc00 0x7f330c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:32.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.876+0000 7f331e7fc640 1 -- 172.21.15.67:0/609690573 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f33100ad2a0 con 0x7f33200b0f10 2024-03-24T00:39:32.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.876+0000 7f3325020640 1 --2- 172.21.15.67:0/609690573 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f330c06bc00 0x7f330c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:32.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.876+0000 7f3325020640 1 --2- 172.21.15.67:0/609690573 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f330c06bc00 0x7f330c06e0c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3314007fd0 tx=0x7f3314009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:32.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:32.880+0000 7f331e7fc640 1 -- 172.21.15.67:0/609690573 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f331004e020 con 0x7f33200b0f10 2024-03-24T00:39:33.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.069+0000 7f3327a85640 1 -- 172.21.15.67:0/609690573 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f33200b1810 con 0x7f33200b0f10 2024-03-24T00:39:33.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.069+0000 7f331e7fc640 1 -- 172.21.15.67:0/609690573 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f331007b400 con 0x7f33200b0f10 2024-03-24T00:39:33.069 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-03-24T00:39:33.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.071+0000 7f3327a85640 1 -- 172.21.15.67:0/609690573 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f330c06bc00 msgr2=0x7f330c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:33.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.071+0000 7f3327a85640 1 --2- 172.21.15.67:0/609690573 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f330c06bc00 0x7f330c06e0c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3314007fd0 tx=0x7f3314009040 comp rx=0 tx=0).stop 2024-03-24T00:39:33.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.071+0000 7f3327a85640 1 -- 172.21.15.67:0/609690573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33200b0f10 msgr2=0x7f33200c8cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:33.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.071+0000 7f3327a85640 1 --2- 172.21.15.67:0/609690573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33200b0f10 0x7f33200c8cd0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f3310009d90 tx=0x7f3310004a40 comp rx=0 tx=0).stop 2024-03-24T00:39:33.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.071+0000 7f3327a85640 1 -- 172.21.15.67:0/609690573 shutdown_connections 2024-03-24T00:39:33.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.071+0000 7f3327a85640 1 --2- 172.21.15.67:0/609690573 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f330c06bc00 0x7f330c06e0c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:33.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.071+0000 7f3327a85640 1 --2- 172.21.15.67:0/609690573 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f33200d20d0 0x7f33200cb5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:33.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.071+0000 7f3327a85640 1 --2- 172.21.15.67:0/609690573 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f33200b2610 0x7f33201560e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:33.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.071+0000 7f3327a85640 1 --2- 172.21.15.67:0/609690573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33200b0f10 0x7f33200c8cd0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:33.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.071+0000 7f3327a85640 1 -- 172.21.15.67:0/609690573 >> 172.21.15.67:0/609690573 conn(0x7f332008b1b0 msgr2=0x7f33200d0040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:33.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.071+0000 7f3327a85640 1 -- 172.21.15.67:0/609690573 shutdown_connections 2024-03-24T00:39:33.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.071+0000 7f3327a85640 1 -- 172.21.15.67:0/609690573 wait complete. 2024-03-24T00:39:33.081 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T00:39:33.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.186+0000 7f9161ca1640 1 Processor -- start 2024-03-24T00:39:33.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.186+0000 7f9161ca1640 1 -- start start 2024-03-24T00:39:33.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.187+0000 7f9161ca1640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f915c15dfd0 0x7f915c0cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:33.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.187+0000 7f9161ca1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f915c0b41e0 0x7f915c0b3620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:33.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.187+0000 7f9161ca1640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f915c0cf090 con 0x7f915c0b3b60 2024-03-24T00:39:33.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.187+0000 7f9161ca1640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f915c0cc260 con 0x7f915c15dfd0 2024-03-24T00:39:33.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.187+0000 7f9161ca1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f915c0d0320 con 0x7f915c0b41e0 2024-03-24T00:39:33.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.187+0000 7f915bfff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f915c0b3b60 0x7f915c161b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47034/0 (socket says 172.21.15.67:47034) 2024-03-24T00:39:33.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.187+0000 7f915bfff640 1 -- 172.21.15.67:0/64520158 learned_addr learned my addr 172.21.15.67:0/64520158 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:33.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.187+0000 7f915affd640 1 --2- 172.21.15.67:0/64520158 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f915c0b41e0 0x7f915c0b3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:33.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.187+0000 7f915b7fe640 1 --2- 172.21.15.67:0/64520158 >> v2:172.21.15.72:3300/0 conn(0x7f915c15dfd0 0x7f915c0cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:33.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.187+0000 7f915a7fc640 1 -- 172.21.15.67:0/64520158 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1707607159 0 0) 0x7f915c0cf090 con 0x7f915c0b3b60 2024-03-24T00:39:33.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.187+0000 7f915a7fc640 1 -- 172.21.15.67:0/64520158 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9144003520 con 0x7f915c0b3b60 2024-03-24T00:39:33.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.187+0000 7f915b7fe640 1 -- 172.21.15.67:0/64520158 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f915c0b41e0 msgr2=0x7f915c0b3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:33.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.187+0000 7f915b7fe640 1 --2- 172.21.15.67:0/64520158 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f915c0b41e0 0x7f915c0b3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:33.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.187+0000 7f915b7fe640 1 -- 172.21.15.67:0/64520158 >> v1:172.21.15.67:6789/0 conn(0x7f915c0b3b60 legacy=0x7f915c161b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:33.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.187+0000 7f915b7fe640 1 -- 172.21.15.67:0/64520158 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f915c0d0600 con 0x7f915c15dfd0 2024-03-24T00:39:33.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.188+0000 7f915b7fe640 1 --2- 172.21.15.67:0/64520158 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f915c15dfd0 0x7f915c0cf980 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f914c0097a0 tx=0x7f914c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c249603535744619 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:33.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.188+0000 7f915a7fc640 1 -- 172.21.15.67:0/64520158 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f914c034070 con 0x7f915c15dfd0 2024-03-24T00:39:33.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.188+0000 7f915a7fc640 1 -- 172.21.15.67:0/64520158 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f914c030610 con 0x7f915c15dfd0 2024-03-24T00:39:33.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.188+0000 7f915a7fc640 1 -- 172.21.15.67:0/64520158 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f914c0309e0 con 0x7f915c15dfd0 2024-03-24T00:39:33.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.189+0000 7f9161ca1640 1 -- 172.21.15.67:0/64520158 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f915c15dfd0 msgr2=0x7f915c0cf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:33.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.189+0000 7f9161ca1640 1 --2- 172.21.15.67:0/64520158 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f915c15dfd0 0x7f915c0cf980 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f914c0097a0 tx=0x7f914c02b960 comp rx=0 tx=0).stop 2024-03-24T00:39:33.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.189+0000 7f9161ca1640 1 -- 172.21.15.67:0/64520158 shutdown_connections 2024-03-24T00:39:33.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.189+0000 7f9161ca1640 1 --2- 172.21.15.67:0/64520158 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f915c0b41e0 0x7f915c0b3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:33.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.189+0000 7f9161ca1640 1 --2- 172.21.15.67:0/64520158 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f915c15dfd0 0x7f915c0cf980 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:33.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.189+0000 7f9161ca1640 1 -- 172.21.15.67:0/64520158 >> 172.21.15.67:0/64520158 conn(0x7f915c06fcf0 msgr2=0x7f915c0700d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:33.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.189+0000 7f9161ca1640 1 -- 172.21.15.67:0/64520158 shutdown_connections 2024-03-24T00:39:33.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.189+0000 7f9161ca1640 1 -- 172.21.15.67:0/64520158 wait complete. 2024-03-24T00:39:33.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.190+0000 7f9161ca1640 1 Processor -- start 2024-03-24T00:39:33.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.190+0000 7f9161ca1640 1 -- start start 2024-03-24T00:39:33.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.191+0000 7f9161ca1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f915c0b3b60 0x7f915c0cad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:33.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.191+0000 7f9161ca1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f915c0b41e0 0x7f915c1f8020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:33.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.191+0000 7f9161ca1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f915c15dfd0 0x7f915c14ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:33.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.191+0000 7f9161ca1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f915c0cfec0 con 0x7f915c15dfd0 2024-03-24T00:39:33.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.191+0000 7f9161ca1640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f915c0cedb0 con 0x7f915c0b3b60 2024-03-24T00:39:33.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.191+0000 7f9161ca1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f915c15e3b0 con 0x7f915c0b41e0 2024-03-24T00:39:33.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.191+0000 7f915bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f915c15dfd0 0x7f915c14ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:33.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.191+0000 7f915b7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f915c0b3b60 0x7f915c0cad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:33.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.191+0000 7f915affd640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f915c0b41e0 0x7f915c1f8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:33.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.191+0000 7f915bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f915c15dfd0 0x7f915c14ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34220/0 (socket says 172.21.15.67:34220) 2024-03-24T00:39:33.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.191+0000 7f915bfff640 1 -- 172.21.15.67:0/4245830243 learned_addr learned my addr 172.21.15.67:0/4245830243 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:33.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.191+0000 7f915bfff640 1 -- 172.21.15.67:0/4245830243 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f915c0b41e0 msgr2=0x7f915c1f8020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:33.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.191+0000 7f915bfff640 1 --2- 172.21.15.67:0/4245830243 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f915c0b41e0 0x7f915c1f8020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:33.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.191+0000 7f915bfff640 1 -- 172.21.15.67:0/4245830243 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f915c0b3b60 msgr2=0x7f915c0cad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:33.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.191+0000 7f915bfff640 1 --2- 172.21.15.67:0/4245830243 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f915c0b3b60 0x7f915c0cad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:33.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.191+0000 7f915bfff640 1 -- 172.21.15.67:0/4245830243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f915c1f8560 con 0x7f915c15dfd0 2024-03-24T00:39:33.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.191+0000 7f915b7fe640 1 --2- 172.21.15.67:0/4245830243 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f915c0b3b60 0x7f915c0cad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:33.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.191+0000 7f915bfff640 1 --2- 172.21.15.67:0/4245830243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f915c15dfd0 0x7f915c14ec10 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f915400cb70 tx=0x7f915400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:33.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.192+0000 7f9143fff640 1 -- 172.21.15.67:0/4245830243 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9154017070 con 0x7f915c15dfd0 2024-03-24T00:39:33.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.192+0000 7f9143fff640 1 -- 172.21.15.67:0/4245830243 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f915400ad40 con 0x7f915c15dfd0 2024-03-24T00:39:33.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.192+0000 7f9161ca1640 1 -- 172.21.15.67:0/4245830243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f915c160300 con 0x7f915c15dfd0 2024-03-24T00:39:33.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.192+0000 7f9143fff640 1 -- 172.21.15.67:0/4245830243 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9154012620 con 0x7f915c15dfd0 2024-03-24T00:39:33.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.192+0000 7f9161ca1640 1 -- 172.21.15.67:0/4245830243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f915c0cb6d0 con 0x7f915c15dfd0 2024-03-24T00:39:33.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.193+0000 7f9161ca1640 1 -- 172.21.15.67:0/4245830243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f915c04f400 con 0x7f915c15dfd0 2024-03-24T00:39:33.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.193+0000 7f9143fff640 1 -- 172.21.15.67:0/4245830243 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f915401a070 con 0x7f915c15dfd0 2024-03-24T00:39:33.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.193+0000 7f9143fff640 1 --2- 172.21.15.67:0/4245830243 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f914806bbd0 0x7f914806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:33.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.194+0000 7f9143fff640 1 -- 172.21.15.67:0/4245830243 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f915408ac90 con 0x7f915c15dfd0 2024-03-24T00:39:33.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.194+0000 7f915b7fe640 1 --2- 172.21.15.67:0/4245830243 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f914806bbd0 0x7f914806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:33.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.194+0000 7f915b7fe640 1 --2- 172.21.15.67:0/4245830243 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f914806bbd0 0x7f914806e090 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f914c036000 tx=0x7f914c03b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:33.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.198+0000 7f9143fff640 1 -- 172.21.15.67:0/4245830243 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f915c1f8560 con 0x7f915c15dfd0 2024-03-24T00:39:33.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.379+0000 7f9161ca1640 1 -- 172.21.15.67:0/4245830243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f915c0b7b20 con 0x7f915c15dfd0 2024-03-24T00:39:33.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.379+0000 7f9143fff640 1 -- 172.21.15.67:0/4245830243 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f9154058cb0 con 0x7f915c15dfd0 2024-03-24T00:39:33.379 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-24T00:39:33.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.381+0000 7f9161ca1640 1 -- 172.21.15.67:0/4245830243 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f914806bbd0 msgr2=0x7f914806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:33.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.381+0000 7f9161ca1640 1 --2- 172.21.15.67:0/4245830243 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f914806bbd0 0x7f914806e090 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f914c036000 tx=0x7f914c03b040 comp rx=0 tx=0).stop 2024-03-24T00:39:33.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.381+0000 7f9161ca1640 1 -- 172.21.15.67:0/4245830243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f915c15dfd0 msgr2=0x7f915c14ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:33.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.381+0000 7f9161ca1640 1 --2- 172.21.15.67:0/4245830243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f915c15dfd0 0x7f915c14ec10 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f915400cb70 tx=0x7f915400a430 comp rx=0 tx=0).stop 2024-03-24T00:39:33.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.381+0000 7f9161ca1640 1 -- 172.21.15.67:0/4245830243 shutdown_connections 2024-03-24T00:39:33.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.381+0000 7f9161ca1640 1 --2- 172.21.15.67:0/4245830243 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f914806bbd0 0x7f914806e090 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:33.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.381+0000 7f9161ca1640 1 --2- 172.21.15.67:0/4245830243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f915c15dfd0 0x7f915c14ec10 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:33.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.381+0000 7f9161ca1640 1 --2- 172.21.15.67:0/4245830243 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f915c0b41e0 0x7f915c1f8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:33.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.381+0000 7f9161ca1640 1 --2- 172.21.15.67:0/4245830243 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f915c0b3b60 0x7f915c0cad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:33.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.381+0000 7f9161ca1640 1 -- 172.21.15.67:0/4245830243 >> 172.21.15.67:0/4245830243 conn(0x7f915c06fcf0 msgr2=0x7f915c0b24b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:33.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.381+0000 7f9161ca1640 1 -- 172.21.15.67:0/4245830243 shutdown_connections 2024-03-24T00:39:33.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.381+0000 7f9161ca1640 1 -- 172.21.15.67:0/4245830243 wait complete. 2024-03-24T00:39:33.392 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:39:33.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.493+0000 7fb3e8419640 1 Processor -- start 2024-03-24T00:39:33.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.493+0000 7fb3e8419640 1 -- start start 2024-03-24T00:39:33.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.494+0000 7fb3e8419640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb3e00b21f0 0x7fb3e00cfe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:33.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.494+0000 7fb3e8419640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb3e006b040 0x7fb3e006b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:33.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.494+0000 7fb3e8419640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3e00cdb00 con 0x7fb3e00cd670 2024-03-24T00:39:33.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.494+0000 7fb3e8419640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3e00b25d0 con 0x7fb3e006b040 2024-03-24T00:39:33.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.494+0000 7fb3e8419640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3e00d1a50 con 0x7fb3e00b21f0 2024-03-24T00:39:33.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.494+0000 7fb3e69b6640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb3e006b040 0x7fb3e006b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:33.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.494+0000 7fb3e69b6640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb3e006b040 0x7fb3e006b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50118/0 (socket says 172.21.15.67:50118) 2024-03-24T00:39:33.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.494+0000 7fb3e69b6640 1 -- 172.21.15.67:0/659668486 learned_addr learned my addr 172.21.15.67:0/659668486 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:33.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.494+0000 7fb3e61b5640 1 --2- 172.21.15.67:0/659668486 >> v2:172.21.15.72:3300/0 conn(0x7fb3e00b21f0 0x7fb3e00cfe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:33.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.495+0000 7fb3e69b6640 1 -- 172.21.15.67:0/659668486 >> v2:172.21.15.72:3300/0 conn(0x7fb3e00b21f0 msgr2=0x7fb3e00cfe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:33.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.495+0000 7fb3e69b6640 1 --2- 172.21.15.67:0/659668486 >> v2:172.21.15.72:3300/0 conn(0x7fb3e00b21f0 0x7fb3e00cfe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:33.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.495+0000 7fb3e69b6640 1 -- 172.21.15.67:0/659668486 >> v1:172.21.15.72:6789/0 conn(0x7fb3e00cd670 legacy=0x7fb3e01624d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:33.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.495+0000 7fb3e69b6640 1 -- 172.21.15.67:0/659668486 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3e0161d10 con 0x7fb3e006b040 2024-03-24T00:39:33.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.495+0000 7fb3e69b6640 1 --2- 172.21.15.67:0/659668486 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3e006b040 0x7fb3e006b4d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb3e00728f0 tx=0x7fb3d802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41f4281501179f52 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:33.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.496+0000 7fb3e51b3640 1 -- 172.21.15.67:0/659668486 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d8035020 con 0x7fb3e006b040 2024-03-24T00:39:33.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.496+0000 7fb3e51b3640 1 -- 172.21.15.67:0/659668486 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3d8031870 con 0x7fb3e006b040 2024-03-24T00:39:33.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.496+0000 7fb3e51b3640 1 -- 172.21.15.67:0/659668486 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d8031c40 con 0x7fb3e006b040 2024-03-24T00:39:33.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.496+0000 7fb3e8419640 1 -- 172.21.15.67:0/659668486 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3e006b040 msgr2=0x7fb3e006b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:33.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.496+0000 7fb3e8419640 1 --2- 172.21.15.67:0/659668486 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3e006b040 0x7fb3e006b4d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb3e00728f0 tx=0x7fb3d802b910 comp rx=0 tx=0).stop 2024-03-24T00:39:33.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.496+0000 7fb3e8419640 1 -- 172.21.15.67:0/659668486 shutdown_connections 2024-03-24T00:39:33.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.496+0000 7fb3e8419640 1 --2- 172.21.15.67:0/659668486 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3e006b040 0x7fb3e006b4d0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:33.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.496+0000 7fb3e8419640 1 --2- 172.21.15.67:0/659668486 >> v2:172.21.15.72:3300/0 conn(0x7fb3e00b21f0 0x7fb3e00cfe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:33.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.496+0000 7fb3e8419640 1 -- 172.21.15.67:0/659668486 >> 172.21.15.67:0/659668486 conn(0x7fb3e0099a30 msgr2=0x7fb3e0099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:33.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.496+0000 7fb3e8419640 1 -- 172.21.15.67:0/659668486 shutdown_connections 2024-03-24T00:39:33.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.496+0000 7fb3e8419640 1 -- 172.21.15.67:0/659668486 wait complete. 2024-03-24T00:39:33.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.497+0000 7fb3e8419640 1 Processor -- start 2024-03-24T00:39:33.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.497+0000 7fb3e8419640 1 -- start start 2024-03-24T00:39:33.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.498+0000 7fb3e8419640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb3e006b040 0x7fb3e00c4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:33.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.498+0000 7fb3e8419640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3e00b21f0 0x7fb3e014a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:33.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.498+0000 7fb3e8419640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3e00cd670 0x7fb3e014a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:33.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.498+0000 7fb3e8419640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e0161fb0 con 0x7fb3e00cd670 2024-03-24T00:39:33.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.498+0000 7fb3e8419640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e00d0760 con 0x7fb3e00b21f0 2024-03-24T00:39:33.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.498+0000 7fb3e8419640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e00ce1a0 con 0x7fb3e006b040 2024-03-24T00:39:33.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.498+0000 7fb3e69b6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3e00cd670 0x7fb3e014a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:33.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.498+0000 7fb3e69b6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3e00cd670 0x7fb3e014a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50130/0 (socket says 172.21.15.67:50130) 2024-03-24T00:39:33.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.498+0000 7fb3e69b6640 1 -- 172.21.15.67:0/2995157983 learned_addr learned my addr 172.21.15.67:0/2995157983 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:33.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.498+0000 7fb3e59b4640 1 --2- 172.21.15.67:0/2995157983 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3e00b21f0 0x7fb3e014a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:33.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.498+0000 7fb3e61b5640 1 --2- 172.21.15.67:0/2995157983 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb3e006b040 0x7fb3e00c4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:33.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.499+0000 7fb3e69b6640 1 -- 172.21.15.67:0/2995157983 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb3e006b040 msgr2=0x7fb3e00c4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:33.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.499+0000 7fb3e69b6640 1 --2- 172.21.15.67:0/2995157983 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb3e006b040 0x7fb3e00c4790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:33.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.499+0000 7fb3e69b6640 1 -- 172.21.15.67:0/2995157983 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3e00b21f0 msgr2=0x7fb3e014a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:33.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.499+0000 7fb3e69b6640 1 --2- 172.21.15.67:0/2995157983 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3e00b21f0 0x7fb3e014a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:33.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.499+0000 7fb3e69b6640 1 -- 172.21.15.67:0/2995157983 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3e00b4a60 con 0x7fb3e00cd670 2024-03-24T00:39:33.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.499+0000 7fb3e69b6640 1 --2- 172.21.15.67:0/2995157983 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3e00cd670 0x7fb3e014a6e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb3d802b950 tx=0x7fb3d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:33.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.499+0000 7fb3deffd640 1 -- 172.21.15.67:0/2995157983 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d803ad60 con 0x7fb3e00cd670 2024-03-24T00:39:33.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.499+0000 7fb3deffd640 1 -- 172.21.15.67:0/2995157983 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3d80443f0 con 0x7fb3e00cd670 2024-03-24T00:39:33.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.499+0000 7fb3e8419640 1 -- 172.21.15.67:0/2995157983 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3e00b4d40 con 0x7fb3e00cd670 2024-03-24T00:39:33.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.499+0000 7fb3deffd640 1 -- 172.21.15.67:0/2995157983 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d80447e0 con 0x7fb3e00cd670 2024-03-24T00:39:33.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.499+0000 7fb3e8419640 1 -- 172.21.15.67:0/2995157983 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3e00b5300 con 0x7fb3e00cd670 2024-03-24T00:39:33.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.500+0000 7fb3e8419640 1 -- 172.21.15.67:0/2995157983 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3e00cf4a0 con 0x7fb3e00cd670 2024-03-24T00:39:33.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.500+0000 7fb3deffd640 1 -- 172.21.15.67:0/2995157983 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb3d8038020 con 0x7fb3e00cd670 2024-03-24T00:39:33.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.501+0000 7fb3deffd640 1 --2- 172.21.15.67:0/2995157983 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb3cc06bbd0 0x7fb3cc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:33.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.501+0000 7fb3deffd640 1 -- 172.21.15.67:0/2995157983 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fb3d80ace50 con 0x7fb3e00cd670 2024-03-24T00:39:33.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.501+0000 7fb3e61b5640 1 --2- 172.21.15.67:0/2995157983 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb3cc06bbd0 0x7fb3cc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:33.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.501+0000 7fb3e61b5640 1 --2- 172.21.15.67:0/2995157983 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb3cc06bbd0 0x7fb3cc06e090 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb3d0011340 tx=0x7fb3d000b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:33.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.505+0000 7fb3deffd640 1 -- 172.21.15.67:0/2995157983 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb3e00cf4a0 con 0x7fb3e00cd670 2024-03-24T00:39:33.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.687+0000 7fb3e8419640 1 -- 172.21.15.67:0/2995157983 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb3e004f400 con 0x7fb3e00cd670 2024-03-24T00:39:33.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.688+0000 7fb3deffd640 1 -- 172.21.15.67:0/2995157983 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18843 (secure 0 0 0) 0x7fb3d803f070 con 0x7fb3e00cd670 2024-03-24T00:39:33.688 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:39:33.688 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":27,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:39:32.904729+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:38:50.077085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:38:51.095454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T00:38:54.807117+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.14","mappings":[{"from":4,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:39:33.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.690+0000 7fb3e8419640 1 -- 172.21.15.67:0/2995157983 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb3cc06bbd0 msgr2=0x7fb3cc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:33.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.690+0000 7fb3e8419640 1 --2- 172.21.15.67:0/2995157983 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb3cc06bbd0 0x7fb3cc06e090 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb3d0011340 tx=0x7fb3d000b040 comp rx=0 tx=0).stop 2024-03-24T00:39:33.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.690+0000 7fb3e8419640 1 -- 172.21.15.67:0/2995157983 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3e00cd670 msgr2=0x7fb3e014a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:33.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.690+0000 7fb3e8419640 1 --2- 172.21.15.67:0/2995157983 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3e00cd670 0x7fb3e014a6e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb3d802b950 tx=0x7fb3d80046c0 comp rx=0 tx=0).stop 2024-03-24T00:39:33.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.690+0000 7fb3e8419640 1 -- 172.21.15.67:0/2995157983 shutdown_connections 2024-03-24T00:39:33.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.690+0000 7fb3e8419640 1 --2- 172.21.15.67:0/2995157983 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb3cc06bbd0 0x7fb3cc06e090 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:33.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.690+0000 7fb3e8419640 1 --2- 172.21.15.67:0/2995157983 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3e00cd670 0x7fb3e014a6e0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:33.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.690+0000 7fb3e8419640 1 --2- 172.21.15.67:0/2995157983 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3e00b21f0 0x7fb3e014a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:33.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.690+0000 7fb3e8419640 1 --2- 172.21.15.67:0/2995157983 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb3e006b040 0x7fb3e00c4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:33.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.690+0000 7fb3e8419640 1 -- 172.21.15.67:0/2995157983 >> 172.21.15.67:0/2995157983 conn(0x7fb3e0099a30 msgr2=0x7fb3e00d09a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:33.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.690+0000 7fb3e8419640 1 -- 172.21.15.67:0/2995157983 shutdown_connections 2024-03-24T00:39:33.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.690+0000 7fb3e8419640 1 -- 172.21.15.67:0/2995157983 wait complete. 2024-03-24T00:39:33.701 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:38:39.364950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:38:50.077085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:38:51.095454+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T00:38:54.807117+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T00:39:33.701 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:39:33.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.806+0000 7fa690bc4640 1 Processor -- start 2024-03-24T00:39:33.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.806+0000 7fa690bc4640 1 -- start start 2024-03-24T00:39:33.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.807+0000 7fa690bc4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa68c15e130 0x7fa68c0ce610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:33.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.807+0000 7fa690bc4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa68c0d0640 0x7fa68c0d0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:33.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.807+0000 7fa690bc4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa68c0b57a0 0x7fa68c0b5c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:33.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.807+0000 7fa690bc4640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa68c0ceb50 con 0x7fa68c15e130 2024-03-24T00:39:33.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.807+0000 7fa690bc4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa68c0d0fe0 con 0x7fa68c0d0640 2024-03-24T00:39:33.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.807+0000 7fa690bc4640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa68c0cf570 con 0x7fa68c0b57a0 2024-03-24T00:39:33.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.807+0000 7fa68a59c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa68c15e130 0x7fa68c0ce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:33.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.807+0000 7fa68a59c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa68c15e130 0x7fa68c0ce610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50138/0 (socket says 172.21.15.67:50138) 2024-03-24T00:39:33.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.807+0000 7fa68a59c640 1 -- 172.21.15.67:0/398652479 learned_addr learned my addr 172.21.15.67:0/398652479 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:33.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.807+0000 7fa68ad9d640 1 --2- 172.21.15.67:0/398652479 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa68c0b57a0 0x7fa68c0b5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:33.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.807+0000 7fa689d9b640 1 --2- 172.21.15.67:0/398652479 >> v2:172.21.15.72:3300/0 conn(0x7fa68c0d0640 0x7fa68c0d0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:33.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.807+0000 7fa68a59c640 1 -- 172.21.15.67:0/398652479 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa68c0b57a0 msgr2=0x7fa68c0b5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:33.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.807+0000 7fa68a59c640 1 --2- 172.21.15.67:0/398652479 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa68c0b57a0 0x7fa68c0b5c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:33.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.807+0000 7fa68a59c640 1 -- 172.21.15.67:0/398652479 >> v2:172.21.15.72:3300/0 conn(0x7fa68c0d0640 msgr2=0x7fa68c0d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:33.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.807+0000 7fa68a59c640 1 --2- 172.21.15.67:0/398652479 >> v2:172.21.15.72:3300/0 conn(0x7fa68c0d0640 0x7fa68c0d0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:33.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.807+0000 7fa68a59c640 1 -- 172.21.15.67:0/398652479 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa68c0cecf0 con 0x7fa68c15e130 2024-03-24T00:39:33.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.808+0000 7fa68a59c640 1 --2- 172.21.15.67:0/398652479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa68c15e130 0x7fa68c0ce610 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa68c161bd0 tx=0x7fa68402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7947660e97deb056 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:33.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.808+0000 7fa68959a640 1 -- 172.21.15.67:0/398652479 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa684035020 con 0x7fa68c15e130 2024-03-24T00:39:33.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.808+0000 7fa68959a640 1 -- 172.21.15.67:0/398652479 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa684031870 con 0x7fa68c15e130 2024-03-24T00:39:33.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.808+0000 7fa68959a640 1 -- 172.21.15.67:0/398652479 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa684031c40 con 0x7fa68c15e130 2024-03-24T00:39:33.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.809+0000 7fa690bc4640 1 -- 172.21.15.67:0/398652479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa68c15e130 msgr2=0x7fa68c0ce610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:33.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.809+0000 7fa690bc4640 1 --2- 172.21.15.67:0/398652479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa68c15e130 0x7fa68c0ce610 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa68c161bd0 tx=0x7fa68402b910 comp rx=0 tx=0).stop 2024-03-24T00:39:33.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.809+0000 7fa690bc4640 1 -- 172.21.15.67:0/398652479 shutdown_connections 2024-03-24T00:39:33.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.809+0000 7fa690bc4640 1 --2- 172.21.15.67:0/398652479 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa68c0b57a0 0x7fa68c0b5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:33.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.809+0000 7fa690bc4640 1 --2- 172.21.15.67:0/398652479 >> v2:172.21.15.72:3300/0 conn(0x7fa68c0d0640 0x7fa68c0d0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:33.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.809+0000 7fa690bc4640 1 --2- 172.21.15.67:0/398652479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa68c15e130 0x7fa68c0ce610 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:33.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.809+0000 7fa690bc4640 1 -- 172.21.15.67:0/398652479 >> 172.21.15.67:0/398652479 conn(0x7fa68c06c8a0 msgr2=0x7fa68c06ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:33.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.809+0000 7fa690bc4640 1 -- 172.21.15.67:0/398652479 shutdown_connections 2024-03-24T00:39:33.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.809+0000 7fa690bc4640 1 -- 172.21.15.67:0/398652479 wait complete. 2024-03-24T00:39:33.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.810+0000 7fa690bc4640 1 Processor -- start 2024-03-24T00:39:33.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.810+0000 7fa690bc4640 1 -- start start 2024-03-24T00:39:33.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.811+0000 7fa690bc4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa68c0b57a0 0x7fa68c0cc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:33.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.811+0000 7fa690bc4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa68c0d0640 0x7fa68c1472c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:33.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.811+0000 7fa690bc4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa68c15e130 0x7fa68c147800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:33.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.811+0000 7fa690bc4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa68c0cf290 con 0x7fa68c15e130 2024-03-24T00:39:33.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.811+0000 7fa690bc4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa68c0cefb0 con 0x7fa68c0d0640 2024-03-24T00:39:33.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.811+0000 7fa690bc4640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa68c15e510 con 0x7fa68c0b57a0 2024-03-24T00:39:33.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.811+0000 7fa68ad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa68c15e130 0x7fa68c147800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:33.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.811+0000 7fa68ad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa68c15e130 0x7fa68c147800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50142/0 (socket says 172.21.15.67:50142) 2024-03-24T00:39:33.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.811+0000 7fa68ad9d640 1 -- 172.21.15.67:0/999871569 learned_addr learned my addr 172.21.15.67:0/999871569 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:33.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.811+0000 7fa68a59c640 1 --2- 172.21.15.67:0/999871569 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa68c0b57a0 0x7fa68c0cc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:33.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.811+0000 7fa689d9b640 1 --2- 172.21.15.67:0/999871569 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa68c0d0640 0x7fa68c1472c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:33.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.811+0000 7fa68ad9d640 1 -- 172.21.15.67:0/999871569 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa68c0b57a0 msgr2=0x7fa68c0cc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:33.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.811+0000 7fa68ad9d640 1 --2- 172.21.15.67:0/999871569 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa68c0b57a0 0x7fa68c0cc0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:33.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.811+0000 7fa68ad9d640 1 -- 172.21.15.67:0/999871569 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa68c0d0640 msgr2=0x7fa68c1472c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:33.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.811+0000 7fa68ad9d640 1 --2- 172.21.15.67:0/999871569 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa68c0d0640 0x7fa68c1472c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:33.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.811+0000 7fa68ad9d640 1 -- 172.21.15.67:0/999871569 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa68c0c7f10 con 0x7fa68c15e130 2024-03-24T00:39:33.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.812+0000 7fa68ad9d640 1 --2- 172.21.15.67:0/999871569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa68c15e130 0x7fa68c147800 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa67c00efc0 tx=0x7fa67c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:33.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.812+0000 7fa682ffd640 1 -- 172.21.15.67:0/999871569 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa67c018020 con 0x7fa68c15e130 2024-03-24T00:39:33.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.812+0000 7fa682ffd640 1 -- 172.21.15.67:0/999871569 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa67c014480 con 0x7fa68c15e130 2024-03-24T00:39:33.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.812+0000 7fa682ffd640 1 -- 172.21.15.67:0/999871569 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa67c014870 con 0x7fa68c15e130 2024-03-24T00:39:33.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.812+0000 7fa690bc4640 1 -- 172.21.15.67:0/999871569 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa68c0c7c30 con 0x7fa68c15e130 2024-03-24T00:39:33.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.812+0000 7fa690bc4640 1 -- 172.21.15.67:0/999871569 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa68c0ccfb0 con 0x7fa68c15e130 2024-03-24T00:39:33.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.813+0000 7fa690bc4640 1 -- 172.21.15.67:0/999871569 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa68c0ccb00 con 0x7fa68c15e130 2024-03-24T00:39:33.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.813+0000 7fa682ffd640 1 -- 172.21.15.67:0/999871569 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa67c005d10 con 0x7fa68c15e130 2024-03-24T00:39:33.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.814+0000 7fa682ffd640 1 --2- 172.21.15.67:0/999871569 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa66c06bc00 0x7fa66c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:33.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.814+0000 7fa682ffd640 1 -- 172.21.15.67:0/999871569 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fa67c08b740 con 0x7fa68c15e130 2024-03-24T00:39:33.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.814+0000 7fa68a59c640 1 --2- 172.21.15.67:0/999871569 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa66c06bc00 0x7fa66c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:33.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.814+0000 7fa68a59c640 1 --2- 172.21.15.67:0/999871569 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa66c06bc00 0x7fa66c06e0c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa684036000 tx=0x7fa68403c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:33.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.818+0000 7fa682ffd640 1 -- 172.21.15.67:0/999871569 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa67c022070 con 0x7fa68c15e130 2024-03-24T00:39:33.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.999+0000 7fa690bc4640 1 -- 172.21.15.67:0/999871569 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa68c04f400 con 0x7fa68c15e130 2024-03-24T00:39:33.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:33.999+0000 7fa682ffd640 1 -- 172.21.15.67:0/999871569 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa68c0c7f10 con 0x7fa68c15e130 2024-03-24T00:39:33.999 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-24T00:39:34.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.001+0000 7fa690bc4640 1 -- 172.21.15.67:0/999871569 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa66c06bc00 msgr2=0x7fa66c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:34.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.001+0000 7fa690bc4640 1 --2- 172.21.15.67:0/999871569 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa66c06bc00 0x7fa66c06e0c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa684036000 tx=0x7fa68403c040 comp rx=0 tx=0).stop 2024-03-24T00:39:34.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.001+0000 7fa690bc4640 1 -- 172.21.15.67:0/999871569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa68c15e130 msgr2=0x7fa68c147800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:34.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.001+0000 7fa690bc4640 1 --2- 172.21.15.67:0/999871569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa68c15e130 0x7fa68c147800 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa67c00efc0 tx=0x7fa67c00c490 comp rx=0 tx=0).stop 2024-03-24T00:39:34.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.001+0000 7fa690bc4640 1 -- 172.21.15.67:0/999871569 shutdown_connections 2024-03-24T00:39:34.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.001+0000 7fa690bc4640 1 --2- 172.21.15.67:0/999871569 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa66c06bc00 0x7fa66c06e0c0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:34.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.001+0000 7fa690bc4640 1 --2- 172.21.15.67:0/999871569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa68c15e130 0x7fa68c147800 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:34.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.001+0000 7fa690bc4640 1 --2- 172.21.15.67:0/999871569 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa68c0d0640 0x7fa68c1472c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:34.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.001+0000 7fa690bc4640 1 --2- 172.21.15.67:0/999871569 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa68c0b57a0 0x7fa68c0cc0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:34.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.001+0000 7fa690bc4640 1 -- 172.21.15.67:0/999871569 >> 172.21.15.67:0/999871569 conn(0x7fa68c06c8a0 msgr2=0x7fa68c0b3760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:34.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.001+0000 7fa690bc4640 1 -- 172.21.15.67:0/999871569 shutdown_connections 2024-03-24T00:39:34.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.001+0000 7fa690bc4640 1 -- 172.21.15.67:0/999871569 wait complete. 2024-03-24T00:39:34.012 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:39:34.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.116+0000 7f6b5e989640 1 Processor -- start 2024-03-24T00:39:34.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.116+0000 7f6b5e989640 1 -- start start 2024-03-24T00:39:34.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.117+0000 7f6b5e989640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6b58161370 0x7f6b58161ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:34.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.117+0000 7f6b5e989640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6b580b9690 0x7f6b580ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:34.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.117+0000 7f6b5e989640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b58161ff0 con 0x7f6b580cef30 2024-03-24T00:39:34.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.117+0000 7f6b5e989640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b580b6e10 con 0x7f6b580b9690 2024-03-24T00:39:34.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.117+0000 7f6b5e989640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b580d2190 con 0x7f6b58161370 2024-03-24T00:39:34.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.117+0000 7f6b577fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6b580b9690 0x7f6b580ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:34.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.117+0000 7f6b577fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6b580b9690 0x7f6b580ce9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50156/0 (socket says 172.21.15.67:50156) 2024-03-24T00:39:34.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.117+0000 7f6b577fe640 1 -- 172.21.15.67:0/2769044703 learned_addr learned my addr 172.21.15.67:0/2769044703 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:34.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.117+0000 7f6b57fff640 1 --2- 172.21.15.67:0/2769044703 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6b58161370 0x7f6b58161ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:34.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.117+0000 7f6b577fe640 1 -- 172.21.15.67:0/2769044703 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6b58161370 msgr2=0x7f6b58161ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:34.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.117+0000 7f6b577fe640 1 --2- 172.21.15.67:0/2769044703 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6b58161370 0x7f6b58161ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:34.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.117+0000 7f6b577fe640 1 -- 172.21.15.67:0/2769044703 >> v1:172.21.15.72:6789/0 conn(0x7f6b580cef30 legacy=0x7f6b580b5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:34.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.117+0000 7f6b56ffd640 1 -- 172.21.15.67:0/2769044703 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3309262488 0 0) 0x7f6b58161ff0 con 0x7f6b580cef30 2024-03-24T00:39:34.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.117+0000 7f6b577fe640 1 -- 172.21.15.67:0/2769044703 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b58162290 con 0x7f6b580b9690 2024-03-24T00:39:34.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.118+0000 7f6b577fe640 1 --2- 172.21.15.67:0/2769044703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b580b9690 0x7f6b580ce9f0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6b480098a0 tx=0x7f6b4802ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fdab720a0f1420e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:34.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.118+0000 7f6b56ffd640 1 -- 172.21.15.67:0/2769044703 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b4802f830 con 0x7f6b580b9690 2024-03-24T00:39:34.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.118+0000 7f6b56ffd640 1 -- 172.21.15.67:0/2769044703 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b4802f9d0 con 0x7f6b580b9690 2024-03-24T00:39:34.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.118+0000 7f6b56ffd640 1 -- 172.21.15.67:0/2769044703 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b4802fda0 con 0x7f6b580b9690 2024-03-24T00:39:34.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.118+0000 7f6b5e989640 1 -- 172.21.15.67:0/2769044703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b580b9690 msgr2=0x7f6b580ce9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:34.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.118+0000 7f6b5e989640 1 --2- 172.21.15.67:0/2769044703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b580b9690 0x7f6b580ce9f0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6b480098a0 tx=0x7f6b4802ba10 comp rx=0 tx=0).stop 2024-03-24T00:39:34.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.119+0000 7f6b5e989640 1 -- 172.21.15.67:0/2769044703 shutdown_connections 2024-03-24T00:39:34.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.119+0000 7f6b5e989640 1 --2- 172.21.15.67:0/2769044703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b580b9690 0x7f6b580ce9f0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:34.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.119+0000 7f6b5e989640 1 --2- 172.21.15.67:0/2769044703 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6b58161370 0x7f6b58161ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:34.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.119+0000 7f6b5e989640 1 -- 172.21.15.67:0/2769044703 >> 172.21.15.67:0/2769044703 conn(0x7f6b5806c880 msgr2=0x7f6b5806cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:34.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.119+0000 7f6b5e989640 1 -- 172.21.15.67:0/2769044703 shutdown_connections 2024-03-24T00:39:34.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.119+0000 7f6b5e989640 1 -- 172.21.15.67:0/2769044703 wait complete. 2024-03-24T00:39:34.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.120+0000 7f6b5e989640 1 Processor -- start 2024-03-24T00:39:34.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.120+0000 7f6b5e989640 1 -- start start 2024-03-24T00:39:34.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.121+0000 7f6b5e989640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6b580b9690 0x7f6b580cd4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:34.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.121+0000 7f6b5e989640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b580cef30 0x7f6b58151940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:34.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.121+0000 7f6b5e989640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b58161370 0x7f6b580cbbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:34.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.121+0000 7f6b5e989640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b58161750 con 0x7f6b580cef30 2024-03-24T00:39:34.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.121+0000 7f6b5e989640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b58162850 con 0x7f6b58161370 2024-03-24T00:39:34.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.121+0000 7f6b5e989640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b580d3630 con 0x7f6b580b9690 2024-03-24T00:39:34.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.121+0000 7f6b577fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b580cef30 0x7f6b58151940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:34.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.121+0000 7f6b577fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b580cef30 0x7f6b58151940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50166/0 (socket says 172.21.15.67:50166) 2024-03-24T00:39:34.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.121+0000 7f6b577fe640 1 -- 172.21.15.67:0/2014173262 learned_addr learned my addr 172.21.15.67:0/2014173262 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:34.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.121+0000 7f6b5cf26640 1 --2- 172.21.15.67:0/2014173262 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b58161370 0x7f6b580cbbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:34.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.121+0000 7f6b57fff640 1 --2- 172.21.15.67:0/2014173262 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6b580b9690 0x7f6b580cd4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:34.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.121+0000 7f6b577fe640 1 -- 172.21.15.67:0/2014173262 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6b580b9690 msgr2=0x7f6b580cd4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:34.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.121+0000 7f6b577fe640 1 --2- 172.21.15.67:0/2014173262 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6b580b9690 0x7f6b580cd4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:34.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.121+0000 7f6b577fe640 1 -- 172.21.15.67:0/2014173262 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b58161370 msgr2=0x7f6b580cbbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:34.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.121+0000 7f6b577fe640 1 --2- 172.21.15.67:0/2014173262 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b58161370 0x7f6b580cbbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:34.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.121+0000 7f6b577fe640 1 -- 172.21.15.67:0/2014173262 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b580cda00 con 0x7f6b580cef30 2024-03-24T00:39:34.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.121+0000 7f6b577fe640 1 --2- 172.21.15.67:0/2014173262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b580cef30 0x7f6b58151940 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6b48000c00 tx=0x7f6b48004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:34.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.121+0000 7f6b54ff9640 1 -- 172.21.15.67:0/2014173262 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b48037c40 con 0x7f6b580cef30 2024-03-24T00:39:34.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.121+0000 7f6b54ff9640 1 -- 172.21.15.67:0/2014173262 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b48046070 con 0x7f6b580cef30 2024-03-24T00:39:34.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.121+0000 7f6b5e989640 1 -- 172.21.15.67:0/2014173262 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b580cdc80 con 0x7f6b580cef30 2024-03-24T00:39:34.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.121+0000 7f6b54ff9640 1 -- 172.21.15.67:0/2014173262 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b48041400 con 0x7f6b580cef30 2024-03-24T00:39:34.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.122+0000 7f6b5e989640 1 -- 172.21.15.67:0/2014173262 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b580cb0d0 con 0x7f6b580cef30 2024-03-24T00:39:34.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.122+0000 7f6b5e989640 1 -- 172.21.15.67:0/2014173262 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b5804f400 con 0x7f6b580cef30 2024-03-24T00:39:34.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.123+0000 7f6b54ff9640 1 -- 172.21.15.67:0/2014173262 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6b48040070 con 0x7f6b580cef30 2024-03-24T00:39:34.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.123+0000 7f6b54ff9640 1 --2- 172.21.15.67:0/2014173262 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6b4406bbd0 0x7f6b4406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:34.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.123+0000 7f6b54ff9640 1 -- 172.21.15.67:0/2014173262 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f6b480ad210 con 0x7f6b580cef30 2024-03-24T00:39:34.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.123+0000 7f6b57fff640 1 --2- 172.21.15.67:0/2014173262 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6b4406bbd0 0x7f6b4406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:34.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.124+0000 7f6b57fff640 1 --2- 172.21.15.67:0/2014173262 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6b4406bbd0 0x7f6b4406e090 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c009aa0 tx=0x7f6b4c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:34.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.128+0000 7f6b54ff9640 1 -- 172.21.15.67:0/2014173262 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6b580cda00 con 0x7f6b580cef30 2024-03-24T00:39:34.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.308+0000 7f6b5e989640 1 -- 172.21.15.67:0/2014173262 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6b580d1aa0 con 0x7f6b580cef30 2024-03-24T00:39:34.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.309+0000 7f6b54ff9640 1 -- 172.21.15.67:0/2014173262 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6b48033030 con 0x7f6b580cef30 2024-03-24T00:39:34.309 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-24T00:39:34.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.311+0000 7f6b5e989640 1 -- 172.21.15.67:0/2014173262 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6b4406bbd0 msgr2=0x7f6b4406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:34.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.311+0000 7f6b5e989640 1 --2- 172.21.15.67:0/2014173262 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6b4406bbd0 0x7f6b4406e090 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c009aa0 tx=0x7f6b4c009190 comp rx=0 tx=0).stop 2024-03-24T00:39:34.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.311+0000 7f6b5e989640 1 -- 172.21.15.67:0/2014173262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b580cef30 msgr2=0x7f6b58151940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:34.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.311+0000 7f6b5e989640 1 --2- 172.21.15.67:0/2014173262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b580cef30 0x7f6b58151940 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6b48000c00 tx=0x7f6b48004730 comp rx=0 tx=0).stop 2024-03-24T00:39:34.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.311+0000 7f6b5e989640 1 -- 172.21.15.67:0/2014173262 shutdown_connections 2024-03-24T00:39:34.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.311+0000 7f6b5e989640 1 --2- 172.21.15.67:0/2014173262 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6b4406bbd0 0x7f6b4406e090 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:34.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.311+0000 7f6b5e989640 1 --2- 172.21.15.67:0/2014173262 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b58161370 0x7f6b580cbbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:34.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.311+0000 7f6b5e989640 1 --2- 172.21.15.67:0/2014173262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b580cef30 0x7f6b58151940 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:34.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.311+0000 7f6b5e989640 1 --2- 172.21.15.67:0/2014173262 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6b580b9690 0x7f6b580cd4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:34.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.311+0000 7f6b5e989640 1 -- 172.21.15.67:0/2014173262 >> 172.21.15.67:0/2014173262 conn(0x7f6b5806c880 msgr2=0x7f6b5815c620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:34.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.311+0000 7f6b5e989640 1 -- 172.21.15.67:0/2014173262 shutdown_connections 2024-03-24T00:39:34.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.311+0000 7f6b5e989640 1 -- 172.21.15.67:0/2014173262 wait complete. 2024-03-24T00:39:34.321 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:39:34.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.426+0000 7fc711e7e640 1 Processor -- start 2024-03-24T00:39:34.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.426+0000 7fc711e7e640 1 -- start start 2024-03-24T00:39:34.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.427+0000 7fc711e7e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc70c15c2a0 0x7fc70c160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:34.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.427+0000 7fc711e7e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc70c06dcf0 con 0x7fc70c145540 2024-03-24T00:39:34.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.427+0000 7fc711e7e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc70c06dfd0 con 0x7fc70c071af0 2024-03-24T00:39:34.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.427+0000 7fc711e7e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc70c06e620 con 0x7fc70c15c2a0 2024-03-24T00:39:34.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.427+0000 7fc70b7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc70c145540 0x7fc70c0cef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41450/0 (socket says 172.21.15.67:41450) 2024-03-24T00:39:34.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.427+0000 7fc70b7fe640 1 -- 172.21.15.67:0/1191554744 learned_addr learned my addr 172.21.15.67:0/1191554744 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:34.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.427+0000 7fc70affd640 1 --2- 172.21.15.67:0/1191554744 >> v2:172.21.15.72:3300/0 conn(0x7fc70c15c2a0 0x7fc70c160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:34.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.427+0000 7fc70a7fc640 1 -- 172.21.15.67:0/1191554744 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4061172933 0 0) 0x7fc70c06dcf0 con 0x7fc70c145540 2024-03-24T00:39:34.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.427+0000 7fc70a7fc640 1 -- 172.21.15.67:0/1191554744 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6f4003520 con 0x7fc70c145540 2024-03-24T00:39:34.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.427+0000 7fc70affd640 1 -- 172.21.15.67:0/1191554744 >> v1:172.21.15.72:6789/0 conn(0x7fc70c071af0 legacy=0x7fc70c071ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:34.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.427+0000 7fc70affd640 1 -- 172.21.15.67:0/1191554744 >> v1:172.21.15.67:6789/0 conn(0x7fc70c145540 legacy=0x7fc70c0cef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:34.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.427+0000 7fc70affd640 1 -- 172.21.15.67:0/1191554744 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc70c06e2b0 con 0x7fc70c15c2a0 2024-03-24T00:39:34.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.428+0000 7fc70affd640 1 --2- 172.21.15.67:0/1191554744 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc70c15c2a0 0x7fc70c160420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc7000097a0 tx=0x7fc70002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a0124a97a19ae99 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:34.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.428+0000 7fc70a7fc640 1 -- 172.21.15.67:0/1191554744 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc700034070 con 0x7fc70c15c2a0 2024-03-24T00:39:34.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.428+0000 7fc70a7fc640 1 -- 172.21.15.67:0/1191554744 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc700030610 con 0x7fc70c15c2a0 2024-03-24T00:39:34.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.428+0000 7fc70a7fc640 1 -- 172.21.15.67:0/1191554744 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7000309e0 con 0x7fc70c15c2a0 2024-03-24T00:39:34.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.429+0000 7fc711e7e640 1 -- 172.21.15.67:0/1191554744 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc70c15c2a0 msgr2=0x7fc70c160420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:34.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.429+0000 7fc711e7e640 1 --2- 172.21.15.67:0/1191554744 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc70c15c2a0 0x7fc70c160420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc7000097a0 tx=0x7fc70002b960 comp rx=0 tx=0).stop 2024-03-24T00:39:34.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.429+0000 7fc711e7e640 1 -- 172.21.15.67:0/1191554744 shutdown_connections 2024-03-24T00:39:34.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.429+0000 7fc711e7e640 1 --2- 172.21.15.67:0/1191554744 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc70c15c2a0 0x7fc70c160420 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:34.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.429+0000 7fc711e7e640 1 -- 172.21.15.67:0/1191554744 >> 172.21.15.67:0/1191554744 conn(0x7fc70c0cf9e0 msgr2=0x7fc70c0cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:34.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.429+0000 7fc711e7e640 1 -- 172.21.15.67:0/1191554744 shutdown_connections 2024-03-24T00:39:34.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.429+0000 7fc711e7e640 1 -- 172.21.15.67:0/1191554744 wait complete. 2024-03-24T00:39:34.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.430+0000 7fc711e7e640 1 Processor -- start 2024-03-24T00:39:34.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.430+0000 7fc711e7e640 1 -- start start 2024-03-24T00:39:34.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.431+0000 7fc711e7e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc70c071af0 0x7fc70c158940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:34.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.431+0000 7fc711e7e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc70c145540 0x7fc70c159040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:34.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.431+0000 7fc711e7e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc70c15c2a0 0x7fc70c162880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:34.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.431+0000 7fc711e7e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc70c166da0 con 0x7fc70c15c2a0 2024-03-24T00:39:34.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.431+0000 7fc711e7e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc70c166c20 con 0x7fc70c145540 2024-03-24T00:39:34.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.431+0000 7fc711e7e640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc70c166f20 con 0x7fc70c071af0 2024-03-24T00:39:34.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.431+0000 7fc70bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc70c15c2a0 0x7fc70c162880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:34.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.431+0000 7fc70bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc70c15c2a0 0x7fc70c162880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50174/0 (socket says 172.21.15.67:50174) 2024-03-24T00:39:34.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.431+0000 7fc70bfff640 1 -- 172.21.15.67:0/241676718 learned_addr learned my addr 172.21.15.67:0/241676718 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:34.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.431+0000 7fc70affd640 1 --2- 172.21.15.67:0/241676718 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc70c145540 0x7fc70c159040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:34.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.431+0000 7fc70b7fe640 1 --2- 172.21.15.67:0/241676718 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc70c071af0 0x7fc70c158940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:34.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.431+0000 7fc70bfff640 1 -- 172.21.15.67:0/241676718 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc70c071af0 msgr2=0x7fc70c158940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:34.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.431+0000 7fc70bfff640 1 --2- 172.21.15.67:0/241676718 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc70c071af0 0x7fc70c158940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:34.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.431+0000 7fc70bfff640 1 -- 172.21.15.67:0/241676718 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc70c145540 msgr2=0x7fc70c159040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:34.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.431+0000 7fc70bfff640 1 --2- 172.21.15.67:0/241676718 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc70c145540 0x7fc70c159040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:34.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.431+0000 7fc70bfff640 1 -- 172.21.15.67:0/241676718 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc70c1599b0 con 0x7fc70c15c2a0 2024-03-24T00:39:34.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.431+0000 7fc70bfff640 1 --2- 172.21.15.67:0/241676718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc70c15c2a0 0x7fc70c162880 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fc70400c7d0 tx=0x7fc70400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:34.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.431+0000 7fc6f3fff640 1 -- 172.21.15.67:0/241676718 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc704003b70 con 0x7fc70c15c2a0 2024-03-24T00:39:34.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.431+0000 7fc6f3fff640 1 -- 172.21.15.67:0/241676718 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc704003d30 con 0x7fc70c15c2a0 2024-03-24T00:39:34.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.431+0000 7fc711e7e640 1 -- 172.21.15.67:0/241676718 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc70c159c90 con 0x7fc70c15c2a0 2024-03-24T00:39:34.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.431+0000 7fc6f3fff640 1 -- 172.21.15.67:0/241676718 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70400aa90 con 0x7fc70c15c2a0 2024-03-24T00:39:34.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.431+0000 7fc711e7e640 1 -- 172.21.15.67:0/241676718 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc70c15a250 con 0x7fc70c15c2a0 2024-03-24T00:39:34.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.433+0000 7fc711e7e640 1 -- 172.21.15.67:0/241676718 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc70c04f400 con 0x7fc70c15c2a0 2024-03-24T00:39:34.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.433+0000 7fc6f3fff640 1 -- 172.21.15.67:0/241676718 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc70400ac30 con 0x7fc70c15c2a0 2024-03-24T00:39:34.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.433+0000 7fc6f3fff640 1 --2- 172.21.15.67:0/241676718 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc6f806bbd0 0x7fc6f806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:34.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.433+0000 7fc6f3fff640 1 -- 172.21.15.67:0/241676718 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fc70408acb0 con 0x7fc70c15c2a0 2024-03-24T00:39:34.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.433+0000 7fc70b7fe640 1 --2- 172.21.15.67:0/241676718 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc6f806bbd0 0x7fc6f806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:34.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.434+0000 7fc70b7fe640 1 --2- 172.21.15.67:0/241676718 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc6f806bbd0 0x7fc6f806e090 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc0011e0 tx=0x7fc6fc007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:34.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.438+0000 7fc6f3fff640 1 -- 172.21.15.67:0/241676718 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc704058d50 con 0x7fc70c15c2a0 2024-03-24T00:39:34.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.620+0000 7fc711e7e640 1 -- 172.21.15.67:0/241676718 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc70c0d0740 con 0x7fc70c15c2a0 2024-03-24T00:39:34.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.621+0000 7fc6f3fff640 1 -- 172.21.15.67:0/241676718 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc70405cdf0 con 0x7fc70c15c2a0 2024-03-24T00:39:34.621 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-03-24T00:39:34.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.623+0000 7fc711e7e640 1 -- 172.21.15.67:0/241676718 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc6f806bbd0 msgr2=0x7fc6f806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:34.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.623+0000 7fc711e7e640 1 --2- 172.21.15.67:0/241676718 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc6f806bbd0 0x7fc6f806e090 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc0011e0 tx=0x7fc6fc007d30 comp rx=0 tx=0).stop 2024-03-24T00:39:34.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.623+0000 7fc711e7e640 1 -- 172.21.15.67:0/241676718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc70c15c2a0 msgr2=0x7fc70c162880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:34.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.623+0000 7fc711e7e640 1 --2- 172.21.15.67:0/241676718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc70c15c2a0 0x7fc70c162880 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fc70400c7d0 tx=0x7fc70400cc90 comp rx=0 tx=0).stop 2024-03-24T00:39:34.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.623+0000 7fc711e7e640 1 -- 172.21.15.67:0/241676718 shutdown_connections 2024-03-24T00:39:34.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.623+0000 7fc711e7e640 1 --2- 172.21.15.67:0/241676718 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc6f806bbd0 0x7fc6f806e090 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:34.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.623+0000 7fc711e7e640 1 --2- 172.21.15.67:0/241676718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc70c15c2a0 0x7fc70c162880 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:34.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.623+0000 7fc711e7e640 1 --2- 172.21.15.67:0/241676718 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc70c145540 0x7fc70c159040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:34.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.623+0000 7fc711e7e640 1 --2- 172.21.15.67:0/241676718 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc70c071af0 0x7fc70c158940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:34.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.623+0000 7fc711e7e640 1 -- 172.21.15.67:0/241676718 >> 172.21.15.67:0/241676718 conn(0x7fc70c0cf9e0 msgr2=0x7fc70c15e400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:34.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.623+0000 7fc711e7e640 1 -- 172.21.15.67:0/241676718 shutdown_connections 2024-03-24T00:39:34.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.623+0000 7fc711e7e640 1 -- 172.21.15.67:0/241676718 wait complete. 2024-03-24T00:39:34.633 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T00:39:34.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.738+0000 7f026aae8640 1 Processor -- start 2024-03-24T00:39:34.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.738+0000 7f026aae8640 1 -- start start 2024-03-24T00:39:34.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.739+0000 7f026aae8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0264161370 0x7f0264161ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:34.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.739+0000 7f026aae8640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f02640cf100 0x7f02640d2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:34.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.739+0000 7f026aae8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0264161ff0 con 0x7f02640b9690 2024-03-24T00:39:34.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.739+0000 7f026aae8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02640b6e10 con 0x7f0264161370 2024-03-24T00:39:34.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.739+0000 7f026aae8640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02640d2190 con 0x7f02640cf100 2024-03-24T00:39:34.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.739+0000 7f0268884640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0264161370 0x7f0264161ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:34.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.739+0000 7f0263fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f02640b9690 0x7f02640ce9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41458/0 (socket says 172.21.15.67:41458) 2024-03-24T00:39:34.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.739+0000 7f0263fff640 1 -- 172.21.15.67:0/1548681925 learned_addr learned my addr 172.21.15.67:0/1548681925 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:34.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.739+0000 7f0269085640 1 --2- 172.21.15.67:0/1548681925 >> v2:172.21.15.72:3300/0 conn(0x7f02640cf100 0x7f02640d2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:34.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.740+0000 7f0268884640 1 -- 172.21.15.67:0/1548681925 >> v2:172.21.15.72:3300/0 conn(0x7f02640cf100 msgr2=0x7f02640d2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:34.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.740+0000 7f0268884640 1 --2- 172.21.15.67:0/1548681925 >> v2:172.21.15.72:3300/0 conn(0x7f02640cf100 0x7f02640d2620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:34.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.740+0000 7f0268884640 1 -- 172.21.15.67:0/1548681925 >> v1:172.21.15.67:6789/0 conn(0x7f02640b9690 legacy=0x7f02640ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:34.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.740+0000 7f0268884640 1 -- 172.21.15.67:0/1548681925 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0264162290 con 0x7f0264161370 2024-03-24T00:39:34.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.740+0000 7f02637fe640 1 -- 172.21.15.67:0/1548681925 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 7407972 0 0) 0x7f0264161ff0 con 0x7f02640b9690 2024-03-24T00:39:34.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.740+0000 7f0268884640 1 --2- 172.21.15.67:0/1548681925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0264161370 0x7f0264161ab0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f02640ce300 tx=0x7f025c02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bbbb20170598d37 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:34.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.740+0000 7f02637fe640 1 -- 172.21.15.67:0/1548681925 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f025c039070 con 0x7f0264161370 2024-03-24T00:39:34.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.741+0000 7f02637fe640 1 -- 172.21.15.67:0/1548681925 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f025c02cd00 con 0x7f0264161370 2024-03-24T00:39:34.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.741+0000 7f02637fe640 1 -- 172.21.15.67:0/1548681925 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f025c034660 con 0x7f0264161370 2024-03-24T00:39:34.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.741+0000 7f026aae8640 1 -- 172.21.15.67:0/1548681925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0264161370 msgr2=0x7f0264161ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:34.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.741+0000 7f026aae8640 1 --2- 172.21.15.67:0/1548681925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0264161370 0x7f0264161ab0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f02640ce300 tx=0x7f025c02bf50 comp rx=0 tx=0).stop 2024-03-24T00:39:34.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.741+0000 7f026aae8640 1 -- 172.21.15.67:0/1548681925 shutdown_connections 2024-03-24T00:39:34.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.741+0000 7f026aae8640 1 --2- 172.21.15.67:0/1548681925 >> v2:172.21.15.72:3300/0 conn(0x7f02640cf100 0x7f02640d2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:34.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.741+0000 7f026aae8640 1 --2- 172.21.15.67:0/1548681925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0264161370 0x7f0264161ab0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:34.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.741+0000 7f026aae8640 1 -- 172.21.15.67:0/1548681925 >> 172.21.15.67:0/1548681925 conn(0x7f026406c880 msgr2=0x7f026406cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:34.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.741+0000 7f026aae8640 1 -- 172.21.15.67:0/1548681925 shutdown_connections 2024-03-24T00:39:34.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.741+0000 7f026aae8640 1 -- 172.21.15.67:0/1548681925 wait complete. 2024-03-24T00:39:34.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.742+0000 7f026aae8640 1 Processor -- start 2024-03-24T00:39:34.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.742+0000 7f026aae8640 1 -- start start 2024-03-24T00:39:34.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.743+0000 7f026aae8640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02640b9690 0x7f02640c68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:34.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.743+0000 7f026aae8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02640cf100 0x7f02640b5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:34.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.743+0000 7f026aae8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0264161370 0x7f026415ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:34.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.743+0000 7f026aae8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0264161750 con 0x7f0264161370 2024-03-24T00:39:34.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.743+0000 7f026aae8640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0264162850 con 0x7f02640cf100 2024-03-24T00:39:34.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.743+0000 7f026aae8640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02640d2b60 con 0x7f02640b9690 2024-03-24T00:39:34.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.743+0000 7f0269085640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0264161370 0x7f026415ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:34.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.743+0000 7f0269085640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0264161370 0x7f026415ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50192/0 (socket says 172.21.15.67:50192) 2024-03-24T00:39:34.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.743+0000 7f0269085640 1 -- 172.21.15.67:0/3544278566 learned_addr learned my addr 172.21.15.67:0/3544278566 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:34.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.743+0000 7f0268884640 1 --2- 172.21.15.67:0/3544278566 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02640b9690 0x7f02640c68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:34.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.743+0000 7f0263fff640 1 --2- 172.21.15.67:0/3544278566 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02640cf100 0x7f02640b5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:34.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.744+0000 7f0269085640 1 -- 172.21.15.67:0/3544278566 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02640b9690 msgr2=0x7f02640c68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:34.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.744+0000 7f0269085640 1 --2- 172.21.15.67:0/3544278566 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02640b9690 0x7f02640c68e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:34.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.744+0000 7f0269085640 1 -- 172.21.15.67:0/3544278566 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02640cf100 msgr2=0x7f02640b5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:34.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.744+0000 7f0269085640 1 --2- 172.21.15.67:0/3544278566 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02640cf100 0x7f02640b5ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:34.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.744+0000 7f0269085640 1 -- 172.21.15.67:0/3544278566 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02640b4400 con 0x7f0264161370 2024-03-24T00:39:34.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.744+0000 7f0268884640 1 --2- 172.21.15.67:0/3544278566 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02640b9690 0x7f02640c68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:34.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.744+0000 7f0269085640 1 --2- 172.21.15.67:0/3544278566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0264161370 0x7f026415ec10 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f025400e850 tx=0x7f025400ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:34.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.744+0000 7f02617fa640 1 -- 172.21.15.67:0/3544278566 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0254017070 con 0x7f0264161370 2024-03-24T00:39:34.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.744+0000 7f02617fa640 1 -- 172.21.15.67:0/3544278566 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f025400c940 con 0x7f0264161370 2024-03-24T00:39:34.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.744+0000 7f026aae8640 1 -- 172.21.15.67:0/3544278566 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02640b46e0 con 0x7f0264161370 2024-03-24T00:39:34.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.744+0000 7f02617fa640 1 -- 172.21.15.67:0/3544278566 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f025400cd30 con 0x7f0264161370 2024-03-24T00:39:34.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.744+0000 7f026aae8640 1 -- 172.21.15.67:0/3544278566 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02640b4ca0 con 0x7f0264161370 2024-03-24T00:39:34.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.745+0000 7f026aae8640 1 -- 172.21.15.67:0/3544278566 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f026404f400 con 0x7f0264161370 2024-03-24T00:39:34.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.745+0000 7f02617fa640 1 -- 172.21.15.67:0/3544278566 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f025401a070 con 0x7f0264161370 2024-03-24T00:39:34.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.746+0000 7f02617fa640 1 --2- 172.21.15.67:0/3544278566 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f025006bbd0 0x7f025006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:34.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.746+0000 7f02617fa640 1 -- 172.21.15.67:0/3544278566 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f025408b0e0 con 0x7f0264161370 2024-03-24T00:39:34.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.746+0000 7f0268884640 1 --2- 172.21.15.67:0/3544278566 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f025006bbd0 0x7f025006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:34.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.746+0000 7f0268884640 1 --2- 172.21.15.67:0/3544278566 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f025006bbd0 0x7f025006e090 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f025c036000 tx=0x7f025c030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:34.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.750+0000 7f02617fa640 1 -- 172.21.15.67:0/3544278566 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0254059210 con 0x7f0264161370 2024-03-24T00:39:34.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.933+0000 7f026aae8640 1 -- 172.21.15.67:0/3544278566 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f02640b9c90 con 0x7f0264161370 2024-03-24T00:39:34.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.934+0000 7f02617fa640 1 -- 172.21.15.67:0/3544278566 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f025405d2b0 con 0x7f0264161370 2024-03-24T00:39:34.933 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-24T00:39:34.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.935+0000 7f026aae8640 1 -- 172.21.15.67:0/3544278566 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f025006bbd0 msgr2=0x7f025006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:34.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.935+0000 7f026aae8640 1 --2- 172.21.15.67:0/3544278566 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f025006bbd0 0x7f025006e090 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f025c036000 tx=0x7f025c030000 comp rx=0 tx=0).stop 2024-03-24T00:39:34.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.935+0000 7f026aae8640 1 -- 172.21.15.67:0/3544278566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0264161370 msgr2=0x7f026415ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:34.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.935+0000 7f026aae8640 1 --2- 172.21.15.67:0/3544278566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0264161370 0x7f026415ec10 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f025400e850 tx=0x7f025400ed10 comp rx=0 tx=0).stop 2024-03-24T00:39:34.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.936+0000 7f026aae8640 1 -- 172.21.15.67:0/3544278566 shutdown_connections 2024-03-24T00:39:34.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.936+0000 7f026aae8640 1 --2- 172.21.15.67:0/3544278566 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f025006bbd0 0x7f025006e090 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:34.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.936+0000 7f026aae8640 1 --2- 172.21.15.67:0/3544278566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0264161370 0x7f026415ec10 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:34.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.936+0000 7f026aae8640 1 --2- 172.21.15.67:0/3544278566 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02640cf100 0x7f02640b5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:34.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.936+0000 7f026aae8640 1 --2- 172.21.15.67:0/3544278566 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02640b9690 0x7f02640c68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:34.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.936+0000 7f026aae8640 1 -- 172.21.15.67:0/3544278566 >> 172.21.15.67:0/3544278566 conn(0x7f026406c880 msgr2=0x7f02641603b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:34.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.936+0000 7f026aae8640 1 -- 172.21.15.67:0/3544278566 shutdown_connections 2024-03-24T00:39:34.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:34.936+0000 7f026aae8640 1 -- 172.21.15.67:0/3544278566 wait complete. 2024-03-24T00:39:34.971 INFO:tasks.cephfs_test_runner:Starting test: test_apply_tag (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2024-03-24T00:39:34.972 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:39:35.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.074+0000 7f9e78577640 1 Processor -- start 2024-03-24T00:39:35.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.074+0000 7f9e78577640 1 -- start start 2024-03-24T00:39:35.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.074+0000 7f9e78577640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9e7015e310 0x7f9e70162610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:35.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.074+0000 7f9e78577640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9e700d0d50 0x7f9e700d2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:35.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.074+0000 7f9e78577640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e700ce920 con 0x7f9e7015df30 2024-03-24T00:39:35.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.074+0000 7f9e78577640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e70162b50 con 0x7f9e7015e310 2024-03-24T00:39:35.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.074+0000 7f9e78577640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e700cef20 con 0x7f9e700d0d50 2024-03-24T00:39:35.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.075+0000 7f9e75b12640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9e7015e310 0x7f9e70162610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:35.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.075+0000 7f9e75b12640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9e7015e310 0x7f9e70162610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50204/0 (socket says 172.21.15.67:50204) 2024-03-24T00:39:35.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.075+0000 7f9e76b14640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9e700d0d50 0x7f9e700d2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:35.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.075+0000 7f9e75b12640 1 -- 172.21.15.67:0/547344301 learned_addr learned my addr 172.21.15.67:0/547344301 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:35.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.075+0000 7f9e75b12640 1 -- 172.21.15.67:0/547344301 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9e700d0d50 msgr2=0x7f9e700d2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:35.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.075+0000 7f9e75b12640 1 --2- 172.21.15.67:0/547344301 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9e700d0d50 0x7f9e700d2490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.075+0000 7f9e75b12640 1 -- 172.21.15.67:0/547344301 >> v1:172.21.15.72:6789/0 conn(0x7f9e7015df30 legacy=0x7f9e700d0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:35.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.075+0000 7f9e75b12640 1 -- 172.21.15.67:0/547344301 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e700cf200 con 0x7f9e7015e310 2024-03-24T00:39:35.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.076+0000 7f9e75b12640 1 --2- 172.21.15.67:0/547344301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e7015e310 0x7f9e70162610 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c009870 tx=0x7f9e6c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7753d9cde6b32231 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:35.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.076+0000 7f9e75311640 1 -- 172.21.15.67:0/547344301 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e6c035020 con 0x7f9e7015e310 2024-03-24T00:39:35.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.076+0000 7f9e75311640 1 -- 172.21.15.67:0/547344301 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e6c031640 con 0x7f9e7015e310 2024-03-24T00:39:35.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.076+0000 7f9e75311640 1 -- 172.21.15.67:0/547344301 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e6c02d040 con 0x7f9e7015e310 2024-03-24T00:39:35.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.076+0000 7f9e78577640 1 -- 172.21.15.67:0/547344301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e7015e310 msgr2=0x7f9e70162610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:35.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.076+0000 7f9e78577640 1 --2- 172.21.15.67:0/547344301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e7015e310 0x7f9e70162610 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c009870 tx=0x7f9e6c02b8f0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.076+0000 7f9e78577640 1 -- 172.21.15.67:0/547344301 shutdown_connections 2024-03-24T00:39:35.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.076+0000 7f9e78577640 1 --2- 172.21.15.67:0/547344301 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9e700d0d50 0x7f9e700d2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.076+0000 7f9e78577640 1 --2- 172.21.15.67:0/547344301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e7015e310 0x7f9e70162610 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.076+0000 7f9e78577640 1 -- 172.21.15.67:0/547344301 >> 172.21.15.67:0/547344301 conn(0x7f9e7006c880 msgr2=0x7f9e7006cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:35.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.076+0000 7f9e78577640 1 -- 172.21.15.67:0/547344301 shutdown_connections 2024-03-24T00:39:35.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.076+0000 7f9e78577640 1 -- 172.21.15.67:0/547344301 wait complete. 2024-03-24T00:39:35.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.078+0000 7f9e78577640 1 Processor -- start 2024-03-24T00:39:35.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.078+0000 7f9e78577640 1 -- start start 2024-03-24T00:39:35.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.078+0000 7f9e78577640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9e700d0d50 0x7f9e700cd370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:35.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.078+0000 7f9e78577640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e7015df30 0x7f9e7015e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:35.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.078+0000 7f9e78577640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9e700cbe40 0x7f9e7015ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:35.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.078+0000 7f9e78577640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e700ce580 con 0x7f9e7015df30 2024-03-24T00:39:35.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.078+0000 7f9e78577640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e700b32f0 con 0x7f9e700cbe40 2024-03-24T00:39:35.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.078+0000 7f9e78577640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e700ce700 con 0x7f9e700d0d50 2024-03-24T00:39:35.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.079+0000 7f9e76313640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9e700d0d50 0x7f9e700cd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:35.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.079+0000 7f9e75b12640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e7015df30 0x7f9e7015e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:35.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.079+0000 7f9e76313640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9e700d0d50 0x7f9e700cd370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.67:33648/0 (socket says 172.21.15.67:33648) 2024-03-24T00:39:35.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.079+0000 7f9e76313640 1 -- 172.21.15.67:0/2952979672 learned_addr learned my addr 172.21.15.67:0/2952979672 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:35.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.079+0000 7f9e76b14640 1 --2- 172.21.15.67:0/2952979672 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9e700cbe40 0x7f9e7015ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:35.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.079+0000 7f9e75b12640 1 -- 172.21.15.67:0/2952979672 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9e700d0d50 msgr2=0x7f9e700cd370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:35.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.079+0000 7f9e75b12640 1 --2- 172.21.15.67:0/2952979672 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9e700d0d50 0x7f9e700cd370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.079+0000 7f9e75b12640 1 -- 172.21.15.67:0/2952979672 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9e700cbe40 msgr2=0x7f9e7015ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:35.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.079+0000 7f9e75b12640 1 --2- 172.21.15.67:0/2952979672 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9e700cbe40 0x7f9e7015ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.079+0000 7f9e75b12640 1 -- 172.21.15.67:0/2952979672 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e700cc2d0 con 0x7f9e7015df30 2024-03-24T00:39:35.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.079+0000 7f9e76313640 1 --2- 172.21.15.67:0/2952979672 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9e700d0d50 0x7f9e700cd370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:35.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.079+0000 7f9e76b14640 1 --2- 172.21.15.67:0/2952979672 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9e700cbe40 0x7f9e7015ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:35.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.079+0000 7f9e75b12640 1 --2- 172.21.15.67:0/2952979672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e7015df30 0x7f9e7015e700 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f9e700b6910 tx=0x7f9e6c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:35.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.079+0000 7f9e66ffd640 1 -- 172.21.15.67:0/2952979672 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e6c040070 con 0x7f9e7015df30 2024-03-24T00:39:35.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.079+0000 7f9e66ffd640 1 -- 172.21.15.67:0/2952979672 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e6c0399f0 con 0x7f9e7015df30 2024-03-24T00:39:35.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.079+0000 7f9e78577640 1 -- 172.21.15.67:0/2952979672 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e700cc5b0 con 0x7f9e7015df30 2024-03-24T00:39:35.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.079+0000 7f9e66ffd640 1 -- 172.21.15.67:0/2952979672 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e6c035040 con 0x7f9e7015df30 2024-03-24T00:39:35.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.079+0000 7f9e78577640 1 -- 172.21.15.67:0/2952979672 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e700ccb70 con 0x7f9e7015df30 2024-03-24T00:39:35.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.080+0000 7f9e78577640 1 -- 172.21.15.67:0/2952979672 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e7004f400 con 0x7f9e7015df30 2024-03-24T00:39:35.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.081+0000 7f9e66ffd640 1 -- 172.21.15.67:0/2952979672 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9e6c038020 con 0x7f9e7015df30 2024-03-24T00:39:35.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.081+0000 7f9e66ffd640 1 --2- 172.21.15.67:0/2952979672 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9e5406bbd0 0x7f9e5406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:35.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.081+0000 7f9e66ffd640 1 -- 172.21.15.67:0/2952979672 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f9e6c0ac6a0 con 0x7f9e7015df30 2024-03-24T00:39:35.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.081+0000 7f9e76313640 1 --2- 172.21.15.67:0/2952979672 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9e5406bbd0 0x7f9e5406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:35.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.082+0000 7f9e76313640 1 --2- 172.21.15.67:0/2952979672 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9e5406bbd0 0x7f9e5406e090 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9e700cdb80 tx=0x7f9e60009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:35.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.085+0000 7f9e66ffd640 1 -- 172.21.15.67:0/2952979672 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9e6c07a740 con 0x7f9e7015df30 2024-03-24T00:39:35.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.267+0000 7f9e78577640 1 -- 172.21.15.67:0/2952979672 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9e700d3380 con 0x7f9e7015df30 2024-03-24T00:39:35.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.269+0000 7f9e66ffd640 1 -- 172.21.15.67:0/2952979672 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18843 (secure 0 0 0) 0x7f9e6c07e7e0 con 0x7f9e7015df30 2024-03-24T00:39:35.268 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:39:35.269 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":27,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:39:32.904729+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:38:50.077085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:38:51.095454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T00:38:54.807117+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.14","mappings":[{"from":4,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:39:35.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.270+0000 7f9e78577640 1 -- 172.21.15.67:0/2952979672 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9e5406bbd0 msgr2=0x7f9e5406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:35.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.270+0000 7f9e78577640 1 --2- 172.21.15.67:0/2952979672 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9e5406bbd0 0x7f9e5406e090 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9e700cdb80 tx=0x7f9e60009040 comp rx=0 tx=0).stop 2024-03-24T00:39:35.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.270+0000 7f9e78577640 1 -- 172.21.15.67:0/2952979672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e7015df30 msgr2=0x7f9e7015e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:35.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.270+0000 7f9e78577640 1 --2- 172.21.15.67:0/2952979672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e7015df30 0x7f9e7015e700 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f9e700b6910 tx=0x7f9e6c0046c0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.270+0000 7f9e78577640 1 -- 172.21.15.67:0/2952979672 shutdown_connections 2024-03-24T00:39:35.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.270+0000 7f9e78577640 1 --2- 172.21.15.67:0/2952979672 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9e5406bbd0 0x7f9e5406e090 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.270+0000 7f9e78577640 1 --2- 172.21.15.67:0/2952979672 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9e700cbe40 0x7f9e7015ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.270+0000 7f9e78577640 1 --2- 172.21.15.67:0/2952979672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e7015df30 0x7f9e7015e700 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.270+0000 7f9e78577640 1 --2- 172.21.15.67:0/2952979672 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9e700d0d50 0x7f9e700cd370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.271+0000 7f9e78577640 1 -- 172.21.15.67:0/2952979672 >> 172.21.15.67:0/2952979672 conn(0x7f9e7006c880 msgr2=0x7f9e700d2320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:35.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.271+0000 7f9e78577640 1 -- 172.21.15.67:0/2952979672 shutdown_connections 2024-03-24T00:39:35.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.271+0000 7f9e78577640 1 -- 172.21.15.67:0/2952979672 wait complete. 2024-03-24T00:39:35.281 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:38:39.364950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:38:50.077085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:38:51.095454+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T00:38:54.807117+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T00:39:35.281 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:39:35.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.386+0000 7fac8b427640 1 Processor -- start 2024-03-24T00:39:35.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.386+0000 7fac8b427640 1 -- start start 2024-03-24T00:39:35.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.387+0000 7fac8b427640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fac8415dfd0 0x7fac840cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:35.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.387+0000 7fac8b427640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fac840b41e0 0x7fac840b3620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:35.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.387+0000 7fac8b427640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac840cf090 con 0x7fac840b3b60 2024-03-24T00:39:35.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.387+0000 7fac8b427640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac840cc260 con 0x7fac8415dfd0 2024-03-24T00:39:35.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.387+0000 7fac8b427640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac840d0320 con 0x7fac840b41e0 2024-03-24T00:39:35.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.387+0000 7fac889c2640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fac840b41e0 0x7fac840b3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:35.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.387+0000 7fac899c4640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fac840b3b60 0x7fac84161b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41460/0 (socket says 172.21.15.67:41460) 2024-03-24T00:39:35.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.387+0000 7fac891c3640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fac8415dfd0 0x7fac840cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:35.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.387+0000 7fac899c4640 1 -- 172.21.15.67:0/570300911 learned_addr learned my addr 172.21.15.67:0/570300911 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:35.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.387+0000 7fac889c2640 1 -- 172.21.15.67:0/570300911 >> v2:172.21.15.67:3300/0 conn(0x7fac8415dfd0 msgr2=0x7fac840cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:35.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.387+0000 7fac889c2640 1 --2- 172.21.15.67:0/570300911 >> v2:172.21.15.67:3300/0 conn(0x7fac8415dfd0 0x7fac840cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.387+0000 7fac889c2640 1 -- 172.21.15.67:0/570300911 >> v1:172.21.15.67:6789/0 conn(0x7fac840b3b60 legacy=0x7fac84161b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:35.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.387+0000 7fac889c2640 1 -- 172.21.15.67:0/570300911 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac840d0600 con 0x7fac840b41e0 2024-03-24T00:39:35.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.387+0000 7fac891c3640 1 --2- 172.21.15.67:0/570300911 >> v2:172.21.15.67:3300/0 conn(0x7fac8415dfd0 0x7fac840cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:35.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.388+0000 7fac889c2640 1 --2- 172.21.15.67:0/570300911 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fac840b41e0 0x7fac840b3620 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fac740097a0 tx=0x7fac7402b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c40201352f35d1dd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:35.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.388+0000 7fac7bfff640 1 -- 172.21.15.67:0/570300911 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac74034070 con 0x7fac840b41e0 2024-03-24T00:39:35.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.388+0000 7fac7bfff640 1 -- 172.21.15.67:0/570300911 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac74030610 con 0x7fac840b41e0 2024-03-24T00:39:35.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.388+0000 7fac7bfff640 1 -- 172.21.15.67:0/570300911 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac740309e0 con 0x7fac840b41e0 2024-03-24T00:39:35.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.389+0000 7fac8b427640 1 -- 172.21.15.67:0/570300911 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fac840b41e0 msgr2=0x7fac840b3620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:35.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.389+0000 7fac8b427640 1 --2- 172.21.15.67:0/570300911 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fac840b41e0 0x7fac840b3620 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fac740097a0 tx=0x7fac7402b960 comp rx=0 tx=0).stop 2024-03-24T00:39:35.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.389+0000 7fac8b427640 1 -- 172.21.15.67:0/570300911 shutdown_connections 2024-03-24T00:39:35.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.389+0000 7fac8b427640 1 --2- 172.21.15.67:0/570300911 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fac840b41e0 0x7fac840b3620 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.389+0000 7fac8b427640 1 --2- 172.21.15.67:0/570300911 >> v2:172.21.15.67:3300/0 conn(0x7fac8415dfd0 0x7fac840cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.389+0000 7fac8b427640 1 -- 172.21.15.67:0/570300911 >> 172.21.15.67:0/570300911 conn(0x7fac8406fcf0 msgr2=0x7fac840700d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:35.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.389+0000 7fac8b427640 1 -- 172.21.15.67:0/570300911 shutdown_connections 2024-03-24T00:39:35.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.389+0000 7fac8b427640 1 -- 172.21.15.67:0/570300911 wait complete. 2024-03-24T00:39:35.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.390+0000 7fac8b427640 1 Processor -- start 2024-03-24T00:39:35.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.390+0000 7fac8b427640 1 -- start start 2024-03-24T00:39:35.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.391+0000 7fac8b427640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fac840b3b60 0x7fac84150090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:35.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.391+0000 7fac8b427640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac840b41e0 0x7fac840cad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:35.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.391+0000 7fac8b427640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac8415dfd0 0x7fac841f7160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:35.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.391+0000 7fac8b427640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac840cfec0 con 0x7fac8415dfd0 2024-03-24T00:39:35.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.391+0000 7fac8b427640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac840cedb0 con 0x7fac840b41e0 2024-03-24T00:39:35.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.391+0000 7fac8b427640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac8415e3b0 con 0x7fac840b3b60 2024-03-24T00:39:35.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.391+0000 7fac899c4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac8415dfd0 0x7fac841f7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:35.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.391+0000 7fac899c4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac8415dfd0 0x7fac841f7160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50240/0 (socket says 172.21.15.67:50240) 2024-03-24T00:39:35.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.391+0000 7fac891c3640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fac840b3b60 0x7fac84150090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:35.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.391+0000 7fac899c4640 1 -- 172.21.15.67:0/199856277 learned_addr learned my addr 172.21.15.67:0/199856277 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:35.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.391+0000 7fac889c2640 1 --2- 172.21.15.67:0/199856277 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac840b41e0 0x7fac840cad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:35.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.391+0000 7fac899c4640 1 -- 172.21.15.67:0/199856277 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fac840b3b60 msgr2=0x7fac84150090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:35.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.391+0000 7fac899c4640 1 --2- 172.21.15.67:0/199856277 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fac840b3b60 0x7fac84150090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.391+0000 7fac899c4640 1 -- 172.21.15.67:0/199856277 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac840b41e0 msgr2=0x7fac840cad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:35.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.391+0000 7fac899c4640 1 --2- 172.21.15.67:0/199856277 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac840b41e0 0x7fac840cad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.391+0000 7fac899c4640 1 -- 172.21.15.67:0/199856277 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac8414f8d0 con 0x7fac8415dfd0 2024-03-24T00:39:35.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.391+0000 7fac899c4640 1 --2- 172.21.15.67:0/199856277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac8415dfd0 0x7fac841f7160 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fac7c00cbf0 tx=0x7fac7c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:35.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.391+0000 7fac79ffb640 1 -- 172.21.15.67:0/199856277 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac7c017070 con 0x7fac8415dfd0 2024-03-24T00:39:35.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.391+0000 7fac79ffb640 1 -- 172.21.15.67:0/199856277 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac7c00ad40 con 0x7fac8415dfd0 2024-03-24T00:39:35.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.391+0000 7fac8b427640 1 -- 172.21.15.67:0/199856277 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac841f7ae0 con 0x7fac8415dfd0 2024-03-24T00:39:35.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.391+0000 7fac79ffb640 1 -- 172.21.15.67:0/199856277 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac7c012600 con 0x7fac8415dfd0 2024-03-24T00:39:35.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.392+0000 7fac8b427640 1 -- 172.21.15.67:0/199856277 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac8414fb70 con 0x7fac8415dfd0 2024-03-24T00:39:35.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.392+0000 7fac8b427640 1 -- 172.21.15.67:0/199856277 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac84150690 con 0x7fac8415dfd0 2024-03-24T00:39:35.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.393+0000 7fac79ffb640 1 -- 172.21.15.67:0/199856277 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fac7c01a070 con 0x7fac8415dfd0 2024-03-24T00:39:35.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.393+0000 7fac79ffb640 1 --2- 172.21.15.67:0/199856277 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fac6806bbd0 0x7fac6806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:35.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.393+0000 7fac79ffb640 1 -- 172.21.15.67:0/199856277 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fac7c08aa60 con 0x7fac8415dfd0 2024-03-24T00:39:35.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.393+0000 7fac891c3640 1 --2- 172.21.15.67:0/199856277 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fac6806bbd0 0x7fac6806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:35.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.394+0000 7fac891c3640 1 --2- 172.21.15.67:0/199856277 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fac6806bbd0 0x7fac6806e090 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fac80009560 tx=0x7fac80009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:35.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.397+0000 7fac79ffb640 1 -- 172.21.15.67:0/199856277 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fac7c058b90 con 0x7fac8415dfd0 2024-03-24T00:39:35.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.580+0000 7fac8b427640 1 -- 172.21.15.67:0/199856277 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fac8404f400 con 0x7fac8415dfd0 2024-03-24T00:39:35.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.580+0000 7fac79ffb640 1 -- 172.21.15.67:0/199856277 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7fac7c05cc30 con 0x7fac8415dfd0 2024-03-24T00:39:35.580 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-24T00:39:35.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.582+0000 7fac8b427640 1 -- 172.21.15.67:0/199856277 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fac6806bbd0 msgr2=0x7fac6806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:35.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.582+0000 7fac8b427640 1 --2- 172.21.15.67:0/199856277 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fac6806bbd0 0x7fac6806e090 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fac80009560 tx=0x7fac80009190 comp rx=0 tx=0).stop 2024-03-24T00:39:35.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.582+0000 7fac8b427640 1 -- 172.21.15.67:0/199856277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac8415dfd0 msgr2=0x7fac841f7160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:35.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.582+0000 7fac8b427640 1 --2- 172.21.15.67:0/199856277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac8415dfd0 0x7fac841f7160 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fac7c00cbf0 tx=0x7fac7c00a430 comp rx=0 tx=0).stop 2024-03-24T00:39:35.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.582+0000 7fac8b427640 1 -- 172.21.15.67:0/199856277 shutdown_connections 2024-03-24T00:39:35.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.582+0000 7fac8b427640 1 --2- 172.21.15.67:0/199856277 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fac6806bbd0 0x7fac6806e090 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.582+0000 7fac8b427640 1 --2- 172.21.15.67:0/199856277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac8415dfd0 0x7fac841f7160 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.582+0000 7fac8b427640 1 --2- 172.21.15.67:0/199856277 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac840b41e0 0x7fac840cad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.582+0000 7fac8b427640 1 --2- 172.21.15.67:0/199856277 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fac840b3b60 0x7fac84150090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.582+0000 7fac8b427640 1 -- 172.21.15.67:0/199856277 >> 172.21.15.67:0/199856277 conn(0x7fac8406fcf0 msgr2=0x7fac840b2c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:35.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.582+0000 7fac8b427640 1 -- 172.21.15.67:0/199856277 shutdown_connections 2024-03-24T00:39:35.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.582+0000 7fac8b427640 1 -- 172.21.15.67:0/199856277 wait complete. 2024-03-24T00:39:35.593 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:39:35.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.697+0000 7faef1c34640 1 Processor -- start 2024-03-24T00:39:35.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.697+0000 7faef1c34640 1 -- start start 2024-03-24T00:39:35.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.698+0000 7faef1c34640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faeec0b9690 0x7faeec0ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:35.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.698+0000 7faef1c34640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faeec0cef30 0x7faeec0d2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:35.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.698+0000 7faef1c34640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeec162290 con 0x7faeec161370 2024-03-24T00:39:35.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.698+0000 7faef1c34640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faeec0b6e10 con 0x7faeec0cef30 2024-03-24T00:39:35.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.698+0000 7faef1c34640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faeec0d2190 con 0x7faeec0b9690 2024-03-24T00:39:35.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.698+0000 7faeebfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faeec0cef30 0x7faeec0d2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:35.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.698+0000 7faeeb7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7faeec161370 0x7faeec161ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41466/0 (socket says 172.21.15.67:41466) 2024-03-24T00:39:35.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.698+0000 7faeeb7fe640 1 -- 172.21.15.67:0/1874923361 learned_addr learned my addr 172.21.15.67:0/1874923361 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:35.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.698+0000 7faeeaffd640 1 --2- 172.21.15.67:0/1874923361 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faeec0b9690 0x7faeec0ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:35.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.698+0000 7faeebfff640 1 -- 172.21.15.67:0/1874923361 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faeec0b9690 msgr2=0x7faeec0ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:35.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.698+0000 7faeebfff640 1 --2- 172.21.15.67:0/1874923361 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faeec0b9690 0x7faeec0ce9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.698+0000 7faeebfff640 1 -- 172.21.15.67:0/1874923361 >> v1:172.21.15.67:6789/0 conn(0x7faeec161370 legacy=0x7faeec161ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:35.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.698+0000 7faeea7fc640 1 -- 172.21.15.67:0/1874923361 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 617629635 0 0) 0x7faeec162290 con 0x7faeec161370 2024-03-24T00:39:35.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.698+0000 7faeebfff640 1 -- 172.21.15.67:0/1874923361 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeec162570 con 0x7faeec0cef30 2024-03-24T00:39:35.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.699+0000 7faeebfff640 1 --2- 172.21.15.67:0/1874923361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faeec0cef30 0x7faeec0d2620 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7faee4009dc0 tx=0x7faee402bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a8a28cd1ed2b9d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:35.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.699+0000 7faeea7fc640 1 -- 172.21.15.67:0/1874923361 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee4039070 con 0x7faeec0cef30 2024-03-24T00:39:35.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.699+0000 7faeea7fc640 1 -- 172.21.15.67:0/1874923361 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faee402ccd0 con 0x7faeec0cef30 2024-03-24T00:39:35.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.699+0000 7faeea7fc640 1 -- 172.21.15.67:0/1874923361 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee4034660 con 0x7faeec0cef30 2024-03-24T00:39:35.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.700+0000 7faef1c34640 1 -- 172.21.15.67:0/1874923361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faeec0cef30 msgr2=0x7faeec0d2620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:35.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.700+0000 7faef1c34640 1 --2- 172.21.15.67:0/1874923361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faeec0cef30 0x7faeec0d2620 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7faee4009dc0 tx=0x7faee402bf20 comp rx=0 tx=0).stop 2024-03-24T00:39:35.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.700+0000 7faef1c34640 1 -- 172.21.15.67:0/1874923361 shutdown_connections 2024-03-24T00:39:35.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.700+0000 7faef1c34640 1 --2- 172.21.15.67:0/1874923361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faeec0cef30 0x7faeec0d2620 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.700+0000 7faef1c34640 1 --2- 172.21.15.67:0/1874923361 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faeec0b9690 0x7faeec0ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.700+0000 7faef1c34640 1 -- 172.21.15.67:0/1874923361 >> 172.21.15.67:0/1874923361 conn(0x7faeec06c880 msgr2=0x7faeec06cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:35.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.700+0000 7faef1c34640 1 -- 172.21.15.67:0/1874923361 shutdown_connections 2024-03-24T00:39:35.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.700+0000 7faef1c34640 1 -- 172.21.15.67:0/1874923361 wait complete. 2024-03-24T00:39:35.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.701+0000 7faef1c34640 1 Processor -- start 2024-03-24T00:39:35.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.701+0000 7faef1c34640 1 -- start start 2024-03-24T00:39:35.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.702+0000 7faef1c34640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faeec0b9690 0x7faeec155ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:35.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.702+0000 7faef1c34640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faeec0cef30 0x7faeec158040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:35.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.702+0000 7faef1c34640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faeec161370 0x7faeec143c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:35.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.702+0000 7faef1c34640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeec0d2b60 con 0x7faeec0cef30 2024-03-24T00:39:35.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.702+0000 7faef1c34640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeec161750 con 0x7faeec161370 2024-03-24T00:39:35.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.702+0000 7faef1c34640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faeec0d2ce0 con 0x7faeec0b9690 2024-03-24T00:39:35.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.702+0000 7faeeaffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faeec0cef30 0x7faeec158040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:35.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.702+0000 7faeeaffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faeec0cef30 0x7faeec158040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50250/0 (socket says 172.21.15.67:50250) 2024-03-24T00:39:35.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.702+0000 7faeeaffd640 1 -- 172.21.15.67:0/2030058734 learned_addr learned my addr 172.21.15.67:0/2030058734 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:35.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.702+0000 7faeeb7fe640 1 --2- 172.21.15.67:0/2030058734 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faeec0b9690 0x7faeec155ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:35.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.702+0000 7faeebfff640 1 --2- 172.21.15.67:0/2030058734 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faeec161370 0x7faeec143c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:35.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.702+0000 7faeeaffd640 1 -- 172.21.15.67:0/2030058734 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faeec0b9690 msgr2=0x7faeec155ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:35.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.702+0000 7faeeaffd640 1 --2- 172.21.15.67:0/2030058734 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faeec0b9690 0x7faeec155ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.702+0000 7faeeaffd640 1 -- 172.21.15.67:0/2030058734 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faeec161370 msgr2=0x7faeec143c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:35.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.702+0000 7faeeaffd640 1 --2- 172.21.15.67:0/2030058734 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faeec161370 0x7faeec143c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.702+0000 7faeeaffd640 1 -- 172.21.15.67:0/2030058734 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeec158880 con 0x7faeec0cef30 2024-03-24T00:39:35.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.702+0000 7faeebfff640 1 --2- 172.21.15.67:0/2030058734 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faeec161370 0x7faeec143c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:35.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.702+0000 7faeeaffd640 1 --2- 172.21.15.67:0/2030058734 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faeec0cef30 0x7faeec158040 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7faee000e9b0 tx=0x7faee000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:35.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.703+0000 7faed3fff640 1 -- 172.21.15.67:0/2030058734 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee0017070 con 0x7faeec0cef30 2024-03-24T00:39:35.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.703+0000 7faed3fff640 1 -- 172.21.15.67:0/2030058734 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faee000c9a0 con 0x7faeec0cef30 2024-03-24T00:39:35.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.703+0000 7faef1c34640 1 -- 172.21.15.67:0/2030058734 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faeec158b60 con 0x7faeec0cef30 2024-03-24T00:39:35.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.703+0000 7faed3fff640 1 -- 172.21.15.67:0/2030058734 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee000cd90 con 0x7faeec0cef30 2024-03-24T00:39:35.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.703+0000 7faef1c34640 1 -- 172.21.15.67:0/2030058734 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faeec158580 con 0x7faeec0cef30 2024-03-24T00:39:35.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.704+0000 7faef1c34640 1 -- 172.21.15.67:0/2030058734 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faeec068a00 con 0x7faeec0cef30 2024-03-24T00:39:35.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.704+0000 7faed3fff640 1 -- 172.21.15.67:0/2030058734 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7faee001a070 con 0x7faeec0cef30 2024-03-24T00:39:35.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.705+0000 7faed3fff640 1 --2- 172.21.15.67:0/2030058734 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7faed806bbd0 0x7faed806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:35.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.705+0000 7faed3fff640 1 -- 172.21.15.67:0/2030058734 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7faee008b150 con 0x7faeec0cef30 2024-03-24T00:39:35.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.705+0000 7faeeb7fe640 1 --2- 172.21.15.67:0/2030058734 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7faed806bbd0 0x7faed806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:35.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.705+0000 7faeeb7fe640 1 --2- 172.21.15.67:0/2030058734 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7faed806bbd0 0x7faed806e090 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7faedc007fd0 tx=0x7faedc007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:35.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.709+0000 7faed3fff640 1 -- 172.21.15.67:0/2030058734 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7faeec068a00 con 0x7faeec0cef30 2024-03-24T00:39:35.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.889+0000 7faef1c34640 1 -- 172.21.15.67:0/2030058734 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7faeec04f400 con 0x7faeec0cef30 2024-03-24T00:39:35.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.889+0000 7faed3fff640 1 -- 172.21.15.67:0/2030058734 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7faee005f070 con 0x7faeec0cef30 2024-03-24T00:39:35.889 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-24T00:39:35.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.891+0000 7faef1c34640 1 -- 172.21.15.67:0/2030058734 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7faed806bbd0 msgr2=0x7faed806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:35.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.891+0000 7faef1c34640 1 --2- 172.21.15.67:0/2030058734 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7faed806bbd0 0x7faed806e090 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7faedc007fd0 tx=0x7faedc007de0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.891+0000 7faef1c34640 1 -- 172.21.15.67:0/2030058734 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faeec0cef30 msgr2=0x7faeec158040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:35.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.891+0000 7faef1c34640 1 --2- 172.21.15.67:0/2030058734 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faeec0cef30 0x7faeec158040 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7faee000e9b0 tx=0x7faee000ee70 comp rx=0 tx=0).stop 2024-03-24T00:39:35.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.891+0000 7faef1c34640 1 -- 172.21.15.67:0/2030058734 shutdown_connections 2024-03-24T00:39:35.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.891+0000 7faef1c34640 1 --2- 172.21.15.67:0/2030058734 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7faed806bbd0 0x7faed806e090 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.891+0000 7faef1c34640 1 --2- 172.21.15.67:0/2030058734 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faeec161370 0x7faeec143c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.891+0000 7faef1c34640 1 --2- 172.21.15.67:0/2030058734 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faeec0cef30 0x7faeec158040 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.891+0000 7faef1c34640 1 --2- 172.21.15.67:0/2030058734 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faeec0b9690 0x7faeec155ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:35.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.891+0000 7faef1c34640 1 -- 172.21.15.67:0/2030058734 >> 172.21.15.67:0/2030058734 conn(0x7faeec06c880 msgr2=0x7faeec15c840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:35.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.891+0000 7faef1c34640 1 -- 172.21.15.67:0/2030058734 shutdown_connections 2024-03-24T00:39:35.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:35.891+0000 7faef1c34640 1 -- 172.21.15.67:0/2030058734 wait complete. 2024-03-24T00:39:35.902 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:39:36.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.007+0000 7f88f7c7e640 1 Processor -- start 2024-03-24T00:39:36.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.007+0000 7f88f7c7e640 1 -- start start 2024-03-24T00:39:36.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.008+0000 7f88f7c7e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f88f0161370 0x7f88f0161ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:36.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.008+0000 7f88f7c7e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f88f00cf100 0x7f88f00d2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:36.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.008+0000 7f88f7c7e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88f0161ff0 con 0x7f88f00b9690 2024-03-24T00:39:36.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.008+0000 7f88f7c7e640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88f00b6e10 con 0x7f88f00cf100 2024-03-24T00:39:36.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.008+0000 7f88f7c7e640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88f00d2190 con 0x7f88f0161370 2024-03-24T00:39:36.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.008+0000 7f88f621b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f88f00cf100 0x7f88f00d2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:36.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.008+0000 7f88f5a1a640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f88f0161370 0x7f88f0161ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:36.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.008+0000 7f88f621b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f88f00cf100 0x7f88f00d2620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50260/0 (socket says 172.21.15.67:50260) 2024-03-24T00:39:36.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.008+0000 7f88f621b640 1 -- 172.21.15.67:0/2857166832 learned_addr learned my addr 172.21.15.67:0/2857166832 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:36.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.009+0000 7f88f5a1a640 1 -- 172.21.15.67:0/2857166832 >> v2:172.21.15.67:3300/0 conn(0x7f88f00cf100 msgr2=0x7f88f00d2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:36.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.009+0000 7f88f5a1a640 1 --2- 172.21.15.67:0/2857166832 >> v2:172.21.15.67:3300/0 conn(0x7f88f00cf100 0x7f88f00d2620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.009+0000 7f88f5a1a640 1 -- 172.21.15.67:0/2857166832 >> v1:172.21.15.72:6789/0 conn(0x7f88f00b9690 legacy=0x7f88f00ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:36.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.009+0000 7f88f5a1a640 1 -- 172.21.15.67:0/2857166832 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f0162290 con 0x7f88f0161370 2024-03-24T00:39:36.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.009+0000 7f88f4a18640 1 -- 172.21.15.67:0/2857166832 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1473765643 0 0) 0x7f88f0161ff0 con 0x7f88f00b9690 2024-03-24T00:39:36.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.009+0000 7f88f621b640 1 --2- 172.21.15.67:0/2857166832 >> v2:172.21.15.67:3300/0 conn(0x7f88f00cf100 0x7f88f00d2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:36.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.009+0000 7f88f5a1a640 1 --2- 172.21.15.67:0/2857166832 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f88f0161370 0x7f88f0161ab0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f88e80097a0 tx=0x7f88e802b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea87788e9ddd8765 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:36.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.009+0000 7f88f4a18640 1 -- 172.21.15.67:0/2857166832 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e8034070 con 0x7f88f0161370 2024-03-24T00:39:36.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.010+0000 7f88f4a18640 1 -- 172.21.15.67:0/2857166832 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88e8030800 con 0x7f88f0161370 2024-03-24T00:39:36.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.010+0000 7f88f4a18640 1 -- 172.21.15.67:0/2857166832 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e8030bd0 con 0x7f88f0161370 2024-03-24T00:39:36.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.010+0000 7f88f7c7e640 1 -- 172.21.15.67:0/2857166832 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f88f0161370 msgr2=0x7f88f0161ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:36.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.010+0000 7f88f7c7e640 1 --2- 172.21.15.67:0/2857166832 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f88f0161370 0x7f88f0161ab0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f88e80097a0 tx=0x7f88e802b940 comp rx=0 tx=0).stop 2024-03-24T00:39:36.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.010+0000 7f88f7c7e640 1 -- 172.21.15.67:0/2857166832 shutdown_connections 2024-03-24T00:39:36.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.010+0000 7f88f7c7e640 1 --2- 172.21.15.67:0/2857166832 >> v2:172.21.15.67:3300/0 conn(0x7f88f00cf100 0x7f88f00d2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.010+0000 7f88f7c7e640 1 --2- 172.21.15.67:0/2857166832 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f88f0161370 0x7f88f0161ab0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.010+0000 7f88f7c7e640 1 -- 172.21.15.67:0/2857166832 >> 172.21.15.67:0/2857166832 conn(0x7f88f006c880 msgr2=0x7f88f006cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:36.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.010+0000 7f88f7c7e640 1 -- 172.21.15.67:0/2857166832 shutdown_connections 2024-03-24T00:39:36.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.010+0000 7f88f7c7e640 1 -- 172.21.15.67:0/2857166832 wait complete. 2024-03-24T00:39:36.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.011+0000 7f88f7c7e640 1 Processor -- start 2024-03-24T00:39:36.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.011+0000 7f88f7c7e640 1 -- start start 2024-03-24T00:39:36.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.012+0000 7f88f7c7e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88f00b9690 0x7f88f01f72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:36.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.012+0000 7f88f7c7e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f88f00cf100 0x7f88f00c94d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:36.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.012+0000 7f88f7c7e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f88f0161370 0x7f88f01e78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:36.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.012+0000 7f88f7c7e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f0161750 con 0x7f88f00b9690 2024-03-24T00:39:36.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.012+0000 7f88f7c7e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f0162850 con 0x7f88f00cf100 2024-03-24T00:39:36.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.012+0000 7f88f7c7e640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88f00d2b60 con 0x7f88f0161370 2024-03-24T00:39:36.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.012+0000 7f88f5a1a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88f00b9690 0x7f88f01f72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:36.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.012+0000 7f88f5a1a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88f00b9690 0x7f88f01f72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50270/0 (socket says 172.21.15.67:50270) 2024-03-24T00:39:36.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.012+0000 7f88f5a1a640 1 -- 172.21.15.67:0/840753612 learned_addr learned my addr 172.21.15.67:0/840753612 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:36.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.012+0000 7f88f5219640 1 --2- 172.21.15.67:0/840753612 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f88f00cf100 0x7f88f00c94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:36.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.012+0000 7f88f621b640 1 --2- 172.21.15.67:0/840753612 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f88f0161370 0x7f88f01e78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:36.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.012+0000 7f88f5a1a640 1 -- 172.21.15.67:0/840753612 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f88f0161370 msgr2=0x7f88f01e78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:36.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.012+0000 7f88f5a1a640 1 --2- 172.21.15.67:0/840753612 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f88f0161370 0x7f88f01e78f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.012+0000 7f88f5a1a640 1 -- 172.21.15.67:0/840753612 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f88f00cf100 msgr2=0x7f88f00c94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:36.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.012+0000 7f88f5a1a640 1 --2- 172.21.15.67:0/840753612 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f88f00cf100 0x7f88f00c94d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.012+0000 7f88f5a1a640 1 -- 172.21.15.67:0/840753612 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f00cba30 con 0x7f88f00b9690 2024-03-24T00:39:36.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.013+0000 7f88f5a1a640 1 --2- 172.21.15.67:0/840753612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88f00b9690 0x7f88f01f72f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f88e802bf50 tx=0x7f88e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:36.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.013+0000 7f88ee7fc640 1 -- 172.21.15.67:0/840753612 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e8047070 con 0x7f88f00b9690 2024-03-24T00:39:36.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.013+0000 7f88ee7fc640 1 -- 172.21.15.67:0/840753612 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88e8030600 con 0x7f88f00b9690 2024-03-24T00:39:36.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.013+0000 7f88f7c7e640 1 -- 172.21.15.67:0/840753612 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88f00cbd10 con 0x7f88f00b9690 2024-03-24T00:39:36.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.013+0000 7f88ee7fc640 1 -- 172.21.15.67:0/840753612 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e8034040 con 0x7f88f00b9690 2024-03-24T00:39:36.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.013+0000 7f88f7c7e640 1 -- 172.21.15.67:0/840753612 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88f00cc2d0 con 0x7f88f00b9690 2024-03-24T00:39:36.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.014+0000 7f88f7c7e640 1 -- 172.21.15.67:0/840753612 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88f00ce120 con 0x7f88f00b9690 2024-03-24T00:39:36.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.014+0000 7f88ee7fc640 1 -- 172.21.15.67:0/840753612 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f88e804a070 con 0x7f88f00b9690 2024-03-24T00:39:36.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.015+0000 7f88ee7fc640 1 --2- 172.21.15.67:0/840753612 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f88dc06bbd0 0x7f88dc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:36.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.015+0000 7f88ee7fc640 1 -- 172.21.15.67:0/840753612 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f88e80ac930 con 0x7f88f00b9690 2024-03-24T00:39:36.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.015+0000 7f88f5219640 1 --2- 172.21.15.67:0/840753612 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f88dc06bbd0 0x7f88dc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:36.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.015+0000 7f88f5219640 1 --2- 172.21.15.67:0/840753612 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f88dc06bbd0 0x7f88dc06e090 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f88f0152cc0 tx=0x7f88e4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:36.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.019+0000 7f88ee7fc640 1 -- 172.21.15.67:0/840753612 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f88f00ce120 con 0x7f88f00b9690 2024-03-24T00:39:36.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.202+0000 7f88f7c7e640 1 -- 172.21.15.67:0/840753612 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f88f004f400 con 0x7f88f00b9690 2024-03-24T00:39:36.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.202+0000 7f88ee7fc640 1 -- 172.21.15.67:0/840753612 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f88e80307d0 con 0x7f88f00b9690 2024-03-24T00:39:36.202 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-03-24T00:39:36.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.204+0000 7f88f7c7e640 1 -- 172.21.15.67:0/840753612 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f88dc06bbd0 msgr2=0x7f88dc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:36.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.204+0000 7f88f7c7e640 1 --2- 172.21.15.67:0/840753612 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f88dc06bbd0 0x7f88dc06e090 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f88f0152cc0 tx=0x7f88e4009040 comp rx=0 tx=0).stop 2024-03-24T00:39:36.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.204+0000 7f88f7c7e640 1 -- 172.21.15.67:0/840753612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88f00b9690 msgr2=0x7f88f01f72f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:36.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.204+0000 7f88f7c7e640 1 --2- 172.21.15.67:0/840753612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88f00b9690 0x7f88f01f72f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f88e802bf50 tx=0x7f88e80046c0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.204+0000 7f88f7c7e640 1 -- 172.21.15.67:0/840753612 shutdown_connections 2024-03-24T00:39:36.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.204+0000 7f88f7c7e640 1 --2- 172.21.15.67:0/840753612 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f88dc06bbd0 0x7f88dc06e090 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.204+0000 7f88f7c7e640 1 --2- 172.21.15.67:0/840753612 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f88f0161370 0x7f88f01e78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.204+0000 7f88f7c7e640 1 --2- 172.21.15.67:0/840753612 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f88f00cf100 0x7f88f00c94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.204+0000 7f88f7c7e640 1 --2- 172.21.15.67:0/840753612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88f00b9690 0x7f88f01f72f0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.204+0000 7f88f7c7e640 1 -- 172.21.15.67:0/840753612 >> 172.21.15.67:0/840753612 conn(0x7f88f006c880 msgr2=0x7f88f00b3dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:36.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.204+0000 7f88f7c7e640 1 -- 172.21.15.67:0/840753612 shutdown_connections 2024-03-24T00:39:36.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.204+0000 7f88f7c7e640 1 -- 172.21.15.67:0/840753612 wait complete. 2024-03-24T00:39:36.214 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T00:39:36.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.319+0000 7f89e8f9d640 1 Processor -- start 2024-03-24T00:39:36.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.319+0000 7f89e8f9d640 1 -- start start 2024-03-24T00:39:36.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.320+0000 7f89e8f9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f89e415bc90 0x7f89e406b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:36.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.320+0000 7f89e8f9d640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f89e40cde60 0x7f89e406b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:36.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.320+0000 7f89e8f9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f89e40b1840 0x7f89e40b1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:36.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.320+0000 7f89e8f9d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89e415c070 con 0x7f89e415bc90 2024-03-24T00:39:36.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.320+0000 7f89e8f9d640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89e40cd0e0 con 0x7f89e40b1840 2024-03-24T00:39:36.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.320+0000 7f89e8f9d640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89e40cd380 con 0x7f89e40cde60 2024-03-24T00:39:36.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.320+0000 7f89e37fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f89e415bc90 0x7f89e406b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:36.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.320+0000 7f89e37fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f89e415bc90 0x7f89e406b040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50272/0 (socket says 172.21.15.67:50272) 2024-03-24T00:39:36.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.320+0000 7f89e37fe640 1 -- 172.21.15.67:0/1604023128 learned_addr learned my addr 172.21.15.67:0/1604023128 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:36.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.320+0000 7f89e2ffd640 1 --2- 172.21.15.67:0/1604023128 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f89e40cde60 0x7f89e406b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:36.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.320+0000 7f89e3fff640 1 --2- 172.21.15.67:0/1604023128 >> v2:172.21.15.72:3300/0 conn(0x7f89e40b1840 0x7f89e40b1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:36.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.321+0000 7f89e37fe640 1 -- 172.21.15.67:0/1604023128 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f89e40cde60 msgr2=0x7f89e406b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:36.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.321+0000 7f89e37fe640 1 --2- 172.21.15.67:0/1604023128 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f89e40cde60 0x7f89e406b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.321+0000 7f89e37fe640 1 -- 172.21.15.67:0/1604023128 >> v2:172.21.15.72:3300/0 conn(0x7f89e40b1840 msgr2=0x7f89e40b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:36.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.321+0000 7f89e37fe640 1 --2- 172.21.15.67:0/1604023128 >> v2:172.21.15.72:3300/0 conn(0x7f89e40b1840 0x7f89e40b1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.321+0000 7f89e37fe640 1 -- 172.21.15.67:0/1604023128 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e40cd660 con 0x7f89e415bc90 2024-03-24T00:39:36.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.321+0000 7f89e37fe640 1 --2- 172.21.15.67:0/1604023128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89e415bc90 0x7f89e406b040 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f89e4072840 tx=0x7f89d402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e2385c043b7403f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:36.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.322+0000 7f89e27fc640 1 -- 172.21.15.67:0/1604023128 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d402ca70 con 0x7f89e415bc90 2024-03-24T00:39:36.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.322+0000 7f89e27fc640 1 -- 172.21.15.67:0/1604023128 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89d402cc10 con 0x7f89e415bc90 2024-03-24T00:39:36.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.322+0000 7f89e27fc640 1 -- 172.21.15.67:0/1604023128 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d40343f0 con 0x7f89e415bc90 2024-03-24T00:39:36.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.322+0000 7f89e8f9d640 1 -- 172.21.15.67:0/1604023128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89e415bc90 msgr2=0x7f89e406b040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:36.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.322+0000 7f89e8f9d640 1 --2- 172.21.15.67:0/1604023128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89e415bc90 0x7f89e406b040 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f89e4072840 tx=0x7f89d402bed0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.322+0000 7f89e8f9d640 1 -- 172.21.15.67:0/1604023128 shutdown_connections 2024-03-24T00:39:36.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.322+0000 7f89e8f9d640 1 --2- 172.21.15.67:0/1604023128 >> v2:172.21.15.72:3300/0 conn(0x7f89e40b1840 0x7f89e40b1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.322+0000 7f89e8f9d640 1 --2- 172.21.15.67:0/1604023128 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f89e40cde60 0x7f89e406b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.322+0000 7f89e8f9d640 1 --2- 172.21.15.67:0/1604023128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89e415bc90 0x7f89e406b040 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.322+0000 7f89e8f9d640 1 -- 172.21.15.67:0/1604023128 >> 172.21.15.67:0/1604023128 conn(0x7f89e40d0890 msgr2=0x7f89e40d0ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:36.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.322+0000 7f89e8f9d640 1 -- 172.21.15.67:0/1604023128 shutdown_connections 2024-03-24T00:39:36.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.322+0000 7f89e8f9d640 1 -- 172.21.15.67:0/1604023128 wait complete. 2024-03-24T00:39:36.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.323+0000 7f89e8f9d640 1 Processor -- start 2024-03-24T00:39:36.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.323+0000 7f89e8f9d640 1 -- start start 2024-03-24T00:39:36.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.324+0000 7f89e8f9d640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f89e40b1840 0x7f89e41f2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:36.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.324+0000 7f89e8f9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89e40cde60 0x7f89e41f0e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:36.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.324+0000 7f89e8f9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89e415bc90 0x7f89e415d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:36.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.324+0000 7f89e8f9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e406bbb0 con 0x7f89e415bc90 2024-03-24T00:39:36.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.324+0000 7f89e8f9d640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e40c4070 con 0x7f89e40cde60 2024-03-24T00:39:36.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.324+0000 7f89e8f9d640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89e40b36e0 con 0x7f89e40b1840 2024-03-24T00:39:36.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.324+0000 7f89e3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89e415bc90 0x7f89e415d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:36.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.324+0000 7f89e3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89e415bc90 0x7f89e415d7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50288/0 (socket says 172.21.15.67:50288) 2024-03-24T00:39:36.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.324+0000 7f89e3fff640 1 -- 172.21.15.67:0/691734387 learned_addr learned my addr 172.21.15.67:0/691734387 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:36.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.324+0000 7f89e37fe640 1 --2- 172.21.15.67:0/691734387 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f89e40b1840 0x7f89e41f2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:36.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.324+0000 7f89e2ffd640 1 --2- 172.21.15.67:0/691734387 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89e40cde60 0x7f89e41f0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:36.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.325+0000 7f89e3fff640 1 -- 172.21.15.67:0/691734387 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f89e40b1840 msgr2=0x7f89e41f2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:36.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.325+0000 7f89e3fff640 1 --2- 172.21.15.67:0/691734387 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f89e40b1840 0x7f89e41f2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.325+0000 7f89e3fff640 1 -- 172.21.15.67:0/691734387 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89e40cde60 msgr2=0x7f89e41f0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:36.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.325+0000 7f89e3fff640 1 --2- 172.21.15.67:0/691734387 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89e40cde60 0x7f89e41f0e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.325+0000 7f89e3fff640 1 -- 172.21.15.67:0/691734387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e41f5a40 con 0x7f89e415bc90 2024-03-24T00:39:36.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.325+0000 7f89e3fff640 1 --2- 172.21.15.67:0/691734387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89e415bc90 0x7f89e415d7e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f89dc00e930 tx=0x7f89dc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:36.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.325+0000 7f89dbfff640 1 -- 172.21.15.67:0/691734387 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89dc017070 con 0x7f89e415bc90 2024-03-24T00:39:36.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.325+0000 7f89dbfff640 1 -- 172.21.15.67:0/691734387 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89dc00c980 con 0x7f89e415bc90 2024-03-24T00:39:36.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.325+0000 7f89dbfff640 1 -- 172.21.15.67:0/691734387 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89dc00cd70 con 0x7f89e415bc90 2024-03-24T00:39:36.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.325+0000 7f89e8f9d640 1 -- 172.21.15.67:0/691734387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89e41f13b0 con 0x7f89e415bc90 2024-03-24T00:39:36.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.325+0000 7f89e8f9d640 1 -- 172.21.15.67:0/691734387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89e41f1670 con 0x7f89e415bc90 2024-03-24T00:39:36.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.326+0000 7f89e8f9d640 1 -- 172.21.15.67:0/691734387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89e4068560 con 0x7f89e415bc90 2024-03-24T00:39:36.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.326+0000 7f89dbfff640 1 -- 172.21.15.67:0/691734387 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f89dc01a070 con 0x7f89e415bc90 2024-03-24T00:39:36.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.327+0000 7f89dbfff640 1 --2- 172.21.15.67:0/691734387 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f89c406bbd0 0x7f89c406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:36.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.327+0000 7f89dbfff640 1 -- 172.21.15.67:0/691734387 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f89dc08adb0 con 0x7f89e415bc90 2024-03-24T00:39:36.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.327+0000 7f89e37fe640 1 --2- 172.21.15.67:0/691734387 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f89c406bbd0 0x7f89c406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:36.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.327+0000 7f89e37fe640 1 --2- 172.21.15.67:0/691734387 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f89c406bbd0 0x7f89c406e090 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f89d4035000 tx=0x7f89d4030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:36.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.332+0000 7f89dbfff640 1 -- 172.21.15.67:0/691734387 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f89dc058fa0 con 0x7f89e415bc90 2024-03-24T00:39:36.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.515+0000 7f89e8f9d640 1 -- 172.21.15.67:0/691734387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f89e40caf70 con 0x7f89e415bc90 2024-03-24T00:39:36.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.516+0000 7f89dbfff640 1 -- 172.21.15.67:0/691734387 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f89dc05d040 con 0x7f89e415bc90 2024-03-24T00:39:36.516 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-24T00:39:36.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.518+0000 7f89e8f9d640 1 -- 172.21.15.67:0/691734387 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f89c406bbd0 msgr2=0x7f89c406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:36.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.518+0000 7f89e8f9d640 1 --2- 172.21.15.67:0/691734387 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f89c406bbd0 0x7f89c406e090 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f89d4035000 tx=0x7f89d4030000 comp rx=0 tx=0).stop 2024-03-24T00:39:36.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.518+0000 7f89e8f9d640 1 -- 172.21.15.67:0/691734387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89e415bc90 msgr2=0x7f89e415d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:36.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.518+0000 7f89e8f9d640 1 --2- 172.21.15.67:0/691734387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89e415bc90 0x7f89e415d7e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f89dc00e930 tx=0x7f89dc00edf0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.518+0000 7f89e8f9d640 1 -- 172.21.15.67:0/691734387 shutdown_connections 2024-03-24T00:39:36.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.518+0000 7f89e8f9d640 1 --2- 172.21.15.67:0/691734387 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f89c406bbd0 0x7f89c406e090 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.518+0000 7f89e8f9d640 1 --2- 172.21.15.67:0/691734387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89e415bc90 0x7f89e415d7e0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.518+0000 7f89e8f9d640 1 --2- 172.21.15.67:0/691734387 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89e40cde60 0x7f89e41f0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.518+0000 7f89e8f9d640 1 --2- 172.21.15.67:0/691734387 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f89e40b1840 0x7f89e41f2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.518+0000 7f89e8f9d640 1 -- 172.21.15.67:0/691734387 >> 172.21.15.67:0/691734387 conn(0x7f89e40d0890 msgr2=0x7f89e40cc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:36.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.518+0000 7f89e8f9d640 1 -- 172.21.15.67:0/691734387 shutdown_connections 2024-03-24T00:39:36.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.518+0000 7f89e8f9d640 1 -- 172.21.15.67:0/691734387 wait complete. 2024-03-24T00:39:36.528 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_apply_tag' 2024-03-24T00:39:36.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.630+0000 7f4d8c6b5640 1 Processor -- start 2024-03-24T00:39:36.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.630+0000 7f4d8c6b5640 1 -- start start 2024-03-24T00:39:36.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.631+0000 7f4d8c6b5640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4d840d1550 0x7f4d8406cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:36.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.631+0000 7f4d8c6b5640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4d840b2780 0x7f4d840b2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:36.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.631+0000 7f4d8c6b5640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d840d1930 con 0x7f4d84099d70 2024-03-24T00:39:36.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.631+0000 7f4d8c6b5640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d8406d420 con 0x7f4d840d1550 2024-03-24T00:39:36.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.631+0000 7f4d8c6b5640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8406ca50 con 0x7f4d840b2780 2024-03-24T00:39:36.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.631+0000 7f4d8ac52640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4d84099d70 0x7f4d840cfd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41480/0 (socket says 172.21.15.67:41480) 2024-03-24T00:39:36.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.631+0000 7f4d8ac52640 1 -- 172.21.15.67:0/335968052 learned_addr learned my addr 172.21.15.67:0/335968052 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:36.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.631+0000 7f4d89c50640 1 --2- 172.21.15.67:0/335968052 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4d840b2780 0x7f4d840b2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:36.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.631+0000 7f4d8a451640 1 --2- 172.21.15.67:0/335968052 >> v2:172.21.15.72:3300/0 conn(0x7f4d840d1550 0x7f4d8406cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:36.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.631+0000 7f4d8944f640 1 -- 172.21.15.67:0/335968052 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3035317823 0 0) 0x7f4d840d1930 con 0x7f4d84099d70 2024-03-24T00:39:36.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.631+0000 7f4d8944f640 1 -- 172.21.15.67:0/335968052 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d70003520 con 0x7f4d84099d70 2024-03-24T00:39:36.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.632+0000 7f4d89c50640 1 -- 172.21.15.67:0/335968052 >> v2:172.21.15.72:3300/0 conn(0x7f4d840d1550 msgr2=0x7f4d8406cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:36.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.632+0000 7f4d89c50640 1 --2- 172.21.15.67:0/335968052 >> v2:172.21.15.72:3300/0 conn(0x7f4d840d1550 0x7f4d8406cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.632+0000 7f4d89c50640 1 -- 172.21.15.67:0/335968052 >> v1:172.21.15.67:6789/0 conn(0x7f4d84099d70 legacy=0x7f4d840cfd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:36.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.632+0000 7f4d89c50640 1 -- 172.21.15.67:0/335968052 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d840b1f60 con 0x7f4d840b2780 2024-03-24T00:39:36.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.632+0000 7f4d8a451640 1 --2- 172.21.15.67:0/335968052 >> v2:172.21.15.72:3300/0 conn(0x7f4d840d1550 0x7f4d8406cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:36.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.632+0000 7f4d89c50640 1 --2- 172.21.15.67:0/335968052 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4d840b2780 0x7f4d840b2c00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4d740097a0 tx=0x7f4d7402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9f73265a04243f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:36.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.633+0000 7f4d8944f640 1 -- 172.21.15.67:0/335968052 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d74035020 con 0x7f4d840b2780 2024-03-24T00:39:36.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.633+0000 7f4d8944f640 1 -- 172.21.15.67:0/335968052 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d74031830 con 0x7f4d840b2780 2024-03-24T00:39:36.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.633+0000 7f4d8944f640 1 -- 172.21.15.67:0/335968052 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d74031be0 con 0x7f4d840b2780 2024-03-24T00:39:36.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.633+0000 7f4d8c6b5640 1 -- 172.21.15.67:0/335968052 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4d840b2780 msgr2=0x7f4d840b2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:36.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.633+0000 7f4d8c6b5640 1 --2- 172.21.15.67:0/335968052 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4d840b2780 0x7f4d840b2c00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4d740097a0 tx=0x7f4d7402b8d0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.633+0000 7f4d8c6b5640 1 -- 172.21.15.67:0/335968052 shutdown_connections 2024-03-24T00:39:36.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.633+0000 7f4d8c6b5640 1 --2- 172.21.15.67:0/335968052 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4d840b2780 0x7f4d840b2c00 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.633+0000 7f4d8c6b5640 1 --2- 172.21.15.67:0/335968052 >> v2:172.21.15.72:3300/0 conn(0x7f4d840d1550 0x7f4d8406cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.633+0000 7f4d8c6b5640 1 -- 172.21.15.67:0/335968052 >> 172.21.15.67:0/335968052 conn(0x7f4d840b36e0 msgr2=0x7f4d840b3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:36.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.633+0000 7f4d8c6b5640 1 -- 172.21.15.67:0/335968052 shutdown_connections 2024-03-24T00:39:36.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.633+0000 7f4d8c6b5640 1 -- 172.21.15.67:0/335968052 wait complete. 2024-03-24T00:39:36.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.634+0000 7f4d8c6b5640 1 Processor -- start 2024-03-24T00:39:36.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.634+0000 7f4d8c6b5640 1 -- start start 2024-03-24T00:39:36.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.635+0000 7f4d8c6b5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d84099d70 0x7f4d840cab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:36.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.635+0000 7f4d8c6b5640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4d840b2780 0x7f4d841f4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:36.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.635+0000 7f4d8c6b5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4d840d1550 0x7f4d840b0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:36.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.635+0000 7f4d8c6b5640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8406c400 con 0x7f4d84099d70 2024-03-24T00:39:36.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.635+0000 7f4d8c6b5640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d840b2520 con 0x7f4d840d1550 2024-03-24T00:39:36.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.635+0000 7f4d8c6b5640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8406c6e0 con 0x7f4d840b2780 2024-03-24T00:39:36.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.635+0000 7f4d8a451640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d84099d70 0x7f4d840cab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:36.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.635+0000 7f4d8a451640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d84099d70 0x7f4d840cab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50304/0 (socket says 172.21.15.67:50304) 2024-03-24T00:39:36.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.635+0000 7f4d8a451640 1 -- 172.21.15.67:0/3706953192 learned_addr learned my addr 172.21.15.67:0/3706953192 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:36.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.635+0000 7f4d89c50640 1 --2- 172.21.15.67:0/3706953192 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4d840b2780 0x7f4d841f4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:36.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.635+0000 7f4d8ac52640 1 --2- 172.21.15.67:0/3706953192 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4d840d1550 0x7f4d840b0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:36.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.635+0000 7f4d8a451640 1 -- 172.21.15.67:0/3706953192 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4d840b2780 msgr2=0x7f4d841f4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:36.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.635+0000 7f4d8a451640 1 --2- 172.21.15.67:0/3706953192 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4d840b2780 0x7f4d841f4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.635+0000 7f4d8a451640 1 -- 172.21.15.67:0/3706953192 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4d840d1550 msgr2=0x7f4d840b0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:36.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.635+0000 7f4d8a451640 1 --2- 172.21.15.67:0/3706953192 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4d840d1550 0x7f4d840b0020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.636+0000 7f4d8a451640 1 -- 172.21.15.67:0/3706953192 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d840cb210 con 0x7f4d84099d70 2024-03-24T00:39:36.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.636+0000 7f4d8a451640 1 --2- 172.21.15.67:0/3706953192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d84099d70 0x7f4d840cab80 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c00e960 tx=0x7f4d7c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:36.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.636+0000 7f4d7affd640 1 -- 172.21.15.67:0/3706953192 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d7c017070 con 0x7f4d84099d70 2024-03-24T00:39:36.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.636+0000 7f4d7affd640 1 -- 172.21.15.67:0/3706953192 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d7c00ca00 con 0x7f4d84099d70 2024-03-24T00:39:36.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.636+0000 7f4d8c6b5640 1 -- 172.21.15.67:0/3706953192 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d840c8df0 con 0x7f4d84099d70 2024-03-24T00:39:36.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.636+0000 7f4d7affd640 1 -- 172.21.15.67:0/3706953192 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d7c00cdd0 con 0x7f4d84099d70 2024-03-24T00:39:36.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.636+0000 7f4d8c6b5640 1 -- 172.21.15.67:0/3706953192 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d840c9370 con 0x7f4d84099d70 2024-03-24T00:39:36.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.637+0000 7f4d8c6b5640 1 -- 172.21.15.67:0/3706953192 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d8404f400 con 0x7f4d84099d70 2024-03-24T00:39:36.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.637+0000 7f4d7affd640 1 -- 172.21.15.67:0/3706953192 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4d7c01a070 con 0x7f4d84099d70 2024-03-24T00:39:36.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.638+0000 7f4d7affd640 1 --2- 172.21.15.67:0/3706953192 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4d6806bbd0 0x7f4d6806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:36.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.638+0000 7f4d7affd640 1 -- 172.21.15.67:0/3706953192 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f4d7c08acb0 con 0x7f4d84099d70 2024-03-24T00:39:36.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.638+0000 7f4d89c50640 1 --2- 172.21.15.67:0/3706953192 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4d6806bbd0 0x7f4d6806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:36.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.638+0000 7f4d89c50640 1 --2- 172.21.15.67:0/3706953192 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4d6806bbd0 0x7f4d6806e090 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4d74036000 tx=0x7f4d7403c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:36.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.642+0000 7f4d7affd640 1 -- 172.21.15.67:0/3706953192 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4d7c058d60 con 0x7f4d84099d70 2024-03-24T00:39:36.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.895+0000 7f4d8c6b5640 1 -- 172.21.15.67:0/3706953192 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_apply_tag"]} v 0) v1 -- 0x7f4d840b1310 con 0x7f4d84099d70 2024-03-24T00:39:36.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.929+0000 7f4d7affd640 1 -- 172.21.15.67:0/3706953192 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_apply_tag"]}]=0 v65) v1 ==== 145+0+0 (secure 0 0 0) 0x7f4d7c05ce00 con 0x7f4d84099d70 2024-03-24T00:39:36.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.931+0000 7f4d8c6b5640 1 -- 172.21.15.67:0/3706953192 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4d6806bbd0 msgr2=0x7f4d6806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:36.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.931+0000 7f4d8c6b5640 1 --2- 172.21.15.67:0/3706953192 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4d6806bbd0 0x7f4d6806e090 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4d74036000 tx=0x7f4d7403c040 comp rx=0 tx=0).stop 2024-03-24T00:39:36.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.931+0000 7f4d8c6b5640 1 -- 172.21.15.67:0/3706953192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d84099d70 msgr2=0x7f4d840cab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:36.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.931+0000 7f4d8c6b5640 1 --2- 172.21.15.67:0/3706953192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d84099d70 0x7f4d840cab80 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c00e960 tx=0x7f4d7c00ee20 comp rx=0 tx=0).stop 2024-03-24T00:39:36.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.931+0000 7f4d8c6b5640 1 -- 172.21.15.67:0/3706953192 shutdown_connections 2024-03-24T00:39:36.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.932+0000 7f4d8c6b5640 1 --2- 172.21.15.67:0/3706953192 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4d6806bbd0 0x7f4d6806e090 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.932+0000 7f4d8c6b5640 1 --2- 172.21.15.67:0/3706953192 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4d840d1550 0x7f4d840b0020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.932+0000 7f4d8c6b5640 1 --2- 172.21.15.67:0/3706953192 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4d840b2780 0x7f4d841f4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.932+0000 7f4d8c6b5640 1 --2- 172.21.15.67:0/3706953192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d84099d70 0x7f4d840cab80 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:36.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.932+0000 7f4d8c6b5640 1 -- 172.21.15.67:0/3706953192 >> 172.21.15.67:0/3706953192 conn(0x7f4d840b36e0 msgr2=0x7f4d84161d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:36.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.932+0000 7f4d8c6b5640 1 -- 172.21.15.67:0/3706953192 shutdown_connections 2024-03-24T00:39:36.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:36.932+0000 7f4d8c6b5640 1 -- 172.21.15.67:0/3706953192 wait complete. 2024-03-24T00:39:36.942 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-24T00:39:37.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.046+0000 7f5f0feb2640 1 Processor -- start 2024-03-24T00:39:37.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.046+0000 7f5f0feb2640 1 -- start start 2024-03-24T00:39:37.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.047+0000 7f5f0feb2640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5f0815dea0 0x7f5f080d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:37.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.047+0000 7f5f0feb2640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5f0815e280 0x7f5f080d3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:37.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.047+0000 7f5f0feb2640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f080d2250 con 0x7f5f080b5a60 2024-03-24T00:39:37.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.047+0000 7f5f0feb2640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f080d2530 con 0x7f5f0815dea0 2024-03-24T00:39:37.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.047+0000 7f5f0feb2640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f080d2810 con 0x7f5f0815e280 2024-03-24T00:39:37.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.047+0000 7f5f0dc4e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5f0815dea0 0x7f5f080d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:37.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.047+0000 7f5f0dc4e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5f0815dea0 0x7f5f080d33d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50306/0 (socket says 172.21.15.67:50306) 2024-03-24T00:39:37.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.047+0000 7f5f0dc4e640 1 -- 172.21.15.67:0/3994066457 learned_addr learned my addr 172.21.15.67:0/3994066457 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:37.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.047+0000 7f5f0d44d640 1 --2- 172.21.15.67:0/3994066457 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5f0815e280 0x7f5f080d3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:37.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.048+0000 7f5f0dc4e640 1 -- 172.21.15.67:0/3994066457 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5f0815e280 msgr2=0x7f5f080d3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:37.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.048+0000 7f5f0dc4e640 1 --2- 172.21.15.67:0/3994066457 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5f0815e280 0x7f5f080d3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.048+0000 7f5f0dc4e640 1 -- 172.21.15.67:0/3994066457 >> v1:172.21.15.72:6789/0 conn(0x7f5f080b5a60 legacy=0x7f5f0806c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:37.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.048+0000 7f5f0dc4e640 1 -- 172.21.15.67:0/3994066457 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f080d2af0 con 0x7f5f0815dea0 2024-03-24T00:39:37.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.048+0000 7f5f0cc4c640 1 -- 172.21.15.67:0/3994066457 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2274017176 0 0) 0x7f5f080d2250 con 0x7f5f080b5a60 2024-03-24T00:39:37.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.048+0000 7f5f0dc4e640 1 --2- 172.21.15.67:0/3994066457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f0815dea0 0x7f5f080d33d0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f5ef8009e50 tx=0x7f5ef802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2c2a13b84c77660 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:37.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.049+0000 7f5f0cc4c640 1 -- 172.21.15.67:0/3994066457 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef802ca70 con 0x7f5f0815dea0 2024-03-24T00:39:37.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.049+0000 7f5f0cc4c640 1 -- 172.21.15.67:0/3994066457 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ef802cc10 con 0x7f5f0815dea0 2024-03-24T00:39:37.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.049+0000 7f5f0cc4c640 1 -- 172.21.15.67:0/3994066457 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef80343f0 con 0x7f5f0815dea0 2024-03-24T00:39:37.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.049+0000 7f5f0feb2640 1 -- 172.21.15.67:0/3994066457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f0815dea0 msgr2=0x7f5f080d33d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:37.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.049+0000 7f5f0feb2640 1 --2- 172.21.15.67:0/3994066457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f0815dea0 0x7f5f080d33d0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f5ef8009e50 tx=0x7f5ef802bed0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.049+0000 7f5f0feb2640 1 -- 172.21.15.67:0/3994066457 shutdown_connections 2024-03-24T00:39:37.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.049+0000 7f5f0feb2640 1 --2- 172.21.15.67:0/3994066457 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5f0815e280 0x7f5f080d3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.049+0000 7f5f0feb2640 1 --2- 172.21.15.67:0/3994066457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f0815dea0 0x7f5f080d33d0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.049+0000 7f5f0feb2640 1 -- 172.21.15.67:0/3994066457 >> 172.21.15.67:0/3994066457 conn(0x7f5f08072da0 msgr2=0x7f5f080d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:37.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.049+0000 7f5f0feb2640 1 -- 172.21.15.67:0/3994066457 shutdown_connections 2024-03-24T00:39:37.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.049+0000 7f5f0feb2640 1 -- 172.21.15.67:0/3994066457 wait complete. 2024-03-24T00:39:37.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.050+0000 7f5f0feb2640 1 Processor -- start 2024-03-24T00:39:37.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.050+0000 7f5f0feb2640 1 -- start start 2024-03-24T00:39:37.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.051+0000 7f5f0feb2640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5f080b5a60 0x7f5f0814d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:37.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.051+0000 7f5f0feb2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f0815dea0 0x7f5f0815a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:37.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.051+0000 7f5f0feb2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5f0815e280 0x7f5f0815ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:37.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.051+0000 7f5f0feb2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f080b5ed0 con 0x7f5f0815dea0 2024-03-24T00:39:37.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.051+0000 7f5f0feb2640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f080b8350 con 0x7f5f0815e280 2024-03-24T00:39:37.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.051+0000 7f5f0feb2640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f080b6050 con 0x7f5f080b5a60 2024-03-24T00:39:37.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.051+0000 7f5f0d44d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f0815dea0 0x7f5f0815a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:37.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.051+0000 7f5f0d44d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f0815dea0 0x7f5f0815a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50314/0 (socket says 172.21.15.67:50314) 2024-03-24T00:39:37.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.051+0000 7f5f0d44d640 1 -- 172.21.15.67:0/1303801559 learned_addr learned my addr 172.21.15.67:0/1303801559 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:37.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.051+0000 7f5f0dc4e640 1 --2- 172.21.15.67:0/1303801559 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5f080b5a60 0x7f5f0814d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:37.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.051+0000 7f5f0e44f640 1 --2- 172.21.15.67:0/1303801559 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5f0815e280 0x7f5f0815ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:37.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.051+0000 7f5f0d44d640 1 -- 172.21.15.67:0/1303801559 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5f080b5a60 msgr2=0x7f5f0814d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:37.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.051+0000 7f5f0d44d640 1 --2- 172.21.15.67:0/1303801559 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5f080b5a60 0x7f5f0814d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.051+0000 7f5f0d44d640 1 -- 172.21.15.67:0/1303801559 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5f0815e280 msgr2=0x7f5f0815ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:37.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.052+0000 7f5f0d44d640 1 --2- 172.21.15.67:0/1303801559 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5f0815e280 0x7f5f0815ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.052+0000 7f5f0d44d640 1 -- 172.21.15.67:0/1303801559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f0815a580 con 0x7f5f0815dea0 2024-03-24T00:39:37.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.052+0000 7f5f0d44d640 1 --2- 172.21.15.67:0/1303801559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f0815dea0 0x7f5f0815a040 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f5efc00e870 tx=0x7f5efc00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:37.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.052+0000 7f5f067fc640 1 -- 172.21.15.67:0/1303801559 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5efc017070 con 0x7f5f0815dea0 2024-03-24T00:39:37.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.052+0000 7f5f067fc640 1 -- 172.21.15.67:0/1303801559 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5efc00c8e0 con 0x7f5f0815dea0 2024-03-24T00:39:37.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.052+0000 7f5f0feb2640 1 -- 172.21.15.67:0/1303801559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f080cb630 con 0x7f5f0815dea0 2024-03-24T00:39:37.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.052+0000 7f5f067fc640 1 -- 172.21.15.67:0/1303801559 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5efc00ccd0 con 0x7f5f0815dea0 2024-03-24T00:39:37.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.052+0000 7f5f0feb2640 1 -- 172.21.15.67:0/1303801559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f080cbbf0 con 0x7f5f0815dea0 2024-03-24T00:39:37.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.053+0000 7f5f0feb2640 1 -- 172.21.15.67:0/1303801559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f0804f400 con 0x7f5f0815dea0 2024-03-24T00:39:37.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.053+0000 7f5f067fc640 1 -- 172.21.15.67:0/1303801559 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5efc01a070 con 0x7f5f0815dea0 2024-03-24T00:39:37.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.054+0000 7f5f067fc640 1 --2- 172.21.15.67:0/1303801559 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5ef406bc00 0x7f5ef406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:37.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.054+0000 7f5f067fc640 1 -- 172.21.15.67:0/1303801559 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f5efc08ab10 con 0x7f5f0815dea0 2024-03-24T00:39:37.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.054+0000 7f5f0dc4e640 1 --2- 172.21.15.67:0/1303801559 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5ef406bc00 0x7f5ef406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:37.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.054+0000 7f5f0dc4e640 1 --2- 172.21.15.67:0/1303801559 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5ef406bc00 0x7f5ef406e0c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5ef8035000 tx=0x7f5ef8030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:37.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.058+0000 7f5f067fc640 1 -- 172.21.15.67:0/1303801559 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5efc058c40 con 0x7f5f0815dea0 2024-03-24T00:39:37.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.242+0000 7f5f0feb2640 1 -- 172.21.15.67:0/1303801559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f5f080683b0 con 0x7f5f0815dea0 2024-03-24T00:39:37.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.247+0000 7f5f067fc640 1 -- 172.21.15.67:0/1303801559 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5efc05cce0 con 0x7f5f0815dea0 2024-03-24T00:39:37.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.249+0000 7f5f0feb2640 1 -- 172.21.15.67:0/1303801559 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5ef406bc00 msgr2=0x7f5ef406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:37.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.249+0000 7f5f0feb2640 1 --2- 172.21.15.67:0/1303801559 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5ef406bc00 0x7f5ef406e0c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5ef8035000 tx=0x7f5ef8030000 comp rx=0 tx=0).stop 2024-03-24T00:39:37.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.249+0000 7f5f0feb2640 1 -- 172.21.15.67:0/1303801559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f0815dea0 msgr2=0x7f5f0815a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:37.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.249+0000 7f5f0feb2640 1 --2- 172.21.15.67:0/1303801559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f0815dea0 0x7f5f0815a040 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f5efc00e870 tx=0x7f5efc00ed30 comp rx=0 tx=0).stop 2024-03-24T00:39:37.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.249+0000 7f5f0feb2640 1 -- 172.21.15.67:0/1303801559 shutdown_connections 2024-03-24T00:39:37.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.249+0000 7f5f0feb2640 1 --2- 172.21.15.67:0/1303801559 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5ef406bc00 0x7f5ef406e0c0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.249+0000 7f5f0feb2640 1 --2- 172.21.15.67:0/1303801559 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5f0815e280 0x7f5f0815ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.249+0000 7f5f0feb2640 1 --2- 172.21.15.67:0/1303801559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f0815dea0 0x7f5f0815a040 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.249+0000 7f5f0feb2640 1 --2- 172.21.15.67:0/1303801559 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5f080b5a60 0x7f5f0814d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.249+0000 7f5f0feb2640 1 -- 172.21.15.67:0/1303801559 >> 172.21.15.67:0/1303801559 conn(0x7f5f08072da0 msgr2=0x7f5f080d3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:37.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.249+0000 7f5f0feb2640 1 -- 172.21.15.67:0/1303801559 shutdown_connections 2024-03-24T00:39:37.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.249+0000 7f5f0feb2640 1 -- 172.21.15.67:0/1303801559 wait complete. 2024-03-24T00:39:37.259 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-24T00:39:37.259 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:39:37.298 DEBUG:teuthology.orchestra.run.smithi072:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:39:37.344 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-24T00:39:37.344 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:39:37.344 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:39:37.359 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:39:37.359 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:39:37.414 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:39:37.414 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:39:37.469 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:39:37.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.610+0000 7fb8401c4640 1 Processor -- start 2024-03-24T00:39:37.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.610+0000 7fb8401c4640 1 -- start start 2024-03-24T00:39:37.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.611+0000 7fb8401c4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb83815dec0 0x7fb8380ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:37.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.611+0000 7fb8401c4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb8380d0ef0 0x7fb838162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:37.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.611+0000 7fb8401c4640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8380cf640 con 0x7fb83815e2a0 2024-03-24T00:39:37.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.611+0000 7fb8401c4640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8380cee30 con 0x7fb8380d0ef0 2024-03-24T00:39:37.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.611+0000 7fb8401c4640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8380d2120 con 0x7fb83815dec0 2024-03-24T00:39:37.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.611+0000 7fb83e761640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb8380d0ef0 0x7fb838162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:37.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.611+0000 7fb83e761640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb8380d0ef0 0x7fb838162630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50320/0 (socket says 172.21.15.67:50320) 2024-03-24T00:39:37.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.611+0000 7fb83df60640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb83815dec0 0x7fb8380ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:37.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.611+0000 7fb83e761640 1 -- 172.21.15.67:0/1114153364 learned_addr learned my addr 172.21.15.67:0/1114153364 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:37.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.612+0000 7fb83e761640 1 -- 172.21.15.67:0/1114153364 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb83815dec0 msgr2=0x7fb8380ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:37.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.612+0000 7fb83cf5e640 1 -- 172.21.15.67:0/1114153364 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 87549943 0 0) 0x7fb828001110 con 0x7fb83815e2a0 2024-03-24T00:39:37.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.612+0000 7fb83e761640 1 --2- 172.21.15.67:0/1114153364 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb83815dec0 0x7fb8380ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.612+0000 7fb83e761640 1 -- 172.21.15.67:0/1114153364 >> v1:172.21.15.72:6789/0 conn(0x7fb83815e2a0 legacy=0x7fb8380d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:37.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.612+0000 7fb83e761640 1 -- 172.21.15.67:0/1114153364 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8380cf080 con 0x7fb8380d0ef0 2024-03-24T00:39:37.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.612+0000 7fb83e761640 1 --2- 172.21.15.67:0/1114153364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8380d0ef0 0x7fb838162630 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fb834009e50 tx=0x7fb83402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f486804f828d9270 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:37.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.613+0000 7fb83cf5e640 1 -- 172.21.15.67:0/1114153364 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb83402ca70 con 0x7fb8380d0ef0 2024-03-24T00:39:37.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.613+0000 7fb83cf5e640 1 -- 172.21.15.67:0/1114153364 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb83402cc10 con 0x7fb8380d0ef0 2024-03-24T00:39:37.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.613+0000 7fb83cf5e640 1 -- 172.21.15.67:0/1114153364 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8340343f0 con 0x7fb8380d0ef0 2024-03-24T00:39:37.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.613+0000 7fb8401c4640 1 -- 172.21.15.67:0/1114153364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8380d0ef0 msgr2=0x7fb838162630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:37.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.613+0000 7fb8401c4640 1 --2- 172.21.15.67:0/1114153364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8380d0ef0 0x7fb838162630 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fb834009e50 tx=0x7fb83402bed0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.613+0000 7fb8401c4640 1 -- 172.21.15.67:0/1114153364 shutdown_connections 2024-03-24T00:39:37.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.613+0000 7fb8401c4640 1 --2- 172.21.15.67:0/1114153364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8380d0ef0 0x7fb838162630 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.613+0000 7fb8401c4640 1 --2- 172.21.15.67:0/1114153364 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb83815dec0 0x7fb8380ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.613+0000 7fb8401c4640 1 -- 172.21.15.67:0/1114153364 >> 172.21.15.67:0/1114153364 conn(0x7fb83806c810 msgr2=0x7fb83806cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:37.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.613+0000 7fb8401c4640 1 -- 172.21.15.67:0/1114153364 shutdown_connections 2024-03-24T00:39:37.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.613+0000 7fb8401c4640 1 -- 172.21.15.67:0/1114153364 wait complete. 2024-03-24T00:39:37.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.614+0000 7fb8401c4640 1 Processor -- start 2024-03-24T00:39:37.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.614+0000 7fb8401c4640 1 -- start start 2024-03-24T00:39:37.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.615+0000 7fb8401c4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb8380d0ef0 0x7fb8381f3a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:37.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.615+0000 7fb8401c4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb83815dec0 0x7fb8380caea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:37.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.615+0000 7fb8401c4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb83815e2a0 0x7fb8380b50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:37.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.615+0000 7fb8401c4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8380d2730 con 0x7fb83815e2a0 2024-03-24T00:39:37.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.615+0000 7fb8401c4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8380d25b0 con 0x7fb8380d0ef0 2024-03-24T00:39:37.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.615+0000 7fb8401c4640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8380d28b0 con 0x7fb83815dec0 2024-03-24T00:39:37.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.615+0000 7fb83e761640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb83815e2a0 0x7fb8380b50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:37.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.615+0000 7fb83e761640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb83815e2a0 0x7fb8380b50d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50322/0 (socket says 172.21.15.67:50322) 2024-03-24T00:39:37.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.615+0000 7fb83e761640 1 -- 172.21.15.67:0/2589480910 learned_addr learned my addr 172.21.15.67:0/2589480910 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:37.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.615+0000 7fb83d75f640 1 --2- 172.21.15.67:0/2589480910 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb83815dec0 0x7fb8380caea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:37.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.615+0000 7fb83df60640 1 --2- 172.21.15.67:0/2589480910 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb8380d0ef0 0x7fb8381f3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:37.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.615+0000 7fb83e761640 1 -- 172.21.15.67:0/2589480910 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb83815dec0 msgr2=0x7fb8380caea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:37.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.615+0000 7fb83e761640 1 --2- 172.21.15.67:0/2589480910 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb83815dec0 0x7fb8380caea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.615+0000 7fb83e761640 1 -- 172.21.15.67:0/2589480910 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb8380d0ef0 msgr2=0x7fb8381f3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:37.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.615+0000 7fb83e761640 1 --2- 172.21.15.67:0/2589480910 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb8380d0ef0 0x7fb8381f3a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.615+0000 7fb83e761640 1 -- 172.21.15.67:0/2589480910 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8380cb960 con 0x7fb83815e2a0 2024-03-24T00:39:37.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.616+0000 7fb83e761640 1 --2- 172.21.15.67:0/2589480910 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb83815e2a0 0x7fb8380b50d0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fb834009e50 tx=0x7fb834004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:37.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.616+0000 7fb82e7fc640 1 -- 172.21.15.67:0/2589480910 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb834043030 con 0x7fb83815e2a0 2024-03-24T00:39:37.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.616+0000 7fb82e7fc640 1 -- 172.21.15.67:0/2589480910 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb83403e450 con 0x7fb83815e2a0 2024-03-24T00:39:37.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.616+0000 7fb8401c4640 1 -- 172.21.15.67:0/2589480910 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8380cbc40 con 0x7fb83815e2a0 2024-03-24T00:39:37.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.616+0000 7fb82e7fc640 1 -- 172.21.15.67:0/2589480910 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb834009510 con 0x7fb83815e2a0 2024-03-24T00:39:37.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.616+0000 7fb8401c4640 1 -- 172.21.15.67:0/2589480910 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8380cc200 con 0x7fb83815e2a0 2024-03-24T00:39:37.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.617+0000 7fb8401c4640 1 -- 172.21.15.67:0/2589480910 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb83804f400 con 0x7fb83815e2a0 2024-03-24T00:39:37.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.617+0000 7fb82e7fc640 1 -- 172.21.15.67:0/2589480910 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb834046070 con 0x7fb83815e2a0 2024-03-24T00:39:37.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.617+0000 7fb82e7fc640 1 --2- 172.21.15.67:0/2589480910 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb81c06bc00 0x7fb81c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:37.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.618+0000 7fb82e7fc640 1 -- 172.21.15.67:0/2589480910 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fb8340ad050 con 0x7fb83815e2a0 2024-03-24T00:39:37.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.618+0000 7fb83df60640 1 --2- 172.21.15.67:0/2589480910 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb81c06bc00 0x7fb81c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:37.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.618+0000 7fb83df60640 1 --2- 172.21.15.67:0/2589480910 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb81c06bc00 0x7fb81c06e0c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb830009b40 tx=0x7fb830009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:37.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.622+0000 7fb82e7fc640 1 -- 172.21.15.67:0/2589480910 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb83404d070 con 0x7fb83815e2a0 2024-03-24T00:39:37.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.805+0000 7fb8401c4640 1 -- 172.21.15.67:0/2589480910 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb8380d1aa0 con 0x7fb83815e2a0 2024-03-24T00:39:37.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.806+0000 7fb82e7fc640 1 -- 172.21.15.67:0/2589480910 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18843 (secure 0 0 0) 0x7fb83407b210 con 0x7fb83815e2a0 2024-03-24T00:39:37.806 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:39:37.806 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":27,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:39:32.904729+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:38:50.077085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:38:51.095454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T00:38:54.807117+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.14","mappings":[{"from":4,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:39:37.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.808+0000 7fb8401c4640 1 -- 172.21.15.67:0/2589480910 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb81c06bc00 msgr2=0x7fb81c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:37.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.808+0000 7fb8401c4640 1 --2- 172.21.15.67:0/2589480910 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb81c06bc00 0x7fb81c06e0c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb830009b40 tx=0x7fb830009190 comp rx=0 tx=0).stop 2024-03-24T00:39:37.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.808+0000 7fb8401c4640 1 -- 172.21.15.67:0/2589480910 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb83815e2a0 msgr2=0x7fb8380b50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:37.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.808+0000 7fb8401c4640 1 --2- 172.21.15.67:0/2589480910 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb83815e2a0 0x7fb8380b50d0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fb834009e50 tx=0x7fb834004a40 comp rx=0 tx=0).stop 2024-03-24T00:39:37.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.808+0000 7fb8401c4640 1 -- 172.21.15.67:0/2589480910 shutdown_connections 2024-03-24T00:39:37.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.808+0000 7fb8401c4640 1 --2- 172.21.15.67:0/2589480910 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb81c06bc00 0x7fb81c06e0c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.808+0000 7fb8401c4640 1 --2- 172.21.15.67:0/2589480910 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb83815e2a0 0x7fb8380b50d0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.808+0000 7fb8401c4640 1 --2- 172.21.15.67:0/2589480910 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb83815dec0 0x7fb8380caea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.808+0000 7fb8401c4640 1 --2- 172.21.15.67:0/2589480910 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb8380d0ef0 0x7fb8381f3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.808+0000 7fb8401c4640 1 -- 172.21.15.67:0/2589480910 >> 172.21.15.67:0/2589480910 conn(0x7fb83806c810 msgr2=0x7fb8380ca1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:37.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.808+0000 7fb8401c4640 1 -- 172.21.15.67:0/2589480910 shutdown_connections 2024-03-24T00:39:37.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.808+0000 7fb8401c4640 1 -- 172.21.15.67:0/2589480910 wait complete. 2024-03-24T00:39:37.819 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:38:39.364950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:38:50.077085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:38:51.095454+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T00:38:54.807117+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T00:39:37.819 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:39:37.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.924+0000 7f9440a59640 1 Processor -- start 2024-03-24T00:39:37.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.924+0000 7f9440a59640 1 -- start start 2024-03-24T00:39:37.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.925+0000 7f9440a59640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f943c099d00 0x7f943c09a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:37.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.925+0000 7f9440a59640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f943c0d1530 0x7f943c15d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:37.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.925+0000 7f9440a59640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f943c09a620 con 0x7f943c0b49b0 2024-03-24T00:39:37.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.925+0000 7f9440a59640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f943c0cc1b0 con 0x7f943c0d1530 2024-03-24T00:39:37.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.925+0000 7f9440a59640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f943c0cc400 con 0x7f943c099d00 2024-03-24T00:39:37.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.925+0000 7f943ad9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f943c0d1530 0x7f943c15d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:37.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.925+0000 7f943a59c640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f943c099d00 0x7f943c09a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:37.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.925+0000 7f9439d9b640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f943c0b49b0 0x7f943c0d0e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:39550/0 (socket says 172.21.15.67:39550) 2024-03-24T00:39:37.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.925+0000 7f9439d9b640 1 -- 172.21.15.67:0/3492726560 learned_addr learned my addr 172.21.15.67:0/3492726560 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:37.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.925+0000 7f943ad9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f943c0d1530 0x7f943c15d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:51042/0 (socket says 172.21.15.67:51042) 2024-03-24T00:39:37.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.925+0000 7f943ad9d640 1 -- 172.21.15.67:0/3492726560 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f943c099d00 msgr2=0x7f943c09a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:37.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.925+0000 7f943ad9d640 1 --2- 172.21.15.67:0/3492726560 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f943c099d00 0x7f943c09a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.925+0000 7f943ad9d640 1 -- 172.21.15.67:0/3492726560 >> v1:172.21.15.72:6789/0 conn(0x7f943c0b49b0 legacy=0x7f943c0d0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:37.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.925+0000 7f943959a640 1 -- 172.21.15.67:0/3492726560 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1281560673 0 0) 0x7f943c09a620 con 0x7f943c0b49b0 2024-03-24T00:39:37.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.925+0000 7f943ad9d640 1 -- 172.21.15.67:0/3492726560 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f943c0cc6e0 con 0x7f943c0d1530 2024-03-24T00:39:37.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.926+0000 7f943a59c640 1 --2- 172.21.15.67:0/3492726560 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f943c099d00 0x7f943c09a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:37.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.926+0000 7f943ad9d640 1 --2- 172.21.15.67:0/3492726560 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f943c0d1530 0x7f943c15d7d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f942c009d90 tx=0x7f942c02bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6396da3afb48f474 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:37.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.926+0000 7f943959a640 1 -- 172.21.15.67:0/3492726560 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f942c039070 con 0x7f943c0d1530 2024-03-24T00:39:37.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.926+0000 7f943959a640 1 -- 172.21.15.67:0/3492726560 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f942c02cc50 con 0x7f943c0d1530 2024-03-24T00:39:37.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.926+0000 7f943959a640 1 -- 172.21.15.67:0/3492726560 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f942c034440 con 0x7f943c0d1530 2024-03-24T00:39:37.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.927+0000 7f9440a59640 1 -- 172.21.15.67:0/3492726560 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f943c0d1530 msgr2=0x7f943c15d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:37.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.927+0000 7f9440a59640 1 --2- 172.21.15.67:0/3492726560 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f943c0d1530 0x7f943c15d7d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f942c009d90 tx=0x7f942c02bea0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.927+0000 7f9440a59640 1 -- 172.21.15.67:0/3492726560 shutdown_connections 2024-03-24T00:39:37.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.927+0000 7f9440a59640 1 --2- 172.21.15.67:0/3492726560 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f943c0d1530 0x7f943c15d7d0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.927+0000 7f9440a59640 1 --2- 172.21.15.67:0/3492726560 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f943c099d00 0x7f943c09a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.927+0000 7f9440a59640 1 -- 172.21.15.67:0/3492726560 >> 172.21.15.67:0/3492726560 conn(0x7f943c0b3650 msgr2=0x7f943c0b3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:37.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.927+0000 7f9440a59640 1 -- 172.21.15.67:0/3492726560 shutdown_connections 2024-03-24T00:39:37.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.927+0000 7f9440a59640 1 -- 172.21.15.67:0/3492726560 wait complete. 2024-03-24T00:39:37.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.928+0000 7f9440a59640 1 Processor -- start 2024-03-24T00:39:37.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.928+0000 7f9440a59640 1 -- start start 2024-03-24T00:39:37.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.929+0000 7f9440a59640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f943c099d00 0x7f943c0cb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:37.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.929+0000 7f9440a59640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f943c0b49b0 0x7f943c156bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:37.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.929+0000 7f9440a59640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f943c1570f0 0x7f943c1f72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:37.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.929+0000 7f9440a59640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f943c0d01c0 con 0x7f943c1570f0 2024-03-24T00:39:37.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.929+0000 7f9440a59640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f943c0ccca0 con 0x7f943c099d00 2024-03-24T00:39:37.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.929+0000 7f9440a59640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f943c0b32a0 con 0x7f943c0b49b0 2024-03-24T00:39:37.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.929+0000 7f943ad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f943c1570f0 0x7f943c1f72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:37.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.929+0000 7f9439d9b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f943c0b49b0 0x7f943c156bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:37.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.929+0000 7f943a59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f943c099d00 0x7f943c0cb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:37.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.929+0000 7f9439d9b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f943c0b49b0 0x7f943c156bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.67:33824/0 (socket says 172.21.15.67:33824) 2024-03-24T00:39:37.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.929+0000 7f943a59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f943c099d00 0x7f943c0cb1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:51048/0 (socket says 172.21.15.67:51048) 2024-03-24T00:39:37.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.929+0000 7f943ad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f943c1570f0 0x7f943c1f72b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50336/0 (socket says 172.21.15.67:50336) 2024-03-24T00:39:37.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.929+0000 7f9439d9b640 1 -- 172.21.15.67:0/851029540 learned_addr learned my addr 172.21.15.67:0/851029540 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:37.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.930+0000 7f9439d9b640 1 -- 172.21.15.67:0/851029540 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f943c099d00 msgr2=0x7f943c0cb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:37.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.930+0000 7f9439d9b640 1 --2- 172.21.15.67:0/851029540 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f943c099d00 0x7f943c0cb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.930+0000 7f9439d9b640 1 -- 172.21.15.67:0/851029540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f943c1570f0 msgr2=0x7f943c1f72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:37.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.930+0000 7f9439d9b640 1 --2- 172.21.15.67:0/851029540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f943c1570f0 0x7f943c1f72b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:37.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.930+0000 7f9439d9b640 1 -- 172.21.15.67:0/851029540 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f943c1f3ce0 con 0x7f943c0b49b0 2024-03-24T00:39:37.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.930+0000 7f943a59c640 1 --2- 172.21.15.67:0/851029540 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f943c099d00 0x7f943c0cb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:37.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.930+0000 7f9439d9b640 1 --2- 172.21.15.67:0/851029540 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f943c0b49b0 0x7f943c156bb0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f942400c4d0 tx=0x7f942400c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:37.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.932+0000 7f9432ffd640 1 -- 172.21.15.67:0/851029540 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9424018070 con 0x7f943c0b49b0 2024-03-24T00:39:37.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.932+0000 7f9432ffd640 1 -- 172.21.15.67:0/851029540 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9424007500 con 0x7f943c0b49b0 2024-03-24T00:39:37.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.932+0000 7f9432ffd640 1 -- 172.21.15.67:0/851029540 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9424004a20 con 0x7f943c0b49b0 2024-03-24T00:39:37.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.932+0000 7f9440a59640 1 -- 172.21.15.67:0/851029540 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f943c1f3fc0 con 0x7f943c0b49b0 2024-03-24T00:39:37.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.932+0000 7f9440a59640 1 -- 172.21.15.67:0/851029540 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f943c1575a0 con 0x7f943c0b49b0 2024-03-24T00:39:37.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.933+0000 7f9440a59640 1 -- 172.21.15.67:0/851029540 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f943c1f4190 con 0x7f943c0b49b0 2024-03-24T00:39:37.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.933+0000 7f9432ffd640 1 -- 172.21.15.67:0/851029540 <== mon.2 v2:172.21.15.72:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f942401b070 con 0x7f943c0b49b0 2024-03-24T00:39:37.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.934+0000 7f9432ffd640 1 --2- 172.21.15.67:0/851029540 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f941c06bbd0 0x7f941c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:37.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.934+0000 7f9432ffd640 1 -- 172.21.15.67:0/851029540 <== mon.2 v2:172.21.15.72:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f942408a190 con 0x7f943c0b49b0 2024-03-24T00:39:37.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.934+0000 7f943a59c640 1 --2- 172.21.15.67:0/851029540 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f941c06bbd0 0x7f941c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:37.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.934+0000 7f943a59c640 1 --2- 172.21.15.67:0/851029540 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f941c06bbd0 0x7f941c06e090 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9434009560 tx=0x7f9434009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:37.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:37.938+0000 7f9432ffd640 1 -- 172.21.15.67:0/851029540 <== mon.2 v2:172.21.15.72:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9424058190 con 0x7f943c0b49b0 2024-03-24T00:39:38.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.120+0000 7f9440a59640 1 -- 172.21.15.67:0/851029540 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f943c04f400 con 0x7f943c0b49b0 2024-03-24T00:39:38.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.120+0000 7f9432ffd640 1 -- 172.21.15.67:0/851029540 <== mon.2 v2:172.21.15.72:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f942405c230 con 0x7f943c0b49b0 2024-03-24T00:39:38.120 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-24T00:39:38.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.122+0000 7f9440a59640 1 -- 172.21.15.67:0/851029540 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f941c06bbd0 msgr2=0x7f941c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:38.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.122+0000 7f9440a59640 1 --2- 172.21.15.67:0/851029540 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f941c06bbd0 0x7f941c06e090 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9434009560 tx=0x7f9434009190 comp rx=0 tx=0).stop 2024-03-24T00:39:38.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.122+0000 7f9440a59640 1 -- 172.21.15.67:0/851029540 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f943c0b49b0 msgr2=0x7f943c156bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:38.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.122+0000 7f9440a59640 1 --2- 172.21.15.67:0/851029540 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f943c0b49b0 0x7f943c156bb0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f942400c4d0 tx=0x7f942400c990 comp rx=0 tx=0).stop 2024-03-24T00:39:38.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.122+0000 7f9440a59640 1 -- 172.21.15.67:0/851029540 shutdown_connections 2024-03-24T00:39:38.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.122+0000 7f9440a59640 1 --2- 172.21.15.67:0/851029540 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f941c06bbd0 0x7f941c06e090 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.122+0000 7f9440a59640 1 --2- 172.21.15.67:0/851029540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f943c1570f0 0x7f943c1f72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.122+0000 7f9440a59640 1 --2- 172.21.15.67:0/851029540 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f943c0b49b0 0x7f943c156bb0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.122+0000 7f9440a59640 1 --2- 172.21.15.67:0/851029540 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f943c099d00 0x7f943c0cb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.122+0000 7f9440a59640 1 -- 172.21.15.67:0/851029540 >> 172.21.15.67:0/851029540 conn(0x7f943c0b3650 msgr2=0x7f943c06cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:38.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.122+0000 7f9440a59640 1 -- 172.21.15.67:0/851029540 shutdown_connections 2024-03-24T00:39:38.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.122+0000 7f9440a59640 1 -- 172.21.15.67:0/851029540 wait complete. 2024-03-24T00:39:38.133 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:39:38.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.237+0000 7f1681fb3640 1 Processor -- start 2024-03-24T00:39:38.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.237+0000 7f1681fb3640 1 -- start start 2024-03-24T00:39:38.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.237+0000 7f1681fb3640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f167c15dfd0 0x7f167c0cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:38.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.237+0000 7f1681fb3640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f167c0b41e0 0x7f167c0b3620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:38.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.237+0000 7f1681fb3640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f167c0b3b60 0x7f167c161b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:38.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.237+0000 7f1681fb3640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f167c0cc260 con 0x7f167c0b41e0 2024-03-24T00:39:38.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.237+0000 7f1681fb3640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f167c0cf090 con 0x7f167c0b3b60 2024-03-24T00:39:38.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.237+0000 7f1681fb3640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f167c0d0320 con 0x7f167c15dfd0 2024-03-24T00:39:38.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.238+0000 7f167affd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f167c0b41e0 0x7f167c0b3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:38.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.238+0000 7f167affd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f167c0b41e0 0x7f167c0b3620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50338/0 (socket says 172.21.15.67:50338) 2024-03-24T00:39:38.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.238+0000 7f167affd640 1 -- 172.21.15.67:0/3860623021 learned_addr learned my addr 172.21.15.67:0/3860623021 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:38.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.238+0000 7f167b7fe640 1 --2- 172.21.15.67:0/3860623021 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f167c15dfd0 0x7f167c0cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:38.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.238+0000 7f167bfff640 1 --2- 172.21.15.67:0/3860623021 >> v2:172.21.15.72:3300/0 conn(0x7f167c0b3b60 0x7f167c161b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:38.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.238+0000 7f167affd640 1 -- 172.21.15.67:0/3860623021 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f167c15dfd0 msgr2=0x7f167c0cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:38.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.238+0000 7f167affd640 1 --2- 172.21.15.67:0/3860623021 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f167c15dfd0 0x7f167c0cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.238+0000 7f167affd640 1 -- 172.21.15.67:0/3860623021 >> v2:172.21.15.72:3300/0 conn(0x7f167c0b3b60 msgr2=0x7f167c161b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:38.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.238+0000 7f167affd640 1 --2- 172.21.15.67:0/3860623021 >> v2:172.21.15.72:3300/0 conn(0x7f167c0b3b60 0x7f167c161b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.238+0000 7f167affd640 1 -- 172.21.15.67:0/3860623021 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f167c0d0600 con 0x7f167c0b41e0 2024-03-24T00:39:38.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.238+0000 7f167bfff640 1 --2- 172.21.15.67:0/3860623021 >> v2:172.21.15.72:3300/0 conn(0x7f167c0b3b60 0x7f167c161b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:38.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.238+0000 7f167b7fe640 1 --2- 172.21.15.67:0/3860623021 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f167c15dfd0 0x7f167c0cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:38.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.239+0000 7f167affd640 1 --2- 172.21.15.67:0/3860623021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f167c0b41e0 0x7f167c0b3620 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f167c070cc0 tx=0x7f166c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f1f19a7c91fba52 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:38.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.239+0000 7f167a7fc640 1 -- 172.21.15.67:0/3860623021 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f166c034070 con 0x7f167c0b41e0 2024-03-24T00:39:38.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.239+0000 7f167a7fc640 1 -- 172.21.15.67:0/3860623021 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f166c030860 con 0x7f167c0b41e0 2024-03-24T00:39:38.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.239+0000 7f167a7fc640 1 -- 172.21.15.67:0/3860623021 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f166c030c30 con 0x7f167c0b41e0 2024-03-24T00:39:38.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.239+0000 7f1681fb3640 1 -- 172.21.15.67:0/3860623021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f167c0b41e0 msgr2=0x7f167c0b3620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:38.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.239+0000 7f1681fb3640 1 --2- 172.21.15.67:0/3860623021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f167c0b41e0 0x7f167c0b3620 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f167c070cc0 tx=0x7f166c02b9a0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.239+0000 7f1681fb3640 1 -- 172.21.15.67:0/3860623021 shutdown_connections 2024-03-24T00:39:38.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.239+0000 7f1681fb3640 1 --2- 172.21.15.67:0/3860623021 >> v2:172.21.15.72:3300/0 conn(0x7f167c0b3b60 0x7f167c161b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.239+0000 7f1681fb3640 1 --2- 172.21.15.67:0/3860623021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f167c0b41e0 0x7f167c0b3620 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.239+0000 7f1681fb3640 1 --2- 172.21.15.67:0/3860623021 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f167c15dfd0 0x7f167c0cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.239+0000 7f1681fb3640 1 -- 172.21.15.67:0/3860623021 >> 172.21.15.67:0/3860623021 conn(0x7f167c06fcf0 msgr2=0x7f167c0700d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:38.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.239+0000 7f1681fb3640 1 -- 172.21.15.67:0/3860623021 shutdown_connections 2024-03-24T00:39:38.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.239+0000 7f1681fb3640 1 -- 172.21.15.67:0/3860623021 wait complete. 2024-03-24T00:39:38.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.240+0000 7f1681fb3640 1 Processor -- start 2024-03-24T00:39:38.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.241+0000 7f1681fb3640 1 -- start start 2024-03-24T00:39:38.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.241+0000 7f1681fb3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f167c0b3b60 0x7f167c1494b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:38.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.241+0000 7f1681fb3640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f167c0b41e0 0x7f167c158cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:38.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.241+0000 7f1681fb3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f167c15dfd0 0x7f167c159230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:38.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.241+0000 7f1681fb3640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f167c0cfec0 con 0x7f167c0b3b60 2024-03-24T00:39:38.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.241+0000 7f1681fb3640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f167c0cedb0 con 0x7f167c15dfd0 2024-03-24T00:39:38.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.241+0000 7f1681fb3640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f167c15e3b0 con 0x7f167c0b41e0 2024-03-24T00:39:38.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.241+0000 7f167b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f167c0b3b60 0x7f167c1494b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:38.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.241+0000 7f167b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f167c0b3b60 0x7f167c1494b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50342/0 (socket says 172.21.15.67:50342) 2024-03-24T00:39:38.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.242+0000 7f167b7fe640 1 -- 172.21.15.67:0/1216726388 learned_addr learned my addr 172.21.15.67:0/1216726388 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:38.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.242+0000 7f167bfff640 1 --2- 172.21.15.67:0/1216726388 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f167c15dfd0 0x7f167c159230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:38.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.242+0000 7f167affd640 1 --2- 172.21.15.67:0/1216726388 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f167c0b41e0 0x7f167c158cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:38.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.242+0000 7f167b7fe640 1 -- 172.21.15.67:0/1216726388 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f167c0b41e0 msgr2=0x7f167c158cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:38.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.242+0000 7f167b7fe640 1 --2- 172.21.15.67:0/1216726388 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f167c0b41e0 0x7f167c158cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.242+0000 7f167b7fe640 1 -- 172.21.15.67:0/1216726388 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f167c15dfd0 msgr2=0x7f167c159230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:38.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.242+0000 7f167b7fe640 1 --2- 172.21.15.67:0/1216726388 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f167c15dfd0 0x7f167c159230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.242+0000 7f167b7fe640 1 -- 172.21.15.67:0/1216726388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f167c15bdc0 con 0x7f167c0b3b60 2024-03-24T00:39:38.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.242+0000 7f167b7fe640 1 --2- 172.21.15.67:0/1216726388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f167c0b3b60 0x7f167c1494b0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f167000e990 tx=0x7f167000ed90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:38.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.242+0000 7f1663fff640 1 -- 172.21.15.67:0/1216726388 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1670017070 con 0x7f167c0b3b60 2024-03-24T00:39:38.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.242+0000 7f1663fff640 1 -- 172.21.15.67:0/1216726388 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f167000c940 con 0x7f167c0b3b60 2024-03-24T00:39:38.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.242+0000 7f1681fb3640 1 -- 172.21.15.67:0/1216726388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f167c0c9df0 con 0x7f167c0b3b60 2024-03-24T00:39:38.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.242+0000 7f1663fff640 1 -- 172.21.15.67:0/1216726388 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f167000cd10 con 0x7f167c0b3b60 2024-03-24T00:39:38.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.242+0000 7f1681fb3640 1 -- 172.21.15.67:0/1216726388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f167c148fb0 con 0x7f167c0b3b60 2024-03-24T00:39:38.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.243+0000 7f1681fb3640 1 -- 172.21.15.67:0/1216726388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f167c04f400 con 0x7f167c0b3b60 2024-03-24T00:39:38.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.244+0000 7f1663fff640 1 -- 172.21.15.67:0/1216726388 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f167001a070 con 0x7f167c0b3b60 2024-03-24T00:39:38.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.244+0000 7f1663fff640 1 --2- 172.21.15.67:0/1216726388 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f166806bc00 0x7f166806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:38.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.244+0000 7f1663fff640 1 -- 172.21.15.67:0/1216726388 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f167008ae90 con 0x7f167c0b3b60 2024-03-24T00:39:38.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.244+0000 7f167affd640 1 --2- 172.21.15.67:0/1216726388 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f166806bc00 0x7f166806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:38.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.245+0000 7f167affd640 1 --2- 172.21.15.67:0/1216726388 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f166806bc00 0x7f166806e0c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f166c036000 tx=0x7f166c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:38.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.249+0000 7f1663fff640 1 -- 172.21.15.67:0/1216726388 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1670058fc0 con 0x7f167c0b3b60 2024-03-24T00:39:38.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.428+0000 7f1681fb3640 1 -- 172.21.15.67:0/1216726388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f167c0b7b20 con 0x7f167c0b3b60 2024-03-24T00:39:38.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.429+0000 7f1663fff640 1 -- 172.21.15.67:0/1216726388 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f167005d060 con 0x7f167c0b3b60 2024-03-24T00:39:38.430 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-24T00:39:38.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.431+0000 7f1681fb3640 1 -- 172.21.15.67:0/1216726388 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f166806bc00 msgr2=0x7f166806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:38.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.431+0000 7f1681fb3640 1 --2- 172.21.15.67:0/1216726388 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f166806bc00 0x7f166806e0c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f166c036000 tx=0x7f166c008030 comp rx=0 tx=0).stop 2024-03-24T00:39:38.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.431+0000 7f1681fb3640 1 -- 172.21.15.67:0/1216726388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f167c0b3b60 msgr2=0x7f167c1494b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:38.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.431+0000 7f1681fb3640 1 --2- 172.21.15.67:0/1216726388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f167c0b3b60 0x7f167c1494b0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f167000e990 tx=0x7f167000ed90 comp rx=0 tx=0).stop 2024-03-24T00:39:38.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.431+0000 7f1681fb3640 1 -- 172.21.15.67:0/1216726388 shutdown_connections 2024-03-24T00:39:38.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.431+0000 7f1681fb3640 1 --2- 172.21.15.67:0/1216726388 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f166806bc00 0x7f166806e0c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.431+0000 7f1681fb3640 1 --2- 172.21.15.67:0/1216726388 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f167c15dfd0 0x7f167c159230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.431+0000 7f1681fb3640 1 --2- 172.21.15.67:0/1216726388 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f167c0b41e0 0x7f167c158cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.431+0000 7f1681fb3640 1 --2- 172.21.15.67:0/1216726388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f167c0b3b60 0x7f167c1494b0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.431+0000 7f1681fb3640 1 -- 172.21.15.67:0/1216726388 >> 172.21.15.67:0/1216726388 conn(0x7f167c06fcf0 msgr2=0x7f167c160750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:38.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.431+0000 7f1681fb3640 1 -- 172.21.15.67:0/1216726388 shutdown_connections 2024-03-24T00:39:38.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.431+0000 7f1681fb3640 1 -- 172.21.15.67:0/1216726388 wait complete. 2024-03-24T00:39:38.442 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:39:38.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.546+0000 7f323dd13640 1 Processor -- start 2024-03-24T00:39:38.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.546+0000 7f323dd13640 1 -- start start 2024-03-24T00:39:38.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.547+0000 7f323dd13640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f323815e130 0x7f32380ce610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:38.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.547+0000 7f323dd13640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f32380d0640 0x7f32380d0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:38.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.547+0000 7f323dd13640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32380cf570 con 0x7f32380b57a0 2024-03-24T00:39:38.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.547+0000 7f323dd13640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32380ceb50 con 0x7f323815e130 2024-03-24T00:39:38.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.547+0000 7f323dd13640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32380d0fe0 con 0x7f32380d0640 2024-03-24T00:39:38.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.547+0000 7f32377fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f323815e130 0x7f32380ce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:38.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.547+0000 7f3237fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f32380b57a0 0x7f32380b5bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41496/0 (socket says 172.21.15.67:41496) 2024-03-24T00:39:38.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.547+0000 7f3237fff640 1 -- 172.21.15.67:0/1273781212 learned_addr learned my addr 172.21.15.67:0/1273781212 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:38.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.547+0000 7f3236ffd640 1 --2- 172.21.15.67:0/1273781212 >> v2:172.21.15.72:3300/0 conn(0x7f32380d0640 0x7f32380d0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:38.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.547+0000 7f32367fc640 1 -- 172.21.15.67:0/1273781212 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1135492910 0 0) 0x7f32380cf570 con 0x7f32380b57a0 2024-03-24T00:39:38.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.547+0000 7f32367fc640 1 -- 172.21.15.67:0/1273781212 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3220003520 con 0x7f32380b57a0 2024-03-24T00:39:38.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.547+0000 7f32377fe640 1 -- 172.21.15.67:0/1273781212 >> v2:172.21.15.72:3300/0 conn(0x7f32380d0640 msgr2=0x7f32380d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:38.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.547+0000 7f32377fe640 1 --2- 172.21.15.67:0/1273781212 >> v2:172.21.15.72:3300/0 conn(0x7f32380d0640 0x7f32380d0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.547+0000 7f32377fe640 1 -- 172.21.15.67:0/1273781212 >> v1:172.21.15.67:6789/0 conn(0x7f32380b57a0 legacy=0x7f32380b5bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:38.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.548+0000 7f32377fe640 1 -- 172.21.15.67:0/1273781212 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32380cecf0 con 0x7f323815e130 2024-03-24T00:39:38.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.548+0000 7f3236ffd640 1 --2- 172.21.15.67:0/1273781212 >> v2:172.21.15.72:3300/0 conn(0x7f32380d0640 0x7f32380d0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:38.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.548+0000 7f32377fe640 1 --2- 172.21.15.67:0/1273781212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f323815e130 0x7f32380ce610 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f3238161bd0 tx=0x7f322802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c6a9b6a645a23bd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:38.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.548+0000 7f32367fc640 1 -- 172.21.15.67:0/1273781212 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3228035020 con 0x7f323815e130 2024-03-24T00:39:38.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.548+0000 7f32367fc640 1 -- 172.21.15.67:0/1273781212 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3228031830 con 0x7f323815e130 2024-03-24T00:39:38.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.548+0000 7f32367fc640 1 -- 172.21.15.67:0/1273781212 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3228031c00 con 0x7f323815e130 2024-03-24T00:39:38.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.549+0000 7f323dd13640 1 -- 172.21.15.67:0/1273781212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f323815e130 msgr2=0x7f32380ce610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:38.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.549+0000 7f323dd13640 1 --2- 172.21.15.67:0/1273781212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f323815e130 0x7f32380ce610 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f3238161bd0 tx=0x7f322802b8d0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.549+0000 7f323dd13640 1 -- 172.21.15.67:0/1273781212 shutdown_connections 2024-03-24T00:39:38.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.549+0000 7f323dd13640 1 --2- 172.21.15.67:0/1273781212 >> v2:172.21.15.72:3300/0 conn(0x7f32380d0640 0x7f32380d0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.549+0000 7f323dd13640 1 --2- 172.21.15.67:0/1273781212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f323815e130 0x7f32380ce610 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.549+0000 7f323dd13640 1 -- 172.21.15.67:0/1273781212 >> 172.21.15.67:0/1273781212 conn(0x7f323806c8a0 msgr2=0x7f323806ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:38.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.549+0000 7f323dd13640 1 -- 172.21.15.67:0/1273781212 shutdown_connections 2024-03-24T00:39:38.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.549+0000 7f323dd13640 1 -- 172.21.15.67:0/1273781212 wait complete. 2024-03-24T00:39:38.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.550+0000 7f323dd13640 1 Processor -- start 2024-03-24T00:39:38.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.550+0000 7f323dd13640 1 -- start start 2024-03-24T00:39:38.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.551+0000 7f323dd13640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32380b57a0 0x7f323815a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:38.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.551+0000 7f323dd13640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f32380d0640 0x7f3238153240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:38.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.551+0000 7f323dd13640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f323815e130 0x7f32380c7850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:38.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.551+0000 7f323dd13640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32380cf290 con 0x7f323815e130 2024-03-24T00:39:38.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.551+0000 7f323dd13640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32380cefb0 con 0x7f32380b57a0 2024-03-24T00:39:38.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.551+0000 7f323dd13640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f323815e510 con 0x7f32380d0640 2024-03-24T00:39:38.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.551+0000 7f3237fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f323815e130 0x7f32380c7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:38.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.551+0000 7f3237fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f323815e130 0x7f32380c7850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50358/0 (socket says 172.21.15.67:50358) 2024-03-24T00:39:38.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.551+0000 7f3237fff640 1 -- 172.21.15.67:0/3265818860 learned_addr learned my addr 172.21.15.67:0/3265818860 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:38.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.551+0000 7f3236ffd640 1 --2- 172.21.15.67:0/3265818860 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f32380d0640 0x7f3238153240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:38.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.551+0000 7f32377fe640 1 --2- 172.21.15.67:0/3265818860 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32380b57a0 0x7f323815a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:38.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.551+0000 7f3237fff640 1 -- 172.21.15.67:0/3265818860 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f32380d0640 msgr2=0x7f3238153240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:38.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.551+0000 7f3237fff640 1 --2- 172.21.15.67:0/3265818860 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f32380d0640 0x7f3238153240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.551+0000 7f3237fff640 1 -- 172.21.15.67:0/3265818860 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32380b57a0 msgr2=0x7f323815a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:38.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.551+0000 7f3237fff640 1 --2- 172.21.15.67:0/3265818860 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32380b57a0 0x7f323815a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.551+0000 7f3237fff640 1 -- 172.21.15.67:0/3265818860 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3238160af0 con 0x7f323815e130 2024-03-24T00:39:38.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.552+0000 7f3237fff640 1 --2- 172.21.15.67:0/3265818860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f323815e130 0x7f32380c7850 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f323000cb30 tx=0x7f323000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:38.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.552+0000 7f321ffff640 1 -- 172.21.15.67:0/3265818860 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3230017070 con 0x7f323815e130 2024-03-24T00:39:38.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.552+0000 7f321ffff640 1 -- 172.21.15.67:0/3265818860 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f323000aca0 con 0x7f323815e130 2024-03-24T00:39:38.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.552+0000 7f323dd13640 1 -- 172.21.15.67:0/3265818860 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32380c8660 con 0x7f323815e130 2024-03-24T00:39:38.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.552+0000 7f321ffff640 1 -- 172.21.15.67:0/3265818860 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3230012600 con 0x7f323815e130 2024-03-24T00:39:38.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.552+0000 7f323dd13640 1 -- 172.21.15.67:0/3265818860 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3238160810 con 0x7f323815e130 2024-03-24T00:39:38.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.553+0000 7f323dd13640 1 -- 172.21.15.67:0/3265818860 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f323804f400 con 0x7f323815e130 2024-03-24T00:39:38.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.553+0000 7f321ffff640 1 -- 172.21.15.67:0/3265818860 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f323001a070 con 0x7f323815e130 2024-03-24T00:39:38.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.554+0000 7f321ffff640 1 --2- 172.21.15.67:0/3265818860 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f322406bbd0 0x7f322406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:38.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.554+0000 7f321ffff640 1 -- 172.21.15.67:0/3265818860 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f323008aa70 con 0x7f323815e130 2024-03-24T00:39:38.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.554+0000 7f32377fe640 1 --2- 172.21.15.67:0/3265818860 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f322406bbd0 0x7f322406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:38.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.554+0000 7f32377fe640 1 --2- 172.21.15.67:0/3265818860 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f322406bbd0 0x7f322406e090 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f3228036000 tx=0x7f322803c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:38.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.558+0000 7f321ffff640 1 -- 172.21.15.67:0/3265818860 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3230058ba0 con 0x7f323815e130 2024-03-24T00:39:38.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.739+0000 7f323dd13640 1 -- 172.21.15.67:0/3265818860 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f32380d19f0 con 0x7f323815e130 2024-03-24T00:39:38.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.739+0000 7f321ffff640 1 -- 172.21.15.67:0/3265818860 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f323005cc40 con 0x7f323815e130 2024-03-24T00:39:38.740 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-03-24T00:39:38.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.741+0000 7f323dd13640 1 -- 172.21.15.67:0/3265818860 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f322406bbd0 msgr2=0x7f322406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:38.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.741+0000 7f323dd13640 1 --2- 172.21.15.67:0/3265818860 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f322406bbd0 0x7f322406e090 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f3228036000 tx=0x7f322803c040 comp rx=0 tx=0).stop 2024-03-24T00:39:38.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.741+0000 7f323dd13640 1 -- 172.21.15.67:0/3265818860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f323815e130 msgr2=0x7f32380c7850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:38.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.741+0000 7f323dd13640 1 --2- 172.21.15.67:0/3265818860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f323815e130 0x7f32380c7850 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f323000cb30 tx=0x7f323000a060 comp rx=0 tx=0).stop 2024-03-24T00:39:38.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.741+0000 7f323dd13640 1 -- 172.21.15.67:0/3265818860 shutdown_connections 2024-03-24T00:39:38.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.741+0000 7f323dd13640 1 --2- 172.21.15.67:0/3265818860 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f322406bbd0 0x7f322406e090 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.741+0000 7f323dd13640 1 --2- 172.21.15.67:0/3265818860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f323815e130 0x7f32380c7850 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.741+0000 7f323dd13640 1 --2- 172.21.15.67:0/3265818860 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f32380d0640 0x7f3238153240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.741+0000 7f323dd13640 1 --2- 172.21.15.67:0/3265818860 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32380b57a0 0x7f323815a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.741+0000 7f323dd13640 1 -- 172.21.15.67:0/3265818860 >> 172.21.15.67:0/3265818860 conn(0x7f323806c8a0 msgr2=0x7f32380ce0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:38.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.741+0000 7f323dd13640 1 -- 172.21.15.67:0/3265818860 shutdown_connections 2024-03-24T00:39:38.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.741+0000 7f323dd13640 1 -- 172.21.15.67:0/3265818860 wait complete. 2024-03-24T00:39:38.752 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T00:39:38.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.856+0000 7f7b0f0d1640 1 Processor -- start 2024-03-24T00:39:38.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.856+0000 7f7b0f0d1640 1 -- start start 2024-03-24T00:39:38.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.856+0000 7f7b0f0d1640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7b0815bc30 0x7f7b080b1a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:38.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.856+0000 7f7b0f0d1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7b0815c010 0x7f7b080b1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:38.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.856+0000 7f7b0f0d1640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b080b3ae0 con 0x7f7b080b2d90 2024-03-24T00:39:38.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.856+0000 7f7b0f0d1640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b080cca60 con 0x7f7b0815bc30 2024-03-24T00:39:38.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.856+0000 7f7b0f0d1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b080b3dc0 con 0x7f7b0815c010 2024-03-24T00:39:38.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.857+0000 7f7b0ce6d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7b0815bc30 0x7f7b080b1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:38.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.857+0000 7f7b0d66e640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7b080b2d90 0x7f7b080b3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41512/0 (socket says 172.21.15.67:41512) 2024-03-24T00:39:38.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.857+0000 7f7b0ce6d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7b0815bc30 0x7f7b080b1a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50366/0 (socket says 172.21.15.67:50366) 2024-03-24T00:39:38.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.857+0000 7f7b0d66e640 1 -- 172.21.15.67:0/3710120621 learned_addr learned my addr 172.21.15.67:0/3710120621 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:38.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.857+0000 7f7b07fff640 1 --2- 172.21.15.67:0/3710120621 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7b0815c010 0x7f7b080b1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:38.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.857+0000 7f7b07fff640 1 -- 172.21.15.67:0/3710120621 >> v2:172.21.15.67:3300/0 conn(0x7f7b0815bc30 msgr2=0x7f7b080b1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:38.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.857+0000 7f7b07fff640 1 --2- 172.21.15.67:0/3710120621 >> v2:172.21.15.67:3300/0 conn(0x7f7b0815bc30 0x7f7b080b1a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.857+0000 7f7b07fff640 1 -- 172.21.15.67:0/3710120621 >> v1:172.21.15.67:6789/0 conn(0x7f7b080b2d90 legacy=0x7f7b080b3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:38.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.857+0000 7f7b07fff640 1 -- 172.21.15.67:0/3710120621 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b080b40a0 con 0x7f7b0815c010 2024-03-24T00:39:38.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.857+0000 7f7b077fe640 1 -- 172.21.15.67:0/3710120621 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 933746901 0 0) 0x7f7b080b3ae0 con 0x7f7b080b2d90 2024-03-24T00:39:38.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.857+0000 7f7b0ce6d640 1 --2- 172.21.15.67:0/3710120621 >> v2:172.21.15.67:3300/0 conn(0x7f7b0815bc30 0x7f7b080b1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:38.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.858+0000 7f7b07fff640 1 --2- 172.21.15.67:0/3710120621 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7b0815c010 0x7f7b080b1f40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7af80097a0 tx=0x7f7af802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f646cb49ea817025 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:38.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.858+0000 7f7b077fe640 1 -- 172.21.15.67:0/3710120621 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af8035020 con 0x7f7b0815c010 2024-03-24T00:39:38.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.858+0000 7f7b077fe640 1 -- 172.21.15.67:0/3710120621 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7af8031810 con 0x7f7b0815c010 2024-03-24T00:39:38.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.858+0000 7f7b077fe640 1 -- 172.21.15.67:0/3710120621 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af8031be0 con 0x7f7b0815c010 2024-03-24T00:39:38.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.858+0000 7f7b0f0d1640 1 -- 172.21.15.67:0/3710120621 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7b0815c010 msgr2=0x7f7b080b1f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:38.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.858+0000 7f7b0f0d1640 1 --2- 172.21.15.67:0/3710120621 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7b0815c010 0x7f7b080b1f40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7af80097a0 tx=0x7f7af802b8b0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.858+0000 7f7b0f0d1640 1 -- 172.21.15.67:0/3710120621 shutdown_connections 2024-03-24T00:39:38.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.858+0000 7f7b0f0d1640 1 --2- 172.21.15.67:0/3710120621 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7b0815c010 0x7f7b080b1f40 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.858+0000 7f7b0f0d1640 1 --2- 172.21.15.67:0/3710120621 >> v2:172.21.15.67:3300/0 conn(0x7f7b0815bc30 0x7f7b080b1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.858+0000 7f7b0f0d1640 1 -- 172.21.15.67:0/3710120621 >> 172.21.15.67:0/3710120621 conn(0x7f7b0808b180 msgr2=0x7f7b0808b590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:38.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.858+0000 7f7b0f0d1640 1 -- 172.21.15.67:0/3710120621 shutdown_connections 2024-03-24T00:39:38.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.858+0000 7f7b0f0d1640 1 -- 172.21.15.67:0/3710120621 wait complete. 2024-03-24T00:39:38.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.860+0000 7f7b0f0d1640 1 Processor -- start 2024-03-24T00:39:38.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.860+0000 7f7b0f0d1640 1 -- start start 2024-03-24T00:39:38.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.860+0000 7f7b0f0d1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b080b2d90 0x7f7b0814ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:38.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.860+0000 7f7b0f0d1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7b0815bc30 0x7f7b080ae690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:38.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.860+0000 7f7b0f0d1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b0815c010 0x7f7b0815db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:38.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.860+0000 7f7b0f0d1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b080cd070 con 0x7f7b0815c010 2024-03-24T00:39:38.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.860+0000 7f7b0f0d1640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b080ccef0 con 0x7f7b080b2d90 2024-03-24T00:39:38.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.860+0000 7f7b0f0d1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b080cd1f0 con 0x7f7b0815bc30 2024-03-24T00:39:38.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.861+0000 7f7b0d66e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b0815c010 0x7f7b0815db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:38.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.861+0000 7f7b0d66e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b0815c010 0x7f7b0815db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50376/0 (socket says 172.21.15.67:50376) 2024-03-24T00:39:38.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.861+0000 7f7b0d66e640 1 -- 172.21.15.67:0/313270404 learned_addr learned my addr 172.21.15.67:0/313270404 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:38.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.861+0000 7f7b07fff640 1 --2- 172.21.15.67:0/313270404 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7b0815bc30 0x7f7b080ae690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:38.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.861+0000 7f7b0ce6d640 1 --2- 172.21.15.67:0/313270404 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b080b2d90 0x7f7b0814ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:38.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.861+0000 7f7b0d66e640 1 -- 172.21.15.67:0/313270404 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7b0815bc30 msgr2=0x7f7b080ae690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:38.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.861+0000 7f7b0d66e640 1 --2- 172.21.15.67:0/313270404 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7b0815bc30 0x7f7b080ae690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.861+0000 7f7b0d66e640 1 -- 172.21.15.67:0/313270404 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b080b2d90 msgr2=0x7f7b0814ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:38.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.861+0000 7f7b0d66e640 1 --2- 172.21.15.67:0/313270404 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b080b2d90 0x7f7b0814ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:38.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.861+0000 7f7b0d66e640 1 -- 172.21.15.67:0/313270404 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b080aeee0 con 0x7f7b0815c010 2024-03-24T00:39:38.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.861+0000 7f7b07fff640 1 --2- 172.21.15.67:0/313270404 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7b0815bc30 0x7f7b080ae690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:38.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.861+0000 7f7b0d66e640 1 --2- 172.21.15.67:0/313270404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b0815c010 0x7f7b0815db40 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f7af000c480 tx=0x7f7af000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:38.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.861+0000 7f7b057fa640 1 -- 172.21.15.67:0/313270404 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af0018070 con 0x7f7b0815c010 2024-03-24T00:39:38.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.861+0000 7f7b057fa640 1 -- 172.21.15.67:0/313270404 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7af0007500 con 0x7f7b0815c010 2024-03-24T00:39:38.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.861+0000 7f7b057fa640 1 -- 172.21.15.67:0/313270404 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af0004a20 con 0x7f7b0815c010 2024-03-24T00:39:38.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.861+0000 7f7b0f0d1640 1 -- 172.21.15.67:0/313270404 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b080af1c0 con 0x7f7b0815c010 2024-03-24T00:39:38.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.861+0000 7f7b0f0d1640 1 -- 172.21.15.67:0/313270404 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b080af780 con 0x7f7b0815c010 2024-03-24T00:39:38.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.862+0000 7f7b0f0d1640 1 -- 172.21.15.67:0/313270404 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b0804f400 con 0x7f7b0815c010 2024-03-24T00:39:38.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.863+0000 7f7b057fa640 1 -- 172.21.15.67:0/313270404 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7af0004bc0 con 0x7f7b0815c010 2024-03-24T00:39:38.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.863+0000 7f7b057fa640 1 --2- 172.21.15.67:0/313270404 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7ae806bbd0 0x7f7ae806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:38.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.863+0000 7f7b057fa640 1 -- 172.21.15.67:0/313270404 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f7af0089ff0 con 0x7f7b0815c010 2024-03-24T00:39:38.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.863+0000 7f7b0ce6d640 1 --2- 172.21.15.67:0/313270404 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7ae806bbd0 0x7f7ae806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:38.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.864+0000 7f7b0ce6d640 1 --2- 172.21.15.67:0/313270404 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7ae806bbd0 0x7f7ae806e090 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7b080b0240 tx=0x7f7b00012040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:38.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:38.867+0000 7f7b057fa640 1 -- 172.21.15.67:0/313270404 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7af00580a0 con 0x7f7b0815c010 2024-03-24T00:39:39.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.049+0000 7f7b0f0d1640 1 -- 172.21.15.67:0/313270404 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f7b08068a20 con 0x7f7b0815c010 2024-03-24T00:39:39.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.049+0000 7f7b057fa640 1 -- 172.21.15.67:0/313270404 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f7af005c140 con 0x7f7b0815c010 2024-03-24T00:39:39.050 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-24T00:39:39.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.051+0000 7f7b0f0d1640 1 -- 172.21.15.67:0/313270404 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7ae806bbd0 msgr2=0x7f7ae806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:39.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.051+0000 7f7b0f0d1640 1 --2- 172.21.15.67:0/313270404 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7ae806bbd0 0x7f7ae806e090 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7b080b0240 tx=0x7f7b00012040 comp rx=0 tx=0).stop 2024-03-24T00:39:39.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.051+0000 7f7b0f0d1640 1 -- 172.21.15.67:0/313270404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b0815c010 msgr2=0x7f7b0815db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:39.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.051+0000 7f7b0f0d1640 1 --2- 172.21.15.67:0/313270404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b0815c010 0x7f7b0815db40 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f7af000c480 tx=0x7f7af000c940 comp rx=0 tx=0).stop 2024-03-24T00:39:39.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.051+0000 7f7b0f0d1640 1 -- 172.21.15.67:0/313270404 shutdown_connections 2024-03-24T00:39:39.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.051+0000 7f7b0f0d1640 1 --2- 172.21.15.67:0/313270404 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7ae806bbd0 0x7f7ae806e090 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:39.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.051+0000 7f7b0f0d1640 1 --2- 172.21.15.67:0/313270404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b0815c010 0x7f7b0815db40 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:39.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.051+0000 7f7b0f0d1640 1 --2- 172.21.15.67:0/313270404 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7b0815bc30 0x7f7b080ae690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:39.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.051+0000 7f7b0f0d1640 1 --2- 172.21.15.67:0/313270404 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b080b2d90 0x7f7b0814ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:39.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.051+0000 7f7b0f0d1640 1 -- 172.21.15.67:0/313270404 >> 172.21.15.67:0/313270404 conn(0x7f7b0808b180 msgr2=0x7f7b08156590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:39.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.051+0000 7f7b0f0d1640 1 -- 172.21.15.67:0/313270404 shutdown_connections 2024-03-24T00:39:39.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.052+0000 7f7b0f0d1640 1 -- 172.21.15.67:0/313270404 wait complete. 2024-03-24T00:39:39.063 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T00:39:39.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.164+0000 7f52cdaa1640 1 Processor -- start 2024-03-24T00:39:39.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.164+0000 7f52cdaa1640 1 -- start start 2024-03-24T00:39:39.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.165+0000 7f52cdaa1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f52c8144430 0x7f52c80cef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:39.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.165+0000 7f52cdaa1640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f52c8072210 0x7f52c80b3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:39.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.165+0000 7f52cdaa1640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52c80b2ad0 con 0x7f52c80cd830 2024-03-24T00:39:39.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.165+0000 7f52cdaa1640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52c80b36e0 con 0x7f52c8072210 2024-03-24T00:39:39.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.165+0000 7f52cdaa1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52c80b2db0 con 0x7f52c8144430 2024-03-24T00:39:39.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.165+0000 7f52c77fe640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f52c8144430 0x7f52c80cef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:39.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.165+0000 7f52c7fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f52c8072210 0x7f52c80b3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:39.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.165+0000 7f52c77fe640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f52c8144430 0x7f52c80cef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.67:33858/0 (socket says 172.21.15.67:33858) 2024-03-24T00:39:39.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.165+0000 7f52c77fe640 1 -- 172.21.15.67:0/2504003997 learned_addr learned my addr 172.21.15.67:0/2504003997 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:39.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.165+0000 7f52c6ffd640 1 --1- 172.21.15.67:0/2504003997 >> v1:172.21.15.72:6789/0 conn(0x7f52c80cd830 0x7f52c8071b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:39554/0 (socket says 172.21.15.67:39554) 2024-03-24T00:39:39.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.166+0000 7f52c77fe640 1 -- 172.21.15.67:0/2504003997 >> v2:172.21.15.72:3300/0 conn(0x7f52c8072210 msgr2=0x7f52c80b3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:39.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.166+0000 7f52c77fe640 1 --2- 172.21.15.67:0/2504003997 >> v2:172.21.15.72:3300/0 conn(0x7f52c8072210 0x7f52c80b3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:39.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.166+0000 7f52c77fe640 1 -- 172.21.15.67:0/2504003997 >> v1:172.21.15.72:6789/0 conn(0x7f52c80cd830 legacy=0x7f52c8071b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:39.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.166+0000 7f52c77fe640 1 -- 172.21.15.67:0/2504003997 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c80b3090 con 0x7f52c8144430 2024-03-24T00:39:39.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.166+0000 7f52c77fe640 1 --2- 172.21.15.67:0/2504003997 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f52c8144430 0x7f52c80cef40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f52b8009870 tx=0x7f52b802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89cc0997bde5a52e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:39.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.166+0000 7f52c67fc640 1 -- 172.21.15.67:0/2504003997 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b8035020 con 0x7f52c8144430 2024-03-24T00:39:39.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.166+0000 7f52c67fc640 1 -- 172.21.15.67:0/2504003997 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52b8031640 con 0x7f52c8144430 2024-03-24T00:39:39.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.166+0000 7f52c67fc640 1 -- 172.21.15.67:0/2504003997 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b802d040 con 0x7f52c8144430 2024-03-24T00:39:39.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.167+0000 7f52cdaa1640 1 -- 172.21.15.67:0/2504003997 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f52c8144430 msgr2=0x7f52c80cef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:39.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.167+0000 7f52cdaa1640 1 --2- 172.21.15.67:0/2504003997 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f52c8144430 0x7f52c80cef40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f52b8009870 tx=0x7f52b802b8f0 comp rx=0 tx=0).stop 2024-03-24T00:39:39.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.167+0000 7f52cdaa1640 1 -- 172.21.15.67:0/2504003997 shutdown_connections 2024-03-24T00:39:39.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.167+0000 7f52cdaa1640 1 --2- 172.21.15.67:0/2504003997 >> v2:172.21.15.72:3300/0 conn(0x7f52c8072210 0x7f52c80b3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:39.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.167+0000 7f52cdaa1640 1 --2- 172.21.15.67:0/2504003997 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f52c8144430 0x7f52c80cef40 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:39.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.167+0000 7f52cdaa1640 1 -- 172.21.15.67:0/2504003997 >> 172.21.15.67:0/2504003997 conn(0x7f52c8067470 msgr2=0x7f52c806b040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:39.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.167+0000 7f52cdaa1640 1 -- 172.21.15.67:0/2504003997 shutdown_connections 2024-03-24T00:39:39.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.167+0000 7f52cdaa1640 1 -- 172.21.15.67:0/2504003997 wait complete. 2024-03-24T00:39:39.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.168+0000 7f52cdaa1640 1 Processor -- start 2024-03-24T00:39:39.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.168+0000 7f52cdaa1640 1 -- start start 2024-03-24T00:39:39.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.169+0000 7f52cdaa1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52c8072210 0x7f52c81ef520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:39.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.169+0000 7f52cdaa1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f52c80cd830 0x7f52c813f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:39.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.169+0000 7f52cdaa1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f52c8144430 0x7f52c80c5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:39.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.169+0000 7f52cdaa1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c80cf480 con 0x7f52c8072210 2024-03-24T00:39:39.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.169+0000 7f52cdaa1640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c8144810 con 0x7f52c80cd830 2024-03-24T00:39:39.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.169+0000 7f52cdaa1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52c80cf600 con 0x7f52c8144430 2024-03-24T00:39:39.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.169+0000 7f52c77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52c8072210 0x7f52c81ef520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:39.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.169+0000 7f52c77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52c8072210 0x7f52c81ef520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50380/0 (socket says 172.21.15.67:50380) 2024-03-24T00:39:39.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.169+0000 7f52c77fe640 1 -- 172.21.15.67:0/3573455855 learned_addr learned my addr 172.21.15.67:0/3573455855 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:39.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.169+0000 7f52c7fff640 1 --2- 172.21.15.67:0/3573455855 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f52c8144430 0x7f52c80c5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:39.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.169+0000 7f52c6ffd640 1 --2- 172.21.15.67:0/3573455855 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f52c80cd830 0x7f52c813f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:39.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.169+0000 7f52c77fe640 1 -- 172.21.15.67:0/3573455855 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f52c8144430 msgr2=0x7f52c80c5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:39.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.169+0000 7f52c77fe640 1 --2- 172.21.15.67:0/3573455855 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f52c8144430 0x7f52c80c5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:39.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.169+0000 7f52c77fe640 1 -- 172.21.15.67:0/3573455855 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f52c80cd830 msgr2=0x7f52c813f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:39.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.169+0000 7f52c77fe640 1 --2- 172.21.15.67:0/3573455855 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f52c80cd830 0x7f52c813f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:39.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.169+0000 7f52c77fe640 1 -- 172.21.15.67:0/3573455855 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c80c4d40 con 0x7f52c8072210 2024-03-24T00:39:39.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.170+0000 7f52c77fe640 1 --2- 172.21.15.67:0/3573455855 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52c8072210 0x7f52c81ef520 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f52b8000c00 tx=0x7f52b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:39.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.170+0000 7f52affff640 1 -- 172.21.15.67:0/3573455855 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b8040070 con 0x7f52c8072210 2024-03-24T00:39:39.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.170+0000 7f52cdaa1640 1 -- 172.21.15.67:0/3573455855 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52c80c7980 con 0x7f52c8072210 2024-03-24T00:39:39.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.170+0000 7f52affff640 1 -- 172.21.15.67:0/3573455855 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52b80399f0 con 0x7f52c8072210 2024-03-24T00:39:39.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.170+0000 7f52affff640 1 -- 172.21.15.67:0/3573455855 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b8035040 con 0x7f52c8072210 2024-03-24T00:39:39.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.170+0000 7f52cdaa1640 1 -- 172.21.15.67:0/3573455855 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52c80c7f40 con 0x7f52c8072210 2024-03-24T00:39:39.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.171+0000 7f52cdaa1640 1 -- 172.21.15.67:0/3573455855 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52c804f400 con 0x7f52c8072210 2024-03-24T00:39:39.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.171+0000 7f52affff640 1 -- 172.21.15.67:0/3573455855 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f52b804b060 con 0x7f52c8072210 2024-03-24T00:39:39.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.172+0000 7f52affff640 1 --2- 172.21.15.67:0/3573455855 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f52b406bbd0 0x7f52b406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:39.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.172+0000 7f52affff640 1 -- 172.21.15.67:0/3573455855 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f52b80ac590 con 0x7f52c8072210 2024-03-24T00:39:39.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.172+0000 7f52c6ffd640 1 --2- 172.21.15.67:0/3573455855 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f52b406bbd0 0x7f52b406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:39.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.172+0000 7f52c6ffd640 1 --2- 172.21.15.67:0/3573455855 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f52b406bbd0 0x7f52b406e090 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f52bc000f60 tx=0x7f52bc0011d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:39.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.176+0000 7f52affff640 1 -- 172.21.15.67:0/3573455855 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f52b807a640 con 0x7f52c8072210 2024-03-24T00:39:39.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.370+0000 7f52cdaa1640 1 -- 172.21.15.67:0/3573455855 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f52c81efa60 con 0x7f52c8072210 2024-03-24T00:39:39.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.370+0000 7f52affff640 1 -- 172.21.15.67:0/3573455855 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7f52b807e6e0 con 0x7f52c8072210 2024-03-24T00:39:39.371 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-03-24T00:39:39.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.372+0000 7f52cdaa1640 1 -- 172.21.15.67:0/3573455855 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f52b406bbd0 msgr2=0x7f52b406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:39.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.372+0000 7f52cdaa1640 1 --2- 172.21.15.67:0/3573455855 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f52b406bbd0 0x7f52b406e090 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f52bc000f60 tx=0x7f52bc0011d0 comp rx=0 tx=0).stop 2024-03-24T00:39:39.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.372+0000 7f52cdaa1640 1 -- 172.21.15.67:0/3573455855 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52c8072210 msgr2=0x7f52c81ef520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:39.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.372+0000 7f52cdaa1640 1 --2- 172.21.15.67:0/3573455855 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52c8072210 0x7f52c81ef520 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f52b8000c00 tx=0x7f52b80046c0 comp rx=0 tx=0).stop 2024-03-24T00:39:39.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.372+0000 7f52cdaa1640 1 -- 172.21.15.67:0/3573455855 shutdown_connections 2024-03-24T00:39:39.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.372+0000 7f52cdaa1640 1 --2- 172.21.15.67:0/3573455855 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f52b406bbd0 0x7f52b406e090 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:39.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.372+0000 7f52cdaa1640 1 --2- 172.21.15.67:0/3573455855 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f52c8144430 0x7f52c80c5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:39.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.372+0000 7f52cdaa1640 1 --2- 172.21.15.67:0/3573455855 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f52c80cd830 0x7f52c813f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:39.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.372+0000 7f52cdaa1640 1 --2- 172.21.15.67:0/3573455855 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52c8072210 0x7f52c81ef520 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:39.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.372+0000 7f52cdaa1640 1 -- 172.21.15.67:0/3573455855 >> 172.21.15.67:0/3573455855 conn(0x7f52c8067470 msgr2=0x7f52c8142260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:39.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.372+0000 7f52cdaa1640 1 -- 172.21.15.67:0/3573455855 shutdown_connections 2024-03-24T00:39:39.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.372+0000 7f52cdaa1640 1 -- 172.21.15.67:0/3573455855 wait complete. 2024-03-24T00:39:39.384 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-24T00:39:39.384 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:39.384 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-03-24T00:39:39.414 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-03-24T00:39:39.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:39.414 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-24T00:39:39.472 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:39:39.473 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:39:39.489 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:39:39.489 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:39:39.544 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:39:39.544 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:39:39.599 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:39:39.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.699+0000 7fe9ecec0640 1 Processor -- start 2024-03-24T00:39:39.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.699+0000 7fe9ecec0640 1 -- start start 2024-03-24T00:39:39.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.700+0000 7fe9ecec0640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe9e815dec0 0x7fe9e80ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:39.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.700+0000 7fe9ecec0640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9e80cf640 con 0x7fe9e815e2a0 2024-03-24T00:39:39.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.700+0000 7fe9ecec0640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9e80cf080 con 0x7fe9e80d0ef0 2024-03-24T00:39:39.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.700+0000 7fe9ecec0640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e80cee30 con 0x7fe9e815dec0 2024-03-24T00:39:39.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.700+0000 7fe9dffff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe9e815e2a0 0x7fe9e80d07e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41522/0 (socket says 172.21.15.67:41522) 2024-03-24T00:39:39.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.700+0000 7fe9dffff640 1 -- 172.21.15.67:0/2181163462 learned_addr learned my addr 172.21.15.67:0/2181163462 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:39.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.700+0000 7fe9e659c640 1 --2- 172.21.15.67:0/2181163462 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe9e815dec0 0x7fe9e80ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:39.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.701+0000 7fe9e5d9b640 1 -- 172.21.15.67:0/2181163462 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2217759200 0 0) 0x7fe9d0001110 con 0x7fe9e815e2a0 2024-03-24T00:39:39.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.701+0000 7fe9e5d9b640 1 -- 172.21.15.67:0/2181163462 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9d4003520 con 0x7fe9e815e2a0 2024-03-24T00:39:39.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.701+0000 7fe9e5d9b640 1 -- 172.21.15.67:0/2181163462 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1247705486 0 0) 0x7fe9e80cf080 con 0x7fe9e80d0ef0 2024-03-24T00:39:39.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.701+0000 7fe9e5d9b640 1 -- 172.21.15.67:0/2181163462 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9d0001110 con 0x7fe9e80d0ef0 2024-03-24T00:39:39.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.701+0000 7fe9e659c640 1 -- 172.21.15.67:0/2181163462 >> v1:172.21.15.72:6789/0 conn(0x7fe9e80d0ef0 legacy=0x7fe9e80d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:39.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.701+0000 7fe9e659c640 1 -- 172.21.15.67:0/2181163462 >> v1:172.21.15.67:6789/0 conn(0x7fe9e815e2a0 legacy=0x7fe9e80d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:39.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.701+0000 7fe9e659c640 1 -- 172.21.15.67:0/2181163462 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e80cf360 con 0x7fe9e815dec0 2024-03-24T00:39:39.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.702+0000 7fe9e659c640 1 --2- 172.21.15.67:0/2181163462 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe9e815dec0 0x7fe9e80ce630 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe9d80097d0 tx=0x7fe9d802b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c86e39795748e60e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:39.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.702+0000 7fe9e5d9b640 1 -- 172.21.15.67:0/2181163462 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d8035020 con 0x7fe9e815dec0 2024-03-24T00:39:39.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.702+0000 7fe9e5d9b640 1 -- 172.21.15.67:0/2181163462 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9d8031800 con 0x7fe9e815dec0 2024-03-24T00:39:39.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.702+0000 7fe9e5d9b640 1 -- 172.21.15.67:0/2181163462 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d8031bd0 con 0x7fe9e815dec0 2024-03-24T00:39:39.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.703+0000 7fe9ecec0640 1 -- 172.21.15.67:0/2181163462 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe9e815dec0 msgr2=0x7fe9e80ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:39.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.703+0000 7fe9ecec0640 1 --2- 172.21.15.67:0/2181163462 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe9e815dec0 0x7fe9e80ce630 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe9d80097d0 tx=0x7fe9d802b8a0 comp rx=0 tx=0).stop 2024-03-24T00:39:39.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.703+0000 7fe9ecec0640 1 -- 172.21.15.67:0/2181163462 shutdown_connections 2024-03-24T00:39:39.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.703+0000 7fe9ecec0640 1 --2- 172.21.15.67:0/2181163462 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe9e815dec0 0x7fe9e80ce630 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:39.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.703+0000 7fe9ecec0640 1 -- 172.21.15.67:0/2181163462 >> 172.21.15.67:0/2181163462 conn(0x7fe9e806c810 msgr2=0x7fe9e806cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:39.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.703+0000 7fe9ecec0640 1 -- 172.21.15.67:0/2181163462 shutdown_connections 2024-03-24T00:39:39.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.703+0000 7fe9ecec0640 1 -- 172.21.15.67:0/2181163462 wait complete. 2024-03-24T00:39:39.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.704+0000 7fe9ecec0640 1 Processor -- start 2024-03-24T00:39:39.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.704+0000 7fe9ecec0640 1 -- start start 2024-03-24T00:39:39.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.705+0000 7fe9ecec0640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe9e80d0ef0 0x7fe9e80b63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:39.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.705+0000 7fe9ecec0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9e815dec0 0x7fe9e814f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:39.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.705+0000 7fe9ecec0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe9e815e2a0 0x7fe9e80cd230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:39.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.705+0000 7fe9ecec0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e8162a40 con 0x7fe9e815dec0 2024-03-24T00:39:39.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.705+0000 7fe9ecec0640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e81628c0 con 0x7fe9e815e2a0 2024-03-24T00:39:39.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.705+0000 7fe9ecec0640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e80d2120 con 0x7fe9e80d0ef0 2024-03-24T00:39:39.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.705+0000 7fe9dffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9e815dec0 0x7fe9e814f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:39.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.705+0000 7fe9dffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9e815dec0 0x7fe9e814f2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50392/0 (socket says 172.21.15.67:50392) 2024-03-24T00:39:39.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.705+0000 7fe9dffff640 1 -- 172.21.15.67:0/376468624 learned_addr learned my addr 172.21.15.67:0/376468624 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:39.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.705+0000 7fe9e6d9d640 1 --2- 172.21.15.67:0/376468624 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe9e815e2a0 0x7fe9e80cd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:39.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.705+0000 7fe9e659c640 1 --2- 172.21.15.67:0/376468624 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe9e80d0ef0 0x7fe9e80b63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:39.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.705+0000 7fe9e6d9d640 1 -- 172.21.15.67:0/376468624 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe9e80d0ef0 msgr2=0x7fe9e80b63e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:39.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.705+0000 7fe9e6d9d640 1 --2- 172.21.15.67:0/376468624 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe9e80d0ef0 0x7fe9e80b63e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:39.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.705+0000 7fe9e6d9d640 1 -- 172.21.15.67:0/376468624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9e815dec0 msgr2=0x7fe9e814f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:39.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.705+0000 7fe9e6d9d640 1 --2- 172.21.15.67:0/376468624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9e815dec0 0x7fe9e814f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:39.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.705+0000 7fe9e6d9d640 1 -- 172.21.15.67:0/376468624 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e80cbf20 con 0x7fe9e815e2a0 2024-03-24T00:39:39.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.705+0000 7fe9dffff640 1 --2- 172.21.15.67:0/376468624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9e815dec0 0x7fe9e814f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:39.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.705+0000 7fe9e6d9d640 1 --2- 172.21.15.67:0/376468624 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe9e815e2a0 0x7fe9e80cd230 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe9e80b6df0 tx=0x7fe9e000cb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:39.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.706+0000 7fe9deffd640 1 -- 172.21.15.67:0/376468624 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e0003b70 con 0x7fe9e815e2a0 2024-03-24T00:39:39.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.706+0000 7fe9deffd640 1 -- 172.21.15.67:0/376468624 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9e0003d30 con 0x7fe9e815e2a0 2024-03-24T00:39:39.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.706+0000 7fe9ecec0640 1 -- 172.21.15.67:0/376468624 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9e80cc200 con 0x7fe9e815e2a0 2024-03-24T00:39:39.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.706+0000 7fe9deffd640 1 -- 172.21.15.67:0/376468624 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e0005310 con 0x7fe9e815e2a0 2024-03-24T00:39:39.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.706+0000 7fe9ecec0640 1 -- 172.21.15.67:0/376468624 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9e81590a0 con 0x7fe9e815e2a0 2024-03-24T00:39:39.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.707+0000 7fe9ecec0640 1 -- 172.21.15.67:0/376468624 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9e804f400 con 0x7fe9e815e2a0 2024-03-24T00:39:39.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.707+0000 7fe9deffd640 1 -- 172.21.15.67:0/376468624 <== mon.1 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe9e00054b0 con 0x7fe9e815e2a0 2024-03-24T00:39:39.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.707+0000 7fe9deffd640 1 --2- 172.21.15.67:0/376468624 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe9c806bc00 0x7fe9c806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:39.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.708+0000 7fe9deffd640 1 -- 172.21.15.67:0/376468624 <== mon.1 v2:172.21.15.72:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fe9e008aa30 con 0x7fe9e815e2a0 2024-03-24T00:39:39.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.708+0000 7fe9e659c640 1 --2- 172.21.15.67:0/376468624 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe9c806bc00 0x7fe9c806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:39.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.708+0000 7fe9e659c640 1 --2- 172.21.15.67:0/376468624 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe9c806bc00 0x7fe9c806e0c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8036000 tx=0x7fe9d8008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:39.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.712+0000 7fe9deffd640 1 -- 172.21.15.67:0/376468624 <== mon.1 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe9e0058b60 con 0x7fe9e815e2a0 2024-03-24T00:39:39.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.899+0000 7fe9ecec0640 1 -- 172.21.15.67:0/376468624 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9e8159360 con 0x7fe9e815e2a0 2024-03-24T00:39:39.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.900+0000 7fe9deffd640 1 -- 172.21.15.67:0/376468624 <== mon.1 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18843 (secure 0 0 0) 0x7fe9e005cc00 con 0x7fe9e815e2a0 2024-03-24T00:39:39.901 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:39:39.901 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":27,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:39:32.904729+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:38:50.077085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:38:51.095454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T00:38:54.807117+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.14","mappings":[{"from":4,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:39:39.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.902+0000 7fe9ecec0640 1 -- 172.21.15.67:0/376468624 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe9c806bc00 msgr2=0x7fe9c806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:39.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.902+0000 7fe9ecec0640 1 --2- 172.21.15.67:0/376468624 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe9c806bc00 0x7fe9c806e0c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8036000 tx=0x7fe9d8008030 comp rx=0 tx=0).stop 2024-03-24T00:39:39.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.902+0000 7fe9ecec0640 1 -- 172.21.15.67:0/376468624 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe9e815e2a0 msgr2=0x7fe9e80cd230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:39.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.902+0000 7fe9ecec0640 1 --2- 172.21.15.67:0/376468624 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe9e815e2a0 0x7fe9e80cd230 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe9e80b6df0 tx=0x7fe9e000cb90 comp rx=0 tx=0).stop 2024-03-24T00:39:39.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.902+0000 7fe9ecec0640 1 -- 172.21.15.67:0/376468624 shutdown_connections 2024-03-24T00:39:39.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.902+0000 7fe9ecec0640 1 --2- 172.21.15.67:0/376468624 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe9c806bc00 0x7fe9c806e0c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:39.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.902+0000 7fe9ecec0640 1 --2- 172.21.15.67:0/376468624 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe9e815e2a0 0x7fe9e80cd230 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:39.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.902+0000 7fe9ecec0640 1 --2- 172.21.15.67:0/376468624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9e815dec0 0x7fe9e814f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:39.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.902+0000 7fe9ecec0640 1 --2- 172.21.15.67:0/376468624 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe9e80d0ef0 0x7fe9e80b63e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:39.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.902+0000 7fe9ecec0640 1 -- 172.21.15.67:0/376468624 >> 172.21.15.67:0/376468624 conn(0x7fe9e806c810 msgr2=0x7fe9e80b93b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:39.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.902+0000 7fe9ecec0640 1 -- 172.21.15.67:0/376468624 shutdown_connections 2024-03-24T00:39:39.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:39.902+0000 7fe9ecec0640 1 -- 172.21.15.67:0/376468624 wait complete. 2024-03-24T00:39:39.914 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:38:39.364950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:38:50.077085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:38:51.095454+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T00:38:54.807117+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T00:39:39.915 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:39:40.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.019+0000 7f5471427640 1 Processor -- start 2024-03-24T00:39:40.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.019+0000 7f5471427640 1 -- start start 2024-03-24T00:39:40.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.020+0000 7f5471427640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f546c0b9690 0x7f546c0ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:40.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.020+0000 7f5471427640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f546c0cef30 0x7f546c0d2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:40.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.020+0000 7f5471427640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f546c162290 con 0x7f546c161370 2024-03-24T00:39:40.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.020+0000 7f5471427640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f546c0b6e10 con 0x7f546c0cef30 2024-03-24T00:39:40.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.020+0000 7f5471427640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f546c0d2190 con 0x7f546c0b9690 2024-03-24T00:39:40.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.020+0000 7f546b7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f546c0cef30 0x7f546c0d2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:40.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.020+0000 7f546affd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f546c161370 0x7f546c161ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41536/0 (socket says 172.21.15.67:41536) 2024-03-24T00:39:40.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.020+0000 7f546affd640 1 -- 172.21.15.67:0/194366808 learned_addr learned my addr 172.21.15.67:0/194366808 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:40.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.020+0000 7f546a7fc640 1 --2- 172.21.15.67:0/194366808 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f546c0b9690 0x7f546c0ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:40.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.020+0000 7f5469ffb640 1 -- 172.21.15.67:0/194366808 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 736897838 0 0) 0x7f546c162290 con 0x7f546c161370 2024-03-24T00:39:40.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.020+0000 7f5469ffb640 1 -- 172.21.15.67:0/194366808 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5458003520 con 0x7f546c161370 2024-03-24T00:39:40.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.020+0000 7f546b7fe640 1 -- 172.21.15.67:0/194366808 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f546c0b9690 msgr2=0x7f546c0ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:40.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.020+0000 7f546b7fe640 1 --2- 172.21.15.67:0/194366808 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f546c0b9690 0x7f546c0ce9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:40.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.020+0000 7f546b7fe640 1 -- 172.21.15.67:0/194366808 >> v1:172.21.15.67:6789/0 conn(0x7f546c161370 legacy=0x7f546c161ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:40.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.020+0000 7f546b7fe640 1 -- 172.21.15.67:0/194366808 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f546c162570 con 0x7f546c0cef30 2024-03-24T00:39:40.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.021+0000 7f546b7fe640 1 --2- 172.21.15.67:0/194366808 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f546c0cef30 0x7f546c0d2620 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f545c009dc0 tx=0x7f545c02bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65c500c33a57b200 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:40.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.021+0000 7f5469ffb640 1 -- 172.21.15.67:0/194366808 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f545c039070 con 0x7f546c0cef30 2024-03-24T00:39:40.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.021+0000 7f5469ffb640 1 -- 172.21.15.67:0/194366808 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f545c02ccd0 con 0x7f546c0cef30 2024-03-24T00:39:40.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.021+0000 7f5469ffb640 1 -- 172.21.15.67:0/194366808 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f545c034660 con 0x7f546c0cef30 2024-03-24T00:39:40.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.022+0000 7f5471427640 1 -- 172.21.15.67:0/194366808 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f546c0cef30 msgr2=0x7f546c0d2620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:40.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.022+0000 7f5471427640 1 --2- 172.21.15.67:0/194366808 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f546c0cef30 0x7f546c0d2620 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f545c009dc0 tx=0x7f545c02bf20 comp rx=0 tx=0).stop 2024-03-24T00:39:40.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.022+0000 7f5471427640 1 -- 172.21.15.67:0/194366808 shutdown_connections 2024-03-24T00:39:40.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.022+0000 7f5471427640 1 --2- 172.21.15.67:0/194366808 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f546c0cef30 0x7f546c0d2620 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:40.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.022+0000 7f5471427640 1 --2- 172.21.15.67:0/194366808 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f546c0b9690 0x7f546c0ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:40.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.022+0000 7f5471427640 1 -- 172.21.15.67:0/194366808 >> 172.21.15.67:0/194366808 conn(0x7f546c06c880 msgr2=0x7f546c06cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:40.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.022+0000 7f5471427640 1 -- 172.21.15.67:0/194366808 shutdown_connections 2024-03-24T00:39:40.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.022+0000 7f5471427640 1 -- 172.21.15.67:0/194366808 wait complete. 2024-03-24T00:39:40.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.023+0000 7f5471427640 1 Processor -- start 2024-03-24T00:39:40.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.023+0000 7f5471427640 1 -- start start 2024-03-24T00:39:40.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.024+0000 7f5471427640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f546c0b9690 0x7f546c155e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:40.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.024+0000 7f5471427640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f546c0cef30 0x7f546c0cc250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:40.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.024+0000 7f5471427640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f546c161370 0x7f546c0cd7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:40.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.024+0000 7f5471427640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f546c0d2b60 con 0x7f546c161370 2024-03-24T00:39:40.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.024+0000 7f5471427640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f546c161750 con 0x7f546c0cef30 2024-03-24T00:39:40.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.024+0000 7f5471427640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f546c0d2ce0 con 0x7f546c0b9690 2024-03-24T00:39:40.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.024+0000 7f546b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f546c161370 0x7f546c0cd7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:40.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.024+0000 7f546b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f546c161370 0x7f546c0cd7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50398/0 (socket says 172.21.15.67:50398) 2024-03-24T00:39:40.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.024+0000 7f546b7fe640 1 -- 172.21.15.67:0/2253200470 learned_addr learned my addr 172.21.15.67:0/2253200470 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:40.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.024+0000 7f546affd640 1 --2- 172.21.15.67:0/2253200470 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f546c0b9690 0x7f546c155e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:40.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.024+0000 7f546a7fc640 1 --2- 172.21.15.67:0/2253200470 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f546c0cef30 0x7f546c0cc250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:40.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.024+0000 7f546b7fe640 1 -- 172.21.15.67:0/2253200470 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f546c0b9690 msgr2=0x7f546c155e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:40.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.024+0000 7f546b7fe640 1 --2- 172.21.15.67:0/2253200470 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f546c0b9690 0x7f546c155e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:40.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.024+0000 7f546b7fe640 1 -- 172.21.15.67:0/2253200470 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f546c0cef30 msgr2=0x7f546c0cc250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:40.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.024+0000 7f546b7fe640 1 --2- 172.21.15.67:0/2253200470 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f546c0cef30 0x7f546c0cc250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:40.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.024+0000 7f546b7fe640 1 -- 172.21.15.67:0/2253200470 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f546c0cdd10 con 0x7f546c161370 2024-03-24T00:39:40.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.024+0000 7f546b7fe640 1 --2- 172.21.15.67:0/2253200470 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f546c161370 0x7f546c0cd7d0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f545c037040 tx=0x7f545c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:40.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.024+0000 7f54637fe640 1 -- 172.21.15.67:0/2253200470 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f545c039050 con 0x7f546c161370 2024-03-24T00:39:40.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.024+0000 7f54637fe640 1 -- 172.21.15.67:0/2253200470 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f545c045070 con 0x7f546c161370 2024-03-24T00:39:40.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.025+0000 7f5471427640 1 -- 172.21.15.67:0/2253200470 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f546c0cd170 con 0x7f546c161370 2024-03-24T00:39:40.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.025+0000 7f54637fe640 1 -- 172.21.15.67:0/2253200470 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f545c009450 con 0x7f546c161370 2024-03-24T00:39:40.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.025+0000 7f5471427640 1 -- 172.21.15.67:0/2253200470 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f546c0cba50 con 0x7f546c161370 2024-03-24T00:39:40.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.026+0000 7f5471427640 1 -- 172.21.15.67:0/2253200470 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f546c06ded0 con 0x7f546c161370 2024-03-24T00:39:40.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.026+0000 7f54637fe640 1 -- 172.21.15.67:0/2253200470 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f545c03e850 con 0x7f546c161370 2024-03-24T00:39:40.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.026+0000 7f54637fe640 1 --2- 172.21.15.67:0/2253200470 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f544c06bbd0 0x7f544c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:40.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.027+0000 7f54637fe640 1 -- 172.21.15.67:0/2253200470 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f545c0ada70 con 0x7f546c161370 2024-03-24T00:39:40.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.027+0000 7f546affd640 1 --2- 172.21.15.67:0/2253200470 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f544c06bbd0 0x7f544c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:40.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.027+0000 7f546affd640 1 --2- 172.21.15.67:0/2253200470 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f544c06bbd0 0x7f544c06e090 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5464005880 tx=0x7f5464009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:40.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.031+0000 7f54637fe640 1 -- 172.21.15.67:0/2253200470 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f546c06ded0 con 0x7f546c161370 2024-03-24T00:39:40.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.212+0000 7f5471427640 1 -- 172.21.15.67:0/2253200470 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f546c0cb850 con 0x7f546c161370 2024-03-24T00:39:40.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.212+0000 7f54637fe640 1 -- 172.21.15.67:0/2253200470 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f546c0cdd10 con 0x7f546c161370 2024-03-24T00:39:40.213 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-24T00:39:40.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.214+0000 7f5471427640 1 -- 172.21.15.67:0/2253200470 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f544c06bbd0 msgr2=0x7f544c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:40.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.214+0000 7f5471427640 1 --2- 172.21.15.67:0/2253200470 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f544c06bbd0 0x7f544c06e090 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5464005880 tx=0x7f5464009040 comp rx=0 tx=0).stop 2024-03-24T00:39:40.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.214+0000 7f5471427640 1 -- 172.21.15.67:0/2253200470 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f546c161370 msgr2=0x7f546c0cd7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:40.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.214+0000 7f5471427640 1 --2- 172.21.15.67:0/2253200470 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f546c161370 0x7f546c0cd7d0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f545c037040 tx=0x7f545c004a40 comp rx=0 tx=0).stop 2024-03-24T00:39:40.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.214+0000 7f5471427640 1 -- 172.21.15.67:0/2253200470 shutdown_connections 2024-03-24T00:39:40.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.214+0000 7f5471427640 1 --2- 172.21.15.67:0/2253200470 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f544c06bbd0 0x7f544c06e090 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:40.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.214+0000 7f5471427640 1 --2- 172.21.15.67:0/2253200470 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f546c161370 0x7f546c0cd7d0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:40.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.214+0000 7f5471427640 1 --2- 172.21.15.67:0/2253200470 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f546c0cef30 0x7f546c0cc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:40.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.214+0000 7f5471427640 1 --2- 172.21.15.67:0/2253200470 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f546c0b9690 0x7f546c155e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:40.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.214+0000 7f5471427640 1 -- 172.21.15.67:0/2253200470 >> 172.21.15.67:0/2253200470 conn(0x7f546c06c880 msgr2=0x7f546c0b6fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:40.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.214+0000 7f5471427640 1 -- 172.21.15.67:0/2253200470 shutdown_connections 2024-03-24T00:39:40.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.214+0000 7f5471427640 1 -- 172.21.15.67:0/2253200470 wait complete. 2024-03-24T00:39:40.226 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:39:40.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.332+0000 7fbec5cd1640 1 Processor -- start 2024-03-24T00:39:40.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.332+0000 7fbec5cd1640 1 -- start start 2024-03-24T00:39:40.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.333+0000 7fbec5cd1640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fbec015e130 0x7fbec00ce610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:40.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.333+0000 7fbec5cd1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbec00d0640 0x7fbec00d0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:40.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.333+0000 7fbec5cd1640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbec00cf570 con 0x7fbec00b57a0 2024-03-24T00:39:40.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.333+0000 7fbec5cd1640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbec00ceb50 con 0x7fbec015e130 2024-03-24T00:39:40.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.333+0000 7fbec5cd1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbec00d0fe0 con 0x7fbec00d0640 2024-03-24T00:39:40.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.333+0000 7fbebeffd640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbec00d0640 0x7fbec00d0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:40.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.333+0000 7fbebf7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fbec015e130 0x7fbec00ce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:40.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.333+0000 7fbebf7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fbec015e130 0x7fbec00ce610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:51154/0 (socket says 172.21.15.67:51154) 2024-03-24T00:39:40.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.333+0000 7fbebeffd640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbec00d0640 0x7fbec00d0aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.67:33928/0 (socket says 172.21.15.67:33928) 2024-03-24T00:39:40.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.333+0000 7fbebf7fe640 1 -- 172.21.15.67:0/2503198635 learned_addr learned my addr 172.21.15.67:0/2503198635 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:40.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.333+0000 7fbebf7fe640 1 -- 172.21.15.67:0/2503198635 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbec00d0640 msgr2=0x7fbec00d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:40.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.333+0000 7fbebf7fe640 1 --2- 172.21.15.67:0/2503198635 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbec00d0640 0x7fbec00d0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:40.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.333+0000 7fbebf7fe640 1 -- 172.21.15.67:0/2503198635 >> v1:172.21.15.72:6789/0 conn(0x7fbec00b57a0 legacy=0x7fbec00b5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:40.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.333+0000 7fbebf7fe640 1 -- 172.21.15.67:0/2503198635 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbec00cecf0 con 0x7fbec015e130 2024-03-24T00:39:40.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.334+0000 7fbebf7fe640 1 --2- 172.21.15.67:0/2503198635 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbec015e130 0x7fbec00ce610 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fbeb0009870 tx=0x7fbeb002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=537d014219c87f50 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:40.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.334+0000 7fbebe7fc640 1 -- 172.21.15.67:0/2503198635 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb0034070 con 0x7fbec015e130 2024-03-24T00:39:40.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.334+0000 7fbebe7fc640 1 -- 172.21.15.67:0/2503198635 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeb0030840 con 0x7fbec015e130 2024-03-24T00:39:40.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.334+0000 7fbebe7fc640 1 -- 172.21.15.67:0/2503198635 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb0030c10 con 0x7fbec015e130 2024-03-24T00:39:40.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.335+0000 7fbec5cd1640 1 -- 172.21.15.67:0/2503198635 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbec015e130 msgr2=0x7fbec00ce610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:40.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.335+0000 7fbec5cd1640 1 --2- 172.21.15.67:0/2503198635 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbec015e130 0x7fbec00ce610 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fbeb0009870 tx=0x7fbeb002b980 comp rx=0 tx=0).stop 2024-03-24T00:39:40.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.335+0000 7fbec5cd1640 1 -- 172.21.15.67:0/2503198635 shutdown_connections 2024-03-24T00:39:40.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.335+0000 7fbec5cd1640 1 --2- 172.21.15.67:0/2503198635 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbec00d0640 0x7fbec00d0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:40.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.335+0000 7fbec5cd1640 1 --2- 172.21.15.67:0/2503198635 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbec015e130 0x7fbec00ce610 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:40.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.335+0000 7fbec5cd1640 1 -- 172.21.15.67:0/2503198635 >> 172.21.15.67:0/2503198635 conn(0x7fbec006c8a0 msgr2=0x7fbec006ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:40.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.335+0000 7fbec5cd1640 1 -- 172.21.15.67:0/2503198635 shutdown_connections 2024-03-24T00:39:40.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.335+0000 7fbec5cd1640 1 -- 172.21.15.67:0/2503198635 wait complete. 2024-03-24T00:39:40.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.336+0000 7fbec5cd1640 1 Processor -- start 2024-03-24T00:39:40.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.336+0000 7fbec5cd1640 1 -- start start 2024-03-24T00:39:40.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.337+0000 7fbec5cd1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbec00b57a0 0x7fbec0160150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:40.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.337+0000 7fbec5cd1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbec00d0640 0x7fbec0157ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:40.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.337+0000 7fbec5cd1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbec015e130 0x7fbec014cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:40.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.337+0000 7fbec5cd1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbec00cf290 con 0x7fbec00b57a0 2024-03-24T00:39:40.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.337+0000 7fbec5cd1640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbec00cefb0 con 0x7fbec015e130 2024-03-24T00:39:40.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.337+0000 7fbec5cd1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbec015e510 con 0x7fbec00d0640 2024-03-24T00:39:40.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.337+0000 7fbebf7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbec00b57a0 0x7fbec0160150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:40.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.337+0000 7fbebeffd640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbec00d0640 0x7fbec0157ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:40.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.337+0000 7fbebf7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbec00b57a0 0x7fbec0160150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50400/0 (socket says 172.21.15.67:50400) 2024-03-24T00:39:40.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.337+0000 7fbebf7fe640 1 -- 172.21.15.67:0/3304693587 learned_addr learned my addr 172.21.15.67:0/3304693587 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:40.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.337+0000 7fbebffff640 1 --2- 172.21.15.67:0/3304693587 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbec015e130 0x7fbec014cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:40.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.337+0000 7fbebf7fe640 1 -- 172.21.15.67:0/3304693587 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbec00d0640 msgr2=0x7fbec0157ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:40.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.337+0000 7fbebf7fe640 1 --2- 172.21.15.67:0/3304693587 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbec00d0640 0x7fbec0157ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:40.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.337+0000 7fbebf7fe640 1 -- 172.21.15.67:0/3304693587 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbec015e130 msgr2=0x7fbec014cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:40.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.337+0000 7fbebf7fe640 1 --2- 172.21.15.67:0/3304693587 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbec015e130 0x7fbec014cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:40.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.337+0000 7fbebf7fe640 1 -- 172.21.15.67:0/3304693587 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbec0160830 con 0x7fbec00b57a0 2024-03-24T00:39:40.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.337+0000 7fbebeffd640 1 --2- 172.21.15.67:0/3304693587 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbec00d0640 0x7fbec0157ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:40.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.338+0000 7fbebf7fe640 1 --2- 172.21.15.67:0/3304693587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbec00b57a0 0x7fbec0160150 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fbeb002b9c0 tx=0x7fbeb00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:40.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.338+0000 7fbea7fff640 1 -- 172.21.15.67:0/3304693587 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb0034040 con 0x7fbec00b57a0 2024-03-24T00:39:40.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.338+0000 7fbea7fff640 1 -- 172.21.15.67:0/3304693587 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeb0039d90 con 0x7fbec00b57a0 2024-03-24T00:39:40.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.338+0000 7fbec5cd1640 1 -- 172.21.15.67:0/3304693587 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbec0160b10 con 0x7fbec00b57a0 2024-03-24T00:39:40.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.338+0000 7fbea7fff640 1 -- 172.21.15.67:0/3304693587 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb0043630 con 0x7fbec00b57a0 2024-03-24T00:39:40.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.338+0000 7fbec5cd1640 1 -- 172.21.15.67:0/3304693587 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbec00c8350 con 0x7fbec00b57a0 2024-03-24T00:39:40.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.339+0000 7fbec5cd1640 1 -- 172.21.15.67:0/3304693587 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbec004f400 con 0x7fbec00b57a0 2024-03-24T00:39:40.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.339+0000 7fbea7fff640 1 -- 172.21.15.67:0/3304693587 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fbeb004a060 con 0x7fbec00b57a0 2024-03-24T00:39:40.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.340+0000 7fbea7fff640 1 --2- 172.21.15.67:0/3304693587 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbeac06bbd0 0x7fbeac06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:40.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.340+0000 7fbea7fff640 1 -- 172.21.15.67:0/3304693587 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fbeb00acbf0 con 0x7fbec00b57a0 2024-03-24T00:39:40.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.340+0000 7fbebeffd640 1 --2- 172.21.15.67:0/3304693587 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbeac06bbd0 0x7fbeac06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:40.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.340+0000 7fbebeffd640 1 --2- 172.21.15.67:0/3304693587 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbeac06bbd0 0x7fbeac06e090 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbec015b160 tx=0x7fbeb4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:40.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.344+0000 7fbea7fff640 1 -- 172.21.15.67:0/3304693587 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fbeb003e070 con 0x7fbec00b57a0 2024-03-24T00:39:40.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.525+0000 7fbec5cd1640 1 -- 172.21.15.67:0/3304693587 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbec00d19f0 con 0x7fbec00b57a0 2024-03-24T00:39:40.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.526+0000 7fbea7fff640 1 -- 172.21.15.67:0/3304693587 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbeb007ad20 con 0x7fbec00b57a0 2024-03-24T00:39:40.526 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-24T00:39:40.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.528+0000 7fbec5cd1640 1 -- 172.21.15.67:0/3304693587 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbeac06bbd0 msgr2=0x7fbeac06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:40.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.528+0000 7fbec5cd1640 1 --2- 172.21.15.67:0/3304693587 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbeac06bbd0 0x7fbeac06e090 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbec015b160 tx=0x7fbeb4009190 comp rx=0 tx=0).stop 2024-03-24T00:39:40.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.528+0000 7fbec5cd1640 1 -- 172.21.15.67:0/3304693587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbec00b57a0 msgr2=0x7fbec0160150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:40.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.528+0000 7fbec5cd1640 1 --2- 172.21.15.67:0/3304693587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbec00b57a0 0x7fbec0160150 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fbeb002b9c0 tx=0x7fbeb00046c0 comp rx=0 tx=0).stop 2024-03-24T00:39:40.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.528+0000 7fbec5cd1640 1 -- 172.21.15.67:0/3304693587 shutdown_connections 2024-03-24T00:39:40.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.528+0000 7fbec5cd1640 1 --2- 172.21.15.67:0/3304693587 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbeac06bbd0 0x7fbeac06e090 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:40.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.528+0000 7fbec5cd1640 1 --2- 172.21.15.67:0/3304693587 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbec015e130 0x7fbec014cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:40.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.528+0000 7fbec5cd1640 1 --2- 172.21.15.67:0/3304693587 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbec00d0640 0x7fbec0157ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:40.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.528+0000 7fbec5cd1640 1 --2- 172.21.15.67:0/3304693587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbec00b57a0 0x7fbec0160150 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:40.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.528+0000 7fbec5cd1640 1 -- 172.21.15.67:0/3304693587 >> 172.21.15.67:0/3304693587 conn(0x7fbec006c8a0 msgr2=0x7fbec00b5c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:40.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.528+0000 7fbec5cd1640 1 -- 172.21.15.67:0/3304693587 shutdown_connections 2024-03-24T00:39:40.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.528+0000 7fbec5cd1640 1 -- 172.21.15.67:0/3304693587 wait complete. 2024-03-24T00:39:40.539 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:39:40.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.643+0000 7f02c6212640 1 Processor -- start 2024-03-24T00:39:40.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.643+0000 7f02c6212640 1 -- start start 2024-03-24T00:39:40.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.644+0000 7f02c6212640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02c015dea0 0x7f02c00d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:40.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.644+0000 7f02c6212640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f02c00b6170 0x7f02c00d3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:40.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.644+0000 7f02c6212640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02c00d2250 con 0x7f02c015e280 2024-03-24T00:39:40.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.644+0000 7f02c6212640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02c00d2530 con 0x7f02c00b6170 2024-03-24T00:39:40.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.644+0000 7f02c6212640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02c00d2810 con 0x7f02c015dea0 2024-03-24T00:39:40.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.644+0000 7f02bf7fe640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02c015dea0 0x7f02c00d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:40.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.644+0000 7f02bffff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f02c00b6170 0x7f02c00d3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:40.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.644+0000 7f02beffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f02c015e280 0x7f02c00b5a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:39580/0 (socket says 172.21.15.67:39580) 2024-03-24T00:39:40.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.644+0000 7f02beffd640 1 -- 172.21.15.67:0/2885484624 learned_addr learned my addr 172.21.15.67:0/2885484624 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:40.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.644+0000 7f02bf7fe640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02c015dea0 0x7f02c00d33d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.67:33938/0 (socket says 172.21.15.67:33938) 2024-03-24T00:39:40.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.645+0000 7f02bffff640 1 -- 172.21.15.67:0/2885484624 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02c015dea0 msgr2=0x7f02c00d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:40.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.645+0000 7f02be7fc640 1 -- 172.21.15.67:0/2885484624 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3161250222 0 0) 0x7f02c00d2250 con 0x7f02c015e280 2024-03-24T00:39:40.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.645+0000 7f02bffff640 1 --2- 172.21.15.67:0/2885484624 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02c015dea0 0x7f02c00d33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:40.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.645+0000 7f02bffff640 1 -- 172.21.15.67:0/2885484624 >> v1:172.21.15.72:6789/0 conn(0x7f02c015e280 legacy=0x7f02c00b5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:40.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.645+0000 7f02bffff640 1 -- 172.21.15.67:0/2885484624 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02c00d2af0 con 0x7f02c00b6170 2024-03-24T00:39:40.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.645+0000 7f02bffff640 1 --2- 172.21.15.67:0/2885484624 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02c00b6170 0x7f02c00d3910 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f02b80097a0 tx=0x7f02b802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb91d7d2d99c2995 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:40.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.646+0000 7f02be7fc640 1 -- 172.21.15.67:0/2885484624 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b80315a0 con 0x7f02c00b6170 2024-03-24T00:39:40.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.646+0000 7f02be7fc640 1 -- 172.21.15.67:0/2885484624 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02b8037070 con 0x7f02c00b6170 2024-03-24T00:39:40.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.646+0000 7f02be7fc640 1 -- 172.21.15.67:0/2885484624 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b8031960 con 0x7f02c00b6170 2024-03-24T00:39:40.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.646+0000 7f02c6212640 1 -- 172.21.15.67:0/2885484624 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02c00b6170 msgr2=0x7f02c00d3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:40.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.646+0000 7f02c6212640 1 --2- 172.21.15.67:0/2885484624 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02c00b6170 0x7f02c00d3910 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f02b80097a0 tx=0x7f02b802b840 comp rx=0 tx=0).stop 2024-03-24T00:39:40.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.646+0000 7f02c6212640 1 -- 172.21.15.67:0/2885484624 shutdown_connections 2024-03-24T00:39:40.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.646+0000 7f02c6212640 1 --2- 172.21.15.67:0/2885484624 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02c00b6170 0x7f02c00d3910 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:40.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.646+0000 7f02c6212640 1 --2- 172.21.15.67:0/2885484624 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02c015dea0 0x7f02c00d33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:40.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.646+0000 7f02c6212640 1 -- 172.21.15.67:0/2885484624 >> 172.21.15.67:0/2885484624 conn(0x7f02c0072da0 msgr2=0x7f02c00d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:40.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.646+0000 7f02c6212640 1 -- 172.21.15.67:0/2885484624 shutdown_connections 2024-03-24T00:39:40.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.646+0000 7f02c6212640 1 -- 172.21.15.67:0/2885484624 wait complete. 2024-03-24T00:39:40.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.647+0000 7f02c6212640 1 Processor -- start 2024-03-24T00:39:40.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.647+0000 7f02c6212640 1 -- start start 2024-03-24T00:39:40.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.648+0000 7f02c6212640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02c00b6170 0x7f02c014e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:40.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.648+0000 7f02c6212640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02c015dea0 0x7f02c00cdde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:40.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.648+0000 7f02c6212640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02c015e280 0x7f02c00b81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:40.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.648+0000 7f02c6212640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c00b7480 con 0x7f02c00b6170 2024-03-24T00:39:40.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.648+0000 7f02c6212640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c00b7300 con 0x7f02c015dea0 2024-03-24T00:39:40.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.648+0000 7f02c6212640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02c00b7600 con 0x7f02c015e280 2024-03-24T00:39:40.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.648+0000 7f02bf7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02c00b6170 0x7f02c014e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:40.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.648+0000 7f02bf7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02c00b6170 0x7f02c014e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50406/0 (socket says 172.21.15.67:50406) 2024-03-24T00:39:40.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.648+0000 7f02bf7fe640 1 -- 172.21.15.67:0/4260420769 learned_addr learned my addr 172.21.15.67:0/4260420769 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:40.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.648+0000 7f02bffff640 1 --2- 172.21.15.67:0/4260420769 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02c015e280 0x7f02c00b81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:40.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.648+0000 7f02beffd640 1 --2- 172.21.15.67:0/4260420769 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02c015dea0 0x7f02c00cdde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:40.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.649+0000 7f02bf7fe640 1 -- 172.21.15.67:0/4260420769 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02c015e280 msgr2=0x7f02c00b81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:40.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.649+0000 7f02bf7fe640 1 --2- 172.21.15.67:0/4260420769 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02c015e280 0x7f02c00b81f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:40.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.649+0000 7f02bf7fe640 1 -- 172.21.15.67:0/4260420769 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02c015dea0 msgr2=0x7f02c00cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:40.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.649+0000 7f02bf7fe640 1 --2- 172.21.15.67:0/4260420769 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02c015dea0 0x7f02c00cdde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:40.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.649+0000 7f02bf7fe640 1 -- 172.21.15.67:0/4260420769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02c014e940 con 0x7f02c00b6170 2024-03-24T00:39:40.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.649+0000 7f02bf7fe640 1 --2- 172.21.15.67:0/4260420769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02c00b6170 0x7f02c014e400 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f02b0009db0 tx=0x7f02b000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:40.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.649+0000 7f02a7fff640 1 -- 172.21.15.67:0/4260420769 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b0018020 con 0x7f02c00b6170 2024-03-24T00:39:40.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.649+0000 7f02a7fff640 1 -- 172.21.15.67:0/4260420769 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02b0014480 con 0x7f02c00b6170 2024-03-24T00:39:40.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.649+0000 7f02c6212640 1 -- 172.21.15.67:0/4260420769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02c01582b0 con 0x7f02c00b6170 2024-03-24T00:39:40.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.649+0000 7f02a7fff640 1 -- 172.21.15.67:0/4260420769 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b0014850 con 0x7f02c00b6170 2024-03-24T00:39:40.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.649+0000 7f02c6212640 1 -- 172.21.15.67:0/4260420769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02c0158870 con 0x7f02c00b6170 2024-03-24T00:39:40.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.650+0000 7f02c6212640 1 -- 172.21.15.67:0/4260420769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02c004f400 con 0x7f02c00b6170 2024-03-24T00:39:40.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.650+0000 7f02a7fff640 1 -- 172.21.15.67:0/4260420769 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f02b0005d10 con 0x7f02c00b6170 2024-03-24T00:39:40.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.651+0000 7f02a7fff640 1 --2- 172.21.15.67:0/4260420769 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f02ac06bc00 0x7f02ac06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:40.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.651+0000 7f02a7fff640 1 -- 172.21.15.67:0/4260420769 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f02b008b7f0 con 0x7f02c00b6170 2024-03-24T00:39:40.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.651+0000 7f02beffd640 1 --2- 172.21.15.67:0/4260420769 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f02ac06bc00 0x7f02ac06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:40.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.651+0000 7f02beffd640 1 --2- 172.21.15.67:0/4260420769 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f02ac06bc00 0x7f02ac06e0c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f02c00b8fa0 tx=0x7f02b4007e40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:40.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.655+0000 7f02a7fff640 1 -- 172.21.15.67:0/4260420769 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f02b0022070 con 0x7f02c00b6170 2024-03-24T00:39:40.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.838+0000 7f02c6212640 1 -- 172.21.15.67:0/4260420769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f02c00ca640 con 0x7f02c00b6170 2024-03-24T00:39:40.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.838+0000 7f02a7fff640 1 -- 172.21.15.67:0/4260420769 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f02b0010040 con 0x7f02c00b6170 2024-03-24T00:39:40.839 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-03-24T00:39:40.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.840+0000 7f02c6212640 1 -- 172.21.15.67:0/4260420769 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f02ac06bc00 msgr2=0x7f02ac06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:40.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.840+0000 7f02c6212640 1 --2- 172.21.15.67:0/4260420769 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f02ac06bc00 0x7f02ac06e0c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f02c00b8fa0 tx=0x7f02b4007e40 comp rx=0 tx=0).stop 2024-03-24T00:39:40.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.840+0000 7f02c6212640 1 -- 172.21.15.67:0/4260420769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02c00b6170 msgr2=0x7f02c014e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:40.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.840+0000 7f02c6212640 1 --2- 172.21.15.67:0/4260420769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02c00b6170 0x7f02c014e400 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f02b0009db0 tx=0x7f02b000c490 comp rx=0 tx=0).stop 2024-03-24T00:39:40.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.840+0000 7f02c6212640 1 -- 172.21.15.67:0/4260420769 shutdown_connections 2024-03-24T00:39:40.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.840+0000 7f02c6212640 1 --2- 172.21.15.67:0/4260420769 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f02ac06bc00 0x7f02ac06e0c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:40.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.840+0000 7f02c6212640 1 --2- 172.21.15.67:0/4260420769 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02c015e280 0x7f02c00b81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:40.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.840+0000 7f02c6212640 1 --2- 172.21.15.67:0/4260420769 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02c015dea0 0x7f02c00cdde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:40.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.840+0000 7f02c6212640 1 --2- 172.21.15.67:0/4260420769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02c00b6170 0x7f02c014e400 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:40.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.840+0000 7f02c6212640 1 -- 172.21.15.67:0/4260420769 >> 172.21.15.67:0/4260420769 conn(0x7f02c0072da0 msgr2=0x7f02c00b4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:40.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.840+0000 7f02c6212640 1 -- 172.21.15.67:0/4260420769 shutdown_connections 2024-03-24T00:39:40.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.840+0000 7f02c6212640 1 -- 172.21.15.67:0/4260420769 wait complete. 2024-03-24T00:39:40.852 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T00:39:40.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.956+0000 7effda085640 1 Processor -- start 2024-03-24T00:39:40.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.956+0000 7effda085640 1 -- start start 2024-03-24T00:39:40.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.957+0000 7effda085640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7effd415dea0 0x7effd40d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:40.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.957+0000 7effda085640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7effd40b6170 0x7effd40d3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:40.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.957+0000 7effda085640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effd40d2250 con 0x7effd415e280 2024-03-24T00:39:40.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.957+0000 7effda085640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effd40d2530 con 0x7effd40b6170 2024-03-24T00:39:40.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.957+0000 7effda085640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effd40d2810 con 0x7effd415dea0 2024-03-24T00:39:40.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.957+0000 7effd3fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7effd40b6170 0x7effd40d3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:40.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.957+0000 7effd3fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7effd40b6170 0x7effd40d3910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50414/0 (socket says 172.21.15.67:50414) 2024-03-24T00:39:40.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.957+0000 7effd37fe640 1 --2- 172.21.15.67:0/1491686440 >> v2:172.21.15.72:3300/0 conn(0x7effd415dea0 0x7effd40d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:40.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.957+0000 7effd3fff640 1 -- 172.21.15.67:0/1491686440 learned_addr learned my addr 172.21.15.67:0/1491686440 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:40.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.958+0000 7effd3fff640 1 -- 172.21.15.67:0/1491686440 >> v2:172.21.15.72:3300/0 conn(0x7effd415dea0 msgr2=0x7effd40d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:40.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.958+0000 7effd3fff640 1 --2- 172.21.15.67:0/1491686440 >> v2:172.21.15.72:3300/0 conn(0x7effd415dea0 0x7effd40d33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:40.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.958+0000 7effd3fff640 1 -- 172.21.15.67:0/1491686440 >> v1:172.21.15.72:6789/0 conn(0x7effd415e280 legacy=0x7effd40b5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:40.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.958+0000 7effd3fff640 1 -- 172.21.15.67:0/1491686440 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effd40d2af0 con 0x7effd40b6170 2024-03-24T00:39:40.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.958+0000 7effd37fe640 1 --2- 172.21.15.67:0/1491686440 >> v2:172.21.15.72:3300/0 conn(0x7effd415dea0 0x7effd40d33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:40.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.958+0000 7effd27fc640 1 -- 172.21.15.67:0/1491686440 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2513368494 0 0) 0x7effd40d2250 con 0x7effd415e280 2024-03-24T00:39:40.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.958+0000 7effd3fff640 1 --2- 172.21.15.67:0/1491686440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effd40b6170 0x7effd40d3910 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7effcc009870 tx=0x7effcc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf9c699c6d446e79 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:40.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.959+0000 7effd27fc640 1 -- 172.21.15.67:0/1491686440 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effcc035020 con 0x7effd40b6170 2024-03-24T00:39:40.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.959+0000 7effd27fc640 1 -- 172.21.15.67:0/1491686440 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effcc031870 con 0x7effd40b6170 2024-03-24T00:39:40.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.959+0000 7effd27fc640 1 -- 172.21.15.67:0/1491686440 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effcc031c40 con 0x7effd40b6170 2024-03-24T00:39:40.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.959+0000 7effda085640 1 -- 172.21.15.67:0/1491686440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effd40b6170 msgr2=0x7effd40d3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:40.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.959+0000 7effda085640 1 --2- 172.21.15.67:0/1491686440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effd40b6170 0x7effd40d3910 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7effcc009870 tx=0x7effcc02b910 comp rx=0 tx=0).stop 2024-03-24T00:39:40.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.959+0000 7effda085640 1 -- 172.21.15.67:0/1491686440 shutdown_connections 2024-03-24T00:39:40.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.959+0000 7effda085640 1 --2- 172.21.15.67:0/1491686440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effd40b6170 0x7effd40d3910 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:40.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.959+0000 7effda085640 1 --2- 172.21.15.67:0/1491686440 >> v2:172.21.15.72:3300/0 conn(0x7effd415dea0 0x7effd40d33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:40.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.959+0000 7effda085640 1 -- 172.21.15.67:0/1491686440 >> 172.21.15.67:0/1491686440 conn(0x7effd4072da0 msgr2=0x7effd40d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:40.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.959+0000 7effda085640 1 -- 172.21.15.67:0/1491686440 shutdown_connections 2024-03-24T00:39:40.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.959+0000 7effda085640 1 -- 172.21.15.67:0/1491686440 wait complete. 2024-03-24T00:39:40.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.960+0000 7effda085640 1 Processor -- start 2024-03-24T00:39:40.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.960+0000 7effda085640 1 -- start start 2024-03-24T00:39:40.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.961+0000 7effda085640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7effd40b6170 0x7effd414ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:40.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.961+0000 7effda085640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effd415dea0 0x7effd41f61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:40.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.961+0000 7effda085640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7effd415e280 0x7effd415ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:40.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.961+0000 7effda085640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd40b7480 con 0x7effd415dea0 2024-03-24T00:39:40.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.961+0000 7effda085640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd40b7300 con 0x7effd415e280 2024-03-24T00:39:40.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.961+0000 7effda085640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effd40b7600 con 0x7effd40b6170 2024-03-24T00:39:40.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.961+0000 7effd2ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effd415dea0 0x7effd41f61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:40.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.961+0000 7effd2ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effd415dea0 0x7effd41f61e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50424/0 (socket says 172.21.15.67:50424) 2024-03-24T00:39:40.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.961+0000 7effd2ffd640 1 -- 172.21.15.67:0/616956832 learned_addr learned my addr 172.21.15.67:0/616956832 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:40.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.961+0000 7effd37fe640 1 --2- 172.21.15.67:0/616956832 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7effd40b6170 0x7effd414ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:40.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.961+0000 7effd3fff640 1 --2- 172.21.15.67:0/616956832 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7effd415e280 0x7effd415ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:40.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.962+0000 7effd2ffd640 1 -- 172.21.15.67:0/616956832 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7effd40b6170 msgr2=0x7effd414ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:40.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.962+0000 7effd2ffd640 1 --2- 172.21.15.67:0/616956832 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7effd40b6170 0x7effd414ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:40.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.962+0000 7effd2ffd640 1 -- 172.21.15.67:0/616956832 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7effd415e280 msgr2=0x7effd415ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:40.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.962+0000 7effd2ffd640 1 --2- 172.21.15.67:0/616956832 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7effd415e280 0x7effd415ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:40.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.962+0000 7effd2ffd640 1 -- 172.21.15.67:0/616956832 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effd415fa70 con 0x7effd415dea0 2024-03-24T00:39:40.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.962+0000 7effd2ffd640 1 --2- 172.21.15.67:0/616956832 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effd415dea0 0x7effd41f61e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7effc800c410 tx=0x7effc800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:40.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.962+0000 7effbbfff640 1 -- 172.21.15.67:0/616956832 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc8017070 con 0x7effd415dea0 2024-03-24T00:39:40.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.962+0000 7effbbfff640 1 -- 172.21.15.67:0/616956832 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effc80035f0 con 0x7effd415dea0 2024-03-24T00:39:40.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.962+0000 7effda085640 1 -- 172.21.15.67:0/616956832 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effd415fd50 con 0x7effd415dea0 2024-03-24T00:39:40.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.962+0000 7effbbfff640 1 -- 172.21.15.67:0/616956832 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc80039e0 con 0x7effd415dea0 2024-03-24T00:39:40.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.962+0000 7effda085640 1 -- 172.21.15.67:0/616956832 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effd4160310 con 0x7effd415dea0 2024-03-24T00:39:40.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.963+0000 7effda085640 1 -- 172.21.15.67:0/616956832 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effd404f400 con 0x7effd415dea0 2024-03-24T00:39:40.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.963+0000 7effbbfff640 1 -- 172.21.15.67:0/616956832 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7effc801a070 con 0x7effd415dea0 2024-03-24T00:39:40.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.964+0000 7effbbfff640 1 --2- 172.21.15.67:0/616956832 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7effc006bbd0 0x7effc006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:40.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.964+0000 7effbbfff640 1 -- 172.21.15.67:0/616956832 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7effc808a190 con 0x7effd415dea0 2024-03-24T00:39:40.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.964+0000 7effd37fe640 1 --2- 172.21.15.67:0/616956832 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7effc006bbd0 0x7effc006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:40.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.965+0000 7effd37fe640 1 --2- 172.21.15.67:0/616956832 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7effc006bbd0 0x7effc006e090 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7effc4009bd0 tx=0x7effc400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:40.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:40.968+0000 7effbbfff640 1 -- 172.21.15.67:0/616956832 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7effc80582c0 con 0x7effd415dea0 2024-03-24T00:39:41.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.151+0000 7effda085640 1 -- 172.21.15.67:0/616956832 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7effd41f6720 con 0x7effd415dea0 2024-03-24T00:39:41.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.152+0000 7effbbfff640 1 -- 172.21.15.67:0/616956832 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7effc805c360 con 0x7effd415dea0 2024-03-24T00:39:41.152 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-24T00:39:41.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.154+0000 7effda085640 1 -- 172.21.15.67:0/616956832 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7effc006bbd0 msgr2=0x7effc006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:41.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.154+0000 7effda085640 1 --2- 172.21.15.67:0/616956832 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7effc006bbd0 0x7effc006e090 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7effc4009bd0 tx=0x7effc400b040 comp rx=0 tx=0).stop 2024-03-24T00:39:41.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.154+0000 7effda085640 1 -- 172.21.15.67:0/616956832 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effd415dea0 msgr2=0x7effd41f61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:41.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.154+0000 7effda085640 1 --2- 172.21.15.67:0/616956832 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effd415dea0 0x7effd41f61e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7effc800c410 tx=0x7effc800c8d0 comp rx=0 tx=0).stop 2024-03-24T00:39:41.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.154+0000 7effda085640 1 -- 172.21.15.67:0/616956832 shutdown_connections 2024-03-24T00:39:41.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.154+0000 7effda085640 1 --2- 172.21.15.67:0/616956832 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7effc006bbd0 0x7effc006e090 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:41.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.154+0000 7effda085640 1 --2- 172.21.15.67:0/616956832 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7effd415e280 0x7effd415ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:41.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.154+0000 7effda085640 1 --2- 172.21.15.67:0/616956832 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effd415dea0 0x7effd41f61e0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:41.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.154+0000 7effda085640 1 --2- 172.21.15.67:0/616956832 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7effd40b6170 0x7effd414ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:41.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.154+0000 7effda085640 1 -- 172.21.15.67:0/616956832 >> 172.21.15.67:0/616956832 conn(0x7effd4072da0 msgr2=0x7effd40d09b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:41.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.154+0000 7effda085640 1 -- 172.21.15.67:0/616956832 shutdown_connections 2024-03-24T00:39:41.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.154+0000 7effda085640 1 -- 172.21.15.67:0/616956832 wait complete. 2024-03-24T00:39:41.165 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T00:39:41.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.267+0000 7fede3dd4640 1 Processor -- start 2024-03-24T00:39:41.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.267+0000 7fede3dd4640 1 -- start start 2024-03-24T00:39:41.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.268+0000 7fede3dd4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7feddc06de60 0x7feddc09ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:41.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.268+0000 7fede3dd4640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feddc0b29b0 con 0x7feddc09a410 2024-03-24T00:39:41.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.268+0000 7fede3dd4640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feddc0b23f0 con 0x7feddc0b1eb0 2024-03-24T00:39:41.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.268+0000 7fede3dd4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feddc0cc9e0 con 0x7feddc06de60 2024-03-24T00:39:41.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.268+0000 7fede2371640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7feddc09a410 0x7feddc0cd410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41552/0 (socket says 172.21.15.67:41552) 2024-03-24T00:39:41.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.268+0000 7fede2371640 1 -- 172.21.15.67:0/1054143327 learned_addr learned my addr 172.21.15.67:0/1054143327 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:41.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.268+0000 7fede1b70640 1 --2- 172.21.15.67:0/1054143327 >> v2:172.21.15.72:3300/0 conn(0x7feddc06de60 0x7feddc09ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:41.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.268+0000 7fede0b6e640 1 -- 172.21.15.67:0/1054143327 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 161180983 0 0) 0x7feddc0b29b0 con 0x7feddc09a410 2024-03-24T00:39:41.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.268+0000 7fede0b6e640 1 -- 172.21.15.67:0/1054143327 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedc4003520 con 0x7feddc09a410 2024-03-24T00:39:41.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.269+0000 7fede1b70640 1 -- 172.21.15.67:0/1054143327 >> v1:172.21.15.72:6789/0 conn(0x7feddc0b1eb0 legacy=0x7feddc099d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:41.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.269+0000 7fede0b6e640 1 -- 172.21.15.67:0/1054143327 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1118174820 0 0) 0x7feddc0b23f0 con 0x7feddc0b1eb0 2024-03-24T00:39:41.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.269+0000 7fede1b70640 1 -- 172.21.15.67:0/1054143327 >> v1:172.21.15.67:6789/0 conn(0x7feddc09a410 legacy=0x7feddc0cd410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:41.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.269+0000 7fede1b70640 1 -- 172.21.15.67:0/1054143327 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feddc0b26d0 con 0x7feddc06de60 2024-03-24T00:39:41.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.269+0000 7fede1b70640 1 --2- 172.21.15.67:0/1054143327 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feddc06de60 0x7feddc09ae80 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fedd40097a0 tx=0x7fedd402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17d33c037ac57d9a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:41.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.269+0000 7fede0b6e640 1 -- 172.21.15.67:0/1054143327 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd4034070 con 0x7feddc06de60 2024-03-24T00:39:41.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.269+0000 7fede0b6e640 1 -- 172.21.15.67:0/1054143327 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedd4030610 con 0x7feddc06de60 2024-03-24T00:39:41.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.270+0000 7fede0b6e640 1 -- 172.21.15.67:0/1054143327 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd40309c0 con 0x7feddc06de60 2024-03-24T00:39:41.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.270+0000 7fede3dd4640 1 -- 172.21.15.67:0/1054143327 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feddc06de60 msgr2=0x7feddc09ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:41.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.270+0000 7fede3dd4640 1 --2- 172.21.15.67:0/1054143327 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feddc06de60 0x7feddc09ae80 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fedd40097a0 tx=0x7fedd402b960 comp rx=0 tx=0).stop 2024-03-24T00:39:41.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.270+0000 7fede3dd4640 1 -- 172.21.15.67:0/1054143327 shutdown_connections 2024-03-24T00:39:41.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.270+0000 7fede3dd4640 1 --2- 172.21.15.67:0/1054143327 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feddc06de60 0x7feddc09ae80 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:41.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.270+0000 7fede3dd4640 1 -- 172.21.15.67:0/1054143327 >> 172.21.15.67:0/1054143327 conn(0x7feddc0b3650 msgr2=0x7feddc0b3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:41.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.270+0000 7fede3dd4640 1 -- 172.21.15.67:0/1054143327 shutdown_connections 2024-03-24T00:39:41.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.270+0000 7fede3dd4640 1 -- 172.21.15.67:0/1054143327 wait complete. 2024-03-24T00:39:41.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.271+0000 7fede3dd4640 1 Processor -- start 2024-03-24T00:39:41.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.271+0000 7fede3dd4640 1 -- start start 2024-03-24T00:39:41.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.272+0000 7fede3dd4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feddc06de60 0x7feddc150630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:41.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.272+0000 7fede3dd4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7feddc09a410 0x7feddc1f3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:41.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.272+0000 7fede3dd4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feddc0b1eb0 0x7feddc09b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:41.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.272+0000 7fede3dd4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feddc09b3c0 con 0x7feddc0b1eb0 2024-03-24T00:39:41.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.272+0000 7fede3dd4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feddc0ccf80 con 0x7feddc06de60 2024-03-24T00:39:41.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.272+0000 7fede3dd4640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feddc06c150 con 0x7feddc09a410 2024-03-24T00:39:41.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.272+0000 7fede2371640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feddc0b1eb0 0x7feddc09b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:41.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.272+0000 7fede2371640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feddc0b1eb0 0x7feddc09b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50440/0 (socket says 172.21.15.67:50440) 2024-03-24T00:39:41.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.272+0000 7fede2371640 1 -- 172.21.15.67:0/3611049704 learned_addr learned my addr 172.21.15.67:0/3611049704 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:41.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.272+0000 7fede1b70640 1 --2- 172.21.15.67:0/3611049704 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feddc06de60 0x7feddc150630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:41.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.272+0000 7fede136f640 1 --2- 172.21.15.67:0/3611049704 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7feddc09a410 0x7feddc1f3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:41.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.272+0000 7fede2371640 1 -- 172.21.15.67:0/3611049704 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7feddc09a410 msgr2=0x7feddc1f3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:41.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.272+0000 7fede2371640 1 --2- 172.21.15.67:0/3611049704 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7feddc09a410 0x7feddc1f3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:41.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.272+0000 7fede2371640 1 -- 172.21.15.67:0/3611049704 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feddc06de60 msgr2=0x7feddc150630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:41.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.272+0000 7fede2371640 1 --2- 172.21.15.67:0/3611049704 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feddc06de60 0x7feddc150630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:41.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.272+0000 7fede2371640 1 -- 172.21.15.67:0/3611049704 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feddc1f43a0 con 0x7feddc0b1eb0 2024-03-24T00:39:41.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.273+0000 7fede2371640 1 --2- 172.21.15.67:0/3611049704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feddc0b1eb0 0x7feddc09b860 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fedcc00caa0 tx=0x7fedcc00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:41.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.273+0000 7fedda7fc640 1 -- 172.21.15.67:0/3611049704 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedcc017070 con 0x7feddc0b1eb0 2024-03-24T00:39:41.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.273+0000 7fedda7fc640 1 -- 172.21.15.67:0/3611049704 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedcc00aba0 con 0x7feddc0b1eb0 2024-03-24T00:39:41.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.273+0000 7fede3dd4640 1 -- 172.21.15.67:0/3611049704 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feddc1f4680 con 0x7feddc0b1eb0 2024-03-24T00:39:41.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.273+0000 7fedda7fc640 1 -- 172.21.15.67:0/3611049704 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedcc012400 con 0x7feddc0b1eb0 2024-03-24T00:39:41.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.273+0000 7fede3dd4640 1 -- 172.21.15.67:0/3611049704 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feddc150de0 con 0x7feddc0b1eb0 2024-03-24T00:39:41.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.274+0000 7fede3dd4640 1 -- 172.21.15.67:0/3611049704 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feddc04f400 con 0x7feddc0b1eb0 2024-03-24T00:39:41.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.274+0000 7fedda7fc640 1 -- 172.21.15.67:0/3611049704 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fedcc01a070 con 0x7feddc0b1eb0 2024-03-24T00:39:41.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.275+0000 7fedda7fc640 1 --2- 172.21.15.67:0/3611049704 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fedc806bbd0 0x7fedc806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:41.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.275+0000 7fedda7fc640 1 -- 172.21.15.67:0/3611049704 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fedcc08aab0 con 0x7feddc0b1eb0 2024-03-24T00:39:41.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.275+0000 7fede1b70640 1 --2- 172.21.15.67:0/3611049704 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fedc806bbd0 0x7fedc806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:41.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.275+0000 7fede1b70640 1 --2- 172.21.15.67:0/3611049704 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fedc806bbd0 0x7fedc806e090 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fedd4036000 tx=0x7fedd403b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:41.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.279+0000 7fedda7fc640 1 -- 172.21.15.67:0/3611049704 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fedcc058be0 con 0x7feddc0b1eb0 2024-03-24T00:39:41.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.473+0000 7fede3dd4640 1 -- 172.21.15.67:0/3611049704 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7feddc0b1210 con 0x7feddc0b1eb0 2024-03-24T00:39:41.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.474+0000 7fedda7fc640 1 -- 172.21.15.67:0/3611049704 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7fedcc05cc80 con 0x7feddc0b1eb0 2024-03-24T00:39:41.475 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-03-24T00:39:41.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.476+0000 7fede3dd4640 1 -- 172.21.15.67:0/3611049704 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fedc806bbd0 msgr2=0x7fedc806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:41.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.476+0000 7fede3dd4640 1 --2- 172.21.15.67:0/3611049704 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fedc806bbd0 0x7fedc806e090 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fedd4036000 tx=0x7fedd403b040 comp rx=0 tx=0).stop 2024-03-24T00:39:41.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.476+0000 7fede3dd4640 1 -- 172.21.15.67:0/3611049704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feddc0b1eb0 msgr2=0x7feddc09b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:41.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.476+0000 7fede3dd4640 1 --2- 172.21.15.67:0/3611049704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feddc0b1eb0 0x7feddc09b860 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fedcc00caa0 tx=0x7fedcc00cf60 comp rx=0 tx=0).stop 2024-03-24T00:39:41.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.476+0000 7fede3dd4640 1 -- 172.21.15.67:0/3611049704 shutdown_connections 2024-03-24T00:39:41.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.476+0000 7fede3dd4640 1 --2- 172.21.15.67:0/3611049704 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fedc806bbd0 0x7fedc806e090 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:41.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.476+0000 7fede3dd4640 1 --2- 172.21.15.67:0/3611049704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feddc0b1eb0 0x7feddc09b860 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:41.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.476+0000 7fede3dd4640 1 --2- 172.21.15.67:0/3611049704 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7feddc09a410 0x7feddc1f3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:41.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.476+0000 7fede3dd4640 1 --2- 172.21.15.67:0/3611049704 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feddc06de60 0x7feddc150630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:41.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.476+0000 7fede3dd4640 1 -- 172.21.15.67:0/3611049704 >> 172.21.15.67:0/3611049704 conn(0x7feddc0b3650 msgr2=0x7feddc0b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:41.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.476+0000 7fede3dd4640 1 -- 172.21.15.67:0/3611049704 shutdown_connections 2024-03-24T00:39:41.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.476+0000 7fede3dd4640 1 -- 172.21.15.67:0/3611049704 wait complete. 2024-03-24T00:39:41.487 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-03-24T00:39:41.487 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:41.487 DEBUG:teuthology.orchestra.run.smithi072:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2024-03-24T00:39:41.516 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi072.front.sepia.ceph.com 2024-03-24T00:39:41.516 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:41.516 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-24T00:39:41.581 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:39:41.581 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:39:41.596 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:39:41.596 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:39:41.650 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:39:41.650 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:39:41.705 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:39:41.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.846+0000 7fda9b33b640 1 Processor -- start 2024-03-24T00:39:41.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.846+0000 7fda9b33b640 1 -- start start 2024-03-24T00:39:41.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.847+0000 7fda9b33b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fda9409a410 0x7fda940cfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:41.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.847+0000 7fda9b33b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fda940b43a0 0x7fda940cc880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:41.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.847+0000 7fda9b33b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda940d0fc0 con 0x7fda9415e1b0 2024-03-24T00:39:41.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.847+0000 7fda9b33b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda9415e590 con 0x7fda940b43a0 2024-03-24T00:39:41.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.847+0000 7fda9b33b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda940d1820 con 0x7fda9409a410 2024-03-24T00:39:41.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.847+0000 7fda998d8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fda940b43a0 0x7fda940cc880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:41.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.847+0000 7fda998d8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fda940b43a0 0x7fda940cc880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50442/0 (socket says 172.21.15.67:50442) 2024-03-24T00:39:41.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.847+0000 7fda998d8640 1 -- 172.21.15.67:0/2806554122 learned_addr learned my addr 172.21.15.67:0/2806554122 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:41.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.847+0000 7fda988d6640 1 --2- 172.21.15.67:0/2806554122 >> v2:172.21.15.72:3300/0 conn(0x7fda9409a410 0x7fda940cfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:41.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.847+0000 7fda998d8640 1 -- 172.21.15.67:0/2806554122 >> v2:172.21.15.72:3300/0 conn(0x7fda9409a410 msgr2=0x7fda940cfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:41.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.847+0000 7fda998d8640 1 --2- 172.21.15.67:0/2806554122 >> v2:172.21.15.72:3300/0 conn(0x7fda9409a410 0x7fda940cfe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:41.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.847+0000 7fda998d8640 1 -- 172.21.15.67:0/2806554122 >> v1:172.21.15.72:6789/0 conn(0x7fda9415e1b0 legacy=0x7fda94099d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:41.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.847+0000 7fda998d8640 1 -- 172.21.15.67:0/2806554122 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda940d1260 con 0x7fda940b43a0 2024-03-24T00:39:41.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.847+0000 7fda988d6640 1 --2- 172.21.15.67:0/2806554122 >> v2:172.21.15.72:3300/0 conn(0x7fda9409a410 0x7fda940cfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:41.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.848+0000 7fda998d8640 1 --2- 172.21.15.67:0/2806554122 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda940b43a0 0x7fda940cc880 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fda940c8c30 tx=0x7fda8c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce37339a5b605e2d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:41.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.848+0000 7fda8bfff640 1 -- 172.21.15.67:0/2806554122 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda8c02c9f0 con 0x7fda940b43a0 2024-03-24T00:39:41.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.848+0000 7fda8bfff640 1 -- 172.21.15.67:0/2806554122 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda8c02cb90 con 0x7fda940b43a0 2024-03-24T00:39:41.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.848+0000 7fda8bfff640 1 -- 172.21.15.67:0/2806554122 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda8c0343f0 con 0x7fda940b43a0 2024-03-24T00:39:41.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.849+0000 7fda9b33b640 1 -- 172.21.15.67:0/2806554122 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda940b43a0 msgr2=0x7fda940cc880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:41.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.849+0000 7fda9b33b640 1 --2- 172.21.15.67:0/2806554122 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda940b43a0 0x7fda940cc880 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fda940c8c30 tx=0x7fda8c02be50 comp rx=0 tx=0).stop 2024-03-24T00:39:41.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.849+0000 7fda9b33b640 1 -- 172.21.15.67:0/2806554122 shutdown_connections 2024-03-24T00:39:41.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.849+0000 7fda9b33b640 1 --2- 172.21.15.67:0/2806554122 >> v2:172.21.15.72:3300/0 conn(0x7fda9409a410 0x7fda940cfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:41.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.849+0000 7fda9b33b640 1 --2- 172.21.15.67:0/2806554122 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda940b43a0 0x7fda940cc880 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:41.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.849+0000 7fda9b33b640 1 -- 172.21.15.67:0/2806554122 >> 172.21.15.67:0/2806554122 conn(0x7fda940b36d0 msgr2=0x7fda940b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:41.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.849+0000 7fda9b33b640 1 -- 172.21.15.67:0/2806554122 shutdown_connections 2024-03-24T00:39:41.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.849+0000 7fda9b33b640 1 -- 172.21.15.67:0/2806554122 wait complete. 2024-03-24T00:39:41.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.850+0000 7fda9b33b640 1 Processor -- start 2024-03-24T00:39:41.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.850+0000 7fda9b33b640 1 -- start start 2024-03-24T00:39:41.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.851+0000 7fda9b33b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fda9409a410 0x7fda941f30e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:41.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.851+0000 7fda9b33b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fda940b43a0 0x7fda941f6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:41.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.851+0000 7fda9b33b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda9415e1b0 0x7fda941f72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:41.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.851+0000 7fda9b33b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda9409a9a0 con 0x7fda9415e1b0 2024-03-24T00:39:41.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.851+0000 7fda9b33b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda940cc360 con 0x7fda940b43a0 2024-03-24T00:39:41.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.851+0000 7fda9b33b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda940ccdc0 con 0x7fda9409a410 2024-03-24T00:39:41.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.851+0000 7fda998d8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda9415e1b0 0x7fda941f72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:41.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.851+0000 7fda998d8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda9415e1b0 0x7fda941f72b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50448/0 (socket says 172.21.15.67:50448) 2024-03-24T00:39:41.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.851+0000 7fda998d8640 1 -- 172.21.15.67:0/2491385255 learned_addr learned my addr 172.21.15.67:0/2491385255 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:41.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.851+0000 7fda988d6640 1 --2- 172.21.15.67:0/2491385255 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fda940b43a0 0x7fda941f6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:41.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.851+0000 7fda990d7640 1 --2- 172.21.15.67:0/2491385255 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fda9409a410 0x7fda941f30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:41.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.851+0000 7fda998d8640 1 -- 172.21.15.67:0/2491385255 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fda9409a410 msgr2=0x7fda941f30e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:41.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.851+0000 7fda998d8640 1 --2- 172.21.15.67:0/2491385255 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fda9409a410 0x7fda941f30e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:41.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.851+0000 7fda998d8640 1 -- 172.21.15.67:0/2491385255 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fda940b43a0 msgr2=0x7fda941f6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:41.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.851+0000 7fda998d8640 1 --2- 172.21.15.67:0/2491385255 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fda940b43a0 0x7fda941f6d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:41.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.851+0000 7fda998d8640 1 -- 172.21.15.67:0/2491385255 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda940c8d50 con 0x7fda9415e1b0 2024-03-24T00:39:41.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.851+0000 7fda988d6640 1 --2- 172.21.15.67:0/2491385255 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fda940b43a0 0x7fda941f6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:41.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.851+0000 7fda998d8640 1 --2- 172.21.15.67:0/2491385255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda9415e1b0 0x7fda941f72b0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fda8c036040 tx=0x7fda8c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:41.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.851+0000 7fda89ffb640 1 -- 172.21.15.67:0/2491385255 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda8c044020 con 0x7fda9415e1b0 2024-03-24T00:39:41.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.852+0000 7fda89ffb640 1 -- 172.21.15.67:0/2491385255 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda8c040470 con 0x7fda9415e1b0 2024-03-24T00:39:41.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.852+0000 7fda9b33b640 1 -- 172.21.15.67:0/2491385255 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda941f3620 con 0x7fda9415e1b0 2024-03-24T00:39:41.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.852+0000 7fda89ffb640 1 -- 172.21.15.67:0/2491385255 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda8c040860 con 0x7fda9415e1b0 2024-03-24T00:39:41.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.852+0000 7fda9b33b640 1 -- 172.21.15.67:0/2491385255 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda940c92a0 con 0x7fda9415e1b0 2024-03-24T00:39:41.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.853+0000 7fda9b33b640 1 -- 172.21.15.67:0/2491385255 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda9404f400 con 0x7fda9415e1b0 2024-03-24T00:39:41.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.853+0000 7fda89ffb640 1 -- 172.21.15.67:0/2491385255 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fda8c047020 con 0x7fda9415e1b0 2024-03-24T00:39:41.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.853+0000 7fda89ffb640 1 --2- 172.21.15.67:0/2491385255 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fda7806bbd0 0x7fda7806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:41.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.854+0000 7fda89ffb640 1 -- 172.21.15.67:0/2491385255 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fda8c0ad550 con 0x7fda9415e1b0 2024-03-24T00:39:41.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.854+0000 7fda990d7640 1 --2- 172.21.15.67:0/2491385255 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fda7806bbd0 0x7fda7806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:41.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.854+0000 7fda990d7640 1 --2- 172.21.15.67:0/2491385255 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fda7806bbd0 0x7fda7806e090 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fda900016e0 tx=0x7fda90009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:41.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:41.858+0000 7fda89ffb640 1 -- 172.21.15.67:0/2491385255 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fda8c04e030 con 0x7fda9415e1b0 2024-03-24T00:39:42.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.041+0000 7fda9b33b640 1 -- 172.21.15.67:0/2491385255 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda940b2da0 con 0x7fda9415e1b0 2024-03-24T00:39:42.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.042+0000 7fda89ffb640 1 -- 172.21.15.67:0/2491385255 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18843 (secure 0 0 0) 0x7fda8c07b680 con 0x7fda9415e1b0 2024-03-24T00:39:42.043 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:39:42.044 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":27,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:39:32.904729+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:38:50.077085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:38:51.095454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T00:38:54.807117+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.14","mappings":[{"from":4,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:39:42.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.044+0000 7fda9b33b640 1 -- 172.21.15.67:0/2491385255 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fda7806bbd0 msgr2=0x7fda7806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:42.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.044+0000 7fda9b33b640 1 --2- 172.21.15.67:0/2491385255 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fda7806bbd0 0x7fda7806e090 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fda900016e0 tx=0x7fda90009040 comp rx=0 tx=0).stop 2024-03-24T00:39:42.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.044+0000 7fda9b33b640 1 -- 172.21.15.67:0/2491385255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda9415e1b0 msgr2=0x7fda941f72b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:42.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.044+0000 7fda9b33b640 1 --2- 172.21.15.67:0/2491385255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda9415e1b0 0x7fda941f72b0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fda8c036040 tx=0x7fda8c004a40 comp rx=0 tx=0).stop 2024-03-24T00:39:42.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.045+0000 7fda9b33b640 1 -- 172.21.15.67:0/2491385255 shutdown_connections 2024-03-24T00:39:42.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.045+0000 7fda9b33b640 1 --2- 172.21.15.67:0/2491385255 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fda7806bbd0 0x7fda7806e090 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:42.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.045+0000 7fda9b33b640 1 --2- 172.21.15.67:0/2491385255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda9415e1b0 0x7fda941f72b0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:42.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.045+0000 7fda9b33b640 1 --2- 172.21.15.67:0/2491385255 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fda940b43a0 0x7fda941f6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:42.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.045+0000 7fda9b33b640 1 --2- 172.21.15.67:0/2491385255 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fda9409a410 0x7fda941f30e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:42.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.045+0000 7fda9b33b640 1 -- 172.21.15.67:0/2491385255 >> 172.21.15.67:0/2491385255 conn(0x7fda940b36d0 msgr2=0x7fda9406fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:42.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.045+0000 7fda9b33b640 1 -- 172.21.15.67:0/2491385255 shutdown_connections 2024-03-24T00:39:42.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.045+0000 7fda9b33b640 1 -- 172.21.15.67:0/2491385255 wait complete. 2024-03-24T00:39:42.056 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:38:39.364950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:38:50.077085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:38:51.095454+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T00:38:54.807117+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T00:39:42.057 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:39:42.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.161+0000 7f61faed1640 1 Processor -- start 2024-03-24T00:39:42.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.161+0000 7f61faed1640 1 -- start start 2024-03-24T00:39:42.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.161+0000 7f61faed1640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f61f40b3ed0 0x7f61f40cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:42.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.161+0000 7f61faed1640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f61f40d0000 0x7f61f406df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:42.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.161+0000 7f61faed1640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f40b2e30 con 0x7f61f41584a0 2024-03-24T00:39:42.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.162+0000 7f61faed1640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61f40d0450 con 0x7f61f40b3ed0 2024-03-24T00:39:42.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.162+0000 7f61faed1640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61f40b4360 con 0x7f61f40d0000 2024-03-24T00:39:42.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.162+0000 7f61f3fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f61f40b3ed0 0x7f61f40cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:42.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.162+0000 7f61f8c6d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f61f41584a0 0x7f61f40728b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41564/0 (socket says 172.21.15.67:41564) 2024-03-24T00:39:42.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.162+0000 7f61f8c6d640 1 -- 172.21.15.67:0/1710760784 learned_addr learned my addr 172.21.15.67:0/1710760784 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:42.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.162+0000 7f61f946e640 1 --2- 172.21.15.67:0/1710760784 >> v2:172.21.15.72:3300/0 conn(0x7f61f40d0000 0x7f61f406df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:42.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.162+0000 7f61f3fff640 1 -- 172.21.15.67:0/1710760784 >> v2:172.21.15.72:3300/0 conn(0x7f61f40d0000 msgr2=0x7f61f406df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:42.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.162+0000 7f61f3fff640 1 --2- 172.21.15.67:0/1710760784 >> v2:172.21.15.72:3300/0 conn(0x7f61f40d0000 0x7f61f406df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:42.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.162+0000 7f61f3fff640 1 -- 172.21.15.67:0/1710760784 >> v1:172.21.15.67:6789/0 conn(0x7f61f41584a0 legacy=0x7f61f40728b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:42.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.162+0000 7f61f37fe640 1 -- 172.21.15.67:0/1710760784 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 205576339 0 0) 0x7f61f40b2e30 con 0x7f61f41584a0 2024-03-24T00:39:42.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.162+0000 7f61f3fff640 1 -- 172.21.15.67:0/1710760784 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f40b3110 con 0x7f61f40b3ed0 2024-03-24T00:39:42.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.163+0000 7f61f3fff640 1 --2- 172.21.15.67:0/1710760784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61f40b3ed0 0x7f61f40cfac0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f61f406feb0 tx=0x7f61dc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=382458b5d3e7837f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:42.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.163+0000 7f61f37fe640 1 -- 172.21.15.67:0/1710760784 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61dc035020 con 0x7f61f40b3ed0 2024-03-24T00:39:42.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.163+0000 7f61f37fe640 1 -- 172.21.15.67:0/1710760784 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61dc031830 con 0x7f61f40b3ed0 2024-03-24T00:39:42.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.163+0000 7f61f37fe640 1 -- 172.21.15.67:0/1710760784 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61dc031c00 con 0x7f61f40b3ed0 2024-03-24T00:39:42.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.163+0000 7f61faed1640 1 -- 172.21.15.67:0/1710760784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61f40b3ed0 msgr2=0x7f61f40cfac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:42.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.163+0000 7f61faed1640 1 --2- 172.21.15.67:0/1710760784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61f40b3ed0 0x7f61f40cfac0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f61f406feb0 tx=0x7f61dc02b8d0 comp rx=0 tx=0).stop 2024-03-24T00:39:42.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.163+0000 7f61faed1640 1 -- 172.21.15.67:0/1710760784 shutdown_connections 2024-03-24T00:39:42.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.163+0000 7f61faed1640 1 --2- 172.21.15.67:0/1710760784 >> v2:172.21.15.72:3300/0 conn(0x7f61f40d0000 0x7f61f406df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:42.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.163+0000 7f61faed1640 1 --2- 172.21.15.67:0/1710760784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61f40b3ed0 0x7f61f40cfac0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:42.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.163+0000 7f61faed1640 1 -- 172.21.15.67:0/1710760784 >> 172.21.15.67:0/1710760784 conn(0x7f61f406b040 msgr2=0x7f61f406b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:42.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.163+0000 7f61faed1640 1 -- 172.21.15.67:0/1710760784 shutdown_connections 2024-03-24T00:39:42.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.163+0000 7f61faed1640 1 -- 172.21.15.67:0/1710760784 wait complete. 2024-03-24T00:39:42.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.165+0000 7f61faed1640 1 Processor -- start 2024-03-24T00:39:42.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.165+0000 7f61faed1640 1 -- start start 2024-03-24T00:39:42.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.165+0000 7f61faed1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f61f40b3ed0 0x7f61f414f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:42.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.165+0000 7f61faed1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61f40d0000 0x7f61f4144960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:42.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.165+0000 7f61faed1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f61f41584a0 0x7f61f4150930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:42.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.165+0000 7f61faed1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61f40b39b0 con 0x7f61f40d0000 2024-03-24T00:39:42.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.165+0000 7f61faed1640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61f40b36d0 con 0x7f61f40b3ed0 2024-03-24T00:39:42.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.165+0000 7f61faed1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61f4158880 con 0x7f61f41584a0 2024-03-24T00:39:42.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.166+0000 7f61f3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61f40d0000 0x7f61f4144960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:42.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.166+0000 7f61f3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61f40d0000 0x7f61f4144960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50474/0 (socket says 172.21.15.67:50474) 2024-03-24T00:39:42.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.166+0000 7f61f3fff640 1 -- 172.21.15.67:0/657163128 learned_addr learned my addr 172.21.15.67:0/657163128 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:42.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.166+0000 7f61f946e640 1 --2- 172.21.15.67:0/657163128 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f61f41584a0 0x7f61f4150930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:42.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.166+0000 7f61f8c6d640 1 --2- 172.21.15.67:0/657163128 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f61f40b3ed0 0x7f61f414f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:42.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.166+0000 7f61f3fff640 1 -- 172.21.15.67:0/657163128 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f61f41584a0 msgr2=0x7f61f4150930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:42.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.166+0000 7f61f3fff640 1 --2- 172.21.15.67:0/657163128 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f61f41584a0 0x7f61f4150930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:42.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.166+0000 7f61f3fff640 1 -- 172.21.15.67:0/657163128 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f61f40b3ed0 msgr2=0x7f61f414f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:42.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.166+0000 7f61f3fff640 1 --2- 172.21.15.67:0/657163128 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f61f40b3ed0 0x7f61f414f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:42.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.166+0000 7f61f3fff640 1 -- 172.21.15.67:0/657163128 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f414fbb0 con 0x7f61f40d0000 2024-03-24T00:39:42.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.166+0000 7f61f3fff640 1 --2- 172.21.15.67:0/657163128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61f40d0000 0x7f61f4144960 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f61dc000c00 tx=0x7f61dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:42.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.166+0000 7f61f17fa640 1 -- 172.21.15.67:0/657163128 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61dc035040 con 0x7f61f40d0000 2024-03-24T00:39:42.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.166+0000 7f61f17fa640 1 -- 172.21.15.67:0/657163128 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61dc004230 con 0x7f61f40d0000 2024-03-24T00:39:42.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.166+0000 7f61faed1640 1 -- 172.21.15.67:0/657163128 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61f414fe50 con 0x7f61f40d0000 2024-03-24T00:39:42.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.166+0000 7f61f17fa640 1 -- 172.21.15.67:0/657163128 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61dc03a640 con 0x7f61f40d0000 2024-03-24T00:39:42.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.166+0000 7f61faed1640 1 -- 172.21.15.67:0/657163128 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61f41564a0 con 0x7f61f40d0000 2024-03-24T00:39:42.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.167+0000 7f61faed1640 1 -- 172.21.15.67:0/657163128 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61f404f400 con 0x7f61f40d0000 2024-03-24T00:39:42.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.168+0000 7f61f17fa640 1 -- 172.21.15.67:0/657163128 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f61dc038020 con 0x7f61f40d0000 2024-03-24T00:39:42.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.168+0000 7f61f17fa640 1 --2- 172.21.15.67:0/657163128 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f61d406bbd0 0x7f61d406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:42.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.168+0000 7f61f17fa640 1 -- 172.21.15.67:0/657163128 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f61dc0aca30 con 0x7f61f40d0000 2024-03-24T00:39:42.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.168+0000 7f61f8c6d640 1 --2- 172.21.15.67:0/657163128 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f61d406bbd0 0x7f61d406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:42.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.168+0000 7f61f8c6d640 1 --2- 172.21.15.67:0/657163128 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f61d406bbd0 0x7f61d406e090 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f61ec0012e0 tx=0x7f61ec001700 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:42.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.172+0000 7f61f17fa640 1 -- 172.21.15.67:0/657163128 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f61dc07aad0 con 0x7f61f40d0000 2024-03-24T00:39:42.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.352+0000 7f61faed1640 1 -- 172.21.15.67:0/657163128 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f61f406b810 con 0x7f61f40d0000 2024-03-24T00:39:42.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.353+0000 7f61f17fa640 1 -- 172.21.15.67:0/657163128 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f61dc07eb70 con 0x7f61f40d0000 2024-03-24T00:39:42.354 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-24T00:39:42.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.355+0000 7f61faed1640 1 -- 172.21.15.67:0/657163128 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f61d406bbd0 msgr2=0x7f61d406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:42.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.355+0000 7f61faed1640 1 --2- 172.21.15.67:0/657163128 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f61d406bbd0 0x7f61d406e090 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f61ec0012e0 tx=0x7f61ec001700 comp rx=0 tx=0).stop 2024-03-24T00:39:42.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.355+0000 7f61faed1640 1 -- 172.21.15.67:0/657163128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61f40d0000 msgr2=0x7f61f4144960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:42.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.355+0000 7f61faed1640 1 --2- 172.21.15.67:0/657163128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61f40d0000 0x7f61f4144960 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f61dc000c00 tx=0x7f61dc0046c0 comp rx=0 tx=0).stop 2024-03-24T00:39:42.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.355+0000 7f61faed1640 1 -- 172.21.15.67:0/657163128 shutdown_connections 2024-03-24T00:39:42.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.355+0000 7f61faed1640 1 --2- 172.21.15.67:0/657163128 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f61d406bbd0 0x7f61d406e090 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:42.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.355+0000 7f61faed1640 1 --2- 172.21.15.67:0/657163128 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f61f41584a0 0x7f61f4150930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:42.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.355+0000 7f61faed1640 1 --2- 172.21.15.67:0/657163128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61f40d0000 0x7f61f4144960 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:42.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.355+0000 7f61faed1640 1 --2- 172.21.15.67:0/657163128 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f61f40b3ed0 0x7f61f414f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:42.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.355+0000 7f61faed1640 1 -- 172.21.15.67:0/657163128 >> 172.21.15.67:0/657163128 conn(0x7f61f406b040 msgr2=0x7f61f406efc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:42.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.355+0000 7f61faed1640 1 -- 172.21.15.67:0/657163128 shutdown_connections 2024-03-24T00:39:42.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.355+0000 7f61faed1640 1 -- 172.21.15.67:0/657163128 wait complete. 2024-03-24T00:39:42.367 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:39:42.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.471+0000 7fde29d5b640 1 Processor -- start 2024-03-24T00:39:42.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.471+0000 7fde29d5b640 1 -- start start 2024-03-24T00:39:42.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.472+0000 7fde29d5b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fde24099d00 0x7fde2409a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:42.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.472+0000 7fde29d5b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fde240b49b0 0x7fde240d0e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:42.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.472+0000 7fde29d5b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fde240d1360 0x7fde2415d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:42.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.472+0000 7fde29d5b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde240cc1b0 con 0x7fde240d1360 2024-03-24T00:39:42.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.472+0000 7fde29d5b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde2409a620 con 0x7fde240b49b0 2024-03-24T00:39:42.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.472+0000 7fde29d5b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde240cc400 con 0x7fde24099d00 2024-03-24T00:39:42.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.472+0000 7fde23fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fde240d1360 0x7fde2415d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:42.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.472+0000 7fde23fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fde240d1360 0x7fde2415d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50488/0 (socket says 172.21.15.67:50488) 2024-03-24T00:39:42.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.472+0000 7fde23fff640 1 -- 172.21.15.67:0/4253763543 learned_addr learned my addr 172.21.15.67:0/4253763543 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:42.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.472+0000 7fde237fe640 1 --2- 172.21.15.67:0/4253763543 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fde24099d00 0x7fde2409a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:42.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.472+0000 7fde22ffd640 1 --2- 172.21.15.67:0/4253763543 >> v2:172.21.15.72:3300/0 conn(0x7fde240b49b0 0x7fde240d0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:42.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.472+0000 7fde23fff640 1 -- 172.21.15.67:0/4253763543 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fde24099d00 msgr2=0x7fde2409a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:42.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.472+0000 7fde23fff640 1 --2- 172.21.15.67:0/4253763543 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fde24099d00 0x7fde2409a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:42.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.472+0000 7fde23fff640 1 -- 172.21.15.67:0/4253763543 >> v2:172.21.15.72:3300/0 conn(0x7fde240b49b0 msgr2=0x7fde240d0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:42.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.472+0000 7fde23fff640 1 --2- 172.21.15.67:0/4253763543 >> v2:172.21.15.72:3300/0 conn(0x7fde240b49b0 0x7fde240d0e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:42.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.472+0000 7fde23fff640 1 -- 172.21.15.67:0/4253763543 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde240cc6e0 con 0x7fde240d1360 2024-03-24T00:39:42.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.473+0000 7fde23fff640 1 --2- 172.21.15.67:0/4253763543 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde240d1360 0x7fde2415d7d0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fde240736c0 tx=0x7fde1c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93cd9428ec403201 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:42.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.473+0000 7fde227fc640 1 -- 172.21.15.67:0/4253763543 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde1c02ca70 con 0x7fde240d1360 2024-03-24T00:39:42.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.473+0000 7fde227fc640 1 -- 172.21.15.67:0/4253763543 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde1c02cc10 con 0x7fde240d1360 2024-03-24T00:39:42.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.473+0000 7fde227fc640 1 -- 172.21.15.67:0/4253763543 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde1c0343f0 con 0x7fde240d1360 2024-03-24T00:39:42.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.474+0000 7fde29d5b640 1 -- 172.21.15.67:0/4253763543 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde240d1360 msgr2=0x7fde2415d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:42.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.474+0000 7fde29d5b640 1 --2- 172.21.15.67:0/4253763543 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde240d1360 0x7fde2415d7d0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fde240736c0 tx=0x7fde1c02bed0 comp rx=0 tx=0).stop 2024-03-24T00:39:42.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.474+0000 7fde29d5b640 1 -- 172.21.15.67:0/4253763543 shutdown_connections 2024-03-24T00:39:42.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.474+0000 7fde29d5b640 1 --2- 172.21.15.67:0/4253763543 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde240d1360 0x7fde2415d7d0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:42.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.474+0000 7fde29d5b640 1 --2- 172.21.15.67:0/4253763543 >> v2:172.21.15.72:3300/0 conn(0x7fde240b49b0 0x7fde240d0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:42.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.474+0000 7fde29d5b640 1 --2- 172.21.15.67:0/4253763543 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fde24099d00 0x7fde2409a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:42.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.474+0000 7fde29d5b640 1 -- 172.21.15.67:0/4253763543 >> 172.21.15.67:0/4253763543 conn(0x7fde240b3650 msgr2=0x7fde240b3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:42.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.474+0000 7fde29d5b640 1 -- 172.21.15.67:0/4253763543 shutdown_connections 2024-03-24T00:39:42.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.474+0000 7fde29d5b640 1 -- 172.21.15.67:0/4253763543 wait complete. 2024-03-24T00:39:42.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.475+0000 7fde29d5b640 1 Processor -- start 2024-03-24T00:39:42.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.475+0000 7fde29d5b640 1 -- start start 2024-03-24T00:39:42.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.476+0000 7fde29d5b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde24099d00 0x7fde241f73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:42.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.476+0000 7fde29d5b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fde240b49b0 0x7fde240cb120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:42.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.476+0000 7fde29d5b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde240d1360 0x7fde240cb660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:42.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.476+0000 7fde29d5b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde240d01c0 con 0x7fde24099d00 2024-03-24T00:39:42.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.476+0000 7fde29d5b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde240ccca0 con 0x7fde240d1360 2024-03-24T00:39:42.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.476+0000 7fde29d5b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde240b32a0 con 0x7fde240b49b0 2024-03-24T00:39:42.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.476+0000 7fde237fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde24099d00 0x7fde241f73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:42.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.476+0000 7fde237fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde24099d00 0x7fde241f73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50498/0 (socket says 172.21.15.67:50498) 2024-03-24T00:39:42.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.476+0000 7fde237fe640 1 -- 172.21.15.67:0/2540870680 learned_addr learned my addr 172.21.15.67:0/2540870680 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:42.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.476+0000 7fde22ffd640 1 --2- 172.21.15.67:0/2540870680 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fde240b49b0 0x7fde240cb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:42.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.476+0000 7fde23fff640 1 --2- 172.21.15.67:0/2540870680 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde240d1360 0x7fde240cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:42.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.476+0000 7fde237fe640 1 -- 172.21.15.67:0/2540870680 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fde240b49b0 msgr2=0x7fde240cb120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:42.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.476+0000 7fde237fe640 1 --2- 172.21.15.67:0/2540870680 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fde240b49b0 0x7fde240cb120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:42.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.476+0000 7fde237fe640 1 -- 172.21.15.67:0/2540870680 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde240d1360 msgr2=0x7fde240cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:42.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.476+0000 7fde237fe640 1 --2- 172.21.15.67:0/2540870680 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde240d1360 0x7fde240cb660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:42.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.476+0000 7fde237fe640 1 -- 172.21.15.67:0/2540870680 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde24157ef0 con 0x7fde24099d00 2024-03-24T00:39:42.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.476+0000 7fde237fe640 1 --2- 172.21.15.67:0/2540870680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde24099d00 0x7fde241f73a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fde1400e890 tx=0x7fde1400ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:42.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.477+0000 7fde0bfff640 1 -- 172.21.15.67:0/2540870680 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde14017070 con 0x7fde24099d00 2024-03-24T00:39:42.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.477+0000 7fde0bfff640 1 -- 172.21.15.67:0/2540870680 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde1400c940 con 0x7fde24099d00 2024-03-24T00:39:42.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.477+0000 7fde0bfff640 1 -- 172.21.15.67:0/2540870680 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde1400cd10 con 0x7fde24099d00 2024-03-24T00:39:42.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.477+0000 7fde29d5b640 1 -- 172.21.15.67:0/2540870680 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde241581d0 con 0x7fde24099d00 2024-03-24T00:39:42.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.477+0000 7fde29d5b640 1 -- 172.21.15.67:0/2540870680 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde24158790 con 0x7fde24099d00 2024-03-24T00:39:42.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.477+0000 7fde29d5b640 1 -- 172.21.15.67:0/2540870680 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde2404f400 con 0x7fde24099d00 2024-03-24T00:39:42.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.478+0000 7fde0bfff640 1 -- 172.21.15.67:0/2540870680 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fde1401a070 con 0x7fde24099d00 2024-03-24T00:39:42.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.478+0000 7fde0bfff640 1 --2- 172.21.15.67:0/2540870680 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fde1006bc00 0x7fde1006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:42.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.478+0000 7fde0bfff640 1 -- 172.21.15.67:0/2540870680 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fde1408b130 con 0x7fde24099d00 2024-03-24T00:39:42.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.478+0000 7fde22ffd640 1 --2- 172.21.15.67:0/2540870680 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fde1006bc00 0x7fde1006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:42.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.479+0000 7fde22ffd640 1 --2- 172.21.15.67:0/2540870680 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fde1006bc00 0x7fde1006e0c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fde241610c0 tx=0x7fde18009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:42.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.483+0000 7fde0bfff640 1 -- 172.21.15.67:0/2540870680 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fde140591d0 con 0x7fde24099d00 2024-03-24T00:39:42.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.663+0000 7fde29d5b640 1 -- 172.21.15.67:0/2540870680 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fde240cbe70 con 0x7fde24099d00 2024-03-24T00:39:42.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.663+0000 7fde0bfff640 1 -- 172.21.15.67:0/2540870680 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7fde1405d270 con 0x7fde24099d00 2024-03-24T00:39:42.664 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-24T00:39:42.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.665+0000 7fde29d5b640 1 -- 172.21.15.67:0/2540870680 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fde1006bc00 msgr2=0x7fde1006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:42.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.665+0000 7fde29d5b640 1 --2- 172.21.15.67:0/2540870680 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fde1006bc00 0x7fde1006e0c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fde241610c0 tx=0x7fde18009190 comp rx=0 tx=0).stop 2024-03-24T00:39:42.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.665+0000 7fde29d5b640 1 -- 172.21.15.67:0/2540870680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde24099d00 msgr2=0x7fde241f73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:42.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.665+0000 7fde29d5b640 1 --2- 172.21.15.67:0/2540870680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde24099d00 0x7fde241f73a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fde1400e890 tx=0x7fde1400ed50 comp rx=0 tx=0).stop 2024-03-24T00:39:42.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.665+0000 7fde29d5b640 1 -- 172.21.15.67:0/2540870680 shutdown_connections 2024-03-24T00:39:42.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.665+0000 7fde29d5b640 1 --2- 172.21.15.67:0/2540870680 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fde1006bc00 0x7fde1006e0c0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:42.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.665+0000 7fde29d5b640 1 --2- 172.21.15.67:0/2540870680 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde240d1360 0x7fde240cb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:42.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.665+0000 7fde29d5b640 1 --2- 172.21.15.67:0/2540870680 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fde240b49b0 0x7fde240cb120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:42.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.665+0000 7fde29d5b640 1 --2- 172.21.15.67:0/2540870680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde24099d00 0x7fde241f73a0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:42.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.665+0000 7fde29d5b640 1 -- 172.21.15.67:0/2540870680 >> 172.21.15.67:0/2540870680 conn(0x7fde240b3650 msgr2=0x7fde24071320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:42.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.665+0000 7fde29d5b640 1 -- 172.21.15.67:0/2540870680 shutdown_connections 2024-03-24T00:39:42.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.665+0000 7fde29d5b640 1 -- 172.21.15.67:0/2540870680 wait complete. 2024-03-24T00:39:42.677 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:39:42.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.780+0000 7f497eaac640 1 Processor -- start 2024-03-24T00:39:42.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.780+0000 7f497eaac640 1 -- start start 2024-03-24T00:39:42.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.781+0000 7f497eaac640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4978161370 0x7f4978161ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:42.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.781+0000 7f497eaac640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f49780cf100 0x7f49780d2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:42.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.781+0000 7f497eaac640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4978161ff0 con 0x7f49780b9690 2024-03-24T00:39:42.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.781+0000 7f497eaac640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49780b6e10 con 0x7f4978161370 2024-03-24T00:39:42.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.781+0000 7f497eaac640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49780d2190 con 0x7f49780cf100 2024-03-24T00:39:42.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.781+0000 7f497c848640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4978161370 0x7f4978161ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:42.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.781+0000 7f497c848640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4978161370 0x7f4978161ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50510/0 (socket says 172.21.15.67:50510) 2024-03-24T00:39:42.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.781+0000 7f497c848640 1 -- 172.21.15.67:0/361819306 learned_addr learned my addr 172.21.15.67:0/361819306 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:42.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.781+0000 7f497d049640 1 --2- 172.21.15.67:0/361819306 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f49780cf100 0x7f49780d2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:42.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.781+0000 7f497c848640 1 -- 172.21.15.67:0/361819306 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f49780cf100 msgr2=0x7f49780d2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:42.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.781+0000 7f497c848640 1 --2- 172.21.15.67:0/361819306 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f49780cf100 0x7f49780d2620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:42.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.781+0000 7f497c848640 1 -- 172.21.15.67:0/361819306 >> v1:172.21.15.72:6789/0 conn(0x7f49780b9690 legacy=0x7f49780ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:42.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.781+0000 7f49777fe640 1 -- 172.21.15.67:0/361819306 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 789297371 0 0) 0x7f4978161ff0 con 0x7f49780b9690 2024-03-24T00:39:42.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.781+0000 7f497c848640 1 -- 172.21.15.67:0/361819306 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4978162290 con 0x7f4978161370 2024-03-24T00:39:42.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.781+0000 7f497d049640 1 --2- 172.21.15.67:0/361819306 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f49780cf100 0x7f49780d2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:42.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.782+0000 7f497c848640 1 --2- 172.21.15.67:0/361819306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4978161370 0x7f4978161ab0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f4970009e50 tx=0x7f497002bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ad0b5e5cd6b0cb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:42.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.782+0000 7f49777fe640 1 -- 172.21.15.67:0/361819306 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f497002cb90 con 0x7f4978161370 2024-03-24T00:39:42.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.782+0000 7f49777fe640 1 -- 172.21.15.67:0/361819306 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f497002cd30 con 0x7f4978161370 2024-03-24T00:39:42.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.782+0000 7f49777fe640 1 -- 172.21.15.67:0/361819306 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4970034610 con 0x7f4978161370 2024-03-24T00:39:42.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.783+0000 7f497eaac640 1 -- 172.21.15.67:0/361819306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4978161370 msgr2=0x7f4978161ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:42.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.783+0000 7f497eaac640 1 --2- 172.21.15.67:0/361819306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4978161370 0x7f4978161ab0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f4970009e50 tx=0x7f497002bff0 comp rx=0 tx=0).stop 2024-03-24T00:39:42.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.783+0000 7f497eaac640 1 -- 172.21.15.67:0/361819306 shutdown_connections 2024-03-24T00:39:42.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.783+0000 7f497eaac640 1 --2- 172.21.15.67:0/361819306 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f49780cf100 0x7f49780d2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:42.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.783+0000 7f497eaac640 1 --2- 172.21.15.67:0/361819306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4978161370 0x7f4978161ab0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:42.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.783+0000 7f497eaac640 1 -- 172.21.15.67:0/361819306 >> 172.21.15.67:0/361819306 conn(0x7f497806c880 msgr2=0x7f497806cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:42.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.783+0000 7f497eaac640 1 -- 172.21.15.67:0/361819306 shutdown_connections 2024-03-24T00:39:42.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.783+0000 7f497eaac640 1 -- 172.21.15.67:0/361819306 wait complete. 2024-03-24T00:39:42.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.784+0000 7f497eaac640 1 Processor -- start 2024-03-24T00:39:42.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.784+0000 7f497eaac640 1 -- start start 2024-03-24T00:39:42.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.785+0000 7f497eaac640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f49780b9690 0x7f49780cd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:42.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.785+0000 7f497eaac640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f49780cf100 0x7f4978155a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:42.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.785+0000 7f497eaac640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4978161370 0x7f49781f7c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:42.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.785+0000 7f497eaac640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4978161750 con 0x7f4978161370 2024-03-24T00:39:42.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.785+0000 7f497eaac640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4978162850 con 0x7f49780b9690 2024-03-24T00:39:42.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.785+0000 7f497eaac640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49780d2b60 con 0x7f49780cf100 2024-03-24T00:39:42.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.785+0000 7f497d049640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4978161370 0x7f49781f7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:42.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.785+0000 7f497d049640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4978161370 0x7f49781f7c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50520/0 (socket says 172.21.15.67:50520) 2024-03-24T00:39:42.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.785+0000 7f497d049640 1 -- 172.21.15.67:0/1520501776 learned_addr learned my addr 172.21.15.67:0/1520501776 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:42.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.785+0000 7f4977fff640 1 --2- 172.21.15.67:0/1520501776 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f49780cf100 0x7f4978155a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:42.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.785+0000 7f497c848640 1 --2- 172.21.15.67:0/1520501776 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f49780b9690 0x7f49780cd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:42.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.785+0000 7f497d049640 1 -- 172.21.15.67:0/1520501776 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f49780cf100 msgr2=0x7f4978155a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:42.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.785+0000 7f497d049640 1 --2- 172.21.15.67:0/1520501776 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f49780cf100 0x7f4978155a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:42.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.785+0000 7f497d049640 1 -- 172.21.15.67:0/1520501776 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f49780b9690 msgr2=0x7f49780cd6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:42.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.785+0000 7f497d049640 1 --2- 172.21.15.67:0/1520501776 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f49780b9690 0x7f49780cd6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:42.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.785+0000 7f497d049640 1 -- 172.21.15.67:0/1520501776 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4978155fb0 con 0x7f4978161370 2024-03-24T00:39:42.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.786+0000 7f497d049640 1 --2- 172.21.15.67:0/1520501776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4978161370 0x7f49781f7c60 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f496800e8c0 tx=0x7f496800ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:42.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.786+0000 7f49757fa640 1 -- 172.21.15.67:0/1520501776 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4968017070 con 0x7f4978161370 2024-03-24T00:39:42.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.786+0000 7f49757fa640 1 -- 172.21.15.67:0/1520501776 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4968004020 con 0x7f4978161370 2024-03-24T00:39:42.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.786+0000 7f497eaac640 1 -- 172.21.15.67:0/1520501776 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4978152ec0 con 0x7f4978161370 2024-03-24T00:39:42.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.786+0000 7f49757fa640 1 -- 172.21.15.67:0/1520501776 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49680043f0 con 0x7f4978161370 2024-03-24T00:39:42.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.786+0000 7f497eaac640 1 -- 172.21.15.67:0/1520501776 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4978155550 con 0x7f4978161370 2024-03-24T00:39:42.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.787+0000 7f497eaac640 1 -- 172.21.15.67:0/1520501776 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f497804f400 con 0x7f4978161370 2024-03-24T00:39:42.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.787+0000 7f49757fa640 1 -- 172.21.15.67:0/1520501776 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f496800c8e0 con 0x7f4978161370 2024-03-24T00:39:42.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.787+0000 7f49757fa640 1 --2- 172.21.15.67:0/1520501776 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f496406bc00 0x7f496406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:42.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.788+0000 7f49757fa640 1 -- 172.21.15.67:0/1520501776 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f496808ab20 con 0x7f4978161370 2024-03-24T00:39:42.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.788+0000 7f497c848640 1 --2- 172.21.15.67:0/1520501776 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f496406bc00 0x7f496406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:42.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.788+0000 7f497c848640 1 --2- 172.21.15.67:0/1520501776 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f496406bc00 0x7f496406e0c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4970035000 tx=0x7f4970030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:42.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.792+0000 7f49757fa640 1 -- 172.21.15.67:0/1520501776 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4968058bc0 con 0x7f4978161370 2024-03-24T00:39:42.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.972+0000 7f497eaac640 1 -- 172.21.15.67:0/1520501776 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f49780ca680 con 0x7f4978161370 2024-03-24T00:39:42.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.973+0000 7f49757fa640 1 -- 172.21.15.67:0/1520501776 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f496805cc60 con 0x7f4978161370 2024-03-24T00:39:42.974 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-03-24T00:39:42.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.975+0000 7f497eaac640 1 -- 172.21.15.67:0/1520501776 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f496406bc00 msgr2=0x7f496406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:42.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.975+0000 7f497eaac640 1 --2- 172.21.15.67:0/1520501776 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f496406bc00 0x7f496406e0c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4970035000 tx=0x7f4970030000 comp rx=0 tx=0).stop 2024-03-24T00:39:42.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.975+0000 7f497eaac640 1 -- 172.21.15.67:0/1520501776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4978161370 msgr2=0x7f49781f7c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:42.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.975+0000 7f497eaac640 1 --2- 172.21.15.67:0/1520501776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4978161370 0x7f49781f7c60 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f496800e8c0 tx=0x7f496800ed80 comp rx=0 tx=0).stop 2024-03-24T00:39:42.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.975+0000 7f497eaac640 1 -- 172.21.15.67:0/1520501776 shutdown_connections 2024-03-24T00:39:42.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.975+0000 7f497eaac640 1 --2- 172.21.15.67:0/1520501776 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f496406bc00 0x7f496406e0c0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:42.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.975+0000 7f497eaac640 1 --2- 172.21.15.67:0/1520501776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4978161370 0x7f49781f7c60 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:42.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.975+0000 7f497eaac640 1 --2- 172.21.15.67:0/1520501776 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f49780cf100 0x7f4978155a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:42.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.975+0000 7f497eaac640 1 --2- 172.21.15.67:0/1520501776 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f49780b9690 0x7f49780cd6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:42.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.975+0000 7f497eaac640 1 -- 172.21.15.67:0/1520501776 >> 172.21.15.67:0/1520501776 conn(0x7f497806c880 msgr2=0x7f497815c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:42.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.975+0000 7f497eaac640 1 -- 172.21.15.67:0/1520501776 shutdown_connections 2024-03-24T00:39:42.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:42.975+0000 7f497eaac640 1 -- 172.21.15.67:0/1520501776 wait complete. 2024-03-24T00:39:42.986 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T00:39:43.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.090+0000 7f4999ab2640 1 Processor -- start 2024-03-24T00:39:43.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.090+0000 7f4999ab2640 1 -- start start 2024-03-24T00:39:43.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.091+0000 7f4999ab2640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f49940b3d30 0x7f4994168e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:43.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.091+0000 7f4999ab2640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49940cf090 con 0x7f49940b41e0 2024-03-24T00:39:43.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.091+0000 7f4999ab2640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49940d0320 con 0x7f499415dfd0 2024-03-24T00:39:43.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.091+0000 7f4999ab2640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49940cc260 con 0x7f49940b3d30 2024-03-24T00:39:43.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.091+0000 7f4993fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f49940b3d30 0x7f4994168e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:43.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.091+0000 7f4992ffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f49940b41e0 0x7f49940b3620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41572/0 (socket says 172.21.15.67:41572) 2024-03-24T00:39:43.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.091+0000 7f4992ffd640 1 -- 172.21.15.67:0/3143476593 learned_addr learned my addr 172.21.15.67:0/3143476593 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:43.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.091+0000 7f49927fc640 1 -- 172.21.15.67:0/3143476593 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1354821295 0 0) 0x7f49940d0320 con 0x7f499415dfd0 2024-03-24T00:39:43.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.091+0000 7f4993fff640 1 -- 172.21.15.67:0/3143476593 >> v1:172.21.15.72:6789/0 conn(0x7f499415dfd0 legacy=0x7f49940cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:43.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.091+0000 7f4993fff640 1 -- 172.21.15.67:0/3143476593 >> v1:172.21.15.67:6789/0 conn(0x7f49940b41e0 legacy=0x7f49940b3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:43.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.091+0000 7f4993fff640 1 -- 172.21.15.67:0/3143476593 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49940d0600 con 0x7f49940b3d30 2024-03-24T00:39:43.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.092+0000 7f4993fff640 1 --2- 172.21.15.67:0/3143476593 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49940b3d30 0x7f4994168e40 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f498c009dc0 tx=0x7f498c02be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6552dc2cb0dd22e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:43.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.092+0000 7f49927fc640 1 -- 172.21.15.67:0/3143476593 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498c039070 con 0x7f49940b3d30 2024-03-24T00:39:43.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.092+0000 7f49927fc640 1 -- 172.21.15.67:0/3143476593 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f498c02cb30 con 0x7f49940b3d30 2024-03-24T00:39:43.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.092+0000 7f49927fc640 1 -- 172.21.15.67:0/3143476593 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498c034440 con 0x7f49940b3d30 2024-03-24T00:39:43.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.093+0000 7f4999ab2640 1 -- 172.21.15.67:0/3143476593 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49940b3d30 msgr2=0x7f4994168e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:43.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.093+0000 7f4999ab2640 1 --2- 172.21.15.67:0/3143476593 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49940b3d30 0x7f4994168e40 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f498c009dc0 tx=0x7f498c02be00 comp rx=0 tx=0).stop 2024-03-24T00:39:43.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.093+0000 7f4999ab2640 1 -- 172.21.15.67:0/3143476593 shutdown_connections 2024-03-24T00:39:43.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.093+0000 7f4999ab2640 1 --2- 172.21.15.67:0/3143476593 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49940b3d30 0x7f4994168e40 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:43.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.093+0000 7f4999ab2640 1 -- 172.21.15.67:0/3143476593 >> 172.21.15.67:0/3143476593 conn(0x7f499406fcf0 msgr2=0x7f49940700d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:43.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.093+0000 7f4999ab2640 1 -- 172.21.15.67:0/3143476593 shutdown_connections 2024-03-24T00:39:43.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.093+0000 7f4999ab2640 1 -- 172.21.15.67:0/3143476593 wait complete. 2024-03-24T00:39:43.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.094+0000 7f4999ab2640 1 Processor -- start 2024-03-24T00:39:43.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.094+0000 7f4999ab2640 1 -- start start 2024-03-24T00:39:43.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.095+0000 7f4999ab2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f49940b3d30 0x7f49941580a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:43.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.095+0000 7f4999ab2640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f49940b41e0 0x7f499415ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:43.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.095+0000 7f4999ab2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f499415dfd0 0x7f4994158c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:43.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.095+0000 7f4999ab2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f499415e3b0 con 0x7f499415dfd0 2024-03-24T00:39:43.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.095+0000 7f4999ab2640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49940cedb0 con 0x7f49940b3d30 2024-03-24T00:39:43.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.095+0000 7f4999ab2640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49940cd6e0 con 0x7f49940b41e0 2024-03-24T00:39:43.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.095+0000 7f4993fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f499415dfd0 0x7f4994158c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:43.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.095+0000 7f4993fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f499415dfd0 0x7f4994158c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50532/0 (socket says 172.21.15.67:50532) 2024-03-24T00:39:43.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.095+0000 7f4993fff640 1 -- 172.21.15.67:0/3290439945 learned_addr learned my addr 172.21.15.67:0/3290439945 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:43.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.095+0000 7f4992ffd640 1 --2- 172.21.15.67:0/3290439945 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f49940b41e0 0x7f499415ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:43.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.095+0000 7f49937fe640 1 --2- 172.21.15.67:0/3290439945 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f49940b3d30 0x7f49941580a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:43.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.095+0000 7f4993fff640 1 -- 172.21.15.67:0/3290439945 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f49940b41e0 msgr2=0x7f499415ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:43.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.095+0000 7f4993fff640 1 --2- 172.21.15.67:0/3290439945 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f49940b41e0 0x7f499415ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:43.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.095+0000 7f4993fff640 1 -- 172.21.15.67:0/3290439945 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f49940b3d30 msgr2=0x7f49941580a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:43.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.095+0000 7f4993fff640 1 --2- 172.21.15.67:0/3290439945 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f49940b3d30 0x7f49941580a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:43.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.095+0000 7f4993fff640 1 -- 172.21.15.67:0/3290439945 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49940c9e10 con 0x7f499415dfd0 2024-03-24T00:39:43.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.095+0000 7f4993fff640 1 --2- 172.21.15.67:0/3290439945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f499415dfd0 0x7f4994158c80 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f498c009dc0 tx=0x7f498c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:43.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.096+0000 7f497bfff640 1 -- 172.21.15.67:0/3290439945 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498c044020 con 0x7f499415dfd0 2024-03-24T00:39:43.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.096+0000 7f497bfff640 1 -- 172.21.15.67:0/3290439945 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f498c03f470 con 0x7f499415dfd0 2024-03-24T00:39:43.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.096+0000 7f4999ab2640 1 -- 172.21.15.67:0/3290439945 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49940c9850 con 0x7f499415dfd0 2024-03-24T00:39:43.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.096+0000 7f497bfff640 1 -- 172.21.15.67:0/3290439945 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498c039040 con 0x7f499415dfd0 2024-03-24T00:39:43.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.096+0000 7f4999ab2640 1 -- 172.21.15.67:0/3290439945 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49940ca1e0 con 0x7f499415dfd0 2024-03-24T00:39:43.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.097+0000 7f4999ab2640 1 -- 172.21.15.67:0/3290439945 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49940d1530 con 0x7f499415dfd0 2024-03-24T00:39:43.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.097+0000 7f497bfff640 1 -- 172.21.15.67:0/3290439945 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f498c047020 con 0x7f499415dfd0 2024-03-24T00:39:43.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.097+0000 7f497bfff640 1 --2- 172.21.15.67:0/3290439945 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f498006bbd0 0x7f498006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:43.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.098+0000 7f497bfff640 1 -- 172.21.15.67:0/3290439945 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f498c0ad900 con 0x7f499415dfd0 2024-03-24T00:39:43.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.098+0000 7f49937fe640 1 --2- 172.21.15.67:0/3290439945 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f498006bbd0 0x7f498006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:43.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.098+0000 7f49937fe640 1 --2- 172.21.15.67:0/3290439945 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f498006bbd0 0x7f498006e090 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4984000f60 tx=0x7f4984009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:43.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.102+0000 7f497bfff640 1 -- 172.21.15.67:0/3290439945 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f49940d1530 con 0x7f499415dfd0 2024-03-24T00:39:43.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.282+0000 7f4999ab2640 1 -- 172.21.15.67:0/3290439945 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f49940c9fe0 con 0x7f499415dfd0 2024-03-24T00:39:43.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.283+0000 7f497bfff640 1 -- 172.21.15.67:0/3290439945 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f498c04e030 con 0x7f499415dfd0 2024-03-24T00:39:43.284 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-24T00:39:43.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.285+0000 7f4999ab2640 1 -- 172.21.15.67:0/3290439945 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f498006bbd0 msgr2=0x7f498006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:43.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.285+0000 7f4999ab2640 1 --2- 172.21.15.67:0/3290439945 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f498006bbd0 0x7f498006e090 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4984000f60 tx=0x7f4984009040 comp rx=0 tx=0).stop 2024-03-24T00:39:43.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.285+0000 7f4999ab2640 1 -- 172.21.15.67:0/3290439945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f499415dfd0 msgr2=0x7f4994158c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:43.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.285+0000 7f4999ab2640 1 --2- 172.21.15.67:0/3290439945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f499415dfd0 0x7f4994158c80 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f498c009dc0 tx=0x7f498c004a40 comp rx=0 tx=0).stop 2024-03-24T00:39:43.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.285+0000 7f4999ab2640 1 -- 172.21.15.67:0/3290439945 shutdown_connections 2024-03-24T00:39:43.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.285+0000 7f4999ab2640 1 --2- 172.21.15.67:0/3290439945 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f498006bbd0 0x7f498006e090 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:43.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.285+0000 7f4999ab2640 1 --2- 172.21.15.67:0/3290439945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f499415dfd0 0x7f4994158c80 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:43.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.285+0000 7f4999ab2640 1 --2- 172.21.15.67:0/3290439945 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f49940b41e0 0x7f499415ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:43.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.285+0000 7f4999ab2640 1 --2- 172.21.15.67:0/3290439945 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f49940b3d30 0x7f49941580a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:43.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.285+0000 7f4999ab2640 1 -- 172.21.15.67:0/3290439945 >> 172.21.15.67:0/3290439945 conn(0x7f499406fcf0 msgr2=0x7f49941606f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:43.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.285+0000 7f4999ab2640 1 -- 172.21.15.67:0/3290439945 shutdown_connections 2024-03-24T00:39:43.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.285+0000 7f4999ab2640 1 -- 172.21.15.67:0/3290439945 wait complete. 2024-03-24T00:39:43.297 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T00:39:43.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.399+0000 7f60f453b640 1 Processor -- start 2024-03-24T00:39:43.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.399+0000 7f60f453b640 1 -- start start 2024-03-24T00:39:43.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.400+0000 7f60f453b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f60ec0cd630 0x7f60ec0d06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:43.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.400+0000 7f60f453b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60ec0d0d10 con 0x7f60ec06cb60 2024-03-24T00:39:43.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.400+0000 7f60f453b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60ec0d0ff0 con 0x7f60ec15e250 2024-03-24T00:39:43.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.400+0000 7f60f453b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60ec0d12d0 con 0x7f60ec0cd630 2024-03-24T00:39:43.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.400+0000 7f60f22d7640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f60ec06cb60 0x7f60ec06cf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41574/0 (socket says 172.21.15.67:41574) 2024-03-24T00:39:43.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.400+0000 7f60f22d7640 1 -- 172.21.15.67:0/1381534008 learned_addr learned my addr 172.21.15.67:0/1381534008 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:43.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.400+0000 7f60f2ad8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f60ec0cd630 0x7f60ec0d06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:43.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.400+0000 7f60f12d5640 1 -- 172.21.15.67:0/1381534008 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 789442929 0 0) 0x7f60ec0d0ff0 con 0x7f60ec15e250 2024-03-24T00:39:43.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.400+0000 7f60f12d5640 1 -- 172.21.15.67:0/1381534008 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60d8003520 con 0x7f60ec15e250 2024-03-24T00:39:43.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.400+0000 7f60f2ad8640 1 -- 172.21.15.67:0/1381534008 >> v1:172.21.15.72:6789/0 conn(0x7f60ec15e250 legacy=0x7f60ec0cffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:43.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.400+0000 7f60f12d5640 1 -- 172.21.15.67:0/1381534008 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3523120395 0 0) 0x7f60ec0d0d10 con 0x7f60ec06cb60 2024-03-24T00:39:43.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.400+0000 7f60f2ad8640 1 -- 172.21.15.67:0/1381534008 >> v1:172.21.15.67:6789/0 conn(0x7f60ec06cb60 legacy=0x7f60ec06cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:43.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.400+0000 7f60f2ad8640 1 -- 172.21.15.67:0/1381534008 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60ec0d15b0 con 0x7f60ec0cd630 2024-03-24T00:39:43.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.401+0000 7f60f2ad8640 1 --2- 172.21.15.67:0/1381534008 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60ec0cd630 0x7f60ec0d06b0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f60e4009d90 tx=0x7f60e402bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c47e654d7a86f02 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:43.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.401+0000 7f60f12d5640 1 -- 172.21.15.67:0/1381534008 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60e4039070 con 0x7f60ec0cd630 2024-03-24T00:39:43.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.401+0000 7f60f12d5640 1 -- 172.21.15.67:0/1381534008 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60e402cd80 con 0x7f60ec0cd630 2024-03-24T00:39:43.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.401+0000 7f60f12d5640 1 -- 172.21.15.67:0/1381534008 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60e4034660 con 0x7f60ec0cd630 2024-03-24T00:39:43.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.402+0000 7f60f453b640 1 -- 172.21.15.67:0/1381534008 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60ec0cd630 msgr2=0x7f60ec0d06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:43.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.402+0000 7f60f453b640 1 --2- 172.21.15.67:0/1381534008 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60ec0cd630 0x7f60ec0d06b0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f60e4009d90 tx=0x7f60e402bfd0 comp rx=0 tx=0).stop 2024-03-24T00:39:43.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.402+0000 7f60f453b640 1 -- 172.21.15.67:0/1381534008 shutdown_connections 2024-03-24T00:39:43.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.402+0000 7f60f453b640 1 --2- 172.21.15.67:0/1381534008 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60ec0cd630 0x7f60ec0d06b0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:43.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.402+0000 7f60f453b640 1 -- 172.21.15.67:0/1381534008 >> 172.21.15.67:0/1381534008 conn(0x7f60ec0d1bf0 msgr2=0x7f60ec0d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:43.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.402+0000 7f60f453b640 1 -- 172.21.15.67:0/1381534008 shutdown_connections 2024-03-24T00:39:43.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.402+0000 7f60f453b640 1 -- 172.21.15.67:0/1381534008 wait complete. 2024-03-24T00:39:43.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.403+0000 7f60f453b640 1 Processor -- start 2024-03-24T00:39:43.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.403+0000 7f60f453b640 1 -- start start 2024-03-24T00:39:43.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.404+0000 7f60f453b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60ec06cb60 0x7f60ec15e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:43.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.404+0000 7f60f453b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60ec0cd630 0x7f60ec154590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:43.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.404+0000 7f60f453b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f60ec15e250 0x7f60ec154ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:43.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.404+0000 7f60f453b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60ec168f30 con 0x7f60ec06cb60 2024-03-24T00:39:43.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.404+0000 7f60f453b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60ec168db0 con 0x7f60ec0cd630 2024-03-24T00:39:43.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.404+0000 7f60f453b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60ec1690b0 con 0x7f60ec15e250 2024-03-24T00:39:43.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.404+0000 7f60f22d7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60ec06cb60 0x7f60ec15e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:43.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.404+0000 7f60f22d7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60ec06cb60 0x7f60ec15e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50544/0 (socket says 172.21.15.67:50544) 2024-03-24T00:39:43.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.404+0000 7f60f22d7640 1 -- 172.21.15.67:0/1513124892 learned_addr learned my addr 172.21.15.67:0/1513124892 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:43.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.404+0000 7f60f2ad8640 1 --2- 172.21.15.67:0/1513124892 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f60ec15e250 0x7f60ec154ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:43.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.404+0000 7f60f1ad6640 1 --2- 172.21.15.67:0/1513124892 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60ec0cd630 0x7f60ec154590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:43.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.404+0000 7f60f22d7640 1 -- 172.21.15.67:0/1513124892 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f60ec15e250 msgr2=0x7f60ec154ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:43.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.404+0000 7f60f22d7640 1 --2- 172.21.15.67:0/1513124892 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f60ec15e250 0x7f60ec154ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:43.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.404+0000 7f60f22d7640 1 -- 172.21.15.67:0/1513124892 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60ec0cd630 msgr2=0x7f60ec154590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:43.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.404+0000 7f60f22d7640 1 --2- 172.21.15.67:0/1513124892 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60ec0cd630 0x7f60ec154590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:43.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.404+0000 7f60f22d7640 1 -- 172.21.15.67:0/1513124892 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60ec15eba0 con 0x7f60ec06cb60 2024-03-24T00:39:43.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.404+0000 7f60f22d7640 1 --2- 172.21.15.67:0/1513124892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60ec06cb60 0x7f60ec15e660 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f60e800c4d0 tx=0x7f60e800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:43.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.404+0000 7f60e2ffd640 1 -- 172.21.15.67:0/1513124892 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60e8018070 con 0x7f60ec06cb60 2024-03-24T00:39:43.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.404+0000 7f60e2ffd640 1 -- 172.21.15.67:0/1513124892 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60e80074e0 con 0x7f60ec06cb60 2024-03-24T00:39:43.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.404+0000 7f60f453b640 1 -- 172.21.15.67:0/1513124892 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60ec0c5800 con 0x7f60ec06cb60 2024-03-24T00:39:43.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.404+0000 7f60e2ffd640 1 -- 172.21.15.67:0/1513124892 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60e8004a20 con 0x7f60ec06cb60 2024-03-24T00:39:43.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.404+0000 7f60f453b640 1 -- 172.21.15.67:0/1513124892 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60ec160ce0 con 0x7f60ec06cb60 2024-03-24T00:39:43.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.405+0000 7f60f453b640 1 -- 172.21.15.67:0/1513124892 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60ec04f400 con 0x7f60ec06cb60 2024-03-24T00:39:43.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.406+0000 7f60e2ffd640 1 -- 172.21.15.67:0/1513124892 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f60e801b070 con 0x7f60ec06cb60 2024-03-24T00:39:43.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.406+0000 7f60e2ffd640 1 --2- 172.21.15.67:0/1513124892 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f60d006bc00 0x7f60d006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:43.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.406+0000 7f60e2ffd640 1 -- 172.21.15.67:0/1513124892 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f60e808a030 con 0x7f60ec06cb60 2024-03-24T00:39:43.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.406+0000 7f60f1ad6640 1 --2- 172.21.15.67:0/1513124892 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f60d006bc00 0x7f60d006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:43.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.407+0000 7f60f1ad6640 1 --2- 172.21.15.67:0/1513124892 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f60d006bc00 0x7f60d006e0c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f60ec15ca50 tx=0x7f60dc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:43.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.410+0000 7f60e2ffd640 1 -- 172.21.15.67:0/1513124892 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f60e80580d0 con 0x7f60ec06cb60 2024-03-24T00:39:43.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.605+0000 7f60f453b640 1 -- 172.21.15.67:0/1513124892 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f60ec06c940 con 0x7f60ec06cb60 2024-03-24T00:39:43.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.606+0000 7f60e2ffd640 1 -- 172.21.15.67:0/1513124892 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7f60e805c170 con 0x7f60ec06cb60 2024-03-24T00:39:43.606 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-03-24T00:39:43.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.607+0000 7f60f453b640 1 -- 172.21.15.67:0/1513124892 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f60d006bc00 msgr2=0x7f60d006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:43.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.607+0000 7f60f453b640 1 --2- 172.21.15.67:0/1513124892 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f60d006bc00 0x7f60d006e0c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f60ec15ca50 tx=0x7f60dc009040 comp rx=0 tx=0).stop 2024-03-24T00:39:43.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.607+0000 7f60f453b640 1 -- 172.21.15.67:0/1513124892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60ec06cb60 msgr2=0x7f60ec15e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:43.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.607+0000 7f60f453b640 1 --2- 172.21.15.67:0/1513124892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60ec06cb60 0x7f60ec15e660 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f60e800c4d0 tx=0x7f60e800c990 comp rx=0 tx=0).stop 2024-03-24T00:39:43.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.607+0000 7f60f453b640 1 -- 172.21.15.67:0/1513124892 shutdown_connections 2024-03-24T00:39:43.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.607+0000 7f60f453b640 1 --2- 172.21.15.67:0/1513124892 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f60d006bc00 0x7f60d006e0c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:43.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.607+0000 7f60f453b640 1 --2- 172.21.15.67:0/1513124892 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f60ec15e250 0x7f60ec154ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:43.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.607+0000 7f60f453b640 1 --2- 172.21.15.67:0/1513124892 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60ec0cd630 0x7f60ec154590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:43.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.607+0000 7f60f453b640 1 --2- 172.21.15.67:0/1513124892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60ec06cb60 0x7f60ec15e660 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:43.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.607+0000 7f60f453b640 1 -- 172.21.15.67:0/1513124892 >> 172.21.15.67:0/1513124892 conn(0x7f60ec0d1bf0 msgr2=0x7f60ec15d3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:43.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.607+0000 7f60f453b640 1 -- 172.21.15.67:0/1513124892 shutdown_connections 2024-03-24T00:39:43.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.608+0000 7f60f453b640 1 -- 172.21.15.67:0/1513124892 wait complete. 2024-03-24T00:39:43.619 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2024-03-24T00:39:43.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:43.619 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2024-03-24T00:39:43.646 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-03-24T00:39:43.646 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:43.647 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-24T00:39:43.701 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:39:43.701 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:39:43.717 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:39:43.717 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:39:43.773 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:39:43.773 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:39:43.829 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:39:43.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.930+0000 7fb2c09b9640 1 Processor -- start 2024-03-24T00:39:43.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.930+0000 7fb2c09b9640 1 -- start start 2024-03-24T00:39:43.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.930+0000 7fb2c09b9640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb2bc15f410 0x7fb2bc0cf7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:43.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.930+0000 7fb2c09b9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb2bc0d0940 0x7fb2bc0cfd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:43.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.930+0000 7fb2c09b9640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb2bc099d70 0x7fb2bc0d0240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:43.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.931+0000 7fb2c09b9640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2bc0cdb50 con 0x7fb2bc0d0940 2024-03-24T00:39:43.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.931+0000 7fb2c09b9640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2bc0cd2b0 con 0x7fb2bc099d70 2024-03-24T00:39:43.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.931+0000 7fb2c09b9640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2bc0cd590 con 0x7fb2bc15f410 2024-03-24T00:39:43.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.931+0000 7fb2b9d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb2bc0d0940 0x7fb2bc0cfd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:43.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.931+0000 7fb2b9d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb2bc0d0940 0x7fb2bc0cfd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49744/0 (socket says 172.21.15.67:49744) 2024-03-24T00:39:43.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.931+0000 7fb2b9d9b640 1 -- 172.21.15.67:0/70152492 learned_addr learned my addr 172.21.15.67:0/70152492 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:43.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.931+0000 7fb2ba59c640 1 --2- 172.21.15.67:0/70152492 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb2bc15f410 0x7fb2bc0cf7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:43.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.931+0000 7fb2bad9d640 1 --2- 172.21.15.67:0/70152492 >> v2:172.21.15.72:3300/0 conn(0x7fb2bc099d70 0x7fb2bc0d0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:43.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.931+0000 7fb2b9d9b640 1 -- 172.21.15.67:0/70152492 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb2bc15f410 msgr2=0x7fb2bc0cf7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:43.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.931+0000 7fb2b9d9b640 1 --2- 172.21.15.67:0/70152492 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb2bc15f410 0x7fb2bc0cf7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:43.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.931+0000 7fb2b9d9b640 1 -- 172.21.15.67:0/70152492 >> v2:172.21.15.72:3300/0 conn(0x7fb2bc099d70 msgr2=0x7fb2bc0d0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:43.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.931+0000 7fb2b9d9b640 1 --2- 172.21.15.67:0/70152492 >> v2:172.21.15.72:3300/0 conn(0x7fb2bc099d70 0x7fb2bc0d0240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:43.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.931+0000 7fb2b9d9b640 1 -- 172.21.15.67:0/70152492 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2bc0cd870 con 0x7fb2bc0d0940 2024-03-24T00:39:43.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.931+0000 7fb2bad9d640 1 --2- 172.21.15.67:0/70152492 >> v2:172.21.15.72:3300/0 conn(0x7fb2bc099d70 0x7fb2bc0d0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:43.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.932+0000 7fb2b9d9b640 1 --2- 172.21.15.67:0/70152492 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2bc0d0940 0x7fb2bc0cfd00 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc0b34a0 tx=0x7fb2ac02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa4dda0c911828fe server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:43.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.932+0000 7fb2b959a640 1 -- 172.21.15.67:0/70152492 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2ac035020 con 0x7fb2bc0d0940 2024-03-24T00:39:43.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.932+0000 7fb2b959a640 1 -- 172.21.15.67:0/70152492 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2ac031870 con 0x7fb2bc0d0940 2024-03-24T00:39:43.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.932+0000 7fb2b959a640 1 -- 172.21.15.67:0/70152492 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2ac031c40 con 0x7fb2bc0d0940 2024-03-24T00:39:43.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.932+0000 7fb2c09b9640 1 -- 172.21.15.67:0/70152492 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2bc0d0940 msgr2=0x7fb2bc0cfd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:43.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.932+0000 7fb2c09b9640 1 --2- 172.21.15.67:0/70152492 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2bc0d0940 0x7fb2bc0cfd00 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc0b34a0 tx=0x7fb2ac02b910 comp rx=0 tx=0).stop 2024-03-24T00:39:43.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.932+0000 7fb2c09b9640 1 -- 172.21.15.67:0/70152492 shutdown_connections 2024-03-24T00:39:43.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.932+0000 7fb2c09b9640 1 --2- 172.21.15.67:0/70152492 >> v2:172.21.15.72:3300/0 conn(0x7fb2bc099d70 0x7fb2bc0d0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:43.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.932+0000 7fb2c09b9640 1 --2- 172.21.15.67:0/70152492 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2bc0d0940 0x7fb2bc0cfd00 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:43.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.932+0000 7fb2c09b9640 1 --2- 172.21.15.67:0/70152492 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb2bc15f410 0x7fb2bc0cf7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:43.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.932+0000 7fb2c09b9640 1 -- 172.21.15.67:0/70152492 >> 172.21.15.67:0/70152492 conn(0x7fb2bc0b36e0 msgr2=0x7fb2bc0b3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:43.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.932+0000 7fb2c09b9640 1 -- 172.21.15.67:0/70152492 shutdown_connections 2024-03-24T00:39:43.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.933+0000 7fb2c09b9640 1 -- 172.21.15.67:0/70152492 wait complete. 2024-03-24T00:39:43.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.934+0000 7fb2c09b9640 1 Processor -- start 2024-03-24T00:39:43.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.934+0000 7fb2c09b9640 1 -- start start 2024-03-24T00:39:43.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.934+0000 7fb2c09b9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb2bc099d70 0x7fb2bc1f5890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:43.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.934+0000 7fb2c09b9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2bc15f410 0x7fb2bc1570e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:43.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.934+0000 7fb2c09b9640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb2bc0b7360 0x7fb2bc0afc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:43.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.934+0000 7fb2c09b9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2bc15f7f0 con 0x7fb2bc15f410 2024-03-24T00:39:43.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.934+0000 7fb2c09b9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2bc15fb70 con 0x7fb2bc099d70 2024-03-24T00:39:43.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.934+0000 7fb2c09b9640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2bc1f5dd0 con 0x7fb2bc0b7360 2024-03-24T00:39:43.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.935+0000 7fb2b9d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2bc15f410 0x7fb2bc1570e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:43.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.935+0000 7fb2b9d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2bc15f410 0x7fb2bc1570e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49748/0 (socket says 172.21.15.67:49748) 2024-03-24T00:39:43.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.935+0000 7fb2b9d9b640 1 -- 172.21.15.67:0/3789001408 learned_addr learned my addr 172.21.15.67:0/3789001408 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:43.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.935+0000 7fb2ba59c640 1 --2- 172.21.15.67:0/3789001408 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb2bc099d70 0x7fb2bc1f5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:43.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.935+0000 7fb2bad9d640 1 --2- 172.21.15.67:0/3789001408 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb2bc0b7360 0x7fb2bc0afc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:43.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.935+0000 7fb2b9d9b640 1 -- 172.21.15.67:0/3789001408 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb2bc0b7360 msgr2=0x7fb2bc0afc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:43.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.935+0000 7fb2b9d9b640 1 --2- 172.21.15.67:0/3789001408 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb2bc0b7360 0x7fb2bc0afc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:43.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.935+0000 7fb2b9d9b640 1 -- 172.21.15.67:0/3789001408 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb2bc099d70 msgr2=0x7fb2bc1f5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:43.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.935+0000 7fb2b9d9b640 1 --2- 172.21.15.67:0/3789001408 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb2bc099d70 0x7fb2bc1f5890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:43.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.935+0000 7fb2b9d9b640 1 -- 172.21.15.67:0/3789001408 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2bc0c6f70 con 0x7fb2bc15f410 2024-03-24T00:39:43.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.935+0000 7fb2b9d9b640 1 --2- 172.21.15.67:0/3789001408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2bc15f410 0x7fb2bc1570e0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac000c00 tx=0x7fb2ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:43.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.935+0000 7fb2b2ffd640 1 -- 172.21.15.67:0/3789001408 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2ac035040 con 0x7fb2bc15f410 2024-03-24T00:39:43.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.935+0000 7fb2b2ffd640 1 -- 172.21.15.67:0/3789001408 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2ac030d00 con 0x7fb2bc15f410 2024-03-24T00:39:43.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.935+0000 7fb2c09b9640 1 -- 172.21.15.67:0/3789001408 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2bc0b7900 con 0x7fb2bc15f410 2024-03-24T00:39:43.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.935+0000 7fb2b2ffd640 1 -- 172.21.15.67:0/3789001408 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2ac0456a0 con 0x7fb2bc15f410 2024-03-24T00:39:43.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.935+0000 7fb2c09b9640 1 -- 172.21.15.67:0/3789001408 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2bc0c7510 con 0x7fb2bc15f410 2024-03-24T00:39:43.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.936+0000 7fb2c09b9640 1 -- 172.21.15.67:0/3789001408 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2bc04f400 con 0x7fb2bc15f410 2024-03-24T00:39:43.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.937+0000 7fb2b2ffd640 1 -- 172.21.15.67:0/3789001408 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb2ac038020 con 0x7fb2bc15f410 2024-03-24T00:39:43.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.937+0000 7fb2b2ffd640 1 --2- 172.21.15.67:0/3789001408 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb29c06bbd0 0x7fb29c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:43.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.937+0000 7fb2b2ffd640 1 -- 172.21.15.67:0/3789001408 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fb2ac0acfa0 con 0x7fb2bc15f410 2024-03-24T00:39:43.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.937+0000 7fb2ba59c640 1 --2- 172.21.15.67:0/3789001408 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb29c06bbd0 0x7fb29c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:43.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.938+0000 7fb2ba59c640 1 --2- 172.21.15.67:0/3789001408 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb29c06bbd0 0x7fb29c06e090 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb2a4009aa0 tx=0x7fb2a4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:43.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:43.942+0000 7fb2b2ffd640 1 -- 172.21.15.67:0/3789001408 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb2ac07b0d0 con 0x7fb2bc15f410 2024-03-24T00:39:44.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.126+0000 7fb2c09b9640 1 -- 172.21.15.67:0/3789001408 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2bc0d19d0 con 0x7fb2bc15f410 2024-03-24T00:39:44.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.127+0000 7fb2b2ffd640 1 -- 172.21.15.67:0/3789001408 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18843 (secure 0 0 0) 0x7fb2ac07f170 con 0x7fb2bc15f410 2024-03-24T00:39:44.128 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:39:44.128 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":27,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:39:32.904729+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:38:50.077085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:38:51.095454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T00:38:54.807117+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.14","mappings":[{"from":4,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:39:44.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.129+0000 7fb2c09b9640 1 -- 172.21.15.67:0/3789001408 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb29c06bbd0 msgr2=0x7fb29c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:44.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.129+0000 7fb2c09b9640 1 --2- 172.21.15.67:0/3789001408 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb29c06bbd0 0x7fb29c06e090 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb2a4009aa0 tx=0x7fb2a4009190 comp rx=0 tx=0).stop 2024-03-24T00:39:44.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.129+0000 7fb2c09b9640 1 -- 172.21.15.67:0/3789001408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2bc15f410 msgr2=0x7fb2bc1570e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:44.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.129+0000 7fb2c09b9640 1 --2- 172.21.15.67:0/3789001408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2bc15f410 0x7fb2bc1570e0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac000c00 tx=0x7fb2ac0046c0 comp rx=0 tx=0).stop 2024-03-24T00:39:44.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.129+0000 7fb2c09b9640 1 -- 172.21.15.67:0/3789001408 shutdown_connections 2024-03-24T00:39:44.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.129+0000 7fb2c09b9640 1 --2- 172.21.15.67:0/3789001408 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb29c06bbd0 0x7fb29c06e090 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:44.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.129+0000 7fb2c09b9640 1 --2- 172.21.15.67:0/3789001408 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb2bc0b7360 0x7fb2bc0afc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:44.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.129+0000 7fb2c09b9640 1 --2- 172.21.15.67:0/3789001408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2bc15f410 0x7fb2bc1570e0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:44.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.129+0000 7fb2c09b9640 1 --2- 172.21.15.67:0/3789001408 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb2bc099d70 0x7fb2bc1f5890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:44.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.129+0000 7fb2c09b9640 1 -- 172.21.15.67:0/3789001408 >> 172.21.15.67:0/3789001408 conn(0x7fb2bc0b36e0 msgr2=0x7fb2bc0b2270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:44.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.129+0000 7fb2c09b9640 1 -- 172.21.15.67:0/3789001408 shutdown_connections 2024-03-24T00:39:44.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.129+0000 7fb2c09b9640 1 -- 172.21.15.67:0/3789001408 wait complete. 2024-03-24T00:39:44.141 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:38:39.364950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:38:50.077085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:38:51.095454+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T00:38:54.807117+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T00:39:44.142 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:39:44.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.246+0000 7f0e68df3640 1 Processor -- start 2024-03-24T00:39:44.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.246+0000 7f0e68df3640 1 -- start start 2024-03-24T00:39:44.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.247+0000 7f0e68df3640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0e6406b750 0x7f0e6415e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:44.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.247+0000 7f0e68df3640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e640d1630 con 0x7f0e6415de90 2024-03-24T00:39:44.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.247+0000 7f0e68df3640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e640ad9f0 con 0x7f0e640af4e0 2024-03-24T00:39:44.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.247+0000 7f0e68df3640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e640d1910 con 0x7f0e6406b750 2024-03-24T00:39:44.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.247+0000 7f0e62d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0e6406b750 0x7f0e6415e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:44.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.247+0000 7f0e6259c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0e6415de90 0x7f0e6406b040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50414/0 (socket says 172.21.15.67:50414) 2024-03-24T00:39:44.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.247+0000 7f0e6259c640 1 -- 172.21.15.67:0/227690487 learned_addr learned my addr 172.21.15.67:0/227690487 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:44.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.247+0000 7f0e62d9d640 1 -- 172.21.15.67:0/227690487 >> v1:172.21.15.72:6789/0 conn(0x7f0e640af4e0 legacy=0x7f0e640cfd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:44.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.247+0000 7f0e62d9d640 1 -- 172.21.15.67:0/227690487 >> v1:172.21.15.67:6789/0 conn(0x7f0e6415de90 legacy=0x7f0e6406b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:44.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.247+0000 7f0e62d9d640 1 -- 172.21.15.67:0/227690487 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e640adcd0 con 0x7f0e6406b750 2024-03-24T00:39:44.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.247+0000 7f0e6159a640 1 -- 172.21.15.67:0/227690487 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2604317980 0 0) 0x7f0e640d1630 con 0x7f0e6415de90 2024-03-24T00:39:44.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.248+0000 7f0e62d9d640 1 --2- 172.21.15.67:0/227690487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e6406b750 0x7f0e6415e270 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c009d90 tx=0x7f0e5c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7760e6b91848aa9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:44.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.248+0000 7f0e6159a640 1 -- 172.21.15.67:0/227690487 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5c039070 con 0x7f0e6406b750 2024-03-24T00:39:44.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.248+0000 7f0e6159a640 1 -- 172.21.15.67:0/227690487 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e5c02cc60 con 0x7f0e6406b750 2024-03-24T00:39:44.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.248+0000 7f0e6159a640 1 -- 172.21.15.67:0/227690487 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5c034660 con 0x7f0e6406b750 2024-03-24T00:39:44.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.248+0000 7f0e68df3640 1 -- 172.21.15.67:0/227690487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e6406b750 msgr2=0x7f0e6415e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:44.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.248+0000 7f0e68df3640 1 --2- 172.21.15.67:0/227690487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e6406b750 0x7f0e6415e270 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c009d90 tx=0x7f0e5c02beb0 comp rx=0 tx=0).stop 2024-03-24T00:39:44.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.249+0000 7f0e68df3640 1 -- 172.21.15.67:0/227690487 shutdown_connections 2024-03-24T00:39:44.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.249+0000 7f0e68df3640 1 --2- 172.21.15.67:0/227690487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e6406b750 0x7f0e6415e270 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:44.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.249+0000 7f0e68df3640 1 -- 172.21.15.67:0/227690487 >> 172.21.15.67:0/227690487 conn(0x7f0e64099a30 msgr2=0x7f0e64099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:44.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.249+0000 7f0e68df3640 1 -- 172.21.15.67:0/227690487 shutdown_connections 2024-03-24T00:39:44.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.249+0000 7f0e68df3640 1 -- 172.21.15.67:0/227690487 wait complete. 2024-03-24T00:39:44.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.250+0000 7f0e68df3640 1 Processor -- start 2024-03-24T00:39:44.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.250+0000 7f0e68df3640 1 -- start start 2024-03-24T00:39:44.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.251+0000 7f0e68df3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e6406b750 0x7f0e640b0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:44.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.251+0000 7f0e68df3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0e640af4e0 0x7f0e640b2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:44.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.251+0000 7f0e68df3640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0e6415de90 0x7f0e6415c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:44.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.251+0000 7f0e68df3640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e640cd460 con 0x7f0e6406b750 2024-03-24T00:39:44.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.251+0000 7f0e68df3640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e640cd2e0 con 0x7f0e640af4e0 2024-03-24T00:39:44.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.251+0000 7f0e68df3640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e640cd5e0 con 0x7f0e6415de90 2024-03-24T00:39:44.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.251+0000 7f0e6259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e6406b750 0x7f0e640b0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:44.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.251+0000 7f0e6259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e6406b750 0x7f0e640b0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49766/0 (socket says 172.21.15.67:49766) 2024-03-24T00:39:44.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.251+0000 7f0e6259c640 1 -- 172.21.15.67:0/2125860908 learned_addr learned my addr 172.21.15.67:0/2125860908 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:44.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.251+0000 7f0e61d9b640 1 --2- 172.21.15.67:0/2125860908 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0e640af4e0 0x7f0e640b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:44.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.251+0000 7f0e62d9d640 1 --2- 172.21.15.67:0/2125860908 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0e6415de90 0x7f0e6415c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:44.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.251+0000 7f0e6259c640 1 -- 172.21.15.67:0/2125860908 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0e6415de90 msgr2=0x7f0e6415c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:44.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.251+0000 7f0e6259c640 1 --2- 172.21.15.67:0/2125860908 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0e6415de90 0x7f0e6415c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:44.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.251+0000 7f0e6259c640 1 -- 172.21.15.67:0/2125860908 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0e640af4e0 msgr2=0x7f0e640b2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:44.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.251+0000 7f0e6259c640 1 --2- 172.21.15.67:0/2125860908 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0e640af4e0 0x7f0e640b2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:44.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.251+0000 7f0e6259c640 1 -- 172.21.15.67:0/2125860908 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e640b0fa0 con 0x7f0e6406b750 2024-03-24T00:39:44.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.251+0000 7f0e6259c640 1 --2- 172.21.15.67:0/2125860908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e6406b750 0x7f0e640b0910 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f0e5400c420 tx=0x7f0e5400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:44.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.251+0000 7f0e52ffd640 1 -- 172.21.15.67:0/2125860908 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e540074e0 con 0x7f0e6406b750 2024-03-24T00:39:44.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.251+0000 7f0e52ffd640 1 -- 172.21.15.67:0/2125860908 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e540047e0 con 0x7f0e6406b750 2024-03-24T00:39:44.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.251+0000 7f0e68df3640 1 -- 172.21.15.67:0/2125860908 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e640b1280 con 0x7f0e6406b750 2024-03-24T00:39:44.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.251+0000 7f0e52ffd640 1 -- 172.21.15.67:0/2125860908 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e54004c10 con 0x7f0e6406b750 2024-03-24T00:39:44.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.251+0000 7f0e68df3640 1 -- 172.21.15.67:0/2125860908 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e640b1840 con 0x7f0e6406b750 2024-03-24T00:39:44.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.253+0000 7f0e68df3640 1 -- 172.21.15.67:0/2125860908 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e6404f400 con 0x7f0e6406b750 2024-03-24T00:39:44.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.253+0000 7f0e52ffd640 1 -- 172.21.15.67:0/2125860908 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0e54011070 con 0x7f0e6406b750 2024-03-24T00:39:44.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.253+0000 7f0e52ffd640 1 --2- 172.21.15.67:0/2125860908 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0e4406bbd0 0x7f0e4406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:44.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.253+0000 7f0e52ffd640 1 -- 172.21.15.67:0/2125860908 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f0e5408a8d0 con 0x7f0e6406b750 2024-03-24T00:39:44.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.253+0000 7f0e61d9b640 1 --2- 172.21.15.67:0/2125860908 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0e4406bbd0 0x7f0e4406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:44.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.254+0000 7f0e61d9b640 1 --2- 172.21.15.67:0/2125860908 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0e4406bbd0 0x7f0e4406e090 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0e6415d5a0 tx=0x7f0e58009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:44.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.258+0000 7f0e52ffd640 1 -- 172.21.15.67:0/2125860908 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0e54058970 con 0x7f0e6406b750 2024-03-24T00:39:44.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.438+0000 7f0e68df3640 1 -- 172.21.15.67:0/2125860908 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0e6406b250 con 0x7f0e6406b750 2024-03-24T00:39:44.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.438+0000 7f0e52ffd640 1 -- 172.21.15.67:0/2125860908 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0e54088e20 con 0x7f0e6406b750 2024-03-24T00:39:44.439 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-24T00:39:44.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.440+0000 7f0e68df3640 1 -- 172.21.15.67:0/2125860908 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0e4406bbd0 msgr2=0x7f0e4406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:44.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.440+0000 7f0e68df3640 1 --2- 172.21.15.67:0/2125860908 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0e4406bbd0 0x7f0e4406e090 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0e6415d5a0 tx=0x7f0e58009040 comp rx=0 tx=0).stop 2024-03-24T00:39:44.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.440+0000 7f0e68df3640 1 -- 172.21.15.67:0/2125860908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e6406b750 msgr2=0x7f0e640b0910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:44.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.440+0000 7f0e68df3640 1 --2- 172.21.15.67:0/2125860908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e6406b750 0x7f0e640b0910 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f0e5400c420 tx=0x7f0e5400c8e0 comp rx=0 tx=0).stop 2024-03-24T00:39:44.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.440+0000 7f0e68df3640 1 -- 172.21.15.67:0/2125860908 shutdown_connections 2024-03-24T00:39:44.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.440+0000 7f0e68df3640 1 --2- 172.21.15.67:0/2125860908 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0e4406bbd0 0x7f0e4406e090 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:44.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.440+0000 7f0e68df3640 1 --2- 172.21.15.67:0/2125860908 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0e6415de90 0x7f0e6415c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:44.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.440+0000 7f0e68df3640 1 --2- 172.21.15.67:0/2125860908 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0e640af4e0 0x7f0e640b2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:44.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.440+0000 7f0e68df3640 1 --2- 172.21.15.67:0/2125860908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e6406b750 0x7f0e640b0910 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:44.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.440+0000 7f0e68df3640 1 -- 172.21.15.67:0/2125860908 >> 172.21.15.67:0/2125860908 conn(0x7f0e64099a30 msgr2=0x7f0e64099e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:44.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.440+0000 7f0e68df3640 1 -- 172.21.15.67:0/2125860908 shutdown_connections 2024-03-24T00:39:44.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.440+0000 7f0e68df3640 1 -- 172.21.15.67:0/2125860908 wait complete. 2024-03-24T00:39:44.452 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:39:44.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.555+0000 7fd63e697640 1 Processor -- start 2024-03-24T00:39:44.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.555+0000 7fd63e697640 1 -- start start 2024-03-24T00:39:44.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.556+0000 7fd63e697640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd638099d00 0x7fd63809a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:44.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.556+0000 7fd63e697640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd6380d1530 0x7fd63815d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:44.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.556+0000 7fd63e697640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd63809a620 con 0x7fd6380b49b0 2024-03-24T00:39:44.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.556+0000 7fd63e697640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6380cc1b0 con 0x7fd6380d1530 2024-03-24T00:39:44.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.556+0000 7fd63e697640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6380cc400 con 0x7fd638099d00 2024-03-24T00:39:44.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.556+0000 7fd63cc34640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd6380d1530 0x7fd63815d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:44.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.556+0000 7fd6377fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd6380b49b0 0x7fd6380d0e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50430/0 (socket says 172.21.15.67:50430) 2024-03-24T00:39:44.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.556+0000 7fd6377fe640 1 -- 172.21.15.67:0/3183651029 learned_addr learned my addr 172.21.15.67:0/3183651029 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:44.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.556+0000 7fd637fff640 1 --2- 172.21.15.67:0/3183651029 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd638099d00 0x7fd63809a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:44.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.557+0000 7fd63cc34640 1 -- 172.21.15.67:0/3183651029 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd638099d00 msgr2=0x7fd63809a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:44.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.557+0000 7fd63cc34640 1 --2- 172.21.15.67:0/3183651029 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd638099d00 0x7fd63809a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:44.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.557+0000 7fd63cc34640 1 -- 172.21.15.67:0/3183651029 >> v1:172.21.15.67:6789/0 conn(0x7fd6380b49b0 legacy=0x7fd6380d0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:44.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.557+0000 7fd63cc34640 1 -- 172.21.15.67:0/3183651029 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6380cc6e0 con 0x7fd6380d1530 2024-03-24T00:39:44.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.557+0000 7fd636ffd640 1 -- 172.21.15.67:0/3183651029 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1193175514 0 0) 0x7fd63809a620 con 0x7fd6380b49b0 2024-03-24T00:39:44.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.557+0000 7fd63cc34640 1 --2- 172.21.15.67:0/3183651029 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6380d1530 0x7fd63815d7d0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fd630009d90 tx=0x7fd63002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e297e25d38d3fd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:44.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.557+0000 7fd636ffd640 1 -- 172.21.15.67:0/3183651029 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd630039070 con 0x7fd6380d1530 2024-03-24T00:39:44.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.557+0000 7fd636ffd640 1 -- 172.21.15.67:0/3183651029 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd63002cc50 con 0x7fd6380d1530 2024-03-24T00:39:44.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.558+0000 7fd636ffd640 1 -- 172.21.15.67:0/3183651029 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd630034440 con 0x7fd6380d1530 2024-03-24T00:39:44.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.558+0000 7fd63e697640 1 -- 172.21.15.67:0/3183651029 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6380d1530 msgr2=0x7fd63815d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:44.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.558+0000 7fd63e697640 1 --2- 172.21.15.67:0/3183651029 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6380d1530 0x7fd63815d7d0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fd630009d90 tx=0x7fd63002bea0 comp rx=0 tx=0).stop 2024-03-24T00:39:44.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.558+0000 7fd63e697640 1 -- 172.21.15.67:0/3183651029 shutdown_connections 2024-03-24T00:39:44.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.558+0000 7fd63e697640 1 --2- 172.21.15.67:0/3183651029 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6380d1530 0x7fd63815d7d0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:44.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.558+0000 7fd63e697640 1 --2- 172.21.15.67:0/3183651029 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd638099d00 0x7fd63809a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:44.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.558+0000 7fd63e697640 1 -- 172.21.15.67:0/3183651029 >> 172.21.15.67:0/3183651029 conn(0x7fd6380b3650 msgr2=0x7fd6380b3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:44.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.558+0000 7fd63e697640 1 -- 172.21.15.67:0/3183651029 shutdown_connections 2024-03-24T00:39:44.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.558+0000 7fd63e697640 1 -- 172.21.15.67:0/3183651029 wait complete. 2024-03-24T00:39:44.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.559+0000 7fd63e697640 1 Processor -- start 2024-03-24T00:39:44.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.559+0000 7fd63e697640 1 -- start start 2024-03-24T00:39:44.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.560+0000 7fd63e697640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd638099d00 0x7fd6381478e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:44.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.560+0000 7fd63e697640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6380b49b0 0x7fd6380c8020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:44.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.560+0000 7fd63e697640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd6381f7f60 0x7fd6381f7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:44.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.560+0000 7fd63e697640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6380d01c0 con 0x7fd6380b49b0 2024-03-24T00:39:44.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.560+0000 7fd63e697640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6380ccca0 con 0x7fd638099d00 2024-03-24T00:39:44.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.560+0000 7fd63e697640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6380b32a0 con 0x7fd6381f7f60 2024-03-24T00:39:44.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.560+0000 7fd6377fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6380b49b0 0x7fd6380c8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:44.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.560+0000 7fd6377fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6380b49b0 0x7fd6380c8020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49776/0 (socket says 172.21.15.67:49776) 2024-03-24T00:39:44.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.560+0000 7fd6377fe640 1 -- 172.21.15.67:0/2030702547 learned_addr learned my addr 172.21.15.67:0/2030702547 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:44.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.560+0000 7fd63cc34640 1 --2- 172.21.15.67:0/2030702547 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd6381f7f60 0x7fd6381f7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:44.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.560+0000 7fd637fff640 1 --2- 172.21.15.67:0/2030702547 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd638099d00 0x7fd6381478e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:44.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.561+0000 7fd6377fe640 1 -- 172.21.15.67:0/2030702547 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd6381f7f60 msgr2=0x7fd6381f7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:44.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.561+0000 7fd6377fe640 1 --2- 172.21.15.67:0/2030702547 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd6381f7f60 0x7fd6381f7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:44.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.561+0000 7fd6377fe640 1 -- 172.21.15.67:0/2030702547 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd638099d00 msgr2=0x7fd6381478e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:44.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.561+0000 7fd6377fe640 1 --2- 172.21.15.67:0/2030702547 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd638099d00 0x7fd6381478e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:44.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.561+0000 7fd6377fe640 1 -- 172.21.15.67:0/2030702547 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6380c62b0 con 0x7fd6380b49b0 2024-03-24T00:39:44.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.561+0000 7fd6377fe640 1 --2- 172.21.15.67:0/2030702547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6380b49b0 0x7fd6380c8020 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fd62000c4d0 tx=0x7fd62000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:44.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.561+0000 7fd634ff9640 1 -- 172.21.15.67:0/2030702547 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd620018070 con 0x7fd6380b49b0 2024-03-24T00:39:44.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.561+0000 7fd634ff9640 1 -- 172.21.15.67:0/2030702547 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd620007500 con 0x7fd6380b49b0 2024-03-24T00:39:44.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.561+0000 7fd63e697640 1 -- 172.21.15.67:0/2030702547 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6380c6590 con 0x7fd6380b49b0 2024-03-24T00:39:44.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.561+0000 7fd634ff9640 1 -- 172.21.15.67:0/2030702547 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd620004a20 con 0x7fd6380b49b0 2024-03-24T00:39:44.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.561+0000 7fd63e697640 1 -- 172.21.15.67:0/2030702547 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6380c6b50 con 0x7fd6380b49b0 2024-03-24T00:39:44.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.562+0000 7fd63e697640 1 -- 172.21.15.67:0/2030702547 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd63804f400 con 0x7fd6380b49b0 2024-03-24T00:39:44.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.562+0000 7fd634ff9640 1 -- 172.21.15.67:0/2030702547 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd62001b070 con 0x7fd6380b49b0 2024-03-24T00:39:44.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.563+0000 7fd634ff9640 1 --2- 172.21.15.67:0/2030702547 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd61806bbd0 0x7fd61806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:44.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.563+0000 7fd634ff9640 1 -- 172.21.15.67:0/2030702547 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fd62008a150 con 0x7fd6380b49b0 2024-03-24T00:39:44.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.563+0000 7fd637fff640 1 --2- 172.21.15.67:0/2030702547 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd61806bbd0 0x7fd61806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:44.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.563+0000 7fd637fff640 1 --2- 172.21.15.67:0/2030702547 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd61806bbd0 0x7fd61806e090 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd628009b40 tx=0x7fd628009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:44.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.567+0000 7fd634ff9640 1 -- 172.21.15.67:0/2030702547 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd6200581f0 con 0x7fd6380b49b0 2024-03-24T00:39:44.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.747+0000 7fd63e697640 1 -- 172.21.15.67:0/2030702547 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd6380b2a30 con 0x7fd6380b49b0 2024-03-24T00:39:44.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.748+0000 7fd634ff9640 1 -- 172.21.15.67:0/2030702547 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd62005e030 con 0x7fd6380b49b0 2024-03-24T00:39:44.748 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-24T00:39:44.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.749+0000 7fd63e697640 1 -- 172.21.15.67:0/2030702547 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd61806bbd0 msgr2=0x7fd61806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:44.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.749+0000 7fd63e697640 1 --2- 172.21.15.67:0/2030702547 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd61806bbd0 0x7fd61806e090 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd628009b40 tx=0x7fd628009190 comp rx=0 tx=0).stop 2024-03-24T00:39:44.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.749+0000 7fd63e697640 1 -- 172.21.15.67:0/2030702547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6380b49b0 msgr2=0x7fd6380c8020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:44.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.749+0000 7fd63e697640 1 --2- 172.21.15.67:0/2030702547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6380b49b0 0x7fd6380c8020 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fd62000c4d0 tx=0x7fd62000c990 comp rx=0 tx=0).stop 2024-03-24T00:39:44.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.749+0000 7fd63e697640 1 -- 172.21.15.67:0/2030702547 shutdown_connections 2024-03-24T00:39:44.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.749+0000 7fd63e697640 1 --2- 172.21.15.67:0/2030702547 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd61806bbd0 0x7fd61806e090 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:44.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.749+0000 7fd63e697640 1 --2- 172.21.15.67:0/2030702547 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd6381f7f60 0x7fd6381f7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:44.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.749+0000 7fd63e697640 1 --2- 172.21.15.67:0/2030702547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6380b49b0 0x7fd6380c8020 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:44.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.749+0000 7fd63e697640 1 --2- 172.21.15.67:0/2030702547 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd638099d00 0x7fd6381478e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:44.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.749+0000 7fd63e697640 1 -- 172.21.15.67:0/2030702547 >> 172.21.15.67:0/2030702547 conn(0x7fd6380b3650 msgr2=0x7fd638070880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:44.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.750+0000 7fd63e697640 1 -- 172.21.15.67:0/2030702547 shutdown_connections 2024-03-24T00:39:44.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.750+0000 7fd63e697640 1 -- 172.21.15.67:0/2030702547 wait complete. 2024-03-24T00:39:44.761 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:39:44.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.866+0000 7f8de5aac640 1 Processor -- start 2024-03-24T00:39:44.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.866+0000 7f8de5aac640 1 -- start start 2024-03-24T00:39:44.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.867+0000 7f8de5aac640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8de00afae0 0x7f8de00ad8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:44.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.867+0000 7f8de5aac640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8de00b0c10 0x7f8de00b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:44.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.867+0000 7f8de5aac640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8de015c220 con 0x7f8de006b040 2024-03-24T00:39:44.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.867+0000 7f8de5aac640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8de00cc950 con 0x7f8de00afae0 2024-03-24T00:39:44.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.867+0000 7f8de5aac640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8de00ccc30 con 0x7f8de00b0c10 2024-03-24T00:39:44.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.867+0000 7f8ddf7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8de00afae0 0x7f8de00ad8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:44.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.867+0000 7f8ddffff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8de006b040 0x7f8de00cd9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50432/0 (socket says 172.21.15.67:50432) 2024-03-24T00:39:44.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.867+0000 7f8ddffff640 1 -- 172.21.15.67:0/3823862223 learned_addr learned my addr 172.21.15.67:0/3823862223 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:44.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.867+0000 7f8ddeffd640 1 --2- 172.21.15.67:0/3823862223 >> v2:172.21.15.72:3300/0 conn(0x7f8de00b0c10 0x7f8de00b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:44.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.867+0000 7f8ddf7fe640 1 -- 172.21.15.67:0/3823862223 >> v2:172.21.15.72:3300/0 conn(0x7f8de00b0c10 msgr2=0x7f8de00b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:44.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.867+0000 7f8ddf7fe640 1 --2- 172.21.15.67:0/3823862223 >> v2:172.21.15.72:3300/0 conn(0x7f8de00b0c10 0x7f8de00b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:44.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.867+0000 7f8ddf7fe640 1 -- 172.21.15.67:0/3823862223 >> v1:172.21.15.67:6789/0 conn(0x7f8de006b040 legacy=0x7f8de00cd9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:44.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.867+0000 7f8ddf7fe640 1 -- 172.21.15.67:0/3823862223 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8de00ccf10 con 0x7f8de00afae0 2024-03-24T00:39:44.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.867+0000 7f8dde7fc640 1 -- 172.21.15.67:0/3823862223 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3640899389 0 0) 0x7f8de015c220 con 0x7f8de006b040 2024-03-24T00:39:44.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.867+0000 7f8ddeffd640 1 --2- 172.21.15.67:0/3823862223 >> v2:172.21.15.72:3300/0 conn(0x7f8de00b0c10 0x7f8de00b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:44.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.868+0000 7f8ddf7fe640 1 --2- 172.21.15.67:0/3823862223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8de00afae0 0x7f8de00ad8c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f8de00b3710 tx=0x7f8dd002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=997475147395be60 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:44.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.868+0000 7f8dde7fc640 1 -- 172.21.15.67:0/3823862223 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd0035020 con 0x7f8de00afae0 2024-03-24T00:39:44.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.868+0000 7f8dde7fc640 1 -- 172.21.15.67:0/3823862223 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dd0031830 con 0x7f8de00afae0 2024-03-24T00:39:44.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.868+0000 7f8dde7fc640 1 -- 172.21.15.67:0/3823862223 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd0031c00 con 0x7f8de00afae0 2024-03-24T00:39:44.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.868+0000 7f8de5aac640 1 -- 172.21.15.67:0/3823862223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8de00afae0 msgr2=0x7f8de00ad8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:44.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.868+0000 7f8de5aac640 1 --2- 172.21.15.67:0/3823862223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8de00afae0 0x7f8de00ad8c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f8de00b3710 tx=0x7f8dd002b8d0 comp rx=0 tx=0).stop 2024-03-24T00:39:44.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.868+0000 7f8de5aac640 1 -- 172.21.15.67:0/3823862223 shutdown_connections 2024-03-24T00:39:44.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.868+0000 7f8de5aac640 1 --2- 172.21.15.67:0/3823862223 >> v2:172.21.15.72:3300/0 conn(0x7f8de00b0c10 0x7f8de00b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:44.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.868+0000 7f8de5aac640 1 --2- 172.21.15.67:0/3823862223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8de00afae0 0x7f8de00ad8c0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:44.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.868+0000 7f8de5aac640 1 -- 172.21.15.67:0/3823862223 >> 172.21.15.67:0/3823862223 conn(0x7f8de0099a30 msgr2=0x7f8de0099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:44.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.869+0000 7f8de5aac640 1 -- 172.21.15.67:0/3823862223 shutdown_connections 2024-03-24T00:39:44.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.869+0000 7f8de5aac640 1 -- 172.21.15.67:0/3823862223 wait complete. 2024-03-24T00:39:44.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.870+0000 7f8de5aac640 1 Processor -- start 2024-03-24T00:39:44.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.870+0000 7f8de5aac640 1 -- start start 2024-03-24T00:39:44.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.871+0000 7f8de5aac640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8de006b040 0x7f8de0155700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:44.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.871+0000 7f8de5aac640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8de00afae0 0x7f8de006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:44.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.871+0000 7f8de5aac640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8de00b0c10 0x7f8de006f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:44.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.871+0000 7f8de5aac640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de00cc600 con 0x7f8de00afae0 2024-03-24T00:39:44.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.871+0000 7f8de5aac640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de00cd4d0 con 0x7f8de006b040 2024-03-24T00:39:44.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.871+0000 7f8de5aac640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8de00ade00 con 0x7f8de00b0c10 2024-03-24T00:39:44.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.871+0000 7f8ddeffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8de00afae0 0x7f8de006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:44.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.871+0000 7f8ddeffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8de00afae0 0x7f8de006ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49800/0 (socket says 172.21.15.67:49800) 2024-03-24T00:39:44.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.871+0000 7f8ddeffd640 1 -- 172.21.15.67:0/2087498279 learned_addr learned my addr 172.21.15.67:0/2087498279 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:44.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.871+0000 7f8ddf7fe640 1 --2- 172.21.15.67:0/2087498279 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8de006b040 0x7f8de0155700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:44.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.871+0000 7f8ddffff640 1 --2- 172.21.15.67:0/2087498279 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8de00b0c10 0x7f8de006f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:44.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.871+0000 7f8ddeffd640 1 -- 172.21.15.67:0/2087498279 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8de00b0c10 msgr2=0x7f8de006f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:44.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.871+0000 7f8ddeffd640 1 --2- 172.21.15.67:0/2087498279 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8de00b0c10 0x7f8de006f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:44.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.871+0000 7f8ddeffd640 1 -- 172.21.15.67:0/2087498279 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8de006b040 msgr2=0x7f8de0155700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:44.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.871+0000 7f8ddeffd640 1 --2- 172.21.15.67:0/2087498279 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8de006b040 0x7f8de0155700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:44.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.871+0000 7f8ddeffd640 1 -- 172.21.15.67:0/2087498279 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8de00b5b50 con 0x7f8de00afae0 2024-03-24T00:39:44.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.871+0000 7f8ddeffd640 1 --2- 172.21.15.67:0/2087498279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8de00afae0 0x7f8de006ec70 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f8dd400e960 tx=0x7f8dd400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:44.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.871+0000 7f8dc7fff640 1 -- 172.21.15.67:0/2087498279 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd4017070 con 0x7f8de00afae0 2024-03-24T00:39:44.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.871+0000 7f8dc7fff640 1 -- 172.21.15.67:0/2087498279 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dd400ca00 con 0x7f8de00afae0 2024-03-24T00:39:44.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.871+0000 7f8dc7fff640 1 -- 172.21.15.67:0/2087498279 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd400cdd0 con 0x7f8de00afae0 2024-03-24T00:39:44.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.871+0000 7f8de5aac640 1 -- 172.21.15.67:0/2087498279 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8de00b5e30 con 0x7f8de00afae0 2024-03-24T00:39:44.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.871+0000 7f8de5aac640 1 -- 172.21.15.67:0/2087498279 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8de00b63f0 con 0x7f8de00afae0 2024-03-24T00:39:44.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.872+0000 7f8de5aac640 1 -- 172.21.15.67:0/2087498279 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8de004f400 con 0x7f8de00afae0 2024-03-24T00:39:44.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.873+0000 7f8dc7fff640 1 -- 172.21.15.67:0/2087498279 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8dd401a070 con 0x7f8de00afae0 2024-03-24T00:39:44.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.873+0000 7f8dc7fff640 1 --2- 172.21.15.67:0/2087498279 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8dcc06bbd0 0x7f8dcc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:44.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.873+0000 7f8dc7fff640 1 -- 172.21.15.67:0/2087498279 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f8dd408b020 con 0x7f8de00afae0 2024-03-24T00:39:44.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.873+0000 7f8ddf7fe640 1 --2- 172.21.15.67:0/2087498279 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8dcc06bbd0 0x7f8dcc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:44.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.874+0000 7f8ddf7fe640 1 --2- 172.21.15.67:0/2087498279 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8dcc06bbd0 0x7f8dcc06e090 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8dd0036000 tx=0x7f8dd0008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:44.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:44.878+0000 7f8dc7fff640 1 -- 172.21.15.67:0/2087498279 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8dd40590c0 con 0x7f8de00afae0 2024-03-24T00:39:45.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.059+0000 7f8de5aac640 1 -- 172.21.15.67:0/2087498279 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8de00b0740 con 0x7f8de00afae0 2024-03-24T00:39:45.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.059+0000 7f8dc7fff640 1 -- 172.21.15.67:0/2087498279 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8dd405d160 con 0x7f8de00afae0 2024-03-24T00:39:45.060 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-03-24T00:39:45.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.061+0000 7f8de5aac640 1 -- 172.21.15.67:0/2087498279 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8dcc06bbd0 msgr2=0x7f8dcc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:45.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.061+0000 7f8de5aac640 1 --2- 172.21.15.67:0/2087498279 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8dcc06bbd0 0x7f8dcc06e090 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8dd0036000 tx=0x7f8dd0008030 comp rx=0 tx=0).stop 2024-03-24T00:39:45.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.061+0000 7f8de5aac640 1 -- 172.21.15.67:0/2087498279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8de00afae0 msgr2=0x7f8de006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:45.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.061+0000 7f8de5aac640 1 --2- 172.21.15.67:0/2087498279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8de00afae0 0x7f8de006ec70 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f8dd400e960 tx=0x7f8dd400ee20 comp rx=0 tx=0).stop 2024-03-24T00:39:45.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.061+0000 7f8de5aac640 1 -- 172.21.15.67:0/2087498279 shutdown_connections 2024-03-24T00:39:45.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.061+0000 7f8de5aac640 1 --2- 172.21.15.67:0/2087498279 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8dcc06bbd0 0x7f8dcc06e090 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.061+0000 7f8de5aac640 1 --2- 172.21.15.67:0/2087498279 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8de00b0c10 0x7f8de006f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.061+0000 7f8de5aac640 1 --2- 172.21.15.67:0/2087498279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8de00afae0 0x7f8de006ec70 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.061+0000 7f8de5aac640 1 --2- 172.21.15.67:0/2087498279 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8de006b040 0x7f8de0155700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.061+0000 7f8de5aac640 1 -- 172.21.15.67:0/2087498279 >> 172.21.15.67:0/2087498279 conn(0x7f8de0099a30 msgr2=0x7f8de00cdcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:45.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.061+0000 7f8de5aac640 1 -- 172.21.15.67:0/2087498279 shutdown_connections 2024-03-24T00:39:45.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.061+0000 7f8de5aac640 1 -- 172.21.15.67:0/2087498279 wait complete. 2024-03-24T00:39:45.072 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T00:39:45.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.176+0000 7f741d190640 1 Processor -- start 2024-03-24T00:39:45.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.176+0000 7f741d190640 1 -- start start 2024-03-24T00:39:45.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.177+0000 7f741d190640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f74180b9690 0x7f74180ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:45.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.177+0000 7f741d190640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f74180cef30 0x7f74180d2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:45.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.177+0000 7f741d190640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7418162290 con 0x7f7418161370 2024-03-24T00:39:45.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.177+0000 7f741d190640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74180b6e10 con 0x7f74180cef30 2024-03-24T00:39:45.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.177+0000 7f741d190640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74180cf380 con 0x7f74180b9690 2024-03-24T00:39:45.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.177+0000 7f741759e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f74180cef30 0x7f74180d2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:45.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.177+0000 7f741759e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f74180cef30 0x7f74180d2620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49812/0 (socket says 172.21.15.67:49812) 2024-03-24T00:39:45.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.177+0000 7f741759e640 1 -- 172.21.15.67:0/1258827293 learned_addr learned my addr 172.21.15.67:0/1258827293 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:45.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.177+0000 7f741659c640 1 --2- 172.21.15.67:0/1258827293 >> v2:172.21.15.72:3300/0 conn(0x7f74180b9690 0x7f74180ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:45.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.177+0000 7f741759e640 1 -- 172.21.15.67:0/1258827293 >> v2:172.21.15.72:3300/0 conn(0x7f74180b9690 msgr2=0x7f74180ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:45.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.177+0000 7f741759e640 1 --2- 172.21.15.67:0/1258827293 >> v2:172.21.15.72:3300/0 conn(0x7f74180b9690 0x7f74180ce9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.177+0000 7f741759e640 1 -- 172.21.15.67:0/1258827293 >> v1:172.21.15.72:6789/0 conn(0x7f7418161370 legacy=0x7f7418161ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:45.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.177+0000 7f741759e640 1 -- 172.21.15.67:0/1258827293 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7418162570 con 0x7f74180cef30 2024-03-24T00:39:45.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.178+0000 7f741759e640 1 --2- 172.21.15.67:0/1258827293 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74180cef30 0x7f74180d2620 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f74180ce690 tx=0x7f740802c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78b86961c33d6067 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:45.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.178+0000 7f7415d9b640 1 -- 172.21.15.67:0/1258827293 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f740802cbb0 con 0x7f74180cef30 2024-03-24T00:39:45.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.178+0000 7f7415d9b640 1 -- 172.21.15.67:0/1258827293 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f740802cd50 con 0x7f74180cef30 2024-03-24T00:39:45.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.178+0000 7f7415d9b640 1 -- 172.21.15.67:0/1258827293 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7408034610 con 0x7f74180cef30 2024-03-24T00:39:45.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.179+0000 7f741d190640 1 -- 172.21.15.67:0/1258827293 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74180cef30 msgr2=0x7f74180d2620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:45.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.179+0000 7f741d190640 1 --2- 172.21.15.67:0/1258827293 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74180cef30 0x7f74180d2620 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f74180ce690 tx=0x7f740802c010 comp rx=0 tx=0).stop 2024-03-24T00:39:45.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.179+0000 7f741d190640 1 -- 172.21.15.67:0/1258827293 shutdown_connections 2024-03-24T00:39:45.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.179+0000 7f741d190640 1 --2- 172.21.15.67:0/1258827293 >> v2:172.21.15.72:3300/0 conn(0x7f74180b9690 0x7f74180ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.179+0000 7f741d190640 1 --2- 172.21.15.67:0/1258827293 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74180cef30 0x7f74180d2620 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.179+0000 7f741d190640 1 -- 172.21.15.67:0/1258827293 >> 172.21.15.67:0/1258827293 conn(0x7f741806c880 msgr2=0x7f741806cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:45.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.179+0000 7f741d190640 1 -- 172.21.15.67:0/1258827293 shutdown_connections 2024-03-24T00:39:45.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.179+0000 7f741d190640 1 -- 172.21.15.67:0/1258827293 wait complete. 2024-03-24T00:39:45.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.180+0000 7f741d190640 1 Processor -- start 2024-03-24T00:39:45.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.181+0000 7f741d190640 1 -- start start 2024-03-24T00:39:45.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.181+0000 7f741d190640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f74180b9690 0x7f74181f3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:45.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.181+0000 7f741d190640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74180cef30 0x7f74180cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:45.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.181+0000 7f741d190640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7418161370 0x7f74180cd900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:45.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.181+0000 7f741d190640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7418161750 con 0x7f74180cef30 2024-03-24T00:39:45.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.181+0000 7f741d190640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74180d2190 con 0x7f7418161370 2024-03-24T00:39:45.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.181+0000 7f741d190640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74180d2b60 con 0x7f74180b9690 2024-03-24T00:39:45.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.181+0000 7f741659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74180cef30 0x7f74180cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:45.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.181+0000 7f741659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74180cef30 0x7f74180cd3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49824/0 (socket says 172.21.15.67:49824) 2024-03-24T00:39:45.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.181+0000 7f741659c640 1 -- 172.21.15.67:0/3252833421 learned_addr learned my addr 172.21.15.67:0/3252833421 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:45.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.181+0000 7f741759e640 1 --2- 172.21.15.67:0/3252833421 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7418161370 0x7f74180cd900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:45.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.181+0000 7f7416d9d640 1 --2- 172.21.15.67:0/3252833421 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f74180b9690 0x7f74181f3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:45.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.182+0000 7f741659c640 1 -- 172.21.15.67:0/3252833421 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f74180b9690 msgr2=0x7f74181f3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:45.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.182+0000 7f741659c640 1 --2- 172.21.15.67:0/3252833421 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f74180b9690 0x7f74181f3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.182+0000 7f741659c640 1 -- 172.21.15.67:0/3252833421 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7418161370 msgr2=0x7f74180cd900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:45.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.182+0000 7f741659c640 1 --2- 172.21.15.67:0/3252833421 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7418161370 0x7f74180cd900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.182+0000 7f741659c640 1 -- 172.21.15.67:0/3252833421 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7418148700 con 0x7f74180cef30 2024-03-24T00:39:45.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.182+0000 7f741759e640 1 --2- 172.21.15.67:0/3252833421 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7418161370 0x7f74180cd900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:45.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.182+0000 7f741659c640 1 --2- 172.21.15.67:0/3252833421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74180cef30 0x7f74180cd3c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f740000e9b0 tx=0x7f740000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:45.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.182+0000 7f740f7fe640 1 -- 172.21.15.67:0/3252833421 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7400017070 con 0x7f74180cef30 2024-03-24T00:39:45.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.182+0000 7f740f7fe640 1 -- 172.21.15.67:0/3252833421 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f740000c9a0 con 0x7f74180cef30 2024-03-24T00:39:45.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.182+0000 7f741d190640 1 -- 172.21.15.67:0/3252833421 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74181542c0 con 0x7f74180cef30 2024-03-24T00:39:45.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.182+0000 7f740f7fe640 1 -- 172.21.15.67:0/3252833421 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f740000cd90 con 0x7f74180cef30 2024-03-24T00:39:45.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.182+0000 7f741d190640 1 -- 172.21.15.67:0/3252833421 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74180c8b20 con 0x7f74180cef30 2024-03-24T00:39:45.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.183+0000 7f741d190640 1 -- 172.21.15.67:0/3252833421 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f741804f400 con 0x7f74180cef30 2024-03-24T00:39:45.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.183+0000 7f740f7fe640 1 -- 172.21.15.67:0/3252833421 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f740001a070 con 0x7f74180cef30 2024-03-24T00:39:45.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.184+0000 7f740f7fe640 1 --2- 172.21.15.67:0/3252833421 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f73f806bbd0 0x7f73f806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:45.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.184+0000 7f740f7fe640 1 -- 172.21.15.67:0/3252833421 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f740008ade0 con 0x7f74180cef30 2024-03-24T00:39:45.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.184+0000 7f7416d9d640 1 --2- 172.21.15.67:0/3252833421 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f73f806bbd0 0x7f73f806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:45.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.184+0000 7f7416d9d640 1 --2- 172.21.15.67:0/3252833421 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f73f806bbd0 0x7f73f806e090 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f74100016e0 tx=0x7f7410009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:45.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.188+0000 7f740f7fe640 1 -- 172.21.15.67:0/3252833421 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7400058e80 con 0x7f74180cef30 2024-03-24T00:39:45.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.369+0000 7f741d190640 1 -- 172.21.15.67:0/3252833421 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f74180b9c90 con 0x7f74180cef30 2024-03-24T00:39:45.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.369+0000 7f740f7fe640 1 -- 172.21.15.67:0/3252833421 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f740005cf20 con 0x7f74180cef30 2024-03-24T00:39:45.370 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-24T00:39:45.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.371+0000 7f741d190640 1 -- 172.21.15.67:0/3252833421 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f73f806bbd0 msgr2=0x7f73f806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:45.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.371+0000 7f741d190640 1 --2- 172.21.15.67:0/3252833421 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f73f806bbd0 0x7f73f806e090 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f74100016e0 tx=0x7f7410009040 comp rx=0 tx=0).stop 2024-03-24T00:39:45.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.371+0000 7f741d190640 1 -- 172.21.15.67:0/3252833421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74180cef30 msgr2=0x7f74180cd3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:45.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.371+0000 7f741d190640 1 --2- 172.21.15.67:0/3252833421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74180cef30 0x7f74180cd3c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f740000e9b0 tx=0x7f740000ee70 comp rx=0 tx=0).stop 2024-03-24T00:39:45.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.371+0000 7f741d190640 1 -- 172.21.15.67:0/3252833421 shutdown_connections 2024-03-24T00:39:45.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.371+0000 7f741d190640 1 --2- 172.21.15.67:0/3252833421 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f73f806bbd0 0x7f73f806e090 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.371+0000 7f741d190640 1 --2- 172.21.15.67:0/3252833421 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7418161370 0x7f74180cd900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.371+0000 7f741d190640 1 --2- 172.21.15.67:0/3252833421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74180cef30 0x7f74180cd3c0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.371+0000 7f741d190640 1 --2- 172.21.15.67:0/3252833421 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f74180b9690 0x7f74181f3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.371+0000 7f741d190640 1 -- 172.21.15.67:0/3252833421 >> 172.21.15.67:0/3252833421 conn(0x7f741806c880 msgr2=0x7f74180d2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:45.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.371+0000 7f741d190640 1 -- 172.21.15.67:0/3252833421 shutdown_connections 2024-03-24T00:39:45.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.371+0000 7f741d190640 1 -- 172.21.15.67:0/3252833421 wait complete. 2024-03-24T00:39:45.383 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T00:39:45.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.485+0000 7f668c0e8640 1 Processor -- start 2024-03-24T00:39:45.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.485+0000 7f668c0e8640 1 -- start start 2024-03-24T00:39:45.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.486+0000 7f668c0e8640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f668415ded0 0x7f66840b96e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:45.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.486+0000 7f668c0e8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66840d2190 con 0x7f668415e2b0 2024-03-24T00:39:45.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.486+0000 7f668c0e8640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66841620f0 con 0x7f66840cf230 2024-03-24T00:39:45.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.486+0000 7f668c0e8640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66841623d0 con 0x7f668415ded0 2024-03-24T00:39:45.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.486+0000 7f6689683640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f668415e2b0 0x7f66840ceb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50442/0 (socket says 172.21.15.67:50442) 2024-03-24T00:39:45.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.486+0000 7f6689683640 1 -- 172.21.15.67:0/1736415249 learned_addr learned my addr 172.21.15.67:0/1736415249 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:45.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.486+0000 7f6689e84640 1 --2- 172.21.15.67:0/1736415249 >> v2:172.21.15.72:3300/0 conn(0x7f668415ded0 0x7f66840b96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:45.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.486+0000 7f6688e82640 1 -- 172.21.15.67:0/1736415249 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1254946230 0 0) 0x7f66840d2190 con 0x7f668415e2b0 2024-03-24T00:39:45.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.486+0000 7f6688e82640 1 -- 172.21.15.67:0/1736415249 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f666c003520 con 0x7f668415e2b0 2024-03-24T00:39:45.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.486+0000 7f6689e84640 1 -- 172.21.15.67:0/1736415249 >> v1:172.21.15.72:6789/0 conn(0x7f66840cf230 legacy=0x7f66840d2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:45.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.486+0000 7f6689e84640 1 -- 172.21.15.67:0/1736415249 >> v1:172.21.15.67:6789/0 conn(0x7f668415e2b0 legacy=0x7f66840ceb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:45.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.486+0000 7f6688e82640 1 -- 172.21.15.67:0/1736415249 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3244864124 0 0) 0x7f66841620f0 con 0x7f66840cf230 2024-03-24T00:39:45.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.486+0000 7f6689e84640 1 -- 172.21.15.67:0/1736415249 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66841626b0 con 0x7f668415ded0 2024-03-24T00:39:45.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.487+0000 7f6689e84640 1 --2- 172.21.15.67:0/1736415249 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f668415ded0 0x7f66840b96e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f667c0097a0 tx=0x7f667c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=efd4a193a228f747 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:45.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.487+0000 7f6688e82640 1 -- 172.21.15.67:0/1736415249 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f667c034070 con 0x7f668415ded0 2024-03-24T00:39:45.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.487+0000 7f6688e82640 1 -- 172.21.15.67:0/1736415249 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f667c030610 con 0x7f668415ded0 2024-03-24T00:39:45.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.487+0000 7f6688e82640 1 -- 172.21.15.67:0/1736415249 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f667c0309c0 con 0x7f668415ded0 2024-03-24T00:39:45.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.488+0000 7f668c0e8640 1 -- 172.21.15.67:0/1736415249 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f668415ded0 msgr2=0x7f66840b96e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:45.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.488+0000 7f668c0e8640 1 --2- 172.21.15.67:0/1736415249 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f668415ded0 0x7f66840b96e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f667c0097a0 tx=0x7f667c02b960 comp rx=0 tx=0).stop 2024-03-24T00:39:45.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.488+0000 7f668c0e8640 1 -- 172.21.15.67:0/1736415249 shutdown_connections 2024-03-24T00:39:45.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.488+0000 7f668c0e8640 1 --2- 172.21.15.67:0/1736415249 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f668415ded0 0x7f66840b96e0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.488+0000 7f668c0e8640 1 -- 172.21.15.67:0/1736415249 >> 172.21.15.67:0/1736415249 conn(0x7f668406c860 msgr2=0x7f668406cc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:45.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.488+0000 7f668c0e8640 1 -- 172.21.15.67:0/1736415249 shutdown_connections 2024-03-24T00:39:45.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.488+0000 7f668c0e8640 1 -- 172.21.15.67:0/1736415249 wait complete. 2024-03-24T00:39:45.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.489+0000 7f668c0e8640 1 Processor -- start 2024-03-24T00:39:45.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.489+0000 7f668c0e8640 1 -- start start 2024-03-24T00:39:45.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.490+0000 7f668c0e8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66840cf230 0x7f668414f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:45.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.490+0000 7f668c0e8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f668415ded0 0x7f66841606a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:45.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.490+0000 7f668c0e8640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f668415e2b0 0x7f6684160be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:45.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.490+0000 7f668c0e8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6684161bc0 con 0x7f66840cf230 2024-03-24T00:39:45.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.490+0000 7f668c0e8640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66840b9c20 con 0x7f668415ded0 2024-03-24T00:39:45.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.490+0000 7f668c0e8640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6684161d40 con 0x7f668415e2b0 2024-03-24T00:39:45.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.490+0000 7f6689e84640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66840cf230 0x7f668414f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:45.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.490+0000 7f6689e84640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66840cf230 0x7f668414f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49832/0 (socket says 172.21.15.67:49832) 2024-03-24T00:39:45.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.490+0000 7f6689e84640 1 -- 172.21.15.67:0/3045556769 learned_addr learned my addr 172.21.15.67:0/3045556769 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:45.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.490+0000 7f6689683640 1 --2- 172.21.15.67:0/3045556769 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f668415ded0 0x7f66841606a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:45.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.490+0000 7f668a685640 1 --2- 172.21.15.67:0/3045556769 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f668415e2b0 0x7f6684160be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:45.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.490+0000 7f6689e84640 1 -- 172.21.15.67:0/3045556769 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f668415e2b0 msgr2=0x7f6684160be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:45.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.490+0000 7f6689e84640 1 --2- 172.21.15.67:0/3045556769 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f668415e2b0 0x7f6684160be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.490+0000 7f6689e84640 1 -- 172.21.15.67:0/3045556769 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f668415ded0 msgr2=0x7f66841606a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:45.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.490+0000 7f6689e84640 1 --2- 172.21.15.67:0/3045556769 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f668415ded0 0x7f66841606a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.490+0000 7f6689e84640 1 -- 172.21.15.67:0/3045556769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f668414f6e0 con 0x7f66840cf230 2024-03-24T00:39:45.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.490+0000 7f6689e84640 1 --2- 172.21.15.67:0/3045556769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66840cf230 0x7f668414f1a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f667c02b9a0 tx=0x7f667c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:45.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.491+0000 7f66827fc640 1 -- 172.21.15.67:0/3045556769 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f667c034040 con 0x7f66840cf230 2024-03-24T00:39:45.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.491+0000 7f66827fc640 1 -- 172.21.15.67:0/3045556769 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f667c030410 con 0x7f66840cf230 2024-03-24T00:39:45.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.491+0000 7f668c0e8640 1 -- 172.21.15.67:0/3045556769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66840b4880 con 0x7f66840cf230 2024-03-24T00:39:45.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.491+0000 7f66827fc640 1 -- 172.21.15.67:0/3045556769 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f667c039c70 con 0x7f66840cf230 2024-03-24T00:39:45.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.491+0000 7f668c0e8640 1 -- 172.21.15.67:0/3045556769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66840b4e40 con 0x7f66840cf230 2024-03-24T00:39:45.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.492+0000 7f668c0e8640 1 -- 172.21.15.67:0/3045556769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66840b5770 con 0x7f66840cf230 2024-03-24T00:39:45.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.492+0000 7f66827fc640 1 -- 172.21.15.67:0/3045556769 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f667c037070 con 0x7f66840cf230 2024-03-24T00:39:45.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.493+0000 7f66827fc640 1 --2- 172.21.15.67:0/3045556769 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f667006bbd0 0x7f667006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:45.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.493+0000 7f66827fc640 1 -- 172.21.15.67:0/3045556769 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f667c0accd0 con 0x7f66840cf230 2024-03-24T00:39:45.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.493+0000 7f6689683640 1 --2- 172.21.15.67:0/3045556769 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f667006bbd0 0x7f667006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:45.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.493+0000 7f6689683640 1 --2- 172.21.15.67:0/3045556769 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f667006bbd0 0x7f667006e090 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f66840b61f0 tx=0x7f6678000bc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:45.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.497+0000 7f66827fc640 1 -- 172.21.15.67:0/3045556769 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f668414f6e0 con 0x7f66840cf230 2024-03-24T00:39:45.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.695+0000 7f668c0e8640 1 -- 172.21.15.67:0/3045556769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f668404f400 con 0x7f66840cf230 2024-03-24T00:39:45.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.695+0000 7f66827fc640 1 -- 172.21.15.67:0/3045556769 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7f667c07ad70 con 0x7f66840cf230 2024-03-24T00:39:45.696 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-03-24T00:39:45.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.698+0000 7f668c0e8640 1 -- 172.21.15.67:0/3045556769 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f667006bbd0 msgr2=0x7f667006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:45.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.698+0000 7f668c0e8640 1 --2- 172.21.15.67:0/3045556769 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f667006bbd0 0x7f667006e090 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f66840b61f0 tx=0x7f6678000bc0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.698+0000 7f668c0e8640 1 -- 172.21.15.67:0/3045556769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66840cf230 msgr2=0x7f668414f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:45.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.698+0000 7f668c0e8640 1 --2- 172.21.15.67:0/3045556769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66840cf230 0x7f668414f1a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f667c02b9a0 tx=0x7f667c0046c0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.698+0000 7f668c0e8640 1 -- 172.21.15.67:0/3045556769 shutdown_connections 2024-03-24T00:39:45.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.698+0000 7f668c0e8640 1 --2- 172.21.15.67:0/3045556769 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f667006bbd0 0x7f667006e090 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.698+0000 7f668c0e8640 1 --2- 172.21.15.67:0/3045556769 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f668415e2b0 0x7f6684160be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.698+0000 7f668c0e8640 1 --2- 172.21.15.67:0/3045556769 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f668415ded0 0x7f66841606a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.698+0000 7f668c0e8640 1 --2- 172.21.15.67:0/3045556769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66840cf230 0x7f668414f1a0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.698+0000 7f668c0e8640 1 -- 172.21.15.67:0/3045556769 >> 172.21.15.67:0/3045556769 conn(0x7f668406c860 msgr2=0x7f66840b8840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:45.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.698+0000 7f668c0e8640 1 -- 172.21.15.67:0/3045556769 shutdown_connections 2024-03-24T00:39:45.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.698+0000 7f668c0e8640 1 -- 172.21.15.67:0/3045556769 wait complete. 2024-03-24T00:39:45.709 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2024-03-24T00:39:45.709 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:45.710 DEBUG:teuthology.orchestra.run.smithi072:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2024-03-24T00:39:45.738 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi072.front.sepia.ceph.com 2024-03-24T00:39:45.738 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:45.738 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-24T00:39:45.793 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-24T00:39:45.793 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:39:45.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.894+0000 7fd9cb7e6640 1 Processor -- start 2024-03-24T00:39:45.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.894+0000 7fd9cb7e6640 1 -- start start 2024-03-24T00:39:45.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.895+0000 7fd9cb7e6640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd9c406b040 0x7fd9c40b2170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:45.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.895+0000 7fd9cb7e6640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd9c406b420 0x7fd9c40cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:45.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.895+0000 7fd9cb7e6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd9c40d0b80 0x7fd9c40d1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:45.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.895+0000 7fd9cb7e6640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9c40ce660 con 0x7fd9c406b420 2024-03-24T00:39:45.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.895+0000 7fd9cb7e6640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9c415c230 con 0x7fd9c406b040 2024-03-24T00:39:45.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.895+0000 7fd9cb7e6640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c40cddc0 con 0x7fd9c40d0b80 2024-03-24T00:39:45.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.895+0000 7fd9c8d81640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd9c406b420 0x7fd9c40cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:45.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.895+0000 7fd9c8d81640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd9c406b420 0x7fd9c40cd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49842/0 (socket says 172.21.15.67:49842) 2024-03-24T00:39:45.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.895+0000 7fd9c8d81640 1 -- 172.21.15.67:0/3192175932 learned_addr learned my addr 172.21.15.67:0/3192175932 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:45.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.895+0000 7fd9c9582640 1 --2- 172.21.15.67:0/3192175932 >> v2:172.21.15.72:3300/0 conn(0x7fd9c406b040 0x7fd9c40b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:45.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.895+0000 7fd9c9d83640 1 --2- 172.21.15.67:0/3192175932 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd9c40d0b80 0x7fd9c40d1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:45.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.896+0000 7fd9c8d81640 1 -- 172.21.15.67:0/3192175932 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd9c40d0b80 msgr2=0x7fd9c40d1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:45.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.896+0000 7fd9c8d81640 1 --2- 172.21.15.67:0/3192175932 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd9c40d0b80 0x7fd9c40d1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.896+0000 7fd9c8d81640 1 -- 172.21.15.67:0/3192175932 >> v2:172.21.15.72:3300/0 conn(0x7fd9c406b040 msgr2=0x7fd9c40b2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:45.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.896+0000 7fd9c8d81640 1 --2- 172.21.15.67:0/3192175932 >> v2:172.21.15.72:3300/0 conn(0x7fd9c406b040 0x7fd9c40b2170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.896+0000 7fd9c8d81640 1 -- 172.21.15.67:0/3192175932 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9c40ce0a0 con 0x7fd9c406b420 2024-03-24T00:39:45.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.896+0000 7fd9c8d81640 1 --2- 172.21.15.67:0/3192175932 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9c406b420 0x7fd9c40cd460 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fd9c40d1cf0 tx=0x7fd9b402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f9681bda5e723a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:45.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.897+0000 7fd9bbfff640 1 -- 172.21.15.67:0/3192175932 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b402e070 con 0x7fd9c406b420 2024-03-24T00:39:45.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.897+0000 7fd9bbfff640 1 -- 172.21.15.67:0/3192175932 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9b4037070 con 0x7fd9c406b420 2024-03-24T00:39:45.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.897+0000 7fd9bbfff640 1 -- 172.21.15.67:0/3192175932 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b4031a00 con 0x7fd9c406b420 2024-03-24T00:39:45.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.897+0000 7fd9cb7e6640 1 -- 172.21.15.67:0/3192175932 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9c406b420 msgr2=0x7fd9c40cd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:45.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.897+0000 7fd9cb7e6640 1 --2- 172.21.15.67:0/3192175932 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9c406b420 0x7fd9c40cd460 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fd9c40d1cf0 tx=0x7fd9b402b880 comp rx=0 tx=0).stop 2024-03-24T00:39:45.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.897+0000 7fd9cb7e6640 1 -- 172.21.15.67:0/3192175932 shutdown_connections 2024-03-24T00:39:45.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.897+0000 7fd9cb7e6640 1 --2- 172.21.15.67:0/3192175932 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd9c40d0b80 0x7fd9c40d1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.897+0000 7fd9cb7e6640 1 --2- 172.21.15.67:0/3192175932 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9c406b420 0x7fd9c40cd460 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.897+0000 7fd9cb7e6640 1 --2- 172.21.15.67:0/3192175932 >> v2:172.21.15.72:3300/0 conn(0x7fd9c406b040 0x7fd9c40b2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.897+0000 7fd9cb7e6640 1 -- 172.21.15.67:0/3192175932 >> 172.21.15.67:0/3192175932 conn(0x7fd9c4099a30 msgr2=0x7fd9c4099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:45.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.897+0000 7fd9cb7e6640 1 -- 172.21.15.67:0/3192175932 shutdown_connections 2024-03-24T00:39:45.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.897+0000 7fd9cb7e6640 1 -- 172.21.15.67:0/3192175932 wait complete. 2024-03-24T00:39:45.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.898+0000 7fd9cb7e6640 1 Processor -- start 2024-03-24T00:39:45.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.898+0000 7fd9cb7e6640 1 -- start start 2024-03-24T00:39:45.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.899+0000 7fd9cb7e6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9c406b040 0x7fd9c40c70b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:45.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.899+0000 7fd9cb7e6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd9c406b420 0x7fd9c40c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:45.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.899+0000 7fd9cb7e6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9c40d0b80 0x7fd9c40c4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:45.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.899+0000 7fd9cb7e6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c406b8a0 con 0x7fd9c40d0b80 2024-03-24T00:39:45.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.899+0000 7fd9cb7e6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c4142be0 con 0x7fd9c406b040 2024-03-24T00:39:45.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.899+0000 7fd9cb7e6640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c40cd9a0 con 0x7fd9c406b420 2024-03-24T00:39:45.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.899+0000 7fd9c9d83640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9c40d0b80 0x7fd9c40c4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:45.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.899+0000 7fd9c9d83640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9c40d0b80 0x7fd9c40c4ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49844/0 (socket says 172.21.15.67:49844) 2024-03-24T00:39:45.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.899+0000 7fd9c9d83640 1 -- 172.21.15.67:0/575548482 learned_addr learned my addr 172.21.15.67:0/575548482 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:45.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.899+0000 7fd9c9582640 1 --2- 172.21.15.67:0/575548482 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9c406b040 0x7fd9c40c70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:45.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.899+0000 7fd9c8d81640 1 --2- 172.21.15.67:0/575548482 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd9c406b420 0x7fd9c40c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:45.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.899+0000 7fd9c9d83640 1 -- 172.21.15.67:0/575548482 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd9c406b420 msgr2=0x7fd9c40c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:45.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.899+0000 7fd9c9d83640 1 --2- 172.21.15.67:0/575548482 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd9c406b420 0x7fd9c40c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.899+0000 7fd9c9d83640 1 -- 172.21.15.67:0/575548482 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9c406b040 msgr2=0x7fd9c40c70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:45.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.899+0000 7fd9c9d83640 1 --2- 172.21.15.67:0/575548482 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9c406b040 0x7fd9c40c70b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:45.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.899+0000 7fd9c9d83640 1 -- 172.21.15.67:0/575548482 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9c40b0170 con 0x7fd9c40d0b80 2024-03-24T00:39:45.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.900+0000 7fd9c9d83640 1 --2- 172.21.15.67:0/575548482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9c40d0b80 0x7fd9c40c4ee0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd9c000efc0 tx=0x7fd9c000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:45.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.900+0000 7fd9b9ffb640 1 -- 172.21.15.67:0/575548482 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c0018020 con 0x7fd9c40d0b80 2024-03-24T00:39:45.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.900+0000 7fd9b9ffb640 1 -- 172.21.15.67:0/575548482 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9c0014480 con 0x7fd9c40d0b80 2024-03-24T00:39:45.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.900+0000 7fd9cb7e6640 1 -- 172.21.15.67:0/575548482 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9c40af8d0 con 0x7fd9c40d0b80 2024-03-24T00:39:45.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.900+0000 7fd9b9ffb640 1 -- 172.21.15.67:0/575548482 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c0014870 con 0x7fd9c40d0b80 2024-03-24T00:39:45.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.900+0000 7fd9cb7e6640 1 -- 172.21.15.67:0/575548482 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9c40afe90 con 0x7fd9c40d0b80 2024-03-24T00:39:45.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.901+0000 7fd9cb7e6640 1 -- 172.21.15.67:0/575548482 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9c404f400 con 0x7fd9c40d0b80 2024-03-24T00:39:45.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.901+0000 7fd9b9ffb640 1 -- 172.21.15.67:0/575548482 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd9c0005d10 con 0x7fd9c40d0b80 2024-03-24T00:39:45.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.902+0000 7fd9b9ffb640 1 --2- 172.21.15.67:0/575548482 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd9a806bbd0 0x7fd9a806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:45.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.902+0000 7fd9b9ffb640 1 -- 172.21.15.67:0/575548482 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fd9c008b760 con 0x7fd9c40d0b80 2024-03-24T00:39:45.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.902+0000 7fd9c9582640 1 --2- 172.21.15.67:0/575548482 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd9a806bbd0 0x7fd9a806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:45.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.902+0000 7fd9c9582640 1 --2- 172.21.15.67:0/575548482 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd9a806bbd0 0x7fd9a806e090 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc009ac0 tx=0x7fd9bc009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:45.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:45.906+0000 7fd9b9ffb640 1 -- 172.21.15.67:0/575548482 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd9c0022070 con 0x7fd9c40d0b80 2024-03-24T00:39:45.926 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:39:45.928 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:39:46.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.130+0000 7fd9cb7e6640 1 -- 172.21.15.67:0/575548482 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd9c40c4310 con 0x7fd9c40d0b80 2024-03-24T00:39:46.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.131+0000 7fd9b9ffb640 1 -- 172.21.15.67:0/575548482 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4970 (secure 0 0 0) 0x7fd9c0010040 con 0x7fd9c40d0b80 2024-03-24T00:39:46.132 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:39:46.132 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4263,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/837528243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":837528243},{"type":"v1","addr":"172.21.15.67:6834","nonce":837528243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3987071456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":3987071456},{"type":"v1","addr":"172.21.15.67:6835","nonce":3987071456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4306,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/3953492281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3953492281},{"type":"v1","addr":"172.21.15.72:6837","nonce":3953492281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:38:52.127871+0000","modified":"2024-03-24T00:39:08.187262+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4309},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4309":{"gid":4309,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6836/3737698654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3737698654},{"type":"v1","addr":"172.21.15.72:6836","nonce":3737698654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4309,"qdb_cluster":[4309]},"id":1}]} 2024-03-24T00:39:46.132 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 9 2024-03-24T00:39:46.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.133+0000 7fd9cb7e6640 1 -- 172.21.15.67:0/575548482 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd9a806bbd0 msgr2=0x7fd9a806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:46.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.133+0000 7fd9cb7e6640 1 --2- 172.21.15.67:0/575548482 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd9a806bbd0 0x7fd9a806e090 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc009ac0 tx=0x7fd9bc009190 comp rx=0 tx=0).stop 2024-03-24T00:39:46.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.133+0000 7fd9cb7e6640 1 -- 172.21.15.67:0/575548482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9c40d0b80 msgr2=0x7fd9c40c4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:46.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.133+0000 7fd9cb7e6640 1 --2- 172.21.15.67:0/575548482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9c40d0b80 0x7fd9c40c4ee0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd9c000efc0 tx=0x7fd9c000c490 comp rx=0 tx=0).stop 2024-03-24T00:39:46.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.133+0000 7fd9cb7e6640 1 -- 172.21.15.67:0/575548482 shutdown_connections 2024-03-24T00:39:46.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.133+0000 7fd9cb7e6640 1 --2- 172.21.15.67:0/575548482 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd9a806bbd0 0x7fd9a806e090 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:46.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.133+0000 7fd9cb7e6640 1 --2- 172.21.15.67:0/575548482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9c40d0b80 0x7fd9c40c4ee0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:46.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.133+0000 7fd9cb7e6640 1 --2- 172.21.15.67:0/575548482 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd9c406b420 0x7fd9c40c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:46.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.133+0000 7fd9cb7e6640 1 --2- 172.21.15.67:0/575548482 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9c406b040 0x7fd9c40c70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:46.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.133+0000 7fd9cb7e6640 1 -- 172.21.15.67:0/575548482 >> 172.21.15.67:0/575548482 conn(0x7fd9c4099a30 msgr2=0x7fd9c4143000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:46.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.133+0000 7fd9cb7e6640 1 -- 172.21.15.67:0/575548482 shutdown_connections 2024-03-24T00:39:46.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.133+0000 7fd9cb7e6640 1 -- 172.21.15.67:0/575548482 wait complete. 2024-03-24T00:39:46.145 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:39:46.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.245+0000 7efcba344640 1 Processor -- start 2024-03-24T00:39:46.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.245+0000 7efcba344640 1 -- start start 2024-03-24T00:39:46.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.246+0000 7efcba344640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7efcb4161bb0 0x7efcb40d2d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:46.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.246+0000 7efcba344640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcb4162120 con 0x7efcb40cf220 2024-03-24T00:39:46.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.246+0000 7efcba344640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcb4162400 con 0x7efcb415dfe0 2024-03-24T00:39:46.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.246+0000 7efcba344640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcb40d21d0 con 0x7efcb4161bb0 2024-03-24T00:39:46.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.246+0000 7efcb37fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7efcb40cf220 0x7efcb40d2660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50454/0 (socket says 172.21.15.67:50454) 2024-03-24T00:39:46.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.246+0000 7efcb37fe640 1 -- 172.21.15.67:0/2461048884 learned_addr learned my addr 172.21.15.67:0/2461048884 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:46.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.246+0000 7efcb88e1640 1 --2- 172.21.15.67:0/2461048884 >> v2:172.21.15.72:3300/0 conn(0x7efcb4161bb0 0x7efcb40d2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:46.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.246+0000 7efcb2ffd640 1 -- 172.21.15.67:0/2461048884 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2904731986 0 0) 0x7efcb4162120 con 0x7efcb40cf220 2024-03-24T00:39:46.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.246+0000 7efcb2ffd640 1 -- 172.21.15.67:0/2461048884 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efca0003520 con 0x7efcb40cf220 2024-03-24T00:39:46.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.247+0000 7efcb2ffd640 1 -- 172.21.15.67:0/2461048884 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1718931962 0 0) 0x7efcb4162400 con 0x7efcb415dfe0 2024-03-24T00:39:46.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.247+0000 7efcb2ffd640 1 -- 172.21.15.67:0/2461048884 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcb4162120 con 0x7efcb415dfe0 2024-03-24T00:39:46.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.247+0000 7efcb2ffd640 1 -- 172.21.15.67:0/2461048884 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3987486930 0 0) 0x7efca0003520 con 0x7efcb40cf220 2024-03-24T00:39:46.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.247+0000 7efcb88e1640 1 -- 172.21.15.67:0/2461048884 >> v1:172.21.15.72:6789/0 conn(0x7efcb415dfe0 legacy=0x7efcb40ceb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:46.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.247+0000 7efcb88e1640 1 -- 172.21.15.67:0/2461048884 >> v1:172.21.15.67:6789/0 conn(0x7efcb40cf220 legacy=0x7efcb40d2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:46.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.247+0000 7efcb88e1640 1 -- 172.21.15.67:0/2461048884 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb41626e0 con 0x7efcb4161bb0 2024-03-24T00:39:46.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.247+0000 7efcb88e1640 1 --2- 172.21.15.67:0/2461048884 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcb4161bb0 0x7efcb40d2d70 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7efcac009d90 tx=0x7efcac02be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c20f8d7f8dea7ce7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:46.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.248+0000 7efcb2ffd640 1 -- 172.21.15.67:0/2461048884 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcac039070 con 0x7efcb4161bb0 2024-03-24T00:39:46.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.248+0000 7efcb2ffd640 1 -- 172.21.15.67:0/2461048884 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcac02c940 con 0x7efcb4161bb0 2024-03-24T00:39:46.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.248+0000 7efcb2ffd640 1 -- 172.21.15.67:0/2461048884 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca4001670 con 0x7efcb4161bb0 2024-03-24T00:39:46.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.248+0000 7efcba344640 1 -- 172.21.15.67:0/2461048884 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcb4161bb0 msgr2=0x7efcb40d2d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:46.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.248+0000 7efcba344640 1 --2- 172.21.15.67:0/2461048884 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcb4161bb0 0x7efcb40d2d70 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7efcac009d90 tx=0x7efcac02be10 comp rx=0 tx=0).stop 2024-03-24T00:39:46.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.248+0000 7efcba344640 1 -- 172.21.15.67:0/2461048884 shutdown_connections 2024-03-24T00:39:46.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.248+0000 7efcba344640 1 --2- 172.21.15.67:0/2461048884 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcb4161bb0 0x7efcb40d2d70 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:46.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.248+0000 7efcba344640 1 -- 172.21.15.67:0/2461048884 >> 172.21.15.67:0/2461048884 conn(0x7efcb406c890 msgr2=0x7efcb406cc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:46.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.248+0000 7efcba344640 1 -- 172.21.15.67:0/2461048884 shutdown_connections 2024-03-24T00:39:46.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.248+0000 7efcba344640 1 -- 172.21.15.67:0/2461048884 wait complete. 2024-03-24T00:39:46.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.249+0000 7efcba344640 1 Processor -- start 2024-03-24T00:39:46.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.250+0000 7efcba344640 1 -- start start 2024-03-24T00:39:46.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.250+0000 7efcba344640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efcb40cf220 0x7efcb4151690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:46.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.250+0000 7efcba344640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcb415dfe0 0x7efcb40cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:46.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.250+0000 7efcba344640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcb4161bb0 0x7efcb40c99c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:46.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.250+0000 7efcba344640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb40d1120 con 0x7efcb4161bb0 2024-03-24T00:39:46.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.250+0000 7efcba344640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb415e3c0 con 0x7efcb415dfe0 2024-03-24T00:39:46.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.250+0000 7efcba344640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcb40d12a0 con 0x7efcb40cf220 2024-03-24T00:39:46.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.250+0000 7efcb88e1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcb4161bb0 0x7efcb40c99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:46.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.251+0000 7efcb3fff640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efcb40cf220 0x7efcb4151690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:46.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.251+0000 7efcb88e1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcb4161bb0 0x7efcb40c99c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49858/0 (socket says 172.21.15.67:49858) 2024-03-24T00:39:46.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.251+0000 7efcb37fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcb415dfe0 0x7efcb40cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:46.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.251+0000 7efcb88e1640 1 -- 172.21.15.67:0/2146297464 learned_addr learned my addr 172.21.15.67:0/2146297464 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:46.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.251+0000 7efcb88e1640 1 -- 172.21.15.67:0/2146297464 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efcb40cf220 msgr2=0x7efcb4151690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:46.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.251+0000 7efcb88e1640 1 --2- 172.21.15.67:0/2146297464 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efcb40cf220 0x7efcb4151690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:46.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.251+0000 7efcb88e1640 1 -- 172.21.15.67:0/2146297464 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcb415dfe0 msgr2=0x7efcb40cae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:46.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.251+0000 7efcb88e1640 1 --2- 172.21.15.67:0/2146297464 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcb415dfe0 0x7efcb40cae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:46.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.251+0000 7efcb88e1640 1 -- 172.21.15.67:0/2146297464 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb415f280 con 0x7efcb4161bb0 2024-03-24T00:39:46.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.251+0000 7efcb88e1640 1 --2- 172.21.15.67:0/2146297464 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcb4161bb0 0x7efcb40c99c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7efcac009d90 tx=0x7efcac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:46.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.251+0000 7efcb0ff9640 1 -- 172.21.15.67:0/2146297464 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcac039050 con 0x7efcb4161bb0 2024-03-24T00:39:46.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.251+0000 7efcb0ff9640 1 -- 172.21.15.67:0/2146297464 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcac03a040 con 0x7efcb4161bb0 2024-03-24T00:39:46.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.251+0000 7efcba344640 1 -- 172.21.15.67:0/2146297464 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcb40b4e70 con 0x7efcb4161bb0 2024-03-24T00:39:46.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.251+0000 7efcb0ff9640 1 -- 172.21.15.67:0/2146297464 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcac009450 con 0x7efcb4161bb0 2024-03-24T00:39:46.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.251+0000 7efcba344640 1 -- 172.21.15.67:0/2146297464 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcb40b5430 con 0x7efcb4161bb0 2024-03-24T00:39:46.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.252+0000 7efcba344640 1 -- 172.21.15.67:0/2146297464 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcb406b180 con 0x7efcb4161bb0 2024-03-24T00:39:46.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.253+0000 7efcb0ff9640 1 -- 172.21.15.67:0/2146297464 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7efcac047020 con 0x7efcb4161bb0 2024-03-24T00:39:46.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.253+0000 7efcb0ff9640 1 --2- 172.21.15.67:0/2146297464 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7efc9406bbd0 0x7efc9406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:46.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.253+0000 7efcb0ff9640 1 -- 172.21.15.67:0/2146297464 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7efcac0acb30 con 0x7efcb4161bb0 2024-03-24T00:39:46.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.253+0000 7efcb3fff640 1 --2- 172.21.15.67:0/2146297464 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7efc9406bbd0 0x7efc9406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:46.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.254+0000 7efcb3fff640 1 --2- 172.21.15.67:0/2146297464 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7efc9406bbd0 0x7efc9406e090 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7efc9c0011e0 tx=0x7efc9c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:46.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.258+0000 7efcb0ff9640 1 -- 172.21.15.67:0/2146297464 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7efcac04e020 con 0x7efcb4161bb0 2024-03-24T00:39:46.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.441+0000 7efcba344640 1 -- 172.21.15.67:0/2146297464 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efcb404f400 con 0x7efcb4161bb0 2024-03-24T00:39:46.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.442+0000 7efcb0ff9640 1 -- 172.21.15.67:0/2146297464 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18843 (secure 0 0 0) 0x7efcb415f280 con 0x7efcb4161bb0 2024-03-24T00:39:46.443 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:39:46.443 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":27,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:39:32.904729+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:38:50.077085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:38:51.095454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T00:38:54.807117+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.14","mappings":[{"from":4,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:39:46.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.444+0000 7efcba344640 1 -- 172.21.15.67:0/2146297464 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7efc9406bbd0 msgr2=0x7efc9406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:46.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.444+0000 7efcba344640 1 --2- 172.21.15.67:0/2146297464 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7efc9406bbd0 0x7efc9406e090 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7efc9c0011e0 tx=0x7efc9c009040 comp rx=0 tx=0).stop 2024-03-24T00:39:46.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.444+0000 7efcba344640 1 -- 172.21.15.67:0/2146297464 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcb4161bb0 msgr2=0x7efcb40c99c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:46.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.444+0000 7efcba344640 1 --2- 172.21.15.67:0/2146297464 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcb4161bb0 0x7efcb40c99c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7efcac009d90 tx=0x7efcac004a40 comp rx=0 tx=0).stop 2024-03-24T00:39:46.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.445+0000 7efcba344640 1 -- 172.21.15.67:0/2146297464 shutdown_connections 2024-03-24T00:39:46.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.445+0000 7efcba344640 1 --2- 172.21.15.67:0/2146297464 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7efc9406bbd0 0x7efc9406e090 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:46.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.445+0000 7efcba344640 1 --2- 172.21.15.67:0/2146297464 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcb4161bb0 0x7efcb40c99c0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:46.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.445+0000 7efcba344640 1 --2- 172.21.15.67:0/2146297464 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcb415dfe0 0x7efcb40cae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:46.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.445+0000 7efcba344640 1 --2- 172.21.15.67:0/2146297464 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efcb40cf220 0x7efcb4151690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:46.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.445+0000 7efcba344640 1 -- 172.21.15.67:0/2146297464 >> 172.21.15.67:0/2146297464 conn(0x7efcb406c890 msgr2=0x7efcb40c7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:46.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.445+0000 7efcba344640 1 -- 172.21.15.67:0/2146297464 shutdown_connections 2024-03-24T00:39:46.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.445+0000 7efcba344640 1 -- 172.21.15.67:0/2146297464 wait complete. 2024-03-24T00:39:46.456 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:38:39.364950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:38:50.077085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:38:51.095454+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T00:38:54.807117+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T00:39:46.457 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:39:46.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.560+0000 7f5df240b640 1 Processor -- start 2024-03-24T00:39:46.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.560+0000 7f5df240b640 1 -- start start 2024-03-24T00:39:46.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.561+0000 7f5df240b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5dec0b3d30 0x7f5dec168e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:46.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.561+0000 7f5df240b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dec0cf090 con 0x7f5dec15dfd0 2024-03-24T00:39:46.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.561+0000 7f5df240b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dec0d0320 con 0x7f5dec0b41e0 2024-03-24T00:39:46.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.561+0000 7f5df240b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dec0cc260 con 0x7f5dec0b3d30 2024-03-24T00:39:46.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.561+0000 7f5debfff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5dec15dfd0 0x7f5dec0cf980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50462/0 (socket says 172.21.15.67:50462) 2024-03-24T00:39:46.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.561+0000 7f5debfff640 1 -- 172.21.15.67:0/713457357 learned_addr learned my addr 172.21.15.67:0/713457357 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:46.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.561+0000 7f5df09a8640 1 --2- 172.21.15.67:0/713457357 >> v2:172.21.15.72:3300/0 conn(0x7f5dec0b3d30 0x7f5dec168e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:46.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.562+0000 7f5deaffd640 1 -- 172.21.15.67:0/713457357 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 217134311 0 0) 0x7f5dec0cf090 con 0x7f5dec15dfd0 2024-03-24T00:39:46.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.562+0000 7f5deaffd640 1 -- 172.21.15.67:0/713457357 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dd4003520 con 0x7f5dec15dfd0 2024-03-24T00:39:46.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.562+0000 7f5deaffd640 1 -- 172.21.15.67:0/713457357 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 184680370 0 0) 0x7f5dec0d0320 con 0x7f5dec0b41e0 2024-03-24T00:39:46.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.562+0000 7f5deaffd640 1 -- 172.21.15.67:0/713457357 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dec0cf090 con 0x7f5dec0b41e0 2024-03-24T00:39:46.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.562+0000 7f5df09a8640 1 -- 172.21.15.67:0/713457357 >> v1:172.21.15.72:6789/0 conn(0x7f5dec0b41e0 legacy=0x7f5dec0b3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:46.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.562+0000 7f5df09a8640 1 -- 172.21.15.67:0/713457357 >> v1:172.21.15.67:6789/0 conn(0x7f5dec15dfd0 legacy=0x7f5dec0cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:46.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.562+0000 7f5df09a8640 1 -- 172.21.15.67:0/713457357 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dec0d0600 con 0x7f5dec0b3d30 2024-03-24T00:39:46.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.563+0000 7f5df09a8640 1 --2- 172.21.15.67:0/713457357 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5dec0b3d30 0x7f5dec168e40 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5de4009dc0 tx=0x7f5de402bfb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=41a02d156f6010fb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:46.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.563+0000 7f5deaffd640 1 -- 172.21.15.67:0/713457357 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de4039070 con 0x7f5dec0b3d30 2024-03-24T00:39:46.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.563+0000 7f5deaffd640 1 -- 172.21.15.67:0/713457357 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5de402cd60 con 0x7f5dec0b3d30 2024-03-24T00:39:46.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.563+0000 7f5deaffd640 1 -- 172.21.15.67:0/713457357 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de4034660 con 0x7f5dec0b3d30 2024-03-24T00:39:46.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.563+0000 7f5df240b640 1 -- 172.21.15.67:0/713457357 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5dec0b3d30 msgr2=0x7f5dec168e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:46.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.563+0000 7f5df240b640 1 --2- 172.21.15.67:0/713457357 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5dec0b3d30 0x7f5dec168e40 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5de4009dc0 tx=0x7f5de402bfb0 comp rx=0 tx=0).stop 2024-03-24T00:39:46.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.563+0000 7f5df240b640 1 -- 172.21.15.67:0/713457357 shutdown_connections 2024-03-24T00:39:46.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.563+0000 7f5df240b640 1 --2- 172.21.15.67:0/713457357 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5dec0b3d30 0x7f5dec168e40 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:46.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.563+0000 7f5df240b640 1 -- 172.21.15.67:0/713457357 >> 172.21.15.67:0/713457357 conn(0x7f5dec06fcf0 msgr2=0x7f5dec0700d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:46.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.563+0000 7f5df240b640 1 -- 172.21.15.67:0/713457357 shutdown_connections 2024-03-24T00:39:46.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.563+0000 7f5df240b640 1 -- 172.21.15.67:0/713457357 wait complete. 2024-03-24T00:39:46.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.564+0000 7f5df240b640 1 Processor -- start 2024-03-24T00:39:46.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.565+0000 7f5df240b640 1 -- start start 2024-03-24T00:39:46.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.565+0000 7f5df240b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5dec0b3d30 0x7f5dec0b73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:46.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.565+0000 7f5df240b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5dec0b41e0 0x7f5dec1e8090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:46.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.565+0000 7f5df240b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5dec15dfd0 0x7f5dec0cb290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:46.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.565+0000 7f5df240b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dec15e3b0 con 0x7f5dec0b3d30 2024-03-24T00:39:46.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.565+0000 7f5df240b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dec0cedb0 con 0x7f5dec15dfd0 2024-03-24T00:39:46.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.565+0000 7f5df240b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dec0cd6e0 con 0x7f5dec0b41e0 2024-03-24T00:39:46.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.565+0000 7f5debfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5dec0b3d30 0x7f5dec0b73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:46.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.566+0000 7f5debfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5dec0b3d30 0x7f5dec0b73f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49872/0 (socket says 172.21.15.67:49872) 2024-03-24T00:39:46.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.566+0000 7f5debfff640 1 -- 172.21.15.67:0/2856959061 learned_addr learned my addr 172.21.15.67:0/2856959061 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:46.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.566+0000 7f5df09a8640 1 --2- 172.21.15.67:0/2856959061 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5dec15dfd0 0x7f5dec0cb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:46.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.566+0000 7f5deb7fe640 1 --2- 172.21.15.67:0/2856959061 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5dec0b41e0 0x7f5dec1e8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:46.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.566+0000 7f5debfff640 1 -- 172.21.15.67:0/2856959061 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5dec0b41e0 msgr2=0x7f5dec1e8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:46.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.566+0000 7f5debfff640 1 --2- 172.21.15.67:0/2856959061 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5dec0b41e0 0x7f5dec1e8090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:46.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.566+0000 7f5debfff640 1 -- 172.21.15.67:0/2856959061 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5dec15dfd0 msgr2=0x7f5dec0cb290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:46.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.566+0000 7f5debfff640 1 --2- 172.21.15.67:0/2856959061 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5dec15dfd0 0x7f5dec0cb290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:46.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.566+0000 7f5debfff640 1 -- 172.21.15.67:0/2856959061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dec0b6c20 con 0x7f5dec0b3d30 2024-03-24T00:39:46.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.566+0000 7f5df09a8640 1 --2- 172.21.15.67:0/2856959061 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5dec15dfd0 0x7f5dec0cb290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:46.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.566+0000 7f5debfff640 1 --2- 172.21.15.67:0/2856959061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5dec0b3d30 0x7f5dec0b73f0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc00c440 tx=0x7f5ddc00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:46.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.566+0000 7f5de8ff9640 1 -- 172.21.15.67:0/2856959061 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ddc017070 con 0x7f5dec0b3d30 2024-03-24T00:39:46.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.566+0000 7f5de8ff9640 1 -- 172.21.15.67:0/2856959061 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ddc005ce0 con 0x7f5dec0b3d30 2024-03-24T00:39:46.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.566+0000 7f5df240b640 1 -- 172.21.15.67:0/2856959061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dec1e72d0 con 0x7f5dec0b3d30 2024-03-24T00:39:46.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.566+0000 7f5de8ff9640 1 -- 172.21.15.67:0/2856959061 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ddc007270 con 0x7f5dec0b3d30 2024-03-24T00:39:46.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.566+0000 7f5df240b640 1 -- 172.21.15.67:0/2856959061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dec1e7890 con 0x7f5dec0b3d30 2024-03-24T00:39:46.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.567+0000 7f5df240b640 1 -- 172.21.15.67:0/2856959061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dec04f400 con 0x7f5dec0b3d30 2024-03-24T00:39:46.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.568+0000 7f5de8ff9640 1 -- 172.21.15.67:0/2856959061 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5ddc01a070 con 0x7f5dec0b3d30 2024-03-24T00:39:46.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.568+0000 7f5de8ff9640 1 --2- 172.21.15.67:0/2856959061 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5dd806bc00 0x7f5dd806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:46.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.568+0000 7f5de8ff9640 1 -- 172.21.15.67:0/2856959061 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f5ddc08a040 con 0x7f5dec0b3d30 2024-03-24T00:39:46.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.568+0000 7f5deb7fe640 1 --2- 172.21.15.67:0/2856959061 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5dd806bc00 0x7f5dd806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:46.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.569+0000 7f5deb7fe640 1 --2- 172.21.15.67:0/2856959061 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5dd806bc00 0x7f5dd806e0c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5de0007fd0 tx=0x7f5de0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:46.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.572+0000 7f5de8ff9640 1 -- 172.21.15.67:0/2856959061 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5ddc0580e0 con 0x7f5dec0b3d30 2024-03-24T00:39:46.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.754+0000 7f5df240b640 1 -- 172.21.15.67:0/2856959061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5dec0b6df0 con 0x7f5dec0b3d30 2024-03-24T00:39:46.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.755+0000 7f5de8ff9640 1 -- 172.21.15.67:0/2856959061 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5ddc05c180 con 0x7f5dec0b3d30 2024-03-24T00:39:46.756 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-24T00:39:46.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.757+0000 7f5df240b640 1 -- 172.21.15.67:0/2856959061 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5dd806bc00 msgr2=0x7f5dd806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:46.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.757+0000 7f5df240b640 1 --2- 172.21.15.67:0/2856959061 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5dd806bc00 0x7f5dd806e0c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5de0007fd0 tx=0x7f5de0009040 comp rx=0 tx=0).stop 2024-03-24T00:39:46.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.757+0000 7f5df240b640 1 -- 172.21.15.67:0/2856959061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5dec0b3d30 msgr2=0x7f5dec0b73f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:46.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.757+0000 7f5df240b640 1 --2- 172.21.15.67:0/2856959061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5dec0b3d30 0x7f5dec0b73f0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc00c440 tx=0x7f5ddc00c900 comp rx=0 tx=0).stop 2024-03-24T00:39:46.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.757+0000 7f5df240b640 1 -- 172.21.15.67:0/2856959061 shutdown_connections 2024-03-24T00:39:46.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.757+0000 7f5df240b640 1 --2- 172.21.15.67:0/2856959061 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5dd806bc00 0x7f5dd806e0c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:46.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.757+0000 7f5df240b640 1 --2- 172.21.15.67:0/2856959061 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5dec15dfd0 0x7f5dec0cb290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:46.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.757+0000 7f5df240b640 1 --2- 172.21.15.67:0/2856959061 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5dec0b41e0 0x7f5dec1e8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:46.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.757+0000 7f5df240b640 1 --2- 172.21.15.67:0/2856959061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5dec0b3d30 0x7f5dec0b73f0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:46.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.757+0000 7f5df240b640 1 -- 172.21.15.67:0/2856959061 >> 172.21.15.67:0/2856959061 conn(0x7f5dec06fcf0 msgr2=0x7f5dec0b27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:46.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.757+0000 7f5df240b640 1 -- 172.21.15.67:0/2856959061 shutdown_connections 2024-03-24T00:39:46.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.757+0000 7f5df240b640 1 -- 172.21.15.67:0/2856959061 wait complete. 2024-03-24T00:39:46.768 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:39:46.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.872+0000 7fd2efcde640 1 Processor -- start 2024-03-24T00:39:46.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.872+0000 7fd2efcde640 1 -- start start 2024-03-24T00:39:46.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.873+0000 7fd2efcde640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd2e815dfd0 0x7fd2e80cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:46.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.873+0000 7fd2efcde640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2e80cf090 con 0x7fd2e80b3d30 2024-03-24T00:39:46.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.873+0000 7fd2efcde640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2e80d0320 con 0x7fd2e80b41e0 2024-03-24T00:39:46.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.873+0000 7fd2efcde640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2e80cc260 con 0x7fd2e815dfd0 2024-03-24T00:39:46.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.873+0000 7fd2ee27b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd2e80b3d30 0x7fd2e80cc6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50478/0 (socket says 172.21.15.67:50478) 2024-03-24T00:39:46.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.873+0000 7fd2ee27b640 1 -- 172.21.15.67:0/2563339584 learned_addr learned my addr 172.21.15.67:0/2563339584 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:46.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.873+0000 7fd2eda7a640 1 --2- 172.21.15.67:0/2563339584 >> v2:172.21.15.72:3300/0 conn(0x7fd2e815dfd0 0x7fd2e80cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:46.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.873+0000 7fd2eca78640 1 -- 172.21.15.67:0/2563339584 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 611242537 0 0) 0x7fd2e80cf090 con 0x7fd2e80b3d30 2024-03-24T00:39:46.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.873+0000 7fd2eca78640 1 -- 172.21.15.67:0/2563339584 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2d4003520 con 0x7fd2e80b3d30 2024-03-24T00:39:46.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.873+0000 7fd2eda7a640 1 -- 172.21.15.67:0/2563339584 >> v1:172.21.15.72:6789/0 conn(0x7fd2e80b41e0 legacy=0x7fd2e80b3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:46.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.873+0000 7fd2eda7a640 1 -- 172.21.15.67:0/2563339584 >> v1:172.21.15.67:6789/0 conn(0x7fd2e80b3d30 legacy=0x7fd2e80cc6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:46.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.873+0000 7fd2eda7a640 1 -- 172.21.15.67:0/2563339584 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e80d0600 con 0x7fd2e815dfd0 2024-03-24T00:39:46.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.874+0000 7fd2eda7a640 1 --2- 172.21.15.67:0/2563339584 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd2e815dfd0 0x7fd2e80cf980 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd2e40097d0 tx=0x7fd2e402b9d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef00d6a6841906ec server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:46.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.874+0000 7fd2eca78640 1 -- 172.21.15.67:0/2563339584 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e4034070 con 0x7fd2e815dfd0 2024-03-24T00:39:46.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.874+0000 7fd2eca78640 1 -- 172.21.15.67:0/2563339584 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2e4030890 con 0x7fd2e815dfd0 2024-03-24T00:39:46.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.874+0000 7fd2eca78640 1 -- 172.21.15.67:0/2563339584 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e4030c40 con 0x7fd2e815dfd0 2024-03-24T00:39:46.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.875+0000 7fd2efcde640 1 -- 172.21.15.67:0/2563339584 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd2e815dfd0 msgr2=0x7fd2e80cf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:46.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.875+0000 7fd2efcde640 1 --2- 172.21.15.67:0/2563339584 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd2e815dfd0 0x7fd2e80cf980 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd2e40097d0 tx=0x7fd2e402b9d0 comp rx=0 tx=0).stop 2024-03-24T00:39:46.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.875+0000 7fd2efcde640 1 -- 172.21.15.67:0/2563339584 shutdown_connections 2024-03-24T00:39:46.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.875+0000 7fd2efcde640 1 --2- 172.21.15.67:0/2563339584 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd2e815dfd0 0x7fd2e80cf980 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:46.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.875+0000 7fd2efcde640 1 -- 172.21.15.67:0/2563339584 >> 172.21.15.67:0/2563339584 conn(0x7fd2e806fcf0 msgr2=0x7fd2e80700d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:46.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.875+0000 7fd2efcde640 1 -- 172.21.15.67:0/2563339584 shutdown_connections 2024-03-24T00:39:46.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.875+0000 7fd2efcde640 1 -- 172.21.15.67:0/2563339584 wait complete. 2024-03-24T00:39:46.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.876+0000 7fd2efcde640 1 Processor -- start 2024-03-24T00:39:46.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.876+0000 7fd2efcde640 1 -- start start 2024-03-24T00:39:46.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.877+0000 7fd2efcde640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd2e80b3d30 0x7fd2e80cbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:46.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.877+0000 7fd2efcde640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2e80b41e0 0x7fd2e815eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:46.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.877+0000 7fd2efcde640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd2e815dfd0 0x7fd2e80c8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:46.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.877+0000 7fd2efcde640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e80cfec0 con 0x7fd2e80b41e0 2024-03-24T00:39:46.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.877+0000 7fd2efcde640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e80cedb0 con 0x7fd2e815dfd0 2024-03-24T00:39:46.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.877+0000 7fd2efcde640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e815e3b0 con 0x7fd2e80b3d30 2024-03-24T00:39:46.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.877+0000 7fd2ed279640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2e80b41e0 0x7fd2e815eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:46.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.877+0000 7fd2ed279640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2e80b41e0 0x7fd2e815eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49878/0 (socket says 172.21.15.67:49878) 2024-03-24T00:39:46.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.877+0000 7fd2ed279640 1 -- 172.21.15.67:0/4239916962 learned_addr learned my addr 172.21.15.67:0/4239916962 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:46.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.877+0000 7fd2ee27b640 1 --2- 172.21.15.67:0/4239916962 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd2e815dfd0 0x7fd2e80c8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:46.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.877+0000 7fd2eda7a640 1 --2- 172.21.15.67:0/4239916962 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd2e80b3d30 0x7fd2e80cbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:46.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.877+0000 7fd2ed279640 1 -- 172.21.15.67:0/4239916962 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd2e80b3d30 msgr2=0x7fd2e80cbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:46.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.877+0000 7fd2ed279640 1 --2- 172.21.15.67:0/4239916962 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd2e80b3d30 0x7fd2e80cbcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:46.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.877+0000 7fd2ed279640 1 -- 172.21.15.67:0/4239916962 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd2e815dfd0 msgr2=0x7fd2e80c8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:46.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.877+0000 7fd2ed279640 1 --2- 172.21.15.67:0/4239916962 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd2e815dfd0 0x7fd2e80c8ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:46.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.877+0000 7fd2ed279640 1 -- 172.21.15.67:0/4239916962 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e815f410 con 0x7fd2e80b41e0 2024-03-24T00:39:46.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.877+0000 7fd2ed279640 1 --2- 172.21.15.67:0/4239916962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2e80b41e0 0x7fd2e815eed0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fd2d800c440 tx=0x7fd2d8001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:46.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.877+0000 7fd2de7fc640 1 -- 172.21.15.67:0/4239916962 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d80052d0 con 0x7fd2e80b41e0 2024-03-24T00:39:46.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.878+0000 7fd2de7fc640 1 -- 172.21.15.67:0/4239916962 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2d8005470 con 0x7fd2e80b41e0 2024-03-24T00:39:46.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.878+0000 7fd2efcde640 1 -- 172.21.15.67:0/4239916962 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2e815f690 con 0x7fd2e80b41e0 2024-03-24T00:39:46.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.878+0000 7fd2de7fc640 1 -- 172.21.15.67:0/4239916962 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d8007260 con 0x7fd2e80b41e0 2024-03-24T00:39:46.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.878+0000 7fd2efcde640 1 -- 172.21.15.67:0/4239916962 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2e815fc50 con 0x7fd2e80b41e0 2024-03-24T00:39:46.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.879+0000 7fd2efcde640 1 -- 172.21.15.67:0/4239916962 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2e80d1810 con 0x7fd2e80b41e0 2024-03-24T00:39:46.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.879+0000 7fd2de7fc640 1 -- 172.21.15.67:0/4239916962 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd2d8007400 con 0x7fd2e80b41e0 2024-03-24T00:39:46.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.879+0000 7fd2de7fc640 1 --2- 172.21.15.67:0/4239916962 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd2cc06bc00 0x7fd2cc06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:46.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.879+0000 7fd2de7fc640 1 -- 172.21.15.67:0/4239916962 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fd2d808a260 con 0x7fd2e80b41e0 2024-03-24T00:39:46.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.880+0000 7fd2eda7a640 1 --2- 172.21.15.67:0/4239916962 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd2cc06bc00 0x7fd2cc06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:46.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.880+0000 7fd2eda7a640 1 --2- 172.21.15.67:0/4239916962 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd2cc06bc00 0x7fd2cc06e0c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd2e4036000 tx=0x7fd2e4008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:46.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:46.884+0000 7fd2de7fc640 1 -- 172.21.15.67:0/4239916962 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd2e80d1810 con 0x7fd2e80b41e0 2024-03-24T00:39:47.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.067+0000 7fd2efcde640 1 -- 172.21.15.67:0/4239916962 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd2e80c84e0 con 0x7fd2e80b41e0 2024-03-24T00:39:47.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.068+0000 7fd2de7fc640 1 -- 172.21.15.67:0/4239916962 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd2e815f410 con 0x7fd2e80b41e0 2024-03-24T00:39:47.069 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-24T00:39:47.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.069+0000 7fd2efcde640 1 -- 172.21.15.67:0/4239916962 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd2cc06bc00 msgr2=0x7fd2cc06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:47.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.069+0000 7fd2efcde640 1 --2- 172.21.15.67:0/4239916962 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd2cc06bc00 0x7fd2cc06e0c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd2e4036000 tx=0x7fd2e4008030 comp rx=0 tx=0).stop 2024-03-24T00:39:47.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.069+0000 7fd2efcde640 1 -- 172.21.15.67:0/4239916962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2e80b41e0 msgr2=0x7fd2e815eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:47.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.069+0000 7fd2efcde640 1 --2- 172.21.15.67:0/4239916962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2e80b41e0 0x7fd2e815eed0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fd2d800c440 tx=0x7fd2d8001e70 comp rx=0 tx=0).stop 2024-03-24T00:39:47.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.070+0000 7fd2efcde640 1 -- 172.21.15.67:0/4239916962 shutdown_connections 2024-03-24T00:39:47.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.070+0000 7fd2efcde640 1 --2- 172.21.15.67:0/4239916962 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd2cc06bc00 0x7fd2cc06e0c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:47.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.070+0000 7fd2efcde640 1 --2- 172.21.15.67:0/4239916962 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd2e815dfd0 0x7fd2e80c8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:47.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.070+0000 7fd2efcde640 1 --2- 172.21.15.67:0/4239916962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2e80b41e0 0x7fd2e815eed0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:47.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.070+0000 7fd2efcde640 1 --2- 172.21.15.67:0/4239916962 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd2e80b3d30 0x7fd2e80cbcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:47.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.070+0000 7fd2efcde640 1 -- 172.21.15.67:0/4239916962 >> 172.21.15.67:0/4239916962 conn(0x7fd2e806fcf0 msgr2=0x7fd2e815e660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:47.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.070+0000 7fd2efcde640 1 -- 172.21.15.67:0/4239916962 shutdown_connections 2024-03-24T00:39:47.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.070+0000 7fd2efcde640 1 -- 172.21.15.67:0/4239916962 wait complete. 2024-03-24T00:39:47.081 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:39:47.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.185+0000 7fd27ccdc640 1 Processor -- start 2024-03-24T00:39:47.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.185+0000 7fd27ccdc640 1 -- start start 2024-03-24T00:39:47.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.186+0000 7fd27ccdc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd2780d3ae0 0x7fd2780b5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:47.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.186+0000 7fd27ccdc640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd27815e280 0x7fd2780b5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:47.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.186+0000 7fd27ccdc640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2780d2250 con 0x7fd27815dea0 2024-03-24T00:39:47.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.186+0000 7fd27ccdc640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2780d2530 con 0x7fd2780d3ae0 2024-03-24T00:39:47.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.186+0000 7fd27ccdc640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2780d2810 con 0x7fd27815e280 2024-03-24T00:39:47.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.186+0000 7fd275d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd2780d3ae0 0x7fd2780b5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:47.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.186+0000 7fd275d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd2780d3ae0 0x7fd2780b5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49890/0 (socket says 172.21.15.67:49890) 2024-03-24T00:39:47.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.186+0000 7fd275d9b640 1 -- 172.21.15.67:0/2437843624 learned_addr learned my addr 172.21.15.67:0/2437843624 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:47.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.186+0000 7fd276d9d640 1 --2- 172.21.15.67:0/2437843624 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd27815e280 0x7fd2780b5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:47.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.187+0000 7fd275d9b640 1 -- 172.21.15.67:0/2437843624 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd27815e280 msgr2=0x7fd2780b5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:47.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.187+0000 7fd275d9b640 1 --2- 172.21.15.67:0/2437843624 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd27815e280 0x7fd2780b5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:47.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.187+0000 7fd275d9b640 1 -- 172.21.15.67:0/2437843624 >> v1:172.21.15.72:6789/0 conn(0x7fd27815dea0 legacy=0x7fd2780d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:47.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.187+0000 7fd275d9b640 1 -- 172.21.15.67:0/2437843624 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2780d2af0 con 0x7fd2780d3ae0 2024-03-24T00:39:47.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.187+0000 7fd275d9b640 1 --2- 172.21.15.67:0/2437843624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2780d3ae0 0x7fd2780b5a60 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fd26c0098a0 tx=0x7fd26c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c80cb534547f688 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:47.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.188+0000 7fd27559a640 1 -- 172.21.15.67:0/2437843624 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd26c035020 con 0x7fd2780d3ae0 2024-03-24T00:39:47.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.188+0000 7fd27559a640 1 -- 172.21.15.67:0/2437843624 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd26c031640 con 0x7fd2780d3ae0 2024-03-24T00:39:47.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.188+0000 7fd27559a640 1 -- 172.21.15.67:0/2437843624 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd26c02d040 con 0x7fd2780d3ae0 2024-03-24T00:39:47.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.188+0000 7fd27ccdc640 1 -- 172.21.15.67:0/2437843624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2780d3ae0 msgr2=0x7fd2780b5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:47.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.188+0000 7fd27ccdc640 1 --2- 172.21.15.67:0/2437843624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2780d3ae0 0x7fd2780b5a60 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fd26c0098a0 tx=0x7fd26c02b8f0 comp rx=0 tx=0).stop 2024-03-24T00:39:47.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.188+0000 7fd27ccdc640 1 -- 172.21.15.67:0/2437843624 shutdown_connections 2024-03-24T00:39:47.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.188+0000 7fd27ccdc640 1 --2- 172.21.15.67:0/2437843624 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd27815e280 0x7fd2780b5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:47.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.188+0000 7fd27ccdc640 1 --2- 172.21.15.67:0/2437843624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2780d3ae0 0x7fd2780b5a60 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:47.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.188+0000 7fd27ccdc640 1 -- 172.21.15.67:0/2437843624 >> 172.21.15.67:0/2437843624 conn(0x7fd278072da0 msgr2=0x7fd2780d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:47.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.188+0000 7fd27ccdc640 1 -- 172.21.15.67:0/2437843624 shutdown_connections 2024-03-24T00:39:47.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.188+0000 7fd27ccdc640 1 -- 172.21.15.67:0/2437843624 wait complete. 2024-03-24T00:39:47.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.189+0000 7fd27ccdc640 1 Processor -- start 2024-03-24T00:39:47.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.190+0000 7fd27ccdc640 1 -- start start 2024-03-24T00:39:47.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.190+0000 7fd27ccdc640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd2780d3ae0 0x7fd2780c9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:47.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.190+0000 7fd27ccdc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd27815dea0 0x7fd278159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:47.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.190+0000 7fd27ccdc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd27815e280 0x7fd27814ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:47.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.190+0000 7fd27ccdc640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2780b7480 con 0x7fd27815dea0 2024-03-24T00:39:47.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.190+0000 7fd27ccdc640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2780b7300 con 0x7fd27815e280 2024-03-24T00:39:47.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.190+0000 7fd27ccdc640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2780b7600 con 0x7fd2780d3ae0 2024-03-24T00:39:47.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.190+0000 7fd275d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd27815dea0 0x7fd278159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:47.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.190+0000 7fd275d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd27815dea0 0x7fd278159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49900/0 (socket says 172.21.15.67:49900) 2024-03-24T00:39:47.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.190+0000 7fd275d9b640 1 -- 172.21.15.67:0/3092366807 learned_addr learned my addr 172.21.15.67:0/3092366807 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:47.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.190+0000 7fd27659c640 1 --2- 172.21.15.67:0/3092366807 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd2780d3ae0 0x7fd2780c9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:47.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.190+0000 7fd276d9d640 1 --2- 172.21.15.67:0/3092366807 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd27815e280 0x7fd27814ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:47.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.191+0000 7fd275d9b640 1 -- 172.21.15.67:0/3092366807 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd2780d3ae0 msgr2=0x7fd2780c9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:47.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.191+0000 7fd275d9b640 1 --2- 172.21.15.67:0/3092366807 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd2780d3ae0 0x7fd2780c9cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:47.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.191+0000 7fd275d9b640 1 -- 172.21.15.67:0/3092366807 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd27815e280 msgr2=0x7fd27814ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:47.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.191+0000 7fd275d9b640 1 --2- 172.21.15.67:0/3092366807 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd27815e280 0x7fd27814ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:47.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.191+0000 7fd275d9b640 1 -- 172.21.15.67:0/3092366807 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd278158f10 con 0x7fd27815dea0 2024-03-24T00:39:47.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.191+0000 7fd275d9b640 1 --2- 172.21.15.67:0/3092366807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd27815dea0 0x7fd278159fb0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fd26c0099c0 tx=0x7fd26c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:47.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.191+0000 7fd266ffd640 1 -- 172.21.15.67:0/3092366807 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd26c040070 con 0x7fd27815dea0 2024-03-24T00:39:47.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.191+0000 7fd266ffd640 1 -- 172.21.15.67:0/3092366807 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd26c03e070 con 0x7fd27815dea0 2024-03-24T00:39:47.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.191+0000 7fd27ccdc640 1 -- 172.21.15.67:0/3092366807 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2781591f0 con 0x7fd27815dea0 2024-03-24T00:39:47.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.191+0000 7fd266ffd640 1 -- 172.21.15.67:0/3092366807 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd26c035040 con 0x7fd27815dea0 2024-03-24T00:39:47.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.191+0000 7fd27ccdc640 1 -- 172.21.15.67:0/3092366807 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2781597b0 con 0x7fd27815dea0 2024-03-24T00:39:47.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.192+0000 7fd27ccdc640 1 -- 172.21.15.67:0/3092366807 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd27804f400 con 0x7fd27815dea0 2024-03-24T00:39:47.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.193+0000 7fd266ffd640 1 -- 172.21.15.67:0/3092366807 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd26c038020 con 0x7fd27815dea0 2024-03-24T00:39:47.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.193+0000 7fd266ffd640 1 --2- 172.21.15.67:0/3092366807 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd25806bbd0 0x7fd25806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:47.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.193+0000 7fd266ffd640 1 -- 172.21.15.67:0/3092366807 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fd26c0ac750 con 0x7fd27815dea0 2024-03-24T00:39:47.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.193+0000 7fd27659c640 1 --2- 172.21.15.67:0/3092366807 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd25806bbd0 0x7fd25806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:47.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.194+0000 7fd27659c640 1 --2- 172.21.15.67:0/3092366807 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd25806bbd0 0x7fd25806e090 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd270001780 tx=0x7fd270009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:47.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.198+0000 7fd266ffd640 1 -- 172.21.15.67:0/3092366807 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd26c07a770 con 0x7fd27815dea0 2024-03-24T00:39:47.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.383+0000 7fd27ccdc640 1 -- 172.21.15.67:0/3092366807 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd2780b3270 con 0x7fd27815dea0 2024-03-24T00:39:47.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.384+0000 7fd266ffd640 1 -- 172.21.15.67:0/3092366807 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd26c07e810 con 0x7fd27815dea0 2024-03-24T00:39:47.385 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-03-24T00:39:47.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.385+0000 7fd27ccdc640 1 -- 172.21.15.67:0/3092366807 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd25806bbd0 msgr2=0x7fd25806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:47.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.385+0000 7fd27ccdc640 1 --2- 172.21.15.67:0/3092366807 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd25806bbd0 0x7fd25806e090 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd270001780 tx=0x7fd270009040 comp rx=0 tx=0).stop 2024-03-24T00:39:47.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.385+0000 7fd27ccdc640 1 -- 172.21.15.67:0/3092366807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd27815dea0 msgr2=0x7fd278159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:47.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.385+0000 7fd27ccdc640 1 --2- 172.21.15.67:0/3092366807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd27815dea0 0x7fd278159fb0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fd26c0099c0 tx=0x7fd26c0046c0 comp rx=0 tx=0).stop 2024-03-24T00:39:47.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.386+0000 7fd27ccdc640 1 -- 172.21.15.67:0/3092366807 shutdown_connections 2024-03-24T00:39:47.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.386+0000 7fd27ccdc640 1 --2- 172.21.15.67:0/3092366807 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd25806bbd0 0x7fd25806e090 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:47.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.386+0000 7fd27ccdc640 1 --2- 172.21.15.67:0/3092366807 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd27815e280 0x7fd27814ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:47.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.386+0000 7fd27ccdc640 1 --2- 172.21.15.67:0/3092366807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd27815dea0 0x7fd278159fb0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:47.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.386+0000 7fd27ccdc640 1 --2- 172.21.15.67:0/3092366807 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd2780d3ae0 0x7fd2780c9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:47.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.386+0000 7fd27ccdc640 1 -- 172.21.15.67:0/3092366807 >> 172.21.15.67:0/3092366807 conn(0x7fd278072da0 msgr2=0x7fd2780b42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:47.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.386+0000 7fd27ccdc640 1 -- 172.21.15.67:0/3092366807 shutdown_connections 2024-03-24T00:39:47.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.386+0000 7fd27ccdc640 1 -- 172.21.15.67:0/3092366807 wait complete. 2024-03-24T00:39:47.397 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T00:39:47.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.501+0000 7fbc3eb10640 1 Processor -- start 2024-03-24T00:39:47.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.501+0000 7fbc3eb10640 1 -- start start 2024-03-24T00:39:47.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.501+0000 7fbc3eb10640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbc380d0d50 0x7fbc380ce610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:47.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.501+0000 7fbc3eb10640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc380cf570 con 0x7fbc3815e130 2024-03-24T00:39:47.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.501+0000 7fbc3eb10640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc380cecf0 con 0x7fbc380d22b0 2024-03-24T00:39:47.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.501+0000 7fbc3eb10640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc380ceb50 con 0x7fbc380d0d50 2024-03-24T00:39:47.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.502+0000 7fbc37fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbc380d0d50 0x7fbc380ce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:47.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.502+0000 7fbc3c8ac640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbc3815e130 0x7fbc380d0640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50490/0 (socket says 172.21.15.67:50490) 2024-03-24T00:39:47.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.502+0000 7fbc3c8ac640 1 -- 172.21.15.67:0/2021558225 learned_addr learned my addr 172.21.15.67:0/2021558225 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:47.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.502+0000 7fbc37fff640 1 -- 172.21.15.67:0/2021558225 >> v1:172.21.15.72:6789/0 conn(0x7fbc380d22b0 legacy=0x7fbc380d2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:47.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.502+0000 7fbc37fff640 1 -- 172.21.15.67:0/2021558225 >> v1:172.21.15.67:6789/0 conn(0x7fbc3815e130 legacy=0x7fbc380d0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:47.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.502+0000 7fbc37fff640 1 -- 172.21.15.67:0/2021558225 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc38162a60 con 0x7fbc380d0d50 2024-03-24T00:39:47.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.502+0000 7fbc377fe640 1 -- 172.21.15.67:0/2021558225 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2035856526 0 0) 0x7fbc380cf570 con 0x7fbc3815e130 2024-03-24T00:39:47.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.503+0000 7fbc37fff640 1 --2- 172.21.15.67:0/2021558225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc380d0d50 0x7fbc380ce610 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fbc240097a0 tx=0x7fbc2402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=255a5488055eb0ed server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:47.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.503+0000 7fbc377fe640 1 -- 172.21.15.67:0/2021558225 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc24034070 con 0x7fbc380d0d50 2024-03-24T00:39:47.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.503+0000 7fbc377fe640 1 -- 172.21.15.67:0/2021558225 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc24030610 con 0x7fbc380d0d50 2024-03-24T00:39:47.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.503+0000 7fbc377fe640 1 -- 172.21.15.67:0/2021558225 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc240309e0 con 0x7fbc380d0d50 2024-03-24T00:39:47.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.503+0000 7fbc3eb10640 1 -- 172.21.15.67:0/2021558225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc380d0d50 msgr2=0x7fbc380ce610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:47.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.503+0000 7fbc3eb10640 1 --2- 172.21.15.67:0/2021558225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc380d0d50 0x7fbc380ce610 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fbc240097a0 tx=0x7fbc2402b960 comp rx=0 tx=0).stop 2024-03-24T00:39:47.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.503+0000 7fbc3eb10640 1 -- 172.21.15.67:0/2021558225 shutdown_connections 2024-03-24T00:39:47.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.503+0000 7fbc3eb10640 1 --2- 172.21.15.67:0/2021558225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc380d0d50 0x7fbc380ce610 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:47.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.503+0000 7fbc3eb10640 1 -- 172.21.15.67:0/2021558225 >> 172.21.15.67:0/2021558225 conn(0x7fbc3806c8a0 msgr2=0x7fbc3806ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:47.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.503+0000 7fbc3eb10640 1 -- 172.21.15.67:0/2021558225 shutdown_connections 2024-03-24T00:39:47.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.503+0000 7fbc3eb10640 1 -- 172.21.15.67:0/2021558225 wait complete. 2024-03-24T00:39:47.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.505+0000 7fbc3eb10640 1 Processor -- start 2024-03-24T00:39:47.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.505+0000 7fbc3eb10640 1 -- start start 2024-03-24T00:39:47.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.505+0000 7fbc3eb10640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc380d0d50 0x7fbc38154f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:47.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.505+0000 7fbc3eb10640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc380d22b0 0x7fbc380ca030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:47.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.505+0000 7fbc3eb10640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbc3815e130 0x7fbc380c7830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:47.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.506+0000 7fbc3eb10640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3815e510 con 0x7fbc380d22b0 2024-03-24T00:39:47.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.506+0000 7fbc3eb10640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc380cf290 con 0x7fbc380d0d50 2024-03-24T00:39:47.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.506+0000 7fbc3eb10640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc380b2e10 con 0x7fbc3815e130 2024-03-24T00:39:47.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.506+0000 7fbc37fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc380d22b0 0x7fbc380ca030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:47.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.506+0000 7fbc37fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc380d22b0 0x7fbc380ca030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49916/0 (socket says 172.21.15.67:49916) 2024-03-24T00:39:47.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.506+0000 7fbc3c8ac640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc380d0d50 0x7fbc38154f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:47.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.506+0000 7fbc37fff640 1 -- 172.21.15.67:0/1343220816 learned_addr learned my addr 172.21.15.67:0/1343220816 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:47.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.506+0000 7fbc3d0ad640 1 --2- 172.21.15.67:0/1343220816 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbc3815e130 0x7fbc380c7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:47.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.506+0000 7fbc37fff640 1 -- 172.21.15.67:0/1343220816 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbc3815e130 msgr2=0x7fbc380c7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:47.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.506+0000 7fbc37fff640 1 --2- 172.21.15.67:0/1343220816 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbc3815e130 0x7fbc380c7830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:47.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.506+0000 7fbc37fff640 1 -- 172.21.15.67:0/1343220816 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc380d0d50 msgr2=0x7fbc38154f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:47.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.506+0000 7fbc37fff640 1 --2- 172.21.15.67:0/1343220816 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc380d0d50 0x7fbc38154f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:47.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.506+0000 7fbc37fff640 1 -- 172.21.15.67:0/1343220816 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc38155450 con 0x7fbc380d22b0 2024-03-24T00:39:47.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.506+0000 7fbc37fff640 1 --2- 172.21.15.67:0/1343220816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc380d22b0 0x7fbc380ca030 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fbc2402bf70 tx=0x7fbc240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:47.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.506+0000 7fbc357fa640 1 -- 172.21.15.67:0/1343220816 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc24034040 con 0x7fbc380d22b0 2024-03-24T00:39:47.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.506+0000 7fbc357fa640 1 -- 172.21.15.67:0/1343220816 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc24039a10 con 0x7fbc380d22b0 2024-03-24T00:39:47.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.506+0000 7fbc3eb10640 1 -- 172.21.15.67:0/1343220816 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc38161140 con 0x7fbc380d22b0 2024-03-24T00:39:47.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.506+0000 7fbc357fa640 1 -- 172.21.15.67:0/1343220816 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc24039de0 con 0x7fbc380d22b0 2024-03-24T00:39:47.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.506+0000 7fbc3eb10640 1 -- 172.21.15.67:0/1343220816 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc380c9930 con 0x7fbc380d22b0 2024-03-24T00:39:47.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.507+0000 7fbc3eb10640 1 -- 172.21.15.67:0/1343220816 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc3804f400 con 0x7fbc380d22b0 2024-03-24T00:39:47.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.508+0000 7fbc357fa640 1 -- 172.21.15.67:0/1343220816 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fbc24037070 con 0x7fbc380d22b0 2024-03-24T00:39:47.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.508+0000 7fbc357fa640 1 --2- 172.21.15.67:0/1343220816 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbc1c06bbd0 0x7fbc1c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:47.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.508+0000 7fbc357fa640 1 -- 172.21.15.67:0/1343220816 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fbc240ace40 con 0x7fbc380d22b0 2024-03-24T00:39:47.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.508+0000 7fbc3c8ac640 1 --2- 172.21.15.67:0/1343220816 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbc1c06bbd0 0x7fbc1c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:47.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.509+0000 7fbc3c8ac640 1 --2- 172.21.15.67:0/1343220816 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbc1c06bbd0 0x7fbc1c06e090 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c007c30 tx=0x7fbc2c007b80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:47.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.513+0000 7fbc357fa640 1 -- 172.21.15.67:0/1343220816 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fbc2407aee0 con 0x7fbc380d22b0 2024-03-24T00:39:47.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.692+0000 7fbc3eb10640 1 -- 172.21.15.67:0/1343220816 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fbc380b6640 con 0x7fbc380d22b0 2024-03-24T00:39:47.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.693+0000 7fbc357fa640 1 -- 172.21.15.67:0/1343220816 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7fbc2407ef80 con 0x7fbc380d22b0 2024-03-24T00:39:47.694 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-24T00:39:47.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.694+0000 7fbc3eb10640 1 -- 172.21.15.67:0/1343220816 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbc1c06bbd0 msgr2=0x7fbc1c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:47.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.694+0000 7fbc3eb10640 1 --2- 172.21.15.67:0/1343220816 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbc1c06bbd0 0x7fbc1c06e090 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c007c30 tx=0x7fbc2c007b80 comp rx=0 tx=0).stop 2024-03-24T00:39:47.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.694+0000 7fbc3eb10640 1 -- 172.21.15.67:0/1343220816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc380d22b0 msgr2=0x7fbc380ca030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:47.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.694+0000 7fbc3eb10640 1 --2- 172.21.15.67:0/1343220816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc380d22b0 0x7fbc380ca030 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fbc2402bf70 tx=0x7fbc240046c0 comp rx=0 tx=0).stop 2024-03-24T00:39:47.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.695+0000 7fbc3eb10640 1 -- 172.21.15.67:0/1343220816 shutdown_connections 2024-03-24T00:39:47.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.695+0000 7fbc3eb10640 1 --2- 172.21.15.67:0/1343220816 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbc1c06bbd0 0x7fbc1c06e090 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:47.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.695+0000 7fbc3eb10640 1 --2- 172.21.15.67:0/1343220816 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbc3815e130 0x7fbc380c7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:47.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.695+0000 7fbc3eb10640 1 --2- 172.21.15.67:0/1343220816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc380d22b0 0x7fbc380ca030 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:47.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.695+0000 7fbc3eb10640 1 --2- 172.21.15.67:0/1343220816 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc380d0d50 0x7fbc38154f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:47.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.695+0000 7fbc3eb10640 1 -- 172.21.15.67:0/1343220816 >> 172.21.15.67:0/1343220816 conn(0x7fbc3806c8a0 msgr2=0x7fbc380b3980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:47.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.695+0000 7fbc3eb10640 1 -- 172.21.15.67:0/1343220816 shutdown_connections 2024-03-24T00:39:47.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.695+0000 7fbc3eb10640 1 -- 172.21.15.67:0/1343220816 wait complete. 2024-03-24T00:39:47.706 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:39:47.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.808+0000 7f53e1362640 1 Processor -- start 2024-03-24T00:39:47.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.808+0000 7f53e1362640 1 -- start start 2024-03-24T00:39:47.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.809+0000 7f53e1362640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f53dc0d1550 0x7f53dc06cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:47.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.809+0000 7f53e1362640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f53dc09a480 0x7f53dc0b2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:47.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.809+0000 7f53e1362640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53dc0d1930 con 0x7f53dc0b2780 2024-03-24T00:39:47.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.809+0000 7f53e1362640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53dc06d420 con 0x7f53dc0d1550 2024-03-24T00:39:47.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.809+0000 7f53e1362640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53dc06ca50 con 0x7f53dc09a480 2024-03-24T00:39:47.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.809+0000 7f53da7fc640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f53dc0b2780 0x7f53dc099d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50504/0 (socket says 172.21.15.67:50504) 2024-03-24T00:39:47.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.809+0000 7f53da7fc640 1 -- 172.21.15.67:0/3948292393 learned_addr learned my addr 172.21.15.67:0/3948292393 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:47.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.809+0000 7f53daffd640 1 --2- 172.21.15.67:0/3948292393 >> v2:172.21.15.72:3300/0 conn(0x7f53dc0d1550 0x7f53dc06cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:47.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.809+0000 7f53db7fe640 1 --2- 172.21.15.67:0/3948292393 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f53dc09a480 0x7f53dc0b2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:47.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.809+0000 7f53d9ffb640 1 -- 172.21.15.67:0/3948292393 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3268371770 0 0) 0x7f53dc0d1930 con 0x7f53dc0b2780 2024-03-24T00:39:47.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.809+0000 7f53d9ffb640 1 -- 172.21.15.67:0/3948292393 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53c8003520 con 0x7f53dc0b2780 2024-03-24T00:39:47.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.809+0000 7f53db7fe640 1 -- 172.21.15.67:0/3948292393 >> v2:172.21.15.72:3300/0 conn(0x7f53dc0d1550 msgr2=0x7f53dc06cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:47.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.809+0000 7f53db7fe640 1 --2- 172.21.15.67:0/3948292393 >> v2:172.21.15.72:3300/0 conn(0x7f53dc0d1550 0x7f53dc06cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:47.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.809+0000 7f53db7fe640 1 -- 172.21.15.67:0/3948292393 >> v1:172.21.15.67:6789/0 conn(0x7f53dc0b2780 legacy=0x7f53dc099d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:47.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.809+0000 7f53db7fe640 1 -- 172.21.15.67:0/3948292393 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53dc0b1f60 con 0x7f53dc09a480 2024-03-24T00:39:47.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.809+0000 7f53daffd640 1 --2- 172.21.15.67:0/3948292393 >> v2:172.21.15.72:3300/0 conn(0x7f53dc0d1550 0x7f53dc06cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:47.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.810+0000 7f53db7fe640 1 --2- 172.21.15.67:0/3948292393 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f53dc09a480 0x7f53dc0b2c30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f53d40097d0 tx=0x7f53d402b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4af576c0c6ba28a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:47.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.810+0000 7f53d9ffb640 1 -- 172.21.15.67:0/3948292393 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d4031770 con 0x7f53dc09a480 2024-03-24T00:39:47.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.810+0000 7f53d9ffb640 1 -- 172.21.15.67:0/3948292393 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53d4037070 con 0x7f53dc09a480 2024-03-24T00:39:47.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.810+0000 7f53d9ffb640 1 -- 172.21.15.67:0/3948292393 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d4031b10 con 0x7f53dc09a480 2024-03-24T00:39:47.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.811+0000 7f53e1362640 1 -- 172.21.15.67:0/3948292393 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f53dc09a480 msgr2=0x7f53dc0b2c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:47.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.811+0000 7f53e1362640 1 --2- 172.21.15.67:0/3948292393 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f53dc09a480 0x7f53dc0b2c30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f53d40097d0 tx=0x7f53d402b810 comp rx=0 tx=0).stop 2024-03-24T00:39:47.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.811+0000 7f53e1362640 1 -- 172.21.15.67:0/3948292393 shutdown_connections 2024-03-24T00:39:47.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.811+0000 7f53e1362640 1 --2- 172.21.15.67:0/3948292393 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f53dc09a480 0x7f53dc0b2c30 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:47.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.811+0000 7f53e1362640 1 --2- 172.21.15.67:0/3948292393 >> v2:172.21.15.72:3300/0 conn(0x7f53dc0d1550 0x7f53dc06cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:47.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.811+0000 7f53e1362640 1 -- 172.21.15.67:0/3948292393 >> 172.21.15.67:0/3948292393 conn(0x7f53dc0b36e0 msgr2=0x7f53dc0b3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:47.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.811+0000 7f53e1362640 1 -- 172.21.15.67:0/3948292393 shutdown_connections 2024-03-24T00:39:47.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.811+0000 7f53e1362640 1 -- 172.21.15.67:0/3948292393 wait complete. 2024-03-24T00:39:47.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.812+0000 7f53e1362640 1 Processor -- start 2024-03-24T00:39:47.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.812+0000 7f53e1362640 1 -- start start 2024-03-24T00:39:47.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.813+0000 7f53e1362640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53dc09a480 0x7f53dc0c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:47.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.813+0000 7f53e1362640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53dc0b2780 0x7f53dc0b5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:47.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.813+0000 7f53e1362640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f53dc0d1550 0x7f53dc1e9340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:47.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.813+0000 7f53e1362640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53dc06c400 con 0x7f53dc0b2780 2024-03-24T00:39:47.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.813+0000 7f53e1362640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53dc0b2520 con 0x7f53dc09a480 2024-03-24T00:39:47.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.813+0000 7f53e1362640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53dc06c6e0 con 0x7f53dc0d1550 2024-03-24T00:39:47.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.813+0000 7f53da7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53dc0b2780 0x7f53dc0b5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:47.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.813+0000 7f53da7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53dc0b2780 0x7f53dc0b5220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49926/0 (socket says 172.21.15.67:49926) 2024-03-24T00:39:47.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.813+0000 7f53da7fc640 1 -- 172.21.15.67:0/3148528411 learned_addr learned my addr 172.21.15.67:0/3148528411 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:47.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.813+0000 7f53db7fe640 1 --2- 172.21.15.67:0/3148528411 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f53dc0d1550 0x7f53dc1e9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:47.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.813+0000 7f53daffd640 1 --2- 172.21.15.67:0/3148528411 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53dc09a480 0x7f53dc0c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:47.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.813+0000 7f53da7fc640 1 -- 172.21.15.67:0/3148528411 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f53dc0d1550 msgr2=0x7f53dc1e9340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:47.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.813+0000 7f53da7fc640 1 --2- 172.21.15.67:0/3148528411 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f53dc0d1550 0x7f53dc1e9340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:47.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.813+0000 7f53da7fc640 1 -- 172.21.15.67:0/3148528411 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53dc09a480 msgr2=0x7f53dc0c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:47.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.813+0000 7f53da7fc640 1 --2- 172.21.15.67:0/3148528411 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53dc09a480 0x7f53dc0c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:47.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.813+0000 7f53da7fc640 1 -- 172.21.15.67:0/3148528411 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53dc0c93c0 con 0x7f53dc0b2780 2024-03-24T00:39:47.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.813+0000 7f53da7fc640 1 --2- 172.21.15.67:0/3148528411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53dc0b2780 0x7f53dc0b5220 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f53c400c430 tx=0x7f53c400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:47.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.814+0000 7f53d37fe640 1 -- 172.21.15.67:0/3148528411 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c4017070 con 0x7f53dc0b2780 2024-03-24T00:39:47.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.814+0000 7f53d37fe640 1 -- 172.21.15.67:0/3148528411 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53c4005ce0 con 0x7f53dc0b2780 2024-03-24T00:39:47.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.814+0000 7f53e1362640 1 -- 172.21.15.67:0/3148528411 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53dc0c96a0 con 0x7f53dc0b2780 2024-03-24T00:39:47.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.814+0000 7f53d37fe640 1 -- 172.21.15.67:0/3148528411 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c40072f0 con 0x7f53dc0b2780 2024-03-24T00:39:47.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.814+0000 7f53e1362640 1 -- 172.21.15.67:0/3148528411 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53dc0c9c60 con 0x7f53dc0b2780 2024-03-24T00:39:47.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.815+0000 7f53e1362640 1 -- 172.21.15.67:0/3148528411 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53dc04f400 con 0x7f53dc0b2780 2024-03-24T00:39:47.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.815+0000 7f53d37fe640 1 -- 172.21.15.67:0/3148528411 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f53c401a070 con 0x7f53dc0b2780 2024-03-24T00:39:47.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.815+0000 7f53d37fe640 1 --2- 172.21.15.67:0/3148528411 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f53bc06bbd0 0x7f53bc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:47.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.816+0000 7f53d37fe640 1 -- 172.21.15.67:0/3148528411 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f53c408a280 con 0x7f53dc0b2780 2024-03-24T00:39:47.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.816+0000 7f53daffd640 1 --2- 172.21.15.67:0/3148528411 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f53bc06bbd0 0x7f53bc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:47.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.816+0000 7f53daffd640 1 --2- 172.21.15.67:0/3148528411 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f53bc06bbd0 0x7f53bc06e090 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f53cc009bd0 tx=0x7f53cc00b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:47.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:47.820+0000 7f53d37fe640 1 -- 172.21.15.67:0/3148528411 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f53c4058320 con 0x7f53dc0b2780 2024-03-24T00:39:48.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.042+0000 7f53e1362640 1 -- 172.21.15.67:0/3148528411 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53dc0b1310 con 0x7f53dc0b2780 2024-03-24T00:39:48.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.042+0000 7f53d37fe640 1 -- 172.21.15.67:0/3148528411 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4970 (secure 0 0 0) 0x7f53c405c3c0 con 0x7f53dc0b2780 2024-03-24T00:39:48.043 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:39:48.043 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4263,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/837528243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":837528243},{"type":"v1","addr":"172.21.15.67:6834","nonce":837528243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3987071456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":3987071456},{"type":"v1","addr":"172.21.15.67:6835","nonce":3987071456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4306,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/3953492281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3953492281},{"type":"v1","addr":"172.21.15.72:6837","nonce":3953492281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:38:52.127871+0000","modified":"2024-03-24T00:39:08.187262+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4309},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4309":{"gid":4309,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6836/3737698654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3737698654},{"type":"v1","addr":"172.21.15.72:6836","nonce":3737698654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4309,"qdb_cluster":[4309]},"id":1}]} 2024-03-24T00:39:48.043 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 9 2024-03-24T00:39:48.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.044+0000 7f53e1362640 1 -- 172.21.15.67:0/3148528411 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f53bc06bbd0 msgr2=0x7f53bc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:48.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.044+0000 7f53e1362640 1 --2- 172.21.15.67:0/3148528411 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f53bc06bbd0 0x7f53bc06e090 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f53cc009bd0 tx=0x7f53cc00b040 comp rx=0 tx=0).stop 2024-03-24T00:39:48.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.044+0000 7f53e1362640 1 -- 172.21.15.67:0/3148528411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53dc0b2780 msgr2=0x7f53dc0b5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:48.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.044+0000 7f53e1362640 1 --2- 172.21.15.67:0/3148528411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53dc0b2780 0x7f53dc0b5220 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f53c400c430 tx=0x7f53c400c8f0 comp rx=0 tx=0).stop 2024-03-24T00:39:48.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.044+0000 7f53e1362640 1 -- 172.21.15.67:0/3148528411 shutdown_connections 2024-03-24T00:39:48.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.044+0000 7f53e1362640 1 --2- 172.21.15.67:0/3148528411 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f53bc06bbd0 0x7f53bc06e090 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:48.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.044+0000 7f53e1362640 1 --2- 172.21.15.67:0/3148528411 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f53dc0d1550 0x7f53dc1e9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:48.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.044+0000 7f53e1362640 1 --2- 172.21.15.67:0/3148528411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53dc0b2780 0x7f53dc0b5220 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:48.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.044+0000 7f53e1362640 1 --2- 172.21.15.67:0/3148528411 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53dc09a480 0x7f53dc0c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:48.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.044+0000 7f53e1362640 1 -- 172.21.15.67:0/3148528411 >> 172.21.15.67:0/3148528411 conn(0x7f53dc0b36e0 msgr2=0x7f53dc0cc220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:48.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.044+0000 7f53e1362640 1 -- 172.21.15.67:0/3148528411 shutdown_connections 2024-03-24T00:39:48.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.044+0000 7f53e1362640 1 -- 172.21.15.67:0/3148528411 wait complete. 2024-03-24T00:39:48.056 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:39:48.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.157+0000 7ff610661640 1 Processor -- start 2024-03-24T00:39:48.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.157+0000 7ff610661640 1 -- start start 2024-03-24T00:39:48.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.158+0000 7ff610661640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff60815beb0 0x7ff6080d12b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:48.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.158+0000 7ff610661640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff6080cfd30 0x7ff6080b0c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:48.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.158+0000 7ff610661640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff6080ce4f0 0x7ff6080ce980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:48.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.158+0000 7ff610661640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6080ceec0 con 0x7ff6080ce4f0 2024-03-24T00:39:48.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.158+0000 7ff610661640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6080d0210 con 0x7ff60815beb0 2024-03-24T00:39:48.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.158+0000 7ff610661640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6080d04f0 con 0x7ff6080cfd30 2024-03-24T00:39:48.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.158+0000 7ff60ebfe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff6080ce4f0 0x7ff6080ce980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:48.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.158+0000 7ff60e3fd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff60815beb0 0x7ff6080d12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:48.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.158+0000 7ff60dbfc640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff6080cfd30 0x7ff6080b0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:48.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.158+0000 7ff60ebfe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff6080ce4f0 0x7ff6080ce980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49938/0 (socket says 172.21.15.67:49938) 2024-03-24T00:39:48.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.158+0000 7ff60ebfe640 1 -- 172.21.15.67:0/3282529894 learned_addr learned my addr 172.21.15.67:0/3282529894 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:48.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.158+0000 7ff60ebfe640 1 -- 172.21.15.67:0/3282529894 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff6080cfd30 msgr2=0x7ff6080b0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:48.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.158+0000 7ff60ebfe640 1 --2- 172.21.15.67:0/3282529894 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff6080cfd30 0x7ff6080b0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:48.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.158+0000 7ff60ebfe640 1 -- 172.21.15.67:0/3282529894 >> v2:172.21.15.72:3300/0 conn(0x7ff60815beb0 msgr2=0x7ff6080d12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:48.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.158+0000 7ff60ebfe640 1 --2- 172.21.15.67:0/3282529894 >> v2:172.21.15.72:3300/0 conn(0x7ff60815beb0 0x7ff6080d12b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:48.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.158+0000 7ff60ebfe640 1 -- 172.21.15.67:0/3282529894 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6080d07d0 con 0x7ff6080ce4f0 2024-03-24T00:39:48.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.158+0000 7ff60e3fd640 1 --2- 172.21.15.67:0/3282529894 >> v2:172.21.15.72:3300/0 conn(0x7ff60815beb0 0x7ff6080d12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:48.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.159+0000 7ff60ebfe640 1 --2- 172.21.15.67:0/3282529894 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6080ce4f0 0x7ff6080ce980 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7ff608070a90 tx=0x7ff60002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5ce5a56e5e40df3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:48.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.159+0000 7ff60d3fb640 1 -- 172.21.15.67:0/3282529894 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff600034070 con 0x7ff6080ce4f0 2024-03-24T00:39:48.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.159+0000 7ff60d3fb640 1 -- 172.21.15.67:0/3282529894 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff600030860 con 0x7ff6080ce4f0 2024-03-24T00:39:48.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.159+0000 7ff60d3fb640 1 -- 172.21.15.67:0/3282529894 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff600030c30 con 0x7ff6080ce4f0 2024-03-24T00:39:48.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.160+0000 7ff610661640 1 -- 172.21.15.67:0/3282529894 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6080ce4f0 msgr2=0x7ff6080ce980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:48.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.160+0000 7ff610661640 1 --2- 172.21.15.67:0/3282529894 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6080ce4f0 0x7ff6080ce980 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7ff608070a90 tx=0x7ff60002b9a0 comp rx=0 tx=0).stop 2024-03-24T00:39:48.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.160+0000 7ff610661640 1 -- 172.21.15.67:0/3282529894 shutdown_connections 2024-03-24T00:39:48.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.160+0000 7ff610661640 1 --2- 172.21.15.67:0/3282529894 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6080ce4f0 0x7ff6080ce980 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:48.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.160+0000 7ff610661640 1 --2- 172.21.15.67:0/3282529894 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff6080cfd30 0x7ff6080b0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:48.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.160+0000 7ff610661640 1 --2- 172.21.15.67:0/3282529894 >> v2:172.21.15.72:3300/0 conn(0x7ff60815beb0 0x7ff6080d12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:48.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.160+0000 7ff610661640 1 -- 172.21.15.67:0/3282529894 >> 172.21.15.67:0/3282529894 conn(0x7ff60806fcf0 msgr2=0x7ff6080700d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:48.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.160+0000 7ff610661640 1 -- 172.21.15.67:0/3282529894 shutdown_connections 2024-03-24T00:39:48.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.160+0000 7ff610661640 1 -- 172.21.15.67:0/3282529894 wait complete. 2024-03-24T00:39:48.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.161+0000 7ff610661640 1 Processor -- start 2024-03-24T00:39:48.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.161+0000 7ff610661640 1 -- start start 2024-03-24T00:39:48.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.162+0000 7ff610661640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6080ce4f0 0x7ff6080ab110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:48.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.162+0000 7ff610661640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff6080cfd30 0x7ff60814a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:48.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.162+0000 7ff610661640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff60815beb0 0x7ff60814a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:48.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.162+0000 7ff610661640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60815c290 con 0x7ff6080ce4f0 2024-03-24T00:39:48.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.162+0000 7ff610661640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6080d17f0 con 0x7ff6080cfd30 2024-03-24T00:39:48.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.162+0000 7ff610661640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6080b1140 con 0x7ff60815beb0 2024-03-24T00:39:48.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.162+0000 7ff60e3fd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6080ce4f0 0x7ff6080ab110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:48.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.162+0000 7ff60e3fd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6080ce4f0 0x7ff6080ab110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49952/0 (socket says 172.21.15.67:49952) 2024-03-24T00:39:48.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.162+0000 7ff60e3fd640 1 -- 172.21.15.67:0/222123696 learned_addr learned my addr 172.21.15.67:0/222123696 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:48.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.162+0000 7ff60dbfc640 1 --2- 172.21.15.67:0/222123696 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff6080cfd30 0x7ff60814a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:48.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.162+0000 7ff60ebfe640 1 --2- 172.21.15.67:0/222123696 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff60815beb0 0x7ff60814a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:48.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.162+0000 7ff60e3fd640 1 -- 172.21.15.67:0/222123696 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff60815beb0 msgr2=0x7ff60814a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:48.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.162+0000 7ff60e3fd640 1 --2- 172.21.15.67:0/222123696 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff60815beb0 0x7ff60814a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:48.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.162+0000 7ff60e3fd640 1 -- 172.21.15.67:0/222123696 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff6080cfd30 msgr2=0x7ff60814a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:48.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.162+0000 7ff60e3fd640 1 --2- 172.21.15.67:0/222123696 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff6080cfd30 0x7ff60814a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:48.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.162+0000 7ff60e3fd640 1 -- 172.21.15.67:0/222123696 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff60815e100 con 0x7ff6080ce4f0 2024-03-24T00:39:48.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.162+0000 7ff60e3fd640 1 --2- 172.21.15.67:0/222123696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6080ce4f0 0x7ff6080ab110 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ff5f8009ec0 tx=0x7ff5f800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:48.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.163+0000 7ff606ffd640 1 -- 172.21.15.67:0/222123696 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f8018020 con 0x7ff6080ce4f0 2024-03-24T00:39:48.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.163+0000 7ff606ffd640 1 -- 172.21.15.67:0/222123696 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5f8014680 con 0x7ff6080ce4f0 2024-03-24T00:39:48.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.163+0000 7ff610661640 1 -- 172.21.15.67:0/222123696 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff60814adf0 con 0x7ff6080ce4f0 2024-03-24T00:39:48.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.163+0000 7ff606ffd640 1 -- 172.21.15.67:0/222123696 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f8014a50 con 0x7ff6080ce4f0 2024-03-24T00:39:48.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.163+0000 7ff610661640 1 -- 172.21.15.67:0/222123696 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff60814b3b0 con 0x7ff6080ce4f0 2024-03-24T00:39:48.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.164+0000 7ff610661640 1 -- 172.21.15.67:0/222123696 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff60804f400 con 0x7ff6080ce4f0 2024-03-24T00:39:48.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.164+0000 7ff606ffd640 1 -- 172.21.15.67:0/222123696 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff5f8005d10 con 0x7ff6080ce4f0 2024-03-24T00:39:48.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.164+0000 7ff606ffd640 1 --2- 172.21.15.67:0/222123696 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff5f406bbd0 0x7ff5f406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:48.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.165+0000 7ff606ffd640 1 -- 172.21.15.67:0/222123696 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7ff5f808bad0 con 0x7ff6080ce4f0 2024-03-24T00:39:48.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.165+0000 7ff60dbfc640 1 --2- 172.21.15.67:0/222123696 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff5f406bbd0 0x7ff5f406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:48.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.165+0000 7ff60dbfc640 1 --2- 172.21.15.67:0/222123696 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff5f406bbd0 0x7ff5f406e090 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff608159dc0 tx=0x7ff5fc009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:48.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.169+0000 7ff606ffd640 1 -- 172.21.15.67:0/222123696 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff5f8022030 con 0x7ff6080ce4f0 2024-03-24T00:39:48.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.350+0000 7ff610661640 1 -- 172.21.15.67:0/222123696 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6080b4dd0 con 0x7ff6080ce4f0 2024-03-24T00:39:48.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.351+0000 7ff606ffd640 1 -- 172.21.15.67:0/222123696 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18843 (secure 0 0 0) 0x7ff5f8010040 con 0x7ff6080ce4f0 2024-03-24T00:39:48.352 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:39:48.352 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":27,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:39:32.904729+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:38:50.077085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:38:51.095454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T00:38:54.807117+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.14","mappings":[{"from":4,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:39:48.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.353+0000 7ff610661640 1 -- 172.21.15.67:0/222123696 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff5f406bbd0 msgr2=0x7ff5f406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:48.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.353+0000 7ff610661640 1 --2- 172.21.15.67:0/222123696 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff5f406bbd0 0x7ff5f406e090 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff608159dc0 tx=0x7ff5fc009190 comp rx=0 tx=0).stop 2024-03-24T00:39:48.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.353+0000 7ff610661640 1 -- 172.21.15.67:0/222123696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6080ce4f0 msgr2=0x7ff6080ab110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:48.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.353+0000 7ff610661640 1 --2- 172.21.15.67:0/222123696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6080ce4f0 0x7ff6080ab110 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ff5f8009ec0 tx=0x7ff5f800c490 comp rx=0 tx=0).stop 2024-03-24T00:39:48.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.353+0000 7ff610661640 1 -- 172.21.15.67:0/222123696 shutdown_connections 2024-03-24T00:39:48.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.353+0000 7ff610661640 1 --2- 172.21.15.67:0/222123696 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff5f406bbd0 0x7ff5f406e090 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:48.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.353+0000 7ff610661640 1 --2- 172.21.15.67:0/222123696 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff60815beb0 0x7ff60814a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:48.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.353+0000 7ff610661640 1 --2- 172.21.15.67:0/222123696 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff6080cfd30 0x7ff60814a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:48.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.353+0000 7ff610661640 1 --2- 172.21.15.67:0/222123696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6080ce4f0 0x7ff6080ab110 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:48.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.353+0000 7ff610661640 1 -- 172.21.15.67:0/222123696 >> 172.21.15.67:0/222123696 conn(0x7ff60806fcf0 msgr2=0x7ff6080b1970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:48.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.353+0000 7ff610661640 1 -- 172.21.15.67:0/222123696 shutdown_connections 2024-03-24T00:39:48.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.353+0000 7ff610661640 1 -- 172.21.15.67:0/222123696 wait complete. 2024-03-24T00:39:48.365 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-24T00:39:48.365 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:39:48.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.466+0000 7f4a76359640 1 Processor -- start 2024-03-24T00:39:48.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.466+0000 7f4a76359640 1 -- start start 2024-03-24T00:39:48.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.467+0000 7f4a76359640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4a7015dec0 0x7f4a700ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:48.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.467+0000 7f4a76359640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4a7015e2a0 0x7f4a70162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:48.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.467+0000 7f4a76359640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a700cf640 con 0x7f4a700d07e0 2024-03-24T00:39:48.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.467+0000 7f4a76359640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a700cee30 con 0x7f4a7015dec0 2024-03-24T00:39:48.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.467+0000 7f4a76359640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a700d2120 con 0x7f4a7015e2a0 2024-03-24T00:39:48.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.467+0000 7f4a6ffff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4a7015dec0 0x7f4a700ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:48.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.467+0000 7f4a6f7fe640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4a7015e2a0 0x7f4a70162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:48.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.467+0000 7f4a748f6640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f4a700d07e0 0x7f4a700b5850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:43784/0 (socket says 172.21.15.67:43784) 2024-03-24T00:39:48.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.467+0000 7f4a748f6640 1 -- 172.21.15.67:0/1888459566 learned_addr learned my addr 172.21.15.67:0/1888459566 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:48.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.468+0000 7f4a6ffff640 1 -- 172.21.15.67:0/1888459566 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4a7015e2a0 msgr2=0x7f4a70162630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:48.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.468+0000 7f4a6ffff640 1 --2- 172.21.15.67:0/1888459566 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4a7015e2a0 0x7f4a70162630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:48.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.468+0000 7f4a6ffff640 1 -- 172.21.15.67:0/1888459566 >> v1:172.21.15.72:6789/0 conn(0x7f4a700d07e0 legacy=0x7f4a700b5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:48.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.468+0000 7f4a6ffff640 1 -- 172.21.15.67:0/1888459566 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a700cf080 con 0x7f4a7015dec0 2024-03-24T00:39:48.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.468+0000 7f4a6effd640 1 -- 172.21.15.67:0/1888459566 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4140446549 0 0) 0x7f4a68001490 con 0x7f4a700d07e0 2024-03-24T00:39:48.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.468+0000 7f4a6ffff640 1 --2- 172.21.15.67:0/1888459566 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a7015dec0 0x7f4a700ce630 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4a700cefb0 tx=0x7f4a6002b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db85a6ae58f0eb9b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:48.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.469+0000 7f4a6effd640 1 -- 172.21.15.67:0/1888459566 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a60031770 con 0x7f4a7015dec0 2024-03-24T00:39:48.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.469+0000 7f4a6effd640 1 -- 172.21.15.67:0/1888459566 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a60037070 con 0x7f4a7015dec0 2024-03-24T00:39:48.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.469+0000 7f4a6effd640 1 -- 172.21.15.67:0/1888459566 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a60031b30 con 0x7f4a7015dec0 2024-03-24T00:39:48.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.469+0000 7f4a76359640 1 -- 172.21.15.67:0/1888459566 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a7015dec0 msgr2=0x7f4a700ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:48.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.469+0000 7f4a76359640 1 --2- 172.21.15.67:0/1888459566 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a7015dec0 0x7f4a700ce630 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4a700cefb0 tx=0x7f4a6002b810 comp rx=0 tx=0).stop 2024-03-24T00:39:48.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.469+0000 7f4a76359640 1 -- 172.21.15.67:0/1888459566 shutdown_connections 2024-03-24T00:39:48.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.469+0000 7f4a76359640 1 --2- 172.21.15.67:0/1888459566 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4a7015e2a0 0x7f4a70162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:48.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.469+0000 7f4a76359640 1 --2- 172.21.15.67:0/1888459566 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a7015dec0 0x7f4a700ce630 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:48.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.469+0000 7f4a76359640 1 -- 172.21.15.67:0/1888459566 >> 172.21.15.67:0/1888459566 conn(0x7f4a7006c810 msgr2=0x7f4a7006cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:48.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.469+0000 7f4a76359640 1 -- 172.21.15.67:0/1888459566 shutdown_connections 2024-03-24T00:39:48.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.469+0000 7f4a76359640 1 -- 172.21.15.67:0/1888459566 wait complete. 2024-03-24T00:39:48.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.470+0000 7f4a76359640 1 Processor -- start 2024-03-24T00:39:48.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.471+0000 7f4a76359640 1 -- start start 2024-03-24T00:39:48.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.471+0000 7f4a76359640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4a700d07e0 0x7f4a701f3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:48.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.471+0000 7f4a76359640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a7015dec0 0x7f4a700b6090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:48.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.471+0000 7f4a76359640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a7015e2a0 0x7f4a700b4190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:48.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.471+0000 7f4a76359640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a700d3740 con 0x7f4a7015e2a0 2024-03-24T00:39:48.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.471+0000 7f4a76359640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a700d35c0 con 0x7f4a7015dec0 2024-03-24T00:39:48.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.471+0000 7f4a76359640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a700d38c0 con 0x7f4a700d07e0 2024-03-24T00:39:48.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.471+0000 7f4a6f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a7015dec0 0x7f4a700b6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:48.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.471+0000 7f4a6ffff640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4a700d07e0 0x7f4a701f3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:48.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.471+0000 7f4a748f6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a7015e2a0 0x7f4a700b4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:48.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.471+0000 7f4a6f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a7015dec0 0x7f4a700b6090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:50684/0 (socket says 172.21.15.67:50684) 2024-03-24T00:39:48.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.471+0000 7f4a6f7fe640 1 -- 172.21.15.67:0/2761858712 learned_addr learned my addr 172.21.15.67:0/2761858712 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:48.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.472+0000 7f4a6f7fe640 1 -- 172.21.15.67:0/2761858712 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4a700d07e0 msgr2=0x7f4a701f3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:48.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.472+0000 7f4a6f7fe640 1 --2- 172.21.15.67:0/2761858712 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4a700d07e0 0x7f4a701f3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:48.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.472+0000 7f4a6f7fe640 1 -- 172.21.15.67:0/2761858712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a7015e2a0 msgr2=0x7f4a700b4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:48.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.472+0000 7f4a6f7fe640 1 --2- 172.21.15.67:0/2761858712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a7015e2a0 0x7f4a700b4190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:48.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.472+0000 7f4a6f7fe640 1 -- 172.21.15.67:0/2761858712 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a700b8790 con 0x7f4a7015dec0 2024-03-24T00:39:48.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.472+0000 7f4a748f6640 1 --2- 172.21.15.67:0/2761858712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a7015e2a0 0x7f4a700b4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:48.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.472+0000 7f4a6f7fe640 1 --2- 172.21.15.67:0/2761858712 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a7015dec0 0x7f4a700b6090 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4a6400e870 tx=0x7f4a6400ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:48.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.472+0000 7f4a6cff9640 1 -- 172.21.15.67:0/2761858712 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a64017070 con 0x7f4a7015dec0 2024-03-24T00:39:48.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.472+0000 7f4a6cff9640 1 -- 172.21.15.67:0/2761858712 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a6400c940 con 0x7f4a7015dec0 2024-03-24T00:39:48.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.472+0000 7f4a6cff9640 1 -- 172.21.15.67:0/2761858712 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a6400cd10 con 0x7f4a7015dec0 2024-03-24T00:39:48.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.472+0000 7f4a76359640 1 -- 172.21.15.67:0/2761858712 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a700b8a70 con 0x7f4a7015dec0 2024-03-24T00:39:48.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.472+0000 7f4a76359640 1 -- 172.21.15.67:0/2761858712 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a700b9030 con 0x7f4a7015dec0 2024-03-24T00:39:48.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.473+0000 7f4a76359640 1 -- 172.21.15.67:0/2761858712 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a7004f400 con 0x7f4a7015dec0 2024-03-24T00:39:48.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.473+0000 7f4a6cff9640 1 -- 172.21.15.67:0/2761858712 <== mon.1 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4a6401a070 con 0x7f4a7015dec0 2024-03-24T00:39:48.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.474+0000 7f4a6cff9640 1 --2- 172.21.15.67:0/2761858712 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4a5c06bbd0 0x7f4a5c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:48.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.474+0000 7f4a6cff9640 1 -- 172.21.15.67:0/2761858712 <== mon.1 v2:172.21.15.72:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f4a6408afd0 con 0x7f4a7015dec0 2024-03-24T00:39:48.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.474+0000 7f4a6ffff640 1 --2- 172.21.15.67:0/2761858712 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4a5c06bbd0 0x7f4a5c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:48.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.474+0000 7f4a6ffff640 1 --2- 172.21.15.67:0/2761858712 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4a5c06bbd0 0x7f4a5c06e090 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4a60036000 tx=0x7f4a6003d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:48.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.479+0000 7f4a6cff9640 1 -- 172.21.15.67:0/2761858712 <== mon.1 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4a64059070 con 0x7f4a7015dec0 2024-03-24T00:39:48.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.698+0000 7f4a76359640 1 -- 172.21.15.67:0/2761858712 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a700d1aa0 con 0x7f4a7015dec0 2024-03-24T00:39:48.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.698+0000 7f4a6cff9640 1 -- 172.21.15.67:0/2761858712 <== mon.1 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4970 (secure 0 0 0) 0x7f4a6405d110 con 0x7f4a7015dec0 2024-03-24T00:39:48.699 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:39:48.699 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4263,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/837528243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":837528243},{"type":"v1","addr":"172.21.15.67:6834","nonce":837528243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3987071456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":3987071456},{"type":"v1","addr":"172.21.15.67:6835","nonce":3987071456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4306,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/3953492281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3953492281},{"type":"v1","addr":"172.21.15.72:6837","nonce":3953492281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:38:52.127871+0000","modified":"2024-03-24T00:39:08.187262+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4309},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4309":{"gid":4309,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6836/3737698654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3737698654},{"type":"v1","addr":"172.21.15.72:6836","nonce":3737698654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4309,"qdb_cluster":[4309]},"id":1}]} 2024-03-24T00:39:48.700 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 9 2024-03-24T00:39:48.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.700+0000 7f4a76359640 1 -- 172.21.15.67:0/2761858712 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4a5c06bbd0 msgr2=0x7f4a5c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:48.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.700+0000 7f4a76359640 1 --2- 172.21.15.67:0/2761858712 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4a5c06bbd0 0x7f4a5c06e090 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4a60036000 tx=0x7f4a6003d040 comp rx=0 tx=0).stop 2024-03-24T00:39:48.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.700+0000 7f4a76359640 1 -- 172.21.15.67:0/2761858712 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a7015dec0 msgr2=0x7f4a700b6090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:48.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.700+0000 7f4a76359640 1 --2- 172.21.15.67:0/2761858712 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a7015dec0 0x7f4a700b6090 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4a6400e870 tx=0x7f4a6400ed30 comp rx=0 tx=0).stop 2024-03-24T00:39:48.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.701+0000 7f4a76359640 1 -- 172.21.15.67:0/2761858712 shutdown_connections 2024-03-24T00:39:48.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.701+0000 7f4a76359640 1 --2- 172.21.15.67:0/2761858712 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4a5c06bbd0 0x7f4a5c06e090 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:48.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.701+0000 7f4a76359640 1 --2- 172.21.15.67:0/2761858712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a7015e2a0 0x7f4a700b4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:48.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.701+0000 7f4a76359640 1 --2- 172.21.15.67:0/2761858712 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a7015dec0 0x7f4a700b6090 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:48.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.701+0000 7f4a76359640 1 --2- 172.21.15.67:0/2761858712 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4a700d07e0 0x7f4a701f3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:48.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.701+0000 7f4a76359640 1 -- 172.21.15.67:0/2761858712 >> 172.21.15.67:0/2761858712 conn(0x7f4a7006c810 msgr2=0x7f4a7015fa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:48.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.701+0000 7f4a76359640 1 -- 172.21.15.67:0/2761858712 shutdown_connections 2024-03-24T00:39:48.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.701+0000 7f4a76359640 1 -- 172.21.15.67:0/2761858712 wait complete. 2024-03-24T00:39:48.712 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:39:48.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.814+0000 7f049f281640 1 Processor -- start 2024-03-24T00:39:48.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.814+0000 7f049f281640 1 -- start start 2024-03-24T00:39:48.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.815+0000 7f049f281640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f049815dec0 0x7f04980ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:48.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.815+0000 7f049f281640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f049815e2a0 0x7f0498162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:48.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.815+0000 7f049f281640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04980d07e0 0x7f04980d0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:48.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.815+0000 7f049f281640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04980d1170 con 0x7f049815e2a0 2024-03-24T00:39:48.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.815+0000 7f049f281640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04980cee30 con 0x7f049815dec0 2024-03-24T00:39:48.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.815+0000 7f049f281640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04980d2120 con 0x7f04980d07e0 2024-03-24T00:39:48.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.815+0000 7f049c81c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f049815e2a0 0x7f0498162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:48.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.815+0000 7f049c81c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f049815e2a0 0x7f0498162630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49968/0 (socket says 172.21.15.67:49968) 2024-03-24T00:39:48.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.815+0000 7f049c81c640 1 -- 172.21.15.67:0/3455591028 learned_addr learned my addr 172.21.15.67:0/3455591028 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:48.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.815+0000 7f049d01d640 1 --2- 172.21.15.67:0/3455591028 >> v2:172.21.15.72:3300/0 conn(0x7f049815dec0 0x7f04980ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:48.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.815+0000 7f049d81e640 1 --2- 172.21.15.67:0/3455591028 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04980d07e0 0x7f04980d0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:48.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.816+0000 7f049c81c640 1 -- 172.21.15.67:0/3455591028 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04980d07e0 msgr2=0x7f04980d0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:48.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.816+0000 7f049c81c640 1 --2- 172.21.15.67:0/3455591028 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04980d07e0 0x7f04980d0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:48.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.816+0000 7f049c81c640 1 -- 172.21.15.67:0/3455591028 >> v2:172.21.15.72:3300/0 conn(0x7f049815dec0 msgr2=0x7f04980ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:48.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.816+0000 7f049c81c640 1 --2- 172.21.15.67:0/3455591028 >> v2:172.21.15.72:3300/0 conn(0x7f049815dec0 0x7f04980ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:48.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.816+0000 7f049c81c640 1 -- 172.21.15.67:0/3455591028 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04980cf640 con 0x7f049815e2a0 2024-03-24T00:39:48.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.816+0000 7f049c81c640 1 --2- 172.21.15.67:0/3455591028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f049815e2a0 0x7f0498162630 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f04980d14a0 tx=0x7f048c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84a974518f83a7ae server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:48.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.817+0000 7f0497fff640 1 -- 172.21.15.67:0/3455591028 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f048c02e070 con 0x7f049815e2a0 2024-03-24T00:39:48.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.817+0000 7f0497fff640 1 -- 172.21.15.67:0/3455591028 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f048c037070 con 0x7f049815e2a0 2024-03-24T00:39:48.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.817+0000 7f0497fff640 1 -- 172.21.15.67:0/3455591028 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f048c031a00 con 0x7f049815e2a0 2024-03-24T00:39:48.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.817+0000 7f049f281640 1 -- 172.21.15.67:0/3455591028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f049815e2a0 msgr2=0x7f0498162630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:48.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.817+0000 7f049f281640 1 --2- 172.21.15.67:0/3455591028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f049815e2a0 0x7f0498162630 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f04980d14a0 tx=0x7f048c02b880 comp rx=0 tx=0).stop 2024-03-24T00:39:48.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.817+0000 7f049f281640 1 -- 172.21.15.67:0/3455591028 shutdown_connections 2024-03-24T00:39:48.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.817+0000 7f049f281640 1 --2- 172.21.15.67:0/3455591028 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04980d07e0 0x7f04980d0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:48.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.817+0000 7f049f281640 1 --2- 172.21.15.67:0/3455591028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f049815e2a0 0x7f0498162630 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:48.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.817+0000 7f049f281640 1 --2- 172.21.15.67:0/3455591028 >> v2:172.21.15.72:3300/0 conn(0x7f049815dec0 0x7f04980ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:48.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.817+0000 7f049f281640 1 -- 172.21.15.67:0/3455591028 >> 172.21.15.67:0/3455591028 conn(0x7f049806c810 msgr2=0x7f049806cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:48.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.817+0000 7f049f281640 1 -- 172.21.15.67:0/3455591028 shutdown_connections 2024-03-24T00:39:48.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.817+0000 7f049f281640 1 -- 172.21.15.67:0/3455591028 wait complete. 2024-03-24T00:39:48.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.818+0000 7f049f281640 1 Processor -- start 2024-03-24T00:39:48.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.818+0000 7f049f281640 1 -- start start 2024-03-24T00:39:48.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.819+0000 7f049f281640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04980d07e0 0x7f04981f4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:48.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.819+0000 7f049f281640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f049815dec0 0x7f04981609c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:48.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.819+0000 7f049f281640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f049815e2a0 0x7f04980b8660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:48.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.819+0000 7f049f281640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04980b5850 con 0x7f049815dec0 2024-03-24T00:39:48.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.819+0000 7f049f281640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04980cf360 con 0x7f049815e2a0 2024-03-24T00:39:48.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.819+0000 7f049f281640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04980b59d0 con 0x7f04980d07e0 2024-03-24T00:39:48.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.819+0000 7f049c81c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f049815dec0 0x7f04981609c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:48.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.819+0000 7f049c81c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f049815dec0 0x7f04981609c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49984/0 (socket says 172.21.15.67:49984) 2024-03-24T00:39:48.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.819+0000 7f049c81c640 1 -- 172.21.15.67:0/1759626892 learned_addr learned my addr 172.21.15.67:0/1759626892 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:48.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.819+0000 7f049d81e640 1 --2- 172.21.15.67:0/1759626892 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f049815e2a0 0x7f04980b8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:48.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.819+0000 7f049d01d640 1 --2- 172.21.15.67:0/1759626892 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04980d07e0 0x7f04981f4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:48.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.819+0000 7f049c81c640 1 -- 172.21.15.67:0/1759626892 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04980d07e0 msgr2=0x7f04981f4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:48.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.820+0000 7f049c81c640 1 --2- 172.21.15.67:0/1759626892 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04980d07e0 0x7f04981f4060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:48.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.820+0000 7f049c81c640 1 -- 172.21.15.67:0/1759626892 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f049815e2a0 msgr2=0x7f04980b8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:48.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.820+0000 7f049c81c640 1 --2- 172.21.15.67:0/1759626892 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f049815e2a0 0x7f04980b8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:48.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.820+0000 7f049c81c640 1 -- 172.21.15.67:0/1759626892 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04981f46b0 con 0x7f049815dec0 2024-03-24T00:39:48.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.820+0000 7f049c81c640 1 --2- 172.21.15.67:0/1759626892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f049815dec0 0x7f04981609c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f048c02b8c0 tx=0x7f048c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:48.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.820+0000 7f0495ffb640 1 -- 172.21.15.67:0/1759626892 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f048c032670 con 0x7f049815dec0 2024-03-24T00:39:48.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.820+0000 7f0495ffb640 1 -- 172.21.15.67:0/1759626892 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f048c032830 con 0x7f049815dec0 2024-03-24T00:39:48.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.820+0000 7f049f281640 1 -- 172.21.15.67:0/1759626892 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04981f4990 con 0x7f049815dec0 2024-03-24T00:39:48.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.820+0000 7f0495ffb640 1 -- 172.21.15.67:0/1759626892 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f048c032c20 con 0x7f049815dec0 2024-03-24T00:39:48.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.820+0000 7f049f281640 1 -- 172.21.15.67:0/1759626892 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0498160f00 con 0x7f049815dec0 2024-03-24T00:39:48.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.821+0000 7f049f281640 1 -- 172.21.15.67:0/1759626892 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04981f5010 con 0x7f049815dec0 2024-03-24T00:39:48.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.822+0000 7f0495ffb640 1 -- 172.21.15.67:0/1759626892 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f048c04c060 con 0x7f049815dec0 2024-03-24T00:39:48.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.822+0000 7f0495ffb640 1 --2- 172.21.15.67:0/1759626892 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f048406bbd0 0x7f048406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:48.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.822+0000 7f0495ffb640 1 -- 172.21.15.67:0/1759626892 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f048c0ac670 con 0x7f049815dec0 2024-03-24T00:39:48.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.822+0000 7f049d01d640 1 --2- 172.21.15.67:0/1759626892 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f048406bbd0 0x7f048406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:48.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.823+0000 7f049d01d640 1 --2- 172.21.15.67:0/1759626892 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f048406bbd0 0x7f048406e090 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0490009ac0 tx=0x7f0490009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:48.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:48.826+0000 7f0495ffb640 1 -- 172.21.15.67:0/1759626892 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f048c040070 con 0x7f049815dec0 2024-03-24T00:39:49.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.047+0000 7f049f281640 1 -- 172.21.15.67:0/1759626892 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04980d1aa0 con 0x7f049815dec0 2024-03-24T00:39:49.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.048+0000 7f0495ffb640 1 -- 172.21.15.67:0/1759626892 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4970 (secure 0 0 0) 0x7f04980cf640 con 0x7f049815dec0 2024-03-24T00:39:49.049 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:39:49.049 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4263,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/837528243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":837528243},{"type":"v1","addr":"172.21.15.67:6834","nonce":837528243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3987071456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":3987071456},{"type":"v1","addr":"172.21.15.67:6835","nonce":3987071456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4306,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/3953492281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3953492281},{"type":"v1","addr":"172.21.15.72:6837","nonce":3953492281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:38:52.127871+0000","modified":"2024-03-24T00:39:08.187262+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4309},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4309":{"gid":4309,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6836/3737698654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3737698654},{"type":"v1","addr":"172.21.15.72:6836","nonce":3737698654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4309,"qdb_cluster":[4309]},"id":1}]} 2024-03-24T00:39:49.049 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 9 2024-03-24T00:39:49.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.050+0000 7f049f281640 1 -- 172.21.15.67:0/1759626892 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f048406bbd0 msgr2=0x7f048406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:49.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.050+0000 7f049f281640 1 --2- 172.21.15.67:0/1759626892 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f048406bbd0 0x7f048406e090 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0490009ac0 tx=0x7f0490009190 comp rx=0 tx=0).stop 2024-03-24T00:39:49.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.050+0000 7f049f281640 1 -- 172.21.15.67:0/1759626892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f049815dec0 msgr2=0x7f04981609c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:49.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.050+0000 7f049f281640 1 --2- 172.21.15.67:0/1759626892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f049815dec0 0x7f04981609c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f048c02b8c0 tx=0x7f048c0046c0 comp rx=0 tx=0).stop 2024-03-24T00:39:49.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.050+0000 7f049f281640 1 -- 172.21.15.67:0/1759626892 shutdown_connections 2024-03-24T00:39:49.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.050+0000 7f049f281640 1 --2- 172.21.15.67:0/1759626892 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f048406bbd0 0x7f048406e090 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:49.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.050+0000 7f049f281640 1 --2- 172.21.15.67:0/1759626892 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f049815e2a0 0x7f04980b8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:49.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.050+0000 7f049f281640 1 --2- 172.21.15.67:0/1759626892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f049815dec0 0x7f04981609c0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:49.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.050+0000 7f049f281640 1 --2- 172.21.15.67:0/1759626892 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04980d07e0 0x7f04981f4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:49.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.050+0000 7f049f281640 1 -- 172.21.15.67:0/1759626892 >> 172.21.15.67:0/1759626892 conn(0x7f049806c810 msgr2=0x7f049815d0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:49.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.050+0000 7f049f281640 1 -- 172.21.15.67:0/1759626892 shutdown_connections 2024-03-24T00:39:49.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.050+0000 7f049f281640 1 -- 172.21.15.67:0/1759626892 wait complete. 2024-03-24T00:39:49.062 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:39:49.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.162+0000 7f3a4df80640 1 Processor -- start 2024-03-24T00:39:49.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.162+0000 7f3a4df80640 1 -- start start 2024-03-24T00:39:49.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.163+0000 7f3a4df80640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3a4815e2a0 0x7f3a480ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:49.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.163+0000 7f3a4df80640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a480cf640 con 0x7f3a4815dec0 2024-03-24T00:39:49.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.163+0000 7f3a4df80640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a480cf080 con 0x7f3a480d0ef0 2024-03-24T00:39:49.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.163+0000 7f3a4df80640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a480cee30 con 0x7f3a4815e2a0 2024-03-24T00:39:49.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.163+0000 7f3a477fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3a4815dec0 0x7f3a480d07e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50510/0 (socket says 172.21.15.67:50510) 2024-03-24T00:39:49.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.163+0000 7f3a477fe640 1 -- 172.21.15.67:0/2936959496 learned_addr learned my addr 172.21.15.67:0/2936959496 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:49.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.163+0000 7f3a46ffd640 1 --2- 172.21.15.67:0/2936959496 >> v2:172.21.15.72:3300/0 conn(0x7f3a4815e2a0 0x7f3a480ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:49.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.163+0000 7f3a467fc640 1 -- 172.21.15.67:0/2936959496 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 951526429 0 0) 0x7f3a38001110 con 0x7f3a4815dec0 2024-03-24T00:39:49.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.164+0000 7f3a467fc640 1 -- 172.21.15.67:0/2936959496 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a30003520 con 0x7f3a4815dec0 2024-03-24T00:39:49.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.164+0000 7f3a467fc640 1 -- 172.21.15.67:0/2936959496 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 833934947 0 0) 0x7f3a480cf080 con 0x7f3a480d0ef0 2024-03-24T00:39:49.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.164+0000 7f3a467fc640 1 -- 172.21.15.67:0/2936959496 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a38001110 con 0x7f3a480d0ef0 2024-03-24T00:39:49.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.164+0000 7f3a46ffd640 1 -- 172.21.15.67:0/2936959496 >> v1:172.21.15.72:6789/0 conn(0x7f3a480d0ef0 legacy=0x7f3a480d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:49.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.164+0000 7f3a46ffd640 1 -- 172.21.15.67:0/2936959496 >> v1:172.21.15.67:6789/0 conn(0x7f3a4815dec0 legacy=0x7f3a480d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:49.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.164+0000 7f3a46ffd640 1 -- 172.21.15.67:0/2936959496 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a480cf360 con 0x7f3a4815e2a0 2024-03-24T00:39:49.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.164+0000 7f3a46ffd640 1 --2- 172.21.15.67:0/2936959496 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a4815e2a0 0x7f3a480ce630 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3a480cefb0 tx=0x7f3a3c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31095f7d78938860 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:49.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.165+0000 7f3a467fc640 1 -- 172.21.15.67:0/2936959496 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c035020 con 0x7f3a4815e2a0 2024-03-24T00:39:49.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.165+0000 7f3a467fc640 1 -- 172.21.15.67:0/2936959496 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a3c031830 con 0x7f3a4815e2a0 2024-03-24T00:39:49.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.165+0000 7f3a467fc640 1 -- 172.21.15.67:0/2936959496 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c031be0 con 0x7f3a4815e2a0 2024-03-24T00:39:49.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.165+0000 7f3a4df80640 1 -- 172.21.15.67:0/2936959496 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a4815e2a0 msgr2=0x7f3a480ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:49.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.165+0000 7f3a4df80640 1 --2- 172.21.15.67:0/2936959496 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a4815e2a0 0x7f3a480ce630 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3a480cefb0 tx=0x7f3a3c02b8d0 comp rx=0 tx=0).stop 2024-03-24T00:39:49.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.165+0000 7f3a4df80640 1 -- 172.21.15.67:0/2936959496 shutdown_connections 2024-03-24T00:39:49.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.165+0000 7f3a4df80640 1 --2- 172.21.15.67:0/2936959496 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a4815e2a0 0x7f3a480ce630 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:49.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.165+0000 7f3a4df80640 1 -- 172.21.15.67:0/2936959496 >> 172.21.15.67:0/2936959496 conn(0x7f3a4806c810 msgr2=0x7f3a4806cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:49.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.165+0000 7f3a4df80640 1 -- 172.21.15.67:0/2936959496 shutdown_connections 2024-03-24T00:39:49.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.165+0000 7f3a4df80640 1 -- 172.21.15.67:0/2936959496 wait complete. 2024-03-24T00:39:49.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.166+0000 7f3a4df80640 1 Processor -- start 2024-03-24T00:39:49.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.167+0000 7f3a4df80640 1 -- start start 2024-03-24T00:39:49.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.167+0000 7f3a4df80640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a480d0ef0 0x7f3a48160bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:49.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.167+0000 7f3a4df80640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3a4815dec0 0x7f3a480b5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:49.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.167+0000 7f3a4df80640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a4815e2a0 0x7f3a481589d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:49.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.167+0000 7f3a4df80640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a48162a40 con 0x7f3a4815e2a0 2024-03-24T00:39:49.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.167+0000 7f3a4df80640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a481628c0 con 0x7f3a480d0ef0 2024-03-24T00:39:49.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.167+0000 7f3a4df80640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a480d2120 con 0x7f3a4815dec0 2024-03-24T00:39:49.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.167+0000 7f3a47fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a4815e2a0 0x7f3a481589d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:49.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.167+0000 7f3a47fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a4815e2a0 0x7f3a481589d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49994/0 (socket says 172.21.15.67:49994) 2024-03-24T00:39:49.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.167+0000 7f3a47fff640 1 -- 172.21.15.67:0/21093418 learned_addr learned my addr 172.21.15.67:0/21093418 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:49.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.167+0000 7f3a477fe640 1 --2- 172.21.15.67:0/21093418 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a480d0ef0 0x7f3a48160bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:49.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.167+0000 7f3a46ffd640 1 --2- 172.21.15.67:0/21093418 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3a4815dec0 0x7f3a480b5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:49.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.168+0000 7f3a47fff640 1 -- 172.21.15.67:0/21093418 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3a4815dec0 msgr2=0x7f3a480b5fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:49.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.168+0000 7f3a47fff640 1 --2- 172.21.15.67:0/21093418 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3a4815dec0 0x7f3a480b5fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:49.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.168+0000 7f3a47fff640 1 -- 172.21.15.67:0/21093418 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a480d0ef0 msgr2=0x7f3a48160bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:49.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.168+0000 7f3a47fff640 1 --2- 172.21.15.67:0/21093418 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a480d0ef0 0x7f3a48160bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:49.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.168+0000 7f3a47fff640 1 -- 172.21.15.67:0/21093418 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a480cb220 con 0x7f3a4815e2a0 2024-03-24T00:39:49.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.168+0000 7f3a47fff640 1 --2- 172.21.15.67:0/21093418 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a4815e2a0 0x7f3a481589d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f3a4000c790 tx=0x7f3a4000cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:49.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.168+0000 7f3a2ffff640 1 -- 172.21.15.67:0/21093418 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a40003b70 con 0x7f3a4815e2a0 2024-03-24T00:39:49.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.168+0000 7f3a2ffff640 1 -- 172.21.15.67:0/21093418 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a40003d10 con 0x7f3a4815e2a0 2024-03-24T00:39:49.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.168+0000 7f3a2ffff640 1 -- 172.21.15.67:0/21093418 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a4000aa50 con 0x7f3a4815e2a0 2024-03-24T00:39:49.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.168+0000 7f3a4df80640 1 -- 172.21.15.67:0/21093418 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a480cb500 con 0x7f3a4815e2a0 2024-03-24T00:39:49.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.168+0000 7f3a4df80640 1 -- 172.21.15.67:0/21093418 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a480cbac0 con 0x7f3a4815e2a0 2024-03-24T00:39:49.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.169+0000 7f3a4df80640 1 -- 172.21.15.67:0/21093418 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a4804f400 con 0x7f3a4815e2a0 2024-03-24T00:39:49.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.169+0000 7f3a2ffff640 1 -- 172.21.15.67:0/21093418 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3a4000abf0 con 0x7f3a4815e2a0 2024-03-24T00:39:49.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.170+0000 7f3a2ffff640 1 --2- 172.21.15.67:0/21093418 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3a3406bbd0 0x7f3a3406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:49.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.170+0000 7f3a2ffff640 1 -- 172.21.15.67:0/21093418 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f3a4008ad40 con 0x7f3a4815e2a0 2024-03-24T00:39:49.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.170+0000 7f3a477fe640 1 --2- 172.21.15.67:0/21093418 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3a3406bbd0 0x7f3a3406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:49.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.170+0000 7f3a477fe640 1 --2- 172.21.15.67:0/21093418 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3a3406bbd0 0x7f3a3406e090 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3a380013b0 tx=0x7f3a38007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:49.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.174+0000 7f3a2ffff640 1 -- 172.21.15.67:0/21093418 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3a40058de0 con 0x7f3a4815e2a0 2024-03-24T00:39:49.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.360+0000 7f3a4df80640 1 -- 172.21.15.67:0/21093418 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3a480d1aa0 con 0x7f3a4815e2a0 2024-03-24T00:39:49.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.361+0000 7f3a2ffff640 1 -- 172.21.15.67:0/21093418 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18843 (secure 0 0 0) 0x7f3a4005ce80 con 0x7f3a4815e2a0 2024-03-24T00:39:49.362 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:39:49.362 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":27,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:39:32.904729+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:38:50.077085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:38:51.095454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T00:38:54.807117+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.14","mappings":[{"from":4,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:39:49.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.363+0000 7f3a4df80640 1 -- 172.21.15.67:0/21093418 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3a3406bbd0 msgr2=0x7f3a3406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:49.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.363+0000 7f3a4df80640 1 --2- 172.21.15.67:0/21093418 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3a3406bbd0 0x7f3a3406e090 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3a380013b0 tx=0x7f3a38007d30 comp rx=0 tx=0).stop 2024-03-24T00:39:49.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.363+0000 7f3a4df80640 1 -- 172.21.15.67:0/21093418 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a4815e2a0 msgr2=0x7f3a481589d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:49.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.363+0000 7f3a4df80640 1 --2- 172.21.15.67:0/21093418 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a4815e2a0 0x7f3a481589d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f3a4000c790 tx=0x7f3a4000cc50 comp rx=0 tx=0).stop 2024-03-24T00:39:49.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.363+0000 7f3a4df80640 1 -- 172.21.15.67:0/21093418 shutdown_connections 2024-03-24T00:39:49.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.363+0000 7f3a4df80640 1 --2- 172.21.15.67:0/21093418 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3a3406bbd0 0x7f3a3406e090 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:49.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.363+0000 7f3a4df80640 1 --2- 172.21.15.67:0/21093418 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a4815e2a0 0x7f3a481589d0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:49.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.363+0000 7f3a4df80640 1 --2- 172.21.15.67:0/21093418 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3a4815dec0 0x7f3a480b5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:49.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.363+0000 7f3a4df80640 1 --2- 172.21.15.67:0/21093418 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a480d0ef0 0x7f3a48160bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:49.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.363+0000 7f3a4df80640 1 -- 172.21.15.67:0/21093418 >> 172.21.15.67:0/21093418 conn(0x7f3a4806c810 msgr2=0x7f3a480b7a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:49.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.364+0000 7f3a4df80640 1 -- 172.21.15.67:0/21093418 shutdown_connections 2024-03-24T00:39:49.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.364+0000 7f3a4df80640 1 -- 172.21.15.67:0/21093418 wait complete. 2024-03-24T00:39:49.375 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-24T00:39:49.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.478+0000 7f27c1459640 1 Processor -- start 2024-03-24T00:39:49.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.478+0000 7f27c1459640 1 -- start start 2024-03-24T00:39:49.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.478+0000 7f27c1459640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f27bc15e250 0x7f27bc0cffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:49.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.478+0000 7f27c1459640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f27bc0d04e0 0x7f27bc0cc620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:49.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.478+0000 7f27c1459640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27bc0d0d10 con 0x7f27bc06cb60 2024-03-24T00:39:49.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.478+0000 7f27c1459640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27bc0d0ff0 con 0x7f27bc0d04e0 2024-03-24T00:39:49.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.478+0000 7f27c1459640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27bc0d12d0 con 0x7f27bc15e250 2024-03-24T00:39:49.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.479+0000 7f27bb7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f27bc0d04e0 0x7f27bc0cc620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:49.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.479+0000 7f27baffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f27bc06cb60 0x7f27bc06cf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50526/0 (socket says 172.21.15.67:50526) 2024-03-24T00:39:49.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.479+0000 7f27baffd640 1 -- 172.21.15.67:0/2730974043 learned_addr learned my addr 172.21.15.67:0/2730974043 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:49.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.479+0000 7f27b3fff640 1 --2- 172.21.15.67:0/2730974043 >> v2:172.21.15.72:3300/0 conn(0x7f27bc15e250 0x7f27bc0cffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:49.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.479+0000 7f27ba7fc640 1 -- 172.21.15.67:0/2730974043 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1920981426 0 0) 0x7f27bc0d0d10 con 0x7f27bc06cb60 2024-03-24T00:39:49.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.479+0000 7f27ba7fc640 1 -- 172.21.15.67:0/2730974043 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27a8003520 con 0x7f27bc06cb60 2024-03-24T00:39:49.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.479+0000 7f27bb7fe640 1 -- 172.21.15.67:0/2730974043 >> v2:172.21.15.72:3300/0 conn(0x7f27bc15e250 msgr2=0x7f27bc0cffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:49.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.479+0000 7f27bb7fe640 1 --2- 172.21.15.67:0/2730974043 >> v2:172.21.15.72:3300/0 conn(0x7f27bc15e250 0x7f27bc0cffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:49.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.479+0000 7f27bb7fe640 1 -- 172.21.15.67:0/2730974043 >> v1:172.21.15.67:6789/0 conn(0x7f27bc06cb60 legacy=0x7f27bc06cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:49.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.479+0000 7f27bb7fe640 1 -- 172.21.15.67:0/2730974043 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27bc0d15b0 con 0x7f27bc0d04e0 2024-03-24T00:39:49.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.480+0000 7f27bb7fe640 1 --2- 172.21.15.67:0/2730974043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27bc0d04e0 0x7f27bc0cc620 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f27bc06fbf0 tx=0x7f27b402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=929f5b09eda8d0d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:49.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.480+0000 7f27ba7fc640 1 -- 172.21.15.67:0/2730974043 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b4039070 con 0x7f27bc0d04e0 2024-03-24T00:39:49.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.480+0000 7f27ba7fc640 1 -- 172.21.15.67:0/2730974043 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27b402cd00 con 0x7f27bc0d04e0 2024-03-24T00:39:49.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.480+0000 7f27ba7fc640 1 -- 172.21.15.67:0/2730974043 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b4034660 con 0x7f27bc0d04e0 2024-03-24T00:39:49.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.481+0000 7f27c1459640 1 -- 172.21.15.67:0/2730974043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27bc0d04e0 msgr2=0x7f27bc0cc620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:49.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.481+0000 7f27c1459640 1 --2- 172.21.15.67:0/2730974043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27bc0d04e0 0x7f27bc0cc620 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f27bc06fbf0 tx=0x7f27b402bf50 comp rx=0 tx=0).stop 2024-03-24T00:39:49.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.481+0000 7f27c1459640 1 -- 172.21.15.67:0/2730974043 shutdown_connections 2024-03-24T00:39:49.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.481+0000 7f27c1459640 1 --2- 172.21.15.67:0/2730974043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27bc0d04e0 0x7f27bc0cc620 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:49.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.481+0000 7f27c1459640 1 --2- 172.21.15.67:0/2730974043 >> v2:172.21.15.72:3300/0 conn(0x7f27bc15e250 0x7f27bc0cffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:49.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.481+0000 7f27c1459640 1 -- 172.21.15.67:0/2730974043 >> 172.21.15.67:0/2730974043 conn(0x7f27bc0d1bf0 msgr2=0x7f27bc0d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:49.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.481+0000 7f27c1459640 1 -- 172.21.15.67:0/2730974043 shutdown_connections 2024-03-24T00:39:49.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.481+0000 7f27c1459640 1 -- 172.21.15.67:0/2730974043 wait complete. 2024-03-24T00:39:49.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.482+0000 7f27c1459640 1 Processor -- start 2024-03-24T00:39:49.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.482+0000 7f27c1459640 1 -- start start 2024-03-24T00:39:49.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.483+0000 7f27c1459640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f27bc06cb60 0x7f27bc151f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:49.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.483+0000 7f27c1459640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27bc0d04e0 0x7f27bc15cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:49.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.483+0000 7f27c1459640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27bc15e250 0x7f27bc15d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:49.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.483+0000 7f27c1459640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc0ccce0 con 0x7f27bc15e250 2024-03-24T00:39:49.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.483+0000 7f27c1459640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc0ccb60 con 0x7f27bc0d04e0 2024-03-24T00:39:49.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.483+0000 7f27c1459640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc0cce60 con 0x7f27bc06cb60 2024-03-24T00:39:49.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.483+0000 7f27bb7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27bc15e250 0x7f27bc15d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:49.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.483+0000 7f27bb7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27bc15e250 0x7f27bc15d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50020/0 (socket says 172.21.15.67:50020) 2024-03-24T00:39:49.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.483+0000 7f27bb7fe640 1 -- 172.21.15.67:0/12413022 learned_addr learned my addr 172.21.15.67:0/12413022 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:49.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.483+0000 7f27b3fff640 1 --2- 172.21.15.67:0/12413022 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27bc0d04e0 0x7f27bc15cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:49.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.483+0000 7f27baffd640 1 --2- 172.21.15.67:0/12413022 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f27bc06cb60 0x7f27bc151f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:49.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.483+0000 7f27bb7fe640 1 -- 172.21.15.67:0/12413022 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f27bc06cb60 msgr2=0x7f27bc151f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:49.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.483+0000 7f27bb7fe640 1 --2- 172.21.15.67:0/12413022 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f27bc06cb60 0x7f27bc151f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:49.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.483+0000 7f27bb7fe640 1 -- 172.21.15.67:0/12413022 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27bc0d04e0 msgr2=0x7f27bc15cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:49.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.483+0000 7f27bb7fe640 1 --2- 172.21.15.67:0/12413022 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27bc0d04e0 0x7f27bc15cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:49.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.483+0000 7f27bb7fe640 1 -- 172.21.15.67:0/12413022 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27bc15e800 con 0x7f27bc15e250 2024-03-24T00:39:49.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.483+0000 7f27bb7fe640 1 --2- 172.21.15.67:0/12413022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27bc15e250 0x7f27bc15d0c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f27b4009e80 tx=0x7f27b4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:49.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.484+0000 7f27b37fe640 1 -- 172.21.15.67:0/12413022 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b4039050 con 0x7f27bc15e250 2024-03-24T00:39:49.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.484+0000 7f27b37fe640 1 -- 172.21.15.67:0/12413022 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27b4045070 con 0x7f27bc15e250 2024-03-24T00:39:49.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.484+0000 7f27b37fe640 1 -- 172.21.15.67:0/12413022 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b4009450 con 0x7f27bc15e250 2024-03-24T00:39:49.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.484+0000 7f27c1459640 1 -- 172.21.15.67:0/12413022 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27bc15eae0 con 0x7f27bc15e250 2024-03-24T00:39:49.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.484+0000 7f27c1459640 1 -- 172.21.15.67:0/12413022 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27bc15f0a0 con 0x7f27bc15e250 2024-03-24T00:39:49.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.485+0000 7f27c1459640 1 -- 172.21.15.67:0/12413022 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27bc04f400 con 0x7f27bc15e250 2024-03-24T00:39:49.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.485+0000 7f27b37fe640 1 -- 172.21.15.67:0/12413022 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f27b403e850 con 0x7f27bc15e250 2024-03-24T00:39:49.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.485+0000 7f27b37fe640 1 --2- 172.21.15.67:0/12413022 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f279c06bc00 0x7f279c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:49.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.486+0000 7f27b37fe640 1 -- 172.21.15.67:0/12413022 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f27b40ad240 con 0x7f27bc15e250 2024-03-24T00:39:49.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.486+0000 7f27baffd640 1 --2- 172.21.15.67:0/12413022 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f279c06bc00 0x7f279c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:49.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.486+0000 7f27baffd640 1 --2- 172.21.15.67:0/12413022 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f279c06bc00 0x7f279c06e0c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f27ac005880 tx=0x7f27ac009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:49.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.490+0000 7f27b37fe640 1 -- 172.21.15.67:0/12413022 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f27b404e020 con 0x7f27bc15e250 2024-03-24T00:39:49.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.709+0000 7f27c1459640 1 -- 172.21.15.67:0/12413022 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f27bc0cd3d0 con 0x7f27bc15e250 2024-03-24T00:39:49.710 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-24T00:39:49.710+0000 7fbfcfa11640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T00:39:49.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.711+0000 7f27b37fe640 1 -- 172.21.15.67:0/12413022 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v10) v1 ==== 161+0+0 (secure 0 0 0) 0x7f27b407b370 con 0x7f27bc15e250 2024-03-24T00:39:49.711 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-24T00:39:49.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.713+0000 7f27c1459640 1 -- 172.21.15.67:0/12413022 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f279c06bc00 msgr2=0x7f279c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:49.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.713+0000 7f27c1459640 1 --2- 172.21.15.67:0/12413022 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f279c06bc00 0x7f279c06e0c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f27ac005880 tx=0x7f27ac009040 comp rx=0 tx=0).stop 2024-03-24T00:39:49.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.713+0000 7f27c1459640 1 -- 172.21.15.67:0/12413022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27bc15e250 msgr2=0x7f27bc15d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:49.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.713+0000 7f27c1459640 1 --2- 172.21.15.67:0/12413022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27bc15e250 0x7f27bc15d0c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f27b4009e80 tx=0x7f27b4004a40 comp rx=0 tx=0).stop 2024-03-24T00:39:49.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.713+0000 7f27c1459640 1 -- 172.21.15.67:0/12413022 shutdown_connections 2024-03-24T00:39:49.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.713+0000 7f27c1459640 1 --2- 172.21.15.67:0/12413022 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f279c06bc00 0x7f279c06e0c0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:49.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.713+0000 7f27c1459640 1 --2- 172.21.15.67:0/12413022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27bc15e250 0x7f27bc15d0c0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:49.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.713+0000 7f27c1459640 1 --2- 172.21.15.67:0/12413022 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27bc0d04e0 0x7f27bc15cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:49.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.713+0000 7f27c1459640 1 --2- 172.21.15.67:0/12413022 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f27bc06cb60 0x7f27bc151f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:49.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.713+0000 7f27c1459640 1 -- 172.21.15.67:0/12413022 >> 172.21.15.67:0/12413022 conn(0x7f27bc0d1bf0 msgr2=0x7f27bc0c7a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:49.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.714+0000 7f27c1459640 1 -- 172.21.15.67:0/12413022 shutdown_connections 2024-03-24T00:39:49.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.714+0000 7f27c1459640 1 -- 172.21.15.67:0/12413022 wait complete. 2024-03-24T00:39:49.716 INFO:tasks.ceph.mds.b.smithi072.stderr: -6112> 2024-03-24T00:38:45.537+0000 7f1f273bb640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2024-03-24T00:39:49.717 INFO:tasks.ceph.mds.b.smithi072.stderr: -4918> 2024-03-24T00:38:56.547+0000 7f1f213af640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T00:39:49.725 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-24T00:39:49.746 INFO:tasks.ceph.mds.b.smithi072.stdout:starting mds.b at 2024-03-24T00:39:49.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.829+0000 7f1936a6c640 1 Processor -- start 2024-03-24T00:39:49.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.829+0000 7f1936a6c640 1 -- start start 2024-03-24T00:39:49.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.830+0000 7f1936a6c640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1930099d00 0x7f193009a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:49.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.830+0000 7f1936a6c640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f193009a620 con 0x7f19300d1530 2024-03-24T00:39:49.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.830+0000 7f1936a6c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19300cc400 con 0x7f19300b49b0 2024-03-24T00:39:49.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.830+0000 7f1936a6c640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19300cc1b0 con 0x7f1930099d00 2024-03-24T00:39:49.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.830+0000 7f1935009640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f19300d1530 0x7f193015d7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50538/0 (socket says 172.21.15.67:50538) 2024-03-24T00:39:49.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.830+0000 7f1935009640 1 -- 172.21.15.67:0/1210353455 learned_addr learned my addr 172.21.15.67:0/1210353455 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:49.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.830+0000 7f1934808640 1 --2- 172.21.15.67:0/1210353455 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1930099d00 0x7f193009a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:49.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.830+0000 7f192f7fe640 1 -- 172.21.15.67:0/1210353455 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3291552849 0 0) 0x7f193009a620 con 0x7f19300d1530 2024-03-24T00:39:49.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.830+0000 7f192f7fe640 1 -- 172.21.15.67:0/1210353455 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1920003520 con 0x7f19300d1530 2024-03-24T00:39:49.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.830+0000 7f192f7fe640 1 -- 172.21.15.67:0/1210353455 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1749193067 0 0) 0x7f19300cc400 con 0x7f19300b49b0 2024-03-24T00:39:49.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.830+0000 7f192f7fe640 1 -- 172.21.15.67:0/1210353455 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f193009a620 con 0x7f19300b49b0 2024-03-24T00:39:49.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.830+0000 7f1934808640 1 -- 172.21.15.67:0/1210353455 >> v1:172.21.15.72:6789/0 conn(0x7f19300b49b0 legacy=0x7f19300d0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:49.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.830+0000 7f1934808640 1 -- 172.21.15.67:0/1210353455 >> v1:172.21.15.67:6789/0 conn(0x7f19300d1530 legacy=0x7f193015d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:49.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.830+0000 7f1934808640 1 -- 172.21.15.67:0/1210353455 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19300cc6e0 con 0x7f1930099d00 2024-03-24T00:39:49.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.831+0000 7f1934808640 1 --2- 172.21.15.67:0/1210353455 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1930099d00 0x7f193009a0e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f19240097d0 tx=0x7f192402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ccfb2697455bcf65 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:49.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.831+0000 7f192f7fe640 1 -- 172.21.15.67:0/1210353455 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1924035020 con 0x7f1930099d00 2024-03-24T00:39:49.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.831+0000 7f192f7fe640 1 -- 172.21.15.67:0/1210353455 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1924031890 con 0x7f1930099d00 2024-03-24T00:39:49.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.831+0000 7f192f7fe640 1 -- 172.21.15.67:0/1210353455 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1924031c60 con 0x7f1930099d00 2024-03-24T00:39:49.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.832+0000 7f1936a6c640 1 -- 172.21.15.67:0/1210353455 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1930099d00 msgr2=0x7f193009a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:49.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.832+0000 7f1936a6c640 1 --2- 172.21.15.67:0/1210353455 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1930099d00 0x7f193009a0e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f19240097d0 tx=0x7f192402b930 comp rx=0 tx=0).stop 2024-03-24T00:39:49.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.832+0000 7f1936a6c640 1 -- 172.21.15.67:0/1210353455 shutdown_connections 2024-03-24T00:39:49.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.832+0000 7f1936a6c640 1 --2- 172.21.15.67:0/1210353455 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1930099d00 0x7f193009a0e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:49.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.832+0000 7f1936a6c640 1 -- 172.21.15.67:0/1210353455 >> 172.21.15.67:0/1210353455 conn(0x7f19300b3650 msgr2=0x7f19300b3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:49.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.832+0000 7f1936a6c640 1 -- 172.21.15.67:0/1210353455 shutdown_connections 2024-03-24T00:39:49.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.832+0000 7f1936a6c640 1 -- 172.21.15.67:0/1210353455 wait complete. 2024-03-24T00:39:49.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.833+0000 7f1936a6c640 1 Processor -- start 2024-03-24T00:39:49.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.833+0000 7f1936a6c640 1 -- start start 2024-03-24T00:39:49.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.834+0000 7f1936a6c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1930099d00 0x7f19300c6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:49.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.834+0000 7f1936a6c640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f19300b49b0 0x7f19300c7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:49.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.834+0000 7f1936a6c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19300b5ec0 0x7f19300b6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:49.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.834+0000 7f1936a6c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19300d01c0 con 0x7f1930099d00 2024-03-24T00:39:49.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.834+0000 7f1936a6c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19300ccca0 con 0x7f19300b5ec0 2024-03-24T00:39:49.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.834+0000 7f1936a6c640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19300b32a0 con 0x7f19300b49b0 2024-03-24T00:39:49.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.834+0000 7f1934808640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1930099d00 0x7f19300c6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:49.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.834+0000 7f1934808640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1930099d00 0x7f19300c6130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50032/0 (socket says 172.21.15.67:50032) 2024-03-24T00:39:49.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.834+0000 7f1934808640 1 -- 172.21.15.67:0/3820574059 learned_addr learned my addr 172.21.15.67:0/3820574059 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:49.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.834+0000 7f1935009640 1 --2- 172.21.15.67:0/3820574059 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19300b5ec0 0x7f19300b6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:49.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.834+0000 7f192ffff640 1 --2- 172.21.15.67:0/3820574059 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f19300b49b0 0x7f19300c7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:49.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.834+0000 7f1934808640 1 -- 172.21.15.67:0/3820574059 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f19300b49b0 msgr2=0x7f19300c7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:49.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.834+0000 7f1934808640 1 --2- 172.21.15.67:0/3820574059 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f19300b49b0 0x7f19300c7910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:49.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.834+0000 7f1934808640 1 -- 172.21.15.67:0/3820574059 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19300b5ec0 msgr2=0x7f19300b6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:49.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.834+0000 7f1934808640 1 --2- 172.21.15.67:0/3820574059 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19300b5ec0 0x7f19300b6350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:49.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.834+0000 7f1934808640 1 -- 172.21.15.67:0/3820574059 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19300c6670 con 0x7f1930099d00 2024-03-24T00:39:49.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.834+0000 7f1934808640 1 --2- 172.21.15.67:0/3820574059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1930099d00 0x7f19300c6130 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f19240098f0 tx=0x7f19240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:49.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.834+0000 7f192d7fa640 1 -- 172.21.15.67:0/3820574059 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1924035040 con 0x7f1930099d00 2024-03-24T00:39:49.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.834+0000 7f192d7fa640 1 -- 172.21.15.67:0/3820574059 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1924030d20 con 0x7f1930099d00 2024-03-24T00:39:49.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.834+0000 7f1936a6c640 1 -- 172.21.15.67:0/3820574059 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19300c6f80 con 0x7f1930099d00 2024-03-24T00:39:49.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.835+0000 7f192d7fa640 1 -- 172.21.15.67:0/3820574059 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1924006e10 con 0x7f1930099d00 2024-03-24T00:39:49.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.835+0000 7f1936a6c640 1 -- 172.21.15.67:0/3820574059 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19300c7520 con 0x7f1930099d00 2024-03-24T00:39:49.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.836+0000 7f1936a6c640 1 -- 172.21.15.67:0/3820574059 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f193004f400 con 0x7f1930099d00 2024-03-24T00:39:49.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.836+0000 7f192d7fa640 1 -- 172.21.15.67:0/3820574059 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1924038020 con 0x7f1930099d00 2024-03-24T00:39:49.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.836+0000 7f192d7fa640 1 --2- 172.21.15.67:0/3820574059 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f191406bc00 0x7f191406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:49.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.836+0000 7f192d7fa640 1 -- 172.21.15.67:0/3820574059 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f19240ad5a0 con 0x7f1930099d00 2024-03-24T00:39:49.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.837+0000 7f192ffff640 1 --2- 172.21.15.67:0/3820574059 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f191406bc00 0x7f191406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:49.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.837+0000 7f192ffff640 1 --2- 172.21.15.67:0/3820574059 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f191406bc00 0x7f191406e0c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f191c007fd0 tx=0x7f191c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:49.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:49.841+0000 7f192d7fa640 1 -- 172.21.15.67:0/3820574059 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f192402d040 con 0x7f1930099d00 2024-03-24T00:39:50.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.062+0000 7f1936a6c640 1 -- 172.21.15.67:0/3820574059 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f19301616a0 con 0x7f1930099d00 2024-03-24T00:39:50.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.712+0000 7f192d7fa640 1 -- 172.21.15.67:0/3820574059 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7f192407b5e0 con 0x7f1930099d00 2024-03-24T00:39:50.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.714+0000 7f1936a6c640 1 -- 172.21.15.67:0/3820574059 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f191406bc00 msgr2=0x7f191406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:50.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.714+0000 7f1936a6c640 1 --2- 172.21.15.67:0/3820574059 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f191406bc00 0x7f191406e0c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f191c007fd0 tx=0x7f191c009040 comp rx=0 tx=0).stop 2024-03-24T00:39:50.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.714+0000 7f1936a6c640 1 -- 172.21.15.67:0/3820574059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1930099d00 msgr2=0x7f19300c6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:50.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.714+0000 7f1936a6c640 1 --2- 172.21.15.67:0/3820574059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1930099d00 0x7f19300c6130 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f19240098f0 tx=0x7f19240046c0 comp rx=0 tx=0).stop 2024-03-24T00:39:50.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.715+0000 7f1936a6c640 1 -- 172.21.15.67:0/3820574059 shutdown_connections 2024-03-24T00:39:50.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.715+0000 7f1936a6c640 1 --2- 172.21.15.67:0/3820574059 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f191406bc00 0x7f191406e0c0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:50.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.715+0000 7f1936a6c640 1 --2- 172.21.15.67:0/3820574059 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19300b5ec0 0x7f19300b6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:50.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.715+0000 7f1936a6c640 1 --2- 172.21.15.67:0/3820574059 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f19300b49b0 0x7f19300c7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:50.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.715+0000 7f1936a6c640 1 --2- 172.21.15.67:0/3820574059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1930099d00 0x7f19300c6130 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:50.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.715+0000 7f1936a6c640 1 -- 172.21.15.67:0/3820574059 >> 172.21.15.67:0/3820574059 conn(0x7f19300b3650 msgr2=0x7f193006d530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:50.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.715+0000 7f1936a6c640 1 -- 172.21.15.67:0/3820574059 shutdown_connections 2024-03-24T00:39:50.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.715+0000 7f1936a6c640 1 -- 172.21.15.67:0/3820574059 wait complete. 2024-03-24T00:39:50.729 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-24T00:39:50.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.839+0000 7f230e7b3640 1 Processor -- start 2024-03-24T00:39:50.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.839+0000 7f230e7b3640 1 -- start start 2024-03-24T00:39:50.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.840+0000 7f230e7b3640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f23081589a0 0x7f23080725b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:50.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.840+0000 7f230e7b3640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f230815bed0 0x7f23080cc390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:50.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.840+0000 7f230e7b3640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23080b4b40 con 0x7f230806ec40 2024-03-24T00:39:50.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.840+0000 7f230e7b3640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2308072af0 con 0x7f23081589a0 2024-03-24T00:39:50.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.840+0000 7f230e7b3640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23080b48f0 con 0x7f230815bed0 2024-03-24T00:39:50.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.840+0000 7f2307fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f23081589a0 0x7f23080725b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:50.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.840+0000 7f2307fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f23081589a0 0x7f23080725b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50044/0 (socket says 172.21.15.67:50044) 2024-03-24T00:39:50.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.840+0000 7f2307fff640 1 -- 172.21.15.67:0/2815953078 learned_addr learned my addr 172.21.15.67:0/2815953078 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:50.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.840+0000 7f23077fe640 1 --2- 172.21.15.67:0/2815953078 >> v2:172.21.15.72:3300/0 conn(0x7f230815bed0 0x7f23080cc390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:50.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.840+0000 7f2307fff640 1 -- 172.21.15.67:0/2815953078 >> v2:172.21.15.72:3300/0 conn(0x7f230815bed0 msgr2=0x7f23080cc390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:50.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.840+0000 7f2307fff640 1 --2- 172.21.15.67:0/2815953078 >> v2:172.21.15.72:3300/0 conn(0x7f230815bed0 0x7f23080cc390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:50.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.840+0000 7f2307fff640 1 -- 172.21.15.67:0/2815953078 >> v1:172.21.15.72:6789/0 conn(0x7f230806ec40 legacy=0x7f23080cddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:50.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.840+0000 7f2307fff640 1 -- 172.21.15.67:0/2815953078 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23080b4e20 con 0x7f23081589a0 2024-03-24T00:39:50.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.841+0000 7f2307fff640 1 --2- 172.21.15.67:0/2815953078 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23081589a0 0x7f23080725b0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f23080b5100 tx=0x7f22fc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7acd6fbc1f918833 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:50.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.841+0000 7f2306ffd640 1 -- 172.21.15.67:0/2815953078 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22fc035020 con 0x7f23081589a0 2024-03-24T00:39:50.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.841+0000 7f2306ffd640 1 -- 172.21.15.67:0/2815953078 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22fc031870 con 0x7f23081589a0 2024-03-24T00:39:50.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.841+0000 7f2306ffd640 1 -- 172.21.15.67:0/2815953078 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22fc031c40 con 0x7f23081589a0 2024-03-24T00:39:50.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.842+0000 7f230e7b3640 1 -- 172.21.15.67:0/2815953078 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23081589a0 msgr2=0x7f23080725b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:50.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.842+0000 7f230e7b3640 1 --2- 172.21.15.67:0/2815953078 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23081589a0 0x7f23080725b0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f23080b5100 tx=0x7f22fc02b910 comp rx=0 tx=0).stop 2024-03-24T00:39:50.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.842+0000 7f230e7b3640 1 -- 172.21.15.67:0/2815953078 shutdown_connections 2024-03-24T00:39:50.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.842+0000 7f230e7b3640 1 --2- 172.21.15.67:0/2815953078 >> v2:172.21.15.72:3300/0 conn(0x7f230815bed0 0x7f23080cc390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:50.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.842+0000 7f230e7b3640 1 --2- 172.21.15.67:0/2815953078 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23081589a0 0x7f23080725b0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:50.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.842+0000 7f230e7b3640 1 -- 172.21.15.67:0/2815953078 >> 172.21.15.67:0/2815953078 conn(0x7f23080cf9e0 msgr2=0x7f23080cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:50.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.842+0000 7f230e7b3640 1 -- 172.21.15.67:0/2815953078 shutdown_connections 2024-03-24T00:39:50.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.842+0000 7f230e7b3640 1 -- 172.21.15.67:0/2815953078 wait complete. 2024-03-24T00:39:50.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.843+0000 7f230e7b3640 1 Processor -- start 2024-03-24T00:39:50.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.843+0000 7f230e7b3640 1 -- start start 2024-03-24T00:39:50.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.844+0000 7f230e7b3640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f230806ec40 0x7f230814acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:50.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.844+0000 7f230e7b3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23081589a0 0x7f230814b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:50.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.844+0000 7f230e7b3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f230815bed0 0x7f230814c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:50.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.844+0000 7f230e7b3640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23080cc8d0 con 0x7f230815bed0 2024-03-24T00:39:50.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.844+0000 7f230e7b3640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2308071db0 con 0x7f23081589a0 2024-03-24T00:39:50.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.844+0000 7f230e7b3640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f230806f090 con 0x7f230806ec40 2024-03-24T00:39:50.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.844+0000 7f230cd50640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f230815bed0 0x7f230814c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:50.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.844+0000 7f230cd50640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f230815bed0 0x7f230814c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50048/0 (socket says 172.21.15.67:50048) 2024-03-24T00:39:50.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.844+0000 7f230cd50640 1 -- 172.21.15.67:0/1644680953 learned_addr learned my addr 172.21.15.67:0/1644680953 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:50.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.844+0000 7f230cd50640 1 -- 172.21.15.67:0/1644680953 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f230806ec40 msgr2=0x7f230814acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:50.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.844+0000 7f230cd50640 1 --2- 172.21.15.67:0/1644680953 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f230806ec40 0x7f230814acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:50.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.844+0000 7f230cd50640 1 -- 172.21.15.67:0/1644680953 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23081589a0 msgr2=0x7f230814b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:50.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.844+0000 7f230cd50640 1 --2- 172.21.15.67:0/1644680953 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23081589a0 0x7f230814b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:50.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.844+0000 7f230cd50640 1 -- 172.21.15.67:0/1644680953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23081f55a0 con 0x7f230815bed0 2024-03-24T00:39:50.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.845+0000 7f230cd50640 1 --2- 172.21.15.67:0/1644680953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f230815bed0 0x7f230814c950 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f230000c780 tx=0x7f230000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:50.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.845+0000 7f2304ff9640 1 -- 172.21.15.67:0/1644680953 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2300017070 con 0x7f230815bed0 2024-03-24T00:39:50.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.845+0000 7f2304ff9640 1 -- 172.21.15.67:0/1644680953 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2300003da0 con 0x7f230815bed0 2024-03-24T00:39:50.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.845+0000 7f230e7b3640 1 -- 172.21.15.67:0/1644680953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23081f5880 con 0x7f230815bed0 2024-03-24T00:39:50.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.845+0000 7f2304ff9640 1 -- 172.21.15.67:0/1644680953 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23000052b0 con 0x7f230815bed0 2024-03-24T00:39:50.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.845+0000 7f230e7b3640 1 -- 172.21.15.67:0/1644680953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23081f5e40 con 0x7f230815bed0 2024-03-24T00:39:50.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.846+0000 7f230e7b3640 1 -- 172.21.15.67:0/1644680953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f230804f400 con 0x7f230815bed0 2024-03-24T00:39:50.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.846+0000 7f2304ff9640 1 -- 172.21.15.67:0/1644680953 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f230001a070 con 0x7f230815bed0 2024-03-24T00:39:50.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.846+0000 7f2304ff9640 1 --2- 172.21.15.67:0/1644680953 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f22f406bbd0 0x7f22f406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:50.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.847+0000 7f2304ff9640 1 -- 172.21.15.67:0/1644680953 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f230008a5c0 con 0x7f230815bed0 2024-03-24T00:39:50.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.847+0000 7f2307fff640 1 --2- 172.21.15.67:0/1644680953 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f22f406bbd0 0x7f22f406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:50.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.847+0000 7f2307fff640 1 --2- 172.21.15.67:0/1644680953 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f22f406bbd0 0x7f22f406e090 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f22fc036000 tx=0x7f22fc030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:50.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:50.851+0000 7f2304ff9640 1 -- 172.21.15.67:0/1644680953 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2300058c50 con 0x7f230815bed0 2024-03-24T00:39:51.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:51.033+0000 7f230e7b3640 1 -- 172.21.15.67:0/1644680953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2308156bf0 con 0x7f230815bed0 2024-03-24T00:39:51.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:51.972+0000 7f2304ff9640 1 -- 172.21.15.67:0/1644680953 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v29) v1 ==== 181+0+0 (secure 0 0 0) 0x7f2300058550 con 0x7f230815bed0 2024-03-24T00:39:51.973 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' removed 2024-03-24T00:39:51.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:51.976+0000 7f230e7b3640 1 -- 172.21.15.67:0/1644680953 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f22f406bbd0 msgr2=0x7f22f406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:51.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:51.976+0000 7f230e7b3640 1 --2- 172.21.15.67:0/1644680953 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f22f406bbd0 0x7f22f406e090 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f22fc036000 tx=0x7f22fc030000 comp rx=0 tx=0).stop 2024-03-24T00:39:51.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:51.976+0000 7f230e7b3640 1 -- 172.21.15.67:0/1644680953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f230815bed0 msgr2=0x7f230814c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:51.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:51.976+0000 7f230e7b3640 1 --2- 172.21.15.67:0/1644680953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f230815bed0 0x7f230814c950 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f230000c780 tx=0x7f230000cc40 comp rx=0 tx=0).stop 2024-03-24T00:39:51.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:51.976+0000 7f230e7b3640 1 -- 172.21.15.67:0/1644680953 shutdown_connections 2024-03-24T00:39:51.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:51.976+0000 7f230e7b3640 1 --2- 172.21.15.67:0/1644680953 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f22f406bbd0 0x7f22f406e090 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:51.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:51.976+0000 7f230e7b3640 1 --2- 172.21.15.67:0/1644680953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f230815bed0 0x7f230814c950 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:51.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:51.976+0000 7f230e7b3640 1 --2- 172.21.15.67:0/1644680953 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23081589a0 0x7f230814b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:51.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:51.976+0000 7f230e7b3640 1 --2- 172.21.15.67:0/1644680953 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f230806ec40 0x7f230814acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:51.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:51.976+0000 7f230e7b3640 1 -- 172.21.15.67:0/1644680953 >> 172.21.15.67:0/1644680953 conn(0x7f23080cf9e0 msgr2=0x7f230815b100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:51.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:51.977+0000 7f230e7b3640 1 -- 172.21.15.67:0/1644680953 shutdown_connections 2024-03-24T00:39:51.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:51.977+0000 7f230e7b3640 1 -- 172.21.15.67:0/1644680953 wait complete. 2024-03-24T00:39:51.994 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-24T00:39:52.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.104+0000 7f7e0f1f4640 1 Processor -- start 2024-03-24T00:39:52.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.104+0000 7f7e0f1f4640 1 -- start start 2024-03-24T00:39:52.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.105+0000 7f7e0f1f4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7e080d0840 0x7f7e080c56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:52.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.105+0000 7f7e0f1f4640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e08099d70 con 0x7f7e0809ae10 2024-03-24T00:39:52.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.105+0000 7f7e0f1f4640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e0809a050 con 0x7f7e0815bc40 2024-03-24T00:39:52.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.105+0000 7f7e0f1f4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e0809b270 con 0x7f7e080d0840 2024-03-24T00:39:52.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.105+0000 7f7e07fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7e0809ae10 0x7f7e080d0130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50552/0 (socket says 172.21.15.67:50552) 2024-03-24T00:39:52.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.105+0000 7f7e07fff640 1 -- 172.21.15.67:0/2375796790 learned_addr learned my addr 172.21.15.67:0/2375796790 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:52.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.105+0000 7f7e0d791640 1 --2- 172.21.15.67:0/2375796790 >> v2:172.21.15.72:3300/0 conn(0x7f7e080d0840 0x7f7e080c56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:52.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.106+0000 7f7e077fe640 1 -- 172.21.15.67:0/2375796790 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 228488502 0 0) 0x7f7e08099d70 con 0x7f7e0809ae10 2024-03-24T00:39:52.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.106+0000 7f7e077fe640 1 -- 172.21.15.67:0/2375796790 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7df0003520 con 0x7f7e0809ae10 2024-03-24T00:39:52.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.106+0000 7f7e077fe640 1 -- 172.21.15.67:0/2375796790 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 567001669 0 0) 0x7f7e0809a050 con 0x7f7e0815bc40 2024-03-24T00:39:52.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.106+0000 7f7e077fe640 1 -- 172.21.15.67:0/2375796790 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e08099d70 con 0x7f7e0815bc40 2024-03-24T00:39:52.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.106+0000 7f7e0d791640 1 -- 172.21.15.67:0/2375796790 >> v1:172.21.15.72:6789/0 conn(0x7f7e0815bc40 legacy=0x7f7e080cd130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:52.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.106+0000 7f7e0d791640 1 -- 172.21.15.67:0/2375796790 >> v1:172.21.15.67:6789/0 conn(0x7f7e0809ae10 legacy=0x7f7e080d0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:52.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.106+0000 7f7e0d791640 1 -- 172.21.15.67:0/2375796790 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e0809a330 con 0x7f7e080d0840 2024-03-24T00:39:52.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.107+0000 7f7e0d791640 1 --2- 172.21.15.67:0/2375796790 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e080d0840 0x7f7e080c56a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7df80097a0 tx=0x7f7df802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc3e34ea3f12de9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:52.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.107+0000 7f7e077fe640 1 -- 172.21.15.67:0/2375796790 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df8034070 con 0x7f7e080d0840 2024-03-24T00:39:52.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.107+0000 7f7e077fe640 1 -- 172.21.15.67:0/2375796790 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7df8030610 con 0x7f7e080d0840 2024-03-24T00:39:52.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.107+0000 7f7e077fe640 1 -- 172.21.15.67:0/2375796790 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df80309c0 con 0x7f7e080d0840 2024-03-24T00:39:52.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.107+0000 7f7e0f1f4640 1 -- 172.21.15.67:0/2375796790 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e080d0840 msgr2=0x7f7e080c56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:52.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.107+0000 7f7e0f1f4640 1 --2- 172.21.15.67:0/2375796790 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e080d0840 0x7f7e080c56a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7df80097a0 tx=0x7f7df802b960 comp rx=0 tx=0).stop 2024-03-24T00:39:52.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.107+0000 7f7e0f1f4640 1 -- 172.21.15.67:0/2375796790 shutdown_connections 2024-03-24T00:39:52.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.107+0000 7f7e0f1f4640 1 --2- 172.21.15.67:0/2375796790 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e080d0840 0x7f7e080c56a0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:52.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.107+0000 7f7e0f1f4640 1 -- 172.21.15.67:0/2375796790 >> 172.21.15.67:0/2375796790 conn(0x7f7e080b3640 msgr2=0x7f7e080b3a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:52.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.107+0000 7f7e0f1f4640 1 -- 172.21.15.67:0/2375796790 shutdown_connections 2024-03-24T00:39:52.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.107+0000 7f7e0f1f4640 1 -- 172.21.15.67:0/2375796790 wait complete. 2024-03-24T00:39:52.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.109+0000 7f7e0f1f4640 1 Processor -- start 2024-03-24T00:39:52.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.109+0000 7f7e0f1f4640 1 -- start start 2024-03-24T00:39:52.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.109+0000 7f7e0f1f4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e0809ae10 0x7f7e081510e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:52.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.109+0000 7f7e0f1f4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7e080d0840 0x7f7e0806d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:52.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.109+0000 7f7e0f1f4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e0815bc40 0x7f7e080b6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:52.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.109+0000 7f7e0f1f4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e080cf950 con 0x7f7e0809ae10 2024-03-24T00:39:52.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.109+0000 7f7e0f1f4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0809a8f0 con 0x7f7e0815bc40 2024-03-24T00:39:52.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.109+0000 7f7e0f1f4640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e080cd840 con 0x7f7e080d0840 2024-03-24T00:39:52.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.110+0000 7f7e0cf90640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e0809ae10 0x7f7e081510e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:52.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.110+0000 7f7e0cf90640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e0809ae10 0x7f7e081510e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50052/0 (socket says 172.21.15.67:50052) 2024-03-24T00:39:52.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.110+0000 7f7e0cf90640 1 -- 172.21.15.67:0/954006220 learned_addr learned my addr 172.21.15.67:0/954006220 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:52.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.110+0000 7f7e0d791640 1 --2- 172.21.15.67:0/954006220 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e0815bc40 0x7f7e080b6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:52.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.110+0000 7f7e07fff640 1 --2- 172.21.15.67:0/954006220 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7e080d0840 0x7f7e0806d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:52.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.110+0000 7f7e0cf90640 1 -- 172.21.15.67:0/954006220 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7e080d0840 msgr2=0x7f7e0806d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:52.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.110+0000 7f7e0cf90640 1 --2- 172.21.15.67:0/954006220 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7e080d0840 0x7f7e0806d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:52.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.110+0000 7f7e0cf90640 1 -- 172.21.15.67:0/954006220 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e0815bc40 msgr2=0x7f7e080b6cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:52.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.110+0000 7f7e0cf90640 1 --2- 172.21.15.67:0/954006220 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e0815bc40 0x7f7e080b6cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:52.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.110+0000 7f7e0cf90640 1 -- 172.21.15.67:0/954006220 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e08151620 con 0x7f7e0809ae10 2024-03-24T00:39:52.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.110+0000 7f7e0cf90640 1 --2- 172.21.15.67:0/954006220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e0809ae10 0x7f7e081510e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f7e0000c7d0 tx=0x7f7e0000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:52.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.110+0000 7f7e057fa640 1 -- 172.21.15.67:0/954006220 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e00003b70 con 0x7f7e0809ae10 2024-03-24T00:39:52.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.110+0000 7f7e057fa640 1 -- 172.21.15.67:0/954006220 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e00003d30 con 0x7f7e0809ae10 2024-03-24T00:39:52.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.110+0000 7f7e0f1f4640 1 -- 172.21.15.67:0/954006220 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e0815e770 con 0x7f7e0809ae10 2024-03-24T00:39:52.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.110+0000 7f7e057fa640 1 -- 172.21.15.67:0/954006220 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e0000aa30 con 0x7f7e0809ae10 2024-03-24T00:39:52.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.110+0000 7f7e0f1f4640 1 -- 172.21.15.67:0/954006220 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e0815ecf0 con 0x7f7e0809ae10 2024-03-24T00:39:52.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.111+0000 7f7e0f1f4640 1 -- 172.21.15.67:0/954006220 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e0804f400 con 0x7f7e0809ae10 2024-03-24T00:39:52.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.112+0000 7f7e057fa640 1 -- 172.21.15.67:0/954006220 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7e0000abd0 con 0x7f7e0809ae10 2024-03-24T00:39:52.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.112+0000 7f7e057fa640 1 --2- 172.21.15.67:0/954006220 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7df406bbd0 0x7f7df406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:52.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.112+0000 7f7e057fa640 1 -- 172.21.15.67:0/954006220 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6383+0+0 (secure 0 0 0) 0x7f7e0008aad0 con 0x7f7e0809ae10 2024-03-24T00:39:52.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.112+0000 7f7e07fff640 1 --2- 172.21.15.67:0/954006220 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7df406bbd0 0x7f7df406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:52.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.113+0000 7f7e07fff640 1 --2- 172.21.15.67:0/954006220 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7df406bbd0 0x7f7df406e090 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f7e080cc0e0 tx=0x7f7dfc007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:52.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.117+0000 7f7e057fa640 1 -- 172.21.15.67:0/954006220 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7e08151620 con 0x7f7e0809ae10 2024-03-24T00:39:52.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.298+0000 7f7e0f1f4640 1 -- 172.21.15.67:0/954006220 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7e080b4550 con 0x7f7e0809ae10 2024-03-24T00:39:52.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.977+0000 7f7e057fa640 1 -- 172.21.15.67:0/954006220 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v30) v1 ==== 169+0+0 (secure 0 0 0) 0x7f7e00058d60 con 0x7f7e0809ae10 2024-03-24T00:39:52.978 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' removed 2024-03-24T00:39:52.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.981+0000 7f7e0f1f4640 1 -- 172.21.15.67:0/954006220 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7df406bbd0 msgr2=0x7f7df406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:52.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.981+0000 7f7e0f1f4640 1 --2- 172.21.15.67:0/954006220 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7df406bbd0 0x7f7df406e090 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f7e080cc0e0 tx=0x7f7dfc007d30 comp rx=0 tx=0).stop 2024-03-24T00:39:52.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.981+0000 7f7e0f1f4640 1 -- 172.21.15.67:0/954006220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e0809ae10 msgr2=0x7f7e081510e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:52.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.981+0000 7f7e0f1f4640 1 --2- 172.21.15.67:0/954006220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e0809ae10 0x7f7e081510e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f7e0000c7d0 tx=0x7f7e0000cc90 comp rx=0 tx=0).stop 2024-03-24T00:39:52.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.982+0000 7f7e0f1f4640 1 -- 172.21.15.67:0/954006220 shutdown_connections 2024-03-24T00:39:52.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.982+0000 7f7e0f1f4640 1 --2- 172.21.15.67:0/954006220 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7df406bbd0 0x7f7df406e090 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:52.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.982+0000 7f7e0f1f4640 1 --2- 172.21.15.67:0/954006220 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e0815bc40 0x7f7e080b6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:52.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.982+0000 7f7e0f1f4640 1 --2- 172.21.15.67:0/954006220 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7e080d0840 0x7f7e0806d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:52.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.982+0000 7f7e0f1f4640 1 --2- 172.21.15.67:0/954006220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e0809ae10 0x7f7e081510e0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:52.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.982+0000 7f7e0f1f4640 1 -- 172.21.15.67:0/954006220 >> 172.21.15.67:0/954006220 conn(0x7f7e080b3640 msgr2=0x7f7e080cfad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:52.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.982+0000 7f7e0f1f4640 1 -- 172.21.15.67:0/954006220 shutdown_connections 2024-03-24T00:39:52.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:52.982+0000 7f7e0f1f4640 1 -- 172.21.15.67:0/954006220 wait complete. 2024-03-24T00:39:53.000 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-03-24T00:39:53.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.111+0000 7ff278737640 1 Processor -- start 2024-03-24T00:39:53.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.111+0000 7ff278737640 1 -- start start 2024-03-24T00:39:53.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.112+0000 7ff278737640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff2700cdc10 0x7ff2700cfd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:53.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.112+0000 7ff278737640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff27006d270 0x7ff2700cc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:53.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.112+0000 7ff278737640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2700cdff0 con 0x7ff270161ac0 2024-03-24T00:39:53.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.112+0000 7ff278737640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2700d06a0 con 0x7ff2700cdc10 2024-03-24T00:39:53.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.112+0000 7ff278737640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2700d0980 con 0x7ff27006d270 2024-03-24T00:39:53.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.112+0000 7ff276cd4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff27006d270 0x7ff2700cc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:53.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.112+0000 7ff2764d3640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff2700cdc10 0x7ff2700cfd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:53.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.112+0000 7ff275cd2640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7ff270161ac0 0x7ff27006cb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:43838/0 (socket says 172.21.15.67:43838) 2024-03-24T00:39:53.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.112+0000 7ff275cd2640 1 -- 172.21.15.67:0/2095233139 learned_addr learned my addr 172.21.15.67:0/2095233139 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:53.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.112+0000 7ff276cd4640 1 -- 172.21.15.67:0/2095233139 >> v2:172.21.15.72:3300/0 conn(0x7ff2700cdc10 msgr2=0x7ff2700cfd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:53.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.112+0000 7ff276cd4640 1 --2- 172.21.15.67:0/2095233139 >> v2:172.21.15.72:3300/0 conn(0x7ff2700cdc10 0x7ff2700cfd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:53.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.112+0000 7ff276cd4640 1 -- 172.21.15.67:0/2095233139 >> v1:172.21.15.72:6789/0 conn(0x7ff270161ac0 legacy=0x7ff27006cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:53.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.112+0000 7ff276cd4640 1 -- 172.21.15.67:0/2095233139 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2700d0c60 con 0x7ff27006d270 2024-03-24T00:39:53.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.112+0000 7ff2764d3640 1 --2- 172.21.15.67:0/2095233139 >> v2:172.21.15.72:3300/0 conn(0x7ff2700cdc10 0x7ff2700cfd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:53.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.113+0000 7ff276cd4640 1 --2- 172.21.15.67:0/2095233139 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff27006d270 0x7ff2700cc1a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff268009d90 tx=0x7ff26802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff525cc9f34e086c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:53.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.113+0000 7ff2754d1640 1 -- 172.21.15.67:0/2095233139 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff268039070 con 0x7ff27006d270 2024-03-24T00:39:53.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.113+0000 7ff2754d1640 1 -- 172.21.15.67:0/2095233139 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff26802cc50 con 0x7ff27006d270 2024-03-24T00:39:53.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.113+0000 7ff2754d1640 1 -- 172.21.15.67:0/2095233139 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff268034440 con 0x7ff27006d270 2024-03-24T00:39:53.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.114+0000 7ff278737640 1 -- 172.21.15.67:0/2095233139 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff27006d270 msgr2=0x7ff2700cc1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:53.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.114+0000 7ff278737640 1 --2- 172.21.15.67:0/2095233139 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff27006d270 0x7ff2700cc1a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff268009d90 tx=0x7ff26802bea0 comp rx=0 tx=0).stop 2024-03-24T00:39:53.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.114+0000 7ff278737640 1 -- 172.21.15.67:0/2095233139 shutdown_connections 2024-03-24T00:39:53.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.114+0000 7ff278737640 1 --2- 172.21.15.67:0/2095233139 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff27006d270 0x7ff2700cc1a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:53.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.114+0000 7ff278737640 1 --2- 172.21.15.67:0/2095233139 >> v2:172.21.15.72:3300/0 conn(0x7ff2700cdc10 0x7ff2700cfd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:53.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.114+0000 7ff278737640 1 -- 172.21.15.67:0/2095233139 >> 172.21.15.67:0/2095233139 conn(0x7ff2700b3640 msgr2=0x7ff2700b3a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:53.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.114+0000 7ff278737640 1 -- 172.21.15.67:0/2095233139 shutdown_connections 2024-03-24T00:39:53.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.114+0000 7ff278737640 1 -- 172.21.15.67:0/2095233139 wait complete. 2024-03-24T00:39:53.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.115+0000 7ff278737640 1 Processor -- start 2024-03-24T00:39:53.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.115+0000 7ff278737640 1 -- start start 2024-03-24T00:39:53.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.116+0000 7ff278737640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff27006d270 0x7ff2700b71a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:53.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.116+0000 7ff278737640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2700cdc10 0x7ff2700b2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:53.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.116+0000 7ff278737640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff270161ac0 0x7ff2700b26d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:53.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.116+0000 7ff278737640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2700d1740 con 0x7ff270161ac0 2024-03-24T00:39:53.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.116+0000 7ff278737640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2700d1220 con 0x7ff2700cdc10 2024-03-24T00:39:53.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.116+0000 7ff278737640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2700d0290 con 0x7ff27006d270 2024-03-24T00:39:53.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.116+0000 7ff276cd4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff270161ac0 0x7ff2700b26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:53.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.116+0000 7ff276cd4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff270161ac0 0x7ff2700b26d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50054/0 (socket says 172.21.15.67:50054) 2024-03-24T00:39:53.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.116+0000 7ff276cd4640 1 -- 172.21.15.67:0/3771297613 learned_addr learned my addr 172.21.15.67:0/3771297613 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:53.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.116+0000 7ff275cd2640 1 --2- 172.21.15.67:0/3771297613 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2700cdc10 0x7ff2700b2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:53.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.116+0000 7ff2764d3640 1 --2- 172.21.15.67:0/3771297613 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff27006d270 0x7ff2700b71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:53.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.116+0000 7ff276cd4640 1 -- 172.21.15.67:0/3771297613 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff27006d270 msgr2=0x7ff2700b71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:53.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.116+0000 7ff276cd4640 1 --2- 172.21.15.67:0/3771297613 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff27006d270 0x7ff2700b71a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:53.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.116+0000 7ff276cd4640 1 -- 172.21.15.67:0/3771297613 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2700cdc10 msgr2=0x7ff2700b2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:53.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.116+0000 7ff276cd4640 1 --2- 172.21.15.67:0/3771297613 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2700cdc10 0x7ff2700b2190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:53.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.116+0000 7ff276cd4640 1 -- 172.21.15.67:0/3771297613 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2700ca9b0 con 0x7ff270161ac0 2024-03-24T00:39:53.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.116+0000 7ff2764d3640 1 --2- 172.21.15.67:0/3771297613 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff27006d270 0x7ff2700b71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:39:53.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.117+0000 7ff276cd4640 1 --2- 172.21.15.67:0/3771297613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff270161ac0 0x7ff2700b26d0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ff268009d90 tx=0x7ff268004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:53.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.117+0000 7ff266ffd640 1 -- 172.21.15.67:0/3771297613 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff268039050 con 0x7ff270161ac0 2024-03-24T00:39:53.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.117+0000 7ff266ffd640 1 -- 172.21.15.67:0/3771297613 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff268045070 con 0x7ff270161ac0 2024-03-24T00:39:53.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.117+0000 7ff278737640 1 -- 172.21.15.67:0/3771297613 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2700cac90 con 0x7ff270161ac0 2024-03-24T00:39:53.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.117+0000 7ff266ffd640 1 -- 172.21.15.67:0/3771297613 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff26803e610 con 0x7ff270161ac0 2024-03-24T00:39:53.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.117+0000 7ff278737640 1 -- 172.21.15.67:0/3771297613 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff27015d840 con 0x7ff270161ac0 2024-03-24T00:39:53.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.118+0000 7ff278737640 1 -- 172.21.15.67:0/3771297613 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff27004f400 con 0x7ff270161ac0 2024-03-24T00:39:53.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.118+0000 7ff266ffd640 1 -- 172.21.15.67:0/3771297613 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff26803e7b0 con 0x7ff270161ac0 2024-03-24T00:39:53.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.119+0000 7ff266ffd640 1 --2- 172.21.15.67:0/3771297613 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff25406bbd0 0x7ff25406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:53.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.119+0000 7ff266ffd640 1 -- 172.21.15.67:0/3771297613 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5853+0+0 (secure 0 0 0) 0x7ff2680ad650 con 0x7ff270161ac0 2024-03-24T00:39:53.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.119+0000 7ff2764d3640 1 --2- 172.21.15.67:0/3771297613 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff25406bbd0 0x7ff25406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:53.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.119+0000 7ff2764d3640 1 --2- 172.21.15.67:0/3771297613 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff25406bbd0 0x7ff25406e090 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff26c009ac0 tx=0x7ff26c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:53.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.123+0000 7ff266ffd640 1 -- 172.21.15.67:0/3771297613 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff26807c140 con 0x7ff270161ac0 2024-03-24T00:39:53.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.307+0000 7ff278737640 1 -- 172.21.15.67:0/3771297613 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff270160980 con 0x7ff270161ac0 2024-03-24T00:39:53.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.982+0000 7ff266ffd640 1 -- 172.21.15.67:0/3771297613 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v31) v1 ==== 178+0+0 (secure 0 0 0) 0x7ff26807ba40 con 0x7ff270161ac0 2024-03-24T00:39:53.984 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data_ec' removed 2024-03-24T00:39:53.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.985+0000 7ff278737640 1 -- 172.21.15.67:0/3771297613 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff25406bbd0 msgr2=0x7ff25406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:53.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.985+0000 7ff278737640 1 --2- 172.21.15.67:0/3771297613 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff25406bbd0 0x7ff25406e090 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff26c009ac0 tx=0x7ff26c008040 comp rx=0 tx=0).stop 2024-03-24T00:39:53.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.985+0000 7ff278737640 1 -- 172.21.15.67:0/3771297613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff270161ac0 msgr2=0x7ff2700b26d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:53.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.985+0000 7ff278737640 1 --2- 172.21.15.67:0/3771297613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff270161ac0 0x7ff2700b26d0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ff268009d90 tx=0x7ff268004a40 comp rx=0 tx=0).stop 2024-03-24T00:39:53.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.985+0000 7ff278737640 1 -- 172.21.15.67:0/3771297613 shutdown_connections 2024-03-24T00:39:53.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.985+0000 7ff278737640 1 --2- 172.21.15.67:0/3771297613 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff25406bbd0 0x7ff25406e090 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:53.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.985+0000 7ff278737640 1 --2- 172.21.15.67:0/3771297613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff270161ac0 0x7ff2700b26d0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:53.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.985+0000 7ff278737640 1 --2- 172.21.15.67:0/3771297613 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2700cdc10 0x7ff2700b2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:53.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.985+0000 7ff278737640 1 --2- 172.21.15.67:0/3771297613 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff27006d270 0x7ff2700b71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:53.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.985+0000 7ff278737640 1 -- 172.21.15.67:0/3771297613 >> 172.21.15.67:0/3771297613 conn(0x7ff2700b3640 msgr2=0x7ff2700b3a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:53.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.986+0000 7ff278737640 1 -- 172.21.15.67:0/3771297613 shutdown_connections 2024-03-24T00:39:53.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:53.986+0000 7ff278737640 1 -- 172.21.15.67:0/3771297613 wait complete. 2024-03-24T00:39:54.001 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-24T00:39:54.001 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-24T00:39:54.002 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-24T00:39:54.002 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:39:54.002 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-24T00:39:54.002 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-24T00:39:54.002 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-24T00:39:54.002 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:39:54.003 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-24T00:39:54.003 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-24T00:39:54.003 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-24T00:39:54.003 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:39:54.003 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-24T00:39:54.003 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-24T00:39:54.003 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-24T00:39:54.003 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:39:54.012 INFO:tasks.ceph.mds.a:Stopped 2024-03-24T00:39:54.012 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-24T00:39:54.013 INFO:tasks.ceph.mds.c:Stopped 2024-03-24T00:39:54.013 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-24T00:39:54.014 INFO:tasks.ceph.mds.b:Stopped 2024-03-24T00:39:54.015 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-24T00:39:54.015 INFO:tasks.ceph.mds.d:Stopped 2024-03-24T00:39:54.015 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-24T00:39:54.055 INFO:tasks.ceph.mds.a:Started 2024-03-24T00:39:54.055 DEBUG:teuthology.parallel:result is None 2024-03-24T00:39:54.056 INFO:tasks.ceph.mds.c:Started 2024-03-24T00:39:54.056 DEBUG:teuthology.parallel:result is None 2024-03-24T00:39:54.058 INFO:tasks.ceph.mds.b:Started 2024-03-24T00:39:54.058 DEBUG:teuthology.parallel:result is None 2024-03-24T00:39:54.058 INFO:tasks.ceph.mds.d:Started 2024-03-24T00:39:54.059 DEBUG:teuthology.parallel:result is None 2024-03-24T00:39:54.059 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-24T00:39:54.133 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-03-24T00:39:54.133 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-03-24T00:39:54.140 INFO:tasks.ceph.mds.d.smithi072.stdout:starting mds.d at 2024-03-24T00:39:54.140 INFO:tasks.ceph.mds.b.smithi072.stdout:starting mds.b at 2024-03-24T00:39:54.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.165+0000 7f8bcfa36640 1 Processor -- start 2024-03-24T00:39:54.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.165+0000 7f8bcfa36640 1 -- start start 2024-03-24T00:39:54.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.166+0000 7f8bcfa36640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8bc815e0f0 0x7f8bc80b95b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:54.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.166+0000 7f8bcfa36640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8bc80b9af0 0x7f8bc80d32d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:54.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.166+0000 7f8bcfa36640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bc80d2230 con 0x7f8bc80d3810 2024-03-24T00:39:54.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.166+0000 7f8bcfa36640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bc80d2510 con 0x7f8bc80b9af0 2024-03-24T00:39:54.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.166+0000 7f8bcfa36640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc80d27f0 con 0x7f8bc815e0f0 2024-03-24T00:39:54.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.166+0000 7f8bcd7d2640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8bc815e0f0 0x7f8bc80b95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:54.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.166+0000 7f8bccfd1640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8bc80b9af0 0x7f8bc80d32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:54.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.166+0000 7f8bccfd1640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8bc80b9af0 0x7f8bc80d32d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:49686/0 (socket says 172.21.15.67:49686) 2024-03-24T00:39:54.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.166+0000 7f8bccfd1640 1 -- 172.21.15.67:0/2243091843 learned_addr learned my addr 172.21.15.67:0/2243091843 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:54.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.166+0000 7f8bcd7d2640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8bc815e0f0 0x7f8bc80b95b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.67:59058/0 (socket says 172.21.15.67:59058) 2024-03-24T00:39:54.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.166+0000 7f8bccfd1640 1 -- 172.21.15.67:0/2243091843 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8bc815e0f0 msgr2=0x7f8bc80b95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:54.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.166+0000 7f8bccfd1640 1 --2- 172.21.15.67:0/2243091843 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8bc815e0f0 0x7f8bc80b95b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:54.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.166+0000 7f8bccfd1640 1 -- 172.21.15.67:0/2243091843 >> v1:172.21.15.72:6789/0 conn(0x7f8bc80d3810 legacy=0x7f8bc80b6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:54.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.166+0000 7f8bccfd1640 1 -- 172.21.15.67:0/2243091843 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bc80d2ad0 con 0x7f8bc80b9af0 2024-03-24T00:39:54.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.166+0000 7f8bbffff640 1 -- 172.21.15.67:0/2243091843 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3125001778 0 0) 0x7f8bc80d2230 con 0x7f8bc80d3810 2024-03-24T00:39:54.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.167+0000 7f8bccfd1640 1 --2- 172.21.15.67:0/2243091843 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8bc80b9af0 0x7f8bc80d32d0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8bb8009850 tx=0x7f8bb802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e6aa7c7deb038aa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:54.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.167+0000 7f8bbffff640 1 -- 172.21.15.67:0/2243091843 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb8004230 con 0x7f8bc80b9af0 2024-03-24T00:39:54.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.167+0000 7f8bbffff640 1 -- 172.21.15.67:0/2243091843 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bb80043d0 con 0x7f8bc80b9af0 2024-03-24T00:39:54.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.167+0000 7f8bbffff640 1 -- 172.21.15.67:0/2243091843 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb8003810 con 0x7f8bc80b9af0 2024-03-24T00:39:54.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.168+0000 7f8bcfa36640 1 -- 172.21.15.67:0/2243091843 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8bc80b9af0 msgr2=0x7f8bc80d32d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:54.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.168+0000 7f8bcfa36640 1 --2- 172.21.15.67:0/2243091843 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8bc80b9af0 0x7f8bc80d32d0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8bb8009850 tx=0x7f8bb802b960 comp rx=0 tx=0).stop 2024-03-24T00:39:54.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.168+0000 7f8bcfa36640 1 -- 172.21.15.67:0/2243091843 shutdown_connections 2024-03-24T00:39:54.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.168+0000 7f8bcfa36640 1 --2- 172.21.15.67:0/2243091843 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8bc80b9af0 0x7f8bc80d32d0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:54.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.168+0000 7f8bcfa36640 1 --2- 172.21.15.67:0/2243091843 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8bc815e0f0 0x7f8bc80b95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:54.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.168+0000 7f8bcfa36640 1 -- 172.21.15.67:0/2243091843 >> 172.21.15.67:0/2243091843 conn(0x7f8bc8072b60 msgr2=0x7f8bc8072f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:54.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.168+0000 7f8bcfa36640 1 -- 172.21.15.67:0/2243091843 shutdown_connections 2024-03-24T00:39:54.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.168+0000 7f8bcfa36640 1 -- 172.21.15.67:0/2243091843 wait complete. 2024-03-24T00:39:54.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.169+0000 7f8bcfa36640 1 Processor -- start 2024-03-24T00:39:54.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.169+0000 7f8bcfa36640 1 -- start start 2024-03-24T00:39:54.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.170+0000 7f8bcfa36640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8bc80b9af0 0x7f8bc80cd6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:54.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.170+0000 7f8bcfa36640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8bc80d3810 0x7f8bc80cad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:54.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.170+0000 7f8bcfa36640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bc815e0f0 0x7f8bc80cb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:54.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.170+0000 7f8bcfa36640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc80b3350 con 0x7f8bc815e0f0 2024-03-24T00:39:54.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.170+0000 7f8bcfa36640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc80b31d0 con 0x7f8bc80b9af0 2024-03-24T00:39:54.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.170+0000 7f8bcfa36640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc80b74d0 con 0x7f8bc80d3810 2024-03-24T00:39:54.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.170+0000 7f8bcdfd3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bc815e0f0 0x7f8bc80cb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:54.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.170+0000 7f8bcd7d2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8bc80b9af0 0x7f8bc80cd6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:54.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.170+0000 7f8bcdfd3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bc815e0f0 0x7f8bc80cb2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39622/0 (socket says 172.21.15.67:39622) 2024-03-24T00:39:54.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.170+0000 7f8bcdfd3640 1 -- 172.21.15.67:0/878290873 learned_addr learned my addr 172.21.15.67:0/878290873 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:54.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.170+0000 7f8bccfd1640 1 --2- 172.21.15.67:0/878290873 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8bc80d3810 0x7f8bc80cad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:54.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.170+0000 7f8bcd7d2640 1 -- 172.21.15.67:0/878290873 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8bc80d3810 msgr2=0x7f8bc80cad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:54.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.170+0000 7f8bcd7d2640 1 --2- 172.21.15.67:0/878290873 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8bc80d3810 0x7f8bc80cad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:54.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.170+0000 7f8bcd7d2640 1 -- 172.21.15.67:0/878290873 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bc815e0f0 msgr2=0x7f8bc80cb2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:54.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.170+0000 7f8bcd7d2640 1 --2- 172.21.15.67:0/878290873 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bc815e0f0 0x7f8bc80cb2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:54.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.170+0000 7f8bcd7d2640 1 -- 172.21.15.67:0/878290873 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bc81f4690 con 0x7f8bc80b9af0 2024-03-24T00:39:54.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.171+0000 7f8bcd7d2640 1 --2- 172.21.15.67:0/878290873 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8bc80b9af0 0x7f8bc80cd6d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8bc400e870 tx=0x7f8bc400ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:54.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.171+0000 7f8bbdffb640 1 -- 172.21.15.67:0/878290873 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bc4017070 con 0x7f8bc80b9af0 2024-03-24T00:39:54.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.171+0000 7f8bbdffb640 1 -- 172.21.15.67:0/878290873 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bc400ee60 con 0x7f8bc80b9af0 2024-03-24T00:39:54.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.171+0000 7f8bcfa36640 1 -- 172.21.15.67:0/878290873 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bc80aa8c0 con 0x7f8bc80b9af0 2024-03-24T00:39:54.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.171+0000 7f8bbdffb640 1 -- 172.21.15.67:0/878290873 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bc400cb60 con 0x7f8bc80b9af0 2024-03-24T00:39:54.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.171+0000 7f8bcfa36640 1 -- 172.21.15.67:0/878290873 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bc80aae80 con 0x7f8bc80b9af0 2024-03-24T00:39:54.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.172+0000 7f8bcfa36640 1 -- 172.21.15.67:0/878290873 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bc804f400 con 0x7f8bc80b9af0 2024-03-24T00:39:54.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.172+0000 7f8bbdffb640 1 -- 172.21.15.67:0/878290873 <== mon.1 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8bc401a070 con 0x7f8bc80b9af0 2024-03-24T00:39:54.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.173+0000 7f8bbdffb640 1 --2- 172.21.15.67:0/878290873 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8bac06bbd0 0x7f8bac06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:54.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.173+0000 7f8bbdffb640 1 -- 172.21.15.67:0/878290873 <== mon.1 v2:172.21.15.72:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f8bc408aa30 con 0x7f8bc80b9af0 2024-03-24T00:39:54.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.173+0000 7f8bccfd1640 1 --2- 172.21.15.67:0/878290873 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8bac06bbd0 0x7f8bac06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:54.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.174+0000 7f8bccfd1640 1 --2- 172.21.15.67:0/878290873 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8bac06bbd0 0x7f8bac06e090 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f8bb8035000 tx=0x7f8bb8008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:54.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.178+0000 7f8bbdffb640 1 -- 172.21.15.67:0/878290873 <== mon.1 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8bc4059020 con 0x7f8bc80b9af0 2024-03-24T00:39:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.376+0000 7f8bcfa36640 1 -- 172.21.15.67:0/878290873 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f8bc80cf400 con 0x7f8bc80b9af0 2024-03-24T00:39:54.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:54.998+0000 7f8bbdffb640 1 -- 172.21.15.67:0/878290873 <== mon.1 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v33) v1 ==== 97+0+0 (secure 0 0 0) 0x7f8bc405d0c0 con 0x7f8bc80b9af0 2024-03-24T00:39:54.999 INFO:teuthology.orchestra.run.smithi067.stderr: removed all blocklist entries 2024-03-24T00:39:55.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.002+0000 7f8bcfa36640 1 -- 172.21.15.67:0/878290873 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8bac06bbd0 msgr2=0x7f8bac06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:55.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.002+0000 7f8bcfa36640 1 --2- 172.21.15.67:0/878290873 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8bac06bbd0 0x7f8bac06e090 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f8bb8035000 tx=0x7f8bb8008030 comp rx=0 tx=0).stop 2024-03-24T00:39:55.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.002+0000 7f8bcfa36640 1 -- 172.21.15.67:0/878290873 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8bc80b9af0 msgr2=0x7f8bc80cd6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:55.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.002+0000 7f8bcfa36640 1 --2- 172.21.15.67:0/878290873 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8bc80b9af0 0x7f8bc80cd6d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8bc400e870 tx=0x7f8bc400ed30 comp rx=0 tx=0).stop 2024-03-24T00:39:55.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.002+0000 7f8bcfa36640 1 -- 172.21.15.67:0/878290873 shutdown_connections 2024-03-24T00:39:55.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.002+0000 7f8bcfa36640 1 --2- 172.21.15.67:0/878290873 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8bac06bbd0 0x7f8bac06e090 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:55.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.002+0000 7f8bcfa36640 1 --2- 172.21.15.67:0/878290873 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bc815e0f0 0x7f8bc80cb2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:55.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.002+0000 7f8bcfa36640 1 --2- 172.21.15.67:0/878290873 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8bc80d3810 0x7f8bc80cad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:55.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.002+0000 7f8bcfa36640 1 --2- 172.21.15.67:0/878290873 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8bc80b9af0 0x7f8bc80cd6d0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:55.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.002+0000 7f8bcfa36640 1 -- 172.21.15.67:0/878290873 >> 172.21.15.67:0/878290873 conn(0x7f8bc8072b60 msgr2=0x7f8bc80b9100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:55.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.002+0000 7f8bcfa36640 1 -- 172.21.15.67:0/878290873 shutdown_connections 2024-03-24T00:39:55.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.002+0000 7f8bcfa36640 1 -- 172.21.15.67:0/878290873 wait complete. 2024-03-24T00:39:55.019 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-24T00:39:55.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.125+0000 7f79a0d7e640 1 Processor -- start 2024-03-24T00:39:55.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.125+0000 7f79a0d7e640 1 -- start start 2024-03-24T00:39:55.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.126+0000 7f79a0d7e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f799c0cef40 0x7f799c0cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:55.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.126+0000 7f79a0d7e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f799c06dce0 0x7f799c06e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:55.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.126+0000 7f79a0d7e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f799c072890 con 0x7f799c0ccce0 2024-03-24T00:39:55.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.126+0000 7f79a0d7e640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f799c15c580 con 0x7f799c06dce0 2024-03-24T00:39:55.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.126+0000 7f79a0d7e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f799c15c860 con 0x7f799c0cef40 2024-03-24T00:39:55.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.126+0000 7f799ad9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f799c06dce0 0x7f799c06e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:55.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.126+0000 7f799a59c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f799c0ccce0 0x7f799c0cd0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49070/0 (socket says 172.21.15.67:49070) 2024-03-24T00:39:55.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.126+0000 7f799a59c640 1 -- 172.21.15.67:0/1545160266 learned_addr learned my addr 172.21.15.67:0/1545160266 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:55.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.126+0000 7f7999d9b640 1 --2- 172.21.15.67:0/1545160266 >> v2:172.21.15.72:3300/0 conn(0x7f799c0cef40 0x7f799c0cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:55.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.126+0000 7f799ad9d640 1 -- 172.21.15.67:0/1545160266 >> v2:172.21.15.72:3300/0 conn(0x7f799c0cef40 msgr2=0x7f799c0cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:55.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.126+0000 7f799ad9d640 1 --2- 172.21.15.67:0/1545160266 >> v2:172.21.15.72:3300/0 conn(0x7f799c0cef40 0x7f799c0cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:55.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.126+0000 7f799ad9d640 1 -- 172.21.15.67:0/1545160266 >> v1:172.21.15.67:6789/0 conn(0x7f799c0ccce0 legacy=0x7f799c0cd0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:55.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.126+0000 7f799ad9d640 1 -- 172.21.15.67:0/1545160266 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f799c15cb40 con 0x7f799c06dce0 2024-03-24T00:39:55.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.126+0000 7f799959a640 1 -- 172.21.15.67:0/1545160266 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3701859562 0 0) 0x7f799c072890 con 0x7f799c0ccce0 2024-03-24T00:39:55.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.127+0000 7f799ad9d640 1 --2- 172.21.15.67:0/1545160266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f799c06dce0 0x7f799c06e150 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f799c06da50 tx=0x7f799402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df52198a90d0f8be server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:55.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.127+0000 7f799959a640 1 -- 172.21.15.67:0/1545160266 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7994039070 con 0x7f799c06dce0 2024-03-24T00:39:55.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.127+0000 7f799959a640 1 -- 172.21.15.67:0/1545160266 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f799402cc80 con 0x7f799c06dce0 2024-03-24T00:39:55.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.127+0000 7f799959a640 1 -- 172.21.15.67:0/1545160266 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7994034660 con 0x7f799c06dce0 2024-03-24T00:39:55.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.128+0000 7f79a0d7e640 1 -- 172.21.15.67:0/1545160266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f799c06dce0 msgr2=0x7f799c06e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:55.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.128+0000 7f79a0d7e640 1 --2- 172.21.15.67:0/1545160266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f799c06dce0 0x7f799c06e150 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f799c06da50 tx=0x7f799402bed0 comp rx=0 tx=0).stop 2024-03-24T00:39:55.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.128+0000 7f79a0d7e640 1 -- 172.21.15.67:0/1545160266 shutdown_connections 2024-03-24T00:39:55.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.128+0000 7f79a0d7e640 1 --2- 172.21.15.67:0/1545160266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f799c06dce0 0x7f799c06e150 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:55.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.128+0000 7f79a0d7e640 1 --2- 172.21.15.67:0/1545160266 >> v2:172.21.15.72:3300/0 conn(0x7f799c0cef40 0x7f799c0cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:55.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.128+0000 7f79a0d7e640 1 -- 172.21.15.67:0/1545160266 >> 172.21.15.67:0/1545160266 conn(0x7f799c0cf9e0 msgr2=0x7f799c0cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:55.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.128+0000 7f79a0d7e640 1 -- 172.21.15.67:0/1545160266 shutdown_connections 2024-03-24T00:39:55.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.128+0000 7f79a0d7e640 1 -- 172.21.15.67:0/1545160266 wait complete. 2024-03-24T00:39:55.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.129+0000 7f79a0d7e640 1 Processor -- start 2024-03-24T00:39:55.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.129+0000 7f79a0d7e640 1 -- start start 2024-03-24T00:39:55.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.130+0000 7f79a0d7e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f799c06dce0 0x7f799c15b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:55.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.130+0000 7f79a0d7e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f799c0ccce0 0x7f799c13d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:55.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.130+0000 7f79a0d7e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f799c0cef40 0x7f799c13dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:55.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.130+0000 7f79a0d7e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f799c145900 con 0x7f799c06dce0 2024-03-24T00:39:55.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.130+0000 7f79a0d7e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f799c15c290 con 0x7f799c0ccce0 2024-03-24T00:39:55.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.130+0000 7f79a0d7e640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f799c06e690 con 0x7f799c0cef40 2024-03-24T00:39:55.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.130+0000 7f799a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f799c06dce0 0x7f799c15b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:55.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.130+0000 7f799a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f799c06dce0 0x7f799c15b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39638/0 (socket says 172.21.15.67:39638) 2024-03-24T00:39:55.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.130+0000 7f799a59c640 1 -- 172.21.15.67:0/2410942536 learned_addr learned my addr 172.21.15.67:0/2410942536 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:55.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.130+0000 7f7999d9b640 1 --2- 172.21.15.67:0/2410942536 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f799c0ccce0 0x7f799c13d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:55.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.130+0000 7f799ad9d640 1 --2- 172.21.15.67:0/2410942536 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f799c0cef40 0x7f799c13dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:55.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.130+0000 7f799a59c640 1 -- 172.21.15.67:0/2410942536 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f799c0cef40 msgr2=0x7f799c13dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:55.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.130+0000 7f799a59c640 1 --2- 172.21.15.67:0/2410942536 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f799c0cef40 0x7f799c13dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:55.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.130+0000 7f799a59c640 1 -- 172.21.15.67:0/2410942536 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f799c0ccce0 msgr2=0x7f799c13d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:55.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.130+0000 7f799a59c640 1 --2- 172.21.15.67:0/2410942536 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f799c0ccce0 0x7f799c13d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:55.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.130+0000 7f799a59c640 1 -- 172.21.15.67:0/2410942536 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f799c0b3f60 con 0x7f799c06dce0 2024-03-24T00:39:55.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.130+0000 7f7999d9b640 1 --2- 172.21.15.67:0/2410942536 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f799c0ccce0 0x7f799c13d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:55.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.130+0000 7f799a59c640 1 --2- 172.21.15.67:0/2410942536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f799c06dce0 0x7f799c15b5a0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f798c00c4d0 tx=0x7f798c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:55.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.130+0000 7f798affd640 1 -- 172.21.15.67:0/2410942536 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f798c018070 con 0x7f799c06dce0 2024-03-24T00:39:55.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.130+0000 7f798affd640 1 -- 172.21.15.67:0/2410942536 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f798c007500 con 0x7f799c06dce0 2024-03-24T00:39:55.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.130+0000 7f798affd640 1 -- 172.21.15.67:0/2410942536 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f798c004a20 con 0x7f799c06dce0 2024-03-24T00:39:55.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.131+0000 7f79a0d7e640 1 -- 172.21.15.67:0/2410942536 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f799c0b4240 con 0x7f799c06dce0 2024-03-24T00:39:55.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.131+0000 7f79a0d7e640 1 -- 172.21.15.67:0/2410942536 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f799c0b4800 con 0x7f799c06dce0 2024-03-24T00:39:55.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.132+0000 7f79a0d7e640 1 -- 172.21.15.67:0/2410942536 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f799c04f400 con 0x7f799c06dce0 2024-03-24T00:39:55.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.132+0000 7f798affd640 1 -- 172.21.15.67:0/2410942536 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f798c01b070 con 0x7f799c06dce0 2024-03-24T00:39:55.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.132+0000 7f798affd640 1 --2- 172.21.15.67:0/2410942536 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f797c06bc00 0x7f797c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:55.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.132+0000 7f798affd640 1 -- 172.21.15.67:0/2410942536 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f798c089be0 con 0x7f799c06dce0 2024-03-24T00:39:55.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.133+0000 7f7999d9b640 1 --2- 172.21.15.67:0/2410942536 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f797c06bc00 0x7f797c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:55.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.133+0000 7f7999d9b640 1 --2- 172.21.15.67:0/2410942536 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f797c06bc00 0x7f797c06e0c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7990006fd0 tx=0x7f7990008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:55.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.137+0000 7f798affd640 1 -- 172.21.15.67:0/2410942536 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f798c058890 con 0x7f799c06dce0 2024-03-24T00:39:55.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.387+0000 7f79a0d7e640 1 -- 172.21.15.67:0/2410942536 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f799c0d0600 con 0x7f799c06dce0 2024-03-24T00:39:55.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.388+0000 7f798affd640 1 -- 172.21.15.67:0/2410942536 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v6) v1 ==== 80+0+6645 (secure 0 0 0) 0x7f798c058190 con 0x7f799c06dce0 2024-03-24T00:39:55.389 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:39:55.389 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:39:55.389 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_dump": [ 2024-03-24T00:39:55.389 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:39:55.389 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.a", 2024-03-24T00:39:55.389 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9l4UENDhAASKUDtqcbSQ7wADj7uo4eyQ==", 2024-03-24T00:39:55.389 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:39:55.389 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:39:55.389 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-03-24T00:39:55.389 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-03-24T00:39:55.389 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:39:55.389 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:39:55.390 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:39:55.390 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:39:55.390 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.b", 2024-03-24T00:39:55.390 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9luhJWGhAAsS6PFHV2XdF9htjIjzdrdA==", 2024-03-24T00:39:55.390 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:39:55.390 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:39:55.390 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-03-24T00:39:55.390 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-03-24T00:39:55.390 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:39:55.390 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:39:55.390 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:39:55.390 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:39:55.390 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.c", 2024-03-24T00:39:55.391 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9l8mRjFRAAQ73yexNddpcyp8ayTcRKCg==", 2024-03-24T00:39:55.391 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:39:55.391 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:39:55.391 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-03-24T00:39:55.391 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-03-24T00:39:55.391 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:39:55.391 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:39:55.391 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:39:55.391 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:39:55.391 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.d", 2024-03-24T00:39:55.391 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9lcJyvIRAA2tpDb6qxn1MtljjEno4qPA==", 2024-03-24T00:39:55.391 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:39:55.391 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:39:55.391 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-03-24T00:39:55.392 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-03-24T00:39:55.392 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:39:55.392 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:39:55.392 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:39:55.392 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:39:55.392 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.0", 2024-03-24T00:39:55.392 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDgdf9lV/NUBRAAyJNajFAnNABJErnBlB8TEA==", 2024-03-24T00:39:55.392 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:39:55.392 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:39:55.392 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:39:55.392 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:39:55.392 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:39:55.392 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:39:55.392 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:39:55.393 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.1", 2024-03-24T00:39:55.393 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDjdf9l0J6lAxAA9RInt/X22jqzthE0vOiiFg==", 2024-03-24T00:39:55.393 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:39:55.393 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:39:55.393 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:39:55.393 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:39:55.393 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:39:55.393 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:39:55.393 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:39:55.393 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.2", 2024-03-24T00:39:55.393 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDmdf9lSmrEARAAwmtSQRNePsn4WIZODnjhfQ==", 2024-03-24T00:39:55.393 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:39:55.393 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:39:55.393 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:39:55.394 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:39:55.394 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:39:55.394 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:39:55.394 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:39:55.394 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.3", 2024-03-24T00:39:55.394 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDpdf9lm59bABAAgJ5AV774lw2BgkSCgaweyA==", 2024-03-24T00:39:55.394 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:39:55.394 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:39:55.394 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:39:55.394 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:39:55.394 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:39:55.394 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:39:55.394 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:39:55.394 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.4", 2024-03-24T00:39:55.395 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDvdf9lCo+pCRAANHkL0pPxxok6IsBCLAFCbg==", 2024-03-24T00:39:55.395 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:39:55.395 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:39:55.395 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:39:55.395 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:39:55.395 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:39:55.395 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:39:55.395 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:39:55.395 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.5", 2024-03-24T00:39:55.395 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDydf9lyGEdCBAAcuszPsthGlLvVryE7+D9iQ==", 2024-03-24T00:39:55.395 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:39:55.395 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:39:55.395 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:39:55.395 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:39:55.396 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:39:55.396 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:39:55.396 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:39:55.396 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.6", 2024-03-24T00:39:55.396 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQD1df9ljuZqBhAACt2OPNkjKOD3rK7g2mVBBA==", 2024-03-24T00:39:55.396 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:39:55.396 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:39:55.396 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:39:55.396 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:39:55.396 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:39:55.396 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:39:55.396 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:39:55.396 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.7", 2024-03-24T00:39:55.396 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQD4df9lYKBfBRAAy/cWFF6pvGDjXZHAuBYw7g==", 2024-03-24T00:39:55.397 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:39:55.397 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:39:55.397 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:39:55.397 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:39:55.397 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:39:55.397 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:39:55.397 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:39:55.397 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.0", 2024-03-24T00:39:55.397 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9lY4nvJRAA6HWOZDWA+SSvJVmyjG+51g==", 2024-03-24T00:39:55.397 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:39:55.397 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:39:55.397 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-03-24T00:39:55.397 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-03-24T00:39:55.397 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-03-24T00:39:55.397 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:39:55.398 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:39:55.398 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:39:55.398 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.1", 2024-03-24T00:39:55.398 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9lonsXLxAAaltVMW3n0qHX8u9X8EkSmg==", 2024-03-24T00:39:55.398 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:39:55.398 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:39:55.398 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-03-24T00:39:55.398 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-03-24T00:39:55.398 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-03-24T00:39:55.398 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:39:55.402 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:39:55.402 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:39:55.402 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.2", 2024-03-24T00:39:55.402 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9lLEm2KxAAkT65mhLrTirlDnOUyvBY7g==", 2024-03-24T00:39:55.402 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:39:55.402 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:39:55.402 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-03-24T00:39:55.402 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-03-24T00:39:55.402 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-03-24T00:39:55.402 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:39:55.402 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:39:55.402 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:39:55.403 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.3", 2024-03-24T00:39:55.403 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9l8njZNBAAaAktZpX2SPcJR9ymPm9ebw==", 2024-03-24T00:39:55.403 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:39:55.403 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:39:55.403 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-03-24T00:39:55.403 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-03-24T00:39:55.403 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-03-24T00:39:55.403 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:39:55.403 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:39:55.403 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:39:55.403 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.admin", 2024-03-24T00:39:55.403 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDbdf9lDTIwLRAAMajqhM/KfUNF/SjVv9OJDw==", 2024-03-24T00:39:55.403 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:39:55.403 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-03-24T00:39:55.403 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-03-24T00:39:55.404 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-03-24T00:39:55.404 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:39:55.404 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:39:55.404 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:39:55.404 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:39:55.404 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.bootstrap-mds", 2024-03-24T00:39:55.404 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQADdv9lvX4vAhAALGEdemxIssQrQLnvBkffUA==", 2024-03-24T00:39:55.404 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:39:55.404 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile bootstrap-mds" 2024-03-24T00:39:55.404 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:39:55.404 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:39:55.404 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:39:55.404 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.bootstrap-mgr", 2024-03-24T00:39:55.404 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQADdv9lEYcvAhAAkPN1znvk30Mx7DVGfjN9hw==", 2024-03-24T00:39:55.405 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:39:55.405 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile bootstrap-mgr" 2024-03-24T00:39:55.405 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:39:55.405 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:39:55.405 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:39:55.405 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.bootstrap-osd", 2024-03-24T00:39:55.405 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQADdv9lsJEvAhAA5uCg25xaQjed1my4fo8eLQ==", 2024-03-24T00:39:55.405 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:39:55.405 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile bootstrap-osd" 2024-03-24T00:39:55.405 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:39:55.405 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:39:55.405 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:39:55.406 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.bootstrap-rbd", 2024-03-24T00:39:55.406 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQADdv9lF5kvAhAAUtQFbXbEJrxj1hIdoUORdw==", 2024-03-24T00:39:55.406 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:39:55.406 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile bootstrap-rbd" 2024-03-24T00:39:55.406 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:39:55.406 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:39:55.406 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:39:55.406 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-03-24T00:39:55.406 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQADdv9lu58vAhAAZN5xP+UbL5nbpop3OMtGhA==", 2024-03-24T00:39:55.406 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:39:55.406 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-03-24T00:39:55.406 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:39:55.406 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:39:55.406 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:39:55.407 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.bootstrap-rgw", 2024-03-24T00:39:55.407 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQADdv9ltKYvAhAAazGqdK5FhgX3hWWUbEQO7A==", 2024-03-24T00:39:55.407 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:39:55.407 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile bootstrap-rgw" 2024-03-24T00:39:55.407 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:39:55.407 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:39:55.407 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:39:55.407 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.x", 2024-03-24T00:39:55.407 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9lGwKvChAAxl+fRnjbAQ7dYJ727QFUgw==", 2024-03-24T00:39:55.407 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:39:55.407 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-03-24T00:39:55.407 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-03-24T00:39:55.407 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:39:55.407 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:39:55.408 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:39:55.408 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:39:55.408 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.y", 2024-03-24T00:39:55.408 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9lSvufBhAAoQ9L1GjLkstjrAw2lqlryA==", 2024-03-24T00:39:55.408 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:39:55.408 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-03-24T00:39:55.408 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-03-24T00:39:55.408 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:39:55.408 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:39:55.408 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:39:55.408 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-24T00:39:55.408 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:39:55.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.389+0000 7f79a0d7e640 1 -- 172.21.15.67:0/2410942536 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f797c06bc00 msgr2=0x7f797c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:55.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.389+0000 7f79a0d7e640 1 --2- 172.21.15.67:0/2410942536 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f797c06bc00 0x7f797c06e0c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7990006fd0 tx=0x7f7990008040 comp rx=0 tx=0).stop 2024-03-24T00:39:55.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.389+0000 7f79a0d7e640 1 -- 172.21.15.67:0/2410942536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f799c06dce0 msgr2=0x7f799c15b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:55.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.389+0000 7f79a0d7e640 1 --2- 172.21.15.67:0/2410942536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f799c06dce0 0x7f799c15b5a0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f798c00c4d0 tx=0x7f798c00c990 comp rx=0 tx=0).stop 2024-03-24T00:39:55.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.389+0000 7f79a0d7e640 1 -- 172.21.15.67:0/2410942536 shutdown_connections 2024-03-24T00:39:55.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.389+0000 7f79a0d7e640 1 --2- 172.21.15.67:0/2410942536 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f797c06bc00 0x7f797c06e0c0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:55.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.389+0000 7f79a0d7e640 1 --2- 172.21.15.67:0/2410942536 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f799c0cef40 0x7f799c13dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:55.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.390+0000 7f79a0d7e640 1 --2- 172.21.15.67:0/2410942536 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f799c0ccce0 0x7f799c13d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:55.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.390+0000 7f79a0d7e640 1 --2- 172.21.15.67:0/2410942536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f799c06dce0 0x7f799c15b5a0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:55.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.390+0000 7f79a0d7e640 1 -- 172.21.15.67:0/2410942536 >> 172.21.15.67:0/2410942536 conn(0x7f799c0cf9e0 msgr2=0x7f799c141d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:55.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.390+0000 7f79a0d7e640 1 -- 172.21.15.67:0/2410942536 shutdown_connections 2024-03-24T00:39:55.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.390+0000 7f79a0d7e640 1 -- 172.21.15.67:0/2410942536 wait complete. 2024-03-24T00:39:55.411 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-03-24T00:39:55.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.513+0000 7f19f08a7640 1 Processor -- start 2024-03-24T00:39:55.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.513+0000 7f19f08a7640 1 -- start start 2024-03-24T00:39:55.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.514+0000 7f19f08a7640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f19ec06de60 0x7f19ec09ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:55.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.514+0000 7f19f08a7640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f19ec09a410 0x7f19ec0cd410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:55.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.514+0000 7f19f08a7640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19ec0b29b0 con 0x7f19ec0b1eb0 2024-03-24T00:39:55.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.514+0000 7f19f08a7640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19ec0cc9e0 con 0x7f19ec06de60 2024-03-24T00:39:55.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.514+0000 7f19f08a7640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19ec0ccf80 con 0x7f19ec09a410 2024-03-24T00:39:55.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.514+0000 7f19ea59c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f19ec06de60 0x7f19ec09ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:55.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.514+0000 7f19ea59c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f19ec06de60 0x7f19ec09ae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39640/0 (socket says 172.21.15.67:39640) 2024-03-24T00:39:55.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.514+0000 7f19ea59c640 1 -- 172.21.15.67:0/2042561351 learned_addr learned my addr 172.21.15.67:0/2042561351 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:55.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.514+0000 7f19ea59c640 1 -- 172.21.15.67:0/2042561351 >> v2:172.21.15.72:3300/0 conn(0x7f19ec09a410 msgr2=0x7f19ec0cd410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:55.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.514+0000 7f19ea59c640 1 --2- 172.21.15.67:0/2042561351 >> v2:172.21.15.72:3300/0 conn(0x7f19ec09a410 0x7f19ec0cd410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:55.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.514+0000 7f19ea59c640 1 -- 172.21.15.67:0/2042561351 >> v1:172.21.15.72:6789/0 conn(0x7f19ec0b1eb0 legacy=0x7f19ec099d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:55.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.514+0000 7f19ea59c640 1 -- 172.21.15.67:0/2042561351 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19ec0b23f0 con 0x7f19ec06de60 2024-03-24T00:39:55.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.515+0000 7f19ea59c640 1 --2- 172.21.15.67:0/2042561351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19ec06de60 0x7f19ec09ae80 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f19ec0731b0 tx=0x7f19dc02b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6ff3418bd714740 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:55.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.515+0000 7f19e9d9b640 1 -- 172.21.15.67:0/2042561351 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19dc035020 con 0x7f19ec06de60 2024-03-24T00:39:55.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.515+0000 7f19e9d9b640 1 -- 172.21.15.67:0/2042561351 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19dc031890 con 0x7f19ec06de60 2024-03-24T00:39:55.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.515+0000 7f19e9d9b640 1 -- 172.21.15.67:0/2042561351 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19dc031c60 con 0x7f19ec06de60 2024-03-24T00:39:55.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.515+0000 7f19f08a7640 1 -- 172.21.15.67:0/2042561351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19ec06de60 msgr2=0x7f19ec09ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:55.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.515+0000 7f19f08a7640 1 --2- 172.21.15.67:0/2042561351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19ec06de60 0x7f19ec09ae80 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f19ec0731b0 tx=0x7f19dc02b930 comp rx=0 tx=0).stop 2024-03-24T00:39:55.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.515+0000 7f19f08a7640 1 -- 172.21.15.67:0/2042561351 shutdown_connections 2024-03-24T00:39:55.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.515+0000 7f19f08a7640 1 --2- 172.21.15.67:0/2042561351 >> v2:172.21.15.72:3300/0 conn(0x7f19ec09a410 0x7f19ec0cd410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:55.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.515+0000 7f19f08a7640 1 --2- 172.21.15.67:0/2042561351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19ec06de60 0x7f19ec09ae80 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:55.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.515+0000 7f19f08a7640 1 -- 172.21.15.67:0/2042561351 >> 172.21.15.67:0/2042561351 conn(0x7f19ec0b3650 msgr2=0x7f19ec0b3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:55.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.515+0000 7f19f08a7640 1 -- 172.21.15.67:0/2042561351 shutdown_connections 2024-03-24T00:39:55.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.515+0000 7f19f08a7640 1 -- 172.21.15.67:0/2042561351 wait complete. 2024-03-24T00:39:55.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.517+0000 7f19f08a7640 1 Processor -- start 2024-03-24T00:39:55.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.517+0000 7f19f08a7640 1 -- start start 2024-03-24T00:39:55.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.517+0000 7f19f08a7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19ec06de60 0x7f19ec15e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:55.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.517+0000 7f19f08a7640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f19ec09a410 0x7f19ec0c5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:55.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.517+0000 7f19f08a7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19ec0b1eb0 0x7f19ec0cb580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:55.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.517+0000 7f19f08a7640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19ec09b3c0 con 0x7f19ec06de60 2024-03-24T00:39:55.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.517+0000 7f19f08a7640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19ec0ccc10 con 0x7f19ec0b1eb0 2024-03-24T00:39:55.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.517+0000 7f19f08a7640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19ec06c150 con 0x7f19ec09a410 2024-03-24T00:39:55.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.518+0000 7f19ea59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19ec06de60 0x7f19ec15e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:55.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.518+0000 7f19ea59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19ec06de60 0x7f19ec15e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39648/0 (socket says 172.21.15.67:39648) 2024-03-24T00:39:55.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.518+0000 7f19ea59c640 1 -- 172.21.15.67:0/3132451478 learned_addr learned my addr 172.21.15.67:0/3132451478 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:55.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.518+0000 7f19e3fff640 1 --2- 172.21.15.67:0/3132451478 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f19ec09a410 0x7f19ec0c5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:55.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.518+0000 7f19ead9d640 1 --2- 172.21.15.67:0/3132451478 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19ec0b1eb0 0x7f19ec0cb580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:55.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.518+0000 7f19ea59c640 1 -- 172.21.15.67:0/3132451478 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f19ec09a410 msgr2=0x7f19ec0c5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:55.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.518+0000 7f19ea59c640 1 --2- 172.21.15.67:0/3132451478 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f19ec09a410 0x7f19ec0c5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:55.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.518+0000 7f19ea59c640 1 -- 172.21.15.67:0/3132451478 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19ec0b1eb0 msgr2=0x7f19ec0cb580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:55.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.518+0000 7f19ea59c640 1 --2- 172.21.15.67:0/3132451478 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19ec0b1eb0 0x7f19ec0cb580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:55.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.518+0000 7f19ea59c640 1 -- 172.21.15.67:0/3132451478 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19ec0aefc0 con 0x7f19ec06de60 2024-03-24T00:39:55.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.518+0000 7f19ead9d640 1 --2- 172.21.15.67:0/3132451478 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19ec0b1eb0 0x7f19ec0cb580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:39:55.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.518+0000 7f19ea59c640 1 --2- 172.21.15.67:0/3132451478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19ec06de60 0x7f19ec15e280 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f19dc0090b0 tx=0x7f19dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:55.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.518+0000 7f19e2ffd640 1 -- 172.21.15.67:0/3132451478 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19dc03ad60 con 0x7f19ec06de60 2024-03-24T00:39:55.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.518+0000 7f19e2ffd640 1 -- 172.21.15.67:0/3132451478 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19dc0443f0 con 0x7f19ec06de60 2024-03-24T00:39:55.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.518+0000 7f19f08a7640 1 -- 172.21.15.67:0/3132451478 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19ec0af2a0 con 0x7f19ec06de60 2024-03-24T00:39:55.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.518+0000 7f19e2ffd640 1 -- 172.21.15.67:0/3132451478 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19dc0447c0 con 0x7f19ec06de60 2024-03-24T00:39:55.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.518+0000 7f19f08a7640 1 -- 172.21.15.67:0/3132451478 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19ec0af860 con 0x7f19ec06de60 2024-03-24T00:39:55.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.519+0000 7f19f08a7640 1 -- 172.21.15.67:0/3132451478 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19ec04f400 con 0x7f19ec06de60 2024-03-24T00:39:55.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.520+0000 7f19e2ffd640 1 -- 172.21.15.67:0/3132451478 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f19dc04b060 con 0x7f19ec06de60 2024-03-24T00:39:55.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.520+0000 7f19e2ffd640 1 --2- 172.21.15.67:0/3132451478 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f19cc06bbd0 0x7f19cc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:55.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.520+0000 7f19e2ffd640 1 -- 172.21.15.67:0/3132451478 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f19dc0acb30 con 0x7f19ec06de60 2024-03-24T00:39:55.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.520+0000 7f19e3fff640 1 --2- 172.21.15.67:0/3132451478 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f19cc06bbd0 0x7f19cc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:55.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.521+0000 7f19e3fff640 1 --2- 172.21.15.67:0/3132451478 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f19cc06bbd0 0x7f19cc06e090 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f19d4007fd0 tx=0x7f19d4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:55.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.524+0000 7f19e2ffd640 1 -- 172.21.15.67:0/3132451478 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f19dc03f070 con 0x7f19ec06de60 2024-03-24T00:39:55.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.778+0000 7f19f08a7640 1 -- 172.21.15.67:0/3132451478 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f19ec0cc020 con 0x7f19ec06de60 2024-03-24T00:39:55.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.780+0000 7f19e2ffd640 1 -- 172.21.15.67:0/3132451478 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7f19dc02d040 con 0x7f19ec06de60 2024-03-24T00:39:55.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.782+0000 7f19f08a7640 1 -- 172.21.15.67:0/3132451478 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f19cc06bbd0 msgr2=0x7f19cc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:55.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.782+0000 7f19f08a7640 1 --2- 172.21.15.67:0/3132451478 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f19cc06bbd0 0x7f19cc06e090 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f19d4007fd0 tx=0x7f19d4009040 comp rx=0 tx=0).stop 2024-03-24T00:39:55.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.782+0000 7f19f08a7640 1 -- 172.21.15.67:0/3132451478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19ec06de60 msgr2=0x7f19ec15e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:55.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.782+0000 7f19f08a7640 1 --2- 172.21.15.67:0/3132451478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19ec06de60 0x7f19ec15e280 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f19dc0090b0 tx=0x7f19dc0046c0 comp rx=0 tx=0).stop 2024-03-24T00:39:55.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.782+0000 7f19f08a7640 1 -- 172.21.15.67:0/3132451478 shutdown_connections 2024-03-24T00:39:55.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.782+0000 7f19f08a7640 1 --2- 172.21.15.67:0/3132451478 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f19cc06bbd0 0x7f19cc06e090 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:55.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.782+0000 7f19f08a7640 1 --2- 172.21.15.67:0/3132451478 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19ec0b1eb0 0x7f19ec0cb580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:55.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.782+0000 7f19f08a7640 1 --2- 172.21.15.67:0/3132451478 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f19ec09a410 0x7f19ec0c5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:55.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.782+0000 7f19f08a7640 1 --2- 172.21.15.67:0/3132451478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19ec06de60 0x7f19ec15e280 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:55.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.782+0000 7f19f08a7640 1 -- 172.21.15.67:0/3132451478 >> 172.21.15.67:0/3132451478 conn(0x7f19ec0b3650 msgr2=0x7f19ec070080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:55.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.782+0000 7f19f08a7640 1 -- 172.21.15.67:0/3132451478 shutdown_connections 2024-03-24T00:39:55.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.782+0000 7f19f08a7640 1 -- 172.21.15.67:0/3132451478 wait complete. 2024-03-24T00:39:55.794 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-03-24T00:39:55.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.895+0000 7feceedf3640 1 Processor -- start 2024-03-24T00:39:55.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.895+0000 7feceedf3640 1 -- start start 2024-03-24T00:39:55.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.896+0000 7feceedf3640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fece815e0f0 0x7fece80b95b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:55.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.896+0000 7feceedf3640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fece80b6dc0 0x7fece80b7230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:55.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.896+0000 7feceedf3640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fece80d2230 con 0x7fece80b9af0 2024-03-24T00:39:55.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.896+0000 7feceedf3640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fece80b7770 con 0x7fece80b6dc0 2024-03-24T00:39:55.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.896+0000 7feceedf3640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fece80d2510 con 0x7fece815e0f0 2024-03-24T00:39:55.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.896+0000 7feced390640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fece80b6dc0 0x7fece80b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:55.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.896+0000 7fececb8f640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fece815e0f0 0x7fece80b95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:55.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.897+0000 7fececb8f640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fece815e0f0 0x7fece80b95b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.67:59110/0 (socket says 172.21.15.67:59110) 2024-03-24T00:39:55.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.897+0000 7feced390640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fece80b6dc0 0x7fece80b7230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:49738/0 (socket says 172.21.15.67:49738) 2024-03-24T00:39:55.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.897+0000 7fececb8f640 1 -- 172.21.15.67:0/181693571 learned_addr learned my addr 172.21.15.67:0/181693571 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:55.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.897+0000 7fece7fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fece80b9af0 0x7fece80d32d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:38302/0 (socket says 172.21.15.67:38302) 2024-03-24T00:39:55.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.897+0000 7fececb8f640 1 -- 172.21.15.67:0/181693571 >> v2:172.21.15.72:3300/0 conn(0x7fece80b6dc0 msgr2=0x7fece80b7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:55.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.897+0000 7fececb8f640 1 --2- 172.21.15.67:0/181693571 >> v2:172.21.15.72:3300/0 conn(0x7fece80b6dc0 0x7fece80b7230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:55.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.897+0000 7fececb8f640 1 -- 172.21.15.67:0/181693571 >> v1:172.21.15.72:6789/0 conn(0x7fece80b9af0 legacy=0x7fece80d32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:55.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.897+0000 7fececb8f640 1 -- 172.21.15.67:0/181693571 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece80d27f0 con 0x7fece815e0f0 2024-03-24T00:39:55.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.897+0000 7feced390640 1 --2- 172.21.15.67:0/181693571 >> v2:172.21.15.72:3300/0 conn(0x7fece80b6dc0 0x7fece80b7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:55.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.898+0000 7fececb8f640 1 --2- 172.21.15.67:0/181693571 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fece815e0f0 0x7fece80b95b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fece0009e50 tx=0x7fece002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8c85a2c7012a7b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:55.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.898+0000 7fece77fe640 1 -- 172.21.15.67:0/181693571 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece002cb00 con 0x7fece815e0f0 2024-03-24T00:39:55.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.898+0000 7fece77fe640 1 -- 172.21.15.67:0/181693571 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fece002cca0 con 0x7fece815e0f0 2024-03-24T00:39:55.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.898+0000 7fece77fe640 1 -- 172.21.15.67:0/181693571 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece0034610 con 0x7fece815e0f0 2024-03-24T00:39:55.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.898+0000 7feceedf3640 1 -- 172.21.15.67:0/181693571 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fece815e0f0 msgr2=0x7fece80b95b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:55.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.898+0000 7feceedf3640 1 --2- 172.21.15.67:0/181693571 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fece815e0f0 0x7fece80b95b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fece0009e50 tx=0x7fece002bf60 comp rx=0 tx=0).stop 2024-03-24T00:39:55.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.898+0000 7feceedf3640 1 -- 172.21.15.67:0/181693571 shutdown_connections 2024-03-24T00:39:55.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.898+0000 7feceedf3640 1 --2- 172.21.15.67:0/181693571 >> v2:172.21.15.72:3300/0 conn(0x7fece80b6dc0 0x7fece80b7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:55.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.898+0000 7feceedf3640 1 --2- 172.21.15.67:0/181693571 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fece815e0f0 0x7fece80b95b0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:55.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.898+0000 7feceedf3640 1 -- 172.21.15.67:0/181693571 >> 172.21.15.67:0/181693571 conn(0x7fece8072b60 msgr2=0x7fece8072f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:55.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.898+0000 7feceedf3640 1 -- 172.21.15.67:0/181693571 shutdown_connections 2024-03-24T00:39:55.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.898+0000 7feceedf3640 1 -- 172.21.15.67:0/181693571 wait complete. 2024-03-24T00:39:55.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.899+0000 7feceedf3640 1 Processor -- start 2024-03-24T00:39:55.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.900+0000 7feceedf3640 1 -- start start 2024-03-24T00:39:55.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.900+0000 7feceedf3640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fece80b6dc0 0x7fece81f4290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:55.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.900+0000 7feceedf3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fece80b9af0 0x7fece814a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:55.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.900+0000 7feceedf3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fece815e0f0 0x7fece814a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:55.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.900+0000 7feceedf3640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece80b31d0 con 0x7fece815e0f0 2024-03-24T00:39:55.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.900+0000 7feceedf3640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece80d2db0 con 0x7fece80b9af0 2024-03-24T00:39:55.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.900+0000 7feceedf3640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fece80b3350 con 0x7fece80b6dc0 2024-03-24T00:39:55.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.901+0000 7feced390640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fece815e0f0 0x7fece814a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:55.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.901+0000 7feced390640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fece815e0f0 0x7fece814a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39658/0 (socket says 172.21.15.67:39658) 2024-03-24T00:39:55.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.901+0000 7feced390640 1 -- 172.21.15.67:0/978765602 learned_addr learned my addr 172.21.15.67:0/978765602 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:55.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.901+0000 7fececb8f640 1 --2- 172.21.15.67:0/978765602 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fece80b6dc0 0x7fece81f4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:55.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.901+0000 7fece7fff640 1 --2- 172.21.15.67:0/978765602 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fece80b9af0 0x7fece814a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:55.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.901+0000 7feced390640 1 -- 172.21.15.67:0/978765602 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fece80b6dc0 msgr2=0x7fece81f4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:55.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.901+0000 7feced390640 1 --2- 172.21.15.67:0/978765602 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fece80b6dc0 0x7fece81f4290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:55.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.901+0000 7feced390640 1 -- 172.21.15.67:0/978765602 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fece80b9af0 msgr2=0x7fece814a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:55.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.901+0000 7feced390640 1 --2- 172.21.15.67:0/978765602 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fece80b9af0 0x7fece814a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:55.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.901+0000 7feced390640 1 -- 172.21.15.67:0/978765602 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece815e690 con 0x7fece815e0f0 2024-03-24T00:39:55.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.901+0000 7feced390640 1 --2- 172.21.15.67:0/978765602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fece815e0f0 0x7fece814a820 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fecd800e930 tx=0x7fecd800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:55.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.901+0000 7fece57fa640 1 -- 172.21.15.67:0/978765602 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd8017070 con 0x7fece815e0f0 2024-03-24T00:39:55.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.901+0000 7fece57fa640 1 -- 172.21.15.67:0/978765602 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecd800ca00 con 0x7fece815e0f0 2024-03-24T00:39:55.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.901+0000 7feceedf3640 1 -- 172.21.15.67:0/978765602 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fece815e970 con 0x7fece815e0f0 2024-03-24T00:39:55.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.901+0000 7fece57fa640 1 -- 172.21.15.67:0/978765602 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd800cdd0 con 0x7fece815e0f0 2024-03-24T00:39:55.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.901+0000 7feceedf3640 1 -- 172.21.15.67:0/978765602 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fece815ef30 con 0x7fece815e0f0 2024-03-24T00:39:55.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.902+0000 7feceedf3640 1 -- 172.21.15.67:0/978765602 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fece804f400 con 0x7fece815e0f0 2024-03-24T00:39:55.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.903+0000 7fece57fa640 1 -- 172.21.15.67:0/978765602 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fecd8013470 con 0x7fece815e0f0 2024-03-24T00:39:55.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.903+0000 7fece57fa640 1 --2- 172.21.15.67:0/978765602 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fecd406bbd0 0x7fecd406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:55.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.903+0000 7fece57fa640 1 -- 172.21.15.67:0/978765602 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fecd808a6b0 con 0x7fece815e0f0 2024-03-24T00:39:55.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.903+0000 7fececb8f640 1 --2- 172.21.15.67:0/978765602 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fecd406bbd0 0x7fecd406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:55.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.904+0000 7fececb8f640 1 --2- 172.21.15.67:0/978765602 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fecd406bbd0 0x7fecd406e090 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fece0035000 tx=0x7fece0030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:55.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:55.907+0000 7fece57fa640 1 -- 172.21.15.67:0/978765602 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fecd8058e60 con 0x7fece815e0f0 2024-03-24T00:39:56.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.159+0000 7feceedf3640 1 -- 172.21.15.67:0/978765602 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7fece80ce700 con 0x7fece815e0f0 2024-03-24T00:39:56.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.161+0000 7fece57fa640 1 -- 172.21.15.67:0/978765602 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7fecd805de20 con 0x7fece815e0f0 2024-03-24T00:39:56.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.163+0000 7feceedf3640 1 -- 172.21.15.67:0/978765602 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fecd406bbd0 msgr2=0x7fecd406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:56.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.163+0000 7feceedf3640 1 --2- 172.21.15.67:0/978765602 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fecd406bbd0 0x7fecd406e090 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fece0035000 tx=0x7fece0030000 comp rx=0 tx=0).stop 2024-03-24T00:39:56.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.163+0000 7feceedf3640 1 -- 172.21.15.67:0/978765602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fece815e0f0 msgr2=0x7fece814a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:56.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.163+0000 7feceedf3640 1 --2- 172.21.15.67:0/978765602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fece815e0f0 0x7fece814a820 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fecd800e930 tx=0x7fecd800edf0 comp rx=0 tx=0).stop 2024-03-24T00:39:56.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.163+0000 7feceedf3640 1 -- 172.21.15.67:0/978765602 shutdown_connections 2024-03-24T00:39:56.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.163+0000 7feceedf3640 1 --2- 172.21.15.67:0/978765602 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fecd406bbd0 0x7fecd406e090 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:56.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.163+0000 7feceedf3640 1 --2- 172.21.15.67:0/978765602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fece815e0f0 0x7fece814a820 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:56.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.163+0000 7feceedf3640 1 --2- 172.21.15.67:0/978765602 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fece80b9af0 0x7fece814a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:56.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.163+0000 7feceedf3640 1 --2- 172.21.15.67:0/978765602 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fece80b6dc0 0x7fece81f4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:56.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.163+0000 7feceedf3640 1 -- 172.21.15.67:0/978765602 >> 172.21.15.67:0/978765602 conn(0x7fece8072b60 msgr2=0x7fece815d330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:56.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.163+0000 7feceedf3640 1 -- 172.21.15.67:0/978765602 shutdown_connections 2024-03-24T00:39:56.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.163+0000 7feceedf3640 1 -- 172.21.15.67:0/978765602 wait complete. 2024-03-24T00:39:56.174 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-03-24T00:39:56.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.276+0000 7f5ebe39f640 1 Processor -- start 2024-03-24T00:39:56.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.277+0000 7f5ebe39f640 1 -- start start 2024-03-24T00:39:56.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.277+0000 7f5ebe39f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5eb815dd90 0x7f5eb80b5690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:56.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.277+0000 7f5ebe39f640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5eb815e170 0x7f5eb80b5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:56.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.277+0000 7f5ebe39f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eb80cf050 con 0x7f5eb80b6e20 2024-03-24T00:39:56.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.277+0000 7f5ebe39f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5eb8161dd0 con 0x7f5eb815dd90 2024-03-24T00:39:56.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.277+0000 7f5ebe39f640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb80d24d0 con 0x7f5eb815e170 2024-03-24T00:39:56.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.278+0000 7f5ebd39d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5eb815dd90 0x7f5eb80b5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:56.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.278+0000 7f5ebcb9c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5eb80b6e20 0x7f5eb80ce470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49084/0 (socket says 172.21.15.67:49084) 2024-03-24T00:39:56.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.278+0000 7f5ebcb9c640 1 -- 172.21.15.67:0/2823917964 learned_addr learned my addr 172.21.15.67:0/2823917964 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:56.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.278+0000 7f5ebd39d640 1 -- 172.21.15.67:0/2823917964 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5eb815e170 msgr2=0x7f5eb80b5bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:56.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.278+0000 7f5ebdb9e640 1 --2- 172.21.15.67:0/2823917964 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5eb815e170 0x7f5eb80b5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:56.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.278+0000 7f5ebd39d640 1 --2- 172.21.15.67:0/2823917964 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5eb815e170 0x7f5eb80b5bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:56.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.278+0000 7f5eaffff640 1 -- 172.21.15.67:0/2823917964 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2980855714 0 0) 0x7f5eb80cf050 con 0x7f5eb80b6e20 2024-03-24T00:39:56.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.278+0000 7f5ebd39d640 1 -- 172.21.15.67:0/2823917964 >> v1:172.21.15.67:6789/0 conn(0x7f5eb80b6e20 legacy=0x7f5eb80ce470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:56.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.278+0000 7f5ebd39d640 1 -- 172.21.15.67:0/2823917964 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb80d1e80 con 0x7f5eb815dd90 2024-03-24T00:39:56.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.278+0000 7f5ebdb9e640 1 --2- 172.21.15.67:0/2823917964 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5eb815e170 0x7f5eb80b5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:39:56.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.279+0000 7f5ebd39d640 1 --2- 172.21.15.67:0/2823917964 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5eb815dd90 0x7f5eb80b5690 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f5eb40097a0 tx=0x7f5eb402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8101828d90407546 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:56.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.279+0000 7f5eaffff640 1 -- 172.21.15.67:0/2823917964 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb4035020 con 0x7f5eb815dd90 2024-03-24T00:39:56.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.279+0000 7f5eaffff640 1 -- 172.21.15.67:0/2823917964 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eb4031830 con 0x7f5eb815dd90 2024-03-24T00:39:56.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.279+0000 7f5eaffff640 1 -- 172.21.15.67:0/2823917964 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb4031c00 con 0x7f5eb815dd90 2024-03-24T00:39:56.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.279+0000 7f5ebe39f640 1 -- 172.21.15.67:0/2823917964 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5eb815dd90 msgr2=0x7f5eb80b5690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:56.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.279+0000 7f5ebe39f640 1 --2- 172.21.15.67:0/2823917964 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5eb815dd90 0x7f5eb80b5690 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f5eb40097a0 tx=0x7f5eb402b8d0 comp rx=0 tx=0).stop 2024-03-24T00:39:56.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.279+0000 7f5ebe39f640 1 -- 172.21.15.67:0/2823917964 shutdown_connections 2024-03-24T00:39:56.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.279+0000 7f5ebe39f640 1 --2- 172.21.15.67:0/2823917964 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5eb815e170 0x7f5eb80b5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:56.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.279+0000 7f5ebe39f640 1 --2- 172.21.15.67:0/2823917964 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5eb815dd90 0x7f5eb80b5690 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:56.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.279+0000 7f5ebe39f640 1 -- 172.21.15.67:0/2823917964 >> 172.21.15.67:0/2823917964 conn(0x7f5eb80d0690 msgr2=0x7f5eb80d0aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:56.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.279+0000 7f5ebe39f640 1 -- 172.21.15.67:0/2823917964 shutdown_connections 2024-03-24T00:39:56.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.279+0000 7f5ebe39f640 1 -- 172.21.15.67:0/2823917964 wait complete. 2024-03-24T00:39:56.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.280+0000 7f5ebe39f640 1 Processor -- start 2024-03-24T00:39:56.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.281+0000 7f5ebe39f640 1 -- start start 2024-03-24T00:39:56.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.281+0000 7f5ebe39f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5eb80b6e20 0x7f5eb8160b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:56.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.281+0000 7f5ebe39f640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5eb815dd90 0x7f5eb814fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:56.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.281+0000 7f5ebe39f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5eb815e170 0x7f5eb8150110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:56.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.282+0000 7f5ebe39f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb80cf4b0 con 0x7f5eb815e170 2024-03-24T00:39:56.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.282+0000 7f5ebe39f640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb80cf330 con 0x7f5eb80b6e20 2024-03-24T00:39:56.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.282+0000 7f5ebe39f640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb80ceb80 con 0x7f5eb815dd90 2024-03-24T00:39:56.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.282+0000 7f5ebdb9e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5eb815e170 0x7f5eb8150110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:56.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.282+0000 7f5ebdb9e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5eb815e170 0x7f5eb8150110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39680/0 (socket says 172.21.15.67:39680) 2024-03-24T00:39:56.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.282+0000 7f5ebdb9e640 1 -- 172.21.15.67:0/3760941779 learned_addr learned my addr 172.21.15.67:0/3760941779 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:56.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.282+0000 7f5ebcb9c640 1 --2- 172.21.15.67:0/3760941779 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5eb815dd90 0x7f5eb814fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:56.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.282+0000 7f5ebd39d640 1 --2- 172.21.15.67:0/3760941779 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5eb80b6e20 0x7f5eb8160b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:56.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.282+0000 7f5ebdb9e640 1 -- 172.21.15.67:0/3760941779 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5eb815dd90 msgr2=0x7f5eb814fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:56.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.282+0000 7f5ebdb9e640 1 --2- 172.21.15.67:0/3760941779 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5eb815dd90 0x7f5eb814fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:56.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.282+0000 7f5ebdb9e640 1 -- 172.21.15.67:0/3760941779 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5eb80b6e20 msgr2=0x7f5eb8160b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:56.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.282+0000 7f5ebdb9e640 1 --2- 172.21.15.67:0/3760941779 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5eb80b6e20 0x7f5eb8160b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:56.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.282+0000 7f5ebdb9e640 1 -- 172.21.15.67:0/3760941779 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb80c9a60 con 0x7f5eb815e170 2024-03-24T00:39:56.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.282+0000 7f5ebdb9e640 1 --2- 172.21.15.67:0/3760941779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5eb815e170 0x7f5eb8150110 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f5eb000cc60 tx=0x7f5eb000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:56.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.282+0000 7f5eadffb640 1 -- 172.21.15.67:0/3760941779 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb0003b90 con 0x7f5eb815e170 2024-03-24T00:39:56.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.282+0000 7f5eadffb640 1 -- 172.21.15.67:0/3760941779 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eb0003d50 con 0x7f5eb815e170 2024-03-24T00:39:56.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.282+0000 7f5eadffb640 1 -- 172.21.15.67:0/3760941779 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb0004260 con 0x7f5eb815e170 2024-03-24T00:39:56.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.282+0000 7f5ebe39f640 1 -- 172.21.15.67:0/3760941779 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5eb80c9d40 con 0x7f5eb815e170 2024-03-24T00:39:56.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.283+0000 7f5ebe39f640 1 -- 172.21.15.67:0/3760941779 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5eb80ca300 con 0x7f5eb815e170 2024-03-24T00:39:56.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.283+0000 7f5ebe39f640 1 -- 172.21.15.67:0/3760941779 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5eb80751c0 con 0x7f5eb815e170 2024-03-24T00:39:56.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.284+0000 7f5eadffb640 1 -- 172.21.15.67:0/3760941779 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5eb0004400 con 0x7f5eb815e170 2024-03-24T00:39:56.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.284+0000 7f5eadffb640 1 --2- 172.21.15.67:0/3760941779 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5e9c06bbd0 0x7f5e9c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:56.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.284+0000 7f5eadffb640 1 -- 172.21.15.67:0/3760941779 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f5eb008b170 con 0x7f5eb815e170 2024-03-24T00:39:56.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.284+0000 7f5ebd39d640 1 --2- 172.21.15.67:0/3760941779 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5e9c06bbd0 0x7f5e9c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:56.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.285+0000 7f5ebd39d640 1 --2- 172.21.15.67:0/3760941779 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5e9c06bbd0 0x7f5e9c06e090 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5eb4036000 tx=0x7f5eb4008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:56.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.288+0000 7f5eadffb640 1 -- 172.21.15.67:0/3760941779 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5eb0014030 con 0x7f5eb815e170 2024-03-24T00:39:56.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.543+0000 7f5ebe39f640 1 -- 172.21.15.67:0/3760941779 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f5eb80ca4d0 con 0x7f5eb815e170 2024-03-24T00:39:56.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.545+0000 7f5eadffb640 1 -- 172.21.15.67:0/3760941779 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v9) v1 ==== 90+0+0 (secure 0 0 0) 0x7f5eb0059860 con 0x7f5eb815e170 2024-03-24T00:39:56.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.547+0000 7f5ebe39f640 1 -- 172.21.15.67:0/3760941779 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5e9c06bbd0 msgr2=0x7f5e9c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:56.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.547+0000 7f5ebe39f640 1 --2- 172.21.15.67:0/3760941779 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5e9c06bbd0 0x7f5e9c06e090 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5eb4036000 tx=0x7f5eb4008030 comp rx=0 tx=0).stop 2024-03-24T00:39:56.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.547+0000 7f5ebe39f640 1 -- 172.21.15.67:0/3760941779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5eb815e170 msgr2=0x7f5eb8150110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:56.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.547+0000 7f5ebe39f640 1 --2- 172.21.15.67:0/3760941779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5eb815e170 0x7f5eb8150110 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f5eb000cc60 tx=0x7f5eb000a650 comp rx=0 tx=0).stop 2024-03-24T00:39:56.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.547+0000 7f5ebe39f640 1 -- 172.21.15.67:0/3760941779 shutdown_connections 2024-03-24T00:39:56.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.547+0000 7f5ebe39f640 1 --2- 172.21.15.67:0/3760941779 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5e9c06bbd0 0x7f5e9c06e090 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:56.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.547+0000 7f5ebe39f640 1 --2- 172.21.15.67:0/3760941779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5eb815e170 0x7f5eb8150110 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:56.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.547+0000 7f5ebe39f640 1 --2- 172.21.15.67:0/3760941779 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5eb815dd90 0x7f5eb814fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:56.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.547+0000 7f5ebe39f640 1 --2- 172.21.15.67:0/3760941779 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5eb80b6e20 0x7f5eb8160b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:56.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.547+0000 7f5ebe39f640 1 -- 172.21.15.67:0/3760941779 >> 172.21.15.67:0/3760941779 conn(0x7f5eb80d0690 msgr2=0x7f5eb80b3670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:56.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.547+0000 7f5ebe39f640 1 -- 172.21.15.67:0/3760941779 shutdown_connections 2024-03-24T00:39:56.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.547+0000 7f5ebe39f640 1 -- 172.21.15.67:0/3760941779 wait complete. 2024-03-24T00:39:56.559 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-03-24T00:39:56.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.660+0000 7fbdb0159640 1 Processor -- start 2024-03-24T00:39:56.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.660+0000 7fbdb0159640 1 -- start start 2024-03-24T00:39:56.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.661+0000 7fbdb0159640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fbda8158990 0x7fbda80d0b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:56.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.661+0000 7fbdb0159640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbda815f140 con 0x7fbda806e3c0 2024-03-24T00:39:56.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.661+0000 7fbdb0159640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbda815f420 con 0x7fbda8099bf0 2024-03-24T00:39:56.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.661+0000 7fbdb0159640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbda80d06f0 con 0x7fbda8158990 2024-03-24T00:39:56.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.661+0000 7fbdadef5640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbda806e3c0 0x7fbda806e7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49086/0 (socket says 172.21.15.67:49086) 2024-03-24T00:39:56.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.661+0000 7fbdadef5640 1 -- 172.21.15.67:0/2163217772 learned_addr learned my addr 172.21.15.67:0/2163217772 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:56.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.661+0000 7fbdad6f4640 1 --2- 172.21.15.67:0/2163217772 >> v2:172.21.15.72:3300/0 conn(0x7fbda8158990 0x7fbda80d0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:56.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.661+0000 7fbdacef3640 1 -- 172.21.15.67:0/2163217772 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1299663624 0 0) 0x7fbda815f140 con 0x7fbda806e3c0 2024-03-24T00:39:56.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.661+0000 7fbdacef3640 1 -- 172.21.15.67:0/2163217772 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd94003520 con 0x7fbda806e3c0 2024-03-24T00:39:56.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.661+0000 7fbdacef3640 1 -- 172.21.15.67:0/2163217772 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 220265767 0 0) 0x7fbd94003520 con 0x7fbda806e3c0 2024-03-24T00:39:56.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.661+0000 7fbdacef3640 1 -- 172.21.15.67:0/2163217772 >> v2:172.21.15.72:3300/0 conn(0x7fbda8158990 msgr2=0x7fbda80d0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:56.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.661+0000 7fbdacef3640 1 --2- 172.21.15.67:0/2163217772 >> v2:172.21.15.72:3300/0 conn(0x7fbda8158990 0x7fbda80d0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:56.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.661+0000 7fbdacef3640 1 -- 172.21.15.67:0/2163217772 >> v1:172.21.15.72:6789/0 conn(0x7fbda8099bf0 legacy=0x7fbda806b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:56.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.661+0000 7fbdacef3640 1 -- 172.21.15.67:0/2163217772 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbda80d00a0 con 0x7fbda806e3c0 2024-03-24T00:39:56.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.661+0000 7fbdacef3640 1 -- 172.21.15.67:0/2163217772 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7fbda4001670 con 0x7fbda806e3c0 2024-03-24T00:39:56.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.661+0000 7fbdacef3640 1 -- 172.21.15.67:0/2163217772 >> v1:172.21.15.67:6789/0 conn(0x7fbda806e3c0 legacy=0x7fbda806e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:56.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.662+0000 7fbdacef3640 1 --2- 172.21.15.67:0/2163217772 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd940039b0 0x7fbd94003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:56.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.662+0000 7fbdacef3640 1 --2- 172.21.15.67:0/2163217772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd940043b0 0x7fbd94004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:56.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.662+0000 7fbdacef3640 1 --2- 172.21.15.67:0/2163217772 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbd94005080 0x7fbd9400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:56.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.662+0000 7fbdacef3640 1 -- 172.21.15.67:0/2163217772 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbda80d06f0 con 0x7fbd940043b0 2024-03-24T00:39:56.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.662+0000 7fbdacef3640 1 -- 172.21.15.67:0/2163217772 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9400bfe0 con 0x7fbd940039b0 2024-03-24T00:39:56.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.662+0000 7fbdacef3640 1 -- 172.21.15.67:0/2163217772 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9400c1e0 con 0x7fbd94005080 2024-03-24T00:39:56.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.662+0000 7fbdad6f4640 1 --2- 172.21.15.67:0/2163217772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd940043b0 0x7fbd94004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:56.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.662+0000 7fbdadef5640 1 --2- 172.21.15.67:0/2163217772 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd940039b0 0x7fbd94003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:56.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.662+0000 7fbdae6f6640 1 --2- 172.21.15.67:0/2163217772 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbd94005080 0x7fbd9400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:56.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.662+0000 7fbdad6f4640 1 -- 172.21.15.67:0/2163217772 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbd94005080 msgr2=0x7fbd9400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:56.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.662+0000 7fbdad6f4640 1 --2- 172.21.15.67:0/2163217772 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbd94005080 0x7fbd9400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:56.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.662+0000 7fbdad6f4640 1 -- 172.21.15.67:0/2163217772 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd940039b0 msgr2=0x7fbd94003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:56.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.662+0000 7fbdad6f4640 1 --2- 172.21.15.67:0/2163217772 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd940039b0 0x7fbd94003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:56.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.662+0000 7fbdad6f4640 1 -- 172.21.15.67:0/2163217772 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd94003520 con 0x7fbd940043b0 2024-03-24T00:39:56.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.662+0000 7fbdae6f6640 1 --2- 172.21.15.67:0/2163217772 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbd94005080 0x7fbd9400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:56.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.662+0000 7fbdae6f6640 1 -- 172.21.15.67:0/2163217772 reap_dead start 2024-03-24T00:39:56.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.663+0000 7fbdad6f4640 1 --2- 172.21.15.67:0/2163217772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd940043b0 0x7fbd94004880 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fbd98009ac0 tx=0x7fbd94005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e0998f3980d802f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:56.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.663+0000 7fbdacef3640 1 -- 172.21.15.67:0/2163217772 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd98039070 con 0x7fbd940043b0 2024-03-24T00:39:56.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.663+0000 7fbdacef3640 1 -- 172.21.15.67:0/2163217772 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd9802c6b0 con 0x7fbd940043b0 2024-03-24T00:39:56.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.663+0000 7fbdacef3640 1 -- 172.21.15.67:0/2163217772 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd98009690 con 0x7fbd940043b0 2024-03-24T00:39:56.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.663+0000 7fbdb0159640 1 -- 172.21.15.67:0/2163217772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd940043b0 msgr2=0x7fbd94004880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:56.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.663+0000 7fbdb0159640 1 --2- 172.21.15.67:0/2163217772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd940043b0 0x7fbd94004880 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fbd98009ac0 tx=0x7fbd94005830 comp rx=0 tx=0).stop 2024-03-24T00:39:56.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.663+0000 7fbdb0159640 1 -- 172.21.15.67:0/2163217772 shutdown_connections 2024-03-24T00:39:56.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.663+0000 7fbdb0159640 1 --2- 172.21.15.67:0/2163217772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd940043b0 0x7fbd94004880 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:56.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.663+0000 7fbdb0159640 1 -- 172.21.15.67:0/2163217772 >> 172.21.15.67:0/2163217772 conn(0x7fbda8072860 msgr2=0x7fbda80b5030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:56.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.663+0000 7fbdb0159640 1 -- 172.21.15.67:0/2163217772 shutdown_connections 2024-03-24T00:39:56.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.663+0000 7fbdb0159640 1 -- 172.21.15.67:0/2163217772 wait complete. 2024-03-24T00:39:56.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.665+0000 7fbdb0159640 1 Processor -- start 2024-03-24T00:39:56.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.665+0000 7fbdb0159640 1 -- start start 2024-03-24T00:39:56.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.665+0000 7fbdb0159640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd940043b0 0x7fbda8153770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:56.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.665+0000 7fbdb0159640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbda80b0040 0x7fbda80b24c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:56.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.665+0000 7fbdb0159640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbda80b0f60 0x7fbda8152770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:56.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.665+0000 7fbdb0159640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbda80d0d00 con 0x7fbda80b0f60 2024-03-24T00:39:56.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.665+0000 7fbdb0159640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbda80d0b80 con 0x7fbd940043b0 2024-03-24T00:39:56.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.665+0000 7fbdb0159640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbda80d0e80 con 0x7fbda80b0040 2024-03-24T00:39:56.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.665+0000 7fbdae6f6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbda80b0f60 0x7fbda8152770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:56.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.665+0000 7fbdae6f6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbda80b0f60 0x7fbda8152770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39704/0 (socket says 172.21.15.67:39704) 2024-03-24T00:39:56.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.665+0000 7fbdae6f6640 1 -- 172.21.15.67:0/73076098 learned_addr learned my addr 172.21.15.67:0/73076098 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:56.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.665+0000 7fbdadef5640 1 --2- 172.21.15.67:0/73076098 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd940043b0 0x7fbda8153770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:56.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.665+0000 7fbdad6f4640 1 --2- 172.21.15.67:0/73076098 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbda80b0040 0x7fbda80b24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:56.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.666+0000 7fbdae6f6640 1 -- 172.21.15.67:0/73076098 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbda80b0040 msgr2=0x7fbda80b24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:56.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.666+0000 7fbdae6f6640 1 --2- 172.21.15.67:0/73076098 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbda80b0040 0x7fbda80b24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:56.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.666+0000 7fbdae6f6640 1 -- 172.21.15.67:0/73076098 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd940043b0 msgr2=0x7fbda8153770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:56.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.666+0000 7fbdae6f6640 1 --2- 172.21.15.67:0/73076098 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd940043b0 0x7fbda8153770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:56.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.666+0000 7fbdae6f6640 1 -- 172.21.15.67:0/73076098 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbda80b1520 con 0x7fbda80b0f60 2024-03-24T00:39:56.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.666+0000 7fbdae6f6640 1 --2- 172.21.15.67:0/73076098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbda80b0f60 0x7fbda8152770 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fbda80730f0 tx=0x7fbda000ebe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:56.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.666+0000 7fbd9e7fc640 1 -- 172.21.15.67:0/73076098 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbda0019070 con 0x7fbda80b0f60 2024-03-24T00:39:56.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.666+0000 7fbd9e7fc640 1 -- 172.21.15.67:0/73076098 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbda0006b50 con 0x7fbda80b0f60 2024-03-24T00:39:56.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.666+0000 7fbdb0159640 1 -- 172.21.15.67:0/73076098 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbda815a740 con 0x7fbda80b0f60 2024-03-24T00:39:56.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.666+0000 7fbd9e7fc640 1 -- 172.21.15.67:0/73076098 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbda0003970 con 0x7fbda80b0f60 2024-03-24T00:39:56.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.666+0000 7fbdb0159640 1 -- 172.21.15.67:0/73076098 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbda815ad00 con 0x7fbda80b0f60 2024-03-24T00:39:56.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.667+0000 7fbdb0159640 1 -- 172.21.15.67:0/73076098 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbda804f400 con 0x7fbda80b0f60 2024-03-24T00:39:56.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.668+0000 7fbd9e7fc640 1 -- 172.21.15.67:0/73076098 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fbda001c070 con 0x7fbda80b0f60 2024-03-24T00:39:56.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.668+0000 7fbd9e7fc640 1 --2- 172.21.15.67:0/73076098 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbd8c06bbd0 0x7fbd8c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:56.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.668+0000 7fbd9e7fc640 1 -- 172.21.15.67:0/73076098 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fbda0089b60 con 0x7fbda80b0f60 2024-03-24T00:39:56.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.668+0000 7fbdadef5640 1 --2- 172.21.15.67:0/73076098 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbd8c06bbd0 0x7fbd8c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:56.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.669+0000 7fbdadef5640 1 --2- 172.21.15.67:0/73076098 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbd8c06bbd0 0x7fbd8c06e090 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fbda4001cf0 tx=0x7fbda4000ed0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:56.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.672+0000 7fbd9e7fc640 1 -- 172.21.15.67:0/73076098 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fbda00581c0 con 0x7fbda80b0f60 2024-03-24T00:39:56.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.923+0000 7fbdb0159640 1 -- 172.21.15.67:0/73076098 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7fbda815aed0 con 0x7fbda80b0f60 2024-03-24T00:39:56.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.925+0000 7fbd9e7fc640 1 -- 172.21.15.67:0/73076098 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7fbda005c260 con 0x7fbda80b0f60 2024-03-24T00:39:56.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.928+0000 7fbdb0159640 1 -- 172.21.15.67:0/73076098 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbd8c06bbd0 msgr2=0x7fbd8c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:56.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.928+0000 7fbdb0159640 1 --2- 172.21.15.67:0/73076098 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbd8c06bbd0 0x7fbd8c06e090 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fbda4001cf0 tx=0x7fbda4000ed0 comp rx=0 tx=0).stop 2024-03-24T00:39:56.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.928+0000 7fbdb0159640 1 -- 172.21.15.67:0/73076098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbda80b0f60 msgr2=0x7fbda8152770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:56.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.928+0000 7fbdb0159640 1 --2- 172.21.15.67:0/73076098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbda80b0f60 0x7fbda8152770 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fbda80730f0 tx=0x7fbda000ebe0 comp rx=0 tx=0).stop 2024-03-24T00:39:56.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.928+0000 7fbdb0159640 1 -- 172.21.15.67:0/73076098 shutdown_connections 2024-03-24T00:39:56.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.928+0000 7fbdb0159640 1 --2- 172.21.15.67:0/73076098 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbd8c06bbd0 0x7fbd8c06e090 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:56.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.928+0000 7fbdb0159640 1 --2- 172.21.15.67:0/73076098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbda80b0f60 0x7fbda8152770 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:56.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.928+0000 7fbdb0159640 1 --2- 172.21.15.67:0/73076098 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbda80b0040 0x7fbda80b24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:56.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.928+0000 7fbdb0159640 1 --2- 172.21.15.67:0/73076098 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd940043b0 0x7fbda8153770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:56.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.928+0000 7fbdb0159640 1 -- 172.21.15.67:0/73076098 >> 172.21.15.67:0/73076098 conn(0x7fbda8072860 msgr2=0x7fbda815da50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:56.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.928+0000 7fbdb0159640 1 -- 172.21.15.67:0/73076098 shutdown_connections 2024-03-24T00:39:56.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:56.928+0000 7fbdb0159640 1 -- 172.21.15.67:0/73076098 wait complete. 2024-03-24T00:39:56.939 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-03-24T00:39:57.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.042+0000 7fb0b644b640 1 Processor -- start 2024-03-24T00:39:57.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.042+0000 7fb0b644b640 1 -- start start 2024-03-24T00:39:57.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.042+0000 7fb0b644b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb0b006de60 0x7fb0b009ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:57.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.043+0000 7fb0b644b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb0b00b1eb0 0x7fb0b0099d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:57.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.043+0000 7fb0b644b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0b00b29b0 con 0x7fb0b009a240 2024-03-24T00:39:57.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.043+0000 7fb0b644b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0b00cc9e0 con 0x7fb0b00b1eb0 2024-03-24T00:39:57.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.043+0000 7fb0b644b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b00ccf80 con 0x7fb0b006de60 2024-03-24T00:39:57.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.043+0000 7fb0af7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb0b00b1eb0 0x7fb0b0099d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:57.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.043+0000 7fb0affff640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb0b006de60 0x7fb0b009ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:57.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.043+0000 7fb0b49e8640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fb0b009a240 0x7fb0b00cff00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:38316/0 (socket says 172.21.15.67:38316) 2024-03-24T00:39:57.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.043+0000 7fb0af7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb0b00b1eb0 0x7fb0b0099d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:49794/0 (socket says 172.21.15.67:49794) 2024-03-24T00:39:57.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.043+0000 7fb0b49e8640 1 -- 172.21.15.67:0/142487647 learned_addr learned my addr 172.21.15.67:0/142487647 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:57.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.043+0000 7fb0af7fe640 1 -- 172.21.15.67:0/142487647 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb0b006de60 msgr2=0x7fb0b009ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:57.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.043+0000 7fb0af7fe640 1 --2- 172.21.15.67:0/142487647 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb0b006de60 0x7fb0b009ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:57.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.043+0000 7fb0af7fe640 1 -- 172.21.15.67:0/142487647 >> v1:172.21.15.72:6789/0 conn(0x7fb0b009a240 legacy=0x7fb0b00cff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:57.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.043+0000 7fb0af7fe640 1 -- 172.21.15.67:0/142487647 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0b00b23f0 con 0x7fb0b00b1eb0 2024-03-24T00:39:57.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.043+0000 7fb0affff640 1 --2- 172.21.15.67:0/142487647 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb0b006de60 0x7fb0b009ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:57.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.044+0000 7fb0af7fe640 1 --2- 172.21.15.67:0/142487647 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb0b00b1eb0 0x7fb0b0099d00 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb0a40097a0 tx=0x7fb0a402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed5b505ed6899e11 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:57.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.044+0000 7fb0aeffd640 1 -- 172.21.15.67:0/142487647 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a4035020 con 0x7fb0b00b1eb0 2024-03-24T00:39:57.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.044+0000 7fb0aeffd640 1 -- 172.21.15.67:0/142487647 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0a4031830 con 0x7fb0b00b1eb0 2024-03-24T00:39:57.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.044+0000 7fb0aeffd640 1 -- 172.21.15.67:0/142487647 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a4031c00 con 0x7fb0b00b1eb0 2024-03-24T00:39:57.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.045+0000 7fb0b644b640 1 -- 172.21.15.67:0/142487647 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb0b00b1eb0 msgr2=0x7fb0b0099d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:57.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.045+0000 7fb0b644b640 1 --2- 172.21.15.67:0/142487647 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb0b00b1eb0 0x7fb0b0099d00 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb0a40097a0 tx=0x7fb0a402b8d0 comp rx=0 tx=0).stop 2024-03-24T00:39:57.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.045+0000 7fb0b644b640 1 -- 172.21.15.67:0/142487647 shutdown_connections 2024-03-24T00:39:57.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.045+0000 7fb0b644b640 1 --2- 172.21.15.67:0/142487647 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb0b00b1eb0 0x7fb0b0099d00 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:57.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.045+0000 7fb0b644b640 1 --2- 172.21.15.67:0/142487647 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb0b006de60 0x7fb0b009ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:57.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.045+0000 7fb0b644b640 1 -- 172.21.15.67:0/142487647 >> 172.21.15.67:0/142487647 conn(0x7fb0b00b3650 msgr2=0x7fb0b00b3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:57.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.045+0000 7fb0b644b640 1 -- 172.21.15.67:0/142487647 shutdown_connections 2024-03-24T00:39:57.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.045+0000 7fb0b644b640 1 -- 172.21.15.67:0/142487647 wait complete. 2024-03-24T00:39:57.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.046+0000 7fb0b644b640 1 Processor -- start 2024-03-24T00:39:57.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.046+0000 7fb0b644b640 1 -- start start 2024-03-24T00:39:57.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.047+0000 7fb0b644b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb0b006de60 0x7fb0b01f38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:57.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.047+0000 7fb0b644b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb0b009a240 0x7fb0b0157150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:57.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.047+0000 7fb0b644b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0b00b1eb0 0x7fb0b00b6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:57.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.047+0000 7fb0b644b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b009b3c0 con 0x7fb0b00b1eb0 2024-03-24T00:39:57.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.047+0000 7fb0b644b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b00ccc10 con 0x7fb0b009a240 2024-03-24T00:39:57.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.047+0000 7fb0b644b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b006c150 con 0x7fb0b006de60 2024-03-24T00:39:57.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.047+0000 7fb0b49e8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0b00b1eb0 0x7fb0b00b6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:57.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.047+0000 7fb0b49e8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0b00b1eb0 0x7fb0b00b6410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39714/0 (socket says 172.21.15.67:39714) 2024-03-24T00:39:57.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.047+0000 7fb0b49e8640 1 -- 172.21.15.67:0/3478492426 learned_addr learned my addr 172.21.15.67:0/3478492426 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:57.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.047+0000 7fb0af7fe640 1 --2- 172.21.15.67:0/3478492426 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb0b009a240 0x7fb0b0157150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:57.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.047+0000 7fb0affff640 1 --2- 172.21.15.67:0/3478492426 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb0b006de60 0x7fb0b01f38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:57.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.047+0000 7fb0b49e8640 1 -- 172.21.15.67:0/3478492426 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb0b006de60 msgr2=0x7fb0b01f38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:57.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.047+0000 7fb0b49e8640 1 --2- 172.21.15.67:0/3478492426 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb0b006de60 0x7fb0b01f38a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:57.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.047+0000 7fb0b49e8640 1 -- 172.21.15.67:0/3478492426 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb0b009a240 msgr2=0x7fb0b0157150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:57.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.047+0000 7fb0b49e8640 1 --2- 172.21.15.67:0/3478492426 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb0b009a240 0x7fb0b0157150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:57.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.047+0000 7fb0b49e8640 1 -- 172.21.15.67:0/3478492426 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0b0157690 con 0x7fb0b00b1eb0 2024-03-24T00:39:57.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.047+0000 7fb0b49e8640 1 --2- 172.21.15.67:0/3478492426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0b00b1eb0 0x7fb0b00b6410 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fb0a800cbc0 tx=0x7fb0a800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:57.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.048+0000 7fb0acff9640 1 -- 172.21.15.67:0/3478492426 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a8017070 con 0x7fb0b00b1eb0 2024-03-24T00:39:57.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.048+0000 7fb0acff9640 1 -- 172.21.15.67:0/3478492426 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0a800ad40 con 0x7fb0b00b1eb0 2024-03-24T00:39:57.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.048+0000 7fb0b644b640 1 -- 172.21.15.67:0/3478492426 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0b01f3ed0 con 0x7fb0b00b1eb0 2024-03-24T00:39:57.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.048+0000 7fb0acff9640 1 -- 172.21.15.67:0/3478492426 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a8012600 con 0x7fb0b00b1eb0 2024-03-24T00:39:57.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.048+0000 7fb0b644b640 1 -- 172.21.15.67:0/3478492426 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0b01f4490 con 0x7fb0b00b1eb0 2024-03-24T00:39:57.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.049+0000 7fb0b644b640 1 -- 172.21.15.67:0/3478492426 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0b004f400 con 0x7fb0b00b1eb0 2024-03-24T00:39:57.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.049+0000 7fb0acff9640 1 -- 172.21.15.67:0/3478492426 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb0a801a070 con 0x7fb0b00b1eb0 2024-03-24T00:39:57.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.049+0000 7fb0acff9640 1 --2- 172.21.15.67:0/3478492426 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb09c06bbd0 0x7fb09c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:57.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.049+0000 7fb0acff9640 1 -- 172.21.15.67:0/3478492426 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fb0a808a530 con 0x7fb0b00b1eb0 2024-03-24T00:39:57.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.050+0000 7fb0affff640 1 --2- 172.21.15.67:0/3478492426 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb09c06bbd0 0x7fb09c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:57.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.050+0000 7fb0affff640 1 --2- 172.21.15.67:0/3478492426 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb09c06bbd0 0x7fb09c06e090 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb0a0009560 tx=0x7fb0a0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:57.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.054+0000 7fb0acff9640 1 -- 172.21.15.67:0/3478492426 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb0b0157690 con 0x7fb0b00b1eb0 2024-03-24T00:39:57.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.309+0000 7fb0b644b640 1 -- 172.21.15.67:0/3478492426 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7fb0b00718d0 con 0x7fb0b00b1eb0 2024-03-24T00:39:57.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.311+0000 7fb0acff9640 1 -- 172.21.15.67:0/3478492426 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v11) v1 ==== 97+0+0 (secure 0 0 0) 0x7fb0a8058b90 con 0x7fb0b00b1eb0 2024-03-24T00:39:57.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.313+0000 7fb0b644b640 1 -- 172.21.15.67:0/3478492426 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb09c06bbd0 msgr2=0x7fb09c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:57.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.313+0000 7fb0b644b640 1 --2- 172.21.15.67:0/3478492426 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb09c06bbd0 0x7fb09c06e090 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb0a0009560 tx=0x7fb0a0009190 comp rx=0 tx=0).stop 2024-03-24T00:39:57.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.313+0000 7fb0b644b640 1 -- 172.21.15.67:0/3478492426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0b00b1eb0 msgr2=0x7fb0b00b6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:57.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.313+0000 7fb0b644b640 1 --2- 172.21.15.67:0/3478492426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0b00b1eb0 0x7fb0b00b6410 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fb0a800cbc0 tx=0x7fb0a800a430 comp rx=0 tx=0).stop 2024-03-24T00:39:57.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.314+0000 7fb0b644b640 1 -- 172.21.15.67:0/3478492426 shutdown_connections 2024-03-24T00:39:57.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.314+0000 7fb0b644b640 1 --2- 172.21.15.67:0/3478492426 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb09c06bbd0 0x7fb09c06e090 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:57.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.314+0000 7fb0b644b640 1 --2- 172.21.15.67:0/3478492426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0b00b1eb0 0x7fb0b00b6410 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:57.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.314+0000 7fb0b644b640 1 --2- 172.21.15.67:0/3478492426 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb0b009a240 0x7fb0b0157150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:57.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.314+0000 7fb0b644b640 1 --2- 172.21.15.67:0/3478492426 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb0b006de60 0x7fb0b01f38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:57.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.314+0000 7fb0b644b640 1 -- 172.21.15.67:0/3478492426 >> 172.21.15.67:0/3478492426 conn(0x7fb0b00b3650 msgr2=0x7fb0b00cde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:57.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.314+0000 7fb0b644b640 1 -- 172.21.15.67:0/3478492426 shutdown_connections 2024-03-24T00:39:57.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.314+0000 7fb0b644b640 1 -- 172.21.15.67:0/3478492426 wait complete. 2024-03-24T00:39:57.325 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-03-24T00:39:57.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.427+0000 7f4080d9d640 1 Processor -- start 2024-03-24T00:39:57.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.427+0000 7f4080d9d640 1 -- start start 2024-03-24T00:39:57.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.428+0000 7f4080d9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f407c161930 0x7f407c0ce8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:57.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.428+0000 7f4080d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f407c0cedf0 0x7f407c0d2540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:57.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.428+0000 7f4080d9d640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f407c161e80 con 0x7f407c15df60 2024-03-24T00:39:57.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.428+0000 7f4080d9d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f407c0cf260 con 0x7f407c0cedf0 2024-03-24T00:39:57.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.428+0000 7f4080d9d640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f407c162160 con 0x7f407c161930 2024-03-24T00:39:57.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.428+0000 7f407bfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f407c0cedf0 0x7f407c0d2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:57.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.428+0000 7f407bfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f407c0cedf0 0x7f407c0d2540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39718/0 (socket says 172.21.15.67:39718) 2024-03-24T00:39:57.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.428+0000 7f407bfff640 1 -- 172.21.15.67:0/3859403070 learned_addr learned my addr 172.21.15.67:0/3859403070 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:57.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.428+0000 7f407affd640 1 --2- 172.21.15.67:0/3859403070 >> v2:172.21.15.72:3300/0 conn(0x7f407c161930 0x7f407c0ce8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:57.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.428+0000 7f407bfff640 1 -- 172.21.15.67:0/3859403070 >> v2:172.21.15.72:3300/0 conn(0x7f407c161930 msgr2=0x7f407c0ce8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:57.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.428+0000 7f407bfff640 1 --2- 172.21.15.67:0/3859403070 >> v2:172.21.15.72:3300/0 conn(0x7f407c161930 0x7f407c0ce8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:57.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.428+0000 7f407bfff640 1 -- 172.21.15.67:0/3859403070 >> v1:172.21.15.72:6789/0 conn(0x7f407c15df60 legacy=0x7f407c0b9570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:57.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.428+0000 7f407bfff640 1 -- 172.21.15.67:0/3859403070 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f407c162440 con 0x7f407c0cedf0 2024-03-24T00:39:57.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.428+0000 7f407affd640 1 --2- 172.21.15.67:0/3859403070 >> v2:172.21.15.72:3300/0 conn(0x7f407c161930 0x7f407c0ce8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:39:57.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.429+0000 7f407bfff640 1 --2- 172.21.15.67:0/3859403070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f407c0cedf0 0x7f407c0d2540 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f4074009e50 tx=0x7f407402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d90fffd9ee8ff228 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:57.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.429+0000 7f407a7fc640 1 -- 172.21.15.67:0/3859403070 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f407402ca80 con 0x7f407c0cedf0 2024-03-24T00:39:57.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.429+0000 7f407a7fc640 1 -- 172.21.15.67:0/3859403070 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f407402cc20 con 0x7f407c0cedf0 2024-03-24T00:39:57.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.429+0000 7f407a7fc640 1 -- 172.21.15.67:0/3859403070 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40740343f0 con 0x7f407c0cedf0 2024-03-24T00:39:57.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.430+0000 7f4080d9d640 1 -- 172.21.15.67:0/3859403070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f407c0cedf0 msgr2=0x7f407c0d2540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:57.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.430+0000 7f4080d9d640 1 --2- 172.21.15.67:0/3859403070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f407c0cedf0 0x7f407c0d2540 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f4074009e50 tx=0x7f407402bee0 comp rx=0 tx=0).stop 2024-03-24T00:39:57.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.430+0000 7f4080d9d640 1 -- 172.21.15.67:0/3859403070 shutdown_connections 2024-03-24T00:39:57.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.430+0000 7f4080d9d640 1 --2- 172.21.15.67:0/3859403070 >> v2:172.21.15.72:3300/0 conn(0x7f407c161930 0x7f407c0ce8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:57.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.430+0000 7f4080d9d640 1 --2- 172.21.15.67:0/3859403070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f407c0cedf0 0x7f407c0d2540 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:57.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.430+0000 7f4080d9d640 1 -- 172.21.15.67:0/3859403070 >> 172.21.15.67:0/3859403070 conn(0x7f407c072b60 msgr2=0x7f407c072f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:57.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.430+0000 7f4080d9d640 1 -- 172.21.15.67:0/3859403070 shutdown_connections 2024-03-24T00:39:57.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.430+0000 7f4080d9d640 1 -- 172.21.15.67:0/3859403070 wait complete. 2024-03-24T00:39:57.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.431+0000 7f4080d9d640 1 Processor -- start 2024-03-24T00:39:57.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.431+0000 7f4080d9d640 1 -- start start 2024-03-24T00:39:57.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.432+0000 7f4080d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f407c0cedf0 0x7f407c0c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:57.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.432+0000 7f4080d9d640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f407c15df60 0x7f407c0cb6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:57.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.432+0000 7f4080d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f407c161930 0x7f407c0cc810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:57.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.432+0000 7f4080d9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f407c0d2a80 con 0x7f407c0cedf0 2024-03-24T00:39:57.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.432+0000 7f4080d9d640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f407c0d2020 con 0x7f407c161930 2024-03-24T00:39:57.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.432+0000 7f4080d9d640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f407c0d2c00 con 0x7f407c15df60 2024-03-24T00:39:57.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.432+0000 7f407b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f407c0cedf0 0x7f407c0c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:57.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.432+0000 7f407b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f407c0cedf0 0x7f407c0c9bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39732/0 (socket says 172.21.15.67:39732) 2024-03-24T00:39:57.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.432+0000 7f407b7fe640 1 -- 172.21.15.67:0/4125389994 learned_addr learned my addr 172.21.15.67:0/4125389994 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:57.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.432+0000 7f407bfff640 1 --2- 172.21.15.67:0/4125389994 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f407c161930 0x7f407c0cc810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:57.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.432+0000 7f407affd640 1 --2- 172.21.15.67:0/4125389994 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f407c15df60 0x7f407c0cb6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:57.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.432+0000 7f407b7fe640 1 -- 172.21.15.67:0/4125389994 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f407c15df60 msgr2=0x7f407c0cb6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:57.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.432+0000 7f407b7fe640 1 --2- 172.21.15.67:0/4125389994 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f407c15df60 0x7f407c0cb6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:57.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.432+0000 7f407b7fe640 1 -- 172.21.15.67:0/4125389994 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f407c161930 msgr2=0x7f407c0cc810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:57.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.432+0000 7f407b7fe640 1 --2- 172.21.15.67:0/4125389994 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f407c161930 0x7f407c0cc810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:57.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.432+0000 7f407b7fe640 1 -- 172.21.15.67:0/4125389994 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f407c0cc080 con 0x7f407c0cedf0 2024-03-24T00:39:57.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.432+0000 7f407affd640 1 --2- 172.21.15.67:0/4125389994 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f407c15df60 0x7f407c0cb6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:39:57.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.432+0000 7f407b7fe640 1 --2- 172.21.15.67:0/4125389994 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f407c0cedf0 0x7f407c0c9bc0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f406c00c430 tx=0x7f406c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:57.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.432+0000 7f4063fff640 1 -- 172.21.15.67:0/4125389994 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f406c011790 con 0x7f407c0cedf0 2024-03-24T00:39:57.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.432+0000 7f4063fff640 1 -- 172.21.15.67:0/4125389994 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f406c017070 con 0x7f407c0cedf0 2024-03-24T00:39:57.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.432+0000 7f4080d9d640 1 -- 172.21.15.67:0/4125389994 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f407c0cc360 con 0x7f407c0cedf0 2024-03-24T00:39:57.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.432+0000 7f4063fff640 1 -- 172.21.15.67:0/4125389994 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f406c011b70 con 0x7f407c0cedf0 2024-03-24T00:39:57.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.432+0000 7f4080d9d640 1 -- 172.21.15.67:0/4125389994 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f407c1e7a90 con 0x7f407c0cedf0 2024-03-24T00:39:57.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.433+0000 7f4080d9d640 1 -- 172.21.15.67:0/4125389994 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f407c0751c0 con 0x7f407c0cedf0 2024-03-24T00:39:57.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.434+0000 7f4063fff640 1 -- 172.21.15.67:0/4125389994 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f406c011d10 con 0x7f407c0cedf0 2024-03-24T00:39:57.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.434+0000 7f4063fff640 1 --2- 172.21.15.67:0/4125389994 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f406806bbd0 0x7f406806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:57.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.434+0000 7f4063fff640 1 -- 172.21.15.67:0/4125389994 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f406c08a6e0 con 0x7f407c0cedf0 2024-03-24T00:39:57.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.434+0000 7f407affd640 1 --2- 172.21.15.67:0/4125389994 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f406806bbd0 0x7f406806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:57.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.435+0000 7f407affd640 1 --2- 172.21.15.67:0/4125389994 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f406806bbd0 0x7f406806e090 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4070006fd0 tx=0x7f4070008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:57.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.438+0000 7f4063fff640 1 -- 172.21.15.67:0/4125389994 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f406c059420 con 0x7f407c0cedf0 2024-03-24T00:39:57.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.689+0000 7f4080d9d640 1 -- 172.21.15.67:0/4125389994 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f407c15c1a0 con 0x7f407c0cedf0 2024-03-24T00:39:57.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.691+0000 7f4063fff640 1 -- 172.21.15.67:0/4125389994 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v12) v1 ==== 90+0+0 (secure 0 0 0) 0x7f406c058d20 con 0x7f407c0cedf0 2024-03-24T00:39:57.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.693+0000 7f4080d9d640 1 -- 172.21.15.67:0/4125389994 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f406806bbd0 msgr2=0x7f406806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:57.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.693+0000 7f4080d9d640 1 --2- 172.21.15.67:0/4125389994 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f406806bbd0 0x7f406806e090 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4070006fd0 tx=0x7f4070008040 comp rx=0 tx=0).stop 2024-03-24T00:39:57.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.693+0000 7f4080d9d640 1 -- 172.21.15.67:0/4125389994 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f407c0cedf0 msgr2=0x7f407c0c9bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:57.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.693+0000 7f4080d9d640 1 --2- 172.21.15.67:0/4125389994 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f407c0cedf0 0x7f407c0c9bc0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f406c00c430 tx=0x7f406c00c8f0 comp rx=0 tx=0).stop 2024-03-24T00:39:57.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.693+0000 7f4080d9d640 1 -- 172.21.15.67:0/4125389994 shutdown_connections 2024-03-24T00:39:57.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.693+0000 7f4080d9d640 1 --2- 172.21.15.67:0/4125389994 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f406806bbd0 0x7f406806e090 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:57.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.693+0000 7f4080d9d640 1 --2- 172.21.15.67:0/4125389994 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f407c161930 0x7f407c0cc810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:57.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.693+0000 7f4080d9d640 1 --2- 172.21.15.67:0/4125389994 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f407c15df60 0x7f407c0cb6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:57.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.693+0000 7f4080d9d640 1 --2- 172.21.15.67:0/4125389994 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f407c0cedf0 0x7f407c0c9bc0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:57.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.693+0000 7f4080d9d640 1 -- 172.21.15.67:0/4125389994 >> 172.21.15.67:0/4125389994 conn(0x7f407c072b60 msgr2=0x7f407c15c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:57.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.693+0000 7f4080d9d640 1 -- 172.21.15.67:0/4125389994 shutdown_connections 2024-03-24T00:39:57.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.693+0000 7f4080d9d640 1 -- 172.21.15.67:0/4125389994 wait complete. 2024-03-24T00:39:57.705 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:39:57.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.805+0000 7f17737c6640 1 Processor -- start 2024-03-24T00:39:57.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.806+0000 7f17737c6640 1 -- start start 2024-03-24T00:39:57.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.806+0000 7f17737c6640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f176c09a410 0x7f176c0cfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:57.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.806+0000 7f17737c6640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f176c0b43a0 0x7f176c0cc880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:57.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.806+0000 7f17737c6640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f176c0d0fc0 con 0x7f176c15e1b0 2024-03-24T00:39:57.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.806+0000 7f17737c6640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f176c15e590 con 0x7f176c09a410 2024-03-24T00:39:57.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.806+0000 7f17737c6640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f176c0d1820 con 0x7f176c0b43a0 2024-03-24T00:39:57.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.806+0000 7f1770d61640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f176c09a410 0x7f176c0cfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:57.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.806+0000 7f1771562640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f176c15e1b0 0x7f176c099d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49102/0 (socket says 172.21.15.67:49102) 2024-03-24T00:39:57.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.806+0000 7f1771562640 1 -- 172.21.15.67:0/2660567468 learned_addr learned my addr 172.21.15.67:0/2660567468 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:57.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.807+0000 7f1771d63640 1 --2- 172.21.15.67:0/2660567468 >> v2:172.21.15.72:3300/0 conn(0x7f176c0b43a0 0x7f176c0cc880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:57.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.807+0000 7f1770d61640 1 -- 172.21.15.67:0/2660567468 >> v2:172.21.15.72:3300/0 conn(0x7f176c0b43a0 msgr2=0x7f176c0cc880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:57.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.807+0000 7f1770d61640 1 --2- 172.21.15.67:0/2660567468 >> v2:172.21.15.72:3300/0 conn(0x7f176c0b43a0 0x7f176c0cc880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:57.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.807+0000 7f1770d61640 1 -- 172.21.15.67:0/2660567468 >> v1:172.21.15.67:6789/0 conn(0x7f176c15e1b0 legacy=0x7f176c099d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:57.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.807+0000 7f1770d61640 1 -- 172.21.15.67:0/2660567468 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f176c0d1260 con 0x7f176c09a410 2024-03-24T00:39:57.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.807+0000 7f1763fff640 1 -- 172.21.15.67:0/2660567468 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2567093975 0 0) 0x7f176c0d0fc0 con 0x7f176c15e1b0 2024-03-24T00:39:57.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.808+0000 7f1770d61640 1 --2- 172.21.15.67:0/2660567468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f176c09a410 0x7f176c0cfe20 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f176c0b36a0 tx=0x7f175c02b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1faf7d81f641e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:57.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.808+0000 7f1763fff640 1 -- 172.21.15.67:0/2660567468 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f175c038070 con 0x7f176c09a410 2024-03-24T00:39:57.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.808+0000 7f1763fff640 1 -- 172.21.15.67:0/2660567468 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f175c031710 con 0x7f176c09a410 2024-03-24T00:39:57.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.808+0000 7f1763fff640 1 -- 172.21.15.67:0/2660567468 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f175c031ae0 con 0x7f176c09a410 2024-03-24T00:39:57.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.808+0000 7f17737c6640 1 -- 172.21.15.67:0/2660567468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f176c09a410 msgr2=0x7f176c0cfe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:57.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.808+0000 7f17737c6640 1 --2- 172.21.15.67:0/2660567468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f176c09a410 0x7f176c0cfe20 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f176c0b36a0 tx=0x7f175c02b7b0 comp rx=0 tx=0).stop 2024-03-24T00:39:57.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.808+0000 7f17737c6640 1 -- 172.21.15.67:0/2660567468 shutdown_connections 2024-03-24T00:39:57.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.808+0000 7f17737c6640 1 --2- 172.21.15.67:0/2660567468 >> v2:172.21.15.72:3300/0 conn(0x7f176c0b43a0 0x7f176c0cc880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:57.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.808+0000 7f17737c6640 1 --2- 172.21.15.67:0/2660567468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f176c09a410 0x7f176c0cfe20 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:57.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.808+0000 7f17737c6640 1 -- 172.21.15.67:0/2660567468 >> 172.21.15.67:0/2660567468 conn(0x7f176c0b36d0 msgr2=0x7f176c0b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:57.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.808+0000 7f17737c6640 1 -- 172.21.15.67:0/2660567468 shutdown_connections 2024-03-24T00:39:57.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.808+0000 7f17737c6640 1 -- 172.21.15.67:0/2660567468 wait complete. 2024-03-24T00:39:57.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.809+0000 7f17737c6640 1 Processor -- start 2024-03-24T00:39:57.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.809+0000 7f17737c6640 1 -- start start 2024-03-24T00:39:57.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.810+0000 7f17737c6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f176c09a410 0x7f176c1e57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:57.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.810+0000 7f17737c6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f176c0b43a0 0x7f176c158630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:57.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.810+0000 7f17737c6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f176c15e1b0 0x7f176c158da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:57.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.810+0000 7f17737c6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f176c09a9a0 con 0x7f176c0b43a0 2024-03-24T00:39:57.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.810+0000 7f17737c6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f176c0cc360 con 0x7f176c09a410 2024-03-24T00:39:57.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.810+0000 7f17737c6640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f176c0ccdc0 con 0x7f176c15e1b0 2024-03-24T00:39:57.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.810+0000 7f1770d61640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f176c0b43a0 0x7f176c158630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:57.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.810+0000 7f1770d61640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f176c0b43a0 0x7f176c158630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39748/0 (socket says 172.21.15.67:39748) 2024-03-24T00:39:57.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.810+0000 7f1770d61640 1 -- 172.21.15.67:0/2842646735 learned_addr learned my addr 172.21.15.67:0/2842646735 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:57.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.810+0000 7f1770d61640 1 -- 172.21.15.67:0/2842646735 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f176c15e1b0 msgr2=0x7f176c158da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:57.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.810+0000 7f1770d61640 1 --2- 172.21.15.67:0/2842646735 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f176c15e1b0 0x7f176c158da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:57.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.810+0000 7f1770d61640 1 -- 172.21.15.67:0/2842646735 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f176c09a410 msgr2=0x7f176c1e57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:57.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.810+0000 7f1770d61640 1 --2- 172.21.15.67:0/2842646735 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f176c09a410 0x7f176c1e57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:57.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.810+0000 7f1770d61640 1 -- 172.21.15.67:0/2842646735 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f176c0c9a40 con 0x7f176c0b43a0 2024-03-24T00:39:57.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.811+0000 7f1770d61640 1 --2- 172.21.15.67:0/2842646735 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f176c0b43a0 0x7f176c158630 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f175c000c00 tx=0x7f175c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:57.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.811+0000 7f1761ffb640 1 -- 172.21.15.67:0/2842646735 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f175c042070 con 0x7f176c0b43a0 2024-03-24T00:39:57.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.811+0000 7f1761ffb640 1 -- 172.21.15.67:0/2842646735 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f175c03d870 con 0x7f176c0b43a0 2024-03-24T00:39:57.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.811+0000 7f17737c6640 1 -- 172.21.15.67:0/2842646735 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f176c0c7500 con 0x7f176c0b43a0 2024-03-24T00:39:57.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.811+0000 7f1761ffb640 1 -- 172.21.15.67:0/2842646735 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f175c038040 con 0x7f176c0b43a0 2024-03-24T00:39:57.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.811+0000 7f17737c6640 1 -- 172.21.15.67:0/2842646735 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f176c0c7ac0 con 0x7f176c0b43a0 2024-03-24T00:39:57.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.812+0000 7f17737c6640 1 -- 172.21.15.67:0/2842646735 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f176c070cc0 con 0x7f176c0b43a0 2024-03-24T00:39:57.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.812+0000 7f1761ffb640 1 -- 172.21.15.67:0/2842646735 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f175c03b070 con 0x7f176c0b43a0 2024-03-24T00:39:57.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.813+0000 7f1761ffb640 1 --2- 172.21.15.67:0/2842646735 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f175006bbd0 0x7f175006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:57.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.813+0000 7f1761ffb640 1 -- 172.21.15.67:0/2842646735 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f175c0ac110 con 0x7f176c0b43a0 2024-03-24T00:39:57.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.813+0000 7f1771562640 1 --2- 172.21.15.67:0/2842646735 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f175006bbd0 0x7f175006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:57.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.814+0000 7f1771562640 1 --2- 172.21.15.67:0/2842646735 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f175006bbd0 0x7f175006e090 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1768007fd0 tx=0x7f1768007960 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:57.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:57.817+0000 7f1761ffb640 1 -- 172.21.15.67:0/2842646735 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f176c070cc0 con 0x7f176c0b43a0 2024-03-24T00:39:58.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.002+0000 7f17737c6640 1 -- 172.21.15.67:0/2842646735 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f176c04f400 con 0x7f176c0b43a0 2024-03-24T00:39:58.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.003+0000 7f1761ffb640 1 -- 172.21.15.67:0/2842646735 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) v1 ==== 74+0+13104 (secure 0 0 0) 0x7f175c07ae50 con 0x7f176c0b43a0 2024-03-24T00:39:58.004 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:39:58.004 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":35,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:39:55.004590+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:39:58.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.006+0000 7f17737c6640 1 -- 172.21.15.67:0/2842646735 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f175006bbd0 msgr2=0x7f175006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:58.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.006+0000 7f17737c6640 1 --2- 172.21.15.67:0/2842646735 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f175006bbd0 0x7f175006e090 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1768007fd0 tx=0x7f1768007960 comp rx=0 tx=0).stop 2024-03-24T00:39:58.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.006+0000 7f17737c6640 1 -- 172.21.15.67:0/2842646735 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f176c0b43a0 msgr2=0x7f176c158630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:58.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.006+0000 7f17737c6640 1 --2- 172.21.15.67:0/2842646735 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f176c0b43a0 0x7f176c158630 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f175c000c00 tx=0x7f175c0046c0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.006+0000 7f17737c6640 1 -- 172.21.15.67:0/2842646735 shutdown_connections 2024-03-24T00:39:58.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.006+0000 7f17737c6640 1 --2- 172.21.15.67:0/2842646735 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f175006bbd0 0x7f175006e090 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.006+0000 7f17737c6640 1 --2- 172.21.15.67:0/2842646735 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f176c15e1b0 0x7f176c158da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.006+0000 7f17737c6640 1 --2- 172.21.15.67:0/2842646735 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f176c0b43a0 0x7f176c158630 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.006+0000 7f17737c6640 1 --2- 172.21.15.67:0/2842646735 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f176c09a410 0x7f176c1e57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.006+0000 7f17737c6640 1 -- 172.21.15.67:0/2842646735 >> 172.21.15.67:0/2842646735 conn(0x7f176c0b36d0 msgr2=0x7f176c06dbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:58.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.006+0000 7f17737c6640 1 -- 172.21.15.67:0/2842646735 shutdown_connections 2024-03-24T00:39:58.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.006+0000 7f17737c6640 1 -- 172.21.15.67:0/2842646735 wait complete. 2024-03-24T00:39:58.018 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:38:39.364950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:39:58.018 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:39:58.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.122+0000 7f614b110640 1 Processor -- start 2024-03-24T00:39:58.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.122+0000 7f614b110640 1 -- start start 2024-03-24T00:39:58.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.123+0000 7f614b110640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f614415a500 0x7f614406f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:58.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.123+0000 7f614b110640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f614406f570 0x7f6144072380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:58.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.123+0000 7f614b110640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f61440728c0 0x7f614406db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:58.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.123+0000 7f614b110640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6144072d30 con 0x7f614415a500 2024-03-24T00:39:58.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.123+0000 7f614b110640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61440b2b40 con 0x7f614406f570 2024-03-24T00:39:58.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.123+0000 7f614b110640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61440b2e20 con 0x7f61440728c0 2024-03-24T00:39:58.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.123+0000 7f6148eac640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f614415a500 0x7f614406f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:58.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.123+0000 7f6148eac640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f614415a500 0x7f614406f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39764/0 (socket says 172.21.15.67:39764) 2024-03-24T00:39:58.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.124+0000 7f6148eac640 1 -- 172.21.15.67:0/2489749339 learned_addr learned my addr 172.21.15.67:0/2489749339 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:58.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.124+0000 7f6143fff640 1 --2- 172.21.15.67:0/2489749339 >> v2:172.21.15.72:3300/0 conn(0x7f614406f570 0x7f6144072380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:58.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.124+0000 7f61496ad640 1 --2- 172.21.15.67:0/2489749339 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f61440728c0 0x7f614406db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:58.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.124+0000 7f6148eac640 1 -- 172.21.15.67:0/2489749339 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f61440728c0 msgr2=0x7f614406db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:58.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.124+0000 7f6148eac640 1 --2- 172.21.15.67:0/2489749339 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f61440728c0 0x7f614406db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.124+0000 7f6148eac640 1 -- 172.21.15.67:0/2489749339 >> v2:172.21.15.72:3300/0 conn(0x7f614406f570 msgr2=0x7f6144072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:58.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.124+0000 7f6148eac640 1 --2- 172.21.15.67:0/2489749339 >> v2:172.21.15.72:3300/0 conn(0x7f614406f570 0x7f6144072380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.124+0000 7f6148eac640 1 -- 172.21.15.67:0/2489749339 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61440b3100 con 0x7f614415a500 2024-03-24T00:39:58.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.124+0000 7f61496ad640 1 --2- 172.21.15.67:0/2489749339 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f61440728c0 0x7f614406db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:58.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.125+0000 7f6148eac640 1 --2- 172.21.15.67:0/2489749339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f614415a500 0x7f614406f030 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f6144071500 tx=0x7f613802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8974717dd1be5f50 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:58.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.125+0000 7f61437fe640 1 -- 172.21.15.67:0/2489749339 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613802e070 con 0x7f614415a500 2024-03-24T00:39:58.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.125+0000 7f61437fe640 1 -- 172.21.15.67:0/2489749339 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6138037070 con 0x7f614415a500 2024-03-24T00:39:58.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.125+0000 7f61437fe640 1 -- 172.21.15.67:0/2489749339 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6138031a00 con 0x7f614415a500 2024-03-24T00:39:58.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.125+0000 7f614b110640 1 -- 172.21.15.67:0/2489749339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f614415a500 msgr2=0x7f614406f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:58.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.125+0000 7f614b110640 1 --2- 172.21.15.67:0/2489749339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f614415a500 0x7f614406f030 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f6144071500 tx=0x7f613802b880 comp rx=0 tx=0).stop 2024-03-24T00:39:58.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.125+0000 7f614b110640 1 -- 172.21.15.67:0/2489749339 shutdown_connections 2024-03-24T00:39:58.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.125+0000 7f614b110640 1 --2- 172.21.15.67:0/2489749339 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f61440728c0 0x7f614406db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.125+0000 7f614b110640 1 --2- 172.21.15.67:0/2489749339 >> v2:172.21.15.72:3300/0 conn(0x7f614406f570 0x7f6144072380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.125+0000 7f614b110640 1 --2- 172.21.15.67:0/2489749339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f614415a500 0x7f614406f030 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.125+0000 7f614b110640 1 -- 172.21.15.67:0/2489749339 >> 172.21.15.67:0/2489749339 conn(0x7f61440cf9e0 msgr2=0x7f61440cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:58.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.125+0000 7f614b110640 1 -- 172.21.15.67:0/2489749339 shutdown_connections 2024-03-24T00:39:58.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.125+0000 7f614b110640 1 -- 172.21.15.67:0/2489749339 wait complete. 2024-03-24T00:39:58.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.127+0000 7f614b110640 1 Processor -- start 2024-03-24T00:39:58.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.127+0000 7f614b110640 1 -- start start 2024-03-24T00:39:58.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.127+0000 7f614b110640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f614406f570 0x7f61441495c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:58.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.127+0000 7f614b110640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61440728c0 0x7f614414be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:58.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.127+0000 7f614b110640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f614415a500 0x7f614415b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:58.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.127+0000 7f614b110640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f614415a8e0 con 0x7f61440728c0 2024-03-24T00:39:58.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.127+0000 7f614b110640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61440b36c0 con 0x7f614406f570 2024-03-24T00:39:58.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.127+0000 7f614b110640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61440d06f0 con 0x7f614415a500 2024-03-24T00:39:58.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.128+0000 7f6143fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61440728c0 0x7f614414be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:58.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.128+0000 7f6143fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61440728c0 0x7f614414be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39778/0 (socket says 172.21.15.67:39778) 2024-03-24T00:39:58.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.128+0000 7f6143fff640 1 -- 172.21.15.67:0/3130053449 learned_addr learned my addr 172.21.15.67:0/3130053449 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:58.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.128+0000 7f61496ad640 1 --2- 172.21.15.67:0/3130053449 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f614415a500 0x7f614415b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:58.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.128+0000 7f6148eac640 1 --2- 172.21.15.67:0/3130053449 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f614406f570 0x7f61441495c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:58.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.128+0000 7f6143fff640 1 -- 172.21.15.67:0/3130053449 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f614415a500 msgr2=0x7f614415b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:58.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.128+0000 7f6143fff640 1 --2- 172.21.15.67:0/3130053449 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f614415a500 0x7f614415b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.128+0000 7f6143fff640 1 -- 172.21.15.67:0/3130053449 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f614406f570 msgr2=0x7f61441495c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:58.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.128+0000 7f6143fff640 1 --2- 172.21.15.67:0/3130053449 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f614406f570 0x7f61441495c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.128+0000 7f6143fff640 1 -- 172.21.15.67:0/3130053449 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6144157b00 con 0x7f61440728c0 2024-03-24T00:39:58.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.128+0000 7f6143fff640 1 --2- 172.21.15.67:0/3130053449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61440728c0 0x7f614414be00 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f613000e870 tx=0x7f613000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:58.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.128+0000 7f61417fa640 1 -- 172.21.15.67:0/3130053449 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6130017070 con 0x7f61440728c0 2024-03-24T00:39:58.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.128+0000 7f61417fa640 1 -- 172.21.15.67:0/3130053449 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f613000ee60 con 0x7f61440728c0 2024-03-24T00:39:58.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.128+0000 7f614b110640 1 -- 172.21.15.67:0/3130053449 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6144157de0 con 0x7f61440728c0 2024-03-24T00:39:58.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.128+0000 7f61417fa640 1 -- 172.21.15.67:0/3130053449 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613000cb80 con 0x7f61440728c0 2024-03-24T00:39:58.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.128+0000 7f614b110640 1 -- 172.21.15.67:0/3130053449 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61441583a0 con 0x7f61440728c0 2024-03-24T00:39:58.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.129+0000 7f614b110640 1 -- 172.21.15.67:0/3130053449 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f614404f400 con 0x7f61440728c0 2024-03-24T00:39:58.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.130+0000 7f61417fa640 1 -- 172.21.15.67:0/3130053449 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f613001a070 con 0x7f61440728c0 2024-03-24T00:39:58.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.130+0000 7f61417fa640 1 --2- 172.21.15.67:0/3130053449 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f612806bbd0 0x7f612806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:58.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.130+0000 7f61417fa640 1 -- 172.21.15.67:0/3130053449 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f613008a600 con 0x7f61440728c0 2024-03-24T00:39:58.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.130+0000 7f6148eac640 1 --2- 172.21.15.67:0/3130053449 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f612806bbd0 0x7f612806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:58.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.131+0000 7f6148eac640 1 --2- 172.21.15.67:0/3130053449 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f612806bbd0 0x7f612806e090 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f6138036000 tx=0x7f613803d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:58.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.135+0000 7f61417fa640 1 -- 172.21.15.67:0/3130053449 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6130058cf0 con 0x7f61440728c0 2024-03-24T00:39:58.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.316+0000 7f614b110640 1 -- 172.21.15.67:0/3130053449 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6144151a70 con 0x7f61440728c0 2024-03-24T00:39:58.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.317+0000 7f61417fa640 1 -- 172.21.15.67:0/3130053449 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v35) v1 ==== 93+0+10 (secure 0 0 0) 0x7f613005cd90 con 0x7f61440728c0 2024-03-24T00:39:58.317 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-24T00:39:58.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.319+0000 7f614b110640 1 -- 172.21.15.67:0/3130053449 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f612806bbd0 msgr2=0x7f612806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:58.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.319+0000 7f614b110640 1 --2- 172.21.15.67:0/3130053449 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f612806bbd0 0x7f612806e090 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f6138036000 tx=0x7f613803d040 comp rx=0 tx=0).stop 2024-03-24T00:39:58.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.319+0000 7f614b110640 1 -- 172.21.15.67:0/3130053449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61440728c0 msgr2=0x7f614414be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:58.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.319+0000 7f614b110640 1 --2- 172.21.15.67:0/3130053449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61440728c0 0x7f614414be00 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f613000e870 tx=0x7f613000ed30 comp rx=0 tx=0).stop 2024-03-24T00:39:58.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.319+0000 7f614b110640 1 -- 172.21.15.67:0/3130053449 shutdown_connections 2024-03-24T00:39:58.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.319+0000 7f614b110640 1 --2- 172.21.15.67:0/3130053449 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f612806bbd0 0x7f612806e090 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.319+0000 7f614b110640 1 --2- 172.21.15.67:0/3130053449 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f614415a500 0x7f614415b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.319+0000 7f614b110640 1 --2- 172.21.15.67:0/3130053449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61440728c0 0x7f614414be00 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.319+0000 7f614b110640 1 --2- 172.21.15.67:0/3130053449 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f614406f570 0x7f61441495c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.319+0000 7f614b110640 1 -- 172.21.15.67:0/3130053449 >> 172.21.15.67:0/3130053449 conn(0x7f61440cf9e0 msgr2=0x7f61440b0550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:58.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.319+0000 7f614b110640 1 -- 172.21.15.67:0/3130053449 shutdown_connections 2024-03-24T00:39:58.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.319+0000 7f614b110640 1 -- 172.21.15.67:0/3130053449 wait complete. 2024-03-24T00:39:58.331 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-24T00:39:58.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.434+0000 7f28632cc640 1 Processor -- start 2024-03-24T00:39:58.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.434+0000 7f28632cc640 1 -- start start 2024-03-24T00:39:58.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.435+0000 7f28632cc640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f285c15dec0 0x7f285c0ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:58.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.435+0000 7f28632cc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f285c15e2a0 0x7f285c162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:58.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.435+0000 7f28632cc640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f285c0cf640 con 0x7f285c0d07e0 2024-03-24T00:39:58.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.435+0000 7f28632cc640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f285c0cee30 con 0x7f285c15e2a0 2024-03-24T00:39:58.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.435+0000 7f28632cc640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f285c0d2120 con 0x7f285c15dec0 2024-03-24T00:39:58.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.435+0000 7f2861869640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f285c0d07e0 0x7f285c0b5850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49110/0 (socket says 172.21.15.67:49110) 2024-03-24T00:39:58.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.435+0000 7f2861869640 1 -- 172.21.15.67:0/452594036 learned_addr learned my addr 172.21.15.67:0/452594036 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:58.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.435+0000 7f2860867640 1 --2- 172.21.15.67:0/452594036 >> v2:172.21.15.72:3300/0 conn(0x7f285c15e2a0 0x7f285c162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:58.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.435+0000 7f2861068640 1 --2- 172.21.15.67:0/452594036 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f285c15dec0 0x7f285c0ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:58.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.435+0000 7f2853fff640 1 -- 172.21.15.67:0/452594036 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 874713330 0 0) 0x7f2854001490 con 0x7f285c0d07e0 2024-03-24T00:39:58.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.435+0000 7f2853fff640 1 -- 172.21.15.67:0/452594036 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2848003520 con 0x7f285c0d07e0 2024-03-24T00:39:58.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.435+0000 7f2860867640 1 -- 172.21.15.67:0/452594036 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f285c15dec0 msgr2=0x7f285c0ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:58.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.435+0000 7f2860867640 1 --2- 172.21.15.67:0/452594036 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f285c15dec0 0x7f285c0ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.435+0000 7f2860867640 1 -- 172.21.15.67:0/452594036 >> v1:172.21.15.67:6789/0 conn(0x7f285c0d07e0 legacy=0x7f285c0b5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:58.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.435+0000 7f2860867640 1 -- 172.21.15.67:0/452594036 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f285c0cf080 con 0x7f285c15e2a0 2024-03-24T00:39:58.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.435+0000 7f2861068640 1 --2- 172.21.15.67:0/452594036 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f285c15dec0 0x7f285c0ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:58.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.436+0000 7f2860867640 1 --2- 172.21.15.67:0/452594036 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f285c15e2a0 0x7f285c162630 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f285c0cefb0 tx=0x7f284c02b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71c75f3fe9610574 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:58.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.436+0000 7f2853fff640 1 -- 172.21.15.67:0/452594036 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f284c031770 con 0x7f285c15e2a0 2024-03-24T00:39:58.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.436+0000 7f2853fff640 1 -- 172.21.15.67:0/452594036 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f284c037070 con 0x7f285c15e2a0 2024-03-24T00:39:58.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.437+0000 7f2853fff640 1 -- 172.21.15.67:0/452594036 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f284c031b30 con 0x7f285c15e2a0 2024-03-24T00:39:58.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.437+0000 7f28632cc640 1 -- 172.21.15.67:0/452594036 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f285c15e2a0 msgr2=0x7f285c162630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:58.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.437+0000 7f28632cc640 1 --2- 172.21.15.67:0/452594036 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f285c15e2a0 0x7f285c162630 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f285c0cefb0 tx=0x7f284c02b810 comp rx=0 tx=0).stop 2024-03-24T00:39:58.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.437+0000 7f28632cc640 1 -- 172.21.15.67:0/452594036 shutdown_connections 2024-03-24T00:39:58.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.437+0000 7f28632cc640 1 --2- 172.21.15.67:0/452594036 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f285c15e2a0 0x7f285c162630 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.437+0000 7f28632cc640 1 --2- 172.21.15.67:0/452594036 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f285c15dec0 0x7f285c0ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.437+0000 7f28632cc640 1 -- 172.21.15.67:0/452594036 >> 172.21.15.67:0/452594036 conn(0x7f285c06c810 msgr2=0x7f285c06cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:58.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.437+0000 7f28632cc640 1 -- 172.21.15.67:0/452594036 shutdown_connections 2024-03-24T00:39:58.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.437+0000 7f28632cc640 1 -- 172.21.15.67:0/452594036 wait complete. 2024-03-24T00:39:58.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.438+0000 7f28632cc640 1 Processor -- start 2024-03-24T00:39:58.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.438+0000 7f28632cc640 1 -- start start 2024-03-24T00:39:58.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.439+0000 7f28632cc640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f285c0d07e0 0x7f285c150330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:58.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.439+0000 7f28632cc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f285c15dec0 0x7f285c0cbf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:58.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.439+0000 7f28632cc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f285c15e2a0 0x7f285c0aa720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:58.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.439+0000 7f28632cc640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f285c0d3740 con 0x7f285c15e2a0 2024-03-24T00:39:58.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.439+0000 7f28632cc640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f285c0d35c0 con 0x7f285c15dec0 2024-03-24T00:39:58.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.439+0000 7f28632cc640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f285c0d38c0 con 0x7f285c0d07e0 2024-03-24T00:39:58.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.439+0000 7f2861869640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f285c15e2a0 0x7f285c0aa720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:58.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.439+0000 7f2861869640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f285c15e2a0 0x7f285c0aa720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39784/0 (socket says 172.21.15.67:39784) 2024-03-24T00:39:58.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.439+0000 7f2861869640 1 -- 172.21.15.67:0/1802334167 learned_addr learned my addr 172.21.15.67:0/1802334167 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:58.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.439+0000 7f2860867640 1 --2- 172.21.15.67:0/1802334167 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f285c15dec0 0x7f285c0cbf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:58.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.439+0000 7f2861068640 1 --2- 172.21.15.67:0/1802334167 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f285c0d07e0 0x7f285c150330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:58.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.440+0000 7f2860867640 1 -- 172.21.15.67:0/1802334167 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f285c0d07e0 msgr2=0x7f285c150330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:58.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.440+0000 7f2860867640 1 --2- 172.21.15.67:0/1802334167 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f285c0d07e0 0x7f285c150330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.440+0000 7f2860867640 1 -- 172.21.15.67:0/1802334167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f285c15e2a0 msgr2=0x7f285c0aa720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:58.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.440+0000 7f2860867640 1 --2- 172.21.15.67:0/1802334167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f285c15e2a0 0x7f285c0aa720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.440+0000 7f2860867640 1 -- 172.21.15.67:0/1802334167 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f285c150870 con 0x7f285c15dec0 2024-03-24T00:39:58.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.440+0000 7f2861869640 1 --2- 172.21.15.67:0/1802334167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f285c15e2a0 0x7f285c0aa720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:58.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.440+0000 7f2861068640 1 --2- 172.21.15.67:0/1802334167 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f285c0d07e0 0x7f285c150330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:58.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.440+0000 7f2860867640 1 --2- 172.21.15.67:0/1802334167 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f285c15dec0 0x7f285c0cbf40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f284c033000 tx=0x7f284c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:58.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.440+0000 7f2851ffb640 1 -- 172.21.15.67:0/1802334167 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f284c045600 con 0x7f285c15dec0 2024-03-24T00:39:58.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.440+0000 7f2851ffb640 1 -- 172.21.15.67:0/1802334167 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f284c0457a0 con 0x7f285c15dec0 2024-03-24T00:39:58.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.440+0000 7f2851ffb640 1 -- 172.21.15.67:0/1802334167 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f284c045b70 con 0x7f285c15dec0 2024-03-24T00:39:58.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.440+0000 7f28632cc640 1 -- 172.21.15.67:0/1802334167 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f285c0b4430 con 0x7f285c15dec0 2024-03-24T00:39:58.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.441+0000 7f28632cc640 1 -- 172.21.15.67:0/1802334167 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f285c0b49f0 con 0x7f285c15dec0 2024-03-24T00:39:58.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.441+0000 7f28632cc640 1 -- 172.21.15.67:0/1802334167 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f285c04f400 con 0x7f285c15dec0 2024-03-24T00:39:58.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.442+0000 7f2851ffb640 1 -- 172.21.15.67:0/1802334167 <== mon.1 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f284c045d10 con 0x7f285c15dec0 2024-03-24T00:39:58.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.442+0000 7f2851ffb640 1 --2- 172.21.15.67:0/1802334167 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f284006bc00 0x7f284006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:58.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.442+0000 7f2851ffb640 1 -- 172.21.15.67:0/1802334167 <== mon.1 v2:172.21.15.72:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f284c0ac480 con 0x7f285c15dec0 2024-03-24T00:39:58.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.443+0000 7f2861068640 1 --2- 172.21.15.67:0/1802334167 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f284006bc00 0x7f284006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:58.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.443+0000 7f2861068640 1 --2- 172.21.15.67:0/1802334167 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f284006bc00 0x7f284006e0c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2858009560 tx=0x7f2858009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:58.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.448+0000 7f2851ffb640 1 -- 172.21.15.67:0/1802334167 <== mon.1 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f284c040070 con 0x7f285c15dec0 2024-03-24T00:39:58.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.634+0000 7f28632cc640 1 -- 172.21.15.67:0/1802334167 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f285c0d1aa0 con 0x7f285c15dec0 2024-03-24T00:39:58.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.635+0000 7f2851ffb640 1 -- 172.21.15.67:0/1802334167 <== mon.1 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v35) v1 ==== 84+0+65 (secure 0 0 0) 0x7f284c07ab60 con 0x7f285c15dec0 2024-03-24T00:39:58.636 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:39:58.636 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-03-24T00:39:58.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:39:58.636 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2024-03-24T00:39:58.636 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2024-03-24T00:39:58.636 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:39:58.636 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-03-24T00:39:58.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.637+0000 7f28632cc640 1 -- 172.21.15.67:0/1802334167 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f284006bc00 msgr2=0x7f284006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:58.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.637+0000 7f28632cc640 1 --2- 172.21.15.67:0/1802334167 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f284006bc00 0x7f284006e0c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2858009560 tx=0x7f2858009190 comp rx=0 tx=0).stop 2024-03-24T00:39:58.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.637+0000 7f28632cc640 1 -- 172.21.15.67:0/1802334167 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f285c15dec0 msgr2=0x7f285c0cbf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:58.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.637+0000 7f28632cc640 1 --2- 172.21.15.67:0/1802334167 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f285c15dec0 0x7f285c0cbf40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f284c033000 tx=0x7f284c0046c0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.637+0000 7f28632cc640 1 -- 172.21.15.67:0/1802334167 shutdown_connections 2024-03-24T00:39:58.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.637+0000 7f28632cc640 1 --2- 172.21.15.67:0/1802334167 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f284006bc00 0x7f284006e0c0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.637+0000 7f28632cc640 1 --2- 172.21.15.67:0/1802334167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f285c15e2a0 0x7f285c0aa720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.637+0000 7f28632cc640 1 --2- 172.21.15.67:0/1802334167 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f285c15dec0 0x7f285c0cbf40 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.637+0000 7f28632cc640 1 --2- 172.21.15.67:0/1802334167 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f285c0d07e0 0x7f285c150330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.637+0000 7f28632cc640 1 -- 172.21.15.67:0/1802334167 >> 172.21.15.67:0/1802334167 conn(0x7f285c06c810 msgr2=0x7f285c0b8780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:58.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.637+0000 7f28632cc640 1 -- 172.21.15.67:0/1802334167 shutdown_connections 2024-03-24T00:39:58.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.637+0000 7f28632cc640 1 -- 172.21.15.67:0/1802334167 wait complete. 2024-03-24T00:39:58.648 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-24T00:39:58.649 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-24T00:39:58.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.755+0000 7f32c31f4640 1 Processor -- start 2024-03-24T00:39:58.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.755+0000 7f32c31f4640 1 -- start start 2024-03-24T00:39:58.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.756+0000 7f32c31f4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f32bc0ced60 0x7f32bc0d0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:58.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.756+0000 7f32c31f4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f32bc0d0b80 0x7f32bc0d2b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:58.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.756+0000 7f32c31f4640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32bc0cf1f0 con 0x7f32bc15dfc0 2024-03-24T00:39:58.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.756+0000 7f32c31f4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32bc15e3a0 con 0x7f32bc0d0b80 2024-03-24T00:39:58.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.756+0000 7f32c31f4640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32bc0cf4d0 con 0x7f32bc0ced60 2024-03-24T00:39:58.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.756+0000 7f32c0f90640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f32bc15dfc0 0x7f32bc1624c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49114/0 (socket says 172.21.15.67:49114) 2024-03-24T00:39:58.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.756+0000 7f32c0f90640 1 -- 172.21.15.67:0/2226778432 learned_addr learned my addr 172.21.15.67:0/2226778432 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:58.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.757+0000 7f32c1791640 1 --2- 172.21.15.67:0/2226778432 >> v2:172.21.15.72:3300/0 conn(0x7f32bc0d0b80 0x7f32bc0d2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:58.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.757+0000 7f32bbfff640 1 --2- 172.21.15.67:0/2226778432 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f32bc0ced60 0x7f32bc0d0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:58.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.757+0000 7f32bb7fe640 1 -- 172.21.15.67:0/2226778432 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4188585858 0 0) 0x7f32bc0cf1f0 con 0x7f32bc15dfc0 2024-03-24T00:39:58.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.757+0000 7f32bb7fe640 1 -- 172.21.15.67:0/2226778432 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32ac003520 con 0x7f32bc15dfc0 2024-03-24T00:39:58.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.757+0000 7f32c1791640 1 -- 172.21.15.67:0/2226778432 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f32bc0ced60 msgr2=0x7f32bc0d0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:58.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.757+0000 7f32c1791640 1 --2- 172.21.15.67:0/2226778432 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f32bc0ced60 0x7f32bc0d0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.757+0000 7f32c1791640 1 -- 172.21.15.67:0/2226778432 >> v1:172.21.15.67:6789/0 conn(0x7f32bc15dfc0 legacy=0x7f32bc1624c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:58.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.757+0000 7f32c1791640 1 -- 172.21.15.67:0/2226778432 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32bc0d2020 con 0x7f32bc0d0b80 2024-03-24T00:39:58.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.758+0000 7f32c1791640 1 --2- 172.21.15.67:0/2226778432 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32bc0d0b80 0x7f32bc0d2b00 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f32b4009d90 tx=0x7f32b402beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a0d95953b797c90 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:58.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.758+0000 7f32bb7fe640 1 -- 172.21.15.67:0/2226778432 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b4039070 con 0x7f32bc0d0b80 2024-03-24T00:39:58.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.758+0000 7f32bb7fe640 1 -- 172.21.15.67:0/2226778432 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32b402cc60 con 0x7f32bc0d0b80 2024-03-24T00:39:58.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.758+0000 7f32bb7fe640 1 -- 172.21.15.67:0/2226778432 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b4034660 con 0x7f32bc0d0b80 2024-03-24T00:39:58.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.758+0000 7f32c31f4640 1 -- 172.21.15.67:0/2226778432 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32bc0d0b80 msgr2=0x7f32bc0d2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:58.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.758+0000 7f32c31f4640 1 --2- 172.21.15.67:0/2226778432 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32bc0d0b80 0x7f32bc0d2b00 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f32b4009d90 tx=0x7f32b402beb0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.758+0000 7f32c31f4640 1 -- 172.21.15.67:0/2226778432 shutdown_connections 2024-03-24T00:39:58.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.758+0000 7f32c31f4640 1 --2- 172.21.15.67:0/2226778432 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32bc0d0b80 0x7f32bc0d2b00 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.758+0000 7f32c31f4640 1 --2- 172.21.15.67:0/2226778432 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f32bc0ced60 0x7f32bc0d0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.758+0000 7f32c31f4640 1 -- 172.21.15.67:0/2226778432 >> 172.21.15.67:0/2226778432 conn(0x7f32bc06c880 msgr2=0x7f32bc06cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:58.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.759+0000 7f32c31f4640 1 -- 172.21.15.67:0/2226778432 shutdown_connections 2024-03-24T00:39:58.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.759+0000 7f32c31f4640 1 -- 172.21.15.67:0/2226778432 wait complete. 2024-03-24T00:39:58.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.760+0000 7f32c31f4640 1 Processor -- start 2024-03-24T00:39:58.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.760+0000 7f32c31f4640 1 -- start start 2024-03-24T00:39:58.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.761+0000 7f32c31f4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32bc0d0b80 0x7f32bc1f6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:58.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.761+0000 7f32c31f4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f32bc15dfc0 0x7f32bc15ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:58.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.761+0000 7f32c31f4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32bc15b0a0 0x7f32bc0cd020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:58.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.761+0000 7f32c31f4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32bc0d0fd0 con 0x7f32bc0d0b80 2024-03-24T00:39:58.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.761+0000 7f32c31f4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32bc0b31f0 con 0x7f32bc15b0a0 2024-03-24T00:39:58.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.761+0000 7f32c31f4640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32bc0d25e0 con 0x7f32bc15dfc0 2024-03-24T00:39:58.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.761+0000 7f32c0f90640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32bc0d0b80 0x7f32bc1f6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:58.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.761+0000 7f32c0f90640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32bc0d0b80 0x7f32bc1f6a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39800/0 (socket says 172.21.15.67:39800) 2024-03-24T00:39:58.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.761+0000 7f32c0f90640 1 -- 172.21.15.67:0/2897038104 learned_addr learned my addr 172.21.15.67:0/2897038104 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:58.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.761+0000 7f32bbfff640 1 --2- 172.21.15.67:0/2897038104 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f32bc15dfc0 0x7f32bc15ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:58.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.761+0000 7f32c1791640 1 --2- 172.21.15.67:0/2897038104 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32bc15b0a0 0x7f32bc0cd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:58.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.761+0000 7f32c0f90640 1 -- 172.21.15.67:0/2897038104 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f32bc15dfc0 msgr2=0x7f32bc15ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:58.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.761+0000 7f32c0f90640 1 --2- 172.21.15.67:0/2897038104 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f32bc15dfc0 0x7f32bc15ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.761+0000 7f32c0f90640 1 -- 172.21.15.67:0/2897038104 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32bc15b0a0 msgr2=0x7f32bc0cd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:58.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.761+0000 7f32c0f90640 1 --2- 172.21.15.67:0/2897038104 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32bc15b0a0 0x7f32bc0cd020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:58.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.761+0000 7f32c0f90640 1 -- 172.21.15.67:0/2897038104 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32bc0ce3a0 con 0x7f32bc0d0b80 2024-03-24T00:39:58.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.761+0000 7f32c1791640 1 --2- 172.21.15.67:0/2897038104 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32bc15b0a0 0x7f32bc0cd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:58.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.761+0000 7f32c0f90640 1 --2- 172.21.15.67:0/2897038104 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32bc0d0b80 0x7f32bc1f6a80 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f32b000c3c0 tx=0x7f32b000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:58.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.761+0000 7f32b97fa640 1 -- 172.21.15.67:0/2897038104 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b0017070 con 0x7f32bc0d0b80 2024-03-24T00:39:58.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.761+0000 7f32b97fa640 1 -- 172.21.15.67:0/2897038104 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32b0005ce0 con 0x7f32bc0d0b80 2024-03-24T00:39:58.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.761+0000 7f32c31f4640 1 -- 172.21.15.67:0/2897038104 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32bc0cdde0 con 0x7f32bc0d0b80 2024-03-24T00:39:58.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.762+0000 7f32b97fa640 1 -- 172.21.15.67:0/2897038104 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b00072f0 con 0x7f32bc0d0b80 2024-03-24T00:39:58.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.762+0000 7f32c31f4640 1 -- 172.21.15.67:0/2897038104 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32bc14aa20 con 0x7f32bc0d0b80 2024-03-24T00:39:58.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.763+0000 7f32c31f4640 1 -- 172.21.15.67:0/2897038104 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32bc15b4b0 con 0x7f32bc0d0b80 2024-03-24T00:39:58.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.763+0000 7f32b97fa640 1 -- 172.21.15.67:0/2897038104 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f32b001a070 con 0x7f32bc0d0b80 2024-03-24T00:39:58.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.763+0000 7f32b97fa640 1 --2- 172.21.15.67:0/2897038104 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f32a006bbd0 0x7f32a006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:58.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.763+0000 7f32b97fa640 1 -- 172.21.15.67:0/2897038104 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f32b0089cd0 con 0x7f32bc0d0b80 2024-03-24T00:39:58.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.764+0000 7f32bbfff640 1 --2- 172.21.15.67:0/2897038104 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f32a006bbd0 0x7f32a006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:58.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.764+0000 7f32bbfff640 1 --2- 172.21.15.67:0/2897038104 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f32a006bbd0 0x7f32a006e090 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f32bc0d1580 tx=0x7f32a8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:58.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.772+0000 7f32b97fa640 1 -- 172.21.15.67:0/2897038104 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f32bc15b4b0 con 0x7f32bc0d0b80 2024-03-24T00:39:58.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:58.955+0000 7f32c31f4640 1 -- 172.21.15.67:0/2897038104 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f32bc04f400 con 0x7f32bc0d0b80 2024-03-24T00:39:59.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.314+0000 7f32b97fa640 1 -- 172.21.15.67:0/2897038104 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v36) v1 ==== 138+0+0 (secure 0 0 0) 0x7f32b0058900 con 0x7f32bc0d0b80 2024-03-24T00:39:59.315 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' created 2024-03-24T00:39:59.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.317+0000 7f32c31f4640 1 -- 172.21.15.67:0/2897038104 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f32a006bbd0 msgr2=0x7f32a006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:59.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.317+0000 7f32c31f4640 1 --2- 172.21.15.67:0/2897038104 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f32a006bbd0 0x7f32a006e090 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f32bc0d1580 tx=0x7f32a8008040 comp rx=0 tx=0).stop 2024-03-24T00:39:59.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.317+0000 7f32c31f4640 1 -- 172.21.15.67:0/2897038104 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32bc0d0b80 msgr2=0x7f32bc1f6a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:59.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.317+0000 7f32c31f4640 1 --2- 172.21.15.67:0/2897038104 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32bc0d0b80 0x7f32bc1f6a80 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f32b000c3c0 tx=0x7f32b000c880 comp rx=0 tx=0).stop 2024-03-24T00:39:59.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.318+0000 7f32c31f4640 1 -- 172.21.15.67:0/2897038104 shutdown_connections 2024-03-24T00:39:59.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.318+0000 7f32c31f4640 1 --2- 172.21.15.67:0/2897038104 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f32a006bbd0 0x7f32a006e090 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:59.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.318+0000 7f32c31f4640 1 --2- 172.21.15.67:0/2897038104 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32bc15b0a0 0x7f32bc0cd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:59.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.318+0000 7f32c31f4640 1 --2- 172.21.15.67:0/2897038104 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f32bc15dfc0 0x7f32bc15ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:59.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.318+0000 7f32c31f4640 1 --2- 172.21.15.67:0/2897038104 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32bc0d0b80 0x7f32bc1f6a80 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:59.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.318+0000 7f32c31f4640 1 -- 172.21.15.67:0/2897038104 >> 172.21.15.67:0/2897038104 conn(0x7f32bc06c880 msgr2=0x7f32bc0c7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:59.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.318+0000 7f32c31f4640 1 -- 172.21.15.67:0/2897038104 shutdown_connections 2024-03-24T00:39:59.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.319+0000 7f32c31f4640 1 -- 172.21.15.67:0/2897038104 wait complete. 2024-03-24T00:39:59.332 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-24T00:39:59.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.440+0000 7fa4a1619640 1 Processor -- start 2024-03-24T00:39:59.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.440+0000 7fa4a1619640 1 -- start start 2024-03-24T00:39:59.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.441+0000 7fa4a1619640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa49c0ceda0 0x7fa49c15ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:59.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.441+0000 7fa4a1619640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa49c0d1380 con 0x7fa49c15de50 2024-03-24T00:39:59.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.441+0000 7fa4a1619640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa49c161b30 con 0x7fa49c0d0bc0 2024-03-24T00:39:59.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.441+0000 7fa4a1619640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa49c161e10 con 0x7fa49c0ceda0 2024-03-24T00:39:59.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.441+0000 7fa49affd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa49c15de50 0x7fa49c0d04b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49126/0 (socket says 172.21.15.67:49126) 2024-03-24T00:39:59.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.441+0000 7fa49affd640 1 -- 172.21.15.67:0/1811734939 learned_addr learned my addr 172.21.15.67:0/1811734939 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:39:59.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.441+0000 7fa49a7fc640 1 --2- 172.21.15.67:0/1811734939 >> v2:172.21.15.72:3300/0 conn(0x7fa49c0ceda0 0x7fa49c15ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:59.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.441+0000 7fa499ffb640 1 -- 172.21.15.67:0/1811734939 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1823540160 0 0) 0x7fa49c0d1380 con 0x7fa49c15de50 2024-03-24T00:39:59.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.441+0000 7fa499ffb640 1 -- 172.21.15.67:0/1811734939 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa488003520 con 0x7fa49c15de50 2024-03-24T00:39:59.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.441+0000 7fa499ffb640 1 -- 172.21.15.67:0/1811734939 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 786763713 0 0) 0x7fa49c161b30 con 0x7fa49c0d0bc0 2024-03-24T00:39:59.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.441+0000 7fa499ffb640 1 -- 172.21.15.67:0/1811734939 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa49c0d1380 con 0x7fa49c0d0bc0 2024-03-24T00:39:59.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.441+0000 7fa49a7fc640 1 -- 172.21.15.67:0/1811734939 >> v1:172.21.15.72:6789/0 conn(0x7fa49c0d0bc0 legacy=0x7fa49c0b8f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:59.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.441+0000 7fa499ffb640 1 -- 172.21.15.67:0/1811734939 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3428403491 0 0) 0x7fa488003520 con 0x7fa49c15de50 2024-03-24T00:39:59.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.441+0000 7fa49a7fc640 1 -- 172.21.15.67:0/1811734939 >> v1:172.21.15.67:6789/0 conn(0x7fa49c15de50 legacy=0x7fa49c0d04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:59.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.442+0000 7fa49a7fc640 1 -- 172.21.15.67:0/1811734939 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa49c1620f0 con 0x7fa49c0ceda0 2024-03-24T00:39:59.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.442+0000 7fa49a7fc640 1 --2- 172.21.15.67:0/1811734939 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa49c0ceda0 0x7fa49c15ef40 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fa4840097a0 tx=0x7fa48402b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=187cee610218e5cf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:59.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.442+0000 7fa499ffb640 1 -- 172.21.15.67:0/1811734939 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa484038070 con 0x7fa49c0ceda0 2024-03-24T00:39:59.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.442+0000 7fa499ffb640 1 -- 172.21.15.67:0/1811734939 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa484031500 con 0x7fa49c0ceda0 2024-03-24T00:39:59.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.443+0000 7fa499ffb640 1 -- 172.21.15.67:0/1811734939 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa494001690 con 0x7fa49c0ceda0 2024-03-24T00:39:59.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.443+0000 7fa4a1619640 1 -- 172.21.15.67:0/1811734939 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa49c0ceda0 msgr2=0x7fa49c15ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:59.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.443+0000 7fa4a1619640 1 --2- 172.21.15.67:0/1811734939 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa49c0ceda0 0x7fa49c15ef40 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fa4840097a0 tx=0x7fa48402b820 comp rx=0 tx=0).stop 2024-03-24T00:39:59.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.443+0000 7fa4a1619640 1 -- 172.21.15.67:0/1811734939 shutdown_connections 2024-03-24T00:39:59.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.443+0000 7fa4a1619640 1 --2- 172.21.15.67:0/1811734939 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa49c0ceda0 0x7fa49c15ef40 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:59.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.443+0000 7fa4a1619640 1 -- 172.21.15.67:0/1811734939 >> 172.21.15.67:0/1811734939 conn(0x7fa49c06c8b0 msgr2=0x7fa49c06cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:59.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.443+0000 7fa4a1619640 1 -- 172.21.15.67:0/1811734939 shutdown_connections 2024-03-24T00:39:59.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.443+0000 7fa4a1619640 1 -- 172.21.15.67:0/1811734939 wait complete. 2024-03-24T00:39:59.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.444+0000 7fa4a1619640 1 Processor -- start 2024-03-24T00:39:59.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.444+0000 7fa4a1619640 1 -- start start 2024-03-24T00:39:59.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.445+0000 7fa4a1619640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa49c0ceda0 0x7fa49c1522e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:59.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.445+0000 7fa4a1619640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa49c0d0bc0 0x7fa49c0b5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:59.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.445+0000 7fa4a1619640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa49c15de50 0x7fa49c0b5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:59.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.445+0000 7fa4a1619640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa49c15e3b0 con 0x7fa49c0ceda0 2024-03-24T00:39:59.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.445+0000 7fa4a1619640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa49c15e230 con 0x7fa49c15de50 2024-03-24T00:39:59.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.445+0000 7fa4a1619640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa49c0d2ef0 con 0x7fa49c0d0bc0 2024-03-24T00:39:59.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.445+0000 7fa49affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa49c0ceda0 0x7fa49c1522e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:59.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.445+0000 7fa49affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa49c0ceda0 0x7fa49c1522e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39808/0 (socket says 172.21.15.67:39808) 2024-03-24T00:39:59.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.445+0000 7fa49affd640 1 -- 172.21.15.67:0/3250635925 learned_addr learned my addr 172.21.15.67:0/3250635925 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:39:59.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.445+0000 7fa49a7fc640 1 --2- 172.21.15.67:0/3250635925 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa49c0d0bc0 0x7fa49c0b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:59.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.445+0000 7fa49b7fe640 1 --2- 172.21.15.67:0/3250635925 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa49c15de50 0x7fa49c0b5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:59.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.445+0000 7fa49affd640 1 -- 172.21.15.67:0/3250635925 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa49c0d0bc0 msgr2=0x7fa49c0b5110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:59.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.445+0000 7fa49affd640 1 --2- 172.21.15.67:0/3250635925 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa49c0d0bc0 0x7fa49c0b5110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:59.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.445+0000 7fa49affd640 1 -- 172.21.15.67:0/3250635925 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa49c15de50 msgr2=0x7fa49c0b5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:59.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.445+0000 7fa49affd640 1 --2- 172.21.15.67:0/3250635925 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa49c15de50 0x7fa49c0b5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:59.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.445+0000 7fa49affd640 1 -- 172.21.15.67:0/3250635925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa49c0cba40 con 0x7fa49c0ceda0 2024-03-24T00:39:59.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.446+0000 7fa49affd640 1 --2- 172.21.15.67:0/3250635925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa49c0ceda0 0x7fa49c1522e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fa49400c730 tx=0x7fa49400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:59.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.446+0000 7fa4937fe640 1 -- 172.21.15.67:0/3250635925 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa494018070 con 0x7fa49c0ceda0 2024-03-24T00:39:59.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.446+0000 7fa4937fe640 1 -- 172.21.15.67:0/3250635925 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4940072d0 con 0x7fa49c0ceda0 2024-03-24T00:39:59.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.446+0000 7fa4a1619640 1 -- 172.21.15.67:0/3250635925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa49c0cbd20 con 0x7fa49c0ceda0 2024-03-24T00:39:59.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.446+0000 7fa4937fe640 1 -- 172.21.15.67:0/3250635925 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4940047e0 con 0x7fa49c0ceda0 2024-03-24T00:39:59.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.446+0000 7fa4a1619640 1 -- 172.21.15.67:0/3250635925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa49c0cc2e0 con 0x7fa49c0ceda0 2024-03-24T00:39:59.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.447+0000 7fa4a1619640 1 -- 172.21.15.67:0/3250635925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa49c0cf4b0 con 0x7fa49c0ceda0 2024-03-24T00:39:59.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.447+0000 7fa4937fe640 1 -- 172.21.15.67:0/3250635925 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa49401b070 con 0x7fa49c0ceda0 2024-03-24T00:39:59.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.448+0000 7fa4937fe640 1 --2- 172.21.15.67:0/3250635925 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa47c06bbd0 0x7fa47c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:59.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.448+0000 7fa4937fe640 1 -- 172.21.15.67:0/3250635925 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5619+0+0 (secure 0 0 0) 0x7fa494089f30 con 0x7fa49c0ceda0 2024-03-24T00:39:59.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.448+0000 7fa49a7fc640 1 --2- 172.21.15.67:0/3250635925 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa47c06bbd0 0x7fa47c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:59.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.448+0000 7fa49a7fc640 1 --2- 172.21.15.67:0/3250635925 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa47c06bbd0 0x7fa47c06e090 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa484035000 tx=0x7fa484008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:59.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.452+0000 7fa4937fe640 1 -- 172.21.15.67:0/3250635925 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa494058a70 con 0x7fa49c0ceda0 2024-03-24T00:39:59.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:39:59.634+0000 7fa4a1619640 1 -- 172.21.15.67:0/3250635925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fa49c04f400 con 0x7fa49c0ceda0 2024-03-24T00:40:00.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.320+0000 7fa4937fe640 1 -- 172.21.15.67:0/3250635925 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v37) v1 ==== 170+0+0 (secure 0 0 0) 0x7fa494058370 con 0x7fa49c0ceda0 2024-03-24T00:40:00.322 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' created 2024-03-24T00:40:00.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.323+0000 7fa4a1619640 1 -- 172.21.15.67:0/3250635925 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa47c06bbd0 msgr2=0x7fa47c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:00.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.323+0000 7fa4a1619640 1 --2- 172.21.15.67:0/3250635925 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa47c06bbd0 0x7fa47c06e090 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa484035000 tx=0x7fa484008030 comp rx=0 tx=0).stop 2024-03-24T00:40:00.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.323+0000 7fa4a1619640 1 -- 172.21.15.67:0/3250635925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa49c0ceda0 msgr2=0x7fa49c1522e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:00.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.323+0000 7fa4a1619640 1 --2- 172.21.15.67:0/3250635925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa49c0ceda0 0x7fa49c1522e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fa49400c730 tx=0x7fa49400cbf0 comp rx=0 tx=0).stop 2024-03-24T00:40:00.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.323+0000 7fa4a1619640 1 -- 172.21.15.67:0/3250635925 shutdown_connections 2024-03-24T00:40:00.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.323+0000 7fa4a1619640 1 --2- 172.21.15.67:0/3250635925 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa47c06bbd0 0x7fa47c06e090 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:00.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.323+0000 7fa4a1619640 1 --2- 172.21.15.67:0/3250635925 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa49c15de50 0x7fa49c0b5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:00.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.323+0000 7fa4a1619640 1 --2- 172.21.15.67:0/3250635925 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa49c0d0bc0 0x7fa49c0b5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:00.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.323+0000 7fa4a1619640 1 --2- 172.21.15.67:0/3250635925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa49c0ceda0 0x7fa49c1522e0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:00.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.323+0000 7fa4a1619640 1 -- 172.21.15.67:0/3250635925 >> 172.21.15.67:0/3250635925 conn(0x7fa49c06c8b0 msgr2=0x7fa49c15e530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:00.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.323+0000 7fa4a1619640 1 -- 172.21.15.67:0/3250635925 shutdown_connections 2024-03-24T00:40:00.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.323+0000 7fa4a1619640 1 -- 172.21.15.67:0/3250635925 wait complete. 2024-03-24T00:40:00.344 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-24T00:40:00.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.454+0000 7fa17ed77640 1 Processor -- start 2024-03-24T00:40:00.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.454+0000 7fa17ed77640 1 -- start start 2024-03-24T00:40:00.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.455+0000 7fa17ed77640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa1780b9690 0x7fa1780ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:00.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.455+0000 7fa17ed77640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa1780cef30 0x7fa1780d2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:00.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.455+0000 7fa17ed77640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa178162290 con 0x7fa178161370 2024-03-24T00:40:00.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.455+0000 7fa17ed77640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1780b6e10 con 0x7fa1780cef30 2024-03-24T00:40:00.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.455+0000 7fa17ed77640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1780d2190 con 0x7fa1780b9690 2024-03-24T00:40:00.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.455+0000 7fa17d314640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa1780cef30 0x7fa1780d2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:00.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.455+0000 7fa17d314640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa1780cef30 0x7fa1780d2620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39816/0 (socket says 172.21.15.67:39816) 2024-03-24T00:40:00.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.455+0000 7fa17d314640 1 -- 172.21.15.67:0/259623147 learned_addr learned my addr 172.21.15.67:0/259623147 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:00.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.455+0000 7fa177fff640 1 --2- 172.21.15.67:0/259623147 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa1780b9690 0x7fa1780ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:00.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.455+0000 7fa17d314640 1 -- 172.21.15.67:0/259623147 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa1780b9690 msgr2=0x7fa1780ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:00.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.455+0000 7fa17d314640 1 --2- 172.21.15.67:0/259623147 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa1780b9690 0x7fa1780ce9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:00.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.455+0000 7fa17d314640 1 -- 172.21.15.67:0/259623147 >> v1:172.21.15.72:6789/0 conn(0x7fa178161370 legacy=0x7fa178161ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:00.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.455+0000 7fa17d314640 1 -- 172.21.15.67:0/259623147 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa178162570 con 0x7fa1780cef30 2024-03-24T00:40:00.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.456+0000 7fa17d314640 1 --2- 172.21.15.67:0/259623147 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1780cef30 0x7fa1780d2620 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fa170009e80 tx=0x7fa17002bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5eec359e4fe9fe3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:00.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.456+0000 7fa1777fe640 1 -- 172.21.15.67:0/259623147 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa17002cb70 con 0x7fa1780cef30 2024-03-24T00:40:00.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.456+0000 7fa1777fe640 1 -- 172.21.15.67:0/259623147 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa17002cd10 con 0x7fa1780cef30 2024-03-24T00:40:00.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.456+0000 7fa1777fe640 1 -- 172.21.15.67:0/259623147 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa170034610 con 0x7fa1780cef30 2024-03-24T00:40:00.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.456+0000 7fa17ed77640 1 -- 172.21.15.67:0/259623147 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1780cef30 msgr2=0x7fa1780d2620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:00.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.456+0000 7fa17ed77640 1 --2- 172.21.15.67:0/259623147 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1780cef30 0x7fa1780d2620 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fa170009e80 tx=0x7fa17002bfd0 comp rx=0 tx=0).stop 2024-03-24T00:40:00.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.457+0000 7fa17ed77640 1 -- 172.21.15.67:0/259623147 shutdown_connections 2024-03-24T00:40:00.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.457+0000 7fa17ed77640 1 --2- 172.21.15.67:0/259623147 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa1780b9690 0x7fa1780ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:00.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.457+0000 7fa17ed77640 1 --2- 172.21.15.67:0/259623147 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1780cef30 0x7fa1780d2620 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:00.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.457+0000 7fa17ed77640 1 -- 172.21.15.67:0/259623147 >> 172.21.15.67:0/259623147 conn(0x7fa17806c880 msgr2=0x7fa17806cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:00.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.457+0000 7fa17ed77640 1 -- 172.21.15.67:0/259623147 shutdown_connections 2024-03-24T00:40:00.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.457+0000 7fa17ed77640 1 -- 172.21.15.67:0/259623147 wait complete. 2024-03-24T00:40:00.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.458+0000 7fa17ed77640 1 Processor -- start 2024-03-24T00:40:00.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.458+0000 7fa17ed77640 1 -- start start 2024-03-24T00:40:00.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.458+0000 7fa17ed77640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa1780b9690 0x7fa1781e9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:00.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.458+0000 7fa17ed77640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1780cef30 0x7fa178155eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:00.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.459+0000 7fa17ed77640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa178161370 0x7fa1780c69e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:00.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.459+0000 7fa17ed77640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1780d2b60 con 0x7fa1780cef30 2024-03-24T00:40:00.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.459+0000 7fa17ed77640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa178161750 con 0x7fa178161370 2024-03-24T00:40:00.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.459+0000 7fa17ed77640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1780d2ce0 con 0x7fa1780b9690 2024-03-24T00:40:00.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.459+0000 7fa177fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1780cef30 0x7fa178155eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:00.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.459+0000 7fa177fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1780cef30 0x7fa178155eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39832/0 (socket says 172.21.15.67:39832) 2024-03-24T00:40:00.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.459+0000 7fa177fff640 1 -- 172.21.15.67:0/1714201225 learned_addr learned my addr 172.21.15.67:0/1714201225 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:00.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.459+0000 7fa17d314640 1 --2- 172.21.15.67:0/1714201225 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa178161370 0x7fa1780c69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:00.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.459+0000 7fa17cb13640 1 --2- 172.21.15.67:0/1714201225 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa1780b9690 0x7fa1781e9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:00.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.459+0000 7fa177fff640 1 -- 172.21.15.67:0/1714201225 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa1780b9690 msgr2=0x7fa1781e9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:00.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.459+0000 7fa177fff640 1 --2- 172.21.15.67:0/1714201225 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa1780b9690 0x7fa1781e9cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:00.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.459+0000 7fa177fff640 1 -- 172.21.15.67:0/1714201225 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa178161370 msgr2=0x7fa1780c69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:00.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.459+0000 7fa177fff640 1 --2- 172.21.15.67:0/1714201225 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa178161370 0x7fa1780c69e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:00.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.459+0000 7fa177fff640 1 -- 172.21.15.67:0/1714201225 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1781563f0 con 0x7fa1780cef30 2024-03-24T00:40:00.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.459+0000 7fa177fff640 1 --2- 172.21.15.67:0/1714201225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1780cef30 0x7fa178155eb0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fa16400d800 tx=0x7fa16400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:00.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.459+0000 7fa1757fa640 1 -- 172.21.15.67:0/1714201225 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16400b660 con 0x7fa1780cef30 2024-03-24T00:40:00.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.459+0000 7fa1757fa640 1 -- 172.21.15.67:0/1714201225 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa164019070 con 0x7fa1780cef30 2024-03-24T00:40:00.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.459+0000 7fa17ed77640 1 -- 172.21.15.67:0/1714201225 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1781e8c30 con 0x7fa1780cef30 2024-03-24T00:40:00.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.459+0000 7fa1757fa640 1 -- 172.21.15.67:0/1714201225 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16400ba20 con 0x7fa1780cef30 2024-03-24T00:40:00.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.459+0000 7fa17ed77640 1 -- 172.21.15.67:0/1714201225 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1781e91f0 con 0x7fa1780cef30 2024-03-24T00:40:00.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.460+0000 7fa17ed77640 1 -- 172.21.15.67:0/1714201225 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa17804f400 con 0x7fa1780cef30 2024-03-24T00:40:00.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.461+0000 7fa1757fa640 1 -- 172.21.15.67:0/1714201225 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa16400bbc0 con 0x7fa1780cef30 2024-03-24T00:40:00.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.461+0000 7fa1757fa640 1 --2- 172.21.15.67:0/1714201225 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa15c06bbd0 0x7fa15c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:00.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.461+0000 7fa1757fa640 1 -- 172.21.15.67:0/1714201225 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6001+0+0 (secure 0 0 0) 0x7fa16408ae90 con 0x7fa1780cef30 2024-03-24T00:40:00.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.461+0000 7fa17cb13640 1 --2- 172.21.15.67:0/1714201225 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa15c06bbd0 0x7fa15c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:00.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.462+0000 7fa17cb13640 1 --2- 172.21.15.67:0/1714201225 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa15c06bbd0 0x7fa15c06e090 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa16c0016e0 tx=0x7fa16c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:00.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.466+0000 7fa1757fa640 1 -- 172.21.15.67:0/1714201225 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa1640591f0 con 0x7fa1780cef30 2024-03-24T00:40:00.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:00.689+0000 7fa17ed77640 1 -- 172.21.15.67:0/1714201225 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fa1780d1aa0 con 0x7fa1780cef30 2024-03-24T00:40:01.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.322+0000 7fa1757fa640 1 -- 172.21.15.67:0/1714201225 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 5 and data pool 6 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7fa16405f030 con 0x7fa1780cef30 2024-03-24T00:40:01.323 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 5 and data pool 6 2024-03-24T00:40:01.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.328+0000 7fa17ed77640 1 -- 172.21.15.67:0/1714201225 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa15c06bbd0 msgr2=0x7fa15c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:01.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.328+0000 7fa17ed77640 1 --2- 172.21.15.67:0/1714201225 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa15c06bbd0 0x7fa15c06e090 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa16c0016e0 tx=0x7fa16c009040 comp rx=0 tx=0).stop 2024-03-24T00:40:01.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.328+0000 7fa17ed77640 1 -- 172.21.15.67:0/1714201225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1780cef30 msgr2=0x7fa178155eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:01.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.328+0000 7fa17ed77640 1 --2- 172.21.15.67:0/1714201225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1780cef30 0x7fa178155eb0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fa16400d800 tx=0x7fa16400dcc0 comp rx=0 tx=0).stop 2024-03-24T00:40:01.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.329+0000 7fa17ed77640 1 -- 172.21.15.67:0/1714201225 shutdown_connections 2024-03-24T00:40:01.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.329+0000 7fa17ed77640 1 --2- 172.21.15.67:0/1714201225 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa15c06bbd0 0x7fa15c06e090 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:01.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.329+0000 7fa17ed77640 1 --2- 172.21.15.67:0/1714201225 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa178161370 0x7fa1780c69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:01.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.329+0000 7fa17ed77640 1 --2- 172.21.15.67:0/1714201225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1780cef30 0x7fa178155eb0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:01.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.329+0000 7fa17ed77640 1 --2- 172.21.15.67:0/1714201225 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa1780b9690 0x7fa1781e9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:01.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.329+0000 7fa17ed77640 1 -- 172.21.15.67:0/1714201225 >> 172.21.15.67:0/1714201225 conn(0x7fa17806c880 msgr2=0x7fa1780c88e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:01.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.329+0000 7fa17ed77640 1 -- 172.21.15.67:0/1714201225 shutdown_connections 2024-03-24T00:40:01.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.329+0000 7fa17ed77640 1 -- 172.21.15.67:0/1714201225 wait complete. 2024-03-24T00:40:01.355 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:40:01.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.461+0000 7f30a9c97640 1 Processor -- start 2024-03-24T00:40:01.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.461+0000 7f30a9c97640 1 -- start start 2024-03-24T00:40:01.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.462+0000 7f30a9c97640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f30a415bf70 0x7f30a40cc690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:01.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.462+0000 7f30a9c97640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f30a40cfc70 0x7f30a40ce220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:01.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.462+0000 7f30a9c97640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30a40ccfe0 con 0x7f30a40ce760 2024-03-24T00:40:01.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.462+0000 7f30a9c97640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30a40cd2c0 con 0x7f30a415bf70 2024-03-24T00:40:01.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.462+0000 7f30a9c97640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30a40cd5a0 con 0x7f30a40cfc70 2024-03-24T00:40:01.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.462+0000 7f30a3fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f30a40ce760 0x7f30a408b060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49138/0 (socket says 172.21.15.67:49138) 2024-03-24T00:40:01.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.462+0000 7f30a3fff640 1 -- 172.21.15.67:0/921617903 learned_addr learned my addr 172.21.15.67:0/921617903 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:40:01.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.462+0000 7f30a2ffd640 1 --2- 172.21.15.67:0/921617903 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f30a40cfc70 0x7f30a40ce220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:01.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.462+0000 7f30a37fe640 1 --2- 172.21.15.67:0/921617903 >> v2:172.21.15.72:3300/0 conn(0x7f30a415bf70 0x7f30a40cc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:01.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.463+0000 7f30a27fc640 1 -- 172.21.15.67:0/921617903 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1087863882 0 0) 0x7f30a40ccfe0 con 0x7f30a40ce760 2024-03-24T00:40:01.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.463+0000 7f30a27fc640 1 -- 172.21.15.67:0/921617903 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f308c003520 con 0x7f30a40ce760 2024-03-24T00:40:01.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.463+0000 7f30a37fe640 1 -- 172.21.15.67:0/921617903 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f30a40cfc70 msgr2=0x7f30a40ce220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:01.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.463+0000 7f30a37fe640 1 --2- 172.21.15.67:0/921617903 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f30a40cfc70 0x7f30a40ce220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:01.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.463+0000 7f30a37fe640 1 -- 172.21.15.67:0/921617903 >> v1:172.21.15.67:6789/0 conn(0x7f30a40ce760 legacy=0x7f30a408b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:01.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.463+0000 7f30a37fe640 1 -- 172.21.15.67:0/921617903 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a40cd880 con 0x7f30a415bf70 2024-03-24T00:40:01.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.464+0000 7f30a37fe640 1 --2- 172.21.15.67:0/921617903 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f30a415bf70 0x7f30a40cc690 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f3094009d90 tx=0x7f309402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cec8e66993fdfb98 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:01.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.464+0000 7f30a27fc640 1 -- 172.21.15.67:0/921617903 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3094039070 con 0x7f30a415bf70 2024-03-24T00:40:01.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.464+0000 7f30a27fc640 1 -- 172.21.15.67:0/921617903 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f309402cc50 con 0x7f30a415bf70 2024-03-24T00:40:01.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.464+0000 7f30a27fc640 1 -- 172.21.15.67:0/921617903 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3094034440 con 0x7f30a415bf70 2024-03-24T00:40:01.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.464+0000 7f30a9c97640 1 -- 172.21.15.67:0/921617903 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f30a415bf70 msgr2=0x7f30a40cc690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:01.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.464+0000 7f30a9c97640 1 --2- 172.21.15.67:0/921617903 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f30a415bf70 0x7f30a40cc690 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f3094009d90 tx=0x7f309402bea0 comp rx=0 tx=0).stop 2024-03-24T00:40:01.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.464+0000 7f30a9c97640 1 -- 172.21.15.67:0/921617903 shutdown_connections 2024-03-24T00:40:01.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.464+0000 7f30a9c97640 1 --2- 172.21.15.67:0/921617903 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f30a40cfc70 0x7f30a40ce220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:01.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.464+0000 7f30a9c97640 1 --2- 172.21.15.67:0/921617903 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f30a415bf70 0x7f30a40cc690 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:01.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.464+0000 7f30a9c97640 1 -- 172.21.15.67:0/921617903 >> 172.21.15.67:0/921617903 conn(0x7f30a40d1bf0 msgr2=0x7f30a40d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:01.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.464+0000 7f30a9c97640 1 -- 172.21.15.67:0/921617903 shutdown_connections 2024-03-24T00:40:01.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.464+0000 7f30a9c97640 1 -- 172.21.15.67:0/921617903 wait complete. 2024-03-24T00:40:01.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.465+0000 7f30a9c97640 1 Processor -- start 2024-03-24T00:40:01.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.466+0000 7f30a9c97640 1 -- start start 2024-03-24T00:40:01.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.466+0000 7f30a9c97640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f30a40ce760 0x7f30a4157e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:01.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.466+0000 7f30a9c97640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30a40cfc70 0x7f30a4070b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:01.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.466+0000 7f30a9c97640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f30a415bf70 0x7f30a4071060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:01.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.466+0000 7f30a9c97640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a40ccd50 con 0x7f30a40cfc70 2024-03-24T00:40:01.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.466+0000 7f30a9c97640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a40ccbd0 con 0x7f30a40ce760 2024-03-24T00:40:01.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.466+0000 7f30a9c97640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30a40b2ce0 con 0x7f30a415bf70 2024-03-24T00:40:01.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.466+0000 7f30a2ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30a40cfc70 0x7f30a4070b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:01.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.467+0000 7f30a2ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30a40cfc70 0x7f30a4070b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39834/0 (socket says 172.21.15.67:39834) 2024-03-24T00:40:01.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.467+0000 7f30a2ffd640 1 -- 172.21.15.67:0/1581789177 learned_addr learned my addr 172.21.15.67:0/1581789177 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:01.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.467+0000 7f30a37fe640 1 --2- 172.21.15.67:0/1581789177 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f30a40ce760 0x7f30a4157e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:01.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.467+0000 7f30a3fff640 1 --2- 172.21.15.67:0/1581789177 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f30a415bf70 0x7f30a4071060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:01.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.467+0000 7f30a2ffd640 1 -- 172.21.15.67:0/1581789177 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f30a415bf70 msgr2=0x7f30a4071060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:01.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.467+0000 7f30a2ffd640 1 --2- 172.21.15.67:0/1581789177 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f30a415bf70 0x7f30a4071060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:01.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.467+0000 7f30a2ffd640 1 -- 172.21.15.67:0/1581789177 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f30a40ce760 msgr2=0x7f30a4157e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:01.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.467+0000 7f30a2ffd640 1 --2- 172.21.15.67:0/1581789177 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f30a40ce760 0x7f30a4157e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:01.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.467+0000 7f30a2ffd640 1 -- 172.21.15.67:0/1581789177 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a406c2f0 con 0x7f30a40cfc70 2024-03-24T00:40:01.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.467+0000 7f30a3fff640 1 --2- 172.21.15.67:0/1581789177 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f30a415bf70 0x7f30a4071060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:40:01.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.467+0000 7f30a2ffd640 1 --2- 172.21.15.67:0/1581789177 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30a40cfc70 0x7f30a4070b20 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f309800e9b0 tx=0x7f309800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:01.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.467+0000 7f308bfff640 1 -- 172.21.15.67:0/1581789177 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3098017070 con 0x7f30a40cfc70 2024-03-24T00:40:01.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.467+0000 7f308bfff640 1 -- 172.21.15.67:0/1581789177 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f309800c9a0 con 0x7f30a40cfc70 2024-03-24T00:40:01.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.467+0000 7f30a9c97640 1 -- 172.21.15.67:0/1581789177 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30a406c5d0 con 0x7f30a40cfc70 2024-03-24T00:40:01.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.467+0000 7f308bfff640 1 -- 172.21.15.67:0/1581789177 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f309800cd90 con 0x7f30a40cfc70 2024-03-24T00:40:01.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.467+0000 7f30a9c97640 1 -- 172.21.15.67:0/1581789177 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30a406cb90 con 0x7f30a40cfc70 2024-03-24T00:40:01.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.468+0000 7f30a9c97640 1 -- 172.21.15.67:0/1581789177 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30a406cfd0 con 0x7f30a40cfc70 2024-03-24T00:40:01.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.469+0000 7f308bfff640 1 -- 172.21.15.67:0/1581789177 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f309801a070 con 0x7f30a40cfc70 2024-03-24T00:40:01.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.469+0000 7f308bfff640 1 --2- 172.21.15.67:0/1581789177 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f309006bbd0 0x7f309006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:01.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.469+0000 7f308bfff640 1 -- 172.21.15.67:0/1581789177 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f309808afe0 con 0x7f30a40cfc70 2024-03-24T00:40:01.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.470+0000 7f30a37fe640 1 --2- 172.21.15.67:0/1581789177 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f309006bbd0 0x7f309006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:01.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.470+0000 7f30a37fe640 1 --2- 172.21.15.67:0/1581789177 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f309006bbd0 0x7f309006e090 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3094036000 tx=0x7f3094030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:01.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.474+0000 7f308bfff640 1 -- 172.21.15.67:0/1581789177 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3098059270 con 0x7f30a40cfc70 2024-03-24T00:40:01.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.656+0000 7f30a9c97640 1 -- 172.21.15.67:0/1581789177 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f30a404f400 con 0x7f30a40cfc70 2024-03-24T00:40:01.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.657+0000 7f308bfff640 1 -- 172.21.15.67:0/1581789177 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+16895 (secure 0 0 0) 0x7f309805d310 con 0x7f30a40cfc70 2024-03-24T00:40:01.658 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:40:01.658 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":38,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:40:00.691111+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:39:58.975995+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-24T00:39:59.653615+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:40:01.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.659+0000 7f30a9c97640 1 -- 172.21.15.67:0/1581789177 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f309006bbd0 msgr2=0x7f309006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:01.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.659+0000 7f30a9c97640 1 --2- 172.21.15.67:0/1581789177 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f309006bbd0 0x7f309006e090 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3094036000 tx=0x7f3094030000 comp rx=0 tx=0).stop 2024-03-24T00:40:01.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.659+0000 7f30a9c97640 1 -- 172.21.15.67:0/1581789177 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30a40cfc70 msgr2=0x7f30a4070b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:01.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.659+0000 7f30a9c97640 1 --2- 172.21.15.67:0/1581789177 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30a40cfc70 0x7f30a4070b20 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f309800e9b0 tx=0x7f309800ee70 comp rx=0 tx=0).stop 2024-03-24T00:40:01.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.659+0000 7f30a9c97640 1 -- 172.21.15.67:0/1581789177 shutdown_connections 2024-03-24T00:40:01.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.659+0000 7f30a9c97640 1 --2- 172.21.15.67:0/1581789177 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f309006bbd0 0x7f309006e090 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:01.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.659+0000 7f30a9c97640 1 --2- 172.21.15.67:0/1581789177 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f30a415bf70 0x7f30a4071060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:01.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.659+0000 7f30a9c97640 1 --2- 172.21.15.67:0/1581789177 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30a40cfc70 0x7f30a4070b20 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:01.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.659+0000 7f30a9c97640 1 --2- 172.21.15.67:0/1581789177 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f30a40ce760 0x7f30a4157e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:01.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.659+0000 7f30a9c97640 1 -- 172.21.15.67:0/1581789177 >> 172.21.15.67:0/1581789177 conn(0x7f30a40d1bf0 msgr2=0x7f30a408b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:01.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.659+0000 7f30a9c97640 1 -- 172.21.15.67:0/1581789177 shutdown_connections 2024-03-24T00:40:01.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.659+0000 7f30a9c97640 1 -- 172.21.15.67:0/1581789177 wait complete. 2024-03-24T00:40:01.671 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:40:01.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.772+0000 7f43d31d3640 1 Processor -- start 2024-03-24T00:40:01.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.772+0000 7f43d31d3640 1 -- start start 2024-03-24T00:40:01.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.773+0000 7f43d31d3640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f43cc15e1b0 0x7f43cc0cfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:01.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.773+0000 7f43d31d3640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f43cc0b43a0 0x7f43cc099d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:01.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.773+0000 7f43d31d3640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f43cc09a240 0x7f43cc06c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:01.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.773+0000 7f43d31d3640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43cc15e590 con 0x7f43cc15e1b0 2024-03-24T00:40:01.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.773+0000 7f43d31d3640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43cc09a690 con 0x7f43cc0b43a0 2024-03-24T00:40:01.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.773+0000 7f43d31d3640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43cc0d1820 con 0x7f43cc09a240 2024-03-24T00:40:01.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.773+0000 7f43d0f6f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f43cc15e1b0 0x7f43cc0cfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:01.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.773+0000 7f43d0f6f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f43cc15e1b0 0x7f43cc0cfe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39838/0 (socket says 172.21.15.67:39838) 2024-03-24T00:40:01.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.773+0000 7f43d0f6f640 1 -- 172.21.15.67:0/1336855968 learned_addr learned my addr 172.21.15.67:0/1336855968 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:01.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.773+0000 7f43cbfff640 1 --2- 172.21.15.67:0/1336855968 >> v2:172.21.15.72:3300/0 conn(0x7f43cc0b43a0 0x7f43cc099d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:01.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.773+0000 7f43d1770640 1 --2- 172.21.15.67:0/1336855968 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f43cc09a240 0x7f43cc06c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:01.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.773+0000 7f43d0f6f640 1 -- 172.21.15.67:0/1336855968 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f43cc09a240 msgr2=0x7f43cc06c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:01.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.773+0000 7f43d0f6f640 1 --2- 172.21.15.67:0/1336855968 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f43cc09a240 0x7f43cc06c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:01.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.773+0000 7f43d0f6f640 1 -- 172.21.15.67:0/1336855968 >> v2:172.21.15.72:3300/0 conn(0x7f43cc0b43a0 msgr2=0x7f43cc099d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:01.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.773+0000 7f43d0f6f640 1 --2- 172.21.15.67:0/1336855968 >> v2:172.21.15.72:3300/0 conn(0x7f43cc0b43a0 0x7f43cc099d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:01.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.773+0000 7f43d0f6f640 1 -- 172.21.15.67:0/1336855968 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43cc0d0fc0 con 0x7f43cc15e1b0 2024-03-24T00:40:01.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.773+0000 7f43cbfff640 1 --2- 172.21.15.67:0/1336855968 >> v2:172.21.15.72:3300/0 conn(0x7f43cc0b43a0 0x7f43cc099d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:40:01.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.774+0000 7f43d0f6f640 1 --2- 172.21.15.67:0/1336855968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43cc15e1b0 0x7f43cc0cfe20 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f43cc0c8c30 tx=0x7f43c402bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=100f1207da95a070 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:01.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.774+0000 7f43cb7fe640 1 -- 172.21.15.67:0/1336855968 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c402c950 con 0x7f43cc15e1b0 2024-03-24T00:40:01.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.774+0000 7f43cb7fe640 1 -- 172.21.15.67:0/1336855968 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43c402caf0 con 0x7f43cc15e1b0 2024-03-24T00:40:01.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.774+0000 7f43cb7fe640 1 -- 172.21.15.67:0/1336855968 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c40343f0 con 0x7f43cc15e1b0 2024-03-24T00:40:01.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.774+0000 7f43d31d3640 1 -- 172.21.15.67:0/1336855968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43cc15e1b0 msgr2=0x7f43cc0cfe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:01.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.774+0000 7f43d31d3640 1 --2- 172.21.15.67:0/1336855968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43cc15e1b0 0x7f43cc0cfe20 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f43cc0c8c30 tx=0x7f43c402bdb0 comp rx=0 tx=0).stop 2024-03-24T00:40:01.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.775+0000 7f43d31d3640 1 -- 172.21.15.67:0/1336855968 shutdown_connections 2024-03-24T00:40:01.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.775+0000 7f43d31d3640 1 --2- 172.21.15.67:0/1336855968 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f43cc09a240 0x7f43cc06c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:01.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.775+0000 7f43d31d3640 1 --2- 172.21.15.67:0/1336855968 >> v2:172.21.15.72:3300/0 conn(0x7f43cc0b43a0 0x7f43cc099d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:01.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.775+0000 7f43d31d3640 1 --2- 172.21.15.67:0/1336855968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43cc15e1b0 0x7f43cc0cfe20 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:01.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.775+0000 7f43d31d3640 1 -- 172.21.15.67:0/1336855968 >> 172.21.15.67:0/1336855968 conn(0x7f43cc0b36d0 msgr2=0x7f43cc0b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:01.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.775+0000 7f43d31d3640 1 -- 172.21.15.67:0/1336855968 shutdown_connections 2024-03-24T00:40:01.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.775+0000 7f43d31d3640 1 -- 172.21.15.67:0/1336855968 wait complete. 2024-03-24T00:40:01.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.776+0000 7f43d31d3640 1 Processor -- start 2024-03-24T00:40:01.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.776+0000 7f43d31d3640 1 -- start start 2024-03-24T00:40:01.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.777+0000 7f43d31d3640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f43cc09a240 0x7f43cc156680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:01.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.777+0000 7f43d31d3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43cc0b43a0 0x7f43cc157180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:01.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.777+0000 7f43d31d3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43cc15e1b0 0x7f43cc0ca2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:01.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.777+0000 7f43d31d3640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43cc0cc360 con 0x7f43cc15e1b0 2024-03-24T00:40:01.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.777+0000 7f43d31d3640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43cc0d1540 con 0x7f43cc0b43a0 2024-03-24T00:40:01.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.777+0000 7f43d31d3640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43cc09a9a0 con 0x7f43cc09a240 2024-03-24T00:40:01.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.777+0000 7f43d1770640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43cc15e1b0 0x7f43cc0ca2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:01.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.777+0000 7f43d1770640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43cc15e1b0 0x7f43cc0ca2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39842/0 (socket says 172.21.15.67:39842) 2024-03-24T00:40:01.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.777+0000 7f43d1770640 1 -- 172.21.15.67:0/312965257 learned_addr learned my addr 172.21.15.67:0/312965257 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:01.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.777+0000 7f43d1770640 1 -- 172.21.15.67:0/312965257 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f43cc09a240 msgr2=0x7f43cc156680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:01.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.777+0000 7f43cbfff640 1 --2- 172.21.15.67:0/312965257 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43cc0b43a0 0x7f43cc157180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:01.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.777+0000 7f43d0f6f640 1 --2- 172.21.15.67:0/312965257 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f43cc09a240 0x7f43cc156680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:01.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.777+0000 7f43d1770640 1 --2- 172.21.15.67:0/312965257 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f43cc09a240 0x7f43cc156680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:01.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.777+0000 7f43d1770640 1 -- 172.21.15.67:0/312965257 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43cc0b43a0 msgr2=0x7f43cc157180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:01.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.777+0000 7f43d1770640 1 --2- 172.21.15.67:0/312965257 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43cc0b43a0 0x7f43cc157180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:01.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.777+0000 7f43d1770640 1 -- 172.21.15.67:0/312965257 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43cc1576c0 con 0x7f43cc15e1b0 2024-03-24T00:40:01.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.777+0000 7f43cbfff640 1 --2- 172.21.15.67:0/312965257 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43cc0b43a0 0x7f43cc157180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:40:01.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.777+0000 7f43d1770640 1 --2- 172.21.15.67:0/312965257 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43cc15e1b0 0x7f43cc0ca2a0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f43bc00c580 tx=0x7f43bc00ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:01.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.777+0000 7f43c97fa640 1 -- 172.21.15.67:0/312965257 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43bc015070 con 0x7f43cc15e1b0 2024-03-24T00:40:01.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.777+0000 7f43c97fa640 1 -- 172.21.15.67:0/312965257 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43bc004220 con 0x7f43cc15e1b0 2024-03-24T00:40:01.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.777+0000 7f43d31d3640 1 -- 172.21.15.67:0/312965257 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43cc1f3bb0 con 0x7f43cc15e1b0 2024-03-24T00:40:01.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.777+0000 7f43c97fa640 1 -- 172.21.15.67:0/312965257 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43bc003610 con 0x7f43cc15e1b0 2024-03-24T00:40:01.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.777+0000 7f43d31d3640 1 -- 172.21.15.67:0/312965257 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43cc1f4170 con 0x7f43cc15e1b0 2024-03-24T00:40:01.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.778+0000 7f43d31d3640 1 -- 172.21.15.67:0/312965257 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43cc04f400 con 0x7f43cc15e1b0 2024-03-24T00:40:01.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.779+0000 7f43c97fa640 1 -- 172.21.15.67:0/312965257 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f43bc0037b0 con 0x7f43cc15e1b0 2024-03-24T00:40:01.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.779+0000 7f43c97fa640 1 --2- 172.21.15.67:0/312965257 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f43b806bc00 0x7f43b806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:01.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.779+0000 7f43c97fa640 1 -- 172.21.15.67:0/312965257 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f43bc08a980 con 0x7f43cc15e1b0 2024-03-24T00:40:01.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.779+0000 7f43d0f6f640 1 --2- 172.21.15.67:0/312965257 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f43b806bc00 0x7f43b806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:01.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.780+0000 7f43d0f6f640 1 --2- 172.21.15.67:0/312965257 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f43b806bc00 0x7f43b806e0c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f43c4035000 tx=0x7f43c4030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:01.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.784+0000 7f43c97fa640 1 -- 172.21.15.67:0/312965257 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f43bc0592f0 con 0x7f43cc15e1b0 2024-03-24T00:40:01.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.968+0000 7f43d31d3640 1 -- 172.21.15.67:0/312965257 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f43cc1f38c0 con 0x7f43cc15e1b0 2024-03-24T00:40:01.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.969+0000 7f43c97fa640 1 -- 172.21.15.67:0/312965257 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+16895 (secure 0 0 0) 0x7f43bc058bf0 con 0x7f43cc15e1b0 2024-03-24T00:40:01.970 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:40:01.971 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":38,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:40:00.691111+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:39:58.975995+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-24T00:39:59.653615+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:40:01.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.971+0000 7f43d31d3640 1 -- 172.21.15.67:0/312965257 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f43b806bc00 msgr2=0x7f43b806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:01.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.971+0000 7f43d31d3640 1 --2- 172.21.15.67:0/312965257 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f43b806bc00 0x7f43b806e0c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f43c4035000 tx=0x7f43c4030000 comp rx=0 tx=0).stop 2024-03-24T00:40:01.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.971+0000 7f43d31d3640 1 -- 172.21.15.67:0/312965257 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43cc15e1b0 msgr2=0x7f43cc0ca2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:01.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.971+0000 7f43d31d3640 1 --2- 172.21.15.67:0/312965257 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43cc15e1b0 0x7f43cc0ca2a0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f43bc00c580 tx=0x7f43bc00ca40 comp rx=0 tx=0).stop 2024-03-24T00:40:01.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.971+0000 7f43d31d3640 1 -- 172.21.15.67:0/312965257 shutdown_connections 2024-03-24T00:40:01.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.971+0000 7f43d31d3640 1 --2- 172.21.15.67:0/312965257 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f43b806bc00 0x7f43b806e0c0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:01.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.971+0000 7f43d31d3640 1 --2- 172.21.15.67:0/312965257 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43cc15e1b0 0x7f43cc0ca2a0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:01.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.971+0000 7f43d31d3640 1 --2- 172.21.15.67:0/312965257 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43cc0b43a0 0x7f43cc157180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:01.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.971+0000 7f43d31d3640 1 --2- 172.21.15.67:0/312965257 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f43cc09a240 0x7f43cc156680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:01.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.972+0000 7f43d31d3640 1 -- 172.21.15.67:0/312965257 >> 172.21.15.67:0/312965257 conn(0x7f43cc0b36d0 msgr2=0x7f43cc0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:01.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.972+0000 7f43d31d3640 1 -- 172.21.15.67:0/312965257 shutdown_connections 2024-03-24T00:40:01.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:01.972+0000 7f43d31d3640 1 -- 172.21.15.67:0/312965257 wait complete. 2024-03-24T00:40:01.983 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-24T00:40:02.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.087+0000 7f19e20dc640 1 Processor -- start 2024-03-24T00:40:02.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.087+0000 7f19e20dc640 1 -- start start 2024-03-24T00:40:02.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.088+0000 7f19e20dc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f19dc099d00 0x7f19dc09a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:02.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.088+0000 7f19e20dc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f19dc0b49b0 0x7f19dc0d0e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:02.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.088+0000 7f19e20dc640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19dc09a620 con 0x7f19dc0d1360 2024-03-24T00:40:02.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.088+0000 7f19e20dc640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19dc0cc1b0 con 0x7f19dc0b49b0 2024-03-24T00:40:02.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.088+0000 7f19e20dc640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19dc0cc400 con 0x7f19dc099d00 2024-03-24T00:40:02.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.088+0000 7f19daffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f19dc0b49b0 0x7f19dc0d0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:02.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.088+0000 7f19daffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f19dc0b49b0 0x7f19dc0d0e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39858/0 (socket says 172.21.15.67:39858) 2024-03-24T00:40:02.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.088+0000 7f19daffd640 1 -- 172.21.15.67:0/398280307 learned_addr learned my addr 172.21.15.67:0/398280307 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:02.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.088+0000 7f19db7fe640 1 --2- 172.21.15.67:0/398280307 >> v2:172.21.15.72:3300/0 conn(0x7f19dc099d00 0x7f19dc09a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:02.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.088+0000 7f19daffd640 1 -- 172.21.15.67:0/398280307 >> v2:172.21.15.72:3300/0 conn(0x7f19dc099d00 msgr2=0x7f19dc09a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:02.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.088+0000 7f19daffd640 1 --2- 172.21.15.67:0/398280307 >> v2:172.21.15.72:3300/0 conn(0x7f19dc099d00 0x7f19dc09a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:02.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.088+0000 7f19daffd640 1 -- 172.21.15.67:0/398280307 >> v1:172.21.15.72:6789/0 conn(0x7f19dc0d1360 legacy=0x7f19dc15d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:02.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.088+0000 7f19daffd640 1 -- 172.21.15.67:0/398280307 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19dc0cc6e0 con 0x7f19dc0b49b0 2024-03-24T00:40:02.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.089+0000 7f19daffd640 1 --2- 172.21.15.67:0/398280307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19dc0b49b0 0x7f19dc0d0e20 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f19dc08b260 tx=0x7f19d002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43c525d3f959b765 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:02.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.089+0000 7f19da7fc640 1 -- 172.21.15.67:0/398280307 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d0034070 con 0x7f19dc0b49b0 2024-03-24T00:40:02.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.089+0000 7f19da7fc640 1 -- 172.21.15.67:0/398280307 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19d0030860 con 0x7f19dc0b49b0 2024-03-24T00:40:02.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.089+0000 7f19da7fc640 1 -- 172.21.15.67:0/398280307 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d0030c30 con 0x7f19dc0b49b0 2024-03-24T00:40:02.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.090+0000 7f19e20dc640 1 -- 172.21.15.67:0/398280307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19dc0b49b0 msgr2=0x7f19dc0d0e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:02.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.090+0000 7f19e20dc640 1 --2- 172.21.15.67:0/398280307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19dc0b49b0 0x7f19dc0d0e20 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f19dc08b260 tx=0x7f19d002b9a0 comp rx=0 tx=0).stop 2024-03-24T00:40:02.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.090+0000 7f19e20dc640 1 -- 172.21.15.67:0/398280307 shutdown_connections 2024-03-24T00:40:02.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.090+0000 7f19e20dc640 1 --2- 172.21.15.67:0/398280307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19dc0b49b0 0x7f19dc0d0e20 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:02.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.090+0000 7f19e20dc640 1 --2- 172.21.15.67:0/398280307 >> v2:172.21.15.72:3300/0 conn(0x7f19dc099d00 0x7f19dc09a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:02.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.090+0000 7f19e20dc640 1 -- 172.21.15.67:0/398280307 >> 172.21.15.67:0/398280307 conn(0x7f19dc0b3650 msgr2=0x7f19dc0b3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:02.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.090+0000 7f19e20dc640 1 -- 172.21.15.67:0/398280307 shutdown_connections 2024-03-24T00:40:02.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.090+0000 7f19e20dc640 1 -- 172.21.15.67:0/398280307 wait complete. 2024-03-24T00:40:02.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.091+0000 7f19e20dc640 1 Processor -- start 2024-03-24T00:40:02.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.091+0000 7f19e20dc640 1 -- start start 2024-03-24T00:40:02.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.092+0000 7f19e20dc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19dc099d00 0x7f19dc0c6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:02.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.092+0000 7f19e20dc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19dc0b49b0 0x7f19dc0b5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:02.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.092+0000 7f19e20dc640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f19dc0d1360 0x7f19dc0b6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:02.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.092+0000 7f19e20dc640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19dc0d01c0 con 0x7f19dc099d00 2024-03-24T00:40:02.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.092+0000 7f19e20dc640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19dc0ccca0 con 0x7f19dc0b49b0 2024-03-24T00:40:02.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.092+0000 7f19e20dc640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19dc0b32a0 con 0x7f19dc0d1360 2024-03-24T00:40:02.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.092+0000 7f19db7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19dc099d00 0x7f19dc0c6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:02.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.092+0000 7f19db7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19dc099d00 0x7f19dc0c6b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39862/0 (socket says 172.21.15.67:39862) 2024-03-24T00:40:02.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.092+0000 7f19db7fe640 1 -- 172.21.15.67:0/393979733 learned_addr learned my addr 172.21.15.67:0/393979733 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:02.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.092+0000 7f19daffd640 1 --2- 172.21.15.67:0/393979733 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19dc0b49b0 0x7f19dc0b5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:02.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.092+0000 7f19dbfff640 1 --2- 172.21.15.67:0/393979733 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f19dc0d1360 0x7f19dc0b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:02.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.092+0000 7f19db7fe640 1 -- 172.21.15.67:0/393979733 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f19dc0d1360 msgr2=0x7f19dc0b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:02.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.092+0000 7f19db7fe640 1 --2- 172.21.15.67:0/393979733 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f19dc0d1360 0x7f19dc0b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:02.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.092+0000 7f19db7fe640 1 -- 172.21.15.67:0/393979733 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19dc0b49b0 msgr2=0x7f19dc0b5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:02.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.092+0000 7f19db7fe640 1 --2- 172.21.15.67:0/393979733 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19dc0b49b0 0x7f19dc0b5e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:02.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.092+0000 7f19db7fe640 1 -- 172.21.15.67:0/393979733 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19dc151ae0 con 0x7f19dc099d00 2024-03-24T00:40:02.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.093+0000 7f19db7fe640 1 --2- 172.21.15.67:0/393979733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19dc099d00 0x7f19dc0c6b30 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f19cc00e9b0 tx=0x7f19cc00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:02.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.093+0000 7f19c3fff640 1 -- 172.21.15.67:0/393979733 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19cc017070 con 0x7f19dc099d00 2024-03-24T00:40:02.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.093+0000 7f19c3fff640 1 -- 172.21.15.67:0/393979733 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19cc00c9a0 con 0x7f19dc099d00 2024-03-24T00:40:02.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.093+0000 7f19e20dc640 1 -- 172.21.15.67:0/393979733 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19dc150a40 con 0x7f19dc099d00 2024-03-24T00:40:02.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.093+0000 7f19c3fff640 1 -- 172.21.15.67:0/393979733 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19cc00cd90 con 0x7f19dc099d00 2024-03-24T00:40:02.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.093+0000 7f19e20dc640 1 -- 172.21.15.67:0/393979733 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19dc151000 con 0x7f19dc099d00 2024-03-24T00:40:02.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.094+0000 7f19e20dc640 1 -- 172.21.15.67:0/393979733 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19dc04f400 con 0x7f19dc099d00 2024-03-24T00:40:02.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.094+0000 7f19c3fff640 1 -- 172.21.15.67:0/393979733 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f19cc01a070 con 0x7f19dc099d00 2024-03-24T00:40:02.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.094+0000 7f19c3fff640 1 --2- 172.21.15.67:0/393979733 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f19c806bbd0 0x7f19c806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:02.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.095+0000 7f19c3fff640 1 -- 172.21.15.67:0/393979733 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f19cc08aea0 con 0x7f19dc099d00 2024-03-24T00:40:02.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.095+0000 7f19daffd640 1 --2- 172.21.15.67:0/393979733 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f19c806bbd0 0x7f19c806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:02.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.095+0000 7f19daffd640 1 --2- 172.21.15.67:0/393979733 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f19c806bbd0 0x7f19c806e090 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f19d0036000 tx=0x7f19d0008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:02.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.099+0000 7f19c3fff640 1 -- 172.21.15.67:0/393979733 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f19dc151ae0 con 0x7f19dc099d00 2024-03-24T00:40:02.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.321+0000 7f19e20dc640 1 -- 172.21.15.67:0/393979733 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f19dc09a210 con 0x7f19dc099d00 2024-03-24T00:40:02.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.788+0000 7f19c3fff640 1 -- 172.21.15.67:0/393979733 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v16) v1 ==== 118+0+0 (secure 0 0 0) 0x7f19cc059240 con 0x7f19dc099d00 2024-03-24T00:40:02.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.791+0000 7f19e20dc640 1 -- 172.21.15.67:0/393979733 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f19c806bbd0 msgr2=0x7f19c806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:02.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.791+0000 7f19e20dc640 1 --2- 172.21.15.67:0/393979733 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f19c806bbd0 0x7f19c806e090 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f19d0036000 tx=0x7f19d0008030 comp rx=0 tx=0).stop 2024-03-24T00:40:02.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.791+0000 7f19e20dc640 1 -- 172.21.15.67:0/393979733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19dc099d00 msgr2=0x7f19dc0c6b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:02.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.791+0000 7f19e20dc640 1 --2- 172.21.15.67:0/393979733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19dc099d00 0x7f19dc0c6b30 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f19cc00e9b0 tx=0x7f19cc00ee70 comp rx=0 tx=0).stop 2024-03-24T00:40:02.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.791+0000 7f19e20dc640 1 -- 172.21.15.67:0/393979733 shutdown_connections 2024-03-24T00:40:02.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.791+0000 7f19e20dc640 1 --2- 172.21.15.67:0/393979733 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f19c806bbd0 0x7f19c806e090 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:02.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.791+0000 7f19e20dc640 1 --2- 172.21.15.67:0/393979733 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f19dc0d1360 0x7f19dc0b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:02.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.791+0000 7f19e20dc640 1 --2- 172.21.15.67:0/393979733 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19dc0b49b0 0x7f19dc0b5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:02.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.791+0000 7f19e20dc640 1 --2- 172.21.15.67:0/393979733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19dc099d00 0x7f19dc0c6b30 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:02.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.792+0000 7f19e20dc640 1 -- 172.21.15.67:0/393979733 >> 172.21.15.67:0/393979733 conn(0x7f19dc0b3650 msgr2=0x7f19dc06cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:02.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.792+0000 7f19e20dc640 1 -- 172.21.15.67:0/393979733 shutdown_connections 2024-03-24T00:40:02.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.792+0000 7f19e20dc640 1 -- 172.21.15.67:0/393979733 wait complete. 2024-03-24T00:40:02.804 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-24T00:40:02.805 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-24T00:40:02.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.912+0000 7f830fdc4640 1 Processor -- start 2024-03-24T00:40:02.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.912+0000 7f830fdc4640 1 -- start start 2024-03-24T00:40:02.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.913+0000 7f830fdc4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f830815e130 0x7f83080ce610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:02.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.913+0000 7f830fdc4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f83080d0640 0x7f83080d0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:02.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.913+0000 7f830fdc4640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83080cf570 con 0x7f83080b57a0 2024-03-24T00:40:02.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.913+0000 7f830fdc4640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83080ceb50 con 0x7f83080d0640 2024-03-24T00:40:02.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.913+0000 7f830fdc4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83080d0fe0 con 0x7f830815e130 2024-03-24T00:40:02.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.913+0000 7f830d35f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f83080d0640 0x7f83080d0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:02.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.913+0000 7f830e361640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f83080b57a0 0x7f83080b5bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49152/0 (socket says 172.21.15.67:49152) 2024-03-24T00:40:02.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.913+0000 7f830e361640 1 -- 172.21.15.67:0/19566860 learned_addr learned my addr 172.21.15.67:0/19566860 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:40:02.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.913+0000 7f830db60640 1 --2- 172.21.15.67:0/19566860 >> v2:172.21.15.72:3300/0 conn(0x7f830815e130 0x7f83080ce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:02.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.913+0000 7f830d35f640 1 -- 172.21.15.67:0/19566860 >> v2:172.21.15.72:3300/0 conn(0x7f830815e130 msgr2=0x7f83080ce610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:02.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.913+0000 7f830d35f640 1 --2- 172.21.15.67:0/19566860 >> v2:172.21.15.72:3300/0 conn(0x7f830815e130 0x7f83080ce610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:02.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.913+0000 7f830d35f640 1 -- 172.21.15.67:0/19566860 >> v1:172.21.15.67:6789/0 conn(0x7f83080b57a0 legacy=0x7f83080b5bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:02.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.913+0000 7f830d35f640 1 -- 172.21.15.67:0/19566860 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83080cecf0 con 0x7f83080d0640 2024-03-24T00:40:02.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.914+0000 7f830d35f640 1 --2- 172.21.15.67:0/19566860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83080d0640 0x7f83080d0aa0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f83080b6d40 tx=0x7f82f802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfc86904fdaebc5b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:02.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.914+0000 7f830cb5e640 1 -- 172.21.15.67:0/19566860 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f8035020 con 0x7f83080d0640 2024-03-24T00:40:02.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.914+0000 7f830cb5e640 1 -- 172.21.15.67:0/19566860 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82f8031830 con 0x7f83080d0640 2024-03-24T00:40:02.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.914+0000 7f830cb5e640 1 -- 172.21.15.67:0/19566860 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f8031c00 con 0x7f83080d0640 2024-03-24T00:40:02.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.914+0000 7f830fdc4640 1 -- 172.21.15.67:0/19566860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83080d0640 msgr2=0x7f83080d0aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:02.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.914+0000 7f830fdc4640 1 --2- 172.21.15.67:0/19566860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83080d0640 0x7f83080d0aa0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f83080b6d40 tx=0x7f82f802b8d0 comp rx=0 tx=0).stop 2024-03-24T00:40:02.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.914+0000 7f830fdc4640 1 -- 172.21.15.67:0/19566860 shutdown_connections 2024-03-24T00:40:02.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.914+0000 7f830fdc4640 1 --2- 172.21.15.67:0/19566860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83080d0640 0x7f83080d0aa0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:02.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.914+0000 7f830fdc4640 1 --2- 172.21.15.67:0/19566860 >> v2:172.21.15.72:3300/0 conn(0x7f830815e130 0x7f83080ce610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:02.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.914+0000 7f830fdc4640 1 -- 172.21.15.67:0/19566860 >> 172.21.15.67:0/19566860 conn(0x7f830806c8a0 msgr2=0x7f830806ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:02.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.915+0000 7f830fdc4640 1 -- 172.21.15.67:0/19566860 shutdown_connections 2024-03-24T00:40:02.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.915+0000 7f830fdc4640 1 -- 172.21.15.67:0/19566860 wait complete. 2024-03-24T00:40:02.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.916+0000 7f830fdc4640 1 Processor -- start 2024-03-24T00:40:02.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.916+0000 7f830fdc4640 1 -- start start 2024-03-24T00:40:02.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.916+0000 7f830fdc4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83080b57a0 0x7f83080cdc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:02.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.916+0000 7f830fdc4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83080d0640 0x7f83080aa950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:02.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.916+0000 7f830fdc4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f830815e130 0x7f830815d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:02.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.916+0000 7f830fdc4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83080cf290 con 0x7f83080d0640 2024-03-24T00:40:02.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.916+0000 7f830fdc4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83080cefb0 con 0x7f83080b57a0 2024-03-24T00:40:02.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.916+0000 7f830fdc4640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f830815e510 con 0x7f830815e130 2024-03-24T00:40:02.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.917+0000 7f830d35f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83080d0640 0x7f83080aa950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:02.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.917+0000 7f830d35f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83080d0640 0x7f83080aa950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39868/0 (socket says 172.21.15.67:39868) 2024-03-24T00:40:02.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.917+0000 7f830d35f640 1 -- 172.21.15.67:0/3759013916 learned_addr learned my addr 172.21.15.67:0/3759013916 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:02.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.917+0000 7f830db60640 1 --2- 172.21.15.67:0/3759013916 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83080b57a0 0x7f83080cdc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:02.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.917+0000 7f830e361640 1 --2- 172.21.15.67:0/3759013916 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f830815e130 0x7f830815d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:02.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.917+0000 7f830d35f640 1 -- 172.21.15.67:0/3759013916 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f830815e130 msgr2=0x7f830815d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:02.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.917+0000 7f830d35f640 1 --2- 172.21.15.67:0/3759013916 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f830815e130 0x7f830815d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:02.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.917+0000 7f830d35f640 1 -- 172.21.15.67:0/3759013916 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83080b57a0 msgr2=0x7f83080cdc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:02.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.917+0000 7f830d35f640 1 --2- 172.21.15.67:0/3759013916 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83080b57a0 0x7f83080cdc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:02.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.917+0000 7f830d35f640 1 -- 172.21.15.67:0/3759013916 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83080ccbf0 con 0x7f83080d0640 2024-03-24T00:40:02.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.917+0000 7f830d35f640 1 --2- 172.21.15.67:0/3759013916 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83080d0640 0x7f83080aa950 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f82f8000c00 tx=0x7f82f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:02.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.917+0000 7f82fe7fc640 1 -- 172.21.15.67:0/3759013916 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f8041070 con 0x7f83080d0640 2024-03-24T00:40:02.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.917+0000 7f82fe7fc640 1 -- 172.21.15.67:0/3759013916 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82f803b880 con 0x7f83080d0640 2024-03-24T00:40:02.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.917+0000 7f830fdc4640 1 -- 172.21.15.67:0/3759013916 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83080cced0 con 0x7f83080d0640 2024-03-24T00:40:02.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.917+0000 7f82fe7fc640 1 -- 172.21.15.67:0/3759013916 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f8035040 con 0x7f83080d0640 2024-03-24T00:40:02.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.917+0000 7f830fdc4640 1 -- 172.21.15.67:0/3759013916 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83080cd490 con 0x7f83080d0640 2024-03-24T00:40:02.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.918+0000 7f830fdc4640 1 -- 172.21.15.67:0/3759013916 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f830804f400 con 0x7f83080d0640 2024-03-24T00:40:02.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.919+0000 7f82fe7fc640 1 -- 172.21.15.67:0/3759013916 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f82f8038020 con 0x7f83080d0640 2024-03-24T00:40:02.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.919+0000 7f82fe7fc640 1 --2- 172.21.15.67:0/3759013916 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f82ec06bbd0 0x7f82ec06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:02.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.919+0000 7f82fe7fc640 1 -- 172.21.15.67:0/3759013916 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f82f80ac410 con 0x7f83080d0640 2024-03-24T00:40:02.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.919+0000 7f830db60640 1 --2- 172.21.15.67:0/3759013916 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f82ec06bbd0 0x7f82ec06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:02.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.920+0000 7f830db60640 1 --2- 172.21.15.67:0/3759013916 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f82ec06bbd0 0x7f82ec06e090 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8304009aa0 tx=0x7f8304009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:02.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:02.923+0000 7f82fe7fc640 1 -- 172.21.15.67:0/3759013916 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f82f8031800 con 0x7f83080d0640 2024-03-24T00:40:03.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.144+0000 7f830fdc4640 1 -- 172.21.15.67:0/3759013916 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f83080d19f0 con 0x7f83080d0640 2024-03-24T00:40:03.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.792+0000 7f82fe7fc640 1 -- 172.21.15.67:0/3759013916 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v17) v1 ==== 122+0+0 (secure 0 0 0) 0x7f82f807a7b0 con 0x7f83080d0640 2024-03-24T00:40:03.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.794+0000 7f830fdc4640 1 -- 172.21.15.67:0/3759013916 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f82ec06bbd0 msgr2=0x7f82ec06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:03.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.794+0000 7f830fdc4640 1 --2- 172.21.15.67:0/3759013916 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f82ec06bbd0 0x7f82ec06e090 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8304009aa0 tx=0x7f8304009190 comp rx=0 tx=0).stop 2024-03-24T00:40:03.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.794+0000 7f830fdc4640 1 -- 172.21.15.67:0/3759013916 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83080d0640 msgr2=0x7f83080aa950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:03.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.794+0000 7f830fdc4640 1 --2- 172.21.15.67:0/3759013916 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83080d0640 0x7f83080aa950 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f82f8000c00 tx=0x7f82f80046c0 comp rx=0 tx=0).stop 2024-03-24T00:40:03.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.795+0000 7f830fdc4640 1 -- 172.21.15.67:0/3759013916 shutdown_connections 2024-03-24T00:40:03.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.795+0000 7f830fdc4640 1 --2- 172.21.15.67:0/3759013916 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f82ec06bbd0 0x7f82ec06e090 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:03.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.795+0000 7f830fdc4640 1 --2- 172.21.15.67:0/3759013916 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f830815e130 0x7f830815d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:03.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.795+0000 7f830fdc4640 1 --2- 172.21.15.67:0/3759013916 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83080d0640 0x7f83080aa950 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:03.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.795+0000 7f830fdc4640 1 --2- 172.21.15.67:0/3759013916 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83080b57a0 0x7f83080cdc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:03.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.795+0000 7f830fdc4640 1 -- 172.21.15.67:0/3759013916 >> 172.21.15.67:0/3759013916 conn(0x7f830806c8a0 msgr2=0x7f8308160700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:03.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.795+0000 7f830fdc4640 1 -- 172.21.15.67:0/3759013916 shutdown_connections 2024-03-24T00:40:03.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.795+0000 7f830fdc4640 1 -- 172.21.15.67:0/3759013916 wait complete. 2024-03-24T00:40:03.807 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-24T00:40:03.807 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:40:03.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.909+0000 7fe70b3b9640 1 Processor -- start 2024-03-24T00:40:03.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.910+0000 7fe70b3b9640 1 -- start start 2024-03-24T00:40:03.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.910+0000 7fe70b3b9640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe70415e1b0 0x7fe7040cfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:03.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.910+0000 7fe70b3b9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe7040b43a0 0x7fe704099d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:03.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.910+0000 7fe70b3b9640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7040d0fc0 con 0x7fe70409a240 2024-03-24T00:40:03.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.910+0000 7fe70b3b9640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe70415e590 con 0x7fe7040b43a0 2024-03-24T00:40:03.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.910+0000 7fe70b3b9640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe70409a690 con 0x7fe70415e1b0 2024-03-24T00:40:03.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.911+0000 7fe708954640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe7040b43a0 0x7fe704099d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:03.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.911+0000 7fe708954640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe7040b43a0 0x7fe704099d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40772/0 (socket says 172.21.15.67:40772) 2024-03-24T00:40:03.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.911+0000 7fe708954640 1 -- 172.21.15.67:0/2479275837 learned_addr learned my addr 172.21.15.67:0/2479275837 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:03.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.911+0000 7fe709155640 1 --2- 172.21.15.67:0/2479275837 >> v2:172.21.15.72:3300/0 conn(0x7fe70415e1b0 0x7fe7040cfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:03.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.911+0000 7fe708954640 1 -- 172.21.15.67:0/2479275837 >> v2:172.21.15.72:3300/0 conn(0x7fe70415e1b0 msgr2=0x7fe7040cfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:03.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.911+0000 7fe708954640 1 --2- 172.21.15.67:0/2479275837 >> v2:172.21.15.72:3300/0 conn(0x7fe70415e1b0 0x7fe7040cfe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:03.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.911+0000 7fe708954640 1 -- 172.21.15.67:0/2479275837 >> v1:172.21.15.72:6789/0 conn(0x7fe70409a240 legacy=0x7fe70406c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:03.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.911+0000 7fe708954640 1 -- 172.21.15.67:0/2479275837 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7040d1260 con 0x7fe7040b43a0 2024-03-24T00:40:03.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.912+0000 7fe708954640 1 --2- 172.21.15.67:0/2479275837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7040b43a0 0x7fe704099d00 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fe704073070 tx=0x7fe6f802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5a694b000407684 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:03.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.912+0000 7fe703fff640 1 -- 172.21.15.67:0/2479275837 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f802e070 con 0x7fe7040b43a0 2024-03-24T00:40:03.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.912+0000 7fe703fff640 1 -- 172.21.15.67:0/2479275837 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6f8037070 con 0x7fe7040b43a0 2024-03-24T00:40:03.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.912+0000 7fe703fff640 1 -- 172.21.15.67:0/2479275837 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f8031a00 con 0x7fe7040b43a0 2024-03-24T00:40:03.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.912+0000 7fe70b3b9640 1 -- 172.21.15.67:0/2479275837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7040b43a0 msgr2=0x7fe704099d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:03.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.912+0000 7fe70b3b9640 1 --2- 172.21.15.67:0/2479275837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7040b43a0 0x7fe704099d00 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fe704073070 tx=0x7fe6f802b880 comp rx=0 tx=0).stop 2024-03-24T00:40:03.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.912+0000 7fe70b3b9640 1 -- 172.21.15.67:0/2479275837 shutdown_connections 2024-03-24T00:40:03.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.912+0000 7fe70b3b9640 1 --2- 172.21.15.67:0/2479275837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7040b43a0 0x7fe704099d00 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:03.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.912+0000 7fe70b3b9640 1 --2- 172.21.15.67:0/2479275837 >> v2:172.21.15.72:3300/0 conn(0x7fe70415e1b0 0x7fe7040cfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:03.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.912+0000 7fe70b3b9640 1 -- 172.21.15.67:0/2479275837 >> 172.21.15.67:0/2479275837 conn(0x7fe7040b36d0 msgr2=0x7fe7040b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:03.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.912+0000 7fe70b3b9640 1 -- 172.21.15.67:0/2479275837 shutdown_connections 2024-03-24T00:40:03.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.912+0000 7fe70b3b9640 1 -- 172.21.15.67:0/2479275837 wait complete. 2024-03-24T00:40:03.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.913+0000 7fe70b3b9640 1 Processor -- start 2024-03-24T00:40:03.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.914+0000 7fe70b3b9640 1 -- start start 2024-03-24T00:40:03.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.914+0000 7fe70b3b9640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe70409a240 0x7fe704155250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:03.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.914+0000 7fe70b3b9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7040b43a0 0x7fe7040ae7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:03.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.914+0000 7fe70b3b9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe70415e1b0 0x7fe7040c9280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:03.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.914+0000 7fe70b3b9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7040cc360 con 0x7fe7040b43a0 2024-03-24T00:40:03.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.914+0000 7fe70b3b9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7040d1820 con 0x7fe70415e1b0 2024-03-24T00:40:03.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.914+0000 7fe70b3b9640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe70409a9a0 con 0x7fe70409a240 2024-03-24T00:40:03.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.914+0000 7fe708954640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7040b43a0 0x7fe7040ae7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:03.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.914+0000 7fe708954640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7040b43a0 0x7fe7040ae7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40786/0 (socket says 172.21.15.67:40786) 2024-03-24T00:40:03.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.914+0000 7fe708954640 1 -- 172.21.15.67:0/2299294554 learned_addr learned my addr 172.21.15.67:0/2299294554 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:03.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.914+0000 7fe709956640 1 --2- 172.21.15.67:0/2299294554 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe70415e1b0 0x7fe7040c9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:03.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.914+0000 7fe709155640 1 --2- 172.21.15.67:0/2299294554 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe70409a240 0x7fe704155250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:03.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.915+0000 7fe708954640 1 -- 172.21.15.67:0/2299294554 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe70409a240 msgr2=0x7fe704155250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:03.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.915+0000 7fe708954640 1 --2- 172.21.15.67:0/2299294554 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe70409a240 0x7fe704155250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:03.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.915+0000 7fe708954640 1 -- 172.21.15.67:0/2299294554 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe70415e1b0 msgr2=0x7fe7040c9280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:03.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.915+0000 7fe708954640 1 --2- 172.21.15.67:0/2299294554 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe70415e1b0 0x7fe7040c9280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:03.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.915+0000 7fe708954640 1 -- 172.21.15.67:0/2299294554 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7040b11c0 con 0x7fe7040b43a0 2024-03-24T00:40:03.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.915+0000 7fe708954640 1 --2- 172.21.15.67:0/2299294554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7040b43a0 0x7fe7040ae7e0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fe6f8000c00 tx=0x7fe6f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:03.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.915+0000 7fe701ffb640 1 -- 172.21.15.67:0/2299294554 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f8032670 con 0x7fe7040b43a0 2024-03-24T00:40:03.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.915+0000 7fe701ffb640 1 -- 172.21.15.67:0/2299294554 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6f8032850 con 0x7fe7040b43a0 2024-03-24T00:40:03.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.915+0000 7fe70b3b9640 1 -- 172.21.15.67:0/2299294554 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7040b14a0 con 0x7fe7040b43a0 2024-03-24T00:40:03.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.915+0000 7fe701ffb640 1 -- 172.21.15.67:0/2299294554 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f8032c40 con 0x7fe7040b43a0 2024-03-24T00:40:03.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.915+0000 7fe70b3b9640 1 -- 172.21.15.67:0/2299294554 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7040b1a60 con 0x7fe7040b43a0 2024-03-24T00:40:03.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.916+0000 7fe70b3b9640 1 -- 172.21.15.67:0/2299294554 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe70404f400 con 0x7fe7040b43a0 2024-03-24T00:40:03.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.917+0000 7fe701ffb640 1 -- 172.21.15.67:0/2299294554 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe6f80453f0 con 0x7fe7040b43a0 2024-03-24T00:40:03.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.917+0000 7fe701ffb640 1 --2- 172.21.15.67:0/2299294554 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe6f006bbd0 0x7fe6f006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:03.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.917+0000 7fe701ffb640 1 -- 172.21.15.67:0/2299294554 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe6f80ac920 con 0x7fe7040b43a0 2024-03-24T00:40:03.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.917+0000 7fe709155640 1 --2- 172.21.15.67:0/2299294554 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe6f006bbd0 0x7fe6f006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:03.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.918+0000 7fe709155640 1 --2- 172.21.15.67:0/2299294554 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe6f006bbd0 0x7fe6f006e090 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc009aa0 tx=0x7fe6fc009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:03.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:03.921+0000 7fe701ffb640 1 -- 172.21.15.67:0/2299294554 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe6f8040070 con 0x7fe7040b43a0 2024-03-24T00:40:04.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.145+0000 7fe70b3b9640 1 -- 172.21.15.67:0/2299294554 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7040cf6e0 con 0x7fe7040b43a0 2024-03-24T00:40:04.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.146+0000 7fe701ffb640 1 -- 172.21.15.67:0/2299294554 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4974 (secure 0 0 0) 0x7fe6f807acc0 con 0x7fe7040b43a0 2024-03-24T00:40:04.147 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:40:04.147 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5039,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/166859817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":166859817},{"type":"v1","addr":"172.21.15.67:6834","nonce":166859817}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5042,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2653663845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2653663845},{"type":"v1","addr":"172.21.15.67:6835","nonce":2653663845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5048,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2355909634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2355909634},{"type":"v1","addr":"172.21.15.72:6837","nonce":2355909634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:40:00.691429+0000","modified":"2024-03-24T00:40:03.146091+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5418},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5418":{"gid":5418,"name":"d","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6836/2551774148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2551774148},{"type":"v1","addr":"172.21.15.72:6836","nonce":2551774148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5418,"qdb_cluster":[5418]},"id":2}]} 2024-03-24T00:40:04.147 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 17 2024-03-24T00:40:04.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.148+0000 7fe70b3b9640 1 -- 172.21.15.67:0/2299294554 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe6f006bbd0 msgr2=0x7fe6f006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:04.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.148+0000 7fe70b3b9640 1 --2- 172.21.15.67:0/2299294554 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe6f006bbd0 0x7fe6f006e090 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc009aa0 tx=0x7fe6fc009190 comp rx=0 tx=0).stop 2024-03-24T00:40:04.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.148+0000 7fe70b3b9640 1 -- 172.21.15.67:0/2299294554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7040b43a0 msgr2=0x7fe7040ae7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:04.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.148+0000 7fe70b3b9640 1 --2- 172.21.15.67:0/2299294554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7040b43a0 0x7fe7040ae7e0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fe6f8000c00 tx=0x7fe6f80046c0 comp rx=0 tx=0).stop 2024-03-24T00:40:04.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.148+0000 7fe70b3b9640 1 -- 172.21.15.67:0/2299294554 shutdown_connections 2024-03-24T00:40:04.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.148+0000 7fe70b3b9640 1 --2- 172.21.15.67:0/2299294554 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe6f006bbd0 0x7fe6f006e090 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:04.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.148+0000 7fe70b3b9640 1 --2- 172.21.15.67:0/2299294554 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe70415e1b0 0x7fe7040c9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:04.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.148+0000 7fe70b3b9640 1 --2- 172.21.15.67:0/2299294554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7040b43a0 0x7fe7040ae7e0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:04.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.148+0000 7fe70b3b9640 1 --2- 172.21.15.67:0/2299294554 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe70409a240 0x7fe704155250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:04.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.148+0000 7fe70b3b9640 1 -- 172.21.15.67:0/2299294554 >> 172.21.15.67:0/2299294554 conn(0x7fe7040b36d0 msgr2=0x7fe7040707b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:04.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.148+0000 7fe70b3b9640 1 -- 172.21.15.67:0/2299294554 shutdown_connections 2024-03-24T00:40:04.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.148+0000 7fe70b3b9640 1 -- 172.21.15.67:0/2299294554 wait complete. 2024-03-24T00:40:04.160 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:40:04.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.260+0000 7f7d8f305640 1 Processor -- start 2024-03-24T00:40:04.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.260+0000 7f7d8f305640 1 -- start start 2024-03-24T00:40:04.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.261+0000 7f7d8f305640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7d8815e2a0 0x7f7d880ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:04.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.261+0000 7f7d8f305640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7d880d0ef0 0x7f7d88162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:04.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.261+0000 7f7d8f305640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d880cf640 con 0x7f7d8815dec0 2024-03-24T00:40:04.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.261+0000 7f7d8f305640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d880cee30 con 0x7f7d880d0ef0 2024-03-24T00:40:04.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.261+0000 7f7d8f305640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d880d2120 con 0x7f7d8815e2a0 2024-03-24T00:40:04.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.261+0000 7f7d8d0a1640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7d8815dec0 0x7f7d880d07e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42392/0 (socket says 172.21.15.67:42392) 2024-03-24T00:40:04.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.261+0000 7f7d8d0a1640 1 -- 172.21.15.67:0/3691490480 learned_addr learned my addr 172.21.15.67:0/3691490480 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:40:04.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.261+0000 7f7d8c8a0640 1 --2- 172.21.15.67:0/3691490480 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7d8815e2a0 0x7f7d880ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:04.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.261+0000 7f7d8d8a2640 1 --2- 172.21.15.67:0/3691490480 >> v2:172.21.15.72:3300/0 conn(0x7f7d880d0ef0 0x7f7d88162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:04.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.261+0000 7f7d7bfff640 1 -- 172.21.15.67:0/3691490480 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2251266376 0 0) 0x7f7d7c001110 con 0x7f7d8815dec0 2024-03-24T00:40:04.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.261+0000 7f7d7bfff640 1 -- 172.21.15.67:0/3691490480 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d74003520 con 0x7f7d8815dec0 2024-03-24T00:40:04.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.261+0000 7f7d8c8a0640 1 -- 172.21.15.67:0/3691490480 >> v2:172.21.15.72:3300/0 conn(0x7f7d880d0ef0 msgr2=0x7f7d88162630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:04.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.261+0000 7f7d8c8a0640 1 --2- 172.21.15.67:0/3691490480 >> v2:172.21.15.72:3300/0 conn(0x7f7d880d0ef0 0x7f7d88162630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:04.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.261+0000 7f7d8c8a0640 1 -- 172.21.15.67:0/3691490480 >> v1:172.21.15.67:6789/0 conn(0x7f7d8815dec0 legacy=0x7f7d880d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:04.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.261+0000 7f7d8c8a0640 1 -- 172.21.15.67:0/3691490480 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d880cf080 con 0x7f7d8815e2a0 2024-03-24T00:40:04.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.261+0000 7f7d8d8a2640 1 --2- 172.21.15.67:0/3691490480 >> v2:172.21.15.72:3300/0 conn(0x7f7d880d0ef0 0x7f7d88162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:40:04.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.262+0000 7f7d8c8a0640 1 --2- 172.21.15.67:0/3691490480 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7d8815e2a0 0x7f7d880ce630 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7d84009d90 tx=0x7f7d8402be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25e60b93ac6c7d4f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:04.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.262+0000 7f7d7bfff640 1 -- 172.21.15.67:0/3691490480 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d84039070 con 0x7f7d8815e2a0 2024-03-24T00:40:04.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.262+0000 7f7d7bfff640 1 -- 172.21.15.67:0/3691490480 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d8402cbc0 con 0x7f7d8815e2a0 2024-03-24T00:40:04.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.262+0000 7f7d7bfff640 1 -- 172.21.15.67:0/3691490480 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d84034440 con 0x7f7d8815e2a0 2024-03-24T00:40:04.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.263+0000 7f7d8f305640 1 -- 172.21.15.67:0/3691490480 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7d8815e2a0 msgr2=0x7f7d880ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:04.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.263+0000 7f7d8f305640 1 --2- 172.21.15.67:0/3691490480 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7d8815e2a0 0x7f7d880ce630 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7d84009d90 tx=0x7f7d8402be10 comp rx=0 tx=0).stop 2024-03-24T00:40:04.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.263+0000 7f7d8f305640 1 -- 172.21.15.67:0/3691490480 shutdown_connections 2024-03-24T00:40:04.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.263+0000 7f7d8f305640 1 --2- 172.21.15.67:0/3691490480 >> v2:172.21.15.72:3300/0 conn(0x7f7d880d0ef0 0x7f7d88162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:04.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.263+0000 7f7d8f305640 1 --2- 172.21.15.67:0/3691490480 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7d8815e2a0 0x7f7d880ce630 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:04.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.263+0000 7f7d8f305640 1 -- 172.21.15.67:0/3691490480 >> 172.21.15.67:0/3691490480 conn(0x7f7d8806c810 msgr2=0x7f7d8806cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:04.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.263+0000 7f7d8f305640 1 -- 172.21.15.67:0/3691490480 shutdown_connections 2024-03-24T00:40:04.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.263+0000 7f7d8f305640 1 -- 172.21.15.67:0/3691490480 wait complete. 2024-03-24T00:40:04.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.264+0000 7f7d8f305640 1 Processor -- start 2024-03-24T00:40:04.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.264+0000 7f7d8f305640 1 -- start start 2024-03-24T00:40:04.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.265+0000 7f7d8f305640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d880d0ef0 0x7f7d8815b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:04.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.265+0000 7f7d8f305640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d8815dec0 0x7f7d881f3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:04.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.265+0000 7f7d8f305640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7d8815e2a0 0x7f7d880ac260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:04.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.265+0000 7f7d8f305640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d880d2730 con 0x7f7d8815dec0 2024-03-24T00:40:04.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.265+0000 7f7d8f305640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d880d25b0 con 0x7f7d880d0ef0 2024-03-24T00:40:04.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.265+0000 7f7d8f305640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d880d28b0 con 0x7f7d8815e2a0 2024-03-24T00:40:04.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.265+0000 7f7d8c8a0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d8815dec0 0x7f7d881f3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:04.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.265+0000 7f7d8c8a0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d8815dec0 0x7f7d881f3fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40800/0 (socket says 172.21.15.67:40800) 2024-03-24T00:40:04.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.265+0000 7f7d8c8a0640 1 -- 172.21.15.67:0/1021032453 learned_addr learned my addr 172.21.15.67:0/1021032453 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:04.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.265+0000 7f7d8d0a1640 1 --2- 172.21.15.67:0/1021032453 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d880d0ef0 0x7f7d8815b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:04.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.265+0000 7f7d8d8a2640 1 --2- 172.21.15.67:0/1021032453 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7d8815e2a0 0x7f7d880ac260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:04.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.266+0000 7f7d8c8a0640 1 -- 172.21.15.67:0/1021032453 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7d8815e2a0 msgr2=0x7f7d880ac260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:04.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.266+0000 7f7d8c8a0640 1 --2- 172.21.15.67:0/1021032453 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7d8815e2a0 0x7f7d880ac260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:04.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.266+0000 7f7d8c8a0640 1 -- 172.21.15.67:0/1021032453 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d880d0ef0 msgr2=0x7f7d8815b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:04.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.266+0000 7f7d8c8a0640 1 --2- 172.21.15.67:0/1021032453 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d880d0ef0 0x7f7d8815b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:04.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.266+0000 7f7d8c8a0640 1 -- 172.21.15.67:0/1021032453 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d880b6800 con 0x7f7d8815dec0 2024-03-24T00:40:04.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.266+0000 7f7d8c8a0640 1 --2- 172.21.15.67:0/1021032453 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d8815dec0 0x7f7d881f3fb0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f7d84009d90 tx=0x7f7d84004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:04.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.266+0000 7f7d79ffb640 1 -- 172.21.15.67:0/1021032453 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d84039050 con 0x7f7d8815dec0 2024-03-24T00:40:04.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.266+0000 7f7d79ffb640 1 -- 172.21.15.67:0/1021032453 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d84045070 con 0x7f7d8815dec0 2024-03-24T00:40:04.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.266+0000 7f7d8f305640 1 -- 172.21.15.67:0/1021032453 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d880b3940 con 0x7f7d8815dec0 2024-03-24T00:40:04.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.266+0000 7f7d79ffb640 1 -- 172.21.15.67:0/1021032453 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d84009450 con 0x7f7d8815dec0 2024-03-24T00:40:04.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.266+0000 7f7d8f305640 1 -- 172.21.15.67:0/1021032453 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d880b3ec0 con 0x7f7d8815dec0 2024-03-24T00:40:04.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.267+0000 7f7d8f305640 1 -- 172.21.15.67:0/1021032453 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d8815bb70 con 0x7f7d8815dec0 2024-03-24T00:40:04.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.267+0000 7f7d79ffb640 1 -- 172.21.15.67:0/1021032453 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7d8403e850 con 0x7f7d8815dec0 2024-03-24T00:40:04.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.268+0000 7f7d79ffb640 1 --2- 172.21.15.67:0/1021032453 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7d6c06bbd0 0x7f7d6c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:04.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.268+0000 7f7d79ffb640 1 -- 172.21.15.67:0/1021032453 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7d840ad880 con 0x7f7d8815dec0 2024-03-24T00:40:04.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.268+0000 7f7d8d0a1640 1 --2- 172.21.15.67:0/1021032453 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7d6c06bbd0 0x7f7d6c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:04.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.268+0000 7f7d8d0a1640 1 --2- 172.21.15.67:0/1021032453 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7d6c06bbd0 0x7f7d6c06e090 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c005880 tx=0x7f7d7c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:04.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.272+0000 7f7d79ffb640 1 -- 172.21.15.67:0/1021032453 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7d8815bb70 con 0x7f7d8815dec0 2024-03-24T00:40:04.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.458+0000 7f7d8f305640 1 -- 172.21.15.67:0/1021032453 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d8804f400 con 0x7f7d8815dec0 2024-03-24T00:40:04.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.459+0000 7f7d79ffb640 1 -- 172.21.15.67:0/1021032453 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) v1 ==== 74+0+16882 (secure 0 0 0) 0x7f7d8407be00 con 0x7f7d8815dec0 2024-03-24T00:40:04.461 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:40:04.461 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":39,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:40:02.323736+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:39:58.975995+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-24T00:39:59.653615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:40:04.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.461+0000 7f7d8f305640 1 -- 172.21.15.67:0/1021032453 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7d6c06bbd0 msgr2=0x7f7d6c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.461+0000 7f7d8f305640 1 --2- 172.21.15.67:0/1021032453 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7d6c06bbd0 0x7f7d6c06e090 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c005880 tx=0x7f7d7c009040 comp rx=0 tx=0).stop 2024-03-24T00:40:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.461+0000 7f7d8f305640 1 -- 172.21.15.67:0/1021032453 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d8815dec0 msgr2=0x7f7d881f3fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.461+0000 7f7d8f305640 1 --2- 172.21.15.67:0/1021032453 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d8815dec0 0x7f7d881f3fb0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f7d84009d90 tx=0x7f7d84004a40 comp rx=0 tx=0).stop 2024-03-24T00:40:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.461+0000 7f7d8f305640 1 -- 172.21.15.67:0/1021032453 shutdown_connections 2024-03-24T00:40:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.461+0000 7f7d8f305640 1 --2- 172.21.15.67:0/1021032453 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7d6c06bbd0 0x7f7d6c06e090 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.461+0000 7f7d8f305640 1 --2- 172.21.15.67:0/1021032453 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7d8815e2a0 0x7f7d880ac260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.461+0000 7f7d8f305640 1 --2- 172.21.15.67:0/1021032453 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d8815dec0 0x7f7d881f3fb0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.461+0000 7f7d8f305640 1 --2- 172.21.15.67:0/1021032453 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d880d0ef0 0x7f7d8815b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.461+0000 7f7d8f305640 1 -- 172.21.15.67:0/1021032453 >> 172.21.15.67:0/1021032453 conn(0x7f7d8806c810 msgr2=0x7f7d8815d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.461+0000 7f7d8f305640 1 -- 172.21.15.67:0/1021032453 shutdown_connections 2024-03-24T00:40:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.461+0000 7f7d8f305640 1 -- 172.21.15.67:0/1021032453 wait complete. 2024-03-24T00:40:04.473 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-24T00:40:04.473 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:40:04.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.573+0000 7f82a953b640 1 Processor -- start 2024-03-24T00:40:04.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.573+0000 7f82a953b640 1 -- start start 2024-03-24T00:40:04.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.574+0000 7f82a953b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f82a41445a0 0x7f82a40cee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:04.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.574+0000 7f82a953b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f82a40b48d0 0x7f82a4072550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:04.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.574+0000 7f82a953b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82a40b4d30 con 0x7f82a406ec40 2024-03-24T00:40:04.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.574+0000 7f82a953b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82a40ce990 con 0x7f82a41445a0 2024-03-24T00:40:04.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.574+0000 7f82a953b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82a40cddc0 con 0x7f82a40b48d0 2024-03-24T00:40:04.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.574+0000 7f82a2ffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f82a41445a0 0x7f82a40cee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:04.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.574+0000 7f82a27fc640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f82a40b48d0 0x7f82a4072550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:04.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.574+0000 7f82a37fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f82a406ec40 0x7f82a40b2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:48952/0 (socket says 172.21.15.67:48952) 2024-03-24T00:40:04.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.574+0000 7f82a2ffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f82a41445a0 0x7f82a40cee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:41106/0 (socket says 172.21.15.67:41106) 2024-03-24T00:40:04.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.574+0000 7f82a37fe640 1 -- 172.21.15.67:0/162346899 learned_addr learned my addr 172.21.15.67:0/162346899 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:40:04.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.574+0000 7f82a27fc640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f82a40b48d0 0x7f82a4072550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.67:48138/0 (socket says 172.21.15.67:48138) 2024-03-24T00:40:04.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.575+0000 7f82a2ffd640 1 -- 172.21.15.67:0/162346899 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f82a40b48d0 msgr2=0x7f82a4072550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:04.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.575+0000 7f82a2ffd640 1 --2- 172.21.15.67:0/162346899 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f82a40b48d0 0x7f82a4072550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:04.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.575+0000 7f82a2ffd640 1 -- 172.21.15.67:0/162346899 >> v1:172.21.15.72:6789/0 conn(0x7f82a406ec40 legacy=0x7f82a40b2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:04.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.575+0000 7f82a2ffd640 1 -- 172.21.15.67:0/162346899 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82a40ce060 con 0x7f82a41445a0 2024-03-24T00:40:04.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.575+0000 7f82a1ffb640 1 -- 172.21.15.67:0/162346899 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3579192531 0 0) 0x7f8294001490 con 0x7f82a406ec40 2024-03-24T00:40:04.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.575+0000 7f82a27fc640 1 --2- 172.21.15.67:0/162346899 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f82a40b48d0 0x7f82a4072550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:40:04.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.575+0000 7f82a2ffd640 1 --2- 172.21.15.67:0/162346899 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f82a41445a0 0x7f82a40cee20 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f829c0097a0 tx=0x7f829c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7dd0c1c2b18094ab server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:04.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.576+0000 7f82a1ffb640 1 -- 172.21.15.67:0/162346899 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f829c035020 con 0x7f82a41445a0 2024-03-24T00:40:04.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.576+0000 7f82a1ffb640 1 -- 172.21.15.67:0/162346899 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f829c031830 con 0x7f82a41445a0 2024-03-24T00:40:04.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.576+0000 7f82a1ffb640 1 -- 172.21.15.67:0/162346899 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f829c031c00 con 0x7f82a41445a0 2024-03-24T00:40:04.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.576+0000 7f82a953b640 1 -- 172.21.15.67:0/162346899 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f82a41445a0 msgr2=0x7f82a40cee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:04.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.576+0000 7f82a953b640 1 --2- 172.21.15.67:0/162346899 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f82a41445a0 0x7f82a40cee20 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f829c0097a0 tx=0x7f829c02b8d0 comp rx=0 tx=0).stop 2024-03-24T00:40:04.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.576+0000 7f82a953b640 1 -- 172.21.15.67:0/162346899 shutdown_connections 2024-03-24T00:40:04.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.576+0000 7f82a953b640 1 --2- 172.21.15.67:0/162346899 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f82a40b48d0 0x7f82a4072550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:04.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.576+0000 7f82a953b640 1 --2- 172.21.15.67:0/162346899 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f82a41445a0 0x7f82a40cee20 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:04.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.576+0000 7f82a953b640 1 -- 172.21.15.67:0/162346899 >> 172.21.15.67:0/162346899 conn(0x7f82a40cf9e0 msgr2=0x7f82a40cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:04.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.576+0000 7f82a953b640 1 -- 172.21.15.67:0/162346899 shutdown_connections 2024-03-24T00:40:04.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.576+0000 7f82a953b640 1 -- 172.21.15.67:0/162346899 wait complete. 2024-03-24T00:40:04.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.577+0000 7f82a953b640 1 Processor -- start 2024-03-24T00:40:04.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.578+0000 7f82a953b640 1 -- start start 2024-03-24T00:40:04.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.578+0000 7f82a953b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82a406ec40 0x7f82a413a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:04.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.578+0000 7f82a953b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f82a40b48d0 0x7f82a40ad520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:04.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.578+0000 7f82a953b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f82a41445a0 0x7f82a415a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:04.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.578+0000 7f82a953b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82a406f0d0 con 0x7f82a406ec40 2024-03-24T00:40:04.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.578+0000 7f82a953b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82a40ce620 con 0x7f82a41445a0 2024-03-24T00:40:04.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.578+0000 7f82a953b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82a406f250 con 0x7f82a40b48d0 2024-03-24T00:40:04.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.578+0000 7f82a2ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82a406ec40 0x7f82a413a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:04.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.578+0000 7f82a27fc640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f82a40b48d0 0x7f82a40ad520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:04.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.578+0000 7f82a2ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82a406ec40 0x7f82a413a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40812/0 (socket says 172.21.15.67:40812) 2024-03-24T00:40:04.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.578+0000 7f82a2ffd640 1 -- 172.21.15.67:0/2513960743 learned_addr learned my addr 172.21.15.67:0/2513960743 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:04.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.578+0000 7f82a37fe640 1 --2- 172.21.15.67:0/2513960743 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f82a41445a0 0x7f82a415a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:04.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.579+0000 7f82a37fe640 1 -- 172.21.15.67:0/2513960743 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f82a40b48d0 msgr2=0x7f82a40ad520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:04.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.579+0000 7f82a37fe640 1 --2- 172.21.15.67:0/2513960743 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f82a40b48d0 0x7f82a40ad520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:04.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.579+0000 7f82a37fe640 1 -- 172.21.15.67:0/2513960743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82a406ec40 msgr2=0x7f82a413a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:04.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.579+0000 7f82a37fe640 1 --2- 172.21.15.67:0/2513960743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82a406ec40 0x7f82a413a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:04.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.579+0000 7f82a37fe640 1 -- 172.21.15.67:0/2513960743 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82a413b920 con 0x7f82a41445a0 2024-03-24T00:40:04.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.579+0000 7f82a2ffd640 1 --2- 172.21.15.67:0/2513960743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82a406ec40 0x7f82a413a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:40:04.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.579+0000 7f82a37fe640 1 --2- 172.21.15.67:0/2513960743 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f82a41445a0 0x7f82a415a1f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f829400c6e0 tx=0x7f829400cba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:04.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.579+0000 7f829b7fe640 1 -- 172.21.15.67:0/2513960743 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8294018070 con 0x7f82a41445a0 2024-03-24T00:40:04.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.579+0000 7f829b7fe640 1 -- 172.21.15.67:0/2513960743 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8294004d60 con 0x7f82a41445a0 2024-03-24T00:40:04.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.579+0000 7f82a953b640 1 -- 172.21.15.67:0/2513960743 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82a415b160 con 0x7f82a41445a0 2024-03-24T00:40:04.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.579+0000 7f829b7fe640 1 -- 172.21.15.67:0/2513960743 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f829400a880 con 0x7f82a41445a0 2024-03-24T00:40:04.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.579+0000 7f82a953b640 1 -- 172.21.15.67:0/2513960743 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82a413b140 con 0x7f82a41445a0 2024-03-24T00:40:04.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.580+0000 7f82a953b640 1 -- 172.21.15.67:0/2513960743 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82a404f400 con 0x7f82a41445a0 2024-03-24T00:40:04.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.580+0000 7f829b7fe640 1 -- 172.21.15.67:0/2513960743 <== mon.1 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f829401b070 con 0x7f82a41445a0 2024-03-24T00:40:04.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.581+0000 7f829b7fe640 1 --2- 172.21.15.67:0/2513960743 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f828406bbd0 0x7f828406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:04.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.581+0000 7f829b7fe640 1 -- 172.21.15.67:0/2513960743 <== mon.1 v2:172.21.15.72:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8294089fd0 con 0x7f82a41445a0 2024-03-24T00:40:04.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.581+0000 7f82a2ffd640 1 --2- 172.21.15.67:0/2513960743 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f828406bbd0 0x7f828406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:04.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.581+0000 7f82a2ffd640 1 --2- 172.21.15.67:0/2513960743 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f828406bbd0 0x7f828406e090 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f829c02b910 tx=0x7f829c03c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:04.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.586+0000 7f829b7fe640 1 -- 172.21.15.67:0/2513960743 <== mon.1 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f82a413b920 con 0x7f82a41445a0 2024-03-24T00:40:04.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.765+0000 7f82a953b640 1 -- 172.21.15.67:0/2513960743 --> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f82a40b08a0 con 0x7f828406bbd0 2024-03-24T00:40:04.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.771+0000 7f829b7fe640 1 -- 172.21.15.67:0/2513960743 <== mgr.4105 v2:172.21.15.72:6832/2768292274 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240073 (secure 0 0 0) 0x7f829c0829e0 con 0x7f828406bbd0 2024-03-24T00:40:04.775 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:40:04.777 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-24T00:40:04.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.773+0000 7f82a953b640 1 -- 172.21.15.67:0/2513960743 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f828406bbd0 msgr2=0x7f828406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:04.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.773+0000 7f82a953b640 1 --2- 172.21.15.67:0/2513960743 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f828406bbd0 0x7f828406e090 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f829c02b910 tx=0x7f829c03c040 comp rx=0 tx=0).stop 2024-03-24T00:40:04.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.773+0000 7f82a953b640 1 -- 172.21.15.67:0/2513960743 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f82a41445a0 msgr2=0x7f82a415a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:04.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.773+0000 7f82a953b640 1 --2- 172.21.15.67:0/2513960743 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f82a41445a0 0x7f82a415a1f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f829400c6e0 tx=0x7f829400cba0 comp rx=0 tx=0).stop 2024-03-24T00:40:04.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.773+0000 7f82a953b640 1 -- 172.21.15.67:0/2513960743 shutdown_connections 2024-03-24T00:40:04.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.773+0000 7f82a953b640 1 --2- 172.21.15.67:0/2513960743 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f828406bbd0 0x7f828406e090 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:04.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.773+0000 7f82a953b640 1 --2- 172.21.15.67:0/2513960743 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f82a41445a0 0x7f82a415a1f0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:04.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.773+0000 7f82a953b640 1 --2- 172.21.15.67:0/2513960743 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f82a40b48d0 0x7f82a40ad520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:04.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.773+0000 7f82a953b640 1 --2- 172.21.15.67:0/2513960743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82a406ec40 0x7f82a413a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:04.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.773+0000 7f82a953b640 1 -- 172.21.15.67:0/2513960743 >> 172.21.15.67:0/2513960743 conn(0x7f82a40cf9e0 msgr2=0x7f82a40724e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:04.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.774+0000 7f82a953b640 1 -- 172.21.15.67:0/2513960743 shutdown_connections 2024-03-24T00:40:04.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.774+0000 7f82a953b640 1 -- 172.21.15.67:0/2513960743 wait complete. 2024-03-24T00:40:04.785 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":74,"stamp":"2024-03-24T00:40:03.338386+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220496,"kb_used_data":4296,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754224,"statfs":{"total":773094113280,"available":772868325376,"internally_reserved":0,"allocated":4399104,"data_stored":2544112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.020855"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.319288+0000","last_change":"2024-03-24T00:39:59.319288+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.319288+0000","last_undegraded":"2024-03-24T00:39:59.319288+0000","last_fullsized":"2024-03-24T00:39:59.319288+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.321016+0000","last_change":"2024-03-24T00:39:59.321016+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.321016+0000","last_undegraded":"2024-03-24T00:39:59.321016+0000","last_fullsized":"2024-03-24T00:39:59.321016+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.322566+0000","last_change":"2024-03-24T00:39:59.322566+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.322566+0000","last_undegraded":"2024-03-24T00:39:59.322566+0000","last_fullsized":"2024-03-24T00:39:59.322566+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.322415+0000","last_change":"2024-03-24T00:39:59.322415+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.322415+0000","last_undegraded":"2024-03-24T00:39:59.322415+0000","last_fullsized":"2024-03-24T00:39:59.322415+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.321245+0000","last_change":"2024-03-24T00:39:59.321245+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.321245+0000","last_undegraded":"2024-03-24T00:39:59.321245+0000","last_fullsized":"2024-03-24T00:39:59.321245+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.324337+0000","last_change":"2024-03-24T00:39:59.324337+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.324337+0000","last_undegraded":"2024-03-24T00:39:59.324337+0000","last_fullsized":"2024-03-24T00:39:59.324337+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.322008+0000","last_change":"2024-03-24T00:39:59.322008+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.322008+0000","last_undegraded":"2024-03-24T00:39:59.322008+0000","last_fullsized":"2024-03-24T00:39:59.322008+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.322442+0000","last_change":"2024-03-24T00:39:59.322442+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.322442+0000","last_undegraded":"2024-03-24T00:39:59.322442+0000","last_fullsized":"2024-03-24T00:39:59.322442+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.320112+0000","last_change":"2024-03-24T00:39:59.320112+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.320112+0000","last_undegraded":"2024-03-24T00:39:59.320112+0000","last_fullsized":"2024-03-24T00:39:59.320112+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.321166+0000","last_change":"2024-03-24T00:39:59.321166+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.321166+0000","last_undegraded":"2024-03-24T00:39:59.321166+0000","last_fullsized":"2024-03-24T00:39:59.321166+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.321389+0000","last_change":"2024-03-24T00:39:59.321389+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.321389+0000","last_undegraded":"2024-03-24T00:39:59.321389+0000","last_fullsized":"2024-03-24T00:39:59.321389+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.319941+0000","last_change":"2024-03-24T00:39:59.319941+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.319941+0000","last_undegraded":"2024-03-24T00:39:59.319941+0000","last_fullsized":"2024-03-24T00:39:59.319941+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.319991+0000","last_change":"2024-03-24T00:39:59.319991+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.319991+0000","last_undegraded":"2024-03-24T00:39:59.319991+0000","last_fullsized":"2024-03-24T00:39:59.319991+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.322129+0000","last_change":"2024-03-24T00:39:59.322129+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.322129+0000","last_undegraded":"2024-03-24T00:39:59.322129+0000","last_fullsized":"2024-03-24T00:39:59.322129+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.319263+0000","last_change":"2024-03-24T00:39:59.319263+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.319263+0000","last_undegraded":"2024-03-24T00:39:59.319263+0000","last_fullsized":"2024-03-24T00:39:59.319263+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.323849+0000","last_change":"2024-03-24T00:39:59.323849+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.323849+0000","last_undegraded":"2024-03-24T00:39:59.323849+0000","last_fullsized":"2024-03-24T00:39:59.323849+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.321729+0000","last_change":"2024-03-24T00:39:59.321729+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.321729+0000","last_undegraded":"2024-03-24T00:39:59.321729+0000","last_fullsized":"2024-03-24T00:39:59.321729+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.323123+0000","last_change":"2024-03-24T00:39:59.323123+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.323123+0000","last_undegraded":"2024-03-24T00:39:59.323123+0000","last_fullsized":"2024-03-24T00:39:59.323123+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.318625+0000","last_change":"2024-03-24T00:39:59.318625+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.318625+0000","last_undegraded":"2024-03-24T00:39:59.318625+0000","last_fullsized":"2024-03-24T00:39:59.318625+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.322385+0000","last_change":"2024-03-24T00:39:59.322385+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.322385+0000","last_undegraded":"2024-03-24T00:39:59.322385+0000","last_fullsized":"2024-03-24T00:39:59.322385+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.318612+0000","last_change":"2024-03-24T00:39:59.318612+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.318612+0000","last_undegraded":"2024-03-24T00:39:59.318612+0000","last_fullsized":"2024-03-24T00:39:59.318612+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.319061+0000","last_change":"2024-03-24T00:39:59.319061+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.319061+0000","last_undegraded":"2024-03-24T00:39:59.319061+0000","last_fullsized":"2024-03-24T00:39:59.319061+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.324013+0000","last_change":"2024-03-24T00:39:59.324013+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.324013+0000","last_undegraded":"2024-03-24T00:39:59.324013+0000","last_fullsized":"2024-03-24T00:39:59.324013+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.318609+0000","last_change":"2024-03-24T00:39:59.318609+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.318609+0000","last_undegraded":"2024-03-24T00:39:59.318609+0000","last_fullsized":"2024-03-24T00:39:59.318609+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.321199+0000","last_change":"2024-03-24T00:39:59.321199+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.321199+0000","last_undegraded":"2024-03-24T00:39:59.321199+0000","last_fullsized":"2024-03-24T00:39:59.321199+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.318620+0000","last_change":"2024-03-24T00:39:59.318620+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.318620+0000","last_undegraded":"2024-03-24T00:39:59.318620+0000","last_fullsized":"2024-03-24T00:39:59.318620+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.320582+0000","last_change":"2024-03-24T00:39:59.320582+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.320582+0000","last_undegraded":"2024-03-24T00:39:59.320582+0000","last_fullsized":"2024-03-24T00:39:59.320582+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.321049+0000","last_change":"2024-03-24T00:39:59.321049+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.321049+0000","last_undegraded":"2024-03-24T00:39:59.321049+0000","last_fullsized":"2024-03-24T00:39:59.321049+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.319260+0000","last_change":"2024-03-24T00:39:59.319260+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.319260+0000","last_undegraded":"2024-03-24T00:39:59.319260+0000","last_fullsized":"2024-03-24T00:39:59.319260+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.322134+0000","last_change":"2024-03-24T00:39:59.322134+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.322134+0000","last_undegraded":"2024-03-24T00:39:59.322134+0000","last_fullsized":"2024-03-24T00:39:59.322134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.319412+0000","last_change":"2024-03-24T00:39:59.319412+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.319412+0000","last_undegraded":"2024-03-24T00:39:59.319412+0000","last_fullsized":"2024-03-24T00:39:59.319412+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.323006+0000","last_change":"2024-03-24T00:39:59.323006+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.323006+0000","last_undegraded":"2024-03-24T00:39:59.323006+0000","last_fullsized":"2024-03-24T00:39:59.323006+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":199,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-24T00:39:59.324075+0000","last_change":"2024-03-24T00:38:40.953871+0000","last_active":"2024-03-24T00:39:59.324075+0000","last_peered":"2024-03-24T00:39:59.324075+0000","last_clean":"2024-03-24T00:39:59.324075+0000","last_became_active":"2024-03-24T00:38:40.952926+0000","last_became_peered":"2024-03-24T00:38:40.952926+0000","last_unstale":"2024-03-24T00:39:59.324075+0000","last_undegraded":"2024-03-24T00:39:59.324075+0000","last_fullsized":"2024-03-24T00:39:59.324075+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_clean_scrub_stamp":"2024-03-24T00:38:39.722194+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:05.397257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27996,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343844,"statfs":{"total":96636764160,"available":96608096256,"internally_reserved":0,"allocated":995328,"data_stored":760790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27416,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344424,"statfs":{"total":96636764160,"available":96608690176,"internally_reserved":0,"allocated":401408,"data_stored":170422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":401408,"data_stored":170422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":401408,"data_stored":170422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":401408,"data_stored":170422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":401408,"data_stored":170422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27412,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":401408,"data_stored":170422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574867,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343848,"statfs":{"total":96636764160,"available":96608100352,"internally_reserved":0,"allocated":995328,"data_stored":760790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:40:04.790 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:40:04.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.926+0000 7f51d7ec0640 1 Processor -- start 2024-03-24T00:40:04.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.926+0000 7f51d7ec0640 1 -- start start 2024-03-24T00:40:04.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.927+0000 7f51d7ec0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f51d015e1b0 0x7f51d00cfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:04.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.927+0000 7f51d7ec0640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f51d009a410 0x7f51d00cc880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:04.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.927+0000 7f51d7ec0640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51d00d0fc0 con 0x7f51d00b43a0 2024-03-24T00:40:04.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.927+0000 7f51d7ec0640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51d015e590 con 0x7f51d015e1b0 2024-03-24T00:40:04.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.927+0000 7f51d7ec0640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51d00d1820 con 0x7f51d009a410 2024-03-24T00:40:04.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.927+0000 7f51d5c5c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f51d015e1b0 0x7f51d00cfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:04.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.927+0000 7f51d545b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f51d00b43a0 0x7f51d0099d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42408/0 (socket says 172.21.15.67:42408) 2024-03-24T00:40:04.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.927+0000 7f51d545b640 1 -- 172.21.15.67:0/2859687992 learned_addr learned my addr 172.21.15.67:0/2859687992 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:40:04.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.927+0000 7f51d5c5c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f51d015e1b0 0x7f51d00cfe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40818/0 (socket says 172.21.15.67:40818) 2024-03-24T00:40:04.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.927+0000 7f51d645d640 1 --2- 172.21.15.67:0/2859687992 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f51d009a410 0x7f51d00cc880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:04.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.928+0000 7f51d5c5c640 1 -- 172.21.15.67:0/2859687992 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f51d009a410 msgr2=0x7f51d00cc880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:04.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.928+0000 7f51d5c5c640 1 --2- 172.21.15.67:0/2859687992 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f51d009a410 0x7f51d00cc880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:04.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.928+0000 7f51d5c5c640 1 -- 172.21.15.67:0/2859687992 >> v1:172.21.15.67:6789/0 conn(0x7f51d00b43a0 legacy=0x7f51d0099d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:04.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.928+0000 7f51d5c5c640 1 -- 172.21.15.67:0/2859687992 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d00d1260 con 0x7f51d015e1b0 2024-03-24T00:40:04.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.928+0000 7f51d5c5c640 1 --2- 172.21.15.67:0/2859687992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51d015e1b0 0x7f51d00cfe20 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f51c0009d90 tx=0x7f51c002bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e21835bf3e840043 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:04.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.928+0000 7f51d4c5a640 1 -- 172.21.15.67:0/2859687992 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c0039070 con 0x7f51d015e1b0 2024-03-24T00:40:04.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.928+0000 7f51d4c5a640 1 -- 172.21.15.67:0/2859687992 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51c002cb30 con 0x7f51d015e1b0 2024-03-24T00:40:04.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.928+0000 7f51d4c5a640 1 -- 172.21.15.67:0/2859687992 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c0034440 con 0x7f51d015e1b0 2024-03-24T00:40:04.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.929+0000 7f51d7ec0640 1 -- 172.21.15.67:0/2859687992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51d015e1b0 msgr2=0x7f51d00cfe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:04.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.929+0000 7f51d7ec0640 1 --2- 172.21.15.67:0/2859687992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51d015e1b0 0x7f51d00cfe20 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f51c0009d90 tx=0x7f51c002bd80 comp rx=0 tx=0).stop 2024-03-24T00:40:04.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.929+0000 7f51d7ec0640 1 -- 172.21.15.67:0/2859687992 shutdown_connections 2024-03-24T00:40:04.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.929+0000 7f51d7ec0640 1 --2- 172.21.15.67:0/2859687992 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f51d009a410 0x7f51d00cc880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:04.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.929+0000 7f51d7ec0640 1 --2- 172.21.15.67:0/2859687992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51d015e1b0 0x7f51d00cfe20 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:04.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.929+0000 7f51d7ec0640 1 -- 172.21.15.67:0/2859687992 >> 172.21.15.67:0/2859687992 conn(0x7f51d00b36d0 msgr2=0x7f51d00b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:04.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.929+0000 7f51d7ec0640 1 -- 172.21.15.67:0/2859687992 shutdown_connections 2024-03-24T00:40:04.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.929+0000 7f51d7ec0640 1 -- 172.21.15.67:0/2859687992 wait complete. 2024-03-24T00:40:04.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.930+0000 7f51d7ec0640 1 Processor -- start 2024-03-24T00:40:04.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.930+0000 7f51d7ec0640 1 -- start start 2024-03-24T00:40:04.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.931+0000 7f51d7ec0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51d009a410 0x7f51d01edc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:04.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.931+0000 7f51d7ec0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51d00b43a0 0x7f51d00b53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:04.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.931+0000 7f51d7ec0640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f51d015e1b0 0x7f51d00c7720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:04.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.931+0000 7f51d7ec0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d009a9a0 con 0x7f51d009a410 2024-03-24T00:40:04.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.931+0000 7f51d7ec0640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d00cc360 con 0x7f51d00b43a0 2024-03-24T00:40:04.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.931+0000 7f51d7ec0640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51d00ccdc0 con 0x7f51d015e1b0 2024-03-24T00:40:04.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.931+0000 7f51d5c5c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51d009a410 0x7f51d01edc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:04.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.931+0000 7f51d5c5c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51d009a410 0x7f51d01edc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40832/0 (socket says 172.21.15.67:40832) 2024-03-24T00:40:04.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.931+0000 7f51d5c5c640 1 -- 172.21.15.67:0/2278946072 learned_addr learned my addr 172.21.15.67:0/2278946072 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:04.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.931+0000 7f51d645d640 1 --2- 172.21.15.67:0/2278946072 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f51d015e1b0 0x7f51d00c7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:04.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.931+0000 7f51d545b640 1 --2- 172.21.15.67:0/2278946072 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51d00b43a0 0x7f51d00b53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:04.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.931+0000 7f51d5c5c640 1 -- 172.21.15.67:0/2278946072 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f51d015e1b0 msgr2=0x7f51d00c7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:04.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.931+0000 7f51d5c5c640 1 --2- 172.21.15.67:0/2278946072 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f51d015e1b0 0x7f51d00c7720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:04.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.931+0000 7f51d5c5c640 1 -- 172.21.15.67:0/2278946072 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51d00b43a0 msgr2=0x7f51d00b53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:04.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.931+0000 7f51d5c5c640 1 --2- 172.21.15.67:0/2278946072 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51d00b43a0 0x7f51d00b53e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:04.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.931+0000 7f51d5c5c640 1 -- 172.21.15.67:0/2278946072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d00b7610 con 0x7f51d009a410 2024-03-24T00:40:04.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.931+0000 7f51d545b640 1 --2- 172.21.15.67:0/2278946072 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51d00b43a0 0x7f51d00b53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:40:04.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.931+0000 7f51d5c5c640 1 --2- 172.21.15.67:0/2278946072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51d009a410 0x7f51d01edc10 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f51c0037040 tx=0x7f51c0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:04.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.932+0000 7f51ce7fc640 1 -- 172.21.15.67:0/2278946072 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c0039050 con 0x7f51d009a410 2024-03-24T00:40:04.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.932+0000 7f51ce7fc640 1 -- 172.21.15.67:0/2278946072 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51c0045070 con 0x7f51d009a410 2024-03-24T00:40:04.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.932+0000 7f51d7ec0640 1 -- 172.21.15.67:0/2278946072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51d00b4900 con 0x7f51d009a410 2024-03-24T00:40:04.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.932+0000 7f51ce7fc640 1 -- 172.21.15.67:0/2278946072 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c003e850 con 0x7f51d009a410 2024-03-24T00:40:04.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.932+0000 7f51d7ec0640 1 -- 172.21.15.67:0/2278946072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51d00b78f0 con 0x7f51d009a410 2024-03-24T00:40:04.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.933+0000 7f51d7ec0640 1 -- 172.21.15.67:0/2278946072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51d004f400 con 0x7f51d009a410 2024-03-24T00:40:04.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.933+0000 7f51ce7fc640 1 -- 172.21.15.67:0/2278946072 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f51c003e9f0 con 0x7f51d009a410 2024-03-24T00:40:04.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.933+0000 7f51ce7fc640 1 --2- 172.21.15.67:0/2278946072 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f51bc06bbd0 0x7f51bc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:04.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.934+0000 7f51ce7fc640 1 -- 172.21.15.67:0/2278946072 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f51c00ad940 con 0x7f51d009a410 2024-03-24T00:40:04.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.934+0000 7f51d545b640 1 --2- 172.21.15.67:0/2278946072 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f51bc06bbd0 0x7f51bc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:04.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.934+0000 7f51d545b640 1 --2- 172.21.15.67:0/2278946072 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f51bc06bbd0 0x7f51bc06e090 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f51d00c85f0 tx=0x7f51c4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:04.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:04.938+0000 7f51ce7fc640 1 -- 172.21.15.67:0/2278946072 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f51c007c3d0 con 0x7f51d009a410 2024-03-24T00:40:05.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.117+0000 7f51d7ec0640 1 -- 172.21.15.67:0/2278946072 --> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f51d006d720 con 0x7f51bc06bbd0 2024-03-24T00:40:05.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.123+0000 7f51ce7fc640 1 -- 172.21.15.67:0/2278946072 <== mgr.4105 v2:172.21.15.72:6832/2768292274 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240073 (secure 0 0 0) 0x7f51d006d720 con 0x7f51bc06bbd0 2024-03-24T00:40:05.126 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:40:05.127 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-24T00:40:05.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.126+0000 7f51d7ec0640 1 -- 172.21.15.67:0/2278946072 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f51bc06bbd0 msgr2=0x7f51bc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:05.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.126+0000 7f51d7ec0640 1 --2- 172.21.15.67:0/2278946072 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f51bc06bbd0 0x7f51bc06e090 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f51d00c85f0 tx=0x7f51c4009040 comp rx=0 tx=0).stop 2024-03-24T00:40:05.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.126+0000 7f51d7ec0640 1 -- 172.21.15.67:0/2278946072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51d009a410 msgr2=0x7f51d01edc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:05.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.126+0000 7f51d7ec0640 1 --2- 172.21.15.67:0/2278946072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51d009a410 0x7f51d01edc10 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f51c0037040 tx=0x7f51c0004a40 comp rx=0 tx=0).stop 2024-03-24T00:40:05.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.126+0000 7f51d7ec0640 1 -- 172.21.15.67:0/2278946072 shutdown_connections 2024-03-24T00:40:05.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.126+0000 7f51d7ec0640 1 --2- 172.21.15.67:0/2278946072 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f51bc06bbd0 0x7f51bc06e090 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:05.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.126+0000 7f51d7ec0640 1 --2- 172.21.15.67:0/2278946072 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f51d015e1b0 0x7f51d00c7720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:05.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.126+0000 7f51d7ec0640 1 --2- 172.21.15.67:0/2278946072 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51d00b43a0 0x7f51d00b53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:05.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.126+0000 7f51d7ec0640 1 --2- 172.21.15.67:0/2278946072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51d009a410 0x7f51d01edc10 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:05.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.126+0000 7f51d7ec0640 1 -- 172.21.15.67:0/2278946072 >> 172.21.15.67:0/2278946072 conn(0x7f51d00b36d0 msgr2=0x7f51d006ca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:05.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.126+0000 7f51d7ec0640 1 -- 172.21.15.67:0/2278946072 shutdown_connections 2024-03-24T00:40:05.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.126+0000 7f51d7ec0640 1 -- 172.21.15.67:0/2278946072 wait complete. 2024-03-24T00:40:05.137 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":74,"stamp":"2024-03-24T00:40:03.338386+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220496,"kb_used_data":4296,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754224,"statfs":{"total":773094113280,"available":772868325376,"internally_reserved":0,"allocated":4399104,"data_stored":2544112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.020855"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.319288+0000","last_change":"2024-03-24T00:39:59.319288+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.319288+0000","last_undegraded":"2024-03-24T00:39:59.319288+0000","last_fullsized":"2024-03-24T00:39:59.319288+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.321016+0000","last_change":"2024-03-24T00:39:59.321016+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.321016+0000","last_undegraded":"2024-03-24T00:39:59.321016+0000","last_fullsized":"2024-03-24T00:39:59.321016+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.322566+0000","last_change":"2024-03-24T00:39:59.322566+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.322566+0000","last_undegraded":"2024-03-24T00:39:59.322566+0000","last_fullsized":"2024-03-24T00:39:59.322566+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.322415+0000","last_change":"2024-03-24T00:39:59.322415+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.322415+0000","last_undegraded":"2024-03-24T00:39:59.322415+0000","last_fullsized":"2024-03-24T00:39:59.322415+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.321245+0000","last_change":"2024-03-24T00:39:59.321245+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.321245+0000","last_undegraded":"2024-03-24T00:39:59.321245+0000","last_fullsized":"2024-03-24T00:39:59.321245+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.324337+0000","last_change":"2024-03-24T00:39:59.324337+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.324337+0000","last_undegraded":"2024-03-24T00:39:59.324337+0000","last_fullsized":"2024-03-24T00:39:59.324337+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.322008+0000","last_change":"2024-03-24T00:39:59.322008+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.322008+0000","last_undegraded":"2024-03-24T00:39:59.322008+0000","last_fullsized":"2024-03-24T00:39:59.322008+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.322442+0000","last_change":"2024-03-24T00:39:59.322442+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.322442+0000","last_undegraded":"2024-03-24T00:39:59.322442+0000","last_fullsized":"2024-03-24T00:39:59.322442+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.320112+0000","last_change":"2024-03-24T00:39:59.320112+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.320112+0000","last_undegraded":"2024-03-24T00:39:59.320112+0000","last_fullsized":"2024-03-24T00:39:59.320112+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.321166+0000","last_change":"2024-03-24T00:39:59.321166+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.321166+0000","last_undegraded":"2024-03-24T00:39:59.321166+0000","last_fullsized":"2024-03-24T00:39:59.321166+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.321389+0000","last_change":"2024-03-24T00:39:59.321389+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.321389+0000","last_undegraded":"2024-03-24T00:39:59.321389+0000","last_fullsized":"2024-03-24T00:39:59.321389+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.319941+0000","last_change":"2024-03-24T00:39:59.319941+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.319941+0000","last_undegraded":"2024-03-24T00:39:59.319941+0000","last_fullsized":"2024-03-24T00:39:59.319941+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.319991+0000","last_change":"2024-03-24T00:39:59.319991+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.319991+0000","last_undegraded":"2024-03-24T00:39:59.319991+0000","last_fullsized":"2024-03-24T00:39:59.319991+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.322129+0000","last_change":"2024-03-24T00:39:59.322129+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.322129+0000","last_undegraded":"2024-03-24T00:39:59.322129+0000","last_fullsized":"2024-03-24T00:39:59.322129+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.319263+0000","last_change":"2024-03-24T00:39:59.319263+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.319263+0000","last_undegraded":"2024-03-24T00:39:59.319263+0000","last_fullsized":"2024-03-24T00:39:59.319263+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.323849+0000","last_change":"2024-03-24T00:39:59.323849+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.323849+0000","last_undegraded":"2024-03-24T00:39:59.323849+0000","last_fullsized":"2024-03-24T00:39:59.323849+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.321729+0000","last_change":"2024-03-24T00:39:59.321729+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.321729+0000","last_undegraded":"2024-03-24T00:39:59.321729+0000","last_fullsized":"2024-03-24T00:39:59.321729+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.323123+0000","last_change":"2024-03-24T00:39:59.323123+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.323123+0000","last_undegraded":"2024-03-24T00:39:59.323123+0000","last_fullsized":"2024-03-24T00:39:59.323123+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.318625+0000","last_change":"2024-03-24T00:39:59.318625+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.318625+0000","last_undegraded":"2024-03-24T00:39:59.318625+0000","last_fullsized":"2024-03-24T00:39:59.318625+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.322385+0000","last_change":"2024-03-24T00:39:59.322385+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.322385+0000","last_undegraded":"2024-03-24T00:39:59.322385+0000","last_fullsized":"2024-03-24T00:39:59.322385+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.318612+0000","last_change":"2024-03-24T00:39:59.318612+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.318612+0000","last_undegraded":"2024-03-24T00:39:59.318612+0000","last_fullsized":"2024-03-24T00:39:59.318612+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.319061+0000","last_change":"2024-03-24T00:39:59.319061+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.319061+0000","last_undegraded":"2024-03-24T00:39:59.319061+0000","last_fullsized":"2024-03-24T00:39:59.319061+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.324013+0000","last_change":"2024-03-24T00:39:59.324013+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.324013+0000","last_undegraded":"2024-03-24T00:39:59.324013+0000","last_fullsized":"2024-03-24T00:39:59.324013+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.318609+0000","last_change":"2024-03-24T00:39:59.318609+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.318609+0000","last_undegraded":"2024-03-24T00:39:59.318609+0000","last_fullsized":"2024-03-24T00:39:59.318609+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.321199+0000","last_change":"2024-03-24T00:39:59.321199+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.321199+0000","last_undegraded":"2024-03-24T00:39:59.321199+0000","last_fullsized":"2024-03-24T00:39:59.321199+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.318620+0000","last_change":"2024-03-24T00:39:59.318620+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.318620+0000","last_undegraded":"2024-03-24T00:39:59.318620+0000","last_fullsized":"2024-03-24T00:39:59.318620+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.320582+0000","last_change":"2024-03-24T00:39:59.320582+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.320582+0000","last_undegraded":"2024-03-24T00:39:59.320582+0000","last_fullsized":"2024-03-24T00:39:59.320582+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.321049+0000","last_change":"2024-03-24T00:39:59.321049+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.321049+0000","last_undegraded":"2024-03-24T00:39:59.321049+0000","last_fullsized":"2024-03-24T00:39:59.321049+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.319260+0000","last_change":"2024-03-24T00:39:59.319260+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.319260+0000","last_undegraded":"2024-03-24T00:39:59.319260+0000","last_fullsized":"2024-03-24T00:39:59.319260+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.322134+0000","last_change":"2024-03-24T00:39:59.322134+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.322134+0000","last_undegraded":"2024-03-24T00:39:59.322134+0000","last_fullsized":"2024-03-24T00:39:59.322134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.319412+0000","last_change":"2024-03-24T00:39:59.319412+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.319412+0000","last_undegraded":"2024-03-24T00:39:59.319412+0000","last_fullsized":"2024-03-24T00:39:59.319412+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:40:00.316527+0000","last_change":"2024-03-24T00:40:00.316527+0000","last_active":"2024-03-24T00:40:00.316527+0000","last_peered":"2024-03-24T00:40:00.316527+0000","last_clean":"2024-03-24T00:40:00.316527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:00.316527+0000","last_undegraded":"2024-03-24T00:40:00.316527+0000","last_fullsized":"2024-03-24T00:40:00.316527+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-24T00:39:59.323006+0000","last_change":"2024-03-24T00:39:59.323006+0000","last_active":"2024-03-24T00:39:59.314474+0000","last_peered":"2024-03-24T00:39:59.314474+0000","last_clean":"2024-03-24T00:39:59.314474+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:39:59.323006+0000","last_undegraded":"2024-03-24T00:39:59.323006+0000","last_fullsized":"2024-03-24T00:39:59.323006+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":199,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-24T00:39:59.324075+0000","last_change":"2024-03-24T00:38:40.953871+0000","last_active":"2024-03-24T00:39:59.324075+0000","last_peered":"2024-03-24T00:39:59.324075+0000","last_clean":"2024-03-24T00:39:59.324075+0000","last_became_active":"2024-03-24T00:38:40.952926+0000","last_became_peered":"2024-03-24T00:38:40.952926+0000","last_unstale":"2024-03-24T00:39:59.324075+0000","last_undegraded":"2024-03-24T00:39:59.324075+0000","last_fullsized":"2024-03-24T00:39:59.324075+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_clean_scrub_stamp":"2024-03-24T00:38:39.722194+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:05.397257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27996,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343844,"statfs":{"total":96636764160,"available":96608096256,"internally_reserved":0,"allocated":995328,"data_stored":760790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27416,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344424,"statfs":{"total":96636764160,"available":96608690176,"internally_reserved":0,"allocated":401408,"data_stored":170422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":401408,"data_stored":170422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":401408,"data_stored":170422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":401408,"data_stored":170422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":401408,"data_stored":170422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27412,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":401408,"data_stored":170422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574867,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343848,"statfs":{"total":96636764160,"available":96608100352,"internally_reserved":0,"allocated":995328,"data_stored":760790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:40:05.141 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:40:05.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.278+0000 7faca33cd640 1 Processor -- start 2024-03-24T00:40:05.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.278+0000 7faca33cd640 1 -- start start 2024-03-24T00:40:05.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.279+0000 7faca33cd640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fac9c0d1550 0x7fac9c06cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:05.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.279+0000 7faca33cd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fac9c0b2780 0x7fac9c0b2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:05.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.279+0000 7faca33cd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fac9c099d70 0x7fac9c09a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:05.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.279+0000 7faca33cd640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac9c06d420 con 0x7fac9c099d70 2024-03-24T00:40:05.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.279+0000 7faca33cd640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac9c06ca50 con 0x7fac9c0b2780 2024-03-24T00:40:05.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.279+0000 7faca33cd640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac9c0d1930 con 0x7fac9c0d1550 2024-03-24T00:40:05.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.279+0000 7faca196a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fac9c099d70 0x7fac9c09a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:05.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.279+0000 7faca196a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fac9c099d70 0x7fac9c09a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40848/0 (socket says 172.21.15.67:40848) 2024-03-24T00:40:05.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.279+0000 7faca196a640 1 -- 172.21.15.67:0/1552329949 learned_addr learned my addr 172.21.15.67:0/1552329949 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:05.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.279+0000 7faca0968640 1 --2- 172.21.15.67:0/1552329949 >> v2:172.21.15.72:3300/0 conn(0x7fac9c0b2780 0x7fac9c0b2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:05.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.279+0000 7faca1169640 1 --2- 172.21.15.67:0/1552329949 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fac9c0d1550 0x7fac9c06cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:05.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.279+0000 7faca196a640 1 -- 172.21.15.67:0/1552329949 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fac9c0d1550 msgr2=0x7fac9c06cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:05.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.279+0000 7faca196a640 1 --2- 172.21.15.67:0/1552329949 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fac9c0d1550 0x7fac9c06cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:05.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.279+0000 7faca196a640 1 -- 172.21.15.67:0/1552329949 >> v2:172.21.15.72:3300/0 conn(0x7fac9c0b2780 msgr2=0x7fac9c0b2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:05.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.279+0000 7faca196a640 1 --2- 172.21.15.67:0/1552329949 >> v2:172.21.15.72:3300/0 conn(0x7fac9c0b2780 0x7fac9c0b2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:05.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.279+0000 7faca196a640 1 -- 172.21.15.67:0/1552329949 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac9c0b1f60 con 0x7fac9c099d70 2024-03-24T00:40:05.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.279+0000 7faca0968640 1 --2- 172.21.15.67:0/1552329949 >> v2:172.21.15.72:3300/0 conn(0x7fac9c0b2780 0x7fac9c0b2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:40:05.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.279+0000 7faca1169640 1 --2- 172.21.15.67:0/1552329949 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fac9c0d1550 0x7fac9c06cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:40:05.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.280+0000 7faca196a640 1 --2- 172.21.15.67:0/1552329949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac9c099d70 0x7fac9c09a200 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fac9c073260 tx=0x7fac9402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=211f11af79084edb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:05.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.280+0000 7fac93fff640 1 -- 172.21.15.67:0/1552329949 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac9402ca70 con 0x7fac9c099d70 2024-03-24T00:40:05.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.280+0000 7fac93fff640 1 -- 172.21.15.67:0/1552329949 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac9402cc10 con 0x7fac9c099d70 2024-03-24T00:40:05.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.280+0000 7fac93fff640 1 -- 172.21.15.67:0/1552329949 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac940343f0 con 0x7fac9c099d70 2024-03-24T00:40:05.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.281+0000 7faca33cd640 1 -- 172.21.15.67:0/1552329949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac9c099d70 msgr2=0x7fac9c09a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:05.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.281+0000 7faca33cd640 1 --2- 172.21.15.67:0/1552329949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac9c099d70 0x7fac9c09a200 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fac9c073260 tx=0x7fac9402bed0 comp rx=0 tx=0).stop 2024-03-24T00:40:05.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.281+0000 7faca33cd640 1 -- 172.21.15.67:0/1552329949 shutdown_connections 2024-03-24T00:40:05.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.281+0000 7faca33cd640 1 --2- 172.21.15.67:0/1552329949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac9c099d70 0x7fac9c09a200 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:05.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.281+0000 7faca33cd640 1 --2- 172.21.15.67:0/1552329949 >> v2:172.21.15.72:3300/0 conn(0x7fac9c0b2780 0x7fac9c0b2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:05.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.281+0000 7faca33cd640 1 --2- 172.21.15.67:0/1552329949 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fac9c0d1550 0x7fac9c06cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:05.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.281+0000 7faca33cd640 1 -- 172.21.15.67:0/1552329949 >> 172.21.15.67:0/1552329949 conn(0x7fac9c0b36e0 msgr2=0x7fac9c0b3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:05.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.281+0000 7faca33cd640 1 -- 172.21.15.67:0/1552329949 shutdown_connections 2024-03-24T00:40:05.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.281+0000 7faca33cd640 1 -- 172.21.15.67:0/1552329949 wait complete. 2024-03-24T00:40:05.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.282+0000 7faca33cd640 1 Processor -- start 2024-03-24T00:40:05.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.282+0000 7faca33cd640 1 -- start start 2024-03-24T00:40:05.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.282+0000 7faca33cd640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fac9c099d70 0x7fac9c0b6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:05.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.283+0000 7faca33cd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac9c0b2780 0x7fac9c160160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:05.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.283+0000 7faca33cd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac9c0d1550 0x7fac9c1606a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:05.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.283+0000 7faca33cd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac9c06c400 con 0x7fac9c0d1550 2024-03-24T00:40:05.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.283+0000 7faca33cd640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac9c0b2520 con 0x7fac9c0b2780 2024-03-24T00:40:05.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.283+0000 7faca33cd640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac9c06c6e0 con 0x7fac9c099d70 2024-03-24T00:40:05.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.283+0000 7faca196a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac9c0d1550 0x7fac9c1606a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:05.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.283+0000 7faca196a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac9c0d1550 0x7fac9c1606a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40850/0 (socket says 172.21.15.67:40850) 2024-03-24T00:40:05.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.283+0000 7faca196a640 1 -- 172.21.15.67:0/4183590851 learned_addr learned my addr 172.21.15.67:0/4183590851 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:05.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.283+0000 7faca0968640 1 --2- 172.21.15.67:0/4183590851 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac9c0b2780 0x7fac9c160160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:05.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.283+0000 7faca1169640 1 --2- 172.21.15.67:0/4183590851 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fac9c099d70 0x7fac9c0b6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:05.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.283+0000 7faca196a640 1 -- 172.21.15.67:0/4183590851 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fac9c099d70 msgr2=0x7fac9c0b6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:05.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.283+0000 7faca196a640 1 --2- 172.21.15.67:0/4183590851 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fac9c099d70 0x7fac9c0b6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:05.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.283+0000 7faca196a640 1 -- 172.21.15.67:0/4183590851 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac9c0b2780 msgr2=0x7fac9c160160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:05.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.283+0000 7faca196a640 1 --2- 172.21.15.67:0/4183590851 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac9c0b2780 0x7fac9c160160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:05.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.283+0000 7faca196a640 1 -- 172.21.15.67:0/4183590851 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac9c14cbe0 con 0x7fac9c0d1550 2024-03-24T00:40:05.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.283+0000 7faca196a640 1 --2- 172.21.15.67:0/4183590851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac9c0d1550 0x7fac9c1606a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fac94036040 tx=0x7fac94004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:05.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.283+0000 7fac91ffb640 1 -- 172.21.15.67:0/4183590851 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac94043020 con 0x7fac9c0d1550 2024-03-24T00:40:05.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.283+0000 7fac91ffb640 1 -- 172.21.15.67:0/4183590851 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac9403e450 con 0x7fac9c0d1550 2024-03-24T00:40:05.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.283+0000 7faca33cd640 1 -- 172.21.15.67:0/4183590851 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac9c0cb570 con 0x7fac9c0d1550 2024-03-24T00:40:05.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.283+0000 7fac91ffb640 1 -- 172.21.15.67:0/4183590851 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac94009510 con 0x7fac9c0d1550 2024-03-24T00:40:05.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.284+0000 7faca33cd640 1 -- 172.21.15.67:0/4183590851 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac9c0b4b40 con 0x7fac9c0d1550 2024-03-24T00:40:05.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.285+0000 7faca33cd640 1 -- 172.21.15.67:0/4183590851 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac9c04f400 con 0x7fac9c0d1550 2024-03-24T00:40:05.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.285+0000 7fac91ffb640 1 -- 172.21.15.67:0/4183590851 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fac94046020 con 0x7fac9c0d1550 2024-03-24T00:40:05.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.285+0000 7fac91ffb640 1 --2- 172.21.15.67:0/4183590851 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fac8006bbd0 0x7fac8006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:05.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.286+0000 7fac91ffb640 1 -- 172.21.15.67:0/4183590851 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fac940ad100 con 0x7fac9c0d1550 2024-03-24T00:40:05.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.286+0000 7faca1169640 1 --2- 172.21.15.67:0/4183590851 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fac8006bbd0 0x7fac8006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:05.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.286+0000 7faca1169640 1 --2- 172.21.15.67:0/4183590851 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fac8006bbd0 0x7fac8006e090 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fac98009560 tx=0x7fac98009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:05.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.290+0000 7fac91ffb640 1 -- 172.21.15.67:0/4183590851 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fac9404d030 con 0x7fac9c0d1550 2024-03-24T00:40:05.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.469+0000 7faca33cd640 1 -- 172.21.15.67:0/4183590851 --> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fac9c0b4510 con 0x7fac8006bbd0 2024-03-24T00:40:05.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.476+0000 7fac91ffb640 1 -- 172.21.15.67:0/4183590851 <== mgr.4105 v2:172.21.15.72:6832/2768292274 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251958 (secure 0 0 0) 0x7fac9c0b4510 con 0x7fac8006bbd0 2024-03-24T00:40:05.479 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:40:05.480 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-24T00:40:05.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.479+0000 7faca33cd640 1 -- 172.21.15.67:0/4183590851 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fac8006bbd0 msgr2=0x7fac8006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:05.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.479+0000 7faca33cd640 1 --2- 172.21.15.67:0/4183590851 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fac8006bbd0 0x7fac8006e090 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fac98009560 tx=0x7fac98009190 comp rx=0 tx=0).stop 2024-03-24T00:40:05.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.479+0000 7faca33cd640 1 -- 172.21.15.67:0/4183590851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac9c0d1550 msgr2=0x7fac9c1606a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:05.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.479+0000 7faca33cd640 1 --2- 172.21.15.67:0/4183590851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac9c0d1550 0x7fac9c1606a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fac94036040 tx=0x7fac94004a40 comp rx=0 tx=0).stop 2024-03-24T00:40:05.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.479+0000 7faca33cd640 1 -- 172.21.15.67:0/4183590851 shutdown_connections 2024-03-24T00:40:05.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.479+0000 7faca33cd640 1 --2- 172.21.15.67:0/4183590851 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fac8006bbd0 0x7fac8006e090 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:05.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.479+0000 7faca33cd640 1 --2- 172.21.15.67:0/4183590851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac9c0d1550 0x7fac9c1606a0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:05.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.479+0000 7faca33cd640 1 --2- 172.21.15.67:0/4183590851 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac9c0b2780 0x7fac9c160160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:05.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.479+0000 7faca33cd640 1 --2- 172.21.15.67:0/4183590851 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fac9c099d70 0x7fac9c0b6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:05.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.479+0000 7faca33cd640 1 -- 172.21.15.67:0/4183590851 >> 172.21.15.67:0/4183590851 conn(0x7fac9c0b36e0 msgr2=0x7fac9c161d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:05.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.479+0000 7faca33cd640 1 -- 172.21.15.67:0/4183590851 shutdown_connections 2024-03-24T00:40:05.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.479+0000 7faca33cd640 1 -- 172.21.15.67:0/4183590851 wait complete. 2024-03-24T00:40:05.490 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":75,"stamp":"2024-03-24T00:40:05.338666+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":174,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220948,"kb_used_data":4744,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753772,"statfs":{"total":773094113280,"available":772867862528,"internally_reserved":0,"allocated":4857856,"data_stored":2762812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000985"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.326693+0000","last_change":"2024-03-24T00:40:01.346232+0000","last_active":"2024-03-24T00:40:02.326693+0000","last_peered":"2024-03-24T00:40:02.326693+0000","last_clean":"2024-03-24T00:40:02.326693+0000","last_became_active":"2024-03-24T00:40:01.345850+0000","last_became_peered":"2024-03-24T00:40:01.345850+0000","last_unstale":"2024-03-24T00:40:02.326693+0000","last_undegraded":"2024-03-24T00:40:02.326693+0000","last_fullsized":"2024-03-24T00:40:02.326693+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:38:18.834926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.326462+0000","last_change":"2024-03-24T00:40:01.334855+0000","last_active":"2024-03-24T00:40:02.326462+0000","last_peered":"2024-03-24T00:40:02.326462+0000","last_clean":"2024-03-24T00:40:02.326462+0000","last_became_active":"2024-03-24T00:40:01.334560+0000","last_became_peered":"2024-03-24T00:40:01.334560+0000","last_unstale":"2024-03-24T00:40:02.326462+0000","last_undegraded":"2024-03-24T00:40:02.326462+0000","last_fullsized":"2024-03-24T00:40:02.326462+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:52:56.182730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.684957+0000","last_change":"2024-03-24T00:40:01.356272+0000","last_active":"2024-03-24T00:40:02.684957+0000","last_peered":"2024-03-24T00:40:02.684957+0000","last_clean":"2024-03-24T00:40:02.684957+0000","last_became_active":"2024-03-24T00:40:01.355995+0000","last_became_peered":"2024-03-24T00:40:01.355995+0000","last_unstale":"2024-03-24T00:40:02.684957+0000","last_undegraded":"2024-03-24T00:40:02.684957+0000","last_fullsized":"2024-03-24T00:40:02.684957+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:51.086346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.329660+0000","last_change":"2024-03-24T00:40:01.349541+0000","last_active":"2024-03-24T00:40:02.329660+0000","last_peered":"2024-03-24T00:40:02.329660+0000","last_clean":"2024-03-24T00:40:02.329660+0000","last_became_active":"2024-03-24T00:40:01.349072+0000","last_became_peered":"2024-03-24T00:40:01.349072+0000","last_unstale":"2024-03-24T00:40:02.329660+0000","last_undegraded":"2024-03-24T00:40:02.329660+0000","last_fullsized":"2024-03-24T00:40:02.329660+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:06:45.735287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.329550+0000","last_change":"2024-03-24T00:40:01.349628+0000","last_active":"2024-03-24T00:40:02.329550+0000","last_peered":"2024-03-24T00:40:02.329550+0000","last_clean":"2024-03-24T00:40:02.329550+0000","last_became_active":"2024-03-24T00:40:01.349371+0000","last_became_peered":"2024-03-24T00:40:01.349371+0000","last_unstale":"2024-03-24T00:40:02.329550+0000","last_undegraded":"2024-03-24T00:40:02.329550+0000","last_fullsized":"2024-03-24T00:40:02.329550+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:09:01.114410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685148+0000","last_change":"2024-03-24T00:40:01.345171+0000","last_active":"2024-03-24T00:40:02.685148+0000","last_peered":"2024-03-24T00:40:02.685148+0000","last_clean":"2024-03-24T00:40:02.685148+0000","last_became_active":"2024-03-24T00:40:01.344909+0000","last_became_peered":"2024-03-24T00:40:01.344909+0000","last_unstale":"2024-03-24T00:40:02.685148+0000","last_undegraded":"2024-03-24T00:40:02.685148+0000","last_fullsized":"2024-03-24T00:40:02.685148+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:11:12.174614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.326916+0000","last_change":"2024-03-24T00:40:01.350464+0000","last_active":"2024-03-24T00:40:02.326916+0000","last_peered":"2024-03-24T00:40:02.326916+0000","last_clean":"2024-03-24T00:40:02.326916+0000","last_became_active":"2024-03-24T00:40:01.350212+0000","last_became_peered":"2024-03-24T00:40:01.350212+0000","last_unstale":"2024-03-24T00:40:02.326916+0000","last_undegraded":"2024-03-24T00:40:02.326916+0000","last_fullsized":"2024-03-24T00:40:02.326916+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:44:25.432200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.326251+0000","last_change":"2024-03-24T00:40:01.356804+0000","last_active":"2024-03-24T00:40:02.326251+0000","last_peered":"2024-03-24T00:40:02.326251+0000","last_clean":"2024-03-24T00:40:02.326251+0000","last_became_active":"2024-03-24T00:40:01.356513+0000","last_became_peered":"2024-03-24T00:40:01.356513+0000","last_unstale":"2024-03-24T00:40:02.326251+0000","last_undegraded":"2024-03-24T00:40:02.326251+0000","last_fullsized":"2024-03-24T00:40:02.326251+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:09:15.867422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685100+0000","last_change":"2024-03-24T00:40:01.356194+0000","last_active":"2024-03-24T00:40:02.685100+0000","last_peered":"2024-03-24T00:40:02.685100+0000","last_clean":"2024-03-24T00:40:02.685100+0000","last_became_active":"2024-03-24T00:40:01.355912+0000","last_became_peered":"2024-03-24T00:40:01.355912+0000","last_unstale":"2024-03-24T00:40:02.685100+0000","last_undegraded":"2024-03-24T00:40:02.685100+0000","last_fullsized":"2024-03-24T00:40:02.685100+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:23:33.705793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.328221+0000","last_change":"2024-03-24T00:40:01.352016+0000","last_active":"2024-03-24T00:40:02.328221+0000","last_peered":"2024-03-24T00:40:02.328221+0000","last_clean":"2024-03-24T00:40:02.328221+0000","last_became_active":"2024-03-24T00:40:01.351496+0000","last_became_peered":"2024-03-24T00:40:01.351496+0000","last_unstale":"2024-03-24T00:40:02.328221+0000","last_undegraded":"2024-03-24T00:40:02.328221+0000","last_fullsized":"2024-03-24T00:40:02.328221+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:54:11.377947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.327897+0000","last_change":"2024-03-24T00:40:01.354643+0000","last_active":"2024-03-24T00:40:02.327897+0000","last_peered":"2024-03-24T00:40:02.327897+0000","last_clean":"2024-03-24T00:40:02.327897+0000","last_became_active":"2024-03-24T00:40:01.354393+0000","last_became_peered":"2024-03-24T00:40:01.354393+0000","last_unstale":"2024-03-24T00:40:02.327897+0000","last_undegraded":"2024-03-24T00:40:02.327897+0000","last_fullsized":"2024-03-24T00:40:02.327897+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:37:43.101486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.684935+0000","last_change":"2024-03-24T00:40:01.342932+0000","last_active":"2024-03-24T00:40:02.684935+0000","last_peered":"2024-03-24T00:40:02.684935+0000","last_clean":"2024-03-24T00:40:02.684935+0000","last_became_active":"2024-03-24T00:40:01.342615+0000","last_became_peered":"2024-03-24T00:40:01.342615+0000","last_unstale":"2024-03-24T00:40:02.684935+0000","last_undegraded":"2024-03-24T00:40:02.684935+0000","last_fullsized":"2024-03-24T00:40:02.684935+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:11:12.174614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.327186+0000","last_change":"2024-03-24T00:40:01.352457+0000","last_active":"2024-03-24T00:40:02.327186+0000","last_peered":"2024-03-24T00:40:02.327186+0000","last_clean":"2024-03-24T00:40:02.327186+0000","last_became_active":"2024-03-24T00:40:01.352201+0000","last_became_peered":"2024-03-24T00:40:01.352201+0000","last_unstale":"2024-03-24T00:40:02.327186+0000","last_undegraded":"2024-03-24T00:40:02.327186+0000","last_fullsized":"2024-03-24T00:40:02.327186+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:52:01.752611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.327238+0000","last_change":"2024-03-24T00:40:01.356693+0000","last_active":"2024-03-24T00:40:02.327238+0000","last_peered":"2024-03-24T00:40:02.327238+0000","last_clean":"2024-03-24T00:40:02.327238+0000","last_became_active":"2024-03-24T00:40:01.356306+0000","last_became_peered":"2024-03-24T00:40:01.356306+0000","last_unstale":"2024-03-24T00:40:02.327238+0000","last_undegraded":"2024-03-24T00:40:02.327238+0000","last_fullsized":"2024-03-24T00:40:02.327238+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:25:54.080003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.327257+0000","last_change":"2024-03-24T00:40:01.354392+0000","last_active":"2024-03-24T00:40:02.327257+0000","last_peered":"2024-03-24T00:40:02.327257+0000","last_clean":"2024-03-24T00:40:02.327257+0000","last_became_active":"2024-03-24T00:40:01.354111+0000","last_became_peered":"2024-03-24T00:40:01.354111+0000","last_unstale":"2024-03-24T00:40:02.327257+0000","last_undegraded":"2024-03-24T00:40:02.327257+0000","last_fullsized":"2024-03-24T00:40:02.327257+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:30:01.709222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685016+0000","last_change":"2024-03-24T00:40:01.354777+0000","last_active":"2024-03-24T00:40:02.685016+0000","last_peered":"2024-03-24T00:40:02.685016+0000","last_clean":"2024-03-24T00:40:02.685016+0000","last_became_active":"2024-03-24T00:40:01.354507+0000","last_became_peered":"2024-03-24T00:40:01.354507+0000","last_unstale":"2024-03-24T00:40:02.685016+0000","last_undegraded":"2024-03-24T00:40:02.685016+0000","last_fullsized":"2024-03-24T00:40:02.685016+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:11:12.174614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685571+0000","last_change":"2024-03-24T00:40:01.356643+0000","last_active":"2024-03-24T00:40:02.685571+0000","last_peered":"2024-03-24T00:40:02.685571+0000","last_clean":"2024-03-24T00:40:02.685571+0000","last_became_active":"2024-03-24T00:40:01.355610+0000","last_became_peered":"2024-03-24T00:40:01.355610+0000","last_unstale":"2024-03-24T00:40:02.685571+0000","last_undegraded":"2024-03-24T00:40:02.685571+0000","last_fullsized":"2024-03-24T00:40:02.685571+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:31:30.760690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685511+0000","last_change":"2024-03-24T00:40:01.349254+0000","last_active":"2024-03-24T00:40:02.685511+0000","last_peered":"2024-03-24T00:40:02.685511+0000","last_clean":"2024-03-24T00:40:02.685511+0000","last_became_active":"2024-03-24T00:40:01.348965+0000","last_became_peered":"2024-03-24T00:40:01.348965+0000","last_unstale":"2024-03-24T00:40:02.685511+0000","last_undegraded":"2024-03-24T00:40:02.685511+0000","last_fullsized":"2024-03-24T00:40:02.685511+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:31:38.754178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.329759+0000","last_change":"2024-03-24T00:40:01.346529+0000","last_active":"2024-03-24T00:40:02.329759+0000","last_peered":"2024-03-24T00:40:02.329759+0000","last_clean":"2024-03-24T00:40:02.329759+0000","last_became_active":"2024-03-24T00:40:01.346262+0000","last_became_peered":"2024-03-24T00:40:01.346262+0000","last_unstale":"2024-03-24T00:40:02.329759+0000","last_undegraded":"2024-03-24T00:40:02.329759+0000","last_fullsized":"2024-03-24T00:40:02.329759+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:40:42.216343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685222+0000","last_change":"2024-03-24T00:40:01.348561+0000","last_active":"2024-03-24T00:40:02.685222+0000","last_peered":"2024-03-24T00:40:02.685222+0000","last_clean":"2024-03-24T00:40:02.685222+0000","last_became_active":"2024-03-24T00:40:01.348304+0000","last_became_peered":"2024-03-24T00:40:01.348304+0000","last_unstale":"2024-03-24T00:40:02.685222+0000","last_undegraded":"2024-03-24T00:40:02.685222+0000","last_fullsized":"2024-03-24T00:40:02.685222+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:30:16.536487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.327350+0000","last_change":"2024-03-24T00:40:01.353646+0000","last_active":"2024-03-24T00:40:02.327350+0000","last_peered":"2024-03-24T00:40:02.327350+0000","last_clean":"2024-03-24T00:40:02.327350+0000","last_became_active":"2024-03-24T00:40:01.352758+0000","last_became_peered":"2024-03-24T00:40:01.352758+0000","last_unstale":"2024-03-24T00:40:02.327350+0000","last_undegraded":"2024-03-24T00:40:02.327350+0000","last_fullsized":"2024-03-24T00:40:02.327350+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:27:49.593844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.327514+0000","last_change":"2024-03-24T00:40:01.354943+0000","last_active":"2024-03-24T00:40:02.327514+0000","last_peered":"2024-03-24T00:40:02.327514+0000","last_clean":"2024-03-24T00:40:02.327514+0000","last_became_active":"2024-03-24T00:40:01.354452+0000","last_became_peered":"2024-03-24T00:40:01.354452+0000","last_unstale":"2024-03-24T00:40:02.327514+0000","last_undegraded":"2024-03-24T00:40:02.327514+0000","last_fullsized":"2024-03-24T00:40:02.327514+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:51.086346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.684979+0000","last_change":"2024-03-24T00:40:01.345157+0000","last_active":"2024-03-24T00:40:02.684979+0000","last_peered":"2024-03-24T00:40:02.684979+0000","last_clean":"2024-03-24T00:40:02.684979+0000","last_became_active":"2024-03-24T00:40:01.344404+0000","last_became_peered":"2024-03-24T00:40:01.344404+0000","last_unstale":"2024-03-24T00:40:02.684979+0000","last_undegraded":"2024-03-24T00:40:02.684979+0000","last_fullsized":"2024-03-24T00:40:02.684979+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:29:16.074361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.329561+0000","last_change":"2024-03-24T00:40:01.337319+0000","last_active":"2024-03-24T00:40:02.329561+0000","last_peered":"2024-03-24T00:40:02.329561+0000","last_clean":"2024-03-24T00:40:02.329561+0000","last_became_active":"2024-03-24T00:40:01.336989+0000","last_became_peered":"2024-03-24T00:40:01.336989+0000","last_unstale":"2024-03-24T00:40:02.329561+0000","last_undegraded":"2024-03-24T00:40:02.329561+0000","last_fullsized":"2024-03-24T00:40:02.329561+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:11:33.082463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685282+0000","last_change":"2024-03-24T00:40:01.349701+0000","last_active":"2024-03-24T00:40:02.685282+0000","last_peered":"2024-03-24T00:40:02.685282+0000","last_clean":"2024-03-24T00:40:02.685282+0000","last_became_active":"2024-03-24T00:40:01.349448+0000","last_became_peered":"2024-03-24T00:40:01.349448+0000","last_unstale":"2024-03-24T00:40:02.685282+0000","last_undegraded":"2024-03-24T00:40:02.685282+0000","last_fullsized":"2024-03-24T00:40:02.685282+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:04:47.673221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685589+0000","last_change":"2024-03-24T00:40:01.356229+0000","last_active":"2024-03-24T00:40:02.685589+0000","last_peered":"2024-03-24T00:40:02.685589+0000","last_clean":"2024-03-24T00:40:02.685589+0000","last_became_active":"2024-03-24T00:40:01.355953+0000","last_became_peered":"2024-03-24T00:40:01.355953+0000","last_unstale":"2024-03-24T00:40:02.685589+0000","last_undegraded":"2024-03-24T00:40:02.685589+0000","last_fullsized":"2024-03-24T00:40:02.685589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:23:33.705793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685303+0000","last_change":"2024-03-24T00:40:01.338189+0000","last_active":"2024-03-24T00:40:02.685303+0000","last_peered":"2024-03-24T00:40:02.685303+0000","last_clean":"2024-03-24T00:40:02.685303+0000","last_became_active":"2024-03-24T00:40:01.337910+0000","last_became_peered":"2024-03-24T00:40:01.337910+0000","last_unstale":"2024-03-24T00:40:02.685303+0000","last_undegraded":"2024-03-24T00:40:02.685303+0000","last_fullsized":"2024-03-24T00:40:02.685303+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:18:05.164294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.327590+0000","last_change":"2024-03-24T00:40:01.353130+0000","last_active":"2024-03-24T00:40:02.327590+0000","last_peered":"2024-03-24T00:40:02.327590+0000","last_clean":"2024-03-24T00:40:02.327590+0000","last_became_active":"2024-03-24T00:40:01.352852+0000","last_became_peered":"2024-03-24T00:40:01.352852+0000","last_unstale":"2024-03-24T00:40:02.327590+0000","last_undegraded":"2024-03-24T00:40:02.327590+0000","last_fullsized":"2024-03-24T00:40:02.327590+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:25:10.580085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685299+0000","last_change":"2024-03-24T00:40:01.345986+0000","last_active":"2024-03-24T00:40:02.685299+0000","last_peered":"2024-03-24T00:40:02.685299+0000","last_clean":"2024-03-24T00:40:02.685299+0000","last_became_active":"2024-03-24T00:40:01.345705+0000","last_became_peered":"2024-03-24T00:40:01.345705+0000","last_unstale":"2024-03-24T00:40:02.685299+0000","last_undegraded":"2024-03-24T00:40:02.685299+0000","last_fullsized":"2024-03-24T00:40:02.685299+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:28:36.656275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685595+0000","last_change":"2024-03-24T00:40:01.331420+0000","last_active":"2024-03-24T00:40:02.685595+0000","last_peered":"2024-03-24T00:40:02.685595+0000","last_clean":"2024-03-24T00:40:02.685595+0000","last_became_active":"2024-03-24T00:40:01.331128+0000","last_became_peered":"2024-03-24T00:40:01.331128+0000","last_unstale":"2024-03-24T00:40:02.685595+0000","last_undegraded":"2024-03-24T00:40:02.685595+0000","last_fullsized":"2024-03-24T00:40:02.685595+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:48:50.540849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685015+0000","last_change":"2024-03-24T00:40:01.344038+0000","last_active":"2024-03-24T00:40:02.685015+0000","last_peered":"2024-03-24T00:40:02.685015+0000","last_clean":"2024-03-24T00:40:02.685015+0000","last_became_active":"2024-03-24T00:40:01.343741+0000","last_became_peered":"2024-03-24T00:40:01.343741+0000","last_unstale":"2024-03-24T00:40:02.685015+0000","last_undegraded":"2024-03-24T00:40:02.685015+0000","last_fullsized":"2024-03-24T00:40:02.685015+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:25:54.080003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.684875+0000","last_change":"2024-03-24T00:40:01.344268+0000","last_active":"2024-03-24T00:40:02.684875+0000","last_peered":"2024-03-24T00:40:02.684875+0000","last_clean":"2024-03-24T00:40:02.684875+0000","last_became_active":"2024-03-24T00:40:01.343972+0000","last_became_peered":"2024-03-24T00:40:01.343972+0000","last_unstale":"2024-03-24T00:40:02.684875+0000","last_undegraded":"2024-03-24T00:40:02.684875+0000","last_fullsized":"2024-03-24T00:40:02.684875+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:25:54.080003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685342+0000","last_change":"2024-03-24T00:40:01.355848+0000","last_active":"2024-03-24T00:40:02.685342+0000","last_peered":"2024-03-24T00:40:02.685342+0000","last_clean":"2024-03-24T00:40:02.685342+0000","last_became_active":"2024-03-24T00:40:01.355593+0000","last_became_peered":"2024-03-24T00:40:01.355593+0000","last_unstale":"2024-03-24T00:40:02.685342+0000","last_undegraded":"2024-03-24T00:40:02.685342+0000","last_fullsized":"2024-03-24T00:40:02.685342+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:34:52.148336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"38'4","reported_seq":18,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685329+0000","last_change":"2024-03-24T00:40:00.346759+0000","last_active":"2024-03-24T00:40:02.685329+0000","last_peered":"2024-03-24T00:40:02.685329+0000","last_clean":"2024-03-24T00:40:02.685329+0000","last_became_active":"2024-03-24T00:40:00.346366+0000","last_became_peered":"2024-03-24T00:40:00.346366+0000","last_unstale":"2024-03-24T00:40:02.685329+0000","last_undegraded":"2024-03-24T00:40:02.685329+0000","last_fullsized":"2024-03-24T00:40:02.685329+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:09:00.112357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685511+0000","last_change":"2024-03-24T00:40:01.356499+0000","last_active":"2024-03-24T00:40:02.685511+0000","last_peered":"2024-03-24T00:40:02.685511+0000","last_clean":"2024-03-24T00:40:02.685511+0000","last_became_active":"2024-03-24T00:40:01.356224+0000","last_became_peered":"2024-03-24T00:40:01.356224+0000","last_unstale":"2024-03-24T00:40:02.685511+0000","last_undegraded":"2024-03-24T00:40:02.685511+0000","last_fullsized":"2024-03-24T00:40:02.685511+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:47:44.580768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.684876+0000","last_change":"2024-03-24T00:40:00.338722+0000","last_active":"2024-03-24T00:40:02.684876+0000","last_peered":"2024-03-24T00:40:02.684876+0000","last_clean":"2024-03-24T00:40:02.684876+0000","last_became_active":"2024-03-24T00:40:00.338449+0000","last_became_peered":"2024-03-24T00:40:00.338449+0000","last_unstale":"2024-03-24T00:40:02.684876+0000","last_undegraded":"2024-03-24T00:40:02.684876+0000","last_fullsized":"2024-03-24T00:40:02.684876+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:50:26.015558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.328071+0000","last_change":"2024-03-24T00:40:01.352156+0000","last_active":"2024-03-24T00:40:02.328071+0000","last_peered":"2024-03-24T00:40:02.328071+0000","last_clean":"2024-03-24T00:40:02.328071+0000","last_became_active":"2024-03-24T00:40:01.351605+0000","last_became_peered":"2024-03-24T00:40:01.351605+0000","last_unstale":"2024-03-24T00:40:02.328071+0000","last_undegraded":"2024-03-24T00:40:02.328071+0000","last_fullsized":"2024-03-24T00:40:02.328071+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:23:33.705793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685584+0000","last_change":"2024-03-24T00:40:00.345390+0000","last_active":"2024-03-24T00:40:02.685584+0000","last_peered":"2024-03-24T00:40:02.685584+0000","last_clean":"2024-03-24T00:40:02.685584+0000","last_became_active":"2024-03-24T00:40:00.345117+0000","last_became_peered":"2024-03-24T00:40:00.345117+0000","last_unstale":"2024-03-24T00:40:02.685584+0000","last_undegraded":"2024-03-24T00:40:02.685584+0000","last_fullsized":"2024-03-24T00:40:02.685584+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:04:46.671168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.326684+0000","last_change":"2024-03-24T00:40:01.339489+0000","last_active":"2024-03-24T00:40:02.326684+0000","last_peered":"2024-03-24T00:40:02.326684+0000","last_clean":"2024-03-24T00:40:02.326684+0000","last_became_active":"2024-03-24T00:40:01.338052+0000","last_became_peered":"2024-03-24T00:40:01.338052+0000","last_unstale":"2024-03-24T00:40:02.326684+0000","last_undegraded":"2024-03-24T00:40:02.326684+0000","last_fullsized":"2024-03-24T00:40:02.326684+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:56:17.522174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.327735+0000","last_change":"2024-03-24T00:40:00.348606+0000","last_active":"2024-03-24T00:40:02.327735+0000","last_peered":"2024-03-24T00:40:02.327735+0000","last_clean":"2024-03-24T00:40:02.327735+0000","last_became_active":"2024-03-24T00:40:00.348365+0000","last_became_peered":"2024-03-24T00:40:00.348365+0000","last_unstale":"2024-03-24T00:40:02.327735+0000","last_undegraded":"2024-03-24T00:40:02.327735+0000","last_fullsized":"2024-03-24T00:40:02.327735+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:30:15.534434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.684685+0000","last_change":"2024-03-24T00:40:01.351153+0000","last_active":"2024-03-24T00:40:02.684685+0000","last_peered":"2024-03-24T00:40:02.684685+0000","last_clean":"2024-03-24T00:40:02.684685+0000","last_became_active":"2024-03-24T00:40:01.350878+0000","last_became_peered":"2024-03-24T00:40:01.350878+0000","last_unstale":"2024-03-24T00:40:02.684685+0000","last_undegraded":"2024-03-24T00:40:02.684685+0000","last_fullsized":"2024-03-24T00:40:02.684685+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:29:16.074361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.327399+0000","last_change":"2024-03-24T00:40:00.342836+0000","last_active":"2024-03-24T00:40:02.327399+0000","last_peered":"2024-03-24T00:40:02.327399+0000","last_clean":"2024-03-24T00:40:02.327399+0000","last_became_active":"2024-03-24T00:40:00.342550+0000","last_became_peered":"2024-03-24T00:40:00.342550+0000","last_unstale":"2024-03-24T00:40:02.327399+0000","last_undegraded":"2024-03-24T00:40:02.327399+0000","last_fullsized":"2024-03-24T00:40:02.327399+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:30:15.534434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.327069+0000","last_change":"2024-03-24T00:40:01.353462+0000","last_active":"2024-03-24T00:40:02.327069+0000","last_peered":"2024-03-24T00:40:02.327069+0000","last_clean":"2024-03-24T00:40:02.327069+0000","last_became_active":"2024-03-24T00:40:01.353183+0000","last_became_peered":"2024-03-24T00:40:01.353183+0000","last_unstale":"2024-03-24T00:40:02.327069+0000","last_undegraded":"2024-03-24T00:40:02.327069+0000","last_fullsized":"2024-03-24T00:40:02.327069+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:31:30.760690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.684915+0000","last_change":"2024-03-24T00:40:00.346329+0000","last_active":"2024-03-24T00:40:02.684915+0000","last_peered":"2024-03-24T00:40:02.684915+0000","last_clean":"2024-03-24T00:40:02.684915+0000","last_became_active":"2024-03-24T00:40:00.346051+0000","last_became_peered":"2024-03-24T00:40:00.346051+0000","last_unstale":"2024-03-24T00:40:02.684915+0000","last_undegraded":"2024-03-24T00:40:02.684915+0000","last_fullsized":"2024-03-24T00:40:02.684915+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:06:44.733235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.327708+0000","last_change":"2024-03-24T00:40:01.357619+0000","last_active":"2024-03-24T00:40:02.327708+0000","last_peered":"2024-03-24T00:40:02.327708+0000","last_clean":"2024-03-24T00:40:02.327708+0000","last_became_active":"2024-03-24T00:40:01.356944+0000","last_became_peered":"2024-03-24T00:40:01.356944+0000","last_unstale":"2024-03-24T00:40:02.327708+0000","last_undegraded":"2024-03-24T00:40:02.327708+0000","last_fullsized":"2024-03-24T00:40:02.327708+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:34:52.148336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685257+0000","last_change":"2024-03-24T00:40:00.341907+0000","last_active":"2024-03-24T00:40:02.685257+0000","last_peered":"2024-03-24T00:40:02.685257+0000","last_clean":"2024-03-24T00:40:02.685257+0000","last_became_active":"2024-03-24T00:40:00.341643+0000","last_became_peered":"2024-03-24T00:40:00.341643+0000","last_unstale":"2024-03-24T00:40:02.685257+0000","last_undegraded":"2024-03-24T00:40:02.685257+0000","last_fullsized":"2024-03-24T00:40:02.685257+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:30:15.534434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.326459+0000","last_change":"2024-03-24T00:40:01.344561+0000","last_active":"2024-03-24T00:40:02.326459+0000","last_peered":"2024-03-24T00:40:02.326459+0000","last_clean":"2024-03-24T00:40:02.326459+0000","last_became_active":"2024-03-24T00:40:01.343777+0000","last_became_peered":"2024-03-24T00:40:01.343777+0000","last_unstale":"2024-03-24T00:40:02.326459+0000","last_undegraded":"2024-03-24T00:40:02.326459+0000","last_fullsized":"2024-03-24T00:40:02.326459+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:09:57.767219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.326786+0000","last_change":"2024-03-24T00:40:00.346231+0000","last_active":"2024-03-24T00:40:02.326786+0000","last_peered":"2024-03-24T00:40:02.326786+0000","last_clean":"2024-03-24T00:40:02.326786+0000","last_became_active":"2024-03-24T00:40:00.345948+0000","last_became_peered":"2024-03-24T00:40:00.345948+0000","last_unstale":"2024-03-24T00:40:02.326786+0000","last_undegraded":"2024-03-24T00:40:02.326786+0000","last_fullsized":"2024-03-24T00:40:02.326786+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:40:41.214290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.329742+0000","last_change":"2024-03-24T00:40:01.349328+0000","last_active":"2024-03-24T00:40:02.329742+0000","last_peered":"2024-03-24T00:40:02.329742+0000","last_clean":"2024-03-24T00:40:02.329742+0000","last_became_active":"2024-03-24T00:40:01.348743+0000","last_became_peered":"2024-03-24T00:40:01.348743+0000","last_unstale":"2024-03-24T00:40:02.329742+0000","last_undegraded":"2024-03-24T00:40:02.329742+0000","last_fullsized":"2024-03-24T00:40:02.329742+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:27:39.492378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.326712+0000","last_change":"2024-03-24T00:40:00.343185+0000","last_active":"2024-03-24T00:40:02.326712+0000","last_peered":"2024-03-24T00:40:02.326712+0000","last_clean":"2024-03-24T00:40:02.326712+0000","last_became_active":"2024-03-24T00:40:00.342931+0000","last_became_peered":"2024-03-24T00:40:00.342931+0000","last_unstale":"2024-03-24T00:40:02.326712+0000","last_undegraded":"2024-03-24T00:40:02.326712+0000","last_fullsized":"2024-03-24T00:40:02.326712+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:23:32.703740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685526+0000","last_change":"2024-03-24T00:40:01.344896+0000","last_active":"2024-03-24T00:40:02.685526+0000","last_peered":"2024-03-24T00:40:02.685526+0000","last_clean":"2024-03-24T00:40:02.685526+0000","last_became_active":"2024-03-24T00:40:01.344627+0000","last_became_peered":"2024-03-24T00:40:01.344627+0000","last_unstale":"2024-03-24T00:40:02.685526+0000","last_undegraded":"2024-03-24T00:40:02.685526+0000","last_fullsized":"2024-03-24T00:40:02.685526+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:44:25.432200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.328239+0000","last_change":"2024-03-24T00:40:00.337581+0000","last_active":"2024-03-24T00:40:02.328239+0000","last_peered":"2024-03-24T00:40:02.328239+0000","last_clean":"2024-03-24T00:40:02.328239+0000","last_became_active":"2024-03-24T00:40:00.336273+0000","last_became_peered":"2024-03-24T00:40:00.336273+0000","last_unstale":"2024-03-24T00:40:02.328239+0000","last_undegraded":"2024-03-24T00:40:02.328239+0000","last_fullsized":"2024-03-24T00:40:02.328239+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:04:46.671168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.684874+0000","last_change":"2024-03-24T00:40:01.346266+0000","last_active":"2024-03-24T00:40:02.684874+0000","last_peered":"2024-03-24T00:40:02.684874+0000","last_clean":"2024-03-24T00:40:02.684874+0000","last_became_active":"2024-03-24T00:40:01.345647+0000","last_became_peered":"2024-03-24T00:40:01.345647+0000","last_unstale":"2024-03-24T00:40:02.684874+0000","last_undegraded":"2024-03-24T00:40:02.684874+0000","last_fullsized":"2024-03-24T00:40:02.684874+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:25:54.080003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"38'3","reported_seq":17,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.327966+0000","last_change":"2024-03-24T00:40:00.337995+0000","last_active":"2024-03-24T00:40:02.327966+0000","last_peered":"2024-03-24T00:40:02.327966+0000","last_clean":"2024-03-24T00:40:02.327966+0000","last_became_active":"2024-03-24T00:40:00.336199+0000","last_became_peered":"2024-03-24T00:40:00.336199+0000","last_unstale":"2024-03-24T00:40:02.327966+0000","last_undegraded":"2024-03-24T00:40:02.327966+0000","last_fullsized":"2024-03-24T00:40:02.327966+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:05:49.943101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.327123+0000","last_change":"2024-03-24T00:40:01.354792+0000","last_active":"2024-03-24T00:40:02.327123+0000","last_peered":"2024-03-24T00:40:02.327123+0000","last_clean":"2024-03-24T00:40:02.327123+0000","last_became_active":"2024-03-24T00:40:01.354518+0000","last_became_peered":"2024-03-24T00:40:01.354518+0000","last_unstale":"2024-03-24T00:40:02.327123+0000","last_undegraded":"2024-03-24T00:40:02.327123+0000","last_fullsized":"2024-03-24T00:40:02.327123+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:29:16.074361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"38'6","reported_seq":20,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.326390+0000","last_change":"2024-03-24T00:40:00.338540+0000","last_active":"2024-03-24T00:40:02.326390+0000","last_peered":"2024-03-24T00:40:02.326390+0000","last_clean":"2024-03-24T00:40:02.326390+0000","last_became_active":"2024-03-24T00:40:00.338230+0000","last_became_peered":"2024-03-24T00:40:00.338230+0000","last_unstale":"2024-03-24T00:40:02.326390+0000","last_undegraded":"2024-03-24T00:40:02.326390+0000","last_fullsized":"2024-03-24T00:40:02.326390+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:34:51.146284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685105+0000","last_change":"2024-03-24T00:40:01.350796+0000","last_active":"2024-03-24T00:40:02.685105+0000","last_peered":"2024-03-24T00:40:02.685105+0000","last_clean":"2024-03-24T00:40:02.685105+0000","last_became_active":"2024-03-24T00:40:01.350249+0000","last_became_peered":"2024-03-24T00:40:01.350249+0000","last_unstale":"2024-03-24T00:40:02.685105+0000","last_undegraded":"2024-03-24T00:40:02.685105+0000","last_fullsized":"2024-03-24T00:40:02.685105+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:44:25.432200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.326548+0000","last_change":"2024-03-24T00:40:00.344040+0000","last_active":"2024-03-24T00:40:02.326548+0000","last_peered":"2024-03-24T00:40:02.326548+0000","last_clean":"2024-03-24T00:40:02.326548+0000","last_became_active":"2024-03-24T00:40:00.343790+0000","last_became_peered":"2024-03-24T00:40:00.343790+0000","last_unstale":"2024-03-24T00:40:02.326548+0000","last_undegraded":"2024-03-24T00:40:02.326548+0000","last_fullsized":"2024-03-24T00:40:02.326548+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:47:43.578715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.326251+0000","last_change":"2024-03-24T00:40:01.346676+0000","last_active":"2024-03-24T00:40:02.326251+0000","last_peered":"2024-03-24T00:40:02.326251+0000","last_clean":"2024-03-24T00:40:02.326251+0000","last_became_active":"2024-03-24T00:40:01.346398+0000","last_became_peered":"2024-03-24T00:40:01.346398+0000","last_unstale":"2024-03-24T00:40:02.326251+0000","last_undegraded":"2024-03-24T00:40:02.326251+0000","last_fullsized":"2024-03-24T00:40:02.326251+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:42:29.481647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685205+0000","last_change":"2024-03-24T00:40:00.343886+0000","last_active":"2024-03-24T00:40:02.685205+0000","last_peered":"2024-03-24T00:40:02.685205+0000","last_clean":"2024-03-24T00:40:02.685205+0000","last_became_active":"2024-03-24T00:40:00.343624+0000","last_became_peered":"2024-03-24T00:40:00.343624+0000","last_unstale":"2024-03-24T00:40:02.685205+0000","last_undegraded":"2024-03-24T00:40:02.685205+0000","last_fullsized":"2024-03-24T00:40:02.685205+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:05:49.943101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685089+0000","last_change":"2024-03-24T00:40:01.343214+0000","last_active":"2024-03-24T00:40:02.685089+0000","last_peered":"2024-03-24T00:40:02.685089+0000","last_clean":"2024-03-24T00:40:02.685089+0000","last_became_active":"2024-03-24T00:40:01.342847+0000","last_became_peered":"2024-03-24T00:40:01.342847+0000","last_unstale":"2024-03-24T00:40:02.685089+0000","last_undegraded":"2024-03-24T00:40:02.685089+0000","last_fullsized":"2024-03-24T00:40:02.685089+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:51.086346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.684866+0000","last_change":"2024-03-24T00:40:00.345927+0000","last_active":"2024-03-24T00:40:02.684866+0000","last_peered":"2024-03-24T00:40:02.684866+0000","last_clean":"2024-03-24T00:40:02.684866+0000","last_became_active":"2024-03-24T00:40:00.345681+0000","last_became_peered":"2024-03-24T00:40:00.345681+0000","last_unstale":"2024-03-24T00:40:02.684866+0000","last_undegraded":"2024-03-24T00:40:02.684866+0000","last_fullsized":"2024-03-24T00:40:02.684866+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:27:53.783517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.326490+0000","last_change":"2024-03-24T00:40:01.347982+0000","last_active":"2024-03-24T00:40:02.326490+0000","last_peered":"2024-03-24T00:40:02.326490+0000","last_clean":"2024-03-24T00:40:02.326490+0000","last_became_active":"2024-03-24T00:40:01.347571+0000","last_became_peered":"2024-03-24T00:40:01.347571+0000","last_unstale":"2024-03-24T00:40:02.326490+0000","last_undegraded":"2024-03-24T00:40:02.326490+0000","last_fullsized":"2024-03-24T00:40:02.326490+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:07:41.182445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"38'8","reported_seq":22,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.684865+0000","last_change":"2024-03-24T00:40:00.345571+0000","last_active":"2024-03-24T00:40:02.684865+0000","last_peered":"2024-03-24T00:40:02.684865+0000","last_clean":"2024-03-24T00:40:02.684865+0000","last_became_active":"2024-03-24T00:40:00.345288+0000","last_became_peered":"2024-03-24T00:40:00.345288+0000","last_unstale":"2024-03-24T00:40:02.684865+0000","last_undegraded":"2024-03-24T00:40:02.684865+0000","last_fullsized":"2024-03-24T00:40:02.684865+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:11:32.080410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.326458+0000","last_change":"2024-03-24T00:40:01.331287+0000","last_active":"2024-03-24T00:40:02.326458+0000","last_peered":"2024-03-24T00:40:02.326458+0000","last_clean":"2024-03-24T00:40:02.326458+0000","last_became_active":"2024-03-24T00:40:01.330903+0000","last_became_peered":"2024-03-24T00:40:01.330903+0000","last_unstale":"2024-03-24T00:40:02.326458+0000","last_undegraded":"2024-03-24T00:40:02.326458+0000","last_fullsized":"2024-03-24T00:40:02.326458+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:41:50.553798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.328239+0000","last_change":"2024-03-24T00:40:00.345924+0000","last_active":"2024-03-24T00:40:02.328239+0000","last_peered":"2024-03-24T00:40:02.328239+0000","last_clean":"2024-03-24T00:40:02.328239+0000","last_became_active":"2024-03-24T00:40:00.345636+0000","last_became_peered":"2024-03-24T00:40:00.345636+0000","last_unstale":"2024-03-24T00:40:02.328239+0000","last_undegraded":"2024-03-24T00:40:02.328239+0000","last_fullsized":"2024-03-24T00:40:02.328239+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:28:35.654223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.328035+0000","last_change":"2024-03-24T00:40:01.353040+0000","last_active":"2024-03-24T00:40:02.328035+0000","last_peered":"2024-03-24T00:40:02.328035+0000","last_clean":"2024-03-24T00:40:02.328035+0000","last_became_active":"2024-03-24T00:40:01.352658+0000","last_became_peered":"2024-03-24T00:40:01.352658+0000","last_unstale":"2024-03-24T00:40:02.328035+0000","last_undegraded":"2024-03-24T00:40:02.328035+0000","last_fullsized":"2024-03-24T00:40:02.328035+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:15:44.062886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.328077+0000","last_change":"2024-03-24T00:40:00.345548+0000","last_active":"2024-03-24T00:40:02.328077+0000","last_peered":"2024-03-24T00:40:02.328077+0000","last_clean":"2024-03-24T00:40:02.328077+0000","last_became_active":"2024-03-24T00:40:00.345250+0000","last_became_peered":"2024-03-24T00:40:00.345250+0000","last_unstale":"2024-03-24T00:40:02.328077+0000","last_undegraded":"2024-03-24T00:40:02.328077+0000","last_fullsized":"2024-03-24T00:40:02.328077+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:05:49.943101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.328021+0000","last_change":"2024-03-24T00:40:01.351854+0000","last_active":"2024-03-24T00:40:02.328021+0000","last_peered":"2024-03-24T00:40:02.328021+0000","last_clean":"2024-03-24T00:40:02.328021+0000","last_became_active":"2024-03-24T00:40:01.351282+0000","last_became_peered":"2024-03-24T00:40:01.351282+0000","last_unstale":"2024-03-24T00:40:02.328021+0000","last_undegraded":"2024-03-24T00:40:02.328021+0000","last_fullsized":"2024-03-24T00:40:02.328021+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:47:44.580768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.329740+0000","last_change":"2024-03-24T00:40:00.335501+0000","last_active":"2024-03-24T00:40:02.329740+0000","last_peered":"2024-03-24T00:40:02.329740+0000","last_clean":"2024-03-24T00:40:02.329740+0000","last_became_active":"2024-03-24T00:40:00.335233+0000","last_became_peered":"2024-03-24T00:40:00.335233+0000","last_unstale":"2024-03-24T00:40:02.329740+0000","last_undegraded":"2024-03-24T00:40:02.329740+0000","last_fullsized":"2024-03-24T00:40:02.329740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:21:02.704842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.329818+0000","last_change":"2024-03-24T00:40:01.334780+0000","last_active":"2024-03-24T00:40:02.329818+0000","last_peered":"2024-03-24T00:40:02.329818+0000","last_clean":"2024-03-24T00:40:02.329818+0000","last_became_active":"2024-03-24T00:40:01.333949+0000","last_became_peered":"2024-03-24T00:40:01.333949+0000","last_unstale":"2024-03-24T00:40:02.329818+0000","last_undegraded":"2024-03-24T00:40:02.329818+0000","last_fullsized":"2024-03-24T00:40:02.329818+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:50:27.017610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685335+0000","last_change":"2024-03-24T00:40:00.345775+0000","last_active":"2024-03-24T00:40:02.685335+0000","last_peered":"2024-03-24T00:40:02.685335+0000","last_clean":"2024-03-24T00:40:02.685335+0000","last_became_active":"2024-03-24T00:40:00.345458+0000","last_became_peered":"2024-03-24T00:40:00.345458+0000","last_unstale":"2024-03-24T00:40:02.685335+0000","last_undegraded":"2024-03-24T00:40:02.685335+0000","last_fullsized":"2024-03-24T00:40:02.685335+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:28:35.654223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685361+0000","last_change":"2024-03-24T00:40:01.345730+0000","last_active":"2024-03-24T00:40:02.685361+0000","last_peered":"2024-03-24T00:40:02.685361+0000","last_clean":"2024-03-24T00:40:02.685361+0000","last_became_active":"2024-03-24T00:40:01.345437+0000","last_became_peered":"2024-03-24T00:40:01.345437+0000","last_unstale":"2024-03-24T00:40:02.685361+0000","last_undegraded":"2024-03-24T00:40:02.685361+0000","last_fullsized":"2024-03-24T00:40:02.685361+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:47:44.580768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.329483+0000","last_change":"2024-03-24T00:40:00.347310+0000","last_active":"2024-03-24T00:40:02.329483+0000","last_peered":"2024-03-24T00:40:02.329483+0000","last_clean":"2024-03-24T00:40:02.329483+0000","last_became_active":"2024-03-24T00:40:00.347030+0000","last_became_peered":"2024-03-24T00:40:00.347030+0000","last_unstale":"2024-03-24T00:40:02.329483+0000","last_undegraded":"2024-03-24T00:40:02.329483+0000","last_fullsized":"2024-03-24T00:40:02.329483+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:16:31.720628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685032+0000","last_change":"2024-03-24T00:40:01.346045+0000","last_active":"2024-03-24T00:40:02.685032+0000","last_peered":"2024-03-24T00:40:02.685032+0000","last_clean":"2024-03-24T00:40:02.685032+0000","last_became_active":"2024-03-24T00:40:01.345693+0000","last_became_peered":"2024-03-24T00:40:01.345693+0000","last_unstale":"2024-03-24T00:40:02.685032+0000","last_undegraded":"2024-03-24T00:40:02.685032+0000","last_fullsized":"2024-03-24T00:40:02.685032+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:47:44.580768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.329491+0000","last_change":"2024-03-24T00:40:00.346403+0000","last_active":"2024-03-24T00:40:02.329491+0000","last_peered":"2024-03-24T00:40:02.329491+0000","last_clean":"2024-03-24T00:40:02.329491+0000","last_became_active":"2024-03-24T00:40:00.346095+0000","last_became_peered":"2024-03-24T00:40:00.346095+0000","last_unstale":"2024-03-24T00:40:02.329491+0000","last_undegraded":"2024-03-24T00:40:02.329491+0000","last_fullsized":"2024-03-24T00:40:02.329491+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:49:32.598338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.327180+0000","last_change":"2024-03-24T00:40:01.351042+0000","last_active":"2024-03-24T00:40:02.327180+0000","last_peered":"2024-03-24T00:40:02.327180+0000","last_clean":"2024-03-24T00:40:02.327180+0000","last_became_active":"2024-03-24T00:40:01.350750+0000","last_became_peered":"2024-03-24T00:40:01.350750+0000","last_unstale":"2024-03-24T00:40:02.327180+0000","last_undegraded":"2024-03-24T00:40:02.327180+0000","last_fullsized":"2024-03-24T00:40:02.327180+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:04:37.096161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685035+0000","last_change":"2024-03-24T00:40:00.335633+0000","last_active":"2024-03-24T00:40:02.685035+0000","last_peered":"2024-03-24T00:40:02.685035+0000","last_clean":"2024-03-24T00:40:02.685035+0000","last_became_active":"2024-03-24T00:40:00.335348+0000","last_became_peered":"2024-03-24T00:40:00.335348+0000","last_unstale":"2024-03-24T00:40:02.685035+0000","last_undegraded":"2024-03-24T00:40:02.685035+0000","last_fullsized":"2024-03-24T00:40:02.685035+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:40:41.214290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.326627+0000","last_change":"2024-03-24T00:40:01.345061+0000","last_active":"2024-03-24T00:40:02.326627+0000","last_peered":"2024-03-24T00:40:02.326627+0000","last_clean":"2024-03-24T00:40:02.326627+0000","last_became_active":"2024-03-24T00:40:01.344793+0000","last_became_peered":"2024-03-24T00:40:01.344793+0000","last_unstale":"2024-03-24T00:40:02.326627+0000","last_undegraded":"2024-03-24T00:40:02.326627+0000","last_fullsized":"2024-03-24T00:40:02.326627+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:17:43.539404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.329538+0000","last_change":"2024-03-24T00:40:00.338765+0000","last_active":"2024-03-24T00:40:02.329538+0000","last_peered":"2024-03-24T00:40:02.329538+0000","last_clean":"2024-03-24T00:40:02.329538+0000","last_became_active":"2024-03-24T00:40:00.337416+0000","last_became_peered":"2024-03-24T00:40:00.337416+0000","last_unstale":"2024-03-24T00:40:02.329538+0000","last_undegraded":"2024-03-24T00:40:02.329538+0000","last_fullsized":"2024-03-24T00:40:02.329538+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:38:41.933921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685292+0000","last_change":"2024-03-24T00:40:01.356678+0000","last_active":"2024-03-24T00:40:02.685292+0000","last_peered":"2024-03-24T00:40:02.685292+0000","last_clean":"2024-03-24T00:40:02.685292+0000","last_became_active":"2024-03-24T00:40:01.356395+0000","last_became_peered":"2024-03-24T00:40:01.356395+0000","last_unstale":"2024-03-24T00:40:02.685292+0000","last_undegraded":"2024-03-24T00:40:02.685292+0000","last_fullsized":"2024-03-24T00:40:02.685292+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:51.086346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.328338+0000","last_change":"2024-03-24T00:40:00.342913+0000","last_active":"2024-03-24T00:40:02.328338+0000","last_peered":"2024-03-24T00:40:02.328338+0000","last_clean":"2024-03-24T00:40:02.328338+0000","last_became_active":"2024-03-24T00:40:00.342633+0000","last_became_peered":"2024-03-24T00:40:00.342633+0000","last_unstale":"2024-03-24T00:40:02.328338+0000","last_undegraded":"2024-03-24T00:40:02.328338+0000","last_fullsized":"2024-03-24T00:40:02.328338+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:18:04.162241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685586+0000","last_change":"2024-03-24T00:40:01.343468+0000","last_active":"2024-03-24T00:40:02.685586+0000","last_peered":"2024-03-24T00:40:02.685586+0000","last_clean":"2024-03-24T00:40:02.685586+0000","last_became_active":"2024-03-24T00:40:01.343167+0000","last_became_peered":"2024-03-24T00:40:01.343167+0000","last_unstale":"2024-03-24T00:40:02.685586+0000","last_undegraded":"2024-03-24T00:40:02.685586+0000","last_fullsized":"2024-03-24T00:40:02.685586+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:04:37.096161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"39'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.329731+0000","last_change":"2024-03-24T00:40:00.335226+0000","last_active":"2024-03-24T00:40:02.329731+0000","last_peered":"2024-03-24T00:40:02.329731+0000","last_clean":"2024-03-24T00:40:02.329731+0000","last_became_active":"2024-03-24T00:40:00.334961+0000","last_became_peered":"2024-03-24T00:40:00.334961+0000","last_unstale":"2024-03-24T00:40:02.329731+0000","last_undegraded":"2024-03-24T00:40:02.329731+0000","last_fullsized":"2024-03-24T00:40:02.329731+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:23:45.494377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.326613+0000","last_change":"2024-03-24T00:40:01.349135+0000","last_active":"2024-03-24T00:40:02.326613+0000","last_peered":"2024-03-24T00:40:02.326613+0000","last_clean":"2024-03-24T00:40:02.326613+0000","last_became_active":"2024-03-24T00:40:01.348712+0000","last_became_peered":"2024-03-24T00:40:01.348712+0000","last_unstale":"2024-03-24T00:40:02.326613+0000","last_undegraded":"2024-03-24T00:40:02.326613+0000","last_fullsized":"2024-03-24T00:40:02.326613+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:00:13.987179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.684740+0000","last_change":"2024-03-24T00:40:00.342013+0000","last_active":"2024-03-24T00:40:02.684740+0000","last_peered":"2024-03-24T00:40:02.684740+0000","last_clean":"2024-03-24T00:40:02.684740+0000","last_became_active":"2024-03-24T00:40:00.341732+0000","last_became_peered":"2024-03-24T00:40:00.341732+0000","last_unstale":"2024-03-24T00:40:02.684740+0000","last_undegraded":"2024-03-24T00:40:02.684740+0000","last_fullsized":"2024-03-24T00:40:02.684740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:40:41.214290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.684998+0000","last_change":"2024-03-24T00:40:01.349804+0000","last_active":"2024-03-24T00:40:02.684998+0000","last_peered":"2024-03-24T00:40:02.684998+0000","last_clean":"2024-03-24T00:40:02.684998+0000","last_became_active":"2024-03-24T00:40:01.349542+0000","last_became_peered":"2024-03-24T00:40:01.349542+0000","last_unstale":"2024-03-24T00:40:02.684998+0000","last_undegraded":"2024-03-24T00:40:02.684998+0000","last_fullsized":"2024-03-24T00:40:02.684998+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:34:52.148336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.684813+0000","last_change":"2024-03-24T00:40:00.345415+0000","last_active":"2024-03-24T00:40:02.684813+0000","last_peered":"2024-03-24T00:40:02.684813+0000","last_clean":"2024-03-24T00:40:02.684813+0000","last_became_active":"2024-03-24T00:40:00.345145+0000","last_became_peered":"2024-03-24T00:40:00.345145+0000","last_unstale":"2024-03-24T00:40:02.684813+0000","last_undegraded":"2024-03-24T00:40:02.684813+0000","last_fullsized":"2024-03-24T00:40:02.684813+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:05:49.943101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685368+0000","last_change":"2024-03-24T00:40:01.344953+0000","last_active":"2024-03-24T00:40:02.685368+0000","last_peered":"2024-03-24T00:40:02.685368+0000","last_clean":"2024-03-24T00:40:02.685368+0000","last_became_active":"2024-03-24T00:40:01.344683+0000","last_became_peered":"2024-03-24T00:40:01.344683+0000","last_unstale":"2024-03-24T00:40:02.685368+0000","last_undegraded":"2024-03-24T00:40:02.685368+0000","last_fullsized":"2024-03-24T00:40:02.685368+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:34:52.148336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685583+0000","last_change":"2024-03-24T00:40:00.335784+0000","last_active":"2024-03-24T00:40:02.685583+0000","last_peered":"2024-03-24T00:40:02.685583+0000","last_clean":"2024-03-24T00:40:02.685583+0000","last_became_active":"2024-03-24T00:40:00.335478+0000","last_became_peered":"2024-03-24T00:40:00.335478+0000","last_unstale":"2024-03-24T00:40:02.685583+0000","last_undegraded":"2024-03-24T00:40:02.685583+0000","last_fullsized":"2024-03-24T00:40:02.685583+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:47:40.706718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.327376+0000","last_change":"2024-03-24T00:40:01.352911+0000","last_active":"2024-03-24T00:40:02.327376+0000","last_peered":"2024-03-24T00:40:02.327376+0000","last_clean":"2024-03-24T00:40:02.327376+0000","last_became_active":"2024-03-24T00:40:01.352668+0000","last_became_peered":"2024-03-24T00:40:01.352668+0000","last_unstale":"2024-03-24T00:40:02.327376+0000","last_undegraded":"2024-03-24T00:40:02.327376+0000","last_fullsized":"2024-03-24T00:40:02.327376+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:11:12.174614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.326903+0000","last_change":"2024-03-24T00:40:00.342133+0000","last_active":"2024-03-24T00:40:02.326903+0000","last_peered":"2024-03-24T00:40:02.326903+0000","last_clean":"2024-03-24T00:40:02.326903+0000","last_became_active":"2024-03-24T00:40:00.341821+0000","last_became_peered":"2024-03-24T00:40:00.341821+0000","last_unstale":"2024-03-24T00:40:02.326903+0000","last_undegraded":"2024-03-24T00:40:02.326903+0000","last_fullsized":"2024-03-24T00:40:02.326903+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:50:26.015558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685436+0000","last_change":"2024-03-24T00:40:01.344138+0000","last_active":"2024-03-24T00:40:02.685436+0000","last_peered":"2024-03-24T00:40:02.685436+0000","last_clean":"2024-03-24T00:40:02.685436+0000","last_became_active":"2024-03-24T00:40:01.343857+0000","last_became_peered":"2024-03-24T00:40:01.343857+0000","last_unstale":"2024-03-24T00:40:02.685436+0000","last_undegraded":"2024-03-24T00:40:02.685436+0000","last_fullsized":"2024-03-24T00:40:02.685436+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:23:33.705793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.326192+0000","last_change":"2024-03-24T00:40:00.344855+0000","last_active":"2024-03-24T00:40:02.326192+0000","last_peered":"2024-03-24T00:40:02.326192+0000","last_clean":"2024-03-24T00:40:02.326192+0000","last_became_active":"2024-03-24T00:40:00.344605+0000","last_became_peered":"2024-03-24T00:40:00.344605+0000","last_unstale":"2024-03-24T00:40:02.326192+0000","last_undegraded":"2024-03-24T00:40:02.326192+0000","last_fullsized":"2024-03-24T00:40:02.326192+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:50.084293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.328267+0000","last_change":"2024-03-24T00:40:01.355150+0000","last_active":"2024-03-24T00:40:02.328267+0000","last_peered":"2024-03-24T00:40:02.328267+0000","last_clean":"2024-03-24T00:40:02.328267+0000","last_became_active":"2024-03-24T00:40:01.354677+0000","last_became_peered":"2024-03-24T00:40:01.354677+0000","last_unstale":"2024-03-24T00:40:02.328267+0000","last_undegraded":"2024-03-24T00:40:02.328267+0000","last_fullsized":"2024-03-24T00:40:02.328267+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:09:15.867422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685395+0000","last_change":"2024-03-24T00:40:00.341415+0000","last_active":"2024-03-24T00:40:02.685395+0000","last_peered":"2024-03-24T00:40:02.685395+0000","last_clean":"2024-03-24T00:40:02.685395+0000","last_became_active":"2024-03-24T00:40:00.341108+0000","last_became_peered":"2024-03-24T00:40:00.341108+0000","last_unstale":"2024-03-24T00:40:02.685395+0000","last_undegraded":"2024-03-24T00:40:02.685395+0000","last_fullsized":"2024-03-24T00:40:02.685395+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:27:38.490325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":205,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685138+0000","last_change":"2024-03-24T00:38:40.953871+0000","last_active":"2024-03-24T00:40:02.685138+0000","last_peered":"2024-03-24T00:40:02.685138+0000","last_clean":"2024-03-24T00:40:02.685138+0000","last_became_active":"2024-03-24T00:38:40.952926+0000","last_became_peered":"2024-03-24T00:38:40.952926+0000","last_unstale":"2024-03-24T00:40:02.685138+0000","last_undegraded":"2024-03-24T00:40:02.685138+0000","last_fullsized":"2024-03-24T00:40:02.685138+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_clean_scrub_stamp":"2024-03-24T00:38:39.722194+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:05.397257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574870,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28060,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343780,"statfs":{"total":96636764160,"available":96608030720,"internally_reserved":0,"allocated":1060864,"data_stored":789207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574869,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27468,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344372,"statfs":{"total":96636764160,"available":96608636928,"internally_reserved":0,"allocated":454656,"data_stored":197739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574869,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27476,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344364,"statfs":{"total":96636764160,"available":96608628736,"internally_reserved":0,"allocated":458752,"data_stored":197269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574868,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27476,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344364,"statfs":{"total":96636764160,"available":96608628736,"internally_reserved":0,"allocated":458752,"data_stored":197761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574868,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27480,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344360,"statfs":{"total":96636764160,"available":96608624640,"internally_reserved":0,"allocated":466944,"data_stored":198803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574869,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27468,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344372,"statfs":{"total":96636764160,"available":96608636928,"internally_reserved":0,"allocated":454656,"data_stored":197227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27472,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344368,"statfs":{"total":96636764160,"available":96608632832,"internally_reserved":0,"allocated":454656,"data_stored":197247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574868,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28048,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343792,"statfs":{"total":96636764160,"available":96608043008,"internally_reserved":0,"allocated":1048576,"data_stored":787559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:40:05.494 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-24T00:40:05.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.630+0000 7f34c671b640 1 Processor -- start 2024-03-24T00:40:05.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.631+0000 7f34c671b640 1 -- start start 2024-03-24T00:40:05.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.631+0000 7f34c671b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f34c01445a0 0x7f34c00cee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:05.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.631+0000 7f34c671b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f34c00b48d0 0x7f34c0072550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:05.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.631+0000 7f34c671b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34c00b4d30 con 0x7f34c006ec40 2024-03-24T00:40:05.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.631+0000 7f34c671b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34c00ce990 con 0x7f34c01445a0 2024-03-24T00:40:05.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.631+0000 7f34c671b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34c00cddc0 con 0x7f34c00b48d0 2024-03-24T00:40:05.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.632+0000 7f34bffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f34c01445a0 0x7f34c00cee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:05.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.632+0000 7f34bffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f34c01445a0 0x7f34c00cee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40866/0 (socket says 172.21.15.67:40866) 2024-03-24T00:40:05.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.632+0000 7f34bffff640 1 -- 172.21.15.67:0/2776475002 learned_addr learned my addr 172.21.15.67:0/2776475002 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:05.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.632+0000 7f34bf7fe640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f34c00b48d0 0x7f34c0072550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:05.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.632+0000 7f34beffd640 1 -- 172.21.15.67:0/2776475002 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2316019890 0 0) 0x7f34b8001490 con 0x7f34c006ec40 2024-03-24T00:40:05.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.632+0000 7f34bffff640 1 -- 172.21.15.67:0/2776475002 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f34c00b48d0 msgr2=0x7f34c0072550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:05.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.632+0000 7f34bffff640 1 --2- 172.21.15.67:0/2776475002 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f34c00b48d0 0x7f34c0072550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:05.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.632+0000 7f34bffff640 1 -- 172.21.15.67:0/2776475002 >> v1:172.21.15.72:6789/0 conn(0x7f34c006ec40 legacy=0x7f34c00b2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:05.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.632+0000 7f34bffff640 1 -- 172.21.15.67:0/2776475002 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c00ce060 con 0x7f34c01445a0 2024-03-24T00:40:05.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.632+0000 7f34bf7fe640 1 --2- 172.21.15.67:0/2776475002 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f34c00b48d0 0x7f34c0072550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:40:05.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.633+0000 7f34bffff640 1 --2- 172.21.15.67:0/2776475002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34c01445a0 0x7f34c00cee20 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f34b0009870 tx=0x7f34b002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f60c2548569aaefa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:05.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.633+0000 7f34beffd640 1 -- 172.21.15.67:0/2776475002 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b0034070 con 0x7f34c01445a0 2024-03-24T00:40:05.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.633+0000 7f34beffd640 1 -- 172.21.15.67:0/2776475002 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34b0030840 con 0x7f34c01445a0 2024-03-24T00:40:05.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.633+0000 7f34beffd640 1 -- 172.21.15.67:0/2776475002 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b0030c10 con 0x7f34c01445a0 2024-03-24T00:40:05.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.633+0000 7f34c671b640 1 -- 172.21.15.67:0/2776475002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34c01445a0 msgr2=0x7f34c00cee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:05.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.633+0000 7f34c671b640 1 --2- 172.21.15.67:0/2776475002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34c01445a0 0x7f34c00cee20 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f34b0009870 tx=0x7f34b002b980 comp rx=0 tx=0).stop 2024-03-24T00:40:05.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.633+0000 7f34c671b640 1 -- 172.21.15.67:0/2776475002 shutdown_connections 2024-03-24T00:40:05.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.633+0000 7f34c671b640 1 --2- 172.21.15.67:0/2776475002 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f34c00b48d0 0x7f34c0072550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:05.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.633+0000 7f34c671b640 1 --2- 172.21.15.67:0/2776475002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34c01445a0 0x7f34c00cee20 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:05.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.633+0000 7f34c671b640 1 -- 172.21.15.67:0/2776475002 >> 172.21.15.67:0/2776475002 conn(0x7f34c00cf9e0 msgr2=0x7f34c00cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:05.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.633+0000 7f34c671b640 1 -- 172.21.15.67:0/2776475002 shutdown_connections 2024-03-24T00:40:05.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.633+0000 7f34c671b640 1 -- 172.21.15.67:0/2776475002 wait complete. 2024-03-24T00:40:05.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.634+0000 7f34c671b640 1 Processor -- start 2024-03-24T00:40:05.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.635+0000 7f34c671b640 1 -- start start 2024-03-24T00:40:05.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.635+0000 7f34c671b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f34c006ec40 0x7f34c01f3b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:05.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.635+0000 7f34c671b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34c00b48d0 0x7f34c01e6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:05.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.635+0000 7f34c671b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34c01445a0 0x7f34c006db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:05.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.635+0000 7f34c671b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c006f0d0 con 0x7f34c00b48d0 2024-03-24T00:40:05.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.635+0000 7f34c671b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c00ce620 con 0x7f34c01445a0 2024-03-24T00:40:05.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.635+0000 7f34c671b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34c006f250 con 0x7f34c006ec40 2024-03-24T00:40:05.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.635+0000 7f34bf7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34c00b48d0 0x7f34c01e6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:05.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.635+0000 7f34bf7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34c00b48d0 0x7f34c01e6320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40876/0 (socket says 172.21.15.67:40876) 2024-03-24T00:40:05.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.635+0000 7f34bffff640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f34c006ec40 0x7f34c01f3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:05.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.635+0000 7f34bf7fe640 1 -- 172.21.15.67:0/2498443192 learned_addr learned my addr 172.21.15.67:0/2498443192 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:05.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.636+0000 7f34c4cb8640 1 --2- 172.21.15.67:0/2498443192 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34c01445a0 0x7f34c006db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:05.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.636+0000 7f34bf7fe640 1 -- 172.21.15.67:0/2498443192 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f34c006ec40 msgr2=0x7f34c01f3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:05.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.636+0000 7f34bf7fe640 1 --2- 172.21.15.67:0/2498443192 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f34c006ec40 0x7f34c01f3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:05.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.636+0000 7f34bf7fe640 1 -- 172.21.15.67:0/2498443192 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34c01445a0 msgr2=0x7f34c006db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:05.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.636+0000 7f34bf7fe640 1 --2- 172.21.15.67:0/2498443192 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34c01445a0 0x7f34c006db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:05.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.636+0000 7f34bf7fe640 1 -- 172.21.15.67:0/2498443192 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c013d5b0 con 0x7f34c00b48d0 2024-03-24T00:40:05.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.636+0000 7f34bf7fe640 1 --2- 172.21.15.67:0/2498443192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34c00b48d0 0x7f34c01e6320 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f34a800ebe0 tx=0x7f34a800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:05.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.636+0000 7f34bcff9640 1 -- 172.21.15.67:0/2498443192 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a8017070 con 0x7f34c00b48d0 2024-03-24T00:40:05.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.636+0000 7f34bcff9640 1 -- 172.21.15.67:0/2498443192 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34a800cc30 con 0x7f34c00b48d0 2024-03-24T00:40:05.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.636+0000 7f34c671b640 1 -- 172.21.15.67:0/2498443192 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34c013d890 con 0x7f34c00b48d0 2024-03-24T00:40:05.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.636+0000 7f34bcff9640 1 -- 172.21.15.67:0/2498443192 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a8013470 con 0x7f34c00b48d0 2024-03-24T00:40:05.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.636+0000 7f34c671b640 1 -- 172.21.15.67:0/2498443192 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34c013de50 con 0x7f34c00b48d0 2024-03-24T00:40:05.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.637+0000 7f34c671b640 1 -- 172.21.15.67:0/2498443192 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34c004f400 con 0x7f34c00b48d0 2024-03-24T00:40:05.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.638+0000 7f34bcff9640 1 -- 172.21.15.67:0/2498443192 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f34a801a070 con 0x7f34c00b48d0 2024-03-24T00:40:05.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.638+0000 7f34bcff9640 1 --2- 172.21.15.67:0/2498443192 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f34a006bbd0 0x7f34a006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:05.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.638+0000 7f34bcff9640 1 -- 172.21.15.67:0/2498443192 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f34a808ae60 con 0x7f34c00b48d0 2024-03-24T00:40:05.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.638+0000 7f34bffff640 1 --2- 172.21.15.67:0/2498443192 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f34a006bbd0 0x7f34a006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:05.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.639+0000 7f34bffff640 1 --2- 172.21.15.67:0/2498443192 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f34a006bbd0 0x7f34a006e090 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f34b0036000 tx=0x7f34b003c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:05.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.642+0000 7f34bcff9640 1 -- 172.21.15.67:0/2498443192 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f34a8059170 con 0x7f34c00b48d0 2024-03-24T00:40:05.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.893+0000 7f34c671b640 1 -- 172.21.15.67:0/2498443192 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f34c0072c20 con 0x7f34c00b48d0 2024-03-24T00:40:05.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.893+0000 7f34bcff9640 1 -- 172.21.15.67:0/2498443192 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1173 (secure 0 0 0) 0x7f34a805d210 con 0x7f34c00b48d0 2024-03-24T00:40:05.894 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:40:05.894 INFO:teuthology.orchestra.run.smithi067.stdout:{"fsid":"de1d0178-808e-49ca-a986-e297e523b976","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":98,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":8,"num_up_osds":8,"osd_up_since":1711240718,"num_in_osds":8,"osd_in_since":1711240709,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":24,"data_bytes":592778,"bytes_used":226250752,"bytes_avail":772867862528,"bytes_total":773094113280,"write_bytes_sec":2388,"read_op_per_sec":0,"write_op_per_sec":7},"fsmap":{"epoch":17,"id":2,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":2,"rank":0,"name":"d","status":"up:active","gid":5418}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-03-24T00:39:47.335006+0000","services":{"mds":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-24T00:40:05.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.895+0000 7f34c671b640 1 -- 172.21.15.67:0/2498443192 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f34a006bbd0 msgr2=0x7f34a006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:05.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.895+0000 7f34c671b640 1 --2- 172.21.15.67:0/2498443192 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f34a006bbd0 0x7f34a006e090 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f34b0036000 tx=0x7f34b003c040 comp rx=0 tx=0).stop 2024-03-24T00:40:05.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.895+0000 7f34c671b640 1 -- 172.21.15.67:0/2498443192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34c00b48d0 msgr2=0x7f34c01e6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:05.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.895+0000 7f34c671b640 1 --2- 172.21.15.67:0/2498443192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34c00b48d0 0x7f34c01e6320 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f34a800ebe0 tx=0x7f34a800c520 comp rx=0 tx=0).stop 2024-03-24T00:40:05.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.895+0000 7f34c671b640 1 -- 172.21.15.67:0/2498443192 shutdown_connections 2024-03-24T00:40:05.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.895+0000 7f34c671b640 1 --2- 172.21.15.67:0/2498443192 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f34a006bbd0 0x7f34a006e090 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:05.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.895+0000 7f34c671b640 1 --2- 172.21.15.67:0/2498443192 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34c01445a0 0x7f34c006db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:05.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.895+0000 7f34c671b640 1 --2- 172.21.15.67:0/2498443192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34c00b48d0 0x7f34c01e6320 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:05.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.895+0000 7f34c671b640 1 --2- 172.21.15.67:0/2498443192 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f34c006ec40 0x7f34c01f3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:05.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.895+0000 7f34c671b640 1 -- 172.21.15.67:0/2498443192 >> 172.21.15.67:0/2498443192 conn(0x7f34c00cf9e0 msgr2=0x7f34c00af660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:05.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.895+0000 7f34c671b640 1 -- 172.21.15.67:0/2498443192 shutdown_connections 2024-03-24T00:40:05.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:05.895+0000 7f34c671b640 1 -- 172.21.15.67:0/2498443192 wait complete. 2024-03-24T00:40:05.906 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-24T00:40:05.907 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:40:06.004 INFO:tasks.ceph.mds.d.smithi072.stderr:2024-03-24T00:40:06.004+0000 7fc75cacc640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T00:40:06.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.007+0000 7f2644f13640 1 Processor -- start 2024-03-24T00:40:06.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.007+0000 7f2644f13640 1 -- start start 2024-03-24T00:40:06.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.008+0000 7f2644f13640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f264015e390 0x7f264006b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:06.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.008+0000 7f2644f13640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f26400cded0 0x7f264006b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:06.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.008+0000 7f2644f13640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26400d07d0 con 0x7f264015dfb0 2024-03-24T00:40:06.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.008+0000 7f2644f13640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26400d0590 con 0x7f26400cded0 2024-03-24T00:40:06.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.008+0000 7f2644f13640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26400d0ab0 con 0x7f264015e390 2024-03-24T00:40:06.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.008+0000 7f263ed9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f26400cded0 0x7f264006b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:06.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.008+0000 7f263ed9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f26400cded0 0x7f264006b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40892/0 (socket says 172.21.15.67:40892) 2024-03-24T00:40:06.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.008+0000 7f263ed9d640 1 -- 172.21.15.67:0/1549074321 learned_addr learned my addr 172.21.15.67:0/1549074321 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:06.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.008+0000 7f263e59c640 1 --1- 172.21.15.67:0/1549074321 >> v1:172.21.15.72:6789/0 conn(0x7f264015dfb0 0x7f26400cfe80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:48958/0 (socket says 172.21.15.67:48958) 2024-03-24T00:40:06.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.008+0000 7f263dd9b640 1 --2- 172.21.15.67:0/1549074321 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f264015e390 0x7f264006b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:06.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.009+0000 7f263ed9d640 1 -- 172.21.15.67:0/1549074321 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f264015e390 msgr2=0x7f264006b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:06.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.009+0000 7f263ed9d640 1 --2- 172.21.15.67:0/1549074321 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f264015e390 0x7f264006b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:06.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.009+0000 7f263ed9d640 1 -- 172.21.15.67:0/1549074321 >> v1:172.21.15.72:6789/0 conn(0x7f264015dfb0 legacy=0x7f26400cfe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:06.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.009+0000 7f263d59a640 1 -- 172.21.15.67:0/1549074321 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 787202352 0 0) 0x7f26400d07d0 con 0x7f264015dfb0 2024-03-24T00:40:06.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.009+0000 7f263ed9d640 1 -- 172.21.15.67:0/1549074321 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26400d0d90 con 0x7f26400cded0 2024-03-24T00:40:06.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.009+0000 7f263ed9d640 1 --2- 172.21.15.67:0/1549074321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26400cded0 0x7f264006b580 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f2638009e50 tx=0x7f263802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=438a293d390b6e82 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:06.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.010+0000 7f263d59a640 1 -- 172.21.15.67:0/1549074321 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f263802cb00 con 0x7f26400cded0 2024-03-24T00:40:06.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.010+0000 7f263d59a640 1 -- 172.21.15.67:0/1549074321 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f263802cca0 con 0x7f26400cded0 2024-03-24T00:40:06.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.010+0000 7f263d59a640 1 -- 172.21.15.67:0/1549074321 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2638034610 con 0x7f26400cded0 2024-03-24T00:40:06.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.010+0000 7f2644f13640 1 -- 172.21.15.67:0/1549074321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26400cded0 msgr2=0x7f264006b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:06.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.010+0000 7f2644f13640 1 --2- 172.21.15.67:0/1549074321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26400cded0 0x7f264006b580 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f2638009e50 tx=0x7f263802bf60 comp rx=0 tx=0).stop 2024-03-24T00:40:06.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.010+0000 7f2644f13640 1 -- 172.21.15.67:0/1549074321 shutdown_connections 2024-03-24T00:40:06.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.010+0000 7f2644f13640 1 --2- 172.21.15.67:0/1549074321 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f264015e390 0x7f264006b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:06.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.010+0000 7f2644f13640 1 --2- 172.21.15.67:0/1549074321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26400cded0 0x7f264006b580 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:06.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.010+0000 7f2644f13640 1 -- 172.21.15.67:0/1549074321 >> 172.21.15.67:0/1549074321 conn(0x7f2640099a30 msgr2=0x7f2640099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:06.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.010+0000 7f2644f13640 1 -- 172.21.15.67:0/1549074321 shutdown_connections 2024-03-24T00:40:06.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.010+0000 7f2644f13640 1 -- 172.21.15.67:0/1549074321 wait complete. 2024-03-24T00:40:06.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.012+0000 7f2644f13640 1 Processor -- start 2024-03-24T00:40:06.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.012+0000 7f2644f13640 1 -- start start 2024-03-24T00:40:06.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.012+0000 7f2644f13640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f26400cded0 0x7f26400ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:06.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.012+0000 7f2644f13640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f264015dfb0 0x7f26401578a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:06.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.012+0000 7f2644f13640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f264015e390 0x7f26401f7100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:06.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.012+0000 7f2644f13640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26400d1870 con 0x7f264015e390 2024-03-24T00:40:06.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.012+0000 7f2644f13640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26400d1350 con 0x7f26400cded0 2024-03-24T00:40:06.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.012+0000 7f2644f13640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26400cb120 con 0x7f264015dfb0 2024-03-24T00:40:06.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.012+0000 7f263ed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f264015e390 0x7f26401f7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:06.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.012+0000 7f263ed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f264015e390 0x7f26401f7100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40902/0 (socket says 172.21.15.67:40902) 2024-03-24T00:40:06.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.012+0000 7f263ed9d640 1 -- 172.21.15.67:0/3177776213 learned_addr learned my addr 172.21.15.67:0/3177776213 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:06.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.012+0000 7f263e59c640 1 --2- 172.21.15.67:0/3177776213 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f26400cded0 0x7f26400ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:06.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.012+0000 7f263dd9b640 1 --2- 172.21.15.67:0/3177776213 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f264015dfb0 0x7f26401578a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:06.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.013+0000 7f263ed9d640 1 -- 172.21.15.67:0/3177776213 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f264015dfb0 msgr2=0x7f26401578a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:06.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.013+0000 7f263ed9d640 1 --2- 172.21.15.67:0/3177776213 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f264015dfb0 0x7f26401578a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:06.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.013+0000 7f263ed9d640 1 -- 172.21.15.67:0/3177776213 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f26400cded0 msgr2=0x7f26400ca6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:06.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.013+0000 7f263ed9d640 1 --2- 172.21.15.67:0/3177776213 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f26400cded0 0x7f26400ca6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:06.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.013+0000 7f263ed9d640 1 -- 172.21.15.67:0/3177776213 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f264015f8c0 con 0x7f264015e390 2024-03-24T00:40:06.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.013+0000 7f263ed9d640 1 --2- 172.21.15.67:0/3177776213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f264015e390 0x7f26401f7100 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f2638009e50 tx=0x7f2638004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:06.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.013+0000 7f2636ffd640 1 -- 172.21.15.67:0/3177776213 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2638043030 con 0x7f264015e390 2024-03-24T00:40:06.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.013+0000 7f2636ffd640 1 -- 172.21.15.67:0/3177776213 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f263802ce50 con 0x7f264015e390 2024-03-24T00:40:06.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.013+0000 7f2636ffd640 1 -- 172.21.15.67:0/3177776213 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2638009510 con 0x7f264015e390 2024-03-24T00:40:06.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.013+0000 7f2644f13640 1 -- 172.21.15.67:0/3177776213 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2640147bd0 con 0x7f264015e390 2024-03-24T00:40:06.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.013+0000 7f2644f13640 1 -- 172.21.15.67:0/3177776213 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f264015a370 con 0x7f264015e390 2024-03-24T00:40:06.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.014+0000 7f2644f13640 1 -- 172.21.15.67:0/3177776213 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f264004f400 con 0x7f264015e390 2024-03-24T00:40:06.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.015+0000 7f2636ffd640 1 -- 172.21.15.67:0/3177776213 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2638046070 con 0x7f264015e390 2024-03-24T00:40:06.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.015+0000 7f2636ffd640 1 --2- 172.21.15.67:0/3177776213 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f262006bc00 0x7f262006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:06.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.015+0000 7f2636ffd640 1 -- 172.21.15.67:0/3177776213 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f26380acc30 con 0x7f264015e390 2024-03-24T00:40:06.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.015+0000 7f263e59c640 1 --2- 172.21.15.67:0/3177776213 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f262006bc00 0x7f262006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:06.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.015+0000 7f263e59c640 1 --2- 172.21.15.67:0/3177776213 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f262006bc00 0x7f262006e0c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2630000f60 tx=0x7f2630007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:06.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.019+0000 7f2636ffd640 1 -- 172.21.15.67:0/3177776213 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f263804d070 con 0x7f264015e390 2024-03-24T00:40:06.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.197+0000 7f2644f13640 1 -- 172.21.15.67:0/3177776213 --> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f26400b6a50 con 0x7f262006bc00 2024-03-24T00:40:06.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.204+0000 7f2636ffd640 1 -- 172.21.15.67:0/3177776213 <== mgr.4105 v2:172.21.15.72:6832/2768292274 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251958 (secure 0 0 0) 0x7f26400b6a50 con 0x7f262006bc00 2024-03-24T00:40:06.206 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:40:06.208 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-24T00:40:06.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.206+0000 7f2644f13640 1 -- 172.21.15.67:0/3177776213 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f262006bc00 msgr2=0x7f262006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:06.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.206+0000 7f2644f13640 1 --2- 172.21.15.67:0/3177776213 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f262006bc00 0x7f262006e0c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2630000f60 tx=0x7f2630007d50 comp rx=0 tx=0).stop 2024-03-24T00:40:06.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.206+0000 7f2644f13640 1 -- 172.21.15.67:0/3177776213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f264015e390 msgr2=0x7f26401f7100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:06.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.206+0000 7f2644f13640 1 --2- 172.21.15.67:0/3177776213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f264015e390 0x7f26401f7100 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f2638009e50 tx=0x7f2638004a40 comp rx=0 tx=0).stop 2024-03-24T00:40:06.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.206+0000 7f2644f13640 1 -- 172.21.15.67:0/3177776213 shutdown_connections 2024-03-24T00:40:06.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.206+0000 7f2644f13640 1 --2- 172.21.15.67:0/3177776213 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f262006bc00 0x7f262006e0c0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:06.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.206+0000 7f2644f13640 1 --2- 172.21.15.67:0/3177776213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f264015e390 0x7f26401f7100 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:06.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.206+0000 7f2644f13640 1 --2- 172.21.15.67:0/3177776213 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f264015dfb0 0x7f26401578a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:06.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.206+0000 7f2644f13640 1 --2- 172.21.15.67:0/3177776213 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f26400cded0 0x7f26400ca6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:06.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.206+0000 7f2644f13640 1 -- 172.21.15.67:0/3177776213 >> 172.21.15.67:0/3177776213 conn(0x7f2640099a30 msgr2=0x7f264015d1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:06.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.206+0000 7f2644f13640 1 -- 172.21.15.67:0/3177776213 shutdown_connections 2024-03-24T00:40:06.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:06.206+0000 7f2644f13640 1 -- 172.21.15.67:0/3177776213 wait complete. 2024-03-24T00:40:06.217 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":75,"stamp":"2024-03-24T00:40:05.338666+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":174,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220948,"kb_used_data":4744,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753772,"statfs":{"total":773094113280,"available":772867862528,"internally_reserved":0,"allocated":4857856,"data_stored":2762812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000985"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.326693+0000","last_change":"2024-03-24T00:40:01.346232+0000","last_active":"2024-03-24T00:40:02.326693+0000","last_peered":"2024-03-24T00:40:02.326693+0000","last_clean":"2024-03-24T00:40:02.326693+0000","last_became_active":"2024-03-24T00:40:01.345850+0000","last_became_peered":"2024-03-24T00:40:01.345850+0000","last_unstale":"2024-03-24T00:40:02.326693+0000","last_undegraded":"2024-03-24T00:40:02.326693+0000","last_fullsized":"2024-03-24T00:40:02.326693+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:38:18.834926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.326462+0000","last_change":"2024-03-24T00:40:01.334855+0000","last_active":"2024-03-24T00:40:02.326462+0000","last_peered":"2024-03-24T00:40:02.326462+0000","last_clean":"2024-03-24T00:40:02.326462+0000","last_became_active":"2024-03-24T00:40:01.334560+0000","last_became_peered":"2024-03-24T00:40:01.334560+0000","last_unstale":"2024-03-24T00:40:02.326462+0000","last_undegraded":"2024-03-24T00:40:02.326462+0000","last_fullsized":"2024-03-24T00:40:02.326462+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:52:56.182730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.684957+0000","last_change":"2024-03-24T00:40:01.356272+0000","last_active":"2024-03-24T00:40:02.684957+0000","last_peered":"2024-03-24T00:40:02.684957+0000","last_clean":"2024-03-24T00:40:02.684957+0000","last_became_active":"2024-03-24T00:40:01.355995+0000","last_became_peered":"2024-03-24T00:40:01.355995+0000","last_unstale":"2024-03-24T00:40:02.684957+0000","last_undegraded":"2024-03-24T00:40:02.684957+0000","last_fullsized":"2024-03-24T00:40:02.684957+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:51.086346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.329660+0000","last_change":"2024-03-24T00:40:01.349541+0000","last_active":"2024-03-24T00:40:02.329660+0000","last_peered":"2024-03-24T00:40:02.329660+0000","last_clean":"2024-03-24T00:40:02.329660+0000","last_became_active":"2024-03-24T00:40:01.349072+0000","last_became_peered":"2024-03-24T00:40:01.349072+0000","last_unstale":"2024-03-24T00:40:02.329660+0000","last_undegraded":"2024-03-24T00:40:02.329660+0000","last_fullsized":"2024-03-24T00:40:02.329660+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:06:45.735287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.329550+0000","last_change":"2024-03-24T00:40:01.349628+0000","last_active":"2024-03-24T00:40:02.329550+0000","last_peered":"2024-03-24T00:40:02.329550+0000","last_clean":"2024-03-24T00:40:02.329550+0000","last_became_active":"2024-03-24T00:40:01.349371+0000","last_became_peered":"2024-03-24T00:40:01.349371+0000","last_unstale":"2024-03-24T00:40:02.329550+0000","last_undegraded":"2024-03-24T00:40:02.329550+0000","last_fullsized":"2024-03-24T00:40:02.329550+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:09:01.114410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685148+0000","last_change":"2024-03-24T00:40:01.345171+0000","last_active":"2024-03-24T00:40:02.685148+0000","last_peered":"2024-03-24T00:40:02.685148+0000","last_clean":"2024-03-24T00:40:02.685148+0000","last_became_active":"2024-03-24T00:40:01.344909+0000","last_became_peered":"2024-03-24T00:40:01.344909+0000","last_unstale":"2024-03-24T00:40:02.685148+0000","last_undegraded":"2024-03-24T00:40:02.685148+0000","last_fullsized":"2024-03-24T00:40:02.685148+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:11:12.174614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.326916+0000","last_change":"2024-03-24T00:40:01.350464+0000","last_active":"2024-03-24T00:40:02.326916+0000","last_peered":"2024-03-24T00:40:02.326916+0000","last_clean":"2024-03-24T00:40:02.326916+0000","last_became_active":"2024-03-24T00:40:01.350212+0000","last_became_peered":"2024-03-24T00:40:01.350212+0000","last_unstale":"2024-03-24T00:40:02.326916+0000","last_undegraded":"2024-03-24T00:40:02.326916+0000","last_fullsized":"2024-03-24T00:40:02.326916+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:44:25.432200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.326251+0000","last_change":"2024-03-24T00:40:01.356804+0000","last_active":"2024-03-24T00:40:02.326251+0000","last_peered":"2024-03-24T00:40:02.326251+0000","last_clean":"2024-03-24T00:40:02.326251+0000","last_became_active":"2024-03-24T00:40:01.356513+0000","last_became_peered":"2024-03-24T00:40:01.356513+0000","last_unstale":"2024-03-24T00:40:02.326251+0000","last_undegraded":"2024-03-24T00:40:02.326251+0000","last_fullsized":"2024-03-24T00:40:02.326251+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:09:15.867422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685100+0000","last_change":"2024-03-24T00:40:01.356194+0000","last_active":"2024-03-24T00:40:02.685100+0000","last_peered":"2024-03-24T00:40:02.685100+0000","last_clean":"2024-03-24T00:40:02.685100+0000","last_became_active":"2024-03-24T00:40:01.355912+0000","last_became_peered":"2024-03-24T00:40:01.355912+0000","last_unstale":"2024-03-24T00:40:02.685100+0000","last_undegraded":"2024-03-24T00:40:02.685100+0000","last_fullsized":"2024-03-24T00:40:02.685100+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:23:33.705793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.328221+0000","last_change":"2024-03-24T00:40:01.352016+0000","last_active":"2024-03-24T00:40:02.328221+0000","last_peered":"2024-03-24T00:40:02.328221+0000","last_clean":"2024-03-24T00:40:02.328221+0000","last_became_active":"2024-03-24T00:40:01.351496+0000","last_became_peered":"2024-03-24T00:40:01.351496+0000","last_unstale":"2024-03-24T00:40:02.328221+0000","last_undegraded":"2024-03-24T00:40:02.328221+0000","last_fullsized":"2024-03-24T00:40:02.328221+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:54:11.377947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.327897+0000","last_change":"2024-03-24T00:40:01.354643+0000","last_active":"2024-03-24T00:40:02.327897+0000","last_peered":"2024-03-24T00:40:02.327897+0000","last_clean":"2024-03-24T00:40:02.327897+0000","last_became_active":"2024-03-24T00:40:01.354393+0000","last_became_peered":"2024-03-24T00:40:01.354393+0000","last_unstale":"2024-03-24T00:40:02.327897+0000","last_undegraded":"2024-03-24T00:40:02.327897+0000","last_fullsized":"2024-03-24T00:40:02.327897+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:37:43.101486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.684935+0000","last_change":"2024-03-24T00:40:01.342932+0000","last_active":"2024-03-24T00:40:02.684935+0000","last_peered":"2024-03-24T00:40:02.684935+0000","last_clean":"2024-03-24T00:40:02.684935+0000","last_became_active":"2024-03-24T00:40:01.342615+0000","last_became_peered":"2024-03-24T00:40:01.342615+0000","last_unstale":"2024-03-24T00:40:02.684935+0000","last_undegraded":"2024-03-24T00:40:02.684935+0000","last_fullsized":"2024-03-24T00:40:02.684935+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:11:12.174614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.327186+0000","last_change":"2024-03-24T00:40:01.352457+0000","last_active":"2024-03-24T00:40:02.327186+0000","last_peered":"2024-03-24T00:40:02.327186+0000","last_clean":"2024-03-24T00:40:02.327186+0000","last_became_active":"2024-03-24T00:40:01.352201+0000","last_became_peered":"2024-03-24T00:40:01.352201+0000","last_unstale":"2024-03-24T00:40:02.327186+0000","last_undegraded":"2024-03-24T00:40:02.327186+0000","last_fullsized":"2024-03-24T00:40:02.327186+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:52:01.752611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.327238+0000","last_change":"2024-03-24T00:40:01.356693+0000","last_active":"2024-03-24T00:40:02.327238+0000","last_peered":"2024-03-24T00:40:02.327238+0000","last_clean":"2024-03-24T00:40:02.327238+0000","last_became_active":"2024-03-24T00:40:01.356306+0000","last_became_peered":"2024-03-24T00:40:01.356306+0000","last_unstale":"2024-03-24T00:40:02.327238+0000","last_undegraded":"2024-03-24T00:40:02.327238+0000","last_fullsized":"2024-03-24T00:40:02.327238+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:25:54.080003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.327257+0000","last_change":"2024-03-24T00:40:01.354392+0000","last_active":"2024-03-24T00:40:02.327257+0000","last_peered":"2024-03-24T00:40:02.327257+0000","last_clean":"2024-03-24T00:40:02.327257+0000","last_became_active":"2024-03-24T00:40:01.354111+0000","last_became_peered":"2024-03-24T00:40:01.354111+0000","last_unstale":"2024-03-24T00:40:02.327257+0000","last_undegraded":"2024-03-24T00:40:02.327257+0000","last_fullsized":"2024-03-24T00:40:02.327257+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:30:01.709222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685016+0000","last_change":"2024-03-24T00:40:01.354777+0000","last_active":"2024-03-24T00:40:02.685016+0000","last_peered":"2024-03-24T00:40:02.685016+0000","last_clean":"2024-03-24T00:40:02.685016+0000","last_became_active":"2024-03-24T00:40:01.354507+0000","last_became_peered":"2024-03-24T00:40:01.354507+0000","last_unstale":"2024-03-24T00:40:02.685016+0000","last_undegraded":"2024-03-24T00:40:02.685016+0000","last_fullsized":"2024-03-24T00:40:02.685016+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:11:12.174614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685571+0000","last_change":"2024-03-24T00:40:01.356643+0000","last_active":"2024-03-24T00:40:02.685571+0000","last_peered":"2024-03-24T00:40:02.685571+0000","last_clean":"2024-03-24T00:40:02.685571+0000","last_became_active":"2024-03-24T00:40:01.355610+0000","last_became_peered":"2024-03-24T00:40:01.355610+0000","last_unstale":"2024-03-24T00:40:02.685571+0000","last_undegraded":"2024-03-24T00:40:02.685571+0000","last_fullsized":"2024-03-24T00:40:02.685571+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:31:30.760690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685511+0000","last_change":"2024-03-24T00:40:01.349254+0000","last_active":"2024-03-24T00:40:02.685511+0000","last_peered":"2024-03-24T00:40:02.685511+0000","last_clean":"2024-03-24T00:40:02.685511+0000","last_became_active":"2024-03-24T00:40:01.348965+0000","last_became_peered":"2024-03-24T00:40:01.348965+0000","last_unstale":"2024-03-24T00:40:02.685511+0000","last_undegraded":"2024-03-24T00:40:02.685511+0000","last_fullsized":"2024-03-24T00:40:02.685511+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:31:38.754178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.329759+0000","last_change":"2024-03-24T00:40:01.346529+0000","last_active":"2024-03-24T00:40:02.329759+0000","last_peered":"2024-03-24T00:40:02.329759+0000","last_clean":"2024-03-24T00:40:02.329759+0000","last_became_active":"2024-03-24T00:40:01.346262+0000","last_became_peered":"2024-03-24T00:40:01.346262+0000","last_unstale":"2024-03-24T00:40:02.329759+0000","last_undegraded":"2024-03-24T00:40:02.329759+0000","last_fullsized":"2024-03-24T00:40:02.329759+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:40:42.216343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685222+0000","last_change":"2024-03-24T00:40:01.348561+0000","last_active":"2024-03-24T00:40:02.685222+0000","last_peered":"2024-03-24T00:40:02.685222+0000","last_clean":"2024-03-24T00:40:02.685222+0000","last_became_active":"2024-03-24T00:40:01.348304+0000","last_became_peered":"2024-03-24T00:40:01.348304+0000","last_unstale":"2024-03-24T00:40:02.685222+0000","last_undegraded":"2024-03-24T00:40:02.685222+0000","last_fullsized":"2024-03-24T00:40:02.685222+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:30:16.536487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.327350+0000","last_change":"2024-03-24T00:40:01.353646+0000","last_active":"2024-03-24T00:40:02.327350+0000","last_peered":"2024-03-24T00:40:02.327350+0000","last_clean":"2024-03-24T00:40:02.327350+0000","last_became_active":"2024-03-24T00:40:01.352758+0000","last_became_peered":"2024-03-24T00:40:01.352758+0000","last_unstale":"2024-03-24T00:40:02.327350+0000","last_undegraded":"2024-03-24T00:40:02.327350+0000","last_fullsized":"2024-03-24T00:40:02.327350+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:27:49.593844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.327514+0000","last_change":"2024-03-24T00:40:01.354943+0000","last_active":"2024-03-24T00:40:02.327514+0000","last_peered":"2024-03-24T00:40:02.327514+0000","last_clean":"2024-03-24T00:40:02.327514+0000","last_became_active":"2024-03-24T00:40:01.354452+0000","last_became_peered":"2024-03-24T00:40:01.354452+0000","last_unstale":"2024-03-24T00:40:02.327514+0000","last_undegraded":"2024-03-24T00:40:02.327514+0000","last_fullsized":"2024-03-24T00:40:02.327514+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:51.086346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.684979+0000","last_change":"2024-03-24T00:40:01.345157+0000","last_active":"2024-03-24T00:40:02.684979+0000","last_peered":"2024-03-24T00:40:02.684979+0000","last_clean":"2024-03-24T00:40:02.684979+0000","last_became_active":"2024-03-24T00:40:01.344404+0000","last_became_peered":"2024-03-24T00:40:01.344404+0000","last_unstale":"2024-03-24T00:40:02.684979+0000","last_undegraded":"2024-03-24T00:40:02.684979+0000","last_fullsized":"2024-03-24T00:40:02.684979+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:29:16.074361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.329561+0000","last_change":"2024-03-24T00:40:01.337319+0000","last_active":"2024-03-24T00:40:02.329561+0000","last_peered":"2024-03-24T00:40:02.329561+0000","last_clean":"2024-03-24T00:40:02.329561+0000","last_became_active":"2024-03-24T00:40:01.336989+0000","last_became_peered":"2024-03-24T00:40:01.336989+0000","last_unstale":"2024-03-24T00:40:02.329561+0000","last_undegraded":"2024-03-24T00:40:02.329561+0000","last_fullsized":"2024-03-24T00:40:02.329561+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:11:33.082463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685282+0000","last_change":"2024-03-24T00:40:01.349701+0000","last_active":"2024-03-24T00:40:02.685282+0000","last_peered":"2024-03-24T00:40:02.685282+0000","last_clean":"2024-03-24T00:40:02.685282+0000","last_became_active":"2024-03-24T00:40:01.349448+0000","last_became_peered":"2024-03-24T00:40:01.349448+0000","last_unstale":"2024-03-24T00:40:02.685282+0000","last_undegraded":"2024-03-24T00:40:02.685282+0000","last_fullsized":"2024-03-24T00:40:02.685282+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:04:47.673221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685589+0000","last_change":"2024-03-24T00:40:01.356229+0000","last_active":"2024-03-24T00:40:02.685589+0000","last_peered":"2024-03-24T00:40:02.685589+0000","last_clean":"2024-03-24T00:40:02.685589+0000","last_became_active":"2024-03-24T00:40:01.355953+0000","last_became_peered":"2024-03-24T00:40:01.355953+0000","last_unstale":"2024-03-24T00:40:02.685589+0000","last_undegraded":"2024-03-24T00:40:02.685589+0000","last_fullsized":"2024-03-24T00:40:02.685589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:23:33.705793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685303+0000","last_change":"2024-03-24T00:40:01.338189+0000","last_active":"2024-03-24T00:40:02.685303+0000","last_peered":"2024-03-24T00:40:02.685303+0000","last_clean":"2024-03-24T00:40:02.685303+0000","last_became_active":"2024-03-24T00:40:01.337910+0000","last_became_peered":"2024-03-24T00:40:01.337910+0000","last_unstale":"2024-03-24T00:40:02.685303+0000","last_undegraded":"2024-03-24T00:40:02.685303+0000","last_fullsized":"2024-03-24T00:40:02.685303+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:18:05.164294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.327590+0000","last_change":"2024-03-24T00:40:01.353130+0000","last_active":"2024-03-24T00:40:02.327590+0000","last_peered":"2024-03-24T00:40:02.327590+0000","last_clean":"2024-03-24T00:40:02.327590+0000","last_became_active":"2024-03-24T00:40:01.352852+0000","last_became_peered":"2024-03-24T00:40:01.352852+0000","last_unstale":"2024-03-24T00:40:02.327590+0000","last_undegraded":"2024-03-24T00:40:02.327590+0000","last_fullsized":"2024-03-24T00:40:02.327590+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:25:10.580085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685299+0000","last_change":"2024-03-24T00:40:01.345986+0000","last_active":"2024-03-24T00:40:02.685299+0000","last_peered":"2024-03-24T00:40:02.685299+0000","last_clean":"2024-03-24T00:40:02.685299+0000","last_became_active":"2024-03-24T00:40:01.345705+0000","last_became_peered":"2024-03-24T00:40:01.345705+0000","last_unstale":"2024-03-24T00:40:02.685299+0000","last_undegraded":"2024-03-24T00:40:02.685299+0000","last_fullsized":"2024-03-24T00:40:02.685299+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:28:36.656275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685595+0000","last_change":"2024-03-24T00:40:01.331420+0000","last_active":"2024-03-24T00:40:02.685595+0000","last_peered":"2024-03-24T00:40:02.685595+0000","last_clean":"2024-03-24T00:40:02.685595+0000","last_became_active":"2024-03-24T00:40:01.331128+0000","last_became_peered":"2024-03-24T00:40:01.331128+0000","last_unstale":"2024-03-24T00:40:02.685595+0000","last_undegraded":"2024-03-24T00:40:02.685595+0000","last_fullsized":"2024-03-24T00:40:02.685595+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:48:50.540849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685015+0000","last_change":"2024-03-24T00:40:01.344038+0000","last_active":"2024-03-24T00:40:02.685015+0000","last_peered":"2024-03-24T00:40:02.685015+0000","last_clean":"2024-03-24T00:40:02.685015+0000","last_became_active":"2024-03-24T00:40:01.343741+0000","last_became_peered":"2024-03-24T00:40:01.343741+0000","last_unstale":"2024-03-24T00:40:02.685015+0000","last_undegraded":"2024-03-24T00:40:02.685015+0000","last_fullsized":"2024-03-24T00:40:02.685015+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:25:54.080003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.684875+0000","last_change":"2024-03-24T00:40:01.344268+0000","last_active":"2024-03-24T00:40:02.684875+0000","last_peered":"2024-03-24T00:40:02.684875+0000","last_clean":"2024-03-24T00:40:02.684875+0000","last_became_active":"2024-03-24T00:40:01.343972+0000","last_became_peered":"2024-03-24T00:40:01.343972+0000","last_unstale":"2024-03-24T00:40:02.684875+0000","last_undegraded":"2024-03-24T00:40:02.684875+0000","last_fullsized":"2024-03-24T00:40:02.684875+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:25:54.080003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685342+0000","last_change":"2024-03-24T00:40:01.355848+0000","last_active":"2024-03-24T00:40:02.685342+0000","last_peered":"2024-03-24T00:40:02.685342+0000","last_clean":"2024-03-24T00:40:02.685342+0000","last_became_active":"2024-03-24T00:40:01.355593+0000","last_became_peered":"2024-03-24T00:40:01.355593+0000","last_unstale":"2024-03-24T00:40:02.685342+0000","last_undegraded":"2024-03-24T00:40:02.685342+0000","last_fullsized":"2024-03-24T00:40:02.685342+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:34:52.148336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"38'4","reported_seq":18,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685329+0000","last_change":"2024-03-24T00:40:00.346759+0000","last_active":"2024-03-24T00:40:02.685329+0000","last_peered":"2024-03-24T00:40:02.685329+0000","last_clean":"2024-03-24T00:40:02.685329+0000","last_became_active":"2024-03-24T00:40:00.346366+0000","last_became_peered":"2024-03-24T00:40:00.346366+0000","last_unstale":"2024-03-24T00:40:02.685329+0000","last_undegraded":"2024-03-24T00:40:02.685329+0000","last_fullsized":"2024-03-24T00:40:02.685329+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:09:00.112357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685511+0000","last_change":"2024-03-24T00:40:01.356499+0000","last_active":"2024-03-24T00:40:02.685511+0000","last_peered":"2024-03-24T00:40:02.685511+0000","last_clean":"2024-03-24T00:40:02.685511+0000","last_became_active":"2024-03-24T00:40:01.356224+0000","last_became_peered":"2024-03-24T00:40:01.356224+0000","last_unstale":"2024-03-24T00:40:02.685511+0000","last_undegraded":"2024-03-24T00:40:02.685511+0000","last_fullsized":"2024-03-24T00:40:02.685511+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:47:44.580768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.684876+0000","last_change":"2024-03-24T00:40:00.338722+0000","last_active":"2024-03-24T00:40:02.684876+0000","last_peered":"2024-03-24T00:40:02.684876+0000","last_clean":"2024-03-24T00:40:02.684876+0000","last_became_active":"2024-03-24T00:40:00.338449+0000","last_became_peered":"2024-03-24T00:40:00.338449+0000","last_unstale":"2024-03-24T00:40:02.684876+0000","last_undegraded":"2024-03-24T00:40:02.684876+0000","last_fullsized":"2024-03-24T00:40:02.684876+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:50:26.015558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.328071+0000","last_change":"2024-03-24T00:40:01.352156+0000","last_active":"2024-03-24T00:40:02.328071+0000","last_peered":"2024-03-24T00:40:02.328071+0000","last_clean":"2024-03-24T00:40:02.328071+0000","last_became_active":"2024-03-24T00:40:01.351605+0000","last_became_peered":"2024-03-24T00:40:01.351605+0000","last_unstale":"2024-03-24T00:40:02.328071+0000","last_undegraded":"2024-03-24T00:40:02.328071+0000","last_fullsized":"2024-03-24T00:40:02.328071+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:23:33.705793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685584+0000","last_change":"2024-03-24T00:40:00.345390+0000","last_active":"2024-03-24T00:40:02.685584+0000","last_peered":"2024-03-24T00:40:02.685584+0000","last_clean":"2024-03-24T00:40:02.685584+0000","last_became_active":"2024-03-24T00:40:00.345117+0000","last_became_peered":"2024-03-24T00:40:00.345117+0000","last_unstale":"2024-03-24T00:40:02.685584+0000","last_undegraded":"2024-03-24T00:40:02.685584+0000","last_fullsized":"2024-03-24T00:40:02.685584+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:04:46.671168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.326684+0000","last_change":"2024-03-24T00:40:01.339489+0000","last_active":"2024-03-24T00:40:02.326684+0000","last_peered":"2024-03-24T00:40:02.326684+0000","last_clean":"2024-03-24T00:40:02.326684+0000","last_became_active":"2024-03-24T00:40:01.338052+0000","last_became_peered":"2024-03-24T00:40:01.338052+0000","last_unstale":"2024-03-24T00:40:02.326684+0000","last_undegraded":"2024-03-24T00:40:02.326684+0000","last_fullsized":"2024-03-24T00:40:02.326684+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:56:17.522174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.327735+0000","last_change":"2024-03-24T00:40:00.348606+0000","last_active":"2024-03-24T00:40:02.327735+0000","last_peered":"2024-03-24T00:40:02.327735+0000","last_clean":"2024-03-24T00:40:02.327735+0000","last_became_active":"2024-03-24T00:40:00.348365+0000","last_became_peered":"2024-03-24T00:40:00.348365+0000","last_unstale":"2024-03-24T00:40:02.327735+0000","last_undegraded":"2024-03-24T00:40:02.327735+0000","last_fullsized":"2024-03-24T00:40:02.327735+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:30:15.534434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.684685+0000","last_change":"2024-03-24T00:40:01.351153+0000","last_active":"2024-03-24T00:40:02.684685+0000","last_peered":"2024-03-24T00:40:02.684685+0000","last_clean":"2024-03-24T00:40:02.684685+0000","last_became_active":"2024-03-24T00:40:01.350878+0000","last_became_peered":"2024-03-24T00:40:01.350878+0000","last_unstale":"2024-03-24T00:40:02.684685+0000","last_undegraded":"2024-03-24T00:40:02.684685+0000","last_fullsized":"2024-03-24T00:40:02.684685+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:29:16.074361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.327399+0000","last_change":"2024-03-24T00:40:00.342836+0000","last_active":"2024-03-24T00:40:02.327399+0000","last_peered":"2024-03-24T00:40:02.327399+0000","last_clean":"2024-03-24T00:40:02.327399+0000","last_became_active":"2024-03-24T00:40:00.342550+0000","last_became_peered":"2024-03-24T00:40:00.342550+0000","last_unstale":"2024-03-24T00:40:02.327399+0000","last_undegraded":"2024-03-24T00:40:02.327399+0000","last_fullsized":"2024-03-24T00:40:02.327399+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:30:15.534434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.327069+0000","last_change":"2024-03-24T00:40:01.353462+0000","last_active":"2024-03-24T00:40:02.327069+0000","last_peered":"2024-03-24T00:40:02.327069+0000","last_clean":"2024-03-24T00:40:02.327069+0000","last_became_active":"2024-03-24T00:40:01.353183+0000","last_became_peered":"2024-03-24T00:40:01.353183+0000","last_unstale":"2024-03-24T00:40:02.327069+0000","last_undegraded":"2024-03-24T00:40:02.327069+0000","last_fullsized":"2024-03-24T00:40:02.327069+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:31:30.760690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.684915+0000","last_change":"2024-03-24T00:40:00.346329+0000","last_active":"2024-03-24T00:40:02.684915+0000","last_peered":"2024-03-24T00:40:02.684915+0000","last_clean":"2024-03-24T00:40:02.684915+0000","last_became_active":"2024-03-24T00:40:00.346051+0000","last_became_peered":"2024-03-24T00:40:00.346051+0000","last_unstale":"2024-03-24T00:40:02.684915+0000","last_undegraded":"2024-03-24T00:40:02.684915+0000","last_fullsized":"2024-03-24T00:40:02.684915+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:06:44.733235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.327708+0000","last_change":"2024-03-24T00:40:01.357619+0000","last_active":"2024-03-24T00:40:02.327708+0000","last_peered":"2024-03-24T00:40:02.327708+0000","last_clean":"2024-03-24T00:40:02.327708+0000","last_became_active":"2024-03-24T00:40:01.356944+0000","last_became_peered":"2024-03-24T00:40:01.356944+0000","last_unstale":"2024-03-24T00:40:02.327708+0000","last_undegraded":"2024-03-24T00:40:02.327708+0000","last_fullsized":"2024-03-24T00:40:02.327708+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:34:52.148336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685257+0000","last_change":"2024-03-24T00:40:00.341907+0000","last_active":"2024-03-24T00:40:02.685257+0000","last_peered":"2024-03-24T00:40:02.685257+0000","last_clean":"2024-03-24T00:40:02.685257+0000","last_became_active":"2024-03-24T00:40:00.341643+0000","last_became_peered":"2024-03-24T00:40:00.341643+0000","last_unstale":"2024-03-24T00:40:02.685257+0000","last_undegraded":"2024-03-24T00:40:02.685257+0000","last_fullsized":"2024-03-24T00:40:02.685257+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:30:15.534434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.326459+0000","last_change":"2024-03-24T00:40:01.344561+0000","last_active":"2024-03-24T00:40:02.326459+0000","last_peered":"2024-03-24T00:40:02.326459+0000","last_clean":"2024-03-24T00:40:02.326459+0000","last_became_active":"2024-03-24T00:40:01.343777+0000","last_became_peered":"2024-03-24T00:40:01.343777+0000","last_unstale":"2024-03-24T00:40:02.326459+0000","last_undegraded":"2024-03-24T00:40:02.326459+0000","last_fullsized":"2024-03-24T00:40:02.326459+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:09:57.767219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.326786+0000","last_change":"2024-03-24T00:40:00.346231+0000","last_active":"2024-03-24T00:40:02.326786+0000","last_peered":"2024-03-24T00:40:02.326786+0000","last_clean":"2024-03-24T00:40:02.326786+0000","last_became_active":"2024-03-24T00:40:00.345948+0000","last_became_peered":"2024-03-24T00:40:00.345948+0000","last_unstale":"2024-03-24T00:40:02.326786+0000","last_undegraded":"2024-03-24T00:40:02.326786+0000","last_fullsized":"2024-03-24T00:40:02.326786+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:40:41.214290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.329742+0000","last_change":"2024-03-24T00:40:01.349328+0000","last_active":"2024-03-24T00:40:02.329742+0000","last_peered":"2024-03-24T00:40:02.329742+0000","last_clean":"2024-03-24T00:40:02.329742+0000","last_became_active":"2024-03-24T00:40:01.348743+0000","last_became_peered":"2024-03-24T00:40:01.348743+0000","last_unstale":"2024-03-24T00:40:02.329742+0000","last_undegraded":"2024-03-24T00:40:02.329742+0000","last_fullsized":"2024-03-24T00:40:02.329742+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:27:39.492378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.326712+0000","last_change":"2024-03-24T00:40:00.343185+0000","last_active":"2024-03-24T00:40:02.326712+0000","last_peered":"2024-03-24T00:40:02.326712+0000","last_clean":"2024-03-24T00:40:02.326712+0000","last_became_active":"2024-03-24T00:40:00.342931+0000","last_became_peered":"2024-03-24T00:40:00.342931+0000","last_unstale":"2024-03-24T00:40:02.326712+0000","last_undegraded":"2024-03-24T00:40:02.326712+0000","last_fullsized":"2024-03-24T00:40:02.326712+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:23:32.703740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685526+0000","last_change":"2024-03-24T00:40:01.344896+0000","last_active":"2024-03-24T00:40:02.685526+0000","last_peered":"2024-03-24T00:40:02.685526+0000","last_clean":"2024-03-24T00:40:02.685526+0000","last_became_active":"2024-03-24T00:40:01.344627+0000","last_became_peered":"2024-03-24T00:40:01.344627+0000","last_unstale":"2024-03-24T00:40:02.685526+0000","last_undegraded":"2024-03-24T00:40:02.685526+0000","last_fullsized":"2024-03-24T00:40:02.685526+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:44:25.432200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.328239+0000","last_change":"2024-03-24T00:40:00.337581+0000","last_active":"2024-03-24T00:40:02.328239+0000","last_peered":"2024-03-24T00:40:02.328239+0000","last_clean":"2024-03-24T00:40:02.328239+0000","last_became_active":"2024-03-24T00:40:00.336273+0000","last_became_peered":"2024-03-24T00:40:00.336273+0000","last_unstale":"2024-03-24T00:40:02.328239+0000","last_undegraded":"2024-03-24T00:40:02.328239+0000","last_fullsized":"2024-03-24T00:40:02.328239+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:04:46.671168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.684874+0000","last_change":"2024-03-24T00:40:01.346266+0000","last_active":"2024-03-24T00:40:02.684874+0000","last_peered":"2024-03-24T00:40:02.684874+0000","last_clean":"2024-03-24T00:40:02.684874+0000","last_became_active":"2024-03-24T00:40:01.345647+0000","last_became_peered":"2024-03-24T00:40:01.345647+0000","last_unstale":"2024-03-24T00:40:02.684874+0000","last_undegraded":"2024-03-24T00:40:02.684874+0000","last_fullsized":"2024-03-24T00:40:02.684874+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:25:54.080003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"38'3","reported_seq":17,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.327966+0000","last_change":"2024-03-24T00:40:00.337995+0000","last_active":"2024-03-24T00:40:02.327966+0000","last_peered":"2024-03-24T00:40:02.327966+0000","last_clean":"2024-03-24T00:40:02.327966+0000","last_became_active":"2024-03-24T00:40:00.336199+0000","last_became_peered":"2024-03-24T00:40:00.336199+0000","last_unstale":"2024-03-24T00:40:02.327966+0000","last_undegraded":"2024-03-24T00:40:02.327966+0000","last_fullsized":"2024-03-24T00:40:02.327966+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:05:49.943101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.327123+0000","last_change":"2024-03-24T00:40:01.354792+0000","last_active":"2024-03-24T00:40:02.327123+0000","last_peered":"2024-03-24T00:40:02.327123+0000","last_clean":"2024-03-24T00:40:02.327123+0000","last_became_active":"2024-03-24T00:40:01.354518+0000","last_became_peered":"2024-03-24T00:40:01.354518+0000","last_unstale":"2024-03-24T00:40:02.327123+0000","last_undegraded":"2024-03-24T00:40:02.327123+0000","last_fullsized":"2024-03-24T00:40:02.327123+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:29:16.074361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"38'6","reported_seq":20,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.326390+0000","last_change":"2024-03-24T00:40:00.338540+0000","last_active":"2024-03-24T00:40:02.326390+0000","last_peered":"2024-03-24T00:40:02.326390+0000","last_clean":"2024-03-24T00:40:02.326390+0000","last_became_active":"2024-03-24T00:40:00.338230+0000","last_became_peered":"2024-03-24T00:40:00.338230+0000","last_unstale":"2024-03-24T00:40:02.326390+0000","last_undegraded":"2024-03-24T00:40:02.326390+0000","last_fullsized":"2024-03-24T00:40:02.326390+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:34:51.146284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685105+0000","last_change":"2024-03-24T00:40:01.350796+0000","last_active":"2024-03-24T00:40:02.685105+0000","last_peered":"2024-03-24T00:40:02.685105+0000","last_clean":"2024-03-24T00:40:02.685105+0000","last_became_active":"2024-03-24T00:40:01.350249+0000","last_became_peered":"2024-03-24T00:40:01.350249+0000","last_unstale":"2024-03-24T00:40:02.685105+0000","last_undegraded":"2024-03-24T00:40:02.685105+0000","last_fullsized":"2024-03-24T00:40:02.685105+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:44:25.432200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.326548+0000","last_change":"2024-03-24T00:40:00.344040+0000","last_active":"2024-03-24T00:40:02.326548+0000","last_peered":"2024-03-24T00:40:02.326548+0000","last_clean":"2024-03-24T00:40:02.326548+0000","last_became_active":"2024-03-24T00:40:00.343790+0000","last_became_peered":"2024-03-24T00:40:00.343790+0000","last_unstale":"2024-03-24T00:40:02.326548+0000","last_undegraded":"2024-03-24T00:40:02.326548+0000","last_fullsized":"2024-03-24T00:40:02.326548+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:47:43.578715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.326251+0000","last_change":"2024-03-24T00:40:01.346676+0000","last_active":"2024-03-24T00:40:02.326251+0000","last_peered":"2024-03-24T00:40:02.326251+0000","last_clean":"2024-03-24T00:40:02.326251+0000","last_became_active":"2024-03-24T00:40:01.346398+0000","last_became_peered":"2024-03-24T00:40:01.346398+0000","last_unstale":"2024-03-24T00:40:02.326251+0000","last_undegraded":"2024-03-24T00:40:02.326251+0000","last_fullsized":"2024-03-24T00:40:02.326251+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:42:29.481647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685205+0000","last_change":"2024-03-24T00:40:00.343886+0000","last_active":"2024-03-24T00:40:02.685205+0000","last_peered":"2024-03-24T00:40:02.685205+0000","last_clean":"2024-03-24T00:40:02.685205+0000","last_became_active":"2024-03-24T00:40:00.343624+0000","last_became_peered":"2024-03-24T00:40:00.343624+0000","last_unstale":"2024-03-24T00:40:02.685205+0000","last_undegraded":"2024-03-24T00:40:02.685205+0000","last_fullsized":"2024-03-24T00:40:02.685205+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:05:49.943101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685089+0000","last_change":"2024-03-24T00:40:01.343214+0000","last_active":"2024-03-24T00:40:02.685089+0000","last_peered":"2024-03-24T00:40:02.685089+0000","last_clean":"2024-03-24T00:40:02.685089+0000","last_became_active":"2024-03-24T00:40:01.342847+0000","last_became_peered":"2024-03-24T00:40:01.342847+0000","last_unstale":"2024-03-24T00:40:02.685089+0000","last_undegraded":"2024-03-24T00:40:02.685089+0000","last_fullsized":"2024-03-24T00:40:02.685089+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:51.086346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.684866+0000","last_change":"2024-03-24T00:40:00.345927+0000","last_active":"2024-03-24T00:40:02.684866+0000","last_peered":"2024-03-24T00:40:02.684866+0000","last_clean":"2024-03-24T00:40:02.684866+0000","last_became_active":"2024-03-24T00:40:00.345681+0000","last_became_peered":"2024-03-24T00:40:00.345681+0000","last_unstale":"2024-03-24T00:40:02.684866+0000","last_undegraded":"2024-03-24T00:40:02.684866+0000","last_fullsized":"2024-03-24T00:40:02.684866+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:27:53.783517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.326490+0000","last_change":"2024-03-24T00:40:01.347982+0000","last_active":"2024-03-24T00:40:02.326490+0000","last_peered":"2024-03-24T00:40:02.326490+0000","last_clean":"2024-03-24T00:40:02.326490+0000","last_became_active":"2024-03-24T00:40:01.347571+0000","last_became_peered":"2024-03-24T00:40:01.347571+0000","last_unstale":"2024-03-24T00:40:02.326490+0000","last_undegraded":"2024-03-24T00:40:02.326490+0000","last_fullsized":"2024-03-24T00:40:02.326490+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:07:41.182445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"38'8","reported_seq":22,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.684865+0000","last_change":"2024-03-24T00:40:00.345571+0000","last_active":"2024-03-24T00:40:02.684865+0000","last_peered":"2024-03-24T00:40:02.684865+0000","last_clean":"2024-03-24T00:40:02.684865+0000","last_became_active":"2024-03-24T00:40:00.345288+0000","last_became_peered":"2024-03-24T00:40:00.345288+0000","last_unstale":"2024-03-24T00:40:02.684865+0000","last_undegraded":"2024-03-24T00:40:02.684865+0000","last_fullsized":"2024-03-24T00:40:02.684865+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:11:32.080410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.326458+0000","last_change":"2024-03-24T00:40:01.331287+0000","last_active":"2024-03-24T00:40:02.326458+0000","last_peered":"2024-03-24T00:40:02.326458+0000","last_clean":"2024-03-24T00:40:02.326458+0000","last_became_active":"2024-03-24T00:40:01.330903+0000","last_became_peered":"2024-03-24T00:40:01.330903+0000","last_unstale":"2024-03-24T00:40:02.326458+0000","last_undegraded":"2024-03-24T00:40:02.326458+0000","last_fullsized":"2024-03-24T00:40:02.326458+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:41:50.553798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.328239+0000","last_change":"2024-03-24T00:40:00.345924+0000","last_active":"2024-03-24T00:40:02.328239+0000","last_peered":"2024-03-24T00:40:02.328239+0000","last_clean":"2024-03-24T00:40:02.328239+0000","last_became_active":"2024-03-24T00:40:00.345636+0000","last_became_peered":"2024-03-24T00:40:00.345636+0000","last_unstale":"2024-03-24T00:40:02.328239+0000","last_undegraded":"2024-03-24T00:40:02.328239+0000","last_fullsized":"2024-03-24T00:40:02.328239+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:28:35.654223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.328035+0000","last_change":"2024-03-24T00:40:01.353040+0000","last_active":"2024-03-24T00:40:02.328035+0000","last_peered":"2024-03-24T00:40:02.328035+0000","last_clean":"2024-03-24T00:40:02.328035+0000","last_became_active":"2024-03-24T00:40:01.352658+0000","last_became_peered":"2024-03-24T00:40:01.352658+0000","last_unstale":"2024-03-24T00:40:02.328035+0000","last_undegraded":"2024-03-24T00:40:02.328035+0000","last_fullsized":"2024-03-24T00:40:02.328035+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:15:44.062886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.328077+0000","last_change":"2024-03-24T00:40:00.345548+0000","last_active":"2024-03-24T00:40:02.328077+0000","last_peered":"2024-03-24T00:40:02.328077+0000","last_clean":"2024-03-24T00:40:02.328077+0000","last_became_active":"2024-03-24T00:40:00.345250+0000","last_became_peered":"2024-03-24T00:40:00.345250+0000","last_unstale":"2024-03-24T00:40:02.328077+0000","last_undegraded":"2024-03-24T00:40:02.328077+0000","last_fullsized":"2024-03-24T00:40:02.328077+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:05:49.943101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.328021+0000","last_change":"2024-03-24T00:40:01.351854+0000","last_active":"2024-03-24T00:40:02.328021+0000","last_peered":"2024-03-24T00:40:02.328021+0000","last_clean":"2024-03-24T00:40:02.328021+0000","last_became_active":"2024-03-24T00:40:01.351282+0000","last_became_peered":"2024-03-24T00:40:01.351282+0000","last_unstale":"2024-03-24T00:40:02.328021+0000","last_undegraded":"2024-03-24T00:40:02.328021+0000","last_fullsized":"2024-03-24T00:40:02.328021+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:47:44.580768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.329740+0000","last_change":"2024-03-24T00:40:00.335501+0000","last_active":"2024-03-24T00:40:02.329740+0000","last_peered":"2024-03-24T00:40:02.329740+0000","last_clean":"2024-03-24T00:40:02.329740+0000","last_became_active":"2024-03-24T00:40:00.335233+0000","last_became_peered":"2024-03-24T00:40:00.335233+0000","last_unstale":"2024-03-24T00:40:02.329740+0000","last_undegraded":"2024-03-24T00:40:02.329740+0000","last_fullsized":"2024-03-24T00:40:02.329740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:21:02.704842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.329818+0000","last_change":"2024-03-24T00:40:01.334780+0000","last_active":"2024-03-24T00:40:02.329818+0000","last_peered":"2024-03-24T00:40:02.329818+0000","last_clean":"2024-03-24T00:40:02.329818+0000","last_became_active":"2024-03-24T00:40:01.333949+0000","last_became_peered":"2024-03-24T00:40:01.333949+0000","last_unstale":"2024-03-24T00:40:02.329818+0000","last_undegraded":"2024-03-24T00:40:02.329818+0000","last_fullsized":"2024-03-24T00:40:02.329818+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:50:27.017610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685335+0000","last_change":"2024-03-24T00:40:00.345775+0000","last_active":"2024-03-24T00:40:02.685335+0000","last_peered":"2024-03-24T00:40:02.685335+0000","last_clean":"2024-03-24T00:40:02.685335+0000","last_became_active":"2024-03-24T00:40:00.345458+0000","last_became_peered":"2024-03-24T00:40:00.345458+0000","last_unstale":"2024-03-24T00:40:02.685335+0000","last_undegraded":"2024-03-24T00:40:02.685335+0000","last_fullsized":"2024-03-24T00:40:02.685335+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:28:35.654223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685361+0000","last_change":"2024-03-24T00:40:01.345730+0000","last_active":"2024-03-24T00:40:02.685361+0000","last_peered":"2024-03-24T00:40:02.685361+0000","last_clean":"2024-03-24T00:40:02.685361+0000","last_became_active":"2024-03-24T00:40:01.345437+0000","last_became_peered":"2024-03-24T00:40:01.345437+0000","last_unstale":"2024-03-24T00:40:02.685361+0000","last_undegraded":"2024-03-24T00:40:02.685361+0000","last_fullsized":"2024-03-24T00:40:02.685361+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:47:44.580768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.329483+0000","last_change":"2024-03-24T00:40:00.347310+0000","last_active":"2024-03-24T00:40:02.329483+0000","last_peered":"2024-03-24T00:40:02.329483+0000","last_clean":"2024-03-24T00:40:02.329483+0000","last_became_active":"2024-03-24T00:40:00.347030+0000","last_became_peered":"2024-03-24T00:40:00.347030+0000","last_unstale":"2024-03-24T00:40:02.329483+0000","last_undegraded":"2024-03-24T00:40:02.329483+0000","last_fullsized":"2024-03-24T00:40:02.329483+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:16:31.720628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685032+0000","last_change":"2024-03-24T00:40:01.346045+0000","last_active":"2024-03-24T00:40:02.685032+0000","last_peered":"2024-03-24T00:40:02.685032+0000","last_clean":"2024-03-24T00:40:02.685032+0000","last_became_active":"2024-03-24T00:40:01.345693+0000","last_became_peered":"2024-03-24T00:40:01.345693+0000","last_unstale":"2024-03-24T00:40:02.685032+0000","last_undegraded":"2024-03-24T00:40:02.685032+0000","last_fullsized":"2024-03-24T00:40:02.685032+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:47:44.580768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.329491+0000","last_change":"2024-03-24T00:40:00.346403+0000","last_active":"2024-03-24T00:40:02.329491+0000","last_peered":"2024-03-24T00:40:02.329491+0000","last_clean":"2024-03-24T00:40:02.329491+0000","last_became_active":"2024-03-24T00:40:00.346095+0000","last_became_peered":"2024-03-24T00:40:00.346095+0000","last_unstale":"2024-03-24T00:40:02.329491+0000","last_undegraded":"2024-03-24T00:40:02.329491+0000","last_fullsized":"2024-03-24T00:40:02.329491+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:49:32.598338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.327180+0000","last_change":"2024-03-24T00:40:01.351042+0000","last_active":"2024-03-24T00:40:02.327180+0000","last_peered":"2024-03-24T00:40:02.327180+0000","last_clean":"2024-03-24T00:40:02.327180+0000","last_became_active":"2024-03-24T00:40:01.350750+0000","last_became_peered":"2024-03-24T00:40:01.350750+0000","last_unstale":"2024-03-24T00:40:02.327180+0000","last_undegraded":"2024-03-24T00:40:02.327180+0000","last_fullsized":"2024-03-24T00:40:02.327180+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:04:37.096161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685035+0000","last_change":"2024-03-24T00:40:00.335633+0000","last_active":"2024-03-24T00:40:02.685035+0000","last_peered":"2024-03-24T00:40:02.685035+0000","last_clean":"2024-03-24T00:40:02.685035+0000","last_became_active":"2024-03-24T00:40:00.335348+0000","last_became_peered":"2024-03-24T00:40:00.335348+0000","last_unstale":"2024-03-24T00:40:02.685035+0000","last_undegraded":"2024-03-24T00:40:02.685035+0000","last_fullsized":"2024-03-24T00:40:02.685035+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:40:41.214290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.326627+0000","last_change":"2024-03-24T00:40:01.345061+0000","last_active":"2024-03-24T00:40:02.326627+0000","last_peered":"2024-03-24T00:40:02.326627+0000","last_clean":"2024-03-24T00:40:02.326627+0000","last_became_active":"2024-03-24T00:40:01.344793+0000","last_became_peered":"2024-03-24T00:40:01.344793+0000","last_unstale":"2024-03-24T00:40:02.326627+0000","last_undegraded":"2024-03-24T00:40:02.326627+0000","last_fullsized":"2024-03-24T00:40:02.326627+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:17:43.539404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.329538+0000","last_change":"2024-03-24T00:40:00.338765+0000","last_active":"2024-03-24T00:40:02.329538+0000","last_peered":"2024-03-24T00:40:02.329538+0000","last_clean":"2024-03-24T00:40:02.329538+0000","last_became_active":"2024-03-24T00:40:00.337416+0000","last_became_peered":"2024-03-24T00:40:00.337416+0000","last_unstale":"2024-03-24T00:40:02.329538+0000","last_undegraded":"2024-03-24T00:40:02.329538+0000","last_fullsized":"2024-03-24T00:40:02.329538+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:38:41.933921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685292+0000","last_change":"2024-03-24T00:40:01.356678+0000","last_active":"2024-03-24T00:40:02.685292+0000","last_peered":"2024-03-24T00:40:02.685292+0000","last_clean":"2024-03-24T00:40:02.685292+0000","last_became_active":"2024-03-24T00:40:01.356395+0000","last_became_peered":"2024-03-24T00:40:01.356395+0000","last_unstale":"2024-03-24T00:40:02.685292+0000","last_undegraded":"2024-03-24T00:40:02.685292+0000","last_fullsized":"2024-03-24T00:40:02.685292+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:51.086346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.328338+0000","last_change":"2024-03-24T00:40:00.342913+0000","last_active":"2024-03-24T00:40:02.328338+0000","last_peered":"2024-03-24T00:40:02.328338+0000","last_clean":"2024-03-24T00:40:02.328338+0000","last_became_active":"2024-03-24T00:40:00.342633+0000","last_became_peered":"2024-03-24T00:40:00.342633+0000","last_unstale":"2024-03-24T00:40:02.328338+0000","last_undegraded":"2024-03-24T00:40:02.328338+0000","last_fullsized":"2024-03-24T00:40:02.328338+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:18:04.162241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685586+0000","last_change":"2024-03-24T00:40:01.343468+0000","last_active":"2024-03-24T00:40:02.685586+0000","last_peered":"2024-03-24T00:40:02.685586+0000","last_clean":"2024-03-24T00:40:02.685586+0000","last_became_active":"2024-03-24T00:40:01.343167+0000","last_became_peered":"2024-03-24T00:40:01.343167+0000","last_unstale":"2024-03-24T00:40:02.685586+0000","last_undegraded":"2024-03-24T00:40:02.685586+0000","last_fullsized":"2024-03-24T00:40:02.685586+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:04:37.096161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"39'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.329731+0000","last_change":"2024-03-24T00:40:00.335226+0000","last_active":"2024-03-24T00:40:02.329731+0000","last_peered":"2024-03-24T00:40:02.329731+0000","last_clean":"2024-03-24T00:40:02.329731+0000","last_became_active":"2024-03-24T00:40:00.334961+0000","last_became_peered":"2024-03-24T00:40:00.334961+0000","last_unstale":"2024-03-24T00:40:02.329731+0000","last_undegraded":"2024-03-24T00:40:02.329731+0000","last_fullsized":"2024-03-24T00:40:02.329731+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:23:45.494377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.326613+0000","last_change":"2024-03-24T00:40:01.349135+0000","last_active":"2024-03-24T00:40:02.326613+0000","last_peered":"2024-03-24T00:40:02.326613+0000","last_clean":"2024-03-24T00:40:02.326613+0000","last_became_active":"2024-03-24T00:40:01.348712+0000","last_became_peered":"2024-03-24T00:40:01.348712+0000","last_unstale":"2024-03-24T00:40:02.326613+0000","last_undegraded":"2024-03-24T00:40:02.326613+0000","last_fullsized":"2024-03-24T00:40:02.326613+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:00:13.987179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.684740+0000","last_change":"2024-03-24T00:40:00.342013+0000","last_active":"2024-03-24T00:40:02.684740+0000","last_peered":"2024-03-24T00:40:02.684740+0000","last_clean":"2024-03-24T00:40:02.684740+0000","last_became_active":"2024-03-24T00:40:00.341732+0000","last_became_peered":"2024-03-24T00:40:00.341732+0000","last_unstale":"2024-03-24T00:40:02.684740+0000","last_undegraded":"2024-03-24T00:40:02.684740+0000","last_fullsized":"2024-03-24T00:40:02.684740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:40:41.214290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.684998+0000","last_change":"2024-03-24T00:40:01.349804+0000","last_active":"2024-03-24T00:40:02.684998+0000","last_peered":"2024-03-24T00:40:02.684998+0000","last_clean":"2024-03-24T00:40:02.684998+0000","last_became_active":"2024-03-24T00:40:01.349542+0000","last_became_peered":"2024-03-24T00:40:01.349542+0000","last_unstale":"2024-03-24T00:40:02.684998+0000","last_undegraded":"2024-03-24T00:40:02.684998+0000","last_fullsized":"2024-03-24T00:40:02.684998+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:34:52.148336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.684813+0000","last_change":"2024-03-24T00:40:00.345415+0000","last_active":"2024-03-24T00:40:02.684813+0000","last_peered":"2024-03-24T00:40:02.684813+0000","last_clean":"2024-03-24T00:40:02.684813+0000","last_became_active":"2024-03-24T00:40:00.345145+0000","last_became_peered":"2024-03-24T00:40:00.345145+0000","last_unstale":"2024-03-24T00:40:02.684813+0000","last_undegraded":"2024-03-24T00:40:02.684813+0000","last_fullsized":"2024-03-24T00:40:02.684813+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:05:49.943101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685368+0000","last_change":"2024-03-24T00:40:01.344953+0000","last_active":"2024-03-24T00:40:02.685368+0000","last_peered":"2024-03-24T00:40:02.685368+0000","last_clean":"2024-03-24T00:40:02.685368+0000","last_became_active":"2024-03-24T00:40:01.344683+0000","last_became_peered":"2024-03-24T00:40:01.344683+0000","last_unstale":"2024-03-24T00:40:02.685368+0000","last_undegraded":"2024-03-24T00:40:02.685368+0000","last_fullsized":"2024-03-24T00:40:02.685368+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:34:52.148336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685583+0000","last_change":"2024-03-24T00:40:00.335784+0000","last_active":"2024-03-24T00:40:02.685583+0000","last_peered":"2024-03-24T00:40:02.685583+0000","last_clean":"2024-03-24T00:40:02.685583+0000","last_became_active":"2024-03-24T00:40:00.335478+0000","last_became_peered":"2024-03-24T00:40:00.335478+0000","last_unstale":"2024-03-24T00:40:02.685583+0000","last_undegraded":"2024-03-24T00:40:02.685583+0000","last_fullsized":"2024-03-24T00:40:02.685583+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:47:40.706718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.327376+0000","last_change":"2024-03-24T00:40:01.352911+0000","last_active":"2024-03-24T00:40:02.327376+0000","last_peered":"2024-03-24T00:40:02.327376+0000","last_clean":"2024-03-24T00:40:02.327376+0000","last_became_active":"2024-03-24T00:40:01.352668+0000","last_became_peered":"2024-03-24T00:40:01.352668+0000","last_unstale":"2024-03-24T00:40:02.327376+0000","last_undegraded":"2024-03-24T00:40:02.327376+0000","last_fullsized":"2024-03-24T00:40:02.327376+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:11:12.174614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.326903+0000","last_change":"2024-03-24T00:40:00.342133+0000","last_active":"2024-03-24T00:40:02.326903+0000","last_peered":"2024-03-24T00:40:02.326903+0000","last_clean":"2024-03-24T00:40:02.326903+0000","last_became_active":"2024-03-24T00:40:00.341821+0000","last_became_peered":"2024-03-24T00:40:00.341821+0000","last_unstale":"2024-03-24T00:40:02.326903+0000","last_undegraded":"2024-03-24T00:40:02.326903+0000","last_fullsized":"2024-03-24T00:40:02.326903+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:50:26.015558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685436+0000","last_change":"2024-03-24T00:40:01.344138+0000","last_active":"2024-03-24T00:40:02.685436+0000","last_peered":"2024-03-24T00:40:02.685436+0000","last_clean":"2024-03-24T00:40:02.685436+0000","last_became_active":"2024-03-24T00:40:01.343857+0000","last_became_peered":"2024-03-24T00:40:01.343857+0000","last_unstale":"2024-03-24T00:40:02.685436+0000","last_undegraded":"2024-03-24T00:40:02.685436+0000","last_fullsized":"2024-03-24T00:40:02.685436+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:23:33.705793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.326192+0000","last_change":"2024-03-24T00:40:00.344855+0000","last_active":"2024-03-24T00:40:02.326192+0000","last_peered":"2024-03-24T00:40:02.326192+0000","last_clean":"2024-03-24T00:40:02.326192+0000","last_became_active":"2024-03-24T00:40:00.344605+0000","last_became_peered":"2024-03-24T00:40:00.344605+0000","last_unstale":"2024-03-24T00:40:02.326192+0000","last_undegraded":"2024-03-24T00:40:02.326192+0000","last_fullsized":"2024-03-24T00:40:02.326192+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:50.084293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.328267+0000","last_change":"2024-03-24T00:40:01.355150+0000","last_active":"2024-03-24T00:40:02.328267+0000","last_peered":"2024-03-24T00:40:02.328267+0000","last_clean":"2024-03-24T00:40:02.328267+0000","last_became_active":"2024-03-24T00:40:01.354677+0000","last_became_peered":"2024-03-24T00:40:01.354677+0000","last_unstale":"2024-03-24T00:40:02.328267+0000","last_undegraded":"2024-03-24T00:40:02.328267+0000","last_fullsized":"2024-03-24T00:40:02.328267+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:09:15.867422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685395+0000","last_change":"2024-03-24T00:40:00.341415+0000","last_active":"2024-03-24T00:40:02.685395+0000","last_peered":"2024-03-24T00:40:02.685395+0000","last_clean":"2024-03-24T00:40:02.685395+0000","last_became_active":"2024-03-24T00:40:00.341108+0000","last_became_peered":"2024-03-24T00:40:00.341108+0000","last_unstale":"2024-03-24T00:40:02.685395+0000","last_undegraded":"2024-03-24T00:40:02.685395+0000","last_fullsized":"2024-03-24T00:40:02.685395+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:27:38.490325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":205,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685138+0000","last_change":"2024-03-24T00:38:40.953871+0000","last_active":"2024-03-24T00:40:02.685138+0000","last_peered":"2024-03-24T00:40:02.685138+0000","last_clean":"2024-03-24T00:40:02.685138+0000","last_became_active":"2024-03-24T00:38:40.952926+0000","last_became_peered":"2024-03-24T00:38:40.952926+0000","last_unstale":"2024-03-24T00:40:02.685138+0000","last_undegraded":"2024-03-24T00:40:02.685138+0000","last_fullsized":"2024-03-24T00:40:02.685138+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_clean_scrub_stamp":"2024-03-24T00:38:39.722194+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:05.397257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574870,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28060,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343780,"statfs":{"total":96636764160,"available":96608030720,"internally_reserved":0,"allocated":1060864,"data_stored":789207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574869,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27468,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344372,"statfs":{"total":96636764160,"available":96608636928,"internally_reserved":0,"allocated":454656,"data_stored":197739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574869,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27476,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344364,"statfs":{"total":96636764160,"available":96608628736,"internally_reserved":0,"allocated":458752,"data_stored":197269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574868,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27476,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344364,"statfs":{"total":96636764160,"available":96608628736,"internally_reserved":0,"allocated":458752,"data_stored":197761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574868,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27480,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344360,"statfs":{"total":96636764160,"available":96608624640,"internally_reserved":0,"allocated":466944,"data_stored":198803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574869,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27468,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344372,"statfs":{"total":96636764160,"available":96608636928,"internally_reserved":0,"allocated":454656,"data_stored":197227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27472,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344368,"statfs":{"total":96636764160,"available":96608632832,"internally_reserved":0,"allocated":454656,"data_stored":197247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574868,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28048,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343792,"statfs":{"total":96636764160,"available":96608043008,"internally_reserved":0,"allocated":1048576,"data_stored":787559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:40:09.222 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:40:09.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.339+0000 7f6dd1f9d640 1 Processor -- start 2024-03-24T00:40:09.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.339+0000 7f6dd1f9d640 1 -- start start 2024-03-24T00:40:09.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.339+0000 7f6dd1f9d640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6dcc15def0 0x7f6dcc161a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:09.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.339+0000 7f6dd1f9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6dcc0cf0c0 0x7f6dcc0d2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:09.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.339+0000 7f6dd1f9d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dcc1623c0 con 0x7f6dcc0b9920 2024-03-24T00:40:09.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.339+0000 7f6dd1f9d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6dcc162980 con 0x7f6dcc0cf0c0 2024-03-24T00:40:09.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.339+0000 7f6dd1f9d640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dcc1626a0 con 0x7f6dcc15def0 2024-03-24T00:40:09.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.340+0000 7f6dcbfff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6dcc0b9920 0x7f6dcc0ce9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42410/0 (socket says 172.21.15.67:42410) 2024-03-24T00:40:09.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.340+0000 7f6dcbfff640 1 -- 172.21.15.67:0/1585081253 learned_addr learned my addr 172.21.15.67:0/1585081253 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:40:09.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.340+0000 7f6dd179c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6dcc0cf0c0 0x7f6dcc0d2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:09.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.340+0000 7f6dd0f9b640 1 --2- 172.21.15.67:0/1585081253 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6dcc15def0 0x7f6dcc161a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:09.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.340+0000 7f6dcb7fe640 1 -- 172.21.15.67:0/1585081253 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2820676516 0 0) 0x7f6dcc1623c0 con 0x7f6dcc0b9920 2024-03-24T00:40:09.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.340+0000 7f6dcb7fe640 1 -- 172.21.15.67:0/1585081253 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6dbc003520 con 0x7f6dcc0b9920 2024-03-24T00:40:09.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.340+0000 7f6dd179c640 1 -- 172.21.15.67:0/1585081253 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6dcc15def0 msgr2=0x7f6dcc161a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:09.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.340+0000 7f6dd179c640 1 --2- 172.21.15.67:0/1585081253 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6dcc15def0 0x7f6dcc161a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.340+0000 7f6dd179c640 1 -- 172.21.15.67:0/1585081253 >> v1:172.21.15.67:6789/0 conn(0x7f6dcc0b9920 legacy=0x7f6dcc0ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:09.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.340+0000 7f6dd179c640 1 -- 172.21.15.67:0/1585081253 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dcc0d1e70 con 0x7f6dcc0cf0c0 2024-03-24T00:40:09.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.340+0000 7f6dd0f9b640 1 --2- 172.21.15.67:0/1585081253 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6dcc15def0 0x7f6dcc161a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:40:09.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.341+0000 7f6dd179c640 1 --2- 172.21.15.67:0/1585081253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6dcc0cf0c0 0x7f6dcc0d2670 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f6dc4009d90 tx=0x7f6dc402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39fa9adfffb22385 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:09.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.341+0000 7f6dcb7fe640 1 -- 172.21.15.67:0/1585081253 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dc4039070 con 0x7f6dcc0cf0c0 2024-03-24T00:40:09.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.341+0000 7f6dcb7fe640 1 -- 172.21.15.67:0/1585081253 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dc402cbc0 con 0x7f6dcc0cf0c0 2024-03-24T00:40:09.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.341+0000 7f6dcb7fe640 1 -- 172.21.15.67:0/1585081253 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dc4034440 con 0x7f6dcc0cf0c0 2024-03-24T00:40:09.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.342+0000 7f6dd1f9d640 1 -- 172.21.15.67:0/1585081253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6dcc0cf0c0 msgr2=0x7f6dcc0d2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:09.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.342+0000 7f6dd1f9d640 1 --2- 172.21.15.67:0/1585081253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6dcc0cf0c0 0x7f6dcc0d2670 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f6dc4009d90 tx=0x7f6dc402be10 comp rx=0 tx=0).stop 2024-03-24T00:40:09.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.342+0000 7f6dd1f9d640 1 -- 172.21.15.67:0/1585081253 shutdown_connections 2024-03-24T00:40:09.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.342+0000 7f6dd1f9d640 1 --2- 172.21.15.67:0/1585081253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6dcc0cf0c0 0x7f6dcc0d2670 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.342+0000 7f6dd1f9d640 1 --2- 172.21.15.67:0/1585081253 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6dcc15def0 0x7f6dcc161a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.342+0000 7f6dd1f9d640 1 -- 172.21.15.67:0/1585081253 >> 172.21.15.67:0/1585081253 conn(0x7f6dcc06c760 msgr2=0x7f6dcc06cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:09.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.342+0000 7f6dd1f9d640 1 -- 172.21.15.67:0/1585081253 shutdown_connections 2024-03-24T00:40:09.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.342+0000 7f6dd1f9d640 1 -- 172.21.15.67:0/1585081253 wait complete. 2024-03-24T00:40:09.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.344+0000 7f6dd1f9d640 1 Processor -- start 2024-03-24T00:40:09.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.344+0000 7f6dd1f9d640 1 -- start start 2024-03-24T00:40:09.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.345+0000 7f6dd1f9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6dcc0b9920 0x7f6dcc1f3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:09.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.345+0000 7f6dd1f9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6dcc0cf0c0 0x7f6dcc160570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:09.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.345+0000 7f6dd1f9d640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6dcc15def0 0x7f6dcc0b8000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:09.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.345+0000 7f6dd1f9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dcc161f90 con 0x7f6dcc0cf0c0 2024-03-24T00:40:09.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.345+0000 7f6dd1f9d640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dcc15e2d0 con 0x7f6dcc0b9920 2024-03-24T00:40:09.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.345+0000 7f6dd1f9d640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dcc162110 con 0x7f6dcc15def0 2024-03-24T00:40:09.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.345+0000 7f6dcbfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6dcc0cf0c0 0x7f6dcc160570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:09.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.345+0000 7f6dd179c640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6dcc15def0 0x7f6dcc0b8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:09.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.345+0000 7f6dcbfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6dcc0cf0c0 0x7f6dcc160570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40916/0 (socket says 172.21.15.67:40916) 2024-03-24T00:40:09.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.345+0000 7f6dcbfff640 1 -- 172.21.15.67:0/167340685 learned_addr learned my addr 172.21.15.67:0/167340685 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:09.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.345+0000 7f6dd0f9b640 1 --2- 172.21.15.67:0/167340685 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6dcc0b9920 0x7f6dcc1f3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:09.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.345+0000 7f6dcbfff640 1 -- 172.21.15.67:0/167340685 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6dcc15def0 msgr2=0x7f6dcc0b8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:09.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.345+0000 7f6dcbfff640 1 --2- 172.21.15.67:0/167340685 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6dcc15def0 0x7f6dcc0b8000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.345+0000 7f6dcbfff640 1 -- 172.21.15.67:0/167340685 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6dcc0b9920 msgr2=0x7f6dcc1f3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:09.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.345+0000 7f6dcbfff640 1 --2- 172.21.15.67:0/167340685 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6dcc0b9920 0x7f6dcc1f3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.345+0000 7f6dcbfff640 1 -- 172.21.15.67:0/167340685 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dcc160e30 con 0x7f6dcc0cf0c0 2024-03-24T00:40:09.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.345+0000 7f6dd0f9b640 1 --2- 172.21.15.67:0/167340685 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6dcc0b9920 0x7f6dcc1f3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:40:09.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.346+0000 7f6dcbfff640 1 --2- 172.21.15.67:0/167340685 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6dcc0cf0c0 0x7f6dcc160570 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f6db800c3c0 tx=0x7f6db800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:09.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.346+0000 7f6dc97fa640 1 -- 172.21.15.67:0/167340685 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db8017070 con 0x7f6dcc0cf0c0 2024-03-24T00:40:09.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.346+0000 7f6dc97fa640 1 -- 172.21.15.67:0/167340685 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6db8005ce0 con 0x7f6dcc0cf0c0 2024-03-24T00:40:09.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.346+0000 7f6dd1f9d640 1 -- 172.21.15.67:0/167340685 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dcc161110 con 0x7f6dcc0cf0c0 2024-03-24T00:40:09.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.346+0000 7f6dc97fa640 1 -- 172.21.15.67:0/167340685 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db80072f0 con 0x7f6dcc0cf0c0 2024-03-24T00:40:09.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.346+0000 7f6dd1f9d640 1 -- 172.21.15.67:0/167340685 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dcc0cad50 con 0x7f6dcc0cf0c0 2024-03-24T00:40:09.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.347+0000 7f6dd1f9d640 1 -- 172.21.15.67:0/167340685 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dcc04f400 con 0x7f6dcc0cf0c0 2024-03-24T00:40:09.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.348+0000 7f6dc97fa640 1 -- 172.21.15.67:0/167340685 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6db801a070 con 0x7f6dcc0cf0c0 2024-03-24T00:40:09.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.348+0000 7f6dc97fa640 1 --2- 172.21.15.67:0/167340685 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6db006bbd0 0x7f6db006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:09.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.348+0000 7f6dc97fa640 1 -- 172.21.15.67:0/167340685 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6db8089e70 con 0x7f6dcc0cf0c0 2024-03-24T00:40:09.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.348+0000 7f6dd0f9b640 1 --2- 172.21.15.67:0/167340685 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6db006bbd0 0x7f6db006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:09.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.349+0000 7f6dd0f9b640 1 --2- 172.21.15.67:0/167340685 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6db006bbd0 0x7f6db006e090 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f6dc0009560 tx=0x7f6dc0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:09.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.353+0000 7f6dc97fa640 1 -- 172.21.15.67:0/167340685 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6db8058210 con 0x7f6dcc0cf0c0 2024-03-24T00:40:09.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.531+0000 7f6dd1f9d640 1 -- 172.21.15.67:0/167340685 --> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6dcc0d18f0 con 0x7f6db006bbd0 2024-03-24T00:40:09.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.538+0000 7f6dc97fa640 1 -- 172.21.15.67:0/167340685 <== mgr.4105 v2:172.21.15.72:6832/2768292274 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251958 (secure 0 0 0) 0x7f6dcc0d18f0 con 0x7f6db006bbd0 2024-03-24T00:40:09.541 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:40:09.542 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-24T00:40:09.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.540+0000 7f6dd1f9d640 1 -- 172.21.15.67:0/167340685 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6db006bbd0 msgr2=0x7f6db006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:09.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.540+0000 7f6dd1f9d640 1 --2- 172.21.15.67:0/167340685 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6db006bbd0 0x7f6db006e090 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f6dc0009560 tx=0x7f6dc0009190 comp rx=0 tx=0).stop 2024-03-24T00:40:09.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.540+0000 7f6dd1f9d640 1 -- 172.21.15.67:0/167340685 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6dcc0cf0c0 msgr2=0x7f6dcc160570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:09.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.540+0000 7f6dd1f9d640 1 --2- 172.21.15.67:0/167340685 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6dcc0cf0c0 0x7f6dcc160570 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f6db800c3c0 tx=0x7f6db800c880 comp rx=0 tx=0).stop 2024-03-24T00:40:09.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.540+0000 7f6dd1f9d640 1 -- 172.21.15.67:0/167340685 shutdown_connections 2024-03-24T00:40:09.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.540+0000 7f6dd1f9d640 1 --2- 172.21.15.67:0/167340685 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6db006bbd0 0x7f6db006e090 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.540+0000 7f6dd1f9d640 1 --2- 172.21.15.67:0/167340685 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6dcc15def0 0x7f6dcc0b8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.540+0000 7f6dd1f9d640 1 --2- 172.21.15.67:0/167340685 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6dcc0cf0c0 0x7f6dcc160570 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.540+0000 7f6dd1f9d640 1 --2- 172.21.15.67:0/167340685 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6dcc0b9920 0x7f6dcc1f3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.540+0000 7f6dd1f9d640 1 -- 172.21.15.67:0/167340685 >> 172.21.15.67:0/167340685 conn(0x7f6dcc06c760 msgr2=0x7f6dcc15d1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:09.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.541+0000 7f6dd1f9d640 1 -- 172.21.15.67:0/167340685 shutdown_connections 2024-03-24T00:40:09.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.541+0000 7f6dd1f9d640 1 -- 172.21.15.67:0/167340685 wait complete. 2024-03-24T00:40:09.552 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":77,"stamp":"2024-03-24T00:40:09.339398+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":174,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220948,"kb_used_data":4744,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753772,"statfs":{"total":773094113280,"available":772867862528,"internally_reserved":0,"allocated":4857856,"data_stored":2762812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.014379"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.326693+0000","last_change":"2024-03-24T00:40:01.346232+0000","last_active":"2024-03-24T00:40:02.326693+0000","last_peered":"2024-03-24T00:40:02.326693+0000","last_clean":"2024-03-24T00:40:02.326693+0000","last_became_active":"2024-03-24T00:40:01.345850+0000","last_became_peered":"2024-03-24T00:40:01.345850+0000","last_unstale":"2024-03-24T00:40:02.326693+0000","last_undegraded":"2024-03-24T00:40:02.326693+0000","last_fullsized":"2024-03-24T00:40:02.326693+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:38:18.834926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.326462+0000","last_change":"2024-03-24T00:40:01.334855+0000","last_active":"2024-03-24T00:40:02.326462+0000","last_peered":"2024-03-24T00:40:02.326462+0000","last_clean":"2024-03-24T00:40:02.326462+0000","last_became_active":"2024-03-24T00:40:01.334560+0000","last_became_peered":"2024-03-24T00:40:01.334560+0000","last_unstale":"2024-03-24T00:40:02.326462+0000","last_undegraded":"2024-03-24T00:40:02.326462+0000","last_fullsized":"2024-03-24T00:40:02.326462+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:52:56.182730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.684957+0000","last_change":"2024-03-24T00:40:01.356272+0000","last_active":"2024-03-24T00:40:02.684957+0000","last_peered":"2024-03-24T00:40:02.684957+0000","last_clean":"2024-03-24T00:40:02.684957+0000","last_became_active":"2024-03-24T00:40:01.355995+0000","last_became_peered":"2024-03-24T00:40:01.355995+0000","last_unstale":"2024-03-24T00:40:02.684957+0000","last_undegraded":"2024-03-24T00:40:02.684957+0000","last_fullsized":"2024-03-24T00:40:02.684957+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:51.086346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.329660+0000","last_change":"2024-03-24T00:40:01.349541+0000","last_active":"2024-03-24T00:40:02.329660+0000","last_peered":"2024-03-24T00:40:02.329660+0000","last_clean":"2024-03-24T00:40:02.329660+0000","last_became_active":"2024-03-24T00:40:01.349072+0000","last_became_peered":"2024-03-24T00:40:01.349072+0000","last_unstale":"2024-03-24T00:40:02.329660+0000","last_undegraded":"2024-03-24T00:40:02.329660+0000","last_fullsized":"2024-03-24T00:40:02.329660+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:06:45.735287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.329550+0000","last_change":"2024-03-24T00:40:01.349628+0000","last_active":"2024-03-24T00:40:02.329550+0000","last_peered":"2024-03-24T00:40:02.329550+0000","last_clean":"2024-03-24T00:40:02.329550+0000","last_became_active":"2024-03-24T00:40:01.349371+0000","last_became_peered":"2024-03-24T00:40:01.349371+0000","last_unstale":"2024-03-24T00:40:02.329550+0000","last_undegraded":"2024-03-24T00:40:02.329550+0000","last_fullsized":"2024-03-24T00:40:02.329550+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:09:01.114410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685148+0000","last_change":"2024-03-24T00:40:01.345171+0000","last_active":"2024-03-24T00:40:02.685148+0000","last_peered":"2024-03-24T00:40:02.685148+0000","last_clean":"2024-03-24T00:40:02.685148+0000","last_became_active":"2024-03-24T00:40:01.344909+0000","last_became_peered":"2024-03-24T00:40:01.344909+0000","last_unstale":"2024-03-24T00:40:02.685148+0000","last_undegraded":"2024-03-24T00:40:02.685148+0000","last_fullsized":"2024-03-24T00:40:02.685148+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:11:12.174614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.326916+0000","last_change":"2024-03-24T00:40:01.350464+0000","last_active":"2024-03-24T00:40:02.326916+0000","last_peered":"2024-03-24T00:40:02.326916+0000","last_clean":"2024-03-24T00:40:02.326916+0000","last_became_active":"2024-03-24T00:40:01.350212+0000","last_became_peered":"2024-03-24T00:40:01.350212+0000","last_unstale":"2024-03-24T00:40:02.326916+0000","last_undegraded":"2024-03-24T00:40:02.326916+0000","last_fullsized":"2024-03-24T00:40:02.326916+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:44:25.432200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.326251+0000","last_change":"2024-03-24T00:40:01.356804+0000","last_active":"2024-03-24T00:40:02.326251+0000","last_peered":"2024-03-24T00:40:02.326251+0000","last_clean":"2024-03-24T00:40:02.326251+0000","last_became_active":"2024-03-24T00:40:01.356513+0000","last_became_peered":"2024-03-24T00:40:01.356513+0000","last_unstale":"2024-03-24T00:40:02.326251+0000","last_undegraded":"2024-03-24T00:40:02.326251+0000","last_fullsized":"2024-03-24T00:40:02.326251+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:09:15.867422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685100+0000","last_change":"2024-03-24T00:40:01.356194+0000","last_active":"2024-03-24T00:40:02.685100+0000","last_peered":"2024-03-24T00:40:02.685100+0000","last_clean":"2024-03-24T00:40:02.685100+0000","last_became_active":"2024-03-24T00:40:01.355912+0000","last_became_peered":"2024-03-24T00:40:01.355912+0000","last_unstale":"2024-03-24T00:40:02.685100+0000","last_undegraded":"2024-03-24T00:40:02.685100+0000","last_fullsized":"2024-03-24T00:40:02.685100+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:23:33.705793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.328221+0000","last_change":"2024-03-24T00:40:01.352016+0000","last_active":"2024-03-24T00:40:02.328221+0000","last_peered":"2024-03-24T00:40:02.328221+0000","last_clean":"2024-03-24T00:40:02.328221+0000","last_became_active":"2024-03-24T00:40:01.351496+0000","last_became_peered":"2024-03-24T00:40:01.351496+0000","last_unstale":"2024-03-24T00:40:02.328221+0000","last_undegraded":"2024-03-24T00:40:02.328221+0000","last_fullsized":"2024-03-24T00:40:02.328221+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:54:11.377947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.327897+0000","last_change":"2024-03-24T00:40:01.354643+0000","last_active":"2024-03-24T00:40:02.327897+0000","last_peered":"2024-03-24T00:40:02.327897+0000","last_clean":"2024-03-24T00:40:02.327897+0000","last_became_active":"2024-03-24T00:40:01.354393+0000","last_became_peered":"2024-03-24T00:40:01.354393+0000","last_unstale":"2024-03-24T00:40:02.327897+0000","last_undegraded":"2024-03-24T00:40:02.327897+0000","last_fullsized":"2024-03-24T00:40:02.327897+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:37:43.101486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.684935+0000","last_change":"2024-03-24T00:40:01.342932+0000","last_active":"2024-03-24T00:40:02.684935+0000","last_peered":"2024-03-24T00:40:02.684935+0000","last_clean":"2024-03-24T00:40:02.684935+0000","last_became_active":"2024-03-24T00:40:01.342615+0000","last_became_peered":"2024-03-24T00:40:01.342615+0000","last_unstale":"2024-03-24T00:40:02.684935+0000","last_undegraded":"2024-03-24T00:40:02.684935+0000","last_fullsized":"2024-03-24T00:40:02.684935+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:11:12.174614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.327186+0000","last_change":"2024-03-24T00:40:01.352457+0000","last_active":"2024-03-24T00:40:02.327186+0000","last_peered":"2024-03-24T00:40:02.327186+0000","last_clean":"2024-03-24T00:40:02.327186+0000","last_became_active":"2024-03-24T00:40:01.352201+0000","last_became_peered":"2024-03-24T00:40:01.352201+0000","last_unstale":"2024-03-24T00:40:02.327186+0000","last_undegraded":"2024-03-24T00:40:02.327186+0000","last_fullsized":"2024-03-24T00:40:02.327186+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:52:01.752611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.327238+0000","last_change":"2024-03-24T00:40:01.356693+0000","last_active":"2024-03-24T00:40:02.327238+0000","last_peered":"2024-03-24T00:40:02.327238+0000","last_clean":"2024-03-24T00:40:02.327238+0000","last_became_active":"2024-03-24T00:40:01.356306+0000","last_became_peered":"2024-03-24T00:40:01.356306+0000","last_unstale":"2024-03-24T00:40:02.327238+0000","last_undegraded":"2024-03-24T00:40:02.327238+0000","last_fullsized":"2024-03-24T00:40:02.327238+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:25:54.080003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.327257+0000","last_change":"2024-03-24T00:40:01.354392+0000","last_active":"2024-03-24T00:40:02.327257+0000","last_peered":"2024-03-24T00:40:02.327257+0000","last_clean":"2024-03-24T00:40:02.327257+0000","last_became_active":"2024-03-24T00:40:01.354111+0000","last_became_peered":"2024-03-24T00:40:01.354111+0000","last_unstale":"2024-03-24T00:40:02.327257+0000","last_undegraded":"2024-03-24T00:40:02.327257+0000","last_fullsized":"2024-03-24T00:40:02.327257+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:30:01.709222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685016+0000","last_change":"2024-03-24T00:40:01.354777+0000","last_active":"2024-03-24T00:40:02.685016+0000","last_peered":"2024-03-24T00:40:02.685016+0000","last_clean":"2024-03-24T00:40:02.685016+0000","last_became_active":"2024-03-24T00:40:01.354507+0000","last_became_peered":"2024-03-24T00:40:01.354507+0000","last_unstale":"2024-03-24T00:40:02.685016+0000","last_undegraded":"2024-03-24T00:40:02.685016+0000","last_fullsized":"2024-03-24T00:40:02.685016+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:11:12.174614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685571+0000","last_change":"2024-03-24T00:40:01.356643+0000","last_active":"2024-03-24T00:40:02.685571+0000","last_peered":"2024-03-24T00:40:02.685571+0000","last_clean":"2024-03-24T00:40:02.685571+0000","last_became_active":"2024-03-24T00:40:01.355610+0000","last_became_peered":"2024-03-24T00:40:01.355610+0000","last_unstale":"2024-03-24T00:40:02.685571+0000","last_undegraded":"2024-03-24T00:40:02.685571+0000","last_fullsized":"2024-03-24T00:40:02.685571+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:31:30.760690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685511+0000","last_change":"2024-03-24T00:40:01.349254+0000","last_active":"2024-03-24T00:40:02.685511+0000","last_peered":"2024-03-24T00:40:02.685511+0000","last_clean":"2024-03-24T00:40:02.685511+0000","last_became_active":"2024-03-24T00:40:01.348965+0000","last_became_peered":"2024-03-24T00:40:01.348965+0000","last_unstale":"2024-03-24T00:40:02.685511+0000","last_undegraded":"2024-03-24T00:40:02.685511+0000","last_fullsized":"2024-03-24T00:40:02.685511+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:31:38.754178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.329759+0000","last_change":"2024-03-24T00:40:01.346529+0000","last_active":"2024-03-24T00:40:02.329759+0000","last_peered":"2024-03-24T00:40:02.329759+0000","last_clean":"2024-03-24T00:40:02.329759+0000","last_became_active":"2024-03-24T00:40:01.346262+0000","last_became_peered":"2024-03-24T00:40:01.346262+0000","last_unstale":"2024-03-24T00:40:02.329759+0000","last_undegraded":"2024-03-24T00:40:02.329759+0000","last_fullsized":"2024-03-24T00:40:02.329759+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:40:42.216343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685222+0000","last_change":"2024-03-24T00:40:01.348561+0000","last_active":"2024-03-24T00:40:02.685222+0000","last_peered":"2024-03-24T00:40:02.685222+0000","last_clean":"2024-03-24T00:40:02.685222+0000","last_became_active":"2024-03-24T00:40:01.348304+0000","last_became_peered":"2024-03-24T00:40:01.348304+0000","last_unstale":"2024-03-24T00:40:02.685222+0000","last_undegraded":"2024-03-24T00:40:02.685222+0000","last_fullsized":"2024-03-24T00:40:02.685222+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:30:16.536487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.327350+0000","last_change":"2024-03-24T00:40:01.353646+0000","last_active":"2024-03-24T00:40:02.327350+0000","last_peered":"2024-03-24T00:40:02.327350+0000","last_clean":"2024-03-24T00:40:02.327350+0000","last_became_active":"2024-03-24T00:40:01.352758+0000","last_became_peered":"2024-03-24T00:40:01.352758+0000","last_unstale":"2024-03-24T00:40:02.327350+0000","last_undegraded":"2024-03-24T00:40:02.327350+0000","last_fullsized":"2024-03-24T00:40:02.327350+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:27:49.593844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.327514+0000","last_change":"2024-03-24T00:40:01.354943+0000","last_active":"2024-03-24T00:40:02.327514+0000","last_peered":"2024-03-24T00:40:02.327514+0000","last_clean":"2024-03-24T00:40:02.327514+0000","last_became_active":"2024-03-24T00:40:01.354452+0000","last_became_peered":"2024-03-24T00:40:01.354452+0000","last_unstale":"2024-03-24T00:40:02.327514+0000","last_undegraded":"2024-03-24T00:40:02.327514+0000","last_fullsized":"2024-03-24T00:40:02.327514+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:51.086346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.684979+0000","last_change":"2024-03-24T00:40:01.345157+0000","last_active":"2024-03-24T00:40:02.684979+0000","last_peered":"2024-03-24T00:40:02.684979+0000","last_clean":"2024-03-24T00:40:02.684979+0000","last_became_active":"2024-03-24T00:40:01.344404+0000","last_became_peered":"2024-03-24T00:40:01.344404+0000","last_unstale":"2024-03-24T00:40:02.684979+0000","last_undegraded":"2024-03-24T00:40:02.684979+0000","last_fullsized":"2024-03-24T00:40:02.684979+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:29:16.074361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.329561+0000","last_change":"2024-03-24T00:40:01.337319+0000","last_active":"2024-03-24T00:40:02.329561+0000","last_peered":"2024-03-24T00:40:02.329561+0000","last_clean":"2024-03-24T00:40:02.329561+0000","last_became_active":"2024-03-24T00:40:01.336989+0000","last_became_peered":"2024-03-24T00:40:01.336989+0000","last_unstale":"2024-03-24T00:40:02.329561+0000","last_undegraded":"2024-03-24T00:40:02.329561+0000","last_fullsized":"2024-03-24T00:40:02.329561+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:11:33.082463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685282+0000","last_change":"2024-03-24T00:40:01.349701+0000","last_active":"2024-03-24T00:40:02.685282+0000","last_peered":"2024-03-24T00:40:02.685282+0000","last_clean":"2024-03-24T00:40:02.685282+0000","last_became_active":"2024-03-24T00:40:01.349448+0000","last_became_peered":"2024-03-24T00:40:01.349448+0000","last_unstale":"2024-03-24T00:40:02.685282+0000","last_undegraded":"2024-03-24T00:40:02.685282+0000","last_fullsized":"2024-03-24T00:40:02.685282+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:04:47.673221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685589+0000","last_change":"2024-03-24T00:40:01.356229+0000","last_active":"2024-03-24T00:40:02.685589+0000","last_peered":"2024-03-24T00:40:02.685589+0000","last_clean":"2024-03-24T00:40:02.685589+0000","last_became_active":"2024-03-24T00:40:01.355953+0000","last_became_peered":"2024-03-24T00:40:01.355953+0000","last_unstale":"2024-03-24T00:40:02.685589+0000","last_undegraded":"2024-03-24T00:40:02.685589+0000","last_fullsized":"2024-03-24T00:40:02.685589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:23:33.705793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685303+0000","last_change":"2024-03-24T00:40:01.338189+0000","last_active":"2024-03-24T00:40:02.685303+0000","last_peered":"2024-03-24T00:40:02.685303+0000","last_clean":"2024-03-24T00:40:02.685303+0000","last_became_active":"2024-03-24T00:40:01.337910+0000","last_became_peered":"2024-03-24T00:40:01.337910+0000","last_unstale":"2024-03-24T00:40:02.685303+0000","last_undegraded":"2024-03-24T00:40:02.685303+0000","last_fullsized":"2024-03-24T00:40:02.685303+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:18:05.164294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.327590+0000","last_change":"2024-03-24T00:40:01.353130+0000","last_active":"2024-03-24T00:40:02.327590+0000","last_peered":"2024-03-24T00:40:02.327590+0000","last_clean":"2024-03-24T00:40:02.327590+0000","last_became_active":"2024-03-24T00:40:01.352852+0000","last_became_peered":"2024-03-24T00:40:01.352852+0000","last_unstale":"2024-03-24T00:40:02.327590+0000","last_undegraded":"2024-03-24T00:40:02.327590+0000","last_fullsized":"2024-03-24T00:40:02.327590+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:25:10.580085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685299+0000","last_change":"2024-03-24T00:40:01.345986+0000","last_active":"2024-03-24T00:40:02.685299+0000","last_peered":"2024-03-24T00:40:02.685299+0000","last_clean":"2024-03-24T00:40:02.685299+0000","last_became_active":"2024-03-24T00:40:01.345705+0000","last_became_peered":"2024-03-24T00:40:01.345705+0000","last_unstale":"2024-03-24T00:40:02.685299+0000","last_undegraded":"2024-03-24T00:40:02.685299+0000","last_fullsized":"2024-03-24T00:40:02.685299+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:28:36.656275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685595+0000","last_change":"2024-03-24T00:40:01.331420+0000","last_active":"2024-03-24T00:40:02.685595+0000","last_peered":"2024-03-24T00:40:02.685595+0000","last_clean":"2024-03-24T00:40:02.685595+0000","last_became_active":"2024-03-24T00:40:01.331128+0000","last_became_peered":"2024-03-24T00:40:01.331128+0000","last_unstale":"2024-03-24T00:40:02.685595+0000","last_undegraded":"2024-03-24T00:40:02.685595+0000","last_fullsized":"2024-03-24T00:40:02.685595+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:48:50.540849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685015+0000","last_change":"2024-03-24T00:40:01.344038+0000","last_active":"2024-03-24T00:40:02.685015+0000","last_peered":"2024-03-24T00:40:02.685015+0000","last_clean":"2024-03-24T00:40:02.685015+0000","last_became_active":"2024-03-24T00:40:01.343741+0000","last_became_peered":"2024-03-24T00:40:01.343741+0000","last_unstale":"2024-03-24T00:40:02.685015+0000","last_undegraded":"2024-03-24T00:40:02.685015+0000","last_fullsized":"2024-03-24T00:40:02.685015+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:25:54.080003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.684875+0000","last_change":"2024-03-24T00:40:01.344268+0000","last_active":"2024-03-24T00:40:02.684875+0000","last_peered":"2024-03-24T00:40:02.684875+0000","last_clean":"2024-03-24T00:40:02.684875+0000","last_became_active":"2024-03-24T00:40:01.343972+0000","last_became_peered":"2024-03-24T00:40:01.343972+0000","last_unstale":"2024-03-24T00:40:02.684875+0000","last_undegraded":"2024-03-24T00:40:02.684875+0000","last_fullsized":"2024-03-24T00:40:02.684875+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:25:54.080003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685342+0000","last_change":"2024-03-24T00:40:01.355848+0000","last_active":"2024-03-24T00:40:02.685342+0000","last_peered":"2024-03-24T00:40:02.685342+0000","last_clean":"2024-03-24T00:40:02.685342+0000","last_became_active":"2024-03-24T00:40:01.355593+0000","last_became_peered":"2024-03-24T00:40:01.355593+0000","last_unstale":"2024-03-24T00:40:02.685342+0000","last_undegraded":"2024-03-24T00:40:02.685342+0000","last_fullsized":"2024-03-24T00:40:02.685342+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:34:52.148336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"38'4","reported_seq":18,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685329+0000","last_change":"2024-03-24T00:40:00.346759+0000","last_active":"2024-03-24T00:40:02.685329+0000","last_peered":"2024-03-24T00:40:02.685329+0000","last_clean":"2024-03-24T00:40:02.685329+0000","last_became_active":"2024-03-24T00:40:00.346366+0000","last_became_peered":"2024-03-24T00:40:00.346366+0000","last_unstale":"2024-03-24T00:40:02.685329+0000","last_undegraded":"2024-03-24T00:40:02.685329+0000","last_fullsized":"2024-03-24T00:40:02.685329+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:09:00.112357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685511+0000","last_change":"2024-03-24T00:40:01.356499+0000","last_active":"2024-03-24T00:40:02.685511+0000","last_peered":"2024-03-24T00:40:02.685511+0000","last_clean":"2024-03-24T00:40:02.685511+0000","last_became_active":"2024-03-24T00:40:01.356224+0000","last_became_peered":"2024-03-24T00:40:01.356224+0000","last_unstale":"2024-03-24T00:40:02.685511+0000","last_undegraded":"2024-03-24T00:40:02.685511+0000","last_fullsized":"2024-03-24T00:40:02.685511+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:47:44.580768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.684876+0000","last_change":"2024-03-24T00:40:00.338722+0000","last_active":"2024-03-24T00:40:02.684876+0000","last_peered":"2024-03-24T00:40:02.684876+0000","last_clean":"2024-03-24T00:40:02.684876+0000","last_became_active":"2024-03-24T00:40:00.338449+0000","last_became_peered":"2024-03-24T00:40:00.338449+0000","last_unstale":"2024-03-24T00:40:02.684876+0000","last_undegraded":"2024-03-24T00:40:02.684876+0000","last_fullsized":"2024-03-24T00:40:02.684876+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:50:26.015558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.328071+0000","last_change":"2024-03-24T00:40:01.352156+0000","last_active":"2024-03-24T00:40:02.328071+0000","last_peered":"2024-03-24T00:40:02.328071+0000","last_clean":"2024-03-24T00:40:02.328071+0000","last_became_active":"2024-03-24T00:40:01.351605+0000","last_became_peered":"2024-03-24T00:40:01.351605+0000","last_unstale":"2024-03-24T00:40:02.328071+0000","last_undegraded":"2024-03-24T00:40:02.328071+0000","last_fullsized":"2024-03-24T00:40:02.328071+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:23:33.705793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685584+0000","last_change":"2024-03-24T00:40:00.345390+0000","last_active":"2024-03-24T00:40:02.685584+0000","last_peered":"2024-03-24T00:40:02.685584+0000","last_clean":"2024-03-24T00:40:02.685584+0000","last_became_active":"2024-03-24T00:40:00.345117+0000","last_became_peered":"2024-03-24T00:40:00.345117+0000","last_unstale":"2024-03-24T00:40:02.685584+0000","last_undegraded":"2024-03-24T00:40:02.685584+0000","last_fullsized":"2024-03-24T00:40:02.685584+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:04:46.671168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.326684+0000","last_change":"2024-03-24T00:40:01.339489+0000","last_active":"2024-03-24T00:40:02.326684+0000","last_peered":"2024-03-24T00:40:02.326684+0000","last_clean":"2024-03-24T00:40:02.326684+0000","last_became_active":"2024-03-24T00:40:01.338052+0000","last_became_peered":"2024-03-24T00:40:01.338052+0000","last_unstale":"2024-03-24T00:40:02.326684+0000","last_undegraded":"2024-03-24T00:40:02.326684+0000","last_fullsized":"2024-03-24T00:40:02.326684+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:56:17.522174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.327735+0000","last_change":"2024-03-24T00:40:00.348606+0000","last_active":"2024-03-24T00:40:02.327735+0000","last_peered":"2024-03-24T00:40:02.327735+0000","last_clean":"2024-03-24T00:40:02.327735+0000","last_became_active":"2024-03-24T00:40:00.348365+0000","last_became_peered":"2024-03-24T00:40:00.348365+0000","last_unstale":"2024-03-24T00:40:02.327735+0000","last_undegraded":"2024-03-24T00:40:02.327735+0000","last_fullsized":"2024-03-24T00:40:02.327735+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:30:15.534434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.684685+0000","last_change":"2024-03-24T00:40:01.351153+0000","last_active":"2024-03-24T00:40:02.684685+0000","last_peered":"2024-03-24T00:40:02.684685+0000","last_clean":"2024-03-24T00:40:02.684685+0000","last_became_active":"2024-03-24T00:40:01.350878+0000","last_became_peered":"2024-03-24T00:40:01.350878+0000","last_unstale":"2024-03-24T00:40:02.684685+0000","last_undegraded":"2024-03-24T00:40:02.684685+0000","last_fullsized":"2024-03-24T00:40:02.684685+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:29:16.074361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.327399+0000","last_change":"2024-03-24T00:40:00.342836+0000","last_active":"2024-03-24T00:40:02.327399+0000","last_peered":"2024-03-24T00:40:02.327399+0000","last_clean":"2024-03-24T00:40:02.327399+0000","last_became_active":"2024-03-24T00:40:00.342550+0000","last_became_peered":"2024-03-24T00:40:00.342550+0000","last_unstale":"2024-03-24T00:40:02.327399+0000","last_undegraded":"2024-03-24T00:40:02.327399+0000","last_fullsized":"2024-03-24T00:40:02.327399+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:30:15.534434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.327069+0000","last_change":"2024-03-24T00:40:01.353462+0000","last_active":"2024-03-24T00:40:02.327069+0000","last_peered":"2024-03-24T00:40:02.327069+0000","last_clean":"2024-03-24T00:40:02.327069+0000","last_became_active":"2024-03-24T00:40:01.353183+0000","last_became_peered":"2024-03-24T00:40:01.353183+0000","last_unstale":"2024-03-24T00:40:02.327069+0000","last_undegraded":"2024-03-24T00:40:02.327069+0000","last_fullsized":"2024-03-24T00:40:02.327069+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:31:30.760690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.684915+0000","last_change":"2024-03-24T00:40:00.346329+0000","last_active":"2024-03-24T00:40:02.684915+0000","last_peered":"2024-03-24T00:40:02.684915+0000","last_clean":"2024-03-24T00:40:02.684915+0000","last_became_active":"2024-03-24T00:40:00.346051+0000","last_became_peered":"2024-03-24T00:40:00.346051+0000","last_unstale":"2024-03-24T00:40:02.684915+0000","last_undegraded":"2024-03-24T00:40:02.684915+0000","last_fullsized":"2024-03-24T00:40:02.684915+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:06:44.733235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.327708+0000","last_change":"2024-03-24T00:40:01.357619+0000","last_active":"2024-03-24T00:40:02.327708+0000","last_peered":"2024-03-24T00:40:02.327708+0000","last_clean":"2024-03-24T00:40:02.327708+0000","last_became_active":"2024-03-24T00:40:01.356944+0000","last_became_peered":"2024-03-24T00:40:01.356944+0000","last_unstale":"2024-03-24T00:40:02.327708+0000","last_undegraded":"2024-03-24T00:40:02.327708+0000","last_fullsized":"2024-03-24T00:40:02.327708+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:34:52.148336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685257+0000","last_change":"2024-03-24T00:40:00.341907+0000","last_active":"2024-03-24T00:40:02.685257+0000","last_peered":"2024-03-24T00:40:02.685257+0000","last_clean":"2024-03-24T00:40:02.685257+0000","last_became_active":"2024-03-24T00:40:00.341643+0000","last_became_peered":"2024-03-24T00:40:00.341643+0000","last_unstale":"2024-03-24T00:40:02.685257+0000","last_undegraded":"2024-03-24T00:40:02.685257+0000","last_fullsized":"2024-03-24T00:40:02.685257+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:30:15.534434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.326459+0000","last_change":"2024-03-24T00:40:01.344561+0000","last_active":"2024-03-24T00:40:02.326459+0000","last_peered":"2024-03-24T00:40:02.326459+0000","last_clean":"2024-03-24T00:40:02.326459+0000","last_became_active":"2024-03-24T00:40:01.343777+0000","last_became_peered":"2024-03-24T00:40:01.343777+0000","last_unstale":"2024-03-24T00:40:02.326459+0000","last_undegraded":"2024-03-24T00:40:02.326459+0000","last_fullsized":"2024-03-24T00:40:02.326459+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:09:57.767219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.326786+0000","last_change":"2024-03-24T00:40:00.346231+0000","last_active":"2024-03-24T00:40:02.326786+0000","last_peered":"2024-03-24T00:40:02.326786+0000","last_clean":"2024-03-24T00:40:02.326786+0000","last_became_active":"2024-03-24T00:40:00.345948+0000","last_became_peered":"2024-03-24T00:40:00.345948+0000","last_unstale":"2024-03-24T00:40:02.326786+0000","last_undegraded":"2024-03-24T00:40:02.326786+0000","last_fullsized":"2024-03-24T00:40:02.326786+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:40:41.214290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.329742+0000","last_change":"2024-03-24T00:40:01.349328+0000","last_active":"2024-03-24T00:40:02.329742+0000","last_peered":"2024-03-24T00:40:02.329742+0000","last_clean":"2024-03-24T00:40:02.329742+0000","last_became_active":"2024-03-24T00:40:01.348743+0000","last_became_peered":"2024-03-24T00:40:01.348743+0000","last_unstale":"2024-03-24T00:40:02.329742+0000","last_undegraded":"2024-03-24T00:40:02.329742+0000","last_fullsized":"2024-03-24T00:40:02.329742+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:27:39.492378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.326712+0000","last_change":"2024-03-24T00:40:00.343185+0000","last_active":"2024-03-24T00:40:02.326712+0000","last_peered":"2024-03-24T00:40:02.326712+0000","last_clean":"2024-03-24T00:40:02.326712+0000","last_became_active":"2024-03-24T00:40:00.342931+0000","last_became_peered":"2024-03-24T00:40:00.342931+0000","last_unstale":"2024-03-24T00:40:02.326712+0000","last_undegraded":"2024-03-24T00:40:02.326712+0000","last_fullsized":"2024-03-24T00:40:02.326712+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:23:32.703740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685526+0000","last_change":"2024-03-24T00:40:01.344896+0000","last_active":"2024-03-24T00:40:02.685526+0000","last_peered":"2024-03-24T00:40:02.685526+0000","last_clean":"2024-03-24T00:40:02.685526+0000","last_became_active":"2024-03-24T00:40:01.344627+0000","last_became_peered":"2024-03-24T00:40:01.344627+0000","last_unstale":"2024-03-24T00:40:02.685526+0000","last_undegraded":"2024-03-24T00:40:02.685526+0000","last_fullsized":"2024-03-24T00:40:02.685526+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:44:25.432200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.328239+0000","last_change":"2024-03-24T00:40:00.337581+0000","last_active":"2024-03-24T00:40:02.328239+0000","last_peered":"2024-03-24T00:40:02.328239+0000","last_clean":"2024-03-24T00:40:02.328239+0000","last_became_active":"2024-03-24T00:40:00.336273+0000","last_became_peered":"2024-03-24T00:40:00.336273+0000","last_unstale":"2024-03-24T00:40:02.328239+0000","last_undegraded":"2024-03-24T00:40:02.328239+0000","last_fullsized":"2024-03-24T00:40:02.328239+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:04:46.671168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.684874+0000","last_change":"2024-03-24T00:40:01.346266+0000","last_active":"2024-03-24T00:40:02.684874+0000","last_peered":"2024-03-24T00:40:02.684874+0000","last_clean":"2024-03-24T00:40:02.684874+0000","last_became_active":"2024-03-24T00:40:01.345647+0000","last_became_peered":"2024-03-24T00:40:01.345647+0000","last_unstale":"2024-03-24T00:40:02.684874+0000","last_undegraded":"2024-03-24T00:40:02.684874+0000","last_fullsized":"2024-03-24T00:40:02.684874+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:25:54.080003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"38'3","reported_seq":17,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.327966+0000","last_change":"2024-03-24T00:40:00.337995+0000","last_active":"2024-03-24T00:40:02.327966+0000","last_peered":"2024-03-24T00:40:02.327966+0000","last_clean":"2024-03-24T00:40:02.327966+0000","last_became_active":"2024-03-24T00:40:00.336199+0000","last_became_peered":"2024-03-24T00:40:00.336199+0000","last_unstale":"2024-03-24T00:40:02.327966+0000","last_undegraded":"2024-03-24T00:40:02.327966+0000","last_fullsized":"2024-03-24T00:40:02.327966+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:05:49.943101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.327123+0000","last_change":"2024-03-24T00:40:01.354792+0000","last_active":"2024-03-24T00:40:02.327123+0000","last_peered":"2024-03-24T00:40:02.327123+0000","last_clean":"2024-03-24T00:40:02.327123+0000","last_became_active":"2024-03-24T00:40:01.354518+0000","last_became_peered":"2024-03-24T00:40:01.354518+0000","last_unstale":"2024-03-24T00:40:02.327123+0000","last_undegraded":"2024-03-24T00:40:02.327123+0000","last_fullsized":"2024-03-24T00:40:02.327123+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:29:16.074361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"38'6","reported_seq":20,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.326390+0000","last_change":"2024-03-24T00:40:00.338540+0000","last_active":"2024-03-24T00:40:02.326390+0000","last_peered":"2024-03-24T00:40:02.326390+0000","last_clean":"2024-03-24T00:40:02.326390+0000","last_became_active":"2024-03-24T00:40:00.338230+0000","last_became_peered":"2024-03-24T00:40:00.338230+0000","last_unstale":"2024-03-24T00:40:02.326390+0000","last_undegraded":"2024-03-24T00:40:02.326390+0000","last_fullsized":"2024-03-24T00:40:02.326390+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:34:51.146284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685105+0000","last_change":"2024-03-24T00:40:01.350796+0000","last_active":"2024-03-24T00:40:02.685105+0000","last_peered":"2024-03-24T00:40:02.685105+0000","last_clean":"2024-03-24T00:40:02.685105+0000","last_became_active":"2024-03-24T00:40:01.350249+0000","last_became_peered":"2024-03-24T00:40:01.350249+0000","last_unstale":"2024-03-24T00:40:02.685105+0000","last_undegraded":"2024-03-24T00:40:02.685105+0000","last_fullsized":"2024-03-24T00:40:02.685105+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:44:25.432200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.326548+0000","last_change":"2024-03-24T00:40:00.344040+0000","last_active":"2024-03-24T00:40:02.326548+0000","last_peered":"2024-03-24T00:40:02.326548+0000","last_clean":"2024-03-24T00:40:02.326548+0000","last_became_active":"2024-03-24T00:40:00.343790+0000","last_became_peered":"2024-03-24T00:40:00.343790+0000","last_unstale":"2024-03-24T00:40:02.326548+0000","last_undegraded":"2024-03-24T00:40:02.326548+0000","last_fullsized":"2024-03-24T00:40:02.326548+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:47:43.578715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.326251+0000","last_change":"2024-03-24T00:40:01.346676+0000","last_active":"2024-03-24T00:40:02.326251+0000","last_peered":"2024-03-24T00:40:02.326251+0000","last_clean":"2024-03-24T00:40:02.326251+0000","last_became_active":"2024-03-24T00:40:01.346398+0000","last_became_peered":"2024-03-24T00:40:01.346398+0000","last_unstale":"2024-03-24T00:40:02.326251+0000","last_undegraded":"2024-03-24T00:40:02.326251+0000","last_fullsized":"2024-03-24T00:40:02.326251+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:42:29.481647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685205+0000","last_change":"2024-03-24T00:40:00.343886+0000","last_active":"2024-03-24T00:40:02.685205+0000","last_peered":"2024-03-24T00:40:02.685205+0000","last_clean":"2024-03-24T00:40:02.685205+0000","last_became_active":"2024-03-24T00:40:00.343624+0000","last_became_peered":"2024-03-24T00:40:00.343624+0000","last_unstale":"2024-03-24T00:40:02.685205+0000","last_undegraded":"2024-03-24T00:40:02.685205+0000","last_fullsized":"2024-03-24T00:40:02.685205+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:05:49.943101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685089+0000","last_change":"2024-03-24T00:40:01.343214+0000","last_active":"2024-03-24T00:40:02.685089+0000","last_peered":"2024-03-24T00:40:02.685089+0000","last_clean":"2024-03-24T00:40:02.685089+0000","last_became_active":"2024-03-24T00:40:01.342847+0000","last_became_peered":"2024-03-24T00:40:01.342847+0000","last_unstale":"2024-03-24T00:40:02.685089+0000","last_undegraded":"2024-03-24T00:40:02.685089+0000","last_fullsized":"2024-03-24T00:40:02.685089+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:51.086346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.684866+0000","last_change":"2024-03-24T00:40:00.345927+0000","last_active":"2024-03-24T00:40:02.684866+0000","last_peered":"2024-03-24T00:40:02.684866+0000","last_clean":"2024-03-24T00:40:02.684866+0000","last_became_active":"2024-03-24T00:40:00.345681+0000","last_became_peered":"2024-03-24T00:40:00.345681+0000","last_unstale":"2024-03-24T00:40:02.684866+0000","last_undegraded":"2024-03-24T00:40:02.684866+0000","last_fullsized":"2024-03-24T00:40:02.684866+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:27:53.783517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.326490+0000","last_change":"2024-03-24T00:40:01.347982+0000","last_active":"2024-03-24T00:40:02.326490+0000","last_peered":"2024-03-24T00:40:02.326490+0000","last_clean":"2024-03-24T00:40:02.326490+0000","last_became_active":"2024-03-24T00:40:01.347571+0000","last_became_peered":"2024-03-24T00:40:01.347571+0000","last_unstale":"2024-03-24T00:40:02.326490+0000","last_undegraded":"2024-03-24T00:40:02.326490+0000","last_fullsized":"2024-03-24T00:40:02.326490+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:07:41.182445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"38'8","reported_seq":22,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.684865+0000","last_change":"2024-03-24T00:40:00.345571+0000","last_active":"2024-03-24T00:40:02.684865+0000","last_peered":"2024-03-24T00:40:02.684865+0000","last_clean":"2024-03-24T00:40:02.684865+0000","last_became_active":"2024-03-24T00:40:00.345288+0000","last_became_peered":"2024-03-24T00:40:00.345288+0000","last_unstale":"2024-03-24T00:40:02.684865+0000","last_undegraded":"2024-03-24T00:40:02.684865+0000","last_fullsized":"2024-03-24T00:40:02.684865+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:11:32.080410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.326458+0000","last_change":"2024-03-24T00:40:01.331287+0000","last_active":"2024-03-24T00:40:02.326458+0000","last_peered":"2024-03-24T00:40:02.326458+0000","last_clean":"2024-03-24T00:40:02.326458+0000","last_became_active":"2024-03-24T00:40:01.330903+0000","last_became_peered":"2024-03-24T00:40:01.330903+0000","last_unstale":"2024-03-24T00:40:02.326458+0000","last_undegraded":"2024-03-24T00:40:02.326458+0000","last_fullsized":"2024-03-24T00:40:02.326458+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:41:50.553798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.328239+0000","last_change":"2024-03-24T00:40:00.345924+0000","last_active":"2024-03-24T00:40:02.328239+0000","last_peered":"2024-03-24T00:40:02.328239+0000","last_clean":"2024-03-24T00:40:02.328239+0000","last_became_active":"2024-03-24T00:40:00.345636+0000","last_became_peered":"2024-03-24T00:40:00.345636+0000","last_unstale":"2024-03-24T00:40:02.328239+0000","last_undegraded":"2024-03-24T00:40:02.328239+0000","last_fullsized":"2024-03-24T00:40:02.328239+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:28:35.654223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.328035+0000","last_change":"2024-03-24T00:40:01.353040+0000","last_active":"2024-03-24T00:40:02.328035+0000","last_peered":"2024-03-24T00:40:02.328035+0000","last_clean":"2024-03-24T00:40:02.328035+0000","last_became_active":"2024-03-24T00:40:01.352658+0000","last_became_peered":"2024-03-24T00:40:01.352658+0000","last_unstale":"2024-03-24T00:40:02.328035+0000","last_undegraded":"2024-03-24T00:40:02.328035+0000","last_fullsized":"2024-03-24T00:40:02.328035+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:15:44.062886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.328077+0000","last_change":"2024-03-24T00:40:00.345548+0000","last_active":"2024-03-24T00:40:02.328077+0000","last_peered":"2024-03-24T00:40:02.328077+0000","last_clean":"2024-03-24T00:40:02.328077+0000","last_became_active":"2024-03-24T00:40:00.345250+0000","last_became_peered":"2024-03-24T00:40:00.345250+0000","last_unstale":"2024-03-24T00:40:02.328077+0000","last_undegraded":"2024-03-24T00:40:02.328077+0000","last_fullsized":"2024-03-24T00:40:02.328077+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:05:49.943101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.328021+0000","last_change":"2024-03-24T00:40:01.351854+0000","last_active":"2024-03-24T00:40:02.328021+0000","last_peered":"2024-03-24T00:40:02.328021+0000","last_clean":"2024-03-24T00:40:02.328021+0000","last_became_active":"2024-03-24T00:40:01.351282+0000","last_became_peered":"2024-03-24T00:40:01.351282+0000","last_unstale":"2024-03-24T00:40:02.328021+0000","last_undegraded":"2024-03-24T00:40:02.328021+0000","last_fullsized":"2024-03-24T00:40:02.328021+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:47:44.580768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.329740+0000","last_change":"2024-03-24T00:40:00.335501+0000","last_active":"2024-03-24T00:40:02.329740+0000","last_peered":"2024-03-24T00:40:02.329740+0000","last_clean":"2024-03-24T00:40:02.329740+0000","last_became_active":"2024-03-24T00:40:00.335233+0000","last_became_peered":"2024-03-24T00:40:00.335233+0000","last_unstale":"2024-03-24T00:40:02.329740+0000","last_undegraded":"2024-03-24T00:40:02.329740+0000","last_fullsized":"2024-03-24T00:40:02.329740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:21:02.704842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.329818+0000","last_change":"2024-03-24T00:40:01.334780+0000","last_active":"2024-03-24T00:40:02.329818+0000","last_peered":"2024-03-24T00:40:02.329818+0000","last_clean":"2024-03-24T00:40:02.329818+0000","last_became_active":"2024-03-24T00:40:01.333949+0000","last_became_peered":"2024-03-24T00:40:01.333949+0000","last_unstale":"2024-03-24T00:40:02.329818+0000","last_undegraded":"2024-03-24T00:40:02.329818+0000","last_fullsized":"2024-03-24T00:40:02.329818+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:50:27.017610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685335+0000","last_change":"2024-03-24T00:40:00.345775+0000","last_active":"2024-03-24T00:40:02.685335+0000","last_peered":"2024-03-24T00:40:02.685335+0000","last_clean":"2024-03-24T00:40:02.685335+0000","last_became_active":"2024-03-24T00:40:00.345458+0000","last_became_peered":"2024-03-24T00:40:00.345458+0000","last_unstale":"2024-03-24T00:40:02.685335+0000","last_undegraded":"2024-03-24T00:40:02.685335+0000","last_fullsized":"2024-03-24T00:40:02.685335+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:28:35.654223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685361+0000","last_change":"2024-03-24T00:40:01.345730+0000","last_active":"2024-03-24T00:40:02.685361+0000","last_peered":"2024-03-24T00:40:02.685361+0000","last_clean":"2024-03-24T00:40:02.685361+0000","last_became_active":"2024-03-24T00:40:01.345437+0000","last_became_peered":"2024-03-24T00:40:01.345437+0000","last_unstale":"2024-03-24T00:40:02.685361+0000","last_undegraded":"2024-03-24T00:40:02.685361+0000","last_fullsized":"2024-03-24T00:40:02.685361+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:47:44.580768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.329483+0000","last_change":"2024-03-24T00:40:00.347310+0000","last_active":"2024-03-24T00:40:02.329483+0000","last_peered":"2024-03-24T00:40:02.329483+0000","last_clean":"2024-03-24T00:40:02.329483+0000","last_became_active":"2024-03-24T00:40:00.347030+0000","last_became_peered":"2024-03-24T00:40:00.347030+0000","last_unstale":"2024-03-24T00:40:02.329483+0000","last_undegraded":"2024-03-24T00:40:02.329483+0000","last_fullsized":"2024-03-24T00:40:02.329483+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:16:31.720628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685032+0000","last_change":"2024-03-24T00:40:01.346045+0000","last_active":"2024-03-24T00:40:02.685032+0000","last_peered":"2024-03-24T00:40:02.685032+0000","last_clean":"2024-03-24T00:40:02.685032+0000","last_became_active":"2024-03-24T00:40:01.345693+0000","last_became_peered":"2024-03-24T00:40:01.345693+0000","last_unstale":"2024-03-24T00:40:02.685032+0000","last_undegraded":"2024-03-24T00:40:02.685032+0000","last_fullsized":"2024-03-24T00:40:02.685032+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:47:44.580768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.329491+0000","last_change":"2024-03-24T00:40:00.346403+0000","last_active":"2024-03-24T00:40:02.329491+0000","last_peered":"2024-03-24T00:40:02.329491+0000","last_clean":"2024-03-24T00:40:02.329491+0000","last_became_active":"2024-03-24T00:40:00.346095+0000","last_became_peered":"2024-03-24T00:40:00.346095+0000","last_unstale":"2024-03-24T00:40:02.329491+0000","last_undegraded":"2024-03-24T00:40:02.329491+0000","last_fullsized":"2024-03-24T00:40:02.329491+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:49:32.598338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.327180+0000","last_change":"2024-03-24T00:40:01.351042+0000","last_active":"2024-03-24T00:40:02.327180+0000","last_peered":"2024-03-24T00:40:02.327180+0000","last_clean":"2024-03-24T00:40:02.327180+0000","last_became_active":"2024-03-24T00:40:01.350750+0000","last_became_peered":"2024-03-24T00:40:01.350750+0000","last_unstale":"2024-03-24T00:40:02.327180+0000","last_undegraded":"2024-03-24T00:40:02.327180+0000","last_fullsized":"2024-03-24T00:40:02.327180+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:04:37.096161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685035+0000","last_change":"2024-03-24T00:40:00.335633+0000","last_active":"2024-03-24T00:40:02.685035+0000","last_peered":"2024-03-24T00:40:02.685035+0000","last_clean":"2024-03-24T00:40:02.685035+0000","last_became_active":"2024-03-24T00:40:00.335348+0000","last_became_peered":"2024-03-24T00:40:00.335348+0000","last_unstale":"2024-03-24T00:40:02.685035+0000","last_undegraded":"2024-03-24T00:40:02.685035+0000","last_fullsized":"2024-03-24T00:40:02.685035+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:40:41.214290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.326627+0000","last_change":"2024-03-24T00:40:01.345061+0000","last_active":"2024-03-24T00:40:02.326627+0000","last_peered":"2024-03-24T00:40:02.326627+0000","last_clean":"2024-03-24T00:40:02.326627+0000","last_became_active":"2024-03-24T00:40:01.344793+0000","last_became_peered":"2024-03-24T00:40:01.344793+0000","last_unstale":"2024-03-24T00:40:02.326627+0000","last_undegraded":"2024-03-24T00:40:02.326627+0000","last_fullsized":"2024-03-24T00:40:02.326627+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:17:43.539404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.329538+0000","last_change":"2024-03-24T00:40:00.338765+0000","last_active":"2024-03-24T00:40:02.329538+0000","last_peered":"2024-03-24T00:40:02.329538+0000","last_clean":"2024-03-24T00:40:02.329538+0000","last_became_active":"2024-03-24T00:40:00.337416+0000","last_became_peered":"2024-03-24T00:40:00.337416+0000","last_unstale":"2024-03-24T00:40:02.329538+0000","last_undegraded":"2024-03-24T00:40:02.329538+0000","last_fullsized":"2024-03-24T00:40:02.329538+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:38:41.933921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685292+0000","last_change":"2024-03-24T00:40:01.356678+0000","last_active":"2024-03-24T00:40:02.685292+0000","last_peered":"2024-03-24T00:40:02.685292+0000","last_clean":"2024-03-24T00:40:02.685292+0000","last_became_active":"2024-03-24T00:40:01.356395+0000","last_became_peered":"2024-03-24T00:40:01.356395+0000","last_unstale":"2024-03-24T00:40:02.685292+0000","last_undegraded":"2024-03-24T00:40:02.685292+0000","last_fullsized":"2024-03-24T00:40:02.685292+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:51.086346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.328338+0000","last_change":"2024-03-24T00:40:00.342913+0000","last_active":"2024-03-24T00:40:02.328338+0000","last_peered":"2024-03-24T00:40:02.328338+0000","last_clean":"2024-03-24T00:40:02.328338+0000","last_became_active":"2024-03-24T00:40:00.342633+0000","last_became_peered":"2024-03-24T00:40:00.342633+0000","last_unstale":"2024-03-24T00:40:02.328338+0000","last_undegraded":"2024-03-24T00:40:02.328338+0000","last_fullsized":"2024-03-24T00:40:02.328338+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:18:04.162241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685586+0000","last_change":"2024-03-24T00:40:01.343468+0000","last_active":"2024-03-24T00:40:02.685586+0000","last_peered":"2024-03-24T00:40:02.685586+0000","last_clean":"2024-03-24T00:40:02.685586+0000","last_became_active":"2024-03-24T00:40:01.343167+0000","last_became_peered":"2024-03-24T00:40:01.343167+0000","last_unstale":"2024-03-24T00:40:02.685586+0000","last_undegraded":"2024-03-24T00:40:02.685586+0000","last_fullsized":"2024-03-24T00:40:02.685586+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:04:37.096161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"39'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.329731+0000","last_change":"2024-03-24T00:40:00.335226+0000","last_active":"2024-03-24T00:40:02.329731+0000","last_peered":"2024-03-24T00:40:02.329731+0000","last_clean":"2024-03-24T00:40:02.329731+0000","last_became_active":"2024-03-24T00:40:00.334961+0000","last_became_peered":"2024-03-24T00:40:00.334961+0000","last_unstale":"2024-03-24T00:40:02.329731+0000","last_undegraded":"2024-03-24T00:40:02.329731+0000","last_fullsized":"2024-03-24T00:40:02.329731+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:23:45.494377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.326613+0000","last_change":"2024-03-24T00:40:01.349135+0000","last_active":"2024-03-24T00:40:02.326613+0000","last_peered":"2024-03-24T00:40:02.326613+0000","last_clean":"2024-03-24T00:40:02.326613+0000","last_became_active":"2024-03-24T00:40:01.348712+0000","last_became_peered":"2024-03-24T00:40:01.348712+0000","last_unstale":"2024-03-24T00:40:02.326613+0000","last_undegraded":"2024-03-24T00:40:02.326613+0000","last_fullsized":"2024-03-24T00:40:02.326613+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:00:13.987179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.684740+0000","last_change":"2024-03-24T00:40:00.342013+0000","last_active":"2024-03-24T00:40:02.684740+0000","last_peered":"2024-03-24T00:40:02.684740+0000","last_clean":"2024-03-24T00:40:02.684740+0000","last_became_active":"2024-03-24T00:40:00.341732+0000","last_became_peered":"2024-03-24T00:40:00.341732+0000","last_unstale":"2024-03-24T00:40:02.684740+0000","last_undegraded":"2024-03-24T00:40:02.684740+0000","last_fullsized":"2024-03-24T00:40:02.684740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:40:41.214290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.684998+0000","last_change":"2024-03-24T00:40:01.349804+0000","last_active":"2024-03-24T00:40:02.684998+0000","last_peered":"2024-03-24T00:40:02.684998+0000","last_clean":"2024-03-24T00:40:02.684998+0000","last_became_active":"2024-03-24T00:40:01.349542+0000","last_became_peered":"2024-03-24T00:40:01.349542+0000","last_unstale":"2024-03-24T00:40:02.684998+0000","last_undegraded":"2024-03-24T00:40:02.684998+0000","last_fullsized":"2024-03-24T00:40:02.684998+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:34:52.148336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.684813+0000","last_change":"2024-03-24T00:40:00.345415+0000","last_active":"2024-03-24T00:40:02.684813+0000","last_peered":"2024-03-24T00:40:02.684813+0000","last_clean":"2024-03-24T00:40:02.684813+0000","last_became_active":"2024-03-24T00:40:00.345145+0000","last_became_peered":"2024-03-24T00:40:00.345145+0000","last_unstale":"2024-03-24T00:40:02.684813+0000","last_undegraded":"2024-03-24T00:40:02.684813+0000","last_fullsized":"2024-03-24T00:40:02.684813+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:05:49.943101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685368+0000","last_change":"2024-03-24T00:40:01.344953+0000","last_active":"2024-03-24T00:40:02.685368+0000","last_peered":"2024-03-24T00:40:02.685368+0000","last_clean":"2024-03-24T00:40:02.685368+0000","last_became_active":"2024-03-24T00:40:01.344683+0000","last_became_peered":"2024-03-24T00:40:01.344683+0000","last_unstale":"2024-03-24T00:40:02.685368+0000","last_undegraded":"2024-03-24T00:40:02.685368+0000","last_fullsized":"2024-03-24T00:40:02.685368+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:34:52.148336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685583+0000","last_change":"2024-03-24T00:40:00.335784+0000","last_active":"2024-03-24T00:40:02.685583+0000","last_peered":"2024-03-24T00:40:02.685583+0000","last_clean":"2024-03-24T00:40:02.685583+0000","last_became_active":"2024-03-24T00:40:00.335478+0000","last_became_peered":"2024-03-24T00:40:00.335478+0000","last_unstale":"2024-03-24T00:40:02.685583+0000","last_undegraded":"2024-03-24T00:40:02.685583+0000","last_fullsized":"2024-03-24T00:40:02.685583+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:47:40.706718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.327376+0000","last_change":"2024-03-24T00:40:01.352911+0000","last_active":"2024-03-24T00:40:02.327376+0000","last_peered":"2024-03-24T00:40:02.327376+0000","last_clean":"2024-03-24T00:40:02.327376+0000","last_became_active":"2024-03-24T00:40:01.352668+0000","last_became_peered":"2024-03-24T00:40:01.352668+0000","last_unstale":"2024-03-24T00:40:02.327376+0000","last_undegraded":"2024-03-24T00:40:02.327376+0000","last_fullsized":"2024-03-24T00:40:02.327376+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:11:12.174614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.326903+0000","last_change":"2024-03-24T00:40:00.342133+0000","last_active":"2024-03-24T00:40:02.326903+0000","last_peered":"2024-03-24T00:40:02.326903+0000","last_clean":"2024-03-24T00:40:02.326903+0000","last_became_active":"2024-03-24T00:40:00.341821+0000","last_became_peered":"2024-03-24T00:40:00.341821+0000","last_unstale":"2024-03-24T00:40:02.326903+0000","last_undegraded":"2024-03-24T00:40:02.326903+0000","last_fullsized":"2024-03-24T00:40:02.326903+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:50:26.015558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685436+0000","last_change":"2024-03-24T00:40:01.344138+0000","last_active":"2024-03-24T00:40:02.685436+0000","last_peered":"2024-03-24T00:40:02.685436+0000","last_clean":"2024-03-24T00:40:02.685436+0000","last_became_active":"2024-03-24T00:40:01.343857+0000","last_became_peered":"2024-03-24T00:40:01.343857+0000","last_unstale":"2024-03-24T00:40:02.685436+0000","last_undegraded":"2024-03-24T00:40:02.685436+0000","last_fullsized":"2024-03-24T00:40:02.685436+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:23:33.705793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.326192+0000","last_change":"2024-03-24T00:40:00.344855+0000","last_active":"2024-03-24T00:40:02.326192+0000","last_peered":"2024-03-24T00:40:02.326192+0000","last_clean":"2024-03-24T00:40:02.326192+0000","last_became_active":"2024-03-24T00:40:00.344605+0000","last_became_peered":"2024-03-24T00:40:00.344605+0000","last_unstale":"2024-03-24T00:40:02.326192+0000","last_undegraded":"2024-03-24T00:40:02.326192+0000","last_fullsized":"2024-03-24T00:40:02.326192+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:50.084293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.328267+0000","last_change":"2024-03-24T00:40:01.355150+0000","last_active":"2024-03-24T00:40:02.328267+0000","last_peered":"2024-03-24T00:40:02.328267+0000","last_clean":"2024-03-24T00:40:02.328267+0000","last_became_active":"2024-03-24T00:40:01.354677+0000","last_became_peered":"2024-03-24T00:40:01.354677+0000","last_unstale":"2024-03-24T00:40:02.328267+0000","last_undegraded":"2024-03-24T00:40:02.328267+0000","last_fullsized":"2024-03-24T00:40:02.328267+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:00.316527+0000","last_clean_scrub_stamp":"2024-03-24T00:40:00.316527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:09:15.867422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685395+0000","last_change":"2024-03-24T00:40:00.341415+0000","last_active":"2024-03-24T00:40:02.685395+0000","last_peered":"2024-03-24T00:40:02.685395+0000","last_clean":"2024-03-24T00:40:02.685395+0000","last_became_active":"2024-03-24T00:40:00.341108+0000","last_became_peered":"2024-03-24T00:40:00.341108+0000","last_unstale":"2024-03-24T00:40:02.685395+0000","last_undegraded":"2024-03-24T00:40:02.685395+0000","last_fullsized":"2024-03-24T00:40:02.685395+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:59.314474+0000","last_clean_scrub_stamp":"2024-03-24T00:39:59.314474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:27:38.490325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":205,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T00:40:02.685138+0000","last_change":"2024-03-24T00:38:40.953871+0000","last_active":"2024-03-24T00:40:02.685138+0000","last_peered":"2024-03-24T00:40:02.685138+0000","last_clean":"2024-03-24T00:40:02.685138+0000","last_became_active":"2024-03-24T00:38:40.952926+0000","last_became_peered":"2024-03-24T00:38:40.952926+0000","last_unstale":"2024-03-24T00:40:02.685138+0000","last_undegraded":"2024-03-24T00:40:02.685138+0000","last_fullsized":"2024-03-24T00:40:02.685138+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_clean_scrub_stamp":"2024-03-24T00:38:39.722194+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:05.397257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574870,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28060,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343780,"statfs":{"total":96636764160,"available":96608030720,"internally_reserved":0,"allocated":1060864,"data_stored":789207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574869,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27468,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344372,"statfs":{"total":96636764160,"available":96608636928,"internally_reserved":0,"allocated":454656,"data_stored":197739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574869,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27476,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344364,"statfs":{"total":96636764160,"available":96608628736,"internally_reserved":0,"allocated":458752,"data_stored":197269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574868,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27476,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344364,"statfs":{"total":96636764160,"available":96608628736,"internally_reserved":0,"allocated":458752,"data_stored":197761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574868,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27480,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344360,"statfs":{"total":96636764160,"available":96608624640,"internally_reserved":0,"allocated":466944,"data_stored":198803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574869,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27468,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344372,"statfs":{"total":96636764160,"available":96608636928,"internally_reserved":0,"allocated":454656,"data_stored":197227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27472,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344368,"statfs":{"total":96636764160,"available":96608632832,"internally_reserved":0,"allocated":454656,"data_stored":197247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574868,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28048,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343792,"statfs":{"total":96636764160,"available":96608043008,"internally_reserved":0,"allocated":1048576,"data_stored":787559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:40:09.556 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-24T00:40:09.556 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-24T00:40:09.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.700+0000 7f73118b2640 1 Processor -- start 2024-03-24T00:40:09.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.700+0000 7f73118b2640 1 -- start start 2024-03-24T00:40:09.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.701+0000 7f73118b2640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f730c15a360 0x7f730c0cd770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:09.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.701+0000 7f73118b2640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f730c0b4d30 0x7f730c072410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:09.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.701+0000 7f73118b2640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f730c0cdd70 con 0x7f730c06f340 2024-03-24T00:40:09.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.701+0000 7f73118b2640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f730c0ce050 con 0x7f730c0b4d30 2024-03-24T00:40:09.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.701+0000 7f73118b2640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f730c0ce330 con 0x7f730c15a360 2024-03-24T00:40:09.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.701+0000 7f730a7fc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f730c0b4d30 0x7f730c072410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:09.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.701+0000 7f730a7fc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f730c0b4d30 0x7f730c072410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40926/0 (socket says 172.21.15.67:40926) 2024-03-24T00:40:09.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.701+0000 7f730a7fc640 1 -- 172.21.15.67:0/4108824934 learned_addr learned my addr 172.21.15.67:0/4108824934 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:09.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.701+0000 7f730affd640 1 --2- 172.21.15.67:0/4108824934 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f730c15a360 0x7f730c0cd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:09.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.701+0000 7f730a7fc640 1 -- 172.21.15.67:0/4108824934 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f730c15a360 msgr2=0x7f730c0cd770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:09.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.701+0000 7f730a7fc640 1 --2- 172.21.15.67:0/4108824934 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f730c15a360 0x7f730c0cd770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.701+0000 7f730a7fc640 1 -- 172.21.15.67:0/4108824934 >> v1:172.21.15.72:6789/0 conn(0x7f730c06f340 legacy=0x7f730c15df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:09.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.701+0000 7f730a7fc640 1 -- 172.21.15.67:0/4108824934 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f730c0ce610 con 0x7f730c0b4d30 2024-03-24T00:40:09.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.702+0000 7f730a7fc640 1 --2- 172.21.15.67:0/4108824934 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f730c0b4d30 0x7f730c072410 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f7300009870 tx=0x7f730002ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=554f591dc38b7ac7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:09.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.702+0000 7f7309ffb640 1 -- 172.21.15.67:0/4108824934 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730002e850 con 0x7f730c0b4d30 2024-03-24T00:40:09.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.702+0000 7f7309ffb640 1 -- 172.21.15.67:0/4108824934 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f730002e9f0 con 0x7f730c0b4d30 2024-03-24T00:40:09.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.702+0000 7f7309ffb640 1 -- 172.21.15.67:0/4108824934 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730002edc0 con 0x7f730c0b4d30 2024-03-24T00:40:09.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.702+0000 7f73118b2640 1 -- 172.21.15.67:0/4108824934 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f730c0b4d30 msgr2=0x7f730c072410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:09.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.702+0000 7f73118b2640 1 --2- 172.21.15.67:0/4108824934 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f730c0b4d30 0x7f730c072410 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f7300009870 tx=0x7f730002ca90 comp rx=0 tx=0).stop 2024-03-24T00:40:09.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.702+0000 7f73118b2640 1 -- 172.21.15.67:0/4108824934 shutdown_connections 2024-03-24T00:40:09.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.702+0000 7f73118b2640 1 --2- 172.21.15.67:0/4108824934 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f730c0b4d30 0x7f730c072410 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.702+0000 7f73118b2640 1 --2- 172.21.15.67:0/4108824934 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f730c15a360 0x7f730c0cd770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.702+0000 7f73118b2640 1 -- 172.21.15.67:0/4108824934 >> 172.21.15.67:0/4108824934 conn(0x7f730c0cfa70 msgr2=0x7f730c0cfe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:09.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.703+0000 7f73118b2640 1 -- 172.21.15.67:0/4108824934 shutdown_connections 2024-03-24T00:40:09.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.703+0000 7f73118b2640 1 -- 172.21.15.67:0/4108824934 wait complete. 2024-03-24T00:40:09.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.704+0000 7f73118b2640 1 Processor -- start 2024-03-24T00:40:09.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.704+0000 7f73118b2640 1 -- start start 2024-03-24T00:40:09.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.705+0000 7f73118b2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f730c06f340 0x7f730c15a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:09.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.705+0000 7f73118b2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f730c0b4d30 0x7f730c15c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:09.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.705+0000 7f73118b2640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f730c15a360 0x7f730c146670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:09.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.705+0000 7f73118b2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f730c06ecf0 con 0x7f730c06f340 2024-03-24T00:40:09.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.705+0000 7f73118b2640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f730c15a740 con 0x7f730c0b4d30 2024-03-24T00:40:09.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.705+0000 7f73118b2640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f730c06ee70 con 0x7f730c15a360 2024-03-24T00:40:09.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.705+0000 7f730affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f730c06f340 0x7f730c15a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:09.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.705+0000 7f730affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f730c06f340 0x7f730c15a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40934/0 (socket says 172.21.15.67:40934) 2024-03-24T00:40:09.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.705+0000 7f730affd640 1 -- 172.21.15.67:0/2261758906 learned_addr learned my addr 172.21.15.67:0/2261758906 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:09.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.705+0000 7f730a7fc640 1 --2- 172.21.15.67:0/2261758906 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f730c0b4d30 0x7f730c15c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:09.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.705+0000 7f730b7fe640 1 --2- 172.21.15.67:0/2261758906 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f730c15a360 0x7f730c146670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:09.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.705+0000 7f730affd640 1 -- 172.21.15.67:0/2261758906 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f730c15a360 msgr2=0x7f730c146670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:09.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.705+0000 7f730affd640 1 --2- 172.21.15.67:0/2261758906 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f730c15a360 0x7f730c146670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.705+0000 7f730affd640 1 -- 172.21.15.67:0/2261758906 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f730c0b4d30 msgr2=0x7f730c15c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:09.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.705+0000 7f730affd640 1 --2- 172.21.15.67:0/2261758906 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f730c0b4d30 0x7f730c15c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.705+0000 7f730affd640 1 -- 172.21.15.67:0/2261758906 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f730c15ae00 con 0x7f730c06f340 2024-03-24T00:40:09.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.705+0000 7f730affd640 1 --2- 172.21.15.67:0/2261758906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f730c06f340 0x7f730c15a8c0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f72fc00e870 tx=0x7f72fc00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:09.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.705+0000 7f72f3fff640 1 -- 172.21.15.67:0/2261758906 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72fc017070 con 0x7f730c06f340 2024-03-24T00:40:09.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.705+0000 7f72f3fff640 1 -- 172.21.15.67:0/2261758906 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72fc00c940 con 0x7f730c06f340 2024-03-24T00:40:09.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.705+0000 7f73118b2640 1 -- 172.21.15.67:0/2261758906 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f730c15c7d0 con 0x7f730c06f340 2024-03-24T00:40:09.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.705+0000 7f72f3fff640 1 -- 172.21.15.67:0/2261758906 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72fc00cd30 con 0x7f730c06f340 2024-03-24T00:40:09.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.705+0000 7f73118b2640 1 -- 172.21.15.67:0/2261758906 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f730c1f5710 con 0x7f730c06f340 2024-03-24T00:40:09.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.706+0000 7f73118b2640 1 -- 172.21.15.67:0/2261758906 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f730c04f400 con 0x7f730c06f340 2024-03-24T00:40:09.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.707+0000 7f72f3fff640 1 -- 172.21.15.67:0/2261758906 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f72fc01a070 con 0x7f730c06f340 2024-03-24T00:40:09.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.707+0000 7f72f3fff640 1 --2- 172.21.15.67:0/2261758906 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f72f806bbd0 0x7f72f806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:09.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.707+0000 7f72f3fff640 1 -- 172.21.15.67:0/2261758906 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f72fc08a970 con 0x7f730c06f340 2024-03-24T00:40:09.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.707+0000 7f730a7fc640 1 --2- 172.21.15.67:0/2261758906 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f72f806bbd0 0x7f72f806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:09.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.708+0000 7f730a7fc640 1 --2- 172.21.15.67:0/2261758906 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f72f806bbd0 0x7f72f806e090 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f7300037000 tx=0x7f730003a040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:09.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.712+0000 7f72f3fff640 1 -- 172.21.15.67:0/2261758906 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f72fc058d10 con 0x7f730c06f340 2024-03-24T00:40:09.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.963+0000 7f73118b2640 1 -- 172.21.15.67:0/2261758906 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f730c06f120 con 0x7f730c06f340 2024-03-24T00:40:09.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.965+0000 7f72f3fff640 1 -- 172.21.15.67:0/2261758906 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) v1 ==== 190+0+147 (secure 0 0 0) 0x7f72fc05cdb0 con 0x7f730c06f340 2024-03-24T00:40:09.966 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-03-24T00:40:09.967 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQDcdf9lY4nvJRAA6HWOZDWA+SSvJVmyjG+51g== 2024-03-24T00:40:09.967 INFO:teuthology.orchestra.run.smithi067.stdout: caps mds = "allow" 2024-03-24T00:40:09.967 INFO:teuthology.orchestra.run.smithi067.stdout: caps mon = "allow r" 2024-03-24T00:40:09.967 INFO:teuthology.orchestra.run.smithi067.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-24T00:40:09.967 INFO:teuthology.orchestra.run.smithi067.stderr:updated caps for client.0 2024-03-24T00:40:09.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.968+0000 7f73118b2640 1 -- 172.21.15.67:0/2261758906 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f72f806bbd0 msgr2=0x7f72f806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:09.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.968+0000 7f73118b2640 1 --2- 172.21.15.67:0/2261758906 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f72f806bbd0 0x7f72f806e090 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f7300037000 tx=0x7f730003a040 comp rx=0 tx=0).stop 2024-03-24T00:40:09.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.968+0000 7f73118b2640 1 -- 172.21.15.67:0/2261758906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f730c06f340 msgr2=0x7f730c15a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:09.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.968+0000 7f73118b2640 1 --2- 172.21.15.67:0/2261758906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f730c06f340 0x7f730c15a8c0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f72fc00e870 tx=0x7f72fc00ed30 comp rx=0 tx=0).stop 2024-03-24T00:40:09.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.968+0000 7f73118b2640 1 -- 172.21.15.67:0/2261758906 shutdown_connections 2024-03-24T00:40:09.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.968+0000 7f73118b2640 1 --2- 172.21.15.67:0/2261758906 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f72f806bbd0 0x7f72f806e090 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.968+0000 7f73118b2640 1 --2- 172.21.15.67:0/2261758906 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f730c15a360 0x7f730c146670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.968+0000 7f73118b2640 1 --2- 172.21.15.67:0/2261758906 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f730c0b4d30 0x7f730c15c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.968+0000 7f73118b2640 1 --2- 172.21.15.67:0/2261758906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f730c06f340 0x7f730c15a8c0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:09.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.968+0000 7f73118b2640 1 -- 172.21.15.67:0/2261758906 >> 172.21.15.67:0/2261758906 conn(0x7f730c0cfa70 msgr2=0x7f730c0b3050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:09.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.968+0000 7f73118b2640 1 -- 172.21.15.67:0/2261758906 shutdown_connections 2024-03-24T00:40:09.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:09.968+0000 7f73118b2640 1 -- 172.21.15.67:0/2261758906 wait complete. 2024-03-24T00:40:09.980 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:40:10.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.081+0000 7f6a653c4640 1 Processor -- start 2024-03-24T00:40:10.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.081+0000 7f6a653c4640 1 -- start start 2024-03-24T00:40:10.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.082+0000 7f6a653c4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f6a601445a0 0x7f6a600cee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:10.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.082+0000 7f6a653c4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6a600b48d0 0x7f6a60072550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:10.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.082+0000 7f6a653c4640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a600b4d30 con 0x7f6a6006ec40 2024-03-24T00:40:10.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.082+0000 7f6a653c4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a600ce990 con 0x7f6a601445a0 2024-03-24T00:40:10.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.082+0000 7f6a653c4640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a600cddc0 con 0x7f6a600b48d0 2024-03-24T00:40:10.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.082+0000 7f6a5f7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6a6006ec40 0x7f6a600b2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42424/0 (socket says 172.21.15.67:42424) 2024-03-24T00:40:10.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.082+0000 7f6a5f7fe640 1 -- 172.21.15.67:0/2255452992 learned_addr learned my addr 172.21.15.67:0/2255452992 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:40:10.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.082+0000 7f6a5effd640 1 --2- 172.21.15.67:0/2255452992 >> v2:172.21.15.72:3300/0 conn(0x7f6a601445a0 0x7f6a600cee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:10.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.082+0000 7f6a5e7fc640 1 --2- 172.21.15.67:0/2255452992 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6a600b48d0 0x7f6a60072550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:10.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.082+0000 7f6a5dffb640 1 -- 172.21.15.67:0/2255452992 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2752652191 0 0) 0x7f6a58001490 con 0x7f6a6006ec40 2024-03-24T00:40:10.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.082+0000 7f6a5dffb640 1 -- 172.21.15.67:0/2255452992 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a48003520 con 0x7f6a6006ec40 2024-03-24T00:40:10.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.082+0000 7f6a5effd640 1 -- 172.21.15.67:0/2255452992 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6a600b48d0 msgr2=0x7f6a60072550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:10.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.082+0000 7f6a5effd640 1 --2- 172.21.15.67:0/2255452992 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6a600b48d0 0x7f6a60072550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.082+0000 7f6a5effd640 1 -- 172.21.15.67:0/2255452992 >> v1:172.21.15.67:6789/0 conn(0x7f6a6006ec40 legacy=0x7f6a600b2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:10.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.083+0000 7f6a5effd640 1 -- 172.21.15.67:0/2255452992 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a600ce060 con 0x7f6a601445a0 2024-03-24T00:40:10.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.083+0000 7f6a5e7fc640 1 --2- 172.21.15.67:0/2255452992 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6a600b48d0 0x7f6a60072550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:40:10.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.083+0000 7f6a5effd640 1 --2- 172.21.15.67:0/2255452992 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a601445a0 0x7f6a600cee20 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f6a500097a0 tx=0x7f6a5002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=271834f1b0178db5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:10.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.083+0000 7f6a5dffb640 1 -- 172.21.15.67:0/2255452992 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a50035020 con 0x7f6a601445a0 2024-03-24T00:40:10.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.083+0000 7f6a5dffb640 1 -- 172.21.15.67:0/2255452992 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a50031830 con 0x7f6a601445a0 2024-03-24T00:40:10.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.083+0000 7f6a5dffb640 1 -- 172.21.15.67:0/2255452992 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a50031be0 con 0x7f6a601445a0 2024-03-24T00:40:10.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.084+0000 7f6a653c4640 1 -- 172.21.15.67:0/2255452992 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a601445a0 msgr2=0x7f6a600cee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:10.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.084+0000 7f6a653c4640 1 --2- 172.21.15.67:0/2255452992 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a601445a0 0x7f6a600cee20 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f6a500097a0 tx=0x7f6a5002b8d0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.084+0000 7f6a653c4640 1 -- 172.21.15.67:0/2255452992 shutdown_connections 2024-03-24T00:40:10.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.084+0000 7f6a653c4640 1 --2- 172.21.15.67:0/2255452992 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6a600b48d0 0x7f6a60072550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.084+0000 7f6a653c4640 1 --2- 172.21.15.67:0/2255452992 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a601445a0 0x7f6a600cee20 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.084+0000 7f6a653c4640 1 -- 172.21.15.67:0/2255452992 >> 172.21.15.67:0/2255452992 conn(0x7f6a600cf9e0 msgr2=0x7f6a600cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:10.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.084+0000 7f6a653c4640 1 -- 172.21.15.67:0/2255452992 shutdown_connections 2024-03-24T00:40:10.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.084+0000 7f6a653c4640 1 -- 172.21.15.67:0/2255452992 wait complete. 2024-03-24T00:40:10.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.085+0000 7f6a653c4640 1 Processor -- start 2024-03-24T00:40:10.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.085+0000 7f6a653c4640 1 -- start start 2024-03-24T00:40:10.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.086+0000 7f6a653c4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a6006ec40 0x7f6a601572f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:10.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.086+0000 7f6a653c4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a600b48d0 0x7f6a60071ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:10.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.086+0000 7f6a653c4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6a601445a0 0x7f6a60072010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:10.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.086+0000 7f6a653c4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6006f0d0 con 0x7f6a6006ec40 2024-03-24T00:40:10.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.086+0000 7f6a653c4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a600ce620 con 0x7f6a600b48d0 2024-03-24T00:40:10.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.086+0000 7f6a653c4640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6006f250 con 0x7f6a601445a0 2024-03-24T00:40:10.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.086+0000 7f6a5effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a6006ec40 0x7f6a601572f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:10.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.086+0000 7f6a5effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a6006ec40 0x7f6a601572f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40948/0 (socket says 172.21.15.67:40948) 2024-03-24T00:40:10.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.086+0000 7f6a5effd640 1 -- 172.21.15.67:0/3063760051 learned_addr learned my addr 172.21.15.67:0/3063760051 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:10.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.086+0000 7f6a5e7fc640 1 --2- 172.21.15.67:0/3063760051 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a600b48d0 0x7f6a60071ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:10.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.086+0000 7f6a5f7fe640 1 --2- 172.21.15.67:0/3063760051 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6a601445a0 0x7f6a60072010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:10.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.086+0000 7f6a5effd640 1 -- 172.21.15.67:0/3063760051 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6a601445a0 msgr2=0x7f6a60072010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:10.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.086+0000 7f6a5effd640 1 --2- 172.21.15.67:0/3063760051 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6a601445a0 0x7f6a60072010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.087+0000 7f6a5effd640 1 -- 172.21.15.67:0/3063760051 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a600b48d0 msgr2=0x7f6a60071ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:10.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.087+0000 7f6a5effd640 1 --2- 172.21.15.67:0/3063760051 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a600b48d0 0x7f6a60071ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.087+0000 7f6a5effd640 1 -- 172.21.15.67:0/3063760051 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a60154dc0 con 0x7f6a6006ec40 2024-03-24T00:40:10.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.087+0000 7f6a5effd640 1 --2- 172.21.15.67:0/3063760051 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a6006ec40 0x7f6a601572f0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f6a50034040 tx=0x7f6a500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:10.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.087+0000 7f6a477fe640 1 -- 172.21.15.67:0/3063760051 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a50040070 con 0x7f6a6006ec40 2024-03-24T00:40:10.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.087+0000 7f6a477fe640 1 -- 172.21.15.67:0/3063760051 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a5003e070 con 0x7f6a6006ec40 2024-03-24T00:40:10.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.087+0000 7f6a477fe640 1 -- 172.21.15.67:0/3063760051 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a50035040 con 0x7f6a6006ec40 2024-03-24T00:40:10.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.087+0000 7f6a653c4640 1 -- 172.21.15.67:0/3063760051 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a601550a0 con 0x7f6a6006ec40 2024-03-24T00:40:10.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.087+0000 7f6a653c4640 1 -- 172.21.15.67:0/3063760051 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a60155660 con 0x7f6a6006ec40 2024-03-24T00:40:10.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.088+0000 7f6a653c4640 1 -- 172.21.15.67:0/3063760051 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a6004f400 con 0x7f6a6006ec40 2024-03-24T00:40:10.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.088+0000 7f6a477fe640 1 -- 172.21.15.67:0/3063760051 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6a50038020 con 0x7f6a6006ec40 2024-03-24T00:40:10.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.089+0000 7f6a477fe640 1 --2- 172.21.15.67:0/3063760051 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6a4c06bbd0 0x7f6a4c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:10.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.089+0000 7f6a477fe640 1 -- 172.21.15.67:0/3063760051 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6a500ac290 con 0x7f6a6006ec40 2024-03-24T00:40:10.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.089+0000 7f6a5e7fc640 1 --2- 172.21.15.67:0/3063760051 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6a4c06bbd0 0x7f6a4c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:10.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.089+0000 7f6a5e7fc640 1 --2- 172.21.15.67:0/3063760051 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6a4c06bbd0 0x7f6a4c06e090 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6a54009560 tx=0x7f6a54009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:10.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.093+0000 7f6a477fe640 1 -- 172.21.15.67:0/3063760051 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6a5007a630 con 0x7f6a6006ec40 2024-03-24T00:40:10.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.313+0000 7f6a653c4640 1 -- 172.21.15.67:0/3063760051 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a600ceec0 con 0x7f6a6006ec40 2024-03-24T00:40:10.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.313+0000 7f6a477fe640 1 -- 172.21.15.67:0/3063760051 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f6a5007e6d0 con 0x7f6a6006ec40 2024-03-24T00:40:10.314 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:40:10.314 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5039,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/166859817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":166859817},{"type":"v1","addr":"172.21.15.67:6834","nonce":166859817}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5042,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2653663845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2653663845},{"type":"v1","addr":"172.21.15.67:6835","nonce":2653663845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5048,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2355909634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2355909634},{"type":"v1","addr":"172.21.15.72:6837","nonce":2355909634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:40:00.691429+0000","modified":"2024-03-24T00:40:03.146091+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5418},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5418":{"gid":5418,"name":"d","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6836/2551774148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2551774148},{"type":"v1","addr":"172.21.15.72:6836","nonce":2551774148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5418,"qdb_cluster":[5418]},"id":2}]} 2024-03-24T00:40:10.315 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 17 2024-03-24T00:40:10.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.316+0000 7f6a653c4640 1 -- 172.21.15.67:0/3063760051 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6a4c06bbd0 msgr2=0x7f6a4c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:10.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.316+0000 7f6a653c4640 1 --2- 172.21.15.67:0/3063760051 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6a4c06bbd0 0x7f6a4c06e090 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6a54009560 tx=0x7f6a54009190 comp rx=0 tx=0).stop 2024-03-24T00:40:10.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.316+0000 7f6a653c4640 1 -- 172.21.15.67:0/3063760051 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a6006ec40 msgr2=0x7f6a601572f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:10.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.316+0000 7f6a653c4640 1 --2- 172.21.15.67:0/3063760051 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a6006ec40 0x7f6a601572f0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f6a50034040 tx=0x7f6a500046c0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.316+0000 7f6a653c4640 1 -- 172.21.15.67:0/3063760051 shutdown_connections 2024-03-24T00:40:10.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.316+0000 7f6a653c4640 1 --2- 172.21.15.67:0/3063760051 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6a4c06bbd0 0x7f6a4c06e090 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.316+0000 7f6a653c4640 1 --2- 172.21.15.67:0/3063760051 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6a601445a0 0x7f6a60072010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.316+0000 7f6a653c4640 1 --2- 172.21.15.67:0/3063760051 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a600b48d0 0x7f6a60071ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.316+0000 7f6a653c4640 1 --2- 172.21.15.67:0/3063760051 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a6006ec40 0x7f6a601572f0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.316+0000 7f6a653c4640 1 -- 172.21.15.67:0/3063760051 >> 172.21.15.67:0/3063760051 conn(0x7f6a600cf9e0 msgr2=0x7f6a600cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:10.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.316+0000 7f6a653c4640 1 -- 172.21.15.67:0/3063760051 shutdown_connections 2024-03-24T00:40:10.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.316+0000 7f6a653c4640 1 -- 172.21.15.67:0/3063760051 wait complete. 2024-03-24T00:40:10.730 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:40:00.691429+0000', 'modified': '2024-03-24T00:40:03.146091+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5418}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5418': {'gid': 5418, 'name': 'd', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6836/2551774148', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2551774148}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2551774148}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5418, 'qdb_cluster': [5418]} 2024-03-24T00:40:10.730 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-24T00:40:10.731 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-24T00:40:10.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.836+0000 7f1208f74640 1 Processor -- start 2024-03-24T00:40:10.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.836+0000 7f1208f74640 1 -- start start 2024-03-24T00:40:10.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.837+0000 7f1208f74640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1204071b00 0x7f1204071ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:10.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.837+0000 7f1208f74640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f12040ce2a0 0x7f120406e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:10.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.837+0000 7f1208f74640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f120406f450 con 0x7f12040cde40 2024-03-24T00:40:10.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.837+0000 7f1208f74640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f120415d270 con 0x7f1204071b00 2024-03-24T00:40:10.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.837+0000 7f1208f74640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f120415f870 con 0x7f12040ce2a0 2024-03-24T00:40:10.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.837+0000 7f120259c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1204071b00 0x7f1204071ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:10.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.837+0000 7f120259c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1204071b00 0x7f1204071ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40960/0 (socket says 172.21.15.67:40960) 2024-03-24T00:40:10.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.837+0000 7f120259c640 1 -- 172.21.15.67:0/2685991695 learned_addr learned my addr 172.21.15.67:0/2685991695 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:10.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.837+0000 7f1202d9d640 1 --2- 172.21.15.67:0/2685991695 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f12040ce2a0 0x7f120406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:10.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.837+0000 7f120259c640 1 -- 172.21.15.67:0/2685991695 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f12040ce2a0 msgr2=0x7f120406e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:10.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.837+0000 7f120259c640 1 --2- 172.21.15.67:0/2685991695 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f12040ce2a0 0x7f120406e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.837+0000 7f120259c640 1 -- 172.21.15.67:0/2685991695 >> v1:172.21.15.72:6789/0 conn(0x7f12040cde40 legacy=0x7f120406dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:10.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.837+0000 7f120259c640 1 -- 172.21.15.67:0/2685991695 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1204072420 con 0x7f1204071b00 2024-03-24T00:40:10.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.838+0000 7f120259c640 1 --2- 172.21.15.67:0/2685991695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1204071b00 0x7f1204071ee0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f11f4009870 tx=0x7f11f402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eda9911f784ad786 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:10.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.838+0000 7f120159a640 1 -- 172.21.15.67:0/2685991695 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f4034070 con 0x7f1204071b00 2024-03-24T00:40:10.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.838+0000 7f120159a640 1 -- 172.21.15.67:0/2685991695 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11f4030840 con 0x7f1204071b00 2024-03-24T00:40:10.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.838+0000 7f120159a640 1 -- 172.21.15.67:0/2685991695 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f4030c10 con 0x7f1204071b00 2024-03-24T00:40:10.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.839+0000 7f1208f74640 1 -- 172.21.15.67:0/2685991695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1204071b00 msgr2=0x7f1204071ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:10.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.839+0000 7f1208f74640 1 --2- 172.21.15.67:0/2685991695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1204071b00 0x7f1204071ee0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f11f4009870 tx=0x7f11f402b980 comp rx=0 tx=0).stop 2024-03-24T00:40:10.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.839+0000 7f1208f74640 1 -- 172.21.15.67:0/2685991695 shutdown_connections 2024-03-24T00:40:10.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.839+0000 7f1208f74640 1 --2- 172.21.15.67:0/2685991695 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f12040ce2a0 0x7f120406e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.839+0000 7f1208f74640 1 --2- 172.21.15.67:0/2685991695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1204071b00 0x7f1204071ee0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.839+0000 7f1208f74640 1 -- 172.21.15.67:0/2685991695 >> 172.21.15.67:0/2685991695 conn(0x7f1204067470 msgr2=0x7f120406b040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:10.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.839+0000 7f1208f74640 1 -- 172.21.15.67:0/2685991695 shutdown_connections 2024-03-24T00:40:10.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.839+0000 7f1208f74640 1 -- 172.21.15.67:0/2685991695 wait complete. 2024-03-24T00:40:10.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.840+0000 7f1208f74640 1 Processor -- start 2024-03-24T00:40:10.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.840+0000 7f1208f74640 1 -- start start 2024-03-24T00:40:10.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.841+0000 7f1208f74640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1204071b00 0x7f12040ae220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:10.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.841+0000 7f1208f74640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12040cde40 0x7f120415a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:10.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.841+0000 7f1208f74640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f12040ce2a0 0x7f120415a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:10.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.841+0000 7f1208f74640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f120415fdf0 con 0x7f12040cde40 2024-03-24T00:40:10.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.841+0000 7f1208f74640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f120415fb10 con 0x7f1204071b00 2024-03-24T00:40:10.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.841+0000 7f1208f74640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12040ce730 con 0x7f12040ce2a0 2024-03-24T00:40:10.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.841+0000 7f1201d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12040cde40 0x7f120415a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:10.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.841+0000 7f1201d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12040cde40 0x7f120415a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40972/0 (socket says 172.21.15.67:40972) 2024-03-24T00:40:10.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.841+0000 7f1201d9b640 1 -- 172.21.15.67:0/3296782353 learned_addr learned my addr 172.21.15.67:0/3296782353 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:10.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.841+0000 7f120259c640 1 --2- 172.21.15.67:0/3296782353 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1204071b00 0x7f12040ae220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:10.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.841+0000 7f1202d9d640 1 --2- 172.21.15.67:0/3296782353 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f12040ce2a0 0x7f120415a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:10.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.841+0000 7f1201d9b640 1 -- 172.21.15.67:0/3296782353 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f12040ce2a0 msgr2=0x7f120415a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:10.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.841+0000 7f1201d9b640 1 --2- 172.21.15.67:0/3296782353 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f12040ce2a0 0x7f120415a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.841+0000 7f1201d9b640 1 -- 172.21.15.67:0/3296782353 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1204071b00 msgr2=0x7f12040ae220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:10.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.841+0000 7f1201d9b640 1 --2- 172.21.15.67:0/3296782353 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1204071b00 0x7f12040ae220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.841+0000 7f1201d9b640 1 -- 172.21.15.67:0/3296782353 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12040c5580 con 0x7f12040cde40 2024-03-24T00:40:10.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.841+0000 7f1201d9b640 1 --2- 172.21.15.67:0/3296782353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12040cde40 0x7f120415a220 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f11f800c400 tx=0x7f11f800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:10.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.842+0000 7f11f2ffd640 1 -- 172.21.15.67:0/3296782353 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f80052d0 con 0x7f12040cde40 2024-03-24T00:40:10.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.842+0000 7f11f2ffd640 1 -- 172.21.15.67:0/3296782353 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11f8005490 con 0x7f12040cde40 2024-03-24T00:40:10.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.842+0000 7f1208f74640 1 -- 172.21.15.67:0/3296782353 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12040c5860 con 0x7f12040cde40 2024-03-24T00:40:10.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.842+0000 7f11f2ffd640 1 -- 172.21.15.67:0/3296782353 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f8007260 con 0x7f12040cde40 2024-03-24T00:40:10.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.842+0000 7f1208f74640 1 -- 172.21.15.67:0/3296782353 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12040c5e20 con 0x7f12040cde40 2024-03-24T00:40:10.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.843+0000 7f11f2ffd640 1 -- 172.21.15.67:0/3296782353 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f11f8007400 con 0x7f12040cde40 2024-03-24T00:40:10.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.844+0000 7f11f2ffd640 1 --2- 172.21.15.67:0/3296782353 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f11e406bbd0 0x7f11e406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:10.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.844+0000 7f11f2ffd640 1 -- 172.21.15.67:0/3296782353 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f11f808a0d0 con 0x7f12040cde40 2024-03-24T00:40:10.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.844+0000 7f120259c640 1 --2- 172.21.15.67:0/3296782353 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f11e406bbd0 0x7f11e406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:10.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.844+0000 7f120259c640 1 --2- 172.21.15.67:0/3296782353 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f11e406bbd0 0x7f11e406e090 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f11f4036000 tx=0x7f11f4008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:10.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.845+0000 7f120bc36740 1 Processor -- start 2024-03-24T00:40:10.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.845+0000 7f120bc36740 1 -- start start 2024-03-24T00:40:10.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.846+0000 7f120bc36740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556185d89c40 0x556185db0770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:10.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.846+0000 7f120bc36740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556185db0cb0 0x556185d88c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:10.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.846+0000 7f120bc36740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556185d8a020 0x556185d891c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:10.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.846+0000 7f120bc36740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x556185cedc20 con 0x556185d8a020 2024-03-24T00:40:10.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.846+0000 7f120bc36740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x556185c96a40 con 0x556185db0cb0 2024-03-24T00:40:10.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.846+0000 7f120bc36740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x556185ce9cc0 con 0x556185d89c40 2024-03-24T00:40:10.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.846+0000 7f1202d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556185d8a020 0x556185d891c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:10.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.846+0000 7f1201d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556185db0cb0 0x556185d88c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:10.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.846+0000 7f1202d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556185d8a020 0x556185d891c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40980/0 (socket says 172.21.15.67:40980) 2024-03-24T00:40:10.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.846+0000 7f1202d9d640 1 -- 172.21.15.67:0/4176857887 learned_addr learned my addr 172.21.15.67:0/4176857887 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:10.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.846+0000 7f120259c640 1 --2- 172.21.15.67:0/4176857887 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556185d89c40 0x556185db0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:10.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.846+0000 7f1202d9d640 1 -- 172.21.15.67:0/4176857887 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556185d89c40 msgr2=0x556185db0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:10.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.846+0000 7f1202d9d640 1 --2- 172.21.15.67:0/4176857887 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556185d89c40 0x556185db0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.846+0000 7f1202d9d640 1 -- 172.21.15.67:0/4176857887 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556185db0cb0 msgr2=0x556185d88c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:10.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.846+0000 7f1202d9d640 1 --2- 172.21.15.67:0/4176857887 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556185db0cb0 0x556185d88c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.846+0000 7f1202d9d640 1 -- 172.21.15.67:0/4176857887 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556185cb8f90 con 0x556185d8a020 2024-03-24T00:40:10.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.846+0000 7f1201d9b640 1 --2- 172.21.15.67:0/4176857887 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556185db0cb0 0x556185d88c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:40:10.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.846+0000 7f1202d9d640 1 --2- 172.21.15.67:0/4176857887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556185d8a020 0x556185d891c0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f11fc009f10 tx=0x7f11fc003d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=462055ebd151a71a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:10.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.846+0000 7f11e3fff640 1 -- 172.21.15.67:0/4176857887 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11fc0047d0 con 0x556185d8a020 2024-03-24T00:40:10.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.846+0000 7f11e3fff640 1 -- 172.21.15.67:0/4176857887 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11fc01a070 con 0x556185d8a020 2024-03-24T00:40:10.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.846+0000 7f11e3fff640 1 -- 172.21.15.67:0/4176857887 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11fc015640 con 0x556185d8a020 2024-03-24T00:40:10.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.847+0000 7f120bc36740 1 -- 172.21.15.67:0/4176857887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556185d8a020 msgr2=0x556185d891c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:10.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.847+0000 7f120bc36740 1 --2- 172.21.15.67:0/4176857887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556185d8a020 0x556185d891c0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f11fc009f10 tx=0x7f11fc003d60 comp rx=0 tx=0).stop 2024-03-24T00:40:10.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.847+0000 7f120bc36740 1 -- 172.21.15.67:0/4176857887 shutdown_connections 2024-03-24T00:40:10.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.847+0000 7f120bc36740 1 --2- 172.21.15.67:0/4176857887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556185d8a020 0x556185d891c0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.847+0000 7f120bc36740 1 --2- 172.21.15.67:0/4176857887 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556185db0cb0 0x556185d88c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.847+0000 7f120bc36740 1 --2- 172.21.15.67:0/4176857887 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556185d89c40 0x556185db0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.847+0000 7f120bc36740 1 -- 172.21.15.67:0/4176857887 >> 172.21.15.67:0/4176857887 conn(0x556185d50a60 msgr2=0x556185dce960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:10.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.847+0000 7f120bc36740 1 -- 172.21.15.67:0/4176857887 shutdown_connections 2024-03-24T00:40:10.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.847+0000 7f120bc36740 1 -- 172.21.15.67:0/4176857887 wait complete. 2024-03-24T00:40:10.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.848+0000 7f120bc36740 1 -- start start 2024-03-24T00:40:10.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.848+0000 7f120bc36740 1 Processor -- start 2024-03-24T00:40:10.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.848+0000 7f120bc36740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556185d89c40 0x556185da7b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:10.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.849+0000 7f120bc36740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556185d8a020 0x556185dcddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:10.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.849+0000 7f120bc36740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556185dcce40 0x556185db66c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:10.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.849+0000 7f120bc36740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x556185db02e0 con 0x556185dcce40 2024-03-24T00:40:10.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.849+0000 7f120bc36740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x556185ca9e90 con 0x556185d89c40 2024-03-24T00:40:10.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.849+0000 7f120bc36740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x556185da9060 con 0x556185d8a020 2024-03-24T00:40:10.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.849+0000 7f120bc36740 10 client.0 _ll_register_callbacks cb 0x556185a01ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:40:10.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.849+0000 7f1202d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556185dcce40 0x556185db66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:10.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.849+0000 7f1202d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556185dcce40 0x556185db66c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40994/0 (socket says 172.21.15.67:40994) 2024-03-24T00:40:10.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.849+0000 7f1202d9d640 1 -- 172.21.15.67:0/435262136 learned_addr learned my addr 172.21.15.67:0/435262136 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:10.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.849+0000 7f120259c640 1 --2- 172.21.15.67:0/435262136 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556185d89c40 0x556185da7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:10.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.849+0000 7f1201d9b640 1 --2- 172.21.15.67:0/435262136 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556185d8a020 0x556185dcddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:10.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.849+0000 7f1202d9d640 1 -- 172.21.15.67:0/435262136 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556185d8a020 msgr2=0x556185dcddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:10.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.849+0000 7f1202d9d640 1 --2- 172.21.15.67:0/435262136 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556185d8a020 0x556185dcddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.849+0000 7f1202d9d640 1 -- 172.21.15.67:0/435262136 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556185d89c40 msgr2=0x556185da7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:10.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.849+0000 7f1202d9d640 1 --2- 172.21.15.67:0/435262136 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556185d89c40 0x556185da7b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.849+0000 7f1202d9d640 1 -- 172.21.15.67:0/435262136 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556185cb69d0 con 0x556185dcce40 2024-03-24T00:40:10.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.849+0000 7f1202d9d640 1 --2- 172.21.15.67:0/435262136 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556185dcce40 0x556185db66c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f11fc007040 tx=0x7f11fc004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df5b34b1bc57c70d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:10.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.849+0000 7f11e17fa640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:40:10.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.849+0000 7f11e17fa640 1 -- 172.21.15.67:0/435262136 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11fc0157f0 con 0x556185dcce40 2024-03-24T00:40:10.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.849+0000 7f11e17fa640 1 -- 172.21.15.67:0/435262136 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f11fc015990 con 0x556185dcce40 2024-03-24T00:40:10.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.849+0000 7f120bc36740 1 -- 172.21.15.67:0/435262136 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f11fc01a070 con 0x556185dcce40 2024-03-24T00:40:10.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.849+0000 7f11e17fa640 1 -- 172.21.15.67:0/435262136 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f11fc015b90 con 0x556185dcce40 2024-03-24T00:40:10.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.849+0000 7f120bc36740 10 client.5607 fetch_fsmap learned FSMap version 17 2024-03-24T00:40:10.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.849+0000 7f120bc36740 1 -- 172.21.15.67:0/435262136 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x556185a022d0 con 0x556185dcce40 2024-03-24T00:40:10.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.850+0000 7f11e17fa640 1 -- 172.21.15.67:0/435262136 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11fc01a070 con 0x556185dcce40 2024-03-24T00:40:10.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.850+0000 7f11e17fa640 1 -- 172.21.15.67:0/435262136 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 17) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f11fc033420 con 0x556185dcce40 2024-03-24T00:40:10.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.850+0000 7f120bc36740 10 client.5607 fetch_fsmap finished waiting for FSMap version 17 2024-03-24T00:40:10.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.850+0000 7f120bc36740 10 client.5607 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-24T00:40:10.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.850+0000 7f120bc36740 20 client.5607 populate_metadata read hostname 'smithi067' 2024-03-24T00:40:10.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.850+0000 7f120bc36740 1 --2- 172.21.15.67:0/435262136 >> [v2:172.21.15.72:6834/2551774148,v1:172.21.15.72:6836/2551774148] conn(0x556185de1eb0 0x556185dec020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:10.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.850+0000 7f120bc36740 4 client.5607 mds_command: new command op to 5418 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:40:10.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.850+0000 7f120bc36740 1 -- 172.21.15.67:0/435262136 --> [v2:172.21.15.72:6834/2551774148,v1:172.21.15.72:6836/2551774148] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556185cfb160 con 0x556185de1eb0 2024-03-24T00:40:10.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.850+0000 7f120259c640 1 --2- 172.21.15.67:0/435262136 >> [v2:172.21.15.72:6834/2551774148,v1:172.21.15.72:6836/2551774148] conn(0x556185de1eb0 0x556185dec020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:10.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.850+0000 7f11e17fa640 10 client.5607 ms_handle_connect on v2:172.21.15.72:6834/2551774148 2024-03-24T00:40:10.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.850+0000 7f120259c640 1 --2- 172.21.15.67:0/435262136 >> [v2:172.21.15.72:6834/2551774148,v1:172.21.15.72:6836/2551774148] conn(0x556185de1eb0 0x556185dec020 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ffbafe6a9c422462 server_cookie=4f7083d809fefc85 in_seq=0 out_seq=0 2024-03-24T00:40:10.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.851+0000 7f11e17fa640 1 -- 172.21.15.67:0/435262136 <== mds.0 v2:172.21.15.72:6834/2551774148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x556185cfb160 con 0x556185de1eb0 2024-03-24T00:40:10.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.851+0000 7f11e17fa640 10 client.5607 handle_command_reply: tid=0 2024-03-24T00:40:10.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.851+0000 7f120bc36740 1 client.5607 shutdown 2024-03-24T00:40:10.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.851+0000 7f120bc36740 1 -- 172.21.15.67:0/435262136 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556185dcce40 msgr2=0x556185db66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:10.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.851+0000 7f120bc36740 1 --2- 172.21.15.67:0/435262136 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556185dcce40 0x556185db66c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f11fc007040 tx=0x7f11fc004ab0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.851+0000 7f120bc36740 1 -- 172.21.15.67:0/435262136 shutdown_connections 2024-03-24T00:40:10.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.851+0000 7f120bc36740 1 --2- 172.21.15.67:0/435262136 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556185dcce40 0x556185db66c0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.851+0000 7f120bc36740 1 --2- 172.21.15.67:0/435262136 >> [v2:172.21.15.72:6834/2551774148,v1:172.21.15.72:6836/2551774148] conn(0x556185de1eb0 0x556185dec020 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.851+0000 7f120bc36740 1 --2- 172.21.15.67:0/435262136 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556185d8a020 0x556185dcddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.851+0000 7f11e17fa640 0 client.5607 ms_handle_reset on v2:172.21.15.72:6834/2551774148 2024-03-24T00:40:10.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.851+0000 7f120bc36740 1 --2- 172.21.15.67:0/435262136 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556185d89c40 0x556185da7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.852+0000 7f120bc36740 1 -- 172.21.15.67:0/435262136 >> 172.21.15.67:0/435262136 conn(0x556185d50a60 msgr2=0x556185db14f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:10.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.852+0000 7f120bc36740 1 -- 172.21.15.67:0/435262136 shutdown_connections 2024-03-24T00:40:10.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.852+0000 7f120bc36740 1 -- 172.21.15.67:0/435262136 wait complete. 2024-03-24T00:40:10.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.852+0000 7f120bc36740 20 client.5607 trim_cache size 0 max 16384 2024-03-24T00:40:10.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.868+0000 7f120bc36740 1 Processor -- start 2024-03-24T00:40:10.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.868+0000 7f120bc36740 1 -- start start 2024-03-24T00:40:10.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.869+0000 7f120bc36740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556185d89c40 0x556185dcc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:10.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.869+0000 7f120bc36740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556185d8a020 0x556185db62a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:10.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.869+0000 7f120bc36740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556185dcce40 0x556185df0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:10.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.869+0000 7f120bc36740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12041ffb90 con 0x556185dcce40 2024-03-24T00:40:10.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.869+0000 7f120bc36740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12041f50d0 con 0x556185d89c40 2024-03-24T00:40:10.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.869+0000 7f120bc36740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12041f56d0 con 0x556185d8a020 2024-03-24T00:40:10.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.869+0000 7f1202d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556185dcce40 0x556185df0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:10.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.869+0000 7f1202d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556185dcce40 0x556185df0040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41008/0 (socket says 172.21.15.67:41008) 2024-03-24T00:40:10.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.869+0000 7f1202d9d640 1 -- 172.21.15.67:0/1478991709 learned_addr learned my addr 172.21.15.67:0/1478991709 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:10.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.869+0000 7f1201d9b640 1 --2- 172.21.15.67:0/1478991709 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556185d8a020 0x556185db62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:10.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.869+0000 7f120259c640 1 --2- 172.21.15.67:0/1478991709 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556185d89c40 0x556185dcc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:10.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.869+0000 7f1202d9d640 1 -- 172.21.15.67:0/1478991709 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556185d8a020 msgr2=0x556185db62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:10.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.869+0000 7f1202d9d640 1 --2- 172.21.15.67:0/1478991709 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556185d8a020 0x556185db62a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.869+0000 7f1202d9d640 1 -- 172.21.15.67:0/1478991709 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556185d89c40 msgr2=0x556185dcc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:10.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.869+0000 7f1202d9d640 1 --2- 172.21.15.67:0/1478991709 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556185d89c40 0x556185dcc270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.869+0000 7f1202d9d640 1 -- 172.21.15.67:0/1478991709 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556185acc6b0 con 0x556185dcce40 2024-03-24T00:40:10.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.870+0000 7f1202d9d640 1 --2- 172.21.15.67:0/1478991709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556185dcce40 0x556185df0040 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f11fc007070 tx=0x7f11fc020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b56ee8a6332313b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:10.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.870+0000 7f11e37fe640 1 -- 172.21.15.67:0/1478991709 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11fc009b30 con 0x556185dcce40 2024-03-24T00:40:10.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.870+0000 7f11e37fe640 1 -- 172.21.15.67:0/1478991709 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11fc01a040 con 0x556185dcce40 2024-03-24T00:40:10.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.870+0000 7f11e37fe640 1 -- 172.21.15.67:0/1478991709 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11fc033c80 con 0x556185dcce40 2024-03-24T00:40:10.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.870+0000 7f120bc36740 1 -- 172.21.15.67:0/1478991709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556185dcce40 msgr2=0x556185df0040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:10.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.870+0000 7f120bc36740 1 --2- 172.21.15.67:0/1478991709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556185dcce40 0x556185df0040 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f11fc007070 tx=0x7f11fc020000 comp rx=0 tx=0).stop 2024-03-24T00:40:10.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.870+0000 7f120bc36740 1 -- 172.21.15.67:0/1478991709 shutdown_connections 2024-03-24T00:40:10.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.870+0000 7f120bc36740 1 --2- 172.21.15.67:0/1478991709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556185dcce40 0x556185df0040 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.870+0000 7f120bc36740 1 --2- 172.21.15.67:0/1478991709 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556185d8a020 0x556185db62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.870+0000 7f120bc36740 1 --2- 172.21.15.67:0/1478991709 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556185d89c40 0x556185dcc270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.870+0000 7f120bc36740 1 -- 172.21.15.67:0/1478991709 >> 172.21.15.67:0/1478991709 conn(0x556185d50a60 msgr2=0x556185dafd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:10.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.870+0000 7f120bc36740 1 -- 172.21.15.67:0/1478991709 shutdown_connections 2024-03-24T00:40:10.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.870+0000 7f120bc36740 1 -- 172.21.15.67:0/1478991709 wait complete. 2024-03-24T00:40:10.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.871+0000 7f120bc36740 1 -- start start 2024-03-24T00:40:10.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.871+0000 7f120bc36740 1 Processor -- start 2024-03-24T00:40:10.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.872+0000 7f120bc36740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556185d89c40 0x556185dc4f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:10.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.872+0000 7f120bc36740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556185d8a020 0x556185e25b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:10.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.872+0000 7f120bc36740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556185dcce40 0x556185df08c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:10.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.872+0000 7f120bc36740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x556185de1d30 con 0x556185d89c40 2024-03-24T00:40:10.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.872+0000 7f120bc36740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x556185dcef30 con 0x556185dcce40 2024-03-24T00:40:10.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.872+0000 7f120bc36740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x556185de2300 con 0x556185d8a020 2024-03-24T00:40:10.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.872+0000 7f120bc36740 10 client.0 _ll_register_callbacks cb 0x7f11fc009650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:40:10.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.872+0000 7f120259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556185d89c40 0x556185dc4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:10.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.872+0000 7f120259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556185d89c40 0x556185dc4f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41010/0 (socket says 172.21.15.67:41010) 2024-03-24T00:40:10.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.872+0000 7f120259c640 1 -- 172.21.15.67:0/626028377 learned_addr learned my addr 172.21.15.67:0/626028377 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:10.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.872+0000 7f1201d9b640 1 --2- 172.21.15.67:0/626028377 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556185d8a020 0x556185e25b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:10.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.872+0000 7f1202d9d640 1 --2- 172.21.15.67:0/626028377 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556185dcce40 0x556185df08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:10.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.872+0000 7f120259c640 1 -- 172.21.15.67:0/626028377 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556185d8a020 msgr2=0x556185e25b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:10.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.872+0000 7f120259c640 1 --2- 172.21.15.67:0/626028377 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556185d8a020 0x556185e25b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.872+0000 7f120259c640 1 -- 172.21.15.67:0/626028377 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556185dcce40 msgr2=0x556185df08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:10.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.872+0000 7f120259c640 1 --2- 172.21.15.67:0/626028377 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556185dcce40 0x556185df08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.872+0000 7f120259c640 1 -- 172.21.15.67:0/626028377 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556185e34380 con 0x556185d89c40 2024-03-24T00:40:10.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.872+0000 7f120259c640 1 --2- 172.21.15.67:0/626028377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556185d89c40 0x556185dc4f50 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f11f4006c80 tx=0x7f11f4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66579931e8b88fad server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:10.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.872+0000 7f11e17fa640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:40:10.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.873+0000 7f11e17fa640 1 -- 172.21.15.67:0/626028377 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f4034040 con 0x556185d89c40 2024-03-24T00:40:10.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.873+0000 7f11e17fa640 1 -- 172.21.15.67:0/626028377 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f11f4044670 con 0x556185d89c40 2024-03-24T00:40:10.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.873+0000 7f120bc36740 1 -- 172.21.15.67:0/626028377 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f11fc01a040 con 0x556185d89c40 2024-03-24T00:40:10.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.873+0000 7f11e17fa640 1 -- 172.21.15.67:0/626028377 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f11f4065050 con 0x556185d89c40 2024-03-24T00:40:10.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.873+0000 7f120bc36740 10 client.5613 fetch_fsmap learned FSMap version 17 2024-03-24T00:40:10.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.873+0000 7f120bc36740 1 -- 172.21.15.67:0/626028377 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x556185dc4750 con 0x556185d89c40 2024-03-24T00:40:10.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.873+0000 7f11e17fa640 1 -- 172.21.15.67:0/626028377 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11fc01a040 con 0x556185d89c40 2024-03-24T00:40:10.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.873+0000 7f11e17fa640 1 -- 172.21.15.67:0/626028377 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 17) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f11f4068020 con 0x556185d89c40 2024-03-24T00:40:10.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.873+0000 7f120bc36740 10 client.5613 fetch_fsmap finished waiting for FSMap version 17 2024-03-24T00:40:10.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.873+0000 7f120bc36740 10 client.5613 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-24T00:40:10.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.873+0000 7f120bc36740 20 client.5613 populate_metadata read hostname 'smithi067' 2024-03-24T00:40:10.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.873+0000 7f120bc36740 1 --2- 172.21.15.67:0/626028377 >> [v2:172.21.15.72:6834/2551774148,v1:172.21.15.72:6836/2551774148] conn(0x556185daf6b0 0x556185e345a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:10.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.873+0000 7f120bc36740 4 client.5613 mds_command: new command op to 5418 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:40:10.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.873+0000 7f120bc36740 1 -- 172.21.15.67:0/626028377 --> [v2:172.21.15.72:6834/2551774148,v1:172.21.15.72:6836/2551774148] -- command(tid 0: {"prefix": "status"}) v1 -- 0x556185cf54b0 con 0x556185daf6b0 2024-03-24T00:40:10.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.873+0000 7f1201d9b640 1 --2- 172.21.15.67:0/626028377 >> [v2:172.21.15.72:6834/2551774148,v1:172.21.15.72:6836/2551774148] conn(0x556185daf6b0 0x556185e345a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:10.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.874+0000 7f11e17fa640 10 client.5613 ms_handle_connect on v2:172.21.15.72:6834/2551774148 2024-03-24T00:40:10.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.874+0000 7f1201d9b640 1 --2- 172.21.15.67:0/626028377 >> [v2:172.21.15.72:6834/2551774148,v1:172.21.15.72:6836/2551774148] conn(0x556185daf6b0 0x556185e345a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee868f83ce590a1 server_cookie=ebd29cc8f11ff122 in_seq=0 out_seq=0 2024-03-24T00:40:10.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.874+0000 7f11e17fa640 1 -- 172.21.15.67:0/626028377 <== mds.0 v2:172.21.15.72:6834/2551774148 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x556185cf54b0 con 0x556185daf6b0 2024-03-24T00:40:10.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.874+0000 7f11e17fa640 10 client.5613 handle_command_reply: tid=0 2024-03-24T00:40:10.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.874+0000 7f120bc36740 1 client.5613 shutdown 2024-03-24T00:40:10.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.874+0000 7f120bc36740 1 -- 172.21.15.67:0/626028377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556185d89c40 msgr2=0x556185dc4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:10.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.874+0000 7f120bc36740 1 --2- 172.21.15.67:0/626028377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556185d89c40 0x556185dc4f50 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f11f4006c80 tx=0x7f11f4004730 comp rx=0 tx=0).stop 2024-03-24T00:40:10.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.874+0000 7f120bc36740 1 -- 172.21.15.67:0/626028377 shutdown_connections 2024-03-24T00:40:10.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.874+0000 7f120bc36740 1 --2- 172.21.15.67:0/626028377 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556185dcce40 0x556185df08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.874+0000 7f120bc36740 1 --2- 172.21.15.67:0/626028377 >> [v2:172.21.15.72:6834/2551774148,v1:172.21.15.72:6836/2551774148] conn(0x556185daf6b0 0x556185e345a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.874+0000 7f120bc36740 1 --2- 172.21.15.67:0/626028377 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556185d8a020 0x556185e25b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.874+0000 7f120bc36740 1 --2- 172.21.15.67:0/626028377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556185d89c40 0x556185dc4f50 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.874+0000 7f11e17fa640 0 client.5613 ms_handle_reset on v2:172.21.15.72:6834/2551774148 2024-03-24T00:40:10.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.874+0000 7f120bc36740 1 -- 172.21.15.67:0/626028377 >> 172.21.15.67:0/626028377 conn(0x556185d50a60 msgr2=0x556185da7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:10.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.874+0000 7f120bc36740 1 -- 172.21.15.67:0/626028377 shutdown_connections 2024-03-24T00:40:10.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.874+0000 7f120bc36740 1 -- 172.21.15.67:0/626028377 wait complete. 2024-03-24T00:40:10.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.874+0000 7f120bc36740 20 client.5613 trim_cache size 0 max 16384 2024-03-24T00:40:10.880 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:40:10.880 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "de1d0178-808e-49ca-a986-e297e523b976", 2024-03-24T00:40:10.880 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-03-24T00:40:10.881 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 5418, 2024-03-24T00:40:10.881 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-03-24T00:40:10.881 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-03-24T00:40:10.881 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-03-24T00:40:10.881 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 9.550079792, 2024-03-24T00:40:10.881 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 17, 2024-03-24T00:40:10.881 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 39, 2024-03-24T00:40:10.881 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 39, 2024-03-24T00:40:10.881 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 16.732016145999999 2024-03-24T00:40:10.881 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:40:10.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.875+0000 7f1208f74640 1 -- 172.21.15.67:0/3296782353 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f11e406bbd0 msgr2=0x7f11e406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:10.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.875+0000 7f1208f74640 1 --2- 172.21.15.67:0/3296782353 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f11e406bbd0 0x7f11e406e090 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f11f4036000 tx=0x7f11f4008030 comp rx=0 tx=0).stop 2024-03-24T00:40:10.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.875+0000 7f1208f74640 1 -- 172.21.15.67:0/3296782353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12040cde40 msgr2=0x7f120415a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:10.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.875+0000 7f1208f74640 1 --2- 172.21.15.67:0/3296782353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12040cde40 0x7f120415a220 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f11f800c400 tx=0x7f11f800c8c0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.875+0000 7f1208f74640 1 -- 172.21.15.67:0/3296782353 shutdown_connections 2024-03-24T00:40:10.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.875+0000 7f1208f74640 1 --2- 172.21.15.67:0/3296782353 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f11e406bbd0 0x7f11e406e090 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.875+0000 7f1208f74640 1 --2- 172.21.15.67:0/3296782353 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f12040ce2a0 0x7f120415a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.875+0000 7f1208f74640 1 --2- 172.21.15.67:0/3296782353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12040cde40 0x7f120415a220 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.875+0000 7f1208f74640 1 --2- 172.21.15.67:0/3296782353 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1204071b00 0x7f12040ae220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.875+0000 7f1208f74640 1 -- 172.21.15.67:0/3296782353 >> 172.21.15.67:0/3296782353 conn(0x7f1204067470 msgr2=0x7f12040b2080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:10.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.875+0000 7f1208f74640 1 -- 172.21.15.67:0/3296782353 shutdown_connections 2024-03-24T00:40:10.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.875+0000 7f1208f74640 1 -- 172.21.15.67:0/3296782353 wait complete. 2024-03-24T00:40:10.885 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T00:40:10.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.987+0000 7f8cb2eea640 1 Processor -- start 2024-03-24T00:40:10.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.987+0000 7f8cb2eea640 1 -- start start 2024-03-24T00:40:10.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.987+0000 7f8cb2eea640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8cac15e130 0x7f8cac0cde00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:10.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.987+0000 7f8cb2eea640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8cac0b5ac0 0x7f8cac0b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:10.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.988+0000 7f8cb2eea640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cac0701e0 con 0x7f8cac0b2770 2024-03-24T00:40:10.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.988+0000 7f8cb2eea640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cac0d00b0 con 0x7f8cac0b5ac0 2024-03-24T00:40:10.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.988+0000 7f8cb2eea640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cac0704c0 con 0x7f8cac15e130 2024-03-24T00:40:10.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.988+0000 7f8cabfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8cac0b5ac0 0x7f8cac0b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:10.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.988+0000 7f8cabfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8cac0b5ac0 0x7f8cac0b4f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41016/0 (socket says 172.21.15.67:41016) 2024-03-24T00:40:10.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.988+0000 7f8cabfff640 1 -- 172.21.15.67:0/2404325569 learned_addr learned my addr 172.21.15.67:0/2404325569 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:10.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.988+0000 7f8cb0c86640 1 --2- 172.21.15.67:0/2404325569 >> v2:172.21.15.72:3300/0 conn(0x7f8cac15e130 0x7f8cac0cde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:10.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.988+0000 7f8cabfff640 1 -- 172.21.15.67:0/2404325569 >> v2:172.21.15.72:3300/0 conn(0x7f8cac15e130 msgr2=0x7f8cac0cde00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:10.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.988+0000 7f8cabfff640 1 --2- 172.21.15.67:0/2404325569 >> v2:172.21.15.72:3300/0 conn(0x7f8cac15e130 0x7f8cac0cde00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.988+0000 7f8cabfff640 1 -- 172.21.15.67:0/2404325569 >> v1:172.21.15.72:6789/0 conn(0x7f8cac0b2770 legacy=0x7f8cac06bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:10.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.988+0000 7f8cabfff640 1 -- 172.21.15.67:0/2404325569 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cac06b170 con 0x7f8cac0b5ac0 2024-03-24T00:40:10.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.988+0000 7f8cb0c86640 1 --2- 172.21.15.67:0/2404325569 >> v2:172.21.15.72:3300/0 conn(0x7f8cac15e130 0x7f8cac0cde00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:40:10.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.989+0000 7f8cabfff640 1 --2- 172.21.15.67:0/2404325569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cac0b5ac0 0x7f8cac0b4f20 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f8cac0b62e0 tx=0x7f8c9802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b67a48b1d28ee1d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:10.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.989+0000 7f8cab7fe640 1 -- 172.21.15.67:0/2404325569 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c98035020 con 0x7f8cac0b5ac0 2024-03-24T00:40:10.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.989+0000 7f8cab7fe640 1 -- 172.21.15.67:0/2404325569 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c98031870 con 0x7f8cac0b5ac0 2024-03-24T00:40:10.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.989+0000 7f8cab7fe640 1 -- 172.21.15.67:0/2404325569 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c98031c40 con 0x7f8cac0b5ac0 2024-03-24T00:40:10.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.989+0000 7f8cb2eea640 1 -- 172.21.15.67:0/2404325569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cac0b5ac0 msgr2=0x7f8cac0b4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:10.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.989+0000 7f8cb2eea640 1 --2- 172.21.15.67:0/2404325569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cac0b5ac0 0x7f8cac0b4f20 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f8cac0b62e0 tx=0x7f8c9802b910 comp rx=0 tx=0).stop 2024-03-24T00:40:10.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.989+0000 7f8cb2eea640 1 -- 172.21.15.67:0/2404325569 shutdown_connections 2024-03-24T00:40:10.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.989+0000 7f8cb2eea640 1 --2- 172.21.15.67:0/2404325569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cac0b5ac0 0x7f8cac0b4f20 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.989+0000 7f8cb2eea640 1 --2- 172.21.15.67:0/2404325569 >> v2:172.21.15.72:3300/0 conn(0x7f8cac15e130 0x7f8cac0cde00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.989+0000 7f8cb2eea640 1 -- 172.21.15.67:0/2404325569 >> 172.21.15.67:0/2404325569 conn(0x7f8cac0b3c40 msgr2=0x7f8cac0b1dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:10.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.990+0000 7f8cb2eea640 1 -- 172.21.15.67:0/2404325569 shutdown_connections 2024-03-24T00:40:10.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.990+0000 7f8cb2eea640 1 -- 172.21.15.67:0/2404325569 wait complete. 2024-03-24T00:40:10.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.991+0000 7f8cb2eea640 1 Processor -- start 2024-03-24T00:40:10.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.991+0000 7f8cb2eea640 1 -- start start 2024-03-24T00:40:10.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.992+0000 7f8cb2eea640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8cac0b2770 0x7f8cac1575a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:10.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.992+0000 7f8cb2eea640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8cac0b5ac0 0x7f8cac0c5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:10.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.992+0000 7f8cb2eea640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cac15e130 0x7f8cac0cee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:10.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.992+0000 7f8cb2eea640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cac15e510 con 0x7f8cac15e130 2024-03-24T00:40:10.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.992+0000 7f8cb2eea640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cac06b7c0 con 0x7f8cac0b5ac0 2024-03-24T00:40:10.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.992+0000 7f8cb2eea640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cac06fe00 con 0x7f8cac0b2770 2024-03-24T00:40:10.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.992+0000 7f8cb1487640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cac15e130 0x7f8cac0cee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:10.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.992+0000 7f8cb0c86640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8cac0b2770 0x7f8cac1575a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:10.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.992+0000 7f8cabfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8cac0b5ac0 0x7f8cac0c5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:10.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.992+0000 7f8cb0c86640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8cac0b2770 0x7f8cac1575a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.67:48312/0 (socket says 172.21.15.67:48312) 2024-03-24T00:40:10.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.992+0000 7f8cb1487640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cac15e130 0x7f8cac0cee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41022/0 (socket says 172.21.15.67:41022) 2024-03-24T00:40:10.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.992+0000 7f8cb0c86640 1 -- 172.21.15.67:0/4159169747 learned_addr learned my addr 172.21.15.67:0/4159169747 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:10.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.992+0000 7f8cabfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8cac0b5ac0 0x7f8cac0c5fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:41250/0 (socket says 172.21.15.67:41250) 2024-03-24T00:40:10.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.992+0000 7f8cb0c86640 1 -- 172.21.15.67:0/4159169747 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8cac0b5ac0 msgr2=0x7f8cac0c5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:10.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.992+0000 7f8cb0c86640 1 --2- 172.21.15.67:0/4159169747 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8cac0b5ac0 0x7f8cac0c5fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.992+0000 7f8cb0c86640 1 -- 172.21.15.67:0/4159169747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cac15e130 msgr2=0x7f8cac0cee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:10.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.992+0000 7f8cb0c86640 1 --2- 172.21.15.67:0/4159169747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cac15e130 0x7f8cac0cee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:10.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.992+0000 7f8cb0c86640 1 -- 172.21.15.67:0/4159169747 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cac157ae0 con 0x7f8cac0b2770 2024-03-24T00:40:10.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.992+0000 7f8cabfff640 1 --2- 172.21.15.67:0/4159169747 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8cac0b5ac0 0x7f8cac0c5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:40:10.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.992+0000 7f8cb1487640 1 --2- 172.21.15.67:0/4159169747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cac15e130 0x7f8cac0cee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:40:10.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.992+0000 7f8cb0c86640 1 --2- 172.21.15.67:0/4159169747 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8cac0b2770 0x7f8cac1575a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8ca000e960 tx=0x7f8ca000ee20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:10.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.992+0000 7f8ca97fa640 1 -- 172.21.15.67:0/4159169747 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca0017070 con 0x7f8cac0b2770 2024-03-24T00:40:10.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.992+0000 7f8ca97fa640 1 -- 172.21.15.67:0/4159169747 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ca000ca00 con 0x7f8cac0b2770 2024-03-24T00:40:10.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.992+0000 7f8cb2eea640 1 -- 172.21.15.67:0/4159169747 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cac0c6990 con 0x7f8cac0b2770 2024-03-24T00:40:10.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.992+0000 7f8ca97fa640 1 -- 172.21.15.67:0/4159169747 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca000cdd0 con 0x7f8cac0b2770 2024-03-24T00:40:10.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.993+0000 7f8cb2eea640 1 -- 172.21.15.67:0/4159169747 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cac1584b0 con 0x7f8cac0b2770 2024-03-24T00:40:10.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.993+0000 7f8cb2eea640 1 -- 172.21.15.67:0/4159169747 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cac04f400 con 0x7f8cac0b2770 2024-03-24T00:40:10.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.994+0000 7f8ca97fa640 1 -- 172.21.15.67:0/4159169747 <== mon.2 v2:172.21.15.72:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8ca001a070 con 0x7f8cac0b2770 2024-03-24T00:40:10.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.994+0000 7f8ca97fa640 1 --2- 172.21.15.67:0/4159169747 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8c9006bbd0 0x7f8c9006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:10.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.994+0000 7f8ca97fa640 1 -- 172.21.15.67:0/4159169747 <== mon.2 v2:172.21.15.72:3301/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8ca008ae10 con 0x7f8cac0b2770 2024-03-24T00:40:10.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.995+0000 7f8cabfff640 1 --2- 172.21.15.67:0/4159169747 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8c9006bbd0 0x7f8c9006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:10.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.995+0000 7f8cabfff640 1 --2- 172.21.15.67:0/4159169747 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8c9006bbd0 0x7f8c9006e090 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f8c98036000 tx=0x7f8c9803c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:10.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:10.998+0000 7f8ca97fa640 1 -- 172.21.15.67:0/4159169747 <== mon.2 v2:172.21.15.72:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8ca0059170 con 0x7f8cac0b2770 2024-03-24T00:40:11.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:11.217+0000 7f8cb2eea640 1 -- 172.21.15.67:0/4159169747 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f8cac0cd9c0 con 0x7f8cac0b2770 2024-03-24T00:40:11.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:11.218+0000 7f8ca97fa640 1 -- 172.21.15.67:0/4159169747 <== mon.2 v2:172.21.15.72:3301/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+73 (secure 0 0 0) 0x7f8ca005d210 con 0x7f8cac0b2770 2024-03-24T00:40:11.218 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-24T00:40:11.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:11.220+0000 7f8cb2eea640 1 -- 172.21.15.67:0/4159169747 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8c9006bbd0 msgr2=0x7f8c9006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:11.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:11.220+0000 7f8cb2eea640 1 --2- 172.21.15.67:0/4159169747 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8c9006bbd0 0x7f8c9006e090 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f8c98036000 tx=0x7f8c9803c040 comp rx=0 tx=0).stop 2024-03-24T00:40:11.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:11.220+0000 7f8cb2eea640 1 -- 172.21.15.67:0/4159169747 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8cac0b2770 msgr2=0x7f8cac1575a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:11.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:11.220+0000 7f8cb2eea640 1 --2- 172.21.15.67:0/4159169747 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8cac0b2770 0x7f8cac1575a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8ca000e960 tx=0x7f8ca000ee20 comp rx=0 tx=0).stop 2024-03-24T00:40:11.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:11.220+0000 7f8cb2eea640 1 -- 172.21.15.67:0/4159169747 shutdown_connections 2024-03-24T00:40:11.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:11.220+0000 7f8cb2eea640 1 --2- 172.21.15.67:0/4159169747 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8c9006bbd0 0x7f8c9006e090 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:11.220+0000 7f8cb2eea640 1 --2- 172.21.15.67:0/4159169747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cac15e130 0x7f8cac0cee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:11.220+0000 7f8cb2eea640 1 --2- 172.21.15.67:0/4159169747 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8cac0b5ac0 0x7f8cac0c5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:11.220+0000 7f8cb2eea640 1 --2- 172.21.15.67:0/4159169747 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8cac0b2770 0x7f8cac1575a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:11.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:11.220+0000 7f8cb2eea640 1 -- 172.21.15.67:0/4159169747 >> 172.21.15.67:0/4159169747 conn(0x7f8cac0b3c40 msgr2=0x7f8cac0c8640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:11.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:11.220+0000 7f8cb2eea640 1 -- 172.21.15.67:0/4159169747 shutdown_connections 2024-03-24T00:40:11.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:11.220+0000 7f8cb2eea640 1 -- 172.21.15.67:0/4159169747 wait complete. 2024-03-24T00:40:11.230 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-24T00:40:11.230 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T00:40:11.230 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-03-24T00:40:11.231 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-24T00:40:11.231 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T00:40:11.231 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-24T00:40:11.231 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-24T00:40:11.231 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T00:40:11.231 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:40:11.231 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:40:11.231 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-03-24T00:40:11.246 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T00:40:11.246 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T00:40:11.247 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T00:40:11.247 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:40:11.247 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-03-24T00:40:11.247 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:40:11.247 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:40:11.247 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-03-24T00:40:11.247 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-03-24T00:40:11.247 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:40:11.247 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-03-24T00:40:11.247 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-03-24T00:40:11.247 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:40:11.247 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-03-24T00:40:11.247 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T00:40:11.247 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-03-24T00:40:11.248 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-24T00:40:11.248 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 42502sec preferred_lft 42502sec 2024-03-24T00:40:11.248 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-03-24T00:40:11.248 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:40:11.248 INFO:teuthology.orchestra.run.smithi067.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T00:40:11.248 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 96:cf:26:c8:bf:62 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:40:11.248 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-24T00:40:11.248 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:40:11.248 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::7c6a:5fff:feaa:d41/64 scope link 2024-03-24T00:40:11.248 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:40:11.248 INFO:teuthology.orchestra.run.smithi067.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T00:40:11.248 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ea:22:4b:d5:48:3d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:40:11.248 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::e822:4bff:fed5:483d/64 scope link 2024-03-24T00:40:11.248 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:40:11.248 INFO:teuthology.orchestra.run.smithi067.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T00:40:11.249 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 96:cf:26:c8:bf:62 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:40:11.249 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::94cf:26ff:fec8:bf62/64 scope link 2024-03-24T00:40:11.249 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:40:11.249 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:40:11.249 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T00:40:11.311 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-03-24T00:40:11.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:40:11.313 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-03-24T00:40:11.368 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-03-24T00:40:11.368 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T00:40:11.369 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-24T00:40:11.369 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-24T00:40:11.369 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T00:40:11.369 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:40:11.369 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-03-24T00:40:11.369 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-03-24T00:40:11.369 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-24T00:40:11.369 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-24T00:40:11.369 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-24T00:40:11.369 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-03-24T00:40:11.464 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:40:11.464 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-03-24T00:40:11.478 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-24T00:40:11.478 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-24T00:40:11.479 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:40:11.479 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:40:11.479 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-03-24T00:40:11.545 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:40:11.545 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-24T00:40:11.600 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:40:11.600 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:40:11.600 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:40:11.654 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-24T00:40:11.654 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:40:11.654 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:40:11.738 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:40:11.738 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T00:40:11.738 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-03-24T00:40:11.746 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:40:11.746 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T00:40:11.747 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-03-24T00:40:11.747 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.72:3300,172.21.15.72:3301" 2024-03-24T00:40:11.747 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-24T00:40:11.751 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:40:11.751 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T00:40:11.751 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-03-24T00:40:11.751 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.72:3300,172.21.15.72:3301" 2024-03-24T00:40:11.751 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-24T00:40:11.751 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:6789,172.21.15.72:6789,172.21.15.72:6790" 2024-03-24T00:40:11.751 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.67:6789,172.21.15.72:6789,172.21.15.72:6790:/ 2024-03-24T00:40:11.751 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=de1d0178-808e-49ca-a986-e297e523b976" will pass to kernel 2024-03-24T00:40:11.752 INFO:teuthology.orchestra.run.smithi067.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-24T00:40:11.752 INFO:teuthology.orchestra.run.smithi067.stdout: You just mounted a file system that supports labels which does not 2024-03-24T00:40:11.752 INFO:teuthology.orchestra.run.smithi067.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-24T00:40:11.752 INFO:teuthology.orchestra.run.smithi067.stdout: applications will generate AVC messages and not be allowed access to 2024-03-24T00:40:11.752 INFO:teuthology.orchestra.run.smithi067.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-24T00:40:11.753 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-24T00:40:11.753 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:40:11.754 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:40:11.820 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:40:11.821 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T00:40:11.880 INFO:teuthology.orchestra.run.smithi067.stdout:client5542 2024-03-24T00:40:11.881 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:40:11.881 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:40:11.935 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:40:11.936 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir parentdir) 2024-03-24T00:40:11.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:40:11.995 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir parentdir/childdir) 2024-03-24T00:40:12.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:40:12.050 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch rfile) 2024-03-24T00:40:12.105 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:40:12.105 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch parentdir/pfile) 2024-03-24T00:40:12.164 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:40:12.164 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch parentdir/childdir/cfile) 2024-03-24T00:40:12.221 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:40:12.221 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec find ./) 2024-03-24T00:40:12.278 INFO:teuthology.orchestra.run.smithi067.stdout:./ 2024-03-24T00:40:12.278 INFO:teuthology.orchestra.run.smithi067.stdout:./rfile 2024-03-24T00:40:12.278 INFO:teuthology.orchestra.run.smithi067.stdout:./parentdir 2024-03-24T00:40:12.278 INFO:teuthology.orchestra.run.smithi067.stdout:./parentdir/pfile 2024-03-24T00:40:12.278 INFO:teuthology.orchestra.run.smithi067.stdout:./parentdir/childdir 2024-03-24T00:40:12.278 INFO:teuthology.orchestra.run.smithi067.stdout:./parentdir/childdir/cfile 2024-03-24T00:40:12.279 DEBUG:teuthology.orchestra.run.smithi067:> stdin-killer -- python3 -c ' 2024-03-24T00:40:12.279 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-03-24T00:40:12.279 DEBUG:teuthology.orchestra.run.smithi067:> import stat 2024-03-24T00:40:12.279 DEBUG:teuthology.orchestra.run.smithi067:> 2024-03-24T00:40:12.279 DEBUG:teuthology.orchestra.run.smithi067:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./").st_ino) 2024-03-24T00:40:12.279 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-03-24T00:40:12.383 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-03-24T00:40:12.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:40:12.391 DEBUG:teuthology.orchestra.run.smithi067:> stdin-killer -- python3 -c ' 2024-03-24T00:40:12.392 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-03-24T00:40:12.392 DEBUG:teuthology.orchestra.run.smithi067:> import stat 2024-03-24T00:40:12.392 DEBUG:teuthology.orchestra.run.smithi067:> 2024-03-24T00:40:12.392 DEBUG:teuthology.orchestra.run.smithi067:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./rfile").st_ino) 2024-03-24T00:40:12.392 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-03-24T00:40:12.496 INFO:teuthology.orchestra.run.smithi067.stdout:1099511627778 2024-03-24T00:40:12.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:40:12.505 DEBUG:teuthology.orchestra.run.smithi067:> stdin-killer -- python3 -c ' 2024-03-24T00:40:12.505 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-03-24T00:40:12.505 DEBUG:teuthology.orchestra.run.smithi067:> import stat 2024-03-24T00:40:12.505 DEBUG:teuthology.orchestra.run.smithi067:> 2024-03-24T00:40:12.505 DEBUG:teuthology.orchestra.run.smithi067:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parentdir").st_ino) 2024-03-24T00:40:12.505 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-03-24T00:40:12.609 INFO:teuthology.orchestra.run.smithi067.stdout:1099511627776 2024-03-24T00:40:12.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:40:12.618 DEBUG:teuthology.orchestra.run.smithi067:> stdin-killer -- python3 -c ' 2024-03-24T00:40:12.619 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-03-24T00:40:12.619 DEBUG:teuthology.orchestra.run.smithi067:> import stat 2024-03-24T00:40:12.619 DEBUG:teuthology.orchestra.run.smithi067:> 2024-03-24T00:40:12.619 DEBUG:teuthology.orchestra.run.smithi067:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parentdir/pfile").st_ino) 2024-03-24T00:40:12.619 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-03-24T00:40:12.723 INFO:teuthology.orchestra.run.smithi067.stdout:1099511628279 2024-03-24T00:40:12.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:40:12.732 DEBUG:teuthology.orchestra.run.smithi067:> stdin-killer -- python3 -c ' 2024-03-24T00:40:12.732 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-03-24T00:40:12.732 DEBUG:teuthology.orchestra.run.smithi067:> import stat 2024-03-24T00:40:12.732 DEBUG:teuthology.orchestra.run.smithi067:> 2024-03-24T00:40:12.732 DEBUG:teuthology.orchestra.run.smithi067:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parentdir/childdir").st_ino) 2024-03-24T00:40:12.732 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-03-24T00:40:12.837 INFO:teuthology.orchestra.run.smithi067.stdout:1099511627777 2024-03-24T00:40:12.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:40:12.846 DEBUG:teuthology.orchestra.run.smithi067:> stdin-killer -- python3 -c ' 2024-03-24T00:40:12.846 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-03-24T00:40:12.846 DEBUG:teuthology.orchestra.run.smithi067:> import stat 2024-03-24T00:40:12.846 DEBUG:teuthology.orchestra.run.smithi067:> 2024-03-24T00:40:12.846 DEBUG:teuthology.orchestra.run.smithi067:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parentdir/childdir/cfile").st_ino) 2024-03-24T00:40:12.846 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-03-24T00:40:12.951 INFO:teuthology.orchestra.run.smithi067.stdout:1099511628280 2024-03-24T00:40:12.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:40:12.960 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:40:12.960 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:40:13.016 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:40:13.016 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:40:13.070 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:40:13.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.212+0000 7f31a9aa1640 1 Processor -- start 2024-03-24T00:40:13.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.212+0000 7f31a9aa1640 1 -- start start 2024-03-24T00:40:13.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.213+0000 7f31a9aa1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f31a415e390 0x7f31a406b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:13.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.213+0000 7f31a9aa1640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f31a40cded0 0x7f31a406b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:13.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.213+0000 7f31a9aa1640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31a40d07d0 con 0x7f31a415dfb0 2024-03-24T00:40:13.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.213+0000 7f31a9aa1640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31a40d0590 con 0x7f31a40cded0 2024-03-24T00:40:13.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.213+0000 7f31a9aa1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31a40d0ab0 con 0x7f31a415e390 2024-03-24T00:40:13.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.213+0000 7f31a3fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f31a40cded0 0x7f31a406b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:13.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.213+0000 7f31a2ffd640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f31a415e390 0x7f31a406b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:13.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.213+0000 7f31a3fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f31a40cded0 0x7f31a406b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:41264/0 (socket says 172.21.15.67:41264) 2024-03-24T00:40:13.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.213+0000 7f31a2ffd640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f31a415e390 0x7f31a406b040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.67:48328/0 (socket says 172.21.15.67:48328) 2024-03-24T00:40:13.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.213+0000 7f31a37fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f31a415dfb0 0x7f31a40cfe80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:49004/0 (socket says 172.21.15.67:49004) 2024-03-24T00:40:13.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.213+0000 7f31a3fff640 1 -- 172.21.15.67:0/3046839147 learned_addr learned my addr 172.21.15.67:0/3046839147 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:13.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.213+0000 7f31a3fff640 1 -- 172.21.15.67:0/3046839147 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f31a415e390 msgr2=0x7f31a406b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:13.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.213+0000 7f31a3fff640 1 --2- 172.21.15.67:0/3046839147 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f31a415e390 0x7f31a406b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:13.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.213+0000 7f31a3fff640 1 -- 172.21.15.67:0/3046839147 >> v1:172.21.15.72:6789/0 conn(0x7f31a415dfb0 legacy=0x7f31a40cfe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:13.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.213+0000 7f31a3fff640 1 -- 172.21.15.67:0/3046839147 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a40d0d90 con 0x7f31a40cded0 2024-03-24T00:40:13.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.214+0000 7f31a3fff640 1 --2- 172.21.15.67:0/3046839147 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31a40cded0 0x7f31a406b580 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f319c009e50 tx=0x7f319c02bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7dddaf48d2c8cf74 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:13.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.214+0000 7f31a27fc640 1 -- 172.21.15.67:0/3046839147 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f319c02cb00 con 0x7f31a40cded0 2024-03-24T00:40:13.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.214+0000 7f31a27fc640 1 -- 172.21.15.67:0/3046839147 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f319c02cca0 con 0x7f31a40cded0 2024-03-24T00:40:13.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.214+0000 7f31a27fc640 1 -- 172.21.15.67:0/3046839147 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f319c034610 con 0x7f31a40cded0 2024-03-24T00:40:13.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.215+0000 7f31a9aa1640 1 -- 172.21.15.67:0/3046839147 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31a40cded0 msgr2=0x7f31a406b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:13.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.215+0000 7f31a9aa1640 1 --2- 172.21.15.67:0/3046839147 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31a40cded0 0x7f31a406b580 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f319c009e50 tx=0x7f319c02bf60 comp rx=0 tx=0).stop 2024-03-24T00:40:13.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.215+0000 7f31a9aa1640 1 -- 172.21.15.67:0/3046839147 shutdown_connections 2024-03-24T00:40:13.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.215+0000 7f31a9aa1640 1 --2- 172.21.15.67:0/3046839147 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31a40cded0 0x7f31a406b580 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:13.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.215+0000 7f31a9aa1640 1 --2- 172.21.15.67:0/3046839147 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f31a415e390 0x7f31a406b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:13.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.215+0000 7f31a9aa1640 1 -- 172.21.15.67:0/3046839147 >> 172.21.15.67:0/3046839147 conn(0x7f31a4099a30 msgr2=0x7f31a4099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:13.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.215+0000 7f31a9aa1640 1 -- 172.21.15.67:0/3046839147 shutdown_connections 2024-03-24T00:40:13.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.215+0000 7f31a9aa1640 1 -- 172.21.15.67:0/3046839147 wait complete. 2024-03-24T00:40:13.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.216+0000 7f31a9aa1640 1 Processor -- start 2024-03-24T00:40:13.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.216+0000 7f31a9aa1640 1 -- start start 2024-03-24T00:40:13.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.217+0000 7f31a9aa1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f31a40cded0 0x7f31a4160980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:13.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.217+0000 7f31a9aa1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31a415dfb0 0x7f31a4150d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:13.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.217+0000 7f31a9aa1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31a415e390 0x7f31a406ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:13.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.217+0000 7f31a9aa1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a40d1870 con 0x7f31a415e390 2024-03-24T00:40:13.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.217+0000 7f31a9aa1640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a40d1350 con 0x7f31a415dfb0 2024-03-24T00:40:13.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.217+0000 7f31a9aa1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31a40cb120 con 0x7f31a40cded0 2024-03-24T00:40:13.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.217+0000 7f31a3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31a415e390 0x7f31a406ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:13.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.217+0000 7f31a3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31a415e390 0x7f31a406ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41024/0 (socket says 172.21.15.67:41024) 2024-03-24T00:40:13.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.217+0000 7f31a3fff640 1 -- 172.21.15.67:0/3842539714 learned_addr learned my addr 172.21.15.67:0/3842539714 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:13.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.217+0000 7f31a37fe640 1 --2- 172.21.15.67:0/3842539714 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f31a40cded0 0x7f31a4160980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:13.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.217+0000 7f31a2ffd640 1 --2- 172.21.15.67:0/3842539714 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31a415dfb0 0x7f31a4150d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:13.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.218+0000 7f31a3fff640 1 -- 172.21.15.67:0/3842539714 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f31a40cded0 msgr2=0x7f31a4160980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:13.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.218+0000 7f31a3fff640 1 --2- 172.21.15.67:0/3842539714 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f31a40cded0 0x7f31a4160980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:13.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.218+0000 7f31a3fff640 1 -- 172.21.15.67:0/3842539714 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31a415dfb0 msgr2=0x7f31a4150d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:13.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.218+0000 7f31a3fff640 1 --2- 172.21.15.67:0/3842539714 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31a415dfb0 0x7f31a4150d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:13.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.218+0000 7f31a3fff640 1 -- 172.21.15.67:0/3842539714 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a40ca910 con 0x7f31a415e390 2024-03-24T00:40:13.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.218+0000 7f31a3fff640 1 --2- 172.21.15.67:0/3842539714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31a415e390 0x7f31a406ec50 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f319c036040 tx=0x7f319c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:13.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.218+0000 7f318bfff640 1 -- 172.21.15.67:0/3842539714 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f319c043020 con 0x7f31a415e390 2024-03-24T00:40:13.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.218+0000 7f318bfff640 1 -- 172.21.15.67:0/3842539714 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f319c02ce50 con 0x7f31a415e390 2024-03-24T00:40:13.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.218+0000 7f31a9aa1640 1 -- 172.21.15.67:0/3842539714 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31a40cabb0 con 0x7f31a415e390 2024-03-24T00:40:13.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.218+0000 7f318bfff640 1 -- 172.21.15.67:0/3842539714 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f319c009510 con 0x7f31a415e390 2024-03-24T00:40:13.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.218+0000 7f31a9aa1640 1 -- 172.21.15.67:0/3842539714 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31a4161380 con 0x7f31a415e390 2024-03-24T00:40:13.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.219+0000 7f31a9aa1640 1 -- 172.21.15.67:0/3842539714 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31a404f400 con 0x7f31a415e390 2024-03-24T00:40:13.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.219+0000 7f318bfff640 1 -- 172.21.15.67:0/3842539714 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f319c046020 con 0x7f31a415e390 2024-03-24T00:40:13.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.220+0000 7f318bfff640 1 --2- 172.21.15.67:0/3842539714 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f319006bbd0 0x7f319006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:13.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.220+0000 7f318bfff640 1 -- 172.21.15.67:0/3842539714 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f319c0acf60 con 0x7f31a415e390 2024-03-24T00:40:13.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.220+0000 7f31a37fe640 1 --2- 172.21.15.67:0/3842539714 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f319006bbd0 0x7f319006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:13.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.220+0000 7f31a37fe640 1 --2- 172.21.15.67:0/3842539714 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f319006bbd0 0x7f319006e090 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3194007fd0 tx=0x7f3194007e80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:13.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.224+0000 7f318bfff640 1 -- 172.21.15.67:0/3842539714 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f319c04d030 con 0x7f31a415e390 2024-03-24T00:40:13.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.406+0000 7f31a9aa1640 1 -- 172.21.15.67:0/3842539714 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f31a40cad80 con 0x7f31a415e390 2024-03-24T00:40:13.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.407+0000 7f318bfff640 1 -- 172.21.15.67:0/3842539714 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) v1 ==== 74+0+16882 (secure 0 0 0) 0x7f319c07b420 con 0x7f31a415e390 2024-03-24T00:40:13.407 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:40:13.407 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":39,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:40:02.323736+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:39:58.975995+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-24T00:39:59.653615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:40:13.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.409+0000 7f31a9aa1640 1 -- 172.21.15.67:0/3842539714 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f319006bbd0 msgr2=0x7f319006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:13.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.409+0000 7f31a9aa1640 1 --2- 172.21.15.67:0/3842539714 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f319006bbd0 0x7f319006e090 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3194007fd0 tx=0x7f3194007e80 comp rx=0 tx=0).stop 2024-03-24T00:40:13.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.409+0000 7f31a9aa1640 1 -- 172.21.15.67:0/3842539714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31a415e390 msgr2=0x7f31a406ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:13.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.409+0000 7f31a9aa1640 1 --2- 172.21.15.67:0/3842539714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31a415e390 0x7f31a406ec50 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f319c036040 tx=0x7f319c004a40 comp rx=0 tx=0).stop 2024-03-24T00:40:13.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.409+0000 7f31a9aa1640 1 -- 172.21.15.67:0/3842539714 shutdown_connections 2024-03-24T00:40:13.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.409+0000 7f31a9aa1640 1 --2- 172.21.15.67:0/3842539714 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f319006bbd0 0x7f319006e090 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:13.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.409+0000 7f31a9aa1640 1 --2- 172.21.15.67:0/3842539714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31a415e390 0x7f31a406ec50 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:13.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.409+0000 7f31a9aa1640 1 --2- 172.21.15.67:0/3842539714 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31a415dfb0 0x7f31a4150d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:13.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.409+0000 7f31a9aa1640 1 --2- 172.21.15.67:0/3842539714 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f31a40cded0 0x7f31a4160980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:13.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.409+0000 7f31a9aa1640 1 -- 172.21.15.67:0/3842539714 >> 172.21.15.67:0/3842539714 conn(0x7f31a4099a30 msgr2=0x7f31a40b61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:13.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.409+0000 7f31a9aa1640 1 -- 172.21.15.67:0/3842539714 shutdown_connections 2024-03-24T00:40:13.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.409+0000 7f31a9aa1640 1 -- 172.21.15.67:0/3842539714 wait complete. 2024-03-24T00:40:13.420 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:38:39.364950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:39:58.975995+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:39:59.653615+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:40:13.420 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:40:13.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.525+0000 7f1261ad1640 1 Processor -- start 2024-03-24T00:40:13.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.525+0000 7f1261ad1640 1 -- start start 2024-03-24T00:40:13.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.526+0000 7f1261ad1640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f125c099d00 0x7f125c09a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:13.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.526+0000 7f1261ad1640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f125c0d1530 0x7f125c15d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:13.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.526+0000 7f1261ad1640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f125c09a620 con 0x7f125c0b49b0 2024-03-24T00:40:13.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.526+0000 7f1261ad1640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f125c0cc1b0 con 0x7f125c0d1530 2024-03-24T00:40:13.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.526+0000 7f1261ad1640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f125c0cc400 con 0x7f125c099d00 2024-03-24T00:40:13.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.526+0000 7f125bfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f125c0d1530 0x7f125c15d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:13.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.526+0000 7f125bfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f125c0d1530 0x7f125c15d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53134/0 (socket says 172.21.15.67:53134) 2024-03-24T00:40:13.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.526+0000 7f125bfff640 1 -- 172.21.15.67:0/1035137074 learned_addr learned my addr 172.21.15.67:0/1035137074 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:13.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.526+0000 7f125b7fe640 1 --2- 172.21.15.67:0/1035137074 >> v2:172.21.15.72:3300/0 conn(0x7f125c099d00 0x7f125c09a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:13.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.526+0000 7f125bfff640 1 -- 172.21.15.67:0/1035137074 >> v2:172.21.15.72:3300/0 conn(0x7f125c099d00 msgr2=0x7f125c09a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:13.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.526+0000 7f125bfff640 1 --2- 172.21.15.67:0/1035137074 >> v2:172.21.15.72:3300/0 conn(0x7f125c099d00 0x7f125c09a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:13.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.526+0000 7f125bfff640 1 -- 172.21.15.67:0/1035137074 >> v1:172.21.15.72:6789/0 conn(0x7f125c0b49b0 legacy=0x7f125c0d0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:13.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.526+0000 7f125a7fc640 1 -- 172.21.15.67:0/1035137074 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 615243582 0 0) 0x7f125c09a620 con 0x7f125c0b49b0 2024-03-24T00:40:13.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.526+0000 7f125bfff640 1 -- 172.21.15.67:0/1035137074 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f125c0cc6e0 con 0x7f125c0d1530 2024-03-24T00:40:13.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.527+0000 7f125bfff640 1 --2- 172.21.15.67:0/1035137074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f125c0d1530 0x7f125c15d7d0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f125c08b260 tx=0x7f125402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=406ff3d3ea2446a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:13.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.527+0000 7f125a7fc640 1 -- 172.21.15.67:0/1035137074 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f125402cb20 con 0x7f125c0d1530 2024-03-24T00:40:13.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.527+0000 7f125a7fc640 1 -- 172.21.15.67:0/1035137074 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f125402ccc0 con 0x7f125c0d1530 2024-03-24T00:40:13.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.527+0000 7f125a7fc640 1 -- 172.21.15.67:0/1035137074 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1254034610 con 0x7f125c0d1530 2024-03-24T00:40:13.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.528+0000 7f1261ad1640 1 -- 172.21.15.67:0/1035137074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f125c0d1530 msgr2=0x7f125c15d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:13.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.528+0000 7f1261ad1640 1 --2- 172.21.15.67:0/1035137074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f125c0d1530 0x7f125c15d7d0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f125c08b260 tx=0x7f125402bf80 comp rx=0 tx=0).stop 2024-03-24T00:40:13.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.528+0000 7f1261ad1640 1 -- 172.21.15.67:0/1035137074 shutdown_connections 2024-03-24T00:40:13.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.528+0000 7f1261ad1640 1 --2- 172.21.15.67:0/1035137074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f125c0d1530 0x7f125c15d7d0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:13.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.528+0000 7f1261ad1640 1 --2- 172.21.15.67:0/1035137074 >> v2:172.21.15.72:3300/0 conn(0x7f125c099d00 0x7f125c09a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:13.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.528+0000 7f1261ad1640 1 -- 172.21.15.67:0/1035137074 >> 172.21.15.67:0/1035137074 conn(0x7f125c0b3650 msgr2=0x7f125c0b3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:13.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.528+0000 7f1261ad1640 1 -- 172.21.15.67:0/1035137074 shutdown_connections 2024-03-24T00:40:13.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.528+0000 7f1261ad1640 1 -- 172.21.15.67:0/1035137074 wait complete. 2024-03-24T00:40:13.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.529+0000 7f1261ad1640 1 Processor -- start 2024-03-24T00:40:13.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.529+0000 7f1261ad1640 1 -- start start 2024-03-24T00:40:13.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.530+0000 7f1261ad1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f125c099d00 0x7f125c150200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:13.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.530+0000 7f1261ad1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f125c0b49b0 0x7f125c0c55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:13.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.530+0000 7f1261ad1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f125c0c5b00 0x7f125c0710d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:13.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.530+0000 7f1261ad1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f125c0d01c0 con 0x7f125c0b49b0 2024-03-24T00:40:13.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.530+0000 7f1261ad1640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f125c0ccca0 con 0x7f125c099d00 2024-03-24T00:40:13.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.530+0000 7f1261ad1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f125c0b32a0 con 0x7f125c0c5b00 2024-03-24T00:40:13.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.530+0000 7f125affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f125c0b49b0 0x7f125c0c55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:13.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.530+0000 7f125affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f125c0b49b0 0x7f125c0c55c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53146/0 (socket says 172.21.15.67:53146) 2024-03-24T00:40:13.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.530+0000 7f125affd640 1 -- 172.21.15.67:0/1583381450 learned_addr learned my addr 172.21.15.67:0/1583381450 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:13.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.530+0000 7f125b7fe640 1 --2- 172.21.15.67:0/1583381450 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f125c099d00 0x7f125c150200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:13.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.530+0000 7f125bfff640 1 --2- 172.21.15.67:0/1583381450 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f125c0c5b00 0x7f125c0710d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:13.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.530+0000 7f125affd640 1 -- 172.21.15.67:0/1583381450 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f125c0c5b00 msgr2=0x7f125c0710d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:13.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.530+0000 7f125affd640 1 --2- 172.21.15.67:0/1583381450 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f125c0c5b00 0x7f125c0710d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:13.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.530+0000 7f125affd640 1 -- 172.21.15.67:0/1583381450 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f125c099d00 msgr2=0x7f125c150200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:13.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.530+0000 7f125affd640 1 --2- 172.21.15.67:0/1583381450 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f125c099d00 0x7f125c150200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:13.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.530+0000 7f125affd640 1 -- 172.21.15.67:0/1583381450 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f125c0c5fa0 con 0x7f125c0b49b0 2024-03-24T00:40:13.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.531+0000 7f125affd640 1 --2- 172.21.15.67:0/1583381450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f125c0b49b0 0x7f125c0c55c0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f125000c520 tx=0x7f125000c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:13.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.531+0000 7f1243fff640 1 -- 172.21.15.67:0/1583381450 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1250017070 con 0x7f125c0b49b0 2024-03-24T00:40:13.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.531+0000 7f1243fff640 1 -- 172.21.15.67:0/1583381450 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12500035f0 con 0x7f125c0b49b0 2024-03-24T00:40:13.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.531+0000 7f1243fff640 1 -- 172.21.15.67:0/1583381450 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12500039e0 con 0x7f125c0b49b0 2024-03-24T00:40:13.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.531+0000 7f1261ad1640 1 -- 172.21.15.67:0/1583381450 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f125c0c6280 con 0x7f125c0b49b0 2024-03-24T00:40:13.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.531+0000 7f1261ad1640 1 -- 172.21.15.67:0/1583381450 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f125c1f5260 con 0x7f125c0b49b0 2024-03-24T00:40:13.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.532+0000 7f1261ad1640 1 -- 172.21.15.67:0/1583381450 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f125c04f400 con 0x7f125c0b49b0 2024-03-24T00:40:13.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.532+0000 7f1243fff640 1 -- 172.21.15.67:0/1583381450 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f125001a070 con 0x7f125c0b49b0 2024-03-24T00:40:13.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.533+0000 7f1243fff640 1 --2- 172.21.15.67:0/1583381450 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f124806bbd0 0x7f124806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:13.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.533+0000 7f1243fff640 1 -- 172.21.15.67:0/1583381450 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1250089f50 con 0x7f125c0b49b0 2024-03-24T00:40:13.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.533+0000 7f125b7fe640 1 --2- 172.21.15.67:0/1583381450 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f124806bbd0 0x7f124806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:13.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.533+0000 7f125b7fe640 1 --2- 172.21.15.67:0/1583381450 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f124806bbd0 0x7f124806e090 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f125c15cd10 tx=0x7f124c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:13.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.537+0000 7f1243fff640 1 -- 172.21.15.67:0/1583381450 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f12500582f0 con 0x7f125c0b49b0 2024-03-24T00:40:13.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.718+0000 7f1261ad1640 1 -- 172.21.15.67:0/1583381450 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f125c0741e0 con 0x7f125c0b49b0 2024-03-24T00:40:13.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.718+0000 7f1243fff640 1 -- 172.21.15.67:0/1583381450 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v39) v1 ==== 93+0+10 (secure 0 0 0) 0x7f125005c390 con 0x7f125c0b49b0 2024-03-24T00:40:13.718 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-24T00:40:13.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.720+0000 7f1261ad1640 1 -- 172.21.15.67:0/1583381450 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f124806bbd0 msgr2=0x7f124806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:13.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.720+0000 7f1261ad1640 1 --2- 172.21.15.67:0/1583381450 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f124806bbd0 0x7f124806e090 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f125c15cd10 tx=0x7f124c009190 comp rx=0 tx=0).stop 2024-03-24T00:40:13.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.720+0000 7f1261ad1640 1 -- 172.21.15.67:0/1583381450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f125c0b49b0 msgr2=0x7f125c0c55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:13.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.720+0000 7f1261ad1640 1 --2- 172.21.15.67:0/1583381450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f125c0b49b0 0x7f125c0c55c0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f125000c520 tx=0x7f125000c9e0 comp rx=0 tx=0).stop 2024-03-24T00:40:13.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.720+0000 7f1261ad1640 1 -- 172.21.15.67:0/1583381450 shutdown_connections 2024-03-24T00:40:13.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.720+0000 7f1261ad1640 1 --2- 172.21.15.67:0/1583381450 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f124806bbd0 0x7f124806e090 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:13.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.720+0000 7f1261ad1640 1 --2- 172.21.15.67:0/1583381450 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f125c0c5b00 0x7f125c0710d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:13.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.720+0000 7f1261ad1640 1 --2- 172.21.15.67:0/1583381450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f125c0b49b0 0x7f125c0c55c0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:13.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.720+0000 7f1261ad1640 1 --2- 172.21.15.67:0/1583381450 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f125c099d00 0x7f125c150200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:13.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.720+0000 7f1261ad1640 1 -- 172.21.15.67:0/1583381450 >> 172.21.15.67:0/1583381450 conn(0x7f125c0b3650 msgr2=0x7f125c1613c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:13.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.720+0000 7f1261ad1640 1 -- 172.21.15.67:0/1583381450 shutdown_connections 2024-03-24T00:40:13.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.720+0000 7f1261ad1640 1 -- 172.21.15.67:0/1583381450 wait complete. 2024-03-24T00:40:13.731 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:40:13.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.836+0000 7fbc65c75640 1 Processor -- start 2024-03-24T00:40:13.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.836+0000 7fbc65c75640 1 -- start start 2024-03-24T00:40:13.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.837+0000 7fbc65c75640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbc6006f370 0x7fbc600b3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:13.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.837+0000 7fbc65c75640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc600b2ab0 con 0x7fbc600b4a00 2024-03-24T00:40:13.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.837+0000 7fbc65c75640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc600b2d90 con 0x7fbc60144670 2024-03-24T00:40:13.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.837+0000 7fbc65c75640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc600b3070 con 0x7fbc6006f370 2024-03-24T00:40:13.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.837+0000 7fbc5ffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbc6006f370 0x7fbc600b3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:13.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.837+0000 7fbc5effd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbc600b4a00 0x7fbc6006ec60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55920/0 (socket says 172.21.15.67:55920) 2024-03-24T00:40:13.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.837+0000 7fbc5effd640 1 -- 172.21.15.67:0/1621494017 learned_addr learned my addr 172.21.15.67:0/1621494017 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:40:13.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.838+0000 7fbc5ffff640 1 -- 172.21.15.67:0/1621494017 >> v1:172.21.15.72:6789/0 conn(0x7fbc60144670 legacy=0x7fbc60072500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:13.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.838+0000 7fbc5ffff640 1 -- 172.21.15.67:0/1621494017 >> v1:172.21.15.67:6789/0 conn(0x7fbc600b4a00 legacy=0x7fbc6006ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:13.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.838+0000 7fbc5ffff640 1 -- 172.21.15.67:0/1621494017 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc600b3350 con 0x7fbc6006f370 2024-03-24T00:40:13.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.838+0000 7fbc5ffff640 1 --2- 172.21.15.67:0/1621494017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc6006f370 0x7fbc600b3870 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fbc5800ba40 tx=0x7fbc5802e130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=487fd5415f157a66 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:13.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.838+0000 7fbc5e7fc640 1 -- 172.21.15.67:0/1621494017 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc5803b070 con 0x7fbc6006f370 2024-03-24T00:40:13.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.838+0000 7fbc5e7fc640 1 -- 172.21.15.67:0/1621494017 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc58036440 con 0x7fbc6006f370 2024-03-24T00:40:13.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.838+0000 7fbc5e7fc640 1 -- 172.21.15.67:0/1621494017 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc58036810 con 0x7fbc6006f370 2024-03-24T00:40:13.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.839+0000 7fbc65c75640 1 -- 172.21.15.67:0/1621494017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc6006f370 msgr2=0x7fbc600b3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:13.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.839+0000 7fbc65c75640 1 --2- 172.21.15.67:0/1621494017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc6006f370 0x7fbc600b3870 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fbc5800ba40 tx=0x7fbc5802e130 comp rx=0 tx=0).stop 2024-03-24T00:40:13.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.839+0000 7fbc65c75640 1 -- 172.21.15.67:0/1621494017 shutdown_connections 2024-03-24T00:40:13.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.839+0000 7fbc65c75640 1 --2- 172.21.15.67:0/1621494017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc6006f370 0x7fbc600b3870 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:13.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.839+0000 7fbc65c75640 1 -- 172.21.15.67:0/1621494017 >> 172.21.15.67:0/1621494017 conn(0x7fbc600cf9e0 msgr2=0x7fbc600cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:13.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.839+0000 7fbc65c75640 1 -- 172.21.15.67:0/1621494017 shutdown_connections 2024-03-24T00:40:13.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.839+0000 7fbc65c75640 1 -- 172.21.15.67:0/1621494017 wait complete. 2024-03-24T00:40:13.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.840+0000 7fbc65c75640 1 Processor -- start 2024-03-24T00:40:13.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.840+0000 7fbc65c75640 1 -- start start 2024-03-24T00:40:13.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.841+0000 7fbc65c75640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbc6006f370 0x7fbc6013f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:13.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.841+0000 7fbc65c75640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc600b4a00 0x7fbc600aa8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:13.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.841+0000 7fbc65c75640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc60144670 0x7fbc600aae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:13.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.841+0000 7fbc65c75640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc600cdfb0 con 0x7fbc600b4a00 2024-03-24T00:40:13.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.841+0000 7fbc65c75640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc600cde30 con 0x7fbc60144670 2024-03-24T00:40:13.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.841+0000 7fbc65c75640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc600ce130 con 0x7fbc6006f370 2024-03-24T00:40:13.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.841+0000 7fbc5effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc600b4a00 0x7fbc600aa8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:13.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.841+0000 7fbc5effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc600b4a00 0x7fbc600aa8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53154/0 (socket says 172.21.15.67:53154) 2024-03-24T00:40:13.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.841+0000 7fbc5effd640 1 -- 172.21.15.67:0/3196802920 learned_addr learned my addr 172.21.15.67:0/3196802920 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:13.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.841+0000 7fbc5f7fe640 1 --2- 172.21.15.67:0/3196802920 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbc6006f370 0x7fbc6013f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:13.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.841+0000 7fbc5ffff640 1 --2- 172.21.15.67:0/3196802920 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc60144670 0x7fbc600aae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:13.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.841+0000 7fbc5effd640 1 -- 172.21.15.67:0/3196802920 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbc6006f370 msgr2=0x7fbc6013f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:13.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.841+0000 7fbc5effd640 1 --2- 172.21.15.67:0/3196802920 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbc6006f370 0x7fbc6013f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:13.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.841+0000 7fbc5effd640 1 -- 172.21.15.67:0/3196802920 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc60144670 msgr2=0x7fbc600aae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:13.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.841+0000 7fbc5effd640 1 --2- 172.21.15.67:0/3196802920 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc60144670 0x7fbc600aae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:13.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.841+0000 7fbc5effd640 1 -- 172.21.15.67:0/3196802920 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc6013fed0 con 0x7fbc600b4a00 2024-03-24T00:40:13.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.842+0000 7fbc5effd640 1 --2- 172.21.15.67:0/3196802920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc600b4a00 0x7fbc600aa8d0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fbc5400c4a0 tx=0x7fbc5400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:13.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.842+0000 7fbc47fff640 1 -- 172.21.15.67:0/3196802920 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc54017070 con 0x7fbc600b4a00 2024-03-24T00:40:13.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.842+0000 7fbc47fff640 1 -- 172.21.15.67:0/3196802920 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc54005ce0 con 0x7fbc600b4a00 2024-03-24T00:40:13.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.842+0000 7fbc65c75640 1 -- 172.21.15.67:0/3196802920 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc6013fbf0 con 0x7fbc600b4a00 2024-03-24T00:40:13.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.842+0000 7fbc47fff640 1 -- 172.21.15.67:0/3196802920 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc54007270 con 0x7fbc600b4a00 2024-03-24T00:40:13.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.842+0000 7fbc65c75640 1 -- 172.21.15.67:0/3196802920 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc600a9220 con 0x7fbc600b4a00 2024-03-24T00:40:13.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.843+0000 7fbc65c75640 1 -- 172.21.15.67:0/3196802920 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc6004f400 con 0x7fbc600b4a00 2024-03-24T00:40:13.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.843+0000 7fbc47fff640 1 -- 172.21.15.67:0/3196802920 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fbc5401a070 con 0x7fbc600b4a00 2024-03-24T00:40:13.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.843+0000 7fbc47fff640 1 --2- 172.21.15.67:0/3196802920 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbc4c06bbd0 0x7fbc4c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:13.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.844+0000 7fbc47fff640 1 -- 172.21.15.67:0/3196802920 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fbc54089df0 con 0x7fbc600b4a00 2024-03-24T00:40:13.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.844+0000 7fbc5f7fe640 1 --2- 172.21.15.67:0/3196802920 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbc4c06bbd0 0x7fbc4c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:13.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.844+0000 7fbc5f7fe640 1 --2- 172.21.15.67:0/3196802920 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbc4c06bbd0 0x7fbc4c06e090 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fbc601f2ce0 tx=0x7fbc50009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:13.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:13.848+0000 7fbc47fff640 1 -- 172.21.15.67:0/3196802920 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fbc54058100 con 0x7fbc600b4a00 2024-03-24T00:40:14.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.028+0000 7fbc65c75640 1 -- 172.21.15.67:0/3196802920 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbc600008d0 con 0x7fbc600b4a00 2024-03-24T00:40:14.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.029+0000 7fbc47fff640 1 -- 172.21.15.67:0/3196802920 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v39) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbc5405c1a0 con 0x7fbc600b4a00 2024-03-24T00:40:14.028 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-24T00:40:14.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.031+0000 7fbc65c75640 1 -- 172.21.15.67:0/3196802920 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbc4c06bbd0 msgr2=0x7fbc4c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:14.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.031+0000 7fbc65c75640 1 --2- 172.21.15.67:0/3196802920 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbc4c06bbd0 0x7fbc4c06e090 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fbc601f2ce0 tx=0x7fbc50009040 comp rx=0 tx=0).stop 2024-03-24T00:40:14.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.031+0000 7fbc65c75640 1 -- 172.21.15.67:0/3196802920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc600b4a00 msgr2=0x7fbc600aa8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:14.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.031+0000 7fbc65c75640 1 --2- 172.21.15.67:0/3196802920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc600b4a00 0x7fbc600aa8d0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fbc5400c4a0 tx=0x7fbc5400c960 comp rx=0 tx=0).stop 2024-03-24T00:40:14.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.031+0000 7fbc65c75640 1 -- 172.21.15.67:0/3196802920 shutdown_connections 2024-03-24T00:40:14.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.031+0000 7fbc65c75640 1 --2- 172.21.15.67:0/3196802920 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbc4c06bbd0 0x7fbc4c06e090 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:14.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.031+0000 7fbc65c75640 1 --2- 172.21.15.67:0/3196802920 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc60144670 0x7fbc600aae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:14.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.031+0000 7fbc65c75640 1 --2- 172.21.15.67:0/3196802920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc600b4a00 0x7fbc600aa8d0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:14.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.031+0000 7fbc65c75640 1 --2- 172.21.15.67:0/3196802920 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbc6006f370 0x7fbc6013f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:14.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.031+0000 7fbc65c75640 1 -- 172.21.15.67:0/3196802920 >> 172.21.15.67:0/3196802920 conn(0x7fbc600cf9e0 msgr2=0x7fbc6015d790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:14.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.031+0000 7fbc65c75640 1 -- 172.21.15.67:0/3196802920 shutdown_connections 2024-03-24T00:40:14.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.031+0000 7fbc65c75640 1 -- 172.21.15.67:0/3196802920 wait complete. 2024-03-24T00:40:14.041 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:40:14.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.146+0000 7efed14b2640 1 Processor -- start 2024-03-24T00:40:14.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.146+0000 7efed14b2640 1 -- start start 2024-03-24T00:40:14.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.147+0000 7efed14b2640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7efecc0b0f10 0x7efecc0d1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:14.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.147+0000 7efed14b2640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efecc0d20d0 0x7efecc0b1f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:14.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.147+0000 7efed14b2640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efecc0b12f0 con 0x7efecc0b2440 2024-03-24T00:40:14.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.147+0000 7efed14b2640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efecc0d2510 con 0x7efecc0b0f10 2024-03-24T00:40:14.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.147+0000 7efed14b2640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efecc0b2c50 con 0x7efecc0d20d0 2024-03-24T00:40:14.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.147+0000 7efecaffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7efecc0b0f10 0x7efecc0d1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:14.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.147+0000 7efec27fc640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efecc0d20d0 0x7efecc0b1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:14.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.147+0000 7efecb7fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7efecc0b2440 0x7efecc0cf9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:53882/0 (socket says 172.21.15.67:53882) 2024-03-24T00:40:14.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.147+0000 7efecb7fe640 1 -- 172.21.15.67:0/1212263085 learned_addr learned my addr 172.21.15.67:0/1212263085 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:40:14.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.147+0000 7efecaffd640 1 --2- 172.21.15.67:0/1212263085 >> v2:172.21.15.72:3300/0 conn(0x7efecc0b0f10 0x7efecc0d1b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:53408/0 (socket says 172.21.15.67:53408) 2024-03-24T00:40:14.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.148+0000 7efecaffd640 1 -- 172.21.15.67:0/1212263085 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efecc0d20d0 msgr2=0x7efecc0b1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:14.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.148+0000 7efecaffd640 1 --2- 172.21.15.67:0/1212263085 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efecc0d20d0 0x7efecc0b1f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:14.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.148+0000 7efecaffd640 1 -- 172.21.15.67:0/1212263085 >> v1:172.21.15.72:6789/0 conn(0x7efecc0b2440 legacy=0x7efecc0cf9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:14.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.148+0000 7efecaffd640 1 -- 172.21.15.67:0/1212263085 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efecc0ce880 con 0x7efecc0b0f10 2024-03-24T00:40:14.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.148+0000 7efec27fc640 1 --2- 172.21.15.67:0/1212263085 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efecc0d20d0 0x7efecc0b1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:40:14.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.148+0000 7efecaffd640 1 --2- 172.21.15.67:0/1212263085 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efecc0b0f10 0x7efecc0d1b90 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7efebc0097a0 tx=0x7efebc02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9b158906d995d87 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:14.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.148+0000 7efeca7fc640 1 -- 172.21.15.67:0/1212263085 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efebc035020 con 0x7efecc0b0f10 2024-03-24T00:40:14.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.149+0000 7efeca7fc640 1 -- 172.21.15.67:0/1212263085 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efebc031830 con 0x7efecc0b0f10 2024-03-24T00:40:14.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.149+0000 7efeca7fc640 1 -- 172.21.15.67:0/1212263085 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efebc031c00 con 0x7efecc0b0f10 2024-03-24T00:40:14.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.149+0000 7efed14b2640 1 -- 172.21.15.67:0/1212263085 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efecc0b0f10 msgr2=0x7efecc0d1b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:14.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.149+0000 7efed14b2640 1 --2- 172.21.15.67:0/1212263085 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efecc0b0f10 0x7efecc0d1b90 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7efebc0097a0 tx=0x7efebc02b8d0 comp rx=0 tx=0).stop 2024-03-24T00:40:14.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.149+0000 7efed14b2640 1 -- 172.21.15.67:0/1212263085 shutdown_connections 2024-03-24T00:40:14.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.149+0000 7efed14b2640 1 --2- 172.21.15.67:0/1212263085 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efecc0d20d0 0x7efecc0b1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:14.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.149+0000 7efed14b2640 1 --2- 172.21.15.67:0/1212263085 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efecc0b0f10 0x7efecc0d1b90 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:14.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.149+0000 7efed14b2640 1 -- 172.21.15.67:0/1212263085 >> 172.21.15.67:0/1212263085 conn(0x7efecc08b1b0 msgr2=0x7efecc08b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:14.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.149+0000 7efed14b2640 1 -- 172.21.15.67:0/1212263085 shutdown_connections 2024-03-24T00:40:14.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.149+0000 7efed14b2640 1 -- 172.21.15.67:0/1212263085 wait complete. 2024-03-24T00:40:14.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.150+0000 7efed14b2640 1 Processor -- start 2024-03-24T00:40:14.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.150+0000 7efed14b2640 1 -- start start 2024-03-24T00:40:14.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.151+0000 7efed14b2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efecc0b0f10 0x7efecc0c5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:14.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.151+0000 7efed14b2640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efecc0b2440 0x7efecc154150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:14.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.151+0000 7efed14b2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efecc0d20d0 0x7efecc1f4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:14.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.151+0000 7efed14b2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efecc0ce560 con 0x7efecc0d20d0 2024-03-24T00:40:14.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.151+0000 7efed14b2640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efecc0cee40 con 0x7efecc0b0f10 2024-03-24T00:40:14.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.151+0000 7efed14b2640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efecc0b28b0 con 0x7efecc0b2440 2024-03-24T00:40:14.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.151+0000 7efecb7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efecc0d20d0 0x7efecc1f4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:14.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.151+0000 7efecaffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efecc0b0f10 0x7efecc0c5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:14.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.151+0000 7efec27fc640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efecc0b2440 0x7efecc154150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:14.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.151+0000 7efecaffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efecc0b0f10 0x7efecc0c5730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:53422/0 (socket says 172.21.15.67:53422) 2024-03-24T00:40:14.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.151+0000 7efecaffd640 1 -- 172.21.15.67:0/2991311623 learned_addr learned my addr 172.21.15.67:0/2991311623 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:14.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.152+0000 7efec27fc640 1 -- 172.21.15.67:0/2991311623 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efecc0b0f10 msgr2=0x7efecc0c5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:14.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.152+0000 7efec27fc640 1 --2- 172.21.15.67:0/2991311623 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efecc0b0f10 0x7efecc0c5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:14.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.152+0000 7efec27fc640 1 -- 172.21.15.67:0/2991311623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efecc0d20d0 msgr2=0x7efecc1f4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:14.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.152+0000 7efec27fc640 1 --2- 172.21.15.67:0/2991311623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efecc0d20d0 0x7efecc1f4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:14.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.152+0000 7efec27fc640 1 -- 172.21.15.67:0/2991311623 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efecc0b5ec0 con 0x7efecc0b2440 2024-03-24T00:40:14.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.152+0000 7efecb7fe640 1 --2- 172.21.15.67:0/2991311623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efecc0d20d0 0x7efecc1f4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:40:14.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.152+0000 7efecaffd640 1 --2- 172.21.15.67:0/2991311623 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efecc0b0f10 0x7efecc0c5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:40:14.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.152+0000 7efec27fc640 1 --2- 172.21.15.67:0/2991311623 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efecc0b2440 0x7efecc154150 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7efeb400e870 tx=0x7efeb400ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:14.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.152+0000 7efec3fff640 1 -- 172.21.15.67:0/2991311623 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb4017070 con 0x7efecc0b2440 2024-03-24T00:40:14.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.152+0000 7efec3fff640 1 -- 172.21.15.67:0/2991311623 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeb400c940 con 0x7efecc0b2440 2024-03-24T00:40:14.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.152+0000 7efed14b2640 1 -- 172.21.15.67:0/2991311623 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efecc0b61a0 con 0x7efecc0b2440 2024-03-24T00:40:14.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.152+0000 7efec3fff640 1 -- 172.21.15.67:0/2991311623 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb400cd10 con 0x7efecc0b2440 2024-03-24T00:40:14.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.152+0000 7efed14b2640 1 -- 172.21.15.67:0/2991311623 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efecc0b6760 con 0x7efecc0b2440 2024-03-24T00:40:14.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.153+0000 7efed14b2640 1 -- 172.21.15.67:0/2991311623 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efecc04f400 con 0x7efecc0b2440 2024-03-24T00:40:14.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.154+0000 7efec3fff640 1 -- 172.21.15.67:0/2991311623 <== mon.2 v2:172.21.15.72:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7efeb401a070 con 0x7efecc0b2440 2024-03-24T00:40:14.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.154+0000 7efec3fff640 1 --2- 172.21.15.67:0/2991311623 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7efeac06bbd0 0x7efeac06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:14.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.154+0000 7efec3fff640 1 -- 172.21.15.67:0/2991311623 <== mon.2 v2:172.21.15.72:3301/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7efeb408ad00 con 0x7efecc0b2440 2024-03-24T00:40:14.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.154+0000 7efecaffd640 1 --2- 172.21.15.67:0/2991311623 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7efeac06bbd0 0x7efeac06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:14.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.155+0000 7efecaffd640 1 --2- 172.21.15.67:0/2991311623 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7efeac06bbd0 0x7efeac06e090 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7efebc02b910 tx=0x7efebc008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:14.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.163+0000 7efec3fff640 1 -- 172.21.15.67:0/2991311623 <== mon.2 v2:172.21.15.72:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7efeb4059010 con 0x7efecc0b2440 2024-03-24T00:40:14.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.346+0000 7efed14b2640 1 -- 172.21.15.67:0/2991311623 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7efecc0b1810 con 0x7efecc0b2440 2024-03-24T00:40:14.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.346+0000 7efec3fff640 1 -- 172.21.15.67:0/2991311623 <== mon.2 v2:172.21.15.72:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v39) v1 ==== 100+0+11 (secure 0 0 0) 0x7efeb405d0b0 con 0x7efecc0b2440 2024-03-24T00:40:14.346 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-03-24T00:40:14.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.349+0000 7efed14b2640 1 -- 172.21.15.67:0/2991311623 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7efeac06bbd0 msgr2=0x7efeac06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:14.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.349+0000 7efed14b2640 1 --2- 172.21.15.67:0/2991311623 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7efeac06bbd0 0x7efeac06e090 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7efebc02b910 tx=0x7efebc008030 comp rx=0 tx=0).stop 2024-03-24T00:40:14.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.349+0000 7efed14b2640 1 -- 172.21.15.67:0/2991311623 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efecc0b2440 msgr2=0x7efecc154150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:14.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.349+0000 7efed14b2640 1 --2- 172.21.15.67:0/2991311623 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efecc0b2440 0x7efecc154150 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7efeb400e870 tx=0x7efeb400ed30 comp rx=0 tx=0).stop 2024-03-24T00:40:14.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.349+0000 7efed14b2640 1 -- 172.21.15.67:0/2991311623 shutdown_connections 2024-03-24T00:40:14.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.349+0000 7efed14b2640 1 --2- 172.21.15.67:0/2991311623 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7efeac06bbd0 0x7efeac06e090 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:14.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.349+0000 7efed14b2640 1 --2- 172.21.15.67:0/2991311623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efecc0d20d0 0x7efecc1f4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:14.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.349+0000 7efed14b2640 1 --2- 172.21.15.67:0/2991311623 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efecc0b2440 0x7efecc154150 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:14.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.349+0000 7efed14b2640 1 --2- 172.21.15.67:0/2991311623 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efecc0b0f10 0x7efecc0c5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:14.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.349+0000 7efed14b2640 1 -- 172.21.15.67:0/2991311623 >> 172.21.15.67:0/2991311623 conn(0x7efecc08b1b0 msgr2=0x7efecc0d0e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:14.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.349+0000 7efed14b2640 1 -- 172.21.15.67:0/2991311623 shutdown_connections 2024-03-24T00:40:14.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.349+0000 7efed14b2640 1 -- 172.21.15.67:0/2991311623 wait complete. 2024-03-24T00:40:14.359 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T00:40:14.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.462+0000 7f7c9bc85640 1 Processor -- start 2024-03-24T00:40:14.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.462+0000 7f7c9bc85640 1 -- start start 2024-03-24T00:40:14.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.463+0000 7f7c9bc85640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7c9415e2b0 0x7f7c940ceb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:14.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.463+0000 7f7c9bc85640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7c94161bc0 0x7f7c940cf060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:14.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.463+0000 7f7c9bc85640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c940d2190 con 0x7f7c9415ded0 2024-03-24T00:40:14.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.463+0000 7f7c9bc85640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c941620f0 con 0x7f7c94161bc0 2024-03-24T00:40:14.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.463+0000 7f7c9bc85640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c941623d0 con 0x7f7c9415e2b0 2024-03-24T00:40:14.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.463+0000 7f7c9a222640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7c94161bc0 0x7f7c940cf060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:14.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.463+0000 7f7c9a222640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7c94161bc0 0x7f7c940cf060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53182/0 (socket says 172.21.15.67:53182) 2024-03-24T00:40:14.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.463+0000 7f7c9a222640 1 -- 172.21.15.67:0/3940790080 learned_addr learned my addr 172.21.15.67:0/3940790080 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:14.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.463+0000 7f7c99a21640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7c9415ded0 0x7f7c940b96e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55936/0 (socket says 172.21.15.67:55936) 2024-03-24T00:40:14.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.463+0000 7f7c99220640 1 --2- 172.21.15.67:0/3940790080 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7c9415e2b0 0x7f7c940ceb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:14.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.463+0000 7f7c9a222640 1 -- 172.21.15.67:0/3940790080 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7c9415e2b0 msgr2=0x7f7c940ceb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:14.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.463+0000 7f7c9a222640 1 --2- 172.21.15.67:0/3940790080 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7c9415e2b0 0x7f7c940ceb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:14.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.463+0000 7f7c9a222640 1 -- 172.21.15.67:0/3940790080 >> v1:172.21.15.67:6789/0 conn(0x7f7c9415ded0 legacy=0x7f7c940b96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:14.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.463+0000 7f7c9a222640 1 -- 172.21.15.67:0/3940790080 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c941626b0 con 0x7f7c94161bc0 2024-03-24T00:40:14.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.463+0000 7f7c98a1f640 1 -- 172.21.15.67:0/3940790080 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1309296530 0 0) 0x7f7c940d2190 con 0x7f7c9415ded0 2024-03-24T00:40:14.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.464+0000 7f7c9a222640 1 --2- 172.21.15.67:0/3940790080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c94161bc0 0x7f7c940cf060 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f7c84009e50 tx=0x7f7c8402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dfceef759678385 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:14.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.464+0000 7f7c98a1f640 1 -- 172.21.15.67:0/3940790080 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c8402cb00 con 0x7f7c94161bc0 2024-03-24T00:40:14.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.464+0000 7f7c98a1f640 1 -- 172.21.15.67:0/3940790080 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c8402cca0 con 0x7f7c94161bc0 2024-03-24T00:40:14.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.464+0000 7f7c98a1f640 1 -- 172.21.15.67:0/3940790080 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c84034610 con 0x7f7c94161bc0 2024-03-24T00:40:14.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.465+0000 7f7c9bc85640 1 -- 172.21.15.67:0/3940790080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c94161bc0 msgr2=0x7f7c940cf060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:14.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.465+0000 7f7c9bc85640 1 --2- 172.21.15.67:0/3940790080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c94161bc0 0x7f7c940cf060 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f7c84009e50 tx=0x7f7c8402bf60 comp rx=0 tx=0).stop 2024-03-24T00:40:14.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.465+0000 7f7c9bc85640 1 -- 172.21.15.67:0/3940790080 shutdown_connections 2024-03-24T00:40:14.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.465+0000 7f7c9bc85640 1 --2- 172.21.15.67:0/3940790080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c94161bc0 0x7f7c940cf060 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:14.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.465+0000 7f7c9bc85640 1 --2- 172.21.15.67:0/3940790080 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7c9415e2b0 0x7f7c940ceb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:14.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.465+0000 7f7c9bc85640 1 -- 172.21.15.67:0/3940790080 >> 172.21.15.67:0/3940790080 conn(0x7f7c9406c860 msgr2=0x7f7c9406cc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:14.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.465+0000 7f7c9bc85640 1 -- 172.21.15.67:0/3940790080 shutdown_connections 2024-03-24T00:40:14.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.465+0000 7f7c9bc85640 1 -- 172.21.15.67:0/3940790080 wait complete. 2024-03-24T00:40:14.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.466+0000 7f7c9bc85640 1 Processor -- start 2024-03-24T00:40:14.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.466+0000 7f7c9bc85640 1 -- start start 2024-03-24T00:40:14.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.467+0000 7f7c9bc85640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7c9415ded0 0x7f7c940b40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:14.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.467+0000 7f7c9bc85640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7c9415e2b0 0x7f7c940ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:14.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.467+0000 7f7c9bc85640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c94161bc0 0x7f7c94155390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:14.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.467+0000 7f7c9bc85640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c940d27f0 con 0x7f7c94161bc0 2024-03-24T00:40:14.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.467+0000 7f7c9bc85640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c940d2670 con 0x7f7c9415e2b0 2024-03-24T00:40:14.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.467+0000 7f7c9bc85640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c940d2970 con 0x7f7c9415ded0 2024-03-24T00:40:14.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.467+0000 7f7c9a222640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c94161bc0 0x7f7c94155390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:14.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.467+0000 7f7c9a222640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c94161bc0 0x7f7c94155390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53188/0 (socket says 172.21.15.67:53188) 2024-03-24T00:40:14.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.467+0000 7f7c9a222640 1 -- 172.21.15.67:0/3475928933 learned_addr learned my addr 172.21.15.67:0/3475928933 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:14.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.467+0000 7f7c99220640 1 --2- 172.21.15.67:0/3475928933 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7c9415e2b0 0x7f7c940ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:14.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.467+0000 7f7c99a21640 1 --2- 172.21.15.67:0/3475928933 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7c9415ded0 0x7f7c940b40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:14.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.467+0000 7f7c9a222640 1 -- 172.21.15.67:0/3475928933 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7c9415ded0 msgr2=0x7f7c940b40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:14.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.467+0000 7f7c9a222640 1 --2- 172.21.15.67:0/3475928933 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7c9415ded0 0x7f7c940b40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:14.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.467+0000 7f7c9a222640 1 -- 172.21.15.67:0/3475928933 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7c9415e2b0 msgr2=0x7f7c940ccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:14.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.467+0000 7f7c9a222640 1 --2- 172.21.15.67:0/3475928933 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7c9415e2b0 0x7f7c940ccab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:14.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.467+0000 7f7c9a222640 1 -- 172.21.15.67:0/3475928933 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c940b47c0 con 0x7f7c94161bc0 2024-03-24T00:40:14.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.467+0000 7f7c9a222640 1 --2- 172.21.15.67:0/3475928933 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c94161bc0 0x7f7c94155390 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f7c84009e50 tx=0x7f7c84004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:14.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.468+0000 7f7c927fc640 1 -- 172.21.15.67:0/3475928933 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c84043020 con 0x7f7c94161bc0 2024-03-24T00:40:14.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.468+0000 7f7c927fc640 1 -- 172.21.15.67:0/3475928933 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c8402ce50 con 0x7f7c94161bc0 2024-03-24T00:40:14.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.468+0000 7f7c9bc85640 1 -- 172.21.15.67:0/3475928933 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c940b4aa0 con 0x7f7c94161bc0 2024-03-24T00:40:14.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.468+0000 7f7c927fc640 1 -- 172.21.15.67:0/3475928933 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c84009510 con 0x7f7c94161bc0 2024-03-24T00:40:14.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.468+0000 7f7c9bc85640 1 -- 172.21.15.67:0/3475928933 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c940b5060 con 0x7f7c94161bc0 2024-03-24T00:40:14.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.469+0000 7f7c9bc85640 1 -- 172.21.15.67:0/3475928933 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c9404f400 con 0x7f7c94161bc0 2024-03-24T00:40:14.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.469+0000 7f7c927fc640 1 -- 172.21.15.67:0/3475928933 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7c84046020 con 0x7f7c94161bc0 2024-03-24T00:40:14.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.469+0000 7f7c927fc640 1 --2- 172.21.15.67:0/3475928933 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7c8006bbd0 0x7f7c8006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:14.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.470+0000 7f7c927fc640 1 -- 172.21.15.67:0/3475928933 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7c840ad010 con 0x7f7c94161bc0 2024-03-24T00:40:14.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.470+0000 7f7c99a21640 1 --2- 172.21.15.67:0/3475928933 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7c8006bbd0 0x7f7c8006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:14.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.470+0000 7f7c99a21640 1 --2- 172.21.15.67:0/3475928933 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7c8006bbd0 0x7f7c8006e090 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c000f60 tx=0x7f7c8c007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:14.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.474+0000 7f7c927fc640 1 -- 172.21.15.67:0/3475928933 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7c8404d030 con 0x7f7c94161bc0 2024-03-24T00:40:14.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.670+0000 7f7c9bc85640 1 -- 172.21.15.67:0/3475928933 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f7c940ca6b0 con 0x7f7c94161bc0 2024-03-24T00:40:14.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.671+0000 7f7c927fc640 1 -- 172.21.15.67:0/3475928933 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v39) v1 ==== 80+0+0 (secure 0 0 0) 0x7f7c8407b3b0 con 0x7f7c94161bc0 2024-03-24T00:40:14.670 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-03-24T00:40:14.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.672+0000 7f7c9bc85640 1 -- 172.21.15.67:0/3475928933 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7c8006bbd0 msgr2=0x7f7c8006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:14.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.673+0000 7f7c9bc85640 1 --2- 172.21.15.67:0/3475928933 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7c8006bbd0 0x7f7c8006e090 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c000f60 tx=0x7f7c8c007d50 comp rx=0 tx=0).stop 2024-03-24T00:40:14.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.673+0000 7f7c9bc85640 1 -- 172.21.15.67:0/3475928933 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c94161bc0 msgr2=0x7f7c94155390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:14.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.673+0000 7f7c9bc85640 1 --2- 172.21.15.67:0/3475928933 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c94161bc0 0x7f7c94155390 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f7c84009e50 tx=0x7f7c84004a40 comp rx=0 tx=0).stop 2024-03-24T00:40:14.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.673+0000 7f7c9bc85640 1 -- 172.21.15.67:0/3475928933 shutdown_connections 2024-03-24T00:40:14.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.673+0000 7f7c9bc85640 1 --2- 172.21.15.67:0/3475928933 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7c8006bbd0 0x7f7c8006e090 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:14.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.673+0000 7f7c9bc85640 1 --2- 172.21.15.67:0/3475928933 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c94161bc0 0x7f7c94155390 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:14.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.673+0000 7f7c9bc85640 1 --2- 172.21.15.67:0/3475928933 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7c9415e2b0 0x7f7c940ccab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:14.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.673+0000 7f7c9bc85640 1 --2- 172.21.15.67:0/3475928933 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7c9415ded0 0x7f7c940b40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:14.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.673+0000 7f7c9bc85640 1 -- 172.21.15.67:0/3475928933 >> 172.21.15.67:0/3475928933 conn(0x7f7c9406c860 msgr2=0x7f7c940b7890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:14.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.673+0000 7f7c9bc85640 1 -- 172.21.15.67:0/3475928933 shutdown_connections 2024-03-24T00:40:14.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.673+0000 7f7c9bc85640 1 -- 172.21.15.67:0/3475928933 wait complete. 2024-03-24T00:40:14.683 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-24T00:40:14.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:40:14.683 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-24T00:40:14.735 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-03-24T00:40:14.735 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:40:14.735 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-24T00:40:14.755 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:40:14.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.897+0000 7f186d2dc640 1 Processor -- start 2024-03-24T00:40:14.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.897+0000 7f186d2dc640 1 -- start start 2024-03-24T00:40:14.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.898+0000 7f186d2dc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f186815e2a0 0x7f18680ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:14.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.898+0000 7f186d2dc640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18680cf640 con 0x7f186815dec0 2024-03-24T00:40:14.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.898+0000 7f186d2dc640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18680cf080 con 0x7f18680d0ef0 2024-03-24T00:40:14.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.898+0000 7f186d2dc640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18680cee30 con 0x7f186815e2a0 2024-03-24T00:40:14.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.898+0000 7f18667fc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f186815e2a0 0x7f18680ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:14.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.898+0000 7f1866ffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f186815dec0 0x7f18680d07e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55942/0 (socket says 172.21.15.67:55942) 2024-03-24T00:40:14.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.898+0000 7f1866ffd640 1 -- 172.21.15.67:0/2395207523 learned_addr learned my addr 172.21.15.67:0/2395207523 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:40:14.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.898+0000 7f18667fc640 1 -- 172.21.15.67:0/2395207523 >> v1:172.21.15.72:6789/0 conn(0x7f18680d0ef0 legacy=0x7f18680d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:14.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.898+0000 7f18667fc640 1 -- 172.21.15.67:0/2395207523 >> v1:172.21.15.67:6789/0 conn(0x7f186815dec0 legacy=0x7f18680d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:14.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.899+0000 7f18667fc640 1 -- 172.21.15.67:0/2395207523 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18680cf360 con 0x7f186815e2a0 2024-03-24T00:40:14.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.899+0000 7f1865ffb640 1 -- 172.21.15.67:0/2395207523 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2050649033 0 0) 0x7f1860001110 con 0x7f186815dec0 2024-03-24T00:40:14.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.899+0000 7f18667fc640 1 --2- 172.21.15.67:0/2395207523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f186815e2a0 0x7f18680ce630 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f18680cefb0 tx=0x7f185002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1925141a3e01ba9b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:14.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.899+0000 7f1865ffb640 1 -- 172.21.15.67:0/2395207523 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1850035020 con 0x7f186815e2a0 2024-03-24T00:40:14.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.899+0000 7f1865ffb640 1 -- 172.21.15.67:0/2395207523 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1850031830 con 0x7f186815e2a0 2024-03-24T00:40:14.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.899+0000 7f1865ffb640 1 -- 172.21.15.67:0/2395207523 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1850031c00 con 0x7f186815e2a0 2024-03-24T00:40:14.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.900+0000 7f186d2dc640 1 -- 172.21.15.67:0/2395207523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f186815e2a0 msgr2=0x7f18680ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:14.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.900+0000 7f186d2dc640 1 --2- 172.21.15.67:0/2395207523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f186815e2a0 0x7f18680ce630 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f18680cefb0 tx=0x7f185002b8d0 comp rx=0 tx=0).stop 2024-03-24T00:40:14.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.900+0000 7f186d2dc640 1 -- 172.21.15.67:0/2395207523 shutdown_connections 2024-03-24T00:40:14.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.900+0000 7f186d2dc640 1 --2- 172.21.15.67:0/2395207523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f186815e2a0 0x7f18680ce630 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:14.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.900+0000 7f186d2dc640 1 -- 172.21.15.67:0/2395207523 >> 172.21.15.67:0/2395207523 conn(0x7f186806c810 msgr2=0x7f186806cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:14.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.900+0000 7f186d2dc640 1 -- 172.21.15.67:0/2395207523 shutdown_connections 2024-03-24T00:40:14.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.900+0000 7f186d2dc640 1 -- 172.21.15.67:0/2395207523 wait complete. 2024-03-24T00:40:14.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.901+0000 7f186d2dc640 1 Processor -- start 2024-03-24T00:40:14.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.901+0000 7f186d2dc640 1 -- start start 2024-03-24T00:40:14.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.902+0000 7f186d2dc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18680d0ef0 0x7f18680b3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:14.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.902+0000 7f186d2dc640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f186815dec0 0x7f186815b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:14.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.902+0000 7f186d2dc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f186815e2a0 0x7f186815bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:14.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.902+0000 7f186d2dc640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1868162a40 con 0x7f186815e2a0 2024-03-24T00:40:14.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.902+0000 7f186d2dc640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18681628c0 con 0x7f18680d0ef0 2024-03-24T00:40:14.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.902+0000 7f186d2dc640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18680d2120 con 0x7f186815dec0 2024-03-24T00:40:14.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.902+0000 7f18677fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f186815e2a0 0x7f186815bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:14.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.902+0000 7f18677fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f186815e2a0 0x7f186815bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53212/0 (socket says 172.21.15.67:53212) 2024-03-24T00:40:14.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.902+0000 7f18677fe640 1 -- 172.21.15.67:0/3927154717 learned_addr learned my addr 172.21.15.67:0/3927154717 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:14.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.902+0000 7f1866ffd640 1 --2- 172.21.15.67:0/3927154717 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18680d0ef0 0x7f18680b3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:14.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.902+0000 7f18667fc640 1 --2- 172.21.15.67:0/3927154717 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f186815dec0 0x7f186815b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:14.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.902+0000 7f18677fe640 1 -- 172.21.15.67:0/3927154717 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f186815dec0 msgr2=0x7f186815b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:14.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.902+0000 7f18677fe640 1 --2- 172.21.15.67:0/3927154717 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f186815dec0 0x7f186815b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:14.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.902+0000 7f18677fe640 1 -- 172.21.15.67:0/3927154717 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18680d0ef0 msgr2=0x7f18680b3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:14.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.902+0000 7f18677fe640 1 --2- 172.21.15.67:0/3927154717 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18680d0ef0 0x7f18680b3a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:14.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.902+0000 7f18677fe640 1 -- 172.21.15.67:0/3927154717 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18681448a0 con 0x7f186815e2a0 2024-03-24T00:40:14.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.903+0000 7f18677fe640 1 --2- 172.21.15.67:0/3927154717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f186815e2a0 0x7f186815bce0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f185800c7a0 tx=0x7f185800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:14.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.903+0000 7f185f7fe640 1 -- 172.21.15.67:0/3927154717 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1858003b70 con 0x7f186815e2a0 2024-03-24T00:40:14.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.903+0000 7f185f7fe640 1 -- 172.21.15.67:0/3927154717 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1858003d30 con 0x7f186815e2a0 2024-03-24T00:40:14.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.903+0000 7f186d2dc640 1 -- 172.21.15.67:0/3927154717 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f186815c4d0 con 0x7f186815e2a0 2024-03-24T00:40:14.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.903+0000 7f185f7fe640 1 -- 172.21.15.67:0/3927154717 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f185800aa90 con 0x7f186815e2a0 2024-03-24T00:40:14.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.903+0000 7f186d2dc640 1 -- 172.21.15.67:0/3927154717 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f186815ca90 con 0x7f186815e2a0 2024-03-24T00:40:14.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.904+0000 7f186d2dc640 1 -- 172.21.15.67:0/3927154717 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18680d0120 con 0x7f186815e2a0 2024-03-24T00:40:14.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.904+0000 7f185f7fe640 1 -- 172.21.15.67:0/3927154717 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f185800ac30 con 0x7f186815e2a0 2024-03-24T00:40:14.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.905+0000 7f185f7fe640 1 --2- 172.21.15.67:0/3927154717 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f184806bbd0 0x7f184806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:14.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.905+0000 7f185f7fe640 1 -- 172.21.15.67:0/3927154717 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f185808a9d0 con 0x7f186815e2a0 2024-03-24T00:40:14.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.905+0000 7f1866ffd640 1 --2- 172.21.15.67:0/3927154717 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f184806bbd0 0x7f184806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:14.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.905+0000 7f1866ffd640 1 --2- 172.21.15.67:0/3927154717 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f184806bbd0 0x7f184806e090 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1860000f60 tx=0x7f1860007b50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:14.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:14.909+0000 7f185f7fe640 1 -- 172.21.15.67:0/3927154717 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f18680d0120 con 0x7f186815e2a0 2024-03-24T00:40:15.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.132+0000 7f186d2dc640 1 -- 172.21.15.67:0/3927154717 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18680ca770 con 0x7f186815e2a0 2024-03-24T00:40:15.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.132+0000 7f185f7fe640 1 -- 172.21.15.67:0/3927154717 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f18680ca770 con 0x7f186815e2a0 2024-03-24T00:40:15.132 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:40:15.132 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5039,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/166859817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":166859817},{"type":"v1","addr":"172.21.15.67:6834","nonce":166859817}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5042,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2653663845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2653663845},{"type":"v1","addr":"172.21.15.67:6835","nonce":2653663845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5048,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2355909634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2355909634},{"type":"v1","addr":"172.21.15.72:6837","nonce":2355909634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:40:00.691429+0000","modified":"2024-03-24T00:40:03.146091+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5418},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5418":{"gid":5418,"name":"d","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6836/2551774148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2551774148},{"type":"v1","addr":"172.21.15.72:6836","nonce":2551774148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5418,"qdb_cluster":[5418]},"id":2}]} 2024-03-24T00:40:15.133 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 17 2024-03-24T00:40:15.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.135+0000 7f186d2dc640 1 -- 172.21.15.67:0/3927154717 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f184806bbd0 msgr2=0x7f184806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:15.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.135+0000 7f186d2dc640 1 --2- 172.21.15.67:0/3927154717 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f184806bbd0 0x7f184806e090 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1860000f60 tx=0x7f1860007b50 comp rx=0 tx=0).stop 2024-03-24T00:40:15.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.135+0000 7f186d2dc640 1 -- 172.21.15.67:0/3927154717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f186815e2a0 msgr2=0x7f186815bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:15.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.135+0000 7f186d2dc640 1 --2- 172.21.15.67:0/3927154717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f186815e2a0 0x7f186815bce0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f185800c7a0 tx=0x7f185800cc60 comp rx=0 tx=0).stop 2024-03-24T00:40:15.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.135+0000 7f186d2dc640 1 -- 172.21.15.67:0/3927154717 shutdown_connections 2024-03-24T00:40:15.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.135+0000 7f186d2dc640 1 --2- 172.21.15.67:0/3927154717 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f184806bbd0 0x7f184806e090 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:15.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.135+0000 7f186d2dc640 1 --2- 172.21.15.67:0/3927154717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f186815e2a0 0x7f186815bce0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:15.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.135+0000 7f186d2dc640 1 --2- 172.21.15.67:0/3927154717 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f186815dec0 0x7f186815b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:15.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.135+0000 7f186d2dc640 1 --2- 172.21.15.67:0/3927154717 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18680d0ef0 0x7f18680b3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:15.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.135+0000 7f186d2dc640 1 -- 172.21.15.67:0/3927154717 >> 172.21.15.67:0/3927154717 conn(0x7f186806c810 msgr2=0x7f186815e760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:15.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.135+0000 7f186d2dc640 1 -- 172.21.15.67:0/3927154717 shutdown_connections 2024-03-24T00:40:15.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.135+0000 7f186d2dc640 1 -- 172.21.15.67:0/3927154717 wait complete. 2024-03-24T00:40:15.145 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json flush journal 2024-03-24T00:40:15.272 INFO:teuthology.orchestra.run.smithi072.stdout:{"message":"","return_code":0} 2024-03-24T00:40:15.273 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-24T00:40:15.273 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:40:15.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.375+0000 7f813957e640 1 Processor -- start 2024-03-24T00:40:15.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.375+0000 7f813957e640 1 -- start start 2024-03-24T00:40:15.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.376+0000 7f813957e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f813415e310 0x7f8134162610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:15.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.376+0000 7f813957e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f81340d0d50 0x7f81340d2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:15.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.376+0000 7f813957e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81340ce920 con 0x7f813415df30 2024-03-24T00:40:15.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.376+0000 7f813957e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8134162b50 con 0x7f813415e310 2024-03-24T00:40:15.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.376+0000 7f813957e640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81340cef20 con 0x7f81340d0d50 2024-03-24T00:40:15.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.376+0000 7f81337fe640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f81340d0d50 0x7f81340d2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:15.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.376+0000 7f812bfff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f813415e310 0x7f8134162610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:15.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.376+0000 7f812bfff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f813415e310 0x7f8134162610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:53440/0 (socket says 172.21.15.67:53440) 2024-03-24T00:40:15.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.376+0000 7f8132ffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f813415df30 0x7f81340d0640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:53900/0 (socket says 172.21.15.67:53900) 2024-03-24T00:40:15.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.376+0000 7f812bfff640 1 -- 172.21.15.67:0/3678758448 learned_addr learned my addr 172.21.15.67:0/3678758448 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:15.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.376+0000 7f81337fe640 1 --2- 172.21.15.67:0/3678758448 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f81340d0d50 0x7f81340d2490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.67:40824/0 (socket says 172.21.15.67:40824) 2024-03-24T00:40:15.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.376+0000 7f81337fe640 1 -- 172.21.15.67:0/3678758448 >> v2:172.21.15.72:3300/0 conn(0x7f813415e310 msgr2=0x7f8134162610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:15.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.376+0000 7f81337fe640 1 --2- 172.21.15.67:0/3678758448 >> v2:172.21.15.72:3300/0 conn(0x7f813415e310 0x7f8134162610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:15.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.376+0000 7f81337fe640 1 -- 172.21.15.67:0/3678758448 >> v1:172.21.15.72:6789/0 conn(0x7f813415df30 legacy=0x7f81340d0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:15.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.376+0000 7f81337fe640 1 -- 172.21.15.67:0/3678758448 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81340cf200 con 0x7f81340d0d50 2024-03-24T00:40:15.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.376+0000 7f812bfff640 1 --2- 172.21.15.67:0/3678758448 >> v2:172.21.15.72:3300/0 conn(0x7f813415e310 0x7f8134162610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:40:15.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.377+0000 7f81337fe640 1 --2- 172.21.15.67:0/3678758448 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f81340d0d50 0x7f81340d2490 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f812c009d90 tx=0x7f812c02be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c47d7c97030aa2b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:15.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.377+0000 7f81327fc640 1 -- 172.21.15.67:0/3678758448 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812c039070 con 0x7f81340d0d50 2024-03-24T00:40:15.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.377+0000 7f81327fc640 1 -- 172.21.15.67:0/3678758448 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f812c02cbc0 con 0x7f81340d0d50 2024-03-24T00:40:15.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.377+0000 7f81327fc640 1 -- 172.21.15.67:0/3678758448 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812c034440 con 0x7f81340d0d50 2024-03-24T00:40:15.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.378+0000 7f813957e640 1 -- 172.21.15.67:0/3678758448 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f81340d0d50 msgr2=0x7f81340d2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:15.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.378+0000 7f813957e640 1 --2- 172.21.15.67:0/3678758448 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f81340d0d50 0x7f81340d2490 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f812c009d90 tx=0x7f812c02be10 comp rx=0 tx=0).stop 2024-03-24T00:40:15.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.378+0000 7f813957e640 1 -- 172.21.15.67:0/3678758448 shutdown_connections 2024-03-24T00:40:15.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.378+0000 7f813957e640 1 --2- 172.21.15.67:0/3678758448 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f81340d0d50 0x7f81340d2490 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:15.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.378+0000 7f813957e640 1 --2- 172.21.15.67:0/3678758448 >> v2:172.21.15.72:3300/0 conn(0x7f813415e310 0x7f8134162610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:15.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.378+0000 7f813957e640 1 -- 172.21.15.67:0/3678758448 >> 172.21.15.67:0/3678758448 conn(0x7f813406c880 msgr2=0x7f813406cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:15.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.378+0000 7f813957e640 1 -- 172.21.15.67:0/3678758448 shutdown_connections 2024-03-24T00:40:15.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.378+0000 7f813957e640 1 -- 172.21.15.67:0/3678758448 wait complete. 2024-03-24T00:40:15.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.379+0000 7f813957e640 1 Processor -- start 2024-03-24T00:40:15.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.379+0000 7f813957e640 1 -- start start 2024-03-24T00:40:15.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.380+0000 7f813957e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f81340d0d50 0x7f81340cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:15.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.380+0000 7f813957e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f813415df30 0x7f81341e8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:15.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.380+0000 7f813957e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81341e8f70 0x7f8134144320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:15.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.380+0000 7f813957e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81340ce580 con 0x7f81341e8f70 2024-03-24T00:40:15.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.380+0000 7f813957e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81340b32f0 con 0x7f81340d0d50 2024-03-24T00:40:15.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.380+0000 7f813957e640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81340ce700 con 0x7f813415df30 2024-03-24T00:40:15.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.380+0000 7f81337fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81341e8f70 0x7f8134144320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:15.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.380+0000 7f812bfff640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f813415df30 0x7f81341e8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:15.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.380+0000 7f81337fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81341e8f70 0x7f8134144320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53226/0 (socket says 172.21.15.67:53226) 2024-03-24T00:40:15.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.380+0000 7f81337fe640 1 -- 172.21.15.67:0/668890747 learned_addr learned my addr 172.21.15.67:0/668890747 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:15.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.380+0000 7f8132ffd640 1 --2- 172.21.15.67:0/668890747 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f81340d0d50 0x7f81340cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:15.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.380+0000 7f81337fe640 1 -- 172.21.15.67:0/668890747 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f813415df30 msgr2=0x7f81341e8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:15.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.380+0000 7f81337fe640 1 --2- 172.21.15.67:0/668890747 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f813415df30 0x7f81341e8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:15.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.380+0000 7f81337fe640 1 -- 172.21.15.67:0/668890747 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f81340d0d50 msgr2=0x7f81340cb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:15.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.380+0000 7f81337fe640 1 --2- 172.21.15.67:0/668890747 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f81340d0d50 0x7f81340cb1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:15.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.380+0000 7f81337fe640 1 -- 172.21.15.67:0/668890747 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81341e9480 con 0x7f81341e8f70 2024-03-24T00:40:15.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.380+0000 7f812bfff640 1 --2- 172.21.15.67:0/668890747 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f813415df30 0x7f81341e8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:40:15.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.380+0000 7f81337fe640 1 --2- 172.21.15.67:0/668890747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81341e8f70 0x7f8134144320 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f812c009d90 tx=0x7f812c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:15.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.380+0000 7f812b7fe640 1 -- 172.21.15.67:0/668890747 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812c039050 con 0x7f81341e8f70 2024-03-24T00:40:15.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.381+0000 7f812b7fe640 1 -- 172.21.15.67:0/668890747 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f812c045070 con 0x7f81341e8f70 2024-03-24T00:40:15.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.381+0000 7f812b7fe640 1 -- 172.21.15.67:0/668890747 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812c009450 con 0x7f81341e8f70 2024-03-24T00:40:15.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.381+0000 7f813957e640 1 -- 172.21.15.67:0/668890747 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81341e9760 con 0x7f81341e8f70 2024-03-24T00:40:15.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.381+0000 7f813957e640 1 -- 172.21.15.67:0/668890747 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81341e9d20 con 0x7f81341e8f70 2024-03-24T00:40:15.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.382+0000 7f813957e640 1 -- 172.21.15.67:0/668890747 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f813404f400 con 0x7f81341e8f70 2024-03-24T00:40:15.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.382+0000 7f812b7fe640 1 -- 172.21.15.67:0/668890747 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f812c03e850 con 0x7f81341e8f70 2024-03-24T00:40:15.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.382+0000 7f812b7fe640 1 --2- 172.21.15.67:0/668890747 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f811406bbd0 0x7f811406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:15.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.382+0000 7f812b7fe640 1 -- 172.21.15.67:0/668890747 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f812c0ad980 con 0x7f81341e8f70 2024-03-24T00:40:15.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.383+0000 7f8132ffd640 1 --2- 172.21.15.67:0/668890747 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f811406bbd0 0x7f811406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:15.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.383+0000 7f8132ffd640 1 --2- 172.21.15.67:0/668890747 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f811406bbd0 0x7f811406e090 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8124007fd0 tx=0x7f8124007e80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:15.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.387+0000 7f812b7fe640 1 -- 172.21.15.67:0/668890747 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f812c07bdb0 con 0x7f81341e8f70 2024-03-24T00:40:15.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.606+0000 7f813957e640 1 -- 172.21.15.67:0/668890747 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81340c7630 con 0x7f81341e8f70 2024-03-24T00:40:15.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.607+0000 7f812b7fe640 1 -- 172.21.15.67:0/668890747 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f812c07fe50 con 0x7f81341e8f70 2024-03-24T00:40:15.607 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:40:15.607 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5039,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/166859817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":166859817},{"type":"v1","addr":"172.21.15.67:6834","nonce":166859817}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5042,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2653663845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2653663845},{"type":"v1","addr":"172.21.15.67:6835","nonce":2653663845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5048,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2355909634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2355909634},{"type":"v1","addr":"172.21.15.72:6837","nonce":2355909634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:40:00.691429+0000","modified":"2024-03-24T00:40:03.146091+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5418},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5418":{"gid":5418,"name":"d","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6836/2551774148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2551774148},{"type":"v1","addr":"172.21.15.72:6836","nonce":2551774148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5418,"qdb_cluster":[5418]},"id":2}]} 2024-03-24T00:40:15.607 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 17 2024-03-24T00:40:15.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.609+0000 7f813957e640 1 -- 172.21.15.67:0/668890747 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f811406bbd0 msgr2=0x7f811406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:15.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.609+0000 7f813957e640 1 --2- 172.21.15.67:0/668890747 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f811406bbd0 0x7f811406e090 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8124007fd0 tx=0x7f8124007e80 comp rx=0 tx=0).stop 2024-03-24T00:40:15.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.609+0000 7f813957e640 1 -- 172.21.15.67:0/668890747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81341e8f70 msgr2=0x7f8134144320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:15.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.609+0000 7f813957e640 1 --2- 172.21.15.67:0/668890747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81341e8f70 0x7f8134144320 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f812c009d90 tx=0x7f812c004a40 comp rx=0 tx=0).stop 2024-03-24T00:40:15.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.609+0000 7f813957e640 1 -- 172.21.15.67:0/668890747 shutdown_connections 2024-03-24T00:40:15.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.609+0000 7f813957e640 1 --2- 172.21.15.67:0/668890747 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f811406bbd0 0x7f811406e090 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:15.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.609+0000 7f813957e640 1 --2- 172.21.15.67:0/668890747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81341e8f70 0x7f8134144320 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:15.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.609+0000 7f813957e640 1 --2- 172.21.15.67:0/668890747 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f813415df30 0x7f81341e8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:15.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.609+0000 7f813957e640 1 --2- 172.21.15.67:0/668890747 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f81340d0d50 0x7f81340cb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:15.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.609+0000 7f813957e640 1 -- 172.21.15.67:0/668890747 >> 172.21.15.67:0/668890747 conn(0x7f813406c880 msgr2=0x7f81340c7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:15.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.609+0000 7f813957e640 1 -- 172.21.15.67:0/668890747 shutdown_connections 2024-03-24T00:40:15.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:15.609+0000 7f813957e640 1 -- 172.21.15.67:0/668890747 wait complete. 2024-03-24T00:40:15.620 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json tag path /parentdir mytag 2024-03-24T00:40:15.743 INFO:teuthology.orchestra.run.smithi072.stdout:{"return_code":0,"scrub_tag":"mytag","mode":"asynchronous"} 2024-03-24T00:40:15.744 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mode": "asynchronous", "return_code": 0, "scrub_tag": "mytag" } 2024-03-24T00:40:16.009 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:40:16.012 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:40:25.745 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getxattr 10000000000.00000000 scrub_tag 2024-03-24T00:40:25.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.793+0000 7fe27da3bf40 1 Processor -- start 2024-03-24T00:40:25.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.793+0000 7fe27da3bf40 1 -- start start 2024-03-24T00:40:25.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.794+0000 7fe27da3bf40 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55dc03f6f1c0 0x55dc03f6c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:25.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.794+0000 7fe27da3bf40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dc03f87d90 0x55dc03f81910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:25.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.794+0000 7fe27da3bf40 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc04014150 0x55dc03f81e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:25.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.794+0000 7fe27da3bf40 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc03ee60d0 con 0x55dc04014150 2024-03-24T00:40:25.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.794+0000 7fe27da3bf40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc03eea030 con 0x55dc03f87d90 2024-03-24T00:40:25.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.794+0000 7fe27da3bf40 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55dc04014780 con 0x55dc03f6f1c0 2024-03-24T00:40:25.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.794+0000 7fe27b504640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc04014150 0x55dc03f81e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:25.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.794+0000 7fe27b504640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc04014150 0x55dc03f81e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39112/0 (socket says 172.21.15.67:39112) 2024-03-24T00:40:25.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.794+0000 7fe273fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dc03f87d90 0x55dc03f81910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:25.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.794+0000 7fe27b504640 1 -- 172.21.15.67:0/232908462 learned_addr learned my addr 172.21.15.67:0/232908462 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:25.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.794+0000 7fe27ad03640 1 --2- 172.21.15.67:0/232908462 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55dc03f6f1c0 0x55dc03f6c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:25.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.794+0000 7fe27b504640 1 -- 172.21.15.67:0/232908462 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55dc03f6f1c0 msgr2=0x55dc03f6c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:25.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.794+0000 7fe27b504640 1 --2- 172.21.15.67:0/232908462 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55dc03f6f1c0 0x55dc03f6c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.794+0000 7fe27b504640 1 -- 172.21.15.67:0/232908462 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dc03f87d90 msgr2=0x55dc03f81910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:25.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.794+0000 7fe27b504640 1 --2- 172.21.15.67:0/232908462 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dc03f87d90 0x55dc03f81910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.794+0000 7fe27b504640 1 -- 172.21.15.67:0/232908462 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dc04003da0 con 0x55dc04014150 2024-03-24T00:40:25.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.794+0000 7fe273fff640 1 --2- 172.21.15.67:0/232908462 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dc03f87d90 0x55dc03f81910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:40:25.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.794+0000 7fe27b504640 1 --2- 172.21.15.67:0/232908462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc04014150 0x55dc03f81e50 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fe274001680 tx=0x7fe274001b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e504185c4ba9dba4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:25.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.795+0000 7fe279d01640 1 -- 172.21.15.67:0/232908462 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe274004520 con 0x55dc04014150 2024-03-24T00:40:25.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.795+0000 7fe279d01640 1 -- 172.21.15.67:0/232908462 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2740046c0 con 0x55dc04014150 2024-03-24T00:40:25.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.795+0000 7fe279d01640 1 -- 172.21.15.67:0/232908462 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe274004b30 con 0x55dc04014150 2024-03-24T00:40:25.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.795+0000 7fe27da3bf40 1 -- 172.21.15.67:0/232908462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc04014150 msgr2=0x55dc03f81e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:25.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.795+0000 7fe27da3bf40 1 --2- 172.21.15.67:0/232908462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc04014150 0x55dc03f81e50 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fe274001680 tx=0x7fe274001b40 comp rx=0 tx=0).stop 2024-03-24T00:40:25.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.795+0000 7fe27da3bf40 1 -- 172.21.15.67:0/232908462 shutdown_connections 2024-03-24T00:40:25.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.795+0000 7fe27da3bf40 1 --2- 172.21.15.67:0/232908462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc04014150 0x55dc03f81e50 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.795+0000 7fe27da3bf40 1 --2- 172.21.15.67:0/232908462 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dc03f87d90 0x55dc03f81910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.795+0000 7fe27da3bf40 1 --2- 172.21.15.67:0/232908462 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55dc03f6f1c0 0x55dc03f6c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.795+0000 7fe27da3bf40 1 -- 172.21.15.67:0/232908462 >> 172.21.15.67:0/232908462 conn(0x55dc03f3de70 msgr2=0x55dc03f422b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:25.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.795+0000 7fe27da3bf40 1 -- 172.21.15.67:0/232908462 shutdown_connections 2024-03-24T00:40:25.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.795+0000 7fe27da3bf40 1 -- 172.21.15.67:0/232908462 wait complete. 2024-03-24T00:40:25.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.796+0000 7fe27da3bf40 1 Processor -- start 2024-03-24T00:40:25.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.796+0000 7fe27da3bf40 1 -- start start 2024-03-24T00:40:25.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.797+0000 7fe27da3bf40 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55dc03f6f1c0 0x55dc04003370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:25.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.797+0000 7fe27da3bf40 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc03f87d90 0x55dc03f40340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:25.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.797+0000 7fe27da3bf40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dc04014150 0x55dc0400f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:25.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.797+0000 7fe27da3bf40 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc03f813e0 con 0x55dc03f87d90 2024-03-24T00:40:25.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.797+0000 7fe27da3bf40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc03f808f0 con 0x55dc04014150 2024-03-24T00:40:25.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.797+0000 7fe27da3bf40 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55dc03f80180 con 0x55dc03f6f1c0 2024-03-24T00:40:25.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.797+0000 7fe273fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc03f87d90 0x55dc03f40340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:25.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.797+0000 7fe27ad03640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55dc03f6f1c0 0x55dc04003370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:25.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.797+0000 7fe273fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc03f87d90 0x55dc03f40340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39120/0 (socket says 172.21.15.67:39120) 2024-03-24T00:40:25.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.797+0000 7fe273fff640 1 -- 172.21.15.67:0/4113303999 learned_addr learned my addr 172.21.15.67:0/4113303999 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:25.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.797+0000 7fe27b504640 1 --2- 172.21.15.67:0/4113303999 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dc04014150 0x55dc0400f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:25.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.797+0000 7fe273fff640 1 -- 172.21.15.67:0/4113303999 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55dc03f6f1c0 msgr2=0x55dc04003370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:25.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.797+0000 7fe273fff640 1 --2- 172.21.15.67:0/4113303999 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55dc03f6f1c0 0x55dc04003370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.797+0000 7fe273fff640 1 -- 172.21.15.67:0/4113303999 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dc04014150 msgr2=0x55dc0400f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:25.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.797+0000 7fe273fff640 1 --2- 172.21.15.67:0/4113303999 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dc04014150 0x55dc0400f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.797+0000 7fe273fff640 1 -- 172.21.15.67:0/4113303999 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dc03f85550 con 0x55dc03f87d90 2024-03-24T00:40:25.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.797+0000 7fe273fff640 1 --2- 172.21.15.67:0/4113303999 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc03f87d90 0x55dc03f40340 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fe264002d30 tx=0x7fe26400c620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:25.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.798+0000 7fe278cff640 1 -- 172.21.15.67:0/4113303999 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe264015020 con 0x55dc03f87d90 2024-03-24T00:40:25.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.798+0000 7fe278cff640 1 -- 172.21.15.67:0/4113303999 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe264011650 con 0x55dc03f87d90 2024-03-24T00:40:25.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.798+0000 7fe278cff640 1 -- 172.21.15.67:0/4113303999 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe264011a40 con 0x55dc03f87d90 2024-03-24T00:40:25.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.798+0000 7fe27da3bf40 1 -- 172.21.15.67:0/4113303999 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55dc03f85830 con 0x55dc03f87d90 2024-03-24T00:40:25.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.798+0000 7fe27da3bf40 1 -- 172.21.15.67:0/4113303999 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dc03f83c60 con 0x55dc03f87d90 2024-03-24T00:40:25.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.799+0000 7fe278cff640 1 -- 172.21.15.67:0/4113303999 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe264018020 con 0x55dc03f87d90 2024-03-24T00:40:25.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.800+0000 7fe278cff640 1 --2- 172.21.15.67:0/4113303999 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe25c06a5b0 0x7fe25c06ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:25.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.800+0000 7fe278cff640 1 -- 172.21.15.67:0/4113303999 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe264089820 con 0x55dc03f87d90 2024-03-24T00:40:25.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.800+0000 7fe27ad03640 1 --2- 172.21.15.67:0/4113303999 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe25c06a5b0 0x7fe25c06ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:25.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.800+0000 7fe27da3bf40 1 --2- 172.21.15.67:0/4113303999 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x55dc0400fb40 0x55dc040af360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:25.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.800+0000 7fe27da3bf40 1 -- 172.21.15.67:0/4113303999 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:1 5.10 5:0cf6ad0f:::10000000000.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39) v9 -- 0x55dc040af8a0 con 0x55dc0400fb40 2024-03-24T00:40:25.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.800+0000 7fe27b504640 1 --2- 172.21.15.67:0/4113303999 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x55dc0400fb40 0x55dc040af360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:25.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.800+0000 7fe27ad03640 1 --2- 172.21.15.67:0/4113303999 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe25c06a5b0 0x7fe25c06ca70 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe26c054000 tx=0x7fe26c04e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:25.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.801+0000 7fe27b504640 1 --2- 172.21.15.67:0/4113303999 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x55dc0400fb40 0x55dc040af360 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:25.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.801+0000 7fe27b504640 1 -- 172.21.15.67:0/4113303999 <== osd.7 v2:172.21.15.72:6824/2893038746 1 ==== osd_op_reply(1 10000000000.00000000 [getxattr (9) out=9b] v0'0 uv11 ondisk = 0) v8 ==== 164+0+9 (crc 0 0 0) 0x7fe27401b040 con 0x55dc0400fb40 2024-03-24T00:40:25.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.801+0000 7fe27da3bf40 1 -- 172.21.15.67:0/4113303999 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x55dc0400fb40 msgr2=0x55dc040af360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:25.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.801+0000 7fe27da3bf40 1 --2- 172.21.15.67:0/4113303999 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x55dc0400fb40 0x55dc040af360 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.801+0000 7fe27da3bf40 1 -- 172.21.15.67:0/4113303999 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe25c06a5b0 msgr2=0x7fe25c06ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:25.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.801+0000 7fe27da3bf40 1 --2- 172.21.15.67:0/4113303999 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe25c06a5b0 0x7fe25c06ca70 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe26c054000 tx=0x7fe26c04e000 comp rx=0 tx=0).stop 2024-03-24T00:40:25.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.801+0000 7fe27da3bf40 1 -- 172.21.15.67:0/4113303999 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc03f87d90 msgr2=0x55dc03f40340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:25.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.801+0000 7fe27da3bf40 1 --2- 172.21.15.67:0/4113303999 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc03f87d90 0x55dc03f40340 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fe264002d30 tx=0x7fe26400c620 comp rx=0 tx=0).stop 2024-03-24T00:40:25.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.801+0000 7fe27b504640 1 -- 172.21.15.67:0/4113303999 reap_dead start 2024-03-24T00:40:25.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.801+0000 7fe27da3bf40 1 -- 172.21.15.67:0/4113303999 shutdown_connections 2024-03-24T00:40:25.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.801+0000 7fe27da3bf40 1 -- 172.21.15.67:0/4113303999 >> 172.21.15.67:0/4113303999 conn(0x55dc03f3de70 msgr2=0x55dc03f6c5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:25.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.801+0000 7fe27da3bf40 1 -- 172.21.15.67:0/4113303999 shutdown_connections 2024-03-24T00:40:25.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.801+0000 7fe27da3bf40 1 -- 172.21.15.67:0/4113303999 wait complete. 2024-03-24T00:40:25.806 INFO:teuthology.orchestra.run.smithi067.stdout:mytag 2024-03-24T00:40:25.806 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getxattr 10000000001.00000000 scrub_tag 2024-03-24T00:40:25.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.850+0000 7f9a2f63bf40 1 Processor -- start 2024-03-24T00:40:25.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.850+0000 7f9a2f63bf40 1 -- start start 2024-03-24T00:40:25.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.851+0000 7f9a2f63bf40 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56531ab151c0 0x56531ab2b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:25.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.851+0000 7f9a2f63bf40 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56531ab2dd90 0x56531abab4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:25.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.851+0000 7f9a2f63bf40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56531ab2afa0 0x56531abab9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:25.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.851+0000 7f9a2f63bf40 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x56531abba780 con 0x56531ab2dd90 2024-03-24T00:40:25.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.851+0000 7f9a2f63bf40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x56531aa8c0d0 con 0x56531ab2afa0 2024-03-24T00:40:25.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.851+0000 7f9a2f63bf40 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x56531ab2e510 con 0x56531ab151c0 2024-03-24T00:40:25.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.851+0000 7f9a2c102640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56531ab2dd90 0x56531abab4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:25.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.851+0000 7f9a2c102640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56531ab2dd90 0x56531abab4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39150/0 (socket says 172.21.15.67:39150) 2024-03-24T00:40:25.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.851+0000 7f9a2c102640 1 -- 172.21.15.67:0/295153693 learned_addr learned my addr 172.21.15.67:0/295153693 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:25.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.851+0000 7f9a2d104640 1 --2- 172.21.15.67:0/295153693 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56531ab2afa0 0x56531abab9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:25.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.851+0000 7f9a2c903640 1 --2- 172.21.15.67:0/295153693 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56531ab151c0 0x56531ab2b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:25.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.851+0000 7f9a2c102640 1 -- 172.21.15.67:0/295153693 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56531ab151c0 msgr2=0x56531ab2b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:25.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.851+0000 7f9a2c102640 1 --2- 172.21.15.67:0/295153693 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56531ab151c0 0x56531ab2b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.851+0000 7f9a2c102640 1 -- 172.21.15.67:0/295153693 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56531ab2afa0 msgr2=0x56531abab9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:25.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.851+0000 7f9a2c102640 1 --2- 172.21.15.67:0/295153693 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56531ab2afa0 0x56531abab9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.851+0000 7f9a2c102640 1 -- 172.21.15.67:0/295153693 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56531ab2a660 con 0x56531ab2dd90 2024-03-24T00:40:25.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.851+0000 7f9a2c102640 1 --2- 172.21.15.67:0/295153693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56531ab2dd90 0x56531abab4b0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f9a20001d30 tx=0x7f9a2000b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e26f72167b9c3868 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:25.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.852+0000 7f9a2b100640 1 -- 172.21.15.67:0/295153693 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a2000bd20 con 0x56531ab2dd90 2024-03-24T00:40:25.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.852+0000 7f9a2b100640 1 -- 172.21.15.67:0/295153693 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a2000f400 con 0x56531ab2dd90 2024-03-24T00:40:25.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.852+0000 7f9a2b100640 1 -- 172.21.15.67:0/295153693 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a2000f830 con 0x56531ab2dd90 2024-03-24T00:40:25.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.852+0000 7f9a2f63bf40 1 -- 172.21.15.67:0/295153693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56531ab2dd90 msgr2=0x56531abab4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:25.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.852+0000 7f9a2f63bf40 1 --2- 172.21.15.67:0/295153693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56531ab2dd90 0x56531abab4b0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f9a20001d30 tx=0x7f9a2000b5b0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.852+0000 7f9a2f63bf40 1 -- 172.21.15.67:0/295153693 shutdown_connections 2024-03-24T00:40:25.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.852+0000 7f9a2f63bf40 1 --2- 172.21.15.67:0/295153693 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56531ab2afa0 0x56531abab9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.852+0000 7f9a2f63bf40 1 --2- 172.21.15.67:0/295153693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56531ab2dd90 0x56531abab4b0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.852+0000 7f9a2f63bf40 1 --2- 172.21.15.67:0/295153693 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56531ab151c0 0x56531ab2b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.852+0000 7f9a2f63bf40 1 -- 172.21.15.67:0/295153693 >> 172.21.15.67:0/295153693 conn(0x56531aae3e70 msgr2=0x56531aae2ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:25.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.852+0000 7f9a2f63bf40 1 -- 172.21.15.67:0/295153693 shutdown_connections 2024-03-24T00:40:25.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.852+0000 7f9a2f63bf40 1 -- 172.21.15.67:0/295153693 wait complete. 2024-03-24T00:40:25.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.853+0000 7f9a2f63bf40 1 Processor -- start 2024-03-24T00:40:25.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.853+0000 7f9a2f63bf40 1 -- start start 2024-03-24T00:40:25.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.854+0000 7f9a2f63bf40 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56531ab151c0 0x56531abb4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:25.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.854+0000 7f9a2f63bf40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56531ab2afa0 0x56531ac55590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:25.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.854+0000 7f9a2f63bf40 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56531ab2dd90 0x56531abaa260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:25.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.854+0000 7f9a2f63bf40 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x56531ab2a070 con 0x56531ab151c0 2024-03-24T00:40:25.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.854+0000 7f9a2f63bf40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x56531ab2a3c0 con 0x56531ab2afa0 2024-03-24T00:40:25.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.854+0000 7f9a2f63bf40 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x56531ab292b0 con 0x56531ab2dd90 2024-03-24T00:40:25.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.854+0000 7f9a2c903640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56531ab151c0 0x56531abb4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:25.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.854+0000 7f9a2c903640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56531ab151c0 0x56531abb4630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39164/0 (socket says 172.21.15.67:39164) 2024-03-24T00:40:25.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.854+0000 7f9a2c903640 1 -- 172.21.15.67:0/3785341494 learned_addr learned my addr 172.21.15.67:0/3785341494 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:25.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.854+0000 7f9a2d104640 1 --2- 172.21.15.67:0/3785341494 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56531ab2dd90 0x56531abaa260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:25.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.854+0000 7f9a2c102640 1 --2- 172.21.15.67:0/3785341494 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56531ab2afa0 0x56531ac55590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:25.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.854+0000 7f9a2c903640 1 -- 172.21.15.67:0/3785341494 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56531ab2dd90 msgr2=0x56531abaa260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:25.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.854+0000 7f9a2c903640 1 --2- 172.21.15.67:0/3785341494 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56531ab2dd90 0x56531abaa260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.854+0000 7f9a2c903640 1 -- 172.21.15.67:0/3785341494 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56531ab2afa0 msgr2=0x56531ac55590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:25.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.854+0000 7f9a2c903640 1 --2- 172.21.15.67:0/3785341494 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56531ab2afa0 0x56531ac55590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.854+0000 7f9a2c903640 1 -- 172.21.15.67:0/3785341494 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56531ab2c1f0 con 0x56531ab151c0 2024-03-24T00:40:25.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.854+0000 7f9a2c903640 1 --2- 172.21.15.67:0/3785341494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56531ab151c0 0x56531abb4630 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f9a2401cf90 tx=0x7f9a24002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:25.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.854+0000 7f9a2a0fe640 1 -- 172.21.15.67:0/3785341494 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a24065070 con 0x56531ab151c0 2024-03-24T00:40:25.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.854+0000 7f9a2a0fe640 1 -- 172.21.15.67:0/3785341494 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a24052bf0 con 0x56531ab151c0 2024-03-24T00:40:25.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.854+0000 7f9a2f63bf40 1 -- 172.21.15.67:0/3785341494 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x56531aae2f80 con 0x56531ab151c0 2024-03-24T00:40:25.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.854+0000 7f9a2a0fe640 1 -- 172.21.15.67:0/3785341494 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a24057040 con 0x56531ab151c0 2024-03-24T00:40:25.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.855+0000 7f9a2f63bf40 1 -- 172.21.15.67:0/3785341494 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56531ab2bb30 con 0x56531ab151c0 2024-03-24T00:40:25.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.856+0000 7f9a2a0fe640 1 -- 172.21.15.67:0/3785341494 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9a24068070 con 0x56531ab151c0 2024-03-24T00:40:25.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.856+0000 7f9a2a0fe640 1 --2- 172.21.15.67:0/3785341494 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9a1806a5e0 0x7f9a1806caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:25.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.856+0000 7f9a2a0fe640 1 -- 172.21.15.67:0/3785341494 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9a240ca890 con 0x56531ab151c0 2024-03-24T00:40:25.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.856+0000 7f9a2c102640 1 --2- 172.21.15.67:0/3785341494 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9a1806a5e0 0x7f9a1806caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:25.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.857+0000 7f9a2f63bf40 1 --2- 172.21.15.67:0/3785341494 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x56531abacd80 0x56531ab26f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:25.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.857+0000 7f9a2f63bf40 1 -- 172.21.15.67:0/3785341494 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:1 5.7 5:e2e07930:::10000000001.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39) v9 -- 0x56531ab27440 con 0x56531abacd80 2024-03-24T00:40:25.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.857+0000 7f9a2d104640 1 --2- 172.21.15.67:0/3785341494 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x56531abacd80 0x56531ab26f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:25.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.857+0000 7f9a2c102640 1 --2- 172.21.15.67:0/3785341494 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9a1806a5e0 0x7f9a1806caa0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9a20001d30 tx=0x7f9a20009000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:25.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.857+0000 7f9a2d104640 1 --2- 172.21.15.67:0/3785341494 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x56531abacd80 0x56531ab26f00 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:25.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.857+0000 7f9a2d104640 1 -- 172.21.15.67:0/3785341494 <== osd.5 v2:172.21.15.72:6800/288486206 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (9) out=9b] v0'0 uv5 ondisk = 0) v8 ==== 164+0+9 (crc 0 0 0) 0x7f9a1c002b60 con 0x56531abacd80 2024-03-24T00:40:25.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.857+0000 7f9a2f63bf40 1 -- 172.21.15.67:0/3785341494 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x56531abacd80 msgr2=0x56531ab26f00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:25.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.857+0000 7f9a2f63bf40 1 --2- 172.21.15.67:0/3785341494 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x56531abacd80 0x56531ab26f00 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.858+0000 7f9a2f63bf40 1 -- 172.21.15.67:0/3785341494 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9a1806a5e0 msgr2=0x7f9a1806caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:25.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.858+0000 7f9a2f63bf40 1 --2- 172.21.15.67:0/3785341494 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9a1806a5e0 0x7f9a1806caa0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9a20001d30 tx=0x7f9a20009000 comp rx=0 tx=0).stop 2024-03-24T00:40:25.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.858+0000 7f9a2f63bf40 1 -- 172.21.15.67:0/3785341494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56531ab151c0 msgr2=0x56531abb4630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:25.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.858+0000 7f9a2f63bf40 1 --2- 172.21.15.67:0/3785341494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56531ab151c0 0x56531abb4630 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f9a2401cf90 tx=0x7f9a24002fb0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.858+0000 7f9a2d104640 1 -- 172.21.15.67:0/3785341494 reap_dead start 2024-03-24T00:40:25.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.858+0000 7f9a2f63bf40 1 -- 172.21.15.67:0/3785341494 shutdown_connections 2024-03-24T00:40:25.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.858+0000 7f9a2f63bf40 1 -- 172.21.15.67:0/3785341494 >> 172.21.15.67:0/3785341494 conn(0x56531aae3e70 msgr2=0x56531abb4be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:25.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.858+0000 7f9a2f63bf40 1 -- 172.21.15.67:0/3785341494 shutdown_connections 2024-03-24T00:40:25.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.858+0000 7f9a2f63bf40 1 -- 172.21.15.67:0/3785341494 wait complete. 2024-03-24T00:40:25.864 INFO:teuthology.orchestra.run.smithi067.stdout:mytag 2024-03-24T00:40:25.864 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 100000001f7.00000000 scrub_tag 2024-03-24T00:40:25.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.906+0000 7f229b03bf40 1 Processor -- start 2024-03-24T00:40:25.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.906+0000 7f229b03bf40 1 -- start start 2024-03-24T00:40:25.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.907+0000 7f229b03bf40 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5629988d8210 0x56299895f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:25.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.907+0000 7f229b03bf40 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5629988d85f0 0x562998942a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:25.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.907+0000 7f229b03bf40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56299895e8a0 0x562998942f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:25.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.907+0000 7f229b03bf40 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629988360d0 con 0x5629988d85f0 2024-03-24T00:40:25.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.907+0000 7f229b03bf40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x562998890700 con 0x56299895e8a0 2024-03-24T00:40:25.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.907+0000 7f229b03bf40 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5629988ba3a0 con 0x5629988d8210 2024-03-24T00:40:25.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.907+0000 7f2297b96640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5629988d85f0 0x562998942a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:25.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.907+0000 7f2297b96640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5629988d85f0 0x562998942a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39190/0 (socket says 172.21.15.67:39190) 2024-03-24T00:40:25.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.907+0000 7f2297b96640 1 -- 172.21.15.67:0/3738445140 learned_addr learned my addr 172.21.15.67:0/3738445140 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:25.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.907+0000 7f2298b98640 1 --2- 172.21.15.67:0/3738445140 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56299895e8a0 0x562998942f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:25.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.907+0000 7f2298397640 1 --2- 172.21.15.67:0/3738445140 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5629988d8210 0x56299895f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:25.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.907+0000 7f2297b96640 1 -- 172.21.15.67:0/3738445140 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5629988d8210 msgr2=0x56299895f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:25.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.907+0000 7f2297b96640 1 --2- 172.21.15.67:0/3738445140 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5629988d8210 0x56299895f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.907+0000 7f2297b96640 1 -- 172.21.15.67:0/3738445140 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56299895e8a0 msgr2=0x562998942f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:25.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.907+0000 7f2297b96640 1 --2- 172.21.15.67:0/3738445140 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56299895e8a0 0x562998942f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.907+0000 7f2297b96640 1 -- 172.21.15.67:0/3738445140 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56299895fde0 con 0x5629988d85f0 2024-03-24T00:40:25.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.907+0000 7f2297b96640 1 --2- 172.21.15.67:0/3738445140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5629988d85f0 0x562998942a40 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f228c01cf70 tx=0x7f228c04c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea3d451810150358 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:25.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.908+0000 7f2296b94640 1 -- 172.21.15.67:0/3738445140 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f228c057040 con 0x5629988d85f0 2024-03-24T00:40:25.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.908+0000 7f2296b94640 1 -- 172.21.15.67:0/3738445140 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f228c062070 con 0x5629988d85f0 2024-03-24T00:40:25.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.908+0000 7f2296b94640 1 -- 172.21.15.67:0/3738445140 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f228c052c50 con 0x5629988d85f0 2024-03-24T00:40:25.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.908+0000 7f229b03bf40 1 -- 172.21.15.67:0/3738445140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5629988d85f0 msgr2=0x562998942a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:25.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.908+0000 7f229b03bf40 1 --2- 172.21.15.67:0/3738445140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5629988d85f0 0x562998942a40 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f228c01cf70 tx=0x7f228c04c000 comp rx=0 tx=0).stop 2024-03-24T00:40:25.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.908+0000 7f229b03bf40 1 -- 172.21.15.67:0/3738445140 shutdown_connections 2024-03-24T00:40:25.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.908+0000 7f229b03bf40 1 --2- 172.21.15.67:0/3738445140 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56299895e8a0 0x562998942f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.908+0000 7f229b03bf40 1 --2- 172.21.15.67:0/3738445140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5629988d85f0 0x562998942a40 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.908+0000 7f229b03bf40 1 --2- 172.21.15.67:0/3738445140 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5629988d8210 0x56299895f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.908+0000 7f229b03bf40 1 -- 172.21.15.67:0/3738445140 >> 172.21.15.67:0/3738445140 conn(0x5629988bb320 msgr2=0x5629988b8320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:25.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.908+0000 7f229b03bf40 1 -- 172.21.15.67:0/3738445140 shutdown_connections 2024-03-24T00:40:25.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.908+0000 7f229b03bf40 1 -- 172.21.15.67:0/3738445140 wait complete. 2024-03-24T00:40:25.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.909+0000 7f229b03bf40 1 Processor -- start 2024-03-24T00:40:25.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.909+0000 7f229b03bf40 1 -- start start 2024-03-24T00:40:25.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.909+0000 7f229b03bf40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5629988d8210 0x5629989671a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:25.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.909+0000 7f229b03bf40 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5629988d85f0 0x56299896f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:25.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.909+0000 7f229b03bf40 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56299895e8a0 0x5629988bd870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:25.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.909+0000 7f229b03bf40 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x562998946f10 con 0x5629988d85f0 2024-03-24T00:40:25.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.909+0000 7f229b03bf40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629988d7370 con 0x5629988d8210 2024-03-24T00:40:25.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.909+0000 7f229b03bf40 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5629988ba610 con 0x56299895e8a0 2024-03-24T00:40:25.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.909+0000 7f2297b96640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5629988d85f0 0x56299896f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:25.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.909+0000 7f2298397640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5629988d8210 0x5629989671a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:25.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.909+0000 7f2298b98640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56299895e8a0 0x5629988bd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:25.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.910+0000 7f2297b96640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5629988d85f0 0x56299896f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39206/0 (socket says 172.21.15.67:39206) 2024-03-24T00:40:25.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.910+0000 7f2297b96640 1 -- 172.21.15.67:0/1064582407 learned_addr learned my addr 172.21.15.67:0/1064582407 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:25.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.910+0000 7f2297b96640 1 -- 172.21.15.67:0/1064582407 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56299895e8a0 msgr2=0x5629988bd870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:25.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.910+0000 7f2297b96640 1 --2- 172.21.15.67:0/1064582407 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56299895e8a0 0x5629988bd870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.910+0000 7f2297b96640 1 -- 172.21.15.67:0/1064582407 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5629988d8210 msgr2=0x5629989671a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:25.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.910+0000 7f2297b96640 1 --2- 172.21.15.67:0/1064582407 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5629988d8210 0x5629989671a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.910+0000 7f2297b96640 1 -- 172.21.15.67:0/1064582407 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562998946c10 con 0x5629988d85f0 2024-03-24T00:40:25.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.910+0000 7f2297b96640 1 --2- 172.21.15.67:0/1064582407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5629988d85f0 0x56299896f6a0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f228c01cfe0 tx=0x7f228c002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:25.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.910+0000 7f2295b92640 1 -- 172.21.15.67:0/1064582407 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f228c057040 con 0x5629988d85f0 2024-03-24T00:40:25.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.910+0000 7f2295b92640 1 -- 172.21.15.67:0/1064582407 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f228c0529f0 con 0x5629988d85f0 2024-03-24T00:40:25.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.910+0000 7f229b03bf40 1 -- 172.21.15.67:0/1064582407 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5629988bcd90 con 0x5629988d85f0 2024-03-24T00:40:25.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.910+0000 7f2295b92640 1 -- 172.21.15.67:0/1064582407 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f228c052de0 con 0x5629988d85f0 2024-03-24T00:40:25.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.910+0000 7f229b03bf40 1 -- 172.21.15.67:0/1064582407 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5629988bcaa0 con 0x5629988d85f0 2024-03-24T00:40:25.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.912+0000 7f2295b92640 1 -- 172.21.15.67:0/1064582407 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f228c06f020 con 0x5629988d85f0 2024-03-24T00:40:25.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.912+0000 7f2295b92640 1 --2- 172.21.15.67:0/1064582407 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f228406a5b0 0x7f228406ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:25.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.912+0000 7f2295b92640 1 -- 172.21.15.67:0/1064582407 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f228c0cc770 con 0x5629988d85f0 2024-03-24T00:40:25.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.912+0000 7f2298397640 1 --2- 172.21.15.67:0/1064582407 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f228406a5b0 0x7f228406ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:25.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.912+0000 7f229b03bf40 1 --2- 172.21.15.67:0/1064582407 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x562998946710 0x5629989ff610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:25.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.912+0000 7f229b03bf40 1 -- 172.21.15.67:0/1064582407 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:1 6.36 6:6fe38dc2:::100000001f7.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39) v9 -- 0x562998965220 con 0x562998946710 2024-03-24T00:40:25.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.912+0000 7f2298b98640 1 --2- 172.21.15.67:0/1064582407 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x562998946710 0x5629989ff610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:25.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.912+0000 7f2298397640 1 --2- 172.21.15.67:0/1064582407 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f228406a5b0 0x7f228406ca70 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f22880059b0 tx=0x7f2288005d80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:25.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.913+0000 7f2298b98640 1 --2- 172.21.15.67:0/1064582407 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x562998946710 0x5629989ff610 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:25.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.913+0000 7f2298b98640 1 -- 172.21.15.67:0/1064582407 <== osd.5 v2:172.21.15.72:6800/288486206 1 ==== osd_op_reply(1 100000001f7.00000000 [getxattr (9) out=9b] v0'0 uv17 ondisk = 0) v8 ==== 164+0+9 (crc 0 0 0) 0x7f22900117e0 con 0x562998946710 2024-03-24T00:40:25.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.913+0000 7f229b03bf40 1 -- 172.21.15.67:0/1064582407 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x562998946710 msgr2=0x5629989ff610 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:25.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.913+0000 7f229b03bf40 1 --2- 172.21.15.67:0/1064582407 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x562998946710 0x5629989ff610 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.913+0000 7f229b03bf40 1 -- 172.21.15.67:0/1064582407 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f228406a5b0 msgr2=0x7f228406ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:25.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.913+0000 7f229b03bf40 1 --2- 172.21.15.67:0/1064582407 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f228406a5b0 0x7f228406ca70 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f22880059b0 tx=0x7f2288005d80 comp rx=0 tx=0).stop 2024-03-24T00:40:25.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.913+0000 7f229b03bf40 1 -- 172.21.15.67:0/1064582407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5629988d85f0 msgr2=0x56299896f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:25.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.913+0000 7f229b03bf40 1 --2- 172.21.15.67:0/1064582407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5629988d85f0 0x56299896f6a0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f228c01cfe0 tx=0x7f228c002fb0 comp rx=0 tx=0).stop 2024-03-24T00:40:25.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.913+0000 7f2298b98640 1 -- 172.21.15.67:0/1064582407 reap_dead start 2024-03-24T00:40:25.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.913+0000 7f229b03bf40 1 -- 172.21.15.67:0/1064582407 shutdown_connections 2024-03-24T00:40:25.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.913+0000 7f229b03bf40 1 -- 172.21.15.67:0/1064582407 >> 172.21.15.67:0/1064582407 conn(0x5629988bb320 msgr2=0x562998961990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:25.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.913+0000 7f229b03bf40 1 -- 172.21.15.67:0/1064582407 shutdown_connections 2024-03-24T00:40:25.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:25.913+0000 7f229b03bf40 1 -- 172.21.15.67:0/1064582407 wait complete. 2024-03-24T00:40:25.918 INFO:teuthology.orchestra.run.smithi067.stdout:mytag 2024-03-24T00:40:25.919 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 100000001f8.00000000 scrub_tag 2024-03-24T00:40:26.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.000+0000 7f3ef0e37f40 1 Processor -- start 2024-03-24T00:40:26.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.000+0000 7f3ef0e37f40 1 -- start start 2024-03-24T00:40:26.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.000+0000 7f3ef0e37f40 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558c5a5db1c0 0x558c5a670ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:26.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.000+0000 7f3ef0e37f40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558c5a5f3d90 0x558c5a5efe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:26.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.000+0000 7f3ef0e37f40 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558c5a680150 0x558c5a5f06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:26.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.000+0000 7f3ef0e37f40 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c5a680740 con 0x558c5a680150 2024-03-24T00:40:26.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.000+0000 7f3ef0e37f40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c5a5520d0 con 0x558c5a5f3d90 2024-03-24T00:40:26.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.000+0000 7f3ef0e37f40 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x558c5a5f4510 con 0x558c5a5db1c0 2024-03-24T00:40:26.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.000+0000 7f3eee904640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558c5a680150 0x558c5a5f06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:26.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.001+0000 7f3eee103640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558c5a5db1c0 0x558c5a670ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:26.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.001+0000 7f3eee904640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558c5a680150 0x558c5a5f06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39222/0 (socket says 172.21.15.67:39222) 2024-03-24T00:40:26.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.001+0000 7f3eee904640 1 -- 172.21.15.67:0/3468905099 learned_addr learned my addr 172.21.15.67:0/3468905099 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:26.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.001+0000 7f3eed902640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558c5a5f3d90 0x558c5a5efe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:26.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.001+0000 7f3eee904640 1 -- 172.21.15.67:0/3468905099 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558c5a5db1c0 msgr2=0x558c5a670ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:26.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.001+0000 7f3eee904640 1 --2- 172.21.15.67:0/3468905099 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558c5a5db1c0 0x558c5a670ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:26.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.001+0000 7f3eee904640 1 -- 172.21.15.67:0/3468905099 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558c5a5f3d90 msgr2=0x558c5a5efe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:26.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.001+0000 7f3eee904640 1 --2- 172.21.15.67:0/3468905099 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558c5a5f3d90 0x558c5a5efe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:26.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.001+0000 7f3eee904640 1 -- 172.21.15.67:0/3468905099 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558c5a5ef6a0 con 0x558c5a680150 2024-03-24T00:40:26.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.001+0000 7f3eee103640 1 --2- 172.21.15.67:0/3468905099 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558c5a5db1c0 0x558c5a670ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:40:26.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.001+0000 7f3eed902640 1 --2- 172.21.15.67:0/3468905099 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558c5a5f3d90 0x558c5a5efe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:40:26.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.001+0000 7f3eee904640 1 --2- 172.21.15.67:0/3468905099 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558c5a680150 0x558c5a5f06e0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f3ee800a4f0 tx=0x7f3ee804c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e168d527cd6d455 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:26.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.001+0000 7f3eec900640 1 -- 172.21.15.67:0/3468905099 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee8057040 con 0x558c5a680150 2024-03-24T00:40:26.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.001+0000 7f3eec900640 1 -- 172.21.15.67:0/3468905099 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ee8062070 con 0x558c5a680150 2024-03-24T00:40:26.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.001+0000 7f3eec900640 1 -- 172.21.15.67:0/3468905099 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee8052c30 con 0x558c5a680150 2024-03-24T00:40:26.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.002+0000 7f3ef0e37f40 1 -- 172.21.15.67:0/3468905099 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558c5a680150 msgr2=0x558c5a5f06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:26.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.002+0000 7f3ef0e37f40 1 --2- 172.21.15.67:0/3468905099 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558c5a680150 0x558c5a5f06e0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f3ee800a4f0 tx=0x7f3ee804c000 comp rx=0 tx=0).stop 2024-03-24T00:40:26.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.002+0000 7f3ef0e37f40 1 -- 172.21.15.67:0/3468905099 shutdown_connections 2024-03-24T00:40:26.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.002+0000 7f3ef0e37f40 1 --2- 172.21.15.67:0/3468905099 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558c5a680150 0x558c5a5f06e0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:26.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.002+0000 7f3ef0e37f40 1 --2- 172.21.15.67:0/3468905099 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558c5a5f3d90 0x558c5a5efe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:26.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.002+0000 7f3ef0e37f40 1 --2- 172.21.15.67:0/3468905099 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558c5a5db1c0 0x558c5a670ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:26.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.002+0000 7f3ef0e37f40 1 -- 172.21.15.67:0/3468905099 >> 172.21.15.67:0/3468905099 conn(0x558c5a5a9e70 msgr2=0x558c5a67ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:26.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.002+0000 7f3ef0e37f40 1 -- 172.21.15.67:0/3468905099 shutdown_connections 2024-03-24T00:40:26.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.002+0000 7f3ef0e37f40 1 -- 172.21.15.67:0/3468905099 wait complete. 2024-03-24T00:40:26.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.002+0000 7f3ef0e37f40 1 Processor -- start 2024-03-24T00:40:26.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.002+0000 7f3ef0e37f40 1 -- start start 2024-03-24T00:40:26.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.003+0000 7f3ef0e37f40 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558c5a5db1c0 0x558c5a5f0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:26.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.003+0000 7f3ef0e37f40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558c5a5f3d90 0x558c5a67aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:26.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.003+0000 7f3ef0e37f40 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558c5a680150 0x558c5a67af60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:26.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.003+0000 7f3ef0e37f40 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c5a5f1990 con 0x558c5a680150 2024-03-24T00:40:26.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.003+0000 7f3ef0e37f40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c5a5f16b0 con 0x558c5a5f3d90 2024-03-24T00:40:26.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.003+0000 7f3ef0e37f40 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x558c5a5f1c70 con 0x558c5a5db1c0 2024-03-24T00:40:26.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.003+0000 7f3eee904640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558c5a680150 0x558c5a67af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:26.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.003+0000 7f3eee904640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558c5a680150 0x558c5a67af60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39238/0 (socket says 172.21.15.67:39238) 2024-03-24T00:40:26.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.003+0000 7f3eee103640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558c5a5db1c0 0x558c5a5f0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:26.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.003+0000 7f3eee904640 1 -- 172.21.15.67:0/3612561623 learned_addr learned my addr 172.21.15.67:0/3612561623 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:26.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.003+0000 7f3eed902640 1 --2- 172.21.15.67:0/3612561623 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558c5a5f3d90 0x558c5a67aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:26.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.003+0000 7f3eee904640 1 -- 172.21.15.67:0/3612561623 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558c5a5db1c0 msgr2=0x558c5a5f0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:26.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.003+0000 7f3eee904640 1 --2- 172.21.15.67:0/3612561623 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558c5a5db1c0 0x558c5a5f0720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:26.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.003+0000 7f3eee904640 1 -- 172.21.15.67:0/3612561623 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558c5a5f3d90 msgr2=0x558c5a67aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:26.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.003+0000 7f3eee904640 1 --2- 172.21.15.67:0/3612561623 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558c5a5f3d90 0x558c5a67aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:26.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.003+0000 7f3eee904640 1 -- 172.21.15.67:0/3612561623 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558c5a671900 con 0x558c5a680150 2024-03-24T00:40:26.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.003+0000 7f3eee103640 1 --2- 172.21.15.67:0/3612561623 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558c5a5db1c0 0x558c5a5f0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:40:26.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.003+0000 7f3eed902640 1 --2- 172.21.15.67:0/3612561623 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558c5a5f3d90 0x558c5a67aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:40:26.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.004+0000 7f3eee904640 1 --2- 172.21.15.67:0/3612561623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558c5a680150 0x558c5a67af60 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f3ee800a4f0 tx=0x7f3ee8002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:26.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.004+0000 7f3ee77fe640 1 -- 172.21.15.67:0/3612561623 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee8057040 con 0x558c5a680150 2024-03-24T00:40:26.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.004+0000 7f3ee77fe640 1 -- 172.21.15.67:0/3612561623 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ee8052ab0 con 0x558c5a680150 2024-03-24T00:40:26.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.004+0000 7f3ef0e37f40 1 -- 172.21.15.67:0/3612561623 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x558c5a670170 con 0x558c5a680150 2024-03-24T00:40:26.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.004+0000 7f3ee77fe640 1 -- 172.21.15.67:0/3612561623 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee805d470 con 0x558c5a680150 2024-03-24T00:40:26.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.004+0000 7f3ef0e37f40 1 -- 172.21.15.67:0/3612561623 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558c5a67a500 con 0x558c5a680150 2024-03-24T00:40:26.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.005+0000 7f3ee77fe640 1 -- 172.21.15.67:0/3612561623 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3ee806f020 con 0x558c5a680150 2024-03-24T00:40:26.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.005+0000 7f3ee77fe640 1 --2- 172.21.15.67:0/3612561623 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3ed006a5b0 0x7f3ed006ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:26.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.006+0000 7f3ee77fe640 1 -- 172.21.15.67:0/3612561623 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3ee80cce00 con 0x558c5a680150 2024-03-24T00:40:26.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.006+0000 7f3eee103640 1 --2- 172.21.15.67:0/3612561623 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3ed006a5b0 0x7f3ed006ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:26.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.006+0000 7f3ef0e37f40 1 --2- 172.21.15.67:0/3612561623 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x558c5a5f0c60 0x558c5a67b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:26.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.006+0000 7f3ef0e37f40 1 -- 172.21.15.67:0/3612561623 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:1 6.19 6:997b7129:::100000001f8.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39) v9 -- 0x558c5a66f060 con 0x558c5a5f0c60 2024-03-24T00:40:26.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.006+0000 7f3eed902640 1 --2- 172.21.15.67:0/3612561623 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x558c5a5f0c60 0x558c5a67b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:26.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.006+0000 7f3eee103640 1 --2- 172.21.15.67:0/3612561623 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3ed006a5b0 0x7f3ed006ca70 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x558c5a6781e0 tx=0x7f3ee0007040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:26.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.006+0000 7f3eed902640 1 --2- 172.21.15.67:0/3612561623 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x558c5a5f0c60 0x558c5a67b9b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:26.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.006+0000 7f3eed902640 1 -- 172.21.15.67:0/3612561623 <== osd.5 v2:172.21.15.72:6800/288486206 1 ==== osd_op_reply(1 100000001f8.00000000 [getxattr (9) out=9b] v0'0 uv12 ondisk = 0) v8 ==== 164+0+9 (crc 0 0 0) 0x7f3ed800ec30 con 0x558c5a5f0c60 2024-03-24T00:40:26.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.007+0000 7f3ef0e37f40 1 -- 172.21.15.67:0/3612561623 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x558c5a5f0c60 msgr2=0x558c5a67b9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:26.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.007+0000 7f3ef0e37f40 1 --2- 172.21.15.67:0/3612561623 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x558c5a5f0c60 0x558c5a67b9b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:26.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.007+0000 7f3ef0e37f40 1 -- 172.21.15.67:0/3612561623 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3ed006a5b0 msgr2=0x7f3ed006ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:26.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.007+0000 7f3ef0e37f40 1 --2- 172.21.15.67:0/3612561623 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3ed006a5b0 0x7f3ed006ca70 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x558c5a6781e0 tx=0x7f3ee0007040 comp rx=0 tx=0).stop 2024-03-24T00:40:26.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.007+0000 7f3ef0e37f40 1 -- 172.21.15.67:0/3612561623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558c5a680150 msgr2=0x558c5a67af60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:26.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.007+0000 7f3ef0e37f40 1 --2- 172.21.15.67:0/3612561623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558c5a680150 0x558c5a67af60 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f3ee800a4f0 tx=0x7f3ee8002fb0 comp rx=0 tx=0).stop 2024-03-24T00:40:26.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.007+0000 7f3eee904640 1 -- 172.21.15.67:0/3612561623 reap_dead start 2024-03-24T00:40:26.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.007+0000 7f3ef0e37f40 1 -- 172.21.15.67:0/3612561623 shutdown_connections 2024-03-24T00:40:26.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.007+0000 7f3ef0e37f40 1 -- 172.21.15.67:0/3612561623 >> 172.21.15.67:0/3612561623 conn(0x558c5a5a9e70 msgr2=0x558c5a5d8680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:26.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.007+0000 7f3ef0e37f40 1 -- 172.21.15.67:0/3612561623 shutdown_connections 2024-03-24T00:40:26.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.007+0000 7f3ef0e37f40 1 -- 172.21.15.67:0/3612561623 wait complete. 2024-03-24T00:40:26.012 INFO:teuthology.orchestra.run.smithi067.stdout:mytag 2024-03-24T00:40:26.013 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000002.00000000 scrub_tag 2024-03-24T00:40:26.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.094+0000 7fdc0a839f40 1 Processor -- start 2024-03-24T00:40:26.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.094+0000 7fdc0a839f40 1 -- start start 2024-03-24T00:40:26.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.095+0000 7fdc0a839f40 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559bb0452d70 0x559bb0527fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:26.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.095+0000 7fdc0a839f40 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559bb0480ab0 0x559bb050ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:26.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.095+0000 7fdc0a839f40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559bb0522ba0 0x559bb04508e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:26.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.095+0000 7fdc0a839f40 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x559bb0529930 con 0x559bb0480ab0 2024-03-24T00:40:26.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.095+0000 7fdc0a839f40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x559bb03f80d0 con 0x559bb0522ba0 2024-03-24T00:40:26.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.095+0000 7fdc0a839f40 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x559bb050ec00 con 0x559bb0452d70 2024-03-24T00:40:26.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.095+0000 7fdc07244640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559bb0480ab0 0x559bb050ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:26.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.095+0000 7fdc07244640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559bb0480ab0 0x559bb050ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39240/0 (socket says 172.21.15.67:39240) 2024-03-24T00:40:26.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.095+0000 7fdc07244640 1 -- 172.21.15.67:0/793126727 learned_addr learned my addr 172.21.15.67:0/793126727 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:26.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.095+0000 7fdc08246640 1 --2- 172.21.15.67:0/793126727 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559bb0522ba0 0x559bb04508e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:26.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.095+0000 7fdc07a45640 1 --2- 172.21.15.67:0/793126727 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559bb0452d70 0x559bb0527fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:26.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.095+0000 7fdc07244640 1 -- 172.21.15.67:0/793126727 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559bb0452d70 msgr2=0x559bb0527fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:26.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.095+0000 7fdc07244640 1 --2- 172.21.15.67:0/793126727 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559bb0452d70 0x559bb0527fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:26.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.095+0000 7fdc07244640 1 -- 172.21.15.67:0/793126727 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559bb0522ba0 msgr2=0x559bb04508e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:26.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.095+0000 7fdc07244640 1 --2- 172.21.15.67:0/793126727 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559bb0522ba0 0x559bb04508e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:26.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.095+0000 7fdc07244640 1 -- 172.21.15.67:0/793126727 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559bb0506b90 con 0x559bb0480ab0 2024-03-24T00:40:26.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.095+0000 7fdc08246640 1 --2- 172.21.15.67:0/793126727 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559bb0522ba0 0x559bb04508e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:40:26.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.095+0000 7fdc07244640 1 --2- 172.21.15.67:0/793126727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559bb0480ab0 0x559bb050ad50 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fdbf00099a0 tx=0x7fdbf0005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52f48ddb1f4af7f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:26.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.095+0000 7fdc06242640 1 -- 172.21.15.67:0/793126727 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf0008a30 con 0x559bb0480ab0 2024-03-24T00:40:26.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.095+0000 7fdc06242640 1 -- 172.21.15.67:0/793126727 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbf0018070 con 0x559bb0480ab0 2024-03-24T00:40:26.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.095+0000 7fdc06242640 1 -- 172.21.15.67:0/793126727 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf0008dd0 con 0x559bb0480ab0 2024-03-24T00:40:26.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.096+0000 7fdc0a839f40 1 -- 172.21.15.67:0/793126727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559bb0480ab0 msgr2=0x559bb050ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:26.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.096+0000 7fdc0a839f40 1 --2- 172.21.15.67:0/793126727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559bb0480ab0 0x559bb050ad50 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fdbf00099a0 tx=0x7fdbf0005a00 comp rx=0 tx=0).stop 2024-03-24T00:40:26.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.096+0000 7fdc0a839f40 1 -- 172.21.15.67:0/793126727 shutdown_connections 2024-03-24T00:40:26.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.096+0000 7fdc0a839f40 1 --2- 172.21.15.67:0/793126727 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559bb0522ba0 0x559bb04508e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:26.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.096+0000 7fdc0a839f40 1 --2- 172.21.15.67:0/793126727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559bb0480ab0 0x559bb050ad50 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:26.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.096+0000 7fdc0a839f40 1 --2- 172.21.15.67:0/793126727 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559bb0452d70 0x559bb0527fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:26.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.096+0000 7fdc0a839f40 1 -- 172.21.15.67:0/793126727 >> 172.21.15.67:0/793126727 conn(0x559bb044fe70 msgr2=0x559bb0453f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:26.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.096+0000 7fdc0a839f40 1 -- 172.21.15.67:0/793126727 shutdown_connections 2024-03-24T00:40:26.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.096+0000 7fdc0a839f40 1 -- 172.21.15.67:0/793126727 wait complete. 2024-03-24T00:40:26.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.096+0000 7fdc0a839f40 1 Processor -- start 2024-03-24T00:40:26.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.096+0000 7fdc0a839f40 1 -- start start 2024-03-24T00:40:26.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.097+0000 7fdc0a839f40 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559bb0452d70 0x559bb0505880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:26.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.097+0000 7fdc0a839f40 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559bb0480ab0 0x559bb0507fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:26.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.097+0000 7fdc0a839f40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559bb0522ba0 0x559bb0509da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:26.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.097+0000 7fdc0a839f40 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x559bb050e850 con 0x559bb0452d70 2024-03-24T00:40:26.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.097+0000 7fdc0a839f40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x559bb0506e70 con 0x559bb0522ba0 2024-03-24T00:40:26.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.097+0000 7fdc0a839f40 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x559bb0495d20 con 0x559bb0480ab0 2024-03-24T00:40:26.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.097+0000 7fdc07a45640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559bb0452d70 0x559bb0505880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:26.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.097+0000 7fdc07a45640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559bb0452d70 0x559bb0505880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39256/0 (socket says 172.21.15.67:39256) 2024-03-24T00:40:26.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.097+0000 7fdc07a45640 1 -- 172.21.15.67:0/2005477757 learned_addr learned my addr 172.21.15.67:0/2005477757 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:26.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.097+0000 7fdc07244640 1 --2- 172.21.15.67:0/2005477757 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559bb0480ab0 0x559bb0507fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:26.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.097+0000 7fdc08246640 1 --2- 172.21.15.67:0/2005477757 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559bb0522ba0 0x559bb0509da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:26.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.098+0000 7fdc07a45640 1 -- 172.21.15.67:0/2005477757 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559bb0480ab0 msgr2=0x559bb0507fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:26.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.098+0000 7fdc07a45640 1 --2- 172.21.15.67:0/2005477757 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559bb0480ab0 0x559bb0507fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:26.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.098+0000 7fdc07a45640 1 -- 172.21.15.67:0/2005477757 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559bb0522ba0 msgr2=0x559bb0509da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:26.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.098+0000 7fdc07a45640 1 --2- 172.21.15.67:0/2005477757 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559bb0522ba0 0x559bb0509da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:26.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.098+0000 7fdc07a45640 1 -- 172.21.15.67:0/2005477757 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559bb0505dc0 con 0x559bb0452d70 2024-03-24T00:40:26.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.098+0000 7fdc07a45640 1 --2- 172.21.15.67:0/2005477757 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559bb0452d70 0x559bb0505880 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fdbf801cca0 tx=0x7fdbf8003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:26.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.098+0000 7fdc05240640 1 -- 172.21.15.67:0/2005477757 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf8068680 con 0x559bb0452d70 2024-03-24T00:40:26.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.098+0000 7fdc05240640 1 -- 172.21.15.67:0/2005477757 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbf8068820 con 0x559bb0452d70 2024-03-24T00:40:26.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.098+0000 7fdc0a839f40 1 -- 172.21.15.67:0/2005477757 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x559bb05084f0 con 0x559bb0452d70 2024-03-24T00:40:26.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.098+0000 7fdc05240640 1 -- 172.21.15.67:0/2005477757 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf8068bd0 con 0x559bb0452d70 2024-03-24T00:40:26.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.098+0000 7fdc0a839f40 1 -- 172.21.15.67:0/2005477757 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559bb0497d70 con 0x559bb0452d70 2024-03-24T00:40:26.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.099+0000 7fdc05240640 1 -- 172.21.15.67:0/2005477757 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fdbf805d410 con 0x559bb0452d70 2024-03-24T00:40:26.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:40:26.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.100+0000 7fdc05240640 1 --2- 172.21.15.67:0/2005477757 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdbe806a5b0 0x7fdbe806ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:26.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.100+0000 7fdc05240640 1 -- 172.21.15.67:0/2005477757 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fdbf80cc630 con 0x559bb0452d70 2024-03-24T00:40:26.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.100+0000 7fdc07244640 1 --2- 172.21.15.67:0/2005477757 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdbe806a5b0 0x7fdbe806ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:26.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.100+0000 7fdc0a839f40 1 --2- 172.21.15.67:0/2005477757 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x559bb0496490 0x559bb0496870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:26.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.100+0000 7fdc0a839f40 1 -- 172.21.15.67:0/2005477757 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:1 6.12 6:4bec3886:::10000000002.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39) v9 -- 0x559bb0496db0 con 0x559bb0496490 2024-03-24T00:40:26.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.100+0000 7fdc08246640 1 --2- 172.21.15.67:0/2005477757 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x559bb0496490 0x559bb0496870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:26.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.100+0000 7fdc07244640 1 --2- 172.21.15.67:0/2005477757 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdbe806a5b0 0x7fdbe806ca70 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x559bb050a980 tx=0x7fdbf0012000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:26.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.100+0000 7fdc08246640 1 --2- 172.21.15.67:0/2005477757 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x559bb0496490 0x559bb0496870 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:26.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.101+0000 7fdc08246640 1 -- 172.21.15.67:0/2005477757 <== osd.7 v2:172.21.15.72:6824/2893038746 1 ==== osd_op_reply(1 10000000002.00000000 [getxattr] v0'0 uv0 ondisk = -61 ((61) No data available)) v8 ==== 164+0+0 (crc 0 0 0) 0x7fdc0000e3e0 con 0x559bb0496490 2024-03-24T00:40:26.108 INFO:teuthology.orchestra.run.smithi067.stderr:error getting xattr cephfs_data/10000000002.00000000/scrub_tag: (61) No data available 2024-03-24T00:40:26.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.101+0000 7fdc0a839f40 1 -- 172.21.15.67:0/2005477757 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x559bb0496490 msgr2=0x559bb0496870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:26.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.101+0000 7fdc0a839f40 1 --2- 172.21.15.67:0/2005477757 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x559bb0496490 0x559bb0496870 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:26.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.101+0000 7fdc0a839f40 1 -- 172.21.15.67:0/2005477757 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdbe806a5b0 msgr2=0x7fdbe806ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:26.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.101+0000 7fdc0a839f40 1 --2- 172.21.15.67:0/2005477757 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdbe806a5b0 0x7fdbe806ca70 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x559bb050a980 tx=0x7fdbf0012000 comp rx=0 tx=0).stop 2024-03-24T00:40:26.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.101+0000 7fdc0a839f40 1 -- 172.21.15.67:0/2005477757 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559bb0452d70 msgr2=0x559bb0505880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:26.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.101+0000 7fdc0a839f40 1 --2- 172.21.15.67:0/2005477757 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559bb0452d70 0x559bb0505880 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fdbf801cca0 tx=0x7fdbf8003330 comp rx=0 tx=0).stop 2024-03-24T00:40:26.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.101+0000 7fdc08246640 1 -- 172.21.15.67:0/2005477757 reap_dead start 2024-03-24T00:40:26.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.101+0000 7fdc0a839f40 1 -- 172.21.15.67:0/2005477757 shutdown_connections 2024-03-24T00:40:26.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.101+0000 7fdc0a839f40 1 -- 172.21.15.67:0/2005477757 >> 172.21.15.67:0/2005477757 conn(0x559bb044fe70 msgr2=0x559bb049a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:26.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.101+0000 7fdc0a839f40 1 -- 172.21.15.67:0/2005477757 shutdown_connections 2024-03-24T00:40:26.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.101+0000 7fdc0a839f40 1 -- 172.21.15.67:0/2005477757 wait complete. 2024-03-24T00:40:26.109 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-24T00:40:26.109 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:40:26.190 DEBUG:teuthology.orchestra.run.smithi072:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:40:26.227 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-24T00:40:26.228 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:40:26.228 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:40:26.245 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056375,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15771 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T00:40:26.245 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:40:26.245 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:40:26.283 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17590 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248808k,nr_inodes=812202,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:40:26.283 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:40:26.283 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:40:26.299 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056375,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15771 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T00:40:26.300 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:40:26.300 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:40:26.338 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17590 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248808k,nr_inodes=812202,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:40:26.338 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:40:26.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.439+0000 7f090dd32640 1 Processor -- start 2024-03-24T00:40:26.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.439+0000 7f090dd32640 1 -- start start 2024-03-24T00:40:26.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.440+0000 7f090dd32640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f09080b43a0 0x7f09080cfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:26.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.440+0000 7f090dd32640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09080d0fc0 con 0x7f090815e1b0 2024-03-24T00:40:26.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.440+0000 7f090dd32640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09080d1260 con 0x7f090809a410 2024-03-24T00:40:26.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.440+0000 7f090dd32640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f090815e590 con 0x7f09080b43a0 2024-03-24T00:40:26.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.440+0000 7f09077fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f090815e1b0 0x7f0908099d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44116/0 (socket says 172.21.15.67:44116) 2024-03-24T00:40:26.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.440+0000 7f09077fe640 1 -- 172.21.15.67:0/1751952560 learned_addr learned my addr 172.21.15.67:0/1751952560 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:40:26.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.440+0000 7f0907fff640 1 --2- 172.21.15.67:0/1751952560 >> v2:172.21.15.72:3300/0 conn(0x7f09080b43a0 0x7f09080cfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:26.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.440+0000 7f09067fc640 1 -- 172.21.15.67:0/1751952560 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3627902506 0 0) 0x7f09080d0fc0 con 0x7f090815e1b0 2024-03-24T00:40:26.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.441+0000 7f09067fc640 1 -- 172.21.15.67:0/1751952560 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08f0003520 con 0x7f090815e1b0 2024-03-24T00:40:26.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.441+0000 7f0907fff640 1 -- 172.21.15.67:0/1751952560 >> v1:172.21.15.72:6789/0 conn(0x7f090809a410 legacy=0x7f09080cc880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:26.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.441+0000 7f0907fff640 1 -- 172.21.15.67:0/1751952560 >> v1:172.21.15.67:6789/0 conn(0x7f090815e1b0 legacy=0x7f0908099d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:26.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.441+0000 7f0907fff640 1 -- 172.21.15.67:0/1751952560 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09080d1540 con 0x7f09080b43a0 2024-03-24T00:40:26.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.441+0000 7f0907fff640 1 --2- 172.21.15.67:0/1751952560 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09080b43a0 0x7f09080cfe20 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0900009d90 tx=0x7f090002be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7e16415dd60a81b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:26.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.442+0000 7f09067fc640 1 -- 172.21.15.67:0/1751952560 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0900039070 con 0x7f09080b43a0 2024-03-24T00:40:26.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.442+0000 7f09067fc640 1 -- 172.21.15.67:0/1751952560 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f090002cbd0 con 0x7f09080b43a0 2024-03-24T00:40:26.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.442+0000 7f09067fc640 1 -- 172.21.15.67:0/1751952560 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0900034440 con 0x7f09080b43a0 2024-03-24T00:40:26.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.442+0000 7f090dd32640 1 -- 172.21.15.67:0/1751952560 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09080b43a0 msgr2=0x7f09080cfe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:26.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.442+0000 7f090dd32640 1 --2- 172.21.15.67:0/1751952560 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09080b43a0 0x7f09080cfe20 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0900009d90 tx=0x7f090002be20 comp rx=0 tx=0).stop 2024-03-24T00:40:26.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.442+0000 7f090dd32640 1 -- 172.21.15.67:0/1751952560 shutdown_connections 2024-03-24T00:40:26.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.442+0000 7f090dd32640 1 --2- 172.21.15.67:0/1751952560 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09080b43a0 0x7f09080cfe20 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:26.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.442+0000 7f090dd32640 1 -- 172.21.15.67:0/1751952560 >> 172.21.15.67:0/1751952560 conn(0x7f09080b36d0 msgr2=0x7f09080b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:26.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.442+0000 7f090dd32640 1 -- 172.21.15.67:0/1751952560 shutdown_connections 2024-03-24T00:40:26.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.442+0000 7f090dd32640 1 -- 172.21.15.67:0/1751952560 wait complete. 2024-03-24T00:40:26.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.443+0000 7f090dd32640 1 Processor -- start 2024-03-24T00:40:26.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.443+0000 7f090dd32640 1 -- start start 2024-03-24T00:40:26.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.444+0000 7f090dd32640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f090809a410 0x7f09080d05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:26.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.444+0000 7f090dd32640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09080b43a0 0x7f09081f45a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:26.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.444+0000 7f090dd32640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f090815e1b0 0x7f09080c6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:26.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.444+0000 7f090dd32640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f090809a9a0 con 0x7f090809a410 2024-03-24T00:40:26.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.444+0000 7f090dd32640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09080d1820 con 0x7f09080b43a0 2024-03-24T00:40:26.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.444+0000 7f090dd32640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09080ccf90 con 0x7f090815e1b0 2024-03-24T00:40:26.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.444+0000 7f09077fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f090809a410 0x7f09080d05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:26.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.444+0000 7f09077fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f090809a410 0x7f09080d05c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39270/0 (socket says 172.21.15.67:39270) 2024-03-24T00:40:26.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.444+0000 7f09077fe640 1 -- 172.21.15.67:0/553094596 learned_addr learned my addr 172.21.15.67:0/553094596 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:26.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.444+0000 7f0906ffd640 1 --2- 172.21.15.67:0/553094596 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09080b43a0 0x7f09081f45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:26.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.444+0000 7f0907fff640 1 --2- 172.21.15.67:0/553094596 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f090815e1b0 0x7f09080c6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:26.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.445+0000 7f09077fe640 1 -- 172.21.15.67:0/553094596 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f090815e1b0 msgr2=0x7f09080c6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:26.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.445+0000 7f09077fe640 1 --2- 172.21.15.67:0/553094596 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f090815e1b0 0x7f09080c6850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:26.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.445+0000 7f09077fe640 1 -- 172.21.15.67:0/553094596 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09080b43a0 msgr2=0x7f09081f45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:26.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.445+0000 7f09077fe640 1 --2- 172.21.15.67:0/553094596 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09080b43a0 0x7f09081f45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:26.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.445+0000 7f09077fe640 1 -- 172.21.15.67:0/553094596 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09080c57b0 con 0x7f090809a410 2024-03-24T00:40:26.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.445+0000 7f09077fe640 1 --2- 172.21.15.67:0/553094596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f090809a410 0x7f09080d05c0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f08f800c440 tx=0x7f08f800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:26.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.445+0000 7f08effff640 1 -- 172.21.15.67:0/553094596 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f8017070 con 0x7f090809a410 2024-03-24T00:40:26.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.445+0000 7f08effff640 1 -- 172.21.15.67:0/553094596 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08f8005ce0 con 0x7f090809a410 2024-03-24T00:40:26.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.445+0000 7f090dd32640 1 -- 172.21.15.67:0/553094596 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09080c5a90 con 0x7f090809a410 2024-03-24T00:40:26.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.445+0000 7f08effff640 1 -- 172.21.15.67:0/553094596 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f80072b0 con 0x7f090809a410 2024-03-24T00:40:26.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.445+0000 7f090dd32640 1 -- 172.21.15.67:0/553094596 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09080c6050 con 0x7f090809a410 2024-03-24T00:40:26.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.446+0000 7f090dd32640 1 -- 172.21.15.67:0/553094596 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f090804f400 con 0x7f090809a410 2024-03-24T00:40:26.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.446+0000 7f08effff640 1 -- 172.21.15.67:0/553094596 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f08f801a070 con 0x7f090809a410 2024-03-24T00:40:26.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.447+0000 7f08effff640 1 --2- 172.21.15.67:0/553094596 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f08f406bbd0 0x7f08f406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:26.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.447+0000 7f08effff640 1 -- 172.21.15.67:0/553094596 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f08f8089e90 con 0x7f090809a410 2024-03-24T00:40:26.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.447+0000 7f0906ffd640 1 --2- 172.21.15.67:0/553094596 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f08f406bbd0 0x7f08f406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:26.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.447+0000 7f0906ffd640 1 --2- 172.21.15.67:0/553094596 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f08f406bbd0 0x7f08f406e090 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f09081f4cc0 tx=0x7f08fc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:26.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.451+0000 7f08effff640 1 -- 172.21.15.67:0/553094596 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f08f8058270 con 0x7f090809a410 2024-03-24T00:40:26.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.671+0000 7f090dd32640 1 -- 172.21.15.67:0/553094596 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0908068520 con 0x7f090809a410 2024-03-24T00:40:26.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.671+0000 7f08effff640 1 -- 172.21.15.67:0/553094596 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f08f805c310 con 0x7f090809a410 2024-03-24T00:40:26.672 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:40:26.672 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5039,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/166859817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":166859817},{"type":"v1","addr":"172.21.15.67:6834","nonce":166859817}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5042,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2653663845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2653663845},{"type":"v1","addr":"172.21.15.67:6835","nonce":2653663845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5048,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2355909634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2355909634},{"type":"v1","addr":"172.21.15.72:6837","nonce":2355909634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:40:00.691429+0000","modified":"2024-03-24T00:40:03.146091+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5418},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5418":{"gid":5418,"name":"d","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6836/2551774148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2551774148},{"type":"v1","addr":"172.21.15.72:6836","nonce":2551774148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5418,"qdb_cluster":[5418]},"id":2}]} 2024-03-24T00:40:26.672 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 17 2024-03-24T00:40:26.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.673+0000 7f090dd32640 1 -- 172.21.15.67:0/553094596 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f08f406bbd0 msgr2=0x7f08f406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:26.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.673+0000 7f090dd32640 1 --2- 172.21.15.67:0/553094596 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f08f406bbd0 0x7f08f406e090 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f09081f4cc0 tx=0x7f08fc009040 comp rx=0 tx=0).stop 2024-03-24T00:40:26.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.673+0000 7f090dd32640 1 -- 172.21.15.67:0/553094596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f090809a410 msgr2=0x7f09080d05c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:26.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.673+0000 7f090dd32640 1 --2- 172.21.15.67:0/553094596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f090809a410 0x7f09080d05c0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f08f800c440 tx=0x7f08f800c900 comp rx=0 tx=0).stop 2024-03-24T00:40:26.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.673+0000 7f090dd32640 1 -- 172.21.15.67:0/553094596 shutdown_connections 2024-03-24T00:40:26.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.673+0000 7f090dd32640 1 --2- 172.21.15.67:0/553094596 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f08f406bbd0 0x7f08f406e090 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:26.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.673+0000 7f090dd32640 1 --2- 172.21.15.67:0/553094596 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f090815e1b0 0x7f09080c6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:26.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.673+0000 7f090dd32640 1 --2- 172.21.15.67:0/553094596 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09080b43a0 0x7f09081f45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:26.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.673+0000 7f090dd32640 1 --2- 172.21.15.67:0/553094596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f090809a410 0x7f09080d05c0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:26.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.673+0000 7f090dd32640 1 -- 172.21.15.67:0/553094596 >> 172.21.15.67:0/553094596 conn(0x7f09080b36d0 msgr2=0x7f0908070780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:26.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.673+0000 7f090dd32640 1 -- 172.21.15.67:0/553094596 shutdown_connections 2024-03-24T00:40:26.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.674+0000 7f090dd32640 1 -- 172.21.15.67:0/553094596 wait complete. 2024-03-24T00:40:26.685 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:40:26.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.785+0000 7fc5eca59640 1 Processor -- start 2024-03-24T00:40:26.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.785+0000 7fc5eca59640 1 -- start start 2024-03-24T00:40:26.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.786+0000 7fc5eca59640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc5e815dec0 0x7fc5e80ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:26.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.786+0000 7fc5eca59640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc5e80d0ef0 0x7fc5e8162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:26.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.786+0000 7fc5eca59640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5e80cf640 con 0x7fc5e815e2a0 2024-03-24T00:40:26.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.786+0000 7fc5eca59640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5e80cee30 con 0x7fc5e815dec0 2024-03-24T00:40:26.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.786+0000 7fc5eca59640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5e80d2120 con 0x7fc5e80d0ef0 2024-03-24T00:40:26.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.786+0000 7fc5e659c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc5e815dec0 0x7fc5e80ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:26.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.786+0000 7fc5e6d9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc5e80d0ef0 0x7fc5e8162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:26.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.786+0000 7fc5e659c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc5e815dec0 0x7fc5e80ce630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39280/0 (socket says 172.21.15.67:39280) 2024-03-24T00:40:26.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.786+0000 7fc5e6d9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc5e80d0ef0 0x7fc5e8162630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:51584/0 (socket says 172.21.15.67:51584) 2024-03-24T00:40:26.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.786+0000 7fc5e659c640 1 -- 172.21.15.67:0/353450871 learned_addr learned my addr 172.21.15.67:0/353450871 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:26.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.786+0000 7fc5e5d9b640 1 --1- 172.21.15.67:0/353450871 >> v1:172.21.15.72:6789/0 conn(0x7fc5e815e2a0 0x7fc5e80d07e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:38802/0 (socket says 172.21.15.67:38802) 2024-03-24T00:40:26.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.787+0000 7fc5e6d9d640 1 -- 172.21.15.67:0/353450871 >> v2:172.21.15.67:3300/0 conn(0x7fc5e815dec0 msgr2=0x7fc5e80ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:26.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.787+0000 7fc5e6d9d640 1 --2- 172.21.15.67:0/353450871 >> v2:172.21.15.67:3300/0 conn(0x7fc5e815dec0 0x7fc5e80ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:26.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.787+0000 7fc5e6d9d640 1 -- 172.21.15.67:0/353450871 >> v1:172.21.15.72:6789/0 conn(0x7fc5e815e2a0 legacy=0x7fc5e80d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:26.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.787+0000 7fc5e6d9d640 1 -- 172.21.15.67:0/353450871 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e80cf080 con 0x7fc5e80d0ef0 2024-03-24T00:40:26.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.787+0000 7fc5e659c640 1 --2- 172.21.15.67:0/353450871 >> v2:172.21.15.67:3300/0 conn(0x7fc5e815dec0 0x7fc5e80ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:40:26.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.787+0000 7fc5e559a640 1 -- 172.21.15.67:0/353450871 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 19176941 0 0) 0x7fc5d0001110 con 0x7fc5e815e2a0 2024-03-24T00:40:26.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.787+0000 7fc5e6d9d640 1 --2- 172.21.15.67:0/353450871 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc5e80d0ef0 0x7fc5e8162630 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fc5e80cae20 tx=0x7fc5d802be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5171e6ccd4097bc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:26.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.788+0000 7fc5e559a640 1 -- 172.21.15.67:0/353450871 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d8039070 con 0x7fc5e80d0ef0 2024-03-24T00:40:26.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.788+0000 7fc5e559a640 1 -- 172.21.15.67:0/353450871 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5d802cbf0 con 0x7fc5e80d0ef0 2024-03-24T00:40:26.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.788+0000 7fc5e559a640 1 -- 172.21.15.67:0/353450871 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d8034440 con 0x7fc5e80d0ef0 2024-03-24T00:40:26.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.788+0000 7fc5eca59640 1 -- 172.21.15.67:0/353450871 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc5e80d0ef0 msgr2=0x7fc5e8162630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:26.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.788+0000 7fc5eca59640 1 --2- 172.21.15.67:0/353450871 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc5e80d0ef0 0x7fc5e8162630 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fc5e80cae20 tx=0x7fc5d802be40 comp rx=0 tx=0).stop 2024-03-24T00:40:26.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.788+0000 7fc5eca59640 1 -- 172.21.15.67:0/353450871 shutdown_connections 2024-03-24T00:40:26.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.788+0000 7fc5eca59640 1 --2- 172.21.15.67:0/353450871 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc5e80d0ef0 0x7fc5e8162630 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:26.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.788+0000 7fc5eca59640 1 --2- 172.21.15.67:0/353450871 >> v2:172.21.15.67:3300/0 conn(0x7fc5e815dec0 0x7fc5e80ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:26.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.788+0000 7fc5eca59640 1 -- 172.21.15.67:0/353450871 >> 172.21.15.67:0/353450871 conn(0x7fc5e806c810 msgr2=0x7fc5e806cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:26.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.788+0000 7fc5eca59640 1 -- 172.21.15.67:0/353450871 shutdown_connections 2024-03-24T00:40:26.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.788+0000 7fc5eca59640 1 -- 172.21.15.67:0/353450871 wait complete. 2024-03-24T00:40:26.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.790+0000 7fc5eca59640 1 Processor -- start 2024-03-24T00:40:26.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.790+0000 7fc5eca59640 1 -- start start 2024-03-24T00:40:26.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.790+0000 7fc5eca59640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc5e80d0ef0 0x7fc5e80aa720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:26.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.790+0000 7fc5eca59640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5e815dec0 0x7fc5e80c9f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:26.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.790+0000 7fc5eca59640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc5e815e2a0 0x7fc5e815ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:26.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.790+0000 7fc5eca59640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e80d2730 con 0x7fc5e815dec0 2024-03-24T00:40:26.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.790+0000 7fc5eca59640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e80d25b0 con 0x7fc5e80d0ef0 2024-03-24T00:40:26.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.790+0000 7fc5eca59640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e80d28b0 con 0x7fc5e815e2a0 2024-03-24T00:40:26.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.790+0000 7fc5e5d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5e815dec0 0x7fc5e80c9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:26.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.790+0000 7fc5e659c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc5e80d0ef0 0x7fc5e80aa720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:26.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.790+0000 7fc5e5d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5e815dec0 0x7fc5e80c9f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39282/0 (socket says 172.21.15.67:39282) 2024-03-24T00:40:26.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.791+0000 7fc5e5d9b640 1 -- 172.21.15.67:0/3711792756 learned_addr learned my addr 172.21.15.67:0/3711792756 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:26.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.791+0000 7fc5e6d9d640 1 --2- 172.21.15.67:0/3711792756 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc5e815e2a0 0x7fc5e815ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:26.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.791+0000 7fc5e659c640 1 -- 172.21.15.67:0/3711792756 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc5e815e2a0 msgr2=0x7fc5e815ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:26.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.791+0000 7fc5e659c640 1 --2- 172.21.15.67:0/3711792756 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc5e815e2a0 0x7fc5e815ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:26.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.791+0000 7fc5e659c640 1 -- 172.21.15.67:0/3711792756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5e815dec0 msgr2=0x7fc5e80c9f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:26.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.791+0000 7fc5e659c640 1 --2- 172.21.15.67:0/3711792756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5e815dec0 0x7fc5e80c9f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:26.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.791+0000 7fc5e659c640 1 -- 172.21.15.67:0/3711792756 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e81e82b0 con 0x7fc5e80d0ef0 2024-03-24T00:40:26.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.791+0000 7fc5e659c640 1 --2- 172.21.15.67:0/3711792756 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc5e80d0ef0 0x7fc5e80aa720 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fc5e000e900 tx=0x7fc5e000edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:26.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.791+0000 7fc5deffd640 1 -- 172.21.15.67:0/3711792756 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e000ca00 con 0x7fc5e80d0ef0 2024-03-24T00:40:26.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.791+0000 7fc5deffd640 1 -- 172.21.15.67:0/3711792756 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5e000cba0 con 0x7fc5e80d0ef0 2024-03-24T00:40:26.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.791+0000 7fc5eca59640 1 -- 172.21.15.67:0/3711792756 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5e81e7cf0 con 0x7fc5e80d0ef0 2024-03-24T00:40:26.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.791+0000 7fc5deffd640 1 -- 172.21.15.67:0/3711792756 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e0004030 con 0x7fc5e80d0ef0 2024-03-24T00:40:26.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.791+0000 7fc5eca59640 1 -- 172.21.15.67:0/3711792756 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5e8149fd0 con 0x7fc5e80d0ef0 2024-03-24T00:40:26.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.792+0000 7fc5eca59640 1 -- 172.21.15.67:0/3711792756 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5e804f400 con 0x7fc5e80d0ef0 2024-03-24T00:40:26.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.793+0000 7fc5deffd640 1 -- 172.21.15.67:0/3711792756 <== mon.1 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc5e00041d0 con 0x7fc5e80d0ef0 2024-03-24T00:40:26.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.793+0000 7fc5deffd640 1 --2- 172.21.15.67:0/3711792756 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc5c806bbd0 0x7fc5c806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:26.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.793+0000 7fc5deffd640 1 -- 172.21.15.67:0/3711792756 <== mon.1 v2:172.21.15.72:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc5e008aa30 con 0x7fc5e80d0ef0 2024-03-24T00:40:26.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.793+0000 7fc5e5d9b640 1 --2- 172.21.15.67:0/3711792756 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc5c806bbd0 0x7fc5c806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:26.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.794+0000 7fc5e5d9b640 1 --2- 172.21.15.67:0/3711792756 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc5c806bbd0 0x7fc5c806e090 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fc5e81f7820 tx=0x7fc5d0007e70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:26.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.798+0000 7fc5deffd640 1 -- 172.21.15.67:0/3711792756 <== mon.1 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc5e0017020 con 0x7fc5e80d0ef0 2024-03-24T00:40:26.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.984+0000 7fc5eca59640 1 -- 172.21.15.67:0/3711792756 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc5e80b8880 con 0x7fc5e80d0ef0 2024-03-24T00:40:26.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.986+0000 7fc5deffd640 1 -- 172.21.15.67:0/3711792756 <== mon.1 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) v1 ==== 74+0+16882 (secure 0 0 0) 0x7fc5e0058dd0 con 0x7fc5e80d0ef0 2024-03-24T00:40:26.986 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:40:26.987 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":39,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:40:02.323736+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:39:58.975995+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-24T00:39:59.653615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:40:26.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.988+0000 7fc5eca59640 1 -- 172.21.15.67:0/3711792756 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc5c806bbd0 msgr2=0x7fc5c806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:26.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.988+0000 7fc5eca59640 1 --2- 172.21.15.67:0/3711792756 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc5c806bbd0 0x7fc5c806e090 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fc5e81f7820 tx=0x7fc5d0007e70 comp rx=0 tx=0).stop 2024-03-24T00:40:26.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.988+0000 7fc5eca59640 1 -- 172.21.15.67:0/3711792756 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc5e80d0ef0 msgr2=0x7fc5e80aa720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:26.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.988+0000 7fc5eca59640 1 --2- 172.21.15.67:0/3711792756 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc5e80d0ef0 0x7fc5e80aa720 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fc5e000e900 tx=0x7fc5e000edc0 comp rx=0 tx=0).stop 2024-03-24T00:40:26.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.988+0000 7fc5eca59640 1 -- 172.21.15.67:0/3711792756 shutdown_connections 2024-03-24T00:40:26.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.988+0000 7fc5eca59640 1 --2- 172.21.15.67:0/3711792756 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc5c806bbd0 0x7fc5c806e090 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:26.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.988+0000 7fc5eca59640 1 --2- 172.21.15.67:0/3711792756 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc5e815e2a0 0x7fc5e815ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:26.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.988+0000 7fc5eca59640 1 --2- 172.21.15.67:0/3711792756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5e815dec0 0x7fc5e80c9f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:26.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.988+0000 7fc5eca59640 1 --2- 172.21.15.67:0/3711792756 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc5e80d0ef0 0x7fc5e80aa720 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:26.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.988+0000 7fc5eca59640 1 -- 172.21.15.67:0/3711792756 >> 172.21.15.67:0/3711792756 conn(0x7fc5e806c810 msgr2=0x7fc5e815fcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:26.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.988+0000 7fc5eca59640 1 -- 172.21.15.67:0/3711792756 shutdown_connections 2024-03-24T00:40:26.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:26.988+0000 7fc5eca59640 1 -- 172.21.15.67:0/3711792756 wait complete. 2024-03-24T00:40:27.000 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:38:39.364950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:39:58.975995+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:39:59.653615+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:40:27.000 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:40:27.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.103+0000 7f65c89d3640 1 Processor -- start 2024-03-24T00:40:27.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.104+0000 7f65c89d3640 1 -- start start 2024-03-24T00:40:27.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.104+0000 7f65c89d3640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f65c40b0f10 0x7f65c40d1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:27.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.104+0000 7f65c89d3640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f65c40b2610 0x7f65c415d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:27.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.104+0000 7f65c89d3640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65c40b12f0 con 0x7f65c40d20d0 2024-03-24T00:40:27.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.104+0000 7f65c89d3640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65c40d2540 con 0x7f65c40b0f10 2024-03-24T00:40:27.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.104+0000 7f65c89d3640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65c40d09f0 con 0x7f65c40b2610 2024-03-24T00:40:27.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.104+0000 7f65c259c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f65c40b0f10 0x7f65c40d1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:27.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.104+0000 7f65c259c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f65c40b0f10 0x7f65c40d1b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39294/0 (socket says 172.21.15.67:39294) 2024-03-24T00:40:27.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.105+0000 7f65c259c640 1 -- 172.21.15.67:0/2872061684 learned_addr learned my addr 172.21.15.67:0/2872061684 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:27.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.105+0000 7f65c2d9d640 1 --2- 172.21.15.67:0/2872061684 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f65c40b2610 0x7f65c415d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:27.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.105+0000 7f65c259c640 1 -- 172.21.15.67:0/2872061684 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f65c40b2610 msgr2=0x7f65c415d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:27.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.105+0000 7f65c259c640 1 --2- 172.21.15.67:0/2872061684 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f65c40b2610 0x7f65c415d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:27.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.105+0000 7f65c259c640 1 -- 172.21.15.67:0/2872061684 >> v1:172.21.15.72:6789/0 conn(0x7f65c40d20d0 legacy=0x7f65c40b1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:27.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.105+0000 7f65c259c640 1 -- 172.21.15.67:0/2872061684 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65c40b2c50 con 0x7f65c40b0f10 2024-03-24T00:40:27.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.105+0000 7f65c1d9b640 1 -- 172.21.15.67:0/2872061684 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2804035624 0 0) 0x7f65c40b12f0 con 0x7f65c40d20d0 2024-03-24T00:40:27.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.106+0000 7f65c259c640 1 --2- 172.21.15.67:0/2872061684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65c40b0f10 0x7f65c40d1b90 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f65b4009870 tx=0x7f65b402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a1da5441824e447 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:27.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.106+0000 7f65c1d9b640 1 -- 172.21.15.67:0/2872061684 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b4034070 con 0x7f65c40b0f10 2024-03-24T00:40:27.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.106+0000 7f65c1d9b640 1 -- 172.21.15.67:0/2872061684 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65b4030840 con 0x7f65c40b0f10 2024-03-24T00:40:27.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.106+0000 7f65c1d9b640 1 -- 172.21.15.67:0/2872061684 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b4030c10 con 0x7f65c40b0f10 2024-03-24T00:40:27.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.106+0000 7f65c89d3640 1 -- 172.21.15.67:0/2872061684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65c40b0f10 msgr2=0x7f65c40d1b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:27.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.106+0000 7f65c89d3640 1 --2- 172.21.15.67:0/2872061684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65c40b0f10 0x7f65c40d1b90 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f65b4009870 tx=0x7f65b402b980 comp rx=0 tx=0).stop 2024-03-24T00:40:27.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.106+0000 7f65c89d3640 1 -- 172.21.15.67:0/2872061684 shutdown_connections 2024-03-24T00:40:27.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.106+0000 7f65c89d3640 1 --2- 172.21.15.67:0/2872061684 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f65c40b2610 0x7f65c415d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:27.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.106+0000 7f65c89d3640 1 --2- 172.21.15.67:0/2872061684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65c40b0f10 0x7f65c40d1b90 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:27.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.106+0000 7f65c89d3640 1 -- 172.21.15.67:0/2872061684 >> 172.21.15.67:0/2872061684 conn(0x7f65c408b1b0 msgr2=0x7f65c408b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:27.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.106+0000 7f65c89d3640 1 -- 172.21.15.67:0/2872061684 shutdown_connections 2024-03-24T00:40:27.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.106+0000 7f65c89d3640 1 -- 172.21.15.67:0/2872061684 wait complete. 2024-03-24T00:40:27.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.108+0000 7f65c89d3640 1 Processor -- start 2024-03-24T00:40:27.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.108+0000 7f65c89d3640 1 -- start start 2024-03-24T00:40:27.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.108+0000 7f65c89d3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65c40b0f10 0x7f65c414f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:27.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.108+0000 7f65c89d3640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f65c40b2610 0x7f65c40b5860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:27.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.108+0000 7f65c89d3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f65c40d20d0 0x7f65c40b5da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:27.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.108+0000 7f65c89d3640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65c40cee40 con 0x7f65c40b0f10 2024-03-24T00:40:27.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.108+0000 7f65c89d3640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65c40ceb60 con 0x7f65c40d20d0 2024-03-24T00:40:27.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.108+0000 7f65c89d3640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65c40ce560 con 0x7f65c40b2610 2024-03-24T00:40:27.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.108+0000 7f65c259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65c40b0f10 0x7f65c414f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:27.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.108+0000 7f65c259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65c40b0f10 0x7f65c414f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39298/0 (socket says 172.21.15.67:39298) 2024-03-24T00:40:27.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.108+0000 7f65c259c640 1 -- 172.21.15.67:0/1795325838 learned_addr learned my addr 172.21.15.67:0/1795325838 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:27.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.108+0000 7f65bbfff640 1 --2- 172.21.15.67:0/1795325838 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f65c40b2610 0x7f65c40b5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:27.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.108+0000 7f65c2d9d640 1 --2- 172.21.15.67:0/1795325838 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f65c40d20d0 0x7f65c40b5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:27.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.109+0000 7f65c259c640 1 -- 172.21.15.67:0/1795325838 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f65c40b2610 msgr2=0x7f65c40b5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:27.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.109+0000 7f65c259c640 1 --2- 172.21.15.67:0/1795325838 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f65c40b2610 0x7f65c40b5860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:27.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.109+0000 7f65c259c640 1 -- 172.21.15.67:0/1795325838 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f65c40d20d0 msgr2=0x7f65c40b5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:27.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.109+0000 7f65c259c640 1 --2- 172.21.15.67:0/1795325838 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f65c40d20d0 0x7f65c40b5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:27.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.109+0000 7f65c259c640 1 -- 172.21.15.67:0/1795325838 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65c414f5b0 con 0x7f65c40b0f10 2024-03-24T00:40:27.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.109+0000 7f65c259c640 1 --2- 172.21.15.67:0/1795325838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65c40b0f10 0x7f65c414f070 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f65b4000c00 tx=0x7f65b4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:27.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.109+0000 7f65baffd640 1 -- 172.21.15.67:0/1795325838 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b4048070 con 0x7f65c40b0f10 2024-03-24T00:40:27.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.109+0000 7f65baffd640 1 -- 172.21.15.67:0/1795325838 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65b4030640 con 0x7f65c40b0f10 2024-03-24T00:40:27.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.109+0000 7f65c89d3640 1 -- 172.21.15.67:0/1795325838 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65c40b6420 con 0x7f65c40b0f10 2024-03-24T00:40:27.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.109+0000 7f65baffd640 1 -- 172.21.15.67:0/1795325838 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b4034040 con 0x7f65c40b0f10 2024-03-24T00:40:27.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.109+0000 7f65c89d3640 1 -- 172.21.15.67:0/1795325838 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65c40b69e0 con 0x7f65c40b0f10 2024-03-24T00:40:27.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.110+0000 7f65c89d3640 1 -- 172.21.15.67:0/1795325838 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65c404f400 con 0x7f65c40b0f10 2024-03-24T00:40:27.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.110+0000 7f65baffd640 1 -- 172.21.15.67:0/1795325838 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f65b404b070 con 0x7f65c40b0f10 2024-03-24T00:40:27.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.111+0000 7f65baffd640 1 --2- 172.21.15.67:0/1795325838 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f65a406bc00 0x7f65a406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:27.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.111+0000 7f65baffd640 1 -- 172.21.15.67:0/1795325838 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f65b40ac970 con 0x7f65c40b0f10 2024-03-24T00:40:27.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.111+0000 7f65bbfff640 1 --2- 172.21.15.67:0/1795325838 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f65a406bc00 0x7f65a406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:27.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.111+0000 7f65bbfff640 1 --2- 172.21.15.67:0/1795325838 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f65a406bc00 0x7f65a406e0c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f65c40d2000 tx=0x7f65ac009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:27.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.115+0000 7f65baffd640 1 -- 172.21.15.67:0/1795325838 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f65c414f5b0 con 0x7f65c40b0f10 2024-03-24T00:40:27.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.297+0000 7f65c89d3640 1 -- 172.21.15.67:0/1795325838 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f65c40b0430 con 0x7f65c40b0f10 2024-03-24T00:40:27.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.297+0000 7f65baffd640 1 -- 172.21.15.67:0/1795325838 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v39) v1 ==== 93+0+10 (secure 0 0 0) 0x7f65b407ae30 con 0x7f65c40b0f10 2024-03-24T00:40:27.298 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-24T00:40:27.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.299+0000 7f65c89d3640 1 -- 172.21.15.67:0/1795325838 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f65a406bc00 msgr2=0x7f65a406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:27.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.299+0000 7f65c89d3640 1 --2- 172.21.15.67:0/1795325838 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f65a406bc00 0x7f65a406e0c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f65c40d2000 tx=0x7f65ac009040 comp rx=0 tx=0).stop 2024-03-24T00:40:27.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.299+0000 7f65c89d3640 1 -- 172.21.15.67:0/1795325838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65c40b0f10 msgr2=0x7f65c414f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:27.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.299+0000 7f65c89d3640 1 --2- 172.21.15.67:0/1795325838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65c40b0f10 0x7f65c414f070 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f65b4000c00 tx=0x7f65b4004730 comp rx=0 tx=0).stop 2024-03-24T00:40:27.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.299+0000 7f65c89d3640 1 -- 172.21.15.67:0/1795325838 shutdown_connections 2024-03-24T00:40:27.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.299+0000 7f65c89d3640 1 --2- 172.21.15.67:0/1795325838 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f65a406bc00 0x7f65a406e0c0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:27.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.299+0000 7f65c89d3640 1 --2- 172.21.15.67:0/1795325838 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f65c40d20d0 0x7f65c40b5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:27.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.299+0000 7f65c89d3640 1 --2- 172.21.15.67:0/1795325838 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f65c40b2610 0x7f65c40b5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:27.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.299+0000 7f65c89d3640 1 --2- 172.21.15.67:0/1795325838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65c40b0f10 0x7f65c414f070 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:27.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.299+0000 7f65c89d3640 1 -- 172.21.15.67:0/1795325838 >> 172.21.15.67:0/1795325838 conn(0x7f65c408b1b0 msgr2=0x7f65c40cf7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:27.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.299+0000 7f65c89d3640 1 -- 172.21.15.67:0/1795325838 shutdown_connections 2024-03-24T00:40:27.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.299+0000 7f65c89d3640 1 -- 172.21.15.67:0/1795325838 wait complete. 2024-03-24T00:40:27.311 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:40:27.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.414+0000 7f8510bb3640 1 Processor -- start 2024-03-24T00:40:27.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.414+0000 7f8510bb3640 1 -- start start 2024-03-24T00:40:27.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.415+0000 7f8510bb3640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f850c15dfd0 0x7f850c0cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:27.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.415+0000 7f8510bb3640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f850c0b3d30 0x7f850c0cc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:27.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.415+0000 7f8510bb3640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f850c0cf090 con 0x7f850c0b41e0 2024-03-24T00:40:27.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.415+0000 7f8510bb3640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f850c0cc260 con 0x7f850c15dfd0 2024-03-24T00:40:27.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.415+0000 7f8510bb3640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f850c0d0320 con 0x7f850c0b3d30 2024-03-24T00:40:27.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.415+0000 7f850a59c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f850c15dfd0 0x7f850c0cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:27.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.415+0000 7f850a59c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f850c15dfd0 0x7f850c0cf980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39312/0 (socket says 172.21.15.67:39312) 2024-03-24T00:40:27.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.415+0000 7f850a59c640 1 -- 172.21.15.67:0/2000411513 learned_addr learned my addr 172.21.15.67:0/2000411513 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:27.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.415+0000 7f850ad9d640 1 --2- 172.21.15.67:0/2000411513 >> v2:172.21.15.72:3300/0 conn(0x7f850c0b3d30 0x7f850c0cc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:27.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.415+0000 7f850a59c640 1 -- 172.21.15.67:0/2000411513 >> v2:172.21.15.72:3300/0 conn(0x7f850c0b3d30 msgr2=0x7f850c0cc6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:27.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.415+0000 7f850a59c640 1 --2- 172.21.15.67:0/2000411513 >> v2:172.21.15.72:3300/0 conn(0x7f850c0b3d30 0x7f850c0cc6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:27.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.415+0000 7f850a59c640 1 -- 172.21.15.67:0/2000411513 >> v1:172.21.15.72:6789/0 conn(0x7f850c0b41e0 legacy=0x7f850c0b3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:27.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.415+0000 7f850a59c640 1 -- 172.21.15.67:0/2000411513 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f850c0d0600 con 0x7f850c15dfd0 2024-03-24T00:40:27.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.416+0000 7f850a59c640 1 --2- 172.21.15.67:0/2000411513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f850c15dfd0 0x7f850c0cf980 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f8504009870 tx=0x7f850402ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52de2b91a4856828 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:27.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.416+0000 7f850959a640 1 -- 172.21.15.67:0/2000411513 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f850402f840 con 0x7f850c15dfd0 2024-03-24T00:40:27.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.416+0000 7f850959a640 1 -- 172.21.15.67:0/2000411513 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f850402f9e0 con 0x7f850c15dfd0 2024-03-24T00:40:27.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.416+0000 7f850959a640 1 -- 172.21.15.67:0/2000411513 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f850402fdb0 con 0x7f850c15dfd0 2024-03-24T00:40:27.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.417+0000 7f8510bb3640 1 -- 172.21.15.67:0/2000411513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f850c15dfd0 msgr2=0x7f850c0cf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:27.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.417+0000 7f8510bb3640 1 --2- 172.21.15.67:0/2000411513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f850c15dfd0 0x7f850c0cf980 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f8504009870 tx=0x7f850402ba20 comp rx=0 tx=0).stop 2024-03-24T00:40:27.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.417+0000 7f8510bb3640 1 -- 172.21.15.67:0/2000411513 shutdown_connections 2024-03-24T00:40:27.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.417+0000 7f8510bb3640 1 --2- 172.21.15.67:0/2000411513 >> v2:172.21.15.72:3300/0 conn(0x7f850c0b3d30 0x7f850c0cc6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:27.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.417+0000 7f8510bb3640 1 --2- 172.21.15.67:0/2000411513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f850c15dfd0 0x7f850c0cf980 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:27.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.417+0000 7f8510bb3640 1 -- 172.21.15.67:0/2000411513 >> 172.21.15.67:0/2000411513 conn(0x7f850c06fcf0 msgr2=0x7f850c0700d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:27.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.417+0000 7f8510bb3640 1 -- 172.21.15.67:0/2000411513 shutdown_connections 2024-03-24T00:40:27.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.417+0000 7f8510bb3640 1 -- 172.21.15.67:0/2000411513 wait complete. 2024-03-24T00:40:27.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.418+0000 7f8510bb3640 1 Processor -- start 2024-03-24T00:40:27.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.418+0000 7f8510bb3640 1 -- start start 2024-03-24T00:40:27.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.419+0000 7f8510bb3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f850c0b3d30 0x7f850c15ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:27.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.419+0000 7f8510bb3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f850c0b41e0 0x7f850c14c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:27.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.419+0000 7f8510bb3640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f850c15dfd0 0x7f850c0cbb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:27.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.419+0000 7f8510bb3640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f850c0cfec0 con 0x7f850c0b3d30 2024-03-24T00:40:27.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.419+0000 7f8510bb3640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f850c0cedb0 con 0x7f850c0b41e0 2024-03-24T00:40:27.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.419+0000 7f8510bb3640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f850c15e3b0 con 0x7f850c15dfd0 2024-03-24T00:40:27.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.419+0000 7f850a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f850c0b3d30 0x7f850c15ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:27.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.419+0000 7f850a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f850c0b3d30 0x7f850c15ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39314/0 (socket says 172.21.15.67:39314) 2024-03-24T00:40:27.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.419+0000 7f850a59c640 1 -- 172.21.15.67:0/1151487039 learned_addr learned my addr 172.21.15.67:0/1151487039 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:27.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.419+0000 7f8509d9b640 1 --2- 172.21.15.67:0/1151487039 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f850c0b41e0 0x7f850c14c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:27.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.419+0000 7f850ad9d640 1 --2- 172.21.15.67:0/1151487039 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f850c15dfd0 0x7f850c0cbb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:27.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.419+0000 7f850a59c640 1 -- 172.21.15.67:0/1151487039 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f850c15dfd0 msgr2=0x7f850c0cbb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:27.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.419+0000 7f850a59c640 1 --2- 172.21.15.67:0/1151487039 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f850c15dfd0 0x7f850c0cbb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:27.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.419+0000 7f850a59c640 1 -- 172.21.15.67:0/1151487039 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f850c0b41e0 msgr2=0x7f850c14c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:27.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.419+0000 7f850a59c640 1 --2- 172.21.15.67:0/1151487039 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f850c0b41e0 0x7f850c14c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:27.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.419+0000 7f850a59c640 1 -- 172.21.15.67:0/1151487039 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f850c0cc060 con 0x7f850c0b3d30 2024-03-24T00:40:27.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.419+0000 7f8509d9b640 1 --2- 172.21.15.67:0/1151487039 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f850c0b41e0 0x7f850c14c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:40:27.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.420+0000 7f850a59c640 1 --2- 172.21.15.67:0/1151487039 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f850c0b3d30 0x7f850c15ef70 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f850402ba60 tx=0x7f8504004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:27.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.420+0000 7f8502ffd640 1 -- 172.21.15.67:0/1151487039 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8504047070 con 0x7f850c0b3d30 2024-03-24T00:40:27.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.420+0000 7f8502ffd640 1 -- 172.21.15.67:0/1151487039 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8504037bc0 con 0x7f850c0b3d30 2024-03-24T00:40:27.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.420+0000 7f8510bb3640 1 -- 172.21.15.67:0/1151487039 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f850c15f600 con 0x7f850c0b3d30 2024-03-24T00:40:27.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.420+0000 7f8502ffd640 1 -- 172.21.15.67:0/1151487039 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8504042480 con 0x7f850c0b3d30 2024-03-24T00:40:27.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.420+0000 7f8510bb3640 1 -- 172.21.15.67:0/1151487039 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f850c15fbc0 con 0x7f850c0b3d30 2024-03-24T00:40:27.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.421+0000 7f8510bb3640 1 -- 172.21.15.67:0/1151487039 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f850c04f400 con 0x7f850c0b3d30 2024-03-24T00:40:27.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.421+0000 7f8502ffd640 1 -- 172.21.15.67:0/1151487039 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f850404a070 con 0x7f850c0b3d30 2024-03-24T00:40:27.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.422+0000 7f8502ffd640 1 --2- 172.21.15.67:0/1151487039 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f84ec06bbd0 0x7f84ec06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:27.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.422+0000 7f8502ffd640 1 -- 172.21.15.67:0/1151487039 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f85040accf0 con 0x7f850c0b3d30 2024-03-24T00:40:27.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.422+0000 7f8509d9b640 1 --2- 172.21.15.67:0/1151487039 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f84ec06bbd0 0x7f84ec06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:27.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.422+0000 7f8509d9b640 1 --2- 172.21.15.67:0/1151487039 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f84ec06bbd0 0x7f84ec06e090 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f850c0c8160 tx=0x7f84f4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:27.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.426+0000 7f8502ffd640 1 -- 172.21.15.67:0/1151487039 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f85040b1070 con 0x7f850c0b3d30 2024-03-24T00:40:27.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.607+0000 7f8510bb3640 1 -- 172.21.15.67:0/1151487039 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f850c0b7b20 con 0x7f850c0b3d30 2024-03-24T00:40:27.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.607+0000 7f8502ffd640 1 -- 172.21.15.67:0/1151487039 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v39) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8504033030 con 0x7f850c0b3d30 2024-03-24T00:40:27.608 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-24T00:40:27.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.609+0000 7f8510bb3640 1 -- 172.21.15.67:0/1151487039 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f84ec06bbd0 msgr2=0x7f84ec06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:27.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.609+0000 7f8510bb3640 1 --2- 172.21.15.67:0/1151487039 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f84ec06bbd0 0x7f84ec06e090 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f850c0c8160 tx=0x7f84f4009040 comp rx=0 tx=0).stop 2024-03-24T00:40:27.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.609+0000 7f8510bb3640 1 -- 172.21.15.67:0/1151487039 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f850c0b3d30 msgr2=0x7f850c15ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:27.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.609+0000 7f8510bb3640 1 --2- 172.21.15.67:0/1151487039 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f850c0b3d30 0x7f850c15ef70 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f850402ba60 tx=0x7f8504004730 comp rx=0 tx=0).stop 2024-03-24T00:40:27.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.609+0000 7f8510bb3640 1 -- 172.21.15.67:0/1151487039 shutdown_connections 2024-03-24T00:40:27.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.609+0000 7f8510bb3640 1 --2- 172.21.15.67:0/1151487039 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f84ec06bbd0 0x7f84ec06e090 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:27.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.609+0000 7f8510bb3640 1 --2- 172.21.15.67:0/1151487039 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f850c15dfd0 0x7f850c0cbb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:27.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.609+0000 7f8510bb3640 1 --2- 172.21.15.67:0/1151487039 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f850c0b41e0 0x7f850c14c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:27.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.609+0000 7f8510bb3640 1 --2- 172.21.15.67:0/1151487039 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f850c0b3d30 0x7f850c15ef70 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:27.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.609+0000 7f8510bb3640 1 -- 172.21.15.67:0/1151487039 >> 172.21.15.67:0/1151487039 conn(0x7f850c06fcf0 msgr2=0x7f850c160e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:27.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.609+0000 7f8510bb3640 1 -- 172.21.15.67:0/1151487039 shutdown_connections 2024-03-24T00:40:27.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.609+0000 7f8510bb3640 1 -- 172.21.15.67:0/1151487039 wait complete. 2024-03-24T00:40:27.621 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:40:27.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.723+0000 7f3f8f7f4640 1 Processor -- start 2024-03-24T00:40:27.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.724+0000 7f3f8f7f4640 1 -- start start 2024-03-24T00:40:27.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.724+0000 7f3f8f7f4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3f880d3ae0 0x7f3f880b5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:27.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.724+0000 7f3f8f7f4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3f8815e280 0x7f3f880b5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:27.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.724+0000 7f3f8f7f4640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f880d2250 con 0x7f3f8815dea0 2024-03-24T00:40:27.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.724+0000 7f3f8f7f4640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f880d2530 con 0x7f3f880d3ae0 2024-03-24T00:40:27.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.724+0000 7f3f8f7f4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f880d2810 con 0x7f3f8815e280 2024-03-24T00:40:27.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.725+0000 7f3f8cd8f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3f880d3ae0 0x7f3f880b5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:27.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.725+0000 7f3f8dd91640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3f8815e280 0x7f3f880b5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:27.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.725+0000 7f3f8dd91640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3f8815e280 0x7f3f880b5fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:51628/0 (socket says 172.21.15.67:51628) 2024-03-24T00:40:27.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.725+0000 7f3f8cd8f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3f880d3ae0 0x7f3f880b5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39318/0 (socket says 172.21.15.67:39318) 2024-03-24T00:40:27.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.725+0000 7f3f8dd91640 1 -- 172.21.15.67:0/424588674 learned_addr learned my addr 172.21.15.67:0/424588674 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:27.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.725+0000 7f3f8cd8f640 1 -- 172.21.15.67:0/424588674 >> v2:172.21.15.72:3300/0 conn(0x7f3f8815e280 msgr2=0x7f3f880b5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:27.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.725+0000 7f3f8cd8f640 1 --2- 172.21.15.67:0/424588674 >> v2:172.21.15.72:3300/0 conn(0x7f3f8815e280 0x7f3f880b5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:27.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.725+0000 7f3f8cd8f640 1 -- 172.21.15.67:0/424588674 >> v1:172.21.15.72:6789/0 conn(0x7f3f8815dea0 legacy=0x7f3f880d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:27.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.725+0000 7f3f8cd8f640 1 -- 172.21.15.67:0/424588674 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f880d2af0 con 0x7f3f880d3ae0 2024-03-24T00:40:27.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.725+0000 7f3f7bfff640 1 -- 172.21.15.67:0/424588674 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4264749730 0 0) 0x7f3f880d2250 con 0x7f3f8815dea0 2024-03-24T00:40:27.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.725+0000 7f3f8dd91640 1 --2- 172.21.15.67:0/424588674 >> v2:172.21.15.72:3300/0 conn(0x7f3f8815e280 0x7f3f880b5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:40:27.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.726+0000 7f3f8cd8f640 1 --2- 172.21.15.67:0/424588674 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f880d3ae0 0x7f3f880b5a60 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c0097a0 tx=0x7f3f7c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43f0f1af6f53b5df server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:27.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.726+0000 7f3f7bfff640 1 -- 172.21.15.67:0/424588674 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f7c0315a0 con 0x7f3f880d3ae0 2024-03-24T00:40:27.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.726+0000 7f3f7bfff640 1 -- 172.21.15.67:0/424588674 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f7c037070 con 0x7f3f880d3ae0 2024-03-24T00:40:27.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.726+0000 7f3f7bfff640 1 -- 172.21.15.67:0/424588674 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f7c031960 con 0x7f3f880d3ae0 2024-03-24T00:40:27.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.726+0000 7f3f8f7f4640 1 -- 172.21.15.67:0/424588674 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f880d3ae0 msgr2=0x7f3f880b5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:27.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.726+0000 7f3f8f7f4640 1 --2- 172.21.15.67:0/424588674 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f880d3ae0 0x7f3f880b5a60 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c0097a0 tx=0x7f3f7c02b840 comp rx=0 tx=0).stop 2024-03-24T00:40:27.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.726+0000 7f3f8f7f4640 1 -- 172.21.15.67:0/424588674 shutdown_connections 2024-03-24T00:40:27.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.726+0000 7f3f8f7f4640 1 --2- 172.21.15.67:0/424588674 >> v2:172.21.15.72:3300/0 conn(0x7f3f8815e280 0x7f3f880b5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:27.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.726+0000 7f3f8f7f4640 1 --2- 172.21.15.67:0/424588674 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f880d3ae0 0x7f3f880b5a60 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:27.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.726+0000 7f3f8f7f4640 1 -- 172.21.15.67:0/424588674 >> 172.21.15.67:0/424588674 conn(0x7f3f88072da0 msgr2=0x7f3f880d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:27.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.726+0000 7f3f8f7f4640 1 -- 172.21.15.67:0/424588674 shutdown_connections 2024-03-24T00:40:27.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.726+0000 7f3f8f7f4640 1 -- 172.21.15.67:0/424588674 wait complete. 2024-03-24T00:40:27.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.727+0000 7f3f8f7f4640 1 Processor -- start 2024-03-24T00:40:27.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.728+0000 7f3f8f7f4640 1 -- start start 2024-03-24T00:40:27.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.728+0000 7f3f8f7f4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3f880d3ae0 0x7f3f880aa990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:27.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.728+0000 7f3f8f7f4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f8815dea0 0x7f3f8815ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:27.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.728+0000 7f3f8f7f4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f8815e280 0x7f3f880b3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:27.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.728+0000 7f3f8f7f4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f880b7480 con 0x7f3f8815dea0 2024-03-24T00:40:27.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.728+0000 7f3f8f7f4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f880b7300 con 0x7f3f8815e280 2024-03-24T00:40:27.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.728+0000 7f3f8f7f4640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f880b7600 con 0x7f3f880d3ae0 2024-03-24T00:40:27.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.728+0000 7f3f8cd8f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f8815dea0 0x7f3f8815ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:27.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.728+0000 7f3f8cd8f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f8815dea0 0x7f3f8815ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39328/0 (socket says 172.21.15.67:39328) 2024-03-24T00:40:27.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.728+0000 7f3f8cd8f640 1 -- 172.21.15.67:0/2337688390 learned_addr learned my addr 172.21.15.67:0/2337688390 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:27.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.728+0000 7f3f8d590640 1 --2- 172.21.15.67:0/2337688390 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3f880d3ae0 0x7f3f880aa990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:27.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.728+0000 7f3f8dd91640 1 --2- 172.21.15.67:0/2337688390 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f8815e280 0x7f3f880b3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:27.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.729+0000 7f3f8cd8f640 1 -- 172.21.15.67:0/2337688390 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3f880d3ae0 msgr2=0x7f3f880aa990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:27.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.729+0000 7f3f8cd8f640 1 --2- 172.21.15.67:0/2337688390 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3f880d3ae0 0x7f3f880aa990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:27.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.729+0000 7f3f8cd8f640 1 -- 172.21.15.67:0/2337688390 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f8815e280 msgr2=0x7f3f880b3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:27.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.729+0000 7f3f8cd8f640 1 --2- 172.21.15.67:0/2337688390 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f8815e280 0x7f3f880b3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:27.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.729+0000 7f3f8cd8f640 1 -- 172.21.15.67:0/2337688390 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f8815d3d0 con 0x7f3f8815dea0 2024-03-24T00:40:27.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.729+0000 7f3f8cd8f640 1 --2- 172.21.15.67:0/2337688390 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f8815dea0 0x7f3f8815ce90 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c000c00 tx=0x7f3f7c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:27.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.729+0000 7f3f79ffb640 1 -- 172.21.15.67:0/2337688390 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f7c032630 con 0x7f3f8815dea0 2024-03-24T00:40:27.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.729+0000 7f3f79ffb640 1 -- 172.21.15.67:0/2337688390 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f7c0327d0 con 0x7f3f8815dea0 2024-03-24T00:40:27.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.729+0000 7f3f8f7f4640 1 -- 172.21.15.67:0/2337688390 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f881f4be0 con 0x7f3f8815dea0 2024-03-24T00:40:27.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.729+0000 7f3f79ffb640 1 -- 172.21.15.67:0/2337688390 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f7c032bc0 con 0x7f3f8815dea0 2024-03-24T00:40:27.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.729+0000 7f3f8f7f4640 1 -- 172.21.15.67:0/2337688390 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f881f5420 con 0x7f3f8815dea0 2024-03-24T00:40:27.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.730+0000 7f3f8f7f4640 1 -- 172.21.15.67:0/2337688390 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f8804f400 con 0x7f3f8815dea0 2024-03-24T00:40:27.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.731+0000 7f3f79ffb640 1 -- 172.21.15.67:0/2337688390 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3f7c04d3f0 con 0x7f3f8815dea0 2024-03-24T00:40:27.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.731+0000 7f3f79ffb640 1 --2- 172.21.15.67:0/2337688390 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3f6c06bbd0 0x7f3f6c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:27.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.731+0000 7f3f79ffb640 1 -- 172.21.15.67:0/2337688390 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3f7c02f030 con 0x7f3f8815dea0 2024-03-24T00:40:27.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.731+0000 7f3f8d590640 1 --2- 172.21.15.67:0/2337688390 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3f6c06bbd0 0x7f3f6c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:27.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.731+0000 7f3f8d590640 1 --2- 172.21.15.67:0/2337688390 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3f6c06bbd0 0x7f3f6c06e090 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f3f84000f60 tx=0x7f3f84009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:27.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.735+0000 7f3f79ffb640 1 -- 172.21.15.67:0/2337688390 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3f881f4be0 con 0x7f3f8815dea0 2024-03-24T00:40:27.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.917+0000 7f3f8f7f4640 1 -- 172.21.15.67:0/2337688390 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3f8806b990 con 0x7f3f8815dea0 2024-03-24T00:40:27.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.917+0000 7f3f79ffb640 1 -- 172.21.15.67:0/2337688390 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v39) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3f7c040070 con 0x7f3f8815dea0 2024-03-24T00:40:27.918 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-03-24T00:40:27.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.919+0000 7f3f8f7f4640 1 -- 172.21.15.67:0/2337688390 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3f6c06bbd0 msgr2=0x7f3f6c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:27.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.919+0000 7f3f8f7f4640 1 --2- 172.21.15.67:0/2337688390 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3f6c06bbd0 0x7f3f6c06e090 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f3f84000f60 tx=0x7f3f84009040 comp rx=0 tx=0).stop 2024-03-24T00:40:27.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.919+0000 7f3f8f7f4640 1 -- 172.21.15.67:0/2337688390 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f8815dea0 msgr2=0x7f3f8815ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:27.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.919+0000 7f3f8f7f4640 1 --2- 172.21.15.67:0/2337688390 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f8815dea0 0x7f3f8815ce90 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c000c00 tx=0x7f3f7c0046c0 comp rx=0 tx=0).stop 2024-03-24T00:40:27.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.919+0000 7f3f8f7f4640 1 -- 172.21.15.67:0/2337688390 shutdown_connections 2024-03-24T00:40:27.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.919+0000 7f3f8f7f4640 1 --2- 172.21.15.67:0/2337688390 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3f6c06bbd0 0x7f3f6c06e090 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:27.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.919+0000 7f3f8f7f4640 1 --2- 172.21.15.67:0/2337688390 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f8815e280 0x7f3f880b3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:27.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.919+0000 7f3f8f7f4640 1 --2- 172.21.15.67:0/2337688390 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f8815dea0 0x7f3f8815ce90 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:27.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.919+0000 7f3f8f7f4640 1 --2- 172.21.15.67:0/2337688390 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3f880d3ae0 0x7f3f880aa990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:27.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.919+0000 7f3f8f7f4640 1 -- 172.21.15.67:0/2337688390 >> 172.21.15.67:0/2337688390 conn(0x7f3f88072da0 msgr2=0x7f3f880b6950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:27.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.919+0000 7f3f8f7f4640 1 -- 172.21.15.67:0/2337688390 shutdown_connections 2024-03-24T00:40:27.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:27.919+0000 7f3f8f7f4640 1 -- 172.21.15.67:0/2337688390 wait complete. 2024-03-24T00:40:27.931 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:40:28.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.031+0000 7f16298e8640 1 Processor -- start 2024-03-24T00:40:28.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.031+0000 7f16298e8640 1 -- start start 2024-03-24T00:40:28.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.032+0000 7f16298e8640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f162415bcd0 0x7f16240ce540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:28.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.032+0000 7f16298e8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f16240d0180 0x7f16240d05f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:28.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.032+0000 7f16298e8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16240cea80 con 0x7f16240cf7b0 2024-03-24T00:40:28.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.032+0000 7f16298e8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16240b1b00 con 0x7f16240d0180 2024-03-24T00:40:28.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.032+0000 7f16298e8640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16240d0b30 con 0x7f162415bcd0 2024-03-24T00:40:28.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.032+0000 7f16237fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f16240d0180 0x7f16240d05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:28.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.032+0000 7f16227fc640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f16240cf7b0 0x7f16240cc760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44122/0 (socket says 172.21.15.67:44122) 2024-03-24T00:40:28.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.032+0000 7f16227fc640 1 -- 172.21.15.67:0/749109760 learned_addr learned my addr 172.21.15.67:0/749109760 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:40:28.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.032+0000 7f16237fe640 1 --2- 172.21.15.67:0/749109760 >> v2:172.21.15.67:3300/0 conn(0x7f16240d0180 0x7f16240d05f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39338/0 (socket says 172.21.15.67:39338) 2024-03-24T00:40:28.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.032+0000 7f1622ffd640 1 --2- 172.21.15.67:0/749109760 >> v2:172.21.15.72:3300/0 conn(0x7f162415bcd0 0x7f16240ce540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:28.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.033+0000 7f16237fe640 1 -- 172.21.15.67:0/749109760 >> v2:172.21.15.72:3300/0 conn(0x7f162415bcd0 msgr2=0x7f16240ce540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:28.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.033+0000 7f16237fe640 1 --2- 172.21.15.67:0/749109760 >> v2:172.21.15.72:3300/0 conn(0x7f162415bcd0 0x7f16240ce540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:28.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.033+0000 7f16237fe640 1 -- 172.21.15.67:0/749109760 >> v1:172.21.15.67:6789/0 conn(0x7f16240cf7b0 legacy=0x7f16240cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:28.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.033+0000 7f16237fe640 1 -- 172.21.15.67:0/749109760 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16240ced20 con 0x7f16240d0180 2024-03-24T00:40:28.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.033+0000 7f16237fe640 1 --2- 172.21.15.67:0/749109760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16240d0180 0x7f16240d05f0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f16240b4760 tx=0x7f161402bd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0df774f1aa9402a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:28.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.034+0000 7f1621ffb640 1 -- 172.21.15.67:0/749109760 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1614039070 con 0x7f16240d0180 2024-03-24T00:40:28.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.034+0000 7f1621ffb640 1 -- 172.21.15.67:0/749109760 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f161402ca30 con 0x7f16240d0180 2024-03-24T00:40:28.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.034+0000 7f1621ffb640 1 -- 172.21.15.67:0/749109760 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f161402ce00 con 0x7f16240d0180 2024-03-24T00:40:28.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.034+0000 7f16298e8640 1 -- 172.21.15.67:0/749109760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16240d0180 msgr2=0x7f16240d05f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:28.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.034+0000 7f16298e8640 1 --2- 172.21.15.67:0/749109760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16240d0180 0x7f16240d05f0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f16240b4760 tx=0x7f161402bd00 comp rx=0 tx=0).stop 2024-03-24T00:40:28.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.034+0000 7f16298e8640 1 -- 172.21.15.67:0/749109760 shutdown_connections 2024-03-24T00:40:28.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.034+0000 7f16298e8640 1 --2- 172.21.15.67:0/749109760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16240d0180 0x7f16240d05f0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:28.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.034+0000 7f16298e8640 1 --2- 172.21.15.67:0/749109760 >> v2:172.21.15.72:3300/0 conn(0x7f162415bcd0 0x7f16240ce540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:28.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.034+0000 7f16298e8640 1 -- 172.21.15.67:0/749109760 >> 172.21.15.67:0/749109760 conn(0x7f162408b1b0 msgr2=0x7f162408b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:28.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.034+0000 7f16298e8640 1 -- 172.21.15.67:0/749109760 shutdown_connections 2024-03-24T00:40:28.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.034+0000 7f16298e8640 1 -- 172.21.15.67:0/749109760 wait complete. 2024-03-24T00:40:28.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.035+0000 7f16298e8640 1 Processor -- start 2024-03-24T00:40:28.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.035+0000 7f16298e8640 1 -- start start 2024-03-24T00:40:28.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.036+0000 7f16298e8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f16240cf7b0 0x7f16241503f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:28.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.036+0000 7f16298e8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16240d0180 0x7f16240b7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:28.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.036+0000 7f16298e8640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f162415bcd0 0x7f162415e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:28.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.036+0000 7f16298e8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f162415c0b0 con 0x7f16240d0180 2024-03-24T00:40:28.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.036+0000 7f16298e8640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16240cf2e0 con 0x7f16240cf7b0 2024-03-24T00:40:28.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.036+0000 7f16298e8640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f162415c230 con 0x7f162415bcd0 2024-03-24T00:40:28.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.036+0000 7f16227fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16240d0180 0x7f16240b7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:28.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.036+0000 7f16227fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16240d0180 0x7f16240b7170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39340/0 (socket says 172.21.15.67:39340) 2024-03-24T00:40:28.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.036+0000 7f16227fc640 1 -- 172.21.15.67:0/1592090323 learned_addr learned my addr 172.21.15.67:0/1592090323 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:28.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.036+0000 7f1622ffd640 1 --2- 172.21.15.67:0/1592090323 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f16240cf7b0 0x7f16241503f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:28.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.036+0000 7f16237fe640 1 --2- 172.21.15.67:0/1592090323 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f162415bcd0 0x7f162415e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:28.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.037+0000 7f16227fc640 1 -- 172.21.15.67:0/1592090323 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f162415bcd0 msgr2=0x7f162415e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:28.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.037+0000 7f16227fc640 1 --2- 172.21.15.67:0/1592090323 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f162415bcd0 0x7f162415e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:28.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.037+0000 7f16227fc640 1 -- 172.21.15.67:0/1592090323 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f16240cf7b0 msgr2=0x7f16241503f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:28.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.037+0000 7f16227fc640 1 --2- 172.21.15.67:0/1592090323 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f16240cf7b0 0x7f16241503f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:28.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.037+0000 7f16227fc640 1 -- 172.21.15.67:0/1592090323 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16240b4890 con 0x7f16240d0180 2024-03-24T00:40:28.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.037+0000 7f16227fc640 1 --2- 172.21.15.67:0/1592090323 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16240d0180 0x7f16240b7170 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f160c00c780 tx=0x7f160c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:28.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.037+0000 7f161bfff640 1 -- 172.21.15.67:0/1592090323 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f160c017070 con 0x7f16240d0180 2024-03-24T00:40:28.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.037+0000 7f161bfff640 1 -- 172.21.15.67:0/1592090323 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f160c005a70 con 0x7f16240d0180 2024-03-24T00:40:28.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.037+0000 7f16298e8640 1 -- 172.21.15.67:0/1592090323 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16240b4b70 con 0x7f16240d0180 2024-03-24T00:40:28.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.037+0000 7f161bfff640 1 -- 172.21.15.67:0/1592090323 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f160c007070 con 0x7f16240d0180 2024-03-24T00:40:28.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.037+0000 7f16298e8640 1 -- 172.21.15.67:0/1592090323 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16240b5130 con 0x7f16240d0180 2024-03-24T00:40:28.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.038+0000 7f16298e8640 1 -- 172.21.15.67:0/1592090323 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f162404f400 con 0x7f16240d0180 2024-03-24T00:40:28.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.038+0000 7f161bfff640 1 -- 172.21.15.67:0/1592090323 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f160c01a070 con 0x7f16240d0180 2024-03-24T00:40:28.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.039+0000 7f161bfff640 1 --2- 172.21.15.67:0/1592090323 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f160406bbd0 0x7f160406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:28.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.039+0000 7f161bfff640 1 -- 172.21.15.67:0/1592090323 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f160c089fc0 con 0x7f16240d0180 2024-03-24T00:40:28.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.039+0000 7f1622ffd640 1 --2- 172.21.15.67:0/1592090323 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f160406bbd0 0x7f160406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:28.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.039+0000 7f1622ffd640 1 --2- 172.21.15.67:0/1592090323 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f160406bbd0 0x7f160406e090 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f161c009aa0 tx=0x7f161c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:28.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.043+0000 7f161bfff640 1 -- 172.21.15.67:0/1592090323 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f160c058360 con 0x7f16240d0180 2024-03-24T00:40:28.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.265+0000 7f16298e8640 1 -- 172.21.15.67:0/1592090323 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f16240b1640 con 0x7f16240d0180 2024-03-24T00:40:28.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.266+0000 7f161bfff640 1 -- 172.21.15.67:0/1592090323 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f160c05c400 con 0x7f16240d0180 2024-03-24T00:40:28.266 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:40:28.266 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5039,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/166859817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":166859817},{"type":"v1","addr":"172.21.15.67:6834","nonce":166859817}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5042,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2653663845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2653663845},{"type":"v1","addr":"172.21.15.67:6835","nonce":2653663845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5048,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2355909634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2355909634},{"type":"v1","addr":"172.21.15.72:6837","nonce":2355909634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:40:00.691429+0000","modified":"2024-03-24T00:40:03.146091+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5418},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5418":{"gid":5418,"name":"d","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6836/2551774148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2551774148},{"type":"v1","addr":"172.21.15.72:6836","nonce":2551774148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5418,"qdb_cluster":[5418]},"id":2}]} 2024-03-24T00:40:28.267 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 17 2024-03-24T00:40:28.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.267+0000 7f16298e8640 1 -- 172.21.15.67:0/1592090323 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f160406bbd0 msgr2=0x7f160406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:28.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.267+0000 7f16298e8640 1 --2- 172.21.15.67:0/1592090323 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f160406bbd0 0x7f160406e090 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f161c009aa0 tx=0x7f161c009190 comp rx=0 tx=0).stop 2024-03-24T00:40:28.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.267+0000 7f16298e8640 1 -- 172.21.15.67:0/1592090323 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16240d0180 msgr2=0x7f16240b7170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:28.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.267+0000 7f16298e8640 1 --2- 172.21.15.67:0/1592090323 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16240d0180 0x7f16240b7170 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f160c00c780 tx=0x7f160c00cc40 comp rx=0 tx=0).stop 2024-03-24T00:40:28.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.267+0000 7f16298e8640 1 -- 172.21.15.67:0/1592090323 shutdown_connections 2024-03-24T00:40:28.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.267+0000 7f16298e8640 1 --2- 172.21.15.67:0/1592090323 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f160406bbd0 0x7f160406e090 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:28.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.267+0000 7f16298e8640 1 --2- 172.21.15.67:0/1592090323 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f162415bcd0 0x7f162415e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:28.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.267+0000 7f16298e8640 1 --2- 172.21.15.67:0/1592090323 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16240d0180 0x7f16240b7170 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:28.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.267+0000 7f16298e8640 1 --2- 172.21.15.67:0/1592090323 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f16240cf7b0 0x7f16241503f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:28.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.267+0000 7f16298e8640 1 -- 172.21.15.67:0/1592090323 >> 172.21.15.67:0/1592090323 conn(0x7f162408b1b0 msgr2=0x7f1624070590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:28.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.268+0000 7f16298e8640 1 -- 172.21.15.67:0/1592090323 shutdown_connections 2024-03-24T00:40:28.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.268+0000 7f16298e8640 1 -- 172.21.15.67:0/1592090323 wait complete. 2024-03-24T00:40:28.279 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:40:28.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.379+0000 7f866f412640 1 Processor -- start 2024-03-24T00:40:28.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.379+0000 7f866f412640 1 -- start start 2024-03-24T00:40:28.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.380+0000 7f866f412640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f866815be40 0x7f866806b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:28.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.380+0000 7f866f412640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f86680cd350 0x7f86680b58f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:28.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.380+0000 7f866f412640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86680b2ca0 con 0x7f86680af2e0 2024-03-24T00:40:28.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.380+0000 7f866f412640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86680afb80 con 0x7f86680cd350 2024-03-24T00:40:28.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.380+0000 7f866f412640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f866815c220 con 0x7f866815be40 2024-03-24T00:40:28.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.380+0000 7f866d9af640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f86680cd350 0x7f86680b58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:28.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.380+0000 7f866d9af640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f86680cd350 0x7f86680b58f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39352/0 (socket says 172.21.15.67:39352) 2024-03-24T00:40:28.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.380+0000 7f866d9af640 1 -- 172.21.15.67:0/1245738639 learned_addr learned my addr 172.21.15.67:0/1245738639 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:28.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.380+0000 7f866d1ae640 1 --2- 172.21.15.67:0/1245738639 >> v2:172.21.15.72:3300/0 conn(0x7f866815be40 0x7f866806b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:28.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.380+0000 7f866d9af640 1 -- 172.21.15.67:0/1245738639 >> v2:172.21.15.72:3300/0 conn(0x7f866815be40 msgr2=0x7f866806b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:28.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.380+0000 7f866d9af640 1 --2- 172.21.15.67:0/1245738639 >> v2:172.21.15.72:3300/0 conn(0x7f866815be40 0x7f866806b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:28.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.380+0000 7f866d9af640 1 -- 172.21.15.67:0/1245738639 >> v1:172.21.15.72:6789/0 conn(0x7f86680af2e0 legacy=0x7f86680ccc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:28.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.380+0000 7f866d9af640 1 -- 172.21.15.67:0/1245738639 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86680b26e0 con 0x7f86680cd350 2024-03-24T00:40:28.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.381+0000 7f866d9af640 1 --2- 172.21.15.67:0/1245738639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86680cd350 0x7f86680b58f0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f86680b60c0 tx=0x7f866002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1f0f7715e49b116 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:28.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.381+0000 7f865ffff640 1 -- 172.21.15.67:0/1245738639 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f866002ca80 con 0x7f86680cd350 2024-03-24T00:40:28.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.381+0000 7f865ffff640 1 -- 172.21.15.67:0/1245738639 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f866002cc20 con 0x7f86680cd350 2024-03-24T00:40:28.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.381+0000 7f865ffff640 1 -- 172.21.15.67:0/1245738639 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86600343f0 con 0x7f86680cd350 2024-03-24T00:40:28.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.382+0000 7f866f412640 1 -- 172.21.15.67:0/1245738639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86680cd350 msgr2=0x7f86680b58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:28.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.382+0000 7f866f412640 1 --2- 172.21.15.67:0/1245738639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86680cd350 0x7f86680b58f0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f86680b60c0 tx=0x7f866002bee0 comp rx=0 tx=0).stop 2024-03-24T00:40:28.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.382+0000 7f866f412640 1 -- 172.21.15.67:0/1245738639 shutdown_connections 2024-03-24T00:40:28.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.382+0000 7f866f412640 1 --2- 172.21.15.67:0/1245738639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86680cd350 0x7f86680b58f0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:28.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.382+0000 7f866f412640 1 --2- 172.21.15.67:0/1245738639 >> v2:172.21.15.72:3300/0 conn(0x7f866815be40 0x7f866806b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:28.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.382+0000 7f866f412640 1 -- 172.21.15.67:0/1245738639 >> 172.21.15.67:0/1245738639 conn(0x7f86680b3c40 msgr2=0x7f86680b1dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:28.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.382+0000 7f866f412640 1 -- 172.21.15.67:0/1245738639 shutdown_connections 2024-03-24T00:40:28.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.382+0000 7f866f412640 1 -- 172.21.15.67:0/1245738639 wait complete. 2024-03-24T00:40:28.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.383+0000 7f866f412640 1 Processor -- start 2024-03-24T00:40:28.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.383+0000 7f866f412640 1 -- start start 2024-03-24T00:40:28.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.384+0000 7f866f412640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86680af2e0 0x7f8668159430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:28.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.384+0000 7f866f412640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f86680cd350 0x7f866815ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:28.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.384+0000 7f866f412640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f866815be40 0x7f866806f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:28.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.384+0000 7f866f412640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86680cf400 con 0x7f86680af2e0 2024-03-24T00:40:28.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.384+0000 7f866f412640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86680ceee0 con 0x7f86680cd350 2024-03-24T00:40:28.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.384+0000 7f866f412640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86680cf580 con 0x7f866815be40 2024-03-24T00:40:28.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.384+0000 7f866d1ae640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86680af2e0 0x7f8668159430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:28.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.384+0000 7f866d1ae640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86680af2e0 0x7f8668159430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39358/0 (socket says 172.21.15.67:39358) 2024-03-24T00:40:28.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.384+0000 7f866d1ae640 1 -- 172.21.15.67:0/3820124836 learned_addr learned my addr 172.21.15.67:0/3820124836 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:28.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.384+0000 7f866c9ad640 1 --2- 172.21.15.67:0/3820124836 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f86680cd350 0x7f866815ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:28.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.384+0000 7f866d9af640 1 --2- 172.21.15.67:0/3820124836 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f866815be40 0x7f866806f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:28.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.384+0000 7f866d1ae640 1 -- 172.21.15.67:0/3820124836 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f866815be40 msgr2=0x7f866806f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:28.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.384+0000 7f866d1ae640 1 --2- 172.21.15.67:0/3820124836 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f866815be40 0x7f866806f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:28.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.384+0000 7f866d1ae640 1 -- 172.21.15.67:0/3820124836 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f86680cd350 msgr2=0x7f866815ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:28.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.384+0000 7f866d1ae640 1 --2- 172.21.15.67:0/3820124836 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f86680cd350 0x7f866815ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:28.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.384+0000 7f866d1ae640 1 -- 172.21.15.67:0/3820124836 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86681f4810 con 0x7f86680af2e0 2024-03-24T00:40:28.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.384+0000 7f866d1ae640 1 --2- 172.21.15.67:0/3820124836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86680af2e0 0x7f8668159430 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f866400e8a0 tx=0x7f866400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:28.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.384+0000 7f865dffb640 1 -- 172.21.15.67:0/3820124836 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8664017070 con 0x7f86680af2e0 2024-03-24T00:40:28.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.384+0000 7f865dffb640 1 -- 172.21.15.67:0/3820124836 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f866400c940 con 0x7f86680af2e0 2024-03-24T00:40:28.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.384+0000 7f866f412640 1 -- 172.21.15.67:0/3820124836 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86681f4af0 con 0x7f86680af2e0 2024-03-24T00:40:28.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.384+0000 7f865dffb640 1 -- 172.21.15.67:0/3820124836 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f866400cd30 con 0x7f86680af2e0 2024-03-24T00:40:28.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.384+0000 7f866f412640 1 -- 172.21.15.67:0/3820124836 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86681f50b0 con 0x7f86680af2e0 2024-03-24T00:40:28.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.386+0000 7f866f412640 1 -- 172.21.15.67:0/3820124836 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86681f52f0 con 0x7f86680af2e0 2024-03-24T00:40:28.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.386+0000 7f865dffb640 1 -- 172.21.15.67:0/3820124836 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f866401a070 con 0x7f86680af2e0 2024-03-24T00:40:28.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.386+0000 7f865dffb640 1 --2- 172.21.15.67:0/3820124836 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f864c06bbd0 0x7f864c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:28.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.386+0000 7f865dffb640 1 -- 172.21.15.67:0/3820124836 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f866408ada0 con 0x7f86680af2e0 2024-03-24T00:40:28.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.386+0000 7f866c9ad640 1 --2- 172.21.15.67:0/3820124836 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f864c06bbd0 0x7f864c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:28.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.387+0000 7f866c9ad640 1 --2- 172.21.15.67:0/3820124836 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f864c06bbd0 0x7f864c06e090 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f86680af260 tx=0x7f8658009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:28.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.390+0000 7f865dffb640 1 -- 172.21.15.67:0/3820124836 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f86681f52f0 con 0x7f86680af2e0 2024-03-24T00:40:28.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.573+0000 7f866f412640 1 -- 172.21.15.67:0/3820124836 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f866815f610 con 0x7f86680af2e0 2024-03-24T00:40:28.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.574+0000 7f865dffb640 1 -- 172.21.15.67:0/3820124836 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) v1 ==== 74+0+16882 (secure 0 0 0) 0x7f86640590b0 con 0x7f86680af2e0 2024-03-24T00:40:28.575 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:40:28.576 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":39,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:40:02.323736+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:39:58.975995+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-24T00:39:59.653615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:40:28.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.576+0000 7f866f412640 1 -- 172.21.15.67:0/3820124836 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f864c06bbd0 msgr2=0x7f864c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:28.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.576+0000 7f866f412640 1 --2- 172.21.15.67:0/3820124836 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f864c06bbd0 0x7f864c06e090 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f86680af260 tx=0x7f8658009040 comp rx=0 tx=0).stop 2024-03-24T00:40:28.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.576+0000 7f866f412640 1 -- 172.21.15.67:0/3820124836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86680af2e0 msgr2=0x7f8668159430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:28.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.576+0000 7f866f412640 1 --2- 172.21.15.67:0/3820124836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86680af2e0 0x7f8668159430 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f866400e8a0 tx=0x7f866400ed60 comp rx=0 tx=0).stop 2024-03-24T00:40:28.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.577+0000 7f866f412640 1 -- 172.21.15.67:0/3820124836 shutdown_connections 2024-03-24T00:40:28.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.577+0000 7f866f412640 1 --2- 172.21.15.67:0/3820124836 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f864c06bbd0 0x7f864c06e090 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:28.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.577+0000 7f866f412640 1 --2- 172.21.15.67:0/3820124836 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f866815be40 0x7f866806f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:28.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.577+0000 7f866f412640 1 --2- 172.21.15.67:0/3820124836 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f86680cd350 0x7f866815ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:28.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.577+0000 7f866f412640 1 --2- 172.21.15.67:0/3820124836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86680af2e0 0x7f8668159430 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:28.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.577+0000 7f866f412640 1 -- 172.21.15.67:0/3820124836 >> 172.21.15.67:0/3820124836 conn(0x7f86680b3c40 msgr2=0x7f86680afd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:28.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.577+0000 7f866f412640 1 -- 172.21.15.67:0/3820124836 shutdown_connections 2024-03-24T00:40:28.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.577+0000 7f866f412640 1 -- 172.21.15.67:0/3820124836 wait complete. 2024-03-24T00:40:28.588 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-24T00:40:28.588 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:40:28.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.689+0000 7f6ba6e27640 1 Processor -- start 2024-03-24T00:40:28.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.689+0000 7f6ba6e27640 1 -- start start 2024-03-24T00:40:28.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.690+0000 7f6ba6e27640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6ba015dfb0 0x7f6ba00cfe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:28.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.690+0000 7f6ba6e27640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6ba015e390 0x7f6ba006b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:28.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.690+0000 7f6ba6e27640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ba00d07d0 con 0x7f6ba00cded0 2024-03-24T00:40:28.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.690+0000 7f6ba6e27640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ba00d0ab0 con 0x7f6ba015e390 2024-03-24T00:40:28.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.690+0000 7f6ba6e27640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba00d0d90 con 0x7f6ba015dfb0 2024-03-24T00:40:28.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.690+0000 7f6b9ffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6ba015e390 0x7f6ba006b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:28.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.690+0000 7f6ba53c4640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6ba00cded0 0x7f6ba0161d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44126/0 (socket says 172.21.15.67:44126) 2024-03-24T00:40:28.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.690+0000 7f6ba53c4640 1 -- 172.21.15.67:0/747183125 learned_addr learned my addr 172.21.15.67:0/747183125 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:40:28.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.690+0000 7f6ba4bc3640 1 --2- 172.21.15.67:0/747183125 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6ba015dfb0 0x7f6ba00cfe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:28.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.690+0000 7f6b9ffff640 1 -- 172.21.15.67:0/747183125 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6ba015dfb0 msgr2=0x7f6ba00cfe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:28.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.690+0000 7f6b9ffff640 1 --2- 172.21.15.67:0/747183125 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6ba015dfb0 0x7f6ba00cfe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:28.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.690+0000 7f6b9ffff640 1 -- 172.21.15.67:0/747183125 >> v1:172.21.15.67:6789/0 conn(0x7f6ba00cded0 legacy=0x7f6ba0161d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:28.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.690+0000 7f6b9ffff640 1 -- 172.21.15.67:0/747183125 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ba00d1070 con 0x7f6ba015e390 2024-03-24T00:40:28.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.690+0000 7f6b9f7fe640 1 -- 172.21.15.67:0/747183125 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 86561954 0 0) 0x7f6ba00d07d0 con 0x7f6ba00cded0 2024-03-24T00:40:28.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.691+0000 7f6b9ffff640 1 --2- 172.21.15.67:0/747183125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ba015e390 0x7f6ba006b040 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c0097a0 tx=0x7f6b8c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af71aa86e121212d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:28.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.691+0000 7f6b9f7fe640 1 -- 172.21.15.67:0/747183125 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b8c035020 con 0x7f6ba015e390 2024-03-24T00:40:28.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.691+0000 7f6b9f7fe640 1 -- 172.21.15.67:0/747183125 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b8c031830 con 0x7f6ba015e390 2024-03-24T00:40:28.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.691+0000 7f6b9f7fe640 1 -- 172.21.15.67:0/747183125 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b8c031c00 con 0x7f6ba015e390 2024-03-24T00:40:28.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.691+0000 7f6ba6e27640 1 -- 172.21.15.67:0/747183125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ba015e390 msgr2=0x7f6ba006b040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:28.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.691+0000 7f6ba6e27640 1 --2- 172.21.15.67:0/747183125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ba015e390 0x7f6ba006b040 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c0097a0 tx=0x7f6b8c02b8d0 comp rx=0 tx=0).stop 2024-03-24T00:40:28.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.692+0000 7f6ba6e27640 1 -- 172.21.15.67:0/747183125 shutdown_connections 2024-03-24T00:40:28.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.692+0000 7f6ba6e27640 1 --2- 172.21.15.67:0/747183125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ba015e390 0x7f6ba006b040 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:28.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.692+0000 7f6ba6e27640 1 --2- 172.21.15.67:0/747183125 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6ba015dfb0 0x7f6ba00cfe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:28.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.692+0000 7f6ba6e27640 1 -- 172.21.15.67:0/747183125 >> 172.21.15.67:0/747183125 conn(0x7f6ba0099a30 msgr2=0x7f6ba0099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:28.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.692+0000 7f6ba6e27640 1 -- 172.21.15.67:0/747183125 shutdown_connections 2024-03-24T00:40:28.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.692+0000 7f6ba6e27640 1 -- 172.21.15.67:0/747183125 wait complete. 2024-03-24T00:40:28.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.693+0000 7f6ba6e27640 1 Processor -- start 2024-03-24T00:40:28.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.693+0000 7f6ba6e27640 1 -- start start 2024-03-24T00:40:28.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.693+0000 7f6ba6e27640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ba00cded0 0x7f6ba00c5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:28.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.694+0000 7f6ba6e27640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6ba015dfb0 0x7f6ba0161360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:28.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.694+0000 7f6ba6e27640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ba015e390 0x7f6ba00c9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:28.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.694+0000 7f6ba6e27640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba00d1870 con 0x7f6ba015e390 2024-03-24T00:40:28.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.694+0000 7f6ba6e27640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba00cc130 con 0x7f6ba00cded0 2024-03-24T00:40:28.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.694+0000 7f6ba6e27640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba00d03c0 con 0x7f6ba015dfb0 2024-03-24T00:40:28.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.694+0000 7f6ba53c4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ba015e390 0x7f6ba00c9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:28.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.694+0000 7f6ba53c4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ba015e390 0x7f6ba00c9ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39384/0 (socket says 172.21.15.67:39384) 2024-03-24T00:40:28.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.694+0000 7f6ba53c4640 1 -- 172.21.15.67:0/1369022592 learned_addr learned my addr 172.21.15.67:0/1369022592 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:28.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.694+0000 7f6ba4bc3640 1 --2- 172.21.15.67:0/1369022592 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ba00cded0 0x7f6ba00c5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:28.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.694+0000 7f6b9ffff640 1 --2- 172.21.15.67:0/1369022592 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6ba015dfb0 0x7f6ba0161360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:28.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.694+0000 7f6ba53c4640 1 -- 172.21.15.67:0/1369022592 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6ba015dfb0 msgr2=0x7f6ba0161360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:28.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.694+0000 7f6ba53c4640 1 --2- 172.21.15.67:0/1369022592 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6ba015dfb0 0x7f6ba0161360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:28.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.694+0000 7f6ba53c4640 1 -- 172.21.15.67:0/1369022592 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ba00cded0 msgr2=0x7f6ba00c5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:28.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.694+0000 7f6ba53c4640 1 --2- 172.21.15.67:0/1369022592 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ba00cded0 0x7f6ba00c5bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:28.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.694+0000 7f6ba53c4640 1 -- 172.21.15.67:0/1369022592 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ba015ea70 con 0x7f6ba015e390 2024-03-24T00:40:28.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.694+0000 7f6ba53c4640 1 --2- 172.21.15.67:0/1369022592 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ba015e390 0x7f6ba00c9ae0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f6b9800c7a0 tx=0x7f6b9800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:28.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.694+0000 7f6b9d7fa640 1 -- 172.21.15.67:0/1369022592 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b98018070 con 0x7f6ba015e390 2024-03-24T00:40:28.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.694+0000 7f6b9d7fa640 1 -- 172.21.15.67:0/1369022592 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b9800a640 con 0x7f6ba015e390 2024-03-24T00:40:28.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.694+0000 7f6ba6e27640 1 -- 172.21.15.67:0/1369022592 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ba015ece0 con 0x7f6ba015e390 2024-03-24T00:40:28.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.694+0000 7f6b9d7fa640 1 -- 172.21.15.67:0/1369022592 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b9800aab0 con 0x7f6ba015e390 2024-03-24T00:40:28.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.694+0000 7f6ba6e27640 1 -- 172.21.15.67:0/1369022592 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ba00ca480 con 0x7f6ba015e390 2024-03-24T00:40:28.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.695+0000 7f6ba6e27640 1 -- 172.21.15.67:0/1369022592 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ba004f400 con 0x7f6ba015e390 2024-03-24T00:40:28.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.696+0000 7f6b9d7fa640 1 -- 172.21.15.67:0/1369022592 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6b9801b070 con 0x7f6ba015e390 2024-03-24T00:40:28.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.696+0000 7f6b9d7fa640 1 --2- 172.21.15.67:0/1369022592 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6b8406bbd0 0x7f6b8406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:28.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.696+0000 7f6b9d7fa640 1 -- 172.21.15.67:0/1369022592 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6b9808a170 con 0x7f6ba015e390 2024-03-24T00:40:28.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.696+0000 7f6ba4bc3640 1 --2- 172.21.15.67:0/1369022592 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6b8406bbd0 0x7f6b8406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:28.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.697+0000 7f6ba4bc3640 1 --2- 172.21.15.67:0/1369022592 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6b8406bbd0 0x7f6b8406e090 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f6b94009b40 tx=0x7f6b94009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:28.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.700+0000 7f6b9d7fa640 1 -- 172.21.15.67:0/1369022592 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6b98058480 con 0x7f6ba015e390 2024-03-24T00:40:28.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.922+0000 7f6ba6e27640 1 -- 172.21.15.67:0/1369022592 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ba00b6a50 con 0x7f6ba015e390 2024-03-24T00:40:28.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.923+0000 7f6b9d7fa640 1 -- 172.21.15.67:0/1369022592 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f6b9805c520 con 0x7f6ba015e390 2024-03-24T00:40:28.924 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:40:28.924 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5039,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/166859817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":166859817},{"type":"v1","addr":"172.21.15.67:6834","nonce":166859817}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5042,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2653663845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2653663845},{"type":"v1","addr":"172.21.15.67:6835","nonce":2653663845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5048,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2355909634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2355909634},{"type":"v1","addr":"172.21.15.72:6837","nonce":2355909634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:40:00.691429+0000","modified":"2024-03-24T00:40:03.146091+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5418},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5418":{"gid":5418,"name":"d","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6836/2551774148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2551774148},{"type":"v1","addr":"172.21.15.72:6836","nonce":2551774148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5418,"qdb_cluster":[5418]},"id":2}]} 2024-03-24T00:40:28.924 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 17 2024-03-24T00:40:28.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.925+0000 7f6ba6e27640 1 -- 172.21.15.67:0/1369022592 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6b8406bbd0 msgr2=0x7f6b8406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:28.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.925+0000 7f6ba6e27640 1 --2- 172.21.15.67:0/1369022592 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6b8406bbd0 0x7f6b8406e090 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f6b94009b40 tx=0x7f6b94009190 comp rx=0 tx=0).stop 2024-03-24T00:40:28.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.925+0000 7f6ba6e27640 1 -- 172.21.15.67:0/1369022592 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ba015e390 msgr2=0x7f6ba00c9ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:28.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.925+0000 7f6ba6e27640 1 --2- 172.21.15.67:0/1369022592 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ba015e390 0x7f6ba00c9ae0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f6b9800c7a0 tx=0x7f6b9800cc60 comp rx=0 tx=0).stop 2024-03-24T00:40:28.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.925+0000 7f6ba6e27640 1 -- 172.21.15.67:0/1369022592 shutdown_connections 2024-03-24T00:40:28.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.925+0000 7f6ba6e27640 1 --2- 172.21.15.67:0/1369022592 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6b8406bbd0 0x7f6b8406e090 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:28.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.925+0000 7f6ba6e27640 1 --2- 172.21.15.67:0/1369022592 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ba015e390 0x7f6ba00c9ae0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:28.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.925+0000 7f6ba6e27640 1 --2- 172.21.15.67:0/1369022592 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6ba015dfb0 0x7f6ba0161360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:28.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.925+0000 7f6ba6e27640 1 --2- 172.21.15.67:0/1369022592 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ba00cded0 0x7f6ba00c5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:28.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.925+0000 7f6ba6e27640 1 -- 172.21.15.67:0/1369022592 >> 172.21.15.67:0/1369022592 conn(0x7f6ba0099a30 msgr2=0x7f6ba00b60a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:28.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.925+0000 7f6ba6e27640 1 -- 172.21.15.67:0/1369022592 shutdown_connections 2024-03-24T00:40:28.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:28.925+0000 7f6ba6e27640 1 -- 172.21.15.67:0/1369022592 wait complete. 2024-03-24T00:40:28.936 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:40:29.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.037+0000 7f02e9b97640 1 Processor -- start 2024-03-24T00:40:29.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.037+0000 7f02e9b97640 1 -- start start 2024-03-24T00:40:29.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.038+0000 7f02e9b97640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f02e415bf70 0x7f02e40cc690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:29.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.038+0000 7f02e9b97640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f02e40cfc70 0x7f02e40ce220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:29.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.038+0000 7f02e9b97640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02e40ce760 0x7f02e408b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:29.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.038+0000 7f02e9b97640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02e40ccfe0 con 0x7f02e40cfc70 2024-03-24T00:40:29.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.038+0000 7f02e9b97640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02e40cd2c0 con 0x7f02e415bf70 2024-03-24T00:40:29.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.038+0000 7f02e9b97640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02e40cd5a0 con 0x7f02e40ce760 2024-03-24T00:40:29.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.038+0000 7f02e2ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f02e40cfc70 0x7f02e40ce220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:29.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.038+0000 7f02e2ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f02e40cfc70 0x7f02e40ce220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39386/0 (socket says 172.21.15.67:39386) 2024-03-24T00:40:29.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.038+0000 7f02e2ffd640 1 -- 172.21.15.67:0/390835046 learned_addr learned my addr 172.21.15.67:0/390835046 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:29.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.038+0000 7f02e3fff640 1 --2- 172.21.15.67:0/390835046 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02e40ce760 0x7f02e408b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:29.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.038+0000 7f02e37fe640 1 --2- 172.21.15.67:0/390835046 >> v2:172.21.15.72:3300/0 conn(0x7f02e415bf70 0x7f02e40cc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:29.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.039+0000 7f02e2ffd640 1 -- 172.21.15.67:0/390835046 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02e40ce760 msgr2=0x7f02e408b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:29.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.039+0000 7f02e2ffd640 1 --2- 172.21.15.67:0/390835046 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02e40ce760 0x7f02e408b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:29.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.039+0000 7f02e2ffd640 1 -- 172.21.15.67:0/390835046 >> v2:172.21.15.72:3300/0 conn(0x7f02e415bf70 msgr2=0x7f02e40cc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:29.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.039+0000 7f02e2ffd640 1 --2- 172.21.15.67:0/390835046 >> v2:172.21.15.72:3300/0 conn(0x7f02e415bf70 0x7f02e40cc690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:29.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.039+0000 7f02e2ffd640 1 -- 172.21.15.67:0/390835046 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e40cd880 con 0x7f02e40cfc70 2024-03-24T00:40:29.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.039+0000 7f02e2ffd640 1 --2- 172.21.15.67:0/390835046 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02e40cfc70 0x7f02e40ce220 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f02e408c2a0 tx=0x7f02d802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df527acbc2b15888 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:29.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.040+0000 7f02e27fc640 1 -- 172.21.15.67:0/390835046 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d8035020 con 0x7f02e40cfc70 2024-03-24T00:40:29.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.040+0000 7f02e27fc640 1 -- 172.21.15.67:0/390835046 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02d8031870 con 0x7f02e40cfc70 2024-03-24T00:40:29.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.040+0000 7f02e27fc640 1 -- 172.21.15.67:0/390835046 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d8031c40 con 0x7f02e40cfc70 2024-03-24T00:40:29.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.040+0000 7f02e9b97640 1 -- 172.21.15.67:0/390835046 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02e40cfc70 msgr2=0x7f02e40ce220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:29.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.040+0000 7f02e9b97640 1 --2- 172.21.15.67:0/390835046 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02e40cfc70 0x7f02e40ce220 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f02e408c2a0 tx=0x7f02d802b910 comp rx=0 tx=0).stop 2024-03-24T00:40:29.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.040+0000 7f02e9b97640 1 -- 172.21.15.67:0/390835046 shutdown_connections 2024-03-24T00:40:29.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.040+0000 7f02e9b97640 1 --2- 172.21.15.67:0/390835046 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02e40ce760 0x7f02e408b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:29.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.040+0000 7f02e9b97640 1 --2- 172.21.15.67:0/390835046 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02e40cfc70 0x7f02e40ce220 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:29.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.040+0000 7f02e9b97640 1 --2- 172.21.15.67:0/390835046 >> v2:172.21.15.72:3300/0 conn(0x7f02e415bf70 0x7f02e40cc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:29.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.040+0000 7f02e9b97640 1 -- 172.21.15.67:0/390835046 >> 172.21.15.67:0/390835046 conn(0x7f02e40d1bf0 msgr2=0x7f02e40d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:29.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.040+0000 7f02e9b97640 1 -- 172.21.15.67:0/390835046 shutdown_connections 2024-03-24T00:40:29.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.040+0000 7f02e9b97640 1 -- 172.21.15.67:0/390835046 wait complete. 2024-03-24T00:40:29.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.041+0000 7f02e9b97640 1 Processor -- start 2024-03-24T00:40:29.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.041+0000 7f02e9b97640 1 -- start start 2024-03-24T00:40:29.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.042+0000 7f02e9b97640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02e40ce760 0x7f02e40afa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:29.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.042+0000 7f02e9b97640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02e40cfc70 0x7f02e4159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:29.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.042+0000 7f02e9b97640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02e415bf70 0x7f02e415a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:29.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.042+0000 7f02e9b97640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02e40ccd50 con 0x7f02e40cfc70 2024-03-24T00:40:29.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.042+0000 7f02e9b97640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02e40ccbd0 con 0x7f02e415bf70 2024-03-24T00:40:29.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.042+0000 7f02e9b97640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02e408b5a0 con 0x7f02e40ce760 2024-03-24T00:40:29.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.042+0000 7f02e2ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02e40cfc70 0x7f02e4159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:29.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.042+0000 7f02e2ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02e40cfc70 0x7f02e4159ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39392/0 (socket says 172.21.15.67:39392) 2024-03-24T00:40:29.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.042+0000 7f02e2ffd640 1 -- 172.21.15.67:0/3035695137 learned_addr learned my addr 172.21.15.67:0/3035695137 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:29.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.042+0000 7f02e37fe640 1 --2- 172.21.15.67:0/3035695137 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02e40ce760 0x7f02e40afa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:29.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.042+0000 7f02e3fff640 1 --2- 172.21.15.67:0/3035695137 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02e415bf70 0x7f02e415a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:29.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.043+0000 7f02e2ffd640 1 -- 172.21.15.67:0/3035695137 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02e40ce760 msgr2=0x7f02e40afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:29.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.043+0000 7f02e2ffd640 1 --2- 172.21.15.67:0/3035695137 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02e40ce760 0x7f02e40afa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:29.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.043+0000 7f02e2ffd640 1 -- 172.21.15.67:0/3035695137 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02e415bf70 msgr2=0x7f02e415a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:29.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.043+0000 7f02e2ffd640 1 --2- 172.21.15.67:0/3035695137 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02e415bf70 0x7f02e415a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:29.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.043+0000 7f02e2ffd640 1 -- 172.21.15.67:0/3035695137 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e41f6f20 con 0x7f02e40cfc70 2024-03-24T00:40:29.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.043+0000 7f02e2ffd640 1 --2- 172.21.15.67:0/3035695137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02e40cfc70 0x7f02e4159ce0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f02d8000c00 tx=0x7f02d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:29.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.043+0000 7f02cbfff640 1 -- 172.21.15.67:0/3035695137 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d8035040 con 0x7f02e40cfc70 2024-03-24T00:40:29.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.043+0000 7f02cbfff640 1 -- 172.21.15.67:0/3035695137 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02d8030d00 con 0x7f02e40cfc70 2024-03-24T00:40:29.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.043+0000 7f02e9b97640 1 -- 172.21.15.67:0/3035695137 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02e40b6560 con 0x7f02e40cfc70 2024-03-24T00:40:29.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.043+0000 7f02cbfff640 1 -- 172.21.15.67:0/3035695137 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d80456a0 con 0x7f02e40cfc70 2024-03-24T00:40:29.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.043+0000 7f02e9b97640 1 -- 172.21.15.67:0/3035695137 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02e41f74c0 con 0x7f02e40cfc70 2024-03-24T00:40:29.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.044+0000 7f02e9b97640 1 -- 172.21.15.67:0/3035695137 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02e404f400 con 0x7f02e40cfc70 2024-03-24T00:40:29.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.044+0000 7f02cbfff640 1 -- 172.21.15.67:0/3035695137 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f02d8038020 con 0x7f02e40cfc70 2024-03-24T00:40:29.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.045+0000 7f02cbfff640 1 --2- 172.21.15.67:0/3035695137 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f02d006bbd0 0x7f02d006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:29.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.045+0000 7f02cbfff640 1 -- 172.21.15.67:0/3035695137 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f02d80ace40 con 0x7f02e40cfc70 2024-03-24T00:40:29.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.045+0000 7f02e37fe640 1 --2- 172.21.15.67:0/3035695137 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f02d006bbd0 0x7f02d006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:29.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.045+0000 7f02e37fe640 1 --2- 172.21.15.67:0/3035695137 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f02d006bbd0 0x7f02d006e090 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f02d4009aa0 tx=0x7f02d4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:29.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.049+0000 7f02cbfff640 1 -- 172.21.15.67:0/3035695137 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f02d807b150 con 0x7f02e40cfc70 2024-03-24T00:40:29.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.270+0000 7f02e9b97640 1 -- 172.21.15.67:0/3035695137 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02e41f7700 con 0x7f02e40cfc70 2024-03-24T00:40:29.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.270+0000 7f02cbfff640 1 -- 172.21.15.67:0/3035695137 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f02d807f1f0 con 0x7f02e40cfc70 2024-03-24T00:40:29.271 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:40:29.271 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5039,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/166859817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":166859817},{"type":"v1","addr":"172.21.15.67:6834","nonce":166859817}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5042,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2653663845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2653663845},{"type":"v1","addr":"172.21.15.67:6835","nonce":2653663845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5048,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2355909634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2355909634},{"type":"v1","addr":"172.21.15.72:6837","nonce":2355909634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:40:00.691429+0000","modified":"2024-03-24T00:40:03.146091+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5418},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5418":{"gid":5418,"name":"d","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6836/2551774148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2551774148},{"type":"v1","addr":"172.21.15.72:6836","nonce":2551774148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5418,"qdb_cluster":[5418]},"id":2}]} 2024-03-24T00:40:29.271 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 17 2024-03-24T00:40:29.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.272+0000 7f02e9b97640 1 -- 172.21.15.67:0/3035695137 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f02d006bbd0 msgr2=0x7f02d006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:29.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.272+0000 7f02e9b97640 1 --2- 172.21.15.67:0/3035695137 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f02d006bbd0 0x7f02d006e090 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f02d4009aa0 tx=0x7f02d4009190 comp rx=0 tx=0).stop 2024-03-24T00:40:29.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.272+0000 7f02e9b97640 1 -- 172.21.15.67:0/3035695137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02e40cfc70 msgr2=0x7f02e4159ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:29.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.272+0000 7f02e9b97640 1 --2- 172.21.15.67:0/3035695137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02e40cfc70 0x7f02e4159ce0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f02d8000c00 tx=0x7f02d80046c0 comp rx=0 tx=0).stop 2024-03-24T00:40:29.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.272+0000 7f02e9b97640 1 -- 172.21.15.67:0/3035695137 shutdown_connections 2024-03-24T00:40:29.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.272+0000 7f02e9b97640 1 --2- 172.21.15.67:0/3035695137 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f02d006bbd0 0x7f02d006e090 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:29.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.272+0000 7f02e9b97640 1 --2- 172.21.15.67:0/3035695137 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02e415bf70 0x7f02e415a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:29.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.272+0000 7f02e9b97640 1 --2- 172.21.15.67:0/3035695137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02e40cfc70 0x7f02e4159ce0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:29.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.272+0000 7f02e9b97640 1 --2- 172.21.15.67:0/3035695137 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02e40ce760 0x7f02e40afa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:29.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.272+0000 7f02e9b97640 1 -- 172.21.15.67:0/3035695137 >> 172.21.15.67:0/3035695137 conn(0x7f02e40d1bf0 msgr2=0x7f02e4070450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:29.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.273+0000 7f02e9b97640 1 -- 172.21.15.67:0/3035695137 shutdown_connections 2024-03-24T00:40:29.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.273+0000 7f02e9b97640 1 -- 172.21.15.67:0/3035695137 wait complete. 2024-03-24T00:40:29.284 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:40:29.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.384+0000 7f649500b640 1 Processor -- start 2024-03-24T00:40:29.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.384+0000 7f649500b640 1 -- start start 2024-03-24T00:40:29.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.385+0000 7f649500b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f64900b5ac0 0x7f64900b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:29.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.385+0000 7f649500b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f64900b2770 0x7f64900b5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:29.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.385+0000 7f649500b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64900701e0 con 0x7f649015e130 2024-03-24T00:40:29.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.385+0000 7f649500b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64900d00b0 con 0x7f64900b2770 2024-03-24T00:40:29.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.385+0000 7f649500b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64900ce510 con 0x7f64900b5ac0 2024-03-24T00:40:29.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.385+0000 7f648f59e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f64900b2770 0x7f64900b5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:29.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.385+0000 7f648ed9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f649015e130 0x7f64900cde00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44138/0 (socket says 172.21.15.67:44138) 2024-03-24T00:40:29.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.385+0000 7f648ed9d640 1 -- 172.21.15.67:0/3890944899 learned_addr learned my addr 172.21.15.67:0/3890944899 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:40:29.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.385+0000 7f648e59c640 1 --2- 172.21.15.67:0/3890944899 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f64900b5ac0 0x7f64900b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:29.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.386+0000 7f648f59e640 1 -- 172.21.15.67:0/3890944899 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f64900b5ac0 msgr2=0x7f64900b4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:29.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.386+0000 7f648f59e640 1 --2- 172.21.15.67:0/3890944899 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f64900b5ac0 0x7f64900b4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:29.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.386+0000 7f648f59e640 1 -- 172.21.15.67:0/3890944899 >> v1:172.21.15.67:6789/0 conn(0x7f649015e130 legacy=0x7f64900cde00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:29.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.386+0000 7f648f59e640 1 -- 172.21.15.67:0/3890944899 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f649006b170 con 0x7f64900b2770 2024-03-24T00:40:29.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.386+0000 7f648dd9b640 1 -- 172.21.15.67:0/3890944899 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3804005350 0 0) 0x7f64900701e0 con 0x7f649015e130 2024-03-24T00:40:29.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.386+0000 7f648f59e640 1 --2- 172.21.15.67:0/3890944899 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64900b2770 0x7f64900b5460 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f6488009d90 tx=0x7f648802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39d4b0f10e073748 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:29.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.386+0000 7f648dd9b640 1 -- 172.21.15.67:0/3890944899 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6488039070 con 0x7f64900b2770 2024-03-24T00:40:29.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.387+0000 7f648dd9b640 1 -- 172.21.15.67:0/3890944899 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f648802cbc0 con 0x7f64900b2770 2024-03-24T00:40:29.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.387+0000 7f648dd9b640 1 -- 172.21.15.67:0/3890944899 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6488034440 con 0x7f64900b2770 2024-03-24T00:40:29.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.387+0000 7f649500b640 1 -- 172.21.15.67:0/3890944899 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64900b2770 msgr2=0x7f64900b5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:29.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.387+0000 7f649500b640 1 --2- 172.21.15.67:0/3890944899 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64900b2770 0x7f64900b5460 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f6488009d90 tx=0x7f648802be10 comp rx=0 tx=0).stop 2024-03-24T00:40:29.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.387+0000 7f649500b640 1 -- 172.21.15.67:0/3890944899 shutdown_connections 2024-03-24T00:40:29.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.387+0000 7f649500b640 1 --2- 172.21.15.67:0/3890944899 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64900b2770 0x7f64900b5460 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:29.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.387+0000 7f649500b640 1 --2- 172.21.15.67:0/3890944899 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f64900b5ac0 0x7f64900b4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:29.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.387+0000 7f649500b640 1 -- 172.21.15.67:0/3890944899 >> 172.21.15.67:0/3890944899 conn(0x7f64900b3c40 msgr2=0x7f64900b1dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:29.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.387+0000 7f649500b640 1 -- 172.21.15.67:0/3890944899 shutdown_connections 2024-03-24T00:40:29.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.387+0000 7f649500b640 1 -- 172.21.15.67:0/3890944899 wait complete. 2024-03-24T00:40:29.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.388+0000 7f649500b640 1 Processor -- start 2024-03-24T00:40:29.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.389+0000 7f649500b640 1 -- start start 2024-03-24T00:40:29.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.389+0000 7f649500b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64900b2770 0x7f64900c6b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:29.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.389+0000 7f649500b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64900b5ac0 0x7f64901f3b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:29.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.389+0000 7f649500b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f649015e130 0x7f649009ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:29.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.389+0000 7f649500b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f649006b7c0 con 0x7f64900b5ac0 2024-03-24T00:40:29.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.389+0000 7f649500b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64900704c0 con 0x7f64900b2770 2024-03-24T00:40:29.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.389+0000 7f649500b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f649015e510 con 0x7f649015e130 2024-03-24T00:40:29.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.389+0000 7f648e59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64900b5ac0 0x7f64901f3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:29.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.389+0000 7f648e59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64900b5ac0 0x7f64901f3b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39410/0 (socket says 172.21.15.67:39410) 2024-03-24T00:40:29.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.389+0000 7f648e59c640 1 -- 172.21.15.67:0/3082775948 learned_addr learned my addr 172.21.15.67:0/3082775948 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:29.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.389+0000 7f648ed9d640 1 --2- 172.21.15.67:0/3082775948 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64900b2770 0x7f64900c6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:29.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.389+0000 7f648f59e640 1 --2- 172.21.15.67:0/3082775948 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f649015e130 0x7f649009ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:29.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.390+0000 7f648e59c640 1 -- 172.21.15.67:0/3082775948 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f649015e130 msgr2=0x7f649009ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:29.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.390+0000 7f648e59c640 1 --2- 172.21.15.67:0/3082775948 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f649015e130 0x7f649009ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:29.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.390+0000 7f648e59c640 1 -- 172.21.15.67:0/3082775948 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64900b2770 msgr2=0x7f64900c6b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:29.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.390+0000 7f648e59c640 1 --2- 172.21.15.67:0/3082775948 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64900b2770 0x7f64900c6b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:29.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.390+0000 7f648e59c640 1 -- 172.21.15.67:0/3082775948 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64901f4250 con 0x7f64900b5ac0 2024-03-24T00:40:29.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.390+0000 7f648f59e640 1 --2- 172.21.15.67:0/3082775948 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f649015e130 0x7f649009ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:40:29.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.390+0000 7f648ed9d640 1 --2- 172.21.15.67:0/3082775948 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64900b2770 0x7f64900c6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:40:29.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.390+0000 7f648e59c640 1 --2- 172.21.15.67:0/3082775948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64900b5ac0 0x7f64901f3b70 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f647800e980 tx=0x7f647800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:29.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.390+0000 7f64877fe640 1 -- 172.21.15.67:0/3082775948 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6478017070 con 0x7f64900b5ac0 2024-03-24T00:40:29.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.390+0000 7f64877fe640 1 -- 172.21.15.67:0/3082775948 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f647800ca00 con 0x7f64900b5ac0 2024-03-24T00:40:29.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.390+0000 7f64877fe640 1 -- 172.21.15.67:0/3082775948 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f647800cdd0 con 0x7f64900b5ac0 2024-03-24T00:40:29.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.390+0000 7f649500b640 1 -- 172.21.15.67:0/3082775948 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64901f4530 con 0x7f64900b5ac0 2024-03-24T00:40:29.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.390+0000 7f649500b640 1 -- 172.21.15.67:0/3082775948 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f649009a350 con 0x7f64900b5ac0 2024-03-24T00:40:29.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.391+0000 7f649500b640 1 -- 172.21.15.67:0/3082775948 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f649004f400 con 0x7f64900b5ac0 2024-03-24T00:40:29.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.392+0000 7f64877fe640 1 -- 172.21.15.67:0/3082775948 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f647801a070 con 0x7f64900b5ac0 2024-03-24T00:40:29.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.392+0000 7f64877fe640 1 --2- 172.21.15.67:0/3082775948 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f647006bbd0 0x7f647006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:29.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.392+0000 7f64877fe640 1 -- 172.21.15.67:0/3082775948 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f647808ae50 con 0x7f64900b5ac0 2024-03-24T00:40:29.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.392+0000 7f648ed9d640 1 --2- 172.21.15.67:0/3082775948 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f647006bbd0 0x7f647006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:29.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.393+0000 7f648ed9d640 1 --2- 172.21.15.67:0/3082775948 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f647006bbd0 0x7f647006e090 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f6480007fd0 tx=0x7f6480007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:29.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.396+0000 7f64877fe640 1 -- 172.21.15.67:0/3082775948 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f64780591f0 con 0x7f64900b5ac0 2024-03-24T00:40:29.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.581+0000 7f649500b640 1 -- 172.21.15.67:0/3082775948 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64900cd9c0 con 0x7f64900b5ac0 2024-03-24T00:40:29.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.582+0000 7f64877fe640 1 -- 172.21.15.67:0/3082775948 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) v1 ==== 74+0+16882 (secure 0 0 0) 0x7f647805f030 con 0x7f64900b5ac0 2024-03-24T00:40:29.583 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:40:29.583 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":39,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:40:02.323736+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:39:58.975995+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-24T00:39:59.653615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:40:29.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.584+0000 7f649500b640 1 -- 172.21.15.67:0/3082775948 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f647006bbd0 msgr2=0x7f647006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:29.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.584+0000 7f649500b640 1 --2- 172.21.15.67:0/3082775948 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f647006bbd0 0x7f647006e090 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f6480007fd0 tx=0x7f6480007de0 comp rx=0 tx=0).stop 2024-03-24T00:40:29.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.584+0000 7f649500b640 1 -- 172.21.15.67:0/3082775948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64900b5ac0 msgr2=0x7f64901f3b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:29.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.584+0000 7f649500b640 1 --2- 172.21.15.67:0/3082775948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64900b5ac0 0x7f64901f3b70 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f647800e980 tx=0x7f647800ee40 comp rx=0 tx=0).stop 2024-03-24T00:40:29.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.584+0000 7f649500b640 1 -- 172.21.15.67:0/3082775948 shutdown_connections 2024-03-24T00:40:29.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.584+0000 7f649500b640 1 --2- 172.21.15.67:0/3082775948 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f647006bbd0 0x7f647006e090 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:29.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.584+0000 7f649500b640 1 --2- 172.21.15.67:0/3082775948 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f649015e130 0x7f649009ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:29.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.584+0000 7f649500b640 1 --2- 172.21.15.67:0/3082775948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64900b5ac0 0x7f64901f3b70 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:29.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.584+0000 7f649500b640 1 --2- 172.21.15.67:0/3082775948 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64900b2770 0x7f64900c6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:29.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.584+0000 7f649500b640 1 -- 172.21.15.67:0/3082775948 >> 172.21.15.67:0/3082775948 conn(0x7f64900b3c40 msgr2=0x7f649015c140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:29.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.584+0000 7f649500b640 1 -- 172.21.15.67:0/3082775948 shutdown_connections 2024-03-24T00:40:29.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.584+0000 7f649500b640 1 -- 172.21.15.67:0/3082775948 wait complete. 2024-03-24T00:40:29.596 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-24T00:40:29.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.698+0000 7fe88a04b640 1 Processor -- start 2024-03-24T00:40:29.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.698+0000 7fe88a04b640 1 -- start start 2024-03-24T00:40:29.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.699+0000 7fe88a04b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe8840b2010 0x7fe88415c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:29.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.699+0000 7fe88a04b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe8840ccd40 0x7fe8840cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:29.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.699+0000 7fe88a04b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe88415f890 con 0x7fe88406b040 2024-03-24T00:40:29.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.699+0000 7fe88a04b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8840b23f0 con 0x7fe8840b2010 2024-03-24T00:40:29.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.699+0000 7fe88a04b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe88415c850 con 0x7fe8840ccd40 2024-03-24T00:40:29.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.699+0000 7fe8837fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe8840b2010 0x7fe88415c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:29.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.699+0000 7fe8837fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe8840b2010 0x7fe88415c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39418/0 (socket says 172.21.15.67:39418) 2024-03-24T00:40:29.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.699+0000 7fe8837fe640 1 -- 172.21.15.67:0/1094041088 learned_addr learned my addr 172.21.15.67:0/1094041088 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:29.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.699+0000 7fe883fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fe88406b040 0x7fe88415fd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:38862/0 (socket says 172.21.15.67:38862) 2024-03-24T00:40:29.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.699+0000 7fe882ffd640 1 --2- 172.21.15.67:0/1094041088 >> v2:172.21.15.72:3300/0 conn(0x7fe8840ccd40 0x7fe8840cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:29.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.699+0000 7fe8827fc640 1 -- 172.21.15.67:0/1094041088 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 998400475 0 0) 0x7fe88415f890 con 0x7fe88406b040 2024-03-24T00:40:29.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.699+0000 7fe8837fe640 1 -- 172.21.15.67:0/1094041088 >> v2:172.21.15.72:3300/0 conn(0x7fe8840ccd40 msgr2=0x7fe8840cd4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:29.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.699+0000 7fe8837fe640 1 --2- 172.21.15.67:0/1094041088 >> v2:172.21.15.72:3300/0 conn(0x7fe8840ccd40 0x7fe8840cd4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:29.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.699+0000 7fe8837fe640 1 -- 172.21.15.67:0/1094041088 >> v1:172.21.15.72:6789/0 conn(0x7fe88406b040 legacy=0x7fe88415fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:29.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.699+0000 7fe8837fe640 1 -- 172.21.15.67:0/1094041088 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe88415cb30 con 0x7fe8840b2010 2024-03-24T00:40:29.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.700+0000 7fe8837fe640 1 --2- 172.21.15.67:0/1094041088 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8840b2010 0x7fe88415c1b0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fe88406e480 tx=0x7fe87402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7632cf282ab35cb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:29.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.700+0000 7fe8827fc640 1 -- 172.21.15.67:0/1094041088 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe874035020 con 0x7fe8840b2010 2024-03-24T00:40:29.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.700+0000 7fe8827fc640 1 -- 172.21.15.67:0/1094041088 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe874031870 con 0x7fe8840b2010 2024-03-24T00:40:29.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.700+0000 7fe8827fc640 1 -- 172.21.15.67:0/1094041088 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe874031c40 con 0x7fe8840b2010 2024-03-24T00:40:29.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.701+0000 7fe88a04b640 1 -- 172.21.15.67:0/1094041088 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8840b2010 msgr2=0x7fe88415c1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:29.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.701+0000 7fe88a04b640 1 --2- 172.21.15.67:0/1094041088 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8840b2010 0x7fe88415c1b0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fe88406e480 tx=0x7fe87402b910 comp rx=0 tx=0).stop 2024-03-24T00:40:29.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.701+0000 7fe88a04b640 1 -- 172.21.15.67:0/1094041088 shutdown_connections 2024-03-24T00:40:29.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.701+0000 7fe88a04b640 1 --2- 172.21.15.67:0/1094041088 >> v2:172.21.15.72:3300/0 conn(0x7fe8840ccd40 0x7fe8840cd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:29.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.701+0000 7fe88a04b640 1 --2- 172.21.15.67:0/1094041088 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8840b2010 0x7fe88415c1b0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:29.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.701+0000 7fe88a04b640 1 -- 172.21.15.67:0/1094041088 >> 172.21.15.67:0/1094041088 conn(0x7fe884099a30 msgr2=0x7fe884099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:29.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.701+0000 7fe88a04b640 1 -- 172.21.15.67:0/1094041088 shutdown_connections 2024-03-24T00:40:29.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.701+0000 7fe88a04b640 1 -- 172.21.15.67:0/1094041088 wait complete. 2024-03-24T00:40:29.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.702+0000 7fe88a04b640 1 Processor -- start 2024-03-24T00:40:29.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.702+0000 7fe88a04b640 1 -- start start 2024-03-24T00:40:29.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.703+0000 7fe88a04b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe88406b040 0x7fe8840b0050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:29.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.703+0000 7fe88a04b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe8840b2010 0x7fe88406eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:29.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.703+0000 7fe88a04b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8840ccd40 0x7fe88406f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:29.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.703+0000 7fe88a04b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe88415d0f0 con 0x7fe88406b040 2024-03-24T00:40:29.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.703+0000 7fe88a04b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8840cd1d0 con 0x7fe8840ccd40 2024-03-24T00:40:29.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.703+0000 7fe88a04b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8840cd9f0 con 0x7fe8840b2010 2024-03-24T00:40:29.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.703+0000 7fe8837fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe88406b040 0x7fe8840b0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:29.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.703+0000 7fe8837fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe88406b040 0x7fe8840b0050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39420/0 (socket says 172.21.15.67:39420) 2024-03-24T00:40:29.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.703+0000 7fe8837fe640 1 -- 172.21.15.67:0/3277121753 learned_addr learned my addr 172.21.15.67:0/3277121753 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:29.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.703+0000 7fe882ffd640 1 --2- 172.21.15.67:0/3277121753 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe8840b2010 0x7fe88406eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:29.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.703+0000 7fe883fff640 1 --2- 172.21.15.67:0/3277121753 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8840ccd40 0x7fe88406f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:29.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.703+0000 7fe8837fe640 1 -- 172.21.15.67:0/3277121753 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe8840b2010 msgr2=0x7fe88406eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:29.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.703+0000 7fe8837fe640 1 --2- 172.21.15.67:0/3277121753 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe8840b2010 0x7fe88406eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:29.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.703+0000 7fe8837fe640 1 -- 172.21.15.67:0/3277121753 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8840ccd40 msgr2=0x7fe88406f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:29.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.703+0000 7fe8837fe640 1 --2- 172.21.15.67:0/3277121753 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8840ccd40 0x7fe88406f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:29.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.703+0000 7fe8837fe640 1 -- 172.21.15.67:0/3277121753 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8840b6590 con 0x7fe88406b040 2024-03-24T00:40:29.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.703+0000 7fe8837fe640 1 --2- 172.21.15.67:0/3277121753 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe88406b040 0x7fe8840b0050 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fe87402b950 tx=0x7fe8740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:29.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.703+0000 7fe86bfff640 1 -- 172.21.15.67:0/3277121753 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe874040070 con 0x7fe88406b040 2024-03-24T00:40:29.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.703+0000 7fe86bfff640 1 -- 172.21.15.67:0/3277121753 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe874030d00 con 0x7fe88406b040 2024-03-24T00:40:29.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.704+0000 7fe86bfff640 1 -- 172.21.15.67:0/3277121753 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe874035040 con 0x7fe88406b040 2024-03-24T00:40:29.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.704+0000 7fe88a04b640 1 -- 172.21.15.67:0/3277121753 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe88406f880 con 0x7fe88406b040 2024-03-24T00:40:29.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.704+0000 7fe88a04b640 1 -- 172.21.15.67:0/3277121753 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8840c93d0 con 0x7fe88406b040 2024-03-24T00:40:29.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.704+0000 7fe88a04b640 1 -- 172.21.15.67:0/3277121753 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe88404f400 con 0x7fe88406b040 2024-03-24T00:40:29.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.705+0000 7fe86bfff640 1 -- 172.21.15.67:0/3277121753 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe874038020 con 0x7fe88406b040 2024-03-24T00:40:29.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.705+0000 7fe86bfff640 1 --2- 172.21.15.67:0/3277121753 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe87006bbd0 0x7fe87006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:29.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.705+0000 7fe86bfff640 1 -- 172.21.15.67:0/3277121753 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe8740ac8c0 con 0x7fe88406b040 2024-03-24T00:40:29.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.705+0000 7fe882ffd640 1 --2- 172.21.15.67:0/3277121753 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe87006bbd0 0x7fe87006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:29.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.706+0000 7fe882ffd640 1 --2- 172.21.15.67:0/3277121753 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe87006bbd0 0x7fe87006e090 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fe88414e240 tx=0x7fe878009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:29.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.710+0000 7fe86bfff640 1 -- 172.21.15.67:0/3277121753 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe8840b6590 con 0x7fe88406b040 2024-03-24T00:40:29.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.932+0000 7fe88a04b640 1 -- 172.21.15.67:0/3277121753 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fe88406be60 con 0x7fe88406b040 2024-03-24T00:40:29.933 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-24T00:40:29.933+0000 7fbfcfa11640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T00:40:29.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.934+0000 7fe86bfff640 1 -- 172.21.15.67:0/3277121753 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v18) v1 ==== 161+0+0 (secure 0 0 0) 0x7fe87407ac60 con 0x7fe88406b040 2024-03-24T00:40:29.935 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-24T00:40:29.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.937+0000 7fe88a04b640 1 -- 172.21.15.67:0/3277121753 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe87006bbd0 msgr2=0x7fe87006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:29.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.937+0000 7fe88a04b640 1 --2- 172.21.15.67:0/3277121753 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe87006bbd0 0x7fe87006e090 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fe88414e240 tx=0x7fe878009190 comp rx=0 tx=0).stop 2024-03-24T00:40:29.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.937+0000 7fe88a04b640 1 -- 172.21.15.67:0/3277121753 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe88406b040 msgr2=0x7fe8840b0050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:29.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.937+0000 7fe88a04b640 1 --2- 172.21.15.67:0/3277121753 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe88406b040 0x7fe8840b0050 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fe87402b950 tx=0x7fe8740046c0 comp rx=0 tx=0).stop 2024-03-24T00:40:29.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.937+0000 7fe88a04b640 1 -- 172.21.15.67:0/3277121753 shutdown_connections 2024-03-24T00:40:29.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.937+0000 7fe88a04b640 1 --2- 172.21.15.67:0/3277121753 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe87006bbd0 0x7fe87006e090 secure :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fe88414e240 tx=0x7fe878009190 comp rx=0 tx=0).stop 2024-03-24T00:40:29.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.937+0000 7fe88a04b640 1 --2- 172.21.15.67:0/3277121753 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8840ccd40 0x7fe88406f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:29.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.937+0000 7fe88a04b640 1 --2- 172.21.15.67:0/3277121753 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe8840b2010 0x7fe88406eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:29.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.937+0000 7fe88a04b640 1 --2- 172.21.15.67:0/3277121753 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe88406b040 0x7fe8840b0050 secure :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fe87402b950 tx=0x7fe8740046c0 comp rx=0 tx=0).stop 2024-03-24T00:40:29.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.937+0000 7fe88a04b640 1 -- 172.21.15.67:0/3277121753 >> 172.21.15.67:0/3277121753 conn(0x7fe884099a30 msgr2=0x7fe88415a270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:29.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.937+0000 7fe88a04b640 1 -- 172.21.15.67:0/3277121753 shutdown_connections 2024-03-24T00:40:29.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:29.937+0000 7fe88a04b640 1 -- 172.21.15.67:0/3277121753 wait complete. 2024-03-24T00:40:29.941 INFO:tasks.ceph.mds.d.smithi072.stderr: -6788> 2024-03-24T00:40:06.004+0000 7fc75cacc640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T00:40:29.949 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-24T00:40:29.970 INFO:tasks.ceph.mds.d.smithi072.stdout:starting mds.d at 2024-03-24T00:40:30.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.051+0000 7f4971ca1640 1 Processor -- start 2024-03-24T00:40:30.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.051+0000 7f4971ca1640 1 -- start start 2024-03-24T00:40:30.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.052+0000 7f4971ca1640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f496c15bc30 0x7f496c0b1a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:30.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.052+0000 7f4971ca1640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f496c15c010 0x7f496c0b1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:30.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.052+0000 7f4971ca1640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f496c0b3ae0 con 0x7f496c0b2d90 2024-03-24T00:40:30.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.052+0000 7f4971ca1640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f496c0cca60 con 0x7f496c15c010 2024-03-24T00:40:30.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.052+0000 7f4971ca1640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f496c0b3dc0 con 0x7f496c15bc30 2024-03-24T00:40:30.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.052+0000 7f496affd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f496c15c010 0x7f496c0b1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:30.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.052+0000 7f496affd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f496c15c010 0x7f496c0b1f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39430/0 (socket says 172.21.15.67:39430) 2024-03-24T00:40:30.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.052+0000 7f496affd640 1 -- 172.21.15.67:0/2908889570 learned_addr learned my addr 172.21.15.67:0/2908889570 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:30.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.052+0000 7f496b7fe640 1 --2- 172.21.15.67:0/2908889570 >> v2:172.21.15.72:3300/0 conn(0x7f496c15bc30 0x7f496c0b1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:30.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.053+0000 7f496affd640 1 -- 172.21.15.67:0/2908889570 >> v2:172.21.15.72:3300/0 conn(0x7f496c15bc30 msgr2=0x7f496c0b1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:30.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.053+0000 7f496affd640 1 --2- 172.21.15.67:0/2908889570 >> v2:172.21.15.72:3300/0 conn(0x7f496c15bc30 0x7f496c0b1a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:30.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.053+0000 7f496affd640 1 -- 172.21.15.67:0/2908889570 >> v1:172.21.15.72:6789/0 conn(0x7f496c0b2d90 legacy=0x7f496c0b3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:30.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.053+0000 7f496affd640 1 -- 172.21.15.67:0/2908889570 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f496c0b40a0 con 0x7f496c15c010 2024-03-24T00:40:30.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.053+0000 7f496affd640 1 --2- 172.21.15.67:0/2908889570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f496c15c010 0x7f496c0b1f40 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f496c0b4700 tx=0x7f495402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6327e3be154bd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:30.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.053+0000 7f496a7fc640 1 -- 172.21.15.67:0/2908889570 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4954034070 con 0x7f496c15c010 2024-03-24T00:40:30.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.053+0000 7f496a7fc640 1 -- 172.21.15.67:0/2908889570 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4954030860 con 0x7f496c15c010 2024-03-24T00:40:30.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.053+0000 7f496a7fc640 1 -- 172.21.15.67:0/2908889570 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4954030c30 con 0x7f496c15c010 2024-03-24T00:40:30.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.054+0000 7f4971ca1640 1 -- 172.21.15.67:0/2908889570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f496c15c010 msgr2=0x7f496c0b1f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:30.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.054+0000 7f4971ca1640 1 --2- 172.21.15.67:0/2908889570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f496c15c010 0x7f496c0b1f40 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f496c0b4700 tx=0x7f495402b9a0 comp rx=0 tx=0).stop 2024-03-24T00:40:30.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.054+0000 7f4971ca1640 1 -- 172.21.15.67:0/2908889570 shutdown_connections 2024-03-24T00:40:30.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.054+0000 7f4971ca1640 1 --2- 172.21.15.67:0/2908889570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f496c15c010 0x7f496c0b1f40 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:30.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.054+0000 7f4971ca1640 1 --2- 172.21.15.67:0/2908889570 >> v2:172.21.15.72:3300/0 conn(0x7f496c15bc30 0x7f496c0b1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:30.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.054+0000 7f4971ca1640 1 -- 172.21.15.67:0/2908889570 >> 172.21.15.67:0/2908889570 conn(0x7f496c08b180 msgr2=0x7f496c08b590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:30.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.054+0000 7f4971ca1640 1 -- 172.21.15.67:0/2908889570 shutdown_connections 2024-03-24T00:40:30.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.054+0000 7f4971ca1640 1 -- 172.21.15.67:0/2908889570 wait complete. 2024-03-24T00:40:30.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.055+0000 7f4971ca1640 1 Processor -- start 2024-03-24T00:40:30.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.055+0000 7f4971ca1640 1 -- start start 2024-03-24T00:40:30.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.056+0000 7f4971ca1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f496c0b2d90 0x7f496c152970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:30.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.056+0000 7f4971ca1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f496c15bc30 0x7f496c15eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:30.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.056+0000 7f4971ca1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f496c15c010 0x7f496c0cfce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:30.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.056+0000 7f4971ca1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f496c0cd070 con 0x7f496c15c010 2024-03-24T00:40:30.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.056+0000 7f4971ca1640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f496c0ccef0 con 0x7f496c15bc30 2024-03-24T00:40:30.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.056+0000 7f4971ca1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f496c0cd1f0 con 0x7f496c0b2d90 2024-03-24T00:40:30.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.056+0000 7f496bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f496c15c010 0x7f496c0cfce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:30.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.056+0000 7f496bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f496c15c010 0x7f496c0cfce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39434/0 (socket says 172.21.15.67:39434) 2024-03-24T00:40:30.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.056+0000 7f496bfff640 1 -- 172.21.15.67:0/1031617656 learned_addr learned my addr 172.21.15.67:0/1031617656 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:30.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.056+0000 7f496b7fe640 1 --2- 172.21.15.67:0/1031617656 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f496c0b2d90 0x7f496c152970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:30.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.056+0000 7f496affd640 1 --2- 172.21.15.67:0/1031617656 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f496c15bc30 0x7f496c15eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:30.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.056+0000 7f496bfff640 1 -- 172.21.15.67:0/1031617656 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f496c0b2d90 msgr2=0x7f496c152970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:30.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.056+0000 7f496bfff640 1 --2- 172.21.15.67:0/1031617656 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f496c0b2d90 0x7f496c152970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:30.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.056+0000 7f496bfff640 1 -- 172.21.15.67:0/1031617656 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f496c15bc30 msgr2=0x7f496c15eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:30.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.056+0000 7f496bfff640 1 --2- 172.21.15.67:0/1031617656 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f496c15bc30 0x7f496c15eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:30.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.056+0000 7f496bfff640 1 -- 172.21.15.67:0/1031617656 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f496c0cb450 con 0x7f496c15c010 2024-03-24T00:40:30.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.057+0000 7f496bfff640 1 --2- 172.21.15.67:0/1031617656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f496c15c010 0x7f496c0cfce0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f496400c400 tx=0x7f496400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:30.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.057+0000 7f4963fff640 1 -- 172.21.15.67:0/1031617656 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4964017070 con 0x7f496c15c010 2024-03-24T00:40:30.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.057+0000 7f4963fff640 1 -- 172.21.15.67:0/1031617656 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49640052d0 con 0x7f496c15c010 2024-03-24T00:40:30.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.057+0000 7f4971ca1640 1 -- 172.21.15.67:0/1031617656 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f496c0cb730 con 0x7f496c15c010 2024-03-24T00:40:30.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.057+0000 7f4963fff640 1 -- 172.21.15.67:0/1031617656 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4964003810 con 0x7f496c15c010 2024-03-24T00:40:30.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.057+0000 7f4971ca1640 1 -- 172.21.15.67:0/1031617656 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f496c152fe0 con 0x7f496c15c010 2024-03-24T00:40:30.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.058+0000 7f4971ca1640 1 -- 172.21.15.67:0/1031617656 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f496c04f400 con 0x7f496c15c010 2024-03-24T00:40:30.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.058+0000 7f4963fff640 1 -- 172.21.15.67:0/1031617656 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f496401a070 con 0x7f496c15c010 2024-03-24T00:40:30.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.059+0000 7f4963fff640 1 --2- 172.21.15.67:0/1031617656 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f494c06bc00 0x7f494c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:30.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.059+0000 7f4963fff640 1 -- 172.21.15.67:0/1031617656 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f4964089fe0 con 0x7f496c15c010 2024-03-24T00:40:30.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.059+0000 7f496b7fe640 1 --2- 172.21.15.67:0/1031617656 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f494c06bc00 0x7f494c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:30.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.059+0000 7f496b7fe640 1 --2- 172.21.15.67:0/1031617656 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f494c06bc00 0x7f494c06e0c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f495c0113e0 tx=0x7f495c00b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:30.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.063+0000 7f4963fff640 1 -- 172.21.15.67:0/1031617656 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4964058210 con 0x7f496c15c010 2024-03-24T00:40:30.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.284+0000 7f4971ca1640 1 -- 172.21.15.67:0/1031617656 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f496c0c8b80 con 0x7f496c15c010 2024-03-24T00:40:30.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.936+0000 7f4963fff640 1 -- 172.21.15.67:0/1031617656 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v19) v1 ==== 104+0+0 (secure 0 0 0) 0x7f496405c2b0 con 0x7f496c15c010 2024-03-24T00:40:30.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.938+0000 7f4971ca1640 1 -- 172.21.15.67:0/1031617656 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f494c06bc00 msgr2=0x7f494c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:30.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.938+0000 7f4971ca1640 1 --2- 172.21.15.67:0/1031617656 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f494c06bc00 0x7f494c06e0c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f495c0113e0 tx=0x7f495c00b040 comp rx=0 tx=0).stop 2024-03-24T00:40:30.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.938+0000 7f4971ca1640 1 -- 172.21.15.67:0/1031617656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f496c15c010 msgr2=0x7f496c0cfce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:30.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.938+0000 7f4971ca1640 1 --2- 172.21.15.67:0/1031617656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f496c15c010 0x7f496c0cfce0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f496400c400 tx=0x7f496400c8c0 comp rx=0 tx=0).stop 2024-03-24T00:40:30.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.938+0000 7f4971ca1640 1 -- 172.21.15.67:0/1031617656 shutdown_connections 2024-03-24T00:40:30.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.938+0000 7f4971ca1640 1 --2- 172.21.15.67:0/1031617656 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f494c06bc00 0x7f494c06e0c0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:30.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.938+0000 7f4971ca1640 1 --2- 172.21.15.67:0/1031617656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f496c15c010 0x7f496c0cfce0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:30.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.938+0000 7f4971ca1640 1 --2- 172.21.15.67:0/1031617656 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f496c15bc30 0x7f496c15eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:30.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.938+0000 7f4971ca1640 1 --2- 172.21.15.67:0/1031617656 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f496c0b2d90 0x7f496c152970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:30.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.938+0000 7f4971ca1640 1 -- 172.21.15.67:0/1031617656 >> 172.21.15.67:0/1031617656 conn(0x7f496c08b180 msgr2=0x7f496c0702d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:30.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.938+0000 7f4971ca1640 1 -- 172.21.15.67:0/1031617656 shutdown_connections 2024-03-24T00:40:30.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:30.939+0000 7f4971ca1640 1 -- 172.21.15.67:0/1031617656 wait complete. 2024-03-24T00:40:30.953 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-24T00:40:31.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.064+0000 7ff04d95b640 1 Processor -- start 2024-03-24T00:40:31.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.064+0000 7ff04d95b640 1 -- start start 2024-03-24T00:40:31.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.065+0000 7ff04d95b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff04815b500 0x7ff04815b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:31.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.065+0000 7ff04d95b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff048071ad0 con 0x7ff0480b2a10 2024-03-24T00:40:31.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.065+0000 7ff04d95b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff048071d90 con 0x7ff04806ec40 2024-03-24T00:40:31.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.065+0000 7ff04d95b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff048072070 con 0x7ff04815b500 2024-03-24T00:40:31.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.065+0000 7ff0477fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff0480b2a10 0x7ff0480b2e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44148/0 (socket says 172.21.15.67:44148) 2024-03-24T00:40:31.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.065+0000 7ff0477fe640 1 -- 172.21.15.67:0/1299890191 learned_addr learned my addr 172.21.15.67:0/1299890191 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:40:31.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.065+0000 7ff0467fc640 1 --2- 172.21.15.67:0/1299890191 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff04815b500 0x7ff04815b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:31.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.065+0000 7ff045ffb640 1 -- 172.21.15.67:0/1299890191 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1142153674 0 0) 0x7ff048071ad0 con 0x7ff0480b2a10 2024-03-24T00:40:31.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.066+0000 7ff045ffb640 1 -- 172.21.15.67:0/1299890191 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff034003520 con 0x7ff0480b2a10 2024-03-24T00:40:31.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.066+0000 7ff045ffb640 1 -- 172.21.15.67:0/1299890191 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3933712698 0 0) 0x7ff034003520 con 0x7ff0480b2a10 2024-03-24T00:40:31.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.066+0000 7ff045ffb640 1 -- 172.21.15.67:0/1299890191 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff04815b500 msgr2=0x7ff04815b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:31.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.066+0000 7ff045ffb640 1 --2- 172.21.15.67:0/1299890191 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff04815b500 0x7ff04815b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:31.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.066+0000 7ff045ffb640 1 -- 172.21.15.67:0/1299890191 >> v1:172.21.15.72:6789/0 conn(0x7ff04806ec40 legacy=0x7ff04806f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:31.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.066+0000 7ff045ffb640 1 -- 172.21.15.67:0/1299890191 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff048072350 con 0x7ff0480b2a10 2024-03-24T00:40:31.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.066+0000 7ff045ffb640 1 -- 172.21.15.67:0/1299890191 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7ff0380019f0 con 0x7ff0480b2a10 2024-03-24T00:40:31.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.066+0000 7ff045ffb640 1 -- 172.21.15.67:0/1299890191 >> v1:172.21.15.67:6789/0 conn(0x7ff0480b2a10 legacy=0x7ff0480b2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:31.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.066+0000 7ff045ffb640 1 --2- 172.21.15.67:0/1299890191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0340039e0 0x7ff034003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:31.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.066+0000 7ff045ffb640 1 --2- 172.21.15.67:0/1299890191 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff034004390 0x7ff034004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:31.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.066+0000 7ff045ffb640 1 --2- 172.21.15.67:0/1299890191 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff04815b500 0x7ff0340050c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:31.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.066+0000 7ff045ffb640 1 -- 172.21.15.67:0/1299890191 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff048072070 con 0x7ff0340039e0 2024-03-24T00:40:31.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.066+0000 7ff045ffb640 1 -- 172.21.15.67:0/1299890191 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff03400b9f0 con 0x7ff034004390 2024-03-24T00:40:31.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.066+0000 7ff045ffb640 1 -- 172.21.15.67:0/1299890191 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff03400bbf0 con 0x7ff04815b500 2024-03-24T00:40:31.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.066+0000 7ff046ffd640 1 --2- 172.21.15.67:0/1299890191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0340039e0 0x7ff034003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:31.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.066+0000 7ff0477fe640 1 --2- 172.21.15.67:0/1299890191 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff04815b500 0x7ff0340050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:31.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.066+0000 7ff0467fc640 1 --2- 172.21.15.67:0/1299890191 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff034004390 0x7ff034004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:31.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.066+0000 7ff046ffd640 1 -- 172.21.15.67:0/1299890191 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff04815b500 msgr2=0x7ff0340050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:31.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.066+0000 7ff046ffd640 1 --2- 172.21.15.67:0/1299890191 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff04815b500 0x7ff0340050c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:31.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.066+0000 7ff046ffd640 1 -- 172.21.15.67:0/1299890191 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff034004390 msgr2=0x7ff034004860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:31.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.066+0000 7ff046ffd640 1 --2- 172.21.15.67:0/1299890191 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff034004390 0x7ff034004860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:31.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.066+0000 7ff046ffd640 1 -- 172.21.15.67:0/1299890191 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff034003520 con 0x7ff0340039e0 2024-03-24T00:40:31.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.066+0000 7ff0467fc640 1 --2- 172.21.15.67:0/1299890191 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff034004390 0x7ff034004860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:40:31.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.067+0000 7ff046ffd640 1 --2- 172.21.15.67:0/1299890191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0340039e0 0x7ff034003dc0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7ff040001860 tx=0x7ff034005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17450f8a10ea7569 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:31.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.067+0000 7ff045ffb640 1 -- 172.21.15.67:0/1299890191 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff04002f050 con 0x7ff0340039e0 2024-03-24T00:40:31.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.067+0000 7ff045ffb640 1 -- 172.21.15.67:0/1299890191 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff040033650 con 0x7ff0340039e0 2024-03-24T00:40:31.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.067+0000 7ff045ffb640 1 -- 172.21.15.67:0/1299890191 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff040001430 con 0x7ff0340039e0 2024-03-24T00:40:31.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.068+0000 7ff04d95b640 1 -- 172.21.15.67:0/1299890191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0340039e0 msgr2=0x7ff034003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:31.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.068+0000 7ff04d95b640 1 --2- 172.21.15.67:0/1299890191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0340039e0 0x7ff034003dc0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7ff040001860 tx=0x7ff034005830 comp rx=0 tx=0).stop 2024-03-24T00:40:31.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.068+0000 7ff0477fe640 1 -- 172.21.15.67:0/1299890191 reap_dead start 2024-03-24T00:40:31.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.068+0000 7ff04d95b640 1 -- 172.21.15.67:0/1299890191 shutdown_connections 2024-03-24T00:40:31.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.068+0000 7ff04d95b640 1 -- 172.21.15.67:0/1299890191 >> 172.21.15.67:0/1299890191 conn(0x7ff0480cf9e0 msgr2=0x7ff0480cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:31.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.068+0000 7ff04d95b640 1 -- 172.21.15.67:0/1299890191 shutdown_connections 2024-03-24T00:40:31.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.068+0000 7ff04d95b640 1 -- 172.21.15.67:0/1299890191 wait complete. 2024-03-24T00:40:31.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.069+0000 7ff04d95b640 1 Processor -- start 2024-03-24T00:40:31.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.069+0000 7ff04d95b640 1 -- start start 2024-03-24T00:40:31.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.070+0000 7ff04d95b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff0340039e0 0x7ff0481e4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:31.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.070+0000 7ff04d95b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff04815df40 0x7ff048159fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:31.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.070+0000 7ff04d95b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff048140ff0 0x7ff048141470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:31.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.070+0000 7ff04d95b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0480b4f80 con 0x7ff048140ff0 2024-03-24T00:40:31.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.070+0000 7ff04d95b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0480b4e00 con 0x7ff04815df40 2024-03-24T00:40:31.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.070+0000 7ff04d95b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff04815b990 con 0x7ff0340039e0 2024-03-24T00:40:31.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.070+0000 7ff0477fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff048140ff0 0x7ff048141470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:31.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.070+0000 7ff0477fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff048140ff0 0x7ff048141470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39450/0 (socket says 172.21.15.67:39450) 2024-03-24T00:40:31.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.070+0000 7ff0477fe640 1 -- 172.21.15.67:0/691279958 learned_addr learned my addr 172.21.15.67:0/691279958 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:31.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.070+0000 7ff046ffd640 1 --2- 172.21.15.67:0/691279958 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff0340039e0 0x7ff0481e4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:31.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.070+0000 7ff0477fe640 1 -- 172.21.15.67:0/691279958 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff0340039e0 msgr2=0x7ff0481e4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:31.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.070+0000 7ff0477fe640 1 --2- 172.21.15.67:0/691279958 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff0340039e0 0x7ff0481e4f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:31.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.070+0000 7ff0467fc640 1 --2- 172.21.15.67:0/691279958 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff04815df40 0x7ff048159fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:31.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.070+0000 7ff0477fe640 1 -- 172.21.15.67:0/691279958 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff04815df40 msgr2=0x7ff048159fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:31.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.070+0000 7ff0477fe640 1 --2- 172.21.15.67:0/691279958 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff04815df40 0x7ff048159fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:31.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.070+0000 7ff0477fe640 1 -- 172.21.15.67:0/691279958 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0481e5630 con 0x7ff048140ff0 2024-03-24T00:40:31.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.070+0000 7ff0467fc640 1 --2- 172.21.15.67:0/691279958 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff04815df40 0x7ff048159fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:40:31.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.071+0000 7ff0477fe640 1 --2- 172.21.15.67:0/691279958 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff048140ff0 0x7ff048141470 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7ff0380013d0 tx=0x7ff038007b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:31.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.071+0000 7ff03ffff640 1 -- 172.21.15.67:0/691279958 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff038017020 con 0x7ff048140ff0 2024-03-24T00:40:31.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.071+0000 7ff03ffff640 1 -- 172.21.15.67:0/691279958 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff03800abe0 con 0x7ff048140ff0 2024-03-24T00:40:31.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.071+0000 7ff04d95b640 1 -- 172.21.15.67:0/691279958 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff04815ce40 con 0x7ff048140ff0 2024-03-24T00:40:31.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.071+0000 7ff03ffff640 1 -- 172.21.15.67:0/691279958 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0380133f0 con 0x7ff048140ff0 2024-03-24T00:40:31.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.071+0000 7ff04d95b640 1 -- 172.21.15.67:0/691279958 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff04815d400 con 0x7ff048140ff0 2024-03-24T00:40:31.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.072+0000 7ff04d95b640 1 -- 172.21.15.67:0/691279958 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff048158480 con 0x7ff048140ff0 2024-03-24T00:40:31.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.072+0000 7ff03ffff640 1 -- 172.21.15.67:0/691279958 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff03801a020 con 0x7ff048140ff0 2024-03-24T00:40:31.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.072+0000 7ff03ffff640 1 --2- 172.21.15.67:0/691279958 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff02806bc00 0x7ff02806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:31.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.073+0000 7ff03ffff640 1 -- 172.21.15.67:0/691279958 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7ff03808b310 con 0x7ff048140ff0 2024-03-24T00:40:31.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.073+0000 7ff046ffd640 1 --2- 172.21.15.67:0/691279958 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff02806bc00 0x7ff02806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:31.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.073+0000 7ff046ffd640 1 --2- 172.21.15.67:0/691279958 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff02806bc00 0x7ff02806e0c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ff040036000 tx=0x7ff040031000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:31.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.077+0000 7ff03ffff640 1 -- 172.21.15.67:0/691279958 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff048158480 con 0x7ff048140ff0 2024-03-24T00:40:31.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.261+0000 7ff04d95b640 1 -- 172.21.15.67:0/691279958 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff04804f400 con 0x7ff048140ff0 2024-03-24T00:40:31.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.941+0000 7ff03ffff640 1 -- 172.21.15.67:0/691279958 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v41) v1 ==== 181+0+0 (secure 0 0 0) 0x7ff038021020 con 0x7ff048140ff0 2024-03-24T00:40:31.942 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' removed 2024-03-24T00:40:31.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.945+0000 7ff04d95b640 1 -- 172.21.15.67:0/691279958 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff02806bc00 msgr2=0x7ff02806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:31.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.945+0000 7ff04d95b640 1 --2- 172.21.15.67:0/691279958 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff02806bc00 0x7ff02806e0c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ff040036000 tx=0x7ff040031000 comp rx=0 tx=0).stop 2024-03-24T00:40:31.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.945+0000 7ff04d95b640 1 -- 172.21.15.67:0/691279958 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff048140ff0 msgr2=0x7ff048141470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:31.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.945+0000 7ff04d95b640 1 --2- 172.21.15.67:0/691279958 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff048140ff0 0x7ff048141470 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7ff0380013d0 tx=0x7ff038007b20 comp rx=0 tx=0).stop 2024-03-24T00:40:31.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.945+0000 7ff04d95b640 1 -- 172.21.15.67:0/691279958 shutdown_connections 2024-03-24T00:40:31.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.945+0000 7ff04d95b640 1 --2- 172.21.15.67:0/691279958 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff02806bc00 0x7ff02806e0c0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:31.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.945+0000 7ff04d95b640 1 --2- 172.21.15.67:0/691279958 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff048140ff0 0x7ff048141470 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:31.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.945+0000 7ff04d95b640 1 --2- 172.21.15.67:0/691279958 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff04815df40 0x7ff048159fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:31.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.945+0000 7ff04d95b640 1 --2- 172.21.15.67:0/691279958 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff0340039e0 0x7ff0481e4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:31.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.945+0000 7ff04d95b640 1 -- 172.21.15.67:0/691279958 >> 172.21.15.67:0/691279958 conn(0x7ff0480cf9e0 msgr2=0x7ff04815e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:31.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.945+0000 7ff04d95b640 1 -- 172.21.15.67:0/691279958 shutdown_connections 2024-03-24T00:40:31.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:31.945+0000 7ff04d95b640 1 -- 172.21.15.67:0/691279958 wait complete. 2024-03-24T00:40:31.963 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-24T00:40:32.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.073+0000 7fd4c507e640 1 Processor -- start 2024-03-24T00:40:32.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.073+0000 7fd4c507e640 1 -- start start 2024-03-24T00:40:32.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.074+0000 7fd4c507e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd4c006f350 0x7fd4c00725b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:32.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.074+0000 7fd4c507e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4c00b4b40 con 0x7fd4c015bed0 2024-03-24T00:40:32.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.074+0000 7fd4c507e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4c00b4e20 con 0x7fd4c01589a0 2024-03-24T00:40:32.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.074+0000 7fd4c507e640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c0072af0 con 0x7fd4c006f350 2024-03-24T00:40:32.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.074+0000 7fd4be59c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd4c015bed0 0x7fd4c006ec40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44150/0 (socket says 172.21.15.67:44150) 2024-03-24T00:40:32.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.074+0000 7fd4be59c640 1 -- 172.21.15.67:0/3998508264 learned_addr learned my addr 172.21.15.67:0/3998508264 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:40:32.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.074+0000 7fd4bdd9b640 1 -- 172.21.15.67:0/3998508264 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2868883870 0 0) 0x7fd4c00b4b40 con 0x7fd4c015bed0 2024-03-24T00:40:32.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.074+0000 7fd4bdd9b640 1 -- 172.21.15.67:0/3998508264 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4ac003520 con 0x7fd4c015bed0 2024-03-24T00:40:32.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.074+0000 7fd4bf59e640 1 --2- 172.21.15.67:0/3998508264 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd4c006f350 0x7fd4c00725b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:32.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.074+0000 7fd4bdd9b640 1 -- 172.21.15.67:0/3998508264 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2194702511 0 0) 0x7fd4ac003520 con 0x7fd4c015bed0 2024-03-24T00:40:32.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.074+0000 7fd4bdd9b640 1 -- 172.21.15.67:0/3998508264 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd4c006f350 msgr2=0x7fd4c00725b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:32.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.074+0000 7fd4bdd9b640 1 --2- 172.21.15.67:0/3998508264 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd4c006f350 0x7fd4c00725b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:32.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.074+0000 7fd4bdd9b640 1 -- 172.21.15.67:0/3998508264 >> v1:172.21.15.72:6789/0 conn(0x7fd4c01589a0 legacy=0x7fd4c00cc390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:32.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.074+0000 7fd4bdd9b640 1 -- 172.21.15.67:0/3998508264 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4c0071ad0 con 0x7fd4c015bed0 2024-03-24T00:40:32.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.074+0000 7fd4bdd9b640 1 -- 172.21.15.67:0/3998508264 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7fd4a8001670 con 0x7fd4c015bed0 2024-03-24T00:40:32.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.075+0000 7fd4bdd9b640 1 -- 172.21.15.67:0/3998508264 >> v1:172.21.15.67:6789/0 conn(0x7fd4c015bed0 legacy=0x7fd4c006ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:32.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.075+0000 7fd4bdd9b640 1 --2- 172.21.15.67:0/3998508264 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd4c006f350 0x7fd4ac0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:32.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.075+0000 7fd4bdd9b640 1 --2- 172.21.15.67:0/3998508264 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4ac003f60 0x7fd4ac004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:32.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.075+0000 7fd4bdd9b640 1 --2- 172.21.15.67:0/3998508264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4ac004d40 0x7fd4ac0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:32.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.075+0000 7fd4bdd9b640 1 -- 172.21.15.67:0/3998508264 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c0072af0 con 0x7fd4ac004d40 2024-03-24T00:40:32.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.075+0000 7fd4bdd9b640 1 -- 172.21.15.67:0/3998508264 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ac00baa0 con 0x7fd4ac003f60 2024-03-24T00:40:32.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.075+0000 7fd4bdd9b640 1 -- 172.21.15.67:0/3998508264 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ac00deb0 con 0x7fd4c006f350 2024-03-24T00:40:32.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.075+0000 7fd4bf59e640 1 --2- 172.21.15.67:0/3998508264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4ac004d40 0x7fd4ac0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:32.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.075+0000 7fd4bed9d640 1 --2- 172.21.15.67:0/3998508264 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd4c006f350 0x7fd4ac0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:32.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.075+0000 7fd4be59c640 1 --2- 172.21.15.67:0/3998508264 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4ac003f60 0x7fd4ac004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:32.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.075+0000 7fd4bf59e640 1 -- 172.21.15.67:0/3998508264 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd4c006f350 msgr2=0x7fd4ac0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:32.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.075+0000 7fd4bf59e640 1 --2- 172.21.15.67:0/3998508264 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd4c006f350 0x7fd4ac0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:32.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.075+0000 7fd4bf59e640 1 -- 172.21.15.67:0/3998508264 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4ac003f60 msgr2=0x7fd4ac004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:32.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.075+0000 7fd4bf59e640 1 --2- 172.21.15.67:0/3998508264 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4ac003f60 0x7fd4ac004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:32.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.075+0000 7fd4bf59e640 1 -- 172.21.15.67:0/3998508264 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4ac003520 con 0x7fd4ac004d40 2024-03-24T00:40:32.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.076+0000 7fd4bf59e640 1 --2- 172.21.15.67:0/3998508264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4ac004d40 0x7fd4ac0051c0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fd4c006d380 tx=0x7fd4ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74076a73559f9f26 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:32.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.076+0000 7fd4bdd9b640 1 -- 172.21.15.67:0/3998508264 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b8039070 con 0x7fd4ac004d40 2024-03-24T00:40:32.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.076+0000 7fd4bdd9b640 1 -- 172.21.15.67:0/3998508264 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4b802c660 con 0x7fd4ac004d40 2024-03-24T00:40:32.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.076+0000 7fd4bdd9b640 1 -- 172.21.15.67:0/3998508264 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b8009690 con 0x7fd4ac004d40 2024-03-24T00:40:32.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.076+0000 7fd4c507e640 1 -- 172.21.15.67:0/3998508264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4ac004d40 msgr2=0x7fd4ac0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:32.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.076+0000 7fd4c507e640 1 --2- 172.21.15.67:0/3998508264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4ac004d40 0x7fd4ac0051c0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fd4c006d380 tx=0x7fd4ac005830 comp rx=0 tx=0).stop 2024-03-24T00:40:32.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.076+0000 7fd4bf59e640 1 -- 172.21.15.67:0/3998508264 reap_dead start 2024-03-24T00:40:32.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.076+0000 7fd4c507e640 1 -- 172.21.15.67:0/3998508264 shutdown_connections 2024-03-24T00:40:32.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.076+0000 7fd4c507e640 1 -- 172.21.15.67:0/3998508264 >> 172.21.15.67:0/3998508264 conn(0x7fd4c00cf9e0 msgr2=0x7fd4c00cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:32.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.076+0000 7fd4c507e640 1 -- 172.21.15.67:0/3998508264 shutdown_connections 2024-03-24T00:40:32.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.076+0000 7fd4c507e640 1 -- 172.21.15.67:0/3998508264 wait complete. 2024-03-24T00:40:32.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.078+0000 7fd4c507e640 1 Processor -- start 2024-03-24T00:40:32.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.078+0000 7fd4c507e640 1 -- start start 2024-03-24T00:40:32.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.079+0000 7fd4c507e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd4c0155d30 0x7fd4c01efd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:32.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.079+0000 7fd4c507e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4c01f0260 0x7fd4c0154b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:32.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.079+0000 7fd4c507e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4c0156110 0x7fd4c0150b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:32.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.079+0000 7fd4c507e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c00725b0 con 0x7fd4c01f0260 2024-03-24T00:40:32.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.079+0000 7fd4c507e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c00b48f0 con 0x7fd4c0156110 2024-03-24T00:40:32.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.079+0000 7fd4c507e640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c0072730 con 0x7fd4c0155d30 2024-03-24T00:40:32.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.079+0000 7fd4be59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4c01f0260 0x7fd4c0154b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:32.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.079+0000 7fd4be59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4c01f0260 0x7fd4c0154b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39468/0 (socket says 172.21.15.67:39468) 2024-03-24T00:40:32.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.079+0000 7fd4be59c640 1 -- 172.21.15.67:0/3369825592 learned_addr learned my addr 172.21.15.67:0/3369825592 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:32.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.079+0000 7fd4bf59e640 1 --2- 172.21.15.67:0/3369825592 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4c0156110 0x7fd4c0150b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:32.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.079+0000 7fd4bed9d640 1 --2- 172.21.15.67:0/3369825592 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd4c0155d30 0x7fd4c01efd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:32.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.079+0000 7fd4be59c640 1 -- 172.21.15.67:0/3369825592 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd4c0155d30 msgr2=0x7fd4c01efd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:32.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.079+0000 7fd4be59c640 1 --2- 172.21.15.67:0/3369825592 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd4c0155d30 0x7fd4c01efd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:32.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.079+0000 7fd4be59c640 1 -- 172.21.15.67:0/3369825592 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4c0156110 msgr2=0x7fd4c0150b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:32.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.079+0000 7fd4be59c640 1 --2- 172.21.15.67:0/3369825592 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4c0156110 0x7fd4c0150b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:32.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.079+0000 7fd4be59c640 1 -- 172.21.15.67:0/3369825592 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4c01f0700 con 0x7fd4c01f0260 2024-03-24T00:40:32.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.079+0000 7fd4be59c640 1 --2- 172.21.15.67:0/3369825592 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4c01f0260 0x7fd4c0154b60 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fd4a8007290 tx=0x7fd4a8007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:32.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.079+0000 7fd4b77fe640 1 -- 172.21.15.67:0/3369825592 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a800f9a0 con 0x7fd4c01f0260 2024-03-24T00:40:32.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.079+0000 7fd4b77fe640 1 -- 172.21.15.67:0/3369825592 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4a8018070 con 0x7fd4c01f0260 2024-03-24T00:40:32.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.079+0000 7fd4c507e640 1 -- 172.21.15.67:0/3369825592 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4c01f09e0 con 0x7fd4c01f0260 2024-03-24T00:40:32.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.079+0000 7fd4b77fe640 1 -- 172.21.15.67:0/3369825592 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a800fda0 con 0x7fd4c01f0260 2024-03-24T00:40:32.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.080+0000 7fd4c507e640 1 -- 172.21.15.67:0/3369825592 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4c0157120 con 0x7fd4c01f0260 2024-03-24T00:40:32.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.081+0000 7fd4c507e640 1 -- 172.21.15.67:0/3369825592 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4c014ddc0 con 0x7fd4c01f0260 2024-03-24T00:40:32.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.081+0000 7fd4b77fe640 1 -- 172.21.15.67:0/3369825592 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd4a801d460 con 0x7fd4c01f0260 2024-03-24T00:40:32.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.081+0000 7fd4b77fe640 1 --2- 172.21.15.67:0/3369825592 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd4a006bbd0 0x7fd4a006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:32.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.081+0000 7fd4b77fe640 1 -- 172.21.15.67:0/3369825592 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5778+0+0 (secure 0 0 0) 0x7fd4a808ad30 con 0x7fd4c01f0260 2024-03-24T00:40:32.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.081+0000 7fd4bed9d640 1 --2- 172.21.15.67:0/3369825592 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd4a006bbd0 0x7fd4a006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:32.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.082+0000 7fd4bed9d640 1 --2- 172.21.15.67:0/3369825592 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd4a006bbd0 0x7fd4a006e090 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd4c00b3fc0 tx=0x7fd4b0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:32.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.086+0000 7fd4b77fe640 1 -- 172.21.15.67:0/3369825592 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd4c014ddc0 con 0x7fd4c01f0260 2024-03-24T00:40:32.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.268+0000 7fd4c507e640 1 -- 172.21.15.67:0/3369825592 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd4c004f400 con 0x7fd4c01f0260 2024-03-24T00:40:32.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.942+0000 7fd4b77fe640 1 -- 172.21.15.67:0/3369825592 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v42) v1 ==== 169+0+0 (secure 0 0 0) 0x7fd4a80597e0 con 0x7fd4c01f0260 2024-03-24T00:40:32.943 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' removed 2024-03-24T00:40:32.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.946+0000 7fd4c507e640 1 -- 172.21.15.67:0/3369825592 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd4a006bbd0 msgr2=0x7fd4a006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:32.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.946+0000 7fd4c507e640 1 --2- 172.21.15.67:0/3369825592 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd4a006bbd0 0x7fd4a006e090 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd4c00b3fc0 tx=0x7fd4b0009040 comp rx=0 tx=0).stop 2024-03-24T00:40:32.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.946+0000 7fd4c507e640 1 -- 172.21.15.67:0/3369825592 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4c01f0260 msgr2=0x7fd4c0154b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:32.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.946+0000 7fd4c507e640 1 --2- 172.21.15.67:0/3369825592 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4c01f0260 0x7fd4c0154b60 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fd4a8007290 tx=0x7fd4a8007750 comp rx=0 tx=0).stop 2024-03-24T00:40:32.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.946+0000 7fd4c507e640 1 -- 172.21.15.67:0/3369825592 shutdown_connections 2024-03-24T00:40:32.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.946+0000 7fd4c507e640 1 --2- 172.21.15.67:0/3369825592 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd4a006bbd0 0x7fd4a006e090 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:32.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.946+0000 7fd4c507e640 1 --2- 172.21.15.67:0/3369825592 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4c0156110 0x7fd4c0150b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:32.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.946+0000 7fd4c507e640 1 --2- 172.21.15.67:0/3369825592 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4c01f0260 0x7fd4c0154b60 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:32.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.946+0000 7fd4c507e640 1 --2- 172.21.15.67:0/3369825592 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd4c0155d30 0x7fd4c01efd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:32.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.946+0000 7fd4c507e640 1 -- 172.21.15.67:0/3369825592 >> 172.21.15.67:0/3369825592 conn(0x7fd4c00cf9e0 msgr2=0x7fd4c006ec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:32.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.946+0000 7fd4c507e640 1 -- 172.21.15.67:0/3369825592 shutdown_connections 2024-03-24T00:40:32.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:32.946+0000 7fd4c507e640 1 -- 172.21.15.67:0/3369825592 wait complete. 2024-03-24T00:40:32.961 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-24T00:40:33.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.070+0000 7f62c4f32640 1 Processor -- start 2024-03-24T00:40:33.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.070+0000 7f62c4f32640 1 -- start start 2024-03-24T00:40:33.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.071+0000 7f62c4f32640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f62c015df00 0x7f62c015ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:33.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.071+0000 7f62c4f32640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f62c015e580 0x7f62c00ce3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:33.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.071+0000 7f62c4f32640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f62c00b2920 0x7f62c00ce900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:33.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.071+0000 7f62c4f32640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62c015e2e0 con 0x7f62c00b2920 2024-03-24T00:40:33.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.071+0000 7f62c4f32640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62c00b1880 con 0x7f62c015df00 2024-03-24T00:40:33.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.071+0000 7f62c4f32640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62c00b1b60 con 0x7f62c015e580 2024-03-24T00:40:33.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.071+0000 7f62bed9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f62c00b2920 0x7f62c00ce900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:33.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.071+0000 7f62bed9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f62c00b2920 0x7f62c00ce900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39484/0 (socket says 172.21.15.67:39484) 2024-03-24T00:40:33.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.071+0000 7f62bed9d640 1 -- 172.21.15.67:0/470318414 learned_addr learned my addr 172.21.15.67:0/470318414 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:33.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.071+0000 7f62be59c640 1 --2- 172.21.15.67:0/470318414 >> v2:172.21.15.72:3300/0 conn(0x7f62c015df00 0x7f62c015ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:33.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.071+0000 7f62bdd9b640 1 --2- 172.21.15.67:0/470318414 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f62c015e580 0x7f62c00ce3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:33.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.071+0000 7f62bed9d640 1 -- 172.21.15.67:0/470318414 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f62c015e580 msgr2=0x7f62c00ce3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:33.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.071+0000 7f62bed9d640 1 --2- 172.21.15.67:0/470318414 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f62c015e580 0x7f62c00ce3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:33.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.072+0000 7f62bed9d640 1 -- 172.21.15.67:0/470318414 >> v2:172.21.15.72:3300/0 conn(0x7f62c015df00 msgr2=0x7f62c015ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:33.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.072+0000 7f62bed9d640 1 --2- 172.21.15.67:0/470318414 >> v2:172.21.15.72:3300/0 conn(0x7f62c015df00 0x7f62c015ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:33.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.072+0000 7f62bed9d640 1 -- 172.21.15.67:0/470318414 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c00b1e40 con 0x7f62c00b2920 2024-03-24T00:40:33.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.072+0000 7f62bed9d640 1 --2- 172.21.15.67:0/470318414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62c00b2920 0x7f62c00ce900 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f62c0072360 tx=0x7f62b002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1de0ad7e1bba425e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:33.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.072+0000 7f62bd59a640 1 -- 172.21.15.67:0/470318414 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b002c9e0 con 0x7f62c00b2920 2024-03-24T00:40:33.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.072+0000 7f62bd59a640 1 -- 172.21.15.67:0/470318414 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62b002cb80 con 0x7f62c00b2920 2024-03-24T00:40:33.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.072+0000 7f62bd59a640 1 -- 172.21.15.67:0/470318414 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b00343f0 con 0x7f62c00b2920 2024-03-24T00:40:33.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.073+0000 7f62c4f32640 1 -- 172.21.15.67:0/470318414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62c00b2920 msgr2=0x7f62c00ce900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:33.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.073+0000 7f62c4f32640 1 --2- 172.21.15.67:0/470318414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62c00b2920 0x7f62c00ce900 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f62c0072360 tx=0x7f62b002be40 comp rx=0 tx=0).stop 2024-03-24T00:40:33.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.073+0000 7f62c4f32640 1 -- 172.21.15.67:0/470318414 shutdown_connections 2024-03-24T00:40:33.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.073+0000 7f62c4f32640 1 --2- 172.21.15.67:0/470318414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62c00b2920 0x7f62c00ce900 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:33.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.073+0000 7f62c4f32640 1 --2- 172.21.15.67:0/470318414 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f62c015e580 0x7f62c00ce3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:33.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.073+0000 7f62c4f32640 1 --2- 172.21.15.67:0/470318414 >> v2:172.21.15.72:3300/0 conn(0x7f62c015df00 0x7f62c015ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:33.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.073+0000 7f62c4f32640 1 -- 172.21.15.67:0/470318414 >> 172.21.15.67:0/470318414 conn(0x7f62c008b180 msgr2=0x7f62c008b590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:33.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.073+0000 7f62c4f32640 1 -- 172.21.15.67:0/470318414 shutdown_connections 2024-03-24T00:40:33.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.073+0000 7f62c4f32640 1 -- 172.21.15.67:0/470318414 wait complete. 2024-03-24T00:40:33.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.074+0000 7f62c4f32640 1 Processor -- start 2024-03-24T00:40:33.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.074+0000 7f62c4f32640 1 -- start start 2024-03-24T00:40:33.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.075+0000 7f62c4f32640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62c00b2920 0x7f62c00b4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:33.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.075+0000 7f62c4f32640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f62c015df00 0x7f62c00af150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:33.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.075+0000 7f62c4f32640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62c015e580 0x7f62c01550d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:33.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.075+0000 7f62c4f32640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c015f2b0 con 0x7f62c00b2920 2024-03-24T00:40:33.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.075+0000 7f62c4f32640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c00b2400 con 0x7f62c015e580 2024-03-24T00:40:33.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.075+0000 7f62c4f32640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62c015f430 con 0x7f62c015df00 2024-03-24T00:40:33.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.075+0000 7f62be59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62c00b2920 0x7f62c00b4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:33.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.075+0000 7f62be59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62c00b2920 0x7f62c00b4760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39490/0 (socket says 172.21.15.67:39490) 2024-03-24T00:40:33.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.075+0000 7f62be59c640 1 -- 172.21.15.67:0/3872300690 learned_addr learned my addr 172.21.15.67:0/3872300690 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:33.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.075+0000 7f62bed9d640 1 --2- 172.21.15.67:0/3872300690 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62c015e580 0x7f62c01550d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:33.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.075+0000 7f62bdd9b640 1 --2- 172.21.15.67:0/3872300690 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f62c015df00 0x7f62c00af150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:33.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.075+0000 7f62be59c640 1 -- 172.21.15.67:0/3872300690 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f62c015df00 msgr2=0x7f62c00af150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:33.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.075+0000 7f62be59c640 1 --2- 172.21.15.67:0/3872300690 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f62c015df00 0x7f62c00af150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:33.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.076+0000 7f62be59c640 1 -- 172.21.15.67:0/3872300690 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62c015e580 msgr2=0x7f62c01550d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:33.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.076+0000 7f62be59c640 1 --2- 172.21.15.67:0/3872300690 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62c015e580 0x7f62c01550d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:33.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.076+0000 7f62be59c640 1 -- 172.21.15.67:0/3872300690 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c00ae690 con 0x7f62c00b2920 2024-03-24T00:40:33.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.076+0000 7f62bed9d640 1 --2- 172.21.15.67:0/3872300690 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62c015e580 0x7f62c01550d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:40:33.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.076+0000 7f62be59c640 1 --2- 172.21.15.67:0/3872300690 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62c00b2920 0x7f62c00b4760 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f62b800e9b0 tx=0x7f62b800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:33.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.076+0000 7f62aeffd640 1 -- 172.21.15.67:0/3872300690 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b8017070 con 0x7f62c00b2920 2024-03-24T00:40:33.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.076+0000 7f62aeffd640 1 -- 172.21.15.67:0/3872300690 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62b800c9a0 con 0x7f62c00b2920 2024-03-24T00:40:33.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.076+0000 7f62c4f32640 1 -- 172.21.15.67:0/3872300690 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62c00b0430 con 0x7f62c00b2920 2024-03-24T00:40:33.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.076+0000 7f62aeffd640 1 -- 172.21.15.67:0/3872300690 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b800cd90 con 0x7f62c00b2920 2024-03-24T00:40:33.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.076+0000 7f62c4f32640 1 -- 172.21.15.67:0/3872300690 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62c00aec30 con 0x7f62c00b2920 2024-03-24T00:40:33.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.077+0000 7f62c4f32640 1 -- 172.21.15.67:0/3872300690 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62c004f400 con 0x7f62c00b2920 2024-03-24T00:40:33.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.077+0000 7f62aeffd640 1 -- 172.21.15.67:0/3872300690 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f62b801a070 con 0x7f62c00b2920 2024-03-24T00:40:33.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.078+0000 7f62aeffd640 1 --2- 172.21.15.67:0/3872300690 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f62a006bbd0 0x7f62a006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:33.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.078+0000 7f62aeffd640 1 -- 172.21.15.67:0/3872300690 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f62b808a970 con 0x7f62c00b2920 2024-03-24T00:40:33.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.078+0000 7f62bdd9b640 1 --2- 172.21.15.67:0/3872300690 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f62a006bbd0 0x7f62a006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:33.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.078+0000 7f62bdd9b640 1 --2- 172.21.15.67:0/3872300690 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f62a006bbd0 0x7f62a006e090 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f62c00d0880 tx=0x7f62b4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:33.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.082+0000 7f62aeffd640 1 -- 172.21.15.67:0/3872300690 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f62c00b0430 con 0x7f62c00b2920 2024-03-24T00:40:33.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.264+0000 7f62c4f32640 1 -- 172.21.15.67:0/3872300690 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f62c00b7ac0 con 0x7f62c00b2920 2024-03-24T00:40:33.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.266+0000 7f62aeffd640 1 -- 172.21.15.67:0/3872300690 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7f62b80595d0 con 0x7f62c00b2920 2024-03-24T00:40:33.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.269+0000 7f62c4f32640 1 -- 172.21.15.67:0/3872300690 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f62a006bbd0 msgr2=0x7f62a006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:33.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.269+0000 7f62c4f32640 1 --2- 172.21.15.67:0/3872300690 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f62a006bbd0 0x7f62a006e090 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f62c00d0880 tx=0x7f62b4008040 comp rx=0 tx=0).stop 2024-03-24T00:40:33.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.269+0000 7f62c4f32640 1 -- 172.21.15.67:0/3872300690 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62c00b2920 msgr2=0x7f62c00b4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:33.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.269+0000 7f62c4f32640 1 --2- 172.21.15.67:0/3872300690 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62c00b2920 0x7f62c00b4760 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f62b800e9b0 tx=0x7f62b800ee70 comp rx=0 tx=0).stop 2024-03-24T00:40:33.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.269+0000 7f62c4f32640 1 -- 172.21.15.67:0/3872300690 shutdown_connections 2024-03-24T00:40:33.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.269+0000 7f62c4f32640 1 --2- 172.21.15.67:0/3872300690 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f62a006bbd0 0x7f62a006e090 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:33.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.269+0000 7f62c4f32640 1 --2- 172.21.15.67:0/3872300690 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62c015e580 0x7f62c01550d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:33.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.269+0000 7f62c4f32640 1 --2- 172.21.15.67:0/3872300690 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f62c015df00 0x7f62c00af150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:33.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.269+0000 7f62c4f32640 1 --2- 172.21.15.67:0/3872300690 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62c00b2920 0x7f62c00b4760 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:33.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.269+0000 7f62c4f32640 1 -- 172.21.15.67:0/3872300690 >> 172.21.15.67:0/3872300690 conn(0x7f62c008b180 msgr2=0x7f62c00d12a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:33.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.269+0000 7f62c4f32640 1 -- 172.21.15.67:0/3872300690 shutdown_connections 2024-03-24T00:40:33.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.269+0000 7f62c4f32640 1 -- 172.21.15.67:0/3872300690 wait complete. 2024-03-24T00:40:33.282 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_apply_tag' 2024-03-24T00:40:33.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.384+0000 7f6f815d4640 1 Processor -- start 2024-03-24T00:40:33.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.384+0000 7f6f815d4640 1 -- start start 2024-03-24T00:40:33.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.385+0000 7f6f815d4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f6f7c15dec0 0x7f6f7c0ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:33.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.385+0000 7f6f815d4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6f7c0d0ef0 0x7f6f7c162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:33.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.385+0000 7f6f815d4640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f7c0cf640 con 0x7f6f7c15e2a0 2024-03-24T00:40:33.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.385+0000 7f6f815d4640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f7c0cee30 con 0x7f6f7c0d0ef0 2024-03-24T00:40:33.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.385+0000 7f6f815d4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f7c0d2120 con 0x7f6f7c15dec0 2024-03-24T00:40:33.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.385+0000 7f6f7b7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6f7c0d0ef0 0x7f6f7c162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:33.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.385+0000 7f6f7b7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6f7c0d0ef0 0x7f6f7c162630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39500/0 (socket says 172.21.15.67:39500) 2024-03-24T00:40:33.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.385+0000 7f6f7b7fe640 1 -- 172.21.15.67:0/3060656957 learned_addr learned my addr 172.21.15.67:0/3060656957 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:33.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.385+0000 7f6f7affd640 1 --2- 172.21.15.67:0/3060656957 >> v2:172.21.15.72:3300/0 conn(0x7f6f7c15dec0 0x7f6f7c0ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:33.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.385+0000 7f6f7b7fe640 1 -- 172.21.15.67:0/3060656957 >> v2:172.21.15.72:3300/0 conn(0x7f6f7c15dec0 msgr2=0x7f6f7c0ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:33.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.385+0000 7f6f7b7fe640 1 --2- 172.21.15.67:0/3060656957 >> v2:172.21.15.72:3300/0 conn(0x7f6f7c15dec0 0x7f6f7c0ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:33.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.385+0000 7f6f7b7fe640 1 -- 172.21.15.67:0/3060656957 >> v1:172.21.15.72:6789/0 conn(0x7f6f7c15e2a0 legacy=0x7f6f7c0d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:33.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.385+0000 7f6f7b7fe640 1 -- 172.21.15.67:0/3060656957 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f7c0cf080 con 0x7f6f7c0d0ef0 2024-03-24T00:40:33.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.386+0000 7f6f7b7fe640 1 --2- 172.21.15.67:0/3060656957 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f7c0d0ef0 0x7f6f7c162630 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c0d14a0 tx=0x7f6f7402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d16184c36d23d246 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:33.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.386+0000 7f6f7a7fc640 1 -- 172.21.15.67:0/3060656957 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f7402ca80 con 0x7f6f7c0d0ef0 2024-03-24T00:40:33.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.386+0000 7f6f7a7fc640 1 -- 172.21.15.67:0/3060656957 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f7402cc20 con 0x7f6f7c0d0ef0 2024-03-24T00:40:33.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.386+0000 7f6f7a7fc640 1 -- 172.21.15.67:0/3060656957 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f740343f0 con 0x7f6f7c0d0ef0 2024-03-24T00:40:33.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.387+0000 7f6f815d4640 1 -- 172.21.15.67:0/3060656957 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f7c0d0ef0 msgr2=0x7f6f7c162630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:33.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.387+0000 7f6f815d4640 1 --2- 172.21.15.67:0/3060656957 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f7c0d0ef0 0x7f6f7c162630 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c0d14a0 tx=0x7f6f7402bee0 comp rx=0 tx=0).stop 2024-03-24T00:40:33.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.387+0000 7f6f815d4640 1 -- 172.21.15.67:0/3060656957 shutdown_connections 2024-03-24T00:40:33.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.387+0000 7f6f815d4640 1 --2- 172.21.15.67:0/3060656957 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f7c0d0ef0 0x7f6f7c162630 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:33.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.387+0000 7f6f815d4640 1 --2- 172.21.15.67:0/3060656957 >> v2:172.21.15.72:3300/0 conn(0x7f6f7c15dec0 0x7f6f7c0ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:33.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.387+0000 7f6f815d4640 1 -- 172.21.15.67:0/3060656957 >> 172.21.15.67:0/3060656957 conn(0x7f6f7c06c810 msgr2=0x7f6f7c06cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:33.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.387+0000 7f6f815d4640 1 -- 172.21.15.67:0/3060656957 shutdown_connections 2024-03-24T00:40:33.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.387+0000 7f6f815d4640 1 -- 172.21.15.67:0/3060656957 wait complete. 2024-03-24T00:40:33.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.388+0000 7f6f815d4640 1 Processor -- start 2024-03-24T00:40:33.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.388+0000 7f6f815d4640 1 -- start start 2024-03-24T00:40:33.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.389+0000 7f6f815d4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f7c0d0ef0 0x7f6f7c1593e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:33.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.389+0000 7f6f815d4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f7c15dec0 0x7f6f7c0b5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:33.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.389+0000 7f6f815d4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6f7c15e2a0 0x7f6f7c1f3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:33.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.389+0000 7f6f815d4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7c0d2730 con 0x7f6f7c15dec0 2024-03-24T00:40:33.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.389+0000 7f6f815d4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7c0d25b0 con 0x7f6f7c0d0ef0 2024-03-24T00:40:33.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.389+0000 7f6f815d4640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7c0d28b0 con 0x7f6f7c15e2a0 2024-03-24T00:40:33.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.389+0000 7f6f73fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f7c15dec0 0x7f6f7c0b5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:33.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.389+0000 7f6f73fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f7c15dec0 0x7f6f7c0b5620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39512/0 (socket says 172.21.15.67:39512) 2024-03-24T00:40:33.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.389+0000 7f6f73fff640 1 -- 172.21.15.67:0/1741172046 learned_addr learned my addr 172.21.15.67:0/1741172046 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:33.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.389+0000 7f6f7b7fe640 1 --2- 172.21.15.67:0/1741172046 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6f7c15e2a0 0x7f6f7c1f3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:33.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.389+0000 7f6f7affd640 1 --2- 172.21.15.67:0/1741172046 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f7c0d0ef0 0x7f6f7c1593e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:33.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.389+0000 7f6f73fff640 1 -- 172.21.15.67:0/1741172046 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6f7c15e2a0 msgr2=0x7f6f7c1f3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:33.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.389+0000 7f6f73fff640 1 --2- 172.21.15.67:0/1741172046 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6f7c15e2a0 0x7f6f7c1f3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:33.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.389+0000 7f6f73fff640 1 -- 172.21.15.67:0/1741172046 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f7c0d0ef0 msgr2=0x7f6f7c1593e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:33.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.389+0000 7f6f73fff640 1 --2- 172.21.15.67:0/1741172046 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f7c0d0ef0 0x7f6f7c1593e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:33.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.389+0000 7f6f73fff640 1 -- 172.21.15.67:0/1741172046 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f7c0b6000 con 0x7f6f7c15dec0 2024-03-24T00:40:33.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.389+0000 7f6f7b7fe640 1 --2- 172.21.15.67:0/1741172046 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6f7c15e2a0 0x7f6f7c1f3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:40:33.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.389+0000 7f6f7affd640 1 --2- 172.21.15.67:0/1741172046 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f7c0d0ef0 0x7f6f7c1593e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:40:33.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.389+0000 7f6f73fff640 1 --2- 172.21.15.67:0/1741172046 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f7c15dec0 0x7f6f7c0b5620 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f6f6400c440 tx=0x7f6f6400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:33.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.389+0000 7f6f737fe640 1 -- 172.21.15.67:0/1741172046 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f640052d0 con 0x7f6f7c15dec0 2024-03-24T00:40:33.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.389+0000 7f6f737fe640 1 -- 172.21.15.67:0/1741172046 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f64005470 con 0x7f6f7c15dec0 2024-03-24T00:40:33.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.389+0000 7f6f815d4640 1 -- 172.21.15.67:0/1741172046 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f7c0b62e0 con 0x7f6f7c15dec0 2024-03-24T00:40:33.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.389+0000 7f6f737fe640 1 -- 172.21.15.67:0/1741172046 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f64007260 con 0x7f6f7c15dec0 2024-03-24T00:40:33.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.389+0000 7f6f815d4640 1 -- 172.21.15.67:0/1741172046 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f7c0cdab0 con 0x7f6f7c15dec0 2024-03-24T00:40:33.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.390+0000 7f6f815d4640 1 -- 172.21.15.67:0/1741172046 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f7c04f400 con 0x7f6f7c15dec0 2024-03-24T00:40:33.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.391+0000 7f6f737fe640 1 -- 172.21.15.67:0/1741172046 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6f640074c0 con 0x7f6f7c15dec0 2024-03-24T00:40:33.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.391+0000 7f6f737fe640 1 --2- 172.21.15.67:0/1741172046 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6f5c06bbd0 0x7f6f5c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:33.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.391+0000 7f6f737fe640 1 -- 172.21.15.67:0/1741172046 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f6f64089cd0 con 0x7f6f7c15dec0 2024-03-24T00:40:33.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.392+0000 7f6f7affd640 1 --2- 172.21.15.67:0/1741172046 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6f5c06bbd0 0x7f6f5c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:33.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.392+0000 7f6f7affd640 1 --2- 172.21.15.67:0/1741172046 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6f5c06bbd0 0x7f6f5c06e090 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c009b90 tx=0x7f6f6c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:33.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.396+0000 7f6f737fe640 1 -- 172.21.15.67:0/1741172046 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6f7c0cf640 con 0x7f6f7c15dec0 2024-03-24T00:40:33.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.650+0000 7f6f815d4640 1 -- 172.21.15.67:0/1741172046 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_apply_tag"]} v 0) v1 -- 0x7f6f7c0d1aa0 con 0x7f6f7c15dec0 2024-03-24T00:40:33.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.944+0000 7f6f737fe640 1 -- 172.21.15.67:0/1741172046 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_apply_tag"]}]=0 v115) v1 ==== 142+0+0 (secure 0 0 0) 0x7f6f640589c0 con 0x7f6f7c15dec0 2024-03-24T00:40:33.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.946+0000 7f6f815d4640 1 -- 172.21.15.67:0/1741172046 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6f5c06bbd0 msgr2=0x7f6f5c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:33.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.946+0000 7f6f815d4640 1 --2- 172.21.15.67:0/1741172046 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6f5c06bbd0 0x7f6f5c06e090 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c009b90 tx=0x7f6f6c008040 comp rx=0 tx=0).stop 2024-03-24T00:40:33.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.946+0000 7f6f815d4640 1 -- 172.21.15.67:0/1741172046 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f7c15dec0 msgr2=0x7f6f7c0b5620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:33.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.946+0000 7f6f815d4640 1 --2- 172.21.15.67:0/1741172046 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f7c15dec0 0x7f6f7c0b5620 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f6f6400c440 tx=0x7f6f6400c900 comp rx=0 tx=0).stop 2024-03-24T00:40:33.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.946+0000 7f6f815d4640 1 -- 172.21.15.67:0/1741172046 shutdown_connections 2024-03-24T00:40:33.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.946+0000 7f6f815d4640 1 --2- 172.21.15.67:0/1741172046 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6f5c06bbd0 0x7f6f5c06e090 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:33.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.946+0000 7f6f815d4640 1 --2- 172.21.15.67:0/1741172046 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6f7c15e2a0 0x7f6f7c1f3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:33.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.946+0000 7f6f815d4640 1 --2- 172.21.15.67:0/1741172046 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f7c15dec0 0x7f6f7c0b5620 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:33.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.946+0000 7f6f815d4640 1 --2- 172.21.15.67:0/1741172046 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f7c0d0ef0 0x7f6f7c1593e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:33.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.946+0000 7f6f815d4640 1 -- 172.21.15.67:0/1741172046 >> 172.21.15.67:0/1741172046 conn(0x7f6f7c06c810 msgr2=0x7f6f7c15eae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:33.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.946+0000 7f6f815d4640 1 -- 172.21.15.67:0/1741172046 shutdown_connections 2024-03-24T00:40:33.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:33.946+0000 7f6f815d4640 1 -- 172.21.15.67:0/1741172046 wait complete. 2024-03-24T00:40:33.958 INFO:tasks.cephfs_test_runner:test_apply_tag (tasks.cephfs.test_forward_scrub.TestForwardScrub) ... ok 2024-03-24T00:40:33.958 INFO:tasks.cephfs_test_runner:Starting test: test_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) That the MDS can repair an inodes backtrace in the data pool 2024-03-24T00:40:33.958 INFO:tasks.cephfs_test_runner:test_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2024-03-24T00:40:33.958 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:40:34.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.060+0000 7f08c040b640 1 Processor -- start 2024-03-24T00:40:34.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.060+0000 7f08c040b640 1 -- start start 2024-03-24T00:40:34.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.061+0000 7f08c040b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f08b815e130 0x7f08b80cde00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:34.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.061+0000 7f08c040b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f08b80b5ac0 0x7f08b80b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:34.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.061+0000 7f08c040b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08b80701e0 con 0x7f08b80b2770 2024-03-24T00:40:34.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.061+0000 7f08c040b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08b80d00b0 con 0x7f08b815e130 2024-03-24T00:40:34.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.061+0000 7f08c040b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08b80704c0 con 0x7f08b80b5ac0 2024-03-24T00:40:34.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.061+0000 7f08be1a7640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f08b815e130 0x7f08b80cde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:34.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.061+0000 7f08be1a7640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f08b815e130 0x7f08b80cde00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34864/0 (socket says 172.21.15.67:34864) 2024-03-24T00:40:34.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.061+0000 7f08be1a7640 1 -- 172.21.15.67:0/3768592697 learned_addr learned my addr 172.21.15.67:0/3768592697 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:34.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.061+0000 7f08be9a8640 1 --1- 172.21.15.67:0/3768592697 >> v1:172.21.15.67:6789/0 conn(0x7f08b80b2770 0x7f08b806bc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52504/0 (socket says 172.21.15.67:52504) 2024-03-24T00:40:34.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.061+0000 7f08bd9a6640 1 --2- 172.21.15.67:0/3768592697 >> v2:172.21.15.72:3300/0 conn(0x7f08b80b5ac0 0x7f08b80b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:34.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.061+0000 7f08be1a7640 1 -- 172.21.15.67:0/3768592697 >> v2:172.21.15.72:3300/0 conn(0x7f08b80b5ac0 msgr2=0x7f08b80b4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:34.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.061+0000 7f08be1a7640 1 --2- 172.21.15.67:0/3768592697 >> v2:172.21.15.72:3300/0 conn(0x7f08b80b5ac0 0x7f08b80b4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:34.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.061+0000 7f08be1a7640 1 -- 172.21.15.67:0/3768592697 >> v1:172.21.15.67:6789/0 conn(0x7f08b80b2770 legacy=0x7f08b806bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:34.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.061+0000 7f08bd1a5640 1 -- 172.21.15.67:0/3768592697 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3701701925 0 0) 0x7f08b80701e0 con 0x7f08b80b2770 2024-03-24T00:40:34.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.062+0000 7f08be1a7640 1 -- 172.21.15.67:0/3768592697 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b806b170 con 0x7f08b815e130 2024-03-24T00:40:34.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.062+0000 7f08be1a7640 1 --2- 172.21.15.67:0/3768592697 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08b815e130 0x7f08b80cde00 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f08b8070d10 tx=0x7f08b002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef39c46c3639d2bd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:34.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.062+0000 7f08bd1a5640 1 -- 172.21.15.67:0/3768592697 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08b0035020 con 0x7f08b815e130 2024-03-24T00:40:34.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.062+0000 7f08bd1a5640 1 -- 172.21.15.67:0/3768592697 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08b0031870 con 0x7f08b815e130 2024-03-24T00:40:34.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.062+0000 7f08bd1a5640 1 -- 172.21.15.67:0/3768592697 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08b0031c40 con 0x7f08b815e130 2024-03-24T00:40:34.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.063+0000 7f08c040b640 1 -- 172.21.15.67:0/3768592697 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08b815e130 msgr2=0x7f08b80cde00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:34.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.063+0000 7f08c040b640 1 --2- 172.21.15.67:0/3768592697 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08b815e130 0x7f08b80cde00 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f08b8070d10 tx=0x7f08b002b910 comp rx=0 tx=0).stop 2024-03-24T00:40:34.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.063+0000 7f08c040b640 1 -- 172.21.15.67:0/3768592697 shutdown_connections 2024-03-24T00:40:34.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.063+0000 7f08c040b640 1 --2- 172.21.15.67:0/3768592697 >> v2:172.21.15.72:3300/0 conn(0x7f08b80b5ac0 0x7f08b80b4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:34.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.063+0000 7f08c040b640 1 --2- 172.21.15.67:0/3768592697 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08b815e130 0x7f08b80cde00 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:34.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.063+0000 7f08c040b640 1 -- 172.21.15.67:0/3768592697 >> 172.21.15.67:0/3768592697 conn(0x7f08b80b3c40 msgr2=0x7f08b80b1dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:34.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.063+0000 7f08c040b640 1 -- 172.21.15.67:0/3768592697 shutdown_connections 2024-03-24T00:40:34.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.063+0000 7f08c040b640 1 -- 172.21.15.67:0/3768592697 wait complete. 2024-03-24T00:40:34.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.064+0000 7f08c040b640 1 Processor -- start 2024-03-24T00:40:34.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.064+0000 7f08c040b640 1 -- start start 2024-03-24T00:40:34.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.065+0000 7f08c040b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f08b80b2770 0x7f08b81f1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:34.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.065+0000 7f08c040b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f08b80b5ac0 0x7f08b8157750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:34.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.065+0000 7f08c040b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08b815e130 0x7f08b8157c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:34.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.065+0000 7f08c040b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08b815e510 con 0x7f08b815e130 2024-03-24T00:40:34.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.065+0000 7f08c040b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08b806b7c0 con 0x7f08b80b2770 2024-03-24T00:40:34.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.065+0000 7f08c040b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08b806fe00 con 0x7f08b80b5ac0 2024-03-24T00:40:34.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.065+0000 7f08be9a8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08b815e130 0x7f08b8157c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:34.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.065+0000 7f08be9a8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08b815e130 0x7f08b8157c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34876/0 (socket says 172.21.15.67:34876) 2024-03-24T00:40:34.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.065+0000 7f08be9a8640 1 -- 172.21.15.67:0/2182085939 learned_addr learned my addr 172.21.15.67:0/2182085939 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:34.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.065+0000 7f08bd9a6640 1 --2- 172.21.15.67:0/2182085939 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f08b80b5ac0 0x7f08b8157750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:34.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.065+0000 7f08be1a7640 1 --2- 172.21.15.67:0/2182085939 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f08b80b2770 0x7f08b81f1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:34.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.065+0000 7f08be9a8640 1 -- 172.21.15.67:0/2182085939 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f08b80b5ac0 msgr2=0x7f08b8157750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:34.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.065+0000 7f08be9a8640 1 --2- 172.21.15.67:0/2182085939 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f08b80b5ac0 0x7f08b8157750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:34.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.065+0000 7f08be9a8640 1 -- 172.21.15.67:0/2182085939 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f08b80b2770 msgr2=0x7f08b81f1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:34.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.065+0000 7f08be9a8640 1 --2- 172.21.15.67:0/2182085939 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f08b80b2770 0x7f08b81f1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:34.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.065+0000 7f08be9a8640 1 -- 172.21.15.67:0/2182085939 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b81582d0 con 0x7f08b815e130 2024-03-24T00:40:34.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.066+0000 7f08be9a8640 1 --2- 172.21.15.67:0/2182085939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08b815e130 0x7f08b8157c90 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f08b400c7b0 tx=0x7f08b400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:34.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.066+0000 7f08aeffd640 1 -- 172.21.15.67:0/2182085939 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08b4018070 con 0x7f08b815e130 2024-03-24T00:40:34.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.066+0000 7f08aeffd640 1 -- 172.21.15.67:0/2182085939 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08b400a640 con 0x7f08b815e130 2024-03-24T00:40:34.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.066+0000 7f08c040b640 1 -- 172.21.15.67:0/2182085939 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08b81585b0 con 0x7f08b815e130 2024-03-24T00:40:34.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.066+0000 7f08aeffd640 1 -- 172.21.15.67:0/2182085939 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08b400aa70 con 0x7f08b815e130 2024-03-24T00:40:34.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.066+0000 7f08c040b640 1 -- 172.21.15.67:0/2182085939 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08b8158b70 con 0x7f08b815e130 2024-03-24T00:40:34.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.067+0000 7f08c040b640 1 -- 172.21.15.67:0/2182085939 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08b804f400 con 0x7f08b815e130 2024-03-24T00:40:34.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.067+0000 7f08aeffd640 1 -- 172.21.15.67:0/2182085939 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f08b401b070 con 0x7f08b815e130 2024-03-24T00:40:34.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.068+0000 7f08aeffd640 1 --2- 172.21.15.67:0/2182085939 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f089c06bbd0 0x7f089c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:34.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.068+0000 7f08aeffd640 1 -- 172.21.15.67:0/2182085939 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f08b4089df0 con 0x7f08b815e130 2024-03-24T00:40:34.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.068+0000 7f08be1a7640 1 --2- 172.21.15.67:0/2182085939 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f089c06bbd0 0x7f089c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:34.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.069+0000 7f08be1a7640 1 --2- 172.21.15.67:0/2182085939 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f089c06bbd0 0x7f089c06e090 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f08b0036000 tx=0x7f08b003c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:34.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.072+0000 7f08aeffd640 1 -- 172.21.15.67:0/2182085939 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f08b4058490 con 0x7f08b815e130 2024-03-24T00:40:34.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.256+0000 7f08c040b640 1 -- 172.21.15.67:0/2182085939 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f08b80cd9c0 con 0x7f08b815e130 2024-03-24T00:40:34.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.257+0000 7f08aeffd640 1 -- 172.21.15.67:0/2182085939 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+13233 (secure 0 0 0) 0x7f08b405c530 con 0x7f08b815e130 2024-03-24T00:40:34.258 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:40:34.258 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":42,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:40:32.942464+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:6836/2551774148":"2024-03-25T00:40:29.934404+0000","172.21.15.72:6834/2551774148":"2024-03-25T00:40:29.934404+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:40:34.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.259+0000 7f08c040b640 1 -- 172.21.15.67:0/2182085939 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f089c06bbd0 msgr2=0x7f089c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:34.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.259+0000 7f08c040b640 1 --2- 172.21.15.67:0/2182085939 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f089c06bbd0 0x7f089c06e090 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f08b0036000 tx=0x7f08b003c040 comp rx=0 tx=0).stop 2024-03-24T00:40:34.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.259+0000 7f08c040b640 1 -- 172.21.15.67:0/2182085939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08b815e130 msgr2=0x7f08b8157c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:34.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.259+0000 7f08c040b640 1 --2- 172.21.15.67:0/2182085939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08b815e130 0x7f08b8157c90 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f08b400c7b0 tx=0x7f08b400cc70 comp rx=0 tx=0).stop 2024-03-24T00:40:34.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.259+0000 7f08c040b640 1 -- 172.21.15.67:0/2182085939 shutdown_connections 2024-03-24T00:40:34.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.259+0000 7f08c040b640 1 --2- 172.21.15.67:0/2182085939 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f089c06bbd0 0x7f089c06e090 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:34.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.259+0000 7f08c040b640 1 --2- 172.21.15.67:0/2182085939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08b815e130 0x7f08b8157c90 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:34.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.259+0000 7f08c040b640 1 --2- 172.21.15.67:0/2182085939 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f08b80b5ac0 0x7f08b8157750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:34.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.259+0000 7f08c040b640 1 --2- 172.21.15.67:0/2182085939 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f08b80b2770 0x7f08b81f1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:34.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.259+0000 7f08c040b640 1 -- 172.21.15.67:0/2182085939 >> 172.21.15.67:0/2182085939 conn(0x7f08b80b3c40 msgr2=0x7f08b80c7d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:34.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.259+0000 7f08c040b640 1 -- 172.21.15.67:0/2182085939 shutdown_connections 2024-03-24T00:40:34.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.259+0000 7f08c040b640 1 -- 172.21.15.67:0/2182085939 wait complete. 2024-03-24T00:40:34.271 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:38:39.364950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:40:34.271 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:40:34.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.375+0000 7f28ba560640 1 Processor -- start 2024-03-24T00:40:34.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.375+0000 7f28ba560640 1 -- start start 2024-03-24T00:40:34.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.376+0000 7f28ba560640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f28b4072200 0x7f28b40ae1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:34.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.376+0000 7f28ba560640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f28b40ae9b0 0x7f28b40b3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:34.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.376+0000 7f28ba560640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28b40ad140 con 0x7f28b415e010 2024-03-24T00:40:34.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.376+0000 7f28ba560640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28b415e3f0 con 0x7f28b4072200 2024-03-24T00:40:34.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.376+0000 7f28ba560640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28b40ad3e0 con 0x7f28b40ae9b0 2024-03-24T00:40:34.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.376+0000 7f28b8afd640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f28b40ae9b0 0x7f28b40b3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:34.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.376+0000 7f28b37fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f28b4072200 0x7f28b40ae1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:34.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.376+0000 7f28b8afd640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f28b40ae9b0 0x7f28b40b3eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.67:52510/0 (socket says 172.21.15.67:52510) 2024-03-24T00:40:34.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.376+0000 7f28b3fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f28b415e010 0x7f28b4071af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:41976/0 (socket says 172.21.15.67:41976) 2024-03-24T00:40:34.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.376+0000 7f28b8afd640 1 -- 172.21.15.67:0/3973315022 learned_addr learned my addr 172.21.15.67:0/3973315022 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:34.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.376+0000 7f28b8afd640 1 -- 172.21.15.67:0/3973315022 >> v2:172.21.15.72:3300/0 conn(0x7f28b4072200 msgr2=0x7f28b40ae1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:34.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.376+0000 7f28b8afd640 1 --2- 172.21.15.67:0/3973315022 >> v2:172.21.15.72:3300/0 conn(0x7f28b4072200 0x7f28b40ae1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:34.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.376+0000 7f28b8afd640 1 -- 172.21.15.67:0/3973315022 >> v1:172.21.15.72:6789/0 conn(0x7f28b415e010 legacy=0x7f28b4071af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:34.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.377+0000 7f28b8afd640 1 -- 172.21.15.67:0/3973315022 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28b40ad6c0 con 0x7f28b40ae9b0 2024-03-24T00:40:34.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.377+0000 7f28b37fe640 1 --2- 172.21.15.67:0/3973315022 >> v2:172.21.15.72:3300/0 conn(0x7f28b4072200 0x7f28b40ae1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:40:34.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.377+0000 7f28b8afd640 1 --2- 172.21.15.67:0/3973315022 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f28b40ae9b0 0x7f28b40b3eb0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f28ac009e50 tx=0x7f28ac02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a6d0e70d1eb10f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:34.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.377+0000 7f28b2ffd640 1 -- 172.21.15.67:0/3973315022 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28ac02cb00 con 0x7f28b40ae9b0 2024-03-24T00:40:34.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.377+0000 7f28b2ffd640 1 -- 172.21.15.67:0/3973315022 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28ac02cca0 con 0x7f28b40ae9b0 2024-03-24T00:40:34.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.377+0000 7f28b2ffd640 1 -- 172.21.15.67:0/3973315022 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28ac034610 con 0x7f28b40ae9b0 2024-03-24T00:40:34.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.378+0000 7f28ba560640 1 -- 172.21.15.67:0/3973315022 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f28b40ae9b0 msgr2=0x7f28b40b3eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:34.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.378+0000 7f28ba560640 1 --2- 172.21.15.67:0/3973315022 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f28b40ae9b0 0x7f28b40b3eb0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f28ac009e50 tx=0x7f28ac02bf60 comp rx=0 tx=0).stop 2024-03-24T00:40:34.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.378+0000 7f28ba560640 1 -- 172.21.15.67:0/3973315022 shutdown_connections 2024-03-24T00:40:34.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.378+0000 7f28ba560640 1 --2- 172.21.15.67:0/3973315022 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f28b40ae9b0 0x7f28b40b3eb0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:34.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.378+0000 7f28ba560640 1 --2- 172.21.15.67:0/3973315022 >> v2:172.21.15.72:3300/0 conn(0x7f28b4072200 0x7f28b40ae1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:34.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.378+0000 7f28ba560640 1 -- 172.21.15.67:0/3973315022 >> 172.21.15.67:0/3973315022 conn(0x7f28b40cf9e0 msgr2=0x7f28b40cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:34.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.378+0000 7f28ba560640 1 -- 172.21.15.67:0/3973315022 shutdown_connections 2024-03-24T00:40:34.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.378+0000 7f28ba560640 1 -- 172.21.15.67:0/3973315022 wait complete. 2024-03-24T00:40:34.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.379+0000 7f28ba560640 1 Processor -- start 2024-03-24T00:40:34.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.379+0000 7f28ba560640 1 -- start start 2024-03-24T00:40:34.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.380+0000 7f28ba560640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f28b4072200 0x7f28b4146600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:34.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.380+0000 7f28ba560640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f28b40ae9b0 0x7f28b41f73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:34.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.380+0000 7f28ba560640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28b415e010 0x7f28b41f78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:34.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.380+0000 7f28ba560640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28b40af290 con 0x7f28b415e010 2024-03-24T00:40:34.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.380+0000 7f28ba560640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28b40adc80 con 0x7f28b40ae9b0 2024-03-24T00:40:34.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.380+0000 7f28ba560640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28b406db50 con 0x7f28b4072200 2024-03-24T00:40:34.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.380+0000 7f28b8afd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28b415e010 0x7f28b41f78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:34.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.380+0000 7f28b8afd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28b415e010 0x7f28b41f78e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34882/0 (socket says 172.21.15.67:34882) 2024-03-24T00:40:34.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.380+0000 7f28b8afd640 1 -- 172.21.15.67:0/2868679443 learned_addr learned my addr 172.21.15.67:0/2868679443 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:34.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.380+0000 7f28b37fe640 1 --2- 172.21.15.67:0/2868679443 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f28b40ae9b0 0x7f28b41f73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:34.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.380+0000 7f28b3fff640 1 --2- 172.21.15.67:0/2868679443 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f28b4072200 0x7f28b4146600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:34.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.380+0000 7f28b8afd640 1 -- 172.21.15.67:0/2868679443 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f28b4072200 msgr2=0x7f28b4146600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:34.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.380+0000 7f28b8afd640 1 --2- 172.21.15.67:0/2868679443 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f28b4072200 0x7f28b4146600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:34.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.380+0000 7f28b8afd640 1 -- 172.21.15.67:0/2868679443 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f28b40ae9b0 msgr2=0x7f28b41f73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:34.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.380+0000 7f28b8afd640 1 --2- 172.21.15.67:0/2868679443 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f28b40ae9b0 0x7f28b41f73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:34.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.380+0000 7f28b8afd640 1 -- 172.21.15.67:0/2868679443 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28b4143240 con 0x7f28b415e010 2024-03-24T00:40:34.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.381+0000 7f28b8afd640 1 --2- 172.21.15.67:0/2868679443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28b415e010 0x7f28b41f78e0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f28ac009e50 tx=0x7f28ac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:34.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.381+0000 7f28b0ff9640 1 -- 172.21.15.67:0/2868679443 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28ac043020 con 0x7f28b415e010 2024-03-24T00:40:34.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.381+0000 7f28b0ff9640 1 -- 172.21.15.67:0/2868679443 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28ac02ce50 con 0x7f28b415e010 2024-03-24T00:40:34.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.381+0000 7f28ba560640 1 -- 172.21.15.67:0/2868679443 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28b4143520 con 0x7f28b415e010 2024-03-24T00:40:34.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.381+0000 7f28b0ff9640 1 -- 172.21.15.67:0/2868679443 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28ac009510 con 0x7f28b415e010 2024-03-24T00:40:34.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.381+0000 7f28ba560640 1 -- 172.21.15.67:0/2868679443 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28b4143ae0 con 0x7f28b415e010 2024-03-24T00:40:34.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.382+0000 7f28ba560640 1 -- 172.21.15.67:0/2868679443 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28b404f400 con 0x7f28b415e010 2024-03-24T00:40:34.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.382+0000 7f28b0ff9640 1 -- 172.21.15.67:0/2868679443 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f28ac046020 con 0x7f28b415e010 2024-03-24T00:40:34.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.383+0000 7f28b0ff9640 1 --2- 172.21.15.67:0/2868679443 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f28a006bbd0 0x7f28a006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:34.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.383+0000 7f28b0ff9640 1 -- 172.21.15.67:0/2868679443 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f28ac0aced0 con 0x7f28b415e010 2024-03-24T00:40:34.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.383+0000 7f28b3fff640 1 --2- 172.21.15.67:0/2868679443 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f28a006bbd0 0x7f28a006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:34.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.383+0000 7f28b3fff640 1 --2- 172.21.15.67:0/2868679443 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f28a006bbd0 0x7f28a006e090 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f28a4000f60 tx=0x7f28a4007ce0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:34.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.387+0000 7f28b0ff9640 1 -- 172.21.15.67:0/2868679443 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f28ac04d030 con 0x7f28b415e010 2024-03-24T00:40:34.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.567+0000 7f28ba560640 1 -- 172.21.15.67:0/2868679443 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f28b4155e90 con 0x7f28b415e010 2024-03-24T00:40:34.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.567+0000 7f28b0ff9640 1 -- 172.21.15.67:0/2868679443 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) v1 ==== 93+0+10 (secure 0 0 0) 0x7f28ac03c350 con 0x7f28b415e010 2024-03-24T00:40:34.568 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-24T00:40:34.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.569+0000 7f28ba560640 1 -- 172.21.15.67:0/2868679443 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f28a006bbd0 msgr2=0x7f28a006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:34.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.569+0000 7f28ba560640 1 --2- 172.21.15.67:0/2868679443 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f28a006bbd0 0x7f28a006e090 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f28a4000f60 tx=0x7f28a4007ce0 comp rx=0 tx=0).stop 2024-03-24T00:40:34.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.569+0000 7f28ba560640 1 -- 172.21.15.67:0/2868679443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28b415e010 msgr2=0x7f28b41f78e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:34.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.569+0000 7f28ba560640 1 --2- 172.21.15.67:0/2868679443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28b415e010 0x7f28b41f78e0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f28ac009e50 tx=0x7f28ac004a40 comp rx=0 tx=0).stop 2024-03-24T00:40:34.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.569+0000 7f28ba560640 1 -- 172.21.15.67:0/2868679443 shutdown_connections 2024-03-24T00:40:34.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.569+0000 7f28ba560640 1 --2- 172.21.15.67:0/2868679443 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f28a006bbd0 0x7f28a006e090 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:34.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.569+0000 7f28ba560640 1 --2- 172.21.15.67:0/2868679443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28b415e010 0x7f28b41f78e0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:34.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.569+0000 7f28ba560640 1 --2- 172.21.15.67:0/2868679443 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f28b40ae9b0 0x7f28b41f73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:34.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.569+0000 7f28ba560640 1 --2- 172.21.15.67:0/2868679443 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f28b4072200 0x7f28b4146600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:34.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.569+0000 7f28ba560640 1 -- 172.21.15.67:0/2868679443 >> 172.21.15.67:0/2868679443 conn(0x7f28b40cf9e0 msgr2=0x7f28b40af8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:34.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.569+0000 7f28ba560640 1 -- 172.21.15.67:0/2868679443 shutdown_connections 2024-03-24T00:40:34.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.569+0000 7f28ba560640 1 -- 172.21.15.67:0/2868679443 wait complete. 2024-03-24T00:40:34.581 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_backtrace_repair' 2024-03-24T00:40:34.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.681+0000 7fcc04b87640 1 Processor -- start 2024-03-24T00:40:34.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.681+0000 7fcc04b87640 1 -- start start 2024-03-24T00:40:34.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.682+0000 7fcc04b87640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcc000d1550 0x7fcc0006cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:34.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.682+0000 7fcc04b87640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcc0009a480 0x7fcc000b2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:34.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.682+0000 7fcc04b87640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc000d1930 con 0x7fcc000b2780 2024-03-24T00:40:34.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.682+0000 7fcc04b87640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc0006d420 con 0x7fcc0009a480 2024-03-24T00:40:34.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.682+0000 7fcc04b87640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0006ca50 con 0x7fcc000d1550 2024-03-24T00:40:34.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.682+0000 7fcbfdd9b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fcc000b2780 0x7fcc00099d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52506/0 (socket says 172.21.15.67:52506) 2024-03-24T00:40:34.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.682+0000 7fcbfdd9b640 1 -- 172.21.15.67:0/4190009797 learned_addr learned my addr 172.21.15.67:0/4190009797 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:40:34.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.682+0000 7fcbfed9d640 1 --2- 172.21.15.67:0/4190009797 >> v2:172.21.15.67:3300/0 conn(0x7fcc0009a480 0x7fcc000b2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:34.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.682+0000 7fcbfed9d640 1 -- 172.21.15.67:0/4190009797 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcc000d1550 msgr2=0x7fcc0006cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:34.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.682+0000 7fcbfe59c640 1 --2- 172.21.15.67:0/4190009797 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcc000d1550 0x7fcc0006cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:34.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.682+0000 7fcbfed9d640 1 --2- 172.21.15.67:0/4190009797 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcc000d1550 0x7fcc0006cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:34.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.682+0000 7fcbfed9d640 1 -- 172.21.15.67:0/4190009797 >> v1:172.21.15.67:6789/0 conn(0x7fcc000b2780 legacy=0x7fcc00099d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:34.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.682+0000 7fcbfd59a640 1 -- 172.21.15.67:0/4190009797 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1483619544 0 0) 0x7fcc000d1930 con 0x7fcc000b2780 2024-03-24T00:40:34.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.682+0000 7fcbfed9d640 1 -- 172.21.15.67:0/4190009797 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc000b1f60 con 0x7fcc0009a480 2024-03-24T00:40:34.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.683+0000 7fcbfed9d640 1 --2- 172.21.15.67:0/4190009797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc0009a480 0x7fcc000b2c00 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fcbf0009d90 tx=0x7fcbf002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fd0ab49f9f9ca45 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:34.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.683+0000 7fcbfd59a640 1 -- 172.21.15.67:0/4190009797 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf0039070 con 0x7fcc0009a480 2024-03-24T00:40:34.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.683+0000 7fcbfd59a640 1 -- 172.21.15.67:0/4190009797 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbf002cc50 con 0x7fcc0009a480 2024-03-24T00:40:34.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.683+0000 7fcbfd59a640 1 -- 172.21.15.67:0/4190009797 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf0034440 con 0x7fcc0009a480 2024-03-24T00:40:34.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.684+0000 7fcc04b87640 1 -- 172.21.15.67:0/4190009797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc0009a480 msgr2=0x7fcc000b2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:34.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.684+0000 7fcc04b87640 1 --2- 172.21.15.67:0/4190009797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc0009a480 0x7fcc000b2c00 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fcbf0009d90 tx=0x7fcbf002bea0 comp rx=0 tx=0).stop 2024-03-24T00:40:34.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.684+0000 7fcc04b87640 1 -- 172.21.15.67:0/4190009797 shutdown_connections 2024-03-24T00:40:34.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.684+0000 7fcc04b87640 1 --2- 172.21.15.67:0/4190009797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc0009a480 0x7fcc000b2c00 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:34.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.684+0000 7fcc04b87640 1 --2- 172.21.15.67:0/4190009797 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcc000d1550 0x7fcc0006cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:34.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.684+0000 7fcc04b87640 1 -- 172.21.15.67:0/4190009797 >> 172.21.15.67:0/4190009797 conn(0x7fcc000b36e0 msgr2=0x7fcc000b3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:34.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.684+0000 7fcc04b87640 1 -- 172.21.15.67:0/4190009797 shutdown_connections 2024-03-24T00:40:34.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.684+0000 7fcc04b87640 1 -- 172.21.15.67:0/4190009797 wait complete. 2024-03-24T00:40:34.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.685+0000 7fcc04b87640 1 Processor -- start 2024-03-24T00:40:34.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.685+0000 7fcc04b87640 1 -- start start 2024-03-24T00:40:34.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.686+0000 7fcc04b87640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcc0009a480 0x7fcc00160750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:34.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.686+0000 7fcc04b87640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc000b2780 0x7fcc0009b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:34.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.686+0000 7fcc04b87640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc000d1550 0x7fcc0009be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:34.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.686+0000 7fcc04b87640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0006c400 con 0x7fcc000b2780 2024-03-24T00:40:34.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.686+0000 7fcc04b87640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc000b2520 con 0x7fcc000d1550 2024-03-24T00:40:34.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.686+0000 7fcc04b87640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0006c6e0 con 0x7fcc0009a480 2024-03-24T00:40:34.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.686+0000 7fcbfdd9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc000b2780 0x7fcc0009b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:34.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.686+0000 7fcbfdd9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc000b2780 0x7fcc0009b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34898/0 (socket says 172.21.15.67:34898) 2024-03-24T00:40:34.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.686+0000 7fcbfdd9b640 1 -- 172.21.15.67:0/1374624650 learned_addr learned my addr 172.21.15.67:0/1374624650 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:34.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.686+0000 7fcbfed9d640 1 --2- 172.21.15.67:0/1374624650 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc000d1550 0x7fcc0009be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:34.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.686+0000 7fcbfe59c640 1 --2- 172.21.15.67:0/1374624650 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcc0009a480 0x7fcc00160750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:34.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.687+0000 7fcbfdd9b640 1 -- 172.21.15.67:0/1374624650 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcc0009a480 msgr2=0x7fcc00160750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:34.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.687+0000 7fcbfdd9b640 1 --2- 172.21.15.67:0/1374624650 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcc0009a480 0x7fcc00160750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:34.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.687+0000 7fcbfdd9b640 1 -- 172.21.15.67:0/1374624650 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc000d1550 msgr2=0x7fcc0009be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:34.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.687+0000 7fcbfdd9b640 1 --2- 172.21.15.67:0/1374624650 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc000d1550 0x7fcc0009be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:34.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.687+0000 7fcbfdd9b640 1 -- 172.21.15.67:0/1374624650 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc000cb920 con 0x7fcc000b2780 2024-03-24T00:40:34.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.687+0000 7fcbfdd9b640 1 --2- 172.21.15.67:0/1374624650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc000b2780 0x7fcc0009b8d0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fcbf400c4d0 tx=0x7fcbf400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:34.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.687+0000 7fcbeeffd640 1 -- 172.21.15.67:0/1374624650 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf4018070 con 0x7fcc000b2780 2024-03-24T00:40:34.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.687+0000 7fcbeeffd640 1 -- 172.21.15.67:0/1374624650 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbf4007500 con 0x7fcc000b2780 2024-03-24T00:40:34.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.687+0000 7fcc04b87640 1 -- 172.21.15.67:0/1374624650 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc000b5640 con 0x7fcc000b2780 2024-03-24T00:40:34.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.687+0000 7fcbeeffd640 1 -- 172.21.15.67:0/1374624650 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf4004a20 con 0x7fcc000b2780 2024-03-24T00:40:34.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.687+0000 7fcc04b87640 1 -- 172.21.15.67:0/1374624650 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc000b5c00 con 0x7fcc000b2780 2024-03-24T00:40:34.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.688+0000 7fcc04b87640 1 -- 172.21.15.67:0/1374624650 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc0004f400 con 0x7fcc000b2780 2024-03-24T00:40:34.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.688+0000 7fcbeeffd640 1 -- 172.21.15.67:0/1374624650 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fcbf401b070 con 0x7fcc000b2780 2024-03-24T00:40:34.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.689+0000 7fcbeeffd640 1 --2- 172.21.15.67:0/1374624650 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcbe006bbd0 0x7fcbe006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:34.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.689+0000 7fcbeeffd640 1 -- 172.21.15.67:0/1374624650 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5335+0+0 (secure 0 0 0) 0x7fcbf4089b80 con 0x7fcc000b2780 2024-03-24T00:40:34.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.689+0000 7fcbfe59c640 1 --2- 172.21.15.67:0/1374624650 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcbe006bbd0 0x7fcbe006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:34.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.689+0000 7fcbfe59c640 1 --2- 172.21.15.67:0/1374624650 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcbe006bbd0 0x7fcbe006e090 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fcbf800aa20 tx=0x7fcbf8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:34.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.693+0000 7fcbeeffd640 1 -- 172.21.15.67:0/1374624650 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fcbf4058190 con 0x7fcc000b2780 2024-03-24T00:40:34.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.941+0000 7fcc04b87640 1 -- 172.21.15.67:0/1374624650 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_backtrace_repair"]} v 0) v1 -- 0x7fcc000b1310 con 0x7fcc000b2780 2024-03-24T00:40:34.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.945+0000 7fcbeeffd640 1 -- 172.21.15.67:0/1374624650 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_backtrace_repair"]}]=0 v116) v1 ==== 152+0+0 (secure 0 0 0) 0x7fcbf405c230 con 0x7fcc000b2780 2024-03-24T00:40:34.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.947+0000 7fcc04b87640 1 -- 172.21.15.67:0/1374624650 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcbe006bbd0 msgr2=0x7fcbe006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:34.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.947+0000 7fcc04b87640 1 --2- 172.21.15.67:0/1374624650 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcbe006bbd0 0x7fcbe006e090 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fcbf800aa20 tx=0x7fcbf8008040 comp rx=0 tx=0).stop 2024-03-24T00:40:34.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.947+0000 7fcc04b87640 1 -- 172.21.15.67:0/1374624650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc000b2780 msgr2=0x7fcc0009b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:34.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.947+0000 7fcc04b87640 1 --2- 172.21.15.67:0/1374624650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc000b2780 0x7fcc0009b8d0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fcbf400c4d0 tx=0x7fcbf400c990 comp rx=0 tx=0).stop 2024-03-24T00:40:34.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.947+0000 7fcc04b87640 1 -- 172.21.15.67:0/1374624650 shutdown_connections 2024-03-24T00:40:34.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.947+0000 7fcc04b87640 1 --2- 172.21.15.67:0/1374624650 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcbe006bbd0 0x7fcbe006e090 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:34.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.947+0000 7fcc04b87640 1 --2- 172.21.15.67:0/1374624650 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc000d1550 0x7fcc0009be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:34.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.947+0000 7fcc04b87640 1 --2- 172.21.15.67:0/1374624650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc000b2780 0x7fcc0009b8d0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:34.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.947+0000 7fcc04b87640 1 --2- 172.21.15.67:0/1374624650 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcc0009a480 0x7fcc00160750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:34.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.947+0000 7fcc04b87640 1 -- 172.21.15.67:0/1374624650 >> 172.21.15.67:0/1374624650 conn(0x7fcc000b36e0 msgr2=0x7fcc000b1530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:34.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.947+0000 7fcc04b87640 1 -- 172.21.15.67:0/1374624650 shutdown_connections 2024-03-24T00:40:34.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:34.947+0000 7fcc04b87640 1 -- 172.21.15.67:0/1374624650 wait complete. 2024-03-24T00:40:34.959 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-24T00:40:35.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.063+0000 7feeaeb77640 1 Processor -- start 2024-03-24T00:40:35.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.063+0000 7feeaeb77640 1 -- start start 2024-03-24T00:40:35.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.064+0000 7feeaeb77640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7feea80b49b0 0x7feea80d0e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:35.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.064+0000 7feeaeb77640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7feea80d1360 0x7feea815d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:35.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.064+0000 7feeaeb77640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feea80cc400 con 0x7feea8099d00 2024-03-24T00:40:35.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.064+0000 7feeaeb77640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feea80cc1b0 con 0x7feea80b49b0 2024-03-24T00:40:35.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.064+0000 7feeaeb77640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feea80cc6e0 con 0x7feea80d1360 2024-03-24T00:40:35.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.064+0000 7feead114640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7feea80d1360 0x7feea815d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:35.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.064+0000 7feea7fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7feea80b49b0 0x7feea80d0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:35.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.064+0000 7feea7fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7feea80b49b0 0x7feea80d0e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:49878/0 (socket says 172.21.15.67:49878) 2024-03-24T00:40:35.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.064+0000 7feea7fff640 1 -- 172.21.15.67:0/2369295107 learned_addr learned my addr 172.21.15.67:0/2369295107 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:35.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.065+0000 7feead114640 1 -- 172.21.15.67:0/2369295107 >> v2:172.21.15.72:3300/0 conn(0x7feea80b49b0 msgr2=0x7feea80d0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:35.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.065+0000 7feead114640 1 --2- 172.21.15.67:0/2369295107 >> v2:172.21.15.72:3300/0 conn(0x7feea80b49b0 0x7feea80d0e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:35.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.065+0000 7feead114640 1 -- 172.21.15.67:0/2369295107 >> v1:172.21.15.72:6789/0 conn(0x7feea8099d00 legacy=0x7feea809a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:35.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.065+0000 7feead114640 1 -- 172.21.15.67:0/2369295107 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feea80cc9c0 con 0x7feea80d1360 2024-03-24T00:40:35.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.065+0000 7feea7fff640 1 --2- 172.21.15.67:0/2369295107 >> v2:172.21.15.72:3300/0 conn(0x7feea80b49b0 0x7feea80d0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:40:35.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.065+0000 7feead114640 1 --2- 172.21.15.67:0/2369295107 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7feea80d1360 0x7feea815d7d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7feea0009d90 tx=0x7feea002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c0036b5d40fa30b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:35.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.066+0000 7feea77fe640 1 -- 172.21.15.67:0/2369295107 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea0039070 con 0x7feea80d1360 2024-03-24T00:40:35.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.066+0000 7feea77fe640 1 -- 172.21.15.67:0/2369295107 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feea002cc50 con 0x7feea80d1360 2024-03-24T00:40:35.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.066+0000 7feea77fe640 1 -- 172.21.15.67:0/2369295107 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea0034440 con 0x7feea80d1360 2024-03-24T00:40:35.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.066+0000 7feeaeb77640 1 -- 172.21.15.67:0/2369295107 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7feea80d1360 msgr2=0x7feea815d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:35.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.066+0000 7feeaeb77640 1 --2- 172.21.15.67:0/2369295107 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7feea80d1360 0x7feea815d7d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7feea0009d90 tx=0x7feea002bea0 comp rx=0 tx=0).stop 2024-03-24T00:40:35.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.066+0000 7feeaeb77640 1 -- 172.21.15.67:0/2369295107 shutdown_connections 2024-03-24T00:40:35.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.066+0000 7feeaeb77640 1 --2- 172.21.15.67:0/2369295107 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7feea80d1360 0x7feea815d7d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:35.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.066+0000 7feeaeb77640 1 --2- 172.21.15.67:0/2369295107 >> v2:172.21.15.72:3300/0 conn(0x7feea80b49b0 0x7feea80d0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:35.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.066+0000 7feeaeb77640 1 -- 172.21.15.67:0/2369295107 >> 172.21.15.67:0/2369295107 conn(0x7feea80b3650 msgr2=0x7feea80b3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:35.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.066+0000 7feeaeb77640 1 -- 172.21.15.67:0/2369295107 shutdown_connections 2024-03-24T00:40:35.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.066+0000 7feeaeb77640 1 -- 172.21.15.67:0/2369295107 wait complete. 2024-03-24T00:40:35.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.067+0000 7feeaeb77640 1 Processor -- start 2024-03-24T00:40:35.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.067+0000 7feeaeb77640 1 -- start start 2024-03-24T00:40:35.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.068+0000 7feeaeb77640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feea8099d00 0x7feea815dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:35.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.068+0000 7feeaeb77640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feea80b49b0 0x7feea80cb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:35.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.068+0000 7feeaeb77640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7feea80d1360 0x7feea80c7730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:35.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.068+0000 7feeaeb77640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea80b32a0 con 0x7feea8099d00 2024-03-24T00:40:35.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.068+0000 7feeaeb77640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea80d01c0 con 0x7feea80b49b0 2024-03-24T00:40:35.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.068+0000 7feeaeb77640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feea80b3420 con 0x7feea80d1360 2024-03-24T00:40:35.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.068+0000 7feeac913640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feea8099d00 0x7feea815dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:35.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.068+0000 7feeac913640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feea8099d00 0x7feea815dc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34914/0 (socket says 172.21.15.67:34914) 2024-03-24T00:40:35.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.068+0000 7feeac913640 1 -- 172.21.15.67:0/929935075 learned_addr learned my addr 172.21.15.67:0/929935075 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:35.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.068+0000 7feea7fff640 1 --2- 172.21.15.67:0/929935075 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feea80b49b0 0x7feea80cb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:35.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.068+0000 7feead114640 1 --2- 172.21.15.67:0/929935075 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7feea80d1360 0x7feea80c7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:35.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.069+0000 7feeac913640 1 -- 172.21.15.67:0/929935075 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7feea80d1360 msgr2=0x7feea80c7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:35.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.069+0000 7feeac913640 1 --2- 172.21.15.67:0/929935075 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7feea80d1360 0x7feea80c7730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:35.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.069+0000 7feeac913640 1 -- 172.21.15.67:0/929935075 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feea80b49b0 msgr2=0x7feea80cb8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:35.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.069+0000 7feeac913640 1 --2- 172.21.15.67:0/929935075 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feea80b49b0 0x7feea80cb8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:35.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.069+0000 7feeac913640 1 -- 172.21.15.67:0/929935075 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feea80c9b90 con 0x7feea8099d00 2024-03-24T00:40:35.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.069+0000 7feeac913640 1 --2- 172.21.15.67:0/929935075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feea8099d00 0x7feea815dc50 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fee9800c3f0 tx=0x7fee9800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:35.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.069+0000 7feea57fa640 1 -- 172.21.15.67:0/929935075 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee980052d0 con 0x7feea8099d00 2024-03-24T00:40:35.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.069+0000 7feea57fa640 1 -- 172.21.15.67:0/929935075 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee98005490 con 0x7feea8099d00 2024-03-24T00:40:35.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.069+0000 7feeaeb77640 1 -- 172.21.15.67:0/929935075 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feea80c9e70 con 0x7feea8099d00 2024-03-24T00:40:35.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.069+0000 7feea57fa640 1 -- 172.21.15.67:0/929935075 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee98007260 con 0x7feea8099d00 2024-03-24T00:40:35.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.069+0000 7feeaeb77640 1 -- 172.21.15.67:0/929935075 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feea80ca430 con 0x7feea8099d00 2024-03-24T00:40:35.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.070+0000 7feeaeb77640 1 -- 172.21.15.67:0/929935075 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feea804f400 con 0x7feea8099d00 2024-03-24T00:40:35.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.070+0000 7feea57fa640 1 -- 172.21.15.67:0/929935075 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fee98007400 con 0x7feea8099d00 2024-03-24T00:40:35.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.071+0000 7feea57fa640 1 --2- 172.21.15.67:0/929935075 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fee9406bbd0 0x7fee9406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:35.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.071+0000 7feea57fa640 1 -- 172.21.15.67:0/929935075 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5335+0+0 (secure 0 0 0) 0x7fee98089c70 con 0x7feea8099d00 2024-03-24T00:40:35.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.071+0000 7feea7fff640 1 --2- 172.21.15.67:0/929935075 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fee9406bbd0 0x7fee9406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:35.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.071+0000 7feea7fff640 1 --2- 172.21.15.67:0/929935075 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fee9406bbd0 0x7fee9406e090 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7feea80ca8a0 tx=0x7fee9c009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:35.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.075+0000 7feea57fa640 1 -- 172.21.15.67:0/929935075 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fee98058310 con 0x7feea8099d00 2024-03-24T00:40:35.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.257+0000 7feeaeb77640 1 -- 172.21.15.67:0/929935075 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7feea8070560 con 0x7feea8099d00 2024-03-24T00:40:35.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.260+0000 7feea57fa640 1 -- 172.21.15.67:0/929935075 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) v1 ==== 122+0+0 (secure 0 0 0) 0x7fee9805c3b0 con 0x7feea8099d00 2024-03-24T00:40:35.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.262+0000 7feeaeb77640 1 -- 172.21.15.67:0/929935075 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fee9406bbd0 msgr2=0x7fee9406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:35.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.262+0000 7feeaeb77640 1 --2- 172.21.15.67:0/929935075 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fee9406bbd0 0x7fee9406e090 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7feea80ca8a0 tx=0x7fee9c009240 comp rx=0 tx=0).stop 2024-03-24T00:40:35.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.262+0000 7feeaeb77640 1 -- 172.21.15.67:0/929935075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feea8099d00 msgr2=0x7feea815dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:35.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.262+0000 7feeaeb77640 1 --2- 172.21.15.67:0/929935075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feea8099d00 0x7feea815dc50 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fee9800c3f0 tx=0x7fee9800c8b0 comp rx=0 tx=0).stop 2024-03-24T00:40:35.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.262+0000 7feeaeb77640 1 -- 172.21.15.67:0/929935075 shutdown_connections 2024-03-24T00:40:35.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.262+0000 7feeaeb77640 1 --2- 172.21.15.67:0/929935075 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fee9406bbd0 0x7fee9406e090 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:35.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.262+0000 7feeaeb77640 1 --2- 172.21.15.67:0/929935075 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7feea80d1360 0x7feea80c7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:35.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.262+0000 7feeaeb77640 1 --2- 172.21.15.67:0/929935075 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feea80b49b0 0x7feea80cb8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:35.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.262+0000 7feeaeb77640 1 --2- 172.21.15.67:0/929935075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feea8099d00 0x7feea815dc50 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:35.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.262+0000 7feeaeb77640 1 -- 172.21.15.67:0/929935075 >> 172.21.15.67:0/929935075 conn(0x7feea80b3650 msgr2=0x7feea80b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:35.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.262+0000 7feeaeb77640 1 -- 172.21.15.67:0/929935075 shutdown_connections 2024-03-24T00:40:35.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.262+0000 7feeaeb77640 1 -- 172.21.15.67:0/929935075 wait complete. 2024-03-24T00:40:35.274 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-24T00:40:35.274 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:40:35.310 DEBUG:teuthology.orchestra.run.smithi072:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:40:35.356 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-24T00:40:35.356 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:40:35.356 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:40:35.371 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056375,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15771 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T00:40:35.371 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:40:35.371 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:40:35.388 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17590 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248808k,nr_inodes=812202,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:40:35.388 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:40:35.388 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:40:35.426 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056375,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15771 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T00:40:35.426 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:40:35.426 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:40:35.445 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17590 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248808k,nr_inodes=812202,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:40:35.445 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-24T00:40:35.445 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:40:35.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.567+0000 7fb745e6c640 1 Processor -- start 2024-03-24T00:40:35.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.567+0000 7fb745e6c640 1 -- start start 2024-03-24T00:40:35.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.567+0000 7fb745e6c640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb7400d1550 0x7fb74006cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:35.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.567+0000 7fb745e6c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb74009a480 0x7fb7400b2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:35.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.567+0000 7fb745e6c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7400d1930 con 0x7fb7400b2780 2024-03-24T00:40:35.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.567+0000 7fb745e6c640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb74006d420 con 0x7fb74009a480 2024-03-24T00:40:35.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.567+0000 7fb745e6c640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb74006ca50 con 0x7fb7400d1550 2024-03-24T00:40:35.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.567+0000 7fb73ffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb74009a480 0x7fb7400b2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:35.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.568+0000 7fb73ffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb74009a480 0x7fb7400b2c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34928/0 (socket says 172.21.15.67:34928) 2024-03-24T00:40:35.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.568+0000 7fb73ffff640 1 -- 172.21.15.67:0/2833809826 learned_addr learned my addr 172.21.15.67:0/2833809826 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:35.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.568+0000 7fb73ffff640 1 -- 172.21.15.67:0/2833809826 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb7400d1550 msgr2=0x7fb74006cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:35.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.568+0000 7fb73f7fe640 1 --2- 172.21.15.67:0/2833809826 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb7400d1550 0x7fb74006cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:35.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.568+0000 7fb73ffff640 1 --2- 172.21.15.67:0/2833809826 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb7400d1550 0x7fb74006cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:35.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.568+0000 7fb73ffff640 1 -- 172.21.15.67:0/2833809826 >> v1:172.21.15.72:6789/0 conn(0x7fb7400b2780 legacy=0x7fb740099d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:35.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.568+0000 7fb73ffff640 1 -- 172.21.15.67:0/2833809826 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7400b1f60 con 0x7fb74009a480 2024-03-24T00:40:35.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.569+0000 7fb73ffff640 1 --2- 172.21.15.67:0/2833809826 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb74009a480 0x7fb7400b2c00 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fb738009e50 tx=0x7fb73802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63e0d52fec169cec server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:35.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.569+0000 7fb73e7fc640 1 -- 172.21.15.67:0/2833809826 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73802ca70 con 0x7fb74009a480 2024-03-24T00:40:35.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.569+0000 7fb73e7fc640 1 -- 172.21.15.67:0/2833809826 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb73802cc10 con 0x7fb74009a480 2024-03-24T00:40:35.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.569+0000 7fb73e7fc640 1 -- 172.21.15.67:0/2833809826 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7380343f0 con 0x7fb74009a480 2024-03-24T00:40:35.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.569+0000 7fb745e6c640 1 -- 172.21.15.67:0/2833809826 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb74009a480 msgr2=0x7fb7400b2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:35.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.569+0000 7fb745e6c640 1 --2- 172.21.15.67:0/2833809826 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb74009a480 0x7fb7400b2c00 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fb738009e50 tx=0x7fb73802bed0 comp rx=0 tx=0).stop 2024-03-24T00:40:35.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.569+0000 7fb745e6c640 1 -- 172.21.15.67:0/2833809826 shutdown_connections 2024-03-24T00:40:35.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.569+0000 7fb745e6c640 1 --2- 172.21.15.67:0/2833809826 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb74009a480 0x7fb7400b2c00 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:35.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.569+0000 7fb745e6c640 1 --2- 172.21.15.67:0/2833809826 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb7400d1550 0x7fb74006cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:35.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.569+0000 7fb745e6c640 1 -- 172.21.15.67:0/2833809826 >> 172.21.15.67:0/2833809826 conn(0x7fb7400b36e0 msgr2=0x7fb7400b3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:35.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.569+0000 7fb745e6c640 1 -- 172.21.15.67:0/2833809826 shutdown_connections 2024-03-24T00:40:35.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.569+0000 7fb745e6c640 1 -- 172.21.15.67:0/2833809826 wait complete. 2024-03-24T00:40:35.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.570+0000 7fb745e6c640 1 Processor -- start 2024-03-24T00:40:35.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.571+0000 7fb745e6c640 1 -- start start 2024-03-24T00:40:35.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.571+0000 7fb745e6c640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb74009a480 0x7fb7400c7650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:35.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.571+0000 7fb745e6c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7400b2780 0x7fb74015b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:35.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.571+0000 7fb745e6c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7400d1550 0x7fb7401606b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:35.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.571+0000 7fb745e6c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74006c400 con 0x7fb7400d1550 2024-03-24T00:40:35.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.571+0000 7fb745e6c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7400b2520 con 0x7fb7400b2780 2024-03-24T00:40:35.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.571+0000 7fb745e6c640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb74006c6e0 con 0x7fb74009a480 2024-03-24T00:40:35.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.571+0000 7fb73ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7400d1550 0x7fb7401606b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:35.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.571+0000 7fb73ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7400d1550 0x7fb7401606b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34942/0 (socket says 172.21.15.67:34942) 2024-03-24T00:40:35.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.571+0000 7fb73ffff640 1 -- 172.21.15.67:0/4259191217 learned_addr learned my addr 172.21.15.67:0/4259191217 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:35.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.572+0000 7fb73effd640 1 --2- 172.21.15.67:0/4259191217 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7400b2780 0x7fb74015b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:35.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.572+0000 7fb73f7fe640 1 --2- 172.21.15.67:0/4259191217 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb74009a480 0x7fb7400c7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:35.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.572+0000 7fb73ffff640 1 -- 172.21.15.67:0/4259191217 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb74009a480 msgr2=0x7fb7400c7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:35.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.572+0000 7fb73ffff640 1 --2- 172.21.15.67:0/4259191217 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb74009a480 0x7fb7400c7650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:35.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.572+0000 7fb73ffff640 1 -- 172.21.15.67:0/4259191217 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7400b2780 msgr2=0x7fb74015b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:35.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.572+0000 7fb73ffff640 1 --2- 172.21.15.67:0/4259191217 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7400b2780 0x7fb74015b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:35.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.572+0000 7fb73ffff640 1 -- 172.21.15.67:0/4259191217 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7400c71b0 con 0x7fb7400d1550 2024-03-24T00:40:35.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.572+0000 7fb73ffff640 1 --2- 172.21.15.67:0/4259191217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7400d1550 0x7fb7401606b0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fb738036040 tx=0x7fb738004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:35.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.572+0000 7fb737fff640 1 -- 172.21.15.67:0/4259191217 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb738043020 con 0x7fb7400d1550 2024-03-24T00:40:35.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.572+0000 7fb737fff640 1 -- 172.21.15.67:0/4259191217 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb73803e450 con 0x7fb7400d1550 2024-03-24T00:40:35.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.572+0000 7fb737fff640 1 -- 172.21.15.67:0/4259191217 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb738009510 con 0x7fb7400d1550 2024-03-24T00:40:35.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.572+0000 7fb745e6c640 1 -- 172.21.15.67:0/4259191217 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7401e8130 con 0x7fb7400d1550 2024-03-24T00:40:35.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.572+0000 7fb745e6c640 1 -- 172.21.15.67:0/4259191217 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7401e86f0 con 0x7fb7400d1550 2024-03-24T00:40:35.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.573+0000 7fb745e6c640 1 -- 172.21.15.67:0/4259191217 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb74004f400 con 0x7fb7400d1550 2024-03-24T00:40:35.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.574+0000 7fb737fff640 1 -- 172.21.15.67:0/4259191217 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb738046020 con 0x7fb7400d1550 2024-03-24T00:40:35.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.574+0000 7fb737fff640 1 --2- 172.21.15.67:0/4259191217 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb72006bc00 0x7fb72006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:35.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.574+0000 7fb737fff640 1 -- 172.21.15.67:0/4259191217 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5335+0+0 (secure 0 0 0) 0x7fb7380ad2f0 con 0x7fb7400d1550 2024-03-24T00:40:35.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.574+0000 7fb73f7fe640 1 --2- 172.21.15.67:0/4259191217 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb72006bc00 0x7fb72006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:35.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.575+0000 7fb73f7fe640 1 --2- 172.21.15.67:0/4259191217 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb72006bc00 0x7fb72006e0c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fb72800a970 tx=0x7fb728008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:35.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.579+0000 7fb737fff640 1 -- 172.21.15.67:0/4259191217 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb73804d030 con 0x7fb7400d1550 2024-03-24T00:40:35.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.796+0000 7fb745e6c640 1 -- 172.21.15.67:0/4259191217 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7400b1310 con 0x7fb7400d1550 2024-03-24T00:40:35.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.797+0000 7fb737fff640 1 -- 172.21.15.67:0/4259191217 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+3696 (secure 0 0 0) 0x7fb73803c350 con 0x7fb7400d1550 2024-03-24T00:40:35.798 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:40:35.798 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":19,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5039,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/166859817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":166859817},{"type":"v1","addr":"172.21.15.67:6834","nonce":166859817}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5042,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2653663845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2653663845},{"type":"v1","addr":"172.21.15.67:6835","nonce":2653663845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5048,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2355909634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2355909634},{"type":"v1","addr":"172.21.15.72:6837","nonce":2355909634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5737,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/2590542549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2590542549},{"type":"v1","addr":"172.21.15.72:6836","nonce":2590542549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[]} 2024-03-24T00:40:35.798 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 19 2024-03-24T00:40:35.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.799+0000 7fb745e6c640 1 -- 172.21.15.67:0/4259191217 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb72006bc00 msgr2=0x7fb72006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:35.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.799+0000 7fb745e6c640 1 --2- 172.21.15.67:0/4259191217 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb72006bc00 0x7fb72006e0c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fb72800a970 tx=0x7fb728008040 comp rx=0 tx=0).stop 2024-03-24T00:40:35.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.799+0000 7fb745e6c640 1 -- 172.21.15.67:0/4259191217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7400d1550 msgr2=0x7fb7401606b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:35.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.799+0000 7fb745e6c640 1 --2- 172.21.15.67:0/4259191217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7400d1550 0x7fb7401606b0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fb738036040 tx=0x7fb738004a40 comp rx=0 tx=0).stop 2024-03-24T00:40:35.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.799+0000 7fb745e6c640 1 -- 172.21.15.67:0/4259191217 shutdown_connections 2024-03-24T00:40:35.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.799+0000 7fb745e6c640 1 --2- 172.21.15.67:0/4259191217 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb72006bc00 0x7fb72006e0c0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:35.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.799+0000 7fb745e6c640 1 --2- 172.21.15.67:0/4259191217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7400d1550 0x7fb7401606b0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:35.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.799+0000 7fb745e6c640 1 --2- 172.21.15.67:0/4259191217 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7400b2780 0x7fb74015b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:35.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.799+0000 7fb745e6c640 1 --2- 172.21.15.67:0/4259191217 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb74009a480 0x7fb7400c7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:35.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.799+0000 7fb745e6c640 1 -- 172.21.15.67:0/4259191217 >> 172.21.15.67:0/4259191217 conn(0x7fb7400b36e0 msgr2=0x7fb7400cd3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:35.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.799+0000 7fb745e6c640 1 -- 172.21.15.67:0/4259191217 shutdown_connections 2024-03-24T00:40:35.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.799+0000 7fb745e6c640 1 -- 172.21.15.67:0/4259191217 wait complete. 2024-03-24T00:40:35.811 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-24T00:40:35.811 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-24T00:40:35.811 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-24T00:40:35.811 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:40:35.811 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-24T00:40:35.811 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-24T00:40:35.812 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-24T00:40:35.812 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:40:35.812 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-24T00:40:35.812 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-24T00:40:35.812 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-24T00:40:35.812 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:40:35.812 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-24T00:40:35.812 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-24T00:40:35.813 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-24T00:40:35.813 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:40:35.820 INFO:tasks.ceph.mds.a:Stopped 2024-03-24T00:40:35.820 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-24T00:40:35.821 INFO:tasks.ceph.mds.c:Stopped 2024-03-24T00:40:35.821 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-24T00:40:35.825 INFO:tasks.ceph.mds.b:Stopped 2024-03-24T00:40:35.825 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-24T00:40:35.825 INFO:tasks.ceph.mds.d:Stopped 2024-03-24T00:40:35.826 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-24T00:40:35.863 INFO:tasks.ceph.mds.a:Started 2024-03-24T00:40:35.863 DEBUG:teuthology.parallel:result is None 2024-03-24T00:40:35.863 INFO:tasks.ceph.mds.c:Started 2024-03-24T00:40:35.863 DEBUG:teuthology.parallel:result is None 2024-03-24T00:40:35.867 INFO:tasks.ceph.mds.b:Started 2024-03-24T00:40:35.868 DEBUG:teuthology.parallel:result is None 2024-03-24T00:40:35.868 INFO:tasks.ceph.mds.d:Started 2024-03-24T00:40:35.868 DEBUG:teuthology.parallel:result is None 2024-03-24T00:40:35.869 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-24T00:40:35.938 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-03-24T00:40:35.939 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-03-24T00:40:35.951 INFO:tasks.ceph.mds.d.smithi072.stdout:starting mds.d at 2024-03-24T00:40:35.952 INFO:tasks.ceph.mds.b.smithi072.stdout:starting mds.b at 2024-03-24T00:40:35.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.976+0000 7fcb8a576640 1 Processor -- start 2024-03-24T00:40:35.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.976+0000 7fcb8a576640 1 -- start start 2024-03-24T00:40:35.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.977+0000 7fcb8a576640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcb840d25f0 0x7fcb84070a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:35.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.977+0000 7fcb8a576640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb8406fb70 con 0x7fcb8415bfa0 2024-03-24T00:40:35.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.977+0000 7fcb8a576640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb840cd270 con 0x7fcb840ce9c0 2024-03-24T00:40:35.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.977+0000 7fcb8a576640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb840d0af0 con 0x7fcb840d25f0 2024-03-24T00:40:35.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.977+0000 7fcb83fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fcb8415bfa0 0x7fcb840ce2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52530/0 (socket says 172.21.15.67:52530) 2024-03-24T00:40:35.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.977+0000 7fcb83fff640 1 -- 172.21.15.67:0/146098520 learned_addr learned my addr 172.21.15.67:0/146098520 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:40:35.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.977+0000 7fcb88b13640 1 --2- 172.21.15.67:0/146098520 >> v2:172.21.15.67:3300/0 conn(0x7fcb840d25f0 0x7fcb84070a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:35.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.978+0000 7fcb88b13640 1 -- 172.21.15.67:0/146098520 >> v1:172.21.15.72:6789/0 conn(0x7fcb840ce9c0 legacy=0x7fcb84070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:35.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.978+0000 7fcb88b13640 1 -- 172.21.15.67:0/146098520 >> v1:172.21.15.67:6789/0 conn(0x7fcb8415bfa0 legacy=0x7fcb840ce2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:35.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.978+0000 7fcb88b13640 1 -- 172.21.15.67:0/146098520 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb840cd4f0 con 0x7fcb840d25f0 2024-03-24T00:40:35.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.978+0000 7fcb82ffd640 1 -- 172.21.15.67:0/146098520 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 371269255 0 0) 0x7fcb78001110 con 0x7fcb840ce9c0 2024-03-24T00:40:35.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.978+0000 7fcb88b13640 1 --2- 172.21.15.67:0/146098520 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb840d25f0 0x7fcb84070a10 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c009d90 tx=0x7fcb7c02bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89a5ffb63bd86558 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:35.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.979+0000 7fcb82ffd640 1 -- 172.21.15.67:0/146098520 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb7c039070 con 0x7fcb840d25f0 2024-03-24T00:40:35.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.979+0000 7fcb82ffd640 1 -- 172.21.15.67:0/146098520 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb7c02cd80 con 0x7fcb840d25f0 2024-03-24T00:40:35.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.979+0000 7fcb82ffd640 1 -- 172.21.15.67:0/146098520 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb7c034660 con 0x7fcb840d25f0 2024-03-24T00:40:35.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.979+0000 7fcb8a576640 1 -- 172.21.15.67:0/146098520 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb840d25f0 msgr2=0x7fcb84070a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:35.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.979+0000 7fcb8a576640 1 --2- 172.21.15.67:0/146098520 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb840d25f0 0x7fcb84070a10 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c009d90 tx=0x7fcb7c02bfd0 comp rx=0 tx=0).stop 2024-03-24T00:40:35.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.979+0000 7fcb8a576640 1 -- 172.21.15.67:0/146098520 shutdown_connections 2024-03-24T00:40:35.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.979+0000 7fcb8a576640 1 --2- 172.21.15.67:0/146098520 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb840d25f0 0x7fcb84070a10 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:35.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.979+0000 7fcb8a576640 1 -- 172.21.15.67:0/146098520 >> 172.21.15.67:0/146098520 conn(0x7fcb840d1ad0 msgr2=0x7fcb840d1ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:35.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.979+0000 7fcb8a576640 1 -- 172.21.15.67:0/146098520 shutdown_connections 2024-03-24T00:40:35.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.979+0000 7fcb8a576640 1 -- 172.21.15.67:0/146098520 wait complete. 2024-03-24T00:40:35.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.980+0000 7fcb8a576640 1 Processor -- start 2024-03-24T00:40:35.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.980+0000 7fcb8a576640 1 -- start start 2024-03-24T00:40:35.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.981+0000 7fcb8a576640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb840ce9c0 0x7fcb8415cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:35.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.981+0000 7fcb8a576640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb840d25f0 0x7fcb8415a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:35.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.981+0000 7fcb8a576640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcb8415bfa0 0x7fcb840b66d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:35.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.981+0000 7fcb8a576640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8415c380 con 0x7fcb840d25f0 2024-03-24T00:40:35.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.981+0000 7fcb8a576640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8406fde0 con 0x7fcb840ce9c0 2024-03-24T00:40:35.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.981+0000 7fcb8a576640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb840cdab0 con 0x7fcb8415bfa0 2024-03-24T00:40:35.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.981+0000 7fcb837fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb840d25f0 0x7fcb8415a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:35.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.981+0000 7fcb837fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb840d25f0 0x7fcb8415a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34964/0 (socket says 172.21.15.67:34964) 2024-03-24T00:40:35.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.981+0000 7fcb837fe640 1 -- 172.21.15.67:0/3743462514 learned_addr learned my addr 172.21.15.67:0/3743462514 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:35.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.982+0000 7fcb83fff640 1 --2- 172.21.15.67:0/3743462514 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb840ce9c0 0x7fcb8415cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:35.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.982+0000 7fcb88b13640 1 --2- 172.21.15.67:0/3743462514 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcb8415bfa0 0x7fcb840b66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:35.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.982+0000 7fcb837fe640 1 -- 172.21.15.67:0/3743462514 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcb8415bfa0 msgr2=0x7fcb840b66d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:35.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.982+0000 7fcb837fe640 1 --2- 172.21.15.67:0/3743462514 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcb8415bfa0 0x7fcb840b66d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:35.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.982+0000 7fcb837fe640 1 -- 172.21.15.67:0/3743462514 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb840ce9c0 msgr2=0x7fcb8415cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:35.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.982+0000 7fcb837fe640 1 --2- 172.21.15.67:0/3743462514 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb840ce9c0 0x7fcb8415cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:35.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.982+0000 7fcb837fe640 1 -- 172.21.15.67:0/3743462514 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb8409f4c0 con 0x7fcb840d25f0 2024-03-24T00:40:35.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.982+0000 7fcb837fe640 1 --2- 172.21.15.67:0/3743462514 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb840d25f0 0x7fcb8415a860 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fcb7800c3c0 tx=0x7fcb7800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:35.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.982+0000 7fcb80ff9640 1 -- 172.21.15.67:0/3743462514 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb78017070 con 0x7fcb840d25f0 2024-03-24T00:40:35.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.982+0000 7fcb80ff9640 1 -- 172.21.15.67:0/3743462514 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb780035f0 con 0x7fcb840d25f0 2024-03-24T00:40:35.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.982+0000 7fcb80ff9640 1 -- 172.21.15.67:0/3743462514 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb780039e0 con 0x7fcb840d25f0 2024-03-24T00:40:35.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.982+0000 7fcb8a576640 1 -- 172.21.15.67:0/3743462514 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb8415d610 con 0x7fcb840d25f0 2024-03-24T00:40:35.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.982+0000 7fcb8a576640 1 -- 172.21.15.67:0/3743462514 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb8415dbd0 con 0x7fcb840d25f0 2024-03-24T00:40:35.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.983+0000 7fcb8a576640 1 -- 172.21.15.67:0/3743462514 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb8415de10 con 0x7fcb840d25f0 2024-03-24T00:40:35.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.984+0000 7fcb80ff9640 1 -- 172.21.15.67:0/3743462514 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fcb7801a070 con 0x7fcb840d25f0 2024-03-24T00:40:35.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.984+0000 7fcb80ff9640 1 --2- 172.21.15.67:0/3743462514 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcb7006bbd0 0x7fcb7006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:35.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.984+0000 7fcb80ff9640 1 -- 172.21.15.67:0/3743462514 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5335+0+0 (secure 0 0 0) 0x7fcb78089be0 con 0x7fcb840d25f0 2024-03-24T00:40:35.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.984+0000 7fcb83fff640 1 --2- 172.21.15.67:0/3743462514 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcb7006bbd0 0x7fcb7006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:35.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.985+0000 7fcb83fff640 1 --2- 172.21.15.67:0/3743462514 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcb7006bbd0 0x7fcb7006e090 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fcb74000f60 tx=0x7fcb74009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:35.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:35.988+0000 7fcb80ff9640 1 -- 172.21.15.67:0/3743462514 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fcb8415de10 con 0x7fcb840d25f0 2024-03-24T00:40:36.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.183+0000 7fcb8a576640 1 -- 172.21.15.67:0/3743462514 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fcb8404f400 con 0x7fcb840d25f0 2024-03-24T00:40:36.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.274+0000 7fcb80ff9640 1 -- 172.21.15.67:0/3743462514 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v45) v1 ==== 97+0+0 (secure 0 0 0) 0x7fcb840cd270 con 0x7fcb840d25f0 2024-03-24T00:40:36.275 INFO:teuthology.orchestra.run.smithi067.stderr: removed all blocklist entries 2024-03-24T00:40:36.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.276+0000 7fcb8a576640 1 -- 172.21.15.67:0/3743462514 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcb7006bbd0 msgr2=0x7fcb7006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:36.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.277+0000 7fcb8a576640 1 --2- 172.21.15.67:0/3743462514 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcb7006bbd0 0x7fcb7006e090 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fcb74000f60 tx=0x7fcb74009040 comp rx=0 tx=0).stop 2024-03-24T00:40:36.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.277+0000 7fcb8a576640 1 -- 172.21.15.67:0/3743462514 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb840d25f0 msgr2=0x7fcb8415a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:36.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.277+0000 7fcb8a576640 1 --2- 172.21.15.67:0/3743462514 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb840d25f0 0x7fcb8415a860 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fcb7800c3c0 tx=0x7fcb7800c880 comp rx=0 tx=0).stop 2024-03-24T00:40:36.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.277+0000 7fcb8a576640 1 -- 172.21.15.67:0/3743462514 shutdown_connections 2024-03-24T00:40:36.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.277+0000 7fcb8a576640 1 --2- 172.21.15.67:0/3743462514 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcb7006bbd0 0x7fcb7006e090 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:36.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.277+0000 7fcb8a576640 1 --2- 172.21.15.67:0/3743462514 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcb8415bfa0 0x7fcb840b66d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:36.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.277+0000 7fcb8a576640 1 --2- 172.21.15.67:0/3743462514 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb840d25f0 0x7fcb8415a860 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:36.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.277+0000 7fcb8a576640 1 --2- 172.21.15.67:0/3743462514 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb840ce9c0 0x7fcb8415cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:36.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.277+0000 7fcb8a576640 1 -- 172.21.15.67:0/3743462514 >> 172.21.15.67:0/3743462514 conn(0x7fcb840d1ad0 msgr2=0x7fcb840b1060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:36.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.277+0000 7fcb8a576640 1 -- 172.21.15.67:0/3743462514 shutdown_connections 2024-03-24T00:40:36.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.277+0000 7fcb8a576640 1 -- 172.21.15.67:0/3743462514 wait complete. 2024-03-24T00:40:36.289 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-24T00:40:36.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.389+0000 7f9a22d1b640 1 Processor -- start 2024-03-24T00:40:36.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.389+0000 7f9a22d1b640 1 -- start start 2024-03-24T00:40:36.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.390+0000 7f9a22d1b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9a1c06cee0 0x7f9a1c099d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:36.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.390+0000 7f9a22d1b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9a1c09a2b0 0x7f9a1c0cfd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:36.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.390+0000 7f9a22d1b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a1c0d1930 con 0x7f9a1c0d1550 2024-03-24T00:40:36.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.390+0000 7f9a22d1b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a1c06ca50 con 0x7f9a1c06cee0 2024-03-24T00:40:36.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.390+0000 7f9a22d1b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1c06d390 con 0x7f9a1c09a2b0 2024-03-24T00:40:36.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.390+0000 7f9a1bfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9a1c06cee0 0x7f9a1c099d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:36.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.390+0000 7f9a20ab7640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9a1c0d1550 0x7f9a1c0b2780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52538/0 (socket says 172.21.15.67:52538) 2024-03-24T00:40:36.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.390+0000 7f9a20ab7640 1 -- 172.21.15.67:0/1314589712 learned_addr learned my addr 172.21.15.67:0/1314589712 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:40:36.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.390+0000 7f9a212b8640 1 --2- 172.21.15.67:0/1314589712 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9a1c09a2b0 0x7f9a1c0cfd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:36.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.390+0000 7f9a1bfff640 1 -- 172.21.15.67:0/1314589712 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9a1c09a2b0 msgr2=0x7f9a1c0cfd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:36.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.390+0000 7f9a1bfff640 1 --2- 172.21.15.67:0/1314589712 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9a1c09a2b0 0x7f9a1c0cfd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:36.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.390+0000 7f9a1bfff640 1 -- 172.21.15.67:0/1314589712 >> v1:172.21.15.67:6789/0 conn(0x7f9a1c0d1550 legacy=0x7f9a1c0b2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:36.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.390+0000 7f9a1bfff640 1 -- 172.21.15.67:0/1314589712 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a1c0b1f60 con 0x7f9a1c06cee0 2024-03-24T00:40:36.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.390+0000 7f9a212b8640 1 --2- 172.21.15.67:0/1314589712 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9a1c09a2b0 0x7f9a1c0cfd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:40:36.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.391+0000 7f9a1bfff640 1 --2- 172.21.15.67:0/1314589712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a1c06cee0 0x7f9a1c099d70 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f9a100097d0 tx=0x7f9a1002b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8252170f2d89a88e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:36.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.391+0000 7f9a1b7fe640 1 -- 172.21.15.67:0/1314589712 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a10035020 con 0x7f9a1c06cee0 2024-03-24T00:40:36.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.391+0000 7f9a1b7fe640 1 -- 172.21.15.67:0/1314589712 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a10031800 con 0x7f9a1c06cee0 2024-03-24T00:40:36.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.391+0000 7f9a1b7fe640 1 -- 172.21.15.67:0/1314589712 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a10031bd0 con 0x7f9a1c06cee0 2024-03-24T00:40:36.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.392+0000 7f9a22d1b640 1 -- 172.21.15.67:0/1314589712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a1c06cee0 msgr2=0x7f9a1c099d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:36.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.392+0000 7f9a22d1b640 1 --2- 172.21.15.67:0/1314589712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a1c06cee0 0x7f9a1c099d70 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f9a100097d0 tx=0x7f9a1002b8a0 comp rx=0 tx=0).stop 2024-03-24T00:40:36.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.392+0000 7f9a22d1b640 1 -- 172.21.15.67:0/1314589712 shutdown_connections 2024-03-24T00:40:36.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.392+0000 7f9a22d1b640 1 --2- 172.21.15.67:0/1314589712 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9a1c09a2b0 0x7f9a1c0cfd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:36.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.392+0000 7f9a22d1b640 1 --2- 172.21.15.67:0/1314589712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a1c06cee0 0x7f9a1c099d70 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:36.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.392+0000 7f9a22d1b640 1 -- 172.21.15.67:0/1314589712 >> 172.21.15.67:0/1314589712 conn(0x7f9a1c0b36e0 msgr2=0x7f9a1c0b3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:36.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.392+0000 7f9a22d1b640 1 -- 172.21.15.67:0/1314589712 shutdown_connections 2024-03-24T00:40:36.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.392+0000 7f9a22d1b640 1 -- 172.21.15.67:0/1314589712 wait complete. 2024-03-24T00:40:36.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.393+0000 7f9a22d1b640 1 Processor -- start 2024-03-24T00:40:36.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.393+0000 7f9a22d1b640 1 -- start start 2024-03-24T00:40:36.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.394+0000 7f9a22d1b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a1c06cee0 0x7f9a1c15b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:36.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.394+0000 7f9a22d1b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a1c09a2b0 0x7f9a1c160650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:36.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.394+0000 7f9a22d1b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9a1c0d1550 0x7f9a1c160b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:36.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.394+0000 7f9a22d1b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1c06c400 con 0x7f9a1c09a2b0 2024-03-24T00:40:36.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.394+0000 7f9a22d1b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1c0b2520 con 0x7f9a1c06cee0 2024-03-24T00:40:36.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.394+0000 7f9a22d1b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1c06c6e0 con 0x7f9a1c0d1550 2024-03-24T00:40:36.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.394+0000 7f9a1bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a1c09a2b0 0x7f9a1c160650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:36.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.394+0000 7f9a1bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a1c09a2b0 0x7f9a1c160650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34972/0 (socket says 172.21.15.67:34972) 2024-03-24T00:40:36.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.394+0000 7f9a1bfff640 1 -- 172.21.15.67:0/2912855434 learned_addr learned my addr 172.21.15.67:0/2912855434 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:36.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.394+0000 7f9a1bfff640 1 -- 172.21.15.67:0/2912855434 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9a1c0d1550 msgr2=0x7f9a1c160b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:36.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.394+0000 7f9a212b8640 1 --2- 172.21.15.67:0/2912855434 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9a1c0d1550 0x7f9a1c160b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:36.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.394+0000 7f9a20ab7640 1 --2- 172.21.15.67:0/2912855434 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a1c06cee0 0x7f9a1c15b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:36.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.394+0000 7f9a1bfff640 1 --2- 172.21.15.67:0/2912855434 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9a1c0d1550 0x7f9a1c160b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:36.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.394+0000 7f9a1bfff640 1 -- 172.21.15.67:0/2912855434 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a1c06cee0 msgr2=0x7f9a1c15b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:36.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.394+0000 7f9a1bfff640 1 --2- 172.21.15.67:0/2912855434 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a1c06cee0 0x7f9a1c15b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:36.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.394+0000 7f9a1bfff640 1 -- 172.21.15.67:0/2912855434 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a1c1f7a50 con 0x7f9a1c09a2b0 2024-03-24T00:40:36.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.394+0000 7f9a20ab7640 1 --2- 172.21.15.67:0/2912855434 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a1c06cee0 0x7f9a1c15b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:40:36.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.394+0000 7f9a212b8640 1 --2- 172.21.15.67:0/2912855434 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9a1c0d1550 0x7f9a1c160b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:40:36.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.395+0000 7f9a1bfff640 1 --2- 172.21.15.67:0/2912855434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a1c09a2b0 0x7f9a1c160650 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f9a100098f0 tx=0x7f9a100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:36.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.395+0000 7f9a197fa640 1 -- 172.21.15.67:0/2912855434 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a10041070 con 0x7f9a1c09a2b0 2024-03-24T00:40:36.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.395+0000 7f9a197fa640 1 -- 172.21.15.67:0/2912855434 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a1003b430 con 0x7f9a1c09a2b0 2024-03-24T00:40:36.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.395+0000 7f9a22d1b640 1 -- 172.21.15.67:0/2912855434 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a1c1f7d30 con 0x7f9a1c09a2b0 2024-03-24T00:40:36.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.395+0000 7f9a197fa640 1 -- 172.21.15.67:0/2912855434 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a10035040 con 0x7f9a1c09a2b0 2024-03-24T00:40:36.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.395+0000 7f9a22d1b640 1 -- 172.21.15.67:0/2912855434 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a1c1f82f0 con 0x7f9a1c09a2b0 2024-03-24T00:40:36.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.396+0000 7f9a22d1b640 1 -- 172.21.15.67:0/2912855434 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a1c1f84f0 con 0x7f9a1c09a2b0 2024-03-24T00:40:36.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.396+0000 7f9a197fa640 1 -- 172.21.15.67:0/2912855434 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9a10038020 con 0x7f9a1c09a2b0 2024-03-24T00:40:36.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.397+0000 7f9a197fa640 1 --2- 172.21.15.67:0/2912855434 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9a0806bbd0 0x7f9a0806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:36.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.397+0000 7f9a197fa640 1 -- 172.21.15.67:0/2912855434 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f9a100ac5b0 con 0x7f9a1c09a2b0 2024-03-24T00:40:36.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.397+0000 7f9a20ab7640 1 --2- 172.21.15.67:0/2912855434 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9a0806bbd0 0x7f9a0806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:36.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.397+0000 7f9a20ab7640 1 --2- 172.21.15.67:0/2912855434 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9a0806bbd0 0x7f9a0806e090 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f9a140012e0 tx=0x7f9a14009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:36.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.401+0000 7f9a197fa640 1 -- 172.21.15.67:0/2912855434 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9a1c1f84f0 con 0x7f9a1c09a2b0 2024-03-24T00:40:36.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.655+0000 7f9a22d1b640 1 -- 172.21.15.67:0/2912855434 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f9a1c04f400 con 0x7f9a1c09a2b0 2024-03-24T00:40:36.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.656+0000 7f9a197fa640 1 -- 172.21.15.67:0/2912855434 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v13) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f9a1007b260 con 0x7f9a1c09a2b0 2024-03-24T00:40:36.657 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:40:36.657 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:40:36.657 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_dump": [ 2024-03-24T00:40:36.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:40:36.657 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.a", 2024-03-24T00:40:36.657 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9l4UENDhAASKUDtqcbSQ7wADj7uo4eyQ==", 2024-03-24T00:40:36.658 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:40:36.658 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:40:36.658 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-03-24T00:40:36.658 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-03-24T00:40:36.658 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:40:36.658 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:40:36.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:40:36.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:40:36.658 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.b", 2024-03-24T00:40:36.658 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9luhJWGhAAsS6PFHV2XdF9htjIjzdrdA==", 2024-03-24T00:40:36.658 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:40:36.658 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:40:36.658 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-03-24T00:40:36.658 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-03-24T00:40:36.658 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:40:36.659 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:40:36.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:40:36.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:40:36.659 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.c", 2024-03-24T00:40:36.659 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9l8mRjFRAAQ73yexNddpcyp8ayTcRKCg==", 2024-03-24T00:40:36.659 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:40:36.659 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:40:36.659 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-03-24T00:40:36.659 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-03-24T00:40:36.659 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:40:36.659 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:40:36.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:40:36.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:40:36.659 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.d", 2024-03-24T00:40:36.659 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9lcJyvIRAA2tpDb6qxn1MtljjEno4qPA==", 2024-03-24T00:40:36.659 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:40:36.659 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:40:36.660 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-03-24T00:40:36.660 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-03-24T00:40:36.660 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:40:36.660 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:40:36.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:40:36.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:40:36.660 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.0", 2024-03-24T00:40:36.660 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDgdf9lV/NUBRAAyJNajFAnNABJErnBlB8TEA==", 2024-03-24T00:40:36.660 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:40:36.660 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:40:36.660 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:40:36.660 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:40:36.660 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:40:36.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:40:36.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:40:36.660 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.1", 2024-03-24T00:40:36.661 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDjdf9l0J6lAxAA9RInt/X22jqzthE0vOiiFg==", 2024-03-24T00:40:36.661 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:40:36.661 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:40:36.661 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:40:36.661 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:40:36.661 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:40:36.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:40:36.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:40:36.661 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.2", 2024-03-24T00:40:36.661 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDmdf9lSmrEARAAwmtSQRNePsn4WIZODnjhfQ==", 2024-03-24T00:40:36.661 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:40:36.661 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:40:36.661 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:40:36.661 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:40:36.661 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:40:36.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:40:36.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:40:36.662 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.3", 2024-03-24T00:40:36.662 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDpdf9lm59bABAAgJ5AV774lw2BgkSCgaweyA==", 2024-03-24T00:40:36.662 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:40:36.662 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:40:36.662 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:40:36.662 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:40:36.662 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:40:36.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:40:36.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:40:36.662 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.4", 2024-03-24T00:40:36.662 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDvdf9lCo+pCRAANHkL0pPxxok6IsBCLAFCbg==", 2024-03-24T00:40:36.662 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:40:36.663 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:40:36.663 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:40:36.663 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:40:36.663 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:40:36.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:40:36.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:40:36.663 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.5", 2024-03-24T00:40:36.663 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDydf9lyGEdCBAAcuszPsthGlLvVryE7+D9iQ==", 2024-03-24T00:40:36.663 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:40:36.663 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:40:36.663 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:40:36.663 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:40:36.663 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:40:36.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:40:36.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:40:36.664 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.6", 2024-03-24T00:40:36.664 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQD1df9ljuZqBhAACt2OPNkjKOD3rK7g2mVBBA==", 2024-03-24T00:40:36.664 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:40:36.664 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:40:36.664 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:40:36.664 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:40:36.664 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:40:36.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:40:36.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:40:36.664 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.7", 2024-03-24T00:40:36.664 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQD4df9lYKBfBRAAy/cWFF6pvGDjXZHAuBYw7g==", 2024-03-24T00:40:36.664 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:40:36.664 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:40:36.664 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:40:36.664 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:40:36.664 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:40:36.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:40:36.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:40:36.665 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.0", 2024-03-24T00:40:36.665 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9lY4nvJRAA6HWOZDWA+SSvJVmyjG+51g==", 2024-03-24T00:40:36.665 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:40:36.665 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:40:36.665 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow r", 2024-03-24T00:40:36.665 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-24T00:40:36.665 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:40:36.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:40:36.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:40:36.665 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.1", 2024-03-24T00:40:36.665 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9lonsXLxAAaltVMW3n0qHX8u9X8EkSmg==", 2024-03-24T00:40:36.665 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:40:36.665 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:40:36.665 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-03-24T00:40:36.665 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-03-24T00:40:36.665 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-03-24T00:40:36.666 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:40:36.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:40:36.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:40:36.666 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.2", 2024-03-24T00:40:36.666 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9lLEm2KxAAkT65mhLrTirlDnOUyvBY7g==", 2024-03-24T00:40:36.666 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:40:36.666 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:40:36.666 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-03-24T00:40:36.666 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-03-24T00:40:36.666 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-03-24T00:40:36.666 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:40:36.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:40:36.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:40:36.666 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.3", 2024-03-24T00:40:36.667 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9l8njZNBAAaAktZpX2SPcJR9ymPm9ebw==", 2024-03-24T00:40:36.667 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:40:36.667 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:40:36.667 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-03-24T00:40:36.667 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-03-24T00:40:36.667 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-03-24T00:40:36.667 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:40:36.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:40:36.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:40:36.667 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.admin", 2024-03-24T00:40:36.667 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDbdf9lDTIwLRAAMajqhM/KfUNF/SjVv9OJDw==", 2024-03-24T00:40:36.667 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:40:36.667 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-03-24T00:40:36.667 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-03-24T00:40:36.667 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-03-24T00:40:36.668 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:40:36.668 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:40:36.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:40:36.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:40:36.668 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.x", 2024-03-24T00:40:36.668 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9lGwKvChAAxl+fRnjbAQ7dYJ727QFUgw==", 2024-03-24T00:40:36.668 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:40:36.668 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-03-24T00:40:36.668 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-03-24T00:40:36.668 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:40:36.668 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:40:36.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:40:36.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:40:36.668 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.y", 2024-03-24T00:40:36.668 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9lSvufBhAAoQ9L1GjLkstjrAw2lqlryA==", 2024-03-24T00:40:36.669 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:40:36.669 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-03-24T00:40:36.669 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-03-24T00:40:36.669 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:40:36.669 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:40:36.669 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:40:36.669 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-24T00:40:36.669 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:40:36.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.658+0000 7f9a22d1b640 1 -- 172.21.15.67:0/2912855434 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9a0806bbd0 msgr2=0x7f9a0806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:36.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.658+0000 7f9a22d1b640 1 --2- 172.21.15.67:0/2912855434 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9a0806bbd0 0x7f9a0806e090 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f9a140012e0 tx=0x7f9a14009040 comp rx=0 tx=0).stop 2024-03-24T00:40:36.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.658+0000 7f9a22d1b640 1 -- 172.21.15.67:0/2912855434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a1c09a2b0 msgr2=0x7f9a1c160650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:36.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.658+0000 7f9a22d1b640 1 --2- 172.21.15.67:0/2912855434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a1c09a2b0 0x7f9a1c160650 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f9a100098f0 tx=0x7f9a100046c0 comp rx=0 tx=0).stop 2024-03-24T00:40:36.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.658+0000 7f9a22d1b640 1 -- 172.21.15.67:0/2912855434 shutdown_connections 2024-03-24T00:40:36.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.658+0000 7f9a22d1b640 1 --2- 172.21.15.67:0/2912855434 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9a0806bbd0 0x7f9a0806e090 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:36.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.658+0000 7f9a22d1b640 1 --2- 172.21.15.67:0/2912855434 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9a1c0d1550 0x7f9a1c160b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:36.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.658+0000 7f9a22d1b640 1 --2- 172.21.15.67:0/2912855434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a1c09a2b0 0x7f9a1c160650 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:36.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.658+0000 7f9a22d1b640 1 --2- 172.21.15.67:0/2912855434 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a1c06cee0 0x7f9a1c15b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:36.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.658+0000 7f9a22d1b640 1 -- 172.21.15.67:0/2912855434 >> 172.21.15.67:0/2912855434 conn(0x7f9a1c0b36e0 msgr2=0x7f9a1c06ffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:36.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.658+0000 7f9a22d1b640 1 -- 172.21.15.67:0/2912855434 shutdown_connections 2024-03-24T00:40:36.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.658+0000 7f9a22d1b640 1 -- 172.21.15.67:0/2912855434 wait complete. 2024-03-24T00:40:36.671 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:40:36.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.772+0000 7f4137485640 1 Processor -- start 2024-03-24T00:40:36.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.772+0000 7f4137485640 1 -- start start 2024-03-24T00:40:36.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.773+0000 7f4137485640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f41300cef40 0x7f41300cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:36.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.773+0000 7f4137485640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4130072890 con 0x7f41300ccce0 2024-03-24T00:40:36.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.773+0000 7f4137485640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f413015c580 con 0x7f413006dce0 2024-03-24T00:40:36.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.773+0000 7f4137485640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f413015c860 con 0x7f41300cef40 2024-03-24T00:40:36.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.773+0000 7f4135221640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f41300ccce0 0x7f41300cd0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52540/0 (socket says 172.21.15.67:52540) 2024-03-24T00:40:36.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.773+0000 7f4135221640 1 -- 172.21.15.67:0/4164674044 learned_addr learned my addr 172.21.15.67:0/4164674044 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:40:36.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.773+0000 7f4127fff640 1 -- 172.21.15.67:0/4164674044 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1754583227 0 0) 0x7f4130072890 con 0x7f41300ccce0 2024-03-24T00:40:36.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.773+0000 7f4127fff640 1 -- 172.21.15.67:0/4164674044 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f411c003520 con 0x7f41300ccce0 2024-03-24T00:40:36.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.773+0000 7f4134a20640 1 --2- 172.21.15.67:0/4164674044 >> v2:172.21.15.72:3300/0 conn(0x7f41300cef40 0x7f41300cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:36.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.773+0000 7f4127fff640 1 -- 172.21.15.67:0/4164674044 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1625035626 0 0) 0x7f411c003520 con 0x7f41300ccce0 2024-03-24T00:40:36.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.773+0000 7f4127fff640 1 -- 172.21.15.67:0/4164674044 >> v2:172.21.15.72:3300/0 conn(0x7f41300cef40 msgr2=0x7f41300cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:36.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.773+0000 7f4127fff640 1 --2- 172.21.15.67:0/4164674044 >> v2:172.21.15.72:3300/0 conn(0x7f41300cef40 0x7f41300cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:36.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.773+0000 7f4127fff640 1 -- 172.21.15.67:0/4164674044 >> v1:172.21.15.72:6789/0 conn(0x7f413006dce0 legacy=0x7f413006e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:36.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.773+0000 7f4127fff640 1 -- 172.21.15.67:0/4164674044 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f413015cb40 con 0x7f41300ccce0 2024-03-24T00:40:36.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.773+0000 7f4127fff640 1 -- 172.21.15.67:0/4164674044 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f412c001670 con 0x7f41300ccce0 2024-03-24T00:40:36.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.773+0000 7f4127fff640 1 -- 172.21.15.67:0/4164674044 >> v1:172.21.15.67:6789/0 conn(0x7f41300ccce0 legacy=0x7f41300cd0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:36.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.774+0000 7f4127fff640 1 --2- 172.21.15.67:0/4164674044 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f411c0039b0 0x7f411c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:36.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.774+0000 7f4127fff640 1 --2- 172.21.15.67:0/4164674044 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f411c004800 0x7f411c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:36.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.774+0000 7f4127fff640 1 --2- 172.21.15.67:0/4164674044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f411c0051c0 0x7f411c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:36.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.774+0000 7f4127fff640 1 -- 172.21.15.67:0/4164674044 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f413015c860 con 0x7f411c0051c0 2024-03-24T00:40:36.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.774+0000 7f4127fff640 1 -- 172.21.15.67:0/4164674044 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f411c00c110 con 0x7f411c004800 2024-03-24T00:40:36.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.774+0000 7f4127fff640 1 -- 172.21.15.67:0/4164674044 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f411c00c310 con 0x7f411c0039b0 2024-03-24T00:40:36.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.774+0000 7f4135a22640 1 --2- 172.21.15.67:0/4164674044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f411c0051c0 0x7f411c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:36.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.774+0000 7f4135221640 1 --2- 172.21.15.67:0/4164674044 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f411c0039b0 0x7f411c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:36.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.774+0000 7f4134a20640 1 --2- 172.21.15.67:0/4164674044 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f411c004800 0x7f411c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:36.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.774+0000 7f4135a22640 1 -- 172.21.15.67:0/4164674044 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f411c0039b0 msgr2=0x7f411c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:36.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.774+0000 7f4135a22640 1 --2- 172.21.15.67:0/4164674044 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f411c0039b0 0x7f411c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:36.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.774+0000 7f4135a22640 1 -- 172.21.15.67:0/4164674044 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f411c004800 msgr2=0x7f411c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:36.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.774+0000 7f4135a22640 1 --2- 172.21.15.67:0/4164674044 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f411c004800 0x7f411c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:36.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.774+0000 7f4135a22640 1 -- 172.21.15.67:0/4164674044 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f411c003520 con 0x7f411c0051c0 2024-03-24T00:40:36.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.775+0000 7f4135a22640 1 -- 172.21.15.67:0/4164674044 reap_dead start 2024-03-24T00:40:36.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.775+0000 7f4135a22640 1 --2- 172.21.15.67:0/4164674044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f411c0051c0 0x7f411c00b9c0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f4128001e50 tx=0x7f411c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c0c67e714c1ee41 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:36.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.775+0000 7f4127fff640 1 -- 172.21.15.67:0/4164674044 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4128001a70 con 0x7f411c0051c0 2024-03-24T00:40:36.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.775+0000 7f4127fff640 1 -- 172.21.15.67:0/4164674044 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4128003bb0 con 0x7f411c0051c0 2024-03-24T00:40:36.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.775+0000 7f4127fff640 1 -- 172.21.15.67:0/4164674044 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41280050d0 con 0x7f411c0051c0 2024-03-24T00:40:36.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.775+0000 7f4137485640 1 -- 172.21.15.67:0/4164674044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f411c0051c0 msgr2=0x7f411c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:36.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.775+0000 7f4137485640 1 --2- 172.21.15.67:0/4164674044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f411c0051c0 0x7f411c00b9c0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f4128001e50 tx=0x7f411c005830 comp rx=0 tx=0).stop 2024-03-24T00:40:36.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.775+0000 7f4137485640 1 -- 172.21.15.67:0/4164674044 shutdown_connections 2024-03-24T00:40:36.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.775+0000 7f4137485640 1 --2- 172.21.15.67:0/4164674044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f411c0051c0 0x7f411c00b9c0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:36.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.775+0000 7f4137485640 1 -- 172.21.15.67:0/4164674044 >> 172.21.15.67:0/4164674044 conn(0x7f41300cf9e0 msgr2=0x7f41300cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:36.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.775+0000 7f4137485640 1 -- 172.21.15.67:0/4164674044 shutdown_connections 2024-03-24T00:40:36.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.775+0000 7f4137485640 1 -- 172.21.15.67:0/4164674044 wait complete. 2024-03-24T00:40:36.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.777+0000 7f4137485640 1 Processor -- start 2024-03-24T00:40:36.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.777+0000 7f4137485640 1 -- start start 2024-03-24T00:40:36.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.777+0000 7f4137485640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f411c0051c0 0x7f41301f4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:36.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.777+0000 7f4137485640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f41300b2b10 0x7f41300cb340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:36.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.777+0000 7f4137485640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f41300c4980 0x7f41300c4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:36.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.777+0000 7f4137485640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4130145900 con 0x7f411c0051c0 2024-03-24T00:40:36.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.777+0000 7f4137485640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f413015c290 con 0x7f41300c4980 2024-03-24T00:40:36.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.777+0000 7f4137485640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41300cf3c0 con 0x7f41300b2b10 2024-03-24T00:40:36.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.777+0000 7f4135221640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f411c0051c0 0x7f41301f4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:36.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.777+0000 7f4135221640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f411c0051c0 0x7f41301f4f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35000/0 (socket says 172.21.15.67:35000) 2024-03-24T00:40:36.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.777+0000 7f4135221640 1 -- 172.21.15.67:0/3836797626 learned_addr learned my addr 172.21.15.67:0/3836797626 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:36.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.777+0000 7f4134a20640 1 --2- 172.21.15.67:0/3836797626 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f41300b2b10 0x7f41300cb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:36.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.778+0000 7f4135a22640 1 --2- 172.21.15.67:0/3836797626 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f41300c4980 0x7f41300c4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:36.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.778+0000 7f4135221640 1 -- 172.21.15.67:0/3836797626 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f41300b2b10 msgr2=0x7f41300cb340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:36.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.778+0000 7f4135221640 1 --2- 172.21.15.67:0/3836797626 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f41300b2b10 0x7f41300cb340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:36.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.778+0000 7f4135221640 1 -- 172.21.15.67:0/3836797626 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f41300c4980 msgr2=0x7f41300c4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:36.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.778+0000 7f4135221640 1 --2- 172.21.15.67:0/3836797626 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f41300c4980 0x7f41300c4e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:36.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.778+0000 7f4135221640 1 -- 172.21.15.67:0/3836797626 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41300b32d0 con 0x7f411c0051c0 2024-03-24T00:40:36.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.778+0000 7f4135221640 1 --2- 172.21.15.67:0/3836797626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f411c0051c0 0x7f41301f4f70 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f412c00e850 tx=0x7f412c00ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:36.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.778+0000 7f4125ffb640 1 -- 172.21.15.67:0/3836797626 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f412c00c870 con 0x7f411c0051c0 2024-03-24T00:40:36.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.778+0000 7f4125ffb640 1 -- 172.21.15.67:0/3836797626 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f412c018070 con 0x7f411c0051c0 2024-03-24T00:40:36.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.778+0000 7f4137485640 1 -- 172.21.15.67:0/3836797626 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41300b35b0 con 0x7f411c0051c0 2024-03-24T00:40:36.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.778+0000 7f4125ffb640 1 -- 172.21.15.67:0/3836797626 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f412c00cc50 con 0x7f411c0051c0 2024-03-24T00:40:36.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.778+0000 7f4137485640 1 -- 172.21.15.67:0/3836797626 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41300b3b70 con 0x7f411c0051c0 2024-03-24T00:40:36.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.779+0000 7f4137485640 1 -- 172.21.15.67:0/3836797626 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f413004f400 con 0x7f411c0051c0 2024-03-24T00:40:36.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.780+0000 7f4125ffb640 1 -- 172.21.15.67:0/3836797626 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f412c01d460 con 0x7f411c0051c0 2024-03-24T00:40:36.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.780+0000 7f4125ffb640 1 --2- 172.21.15.67:0/3836797626 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f411406bbd0 0x7f411406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:36.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.780+0000 7f4125ffb640 1 -- 172.21.15.67:0/3836797626 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f412c08a6d0 con 0x7f411c0051c0 2024-03-24T00:40:36.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.780+0000 7f4134a20640 1 --2- 172.21.15.67:0/3836797626 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f411406bbd0 0x7f411406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:36.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.781+0000 7f4134a20640 1 --2- 172.21.15.67:0/3836797626 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f411406bbd0 0x7f411406e090 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f41300b4cf0 tx=0x7f4120008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:36.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.784+0000 7f4125ffb640 1 -- 172.21.15.67:0/3836797626 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f412c058d30 con 0x7f411c0051c0 2024-03-24T00:40:36.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.967+0000 7f4137485640 1 -- 172.21.15.67:0/3836797626 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4130140fb0 con 0x7f411c0051c0 2024-03-24T00:40:36.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.968+0000 7f4125ffb640 1 -- 172.21.15.67:0/3836797626 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+13104 (secure 0 0 0) 0x7f412c05cdd0 con 0x7f411c0051c0 2024-03-24T00:40:36.969 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:40:36.969 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":46,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:40:36.277921+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:40:36.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.970+0000 7f4137485640 1 -- 172.21.15.67:0/3836797626 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f411406bbd0 msgr2=0x7f411406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:36.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.970+0000 7f4137485640 1 --2- 172.21.15.67:0/3836797626 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f411406bbd0 0x7f411406e090 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f41300b4cf0 tx=0x7f4120008040 comp rx=0 tx=0).stop 2024-03-24T00:40:36.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.970+0000 7f4137485640 1 -- 172.21.15.67:0/3836797626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f411c0051c0 msgr2=0x7f41301f4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:36.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.970+0000 7f4137485640 1 --2- 172.21.15.67:0/3836797626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f411c0051c0 0x7f41301f4f70 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f412c00e850 tx=0x7f412c00ed10 comp rx=0 tx=0).stop 2024-03-24T00:40:36.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.970+0000 7f4137485640 1 -- 172.21.15.67:0/3836797626 shutdown_connections 2024-03-24T00:40:36.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.970+0000 7f4137485640 1 --2- 172.21.15.67:0/3836797626 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f411406bbd0 0x7f411406e090 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:36.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.970+0000 7f4137485640 1 --2- 172.21.15.67:0/3836797626 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f41300c4980 0x7f41300c4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:36.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.970+0000 7f4137485640 1 --2- 172.21.15.67:0/3836797626 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f41300b2b10 0x7f41300cb340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:36.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.970+0000 7f4137485640 1 --2- 172.21.15.67:0/3836797626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f411c0051c0 0x7f41301f4f70 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:36.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.970+0000 7f4137485640 1 -- 172.21.15.67:0/3836797626 >> 172.21.15.67:0/3836797626 conn(0x7f41300cf9e0 msgr2=0x7f41300af840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:36.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.970+0000 7f4137485640 1 -- 172.21.15.67:0/3836797626 shutdown_connections 2024-03-24T00:40:36.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:36.970+0000 7f4137485640 1 -- 172.21.15.67:0/3836797626 wait complete. 2024-03-24T00:40:36.982 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:38:39.364950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:40:36.982 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:40:37.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.086+0000 7f565909a640 1 Processor -- start 2024-03-24T00:40:37.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.086+0000 7f565909a640 1 -- start start 2024-03-24T00:40:37.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.087+0000 7f565909a640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f56540d0d50 0x7f56540ce610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:37.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.087+0000 7f565909a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f56540d22b0 0x7f56540d2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:37.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.087+0000 7f565909a640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56540cf570 con 0x7f565415e130 2024-03-24T00:40:37.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.087+0000 7f565909a640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56540ceb50 con 0x7f56540d22b0 2024-03-24T00:40:37.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.087+0000 7f565909a640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56540cecf0 con 0x7f56540d0d50 2024-03-24T00:40:37.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.087+0000 7f565359e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f56540d22b0 0x7f56540d2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:37.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.087+0000 7f5652d9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f565415e130 0x7f56540d0640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52544/0 (socket says 172.21.15.67:52544) 2024-03-24T00:40:37.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.087+0000 7f5652d9d640 1 -- 172.21.15.67:0/3048342437 learned_addr learned my addr 172.21.15.67:0/3048342437 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:40:37.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.087+0000 7f565259c640 1 --2- 172.21.15.67:0/3048342437 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f56540d0d50 0x7f56540ce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:37.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.087+0000 7f565359e640 1 -- 172.21.15.67:0/3048342437 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f56540d0d50 msgr2=0x7f56540ce610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:37.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.087+0000 7f565359e640 1 --2- 172.21.15.67:0/3048342437 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f56540d0d50 0x7f56540ce610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:37.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.087+0000 7f565359e640 1 -- 172.21.15.67:0/3048342437 >> v1:172.21.15.67:6789/0 conn(0x7f565415e130 legacy=0x7f56540d0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:37.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.087+0000 7f565359e640 1 -- 172.21.15.67:0/3048342437 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5654162a60 con 0x7f56540d22b0 2024-03-24T00:40:37.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.087+0000 7f5651d9b640 1 -- 172.21.15.67:0/3048342437 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1076344744 0 0) 0x7f56540cf570 con 0x7f565415e130 2024-03-24T00:40:37.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.088+0000 7f565359e640 1 --2- 172.21.15.67:0/3048342437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56540d22b0 0x7f56540d2760 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f5644009d90 tx=0x7f564402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ba8934ac21c5fe4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:37.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.088+0000 7f5651d9b640 1 -- 172.21.15.67:0/3048342437 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5644039070 con 0x7f56540d22b0 2024-03-24T00:40:37.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.088+0000 7f5651d9b640 1 -- 172.21.15.67:0/3048342437 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f564402cc50 con 0x7f56540d22b0 2024-03-24T00:40:37.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.088+0000 7f5651d9b640 1 -- 172.21.15.67:0/3048342437 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5644034440 con 0x7f56540d22b0 2024-03-24T00:40:37.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.088+0000 7f565909a640 1 -- 172.21.15.67:0/3048342437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56540d22b0 msgr2=0x7f56540d2760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:37.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.088+0000 7f565909a640 1 --2- 172.21.15.67:0/3048342437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56540d22b0 0x7f56540d2760 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f5644009d90 tx=0x7f564402bea0 comp rx=0 tx=0).stop 2024-03-24T00:40:37.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.088+0000 7f565909a640 1 -- 172.21.15.67:0/3048342437 shutdown_connections 2024-03-24T00:40:37.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.088+0000 7f565909a640 1 --2- 172.21.15.67:0/3048342437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56540d22b0 0x7f56540d2760 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:37.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.088+0000 7f565909a640 1 --2- 172.21.15.67:0/3048342437 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f56540d0d50 0x7f56540ce610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:37.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.088+0000 7f565909a640 1 -- 172.21.15.67:0/3048342437 >> 172.21.15.67:0/3048342437 conn(0x7f565406c8a0 msgr2=0x7f565406ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:37.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.089+0000 7f565909a640 1 -- 172.21.15.67:0/3048342437 shutdown_connections 2024-03-24T00:40:37.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.089+0000 7f565909a640 1 -- 172.21.15.67:0/3048342437 wait complete. 2024-03-24T00:40:37.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.090+0000 7f565909a640 1 Processor -- start 2024-03-24T00:40:37.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.090+0000 7f565909a640 1 -- start start 2024-03-24T00:40:37.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.091+0000 7f565909a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f56540d0d50 0x7f56540aa990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:37.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.091+0000 7f565909a640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f56540d22b0 0x7f56541528c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:37.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.091+0000 7f565909a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f565415e130 0x7f5654152e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:37.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.091+0000 7f565909a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f565415e510 con 0x7f565415e130 2024-03-24T00:40:37.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.091+0000 7f565909a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56540cf290 con 0x7f56540d0d50 2024-03-24T00:40:37.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.091+0000 7f565909a640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56540b2e10 con 0x7f56540d22b0 2024-03-24T00:40:37.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.091+0000 7f565359e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f565415e130 0x7f5654152e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:37.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.091+0000 7f565359e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f565415e130 0x7f5654152e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35020/0 (socket says 172.21.15.67:35020) 2024-03-24T00:40:37.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.091+0000 7f565359e640 1 -- 172.21.15.67:0/532053009 learned_addr learned my addr 172.21.15.67:0/532053009 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:37.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.091+0000 7f5652d9d640 1 --2- 172.21.15.67:0/532053009 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f56540d0d50 0x7f56540aa990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:37.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.091+0000 7f565359e640 1 -- 172.21.15.67:0/532053009 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f56540d22b0 msgr2=0x7f56541528c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:37.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.091+0000 7f565259c640 1 --2- 172.21.15.67:0/532053009 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f56540d22b0 0x7f56541528c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:37.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.091+0000 7f565359e640 1 --2- 172.21.15.67:0/532053009 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f56540d22b0 0x7f56541528c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:37.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.091+0000 7f565359e640 1 -- 172.21.15.67:0/532053009 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f56540d0d50 msgr2=0x7f56540aa990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:37.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.091+0000 7f565359e640 1 --2- 172.21.15.67:0/532053009 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f56540d0d50 0x7f56540aa990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:37.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.091+0000 7f565359e640 1 -- 172.21.15.67:0/532053009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56540cb980 con 0x7f565415e130 2024-03-24T00:40:37.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.091+0000 7f5652d9d640 1 --2- 172.21.15.67:0/532053009 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f56540d0d50 0x7f56540aa990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:40:37.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.091+0000 7f565359e640 1 --2- 172.21.15.67:0/532053009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f565415e130 0x7f5654152e00 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f5644009d90 tx=0x7f5644004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:37.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.091+0000 7f564b7fe640 1 -- 172.21.15.67:0/532053009 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5644039050 con 0x7f565415e130 2024-03-24T00:40:37.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.091+0000 7f564b7fe640 1 -- 172.21.15.67:0/532053009 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5644045070 con 0x7f565415e130 2024-03-24T00:40:37.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.092+0000 7f565909a640 1 -- 172.21.15.67:0/532053009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56540cbc60 con 0x7f565415e130 2024-03-24T00:40:37.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.092+0000 7f564b7fe640 1 -- 172.21.15.67:0/532053009 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f564403e610 con 0x7f565415e130 2024-03-24T00:40:37.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.092+0000 7f565909a640 1 -- 172.21.15.67:0/532053009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56540cc220 con 0x7f565415e130 2024-03-24T00:40:37.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.092+0000 7f565909a640 1 -- 172.21.15.67:0/532053009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56540689b0 con 0x7f565415e130 2024-03-24T00:40:37.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.093+0000 7f564b7fe640 1 -- 172.21.15.67:0/532053009 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f564403e7b0 con 0x7f565415e130 2024-03-24T00:40:37.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.093+0000 7f564b7fe640 1 --2- 172.21.15.67:0/532053009 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f563406bbd0 0x7f563406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:37.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.093+0000 7f564b7fe640 1 -- 172.21.15.67:0/532053009 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f56440ad3c0 con 0x7f565415e130 2024-03-24T00:40:37.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.093+0000 7f5652d9d640 1 --2- 172.21.15.67:0/532053009 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f563406bbd0 0x7f563406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:37.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.094+0000 7f5652d9d640 1 --2- 172.21.15.67:0/532053009 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f563406bbd0 0x7f563406e090 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f564c000f60 tx=0x7f564c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:37.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.098+0000 7f564b7fe640 1 -- 172.21.15.67:0/532053009 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f56540689b0 con 0x7f565415e130 2024-03-24T00:40:37.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.280+0000 7f565909a640 1 -- 172.21.15.67:0/532053009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f565406d1b0 con 0x7f565415e130 2024-03-24T00:40:37.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.280+0000 7f564b7fe640 1 -- 172.21.15.67:0/532053009 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v46) v1 ==== 93+0+10 (secure 0 0 0) 0x7f565406d1b0 con 0x7f565415e130 2024-03-24T00:40:37.281 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-24T00:40:37.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.282+0000 7f565909a640 1 -- 172.21.15.67:0/532053009 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f563406bbd0 msgr2=0x7f563406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:37.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.282+0000 7f565909a640 1 --2- 172.21.15.67:0/532053009 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f563406bbd0 0x7f563406e090 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f564c000f60 tx=0x7f564c009040 comp rx=0 tx=0).stop 2024-03-24T00:40:37.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.282+0000 7f565909a640 1 -- 172.21.15.67:0/532053009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f565415e130 msgr2=0x7f5654152e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:37.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.282+0000 7f565909a640 1 --2- 172.21.15.67:0/532053009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f565415e130 0x7f5654152e00 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f5644009d90 tx=0x7f5644004a40 comp rx=0 tx=0).stop 2024-03-24T00:40:37.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.282+0000 7f565909a640 1 -- 172.21.15.67:0/532053009 shutdown_connections 2024-03-24T00:40:37.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.282+0000 7f565909a640 1 --2- 172.21.15.67:0/532053009 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f563406bbd0 0x7f563406e090 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:37.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.282+0000 7f565909a640 1 --2- 172.21.15.67:0/532053009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f565415e130 0x7f5654152e00 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:37.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.282+0000 7f565909a640 1 --2- 172.21.15.67:0/532053009 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f56540d22b0 0x7f56541528c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:37.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.282+0000 7f565909a640 1 --2- 172.21.15.67:0/532053009 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f56540d0d50 0x7f56540aa990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:37.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.282+0000 7f565909a640 1 -- 172.21.15.67:0/532053009 >> 172.21.15.67:0/532053009 conn(0x7f565406c8a0 msgr2=0x7f56540d2690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:37.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.283+0000 7f565909a640 1 -- 172.21.15.67:0/532053009 shutdown_connections 2024-03-24T00:40:37.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.283+0000 7f565909a640 1 -- 172.21.15.67:0/532053009 wait complete. 2024-03-24T00:40:37.294 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-24T00:40:37.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.395+0000 7fcdef7df640 1 Processor -- start 2024-03-24T00:40:37.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.395+0000 7fcdef7df640 1 -- start start 2024-03-24T00:40:37.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.396+0000 7fcdef7df640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcde815e310 0x7fcde8162610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:37.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.396+0000 7fcdef7df640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcde80ce920 con 0x7fcde80d0d50 2024-03-24T00:40:37.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.396+0000 7fcdef7df640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcde80cef20 con 0x7fcde815df30 2024-03-24T00:40:37.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.396+0000 7fcdef7df640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcde8162b50 con 0x7fcde815e310 2024-03-24T00:40:37.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.396+0000 7fcdecd7a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcde815e310 0x7fcde8162610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:37.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.396+0000 7fcdedd7c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fcde80d0d50 0x7fcde80d2490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52552/0 (socket says 172.21.15.67:52552) 2024-03-24T00:40:37.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.396+0000 7fcdedd7c640 1 -- 172.21.15.67:0/4015490920 learned_addr learned my addr 172.21.15.67:0/4015490920 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:40:37.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.396+0000 7fcdecd7a640 1 -- 172.21.15.67:0/4015490920 >> v1:172.21.15.72:6789/0 conn(0x7fcde815df30 legacy=0x7fcde80d0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:37.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.396+0000 7fcdecd7a640 1 -- 172.21.15.67:0/4015490920 >> v1:172.21.15.67:6789/0 conn(0x7fcde80d0d50 legacy=0x7fcde80d2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:37.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.397+0000 7fcdecd7a640 1 -- 172.21.15.67:0/4015490920 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcde80cf200 con 0x7fcde815e310 2024-03-24T00:40:37.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.397+0000 7fcde7fff640 1 -- 172.21.15.67:0/4015490920 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2565029387 0 0) 0x7fcde80ce920 con 0x7fcde80d0d50 2024-03-24T00:40:37.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.397+0000 7fcdecd7a640 1 --2- 172.21.15.67:0/4015490920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcde815e310 0x7fcde8162610 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fcddc0097a0 tx=0x7fcddc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9c28e3e610a7903 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:37.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.397+0000 7fcde7fff640 1 -- 172.21.15.67:0/4015490920 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcddc035020 con 0x7fcde815e310 2024-03-24T00:40:37.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.397+0000 7fcde7fff640 1 -- 172.21.15.67:0/4015490920 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcddc031830 con 0x7fcde815e310 2024-03-24T00:40:37.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.397+0000 7fcde7fff640 1 -- 172.21.15.67:0/4015490920 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcddc031c00 con 0x7fcde815e310 2024-03-24T00:40:37.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.398+0000 7fcdef7df640 1 -- 172.21.15.67:0/4015490920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcde815e310 msgr2=0x7fcde8162610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:37.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.398+0000 7fcdef7df640 1 --2- 172.21.15.67:0/4015490920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcde815e310 0x7fcde8162610 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fcddc0097a0 tx=0x7fcddc02b8d0 comp rx=0 tx=0).stop 2024-03-24T00:40:37.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.398+0000 7fcdef7df640 1 -- 172.21.15.67:0/4015490920 shutdown_connections 2024-03-24T00:40:37.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.398+0000 7fcdef7df640 1 --2- 172.21.15.67:0/4015490920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcde815e310 0x7fcde8162610 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:37.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.398+0000 7fcdef7df640 1 -- 172.21.15.67:0/4015490920 >> 172.21.15.67:0/4015490920 conn(0x7fcde806c880 msgr2=0x7fcde806cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:37.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.398+0000 7fcdef7df640 1 -- 172.21.15.67:0/4015490920 shutdown_connections 2024-03-24T00:40:37.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.398+0000 7fcdef7df640 1 -- 172.21.15.67:0/4015490920 wait complete. 2024-03-24T00:40:37.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.399+0000 7fcdef7df640 1 Processor -- start 2024-03-24T00:40:37.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.399+0000 7fcdef7df640 1 -- start start 2024-03-24T00:40:37.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.400+0000 7fcdef7df640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcde80d0d50 0x7fcde815f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:37.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.400+0000 7fcdef7df640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcde815df30 0x7fcde815ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:37.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.400+0000 7fcdef7df640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcde80b6040 0x7fcde80cd350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:37.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.400+0000 7fcdef7df640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde80ce580 con 0x7fcde815df30 2024-03-24T00:40:37.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.400+0000 7fcdef7df640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde80b32f0 con 0x7fcde80d0d50 2024-03-24T00:40:37.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.400+0000 7fcdef7df640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcde80ce700 con 0x7fcde80b6040 2024-03-24T00:40:37.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.400+0000 7fcdecd7a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcde815df30 0x7fcde815ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:37.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.400+0000 7fcdecd7a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcde815df30 0x7fcde815ff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35042/0 (socket says 172.21.15.67:35042) 2024-03-24T00:40:37.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.400+0000 7fcdecd7a640 1 -- 172.21.15.67:0/2766270839 learned_addr learned my addr 172.21.15.67:0/2766270839 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:37.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.400+0000 7fcdedd7c640 1 --2- 172.21.15.67:0/2766270839 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcde80b6040 0x7fcde80cd350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:37.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.400+0000 7fcded57b640 1 --2- 172.21.15.67:0/2766270839 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcde80d0d50 0x7fcde815f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:37.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.400+0000 7fcdecd7a640 1 -- 172.21.15.67:0/2766270839 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcde80b6040 msgr2=0x7fcde80cd350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:37.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.400+0000 7fcdecd7a640 1 --2- 172.21.15.67:0/2766270839 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcde80b6040 0x7fcde80cd350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:37.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.400+0000 7fcdecd7a640 1 -- 172.21.15.67:0/2766270839 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcde80d0d50 msgr2=0x7fcde815f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:37.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.400+0000 7fcdecd7a640 1 --2- 172.21.15.67:0/2766270839 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcde80d0d50 0x7fcde815f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:37.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.400+0000 7fcdecd7a640 1 -- 172.21.15.67:0/2766270839 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcde80cc5d0 con 0x7fcde815df30 2024-03-24T00:40:37.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.400+0000 7fcded57b640 1 --2- 172.21.15.67:0/2766270839 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcde80d0d50 0x7fcde815f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:40:37.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.400+0000 7fcdecd7a640 1 --2- 172.21.15.67:0/2766270839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcde815df30 0x7fcde815ff00 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fcddc0097a0 tx=0x7fcddc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:37.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.401+0000 7fcde5ffb640 1 -- 172.21.15.67:0/2766270839 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcddc004230 con 0x7fcde815df30 2024-03-24T00:40:37.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.401+0000 7fcde5ffb640 1 -- 172.21.15.67:0/2766270839 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcddc0043d0 con 0x7fcde815df30 2024-03-24T00:40:37.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.401+0000 7fcde5ffb640 1 -- 172.21.15.67:0/2766270839 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcddc003850 con 0x7fcde815df30 2024-03-24T00:40:37.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.401+0000 7fcdef7df640 1 -- 172.21.15.67:0/2766270839 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcde80b64c0 con 0x7fcde815df30 2024-03-24T00:40:37.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.401+0000 7fcdef7df640 1 -- 172.21.15.67:0/2766270839 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcde80ccb50 con 0x7fcde815df30 2024-03-24T00:40:37.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.402+0000 7fcdef7df640 1 -- 172.21.15.67:0/2766270839 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcde804f400 con 0x7fcde815df30 2024-03-24T00:40:37.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.402+0000 7fcde5ffb640 1 -- 172.21.15.67:0/2766270839 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fcddc04c060 con 0x7fcde815df30 2024-03-24T00:40:37.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.403+0000 7fcde5ffb640 1 --2- 172.21.15.67:0/2766270839 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcdd406bbd0 0x7fcdd406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:37.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.403+0000 7fcde5ffb640 1 -- 172.21.15.67:0/2766270839 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fcddc0ac420 con 0x7fcde815df30 2024-03-24T00:40:37.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.403+0000 7fcded57b640 1 --2- 172.21.15.67:0/2766270839 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcdd406bbd0 0x7fcdd406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:37.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.403+0000 7fcded57b640 1 --2- 172.21.15.67:0/2766270839 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcdd406bbd0 0x7fcdd406e090 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fcde0000f60 tx=0x7fcde0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:37.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.407+0000 7fcde5ffb640 1 -- 172.21.15.67:0/2766270839 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fcddc040070 con 0x7fcde815df30 2024-03-24T00:40:37.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.592+0000 7fcdef7df640 1 -- 172.21.15.67:0/2766270839 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fcde80ca360 con 0x7fcde815df30 2024-03-24T00:40:37.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.592+0000 7fcde5ffb640 1 -- 172.21.15.67:0/2766270839 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v46) v1 ==== 84+0+65 (secure 0 0 0) 0x7fcddc07b160 con 0x7fcde815df30 2024-03-24T00:40:37.593 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:40:37.593 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-03-24T00:40:37.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:40:37.593 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2024-03-24T00:40:37.593 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2024-03-24T00:40:37.593 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:40:37.593 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-03-24T00:40:37.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.594+0000 7fcdef7df640 1 -- 172.21.15.67:0/2766270839 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcdd406bbd0 msgr2=0x7fcdd406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:37.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.594+0000 7fcdef7df640 1 --2- 172.21.15.67:0/2766270839 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcdd406bbd0 0x7fcdd406e090 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fcde0000f60 tx=0x7fcde0009040 comp rx=0 tx=0).stop 2024-03-24T00:40:37.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.594+0000 7fcdef7df640 1 -- 172.21.15.67:0/2766270839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcde815df30 msgr2=0x7fcde815ff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:37.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.594+0000 7fcdef7df640 1 --2- 172.21.15.67:0/2766270839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcde815df30 0x7fcde815ff00 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fcddc0097a0 tx=0x7fcddc0046c0 comp rx=0 tx=0).stop 2024-03-24T00:40:37.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.594+0000 7fcdef7df640 1 -- 172.21.15.67:0/2766270839 shutdown_connections 2024-03-24T00:40:37.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.594+0000 7fcdef7df640 1 --2- 172.21.15.67:0/2766270839 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcdd406bbd0 0x7fcdd406e090 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:37.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.594+0000 7fcdef7df640 1 --2- 172.21.15.67:0/2766270839 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcde80b6040 0x7fcde80cd350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:37.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.594+0000 7fcdef7df640 1 --2- 172.21.15.67:0/2766270839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcde815df30 0x7fcde815ff00 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:37.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.594+0000 7fcdef7df640 1 --2- 172.21.15.67:0/2766270839 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcde80d0d50 0x7fcde815f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:37.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.594+0000 7fcdef7df640 1 -- 172.21.15.67:0/2766270839 >> 172.21.15.67:0/2766270839 conn(0x7fcde806c880 msgr2=0x7fcde80c8580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:37.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.594+0000 7fcdef7df640 1 -- 172.21.15.67:0/2766270839 shutdown_connections 2024-03-24T00:40:37.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.594+0000 7fcdef7df640 1 -- 172.21.15.67:0/2766270839 wait complete. 2024-03-24T00:40:37.606 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-24T00:40:37.606 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-24T00:40:37.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.711+0000 7f5eb0836640 1 Processor -- start 2024-03-24T00:40:37.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.711+0000 7f5eb0836640 1 -- start start 2024-03-24T00:40:37.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.712+0000 7f5eb0836640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5eac09adb0 0x7f5eac0cd6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:37.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.712+0000 7f5eb0836640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5eac15cd40 0x7f5eac0cf7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:37.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.712+0000 7f5eb0836640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eac15d1b0 con 0x7f5eac099d70 2024-03-24T00:40:37.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.712+0000 7f5eb0836640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5eac0cc6e0 con 0x7f5eac15cd40 2024-03-24T00:40:37.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.712+0000 7f5eb0836640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5eac15d470 con 0x7f5eac09adb0 2024-03-24T00:40:37.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.712+0000 7f5eaad9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5eac15cd40 0x7f5eac0cf7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:37.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.712+0000 7f5eaa59c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5eac099d70 0x7f5eac09a150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52558/0 (socket says 172.21.15.67:52558) 2024-03-24T00:40:37.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.712+0000 7f5eaad9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5eac15cd40 0x7f5eac0cf7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35044/0 (socket says 172.21.15.67:35044) 2024-03-24T00:40:37.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.712+0000 7f5eaa59c640 1 -- 172.21.15.67:0/2576893371 learned_addr learned my addr 172.21.15.67:0/2576893371 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:40:37.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.712+0000 7f5ea3fff640 1 --2- 172.21.15.67:0/2576893371 >> v2:172.21.15.72:3300/0 conn(0x7f5eac09adb0 0x7f5eac0cd6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:37.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.712+0000 7f5eaad9d640 1 -- 172.21.15.67:0/2576893371 >> v2:172.21.15.72:3300/0 conn(0x7f5eac09adb0 msgr2=0x7f5eac0cd6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:37.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.712+0000 7f5eaad9d640 1 --2- 172.21.15.67:0/2576893371 >> v2:172.21.15.72:3300/0 conn(0x7f5eac09adb0 0x7f5eac0cd6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:37.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.712+0000 7f5eaad9d640 1 -- 172.21.15.67:0/2576893371 >> v1:172.21.15.67:6789/0 conn(0x7f5eac099d70 legacy=0x7f5eac09a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:37.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.712+0000 7f5eaad9d640 1 -- 172.21.15.67:0/2576893371 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eac0cc910 con 0x7f5eac15cd40 2024-03-24T00:40:37.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.712+0000 7f5ea9d9b640 1 -- 172.21.15.67:0/2576893371 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1051238295 0 0) 0x7f5eac15d1b0 con 0x7f5eac099d70 2024-03-24T00:40:37.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.713+0000 7f5eaad9d640 1 --2- 172.21.15.67:0/2576893371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5eac15cd40 0x7f5eac0cf7a0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f5eac08b2d0 tx=0x7f5ea402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab3298b960c9ceeb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:37.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.713+0000 7f5ea9d9b640 1 -- 172.21.15.67:0/2576893371 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea4039070 con 0x7f5eac15cd40 2024-03-24T00:40:37.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.713+0000 7f5ea9d9b640 1 -- 172.21.15.67:0/2576893371 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ea402cc60 con 0x7f5eac15cd40 2024-03-24T00:40:37.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.713+0000 7f5ea9d9b640 1 -- 172.21.15.67:0/2576893371 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea4034660 con 0x7f5eac15cd40 2024-03-24T00:40:37.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.714+0000 7f5eb0836640 1 -- 172.21.15.67:0/2576893371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5eac15cd40 msgr2=0x7f5eac0cf7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:37.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.714+0000 7f5eb0836640 1 --2- 172.21.15.67:0/2576893371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5eac15cd40 0x7f5eac0cf7a0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f5eac08b2d0 tx=0x7f5ea402beb0 comp rx=0 tx=0).stop 2024-03-24T00:40:37.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.714+0000 7f5eb0836640 1 -- 172.21.15.67:0/2576893371 shutdown_connections 2024-03-24T00:40:37.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.714+0000 7f5eb0836640 1 --2- 172.21.15.67:0/2576893371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5eac15cd40 0x7f5eac0cf7a0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:37.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.714+0000 7f5eb0836640 1 --2- 172.21.15.67:0/2576893371 >> v2:172.21.15.72:3300/0 conn(0x7f5eac09adb0 0x7f5eac0cd6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:37.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.714+0000 7f5eb0836640 1 -- 172.21.15.67:0/2576893371 >> 172.21.15.67:0/2576893371 conn(0x7f5eac0b36c0 msgr2=0x7f5eac0b3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:37.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.714+0000 7f5eb0836640 1 -- 172.21.15.67:0/2576893371 shutdown_connections 2024-03-24T00:40:37.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.714+0000 7f5eb0836640 1 -- 172.21.15.67:0/2576893371 wait complete. 2024-03-24T00:40:37.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.715+0000 7f5eb0836640 1 Processor -- start 2024-03-24T00:40:37.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.715+0000 7f5eb0836640 1 -- start start 2024-03-24T00:40:37.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.716+0000 7f5eb0836640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5eac099d70 0x7f5eac157d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:37.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.716+0000 7f5eb0836640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5eac09adb0 0x7f5eac1e7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:37.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.716+0000 7f5eb0836640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5eac15cd40 0x7f5eac0c9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:37.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.716+0000 7f5eb0836640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eac0cd1b0 con 0x7f5eac09adb0 2024-03-24T00:40:37.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.716+0000 7f5eb0836640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eac0cced0 con 0x7f5eac099d70 2024-03-24T00:40:37.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.716+0000 7f5eb0836640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5eac0cfce0 con 0x7f5eac15cd40 2024-03-24T00:40:37.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.716+0000 7f5ea3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5eac09adb0 0x7f5eac1e7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:37.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.716+0000 7f5ea3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5eac09adb0 0x7f5eac1e7bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35048/0 (socket says 172.21.15.67:35048) 2024-03-24T00:40:37.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.716+0000 7f5ea3fff640 1 -- 172.21.15.67:0/645634673 learned_addr learned my addr 172.21.15.67:0/645634673 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:37.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.716+0000 7f5eaa59c640 1 --2- 172.21.15.67:0/645634673 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5eac099d70 0x7f5eac157d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:37.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.716+0000 7f5ea3fff640 1 -- 172.21.15.67:0/645634673 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5eac15cd40 msgr2=0x7f5eac0c9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:37.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.716+0000 7f5ea3fff640 1 --2- 172.21.15.67:0/645634673 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5eac15cd40 0x7f5eac0c9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:37.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.716+0000 7f5ea3fff640 1 -- 172.21.15.67:0/645634673 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5eac099d70 msgr2=0x7f5eac157d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:37.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.716+0000 7f5ea3fff640 1 --2- 172.21.15.67:0/645634673 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5eac099d70 0x7f5eac157d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:37.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.716+0000 7f5ea3fff640 1 -- 172.21.15.67:0/645634673 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eac158250 con 0x7f5eac09adb0 2024-03-24T00:40:37.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.716+0000 7f5ea3fff640 1 --2- 172.21.15.67:0/645634673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5eac09adb0 0x7f5eac1e7bd0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f5e9400e9b0 tx=0x7f5e9400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:37.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.716+0000 7f5ea2ffd640 1 -- 172.21.15.67:0/645634673 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e94017070 con 0x7f5eac09adb0 2024-03-24T00:40:37.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.716+0000 7f5ea2ffd640 1 -- 172.21.15.67:0/645634673 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e9400c9a0 con 0x7f5eac09adb0 2024-03-24T00:40:37.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.716+0000 7f5eb0836640 1 -- 172.21.15.67:0/645634673 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5eac1e6850 con 0x7f5eac09adb0 2024-03-24T00:40:37.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.716+0000 7f5ea2ffd640 1 -- 172.21.15.67:0/645634673 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e9400cd90 con 0x7f5eac09adb0 2024-03-24T00:40:37.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.716+0000 7f5eb0836640 1 -- 172.21.15.67:0/645634673 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5eac1e6e10 con 0x7f5eac09adb0 2024-03-24T00:40:37.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.717+0000 7f5eb0836640 1 -- 172.21.15.67:0/645634673 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5eac067970 con 0x7f5eac09adb0 2024-03-24T00:40:37.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.718+0000 7f5ea2ffd640 1 -- 172.21.15.67:0/645634673 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5e9401a070 con 0x7f5eac09adb0 2024-03-24T00:40:37.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.718+0000 7f5ea2ffd640 1 --2- 172.21.15.67:0/645634673 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5e8c06bc00 0x7f5e8c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:37.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.718+0000 7f5ea2ffd640 1 -- 172.21.15.67:0/645634673 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f5e9408a8d0 con 0x7f5eac09adb0 2024-03-24T00:40:37.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.718+0000 7f5eaa59c640 1 --2- 172.21.15.67:0/645634673 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5e8c06bc00 0x7f5e8c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:37.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.719+0000 7f5eaa59c640 1 --2- 172.21.15.67:0/645634673 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5e8c06bc00 0x7f5e8c06e0c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c000f60 tx=0x7f5e9c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:37.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.723+0000 7f5ea2ffd640 1 -- 172.21.15.67:0/645634673 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5eac067970 con 0x7f5eac09adb0 2024-03-24T00:40:37.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:37.907+0000 7f5eb0836640 1 -- 172.21.15.67:0/645634673 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f5eac070050 con 0x7f5eac09adb0 2024-03-24T00:40:38.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.276+0000 7f5ea2ffd640 1 -- 172.21.15.67:0/645634673 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v47) v1 ==== 138+0+0 (secure 0 0 0) 0x7f5e94059580 con 0x7f5eac09adb0 2024-03-24T00:40:38.277 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' created 2024-03-24T00:40:38.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.279+0000 7f5eb0836640 1 -- 172.21.15.67:0/645634673 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5e8c06bc00 msgr2=0x7f5e8c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:38.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.279+0000 7f5eb0836640 1 --2- 172.21.15.67:0/645634673 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5e8c06bc00 0x7f5e8c06e0c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c000f60 tx=0x7f5e9c009040 comp rx=0 tx=0).stop 2024-03-24T00:40:38.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.279+0000 7f5eb0836640 1 -- 172.21.15.67:0/645634673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5eac09adb0 msgr2=0x7f5eac1e7bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:38.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.279+0000 7f5eb0836640 1 --2- 172.21.15.67:0/645634673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5eac09adb0 0x7f5eac1e7bd0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f5e9400e9b0 tx=0x7f5e9400ee70 comp rx=0 tx=0).stop 2024-03-24T00:40:38.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.279+0000 7f5eb0836640 1 -- 172.21.15.67:0/645634673 shutdown_connections 2024-03-24T00:40:38.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.279+0000 7f5eb0836640 1 --2- 172.21.15.67:0/645634673 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5e8c06bc00 0x7f5e8c06e0c0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:38.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.279+0000 7f5eb0836640 1 --2- 172.21.15.67:0/645634673 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5eac15cd40 0x7f5eac0c9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:38.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.279+0000 7f5eb0836640 1 --2- 172.21.15.67:0/645634673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5eac09adb0 0x7f5eac1e7bd0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:38.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.279+0000 7f5eb0836640 1 --2- 172.21.15.67:0/645634673 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5eac099d70 0x7f5eac157d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:38.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.279+0000 7f5eb0836640 1 -- 172.21.15.67:0/645634673 >> 172.21.15.67:0/645634673 conn(0x7f5eac0b36c0 msgr2=0x7f5eac0d00a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:38.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.279+0000 7f5eb0836640 1 -- 172.21.15.67:0/645634673 shutdown_connections 2024-03-24T00:40:38.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.279+0000 7f5eb0836640 1 -- 172.21.15.67:0/645634673 wait complete. 2024-03-24T00:40:38.294 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-24T00:40:38.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.402+0000 7fe3ffb77640 1 Processor -- start 2024-03-24T00:40:38.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.402+0000 7fe3ffb77640 1 -- start start 2024-03-24T00:40:38.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.403+0000 7fe3ffb77640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe3f815e040 0x7fe3f806cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:38.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.403+0000 7fe3ffb77640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3f815f170 con 0x7fe3f815eb30 2024-03-24T00:40:38.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.403+0000 7fe3ffb77640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3f815f450 con 0x7fe3f806d0a0 2024-03-24T00:40:38.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.403+0000 7fe3ffb77640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3f806d4f0 con 0x7fe3f815e040 2024-03-24T00:40:38.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.403+0000 7fe3fe114640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe3f815eb30 0x7fe3f80d07f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52566/0 (socket says 172.21.15.67:52566) 2024-03-24T00:40:38.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.403+0000 7fe3fe114640 1 -- 172.21.15.67:0/4169025225 learned_addr learned my addr 172.21.15.67:0/4169025225 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:40:38.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.403+0000 7fe3fd913640 1 --2- 172.21.15.67:0/4169025225 >> v2:172.21.15.72:3300/0 conn(0x7fe3f815e040 0x7fe3f806cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:38.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.404+0000 7fe3fc911640 1 -- 172.21.15.67:0/4169025225 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 314468788 0 0) 0x7fe3f815f170 con 0x7fe3f815eb30 2024-03-24T00:40:38.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.404+0000 7fe3fc911640 1 -- 172.21.15.67:0/4169025225 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3e0003520 con 0x7fe3f815eb30 2024-03-24T00:40:38.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.404+0000 7fe3fc911640 1 -- 172.21.15.67:0/4169025225 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2160339676 0 0) 0x7fe3e0003520 con 0x7fe3f815eb30 2024-03-24T00:40:38.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.404+0000 7fe3fc911640 1 -- 172.21.15.67:0/4169025225 >> v2:172.21.15.72:3300/0 conn(0x7fe3f815e040 msgr2=0x7fe3f806cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:38.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.404+0000 7fe3fc911640 1 --2- 172.21.15.67:0/4169025225 >> v2:172.21.15.72:3300/0 conn(0x7fe3f815e040 0x7fe3f806cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:38.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.404+0000 7fe3fc911640 1 -- 172.21.15.67:0/4169025225 >> v1:172.21.15.72:6789/0 conn(0x7fe3f806d0a0 legacy=0x7fe3f815e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:38.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.404+0000 7fe3fc911640 1 -- 172.21.15.67:0/4169025225 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3f815f730 con 0x7fe3f815eb30 2024-03-24T00:40:38.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.404+0000 7fe3fc911640 1 -- 172.21.15.67:0/4169025225 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7fe3f0001670 con 0x7fe3f815eb30 2024-03-24T00:40:38.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.404+0000 7fe3fd913640 1 --2- 172.21.15.67:0/4169025225 >> v2:172.21.15.72:3300/0 conn(0x7fe3f815e040 0x7fe3f806cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:40:38.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.404+0000 7fe3fc911640 1 -- 172.21.15.67:0/4169025225 >> v1:172.21.15.67:6789/0 conn(0x7fe3f815eb30 legacy=0x7fe3f80d07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:38.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.404+0000 7fe3fc911640 1 --2- 172.21.15.67:0/4169025225 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe3e00039b0 0x7fe3e0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:38.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.404+0000 7fe3fc911640 1 --2- 172.21.15.67:0/4169025225 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3e0004800 0x7fe3e0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:38.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.404+0000 7fe3fc911640 1 --2- 172.21.15.67:0/4169025225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3e00051c0 0x7fe3e000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:38.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.404+0000 7fe3fc911640 1 -- 172.21.15.67:0/4169025225 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f806d4f0 con 0x7fe3e00051c0 2024-03-24T00:40:38.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.404+0000 7fe3fc911640 1 -- 172.21.15.67:0/4169025225 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e000c070 con 0x7fe3e0004800 2024-03-24T00:40:38.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.404+0000 7fe3fc911640 1 -- 172.21.15.67:0/4169025225 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e000c270 con 0x7fe3e00039b0 2024-03-24T00:40:38.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.404+0000 7fe3fe114640 1 --2- 172.21.15.67:0/4169025225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3e00051c0 0x7fe3e000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:38.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.404+0000 7fe3fd112640 1 --2- 172.21.15.67:0/4169025225 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3e0004800 0x7fe3e0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:38.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.404+0000 7fe3fd913640 1 --2- 172.21.15.67:0/4169025225 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe3e00039b0 0x7fe3e0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:38.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.404+0000 7fe3fe114640 1 -- 172.21.15.67:0/4169025225 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe3e00039b0 msgr2=0x7fe3e0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:38.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.404+0000 7fe3fe114640 1 --2- 172.21.15.67:0/4169025225 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe3e00039b0 0x7fe3e0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:38.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.404+0000 7fe3fe114640 1 -- 172.21.15.67:0/4169025225 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3e0004800 msgr2=0x7fe3e0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:38.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.404+0000 7fe3fe114640 1 --2- 172.21.15.67:0/4169025225 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3e0004800 0x7fe3e0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:38.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.404+0000 7fe3fe114640 1 -- 172.21.15.67:0/4169025225 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3e0003520 con 0x7fe3e00051c0 2024-03-24T00:40:38.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.405+0000 7fe3fe114640 1 -- 172.21.15.67:0/4169025225 reap_dead start 2024-03-24T00:40:38.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.405+0000 7fe3fe114640 1 --2- 172.21.15.67:0/4169025225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3e00051c0 0x7fe3e000b9c0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fe3f000c240 tx=0x7fe3e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b85c3fd37f5677ef server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:38.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.405+0000 7fe3fc911640 1 -- 172.21.15.67:0/4169025225 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f0007cf0 con 0x7fe3e00051c0 2024-03-24T00:40:38.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.405+0000 7fe3fc911640 1 -- 172.21.15.67:0/4169025225 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3f0030070 con 0x7fe3e00051c0 2024-03-24T00:40:38.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.405+0000 7fe3fc911640 1 -- 172.21.15.67:0/4169025225 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f0005d30 con 0x7fe3e00051c0 2024-03-24T00:40:38.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.406+0000 7fe3ffb77640 1 -- 172.21.15.67:0/4169025225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3e00051c0 msgr2=0x7fe3e000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:38.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.406+0000 7fe3ffb77640 1 --2- 172.21.15.67:0/4169025225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3e00051c0 0x7fe3e000b9c0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fe3f000c240 tx=0x7fe3e0005830 comp rx=0 tx=0).stop 2024-03-24T00:40:38.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.406+0000 7fe3ffb77640 1 -- 172.21.15.67:0/4169025225 shutdown_connections 2024-03-24T00:40:38.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.406+0000 7fe3ffb77640 1 --2- 172.21.15.67:0/4169025225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3e00051c0 0x7fe3e000b9c0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:38.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.406+0000 7fe3ffb77640 1 -- 172.21.15.67:0/4169025225 >> 172.21.15.67:0/4169025225 conn(0x7fe3f80d1c10 msgr2=0x7fe3f80d2020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:38.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.406+0000 7fe3ffb77640 1 -- 172.21.15.67:0/4169025225 shutdown_connections 2024-03-24T00:40:38.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.406+0000 7fe3ffb77640 1 -- 172.21.15.67:0/4169025225 wait complete. 2024-03-24T00:40:38.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.407+0000 7fe3ffb77640 1 Processor -- start 2024-03-24T00:40:38.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.407+0000 7fe3ffb77640 1 -- start start 2024-03-24T00:40:38.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.408+0000 7fe3ffb77640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3e00051c0 0x7fe3f80b6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:38.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.408+0000 7fe3ffb77640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe3f8156d40 0x7fe3f8159d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:38.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.408+0000 7fe3ffb77640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3f815a250 0x7fe3f81f2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:38.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.408+0000 7fe3ffb77640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f80b1120 con 0x7fe3f815a250 2024-03-24T00:40:38.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.408+0000 7fe3ffb77640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f80d02d0 con 0x7fe3e00051c0 2024-03-24T00:40:38.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.408+0000 7fe3ffb77640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f806c510 con 0x7fe3f8156d40 2024-03-24T00:40:38.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.408+0000 7fe3fe114640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3f815a250 0x7fe3f81f2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:38.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.408+0000 7fe3fe114640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3f815a250 0x7fe3f81f2d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35078/0 (socket says 172.21.15.67:35078) 2024-03-24T00:40:38.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.408+0000 7fe3fe114640 1 -- 172.21.15.67:0/2809084778 learned_addr learned my addr 172.21.15.67:0/2809084778 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:38.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.408+0000 7fe3fd112640 1 --2- 172.21.15.67:0/2809084778 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe3f8156d40 0x7fe3f8159d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:38.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.408+0000 7fe3fe114640 1 -- 172.21.15.67:0/2809084778 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe3f8156d40 msgr2=0x7fe3f8159d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:38.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.408+0000 7fe3fe114640 1 --2- 172.21.15.67:0/2809084778 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe3f8156d40 0x7fe3f8159d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:38.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.408+0000 7fe3fe114640 1 -- 172.21.15.67:0/2809084778 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3e00051c0 msgr2=0x7fe3f80b6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:38.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.408+0000 7fe3fd913640 1 --2- 172.21.15.67:0/2809084778 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3e00051c0 0x7fe3f80b6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:38.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.408+0000 7fe3fe114640 1 --2- 172.21.15.67:0/2809084778 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3e00051c0 0x7fe3f80b6410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:38.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.408+0000 7fe3fe114640 1 -- 172.21.15.67:0/2809084778 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3f8155f80 con 0x7fe3f815a250 2024-03-24T00:40:38.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.408+0000 7fe3fe114640 1 --2- 172.21.15.67:0/2809084778 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3f815a250 0x7fe3f81f2d30 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fe3f000c360 tx=0x7fe3f815f620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:38.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.409+0000 7fe3f67fc640 1 -- 172.21.15.67:0/2809084778 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f0007cf0 con 0x7fe3f815a250 2024-03-24T00:40:38.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.409+0000 7fe3f67fc640 1 -- 172.21.15.67:0/2809084778 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3f00049f0 con 0x7fe3f815a250 2024-03-24T00:40:38.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.409+0000 7fe3ffb77640 1 -- 172.21.15.67:0/2809084778 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3f8156260 con 0x7fe3f815a250 2024-03-24T00:40:38.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.409+0000 7fe3f67fc640 1 -- 172.21.15.67:0/2809084778 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f0001c30 con 0x7fe3f815a250 2024-03-24T00:40:38.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.409+0000 7fe3ffb77640 1 -- 172.21.15.67:0/2809084778 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3f8156820 con 0x7fe3f815a250 2024-03-24T00:40:38.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.410+0000 7fe3ffb77640 1 -- 172.21.15.67:0/2809084778 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3f804f400 con 0x7fe3f815a250 2024-03-24T00:40:38.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.410+0000 7fe3f67fc640 1 -- 172.21.15.67:0/2809084778 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe3f0007070 con 0x7fe3f815a250 2024-03-24T00:40:38.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.410+0000 7fe3f67fc640 1 --2- 172.21.15.67:0/2809084778 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe3e406bc00 0x7fe3e406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:38.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.410+0000 7fe3f67fc640 1 -- 172.21.15.67:0/2809084778 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5619+0+0 (secure 0 0 0) 0x7fe3f00ab910 con 0x7fe3f815a250 2024-03-24T00:40:38.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.411+0000 7fe3fd913640 1 --2- 172.21.15.67:0/2809084778 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe3e406bc00 0x7fe3e406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:38.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.411+0000 7fe3fd913640 1 --2- 172.21.15.67:0/2809084778 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe3e406bc00 0x7fe3e406e0c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fe3e8007520 tx=0x7fe3e800b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:38.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.415+0000 7fe3f67fc640 1 -- 172.21.15.67:0/2809084778 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe3f0046060 con 0x7fe3f815a250 2024-03-24T00:40:38.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:38.596+0000 7fe3ffb77640 1 -- 172.21.15.67:0/2809084778 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fe3f81569f0 con 0x7fe3f815a250 2024-03-24T00:40:39.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.278+0000 7fe3f67fc640 1 -- 172.21.15.67:0/2809084778 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v48) v1 ==== 170+0+0 (secure 0 0 0) 0x7fe3f007a450 con 0x7fe3f815a250 2024-03-24T00:40:39.279 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' created 2024-03-24T00:40:39.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.281+0000 7fe3ffb77640 1 -- 172.21.15.67:0/2809084778 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe3e406bc00 msgr2=0x7fe3e406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:39.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.281+0000 7fe3ffb77640 1 --2- 172.21.15.67:0/2809084778 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe3e406bc00 0x7fe3e406e0c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fe3e8007520 tx=0x7fe3e800b040 comp rx=0 tx=0).stop 2024-03-24T00:40:39.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.281+0000 7fe3ffb77640 1 -- 172.21.15.67:0/2809084778 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3f815a250 msgr2=0x7fe3f81f2d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:39.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.281+0000 7fe3ffb77640 1 --2- 172.21.15.67:0/2809084778 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3f815a250 0x7fe3f81f2d30 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fe3f000c360 tx=0x7fe3f815f620 comp rx=0 tx=0).stop 2024-03-24T00:40:39.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.281+0000 7fe3ffb77640 1 -- 172.21.15.67:0/2809084778 shutdown_connections 2024-03-24T00:40:39.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.281+0000 7fe3ffb77640 1 --2- 172.21.15.67:0/2809084778 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe3e406bc00 0x7fe3e406e0c0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:39.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.281+0000 7fe3ffb77640 1 --2- 172.21.15.67:0/2809084778 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3f815a250 0x7fe3f81f2d30 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:39.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.281+0000 7fe3ffb77640 1 --2- 172.21.15.67:0/2809084778 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe3f8156d40 0x7fe3f8159d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:39.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.281+0000 7fe3ffb77640 1 --2- 172.21.15.67:0/2809084778 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3e00051c0 0x7fe3f80b6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:39.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.281+0000 7fe3ffb77640 1 -- 172.21.15.67:0/2809084778 >> 172.21.15.67:0/2809084778 conn(0x7fe3f80d1c10 msgr2=0x7fe3f80b07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:39.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.281+0000 7fe3ffb77640 1 -- 172.21.15.67:0/2809084778 shutdown_connections 2024-03-24T00:40:39.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.281+0000 7fe3ffb77640 1 -- 172.21.15.67:0/2809084778 wait complete. 2024-03-24T00:40:39.308 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-24T00:40:39.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.414+0000 7f4f00e85640 1 Processor -- start 2024-03-24T00:40:39.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.414+0000 7f4f00e85640 1 -- start start 2024-03-24T00:40:39.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.414+0000 7f4f00e85640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4efc15dfd0 0x7f4efc0cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:39.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.414+0000 7f4f00e85640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4efc0b3d30 0x7f4efc0cc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:39.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.414+0000 7f4f00e85640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4efc0cf090 con 0x7f4efc0b41e0 2024-03-24T00:40:39.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.414+0000 7f4f00e85640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4efc0cc260 con 0x7f4efc0b3d30 2024-03-24T00:40:39.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.414+0000 7f4f00e85640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4efc0d0320 con 0x7f4efc15dfd0 2024-03-24T00:40:39.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.415+0000 7f4efa59c640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4efc15dfd0 0x7f4efc0cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:39.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.415+0000 7f4efa59c640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4efc15dfd0 0x7f4efc0cf980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.67:52704/0 (socket says 172.21.15.67:52704) 2024-03-24T00:40:39.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.415+0000 7f4efad9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4efc0b3d30 0x7f4efc0cc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:39.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.415+0000 7f4efa59c640 1 -- 172.21.15.67:0/3610943016 learned_addr learned my addr 172.21.15.67:0/3610943016 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:39.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.415+0000 7f4efa59c640 1 -- 172.21.15.67:0/3610943016 >> v2:172.21.15.72:3300/0 conn(0x7f4efc0b3d30 msgr2=0x7f4efc0cc6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:39.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.415+0000 7f4efa59c640 1 --2- 172.21.15.67:0/3610943016 >> v2:172.21.15.72:3300/0 conn(0x7f4efc0b3d30 0x7f4efc0cc6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:39.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.415+0000 7f4efa59c640 1 -- 172.21.15.67:0/3610943016 >> v1:172.21.15.72:6789/0 conn(0x7f4efc0b41e0 legacy=0x7f4efc0b3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:39.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.415+0000 7f4efa59c640 1 -- 172.21.15.67:0/3610943016 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4efc0d0600 con 0x7f4efc15dfd0 2024-03-24T00:40:39.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.416+0000 7f4efa59c640 1 --2- 172.21.15.67:0/3610943016 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4efc15dfd0 0x7f4efc0cf980 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4eec009870 tx=0x7f4eec02ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1091f32ef9a1ab3c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:39.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.416+0000 7f4ef9d9b640 1 -- 172.21.15.67:0/3610943016 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eec02f840 con 0x7f4efc15dfd0 2024-03-24T00:40:39.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.416+0000 7f4ef9d9b640 1 -- 172.21.15.67:0/3610943016 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4eec02f9e0 con 0x7f4efc15dfd0 2024-03-24T00:40:39.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.416+0000 7f4ef9d9b640 1 -- 172.21.15.67:0/3610943016 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eec02fdb0 con 0x7f4efc15dfd0 2024-03-24T00:40:39.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.417+0000 7f4f00e85640 1 -- 172.21.15.67:0/3610943016 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4efc15dfd0 msgr2=0x7f4efc0cf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:39.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.417+0000 7f4f00e85640 1 --2- 172.21.15.67:0/3610943016 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4efc15dfd0 0x7f4efc0cf980 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4eec009870 tx=0x7f4eec02ba20 comp rx=0 tx=0).stop 2024-03-24T00:40:39.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.417+0000 7f4f00e85640 1 -- 172.21.15.67:0/3610943016 shutdown_connections 2024-03-24T00:40:39.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.417+0000 7f4f00e85640 1 --2- 172.21.15.67:0/3610943016 >> v2:172.21.15.72:3300/0 conn(0x7f4efc0b3d30 0x7f4efc0cc6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:39.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.417+0000 7f4f00e85640 1 --2- 172.21.15.67:0/3610943016 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4efc15dfd0 0x7f4efc0cf980 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:39.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.417+0000 7f4f00e85640 1 -- 172.21.15.67:0/3610943016 >> 172.21.15.67:0/3610943016 conn(0x7f4efc06fcf0 msgr2=0x7f4efc0700d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:39.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.417+0000 7f4f00e85640 1 -- 172.21.15.67:0/3610943016 shutdown_connections 2024-03-24T00:40:39.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.417+0000 7f4f00e85640 1 -- 172.21.15.67:0/3610943016 wait complete. 2024-03-24T00:40:39.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.418+0000 7f4f00e85640 1 Processor -- start 2024-03-24T00:40:39.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.418+0000 7f4f00e85640 1 -- start start 2024-03-24T00:40:39.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.419+0000 7f4f00e85640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4efc0b3d30 0x7f4efc15b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:39.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.419+0000 7f4f00e85640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4efc0b41e0 0x7f4efc1f8020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:39.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.419+0000 7f4f00e85640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4efc15dfd0 0x7f4efc0b70b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:39.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.419+0000 7f4f00e85640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4efc0cfec0 con 0x7f4efc0b3d30 2024-03-24T00:40:39.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.419+0000 7f4f00e85640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4efc0cedb0 con 0x7f4efc15dfd0 2024-03-24T00:40:39.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.419+0000 7f4f00e85640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4efc15e3b0 con 0x7f4efc0b41e0 2024-03-24T00:40:39.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.419+0000 7f4efa59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4efc0b3d30 0x7f4efc15b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:39.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.419+0000 7f4efa59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4efc0b3d30 0x7f4efc15b810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35094/0 (socket says 172.21.15.67:35094) 2024-03-24T00:40:39.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.419+0000 7f4efa59c640 1 -- 172.21.15.67:0/3647512121 learned_addr learned my addr 172.21.15.67:0/3647512121 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:39.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.419+0000 7f4efad9d640 1 --2- 172.21.15.67:0/3647512121 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4efc15dfd0 0x7f4efc0b70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:39.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.419+0000 7f4ef3fff640 1 --2- 172.21.15.67:0/3647512121 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4efc0b41e0 0x7f4efc1f8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:39.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.419+0000 7f4efa59c640 1 -- 172.21.15.67:0/3647512121 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4efc0b41e0 msgr2=0x7f4efc1f8020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:39.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.419+0000 7f4efa59c640 1 --2- 172.21.15.67:0/3647512121 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4efc0b41e0 0x7f4efc1f8020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:39.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.419+0000 7f4efa59c640 1 -- 172.21.15.67:0/3647512121 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4efc15dfd0 msgr2=0x7f4efc0b70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:39.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.419+0000 7f4efa59c640 1 --2- 172.21.15.67:0/3647512121 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4efc15dfd0 0x7f4efc0b70b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:39.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.419+0000 7f4efa59c640 1 -- 172.21.15.67:0/3647512121 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4efc158730 con 0x7f4efc0b3d30 2024-03-24T00:40:39.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.419+0000 7f4ef3fff640 1 --2- 172.21.15.67:0/3647512121 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4efc0b41e0 0x7f4efc1f8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:40:39.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.419+0000 7f4efad9d640 1 --2- 172.21.15.67:0/3647512121 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4efc15dfd0 0x7f4efc0b70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:40:39.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.419+0000 7f4efa59c640 1 --2- 172.21.15.67:0/3647512121 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4efc0b3d30 0x7f4efc15b810 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f4eec02ba60 tx=0x7f4eec004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:39.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.420+0000 7f4ef2ffd640 1 -- 172.21.15.67:0/3647512121 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eec047070 con 0x7f4efc0b3d30 2024-03-24T00:40:39.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.420+0000 7f4ef2ffd640 1 -- 172.21.15.67:0/3647512121 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4eec037bc0 con 0x7f4efc0b3d30 2024-03-24T00:40:39.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.420+0000 7f4ef2ffd640 1 -- 172.21.15.67:0/3647512121 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eec042480 con 0x7f4efc0b3d30 2024-03-24T00:40:39.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.420+0000 7f4f00e85640 1 -- 172.21.15.67:0/3647512121 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4efc159bb0 con 0x7f4efc0b3d30 2024-03-24T00:40:39.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.420+0000 7f4f00e85640 1 -- 172.21.15.67:0/3647512121 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4efc15a170 con 0x7f4efc0b3d30 2024-03-24T00:40:39.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.421+0000 7f4f00e85640 1 -- 172.21.15.67:0/3647512121 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4efc0cf6c0 con 0x7f4efc0b3d30 2024-03-24T00:40:39.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.421+0000 7f4ef2ffd640 1 -- 172.21.15.67:0/3647512121 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4eec04a070 con 0x7f4efc0b3d30 2024-03-24T00:40:39.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.422+0000 7f4ef2ffd640 1 --2- 172.21.15.67:0/3647512121 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4edc06bc00 0x7f4edc06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:39.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.422+0000 7f4ef2ffd640 1 -- 172.21.15.67:0/3647512121 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f4eec0acd10 con 0x7f4efc0b3d30 2024-03-24T00:40:39.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.422+0000 7f4ef3fff640 1 --2- 172.21.15.67:0/3647512121 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4edc06bc00 0x7f4edc06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:39.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.422+0000 7f4ef3fff640 1 --2- 172.21.15.67:0/3647512121 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4edc06bc00 0x7f4edc06e0c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f4efc0c8c20 tx=0x7f4ee4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:39.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.426+0000 7f4ef2ffd640 1 -- 172.21.15.67:0/3647512121 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4efc158730 con 0x7f4efc0b3d30 2024-03-24T00:40:39.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:39.647+0000 7f4f00e85640 1 -- 172.21.15.67:0/3647512121 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f4efc04f400 con 0x7f4efc0b3d30 2024-03-24T00:40:40.293 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 7 and data pool 8 2024-03-24T00:40:40.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.292+0000 7f4ef2ffd640 1 -- 172.21.15.67:0/3647512121 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 7 and data pool 8 v22) v1 ==== 172+0+0 (secure 0 0 0) 0x7f4eec033030 con 0x7f4efc0b3d30 2024-03-24T00:40:40.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.296+0000 7f4f00e85640 1 -- 172.21.15.67:0/3647512121 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4edc06bc00 msgr2=0x7f4edc06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:40.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.296+0000 7f4f00e85640 1 --2- 172.21.15.67:0/3647512121 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4edc06bc00 0x7f4edc06e0c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f4efc0c8c20 tx=0x7f4ee4009040 comp rx=0 tx=0).stop 2024-03-24T00:40:40.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.296+0000 7f4f00e85640 1 -- 172.21.15.67:0/3647512121 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4efc0b3d30 msgr2=0x7f4efc15b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:40.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.296+0000 7f4f00e85640 1 --2- 172.21.15.67:0/3647512121 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4efc0b3d30 0x7f4efc15b810 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f4eec02ba60 tx=0x7f4eec004730 comp rx=0 tx=0).stop 2024-03-24T00:40:40.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.298+0000 7f4f00e85640 1 -- 172.21.15.67:0/3647512121 shutdown_connections 2024-03-24T00:40:40.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.298+0000 7f4f00e85640 1 --2- 172.21.15.67:0/3647512121 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4edc06bc00 0x7f4edc06e0c0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:40.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.298+0000 7f4f00e85640 1 --2- 172.21.15.67:0/3647512121 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4efc15dfd0 0x7f4efc0b70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:40.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.298+0000 7f4f00e85640 1 --2- 172.21.15.67:0/3647512121 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4efc0b41e0 0x7f4efc1f8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:40.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.298+0000 7f4f00e85640 1 --2- 172.21.15.67:0/3647512121 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4efc0b3d30 0x7f4efc15b810 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:40.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.298+0000 7f4f00e85640 1 -- 172.21.15.67:0/3647512121 >> 172.21.15.67:0/3647512121 conn(0x7f4efc06fcf0 msgr2=0x7f4efc06c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:40.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.299+0000 7f4f00e85640 1 -- 172.21.15.67:0/3647512121 shutdown_connections 2024-03-24T00:40:40.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.304+0000 7f4f00e85640 1 -- 172.21.15.67:0/3647512121 wait complete. 2024-03-24T00:40:40.324 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:40:40.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.426+0000 7f58ab577640 1 Processor -- start 2024-03-24T00:40:40.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.426+0000 7f58ab577640 1 -- start start 2024-03-24T00:40:40.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.427+0000 7f58ab577640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f58a409a410 0x7f58a40cfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:40.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.427+0000 7f58ab577640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58a40d0fc0 con 0x7f58a40b43a0 2024-03-24T00:40:40.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.427+0000 7f58ab577640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58a40d1260 con 0x7f58a415e1b0 2024-03-24T00:40:40.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.427+0000 7f58ab577640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58a415e590 con 0x7f58a409a410 2024-03-24T00:40:40.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.427+0000 7f58a9b14640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f58a40b43a0 0x7f58a40cc880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52574/0 (socket says 172.21.15.67:52574) 2024-03-24T00:40:40.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.427+0000 7f58a9b14640 1 -- 172.21.15.67:0/926928353 learned_addr learned my addr 172.21.15.67:0/926928353 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:40:40.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.427+0000 7f58a8b12640 1 --2- 172.21.15.67:0/926928353 >> v2:172.21.15.72:3300/0 conn(0x7f58a409a410 0x7f58a40cfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:40.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.427+0000 7f58a3fff640 1 -- 172.21.15.67:0/926928353 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1088378629 0 0) 0x7f58a40d0fc0 con 0x7f58a40b43a0 2024-03-24T00:40:40.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.427+0000 7f58a3fff640 1 -- 172.21.15.67:0/926928353 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f588c003520 con 0x7f58a40b43a0 2024-03-24T00:40:40.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.427+0000 7f58a3fff640 1 -- 172.21.15.67:0/926928353 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3430207999 0 0) 0x7f58a40d1260 con 0x7f58a415e1b0 2024-03-24T00:40:40.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.427+0000 7f58a3fff640 1 -- 172.21.15.67:0/926928353 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58a40d0fc0 con 0x7f58a415e1b0 2024-03-24T00:40:40.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.428+0000 7f58a3fff640 1 -- 172.21.15.67:0/926928353 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1401165191 0 0) 0x7f588c003520 con 0x7f58a40b43a0 2024-03-24T00:40:40.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.428+0000 7f58a3fff640 1 -- 172.21.15.67:0/926928353 >> v2:172.21.15.72:3300/0 conn(0x7f58a409a410 msgr2=0x7f58a40cfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:40.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.428+0000 7f58a3fff640 1 --2- 172.21.15.67:0/926928353 >> v2:172.21.15.72:3300/0 conn(0x7f58a409a410 0x7f58a40cfe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:40.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.428+0000 7f58a3fff640 1 -- 172.21.15.67:0/926928353 >> v1:172.21.15.72:6789/0 conn(0x7f58a415e1b0 legacy=0x7f58a4099d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:40.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.428+0000 7f58a3fff640 1 -- 172.21.15.67:0/926928353 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a40d1540 con 0x7f58a40b43a0 2024-03-24T00:40:40.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.428+0000 7f58a3fff640 1 -- 172.21.15.67:0/926928353 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f5894001670 con 0x7f58a40b43a0 2024-03-24T00:40:40.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.428+0000 7f58a8b12640 1 --2- 172.21.15.67:0/926928353 >> v2:172.21.15.72:3300/0 conn(0x7f58a409a410 0x7f58a40cfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:40:40.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.428+0000 7f58a3fff640 1 -- 172.21.15.67:0/926928353 >> v1:172.21.15.67:6789/0 conn(0x7f58a40b43a0 legacy=0x7f58a40cc880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:40.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.428+0000 7f58a3fff640 1 --2- 172.21.15.67:0/926928353 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f588c003b40 0x7f588c003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:40.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.428+0000 7f58a3fff640 1 --2- 172.21.15.67:0/926928353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f588c004800 0x7f588c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:40.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.428+0000 7f58a3fff640 1 --2- 172.21.15.67:0/926928353 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f588c005230 0x7f588c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:40.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.428+0000 7f58a3fff640 1 -- 172.21.15.67:0/926928353 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a415e590 con 0x7f588c004800 2024-03-24T00:40:40.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.428+0000 7f58a3fff640 1 -- 172.21.15.67:0/926928353 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f588c00c190 con 0x7f588c003b40 2024-03-24T00:40:40.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.428+0000 7f58a3fff640 1 -- 172.21.15.67:0/926928353 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f588c00c390 con 0x7f588c005230 2024-03-24T00:40:40.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.428+0000 7f58a8b12640 1 --2- 172.21.15.67:0/926928353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f588c004800 0x7f588c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:40.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.428+0000 7f58a9313640 1 --2- 172.21.15.67:0/926928353 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f588c003b40 0x7f588c003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:40.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.428+0000 7f58a9b14640 1 --2- 172.21.15.67:0/926928353 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f588c005230 0x7f588c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:40.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.428+0000 7f58a8b12640 1 -- 172.21.15.67:0/926928353 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f588c005230 msgr2=0x7f588c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:40.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.428+0000 7f58a8b12640 1 --2- 172.21.15.67:0/926928353 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f588c005230 0x7f588c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:40.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.428+0000 7f58a8b12640 1 -- 172.21.15.67:0/926928353 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f588c003b40 msgr2=0x7f588c003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:40.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.428+0000 7f58a8b12640 1 --2- 172.21.15.67:0/926928353 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f588c003b40 0x7f588c003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:40.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.428+0000 7f58a8b12640 1 -- 172.21.15.67:0/926928353 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f588c003520 con 0x7f588c004800 2024-03-24T00:40:40.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.428+0000 7f58a9b14640 1 --2- 172.21.15.67:0/926928353 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f588c005230 0x7f588c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:40:40.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.428+0000 7f58a9b14640 1 -- 172.21.15.67:0/926928353 reap_dead start 2024-03-24T00:40:40.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.429+0000 7f58a8b12640 1 --2- 172.21.15.67:0/926928353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f588c004800 0x7f588c004c80 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f5898009c60 tx=0x7f588c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e136cd5b24f0a674 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:40.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.429+0000 7f58a3fff640 1 -- 172.21.15.67:0/926928353 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5898039070 con 0x7f588c004800 2024-03-24T00:40:40.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.429+0000 7f58a3fff640 1 -- 172.21.15.67:0/926928353 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f589802c8d0 con 0x7f588c004800 2024-03-24T00:40:40.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.429+0000 7f58a3fff640 1 -- 172.21.15.67:0/926928353 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5898009860 con 0x7f588c004800 2024-03-24T00:40:40.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.429+0000 7f58ab577640 1 -- 172.21.15.67:0/926928353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f588c004800 msgr2=0x7f588c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:40.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.429+0000 7f58ab577640 1 --2- 172.21.15.67:0/926928353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f588c004800 0x7f588c004c80 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f5898009c60 tx=0x7f588c005830 comp rx=0 tx=0).stop 2024-03-24T00:40:40.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.429+0000 7f58ab577640 1 -- 172.21.15.67:0/926928353 shutdown_connections 2024-03-24T00:40:40.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.429+0000 7f58ab577640 1 --2- 172.21.15.67:0/926928353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f588c004800 0x7f588c004c80 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:40.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.429+0000 7f58ab577640 1 -- 172.21.15.67:0/926928353 >> 172.21.15.67:0/926928353 conn(0x7f58a40b36d0 msgr2=0x7f58a40b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:40.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.430+0000 7f58ab577640 1 -- 172.21.15.67:0/926928353 shutdown_connections 2024-03-24T00:40:40.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.430+0000 7f58ab577640 1 -- 172.21.15.67:0/926928353 wait complete. 2024-03-24T00:40:40.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.431+0000 7f58ab577640 1 Processor -- start 2024-03-24T00:40:40.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.431+0000 7f58ab577640 1 -- start start 2024-03-24T00:40:40.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.431+0000 7f58ab577640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f588c004800 0x7f58a4152f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:40.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.431+0000 7f58ab577640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f58a40caab0 0x7f58a40b47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:40.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.431+0000 7f58ab577640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f58a40cc780 0x7f58a40ccc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:40.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.431+0000 7f58ab577640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a409a9a0 con 0x7f588c004800 2024-03-24T00:40:40.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.431+0000 7f58ab577640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a40d1820 con 0x7f58a40caab0 2024-03-24T00:40:40.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.431+0000 7f58ab577640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58a40cfe20 con 0x7f58a40cc780 2024-03-24T00:40:40.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.431+0000 7f58a9313640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f588c004800 0x7f58a4152f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:40.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.432+0000 7f58a9313640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f588c004800 0x7f58a4152f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35114/0 (socket says 172.21.15.67:35114) 2024-03-24T00:40:40.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.432+0000 7f58a9313640 1 -- 172.21.15.67:0/1806651200 learned_addr learned my addr 172.21.15.67:0/1806651200 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:40.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.432+0000 7f58a8b12640 1 --2- 172.21.15.67:0/1806651200 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f58a40caab0 0x7f58a40b47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:40.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.432+0000 7f58a9b14640 1 --2- 172.21.15.67:0/1806651200 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f58a40cc780 0x7f58a40ccc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:40.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.432+0000 7f58a9313640 1 -- 172.21.15.67:0/1806651200 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f58a40cc780 msgr2=0x7f58a40ccc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:40.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.432+0000 7f58a9313640 1 --2- 172.21.15.67:0/1806651200 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f58a40cc780 0x7f58a40ccc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:40.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.432+0000 7f58a9313640 1 -- 172.21.15.67:0/1806651200 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f58a40caab0 msgr2=0x7f58a40b47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:40.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.432+0000 7f58a9313640 1 --2- 172.21.15.67:0/1806651200 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f58a40caab0 0x7f58a40b47d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:40.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.432+0000 7f58a9313640 1 -- 172.21.15.67:0/1806651200 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a40b50d0 con 0x7f588c004800 2024-03-24T00:40:40.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.432+0000 7f58a8b12640 1 --2- 172.21.15.67:0/1806651200 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f58a40caab0 0x7f58a40b47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:40:40.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.432+0000 7f58a9313640 1 --2- 172.21.15.67:0/1806651200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f588c004800 0x7f58a4152f60 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f589c00fae0 tx=0x7f589c002190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:40.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.432+0000 7f58a1ffb640 1 -- 172.21.15.67:0/1806651200 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f589c003b80 con 0x7f588c004800 2024-03-24T00:40:40.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.432+0000 7f58a1ffb640 1 -- 172.21.15.67:0/1806651200 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f589c018070 con 0x7f588c004800 2024-03-24T00:40:40.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.432+0000 7f58ab577640 1 -- 172.21.15.67:0/1806651200 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58a40b53b0 con 0x7f588c004800 2024-03-24T00:40:40.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.432+0000 7f58a1ffb640 1 -- 172.21.15.67:0/1806651200 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f589c00d850 con 0x7f588c004800 2024-03-24T00:40:40.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.432+0000 7f58ab577640 1 -- 172.21.15.67:0/1806651200 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58a40b4d10 con 0x7f588c004800 2024-03-24T00:40:40.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.433+0000 7f58ab577640 1 -- 172.21.15.67:0/1806651200 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58a4067300 con 0x7f588c004800 2024-03-24T00:40:40.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.434+0000 7f58a1ffb640 1 -- 172.21.15.67:0/1806651200 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f589c00d9f0 con 0x7f588c004800 2024-03-24T00:40:40.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.434+0000 7f58a1ffb640 1 --2- 172.21.15.67:0/1806651200 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f589006bbd0 0x7f589006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:40.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.434+0000 7f58a1ffb640 1 -- 172.21.15.67:0/1806651200 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f589c08ae50 con 0x7f588c004800 2024-03-24T00:40:40.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.434+0000 7f58a8b12640 1 --2- 172.21.15.67:0/1806651200 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f589006bbd0 0x7f589006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:40.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.435+0000 7f58a8b12640 1 --2- 172.21.15.67:0/1806651200 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f589006bbd0 0x7f589006e090 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f5898036000 tx=0x7f5898030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:40.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.439+0000 7f58a1ffb640 1 -- 172.21.15.67:0/1806651200 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f58a4067300 con 0x7f588c004800 2024-03-24T00:40:40.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.624+0000 7f58ab577640 1 -- 172.21.15.67:0/1806651200 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f58a404f400 con 0x7f588c004800 2024-03-24T00:40:40.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.625+0000 7f58a1ffb640 1 -- 172.21.15.67:0/1806651200 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v49) v1 ==== 74+0+16895 (secure 0 0 0) 0x7f589c05f020 con 0x7f588c004800 2024-03-24T00:40:40.626 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:40:40.626 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":49,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:40:39.649329+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:40:37.927587+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-24T00:40:38.616521+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:40:40.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.627+0000 7f58ab577640 1 -- 172.21.15.67:0/1806651200 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f589006bbd0 msgr2=0x7f589006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:40.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.627+0000 7f58ab577640 1 --2- 172.21.15.67:0/1806651200 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f589006bbd0 0x7f589006e090 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f5898036000 tx=0x7f5898030000 comp rx=0 tx=0).stop 2024-03-24T00:40:40.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.627+0000 7f58ab577640 1 -- 172.21.15.67:0/1806651200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f588c004800 msgr2=0x7f58a4152f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:40.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.627+0000 7f58ab577640 1 --2- 172.21.15.67:0/1806651200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f588c004800 0x7f58a4152f60 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f589c00fae0 tx=0x7f589c002190 comp rx=0 tx=0).stop 2024-03-24T00:40:40.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.627+0000 7f58ab577640 1 -- 172.21.15.67:0/1806651200 shutdown_connections 2024-03-24T00:40:40.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.627+0000 7f58ab577640 1 --2- 172.21.15.67:0/1806651200 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f589006bbd0 0x7f589006e090 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:40.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.627+0000 7f58ab577640 1 --2- 172.21.15.67:0/1806651200 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f58a40cc780 0x7f58a40ccc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:40.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.627+0000 7f58ab577640 1 --2- 172.21.15.67:0/1806651200 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f58a40caab0 0x7f58a40b47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:40.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.627+0000 7f58ab577640 1 --2- 172.21.15.67:0/1806651200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f588c004800 0x7f58a4152f60 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:40.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.627+0000 7f58ab577640 1 -- 172.21.15.67:0/1806651200 >> 172.21.15.67:0/1806651200 conn(0x7f58a40b36d0 msgr2=0x7f58a4071410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:40.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.627+0000 7f58ab577640 1 -- 172.21.15.67:0/1806651200 shutdown_connections 2024-03-24T00:40:40.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.627+0000 7f58ab577640 1 -- 172.21.15.67:0/1806651200 wait complete. 2024-03-24T00:40:40.639 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:40:40.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.740+0000 7f7e36fb3640 1 Processor -- start 2024-03-24T00:40:40.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.740+0000 7f7e36fb3640 1 -- start start 2024-03-24T00:40:40.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.741+0000 7f7e36fb3640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7e300d1550 0x7f7e3006cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:40.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.741+0000 7f7e36fb3640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7e300b2780 0x7f7e300b2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:40.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.741+0000 7f7e36fb3640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7e30099d70 0x7f7e3009a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:40.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.741+0000 7f7e36fb3640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e3006d420 con 0x7f7e300d1550 2024-03-24T00:40:40.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.741+0000 7f7e36fb3640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e3006ca50 con 0x7f7e30099d70 2024-03-24T00:40:40.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.741+0000 7f7e36fb3640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e300d1930 con 0x7f7e300b2780 2024-03-24T00:40:40.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.741+0000 7f7e34d4f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7e300d1550 0x7f7e3006cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:40.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.741+0000 7f7e34d4f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7e300d1550 0x7f7e3006cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35120/0 (socket says 172.21.15.67:35120) 2024-03-24T00:40:40.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.741+0000 7f7e34d4f640 1 -- 172.21.15.67:0/947467005 learned_addr learned my addr 172.21.15.67:0/947467005 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:40.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.741+0000 7f7e35550640 1 --2- 172.21.15.67:0/947467005 >> v2:172.21.15.72:3300/0 conn(0x7f7e30099d70 0x7f7e3009a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:40.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.741+0000 7f7e2ffff640 1 --2- 172.21.15.67:0/947467005 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7e300b2780 0x7f7e300b2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:40.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.741+0000 7f7e34d4f640 1 -- 172.21.15.67:0/947467005 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7e300b2780 msgr2=0x7f7e300b2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:40.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.741+0000 7f7e34d4f640 1 --2- 172.21.15.67:0/947467005 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7e300b2780 0x7f7e300b2c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:40.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.741+0000 7f7e34d4f640 1 -- 172.21.15.67:0/947467005 >> v2:172.21.15.72:3300/0 conn(0x7f7e30099d70 msgr2=0x7f7e3009a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:40.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.741+0000 7f7e34d4f640 1 --2- 172.21.15.67:0/947467005 >> v2:172.21.15.72:3300/0 conn(0x7f7e30099d70 0x7f7e3009a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:40.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.741+0000 7f7e34d4f640 1 -- 172.21.15.67:0/947467005 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e300b1f60 con 0x7f7e300d1550 2024-03-24T00:40:40.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.742+0000 7f7e34d4f640 1 --2- 172.21.15.67:0/947467005 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e300d1550 0x7f7e3006cee0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f7e30073290 tx=0x7f7e2802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9978eca53e403a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:40.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.742+0000 7f7e2f7fe640 1 -- 172.21.15.67:0/947467005 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e2802ca70 con 0x7f7e300d1550 2024-03-24T00:40:40.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.742+0000 7f7e2f7fe640 1 -- 172.21.15.67:0/947467005 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e2802cc10 con 0x7f7e300d1550 2024-03-24T00:40:40.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.742+0000 7f7e2f7fe640 1 -- 172.21.15.67:0/947467005 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e280343f0 con 0x7f7e300d1550 2024-03-24T00:40:40.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.743+0000 7f7e36fb3640 1 -- 172.21.15.67:0/947467005 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e300d1550 msgr2=0x7f7e3006cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:40.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.743+0000 7f7e36fb3640 1 --2- 172.21.15.67:0/947467005 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e300d1550 0x7f7e3006cee0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f7e30073290 tx=0x7f7e2802bed0 comp rx=0 tx=0).stop 2024-03-24T00:40:40.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.743+0000 7f7e36fb3640 1 -- 172.21.15.67:0/947467005 shutdown_connections 2024-03-24T00:40:40.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.743+0000 7f7e36fb3640 1 --2- 172.21.15.67:0/947467005 >> v2:172.21.15.72:3300/0 conn(0x7f7e30099d70 0x7f7e3009a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:40.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.743+0000 7f7e36fb3640 1 --2- 172.21.15.67:0/947467005 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7e300b2780 0x7f7e300b2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:40.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.743+0000 7f7e36fb3640 1 --2- 172.21.15.67:0/947467005 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e300d1550 0x7f7e3006cee0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:40.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.743+0000 7f7e36fb3640 1 -- 172.21.15.67:0/947467005 >> 172.21.15.67:0/947467005 conn(0x7f7e300b36e0 msgr2=0x7f7e300b3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:40.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.743+0000 7f7e36fb3640 1 -- 172.21.15.67:0/947467005 shutdown_connections 2024-03-24T00:40:40.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.743+0000 7f7e36fb3640 1 -- 172.21.15.67:0/947467005 wait complete. 2024-03-24T00:40:40.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.744+0000 7f7e36fb3640 1 Processor -- start 2024-03-24T00:40:40.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.744+0000 7f7e36fb3640 1 -- start start 2024-03-24T00:40:40.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.745+0000 7f7e36fb3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e30099d70 0x7f7e300c6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:40.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.745+0000 7f7e36fb3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e300b2780 0x7f7e30160810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:40.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.745+0000 7f7e36fb3640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7e300d1550 0x7f7e3015ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:40.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.745+0000 7f7e36fb3640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3006c400 con 0x7f7e30099d70 2024-03-24T00:40:40.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.745+0000 7f7e36fb3640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e300b2520 con 0x7f7e300b2780 2024-03-24T00:40:40.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.745+0000 7f7e36fb3640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3006c6e0 con 0x7f7e300d1550 2024-03-24T00:40:40.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.745+0000 7f7e34d4f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e30099d70 0x7f7e300c6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:40.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.745+0000 7f7e34d4f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e30099d70 0x7f7e300c6130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35124/0 (socket says 172.21.15.67:35124) 2024-03-24T00:40:40.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.745+0000 7f7e34d4f640 1 -- 172.21.15.67:0/2284195168 learned_addr learned my addr 172.21.15.67:0/2284195168 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:40.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.745+0000 7f7e2ffff640 1 --2- 172.21.15.67:0/2284195168 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e300b2780 0x7f7e30160810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:40.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.745+0000 7f7e35550640 1 --2- 172.21.15.67:0/2284195168 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7e300d1550 0x7f7e3015ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:40.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.745+0000 7f7e34d4f640 1 -- 172.21.15.67:0/2284195168 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7e300d1550 msgr2=0x7f7e3015ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:40.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.745+0000 7f7e34d4f640 1 --2- 172.21.15.67:0/2284195168 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7e300d1550 0x7f7e3015ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:40.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.745+0000 7f7e34d4f640 1 -- 172.21.15.67:0/2284195168 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e300b2780 msgr2=0x7f7e30160810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:40.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.745+0000 7f7e34d4f640 1 --2- 172.21.15.67:0/2284195168 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e300b2780 0x7f7e30160810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:40.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.745+0000 7f7e34d4f640 1 -- 172.21.15.67:0/2284195168 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e300c6e10 con 0x7f7e30099d70 2024-03-24T00:40:40.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.745+0000 7f7e34d4f640 1 --2- 172.21.15.67:0/2284195168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e30099d70 0x7f7e300c6130 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f7e28036040 tx=0x7f7e28004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:40.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.745+0000 7f7e2d7fa640 1 -- 172.21.15.67:0/2284195168 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e28043020 con 0x7f7e30099d70 2024-03-24T00:40:40.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.746+0000 7f7e2d7fa640 1 -- 172.21.15.67:0/2284195168 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e2803e450 con 0x7f7e30099d70 2024-03-24T00:40:40.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.746+0000 7f7e36fb3640 1 -- 172.21.15.67:0/2284195168 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e300c70f0 con 0x7f7e30099d70 2024-03-24T00:40:40.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.746+0000 7f7e2d7fa640 1 -- 172.21.15.67:0/2284195168 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e2803e840 con 0x7f7e30099d70 2024-03-24T00:40:40.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.746+0000 7f7e36fb3640 1 -- 172.21.15.67:0/2284195168 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e300c76b0 con 0x7f7e30099d70 2024-03-24T00:40:40.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.747+0000 7f7e36fb3640 1 -- 172.21.15.67:0/2284195168 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e3004f400 con 0x7f7e30099d70 2024-03-24T00:40:40.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.747+0000 7f7e2d7fa640 1 -- 172.21.15.67:0/2284195168 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7e28046020 con 0x7f7e30099d70 2024-03-24T00:40:40.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.747+0000 7f7e2d7fa640 1 --2- 172.21.15.67:0/2284195168 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7e1006bc00 0x7f7e1006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:40.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.747+0000 7f7e2d7fa640 1 -- 172.21.15.67:0/2284195168 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7e280ad330 con 0x7f7e30099d70 2024-03-24T00:40:40.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.747+0000 7f7e2ffff640 1 --2- 172.21.15.67:0/2284195168 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7e1006bc00 0x7f7e1006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:40.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.748+0000 7f7e2ffff640 1 --2- 172.21.15.67:0/2284195168 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7e1006bc00 0x7f7e1006e0c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f7e20006fd0 tx=0x7f7e20008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:40.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.752+0000 7f7e2d7fa640 1 -- 172.21.15.67:0/2284195168 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7e2804d070 con 0x7f7e30099d70 2024-03-24T00:40:40.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.935+0000 7f7e36fb3640 1 -- 172.21.15.67:0/2284195168 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e300b1310 con 0x7f7e30099d70 2024-03-24T00:40:40.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.936+0000 7f7e2d7fa640 1 -- 172.21.15.67:0/2284195168 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v49) v1 ==== 74+0+16895 (secure 0 0 0) 0x7f7e2807bc00 con 0x7f7e30099d70 2024-03-24T00:40:40.937 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:40:40.937 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":49,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:40:39.649329+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:40:37.927587+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-24T00:40:38.616521+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:40:40.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.938+0000 7f7e36fb3640 1 -- 172.21.15.67:0/2284195168 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7e1006bc00 msgr2=0x7f7e1006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:40.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.938+0000 7f7e36fb3640 1 --2- 172.21.15.67:0/2284195168 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7e1006bc00 0x7f7e1006e0c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f7e20006fd0 tx=0x7f7e20008040 comp rx=0 tx=0).stop 2024-03-24T00:40:40.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.938+0000 7f7e36fb3640 1 -- 172.21.15.67:0/2284195168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e30099d70 msgr2=0x7f7e300c6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:40.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.938+0000 7f7e36fb3640 1 --2- 172.21.15.67:0/2284195168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e30099d70 0x7f7e300c6130 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f7e28036040 tx=0x7f7e28004a40 comp rx=0 tx=0).stop 2024-03-24T00:40:40.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.938+0000 7f7e36fb3640 1 -- 172.21.15.67:0/2284195168 shutdown_connections 2024-03-24T00:40:40.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.938+0000 7f7e36fb3640 1 --2- 172.21.15.67:0/2284195168 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7e1006bc00 0x7f7e1006e0c0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:40.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.938+0000 7f7e36fb3640 1 --2- 172.21.15.67:0/2284195168 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7e300d1550 0x7f7e3015ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:40.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.938+0000 7f7e36fb3640 1 --2- 172.21.15.67:0/2284195168 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e300b2780 0x7f7e30160810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:40.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.938+0000 7f7e36fb3640 1 --2- 172.21.15.67:0/2284195168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e30099d70 0x7f7e300c6130 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:40.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.938+0000 7f7e36fb3640 1 -- 172.21.15.67:0/2284195168 >> 172.21.15.67:0/2284195168 conn(0x7f7e300b36e0 msgr2=0x7f7e30161d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:40.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.938+0000 7f7e36fb3640 1 -- 172.21.15.67:0/2284195168 shutdown_connections 2024-03-24T00:40:40.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:40.938+0000 7f7e36fb3640 1 -- 172.21.15.67:0/2284195168 wait complete. 2024-03-24T00:40:40.950 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-24T00:40:41.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.054+0000 7f940553b640 1 Processor -- start 2024-03-24T00:40:41.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.054+0000 7f940553b640 1 -- start start 2024-03-24T00:40:41.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.055+0000 7f940553b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f94000b0f10 0x7f94000d1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:41.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.055+0000 7f940553b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f94000b2610 0x7f940015d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:41.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.055+0000 7f940553b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94000b12f0 con 0x7f94000d20d0 2024-03-24T00:40:41.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.055+0000 7f940553b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94000d2540 con 0x7f94000b2610 2024-03-24T00:40:41.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.055+0000 7f940553b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94000d09f0 con 0x7f94000b0f10 2024-03-24T00:40:41.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.055+0000 7f93ff7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f94000b2610 0x7f940015d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:41.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.055+0000 7f93fe7fc640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f94000d20d0 0x7f94000b1f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52576/0 (socket says 172.21.15.67:52576) 2024-03-24T00:40:41.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.055+0000 7f93fe7fc640 1 -- 172.21.15.67:0/1296815491 learned_addr learned my addr 172.21.15.67:0/1296815491 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:40:41.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.055+0000 7f93feffd640 1 --2- 172.21.15.67:0/1296815491 >> v2:172.21.15.72:3300/0 conn(0x7f94000b0f10 0x7f94000d1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:41.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.056+0000 7f93ff7fe640 1 -- 172.21.15.67:0/1296815491 >> v2:172.21.15.72:3300/0 conn(0x7f94000b0f10 msgr2=0x7f94000d1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:41.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.056+0000 7f93ff7fe640 1 --2- 172.21.15.67:0/1296815491 >> v2:172.21.15.72:3300/0 conn(0x7f94000b0f10 0x7f94000d1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:41.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.056+0000 7f93ff7fe640 1 -- 172.21.15.67:0/1296815491 >> v1:172.21.15.67:6789/0 conn(0x7f94000d20d0 legacy=0x7f94000b1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:41.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.056+0000 7f93ff7fe640 1 -- 172.21.15.67:0/1296815491 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94000b2c50 con 0x7f94000b2610 2024-03-24T00:40:41.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.056+0000 7f93fdffb640 1 -- 172.21.15.67:0/1296815491 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1430397706 0 0) 0x7f94000b12f0 con 0x7f94000d20d0 2024-03-24T00:40:41.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.056+0000 7f93ff7fe640 1 --2- 172.21.15.67:0/1296815491 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94000b2610 0x7f940015d870 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f940008bf10 tx=0x7f93f802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26152bded792342d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:41.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.057+0000 7f93fdffb640 1 -- 172.21.15.67:0/1296815491 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f8039070 con 0x7f94000b2610 2024-03-24T00:40:41.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.057+0000 7f93fdffb640 1 -- 172.21.15.67:0/1296815491 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93f802cc80 con 0x7f94000b2610 2024-03-24T00:40:41.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.057+0000 7f93fdffb640 1 -- 172.21.15.67:0/1296815491 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f8034660 con 0x7f94000b2610 2024-03-24T00:40:41.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.057+0000 7f940553b640 1 -- 172.21.15.67:0/1296815491 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94000b2610 msgr2=0x7f940015d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:41.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.057+0000 7f940553b640 1 --2- 172.21.15.67:0/1296815491 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94000b2610 0x7f940015d870 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f940008bf10 tx=0x7f93f802bed0 comp rx=0 tx=0).stop 2024-03-24T00:40:41.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.057+0000 7f940553b640 1 -- 172.21.15.67:0/1296815491 shutdown_connections 2024-03-24T00:40:41.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.057+0000 7f940553b640 1 --2- 172.21.15.67:0/1296815491 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94000b2610 0x7f940015d870 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:41.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.057+0000 7f940553b640 1 --2- 172.21.15.67:0/1296815491 >> v2:172.21.15.72:3300/0 conn(0x7f94000b0f10 0x7f94000d1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:41.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.057+0000 7f940553b640 1 -- 172.21.15.67:0/1296815491 >> 172.21.15.67:0/1296815491 conn(0x7f940008b1b0 msgr2=0x7f940008b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:41.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.057+0000 7f940553b640 1 -- 172.21.15.67:0/1296815491 shutdown_connections 2024-03-24T00:40:41.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.057+0000 7f940553b640 1 -- 172.21.15.67:0/1296815491 wait complete. 2024-03-24T00:40:41.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.058+0000 7f940553b640 1 Processor -- start 2024-03-24T00:40:41.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.058+0000 7f940553b640 1 -- start start 2024-03-24T00:40:41.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.059+0000 7f940553b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94000b0f10 0x7f9400151000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:41.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.059+0000 7f940553b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f94000b2610 0x7f94000b6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:41.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.059+0000 7f940553b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f94000d20d0 0x7f94000c5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:41.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.059+0000 7f940553b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94000ceb60 con 0x7f94000b0f10 2024-03-24T00:40:41.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.059+0000 7f940553b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94000cee40 con 0x7f94000b2610 2024-03-24T00:40:41.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.059+0000 7f940553b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94000ce560 con 0x7f94000d20d0 2024-03-24T00:40:41.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.059+0000 7f93feffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94000b0f10 0x7f9400151000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:41.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.059+0000 7f93feffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94000b0f10 0x7f9400151000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35134/0 (socket says 172.21.15.67:35134) 2024-03-24T00:40:41.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.059+0000 7f93feffd640 1 -- 172.21.15.67:0/3517039727 learned_addr learned my addr 172.21.15.67:0/3517039727 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:41.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.059+0000 7f93fe7fc640 1 --2- 172.21.15.67:0/3517039727 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f94000b2610 0x7f94000b6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:41.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.059+0000 7f93ff7fe640 1 --2- 172.21.15.67:0/3517039727 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f94000d20d0 0x7f94000c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:41.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.060+0000 7f93feffd640 1 -- 172.21.15.67:0/3517039727 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f94000d20d0 msgr2=0x7f94000c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:41.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.060+0000 7f93feffd640 1 --2- 172.21.15.67:0/3517039727 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f94000d20d0 0x7f94000c5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:41.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.060+0000 7f93feffd640 1 -- 172.21.15.67:0/3517039727 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f94000b2610 msgr2=0x7f94000b6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:41.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.060+0000 7f93feffd640 1 --2- 172.21.15.67:0/3517039727 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f94000b2610 0x7f94000b6ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:41.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.060+0000 7f93feffd640 1 -- 172.21.15.67:0/3517039727 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94000b5e40 con 0x7f94000b0f10 2024-03-24T00:40:41.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.060+0000 7f93ff7fe640 1 --2- 172.21.15.67:0/3517039727 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f94000d20d0 0x7f94000c5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:40:41.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.060+0000 7f93feffd640 1 --2- 172.21.15.67:0/3517039727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94000b0f10 0x7f9400151000 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f93f000e9b0 tx=0x7f93f000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:41.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.060+0000 7f93f77fe640 1 -- 172.21.15.67:0/3517039727 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f0017070 con 0x7f94000b0f10 2024-03-24T00:40:41.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.060+0000 7f93f77fe640 1 -- 172.21.15.67:0/3517039727 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93f000c9a0 con 0x7f94000b0f10 2024-03-24T00:40:41.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.060+0000 7f940553b640 1 -- 172.21.15.67:0/3517039727 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94000b6120 con 0x7f94000b0f10 2024-03-24T00:40:41.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.060+0000 7f93f77fe640 1 -- 172.21.15.67:0/3517039727 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f000cd90 con 0x7f94000b0f10 2024-03-24T00:40:41.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.060+0000 7f940553b640 1 -- 172.21.15.67:0/3517039727 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94000b66e0 con 0x7f94000b0f10 2024-03-24T00:40:41.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.061+0000 7f940553b640 1 -- 172.21.15.67:0/3517039727 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f940004f400 con 0x7f94000b0f10 2024-03-24T00:40:41.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.061+0000 7f93f77fe640 1 -- 172.21.15.67:0/3517039727 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f93f001a070 con 0x7f94000b0f10 2024-03-24T00:40:41.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.062+0000 7f93f77fe640 1 --2- 172.21.15.67:0/3517039727 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f93e006bbd0 0x7f93e006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:41.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.062+0000 7f93f77fe640 1 -- 172.21.15.67:0/3517039727 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f93f008ab70 con 0x7f94000b0f10 2024-03-24T00:40:41.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.062+0000 7f93fe7fc640 1 --2- 172.21.15.67:0/3517039727 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f93e006bbd0 0x7f93e006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:41.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.063+0000 7f93fe7fc640 1 --2- 172.21.15.67:0/3517039727 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f93e006bbd0 0x7f93e006e090 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f9400161740 tx=0x7f93e8007e40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:41.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.066+0000 7f93f77fe640 1 -- 172.21.15.67:0/3517039727 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f93f0058e80 con 0x7f94000b0f10 2024-03-24T00:40:41.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:41.288+0000 7f940553b640 1 -- 172.21.15.67:0/3517039727 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f94000b1810 con 0x7f94000b0f10 2024-03-24T00:40:42.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.301+0000 7f93f77fe640 1 -- 172.21.15.67:0/3517039727 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v25) v1 ==== 118+0+0 (secure 0 0 0) 0x7f93f005cf20 con 0x7f94000b0f10 2024-03-24T00:40:42.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.303+0000 7f940553b640 1 -- 172.21.15.67:0/3517039727 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f93e006bbd0 msgr2=0x7f93e006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:42.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.303+0000 7f940553b640 1 --2- 172.21.15.67:0/3517039727 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f93e006bbd0 0x7f93e006e090 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f9400161740 tx=0x7f93e8007e40 comp rx=0 tx=0).stop 2024-03-24T00:40:42.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.303+0000 7f940553b640 1 -- 172.21.15.67:0/3517039727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94000b0f10 msgr2=0x7f9400151000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:42.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.303+0000 7f940553b640 1 --2- 172.21.15.67:0/3517039727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94000b0f10 0x7f9400151000 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f93f000e9b0 tx=0x7f93f000ee70 comp rx=0 tx=0).stop 2024-03-24T00:40:42.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.303+0000 7f940553b640 1 -- 172.21.15.67:0/3517039727 shutdown_connections 2024-03-24T00:40:42.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.303+0000 7f940553b640 1 --2- 172.21.15.67:0/3517039727 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f93e006bbd0 0x7f93e006e090 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:42.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.303+0000 7f940553b640 1 --2- 172.21.15.67:0/3517039727 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f94000d20d0 0x7f94000c5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:42.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.303+0000 7f940553b640 1 --2- 172.21.15.67:0/3517039727 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f94000b2610 0x7f94000b6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:42.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.303+0000 7f940553b640 1 --2- 172.21.15.67:0/3517039727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94000b0f10 0x7f9400151000 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:42.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.303+0000 7f940553b640 1 -- 172.21.15.67:0/3517039727 >> 172.21.15.67:0/3517039727 conn(0x7f940008b1b0 msgr2=0x7f94000cf7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:42.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.303+0000 7f940553b640 1 -- 172.21.15.67:0/3517039727 shutdown_connections 2024-03-24T00:40:42.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.303+0000 7f940553b640 1 -- 172.21.15.67:0/3517039727 wait complete. 2024-03-24T00:40:42.315 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-24T00:40:42.315 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-24T00:40:42.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.418+0000 7f88a8e85640 1 Processor -- start 2024-03-24T00:40:42.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.418+0000 7f88a8e85640 1 -- start start 2024-03-24T00:40:42.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.419+0000 7f88a8e85640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f88a40b6170 0x7f88a4168c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:42.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.419+0000 7f88a8e85640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88a40d2250 con 0x7f88a415dea0 2024-03-24T00:40:42.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.419+0000 7f88a8e85640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88a40d2530 con 0x7f88a40d3ae0 2024-03-24T00:40:42.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.419+0000 7f88a8e85640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88a40d2810 con 0x7f88a40b6170 2024-03-24T00:40:42.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.419+0000 7f88a259c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f88a415dea0 0x7f88a40d33d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52580/0 (socket says 172.21.15.67:52580) 2024-03-24T00:40:42.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.419+0000 7f88a259c640 1 -- 172.21.15.67:0/4072825404 learned_addr learned my addr 172.21.15.67:0/4072825404 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:40:42.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.419+0000 7f88a2d9d640 1 --2- 172.21.15.67:0/4072825404 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f88a40b6170 0x7f88a4168c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:42.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.420+0000 7f88a159a640 1 -- 172.21.15.67:0/4072825404 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 130876061 0 0) 0x7f88a40d2250 con 0x7f88a415dea0 2024-03-24T00:40:42.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.420+0000 7f88a159a640 1 -- 172.21.15.67:0/4072825404 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f888c003520 con 0x7f88a415dea0 2024-03-24T00:40:42.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.420+0000 7f88a159a640 1 -- 172.21.15.67:0/4072825404 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3151763269 0 0) 0x7f88a40d2530 con 0x7f88a40d3ae0 2024-03-24T00:40:42.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.420+0000 7f88a159a640 1 -- 172.21.15.67:0/4072825404 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88a40d2250 con 0x7f88a40d3ae0 2024-03-24T00:40:42.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.420+0000 7f88a159a640 1 -- 172.21.15.67:0/4072825404 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 129325675 0 0) 0x7f888c003520 con 0x7f88a415dea0 2024-03-24T00:40:42.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.420+0000 7f88a2d9d640 1 -- 172.21.15.67:0/4072825404 >> v1:172.21.15.72:6789/0 conn(0x7f88a40d3ae0 legacy=0x7f88a40b5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:42.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.420+0000 7f88a2d9d640 1 -- 172.21.15.67:0/4072825404 >> v1:172.21.15.67:6789/0 conn(0x7f88a415dea0 legacy=0x7f88a40d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:42.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.420+0000 7f88a2d9d640 1 -- 172.21.15.67:0/4072825404 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a40d2af0 con 0x7f88a40b6170 2024-03-24T00:40:42.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.421+0000 7f88a2d9d640 1 --2- 172.21.15.67:0/4072825404 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f88a40b6170 0x7f88a4168c80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f889c009df0 tx=0x7f889c02bd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24c21ac9f3d455f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:42.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.421+0000 7f88a159a640 1 -- 172.21.15.67:0/4072825404 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f889c039070 con 0x7f88a40b6170 2024-03-24T00:40:42.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.421+0000 7f88a159a640 1 -- 172.21.15.67:0/4072825404 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f889c02c880 con 0x7f88a40b6170 2024-03-24T00:40:42.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.421+0000 7f88a159a640 1 -- 172.21.15.67:0/4072825404 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8894001670 con 0x7f88a40b6170 2024-03-24T00:40:42.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.421+0000 7f88a8e85640 1 -- 172.21.15.67:0/4072825404 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f88a40b6170 msgr2=0x7f88a4168c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:42.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.421+0000 7f88a8e85640 1 --2- 172.21.15.67:0/4072825404 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f88a40b6170 0x7f88a4168c80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f889c009df0 tx=0x7f889c02bd50 comp rx=0 tx=0).stop 2024-03-24T00:40:42.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.421+0000 7f88a8e85640 1 -- 172.21.15.67:0/4072825404 shutdown_connections 2024-03-24T00:40:42.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.421+0000 7f88a8e85640 1 --2- 172.21.15.67:0/4072825404 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f88a40b6170 0x7f88a4168c80 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:42.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.422+0000 7f88a8e85640 1 -- 172.21.15.67:0/4072825404 >> 172.21.15.67:0/4072825404 conn(0x7f88a4072da0 msgr2=0x7f88a40d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:42.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.422+0000 7f88a8e85640 1 -- 172.21.15.67:0/4072825404 shutdown_connections 2024-03-24T00:40:42.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.422+0000 7f88a8e85640 1 -- 172.21.15.67:0/4072825404 wait complete. 2024-03-24T00:40:42.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.423+0000 7f88a8e85640 1 Processor -- start 2024-03-24T00:40:42.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.423+0000 7f88a8e85640 1 -- start start 2024-03-24T00:40:42.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.423+0000 7f88a8e85640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f88a40b6170 0x7f88a415ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:42.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.423+0000 7f88a8e85640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f88a40d3ae0 0x7f88a4158430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:42.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.423+0000 7f88a8e85640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88a415dea0 0x7f88a41f7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:42.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.424+0000 7f88a8e85640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a415e280 con 0x7f88a415dea0 2024-03-24T00:40:42.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.424+0000 7f88a8e85640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a40b8310 con 0x7f88a40d3ae0 2024-03-24T00:40:42.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.424+0000 7f88a8e85640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88a415e400 con 0x7f88a40b6170 2024-03-24T00:40:42.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.424+0000 7f88a2d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88a415dea0 0x7f88a41f7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:42.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.424+0000 7f88a2d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88a415dea0 0x7f88a41f7330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35138/0 (socket says 172.21.15.67:35138) 2024-03-24T00:40:42.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.424+0000 7f88a2d9d640 1 -- 172.21.15.67:0/3411693955 learned_addr learned my addr 172.21.15.67:0/3411693955 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:42.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.424+0000 7f88a259c640 1 --2- 172.21.15.67:0/3411693955 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f88a40b6170 0x7f88a415ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:42.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.424+0000 7f88a1d9b640 1 --2- 172.21.15.67:0/3411693955 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f88a40d3ae0 0x7f88a4158430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:42.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.424+0000 7f88a2d9d640 1 -- 172.21.15.67:0/3411693955 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f88a40b6170 msgr2=0x7f88a415ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:42.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.424+0000 7f88a2d9d640 1 --2- 172.21.15.67:0/3411693955 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f88a40b6170 0x7f88a415ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:42.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.424+0000 7f88a2d9d640 1 -- 172.21.15.67:0/3411693955 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f88a40d3ae0 msgr2=0x7f88a4158430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:42.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.424+0000 7f88a2d9d640 1 --2- 172.21.15.67:0/3411693955 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f88a40d3ae0 0x7f88a4158430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:42.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.424+0000 7f88a2d9d640 1 -- 172.21.15.67:0/3411693955 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a415b540 con 0x7f88a415dea0 2024-03-24T00:40:42.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.424+0000 7f88a1d9b640 1 --2- 172.21.15.67:0/3411693955 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f88a40d3ae0 0x7f88a4158430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:40:42.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.424+0000 7f88a2d9d640 1 --2- 172.21.15.67:0/3411693955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88a415dea0 0x7f88a41f7330 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f889c037040 tx=0x7f889c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:42.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.424+0000 7f8892ffd640 1 -- 172.21.15.67:0/3411693955 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f889c039050 con 0x7f88a415dea0 2024-03-24T00:40:42.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.424+0000 7f8892ffd640 1 -- 172.21.15.67:0/3411693955 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f889c045070 con 0x7f88a415dea0 2024-03-24T00:40:42.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.424+0000 7f88a8e85640 1 -- 172.21.15.67:0/3411693955 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88a415b820 con 0x7f88a415dea0 2024-03-24T00:40:42.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.424+0000 7f8892ffd640 1 -- 172.21.15.67:0/3411693955 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f889c009450 con 0x7f88a415dea0 2024-03-24T00:40:42.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.424+0000 7f88a8e85640 1 -- 172.21.15.67:0/3411693955 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88a40cb430 con 0x7f88a415dea0 2024-03-24T00:40:42.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.425+0000 7f88a8e85640 1 -- 172.21.15.67:0/3411693955 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88a4068980 con 0x7f88a415dea0 2024-03-24T00:40:42.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.426+0000 7f8892ffd640 1 -- 172.21.15.67:0/3411693955 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f889c040690 con 0x7f88a415dea0 2024-03-24T00:40:42.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.426+0000 7f8892ffd640 1 --2- 172.21.15.67:0/3411693955 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f888406bbd0 0x7f888406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:42.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.426+0000 7f8892ffd640 1 -- 172.21.15.67:0/3411693955 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f889c0ad2b0 con 0x7f88a415dea0 2024-03-24T00:40:42.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.426+0000 7f88a259c640 1 --2- 172.21.15.67:0/3411693955 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f888406bbd0 0x7f888406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:42.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.427+0000 7f88a259c640 1 --2- 172.21.15.67:0/3411693955 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f888406bbd0 0x7f888406e090 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8894000fd0 tx=0x7f8894001150 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:42.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.435+0000 7f8892ffd640 1 -- 172.21.15.67:0/3411693955 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f88a4068980 con 0x7f88a415dea0 2024-03-24T00:40:42.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:42.656+0000 7f88a8e85640 1 -- 172.21.15.67:0/3411693955 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f88a404f400 con 0x7f88a415dea0 2024-03-24T00:40:43.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.305+0000 7f8892ffd640 1 -- 172.21.15.67:0/3411693955 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v26) v1 ==== 122+0+0 (secure 0 0 0) 0x7f889c07b6f0 con 0x7f88a415dea0 2024-03-24T00:40:43.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.307+0000 7f88a8e85640 1 -- 172.21.15.67:0/3411693955 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f888406bbd0 msgr2=0x7f888406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:43.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.307+0000 7f88a8e85640 1 --2- 172.21.15.67:0/3411693955 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f888406bbd0 0x7f888406e090 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8894000fd0 tx=0x7f8894001150 comp rx=0 tx=0).stop 2024-03-24T00:40:43.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.307+0000 7f88a8e85640 1 -- 172.21.15.67:0/3411693955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88a415dea0 msgr2=0x7f88a41f7330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:43.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.307+0000 7f88a8e85640 1 --2- 172.21.15.67:0/3411693955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88a415dea0 0x7f88a41f7330 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f889c037040 tx=0x7f889c004a40 comp rx=0 tx=0).stop 2024-03-24T00:40:43.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.308+0000 7f88a8e85640 1 -- 172.21.15.67:0/3411693955 shutdown_connections 2024-03-24T00:40:43.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.308+0000 7f88a8e85640 1 --2- 172.21.15.67:0/3411693955 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f888406bbd0 0x7f888406e090 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:43.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.308+0000 7f88a8e85640 1 --2- 172.21.15.67:0/3411693955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88a415dea0 0x7f88a41f7330 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:43.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.308+0000 7f88a8e85640 1 --2- 172.21.15.67:0/3411693955 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f88a40d3ae0 0x7f88a4158430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:43.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.308+0000 7f88a8e85640 1 --2- 172.21.15.67:0/3411693955 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f88a40b6170 0x7f88a415ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:43.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.308+0000 7f88a8e85640 1 -- 172.21.15.67:0/3411693955 >> 172.21.15.67:0/3411693955 conn(0x7f88a4072da0 msgr2=0x7f88a40b59f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:43.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.308+0000 7f88a8e85640 1 -- 172.21.15.67:0/3411693955 shutdown_connections 2024-03-24T00:40:43.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.308+0000 7f88a8e85640 1 -- 172.21.15.67:0/3411693955 wait complete. 2024-03-24T00:40:43.319 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-24T00:40:43.319 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:40:43.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.421+0000 7ff4f28f1640 1 Processor -- start 2024-03-24T00:40:43.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.421+0000 7ff4f28f1640 1 -- start start 2024-03-24T00:40:43.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.421+0000 7ff4f28f1640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff4ec15dec0 0x7ff4ec0ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:43.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.421+0000 7ff4f28f1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff4ec15e2a0 0x7ff4ec162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:43.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.421+0000 7ff4f28f1640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4ec0cf640 con 0x7ff4ec0d07e0 2024-03-24T00:40:43.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.421+0000 7ff4f28f1640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4ec0cee30 con 0x7ff4ec15dec0 2024-03-24T00:40:43.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.421+0000 7ff4f28f1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ec0d2120 con 0x7ff4ec15e2a0 2024-03-24T00:40:43.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.422+0000 7ff4f0e8e640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff4ec0d07e0 0x7ff4ec0b5850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52582/0 (socket says 172.21.15.67:52582) 2024-03-24T00:40:43.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.422+0000 7ff4f0e8e640 1 -- 172.21.15.67:0/3378389507 learned_addr learned my addr 172.21.15.67:0/3378389507 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:40:43.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.422+0000 7ff4e3fff640 1 --2- 172.21.15.67:0/3378389507 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff4ec15e2a0 0x7ff4ec162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:43.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.422+0000 7ff4ebfff640 1 --2- 172.21.15.67:0/3378389507 >> v2:172.21.15.72:3300/0 conn(0x7ff4ec15dec0 0x7ff4ec0ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:43.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.422+0000 7ff4eb7fe640 1 -- 172.21.15.67:0/3378389507 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1921092123 0 0) 0x7ff4e4001490 con 0x7ff4ec0d07e0 2024-03-24T00:40:43.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.422+0000 7ff4eb7fe640 1 -- 172.21.15.67:0/3378389507 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4d8003520 con 0x7ff4ec0d07e0 2024-03-24T00:40:43.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.422+0000 7ff4eb7fe640 1 -- 172.21.15.67:0/3378389507 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3811790789 0 0) 0x7ff4d8003520 con 0x7ff4ec0d07e0 2024-03-24T00:40:43.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.422+0000 7ff4eb7fe640 1 -- 172.21.15.67:0/3378389507 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff4ec15e2a0 msgr2=0x7ff4ec162630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:43.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.422+0000 7ff4eb7fe640 1 --2- 172.21.15.67:0/3378389507 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff4ec15e2a0 0x7ff4ec162630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:43.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.422+0000 7ff4eb7fe640 1 -- 172.21.15.67:0/3378389507 >> v2:172.21.15.72:3300/0 conn(0x7ff4ec15dec0 msgr2=0x7ff4ec0ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:43.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.422+0000 7ff4eb7fe640 1 --2- 172.21.15.67:0/3378389507 >> v2:172.21.15.72:3300/0 conn(0x7ff4ec15dec0 0x7ff4ec0ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:43.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.422+0000 7ff4eb7fe640 1 -- 172.21.15.67:0/3378389507 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4ec0cf080 con 0x7ff4ec0d07e0 2024-03-24T00:40:43.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.422+0000 7ff4eb7fe640 1 -- 172.21.15.67:0/3378389507 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7ff4e4001bc0 con 0x7ff4ec0d07e0 2024-03-24T00:40:43.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.422+0000 7ff4e3fff640 1 --2- 172.21.15.67:0/3378389507 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff4ec15e2a0 0x7ff4ec162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:40:43.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.422+0000 7ff4eb7fe640 1 -- 172.21.15.67:0/3378389507 >> v1:172.21.15.67:6789/0 conn(0x7ff4ec0d07e0 legacy=0x7ff4ec0b5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:43.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.422+0000 7ff4eb7fe640 1 --2- 172.21.15.67:0/3378389507 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff4ec15e2a0 0x7ff4d80039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:43.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.422+0000 7ff4eb7fe640 1 --2- 172.21.15.67:0/3378389507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4d8003f10 0x7ff4d8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:43.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.422+0000 7ff4eb7fe640 1 --2- 172.21.15.67:0/3378389507 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff4d8004d40 0x7ff4d80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:43.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.422+0000 7ff4eb7fe640 1 -- 172.21.15.67:0/3378389507 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ec0cee30 con 0x7ff4d8003f10 2024-03-24T00:40:43.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.422+0000 7ff4eb7fe640 1 -- 172.21.15.67:0/3378389507 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ec0d2120 con 0x7ff4d8004d40 2024-03-24T00:40:43.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.422+0000 7ff4eb7fe640 1 -- 172.21.15.67:0/3378389507 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d800b9f0 con 0x7ff4ec15e2a0 2024-03-24T00:40:43.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.422+0000 7ff4ebfff640 1 --2- 172.21.15.67:0/3378389507 >> v2:172.21.15.72:3300/0 conn(0x7ff4ec15dec0 0x7ff4ec0ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:40:43.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.422+0000 7ff4e3fff640 1 --2- 172.21.15.67:0/3378389507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4d8003f10 0x7ff4d8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:43.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.422+0000 7ff4f0e8e640 1 --2- 172.21.15.67:0/3378389507 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff4d8004d40 0x7ff4d80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:43.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.423+0000 7ff4ebfff640 1 --2- 172.21.15.67:0/3378389507 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff4ec15e2a0 0x7ff4d80039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:43.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.423+0000 7ff4e3fff640 1 -- 172.21.15.67:0/3378389507 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff4ec15e2a0 msgr2=0x7ff4d80039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:43.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.423+0000 7ff4e3fff640 1 --2- 172.21.15.67:0/3378389507 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff4ec15e2a0 0x7ff4d80039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:43.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.423+0000 7ff4e3fff640 1 -- 172.21.15.67:0/3378389507 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff4d8004d40 msgr2=0x7ff4d80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:43.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.423+0000 7ff4e3fff640 1 --2- 172.21.15.67:0/3378389507 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff4d8004d40 0x7ff4d80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:43.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.423+0000 7ff4e3fff640 1 -- 172.21.15.67:0/3378389507 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d8003520 con 0x7ff4d8003f10 2024-03-24T00:40:43.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.423+0000 7ff4f0e8e640 1 --2- 172.21.15.67:0/3378389507 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff4d8004d40 0x7ff4d80051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:40:43.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.424+0000 7ff4e3fff640 1 --2- 172.21.15.67:0/3378389507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4d8003f10 0x7ff4d8004800 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7ff4d4009c60 tx=0x7ff4d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc464ae664a86871 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:43.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.424+0000 7ff4eb7fe640 1 -- 172.21.15.67:0/3378389507 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d4039070 con 0x7ff4d8003f10 2024-03-24T00:40:43.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.424+0000 7ff4eb7fe640 1 -- 172.21.15.67:0/3378389507 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4d402c8b0 con 0x7ff4d8003f10 2024-03-24T00:40:43.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.424+0000 7ff4eb7fe640 1 -- 172.21.15.67:0/3378389507 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d4009860 con 0x7ff4d8003f10 2024-03-24T00:40:43.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.425+0000 7ff4f28f1640 1 -- 172.21.15.67:0/3378389507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4d8003f10 msgr2=0x7ff4d8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:43.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.425+0000 7ff4f28f1640 1 --2- 172.21.15.67:0/3378389507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4d8003f10 0x7ff4d8004800 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7ff4d4009c60 tx=0x7ff4d8005830 comp rx=0 tx=0).stop 2024-03-24T00:40:43.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.425+0000 7ff4f0e8e640 1 -- 172.21.15.67:0/3378389507 reap_dead start 2024-03-24T00:40:43.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.425+0000 7ff4f28f1640 1 -- 172.21.15.67:0/3378389507 shutdown_connections 2024-03-24T00:40:43.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.425+0000 7ff4f28f1640 1 -- 172.21.15.67:0/3378389507 >> 172.21.15.67:0/3378389507 conn(0x7ff4ec06c810 msgr2=0x7ff4ec06cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:43.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.425+0000 7ff4f28f1640 1 -- 172.21.15.67:0/3378389507 shutdown_connections 2024-03-24T00:40:43.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.425+0000 7ff4f28f1640 1 -- 172.21.15.67:0/3378389507 wait complete. 2024-03-24T00:40:43.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.427+0000 7ff4f28f1640 1 Processor -- start 2024-03-24T00:40:43.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.427+0000 7ff4f28f1640 1 -- start start 2024-03-24T00:40:43.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.428+0000 7ff4f28f1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff4d8003f10 0x7ff4ec155e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:43.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.428+0000 7ff4f28f1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff4ec0b5610 0x7ff4ec0b5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:43.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.428+0000 7ff4f28f1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4ec0b3f90 0x7ff4ec0b4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:43.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.428+0000 7ff4f28f1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ec0d3740 con 0x7ff4ec0b3f90 2024-03-24T00:40:43.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.428+0000 7ff4f28f1640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ec0d35c0 con 0x7ff4d8003f10 2024-03-24T00:40:43.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.428+0000 7ff4f28f1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ec0d38c0 con 0x7ff4ec0b5610 2024-03-24T00:40:43.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.428+0000 7ff4f0e8e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4ec0b3f90 0x7ff4ec0b4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:43.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.428+0000 7ff4f0e8e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4ec0b3f90 0x7ff4ec0b4440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35146/0 (socket says 172.21.15.67:35146) 2024-03-24T00:40:43.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.428+0000 7ff4f0e8e640 1 -- 172.21.15.67:0/1916010439 learned_addr learned my addr 172.21.15.67:0/1916010439 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:43.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.428+0000 7ff4ebfff640 1 --2- 172.21.15.67:0/1916010439 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff4d8003f10 0x7ff4ec155e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:43.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.428+0000 7ff4e3fff640 1 --2- 172.21.15.67:0/1916010439 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff4ec0b5610 0x7ff4ec0b5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:43.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.428+0000 7ff4f0e8e640 1 -- 172.21.15.67:0/1916010439 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff4ec0b5610 msgr2=0x7ff4ec0b5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:43.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.428+0000 7ff4f0e8e640 1 --2- 172.21.15.67:0/1916010439 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff4ec0b5610 0x7ff4ec0b5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:43.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.428+0000 7ff4f0e8e640 1 -- 172.21.15.67:0/1916010439 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff4d8003f10 msgr2=0x7ff4ec155e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:43.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.428+0000 7ff4f0e8e640 1 --2- 172.21.15.67:0/1916010439 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff4d8003f10 0x7ff4ec155e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:43.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.428+0000 7ff4f0e8e640 1 -- 172.21.15.67:0/1916010439 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4ec156370 con 0x7ff4ec0b3f90 2024-03-24T00:40:43.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.428+0000 7ff4f0e8e640 1 --2- 172.21.15.67:0/1916010439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4ec0b3f90 0x7ff4ec0b4440 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7ff4ec0b37e0 tx=0x7ff4e400a550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:43.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.429+0000 7ff4e97fa640 1 -- 172.21.15.67:0/1916010439 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e400abb0 con 0x7ff4ec0b3f90 2024-03-24T00:40:43.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.429+0000 7ff4e97fa640 1 -- 172.21.15.67:0/1916010439 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4e4018070 con 0x7ff4ec0b3f90 2024-03-24T00:40:43.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.429+0000 7ff4f28f1640 1 -- 172.21.15.67:0/1916010439 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4ec150680 con 0x7ff4ec0b3f90 2024-03-24T00:40:43.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.429+0000 7ff4e97fa640 1 -- 172.21.15.67:0/1916010439 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e4013420 con 0x7ff4ec0b3f90 2024-03-24T00:40:43.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.429+0000 7ff4f28f1640 1 -- 172.21.15.67:0/1916010439 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4ec1538d0 con 0x7ff4ec0b3f90 2024-03-24T00:40:43.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.430+0000 7ff4f28f1640 1 -- 172.21.15.67:0/1916010439 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4ec04f400 con 0x7ff4ec0b3f90 2024-03-24T00:40:43.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.430+0000 7ff4e97fa640 1 -- 172.21.15.67:0/1916010439 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff4e40135c0 con 0x7ff4ec0b3f90 2024-03-24T00:40:43.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.430+0000 7ff4e97fa640 1 --2- 172.21.15.67:0/1916010439 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff4cc06bc00 0x7ff4cc06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:43.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.431+0000 7ff4e97fa640 1 -- 172.21.15.67:0/1916010439 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff4e408b010 con 0x7ff4ec0b3f90 2024-03-24T00:40:43.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.431+0000 7ff4ebfff640 1 --2- 172.21.15.67:0/1916010439 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff4cc06bc00 0x7ff4cc06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:43.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.431+0000 7ff4ebfff640 1 --2- 172.21.15.67:0/1916010439 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff4cc06bc00 0x7ff4cc06e0c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc009c20 tx=0x7ff4dc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:43.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.435+0000 7ff4e97fa640 1 -- 172.21.15.67:0/1916010439 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff4ec0cf640 con 0x7ff4ec0b3f90 2024-03-24T00:40:43.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.655+0000 7ff4f28f1640 1 -- 172.21.15.67:0/1916010439 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff4ec0d1aa0 con 0x7ff4ec0b3f90 2024-03-24T00:40:43.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.655+0000 7ff4e97fa640 1 -- 172.21.15.67:0/1916010439 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4977 (secure 0 0 0) 0x7ff4e4059970 con 0x7ff4ec0b3f90 2024-03-24T00:40:43.656 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:40:43.656 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5360,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3312887486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":3312887486},{"type":"v1","addr":"172.21.15.67:6835","nonce":3312887486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5821,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/4171866637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":4171866637},{"type":"v1","addr":"172.21.15.72:6836","nonce":4171866637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5824,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/4106412452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":4106412452},{"type":"v1","addr":"172.21.15.72:6837","nonce":4106412452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:40:39.649631+0000","modified":"2024-03-24T00:40:42.657888+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5853},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5853":{"gid":5853,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6834/2741818214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2741818214},{"type":"v1","addr":"172.21.15.67:6834","nonce":2741818214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5853,"qdb_cluster":[5853]},"id":3}]} 2024-03-24T00:40:43.657 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 26 2024-03-24T00:40:43.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.657+0000 7ff4f28f1640 1 -- 172.21.15.67:0/1916010439 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff4cc06bc00 msgr2=0x7ff4cc06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:43.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.657+0000 7ff4f28f1640 1 --2- 172.21.15.67:0/1916010439 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff4cc06bc00 0x7ff4cc06e0c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc009c20 tx=0x7ff4dc008040 comp rx=0 tx=0).stop 2024-03-24T00:40:43.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.657+0000 7ff4f28f1640 1 -- 172.21.15.67:0/1916010439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4ec0b3f90 msgr2=0x7ff4ec0b4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:43.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.657+0000 7ff4f28f1640 1 --2- 172.21.15.67:0/1916010439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4ec0b3f90 0x7ff4ec0b4440 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7ff4ec0b37e0 tx=0x7ff4e400a550 comp rx=0 tx=0).stop 2024-03-24T00:40:43.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.657+0000 7ff4f28f1640 1 -- 172.21.15.67:0/1916010439 shutdown_connections 2024-03-24T00:40:43.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.657+0000 7ff4f28f1640 1 --2- 172.21.15.67:0/1916010439 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff4cc06bc00 0x7ff4cc06e0c0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:43.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.657+0000 7ff4f28f1640 1 --2- 172.21.15.67:0/1916010439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4ec0b3f90 0x7ff4ec0b4440 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:43.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.657+0000 7ff4f28f1640 1 --2- 172.21.15.67:0/1916010439 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff4ec0b5610 0x7ff4ec0b5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:43.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.657+0000 7ff4f28f1640 1 --2- 172.21.15.67:0/1916010439 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff4d8003f10 0x7ff4ec155e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:43.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.657+0000 7ff4f28f1640 1 -- 172.21.15.67:0/1916010439 >> 172.21.15.67:0/1916010439 conn(0x7ff4ec06c810 msgr2=0x7ff4ec15f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:43.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.657+0000 7ff4f28f1640 1 -- 172.21.15.67:0/1916010439 shutdown_connections 2024-03-24T00:40:43.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.657+0000 7ff4f28f1640 1 -- 172.21.15.67:0/1916010439 wait complete. 2024-03-24T00:40:43.669 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:40:43.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.773+0000 7f9db9ddf640 1 Processor -- start 2024-03-24T00:40:43.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.773+0000 7f9db9ddf640 1 -- start start 2024-03-24T00:40:43.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.774+0000 7f9db9ddf640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f9db415dec0 0x7f9db40ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:43.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.774+0000 7f9db9ddf640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9db40d0ef0 0x7f9db4162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:43.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.774+0000 7f9db9ddf640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9db40cf640 con 0x7f9db415e2a0 2024-03-24T00:40:43.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.774+0000 7f9db9ddf640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9db40cee30 con 0x7f9db40d0ef0 2024-03-24T00:40:43.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.774+0000 7f9db9ddf640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9db40d2120 con 0x7f9db415dec0 2024-03-24T00:40:43.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.774+0000 7f9db3fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9db40d0ef0 0x7f9db4162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:43.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.774+0000 7f9db3fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9db40d0ef0 0x7f9db4162630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40730/0 (socket says 172.21.15.67:40730) 2024-03-24T00:40:43.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.774+0000 7f9db3fff640 1 -- 172.21.15.67:0/517508960 learned_addr learned my addr 172.21.15.67:0/517508960 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:43.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.774+0000 7f9db37fe640 1 --2- 172.21.15.67:0/517508960 >> v2:172.21.15.72:3300/0 conn(0x7f9db415dec0 0x7f9db40ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:43.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.774+0000 7f9db3fff640 1 -- 172.21.15.67:0/517508960 >> v2:172.21.15.72:3300/0 conn(0x7f9db415dec0 msgr2=0x7f9db40ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:43.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.774+0000 7f9db3fff640 1 --2- 172.21.15.67:0/517508960 >> v2:172.21.15.72:3300/0 conn(0x7f9db415dec0 0x7f9db40ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:43.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.774+0000 7f9db3fff640 1 -- 172.21.15.67:0/517508960 >> v1:172.21.15.72:6789/0 conn(0x7f9db415e2a0 legacy=0x7f9db40d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:43.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.774+0000 7f9db3fff640 1 -- 172.21.15.67:0/517508960 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9db40cf080 con 0x7f9db40d0ef0 2024-03-24T00:40:43.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.775+0000 7f9db3fff640 1 --2- 172.21.15.67:0/517508960 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9db40d0ef0 0x7f9db4162630 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f9db40d14a0 tx=0x7f9dac02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3caeac3ceca1c22d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:43.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.775+0000 7f9db27fc640 1 -- 172.21.15.67:0/517508960 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dac02c9f0 con 0x7f9db40d0ef0 2024-03-24T00:40:43.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.775+0000 7f9db27fc640 1 -- 172.21.15.67:0/517508960 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dac02cb90 con 0x7f9db40d0ef0 2024-03-24T00:40:43.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.775+0000 7f9db27fc640 1 -- 172.21.15.67:0/517508960 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dac0343f0 con 0x7f9db40d0ef0 2024-03-24T00:40:43.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.776+0000 7f9db9ddf640 1 -- 172.21.15.67:0/517508960 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9db40d0ef0 msgr2=0x7f9db4162630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:43.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.776+0000 7f9db9ddf640 1 --2- 172.21.15.67:0/517508960 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9db40d0ef0 0x7f9db4162630 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f9db40d14a0 tx=0x7f9dac02be50 comp rx=0 tx=0).stop 2024-03-24T00:40:43.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.776+0000 7f9db9ddf640 1 -- 172.21.15.67:0/517508960 shutdown_connections 2024-03-24T00:40:43.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.776+0000 7f9db9ddf640 1 --2- 172.21.15.67:0/517508960 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9db40d0ef0 0x7f9db4162630 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:43.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.776+0000 7f9db9ddf640 1 --2- 172.21.15.67:0/517508960 >> v2:172.21.15.72:3300/0 conn(0x7f9db415dec0 0x7f9db40ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:43.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.776+0000 7f9db9ddf640 1 -- 172.21.15.67:0/517508960 >> 172.21.15.67:0/517508960 conn(0x7f9db406c810 msgr2=0x7f9db406cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:43.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.776+0000 7f9db9ddf640 1 -- 172.21.15.67:0/517508960 shutdown_connections 2024-03-24T00:40:43.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.776+0000 7f9db9ddf640 1 -- 172.21.15.67:0/517508960 wait complete. 2024-03-24T00:40:43.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.777+0000 7f9db9ddf640 1 Processor -- start 2024-03-24T00:40:43.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.777+0000 7f9db9ddf640 1 -- start start 2024-03-24T00:40:43.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.779+0000 7f9db9ddf640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9db40d0ef0 0x7f9db415b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:43.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.779+0000 7f9db9ddf640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9db415dec0 0x7f9db40b7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:43.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.779+0000 7f9db9ddf640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9db415e2a0 0x7f9db40b7540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:43.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.779+0000 7f9db9ddf640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db40d2730 con 0x7f9db415e2a0 2024-03-24T00:40:43.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.779+0000 7f9db9ddf640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db40d25b0 con 0x7f9db40d0ef0 2024-03-24T00:40:43.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.779+0000 7f9db9ddf640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9db40d28b0 con 0x7f9db415dec0 2024-03-24T00:40:43.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.779+0000 7f9db3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9db415e2a0 0x7f9db40b7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:43.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.779+0000 7f9db3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9db415e2a0 0x7f9db40b7540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40734/0 (socket says 172.21.15.67:40734) 2024-03-24T00:40:43.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.779+0000 7f9db3fff640 1 -- 172.21.15.67:0/907263626 learned_addr learned my addr 172.21.15.67:0/907263626 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:43.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.779+0000 7f9db37fe640 1 --2- 172.21.15.67:0/907263626 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9db40d0ef0 0x7f9db415b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:43.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.779+0000 7f9db2ffd640 1 --2- 172.21.15.67:0/907263626 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9db415dec0 0x7f9db40b7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:43.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.779+0000 7f9db3fff640 1 -- 172.21.15.67:0/907263626 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9db415dec0 msgr2=0x7f9db40b7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:43.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.779+0000 7f9db3fff640 1 --2- 172.21.15.67:0/907263626 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9db415dec0 0x7f9db40b7000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:43.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.779+0000 7f9db3fff640 1 -- 172.21.15.67:0/907263626 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9db40d0ef0 msgr2=0x7f9db415b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:43.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.779+0000 7f9db3fff640 1 --2- 172.21.15.67:0/907263626 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9db40d0ef0 0x7f9db415b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:43.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.779+0000 7f9db3fff640 1 -- 172.21.15.67:0/907263626 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9db40ac0f0 con 0x7f9db415e2a0 2024-03-24T00:40:43.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.779+0000 7f9db3fff640 1 --2- 172.21.15.67:0/907263626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9db415e2a0 0x7f9db40b7540 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f9dac036040 tx=0x7f9dac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:43.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.779+0000 7f9d9bfff640 1 -- 172.21.15.67:0/907263626 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dac043020 con 0x7f9db415e2a0 2024-03-24T00:40:43.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.780+0000 7f9d9bfff640 1 -- 172.21.15.67:0/907263626 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dac03e450 con 0x7f9db415e2a0 2024-03-24T00:40:43.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.780+0000 7f9db9ddf640 1 -- 172.21.15.67:0/907263626 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9db40b4860 con 0x7f9db415e2a0 2024-03-24T00:40:43.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.780+0000 7f9d9bfff640 1 -- 172.21.15.67:0/907263626 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dac009510 con 0x7f9db415e2a0 2024-03-24T00:40:43.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.780+0000 7f9db9ddf640 1 -- 172.21.15.67:0/907263626 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9db40b4e20 con 0x7f9db415e2a0 2024-03-24T00:40:43.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.781+0000 7f9db9ddf640 1 -- 172.21.15.67:0/907263626 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9db404f400 con 0x7f9db415e2a0 2024-03-24T00:40:43.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.781+0000 7f9d9bfff640 1 -- 172.21.15.67:0/907263626 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9dac046020 con 0x7f9db415e2a0 2024-03-24T00:40:43.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.782+0000 7f9d9bfff640 1 --2- 172.21.15.67:0/907263626 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9da006bbd0 0x7f9da006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:43.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.782+0000 7f9d9bfff640 1 -- 172.21.15.67:0/907263626 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9dac0ad210 con 0x7f9db415e2a0 2024-03-24T00:40:43.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.782+0000 7f9db37fe640 1 --2- 172.21.15.67:0/907263626 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9da006bbd0 0x7f9da006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:43.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.782+0000 7f9db37fe640 1 --2- 172.21.15.67:0/907263626 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9da006bbd0 0x7f9da006e090 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f9da8006fd0 tx=0x7f9da8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:43.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.790+0000 7f9d9bfff640 1 -- 172.21.15.67:0/907263626 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9dac04d030 con 0x7f9db415e2a0 2024-03-24T00:40:43.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.972+0000 7f9db9ddf640 1 -- 172.21.15.67:0/907263626 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9db40b9c70 con 0x7f9db415e2a0 2024-03-24T00:40:43.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.974+0000 7f9d9bfff640 1 -- 172.21.15.67:0/907263626 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) v1 ==== 74+0+16882 (secure 0 0 0) 0x7f9db40cf640 con 0x7f9db415e2a0 2024-03-24T00:40:43.975 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:40:43.975 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":50,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:40:41.288503+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:40:37.927587+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-24T00:40:38.616521+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:40:43.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.976+0000 7f9db9ddf640 1 -- 172.21.15.67:0/907263626 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9da006bbd0 msgr2=0x7f9da006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:43.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.976+0000 7f9db9ddf640 1 --2- 172.21.15.67:0/907263626 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9da006bbd0 0x7f9da006e090 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f9da8006fd0 tx=0x7f9da8008040 comp rx=0 tx=0).stop 2024-03-24T00:40:43.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.977+0000 7f9db9ddf640 1 -- 172.21.15.67:0/907263626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9db415e2a0 msgr2=0x7f9db40b7540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:43.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.977+0000 7f9db9ddf640 1 --2- 172.21.15.67:0/907263626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9db415e2a0 0x7f9db40b7540 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f9dac036040 tx=0x7f9dac004a40 comp rx=0 tx=0).stop 2024-03-24T00:40:43.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.977+0000 7f9db9ddf640 1 -- 172.21.15.67:0/907263626 shutdown_connections 2024-03-24T00:40:43.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.977+0000 7f9db9ddf640 1 --2- 172.21.15.67:0/907263626 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9da006bbd0 0x7f9da006e090 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:43.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.977+0000 7f9db9ddf640 1 --2- 172.21.15.67:0/907263626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9db415e2a0 0x7f9db40b7540 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:43.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.977+0000 7f9db9ddf640 1 --2- 172.21.15.67:0/907263626 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9db415dec0 0x7f9db40b7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:43.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.977+0000 7f9db9ddf640 1 --2- 172.21.15.67:0/907263626 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9db40d0ef0 0x7f9db415b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:43.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.977+0000 7f9db9ddf640 1 -- 172.21.15.67:0/907263626 >> 172.21.15.67:0/907263626 conn(0x7f9db406c810 msgr2=0x7f9db406ce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:43.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.977+0000 7f9db9ddf640 1 -- 172.21.15.67:0/907263626 shutdown_connections 2024-03-24T00:40:43.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:43.977+0000 7f9db9ddf640 1 -- 172.21.15.67:0/907263626 wait complete. 2024-03-24T00:40:43.988 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-24T00:40:43.989 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:40:44.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.091+0000 7f1d64f09640 1 Processor -- start 2024-03-24T00:40:44.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.091+0000 7f1d64f09640 1 -- start start 2024-03-24T00:40:44.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.092+0000 7f1d64f09640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1d6015dec0 0x7f1d600ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:44.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.092+0000 7f1d64f09640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1d6015e2a0 0x7f1d60162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:44.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.092+0000 7f1d64f09640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1d600d07e0 0x7f1d600d0c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:44.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.092+0000 7f1d64f09640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d600d1190 con 0x7f1d600d07e0 2024-03-24T00:40:44.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.092+0000 7f1d64f09640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d600cee30 con 0x7f1d6015e2a0 2024-03-24T00:40:44.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.092+0000 7f1d64f09640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d600d2120 con 0x7f1d6015dec0 2024-03-24T00:40:44.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.092+0000 7f1d5ed9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1d600d07e0 0x7f1d600d0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:44.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.092+0000 7f1d5ed9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1d600d07e0 0x7f1d600d0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40746/0 (socket says 172.21.15.67:40746) 2024-03-24T00:40:44.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.092+0000 7f1d5ed9d640 1 -- 172.21.15.67:0/440522893 learned_addr learned my addr 172.21.15.67:0/440522893 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:44.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.092+0000 7f1d57fff640 1 --2- 172.21.15.67:0/440522893 >> v2:172.21.15.72:3300/0 conn(0x7f1d6015e2a0 0x7f1d60162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:44.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.092+0000 7f1d5e59c640 1 --2- 172.21.15.67:0/440522893 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1d6015dec0 0x7f1d600ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:44.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.092+0000 7f1d5ed9d640 1 -- 172.21.15.67:0/440522893 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1d6015dec0 msgr2=0x7f1d600ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:44.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.092+0000 7f1d5ed9d640 1 --2- 172.21.15.67:0/440522893 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1d6015dec0 0x7f1d600ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:44.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.092+0000 7f1d5ed9d640 1 -- 172.21.15.67:0/440522893 >> v2:172.21.15.72:3300/0 conn(0x7f1d6015e2a0 msgr2=0x7f1d60162630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:44.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.092+0000 7f1d5ed9d640 1 --2- 172.21.15.67:0/440522893 >> v2:172.21.15.72:3300/0 conn(0x7f1d6015e2a0 0x7f1d60162630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:44.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.092+0000 7f1d5ed9d640 1 -- 172.21.15.67:0/440522893 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d600cf640 con 0x7f1d600d07e0 2024-03-24T00:40:44.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.093+0000 7f1d5ed9d640 1 --2- 172.21.15.67:0/440522893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d600d07e0 0x7f1d600d0c50 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f1d600d4e30 tx=0x7f1d5802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dff91acb36c65e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:44.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.093+0000 7f1d5dd9b640 1 -- 172.21.15.67:0/440522893 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d5802c9e0 con 0x7f1d600d07e0 2024-03-24T00:40:44.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.093+0000 7f1d5dd9b640 1 -- 172.21.15.67:0/440522893 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d5802cb80 con 0x7f1d600d07e0 2024-03-24T00:40:44.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.093+0000 7f1d5dd9b640 1 -- 172.21.15.67:0/440522893 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d580343f0 con 0x7f1d600d07e0 2024-03-24T00:40:44.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.094+0000 7f1d64f09640 1 -- 172.21.15.67:0/440522893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d600d07e0 msgr2=0x7f1d600d0c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:44.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.094+0000 7f1d64f09640 1 --2- 172.21.15.67:0/440522893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d600d07e0 0x7f1d600d0c50 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f1d600d4e30 tx=0x7f1d5802be40 comp rx=0 tx=0).stop 2024-03-24T00:40:44.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.094+0000 7f1d64f09640 1 -- 172.21.15.67:0/440522893 shutdown_connections 2024-03-24T00:40:44.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.094+0000 7f1d64f09640 1 --2- 172.21.15.67:0/440522893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d600d07e0 0x7f1d600d0c50 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:44.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.094+0000 7f1d64f09640 1 --2- 172.21.15.67:0/440522893 >> v2:172.21.15.72:3300/0 conn(0x7f1d6015e2a0 0x7f1d60162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:44.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.094+0000 7f1d64f09640 1 --2- 172.21.15.67:0/440522893 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1d6015dec0 0x7f1d600ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:44.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.094+0000 7f1d64f09640 1 -- 172.21.15.67:0/440522893 >> 172.21.15.67:0/440522893 conn(0x7f1d6006c810 msgr2=0x7f1d6006cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:44.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.094+0000 7f1d64f09640 1 -- 172.21.15.67:0/440522893 shutdown_connections 2024-03-24T00:40:44.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.094+0000 7f1d64f09640 1 -- 172.21.15.67:0/440522893 wait complete. 2024-03-24T00:40:44.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.095+0000 7f1d64f09640 1 Processor -- start 2024-03-24T00:40:44.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.095+0000 7f1d64f09640 1 -- start start 2024-03-24T00:40:44.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.096+0000 7f1d64f09640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1d600d07e0 0x7f1d60153e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:44.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.096+0000 7f1d64f09640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d6015dec0 0x7f1d6015feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:44.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.096+0000 7f1d64f09640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1d6015e2a0 0x7f1d60158030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:44.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.096+0000 7f1d64f09640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d600b5850 con 0x7f1d6015dec0 2024-03-24T00:40:44.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.096+0000 7f1d64f09640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d600cf360 con 0x7f1d6015e2a0 2024-03-24T00:40:44.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.096+0000 7f1d64f09640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d600b59d0 con 0x7f1d600d07e0 2024-03-24T00:40:44.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.096+0000 7f1d57fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d6015dec0 0x7f1d6015feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:44.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.096+0000 7f1d57fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d6015dec0 0x7f1d6015feb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40760/0 (socket says 172.21.15.67:40760) 2024-03-24T00:40:44.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.096+0000 7f1d57fff640 1 -- 172.21.15.67:0/4006683228 learned_addr learned my addr 172.21.15.67:0/4006683228 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:44.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.096+0000 7f1d5e59c640 1 --2- 172.21.15.67:0/4006683228 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1d600d07e0 0x7f1d60153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:44.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.096+0000 7f1d5ed9d640 1 --2- 172.21.15.67:0/4006683228 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1d6015e2a0 0x7f1d60158030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:44.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.096+0000 7f1d57fff640 1 -- 172.21.15.67:0/4006683228 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1d600d07e0 msgr2=0x7f1d60153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:44.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.096+0000 7f1d57fff640 1 --2- 172.21.15.67:0/4006683228 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1d600d07e0 0x7f1d60153e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:44.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.096+0000 7f1d57fff640 1 -- 172.21.15.67:0/4006683228 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1d6015e2a0 msgr2=0x7f1d60158030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:44.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.096+0000 7f1d57fff640 1 --2- 172.21.15.67:0/4006683228 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1d6015e2a0 0x7f1d60158030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:44.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.096+0000 7f1d57fff640 1 -- 172.21.15.67:0/4006683228 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d601e8850 con 0x7f1d6015dec0 2024-03-24T00:40:44.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.096+0000 7f1d5e59c640 1 --2- 172.21.15.67:0/4006683228 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1d600d07e0 0x7f1d60153e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:40:44.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.096+0000 7f1d5ed9d640 1 --2- 172.21.15.67:0/4006683228 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1d6015e2a0 0x7f1d60158030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:40:44.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.097+0000 7f1d57fff640 1 --2- 172.21.15.67:0/4006683228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d6015dec0 0x7f1d6015feb0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f1d4800d5c0 tx=0x7f1d4800da80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:44.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.097+0000 7f1d56ffd640 1 -- 172.21.15.67:0/4006683228 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d480125f0 con 0x7f1d6015dec0 2024-03-24T00:40:44.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.097+0000 7f1d56ffd640 1 -- 172.21.15.67:0/4006683228 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d48019070 con 0x7f1d6015dec0 2024-03-24T00:40:44.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.097+0000 7f1d64f09640 1 -- 172.21.15.67:0/4006683228 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d601e8b30 con 0x7f1d6015dec0 2024-03-24T00:40:44.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.097+0000 7f1d56ffd640 1 -- 172.21.15.67:0/4006683228 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d480129d0 con 0x7f1d6015dec0 2024-03-24T00:40:44.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.097+0000 7f1d64f09640 1 -- 172.21.15.67:0/4006683228 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d601e90f0 con 0x7f1d6015dec0 2024-03-24T00:40:44.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.098+0000 7f1d64f09640 1 -- 172.21.15.67:0/4006683228 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d6004f400 con 0x7f1d6015dec0 2024-03-24T00:40:44.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.098+0000 7f1d56ffd640 1 -- 172.21.15.67:0/4006683228 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1d48012b70 con 0x7f1d6015dec0 2024-03-24T00:40:44.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.099+0000 7f1d56ffd640 1 --2- 172.21.15.67:0/4006683228 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1d4006bbd0 0x7f1d4006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:44.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.099+0000 7f1d56ffd640 1 -- 172.21.15.67:0/4006683228 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1d4808af20 con 0x7f1d6015dec0 2024-03-24T00:40:44.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.099+0000 7f1d5e59c640 1 --2- 172.21.15.67:0/4006683228 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1d4006bbd0 0x7f1d4006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:44.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.099+0000 7f1d5e59c640 1 --2- 172.21.15.67:0/4006683228 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1d4006bbd0 0x7f1d4006e090 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f1d50006fd0 tx=0x7f1d50008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:44.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.103+0000 7f1d56ffd640 1 -- 172.21.15.67:0/4006683228 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1d48011340 con 0x7f1d6015dec0 2024-03-24T00:40:44.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.284+0000 7f1d64f09640 1 -- 172.21.15.67:0/4006683228 --> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1d601e9580 con 0x7f1d4006bbd0 2024-03-24T00:40:44.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.290+0000 7f1d56ffd640 1 -- 172.21.15.67:0/4006683228 <== mgr.4105 v2:172.21.15.72:6832/2768292274 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246638 (secure 0 0 0) 0x7f1d601e9580 con 0x7f1d4006bbd0 2024-03-24T00:40:44.293 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:40:44.295 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-24T00:40:44.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.293+0000 7f1d64f09640 1 -- 172.21.15.67:0/4006683228 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1d4006bbd0 msgr2=0x7f1d4006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:44.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.293+0000 7f1d64f09640 1 --2- 172.21.15.67:0/4006683228 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1d4006bbd0 0x7f1d4006e090 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f1d50006fd0 tx=0x7f1d50008040 comp rx=0 tx=0).stop 2024-03-24T00:40:44.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.293+0000 7f1d64f09640 1 -- 172.21.15.67:0/4006683228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d6015dec0 msgr2=0x7f1d6015feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:44.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.293+0000 7f1d64f09640 1 --2- 172.21.15.67:0/4006683228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d6015dec0 0x7f1d6015feb0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f1d4800d5c0 tx=0x7f1d4800da80 comp rx=0 tx=0).stop 2024-03-24T00:40:44.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.293+0000 7f1d64f09640 1 -- 172.21.15.67:0/4006683228 shutdown_connections 2024-03-24T00:40:44.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.293+0000 7f1d64f09640 1 --2- 172.21.15.67:0/4006683228 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1d4006bbd0 0x7f1d4006e090 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:44.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.293+0000 7f1d64f09640 1 --2- 172.21.15.67:0/4006683228 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1d6015e2a0 0x7f1d60158030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:44.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.293+0000 7f1d64f09640 1 --2- 172.21.15.67:0/4006683228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d6015dec0 0x7f1d6015feb0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:44.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.293+0000 7f1d64f09640 1 --2- 172.21.15.67:0/4006683228 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1d600d07e0 0x7f1d60153e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:44.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.293+0000 7f1d64f09640 1 -- 172.21.15.67:0/4006683228 >> 172.21.15.67:0/4006683228 conn(0x7f1d6006c810 msgr2=0x7f1d600b5d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:44.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.294+0000 7f1d64f09640 1 -- 172.21.15.67:0/4006683228 shutdown_connections 2024-03-24T00:40:44.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.294+0000 7f1d64f09640 1 -- 172.21.15.67:0/4006683228 wait complete. 2024-03-24T00:40:44.305 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":105,"stamp":"2024-03-24T00:40:43.345501+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":221684,"kb_used_data":5512,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753036,"statfs":{"total":773094113280,"available":772867108864,"internally_reserved":0,"allocated":5644288,"data_stored":3135232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.066578"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.290314+0000","last_change":"2024-03-24T00:40:39.290314+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.290314+0000","last_undegraded":"2024-03-24T00:40:39.290314+0000","last_fullsized":"2024-03-24T00:40:39.290314+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.290184+0000","last_change":"2024-03-24T00:40:39.290184+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.290184+0000","last_undegraded":"2024-03-24T00:40:39.290184+0000","last_fullsized":"2024-03-24T00:40:39.290184+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.290644+0000","last_change":"2024-03-24T00:40:39.290644+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.290644+0000","last_undegraded":"2024-03-24T00:40:39.290644+0000","last_fullsized":"2024-03-24T00:40:39.290644+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.286759+0000","last_change":"2024-03-24T00:40:39.286759+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.286759+0000","last_undegraded":"2024-03-24T00:40:39.286759+0000","last_fullsized":"2024-03-24T00:40:39.286759+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.288142+0000","last_change":"2024-03-24T00:40:39.288142+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.288142+0000","last_undegraded":"2024-03-24T00:40:39.288142+0000","last_fullsized":"2024-03-24T00:40:39.288142+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.289818+0000","last_change":"2024-03-24T00:40:39.289818+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.289818+0000","last_undegraded":"2024-03-24T00:40:39.289818+0000","last_fullsized":"2024-03-24T00:40:39.289818+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.292120+0000","last_change":"2024-03-24T00:40:39.292120+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.292120+0000","last_undegraded":"2024-03-24T00:40:39.292120+0000","last_fullsized":"2024-03-24T00:40:39.292120+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.285842+0000","last_change":"2024-03-24T00:40:39.285842+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.285842+0000","last_undegraded":"2024-03-24T00:40:39.285842+0000","last_fullsized":"2024-03-24T00:40:39.285842+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.287182+0000","last_change":"2024-03-24T00:40:39.287182+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.287182+0000","last_undegraded":"2024-03-24T00:40:39.287182+0000","last_fullsized":"2024-03-24T00:40:39.287182+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.287090+0000","last_change":"2024-03-24T00:40:39.287090+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.287090+0000","last_undegraded":"2024-03-24T00:40:39.287090+0000","last_fullsized":"2024-03-24T00:40:39.287090+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.293731+0000","last_change":"2024-03-24T00:40:39.293731+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.293731+0000","last_undegraded":"2024-03-24T00:40:39.293731+0000","last_fullsized":"2024-03-24T00:40:39.293731+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.291962+0000","last_change":"2024-03-24T00:40:39.291962+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.291962+0000","last_undegraded":"2024-03-24T00:40:39.291962+0000","last_fullsized":"2024-03-24T00:40:39.291962+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.285910+0000","last_change":"2024-03-24T00:40:39.285910+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.285910+0000","last_undegraded":"2024-03-24T00:40:39.285910+0000","last_fullsized":"2024-03-24T00:40:39.285910+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.291223+0000","last_change":"2024-03-24T00:40:39.291223+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.291223+0000","last_undegraded":"2024-03-24T00:40:39.291223+0000","last_fullsized":"2024-03-24T00:40:39.291223+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.287287+0000","last_change":"2024-03-24T00:40:39.287287+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.287287+0000","last_undegraded":"2024-03-24T00:40:39.287287+0000","last_fullsized":"2024-03-24T00:40:39.287287+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.286183+0000","last_change":"2024-03-24T00:40:39.286183+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.286183+0000","last_undegraded":"2024-03-24T00:40:39.286183+0000","last_fullsized":"2024-03-24T00:40:39.286183+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.290840+0000","last_change":"2024-03-24T00:40:39.290840+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.290840+0000","last_undegraded":"2024-03-24T00:40:39.290840+0000","last_fullsized":"2024-03-24T00:40:39.290840+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.287128+0000","last_change":"2024-03-24T00:40:39.287128+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.287128+0000","last_undegraded":"2024-03-24T00:40:39.287128+0000","last_fullsized":"2024-03-24T00:40:39.287128+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.290088+0000","last_change":"2024-03-24T00:40:39.290088+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.290088+0000","last_undegraded":"2024-03-24T00:40:39.290088+0000","last_fullsized":"2024-03-24T00:40:39.290088+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.287773+0000","last_change":"2024-03-24T00:40:39.287773+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.287773+0000","last_undegraded":"2024-03-24T00:40:39.287773+0000","last_fullsized":"2024-03-24T00:40:39.287773+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.289216+0000","last_change":"2024-03-24T00:40:39.289216+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.289216+0000","last_undegraded":"2024-03-24T00:40:39.289216+0000","last_fullsized":"2024-03-24T00:40:39.289216+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.286528+0000","last_change":"2024-03-24T00:40:39.286528+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.286528+0000","last_undegraded":"2024-03-24T00:40:39.286528+0000","last_fullsized":"2024-03-24T00:40:39.286528+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.293164+0000","last_change":"2024-03-24T00:40:39.293164+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.293164+0000","last_undegraded":"2024-03-24T00:40:39.293164+0000","last_fullsized":"2024-03-24T00:40:39.293164+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.289958+0000","last_change":"2024-03-24T00:40:39.289958+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.289958+0000","last_undegraded":"2024-03-24T00:40:39.289958+0000","last_fullsized":"2024-03-24T00:40:39.289958+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.290445+0000","last_change":"2024-03-24T00:40:39.290445+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.290445+0000","last_undegraded":"2024-03-24T00:40:39.290445+0000","last_fullsized":"2024-03-24T00:40:39.290445+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.287297+0000","last_change":"2024-03-24T00:40:39.287297+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.287297+0000","last_undegraded":"2024-03-24T00:40:39.287297+0000","last_fullsized":"2024-03-24T00:40:39.287297+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.286083+0000","last_change":"2024-03-24T00:40:39.286083+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.286083+0000","last_undegraded":"2024-03-24T00:40:39.286083+0000","last_fullsized":"2024-03-24T00:40:39.286083+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.287284+0000","last_change":"2024-03-24T00:40:39.287284+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.287284+0000","last_undegraded":"2024-03-24T00:40:39.287284+0000","last_fullsized":"2024-03-24T00:40:39.287284+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.291689+0000","last_change":"2024-03-24T00:40:39.291689+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.291689+0000","last_undegraded":"2024-03-24T00:40:39.291689+0000","last_fullsized":"2024-03-24T00:40:39.291689+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.286764+0000","last_change":"2024-03-24T00:40:39.286764+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.286764+0000","last_undegraded":"2024-03-24T00:40:39.286764+0000","last_fullsized":"2024-03-24T00:40:39.286764+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.289986+0000","last_change":"2024-03-24T00:40:39.289986+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.289986+0000","last_undegraded":"2024-03-24T00:40:39.289986+0000","last_fullsized":"2024-03-24T00:40:39.289986+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.284737+0000","last_change":"2024-03-24T00:40:39.284737+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.284737+0000","last_undegraded":"2024-03-24T00:40:39.284737+0000","last_fullsized":"2024-03-24T00:40:39.284737+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.290205+0000","last_change":"2024-03-24T00:40:39.290205+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.290205+0000","last_undegraded":"2024-03-24T00:40:39.290205+0000","last_fullsized":"2024-03-24T00:40:39.290205+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.293699+0000","last_change":"2024-03-24T00:40:39.291631+0000","last_active":"2024-03-24T00:40:39.293699+0000","last_peered":"2024-03-24T00:40:39.293699+0000","last_clean":"2024-03-24T00:40:39.293699+0000","last_became_active":"2024-03-24T00:40:39.291349+0000","last_became_peered":"2024-03-24T00:40:39.291349+0000","last_unstale":"2024-03-24T00:40:39.293699+0000","last_undegraded":"2024-03-24T00:40:39.293699+0000","last_fullsized":"2024-03-24T00:40:39.293699+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:32:08.714068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.283698+0000","last_change":"2024-03-24T00:40:39.283698+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.283698+0000","last_undegraded":"2024-03-24T00:40:39.283698+0000","last_fullsized":"2024-03-24T00:40:39.283698+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.297930+0000","last_change":"2024-03-24T00:40:39.295765+0000","last_active":"2024-03-24T00:40:39.297930+0000","last_peered":"2024-03-24T00:40:39.297930+0000","last_clean":"2024-03-24T00:40:39.297930+0000","last_became_active":"2024-03-24T00:40:39.295486+0000","last_became_peered":"2024-03-24T00:40:39.295486+0000","last_unstale":"2024-03-24T00:40:39.297930+0000","last_undegraded":"2024-03-24T00:40:39.297930+0000","last_fullsized":"2024-03-24T00:40:39.297930+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:09:53.820800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.289688+0000","last_change":"2024-03-24T00:40:39.289688+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.289688+0000","last_undegraded":"2024-03-24T00:40:39.289688+0000","last_fullsized":"2024-03-24T00:40:39.289688+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.295662+0000","last_change":"2024-03-24T00:40:39.294921+0000","last_active":"2024-03-24T00:40:39.295662+0000","last_peered":"2024-03-24T00:40:39.295662+0000","last_clean":"2024-03-24T00:40:39.295662+0000","last_became_active":"2024-03-24T00:40:39.294579+0000","last_became_peered":"2024-03-24T00:40:39.294579+0000","last_unstale":"2024-03-24T00:40:39.295662+0000","last_undegraded":"2024-03-24T00:40:39.295662+0000","last_fullsized":"2024-03-24T00:40:39.295662+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:54:49.331324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.295225+0000","last_change":"2024-03-24T00:40:39.295225+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.295225+0000","last_undegraded":"2024-03-24T00:40:39.295225+0000","last_fullsized":"2024-03-24T00:40:39.295225+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.299268+0000","last_change":"2024-03-24T00:40:39.297839+0000","last_active":"2024-03-24T00:40:39.299268+0000","last_peered":"2024-03-24T00:40:39.299268+0000","last_clean":"2024-03-24T00:40:39.299268+0000","last_became_active":"2024-03-24T00:40:39.297536+0000","last_became_peered":"2024-03-24T00:40:39.297536+0000","last_unstale":"2024-03-24T00:40:39.299268+0000","last_undegraded":"2024-03-24T00:40:39.299268+0000","last_fullsized":"2024-03-24T00:40:39.299268+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:58:21.161844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.286437+0000","last_change":"2024-03-24T00:40:39.286437+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.286437+0000","last_undegraded":"2024-03-24T00:40:39.286437+0000","last_fullsized":"2024-03-24T00:40:39.286437+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.305288+0000","last_change":"2024-03-24T00:40:39.304061+0000","last_active":"2024-03-24T00:40:39.305288+0000","last_peered":"2024-03-24T00:40:39.305288+0000","last_clean":"2024-03-24T00:40:39.305288+0000","last_became_active":"2024-03-24T00:40:39.303769+0000","last_became_peered":"2024-03-24T00:40:39.303769+0000","last_unstale":"2024-03-24T00:40:39.305288+0000","last_undegraded":"2024-03-24T00:40:39.305288+0000","last_fullsized":"2024-03-24T00:40:39.305288+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:00:51.940556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.286866+0000","last_change":"2024-03-24T00:40:39.286866+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.286866+0000","last_undegraded":"2024-03-24T00:40:39.286866+0000","last_fullsized":"2024-03-24T00:40:39.286866+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.295132+0000","last_change":"2024-03-24T00:40:39.293463+0000","last_active":"2024-03-24T00:40:39.295132+0000","last_peered":"2024-03-24T00:40:39.295132+0000","last_clean":"2024-03-24T00:40:39.295132+0000","last_became_active":"2024-03-24T00:40:39.293196+0000","last_became_peered":"2024-03-24T00:40:39.293196+0000","last_unstale":"2024-03-24T00:40:39.295132+0000","last_undegraded":"2024-03-24T00:40:39.295132+0000","last_fullsized":"2024-03-24T00:40:39.295132+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:16:22.016263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.285844+0000","last_change":"2024-03-24T00:40:39.285844+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.285844+0000","last_undegraded":"2024-03-24T00:40:39.285844+0000","last_fullsized":"2024-03-24T00:40:39.285844+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.296000+0000","last_change":"2024-03-24T00:40:39.294468+0000","last_active":"2024-03-24T00:40:39.296000+0000","last_peered":"2024-03-24T00:40:39.296000+0000","last_clean":"2024-03-24T00:40:39.296000+0000","last_became_active":"2024-03-24T00:40:39.294130+0000","last_became_peered":"2024-03-24T00:40:39.294130+0000","last_unstale":"2024-03-24T00:40:39.296000+0000","last_undegraded":"2024-03-24T00:40:39.296000+0000","last_fullsized":"2024-03-24T00:40:39.296000+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:52:39.705988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.290406+0000","last_change":"2024-03-24T00:40:39.290406+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.290406+0000","last_undegraded":"2024-03-24T00:40:39.290406+0000","last_fullsized":"2024-03-24T00:40:39.290406+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.298578+0000","last_change":"2024-03-24T00:40:39.296539+0000","last_active":"2024-03-24T00:40:39.298578+0000","last_peered":"2024-03-24T00:40:39.298578+0000","last_clean":"2024-03-24T00:40:39.298578+0000","last_became_active":"2024-03-24T00:40:39.296272+0000","last_became_peered":"2024-03-24T00:40:39.296272+0000","last_unstale":"2024-03-24T00:40:39.298578+0000","last_undegraded":"2024-03-24T00:40:39.298578+0000","last_fullsized":"2024-03-24T00:40:39.298578+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:09:53.820800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.286586+0000","last_change":"2024-03-24T00:40:39.286586+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.286586+0000","last_undegraded":"2024-03-24T00:40:39.286586+0000","last_fullsized":"2024-03-24T00:40:39.286586+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.297968+0000","last_change":"2024-03-24T00:40:39.296067+0000","last_active":"2024-03-24T00:40:39.297968+0000","last_peered":"2024-03-24T00:40:39.297968+0000","last_clean":"2024-03-24T00:40:39.297968+0000","last_became_active":"2024-03-24T00:40:39.295773+0000","last_became_peered":"2024-03-24T00:40:39.295773+0000","last_unstale":"2024-03-24T00:40:39.297968+0000","last_undegraded":"2024-03-24T00:40:39.297968+0000","last_fullsized":"2024-03-24T00:40:39.297968+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:53:34.136108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.284282+0000","last_change":"2024-03-24T00:40:39.284282+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.284282+0000","last_undegraded":"2024-03-24T00:40:39.284282+0000","last_fullsized":"2024-03-24T00:40:39.284282+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.298719+0000","last_change":"2024-03-24T00:40:39.296711+0000","last_active":"2024-03-24T00:40:39.298719+0000","last_peered":"2024-03-24T00:40:39.298719+0000","last_clean":"2024-03-24T00:40:39.298719+0000","last_became_active":"2024-03-24T00:40:39.296435+0000","last_became_peered":"2024-03-24T00:40:39.296435+0000","last_unstale":"2024-03-24T00:40:39.298719+0000","last_undegraded":"2024-03-24T00:40:39.298719+0000","last_fullsized":"2024-03-24T00:40:39.298719+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:54:49.331324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.285258+0000","last_change":"2024-03-24T00:40:39.285258+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.285258+0000","last_undegraded":"2024-03-24T00:40:39.285258+0000","last_fullsized":"2024-03-24T00:40:39.285258+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.303989+0000","last_change":"2024-03-24T00:40:39.301123+0000","last_active":"2024-03-24T00:40:39.303989+0000","last_peered":"2024-03-24T00:40:39.303989+0000","last_clean":"2024-03-24T00:40:39.303989+0000","last_became_active":"2024-03-24T00:40:39.300851+0000","last_became_peered":"2024-03-24T00:40:39.300851+0000","last_unstale":"2024-03-24T00:40:39.303989+0000","last_undegraded":"2024-03-24T00:40:39.303989+0000","last_fullsized":"2024-03-24T00:40:39.303989+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:00:51.940556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.287678+0000","last_change":"2024-03-24T00:40:39.287678+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.287678+0000","last_undegraded":"2024-03-24T00:40:39.287678+0000","last_fullsized":"2024-03-24T00:40:39.287678+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.304078+0000","last_change":"2024-03-24T00:40:39.302117+0000","last_active":"2024-03-24T00:40:39.304078+0000","last_peered":"2024-03-24T00:40:39.304078+0000","last_clean":"2024-03-24T00:40:39.304078+0000","last_became_active":"2024-03-24T00:40:39.301875+0000","last_became_peered":"2024-03-24T00:40:39.301875+0000","last_unstale":"2024-03-24T00:40:39.304078+0000","last_undegraded":"2024-03-24T00:40:39.304078+0000","last_fullsized":"2024-03-24T00:40:39.304078+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:48:02.396386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.284454+0000","last_change":"2024-03-24T00:40:39.284454+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.284454+0000","last_undegraded":"2024-03-24T00:40:39.284454+0000","last_fullsized":"2024-03-24T00:40:39.284454+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.303378+0000","last_change":"2024-03-24T00:40:39.298667+0000","last_active":"2024-03-24T00:40:39.303378+0000","last_peered":"2024-03-24T00:40:39.303378+0000","last_clean":"2024-03-24T00:40:39.303378+0000","last_became_active":"2024-03-24T00:40:39.298067+0000","last_became_peered":"2024-03-24T00:40:39.298067+0000","last_unstale":"2024-03-24T00:40:39.303378+0000","last_undegraded":"2024-03-24T00:40:39.303378+0000","last_fullsized":"2024-03-24T00:40:39.303378+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:35.720596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.286589+0000","last_change":"2024-03-24T00:40:39.286589+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.286589+0000","last_undegraded":"2024-03-24T00:40:39.286589+0000","last_fullsized":"2024-03-24T00:40:39.286589+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.301055+0000","last_change":"2024-03-24T00:40:39.299651+0000","last_active":"2024-03-24T00:40:39.301055+0000","last_peered":"2024-03-24T00:40:39.301055+0000","last_clean":"2024-03-24T00:40:39.301055+0000","last_became_active":"2024-03-24T00:40:39.299388+0000","last_became_peered":"2024-03-24T00:40:39.299388+0000","last_unstale":"2024-03-24T00:40:39.301055+0000","last_undegraded":"2024-03-24T00:40:39.301055+0000","last_fullsized":"2024-03-24T00:40:39.301055+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:30:39.662599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.286895+0000","last_change":"2024-03-24T00:40:39.286895+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.286895+0000","last_undegraded":"2024-03-24T00:40:39.286895+0000","last_fullsized":"2024-03-24T00:40:39.286895+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.298000+0000","last_change":"2024-03-24T00:40:39.295906+0000","last_active":"2024-03-24T00:40:39.298000+0000","last_peered":"2024-03-24T00:40:39.298000+0000","last_clean":"2024-03-24T00:40:39.298000+0000","last_became_active":"2024-03-24T00:40:39.295604+0000","last_became_peered":"2024-03-24T00:40:39.295604+0000","last_unstale":"2024-03-24T00:40:39.298000+0000","last_undegraded":"2024-03-24T00:40:39.298000+0000","last_fullsized":"2024-03-24T00:40:39.298000+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:35.720596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.288224+0000","last_change":"2024-03-24T00:40:39.288224+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.288224+0000","last_undegraded":"2024-03-24T00:40:39.288224+0000","last_fullsized":"2024-03-24T00:40:39.288224+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.302950+0000","last_change":"2024-03-24T00:40:39.298619+0000","last_active":"2024-03-24T00:40:39.302950+0000","last_peered":"2024-03-24T00:40:39.302950+0000","last_clean":"2024-03-24T00:40:39.302950+0000","last_became_active":"2024-03-24T00:40:39.298350+0000","last_became_peered":"2024-03-24T00:40:39.298350+0000","last_unstale":"2024-03-24T00:40:39.302950+0000","last_undegraded":"2024-03-24T00:40:39.302950+0000","last_fullsized":"2024-03-24T00:40:39.302950+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:04:49.487506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.285444+0000","last_change":"2024-03-24T00:40:39.285444+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.285444+0000","last_undegraded":"2024-03-24T00:40:39.285444+0000","last_fullsized":"2024-03-24T00:40:39.285444+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.303365+0000","last_change":"2024-03-24T00:40:39.299640+0000","last_active":"2024-03-24T00:40:39.303365+0000","last_peered":"2024-03-24T00:40:39.303365+0000","last_clean":"2024-03-24T00:40:39.303365+0000","last_became_active":"2024-03-24T00:40:39.299375+0000","last_became_peered":"2024-03-24T00:40:39.299375+0000","last_unstale":"2024-03-24T00:40:39.303365+0000","last_undegraded":"2024-03-24T00:40:39.303365+0000","last_fullsized":"2024-03-24T00:40:39.303365+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:53:34.136108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.284942+0000","last_change":"2024-03-24T00:40:39.284942+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.284942+0000","last_undegraded":"2024-03-24T00:40:39.284942+0000","last_fullsized":"2024-03-24T00:40:39.284942+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.303123+0000","last_change":"2024-03-24T00:40:39.302072+0000","last_active":"2024-03-24T00:40:39.303123+0000","last_peered":"2024-03-24T00:40:39.303123+0000","last_clean":"2024-03-24T00:40:39.303123+0000","last_became_active":"2024-03-24T00:40:39.301790+0000","last_became_peered":"2024-03-24T00:40:39.301790+0000","last_unstale":"2024-03-24T00:40:39.303123+0000","last_undegraded":"2024-03-24T00:40:39.303123+0000","last_fullsized":"2024-03-24T00:40:39.303123+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:48:22.534145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.287303+0000","last_change":"2024-03-24T00:40:39.287303+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.287303+0000","last_undegraded":"2024-03-24T00:40:39.287303+0000","last_fullsized":"2024-03-24T00:40:39.287303+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.293730+0000","last_change":"2024-03-24T00:40:39.292046+0000","last_active":"2024-03-24T00:40:39.293730+0000","last_peered":"2024-03-24T00:40:39.293730+0000","last_clean":"2024-03-24T00:40:39.293730+0000","last_became_active":"2024-03-24T00:40:39.291744+0000","last_became_peered":"2024-03-24T00:40:39.291744+0000","last_unstale":"2024-03-24T00:40:39.293730+0000","last_undegraded":"2024-03-24T00:40:39.293730+0000","last_fullsized":"2024-03-24T00:40:39.293730+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:05:15.049538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.284674+0000","last_change":"2024-03-24T00:40:39.284674+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.284674+0000","last_undegraded":"2024-03-24T00:40:39.284674+0000","last_fullsized":"2024-03-24T00:40:39.284674+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.309077+0000","last_change":"2024-03-24T00:40:39.304591+0000","last_active":"2024-03-24T00:40:39.309077+0000","last_peered":"2024-03-24T00:40:39.309077+0000","last_clean":"2024-03-24T00:40:39.309077+0000","last_became_active":"2024-03-24T00:40:39.304330+0000","last_became_peered":"2024-03-24T00:40:39.304330+0000","last_unstale":"2024-03-24T00:40:39.309077+0000","last_undegraded":"2024-03-24T00:40:39.309077+0000","last_fullsized":"2024-03-24T00:40:39.309077+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:25:48.533462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.283875+0000","last_change":"2024-03-24T00:40:39.283875+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.283875+0000","last_undegraded":"2024-03-24T00:40:39.283875+0000","last_fullsized":"2024-03-24T00:40:39.283875+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.289779+0000","last_change":"2024-03-24T00:40:39.287554+0000","last_active":"2024-03-24T00:40:39.289779+0000","last_peered":"2024-03-24T00:40:39.289779+0000","last_clean":"2024-03-24T00:40:39.289779+0000","last_became_active":"2024-03-24T00:40:39.287273+0000","last_became_peered":"2024-03-24T00:40:39.287273+0000","last_unstale":"2024-03-24T00:40:39.289779+0000","last_undegraded":"2024-03-24T00:40:39.289779+0000","last_fullsized":"2024-03-24T00:40:39.289779+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:16:22.016263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.289204+0000","last_change":"2024-03-24T00:40:39.289204+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.289204+0000","last_undegraded":"2024-03-24T00:40:39.289204+0000","last_fullsized":"2024-03-24T00:40:39.289204+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.302927+0000","last_change":"2024-03-24T00:40:39.298506+0000","last_active":"2024-03-24T00:40:39.302927+0000","last_peered":"2024-03-24T00:40:39.302927+0000","last_clean":"2024-03-24T00:40:39.302927+0000","last_became_active":"2024-03-24T00:40:39.297935+0000","last_became_peered":"2024-03-24T00:40:39.297935+0000","last_unstale":"2024-03-24T00:40:39.302927+0000","last_undegraded":"2024-03-24T00:40:39.302927+0000","last_fullsized":"2024-03-24T00:40:39.302927+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:23:25.616752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.287275+0000","last_change":"2024-03-24T00:40:39.287275+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.287275+0000","last_undegraded":"2024-03-24T00:40:39.287275+0000","last_fullsized":"2024-03-24T00:40:39.287275+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.298445+0000","last_change":"2024-03-24T00:40:39.296787+0000","last_active":"2024-03-24T00:40:39.298445+0000","last_peered":"2024-03-24T00:40:39.298445+0000","last_clean":"2024-03-24T00:40:39.298445+0000","last_became_active":"2024-03-24T00:40:39.296501+0000","last_became_peered":"2024-03-24T00:40:39.296501+0000","last_unstale":"2024-03-24T00:40:39.298445+0000","last_undegraded":"2024-03-24T00:40:39.298445+0000","last_fullsized":"2024-03-24T00:40:39.298445+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:43:07.435024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.285594+0000","last_change":"2024-03-24T00:40:39.285594+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.285594+0000","last_undegraded":"2024-03-24T00:40:39.285594+0000","last_fullsized":"2024-03-24T00:40:39.285594+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.302585+0000","last_change":"2024-03-24T00:40:39.300361+0000","last_active":"2024-03-24T00:40:39.302585+0000","last_peered":"2024-03-24T00:40:39.302585+0000","last_clean":"2024-03-24T00:40:39.302585+0000","last_became_active":"2024-03-24T00:40:39.299954+0000","last_became_peered":"2024-03-24T00:40:39.299954+0000","last_unstale":"2024-03-24T00:40:39.302585+0000","last_undegraded":"2024-03-24T00:40:39.302585+0000","last_fullsized":"2024-03-24T00:40:39.302585+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:35:30.101713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.283792+0000","last_change":"2024-03-24T00:40:39.283792+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.283792+0000","last_undegraded":"2024-03-24T00:40:39.283792+0000","last_fullsized":"2024-03-24T00:40:39.283792+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.300944+0000","last_change":"2024-03-24T00:40:39.299797+0000","last_active":"2024-03-24T00:40:39.300944+0000","last_peered":"2024-03-24T00:40:39.300944+0000","last_clean":"2024-03-24T00:40:39.300944+0000","last_became_active":"2024-03-24T00:40:39.299456+0000","last_became_peered":"2024-03-24T00:40:39.299456+0000","last_unstale":"2024-03-24T00:40:39.300944+0000","last_undegraded":"2024-03-24T00:40:39.300944+0000","last_fullsized":"2024-03-24T00:40:39.300944+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:28:27.547221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.285596+0000","last_change":"2024-03-24T00:40:39.285596+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.285596+0000","last_undegraded":"2024-03-24T00:40:39.285596+0000","last_fullsized":"2024-03-24T00:40:39.285596+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.299923+0000","last_change":"2024-03-24T00:40:39.299014+0000","last_active":"2024-03-24T00:40:39.299923+0000","last_peered":"2024-03-24T00:40:39.299923+0000","last_clean":"2024-03-24T00:40:39.299923+0000","last_became_active":"2024-03-24T00:40:39.298750+0000","last_became_peered":"2024-03-24T00:40:39.298750+0000","last_unstale":"2024-03-24T00:40:39.299923+0000","last_undegraded":"2024-03-24T00:40:39.299923+0000","last_fullsized":"2024-03-24T00:40:39.299923+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:08:19.135822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.289013+0000","last_change":"2024-03-24T00:40:39.289013+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.289013+0000","last_undegraded":"2024-03-24T00:40:39.289013+0000","last_fullsized":"2024-03-24T00:40:39.289013+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.304762+0000","last_change":"2024-03-24T00:40:39.303224+0000","last_active":"2024-03-24T00:40:39.304762+0000","last_peered":"2024-03-24T00:40:39.304762+0000","last_clean":"2024-03-24T00:40:39.304762+0000","last_became_active":"2024-03-24T00:40:39.302950+0000","last_became_peered":"2024-03-24T00:40:39.302950+0000","last_unstale":"2024-03-24T00:40:39.304762+0000","last_undegraded":"2024-03-24T00:40:39.304762+0000","last_fullsized":"2024-03-24T00:40:39.304762+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:08:19.135822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.285991+0000","last_change":"2024-03-24T00:40:39.285991+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.285991+0000","last_undegraded":"2024-03-24T00:40:39.285991+0000","last_fullsized":"2024-03-24T00:40:39.285991+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.304018+0000","last_change":"2024-03-24T00:40:39.299484+0000","last_active":"2024-03-24T00:40:39.304018+0000","last_peered":"2024-03-24T00:40:39.304018+0000","last_clean":"2024-03-24T00:40:39.304018+0000","last_became_active":"2024-03-24T00:40:39.299184+0000","last_became_peered":"2024-03-24T00:40:39.299184+0000","last_unstale":"2024-03-24T00:40:39.304018+0000","last_undegraded":"2024-03-24T00:40:39.304018+0000","last_fullsized":"2024-03-24T00:40:39.304018+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:57:35.870209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.283715+0000","last_change":"2024-03-24T00:40:39.283715+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.283715+0000","last_undegraded":"2024-03-24T00:40:39.283715+0000","last_fullsized":"2024-03-24T00:40:39.283715+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.302907+0000","last_change":"2024-03-24T00:40:39.298278+0000","last_active":"2024-03-24T00:40:39.302907+0000","last_peered":"2024-03-24T00:40:39.302907+0000","last_clean":"2024-03-24T00:40:39.302907+0000","last_became_active":"2024-03-24T00:40:39.297588+0000","last_became_peered":"2024-03-24T00:40:39.297588+0000","last_unstale":"2024-03-24T00:40:39.302907+0000","last_undegraded":"2024-03-24T00:40:39.302907+0000","last_fullsized":"2024-03-24T00:40:39.302907+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:06:05.426065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.287710+0000","last_change":"2024-03-24T00:40:39.287710+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.287710+0000","last_undegraded":"2024-03-24T00:40:39.287710+0000","last_fullsized":"2024-03-24T00:40:39.287710+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.301019+0000","last_change":"2024-03-24T00:40:39.300219+0000","last_active":"2024-03-24T00:40:39.301019+0000","last_peered":"2024-03-24T00:40:39.301019+0000","last_clean":"2024-03-24T00:40:39.301019+0000","last_became_active":"2024-03-24T00:40:39.299897+0000","last_became_peered":"2024-03-24T00:40:39.299897+0000","last_unstale":"2024-03-24T00:40:39.301019+0000","last_undegraded":"2024-03-24T00:40:39.301019+0000","last_fullsized":"2024-03-24T00:40:39.301019+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:09:53.820800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.286808+0000","last_change":"2024-03-24T00:40:39.286808+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.286808+0000","last_undegraded":"2024-03-24T00:40:39.286808+0000","last_fullsized":"2024-03-24T00:40:39.286808+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.295357+0000","last_change":"2024-03-24T00:40:39.294249+0000","last_active":"2024-03-24T00:40:39.295357+0000","last_peered":"2024-03-24T00:40:39.295357+0000","last_clean":"2024-03-24T00:40:39.295357+0000","last_became_active":"2024-03-24T00:40:39.293915+0000","last_became_peered":"2024-03-24T00:40:39.293915+0000","last_unstale":"2024-03-24T00:40:39.295357+0000","last_undegraded":"2024-03-24T00:40:39.295357+0000","last_fullsized":"2024-03-24T00:40:39.295357+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:05:32.540043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.289541+0000","last_change":"2024-03-24T00:40:39.289541+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.289541+0000","last_undegraded":"2024-03-24T00:40:39.289541+0000","last_fullsized":"2024-03-24T00:40:39.289541+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.302298+0000","last_change":"2024-03-24T00:40:39.297685+0000","last_active":"2024-03-24T00:40:39.302298+0000","last_peered":"2024-03-24T00:40:39.302298+0000","last_clean":"2024-03-24T00:40:39.302298+0000","last_became_active":"2024-03-24T00:40:39.297097+0000","last_became_peered":"2024-03-24T00:40:39.297097+0000","last_unstale":"2024-03-24T00:40:39.302298+0000","last_undegraded":"2024-03-24T00:40:39.302298+0000","last_fullsized":"2024-03-24T00:40:39.302298+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:42:28.507176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":220,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.291524+0000","last_change":"2024-03-24T00:38:40.953871+0000","last_active":"2024-03-24T00:40:39.291524+0000","last_peered":"2024-03-24T00:40:39.291524+0000","last_clean":"2024-03-24T00:40:39.291524+0000","last_became_active":"2024-03-24T00:38:40.952926+0000","last_became_peered":"2024-03-24T00:38:40.952926+0000","last_unstale":"2024-03-24T00:40:39.291524+0000","last_undegraded":"2024-03-24T00:40:39.291524+0000","last_fullsized":"2024-03-24T00:40:39.291524+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_clean_scrub_stamp":"2024-03-24T00:38:39.722194+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:05.397257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574877,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28144,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343696,"statfs":{"total":96636764160,"available":96607944704,"internally_reserved":0,"allocated":1150976,"data_stored":834680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27568,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344272,"statfs":{"total":96636764160,"available":96608534528,"internally_reserved":0,"allocated":557056,"data_stored":244312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":557056,"data_stored":244312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":557056,"data_stored":244312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574875,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27568,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344272,"statfs":{"total":96636764160,"available":96608534528,"internally_reserved":0,"allocated":557056,"data_stored":244312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27568,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344272,"statfs":{"total":96636764160,"available":96608534528,"internally_reserved":0,"allocated":557056,"data_stored":244312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":557056,"data_stored":244312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574875,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28144,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343696,"statfs":{"total":96636764160,"available":96607944704,"internally_reserved":0,"allocated":1150976,"data_stored":834680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:40:44.309 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:40:44.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.446+0000 7f6599f80640 1 Processor -- start 2024-03-24T00:40:44.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.446+0000 7f6599f80640 1 -- start start 2024-03-24T00:40:44.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.448+0000 7f6599f80640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f659406ba50 0x7f65940ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:44.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.448+0000 7f6599f80640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f65940cd240 0x7f65940b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:44.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.448+0000 7f6599f80640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f659406c210 con 0x7f65940cc360 2024-03-24T00:40:44.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.448+0000 7f6599f80640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65940cd6b0 con 0x7f65940cd240 2024-03-24T00:40:44.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.448+0000 7f6599f80640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65940caad0 con 0x7f659406ba50 2024-03-24T00:40:44.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.448+0000 7f658bfff640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f659406ba50 0x7f65940ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:44.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.448+0000 7f6593fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f65940cd240 0x7f65940b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:44.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.448+0000 7f658bfff640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f659406ba50 0x7f65940ccd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.67:43584/0 (socket says 172.21.15.67:43584) 2024-03-24T00:40:44.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.448+0000 7f658bfff640 1 -- 172.21.15.67:0/3047718255 learned_addr learned my addr 172.21.15.67:0/3047718255 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:44.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.448+0000 7f65937fe640 1 --1- 172.21.15.67:0/3047718255 >> v1:172.21.15.72:6789/0 conn(0x7f65940cc360 0x7f6594066e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:51032/0 (socket says 172.21.15.67:51032) 2024-03-24T00:40:44.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.448+0000 7f6593fff640 1 -- 172.21.15.67:0/3047718255 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f659406ba50 msgr2=0x7f65940ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:44.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.448+0000 7f6593fff640 1 --2- 172.21.15.67:0/3047718255 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f659406ba50 0x7f65940ccd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:44.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.448+0000 7f6593fff640 1 -- 172.21.15.67:0/3047718255 >> v1:172.21.15.72:6789/0 conn(0x7f65940cc360 legacy=0x7f6594066e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:44.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.448+0000 7f6593fff640 1 -- 172.21.15.67:0/3047718255 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65940b4240 con 0x7f65940cd240 2024-03-24T00:40:44.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.448+0000 7f658bfff640 1 --2- 172.21.15.67:0/3047718255 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f659406ba50 0x7f65940ccd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:40:44.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.449+0000 7f6593fff640 1 --2- 172.21.15.67:0/3047718255 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f65940cd240 0x7f65940b0c30 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f658c009d90 tx=0x7f658c02bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=50f128cb4d9746f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:44.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.449+0000 7f6592ffd640 1 -- 172.21.15.67:0/3047718255 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f658c039070 con 0x7f65940cd240 2024-03-24T00:40:44.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.450+0000 7f6592ffd640 1 -- 172.21.15.67:0/3047718255 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f658c02cc50 con 0x7f65940cd240 2024-03-24T00:40:44.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.450+0000 7f6592ffd640 1 -- 172.21.15.67:0/3047718255 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f658c034440 con 0x7f65940cd240 2024-03-24T00:40:44.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.450+0000 7f6599f80640 1 -- 172.21.15.67:0/3047718255 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f65940cd240 msgr2=0x7f65940b0c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:44.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.450+0000 7f6599f80640 1 --2- 172.21.15.67:0/3047718255 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f65940cd240 0x7f65940b0c30 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f658c009d90 tx=0x7f658c02bea0 comp rx=0 tx=0).stop 2024-03-24T00:40:44.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.450+0000 7f6599f80640 1 -- 172.21.15.67:0/3047718255 shutdown_connections 2024-03-24T00:40:44.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.450+0000 7f6599f80640 1 --2- 172.21.15.67:0/3047718255 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f65940cd240 0x7f65940b0c30 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:44.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.450+0000 7f6599f80640 1 --2- 172.21.15.67:0/3047718255 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f659406ba50 0x7f65940ccd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:44.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.450+0000 7f6599f80640 1 -- 172.21.15.67:0/3047718255 >> 172.21.15.67:0/3047718255 conn(0x7f65940b5440 msgr2=0x7f65940b33d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:44.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.450+0000 7f6599f80640 1 -- 172.21.15.67:0/3047718255 shutdown_connections 2024-03-24T00:40:44.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.450+0000 7f6599f80640 1 -- 172.21.15.67:0/3047718255 wait complete. 2024-03-24T00:40:44.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.451+0000 7f6599f80640 1 Processor -- start 2024-03-24T00:40:44.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.451+0000 7f6599f80640 1 -- start start 2024-03-24T00:40:44.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.452+0000 7f6599f80640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f659406ba50 0x7f659414afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:44.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.452+0000 7f6599f80640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f65940cc360 0x7f65941e1490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:44.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.452+0000 7f6599f80640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65940cd240 0x7f65940c9050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:44.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.452+0000 7f6599f80640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65940b3f60 con 0x7f65940cd240 2024-03-24T00:40:44.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.452+0000 7f6599f80640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65940b3c80 con 0x7f65940cc360 2024-03-24T00:40:44.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.452+0000 7f6599f80640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65940cc740 con 0x7f659406ba50 2024-03-24T00:40:44.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.452+0000 7f6593fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65940cd240 0x7f65940c9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:44.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.452+0000 7f6593fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65940cd240 0x7f65940c9050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40766/0 (socket says 172.21.15.67:40766) 2024-03-24T00:40:44.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.452+0000 7f6593fff640 1 -- 172.21.15.67:0/853874785 learned_addr learned my addr 172.21.15.67:0/853874785 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:44.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.452+0000 7f65937fe640 1 --2- 172.21.15.67:0/853874785 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f659406ba50 0x7f659414afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:44.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.452+0000 7f658bfff640 1 --2- 172.21.15.67:0/853874785 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f65940cc360 0x7f65941e1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:44.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.453+0000 7f6593fff640 1 -- 172.21.15.67:0/853874785 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f659406ba50 msgr2=0x7f659414afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:44.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.453+0000 7f6593fff640 1 --2- 172.21.15.67:0/853874785 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f659406ba50 0x7f659414afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:44.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.453+0000 7f6593fff640 1 -- 172.21.15.67:0/853874785 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f65940cc360 msgr2=0x7f65941e1490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:44.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.453+0000 7f6593fff640 1 --2- 172.21.15.67:0/853874785 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f65940cc360 0x7f65941e1490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:44.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.453+0000 7f6593fff640 1 -- 172.21.15.67:0/853874785 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f659414bb60 con 0x7f65940cd240 2024-03-24T00:40:44.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.453+0000 7f658bfff640 1 --2- 172.21.15.67:0/853874785 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f65940cc360 0x7f65941e1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:40:44.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.453+0000 7f6593fff640 1 --2- 172.21.15.67:0/853874785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65940cd240 0x7f65940c9050 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f658c009d90 tx=0x7f658c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:44.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.453+0000 7f6590ff9640 1 -- 172.21.15.67:0/853874785 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f658c039050 con 0x7f65940cd240 2024-03-24T00:40:44.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.453+0000 7f6590ff9640 1 -- 172.21.15.67:0/853874785 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f658c045070 con 0x7f65940cd240 2024-03-24T00:40:44.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.453+0000 7f6599f80640 1 -- 172.21.15.67:0/853874785 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f659414be40 con 0x7f65940cd240 2024-03-24T00:40:44.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.453+0000 7f6590ff9640 1 -- 172.21.15.67:0/853874785 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f658c009450 con 0x7f65940cd240 2024-03-24T00:40:44.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.453+0000 7f6599f80640 1 -- 172.21.15.67:0/853874785 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f659414c400 con 0x7f65940cd240 2024-03-24T00:40:44.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.454+0000 7f6599f80640 1 -- 172.21.15.67:0/853874785 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f659414c600 con 0x7f65940cd240 2024-03-24T00:40:44.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.454+0000 7f6590ff9640 1 -- 172.21.15.67:0/853874785 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f658c03e610 con 0x7f65940cd240 2024-03-24T00:40:44.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.455+0000 7f6590ff9640 1 --2- 172.21.15.67:0/853874785 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f657406bbd0 0x7f657406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:44.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.455+0000 7f6590ff9640 1 -- 172.21.15.67:0/853874785 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f658c0ad480 con 0x7f65940cd240 2024-03-24T00:40:44.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.455+0000 7f65937fe640 1 --2- 172.21.15.67:0/853874785 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f657406bbd0 0x7f657406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:44.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.456+0000 7f65937fe640 1 --2- 172.21.15.67:0/853874785 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f657406bbd0 0x7f657406e090 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f6584007fd0 tx=0x7f6584007e80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:44.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.459+0000 7f6590ff9640 1 -- 172.21.15.67:0/853874785 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f659414c600 con 0x7f65940cd240 2024-03-24T00:40:44.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.637+0000 7f6599f80640 1 -- 172.21.15.67:0/853874785 --> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f659404f400 con 0x7f657406bbd0 2024-03-24T00:40:44.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.644+0000 7f6590ff9640 1 -- 172.21.15.67:0/853874785 <== mgr.4105 v2:172.21.15.72:6832/2768292274 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246638 (secure 0 0 0) 0x7f659404f400 con 0x7f657406bbd0 2024-03-24T00:40:44.648 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:40:44.649 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-24T00:40:44.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.646+0000 7f6599f80640 1 -- 172.21.15.67:0/853874785 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f657406bbd0 msgr2=0x7f657406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:44.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.646+0000 7f6599f80640 1 --2- 172.21.15.67:0/853874785 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f657406bbd0 0x7f657406e090 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f6584007fd0 tx=0x7f6584007e80 comp rx=0 tx=0).stop 2024-03-24T00:40:44.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.647+0000 7f6599f80640 1 -- 172.21.15.67:0/853874785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65940cd240 msgr2=0x7f65940c9050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:44.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.647+0000 7f6599f80640 1 --2- 172.21.15.67:0/853874785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65940cd240 0x7f65940c9050 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f658c009d90 tx=0x7f658c004a40 comp rx=0 tx=0).stop 2024-03-24T00:40:44.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.647+0000 7f6599f80640 1 -- 172.21.15.67:0/853874785 shutdown_connections 2024-03-24T00:40:44.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.647+0000 7f6599f80640 1 --2- 172.21.15.67:0/853874785 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f657406bbd0 0x7f657406e090 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:44.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.647+0000 7f6599f80640 1 --2- 172.21.15.67:0/853874785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65940cd240 0x7f65940c9050 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:44.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.647+0000 7f6599f80640 1 --2- 172.21.15.67:0/853874785 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f65940cc360 0x7f65941e1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:44.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.647+0000 7f6599f80640 1 --2- 172.21.15.67:0/853874785 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f659406ba50 0x7f659414afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:44.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.647+0000 7f6599f80640 1 -- 172.21.15.67:0/853874785 >> 172.21.15.67:0/853874785 conn(0x7f65940b5440 msgr2=0x7f65940cac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:44.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.647+0000 7f6599f80640 1 -- 172.21.15.67:0/853874785 shutdown_connections 2024-03-24T00:40:44.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.647+0000 7f6599f80640 1 -- 172.21.15.67:0/853874785 wait complete. 2024-03-24T00:40:44.658 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":105,"stamp":"2024-03-24T00:40:43.345501+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":221684,"kb_used_data":5512,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753036,"statfs":{"total":773094113280,"available":772867108864,"internally_reserved":0,"allocated":5644288,"data_stored":3135232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.066578"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.290314+0000","last_change":"2024-03-24T00:40:39.290314+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.290314+0000","last_undegraded":"2024-03-24T00:40:39.290314+0000","last_fullsized":"2024-03-24T00:40:39.290314+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.290184+0000","last_change":"2024-03-24T00:40:39.290184+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.290184+0000","last_undegraded":"2024-03-24T00:40:39.290184+0000","last_fullsized":"2024-03-24T00:40:39.290184+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.290644+0000","last_change":"2024-03-24T00:40:39.290644+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.290644+0000","last_undegraded":"2024-03-24T00:40:39.290644+0000","last_fullsized":"2024-03-24T00:40:39.290644+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.286759+0000","last_change":"2024-03-24T00:40:39.286759+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.286759+0000","last_undegraded":"2024-03-24T00:40:39.286759+0000","last_fullsized":"2024-03-24T00:40:39.286759+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.288142+0000","last_change":"2024-03-24T00:40:39.288142+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.288142+0000","last_undegraded":"2024-03-24T00:40:39.288142+0000","last_fullsized":"2024-03-24T00:40:39.288142+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.289818+0000","last_change":"2024-03-24T00:40:39.289818+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.289818+0000","last_undegraded":"2024-03-24T00:40:39.289818+0000","last_fullsized":"2024-03-24T00:40:39.289818+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.292120+0000","last_change":"2024-03-24T00:40:39.292120+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.292120+0000","last_undegraded":"2024-03-24T00:40:39.292120+0000","last_fullsized":"2024-03-24T00:40:39.292120+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.285842+0000","last_change":"2024-03-24T00:40:39.285842+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.285842+0000","last_undegraded":"2024-03-24T00:40:39.285842+0000","last_fullsized":"2024-03-24T00:40:39.285842+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.287182+0000","last_change":"2024-03-24T00:40:39.287182+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.287182+0000","last_undegraded":"2024-03-24T00:40:39.287182+0000","last_fullsized":"2024-03-24T00:40:39.287182+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.287090+0000","last_change":"2024-03-24T00:40:39.287090+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.287090+0000","last_undegraded":"2024-03-24T00:40:39.287090+0000","last_fullsized":"2024-03-24T00:40:39.287090+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.293731+0000","last_change":"2024-03-24T00:40:39.293731+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.293731+0000","last_undegraded":"2024-03-24T00:40:39.293731+0000","last_fullsized":"2024-03-24T00:40:39.293731+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.291962+0000","last_change":"2024-03-24T00:40:39.291962+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.291962+0000","last_undegraded":"2024-03-24T00:40:39.291962+0000","last_fullsized":"2024-03-24T00:40:39.291962+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.285910+0000","last_change":"2024-03-24T00:40:39.285910+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.285910+0000","last_undegraded":"2024-03-24T00:40:39.285910+0000","last_fullsized":"2024-03-24T00:40:39.285910+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.291223+0000","last_change":"2024-03-24T00:40:39.291223+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.291223+0000","last_undegraded":"2024-03-24T00:40:39.291223+0000","last_fullsized":"2024-03-24T00:40:39.291223+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.287287+0000","last_change":"2024-03-24T00:40:39.287287+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.287287+0000","last_undegraded":"2024-03-24T00:40:39.287287+0000","last_fullsized":"2024-03-24T00:40:39.287287+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.286183+0000","last_change":"2024-03-24T00:40:39.286183+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.286183+0000","last_undegraded":"2024-03-24T00:40:39.286183+0000","last_fullsized":"2024-03-24T00:40:39.286183+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.290840+0000","last_change":"2024-03-24T00:40:39.290840+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.290840+0000","last_undegraded":"2024-03-24T00:40:39.290840+0000","last_fullsized":"2024-03-24T00:40:39.290840+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.287128+0000","last_change":"2024-03-24T00:40:39.287128+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.287128+0000","last_undegraded":"2024-03-24T00:40:39.287128+0000","last_fullsized":"2024-03-24T00:40:39.287128+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.290088+0000","last_change":"2024-03-24T00:40:39.290088+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.290088+0000","last_undegraded":"2024-03-24T00:40:39.290088+0000","last_fullsized":"2024-03-24T00:40:39.290088+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.287773+0000","last_change":"2024-03-24T00:40:39.287773+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.287773+0000","last_undegraded":"2024-03-24T00:40:39.287773+0000","last_fullsized":"2024-03-24T00:40:39.287773+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.289216+0000","last_change":"2024-03-24T00:40:39.289216+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.289216+0000","last_undegraded":"2024-03-24T00:40:39.289216+0000","last_fullsized":"2024-03-24T00:40:39.289216+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.286528+0000","last_change":"2024-03-24T00:40:39.286528+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.286528+0000","last_undegraded":"2024-03-24T00:40:39.286528+0000","last_fullsized":"2024-03-24T00:40:39.286528+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.293164+0000","last_change":"2024-03-24T00:40:39.293164+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.293164+0000","last_undegraded":"2024-03-24T00:40:39.293164+0000","last_fullsized":"2024-03-24T00:40:39.293164+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.289958+0000","last_change":"2024-03-24T00:40:39.289958+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.289958+0000","last_undegraded":"2024-03-24T00:40:39.289958+0000","last_fullsized":"2024-03-24T00:40:39.289958+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.290445+0000","last_change":"2024-03-24T00:40:39.290445+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.290445+0000","last_undegraded":"2024-03-24T00:40:39.290445+0000","last_fullsized":"2024-03-24T00:40:39.290445+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.287297+0000","last_change":"2024-03-24T00:40:39.287297+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.287297+0000","last_undegraded":"2024-03-24T00:40:39.287297+0000","last_fullsized":"2024-03-24T00:40:39.287297+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.286083+0000","last_change":"2024-03-24T00:40:39.286083+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.286083+0000","last_undegraded":"2024-03-24T00:40:39.286083+0000","last_fullsized":"2024-03-24T00:40:39.286083+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.287284+0000","last_change":"2024-03-24T00:40:39.287284+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.287284+0000","last_undegraded":"2024-03-24T00:40:39.287284+0000","last_fullsized":"2024-03-24T00:40:39.287284+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.291689+0000","last_change":"2024-03-24T00:40:39.291689+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.291689+0000","last_undegraded":"2024-03-24T00:40:39.291689+0000","last_fullsized":"2024-03-24T00:40:39.291689+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.286764+0000","last_change":"2024-03-24T00:40:39.286764+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.286764+0000","last_undegraded":"2024-03-24T00:40:39.286764+0000","last_fullsized":"2024-03-24T00:40:39.286764+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.289986+0000","last_change":"2024-03-24T00:40:39.289986+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.289986+0000","last_undegraded":"2024-03-24T00:40:39.289986+0000","last_fullsized":"2024-03-24T00:40:39.289986+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.284737+0000","last_change":"2024-03-24T00:40:39.284737+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.284737+0000","last_undegraded":"2024-03-24T00:40:39.284737+0000","last_fullsized":"2024-03-24T00:40:39.284737+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.290205+0000","last_change":"2024-03-24T00:40:39.290205+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.290205+0000","last_undegraded":"2024-03-24T00:40:39.290205+0000","last_fullsized":"2024-03-24T00:40:39.290205+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.293699+0000","last_change":"2024-03-24T00:40:39.291631+0000","last_active":"2024-03-24T00:40:39.293699+0000","last_peered":"2024-03-24T00:40:39.293699+0000","last_clean":"2024-03-24T00:40:39.293699+0000","last_became_active":"2024-03-24T00:40:39.291349+0000","last_became_peered":"2024-03-24T00:40:39.291349+0000","last_unstale":"2024-03-24T00:40:39.293699+0000","last_undegraded":"2024-03-24T00:40:39.293699+0000","last_fullsized":"2024-03-24T00:40:39.293699+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:32:08.714068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.283698+0000","last_change":"2024-03-24T00:40:39.283698+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.283698+0000","last_undegraded":"2024-03-24T00:40:39.283698+0000","last_fullsized":"2024-03-24T00:40:39.283698+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.297930+0000","last_change":"2024-03-24T00:40:39.295765+0000","last_active":"2024-03-24T00:40:39.297930+0000","last_peered":"2024-03-24T00:40:39.297930+0000","last_clean":"2024-03-24T00:40:39.297930+0000","last_became_active":"2024-03-24T00:40:39.295486+0000","last_became_peered":"2024-03-24T00:40:39.295486+0000","last_unstale":"2024-03-24T00:40:39.297930+0000","last_undegraded":"2024-03-24T00:40:39.297930+0000","last_fullsized":"2024-03-24T00:40:39.297930+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:09:53.820800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.289688+0000","last_change":"2024-03-24T00:40:39.289688+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.289688+0000","last_undegraded":"2024-03-24T00:40:39.289688+0000","last_fullsized":"2024-03-24T00:40:39.289688+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.295662+0000","last_change":"2024-03-24T00:40:39.294921+0000","last_active":"2024-03-24T00:40:39.295662+0000","last_peered":"2024-03-24T00:40:39.295662+0000","last_clean":"2024-03-24T00:40:39.295662+0000","last_became_active":"2024-03-24T00:40:39.294579+0000","last_became_peered":"2024-03-24T00:40:39.294579+0000","last_unstale":"2024-03-24T00:40:39.295662+0000","last_undegraded":"2024-03-24T00:40:39.295662+0000","last_fullsized":"2024-03-24T00:40:39.295662+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:54:49.331324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.295225+0000","last_change":"2024-03-24T00:40:39.295225+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.295225+0000","last_undegraded":"2024-03-24T00:40:39.295225+0000","last_fullsized":"2024-03-24T00:40:39.295225+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.299268+0000","last_change":"2024-03-24T00:40:39.297839+0000","last_active":"2024-03-24T00:40:39.299268+0000","last_peered":"2024-03-24T00:40:39.299268+0000","last_clean":"2024-03-24T00:40:39.299268+0000","last_became_active":"2024-03-24T00:40:39.297536+0000","last_became_peered":"2024-03-24T00:40:39.297536+0000","last_unstale":"2024-03-24T00:40:39.299268+0000","last_undegraded":"2024-03-24T00:40:39.299268+0000","last_fullsized":"2024-03-24T00:40:39.299268+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:58:21.161844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.286437+0000","last_change":"2024-03-24T00:40:39.286437+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.286437+0000","last_undegraded":"2024-03-24T00:40:39.286437+0000","last_fullsized":"2024-03-24T00:40:39.286437+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.305288+0000","last_change":"2024-03-24T00:40:39.304061+0000","last_active":"2024-03-24T00:40:39.305288+0000","last_peered":"2024-03-24T00:40:39.305288+0000","last_clean":"2024-03-24T00:40:39.305288+0000","last_became_active":"2024-03-24T00:40:39.303769+0000","last_became_peered":"2024-03-24T00:40:39.303769+0000","last_unstale":"2024-03-24T00:40:39.305288+0000","last_undegraded":"2024-03-24T00:40:39.305288+0000","last_fullsized":"2024-03-24T00:40:39.305288+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:00:51.940556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.286866+0000","last_change":"2024-03-24T00:40:39.286866+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.286866+0000","last_undegraded":"2024-03-24T00:40:39.286866+0000","last_fullsized":"2024-03-24T00:40:39.286866+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.295132+0000","last_change":"2024-03-24T00:40:39.293463+0000","last_active":"2024-03-24T00:40:39.295132+0000","last_peered":"2024-03-24T00:40:39.295132+0000","last_clean":"2024-03-24T00:40:39.295132+0000","last_became_active":"2024-03-24T00:40:39.293196+0000","last_became_peered":"2024-03-24T00:40:39.293196+0000","last_unstale":"2024-03-24T00:40:39.295132+0000","last_undegraded":"2024-03-24T00:40:39.295132+0000","last_fullsized":"2024-03-24T00:40:39.295132+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:16:22.016263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.285844+0000","last_change":"2024-03-24T00:40:39.285844+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.285844+0000","last_undegraded":"2024-03-24T00:40:39.285844+0000","last_fullsized":"2024-03-24T00:40:39.285844+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.296000+0000","last_change":"2024-03-24T00:40:39.294468+0000","last_active":"2024-03-24T00:40:39.296000+0000","last_peered":"2024-03-24T00:40:39.296000+0000","last_clean":"2024-03-24T00:40:39.296000+0000","last_became_active":"2024-03-24T00:40:39.294130+0000","last_became_peered":"2024-03-24T00:40:39.294130+0000","last_unstale":"2024-03-24T00:40:39.296000+0000","last_undegraded":"2024-03-24T00:40:39.296000+0000","last_fullsized":"2024-03-24T00:40:39.296000+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:52:39.705988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.290406+0000","last_change":"2024-03-24T00:40:39.290406+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.290406+0000","last_undegraded":"2024-03-24T00:40:39.290406+0000","last_fullsized":"2024-03-24T00:40:39.290406+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.298578+0000","last_change":"2024-03-24T00:40:39.296539+0000","last_active":"2024-03-24T00:40:39.298578+0000","last_peered":"2024-03-24T00:40:39.298578+0000","last_clean":"2024-03-24T00:40:39.298578+0000","last_became_active":"2024-03-24T00:40:39.296272+0000","last_became_peered":"2024-03-24T00:40:39.296272+0000","last_unstale":"2024-03-24T00:40:39.298578+0000","last_undegraded":"2024-03-24T00:40:39.298578+0000","last_fullsized":"2024-03-24T00:40:39.298578+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:09:53.820800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.286586+0000","last_change":"2024-03-24T00:40:39.286586+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.286586+0000","last_undegraded":"2024-03-24T00:40:39.286586+0000","last_fullsized":"2024-03-24T00:40:39.286586+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.297968+0000","last_change":"2024-03-24T00:40:39.296067+0000","last_active":"2024-03-24T00:40:39.297968+0000","last_peered":"2024-03-24T00:40:39.297968+0000","last_clean":"2024-03-24T00:40:39.297968+0000","last_became_active":"2024-03-24T00:40:39.295773+0000","last_became_peered":"2024-03-24T00:40:39.295773+0000","last_unstale":"2024-03-24T00:40:39.297968+0000","last_undegraded":"2024-03-24T00:40:39.297968+0000","last_fullsized":"2024-03-24T00:40:39.297968+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:53:34.136108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.284282+0000","last_change":"2024-03-24T00:40:39.284282+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.284282+0000","last_undegraded":"2024-03-24T00:40:39.284282+0000","last_fullsized":"2024-03-24T00:40:39.284282+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.298719+0000","last_change":"2024-03-24T00:40:39.296711+0000","last_active":"2024-03-24T00:40:39.298719+0000","last_peered":"2024-03-24T00:40:39.298719+0000","last_clean":"2024-03-24T00:40:39.298719+0000","last_became_active":"2024-03-24T00:40:39.296435+0000","last_became_peered":"2024-03-24T00:40:39.296435+0000","last_unstale":"2024-03-24T00:40:39.298719+0000","last_undegraded":"2024-03-24T00:40:39.298719+0000","last_fullsized":"2024-03-24T00:40:39.298719+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:54:49.331324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.285258+0000","last_change":"2024-03-24T00:40:39.285258+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.285258+0000","last_undegraded":"2024-03-24T00:40:39.285258+0000","last_fullsized":"2024-03-24T00:40:39.285258+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.303989+0000","last_change":"2024-03-24T00:40:39.301123+0000","last_active":"2024-03-24T00:40:39.303989+0000","last_peered":"2024-03-24T00:40:39.303989+0000","last_clean":"2024-03-24T00:40:39.303989+0000","last_became_active":"2024-03-24T00:40:39.300851+0000","last_became_peered":"2024-03-24T00:40:39.300851+0000","last_unstale":"2024-03-24T00:40:39.303989+0000","last_undegraded":"2024-03-24T00:40:39.303989+0000","last_fullsized":"2024-03-24T00:40:39.303989+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:00:51.940556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.287678+0000","last_change":"2024-03-24T00:40:39.287678+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.287678+0000","last_undegraded":"2024-03-24T00:40:39.287678+0000","last_fullsized":"2024-03-24T00:40:39.287678+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.304078+0000","last_change":"2024-03-24T00:40:39.302117+0000","last_active":"2024-03-24T00:40:39.304078+0000","last_peered":"2024-03-24T00:40:39.304078+0000","last_clean":"2024-03-24T00:40:39.304078+0000","last_became_active":"2024-03-24T00:40:39.301875+0000","last_became_peered":"2024-03-24T00:40:39.301875+0000","last_unstale":"2024-03-24T00:40:39.304078+0000","last_undegraded":"2024-03-24T00:40:39.304078+0000","last_fullsized":"2024-03-24T00:40:39.304078+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:48:02.396386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.284454+0000","last_change":"2024-03-24T00:40:39.284454+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.284454+0000","last_undegraded":"2024-03-24T00:40:39.284454+0000","last_fullsized":"2024-03-24T00:40:39.284454+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.303378+0000","last_change":"2024-03-24T00:40:39.298667+0000","last_active":"2024-03-24T00:40:39.303378+0000","last_peered":"2024-03-24T00:40:39.303378+0000","last_clean":"2024-03-24T00:40:39.303378+0000","last_became_active":"2024-03-24T00:40:39.298067+0000","last_became_peered":"2024-03-24T00:40:39.298067+0000","last_unstale":"2024-03-24T00:40:39.303378+0000","last_undegraded":"2024-03-24T00:40:39.303378+0000","last_fullsized":"2024-03-24T00:40:39.303378+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:35.720596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.286589+0000","last_change":"2024-03-24T00:40:39.286589+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.286589+0000","last_undegraded":"2024-03-24T00:40:39.286589+0000","last_fullsized":"2024-03-24T00:40:39.286589+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.301055+0000","last_change":"2024-03-24T00:40:39.299651+0000","last_active":"2024-03-24T00:40:39.301055+0000","last_peered":"2024-03-24T00:40:39.301055+0000","last_clean":"2024-03-24T00:40:39.301055+0000","last_became_active":"2024-03-24T00:40:39.299388+0000","last_became_peered":"2024-03-24T00:40:39.299388+0000","last_unstale":"2024-03-24T00:40:39.301055+0000","last_undegraded":"2024-03-24T00:40:39.301055+0000","last_fullsized":"2024-03-24T00:40:39.301055+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:30:39.662599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.286895+0000","last_change":"2024-03-24T00:40:39.286895+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.286895+0000","last_undegraded":"2024-03-24T00:40:39.286895+0000","last_fullsized":"2024-03-24T00:40:39.286895+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.298000+0000","last_change":"2024-03-24T00:40:39.295906+0000","last_active":"2024-03-24T00:40:39.298000+0000","last_peered":"2024-03-24T00:40:39.298000+0000","last_clean":"2024-03-24T00:40:39.298000+0000","last_became_active":"2024-03-24T00:40:39.295604+0000","last_became_peered":"2024-03-24T00:40:39.295604+0000","last_unstale":"2024-03-24T00:40:39.298000+0000","last_undegraded":"2024-03-24T00:40:39.298000+0000","last_fullsized":"2024-03-24T00:40:39.298000+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:35.720596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.288224+0000","last_change":"2024-03-24T00:40:39.288224+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.288224+0000","last_undegraded":"2024-03-24T00:40:39.288224+0000","last_fullsized":"2024-03-24T00:40:39.288224+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.302950+0000","last_change":"2024-03-24T00:40:39.298619+0000","last_active":"2024-03-24T00:40:39.302950+0000","last_peered":"2024-03-24T00:40:39.302950+0000","last_clean":"2024-03-24T00:40:39.302950+0000","last_became_active":"2024-03-24T00:40:39.298350+0000","last_became_peered":"2024-03-24T00:40:39.298350+0000","last_unstale":"2024-03-24T00:40:39.302950+0000","last_undegraded":"2024-03-24T00:40:39.302950+0000","last_fullsized":"2024-03-24T00:40:39.302950+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:04:49.487506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.285444+0000","last_change":"2024-03-24T00:40:39.285444+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.285444+0000","last_undegraded":"2024-03-24T00:40:39.285444+0000","last_fullsized":"2024-03-24T00:40:39.285444+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.303365+0000","last_change":"2024-03-24T00:40:39.299640+0000","last_active":"2024-03-24T00:40:39.303365+0000","last_peered":"2024-03-24T00:40:39.303365+0000","last_clean":"2024-03-24T00:40:39.303365+0000","last_became_active":"2024-03-24T00:40:39.299375+0000","last_became_peered":"2024-03-24T00:40:39.299375+0000","last_unstale":"2024-03-24T00:40:39.303365+0000","last_undegraded":"2024-03-24T00:40:39.303365+0000","last_fullsized":"2024-03-24T00:40:39.303365+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:53:34.136108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.284942+0000","last_change":"2024-03-24T00:40:39.284942+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.284942+0000","last_undegraded":"2024-03-24T00:40:39.284942+0000","last_fullsized":"2024-03-24T00:40:39.284942+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.303123+0000","last_change":"2024-03-24T00:40:39.302072+0000","last_active":"2024-03-24T00:40:39.303123+0000","last_peered":"2024-03-24T00:40:39.303123+0000","last_clean":"2024-03-24T00:40:39.303123+0000","last_became_active":"2024-03-24T00:40:39.301790+0000","last_became_peered":"2024-03-24T00:40:39.301790+0000","last_unstale":"2024-03-24T00:40:39.303123+0000","last_undegraded":"2024-03-24T00:40:39.303123+0000","last_fullsized":"2024-03-24T00:40:39.303123+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:48:22.534145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.287303+0000","last_change":"2024-03-24T00:40:39.287303+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.287303+0000","last_undegraded":"2024-03-24T00:40:39.287303+0000","last_fullsized":"2024-03-24T00:40:39.287303+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.293730+0000","last_change":"2024-03-24T00:40:39.292046+0000","last_active":"2024-03-24T00:40:39.293730+0000","last_peered":"2024-03-24T00:40:39.293730+0000","last_clean":"2024-03-24T00:40:39.293730+0000","last_became_active":"2024-03-24T00:40:39.291744+0000","last_became_peered":"2024-03-24T00:40:39.291744+0000","last_unstale":"2024-03-24T00:40:39.293730+0000","last_undegraded":"2024-03-24T00:40:39.293730+0000","last_fullsized":"2024-03-24T00:40:39.293730+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:05:15.049538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.284674+0000","last_change":"2024-03-24T00:40:39.284674+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.284674+0000","last_undegraded":"2024-03-24T00:40:39.284674+0000","last_fullsized":"2024-03-24T00:40:39.284674+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.309077+0000","last_change":"2024-03-24T00:40:39.304591+0000","last_active":"2024-03-24T00:40:39.309077+0000","last_peered":"2024-03-24T00:40:39.309077+0000","last_clean":"2024-03-24T00:40:39.309077+0000","last_became_active":"2024-03-24T00:40:39.304330+0000","last_became_peered":"2024-03-24T00:40:39.304330+0000","last_unstale":"2024-03-24T00:40:39.309077+0000","last_undegraded":"2024-03-24T00:40:39.309077+0000","last_fullsized":"2024-03-24T00:40:39.309077+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:25:48.533462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.283875+0000","last_change":"2024-03-24T00:40:39.283875+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.283875+0000","last_undegraded":"2024-03-24T00:40:39.283875+0000","last_fullsized":"2024-03-24T00:40:39.283875+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.289779+0000","last_change":"2024-03-24T00:40:39.287554+0000","last_active":"2024-03-24T00:40:39.289779+0000","last_peered":"2024-03-24T00:40:39.289779+0000","last_clean":"2024-03-24T00:40:39.289779+0000","last_became_active":"2024-03-24T00:40:39.287273+0000","last_became_peered":"2024-03-24T00:40:39.287273+0000","last_unstale":"2024-03-24T00:40:39.289779+0000","last_undegraded":"2024-03-24T00:40:39.289779+0000","last_fullsized":"2024-03-24T00:40:39.289779+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:16:22.016263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.289204+0000","last_change":"2024-03-24T00:40:39.289204+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.289204+0000","last_undegraded":"2024-03-24T00:40:39.289204+0000","last_fullsized":"2024-03-24T00:40:39.289204+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.302927+0000","last_change":"2024-03-24T00:40:39.298506+0000","last_active":"2024-03-24T00:40:39.302927+0000","last_peered":"2024-03-24T00:40:39.302927+0000","last_clean":"2024-03-24T00:40:39.302927+0000","last_became_active":"2024-03-24T00:40:39.297935+0000","last_became_peered":"2024-03-24T00:40:39.297935+0000","last_unstale":"2024-03-24T00:40:39.302927+0000","last_undegraded":"2024-03-24T00:40:39.302927+0000","last_fullsized":"2024-03-24T00:40:39.302927+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:23:25.616752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.287275+0000","last_change":"2024-03-24T00:40:39.287275+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.287275+0000","last_undegraded":"2024-03-24T00:40:39.287275+0000","last_fullsized":"2024-03-24T00:40:39.287275+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.298445+0000","last_change":"2024-03-24T00:40:39.296787+0000","last_active":"2024-03-24T00:40:39.298445+0000","last_peered":"2024-03-24T00:40:39.298445+0000","last_clean":"2024-03-24T00:40:39.298445+0000","last_became_active":"2024-03-24T00:40:39.296501+0000","last_became_peered":"2024-03-24T00:40:39.296501+0000","last_unstale":"2024-03-24T00:40:39.298445+0000","last_undegraded":"2024-03-24T00:40:39.298445+0000","last_fullsized":"2024-03-24T00:40:39.298445+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:43:07.435024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.285594+0000","last_change":"2024-03-24T00:40:39.285594+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.285594+0000","last_undegraded":"2024-03-24T00:40:39.285594+0000","last_fullsized":"2024-03-24T00:40:39.285594+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.302585+0000","last_change":"2024-03-24T00:40:39.300361+0000","last_active":"2024-03-24T00:40:39.302585+0000","last_peered":"2024-03-24T00:40:39.302585+0000","last_clean":"2024-03-24T00:40:39.302585+0000","last_became_active":"2024-03-24T00:40:39.299954+0000","last_became_peered":"2024-03-24T00:40:39.299954+0000","last_unstale":"2024-03-24T00:40:39.302585+0000","last_undegraded":"2024-03-24T00:40:39.302585+0000","last_fullsized":"2024-03-24T00:40:39.302585+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:35:30.101713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.283792+0000","last_change":"2024-03-24T00:40:39.283792+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.283792+0000","last_undegraded":"2024-03-24T00:40:39.283792+0000","last_fullsized":"2024-03-24T00:40:39.283792+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.300944+0000","last_change":"2024-03-24T00:40:39.299797+0000","last_active":"2024-03-24T00:40:39.300944+0000","last_peered":"2024-03-24T00:40:39.300944+0000","last_clean":"2024-03-24T00:40:39.300944+0000","last_became_active":"2024-03-24T00:40:39.299456+0000","last_became_peered":"2024-03-24T00:40:39.299456+0000","last_unstale":"2024-03-24T00:40:39.300944+0000","last_undegraded":"2024-03-24T00:40:39.300944+0000","last_fullsized":"2024-03-24T00:40:39.300944+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:28:27.547221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.285596+0000","last_change":"2024-03-24T00:40:39.285596+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.285596+0000","last_undegraded":"2024-03-24T00:40:39.285596+0000","last_fullsized":"2024-03-24T00:40:39.285596+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.299923+0000","last_change":"2024-03-24T00:40:39.299014+0000","last_active":"2024-03-24T00:40:39.299923+0000","last_peered":"2024-03-24T00:40:39.299923+0000","last_clean":"2024-03-24T00:40:39.299923+0000","last_became_active":"2024-03-24T00:40:39.298750+0000","last_became_peered":"2024-03-24T00:40:39.298750+0000","last_unstale":"2024-03-24T00:40:39.299923+0000","last_undegraded":"2024-03-24T00:40:39.299923+0000","last_fullsized":"2024-03-24T00:40:39.299923+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:08:19.135822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.289013+0000","last_change":"2024-03-24T00:40:39.289013+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.289013+0000","last_undegraded":"2024-03-24T00:40:39.289013+0000","last_fullsized":"2024-03-24T00:40:39.289013+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.304762+0000","last_change":"2024-03-24T00:40:39.303224+0000","last_active":"2024-03-24T00:40:39.304762+0000","last_peered":"2024-03-24T00:40:39.304762+0000","last_clean":"2024-03-24T00:40:39.304762+0000","last_became_active":"2024-03-24T00:40:39.302950+0000","last_became_peered":"2024-03-24T00:40:39.302950+0000","last_unstale":"2024-03-24T00:40:39.304762+0000","last_undegraded":"2024-03-24T00:40:39.304762+0000","last_fullsized":"2024-03-24T00:40:39.304762+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:08:19.135822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.285991+0000","last_change":"2024-03-24T00:40:39.285991+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.285991+0000","last_undegraded":"2024-03-24T00:40:39.285991+0000","last_fullsized":"2024-03-24T00:40:39.285991+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.304018+0000","last_change":"2024-03-24T00:40:39.299484+0000","last_active":"2024-03-24T00:40:39.304018+0000","last_peered":"2024-03-24T00:40:39.304018+0000","last_clean":"2024-03-24T00:40:39.304018+0000","last_became_active":"2024-03-24T00:40:39.299184+0000","last_became_peered":"2024-03-24T00:40:39.299184+0000","last_unstale":"2024-03-24T00:40:39.304018+0000","last_undegraded":"2024-03-24T00:40:39.304018+0000","last_fullsized":"2024-03-24T00:40:39.304018+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:57:35.870209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.283715+0000","last_change":"2024-03-24T00:40:39.283715+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.283715+0000","last_undegraded":"2024-03-24T00:40:39.283715+0000","last_fullsized":"2024-03-24T00:40:39.283715+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.302907+0000","last_change":"2024-03-24T00:40:39.298278+0000","last_active":"2024-03-24T00:40:39.302907+0000","last_peered":"2024-03-24T00:40:39.302907+0000","last_clean":"2024-03-24T00:40:39.302907+0000","last_became_active":"2024-03-24T00:40:39.297588+0000","last_became_peered":"2024-03-24T00:40:39.297588+0000","last_unstale":"2024-03-24T00:40:39.302907+0000","last_undegraded":"2024-03-24T00:40:39.302907+0000","last_fullsized":"2024-03-24T00:40:39.302907+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:06:05.426065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.287710+0000","last_change":"2024-03-24T00:40:39.287710+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.287710+0000","last_undegraded":"2024-03-24T00:40:39.287710+0000","last_fullsized":"2024-03-24T00:40:39.287710+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.301019+0000","last_change":"2024-03-24T00:40:39.300219+0000","last_active":"2024-03-24T00:40:39.301019+0000","last_peered":"2024-03-24T00:40:39.301019+0000","last_clean":"2024-03-24T00:40:39.301019+0000","last_became_active":"2024-03-24T00:40:39.299897+0000","last_became_peered":"2024-03-24T00:40:39.299897+0000","last_unstale":"2024-03-24T00:40:39.301019+0000","last_undegraded":"2024-03-24T00:40:39.301019+0000","last_fullsized":"2024-03-24T00:40:39.301019+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:09:53.820800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.286808+0000","last_change":"2024-03-24T00:40:39.286808+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.286808+0000","last_undegraded":"2024-03-24T00:40:39.286808+0000","last_fullsized":"2024-03-24T00:40:39.286808+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.295357+0000","last_change":"2024-03-24T00:40:39.294249+0000","last_active":"2024-03-24T00:40:39.295357+0000","last_peered":"2024-03-24T00:40:39.295357+0000","last_clean":"2024-03-24T00:40:39.295357+0000","last_became_active":"2024-03-24T00:40:39.293915+0000","last_became_peered":"2024-03-24T00:40:39.293915+0000","last_unstale":"2024-03-24T00:40:39.295357+0000","last_undegraded":"2024-03-24T00:40:39.295357+0000","last_fullsized":"2024-03-24T00:40:39.295357+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:05:32.540043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.289541+0000","last_change":"2024-03-24T00:40:39.289541+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.289541+0000","last_undegraded":"2024-03-24T00:40:39.289541+0000","last_fullsized":"2024-03-24T00:40:39.289541+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.302298+0000","last_change":"2024-03-24T00:40:39.297685+0000","last_active":"2024-03-24T00:40:39.302298+0000","last_peered":"2024-03-24T00:40:39.302298+0000","last_clean":"2024-03-24T00:40:39.302298+0000","last_became_active":"2024-03-24T00:40:39.297097+0000","last_became_peered":"2024-03-24T00:40:39.297097+0000","last_unstale":"2024-03-24T00:40:39.302298+0000","last_undegraded":"2024-03-24T00:40:39.302298+0000","last_fullsized":"2024-03-24T00:40:39.302298+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:42:28.507176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":220,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.291524+0000","last_change":"2024-03-24T00:38:40.953871+0000","last_active":"2024-03-24T00:40:39.291524+0000","last_peered":"2024-03-24T00:40:39.291524+0000","last_clean":"2024-03-24T00:40:39.291524+0000","last_became_active":"2024-03-24T00:38:40.952926+0000","last_became_peered":"2024-03-24T00:38:40.952926+0000","last_unstale":"2024-03-24T00:40:39.291524+0000","last_undegraded":"2024-03-24T00:40:39.291524+0000","last_fullsized":"2024-03-24T00:40:39.291524+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_clean_scrub_stamp":"2024-03-24T00:38:39.722194+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:05.397257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574877,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28144,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343696,"statfs":{"total":96636764160,"available":96607944704,"internally_reserved":0,"allocated":1150976,"data_stored":834680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27568,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344272,"statfs":{"total":96636764160,"available":96608534528,"internally_reserved":0,"allocated":557056,"data_stored":244312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":557056,"data_stored":244312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":557056,"data_stored":244312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574875,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27568,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344272,"statfs":{"total":96636764160,"available":96608534528,"internally_reserved":0,"allocated":557056,"data_stored":244312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27568,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344272,"statfs":{"total":96636764160,"available":96608534528,"internally_reserved":0,"allocated":557056,"data_stored":244312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":557056,"data_stored":244312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574875,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28144,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343696,"statfs":{"total":96636764160,"available":96607944704,"internally_reserved":0,"allocated":1150976,"data_stored":834680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:40:44.662 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:40:44.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.799+0000 7f78afba9640 1 Processor -- start 2024-03-24T00:40:44.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.799+0000 7f78afba9640 1 -- start start 2024-03-24T00:40:44.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.801+0000 7f78afba9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f78a80b15d0 0x7f78a80cd710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:44.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.801+0000 7f78afba9640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f78a80b19b0 0x7f78a815d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:44.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.801+0000 7f78afba9640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78a815e0a0 con 0x7f78a815dc50 2024-03-24T00:40:44.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.801+0000 7f78afba9640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78a80cb100 con 0x7f78a80b15d0 2024-03-24T00:40:44.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.801+0000 7f78afba9640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78a80b4270 con 0x7f78a80b19b0 2024-03-24T00:40:44.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.801+0000 7f78ad945640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f78a80b15d0 0x7f78a80cd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:44.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.801+0000 7f78ae146640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f78a815dc50 0x7f78a806bd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39616/0 (socket says 172.21.15.67:39616) 2024-03-24T00:40:44.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.801+0000 7f78ae146640 1 -- 172.21.15.67:0/2743461681 learned_addr learned my addr 172.21.15.67:0/2743461681 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:40:44.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.801+0000 7f78ad144640 1 --2- 172.21.15.67:0/2743461681 >> v2:172.21.15.72:3300/0 conn(0x7f78a80b19b0 0x7f78a815d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:44.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.801+0000 7f78ad945640 1 -- 172.21.15.67:0/2743461681 >> v2:172.21.15.72:3300/0 conn(0x7f78a80b19b0 msgr2=0x7f78a815d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:44.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.801+0000 7f78ad945640 1 --2- 172.21.15.67:0/2743461681 >> v2:172.21.15.72:3300/0 conn(0x7f78a80b19b0 0x7f78a815d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:44.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.801+0000 7f78ad945640 1 -- 172.21.15.67:0/2743461681 >> v1:172.21.15.67:6789/0 conn(0x7f78a815dc50 legacy=0x7f78a806bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:44.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.801+0000 7f78ad945640 1 -- 172.21.15.67:0/2743461681 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a80caad0 con 0x7f78a80b15d0 2024-03-24T00:40:44.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.801+0000 7f78ac943640 1 -- 172.21.15.67:0/2743461681 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2237591550 0 0) 0x7f78a0001490 con 0x7f78a815dc50 2024-03-24T00:40:44.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.801+0000 7f78ad144640 1 --2- 172.21.15.67:0/2743461681 >> v2:172.21.15.72:3300/0 conn(0x7f78a80b19b0 0x7f78a815d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:40:44.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.802+0000 7f78ad945640 1 --2- 172.21.15.67:0/2743461681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78a80b15d0 0x7f78a80cd710 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f78a8072d60 tx=0x7f78a402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=412ee4a856915a05 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:44.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.802+0000 7f78ac943640 1 -- 172.21.15.67:0/2743461681 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a40315a0 con 0x7f78a80b15d0 2024-03-24T00:40:44.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.802+0000 7f78ac943640 1 -- 172.21.15.67:0/2743461681 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78a4037070 con 0x7f78a80b15d0 2024-03-24T00:40:44.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.802+0000 7f78ac943640 1 -- 172.21.15.67:0/2743461681 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a4031960 con 0x7f78a80b15d0 2024-03-24T00:40:44.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.803+0000 7f78afba9640 1 -- 172.21.15.67:0/2743461681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78a80b15d0 msgr2=0x7f78a80cd710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:44.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.803+0000 7f78afba9640 1 --2- 172.21.15.67:0/2743461681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78a80b15d0 0x7f78a80cd710 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f78a8072d60 tx=0x7f78a402b840 comp rx=0 tx=0).stop 2024-03-24T00:40:44.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.803+0000 7f78afba9640 1 -- 172.21.15.67:0/2743461681 shutdown_connections 2024-03-24T00:40:44.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.803+0000 7f78afba9640 1 --2- 172.21.15.67:0/2743461681 >> v2:172.21.15.72:3300/0 conn(0x7f78a80b19b0 0x7f78a815d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:44.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.803+0000 7f78afba9640 1 --2- 172.21.15.67:0/2743461681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78a80b15d0 0x7f78a80cd710 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:44.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.803+0000 7f78afba9640 1 -- 172.21.15.67:0/2743461681 >> 172.21.15.67:0/2743461681 conn(0x7f78a80b5440 msgr2=0x7f78a80b33d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:44.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.803+0000 7f78afba9640 1 -- 172.21.15.67:0/2743461681 shutdown_connections 2024-03-24T00:40:44.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.803+0000 7f78afba9640 1 -- 172.21.15.67:0/2743461681 wait complete. 2024-03-24T00:40:44.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.804+0000 7f78afba9640 1 Processor -- start 2024-03-24T00:40:44.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.804+0000 7f78afba9640 1 -- start start 2024-03-24T00:40:44.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.805+0000 7f78afba9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78a80b15d0 0x7f78a80cfc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:44.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.805+0000 7f78afba9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78a80b19b0 0x7f78a8152eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:44.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.805+0000 7f78afba9640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f78a815dc50 0x7f78a81533f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:44.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.805+0000 7f78afba9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a80cad90 con 0x7f78a80b15d0 2024-03-24T00:40:44.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.805+0000 7f78afba9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a806b870 con 0x7f78a80b19b0 2024-03-24T00:40:44.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.805+0000 7f78afba9640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78a806b260 con 0x7f78a815dc50 2024-03-24T00:40:44.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.805+0000 7f78ad945640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78a80b15d0 0x7f78a80cfc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:44.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.805+0000 7f78ad945640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78a80b15d0 0x7f78a80cfc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40784/0 (socket says 172.21.15.67:40784) 2024-03-24T00:40:44.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.805+0000 7f78ad945640 1 -- 172.21.15.67:0/1912381054 learned_addr learned my addr 172.21.15.67:0/1912381054 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:44.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.805+0000 7f78ae146640 1 --2- 172.21.15.67:0/1912381054 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f78a815dc50 0x7f78a81533f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:44.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.805+0000 7f78ad144640 1 --2- 172.21.15.67:0/1912381054 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78a80b19b0 0x7f78a8152eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:44.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.805+0000 7f78ad945640 1 -- 172.21.15.67:0/1912381054 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f78a815dc50 msgr2=0x7f78a81533f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:44.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.805+0000 7f78ad945640 1 --2- 172.21.15.67:0/1912381054 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f78a815dc50 0x7f78a81533f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:44.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.805+0000 7f78ad945640 1 -- 172.21.15.67:0/1912381054 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78a80b19b0 msgr2=0x7f78a8152eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:44.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.805+0000 7f78ad945640 1 --2- 172.21.15.67:0/1912381054 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78a80b19b0 0x7f78a8152eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:44.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.805+0000 7f78ad945640 1 -- 172.21.15.67:0/1912381054 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a80adfa0 con 0x7f78a80b15d0 2024-03-24T00:40:44.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.806+0000 7f78ad945640 1 --2- 172.21.15.67:0/1912381054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78a80b15d0 0x7f78a80cfc10 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f78a4009890 tx=0x7f78a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:44.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.806+0000 7f789e7fc640 1 -- 172.21.15.67:0/1912381054 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a4032840 con 0x7f78a80b15d0 2024-03-24T00:40:44.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.806+0000 7f789e7fc640 1 -- 172.21.15.67:0/1912381054 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78a40329e0 con 0x7f78a80b15d0 2024-03-24T00:40:44.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.806+0000 7f78afba9640 1 -- 172.21.15.67:0/1912381054 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78a80ae280 con 0x7f78a80b15d0 2024-03-24T00:40:44.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.806+0000 7f789e7fc640 1 -- 172.21.15.67:0/1912381054 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a4032dd0 con 0x7f78a80b15d0 2024-03-24T00:40:44.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.806+0000 7f78afba9640 1 -- 172.21.15.67:0/1912381054 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78a80ae840 con 0x7f78a80b15d0 2024-03-24T00:40:44.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.807+0000 7f78afba9640 1 -- 172.21.15.67:0/1912381054 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78a804f400 con 0x7f78a80b15d0 2024-03-24T00:40:44.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.807+0000 7f789e7fc640 1 -- 172.21.15.67:0/1912381054 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f78a404c020 con 0x7f78a80b15d0 2024-03-24T00:40:44.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.808+0000 7f789e7fc640 1 --2- 172.21.15.67:0/1912381054 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f788c06bbd0 0x7f788c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:44.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.808+0000 7f789e7fc640 1 -- 172.21.15.67:0/1912381054 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f78a402f020 con 0x7f78a80b15d0 2024-03-24T00:40:44.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.808+0000 7f78ad144640 1 --2- 172.21.15.67:0/1912381054 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f788c06bbd0 0x7f788c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:44.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.808+0000 7f78ad144640 1 --2- 172.21.15.67:0/1912381054 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f788c06bbd0 0x7f788c06e090 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f78a80d0680 tx=0x7f7898008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:44.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.812+0000 7f789e7fc640 1 -- 172.21.15.67:0/1912381054 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f78a4040070 con 0x7f78a80b15d0 2024-03-24T00:40:44.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.989+0000 7f78afba9640 1 -- 172.21.15.67:0/1912381054 --> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f78a81556a0 con 0x7f788c06bbd0 2024-03-24T00:40:44.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.996+0000 7f789e7fc640 1 -- 172.21.15.67:0/1912381054 <== mgr.4105 v2:172.21.15.72:6832/2768292274 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246638 (secure 0 0 0) 0x7f78a81556a0 con 0x7f788c06bbd0 2024-03-24T00:40:45.001 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:40:45.002 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-24T00:40:45.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.999+0000 7f78afba9640 1 -- 172.21.15.67:0/1912381054 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f788c06bbd0 msgr2=0x7f788c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:45.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.999+0000 7f78afba9640 1 --2- 172.21.15.67:0/1912381054 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f788c06bbd0 0x7f788c06e090 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f78a80d0680 tx=0x7f7898008040 comp rx=0 tx=0).stop 2024-03-24T00:40:45.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.999+0000 7f78afba9640 1 -- 172.21.15.67:0/1912381054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78a80b15d0 msgr2=0x7f78a80cfc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:45.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.999+0000 7f78afba9640 1 --2- 172.21.15.67:0/1912381054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78a80b15d0 0x7f78a80cfc10 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f78a4009890 tx=0x7f78a40046c0 comp rx=0 tx=0).stop 2024-03-24T00:40:45.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.999+0000 7f78afba9640 1 -- 172.21.15.67:0/1912381054 shutdown_connections 2024-03-24T00:40:45.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.999+0000 7f78afba9640 1 --2- 172.21.15.67:0/1912381054 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f788c06bbd0 0x7f788c06e090 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:45.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.999+0000 7f78afba9640 1 --2- 172.21.15.67:0/1912381054 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f78a815dc50 0x7f78a81533f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:45.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.999+0000 7f78afba9640 1 --2- 172.21.15.67:0/1912381054 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78a80b19b0 0x7f78a8152eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:45.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.999+0000 7f78afba9640 1 --2- 172.21.15.67:0/1912381054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78a80b15d0 0x7f78a80cfc10 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:45.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.999+0000 7f78afba9640 1 -- 172.21.15.67:0/1912381054 >> 172.21.15.67:0/1912381054 conn(0x7f78a80b5440 msgr2=0x7f78a80c7550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:45.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.999+0000 7f78afba9640 1 -- 172.21.15.67:0/1912381054 shutdown_connections 2024-03-24T00:40:45.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:44.999+0000 7f78afba9640 1 -- 172.21.15.67:0/1912381054 wait complete. 2024-03-24T00:40:45.010 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":105,"stamp":"2024-03-24T00:40:43.345501+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":221684,"kb_used_data":5512,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753036,"statfs":{"total":773094113280,"available":772867108864,"internally_reserved":0,"allocated":5644288,"data_stored":3135232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.066578"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.290314+0000","last_change":"2024-03-24T00:40:39.290314+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.290314+0000","last_undegraded":"2024-03-24T00:40:39.290314+0000","last_fullsized":"2024-03-24T00:40:39.290314+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.290184+0000","last_change":"2024-03-24T00:40:39.290184+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.290184+0000","last_undegraded":"2024-03-24T00:40:39.290184+0000","last_fullsized":"2024-03-24T00:40:39.290184+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.290644+0000","last_change":"2024-03-24T00:40:39.290644+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.290644+0000","last_undegraded":"2024-03-24T00:40:39.290644+0000","last_fullsized":"2024-03-24T00:40:39.290644+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.286759+0000","last_change":"2024-03-24T00:40:39.286759+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.286759+0000","last_undegraded":"2024-03-24T00:40:39.286759+0000","last_fullsized":"2024-03-24T00:40:39.286759+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.288142+0000","last_change":"2024-03-24T00:40:39.288142+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.288142+0000","last_undegraded":"2024-03-24T00:40:39.288142+0000","last_fullsized":"2024-03-24T00:40:39.288142+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.289818+0000","last_change":"2024-03-24T00:40:39.289818+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.289818+0000","last_undegraded":"2024-03-24T00:40:39.289818+0000","last_fullsized":"2024-03-24T00:40:39.289818+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.292120+0000","last_change":"2024-03-24T00:40:39.292120+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.292120+0000","last_undegraded":"2024-03-24T00:40:39.292120+0000","last_fullsized":"2024-03-24T00:40:39.292120+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.285842+0000","last_change":"2024-03-24T00:40:39.285842+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.285842+0000","last_undegraded":"2024-03-24T00:40:39.285842+0000","last_fullsized":"2024-03-24T00:40:39.285842+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.287182+0000","last_change":"2024-03-24T00:40:39.287182+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.287182+0000","last_undegraded":"2024-03-24T00:40:39.287182+0000","last_fullsized":"2024-03-24T00:40:39.287182+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.287090+0000","last_change":"2024-03-24T00:40:39.287090+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.287090+0000","last_undegraded":"2024-03-24T00:40:39.287090+0000","last_fullsized":"2024-03-24T00:40:39.287090+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.293731+0000","last_change":"2024-03-24T00:40:39.293731+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.293731+0000","last_undegraded":"2024-03-24T00:40:39.293731+0000","last_fullsized":"2024-03-24T00:40:39.293731+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.291962+0000","last_change":"2024-03-24T00:40:39.291962+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.291962+0000","last_undegraded":"2024-03-24T00:40:39.291962+0000","last_fullsized":"2024-03-24T00:40:39.291962+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.285910+0000","last_change":"2024-03-24T00:40:39.285910+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.285910+0000","last_undegraded":"2024-03-24T00:40:39.285910+0000","last_fullsized":"2024-03-24T00:40:39.285910+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.291223+0000","last_change":"2024-03-24T00:40:39.291223+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.291223+0000","last_undegraded":"2024-03-24T00:40:39.291223+0000","last_fullsized":"2024-03-24T00:40:39.291223+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.287287+0000","last_change":"2024-03-24T00:40:39.287287+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.287287+0000","last_undegraded":"2024-03-24T00:40:39.287287+0000","last_fullsized":"2024-03-24T00:40:39.287287+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.286183+0000","last_change":"2024-03-24T00:40:39.286183+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.286183+0000","last_undegraded":"2024-03-24T00:40:39.286183+0000","last_fullsized":"2024-03-24T00:40:39.286183+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.290840+0000","last_change":"2024-03-24T00:40:39.290840+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.290840+0000","last_undegraded":"2024-03-24T00:40:39.290840+0000","last_fullsized":"2024-03-24T00:40:39.290840+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.287128+0000","last_change":"2024-03-24T00:40:39.287128+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.287128+0000","last_undegraded":"2024-03-24T00:40:39.287128+0000","last_fullsized":"2024-03-24T00:40:39.287128+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.290088+0000","last_change":"2024-03-24T00:40:39.290088+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.290088+0000","last_undegraded":"2024-03-24T00:40:39.290088+0000","last_fullsized":"2024-03-24T00:40:39.290088+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.287773+0000","last_change":"2024-03-24T00:40:39.287773+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.287773+0000","last_undegraded":"2024-03-24T00:40:39.287773+0000","last_fullsized":"2024-03-24T00:40:39.287773+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.289216+0000","last_change":"2024-03-24T00:40:39.289216+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.289216+0000","last_undegraded":"2024-03-24T00:40:39.289216+0000","last_fullsized":"2024-03-24T00:40:39.289216+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.286528+0000","last_change":"2024-03-24T00:40:39.286528+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.286528+0000","last_undegraded":"2024-03-24T00:40:39.286528+0000","last_fullsized":"2024-03-24T00:40:39.286528+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.293164+0000","last_change":"2024-03-24T00:40:39.293164+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.293164+0000","last_undegraded":"2024-03-24T00:40:39.293164+0000","last_fullsized":"2024-03-24T00:40:39.293164+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.289958+0000","last_change":"2024-03-24T00:40:39.289958+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.289958+0000","last_undegraded":"2024-03-24T00:40:39.289958+0000","last_fullsized":"2024-03-24T00:40:39.289958+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.290445+0000","last_change":"2024-03-24T00:40:39.290445+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.290445+0000","last_undegraded":"2024-03-24T00:40:39.290445+0000","last_fullsized":"2024-03-24T00:40:39.290445+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.287297+0000","last_change":"2024-03-24T00:40:39.287297+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.287297+0000","last_undegraded":"2024-03-24T00:40:39.287297+0000","last_fullsized":"2024-03-24T00:40:39.287297+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.286083+0000","last_change":"2024-03-24T00:40:39.286083+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.286083+0000","last_undegraded":"2024-03-24T00:40:39.286083+0000","last_fullsized":"2024-03-24T00:40:39.286083+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.287284+0000","last_change":"2024-03-24T00:40:39.287284+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.287284+0000","last_undegraded":"2024-03-24T00:40:39.287284+0000","last_fullsized":"2024-03-24T00:40:39.287284+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.291689+0000","last_change":"2024-03-24T00:40:39.291689+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.291689+0000","last_undegraded":"2024-03-24T00:40:39.291689+0000","last_fullsized":"2024-03-24T00:40:39.291689+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.286764+0000","last_change":"2024-03-24T00:40:39.286764+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.286764+0000","last_undegraded":"2024-03-24T00:40:39.286764+0000","last_fullsized":"2024-03-24T00:40:39.286764+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.289986+0000","last_change":"2024-03-24T00:40:39.289986+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.289986+0000","last_undegraded":"2024-03-24T00:40:39.289986+0000","last_fullsized":"2024-03-24T00:40:39.289986+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.284737+0000","last_change":"2024-03-24T00:40:39.284737+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.284737+0000","last_undegraded":"2024-03-24T00:40:39.284737+0000","last_fullsized":"2024-03-24T00:40:39.284737+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.290205+0000","last_change":"2024-03-24T00:40:39.290205+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.290205+0000","last_undegraded":"2024-03-24T00:40:39.290205+0000","last_fullsized":"2024-03-24T00:40:39.290205+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.293699+0000","last_change":"2024-03-24T00:40:39.291631+0000","last_active":"2024-03-24T00:40:39.293699+0000","last_peered":"2024-03-24T00:40:39.293699+0000","last_clean":"2024-03-24T00:40:39.293699+0000","last_became_active":"2024-03-24T00:40:39.291349+0000","last_became_peered":"2024-03-24T00:40:39.291349+0000","last_unstale":"2024-03-24T00:40:39.293699+0000","last_undegraded":"2024-03-24T00:40:39.293699+0000","last_fullsized":"2024-03-24T00:40:39.293699+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:32:08.714068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.283698+0000","last_change":"2024-03-24T00:40:39.283698+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.283698+0000","last_undegraded":"2024-03-24T00:40:39.283698+0000","last_fullsized":"2024-03-24T00:40:39.283698+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.297930+0000","last_change":"2024-03-24T00:40:39.295765+0000","last_active":"2024-03-24T00:40:39.297930+0000","last_peered":"2024-03-24T00:40:39.297930+0000","last_clean":"2024-03-24T00:40:39.297930+0000","last_became_active":"2024-03-24T00:40:39.295486+0000","last_became_peered":"2024-03-24T00:40:39.295486+0000","last_unstale":"2024-03-24T00:40:39.297930+0000","last_undegraded":"2024-03-24T00:40:39.297930+0000","last_fullsized":"2024-03-24T00:40:39.297930+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:09:53.820800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.289688+0000","last_change":"2024-03-24T00:40:39.289688+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.289688+0000","last_undegraded":"2024-03-24T00:40:39.289688+0000","last_fullsized":"2024-03-24T00:40:39.289688+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.295662+0000","last_change":"2024-03-24T00:40:39.294921+0000","last_active":"2024-03-24T00:40:39.295662+0000","last_peered":"2024-03-24T00:40:39.295662+0000","last_clean":"2024-03-24T00:40:39.295662+0000","last_became_active":"2024-03-24T00:40:39.294579+0000","last_became_peered":"2024-03-24T00:40:39.294579+0000","last_unstale":"2024-03-24T00:40:39.295662+0000","last_undegraded":"2024-03-24T00:40:39.295662+0000","last_fullsized":"2024-03-24T00:40:39.295662+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:54:49.331324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.295225+0000","last_change":"2024-03-24T00:40:39.295225+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.295225+0000","last_undegraded":"2024-03-24T00:40:39.295225+0000","last_fullsized":"2024-03-24T00:40:39.295225+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.299268+0000","last_change":"2024-03-24T00:40:39.297839+0000","last_active":"2024-03-24T00:40:39.299268+0000","last_peered":"2024-03-24T00:40:39.299268+0000","last_clean":"2024-03-24T00:40:39.299268+0000","last_became_active":"2024-03-24T00:40:39.297536+0000","last_became_peered":"2024-03-24T00:40:39.297536+0000","last_unstale":"2024-03-24T00:40:39.299268+0000","last_undegraded":"2024-03-24T00:40:39.299268+0000","last_fullsized":"2024-03-24T00:40:39.299268+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:58:21.161844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.286437+0000","last_change":"2024-03-24T00:40:39.286437+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.286437+0000","last_undegraded":"2024-03-24T00:40:39.286437+0000","last_fullsized":"2024-03-24T00:40:39.286437+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.305288+0000","last_change":"2024-03-24T00:40:39.304061+0000","last_active":"2024-03-24T00:40:39.305288+0000","last_peered":"2024-03-24T00:40:39.305288+0000","last_clean":"2024-03-24T00:40:39.305288+0000","last_became_active":"2024-03-24T00:40:39.303769+0000","last_became_peered":"2024-03-24T00:40:39.303769+0000","last_unstale":"2024-03-24T00:40:39.305288+0000","last_undegraded":"2024-03-24T00:40:39.305288+0000","last_fullsized":"2024-03-24T00:40:39.305288+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:00:51.940556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.286866+0000","last_change":"2024-03-24T00:40:39.286866+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.286866+0000","last_undegraded":"2024-03-24T00:40:39.286866+0000","last_fullsized":"2024-03-24T00:40:39.286866+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.295132+0000","last_change":"2024-03-24T00:40:39.293463+0000","last_active":"2024-03-24T00:40:39.295132+0000","last_peered":"2024-03-24T00:40:39.295132+0000","last_clean":"2024-03-24T00:40:39.295132+0000","last_became_active":"2024-03-24T00:40:39.293196+0000","last_became_peered":"2024-03-24T00:40:39.293196+0000","last_unstale":"2024-03-24T00:40:39.295132+0000","last_undegraded":"2024-03-24T00:40:39.295132+0000","last_fullsized":"2024-03-24T00:40:39.295132+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:16:22.016263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.285844+0000","last_change":"2024-03-24T00:40:39.285844+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.285844+0000","last_undegraded":"2024-03-24T00:40:39.285844+0000","last_fullsized":"2024-03-24T00:40:39.285844+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.296000+0000","last_change":"2024-03-24T00:40:39.294468+0000","last_active":"2024-03-24T00:40:39.296000+0000","last_peered":"2024-03-24T00:40:39.296000+0000","last_clean":"2024-03-24T00:40:39.296000+0000","last_became_active":"2024-03-24T00:40:39.294130+0000","last_became_peered":"2024-03-24T00:40:39.294130+0000","last_unstale":"2024-03-24T00:40:39.296000+0000","last_undegraded":"2024-03-24T00:40:39.296000+0000","last_fullsized":"2024-03-24T00:40:39.296000+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:52:39.705988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.290406+0000","last_change":"2024-03-24T00:40:39.290406+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.290406+0000","last_undegraded":"2024-03-24T00:40:39.290406+0000","last_fullsized":"2024-03-24T00:40:39.290406+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.298578+0000","last_change":"2024-03-24T00:40:39.296539+0000","last_active":"2024-03-24T00:40:39.298578+0000","last_peered":"2024-03-24T00:40:39.298578+0000","last_clean":"2024-03-24T00:40:39.298578+0000","last_became_active":"2024-03-24T00:40:39.296272+0000","last_became_peered":"2024-03-24T00:40:39.296272+0000","last_unstale":"2024-03-24T00:40:39.298578+0000","last_undegraded":"2024-03-24T00:40:39.298578+0000","last_fullsized":"2024-03-24T00:40:39.298578+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:09:53.820800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.286586+0000","last_change":"2024-03-24T00:40:39.286586+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.286586+0000","last_undegraded":"2024-03-24T00:40:39.286586+0000","last_fullsized":"2024-03-24T00:40:39.286586+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.297968+0000","last_change":"2024-03-24T00:40:39.296067+0000","last_active":"2024-03-24T00:40:39.297968+0000","last_peered":"2024-03-24T00:40:39.297968+0000","last_clean":"2024-03-24T00:40:39.297968+0000","last_became_active":"2024-03-24T00:40:39.295773+0000","last_became_peered":"2024-03-24T00:40:39.295773+0000","last_unstale":"2024-03-24T00:40:39.297968+0000","last_undegraded":"2024-03-24T00:40:39.297968+0000","last_fullsized":"2024-03-24T00:40:39.297968+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:53:34.136108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.284282+0000","last_change":"2024-03-24T00:40:39.284282+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.284282+0000","last_undegraded":"2024-03-24T00:40:39.284282+0000","last_fullsized":"2024-03-24T00:40:39.284282+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.298719+0000","last_change":"2024-03-24T00:40:39.296711+0000","last_active":"2024-03-24T00:40:39.298719+0000","last_peered":"2024-03-24T00:40:39.298719+0000","last_clean":"2024-03-24T00:40:39.298719+0000","last_became_active":"2024-03-24T00:40:39.296435+0000","last_became_peered":"2024-03-24T00:40:39.296435+0000","last_unstale":"2024-03-24T00:40:39.298719+0000","last_undegraded":"2024-03-24T00:40:39.298719+0000","last_fullsized":"2024-03-24T00:40:39.298719+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:54:49.331324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.285258+0000","last_change":"2024-03-24T00:40:39.285258+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.285258+0000","last_undegraded":"2024-03-24T00:40:39.285258+0000","last_fullsized":"2024-03-24T00:40:39.285258+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.303989+0000","last_change":"2024-03-24T00:40:39.301123+0000","last_active":"2024-03-24T00:40:39.303989+0000","last_peered":"2024-03-24T00:40:39.303989+0000","last_clean":"2024-03-24T00:40:39.303989+0000","last_became_active":"2024-03-24T00:40:39.300851+0000","last_became_peered":"2024-03-24T00:40:39.300851+0000","last_unstale":"2024-03-24T00:40:39.303989+0000","last_undegraded":"2024-03-24T00:40:39.303989+0000","last_fullsized":"2024-03-24T00:40:39.303989+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:00:51.940556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.287678+0000","last_change":"2024-03-24T00:40:39.287678+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.287678+0000","last_undegraded":"2024-03-24T00:40:39.287678+0000","last_fullsized":"2024-03-24T00:40:39.287678+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.304078+0000","last_change":"2024-03-24T00:40:39.302117+0000","last_active":"2024-03-24T00:40:39.304078+0000","last_peered":"2024-03-24T00:40:39.304078+0000","last_clean":"2024-03-24T00:40:39.304078+0000","last_became_active":"2024-03-24T00:40:39.301875+0000","last_became_peered":"2024-03-24T00:40:39.301875+0000","last_unstale":"2024-03-24T00:40:39.304078+0000","last_undegraded":"2024-03-24T00:40:39.304078+0000","last_fullsized":"2024-03-24T00:40:39.304078+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:48:02.396386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.284454+0000","last_change":"2024-03-24T00:40:39.284454+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.284454+0000","last_undegraded":"2024-03-24T00:40:39.284454+0000","last_fullsized":"2024-03-24T00:40:39.284454+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.303378+0000","last_change":"2024-03-24T00:40:39.298667+0000","last_active":"2024-03-24T00:40:39.303378+0000","last_peered":"2024-03-24T00:40:39.303378+0000","last_clean":"2024-03-24T00:40:39.303378+0000","last_became_active":"2024-03-24T00:40:39.298067+0000","last_became_peered":"2024-03-24T00:40:39.298067+0000","last_unstale":"2024-03-24T00:40:39.303378+0000","last_undegraded":"2024-03-24T00:40:39.303378+0000","last_fullsized":"2024-03-24T00:40:39.303378+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:35.720596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.286589+0000","last_change":"2024-03-24T00:40:39.286589+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.286589+0000","last_undegraded":"2024-03-24T00:40:39.286589+0000","last_fullsized":"2024-03-24T00:40:39.286589+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.301055+0000","last_change":"2024-03-24T00:40:39.299651+0000","last_active":"2024-03-24T00:40:39.301055+0000","last_peered":"2024-03-24T00:40:39.301055+0000","last_clean":"2024-03-24T00:40:39.301055+0000","last_became_active":"2024-03-24T00:40:39.299388+0000","last_became_peered":"2024-03-24T00:40:39.299388+0000","last_unstale":"2024-03-24T00:40:39.301055+0000","last_undegraded":"2024-03-24T00:40:39.301055+0000","last_fullsized":"2024-03-24T00:40:39.301055+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:30:39.662599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.286895+0000","last_change":"2024-03-24T00:40:39.286895+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.286895+0000","last_undegraded":"2024-03-24T00:40:39.286895+0000","last_fullsized":"2024-03-24T00:40:39.286895+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.298000+0000","last_change":"2024-03-24T00:40:39.295906+0000","last_active":"2024-03-24T00:40:39.298000+0000","last_peered":"2024-03-24T00:40:39.298000+0000","last_clean":"2024-03-24T00:40:39.298000+0000","last_became_active":"2024-03-24T00:40:39.295604+0000","last_became_peered":"2024-03-24T00:40:39.295604+0000","last_unstale":"2024-03-24T00:40:39.298000+0000","last_undegraded":"2024-03-24T00:40:39.298000+0000","last_fullsized":"2024-03-24T00:40:39.298000+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:35.720596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.288224+0000","last_change":"2024-03-24T00:40:39.288224+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.288224+0000","last_undegraded":"2024-03-24T00:40:39.288224+0000","last_fullsized":"2024-03-24T00:40:39.288224+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.302950+0000","last_change":"2024-03-24T00:40:39.298619+0000","last_active":"2024-03-24T00:40:39.302950+0000","last_peered":"2024-03-24T00:40:39.302950+0000","last_clean":"2024-03-24T00:40:39.302950+0000","last_became_active":"2024-03-24T00:40:39.298350+0000","last_became_peered":"2024-03-24T00:40:39.298350+0000","last_unstale":"2024-03-24T00:40:39.302950+0000","last_undegraded":"2024-03-24T00:40:39.302950+0000","last_fullsized":"2024-03-24T00:40:39.302950+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:04:49.487506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.285444+0000","last_change":"2024-03-24T00:40:39.285444+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.285444+0000","last_undegraded":"2024-03-24T00:40:39.285444+0000","last_fullsized":"2024-03-24T00:40:39.285444+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.303365+0000","last_change":"2024-03-24T00:40:39.299640+0000","last_active":"2024-03-24T00:40:39.303365+0000","last_peered":"2024-03-24T00:40:39.303365+0000","last_clean":"2024-03-24T00:40:39.303365+0000","last_became_active":"2024-03-24T00:40:39.299375+0000","last_became_peered":"2024-03-24T00:40:39.299375+0000","last_unstale":"2024-03-24T00:40:39.303365+0000","last_undegraded":"2024-03-24T00:40:39.303365+0000","last_fullsized":"2024-03-24T00:40:39.303365+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:53:34.136108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.284942+0000","last_change":"2024-03-24T00:40:39.284942+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.284942+0000","last_undegraded":"2024-03-24T00:40:39.284942+0000","last_fullsized":"2024-03-24T00:40:39.284942+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.303123+0000","last_change":"2024-03-24T00:40:39.302072+0000","last_active":"2024-03-24T00:40:39.303123+0000","last_peered":"2024-03-24T00:40:39.303123+0000","last_clean":"2024-03-24T00:40:39.303123+0000","last_became_active":"2024-03-24T00:40:39.301790+0000","last_became_peered":"2024-03-24T00:40:39.301790+0000","last_unstale":"2024-03-24T00:40:39.303123+0000","last_undegraded":"2024-03-24T00:40:39.303123+0000","last_fullsized":"2024-03-24T00:40:39.303123+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:48:22.534145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.287303+0000","last_change":"2024-03-24T00:40:39.287303+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.287303+0000","last_undegraded":"2024-03-24T00:40:39.287303+0000","last_fullsized":"2024-03-24T00:40:39.287303+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.293730+0000","last_change":"2024-03-24T00:40:39.292046+0000","last_active":"2024-03-24T00:40:39.293730+0000","last_peered":"2024-03-24T00:40:39.293730+0000","last_clean":"2024-03-24T00:40:39.293730+0000","last_became_active":"2024-03-24T00:40:39.291744+0000","last_became_peered":"2024-03-24T00:40:39.291744+0000","last_unstale":"2024-03-24T00:40:39.293730+0000","last_undegraded":"2024-03-24T00:40:39.293730+0000","last_fullsized":"2024-03-24T00:40:39.293730+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:05:15.049538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.284674+0000","last_change":"2024-03-24T00:40:39.284674+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.284674+0000","last_undegraded":"2024-03-24T00:40:39.284674+0000","last_fullsized":"2024-03-24T00:40:39.284674+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.309077+0000","last_change":"2024-03-24T00:40:39.304591+0000","last_active":"2024-03-24T00:40:39.309077+0000","last_peered":"2024-03-24T00:40:39.309077+0000","last_clean":"2024-03-24T00:40:39.309077+0000","last_became_active":"2024-03-24T00:40:39.304330+0000","last_became_peered":"2024-03-24T00:40:39.304330+0000","last_unstale":"2024-03-24T00:40:39.309077+0000","last_undegraded":"2024-03-24T00:40:39.309077+0000","last_fullsized":"2024-03-24T00:40:39.309077+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:25:48.533462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.283875+0000","last_change":"2024-03-24T00:40:39.283875+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.283875+0000","last_undegraded":"2024-03-24T00:40:39.283875+0000","last_fullsized":"2024-03-24T00:40:39.283875+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.289779+0000","last_change":"2024-03-24T00:40:39.287554+0000","last_active":"2024-03-24T00:40:39.289779+0000","last_peered":"2024-03-24T00:40:39.289779+0000","last_clean":"2024-03-24T00:40:39.289779+0000","last_became_active":"2024-03-24T00:40:39.287273+0000","last_became_peered":"2024-03-24T00:40:39.287273+0000","last_unstale":"2024-03-24T00:40:39.289779+0000","last_undegraded":"2024-03-24T00:40:39.289779+0000","last_fullsized":"2024-03-24T00:40:39.289779+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:16:22.016263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.289204+0000","last_change":"2024-03-24T00:40:39.289204+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.289204+0000","last_undegraded":"2024-03-24T00:40:39.289204+0000","last_fullsized":"2024-03-24T00:40:39.289204+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.302927+0000","last_change":"2024-03-24T00:40:39.298506+0000","last_active":"2024-03-24T00:40:39.302927+0000","last_peered":"2024-03-24T00:40:39.302927+0000","last_clean":"2024-03-24T00:40:39.302927+0000","last_became_active":"2024-03-24T00:40:39.297935+0000","last_became_peered":"2024-03-24T00:40:39.297935+0000","last_unstale":"2024-03-24T00:40:39.302927+0000","last_undegraded":"2024-03-24T00:40:39.302927+0000","last_fullsized":"2024-03-24T00:40:39.302927+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:23:25.616752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.287275+0000","last_change":"2024-03-24T00:40:39.287275+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.287275+0000","last_undegraded":"2024-03-24T00:40:39.287275+0000","last_fullsized":"2024-03-24T00:40:39.287275+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.298445+0000","last_change":"2024-03-24T00:40:39.296787+0000","last_active":"2024-03-24T00:40:39.298445+0000","last_peered":"2024-03-24T00:40:39.298445+0000","last_clean":"2024-03-24T00:40:39.298445+0000","last_became_active":"2024-03-24T00:40:39.296501+0000","last_became_peered":"2024-03-24T00:40:39.296501+0000","last_unstale":"2024-03-24T00:40:39.298445+0000","last_undegraded":"2024-03-24T00:40:39.298445+0000","last_fullsized":"2024-03-24T00:40:39.298445+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:43:07.435024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.285594+0000","last_change":"2024-03-24T00:40:39.285594+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.285594+0000","last_undegraded":"2024-03-24T00:40:39.285594+0000","last_fullsized":"2024-03-24T00:40:39.285594+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.302585+0000","last_change":"2024-03-24T00:40:39.300361+0000","last_active":"2024-03-24T00:40:39.302585+0000","last_peered":"2024-03-24T00:40:39.302585+0000","last_clean":"2024-03-24T00:40:39.302585+0000","last_became_active":"2024-03-24T00:40:39.299954+0000","last_became_peered":"2024-03-24T00:40:39.299954+0000","last_unstale":"2024-03-24T00:40:39.302585+0000","last_undegraded":"2024-03-24T00:40:39.302585+0000","last_fullsized":"2024-03-24T00:40:39.302585+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:35:30.101713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.283792+0000","last_change":"2024-03-24T00:40:39.283792+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.283792+0000","last_undegraded":"2024-03-24T00:40:39.283792+0000","last_fullsized":"2024-03-24T00:40:39.283792+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.300944+0000","last_change":"2024-03-24T00:40:39.299797+0000","last_active":"2024-03-24T00:40:39.300944+0000","last_peered":"2024-03-24T00:40:39.300944+0000","last_clean":"2024-03-24T00:40:39.300944+0000","last_became_active":"2024-03-24T00:40:39.299456+0000","last_became_peered":"2024-03-24T00:40:39.299456+0000","last_unstale":"2024-03-24T00:40:39.300944+0000","last_undegraded":"2024-03-24T00:40:39.300944+0000","last_fullsized":"2024-03-24T00:40:39.300944+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:28:27.547221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.285596+0000","last_change":"2024-03-24T00:40:39.285596+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.285596+0000","last_undegraded":"2024-03-24T00:40:39.285596+0000","last_fullsized":"2024-03-24T00:40:39.285596+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.299923+0000","last_change":"2024-03-24T00:40:39.299014+0000","last_active":"2024-03-24T00:40:39.299923+0000","last_peered":"2024-03-24T00:40:39.299923+0000","last_clean":"2024-03-24T00:40:39.299923+0000","last_became_active":"2024-03-24T00:40:39.298750+0000","last_became_peered":"2024-03-24T00:40:39.298750+0000","last_unstale":"2024-03-24T00:40:39.299923+0000","last_undegraded":"2024-03-24T00:40:39.299923+0000","last_fullsized":"2024-03-24T00:40:39.299923+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:08:19.135822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.289013+0000","last_change":"2024-03-24T00:40:39.289013+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.289013+0000","last_undegraded":"2024-03-24T00:40:39.289013+0000","last_fullsized":"2024-03-24T00:40:39.289013+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.304762+0000","last_change":"2024-03-24T00:40:39.303224+0000","last_active":"2024-03-24T00:40:39.304762+0000","last_peered":"2024-03-24T00:40:39.304762+0000","last_clean":"2024-03-24T00:40:39.304762+0000","last_became_active":"2024-03-24T00:40:39.302950+0000","last_became_peered":"2024-03-24T00:40:39.302950+0000","last_unstale":"2024-03-24T00:40:39.304762+0000","last_undegraded":"2024-03-24T00:40:39.304762+0000","last_fullsized":"2024-03-24T00:40:39.304762+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:08:19.135822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.285991+0000","last_change":"2024-03-24T00:40:39.285991+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.285991+0000","last_undegraded":"2024-03-24T00:40:39.285991+0000","last_fullsized":"2024-03-24T00:40:39.285991+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.304018+0000","last_change":"2024-03-24T00:40:39.299484+0000","last_active":"2024-03-24T00:40:39.304018+0000","last_peered":"2024-03-24T00:40:39.304018+0000","last_clean":"2024-03-24T00:40:39.304018+0000","last_became_active":"2024-03-24T00:40:39.299184+0000","last_became_peered":"2024-03-24T00:40:39.299184+0000","last_unstale":"2024-03-24T00:40:39.304018+0000","last_undegraded":"2024-03-24T00:40:39.304018+0000","last_fullsized":"2024-03-24T00:40:39.304018+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:57:35.870209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.283715+0000","last_change":"2024-03-24T00:40:39.283715+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.283715+0000","last_undegraded":"2024-03-24T00:40:39.283715+0000","last_fullsized":"2024-03-24T00:40:39.283715+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.302907+0000","last_change":"2024-03-24T00:40:39.298278+0000","last_active":"2024-03-24T00:40:39.302907+0000","last_peered":"2024-03-24T00:40:39.302907+0000","last_clean":"2024-03-24T00:40:39.302907+0000","last_became_active":"2024-03-24T00:40:39.297588+0000","last_became_peered":"2024-03-24T00:40:39.297588+0000","last_unstale":"2024-03-24T00:40:39.302907+0000","last_undegraded":"2024-03-24T00:40:39.302907+0000","last_fullsized":"2024-03-24T00:40:39.302907+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:06:05.426065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.287710+0000","last_change":"2024-03-24T00:40:39.287710+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.287710+0000","last_undegraded":"2024-03-24T00:40:39.287710+0000","last_fullsized":"2024-03-24T00:40:39.287710+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.301019+0000","last_change":"2024-03-24T00:40:39.300219+0000","last_active":"2024-03-24T00:40:39.301019+0000","last_peered":"2024-03-24T00:40:39.301019+0000","last_clean":"2024-03-24T00:40:39.301019+0000","last_became_active":"2024-03-24T00:40:39.299897+0000","last_became_peered":"2024-03-24T00:40:39.299897+0000","last_unstale":"2024-03-24T00:40:39.301019+0000","last_undegraded":"2024-03-24T00:40:39.301019+0000","last_fullsized":"2024-03-24T00:40:39.301019+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:09:53.820800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.286808+0000","last_change":"2024-03-24T00:40:39.286808+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.286808+0000","last_undegraded":"2024-03-24T00:40:39.286808+0000","last_fullsized":"2024-03-24T00:40:39.286808+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.295357+0000","last_change":"2024-03-24T00:40:39.294249+0000","last_active":"2024-03-24T00:40:39.295357+0000","last_peered":"2024-03-24T00:40:39.295357+0000","last_clean":"2024-03-24T00:40:39.295357+0000","last_became_active":"2024-03-24T00:40:39.293915+0000","last_became_peered":"2024-03-24T00:40:39.293915+0000","last_unstale":"2024-03-24T00:40:39.295357+0000","last_undegraded":"2024-03-24T00:40:39.295357+0000","last_fullsized":"2024-03-24T00:40:39.295357+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:05:32.540043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T00:40:39.289541+0000","last_change":"2024-03-24T00:40:39.289541+0000","last_active":"2024-03-24T00:40:39.278047+0000","last_peered":"2024-03-24T00:40:39.278047+0000","last_clean":"2024-03-24T00:40:39.278047+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:40:39.289541+0000","last_undegraded":"2024-03-24T00:40:39.289541+0000","last_fullsized":"2024-03-24T00:40:39.289541+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.302298+0000","last_change":"2024-03-24T00:40:39.297685+0000","last_active":"2024-03-24T00:40:39.302298+0000","last_peered":"2024-03-24T00:40:39.302298+0000","last_clean":"2024-03-24T00:40:39.302298+0000","last_became_active":"2024-03-24T00:40:39.297097+0000","last_became_peered":"2024-03-24T00:40:39.297097+0000","last_unstale":"2024-03-24T00:40:39.302298+0000","last_undegraded":"2024-03-24T00:40:39.302298+0000","last_fullsized":"2024-03-24T00:40:39.302298+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:42:28.507176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":220,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T00:40:39.291524+0000","last_change":"2024-03-24T00:38:40.953871+0000","last_active":"2024-03-24T00:40:39.291524+0000","last_peered":"2024-03-24T00:40:39.291524+0000","last_clean":"2024-03-24T00:40:39.291524+0000","last_became_active":"2024-03-24T00:38:40.952926+0000","last_became_peered":"2024-03-24T00:38:40.952926+0000","last_unstale":"2024-03-24T00:40:39.291524+0000","last_undegraded":"2024-03-24T00:40:39.291524+0000","last_fullsized":"2024-03-24T00:40:39.291524+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_clean_scrub_stamp":"2024-03-24T00:38:39.722194+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:05.397257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574877,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28144,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343696,"statfs":{"total":96636764160,"available":96607944704,"internally_reserved":0,"allocated":1150976,"data_stored":834680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27568,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344272,"statfs":{"total":96636764160,"available":96608534528,"internally_reserved":0,"allocated":557056,"data_stored":244312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":557056,"data_stored":244312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":557056,"data_stored":244312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574875,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27568,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344272,"statfs":{"total":96636764160,"available":96608534528,"internally_reserved":0,"allocated":557056,"data_stored":244312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27568,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344272,"statfs":{"total":96636764160,"available":96608534528,"internally_reserved":0,"allocated":557056,"data_stored":244312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":557056,"data_stored":244312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574875,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28144,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343696,"statfs":{"total":96636764160,"available":96607944704,"internally_reserved":0,"allocated":1150976,"data_stored":834680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:40:45.014 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3f is not active+clean 2024-03-24T00:40:45.014 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.290314+0000', 'last_change': '2024-03-24T00:40:39.290314+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.290314+0000', 'last_undegraded': '2024-03-24T00:40:39.290314+0000', 'last_fullsized': '2024-03-24T00:40:39.290314+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-24T00:40:45.014 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3e is not active+clean 2024-03-24T00:40:45.014 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.290184+0000', 'last_change': '2024-03-24T00:40:39.290184+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.290184+0000', 'last_undegraded': '2024-03-24T00:40:39.290184+0000', 'last_fullsized': '2024-03-24T00:40:39.290184+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T00:40:45.014 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3d is not active+clean 2024-03-24T00:40:45.014 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.290644+0000', 'last_change': '2024-03-24T00:40:39.290644+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.290644+0000', 'last_undegraded': '2024-03-24T00:40:39.290644+0000', 'last_fullsized': '2024-03-24T00:40:39.290644+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T00:40:45.014 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3c is not active+clean 2024-03-24T00:40:45.014 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.286759+0000', 'last_change': '2024-03-24T00:40:39.286759+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.286759+0000', 'last_undegraded': '2024-03-24T00:40:39.286759+0000', 'last_fullsized': '2024-03-24T00:40:39.286759+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T00:40:45.015 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3b is not active+clean 2024-03-24T00:40:45.015 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.288142+0000', 'last_change': '2024-03-24T00:40:39.288142+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.288142+0000', 'last_undegraded': '2024-03-24T00:40:39.288142+0000', 'last_fullsized': '2024-03-24T00:40:39.288142+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-24T00:40:45.015 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3a is not active+clean 2024-03-24T00:40:45.015 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.289818+0000', 'last_change': '2024-03-24T00:40:39.289818+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.289818+0000', 'last_undegraded': '2024-03-24T00:40:39.289818+0000', 'last_fullsized': '2024-03-24T00:40:39.289818+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-24T00:40:45.015 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.39 is not active+clean 2024-03-24T00:40:45.015 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.292120+0000', 'last_change': '2024-03-24T00:40:39.292120+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.292120+0000', 'last_undegraded': '2024-03-24T00:40:39.292120+0000', 'last_fullsized': '2024-03-24T00:40:39.292120+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T00:40:45.015 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.38 is not active+clean 2024-03-24T00:40:45.015 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.285842+0000', 'last_change': '2024-03-24T00:40:39.285842+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.285842+0000', 'last_undegraded': '2024-03-24T00:40:39.285842+0000', 'last_fullsized': '2024-03-24T00:40:39.285842+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T00:40:45.015 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.37 is not active+clean 2024-03-24T00:40:45.015 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.287182+0000', 'last_change': '2024-03-24T00:40:39.287182+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.287182+0000', 'last_undegraded': '2024-03-24T00:40:39.287182+0000', 'last_fullsized': '2024-03-24T00:40:39.287182+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T00:40:45.015 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.36 is not active+clean 2024-03-24T00:40:45.015 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.287090+0000', 'last_change': '2024-03-24T00:40:39.287090+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.287090+0000', 'last_undegraded': '2024-03-24T00:40:39.287090+0000', 'last_fullsized': '2024-03-24T00:40:39.287090+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-24T00:40:45.016 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.35 is not active+clean 2024-03-24T00:40:45.016 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.293731+0000', 'last_change': '2024-03-24T00:40:39.293731+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.293731+0000', 'last_undegraded': '2024-03-24T00:40:39.293731+0000', 'last_fullsized': '2024-03-24T00:40:39.293731+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-24T00:40:45.016 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.34 is not active+clean 2024-03-24T00:40:45.016 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.291962+0000', 'last_change': '2024-03-24T00:40:39.291962+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.291962+0000', 'last_undegraded': '2024-03-24T00:40:39.291962+0000', 'last_fullsized': '2024-03-24T00:40:39.291962+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T00:40:45.016 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.33 is not active+clean 2024-03-24T00:40:45.016 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.285910+0000', 'last_change': '2024-03-24T00:40:39.285910+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.285910+0000', 'last_undegraded': '2024-03-24T00:40:39.285910+0000', 'last_fullsized': '2024-03-24T00:40:39.285910+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-24T00:40:45.016 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.32 is not active+clean 2024-03-24T00:40:45.016 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.291223+0000', 'last_change': '2024-03-24T00:40:39.291223+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.291223+0000', 'last_undegraded': '2024-03-24T00:40:39.291223+0000', 'last_fullsized': '2024-03-24T00:40:39.291223+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T00:40:45.016 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.31 is not active+clean 2024-03-24T00:40:45.016 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.287287+0000', 'last_change': '2024-03-24T00:40:39.287287+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.287287+0000', 'last_undegraded': '2024-03-24T00:40:39.287287+0000', 'last_fullsized': '2024-03-24T00:40:39.287287+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T00:40:45.016 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.30 is not active+clean 2024-03-24T00:40:45.016 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.286183+0000', 'last_change': '2024-03-24T00:40:39.286183+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.286183+0000', 'last_undegraded': '2024-03-24T00:40:39.286183+0000', 'last_fullsized': '2024-03-24T00:40:39.286183+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-24T00:40:45.017 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2f is not active+clean 2024-03-24T00:40:45.017 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.290840+0000', 'last_change': '2024-03-24T00:40:39.290840+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.290840+0000', 'last_undegraded': '2024-03-24T00:40:39.290840+0000', 'last_fullsized': '2024-03-24T00:40:39.290840+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T00:40:45.017 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2e is not active+clean 2024-03-24T00:40:45.017 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.287128+0000', 'last_change': '2024-03-24T00:40:39.287128+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.287128+0000', 'last_undegraded': '2024-03-24T00:40:39.287128+0000', 'last_fullsized': '2024-03-24T00:40:39.287128+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T00:40:45.017 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2d is not active+clean 2024-03-24T00:40:45.017 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.290088+0000', 'last_change': '2024-03-24T00:40:39.290088+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.290088+0000', 'last_undegraded': '2024-03-24T00:40:39.290088+0000', 'last_fullsized': '2024-03-24T00:40:39.290088+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T00:40:45.017 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2c is not active+clean 2024-03-24T00:40:45.017 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.287773+0000', 'last_change': '2024-03-24T00:40:39.287773+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.287773+0000', 'last_undegraded': '2024-03-24T00:40:39.287773+0000', 'last_fullsized': '2024-03-24T00:40:39.287773+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-24T00:40:45.017 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2b is not active+clean 2024-03-24T00:40:45.017 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.289216+0000', 'last_change': '2024-03-24T00:40:39.289216+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.289216+0000', 'last_undegraded': '2024-03-24T00:40:39.289216+0000', 'last_fullsized': '2024-03-24T00:40:39.289216+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-24T00:40:45.017 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2a is not active+clean 2024-03-24T00:40:45.017 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.286528+0000', 'last_change': '2024-03-24T00:40:39.286528+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.286528+0000', 'last_undegraded': '2024-03-24T00:40:39.286528+0000', 'last_fullsized': '2024-03-24T00:40:39.286528+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T00:40:45.018 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.29 is not active+clean 2024-03-24T00:40:45.018 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.293164+0000', 'last_change': '2024-03-24T00:40:39.293164+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.293164+0000', 'last_undegraded': '2024-03-24T00:40:39.293164+0000', 'last_fullsized': '2024-03-24T00:40:39.293164+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-24T00:40:45.018 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.28 is not active+clean 2024-03-24T00:40:45.018 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.289958+0000', 'last_change': '2024-03-24T00:40:39.289958+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.289958+0000', 'last_undegraded': '2024-03-24T00:40:39.289958+0000', 'last_fullsized': '2024-03-24T00:40:39.289958+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T00:40:45.018 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.27 is not active+clean 2024-03-24T00:40:45.018 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.290445+0000', 'last_change': '2024-03-24T00:40:39.290445+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.290445+0000', 'last_undegraded': '2024-03-24T00:40:39.290445+0000', 'last_fullsized': '2024-03-24T00:40:39.290445+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T00:40:45.018 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.26 is not active+clean 2024-03-24T00:40:45.018 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.287297+0000', 'last_change': '2024-03-24T00:40:39.287297+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.287297+0000', 'last_undegraded': '2024-03-24T00:40:39.287297+0000', 'last_fullsized': '2024-03-24T00:40:39.287297+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T00:40:45.018 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.25 is not active+clean 2024-03-24T00:40:45.018 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.286083+0000', 'last_change': '2024-03-24T00:40:39.286083+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.286083+0000', 'last_undegraded': '2024-03-24T00:40:39.286083+0000', 'last_fullsized': '2024-03-24T00:40:39.286083+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-24T00:40:45.018 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.24 is not active+clean 2024-03-24T00:40:45.019 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.287284+0000', 'last_change': '2024-03-24T00:40:39.287284+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.287284+0000', 'last_undegraded': '2024-03-24T00:40:39.287284+0000', 'last_fullsized': '2024-03-24T00:40:39.287284+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-24T00:40:45.019 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.23 is not active+clean 2024-03-24T00:40:45.019 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.291689+0000', 'last_change': '2024-03-24T00:40:39.291689+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.291689+0000', 'last_undegraded': '2024-03-24T00:40:39.291689+0000', 'last_fullsized': '2024-03-24T00:40:39.291689+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T00:40:45.019 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.22 is not active+clean 2024-03-24T00:40:45.019 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.286764+0000', 'last_change': '2024-03-24T00:40:39.286764+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.286764+0000', 'last_undegraded': '2024-03-24T00:40:39.286764+0000', 'last_fullsized': '2024-03-24T00:40:39.286764+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T00:40:45.019 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.21 is not active+clean 2024-03-24T00:40:45.019 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.289986+0000', 'last_change': '2024-03-24T00:40:39.289986+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.289986+0000', 'last_undegraded': '2024-03-24T00:40:39.289986+0000', 'last_fullsized': '2024-03-24T00:40:39.289986+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T00:40:45.019 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.20 is not active+clean 2024-03-24T00:40:45.019 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.284737+0000', 'last_change': '2024-03-24T00:40:39.284737+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.284737+0000', 'last_undegraded': '2024-03-24T00:40:39.284737+0000', 'last_fullsized': '2024-03-24T00:40:39.284737+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-24T00:40:45.019 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.10 is not active+clean 2024-03-24T00:40:45.019 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.290205+0000', 'last_change': '2024-03-24T00:40:39.290205+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.290205+0000', 'last_undegraded': '2024-03-24T00:40:39.290205+0000', 'last_fullsized': '2024-03-24T00:40:39.290205+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T00:40:45.019 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.11 is not active+clean 2024-03-24T00:40:45.020 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.283698+0000', 'last_change': '2024-03-24T00:40:39.283698+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.283698+0000', 'last_undegraded': '2024-03-24T00:40:39.283698+0000', 'last_fullsized': '2024-03-24T00:40:39.283698+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-24T00:40:45.020 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.12 is not active+clean 2024-03-24T00:40:45.020 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.289688+0000', 'last_change': '2024-03-24T00:40:39.289688+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.289688+0000', 'last_undegraded': '2024-03-24T00:40:39.289688+0000', 'last_fullsized': '2024-03-24T00:40:39.289688+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-24T00:40:45.020 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.13 is not active+clean 2024-03-24T00:40:45.020 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.295225+0000', 'last_change': '2024-03-24T00:40:39.295225+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.295225+0000', 'last_undegraded': '2024-03-24T00:40:39.295225+0000', 'last_fullsized': '2024-03-24T00:40:39.295225+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T00:40:45.020 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.14 is not active+clean 2024-03-24T00:40:45.020 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.286437+0000', 'last_change': '2024-03-24T00:40:39.286437+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.286437+0000', 'last_undegraded': '2024-03-24T00:40:39.286437+0000', 'last_fullsized': '2024-03-24T00:40:39.286437+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-24T00:40:45.020 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.15 is not active+clean 2024-03-24T00:40:45.020 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.286866+0000', 'last_change': '2024-03-24T00:40:39.286866+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.286866+0000', 'last_undegraded': '2024-03-24T00:40:39.286866+0000', 'last_fullsized': '2024-03-24T00:40:39.286866+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T00:40:45.020 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.16 is not active+clean 2024-03-24T00:40:45.020 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.285844+0000', 'last_change': '2024-03-24T00:40:39.285844+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.285844+0000', 'last_undegraded': '2024-03-24T00:40:39.285844+0000', 'last_fullsized': '2024-03-24T00:40:39.285844+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T00:40:45.021 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.17 is not active+clean 2024-03-24T00:40:45.021 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.290406+0000', 'last_change': '2024-03-24T00:40:39.290406+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.290406+0000', 'last_undegraded': '2024-03-24T00:40:39.290406+0000', 'last_fullsized': '2024-03-24T00:40:39.290406+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T00:40:45.021 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.18 is not active+clean 2024-03-24T00:40:45.021 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.286586+0000', 'last_change': '2024-03-24T00:40:39.286586+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.286586+0000', 'last_undegraded': '2024-03-24T00:40:39.286586+0000', 'last_fullsized': '2024-03-24T00:40:39.286586+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-24T00:40:45.021 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.19 is not active+clean 2024-03-24T00:40:45.021 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.284282+0000', 'last_change': '2024-03-24T00:40:39.284282+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.284282+0000', 'last_undegraded': '2024-03-24T00:40:39.284282+0000', 'last_fullsized': '2024-03-24T00:40:39.284282+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-24T00:40:45.021 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1a is not active+clean 2024-03-24T00:40:45.021 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.285258+0000', 'last_change': '2024-03-24T00:40:39.285258+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.285258+0000', 'last_undegraded': '2024-03-24T00:40:39.285258+0000', 'last_fullsized': '2024-03-24T00:40:39.285258+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T00:40:45.021 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1b is not active+clean 2024-03-24T00:40:45.021 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.287678+0000', 'last_change': '2024-03-24T00:40:39.287678+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.287678+0000', 'last_undegraded': '2024-03-24T00:40:39.287678+0000', 'last_fullsized': '2024-03-24T00:40:39.287678+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-24T00:40:45.021 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1c is not active+clean 2024-03-24T00:40:45.021 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.284454+0000', 'last_change': '2024-03-24T00:40:39.284454+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.284454+0000', 'last_undegraded': '2024-03-24T00:40:39.284454+0000', 'last_fullsized': '2024-03-24T00:40:39.284454+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-24T00:40:45.022 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1d is not active+clean 2024-03-24T00:40:45.022 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.286589+0000', 'last_change': '2024-03-24T00:40:39.286589+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.286589+0000', 'last_undegraded': '2024-03-24T00:40:39.286589+0000', 'last_fullsized': '2024-03-24T00:40:39.286589+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T00:40:45.022 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1e is not active+clean 2024-03-24T00:40:45.022 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.286895+0000', 'last_change': '2024-03-24T00:40:39.286895+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.286895+0000', 'last_undegraded': '2024-03-24T00:40:39.286895+0000', 'last_fullsized': '2024-03-24T00:40:39.286895+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T00:40:45.022 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1f is not active+clean 2024-03-24T00:40:45.022 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.288224+0000', 'last_change': '2024-03-24T00:40:39.288224+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.288224+0000', 'last_undegraded': '2024-03-24T00:40:39.288224+0000', 'last_fullsized': '2024-03-24T00:40:39.288224+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T00:40:45.022 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.0 is not active+clean 2024-03-24T00:40:45.022 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.285444+0000', 'last_change': '2024-03-24T00:40:39.285444+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.285444+0000', 'last_undegraded': '2024-03-24T00:40:39.285444+0000', 'last_fullsized': '2024-03-24T00:40:39.285444+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T00:40:45.022 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1 is not active+clean 2024-03-24T00:40:45.022 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.284942+0000', 'last_change': '2024-03-24T00:40:39.284942+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.284942+0000', 'last_undegraded': '2024-03-24T00:40:39.284942+0000', 'last_fullsized': '2024-03-24T00:40:39.284942+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-24T00:40:45.022 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2 is not active+clean 2024-03-24T00:40:45.023 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.287303+0000', 'last_change': '2024-03-24T00:40:39.287303+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.287303+0000', 'last_undegraded': '2024-03-24T00:40:39.287303+0000', 'last_fullsized': '2024-03-24T00:40:39.287303+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-24T00:40:45.023 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3 is not active+clean 2024-03-24T00:40:45.023 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.284674+0000', 'last_change': '2024-03-24T00:40:39.284674+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.284674+0000', 'last_undegraded': '2024-03-24T00:40:39.284674+0000', 'last_fullsized': '2024-03-24T00:40:39.284674+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-24T00:40:45.023 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.4 is not active+clean 2024-03-24T00:40:45.023 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.283875+0000', 'last_change': '2024-03-24T00:40:39.283875+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.283875+0000', 'last_undegraded': '2024-03-24T00:40:39.283875+0000', 'last_fullsized': '2024-03-24T00:40:39.283875+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-24T00:40:45.023 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.5 is not active+clean 2024-03-24T00:40:45.023 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.289204+0000', 'last_change': '2024-03-24T00:40:39.289204+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.289204+0000', 'last_undegraded': '2024-03-24T00:40:39.289204+0000', 'last_fullsized': '2024-03-24T00:40:39.289204+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-24T00:40:45.023 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.6 is not active+clean 2024-03-24T00:40:45.023 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.287275+0000', 'last_change': '2024-03-24T00:40:39.287275+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.287275+0000', 'last_undegraded': '2024-03-24T00:40:39.287275+0000', 'last_fullsized': '2024-03-24T00:40:39.287275+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T00:40:45.023 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.7 is not active+clean 2024-03-24T00:40:45.023 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.285594+0000', 'last_change': '2024-03-24T00:40:39.285594+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.285594+0000', 'last_undegraded': '2024-03-24T00:40:39.285594+0000', 'last_fullsized': '2024-03-24T00:40:39.285594+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-24T00:40:45.024 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.8 is not active+clean 2024-03-24T00:40:45.024 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.283792+0000', 'last_change': '2024-03-24T00:40:39.283792+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.283792+0000', 'last_undegraded': '2024-03-24T00:40:39.283792+0000', 'last_fullsized': '2024-03-24T00:40:39.283792+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-24T00:40:45.024 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.a is not active+clean 2024-03-24T00:40:45.024 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.285596+0000', 'last_change': '2024-03-24T00:40:39.285596+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.285596+0000', 'last_undegraded': '2024-03-24T00:40:39.285596+0000', 'last_fullsized': '2024-03-24T00:40:39.285596+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-24T00:40:45.024 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.b is not active+clean 2024-03-24T00:40:45.024 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.289013+0000', 'last_change': '2024-03-24T00:40:39.289013+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.289013+0000', 'last_undegraded': '2024-03-24T00:40:39.289013+0000', 'last_fullsized': '2024-03-24T00:40:39.289013+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T00:40:45.024 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.c is not active+clean 2024-03-24T00:40:45.024 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.285991+0000', 'last_change': '2024-03-24T00:40:39.285991+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.285991+0000', 'last_undegraded': '2024-03-24T00:40:39.285991+0000', 'last_fullsized': '2024-03-24T00:40:39.285991+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T00:40:45.024 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.d is not active+clean 2024-03-24T00:40:45.024 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.283715+0000', 'last_change': '2024-03-24T00:40:39.283715+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.283715+0000', 'last_undegraded': '2024-03-24T00:40:39.283715+0000', 'last_fullsized': '2024-03-24T00:40:39.283715+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-24T00:40:45.024 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.e is not active+clean 2024-03-24T00:40:45.024 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.287710+0000', 'last_change': '2024-03-24T00:40:39.287710+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.287710+0000', 'last_undegraded': '2024-03-24T00:40:39.287710+0000', 'last_fullsized': '2024-03-24T00:40:39.287710+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T00:40:45.024 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.f is not active+clean 2024-03-24T00:40:45.025 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.286808+0000', 'last_change': '2024-03-24T00:40:39.286808+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.286808+0000', 'last_undegraded': '2024-03-24T00:40:39.286808+0000', 'last_fullsized': '2024-03-24T00:40:39.286808+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T00:40:45.025 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.9 is not active+clean 2024-03-24T00:40:45.025 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:40:39.289541+0000', 'last_change': '2024-03-24T00:40:39.289541+0000', 'last_active': '2024-03-24T00:40:39.278047+0000', 'last_peered': '2024-03-24T00:40:39.278047+0000', 'last_clean': '2024-03-24T00:40:39.278047+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:40:39.289541+0000', 'last_undegraded': '2024-03-24T00:40:39.289541+0000', 'last_fullsized': '2024-03-24T00:40:39.289541+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'last_clean_scrub_stamp': '2024-03-24T00:40:39.278047+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-24T00:40:45.025 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-24T00:40:45.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.153+0000 7f771c0c6640 1 Processor -- start 2024-03-24T00:40:45.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.154+0000 7f771c0c6640 1 -- start start 2024-03-24T00:40:45.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.154+0000 7f771c0c6640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f771415dec0 0x7f77140ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:45.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.154+0000 7f771c0c6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f77140d0ef0 0x7f7714162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:45.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.154+0000 7f771c0c6640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77140cf640 con 0x7f771415e2a0 2024-03-24T00:40:45.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.154+0000 7f771c0c6640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77140cee30 con 0x7f771415dec0 2024-03-24T00:40:45.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.154+0000 7f771c0c6640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77140d2120 con 0x7f77140d0ef0 2024-03-24T00:40:45.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.155+0000 7f7719e62640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f771415dec0 0x7f77140ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:45.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.155+0000 7f7719661640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f771415e2a0 0x7f77140d07e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39622/0 (socket says 172.21.15.67:39622) 2024-03-24T00:40:45.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.155+0000 7f7719661640 1 -- 172.21.15.67:0/813430251 learned_addr learned my addr 172.21.15.67:0/813430251 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:40:45.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.155+0000 7f771a663640 1 --2- 172.21.15.67:0/813430251 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f77140d0ef0 0x7f7714162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:45.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.155+0000 7f7719e62640 1 -- 172.21.15.67:0/813430251 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f77140d0ef0 msgr2=0x7f7714162630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:45.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.155+0000 7f7719e62640 1 --2- 172.21.15.67:0/813430251 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f77140d0ef0 0x7f7714162630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:45.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.155+0000 7f7719e62640 1 -- 172.21.15.67:0/813430251 >> v1:172.21.15.67:6789/0 conn(0x7f771415e2a0 legacy=0x7f77140d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:45.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.155+0000 7f7719e62640 1 -- 172.21.15.67:0/813430251 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77140cf080 con 0x7f771415dec0 2024-03-24T00:40:45.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.155+0000 7f7718e60640 1 -- 172.21.15.67:0/813430251 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4165280431 0 0) 0x7f7704001110 con 0x7f771415e2a0 2024-03-24T00:40:45.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.155+0000 7f771a663640 1 --2- 172.21.15.67:0/813430251 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f77140d0ef0 0x7f7714162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:40:45.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.156+0000 7f7719e62640 1 --2- 172.21.15.67:0/813430251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f771415dec0 0x7f77140ce630 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f77100097a0 tx=0x7f771002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad4a89013ba9dc3b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:45.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.156+0000 7f7718e60640 1 -- 172.21.15.67:0/813430251 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77100315a0 con 0x7f771415dec0 2024-03-24T00:40:45.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.156+0000 7f7718e60640 1 -- 172.21.15.67:0/813430251 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7710037070 con 0x7f771415dec0 2024-03-24T00:40:45.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.156+0000 7f7718e60640 1 -- 172.21.15.67:0/813430251 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7710031960 con 0x7f771415dec0 2024-03-24T00:40:45.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.156+0000 7f771c0c6640 1 -- 172.21.15.67:0/813430251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f771415dec0 msgr2=0x7f77140ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:45.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.156+0000 7f771c0c6640 1 --2- 172.21.15.67:0/813430251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f771415dec0 0x7f77140ce630 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f77100097a0 tx=0x7f771002b840 comp rx=0 tx=0).stop 2024-03-24T00:40:45.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.156+0000 7f771c0c6640 1 -- 172.21.15.67:0/813430251 shutdown_connections 2024-03-24T00:40:45.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.156+0000 7f771c0c6640 1 --2- 172.21.15.67:0/813430251 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f77140d0ef0 0x7f7714162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:45.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.156+0000 7f771c0c6640 1 --2- 172.21.15.67:0/813430251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f771415dec0 0x7f77140ce630 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:45.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.156+0000 7f771c0c6640 1 -- 172.21.15.67:0/813430251 >> 172.21.15.67:0/813430251 conn(0x7f771406c810 msgr2=0x7f771406cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:45.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.156+0000 7f771c0c6640 1 -- 172.21.15.67:0/813430251 shutdown_connections 2024-03-24T00:40:45.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.156+0000 7f771c0c6640 1 -- 172.21.15.67:0/813430251 wait complete. 2024-03-24T00:40:45.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.158+0000 7f771c0c6640 1 Processor -- start 2024-03-24T00:40:45.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.158+0000 7f771c0c6640 1 -- start start 2024-03-24T00:40:45.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.159+0000 7f771c0c6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f77140d0ef0 0x7f77140ae1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:45.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.159+0000 7f771c0c6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f771415dec0 0x7f77140ab280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:45.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.159+0000 7f771c0c6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f771415e2a0 0x7f77140cd2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:45.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.159+0000 7f771c0c6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77140d2730 con 0x7f771415e2a0 2024-03-24T00:40:45.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.159+0000 7f771c0c6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77140d25b0 con 0x7f771415dec0 2024-03-24T00:40:45.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.159+0000 7f771c0c6640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77140d28b0 con 0x7f77140d0ef0 2024-03-24T00:40:45.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.160+0000 7f771a663640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f771415e2a0 0x7f77140cd2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:45.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.160+0000 7f771a663640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f771415e2a0 0x7f77140cd2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40810/0 (socket says 172.21.15.67:40810) 2024-03-24T00:40:45.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.160+0000 7f771a663640 1 -- 172.21.15.67:0/144251981 learned_addr learned my addr 172.21.15.67:0/144251981 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:45.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.160+0000 7f7719e62640 1 --2- 172.21.15.67:0/144251981 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f77140d0ef0 0x7f77140ae1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:45.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.160+0000 7f7719661640 1 --2- 172.21.15.67:0/144251981 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f771415dec0 0x7f77140ab280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:45.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.160+0000 7f771a663640 1 -- 172.21.15.67:0/144251981 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f77140d0ef0 msgr2=0x7f77140ae1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:45.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.160+0000 7f771a663640 1 --2- 172.21.15.67:0/144251981 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f77140d0ef0 0x7f77140ae1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:45.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.160+0000 7f771a663640 1 -- 172.21.15.67:0/144251981 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f771415dec0 msgr2=0x7f77140ab280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:45.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.160+0000 7f771a663640 1 --2- 172.21.15.67:0/144251981 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f771415dec0 0x7f77140ab280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:45.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.160+0000 7f771a663640 1 -- 172.21.15.67:0/144251981 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77140ae850 con 0x7f771415e2a0 2024-03-24T00:40:45.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.160+0000 7f7719661640 1 --2- 172.21.15.67:0/144251981 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f771415dec0 0x7f77140ab280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:40:45.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.160+0000 7f771a663640 1 --2- 172.21.15.67:0/144251981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f771415e2a0 0x7f77140cd2e0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f770c009f70 tx=0x7f770c00d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:45.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.160+0000 7f770a7fc640 1 -- 172.21.15.67:0/144251981 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f770c01a070 con 0x7f771415e2a0 2024-03-24T00:40:45.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.160+0000 7f770a7fc640 1 -- 172.21.15.67:0/144251981 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f770c003bb0 con 0x7f771415e2a0 2024-03-24T00:40:45.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.160+0000 7f770a7fc640 1 -- 172.21.15.67:0/144251981 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f770c004030 con 0x7f771415e2a0 2024-03-24T00:40:45.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.160+0000 7f771c0c6640 1 -- 172.21.15.67:0/144251981 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77140aeb30 con 0x7f771415e2a0 2024-03-24T00:40:45.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.160+0000 7f771c0c6640 1 -- 172.21.15.67:0/144251981 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77140af0f0 con 0x7f771415e2a0 2024-03-24T00:40:45.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.161+0000 7f771c0c6640 1 -- 172.21.15.67:0/144251981 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f771404f400 con 0x7f771415e2a0 2024-03-24T00:40:45.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.162+0000 7f770a7fc640 1 -- 172.21.15.67:0/144251981 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f770c01d070 con 0x7f771415e2a0 2024-03-24T00:40:45.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.162+0000 7f770a7fc640 1 --2- 172.21.15.67:0/144251981 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f76f806bbd0 0x7f76f806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:45.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.162+0000 7f770a7fc640 1 -- 172.21.15.67:0/144251981 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f770c08ae90 con 0x7f771415e2a0 2024-03-24T00:40:45.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.162+0000 7f7719e62640 1 --2- 172.21.15.67:0/144251981 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f76f806bbd0 0x7f76f806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:45.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.163+0000 7f7719e62640 1 --2- 172.21.15.67:0/144251981 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f76f806bbd0 0x7f76f806e090 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f7710036000 tx=0x7f7710008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:45.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.166+0000 7f770a7fc640 1 -- 172.21.15.67:0/144251981 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f770c010070 con 0x7f771415e2a0 2024-03-24T00:40:45.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.421+0000 7f771c0c6640 1 -- 172.21.15.67:0/144251981 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f77140d1aa0 con 0x7f771415e2a0 2024-03-24T00:40:45.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.421+0000 7f770a7fc640 1 -- 172.21.15.67:0/144251981 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1195 (secure 0 0 0) 0x7f770c059880 con 0x7f771415e2a0 2024-03-24T00:40:45.422 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:40:45.422 INFO:teuthology.orchestra.run.smithi067.stdout:{"fsid":"de1d0178-808e-49ca-a986-e297e523b976","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":138,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":50,"num_osds":8,"num_up_osds":8,"osd_up_since":1711240718,"num_in_osds":8,"osd_in_since":1711240709,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":33}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":227004416,"bytes_avail":772867108864,"bytes_total":773094113280,"inactive_pgs_ratio":0.65979379415512085},"fsmap":{"epoch":26,"id":3,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":3,"rank":0,"name":"a","status":"up:active","gid":5853}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-03-24T00:39:47.335006+0000","services":{"mds":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-24T00:40:45.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.423+0000 7f771c0c6640 1 -- 172.21.15.67:0/144251981 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f76f806bbd0 msgr2=0x7f76f806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:45.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.423+0000 7f771c0c6640 1 --2- 172.21.15.67:0/144251981 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f76f806bbd0 0x7f76f806e090 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f7710036000 tx=0x7f7710008030 comp rx=0 tx=0).stop 2024-03-24T00:40:45.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.423+0000 7f771c0c6640 1 -- 172.21.15.67:0/144251981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f771415e2a0 msgr2=0x7f77140cd2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:45.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.423+0000 7f771c0c6640 1 --2- 172.21.15.67:0/144251981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f771415e2a0 0x7f77140cd2e0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f770c009f70 tx=0x7f770c00d560 comp rx=0 tx=0).stop 2024-03-24T00:40:45.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.423+0000 7f771c0c6640 1 -- 172.21.15.67:0/144251981 shutdown_connections 2024-03-24T00:40:45.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.423+0000 7f771c0c6640 1 --2- 172.21.15.67:0/144251981 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f76f806bbd0 0x7f76f806e090 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:45.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.423+0000 7f771c0c6640 1 --2- 172.21.15.67:0/144251981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f771415e2a0 0x7f77140cd2e0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:45.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.423+0000 7f771c0c6640 1 --2- 172.21.15.67:0/144251981 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f771415dec0 0x7f77140ab280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:45.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.423+0000 7f771c0c6640 1 --2- 172.21.15.67:0/144251981 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f77140d0ef0 0x7f77140ae1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:45.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.423+0000 7f771c0c6640 1 -- 172.21.15.67:0/144251981 >> 172.21.15.67:0/144251981 conn(0x7f771406c810 msgr2=0x7f77140b61c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:45.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.423+0000 7f771c0c6640 1 -- 172.21.15.67:0/144251981 shutdown_connections 2024-03-24T00:40:45.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.423+0000 7f771c0c6640 1 -- 172.21.15.67:0/144251981 wait complete. 2024-03-24T00:40:45.435 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-24T00:40:45.435 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:40:45.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.535+0000 7f5aceddf640 1 Processor -- start 2024-03-24T00:40:45.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.535+0000 7f5aceddf640 1 -- start start 2024-03-24T00:40:45.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.537+0000 7f5aceddf640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5ac815e2a0 0x7f5ac80ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:45.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.537+0000 7f5aceddf640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ac80cf640 con 0x7f5ac80d0ef0 2024-03-24T00:40:45.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.537+0000 7f5aceddf640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ac80cf080 con 0x7f5ac815dec0 2024-03-24T00:40:45.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.537+0000 7f5aceddf640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ac80cee30 con 0x7f5ac815e2a0 2024-03-24T00:40:45.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.537+0000 7f5ac7fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5ac815e2a0 0x7f5ac80ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:45.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.537+0000 7f5acd37c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5ac80d0ef0 0x7f5ac80d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39638/0 (socket says 172.21.15.67:39638) 2024-03-24T00:40:45.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.537+0000 7f5acd37c640 1 -- 172.21.15.67:0/1227534829 learned_addr learned my addr 172.21.15.67:0/1227534829 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:40:45.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.537+0000 7f5ac77fe640 1 -- 172.21.15.67:0/1227534829 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1509668028 0 0) 0x7f5ab0001110 con 0x7f5ac80d0ef0 2024-03-24T00:40:45.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.537+0000 7f5ac77fe640 1 -- 172.21.15.67:0/1227534829 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ab4003520 con 0x7f5ac80d0ef0 2024-03-24T00:40:45.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.537+0000 7f5ac7fff640 1 -- 172.21.15.67:0/1227534829 >> v1:172.21.15.72:6789/0 conn(0x7f5ac815dec0 legacy=0x7f5ac80d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:45.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.537+0000 7f5ac7fff640 1 -- 172.21.15.67:0/1227534829 >> v1:172.21.15.67:6789/0 conn(0x7f5ac80d0ef0 legacy=0x7f5ac80d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:45.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.537+0000 7f5ac7fff640 1 -- 172.21.15.67:0/1227534829 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ac80cf360 con 0x7f5ac815e2a0 2024-03-24T00:40:45.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.538+0000 7f5ac7fff640 1 --2- 172.21.15.67:0/1227534829 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ac815e2a0 0x7f5ac80ce630 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f5ac80cefb0 tx=0x7f5ab802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc08b5854f6a9728 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:45.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.538+0000 7f5ac77fe640 1 -- 172.21.15.67:0/1227534829 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab8039070 con 0x7f5ac815e2a0 2024-03-24T00:40:45.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.538+0000 7f5ac77fe640 1 -- 172.21.15.67:0/1227534829 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ab802cc50 con 0x7f5ac815e2a0 2024-03-24T00:40:45.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.538+0000 7f5ac77fe640 1 -- 172.21.15.67:0/1227534829 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab8034440 con 0x7f5ac815e2a0 2024-03-24T00:40:45.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.539+0000 7f5aceddf640 1 -- 172.21.15.67:0/1227534829 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ac815e2a0 msgr2=0x7f5ac80ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:45.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.539+0000 7f5aceddf640 1 --2- 172.21.15.67:0/1227534829 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ac815e2a0 0x7f5ac80ce630 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f5ac80cefb0 tx=0x7f5ab802bea0 comp rx=0 tx=0).stop 2024-03-24T00:40:45.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.539+0000 7f5aceddf640 1 -- 172.21.15.67:0/1227534829 shutdown_connections 2024-03-24T00:40:45.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.539+0000 7f5aceddf640 1 --2- 172.21.15.67:0/1227534829 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ac815e2a0 0x7f5ac80ce630 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:45.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.539+0000 7f5aceddf640 1 -- 172.21.15.67:0/1227534829 >> 172.21.15.67:0/1227534829 conn(0x7f5ac806c810 msgr2=0x7f5ac806cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:45.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.539+0000 7f5aceddf640 1 -- 172.21.15.67:0/1227534829 shutdown_connections 2024-03-24T00:40:45.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.539+0000 7f5aceddf640 1 -- 172.21.15.67:0/1227534829 wait complete. 2024-03-24T00:40:45.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.540+0000 7f5aceddf640 1 Processor -- start 2024-03-24T00:40:45.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.540+0000 7f5aceddf640 1 -- start start 2024-03-24T00:40:45.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.541+0000 7f5aceddf640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ac80d0ef0 0x7f5ac81e9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:45.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.541+0000 7f5aceddf640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5ac815dec0 0x7f5ac81ea020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:45.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.541+0000 7f5aceddf640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ac815e2a0 0x7f5ac81e8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:45.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.541+0000 7f5aceddf640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac8162a40 con 0x7f5ac815e2a0 2024-03-24T00:40:45.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.541+0000 7f5aceddf640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac81628c0 con 0x7f5ac80d0ef0 2024-03-24T00:40:45.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.541+0000 7f5aceddf640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac80d2120 con 0x7f5ac815dec0 2024-03-24T00:40:45.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.541+0000 7f5acd37c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ac815e2a0 0x7f5ac81e8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:45.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.541+0000 7f5acd37c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ac815e2a0 0x7f5ac81e8a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40830/0 (socket says 172.21.15.67:40830) 2024-03-24T00:40:45.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.541+0000 7f5acd37c640 1 -- 172.21.15.67:0/1370288238 learned_addr learned my addr 172.21.15.67:0/1370288238 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:45.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.541+0000 7f5accb7b640 1 --2- 172.21.15.67:0/1370288238 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ac80d0ef0 0x7f5ac81e9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:45.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.541+0000 7f5ac7fff640 1 --2- 172.21.15.67:0/1370288238 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5ac815dec0 0x7f5ac81ea020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:45.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.541+0000 7f5acd37c640 1 -- 172.21.15.67:0/1370288238 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5ac815dec0 msgr2=0x7f5ac81ea020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:45.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.541+0000 7f5acd37c640 1 --2- 172.21.15.67:0/1370288238 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5ac815dec0 0x7f5ac81ea020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:45.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.541+0000 7f5acd37c640 1 -- 172.21.15.67:0/1370288238 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ac80d0ef0 msgr2=0x7f5ac81e9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:45.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.541+0000 7f5acd37c640 1 --2- 172.21.15.67:0/1370288238 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ac80d0ef0 0x7f5ac81e9ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:45.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.541+0000 7f5acd37c640 1 -- 172.21.15.67:0/1370288238 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ac8157880 con 0x7f5ac815e2a0 2024-03-24T00:40:45.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.541+0000 7f5accb7b640 1 --2- 172.21.15.67:0/1370288238 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ac80d0ef0 0x7f5ac81e9ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:40:45.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.541+0000 7f5ac7fff640 1 --2- 172.21.15.67:0/1370288238 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5ac815dec0 0x7f5ac81ea020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:40:45.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.542+0000 7f5acd37c640 1 --2- 172.21.15.67:0/1370288238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ac815e2a0 0x7f5ac81e8a20 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f5ab000c730 tx=0x7f5ab000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:45.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.542+0000 7f5ac57fa640 1 -- 172.21.15.67:0/1370288238 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab0017070 con 0x7f5ac815e2a0 2024-03-24T00:40:45.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.542+0000 7f5ac57fa640 1 -- 172.21.15.67:0/1370288238 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ab00058b0 con 0x7f5ac815e2a0 2024-03-24T00:40:45.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.542+0000 7f5aceddf640 1 -- 172.21.15.67:0/1370288238 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ac80b4820 con 0x7f5ac815e2a0 2024-03-24T00:40:45.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.542+0000 7f5ac57fa640 1 -- 172.21.15.67:0/1370288238 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab0005ce0 con 0x7f5ac815e2a0 2024-03-24T00:40:45.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.542+0000 7f5aceddf640 1 -- 172.21.15.67:0/1370288238 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ac80b4de0 con 0x7f5ac815e2a0 2024-03-24T00:40:45.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.543+0000 7f5aceddf640 1 -- 172.21.15.67:0/1370288238 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ac804f400 con 0x7f5ac815e2a0 2024-03-24T00:40:45.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.543+0000 7f5ac57fa640 1 -- 172.21.15.67:0/1370288238 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5ab001a070 con 0x7f5ac815e2a0 2024-03-24T00:40:45.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.544+0000 7f5ac57fa640 1 --2- 172.21.15.67:0/1370288238 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5aa806bc00 0x7f5aa806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:45.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.544+0000 7f5ac57fa640 1 -- 172.21.15.67:0/1370288238 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5ab0089ee0 con 0x7f5ac815e2a0 2024-03-24T00:40:45.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.544+0000 7f5accb7b640 1 --2- 172.21.15.67:0/1370288238 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5aa806bc00 0x7f5aa806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:45.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.544+0000 7f5accb7b640 1 --2- 172.21.15.67:0/1370288238 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5aa806bc00 0x7f5aa806e0c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f5ac80ca920 tx=0x7f5ac0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:45.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.548+0000 7f5ac57fa640 1 -- 172.21.15.67:0/1370288238 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5ac80cf640 con 0x7f5ac815e2a0 2024-03-24T00:40:45.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.875+0000 7f5aceddf640 1 -- 172.21.15.67:0/1370288238 --> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5ac80b45d0 con 0x7f5aa806bc00 2024-03-24T00:40:45.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.882+0000 7f5ac57fa640 1 -- 172.21.15.67:0/1370288238 <== mgr.4105 v2:172.21.15.72:6832/2768292274 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251957 (secure 0 0 0) 0x7f5ac80b45d0 con 0x7f5aa806bc00 2024-03-24T00:40:45.885 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:40:45.886 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-24T00:40:45.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.885+0000 7f5aceddf640 1 -- 172.21.15.67:0/1370288238 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5aa806bc00 msgr2=0x7f5aa806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:45.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.885+0000 7f5aceddf640 1 --2- 172.21.15.67:0/1370288238 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5aa806bc00 0x7f5aa806e0c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f5ac80ca920 tx=0x7f5ac0009040 comp rx=0 tx=0).stop 2024-03-24T00:40:45.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.885+0000 7f5aceddf640 1 -- 172.21.15.67:0/1370288238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ac815e2a0 msgr2=0x7f5ac81e8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:45.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.885+0000 7f5aceddf640 1 --2- 172.21.15.67:0/1370288238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ac815e2a0 0x7f5ac81e8a20 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f5ab000c730 tx=0x7f5ab000cbf0 comp rx=0 tx=0).stop 2024-03-24T00:40:45.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.885+0000 7f5aceddf640 1 -- 172.21.15.67:0/1370288238 shutdown_connections 2024-03-24T00:40:45.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.885+0000 7f5aceddf640 1 --2- 172.21.15.67:0/1370288238 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5aa806bc00 0x7f5aa806e0c0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:45.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.885+0000 7f5aceddf640 1 --2- 172.21.15.67:0/1370288238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ac815e2a0 0x7f5ac81e8a20 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:45.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.885+0000 7f5aceddf640 1 --2- 172.21.15.67:0/1370288238 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5ac815dec0 0x7f5ac81ea020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:45.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.885+0000 7f5aceddf640 1 --2- 172.21.15.67:0/1370288238 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ac80d0ef0 0x7f5ac81e9ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:45.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.885+0000 7f5aceddf640 1 -- 172.21.15.67:0/1370288238 >> 172.21.15.67:0/1370288238 conn(0x7f5ac806c810 msgr2=0x7f5ac80b7090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:45.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.885+0000 7f5aceddf640 1 -- 172.21.15.67:0/1370288238 shutdown_connections 2024-03-24T00:40:45.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:45.885+0000 7f5aceddf640 1 -- 172.21.15.67:0/1370288238 wait complete. 2024-03-24T00:40:45.896 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":106,"stamp":"2024-03-24T00:40:45.345928+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222052,"kb_used_data":5864,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752668,"statfs":{"total":773094113280,"available":772866732032,"internally_reserved":0,"allocated":6004736,"data_stored":3304260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.064504"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.586509+0000","last_change":"2024-03-24T00:40:40.304215+0000","last_active":"2024-03-24T00:40:41.586509+0000","last_peered":"2024-03-24T00:40:41.586509+0000","last_clean":"2024-03-24T00:40:41.586509+0000","last_became_active":"2024-03-24T00:40:40.303929+0000","last_became_peered":"2024-03-24T00:40:40.303929+0000","last_unstale":"2024-03-24T00:40:41.586509+0000","last_undegraded":"2024-03-24T00:40:41.586509+0000","last_fullsized":"2024-03-24T00:40:41.586509+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:57:36.878352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.302282+0000","last_change":"2024-03-24T00:40:40.300086+0000","last_active":"2024-03-24T00:40:41.302282+0000","last_peered":"2024-03-24T00:40:41.302282+0000","last_clean":"2024-03-24T00:40:41.302282+0000","last_became_active":"2024-03-24T00:40:40.299601+0000","last_became_peered":"2024-03-24T00:40:40.299601+0000","last_unstale":"2024-03-24T00:40:41.302282+0000","last_undegraded":"2024-03-24T00:40:41.302282+0000","last_fullsized":"2024-03-24T00:40:41.302282+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:47:38.271047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.298821+0000","last_change":"2024-03-24T00:40:40.289947+0000","last_active":"2024-03-24T00:40:41.298821+0000","last_peered":"2024-03-24T00:40:41.298821+0000","last_clean":"2024-03-24T00:40:41.298821+0000","last_became_active":"2024-03-24T00:40:40.289632+0000","last_became_peered":"2024-03-24T00:40:40.289632+0000","last_unstale":"2024-03-24T00:40:41.298821+0000","last_undegraded":"2024-03-24T00:40:41.298821+0000","last_fullsized":"2024-03-24T00:40:41.298821+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:45:44.359862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.301829+0000","last_change":"2024-03-24T00:40:40.305533+0000","last_active":"2024-03-24T00:40:41.301829+0000","last_peered":"2024-03-24T00:40:41.301829+0000","last_clean":"2024-03-24T00:40:41.301829+0000","last_became_active":"2024-03-24T00:40:40.305056+0000","last_became_peered":"2024-03-24T00:40:40.305056+0000","last_unstale":"2024-03-24T00:40:41.301829+0000","last_undegraded":"2024-03-24T00:40:41.301829+0000","last_fullsized":"2024-03-24T00:40:41.301829+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:04:50.495649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.845013+0000","last_change":"2024-03-24T00:40:40.300648+0000","last_active":"2024-03-24T00:40:41.845013+0000","last_peered":"2024-03-24T00:40:41.845013+0000","last_clean":"2024-03-24T00:40:41.845013+0000","last_became_active":"2024-03-24T00:40:40.300395+0000","last_became_peered":"2024-03-24T00:40:40.300395+0000","last_unstale":"2024-03-24T00:40:41.845013+0000","last_undegraded":"2024-03-24T00:40:41.845013+0000","last_fullsized":"2024-03-24T00:40:41.845013+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:19:39.784710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.586217+0000","last_change":"2024-03-24T00:40:40.304050+0000","last_active":"2024-03-24T00:40:41.586217+0000","last_peered":"2024-03-24T00:40:41.586217+0000","last_clean":"2024-03-24T00:40:41.586217+0000","last_became_active":"2024-03-24T00:40:40.303730+0000","last_became_peered":"2024-03-24T00:40:40.303730+0000","last_unstale":"2024-03-24T00:40:41.586217+0000","last_undegraded":"2024-03-24T00:40:41.586217+0000","last_fullsized":"2024-03-24T00:40:41.586217+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:39:53.235649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.298368+0000","last_change":"2024-03-24T00:40:40.289712+0000","last_active":"2024-03-24T00:40:41.298368+0000","last_peered":"2024-03-24T00:40:41.298368+0000","last_clean":"2024-03-24T00:40:41.298368+0000","last_became_active":"2024-03-24T00:40:40.289451+0000","last_became_peered":"2024-03-24T00:40:40.289451+0000","last_unstale":"2024-03-24T00:40:41.298368+0000","last_undegraded":"2024-03-24T00:40:41.298368+0000","last_fullsized":"2024-03-24T00:40:41.298368+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:16:13.256691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.295427+0000","last_change":"2024-03-24T00:40:40.307005+0000","last_active":"2024-03-24T00:40:41.295427+0000","last_peered":"2024-03-24T00:40:41.295427+0000","last_clean":"2024-03-24T00:40:41.295427+0000","last_became_active":"2024-03-24T00:40:40.306746+0000","last_became_peered":"2024-03-24T00:40:40.306746+0000","last_unstale":"2024-03-24T00:40:41.295427+0000","last_undegraded":"2024-03-24T00:40:41.295427+0000","last_fullsized":"2024-03-24T00:40:41.295427+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:42:06.753248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.295972+0000","last_change":"2024-03-24T00:40:40.300275+0000","last_active":"2024-03-24T00:40:41.295972+0000","last_peered":"2024-03-24T00:40:41.295972+0000","last_clean":"2024-03-24T00:40:41.295972+0000","last_became_active":"2024-03-24T00:40:40.298907+0000","last_became_peered":"2024-03-24T00:40:40.298907+0000","last_unstale":"2024-03-24T00:40:41.295972+0000","last_undegraded":"2024-03-24T00:40:41.295972+0000","last_fullsized":"2024-03-24T00:40:41.295972+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:23:33.487671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.586460+0000","last_change":"2024-03-24T00:40:40.302765+0000","last_active":"2024-03-24T00:40:41.586460+0000","last_peered":"2024-03-24T00:40:41.586460+0000","last_clean":"2024-03-24T00:40:41.586460+0000","last_became_active":"2024-03-24T00:40:40.302496+0000","last_became_peered":"2024-03-24T00:40:40.302496+0000","last_unstale":"2024-03-24T00:40:41.586460+0000","last_undegraded":"2024-03-24T00:40:41.586460+0000","last_fullsized":"2024-03-24T00:40:41.586460+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:04:50.495649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.300764+0000","last_change":"2024-03-24T00:40:40.297831+0000","last_active":"2024-03-24T00:40:41.300764+0000","last_peered":"2024-03-24T00:40:41.300764+0000","last_clean":"2024-03-24T00:40:41.300764+0000","last_became_active":"2024-03-24T00:40:40.297510+0000","last_became_peered":"2024-03-24T00:40:40.297510+0000","last_unstale":"2024-03-24T00:40:41.300764+0000","last_undegraded":"2024-03-24T00:40:41.300764+0000","last_fullsized":"2024-03-24T00:40:41.300764+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:30:40.670742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.295748+0000","last_change":"2024-03-24T00:40:40.307709+0000","last_active":"2024-03-24T00:40:41.295748+0000","last_peered":"2024-03-24T00:40:41.295748+0000","last_clean":"2024-03-24T00:40:41.295748+0000","last_became_active":"2024-03-24T00:40:40.307429+0000","last_became_peered":"2024-03-24T00:40:40.307429+0000","last_unstale":"2024-03-24T00:40:41.295748+0000","last_undegraded":"2024-03-24T00:40:41.295748+0000","last_fullsized":"2024-03-24T00:40:41.295748+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:38:20.268729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.586358+0000","last_change":"2024-03-24T00:40:40.303874+0000","last_active":"2024-03-24T00:40:41.586358+0000","last_peered":"2024-03-24T00:40:41.586358+0000","last_clean":"2024-03-24T00:40:41.586358+0000","last_became_active":"2024-03-24T00:40:40.303425+0000","last_became_peered":"2024-03-24T00:40:40.303425+0000","last_unstale":"2024-03-24T00:40:41.586358+0000","last_undegraded":"2024-03-24T00:40:41.586358+0000","last_fullsized":"2024-03-24T00:40:41.586358+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:23:26.624895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.298574+0000","last_change":"2024-03-24T00:40:40.289246+0000","last_active":"2024-03-24T00:40:41.298574+0000","last_peered":"2024-03-24T00:40:41.298574+0000","last_clean":"2024-03-24T00:40:41.298574+0000","last_became_active":"2024-03-24T00:40:40.288979+0000","last_became_peered":"2024-03-24T00:40:40.288979+0000","last_unstale":"2024-03-24T00:40:41.298574+0000","last_undegraded":"2024-03-24T00:40:41.298574+0000","last_fullsized":"2024-03-24T00:40:41.298574+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:18:03.493963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.301784+0000","last_change":"2024-03-24T00:40:40.298411+0000","last_active":"2024-03-24T00:40:41.301784+0000","last_peered":"2024-03-24T00:40:41.301784+0000","last_clean":"2024-03-24T00:40:41.301784+0000","last_became_active":"2024-03-24T00:40:40.298130+0000","last_became_peered":"2024-03-24T00:40:40.298130+0000","last_unstale":"2024-03-24T00:40:41.301784+0000","last_undegraded":"2024-03-24T00:40:41.301784+0000","last_fullsized":"2024-03-24T00:40:41.301784+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:23:26.624895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.586142+0000","last_change":"2024-03-24T00:40:40.301978+0000","last_active":"2024-03-24T00:40:41.586142+0000","last_peered":"2024-03-24T00:40:41.586142+0000","last_clean":"2024-03-24T00:40:41.586142+0000","last_became_active":"2024-03-24T00:40:40.301689+0000","last_became_peered":"2024-03-24T00:40:40.301689+0000","last_unstale":"2024-03-24T00:40:41.586142+0000","last_undegraded":"2024-03-24T00:40:41.586142+0000","last_fullsized":"2024-03-24T00:40:41.586142+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:05:33.548187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.299048+0000","last_change":"2024-03-24T00:40:40.292724+0000","last_active":"2024-03-24T00:40:41.299048+0000","last_peered":"2024-03-24T00:40:41.299048+0000","last_clean":"2024-03-24T00:40:41.299048+0000","last_became_active":"2024-03-24T00:40:40.291898+0000","last_became_peered":"2024-03-24T00:40:40.291898+0000","last_unstale":"2024-03-24T00:40:41.299048+0000","last_undegraded":"2024-03-24T00:40:41.299048+0000","last_fullsized":"2024-03-24T00:40:41.299048+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:34:06.342935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.302318+0000","last_change":"2024-03-24T00:40:40.299860+0000","last_active":"2024-03-24T00:40:41.302318+0000","last_peered":"2024-03-24T00:40:41.302318+0000","last_clean":"2024-03-24T00:40:41.302318+0000","last_became_active":"2024-03-24T00:40:40.299305+0000","last_became_peered":"2024-03-24T00:40:40.299305+0000","last_unstale":"2024-03-24T00:40:41.302318+0000","last_undegraded":"2024-03-24T00:40:41.302318+0000","last_fullsized":"2024-03-24T00:40:41.302318+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:35:52.681866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.298867+0000","last_change":"2024-03-24T00:40:40.296936+0000","last_active":"2024-03-24T00:40:41.298867+0000","last_peered":"2024-03-24T00:40:41.298867+0000","last_clean":"2024-03-24T00:40:41.298867+0000","last_became_active":"2024-03-24T00:40:40.296637+0000","last_became_peered":"2024-03-24T00:40:40.296637+0000","last_unstale":"2024-03-24T00:40:41.298867+0000","last_undegraded":"2024-03-24T00:40:41.298867+0000","last_fullsized":"2024-03-24T00:40:41.298867+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:04:47.728206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:42.237877+0000","last_change":"2024-03-24T00:40:40.300651+0000","last_active":"2024-03-24T00:40:42.237877+0000","last_peered":"2024-03-24T00:40:42.237877+0000","last_clean":"2024-03-24T00:40:42.237877+0000","last_became_active":"2024-03-24T00:40:40.300376+0000","last_became_peered":"2024-03-24T00:40:40.300376+0000","last_unstale":"2024-03-24T00:40:42.237877+0000","last_undegraded":"2024-03-24T00:40:42.237877+0000","last_fullsized":"2024-03-24T00:40:42.237877+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:27:10.311900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:42.237696+0000","last_change":"2024-03-24T00:40:40.303232+0000","last_active":"2024-03-24T00:40:42.237696+0000","last_peered":"2024-03-24T00:40:42.237696+0000","last_clean":"2024-03-24T00:40:42.237696+0000","last_became_active":"2024-03-24T00:40:40.302929+0000","last_became_peered":"2024-03-24T00:40:40.302929+0000","last_unstale":"2024-03-24T00:40:42.237696+0000","last_undegraded":"2024-03-24T00:40:42.237696+0000","last_fullsized":"2024-03-24T00:40:42.237696+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:05:46.708016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.295395+0000","last_change":"2024-03-24T00:40:40.308988+0000","last_active":"2024-03-24T00:40:41.295395+0000","last_peered":"2024-03-24T00:40:41.295395+0000","last_clean":"2024-03-24T00:40:41.295395+0000","last_became_active":"2024-03-24T00:40:40.308147+0000","last_became_peered":"2024-03-24T00:40:40.308147+0000","last_unstale":"2024-03-24T00:40:41.295395+0000","last_undegraded":"2024-03-24T00:40:41.295395+0000","last_fullsized":"2024-03-24T00:40:41.295395+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:34:06.342935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.300513+0000","last_change":"2024-03-24T00:40:40.298134+0000","last_active":"2024-03-24T00:40:41.300513+0000","last_peered":"2024-03-24T00:40:41.300513+0000","last_clean":"2024-03-24T00:40:41.300513+0000","last_became_active":"2024-03-24T00:40:40.297569+0000","last_became_peered":"2024-03-24T00:40:40.297569+0000","last_unstale":"2024-03-24T00:40:41.300513+0000","last_undegraded":"2024-03-24T00:40:41.300513+0000","last_fullsized":"2024-03-24T00:40:41.300513+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:52:40.714131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.298324+0000","last_change":"2024-03-24T00:40:40.290573+0000","last_active":"2024-03-24T00:40:41.298324+0000","last_peered":"2024-03-24T00:40:41.298324+0000","last_clean":"2024-03-24T00:40:41.298324+0000","last_became_active":"2024-03-24T00:40:40.290317+0000","last_became_peered":"2024-03-24T00:40:40.290317+0000","last_unstale":"2024-03-24T00:40:41.298324+0000","last_undegraded":"2024-03-24T00:40:41.298324+0000","last_fullsized":"2024-03-24T00:40:41.298324+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:11:36.428206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.299193+0000","last_change":"2024-03-24T00:40:40.301695+0000","last_active":"2024-03-24T00:40:41.299193+0000","last_peered":"2024-03-24T00:40:41.299193+0000","last_clean":"2024-03-24T00:40:41.299193+0000","last_became_active":"2024-03-24T00:40:40.301421+0000","last_became_peered":"2024-03-24T00:40:40.301421+0000","last_unstale":"2024-03-24T00:40:41.299193+0000","last_undegraded":"2024-03-24T00:40:41.299193+0000","last_fullsized":"2024-03-24T00:40:41.299193+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:42:06.753248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.295697+0000","last_change":"2024-03-24T00:40:40.307744+0000","last_active":"2024-03-24T00:40:41.295697+0000","last_peered":"2024-03-24T00:40:41.295697+0000","last_clean":"2024-03-24T00:40:41.295697+0000","last_became_active":"2024-03-24T00:40:40.307484+0000","last_became_peered":"2024-03-24T00:40:40.307484+0000","last_unstale":"2024-03-24T00:40:41.295697+0000","last_undegraded":"2024-03-24T00:40:41.295697+0000","last_fullsized":"2024-03-24T00:40:41.295697+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:07:01.023388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.586544+0000","last_change":"2024-03-24T00:40:40.306066+0000","last_active":"2024-03-24T00:40:41.586544+0000","last_peered":"2024-03-24T00:40:41.586544+0000","last_clean":"2024-03-24T00:40:41.586544+0000","last_became_active":"2024-03-24T00:40:40.305795+0000","last_became_peered":"2024-03-24T00:40:40.305795+0000","last_unstale":"2024-03-24T00:40:41.586544+0000","last_undegraded":"2024-03-24T00:40:41.586544+0000","last_fullsized":"2024-03-24T00:40:41.586544+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:16:51.931733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:42.237910+0000","last_change":"2024-03-24T00:40:40.303970+0000","last_active":"2024-03-24T00:40:42.237910+0000","last_peered":"2024-03-24T00:40:42.237910+0000","last_clean":"2024-03-24T00:40:42.237910+0000","last_became_active":"2024-03-24T00:40:40.303686+0000","last_became_peered":"2024-03-24T00:40:40.303686+0000","last_unstale":"2024-03-24T00:40:42.237910+0000","last_undegraded":"2024-03-24T00:40:42.237910+0000","last_fullsized":"2024-03-24T00:40:42.237910+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:55:02.465830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.298467+0000","last_change":"2024-03-24T00:40:40.295547+0000","last_active":"2024-03-24T00:40:41.298467+0000","last_peered":"2024-03-24T00:40:41.298467+0000","last_clean":"2024-03-24T00:40:41.298467+0000","last_became_active":"2024-03-24T00:40:40.294692+0000","last_became_peered":"2024-03-24T00:40:40.294692+0000","last_unstale":"2024-03-24T00:40:41.298467+0000","last_undegraded":"2024-03-24T00:40:41.298467+0000","last_fullsized":"2024-03-24T00:40:41.298467+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:14:05.593326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.302222+0000","last_change":"2024-03-24T00:40:40.304565+0000","last_active":"2024-03-24T00:40:41.302222+0000","last_peered":"2024-03-24T00:40:41.302222+0000","last_clean":"2024-03-24T00:40:41.302222+0000","last_became_active":"2024-03-24T00:40:40.304155+0000","last_became_peered":"2024-03-24T00:40:40.304155+0000","last_unstale":"2024-03-24T00:40:41.302222+0000","last_undegraded":"2024-03-24T00:40:41.302222+0000","last_fullsized":"2024-03-24T00:40:41.302222+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:16:51.931733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.298428+0000","last_change":"2024-03-24T00:40:40.288756+0000","last_active":"2024-03-24T00:40:41.298428+0000","last_peered":"2024-03-24T00:40:41.298428+0000","last_clean":"2024-03-24T00:40:41.298428+0000","last_became_active":"2024-03-24T00:40:40.288395+0000","last_became_peered":"2024-03-24T00:40:40.288395+0000","last_unstale":"2024-03-24T00:40:41.298428+0000","last_undegraded":"2024-03-24T00:40:41.298428+0000","last_fullsized":"2024-03-24T00:40:41.298428+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:14:36.606882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.301101+0000","last_change":"2024-03-24T00:40:40.302381+0000","last_active":"2024-03-24T00:40:41.301101+0000","last_peered":"2024-03-24T00:40:41.301101+0000","last_clean":"2024-03-24T00:40:41.301101+0000","last_became_active":"2024-03-24T00:40:40.302104+0000","last_became_peered":"2024-03-24T00:40:40.302104+0000","last_unstale":"2024-03-24T00:40:41.301101+0000","last_undegraded":"2024-03-24T00:40:41.301101+0000","last_fullsized":"2024-03-24T00:40:41.301101+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:02:02.195271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.301787+0000","last_change":"2024-03-24T00:40:40.297972+0000","last_active":"2024-03-24T00:40:41.301787+0000","last_peered":"2024-03-24T00:40:41.301787+0000","last_clean":"2024-03-24T00:40:41.301787+0000","last_became_active":"2024-03-24T00:40:40.297630+0000","last_became_peered":"2024-03-24T00:40:40.297630+0000","last_unstale":"2024-03-24T00:40:41.301787+0000","last_undegraded":"2024-03-24T00:40:41.301787+0000","last_fullsized":"2024-03-24T00:40:41.301787+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:06:06.434208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"49'4","reported_seq":18,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:42.237862+0000","last_change":"2024-03-24T00:40:39.291631+0000","last_active":"2024-03-24T00:40:42.237862+0000","last_peered":"2024-03-24T00:40:42.237862+0000","last_clean":"2024-03-24T00:40:42.237862+0000","last_became_active":"2024-03-24T00:40:39.291349+0000","last_became_peered":"2024-03-24T00:40:39.291349+0000","last_unstale":"2024-03-24T00:40:42.237862+0000","last_undegraded":"2024-03-24T00:40:42.237862+0000","last_fullsized":"2024-03-24T00:40:42.237862+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:38:21.054863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.301772+0000","last_change":"2024-03-24T00:40:40.294998+0000","last_active":"2024-03-24T00:40:41.301772+0000","last_peered":"2024-03-24T00:40:41.301772+0000","last_clean":"2024-03-24T00:40:41.301772+0000","last_became_active":"2024-03-24T00:40:40.294691+0000","last_became_peered":"2024-03-24T00:40:40.294691+0000","last_unstale":"2024-03-24T00:40:41.301772+0000","last_undegraded":"2024-03-24T00:40:41.301772+0000","last_fullsized":"2024-03-24T00:40:41.301772+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:14:05.593326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"49'1","reported_seq":15,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.586414+0000","last_change":"2024-03-24T00:40:39.295765+0000","last_active":"2024-03-24T00:40:41.586414+0000","last_peered":"2024-03-24T00:40:41.586414+0000","last_clean":"2024-03-24T00:40:41.586414+0000","last_became_active":"2024-03-24T00:40:39.295486+0000","last_became_peered":"2024-03-24T00:40:39.295486+0000","last_unstale":"2024-03-24T00:40:41.586414+0000","last_undegraded":"2024-03-24T00:40:41.586414+0000","last_fullsized":"2024-03-24T00:40:41.586414+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:42:28.507176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:42.237643+0000","last_change":"2024-03-24T00:40:40.301600+0000","last_active":"2024-03-24T00:40:42.237643+0000","last_peered":"2024-03-24T00:40:42.237643+0000","last_clean":"2024-03-24T00:40:42.237643+0000","last_became_active":"2024-03-24T00:40:40.301314+0000","last_became_peered":"2024-03-24T00:40:40.301314+0000","last_unstale":"2024-03-24T00:40:42.237643+0000","last_undegraded":"2024-03-24T00:40:42.237643+0000","last_fullsized":"2024-03-24T00:40:42.237643+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:38:57.796446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"49'2","reported_seq":16,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.586566+0000","last_change":"2024-03-24T00:40:39.294921+0000","last_active":"2024-03-24T00:40:41.586566+0000","last_peered":"2024-03-24T00:40:41.586566+0000","last_clean":"2024-03-24T00:40:41.586566+0000","last_became_active":"2024-03-24T00:40:39.294579+0000","last_became_peered":"2024-03-24T00:40:39.294579+0000","last_unstale":"2024-03-24T00:40:41.586566+0000","last_undegraded":"2024-03-24T00:40:41.586566+0000","last_fullsized":"2024-03-24T00:40:41.586566+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:35.720596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.295490+0000","last_change":"2024-03-24T00:40:40.303366+0000","last_active":"2024-03-24T00:40:41.295490+0000","last_peered":"2024-03-24T00:40:41.295490+0000","last_clean":"2024-03-24T00:40:41.295490+0000","last_became_active":"2024-03-24T00:40:40.303100+0000","last_became_peered":"2024-03-24T00:40:40.303100+0000","last_unstale":"2024-03-24T00:40:41.295490+0000","last_undegraded":"2024-03-24T00:40:41.295490+0000","last_fullsized":"2024-03-24T00:40:41.295490+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:20:37.376789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"49'2","reported_seq":16,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.301845+0000","last_change":"2024-03-24T00:40:39.297839+0000","last_active":"2024-03-24T00:40:41.301845+0000","last_peered":"2024-03-24T00:40:41.301845+0000","last_clean":"2024-03-24T00:40:41.301845+0000","last_became_active":"2024-03-24T00:40:39.297536+0000","last_became_peered":"2024-03-24T00:40:39.297536+0000","last_unstale":"2024-03-24T00:40:41.301845+0000","last_undegraded":"2024-03-24T00:40:41.301845+0000","last_fullsized":"2024-03-24T00:40:41.301845+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:39:52.227506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.845005+0000","last_change":"2024-03-24T00:40:40.298734+0000","last_active":"2024-03-24T00:40:41.845005+0000","last_peered":"2024-03-24T00:40:41.845005+0000","last_clean":"2024-03-24T00:40:41.845005+0000","last_became_active":"2024-03-24T00:40:40.298356+0000","last_became_peered":"2024-03-24T00:40:40.298356+0000","last_unstale":"2024-03-24T00:40:41.845005+0000","last_undegraded":"2024-03-24T00:40:41.845005+0000","last_fullsized":"2024-03-24T00:40:41.845005+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:55:02.465830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"49'1","reported_seq":15,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.298540+0000","last_change":"2024-03-24T00:40:39.304061+0000","last_active":"2024-03-24T00:40:41.298540+0000","last_peered":"2024-03-24T00:40:41.298540+0000","last_clean":"2024-03-24T00:40:41.298540+0000","last_became_active":"2024-03-24T00:40:39.303769+0000","last_became_peered":"2024-03-24T00:40:39.303769+0000","last_unstale":"2024-03-24T00:40:41.298540+0000","last_undegraded":"2024-03-24T00:40:41.298540+0000","last_fullsized":"2024-03-24T00:40:41.298540+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:38:56.788303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.302239+0000","last_change":"2024-03-24T00:40:40.304859+0000","last_active":"2024-03-24T00:40:41.302239+0000","last_peered":"2024-03-24T00:40:41.302239+0000","last_clean":"2024-03-24T00:40:41.302239+0000","last_became_active":"2024-03-24T00:40:40.304576+0000","last_became_peered":"2024-03-24T00:40:40.304576+0000","last_unstale":"2024-03-24T00:40:41.302239+0000","last_undegraded":"2024-03-24T00:40:41.302239+0000","last_fullsized":"2024-03-24T00:40:41.302239+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:48:03.404529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"49'1","reported_seq":15,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.844745+0000","last_change":"2024-03-24T00:40:39.293463+0000","last_active":"2024-03-24T00:40:41.844745+0000","last_peered":"2024-03-24T00:40:41.844745+0000","last_clean":"2024-03-24T00:40:41.844745+0000","last_became_active":"2024-03-24T00:40:39.293196+0000","last_became_peered":"2024-03-24T00:40:39.293196+0000","last_unstale":"2024-03-24T00:40:41.844745+0000","last_undegraded":"2024-03-24T00:40:41.844745+0000","last_fullsized":"2024-03-24T00:40:41.844745+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:35.720596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.295842+0000","last_change":"2024-03-24T00:40:40.309098+0000","last_active":"2024-03-24T00:40:41.295842+0000","last_peered":"2024-03-24T00:40:41.295842+0000","last_clean":"2024-03-24T00:40:41.295842+0000","last_became_active":"2024-03-24T00:40:40.307213+0000","last_became_peered":"2024-03-24T00:40:40.307213+0000","last_unstale":"2024-03-24T00:40:41.295842+0000","last_undegraded":"2024-03-24T00:40:41.295842+0000","last_fullsized":"2024-03-24T00:40:41.295842+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:26:47.125317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:42.237610+0000","last_change":"2024-03-24T00:40:39.294468+0000","last_active":"2024-03-24T00:40:42.237610+0000","last_peered":"2024-03-24T00:40:42.237610+0000","last_clean":"2024-03-24T00:40:42.237610+0000","last_became_active":"2024-03-24T00:40:39.294130+0000","last_became_peered":"2024-03-24T00:40:39.294130+0000","last_unstale":"2024-03-24T00:40:42.237610+0000","last_undegraded":"2024-03-24T00:40:42.237610+0000","last_fullsized":"2024-03-24T00:40:42.237610+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:54:49.331324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.301979+0000","last_change":"2024-03-24T00:40:40.299646+0000","last_active":"2024-03-24T00:40:41.301979+0000","last_peered":"2024-03-24T00:40:41.301979+0000","last_clean":"2024-03-24T00:40:41.301979+0000","last_became_active":"2024-03-24T00:40:40.299374+0000","last_became_peered":"2024-03-24T00:40:40.299374+0000","last_unstale":"2024-03-24T00:40:41.301979+0000","last_undegraded":"2024-03-24T00:40:41.301979+0000","last_fullsized":"2024-03-24T00:40:41.301979+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:57:36.878352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.298326+0000","last_change":"2024-03-24T00:40:39.296539+0000","last_active":"2024-03-24T00:40:41.298326+0000","last_peered":"2024-03-24T00:40:41.298326+0000","last_clean":"2024-03-24T00:40:41.298326+0000","last_became_active":"2024-03-24T00:40:39.296272+0000","last_became_peered":"2024-03-24T00:40:39.296272+0000","last_unstale":"2024-03-24T00:40:41.298326+0000","last_undegraded":"2024-03-24T00:40:41.298326+0000","last_fullsized":"2024-03-24T00:40:41.298326+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:18:21.492781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.844764+0000","last_change":"2024-03-24T00:40:40.307197+0000","last_active":"2024-03-24T00:40:41.844764+0000","last_peered":"2024-03-24T00:40:41.844764+0000","last_clean":"2024-03-24T00:40:41.844764+0000","last_became_active":"2024-03-24T00:40:40.306945+0000","last_became_peered":"2024-03-24T00:40:40.306945+0000","last_unstale":"2024-03-24T00:40:41.844764+0000","last_undegraded":"2024-03-24T00:40:41.844764+0000","last_fullsized":"2024-03-24T00:40:41.844764+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:10:00.938576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.301825+0000","last_change":"2024-03-24T00:40:39.296067+0000","last_active":"2024-03-24T00:40:41.301825+0000","last_peered":"2024-03-24T00:40:41.301825+0000","last_clean":"2024-03-24T00:40:41.301825+0000","last_became_active":"2024-03-24T00:40:39.295773+0000","last_became_peered":"2024-03-24T00:40:39.295773+0000","last_unstale":"2024-03-24T00:40:41.301825+0000","last_undegraded":"2024-03-24T00:40:41.301825+0000","last_fullsized":"2024-03-24T00:40:41.301825+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:38:56.788303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.301213+0000","last_change":"2024-03-24T00:40:40.295248+0000","last_active":"2024-03-24T00:40:41.301213+0000","last_peered":"2024-03-24T00:40:41.301213+0000","last_clean":"2024-03-24T00:40:41.301213+0000","last_became_active":"2024-03-24T00:40:40.294957+0000","last_became_peered":"2024-03-24T00:40:40.294957+0000","last_unstale":"2024-03-24T00:40:41.301213+0000","last_undegraded":"2024-03-24T00:40:41.301213+0000","last_fullsized":"2024-03-24T00:40:41.301213+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:45:21.056006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.844836+0000","last_change":"2024-03-24T00:40:39.296711+0000","last_active":"2024-03-24T00:40:41.844836+0000","last_peered":"2024-03-24T00:40:41.844836+0000","last_clean":"2024-03-24T00:40:41.844836+0000","last_became_active":"2024-03-24T00:40:39.296435+0000","last_became_peered":"2024-03-24T00:40:39.296435+0000","last_unstale":"2024-03-24T00:40:41.844836+0000","last_undegraded":"2024-03-24T00:40:41.844836+0000","last_fullsized":"2024-03-24T00:40:41.844836+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:43:07.435024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.298596+0000","last_change":"2024-03-24T00:40:40.294122+0000","last_active":"2024-03-24T00:40:41.298596+0000","last_peered":"2024-03-24T00:40:41.298596+0000","last_clean":"2024-03-24T00:40:41.298596+0000","last_became_active":"2024-03-24T00:40:40.293863+0000","last_became_peered":"2024-03-24T00:40:40.293863+0000","last_unstale":"2024-03-24T00:40:41.298596+0000","last_undegraded":"2024-03-24T00:40:41.298596+0000","last_fullsized":"2024-03-24T00:40:41.298596+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:44:34.057574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"49'3","reported_seq":17,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.302398+0000","last_change":"2024-03-24T00:40:39.301123+0000","last_active":"2024-03-24T00:40:41.302398+0000","last_peered":"2024-03-24T00:40:41.302398+0000","last_clean":"2024-03-24T00:40:41.302398+0000","last_became_active":"2024-03-24T00:40:39.300851+0000","last_became_peered":"2024-03-24T00:40:39.300851+0000","last_unstale":"2024-03-24T00:40:41.302398+0000","last_undegraded":"2024-03-24T00:40:41.302398+0000","last_fullsized":"2024-03-24T00:40:41.302398+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:56:55.475551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.845014+0000","last_change":"2024-03-24T00:40:40.302333+0000","last_active":"2024-03-24T00:40:41.845014+0000","last_peered":"2024-03-24T00:40:41.845014+0000","last_clean":"2024-03-24T00:40:41.845014+0000","last_became_active":"2024-03-24T00:40:40.301837+0000","last_became_peered":"2024-03-24T00:40:40.301837+0000","last_unstale":"2024-03-24T00:40:41.845014+0000","last_undegraded":"2024-03-24T00:40:41.845014+0000","last_fullsized":"2024-03-24T00:40:41.845014+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:27:10.311900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"49'6","reported_seq":20,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.295703+0000","last_change":"2024-03-24T00:40:39.302117+0000","last_active":"2024-03-24T00:40:41.295703+0000","last_peered":"2024-03-24T00:40:41.295703+0000","last_clean":"2024-03-24T00:40:41.295703+0000","last_became_active":"2024-03-24T00:40:39.301875+0000","last_became_peered":"2024-03-24T00:40:39.301875+0000","last_unstale":"2024-03-24T00:40:41.295703+0000","last_undegraded":"2024-03-24T00:40:41.295703+0000","last_fullsized":"2024-03-24T00:40:41.295703+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:11:35.420062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.301101+0000","last_change":"2024-03-24T00:40:40.296786+0000","last_active":"2024-03-24T00:40:41.301101+0000","last_peered":"2024-03-24T00:40:41.301101+0000","last_clean":"2024-03-24T00:40:41.301101+0000","last_became_active":"2024-03-24T00:40:40.294719+0000","last_became_peered":"2024-03-24T00:40:40.294719+0000","last_unstale":"2024-03-24T00:40:41.301101+0000","last_undegraded":"2024-03-24T00:40:41.301101+0000","last_fullsized":"2024-03-24T00:40:41.301101+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:35:52.681866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"49'2","reported_seq":16,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.298548+0000","last_change":"2024-03-24T00:40:39.298667+0000","last_active":"2024-03-24T00:40:41.298548+0000","last_peered":"2024-03-24T00:40:41.298548+0000","last_clean":"2024-03-24T00:40:41.298548+0000","last_became_active":"2024-03-24T00:40:39.298067+0000","last_became_peered":"2024-03-24T00:40:39.298067+0000","last_unstale":"2024-03-24T00:40:41.298548+0000","last_undegraded":"2024-03-24T00:40:41.298548+0000","last_fullsized":"2024-03-24T00:40:41.298548+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:09:59.930432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.299005+0000","last_change":"2024-03-24T00:40:40.303321+0000","last_active":"2024-03-24T00:40:41.299005+0000","last_peered":"2024-03-24T00:40:41.299005+0000","last_clean":"2024-03-24T00:40:41.299005+0000","last_became_active":"2024-03-24T00:40:40.303025+0000","last_became_peered":"2024-03-24T00:40:40.303025+0000","last_unstale":"2024-03-24T00:40:41.299005+0000","last_undegraded":"2024-03-24T00:40:41.299005+0000","last_fullsized":"2024-03-24T00:40:41.299005+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:03:06.246605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:42.237758+0000","last_change":"2024-03-24T00:40:39.299651+0000","last_active":"2024-03-24T00:40:42.237758+0000","last_peered":"2024-03-24T00:40:42.237758+0000","last_clean":"2024-03-24T00:40:42.237758+0000","last_became_active":"2024-03-24T00:40:39.299388+0000","last_became_peered":"2024-03-24T00:40:39.299388+0000","last_unstale":"2024-03-24T00:40:42.237758+0000","last_undegraded":"2024-03-24T00:40:42.237758+0000","last_fullsized":"2024-03-24T00:40:42.237758+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:43:07.435024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.295782+0000","last_change":"2024-03-24T00:40:40.304133+0000","last_active":"2024-03-24T00:40:41.295782+0000","last_peered":"2024-03-24T00:40:41.295782+0000","last_clean":"2024-03-24T00:40:41.295782+0000","last_became_active":"2024-03-24T00:40:40.303887+0000","last_became_peered":"2024-03-24T00:40:40.303887+0000","last_unstale":"2024-03-24T00:40:41.295782+0000","last_undegraded":"2024-03-24T00:40:41.295782+0000","last_fullsized":"2024-03-24T00:40:41.295782+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:40:46.140802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.301877+0000","last_change":"2024-03-24T00:40:39.295906+0000","last_active":"2024-03-24T00:40:41.301877+0000","last_peered":"2024-03-24T00:40:41.301877+0000","last_clean":"2024-03-24T00:40:41.301877+0000","last_became_active":"2024-03-24T00:40:39.295604+0000","last_became_peered":"2024-03-24T00:40:39.295604+0000","last_unstale":"2024-03-24T00:40:41.301877+0000","last_undegraded":"2024-03-24T00:40:41.301877+0000","last_fullsized":"2024-03-24T00:40:41.301877+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:18:21.492781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.299050+0000","last_change":"2024-03-24T00:40:40.291927+0000","last_active":"2024-03-24T00:40:41.299050+0000","last_peered":"2024-03-24T00:40:41.299050+0000","last_clean":"2024-03-24T00:40:41.299050+0000","last_became_active":"2024-03-24T00:40:40.291654+0000","last_became_peered":"2024-03-24T00:40:40.291654+0000","last_unstale":"2024-03-24T00:40:41.299050+0000","last_undegraded":"2024-03-24T00:40:41.299050+0000","last_fullsized":"2024-03-24T00:40:41.299050+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:27:08.995457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"49'8","reported_seq":22,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.295528+0000","last_change":"2024-03-24T00:40:39.298619+0000","last_active":"2024-03-24T00:40:41.295528+0000","last_peered":"2024-03-24T00:40:41.295528+0000","last_clean":"2024-03-24T00:40:41.295528+0000","last_became_active":"2024-03-24T00:40:39.298350+0000","last_became_peered":"2024-03-24T00:40:39.298350+0000","last_unstale":"2024-03-24T00:40:41.295528+0000","last_undegraded":"2024-03-24T00:40:41.295528+0000","last_fullsized":"2024-03-24T00:40:41.295528+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:45:20.047863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.295420+0000","last_change":"2024-03-24T00:40:40.299967+0000","last_active":"2024-03-24T00:40:41.295420+0000","last_peered":"2024-03-24T00:40:41.295420+0000","last_clean":"2024-03-24T00:40:41.295420+0000","last_became_active":"2024-03-24T00:40:40.298988+0000","last_became_peered":"2024-03-24T00:40:40.298988+0000","last_unstale":"2024-03-24T00:40:41.295420+0000","last_undegraded":"2024-03-24T00:40:41.295420+0000","last_fullsized":"2024-03-24T00:40:41.295420+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:27:08.995457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"49'1","reported_seq":15,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.299160+0000","last_change":"2024-03-24T00:40:39.299640+0000","last_active":"2024-03-24T00:40:41.299160+0000","last_peered":"2024-03-24T00:40:41.299160+0000","last_clean":"2024-03-24T00:40:41.299160+0000","last_became_active":"2024-03-24T00:40:39.299375+0000","last_became_peered":"2024-03-24T00:40:39.299375+0000","last_unstale":"2024-03-24T00:40:41.299160+0000","last_undegraded":"2024-03-24T00:40:41.299160+0000","last_fullsized":"2024-03-24T00:40:41.299160+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:27:09.303757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.586176+0000","last_change":"2024-03-24T00:40:40.299147+0000","last_active":"2024-03-24T00:40:41.586176+0000","last_peered":"2024-03-24T00:40:41.586176+0000","last_clean":"2024-03-24T00:40:41.586176+0000","last_became_active":"2024-03-24T00:40:40.298855+0000","last_became_peered":"2024-03-24T00:40:40.298855+0000","last_unstale":"2024-03-24T00:40:41.586176+0000","last_undegraded":"2024-03-24T00:40:41.586176+0000","last_fullsized":"2024-03-24T00:40:41.586176+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:58:22.169987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"49'2","reported_seq":16,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.300861+0000","last_change":"2024-03-24T00:40:39.302072+0000","last_active":"2024-03-24T00:40:41.300861+0000","last_peered":"2024-03-24T00:40:41.300861+0000","last_clean":"2024-03-24T00:40:41.300861+0000","last_became_active":"2024-03-24T00:40:39.301790+0000","last_became_peered":"2024-03-24T00:40:39.301790+0000","last_unstale":"2024-03-24T00:40:41.300861+0000","last_undegraded":"2024-03-24T00:40:41.300861+0000","last_fullsized":"2024-03-24T00:40:41.300861+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:24:11.659170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.844868+0000","last_change":"2024-03-24T00:40:40.299943+0000","last_active":"2024-03-24T00:40:41.844868+0000","last_peered":"2024-03-24T00:40:41.844868+0000","last_clean":"2024-03-24T00:40:41.844868+0000","last_became_active":"2024-03-24T00:40:40.299622+0000","last_became_peered":"2024-03-24T00:40:40.299622+0000","last_unstale":"2024-03-24T00:40:41.844868+0000","last_undegraded":"2024-03-24T00:40:41.844868+0000","last_fullsized":"2024-03-24T00:40:41.844868+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:05:46.708016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"49'1","reported_seq":15,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:42.237865+0000","last_change":"2024-03-24T00:40:39.292046+0000","last_active":"2024-03-24T00:40:42.237865+0000","last_peered":"2024-03-24T00:40:42.237865+0000","last_clean":"2024-03-24T00:40:42.237865+0000","last_became_active":"2024-03-24T00:40:39.291744+0000","last_became_peered":"2024-03-24T00:40:39.291744+0000","last_unstale":"2024-03-24T00:40:42.237865+0000","last_undegraded":"2024-03-24T00:40:42.237865+0000","last_fullsized":"2024-03-24T00:40:42.237865+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:09:53.820800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.586355+0000","last_change":"2024-03-24T00:40:40.304100+0000","last_active":"2024-03-24T00:40:41.586355+0000","last_peered":"2024-03-24T00:40:41.586355+0000","last_clean":"2024-03-24T00:40:41.586355+0000","last_became_active":"2024-03-24T00:40:40.303792+0000","last_became_peered":"2024-03-24T00:40:40.303792+0000","last_unstale":"2024-03-24T00:40:41.586355+0000","last_undegraded":"2024-03-24T00:40:41.586355+0000","last_fullsized":"2024-03-24T00:40:41.586355+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:00:25.379976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:42.238002+0000","last_change":"2024-03-24T00:40:39.304591+0000","last_active":"2024-03-24T00:40:42.238002+0000","last_peered":"2024-03-24T00:40:42.238002+0000","last_clean":"2024-03-24T00:40:42.238002+0000","last_became_active":"2024-03-24T00:40:39.304330+0000","last_became_peered":"2024-03-24T00:40:39.304330+0000","last_unstale":"2024-03-24T00:40:42.238002+0000","last_undegraded":"2024-03-24T00:40:42.238002+0000","last_fullsized":"2024-03-24T00:40:42.238002+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:42:28.507176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.301287+0000","last_change":"2024-03-24T00:40:40.295216+0000","last_active":"2024-03-24T00:40:41.301287+0000","last_peered":"2024-03-24T00:40:41.301287+0000","last_clean":"2024-03-24T00:40:41.301287+0000","last_became_active":"2024-03-24T00:40:40.294902+0000","last_became_peered":"2024-03-24T00:40:40.294902+0000","last_unstale":"2024-03-24T00:40:41.301287+0000","last_undegraded":"2024-03-24T00:40:41.301287+0000","last_fullsized":"2024-03-24T00:40:41.301287+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:11:36.428206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"49'2","reported_seq":16,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.586447+0000","last_change":"2024-03-24T00:40:39.287554+0000","last_active":"2024-03-24T00:40:41.586447+0000","last_peered":"2024-03-24T00:40:41.586447+0000","last_clean":"2024-03-24T00:40:41.586447+0000","last_became_active":"2024-03-24T00:40:39.287273+0000","last_became_peered":"2024-03-24T00:40:39.287273+0000","last_unstale":"2024-03-24T00:40:41.586447+0000","last_undegraded":"2024-03-24T00:40:41.586447+0000","last_fullsized":"2024-03-24T00:40:41.586447+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:43:07.435024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:42.237855+0000","last_change":"2024-03-24T00:40:40.300618+0000","last_active":"2024-03-24T00:40:42.237855+0000","last_peered":"2024-03-24T00:40:42.237855+0000","last_clean":"2024-03-24T00:40:42.237855+0000","last_became_active":"2024-03-24T00:40:40.300350+0000","last_became_peered":"2024-03-24T00:40:40.300350+0000","last_unstale":"2024-03-24T00:40:42.237855+0000","last_undegraded":"2024-03-24T00:40:42.237855+0000","last_fullsized":"2024-03-24T00:40:42.237855+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:19:39.784710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.295352+0000","last_change":"2024-03-24T00:40:39.298506+0000","last_active":"2024-03-24T00:40:41.295352+0000","last_peered":"2024-03-24T00:40:41.295352+0000","last_clean":"2024-03-24T00:40:41.295352+0000","last_became_active":"2024-03-24T00:40:39.297935+0000","last_became_peered":"2024-03-24T00:40:39.297935+0000","last_unstale":"2024-03-24T00:40:41.295352+0000","last_undegraded":"2024-03-24T00:40:41.295352+0000","last_fullsized":"2024-03-24T00:40:41.295352+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:35:51.673723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.298904+0000","last_change":"2024-03-24T00:40:40.294916+0000","last_active":"2024-03-24T00:40:41.298904+0000","last_peered":"2024-03-24T00:40:41.298904+0000","last_clean":"2024-03-24T00:40:41.298904+0000","last_became_active":"2024-03-24T00:40:40.294654+0000","last_became_peered":"2024-03-24T00:40:40.294654+0000","last_unstale":"2024-03-24T00:40:41.298904+0000","last_undegraded":"2024-03-24T00:40:41.298904+0000","last_fullsized":"2024-03-24T00:40:41.298904+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:04:44.682407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.298476+0000","last_change":"2024-03-24T00:40:39.296787+0000","last_active":"2024-03-24T00:40:41.298476+0000","last_peered":"2024-03-24T00:40:41.298476+0000","last_clean":"2024-03-24T00:40:41.298476+0000","last_became_active":"2024-03-24T00:40:39.296501+0000","last_became_peered":"2024-03-24T00:40:39.296501+0000","last_unstale":"2024-03-24T00:40:41.298476+0000","last_undegraded":"2024-03-24T00:40:41.298476+0000","last_fullsized":"2024-03-24T00:40:41.298476+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:56:55.475551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.586528+0000","last_change":"2024-03-24T00:40:40.303603+0000","last_active":"2024-03-24T00:40:41.586528+0000","last_peered":"2024-03-24T00:40:41.586528+0000","last_clean":"2024-03-24T00:40:41.586528+0000","last_became_active":"2024-03-24T00:40:40.303280+0000","last_became_peered":"2024-03-24T00:40:40.303280+0000","last_unstale":"2024-03-24T00:40:41.586528+0000","last_undegraded":"2024-03-24T00:40:41.586528+0000","last_fullsized":"2024-03-24T00:40:41.586528+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:48:03.404529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.300429+0000","last_change":"2024-03-24T00:40:39.300361+0000","last_active":"2024-03-24T00:40:41.300429+0000","last_peered":"2024-03-24T00:40:41.300429+0000","last_clean":"2024-03-24T00:40:41.300429+0000","last_became_active":"2024-03-24T00:40:39.299954+0000","last_became_peered":"2024-03-24T00:40:39.299954+0000","last_unstale":"2024-03-24T00:40:41.300429+0000","last_undegraded":"2024-03-24T00:40:41.300429+0000","last_fullsized":"2024-03-24T00:40:41.300429+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:45:03.385577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.301399+0000","last_change":"2024-03-24T00:40:40.301450+0000","last_active":"2024-03-24T00:40:41.301399+0000","last_peered":"2024-03-24T00:40:41.301399+0000","last_clean":"2024-03-24T00:40:41.301399+0000","last_became_active":"2024-03-24T00:40:40.301165+0000","last_became_peered":"2024-03-24T00:40:40.301165+0000","last_unstale":"2024-03-24T00:40:41.301399+0000","last_undegraded":"2024-03-24T00:40:41.301399+0000","last_fullsized":"2024-03-24T00:40:41.301399+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:16:13.256691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"49'2","reported_seq":16,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:42.237900+0000","last_change":"2024-03-24T00:40:39.299797+0000","last_active":"2024-03-24T00:40:42.237900+0000","last_peered":"2024-03-24T00:40:42.237900+0000","last_clean":"2024-03-24T00:40:42.237900+0000","last_became_active":"2024-03-24T00:40:39.299456+0000","last_became_peered":"2024-03-24T00:40:39.299456+0000","last_unstale":"2024-03-24T00:40:42.237900+0000","last_undegraded":"2024-03-24T00:40:42.237900+0000","last_fullsized":"2024-03-24T00:40:42.237900+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:16:22.016263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.586362+0000","last_change":"2024-03-24T00:40:40.308806+0000","last_active":"2024-03-24T00:40:41.586362+0000","last_peered":"2024-03-24T00:40:41.586362+0000","last_clean":"2024-03-24T00:40:41.586362+0000","last_became_active":"2024-03-24T00:40:40.308549+0000","last_became_peered":"2024-03-24T00:40:40.308549+0000","last_unstale":"2024-03-24T00:40:41.586362+0000","last_undegraded":"2024-03-24T00:40:41.586362+0000","last_fullsized":"2024-03-24T00:40:41.586362+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:06:06.434208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.302425+0000","last_change":"2024-03-24T00:40:39.299014+0000","last_active":"2024-03-24T00:40:41.302425+0000","last_peered":"2024-03-24T00:40:41.302425+0000","last_clean":"2024-03-24T00:40:41.302425+0000","last_became_active":"2024-03-24T00:40:39.298750+0000","last_became_peered":"2024-03-24T00:40:39.298750+0000","last_unstale":"2024-03-24T00:40:41.302425+0000","last_undegraded":"2024-03-24T00:40:41.302425+0000","last_fullsized":"2024-03-24T00:40:41.302425+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:09:59.930432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.295520+0000","last_change":"2024-03-24T00:40:40.307656+0000","last_active":"2024-03-24T00:40:41.295520+0000","last_peered":"2024-03-24T00:40:41.295520+0000","last_clean":"2024-03-24T00:40:41.295520+0000","last_became_active":"2024-03-24T00:40:40.307394+0000","last_became_peered":"2024-03-24T00:40:40.307394+0000","last_unstale":"2024-03-24T00:40:41.295520+0000","last_undegraded":"2024-03-24T00:40:41.295520+0000","last_fullsized":"2024-03-24T00:40:41.295520+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:41:32.183220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.298715+0000","last_change":"2024-03-24T00:40:39.303224+0000","last_active":"2024-03-24T00:40:41.298715+0000","last_peered":"2024-03-24T00:40:41.298715+0000","last_clean":"2024-03-24T00:40:41.298715+0000","last_became_active":"2024-03-24T00:40:39.302950+0000","last_became_peered":"2024-03-24T00:40:39.302950+0000","last_unstale":"2024-03-24T00:40:41.298715+0000","last_undegraded":"2024-03-24T00:40:41.298715+0000","last_fullsized":"2024-03-24T00:40:41.298715+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:05:45.699873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.298735+0000","last_change":"2024-03-24T00:40:40.305492+0000","last_active":"2024-03-24T00:40:41.298735+0000","last_peered":"2024-03-24T00:40:41.298735+0000","last_clean":"2024-03-24T00:40:41.298735+0000","last_became_active":"2024-03-24T00:40:40.305159+0000","last_became_peered":"2024-03-24T00:40:40.305159+0000","last_unstale":"2024-03-24T00:40:41.298735+0000","last_undegraded":"2024-03-24T00:40:41.298735+0000","last_fullsized":"2024-03-24T00:40:41.298735+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:27:01.459529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.295589+0000","last_change":"2024-03-24T00:40:39.299484+0000","last_active":"2024-03-24T00:40:41.295589+0000","last_peered":"2024-03-24T00:40:41.295589+0000","last_clean":"2024-03-24T00:40:41.295589+0000","last_became_active":"2024-03-24T00:40:39.299184+0000","last_became_peered":"2024-03-24T00:40:39.299184+0000","last_unstale":"2024-03-24T00:40:41.295589+0000","last_undegraded":"2024-03-24T00:40:41.295589+0000","last_fullsized":"2024-03-24T00:40:41.295589+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:47:37.262904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.301497+0000","last_change":"2024-03-24T00:40:40.295093+0000","last_active":"2024-03-24T00:40:41.301497+0000","last_peered":"2024-03-24T00:40:41.301497+0000","last_clean":"2024-03-24T00:40:41.301497+0000","last_became_active":"2024-03-24T00:40:40.294745+0000","last_became_peered":"2024-03-24T00:40:40.294745+0000","last_unstale":"2024-03-24T00:40:41.301497+0000","last_undegraded":"2024-03-24T00:40:41.301497+0000","last_fullsized":"2024-03-24T00:40:41.301497+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:14:36.606882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.295453+0000","last_change":"2024-03-24T00:40:39.298278+0000","last_active":"2024-03-24T00:40:41.295453+0000","last_peered":"2024-03-24T00:40:41.295453+0000","last_clean":"2024-03-24T00:40:41.295453+0000","last_became_active":"2024-03-24T00:40:39.297588+0000","last_became_peered":"2024-03-24T00:40:39.297588+0000","last_unstale":"2024-03-24T00:40:41.295453+0000","last_undegraded":"2024-03-24T00:40:41.295453+0000","last_fullsized":"2024-03-24T00:40:41.295453+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:02:01.187128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.298904+0000","last_change":"2024-03-24T00:40:40.290083+0000","last_active":"2024-03-24T00:40:41.298904+0000","last_peered":"2024-03-24T00:40:41.298904+0000","last_clean":"2024-03-24T00:40:41.298904+0000","last_became_active":"2024-03-24T00:40:40.288984+0000","last_became_peered":"2024-03-24T00:40:40.288984+0000","last_unstale":"2024-03-24T00:40:41.298904+0000","last_undegraded":"2024-03-24T00:40:41.298904+0000","last_fullsized":"2024-03-24T00:40:41.298904+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:02:01.565488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.844630+0000","last_change":"2024-03-24T00:40:39.300219+0000","last_active":"2024-03-24T00:40:41.844630+0000","last_peered":"2024-03-24T00:40:41.844630+0000","last_clean":"2024-03-24T00:40:41.844630+0000","last_became_active":"2024-03-24T00:40:39.299897+0000","last_became_peered":"2024-03-24T00:40:39.299897+0000","last_unstale":"2024-03-24T00:40:41.844630+0000","last_undegraded":"2024-03-24T00:40:41.844630+0000","last_fullsized":"2024-03-24T00:40:41.844630+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:42:28.507176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.295880+0000","last_change":"2024-03-24T00:40:40.302135+0000","last_active":"2024-03-24T00:40:41.295880+0000","last_peered":"2024-03-24T00:40:41.295880+0000","last_clean":"2024-03-24T00:40:41.295880+0000","last_became_active":"2024-03-24T00:40:40.301876+0000","last_became_peered":"2024-03-24T00:40:40.301876+0000","last_unstale":"2024-03-24T00:40:41.295880+0000","last_undegraded":"2024-03-24T00:40:41.295880+0000","last_fullsized":"2024-03-24T00:40:41.295880+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:03:47.424870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.301265+0000","last_change":"2024-03-24T00:40:39.294249+0000","last_active":"2024-03-24T00:40:41.301265+0000","last_peered":"2024-03-24T00:40:41.301265+0000","last_clean":"2024-03-24T00:40:41.301265+0000","last_became_active":"2024-03-24T00:40:39.293915+0000","last_became_peered":"2024-03-24T00:40:39.293915+0000","last_unstale":"2024-03-24T00:40:41.301265+0000","last_undegraded":"2024-03-24T00:40:41.301265+0000","last_fullsized":"2024-03-24T00:40:41.301265+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:00:24.371833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:42.237678+0000","last_change":"2024-03-24T00:40:40.296067+0000","last_active":"2024-03-24T00:40:42.237678+0000","last_peered":"2024-03-24T00:40:42.237678+0000","last_clean":"2024-03-24T00:40:42.237678+0000","last_became_active":"2024-03-24T00:40:40.295784+0000","last_became_peered":"2024-03-24T00:40:40.295784+0000","last_unstale":"2024-03-24T00:40:42.237678+0000","last_undegraded":"2024-03-24T00:40:42.237678+0000","last_fullsized":"2024-03-24T00:40:42.237678+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:10:00.938576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"50'1","reported_seq":15,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.299018+0000","last_change":"2024-03-24T00:40:39.297685+0000","last_active":"2024-03-24T00:40:41.299018+0000","last_peered":"2024-03-24T00:40:41.299018+0000","last_clean":"2024-03-24T00:40:41.299018+0000","last_became_active":"2024-03-24T00:40:39.297097+0000","last_became_peered":"2024-03-24T00:40:39.297097+0000","last_unstale":"2024-03-24T00:40:41.299018+0000","last_undegraded":"2024-03-24T00:40:41.299018+0000","last_fullsized":"2024-03-24T00:40:41.299018+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:19:38.776567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":225,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.301938+0000","last_change":"2024-03-24T00:38:40.953871+0000","last_active":"2024-03-24T00:40:41.301938+0000","last_peered":"2024-03-24T00:40:41.301938+0000","last_clean":"2024-03-24T00:40:41.301938+0000","last_became_active":"2024-03-24T00:38:40.952926+0000","last_became_peered":"2024-03-24T00:38:40.952926+0000","last_unstale":"2024-03-24T00:40:41.301938+0000","last_undegraded":"2024-03-24T00:40:41.301938+0000","last_fullsized":"2024-03-24T00:40:41.301938+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_clean_scrub_stamp":"2024-03-24T00:38:39.722194+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:05.397257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574878,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28192,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343648,"statfs":{"total":96636764160,"available":96607895552,"internally_reserved":0,"allocated":1196032,"data_stored":855330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574877,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27612,"kb_used_data":588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344228,"statfs":{"total":96636764160,"available":96608489472,"internally_reserved":0,"allocated":602112,"data_stored":265490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574877,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27604,"kb_used_data":580,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344236,"statfs":{"total":96636764160,"available":96608497664,"internally_reserved":0,"allocated":593920,"data_stored":264838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574876,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27608,"kb_used_data":584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344232,"statfs":{"total":96636764160,"available":96608493568,"internally_reserved":0,"allocated":598016,"data_stored":265420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574876,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27620,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":610304,"data_stored":266472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574877,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":606208,"data_stored":266450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574877,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27612,"kb_used_data":588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344228,"statfs":{"total":96636764160,"available":96608489472,"internally_reserved":0,"allocated":602112,"data_stored":264998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574876,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28188,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343652,"statfs":{"total":96636764160,"available":96607899648,"internally_reserved":0,"allocated":1196032,"data_stored":855262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:40:46.070 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:40:46.072 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:40:46.272 INFO:tasks.ceph.mds.a.smithi067.stderr:2024-03-24T00:40:46.272+0000 7fbe5e6b5640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T00:40:48.901 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:40:49.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.025+0000 7ffacb819640 1 Processor -- start 2024-03-24T00:40:49.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.025+0000 7ffacb819640 1 -- start start 2024-03-24T00:40:49.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.025+0000 7ffacb819640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ffac40b0de0 0x7ffac415cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:49.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.026+0000 7ffacb819640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ffac406b040 0x7ffac406b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:49.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.026+0000 7ffacb819640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac415d330 con 0x7ffac40cc700 2024-03-24T00:40:49.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.026+0000 7ffacb819640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffac40ce400 con 0x7ffac40b0de0 2024-03-24T00:40:49.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.026+0000 7ffacb819640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffac40b1a10 con 0x7ffac406b040 2024-03-24T00:40:49.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.026+0000 7ffac95b5640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ffac40b0de0 0x7ffac415cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:49.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.026+0000 7ffac8db4640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ffac40cc700 0x7ffac40cdcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39648/0 (socket says 172.21.15.67:39648) 2024-03-24T00:40:49.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.026+0000 7ffac8db4640 1 -- 172.21.15.67:0/3095096479 learned_addr learned my addr 172.21.15.67:0/3095096479 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:40:49.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.026+0000 7ffac9db6640 1 --2- 172.21.15.67:0/3095096479 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ffac406b040 0x7ffac406b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:49.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.026+0000 7ffac95b5640 1 -- 172.21.15.67:0/3095096479 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ffac406b040 msgr2=0x7ffac406b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:49.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.026+0000 7ffac95b5640 1 --2- 172.21.15.67:0/3095096479 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ffac406b040 0x7ffac406b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:49.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.026+0000 7ffac95b5640 1 -- 172.21.15.67:0/3095096479 >> v1:172.21.15.67:6789/0 conn(0x7ffac40cc700 legacy=0x7ffac40cdcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:49.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.026+0000 7ffab7fff640 1 -- 172.21.15.67:0/3095096479 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 202277697 0 0) 0x7ffac0001110 con 0x7ffac40cc700 2024-03-24T00:40:49.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.026+0000 7ffac95b5640 1 -- 172.21.15.67:0/3095096479 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac40ccc50 con 0x7ffac40b0de0 2024-03-24T00:40:49.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.027+0000 7ffac95b5640 1 --2- 172.21.15.67:0/3095096479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffac40b0de0 0x7ffac415cdf0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7ffab80097a0 tx=0x7ffab802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5fecc927e59e850 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:49.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.027+0000 7ffab7fff640 1 -- 172.21.15.67:0/3095096479 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab8035020 con 0x7ffac40b0de0 2024-03-24T00:40:49.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.027+0000 7ffab7fff640 1 -- 172.21.15.67:0/3095096479 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffab8031830 con 0x7ffac40b0de0 2024-03-24T00:40:49.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.027+0000 7ffab7fff640 1 -- 172.21.15.67:0/3095096479 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab8031c00 con 0x7ffac40b0de0 2024-03-24T00:40:49.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.028+0000 7ffacb819640 1 -- 172.21.15.67:0/3095096479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffac40b0de0 msgr2=0x7ffac415cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:49.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.028+0000 7ffacb819640 1 --2- 172.21.15.67:0/3095096479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffac40b0de0 0x7ffac415cdf0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7ffab80097a0 tx=0x7ffab802b8d0 comp rx=0 tx=0).stop 2024-03-24T00:40:49.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.028+0000 7ffacb819640 1 -- 172.21.15.67:0/3095096479 shutdown_connections 2024-03-24T00:40:49.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.028+0000 7ffacb819640 1 --2- 172.21.15.67:0/3095096479 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ffac406b040 0x7ffac406b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:49.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.028+0000 7ffacb819640 1 --2- 172.21.15.67:0/3095096479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffac40b0de0 0x7ffac415cdf0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:49.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.028+0000 7ffacb819640 1 -- 172.21.15.67:0/3095096479 >> 172.21.15.67:0/3095096479 conn(0x7ffac4099a30 msgr2=0x7ffac4099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:49.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.028+0000 7ffacb819640 1 -- 172.21.15.67:0/3095096479 shutdown_connections 2024-03-24T00:40:49.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.028+0000 7ffacb819640 1 -- 172.21.15.67:0/3095096479 wait complete. 2024-03-24T00:40:49.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.029+0000 7ffacb819640 1 Processor -- start 2024-03-24T00:40:49.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.029+0000 7ffacb819640 1 -- start start 2024-03-24T00:40:49.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.030+0000 7ffacb819640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffac406b040 0x7ffac41f4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:49.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.030+0000 7ffacb819640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ffac40b0de0 0x7ffac41f1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:49.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.030+0000 7ffacb819640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffac40cc700 0x7ffac409c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:49.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.030+0000 7ffacb819640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac40cd4f0 con 0x7ffac40cc700 2024-03-24T00:40:49.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.030+0000 7ffacb819640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac40cd210 con 0x7ffac406b040 2024-03-24T00:40:49.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.030+0000 7ffacb819640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffac40cd7d0 con 0x7ffac40b0de0 2024-03-24T00:40:49.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.030+0000 7ffac9db6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffac40cc700 0x7ffac409c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:49.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.030+0000 7ffac9db6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffac40cc700 0x7ffac409c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40852/0 (socket says 172.21.15.67:40852) 2024-03-24T00:40:49.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.030+0000 7ffac9db6640 1 -- 172.21.15.67:0/1676159125 learned_addr learned my addr 172.21.15.67:0/1676159125 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:49.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.030+0000 7ffac95b5640 1 --2- 172.21.15.67:0/1676159125 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffac406b040 0x7ffac41f4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:49.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.030+0000 7ffac8db4640 1 --2- 172.21.15.67:0/1676159125 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ffac40b0de0 0x7ffac41f1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:49.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.030+0000 7ffac9db6640 1 -- 172.21.15.67:0/1676159125 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ffac40b0de0 msgr2=0x7ffac41f1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:49.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.030+0000 7ffac9db6640 1 --2- 172.21.15.67:0/1676159125 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ffac40b0de0 0x7ffac41f1e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:49.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.030+0000 7ffac9db6640 1 -- 172.21.15.67:0/1676159125 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffac406b040 msgr2=0x7ffac41f4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:49.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.030+0000 7ffac9db6640 1 --2- 172.21.15.67:0/1676159125 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffac406b040 0x7ffac41f4df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:49.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.030+0000 7ffac9db6640 1 -- 172.21.15.67:0/1676159125 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac409cf70 con 0x7ffac40cc700 2024-03-24T00:40:49.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.030+0000 7ffac95b5640 1 --2- 172.21.15.67:0/1676159125 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffac406b040 0x7ffac41f4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:40:49.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.031+0000 7ffac9db6640 1 --2- 172.21.15.67:0/1676159125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffac40cc700 0x7ffac409c690 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7ffabc007e60 tx=0x7ffabc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:49.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.031+0000 7ffab5ffb640 1 -- 172.21.15.67:0/1676159125 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffabc014420 con 0x7ffac40cc700 2024-03-24T00:40:49.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.031+0000 7ffab5ffb640 1 -- 172.21.15.67:0/1676159125 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffabc019070 con 0x7ffac40cc700 2024-03-24T00:40:49.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.031+0000 7ffab5ffb640 1 -- 172.21.15.67:0/1676159125 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffabc014820 con 0x7ffac40cc700 2024-03-24T00:40:49.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.031+0000 7ffacb819640 1 -- 172.21.15.67:0/1676159125 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffac40c8d30 con 0x7ffac40cc700 2024-03-24T00:40:49.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.031+0000 7ffacb819640 1 -- 172.21.15.67:0/1676159125 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffac40c92f0 con 0x7ffac40cc700 2024-03-24T00:40:49.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.032+0000 7ffacb819640 1 -- 172.21.15.67:0/1676159125 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffac40b1770 con 0x7ffac40cc700 2024-03-24T00:40:49.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.033+0000 7ffab5ffb640 1 -- 172.21.15.67:0/1676159125 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ffabc0149c0 con 0x7ffac40cc700 2024-03-24T00:40:49.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.033+0000 7ffab5ffb640 1 --2- 172.21.15.67:0/1676159125 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ffaa806bbd0 0x7ffaa806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:49.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.033+0000 7ffab5ffb640 1 -- 172.21.15.67:0/1676159125 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ffabc08b400 con 0x7ffac40cc700 2024-03-24T00:40:49.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.033+0000 7ffac95b5640 1 --2- 172.21.15.67:0/1676159125 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ffaa806bbd0 0x7ffaa806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:49.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.034+0000 7ffac95b5640 1 --2- 172.21.15.67:0/1676159125 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ffaa806bbd0 0x7ffaa806e090 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ffab8036000 tx=0x7ffab8008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:49.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.037+0000 7ffab5ffb640 1 -- 172.21.15.67:0/1676159125 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ffac40b1770 con 0x7ffac40cc700 2024-03-24T00:40:49.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.261+0000 7ffacb819640 1 -- 172.21.15.67:0/1676159125 --> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ffac404f400 con 0x7ffaa806bbd0 2024-03-24T00:40:49.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.267+0000 7ffab5ffb640 1 -- 172.21.15.67:0/1676159125 <== mgr.4105 v2:172.21.15.72:6832/2768292274 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251957 (secure 0 0 0) 0x7ffac404f400 con 0x7ffaa806bbd0 2024-03-24T00:40:49.270 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:40:49.271 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-24T00:40:49.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.271+0000 7ffacb819640 1 -- 172.21.15.67:0/1676159125 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ffaa806bbd0 msgr2=0x7ffaa806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:49.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.271+0000 7ffacb819640 1 --2- 172.21.15.67:0/1676159125 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ffaa806bbd0 0x7ffaa806e090 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ffab8036000 tx=0x7ffab8008030 comp rx=0 tx=0).stop 2024-03-24T00:40:49.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.271+0000 7ffacb819640 1 -- 172.21.15.67:0/1676159125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffac40cc700 msgr2=0x7ffac409c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:49.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.271+0000 7ffacb819640 1 --2- 172.21.15.67:0/1676159125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffac40cc700 0x7ffac409c690 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7ffabc007e60 tx=0x7ffabc00e4b0 comp rx=0 tx=0).stop 2024-03-24T00:40:49.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.271+0000 7ffacb819640 1 -- 172.21.15.67:0/1676159125 shutdown_connections 2024-03-24T00:40:49.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.271+0000 7ffacb819640 1 --2- 172.21.15.67:0/1676159125 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ffaa806bbd0 0x7ffaa806e090 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:49.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.271+0000 7ffacb819640 1 --2- 172.21.15.67:0/1676159125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffac40cc700 0x7ffac409c690 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:49.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.271+0000 7ffacb819640 1 --2- 172.21.15.67:0/1676159125 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ffac40b0de0 0x7ffac41f1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:49.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.271+0000 7ffacb819640 1 --2- 172.21.15.67:0/1676159125 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffac406b040 0x7ffac41f4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:49.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.271+0000 7ffacb819640 1 -- 172.21.15.67:0/1676159125 >> 172.21.15.67:0/1676159125 conn(0x7ffac4099a30 msgr2=0x7ffac406ec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:49.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.271+0000 7ffacb819640 1 -- 172.21.15.67:0/1676159125 shutdown_connections 2024-03-24T00:40:49.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.271+0000 7ffacb819640 1 -- 172.21.15.67:0/1676159125 wait complete. 2024-03-24T00:40:49.286 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":107,"stamp":"2024-03-24T00:40:47.346620+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222052,"kb_used_data":5864,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752668,"statfs":{"total":773094113280,"available":772866732032,"internally_reserved":0,"allocated":6004736,"data_stored":3304260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001899"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.586509+0000","last_change":"2024-03-24T00:40:40.304215+0000","last_active":"2024-03-24T00:40:41.586509+0000","last_peered":"2024-03-24T00:40:41.586509+0000","last_clean":"2024-03-24T00:40:41.586509+0000","last_became_active":"2024-03-24T00:40:40.303929+0000","last_became_peered":"2024-03-24T00:40:40.303929+0000","last_unstale":"2024-03-24T00:40:41.586509+0000","last_undegraded":"2024-03-24T00:40:41.586509+0000","last_fullsized":"2024-03-24T00:40:41.586509+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:57:36.878352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.302282+0000","last_change":"2024-03-24T00:40:40.300086+0000","last_active":"2024-03-24T00:40:41.302282+0000","last_peered":"2024-03-24T00:40:41.302282+0000","last_clean":"2024-03-24T00:40:41.302282+0000","last_became_active":"2024-03-24T00:40:40.299601+0000","last_became_peered":"2024-03-24T00:40:40.299601+0000","last_unstale":"2024-03-24T00:40:41.302282+0000","last_undegraded":"2024-03-24T00:40:41.302282+0000","last_fullsized":"2024-03-24T00:40:41.302282+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:47:38.271047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.298821+0000","last_change":"2024-03-24T00:40:40.289947+0000","last_active":"2024-03-24T00:40:41.298821+0000","last_peered":"2024-03-24T00:40:41.298821+0000","last_clean":"2024-03-24T00:40:41.298821+0000","last_became_active":"2024-03-24T00:40:40.289632+0000","last_became_peered":"2024-03-24T00:40:40.289632+0000","last_unstale":"2024-03-24T00:40:41.298821+0000","last_undegraded":"2024-03-24T00:40:41.298821+0000","last_fullsized":"2024-03-24T00:40:41.298821+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:45:44.359862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.301829+0000","last_change":"2024-03-24T00:40:40.305533+0000","last_active":"2024-03-24T00:40:41.301829+0000","last_peered":"2024-03-24T00:40:41.301829+0000","last_clean":"2024-03-24T00:40:41.301829+0000","last_became_active":"2024-03-24T00:40:40.305056+0000","last_became_peered":"2024-03-24T00:40:40.305056+0000","last_unstale":"2024-03-24T00:40:41.301829+0000","last_undegraded":"2024-03-24T00:40:41.301829+0000","last_fullsized":"2024-03-24T00:40:41.301829+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:04:50.495649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.845013+0000","last_change":"2024-03-24T00:40:40.300648+0000","last_active":"2024-03-24T00:40:41.845013+0000","last_peered":"2024-03-24T00:40:41.845013+0000","last_clean":"2024-03-24T00:40:41.845013+0000","last_became_active":"2024-03-24T00:40:40.300395+0000","last_became_peered":"2024-03-24T00:40:40.300395+0000","last_unstale":"2024-03-24T00:40:41.845013+0000","last_undegraded":"2024-03-24T00:40:41.845013+0000","last_fullsized":"2024-03-24T00:40:41.845013+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:19:39.784710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.586217+0000","last_change":"2024-03-24T00:40:40.304050+0000","last_active":"2024-03-24T00:40:41.586217+0000","last_peered":"2024-03-24T00:40:41.586217+0000","last_clean":"2024-03-24T00:40:41.586217+0000","last_became_active":"2024-03-24T00:40:40.303730+0000","last_became_peered":"2024-03-24T00:40:40.303730+0000","last_unstale":"2024-03-24T00:40:41.586217+0000","last_undegraded":"2024-03-24T00:40:41.586217+0000","last_fullsized":"2024-03-24T00:40:41.586217+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:39:53.235649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.298368+0000","last_change":"2024-03-24T00:40:40.289712+0000","last_active":"2024-03-24T00:40:41.298368+0000","last_peered":"2024-03-24T00:40:41.298368+0000","last_clean":"2024-03-24T00:40:41.298368+0000","last_became_active":"2024-03-24T00:40:40.289451+0000","last_became_peered":"2024-03-24T00:40:40.289451+0000","last_unstale":"2024-03-24T00:40:41.298368+0000","last_undegraded":"2024-03-24T00:40:41.298368+0000","last_fullsized":"2024-03-24T00:40:41.298368+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:16:13.256691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.295427+0000","last_change":"2024-03-24T00:40:40.307005+0000","last_active":"2024-03-24T00:40:41.295427+0000","last_peered":"2024-03-24T00:40:41.295427+0000","last_clean":"2024-03-24T00:40:41.295427+0000","last_became_active":"2024-03-24T00:40:40.306746+0000","last_became_peered":"2024-03-24T00:40:40.306746+0000","last_unstale":"2024-03-24T00:40:41.295427+0000","last_undegraded":"2024-03-24T00:40:41.295427+0000","last_fullsized":"2024-03-24T00:40:41.295427+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:42:06.753248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.295972+0000","last_change":"2024-03-24T00:40:40.300275+0000","last_active":"2024-03-24T00:40:41.295972+0000","last_peered":"2024-03-24T00:40:41.295972+0000","last_clean":"2024-03-24T00:40:41.295972+0000","last_became_active":"2024-03-24T00:40:40.298907+0000","last_became_peered":"2024-03-24T00:40:40.298907+0000","last_unstale":"2024-03-24T00:40:41.295972+0000","last_undegraded":"2024-03-24T00:40:41.295972+0000","last_fullsized":"2024-03-24T00:40:41.295972+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:23:33.487671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.586460+0000","last_change":"2024-03-24T00:40:40.302765+0000","last_active":"2024-03-24T00:40:41.586460+0000","last_peered":"2024-03-24T00:40:41.586460+0000","last_clean":"2024-03-24T00:40:41.586460+0000","last_became_active":"2024-03-24T00:40:40.302496+0000","last_became_peered":"2024-03-24T00:40:40.302496+0000","last_unstale":"2024-03-24T00:40:41.586460+0000","last_undegraded":"2024-03-24T00:40:41.586460+0000","last_fullsized":"2024-03-24T00:40:41.586460+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:04:50.495649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.300764+0000","last_change":"2024-03-24T00:40:40.297831+0000","last_active":"2024-03-24T00:40:41.300764+0000","last_peered":"2024-03-24T00:40:41.300764+0000","last_clean":"2024-03-24T00:40:41.300764+0000","last_became_active":"2024-03-24T00:40:40.297510+0000","last_became_peered":"2024-03-24T00:40:40.297510+0000","last_unstale":"2024-03-24T00:40:41.300764+0000","last_undegraded":"2024-03-24T00:40:41.300764+0000","last_fullsized":"2024-03-24T00:40:41.300764+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:30:40.670742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.295748+0000","last_change":"2024-03-24T00:40:40.307709+0000","last_active":"2024-03-24T00:40:41.295748+0000","last_peered":"2024-03-24T00:40:41.295748+0000","last_clean":"2024-03-24T00:40:41.295748+0000","last_became_active":"2024-03-24T00:40:40.307429+0000","last_became_peered":"2024-03-24T00:40:40.307429+0000","last_unstale":"2024-03-24T00:40:41.295748+0000","last_undegraded":"2024-03-24T00:40:41.295748+0000","last_fullsized":"2024-03-24T00:40:41.295748+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:38:20.268729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.586358+0000","last_change":"2024-03-24T00:40:40.303874+0000","last_active":"2024-03-24T00:40:41.586358+0000","last_peered":"2024-03-24T00:40:41.586358+0000","last_clean":"2024-03-24T00:40:41.586358+0000","last_became_active":"2024-03-24T00:40:40.303425+0000","last_became_peered":"2024-03-24T00:40:40.303425+0000","last_unstale":"2024-03-24T00:40:41.586358+0000","last_undegraded":"2024-03-24T00:40:41.586358+0000","last_fullsized":"2024-03-24T00:40:41.586358+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:23:26.624895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.298574+0000","last_change":"2024-03-24T00:40:40.289246+0000","last_active":"2024-03-24T00:40:41.298574+0000","last_peered":"2024-03-24T00:40:41.298574+0000","last_clean":"2024-03-24T00:40:41.298574+0000","last_became_active":"2024-03-24T00:40:40.288979+0000","last_became_peered":"2024-03-24T00:40:40.288979+0000","last_unstale":"2024-03-24T00:40:41.298574+0000","last_undegraded":"2024-03-24T00:40:41.298574+0000","last_fullsized":"2024-03-24T00:40:41.298574+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:18:03.493963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.301784+0000","last_change":"2024-03-24T00:40:40.298411+0000","last_active":"2024-03-24T00:40:41.301784+0000","last_peered":"2024-03-24T00:40:41.301784+0000","last_clean":"2024-03-24T00:40:41.301784+0000","last_became_active":"2024-03-24T00:40:40.298130+0000","last_became_peered":"2024-03-24T00:40:40.298130+0000","last_unstale":"2024-03-24T00:40:41.301784+0000","last_undegraded":"2024-03-24T00:40:41.301784+0000","last_fullsized":"2024-03-24T00:40:41.301784+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:23:26.624895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.586142+0000","last_change":"2024-03-24T00:40:40.301978+0000","last_active":"2024-03-24T00:40:41.586142+0000","last_peered":"2024-03-24T00:40:41.586142+0000","last_clean":"2024-03-24T00:40:41.586142+0000","last_became_active":"2024-03-24T00:40:40.301689+0000","last_became_peered":"2024-03-24T00:40:40.301689+0000","last_unstale":"2024-03-24T00:40:41.586142+0000","last_undegraded":"2024-03-24T00:40:41.586142+0000","last_fullsized":"2024-03-24T00:40:41.586142+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:05:33.548187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.299048+0000","last_change":"2024-03-24T00:40:40.292724+0000","last_active":"2024-03-24T00:40:41.299048+0000","last_peered":"2024-03-24T00:40:41.299048+0000","last_clean":"2024-03-24T00:40:41.299048+0000","last_became_active":"2024-03-24T00:40:40.291898+0000","last_became_peered":"2024-03-24T00:40:40.291898+0000","last_unstale":"2024-03-24T00:40:41.299048+0000","last_undegraded":"2024-03-24T00:40:41.299048+0000","last_fullsized":"2024-03-24T00:40:41.299048+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:34:06.342935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.302318+0000","last_change":"2024-03-24T00:40:40.299860+0000","last_active":"2024-03-24T00:40:41.302318+0000","last_peered":"2024-03-24T00:40:41.302318+0000","last_clean":"2024-03-24T00:40:41.302318+0000","last_became_active":"2024-03-24T00:40:40.299305+0000","last_became_peered":"2024-03-24T00:40:40.299305+0000","last_unstale":"2024-03-24T00:40:41.302318+0000","last_undegraded":"2024-03-24T00:40:41.302318+0000","last_fullsized":"2024-03-24T00:40:41.302318+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:35:52.681866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.298867+0000","last_change":"2024-03-24T00:40:40.296936+0000","last_active":"2024-03-24T00:40:41.298867+0000","last_peered":"2024-03-24T00:40:41.298867+0000","last_clean":"2024-03-24T00:40:41.298867+0000","last_became_active":"2024-03-24T00:40:40.296637+0000","last_became_peered":"2024-03-24T00:40:40.296637+0000","last_unstale":"2024-03-24T00:40:41.298867+0000","last_undegraded":"2024-03-24T00:40:41.298867+0000","last_fullsized":"2024-03-24T00:40:41.298867+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:04:47.728206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:42.237877+0000","last_change":"2024-03-24T00:40:40.300651+0000","last_active":"2024-03-24T00:40:42.237877+0000","last_peered":"2024-03-24T00:40:42.237877+0000","last_clean":"2024-03-24T00:40:42.237877+0000","last_became_active":"2024-03-24T00:40:40.300376+0000","last_became_peered":"2024-03-24T00:40:40.300376+0000","last_unstale":"2024-03-24T00:40:42.237877+0000","last_undegraded":"2024-03-24T00:40:42.237877+0000","last_fullsized":"2024-03-24T00:40:42.237877+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:27:10.311900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:42.237696+0000","last_change":"2024-03-24T00:40:40.303232+0000","last_active":"2024-03-24T00:40:42.237696+0000","last_peered":"2024-03-24T00:40:42.237696+0000","last_clean":"2024-03-24T00:40:42.237696+0000","last_became_active":"2024-03-24T00:40:40.302929+0000","last_became_peered":"2024-03-24T00:40:40.302929+0000","last_unstale":"2024-03-24T00:40:42.237696+0000","last_undegraded":"2024-03-24T00:40:42.237696+0000","last_fullsized":"2024-03-24T00:40:42.237696+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:05:46.708016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.295395+0000","last_change":"2024-03-24T00:40:40.308988+0000","last_active":"2024-03-24T00:40:41.295395+0000","last_peered":"2024-03-24T00:40:41.295395+0000","last_clean":"2024-03-24T00:40:41.295395+0000","last_became_active":"2024-03-24T00:40:40.308147+0000","last_became_peered":"2024-03-24T00:40:40.308147+0000","last_unstale":"2024-03-24T00:40:41.295395+0000","last_undegraded":"2024-03-24T00:40:41.295395+0000","last_fullsized":"2024-03-24T00:40:41.295395+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:34:06.342935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.300513+0000","last_change":"2024-03-24T00:40:40.298134+0000","last_active":"2024-03-24T00:40:41.300513+0000","last_peered":"2024-03-24T00:40:41.300513+0000","last_clean":"2024-03-24T00:40:41.300513+0000","last_became_active":"2024-03-24T00:40:40.297569+0000","last_became_peered":"2024-03-24T00:40:40.297569+0000","last_unstale":"2024-03-24T00:40:41.300513+0000","last_undegraded":"2024-03-24T00:40:41.300513+0000","last_fullsized":"2024-03-24T00:40:41.300513+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:52:40.714131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.298324+0000","last_change":"2024-03-24T00:40:40.290573+0000","last_active":"2024-03-24T00:40:41.298324+0000","last_peered":"2024-03-24T00:40:41.298324+0000","last_clean":"2024-03-24T00:40:41.298324+0000","last_became_active":"2024-03-24T00:40:40.290317+0000","last_became_peered":"2024-03-24T00:40:40.290317+0000","last_unstale":"2024-03-24T00:40:41.298324+0000","last_undegraded":"2024-03-24T00:40:41.298324+0000","last_fullsized":"2024-03-24T00:40:41.298324+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:11:36.428206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.299193+0000","last_change":"2024-03-24T00:40:40.301695+0000","last_active":"2024-03-24T00:40:41.299193+0000","last_peered":"2024-03-24T00:40:41.299193+0000","last_clean":"2024-03-24T00:40:41.299193+0000","last_became_active":"2024-03-24T00:40:40.301421+0000","last_became_peered":"2024-03-24T00:40:40.301421+0000","last_unstale":"2024-03-24T00:40:41.299193+0000","last_undegraded":"2024-03-24T00:40:41.299193+0000","last_fullsized":"2024-03-24T00:40:41.299193+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:42:06.753248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.295697+0000","last_change":"2024-03-24T00:40:40.307744+0000","last_active":"2024-03-24T00:40:41.295697+0000","last_peered":"2024-03-24T00:40:41.295697+0000","last_clean":"2024-03-24T00:40:41.295697+0000","last_became_active":"2024-03-24T00:40:40.307484+0000","last_became_peered":"2024-03-24T00:40:40.307484+0000","last_unstale":"2024-03-24T00:40:41.295697+0000","last_undegraded":"2024-03-24T00:40:41.295697+0000","last_fullsized":"2024-03-24T00:40:41.295697+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:07:01.023388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.586544+0000","last_change":"2024-03-24T00:40:40.306066+0000","last_active":"2024-03-24T00:40:41.586544+0000","last_peered":"2024-03-24T00:40:41.586544+0000","last_clean":"2024-03-24T00:40:41.586544+0000","last_became_active":"2024-03-24T00:40:40.305795+0000","last_became_peered":"2024-03-24T00:40:40.305795+0000","last_unstale":"2024-03-24T00:40:41.586544+0000","last_undegraded":"2024-03-24T00:40:41.586544+0000","last_fullsized":"2024-03-24T00:40:41.586544+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:16:51.931733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:42.237910+0000","last_change":"2024-03-24T00:40:40.303970+0000","last_active":"2024-03-24T00:40:42.237910+0000","last_peered":"2024-03-24T00:40:42.237910+0000","last_clean":"2024-03-24T00:40:42.237910+0000","last_became_active":"2024-03-24T00:40:40.303686+0000","last_became_peered":"2024-03-24T00:40:40.303686+0000","last_unstale":"2024-03-24T00:40:42.237910+0000","last_undegraded":"2024-03-24T00:40:42.237910+0000","last_fullsized":"2024-03-24T00:40:42.237910+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:55:02.465830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.298467+0000","last_change":"2024-03-24T00:40:40.295547+0000","last_active":"2024-03-24T00:40:41.298467+0000","last_peered":"2024-03-24T00:40:41.298467+0000","last_clean":"2024-03-24T00:40:41.298467+0000","last_became_active":"2024-03-24T00:40:40.294692+0000","last_became_peered":"2024-03-24T00:40:40.294692+0000","last_unstale":"2024-03-24T00:40:41.298467+0000","last_undegraded":"2024-03-24T00:40:41.298467+0000","last_fullsized":"2024-03-24T00:40:41.298467+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:14:05.593326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.302222+0000","last_change":"2024-03-24T00:40:40.304565+0000","last_active":"2024-03-24T00:40:41.302222+0000","last_peered":"2024-03-24T00:40:41.302222+0000","last_clean":"2024-03-24T00:40:41.302222+0000","last_became_active":"2024-03-24T00:40:40.304155+0000","last_became_peered":"2024-03-24T00:40:40.304155+0000","last_unstale":"2024-03-24T00:40:41.302222+0000","last_undegraded":"2024-03-24T00:40:41.302222+0000","last_fullsized":"2024-03-24T00:40:41.302222+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:16:51.931733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.298428+0000","last_change":"2024-03-24T00:40:40.288756+0000","last_active":"2024-03-24T00:40:41.298428+0000","last_peered":"2024-03-24T00:40:41.298428+0000","last_clean":"2024-03-24T00:40:41.298428+0000","last_became_active":"2024-03-24T00:40:40.288395+0000","last_became_peered":"2024-03-24T00:40:40.288395+0000","last_unstale":"2024-03-24T00:40:41.298428+0000","last_undegraded":"2024-03-24T00:40:41.298428+0000","last_fullsized":"2024-03-24T00:40:41.298428+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:14:36.606882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.301101+0000","last_change":"2024-03-24T00:40:40.302381+0000","last_active":"2024-03-24T00:40:41.301101+0000","last_peered":"2024-03-24T00:40:41.301101+0000","last_clean":"2024-03-24T00:40:41.301101+0000","last_became_active":"2024-03-24T00:40:40.302104+0000","last_became_peered":"2024-03-24T00:40:40.302104+0000","last_unstale":"2024-03-24T00:40:41.301101+0000","last_undegraded":"2024-03-24T00:40:41.301101+0000","last_fullsized":"2024-03-24T00:40:41.301101+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:02:02.195271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.301787+0000","last_change":"2024-03-24T00:40:40.297972+0000","last_active":"2024-03-24T00:40:41.301787+0000","last_peered":"2024-03-24T00:40:41.301787+0000","last_clean":"2024-03-24T00:40:41.301787+0000","last_became_active":"2024-03-24T00:40:40.297630+0000","last_became_peered":"2024-03-24T00:40:40.297630+0000","last_unstale":"2024-03-24T00:40:41.301787+0000","last_undegraded":"2024-03-24T00:40:41.301787+0000","last_fullsized":"2024-03-24T00:40:41.301787+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:06:06.434208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"49'4","reported_seq":18,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:42.237862+0000","last_change":"2024-03-24T00:40:39.291631+0000","last_active":"2024-03-24T00:40:42.237862+0000","last_peered":"2024-03-24T00:40:42.237862+0000","last_clean":"2024-03-24T00:40:42.237862+0000","last_became_active":"2024-03-24T00:40:39.291349+0000","last_became_peered":"2024-03-24T00:40:39.291349+0000","last_unstale":"2024-03-24T00:40:42.237862+0000","last_undegraded":"2024-03-24T00:40:42.237862+0000","last_fullsized":"2024-03-24T00:40:42.237862+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:38:21.054863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.301772+0000","last_change":"2024-03-24T00:40:40.294998+0000","last_active":"2024-03-24T00:40:41.301772+0000","last_peered":"2024-03-24T00:40:41.301772+0000","last_clean":"2024-03-24T00:40:41.301772+0000","last_became_active":"2024-03-24T00:40:40.294691+0000","last_became_peered":"2024-03-24T00:40:40.294691+0000","last_unstale":"2024-03-24T00:40:41.301772+0000","last_undegraded":"2024-03-24T00:40:41.301772+0000","last_fullsized":"2024-03-24T00:40:41.301772+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:14:05.593326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"49'1","reported_seq":15,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.586414+0000","last_change":"2024-03-24T00:40:39.295765+0000","last_active":"2024-03-24T00:40:41.586414+0000","last_peered":"2024-03-24T00:40:41.586414+0000","last_clean":"2024-03-24T00:40:41.586414+0000","last_became_active":"2024-03-24T00:40:39.295486+0000","last_became_peered":"2024-03-24T00:40:39.295486+0000","last_unstale":"2024-03-24T00:40:41.586414+0000","last_undegraded":"2024-03-24T00:40:41.586414+0000","last_fullsized":"2024-03-24T00:40:41.586414+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:42:28.507176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:42.237643+0000","last_change":"2024-03-24T00:40:40.301600+0000","last_active":"2024-03-24T00:40:42.237643+0000","last_peered":"2024-03-24T00:40:42.237643+0000","last_clean":"2024-03-24T00:40:42.237643+0000","last_became_active":"2024-03-24T00:40:40.301314+0000","last_became_peered":"2024-03-24T00:40:40.301314+0000","last_unstale":"2024-03-24T00:40:42.237643+0000","last_undegraded":"2024-03-24T00:40:42.237643+0000","last_fullsized":"2024-03-24T00:40:42.237643+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:38:57.796446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"49'2","reported_seq":16,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.586566+0000","last_change":"2024-03-24T00:40:39.294921+0000","last_active":"2024-03-24T00:40:41.586566+0000","last_peered":"2024-03-24T00:40:41.586566+0000","last_clean":"2024-03-24T00:40:41.586566+0000","last_became_active":"2024-03-24T00:40:39.294579+0000","last_became_peered":"2024-03-24T00:40:39.294579+0000","last_unstale":"2024-03-24T00:40:41.586566+0000","last_undegraded":"2024-03-24T00:40:41.586566+0000","last_fullsized":"2024-03-24T00:40:41.586566+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:35.720596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.295490+0000","last_change":"2024-03-24T00:40:40.303366+0000","last_active":"2024-03-24T00:40:41.295490+0000","last_peered":"2024-03-24T00:40:41.295490+0000","last_clean":"2024-03-24T00:40:41.295490+0000","last_became_active":"2024-03-24T00:40:40.303100+0000","last_became_peered":"2024-03-24T00:40:40.303100+0000","last_unstale":"2024-03-24T00:40:41.295490+0000","last_undegraded":"2024-03-24T00:40:41.295490+0000","last_fullsized":"2024-03-24T00:40:41.295490+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:20:37.376789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"49'2","reported_seq":16,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.301845+0000","last_change":"2024-03-24T00:40:39.297839+0000","last_active":"2024-03-24T00:40:41.301845+0000","last_peered":"2024-03-24T00:40:41.301845+0000","last_clean":"2024-03-24T00:40:41.301845+0000","last_became_active":"2024-03-24T00:40:39.297536+0000","last_became_peered":"2024-03-24T00:40:39.297536+0000","last_unstale":"2024-03-24T00:40:41.301845+0000","last_undegraded":"2024-03-24T00:40:41.301845+0000","last_fullsized":"2024-03-24T00:40:41.301845+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:39:52.227506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.845005+0000","last_change":"2024-03-24T00:40:40.298734+0000","last_active":"2024-03-24T00:40:41.845005+0000","last_peered":"2024-03-24T00:40:41.845005+0000","last_clean":"2024-03-24T00:40:41.845005+0000","last_became_active":"2024-03-24T00:40:40.298356+0000","last_became_peered":"2024-03-24T00:40:40.298356+0000","last_unstale":"2024-03-24T00:40:41.845005+0000","last_undegraded":"2024-03-24T00:40:41.845005+0000","last_fullsized":"2024-03-24T00:40:41.845005+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:55:02.465830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"49'1","reported_seq":15,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.298540+0000","last_change":"2024-03-24T00:40:39.304061+0000","last_active":"2024-03-24T00:40:41.298540+0000","last_peered":"2024-03-24T00:40:41.298540+0000","last_clean":"2024-03-24T00:40:41.298540+0000","last_became_active":"2024-03-24T00:40:39.303769+0000","last_became_peered":"2024-03-24T00:40:39.303769+0000","last_unstale":"2024-03-24T00:40:41.298540+0000","last_undegraded":"2024-03-24T00:40:41.298540+0000","last_fullsized":"2024-03-24T00:40:41.298540+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:38:56.788303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.302239+0000","last_change":"2024-03-24T00:40:40.304859+0000","last_active":"2024-03-24T00:40:41.302239+0000","last_peered":"2024-03-24T00:40:41.302239+0000","last_clean":"2024-03-24T00:40:41.302239+0000","last_became_active":"2024-03-24T00:40:40.304576+0000","last_became_peered":"2024-03-24T00:40:40.304576+0000","last_unstale":"2024-03-24T00:40:41.302239+0000","last_undegraded":"2024-03-24T00:40:41.302239+0000","last_fullsized":"2024-03-24T00:40:41.302239+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:48:03.404529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"49'1","reported_seq":15,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.844745+0000","last_change":"2024-03-24T00:40:39.293463+0000","last_active":"2024-03-24T00:40:41.844745+0000","last_peered":"2024-03-24T00:40:41.844745+0000","last_clean":"2024-03-24T00:40:41.844745+0000","last_became_active":"2024-03-24T00:40:39.293196+0000","last_became_peered":"2024-03-24T00:40:39.293196+0000","last_unstale":"2024-03-24T00:40:41.844745+0000","last_undegraded":"2024-03-24T00:40:41.844745+0000","last_fullsized":"2024-03-24T00:40:41.844745+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:35.720596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.295842+0000","last_change":"2024-03-24T00:40:40.309098+0000","last_active":"2024-03-24T00:40:41.295842+0000","last_peered":"2024-03-24T00:40:41.295842+0000","last_clean":"2024-03-24T00:40:41.295842+0000","last_became_active":"2024-03-24T00:40:40.307213+0000","last_became_peered":"2024-03-24T00:40:40.307213+0000","last_unstale":"2024-03-24T00:40:41.295842+0000","last_undegraded":"2024-03-24T00:40:41.295842+0000","last_fullsized":"2024-03-24T00:40:41.295842+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:26:47.125317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:42.237610+0000","last_change":"2024-03-24T00:40:39.294468+0000","last_active":"2024-03-24T00:40:42.237610+0000","last_peered":"2024-03-24T00:40:42.237610+0000","last_clean":"2024-03-24T00:40:42.237610+0000","last_became_active":"2024-03-24T00:40:39.294130+0000","last_became_peered":"2024-03-24T00:40:39.294130+0000","last_unstale":"2024-03-24T00:40:42.237610+0000","last_undegraded":"2024-03-24T00:40:42.237610+0000","last_fullsized":"2024-03-24T00:40:42.237610+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:54:49.331324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.301979+0000","last_change":"2024-03-24T00:40:40.299646+0000","last_active":"2024-03-24T00:40:41.301979+0000","last_peered":"2024-03-24T00:40:41.301979+0000","last_clean":"2024-03-24T00:40:41.301979+0000","last_became_active":"2024-03-24T00:40:40.299374+0000","last_became_peered":"2024-03-24T00:40:40.299374+0000","last_unstale":"2024-03-24T00:40:41.301979+0000","last_undegraded":"2024-03-24T00:40:41.301979+0000","last_fullsized":"2024-03-24T00:40:41.301979+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:57:36.878352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.298326+0000","last_change":"2024-03-24T00:40:39.296539+0000","last_active":"2024-03-24T00:40:41.298326+0000","last_peered":"2024-03-24T00:40:41.298326+0000","last_clean":"2024-03-24T00:40:41.298326+0000","last_became_active":"2024-03-24T00:40:39.296272+0000","last_became_peered":"2024-03-24T00:40:39.296272+0000","last_unstale":"2024-03-24T00:40:41.298326+0000","last_undegraded":"2024-03-24T00:40:41.298326+0000","last_fullsized":"2024-03-24T00:40:41.298326+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:18:21.492781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.844764+0000","last_change":"2024-03-24T00:40:40.307197+0000","last_active":"2024-03-24T00:40:41.844764+0000","last_peered":"2024-03-24T00:40:41.844764+0000","last_clean":"2024-03-24T00:40:41.844764+0000","last_became_active":"2024-03-24T00:40:40.306945+0000","last_became_peered":"2024-03-24T00:40:40.306945+0000","last_unstale":"2024-03-24T00:40:41.844764+0000","last_undegraded":"2024-03-24T00:40:41.844764+0000","last_fullsized":"2024-03-24T00:40:41.844764+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:10:00.938576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.301825+0000","last_change":"2024-03-24T00:40:39.296067+0000","last_active":"2024-03-24T00:40:41.301825+0000","last_peered":"2024-03-24T00:40:41.301825+0000","last_clean":"2024-03-24T00:40:41.301825+0000","last_became_active":"2024-03-24T00:40:39.295773+0000","last_became_peered":"2024-03-24T00:40:39.295773+0000","last_unstale":"2024-03-24T00:40:41.301825+0000","last_undegraded":"2024-03-24T00:40:41.301825+0000","last_fullsized":"2024-03-24T00:40:41.301825+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:38:56.788303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.301213+0000","last_change":"2024-03-24T00:40:40.295248+0000","last_active":"2024-03-24T00:40:41.301213+0000","last_peered":"2024-03-24T00:40:41.301213+0000","last_clean":"2024-03-24T00:40:41.301213+0000","last_became_active":"2024-03-24T00:40:40.294957+0000","last_became_peered":"2024-03-24T00:40:40.294957+0000","last_unstale":"2024-03-24T00:40:41.301213+0000","last_undegraded":"2024-03-24T00:40:41.301213+0000","last_fullsized":"2024-03-24T00:40:41.301213+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:45:21.056006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.844836+0000","last_change":"2024-03-24T00:40:39.296711+0000","last_active":"2024-03-24T00:40:41.844836+0000","last_peered":"2024-03-24T00:40:41.844836+0000","last_clean":"2024-03-24T00:40:41.844836+0000","last_became_active":"2024-03-24T00:40:39.296435+0000","last_became_peered":"2024-03-24T00:40:39.296435+0000","last_unstale":"2024-03-24T00:40:41.844836+0000","last_undegraded":"2024-03-24T00:40:41.844836+0000","last_fullsized":"2024-03-24T00:40:41.844836+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:43:07.435024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.298596+0000","last_change":"2024-03-24T00:40:40.294122+0000","last_active":"2024-03-24T00:40:41.298596+0000","last_peered":"2024-03-24T00:40:41.298596+0000","last_clean":"2024-03-24T00:40:41.298596+0000","last_became_active":"2024-03-24T00:40:40.293863+0000","last_became_peered":"2024-03-24T00:40:40.293863+0000","last_unstale":"2024-03-24T00:40:41.298596+0000","last_undegraded":"2024-03-24T00:40:41.298596+0000","last_fullsized":"2024-03-24T00:40:41.298596+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:44:34.057574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"49'3","reported_seq":17,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.302398+0000","last_change":"2024-03-24T00:40:39.301123+0000","last_active":"2024-03-24T00:40:41.302398+0000","last_peered":"2024-03-24T00:40:41.302398+0000","last_clean":"2024-03-24T00:40:41.302398+0000","last_became_active":"2024-03-24T00:40:39.300851+0000","last_became_peered":"2024-03-24T00:40:39.300851+0000","last_unstale":"2024-03-24T00:40:41.302398+0000","last_undegraded":"2024-03-24T00:40:41.302398+0000","last_fullsized":"2024-03-24T00:40:41.302398+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:56:55.475551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.845014+0000","last_change":"2024-03-24T00:40:40.302333+0000","last_active":"2024-03-24T00:40:41.845014+0000","last_peered":"2024-03-24T00:40:41.845014+0000","last_clean":"2024-03-24T00:40:41.845014+0000","last_became_active":"2024-03-24T00:40:40.301837+0000","last_became_peered":"2024-03-24T00:40:40.301837+0000","last_unstale":"2024-03-24T00:40:41.845014+0000","last_undegraded":"2024-03-24T00:40:41.845014+0000","last_fullsized":"2024-03-24T00:40:41.845014+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:27:10.311900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"49'6","reported_seq":20,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.295703+0000","last_change":"2024-03-24T00:40:39.302117+0000","last_active":"2024-03-24T00:40:41.295703+0000","last_peered":"2024-03-24T00:40:41.295703+0000","last_clean":"2024-03-24T00:40:41.295703+0000","last_became_active":"2024-03-24T00:40:39.301875+0000","last_became_peered":"2024-03-24T00:40:39.301875+0000","last_unstale":"2024-03-24T00:40:41.295703+0000","last_undegraded":"2024-03-24T00:40:41.295703+0000","last_fullsized":"2024-03-24T00:40:41.295703+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:11:35.420062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.301101+0000","last_change":"2024-03-24T00:40:40.296786+0000","last_active":"2024-03-24T00:40:41.301101+0000","last_peered":"2024-03-24T00:40:41.301101+0000","last_clean":"2024-03-24T00:40:41.301101+0000","last_became_active":"2024-03-24T00:40:40.294719+0000","last_became_peered":"2024-03-24T00:40:40.294719+0000","last_unstale":"2024-03-24T00:40:41.301101+0000","last_undegraded":"2024-03-24T00:40:41.301101+0000","last_fullsized":"2024-03-24T00:40:41.301101+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:35:52.681866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"49'2","reported_seq":16,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.298548+0000","last_change":"2024-03-24T00:40:39.298667+0000","last_active":"2024-03-24T00:40:41.298548+0000","last_peered":"2024-03-24T00:40:41.298548+0000","last_clean":"2024-03-24T00:40:41.298548+0000","last_became_active":"2024-03-24T00:40:39.298067+0000","last_became_peered":"2024-03-24T00:40:39.298067+0000","last_unstale":"2024-03-24T00:40:41.298548+0000","last_undegraded":"2024-03-24T00:40:41.298548+0000","last_fullsized":"2024-03-24T00:40:41.298548+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:09:59.930432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.299005+0000","last_change":"2024-03-24T00:40:40.303321+0000","last_active":"2024-03-24T00:40:41.299005+0000","last_peered":"2024-03-24T00:40:41.299005+0000","last_clean":"2024-03-24T00:40:41.299005+0000","last_became_active":"2024-03-24T00:40:40.303025+0000","last_became_peered":"2024-03-24T00:40:40.303025+0000","last_unstale":"2024-03-24T00:40:41.299005+0000","last_undegraded":"2024-03-24T00:40:41.299005+0000","last_fullsized":"2024-03-24T00:40:41.299005+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:03:06.246605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:42.237758+0000","last_change":"2024-03-24T00:40:39.299651+0000","last_active":"2024-03-24T00:40:42.237758+0000","last_peered":"2024-03-24T00:40:42.237758+0000","last_clean":"2024-03-24T00:40:42.237758+0000","last_became_active":"2024-03-24T00:40:39.299388+0000","last_became_peered":"2024-03-24T00:40:39.299388+0000","last_unstale":"2024-03-24T00:40:42.237758+0000","last_undegraded":"2024-03-24T00:40:42.237758+0000","last_fullsized":"2024-03-24T00:40:42.237758+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:43:07.435024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.295782+0000","last_change":"2024-03-24T00:40:40.304133+0000","last_active":"2024-03-24T00:40:41.295782+0000","last_peered":"2024-03-24T00:40:41.295782+0000","last_clean":"2024-03-24T00:40:41.295782+0000","last_became_active":"2024-03-24T00:40:40.303887+0000","last_became_peered":"2024-03-24T00:40:40.303887+0000","last_unstale":"2024-03-24T00:40:41.295782+0000","last_undegraded":"2024-03-24T00:40:41.295782+0000","last_fullsized":"2024-03-24T00:40:41.295782+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:40:46.140802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.301877+0000","last_change":"2024-03-24T00:40:39.295906+0000","last_active":"2024-03-24T00:40:41.301877+0000","last_peered":"2024-03-24T00:40:41.301877+0000","last_clean":"2024-03-24T00:40:41.301877+0000","last_became_active":"2024-03-24T00:40:39.295604+0000","last_became_peered":"2024-03-24T00:40:39.295604+0000","last_unstale":"2024-03-24T00:40:41.301877+0000","last_undegraded":"2024-03-24T00:40:41.301877+0000","last_fullsized":"2024-03-24T00:40:41.301877+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:18:21.492781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.299050+0000","last_change":"2024-03-24T00:40:40.291927+0000","last_active":"2024-03-24T00:40:41.299050+0000","last_peered":"2024-03-24T00:40:41.299050+0000","last_clean":"2024-03-24T00:40:41.299050+0000","last_became_active":"2024-03-24T00:40:40.291654+0000","last_became_peered":"2024-03-24T00:40:40.291654+0000","last_unstale":"2024-03-24T00:40:41.299050+0000","last_undegraded":"2024-03-24T00:40:41.299050+0000","last_fullsized":"2024-03-24T00:40:41.299050+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:27:08.995457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"49'8","reported_seq":22,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.295528+0000","last_change":"2024-03-24T00:40:39.298619+0000","last_active":"2024-03-24T00:40:41.295528+0000","last_peered":"2024-03-24T00:40:41.295528+0000","last_clean":"2024-03-24T00:40:41.295528+0000","last_became_active":"2024-03-24T00:40:39.298350+0000","last_became_peered":"2024-03-24T00:40:39.298350+0000","last_unstale":"2024-03-24T00:40:41.295528+0000","last_undegraded":"2024-03-24T00:40:41.295528+0000","last_fullsized":"2024-03-24T00:40:41.295528+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:45:20.047863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.295420+0000","last_change":"2024-03-24T00:40:40.299967+0000","last_active":"2024-03-24T00:40:41.295420+0000","last_peered":"2024-03-24T00:40:41.295420+0000","last_clean":"2024-03-24T00:40:41.295420+0000","last_became_active":"2024-03-24T00:40:40.298988+0000","last_became_peered":"2024-03-24T00:40:40.298988+0000","last_unstale":"2024-03-24T00:40:41.295420+0000","last_undegraded":"2024-03-24T00:40:41.295420+0000","last_fullsized":"2024-03-24T00:40:41.295420+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:27:08.995457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"49'1","reported_seq":15,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.299160+0000","last_change":"2024-03-24T00:40:39.299640+0000","last_active":"2024-03-24T00:40:41.299160+0000","last_peered":"2024-03-24T00:40:41.299160+0000","last_clean":"2024-03-24T00:40:41.299160+0000","last_became_active":"2024-03-24T00:40:39.299375+0000","last_became_peered":"2024-03-24T00:40:39.299375+0000","last_unstale":"2024-03-24T00:40:41.299160+0000","last_undegraded":"2024-03-24T00:40:41.299160+0000","last_fullsized":"2024-03-24T00:40:41.299160+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:27:09.303757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.586176+0000","last_change":"2024-03-24T00:40:40.299147+0000","last_active":"2024-03-24T00:40:41.586176+0000","last_peered":"2024-03-24T00:40:41.586176+0000","last_clean":"2024-03-24T00:40:41.586176+0000","last_became_active":"2024-03-24T00:40:40.298855+0000","last_became_peered":"2024-03-24T00:40:40.298855+0000","last_unstale":"2024-03-24T00:40:41.586176+0000","last_undegraded":"2024-03-24T00:40:41.586176+0000","last_fullsized":"2024-03-24T00:40:41.586176+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:58:22.169987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"49'2","reported_seq":16,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.300861+0000","last_change":"2024-03-24T00:40:39.302072+0000","last_active":"2024-03-24T00:40:41.300861+0000","last_peered":"2024-03-24T00:40:41.300861+0000","last_clean":"2024-03-24T00:40:41.300861+0000","last_became_active":"2024-03-24T00:40:39.301790+0000","last_became_peered":"2024-03-24T00:40:39.301790+0000","last_unstale":"2024-03-24T00:40:41.300861+0000","last_undegraded":"2024-03-24T00:40:41.300861+0000","last_fullsized":"2024-03-24T00:40:41.300861+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:24:11.659170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.844868+0000","last_change":"2024-03-24T00:40:40.299943+0000","last_active":"2024-03-24T00:40:41.844868+0000","last_peered":"2024-03-24T00:40:41.844868+0000","last_clean":"2024-03-24T00:40:41.844868+0000","last_became_active":"2024-03-24T00:40:40.299622+0000","last_became_peered":"2024-03-24T00:40:40.299622+0000","last_unstale":"2024-03-24T00:40:41.844868+0000","last_undegraded":"2024-03-24T00:40:41.844868+0000","last_fullsized":"2024-03-24T00:40:41.844868+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:05:46.708016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"49'1","reported_seq":15,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:42.237865+0000","last_change":"2024-03-24T00:40:39.292046+0000","last_active":"2024-03-24T00:40:42.237865+0000","last_peered":"2024-03-24T00:40:42.237865+0000","last_clean":"2024-03-24T00:40:42.237865+0000","last_became_active":"2024-03-24T00:40:39.291744+0000","last_became_peered":"2024-03-24T00:40:39.291744+0000","last_unstale":"2024-03-24T00:40:42.237865+0000","last_undegraded":"2024-03-24T00:40:42.237865+0000","last_fullsized":"2024-03-24T00:40:42.237865+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:09:53.820800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.586355+0000","last_change":"2024-03-24T00:40:40.304100+0000","last_active":"2024-03-24T00:40:41.586355+0000","last_peered":"2024-03-24T00:40:41.586355+0000","last_clean":"2024-03-24T00:40:41.586355+0000","last_became_active":"2024-03-24T00:40:40.303792+0000","last_became_peered":"2024-03-24T00:40:40.303792+0000","last_unstale":"2024-03-24T00:40:41.586355+0000","last_undegraded":"2024-03-24T00:40:41.586355+0000","last_fullsized":"2024-03-24T00:40:41.586355+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:00:25.379976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:42.238002+0000","last_change":"2024-03-24T00:40:39.304591+0000","last_active":"2024-03-24T00:40:42.238002+0000","last_peered":"2024-03-24T00:40:42.238002+0000","last_clean":"2024-03-24T00:40:42.238002+0000","last_became_active":"2024-03-24T00:40:39.304330+0000","last_became_peered":"2024-03-24T00:40:39.304330+0000","last_unstale":"2024-03-24T00:40:42.238002+0000","last_undegraded":"2024-03-24T00:40:42.238002+0000","last_fullsized":"2024-03-24T00:40:42.238002+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:42:28.507176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.301287+0000","last_change":"2024-03-24T00:40:40.295216+0000","last_active":"2024-03-24T00:40:41.301287+0000","last_peered":"2024-03-24T00:40:41.301287+0000","last_clean":"2024-03-24T00:40:41.301287+0000","last_became_active":"2024-03-24T00:40:40.294902+0000","last_became_peered":"2024-03-24T00:40:40.294902+0000","last_unstale":"2024-03-24T00:40:41.301287+0000","last_undegraded":"2024-03-24T00:40:41.301287+0000","last_fullsized":"2024-03-24T00:40:41.301287+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:11:36.428206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"49'2","reported_seq":16,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.586447+0000","last_change":"2024-03-24T00:40:39.287554+0000","last_active":"2024-03-24T00:40:41.586447+0000","last_peered":"2024-03-24T00:40:41.586447+0000","last_clean":"2024-03-24T00:40:41.586447+0000","last_became_active":"2024-03-24T00:40:39.287273+0000","last_became_peered":"2024-03-24T00:40:39.287273+0000","last_unstale":"2024-03-24T00:40:41.586447+0000","last_undegraded":"2024-03-24T00:40:41.586447+0000","last_fullsized":"2024-03-24T00:40:41.586447+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:43:07.435024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:42.237855+0000","last_change":"2024-03-24T00:40:40.300618+0000","last_active":"2024-03-24T00:40:42.237855+0000","last_peered":"2024-03-24T00:40:42.237855+0000","last_clean":"2024-03-24T00:40:42.237855+0000","last_became_active":"2024-03-24T00:40:40.300350+0000","last_became_peered":"2024-03-24T00:40:40.300350+0000","last_unstale":"2024-03-24T00:40:42.237855+0000","last_undegraded":"2024-03-24T00:40:42.237855+0000","last_fullsized":"2024-03-24T00:40:42.237855+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:19:39.784710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.295352+0000","last_change":"2024-03-24T00:40:39.298506+0000","last_active":"2024-03-24T00:40:41.295352+0000","last_peered":"2024-03-24T00:40:41.295352+0000","last_clean":"2024-03-24T00:40:41.295352+0000","last_became_active":"2024-03-24T00:40:39.297935+0000","last_became_peered":"2024-03-24T00:40:39.297935+0000","last_unstale":"2024-03-24T00:40:41.295352+0000","last_undegraded":"2024-03-24T00:40:41.295352+0000","last_fullsized":"2024-03-24T00:40:41.295352+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:35:51.673723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.298904+0000","last_change":"2024-03-24T00:40:40.294916+0000","last_active":"2024-03-24T00:40:41.298904+0000","last_peered":"2024-03-24T00:40:41.298904+0000","last_clean":"2024-03-24T00:40:41.298904+0000","last_became_active":"2024-03-24T00:40:40.294654+0000","last_became_peered":"2024-03-24T00:40:40.294654+0000","last_unstale":"2024-03-24T00:40:41.298904+0000","last_undegraded":"2024-03-24T00:40:41.298904+0000","last_fullsized":"2024-03-24T00:40:41.298904+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:04:44.682407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.298476+0000","last_change":"2024-03-24T00:40:39.296787+0000","last_active":"2024-03-24T00:40:41.298476+0000","last_peered":"2024-03-24T00:40:41.298476+0000","last_clean":"2024-03-24T00:40:41.298476+0000","last_became_active":"2024-03-24T00:40:39.296501+0000","last_became_peered":"2024-03-24T00:40:39.296501+0000","last_unstale":"2024-03-24T00:40:41.298476+0000","last_undegraded":"2024-03-24T00:40:41.298476+0000","last_fullsized":"2024-03-24T00:40:41.298476+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:56:55.475551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.586528+0000","last_change":"2024-03-24T00:40:40.303603+0000","last_active":"2024-03-24T00:40:41.586528+0000","last_peered":"2024-03-24T00:40:41.586528+0000","last_clean":"2024-03-24T00:40:41.586528+0000","last_became_active":"2024-03-24T00:40:40.303280+0000","last_became_peered":"2024-03-24T00:40:40.303280+0000","last_unstale":"2024-03-24T00:40:41.586528+0000","last_undegraded":"2024-03-24T00:40:41.586528+0000","last_fullsized":"2024-03-24T00:40:41.586528+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:48:03.404529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.300429+0000","last_change":"2024-03-24T00:40:39.300361+0000","last_active":"2024-03-24T00:40:41.300429+0000","last_peered":"2024-03-24T00:40:41.300429+0000","last_clean":"2024-03-24T00:40:41.300429+0000","last_became_active":"2024-03-24T00:40:39.299954+0000","last_became_peered":"2024-03-24T00:40:39.299954+0000","last_unstale":"2024-03-24T00:40:41.300429+0000","last_undegraded":"2024-03-24T00:40:41.300429+0000","last_fullsized":"2024-03-24T00:40:41.300429+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:45:03.385577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.301399+0000","last_change":"2024-03-24T00:40:40.301450+0000","last_active":"2024-03-24T00:40:41.301399+0000","last_peered":"2024-03-24T00:40:41.301399+0000","last_clean":"2024-03-24T00:40:41.301399+0000","last_became_active":"2024-03-24T00:40:40.301165+0000","last_became_peered":"2024-03-24T00:40:40.301165+0000","last_unstale":"2024-03-24T00:40:41.301399+0000","last_undegraded":"2024-03-24T00:40:41.301399+0000","last_fullsized":"2024-03-24T00:40:41.301399+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:16:13.256691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"49'2","reported_seq":16,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:42.237900+0000","last_change":"2024-03-24T00:40:39.299797+0000","last_active":"2024-03-24T00:40:42.237900+0000","last_peered":"2024-03-24T00:40:42.237900+0000","last_clean":"2024-03-24T00:40:42.237900+0000","last_became_active":"2024-03-24T00:40:39.299456+0000","last_became_peered":"2024-03-24T00:40:39.299456+0000","last_unstale":"2024-03-24T00:40:42.237900+0000","last_undegraded":"2024-03-24T00:40:42.237900+0000","last_fullsized":"2024-03-24T00:40:42.237900+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:16:22.016263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.586362+0000","last_change":"2024-03-24T00:40:40.308806+0000","last_active":"2024-03-24T00:40:41.586362+0000","last_peered":"2024-03-24T00:40:41.586362+0000","last_clean":"2024-03-24T00:40:41.586362+0000","last_became_active":"2024-03-24T00:40:40.308549+0000","last_became_peered":"2024-03-24T00:40:40.308549+0000","last_unstale":"2024-03-24T00:40:41.586362+0000","last_undegraded":"2024-03-24T00:40:41.586362+0000","last_fullsized":"2024-03-24T00:40:41.586362+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:06:06.434208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.302425+0000","last_change":"2024-03-24T00:40:39.299014+0000","last_active":"2024-03-24T00:40:41.302425+0000","last_peered":"2024-03-24T00:40:41.302425+0000","last_clean":"2024-03-24T00:40:41.302425+0000","last_became_active":"2024-03-24T00:40:39.298750+0000","last_became_peered":"2024-03-24T00:40:39.298750+0000","last_unstale":"2024-03-24T00:40:41.302425+0000","last_undegraded":"2024-03-24T00:40:41.302425+0000","last_fullsized":"2024-03-24T00:40:41.302425+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:09:59.930432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.295520+0000","last_change":"2024-03-24T00:40:40.307656+0000","last_active":"2024-03-24T00:40:41.295520+0000","last_peered":"2024-03-24T00:40:41.295520+0000","last_clean":"2024-03-24T00:40:41.295520+0000","last_became_active":"2024-03-24T00:40:40.307394+0000","last_became_peered":"2024-03-24T00:40:40.307394+0000","last_unstale":"2024-03-24T00:40:41.295520+0000","last_undegraded":"2024-03-24T00:40:41.295520+0000","last_fullsized":"2024-03-24T00:40:41.295520+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:41:32.183220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.298715+0000","last_change":"2024-03-24T00:40:39.303224+0000","last_active":"2024-03-24T00:40:41.298715+0000","last_peered":"2024-03-24T00:40:41.298715+0000","last_clean":"2024-03-24T00:40:41.298715+0000","last_became_active":"2024-03-24T00:40:39.302950+0000","last_became_peered":"2024-03-24T00:40:39.302950+0000","last_unstale":"2024-03-24T00:40:41.298715+0000","last_undegraded":"2024-03-24T00:40:41.298715+0000","last_fullsized":"2024-03-24T00:40:41.298715+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:05:45.699873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.298735+0000","last_change":"2024-03-24T00:40:40.305492+0000","last_active":"2024-03-24T00:40:41.298735+0000","last_peered":"2024-03-24T00:40:41.298735+0000","last_clean":"2024-03-24T00:40:41.298735+0000","last_became_active":"2024-03-24T00:40:40.305159+0000","last_became_peered":"2024-03-24T00:40:40.305159+0000","last_unstale":"2024-03-24T00:40:41.298735+0000","last_undegraded":"2024-03-24T00:40:41.298735+0000","last_fullsized":"2024-03-24T00:40:41.298735+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:27:01.459529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.295589+0000","last_change":"2024-03-24T00:40:39.299484+0000","last_active":"2024-03-24T00:40:41.295589+0000","last_peered":"2024-03-24T00:40:41.295589+0000","last_clean":"2024-03-24T00:40:41.295589+0000","last_became_active":"2024-03-24T00:40:39.299184+0000","last_became_peered":"2024-03-24T00:40:39.299184+0000","last_unstale":"2024-03-24T00:40:41.295589+0000","last_undegraded":"2024-03-24T00:40:41.295589+0000","last_fullsized":"2024-03-24T00:40:41.295589+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:47:37.262904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.301497+0000","last_change":"2024-03-24T00:40:40.295093+0000","last_active":"2024-03-24T00:40:41.301497+0000","last_peered":"2024-03-24T00:40:41.301497+0000","last_clean":"2024-03-24T00:40:41.301497+0000","last_became_active":"2024-03-24T00:40:40.294745+0000","last_became_peered":"2024-03-24T00:40:40.294745+0000","last_unstale":"2024-03-24T00:40:41.301497+0000","last_undegraded":"2024-03-24T00:40:41.301497+0000","last_fullsized":"2024-03-24T00:40:41.301497+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:14:36.606882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.295453+0000","last_change":"2024-03-24T00:40:39.298278+0000","last_active":"2024-03-24T00:40:41.295453+0000","last_peered":"2024-03-24T00:40:41.295453+0000","last_clean":"2024-03-24T00:40:41.295453+0000","last_became_active":"2024-03-24T00:40:39.297588+0000","last_became_peered":"2024-03-24T00:40:39.297588+0000","last_unstale":"2024-03-24T00:40:41.295453+0000","last_undegraded":"2024-03-24T00:40:41.295453+0000","last_fullsized":"2024-03-24T00:40:41.295453+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:02:01.187128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.298904+0000","last_change":"2024-03-24T00:40:40.290083+0000","last_active":"2024-03-24T00:40:41.298904+0000","last_peered":"2024-03-24T00:40:41.298904+0000","last_clean":"2024-03-24T00:40:41.298904+0000","last_became_active":"2024-03-24T00:40:40.288984+0000","last_became_peered":"2024-03-24T00:40:40.288984+0000","last_unstale":"2024-03-24T00:40:41.298904+0000","last_undegraded":"2024-03-24T00:40:41.298904+0000","last_fullsized":"2024-03-24T00:40:41.298904+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:02:01.565488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.844630+0000","last_change":"2024-03-24T00:40:39.300219+0000","last_active":"2024-03-24T00:40:41.844630+0000","last_peered":"2024-03-24T00:40:41.844630+0000","last_clean":"2024-03-24T00:40:41.844630+0000","last_became_active":"2024-03-24T00:40:39.299897+0000","last_became_peered":"2024-03-24T00:40:39.299897+0000","last_unstale":"2024-03-24T00:40:41.844630+0000","last_undegraded":"2024-03-24T00:40:41.844630+0000","last_fullsized":"2024-03-24T00:40:41.844630+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:42:28.507176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.295880+0000","last_change":"2024-03-24T00:40:40.302135+0000","last_active":"2024-03-24T00:40:41.295880+0000","last_peered":"2024-03-24T00:40:41.295880+0000","last_clean":"2024-03-24T00:40:41.295880+0000","last_became_active":"2024-03-24T00:40:40.301876+0000","last_became_peered":"2024-03-24T00:40:40.301876+0000","last_unstale":"2024-03-24T00:40:41.295880+0000","last_undegraded":"2024-03-24T00:40:41.295880+0000","last_fullsized":"2024-03-24T00:40:41.295880+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:03:47.424870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.301265+0000","last_change":"2024-03-24T00:40:39.294249+0000","last_active":"2024-03-24T00:40:41.301265+0000","last_peered":"2024-03-24T00:40:41.301265+0000","last_clean":"2024-03-24T00:40:41.301265+0000","last_became_active":"2024-03-24T00:40:39.293915+0000","last_became_peered":"2024-03-24T00:40:39.293915+0000","last_unstale":"2024-03-24T00:40:41.301265+0000","last_undegraded":"2024-03-24T00:40:41.301265+0000","last_fullsized":"2024-03-24T00:40:41.301265+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:00:24.371833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:42.237678+0000","last_change":"2024-03-24T00:40:40.296067+0000","last_active":"2024-03-24T00:40:42.237678+0000","last_peered":"2024-03-24T00:40:42.237678+0000","last_clean":"2024-03-24T00:40:42.237678+0000","last_became_active":"2024-03-24T00:40:40.295784+0000","last_became_peered":"2024-03-24T00:40:40.295784+0000","last_unstale":"2024-03-24T00:40:42.237678+0000","last_undegraded":"2024-03-24T00:40:42.237678+0000","last_fullsized":"2024-03-24T00:40:42.237678+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:39.278047+0000","last_clean_scrub_stamp":"2024-03-24T00:40:39.278047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:10:00.938576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"50'1","reported_seq":15,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.299018+0000","last_change":"2024-03-24T00:40:39.297685+0000","last_active":"2024-03-24T00:40:41.299018+0000","last_peered":"2024-03-24T00:40:41.299018+0000","last_clean":"2024-03-24T00:40:41.299018+0000","last_became_active":"2024-03-24T00:40:39.297097+0000","last_became_peered":"2024-03-24T00:40:39.297097+0000","last_unstale":"2024-03-24T00:40:41.299018+0000","last_undegraded":"2024-03-24T00:40:41.299018+0000","last_fullsized":"2024-03-24T00:40:41.299018+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:40:38.269904+0000","last_clean_scrub_stamp":"2024-03-24T00:40:38.269904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:19:38.776567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":225,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T00:40:41.301938+0000","last_change":"2024-03-24T00:38:40.953871+0000","last_active":"2024-03-24T00:40:41.301938+0000","last_peered":"2024-03-24T00:40:41.301938+0000","last_clean":"2024-03-24T00:40:41.301938+0000","last_became_active":"2024-03-24T00:38:40.952926+0000","last_became_peered":"2024-03-24T00:38:40.952926+0000","last_unstale":"2024-03-24T00:40:41.301938+0000","last_undegraded":"2024-03-24T00:40:41.301938+0000","last_fullsized":"2024-03-24T00:40:41.301938+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_clean_scrub_stamp":"2024-03-24T00:38:39.722194+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:05.397257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574878,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28192,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343648,"statfs":{"total":96636764160,"available":96607895552,"internally_reserved":0,"allocated":1196032,"data_stored":855330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574877,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27612,"kb_used_data":588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344228,"statfs":{"total":96636764160,"available":96608489472,"internally_reserved":0,"allocated":602112,"data_stored":265490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574877,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27604,"kb_used_data":580,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344236,"statfs":{"total":96636764160,"available":96608497664,"internally_reserved":0,"allocated":593920,"data_stored":264838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574876,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27608,"kb_used_data":584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344232,"statfs":{"total":96636764160,"available":96608493568,"internally_reserved":0,"allocated":598016,"data_stored":265420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574876,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27620,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":610304,"data_stored":266472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574877,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":606208,"data_stored":266450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574877,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27612,"kb_used_data":588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344228,"statfs":{"total":96636764160,"available":96608489472,"internally_reserved":0,"allocated":602112,"data_stored":264998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574876,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28188,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343652,"statfs":{"total":96636764160,"available":96607899648,"internally_reserved":0,"allocated":1196032,"data_stored":855262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:40:49.289 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-24T00:40:49.289 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-24T00:40:49.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.437+0000 7fba7fec6640 1 Processor -- start 2024-03-24T00:40:49.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.438+0000 7fba7fec6640 1 -- start start 2024-03-24T00:40:49.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.438+0000 7fba7fec6640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fba7815e170 0x7fba780d21c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:49.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.438+0000 7fba7fec6640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fba780b9b50 0x7fba780cebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:49.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.438+0000 7fba7fec6640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba780d2af0 con 0x7fba780cf100 2024-03-24T00:40:49.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.438+0000 7fba7fec6640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba780d2dd0 con 0x7fba780b9b50 2024-03-24T00:40:49.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.438+0000 7fba7fec6640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba780d30b0 con 0x7fba7815e170 2024-03-24T00:40:49.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.438+0000 7fba7d461640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fba780b9b50 0x7fba780cebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:49.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.438+0000 7fba7e463640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fba780cf100 0x7fba780b5870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39658/0 (socket says 172.21.15.67:39658) 2024-03-24T00:40:49.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.438+0000 7fba7d461640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fba780b9b50 0x7fba780cebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40864/0 (socket says 172.21.15.67:40864) 2024-03-24T00:40:49.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.438+0000 7fba7e463640 1 -- 172.21.15.67:0/1172742681 learned_addr learned my addr 172.21.15.67:0/1172742681 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:40:49.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.439+0000 7fba7dc62640 1 --2- 172.21.15.67:0/1172742681 >> v2:172.21.15.72:3300/0 conn(0x7fba7815e170 0x7fba780d21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:49.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.439+0000 7fba7d461640 1 -- 172.21.15.67:0/1172742681 >> v2:172.21.15.72:3300/0 conn(0x7fba7815e170 msgr2=0x7fba780d21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:49.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.439+0000 7fba7d461640 1 --2- 172.21.15.67:0/1172742681 >> v2:172.21.15.72:3300/0 conn(0x7fba7815e170 0x7fba780d21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:49.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.439+0000 7fba7d461640 1 -- 172.21.15.67:0/1172742681 >> v1:172.21.15.67:6789/0 conn(0x7fba780cf100 legacy=0x7fba780b5870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:49.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.439+0000 7fba7d461640 1 -- 172.21.15.67:0/1172742681 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba780d3390 con 0x7fba780b9b50 2024-03-24T00:40:49.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.439+0000 7fba7cc60640 1 -- 172.21.15.67:0/1172742681 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1995571756 0 0) 0x7fba780d2af0 con 0x7fba780cf100 2024-03-24T00:40:49.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.439+0000 7fba7dc62640 1 --2- 172.21.15.67:0/1172742681 >> v2:172.21.15.72:3300/0 conn(0x7fba7815e170 0x7fba780d21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:40:49.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.440+0000 7fba7d461640 1 --2- 172.21.15.67:0/1172742681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba780b9b50 0x7fba780cebc0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fba780b6950 tx=0x7fba6c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a98d193c36c6240 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:49.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.440+0000 7fba7cc60640 1 -- 172.21.15.67:0/1172742681 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba6c035020 con 0x7fba780b9b50 2024-03-24T00:40:49.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.440+0000 7fba7cc60640 1 -- 172.21.15.67:0/1172742681 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba6c031830 con 0x7fba780b9b50 2024-03-24T00:40:49.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.440+0000 7fba7cc60640 1 -- 172.21.15.67:0/1172742681 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba6c031c00 con 0x7fba780b9b50 2024-03-24T00:40:49.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.440+0000 7fba7fec6640 1 -- 172.21.15.67:0/1172742681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba780b9b50 msgr2=0x7fba780cebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:49.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.440+0000 7fba7fec6640 1 --2- 172.21.15.67:0/1172742681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba780b9b50 0x7fba780cebc0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fba780b6950 tx=0x7fba6c02b8d0 comp rx=0 tx=0).stop 2024-03-24T00:40:49.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.440+0000 7fba7fec6640 1 -- 172.21.15.67:0/1172742681 shutdown_connections 2024-03-24T00:40:49.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.440+0000 7fba7fec6640 1 --2- 172.21.15.67:0/1172742681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba780b9b50 0x7fba780cebc0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:49.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.440+0000 7fba7fec6640 1 --2- 172.21.15.67:0/1172742681 >> v2:172.21.15.72:3300/0 conn(0x7fba7815e170 0x7fba780d21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:49.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.440+0000 7fba7fec6640 1 -- 172.21.15.67:0/1172742681 >> 172.21.15.67:0/1172742681 conn(0x7fba7806c810 msgr2=0x7fba7806cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:49.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.440+0000 7fba7fec6640 1 -- 172.21.15.67:0/1172742681 shutdown_connections 2024-03-24T00:40:49.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.440+0000 7fba7fec6640 1 -- 172.21.15.67:0/1172742681 wait complete. 2024-03-24T00:40:49.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.441+0000 7fba7fec6640 1 Processor -- start 2024-03-24T00:40:49.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.441+0000 7fba7fec6640 1 -- start start 2024-03-24T00:40:49.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.442+0000 7fba7fec6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fba780b9b50 0x7fba7814e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:49.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.442+0000 7fba7fec6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba780cf100 0x7fba780b8840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:49.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.442+0000 7fba7fec6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba7815e170 0x7fba780b8d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:49.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.442+0000 7fba7fec6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba780d2880 con 0x7fba780cf100 2024-03-24T00:40:49.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.442+0000 7fba7fec6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba780d2700 con 0x7fba7815e170 2024-03-24T00:40:49.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.442+0000 7fba7fec6640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba780b7f30 con 0x7fba780b9b50 2024-03-24T00:40:49.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.442+0000 7fba7d461640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba780cf100 0x7fba780b8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:49.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.442+0000 7fba7d461640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba780cf100 0x7fba780b8840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40878/0 (socket says 172.21.15.67:40878) 2024-03-24T00:40:49.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.442+0000 7fba7d461640 1 -- 172.21.15.67:0/1707765204 learned_addr learned my addr 172.21.15.67:0/1707765204 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:49.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.442+0000 7fba7e463640 1 --2- 172.21.15.67:0/1707765204 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba7815e170 0x7fba780b8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:49.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.443+0000 7fba7d461640 1 -- 172.21.15.67:0/1707765204 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fba780b9b50 msgr2=0x7fba7814e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:49.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.443+0000 7fba7d461640 1 --2- 172.21.15.67:0/1707765204 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fba780b9b50 0x7fba7814e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:49.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.443+0000 7fba7d461640 1 -- 172.21.15.67:0/1707765204 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba7815e170 msgr2=0x7fba780b8d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:49.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.443+0000 7fba7d461640 1 --2- 172.21.15.67:0/1707765204 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba7815e170 0x7fba780b8d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:49.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.443+0000 7fba7d461640 1 -- 172.21.15.67:0/1707765204 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba780cbd20 con 0x7fba780cf100 2024-03-24T00:40:49.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.443+0000 7fba7e463640 1 --2- 172.21.15.67:0/1707765204 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba7815e170 0x7fba780b8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:40:49.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.443+0000 7fba7d461640 1 --2- 172.21.15.67:0/1707765204 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba780cf100 0x7fba780b8840 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fba6c009890 tx=0x7fba6c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:49.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.443+0000 7fba767fc640 1 -- 172.21.15.67:0/1707765204 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba6c041070 con 0x7fba780cf100 2024-03-24T00:40:49.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.443+0000 7fba767fc640 1 -- 172.21.15.67:0/1707765204 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba6c03aa30 con 0x7fba780cf100 2024-03-24T00:40:49.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.443+0000 7fba767fc640 1 -- 172.21.15.67:0/1707765204 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba6c035040 con 0x7fba780cf100 2024-03-24T00:40:49.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.443+0000 7fba7fec6640 1 -- 172.21.15.67:0/1707765204 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba7815e7a0 con 0x7fba780cf100 2024-03-24T00:40:49.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.443+0000 7fba7fec6640 1 -- 172.21.15.67:0/1707765204 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba7815ed60 con 0x7fba780cf100 2024-03-24T00:40:49.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.444+0000 7fba7fec6640 1 -- 172.21.15.67:0/1707765204 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba7804f400 con 0x7fba780cf100 2024-03-24T00:40:49.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.444+0000 7fba767fc640 1 -- 172.21.15.67:0/1707765204 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fba6c038020 con 0x7fba780cf100 2024-03-24T00:40:49.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.445+0000 7fba767fc640 1 --2- 172.21.15.67:0/1707765204 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fba6406bbd0 0x7fba6406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:49.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.445+0000 7fba767fc640 1 -- 172.21.15.67:0/1707765204 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fba6c0ac7d0 con 0x7fba780cf100 2024-03-24T00:40:49.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.445+0000 7fba7dc62640 1 --2- 172.21.15.67:0/1707765204 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fba6406bbd0 0x7fba6406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:49.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.445+0000 7fba7dc62640 1 --2- 172.21.15.67:0/1707765204 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fba6406bbd0 0x7fba6406e090 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fba68008e00 tx=0x7fba68008330 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:49.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.449+0000 7fba767fc640 1 -- 172.21.15.67:0/1707765204 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fba6c02f2e0 con 0x7fba780cf100 2024-03-24T00:40:49.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.743+0000 7fba7fec6640 1 -- 172.21.15.67:0/1707765204 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fba7815d4d0 con 0x7fba780cf100 2024-03-24T00:40:49.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.745+0000 7fba767fc640 1 -- 172.21.15.67:0/1707765204 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v14) v1 ==== 190+0+147 (secure 0 0 0) 0x7fba6c07b130 con 0x7fba780cf100 2024-03-24T00:40:49.746 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-03-24T00:40:49.746 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQDcdf9lY4nvJRAA6HWOZDWA+SSvJVmyjG+51g== 2024-03-24T00:40:49.746 INFO:teuthology.orchestra.run.smithi067.stdout: caps mds = "allow" 2024-03-24T00:40:49.746 INFO:teuthology.orchestra.run.smithi067.stdout: caps mon = "allow r" 2024-03-24T00:40:49.746 INFO:teuthology.orchestra.run.smithi067.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-24T00:40:49.746 INFO:teuthology.orchestra.run.smithi067.stderr:updated caps for client.0 2024-03-24T00:40:49.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.747+0000 7fba7fec6640 1 -- 172.21.15.67:0/1707765204 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fba6406bbd0 msgr2=0x7fba6406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:49.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.747+0000 7fba7fec6640 1 --2- 172.21.15.67:0/1707765204 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fba6406bbd0 0x7fba6406e090 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fba68008e00 tx=0x7fba68008330 comp rx=0 tx=0).stop 2024-03-24T00:40:49.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.747+0000 7fba7fec6640 1 -- 172.21.15.67:0/1707765204 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba780cf100 msgr2=0x7fba780b8840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:49.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.747+0000 7fba7fec6640 1 --2- 172.21.15.67:0/1707765204 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba780cf100 0x7fba780b8840 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fba6c009890 tx=0x7fba6c0046c0 comp rx=0 tx=0).stop 2024-03-24T00:40:49.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.747+0000 7fba7fec6640 1 -- 172.21.15.67:0/1707765204 shutdown_connections 2024-03-24T00:40:49.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.747+0000 7fba7fec6640 1 --2- 172.21.15.67:0/1707765204 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fba6406bbd0 0x7fba6406e090 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:49.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.747+0000 7fba7fec6640 1 --2- 172.21.15.67:0/1707765204 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba7815e170 0x7fba780b8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:49.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.747+0000 7fba7fec6640 1 --2- 172.21.15.67:0/1707765204 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba780cf100 0x7fba780b8840 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:49.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.747+0000 7fba7fec6640 1 --2- 172.21.15.67:0/1707765204 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fba780b9b50 0x7fba7814e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:49.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.747+0000 7fba7fec6640 1 -- 172.21.15.67:0/1707765204 >> 172.21.15.67:0/1707765204 conn(0x7fba7806c810 msgr2=0x7fba780ce660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:49.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.747+0000 7fba7fec6640 1 -- 172.21.15.67:0/1707765204 shutdown_connections 2024-03-24T00:40:49.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.747+0000 7fba7fec6640 1 -- 172.21.15.67:0/1707765204 wait complete. 2024-03-24T00:40:49.758 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:40:49.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.859+0000 7f5bccbe6640 1 Processor -- start 2024-03-24T00:40:49.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.859+0000 7f5bccbe6640 1 -- start start 2024-03-24T00:40:49.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.860+0000 7f5bccbe6640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5bc815def0 0x7f5bc806c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:49.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.860+0000 7f5bccbe6640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5bc815e2d0 0x7f5bc80cff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:49.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.860+0000 7f5bccbe6640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bc80d0490 con 0x7f5bc80ce460 2024-03-24T00:40:49.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.860+0000 7f5bccbe6640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bc80cecb0 con 0x7f5bc815def0 2024-03-24T00:40:49.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.860+0000 7f5bccbe6640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bc80cef90 con 0x7f5bc815e2d0 2024-03-24T00:40:49.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.860+0000 7f5bc659c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5bc815def0 0x7f5bc806c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:49.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.860+0000 7f5bc659c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5bc815def0 0x7f5bc806c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40886/0 (socket says 172.21.15.67:40886) 2024-03-24T00:40:49.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.860+0000 7f5bc659c640 1 -- 172.21.15.67:0/317315525 learned_addr learned my addr 172.21.15.67:0/317315525 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:49.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.860+0000 7f5bbdd9b640 1 --2- 172.21.15.67:0/317315525 >> v2:172.21.15.72:3300/0 conn(0x7f5bc815e2d0 0x7f5bc80cff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:49.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.860+0000 7f5bc659c640 1 -- 172.21.15.67:0/317315525 >> v2:172.21.15.72:3300/0 conn(0x7f5bc815e2d0 msgr2=0x7f5bc80cff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:49.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.860+0000 7f5bc659c640 1 --2- 172.21.15.67:0/317315525 >> v2:172.21.15.72:3300/0 conn(0x7f5bc815e2d0 0x7f5bc80cff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:49.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.860+0000 7f5bc659c640 1 -- 172.21.15.67:0/317315525 >> v1:172.21.15.72:6789/0 conn(0x7f5bc80ce460 legacy=0x7f5bc806cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:49.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.860+0000 7f5bc659c640 1 -- 172.21.15.67:0/317315525 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bc80cf270 con 0x7f5bc815def0 2024-03-24T00:40:49.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.861+0000 7f5bc659c640 1 --2- 172.21.15.67:0/317315525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bc815def0 0x7f5bc806c5d0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f5bc80cf730 tx=0x7f5bb802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c337ae4b6c4c1b69 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:49.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.861+0000 7f5bc5d9b640 1 -- 172.21.15.67:0/317315525 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bb802e070 con 0x7f5bc815def0 2024-03-24T00:40:49.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.861+0000 7f5bc5d9b640 1 -- 172.21.15.67:0/317315525 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bb8037070 con 0x7f5bc815def0 2024-03-24T00:40:49.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.861+0000 7f5bc5d9b640 1 -- 172.21.15.67:0/317315525 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bb8031a00 con 0x7f5bc815def0 2024-03-24T00:40:49.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.862+0000 7f5bccbe6640 1 -- 172.21.15.67:0/317315525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bc815def0 msgr2=0x7f5bc806c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:49.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.862+0000 7f5bccbe6640 1 --2- 172.21.15.67:0/317315525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bc815def0 0x7f5bc806c5d0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f5bc80cf730 tx=0x7f5bb802b880 comp rx=0 tx=0).stop 2024-03-24T00:40:49.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.862+0000 7f5bccbe6640 1 -- 172.21.15.67:0/317315525 shutdown_connections 2024-03-24T00:40:49.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.862+0000 7f5bccbe6640 1 --2- 172.21.15.67:0/317315525 >> v2:172.21.15.72:3300/0 conn(0x7f5bc815e2d0 0x7f5bc80cff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:49.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.862+0000 7f5bccbe6640 1 --2- 172.21.15.67:0/317315525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bc815def0 0x7f5bc806c5d0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:49.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.862+0000 7f5bccbe6640 1 -- 172.21.15.67:0/317315525 >> 172.21.15.67:0/317315525 conn(0x7f5bc808b1b0 msgr2=0x7f5bc808b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:49.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.862+0000 7f5bccbe6640 1 -- 172.21.15.67:0/317315525 shutdown_connections 2024-03-24T00:40:49.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.862+0000 7f5bccbe6640 1 -- 172.21.15.67:0/317315525 wait complete. 2024-03-24T00:40:49.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.863+0000 7f5bccbe6640 1 Processor -- start 2024-03-24T00:40:49.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.863+0000 7f5bccbe6640 1 -- start start 2024-03-24T00:40:49.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.864+0000 7f5bccbe6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bc80ce460 0x7f5bc81473c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:49.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.864+0000 7f5bccbe6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5bc815def0 0x7f5bc81f3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:49.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.864+0000 7f5bccbe6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5bc8147900 0x7f5bc8160890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:49.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.864+0000 7f5bccbe6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc80cea10 con 0x7f5bc80ce460 2024-03-24T00:40:49.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.864+0000 7f5bccbe6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc80ce890 con 0x7f5bc815def0 2024-03-24T00:40:49.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.864+0000 7f5bccbe6640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc806d270 con 0x7f5bc8147900 2024-03-24T00:40:49.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.864+0000 7f5bc659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bc80ce460 0x7f5bc81473c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:49.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.864+0000 7f5bc659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bc80ce460 0x7f5bc81473c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40894/0 (socket says 172.21.15.67:40894) 2024-03-24T00:40:49.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.864+0000 7f5bc659c640 1 -- 172.21.15.67:0/2134385027 learned_addr learned my addr 172.21.15.67:0/2134385027 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:49.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.864+0000 7f5bc659c640 1 -- 172.21.15.67:0/2134385027 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5bc8147900 msgr2=0x7f5bc8160890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:49.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.864+0000 7f5bbdd9b640 1 --2- 172.21.15.67:0/2134385027 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5bc815def0 0x7f5bc81f3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:49.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.864+0000 7f5bc659c640 1 --2- 172.21.15.67:0/2134385027 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5bc8147900 0x7f5bc8160890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:49.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.864+0000 7f5bc659c640 1 -- 172.21.15.67:0/2134385027 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5bc815def0 msgr2=0x7f5bc81f3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:49.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.864+0000 7f5bc659c640 1 --2- 172.21.15.67:0/2134385027 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5bc815def0 0x7f5bc81f3800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:49.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.864+0000 7f5bc659c640 1 -- 172.21.15.67:0/2134385027 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bc8147d80 con 0x7f5bc80ce460 2024-03-24T00:40:49.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.865+0000 7f5bc659c640 1 --2- 172.21.15.67:0/2134385027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bc80ce460 0x7f5bc81473c0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f5bb8000c00 tx=0x7f5bb80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:49.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.865+0000 7f5bbf7fe640 1 -- 172.21.15.67:0/2134385027 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bb8032880 con 0x7f5bc80ce460 2024-03-24T00:40:49.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.865+0000 7f5bbf7fe640 1 -- 172.21.15.67:0/2134385027 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bb8032a20 con 0x7f5bc80ce460 2024-03-24T00:40:49.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.865+0000 7f5bccbe6640 1 -- 172.21.15.67:0/2134385027 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bc8148060 con 0x7f5bc80ce460 2024-03-24T00:40:49.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.865+0000 7f5bbf7fe640 1 -- 172.21.15.67:0/2134385027 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bb8032e10 con 0x7f5bc80ce460 2024-03-24T00:40:49.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.865+0000 7f5bccbe6640 1 -- 172.21.15.67:0/2134385027 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bc80d1a00 con 0x7f5bc80ce460 2024-03-24T00:40:49.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.866+0000 7f5bccbe6640 1 -- 172.21.15.67:0/2134385027 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bc806cb40 con 0x7f5bc80ce460 2024-03-24T00:40:49.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.866+0000 7f5bbf7fe640 1 -- 172.21.15.67:0/2134385027 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5bb804c060 con 0x7f5bc80ce460 2024-03-24T00:40:49.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.867+0000 7f5bbf7fe640 1 --2- 172.21.15.67:0/2134385027 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5ba806bbd0 0x7f5ba806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:49.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.867+0000 7f5bbf7fe640 1 -- 172.21.15.67:0/2134385027 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5bb80ac290 con 0x7f5bc80ce460 2024-03-24T00:40:49.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.867+0000 7f5bbdd9b640 1 --2- 172.21.15.67:0/2134385027 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5ba806bbd0 0x7f5ba806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:49.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.867+0000 7f5bbdd9b640 1 --2- 172.21.15.67:0/2134385027 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5ba806bbd0 0x7f5ba806e090 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f5bb0006fd0 tx=0x7f5bb0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:49.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:49.871+0000 7f5bbf7fe640 1 -- 172.21.15.67:0/2134385027 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5bc806cb40 con 0x7f5bc80ce460 2024-03-24T00:40:50.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.091+0000 7f5bccbe6640 1 -- 172.21.15.67:0/2134385027 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5bc804f400 con 0x7f5bc80ce460 2024-03-24T00:40:50.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.091+0000 7f5bbf7fe640 1 -- 172.21.15.67:0/2134385027 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f5bb8040070 con 0x7f5bc80ce460 2024-03-24T00:40:50.092 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:40:50.092 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5360,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3312887486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":3312887486},{"type":"v1","addr":"172.21.15.67:6835","nonce":3312887486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5821,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/4171866637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":4171866637},{"type":"v1","addr":"172.21.15.72:6836","nonce":4171866637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5824,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/4106412452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":4106412452},{"type":"v1","addr":"172.21.15.72:6837","nonce":4106412452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:40:39.649631+0000","modified":"2024-03-24T00:40:42.657888+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5853},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5853":{"gid":5853,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6834/2741818214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2741818214},{"type":"v1","addr":"172.21.15.67:6834","nonce":2741818214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5853,"qdb_cluster":[5853]},"id":3}]} 2024-03-24T00:40:50.093 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 26 2024-03-24T00:40:50.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.093+0000 7f5bccbe6640 1 -- 172.21.15.67:0/2134385027 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5ba806bbd0 msgr2=0x7f5ba806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:50.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.093+0000 7f5bccbe6640 1 --2- 172.21.15.67:0/2134385027 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5ba806bbd0 0x7f5ba806e090 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f5bb0006fd0 tx=0x7f5bb0008040 comp rx=0 tx=0).stop 2024-03-24T00:40:50.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.093+0000 7f5bccbe6640 1 -- 172.21.15.67:0/2134385027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bc80ce460 msgr2=0x7f5bc81473c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:50.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.093+0000 7f5bccbe6640 1 --2- 172.21.15.67:0/2134385027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bc80ce460 0x7f5bc81473c0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f5bb8000c00 tx=0x7f5bb80046c0 comp rx=0 tx=0).stop 2024-03-24T00:40:50.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.093+0000 7f5bccbe6640 1 -- 172.21.15.67:0/2134385027 shutdown_connections 2024-03-24T00:40:50.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.093+0000 7f5bccbe6640 1 --2- 172.21.15.67:0/2134385027 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5ba806bbd0 0x7f5ba806e090 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:50.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.093+0000 7f5bccbe6640 1 --2- 172.21.15.67:0/2134385027 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5bc8147900 0x7f5bc8160890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:50.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.093+0000 7f5bccbe6640 1 --2- 172.21.15.67:0/2134385027 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5bc815def0 0x7f5bc81f3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:50.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.093+0000 7f5bccbe6640 1 --2- 172.21.15.67:0/2134385027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bc80ce460 0x7f5bc81473c0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:50.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.093+0000 7f5bccbe6640 1 -- 172.21.15.67:0/2134385027 >> 172.21.15.67:0/2134385027 conn(0x7f5bc808b1b0 msgr2=0x7f5bc80b2030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:50.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.093+0000 7f5bccbe6640 1 -- 172.21.15.67:0/2134385027 shutdown_connections 2024-03-24T00:40:50.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.093+0000 7f5bccbe6640 1 -- 172.21.15.67:0/2134385027 wait complete. 2024-03-24T00:40:50.507 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:40:39.649631+0000', 'modified': '2024-03-24T00:40:42.657888+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5853}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5853': {'gid': 5853, 'name': 'a', 'rank': 0, 'incarnation': 22, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.67:6834/2741818214', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6832', 'nonce': 2741818214}, {'type': 'v1', 'addr': '172.21.15.67:6834', 'nonce': 2741818214}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5853, 'qdb_cluster': [5853]} 2024-03-24T00:40:50.507 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-24T00:40:50.507 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-24T00:40:50.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.613+0000 7f6411d5b640 1 Processor -- start 2024-03-24T00:40:50.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.613+0000 7f6411d5b640 1 -- start start 2024-03-24T00:40:50.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.614+0000 7f6411d5b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f640c0b74d0 0x7f640c0d3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:50.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.614+0000 7f6411d5b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f640c0b9cc0 con 0x7f640c15e0f0 2024-03-24T00:40:50.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.614+0000 7f6411d5b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f640c0d2230 con 0x7f640c0d32d0 2024-03-24T00:40:50.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.614+0000 7f6411d5b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f640c0d2510 con 0x7f640c0b74d0 2024-03-24T00:40:50.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.614+0000 7f640b7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f640c15e0f0 0x7f640c0b95b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39666/0 (socket says 172.21.15.67:39666) 2024-03-24T00:40:50.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.614+0000 7f640b7fe640 1 -- 172.21.15.67:0/900536607 learned_addr learned my addr 172.21.15.67:0/900536607 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:40:50.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.614+0000 7f640bfff640 1 --2- 172.21.15.67:0/900536607 >> v2:172.21.15.72:3300/0 conn(0x7f640c0b74d0 0x7f640c0d3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:50.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.615+0000 7f640a7fc640 1 -- 172.21.15.67:0/900536607 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3975535472 0 0) 0x7f640c0b9cc0 con 0x7f640c15e0f0 2024-03-24T00:40:50.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.615+0000 7f640a7fc640 1 -- 172.21.15.67:0/900536607 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63f4003520 con 0x7f640c15e0f0 2024-03-24T00:40:50.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.615+0000 7f640a7fc640 1 -- 172.21.15.67:0/900536607 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2423398057 0 0) 0x7f640c0d2230 con 0x7f640c0d32d0 2024-03-24T00:40:50.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.615+0000 7f640a7fc640 1 -- 172.21.15.67:0/900536607 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f640c0b9cc0 con 0x7f640c0d32d0 2024-03-24T00:40:50.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.615+0000 7f640a7fc640 1 -- 172.21.15.67:0/900536607 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2930109746 0 0) 0x7f63f4003520 con 0x7f640c15e0f0 2024-03-24T00:40:50.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.615+0000 7f640a7fc640 1 -- 172.21.15.67:0/900536607 >> v2:172.21.15.72:3300/0 conn(0x7f640c0b74d0 msgr2=0x7f640c0d3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:50.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.615+0000 7f640a7fc640 1 --2- 172.21.15.67:0/900536607 >> v2:172.21.15.72:3300/0 conn(0x7f640c0b74d0 0x7f640c0d3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:50.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.615+0000 7f640a7fc640 1 -- 172.21.15.67:0/900536607 >> v1:172.21.15.72:6789/0 conn(0x7f640c0d32d0 legacy=0x7f640c0b6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:50.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.615+0000 7f640a7fc640 1 -- 172.21.15.67:0/900536607 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f640c0d27f0 con 0x7f640c15e0f0 2024-03-24T00:40:50.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.615+0000 7f640a7fc640 1 -- 172.21.15.67:0/900536607 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f63fc001690 con 0x7f640c15e0f0 2024-03-24T00:40:50.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.615+0000 7f640a7fc640 1 -- 172.21.15.67:0/900536607 >> v1:172.21.15.67:6789/0 conn(0x7f640c15e0f0 legacy=0x7f640c0b95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:50.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.615+0000 7f640a7fc640 1 --2- 172.21.15.67:0/900536607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63f4003b40 0x7f63f4003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:50.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.615+0000 7f640a7fc640 1 --2- 172.21.15.67:0/900536607 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63f4004800 0x7f63f4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:50.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.615+0000 7f640a7fc640 1 --2- 172.21.15.67:0/900536607 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f63f40051c0 0x7f63f400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:50.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.615+0000 7f640a7fc640 1 -- 172.21.15.67:0/900536607 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f640c0d2510 con 0x7f63f4003b40 2024-03-24T00:40:50.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.615+0000 7f640a7fc640 1 -- 172.21.15.67:0/900536607 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f400c040 con 0x7f63f4004800 2024-03-24T00:40:50.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.615+0000 7f640a7fc640 1 -- 172.21.15.67:0/900536607 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63f400c240 con 0x7f63f40051c0 2024-03-24T00:40:50.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.615+0000 7f640b7fe640 1 --2- 172.21.15.67:0/900536607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63f4003b40 0x7f63f4003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:50.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.615+0000 7f640affd640 1 --2- 172.21.15.67:0/900536607 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63f4004800 0x7f63f4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:50.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.615+0000 7f640bfff640 1 --2- 172.21.15.67:0/900536607 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f63f40051c0 0x7f63f400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:50.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.615+0000 7f640b7fe640 1 -- 172.21.15.67:0/900536607 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f63f40051c0 msgr2=0x7f63f400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:50.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.615+0000 7f640b7fe640 1 --2- 172.21.15.67:0/900536607 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f63f40051c0 0x7f63f400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:50.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.615+0000 7f640b7fe640 1 -- 172.21.15.67:0/900536607 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63f4004800 msgr2=0x7f63f4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:50.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.615+0000 7f640b7fe640 1 --2- 172.21.15.67:0/900536607 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63f4004800 0x7f63f4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:50.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.615+0000 7f640b7fe640 1 -- 172.21.15.67:0/900536607 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f4003520 con 0x7f63f4003b40 2024-03-24T00:40:50.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.615+0000 7f640bfff640 1 -- 172.21.15.67:0/900536607 reap_dead start 2024-03-24T00:40:50.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.615+0000 7f640affd640 1 --2- 172.21.15.67:0/900536607 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63f4004800 0x7f63f4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:40:50.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.616+0000 7f640b7fe640 1 --2- 172.21.15.67:0/900536607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63f4003b40 0x7f63f4003f40 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f63fc00c210 tx=0x7f63f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe88a7720133942d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:50.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.616+0000 7f640a7fc640 1 -- 172.21.15.67:0/900536607 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc007ce0 con 0x7f63f4003b40 2024-03-24T00:40:50.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.616+0000 7f640a7fc640 1 -- 172.21.15.67:0/900536607 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc030070 con 0x7f63f4003b40 2024-03-24T00:40:50.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.616+0000 7f640a7fc640 1 -- 172.21.15.67:0/900536607 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc005d30 con 0x7f63f4003b40 2024-03-24T00:40:50.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.617+0000 7f6411d5b640 1 -- 172.21.15.67:0/900536607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63f4003b40 msgr2=0x7f63f4003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:50.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.617+0000 7f6411d5b640 1 --2- 172.21.15.67:0/900536607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63f4003b40 0x7f63f4003f40 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f63fc00c210 tx=0x7f63f4005830 comp rx=0 tx=0).stop 2024-03-24T00:40:50.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.617+0000 7f6411d5b640 1 -- 172.21.15.67:0/900536607 shutdown_connections 2024-03-24T00:40:50.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.617+0000 7f6411d5b640 1 --2- 172.21.15.67:0/900536607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63f4003b40 0x7f63f4003f40 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:50.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.617+0000 7f6411d5b640 1 -- 172.21.15.67:0/900536607 >> 172.21.15.67:0/900536607 conn(0x7f640c072b60 msgr2=0x7f640c072f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:50.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.617+0000 7f6411d5b640 1 -- 172.21.15.67:0/900536607 shutdown_connections 2024-03-24T00:40:50.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.617+0000 7f6411d5b640 1 -- 172.21.15.67:0/900536607 wait complete. 2024-03-24T00:40:50.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.618+0000 7f6411d5b640 1 Processor -- start 2024-03-24T00:40:50.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.618+0000 7f6411d5b640 1 -- start start 2024-03-24T00:40:50.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.619+0000 7f6411d5b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63f4003b40 0x7f640c15d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:50.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.619+0000 7f6411d5b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f640c0ab0c0 0x7f640c0ab540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:50.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.619+0000 7f6411d5b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f640c0ccb70 0x7f640c0cd020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:50.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.619+0000 7f6411d5b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f640c0b31d0 con 0x7f640c0ccb70 2024-03-24T00:40:50.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.619+0000 7f6411d5b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f640c0d2db0 con 0x7f63f4003b40 2024-03-24T00:40:50.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.619+0000 7f6411d5b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f640c0b3350 con 0x7f640c0ab0c0 2024-03-24T00:40:50.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.619+0000 7f640bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f640c0ccb70 0x7f640c0cd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:50.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.619+0000 7f640bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f640c0ccb70 0x7f640c0cd020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40912/0 (socket says 172.21.15.67:40912) 2024-03-24T00:40:50.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.619+0000 7f640bfff640 1 -- 172.21.15.67:0/221867076 learned_addr learned my addr 172.21.15.67:0/221867076 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:50.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.619+0000 7f640b7fe640 1 --2- 172.21.15.67:0/221867076 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63f4003b40 0x7f640c15d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:50.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.619+0000 7f640affd640 1 --2- 172.21.15.67:0/221867076 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f640c0ab0c0 0x7f640c0ab540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:50.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.619+0000 7f640bfff640 1 -- 172.21.15.67:0/221867076 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f640c0ab0c0 msgr2=0x7f640c0ab540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:50.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.619+0000 7f640bfff640 1 --2- 172.21.15.67:0/221867076 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f640c0ab0c0 0x7f640c0ab540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:50.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.619+0000 7f640bfff640 1 -- 172.21.15.67:0/221867076 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63f4003b40 msgr2=0x7f640c15d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:50.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.619+0000 7f640bfff640 1 --2- 172.21.15.67:0/221867076 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63f4003b40 0x7f640c15d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:50.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.619+0000 7f640bfff640 1 -- 172.21.15.67:0/221867076 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f640c0cd910 con 0x7f640c0ccb70 2024-03-24T00:40:50.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.620+0000 7f640bfff640 1 --2- 172.21.15.67:0/221867076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f640c0ccb70 0x7f640c0cd020 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f640c161ad0 tx=0x7f64040115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:50.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.620+0000 7f63f3fff640 1 -- 172.21.15.67:0/221867076 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6404011e40 con 0x7f640c0ccb70 2024-03-24T00:40:50.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.620+0000 7f63f3fff640 1 -- 172.21.15.67:0/221867076 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f640401b070 con 0x7f640c0ccb70 2024-03-24T00:40:50.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.620+0000 7f63f3fff640 1 -- 172.21.15.67:0/221867076 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6404003990 con 0x7f640c0ccb70 2024-03-24T00:40:50.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.620+0000 7f6411d5b640 1 -- 172.21.15.67:0/221867076 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f640c15f800 con 0x7f640c0ccb70 2024-03-24T00:40:50.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.620+0000 7f6411d5b640 1 -- 172.21.15.67:0/221867076 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f640c0aba80 con 0x7f640c0ccb70 2024-03-24T00:40:50.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.621+0000 7f63f3fff640 1 -- 172.21.15.67:0/221867076 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6404005d10 con 0x7f640c0ccb70 2024-03-24T00:40:50.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.622+0000 7f63f3fff640 1 --2- 172.21.15.67:0/221867076 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f63f806bc00 0x7f63f806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:50.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.622+0000 7f63f3fff640 1 -- 172.21.15.67:0/221867076 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f640408b240 con 0x7f640c0ccb70 2024-03-24T00:40:50.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.622+0000 7f640b7fe640 1 --2- 172.21.15.67:0/221867076 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f63f806bc00 0x7f63f806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:50.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.623+0000 7f6414a29740 1 Processor -- start 2024-03-24T00:40:50.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.623+0000 7f6414a29740 1 -- start start 2024-03-24T00:40:50.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.623+0000 7f640b7fe640 1 --2- 172.21.15.67:0/221867076 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f63f806bc00 0x7f63f806e0c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f63fc034000 tx=0x7f63fc007000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:50.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.623+0000 7f6414a29740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a667506c40 0x55a66752d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:50.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.623+0000 7f6414a29740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55a66752dcb0 0x55a667505c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:50.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.624+0000 7f6414a29740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a667507020 0x55a6675061c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:50.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.624+0000 7f6414a29740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a66746ac20 con 0x55a667506c40 2024-03-24T00:40:50.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.624+0000 7f6414a29740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a667413a40 con 0x55a667507020 2024-03-24T00:40:50.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.624+0000 7f6414a29740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a667466cc0 con 0x55a66752dcb0 2024-03-24T00:40:50.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.624+0000 7f640b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a667506c40 0x55a66752d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:50.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.624+0000 7f640b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a667506c40 0x55a66752d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40928/0 (socket says 172.21.15.67:40928) 2024-03-24T00:40:50.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.624+0000 7f640b7fe640 1 -- 172.21.15.67:0/2123141860 learned_addr learned my addr 172.21.15.67:0/2123141860 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:50.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.624+0000 7f640bfff640 1 --2- 172.21.15.67:0/2123141860 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a667507020 0x55a6675061c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:50.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.624+0000 7f640affd640 1 --2- 172.21.15.67:0/2123141860 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55a66752dcb0 0x55a667505c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:50.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.624+0000 7f640b7fe640 1 -- 172.21.15.67:0/2123141860 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55a66752dcb0 msgr2=0x55a667505c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:50.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.624+0000 7f640b7fe640 1 --2- 172.21.15.67:0/2123141860 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55a66752dcb0 0x55a667505c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:50.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.624+0000 7f640b7fe640 1 -- 172.21.15.67:0/2123141860 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a667507020 msgr2=0x55a6675061c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:50.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.624+0000 7f640b7fe640 1 --2- 172.21.15.67:0/2123141860 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a667507020 0x55a6675061c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:50.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.624+0000 7f640b7fe640 1 -- 172.21.15.67:0/2123141860 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a667435f90 con 0x55a667506c40 2024-03-24T00:40:50.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.624+0000 7f640affd640 1 --2- 172.21.15.67:0/2123141860 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55a66752dcb0 0x55a667505c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:40:50.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.624+0000 7f640b7fe640 1 --2- 172.21.15.67:0/2123141860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a667506c40 0x55a66752d770 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f63fc02c4c0 tx=0x7f63fc02e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fe71e739d649c52 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:50.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.624+0000 7f63f17fa640 1 -- 172.21.15.67:0/2123141860 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0049e0 con 0x55a667506c40 2024-03-24T00:40:50.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.624+0000 7f63f17fa640 1 -- 172.21.15.67:0/2123141860 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc030070 con 0x55a667506c40 2024-03-24T00:40:50.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.624+0000 7f63f17fa640 1 -- 172.21.15.67:0/2123141860 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc004dc0 con 0x55a667506c40 2024-03-24T00:40:50.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.625+0000 7f6414a29740 1 -- 172.21.15.67:0/2123141860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a667506c40 msgr2=0x55a66752d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:50.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.625+0000 7f6414a29740 1 --2- 172.21.15.67:0/2123141860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a667506c40 0x55a66752d770 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f63fc02c4c0 tx=0x7f63fc02e000 comp rx=0 tx=0).stop 2024-03-24T00:40:50.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.625+0000 7f6414a29740 1 -- 172.21.15.67:0/2123141860 shutdown_connections 2024-03-24T00:40:50.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.625+0000 7f6414a29740 1 --2- 172.21.15.67:0/2123141860 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a667507020 0x55a6675061c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:50.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.625+0000 7f6414a29740 1 --2- 172.21.15.67:0/2123141860 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55a66752dcb0 0x55a667505c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:50.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.625+0000 7f6414a29740 1 --2- 172.21.15.67:0/2123141860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a667506c40 0x55a66752d770 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:50.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.625+0000 7f6414a29740 1 -- 172.21.15.67:0/2123141860 >> 172.21.15.67:0/2123141860 conn(0x55a6674cda60 msgr2=0x55a66754b960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:50.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.625+0000 7f6414a29740 1 -- 172.21.15.67:0/2123141860 shutdown_connections 2024-03-24T00:40:50.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.625+0000 7f6414a29740 1 -- 172.21.15.67:0/2123141860 wait complete. 2024-03-24T00:40:50.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.626+0000 7f6414a29740 1 -- start start 2024-03-24T00:40:50.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.626+0000 7f6414a29740 1 Processor -- start 2024-03-24T00:40:50.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.626+0000 7f6414a29740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a667506c40 0x55a6675324c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:50.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.626+0000 7f6414a29740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a667507020 0x55a66755d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:50.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.626+0000 7f6414a29740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55a667534c20 0x55a66755d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:50.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.626+0000 7f6414a29740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a66752d2e0 con 0x55a667506c40 2024-03-24T00:40:50.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.626+0000 7f6414a29740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a667426e90 con 0x55a667507020 2024-03-24T00:40:50.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.626+0000 7f6414a29740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a667526060 con 0x55a667534c20 2024-03-24T00:40:50.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.626+0000 7f6414a29740 10 client.0 _ll_register_callbacks cb 0x55a66717eac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:40:50.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.626+0000 7f640b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a667506c40 0x55a6675324c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:50.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.627+0000 7f640b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a667506c40 0x55a6675324c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40942/0 (socket says 172.21.15.67:40942) 2024-03-24T00:40:50.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.627+0000 7f640b7fe640 1 -- 172.21.15.67:0/1936654170 learned_addr learned my addr 172.21.15.67:0/1936654170 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:50.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.627+0000 7f640bfff640 1 --2- 172.21.15.67:0/1936654170 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55a667534c20 0x55a66755d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:50.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.627+0000 7f640affd640 1 --2- 172.21.15.67:0/1936654170 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a667507020 0x55a66755d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:50.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.627+0000 7f640b7fe640 1 -- 172.21.15.67:0/1936654170 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55a667534c20 msgr2=0x55a66755d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:50.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.627+0000 7f640b7fe640 1 --2- 172.21.15.67:0/1936654170 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55a667534c20 0x55a66755d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:50.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.627+0000 7f640b7fe640 1 -- 172.21.15.67:0/1936654170 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a667507020 msgr2=0x55a66755d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:50.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.627+0000 7f640b7fe640 1 --2- 172.21.15.67:0/1936654170 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a667507020 0x55a66755d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:50.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.627+0000 7f640b7fe640 1 -- 172.21.15.67:0/1936654170 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a6674339d0 con 0x55a667506c40 2024-03-24T00:40:50.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.627+0000 7f640b7fe640 1 --2- 172.21.15.67:0/1936654170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a667506c40 0x55a6675324c0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f63fc00c330 tx=0x7f63fc0019e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f53014d4fa92d235 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:50.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.627+0000 7f63ea7fc640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:40:50.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.627+0000 7f63ea7fc640 1 -- 172.21.15.67:0/1936654170 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc035470 con 0x55a667506c40 2024-03-24T00:40:50.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.627+0000 7f63ea7fc640 1 -- 172.21.15.67:0/1936654170 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(50..50 src has 1..50) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f63fc035610 con 0x55a667506c40 2024-03-24T00:40:50.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.627+0000 7f6414a29740 1 -- 172.21.15.67:0/1936654170 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f63fc030070 con 0x55a667506c40 2024-03-24T00:40:50.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.627+0000 7f63ea7fc640 1 -- 172.21.15.67:0/1936654170 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f63fc062050 con 0x55a667506c40 2024-03-24T00:40:50.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.627+0000 7f6414a29740 10 client.6039 fetch_fsmap learned FSMap version 26 2024-03-24T00:40:50.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.627+0000 7f6414a29740 1 -- 172.21.15.67:0/1936654170 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=2+}) v3 -- 0x55a66717f2d0 con 0x55a667506c40 2024-03-24T00:40:50.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.628+0000 7f63ea7fc640 1 -- 172.21.15.67:0/1936654170 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc030070 con 0x55a667506c40 2024-03-24T00:40:50.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.628+0000 7f63ea7fc640 1 -- 172.21.15.67:0/1936654170 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 26) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f63fc065020 con 0x55a667506c40 2024-03-24T00:40:50.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.628+0000 7f6414a29740 10 client.6039 fetch_fsmap finished waiting for FSMap version 26 2024-03-24T00:40:50.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.628+0000 7f6414a29740 10 client.6039 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-24T00:40:50.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.628+0000 7f6414a29740 20 client.6039 populate_metadata read hostname 'smithi067' 2024-03-24T00:40:50.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.628+0000 7f6414a29740 1 --2- 172.21.15.67:0/1936654170 >> [v2:172.21.15.67:6832/2741818214,v1:172.21.15.67:6834/2741818214] conn(0x55a6675671c0 0x55a6675695b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:50.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.628+0000 7f6414a29740 4 client.6039 mds_command: new command op to 5853 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:40:50.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.628+0000 7f6414a29740 1 -- 172.21.15.67:0/1936654170 --> [v2:172.21.15.67:6832/2741818214,v1:172.21.15.67:6834/2741818214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a667478160 con 0x55a6675671c0 2024-03-24T00:40:50.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.628+0000 7f640affd640 1 --2- 172.21.15.67:0/1936654170 >> [v2:172.21.15.67:6832/2741818214,v1:172.21.15.67:6834/2741818214] conn(0x55a6675671c0 0x55a6675695b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:50.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.628+0000 7f640affd640 1 --2- 172.21.15.67:0/1936654170 >> [v2:172.21.15.67:6832/2741818214,v1:172.21.15.67:6834/2741818214] conn(0x55a6675671c0 0x55a6675695b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=941dcd2490995d86 server_cookie=8a4a7e641838500 in_seq=0 out_seq=0 2024-03-24T00:40:50.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.628+0000 7f63ea7fc640 10 client.6039 ms_handle_connect on v2:172.21.15.67:6832/2741818214 2024-03-24T00:40:50.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.629+0000 7f63ea7fc640 1 -- 172.21.15.67:0/1936654170 <== mds.0 v2:172.21.15.67:6832/2741818214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a667478160 con 0x55a6675671c0 2024-03-24T00:40:50.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.629+0000 7f63ea7fc640 10 client.6039 handle_command_reply: tid=0 2024-03-24T00:40:50.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.629+0000 7f6414a29740 1 client.6039 shutdown 2024-03-24T00:40:50.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.629+0000 7f6414a29740 1 -- 172.21.15.67:0/1936654170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a667506c40 msgr2=0x55a6675324c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:50.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.629+0000 7f6414a29740 1 --2- 172.21.15.67:0/1936654170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a667506c40 0x55a6675324c0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f63fc00c330 tx=0x7f63fc0019e0 comp rx=0 tx=0).stop 2024-03-24T00:40:50.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.629+0000 7f6414a29740 1 -- 172.21.15.67:0/1936654170 shutdown_connections 2024-03-24T00:40:50.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.629+0000 7f6414a29740 1 --2- 172.21.15.67:0/1936654170 >> [v2:172.21.15.67:6832/2741818214,v1:172.21.15.67:6834/2741818214] conn(0x55a6675671c0 0x55a6675695b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:50.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.629+0000 7f6414a29740 1 --2- 172.21.15.67:0/1936654170 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55a667534c20 0x55a66755d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:50.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.629+0000 7f6414a29740 1 --2- 172.21.15.67:0/1936654170 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a667507020 0x55a66755d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:50.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.629+0000 7f63ea7fc640 0 client.6039 ms_handle_reset on v2:172.21.15.67:6832/2741818214 2024-03-24T00:40:50.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.629+0000 7f6414a29740 1 --2- 172.21.15.67:0/1936654170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a667506c40 0x55a6675324c0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:50.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.629+0000 7f6414a29740 1 -- 172.21.15.67:0/1936654170 >> 172.21.15.67:0/1936654170 conn(0x55a6674cda60 msgr2=0x55a66752ca20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:50.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.629+0000 7f6414a29740 1 -- 172.21.15.67:0/1936654170 shutdown_connections 2024-03-24T00:40:50.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.629+0000 7f6414a29740 1 -- 172.21.15.67:0/1936654170 wait complete. 2024-03-24T00:40:50.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.629+0000 7f6414a29740 20 client.6039 trim_cache size 0 max 16384 2024-03-24T00:40:50.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.646+0000 7f6414a29740 1 Processor -- start 2024-03-24T00:40:50.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.646+0000 7f6414a29740 1 -- start start 2024-03-24T00:40:50.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.646+0000 7f6414a29740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a667506c40 0x55a667564e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:50.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.647+0000 7f6414a29740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55a667507020 0x55a667530a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:50.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.647+0000 7f6414a29740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a667534c20 0x55a667530f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:50.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.647+0000 7f6414a29740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f640c200220 con 0x55a667534c20 2024-03-24T00:40:50.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.647+0000 7f6414a29740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f640c1ff620 con 0x55a667506c40 2024-03-24T00:40:50.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.647+0000 7f6414a29740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f640c1ffc20 con 0x55a667507020 2024-03-24T00:40:50.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.647+0000 7f640bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a667534c20 0x55a667530f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:50.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.647+0000 7f640bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a667534c20 0x55a667530f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40956/0 (socket says 172.21.15.67:40956) 2024-03-24T00:40:50.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.647+0000 7f640bfff640 1 -- 172.21.15.67:0/1271297458 learned_addr learned my addr 172.21.15.67:0/1271297458 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:50.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.647+0000 7f640b7fe640 1 --2- 172.21.15.67:0/1271297458 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a667506c40 0x55a667564e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:50.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.647+0000 7f640affd640 1 --2- 172.21.15.67:0/1271297458 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55a667507020 0x55a667530a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:50.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.647+0000 7f640bfff640 1 -- 172.21.15.67:0/1271297458 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55a667507020 msgr2=0x55a667530a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:50.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.647+0000 7f640bfff640 1 --2- 172.21.15.67:0/1271297458 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55a667507020 0x55a667530a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:50.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.647+0000 7f640bfff640 1 -- 172.21.15.67:0/1271297458 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a667506c40 msgr2=0x55a667564e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:50.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.647+0000 7f640bfff640 1 --2- 172.21.15.67:0/1271297458 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a667506c40 0x55a667564e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:50.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.647+0000 7f640bfff640 1 -- 172.21.15.67:0/1271297458 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a6672496b0 con 0x55a667534c20 2024-03-24T00:40:50.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.647+0000 7f640bfff640 1 --2- 172.21.15.67:0/1271297458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a667534c20 0x55a667530f40 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f64040591a0 tx=0x7f640408b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0e4ab7f85de8395 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:50.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.647+0000 7f63f0ff9640 1 -- 172.21.15.67:0/1271297458 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6404089310 con 0x55a667534c20 2024-03-24T00:40:50.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.647+0000 7f63f0ff9640 1 -- 172.21.15.67:0/1271297458 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6404087070 con 0x55a667534c20 2024-03-24T00:40:50.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.647+0000 7f63f0ff9640 1 -- 172.21.15.67:0/1271297458 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6404059ba0 con 0x55a667534c20 2024-03-24T00:40:50.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.648+0000 7f6414a29740 1 -- 172.21.15.67:0/1271297458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a667534c20 msgr2=0x55a667530f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:50.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.648+0000 7f6414a29740 1 --2- 172.21.15.67:0/1271297458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a667534c20 0x55a667530f40 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f64040591a0 tx=0x7f640408b040 comp rx=0 tx=0).stop 2024-03-24T00:40:50.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.648+0000 7f6414a29740 1 -- 172.21.15.67:0/1271297458 shutdown_connections 2024-03-24T00:40:50.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.648+0000 7f6414a29740 1 --2- 172.21.15.67:0/1271297458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a667534c20 0x55a667530f40 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:50.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.648+0000 7f6414a29740 1 --2- 172.21.15.67:0/1271297458 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55a667507020 0x55a667530a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:50.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.648+0000 7f6414a29740 1 --2- 172.21.15.67:0/1271297458 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a667506c40 0x55a667564e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:50.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.648+0000 7f6414a29740 1 -- 172.21.15.67:0/1271297458 >> 172.21.15.67:0/1271297458 conn(0x55a6674cda60 msgr2=0x55a66754b280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:50.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.648+0000 7f6414a29740 1 -- 172.21.15.67:0/1271297458 shutdown_connections 2024-03-24T00:40:50.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.648+0000 7f6414a29740 1 -- 172.21.15.67:0/1271297458 wait complete. 2024-03-24T00:40:50.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.649+0000 7f6414a29740 1 -- start start 2024-03-24T00:40:50.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.649+0000 7f6414a29740 1 Processor -- start 2024-03-24T00:40:50.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.649+0000 7f6414a29740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55a667506c40 0x55a66755a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:50.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.649+0000 7f6414a29740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a667507020 0x55a66755ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:50.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.649+0000 7f6414a29740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a667534c20 0x55a667500400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:50.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.649+0000 7f6414a29740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a66755c970 con 0x55a667534c20 2024-03-24T00:40:50.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.649+0000 7f6414a29740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a66752d2e0 con 0x55a667507020 2024-03-24T00:40:50.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.649+0000 7f6414a29740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a667565490 con 0x55a667506c40 2024-03-24T00:40:50.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.649+0000 7f6414a29740 10 client.0 _ll_register_callbacks cb 0x7f6404055740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:40:50.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.650+0000 7f640bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a667534c20 0x55a667500400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:50.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.650+0000 7f640bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a667534c20 0x55a667500400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40966/0 (socket says 172.21.15.67:40966) 2024-03-24T00:40:50.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.650+0000 7f640bfff640 1 -- 172.21.15.67:0/3917407010 learned_addr learned my addr 172.21.15.67:0/3917407010 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:50.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.650+0000 7f640b7fe640 1 --2- 172.21.15.67:0/3917407010 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55a667506c40 0x55a66755a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:50.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.650+0000 7f640affd640 1 --2- 172.21.15.67:0/3917407010 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a667507020 0x55a66755ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:50.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.650+0000 7f640bfff640 1 -- 172.21.15.67:0/3917407010 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55a667506c40 msgr2=0x55a66755a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:50.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.650+0000 7f640bfff640 1 --2- 172.21.15.67:0/3917407010 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55a667506c40 0x55a66755a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:50.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.650+0000 7f640bfff640 1 -- 172.21.15.67:0/3917407010 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a667507020 msgr2=0x55a66755ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:50.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.650+0000 7f640bfff640 1 --2- 172.21.15.67:0/3917407010 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a667507020 0x55a66755ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:50.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.650+0000 7f640bfff640 1 -- 172.21.15.67:0/3917407010 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a66752c810 con 0x55a667534c20 2024-03-24T00:40:50.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.650+0000 7f640bfff640 1 --2- 172.21.15.67:0/3917407010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a667534c20 0x55a667500400 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f6404089630 tx=0x7f6404055870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57a2e3442f631fcb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:50.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.650+0000 7f63ea7fc640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:40:50.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.650+0000 7f63ea7fc640 1 -- 172.21.15.67:0/3917407010 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f640408b070 con 0x55a667534c20 2024-03-24T00:40:50.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.650+0000 7f63ea7fc640 1 -- 172.21.15.67:0/3917407010 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(50..50 src has 1..50) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f64040711d0 con 0x55a667534c20 2024-03-24T00:40:50.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.650+0000 7f6414a29740 1 -- 172.21.15.67:0/3917407010 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6404087070 con 0x55a667534c20 2024-03-24T00:40:50.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.650+0000 7f63ea7fc640 1 -- 172.21.15.67:0/3917407010 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f640407f050 con 0x55a667534c20 2024-03-24T00:40:50.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.650+0000 7f6414a29740 10 client.6045 fetch_fsmap learned FSMap version 26 2024-03-24T00:40:50.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.650+0000 7f6414a29740 1 -- 172.21.15.67:0/3917407010 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=2+}) v3 -- 0x55a6675246e0 con 0x55a667534c20 2024-03-24T00:40:50.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.650+0000 7f63ea7fc640 1 -- 172.21.15.67:0/3917407010 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6404087070 con 0x55a667534c20 2024-03-24T00:40:50.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.651+0000 7f63ea7fc640 1 -- 172.21.15.67:0/3917407010 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 26) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f6404082020 con 0x55a667534c20 2024-03-24T00:40:50.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.651+0000 7f6414a29740 10 client.6045 fetch_fsmap finished waiting for FSMap version 26 2024-03-24T00:40:50.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.651+0000 7f6414a29740 10 client.6045 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-24T00:40:50.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.651+0000 7f6414a29740 20 client.6045 populate_metadata read hostname 'smithi067' 2024-03-24T00:40:50.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.651+0000 7f6414a29740 1 --2- 172.21.15.67:0/3917407010 >> [v2:172.21.15.67:6832/2741818214,v1:172.21.15.67:6834/2741818214] conn(0x55a667500940 0x55a667500d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:50.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.651+0000 7f6414a29740 4 client.6045 mds_command: new command op to 5853 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:40:50.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.651+0000 7f6414a29740 1 -- 172.21.15.67:0/3917407010 --> [v2:172.21.15.67:6832/2741818214,v1:172.21.15.67:6834/2741818214] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55a6674724b0 con 0x55a667500940 2024-03-24T00:40:50.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.651+0000 7f640b7fe640 1 --2- 172.21.15.67:0/3917407010 >> [v2:172.21.15.67:6832/2741818214,v1:172.21.15.67:6834/2741818214] conn(0x55a667500940 0x55a667500d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:50.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.651+0000 7f640b7fe640 1 --2- 172.21.15.67:0/3917407010 >> [v2:172.21.15.67:6832/2741818214,v1:172.21.15.67:6834/2741818214] conn(0x55a667500940 0x55a667500d20 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d85f347c43644fec server_cookie=795c5d2858606bf0 in_seq=0 out_seq=0 2024-03-24T00:40:50.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.651+0000 7f63ea7fc640 10 client.6045 ms_handle_connect on v2:172.21.15.67:6832/2741818214 2024-03-24T00:40:50.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.651+0000 7f63ea7fc640 1 -- 172.21.15.67:0/3917407010 <== mds.0 v2:172.21.15.67:6832/2741818214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55a6674724b0 con 0x55a667500940 2024-03-24T00:40:50.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.651+0000 7f63ea7fc640 10 client.6045 handle_command_reply: tid=0 2024-03-24T00:40:50.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.651+0000 7f6414a29740 1 client.6045 shutdown 2024-03-24T00:40:50.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.651+0000 7f6414a29740 1 -- 172.21.15.67:0/3917407010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a667534c20 msgr2=0x55a667500400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:50.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.651+0000 7f6414a29740 1 --2- 172.21.15.67:0/3917407010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a667534c20 0x55a667500400 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f6404089630 tx=0x7f6404055870 comp rx=0 tx=0).stop 2024-03-24T00:40:50.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.651+0000 7f6414a29740 1 -- 172.21.15.67:0/3917407010 shutdown_connections 2024-03-24T00:40:50.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.651+0000 7f6414a29740 1 --2- 172.21.15.67:0/3917407010 >> [v2:172.21.15.67:6832/2741818214,v1:172.21.15.67:6834/2741818214] conn(0x55a667500940 0x55a667500d20 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:50.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.651+0000 7f6414a29740 1 --2- 172.21.15.67:0/3917407010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a667534c20 0x55a667500400 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:50.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.651+0000 7f6414a29740 1 --2- 172.21.15.67:0/3917407010 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a667507020 0x55a66755ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:50.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.651+0000 7f63ea7fc640 0 client.6045 ms_handle_reset on v2:172.21.15.67:6832/2741818214 2024-03-24T00:40:50.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.651+0000 7f6414a29740 1 --2- 172.21.15.67:0/3917407010 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55a667506c40 0x55a66755a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:50.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.651+0000 7f6414a29740 1 -- 172.21.15.67:0/3917407010 >> 172.21.15.67:0/3917407010 conn(0x55a6674cda60 msgr2=0x55a66752be10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:50.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.651+0000 7f6414a29740 1 -- 172.21.15.67:0/3917407010 shutdown_connections 2024-03-24T00:40:50.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.651+0000 7f6414a29740 1 -- 172.21.15.67:0/3917407010 wait complete. 2024-03-24T00:40:50.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.652+0000 7f6414a29740 20 client.6045 trim_cache size 0 max 16384 2024-03-24T00:40:50.658 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:40:50.658 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "de1d0178-808e-49ca-a986-e297e523b976", 2024-03-24T00:40:50.658 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-03-24T00:40:50.658 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 5853, 2024-03-24T00:40:50.658 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-03-24T00:40:50.658 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-03-24T00:40:50.658 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-03-24T00:40:50.658 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 10.360995945000001, 2024-03-24T00:40:50.659 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 26, 2024-03-24T00:40:50.659 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 50, 2024-03-24T00:40:50.659 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 50, 2024-03-24T00:40:50.659 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 14.712053396 2024-03-24T00:40:50.659 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:40:50.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.652+0000 7f6411d5b640 1 -- 172.21.15.67:0/221867076 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f63f806bc00 msgr2=0x7f63f806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:50.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.652+0000 7f6411d5b640 1 --2- 172.21.15.67:0/221867076 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f63f806bc00 0x7f63f806e0c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f63fc034000 tx=0x7f63fc007000 comp rx=0 tx=0).stop 2024-03-24T00:40:50.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.652+0000 7f6411d5b640 1 -- 172.21.15.67:0/221867076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f640c0ccb70 msgr2=0x7f640c0cd020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:50.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.652+0000 7f6411d5b640 1 --2- 172.21.15.67:0/221867076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f640c0ccb70 0x7f640c0cd020 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f640c161ad0 tx=0x7f64040115a0 comp rx=0 tx=0).stop 2024-03-24T00:40:50.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.652+0000 7f6411d5b640 1 -- 172.21.15.67:0/221867076 shutdown_connections 2024-03-24T00:40:50.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.652+0000 7f6411d5b640 1 --2- 172.21.15.67:0/221867076 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f63f806bc00 0x7f63f806e0c0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:50.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.652+0000 7f6411d5b640 1 --2- 172.21.15.67:0/221867076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f640c0ccb70 0x7f640c0cd020 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:50.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.652+0000 7f6411d5b640 1 --2- 172.21.15.67:0/221867076 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f640c0ab0c0 0x7f640c0ab540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:50.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.652+0000 7f6411d5b640 1 --2- 172.21.15.67:0/221867076 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63f4003b40 0x7f640c15d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:50.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.652+0000 7f6411d5b640 1 -- 172.21.15.67:0/221867076 >> 172.21.15.67:0/221867076 conn(0x7f640c072b60 msgr2=0x7f640c0d06a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:50.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.652+0000 7f6411d5b640 1 -- 172.21.15.67:0/221867076 shutdown_connections 2024-03-24T00:40:50.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.652+0000 7f6411d5b640 1 -- 172.21.15.67:0/221867076 wait complete. 2024-03-24T00:40:50.665 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T00:40:50.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.769+0000 7fdde8b1b640 1 Processor -- start 2024-03-24T00:40:50.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.769+0000 7fdde8b1b640 1 -- start start 2024-03-24T00:40:50.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.770+0000 7fdde8b1b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdde415dec0 0x7fdde40ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:50.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.770+0000 7fdde8b1b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdde40cf640 con 0x7fdde40d0ef0 2024-03-24T00:40:50.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.770+0000 7fdde8b1b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdde40cf080 con 0x7fdde415e2a0 2024-03-24T00:40:50.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.770+0000 7fdde8b1b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdde40cee30 con 0x7fdde415dec0 2024-03-24T00:40:50.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.770+0000 7fdde259c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdde415dec0 0x7fdde40ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:50.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.770+0000 7fdde2d9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fdde40d0ef0 0x7fdde40d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39672/0 (socket says 172.21.15.67:39672) 2024-03-24T00:40:50.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.770+0000 7fdde2d9d640 1 -- 172.21.15.67:0/1660339044 learned_addr learned my addr 172.21.15.67:0/1660339044 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:40:50.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.770+0000 7fdde259c640 1 -- 172.21.15.67:0/1660339044 >> v1:172.21.15.72:6789/0 conn(0x7fdde415e2a0 legacy=0x7fdde40d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:50.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.770+0000 7fdde259c640 1 -- 172.21.15.67:0/1660339044 >> v1:172.21.15.67:6789/0 conn(0x7fdde40d0ef0 legacy=0x7fdde40d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:50.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.770+0000 7fdde259c640 1 -- 172.21.15.67:0/1660339044 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde40cf360 con 0x7fdde415dec0 2024-03-24T00:40:50.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.770+0000 7fdde159a640 1 -- 172.21.15.67:0/1660339044 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3081303028 0 0) 0x7fdddc001490 con 0x7fdde40d0ef0 2024-03-24T00:40:50.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.771+0000 7fdde259c640 1 --2- 172.21.15.67:0/1660339044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdde415dec0 0x7fdde40ce630 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fdde40cefb0 tx=0x7fddd402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=548eac1bf351716d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:50.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.771+0000 7fdde159a640 1 -- 172.21.15.67:0/1660339044 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd40315a0 con 0x7fdde415dec0 2024-03-24T00:40:50.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.771+0000 7fdde159a640 1 -- 172.21.15.67:0/1660339044 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddd4037070 con 0x7fdde415dec0 2024-03-24T00:40:50.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.771+0000 7fdde159a640 1 -- 172.21.15.67:0/1660339044 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd4031960 con 0x7fdde415dec0 2024-03-24T00:40:50.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.772+0000 7fdde8b1b640 1 -- 172.21.15.67:0/1660339044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdde415dec0 msgr2=0x7fdde40ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:50.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.772+0000 7fdde8b1b640 1 --2- 172.21.15.67:0/1660339044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdde415dec0 0x7fdde40ce630 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fdde40cefb0 tx=0x7fddd402b840 comp rx=0 tx=0).stop 2024-03-24T00:40:50.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.772+0000 7fdde8b1b640 1 -- 172.21.15.67:0/1660339044 shutdown_connections 2024-03-24T00:40:50.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.772+0000 7fdde8b1b640 1 --2- 172.21.15.67:0/1660339044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdde415dec0 0x7fdde40ce630 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:50.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.772+0000 7fdde8b1b640 1 -- 172.21.15.67:0/1660339044 >> 172.21.15.67:0/1660339044 conn(0x7fdde406c810 msgr2=0x7fdde406cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:50.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.772+0000 7fdde8b1b640 1 -- 172.21.15.67:0/1660339044 shutdown_connections 2024-03-24T00:40:50.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.772+0000 7fdde8b1b640 1 -- 172.21.15.67:0/1660339044 wait complete. 2024-03-24T00:40:50.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.773+0000 7fdde8b1b640 1 Processor -- start 2024-03-24T00:40:50.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.773+0000 7fdde8b1b640 1 -- start start 2024-03-24T00:40:50.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.774+0000 7fdde8b1b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdde40d0ef0 0x7fdde41e8530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:50.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.774+0000 7fdde8b1b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdde415dec0 0x7fdde415c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:50.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.774+0000 7fdde8b1b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdde415e2a0 0x7fdde40aa720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:50.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.774+0000 7fdde8b1b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde4162a40 con 0x7fdde40d0ef0 2024-03-24T00:40:50.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.774+0000 7fdde8b1b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde41628c0 con 0x7fdde415e2a0 2024-03-24T00:40:50.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.774+0000 7fdde8b1b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdde40d2120 con 0x7fdde415dec0 2024-03-24T00:40:50.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.774+0000 7fdde259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdde40d0ef0 0x7fdde41e8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:50.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.774+0000 7fdde259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdde40d0ef0 0x7fdde41e8530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40992/0 (socket says 172.21.15.67:40992) 2024-03-24T00:40:50.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.774+0000 7fdde259c640 1 -- 172.21.15.67:0/1682344724 learned_addr learned my addr 172.21.15.67:0/1682344724 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:50.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.774+0000 7fdde2d9d640 1 --2- 172.21.15.67:0/1682344724 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdde415e2a0 0x7fdde40aa720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:50.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.774+0000 7fdde1d9b640 1 --2- 172.21.15.67:0/1682344724 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdde415dec0 0x7fdde415c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:50.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.774+0000 7fdde259c640 1 -- 172.21.15.67:0/1682344724 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdde415dec0 msgr2=0x7fdde415c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:50.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.774+0000 7fdde259c640 1 --2- 172.21.15.67:0/1682344724 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdde415dec0 0x7fdde415c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:50.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.774+0000 7fdde259c640 1 -- 172.21.15.67:0/1682344724 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdde415e2a0 msgr2=0x7fdde40aa720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:50.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.774+0000 7fdde259c640 1 --2- 172.21.15.67:0/1682344724 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdde415e2a0 0x7fdde40aa720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:50.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.774+0000 7fdde259c640 1 -- 172.21.15.67:0/1682344724 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde41e8bf0 con 0x7fdde40d0ef0 2024-03-24T00:40:50.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.775+0000 7fdde259c640 1 --2- 172.21.15.67:0/1682344724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdde40d0ef0 0x7fdde41e8530 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fddd4000c00 tx=0x7fddd40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:50.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.775+0000 7fddd2ffd640 1 -- 172.21.15.67:0/1682344724 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd4032840 con 0x7fdde40d0ef0 2024-03-24T00:40:50.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.775+0000 7fddd2ffd640 1 -- 172.21.15.67:0/1682344724 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddd4032a00 con 0x7fdde40d0ef0 2024-03-24T00:40:50.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.775+0000 7fdde8b1b640 1 -- 172.21.15.67:0/1682344724 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdde41e8ed0 con 0x7fdde40d0ef0 2024-03-24T00:40:50.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.775+0000 7fddd2ffd640 1 -- 172.21.15.67:0/1682344724 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd4032df0 con 0x7fdde40d0ef0 2024-03-24T00:40:50.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.775+0000 7fdde8b1b640 1 -- 172.21.15.67:0/1682344724 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdde41e9490 con 0x7fdde40d0ef0 2024-03-24T00:40:50.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.776+0000 7fdde8b1b640 1 -- 172.21.15.67:0/1682344724 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdde404f400 con 0x7fdde40d0ef0 2024-03-24T00:40:50.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.776+0000 7fddd2ffd640 1 -- 172.21.15.67:0/1682344724 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fddd403b3f0 con 0x7fdde40d0ef0 2024-03-24T00:40:50.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.777+0000 7fddd2ffd640 1 --2- 172.21.15.67:0/1682344724 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fddc406bbd0 0x7fddc406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:50.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.777+0000 7fddd2ffd640 1 -- 172.21.15.67:0/1682344724 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fddd402f030 con 0x7fdde40d0ef0 2024-03-24T00:40:50.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.777+0000 7fdde1d9b640 1 --2- 172.21.15.67:0/1682344724 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fddc406bbd0 0x7fddc406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:50.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.777+0000 7fdde1d9b640 1 --2- 172.21.15.67:0/1682344724 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fddc406bbd0 0x7fddc406e090 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fdde40b5b90 tx=0x7fddd8009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:50.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:50.781+0000 7fddd2ffd640 1 -- 172.21.15.67:0/1682344724 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fddd4040070 con 0x7fdde40d0ef0 2024-03-24T00:40:51.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:51.003+0000 7fdde8b1b640 1 -- 172.21.15.67:0/1682344724 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fdde40d1aa0 con 0x7fdde40d0ef0 2024-03-24T00:40:51.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:51.003+0000 7fddd2ffd640 1 -- 172.21.15.67:0/1682344724 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v26) v1 ==== 53+0+73 (secure 0 0 0) 0x7fddd407add0 con 0x7fdde40d0ef0 2024-03-24T00:40:51.004 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-24T00:40:51.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:51.005+0000 7fdde8b1b640 1 -- 172.21.15.67:0/1682344724 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fddc406bbd0 msgr2=0x7fddc406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:51.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:51.005+0000 7fdde8b1b640 1 --2- 172.21.15.67:0/1682344724 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fddc406bbd0 0x7fddc406e090 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fdde40b5b90 tx=0x7fddd8009060 comp rx=0 tx=0).stop 2024-03-24T00:40:51.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:51.005+0000 7fdde8b1b640 1 -- 172.21.15.67:0/1682344724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdde40d0ef0 msgr2=0x7fdde41e8530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:51.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:51.005+0000 7fdde8b1b640 1 --2- 172.21.15.67:0/1682344724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdde40d0ef0 0x7fdde41e8530 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fddd4000c00 tx=0x7fddd40046c0 comp rx=0 tx=0).stop 2024-03-24T00:40:51.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:51.006+0000 7fdde8b1b640 1 -- 172.21.15.67:0/1682344724 shutdown_connections 2024-03-24T00:40:51.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:51.006+0000 7fdde8b1b640 1 --2- 172.21.15.67:0/1682344724 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fddc406bbd0 0x7fddc406e090 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:51.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:51.006+0000 7fdde8b1b640 1 --2- 172.21.15.67:0/1682344724 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdde415e2a0 0x7fdde40aa720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:51.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:51.006+0000 7fdde8b1b640 1 --2- 172.21.15.67:0/1682344724 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdde415dec0 0x7fdde415c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:51.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:51.006+0000 7fdde8b1b640 1 --2- 172.21.15.67:0/1682344724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdde40d0ef0 0x7fdde41e8530 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:51.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:51.006+0000 7fdde8b1b640 1 -- 172.21.15.67:0/1682344724 >> 172.21.15.67:0/1682344724 conn(0x7fdde406c810 msgr2=0x7fdde40d32f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:51.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:51.006+0000 7fdde8b1b640 1 -- 172.21.15.67:0/1682344724 shutdown_connections 2024-03-24T00:40:51.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:51.006+0000 7fdde8b1b640 1 -- 172.21.15.67:0/1682344724 wait complete. 2024-03-24T00:40:51.021 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-24T00:40:51.021 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T00:40:51.021 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-03-24T00:40:51.021 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-24T00:40:51.021 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T00:40:51.021 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-24T00:40:51.021 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-24T00:40:51.021 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T00:40:51.021 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:40:51.021 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:40:51.022 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-03-24T00:40:51.036 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T00:40:51.037 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T00:40:51.037 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T00:40:51.037 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:40:51.037 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-03-24T00:40:51.037 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:40:51.037 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:40:51.037 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-03-24T00:40:51.037 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-03-24T00:40:51.037 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:40:51.037 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-03-24T00:40:51.037 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-03-24T00:40:51.037 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:40:51.038 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-03-24T00:40:51.038 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T00:40:51.038 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-03-24T00:40:51.038 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-24T00:40:51.038 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 42462sec preferred_lft 42462sec 2024-03-24T00:40:51.038 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-03-24T00:40:51.038 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:40:51.038 INFO:teuthology.orchestra.run.smithi067.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T00:40:51.038 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 96:cf:26:c8:bf:62 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:40:51.038 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-24T00:40:51.038 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:40:51.038 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::7c6a:5fff:feaa:d41/64 scope link 2024-03-24T00:40:51.038 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:40:51.039 INFO:teuthology.orchestra.run.smithi067.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T00:40:51.039 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ea:22:4b:d5:48:3d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:40:51.039 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::e822:4bff:fed5:483d/64 scope link 2024-03-24T00:40:51.039 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:40:51.039 INFO:teuthology.orchestra.run.smithi067.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T00:40:51.039 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 96:cf:26:c8:bf:62 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:40:51.039 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::94cf:26ff:fec8:bf62/64 scope link 2024-03-24T00:40:51.039 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:40:51.039 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:40:51.039 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T00:40:51.101 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-03-24T00:40:51.103 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:40:51.103 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-03-24T00:40:51.158 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-03-24T00:40:51.158 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T00:40:51.158 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-24T00:40:51.159 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-24T00:40:51.159 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T00:40:51.159 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:40:51.159 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-03-24T00:40:51.159 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-03-24T00:40:51.159 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-24T00:40:51.159 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-24T00:40:51.159 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-24T00:40:51.159 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-03-24T00:40:51.253 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:40:51.253 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-03-24T00:40:51.268 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-24T00:40:51.268 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-24T00:40:51.268 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:40:51.268 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:40:51.268 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-03-24T00:40:51.336 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:40:51.336 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-24T00:40:51.390 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:40:51.391 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:40:51.391 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:40:51.445 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-24T00:40:51.445 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:40:51.445 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:40:51.528 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:40:51.529 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T00:40:51.529 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-03-24T00:40:51.537 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:40:51.537 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T00:40:51.537 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-03-24T00:40:51.537 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.72:3300,172.21.15.72:3301" 2024-03-24T00:40:51.537 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-24T00:40:51.542 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:40:51.542 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T00:40:51.542 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-03-24T00:40:51.542 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.72:3300,172.21.15.72:3301" 2024-03-24T00:40:51.542 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-24T00:40:51.542 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:6789,172.21.15.72:6789,172.21.15.72:6790" 2024-03-24T00:40:51.542 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.67:6789,172.21.15.72:6789,172.21.15.72:6790:/ 2024-03-24T00:40:51.542 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=de1d0178-808e-49ca-a986-e297e523b976" will pass to kernel 2024-03-24T00:40:51.543 INFO:teuthology.orchestra.run.smithi067.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-24T00:40:51.543 INFO:teuthology.orchestra.run.smithi067.stdout: You just mounted a file system that supports labels which does not 2024-03-24T00:40:51.543 INFO:teuthology.orchestra.run.smithi067.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-24T00:40:51.543 INFO:teuthology.orchestra.run.smithi067.stdout: applications will generate AVC messages and not be allowed access to 2024-03-24T00:40:51.543 INFO:teuthology.orchestra.run.smithi067.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-24T00:40:51.544 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-24T00:40:51.544 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:40:51.544 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:40:51.612 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:40:51.612 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T00:40:51.671 INFO:teuthology.orchestra.run.smithi067.stdout:client5453 2024-03-24T00:40:51.671 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:40:51.671 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:40:51.727 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:40:51.727 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir parent_a) 2024-03-24T00:40:51.787 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:40:51.788 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch parent_a/alpha) 2024-03-24T00:40:51.843 DEBUG:teuthology.orchestra.run.smithi067:> stdin-killer -- python3 -c ' 2024-03-24T00:40:51.843 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-03-24T00:40:51.843 DEBUG:teuthology.orchestra.run.smithi067:> import stat 2024-03-24T00:40:51.843 DEBUG:teuthology.orchestra.run.smithi067:> 2024-03-24T00:40:51.843 DEBUG:teuthology.orchestra.run.smithi067:> print(os.stat("/home/ubuntu/cephtest/mnt.0/parent_a/alpha").st_ino) 2024-03-24T00:40:51.843 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-03-24T00:40:51.954 INFO:teuthology.orchestra.run.smithi067.stdout:1099511627777 2024-03-24T00:40:51.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:40:51.964 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:40:52.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.104+0000 7f9528412640 1 Processor -- start 2024-03-24T00:40:52.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.104+0000 7f9528412640 1 -- start start 2024-03-24T00:40:52.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.105+0000 7f9528412640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f95200b18f0 0x7f95200b1d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:52.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.105+0000 7f9528412640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95200b2590 con 0x7f952006ca40 2024-03-24T00:40:52.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.105+0000 7f9528412640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95200b2870 con 0x7f95200d0440 2024-03-24T00:40:52.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.105+0000 7f9528412640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95200b2b50 con 0x7f95200b18f0 2024-03-24T00:40:52.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.105+0000 7f95261ae640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f952006ca40 0x7f952006ce20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39686/0 (socket says 172.21.15.67:39686) 2024-03-24T00:40:52.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.105+0000 7f95261ae640 1 -- 172.21.15.67:0/4272548103 learned_addr learned my addr 172.21.15.67:0/4272548103 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:40:52.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.106+0000 7f95251ac640 1 -- 172.21.15.67:0/4272548103 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3201463592 0 0) 0x7f95200b2590 con 0x7f952006ca40 2024-03-24T00:40:52.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.106+0000 7f95259ad640 1 --2- 172.21.15.67:0/4272548103 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f95200b18f0 0x7f95200b1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:52.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.106+0000 7f95251ac640 1 -- 172.21.15.67:0/4272548103 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f950c003520 con 0x7f952006ca40 2024-03-24T00:40:52.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.106+0000 7f95251ac640 1 -- 172.21.15.67:0/4272548103 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3557601474 0 0) 0x7f950c003520 con 0x7f952006ca40 2024-03-24T00:40:52.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.106+0000 7f95251ac640 1 -- 172.21.15.67:0/4272548103 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f95200b18f0 msgr2=0x7f95200b1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:52.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.106+0000 7f95251ac640 1 --2- 172.21.15.67:0/4272548103 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f95200b18f0 0x7f95200b1d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.106+0000 7f95251ac640 1 -- 172.21.15.67:0/4272548103 >> v1:172.21.15.72:6789/0 conn(0x7f95200d0440 legacy=0x7f95200d08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:52.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.106+0000 7f95251ac640 1 -- 172.21.15.67:0/4272548103 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95200b2e30 con 0x7f952006ca40 2024-03-24T00:40:52.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.106+0000 7f95251ac640 1 -- 172.21.15.67:0/4272548103 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f951c001670 con 0x7f952006ca40 2024-03-24T00:40:52.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.106+0000 7f95251ac640 1 -- 172.21.15.67:0/4272548103 >> v1:172.21.15.67:6789/0 conn(0x7f952006ca40 legacy=0x7f952006ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:52.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.106+0000 7f95251ac640 1 --2- 172.21.15.67:0/4272548103 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f95200b18f0 0x7f950c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:52.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.106+0000 7f95251ac640 1 --2- 172.21.15.67:0/4272548103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f950c003f20 0x7f950c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:52.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.106+0000 7f95251ac640 1 --2- 172.21.15.67:0/4272548103 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f950c004d40 0x7f950c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:52.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.106+0000 7f95251ac640 1 -- 172.21.15.67:0/4272548103 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95200b2b50 con 0x7f950c003f20 2024-03-24T00:40:52.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.106+0000 7f95251ac640 1 -- 172.21.15.67:0/4272548103 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950c00ba20 con 0x7f950c004d40 2024-03-24T00:40:52.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.106+0000 7f95251ac640 1 -- 172.21.15.67:0/4272548103 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f950c00bc20 con 0x7f95200b18f0 2024-03-24T00:40:52.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.106+0000 7f95259ad640 1 --2- 172.21.15.67:0/4272548103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f950c003f20 0x7f950c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:52.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.106+0000 7f95261ae640 1 --2- 172.21.15.67:0/4272548103 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f95200b18f0 0x7f950c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:52.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.106+0000 7f95269af640 1 --2- 172.21.15.67:0/4272548103 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f950c004d40 0x7f950c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:52.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.107+0000 7f95259ad640 1 -- 172.21.15.67:0/4272548103 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f95200b18f0 msgr2=0x7f950c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:52.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.107+0000 7f95259ad640 1 --2- 172.21.15.67:0/4272548103 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f95200b18f0 0x7f950c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.107+0000 7f95259ad640 1 -- 172.21.15.67:0/4272548103 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f950c004d40 msgr2=0x7f950c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:52.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.107+0000 7f95259ad640 1 --2- 172.21.15.67:0/4272548103 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f950c004d40 0x7f950c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.107+0000 7f95259ad640 1 -- 172.21.15.67:0/4272548103 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f950c003520 con 0x7f950c003f20 2024-03-24T00:40:52.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.107+0000 7f95259ad640 1 --2- 172.21.15.67:0/4272548103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f950c003f20 0x7f950c004800 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f9510009ac0 tx=0x7f950c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fa3ce00f7c47165 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:52.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.107+0000 7f95251ac640 1 -- 172.21.15.67:0/4272548103 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9510039070 con 0x7f950c003f20 2024-03-24T00:40:52.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.108+0000 7f95251ac640 1 -- 172.21.15.67:0/4272548103 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f951002c600 con 0x7f950c003f20 2024-03-24T00:40:52.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.108+0000 7f95251ac640 1 -- 172.21.15.67:0/4272548103 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9510009690 con 0x7f950c003f20 2024-03-24T00:40:52.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.108+0000 7f9528412640 1 -- 172.21.15.67:0/4272548103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f950c003f20 msgr2=0x7f950c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:52.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.108+0000 7f9528412640 1 --2- 172.21.15.67:0/4272548103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f950c003f20 0x7f950c004800 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f9510009ac0 tx=0x7f950c005830 comp rx=0 tx=0).stop 2024-03-24T00:40:52.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.108+0000 7f95269af640 1 -- 172.21.15.67:0/4272548103 reap_dead start 2024-03-24T00:40:52.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.108+0000 7f9528412640 1 -- 172.21.15.67:0/4272548103 shutdown_connections 2024-03-24T00:40:52.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.108+0000 7f9528412640 1 -- 172.21.15.67:0/4272548103 >> 172.21.15.67:0/4272548103 conn(0x7f95200d1bf0 msgr2=0x7f95200d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:52.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.108+0000 7f9528412640 1 -- 172.21.15.67:0/4272548103 shutdown_connections 2024-03-24T00:40:52.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.108+0000 7f9528412640 1 -- 172.21.15.67:0/4272548103 wait complete. 2024-03-24T00:40:52.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.109+0000 7f9528412640 1 Processor -- start 2024-03-24T00:40:52.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.109+0000 7f9528412640 1 -- start start 2024-03-24T00:40:52.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.110+0000 7f9528412640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f950c003f20 0x7f9520151a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:52.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.110+0000 7f9528412640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95201520c0 0x7f95200b00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:52.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.111+0000 7f9528412640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f95201e7930 0x7f95201e7db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:52.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.111+0000 7f9528412640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f952015e4f0 con 0x7f95201520c0 2024-03-24T00:40:52.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.111+0000 7f9528412640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f952015e370 con 0x7f95201e7930 2024-03-24T00:40:52.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.111+0000 7f9528412640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95200b1d30 con 0x7f950c003f20 2024-03-24T00:40:52.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.111+0000 7f95259ad640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95201520c0 0x7f95200b00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:52.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.111+0000 7f95259ad640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95201520c0 0x7f95200b00e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41014/0 (socket says 172.21.15.67:41014) 2024-03-24T00:40:52.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.111+0000 7f95259ad640 1 -- 172.21.15.67:0/3590282910 learned_addr learned my addr 172.21.15.67:0/3590282910 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:52.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.111+0000 7f95269af640 1 --2- 172.21.15.67:0/3590282910 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f95201e7930 0x7f95201e7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:52.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.111+0000 7f95261ae640 1 --2- 172.21.15.67:0/3590282910 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f950c003f20 0x7f9520151a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:52.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.111+0000 7f95259ad640 1 -- 172.21.15.67:0/3590282910 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f950c003f20 msgr2=0x7f9520151a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:52.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.111+0000 7f95259ad640 1 --2- 172.21.15.67:0/3590282910 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f950c003f20 0x7f9520151a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.111+0000 7f95259ad640 1 -- 172.21.15.67:0/3590282910 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f95201e7930 msgr2=0x7f95201e7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:52.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.111+0000 7f95259ad640 1 --2- 172.21.15.67:0/3590282910 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f95201e7930 0x7f95201e7db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.111+0000 7f95259ad640 1 -- 172.21.15.67:0/3590282910 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95200b0620 con 0x7f95201520c0 2024-03-24T00:40:52.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.111+0000 7f95261ae640 1 --2- 172.21.15.67:0/3590282910 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f950c003f20 0x7f9520151a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:40:52.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.111+0000 7f95259ad640 1 --2- 172.21.15.67:0/3590282910 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95201520c0 0x7f95200b00e0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f9510009ac0 tx=0x7f9510004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:52.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.111+0000 7f9516ffd640 1 -- 172.21.15.67:0/3590282910 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9510039050 con 0x7f95201520c0 2024-03-24T00:40:52.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.111+0000 7f9516ffd640 1 -- 172.21.15.67:0/3590282910 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9510045070 con 0x7f95201520c0 2024-03-24T00:40:52.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.111+0000 7f9516ffd640 1 -- 172.21.15.67:0/3590282910 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95100044e0 con 0x7f95201520c0 2024-03-24T00:40:52.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.111+0000 7f9528412640 1 -- 172.21.15.67:0/3590282910 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95200c8ed0 con 0x7f95201520c0 2024-03-24T00:40:52.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.111+0000 7f9528412640 1 -- 172.21.15.67:0/3590282910 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95200c9490 con 0x7f95201520c0 2024-03-24T00:40:52.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.112+0000 7f9528412640 1 -- 172.21.15.67:0/3590282910 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f952015c770 con 0x7f95201520c0 2024-03-24T00:40:52.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.113+0000 7f9516ffd640 1 -- 172.21.15.67:0/3590282910 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9510003850 con 0x7f95201520c0 2024-03-24T00:40:52.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.113+0000 7f9516ffd640 1 --2- 172.21.15.67:0/3590282910 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f950406bbd0 0x7f950406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:52.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.113+0000 7f9516ffd640 1 -- 172.21.15.67:0/3590282910 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f95100ac6d0 con 0x7f95201520c0 2024-03-24T00:40:52.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.113+0000 7f95261ae640 1 --2- 172.21.15.67:0/3590282910 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f950406bbd0 0x7f950406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:52.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.114+0000 7f95261ae640 1 --2- 172.21.15.67:0/3590282910 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f950406bbd0 0x7f950406e090 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f951c001550 tx=0x7f951c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:52.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.118+0000 7f9516ffd640 1 -- 172.21.15.67:0/3590282910 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f952015c770 con 0x7f95201520c0 2024-03-24T00:40:52.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.372+0000 7f9528412640 1 -- 172.21.15.67:0/3590282910 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f95200b2870 con 0x7f95201520c0 2024-03-24T00:40:52.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.372+0000 7f9516ffd640 1 -- 172.21.15.67:0/3590282910 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f95200b2870 con 0x7f95201520c0 2024-03-24T00:40:52.373 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:40:52.373 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5360,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3312887486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":3312887486},{"type":"v1","addr":"172.21.15.67:6835","nonce":3312887486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5821,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/4171866637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":4171866637},{"type":"v1","addr":"172.21.15.72:6836","nonce":4171866637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5824,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/4106412452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":4106412452},{"type":"v1","addr":"172.21.15.72:6837","nonce":4106412452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:40:39.649631+0000","modified":"2024-03-24T00:40:42.657888+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5853},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5853":{"gid":5853,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6834/2741818214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2741818214},{"type":"v1","addr":"172.21.15.67:6834","nonce":2741818214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5853,"qdb_cluster":[5853]},"id":3}]} 2024-03-24T00:40:52.373 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 26 2024-03-24T00:40:52.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.374+0000 7f9528412640 1 -- 172.21.15.67:0/3590282910 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f950406bbd0 msgr2=0x7f950406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:52.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.374+0000 7f9528412640 1 --2- 172.21.15.67:0/3590282910 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f950406bbd0 0x7f950406e090 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f951c001550 tx=0x7f951c009040 comp rx=0 tx=0).stop 2024-03-24T00:40:52.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.374+0000 7f9528412640 1 -- 172.21.15.67:0/3590282910 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95201520c0 msgr2=0x7f95200b00e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:52.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.374+0000 7f9528412640 1 --2- 172.21.15.67:0/3590282910 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95201520c0 0x7f95200b00e0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f9510009ac0 tx=0x7f9510004730 comp rx=0 tx=0).stop 2024-03-24T00:40:52.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.374+0000 7f9528412640 1 -- 172.21.15.67:0/3590282910 shutdown_connections 2024-03-24T00:40:52.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.374+0000 7f9528412640 1 --2- 172.21.15.67:0/3590282910 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f950406bbd0 0x7f950406e090 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.374+0000 7f9528412640 1 --2- 172.21.15.67:0/3590282910 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f95201e7930 0x7f95201e7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.374+0000 7f9528412640 1 --2- 172.21.15.67:0/3590282910 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95201520c0 0x7f95200b00e0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.374+0000 7f9528412640 1 --2- 172.21.15.67:0/3590282910 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f950c003f20 0x7f9520151a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.374+0000 7f9528412640 1 -- 172.21.15.67:0/3590282910 >> 172.21.15.67:0/3590282910 conn(0x7f95200d1bf0 msgr2=0x7f952008ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:52.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.375+0000 7f9528412640 1 -- 172.21.15.67:0/3590282910 shutdown_connections 2024-03-24T00:40:52.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.375+0000 7f9528412640 1 -- 172.21.15.67:0/3590282910 wait complete. 2024-03-24T00:40:52.386 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-24T00:40:52.511 INFO:teuthology.orchestra.run.smithi067.stdout:{"message":"","return_code":0} 2024-03-24T00:40:52.512 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-24T00:40:52.512 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-24T00:40:52.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.554+0000 7f39a126cf40 1 Processor -- start 2024-03-24T00:40:52.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.554+0000 7f39a126cf40 1 -- start start 2024-03-24T00:40:52.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.555+0000 7f39a126cf40 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39880030f0 0x55fe24fdc260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:52.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.555+0000 7f39a126cf40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55fe24ffcc30 0x55fe24f232d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:52.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.555+0000 7f39a126cf40 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55fe24ff8ad0 0x55fe24ff8f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:52.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.555+0000 7f39a126cf40 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe24ec90d0 con 0x7f39880030f0 2024-03-24T00:40:52.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.555+0000 7f39a126cf40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe24ecd030 con 0x55fe24ffcc30 2024-03-24T00:40:52.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.555+0000 7f39a126cf40 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fe24ffa930 con 0x55fe24ff8ad0 2024-03-24T00:40:52.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.555+0000 7f399ed6a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39880030f0 0x55fe24fdc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:52.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.555+0000 7f399ed6a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39880030f0 0x55fe24fdc260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41032/0 (socket says 172.21.15.67:41032) 2024-03-24T00:40:52.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.555+0000 7f399ed6a640 1 -- 172.21.15.67:0/478510597 learned_addr learned my addr 172.21.15.67:0/478510597 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:52.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.555+0000 7f399ed6a640 1 -- 172.21.15.67:0/478510597 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55fe24ff8ad0 msgr2=0x55fe24ff8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:52.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.555+0000 7f399ed6a640 1 --2- 172.21.15.67:0/478510597 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55fe24ff8ad0 0x55fe24ff8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.555+0000 7f399e569640 1 --2- 172.21.15.67:0/478510597 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55fe24ffcc30 0x55fe24f232d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:52.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.555+0000 7f399ed6a640 1 -- 172.21.15.67:0/478510597 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55fe24ffcc30 msgr2=0x55fe24f232d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:52.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.555+0000 7f399ed6a640 1 --2- 172.21.15.67:0/478510597 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55fe24ffcc30 0x55fe24f232d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.555+0000 7f399ed6a640 1 -- 172.21.15.67:0/478510597 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fe24ff9770 con 0x7f39880030f0 2024-03-24T00:40:52.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.555+0000 7f399e569640 1 --2- 172.21.15.67:0/478510597 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55fe24ffcc30 0x55fe24f232d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:40:52.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.556+0000 7f399ed6a640 1 --2- 172.21.15.67:0/478510597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39880030f0 0x55fe24fdc260 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f3990009730 tx=0x7f3990005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95628e329d4356df server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:52.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.556+0000 7f399d567640 1 -- 172.21.15.67:0/478510597 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39900078f0 con 0x7f39880030f0 2024-03-24T00:40:52.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.556+0000 7f399d567640 1 -- 172.21.15.67:0/478510597 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3990007a90 con 0x7f39880030f0 2024-03-24T00:40:52.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.556+0000 7f399d567640 1 -- 172.21.15.67:0/478510597 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3990007e60 con 0x7f39880030f0 2024-03-24T00:40:52.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.556+0000 7f39a126cf40 1 -- 172.21.15.67:0/478510597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39880030f0 msgr2=0x55fe24fdc260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:52.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.556+0000 7f39a126cf40 1 --2- 172.21.15.67:0/478510597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39880030f0 0x55fe24fdc260 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f3990009730 tx=0x7f3990005a00 comp rx=0 tx=0).stop 2024-03-24T00:40:52.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.556+0000 7f39a126cf40 1 -- 172.21.15.67:0/478510597 shutdown_connections 2024-03-24T00:40:52.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.556+0000 7f39a126cf40 1 --2- 172.21.15.67:0/478510597 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55fe24ff8ad0 0x55fe24ff8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.556+0000 7f39a126cf40 1 --2- 172.21.15.67:0/478510597 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55fe24ffcc30 0x55fe24f232d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.556+0000 7f39a126cf40 1 --2- 172.21.15.67:0/478510597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39880030f0 0x55fe24fdc260 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.556+0000 7f39a126cf40 1 -- 172.21.15.67:0/478510597 >> 172.21.15.67:0/478510597 conn(0x55fe24f20e70 msgr2=0x55fe24ff7730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:52.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.556+0000 7f39a126cf40 1 -- 172.21.15.67:0/478510597 shutdown_connections 2024-03-24T00:40:52.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.556+0000 7f39a126cf40 1 -- 172.21.15.67:0/478510597 wait complete. 2024-03-24T00:40:52.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.557+0000 7f39a126cf40 1 Processor -- start 2024-03-24T00:40:52.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.557+0000 7f39a126cf40 1 -- start start 2024-03-24T00:40:52.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.558+0000 7f39a126cf40 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55fe24ff8ad0 0x55fe24fdb360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:52.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.558+0000 7f39a126cf40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55fe24ffcc30 0x55fe24fdacc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:52.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.558+0000 7f39a126cf40 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39880030f0 0x55fe24f24b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:52.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.558+0000 7f39a126cf40 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe24fdf760 con 0x7f39880030f0 2024-03-24T00:40:52.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.558+0000 7f39a126cf40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe24f22000 con 0x55fe24ffcc30 2024-03-24T00:40:52.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.558+0000 7f39a126cf40 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fe24fdf480 con 0x55fe24ff8ad0 2024-03-24T00:40:52.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.559+0000 7f399f56b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39880030f0 0x55fe24f24b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:52.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.559+0000 7f399f56b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39880030f0 0x55fe24f24b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41034/0 (socket says 172.21.15.67:41034) 2024-03-24T00:40:52.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.559+0000 7f399f56b640 1 -- 172.21.15.67:0/89971167 learned_addr learned my addr 172.21.15.67:0/89971167 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:52.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.559+0000 7f399e569640 1 --2- 172.21.15.67:0/89971167 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55fe24ffcc30 0x55fe24fdacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:52.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.559+0000 7f399ed6a640 1 --2- 172.21.15.67:0/89971167 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55fe24ff8ad0 0x55fe24fdb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:52.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.559+0000 7f399f56b640 1 -- 172.21.15.67:0/89971167 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55fe24ff8ad0 msgr2=0x55fe24fdb360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:52.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.559+0000 7f399f56b640 1 --2- 172.21.15.67:0/89971167 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55fe24ff8ad0 0x55fe24fdb360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.559+0000 7f399f56b640 1 -- 172.21.15.67:0/89971167 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55fe24ffcc30 msgr2=0x55fe24fdacc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:52.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.559+0000 7f399f56b640 1 --2- 172.21.15.67:0/89971167 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55fe24ffcc30 0x55fe24fdacc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.559+0000 7f399f56b640 1 -- 172.21.15.67:0/89971167 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fe24fdef50 con 0x7f39880030f0 2024-03-24T00:40:52.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.559+0000 7f399e569640 1 --2- 172.21.15.67:0/89971167 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55fe24ffcc30 0x55fe24fdacc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:40:52.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.559+0000 7f399f56b640 1 --2- 172.21.15.67:0/89971167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39880030f0 0x55fe24f24b40 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f3998002230 tx=0x7f3998010f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:52.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.559+0000 7f3987fff640 1 -- 172.21.15.67:0/89971167 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f399800f040 con 0x7f39880030f0 2024-03-24T00:40:52.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.559+0000 7f3987fff640 1 -- 172.21.15.67:0/89971167 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3998019070 con 0x7f39880030f0 2024-03-24T00:40:52.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.559+0000 7f39a126cf40 1 -- 172.21.15.67:0/89971167 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55fe24ff9490 con 0x7f39880030f0 2024-03-24T00:40:52.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.559+0000 7f3987fff640 1 -- 172.21.15.67:0/89971167 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f399800ed90 con 0x7f39880030f0 2024-03-24T00:40:52.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.559+0000 7f39a126cf40 1 -- 172.21.15.67:0/89971167 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fe24fde470 con 0x7f39880030f0 2024-03-24T00:40:52.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.561+0000 7f3987fff640 1 -- 172.21.15.67:0/89971167 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f399801e460 con 0x7f39880030f0 2024-03-24T00:40:52.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.561+0000 7f3987fff640 1 --2- 172.21.15.67:0/89971167 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f398c06a5e0 0x7f398c06caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:52.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.561+0000 7f3987fff640 1 -- 172.21.15.67:0/89971167 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f399808be80 con 0x7f39880030f0 2024-03-24T00:40:52.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.562+0000 7f399ed6a640 1 --2- 172.21.15.67:0/89971167 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f398c06a5e0 0x7f398c06caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:52.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.562+0000 7f39a126cf40 1 --2- 172.21.15.67:0/89971167 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x55fe24ff8f60 0x55fe24f67e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:52.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.562+0000 7f39a126cf40 1 -- 172.21.15.67:0/89971167 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:1 8.7 8:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50) v9 -- 0x55fe24f683a0 con 0x55fe24ff8f60 2024-03-24T00:40:52.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.562+0000 7f399e569640 1 --2- 172.21.15.67:0/89971167 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x55fe24ff8f60 0x55fe24f67e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:52.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.562+0000 7f399e569640 1 --2- 172.21.15.67:0/89971167 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x55fe24ff8f60 0x55fe24f67e60 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:52.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.562+0000 7f399ed6a640 1 --2- 172.21.15.67:0/89971167 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f398c06a5e0 0x7f398c06caa0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f3990006010 tx=0x7f3990007000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:52.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.562+0000 7f399e569640 1 -- 172.21.15.67:0/89971167 <== osd.1 v2:172.21.15.67:6800/4062510404 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (95) out=95b] v0'0 uv1 ondisk = 0) v8 ==== 164+0+95 (crc 0 0 0) 0x7f399405e430 con 0x55fe24ff8f60 2024-03-24T00:40:52.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.562+0000 7f39a126cf40 1 -- 172.21.15.67:0/89971167 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x55fe24ff8f60 msgr2=0x55fe24f67e60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:52.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.562+0000 7f39a126cf40 1 --2- 172.21.15.67:0/89971167 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x55fe24ff8f60 0x55fe24f67e60 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.563+0000 7f39a126cf40 1 -- 172.21.15.67:0/89971167 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f398c06a5e0 msgr2=0x7f398c06caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:52.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.563+0000 7f39a126cf40 1 --2- 172.21.15.67:0/89971167 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f398c06a5e0 0x7f398c06caa0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f3990006010 tx=0x7f3990007000 comp rx=0 tx=0).stop 2024-03-24T00:40:52.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.563+0000 7f39a126cf40 1 -- 172.21.15.67:0/89971167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39880030f0 msgr2=0x55fe24f24b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:52.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.563+0000 7f39a126cf40 1 --2- 172.21.15.67:0/89971167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39880030f0 0x55fe24f24b40 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f3998002230 tx=0x7f3998010f70 comp rx=0 tx=0).stop 2024-03-24T00:40:52.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.563+0000 7f399f56b640 1 -- 172.21.15.67:0/89971167 reap_dead start 2024-03-24T00:40:52.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.563+0000 7f39a126cf40 1 -- 172.21.15.67:0/89971167 shutdown_connections 2024-03-24T00:40:52.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.563+0000 7f39a126cf40 1 -- 172.21.15.67:0/89971167 >> 172.21.15.67:0/89971167 conn(0x55fe24f20e70 msgr2=0x55fe24ff7730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:52.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.563+0000 7f39a126cf40 1 -- 172.21.15.67:0/89971167 shutdown_connections 2024-03-24T00:40:52.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.563+0000 7f39a126cf40 1 -- 172.21.15.67:0/89971167 wait complete. 2024-03-24T00:40:52.567 INFO:teuthology.orchestra.run.smithi067.stdout:Yalphaparent_a 2024-03-24T00:40:52.568 DEBUG:teuthology.orchestra.run.smithi067:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-24T00:40:52.700 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:40:52.700 INFO:teuthology.orchestra.run.smithi067.stdout: "ino": 1099511627777, 2024-03-24T00:40:52.700 INFO:teuthology.orchestra.run.smithi067.stdout: "ancestors": [ 2024-03-24T00:40:52.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:40:52.700 INFO:teuthology.orchestra.run.smithi067.stdout: "dirino": 1099511627776, 2024-03-24T00:40:52.700 INFO:teuthology.orchestra.run.smithi067.stdout: "dname": "alpha", 2024-03-24T00:40:52.701 INFO:teuthology.orchestra.run.smithi067.stdout: "version": 2 2024-03-24T00:40:52.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:40:52.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:40:52.701 INFO:teuthology.orchestra.run.smithi067.stdout: "dirino": 1, 2024-03-24T00:40:52.701 INFO:teuthology.orchestra.run.smithi067.stdout: "dname": "parent_a", 2024-03-24T00:40:52.701 INFO:teuthology.orchestra.run.smithi067.stdout: "version": 4 2024-03-24T00:40:52.701 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:40:52.701 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-24T00:40:52.701 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 8, 2024-03-24T00:40:52.701 INFO:teuthology.orchestra.run.smithi067.stdout: "old_pools": [] 2024-03-24T00:40:52.701 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:40:52.701 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:40:52.733 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data setxattr 10000000001.00000000 parent 'oh i'"'"'m sorry did i overwrite your xattr?' 2024-03-24T00:40:52.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.774+0000 7f8ada932f40 1 Processor -- start 2024-03-24T00:40:52.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.774+0000 7f8ada932f40 1 -- start start 2024-03-24T00:40:52.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.775+0000 7f8ada932f40 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5610274cf990 0x56102749ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:52.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.775+0000 7f8ada932f40 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5610274cfd70 0x5610274a32d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:52.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.775+0000 7f8ada932f40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5610274a3810 0x5610274ea0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:52.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.775+0000 7f8ada932f40 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610274d1960 con 0x5610274cf990 2024-03-24T00:40:52.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.775+0000 7f8ada932f40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610274480d0 con 0x5610274a3810 2024-03-24T00:40:52.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.775+0000 7f8ada932f40 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5610274d1c00 con 0x5610274cfd70 2024-03-24T00:40:52.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.775+0000 7f8ad8432640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5610274cf990 0x56102749ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:52.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.776+0000 7f8ad8432640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5610274cf990 0x56102749ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41062/0 (socket says 172.21.15.67:41062) 2024-03-24T00:40:52.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.776+0000 7f8ad8432640 1 -- 172.21.15.67:0/3238864982 learned_addr learned my addr 172.21.15.67:0/3238864982 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:52.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.776+0000 7f8ad8432640 1 -- 172.21.15.67:0/3238864982 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5610274cfd70 msgr2=0x5610274a32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:52.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.776+0000 7f8ad7c31640 1 --2- 172.21.15.67:0/3238864982 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5610274cfd70 0x5610274a32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:52.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.776+0000 7f8ad8432640 1 --2- 172.21.15.67:0/3238864982 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5610274cfd70 0x5610274a32d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.776+0000 7f8ad8432640 1 -- 172.21.15.67:0/3238864982 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5610274a3810 msgr2=0x5610274ea0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:52.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.776+0000 7f8ad8432640 1 --2- 172.21.15.67:0/3238864982 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5610274a3810 0x5610274ea0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.776+0000 7f8ad8432640 1 -- 172.21.15.67:0/3238864982 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5610274d14b0 con 0x5610274cf990 2024-03-24T00:40:52.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.776+0000 7f8ad8432640 1 --2- 172.21.15.67:0/3238864982 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5610274cf990 0x56102749ef50 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f8ac800a4f0 tx=0x7f8ac804c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ac3a8453bac4cd3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:52.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.776+0000 7f8ad6c2f640 1 -- 172.21.15.67:0/3238864982 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac8057070 con 0x5610274cf990 2024-03-24T00:40:52.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.776+0000 7f8ad6c2f640 1 -- 172.21.15.67:0/3238864982 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ac8061070 con 0x5610274cf990 2024-03-24T00:40:52.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.776+0000 7f8ad6c2f640 1 -- 172.21.15.67:0/3238864982 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac8052b30 con 0x5610274cf990 2024-03-24T00:40:52.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.777+0000 7f8ada932f40 1 -- 172.21.15.67:0/3238864982 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5610274cf990 msgr2=0x56102749ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:52.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.777+0000 7f8ada932f40 1 --2- 172.21.15.67:0/3238864982 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5610274cf990 0x56102749ef50 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f8ac800a4f0 tx=0x7f8ac804c000 comp rx=0 tx=0).stop 2024-03-24T00:40:52.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.777+0000 7f8ada932f40 1 -- 172.21.15.67:0/3238864982 shutdown_connections 2024-03-24T00:40:52.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.777+0000 7f8ada932f40 1 --2- 172.21.15.67:0/3238864982 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5610274a3810 0x5610274ea0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.777+0000 7f8ada932f40 1 --2- 172.21.15.67:0/3238864982 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5610274cfd70 0x5610274a32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.777+0000 7f8ada932f40 1 --2- 172.21.15.67:0/3238864982 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5610274cf990 0x56102749ef50 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.777+0000 7f8ada932f40 1 -- 172.21.15.67:0/3238864982 >> 172.21.15.67:0/3238864982 conn(0x5610274cc600 msgr2=0x5610274eb550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:52.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.777+0000 7f8ada932f40 1 -- 172.21.15.67:0/3238864982 shutdown_connections 2024-03-24T00:40:52.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.777+0000 7f8ada932f40 1 -- 172.21.15.67:0/3238864982 wait complete. 2024-03-24T00:40:52.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.777+0000 7f8ada932f40 1 Processor -- start 2024-03-24T00:40:52.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.777+0000 7f8ada932f40 1 -- start start 2024-03-24T00:40:52.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.778+0000 7f8ada932f40 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5610274cf990 0x5610274e88f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:52.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.778+0000 7f8ada932f40 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5610274cfd70 0x561027613ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:52.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.778+0000 7f8ada932f40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561027578ac0 0x5610274cc9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:52.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.778+0000 7f8ada932f40 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x561027571dd0 con 0x5610274cf990 2024-03-24T00:40:52.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.778+0000 7f8ada932f40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610274e91c0 con 0x561027578ac0 2024-03-24T00:40:52.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.778+0000 7f8ada932f40 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5610275720b0 con 0x5610274cfd70 2024-03-24T00:40:52.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.778+0000 7f8ad8432640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5610274cf990 0x5610274e88f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:52.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.778+0000 7f8ad8432640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5610274cf990 0x5610274e88f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41072/0 (socket says 172.21.15.67:41072) 2024-03-24T00:40:52.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.778+0000 7f8ad8432640 1 -- 172.21.15.67:0/2058710870 learned_addr learned my addr 172.21.15.67:0/2058710870 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:52.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.778+0000 7f8ad7c31640 1 --2- 172.21.15.67:0/2058710870 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5610274cfd70 0x561027613ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:52.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.778+0000 7f8ad8c33640 1 --2- 172.21.15.67:0/2058710870 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561027578ac0 0x5610274cc9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:52.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.778+0000 7f8ad8432640 1 -- 172.21.15.67:0/2058710870 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5610274cfd70 msgr2=0x561027613ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:52.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.778+0000 7f8ad8432640 1 --2- 172.21.15.67:0/2058710870 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5610274cfd70 0x561027613ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.778+0000 7f8ad8432640 1 -- 172.21.15.67:0/2058710870 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561027578ac0 msgr2=0x5610274cc9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:52.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.778+0000 7f8ad8432640 1 --2- 172.21.15.67:0/2058710870 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561027578ac0 0x5610274cc9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.778+0000 7f8ad8432640 1 -- 172.21.15.67:0/2058710870 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5610274e8e30 con 0x5610274cf990 2024-03-24T00:40:52.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.779+0000 7f8ad8432640 1 --2- 172.21.15.67:0/2058710870 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5610274cf990 0x5610274e88f0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f8ac800a4f0 tx=0x7f8ac8002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:52.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.779+0000 7f8ad5c2d640 1 -- 172.21.15.67:0/2058710870 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac8057050 con 0x5610274cf990 2024-03-24T00:40:52.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.779+0000 7f8ad5c2d640 1 -- 172.21.15.67:0/2058710870 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ac8058040 con 0x5610274cf990 2024-03-24T00:40:52.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.779+0000 7f8ada932f40 1 -- 172.21.15.67:0/2058710870 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x56102756fcc0 con 0x5610274cf990 2024-03-24T00:40:52.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.779+0000 7f8ad5c2d640 1 -- 172.21.15.67:0/2058710870 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac8065880 con 0x5610274cf990 2024-03-24T00:40:52.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.779+0000 7f8ada932f40 1 -- 172.21.15.67:0/2058710870 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561027570280 con 0x5610274cf990 2024-03-24T00:40:52.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.780+0000 7f8ad5c2d640 1 -- 172.21.15.67:0/2058710870 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8ac8065a20 con 0x5610274cf990 2024-03-24T00:40:52.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.781+0000 7f8ad5c2d640 1 --2- 172.21.15.67:0/2058710870 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8ac406a5b0 0x7f8ac406ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:52.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.781+0000 7f8ad5c2d640 1 -- 172.21.15.67:0/2058710870 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8ac80cc760 con 0x5610274cf990 2024-03-24T00:40:52.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.781+0000 7f8ad7c31640 1 --2- 172.21.15.67:0/2058710870 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8ac406a5b0 0x7f8ac406ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:52.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.781+0000 7f8ada932f40 1 --2- 172.21.15.67:0/2058710870 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x561027612340 0x56102761dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:52.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.781+0000 7f8ada932f40 1 -- 172.21.15.67:0/2058710870 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:1 8.7 8:e2e07930:::10000000001.00000000:head [setxattr parent (40) in=46b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50) v9 -- 0x56102761e260 con 0x561027612340 2024-03-24T00:40:52.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.781+0000 7f8ad8c33640 1 --2- 172.21.15.67:0/2058710870 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x561027612340 0x56102761dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:52.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.781+0000 7f8ad7c31640 1 --2- 172.21.15.67:0/2058710870 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8ac406a5b0 0x7f8ac406ca70 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x5610274d1360 tx=0x7f8acc001710 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:52.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.782+0000 7f8ad8c33640 1 --2- 172.21.15.67:0/2058710870 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x561027612340 0x56102761dd20 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:52.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.783+0000 7f8ad8c33640 1 -- 172.21.15.67:0/2058710870 <== osd.1 v2:172.21.15.67:6800/4062510404 1 ==== osd_op_reply(1 10000000001.00000000 [setxattr (40)] v50'2 uv2 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7f8ad000d790 con 0x561027612340 2024-03-24T00:40:52.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.783+0000 7f8ada932f40 1 -- 172.21.15.67:0/2058710870 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x561027612340 msgr2=0x56102761dd20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:52.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.783+0000 7f8ada932f40 1 --2- 172.21.15.67:0/2058710870 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x561027612340 0x56102761dd20 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.783+0000 7f8ada932f40 1 -- 172.21.15.67:0/2058710870 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8ac406a5b0 msgr2=0x7f8ac406ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:52.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.783+0000 7f8ada932f40 1 --2- 172.21.15.67:0/2058710870 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8ac406a5b0 0x7f8ac406ca70 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x5610274d1360 tx=0x7f8acc001710 comp rx=0 tx=0).stop 2024-03-24T00:40:52.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.783+0000 7f8ada932f40 1 -- 172.21.15.67:0/2058710870 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5610274cf990 msgr2=0x5610274e88f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:52.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.783+0000 7f8ada932f40 1 --2- 172.21.15.67:0/2058710870 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5610274cf990 0x5610274e88f0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f8ac800a4f0 tx=0x7f8ac8002fb0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.783+0000 7f8ad8c33640 1 -- 172.21.15.67:0/2058710870 reap_dead start 2024-03-24T00:40:52.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.783+0000 7f8ada932f40 1 -- 172.21.15.67:0/2058710870 shutdown_connections 2024-03-24T00:40:52.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.783+0000 7f8ada932f40 1 -- 172.21.15.67:0/2058710870 >> 172.21.15.67:0/2058710870 conn(0x5610274cc600 msgr2=0x5610274d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:52.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.783+0000 7f8ada932f40 1 -- 172.21.15.67:0/2058710870 shutdown_connections 2024-03-24T00:40:52.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.783+0000 7f8ada932f40 1 -- 172.21.15.67:0/2058710870 wait complete. 2024-03-24T00:40:52.788 DEBUG:teuthology.orchestra.run.smithi067:> sudo daemon-helper kill ceph --cluster ceph -w 2024-03-24T00:40:52.830 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:40:52.830 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.3:0 scrub start / repair,recursive 2024-03-24T00:40:52.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.935+0000 7fe8d7777640 1 Processor -- start 2024-03-24T00:40:52.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.936+0000 7fe8d7777640 1 -- start start 2024-03-24T00:40:52.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.937+0000 7fe8d7777640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe8d00d0d50 0x7fe8d0168e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:52.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.937+0000 7fe8d7777640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d00cf1f0 con 0x7fe8d015dfc0 2024-03-24T00:40:52.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.937+0000 7fe8d7777640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d00d2020 con 0x7fe8d00ced60 2024-03-24T00:40:52.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.937+0000 7fe8d7777640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d015e3a0 con 0x7fe8d00d0d50 2024-03-24T00:40:52.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.937+0000 7fe8d5513640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe8d015dfc0 0x7fe8d01624c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39704/0 (socket says 172.21.15.67:39704) 2024-03-24T00:40:52.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.937+0000 7fe8d5513640 1 -- 172.21.15.67:0/3891087996 learned_addr learned my addr 172.21.15.67:0/3891087996 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:40:52.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.937+0000 7fe8d5d14640 1 --2- 172.21.15.67:0/3891087996 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe8d00d0d50 0x7fe8d0168e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:52.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.937+0000 7fe8c7fff640 1 -- 172.21.15.67:0/3891087996 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3638947718 0 0) 0x7fe8d00cf1f0 con 0x7fe8d015dfc0 2024-03-24T00:40:52.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.937+0000 7fe8c7fff640 1 -- 172.21.15.67:0/3891087996 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8bc003520 con 0x7fe8d015dfc0 2024-03-24T00:40:52.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.937+0000 7fe8c7fff640 1 -- 172.21.15.67:0/3891087996 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 891201625 0 0) 0x7fe8d00d2020 con 0x7fe8d00ced60 2024-03-24T00:40:52.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.937+0000 7fe8c7fff640 1 -- 172.21.15.67:0/3891087996 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8d00cf1f0 con 0x7fe8d00ced60 2024-03-24T00:40:52.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.938+0000 7fe8c7fff640 1 -- 172.21.15.67:0/3891087996 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 51130495 0 0) 0x7fe8bc003520 con 0x7fe8d015dfc0 2024-03-24T00:40:52.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.938+0000 7fe8d5d14640 1 -- 172.21.15.67:0/3891087996 >> v1:172.21.15.72:6789/0 conn(0x7fe8d00ced60 legacy=0x7fe8d00d0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:52.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.938+0000 7fe8d5d14640 1 -- 172.21.15.67:0/3891087996 >> v1:172.21.15.67:6789/0 conn(0x7fe8d015dfc0 legacy=0x7fe8d01624c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:52.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.938+0000 7fe8d5d14640 1 -- 172.21.15.67:0/3891087996 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d00d2300 con 0x7fe8d00d0d50 2024-03-24T00:40:52.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.939+0000 7fe8d5d14640 1 --2- 172.21.15.67:0/3891087996 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe8d00d0d50 0x7fe8d0168e40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe8d015e520 tx=0x7fe8c802bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d1bfd36dcac0f10 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:52.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.939+0000 7fe8c7fff640 1 -- 172.21.15.67:0/3891087996 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c8039070 con 0x7fe8d00d0d50 2024-03-24T00:40:52.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.939+0000 7fe8c7fff640 1 -- 172.21.15.67:0/3891087996 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8c802c910 con 0x7fe8d00d0d50 2024-03-24T00:40:52.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.939+0000 7fe8c7fff640 1 -- 172.21.15.67:0/3891087996 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8cc001670 con 0x7fe8d00d0d50 2024-03-24T00:40:52.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.940+0000 7fe8d7777640 1 -- 172.21.15.67:0/3891087996 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe8d00d0d50 msgr2=0x7fe8d0168e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:52.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.940+0000 7fe8d7777640 1 --2- 172.21.15.67:0/3891087996 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe8d00d0d50 0x7fe8d0168e40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe8d015e520 tx=0x7fe8c802bde0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.940+0000 7fe8d7777640 1 -- 172.21.15.67:0/3891087996 shutdown_connections 2024-03-24T00:40:52.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.940+0000 7fe8d7777640 1 --2- 172.21.15.67:0/3891087996 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe8d00d0d50 0x7fe8d0168e40 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.940+0000 7fe8d7777640 1 -- 172.21.15.67:0/3891087996 >> 172.21.15.67:0/3891087996 conn(0x7fe8d006c880 msgr2=0x7fe8d006cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:52.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.940+0000 7fe8d7777640 1 -- 172.21.15.67:0/3891087996 shutdown_connections 2024-03-24T00:40:52.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.940+0000 7fe8d7777640 1 -- 172.21.15.67:0/3891087996 wait complete. 2024-03-24T00:40:52.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.941+0000 7fe8d7777640 1 Processor -- start 2024-03-24T00:40:52.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.941+0000 7fe8d7777640 1 -- start start 2024-03-24T00:40:52.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.942+0000 7fe8d7777640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8d00d0d50 0x7fe8d00cb4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:52.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.942+0000 7fe8d7777640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8d015dfc0 0x7fe8d00b5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:52.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.942+0000 7fe8d7777640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe8d01f8250 0x7fe8d0152b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:52.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.942+0000 7fe8d7777640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d00d3b10 con 0x7fe8d015dfc0 2024-03-24T00:40:52.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.942+0000 7fe8d7777640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d00b31f0 con 0x7fe8d00d0d50 2024-03-24T00:40:52.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.942+0000 7fe8d7777640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d00b6ea0 con 0x7fe8d01f8250 2024-03-24T00:40:52.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.942+0000 7fe8d4d12640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8d015dfc0 0x7fe8d00b5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:52.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.942+0000 7fe8d4d12640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8d015dfc0 0x7fe8d00b5c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41074/0 (socket says 172.21.15.67:41074) 2024-03-24T00:40:52.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.942+0000 7fe8d4d12640 1 -- 172.21.15.67:0/2501958461 learned_addr learned my addr 172.21.15.67:0/2501958461 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:52.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.942+0000 7fe8d5513640 1 --2- 172.21.15.67:0/2501958461 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8d00d0d50 0x7fe8d00cb4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:52.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.942+0000 7fe8d5d14640 1 --2- 172.21.15.67:0/2501958461 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe8d01f8250 0x7fe8d0152b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:52.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.942+0000 7fe8d4d12640 1 -- 172.21.15.67:0/2501958461 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe8d01f8250 msgr2=0x7fe8d0152b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:52.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.942+0000 7fe8d4d12640 1 --2- 172.21.15.67:0/2501958461 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe8d01f8250 0x7fe8d0152b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.942+0000 7fe8d4d12640 1 -- 172.21.15.67:0/2501958461 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8d00d0d50 msgr2=0x7fe8d00cb4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:52.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.942+0000 7fe8d4d12640 1 --2- 172.21.15.67:0/2501958461 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8d00d0d50 0x7fe8d00cb4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.942+0000 7fe8d4d12640 1 -- 172.21.15.67:0/2501958461 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d015e6a0 con 0x7fe8d015dfc0 2024-03-24T00:40:52.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.942+0000 7fe8d4d12640 1 --2- 172.21.15.67:0/2501958461 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8d015dfc0 0x7fe8d00b5c50 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fe8c000c3c0 tx=0x7fe8c000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:52.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.943+0000 7fe8c5ffb640 1 -- 172.21.15.67:0/2501958461 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c0017070 con 0x7fe8d015dfc0 2024-03-24T00:40:52.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.943+0000 7fe8c5ffb640 1 -- 172.21.15.67:0/2501958461 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8c00052f0 con 0x7fe8d015dfc0 2024-03-24T00:40:52.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.943+0000 7fe8d7777640 1 -- 172.21.15.67:0/2501958461 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8d015e980 con 0x7fe8d015dfc0 2024-03-24T00:40:52.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.943+0000 7fe8c5ffb640 1 -- 172.21.15.67:0/2501958461 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c0003830 con 0x7fe8d015dfc0 2024-03-24T00:40:52.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.943+0000 7fe8d7777640 1 -- 172.21.15.67:0/2501958461 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8d015ef40 con 0x7fe8d015dfc0 2024-03-24T00:40:52.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.945+0000 7fe8c5ffb640 1 -- 172.21.15.67:0/2501958461 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe8c001a070 con 0x7fe8d015dfc0 2024-03-24T00:40:52.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.945+0000 7fe8c5ffb640 1 --2- 172.21.15.67:0/2501958461 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe8b406bc00 0x7fe8b406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:52.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.945+0000 7fe8c5ffb640 1 -- 172.21.15.67:0/2501958461 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe8c0089f70 con 0x7fe8d015dfc0 2024-03-24T00:40:52.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.945+0000 7fe8d5513640 1 --2- 172.21.15.67:0/2501958461 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe8b406bc00 0x7fe8b406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:52.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.946+0000 7fe8d5513640 1 --2- 172.21.15.67:0/2501958461 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe8b406bc00 0x7fe8b406e0c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc0011e0 tx=0x7fe8cc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:52.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.946+0000 7fe8da367740 1 Processor -- start 2024-03-24T00:40:52.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.946+0000 7fe8da367740 1 -- start start 2024-03-24T00:40:52.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.947+0000 7fe8da367740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d723b94e60 0x55d723bbb990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:52.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.947+0000 7fe8da367740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d723bbbed0 0x55d723b93ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:52.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.947+0000 7fe8da367740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d723b95240 0x55d723b943e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:52.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.947+0000 7fe8da367740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d723ad3c90 con 0x55d723b94e60 2024-03-24T00:40:52.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.947+0000 7fe8da367740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d723af9070 con 0x55d723bbbed0 2024-03-24T00:40:52.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.947+0000 7fe8da367740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d723aa1c80 con 0x55d723b95240 2024-03-24T00:40:52.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.947+0000 7fe8d5513640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d723b94e60 0x55d723bbb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:52.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.947+0000 7fe8d5513640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d723b94e60 0x55d723bbb990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41090/0 (socket says 172.21.15.67:41090) 2024-03-24T00:40:52.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.947+0000 7fe8d5513640 1 -- 172.21.15.67:0/1539361669 learned_addr learned my addr 172.21.15.67:0/1539361669 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:52.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.947+0000 7fe8d5d14640 1 --2- 172.21.15.67:0/1539361669 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d723b95240 0x55d723b943e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:52.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.947+0000 7fe8d4d12640 1 --2- 172.21.15.67:0/1539361669 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d723bbbed0 0x55d723b93ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:52.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.947+0000 7fe8d5513640 1 -- 172.21.15.67:0/1539361669 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d723b95240 msgr2=0x55d723b943e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:52.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.948+0000 7fe8d5513640 1 --2- 172.21.15.67:0/1539361669 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d723b95240 0x55d723b943e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.948+0000 7fe8d5513640 1 -- 172.21.15.67:0/1539361669 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d723bbbed0 msgr2=0x55d723b93ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:52.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.948+0000 7fe8d5513640 1 --2- 172.21.15.67:0/1539361669 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d723bbbed0 0x55d723b93ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.948+0000 7fe8d5513640 1 -- 172.21.15.67:0/1539361669 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d723a99d10 con 0x55d723b94e60 2024-03-24T00:40:52.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.948+0000 7fe8d5513640 1 --2- 172.21.15.67:0/1539361669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d723b94e60 0x55d723bbb990 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc005a00 tx=0x7fe8cc011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1e64247e07d168c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:52.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.948+0000 7fe8baffd640 1 -- 172.21.15.67:0/1539361669 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8cc00e7b0 con 0x55d723b94e60 2024-03-24T00:40:52.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.948+0000 7fe8baffd640 1 -- 172.21.15.67:0/1539361669 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8cc00e950 con 0x55d723b94e60 2024-03-24T00:40:52.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.949+0000 7fe8baffd640 1 -- 172.21.15.67:0/1539361669 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8cc00ed00 con 0x55d723b94e60 2024-03-24T00:40:52.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.949+0000 7fe8da367740 1 -- 172.21.15.67:0/1539361669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d723b94e60 msgr2=0x55d723bbb990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:52.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.949+0000 7fe8da367740 1 --2- 172.21.15.67:0/1539361669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d723b94e60 0x55d723bbb990 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc005a00 tx=0x7fe8cc011000 comp rx=0 tx=0).stop 2024-03-24T00:40:52.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.949+0000 7fe8da367740 1 -- 172.21.15.67:0/1539361669 shutdown_connections 2024-03-24T00:40:52.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.949+0000 7fe8da367740 1 --2- 172.21.15.67:0/1539361669 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d723b95240 0x55d723b943e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.949+0000 7fe8da367740 1 --2- 172.21.15.67:0/1539361669 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d723bbbed0 0x55d723b93ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.949+0000 7fe8da367740 1 --2- 172.21.15.67:0/1539361669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d723b94e60 0x55d723bbb990 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.949+0000 7fe8da367740 1 -- 172.21.15.67:0/1539361669 >> 172.21.15.67:0/1539361669 conn(0x55d723b5c1a0 msgr2=0x55d723bd9b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:52.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.949+0000 7fe8da367740 1 -- 172.21.15.67:0/1539361669 shutdown_connections 2024-03-24T00:40:52.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.949+0000 7fe8da367740 1 -- 172.21.15.67:0/1539361669 wait complete. 2024-03-24T00:40:52.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.950+0000 7fe8da367740 1 -- start start 2024-03-24T00:40:52.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.950+0000 7fe8da367740 1 Processor -- start 2024-03-24T00:40:52.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.951+0000 7fe8da367740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d723b94e60 0x55d723bdd0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:52.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.951+0000 7fe8da367740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d723b95240 0x55d723bed070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:52.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.951+0000 7fe8da367740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d723bb2f50 0x55d723bed5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:52.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.951+0000 7fe8da367740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d723ab50d0 con 0x55d723b94e60 2024-03-24T00:40:52.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.951+0000 7fe8da367740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d723af5110 con 0x55d723b95240 2024-03-24T00:40:52.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.951+0000 7fe8da367740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d723bbb500 con 0x55d723bb2f50 2024-03-24T00:40:52.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.951+0000 7fe8da367740 10 client.0 _ll_register_callbacks cb 0x55d723af0e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:40:52.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.951+0000 7fe8d5513640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d723b94e60 0x55d723bdd0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:52.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.951+0000 7fe8d5513640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d723b94e60 0x55d723bdd0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41106/0 (socket says 172.21.15.67:41106) 2024-03-24T00:40:52.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.951+0000 7fe8d5513640 1 -- 172.21.15.67:0/790743101 learned_addr learned my addr 172.21.15.67:0/790743101 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:52.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.952+0000 7fe8d4d12640 1 --2- 172.21.15.67:0/790743101 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d723b95240 0x55d723bed070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:52.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.952+0000 7fe8d5d14640 1 --2- 172.21.15.67:0/790743101 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d723bb2f50 0x55d723bed5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:52.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.952+0000 7fe8d5513640 1 -- 172.21.15.67:0/790743101 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d723bb2f50 msgr2=0x55d723bed5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:52.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.952+0000 7fe8d5513640 1 --2- 172.21.15.67:0/790743101 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d723bb2f50 0x55d723bed5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.952+0000 7fe8d5513640 1 -- 172.21.15.67:0/790743101 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d723b95240 msgr2=0x55d723bed070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:52.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.952+0000 7fe8d5513640 1 --2- 172.21.15.67:0/790743101 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d723b95240 0x55d723bed070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.952+0000 7fe8d5513640 1 -- 172.21.15.67:0/790743101 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d7238d57e0 con 0x55d723b94e60 2024-03-24T00:40:52.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.952+0000 7fe8d5d14640 1 --2- 172.21.15.67:0/790743101 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d723bb2f50 0x55d723bed5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:40:52.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.952+0000 7fe8d5513640 1 --2- 172.21.15.67:0/790743101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d723b94e60 0x55d723bdd0c0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc014000 tx=0x7fe8cc013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffdaa201561d7183 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:52.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.952+0000 7fe8affff640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:40:52.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.952+0000 7fe8affff640 1 -- 172.21.15.67:0/790743101 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8cc00e7b0 con 0x55d723b94e60 2024-03-24T00:40:52.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.952+0000 7fe8affff640 1 -- 172.21.15.67:0/790743101 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(50..50 src has 1..50) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe8cc02cc80 con 0x55d723b94e60 2024-03-24T00:40:52.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.952+0000 7fe8da367740 1 -- 172.21.15.67:0/790743101 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe8cc00e950 con 0x55d723b94e60 2024-03-24T00:40:52.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.952+0000 7fe8affff640 1 -- 172.21.15.67:0/790743101 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe8cc033050 con 0x55d723b94e60 2024-03-24T00:40:52.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.952+0000 7fe8da367740 10 client.6090 fetch_fsmap learned FSMap version 26 2024-03-24T00:40:52.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.952+0000 7fe8da367740 1 -- 172.21.15.67:0/790743101 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=2+}) v3 -- 0x55d723935f70 con 0x55d723b94e60 2024-03-24T00:40:52.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.952+0000 7fe8affff640 1 -- 172.21.15.67:0/790743101 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8cc00e950 con 0x55d723b94e60 2024-03-24T00:40:52.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.953+0000 7fe8affff640 1 -- 172.21.15.67:0/790743101 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 26) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fe8cc036020 con 0x55d723b94e60 2024-03-24T00:40:52.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.953+0000 7fe8da367740 10 client.6090 fetch_fsmap finished waiting for FSMap version 26 2024-03-24T00:40:52.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.953+0000 7fe8da367740 10 client.6090 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.a 2024-03-24T00:40:52.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.953+0000 7fe8da367740 20 client.6090 populate_metadata read hostname 'smithi067' 2024-03-24T00:40:52.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.953+0000 7fe8da367740 1 --2- 172.21.15.67:0/790743101 >> [v2:172.21.15.67:6832/2741818214,v1:172.21.15.67:6834/2741818214] conn(0x55d723bc1670 0x55d723bc1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:52.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.953+0000 7fe8da367740 4 client.6090 mds_command: new command op to 5853 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:40:52.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.953+0000 7fe8da367740 1 -- 172.21.15.67:0/790743101 --> [v2:172.21.15.67:6832/2741818214,v1:172.21.15.67:6834/2741818214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d723b065b0 con 0x55d723bc1670 2024-03-24T00:40:52.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.953+0000 7fe8d4d12640 1 --2- 172.21.15.67:0/790743101 >> [v2:172.21.15.67:6832/2741818214,v1:172.21.15.67:6834/2741818214] conn(0x55d723bc1670 0x55d723bc1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:52.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.953+0000 7fe8d4d12640 1 --2- 172.21.15.67:0/790743101 >> [v2:172.21.15.67:6832/2741818214,v1:172.21.15.67:6834/2741818214] conn(0x55d723bc1670 0x55d723bc1ab0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6900acf9b2eaaa5 server_cookie=fac4be88aa6301f7 in_seq=0 out_seq=0 2024-03-24T00:40:52.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.953+0000 7fe8affff640 10 client.6090 ms_handle_connect on v2:172.21.15.67:6832/2741818214 2024-03-24T00:40:52.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.954+0000 7fe8affff640 1 -- 172.21.15.67:0/790743101 <== mds.0 v2:172.21.15.67:6832/2741818214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d723b065b0 con 0x55d723bc1670 2024-03-24T00:40:52.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.954+0000 7fe8affff640 10 client.6090 handle_command_reply: tid=0 2024-03-24T00:40:52.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.954+0000 7fe8da367740 1 client.6090 shutdown 2024-03-24T00:40:52.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.954+0000 7fe8da367740 1 -- 172.21.15.67:0/790743101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d723b94e60 msgr2=0x55d723bdd0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:52.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.954+0000 7fe8da367740 1 --2- 172.21.15.67:0/790743101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d723b94e60 0x55d723bdd0c0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc014000 tx=0x7fe8cc013000 comp rx=0 tx=0).stop 2024-03-24T00:40:52.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.954+0000 7fe8da367740 1 -- 172.21.15.67:0/790743101 shutdown_connections 2024-03-24T00:40:52.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.954+0000 7fe8da367740 1 --2- 172.21.15.67:0/790743101 >> [v2:172.21.15.67:6832/2741818214,v1:172.21.15.67:6834/2741818214] conn(0x55d723bc1670 0x55d723bc1ab0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.954+0000 7fe8da367740 1 --2- 172.21.15.67:0/790743101 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d723bb2f50 0x55d723bed5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.954+0000 7fe8da367740 1 --2- 172.21.15.67:0/790743101 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d723b95240 0x55d723bed070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.954+0000 7fe8affff640 0 client.6090 ms_handle_reset on v2:172.21.15.67:6832/2741818214 2024-03-24T00:40:52.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.954+0000 7fe8da367740 1 --2- 172.21.15.67:0/790743101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d723b94e60 0x55d723bdd0c0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.954+0000 7fe8da367740 1 -- 172.21.15.67:0/790743101 >> 172.21.15.67:0/790743101 conn(0x55d723b5c1a0 msgr2=0x55d723bbf0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:52.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.954+0000 7fe8da367740 1 -- 172.21.15.67:0/790743101 shutdown_connections 2024-03-24T00:40:52.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.954+0000 7fe8da367740 1 -- 172.21.15.67:0/790743101 wait complete. 2024-03-24T00:40:52.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.954+0000 7fe8da367740 20 client.6090 trim_cache size 0 max 16384 2024-03-24T00:40:52.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.971+0000 7fe8da367740 1 Processor -- start 2024-03-24T00:40:52.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.971+0000 7fe8da367740 1 -- start start 2024-03-24T00:40:52.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.971+0000 7fe8da367740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d723b94e60 0x55d723bbed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:52.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.971+0000 7fe8da367740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d723b95240 0x55d723becd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:52.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.971+0000 7fe8da367740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d723bb2f50 0x55d723bed2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:52.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.971+0000 7fe8da367740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d0200770 con 0x55d723bb2f50 2024-03-24T00:40:52.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.971+0000 7fe8da367740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d01ff8d0 con 0x55d723b95240 2024-03-24T00:40:52.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.971+0000 7fe8da367740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d01607f0 con 0x55d723b94e60 2024-03-24T00:40:52.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.971+0000 7fe8d5d14640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d723bb2f50 0x55d723bed2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:52.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.971+0000 7fe8d5d14640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d723bb2f50 0x55d723bed2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41120/0 (socket says 172.21.15.67:41120) 2024-03-24T00:40:52.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.971+0000 7fe8d5d14640 1 -- 172.21.15.67:0/2299896519 learned_addr learned my addr 172.21.15.67:0/2299896519 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:52.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.972+0000 7fe8d5513640 1 --2- 172.21.15.67:0/2299896519 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d723b94e60 0x55d723bbed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:52.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.972+0000 7fe8d4d12640 1 --2- 172.21.15.67:0/2299896519 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d723b95240 0x55d723becd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:52.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.972+0000 7fe8d5d14640 1 -- 172.21.15.67:0/2299896519 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d723b94e60 msgr2=0x55d723bbed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:52.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.972+0000 7fe8d5d14640 1 --2- 172.21.15.67:0/2299896519 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d723b94e60 0x55d723bbed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.972+0000 7fe8d5d14640 1 -- 172.21.15.67:0/2299896519 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d723b95240 msgr2=0x55d723becd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:52.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.972+0000 7fe8d5d14640 1 --2- 172.21.15.67:0/2299896519 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d723b95240 0x55d723becd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.972+0000 7fe8d5d14640 1 -- 172.21.15.67:0/2299896519 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d7237cd170 con 0x55d723bb2f50 2024-03-24T00:40:52.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.972+0000 7fe8d4d12640 1 --2- 172.21.15.67:0/2299896519 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d723b95240 0x55d723becd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:40:52.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.972+0000 7fe8d5513640 1 --2- 172.21.15.67:0/2299896519 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d723b94e60 0x55d723bbed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:40:52.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.972+0000 7fe8d5d14640 1 --2- 172.21.15.67:0/2299896519 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d723bb2f50 0x55d723bed2a0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fe8c8009d90 tx=0x7fe8c802e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa836e168b561219 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:52.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.972+0000 7fe8ba7fc640 1 -- 172.21.15.67:0/2299896519 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c8039040 con 0x55d723bb2f50 2024-03-24T00:40:52.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.972+0000 7fe8ba7fc640 1 -- 172.21.15.67:0/2299896519 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8c8045070 con 0x55d723bb2f50 2024-03-24T00:40:52.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.972+0000 7fe8ba7fc640 1 -- 172.21.15.67:0/2299896519 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c802cd50 con 0x55d723bb2f50 2024-03-24T00:40:52.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.973+0000 7fe8da367740 1 -- 172.21.15.67:0/2299896519 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d723bb2f50 msgr2=0x55d723bed2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:52.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.973+0000 7fe8da367740 1 --2- 172.21.15.67:0/2299896519 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d723bb2f50 0x55d723bed2a0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fe8c8009d90 tx=0x7fe8c802e000 comp rx=0 tx=0).stop 2024-03-24T00:40:52.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.973+0000 7fe8da367740 1 -- 172.21.15.67:0/2299896519 shutdown_connections 2024-03-24T00:40:52.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.973+0000 7fe8da367740 1 --2- 172.21.15.67:0/2299896519 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d723bb2f50 0x55d723bed2a0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.973+0000 7fe8da367740 1 --2- 172.21.15.67:0/2299896519 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d723b95240 0x55d723becd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.973+0000 7fe8da367740 1 --2- 172.21.15.67:0/2299896519 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d723b94e60 0x55d723bbed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.973+0000 7fe8da367740 1 -- 172.21.15.67:0/2299896519 >> 172.21.15.67:0/2299896519 conn(0x55d723b5c1a0 msgr2=0x55d723bc1a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:52.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.973+0000 7fe8da367740 1 -- 172.21.15.67:0/2299896519 shutdown_connections 2024-03-24T00:40:52.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.973+0000 7fe8da367740 1 -- 172.21.15.67:0/2299896519 wait complete. 2024-03-24T00:40:52.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.973+0000 7fe8da367740 1 -- start start 2024-03-24T00:40:52.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.973+0000 7fe8da367740 1 Processor -- start 2024-03-24T00:40:52.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.974+0000 7fe8da367740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d723b94e60 0x55d723b928f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:52.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.974+0000 7fe8da367740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d723b95240 0x55d723bba120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:52.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.974+0000 7fe8da367740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d723bb2f50 0x55d723bba660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:52.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.974+0000 7fe8da367740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d723bdb200 con 0x55d723b94e60 2024-03-24T00:40:52.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.974+0000 7fe8da367740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d015d0a0 con 0x55d723b95240 2024-03-24T00:40:52.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.974+0000 7fe8da367740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d723bbb7e0 con 0x55d723bb2f50 2024-03-24T00:40:52.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.974+0000 7fe8da367740 10 client.0 _ll_register_callbacks cb 0x7fe8c00545b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:40:52.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.974+0000 7fe8d5513640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d723b94e60 0x55d723b928f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:52.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.975+0000 7fe8d5513640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d723b94e60 0x55d723b928f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41122/0 (socket says 172.21.15.67:41122) 2024-03-24T00:40:52.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.975+0000 7fe8d5513640 1 -- 172.21.15.67:0/1451174634 learned_addr learned my addr 172.21.15.67:0/1451174634 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:52.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.975+0000 7fe8d4d12640 1 --2- 172.21.15.67:0/1451174634 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d723b95240 0x55d723bba120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:52.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.975+0000 7fe8d5d14640 1 --2- 172.21.15.67:0/1451174634 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d723bb2f50 0x55d723bba660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:52.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.975+0000 7fe8d5513640 1 -- 172.21.15.67:0/1451174634 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d723bb2f50 msgr2=0x55d723bba660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:52.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.975+0000 7fe8d5513640 1 --2- 172.21.15.67:0/1451174634 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d723bb2f50 0x55d723bba660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.975+0000 7fe8d5513640 1 -- 172.21.15.67:0/1451174634 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d723b95240 msgr2=0x55d723bba120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:52.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.975+0000 7fe8d5513640 1 --2- 172.21.15.67:0/1451174634 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d723b95240 0x55d723bba120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.975+0000 7fe8d5513640 1 -- 172.21.15.67:0/1451174634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d723be9850 con 0x55d723b94e60 2024-03-24T00:40:52.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.975+0000 7fe8d5513640 1 --2- 172.21.15.67:0/1451174634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d723b94e60 0x55d723b928f0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc007040 tx=0x7fe8cc001150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98f81274a9299d46 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:52.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.975+0000 7fe8affff640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:40:52.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.975+0000 7fe8affff640 1 -- 172.21.15.67:0/1451174634 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8cc03d020 con 0x55d723b94e60 2024-03-24T00:40:52.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.975+0000 7fe8affff640 1 -- 172.21.15.67:0/1451174634 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(50..50 src has 1..50) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe8cc00efb0 con 0x55d723b94e60 2024-03-24T00:40:52.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.975+0000 7fe8da367740 1 -- 172.21.15.67:0/1451174634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe8c8045070 con 0x55d723b94e60 2024-03-24T00:40:52.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.975+0000 7fe8affff640 1 -- 172.21.15.67:0/1451174634 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe8cc0218b0 con 0x55d723b94e60 2024-03-24T00:40:52.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.976+0000 7fe8da367740 10 client.6096 fetch_fsmap learned FSMap version 26 2024-03-24T00:40:52.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.976+0000 7fe8da367740 1 -- 172.21.15.67:0/1451174634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=2+}) v3 -- 0x55d723bdbc80 con 0x55d723b94e60 2024-03-24T00:40:52.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.976+0000 7fe8affff640 1 -- 172.21.15.67:0/1451174634 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8c8045070 con 0x55d723b94e60 2024-03-24T00:40:52.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.976+0000 7fe8affff640 1 -- 172.21.15.67:0/1451174634 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 26) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fe8cc036040 con 0x55d723b94e60 2024-03-24T00:40:52.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.976+0000 7fe8da367740 10 client.6096 fetch_fsmap finished waiting for FSMap version 26 2024-03-24T00:40:52.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.976+0000 7fe8da367740 10 client.6096 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.a 2024-03-24T00:40:52.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.976+0000 7fe8da367740 20 client.6096 populate_metadata read hostname 'smithi067' 2024-03-24T00:40:52.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.976+0000 7fe8da367740 1 --2- 172.21.15.67:0/1451174634 >> [v2:172.21.15.67:6832/2741818214,v1:172.21.15.67:6834/2741818214] conn(0x55d723b93ae0 0x55d723c521f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:52.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.976+0000 7fe8da367740 4 client.6096 mds_command: new command op to 5853 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive"]}] 2024-03-24T00:40:52.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.976+0000 7fe8da367740 1 -- 172.21.15.67:0/1451174634 --> [v2:172.21.15.67:6832/2741818214,v1:172.21.15.67:6834/2741818214] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive"]}) v1 -- 0x55d723b00900 con 0x55d723b93ae0 2024-03-24T00:40:52.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.976+0000 7fe8d4d12640 1 --2- 172.21.15.67:0/1451174634 >> [v2:172.21.15.67:6832/2741818214,v1:172.21.15.67:6834/2741818214] conn(0x55d723b93ae0 0x55d723c521f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:52.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.976+0000 7fe8affff640 10 client.6096 ms_handle_connect on v2:172.21.15.67:6832/2741818214 2024-03-24T00:40:52.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.977+0000 7fe8d4d12640 1 --2- 172.21.15.67:0/1451174634 >> [v2:172.21.15.67:6832/2741818214,v1:172.21.15.67:6834/2741818214] conn(0x55d723b93ae0 0x55d723c521f0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f6e654393bc7a25 server_cookie=55010dec1c6f2802 in_seq=0 out_seq=0 2024-03-24T00:40:52.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.977+0000 7fe8affff640 1 -- 172.21.15.67:0/1451174634 <== mds.0 v2:172.21.15.67:6832/2741818214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55d723b00900 con 0x55d723b93ae0 2024-03-24T00:40:52.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.977+0000 7fe8affff640 10 client.6096 handle_command_reply: tid=0 2024-03-24T00:40:52.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.977+0000 7fe8da367740 1 client.6096 shutdown 2024-03-24T00:40:52.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.977+0000 7fe8da367740 1 -- 172.21.15.67:0/1451174634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d723b94e60 msgr2=0x55d723b928f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:52.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.977+0000 7fe8da367740 1 --2- 172.21.15.67:0/1451174634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d723b94e60 0x55d723b928f0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc007040 tx=0x7fe8cc001150 comp rx=0 tx=0).stop 2024-03-24T00:40:52.982 INFO:tasks.ceph.mds.a.smithi067.stderr:2024-03-24T00:40:52.978+0000 7fbe5a6ad640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [2,head] /parent_a/alpha auth v2 ap=1 DIRTYPARENT s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={5453=0-4194304@1} caps={5453=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=5453 | request=0 caps=1 dirtyparent=1 dirty=0 authpin=1 scrubqueue=0 0x564e71175700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(8)0x10000000001:[<0x10000000000/alpha v2>,<0x1/parent_a v4>]//[]","error_str":"failed to decode on-disk backtrace (40 bytes)!"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-24T00:40:52.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.978+0000 7fe8da367740 1 -- 172.21.15.67:0/1451174634 shutdown_connections 2024-03-24T00:40:52.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.978+0000 7fe8da367740 1 --2- 172.21.15.67:0/1451174634 >> [v2:172.21.15.67:6832/2741818214,v1:172.21.15.67:6834/2741818214] conn(0x55d723b93ae0 0x55d723c521f0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.978+0000 7fe8da367740 1 --2- 172.21.15.67:0/1451174634 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d723bb2f50 0x55d723bba660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.978+0000 7fe8da367740 1 --2- 172.21.15.67:0/1451174634 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d723b95240 0x55d723bba120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.978+0000 7fe8da367740 1 --2- 172.21.15.67:0/1451174634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d723b94e60 0x55d723b928f0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.978+0000 7fe8da367740 1 -- 172.21.15.67:0/1451174634 >> 172.21.15.67:0/1451174634 conn(0x55d723b5c1a0 msgr2=0x55d723bed3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:52.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.978+0000 7fe8affff640 0 client.6096 ms_handle_reset on v2:172.21.15.67:6832/2741818214 2024-03-24T00:40:52.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.978+0000 7fe8da367740 1 -- 172.21.15.67:0/1451174634 shutdown_connections 2024-03-24T00:40:52.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.978+0000 7fe8da367740 1 -- 172.21.15.67:0/1451174634 wait complete. 2024-03-24T00:40:52.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.978+0000 7fe8da367740 20 client.6096 trim_cache size 0 max 16384 2024-03-24T00:40:52.984 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:40:52.984 INFO:teuthology.orchestra.run.smithi067.stdout: "return_code": 0, 2024-03-24T00:40:52.984 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_tag": "56385fa2-25bc-4ca2-a9a5-2eab70b67325", 2024-03-24T00:40:52.985 INFO:teuthology.orchestra.run.smithi067.stdout: "mode": "asynchronous" 2024-03-24T00:40:52.985 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:40:52.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.979+0000 7fe8d7777640 1 -- 172.21.15.67:0/2501958461 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe8b406bc00 msgr2=0x7fe8b406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:52.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.979+0000 7fe8d7777640 1 --2- 172.21.15.67:0/2501958461 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe8b406bc00 0x7fe8b406e0c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc0011e0 tx=0x7fe8cc009040 comp rx=0 tx=0).stop 2024-03-24T00:40:52.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.979+0000 7fe8d7777640 1 -- 172.21.15.67:0/2501958461 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8d015dfc0 msgr2=0x7fe8d00b5c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:52.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.979+0000 7fe8d7777640 1 --2- 172.21.15.67:0/2501958461 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8d015dfc0 0x7fe8d00b5c50 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fe8c000c3c0 tx=0x7fe8c000c880 comp rx=0 tx=0).stop 2024-03-24T00:40:52.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.979+0000 7fe8d7777640 1 -- 172.21.15.67:0/2501958461 shutdown_connections 2024-03-24T00:40:52.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.979+0000 7fe8d7777640 1 --2- 172.21.15.67:0/2501958461 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe8b406bc00 0x7fe8b406e0c0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.979+0000 7fe8d7777640 1 --2- 172.21.15.67:0/2501958461 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe8d01f8250 0x7fe8d0152b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.979+0000 7fe8d7777640 1 --2- 172.21.15.67:0/2501958461 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8d015dfc0 0x7fe8d00b5c50 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.979+0000 7fe8d7777640 1 --2- 172.21.15.67:0/2501958461 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8d00d0d50 0x7fe8d00cb4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.979+0000 7fe8d7777640 1 -- 172.21.15.67:0/2501958461 >> 172.21.15.67:0/2501958461 conn(0x7fe8d006c880 msgr2=0x7fe8d00d3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:52.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.979+0000 7fe8d7777640 1 -- 172.21.15.67:0/2501958461 shutdown_connections 2024-03-24T00:40:52.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.979+0000 7fe8d7777640 1 -- 172.21.15.67:0/2501958461 wait complete. 2024-03-24T00:40:52.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.980+0000 7f5faf7a9640 1 Processor -- start 2024-03-24T00:40:52.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.981+0000 7f5faf7a9640 1 -- start start 2024-03-24T00:40:52.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.981+0000 7f5faf7a9640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5fa815e3c0 0x7f5fa80cd490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:52.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.981+0000 7f5faf7a9640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5fa80cf710 0x7f5fa80cd9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:52.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.981+0000 7f5faf7a9640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fa80b0400 con 0x7f5fa80b0bc0 2024-03-24T00:40:52.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.981+0000 7f5faf7a9640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fa80ce710 con 0x7f5fa815e3c0 2024-03-24T00:40:52.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.981+0000 7f5faf7a9640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa80ce9f0 con 0x7f5fa80cf710 2024-03-24T00:40:52.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.982+0000 7f5fadd46640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5fa80b0bc0 0x7f5fa80b1050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39716/0 (socket says 172.21.15.67:39716) 2024-03-24T00:40:52.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.982+0000 7f5fadd46640 1 -- 172.21.15.67:0/2131010764 learned_addr learned my addr 172.21.15.67:0/2131010764 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:40:52.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.982+0000 7f5fad545640 1 --2- 172.21.15.67:0/2131010764 >> v2:172.21.15.72:3300/0 conn(0x7f5fa815e3c0 0x7f5fa80cd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:52.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.982+0000 7f5facd44640 1 --2- 172.21.15.67:0/2131010764 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5fa80cf710 0x7f5fa80cd9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:52.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.982+0000 7f5f9ffff640 1 -- 172.21.15.67:0/2131010764 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2568023727 0 0) 0x7f5fa80b0400 con 0x7f5fa80b0bc0 2024-03-24T00:40:52.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.982+0000 7f5f9ffff640 1 -- 172.21.15.67:0/2131010764 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f94003520 con 0x7f5fa80b0bc0 2024-03-24T00:40:52.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.982+0000 7f5f9ffff640 1 -- 172.21.15.67:0/2131010764 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4009428158 0 0) 0x7f5f94003520 con 0x7f5fa80b0bc0 2024-03-24T00:40:52.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.982+0000 7f5facd44640 1 -- 172.21.15.67:0/2131010764 >> v2:172.21.15.72:3300/0 conn(0x7f5fa815e3c0 msgr2=0x7f5fa80cd490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:52.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.982+0000 7f5facd44640 1 --2- 172.21.15.67:0/2131010764 >> v2:172.21.15.72:3300/0 conn(0x7f5fa815e3c0 0x7f5fa80cd490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.982+0000 7f5facd44640 1 -- 172.21.15.67:0/2131010764 >> v1:172.21.15.67:6789/0 conn(0x7f5fa80b0bc0 legacy=0x7f5fa80b1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:52.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.982+0000 7f5facd44640 1 -- 172.21.15.67:0/2131010764 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa80b06a0 con 0x7f5fa80cf710 2024-03-24T00:40:52.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.982+0000 7f5fad545640 1 --2- 172.21.15.67:0/2131010764 >> v2:172.21.15.72:3300/0 conn(0x7f5fa815e3c0 0x7f5fa80cd490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:40:52.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.983+0000 7f5facd44640 1 --2- 172.21.15.67:0/2131010764 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5fa80cf710 0x7f5fa80cd9d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5f980097a0 tx=0x7f5f9802b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9bb313a3a509a3d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:52.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.983+0000 7f5f9ffff640 1 -- 172.21.15.67:0/2131010764 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f98037070 con 0x7f5fa80cf710 2024-03-24T00:40:52.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.983+0000 7f5f9ffff640 1 -- 172.21.15.67:0/2131010764 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f98030510 con 0x7f5fa80cf710 2024-03-24T00:40:52.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.983+0000 7f5f9ffff640 1 -- 172.21.15.67:0/2131010764 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa00019f0 con 0x7f5fa80cf710 2024-03-24T00:40:52.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.984+0000 7f5faf7a9640 1 -- 172.21.15.67:0/2131010764 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5fa80cf710 msgr2=0x7f5fa80cd9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:52.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.984+0000 7f5faf7a9640 1 --2- 172.21.15.67:0/2131010764 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5fa80cf710 0x7f5fa80cd9d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5f980097a0 tx=0x7f5f9802b790 comp rx=0 tx=0).stop 2024-03-24T00:40:52.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.984+0000 7f5faf7a9640 1 -- 172.21.15.67:0/2131010764 shutdown_connections 2024-03-24T00:40:52.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.984+0000 7f5faf7a9640 1 --2- 172.21.15.67:0/2131010764 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5fa80cf710 0x7f5fa80cd9d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.984+0000 7f5faf7a9640 1 --2- 172.21.15.67:0/2131010764 >> v2:172.21.15.72:3300/0 conn(0x7f5fa815e3c0 0x7f5fa80cd490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.984+0000 7f5faf7a9640 1 -- 172.21.15.67:0/2131010764 >> 172.21.15.67:0/2131010764 conn(0x7f5fa80d0aa0 msgr2=0x7f5fa80d0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:52.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.984+0000 7f5faf7a9640 1 -- 172.21.15.67:0/2131010764 shutdown_connections 2024-03-24T00:40:52.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.984+0000 7f5faf7a9640 1 -- 172.21.15.67:0/2131010764 wait complete. 2024-03-24T00:40:52.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.985+0000 7f5faf7a9640 1 Processor -- start 2024-03-24T00:40:52.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.985+0000 7f5faf7a9640 1 -- start start 2024-03-24T00:40:52.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.986+0000 7f5faf7a9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fa80b0bc0 0x7f5fa80b43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:52.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.986+0000 7f5faf7a9640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5fa80cf710 0x7f5fa81eea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:52.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.986+0000 7f5faf7a9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fa815e3c0 0x7f5fa81eefa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:52.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.986+0000 7f5faf7a9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa80d15c0 con 0x7f5fa815e3c0 2024-03-24T00:40:52.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.986+0000 7f5faf7a9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa80d1930 con 0x7f5fa80b0bc0 2024-03-24T00:40:52.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.986+0000 7f5faf7a9640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa815e7a0 con 0x7f5fa80cf710 2024-03-24T00:40:52.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.986+0000 7f5fadd46640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fa815e3c0 0x7f5fa81eefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:52.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.986+0000 7f5fadd46640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fa815e3c0 0x7f5fa81eefa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41136/0 (socket says 172.21.15.67:41136) 2024-03-24T00:40:52.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.986+0000 7f5fadd46640 1 -- 172.21.15.67:0/1383122620 learned_addr learned my addr 172.21.15.67:0/1383122620 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:52.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.986+0000 7f5facd44640 1 --2- 172.21.15.67:0/1383122620 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5fa80cf710 0x7f5fa81eea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:52.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.986+0000 7f5fad545640 1 --2- 172.21.15.67:0/1383122620 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fa80b0bc0 0x7f5fa80b43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:52.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.987+0000 7f5fadd46640 1 -- 172.21.15.67:0/1383122620 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5fa80cf710 msgr2=0x7f5fa81eea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:52.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.987+0000 7f5fadd46640 1 --2- 172.21.15.67:0/1383122620 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5fa80cf710 0x7f5fa81eea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.987+0000 7f5fadd46640 1 -- 172.21.15.67:0/1383122620 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fa80b0bc0 msgr2=0x7f5fa80b43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:52.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.987+0000 7f5fadd46640 1 --2- 172.21.15.67:0/1383122620 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fa80b0bc0 0x7f5fa80b43e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:52.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.987+0000 7f5fadd46640 1 -- 172.21.15.67:0/1383122620 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa80ae440 con 0x7f5fa815e3c0 2024-03-24T00:40:52.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.987+0000 7f5fadd46640 1 --2- 172.21.15.67:0/1383122620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fa815e3c0 0x7f5fa81eefa0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f5fa000cc90 tx=0x7f5fa000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:52.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.987+0000 7f5f9dffb640 1 -- 172.21.15.67:0/1383122620 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa000ce40 con 0x7f5fa815e3c0 2024-03-24T00:40:52.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.987+0000 7f5f9dffb640 1 -- 172.21.15.67:0/1383122620 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fa0012400 con 0x7f5fa815e3c0 2024-03-24T00:40:52.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.987+0000 7f5f9dffb640 1 -- 172.21.15.67:0/1383122620 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa00127f0 con 0x7f5fa815e3c0 2024-03-24T00:40:52.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.987+0000 7f5faf7a9640 1 -- 172.21.15.67:0/1383122620 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fa80ae720 con 0x7f5fa815e3c0 2024-03-24T00:40:52.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.987+0000 7f5faf7a9640 1 -- 172.21.15.67:0/1383122620 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fa80aece0 con 0x7f5fa815e3c0 2024-03-24T00:40:52.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.988+0000 7f5faf7a9640 1 -- 172.21.15.67:0/1383122620 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f5fa804f400 con 0x7f5fa815e3c0 2024-03-24T00:40:52.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.989+0000 7f5f9dffb640 1 -- 172.21.15.67:0/1383122620 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5fa0012990 con 0x7f5fa815e3c0 2024-03-24T00:40:52.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.990+0000 7f5f9dffb640 1 --2- 172.21.15.67:0/1383122620 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5f8c06bbd0 0x7f5f8c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:52.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.990+0000 7f5f9dffb640 1 -- 172.21.15.67:0/1383122620 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5fa008b230 con 0x7f5fa815e3c0 2024-03-24T00:40:52.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.990+0000 7f5f9dffb640 1 -- 172.21.15.67:0/1383122620 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+501 (secure 0 0 0) 0x7f5fa008d8b0 con 0x7f5fa815e3c0 2024-03-24T00:40:52.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.990+0000 7f5faf7a9640 1 -- 172.21.15.67:0/1383122620 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({log-info=0+}) v3 -- 0x7f5fa80b6780 con 0x7f5fa815e3c0 2024-03-24T00:40:52.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.990+0000 7f5fad545640 1 --2- 172.21.15.67:0/1383122620 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5f8c06bbd0 0x7f5f8c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:52.996 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:40:52.996 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.3:0 scrub status 2024-03-24T00:40:52.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:52.991+0000 7f5fad545640 1 --2- 172.21.15.67:0/1383122620 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5f8c06bbd0 0x7f5f8c06e090 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f5fa4009560 tx=0x7f5fa4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:53.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.099+0000 7f638adab640 1 Processor -- start 2024-03-24T00:40:53.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.099+0000 7f638adab640 1 -- start start 2024-03-24T00:40:53.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.100+0000 7f638adab640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f638415dea0 0x7f63840d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:53.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.100+0000 7f638adab640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f638415e280 0x7f63840d3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:53.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.100+0000 7f638adab640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f63840b5a60 0x7f63840b5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:53.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.100+0000 7f638adab640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63840b63f0 con 0x7f638415dea0 2024-03-24T00:40:53.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.100+0000 7f638adab640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63840d2250 con 0x7f638415e280 2024-03-24T00:40:53.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.100+0000 7f638adab640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63840d2530 con 0x7f63840b5a60 2024-03-24T00:40:53.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.100+0000 7f6388b47640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f638415dea0 0x7f63840d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:53.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.100+0000 7f6388b47640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f638415dea0 0x7f63840d33d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41146/0 (socket says 172.21.15.67:41146) 2024-03-24T00:40:53.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.100+0000 7f6388b47640 1 -- 172.21.15.67:0/1149597699 learned_addr learned my addr 172.21.15.67:0/1149597699 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:53.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.100+0000 7f6388b47640 1 -- 172.21.15.67:0/1149597699 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f63840b5a60 msgr2=0x7f63840b5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:53.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.100+0000 7f6388b47640 1 --2- 172.21.15.67:0/1149597699 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f63840b5a60 0x7f63840b5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:53.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.100+0000 7f6388b47640 1 -- 172.21.15.67:0/1149597699 >> v2:172.21.15.72:3300/0 conn(0x7f638415e280 msgr2=0x7f63840d3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:53.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.100+0000 7f6388b47640 1 --2- 172.21.15.67:0/1149597699 >> v2:172.21.15.72:3300/0 conn(0x7f638415e280 0x7f63840d3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:53.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.100+0000 7f6388b47640 1 -- 172.21.15.67:0/1149597699 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63840d2810 con 0x7f638415dea0 2024-03-24T00:40:53.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.101+0000 7f6388b47640 1 --2- 172.21.15.67:0/1149597699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f638415dea0 0x7f63840d33d0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f63840d04d0 tx=0x7f637802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=850b2b91ac886092 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:53.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.101+0000 7f63837fe640 1 -- 172.21.15.67:0/1149597699 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f637802e070 con 0x7f638415dea0 2024-03-24T00:40:53.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.101+0000 7f63837fe640 1 -- 172.21.15.67:0/1149597699 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6378037070 con 0x7f638415dea0 2024-03-24T00:40:53.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.101+0000 7f63837fe640 1 -- 172.21.15.67:0/1149597699 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6378031a00 con 0x7f638415dea0 2024-03-24T00:40:53.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.102+0000 7f638adab640 1 -- 172.21.15.67:0/1149597699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f638415dea0 msgr2=0x7f63840d33d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:53.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.102+0000 7f638adab640 1 --2- 172.21.15.67:0/1149597699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f638415dea0 0x7f63840d33d0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f63840d04d0 tx=0x7f637802b880 comp rx=0 tx=0).stop 2024-03-24T00:40:53.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.102+0000 7f638adab640 1 -- 172.21.15.67:0/1149597699 shutdown_connections 2024-03-24T00:40:53.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.102+0000 7f638adab640 1 --2- 172.21.15.67:0/1149597699 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f63840b5a60 0x7f63840b5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:53.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.102+0000 7f638adab640 1 --2- 172.21.15.67:0/1149597699 >> v2:172.21.15.72:3300/0 conn(0x7f638415e280 0x7f63840d3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:53.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.102+0000 7f638adab640 1 --2- 172.21.15.67:0/1149597699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f638415dea0 0x7f63840d33d0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:53.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.102+0000 7f638adab640 1 -- 172.21.15.67:0/1149597699 >> 172.21.15.67:0/1149597699 conn(0x7f6384072da0 msgr2=0x7f63840d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:53.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.102+0000 7f638adab640 1 -- 172.21.15.67:0/1149597699 shutdown_connections 2024-03-24T00:40:53.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.102+0000 7f638adab640 1 -- 172.21.15.67:0/1149597699 wait complete. 2024-03-24T00:40:53.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.103+0000 7f638adab640 1 Processor -- start 2024-03-24T00:40:53.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.103+0000 7f638adab640 1 -- start start 2024-03-24T00:40:53.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.104+0000 7f638adab640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63840b5a60 0x7f63840b6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:53.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.104+0000 7f638adab640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f638415dea0 0x7f63840b71a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:53.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.104+0000 7f638adab640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f638415e280 0x7f63840cddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:53.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.104+0000 7f638adab640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f638406c7c0 con 0x7f638415e280 2024-03-24T00:40:53.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.104+0000 7f638adab640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f638406c640 con 0x7f63840b5a60 2024-03-24T00:40:53.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.104+0000 7f638adab640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f638406c940 con 0x7f638415dea0 2024-03-24T00:40:53.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.104+0000 7f6389348640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f638415e280 0x7f63840cddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:53.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.104+0000 7f6389348640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f638415e280 0x7f63840cddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41150/0 (socket says 172.21.15.67:41150) 2024-03-24T00:40:53.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.104+0000 7f6389348640 1 -- 172.21.15.67:0/1659736699 learned_addr learned my addr 172.21.15.67:0/1659736699 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:53.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.104+0000 7f6389348640 1 -- 172.21.15.67:0/1659736699 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f638415dea0 msgr2=0x7f63840b71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:53.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.104+0000 7f6389348640 1 --2- 172.21.15.67:0/1659736699 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f638415dea0 0x7f63840b71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:53.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.104+0000 7f6389348640 1 -- 172.21.15.67:0/1659736699 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63840b5a60 msgr2=0x7f63840b6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:53.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.104+0000 7f6388b47640 1 --2- 172.21.15.67:0/1659736699 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63840b5a60 0x7f63840b6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:53.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.104+0000 7f6389348640 1 --2- 172.21.15.67:0/1659736699 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63840b5a60 0x7f63840b6c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:53.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.104+0000 7f6389348640 1 -- 172.21.15.67:0/1659736699 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63841f3e30 con 0x7f638415e280 2024-03-24T00:40:53.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.104+0000 7f6389348640 1 --2- 172.21.15.67:0/1659736699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f638415e280 0x7f63840cddc0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f637c00b9e0 tx=0x7f637c00bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:53.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.105+0000 7f63817fa640 1 -- 172.21.15.67:0/1659736699 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f637c019070 con 0x7f638415e280 2024-03-24T00:40:53.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.105+0000 7f63817fa640 1 -- 172.21.15.67:0/1659736699 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f637c00cac0 con 0x7f638415e280 2024-03-24T00:40:53.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.105+0000 7f638adab640 1 -- 172.21.15.67:0/1659736699 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63841f4110 con 0x7f638415e280 2024-03-24T00:40:53.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.105+0000 7f63817fa640 1 -- 172.21.15.67:0/1659736699 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f637c014440 con 0x7f638415e280 2024-03-24T00:40:53.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.105+0000 7f638adab640 1 -- 172.21.15.67:0/1659736699 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63841f46d0 con 0x7f638415e280 2024-03-24T00:40:53.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.106+0000 7f63817fa640 1 -- 172.21.15.67:0/1659736699 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f637c01c070 con 0x7f638415e280 2024-03-24T00:40:53.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.106+0000 7f63817fa640 1 --2- 172.21.15.67:0/1659736699 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f636806bbd0 0x7f636806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:53.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.107+0000 7f63817fa640 1 -- 172.21.15.67:0/1659736699 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f637c08a8a0 con 0x7f638415e280 2024-03-24T00:40:53.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.107+0000 7f6388b47640 1 --2- 172.21.15.67:0/1659736699 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f636806bbd0 0x7f636806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:53.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.107+0000 7f6388b47640 1 --2- 172.21.15.67:0/1659736699 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f636806bbd0 0x7f636806e090 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f6378036000 tx=0x7f6378008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:53.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.108+0000 7f638d98b740 1 Processor -- start 2024-03-24T00:40:53.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.108+0000 7f638d98b740 1 -- start start 2024-03-24T00:40:53.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.108+0000 7f638d98b740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b9d5884e50 0x55b9d5857df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:53.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.108+0000 7f638d98b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9d5885230 0x55b9d5887be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:53.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.108+0000 7f638d98b740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b9d5858330 0x55b9d5888120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:53.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.108+0000 7f638d98b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9d57c23d0 con 0x55b9d5885230 2024-03-24T00:40:53.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.108+0000 7f638d98b740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9d576aad0 con 0x55b9d5858330 2024-03-24T00:40:53.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.108+0000 7f638d98b740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b9d57be470 con 0x55b9d5884e50 2024-03-24T00:40:53.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.109+0000 7f6383fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9d5885230 0x55b9d5887be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:53.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.109+0000 7f6383fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9d5885230 0x55b9d5887be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41160/0 (socket says 172.21.15.67:41160) 2024-03-24T00:40:53.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.109+0000 7f6383fff640 1 -- 172.21.15.67:0/1045759064 learned_addr learned my addr 172.21.15.67:0/1045759064 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:53.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.109+0000 7f6389348640 1 --2- 172.21.15.67:0/1045759064 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b9d5858330 0x55b9d5888120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:53.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.109+0000 7f6388b47640 1 --2- 172.21.15.67:0/1045759064 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b9d5884e50 0x55b9d5857df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:53.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.109+0000 7f6383fff640 1 -- 172.21.15.67:0/1045759064 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b9d5884e50 msgr2=0x55b9d5857df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:53.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.109+0000 7f6383fff640 1 --2- 172.21.15.67:0/1045759064 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b9d5884e50 0x55b9d5857df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:53.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.109+0000 7f6383fff640 1 -- 172.21.15.67:0/1045759064 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b9d5858330 msgr2=0x55b9d5888120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:53.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.109+0000 7f6383fff640 1 --2- 172.21.15.67:0/1045759064 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b9d5858330 0x55b9d5888120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:53.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.109+0000 7f6383fff640 1 -- 172.21.15.67:0/1045759064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b9d55a07d0 con 0x55b9d5885230 2024-03-24T00:40:53.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.109+0000 7f6383fff640 1 --2- 172.21.15.67:0/1045759064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9d5885230 0x55b9d5887be0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f6370009710 tx=0x7f6370009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf593393153086ab server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:53.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.109+0000 7f636e7fc640 1 -- 172.21.15.67:0/1045759064 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6370015020 con 0x55b9d5885230 2024-03-24T00:40:53.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.109+0000 7f636e7fc640 1 -- 172.21.15.67:0/1045759064 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f637000b880 con 0x55b9d5885230 2024-03-24T00:40:53.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.109+0000 7f636e7fc640 1 -- 172.21.15.67:0/1045759064 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f637000bc70 con 0x55b9d5885230 2024-03-24T00:40:53.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.110+0000 7f638d98b740 1 -- 172.21.15.67:0/1045759064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9d5885230 msgr2=0x55b9d5887be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:53.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.110+0000 7f638d98b740 1 --2- 172.21.15.67:0/1045759064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9d5885230 0x55b9d5887be0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f6370009710 tx=0x7f6370009bd0 comp rx=0 tx=0).stop 2024-03-24T00:40:53.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.110+0000 7f638d98b740 1 -- 172.21.15.67:0/1045759064 shutdown_connections 2024-03-24T00:40:53.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.110+0000 7f638d98b740 1 --2- 172.21.15.67:0/1045759064 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b9d5858330 0x55b9d5888120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:53.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.110+0000 7f638d98b740 1 --2- 172.21.15.67:0/1045759064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9d5885230 0x55b9d5887be0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:53.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.110+0000 7f638d98b740 1 --2- 172.21.15.67:0/1045759064 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b9d5884e50 0x55b9d5857df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:53.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.110+0000 7f638d98b740 1 -- 172.21.15.67:0/1045759064 >> 172.21.15.67:0/1045759064 conn(0x55b9d585a060 msgr2=0x55b9d58a2be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:53.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.110+0000 7f638d98b740 1 -- 172.21.15.67:0/1045759064 shutdown_connections 2024-03-24T00:40:53.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.110+0000 7f638d98b740 1 -- 172.21.15.67:0/1045759064 wait complete. 2024-03-24T00:40:53.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.110+0000 7f638d98b740 1 -- start start 2024-03-24T00:40:53.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.111+0000 7f638d98b740 1 Processor -- start 2024-03-24T00:40:53.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.111+0000 7f638d98b740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b9d5884e50 0x55b9d588dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:53.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.111+0000 7f638d98b740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b9d5885230 0x55b9d588a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:53.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.111+0000 7f638d98b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9d587fab0 0x55b9d587ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:53.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.111+0000 7f638d98b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9d585c700 con 0x55b9d587fab0 2024-03-24T00:40:53.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.111+0000 7f638d98b740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9d577e060 con 0x55b9d5885230 2024-03-24T00:40:53.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.111+0000 7f638d98b740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b9d585ef40 con 0x55b9d5884e50 2024-03-24T00:40:53.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.111+0000 7f638d98b740 10 client.0 _ll_register_callbacks cb 0x55b9d57606a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:40:53.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.111+0000 7f6389348640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9d587fab0 0x55b9d587ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:53.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.111+0000 7f6389348640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9d587fab0 0x55b9d587ff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41170/0 (socket says 172.21.15.67:41170) 2024-03-24T00:40:53.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.111+0000 7f6389348640 1 -- 172.21.15.67:0/3584638098 learned_addr learned my addr 172.21.15.67:0/3584638098 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:53.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.111+0000 7f6388b47640 1 --2- 172.21.15.67:0/3584638098 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b9d5884e50 0x55b9d588dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:53.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.111+0000 7f6383fff640 1 --2- 172.21.15.67:0/3584638098 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b9d5885230 0x55b9d588a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:53.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.112+0000 7f6389348640 1 -- 172.21.15.67:0/3584638098 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b9d5884e50 msgr2=0x55b9d588dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:53.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.112+0000 7f6389348640 1 --2- 172.21.15.67:0/3584638098 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b9d5884e50 0x55b9d588dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:53.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.112+0000 7f6389348640 1 -- 172.21.15.67:0/3584638098 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b9d5885230 msgr2=0x55b9d588a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:53.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.112+0000 7f6389348640 1 --2- 172.21.15.67:0/3584638098 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b9d5885230 0x55b9d588a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:53.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.112+0000 7f6389348640 1 -- 172.21.15.67:0/3584638098 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b9d54d63f0 con 0x55b9d587fab0 2024-03-24T00:40:53.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.112+0000 7f6389348640 1 --2- 172.21.15.67:0/3584638098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9d587fab0 0x55b9d587ff80 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f637c058a40 tx=0x7f637c058c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b149dd818bb60efa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:53.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.112+0000 7f63637fe640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:40:53.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.112+0000 7f63637fe640 1 -- 172.21.15.67:0/3584638098 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f637c05db70 con 0x55b9d587fab0 2024-03-24T00:40:53.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.112+0000 7f63637fe640 1 -- 172.21.15.67:0/3584638098 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(50..50 src has 1..50) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f637c05dd10 con 0x55b9d587fab0 2024-03-24T00:40:53.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.112+0000 7f638d98b740 1 -- 172.21.15.67:0/3584638098 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f637000b880 con 0x55b9d587fab0 2024-03-24T00:40:53.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.112+0000 7f63637fe640 1 -- 172.21.15.67:0/3584638098 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f637c057f50 con 0x55b9d587fab0 2024-03-24T00:40:53.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.112+0000 7f638d98b740 10 client.6114 fetch_fsmap learned FSMap version 26 2024-03-24T00:40:53.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.112+0000 7f638d98b740 1 -- 172.21.15.67:0/3584638098 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=2+}) v3 -- 0x55b9d54960f0 con 0x55b9d587fab0 2024-03-24T00:40:53.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.112+0000 7f63637fe640 1 -- 172.21.15.67:0/3584638098 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f637000b880 con 0x55b9d587fab0 2024-03-24T00:40:53.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.112+0000 7f63637fe640 1 -- 172.21.15.67:0/3584638098 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 26) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f637c0580f0 con 0x55b9d587fab0 2024-03-24T00:40:53.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.112+0000 7f638d98b740 10 client.6114 fetch_fsmap finished waiting for FSMap version 26 2024-03-24T00:40:53.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.112+0000 7f638d98b740 10 client.6114 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.a 2024-03-24T00:40:53.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.112+0000 7f638d98b740 20 client.6114 populate_metadata read hostname 'smithi067' 2024-03-24T00:40:53.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.112+0000 7f638d98b740 1 --2- 172.21.15.67:0/3584638098 >> [v2:172.21.15.67:6832/2741818214,v1:172.21.15.67:6834/2741818214] conn(0x55b9d588ece0 0x55b9d588f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:53.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.112+0000 7f638d98b740 4 client.6114 mds_command: new command op to 5853 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:40:53.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.113+0000 7f638d98b740 1 -- 172.21.15.67:0/3584638098 --> [v2:172.21.15.67:6832/2741818214,v1:172.21.15.67:6834/2741818214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b9d57cf910 con 0x55b9d588ece0 2024-03-24T00:40:53.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.113+0000 7f6388b47640 1 --2- 172.21.15.67:0/3584638098 >> [v2:172.21.15.67:6832/2741818214,v1:172.21.15.67:6834/2741818214] conn(0x55b9d588ece0 0x55b9d588f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:53.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.113+0000 7f63637fe640 10 client.6114 ms_handle_connect on v2:172.21.15.67:6832/2741818214 2024-03-24T00:40:53.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.113+0000 7f6388b47640 1 --2- 172.21.15.67:0/3584638098 >> [v2:172.21.15.67:6832/2741818214,v1:172.21.15.67:6834/2741818214] conn(0x55b9d588ece0 0x55b9d588f140 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a380c7b2297df48 server_cookie=789d7b78f3b65ff1 in_seq=0 out_seq=0 2024-03-24T00:40:53.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.114+0000 7f63637fe640 1 -- 172.21.15.67:0/3584638098 <== mds.0 v2:172.21.15.67:6832/2741818214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b9d57cf910 con 0x55b9d588ece0 2024-03-24T00:40:53.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.114+0000 7f63637fe640 10 client.6114 handle_command_reply: tid=0 2024-03-24T00:40:53.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.114+0000 7f638d98b740 1 client.6114 shutdown 2024-03-24T00:40:53.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.114+0000 7f638d98b740 1 -- 172.21.15.67:0/3584638098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9d587fab0 msgr2=0x55b9d587ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:53.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.114+0000 7f638d98b740 1 --2- 172.21.15.67:0/3584638098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9d587fab0 0x55b9d587ff80 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f637c058a40 tx=0x7f637c058c80 comp rx=0 tx=0).stop 2024-03-24T00:40:53.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.114+0000 7f638d98b740 1 -- 172.21.15.67:0/3584638098 shutdown_connections 2024-03-24T00:40:53.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.114+0000 7f638d98b740 1 --2- 172.21.15.67:0/3584638098 >> [v2:172.21.15.67:6832/2741818214,v1:172.21.15.67:6834/2741818214] conn(0x55b9d588ece0 0x55b9d588f140 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:53.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.114+0000 7f638d98b740 1 --2- 172.21.15.67:0/3584638098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9d587fab0 0x55b9d587ff80 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:53.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.114+0000 7f638d98b740 1 --2- 172.21.15.67:0/3584638098 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b9d5885230 0x55b9d588a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:53.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.114+0000 7f63637fe640 0 client.6114 ms_handle_reset on v2:172.21.15.67:6832/2741818214 2024-03-24T00:40:53.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.114+0000 7f638d98b740 1 --2- 172.21.15.67:0/3584638098 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b9d5884e50 0x55b9d588dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:53.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.114+0000 7f638d98b740 1 -- 172.21.15.67:0/3584638098 >> 172.21.15.67:0/3584638098 conn(0x55b9d5858330 msgr2=0x55b9d585bdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:53.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.114+0000 7f638d98b740 1 -- 172.21.15.67:0/3584638098 shutdown_connections 2024-03-24T00:40:53.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.114+0000 7f638d98b740 1 -- 172.21.15.67:0/3584638098 wait complete. 2024-03-24T00:40:53.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.114+0000 7f638d98b740 20 client.6114 trim_cache size 0 max 16384 2024-03-24T00:40:53.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.130+0000 7f638d98b740 1 Processor -- start 2024-03-24T00:40:53.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.130+0000 7f638d98b740 1 -- start start 2024-03-24T00:40:53.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.131+0000 7f638d98b740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b9d587fab0 0x55b9d585d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:53.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.131+0000 7f638d98b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9d5884e50 0x55b9d58726d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:53.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.131+0000 7f638d98b740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b9d5885230 0x55b9d5872c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:53.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.131+0000 7f638d98b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6384200900 con 0x55b9d5884e50 2024-03-24T00:40:53.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.131+0000 7f638d98b740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63841ffad0 con 0x55b9d587fab0 2024-03-24T00:40:53.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.131+0000 7f638d98b740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63840ab890 con 0x55b9d5885230 2024-03-24T00:40:53.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.131+0000 7f6383fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9d5884e50 0x55b9d58726d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:53.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.131+0000 7f6383fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9d5884e50 0x55b9d58726d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41180/0 (socket says 172.21.15.67:41180) 2024-03-24T00:40:53.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.131+0000 7f6383fff640 1 -- 172.21.15.67:0/4084934673 learned_addr learned my addr 172.21.15.67:0/4084934673 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:53.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.131+0000 7f6383fff640 1 -- 172.21.15.67:0/4084934673 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b9d5885230 msgr2=0x55b9d5872c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:53.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.131+0000 7f6388b47640 1 --2- 172.21.15.67:0/4084934673 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b9d587fab0 0x55b9d585d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:53.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.131+0000 7f6389348640 1 --2- 172.21.15.67:0/4084934673 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b9d5885230 0x55b9d5872c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:53.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.131+0000 7f6383fff640 1 --2- 172.21.15.67:0/4084934673 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b9d5885230 0x55b9d5872c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:53.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.131+0000 7f6383fff640 1 -- 172.21.15.67:0/4084934673 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b9d587fab0 msgr2=0x55b9d585d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:53.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.131+0000 7f6383fff640 1 --2- 172.21.15.67:0/4084934673 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b9d587fab0 0x55b9d585d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:53.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.131+0000 7f6383fff640 1 -- 172.21.15.67:0/4084934673 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b9d587f690 con 0x55b9d5884e50 2024-03-24T00:40:53.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.131+0000 7f6388b47640 1 --2- 172.21.15.67:0/4084934673 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b9d587fab0 0x55b9d585d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:40:53.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.131+0000 7f6389348640 1 --2- 172.21.15.67:0/4084934673 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b9d5885230 0x55b9d5872c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:40:53.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.132+0000 7f6383fff640 1 --2- 172.21.15.67:0/4084934673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9d5884e50 0x55b9d58726d0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f637000a010 tx=0x7f6370008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84ed8ab17aa53fb4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:53.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.132+0000 7f636dffb640 1 -- 172.21.15.67:0/4084934673 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6370015040 con 0x55b9d5884e50 2024-03-24T00:40:53.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.132+0000 7f636dffb640 1 -- 172.21.15.67:0/4084934673 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6370029070 con 0x55b9d5884e50 2024-03-24T00:40:53.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.132+0000 7f636dffb640 1 -- 172.21.15.67:0/4084934673 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f637000bb60 con 0x55b9d5884e50 2024-03-24T00:40:53.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.132+0000 7f638d98b740 1 -- 172.21.15.67:0/4084934673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9d5884e50 msgr2=0x55b9d58726d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:53.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.132+0000 7f638d98b740 1 --2- 172.21.15.67:0/4084934673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9d5884e50 0x55b9d58726d0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f637000a010 tx=0x7f6370008030 comp rx=0 tx=0).stop 2024-03-24T00:40:53.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.132+0000 7f638d98b740 1 -- 172.21.15.67:0/4084934673 shutdown_connections 2024-03-24T00:40:53.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.132+0000 7f638d98b740 1 --2- 172.21.15.67:0/4084934673 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b9d5885230 0x55b9d5872c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:53.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.132+0000 7f638d98b740 1 --2- 172.21.15.67:0/4084934673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9d5884e50 0x55b9d58726d0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:53.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.132+0000 7f638d98b740 1 --2- 172.21.15.67:0/4084934673 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b9d587fab0 0x55b9d585d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:53.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.132+0000 7f638d98b740 1 -- 172.21.15.67:0/4084934673 >> 172.21.15.67:0/4084934673 conn(0x55b9d5858330 msgr2=0x55b9d5888be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:53.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.132+0000 7f638d98b740 1 -- 172.21.15.67:0/4084934673 shutdown_connections 2024-03-24T00:40:53.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.132+0000 7f638d98b740 1 -- 172.21.15.67:0/4084934673 wait complete. 2024-03-24T00:40:53.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.133+0000 7f638d98b740 1 -- start start 2024-03-24T00:40:53.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.133+0000 7f638d98b740 1 Processor -- start 2024-03-24T00:40:53.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.134+0000 7f638d98b740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b9d587fab0 0x55b9d5914e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:53.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.134+0000 7f638d98b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9d5884e50 0x55b9d5888be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:53.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.134+0000 7f638d98b740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b9d5885230 0x55b9d5871e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:53.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.134+0000 7f638d98b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63841ff4d0 con 0x55b9d5884e50 2024-03-24T00:40:53.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.134+0000 7f638d98b740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9d588d970 con 0x55b9d587fab0 2024-03-24T00:40:53.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.134+0000 7f638d98b740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b9d58c5340 con 0x55b9d5885230 2024-03-24T00:40:53.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.134+0000 7f638d98b740 10 client.0 _ll_register_callbacks cb 0x7f637c059200 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:40:53.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.134+0000 7f6383fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9d5884e50 0x55b9d5888be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:53.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.134+0000 7f6383fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9d5884e50 0x55b9d5888be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41192/0 (socket says 172.21.15.67:41192) 2024-03-24T00:40:53.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.134+0000 7f6383fff640 1 -- 172.21.15.67:0/1536323571 learned_addr learned my addr 172.21.15.67:0/1536323571 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:40:53.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.134+0000 7f6388b47640 1 --2- 172.21.15.67:0/1536323571 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b9d587fab0 0x55b9d5914e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:53.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.134+0000 7f6389348640 1 --2- 172.21.15.67:0/1536323571 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b9d5885230 0x55b9d5871e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:53.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.134+0000 7f6383fff640 1 -- 172.21.15.67:0/1536323571 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b9d5885230 msgr2=0x55b9d5871e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:53.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.134+0000 7f6383fff640 1 --2- 172.21.15.67:0/1536323571 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b9d5885230 0x55b9d5871e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:53.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.134+0000 7f6383fff640 1 -- 172.21.15.67:0/1536323571 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b9d587fab0 msgr2=0x55b9d5914e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:40:53.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.134+0000 7f6383fff640 1 --2- 172.21.15.67:0/1536323571 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b9d587fab0 0x55b9d5914e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:53.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.134+0000 7f6383fff640 1 -- 172.21.15.67:0/1536323571 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b9d585cf30 con 0x55b9d5884e50 2024-03-24T00:40:53.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.134+0000 7f6383fff640 1 --2- 172.21.15.67:0/1536323571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9d5884e50 0x55b9d5888be0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f6370014040 tx=0x7f63700047a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9b6711d8aa196b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:53.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.134+0000 7f63637fe640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:40:53.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.135+0000 7f63637fe640 1 -- 172.21.15.67:0/1536323571 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6370015040 con 0x55b9d5884e50 2024-03-24T00:40:53.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.135+0000 7f63637fe640 1 -- 172.21.15.67:0/1536323571 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(50..50 src has 1..50) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f637002f460 con 0x55b9d5884e50 2024-03-24T00:40:53.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.135+0000 7f638d98b740 1 -- 172.21.15.67:0/1536323571 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6370029070 con 0x55b9d5884e50 2024-03-24T00:40:53.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.135+0000 7f63637fe640 1 -- 172.21.15.67:0/1536323571 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f637003c050 con 0x55b9d5884e50 2024-03-24T00:40:53.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.135+0000 7f638d98b740 10 client.6120 fetch_fsmap learned FSMap version 26 2024-03-24T00:40:53.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.135+0000 7f638d98b740 1 -- 172.21.15.67:0/1536323571 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=2+}) v3 -- 0x55b9d58c4fc0 con 0x55b9d5884e50 2024-03-24T00:40:53.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.135+0000 7f63637fe640 1 -- 172.21.15.67:0/1536323571 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6370029070 con 0x55b9d5884e50 2024-03-24T00:40:53.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.135+0000 7f63637fe640 1 -- 172.21.15.67:0/1536323571 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 26) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f637003f020 con 0x55b9d5884e50 2024-03-24T00:40:53.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.135+0000 7f638d98b740 10 client.6120 fetch_fsmap finished waiting for FSMap version 26 2024-03-24T00:40:53.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.135+0000 7f638d98b740 10 client.6120 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.a 2024-03-24T00:40:53.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.135+0000 7f638d98b740 20 client.6120 populate_metadata read hostname 'smithi067' 2024-03-24T00:40:53.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.135+0000 7f638d98b740 1 --2- 172.21.15.67:0/1536323571 >> [v2:172.21.15.67:6832/2741818214,v1:172.21.15.67:6834/2741818214] conn(0x55b9d58a32e0 0x55b9d5872370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:53.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.135+0000 7f638d98b740 4 client.6120 mds_command: new command op to 5853 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:40:53.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.135+0000 7f638d98b740 1 -- 172.21.15.67:0/1536323571 --> [v2:172.21.15.67:6832/2741818214,v1:172.21.15.67:6834/2741818214] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55b9d57c9c60 con 0x55b9d58a32e0 2024-03-24T00:40:53.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.135+0000 7f6388b47640 1 --2- 172.21.15.67:0/1536323571 >> [v2:172.21.15.67:6832/2741818214,v1:172.21.15.67:6834/2741818214] conn(0x55b9d58a32e0 0x55b9d5872370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:53.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.135+0000 7f63637fe640 10 client.6120 ms_handle_connect on v2:172.21.15.67:6832/2741818214 2024-03-24T00:40:53.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.135+0000 7f6388b47640 1 --2- 172.21.15.67:0/1536323571 >> [v2:172.21.15.67:6832/2741818214,v1:172.21.15.67:6834/2741818214] conn(0x55b9d58a32e0 0x55b9d5872370 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d9a6fddd5e3b2a9 server_cookie=c423ffff9e6b1809 in_seq=0 out_seq=0 2024-03-24T00:40:53.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.136+0000 7f63637fe640 1 -- 172.21.15.67:0/1536323571 <== mds.0 v2:172.21.15.67:6832/2741818214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+260 (crc 0 0 0) 0x55b9d57c9c60 con 0x55b9d58a32e0 2024-03-24T00:40:53.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.136+0000 7f63637fe640 10 client.6120 handle_command_reply: tid=0 2024-03-24T00:40:53.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.136+0000 7f638d98b740 1 client.6120 shutdown 2024-03-24T00:40:53.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.136+0000 7f638d98b740 1 -- 172.21.15.67:0/1536323571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9d5884e50 msgr2=0x55b9d5888be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:53.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.136+0000 7f638d98b740 1 --2- 172.21.15.67:0/1536323571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9d5884e50 0x55b9d5888be0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f6370014040 tx=0x7f63700047a0 comp rx=0 tx=0).stop 2024-03-24T00:40:53.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.136+0000 7f638d98b740 1 -- 172.21.15.67:0/1536323571 shutdown_connections 2024-03-24T00:40:53.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.136+0000 7f638d98b740 1 --2- 172.21.15.67:0/1536323571 >> [v2:172.21.15.67:6832/2741818214,v1:172.21.15.67:6834/2741818214] conn(0x55b9d58a32e0 0x55b9d5872370 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:53.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.136+0000 7f638d98b740 1 --2- 172.21.15.67:0/1536323571 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b9d5885230 0x55b9d5871e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:53.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.136+0000 7f63637fe640 0 client.6120 ms_handle_reset on v2:172.21.15.67:6832/2741818214 2024-03-24T00:40:53.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.136+0000 7f638d98b740 1 --2- 172.21.15.67:0/1536323571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9d5884e50 0x55b9d5888be0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:53.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.136+0000 7f638d98b740 1 --2- 172.21.15.67:0/1536323571 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b9d587fab0 0x55b9d5914e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:53.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.136+0000 7f638d98b740 1 -- 172.21.15.67:0/1536323571 >> 172.21.15.67:0/1536323571 conn(0x55b9d5858330 msgr2=0x55b9d5872ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:53.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.136+0000 7f638d98b740 1 -- 172.21.15.67:0/1536323571 shutdown_connections 2024-03-24T00:40:53.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.136+0000 7f638d98b740 1 -- 172.21.15.67:0/1536323571 wait complete. 2024-03-24T00:40:53.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.136+0000 7f638d98b740 20 client.6120 trim_cache size 0 max 16384 2024-03-24T00:40:53.143 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:40:53.143 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T00:40:53.144 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": { 2024-03-24T00:40:53.144 INFO:teuthology.orchestra.run.smithi067.stdout: "56385fa2-25bc-4ca2-a9a5-2eab70b67325": { 2024-03-24T00:40:53.144 INFO:teuthology.orchestra.run.smithi067.stdout: "path": "/", 2024-03-24T00:40:53.144 INFO:teuthology.orchestra.run.smithi067.stdout: "tag": "56385fa2-25bc-4ca2-a9a5-2eab70b67325", 2024-03-24T00:40:53.144 INFO:teuthology.orchestra.run.smithi067.stdout: "options": "recursive,repair" 2024-03-24T00:40:53.144 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:40:53.144 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:40:53.144 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:40:53.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.136+0000 7f638adab640 1 -- 172.21.15.67:0/1659736699 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f636806bbd0 msgr2=0x7f636806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:53.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.137+0000 7f638adab640 1 --2- 172.21.15.67:0/1659736699 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f636806bbd0 0x7f636806e090 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f6378036000 tx=0x7f6378008030 comp rx=0 tx=0).stop 2024-03-24T00:40:53.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.137+0000 7f638adab640 1 -- 172.21.15.67:0/1659736699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f638415e280 msgr2=0x7f63840cddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:40:53.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.137+0000 7f638adab640 1 --2- 172.21.15.67:0/1659736699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f638415e280 0x7f63840cddc0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f637c00b9e0 tx=0x7f637c00bea0 comp rx=0 tx=0).stop 2024-03-24T00:40:53.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.137+0000 7f638adab640 1 -- 172.21.15.67:0/1659736699 shutdown_connections 2024-03-24T00:40:53.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.137+0000 7f638adab640 1 --2- 172.21.15.67:0/1659736699 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f636806bbd0 0x7f636806e090 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:53.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.137+0000 7f638adab640 1 --2- 172.21.15.67:0/1659736699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f638415e280 0x7f63840cddc0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:53.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.137+0000 7f638adab640 1 --2- 172.21.15.67:0/1659736699 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f638415dea0 0x7f63840b71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:53.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.137+0000 7f638adab640 1 --2- 172.21.15.67:0/1659736699 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63840b5a60 0x7f63840b6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:53.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.137+0000 7f638adab640 1 -- 172.21.15.67:0/1659736699 >> 172.21.15.67:0/1659736699 conn(0x7f6384072da0 msgr2=0x7f63840b7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:40:53.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.137+0000 7f638adab640 1 -- 172.21.15.67:0/1659736699 shutdown_connections 2024-03-24T00:40:53.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:53.137+0000 7f638adab640 1 -- 172.21.15.67:0/1659736699 wait complete. 2024-03-24T00:40:53.148 INFO:tasks.cephfs.filesystem:scrub status for tag:56385fa2-25bc-4ca2-a9a5-2eab70b67325 - {'path': '/', 'tag': '56385fa2-25bc-4ca2-a9a5-2eab70b67325', 'options': 'recursive,repair'} 2024-03-24T00:40:54.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:54.413+0000 7f5f9dffb640 1 -- 172.21.15.67:0/1383122620 <== mon.0 v2:172.21.15.67:3300/0 7 ==== log(6 entries from seq 1 at 2024-03-24T00:40:52.978535+0000) v1 ==== 1078+0+0 (secure 0 0 0) 0x7f5fa00896e0 con 0x7f5fa815e3c0 2024-03-24T00:40:54.415 INFO:teuthology.orchestra.run.smithi067.stdout: cluster: 2024-03-24T00:40:54.416 INFO:teuthology.orchestra.run.smithi067.stdout: id: de1d0178-808e-49ca-a986-e297e523b976 2024-03-24T00:40:54.416 INFO:teuthology.orchestra.run.smithi067.stdout: health: HEALTH_OK 2024-03-24T00:40:54.416 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:40:54.416 INFO:teuthology.orchestra.run.smithi067.stdout: services: 2024-03-24T00:40:54.416 INFO:teuthology.orchestra.run.smithi067.stdout: mon: 3 daemons, quorum a,b,c (age 2m) 2024-03-24T00:40:54.416 INFO:teuthology.orchestra.run.smithi067.stdout: mgr: x(active, since 2m), standbys: y 2024-03-24T00:40:54.416 INFO:teuthology.orchestra.run.smithi067.stdout: mds: 1/1 daemons up, 3 standby 2024-03-24T00:40:54.416 INFO:teuthology.orchestra.run.smithi067.stdout: osd: 8 osds: 8 up (since 2m), 8 in (since 2m) 2024-03-24T00:40:54.416 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:40:54.416 INFO:teuthology.orchestra.run.smithi067.stdout: data: 2024-03-24T00:40:54.416 INFO:teuthology.orchestra.run.smithi067.stdout: volumes: 1/1 healthy 2024-03-24T00:40:54.416 INFO:teuthology.orchestra.run.smithi067.stdout: pools: 3 pools, 97 pgs 2024-03-24T00:40:54.417 INFO:teuthology.orchestra.run.smithi067.stdout: objects: 24 objects, 579 KiB 2024-03-24T00:40:54.417 INFO:teuthology.orchestra.run.smithi067.stdout: usage: 217 MiB used, 720 GiB / 720 GiB avail 2024-03-24T00:40:54.417 INFO:teuthology.orchestra.run.smithi067.stdout: pgs: 97 active+clean 2024-03-24T00:40:54.417 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:40:54.417 INFO:teuthology.orchestra.run.smithi067.stdout: io: 2024-03-24T00:40:54.417 INFO:teuthology.orchestra.run.smithi067.stdout: client: 1.4 KiB/s wr, 0 op/s rd, 4 op/s wr 2024-03-24T00:40:54.417 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:40:54.417 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:40:54.417 INFO:teuthology.orchestra.run.smithi067.stdout:2024-03-24T00:40:52.978535+0000 mds.a [INF] scrub queued for path: / 2024-03-24T00:40:54.417 INFO:teuthology.orchestra.run.smithi067.stdout:2024-03-24T00:40:52.978538+0000 mds.a [INF] scrub summary: idle+waiting paths [/] 2024-03-24T00:40:54.417 INFO:teuthology.orchestra.run.smithi067.stdout:2024-03-24T00:40:52.978542+0000 mds.a [INF] scrub summary: active paths [/] 2024-03-24T00:40:54.417 INFO:teuthology.orchestra.run.smithi067.stdout:2024-03-24T00:40:52.979274+0000 mds.a [WRN] bad backtrace on inode 0x10000000001(/parent_a/alpha), rewriting it 2024-03-24T00:40:54.417 INFO:teuthology.orchestra.run.smithi067.stdout:2024-03-24T00:40:52.979288+0000 mds.a [INF] Scrub repaired inode 0x10000000001 (/parent_a/alpha) 2024-03-24T00:40:54.417 INFO:teuthology.orchestra.run.smithi067.stdout:2024-03-24T00:40:52.979996+0000 mds.a [INF] scrub summary: idle+waiting paths [/] 2024-03-24T00:40:57.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:40:57.420+0000 7f5f9dffb640 1 -- 172.21.15.67:0/1383122620 <== mon.0 v2:172.21.15.67:3300/0 8 ==== log(1 entries from seq 7 at 2024-03-24T00:40:55.943389+0000) v1 ==== 189+0+0 (secure 0 0 0) 0x7f5fa0057670 con 0x7f5fa815e3c0 2024-03-24T00:40:57.420 INFO:teuthology.orchestra.run.smithi067.stdout:2024-03-24T00:40:55.943389+0000 mds.a [INF] scrub summary: idle 2024-03-24T00:41:16.129 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:41:16.132 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:41:23.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:41:23.150 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.3:0 scrub status 2024-03-24T00:41:23.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.253+0000 7fb7a013b640 1 Processor -- start 2024-03-24T00:41:23.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.253+0000 7fb7a013b640 1 -- start start 2024-03-24T00:41:23.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.254+0000 7fb7a013b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb7980b6170 0x7fb798168c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:23.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.254+0000 7fb7a013b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7980d2250 con 0x7fb79815dea0 2024-03-24T00:41:23.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.254+0000 7fb7a013b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7980d2530 con 0x7fb7980d3ae0 2024-03-24T00:41:23.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.254+0000 7fb7a013b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7980d2810 con 0x7fb7980b6170 2024-03-24T00:41:23.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.254+0000 7fb79ded7640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb79815dea0 0x7fb7980d33d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39406/0 (socket says 172.21.15.67:39406) 2024-03-24T00:41:23.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.254+0000 7fb79ded7640 1 -- 172.21.15.67:0/2227088259 learned_addr learned my addr 172.21.15.67:0/2227088259 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:41:23.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.254+0000 7fb79e6d8640 1 --2- 172.21.15.67:0/2227088259 >> v2:172.21.15.72:3300/0 conn(0x7fb7980b6170 0x7fb798168c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:23.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.254+0000 7fb79ced5640 1 -- 172.21.15.67:0/2227088259 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2154830646 0 0) 0x7fb7980d2250 con 0x7fb79815dea0 2024-03-24T00:41:23.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.254+0000 7fb79ced5640 1 -- 172.21.15.67:0/2227088259 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb784003520 con 0x7fb79815dea0 2024-03-24T00:41:23.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.254+0000 7fb79ced5640 1 -- 172.21.15.67:0/2227088259 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3195258244 0 0) 0x7fb7980d2530 con 0x7fb7980d3ae0 2024-03-24T00:41:23.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.254+0000 7fb79ced5640 1 -- 172.21.15.67:0/2227088259 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7980d2250 con 0x7fb7980d3ae0 2024-03-24T00:41:23.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.254+0000 7fb79e6d8640 1 -- 172.21.15.67:0/2227088259 >> v1:172.21.15.72:6789/0 conn(0x7fb7980d3ae0 legacy=0x7fb7980b5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:23.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.254+0000 7fb79e6d8640 1 -- 172.21.15.67:0/2227088259 >> v1:172.21.15.67:6789/0 conn(0x7fb79815dea0 legacy=0x7fb7980d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:23.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.254+0000 7fb79e6d8640 1 -- 172.21.15.67:0/2227088259 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7980d2af0 con 0x7fb7980b6170 2024-03-24T00:41:23.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.255+0000 7fb79e6d8640 1 --2- 172.21.15.67:0/2227088259 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7980b6170 0x7fb798168c80 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fb790009dc0 tx=0x7fb79002be90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d0b4f442ed65d57 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:23.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.255+0000 7fb79ced5640 1 -- 172.21.15.67:0/2227088259 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb790039070 con 0x7fb7980b6170 2024-03-24T00:41:23.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.255+0000 7fb79ced5640 1 -- 172.21.15.67:0/2227088259 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb79002cc40 con 0x7fb7980b6170 2024-03-24T00:41:23.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.255+0000 7fb79ced5640 1 -- 172.21.15.67:0/2227088259 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb790034440 con 0x7fb7980b6170 2024-03-24T00:41:23.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.256+0000 7fb7a013b640 1 -- 172.21.15.67:0/2227088259 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7980b6170 msgr2=0x7fb798168c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:23.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.256+0000 7fb7a013b640 1 --2- 172.21.15.67:0/2227088259 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7980b6170 0x7fb798168c80 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fb790009dc0 tx=0x7fb79002be90 comp rx=0 tx=0).stop 2024-03-24T00:41:23.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.256+0000 7fb7a013b640 1 -- 172.21.15.67:0/2227088259 shutdown_connections 2024-03-24T00:41:23.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.256+0000 7fb7a013b640 1 --2- 172.21.15.67:0/2227088259 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7980b6170 0x7fb798168c80 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:23.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.256+0000 7fb7a013b640 1 -- 172.21.15.67:0/2227088259 >> 172.21.15.67:0/2227088259 conn(0x7fb798072da0 msgr2=0x7fb7980d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:23.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.256+0000 7fb7a013b640 1 -- 172.21.15.67:0/2227088259 shutdown_connections 2024-03-24T00:41:23.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.256+0000 7fb7a013b640 1 -- 172.21.15.67:0/2227088259 wait complete. 2024-03-24T00:41:23.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.257+0000 7fb7a013b640 1 Processor -- start 2024-03-24T00:41:23.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.257+0000 7fb7a013b640 1 -- start start 2024-03-24T00:41:23.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.258+0000 7fb7a013b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7980b6170 0x7fb798150610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:23.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.258+0000 7fb7a013b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7980d3ae0 0x7fb79814d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:23.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.258+0000 7fb7a013b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb79815dea0 0x7fb7980ccc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:23.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.258+0000 7fb7a013b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb79815e280 con 0x7fb7980b6170 2024-03-24T00:41:23.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.258+0000 7fb7a013b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7980b8350 con 0x7fb7980d3ae0 2024-03-24T00:41:23.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.258+0000 7fb7a013b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb79815e400 con 0x7fb79815dea0 2024-03-24T00:41:23.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.258+0000 7fb79ded7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7980b6170 0x7fb798150610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:23.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.258+0000 7fb79ded7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7980b6170 0x7fb798150610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47238/0 (socket says 172.21.15.67:47238) 2024-03-24T00:41:23.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.258+0000 7fb79ded7640 1 -- 172.21.15.67:0/2644958209 learned_addr learned my addr 172.21.15.67:0/2644958209 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:23.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.258+0000 7fb79d6d6640 1 --2- 172.21.15.67:0/2644958209 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7980d3ae0 0x7fb79814d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:23.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.258+0000 7fb79e6d8640 1 --2- 172.21.15.67:0/2644958209 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb79815dea0 0x7fb7980ccc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:23.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.258+0000 7fb79ded7640 1 -- 172.21.15.67:0/2644958209 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb79815dea0 msgr2=0x7fb7980ccc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:23.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.258+0000 7fb79ded7640 1 --2- 172.21.15.67:0/2644958209 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb79815dea0 0x7fb7980ccc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:23.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.258+0000 7fb79ded7640 1 -- 172.21.15.67:0/2644958209 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7980d3ae0 msgr2=0x7fb79814d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:23.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.258+0000 7fb79ded7640 1 --2- 172.21.15.67:0/2644958209 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7980d3ae0 0x7fb79814d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:23.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.258+0000 7fb79ded7640 1 -- 172.21.15.67:0/2644958209 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7980cd3b0 con 0x7fb7980b6170 2024-03-24T00:41:23.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.258+0000 7fb79ded7640 1 --2- 172.21.15.67:0/2644958209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7980b6170 0x7fb798150610 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fb79400c530 tx=0x7fb79400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:23.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.259+0000 7fb78e7fc640 1 -- 172.21.15.67:0/2644958209 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb794017070 con 0x7fb7980b6170 2024-03-24T00:41:23.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.259+0000 7fb78e7fc640 1 -- 172.21.15.67:0/2644958209 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb794005ce0 con 0x7fb7980b6170 2024-03-24T00:41:23.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.259+0000 7fb7a013b640 1 -- 172.21.15.67:0/2644958209 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7980cd690 con 0x7fb7980b6170 2024-03-24T00:41:23.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.259+0000 7fb78e7fc640 1 -- 172.21.15.67:0/2644958209 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb794007270 con 0x7fb7980b6170 2024-03-24T00:41:23.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.259+0000 7fb7a013b640 1 -- 172.21.15.67:0/2644958209 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7980cdc50 con 0x7fb7980b6170 2024-03-24T00:41:23.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.260+0000 7fb78e7fc640 1 -- 172.21.15.67:0/2644958209 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb79401a070 con 0x7fb7980b6170 2024-03-24T00:41:23.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.261+0000 7fb78e7fc640 1 --2- 172.21.15.67:0/2644958209 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb77c06bbd0 0x7fb77c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:23.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.261+0000 7fb78e7fc640 1 -- 172.21.15.67:0/2644958209 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb794089f00 con 0x7fb7980b6170 2024-03-24T00:41:23.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.261+0000 7fb79d6d6640 1 --2- 172.21.15.67:0/2644958209 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb77c06bbd0 0x7fb77c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:23.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.261+0000 7fb7a2e02740 1 Processor -- start 2024-03-24T00:41:23.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.262+0000 7fb7a2e02740 1 -- start start 2024-03-24T00:41:23.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.262+0000 7fb79d6d6640 1 --2- 172.21.15.67:0/2644958209 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb77c06bbd0 0x7fb77c06e090 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fb7981608c0 tx=0x7fb788009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:23.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.262+0000 7fb7a2e02740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559945d38e50 0x559945d0bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:23.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.262+0000 7fb7a2e02740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559945d39230 0x559945d3bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:23.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.262+0000 7fb7a2e02740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559945d0c330 0x559945d3c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:23.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.262+0000 7fb7a2e02740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x559945c763d0 con 0x559945d0c330 2024-03-24T00:41:23.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.262+0000 7fb7a2e02740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x559945c1ead0 con 0x559945d38e50 2024-03-24T00:41:23.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.262+0000 7fb7a2e02740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x559945c72470 con 0x559945d39230 2024-03-24T00:41:23.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.262+0000 7fb79e6d8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559945d0c330 0x559945d3c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:23.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.262+0000 7fb79e6d8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559945d0c330 0x559945d3c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47242/0 (socket says 172.21.15.67:47242) 2024-03-24T00:41:23.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.262+0000 7fb79e6d8640 1 -- 172.21.15.67:0/3054649979 learned_addr learned my addr 172.21.15.67:0/3054649979 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:23.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.262+0000 7fb79ded7640 1 --2- 172.21.15.67:0/3054649979 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559945d38e50 0x559945d0bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:23.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.262+0000 7fb79d6d6640 1 --2- 172.21.15.67:0/3054649979 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559945d39230 0x559945d3bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:23.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.263+0000 7fb79e6d8640 1 -- 172.21.15.67:0/3054649979 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559945d39230 msgr2=0x559945d3bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:23.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.263+0000 7fb79e6d8640 1 --2- 172.21.15.67:0/3054649979 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559945d39230 0x559945d3bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:23.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.263+0000 7fb79e6d8640 1 -- 172.21.15.67:0/3054649979 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559945d38e50 msgr2=0x559945d0bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:23.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.263+0000 7fb79e6d8640 1 --2- 172.21.15.67:0/3054649979 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559945d38e50 0x559945d0bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:23.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.263+0000 7fb79e6d8640 1 -- 172.21.15.67:0/3054649979 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559945a547d0 con 0x559945d0c330 2024-03-24T00:41:23.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.263+0000 7fb79e6d8640 1 --2- 172.21.15.67:0/3054649979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559945d0c330 0x559945d3c120 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fb790009ee0 tx=0x7fb79002e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f85e6cf49d106d13 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:23.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.263+0000 7fb77b7fe640 1 -- 172.21.15.67:0/3054649979 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb790039040 con 0x559945d0c330 2024-03-24T00:41:23.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.263+0000 7fb77b7fe640 1 -- 172.21.15.67:0/3054649979 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb79003f440 con 0x559945d0c330 2024-03-24T00:41:23.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.263+0000 7fb77b7fe640 1 -- 172.21.15.67:0/3054649979 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79003f7f0 con 0x559945d0c330 2024-03-24T00:41:23.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.264+0000 7fb7a2e02740 1 -- 172.21.15.67:0/3054649979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559945d0c330 msgr2=0x559945d3c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:23.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.264+0000 7fb7a2e02740 1 --2- 172.21.15.67:0/3054649979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559945d0c330 0x559945d3c120 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fb790009ee0 tx=0x7fb79002e000 comp rx=0 tx=0).stop 2024-03-24T00:41:23.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.264+0000 7fb7a2e02740 1 -- 172.21.15.67:0/3054649979 shutdown_connections 2024-03-24T00:41:23.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.264+0000 7fb7a2e02740 1 --2- 172.21.15.67:0/3054649979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559945d0c330 0x559945d3c120 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:23.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.264+0000 7fb7a2e02740 1 --2- 172.21.15.67:0/3054649979 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559945d39230 0x559945d3bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:23.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.264+0000 7fb7a2e02740 1 --2- 172.21.15.67:0/3054649979 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559945d38e50 0x559945d0bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:23.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.264+0000 7fb7a2e02740 1 -- 172.21.15.67:0/3054649979 >> 172.21.15.67:0/3054649979 conn(0x559945d0e060 msgr2=0x559945d56be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:23.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.264+0000 7fb7a2e02740 1 -- 172.21.15.67:0/3054649979 shutdown_connections 2024-03-24T00:41:23.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.264+0000 7fb7a2e02740 1 -- 172.21.15.67:0/3054649979 wait complete. 2024-03-24T00:41:23.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.264+0000 7fb7a2e02740 1 -- start start 2024-03-24T00:41:23.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.264+0000 7fb7a2e02740 1 Processor -- start 2024-03-24T00:41:23.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.265+0000 7fb7a2e02740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559945d38e50 0x559945d67040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:23.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.265+0000 7fb7a2e02740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559945d39230 0x559945d569a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:23.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.265+0000 7fb7a2e02740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559945d56ee0 0x559945d31ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:23.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.265+0000 7fb7a2e02740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x559945d10700 con 0x559945d39230 2024-03-24T00:41:23.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.265+0000 7fb7a2e02740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x559945c32060 con 0x559945d38e50 2024-03-24T00:41:23.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.265+0000 7fb7a2e02740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x559945d12f40 con 0x559945d56ee0 2024-03-24T00:41:23.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.265+0000 7fb7a2e02740 10 client.0 _ll_register_callbacks cb 0x559945c146a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:41:23.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.265+0000 7fb79d6d6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559945d39230 0x559945d569a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:23.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.265+0000 7fb79d6d6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559945d39230 0x559945d569a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47248/0 (socket says 172.21.15.67:47248) 2024-03-24T00:41:23.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.265+0000 7fb79d6d6640 1 -- 172.21.15.67:0/662564249 learned_addr learned my addr 172.21.15.67:0/662564249 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:23.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.265+0000 7fb79e6d8640 1 --2- 172.21.15.67:0/662564249 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559945d56ee0 0x559945d31ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:23.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.265+0000 7fb79ded7640 1 --2- 172.21.15.67:0/662564249 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559945d38e50 0x559945d67040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:23.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.266+0000 7fb79d6d6640 1 -- 172.21.15.67:0/662564249 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559945d56ee0 msgr2=0x559945d31ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:23.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.266+0000 7fb79d6d6640 1 --2- 172.21.15.67:0/662564249 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559945d56ee0 0x559945d31ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:23.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.266+0000 7fb79d6d6640 1 -- 172.21.15.67:0/662564249 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559945d38e50 msgr2=0x559945d67040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:23.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.266+0000 7fb79d6d6640 1 --2- 172.21.15.67:0/662564249 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559945d38e50 0x559945d67040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:23.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.266+0000 7fb79d6d6640 1 -- 172.21.15.67:0/662564249 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55994598a3f0 con 0x559945d39230 2024-03-24T00:41:23.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.266+0000 7fb79d6d6640 1 --2- 172.21.15.67:0/662564249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559945d39230 0x559945d569a0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fb788003940 tx=0x7fb788000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e478537b9d838d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:23.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.266+0000 7fb778ff9640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:41:23.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.266+0000 7fb778ff9640 1 -- 172.21.15.67:0/662564249 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb788027060 con 0x559945d39230 2024-03-24T00:41:23.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.266+0000 7fb778ff9640 1 -- 172.21.15.67:0/662564249 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(50..50 src has 1..50) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb788010040 con 0x559945d39230 2024-03-24T00:41:23.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.266+0000 7fb7a2e02740 1 -- 172.21.15.67:0/662564249 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb79003f440 con 0x559945d39230 2024-03-24T00:41:23.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.266+0000 7fb778ff9640 1 -- 172.21.15.67:0/662564249 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb788003e00 con 0x559945d39230 2024-03-24T00:41:23.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.266+0000 7fb7a2e02740 10 client.6132 fetch_fsmap learned FSMap version 26 2024-03-24T00:41:23.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.266+0000 7fb7a2e02740 1 -- 172.21.15.67:0/662564249 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=2+}) v3 -- 0x55994594a0f0 con 0x559945d39230 2024-03-24T00:41:23.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.266+0000 7fb778ff9640 1 -- 172.21.15.67:0/662564249 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb79003f440 con 0x559945d39230 2024-03-24T00:41:23.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.266+0000 7fb778ff9640 1 -- 172.21.15.67:0/662564249 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 26) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fb78800ec90 con 0x559945d39230 2024-03-24T00:41:23.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.267+0000 7fb7a2e02740 10 client.6132 fetch_fsmap finished waiting for FSMap version 26 2024-03-24T00:41:23.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.267+0000 7fb7a2e02740 10 client.6132 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.a 2024-03-24T00:41:23.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.267+0000 7fb7a2e02740 20 client.6132 populate_metadata read hostname 'smithi067' 2024-03-24T00:41:23.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.267+0000 7fb7a2e02740 1 --2- 172.21.15.67:0/662564249 >> [v2:172.21.15.67:6832/2741818214,v1:172.21.15.67:6834/2741818214] conn(0x559945d6a630 0x559945d6aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:23.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.267+0000 7fb7a2e02740 4 client.6132 mds_command: new command op to 5853 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:41:23.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.267+0000 7fb7a2e02740 1 -- 172.21.15.67:0/662564249 --> [v2:172.21.15.67:6832/2741818214,v1:172.21.15.67:6834/2741818214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559945c83910 con 0x559945d6a630 2024-03-24T00:41:23.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.267+0000 7fb79ded7640 1 --2- 172.21.15.67:0/662564249 >> [v2:172.21.15.67:6832/2741818214,v1:172.21.15.67:6834/2741818214] conn(0x559945d6a630 0x559945d6aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:23.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.267+0000 7fb778ff9640 10 client.6132 ms_handle_connect on v2:172.21.15.67:6832/2741818214 2024-03-24T00:41:23.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.267+0000 7fb79ded7640 1 --2- 172.21.15.67:0/662564249 >> [v2:172.21.15.67:6832/2741818214,v1:172.21.15.67:6834/2741818214] conn(0x559945d6a630 0x559945d6aa40 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c31ed9e93082012 server_cookie=31a708c553c64fd2 in_seq=0 out_seq=0 2024-03-24T00:41:23.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.267+0000 7fb778ff9640 1 -- 172.21.15.67:0/662564249 <== mds.0 v2:172.21.15.67:6832/2741818214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x559945c83910 con 0x559945d6a630 2024-03-24T00:41:23.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.267+0000 7fb778ff9640 10 client.6132 handle_command_reply: tid=0 2024-03-24T00:41:23.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.267+0000 7fb7a2e02740 1 client.6132 shutdown 2024-03-24T00:41:23.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.268+0000 7fb7a2e02740 1 -- 172.21.15.67:0/662564249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559945d39230 msgr2=0x559945d569a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:23.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.268+0000 7fb7a2e02740 1 --2- 172.21.15.67:0/662564249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559945d39230 0x559945d569a0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fb788003940 tx=0x7fb788000ed0 comp rx=0 tx=0).stop 2024-03-24T00:41:23.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.268+0000 7fb7a2e02740 1 -- 172.21.15.67:0/662564249 shutdown_connections 2024-03-24T00:41:23.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.268+0000 7fb7a2e02740 1 --2- 172.21.15.67:0/662564249 >> [v2:172.21.15.67:6832/2741818214,v1:172.21.15.67:6834/2741818214] conn(0x559945d6a630 0x559945d6aa40 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:23.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.268+0000 7fb7a2e02740 1 --2- 172.21.15.67:0/662564249 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559945d56ee0 0x559945d31ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:23.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.268+0000 7fb7a2e02740 1 --2- 172.21.15.67:0/662564249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559945d39230 0x559945d569a0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:23.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.268+0000 7fb778ff9640 0 client.6132 ms_handle_reset on v2:172.21.15.67:6832/2741818214 2024-03-24T00:41:23.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.268+0000 7fb7a2e02740 1 --2- 172.21.15.67:0/662564249 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559945d38e50 0x559945d67040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:23.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.268+0000 7fb7a2e02740 1 -- 172.21.15.67:0/662564249 >> 172.21.15.67:0/662564249 conn(0x559945d0c330 msgr2=0x559945d0fd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:23.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.268+0000 7fb7a2e02740 1 -- 172.21.15.67:0/662564249 shutdown_connections 2024-03-24T00:41:23.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.268+0000 7fb7a2e02740 1 -- 172.21.15.67:0/662564249 wait complete. 2024-03-24T00:41:23.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.268+0000 7fb7a2e02740 20 client.6132 trim_cache size 0 max 16384 2024-03-24T00:41:23.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.284+0000 7fb7a2e02740 1 Processor -- start 2024-03-24T00:41:23.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.284+0000 7fb7a2e02740 1 -- start start 2024-03-24T00:41:23.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.285+0000 7fb7a2e02740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559945d38e50 0x559945d37550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:23.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.285+0000 7fb7a2e02740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559945d39230 0x559945d69a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:23.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.285+0000 7fb7a2e02740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559945d56ee0 0x559945d3cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:23.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.285+0000 7fb7a2e02740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x559945d613c0 con 0x559945d38e50 2024-03-24T00:41:23.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.285+0000 7fb7a2e02740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb798200660 con 0x559945d56ee0 2024-03-24T00:41:23.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.285+0000 7fb7a2e02740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7981ff7d0 con 0x559945d39230 2024-03-24T00:41:23.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.285+0000 7fb79ded7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559945d38e50 0x559945d37550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:23.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.285+0000 7fb79ded7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559945d38e50 0x559945d37550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47262/0 (socket says 172.21.15.67:47262) 2024-03-24T00:41:23.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.285+0000 7fb79ded7640 1 -- 172.21.15.67:0/562814042 learned_addr learned my addr 172.21.15.67:0/562814042 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:23.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.285+0000 7fb79e6d8640 1 --2- 172.21.15.67:0/562814042 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559945d56ee0 0x559945d3cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:23.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.285+0000 7fb79d6d6640 1 --2- 172.21.15.67:0/562814042 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559945d39230 0x559945d69a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:23.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.285+0000 7fb79ded7640 1 -- 172.21.15.67:0/562814042 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559945d39230 msgr2=0x559945d69a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:23.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.285+0000 7fb79ded7640 1 --2- 172.21.15.67:0/562814042 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559945d39230 0x559945d69a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:23.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.285+0000 7fb79ded7640 1 -- 172.21.15.67:0/562814042 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559945d56ee0 msgr2=0x559945d3cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:23.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.285+0000 7fb79ded7640 1 --2- 172.21.15.67:0/562814042 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559945d56ee0 0x559945d3cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:23.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.285+0000 7fb79ded7640 1 -- 172.21.15.67:0/562814042 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559945db8440 con 0x559945d38e50 2024-03-24T00:41:23.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.286+0000 7fb79ded7640 1 --2- 172.21.15.67:0/562814042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559945d38e50 0x559945d37550 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fb79400c500 tx=0x7fb79405b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbce52eade160d5b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:23.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.286+0000 7fb77affd640 1 -- 172.21.15.67:0/562814042 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb794085070 con 0x559945d38e50 2024-03-24T00:41:23.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.286+0000 7fb77affd640 1 -- 172.21.15.67:0/562814042 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7940562b0 con 0x559945d38e50 2024-03-24T00:41:23.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.286+0000 7fb77affd640 1 -- 172.21.15.67:0/562814042 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb794054880 con 0x559945d38e50 2024-03-24T00:41:23.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.286+0000 7fb7a2e02740 1 -- 172.21.15.67:0/562814042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559945d38e50 msgr2=0x559945d37550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:23.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.286+0000 7fb7a2e02740 1 --2- 172.21.15.67:0/562814042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559945d38e50 0x559945d37550 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fb79400c500 tx=0x7fb79405b040 comp rx=0 tx=0).stop 2024-03-24T00:41:23.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.286+0000 7fb7a2e02740 1 -- 172.21.15.67:0/562814042 shutdown_connections 2024-03-24T00:41:23.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.286+0000 7fb7a2e02740 1 --2- 172.21.15.67:0/562814042 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559945d56ee0 0x559945d3cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:23.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.286+0000 7fb7a2e02740 1 --2- 172.21.15.67:0/562814042 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559945d39230 0x559945d69a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:23.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.286+0000 7fb7a2e02740 1 --2- 172.21.15.67:0/562814042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559945d38e50 0x559945d37550 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:23.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.286+0000 7fb7a2e02740 1 -- 172.21.15.67:0/562814042 >> 172.21.15.67:0/562814042 conn(0x559945d0c330 msgr2=0x559945d70e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:23.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.286+0000 7fb7a2e02740 1 -- 172.21.15.67:0/562814042 shutdown_connections 2024-03-24T00:41:23.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.286+0000 7fb7a2e02740 1 -- 172.21.15.67:0/562814042 wait complete. 2024-03-24T00:41:23.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.287+0000 7fb7a2e02740 1 -- start start 2024-03-24T00:41:23.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.287+0000 7fb7a2e02740 1 Processor -- start 2024-03-24T00:41:23.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.288+0000 7fb7a2e02740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559945d38e50 0x559945d7a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:23.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.288+0000 7fb7a2e02740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559945d39230 0x559945daaff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:23.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.288+0000 7fb7a2e02740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559945d56ee0 0x559945d63710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:23.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.288+0000 7fb7a2e02740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7981ff1d0 con 0x559945d39230 2024-03-24T00:41:23.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.288+0000 7fb7a2e02740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb79814f080 con 0x559945d56ee0 2024-03-24T00:41:23.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.288+0000 7fb7a2e02740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x559945dab530 con 0x559945d38e50 2024-03-24T00:41:23.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.288+0000 7fb7a2e02740 10 client.0 _ll_register_callbacks cb 0x7fb788001420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:41:23.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.288+0000 7fb79d6d6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559945d39230 0x559945daaff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:23.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.288+0000 7fb79d6d6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559945d39230 0x559945daaff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47278/0 (socket says 172.21.15.67:47278) 2024-03-24T00:41:23.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.288+0000 7fb79d6d6640 1 -- 172.21.15.67:0/1687264256 learned_addr learned my addr 172.21.15.67:0/1687264256 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:23.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.288+0000 7fb79ded7640 1 --2- 172.21.15.67:0/1687264256 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559945d38e50 0x559945d7a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:23.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.288+0000 7fb79e6d8640 1 --2- 172.21.15.67:0/1687264256 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559945d56ee0 0x559945d63710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:23.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.288+0000 7fb79d6d6640 1 -- 172.21.15.67:0/1687264256 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559945d38e50 msgr2=0x559945d7a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:23.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.288+0000 7fb79d6d6640 1 --2- 172.21.15.67:0/1687264256 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559945d38e50 0x559945d7a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:23.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.288+0000 7fb79d6d6640 1 -- 172.21.15.67:0/1687264256 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559945d56ee0 msgr2=0x559945d63710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:23.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.288+0000 7fb79d6d6640 1 --2- 172.21.15.67:0/1687264256 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559945d56ee0 0x559945d63710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:23.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.288+0000 7fb79d6d6640 1 -- 172.21.15.67:0/1687264256 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559945d13250 con 0x559945d39230 2024-03-24T00:41:23.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.288+0000 7fb79d6d6640 1 --2- 172.21.15.67:0/1687264256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559945d39230 0x559945daaff0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fb788005330 tx=0x7fb788013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11c22ca5fefa2dda server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:23.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.288+0000 7fb778ff9640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:41:23.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.289+0000 7fb778ff9640 1 -- 172.21.15.67:0/1687264256 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb78800e8d0 con 0x559945d39230 2024-03-24T00:41:23.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.289+0000 7fb778ff9640 1 -- 172.21.15.67:0/1687264256 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(50..50 src has 1..50) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb788010040 con 0x559945d39230 2024-03-24T00:41:23.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.289+0000 7fb7a2e02740 1 -- 172.21.15.67:0/1687264256 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb7940562b0 con 0x559945d39230 2024-03-24T00:41:23.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.289+0000 7fb778ff9640 1 -- 172.21.15.67:0/1687264256 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb78803c050 con 0x559945d39230 2024-03-24T00:41:23.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.289+0000 7fb7a2e02740 10 client.6138 fetch_fsmap learned FSMap version 26 2024-03-24T00:41:23.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.289+0000 7fb7a2e02740 1 -- 172.21.15.67:0/1687264256 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=2+}) v3 -- 0x559945d65dc0 con 0x559945d39230 2024-03-24T00:41:23.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.289+0000 7fb778ff9640 1 -- 172.21.15.67:0/1687264256 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7940562b0 con 0x559945d39230 2024-03-24T00:41:23.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.289+0000 7fb778ff9640 1 -- 172.21.15.67:0/1687264256 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 26) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fb78803f020 con 0x559945d39230 2024-03-24T00:41:23.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.289+0000 7fb7a2e02740 10 client.6138 fetch_fsmap finished waiting for FSMap version 26 2024-03-24T00:41:23.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.289+0000 7fb7a2e02740 10 client.6138 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.a 2024-03-24T00:41:23.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.289+0000 7fb7a2e02740 20 client.6138 populate_metadata read hostname 'smithi067' 2024-03-24T00:41:23.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.289+0000 7fb7a2e02740 1 --2- 172.21.15.67:0/1687264256 >> [v2:172.21.15.67:6832/2741818214,v1:172.21.15.67:6834/2741818214] conn(0x559945d65290 0x559945d656a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:23.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.289+0000 7fb7a2e02740 4 client.6138 mds_command: new command op to 5853 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:41:23.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.289+0000 7fb7a2e02740 1 -- 172.21.15.67:0/1687264256 --> [v2:172.21.15.67:6832/2741818214,v1:172.21.15.67:6834/2741818214] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x559945c7dc60 con 0x559945d65290 2024-03-24T00:41:23.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.289+0000 7fb79ded7640 1 --2- 172.21.15.67:0/1687264256 >> [v2:172.21.15.67:6832/2741818214,v1:172.21.15.67:6834/2741818214] conn(0x559945d65290 0x559945d656a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:23.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.289+0000 7fb778ff9640 10 client.6138 ms_handle_connect on v2:172.21.15.67:6832/2741818214 2024-03-24T00:41:23.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.289+0000 7fb79ded7640 1 --2- 172.21.15.67:0/1687264256 >> [v2:172.21.15.67:6832/2741818214,v1:172.21.15.67:6834/2741818214] conn(0x559945d65290 0x559945d656a0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2973714e0e81d64 server_cookie=abe5d4adcc47002a in_seq=0 out_seq=0 2024-03-24T00:41:23.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.290+0000 7fb778ff9640 1 -- 172.21.15.67:0/1687264256 <== mds.0 v2:172.21.15.67:6832/2741818214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x559945c7dc60 con 0x559945d65290 2024-03-24T00:41:23.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.290+0000 7fb778ff9640 10 client.6138 handle_command_reply: tid=0 2024-03-24T00:41:23.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.290+0000 7fb7a2e02740 1 client.6138 shutdown 2024-03-24T00:41:23.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.290+0000 7fb7a2e02740 1 -- 172.21.15.67:0/1687264256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559945d39230 msgr2=0x559945daaff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:23.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.290+0000 7fb7a2e02740 1 --2- 172.21.15.67:0/1687264256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559945d39230 0x559945daaff0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fb788005330 tx=0x7fb788013000 comp rx=0 tx=0).stop 2024-03-24T00:41:23.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.290+0000 7fb7a2e02740 1 -- 172.21.15.67:0/1687264256 shutdown_connections 2024-03-24T00:41:23.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.290+0000 7fb7a2e02740 1 --2- 172.21.15.67:0/1687264256 >> [v2:172.21.15.67:6832/2741818214,v1:172.21.15.67:6834/2741818214] conn(0x559945d65290 0x559945d656a0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:23.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.290+0000 7fb7a2e02740 1 --2- 172.21.15.67:0/1687264256 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559945d56ee0 0x559945d63710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:23.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.290+0000 7fb7a2e02740 1 --2- 172.21.15.67:0/1687264256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559945d39230 0x559945daaff0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:23.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.290+0000 7fb7a2e02740 1 --2- 172.21.15.67:0/1687264256 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559945d38e50 0x559945d7a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:23.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.290+0000 7fb778ff9640 0 client.6138 ms_handle_reset on v2:172.21.15.67:6832/2741818214 2024-03-24T00:41:23.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.290+0000 7fb7a2e02740 1 -- 172.21.15.67:0/1687264256 >> 172.21.15.67:0/1687264256 conn(0x559945d0c330 msgr2=0x559945d70a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:23.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.290+0000 7fb7a2e02740 1 -- 172.21.15.67:0/1687264256 shutdown_connections 2024-03-24T00:41:23.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.290+0000 7fb7a2e02740 1 -- 172.21.15.67:0/1687264256 wait complete. 2024-03-24T00:41:23.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.290+0000 7fb7a2e02740 20 client.6138 trim_cache size 0 max 16384 2024-03-24T00:41:23.302 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:41:23.302 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "no active scrubs running", 2024-03-24T00:41:23.302 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": {} 2024-03-24T00:41:23.302 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:41:23.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.290+0000 7fb7a013b640 1 -- 172.21.15.67:0/2644958209 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb77c06bbd0 msgr2=0x7fb77c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:23.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.290+0000 7fb7a013b640 1 --2- 172.21.15.67:0/2644958209 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb77c06bbd0 0x7fb77c06e090 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fb7981608c0 tx=0x7fb788009040 comp rx=0 tx=0).stop 2024-03-24T00:41:23.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.290+0000 7fb7a013b640 1 -- 172.21.15.67:0/2644958209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7980b6170 msgr2=0x7fb798150610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:23.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.290+0000 7fb7a013b640 1 --2- 172.21.15.67:0/2644958209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7980b6170 0x7fb798150610 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fb79400c530 tx=0x7fb79400c9f0 comp rx=0 tx=0).stop 2024-03-24T00:41:23.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.290+0000 7fb7a013b640 1 -- 172.21.15.67:0/2644958209 shutdown_connections 2024-03-24T00:41:23.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.290+0000 7fb7a013b640 1 --2- 172.21.15.67:0/2644958209 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb77c06bbd0 0x7fb77c06e090 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:23.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.291+0000 7fb7a013b640 1 --2- 172.21.15.67:0/2644958209 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb79815dea0 0x7fb7980ccc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:23.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.291+0000 7fb7a013b640 1 --2- 172.21.15.67:0/2644958209 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7980d3ae0 0x7fb79814d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:23.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.291+0000 7fb7a013b640 1 --2- 172.21.15.67:0/2644958209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7980b6170 0x7fb798150610 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:23.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.291+0000 7fb7a013b640 1 -- 172.21.15.67:0/2644958209 >> 172.21.15.67:0/2644958209 conn(0x7fb798072da0 msgr2=0x7fb7980b5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:23.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.291+0000 7fb7a013b640 1 -- 172.21.15.67:0/2644958209 shutdown_connections 2024-03-24T00:41:23.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.291+0000 7fb7a013b640 1 -- 172.21.15.67:0/2644958209 wait complete. 2024-03-24T00:41:23.305 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T00:41:23.405 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:41:23.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.507+0000 7f615f159640 1 Processor -- start 2024-03-24T00:41:23.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.507+0000 7f615f159640 1 -- start start 2024-03-24T00:41:23.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.508+0000 7f615f159640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f615815dec0 0x7f61580ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:23.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.508+0000 7f615f159640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f61580d0ef0 0x7f6158162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:23.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.508+0000 7f615f159640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61580cf640 con 0x7f615815e2a0 2024-03-24T00:41:23.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.509+0000 7f615f159640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61580cee30 con 0x7f61580d0ef0 2024-03-24T00:41:23.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.509+0000 7f615f159640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61580d2120 con 0x7f615815dec0 2024-03-24T00:41:23.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.509+0000 7f615d6f6640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f61580d0ef0 0x7f6158162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:23.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.509+0000 7f615d6f6640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f61580d0ef0 0x7f6158162630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37306/0 (socket says 172.21.15.67:37306) 2024-03-24T00:41:23.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.509+0000 7f615d6f6640 1 -- 172.21.15.67:0/1741057215 learned_addr learned my addr 172.21.15.67:0/1741057215 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:23.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.509+0000 7f615cef5640 1 --2- 172.21.15.67:0/1741057215 >> v2:172.21.15.72:3300/0 conn(0x7f615815dec0 0x7f61580ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:23.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.509+0000 7f615d6f6640 1 -- 172.21.15.67:0/1741057215 >> v2:172.21.15.72:3300/0 conn(0x7f615815dec0 msgr2=0x7f61580ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:23.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.509+0000 7f615d6f6640 1 --2- 172.21.15.67:0/1741057215 >> v2:172.21.15.72:3300/0 conn(0x7f615815dec0 0x7f61580ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:23.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.509+0000 7f615d6f6640 1 -- 172.21.15.67:0/1741057215 >> v1:172.21.15.72:6789/0 conn(0x7f615815e2a0 legacy=0x7f61580d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:23.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.509+0000 7f615d6f6640 1 -- 172.21.15.67:0/1741057215 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61580cf080 con 0x7f61580d0ef0 2024-03-24T00:41:23.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.510+0000 7f615d6f6640 1 --2- 172.21.15.67:0/1741057215 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61580d0ef0 0x7f6158162630 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f61580d14a0 tx=0x7f615002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faef1db26554c0a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:23.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.510+0000 7f614f7fe640 1 -- 172.21.15.67:0/1741057215 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615002c9f0 con 0x7f61580d0ef0 2024-03-24T00:41:23.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.510+0000 7f614f7fe640 1 -- 172.21.15.67:0/1741057215 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f615002cb90 con 0x7f61580d0ef0 2024-03-24T00:41:23.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.510+0000 7f614f7fe640 1 -- 172.21.15.67:0/1741057215 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61500343f0 con 0x7f61580d0ef0 2024-03-24T00:41:23.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.511+0000 7f615f159640 1 -- 172.21.15.67:0/1741057215 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61580d0ef0 msgr2=0x7f6158162630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:23.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.511+0000 7f615f159640 1 --2- 172.21.15.67:0/1741057215 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61580d0ef0 0x7f6158162630 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f61580d14a0 tx=0x7f615002be50 comp rx=0 tx=0).stop 2024-03-24T00:41:23.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.511+0000 7f615f159640 1 -- 172.21.15.67:0/1741057215 shutdown_connections 2024-03-24T00:41:23.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.511+0000 7f615f159640 1 --2- 172.21.15.67:0/1741057215 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61580d0ef0 0x7f6158162630 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:23.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.511+0000 7f615f159640 1 --2- 172.21.15.67:0/1741057215 >> v2:172.21.15.72:3300/0 conn(0x7f615815dec0 0x7f61580ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:23.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.511+0000 7f615f159640 1 -- 172.21.15.67:0/1741057215 >> 172.21.15.67:0/1741057215 conn(0x7f615806c810 msgr2=0x7f615806cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:23.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.511+0000 7f615f159640 1 -- 172.21.15.67:0/1741057215 shutdown_connections 2024-03-24T00:41:23.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.511+0000 7f615f159640 1 -- 172.21.15.67:0/1741057215 wait complete. 2024-03-24T00:41:23.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.512+0000 7f615f159640 1 Processor -- start 2024-03-24T00:41:23.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.512+0000 7f615f159640 1 -- start start 2024-03-24T00:41:23.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.513+0000 7f615f159640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61580d0ef0 0x7f615815b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:23.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.513+0000 7f615f159640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f615815dec0 0x7f61580b7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:23.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.513+0000 7f615f159640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f615815e2a0 0x7f61580b7540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:23.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.513+0000 7f615f159640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61580d2730 con 0x7f61580d0ef0 2024-03-24T00:41:23.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.513+0000 7f615f159640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61580d25b0 con 0x7f615815dec0 2024-03-24T00:41:23.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.513+0000 7f615f159640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61580d28b0 con 0x7f615815e2a0 2024-03-24T00:41:23.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.513+0000 7f615cef5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61580d0ef0 0x7f615815b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:23.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.513+0000 7f615cef5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61580d0ef0 0x7f615815b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37322/0 (socket says 172.21.15.67:37322) 2024-03-24T00:41:23.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.513+0000 7f615cef5640 1 -- 172.21.15.67:0/743036247 learned_addr learned my addr 172.21.15.67:0/743036247 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:23.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.513+0000 7f615d6f6640 1 --2- 172.21.15.67:0/743036247 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f615815e2a0 0x7f61580b7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:23.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.513+0000 7f614ffff640 1 --2- 172.21.15.67:0/743036247 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f615815dec0 0x7f61580b7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:23.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.513+0000 7f615cef5640 1 -- 172.21.15.67:0/743036247 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f615815e2a0 msgr2=0x7f61580b7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:23.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.513+0000 7f615cef5640 1 --2- 172.21.15.67:0/743036247 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f615815e2a0 0x7f61580b7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:23.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.513+0000 7f615cef5640 1 -- 172.21.15.67:0/743036247 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f615815dec0 msgr2=0x7f61580b7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:23.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.513+0000 7f615cef5640 1 --2- 172.21.15.67:0/743036247 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f615815dec0 0x7f61580b7000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:23.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.513+0000 7f615cef5640 1 -- 172.21.15.67:0/743036247 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61580ac0f0 con 0x7f61580d0ef0 2024-03-24T00:41:23.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.513+0000 7f615d6f6640 1 --2- 172.21.15.67:0/743036247 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f615815e2a0 0x7f61580b7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:41:23.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.513+0000 7f614ffff640 1 --2- 172.21.15.67:0/743036247 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f615815dec0 0x7f61580b7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:41:23.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.513+0000 7f615cef5640 1 --2- 172.21.15.67:0/743036247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61580d0ef0 0x7f615815b500 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f615400d800 tx=0x7f615400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:23.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.514+0000 7f614d7fa640 1 -- 172.21.15.67:0/743036247 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615400b6c0 con 0x7f61580d0ef0 2024-03-24T00:41:23.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.514+0000 7f614d7fa640 1 -- 172.21.15.67:0/743036247 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6154019070 con 0x7f61580d0ef0 2024-03-24T00:41:23.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.514+0000 7f615f159640 1 -- 172.21.15.67:0/743036247 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61580b4860 con 0x7f61580d0ef0 2024-03-24T00:41:23.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.514+0000 7f614d7fa640 1 -- 172.21.15.67:0/743036247 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615400ba80 con 0x7f61580d0ef0 2024-03-24T00:41:23.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.514+0000 7f615f159640 1 -- 172.21.15.67:0/743036247 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61580b4e20 con 0x7f61580d0ef0 2024-03-24T00:41:23.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.515+0000 7f615f159640 1 -- 172.21.15.67:0/743036247 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f615804f400 con 0x7f61580d0ef0 2024-03-24T00:41:23.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.515+0000 7f614d7fa640 1 -- 172.21.15.67:0/743036247 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f615400bc20 con 0x7f61580d0ef0 2024-03-24T00:41:23.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.515+0000 7f614d7fa640 1 --2- 172.21.15.67:0/743036247 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f614406bbd0 0x7f614406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:23.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.516+0000 7f614d7fa640 1 -- 172.21.15.67:0/743036247 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f615408b060 con 0x7f61580d0ef0 2024-03-24T00:41:23.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.516+0000 7f614ffff640 1 --2- 172.21.15.67:0/743036247 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f614406bbd0 0x7f614406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:23.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.516+0000 7f614ffff640 1 --2- 172.21.15.67:0/743036247 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f614406bbd0 0x7f614406e090 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f61581613e0 tx=0x7f6148008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:23.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.520+0000 7f614d7fa640 1 -- 172.21.15.67:0/743036247 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6154059a50 con 0x7f61580d0ef0 2024-03-24T00:41:23.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.889+0000 7f615f159640 1 -- 172.21.15.67:0/743036247 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f61580d1aa0 con 0x7f61580d0ef0 2024-03-24T00:41:23.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.889+0000 7f614d7fa640 1 -- 172.21.15.67:0/743036247 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f6154059350 con 0x7f61580d0ef0 2024-03-24T00:41:23.890 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:41:23.891 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5360,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3312887486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":3312887486},{"type":"v1","addr":"172.21.15.67:6835","nonce":3312887486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5821,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/4171866637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":4171866637},{"type":"v1","addr":"172.21.15.72:6836","nonce":4171866637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5824,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/4106412452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":4106412452},{"type":"v1","addr":"172.21.15.72:6837","nonce":4106412452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:40:39.649631+0000","modified":"2024-03-24T00:40:42.657888+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5853},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5853":{"gid":5853,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6834/2741818214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2741818214},{"type":"v1","addr":"172.21.15.67:6834","nonce":2741818214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5853,"qdb_cluster":[5853]},"id":3}]} 2024-03-24T00:41:23.891 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 26 2024-03-24T00:41:23.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.892+0000 7f615f159640 1 -- 172.21.15.67:0/743036247 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f614406bbd0 msgr2=0x7f614406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:23.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.892+0000 7f615f159640 1 --2- 172.21.15.67:0/743036247 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f614406bbd0 0x7f614406e090 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f61581613e0 tx=0x7f6148008040 comp rx=0 tx=0).stop 2024-03-24T00:41:23.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.892+0000 7f615f159640 1 -- 172.21.15.67:0/743036247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61580d0ef0 msgr2=0x7f615815b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:23.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.892+0000 7f615f159640 1 --2- 172.21.15.67:0/743036247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61580d0ef0 0x7f615815b500 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f615400d800 tx=0x7f615400dcc0 comp rx=0 tx=0).stop 2024-03-24T00:41:23.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.892+0000 7f615f159640 1 -- 172.21.15.67:0/743036247 shutdown_connections 2024-03-24T00:41:23.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.892+0000 7f615f159640 1 --2- 172.21.15.67:0/743036247 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f614406bbd0 0x7f614406e090 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:23.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.892+0000 7f615f159640 1 --2- 172.21.15.67:0/743036247 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f615815e2a0 0x7f61580b7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:23.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.892+0000 7f615f159640 1 --2- 172.21.15.67:0/743036247 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f615815dec0 0x7f61580b7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:23.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.892+0000 7f615f159640 1 --2- 172.21.15.67:0/743036247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61580d0ef0 0x7f615815b500 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:23.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.892+0000 7f615f159640 1 -- 172.21.15.67:0/743036247 >> 172.21.15.67:0/743036247 conn(0x7f615806c810 msgr2=0x7f615806ce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:23.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.892+0000 7f615f159640 1 -- 172.21.15.67:0/743036247 shutdown_connections 2024-03-24T00:41:23.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:23.892+0000 7f615f159640 1 -- 172.21.15.67:0/743036247 wait complete. 2024-03-24T00:41:23.903 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-24T00:41:24.029 INFO:teuthology.orchestra.run.smithi067.stdout:{"message":"","return_code":0} 2024-03-24T00:41:24.029 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-24T00:41:24.029 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-24T00:41:24.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.071+0000 7fa69875bf40 1 Processor -- start 2024-03-24T00:41:24.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.071+0000 7fa69875bf40 1 -- start start 2024-03-24T00:41:24.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.072+0000 7fa69875bf40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa680002460 0x56040fdc5070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:24.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.072+0000 7fa69875bf40 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56040fdc12c0 0x56040fdbf300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:24.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.072+0000 7fa69875bf40 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56040fdbf840 0x56040fd1d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:24.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.072+0000 7fa69875bf40 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x56040fc980d0 con 0x56040fdc12c0 2024-03-24T00:41:24.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.072+0000 7fa69875bf40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x56040fc9c030 con 0x7fa680002460 2024-03-24T00:41:24.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.072+0000 7fa69875bf40 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x56040fdc6740 con 0x56040fdbf840 2024-03-24T00:41:24.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.072+0000 7fa695a58640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56040fdc12c0 0x56040fdbf300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:24.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.072+0000 7fa695a58640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56040fdc12c0 0x56040fdbf300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37340/0 (socket says 172.21.15.67:37340) 2024-03-24T00:41:24.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.072+0000 7fa695a58640 1 -- 172.21.15.67:0/3742795323 learned_addr learned my addr 172.21.15.67:0/3742795323 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:24.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.072+0000 7fa695a58640 1 -- 172.21.15.67:0/3742795323 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56040fdbf840 msgr2=0x56040fd1d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:24.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.072+0000 7fa695a58640 1 --2- 172.21.15.67:0/3742795323 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56040fdbf840 0x56040fd1d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:24.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.072+0000 7fa695a58640 1 -- 172.21.15.67:0/3742795323 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa680002460 msgr2=0x56040fdc5070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:24.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.072+0000 7fa695a58640 1 --2- 172.21.15.67:0/3742795323 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa680002460 0x56040fdc5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:24.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.072+0000 7fa695a58640 1 -- 172.21.15.67:0/3742795323 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56040fd20800 con 0x56040fdc12c0 2024-03-24T00:41:24.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.072+0000 7fa695a58640 1 --2- 172.21.15.67:0/3742795323 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56040fdc12c0 0x56040fdbf300 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fa68c009690 tx=0x7fa68c009a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81050050da9d98a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:24.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.072+0000 7fa694a56640 1 -- 172.21.15.67:0/3742795323 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa68c00e860 con 0x56040fdc12c0 2024-03-24T00:41:24.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.072+0000 7fa694a56640 1 -- 172.21.15.67:0/3742795323 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa68c013030 con 0x56040fdc12c0 2024-03-24T00:41:24.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.072+0000 7fa694a56640 1 -- 172.21.15.67:0/3742795323 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa68c00ec20 con 0x56040fdc12c0 2024-03-24T00:41:24.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.073+0000 7fa69875bf40 1 -- 172.21.15.67:0/3742795323 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56040fdc12c0 msgr2=0x56040fdbf300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:24.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.073+0000 7fa69875bf40 1 --2- 172.21.15.67:0/3742795323 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56040fdc12c0 0x56040fdbf300 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fa68c009690 tx=0x7fa68c009a90 comp rx=0 tx=0).stop 2024-03-24T00:41:24.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.073+0000 7fa69875bf40 1 -- 172.21.15.67:0/3742795323 shutdown_connections 2024-03-24T00:41:24.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.073+0000 7fa69875bf40 1 --2- 172.21.15.67:0/3742795323 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56040fdbf840 0x56040fd1d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:24.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.073+0000 7fa69875bf40 1 --2- 172.21.15.67:0/3742795323 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56040fdc12c0 0x56040fdbf300 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:24.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.073+0000 7fa69875bf40 1 --2- 172.21.15.67:0/3742795323 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa680002460 0x56040fdc5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:24.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.073+0000 7fa69875bf40 1 -- 172.21.15.67:0/3742795323 >> 172.21.15.67:0/3742795323 conn(0x56040fcefe70 msgr2=0x56040fd1e580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:24.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.073+0000 7fa69875bf40 1 -- 172.21.15.67:0/3742795323 shutdown_connections 2024-03-24T00:41:24.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.073+0000 7fa69875bf40 1 -- 172.21.15.67:0/3742795323 wait complete. 2024-03-24T00:41:24.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.074+0000 7fa69875bf40 1 Processor -- start 2024-03-24T00:41:24.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.074+0000 7fa69875bf40 1 -- start start 2024-03-24T00:41:24.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.074+0000 7fa69875bf40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56040fdbf840 0x56040fdb8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:24.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.074+0000 7fa69875bf40 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56040fdc12c0 0x56040fd1e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:24.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.074+0000 7fa69875bf40 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa680002460 0x56040fdc1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:24.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.074+0000 7fa69875bf40 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x56040fdc2d00 con 0x7fa680002460 2024-03-24T00:41:24.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.074+0000 7fa69875bf40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x56040fd205b0 con 0x56040fdbf840 2024-03-24T00:41:24.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.074+0000 7fa69875bf40 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x56040fdc3350 con 0x56040fdc12c0 2024-03-24T00:41:24.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.074+0000 7fa696a5a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa680002460 0x56040fdc1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:24.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.074+0000 7fa696a5a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa680002460 0x56040fdc1750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37344/0 (socket says 172.21.15.67:37344) 2024-03-24T00:41:24.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.074+0000 7fa696a5a640 1 -- 172.21.15.67:0/4081584480 learned_addr learned my addr 172.21.15.67:0/4081584480 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:24.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.074+0000 7fa695a58640 1 --2- 172.21.15.67:0/4081584480 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56040fdc12c0 0x56040fd1e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:24.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.074+0000 7fa696259640 1 --2- 172.21.15.67:0/4081584480 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56040fdbf840 0x56040fdb8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:24.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.075+0000 7fa696a5a640 1 -- 172.21.15.67:0/4081584480 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56040fdc12c0 msgr2=0x56040fd1e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:24.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.075+0000 7fa696a5a640 1 --2- 172.21.15.67:0/4081584480 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56040fdc12c0 0x56040fd1e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:24.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.075+0000 7fa696a5a640 1 -- 172.21.15.67:0/4081584480 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56040fdbf840 msgr2=0x56040fdb8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:24.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.075+0000 7fa696a5a640 1 --2- 172.21.15.67:0/4081584480 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56040fdbf840 0x56040fdb8ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:24.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.075+0000 7fa696a5a640 1 -- 172.21.15.67:0/4081584480 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56040fd1d7b0 con 0x7fa680002460 2024-03-24T00:41:24.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.075+0000 7fa696a5a640 1 --2- 172.21.15.67:0/4081584480 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa680002460 0x56040fdc1750 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fa690005fd0 tx=0x7fa690010980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:24.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.075+0000 7fa6877fe640 1 -- 172.21.15.67:0/4081584480 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa690017030 con 0x7fa680002460 2024-03-24T00:41:24.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.075+0000 7fa6877fe640 1 -- 172.21.15.67:0/4081584480 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa690012730 con 0x7fa680002460 2024-03-24T00:41:24.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.075+0000 7fa69875bf40 1 -- 172.21.15.67:0/4081584480 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x56040fd346f0 con 0x7fa680002460 2024-03-24T00:41:24.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.075+0000 7fa6877fe640 1 -- 172.21.15.67:0/4081584480 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa690012920 con 0x7fa680002460 2024-03-24T00:41:24.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.075+0000 7fa69875bf40 1 -- 172.21.15.67:0/4081584480 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56040fd1dd50 con 0x7fa680002460 2024-03-24T00:41:24.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.076+0000 7fa6877fe640 1 -- 172.21.15.67:0/4081584480 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa69001a030 con 0x7fa680002460 2024-03-24T00:41:24.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.077+0000 7fa6877fe640 1 --2- 172.21.15.67:0/4081584480 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa67806a5b0 0x7fa67806ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:24.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.077+0000 7fa6877fe640 1 -- 172.21.15.67:0/4081584480 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa690089960 con 0x7fa680002460 2024-03-24T00:41:24.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.077+0000 7fa696259640 1 --2- 172.21.15.67:0/4081584480 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa67806a5b0 0x7fa67806ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:24.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.077+0000 7fa69875bf40 1 --2- 172.21.15.67:0/4081584480 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x56040fd34260 0x56040fe5f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:24.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.077+0000 7fa69875bf40 1 -- 172.21.15.67:0/4081584480 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:1 8.7 8:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50) v9 -- 0x56040fd1ea90 con 0x56040fd34260 2024-03-24T00:41:24.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.077+0000 7fa695a58640 1 --2- 172.21.15.67:0/4081584480 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x56040fd34260 0x56040fe5f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:24.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.078+0000 7fa695a58640 1 --2- 172.21.15.67:0/4081584480 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x56040fd34260 0x56040fe5f820 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:24.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.078+0000 7fa696259640 1 --2- 172.21.15.67:0/4081584480 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa67806a5b0 0x7fa67806ca70 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fa68800b590 tx=0x7fa68804c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:24.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.078+0000 7fa695a58640 1 -- 172.21.15.67:0/4081584480 <== osd.1 v2:172.21.15.67:6800/4062510404 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (95) out=95b] v0'0 uv3 ondisk = 0) v8 ==== 164+0+95 (crc 0 0 0) 0x7fa68c00e9e0 con 0x56040fd34260 2024-03-24T00:41:24.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.078+0000 7fa69875bf40 1 -- 172.21.15.67:0/4081584480 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x56040fd34260 msgr2=0x56040fe5f820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:24.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.078+0000 7fa69875bf40 1 --2- 172.21.15.67:0/4081584480 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x56040fd34260 0x56040fe5f820 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:24.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.078+0000 7fa69875bf40 1 -- 172.21.15.67:0/4081584480 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa67806a5b0 msgr2=0x7fa67806ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:24.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.078+0000 7fa69875bf40 1 --2- 172.21.15.67:0/4081584480 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa67806a5b0 0x7fa67806ca70 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fa68800b590 tx=0x7fa68804c000 comp rx=0 tx=0).stop 2024-03-24T00:41:24.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.078+0000 7fa69875bf40 1 -- 172.21.15.67:0/4081584480 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa680002460 msgr2=0x56040fdc1750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:24.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.078+0000 7fa69875bf40 1 --2- 172.21.15.67:0/4081584480 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa680002460 0x56040fdc1750 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fa690005fd0 tx=0x7fa690010980 comp rx=0 tx=0).stop 2024-03-24T00:41:24.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.078+0000 7fa696a5a640 1 -- 172.21.15.67:0/4081584480 reap_dead start 2024-03-24T00:41:24.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.078+0000 7fa69875bf40 1 -- 172.21.15.67:0/4081584480 shutdown_connections 2024-03-24T00:41:24.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.078+0000 7fa69875bf40 1 -- 172.21.15.67:0/4081584480 >> 172.21.15.67:0/4081584480 conn(0x56040fcefe70 msgr2=0x56040fd20a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:24.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.078+0000 7fa69875bf40 1 -- 172.21.15.67:0/4081584480 shutdown_connections 2024-03-24T00:41:24.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.078+0000 7fa69875bf40 1 -- 172.21.15.67:0/4081584480 wait complete. 2024-03-24T00:41:24.084 INFO:teuthology.orchestra.run.smithi067.stdout:Yalphaparent_a 2024-03-24T00:41:24.084 DEBUG:teuthology.orchestra.run.smithi067:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-24T00:41:24.170 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:41:24.170 INFO:teuthology.orchestra.run.smithi067.stdout: "ino": 1099511627777, 2024-03-24T00:41:24.170 INFO:teuthology.orchestra.run.smithi067.stdout: "ancestors": [ 2024-03-24T00:41:24.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:41:24.171 INFO:teuthology.orchestra.run.smithi067.stdout: "dirino": 1099511627776, 2024-03-24T00:41:24.171 INFO:teuthology.orchestra.run.smithi067.stdout: "dname": "alpha", 2024-03-24T00:41:24.171 INFO:teuthology.orchestra.run.smithi067.stdout: "version": 4 2024-03-24T00:41:24.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:41:24.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:41:24.171 INFO:teuthology.orchestra.run.smithi067.stdout: "dirino": 1, 2024-03-24T00:41:24.171 INFO:teuthology.orchestra.run.smithi067.stdout: "dname": "parent_a", 2024-03-24T00:41:24.171 INFO:teuthology.orchestra.run.smithi067.stdout: "version": 7 2024-03-24T00:41:24.171 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:41:24.171 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-24T00:41:24.171 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 8, 2024-03-24T00:41:24.171 INFO:teuthology.orchestra.run.smithi067.stdout: "old_pools": [] 2024-03-24T00:41:24.171 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:41:24.171 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:41:24.173 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-24T00:41:24.173 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:41:24.254 DEBUG:teuthology.orchestra.run.smithi072:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:41:24.298 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-24T00:41:24.298 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:41:24.298 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:41:24.314 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:41:24.314 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:41:24.369 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:41:24.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.532+0000 7f5b6793b640 1 Processor -- start 2024-03-24T00:41:24.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.533+0000 7f5b6793b640 1 -- start start 2024-03-24T00:41:24.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.533+0000 7f5b6793b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5b6015e2a0 0x7f5b600ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:24.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.533+0000 7f5b6793b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b600cf640 con 0x7f5b600d0ef0 2024-03-24T00:41:24.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.533+0000 7f5b6793b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b600cf080 con 0x7f5b6015dec0 2024-03-24T00:41:24.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.533+0000 7f5b6793b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b600cee30 con 0x7f5b6015e2a0 2024-03-24T00:41:24.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.533+0000 7f5b64ed6640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5b6015e2a0 0x7f5b600ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:24.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.533+0000 7f5b65ed8640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5b600d0ef0 0x7f5b600d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33130/0 (socket says 172.21.15.67:33130) 2024-03-24T00:41:24.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.533+0000 7f5b65ed8640 1 -- 172.21.15.67:0/2556472769 learned_addr learned my addr 172.21.15.67:0/2556472769 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:41:24.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.534+0000 7f5b64ed6640 1 -- 172.21.15.67:0/2556472769 >> v1:172.21.15.72:6789/0 conn(0x7f5b6015dec0 legacy=0x7f5b600d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:24.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.534+0000 7f5b64ed6640 1 -- 172.21.15.67:0/2556472769 >> v1:172.21.15.67:6789/0 conn(0x7f5b600d0ef0 legacy=0x7f5b600d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:24.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.534+0000 7f5b64ed6640 1 -- 172.21.15.67:0/2556472769 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b600cf360 con 0x7f5b6015e2a0 2024-03-24T00:41:24.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.534+0000 7f5b5ffff640 1 -- 172.21.15.67:0/2556472769 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4106549878 0 0) 0x7f5b600cf080 con 0x7f5b6015dec0 2024-03-24T00:41:24.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.535+0000 7f5b64ed6640 1 --2- 172.21.15.67:0/2556472769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b6015e2a0 0x7f5b600ce630 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f5b600cefb0 tx=0x7f5b5402b6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ae9b6bce011e6e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:24.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.535+0000 7f5b5ffff640 1 -- 172.21.15.67:0/2556472769 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b54037070 con 0x7f5b6015e2a0 2024-03-24T00:41:24.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.535+0000 7f5b5ffff640 1 -- 172.21.15.67:0/2556472769 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b54030710 con 0x7f5b6015e2a0 2024-03-24T00:41:24.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.535+0000 7f5b5ffff640 1 -- 172.21.15.67:0/2556472769 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b54030ae0 con 0x7f5b6015e2a0 2024-03-24T00:41:24.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.536+0000 7f5b6793b640 1 -- 172.21.15.67:0/2556472769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b6015e2a0 msgr2=0x7f5b600ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:24.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.536+0000 7f5b6793b640 1 --2- 172.21.15.67:0/2556472769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b6015e2a0 0x7f5b600ce630 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f5b600cefb0 tx=0x7f5b5402b6f0 comp rx=0 tx=0).stop 2024-03-24T00:41:24.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.536+0000 7f5b6793b640 1 -- 172.21.15.67:0/2556472769 shutdown_connections 2024-03-24T00:41:24.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.536+0000 7f5b6793b640 1 --2- 172.21.15.67:0/2556472769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b6015e2a0 0x7f5b600ce630 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:24.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.536+0000 7f5b6793b640 1 -- 172.21.15.67:0/2556472769 >> 172.21.15.67:0/2556472769 conn(0x7f5b6006c810 msgr2=0x7f5b6006cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:24.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.536+0000 7f5b6793b640 1 -- 172.21.15.67:0/2556472769 shutdown_connections 2024-03-24T00:41:24.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.536+0000 7f5b6793b640 1 -- 172.21.15.67:0/2556472769 wait complete. 2024-03-24T00:41:24.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.537+0000 7f5b6793b640 1 Processor -- start 2024-03-24T00:41:24.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.537+0000 7f5b6793b640 1 -- start start 2024-03-24T00:41:24.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.538+0000 7f5b6793b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5b600d0ef0 0x7f5b6014c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:24.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.538+0000 7f5b6793b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b6015dec0 0x7f5b600ad2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:24.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.538+0000 7f5b6793b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b6015e2a0 0x7f5b600ad7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:24.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.538+0000 7f5b6793b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b60162a40 con 0x7f5b6015dec0 2024-03-24T00:41:24.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.538+0000 7f5b6793b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b601628c0 con 0x7f5b6015e2a0 2024-03-24T00:41:24.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.538+0000 7f5b6793b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b600d2120 con 0x7f5b600d0ef0 2024-03-24T00:41:24.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.538+0000 7f5b64ed6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b6015dec0 0x7f5b600ad2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:24.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.538+0000 7f5b64ed6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b6015dec0 0x7f5b600ad2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37364/0 (socket says 172.21.15.67:37364) 2024-03-24T00:41:24.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.538+0000 7f5b64ed6640 1 -- 172.21.15.67:0/3963818559 learned_addr learned my addr 172.21.15.67:0/3963818559 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:24.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.538+0000 7f5b656d7640 1 --2- 172.21.15.67:0/3963818559 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5b600d0ef0 0x7f5b6014c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:24.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.538+0000 7f5b65ed8640 1 --2- 172.21.15.67:0/3963818559 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b6015e2a0 0x7f5b600ad7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:24.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.538+0000 7f5b64ed6640 1 -- 172.21.15.67:0/3963818559 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5b600d0ef0 msgr2=0x7f5b6014c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:24.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.538+0000 7f5b64ed6640 1 --2- 172.21.15.67:0/3963818559 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5b600d0ef0 0x7f5b6014c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:24.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.538+0000 7f5b64ed6640 1 -- 172.21.15.67:0/3963818559 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b6015e2a0 msgr2=0x7f5b600ad7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:24.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.538+0000 7f5b64ed6640 1 --2- 172.21.15.67:0/3963818559 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b6015e2a0 0x7f5b600ad7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:24.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.538+0000 7f5b64ed6640 1 -- 172.21.15.67:0/3963818559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b600ccec0 con 0x7f5b6015dec0 2024-03-24T00:41:24.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.538+0000 7f5b64ed6640 1 --2- 172.21.15.67:0/3963818559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b6015dec0 0x7f5b600ad2a0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f5b54009890 tx=0x7f5b540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:24.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.538+0000 7f5b5dffb640 1 -- 172.21.15.67:0/3963818559 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b54042070 con 0x7f5b6015dec0 2024-03-24T00:41:24.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.538+0000 7f5b5dffb640 1 -- 172.21.15.67:0/3963818559 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b54032a40 con 0x7f5b6015dec0 2024-03-24T00:41:24.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.538+0000 7f5b6793b640 1 -- 172.21.15.67:0/3963818559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b6014cad0 con 0x7f5b6015dec0 2024-03-24T00:41:24.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.539+0000 7f5b5dffb640 1 -- 172.21.15.67:0/3963818559 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b54037040 con 0x7f5b6015dec0 2024-03-24T00:41:24.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.539+0000 7f5b6793b640 1 -- 172.21.15.67:0/3963818559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b600cc190 con 0x7f5b6015dec0 2024-03-24T00:41:24.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.540+0000 7f5b6793b640 1 -- 172.21.15.67:0/3963818559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b600cd1a0 con 0x7f5b6015dec0 2024-03-24T00:41:24.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.540+0000 7f5b5dffb640 1 -- 172.21.15.67:0/3963818559 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5b54045070 con 0x7f5b6015dec0 2024-03-24T00:41:24.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.540+0000 7f5b5dffb640 1 --2- 172.21.15.67:0/3963818559 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5b4c06bc00 0x7f5b4c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:24.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.541+0000 7f5b5dffb640 1 -- 172.21.15.67:0/3963818559 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5b540ac9e0 con 0x7f5b6015dec0 2024-03-24T00:41:24.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.541+0000 7f5b656d7640 1 --2- 172.21.15.67:0/3963818559 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5b4c06bc00 0x7f5b4c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:24.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.541+0000 7f5b656d7640 1 --2- 172.21.15.67:0/3963818559 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5b4c06bc00 0x7f5b4c06e0c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f5b500012e0 tx=0x7f5b50009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:24.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.545+0000 7f5b5dffb640 1 -- 172.21.15.67:0/3963818559 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5b5407ae10 con 0x7f5b6015dec0 2024-03-24T00:41:24.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.727+0000 7f5b6793b640 1 -- 172.21.15.67:0/3963818559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b6004f400 con 0x7f5b6015dec0 2024-03-24T00:41:24.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.728+0000 7f5b5dffb640 1 -- 172.21.15.67:0/3963818559 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) v1 ==== 74+0+16882 (secure 0 0 0) 0x7f5b5407eeb0 con 0x7f5b6015dec0 2024-03-24T00:41:24.729 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:41:24.729 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":50,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:40:41.288503+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:40:37.927587+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-24T00:40:38.616521+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:41:24.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.730+0000 7f5b6793b640 1 -- 172.21.15.67:0/3963818559 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5b4c06bc00 msgr2=0x7f5b4c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:24.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.730+0000 7f5b6793b640 1 --2- 172.21.15.67:0/3963818559 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5b4c06bc00 0x7f5b4c06e0c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f5b500012e0 tx=0x7f5b50009040 comp rx=0 tx=0).stop 2024-03-24T00:41:24.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.730+0000 7f5b6793b640 1 -- 172.21.15.67:0/3963818559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b6015dec0 msgr2=0x7f5b600ad2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:24.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.730+0000 7f5b6793b640 1 --2- 172.21.15.67:0/3963818559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b6015dec0 0x7f5b600ad2a0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f5b54009890 tx=0x7f5b540046c0 comp rx=0 tx=0).stop 2024-03-24T00:41:24.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.730+0000 7f5b6793b640 1 -- 172.21.15.67:0/3963818559 shutdown_connections 2024-03-24T00:41:24.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.730+0000 7f5b6793b640 1 --2- 172.21.15.67:0/3963818559 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5b4c06bc00 0x7f5b4c06e0c0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:24.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.730+0000 7f5b6793b640 1 --2- 172.21.15.67:0/3963818559 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b6015e2a0 0x7f5b600ad7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:24.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.730+0000 7f5b6793b640 1 --2- 172.21.15.67:0/3963818559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b6015dec0 0x7f5b600ad2a0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:24.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.730+0000 7f5b6793b640 1 --2- 172.21.15.67:0/3963818559 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5b600d0ef0 0x7f5b6014c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:24.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.730+0000 7f5b6793b640 1 -- 172.21.15.67:0/3963818559 >> 172.21.15.67:0/3963818559 conn(0x7f5b6006c810 msgr2=0x7f5b6015eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:24.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.730+0000 7f5b6793b640 1 -- 172.21.15.67:0/3963818559 shutdown_connections 2024-03-24T00:41:24.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.730+0000 7f5b6793b640 1 -- 172.21.15.67:0/3963818559 wait complete. 2024-03-24T00:41:24.742 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:38:39.364950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:40:37.927587+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '49', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:40:38.616521+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '50', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:41:24.742 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:41:24.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.845+0000 7f536cfb9640 1 Processor -- start 2024-03-24T00:41:24.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.845+0000 7f536cfb9640 1 -- start start 2024-03-24T00:41:24.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.846+0000 7f536cfb9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f53680d3ae0 0x7f53680b5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:24.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.846+0000 7f536cfb9640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53680d2250 con 0x7f536815dea0 2024-03-24T00:41:24.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.846+0000 7f536cfb9640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53680d2530 con 0x7f536815e280 2024-03-24T00:41:24.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.846+0000 7f536cfb9640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53680d2810 con 0x7f53680d3ae0 2024-03-24T00:41:24.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.846+0000 7f5365d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f53680d3ae0 0x7f53680b5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:24.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.846+0000 7f536659c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f536815dea0 0x7f53680d33d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33136/0 (socket says 172.21.15.67:33136) 2024-03-24T00:41:24.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.846+0000 7f536659c640 1 -- 172.21.15.67:0/2555529981 learned_addr learned my addr 172.21.15.67:0/2555529981 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:41:24.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.847+0000 7f5365d9b640 1 -- 172.21.15.67:0/2555529981 >> v1:172.21.15.72:6789/0 conn(0x7f536815e280 legacy=0x7f53680b70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:24.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.847+0000 7f5365d9b640 1 -- 172.21.15.67:0/2555529981 >> v1:172.21.15.67:6789/0 conn(0x7f536815dea0 legacy=0x7f53680d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:24.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.847+0000 7f5365d9b640 1 -- 172.21.15.67:0/2555529981 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53680d2af0 con 0x7f53680d3ae0 2024-03-24T00:41:24.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.847+0000 7f536559a640 1 -- 172.21.15.67:0/2555529981 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4251758741 0 0) 0x7f53680d2250 con 0x7f536815dea0 2024-03-24T00:41:24.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.847+0000 7f5365d9b640 1 --2- 172.21.15.67:0/2555529981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53680d3ae0 0x7f53680b5a60 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f535c0097d0 tx=0x7f535c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7af8f50a715cfaeb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:24.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.848+0000 7f536559a640 1 -- 172.21.15.67:0/2555529981 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f535c035020 con 0x7f53680d3ae0 2024-03-24T00:41:24.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.848+0000 7f536559a640 1 -- 172.21.15.67:0/2555529981 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f535c031830 con 0x7f53680d3ae0 2024-03-24T00:41:24.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.848+0000 7f536559a640 1 -- 172.21.15.67:0/2555529981 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f535c031c00 con 0x7f53680d3ae0 2024-03-24T00:41:24.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.848+0000 7f536cfb9640 1 -- 172.21.15.67:0/2555529981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53680d3ae0 msgr2=0x7f53680b5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:24.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.848+0000 7f536cfb9640 1 --2- 172.21.15.67:0/2555529981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53680d3ae0 0x7f53680b5a60 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f535c0097d0 tx=0x7f535c02b8d0 comp rx=0 tx=0).stop 2024-03-24T00:41:24.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.848+0000 7f536cfb9640 1 -- 172.21.15.67:0/2555529981 shutdown_connections 2024-03-24T00:41:24.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.848+0000 7f536cfb9640 1 --2- 172.21.15.67:0/2555529981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53680d3ae0 0x7f53680b5a60 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:24.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.848+0000 7f536cfb9640 1 -- 172.21.15.67:0/2555529981 >> 172.21.15.67:0/2555529981 conn(0x7f5368072da0 msgr2=0x7f53680d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:24.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.848+0000 7f536cfb9640 1 -- 172.21.15.67:0/2555529981 shutdown_connections 2024-03-24T00:41:24.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.848+0000 7f536cfb9640 1 -- 172.21.15.67:0/2555529981 wait complete. 2024-03-24T00:41:24.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.850+0000 7f536cfb9640 1 Processor -- start 2024-03-24T00:41:24.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.850+0000 7f536cfb9640 1 -- start start 2024-03-24T00:41:24.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.852+0000 7f536cfb9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53680d3ae0 0x7f536815b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:24.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.852+0000 7f536cfb9640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f536815dea0 0x7f53680b8e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:24.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.852+0000 7f536cfb9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f536815e280 0x7f53681e8580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:24.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.852+0000 7f536cfb9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53680b6370 con 0x7f536815e280 2024-03-24T00:41:24.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.852+0000 7f536cfb9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53680b61f0 con 0x7f53680d3ae0 2024-03-24T00:41:24.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.852+0000 7f536cfb9640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53680b7800 con 0x7f536815dea0 2024-03-24T00:41:24.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.852+0000 7f5366d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f536815e280 0x7f53681e8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:24.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.852+0000 7f5366d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f536815e280 0x7f53681e8580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37384/0 (socket says 172.21.15.67:37384) 2024-03-24T00:41:24.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.852+0000 7f5366d9d640 1 -- 172.21.15.67:0/346521182 learned_addr learned my addr 172.21.15.67:0/346521182 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:24.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.852+0000 7f536659c640 1 --2- 172.21.15.67:0/346521182 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53680d3ae0 0x7f536815b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:24.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.852+0000 7f5365d9b640 1 --2- 172.21.15.67:0/346521182 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f536815dea0 0x7f53680b8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:24.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.852+0000 7f5366d9d640 1 -- 172.21.15.67:0/346521182 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f536815dea0 msgr2=0x7f53680b8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:24.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.852+0000 7f5366d9d640 1 --2- 172.21.15.67:0/346521182 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f536815dea0 0x7f53680b8e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:24.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.852+0000 7f5366d9d640 1 -- 172.21.15.67:0/346521182 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53680d3ae0 msgr2=0x7f536815b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:24.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.852+0000 7f5366d9d640 1 --2- 172.21.15.67:0/346521182 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53680d3ae0 0x7f536815b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:24.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.852+0000 7f5366d9d640 1 -- 172.21.15.67:0/346521182 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53680ccdf0 con 0x7f536815e280 2024-03-24T00:41:24.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.852+0000 7f5366d9d640 1 --2- 172.21.15.67:0/346521182 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f536815e280 0x7f53681e8580 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f536000c390 tx=0x7f536000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:24.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.852+0000 7f5356ffd640 1 -- 172.21.15.67:0/346521182 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5360017070 con 0x7f536815e280 2024-03-24T00:41:24.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.852+0000 7f5356ffd640 1 -- 172.21.15.67:0/346521182 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53600052d0 con 0x7f536815e280 2024-03-24T00:41:24.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.852+0000 7f536cfb9640 1 -- 172.21.15.67:0/346521182 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53680cd0d0 con 0x7f536815e280 2024-03-24T00:41:24.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.852+0000 7f5356ffd640 1 -- 172.21.15.67:0/346521182 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53600037f0 con 0x7f536815e280 2024-03-24T00:41:24.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.853+0000 7f536cfb9640 1 -- 172.21.15.67:0/346521182 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53680cd690 con 0x7f536815e280 2024-03-24T00:41:24.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.853+0000 7f536cfb9640 1 -- 172.21.15.67:0/346521182 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53680cd890 con 0x7f536815e280 2024-03-24T00:41:24.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.854+0000 7f5356ffd640 1 -- 172.21.15.67:0/346521182 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f536001a070 con 0x7f536815e280 2024-03-24T00:41:24.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.854+0000 7f5356ffd640 1 --2- 172.21.15.67:0/346521182 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f534806bbd0 0x7f534806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:24.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.854+0000 7f5356ffd640 1 -- 172.21.15.67:0/346521182 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5360089cd0 con 0x7f536815e280 2024-03-24T00:41:24.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.855+0000 7f536659c640 1 --2- 172.21.15.67:0/346521182 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f534806bbd0 0x7f534806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:24.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.855+0000 7f536659c640 1 --2- 172.21.15.67:0/346521182 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f534806bbd0 0x7f534806e090 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f5358005fd0 tx=0x7f5358007f10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:24.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:24.859+0000 7f5356ffd640 1 -- 172.21.15.67:0/346521182 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f53680cd890 con 0x7f536815e280 2024-03-24T00:41:25.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.039+0000 7f536cfb9640 1 -- 172.21.15.67:0/346521182 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f53680ca510 con 0x7f536815e280 2024-03-24T00:41:25.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.039+0000 7f5356ffd640 1 -- 172.21.15.67:0/346521182 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v50) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5360057fe0 con 0x7f536815e280 2024-03-24T00:41:25.040 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-24T00:41:25.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.041+0000 7f536cfb9640 1 -- 172.21.15.67:0/346521182 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f534806bbd0 msgr2=0x7f534806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:25.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.041+0000 7f536cfb9640 1 --2- 172.21.15.67:0/346521182 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f534806bbd0 0x7f534806e090 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f5358005fd0 tx=0x7f5358007f10 comp rx=0 tx=0).stop 2024-03-24T00:41:25.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.041+0000 7f536cfb9640 1 -- 172.21.15.67:0/346521182 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f536815e280 msgr2=0x7f53681e8580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:25.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.041+0000 7f536cfb9640 1 --2- 172.21.15.67:0/346521182 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f536815e280 0x7f53681e8580 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f536000c390 tx=0x7f536000c850 comp rx=0 tx=0).stop 2024-03-24T00:41:25.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.041+0000 7f536cfb9640 1 -- 172.21.15.67:0/346521182 shutdown_connections 2024-03-24T00:41:25.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.041+0000 7f536cfb9640 1 --2- 172.21.15.67:0/346521182 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f534806bbd0 0x7f534806e090 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:25.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.041+0000 7f536cfb9640 1 --2- 172.21.15.67:0/346521182 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f536815e280 0x7f53681e8580 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:25.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.041+0000 7f536cfb9640 1 --2- 172.21.15.67:0/346521182 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f536815dea0 0x7f53680b8e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:25.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.041+0000 7f536cfb9640 1 --2- 172.21.15.67:0/346521182 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53680d3ae0 0x7f536815b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:25.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.041+0000 7f536cfb9640 1 -- 172.21.15.67:0/346521182 >> 172.21.15.67:0/346521182 conn(0x7f5368072da0 msgr2=0x7f53680d3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:25.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.041+0000 7f536cfb9640 1 -- 172.21.15.67:0/346521182 shutdown_connections 2024-03-24T00:41:25.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.041+0000 7f536cfb9640 1 -- 172.21.15.67:0/346521182 wait complete. 2024-03-24T00:41:25.056 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:41:25.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.160+0000 7fe0cf0e8640 1 Processor -- start 2024-03-24T00:41:25.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.160+0000 7fe0cf0e8640 1 -- start start 2024-03-24T00:41:25.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.161+0000 7fe0cf0e8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe0c815df00 0x7fe0c815ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:25.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.161+0000 7fe0cf0e8640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe0c80cead0 0x7fe0c806c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:25.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.161+0000 7fe0cf0e8640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0c815e2e0 con 0x7fe0c815e580 2024-03-24T00:41:25.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.161+0000 7fe0cf0e8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0c80b1880 con 0x7fe0c815df00 2024-03-24T00:41:25.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.161+0000 7fe0cf0e8640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0c80b1b60 con 0x7fe0c80cead0 2024-03-24T00:41:25.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.162+0000 7fe0cce84640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe0c815df00 0x7fe0c815ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:25.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.162+0000 7fe0cce84640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe0c815df00 0x7fe0c815ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37392/0 (socket says 172.21.15.67:37392) 2024-03-24T00:41:25.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.162+0000 7fe0cce84640 1 -- 172.21.15.67:0/44479072 learned_addr learned my addr 172.21.15.67:0/44479072 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:25.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.162+0000 7fe0cd685640 1 --2- 172.21.15.67:0/44479072 >> v2:172.21.15.72:3300/0 conn(0x7fe0c80cead0 0x7fe0c806c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:25.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.162+0000 7fe0cce84640 1 -- 172.21.15.67:0/44479072 >> v2:172.21.15.72:3300/0 conn(0x7fe0c80cead0 msgr2=0x7fe0c806c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:25.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.162+0000 7fe0cce84640 1 --2- 172.21.15.67:0/44479072 >> v2:172.21.15.72:3300/0 conn(0x7fe0c80cead0 0x7fe0c806c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:25.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.162+0000 7fe0cce84640 1 -- 172.21.15.67:0/44479072 >> v1:172.21.15.72:6789/0 conn(0x7fe0c815e580 legacy=0x7fe0c80ce3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:25.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.162+0000 7fe0cce84640 1 -- 172.21.15.67:0/44479072 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c80b1e40 con 0x7fe0c815df00 2024-03-24T00:41:25.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.162+0000 7fe0cd685640 1 --2- 172.21.15.67:0/44479072 >> v2:172.21.15.72:3300/0 conn(0x7fe0c80cead0 0x7fe0c806c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:41:25.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.163+0000 7fe0cce84640 1 --2- 172.21.15.67:0/44479072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0c815df00 0x7fe0c815ed70 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fe0c80b16c0 tx=0x7fe0bc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12ecf7fa3b6532fa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:25.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.163+0000 7fe0c77fe640 1 -- 172.21.15.67:0/44479072 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0bc02e070 con 0x7fe0c815df00 2024-03-24T00:41:25.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.163+0000 7fe0c77fe640 1 -- 172.21.15.67:0/44479072 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0bc037070 con 0x7fe0c815df00 2024-03-24T00:41:25.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.163+0000 7fe0c77fe640 1 -- 172.21.15.67:0/44479072 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0bc031a00 con 0x7fe0c815df00 2024-03-24T00:41:25.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.163+0000 7fe0cf0e8640 1 -- 172.21.15.67:0/44479072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0c815df00 msgr2=0x7fe0c815ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:25.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.163+0000 7fe0cf0e8640 1 --2- 172.21.15.67:0/44479072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0c815df00 0x7fe0c815ed70 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fe0c80b16c0 tx=0x7fe0bc02b880 comp rx=0 tx=0).stop 2024-03-24T00:41:25.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.163+0000 7fe0cf0e8640 1 -- 172.21.15.67:0/44479072 shutdown_connections 2024-03-24T00:41:25.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.163+0000 7fe0cf0e8640 1 --2- 172.21.15.67:0/44479072 >> v2:172.21.15.72:3300/0 conn(0x7fe0c80cead0 0x7fe0c806c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:25.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.163+0000 7fe0cf0e8640 1 --2- 172.21.15.67:0/44479072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0c815df00 0x7fe0c815ed70 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:25.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.163+0000 7fe0cf0e8640 1 -- 172.21.15.67:0/44479072 >> 172.21.15.67:0/44479072 conn(0x7fe0c808b180 msgr2=0x7fe0c808b590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:25.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.163+0000 7fe0cf0e8640 1 -- 172.21.15.67:0/44479072 shutdown_connections 2024-03-24T00:41:25.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.164+0000 7fe0cf0e8640 1 -- 172.21.15.67:0/44479072 wait complete. 2024-03-24T00:41:25.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.165+0000 7fe0cf0e8640 1 Processor -- start 2024-03-24T00:41:25.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.165+0000 7fe0cf0e8640 1 -- start start 2024-03-24T00:41:25.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.165+0000 7fe0cf0e8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0c80cead0 0x7fe0c80b01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:25.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.165+0000 7fe0cf0e8640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe0c815df00 0x7fe0c815bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:25.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.165+0000 7fe0cf0e8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0c815e580 0x7fe0c80ae690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:25.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.165+0000 7fe0cf0e8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c815f2b0 con 0x7fe0c80cead0 2024-03-24T00:41:25.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.165+0000 7fe0cf0e8640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c80b2400 con 0x7fe0c815e580 2024-03-24T00:41:25.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.165+0000 7fe0cf0e8640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c815f430 con 0x7fe0c815df00 2024-03-24T00:41:25.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.166+0000 7fe0cce84640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0c80cead0 0x7fe0c80b01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:25.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.166+0000 7fe0cce84640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0c80cead0 0x7fe0c80b01d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37408/0 (socket says 172.21.15.67:37408) 2024-03-24T00:41:25.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.166+0000 7fe0cce84640 1 -- 172.21.15.67:0/1613768405 learned_addr learned my addr 172.21.15.67:0/1613768405 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:25.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.166+0000 7fe0cd685640 1 --2- 172.21.15.67:0/1613768405 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0c815e580 0x7fe0c80ae690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:25.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.166+0000 7fe0c7fff640 1 --2- 172.21.15.67:0/1613768405 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe0c815df00 0x7fe0c815bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:25.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.166+0000 7fe0cce84640 1 -- 172.21.15.67:0/1613768405 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe0c815df00 msgr2=0x7fe0c815bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:25.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.166+0000 7fe0cce84640 1 --2- 172.21.15.67:0/1613768405 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe0c815df00 0x7fe0c815bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:25.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.166+0000 7fe0cce84640 1 -- 172.21.15.67:0/1613768405 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0c815e580 msgr2=0x7fe0c80ae690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:25.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.166+0000 7fe0cce84640 1 --2- 172.21.15.67:0/1613768405 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0c815e580 0x7fe0c80ae690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:25.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.166+0000 7fe0cce84640 1 -- 172.21.15.67:0/1613768405 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c80af130 con 0x7fe0c80cead0 2024-03-24T00:41:25.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.166+0000 7fe0cce84640 1 --2- 172.21.15.67:0/1613768405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0c80cead0 0x7fe0c80b01d0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc000c00 tx=0x7fe0bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:25.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.166+0000 7fe0c57fa640 1 -- 172.21.15.67:0/1613768405 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0bc032880 con 0x7fe0c80cead0 2024-03-24T00:41:25.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.166+0000 7fe0c57fa640 1 -- 172.21.15.67:0/1613768405 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0bc032a20 con 0x7fe0c80cead0 2024-03-24T00:41:25.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.166+0000 7fe0cf0e8640 1 -- 172.21.15.67:0/1613768405 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0c80af410 con 0x7fe0c80cead0 2024-03-24T00:41:25.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.166+0000 7fe0c57fa640 1 -- 172.21.15.67:0/1613768405 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0bc032e10 con 0x7fe0c80cead0 2024-03-24T00:41:25.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.166+0000 7fe0cf0e8640 1 -- 172.21.15.67:0/1613768405 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0c80af9d0 con 0x7fe0c80cead0 2024-03-24T00:41:25.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.167+0000 7fe0cf0e8640 1 -- 172.21.15.67:0/1613768405 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0c804f400 con 0x7fe0c80cead0 2024-03-24T00:41:25.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.168+0000 7fe0c57fa640 1 -- 172.21.15.67:0/1613768405 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe0bc04c060 con 0x7fe0c80cead0 2024-03-24T00:41:25.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.168+0000 7fe0c57fa640 1 --2- 172.21.15.67:0/1613768405 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe0ac06bbd0 0x7fe0ac06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:25.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.168+0000 7fe0c57fa640 1 -- 172.21.15.67:0/1613768405 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe0bc0ac4b0 con 0x7fe0c80cead0 2024-03-24T00:41:25.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.168+0000 7fe0c7fff640 1 --2- 172.21.15.67:0/1613768405 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe0ac06bbd0 0x7fe0ac06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:25.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.169+0000 7fe0c7fff640 1 --2- 172.21.15.67:0/1613768405 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe0ac06bbd0 0x7fe0ac06e090 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fe0b4006fd0 tx=0x7fe0b4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:25.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.177+0000 7fe0c57fa640 1 -- 172.21.15.67:0/1613768405 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe0bc040070 con 0x7fe0c80cead0 2024-03-24T00:41:25.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.482+0000 7fe0cf0e8640 1 -- 172.21.15.67:0/1613768405 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe0c806eb90 con 0x7fe0c80cead0 2024-03-24T00:41:25.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.482+0000 7fe0c57fa640 1 -- 172.21.15.67:0/1613768405 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v50) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe0bc07ae10 con 0x7fe0c80cead0 2024-03-24T00:41:25.483 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-24T00:41:25.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.485+0000 7fe0cf0e8640 1 -- 172.21.15.67:0/1613768405 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe0ac06bbd0 msgr2=0x7fe0ac06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:25.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.485+0000 7fe0cf0e8640 1 --2- 172.21.15.67:0/1613768405 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe0ac06bbd0 0x7fe0ac06e090 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fe0b4006fd0 tx=0x7fe0b4008040 comp rx=0 tx=0).stop 2024-03-24T00:41:25.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.485+0000 7fe0cf0e8640 1 -- 172.21.15.67:0/1613768405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0c80cead0 msgr2=0x7fe0c80b01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:25.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.485+0000 7fe0cf0e8640 1 --2- 172.21.15.67:0/1613768405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0c80cead0 0x7fe0c80b01d0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc000c00 tx=0x7fe0bc0046c0 comp rx=0 tx=0).stop 2024-03-24T00:41:25.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.485+0000 7fe0cf0e8640 1 -- 172.21.15.67:0/1613768405 shutdown_connections 2024-03-24T00:41:25.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.485+0000 7fe0cf0e8640 1 --2- 172.21.15.67:0/1613768405 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe0ac06bbd0 0x7fe0ac06e090 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:25.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.485+0000 7fe0cf0e8640 1 --2- 172.21.15.67:0/1613768405 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0c815e580 0x7fe0c80ae690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:25.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.485+0000 7fe0cf0e8640 1 --2- 172.21.15.67:0/1613768405 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe0c815df00 0x7fe0c815bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:25.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.485+0000 7fe0cf0e8640 1 --2- 172.21.15.67:0/1613768405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0c80cead0 0x7fe0c80b01d0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:25.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.485+0000 7fe0cf0e8640 1 -- 172.21.15.67:0/1613768405 >> 172.21.15.67:0/1613768405 conn(0x7fe0c808b180 msgr2=0x7fe0c80d0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:25.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.485+0000 7fe0cf0e8640 1 -- 172.21.15.67:0/1613768405 shutdown_connections 2024-03-24T00:41:25.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.485+0000 7fe0cf0e8640 1 -- 172.21.15.67:0/1613768405 wait complete. 2024-03-24T00:41:25.500 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:41:25.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.603+0000 7f5b8c052640 1 Processor -- start 2024-03-24T00:41:25.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.604+0000 7f5b8c052640 1 -- start start 2024-03-24T00:41:25.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.604+0000 7f5b8c052640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5b8415bc30 0x7f5b840b1a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:25.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.604+0000 7f5b8c052640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b840b3ae0 con 0x7f5b8415c010 2024-03-24T00:41:25.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.604+0000 7f5b8c052640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b840b3dc0 con 0x7f5b840b34a0 2024-03-24T00:41:25.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.604+0000 7f5b8c052640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b840cca60 con 0x7f5b8415bc30 2024-03-24T00:41:25.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.605+0000 7f5b89dee640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5b8415bc30 0x7f5b840b1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:25.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.605+0000 7f5b895ed640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5b8415c010 0x7f5b840b2d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33142/0 (socket says 172.21.15.67:33142) 2024-03-24T00:41:25.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.605+0000 7f5b895ed640 1 -- 172.21.15.67:0/4160342457 learned_addr learned my addr 172.21.15.67:0/4160342457 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:41:25.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.605+0000 7f5b89dee640 1 -- 172.21.15.67:0/4160342457 >> v1:172.21.15.72:6789/0 conn(0x7f5b840b34a0 legacy=0x7f5b840ccef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:25.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.605+0000 7f5b89dee640 1 -- 172.21.15.67:0/4160342457 >> v1:172.21.15.67:6789/0 conn(0x7f5b8415c010 legacy=0x7f5b840b2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:25.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.605+0000 7f5b88dec640 1 -- 172.21.15.67:0/4160342457 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3365993696 0 0) 0x7f5b840b3ae0 con 0x7f5b8415c010 2024-03-24T00:41:25.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.605+0000 7f5b89dee640 1 -- 172.21.15.67:0/4160342457 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b840b40a0 con 0x7f5b8415bc30 2024-03-24T00:41:25.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.606+0000 7f5b89dee640 1 --2- 172.21.15.67:0/4160342457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b8415bc30 0x7f5b840b1a00 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f5b800097a0 tx=0x7f5b8002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5478e8bc4a0d193c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:25.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.606+0000 7f5b88dec640 1 -- 172.21.15.67:0/4160342457 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b80034070 con 0x7f5b8415bc30 2024-03-24T00:41:25.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.606+0000 7f5b88dec640 1 -- 172.21.15.67:0/4160342457 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b80030610 con 0x7f5b8415bc30 2024-03-24T00:41:25.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.606+0000 7f5b88dec640 1 -- 172.21.15.67:0/4160342457 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b800309e0 con 0x7f5b8415bc30 2024-03-24T00:41:25.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.606+0000 7f5b8c052640 1 -- 172.21.15.67:0/4160342457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b8415bc30 msgr2=0x7f5b840b1a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:25.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.606+0000 7f5b8c052640 1 --2- 172.21.15.67:0/4160342457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b8415bc30 0x7f5b840b1a00 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f5b800097a0 tx=0x7f5b8002b960 comp rx=0 tx=0).stop 2024-03-24T00:41:25.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.606+0000 7f5b8c052640 1 -- 172.21.15.67:0/4160342457 shutdown_connections 2024-03-24T00:41:25.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.606+0000 7f5b8c052640 1 --2- 172.21.15.67:0/4160342457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b8415bc30 0x7f5b840b1a00 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:25.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.606+0000 7f5b8c052640 1 -- 172.21.15.67:0/4160342457 >> 172.21.15.67:0/4160342457 conn(0x7f5b8408b180 msgr2=0x7f5b8408b590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:25.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.607+0000 7f5b8c052640 1 -- 172.21.15.67:0/4160342457 shutdown_connections 2024-03-24T00:41:25.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.607+0000 7f5b8c052640 1 -- 172.21.15.67:0/4160342457 wait complete. 2024-03-24T00:41:25.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.608+0000 7f5b8c052640 1 Processor -- start 2024-03-24T00:41:25.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.608+0000 7f5b8c052640 1 -- start start 2024-03-24T00:41:25.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.608+0000 7f5b8c052640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b840b34a0 0x7f5b84142b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:25.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.608+0000 7f5b8c052640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b8415bc30 0x7f5b841430b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:25.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.608+0000 7f5b8c052640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5b8415c010 0x7f5b84144ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:25.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.608+0000 7f5b8c052640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b840b20c0 con 0x7f5b8415bc30 2024-03-24T00:41:25.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.608+0000 7f5b8c052640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b840b1f40 con 0x7f5b840b34a0 2024-03-24T00:41:25.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.608+0000 7f5b8c052640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b840b2240 con 0x7f5b8415c010 2024-03-24T00:41:25.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.608+0000 7f5b895ed640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b8415bc30 0x7f5b841430b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:25.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.608+0000 7f5b895ed640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b8415bc30 0x7f5b841430b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37428/0 (socket says 172.21.15.67:37428) 2024-03-24T00:41:25.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.608+0000 7f5b895ed640 1 -- 172.21.15.67:0/4093414598 learned_addr learned my addr 172.21.15.67:0/4093414598 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:25.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.609+0000 7f5b89dee640 1 --2- 172.21.15.67:0/4093414598 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b840b34a0 0x7f5b84142b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:25.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.609+0000 7f5b8a5ef640 1 --2- 172.21.15.67:0/4093414598 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5b8415c010 0x7f5b84144ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:25.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.609+0000 7f5b895ed640 1 -- 172.21.15.67:0/4093414598 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5b8415c010 msgr2=0x7f5b84144ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:25.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.609+0000 7f5b895ed640 1 --2- 172.21.15.67:0/4093414598 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5b8415c010 0x7f5b84144ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:25.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.609+0000 7f5b895ed640 1 -- 172.21.15.67:0/4093414598 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b840b34a0 msgr2=0x7f5b84142b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:25.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.609+0000 7f5b895ed640 1 --2- 172.21.15.67:0/4093414598 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b840b34a0 0x7f5b84142b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:25.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.609+0000 7f5b895ed640 1 -- 172.21.15.67:0/4093414598 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b840b1390 con 0x7f5b8415bc30 2024-03-24T00:41:25.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.609+0000 7f5b8a5ef640 1 --2- 172.21.15.67:0/4093414598 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5b8415c010 0x7f5b84144ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:41:25.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.609+0000 7f5b895ed640 1 --2- 172.21.15.67:0/4093414598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b8415bc30 0x7f5b841430b0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f5b7400c3f0 tx=0x7f5b7400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:25.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.609+0000 7f5b7a7fc640 1 -- 172.21.15.67:0/4093414598 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b740074e0 con 0x7f5b8415bc30 2024-03-24T00:41:25.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.609+0000 7f5b7a7fc640 1 -- 172.21.15.67:0/4093414598 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b740047e0 con 0x7f5b8415bc30 2024-03-24T00:41:25.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.609+0000 7f5b8c052640 1 -- 172.21.15.67:0/4093414598 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b840b6360 con 0x7f5b8415bc30 2024-03-24T00:41:25.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.609+0000 7f5b7a7fc640 1 -- 172.21.15.67:0/4093414598 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b74004c10 con 0x7f5b8415bc30 2024-03-24T00:41:25.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.609+0000 7f5b8c052640 1 -- 172.21.15.67:0/4093414598 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b840b6920 con 0x7f5b8415bc30 2024-03-24T00:41:25.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.610+0000 7f5b8c052640 1 -- 172.21.15.67:0/4093414598 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b8404f400 con 0x7f5b8415bc30 2024-03-24T00:41:25.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.611+0000 7f5b7a7fc640 1 -- 172.21.15.67:0/4093414598 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5b74011070 con 0x7f5b8415bc30 2024-03-24T00:41:25.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.611+0000 7f5b7a7fc640 1 --2- 172.21.15.67:0/4093414598 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5b6806bbd0 0x7f5b6806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:25.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.612+0000 7f5b7a7fc640 1 -- 172.21.15.67:0/4093414598 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5b7408a710 con 0x7f5b8415bc30 2024-03-24T00:41:25.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.612+0000 7f5b89dee640 1 --2- 172.21.15.67:0/4093414598 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5b6806bbd0 0x7f5b6806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:25.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.612+0000 7f5b89dee640 1 --2- 172.21.15.67:0/4093414598 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5b6806bbd0 0x7f5b6806e090 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f5b80036000 tx=0x7f5b80008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:25.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.616+0000 7f5b7a7fc640 1 -- 172.21.15.67:0/4093414598 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5b74059070 con 0x7f5b8415bc30 2024-03-24T00:41:25.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.892+0000 7f5b8c052640 1 -- 172.21.15.67:0/4093414598 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5b840ceed0 con 0x7f5b8415bc30 2024-03-24T00:41:25.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.892+0000 7f5b7a7fc640 1 -- 172.21.15.67:0/4093414598 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v50) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5b74058970 con 0x7f5b8415bc30 2024-03-24T00:41:25.893 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-03-24T00:41:25.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.894+0000 7f5b8c052640 1 -- 172.21.15.67:0/4093414598 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5b6806bbd0 msgr2=0x7f5b6806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:25.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.894+0000 7f5b8c052640 1 --2- 172.21.15.67:0/4093414598 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5b6806bbd0 0x7f5b6806e090 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f5b80036000 tx=0x7f5b80008030 comp rx=0 tx=0).stop 2024-03-24T00:41:25.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.894+0000 7f5b8c052640 1 -- 172.21.15.67:0/4093414598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b8415bc30 msgr2=0x7f5b841430b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:25.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.894+0000 7f5b8c052640 1 --2- 172.21.15.67:0/4093414598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b8415bc30 0x7f5b841430b0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f5b7400c3f0 tx=0x7f5b7400c8b0 comp rx=0 tx=0).stop 2024-03-24T00:41:25.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.894+0000 7f5b8c052640 1 -- 172.21.15.67:0/4093414598 shutdown_connections 2024-03-24T00:41:25.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.894+0000 7f5b8c052640 1 --2- 172.21.15.67:0/4093414598 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5b6806bbd0 0x7f5b6806e090 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:25.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.894+0000 7f5b8c052640 1 --2- 172.21.15.67:0/4093414598 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5b8415c010 0x7f5b84144ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:25.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.894+0000 7f5b8c052640 1 --2- 172.21.15.67:0/4093414598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b8415bc30 0x7f5b841430b0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:25.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.894+0000 7f5b8c052640 1 --2- 172.21.15.67:0/4093414598 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b840b34a0 0x7f5b84142b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:25.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.894+0000 7f5b8c052640 1 -- 172.21.15.67:0/4093414598 >> 172.21.15.67:0/4093414598 conn(0x7f5b8408b180 msgr2=0x7f5b840cd260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:25.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.894+0000 7f5b8c052640 1 -- 172.21.15.67:0/4093414598 shutdown_connections 2024-03-24T00:41:25.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:25.894+0000 7f5b8c052640 1 -- 172.21.15.67:0/4093414598 wait complete. 2024-03-24T00:41:25.906 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T00:41:26.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.009+0000 7f8a5fef8640 1 Processor -- start 2024-03-24T00:41:26.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.009+0000 7f8a5fef8640 1 -- start start 2024-03-24T00:41:26.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.010+0000 7f8a5fef8640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8a580d32d0 0x7f8a580d3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:26.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.010+0000 7f8a5fef8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8a580b6dc0 0x7f8a580b7210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:26.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.010+0000 7f8a5fef8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a580b9cc0 con 0x7f8a5815e0f0 2024-03-24T00:41:26.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.010+0000 7f8a5fef8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a580b7750 con 0x7f8a580b6dc0 2024-03-24T00:41:26.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.010+0000 7f8a5fef8640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a580d2230 con 0x7f8a580d32d0 2024-03-24T00:41:26.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.010+0000 7f8a5e495640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8a580b6dc0 0x7f8a580b7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:26.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.010+0000 7f8a5dc94640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8a5815e0f0 0x7f8a580b95b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33152/0 (socket says 172.21.15.67:33152) 2024-03-24T00:41:26.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.010+0000 7f8a5dc94640 1 -- 172.21.15.67:0/3217242329 learned_addr learned my addr 172.21.15.67:0/3217242329 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:41:26.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.010+0000 7f8a5e495640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8a580b6dc0 0x7f8a580b7210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37430/0 (socket says 172.21.15.67:37430) 2024-03-24T00:41:26.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.010+0000 7f8a5d493640 1 --2- 172.21.15.67:0/3217242329 >> v2:172.21.15.72:3300/0 conn(0x7f8a580d32d0 0x7f8a580d3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:26.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.010+0000 7f8a5e495640 1 -- 172.21.15.67:0/3217242329 >> v2:172.21.15.72:3300/0 conn(0x7f8a580d32d0 msgr2=0x7f8a580d3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:26.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.010+0000 7f8a5e495640 1 --2- 172.21.15.67:0/3217242329 >> v2:172.21.15.72:3300/0 conn(0x7f8a580d32d0 0x7f8a580d3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:26.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.010+0000 7f8a5e495640 1 -- 172.21.15.67:0/3217242329 >> v1:172.21.15.67:6789/0 conn(0x7f8a5815e0f0 legacy=0x7f8a580b95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:26.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.010+0000 7f8a5e495640 1 -- 172.21.15.67:0/3217242329 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a580d2510 con 0x7f8a580b6dc0 2024-03-24T00:41:26.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.011+0000 7f8a5e495640 1 --2- 172.21.15.67:0/3217242329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a580b6dc0 0x7f8a580b7210 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f8a580cacc0 tx=0x7f8a5002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=291a3f404ff4b5f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:26.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.011+0000 7f8a5cc92640 1 -- 172.21.15.67:0/3217242329 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a50039070 con 0x7f8a580b6dc0 2024-03-24T00:41:26.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.011+0000 7f8a5cc92640 1 -- 172.21.15.67:0/3217242329 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a5002cc80 con 0x7f8a580b6dc0 2024-03-24T00:41:26.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.011+0000 7f8a5cc92640 1 -- 172.21.15.67:0/3217242329 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a50034660 con 0x7f8a580b6dc0 2024-03-24T00:41:26.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.012+0000 7f8a5fef8640 1 -- 172.21.15.67:0/3217242329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a580b6dc0 msgr2=0x7f8a580b7210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:26.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.012+0000 7f8a5fef8640 1 --2- 172.21.15.67:0/3217242329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a580b6dc0 0x7f8a580b7210 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f8a580cacc0 tx=0x7f8a5002bed0 comp rx=0 tx=0).stop 2024-03-24T00:41:26.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.012+0000 7f8a5fef8640 1 -- 172.21.15.67:0/3217242329 shutdown_connections 2024-03-24T00:41:26.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.012+0000 7f8a5fef8640 1 --2- 172.21.15.67:0/3217242329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a580b6dc0 0x7f8a580b7210 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:26.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.012+0000 7f8a5fef8640 1 --2- 172.21.15.67:0/3217242329 >> v2:172.21.15.72:3300/0 conn(0x7f8a580d32d0 0x7f8a580d3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:26.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.012+0000 7f8a5fef8640 1 -- 172.21.15.67:0/3217242329 >> 172.21.15.67:0/3217242329 conn(0x7f8a58072b60 msgr2=0x7f8a58072f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:26.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.012+0000 7f8a5fef8640 1 -- 172.21.15.67:0/3217242329 shutdown_connections 2024-03-24T00:41:26.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.012+0000 7f8a5fef8640 1 -- 172.21.15.67:0/3217242329 wait complete. 2024-03-24T00:41:26.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.014+0000 7f8a5fef8640 1 Processor -- start 2024-03-24T00:41:26.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.014+0000 7f8a5fef8640 1 -- start start 2024-03-24T00:41:26.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.015+0000 7f8a5fef8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8a580b6dc0 0x7f8a58150080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:26.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.015+0000 7f8a5fef8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a580d32d0 0x7f8a580aa720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:26.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.015+0000 7f8a5fef8640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8a5815e0f0 0x7f8a580c77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:26.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.015+0000 7f8a5fef8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a580d2db0 con 0x7f8a580d32d0 2024-03-24T00:41:26.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.015+0000 7f8a5fef8640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a580d2ad0 con 0x7f8a580b6dc0 2024-03-24T00:41:26.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.015+0000 7f8a5fef8640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a580b31d0 con 0x7f8a5815e0f0 2024-03-24T00:41:26.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.015+0000 7f8a5d493640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a580d32d0 0x7f8a580aa720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:26.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.015+0000 7f8a5d493640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a580d32d0 0x7f8a580aa720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37446/0 (socket says 172.21.15.67:37446) 2024-03-24T00:41:26.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.015+0000 7f8a5d493640 1 -- 172.21.15.67:0/4248108956 learned_addr learned my addr 172.21.15.67:0/4248108956 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:26.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.015+0000 7f8a5dc94640 1 --2- 172.21.15.67:0/4248108956 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8a580b6dc0 0x7f8a58150080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:26.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.015+0000 7f8a5e495640 1 --2- 172.21.15.67:0/4248108956 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8a5815e0f0 0x7f8a580c77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:26.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.015+0000 7f8a5d493640 1 -- 172.21.15.67:0/4248108956 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8a5815e0f0 msgr2=0x7f8a580c77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:26.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.015+0000 7f8a5d493640 1 --2- 172.21.15.67:0/4248108956 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8a5815e0f0 0x7f8a580c77b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:26.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.015+0000 7f8a5d493640 1 -- 172.21.15.67:0/4248108956 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8a580b6dc0 msgr2=0x7f8a58150080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:26.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.015+0000 7f8a5d493640 1 --2- 172.21.15.67:0/4248108956 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8a580b6dc0 0x7f8a58150080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:26.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.015+0000 7f8a5d493640 1 -- 172.21.15.67:0/4248108956 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a580cc850 con 0x7f8a580d32d0 2024-03-24T00:41:26.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.016+0000 7f8a5d493640 1 --2- 172.21.15.67:0/4248108956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a580d32d0 0x7f8a580aa720 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f8a4800d800 tx=0x7f8a4800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:26.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.016+0000 7f8a4e7fc640 1 -- 172.21.15.67:0/4248108956 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a4800b6c0 con 0x7f8a580d32d0 2024-03-24T00:41:26.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.016+0000 7f8a4e7fc640 1 -- 172.21.15.67:0/4248108956 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a48019070 con 0x7f8a580d32d0 2024-03-24T00:41:26.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.016+0000 7f8a4e7fc640 1 -- 172.21.15.67:0/4248108956 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a4800baa0 con 0x7f8a580d32d0 2024-03-24T00:41:26.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.016+0000 7f8a5fef8640 1 -- 172.21.15.67:0/4248108956 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a580ccaf0 con 0x7f8a580d32d0 2024-03-24T00:41:26.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.016+0000 7f8a5fef8640 1 -- 172.21.15.67:0/4248108956 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a580cd390 con 0x7f8a580d32d0 2024-03-24T00:41:26.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.017+0000 7f8a5fef8640 1 -- 172.21.15.67:0/4248108956 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a580cccc0 con 0x7f8a580d32d0 2024-03-24T00:41:26.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.017+0000 7f8a4e7fc640 1 -- 172.21.15.67:0/4248108956 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8a4800bc40 con 0x7f8a580d32d0 2024-03-24T00:41:26.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.018+0000 7f8a4e7fc640 1 --2- 172.21.15.67:0/4248108956 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8a3c06bbd0 0x7f8a3c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:26.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.018+0000 7f8a4e7fc640 1 -- 172.21.15.67:0/4248108956 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8a4808b0a0 con 0x7f8a580d32d0 2024-03-24T00:41:26.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.018+0000 7f8a5dc94640 1 --2- 172.21.15.67:0/4248108956 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8a3c06bbd0 0x7f8a3c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:26.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.018+0000 7f8a5dc94640 1 --2- 172.21.15.67:0/4248108956 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8a3c06bbd0 0x7f8a3c06e090 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f8a54000f60 tx=0x7f8a54009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:26.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.022+0000 7f8a4e7fc640 1 -- 172.21.15.67:0/4248108956 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8a48059a00 con 0x7f8a580d32d0 2024-03-24T00:41:26.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.220+0000 7f8a5fef8640 1 -- 172.21.15.67:0/4248108956 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f8a5804f400 con 0x7f8a580d32d0 2024-03-24T00:41:26.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.220+0000 7f8a4e7fc640 1 -- 172.21.15.67:0/4248108956 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v50) v1 ==== 80+0+0 (secure 0 0 0) 0x7f8a48059300 con 0x7f8a580d32d0 2024-03-24T00:41:26.221 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-03-24T00:41:26.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.222+0000 7f8a5fef8640 1 -- 172.21.15.67:0/4248108956 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8a3c06bbd0 msgr2=0x7f8a3c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:26.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.222+0000 7f8a5fef8640 1 --2- 172.21.15.67:0/4248108956 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8a3c06bbd0 0x7f8a3c06e090 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f8a54000f60 tx=0x7f8a54009040 comp rx=0 tx=0).stop 2024-03-24T00:41:26.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.222+0000 7f8a5fef8640 1 -- 172.21.15.67:0/4248108956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a580d32d0 msgr2=0x7f8a580aa720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:26.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.222+0000 7f8a5fef8640 1 --2- 172.21.15.67:0/4248108956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a580d32d0 0x7f8a580aa720 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f8a4800d800 tx=0x7f8a4800dcc0 comp rx=0 tx=0).stop 2024-03-24T00:41:26.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.222+0000 7f8a5fef8640 1 -- 172.21.15.67:0/4248108956 shutdown_connections 2024-03-24T00:41:26.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.222+0000 7f8a5fef8640 1 --2- 172.21.15.67:0/4248108956 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8a3c06bbd0 0x7f8a3c06e090 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:26.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.222+0000 7f8a5fef8640 1 --2- 172.21.15.67:0/4248108956 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8a5815e0f0 0x7f8a580c77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:26.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.222+0000 7f8a5fef8640 1 --2- 172.21.15.67:0/4248108956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a580d32d0 0x7f8a580aa720 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:26.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.222+0000 7f8a5fef8640 1 --2- 172.21.15.67:0/4248108956 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8a580b6dc0 0x7f8a58150080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:26.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.222+0000 7f8a5fef8640 1 -- 172.21.15.67:0/4248108956 >> 172.21.15.67:0/4248108956 conn(0x7f8a58072b60 msgr2=0x7f8a580ceb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:26.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.222+0000 7f8a5fef8640 1 -- 172.21.15.67:0/4248108956 shutdown_connections 2024-03-24T00:41:26.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.222+0000 7f8a5fef8640 1 -- 172.21.15.67:0/4248108956 wait complete. 2024-03-24T00:41:26.233 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-24T00:41:26.234 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:41:26.234 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-24T00:41:26.274 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-03-24T00:41:26.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:41:26.275 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-24T00:41:26.330 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:41:26.330 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:41:26.348 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17590 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248808k,nr_inodes=812202,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:41:26.348 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:41:26.348 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:41:26.386 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056375,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15771 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T00:41:26.386 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:41:26.386 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:41:26.404 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17590 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248808k,nr_inodes=812202,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:41:26.405 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:41:26.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.527+0000 7f013fa7e640 1 Processor -- start 2024-03-24T00:41:26.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.527+0000 7f013fa7e640 1 -- start start 2024-03-24T00:41:26.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.528+0000 7f013fa7e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f013815dec0 0x7f01380ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:26.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.528+0000 7f013fa7e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f013815e2a0 0x7f0138162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:26.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.528+0000 7f013fa7e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01380cf640 con 0x7f01380d07e0 2024-03-24T00:41:26.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.528+0000 7f013fa7e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01380cee30 con 0x7f013815e2a0 2024-03-24T00:41:26.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.528+0000 7f013fa7e640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01380d2120 con 0x7f013815dec0 2024-03-24T00:41:26.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.528+0000 7f013e01b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f01380d07e0 0x7f01380b5850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33154/0 (socket says 172.21.15.67:33154) 2024-03-24T00:41:26.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.528+0000 7f013e01b640 1 -- 172.21.15.67:0/1228953176 learned_addr learned my addr 172.21.15.67:0/1228953176 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:41:26.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.529+0000 7f013d81a640 1 --2- 172.21.15.67:0/1228953176 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f013815dec0 0x7f01380ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:26.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.529+0000 7f013d019640 1 --2- 172.21.15.67:0/1228953176 >> v2:172.21.15.72:3300/0 conn(0x7f013815e2a0 0x7f0138162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:26.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.529+0000 7f013c818640 1 -- 172.21.15.67:0/1228953176 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1915051796 0 0) 0x7f0130001490 con 0x7f01380d07e0 2024-03-24T00:41:26.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.529+0000 7f013c818640 1 -- 172.21.15.67:0/1228953176 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0124003520 con 0x7f01380d07e0 2024-03-24T00:41:26.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.529+0000 7f013c818640 1 -- 172.21.15.67:0/1228953176 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2649063401 0 0) 0x7f0124003520 con 0x7f01380d07e0 2024-03-24T00:41:26.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.529+0000 7f013c818640 1 -- 172.21.15.67:0/1228953176 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f013815dec0 msgr2=0x7f01380ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:26.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.529+0000 7f013c818640 1 --2- 172.21.15.67:0/1228953176 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f013815dec0 0x7f01380ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:26.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.529+0000 7f013c818640 1 -- 172.21.15.67:0/1228953176 >> v2:172.21.15.72:3300/0 conn(0x7f013815e2a0 msgr2=0x7f0138162630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:26.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.529+0000 7f013c818640 1 --2- 172.21.15.67:0/1228953176 >> v2:172.21.15.72:3300/0 conn(0x7f013815e2a0 0x7f0138162630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:26.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.529+0000 7f013c818640 1 -- 172.21.15.67:0/1228953176 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01380cf080 con 0x7f01380d07e0 2024-03-24T00:41:26.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.529+0000 7f013d81a640 1 --2- 172.21.15.67:0/1228953176 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f013815dec0 0x7f01380ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:41:26.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.529+0000 7f013c818640 1 -- 172.21.15.67:0/1228953176 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f0130001bc0 con 0x7f01380d07e0 2024-03-24T00:41:26.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.529+0000 7f013d019640 1 --2- 172.21.15.67:0/1228953176 >> v2:172.21.15.72:3300/0 conn(0x7f013815e2a0 0x7f0138162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:41:26.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.529+0000 7f013c818640 1 -- 172.21.15.67:0/1228953176 >> v1:172.21.15.67:6789/0 conn(0x7f01380d07e0 legacy=0x7f01380b5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:26.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.529+0000 7f013c818640 1 --2- 172.21.15.67:0/1228953176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01240039b0 0x7f0124003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:26.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.529+0000 7f013c818640 1 --2- 172.21.15.67:0/1228953176 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f013815dec0 0x7f0124004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:26.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.529+0000 7f013c818640 1 --2- 172.21.15.67:0/1228953176 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0124004d40 0x7f01240051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:26.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.529+0000 7f013c818640 1 -- 172.21.15.67:0/1228953176 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01380cee30 con 0x7f01240039b0 2024-03-24T00:41:26.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.529+0000 7f013c818640 1 -- 172.21.15.67:0/1228953176 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01380d2120 con 0x7f0124004d40 2024-03-24T00:41:26.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.529+0000 7f013c818640 1 -- 172.21.15.67:0/1228953176 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f012400b9c0 con 0x7f013815dec0 2024-03-24T00:41:26.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.529+0000 7f013d81a640 1 --2- 172.21.15.67:0/1228953176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01240039b0 0x7f0124003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:26.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.529+0000 7f013d019640 1 --2- 172.21.15.67:0/1228953176 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f013815dec0 0x7f0124004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:26.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.529+0000 7f013e01b640 1 --2- 172.21.15.67:0/1228953176 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0124004d40 0x7f01240051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:26.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.529+0000 7f013d81a640 1 -- 172.21.15.67:0/1228953176 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f013815dec0 msgr2=0x7f0124004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:26.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.529+0000 7f013d81a640 1 --2- 172.21.15.67:0/1228953176 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f013815dec0 0x7f0124004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:26.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.529+0000 7f013d81a640 1 -- 172.21.15.67:0/1228953176 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0124004d40 msgr2=0x7f01240051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:26.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.529+0000 7f013d81a640 1 --2- 172.21.15.67:0/1228953176 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0124004d40 0x7f01240051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:26.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.529+0000 7f013d81a640 1 -- 172.21.15.67:0/1228953176 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0124003520 con 0x7f01240039b0 2024-03-24T00:41:26.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.529+0000 7f013e01b640 1 --2- 172.21.15.67:0/1228953176 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0124004d40 0x7f01240051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:41:26.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.530+0000 7f013d81a640 1 --2- 172.21.15.67:0/1228953176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01240039b0 0x7f0124003df0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f0134009cc0 tx=0x7f0124005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77a24d36a26f39dc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:26.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.530+0000 7f013c818640 1 -- 172.21.15.67:0/1228953176 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0134039070 con 0x7f01240039b0 2024-03-24T00:41:26.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.530+0000 7f013c818640 1 -- 172.21.15.67:0/1228953176 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f013402c8f0 con 0x7f01240039b0 2024-03-24T00:41:26.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.530+0000 7f013c818640 1 -- 172.21.15.67:0/1228953176 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0134009860 con 0x7f01240039b0 2024-03-24T00:41:26.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.531+0000 7f013fa7e640 1 -- 172.21.15.67:0/1228953176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01240039b0 msgr2=0x7f0124003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:26.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.531+0000 7f013fa7e640 1 --2- 172.21.15.67:0/1228953176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01240039b0 0x7f0124003df0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f0134009cc0 tx=0x7f0124005830 comp rx=0 tx=0).stop 2024-03-24T00:41:26.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.531+0000 7f013e01b640 1 -- 172.21.15.67:0/1228953176 reap_dead start 2024-03-24T00:41:26.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.531+0000 7f013fa7e640 1 -- 172.21.15.67:0/1228953176 shutdown_connections 2024-03-24T00:41:26.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.531+0000 7f013fa7e640 1 -- 172.21.15.67:0/1228953176 >> 172.21.15.67:0/1228953176 conn(0x7f013806c810 msgr2=0x7f013806cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:26.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.531+0000 7f013fa7e640 1 -- 172.21.15.67:0/1228953176 shutdown_connections 2024-03-24T00:41:26.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.531+0000 7f013fa7e640 1 -- 172.21.15.67:0/1228953176 wait complete. 2024-03-24T00:41:26.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.532+0000 7f013fa7e640 1 Processor -- start 2024-03-24T00:41:26.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.532+0000 7f013fa7e640 1 -- start start 2024-03-24T00:41:26.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.533+0000 7f013fa7e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f01240039b0 0x7f01380b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:26.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.533+0000 7f013fa7e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f013814cd60 0x7f01380b55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:26.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.533+0000 7f013fa7e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f013814d1e0 0x7f01380b5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:26.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.533+0000 7f013fa7e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01380d3740 con 0x7f013814d1e0 2024-03-24T00:41:26.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.533+0000 7f013fa7e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01380d35c0 con 0x7f013814cd60 2024-03-24T00:41:26.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.533+0000 7f013fa7e640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01380d38c0 con 0x7f01240039b0 2024-03-24T00:41:26.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.533+0000 7f013e01b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f013814d1e0 0x7f01380b5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:26.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.533+0000 7f013e01b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f013814d1e0 0x7f01380b5b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37454/0 (socket says 172.21.15.67:37454) 2024-03-24T00:41:26.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.533+0000 7f013e01b640 1 -- 172.21.15.67:0/4252999987 learned_addr learned my addr 172.21.15.67:0/4252999987 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:26.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.533+0000 7f013d019640 1 --2- 172.21.15.67:0/4252999987 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f013814cd60 0x7f01380b55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:26.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.533+0000 7f013d81a640 1 --2- 172.21.15.67:0/4252999987 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f01240039b0 0x7f01380b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:26.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.533+0000 7f013e01b640 1 -- 172.21.15.67:0/4252999987 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f01240039b0 msgr2=0x7f01380b21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:26.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.533+0000 7f013e01b640 1 --2- 172.21.15.67:0/4252999987 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f01240039b0 0x7f01380b21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:26.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.533+0000 7f013e01b640 1 -- 172.21.15.67:0/4252999987 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f013814cd60 msgr2=0x7f01380b55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:26.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.533+0000 7f013e01b640 1 --2- 172.21.15.67:0/4252999987 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f013814cd60 0x7f01380b55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:26.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.533+0000 7f013e01b640 1 -- 172.21.15.67:0/4252999987 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01380b2730 con 0x7f013814d1e0 2024-03-24T00:41:26.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.533+0000 7f013d81a640 1 --2- 172.21.15.67:0/4252999987 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f01240039b0 0x7f01380b21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:41:26.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.533+0000 7f013e01b640 1 --2- 172.21.15.67:0/4252999987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f013814d1e0 0x7f01380b5b20 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f01380cae20 tx=0x7f013000a550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:26.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.533+0000 7f012e7fc640 1 -- 172.21.15.67:0/4252999987 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0130019070 con 0x7f013814d1e0 2024-03-24T00:41:26.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.533+0000 7f012e7fc640 1 -- 172.21.15.67:0/4252999987 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f013000abb0 con 0x7f013814d1e0 2024-03-24T00:41:26.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.533+0000 7f013fa7e640 1 -- 172.21.15.67:0/4252999987 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01380b4540 con 0x7f013814d1e0 2024-03-24T00:41:26.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.533+0000 7f012e7fc640 1 -- 172.21.15.67:0/4252999987 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0130014480 con 0x7f013814d1e0 2024-03-24T00:41:26.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.533+0000 7f013fa7e640 1 -- 172.21.15.67:0/4252999987 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01380b4b00 con 0x7f013814d1e0 2024-03-24T00:41:26.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.534+0000 7f013fa7e640 1 -- 172.21.15.67:0/4252999987 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f013804f400 con 0x7f013814d1e0 2024-03-24T00:41:26.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.535+0000 7f012e7fc640 1 -- 172.21.15.67:0/4252999987 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f013001c070 con 0x7f013814d1e0 2024-03-24T00:41:26.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.535+0000 7f012e7fc640 1 --2- 172.21.15.67:0/4252999987 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f011c06bbd0 0x7f011c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:26.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.536+0000 7f012e7fc640 1 -- 172.21.15.67:0/4252999987 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f013008a900 con 0x7f013814d1e0 2024-03-24T00:41:26.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.536+0000 7f013d81a640 1 --2- 172.21.15.67:0/4252999987 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f011c06bbd0 0x7f011c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:26.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.536+0000 7f013d81a640 1 --2- 172.21.15.67:0/4252999987 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f011c06bbd0 0x7f011c06e090 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0134036000 tx=0x7f0134030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:26.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.540+0000 7f012e7fc640 1 -- 172.21.15.67:0/4252999987 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f01380cf640 con 0x7f013814d1e0 2024-03-24T00:41:26.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.758+0000 7f013fa7e640 1 -- 172.21.15.67:0/4252999987 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01380d1aa0 con 0x7f013814d1e0 2024-03-24T00:41:26.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.759+0000 7f012e7fc640 1 -- 172.21.15.67:0/4252999987 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f0130058c10 con 0x7f013814d1e0 2024-03-24T00:41:26.760 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:41:26.760 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5360,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3312887486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":3312887486},{"type":"v1","addr":"172.21.15.67:6835","nonce":3312887486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5821,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/4171866637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":4171866637},{"type":"v1","addr":"172.21.15.72:6836","nonce":4171866637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5824,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/4106412452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":4106412452},{"type":"v1","addr":"172.21.15.72:6837","nonce":4106412452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:40:39.649631+0000","modified":"2024-03-24T00:40:42.657888+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5853},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5853":{"gid":5853,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6834/2741818214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2741818214},{"type":"v1","addr":"172.21.15.67:6834","nonce":2741818214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5853,"qdb_cluster":[5853]},"id":3}]} 2024-03-24T00:41:26.760 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 26 2024-03-24T00:41:26.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.761+0000 7f013fa7e640 1 -- 172.21.15.67:0/4252999987 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f011c06bbd0 msgr2=0x7f011c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:26.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.761+0000 7f013fa7e640 1 --2- 172.21.15.67:0/4252999987 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f011c06bbd0 0x7f011c06e090 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0134036000 tx=0x7f0134030000 comp rx=0 tx=0).stop 2024-03-24T00:41:26.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.761+0000 7f013fa7e640 1 -- 172.21.15.67:0/4252999987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f013814d1e0 msgr2=0x7f01380b5b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:26.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.761+0000 7f013fa7e640 1 --2- 172.21.15.67:0/4252999987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f013814d1e0 0x7f01380b5b20 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f01380cae20 tx=0x7f013000a550 comp rx=0 tx=0).stop 2024-03-24T00:41:26.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.761+0000 7f013fa7e640 1 -- 172.21.15.67:0/4252999987 shutdown_connections 2024-03-24T00:41:26.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.761+0000 7f013fa7e640 1 --2- 172.21.15.67:0/4252999987 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f011c06bbd0 0x7f011c06e090 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:26.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.761+0000 7f013fa7e640 1 --2- 172.21.15.67:0/4252999987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f013814d1e0 0x7f01380b5b20 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:26.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.761+0000 7f013fa7e640 1 --2- 172.21.15.67:0/4252999987 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f013814cd60 0x7f01380b55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:26.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.761+0000 7f013fa7e640 1 --2- 172.21.15.67:0/4252999987 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f01240039b0 0x7f01380b21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:26.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.761+0000 7f013fa7e640 1 -- 172.21.15.67:0/4252999987 >> 172.21.15.67:0/4252999987 conn(0x7f013806c810 msgr2=0x7f013815e6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:26.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.761+0000 7f013fa7e640 1 -- 172.21.15.67:0/4252999987 shutdown_connections 2024-03-24T00:41:26.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.761+0000 7f013fa7e640 1 -- 172.21.15.67:0/4252999987 wait complete. 2024-03-24T00:41:26.775 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:41:26.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.910+0000 7f480e2f1640 1 Processor -- start 2024-03-24T00:41:26.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.910+0000 7f480e2f1640 1 -- start start 2024-03-24T00:41:26.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.911+0000 7f480e2f1640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f480815e2a0 0x7f48080ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:26.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.911+0000 7f480e2f1640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f48080d0ef0 0x7f4808162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:26.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.911+0000 7f480e2f1640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48080cf640 con 0x7f480815dec0 2024-03-24T00:41:26.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.911+0000 7f480e2f1640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48080cee30 con 0x7f48080d0ef0 2024-03-24T00:41:26.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.911+0000 7f480e2f1640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48080d2120 con 0x7f480815e2a0 2024-03-24T00:41:26.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.911+0000 7f480c88e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f48080d0ef0 0x7f4808162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:26.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.911+0000 7f480c88e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f48080d0ef0 0x7f4808162630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37460/0 (socket says 172.21.15.67:37460) 2024-03-24T00:41:26.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.911+0000 7f480c88e640 1 -- 172.21.15.67:0/1880693926 learned_addr learned my addr 172.21.15.67:0/1880693926 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:26.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.911+0000 7f48077fe640 1 --2- 172.21.15.67:0/1880693926 >> v2:172.21.15.72:3300/0 conn(0x7f480815e2a0 0x7f48080ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:26.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.912+0000 7f480c88e640 1 -- 172.21.15.67:0/1880693926 >> v2:172.21.15.72:3300/0 conn(0x7f480815e2a0 msgr2=0x7f48080ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:26.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.912+0000 7f480c88e640 1 --2- 172.21.15.67:0/1880693926 >> v2:172.21.15.72:3300/0 conn(0x7f480815e2a0 0x7f48080ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:26.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.912+0000 7f480c88e640 1 -- 172.21.15.67:0/1880693926 >> v1:172.21.15.72:6789/0 conn(0x7f480815dec0 legacy=0x7f48080d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:26.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.912+0000 7f480c88e640 1 -- 172.21.15.67:0/1880693926 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48080cf080 con 0x7f48080d0ef0 2024-03-24T00:41:26.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.913+0000 7f480c88e640 1 --2- 172.21.15.67:0/1880693926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48080d0ef0 0x7f4808162630 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f48080cede0 tx=0x7f480002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=487284a10afa0a73 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:26.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.913+0000 7f4806ffd640 1 -- 172.21.15.67:0/1880693926 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f480002c9f0 con 0x7f48080d0ef0 2024-03-24T00:41:26.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.913+0000 7f4806ffd640 1 -- 172.21.15.67:0/1880693926 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f480002cb90 con 0x7f48080d0ef0 2024-03-24T00:41:26.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.913+0000 7f4806ffd640 1 -- 172.21.15.67:0/1880693926 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48000343f0 con 0x7f48080d0ef0 2024-03-24T00:41:26.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.914+0000 7f480e2f1640 1 -- 172.21.15.67:0/1880693926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48080d0ef0 msgr2=0x7f4808162630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:26.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.914+0000 7f480e2f1640 1 --2- 172.21.15.67:0/1880693926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48080d0ef0 0x7f4808162630 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f48080cede0 tx=0x7f480002be50 comp rx=0 tx=0).stop 2024-03-24T00:41:26.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.914+0000 7f480e2f1640 1 -- 172.21.15.67:0/1880693926 shutdown_connections 2024-03-24T00:41:26.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.914+0000 7f480e2f1640 1 --2- 172.21.15.67:0/1880693926 >> v2:172.21.15.72:3300/0 conn(0x7f480815e2a0 0x7f48080ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:26.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.914+0000 7f480e2f1640 1 --2- 172.21.15.67:0/1880693926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48080d0ef0 0x7f4808162630 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:26.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.914+0000 7f480e2f1640 1 -- 172.21.15.67:0/1880693926 >> 172.21.15.67:0/1880693926 conn(0x7f480806c810 msgr2=0x7f480806cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:26.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.914+0000 7f480e2f1640 1 -- 172.21.15.67:0/1880693926 shutdown_connections 2024-03-24T00:41:26.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.914+0000 7f480e2f1640 1 -- 172.21.15.67:0/1880693926 wait complete. 2024-03-24T00:41:26.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.915+0000 7f480e2f1640 1 Processor -- start 2024-03-24T00:41:26.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.915+0000 7f480e2f1640 1 -- start start 2024-03-24T00:41:26.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.916+0000 7f480e2f1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f48080d0ef0 0x7f48081592f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:26.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.916+0000 7f480e2f1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f480815dec0 0x7f4808156870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:26.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.916+0000 7f480e2f1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f480815e2a0 0x7f48081585c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:26.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.916+0000 7f480e2f1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48080d2730 con 0x7f480815e2a0 2024-03-24T00:41:26.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.916+0000 7f480e2f1640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48080d25b0 con 0x7f480815dec0 2024-03-24T00:41:26.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.916+0000 7f480e2f1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48080d28b0 con 0x7f48080d0ef0 2024-03-24T00:41:26.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.916+0000 7f480c88e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f480815e2a0 0x7f48081585c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:26.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.916+0000 7f480c88e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f480815e2a0 0x7f48081585c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37476/0 (socket says 172.21.15.67:37476) 2024-03-24T00:41:26.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.916+0000 7f480c88e640 1 -- 172.21.15.67:0/2994153420 learned_addr learned my addr 172.21.15.67:0/2994153420 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:26.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.916+0000 7f48077fe640 1 --2- 172.21.15.67:0/2994153420 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f480815dec0 0x7f4808156870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:26.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.916+0000 7f4807fff640 1 --2- 172.21.15.67:0/2994153420 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f48080d0ef0 0x7f48081592f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:26.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.917+0000 7f480c88e640 1 -- 172.21.15.67:0/2994153420 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f48080d0ef0 msgr2=0x7f48081592f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:26.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.917+0000 7f480c88e640 1 --2- 172.21.15.67:0/2994153420 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f48080d0ef0 0x7f48081592f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:26.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.917+0000 7f480c88e640 1 -- 172.21.15.67:0/2994153420 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f480815dec0 msgr2=0x7f4808156870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:26.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.917+0000 7f480c88e640 1 --2- 172.21.15.67:0/2994153420 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f480815dec0 0x7f4808156870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:26.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.917+0000 7f480c88e640 1 -- 172.21.15.67:0/2994153420 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48080b3ca0 con 0x7f480815e2a0 2024-03-24T00:41:26.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.917+0000 7f480c88e640 1 --2- 172.21.15.67:0/2994153420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f480815e2a0 0x7f48081585c0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f480002c460 tx=0x7f4800004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:26.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.917+0000 7f4804ff9640 1 -- 172.21.15.67:0/2994153420 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4800043020 con 0x7f480815e2a0 2024-03-24T00:41:26.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.917+0000 7f4804ff9640 1 -- 172.21.15.67:0/2994153420 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f480003e450 con 0x7f480815e2a0 2024-03-24T00:41:26.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.917+0000 7f480e2f1640 1 -- 172.21.15.67:0/2994153420 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48080b3f80 con 0x7f480815e2a0 2024-03-24T00:41:26.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.917+0000 7f4804ff9640 1 -- 172.21.15.67:0/2994153420 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4800009510 con 0x7f480815e2a0 2024-03-24T00:41:26.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.917+0000 7f480e2f1640 1 -- 172.21.15.67:0/2994153420 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48080b4820 con 0x7f480815e2a0 2024-03-24T00:41:26.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.918+0000 7f480e2f1640 1 -- 172.21.15.67:0/2994153420 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48080b4150 con 0x7f480815e2a0 2024-03-24T00:41:26.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.918+0000 7f4804ff9640 1 -- 172.21.15.67:0/2994153420 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4800046020 con 0x7f480815e2a0 2024-03-24T00:41:26.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.919+0000 7f4804ff9640 1 --2- 172.21.15.67:0/2994153420 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f47f406bbd0 0x7f47f406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:26.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.919+0000 7f4804ff9640 1 -- 172.21.15.67:0/2994153420 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f48000ad2c0 con 0x7f480815e2a0 2024-03-24T00:41:26.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.919+0000 7f4807fff640 1 --2- 172.21.15.67:0/2994153420 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f47f406bbd0 0x7f47f406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:26.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.920+0000 7f4807fff640 1 --2- 172.21.15.67:0/2994153420 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f47f406bbd0 0x7f47f406e090 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f47f8004f60 tx=0x7f47f8007e30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:26.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:26.923+0000 7f4804ff9640 1 -- 172.21.15.67:0/2994153420 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f480004d030 con 0x7f480815e2a0 2024-03-24T00:41:27.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.108+0000 7f480e2f1640 1 -- 172.21.15.67:0/2994153420 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f48080b4a20 con 0x7f480815e2a0 2024-03-24T00:41:27.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.109+0000 7f4804ff9640 1 -- 172.21.15.67:0/2994153420 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) v1 ==== 74+0+16882 (secure 0 0 0) 0x7f48080b4a20 con 0x7f480815e2a0 2024-03-24T00:41:27.110 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:41:27.110 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":50,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:40:41.288503+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:40:37.927587+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-24T00:40:38.616521+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:41:27.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.111+0000 7f480e2f1640 1 -- 172.21.15.67:0/2994153420 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f47f406bbd0 msgr2=0x7f47f406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:27.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.111+0000 7f480e2f1640 1 --2- 172.21.15.67:0/2994153420 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f47f406bbd0 0x7f47f406e090 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f47f8004f60 tx=0x7f47f8007e30 comp rx=0 tx=0).stop 2024-03-24T00:41:27.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.111+0000 7f480e2f1640 1 -- 172.21.15.67:0/2994153420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f480815e2a0 msgr2=0x7f48081585c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:27.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.111+0000 7f480e2f1640 1 --2- 172.21.15.67:0/2994153420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f480815e2a0 0x7f48081585c0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f480002c460 tx=0x7f4800004a40 comp rx=0 tx=0).stop 2024-03-24T00:41:27.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.111+0000 7f480e2f1640 1 -- 172.21.15.67:0/2994153420 shutdown_connections 2024-03-24T00:41:27.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.111+0000 7f480e2f1640 1 --2- 172.21.15.67:0/2994153420 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f47f406bbd0 0x7f47f406e090 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.111+0000 7f480e2f1640 1 --2- 172.21.15.67:0/2994153420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f480815e2a0 0x7f48081585c0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.111+0000 7f480e2f1640 1 --2- 172.21.15.67:0/2994153420 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f480815dec0 0x7f4808156870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.111+0000 7f480e2f1640 1 --2- 172.21.15.67:0/2994153420 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f48080d0ef0 0x7f48081592f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.111+0000 7f480e2f1640 1 -- 172.21.15.67:0/2994153420 >> 172.21.15.67:0/2994153420 conn(0x7f480806c810 msgr2=0x7f48080ca210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:27.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.111+0000 7f480e2f1640 1 -- 172.21.15.67:0/2994153420 shutdown_connections 2024-03-24T00:41:27.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.111+0000 7f480e2f1640 1 -- 172.21.15.67:0/2994153420 wait complete. 2024-03-24T00:41:27.122 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:38:39.364950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:40:37.927587+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '49', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:40:38.616521+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '50', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:41:27.123 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:41:27.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.226+0000 7f98c2922640 1 Processor -- start 2024-03-24T00:41:27.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.226+0000 7f98c2922640 1 -- start start 2024-03-24T00:41:27.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.227+0000 7f98c2922640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f98bc15dea0 0x7f98bc0d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:27.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.227+0000 7f98c2922640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f98bc15e280 0x7f98bc0d3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:27.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.227+0000 7f98c2922640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98bc0d2250 con 0x7f98bc0b5a60 2024-03-24T00:41:27.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.227+0000 7f98c2922640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98bc0d2530 con 0x7f98bc15dea0 2024-03-24T00:41:27.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.227+0000 7f98c2922640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98bc0d2810 con 0x7f98bc15e280 2024-03-24T00:41:27.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.228+0000 7f98bbfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f98bc15dea0 0x7f98bc0d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:27.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.228+0000 7f98bbfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f98bc15dea0 0x7f98bc0d33d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37492/0 (socket says 172.21.15.67:37492) 2024-03-24T00:41:27.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.228+0000 7f98bbfff640 1 -- 172.21.15.67:0/3526107082 learned_addr learned my addr 172.21.15.67:0/3526107082 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:27.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.228+0000 7f98bb7fe640 1 --2- 172.21.15.67:0/3526107082 >> v2:172.21.15.72:3300/0 conn(0x7f98bc15e280 0x7f98bc0d3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:27.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.228+0000 7f98bbfff640 1 -- 172.21.15.67:0/3526107082 >> v2:172.21.15.72:3300/0 conn(0x7f98bc15e280 msgr2=0x7f98bc0d3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:27.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.228+0000 7f98bbfff640 1 --2- 172.21.15.67:0/3526107082 >> v2:172.21.15.72:3300/0 conn(0x7f98bc15e280 0x7f98bc0d3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.228+0000 7f98bbfff640 1 -- 172.21.15.67:0/3526107082 >> v1:172.21.15.72:6789/0 conn(0x7f98bc0b5a60 legacy=0x7f98bc06c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:27.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.228+0000 7f98bbfff640 1 -- 172.21.15.67:0/3526107082 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98bc0d2af0 con 0x7f98bc15dea0 2024-03-24T00:41:27.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.229+0000 7f98bbfff640 1 --2- 172.21.15.67:0/3526107082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98bc15dea0 0x7f98bc0d33d0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f98bc0d04d0 tx=0x7f98ac02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc591e442c42edc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:27.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.229+0000 7f98baffd640 1 -- 172.21.15.67:0/3526107082 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98ac02e070 con 0x7f98bc15dea0 2024-03-24T00:41:27.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.229+0000 7f98baffd640 1 -- 172.21.15.67:0/3526107082 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98ac037070 con 0x7f98bc15dea0 2024-03-24T00:41:27.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.229+0000 7f98baffd640 1 -- 172.21.15.67:0/3526107082 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98ac031a00 con 0x7f98bc15dea0 2024-03-24T00:41:27.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.229+0000 7f98c2922640 1 -- 172.21.15.67:0/3526107082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98bc15dea0 msgr2=0x7f98bc0d33d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:27.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.229+0000 7f98c2922640 1 --2- 172.21.15.67:0/3526107082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98bc15dea0 0x7f98bc0d33d0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f98bc0d04d0 tx=0x7f98ac02b880 comp rx=0 tx=0).stop 2024-03-24T00:41:27.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.229+0000 7f98c2922640 1 -- 172.21.15.67:0/3526107082 shutdown_connections 2024-03-24T00:41:27.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.229+0000 7f98c2922640 1 --2- 172.21.15.67:0/3526107082 >> v2:172.21.15.72:3300/0 conn(0x7f98bc15e280 0x7f98bc0d3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.229+0000 7f98c2922640 1 --2- 172.21.15.67:0/3526107082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98bc15dea0 0x7f98bc0d33d0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.229+0000 7f98c2922640 1 -- 172.21.15.67:0/3526107082 >> 172.21.15.67:0/3526107082 conn(0x7f98bc072da0 msgr2=0x7f98bc0d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:27.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.229+0000 7f98c2922640 1 -- 172.21.15.67:0/3526107082 shutdown_connections 2024-03-24T00:41:27.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.229+0000 7f98c2922640 1 -- 172.21.15.67:0/3526107082 wait complete. 2024-03-24T00:41:27.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.230+0000 7f98c2922640 1 Processor -- start 2024-03-24T00:41:27.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.231+0000 7f98c2922640 1 -- start start 2024-03-24T00:41:27.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.231+0000 7f98c2922640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98bc0b5a60 0x7f98bc0aa990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:27.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.231+0000 7f98c2922640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f98bc15dea0 0x7f98bc1f77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:27.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.231+0000 7f98c2922640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98bc15e280 0x7f98bc1533c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:27.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.231+0000 7f98c2922640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98bc0b5eb0 con 0x7f98bc0b5a60 2024-03-24T00:41:27.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.231+0000 7f98c2922640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98bc0b8310 con 0x7f98bc15e280 2024-03-24T00:41:27.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.231+0000 7f98c2922640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98bc0b6030 con 0x7f98bc15dea0 2024-03-24T00:41:27.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.231+0000 7f98bbfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98bc0b5a60 0x7f98bc0aa990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:27.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.232+0000 7f98bbfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98bc0b5a60 0x7f98bc0aa990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37496/0 (socket says 172.21.15.67:37496) 2024-03-24T00:41:27.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.232+0000 7f98bbfff640 1 -- 172.21.15.67:0/2788128555 learned_addr learned my addr 172.21.15.67:0/2788128555 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:27.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.232+0000 7f98bb7fe640 1 --2- 172.21.15.67:0/2788128555 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f98bc15dea0 0x7f98bc1f77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:27.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.232+0000 7f98c0ebf640 1 --2- 172.21.15.67:0/2788128555 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98bc15e280 0x7f98bc1533c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:27.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.232+0000 7f98bbfff640 1 -- 172.21.15.67:0/2788128555 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f98bc15dea0 msgr2=0x7f98bc1f77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:27.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.232+0000 7f98bbfff640 1 --2- 172.21.15.67:0/2788128555 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f98bc15dea0 0x7f98bc1f77a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.232+0000 7f98bbfff640 1 -- 172.21.15.67:0/2788128555 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98bc15e280 msgr2=0x7f98bc1533c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:27.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.232+0000 7f98bbfff640 1 --2- 172.21.15.67:0/2788128555 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98bc15e280 0x7f98bc1533c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.232+0000 7f98bbfff640 1 -- 172.21.15.67:0/2788128555 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98bc0ab020 con 0x7f98bc0b5a60 2024-03-24T00:41:27.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.232+0000 7f98bbfff640 1 --2- 172.21.15.67:0/2788128555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98bc0b5a60 0x7f98bc0aa990 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f98ac02b8c0 tx=0x7f98ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:27.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.232+0000 7f98b8ff9640 1 -- 172.21.15.67:0/2788128555 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98ac032880 con 0x7f98bc0b5a60 2024-03-24T00:41:27.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.232+0000 7f98b8ff9640 1 -- 172.21.15.67:0/2788128555 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98ac032a20 con 0x7f98bc0b5a60 2024-03-24T00:41:27.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.232+0000 7f98c2922640 1 -- 172.21.15.67:0/2788128555 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98bc0ab300 con 0x7f98bc0b5a60 2024-03-24T00:41:27.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.232+0000 7f98b8ff9640 1 -- 172.21.15.67:0/2788128555 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98ac032e10 con 0x7f98bc0b5a60 2024-03-24T00:41:27.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.232+0000 7f98c2922640 1 -- 172.21.15.67:0/2788128555 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98bc0ab8c0 con 0x7f98bc0b5a60 2024-03-24T00:41:27.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.234+0000 7f98c2922640 1 -- 172.21.15.67:0/2788128555 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98bc04f400 con 0x7f98bc0b5a60 2024-03-24T00:41:27.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.234+0000 7f98b8ff9640 1 -- 172.21.15.67:0/2788128555 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f98ac0453f0 con 0x7f98bc0b5a60 2024-03-24T00:41:27.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.234+0000 7f98b8ff9640 1 --2- 172.21.15.67:0/2788128555 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f98a806bbd0 0x7f98a806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:27.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.234+0000 7f98b8ff9640 1 -- 172.21.15.67:0/2788128555 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f98ac0acc00 con 0x7f98bc0b5a60 2024-03-24T00:41:27.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.234+0000 7f98bb7fe640 1 --2- 172.21.15.67:0/2788128555 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f98a806bbd0 0x7f98a806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:27.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.235+0000 7f98bb7fe640 1 --2- 172.21.15.67:0/2788128555 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f98a806bbd0 0x7f98a806e090 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f98bc0ce7f0 tx=0x7f98b0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:27.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.238+0000 7f98b8ff9640 1 -- 172.21.15.67:0/2788128555 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f98ac0b3080 con 0x7f98bc0b5a60 2024-03-24T00:41:27.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.423+0000 7f98c2922640 1 -- 172.21.15.67:0/2788128555 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f98bc0b8c80 con 0x7f98bc0b5a60 2024-03-24T00:41:27.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.424+0000 7f98b8ff9640 1 -- 172.21.15.67:0/2788128555 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v50) v1 ==== 93+0+10 (secure 0 0 0) 0x7f98ac040070 con 0x7f98bc0b5a60 2024-03-24T00:41:27.425 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-24T00:41:27.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.426+0000 7f98c2922640 1 -- 172.21.15.67:0/2788128555 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f98a806bbd0 msgr2=0x7f98a806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:27.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.426+0000 7f98c2922640 1 --2- 172.21.15.67:0/2788128555 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f98a806bbd0 0x7f98a806e090 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f98bc0ce7f0 tx=0x7f98b0008040 comp rx=0 tx=0).stop 2024-03-24T00:41:27.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.426+0000 7f98c2922640 1 -- 172.21.15.67:0/2788128555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98bc0b5a60 msgr2=0x7f98bc0aa990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:27.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.426+0000 7f98c2922640 1 --2- 172.21.15.67:0/2788128555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98bc0b5a60 0x7f98bc0aa990 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f98ac02b8c0 tx=0x7f98ac0046c0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.426+0000 7f98c2922640 1 -- 172.21.15.67:0/2788128555 shutdown_connections 2024-03-24T00:41:27.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.426+0000 7f98c2922640 1 --2- 172.21.15.67:0/2788128555 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f98a806bbd0 0x7f98a806e090 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.426+0000 7f98c2922640 1 --2- 172.21.15.67:0/2788128555 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98bc15e280 0x7f98bc1533c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.426+0000 7f98c2922640 1 --2- 172.21.15.67:0/2788128555 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f98bc15dea0 0x7f98bc1f77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.426+0000 7f98c2922640 1 --2- 172.21.15.67:0/2788128555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98bc0b5a60 0x7f98bc0aa990 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.426+0000 7f98c2922640 1 -- 172.21.15.67:0/2788128555 >> 172.21.15.67:0/2788128555 conn(0x7f98bc072da0 msgr2=0x7f98bc160750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:27.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.426+0000 7f98c2922640 1 -- 172.21.15.67:0/2788128555 shutdown_connections 2024-03-24T00:41:27.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.426+0000 7f98c2922640 1 -- 172.21.15.67:0/2788128555 wait complete. 2024-03-24T00:41:27.437 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:41:27.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.541+0000 7ff153c85640 1 Processor -- start 2024-03-24T00:41:27.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.541+0000 7ff153c85640 1 -- start start 2024-03-24T00:41:27.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.542+0000 7ff153c85640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff14c15dea0 0x7ff14c0d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:27.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.542+0000 7ff153c85640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff14c0d2250 con 0x7ff14c0b6170 2024-03-24T00:41:27.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.542+0000 7ff153c85640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff14c0d2530 con 0x7ff14c15e280 2024-03-24T00:41:27.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.542+0000 7ff153c85640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff14c0d2810 con 0x7ff14c15dea0 2024-03-24T00:41:27.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.542+0000 7ff152222640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff14c0b6170 0x7ff14c0b70f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33156/0 (socket says 172.21.15.67:33156) 2024-03-24T00:41:27.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.542+0000 7ff152222640 1 -- 172.21.15.67:0/4131018665 learned_addr learned my addr 172.21.15.67:0/4131018665 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:41:27.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.542+0000 7ff151a21640 1 --2- 172.21.15.67:0/4131018665 >> v2:172.21.15.72:3300/0 conn(0x7ff14c15dea0 0x7ff14c0d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:27.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.542+0000 7ff150a1f640 1 -- 172.21.15.67:0/4131018665 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3539059288 0 0) 0x7ff14c0d2250 con 0x7ff14c0b6170 2024-03-24T00:41:27.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.542+0000 7ff150a1f640 1 -- 172.21.15.67:0/4131018665 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff134003520 con 0x7ff14c0b6170 2024-03-24T00:41:27.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.543+0000 7ff150a1f640 1 -- 172.21.15.67:0/4131018665 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2762719139 0 0) 0x7ff14c0d2530 con 0x7ff14c15e280 2024-03-24T00:41:27.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.543+0000 7ff150a1f640 1 -- 172.21.15.67:0/4131018665 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff14c0d2250 con 0x7ff14c15e280 2024-03-24T00:41:27.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.543+0000 7ff150a1f640 1 -- 172.21.15.67:0/4131018665 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2800485695 0 0) 0x7ff134003520 con 0x7ff14c0b6170 2024-03-24T00:41:27.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.543+0000 7ff150a1f640 1 -- 172.21.15.67:0/4131018665 >> v2:172.21.15.72:3300/0 conn(0x7ff14c15dea0 msgr2=0x7ff14c0d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:27.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.543+0000 7ff150a1f640 1 --2- 172.21.15.67:0/4131018665 >> v2:172.21.15.72:3300/0 conn(0x7ff14c15dea0 0x7ff14c0d33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.543+0000 7ff150a1f640 1 -- 172.21.15.67:0/4131018665 >> v1:172.21.15.72:6789/0 conn(0x7ff14c15e280 legacy=0x7ff14c0b5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:27.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.543+0000 7ff150a1f640 1 -- 172.21.15.67:0/4131018665 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff14c0d2af0 con 0x7ff14c0b6170 2024-03-24T00:41:27.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.543+0000 7ff150a1f640 1 -- 172.21.15.67:0/4131018665 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7ff144001670 con 0x7ff14c0b6170 2024-03-24T00:41:27.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.543+0000 7ff151a21640 1 --2- 172.21.15.67:0/4131018665 >> v2:172.21.15.72:3300/0 conn(0x7ff14c15dea0 0x7ff14c0d33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:41:27.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.543+0000 7ff150a1f640 1 -- 172.21.15.67:0/4131018665 >> v1:172.21.15.67:6789/0 conn(0x7ff14c0b6170 legacy=0x7ff14c0b70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:27.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.543+0000 7ff150a1f640 1 --2- 172.21.15.67:0/4131018665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff134003b40 0x7ff134003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:27.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.543+0000 7ff150a1f640 1 --2- 172.21.15.67:0/4131018665 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff134004800 0x7ff134004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:27.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.543+0000 7ff150a1f640 1 --2- 172.21.15.67:0/4131018665 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1340051c0 0x7ff13400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:27.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.543+0000 7ff150a1f640 1 -- 172.21.15.67:0/4131018665 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c0d2810 con 0x7ff134003b40 2024-03-24T00:41:27.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.543+0000 7ff150a1f640 1 -- 172.21.15.67:0/4131018665 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff13400c0f0 con 0x7ff1340051c0 2024-03-24T00:41:27.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.543+0000 7ff150a1f640 1 -- 172.21.15.67:0/4131018665 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff13400c2f0 con 0x7ff134004800 2024-03-24T00:41:27.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.543+0000 7ff151a21640 1 --2- 172.21.15.67:0/4131018665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff134003b40 0x7ff134003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:27.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.543+0000 7ff152222640 1 --2- 172.21.15.67:0/4131018665 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1340051c0 0x7ff13400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:27.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.543+0000 7ff151220640 1 --2- 172.21.15.67:0/4131018665 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff134004800 0x7ff134004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:27.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.543+0000 7ff151a21640 1 -- 172.21.15.67:0/4131018665 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff134004800 msgr2=0x7ff134004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:27.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.543+0000 7ff151a21640 1 --2- 172.21.15.67:0/4131018665 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff134004800 0x7ff134004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.543+0000 7ff151a21640 1 -- 172.21.15.67:0/4131018665 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1340051c0 msgr2=0x7ff13400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:27.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.543+0000 7ff151a21640 1 --2- 172.21.15.67:0/4131018665 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1340051c0 0x7ff13400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.543+0000 7ff152222640 1 -- 172.21.15.67:0/4131018665 reap_dead start 2024-03-24T00:41:27.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.543+0000 7ff151a21640 1 -- 172.21.15.67:0/4131018665 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff134003520 con 0x7ff134003b40 2024-03-24T00:41:27.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.543+0000 7ff151220640 1 --2- 172.21.15.67:0/4131018665 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff134004800 0x7ff134004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:41:27.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.544+0000 7ff151a21640 1 --2- 172.21.15.67:0/4131018665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff134003b40 0x7ff134003f40 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7ff13c00e350 tx=0x7ff134005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4524600319a69763 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:27.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.544+0000 7ff150a1f640 1 -- 172.21.15.67:0/4131018665 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff13c039070 con 0x7ff134003b40 2024-03-24T00:41:27.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.544+0000 7ff150a1f640 1 -- 172.21.15.67:0/4131018665 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff13c02cd70 con 0x7ff134003b40 2024-03-24T00:41:27.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.544+0000 7ff150a1f640 1 -- 172.21.15.67:0/4131018665 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff13c009de0 con 0x7ff134003b40 2024-03-24T00:41:27.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.545+0000 7ff153c85640 1 -- 172.21.15.67:0/4131018665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff134003b40 msgr2=0x7ff134003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:27.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.545+0000 7ff153c85640 1 --2- 172.21.15.67:0/4131018665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff134003b40 0x7ff134003f40 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7ff13c00e350 tx=0x7ff134005830 comp rx=0 tx=0).stop 2024-03-24T00:41:27.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.545+0000 7ff153c85640 1 -- 172.21.15.67:0/4131018665 shutdown_connections 2024-03-24T00:41:27.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.545+0000 7ff153c85640 1 --2- 172.21.15.67:0/4131018665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff134003b40 0x7ff134003f40 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.545+0000 7ff153c85640 1 -- 172.21.15.67:0/4131018665 >> 172.21.15.67:0/4131018665 conn(0x7ff14c072da0 msgr2=0x7ff14c0d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:27.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.545+0000 7ff153c85640 1 -- 172.21.15.67:0/4131018665 shutdown_connections 2024-03-24T00:41:27.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.545+0000 7ff153c85640 1 -- 172.21.15.67:0/4131018665 wait complete. 2024-03-24T00:41:27.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.546+0000 7ff153c85640 1 Processor -- start 2024-03-24T00:41:27.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.546+0000 7ff153c85640 1 -- start start 2024-03-24T00:41:27.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.547+0000 7ff153c85640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff134003b40 0x7ff14c0cb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:27.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.547+0000 7ff153c85640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff14c15a100 0x7ff14c0cc730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:27.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.547+0000 7ff153c85640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff14c159860 0x7ff14c0cd710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:27.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.547+0000 7ff153c85640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c0b5df0 con 0x7ff14c15a100 2024-03-24T00:41:27.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.547+0000 7ff153c85640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c0b5c70 con 0x7ff14c159860 2024-03-24T00:41:27.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.547+0000 7ff153c85640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c0b5f70 con 0x7ff134003b40 2024-03-24T00:41:27.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.547+0000 7ff151220640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff14c15a100 0x7ff14c0cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:27.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.547+0000 7ff151220640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff14c15a100 0x7ff14c0cc730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37514/0 (socket says 172.21.15.67:37514) 2024-03-24T00:41:27.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.547+0000 7ff151220640 1 -- 172.21.15.67:0/678988073 learned_addr learned my addr 172.21.15.67:0/678988073 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:27.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.547+0000 7ff151a21640 1 --2- 172.21.15.67:0/678988073 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff134003b40 0x7ff14c0cb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:27.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.547+0000 7ff152222640 1 --2- 172.21.15.67:0/678988073 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff14c159860 0x7ff14c0cd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:27.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.547+0000 7ff151220640 1 -- 172.21.15.67:0/678988073 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff134003b40 msgr2=0x7ff14c0cb0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:27.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.547+0000 7ff151220640 1 --2- 172.21.15.67:0/678988073 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff134003b40 0x7ff14c0cb0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.547+0000 7ff151220640 1 -- 172.21.15.67:0/678988073 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff14c159860 msgr2=0x7ff14c0cd710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:27.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.547+0000 7ff151220640 1 --2- 172.21.15.67:0/678988073 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff14c159860 0x7ff14c0cd710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.547+0000 7ff151220640 1 -- 172.21.15.67:0/678988073 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff14c0cb690 con 0x7ff14c15a100 2024-03-24T00:41:27.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.548+0000 7ff151220640 1 --2- 172.21.15.67:0/678988073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff14c15a100 0x7ff14c0cc730 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7ff14000e4d0 tx=0x7ff14000e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:27.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.548+0000 7ff14a7fc640 1 -- 172.21.15.67:0/678988073 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1400052e0 con 0x7ff14c15a100 2024-03-24T00:41:27.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.548+0000 7ff14a7fc640 1 -- 172.21.15.67:0/678988073 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff140005480 con 0x7ff14c15a100 2024-03-24T00:41:27.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.548+0000 7ff153c85640 1 -- 172.21.15.67:0/678988073 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff14c0cb970 con 0x7ff14c15a100 2024-03-24T00:41:27.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.548+0000 7ff14a7fc640 1 -- 172.21.15.67:0/678988073 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff140007020 con 0x7ff14c15a100 2024-03-24T00:41:27.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.548+0000 7ff153c85640 1 -- 172.21.15.67:0/678988073 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff14c0cbf30 con 0x7ff14c15a100 2024-03-24T00:41:27.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.549+0000 7ff153c85640 1 -- 172.21.15.67:0/678988073 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff14c04f400 con 0x7ff14c15a100 2024-03-24T00:41:27.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.549+0000 7ff14a7fc640 1 -- 172.21.15.67:0/678988073 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff14001a020 con 0x7ff14c15a100 2024-03-24T00:41:27.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.550+0000 7ff14a7fc640 1 --2- 172.21.15.67:0/678988073 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff13806bbd0 0x7ff13806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:27.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.550+0000 7ff14a7fc640 1 -- 172.21.15.67:0/678988073 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff14008a090 con 0x7ff14c15a100 2024-03-24T00:41:27.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.550+0000 7ff151a21640 1 --2- 172.21.15.67:0/678988073 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff13806bbd0 0x7ff13806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:27.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.551+0000 7ff151a21640 1 --2- 172.21.15.67:0/678988073 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff13806bbd0 0x7ff13806e090 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ff13c036000 tx=0x7ff13c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:27.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.554+0000 7ff14a7fc640 1 -- 172.21.15.67:0/678988073 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff140058430 con 0x7ff14c15a100 2024-03-24T00:41:27.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.734+0000 7ff153c85640 1 -- 172.21.15.67:0/678988073 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff14c06b990 con 0x7ff14c15a100 2024-03-24T00:41:27.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.734+0000 7ff14a7fc640 1 -- 172.21.15.67:0/678988073 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v50) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff14005c4d0 con 0x7ff14c15a100 2024-03-24T00:41:27.735 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-24T00:41:27.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.736+0000 7ff153c85640 1 -- 172.21.15.67:0/678988073 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff13806bbd0 msgr2=0x7ff13806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:27.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.736+0000 7ff153c85640 1 --2- 172.21.15.67:0/678988073 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff13806bbd0 0x7ff13806e090 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ff13c036000 tx=0x7ff13c02e000 comp rx=0 tx=0).stop 2024-03-24T00:41:27.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.736+0000 7ff153c85640 1 -- 172.21.15.67:0/678988073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff14c15a100 msgr2=0x7ff14c0cc730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:27.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.736+0000 7ff153c85640 1 --2- 172.21.15.67:0/678988073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff14c15a100 0x7ff14c0cc730 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7ff14000e4d0 tx=0x7ff14000e990 comp rx=0 tx=0).stop 2024-03-24T00:41:27.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.736+0000 7ff153c85640 1 -- 172.21.15.67:0/678988073 shutdown_connections 2024-03-24T00:41:27.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.736+0000 7ff153c85640 1 --2- 172.21.15.67:0/678988073 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff13806bbd0 0x7ff13806e090 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.736+0000 7ff153c85640 1 --2- 172.21.15.67:0/678988073 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff14c159860 0x7ff14c0cd710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.736+0000 7ff153c85640 1 --2- 172.21.15.67:0/678988073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff14c15a100 0x7ff14c0cc730 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.736+0000 7ff153c85640 1 --2- 172.21.15.67:0/678988073 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff134003b40 0x7ff14c0cb0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.736+0000 7ff153c85640 1 -- 172.21.15.67:0/678988073 >> 172.21.15.67:0/678988073 conn(0x7ff14c072da0 msgr2=0x7ff14c0ce510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:27.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.736+0000 7ff153c85640 1 -- 172.21.15.67:0/678988073 shutdown_connections 2024-03-24T00:41:27.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.736+0000 7ff153c85640 1 -- 172.21.15.67:0/678988073 wait complete. 2024-03-24T00:41:27.748 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:41:27.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.852+0000 7f3c22c97640 1 Processor -- start 2024-03-24T00:41:27.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.852+0000 7f3c22c97640 1 -- start start 2024-03-24T00:41:27.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.853+0000 7f3c22c97640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3c1c06eb30 0x7f3c1c0ad140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:27.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.853+0000 7f3c22c97640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c1c0b6150 con 0x7f3c1c0b51d0 2024-03-24T00:41:27.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.853+0000 7f3c22c97640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c1c06ba50 con 0x7f3c1c15de30 2024-03-24T00:41:27.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.853+0000 7f3c22c97640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c1c06c380 con 0x7f3c1c06eb30 2024-03-24T00:41:27.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.853+0000 7f3c1bfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3c1c06eb30 0x7f3c1c0ad140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:27.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.853+0000 7f3c1bfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3c1c06eb30 0x7f3c1c0ad140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37518/0 (socket says 172.21.15.67:37518) 2024-03-24T00:41:27.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.853+0000 7f3c1bfff640 1 -- 172.21.15.67:0/3706745043 learned_addr learned my addr 172.21.15.67:0/3706745043 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:27.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.853+0000 7f3c21234640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3c1c0b51d0 0x7f3c1c0ad680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33164/0 (socket says 172.21.15.67:33164) 2024-03-24T00:41:27.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.853+0000 7f3c1bfff640 1 -- 172.21.15.67:0/3706745043 >> v1:172.21.15.72:6789/0 conn(0x7f3c1c15de30 legacy=0x7f3c1c06e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:27.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.853+0000 7f3c1bfff640 1 -- 172.21.15.67:0/3706745043 >> v1:172.21.15.67:6789/0 conn(0x7f3c1c0b51d0 legacy=0x7f3c1c0ad680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:27.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.853+0000 7f3c1bfff640 1 -- 172.21.15.67:0/3706745043 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c1c06bd30 con 0x7f3c1c06eb30 2024-03-24T00:41:27.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.853+0000 7f3c1b7fe640 1 -- 172.21.15.67:0/3706745043 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3508995874 0 0) 0x7f3c1c0b6150 con 0x7f3c1c0b51d0 2024-03-24T00:41:27.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.854+0000 7f3c1bfff640 1 --2- 172.21.15.67:0/3706745043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c1c06eb30 0x7f3c1c0ad140 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c06c500 tx=0x7f3c0802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56b1e9f09a25108b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:27.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.854+0000 7f3c1b7fe640 1 -- 172.21.15.67:0/3706745043 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c08034070 con 0x7f3c1c06eb30 2024-03-24T00:41:27.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.854+0000 7f3c1b7fe640 1 -- 172.21.15.67:0/3706745043 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c08030840 con 0x7f3c1c06eb30 2024-03-24T00:41:27.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.854+0000 7f3c1b7fe640 1 -- 172.21.15.67:0/3706745043 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c08030c10 con 0x7f3c1c06eb30 2024-03-24T00:41:27.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.854+0000 7f3c22c97640 1 -- 172.21.15.67:0/3706745043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c1c06eb30 msgr2=0x7f3c1c0ad140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:27.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.854+0000 7f3c22c97640 1 --2- 172.21.15.67:0/3706745043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c1c06eb30 0x7f3c1c0ad140 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c06c500 tx=0x7f3c0802b980 comp rx=0 tx=0).stop 2024-03-24T00:41:27.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.854+0000 7f3c22c97640 1 -- 172.21.15.67:0/3706745043 shutdown_connections 2024-03-24T00:41:27.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.854+0000 7f3c22c97640 1 --2- 172.21.15.67:0/3706745043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c1c06eb30 0x7f3c1c0ad140 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.854+0000 7f3c22c97640 1 -- 172.21.15.67:0/3706745043 >> 172.21.15.67:0/3706745043 conn(0x7f3c1c099a30 msgr2=0x7f3c1c0b5680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:27.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.854+0000 7f3c22c97640 1 -- 172.21.15.67:0/3706745043 shutdown_connections 2024-03-24T00:41:27.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.855+0000 7f3c22c97640 1 -- 172.21.15.67:0/3706745043 wait complete. 2024-03-24T00:41:27.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.856+0000 7f3c22c97640 1 Processor -- start 2024-03-24T00:41:27.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.856+0000 7f3c22c97640 1 -- start start 2024-03-24T00:41:27.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.856+0000 7f3c22c97640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3c1c06eb30 0x7f3c1c154670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:27.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.856+0000 7f3c22c97640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3c1c0b51d0 0x7f3c1c1f7030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:27.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.856+0000 7f3c22c97640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c1c15de30 0x7f3c1c1f7570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:27.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.856+0000 7f3c22c97640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1c15e210 con 0x7f3c1c15de30 2024-03-24T00:41:27.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.856+0000 7f3c22c97640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1c06c810 con 0x7f3c1c06eb30 2024-03-24T00:41:27.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.856+0000 7f3c22c97640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1c154ee0 con 0x7f3c1c0b51d0 2024-03-24T00:41:27.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.857+0000 7f3c21234640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c1c15de30 0x7f3c1c1f7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:27.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.857+0000 7f3c21234640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c1c15de30 0x7f3c1c1f7570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37524/0 (socket says 172.21.15.67:37524) 2024-03-24T00:41:27.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.857+0000 7f3c21234640 1 -- 172.21.15.67:0/2527110369 learned_addr learned my addr 172.21.15.67:0/2527110369 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:27.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.857+0000 7f3c20a33640 1 --2- 172.21.15.67:0/2527110369 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3c1c06eb30 0x7f3c1c154670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:27.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.857+0000 7f3c1bfff640 1 --2- 172.21.15.67:0/2527110369 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3c1c0b51d0 0x7f3c1c1f7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:27.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.857+0000 7f3c21234640 1 -- 172.21.15.67:0/2527110369 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3c1c0b51d0 msgr2=0x7f3c1c1f7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:27.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.857+0000 7f3c21234640 1 --2- 172.21.15.67:0/2527110369 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3c1c0b51d0 0x7f3c1c1f7030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.857+0000 7f3c21234640 1 -- 172.21.15.67:0/2527110369 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3c1c06eb30 msgr2=0x7f3c1c154670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:27.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.857+0000 7f3c21234640 1 --2- 172.21.15.67:0/2527110369 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3c1c06eb30 0x7f3c1c154670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:27.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.857+0000 7f3c21234640 1 -- 172.21.15.67:0/2527110369 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c1c1551c0 con 0x7f3c1c15de30 2024-03-24T00:41:27.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.857+0000 7f3c20a33640 1 --2- 172.21.15.67:0/2527110369 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3c1c06eb30 0x7f3c1c154670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:41:27.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.857+0000 7f3c1bfff640 1 --2- 172.21.15.67:0/2527110369 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3c1c0b51d0 0x7f3c1c1f7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:41:27.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.857+0000 7f3c21234640 1 --2- 172.21.15.67:0/2527110369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c1c15de30 0x7f3c1c1f7570 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f3c1400c7a0 tx=0x7f3c1400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:27.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.857+0000 7f3c197fa640 1 -- 172.21.15.67:0/2527110369 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c14018070 con 0x7f3c1c15de30 2024-03-24T00:41:27.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.857+0000 7f3c197fa640 1 -- 172.21.15.67:0/2527110369 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c1400a640 con 0x7f3c1c15de30 2024-03-24T00:41:27.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.857+0000 7f3c22c97640 1 -- 172.21.15.67:0/2527110369 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c1c1554a0 con 0x7f3c1c15de30 2024-03-24T00:41:27.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.857+0000 7f3c197fa640 1 -- 172.21.15.67:0/2527110369 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c1400aa70 con 0x7f3c1c15de30 2024-03-24T00:41:27.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.857+0000 7f3c22c97640 1 -- 172.21.15.67:0/2527110369 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c1c155a60 con 0x7f3c1c15de30 2024-03-24T00:41:27.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.859+0000 7f3c22c97640 1 -- 172.21.15.67:0/2527110369 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c1c15e390 con 0x7f3c1c15de30 2024-03-24T00:41:27.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.859+0000 7f3c197fa640 1 -- 172.21.15.67:0/2527110369 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3c1401b070 con 0x7f3c1c15de30 2024-03-24T00:41:27.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.859+0000 7f3c197fa640 1 --2- 172.21.15.67:0/2527110369 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3c0006bbd0 0x7f3c0006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:27.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.859+0000 7f3c197fa640 1 -- 172.21.15.67:0/2527110369 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3c1408a190 con 0x7f3c1c15de30 2024-03-24T00:41:27.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.859+0000 7f3c20a33640 1 --2- 172.21.15.67:0/2527110369 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3c0006bbd0 0x7f3c0006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:27.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.860+0000 7f3c20a33640 1 --2- 172.21.15.67:0/2527110369 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3c0006bbd0 0x7f3c0006e090 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c0cc5e0 tx=0x7f3c10009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:27.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:27.863+0000 7f3c197fa640 1 -- 172.21.15.67:0/2527110369 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3c1c15e390 con 0x7f3c1c15de30 2024-03-24T00:41:28.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.061+0000 7f3c22c97640 1 -- 172.21.15.67:0/2527110369 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3c1c04f400 con 0x7f3c1c15de30 2024-03-24T00:41:28.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.061+0000 7f3c197fa640 1 -- 172.21.15.67:0/2527110369 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v50) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3c14058b80 con 0x7f3c1c15de30 2024-03-24T00:41:28.062 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-03-24T00:41:28.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.063+0000 7f3c22c97640 1 -- 172.21.15.67:0/2527110369 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3c0006bbd0 msgr2=0x7f3c0006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:28.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.063+0000 7f3c22c97640 1 --2- 172.21.15.67:0/2527110369 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3c0006bbd0 0x7f3c0006e090 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c0cc5e0 tx=0x7f3c10009040 comp rx=0 tx=0).stop 2024-03-24T00:41:28.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.063+0000 7f3c22c97640 1 -- 172.21.15.67:0/2527110369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c1c15de30 msgr2=0x7f3c1c1f7570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:28.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.063+0000 7f3c22c97640 1 --2- 172.21.15.67:0/2527110369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c1c15de30 0x7f3c1c1f7570 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f3c1400c7a0 tx=0x7f3c1400cc60 comp rx=0 tx=0).stop 2024-03-24T00:41:28.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.063+0000 7f3c22c97640 1 -- 172.21.15.67:0/2527110369 shutdown_connections 2024-03-24T00:41:28.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.063+0000 7f3c22c97640 1 --2- 172.21.15.67:0/2527110369 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3c0006bbd0 0x7f3c0006e090 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.063+0000 7f3c22c97640 1 --2- 172.21.15.67:0/2527110369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c1c15de30 0x7f3c1c1f7570 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.063+0000 7f3c22c97640 1 --2- 172.21.15.67:0/2527110369 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3c1c0b51d0 0x7f3c1c1f7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.063+0000 7f3c22c97640 1 --2- 172.21.15.67:0/2527110369 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3c1c06eb30 0x7f3c1c154670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.063+0000 7f3c22c97640 1 -- 172.21.15.67:0/2527110369 >> 172.21.15.67:0/2527110369 conn(0x7f3c1c099a30 msgr2=0x7f3c1c06b040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:28.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.063+0000 7f3c22c97640 1 -- 172.21.15.67:0/2527110369 shutdown_connections 2024-03-24T00:41:28.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.063+0000 7f3c22c97640 1 -- 172.21.15.67:0/2527110369 wait complete. 2024-03-24T00:41:28.074 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:41:28.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.177+0000 7f91f34b8640 1 Processor -- start 2024-03-24T00:41:28.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.177+0000 7f91f34b8640 1 -- start start 2024-03-24T00:41:28.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.178+0000 7f91f34b8640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f91ec15dec0 0x7f91ec0ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:28.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.178+0000 7f91f34b8640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f91ec15e2a0 0x7f91ec162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:28.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.178+0000 7f91f34b8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f91ec0d07e0 0x7f91ec0d0c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:28.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.178+0000 7f91f34b8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91ec0d1190 con 0x7f91ec0d07e0 2024-03-24T00:41:28.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.178+0000 7f91f34b8640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91ec0cee30 con 0x7f91ec15dec0 2024-03-24T00:41:28.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.178+0000 7f91f34b8640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91ec0d2120 con 0x7f91ec15e2a0 2024-03-24T00:41:28.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.178+0000 7f91f1a55640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f91ec0d07e0 0x7f91ec0d0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:28.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.178+0000 7f91f1a55640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f91ec0d07e0 0x7f91ec0d0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37528/0 (socket says 172.21.15.67:37528) 2024-03-24T00:41:28.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.178+0000 7f91f1a55640 1 -- 172.21.15.67:0/2714968320 learned_addr learned my addr 172.21.15.67:0/2714968320 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:28.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.179+0000 7f91f1254640 1 --2- 172.21.15.67:0/2714968320 >> v2:172.21.15.72:3300/0 conn(0x7f91ec15dec0 0x7f91ec0ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:28.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.179+0000 7f91f0a53640 1 --2- 172.21.15.67:0/2714968320 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f91ec15e2a0 0x7f91ec162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:28.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.179+0000 7f91f1a55640 1 -- 172.21.15.67:0/2714968320 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f91ec15e2a0 msgr2=0x7f91ec162630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:28.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.179+0000 7f91f1a55640 1 --2- 172.21.15.67:0/2714968320 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f91ec15e2a0 0x7f91ec162630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.179+0000 7f91f1a55640 1 -- 172.21.15.67:0/2714968320 >> v2:172.21.15.72:3300/0 conn(0x7f91ec15dec0 msgr2=0x7f91ec0ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:28.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.179+0000 7f91f1a55640 1 --2- 172.21.15.67:0/2714968320 >> v2:172.21.15.72:3300/0 conn(0x7f91ec15dec0 0x7f91ec0ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.179+0000 7f91f1a55640 1 -- 172.21.15.67:0/2714968320 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91ec0cf640 con 0x7f91ec0d07e0 2024-03-24T00:41:28.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.179+0000 7f91f0a53640 1 --2- 172.21.15.67:0/2714968320 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f91ec15e2a0 0x7f91ec162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:41:28.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.179+0000 7f91f1a55640 1 --2- 172.21.15.67:0/2714968320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91ec0d07e0 0x7f91ec0d0c50 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f91ec0d14a0 tx=0x7f91e402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6936377f2813bbe0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:28.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.180+0000 7f91e3fff640 1 -- 172.21.15.67:0/2714968320 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e402c9e0 con 0x7f91ec0d07e0 2024-03-24T00:41:28.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.180+0000 7f91e3fff640 1 -- 172.21.15.67:0/2714968320 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91e402cb80 con 0x7f91ec0d07e0 2024-03-24T00:41:28.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.180+0000 7f91e3fff640 1 -- 172.21.15.67:0/2714968320 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e40343f0 con 0x7f91ec0d07e0 2024-03-24T00:41:28.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.180+0000 7f91f34b8640 1 -- 172.21.15.67:0/2714968320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91ec0d07e0 msgr2=0x7f91ec0d0c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:28.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.180+0000 7f91f34b8640 1 --2- 172.21.15.67:0/2714968320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91ec0d07e0 0x7f91ec0d0c50 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f91ec0d14a0 tx=0x7f91e402be40 comp rx=0 tx=0).stop 2024-03-24T00:41:28.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.180+0000 7f91f34b8640 1 -- 172.21.15.67:0/2714968320 shutdown_connections 2024-03-24T00:41:28.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.180+0000 7f91f34b8640 1 --2- 172.21.15.67:0/2714968320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91ec0d07e0 0x7f91ec0d0c50 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.180+0000 7f91f34b8640 1 --2- 172.21.15.67:0/2714968320 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f91ec15e2a0 0x7f91ec162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.180+0000 7f91f34b8640 1 --2- 172.21.15.67:0/2714968320 >> v2:172.21.15.72:3300/0 conn(0x7f91ec15dec0 0x7f91ec0ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.180+0000 7f91f34b8640 1 -- 172.21.15.67:0/2714968320 >> 172.21.15.67:0/2714968320 conn(0x7f91ec06c810 msgr2=0x7f91ec06cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:28.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.180+0000 7f91f34b8640 1 -- 172.21.15.67:0/2714968320 shutdown_connections 2024-03-24T00:41:28.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.180+0000 7f91f34b8640 1 -- 172.21.15.67:0/2714968320 wait complete. 2024-03-24T00:41:28.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.181+0000 7f91f34b8640 1 Processor -- start 2024-03-24T00:41:28.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.181+0000 7f91f34b8640 1 -- start start 2024-03-24T00:41:28.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.182+0000 7f91f34b8640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f91ec0d07e0 0x7f91ec0cc0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:28.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.182+0000 7f91f34b8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91ec15dec0 0x7f91ec0ab610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:28.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.182+0000 7f91f34b8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91ec15e2a0 0x7f91ec0b3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:28.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.182+0000 7f91f34b8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91ec0b5850 con 0x7f91ec15dec0 2024-03-24T00:41:28.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.182+0000 7f91f34b8640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91ec0cf360 con 0x7f91ec15e2a0 2024-03-24T00:41:28.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.182+0000 7f91f34b8640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91ec0b59d0 con 0x7f91ec0d07e0 2024-03-24T00:41:28.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.182+0000 7f91f0a53640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91ec15dec0 0x7f91ec0ab610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:28.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.182+0000 7f91f0a53640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91ec15dec0 0x7f91ec0ab610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37540/0 (socket says 172.21.15.67:37540) 2024-03-24T00:41:28.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.182+0000 7f91f0a53640 1 -- 172.21.15.67:0/3589942894 learned_addr learned my addr 172.21.15.67:0/3589942894 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:28.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.182+0000 7f91f0a53640 1 -- 172.21.15.67:0/3589942894 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f91ec0d07e0 msgr2=0x7f91ec0cc0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:28.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.182+0000 7f91f1254640 1 --2- 172.21.15.67:0/3589942894 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f91ec0d07e0 0x7f91ec0cc0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:28.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.182+0000 7f91f1a55640 1 --2- 172.21.15.67:0/3589942894 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91ec15e2a0 0x7f91ec0b3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:28.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.182+0000 7f91f0a53640 1 --2- 172.21.15.67:0/3589942894 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f91ec0d07e0 0x7f91ec0cc0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.182+0000 7f91f0a53640 1 -- 172.21.15.67:0/3589942894 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91ec15e2a0 msgr2=0x7f91ec0b3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:28.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.182+0000 7f91f0a53640 1 --2- 172.21.15.67:0/3589942894 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91ec15e2a0 0x7f91ec0b3df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.182+0000 7f91f0a53640 1 -- 172.21.15.67:0/3589942894 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91ec0ccd80 con 0x7f91ec15dec0 2024-03-24T00:41:28.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.183+0000 7f91f0a53640 1 --2- 172.21.15.67:0/3589942894 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91ec15dec0 0x7f91ec0ab610 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f91dc00c7c0 tx=0x7f91dc00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:28.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.183+0000 7f91e1ffb640 1 -- 172.21.15.67:0/3589942894 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91dc015070 con 0x7f91ec15dec0 2024-03-24T00:41:28.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.183+0000 7f91e1ffb640 1 -- 172.21.15.67:0/3589942894 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91dc007820 con 0x7f91ec15dec0 2024-03-24T00:41:28.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.183+0000 7f91f34b8640 1 -- 172.21.15.67:0/3589942894 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91ec0cd060 con 0x7f91ec15dec0 2024-03-24T00:41:28.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.183+0000 7f91e1ffb640 1 -- 172.21.15.67:0/3589942894 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91dc007bf0 con 0x7f91ec15dec0 2024-03-24T00:41:28.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.183+0000 7f91f34b8640 1 -- 172.21.15.67:0/3589942894 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91ec0cd620 con 0x7f91ec15dec0 2024-03-24T00:41:28.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.184+0000 7f91f34b8640 1 -- 172.21.15.67:0/3589942894 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91ec04f400 con 0x7f91ec15dec0 2024-03-24T00:41:28.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.184+0000 7f91e1ffb640 1 -- 172.21.15.67:0/3589942894 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f91dc018070 con 0x7f91ec15dec0 2024-03-24T00:41:28.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.185+0000 7f91e1ffb640 1 --2- 172.21.15.67:0/3589942894 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f91d006bbd0 0x7f91d006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:28.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.185+0000 7f91e1ffb640 1 -- 172.21.15.67:0/3589942894 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f91dc08aa70 con 0x7f91ec15dec0 2024-03-24T00:41:28.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.185+0000 7f91f1254640 1 --2- 172.21.15.67:0/3589942894 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f91d006bbd0 0x7f91d006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:28.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.186+0000 7f91f1254640 1 --2- 172.21.15.67:0/3589942894 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f91d006bbd0 0x7f91d006e090 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f91e8006fd0 tx=0x7f91e8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:28.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.193+0000 7f91e1ffb640 1 -- 172.21.15.67:0/3589942894 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f91dc0593d0 con 0x7f91ec15dec0 2024-03-24T00:41:28.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.417+0000 7f91f34b8640 1 -- 172.21.15.67:0/3589942894 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91ec0d1aa0 con 0x7f91ec15dec0 2024-03-24T00:41:28.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.418+0000 7f91e1ffb640 1 -- 172.21.15.67:0/3589942894 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f91dc058cd0 con 0x7f91ec15dec0 2024-03-24T00:41:28.419 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:41:28.419 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5360,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3312887486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":3312887486},{"type":"v1","addr":"172.21.15.67:6835","nonce":3312887486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5821,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/4171866637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":4171866637},{"type":"v1","addr":"172.21.15.72:6836","nonce":4171866637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5824,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/4106412452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":4106412452},{"type":"v1","addr":"172.21.15.72:6837","nonce":4106412452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:40:39.649631+0000","modified":"2024-03-24T00:40:42.657888+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5853},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5853":{"gid":5853,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6834/2741818214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2741818214},{"type":"v1","addr":"172.21.15.67:6834","nonce":2741818214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5853,"qdb_cluster":[5853]},"id":3}]} 2024-03-24T00:41:28.419 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 26 2024-03-24T00:41:28.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.420+0000 7f91f34b8640 1 -- 172.21.15.67:0/3589942894 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f91d006bbd0 msgr2=0x7f91d006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:28.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.420+0000 7f91f34b8640 1 --2- 172.21.15.67:0/3589942894 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f91d006bbd0 0x7f91d006e090 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f91e8006fd0 tx=0x7f91e8008040 comp rx=0 tx=0).stop 2024-03-24T00:41:28.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.420+0000 7f91f34b8640 1 -- 172.21.15.67:0/3589942894 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91ec15dec0 msgr2=0x7f91ec0ab610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:28.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.420+0000 7f91f34b8640 1 --2- 172.21.15.67:0/3589942894 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91ec15dec0 0x7f91ec0ab610 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f91dc00c7c0 tx=0x7f91dc00cc80 comp rx=0 tx=0).stop 2024-03-24T00:41:28.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.420+0000 7f91f34b8640 1 -- 172.21.15.67:0/3589942894 shutdown_connections 2024-03-24T00:41:28.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.420+0000 7f91f34b8640 1 --2- 172.21.15.67:0/3589942894 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f91d006bbd0 0x7f91d006e090 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.420+0000 7f91f34b8640 1 --2- 172.21.15.67:0/3589942894 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91ec15e2a0 0x7f91ec0b3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.420+0000 7f91f34b8640 1 --2- 172.21.15.67:0/3589942894 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91ec15dec0 0x7f91ec0ab610 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.420+0000 7f91f34b8640 1 --2- 172.21.15.67:0/3589942894 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f91ec0d07e0 0x7f91ec0cc0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.420+0000 7f91f34b8640 1 -- 172.21.15.67:0/3589942894 >> 172.21.15.67:0/3589942894 conn(0x7f91ec06c810 msgr2=0x7f91ec15cfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:28.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.420+0000 7f91f34b8640 1 -- 172.21.15.67:0/3589942894 shutdown_connections 2024-03-24T00:41:28.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.420+0000 7f91f34b8640 1 -- 172.21.15.67:0/3589942894 wait complete. 2024-03-24T00:41:28.431 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:41:28.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.566+0000 7efe07861640 1 Processor -- start 2024-03-24T00:41:28.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.567+0000 7efe07861640 1 -- start start 2024-03-24T00:41:28.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.567+0000 7efe07861640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7efe0015f410 0x7efe000cf7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:28.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.567+0000 7efe07861640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efe000d0940 0x7efe000cfd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:28.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.567+0000 7efe07861640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe000cdb50 con 0x7efe00099d70 2024-03-24T00:41:28.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.567+0000 7efe07861640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe000cd2b0 con 0x7efe000d0940 2024-03-24T00:41:28.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.567+0000 7efe07861640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe000cd590 con 0x7efe0015f410 2024-03-24T00:41:28.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.568+0000 7efe04dfc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efe000d0940 0x7efe000cfd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:28.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.568+0000 7efe04dfc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efe000d0940 0x7efe000cfd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37544/0 (socket says 172.21.15.67:37544) 2024-03-24T00:41:28.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.568+0000 7efe04dfc640 1 -- 172.21.15.67:0/1098153987 learned_addr learned my addr 172.21.15.67:0/1098153987 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:28.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.568+0000 7efe055fd640 1 --2- 172.21.15.67:0/1098153987 >> v2:172.21.15.72:3300/0 conn(0x7efe0015f410 0x7efe000cf7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:28.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.568+0000 7efe04dfc640 1 -- 172.21.15.67:0/1098153987 >> v2:172.21.15.72:3300/0 conn(0x7efe0015f410 msgr2=0x7efe000cf7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:28.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.568+0000 7efe04dfc640 1 --2- 172.21.15.67:0/1098153987 >> v2:172.21.15.72:3300/0 conn(0x7efe0015f410 0x7efe000cf7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.568+0000 7efe04dfc640 1 -- 172.21.15.67:0/1098153987 >> v1:172.21.15.72:6789/0 conn(0x7efe00099d70 legacy=0x7efe0015fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:28.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.568+0000 7efe04dfc640 1 -- 172.21.15.67:0/1098153987 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe000cd870 con 0x7efe000d0940 2024-03-24T00:41:28.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.569+0000 7efe04dfc640 1 --2- 172.21.15.67:0/1098153987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe000d0940 0x7efe000cfd00 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7efe000733e0 tx=0x7efdf002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=839f69f79f509a99 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:28.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.569+0000 7efdf7fff640 1 -- 172.21.15.67:0/1098153987 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf0034070 con 0x7efe000d0940 2024-03-24T00:41:28.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.569+0000 7efdf7fff640 1 -- 172.21.15.67:0/1098153987 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdf0030860 con 0x7efe000d0940 2024-03-24T00:41:28.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.569+0000 7efdf7fff640 1 -- 172.21.15.67:0/1098153987 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf0030c30 con 0x7efe000d0940 2024-03-24T00:41:28.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.569+0000 7efe07861640 1 -- 172.21.15.67:0/1098153987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe000d0940 msgr2=0x7efe000cfd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:28.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.569+0000 7efe07861640 1 --2- 172.21.15.67:0/1098153987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe000d0940 0x7efe000cfd00 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7efe000733e0 tx=0x7efdf002b9a0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.569+0000 7efe07861640 1 -- 172.21.15.67:0/1098153987 shutdown_connections 2024-03-24T00:41:28.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.569+0000 7efe07861640 1 --2- 172.21.15.67:0/1098153987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe000d0940 0x7efe000cfd00 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.569+0000 7efe07861640 1 --2- 172.21.15.67:0/1098153987 >> v2:172.21.15.72:3300/0 conn(0x7efe0015f410 0x7efe000cf7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.569+0000 7efe07861640 1 -- 172.21.15.67:0/1098153987 >> 172.21.15.67:0/1098153987 conn(0x7efe000b36e0 msgr2=0x7efe000b3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:28.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.569+0000 7efe07861640 1 -- 172.21.15.67:0/1098153987 shutdown_connections 2024-03-24T00:41:28.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.569+0000 7efe07861640 1 -- 172.21.15.67:0/1098153987 wait complete. 2024-03-24T00:41:28.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.571+0000 7efe07861640 1 Processor -- start 2024-03-24T00:41:28.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.571+0000 7efe07861640 1 -- start start 2024-03-24T00:41:28.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.571+0000 7efe07861640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe00099d70 0x7efe001585d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:28.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.571+0000 7efe07861640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efe0015f410 0x7efe00159c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:28.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.571+0000 7efe07861640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efe00153a30 0x7efe000c9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:28.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.571+0000 7efe07861640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe0015f7f0 con 0x7efe00099d70 2024-03-24T00:41:28.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.571+0000 7efe07861640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe000d0240 con 0x7efe00153a30 2024-03-24T00:41:28.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.571+0000 7efe07861640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe00158bb0 con 0x7efe0015f410 2024-03-24T00:41:28.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.571+0000 7efe055fd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe00099d70 0x7efe001585d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:28.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.571+0000 7efe055fd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe00099d70 0x7efe001585d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37552/0 (socket says 172.21.15.67:37552) 2024-03-24T00:41:28.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.571+0000 7efe055fd640 1 -- 172.21.15.67:0/1408072150 learned_addr learned my addr 172.21.15.67:0/1408072150 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:28.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.571+0000 7efe04dfc640 1 --2- 172.21.15.67:0/1408072150 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efe0015f410 0x7efe00159c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:28.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.571+0000 7efe05dfe640 1 --2- 172.21.15.67:0/1408072150 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efe00153a30 0x7efe000c9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:28.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.572+0000 7efe055fd640 1 -- 172.21.15.67:0/1408072150 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efe0015f410 msgr2=0x7efe00159c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:28.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.572+0000 7efe055fd640 1 --2- 172.21.15.67:0/1408072150 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efe0015f410 0x7efe00159c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.572+0000 7efe055fd640 1 -- 172.21.15.67:0/1408072150 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efe00153a30 msgr2=0x7efe000c9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:28.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.572+0000 7efe055fd640 1 --2- 172.21.15.67:0/1408072150 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efe00153a30 0x7efe000c9fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.572+0000 7efe055fd640 1 -- 172.21.15.67:0/1408072150 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe00158e90 con 0x7efe00099d70 2024-03-24T00:41:28.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.572+0000 7efe055fd640 1 --2- 172.21.15.67:0/1408072150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe00099d70 0x7efe001585d0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7efdfc00d5f0 tx=0x7efdfc00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:28.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.572+0000 7efdf5ffb640 1 -- 172.21.15.67:0/1408072150 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdfc0125f0 con 0x7efe00099d70 2024-03-24T00:41:28.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.572+0000 7efdf5ffb640 1 -- 172.21.15.67:0/1408072150 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdfc019070 con 0x7efe00099d70 2024-03-24T00:41:28.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.572+0000 7efe07861640 1 -- 172.21.15.67:0/1408072150 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe00159170 con 0x7efe00099d70 2024-03-24T00:41:28.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.572+0000 7efdf5ffb640 1 -- 172.21.15.67:0/1408072150 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdfc0129d0 con 0x7efe00099d70 2024-03-24T00:41:28.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.572+0000 7efe07861640 1 -- 172.21.15.67:0/1408072150 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe000ca500 con 0x7efe00099d70 2024-03-24T00:41:28.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.573+0000 7efe07861640 1 -- 172.21.15.67:0/1408072150 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe0004f400 con 0x7efe00099d70 2024-03-24T00:41:28.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.574+0000 7efdf5ffb640 1 -- 172.21.15.67:0/1408072150 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7efdfc012b70 con 0x7efe00099d70 2024-03-24T00:41:28.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.574+0000 7efdf5ffb640 1 --2- 172.21.15.67:0/1408072150 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7efde406bbd0 0x7efde406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:28.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.574+0000 7efdf5ffb640 1 -- 172.21.15.67:0/1408072150 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6101+0+0 (secure 0 0 0) 0x7efdfc08af50 con 0x7efe00099d70 2024-03-24T00:41:28.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.574+0000 7efe04dfc640 1 --2- 172.21.15.67:0/1408072150 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7efde406bbd0 0x7efde406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:28.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.575+0000 7efe04dfc640 1 --2- 172.21.15.67:0/1408072150 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7efde406bbd0 0x7efde406e090 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7efdf0036000 tx=0x7efdf0008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:28.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.578+0000 7efdf5ffb640 1 -- 172.21.15.67:0/1408072150 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7efdfc0591d0 con 0x7efe00099d70 2024-03-24T00:41:28.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.764+0000 7efe07861640 1 -- 172.21.15.67:0/1408072150 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efe000d19d0 con 0x7efe00099d70 2024-03-24T00:41:28.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.765+0000 7efdf5ffb640 1 -- 172.21.15.67:0/1408072150 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) v1 ==== 74+0+16882 (secure 0 0 0) 0x7efdfc05d270 con 0x7efe00099d70 2024-03-24T00:41:28.766 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:41:28.766 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":50,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:40:41.288503+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:40:37.927587+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-24T00:40:38.616521+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:41:28.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.768+0000 7efe07861640 1 -- 172.21.15.67:0/1408072150 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7efde406bbd0 msgr2=0x7efde406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:28.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.768+0000 7efe07861640 1 --2- 172.21.15.67:0/1408072150 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7efde406bbd0 0x7efde406e090 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7efdf0036000 tx=0x7efdf0008030 comp rx=0 tx=0).stop 2024-03-24T00:41:28.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.768+0000 7efe07861640 1 -- 172.21.15.67:0/1408072150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe00099d70 msgr2=0x7efe001585d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:28.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.768+0000 7efe07861640 1 --2- 172.21.15.67:0/1408072150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe00099d70 0x7efe001585d0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7efdfc00d5f0 tx=0x7efdfc00dab0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.768+0000 7efe07861640 1 -- 172.21.15.67:0/1408072150 shutdown_connections 2024-03-24T00:41:28.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.768+0000 7efe07861640 1 --2- 172.21.15.67:0/1408072150 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7efde406bbd0 0x7efde406e090 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.768+0000 7efe07861640 1 --2- 172.21.15.67:0/1408072150 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efe00153a30 0x7efe000c9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.768+0000 7efe07861640 1 --2- 172.21.15.67:0/1408072150 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efe0015f410 0x7efe00159c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.768+0000 7efe07861640 1 --2- 172.21.15.67:0/1408072150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe00099d70 0x7efe001585d0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.768+0000 7efe07861640 1 -- 172.21.15.67:0/1408072150 >> 172.21.15.67:0/1408072150 conn(0x7efe000b36e0 msgr2=0x7efe000b1e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:28.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.768+0000 7efe07861640 1 -- 172.21.15.67:0/1408072150 shutdown_connections 2024-03-24T00:41:28.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.768+0000 7efe07861640 1 -- 172.21.15.67:0/1408072150 wait complete. 2024-03-24T00:41:28.779 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-24T00:41:28.780 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:41:28.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.883+0000 7fd208ae8640 1 Processor -- start 2024-03-24T00:41:28.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.883+0000 7fd208ae8640 1 -- start start 2024-03-24T00:41:28.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.884+0000 7fd208ae8640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd20415e2a0 0x7fd2040ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:28.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.884+0000 7fd208ae8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd2040d0ef0 0x7fd204162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:28.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.884+0000 7fd208ae8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2040cf640 con 0x7fd20415dec0 2024-03-24T00:41:28.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.884+0000 7fd208ae8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2040cee30 con 0x7fd2040d0ef0 2024-03-24T00:41:28.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.884+0000 7fd208ae8640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2040d2120 con 0x7fd20415e2a0 2024-03-24T00:41:28.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.884+0000 7fd202d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd2040d0ef0 0x7fd204162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:28.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.884+0000 7fd20259c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd20415dec0 0x7fd2040d07e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33178/0 (socket says 172.21.15.67:33178) 2024-03-24T00:41:28.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.884+0000 7fd202d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd2040d0ef0 0x7fd204162630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37558/0 (socket says 172.21.15.67:37558) 2024-03-24T00:41:28.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.884+0000 7fd20259c640 1 -- 172.21.15.67:0/2545212909 learned_addr learned my addr 172.21.15.67:0/2545212909 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:41:28.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.884+0000 7fd201d9b640 1 --2- 172.21.15.67:0/2545212909 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd20415e2a0 0x7fd2040ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:28.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.885+0000 7fd202d9d640 1 -- 172.21.15.67:0/2545212909 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd20415e2a0 msgr2=0x7fd2040ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:28.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.885+0000 7fd202d9d640 1 --2- 172.21.15.67:0/2545212909 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd20415e2a0 0x7fd2040ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.885+0000 7fd202d9d640 1 -- 172.21.15.67:0/2545212909 >> v1:172.21.15.67:6789/0 conn(0x7fd20415dec0 legacy=0x7fd2040d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:28.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.885+0000 7fd202d9d640 1 -- 172.21.15.67:0/2545212909 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2040cf080 con 0x7fd2040d0ef0 2024-03-24T00:41:28.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.885+0000 7fd20159a640 1 -- 172.21.15.67:0/2545212909 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1322237343 0 0) 0x7fd1f4001110 con 0x7fd20415dec0 2024-03-24T00:41:28.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.885+0000 7fd202d9d640 1 --2- 172.21.15.67:0/2545212909 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2040d0ef0 0x7fd204162630 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc009d90 tx=0x7fd1fc02be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f729d5065a1db70b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:28.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.885+0000 7fd20159a640 1 -- 172.21.15.67:0/2545212909 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1fc039070 con 0x7fd2040d0ef0 2024-03-24T00:41:28.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.886+0000 7fd20159a640 1 -- 172.21.15.67:0/2545212909 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1fc02cbc0 con 0x7fd2040d0ef0 2024-03-24T00:41:28.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.886+0000 7fd20159a640 1 -- 172.21.15.67:0/2545212909 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1fc034440 con 0x7fd2040d0ef0 2024-03-24T00:41:28.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.886+0000 7fd208ae8640 1 -- 172.21.15.67:0/2545212909 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2040d0ef0 msgr2=0x7fd204162630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:28.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.886+0000 7fd208ae8640 1 --2- 172.21.15.67:0/2545212909 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2040d0ef0 0x7fd204162630 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc009d90 tx=0x7fd1fc02be10 comp rx=0 tx=0).stop 2024-03-24T00:41:28.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.886+0000 7fd208ae8640 1 -- 172.21.15.67:0/2545212909 shutdown_connections 2024-03-24T00:41:28.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.886+0000 7fd208ae8640 1 --2- 172.21.15.67:0/2545212909 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2040d0ef0 0x7fd204162630 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.886+0000 7fd208ae8640 1 --2- 172.21.15.67:0/2545212909 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd20415e2a0 0x7fd2040ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.886+0000 7fd208ae8640 1 -- 172.21.15.67:0/2545212909 >> 172.21.15.67:0/2545212909 conn(0x7fd20406c810 msgr2=0x7fd20406cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:28.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.886+0000 7fd208ae8640 1 -- 172.21.15.67:0/2545212909 shutdown_connections 2024-03-24T00:41:28.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.886+0000 7fd208ae8640 1 -- 172.21.15.67:0/2545212909 wait complete. 2024-03-24T00:41:28.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.887+0000 7fd208ae8640 1 Processor -- start 2024-03-24T00:41:28.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.887+0000 7fd208ae8640 1 -- start start 2024-03-24T00:41:28.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.888+0000 7fd208ae8640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd2040d0ef0 0x7fd2041f3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:28.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.888+0000 7fd208ae8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd20415dec0 0x7fd2040b3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:28.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.888+0000 7fd208ae8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd20415e2a0 0x7fd20415fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:28.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.888+0000 7fd208ae8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2040d2730 con 0x7fd20415dec0 2024-03-24T00:41:28.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.888+0000 7fd208ae8640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2040d25b0 con 0x7fd20415e2a0 2024-03-24T00:41:28.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.888+0000 7fd208ae8640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2040d28b0 con 0x7fd2040d0ef0 2024-03-24T00:41:28.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.888+0000 7fd201d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd20415dec0 0x7fd2040b3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:28.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.888+0000 7fd201d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd20415dec0 0x7fd2040b3810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37574/0 (socket says 172.21.15.67:37574) 2024-03-24T00:41:28.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.888+0000 7fd201d9b640 1 -- 172.21.15.67:0/1393055779 learned_addr learned my addr 172.21.15.67:0/1393055779 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:28.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.888+0000 7fd20259c640 1 --2- 172.21.15.67:0/1393055779 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd2040d0ef0 0x7fd2041f3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:28.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.889+0000 7fd202d9d640 1 --2- 172.21.15.67:0/1393055779 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd20415e2a0 0x7fd20415fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:28.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.889+0000 7fd201d9b640 1 -- 172.21.15.67:0/1393055779 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd2040d0ef0 msgr2=0x7fd2041f3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:28.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.889+0000 7fd201d9b640 1 --2- 172.21.15.67:0/1393055779 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd2040d0ef0 0x7fd2041f3b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.889+0000 7fd201d9b640 1 -- 172.21.15.67:0/1393055779 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd20415e2a0 msgr2=0x7fd20415fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:28.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.889+0000 7fd201d9b640 1 --2- 172.21.15.67:0/1393055779 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd20415e2a0 0x7fd20415fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:28.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.889+0000 7fd201d9b640 1 -- 172.21.15.67:0/1393055779 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2040b4140 con 0x7fd20415dec0 2024-03-24T00:41:28.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.889+0000 7fd201d9b640 1 --2- 172.21.15.67:0/1393055779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd20415dec0 0x7fd2040b3810 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec00d5f0 tx=0x7fd1ec00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:28.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.889+0000 7fd1faffd640 1 -- 172.21.15.67:0/1393055779 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1ec0125f0 con 0x7fd20415dec0 2024-03-24T00:41:28.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.889+0000 7fd1faffd640 1 -- 172.21.15.67:0/1393055779 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1ec019070 con 0x7fd20415dec0 2024-03-24T00:41:28.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.889+0000 7fd208ae8640 1 -- 172.21.15.67:0/1393055779 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2040b4420 con 0x7fd20415dec0 2024-03-24T00:41:28.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.889+0000 7fd1faffd640 1 -- 172.21.15.67:0/1393055779 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1ec0129d0 con 0x7fd20415dec0 2024-03-24T00:41:28.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.889+0000 7fd208ae8640 1 -- 172.21.15.67:0/1393055779 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2040b4c20 con 0x7fd20415dec0 2024-03-24T00:41:28.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.890+0000 7fd208ae8640 1 -- 172.21.15.67:0/1393055779 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd20404f400 con 0x7fd20415dec0 2024-03-24T00:41:28.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.890+0000 7fd1faffd640 1 -- 172.21.15.67:0/1393055779 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd1ec012b70 con 0x7fd20415dec0 2024-03-24T00:41:28.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.891+0000 7fd1faffd640 1 --2- 172.21.15.67:0/1393055779 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd1e406bbd0 0x7fd1e406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:28.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.891+0000 7fd1faffd640 1 -- 172.21.15.67:0/1393055779 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd1ec08afb0 con 0x7fd20415dec0 2024-03-24T00:41:28.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.891+0000 7fd20259c640 1 --2- 172.21.15.67:0/1393055779 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd1e406bbd0 0x7fd1e406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:28.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.891+0000 7fd20259c640 1 --2- 172.21.15.67:0/1393055779 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd1e406bbd0 0x7fd1e406e090 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd1f4000f60 tx=0x7fd1f4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:28.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:28.895+0000 7fd1faffd640 1 -- 172.21.15.67:0/1393055779 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd1ec059910 con 0x7fd20415dec0 2024-03-24T00:41:29.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.179+0000 7fd208ae8640 1 -- 172.21.15.67:0/1393055779 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd2040d1aa0 con 0x7fd20415dec0 2024-03-24T00:41:29.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.180+0000 7fd1faffd640 1 -- 172.21.15.67:0/1393055779 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4977 (secure 0 0 0) 0x7fd1ec059210 con 0x7fd20415dec0 2024-03-24T00:41:29.181 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:41:29.181 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5360,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3312887486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":3312887486},{"type":"v1","addr":"172.21.15.67:6835","nonce":3312887486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5821,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/4171866637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":4171866637},{"type":"v1","addr":"172.21.15.72:6836","nonce":4171866637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5824,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/4106412452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":4106412452},{"type":"v1","addr":"172.21.15.72:6837","nonce":4106412452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:40:39.649631+0000","modified":"2024-03-24T00:40:42.657888+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5853},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5853":{"gid":5853,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6834/2741818214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2741818214},{"type":"v1","addr":"172.21.15.67:6834","nonce":2741818214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5853,"qdb_cluster":[5853]},"id":3}]} 2024-03-24T00:41:29.181 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 26 2024-03-24T00:41:29.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.182+0000 7fd208ae8640 1 -- 172.21.15.67:0/1393055779 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd1e406bbd0 msgr2=0x7fd1e406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:29.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.182+0000 7fd208ae8640 1 --2- 172.21.15.67:0/1393055779 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd1e406bbd0 0x7fd1e406e090 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd1f4000f60 tx=0x7fd1f4009040 comp rx=0 tx=0).stop 2024-03-24T00:41:29.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.182+0000 7fd208ae8640 1 -- 172.21.15.67:0/1393055779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd20415dec0 msgr2=0x7fd2040b3810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:29.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.182+0000 7fd208ae8640 1 --2- 172.21.15.67:0/1393055779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd20415dec0 0x7fd2040b3810 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec00d5f0 tx=0x7fd1ec00dab0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.182+0000 7fd208ae8640 1 -- 172.21.15.67:0/1393055779 shutdown_connections 2024-03-24T00:41:29.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.182+0000 7fd208ae8640 1 --2- 172.21.15.67:0/1393055779 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd1e406bbd0 0x7fd1e406e090 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.182+0000 7fd208ae8640 1 --2- 172.21.15.67:0/1393055779 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd20415e2a0 0x7fd20415fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.182+0000 7fd208ae8640 1 --2- 172.21.15.67:0/1393055779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd20415dec0 0x7fd2040b3810 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.182+0000 7fd208ae8640 1 --2- 172.21.15.67:0/1393055779 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd2040d0ef0 0x7fd2041f3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.182+0000 7fd208ae8640 1 -- 172.21.15.67:0/1393055779 >> 172.21.15.67:0/1393055779 conn(0x7fd20406c810 msgr2=0x7fd2040ca1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:29.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.183+0000 7fd208ae8640 1 -- 172.21.15.67:0/1393055779 shutdown_connections 2024-03-24T00:41:29.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.183+0000 7fd208ae8640 1 -- 172.21.15.67:0/1393055779 wait complete. 2024-03-24T00:41:29.197 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:41:29.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.298+0000 7f3120d77640 1 Processor -- start 2024-03-24T00:41:29.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.298+0000 7f3120d77640 1 -- start start 2024-03-24T00:41:29.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.299+0000 7f3120d77640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f311c15dec0 0x7f311c0ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:29.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.299+0000 7f3120d77640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f311c15e2a0 0x7f311c162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:29.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.299+0000 7f3120d77640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f311c0d07e0 0x7f311c0d0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:29.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.299+0000 7f3120d77640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f311c0d1170 con 0x7f311c15dec0 2024-03-24T00:41:29.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.299+0000 7f3120d77640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f311c0cee30 con 0x7f311c15e2a0 2024-03-24T00:41:29.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.299+0000 7f3120d77640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f311c0d2120 con 0x7f311c0d07e0 2024-03-24T00:41:29.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.299+0000 7f311a59c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f311c15dec0 0x7f311c0ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:29.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.299+0000 7f311a59c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f311c15dec0 0x7f311c0ce630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37580/0 (socket says 172.21.15.67:37580) 2024-03-24T00:41:29.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.299+0000 7f311a59c640 1 -- 172.21.15.67:0/297846237 learned_addr learned my addr 172.21.15.67:0/297846237 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:29.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.299+0000 7f311ad9d640 1 --2- 172.21.15.67:0/297846237 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f311c0d07e0 0x7f311c0d0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:29.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.299+0000 7f3119d9b640 1 --2- 172.21.15.67:0/297846237 >> v2:172.21.15.72:3300/0 conn(0x7f311c15e2a0 0x7f311c162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:29.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.299+0000 7f311a59c640 1 -- 172.21.15.67:0/297846237 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f311c0d07e0 msgr2=0x7f311c0d0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:29.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.299+0000 7f311a59c640 1 --2- 172.21.15.67:0/297846237 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f311c0d07e0 0x7f311c0d0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.299+0000 7f311a59c640 1 -- 172.21.15.67:0/297846237 >> v2:172.21.15.72:3300/0 conn(0x7f311c15e2a0 msgr2=0x7f311c162630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:29.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.299+0000 7f311a59c640 1 --2- 172.21.15.67:0/297846237 >> v2:172.21.15.72:3300/0 conn(0x7f311c15e2a0 0x7f311c162630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.299+0000 7f311a59c640 1 -- 172.21.15.67:0/297846237 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f311c0cf640 con 0x7f311c15dec0 2024-03-24T00:41:29.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.300+0000 7f311a59c640 1 --2- 172.21.15.67:0/297846237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f311c15dec0 0x7f311c0ce630 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f311c0cede0 tx=0x7f310c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b522f22e3e0a93ef server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:29.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.300+0000 7f311959a640 1 -- 172.21.15.67:0/297846237 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310c02c9e0 con 0x7f311c15dec0 2024-03-24T00:41:29.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.300+0000 7f311959a640 1 -- 172.21.15.67:0/297846237 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f310c02cb80 con 0x7f311c15dec0 2024-03-24T00:41:29.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.300+0000 7f311959a640 1 -- 172.21.15.67:0/297846237 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310c0343f0 con 0x7f311c15dec0 2024-03-24T00:41:29.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.301+0000 7f3120d77640 1 -- 172.21.15.67:0/297846237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f311c15dec0 msgr2=0x7f311c0ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:29.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.301+0000 7f3120d77640 1 --2- 172.21.15.67:0/297846237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f311c15dec0 0x7f311c0ce630 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f311c0cede0 tx=0x7f310c02be40 comp rx=0 tx=0).stop 2024-03-24T00:41:29.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.301+0000 7f3120d77640 1 -- 172.21.15.67:0/297846237 shutdown_connections 2024-03-24T00:41:29.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.301+0000 7f3120d77640 1 --2- 172.21.15.67:0/297846237 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f311c0d07e0 0x7f311c0d0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.301+0000 7f3120d77640 1 --2- 172.21.15.67:0/297846237 >> v2:172.21.15.72:3300/0 conn(0x7f311c15e2a0 0x7f311c162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.301+0000 7f3120d77640 1 --2- 172.21.15.67:0/297846237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f311c15dec0 0x7f311c0ce630 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.301+0000 7f3120d77640 1 -- 172.21.15.67:0/297846237 >> 172.21.15.67:0/297846237 conn(0x7f311c06c810 msgr2=0x7f311c06cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:29.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.301+0000 7f3120d77640 1 -- 172.21.15.67:0/297846237 shutdown_connections 2024-03-24T00:41:29.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.301+0000 7f3120d77640 1 -- 172.21.15.67:0/297846237 wait complete. 2024-03-24T00:41:29.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.303+0000 7f3120d77640 1 Processor -- start 2024-03-24T00:41:29.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.303+0000 7f3120d77640 1 -- start start 2024-03-24T00:41:29.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.304+0000 7f3120d77640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f311c0d07e0 0x7f311c1f3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:29.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.304+0000 7f3120d77640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f311c15dec0 0x7f311c0cd330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:29.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.304+0000 7f3120d77640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f311c15e2a0 0x7f311c0cd870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:29.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.304+0000 7f3120d77640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311c0b5850 con 0x7f311c15e2a0 2024-03-24T00:41:29.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.304+0000 7f3120d77640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311c0cf360 con 0x7f311c0d07e0 2024-03-24T00:41:29.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.304+0000 7f3120d77640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f311c0b59d0 con 0x7f311c15dec0 2024-03-24T00:41:29.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.304+0000 7f311ad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f311c15e2a0 0x7f311c0cd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:29.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.305+0000 7f311ad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f311c15e2a0 0x7f311c0cd870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37592/0 (socket says 172.21.15.67:37592) 2024-03-24T00:41:29.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.305+0000 7f311ad9d640 1 -- 172.21.15.67:0/3352876280 learned_addr learned my addr 172.21.15.67:0/3352876280 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:29.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.305+0000 7f311a59c640 1 --2- 172.21.15.67:0/3352876280 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f311c0d07e0 0x7f311c1f3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:29.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.305+0000 7f3119d9b640 1 --2- 172.21.15.67:0/3352876280 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f311c15dec0 0x7f311c0cd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:29.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.305+0000 7f311ad9d640 1 -- 172.21.15.67:0/3352876280 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f311c15dec0 msgr2=0x7f311c0cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:29.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.305+0000 7f311ad9d640 1 --2- 172.21.15.67:0/3352876280 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f311c15dec0 0x7f311c0cd330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.305+0000 7f311ad9d640 1 -- 172.21.15.67:0/3352876280 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f311c0d07e0 msgr2=0x7f311c1f3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:29.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.305+0000 7f311ad9d640 1 --2- 172.21.15.67:0/3352876280 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f311c0d07e0 0x7f311c1f3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.305+0000 7f311ad9d640 1 -- 172.21.15.67:0/3352876280 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f311c0cbef0 con 0x7f311c15e2a0 2024-03-24T00:41:29.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.305+0000 7f311ad9d640 1 --2- 172.21.15.67:0/3352876280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f311c15e2a0 0x7f311c0cd870 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f311400e930 tx=0x7f311400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:29.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.305+0000 7f3112ffd640 1 -- 172.21.15.67:0/3352876280 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3114017070 con 0x7f311c15e2a0 2024-03-24T00:41:29.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.305+0000 7f3112ffd640 1 -- 172.21.15.67:0/3352876280 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f311400c980 con 0x7f311c15e2a0 2024-03-24T00:41:29.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.305+0000 7f3120d77640 1 -- 172.21.15.67:0/3352876280 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f311c0cc190 con 0x7f311c15e2a0 2024-03-24T00:41:29.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.305+0000 7f3112ffd640 1 -- 172.21.15.67:0/3352876280 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f311400cd70 con 0x7f311c15e2a0 2024-03-24T00:41:29.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.305+0000 7f3120d77640 1 -- 172.21.15.67:0/3352876280 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f311c0cc750 con 0x7f311c15e2a0 2024-03-24T00:41:29.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.306+0000 7f3120d77640 1 -- 172.21.15.67:0/3352876280 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f311c04f400 con 0x7f311c15e2a0 2024-03-24T00:41:29.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.307+0000 7f3112ffd640 1 -- 172.21.15.67:0/3352876280 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f311401a070 con 0x7f311c15e2a0 2024-03-24T00:41:29.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.307+0000 7f3112ffd640 1 --2- 172.21.15.67:0/3352876280 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f30fc06bbd0 0x7f30fc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:29.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.307+0000 7f3112ffd640 1 -- 172.21.15.67:0/3352876280 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f311408ab60 con 0x7f311c15e2a0 2024-03-24T00:41:29.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.307+0000 7f311a59c640 1 --2- 172.21.15.67:0/3352876280 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f30fc06bbd0 0x7f30fc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:29.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.308+0000 7f311a59c640 1 --2- 172.21.15.67:0/3352876280 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f30fc06bbd0 0x7f30fc06e090 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f310c035000 tx=0x7f310c030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:29.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.312+0000 7f3112ffd640 1 -- 172.21.15.67:0/3352876280 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3114058f30 con 0x7f311c15e2a0 2024-03-24T00:41:29.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.539+0000 7f3120d77640 1 -- 172.21.15.67:0/3352876280 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f311c0cfb00 con 0x7f311c15e2a0 2024-03-24T00:41:29.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.540+0000 7f3112ffd640 1 -- 172.21.15.67:0/3352876280 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f311405cfd0 con 0x7f311c15e2a0 2024-03-24T00:41:29.540 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:41:29.540 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5360,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3312887486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":3312887486},{"type":"v1","addr":"172.21.15.67:6835","nonce":3312887486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5821,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/4171866637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":4171866637},{"type":"v1","addr":"172.21.15.72:6836","nonce":4171866637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5824,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/4106412452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":4106412452},{"type":"v1","addr":"172.21.15.72:6837","nonce":4106412452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:40:39.649631+0000","modified":"2024-03-24T00:40:42.657888+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5853},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5853":{"gid":5853,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6834/2741818214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2741818214},{"type":"v1","addr":"172.21.15.67:6834","nonce":2741818214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5853,"qdb_cluster":[5853]},"id":3}]} 2024-03-24T00:41:29.541 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 26 2024-03-24T00:41:29.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.542+0000 7f3120d77640 1 -- 172.21.15.67:0/3352876280 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f30fc06bbd0 msgr2=0x7f30fc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:29.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.542+0000 7f3120d77640 1 --2- 172.21.15.67:0/3352876280 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f30fc06bbd0 0x7f30fc06e090 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f310c035000 tx=0x7f310c030000 comp rx=0 tx=0).stop 2024-03-24T00:41:29.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.542+0000 7f3120d77640 1 -- 172.21.15.67:0/3352876280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f311c15e2a0 msgr2=0x7f311c0cd870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:29.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.542+0000 7f3120d77640 1 --2- 172.21.15.67:0/3352876280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f311c15e2a0 0x7f311c0cd870 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f311400e930 tx=0x7f311400edf0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.542+0000 7f3120d77640 1 -- 172.21.15.67:0/3352876280 shutdown_connections 2024-03-24T00:41:29.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.542+0000 7f3120d77640 1 --2- 172.21.15.67:0/3352876280 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f30fc06bbd0 0x7f30fc06e090 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.542+0000 7f3120d77640 1 --2- 172.21.15.67:0/3352876280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f311c15e2a0 0x7f311c0cd870 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.542+0000 7f3120d77640 1 --2- 172.21.15.67:0/3352876280 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f311c15dec0 0x7f311c0cd330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.542+0000 7f3120d77640 1 --2- 172.21.15.67:0/3352876280 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f311c0d07e0 0x7f311c1f3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.542+0000 7f3120d77640 1 -- 172.21.15.67:0/3352876280 >> 172.21.15.67:0/3352876280 conn(0x7f311c06c810 msgr2=0x7f311c0b5d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:29.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.542+0000 7f3120d77640 1 -- 172.21.15.67:0/3352876280 shutdown_connections 2024-03-24T00:41:29.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.542+0000 7f3120d77640 1 -- 172.21.15.67:0/3352876280 wait complete. 2024-03-24T00:41:29.553 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:41:29.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.653+0000 7f24afb09640 1 Processor -- start 2024-03-24T00:41:29.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.654+0000 7f24afb09640 1 -- start start 2024-03-24T00:41:29.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.654+0000 7f24afb09640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f24a815e1b0 0x7f24a80cfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:29.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.654+0000 7f24afb09640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24a80d0fc0 con 0x7f24a80b43a0 2024-03-24T00:41:29.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.654+0000 7f24afb09640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24a80d1260 con 0x7f24a809a410 2024-03-24T00:41:29.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.654+0000 7f24afb09640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24a815e590 con 0x7f24a815e1b0 2024-03-24T00:41:29.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.655+0000 7f24ad8a5640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f24a815e1b0 0x7f24a80cfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:29.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.655+0000 7f24ad0a4640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f24a80b43a0 0x7f24a8099d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33194/0 (socket says 172.21.15.67:33194) 2024-03-24T00:41:29.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.655+0000 7f24ad0a4640 1 -- 172.21.15.67:0/3469378926 learned_addr learned my addr 172.21.15.67:0/3469378926 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:41:29.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.655+0000 7f24ad8a5640 1 -- 172.21.15.67:0/3469378926 >> v1:172.21.15.72:6789/0 conn(0x7f24a809a410 legacy=0x7f24a80cc880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:29.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.655+0000 7f24ad8a5640 1 -- 172.21.15.67:0/3469378926 >> v1:172.21.15.67:6789/0 conn(0x7f24a80b43a0 legacy=0x7f24a8099d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:29.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.655+0000 7f24ad8a5640 1 -- 172.21.15.67:0/3469378926 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24a80d1540 con 0x7f24a815e1b0 2024-03-24T00:41:29.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.655+0000 7f24ac8a3640 1 -- 172.21.15.67:0/3469378926 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2566045323 0 0) 0x7f24a80d0fc0 con 0x7f24a80b43a0 2024-03-24T00:41:29.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.656+0000 7f24ad8a5640 1 --2- 172.21.15.67:0/3469378926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24a815e1b0 0x7f24a80cfe20 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f2498009d90 tx=0x7f249802bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73bd89bbf246c34f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:29.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.656+0000 7f24ac8a3640 1 -- 172.21.15.67:0/3469378926 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2498039070 con 0x7f24a815e1b0 2024-03-24T00:41:29.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.656+0000 7f24ac8a3640 1 -- 172.21.15.67:0/3469378926 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f249802cb40 con 0x7f24a815e1b0 2024-03-24T00:41:29.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.656+0000 7f24ac8a3640 1 -- 172.21.15.67:0/3469378926 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2498034440 con 0x7f24a815e1b0 2024-03-24T00:41:29.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.657+0000 7f24afb09640 1 -- 172.21.15.67:0/3469378926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24a815e1b0 msgr2=0x7f24a80cfe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:29.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.657+0000 7f24afb09640 1 --2- 172.21.15.67:0/3469378926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24a815e1b0 0x7f24a80cfe20 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f2498009d90 tx=0x7f249802bd90 comp rx=0 tx=0).stop 2024-03-24T00:41:29.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.657+0000 7f24afb09640 1 -- 172.21.15.67:0/3469378926 shutdown_connections 2024-03-24T00:41:29.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.657+0000 7f24afb09640 1 --2- 172.21.15.67:0/3469378926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24a815e1b0 0x7f24a80cfe20 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.657+0000 7f24afb09640 1 -- 172.21.15.67:0/3469378926 >> 172.21.15.67:0/3469378926 conn(0x7f24a80b36d0 msgr2=0x7f24a80b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:29.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.657+0000 7f24afb09640 1 -- 172.21.15.67:0/3469378926 shutdown_connections 2024-03-24T00:41:29.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.657+0000 7f24afb09640 1 -- 172.21.15.67:0/3469378926 wait complete. 2024-03-24T00:41:29.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.658+0000 7f24afb09640 1 Processor -- start 2024-03-24T00:41:29.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.658+0000 7f24afb09640 1 -- start start 2024-03-24T00:41:29.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.659+0000 7f24afb09640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24a809a410 0x7f24a814b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:29.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.659+0000 7f24afb09640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f24a80b43a0 0x7f24a8070190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:29.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.659+0000 7f24afb09640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f24a815e1b0 0x7f24a80706d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:29.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.659+0000 7f24afb09640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a809a9a0 con 0x7f24a809a410 2024-03-24T00:41:29.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.659+0000 7f24afb09640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a80d1820 con 0x7f24a815e1b0 2024-03-24T00:41:29.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.659+0000 7f24afb09640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24a80ccf90 con 0x7f24a80b43a0 2024-03-24T00:41:29.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.659+0000 7f24ad8a5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24a809a410 0x7f24a814b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:29.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.659+0000 7f24ad8a5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24a809a410 0x7f24a814b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37620/0 (socket says 172.21.15.67:37620) 2024-03-24T00:41:29.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.659+0000 7f24ad8a5640 1 -- 172.21.15.67:0/4000350810 learned_addr learned my addr 172.21.15.67:0/4000350810 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:29.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.659+0000 7f24ae0a6640 1 --2- 172.21.15.67:0/4000350810 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f24a815e1b0 0x7f24a80706d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:29.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.659+0000 7f24ad0a4640 1 --2- 172.21.15.67:0/4000350810 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f24a80b43a0 0x7f24a8070190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:29.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.660+0000 7f24ad8a5640 1 -- 172.21.15.67:0/4000350810 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f24a80b43a0 msgr2=0x7f24a8070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:29.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.660+0000 7f24ad8a5640 1 --2- 172.21.15.67:0/4000350810 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f24a80b43a0 0x7f24a8070190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.660+0000 7f24ad8a5640 1 -- 172.21.15.67:0/4000350810 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f24a815e1b0 msgr2=0x7f24a80706d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:29.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.660+0000 7f24ad8a5640 1 --2- 172.21.15.67:0/4000350810 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f24a815e1b0 0x7f24a80706d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.660+0000 7f24ad8a5640 1 -- 172.21.15.67:0/4000350810 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24a80cba60 con 0x7f24a809a410 2024-03-24T00:41:29.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.660+0000 7f24ae0a6640 1 --2- 172.21.15.67:0/4000350810 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f24a815e1b0 0x7f24a80706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:41:29.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.660+0000 7f24ad8a5640 1 --2- 172.21.15.67:0/4000350810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24a809a410 0x7f24a814b080 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f2498009d90 tx=0x7f2498004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:29.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.660+0000 7f24a67fc640 1 -- 172.21.15.67:0/4000350810 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2498044020 con 0x7f24a809a410 2024-03-24T00:41:29.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.660+0000 7f24a67fc640 1 -- 172.21.15.67:0/4000350810 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f249803f670 con 0x7f24a809a410 2024-03-24T00:41:29.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.660+0000 7f24afb09640 1 -- 172.21.15.67:0/4000350810 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24a80b4950 con 0x7f24a809a410 2024-03-24T00:41:29.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.660+0000 7f24a67fc640 1 -- 172.21.15.67:0/4000350810 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2498039040 con 0x7f24a809a410 2024-03-24T00:41:29.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.660+0000 7f24afb09640 1 -- 172.21.15.67:0/4000350810 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24a80cad00 con 0x7f24a809a410 2024-03-24T00:41:29.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.661+0000 7f24afb09640 1 -- 172.21.15.67:0/4000350810 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24a804f400 con 0x7f24a809a410 2024-03-24T00:41:29.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.662+0000 7f24a67fc640 1 -- 172.21.15.67:0/4000350810 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2498047020 con 0x7f24a809a410 2024-03-24T00:41:29.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.662+0000 7f24a67fc640 1 --2- 172.21.15.67:0/4000350810 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f249406bbd0 0x7f249406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:29.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.663+0000 7f24a67fc640 1 -- 172.21.15.67:0/4000350810 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f24980ad7e0 con 0x7f24a809a410 2024-03-24T00:41:29.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.663+0000 7f24ad0a4640 1 --2- 172.21.15.67:0/4000350810 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f249406bbd0 0x7f249406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:29.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.663+0000 7f24ad0a4640 1 --2- 172.21.15.67:0/4000350810 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f249406bbd0 0x7f249406e090 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f24a80cc060 tx=0x7f249c007b50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:29.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.667+0000 7f24a67fc640 1 -- 172.21.15.67:0/4000350810 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f249804e030 con 0x7f24a809a410 2024-03-24T00:41:29.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.849+0000 7f24afb09640 1 -- 172.21.15.67:0/4000350810 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f24a806e0a0 con 0x7f24a809a410 2024-03-24T00:41:29.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.850+0000 7f24a67fc640 1 -- 172.21.15.67:0/4000350810 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) v1 ==== 74+0+16882 (secure 0 0 0) 0x7f249807c1d0 con 0x7f24a809a410 2024-03-24T00:41:29.851 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:41:29.851 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":50,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:40:41.288503+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:40:37.927587+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-24T00:40:38.616521+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:41:29.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.852+0000 7f24afb09640 1 -- 172.21.15.67:0/4000350810 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f249406bbd0 msgr2=0x7f249406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:29.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.852+0000 7f24afb09640 1 --2- 172.21.15.67:0/4000350810 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f249406bbd0 0x7f249406e090 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f24a80cc060 tx=0x7f249c007b50 comp rx=0 tx=0).stop 2024-03-24T00:41:29.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.852+0000 7f24afb09640 1 -- 172.21.15.67:0/4000350810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24a809a410 msgr2=0x7f24a814b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:29.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.852+0000 7f24afb09640 1 --2- 172.21.15.67:0/4000350810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24a809a410 0x7f24a814b080 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f2498009d90 tx=0x7f2498004a40 comp rx=0 tx=0).stop 2024-03-24T00:41:29.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.852+0000 7f24afb09640 1 -- 172.21.15.67:0/4000350810 shutdown_connections 2024-03-24T00:41:29.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.852+0000 7f24afb09640 1 --2- 172.21.15.67:0/4000350810 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f249406bbd0 0x7f249406e090 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.852+0000 7f24afb09640 1 --2- 172.21.15.67:0/4000350810 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f24a815e1b0 0x7f24a80706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.852+0000 7f24afb09640 1 --2- 172.21.15.67:0/4000350810 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f24a80b43a0 0x7f24a8070190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.852+0000 7f24afb09640 1 --2- 172.21.15.67:0/4000350810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24a809a410 0x7f24a814b080 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.852+0000 7f24afb09640 1 -- 172.21.15.67:0/4000350810 >> 172.21.15.67:0/4000350810 conn(0x7f24a80b36d0 msgr2=0x7f24a806caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:29.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.852+0000 7f24afb09640 1 -- 172.21.15.67:0/4000350810 shutdown_connections 2024-03-24T00:41:29.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.852+0000 7f24afb09640 1 -- 172.21.15.67:0/4000350810 wait complete. 2024-03-24T00:41:29.863 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-24T00:41:29.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.967+0000 7f17587bc640 1 Processor -- start 2024-03-24T00:41:29.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.967+0000 7f17587bc640 1 -- start start 2024-03-24T00:41:29.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.968+0000 7f17587bc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f17500ad1b0 0x7f17500cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:29.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.968+0000 7f17587bc640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17500ccea0 con 0x7f17500cca10 2024-03-24T00:41:29.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.968+0000 7f17587bc640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1750067330 con 0x7f17500d04d0 2024-03-24T00:41:29.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.968+0000 7f17587bc640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17500678f0 con 0x7f17500ad1b0 2024-03-24T00:41:29.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.969+0000 7f1756558640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f17500ad1b0 0x7f17500cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:29.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.969+0000 7f1755d57640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f17500cca10 0x7f17500ad590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33198/0 (socket says 172.21.15.67:33198) 2024-03-24T00:41:29.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.969+0000 7f1755d57640 1 -- 172.21.15.67:0/34044984 learned_addr learned my addr 172.21.15.67:0/34044984 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:41:29.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.969+0000 7f1756558640 1 -- 172.21.15.67:0/34044984 >> v1:172.21.15.72:6789/0 conn(0x7f17500d04d0 legacy=0x7f17500d08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:29.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.969+0000 7f1756558640 1 -- 172.21.15.67:0/34044984 >> v1:172.21.15.67:6789/0 conn(0x7f17500cca10 legacy=0x7f17500ad590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:29.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.969+0000 7f1756558640 1 -- 172.21.15.67:0/34044984 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1750067610 con 0x7f17500ad1b0 2024-03-24T00:41:29.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.970+0000 7f1756558640 1 --2- 172.21.15.67:0/34044984 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17500ad1b0 0x7f17500cc3c0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f1748009d90 tx=0x7f174802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5db584b8f9646f32 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:29.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.970+0000 7f1755556640 1 -- 172.21.15.67:0/34044984 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1748039070 con 0x7f17500ad1b0 2024-03-24T00:41:29.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.970+0000 7f1755556640 1 -- 172.21.15.67:0/34044984 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f174802cd00 con 0x7f17500ad1b0 2024-03-24T00:41:29.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.970+0000 7f1755556640 1 -- 172.21.15.67:0/34044984 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1748034660 con 0x7f17500ad1b0 2024-03-24T00:41:29.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.970+0000 7f17587bc640 1 -- 172.21.15.67:0/34044984 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17500ad1b0 msgr2=0x7f17500cc3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:29.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.970+0000 7f17587bc640 1 --2- 172.21.15.67:0/34044984 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17500ad1b0 0x7f17500cc3c0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f1748009d90 tx=0x7f174802bf50 comp rx=0 tx=0).stop 2024-03-24T00:41:29.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.970+0000 7f17587bc640 1 -- 172.21.15.67:0/34044984 shutdown_connections 2024-03-24T00:41:29.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.970+0000 7f17587bc640 1 --2- 172.21.15.67:0/34044984 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17500ad1b0 0x7f17500cc3c0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.970+0000 7f17587bc640 1 -- 172.21.15.67:0/34044984 >> 172.21.15.67:0/34044984 conn(0x7f1750072860 msgr2=0x7f17500b5030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:29.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.970+0000 7f17587bc640 1 -- 172.21.15.67:0/34044984 shutdown_connections 2024-03-24T00:41:29.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.970+0000 7f17587bc640 1 -- 172.21.15.67:0/34044984 wait complete. 2024-03-24T00:41:29.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.972+0000 7f17587bc640 1 Processor -- start 2024-03-24T00:41:29.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.972+0000 7f17587bc640 1 -- start start 2024-03-24T00:41:29.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.972+0000 7f17587bc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17500ad1b0 0x7f17500b2820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:29.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.972+0000 7f17587bc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17500cca10 0x7f1750148f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:29.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.972+0000 7f17587bc640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f17500d04d0 0x7f175014c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:29.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.972+0000 7f17587bc640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1750099a30 con 0x7f17500ad1b0 2024-03-24T00:41:29.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.972+0000 7f17587bc640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1750099df0 con 0x7f17500cca10 2024-03-24T00:41:29.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.972+0000 7f17587bc640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1750099bb0 con 0x7f17500d04d0 2024-03-24T00:41:29.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.972+0000 7f1756558640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17500ad1b0 0x7f17500b2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:29.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.972+0000 7f1756558640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17500ad1b0 0x7f17500b2820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37628/0 (socket says 172.21.15.67:37628) 2024-03-24T00:41:29.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.972+0000 7f1756558640 1 -- 172.21.15.67:0/1129512499 learned_addr learned my addr 172.21.15.67:0/1129512499 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:29.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.973+0000 7f1755d57640 1 --2- 172.21.15.67:0/1129512499 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17500cca10 0x7f1750148f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:29.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.973+0000 7f1756d59640 1 --2- 172.21.15.67:0/1129512499 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f17500d04d0 0x7f175014c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:29.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.973+0000 7f1756558640 1 -- 172.21.15.67:0/1129512499 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f17500d04d0 msgr2=0x7f175014c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:29.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.973+0000 7f1756558640 1 --2- 172.21.15.67:0/1129512499 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f17500d04d0 0x7f175014c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.973+0000 7f1756558640 1 -- 172.21.15.67:0/1129512499 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17500cca10 msgr2=0x7f1750148f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:29.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.973+0000 7f1756558640 1 --2- 172.21.15.67:0/1129512499 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17500cca10 0x7f1750148f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:29.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.973+0000 7f1756558640 1 -- 172.21.15.67:0/1129512499 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1750149460 con 0x7f17500ad1b0 2024-03-24T00:41:29.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.973+0000 7f1755d57640 1 --2- 172.21.15.67:0/1129512499 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17500cca10 0x7f1750148f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:41:29.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.973+0000 7f1756d59640 1 --2- 172.21.15.67:0/1129512499 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f17500d04d0 0x7f175014c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:41:29.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.973+0000 7f1756558640 1 --2- 172.21.15.67:0/1129512499 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17500ad1b0 0x7f17500b2820 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f1748037040 tx=0x7f1748004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:29.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.973+0000 7f174effd640 1 -- 172.21.15.67:0/1129512499 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1748039050 con 0x7f17500ad1b0 2024-03-24T00:41:29.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.973+0000 7f174effd640 1 -- 172.21.15.67:0/1129512499 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1748045070 con 0x7f17500ad1b0 2024-03-24T00:41:29.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.973+0000 7f17587bc640 1 -- 172.21.15.67:0/1129512499 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f175014cc20 con 0x7f17500ad1b0 2024-03-24T00:41:29.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.973+0000 7f174effd640 1 -- 172.21.15.67:0/1129512499 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f174803ea50 con 0x7f17500ad1b0 2024-03-24T00:41:29.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.973+0000 7f17587bc640 1 -- 172.21.15.67:0/1129512499 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f175014d1e0 con 0x7f17500ad1b0 2024-03-24T00:41:29.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.974+0000 7f17587bc640 1 -- 172.21.15.67:0/1129512499 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f175004f400 con 0x7f17500ad1b0 2024-03-24T00:41:29.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.975+0000 7f174effd640 1 -- 172.21.15.67:0/1129512499 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f174802cb80 con 0x7f17500ad1b0 2024-03-24T00:41:29.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.975+0000 7f174effd640 1 --2- 172.21.15.67:0/1129512499 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f173c06bc00 0x7f173c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:29.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.975+0000 7f174effd640 1 -- 172.21.15.67:0/1129512499 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f17480adbf0 con 0x7f17500ad1b0 2024-03-24T00:41:29.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.975+0000 7f1755d57640 1 --2- 172.21.15.67:0/1129512499 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f173c06bc00 0x7f173c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:29.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.976+0000 7f1755d57640 1 --2- 172.21.15.67:0/1129512499 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f173c06bc00 0x7f173c06e0c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f1750159260 tx=0x7f1744009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:29.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:29.980+0000 7f174effd640 1 -- 172.21.15.67:0/1129512499 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f174807c670 con 0x7f17500ad1b0 2024-03-24T00:41:30.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.200+0000 7f17587bc640 1 -- 172.21.15.67:0/1129512499 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f17500b4130 con 0x7f17500ad1b0 2024-03-24T00:41:30.201 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-24T00:41:30.201+0000 7fbfcfa11640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T00:41:30.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.202+0000 7f174effd640 1 -- 172.21.15.67:0/1129512499 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v27) v1 ==== 161+0+0 (secure 0 0 0) 0x7f174807bf70 con 0x7f17500ad1b0 2024-03-24T00:41:30.203 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-24T00:41:30.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.205+0000 7f17587bc640 1 -- 172.21.15.67:0/1129512499 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f173c06bc00 msgr2=0x7f173c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:30.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.205+0000 7f17587bc640 1 --2- 172.21.15.67:0/1129512499 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f173c06bc00 0x7f173c06e0c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f1750159260 tx=0x7f1744009040 comp rx=0 tx=0).stop 2024-03-24T00:41:30.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.205+0000 7f17587bc640 1 -- 172.21.15.67:0/1129512499 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17500ad1b0 msgr2=0x7f17500b2820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:30.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.205+0000 7f17587bc640 1 --2- 172.21.15.67:0/1129512499 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17500ad1b0 0x7f17500b2820 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f1748037040 tx=0x7f1748004a40 comp rx=0 tx=0).stop 2024-03-24T00:41:30.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.205+0000 7f17587bc640 1 -- 172.21.15.67:0/1129512499 shutdown_connections 2024-03-24T00:41:30.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.205+0000 7f17587bc640 1 --2- 172.21.15.67:0/1129512499 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f173c06bc00 0x7f173c06e0c0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:30.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.205+0000 7f17587bc640 1 --2- 172.21.15.67:0/1129512499 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f17500d04d0 0x7f175014c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:30.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.205+0000 7f17587bc640 1 --2- 172.21.15.67:0/1129512499 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17500cca10 0x7f1750148f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:30.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.205+0000 7f17587bc640 1 --2- 172.21.15.67:0/1129512499 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17500ad1b0 0x7f17500b2820 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:30.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.205+0000 7f17587bc640 1 -- 172.21.15.67:0/1129512499 >> 172.21.15.67:0/1129512499 conn(0x7f1750072860 msgr2=0x7f17500ce920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:30.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.206+0000 7f17587bc640 1 -- 172.21.15.67:0/1129512499 shutdown_connections 2024-03-24T00:41:30.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.206+0000 7f17587bc640 1 -- 172.21.15.67:0/1129512499 wait complete. 2024-03-24T00:41:30.209 INFO:tasks.ceph.mds.a.smithi067.stderr: -5863> 2024-03-24T00:40:46.272+0000 7fbe5e6b5640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T00:41:30.212 INFO:tasks.ceph.mds.a.smithi067.stderr: -3145> 2024-03-24T00:40:52.978+0000 7fbe5a6ad640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [2,head] /parent_a/alpha auth v2 ap=1 DIRTYPARENT s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={5453=0-4194304@1} caps={5453=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=5453 | request=0 caps=1 dirtyparent=1 dirty=0 authpin=1 scrubqueue=0 0x564e71175700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(8)0x10000000001:[<0x10000000000/alpha v2>,<0x1/parent_a v4>]//[]","error_str":"failed to decode on-disk backtrace (40 bytes)!"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-24T00:41:30.217 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-24T00:41:30.240 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-03-24T00:41:30.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.320+0000 7f2fd2f90640 1 Processor -- start 2024-03-24T00:41:30.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.320+0000 7f2fd2f90640 1 -- start start 2024-03-24T00:41:30.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.321+0000 7f2fd2f90640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2fcc072380 0x7f2fcc0727e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:30.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.321+0000 7f2fd2f90640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fcc0b2b40 con 0x7f2fcc15a500 2024-03-24T00:41:30.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.321+0000 7f2fd2f90640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fcc0b2e20 con 0x7f2fcc0cd7a0 2024-03-24T00:41:30.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.321+0000 7f2fd2f90640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc072d20 con 0x7f2fcc072380 2024-03-24T00:41:30.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.321+0000 7f2fd0d2c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2fcc15a500 0x7f2fcc06f030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33216/0 (socket says 172.21.15.67:33216) 2024-03-24T00:41:30.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.321+0000 7f2fd0d2c640 1 -- 172.21.15.67:0/1346267403 learned_addr learned my addr 172.21.15.67:0/1346267403 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:41:30.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.321+0000 7f2fcb7fe640 1 -- 172.21.15.67:0/1346267403 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3149516574 0 0) 0x7f2fcc0b2b40 con 0x7f2fcc15a500 2024-03-24T00:41:30.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.321+0000 7f2fcb7fe640 1 -- 172.21.15.67:0/1346267403 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fb8003440 con 0x7f2fcc15a500 2024-03-24T00:41:30.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.321+0000 7f2fcbfff640 1 --2- 172.21.15.67:0/1346267403 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2fcc072380 0x7f2fcc0727e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:30.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.322+0000 7f2fcb7fe640 1 -- 172.21.15.67:0/1346267403 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4081383818 0 0) 0x7f2fb8003440 con 0x7f2fcc15a500 2024-03-24T00:41:30.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.322+0000 7f2fcb7fe640 1 -- 172.21.15.67:0/1346267403 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2fcc072380 msgr2=0x7f2fcc0727e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:30.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.322+0000 7f2fcb7fe640 1 --2- 172.21.15.67:0/1346267403 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2fcc072380 0x7f2fcc0727e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:30.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.322+0000 7f2fcb7fe640 1 -- 172.21.15.67:0/1346267403 >> v1:172.21.15.72:6789/0 conn(0x7f2fcc0cd7a0 legacy=0x7f2fcc0cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:30.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.322+0000 7f2fcb7fe640 1 -- 172.21.15.67:0/1346267403 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fcc0b3100 con 0x7f2fcc15a500 2024-03-24T00:41:30.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.322+0000 7f2fcb7fe640 1 -- 172.21.15.67:0/1346267403 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f2fc40019f0 con 0x7f2fcc15a500 2024-03-24T00:41:30.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.322+0000 7f2fcb7fe640 1 -- 172.21.15.67:0/1346267403 >> v1:172.21.15.67:6789/0 conn(0x7f2fcc15a500 legacy=0x7f2fcc06f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:30.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.322+0000 7f2fcb7fe640 1 --2- 172.21.15.67:0/1346267403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fb8003ad0 0x7f2fb8003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:30.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.322+0000 7f2fcb7fe640 1 --2- 172.21.15.67:0/1346267403 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2fb8008850 0x7f2fb800ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:30.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.322+0000 7f2fcb7fe640 1 --2- 172.21.15.67:0/1346267403 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2fcc072380 0x7f2fb800d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:30.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.322+0000 7f2fcb7fe640 1 -- 172.21.15.67:0/1346267403 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb800dc40 con 0x7f2fb8003ad0 2024-03-24T00:41:30.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.322+0000 7f2fcb7fe640 1 -- 172.21.15.67:0/1346267403 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb800ddf0 con 0x7f2fb8008850 2024-03-24T00:41:30.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.322+0000 7f2fcb7fe640 1 -- 172.21.15.67:0/1346267403 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb800dff0 con 0x7f2fcc072380 2024-03-24T00:41:30.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.322+0000 7f2fd0d2c640 1 --2- 172.21.15.67:0/1346267403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fb8003ad0 0x7f2fb8003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:30.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.322+0000 7f2fcbfff640 1 --2- 172.21.15.67:0/1346267403 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2fb8008850 0x7f2fb800ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:30.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.322+0000 7f2fd152d640 1 --2- 172.21.15.67:0/1346267403 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2fcc072380 0x7f2fb800d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:30.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.322+0000 7f2fd0d2c640 1 -- 172.21.15.67:0/1346267403 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2fcc072380 msgr2=0x7f2fb800d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:30.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.322+0000 7f2fd0d2c640 1 --2- 172.21.15.67:0/1346267403 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2fcc072380 0x7f2fb800d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:30.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.322+0000 7f2fd0d2c640 1 -- 172.21.15.67:0/1346267403 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2fb8008850 msgr2=0x7f2fb800ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:30.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.322+0000 7f2fd0d2c640 1 --2- 172.21.15.67:0/1346267403 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2fb8008850 0x7f2fb800ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:30.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.322+0000 7f2fd0d2c640 1 -- 172.21.15.67:0/1346267403 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb8003440 con 0x7f2fb8003ad0 2024-03-24T00:41:30.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.322+0000 7f2fd152d640 1 --2- 172.21.15.67:0/1346267403 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2fcc072380 0x7f2fb800d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:41:30.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.322+0000 7f2fcbfff640 1 --2- 172.21.15.67:0/1346267403 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2fb8008850 0x7f2fb800ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:41:30.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.323+0000 7f2fd0d2c640 1 --2- 172.21.15.67:0/1346267403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fb8003ad0 0x7f2fb8003eb0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f2fc400a800 tx=0x7f2fb8003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27a13ba2f6205211 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:30.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.323+0000 7f2fcb7fe640 1 -- 172.21.15.67:0/1346267403 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc400a450 con 0x7f2fb8003ad0 2024-03-24T00:41:30.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.323+0000 7f2fcb7fe640 1 -- 172.21.15.67:0/1346267403 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fc4034650 con 0x7f2fb8003ad0 2024-03-24T00:41:30.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.323+0000 7f2fcb7fe640 1 -- 172.21.15.67:0/1346267403 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc4034a40 con 0x7f2fb8003ad0 2024-03-24T00:41:30.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.323+0000 7f2fd2f90640 1 -- 172.21.15.67:0/1346267403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fb8003ad0 msgr2=0x7f2fb8003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:30.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.323+0000 7f2fd2f90640 1 --2- 172.21.15.67:0/1346267403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fb8003ad0 0x7f2fb8003eb0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f2fc400a800 tx=0x7f2fb8003730 comp rx=0 tx=0).stop 2024-03-24T00:41:30.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.323+0000 7f2fd152d640 1 -- 172.21.15.67:0/1346267403 reap_dead start 2024-03-24T00:41:30.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.324+0000 7f2fd2f90640 1 -- 172.21.15.67:0/1346267403 shutdown_connections 2024-03-24T00:41:30.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.324+0000 7f2fd2f90640 1 -- 172.21.15.67:0/1346267403 >> 172.21.15.67:0/1346267403 conn(0x7f2fcc0cf9e0 msgr2=0x7f2fcc0cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:30.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.324+0000 7f2fd2f90640 1 -- 172.21.15.67:0/1346267403 shutdown_connections 2024-03-24T00:41:30.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.324+0000 7f2fd2f90640 1 -- 172.21.15.67:0/1346267403 wait complete. 2024-03-24T00:41:30.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.325+0000 7f2fd2f90640 1 Processor -- start 2024-03-24T00:41:30.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.325+0000 7f2fd2f90640 1 -- start start 2024-03-24T00:41:30.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.326+0000 7f2fd2f90640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2fb8003ad0 0x7f2fcc1f35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:30.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.326+0000 7f2fd2f90640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2fcc1eeee0 0x7f2fcc14ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:30.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.326+0000 7f2fd2f90640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fcc14bf80 0x7f2fcc0af660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:30.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.326+0000 7f2fd2f90640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc15a8e0 con 0x7f2fcc14bf80 2024-03-24T00:41:30.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.326+0000 7f2fd2f90640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc0b36c0 con 0x7f2fb8003ad0 2024-03-24T00:41:30.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.326+0000 7f2fd2f90640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc06f740 con 0x7f2fcc1eeee0 2024-03-24T00:41:30.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.326+0000 7f2fd152d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fcc14bf80 0x7f2fcc0af660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:30.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.326+0000 7f2fd152d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fcc14bf80 0x7f2fcc0af660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37658/0 (socket says 172.21.15.67:37658) 2024-03-24T00:41:30.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.326+0000 7f2fd152d640 1 -- 172.21.15.67:0/3253971138 learned_addr learned my addr 172.21.15.67:0/3253971138 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:30.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.326+0000 7f2fd152d640 1 -- 172.21.15.67:0/3253971138 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2fcc1eeee0 msgr2=0x7f2fcc14ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:30.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.326+0000 7f2fd152d640 1 --2- 172.21.15.67:0/3253971138 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2fcc1eeee0 0x7f2fcc14ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:30.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.326+0000 7f2fd152d640 1 -- 172.21.15.67:0/3253971138 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2fb8003ad0 msgr2=0x7f2fcc1f35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:30.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.326+0000 7f2fd0d2c640 1 --2- 172.21.15.67:0/3253971138 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2fb8003ad0 0x7f2fcc1f35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:30.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.326+0000 7f2fd152d640 1 --2- 172.21.15.67:0/3253971138 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2fb8003ad0 0x7f2fcc1f35e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:30.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.326+0000 7f2fd152d640 1 -- 172.21.15.67:0/3253971138 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fcc13a260 con 0x7f2fcc14bf80 2024-03-24T00:41:30.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.326+0000 7f2fd0d2c640 1 --2- 172.21.15.67:0/3253971138 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2fb8003ad0 0x7f2fcc1f35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:41:30.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.326+0000 7f2fd152d640 1 --2- 172.21.15.67:0/3253971138 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fcc14bf80 0x7f2fcc0af660 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc0b3aa0 tx=0x7f2fbc00e820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:30.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.326+0000 7f2fc97fa640 1 -- 172.21.15.67:0/3253971138 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc019070 con 0x7f2fcc14bf80 2024-03-24T00:41:30.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.326+0000 7f2fc97fa640 1 -- 172.21.15.67:0/3253971138 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fbc0052e0 con 0x7f2fcc14bf80 2024-03-24T00:41:30.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.326+0000 7f2fc97fa640 1 -- 172.21.15.67:0/3253971138 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc005490 con 0x7f2fcc14bf80 2024-03-24T00:41:30.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.326+0000 7f2fd2f90640 1 -- 172.21.15.67:0/3253971138 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fcc13a540 con 0x7f2fcc14bf80 2024-03-24T00:41:30.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.326+0000 7f2fd2f90640 1 -- 172.21.15.67:0/3253971138 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fcc13ab00 con 0x7f2fcc14bf80 2024-03-24T00:41:30.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.327+0000 7f2fd2f90640 1 -- 172.21.15.67:0/3253971138 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fcc04f400 con 0x7f2fcc14bf80 2024-03-24T00:41:30.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.328+0000 7f2fc97fa640 1 -- 172.21.15.67:0/3253971138 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2fbc01c070 con 0x7f2fcc14bf80 2024-03-24T00:41:30.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.328+0000 7f2fc97fa640 1 --2- 172.21.15.67:0/3253971138 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2fac06bc00 0x7f2fac06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:30.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.328+0000 7f2fc97fa640 1 -- 172.21.15.67:0/3253971138 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f2fbc08a0b0 con 0x7f2fcc14bf80 2024-03-24T00:41:30.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.328+0000 7f2fd0d2c640 1 --2- 172.21.15.67:0/3253971138 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2fac06bc00 0x7f2fac06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:30.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.329+0000 7f2fd0d2c640 1 --2- 172.21.15.67:0/3253971138 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2fac06bc00 0x7f2fac06e0c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f2fc4034000 tx=0x7f2fc402f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:30.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.333+0000 7f2fc97fa640 1 -- 172.21.15.67:0/3253971138 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2fbc0589c0 con 0x7f2fcc14bf80 2024-03-24T00:41:30.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:30.554+0000 7f2fd2f90640 1 -- 172.21.15.67:0/3253971138 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f2fcc06e6c0 con 0x7f2fcc14bf80 2024-03-24T00:41:31.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.204+0000 7f2fc97fa640 1 -- 172.21.15.67:0/3253971138 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v28) v1 ==== 104+0+0 (secure 0 0 0) 0x7f2fbc0582c0 con 0x7f2fcc14bf80 2024-03-24T00:41:31.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.207+0000 7f2fd2f90640 1 -- 172.21.15.67:0/3253971138 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2fac06bc00 msgr2=0x7f2fac06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:31.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.207+0000 7f2fd2f90640 1 --2- 172.21.15.67:0/3253971138 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2fac06bc00 0x7f2fac06e0c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f2fc4034000 tx=0x7f2fc402f000 comp rx=0 tx=0).stop 2024-03-24T00:41:31.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.207+0000 7f2fd2f90640 1 -- 172.21.15.67:0/3253971138 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fcc14bf80 msgr2=0x7f2fcc0af660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:31.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.207+0000 7f2fd2f90640 1 --2- 172.21.15.67:0/3253971138 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fcc14bf80 0x7f2fcc0af660 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc0b3aa0 tx=0x7f2fbc00e820 comp rx=0 tx=0).stop 2024-03-24T00:41:31.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.207+0000 7f2fd2f90640 1 -- 172.21.15.67:0/3253971138 shutdown_connections 2024-03-24T00:41:31.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.207+0000 7f2fd2f90640 1 --2- 172.21.15.67:0/3253971138 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2fac06bc00 0x7f2fac06e0c0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:31.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.207+0000 7f2fd2f90640 1 --2- 172.21.15.67:0/3253971138 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fcc14bf80 0x7f2fcc0af660 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:31.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.207+0000 7f2fd2f90640 1 --2- 172.21.15.67:0/3253971138 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2fcc1eeee0 0x7f2fcc14ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:31.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.207+0000 7f2fd2f90640 1 --2- 172.21.15.67:0/3253971138 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2fb8003ad0 0x7f2fcc1f35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:31.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.207+0000 7f2fd2f90640 1 -- 172.21.15.67:0/3253971138 >> 172.21.15.67:0/3253971138 conn(0x7f2fcc0cf9e0 msgr2=0x7f2fcc0cea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:31.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.207+0000 7f2fd2f90640 1 -- 172.21.15.67:0/3253971138 shutdown_connections 2024-03-24T00:41:31.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.207+0000 7f2fd2f90640 1 -- 172.21.15.67:0/3253971138 wait complete. 2024-03-24T00:41:31.221 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-24T00:41:31.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.376+0000 7f5fc1cf1640 1 Processor -- start 2024-03-24T00:41:31.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.376+0000 7f5fc1cf1640 1 -- start start 2024-03-24T00:41:31.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.377+0000 7f5fc1cf1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5fbc06cb60 0x7f5fbc09a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:31.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.377+0000 7f5fc1cf1640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5fbc06cf40 0x7f5fbc0cfe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:31.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.377+0000 7f5fc1cf1640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5fbc0b2800 0x7f5fbc0b2c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:31.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.377+0000 7f5fc1cf1640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fbc0d06d0 con 0x7f5fbc0b2800 2024-03-24T00:41:31.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.377+0000 7f5fc1cf1640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fbc0d09b0 con 0x7f5fbc06cf40 2024-03-24T00:41:31.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.377+0000 7f5fc1cf1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fbc0d0c90 con 0x7f5fbc06cb60 2024-03-24T00:41:31.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.377+0000 7f5fbbfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5fbc0b2800 0x7f5fbc0b2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:31.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.377+0000 7f5fbbfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5fbc0b2800 0x7f5fbc0b2c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37664/0 (socket says 172.21.15.67:37664) 2024-03-24T00:41:31.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.377+0000 7f5fbbfff640 1 -- 172.21.15.67:0/2533404055 learned_addr learned my addr 172.21.15.67:0/2533404055 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:31.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.377+0000 7f5fbb7fe640 1 --2- 172.21.15.67:0/2533404055 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5fbc06cb60 0x7f5fbc09a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:31.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.377+0000 7f5fbaffd640 1 --2- 172.21.15.67:0/2533404055 >> v2:172.21.15.72:3300/0 conn(0x7f5fbc06cf40 0x7f5fbc0cfe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:31.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.377+0000 7f5fbbfff640 1 -- 172.21.15.67:0/2533404055 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5fbc06cb60 msgr2=0x7f5fbc09a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:31.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.377+0000 7f5fbbfff640 1 --2- 172.21.15.67:0/2533404055 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5fbc06cb60 0x7f5fbc09a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:31.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.377+0000 7f5fbbfff640 1 -- 172.21.15.67:0/2533404055 >> v2:172.21.15.72:3300/0 conn(0x7f5fbc06cf40 msgr2=0x7f5fbc0cfe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:31.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.377+0000 7f5fbbfff640 1 --2- 172.21.15.67:0/2533404055 >> v2:172.21.15.72:3300/0 conn(0x7f5fbc06cf40 0x7f5fbc0cfe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:31.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.377+0000 7f5fbbfff640 1 -- 172.21.15.67:0/2533404055 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fbc0d0f70 con 0x7f5fbc0b2800 2024-03-24T00:41:31.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.378+0000 7f5fbbfff640 1 --2- 172.21.15.67:0/2533404055 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fbc0b2800 0x7f5fbc0b2c90 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc073280 tx=0x7f5fb402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cd6146b02555938 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:31.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.378+0000 7f5fba7fc640 1 -- 172.21.15.67:0/2533404055 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb402c9e0 con 0x7f5fbc0b2800 2024-03-24T00:41:31.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.378+0000 7f5fba7fc640 1 -- 172.21.15.67:0/2533404055 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fb402cb80 con 0x7f5fbc0b2800 2024-03-24T00:41:31.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.378+0000 7f5fba7fc640 1 -- 172.21.15.67:0/2533404055 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb40343f0 con 0x7f5fbc0b2800 2024-03-24T00:41:31.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.379+0000 7f5fc1cf1640 1 -- 172.21.15.67:0/2533404055 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fbc0b2800 msgr2=0x7f5fbc0b2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:31.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.379+0000 7f5fc1cf1640 1 --2- 172.21.15.67:0/2533404055 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fbc0b2800 0x7f5fbc0b2c90 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc073280 tx=0x7f5fb402be40 comp rx=0 tx=0).stop 2024-03-24T00:41:31.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.379+0000 7f5fc1cf1640 1 -- 172.21.15.67:0/2533404055 shutdown_connections 2024-03-24T00:41:31.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.379+0000 7f5fc1cf1640 1 --2- 172.21.15.67:0/2533404055 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fbc0b2800 0x7f5fbc0b2c90 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:31.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.379+0000 7f5fc1cf1640 1 --2- 172.21.15.67:0/2533404055 >> v2:172.21.15.72:3300/0 conn(0x7f5fbc06cf40 0x7f5fbc0cfe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:31.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.379+0000 7f5fc1cf1640 1 --2- 172.21.15.67:0/2533404055 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5fbc06cb60 0x7f5fbc09a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:31.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.379+0000 7f5fc1cf1640 1 -- 172.21.15.67:0/2533404055 >> 172.21.15.67:0/2533404055 conn(0x7f5fbc0b36c0 msgr2=0x7f5fbc0b3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:31.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.379+0000 7f5fc1cf1640 1 -- 172.21.15.67:0/2533404055 shutdown_connections 2024-03-24T00:41:31.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.379+0000 7f5fc1cf1640 1 -- 172.21.15.67:0/2533404055 wait complete. 2024-03-24T00:41:31.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.380+0000 7f5fc1cf1640 1 Processor -- start 2024-03-24T00:41:31.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.380+0000 7f5fc1cf1640 1 -- start start 2024-03-24T00:41:31.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.381+0000 7f5fc1cf1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fbc06cb60 0x7f5fbc0ae7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:31.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.381+0000 7f5fc1cf1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5fbc06cf40 0x7f5fbc0aed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:31.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.381+0000 7f5fc1cf1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fbc0b2800 0x7f5fbc0c6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:31.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.381+0000 7f5fc1cf1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fbc06d3c0 con 0x7f5fbc06cb60 2024-03-24T00:41:31.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.381+0000 7f5fc1cf1640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fbc0d1770 con 0x7f5fbc0b2800 2024-03-24T00:41:31.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.381+0000 7f5fc1cf1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fbc0d0350 con 0x7f5fbc06cf40 2024-03-24T00:41:31.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.381+0000 7f5fbb7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fbc06cb60 0x7f5fbc0ae7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:31.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.381+0000 7f5fbb7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fbc06cb60 0x7f5fbc0ae7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37676/0 (socket says 172.21.15.67:37676) 2024-03-24T00:41:31.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.381+0000 7f5fbb7fe640 1 -- 172.21.15.67:0/2065154951 learned_addr learned my addr 172.21.15.67:0/2065154951 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:31.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.381+0000 7f5fbbfff640 1 --2- 172.21.15.67:0/2065154951 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fbc0b2800 0x7f5fbc0c6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:31.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.381+0000 7f5fbaffd640 1 --2- 172.21.15.67:0/2065154951 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5fbc06cf40 0x7f5fbc0aed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:31.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.381+0000 7f5fbb7fe640 1 -- 172.21.15.67:0/2065154951 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5fbc06cf40 msgr2=0x7f5fbc0aed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:31.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.381+0000 7f5fbb7fe640 1 --2- 172.21.15.67:0/2065154951 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5fbc06cf40 0x7f5fbc0aed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:31.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.381+0000 7f5fbb7fe640 1 -- 172.21.15.67:0/2065154951 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fbc0b2800 msgr2=0x7f5fbc0c6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:31.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.381+0000 7f5fbb7fe640 1 --2- 172.21.15.67:0/2065154951 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fbc0b2800 0x7f5fbc0c6a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:31.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.381+0000 7f5fbb7fe640 1 -- 172.21.15.67:0/2065154951 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fbc0c7840 con 0x7f5fbc06cb60 2024-03-24T00:41:31.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.382+0000 7f5fbbfff640 1 --2- 172.21.15.67:0/2065154951 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fbc0b2800 0x7f5fbc0c6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:41:31.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.382+0000 7f5fbb7fe640 1 --2- 172.21.15.67:0/2065154951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fbc06cb60 0x7f5fbc0ae7c0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f5fac00d5f0 tx=0x7f5fac00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:31.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.382+0000 7f5fa3fff640 1 -- 172.21.15.67:0/2065154951 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fac0125f0 con 0x7f5fbc06cb60 2024-03-24T00:41:31.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.382+0000 7f5fa3fff640 1 -- 172.21.15.67:0/2065154951 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fac019070 con 0x7f5fbc06cb60 2024-03-24T00:41:31.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.382+0000 7f5fc1cf1640 1 -- 172.21.15.67:0/2065154951 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fbc0c7b20 con 0x7f5fbc06cb60 2024-03-24T00:41:31.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.382+0000 7f5fa3fff640 1 -- 172.21.15.67:0/2065154951 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fac0129d0 con 0x7f5fbc06cb60 2024-03-24T00:41:31.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.382+0000 7f5fc1cf1640 1 -- 172.21.15.67:0/2065154951 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fbc0b0ae0 con 0x7f5fbc06cb60 2024-03-24T00:41:31.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.383+0000 7f5fc1cf1640 1 -- 172.21.15.67:0/2065154951 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fbc04f400 con 0x7f5fbc06cb60 2024-03-24T00:41:31.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.383+0000 7f5fa3fff640 1 -- 172.21.15.67:0/2065154951 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5fac012b70 con 0x7f5fbc06cb60 2024-03-24T00:41:31.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.384+0000 7f5fa3fff640 1 --2- 172.21.15.67:0/2065154951 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5fa806bc00 0x7f5fa806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:31.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.384+0000 7f5fa3fff640 1 -- 172.21.15.67:0/2065154951 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f5fac08b030 con 0x7f5fbc06cb60 2024-03-24T00:41:31.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.384+0000 7f5fbaffd640 1 --2- 172.21.15.67:0/2065154951 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5fa806bc00 0x7f5fa806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:31.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.385+0000 7f5fbaffd640 1 --2- 172.21.15.67:0/2065154951 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5fa806bc00 0x7f5fa806e0c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f5fb0006fd0 tx=0x7f5fb0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:31.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.388+0000 7f5fa3fff640 1 -- 172.21.15.67:0/2065154951 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5fac059940 con 0x7f5fbc06cb60 2024-03-24T00:41:31.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:31.571+0000 7f5fc1cf1640 1 -- 172.21.15.67:0/2065154951 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5fbc0b3170 con 0x7f5fbc06cb60 2024-03-24T00:41:32.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.481+0000 7f5fa3fff640 1 -- 172.21.15.67:0/2065154951 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v52) v1 ==== 181+0+0 (secure 0 0 0) 0x7f5fac059240 con 0x7f5fbc06cb60 2024-03-24T00:41:32.482 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' removed 2024-03-24T00:41:32.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.485+0000 7f5fc1cf1640 1 -- 172.21.15.67:0/2065154951 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5fa806bc00 msgr2=0x7f5fa806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:32.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.485+0000 7f5fc1cf1640 1 --2- 172.21.15.67:0/2065154951 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5fa806bc00 0x7f5fa806e0c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f5fb0006fd0 tx=0x7f5fb0008040 comp rx=0 tx=0).stop 2024-03-24T00:41:32.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.485+0000 7f5fc1cf1640 1 -- 172.21.15.67:0/2065154951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fbc06cb60 msgr2=0x7f5fbc0ae7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:32.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.485+0000 7f5fc1cf1640 1 --2- 172.21.15.67:0/2065154951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fbc06cb60 0x7f5fbc0ae7c0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f5fac00d5f0 tx=0x7f5fac00dab0 comp rx=0 tx=0).stop 2024-03-24T00:41:32.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.485+0000 7f5fc1cf1640 1 -- 172.21.15.67:0/2065154951 shutdown_connections 2024-03-24T00:41:32.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.485+0000 7f5fc1cf1640 1 --2- 172.21.15.67:0/2065154951 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5fa806bc00 0x7f5fa806e0c0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:32.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.485+0000 7f5fc1cf1640 1 --2- 172.21.15.67:0/2065154951 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fbc0b2800 0x7f5fbc0c6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:32.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.485+0000 7f5fc1cf1640 1 --2- 172.21.15.67:0/2065154951 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5fbc06cf40 0x7f5fbc0aed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:32.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.485+0000 7f5fc1cf1640 1 --2- 172.21.15.67:0/2065154951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fbc06cb60 0x7f5fbc0ae7c0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:32.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.485+0000 7f5fc1cf1640 1 -- 172.21.15.67:0/2065154951 >> 172.21.15.67:0/2065154951 conn(0x7f5fbc0b36c0 msgr2=0x7f5fbc15e0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:32.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.485+0000 7f5fc1cf1640 1 -- 172.21.15.67:0/2065154951 shutdown_connections 2024-03-24T00:41:32.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.485+0000 7f5fc1cf1640 1 -- 172.21.15.67:0/2065154951 wait complete. 2024-03-24T00:41:32.500 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-24T00:41:32.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.610+0000 7f9024be8640 1 Processor -- start 2024-03-24T00:41:32.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.610+0000 7f9024be8640 1 -- start start 2024-03-24T00:41:32.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.611+0000 7f9024be8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f90200b3c90 0x7f90200cf160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:32.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.611+0000 7f9024be8640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f902006d1f0 0x7f902015dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:32.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.611+0000 7f9024be8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f902006e110 con 0x7f90200b38b0 2024-03-24T00:41:32.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.611+0000 7f9024be8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90200cf820 con 0x7f90200b3c90 2024-03-24T00:41:32.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.611+0000 7f9024be8640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f902006de30 con 0x7f902006d1f0 2024-03-24T00:41:32.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.611+0000 7f901dd9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f90200b3c90 0x7f90200cf160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:32.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.611+0000 7f901e59c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f90200b38b0 0x7f90200cc3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33218/0 (socket says 172.21.15.67:33218) 2024-03-24T00:41:32.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.611+0000 7f901e59c640 1 -- 172.21.15.67:0/3067259284 learned_addr learned my addr 172.21.15.67:0/3067259284 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:41:32.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.611+0000 7f901ed9d640 1 --2- 172.21.15.67:0/3067259284 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f902006d1f0 0x7f902015dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:32.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.612+0000 7f901dd9b640 1 -- 172.21.15.67:0/3067259284 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f902006d1f0 msgr2=0x7f902015dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:32.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.612+0000 7f901dd9b640 1 --2- 172.21.15.67:0/3067259284 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f902006d1f0 0x7f902015dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:32.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.612+0000 7f901dd9b640 1 -- 172.21.15.67:0/3067259284 >> v1:172.21.15.67:6789/0 conn(0x7f90200b38b0 legacy=0x7f90200cc3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:32.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.612+0000 7f901dd9b640 1 -- 172.21.15.67:0/3067259284 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90201629d0 con 0x7f90200b3c90 2024-03-24T00:41:32.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.612+0000 7f901dd9b640 1 --2- 172.21.15.67:0/3067259284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90200b3c90 0x7f90200cf160 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f90080097a0 tx=0x7f900802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5729f52d2a9887a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:32.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.612+0000 7f901d59a640 1 -- 172.21.15.67:0/3067259284 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9008035020 con 0x7f90200b3c90 2024-03-24T00:41:32.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.612+0000 7f901d59a640 1 -- 172.21.15.67:0/3067259284 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9008031830 con 0x7f90200b3c90 2024-03-24T00:41:32.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.612+0000 7f901d59a640 1 -- 172.21.15.67:0/3067259284 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9008031c00 con 0x7f90200b3c90 2024-03-24T00:41:32.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.613+0000 7f9024be8640 1 -- 172.21.15.67:0/3067259284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90200b3c90 msgr2=0x7f90200cf160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:32.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.613+0000 7f9024be8640 1 --2- 172.21.15.67:0/3067259284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90200b3c90 0x7f90200cf160 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f90080097a0 tx=0x7f900802b8d0 comp rx=0 tx=0).stop 2024-03-24T00:41:32.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.613+0000 7f9024be8640 1 -- 172.21.15.67:0/3067259284 shutdown_connections 2024-03-24T00:41:32.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.613+0000 7f9024be8640 1 --2- 172.21.15.67:0/3067259284 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f902006d1f0 0x7f902015dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:32.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.613+0000 7f9024be8640 1 --2- 172.21.15.67:0/3067259284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90200b3c90 0x7f90200cf160 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:32.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.613+0000 7f9024be8640 1 -- 172.21.15.67:0/3067259284 >> 172.21.15.67:0/3067259284 conn(0x7f90200ce640 msgr2=0x7f90200cea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:32.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.613+0000 7f9024be8640 1 -- 172.21.15.67:0/3067259284 shutdown_connections 2024-03-24T00:41:32.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.613+0000 7f9024be8640 1 -- 172.21.15.67:0/3067259284 wait complete. 2024-03-24T00:41:32.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.614+0000 7f9024be8640 1 Processor -- start 2024-03-24T00:41:32.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.614+0000 7f9024be8640 1 -- start start 2024-03-24T00:41:32.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.615+0000 7f9024be8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f902006d1f0 0x7f90201590d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:32.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.615+0000 7f9024be8640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f90200b38b0 0x7f90200cad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:32.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.615+0000 7f9024be8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90200b3c90 0x7f9020160d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:32.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.615+0000 7f9024be8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f902015e440 con 0x7f902006d1f0 2024-03-24T00:41:32.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.615+0000 7f9024be8640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f902015e2c0 con 0x7f90200b3c90 2024-03-24T00:41:32.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.615+0000 7f9024be8640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f902015e5c0 con 0x7f90200b38b0 2024-03-24T00:41:32.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.615+0000 7f901e59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f902006d1f0 0x7f90201590d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:32.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.615+0000 7f901e59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f902006d1f0 0x7f90201590d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37692/0 (socket says 172.21.15.67:37692) 2024-03-24T00:41:32.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.615+0000 7f901e59c640 1 -- 172.21.15.67:0/1213771641 learned_addr learned my addr 172.21.15.67:0/1213771641 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:32.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.616+0000 7f901dd9b640 1 --2- 172.21.15.67:0/1213771641 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f90200b38b0 0x7f90200cad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:32.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.616+0000 7f901e59c640 1 -- 172.21.15.67:0/1213771641 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f90200b38b0 msgr2=0x7f90200cad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:32.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.616+0000 7f901ed9d640 1 --2- 172.21.15.67:0/1213771641 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90200b3c90 0x7f9020160d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:32.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.616+0000 7f901e59c640 1 --2- 172.21.15.67:0/1213771641 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f90200b38b0 0x7f90200cad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:32.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.616+0000 7f901e59c640 1 -- 172.21.15.67:0/1213771641 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90200b3c90 msgr2=0x7f9020160d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:32.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.616+0000 7f901e59c640 1 --2- 172.21.15.67:0/1213771641 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90200b3c90 0x7f9020160d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:32.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.616+0000 7f901e59c640 1 -- 172.21.15.67:0/1213771641 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f902015f8c0 con 0x7f902006d1f0 2024-03-24T00:41:32.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.616+0000 7f901ed9d640 1 --2- 172.21.15.67:0/1213771641 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90200b3c90 0x7f9020160d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:41:32.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.616+0000 7f901dd9b640 1 --2- 172.21.15.67:0/1213771641 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f90200b38b0 0x7f90200cad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:41:32.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.616+0000 7f901e59c640 1 --2- 172.21.15.67:0/1213771641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f902006d1f0 0x7f90201590d0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f901800c430 tx=0x7f901800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:32.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.616+0000 7f9016ffd640 1 -- 172.21.15.67:0/1213771641 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9018017070 con 0x7f902006d1f0 2024-03-24T00:41:32.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.616+0000 7f9016ffd640 1 -- 172.21.15.67:0/1213771641 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9018005ce0 con 0x7f902006d1f0 2024-03-24T00:41:32.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.616+0000 7f9024be8640 1 -- 172.21.15.67:0/1213771641 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f902015fba0 con 0x7f902006d1f0 2024-03-24T00:41:32.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.616+0000 7f9016ffd640 1 -- 172.21.15.67:0/1213771641 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90180072f0 con 0x7f902006d1f0 2024-03-24T00:41:32.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.616+0000 7f9024be8640 1 -- 172.21.15.67:0/1213771641 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9020160160 con 0x7f902006d1f0 2024-03-24T00:41:32.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.617+0000 7f9024be8640 1 -- 172.21.15.67:0/1213771641 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f902004f400 con 0x7f902006d1f0 2024-03-24T00:41:32.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.617+0000 7f9016ffd640 1 -- 172.21.15.67:0/1213771641 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f901801a070 con 0x7f902006d1f0 2024-03-24T00:41:32.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.618+0000 7f9016ffd640 1 --2- 172.21.15.67:0/1213771641 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f900006bc00 0x7f900006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:32.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.618+0000 7f9016ffd640 1 -- 172.21.15.67:0/1213771641 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5807+0+0 (secure 0 0 0) 0x7f9018089e70 con 0x7f902006d1f0 2024-03-24T00:41:32.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.618+0000 7f901dd9b640 1 --2- 172.21.15.67:0/1213771641 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f900006bc00 0x7f900006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:32.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.618+0000 7f901dd9b640 1 --2- 172.21.15.67:0/1213771641 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f900006bc00 0x7f900006e0c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f9008036000 tx=0x7f9008008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:32.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.622+0000 7f9016ffd640 1 -- 172.21.15.67:0/1213771641 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9018058990 con 0x7f902006d1f0 2024-03-24T00:41:32.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:32.812+0000 7f9024be8640 1 -- 172.21.15.67:0/1213771641 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9020161bd0 con 0x7f902006d1f0 2024-03-24T00:41:33.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.485+0000 7f9016ffd640 1 -- 172.21.15.67:0/1213771641 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v53) v1 ==== 169+0+0 (secure 0 0 0) 0x7f9018058290 con 0x7f902006d1f0 2024-03-24T00:41:33.486 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' removed 2024-03-24T00:41:33.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.489+0000 7f9024be8640 1 -- 172.21.15.67:0/1213771641 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f900006bc00 msgr2=0x7f900006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:33.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.489+0000 7f9024be8640 1 --2- 172.21.15.67:0/1213771641 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f900006bc00 0x7f900006e0c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f9008036000 tx=0x7f9008008030 comp rx=0 tx=0).stop 2024-03-24T00:41:33.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.489+0000 7f9024be8640 1 -- 172.21.15.67:0/1213771641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f902006d1f0 msgr2=0x7f90201590d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:33.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.489+0000 7f9024be8640 1 --2- 172.21.15.67:0/1213771641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f902006d1f0 0x7f90201590d0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f901800c430 tx=0x7f901800c8f0 comp rx=0 tx=0).stop 2024-03-24T00:41:33.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.489+0000 7f9024be8640 1 -- 172.21.15.67:0/1213771641 shutdown_connections 2024-03-24T00:41:33.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.489+0000 7f9024be8640 1 --2- 172.21.15.67:0/1213771641 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f900006bc00 0x7f900006e0c0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:33.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.489+0000 7f9024be8640 1 --2- 172.21.15.67:0/1213771641 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90200b3c90 0x7f9020160d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:33.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.489+0000 7f9024be8640 1 --2- 172.21.15.67:0/1213771641 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f90200b38b0 0x7f90200cad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:33.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.489+0000 7f9024be8640 1 --2- 172.21.15.67:0/1213771641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f902006d1f0 0x7f90201590d0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:33.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.489+0000 7f9024be8640 1 -- 172.21.15.67:0/1213771641 >> 172.21.15.67:0/1213771641 conn(0x7f90200ce640 msgr2=0x7f90200b47e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:33.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.489+0000 7f9024be8640 1 -- 172.21.15.67:0/1213771641 shutdown_connections 2024-03-24T00:41:33.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.489+0000 7f9024be8640 1 -- 172.21.15.67:0/1213771641 wait complete. 2024-03-24T00:41:33.504 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-24T00:41:33.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.617+0000 7f02cee27640 1 Processor -- start 2024-03-24T00:41:33.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.617+0000 7f02cee27640 1 -- start start 2024-03-24T00:41:33.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.618+0000 7f02cee27640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02c80b41e0 0x7f02c80b3620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:33.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.618+0000 7f02cee27640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f02c80b3b60 0x7f02c80cc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:33.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.618+0000 7f02cee27640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02c80cf090 con 0x7f02c815dfd0 2024-03-24T00:41:33.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.618+0000 7f02cee27640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02c80cc260 con 0x7f02c80b3b60 2024-03-24T00:41:33.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.618+0000 7f02cee27640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02c80d0320 con 0x7f02c80b41e0 2024-03-24T00:41:33.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.618+0000 7f02cd3c4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f02c80b3b60 0x7f02c80cc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:33.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.618+0000 7f02cd3c4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f02c80b3b60 0x7f02c80cc6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43650/0 (socket says 172.21.15.67:43650) 2024-03-24T00:41:33.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.618+0000 7f02cd3c4640 1 -- 172.21.15.67:0/1466114210 learned_addr learned my addr 172.21.15.67:0/1466114210 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:33.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.618+0000 7f02c7fff640 1 --2- 172.21.15.67:0/1466114210 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02c80b41e0 0x7f02c80b3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:33.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.619+0000 7f02cd3c4640 1 -- 172.21.15.67:0/1466114210 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02c80b41e0 msgr2=0x7f02c80b3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:33.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.619+0000 7f02cd3c4640 1 --2- 172.21.15.67:0/1466114210 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02c80b41e0 0x7f02c80b3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:33.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.619+0000 7f02cd3c4640 1 -- 172.21.15.67:0/1466114210 >> v1:172.21.15.72:6789/0 conn(0x7f02c815dfd0 legacy=0x7f02c80cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:33.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.619+0000 7f02cd3c4640 1 -- 172.21.15.67:0/1466114210 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02c80d0600 con 0x7f02c80b3b60 2024-03-24T00:41:33.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.619+0000 7f02cd3c4640 1 --2- 172.21.15.67:0/1466114210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02c80b3b60 0x7f02c80cc6d0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f02c0009870 tx=0x7f02c002ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fea9bc1af8560829 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:33.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.620+0000 7f02c77fe640 1 -- 172.21.15.67:0/1466114210 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c002f840 con 0x7f02c80b3b60 2024-03-24T00:41:33.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.620+0000 7f02c77fe640 1 -- 172.21.15.67:0/1466114210 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02c002f9e0 con 0x7f02c80b3b60 2024-03-24T00:41:33.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.620+0000 7f02c77fe640 1 -- 172.21.15.67:0/1466114210 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c002fdb0 con 0x7f02c80b3b60 2024-03-24T00:41:33.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.620+0000 7f02cee27640 1 -- 172.21.15.67:0/1466114210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02c80b3b60 msgr2=0x7f02c80cc6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:33.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.620+0000 7f02cee27640 1 --2- 172.21.15.67:0/1466114210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02c80b3b60 0x7f02c80cc6d0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f02c0009870 tx=0x7f02c002ba20 comp rx=0 tx=0).stop 2024-03-24T00:41:33.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.620+0000 7f02cee27640 1 -- 172.21.15.67:0/1466114210 shutdown_connections 2024-03-24T00:41:33.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.620+0000 7f02cee27640 1 --2- 172.21.15.67:0/1466114210 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02c80b41e0 0x7f02c80b3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:33.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.620+0000 7f02cee27640 1 --2- 172.21.15.67:0/1466114210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02c80b3b60 0x7f02c80cc6d0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:33.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.620+0000 7f02cee27640 1 -- 172.21.15.67:0/1466114210 >> 172.21.15.67:0/1466114210 conn(0x7f02c806fcf0 msgr2=0x7f02c80700d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:33.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.620+0000 7f02cee27640 1 -- 172.21.15.67:0/1466114210 shutdown_connections 2024-03-24T00:41:33.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.620+0000 7f02cee27640 1 -- 172.21.15.67:0/1466114210 wait complete. 2024-03-24T00:41:33.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.621+0000 7f02cee27640 1 Processor -- start 2024-03-24T00:41:33.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.621+0000 7f02cee27640 1 -- start start 2024-03-24T00:41:33.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.622+0000 7f02cee27640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02c80b3b60 0x7f02c8154f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:33.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.622+0000 7f02cee27640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02c80b41e0 0x7f02c8158e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:33.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.622+0000 7f02cee27640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02c815dfd0 0x7f02c815b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:33.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.622+0000 7f02cee27640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c815e3b0 con 0x7f02c815dfd0 2024-03-24T00:41:33.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.622+0000 7f02cee27640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c80cedb0 con 0x7f02c80b3b60 2024-03-24T00:41:33.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.622+0000 7f02cee27640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02c80ccc10 con 0x7f02c80b41e0 2024-03-24T00:41:33.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.622+0000 7f02cd3c4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02c815dfd0 0x7f02c815b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:33.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.622+0000 7f02cd3c4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02c815dfd0 0x7f02c815b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43666/0 (socket says 172.21.15.67:43666) 2024-03-24T00:41:33.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.622+0000 7f02cd3c4640 1 -- 172.21.15.67:0/3905335905 learned_addr learned my addr 172.21.15.67:0/3905335905 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:33.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.622+0000 7f02ccbc3640 1 --2- 172.21.15.67:0/3905335905 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02c80b3b60 0x7f02c8154f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:33.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.622+0000 7f02c7fff640 1 --2- 172.21.15.67:0/3905335905 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02c80b41e0 0x7f02c8158e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:33.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.623+0000 7f02cd3c4640 1 -- 172.21.15.67:0/3905335905 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02c80b41e0 msgr2=0x7f02c8158e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:33.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.623+0000 7f02cd3c4640 1 --2- 172.21.15.67:0/3905335905 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02c80b41e0 0x7f02c8158e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:33.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.623+0000 7f02cd3c4640 1 -- 172.21.15.67:0/3905335905 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02c80b3b60 msgr2=0x7f02c8154f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:33.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.623+0000 7f02cd3c4640 1 --2- 172.21.15.67:0/3905335905 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02c80b3b60 0x7f02c8154f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:33.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.623+0000 7f02cd3c4640 1 -- 172.21.15.67:0/3905335905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02c8159a00 con 0x7f02c815dfd0 2024-03-24T00:41:33.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.623+0000 7f02cd3c4640 1 --2- 172.21.15.67:0/3905335905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02c815dfd0 0x7f02c815b330 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f02c0009870 tx=0x7f02c0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:33.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.623+0000 7f02c57fa640 1 -- 172.21.15.67:0/3905335905 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c0037d10 con 0x7f02c815dfd0 2024-03-24T00:41:33.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.623+0000 7f02c57fa640 1 -- 172.21.15.67:0/3905335905 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02c0046070 con 0x7f02c815dfd0 2024-03-24T00:41:33.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.623+0000 7f02c57fa640 1 -- 172.21.15.67:0/3905335905 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c0041640 con 0x7f02c815dfd0 2024-03-24T00:41:33.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.623+0000 7f02cee27640 1 -- 172.21.15.67:0/3905335905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02c8159ce0 con 0x7f02c815dfd0 2024-03-24T00:41:33.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.623+0000 7f02cee27640 1 -- 172.21.15.67:0/3905335905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02c80c9e80 con 0x7f02c815dfd0 2024-03-24T00:41:33.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.624+0000 7f02cee27640 1 -- 172.21.15.67:0/3905335905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02c806d6b0 con 0x7f02c815dfd0 2024-03-24T00:41:33.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.624+0000 7f02c57fa640 1 -- 172.21.15.67:0/3905335905 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f02c0040070 con 0x7f02c815dfd0 2024-03-24T00:41:33.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.625+0000 7f02c57fa640 1 --2- 172.21.15.67:0/3905335905 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f02a806bbd0 0x7f02a806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:33.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.625+0000 7f02c57fa640 1 -- 172.21.15.67:0/3905335905 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f02c00ace20 con 0x7f02c815dfd0 2024-03-24T00:41:33.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.625+0000 7f02ccbc3640 1 --2- 172.21.15.67:0/3905335905 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f02a806bbd0 0x7f02a806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:33.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.626+0000 7f02ccbc3640 1 --2- 172.21.15.67:0/3905335905 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f02a806bbd0 0x7f02a806e090 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f02b00016e0 tx=0x7f02b0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:33.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.629+0000 7f02c57fa640 1 -- 172.21.15.67:0/3905335905 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f02c806d6b0 con 0x7f02c815dfd0 2024-03-24T00:41:33.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.809+0000 7f02cee27640 1 -- 172.21.15.67:0/3905335905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f02c804f400 con 0x7f02c815dfd0 2024-03-24T00:41:33.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.813+0000 7f02c57fa640 1 -- 172.21.15.67:0/3905335905 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) v1 ==== 104+0+0 (secure 0 0 0) 0x7f02c0033030 con 0x7f02c815dfd0 2024-03-24T00:41:33.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.815+0000 7f02cee27640 1 -- 172.21.15.67:0/3905335905 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f02a806bbd0 msgr2=0x7f02a806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:33.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.815+0000 7f02cee27640 1 --2- 172.21.15.67:0/3905335905 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f02a806bbd0 0x7f02a806e090 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f02b00016e0 tx=0x7f02b0009040 comp rx=0 tx=0).stop 2024-03-24T00:41:33.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.815+0000 7f02cee27640 1 -- 172.21.15.67:0/3905335905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02c815dfd0 msgr2=0x7f02c815b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:33.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.815+0000 7f02cee27640 1 --2- 172.21.15.67:0/3905335905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02c815dfd0 0x7f02c815b330 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f02c0009870 tx=0x7f02c0004730 comp rx=0 tx=0).stop 2024-03-24T00:41:33.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.815+0000 7f02cee27640 1 -- 172.21.15.67:0/3905335905 shutdown_connections 2024-03-24T00:41:33.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.815+0000 7f02cee27640 1 --2- 172.21.15.67:0/3905335905 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f02a806bbd0 0x7f02a806e090 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:33.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.815+0000 7f02cee27640 1 --2- 172.21.15.67:0/3905335905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02c815dfd0 0x7f02c815b330 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:33.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.815+0000 7f02cee27640 1 --2- 172.21.15.67:0/3905335905 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02c80b41e0 0x7f02c8158e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:33.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.815+0000 7f02cee27640 1 --2- 172.21.15.67:0/3905335905 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02c80b3b60 0x7f02c8154f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:33.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.815+0000 7f02cee27640 1 -- 172.21.15.67:0/3905335905 >> 172.21.15.67:0/3905335905 conn(0x7f02c806fcf0 msgr2=0x7f02c80b3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:33.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.815+0000 7f02cee27640 1 -- 172.21.15.67:0/3905335905 shutdown_connections 2024-03-24T00:41:33.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.815+0000 7f02cee27640 1 -- 172.21.15.67:0/3905335905 wait complete. 2024-03-24T00:41:33.826 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_backtrace_repair' 2024-03-24T00:41:33.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.927+0000 7f03a94ea640 1 Processor -- start 2024-03-24T00:41:33.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.927+0000 7f03a94ea640 1 -- start start 2024-03-24T00:41:33.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.928+0000 7f03a94ea640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f03a415e1b0 0x7f03a40cfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:33.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.928+0000 7f03a94ea640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f03a40b43a0 0x7f03a4099d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:33.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.928+0000 7f03a94ea640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03a409a650 con 0x7f03a409a240 2024-03-24T00:41:33.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.928+0000 7f03a94ea640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03a415e590 con 0x7f03a40b43a0 2024-03-24T00:41:33.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.928+0000 7f03a94ea640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03a40d1820 con 0x7f03a415e1b0 2024-03-24T00:41:33.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.928+0000 7f03a2ffd640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f03a415e1b0 0x7f03a40cfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:33.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.928+0000 7f03a27fc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f03a40b43a0 0x7f03a4099d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:33.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.928+0000 7f03a37fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f03a409a240 0x7f03a406c360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:49618/0 (socket says 172.21.15.67:49618) 2024-03-24T00:41:33.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.928+0000 7f03a37fe640 1 -- 172.21.15.67:0/2388971254 learned_addr learned my addr 172.21.15.67:0/2388971254 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:41:33.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.928+0000 7f03a1ffb640 1 -- 172.21.15.67:0/2388971254 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 123277527 0 0) 0x7f0394001490 con 0x7f03a409a240 2024-03-24T00:41:33.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.928+0000 7f03a1ffb640 1 -- 172.21.15.67:0/2388971254 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0390003520 con 0x7f03a409a240 2024-03-24T00:41:33.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.928+0000 7f03a2ffd640 1 -- 172.21.15.67:0/2388971254 >> v2:172.21.15.72:3300/0 conn(0x7f03a40b43a0 msgr2=0x7f03a4099d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:33.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.928+0000 7f03a2ffd640 1 --2- 172.21.15.67:0/2388971254 >> v2:172.21.15.72:3300/0 conn(0x7f03a40b43a0 0x7f03a4099d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:33.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.928+0000 7f03a2ffd640 1 -- 172.21.15.67:0/2388971254 >> v1:172.21.15.72:6789/0 conn(0x7f03a409a240 legacy=0x7f03a406c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:33.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.928+0000 7f03a2ffd640 1 -- 172.21.15.67:0/2388971254 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03a40d0fc0 con 0x7f03a415e1b0 2024-03-24T00:41:33.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.928+0000 7f03a27fc640 1 --2- 172.21.15.67:0/2388971254 >> v2:172.21.15.72:3300/0 conn(0x7f03a40b43a0 0x7f03a4099d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:41:33.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.929+0000 7f03a2ffd640 1 --2- 172.21.15.67:0/2388971254 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f03a415e1b0 0x7f03a40cfe20 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f039c0097a0 tx=0x7f039c02b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b875eb06295d0dc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:33.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.929+0000 7f03a1ffb640 1 -- 172.21.15.67:0/2388971254 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f039c038070 con 0x7f03a415e1b0 2024-03-24T00:41:33.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.929+0000 7f03a1ffb640 1 -- 172.21.15.67:0/2388971254 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f039c031700 con 0x7f03a415e1b0 2024-03-24T00:41:33.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.929+0000 7f03a1ffb640 1 -- 172.21.15.67:0/2388971254 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f039c031ab0 con 0x7f03a415e1b0 2024-03-24T00:41:33.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.930+0000 7f03a94ea640 1 -- 172.21.15.67:0/2388971254 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f03a415e1b0 msgr2=0x7f03a40cfe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:33.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.930+0000 7f03a94ea640 1 --2- 172.21.15.67:0/2388971254 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f03a415e1b0 0x7f03a40cfe20 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f039c0097a0 tx=0x7f039c02b7a0 comp rx=0 tx=0).stop 2024-03-24T00:41:33.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.930+0000 7f03a94ea640 1 -- 172.21.15.67:0/2388971254 shutdown_connections 2024-03-24T00:41:33.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.930+0000 7f03a94ea640 1 --2- 172.21.15.67:0/2388971254 >> v2:172.21.15.72:3300/0 conn(0x7f03a40b43a0 0x7f03a4099d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:33.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.930+0000 7f03a94ea640 1 --2- 172.21.15.67:0/2388971254 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f03a415e1b0 0x7f03a40cfe20 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:33.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.930+0000 7f03a94ea640 1 -- 172.21.15.67:0/2388971254 >> 172.21.15.67:0/2388971254 conn(0x7f03a40b36d0 msgr2=0x7f03a40b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:33.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.930+0000 7f03a94ea640 1 -- 172.21.15.67:0/2388971254 shutdown_connections 2024-03-24T00:41:33.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.930+0000 7f03a94ea640 1 -- 172.21.15.67:0/2388971254 wait complete. 2024-03-24T00:41:33.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.931+0000 7f03a94ea640 1 Processor -- start 2024-03-24T00:41:33.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.931+0000 7f03a94ea640 1 -- start start 2024-03-24T00:41:33.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.932+0000 7f03a94ea640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03a409a240 0x7f03a41601b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:33.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.932+0000 7f03a94ea640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f03a40b43a0 0x7f03a4070340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:33.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.932+0000 7f03a94ea640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03a415e1b0 0x7f03a4070880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:33.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.932+0000 7f03a94ea640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03a40cc360 con 0x7f03a415e1b0 2024-03-24T00:41:33.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.932+0000 7f03a94ea640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03a40d1540 con 0x7f03a409a240 2024-03-24T00:41:33.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.932+0000 7f03a94ea640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03a409a9a0 con 0x7f03a40b43a0 2024-03-24T00:41:33.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.932+0000 7f03a37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03a415e1b0 0x7f03a4070880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:33.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.932+0000 7f03a37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03a415e1b0 0x7f03a4070880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43668/0 (socket says 172.21.15.67:43668) 2024-03-24T00:41:33.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.932+0000 7f03a37fe640 1 -- 172.21.15.67:0/3349453080 learned_addr learned my addr 172.21.15.67:0/3349453080 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:33.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.932+0000 7f03a2ffd640 1 --2- 172.21.15.67:0/3349453080 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03a409a240 0x7f03a41601b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:33.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.932+0000 7f03a27fc640 1 --2- 172.21.15.67:0/3349453080 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f03a40b43a0 0x7f03a4070340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:33.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.932+0000 7f03a37fe640 1 -- 172.21.15.67:0/3349453080 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f03a40b43a0 msgr2=0x7f03a4070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:33.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.932+0000 7f03a37fe640 1 --2- 172.21.15.67:0/3349453080 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f03a40b43a0 0x7f03a4070340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:33.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.932+0000 7f03a37fe640 1 -- 172.21.15.67:0/3349453080 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03a409a240 msgr2=0x7f03a41601b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:33.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.932+0000 7f03a37fe640 1 --2- 172.21.15.67:0/3349453080 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03a409a240 0x7f03a41601b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:33.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.932+0000 7f03a37fe640 1 -- 172.21.15.67:0/3349453080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03a4160c70 con 0x7f03a415e1b0 2024-03-24T00:41:33.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.932+0000 7f03a2ffd640 1 --2- 172.21.15.67:0/3349453080 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03a409a240 0x7f03a41601b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:41:33.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.932+0000 7f03a27fc640 1 --2- 172.21.15.67:0/3349453080 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f03a40b43a0 0x7f03a4070340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:41:33.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.933+0000 7f03a37fe640 1 --2- 172.21.15.67:0/3349453080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03a415e1b0 0x7f03a4070880 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f039400cb40 tx=0x7f039400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:33.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.933+0000 7f039b7fe640 1 -- 172.21.15.67:0/3349453080 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0394017070 con 0x7f03a415e1b0 2024-03-24T00:41:33.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.933+0000 7f039b7fe640 1 -- 172.21.15.67:0/3349453080 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f039400ac40 con 0x7f03a415e1b0 2024-03-24T00:41:33.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.933+0000 7f03a94ea640 1 -- 172.21.15.67:0/3349453080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03a41f73d0 con 0x7f03a415e1b0 2024-03-24T00:41:33.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.933+0000 7f039b7fe640 1 -- 172.21.15.67:0/3349453080 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0394012400 con 0x7f03a415e1b0 2024-03-24T00:41:33.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.933+0000 7f03a94ea640 1 -- 172.21.15.67:0/3349453080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03a41f7990 con 0x7f03a415e1b0 2024-03-24T00:41:33.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.934+0000 7f03a94ea640 1 -- 172.21.15.67:0/3349453080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03a404f400 con 0x7f03a415e1b0 2024-03-24T00:41:33.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.934+0000 7f039b7fe640 1 -- 172.21.15.67:0/3349453080 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f039401a070 con 0x7f03a415e1b0 2024-03-24T00:41:33.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.935+0000 7f039b7fe640 1 --2- 172.21.15.67:0/3349453080 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f038406bbd0 0x7f038406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:33.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.935+0000 7f039b7fe640 1 -- 172.21.15.67:0/3349453080 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f039408a6f0 con 0x7f03a415e1b0 2024-03-24T00:41:33.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.935+0000 7f03a2ffd640 1 --2- 172.21.15.67:0/3349453080 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f038406bbd0 0x7f038406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:33.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.935+0000 7f03a2ffd640 1 --2- 172.21.15.67:0/3349453080 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f038406bbd0 0x7f038406e090 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f039c035000 tx=0x7f039c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:33.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:33.939+0000 7f039b7fe640 1 -- 172.21.15.67:0/3349453080 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f03a409a650 con 0x7f03a415e1b0 2024-03-24T00:41:34.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.195+0000 7f03a94ea640 1 -- 172.21.15.67:0/3349453080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_backtrace_repair"]} v 0) v1 -- 0x7f03a40b59b0 con 0x7f03a415e1b0 2024-03-24T00:41:34.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.487+0000 7f039b7fe640 1 -- 172.21.15.67:0/3349453080 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_backtrace_repair"]}]=0 v159) v1 ==== 149+0+0 (secure 0 0 0) 0x7f039408a070 con 0x7f03a415e1b0 2024-03-24T00:41:34.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.490+0000 7f03a94ea640 1 -- 172.21.15.67:0/3349453080 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f038406bbd0 msgr2=0x7f038406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:34.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.490+0000 7f03a94ea640 1 --2- 172.21.15.67:0/3349453080 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f038406bbd0 0x7f038406e090 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f039c035000 tx=0x7f039c008030 comp rx=0 tx=0).stop 2024-03-24T00:41:34.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.490+0000 7f03a94ea640 1 -- 172.21.15.67:0/3349453080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03a415e1b0 msgr2=0x7f03a4070880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:34.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.490+0000 7f03a94ea640 1 --2- 172.21.15.67:0/3349453080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03a415e1b0 0x7f03a4070880 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f039400cb40 tx=0x7f039400a060 comp rx=0 tx=0).stop 2024-03-24T00:41:34.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.490+0000 7f03a94ea640 1 -- 172.21.15.67:0/3349453080 shutdown_connections 2024-03-24T00:41:34.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.490+0000 7f03a94ea640 1 --2- 172.21.15.67:0/3349453080 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f038406bbd0 0x7f038406e090 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:34.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.490+0000 7f03a94ea640 1 --2- 172.21.15.67:0/3349453080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03a415e1b0 0x7f03a4070880 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:34.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.490+0000 7f03a94ea640 1 --2- 172.21.15.67:0/3349453080 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f03a40b43a0 0x7f03a4070340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:34.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.490+0000 7f03a94ea640 1 --2- 172.21.15.67:0/3349453080 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03a409a240 0x7f03a41601b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:34.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.490+0000 7f03a94ea640 1 -- 172.21.15.67:0/3349453080 >> 172.21.15.67:0/3349453080 conn(0x7f03a40b36d0 msgr2=0x7f03a415d250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:34.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.490+0000 7f03a94ea640 1 -- 172.21.15.67:0/3349453080 shutdown_connections 2024-03-24T00:41:34.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.490+0000 7f03a94ea640 1 -- 172.21.15.67:0/3349453080 wait complete. 2024-03-24T00:41:34.501 INFO:tasks.cephfs_test_runner:That the MDS can repair an inodes backtrace in the data pool ... ok 2024-03-24T00:41:34.502 INFO:tasks.cephfs_test_runner:Starting test: test_health_status_after_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) Test that the damage health status is cleared 2024-03-24T00:41:34.502 INFO:tasks.cephfs_test_runner:test_health_status_after_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2024-03-24T00:41:34.502 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:41:34.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.604+0000 7f2689bc6640 1 Processor -- start 2024-03-24T00:41:34.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.604+0000 7f2689bc6640 1 -- start start 2024-03-24T00:41:34.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.605+0000 7f2689bc6640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f26840cf220 0x7f26840d2660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:34.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.605+0000 7f2689bc6640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2684162120 con 0x7f2684161bb0 2024-03-24T00:41:34.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.605+0000 7f2689bc6640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2684162400 con 0x7f268415dfe0 2024-03-24T00:41:34.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.605+0000 7f2689bc6640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26840d21d0 con 0x7f26840cf220 2024-03-24T00:41:34.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.605+0000 7f2683fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2684161bb0 0x7f26840d2ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58334/0 (socket says 172.21.15.67:58334) 2024-03-24T00:41:34.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.605+0000 7f2683fff640 1 -- 172.21.15.67:0/274955653 learned_addr learned my addr 172.21.15.67:0/274955653 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:41:34.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.605+0000 7f2682ffd640 1 --2- 172.21.15.67:0/274955653 >> v2:172.21.15.72:3300/0 conn(0x7f26840cf220 0x7f26840d2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:34.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.605+0000 7f26827fc640 1 -- 172.21.15.67:0/274955653 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2321288089 0 0) 0x7f2684162120 con 0x7f2684161bb0 2024-03-24T00:41:34.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.605+0000 7f26827fc640 1 -- 172.21.15.67:0/274955653 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f266c003520 con 0x7f2684161bb0 2024-03-24T00:41:34.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.606+0000 7f26827fc640 1 -- 172.21.15.67:0/274955653 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2968466178 0 0) 0x7f266c003520 con 0x7f2684161bb0 2024-03-24T00:41:34.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.606+0000 7f2682ffd640 1 -- 172.21.15.67:0/274955653 >> v1:172.21.15.72:6789/0 conn(0x7f268415dfe0 legacy=0x7f26840ceb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:34.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.606+0000 7f2682ffd640 1 -- 172.21.15.67:0/274955653 >> v1:172.21.15.67:6789/0 conn(0x7f2684161bb0 legacy=0x7f26840d2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:34.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.606+0000 7f2682ffd640 1 -- 172.21.15.67:0/274955653 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26841626e0 con 0x7f26840cf220 2024-03-24T00:41:34.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.606+0000 7f2682ffd640 1 --2- 172.21.15.67:0/274955653 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f26840cf220 0x7f26840d2660 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f26780097a0 tx=0x7f267802b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5be625f255f37476 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:34.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.607+0000 7f26827fc640 1 -- 172.21.15.67:0/274955653 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2678038070 con 0x7f26840cf220 2024-03-24T00:41:34.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.607+0000 7f26827fc640 1 -- 172.21.15.67:0/274955653 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2678031500 con 0x7f26840cf220 2024-03-24T00:41:34.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.607+0000 7f26827fc640 1 -- 172.21.15.67:0/274955653 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f267c0019f0 con 0x7f26840cf220 2024-03-24T00:41:34.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.607+0000 7f2689bc6640 1 -- 172.21.15.67:0/274955653 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f26840cf220 msgr2=0x7f26840d2660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:34.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.607+0000 7f2689bc6640 1 --2- 172.21.15.67:0/274955653 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f26840cf220 0x7f26840d2660 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f26780097a0 tx=0x7f267802b820 comp rx=0 tx=0).stop 2024-03-24T00:41:34.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.607+0000 7f2689bc6640 1 -- 172.21.15.67:0/274955653 shutdown_connections 2024-03-24T00:41:34.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.607+0000 7f2689bc6640 1 --2- 172.21.15.67:0/274955653 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f26840cf220 0x7f26840d2660 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:34.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.607+0000 7f2689bc6640 1 -- 172.21.15.67:0/274955653 >> 172.21.15.67:0/274955653 conn(0x7f268406c890 msgr2=0x7f268406cc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:34.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.607+0000 7f2689bc6640 1 -- 172.21.15.67:0/274955653 shutdown_connections 2024-03-24T00:41:34.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.607+0000 7f2689bc6640 1 -- 172.21.15.67:0/274955653 wait complete. 2024-03-24T00:41:34.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.608+0000 7f2689bc6640 1 Processor -- start 2024-03-24T00:41:34.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.609+0000 7f2689bc6640 1 -- start start 2024-03-24T00:41:34.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.609+0000 7f2689bc6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26840cf220 0x7f26840cb7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:34.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.609+0000 7f2689bc6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f268415dfe0 0x7f268415b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:34.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.609+0000 7f2689bc6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2684161bb0 0x7f26840b5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:34.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.609+0000 7f2689bc6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26840d1120 con 0x7f26840cf220 2024-03-24T00:41:34.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.609+0000 7f2689bc6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f268415e3c0 con 0x7f2684161bb0 2024-03-24T00:41:34.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.609+0000 7f2689bc6640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26840d12a0 con 0x7f268415dfe0 2024-03-24T00:41:34.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.609+0000 7f26837fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26840cf220 0x7f26840cb7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:34.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.609+0000 7f26837fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26840cf220 0x7f26840cb7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43674/0 (socket says 172.21.15.67:43674) 2024-03-24T00:41:34.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.609+0000 7f26837fe640 1 -- 172.21.15.67:0/420769882 learned_addr learned my addr 172.21.15.67:0/420769882 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:34.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.609+0000 7f2683fff640 1 --2- 172.21.15.67:0/420769882 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2684161bb0 0x7f26840b5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:34.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.609+0000 7f2682ffd640 1 --2- 172.21.15.67:0/420769882 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f268415dfe0 0x7f268415b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:34.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.610+0000 7f26837fe640 1 -- 172.21.15.67:0/420769882 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f268415dfe0 msgr2=0x7f268415b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:34.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.610+0000 7f26837fe640 1 --2- 172.21.15.67:0/420769882 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f268415dfe0 0x7f268415b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:34.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.610+0000 7f26837fe640 1 -- 172.21.15.67:0/420769882 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2684161bb0 msgr2=0x7f26840b5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:34.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.610+0000 7f26837fe640 1 --2- 172.21.15.67:0/420769882 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2684161bb0 0x7f26840b5800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:34.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.610+0000 7f26837fe640 1 -- 172.21.15.67:0/420769882 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26840cbe00 con 0x7f26840cf220 2024-03-24T00:41:34.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.610+0000 7f26837fe640 1 --2- 172.21.15.67:0/420769882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26840cf220 0x7f26840cb7f0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f267400c480 tx=0x7f267400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:34.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.610+0000 7f266bfff640 1 -- 172.21.15.67:0/420769882 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26740052d0 con 0x7f26840cf220 2024-03-24T00:41:34.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.610+0000 7f266bfff640 1 -- 172.21.15.67:0/420769882 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2674005470 con 0x7f26840cf220 2024-03-24T00:41:34.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.610+0000 7f2689bc6640 1 -- 172.21.15.67:0/420769882 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26840cc0e0 con 0x7f26840cf220 2024-03-24T00:41:34.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.610+0000 7f266bfff640 1 -- 172.21.15.67:0/420769882 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2674012870 con 0x7f26840cf220 2024-03-24T00:41:34.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.610+0000 7f2689bc6640 1 -- 172.21.15.67:0/420769882 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26840cc6a0 con 0x7f26840cf220 2024-03-24T00:41:34.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.611+0000 7f2689bc6640 1 -- 172.21.15.67:0/420769882 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f268406b180 con 0x7f26840cf220 2024-03-24T00:41:34.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.611+0000 7f266bfff640 1 -- 172.21.15.67:0/420769882 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2674012a10 con 0x7f26840cf220 2024-03-24T00:41:34.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.612+0000 7f266bfff640 1 --2- 172.21.15.67:0/420769882 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f267006bbd0 0x7f267006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:34.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.612+0000 7f266bfff640 1 -- 172.21.15.67:0/420769882 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f267408a450 con 0x7f26840cf220 2024-03-24T00:41:34.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.612+0000 7f2682ffd640 1 --2- 172.21.15.67:0/420769882 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f267006bbd0 0x7f267006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:34.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.613+0000 7f2682ffd640 1 --2- 172.21.15.67:0/420769882 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f267006bbd0 0x7f267006e090 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f2678035000 tx=0x7f2678008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:34.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.616+0000 7f266bfff640 1 -- 172.21.15.67:0/420769882 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2674059140 con 0x7f26840cf220 2024-03-24T00:41:34.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.800+0000 7f2689bc6640 1 -- 172.21.15.67:0/420769882 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f26840d1a10 con 0x7f26840cf220 2024-03-24T00:41:34.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.800+0000 7f266bfff640 1 -- 172.21.15.67:0/420769882 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v53) v1 ==== 74+0+13233 (secure 0 0 0) 0x7f2674058a40 con 0x7f26840cf220 2024-03-24T00:41:34.801 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:41:34.801 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":53,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:41:33.482933+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6834/2741818214":"2024-03-25T00:41:30.202181+0000","172.21.15.67:6832/2741818214":"2024-03-25T00:41:30.202181+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:41:34.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.802+0000 7f2689bc6640 1 -- 172.21.15.67:0/420769882 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f267006bbd0 msgr2=0x7f267006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:34.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.802+0000 7f2689bc6640 1 --2- 172.21.15.67:0/420769882 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f267006bbd0 0x7f267006e090 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f2678035000 tx=0x7f2678008030 comp rx=0 tx=0).stop 2024-03-24T00:41:34.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.802+0000 7f2689bc6640 1 -- 172.21.15.67:0/420769882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26840cf220 msgr2=0x7f26840cb7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:34.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.802+0000 7f2689bc6640 1 --2- 172.21.15.67:0/420769882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26840cf220 0x7f26840cb7f0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f267400c480 tx=0x7f267400c940 comp rx=0 tx=0).stop 2024-03-24T00:41:34.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.802+0000 7f2689bc6640 1 -- 172.21.15.67:0/420769882 shutdown_connections 2024-03-24T00:41:34.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.802+0000 7f2689bc6640 1 --2- 172.21.15.67:0/420769882 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f267006bbd0 0x7f267006e090 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:34.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.802+0000 7f2689bc6640 1 --2- 172.21.15.67:0/420769882 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2684161bb0 0x7f26840b5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:34.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.802+0000 7f2689bc6640 1 --2- 172.21.15.67:0/420769882 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f268415dfe0 0x7f268415b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:34.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.802+0000 7f2689bc6640 1 --2- 172.21.15.67:0/420769882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26840cf220 0x7f26840cb7f0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:34.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.802+0000 7f2689bc6640 1 -- 172.21.15.67:0/420769882 >> 172.21.15.67:0/420769882 conn(0x7f268406c890 msgr2=0x7f26840d2560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:34.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.803+0000 7f2689bc6640 1 -- 172.21.15.67:0/420769882 shutdown_connections 2024-03-24T00:41:34.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.803+0000 7f2689bc6640 1 -- 172.21.15.67:0/420769882 wait complete. 2024-03-24T00:41:34.814 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:38:39.364950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:41:34.814 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:41:34.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.917+0000 7ff567bbc640 1 Processor -- start 2024-03-24T00:41:34.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.918+0000 7ff567bbc640 1 -- start start 2024-03-24T00:41:34.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.918+0000 7ff567bbc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff5600b2610 0x7ff56015e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:34.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.918+0000 7ff567bbc640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5600b12f0 con 0x7ff5600d22a0 2024-03-24T00:41:34.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.918+0000 7ff567bbc640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5600b2c50 con 0x7ff5600b0f10 2024-03-24T00:41:34.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.918+0000 7ff567bbc640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5600d09f0 con 0x7ff5600b2610 2024-03-24T00:41:34.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.919+0000 7ff566159640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff5600b2610 0x7ff56015e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:34.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.919+0000 7ff565157640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff5600d22a0 0x7ff5600b1f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58346/0 (socket says 172.21.15.67:58346) 2024-03-24T00:41:34.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.919+0000 7ff565157640 1 -- 172.21.15.67:0/46103818 learned_addr learned my addr 172.21.15.67:0/46103818 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:41:34.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.919+0000 7ff564956640 1 -- 172.21.15.67:0/46103818 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1909050315 0 0) 0x7ff5600b12f0 con 0x7ff5600d22a0 2024-03-24T00:41:34.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.919+0000 7ff564956640 1 -- 172.21.15.67:0/46103818 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff54c003520 con 0x7ff5600d22a0 2024-03-24T00:41:34.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.919+0000 7ff566159640 1 -- 172.21.15.67:0/46103818 >> v1:172.21.15.72:6789/0 conn(0x7ff5600b0f10 legacy=0x7ff5600d1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:34.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.919+0000 7ff566159640 1 -- 172.21.15.67:0/46103818 >> v1:172.21.15.67:6789/0 conn(0x7ff5600d22a0 legacy=0x7ff5600b1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:34.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.919+0000 7ff566159640 1 -- 172.21.15.67:0/46103818 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5600ce880 con 0x7ff5600b2610 2024-03-24T00:41:34.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.920+0000 7ff566159640 1 --2- 172.21.15.67:0/46103818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5600b2610 0x7ff56015e880 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7ff558009d90 tx=0x7ff55802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a9d7b25a78c2311 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:34.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.920+0000 7ff564956640 1 -- 172.21.15.67:0/46103818 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff558039070 con 0x7ff5600b2610 2024-03-24T00:41:34.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.920+0000 7ff564956640 1 -- 172.21.15.67:0/46103818 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff55802cc60 con 0x7ff5600b2610 2024-03-24T00:41:34.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.920+0000 7ff564956640 1 -- 172.21.15.67:0/46103818 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff558034660 con 0x7ff5600b2610 2024-03-24T00:41:34.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.920+0000 7ff567bbc640 1 -- 172.21.15.67:0/46103818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5600b2610 msgr2=0x7ff56015e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:34.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.920+0000 7ff567bbc640 1 --2- 172.21.15.67:0/46103818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5600b2610 0x7ff56015e880 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7ff558009d90 tx=0x7ff55802beb0 comp rx=0 tx=0).stop 2024-03-24T00:41:34.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.920+0000 7ff567bbc640 1 -- 172.21.15.67:0/46103818 shutdown_connections 2024-03-24T00:41:34.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.920+0000 7ff567bbc640 1 --2- 172.21.15.67:0/46103818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5600b2610 0x7ff56015e880 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:34.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.920+0000 7ff567bbc640 1 -- 172.21.15.67:0/46103818 >> 172.21.15.67:0/46103818 conn(0x7ff56008b1b0 msgr2=0x7ff56008b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:34.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.920+0000 7ff567bbc640 1 -- 172.21.15.67:0/46103818 shutdown_connections 2024-03-24T00:41:34.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.920+0000 7ff567bbc640 1 -- 172.21.15.67:0/46103818 wait complete. 2024-03-24T00:41:34.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.922+0000 7ff567bbc640 1 Processor -- start 2024-03-24T00:41:34.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.922+0000 7ff567bbc640 1 -- start start 2024-03-24T00:41:34.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.922+0000 7ff567bbc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5600b0f10 0x7ff5600c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:34.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.922+0000 7ff567bbc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff5600b2610 0x7ff5600c8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:34.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.922+0000 7ff567bbc640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff5600d22a0 0x7ff56015de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:34.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.922+0000 7ff567bbc640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5600cee40 con 0x7ff5600b0f10 2024-03-24T00:41:34.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.922+0000 7ff567bbc640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5600ce560 con 0x7ff5600b2610 2024-03-24T00:41:34.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.922+0000 7ff567bbc640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5600cf360 con 0x7ff5600d22a0 2024-03-24T00:41:34.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.922+0000 7ff565958640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5600b0f10 0x7ff5600c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:34.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.922+0000 7ff565958640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5600b0f10 0x7ff5600c7df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43684/0 (socket says 172.21.15.67:43684) 2024-03-24T00:41:34.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.922+0000 7ff565958640 1 -- 172.21.15.67:0/1082659238 learned_addr learned my addr 172.21.15.67:0/1082659238 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:34.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.923+0000 7ff565157640 1 --2- 172.21.15.67:0/1082659238 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff5600b2610 0x7ff5600c8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:34.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.923+0000 7ff566159640 1 --2- 172.21.15.67:0/1082659238 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff5600d22a0 0x7ff56015de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:34.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.923+0000 7ff565958640 1 -- 172.21.15.67:0/1082659238 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff5600d22a0 msgr2=0x7ff56015de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:34.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.923+0000 7ff565958640 1 --2- 172.21.15.67:0/1082659238 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff5600d22a0 0x7ff56015de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:34.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.923+0000 7ff565958640 1 -- 172.21.15.67:0/1082659238 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff5600b2610 msgr2=0x7ff5600c8ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:34.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.923+0000 7ff565958640 1 --2- 172.21.15.67:0/1082659238 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff5600b2610 0x7ff5600c8ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:34.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.923+0000 7ff565958640 1 -- 172.21.15.67:0/1082659238 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5600c9430 con 0x7ff5600b0f10 2024-03-24T00:41:34.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.923+0000 7ff565958640 1 --2- 172.21.15.67:0/1082659238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5600b0f10 0x7ff5600c7df0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7ff55c00b420 tx=0x7ff55c00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:34.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.923+0000 7ff5567fc640 1 -- 172.21.15.67:0/1082659238 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff55c011830 con 0x7ff5600b0f10 2024-03-24T00:41:34.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.923+0000 7ff5567fc640 1 -- 172.21.15.67:0/1082659238 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff55c016070 con 0x7ff5600b0f10 2024-03-24T00:41:34.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.923+0000 7ff567bbc640 1 -- 172.21.15.67:0/1082659238 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5600c8480 con 0x7ff5600b0f10 2024-03-24T00:41:34.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.923+0000 7ff5567fc640 1 -- 172.21.15.67:0/1082659238 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff55c011c10 con 0x7ff5600b0f10 2024-03-24T00:41:34.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.923+0000 7ff567bbc640 1 -- 172.21.15.67:0/1082659238 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5600c9c70 con 0x7ff5600b0f10 2024-03-24T00:41:34.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.924+0000 7ff567bbc640 1 -- 172.21.15.67:0/1082659238 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff56004f400 con 0x7ff5600b0f10 2024-03-24T00:41:34.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.925+0000 7ff5567fc640 1 -- 172.21.15.67:0/1082659238 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff55c01b460 con 0x7ff5600b0f10 2024-03-24T00:41:34.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.925+0000 7ff5567fc640 1 --2- 172.21.15.67:0/1082659238 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff54406bc00 0x7ff54406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:34.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.925+0000 7ff5567fc640 1 -- 172.21.15.67:0/1082659238 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5335+0+0 (secure 0 0 0) 0x7ff55c08a2c0 con 0x7ff5600b0f10 2024-03-24T00:41:34.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.925+0000 7ff565157640 1 --2- 172.21.15.67:0/1082659238 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff54406bc00 0x7ff54406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:34.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.926+0000 7ff565157640 1 --2- 172.21.15.67:0/1082659238 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff54406bc00 0x7ff54406e0c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7ff5600b7c30 tx=0x7ff550009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:34.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:34.929+0000 7ff5567fc640 1 -- 172.21.15.67:0/1082659238 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff55c020020 con 0x7ff5600b0f10 2024-03-24T00:41:35.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.109+0000 7ff567bbc640 1 -- 172.21.15.67:0/1082659238 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff56008b600 con 0x7ff5600b0f10 2024-03-24T00:41:35.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.109+0000 7ff5567fc640 1 -- 172.21.15.67:0/1082659238 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v53) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff55c0588d0 con 0x7ff5600b0f10 2024-03-24T00:41:35.110 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-24T00:41:35.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.111+0000 7ff567bbc640 1 -- 172.21.15.67:0/1082659238 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff54406bc00 msgr2=0x7ff54406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:35.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.111+0000 7ff567bbc640 1 --2- 172.21.15.67:0/1082659238 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff54406bc00 0x7ff54406e0c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7ff5600b7c30 tx=0x7ff550009040 comp rx=0 tx=0).stop 2024-03-24T00:41:35.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.111+0000 7ff567bbc640 1 -- 172.21.15.67:0/1082659238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5600b0f10 msgr2=0x7ff5600c7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:35.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.111+0000 7ff567bbc640 1 --2- 172.21.15.67:0/1082659238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5600b0f10 0x7ff5600c7df0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7ff55c00b420 tx=0x7ff55c00b8e0 comp rx=0 tx=0).stop 2024-03-24T00:41:35.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.111+0000 7ff567bbc640 1 -- 172.21.15.67:0/1082659238 shutdown_connections 2024-03-24T00:41:35.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.111+0000 7ff567bbc640 1 --2- 172.21.15.67:0/1082659238 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff54406bc00 0x7ff54406e0c0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:35.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.111+0000 7ff567bbc640 1 --2- 172.21.15.67:0/1082659238 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff5600d22a0 0x7ff56015de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:35.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.111+0000 7ff567bbc640 1 --2- 172.21.15.67:0/1082659238 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff5600b2610 0x7ff5600c8ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:35.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.111+0000 7ff567bbc640 1 --2- 172.21.15.67:0/1082659238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5600b0f10 0x7ff5600c7df0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:35.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.111+0000 7ff567bbc640 1 -- 172.21.15.67:0/1082659238 >> 172.21.15.67:0/1082659238 conn(0x7ff56008b1b0 msgr2=0x7ff5600cd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:35.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.111+0000 7ff567bbc640 1 -- 172.21.15.67:0/1082659238 shutdown_connections 2024-03-24T00:41:35.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.111+0000 7ff567bbc640 1 -- 172.21.15.67:0/1082659238 wait complete. 2024-03-24T00:41:35.123 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_backtrace_repair' 2024-03-24T00:41:35.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.223+0000 7fd1d9db3640 1 Processor -- start 2024-03-24T00:41:35.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.223+0000 7fd1d9db3640 1 -- start start 2024-03-24T00:41:35.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.224+0000 7fd1d9db3640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd1d40ccce0 0x7fd1d40cd0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:35.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.224+0000 7fd1d9db3640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd1d406ecf0 0x7fd1d40cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:35.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.224+0000 7fd1d9db3640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1d40cd600 con 0x7fd1d40cef40 2024-03-24T00:41:35.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.224+0000 7fd1d9db3640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1d4072890 con 0x7fd1d406ecf0 2024-03-24T00:41:35.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.224+0000 7fd1d9db3640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d415c580 con 0x7fd1d40ccce0 2024-03-24T00:41:35.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.224+0000 7fd1d2ffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd1d40cef40 0x7fd1d415f8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58356/0 (socket says 172.21.15.67:58356) 2024-03-24T00:41:35.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.224+0000 7fd1d2ffd640 1 -- 172.21.15.67:0/2205113954 learned_addr learned my addr 172.21.15.67:0/2205113954 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:41:35.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.224+0000 7fd1d37fe640 1 --2- 172.21.15.67:0/2205113954 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd1d40ccce0 0x7fd1d40cd0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:35.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.224+0000 7fd1d3fff640 1 --2- 172.21.15.67:0/2205113954 >> v2:172.21.15.72:3300/0 conn(0x7fd1d406ecf0 0x7fd1d40cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:35.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.224+0000 7fd1d27fc640 1 -- 172.21.15.67:0/2205113954 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 912000173 0 0) 0x7fd1d40cd600 con 0x7fd1d40cef40 2024-03-24T00:41:35.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.224+0000 7fd1d27fc640 1 -- 172.21.15.67:0/2205113954 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1bc003520 con 0x7fd1d40cef40 2024-03-24T00:41:35.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.225+0000 7fd1d3fff640 1 -- 172.21.15.67:0/2205113954 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd1d40ccce0 msgr2=0x7fd1d40cd0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:35.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.225+0000 7fd1d3fff640 1 --2- 172.21.15.67:0/2205113954 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd1d40ccce0 0x7fd1d40cd0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:35.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.225+0000 7fd1d3fff640 1 -- 172.21.15.67:0/2205113954 >> v1:172.21.15.67:6789/0 conn(0x7fd1d40cef40 legacy=0x7fd1d415f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:35.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.225+0000 7fd1d27fc640 1 -- 172.21.15.67:0/2205113954 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 540100213 0 0) 0x7fd1bc003520 con 0x7fd1d40cef40 2024-03-24T00:41:35.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.225+0000 7fd1d3fff640 1 -- 172.21.15.67:0/2205113954 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d415c860 con 0x7fd1d406ecf0 2024-03-24T00:41:35.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.225+0000 7fd1d3fff640 1 --2- 172.21.15.67:0/2205113954 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1d406ecf0 0x7fd1d40cf3c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc009d90 tx=0x7fd1cc02bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=942e3222f904caa8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:35.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.226+0000 7fd1d27fc640 1 -- 172.21.15.67:0/2205113954 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1cc039070 con 0x7fd1d406ecf0 2024-03-24T00:41:35.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.226+0000 7fd1d27fc640 1 -- 172.21.15.67:0/2205113954 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1cc02c8a0 con 0x7fd1d406ecf0 2024-03-24T00:41:35.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.226+0000 7fd1d27fc640 1 -- 172.21.15.67:0/2205113954 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c8001670 con 0x7fd1d406ecf0 2024-03-24T00:41:35.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.226+0000 7fd1d9db3640 1 -- 172.21.15.67:0/2205113954 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1d406ecf0 msgr2=0x7fd1d40cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:35.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.226+0000 7fd1d9db3640 1 --2- 172.21.15.67:0/2205113954 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1d406ecf0 0x7fd1d40cf3c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc009d90 tx=0x7fd1cc02bd70 comp rx=0 tx=0).stop 2024-03-24T00:41:35.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.226+0000 7fd1d9db3640 1 -- 172.21.15.67:0/2205113954 shutdown_connections 2024-03-24T00:41:35.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.226+0000 7fd1d9db3640 1 --2- 172.21.15.67:0/2205113954 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1d406ecf0 0x7fd1d40cf3c0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:35.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.226+0000 7fd1d9db3640 1 --2- 172.21.15.67:0/2205113954 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd1d40ccce0 0x7fd1d40cd0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:35.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.226+0000 7fd1d9db3640 1 -- 172.21.15.67:0/2205113954 >> 172.21.15.67:0/2205113954 conn(0x7fd1d40cf9e0 msgr2=0x7fd1d40cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:35.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.226+0000 7fd1d9db3640 1 -- 172.21.15.67:0/2205113954 shutdown_connections 2024-03-24T00:41:35.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.226+0000 7fd1d9db3640 1 -- 172.21.15.67:0/2205113954 wait complete. 2024-03-24T00:41:35.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.227+0000 7fd1d9db3640 1 Processor -- start 2024-03-24T00:41:35.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.227+0000 7fd1d9db3640 1 -- start start 2024-03-24T00:41:35.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.228+0000 7fd1d9db3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1d406ecf0 0x7fd1d40a8490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:35.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.228+0000 7fd1d9db3640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd1d40ccce0 0x7fd1d40aada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:35.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.228+0000 7fd1d9db3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1d40cef40 0x7fd1d41e3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:35.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.228+0000 7fd1d9db3640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d415c290 con 0x7fd1d40cef40 2024-03-24T00:41:35.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.228+0000 7fd1d9db3640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d415ce20 con 0x7fd1d406ecf0 2024-03-24T00:41:35.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.228+0000 7fd1d9db3640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d4145900 con 0x7fd1d40ccce0 2024-03-24T00:41:35.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.228+0000 7fd1d3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1d40cef40 0x7fd1d41e3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:35.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.228+0000 7fd1d3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1d40cef40 0x7fd1d41e3090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43700/0 (socket says 172.21.15.67:43700) 2024-03-24T00:41:35.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.228+0000 7fd1d3fff640 1 -- 172.21.15.67:0/906490889 learned_addr learned my addr 172.21.15.67:0/906490889 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:35.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.228+0000 7fd1d37fe640 1 --2- 172.21.15.67:0/906490889 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1d406ecf0 0x7fd1d40a8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:35.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.228+0000 7fd1d2ffd640 1 --2- 172.21.15.67:0/906490889 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd1d40ccce0 0x7fd1d40aada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:35.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.229+0000 7fd1d3fff640 1 -- 172.21.15.67:0/906490889 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd1d40ccce0 msgr2=0x7fd1d40aada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:35.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.229+0000 7fd1d3fff640 1 --2- 172.21.15.67:0/906490889 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd1d40ccce0 0x7fd1d40aada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:35.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.229+0000 7fd1d3fff640 1 -- 172.21.15.67:0/906490889 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1d406ecf0 msgr2=0x7fd1d40a8490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:35.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.229+0000 7fd1d3fff640 1 --2- 172.21.15.67:0/906490889 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1d406ecf0 0x7fd1d40a8490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:35.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.229+0000 7fd1d3fff640 1 -- 172.21.15.67:0/906490889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d412e210 con 0x7fd1d40cef40 2024-03-24T00:41:35.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.229+0000 7fd1d2ffd640 1 --2- 172.21.15.67:0/906490889 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd1d40ccce0 0x7fd1d40aada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:41:35.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.229+0000 7fd1d3fff640 1 --2- 172.21.15.67:0/906490889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1d40cef40 0x7fd1d41e3090 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc009eb0 tx=0x7fd1cc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:35.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.229+0000 7fd1bbfff640 1 -- 172.21.15.67:0/906490889 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1cc039050 con 0x7fd1d40cef40 2024-03-24T00:41:35.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.229+0000 7fd1bbfff640 1 -- 172.21.15.67:0/906490889 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1cc045070 con 0x7fd1d40cef40 2024-03-24T00:41:35.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.229+0000 7fd1d9db3640 1 -- 172.21.15.67:0/906490889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1d412e4f0 con 0x7fd1d40cef40 2024-03-24T00:41:35.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.229+0000 7fd1bbfff640 1 -- 172.21.15.67:0/906490889 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1cc009450 con 0x7fd1d40cef40 2024-03-24T00:41:35.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.229+0000 7fd1d9db3640 1 -- 172.21.15.67:0/906490889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1d412eab0 con 0x7fd1d40cef40 2024-03-24T00:41:35.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.230+0000 7fd1d9db3640 1 -- 172.21.15.67:0/906490889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1d40cdfd0 con 0x7fd1d40cef40 2024-03-24T00:41:35.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.230+0000 7fd1bbfff640 1 -- 172.21.15.67:0/906490889 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd1cc040670 con 0x7fd1d40cef40 2024-03-24T00:41:35.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.231+0000 7fd1bbfff640 1 --2- 172.21.15.67:0/906490889 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd1c006bbd0 0x7fd1c006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:35.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.231+0000 7fd1bbfff640 1 -- 172.21.15.67:0/906490889 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5335+0+0 (secure 0 0 0) 0x7fd1cc0ad1e0 con 0x7fd1d40cef40 2024-03-24T00:41:35.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.231+0000 7fd1d37fe640 1 --2- 172.21.15.67:0/906490889 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd1c006bbd0 0x7fd1c006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:35.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.231+0000 7fd1d37fe640 1 --2- 172.21.15.67:0/906490889 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd1c006bbd0 0x7fd1c006e090 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fd1d41e4ec0 tx=0x7fd1c4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:35.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.235+0000 7fd1bbfff640 1 -- 172.21.15.67:0/906490889 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd1d40cdfd0 con 0x7fd1d40cef40 2024-03-24T00:41:35.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:35.498+0000 7fd1d9db3640 1 -- 172.21.15.67:0/906490889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_backtrace_repair"]} v 0) v1 -- 0x7fd1d40d0600 con 0x7fd1d40cef40 2024-03-24T00:41:36.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.495+0000 7fd1bbfff640 1 -- 172.21.15.67:0/906490889 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_backtrace_repair"]}]=0 v161) v1 ==== 172+0+0 (secure 0 0 0) 0x7fd1cc07be30 con 0x7fd1d40cef40 2024-03-24T00:41:36.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.498+0000 7fd1d9db3640 1 -- 172.21.15.67:0/906490889 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd1c006bbd0 msgr2=0x7fd1c006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:36.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.498+0000 7fd1d9db3640 1 --2- 172.21.15.67:0/906490889 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd1c006bbd0 0x7fd1c006e090 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fd1d41e4ec0 tx=0x7fd1c4008040 comp rx=0 tx=0).stop 2024-03-24T00:41:36.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.498+0000 7fd1d9db3640 1 -- 172.21.15.67:0/906490889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1d40cef40 msgr2=0x7fd1d41e3090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:36.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.498+0000 7fd1d9db3640 1 --2- 172.21.15.67:0/906490889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1d40cef40 0x7fd1d41e3090 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc009eb0 tx=0x7fd1cc004a40 comp rx=0 tx=0).stop 2024-03-24T00:41:36.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.498+0000 7fd1d9db3640 1 -- 172.21.15.67:0/906490889 shutdown_connections 2024-03-24T00:41:36.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.498+0000 7fd1d9db3640 1 --2- 172.21.15.67:0/906490889 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd1c006bbd0 0x7fd1c006e090 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:36.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.498+0000 7fd1d9db3640 1 --2- 172.21.15.67:0/906490889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1d40cef40 0x7fd1d41e3090 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:36.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.498+0000 7fd1d9db3640 1 --2- 172.21.15.67:0/906490889 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd1d40ccce0 0x7fd1d40aada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:36.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.498+0000 7fd1d9db3640 1 --2- 172.21.15.67:0/906490889 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1d406ecf0 0x7fd1d40a8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:36.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.498+0000 7fd1d9db3640 1 -- 172.21.15.67:0/906490889 >> 172.21.15.67:0/906490889 conn(0x7fd1d40cf9e0 msgr2=0x7fd1d406dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:36.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.498+0000 7fd1d9db3640 1 -- 172.21.15.67:0/906490889 shutdown_connections 2024-03-24T00:41:36.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.499+0000 7fd1d9db3640 1 -- 172.21.15.67:0/906490889 wait complete. 2024-03-24T00:41:36.513 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-24T00:41:36.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.622+0000 7f4d0d8c0640 1 Processor -- start 2024-03-24T00:41:36.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.622+0000 7f4d0d8c0640 1 -- start start 2024-03-24T00:41:36.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.623+0000 7f4d0d8c0640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4d0815dea0 0x7f4d080d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:36.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.623+0000 7f4d0d8c0640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4d0815e280 0x7f4d080d3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:36.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.623+0000 7f4d0d8c0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4d080b5a60 0x7f4d080b5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:36.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.623+0000 7f4d0d8c0640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d080b6410 con 0x7f4d080b5a60 2024-03-24T00:41:36.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.623+0000 7f4d0d8c0640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d080d2250 con 0x7f4d0815e280 2024-03-24T00:41:36.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.623+0000 7f4d0d8c0640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d080d2530 con 0x7f4d0815dea0 2024-03-24T00:41:36.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.623+0000 7f4d077fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4d080b5a60 0x7f4d080b5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:36.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.623+0000 7f4d077fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4d080b5a60 0x7f4d080b5ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43716/0 (socket says 172.21.15.67:43716) 2024-03-24T00:41:36.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.623+0000 7f4d077fe640 1 -- 172.21.15.67:0/3952608963 learned_addr learned my addr 172.21.15.67:0/3952608963 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:36.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.623+0000 7f4d067fc640 1 --2- 172.21.15.67:0/3952608963 >> v2:172.21.15.72:3300/0 conn(0x7f4d0815e280 0x7f4d080d3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:36.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.623+0000 7f4d06ffd640 1 --2- 172.21.15.67:0/3952608963 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4d0815dea0 0x7f4d080d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:36.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.623+0000 7f4d077fe640 1 -- 172.21.15.67:0/3952608963 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4d0815dea0 msgr2=0x7f4d080d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:36.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.623+0000 7f4d077fe640 1 --2- 172.21.15.67:0/3952608963 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4d0815dea0 0x7f4d080d33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:36.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.623+0000 7f4d077fe640 1 -- 172.21.15.67:0/3952608963 >> v2:172.21.15.72:3300/0 conn(0x7f4d0815e280 msgr2=0x7f4d080d3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:36.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.623+0000 7f4d077fe640 1 --2- 172.21.15.67:0/3952608963 >> v2:172.21.15.72:3300/0 conn(0x7f4d0815e280 0x7f4d080d3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:36.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.623+0000 7f4d077fe640 1 -- 172.21.15.67:0/3952608963 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d080d2810 con 0x7f4d080b5a60 2024-03-24T00:41:36.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.623+0000 7f4d06ffd640 1 --2- 172.21.15.67:0/3952608963 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4d0815dea0 0x7f4d080d33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:41:36.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.623+0000 7f4d067fc640 1 --2- 172.21.15.67:0/3952608963 >> v2:172.21.15.72:3300/0 conn(0x7f4d0815e280 0x7f4d080d3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:41:36.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.624+0000 7f4d077fe640 1 --2- 172.21.15.67:0/3952608963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d080b5a60 0x7f4d080b5ed0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f4d080d2460 tx=0x7f4d0002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=878a1fbfdc2e826d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:36.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.624+0000 7f4d05ffb640 1 -- 172.21.15.67:0/3952608963 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d000317c0 con 0x7f4d080b5a60 2024-03-24T00:41:36.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.624+0000 7f4d05ffb640 1 -- 172.21.15.67:0/3952608963 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d00037070 con 0x7f4d080b5a60 2024-03-24T00:41:36.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.624+0000 7f4d05ffb640 1 -- 172.21.15.67:0/3952608963 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d00031b80 con 0x7f4d080b5a60 2024-03-24T00:41:36.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.625+0000 7f4d0d8c0640 1 -- 172.21.15.67:0/3952608963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d080b5a60 msgr2=0x7f4d080b5ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:36.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.625+0000 7f4d0d8c0640 1 --2- 172.21.15.67:0/3952608963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d080b5a60 0x7f4d080b5ed0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f4d080d2460 tx=0x7f4d0002b860 comp rx=0 tx=0).stop 2024-03-24T00:41:36.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.625+0000 7f4d0d8c0640 1 -- 172.21.15.67:0/3952608963 shutdown_connections 2024-03-24T00:41:36.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.625+0000 7f4d0d8c0640 1 --2- 172.21.15.67:0/3952608963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d080b5a60 0x7f4d080b5ed0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:36.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.625+0000 7f4d0d8c0640 1 --2- 172.21.15.67:0/3952608963 >> v2:172.21.15.72:3300/0 conn(0x7f4d0815e280 0x7f4d080d3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:36.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.625+0000 7f4d0d8c0640 1 --2- 172.21.15.67:0/3952608963 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4d0815dea0 0x7f4d080d33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:36.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.625+0000 7f4d0d8c0640 1 -- 172.21.15.67:0/3952608963 >> 172.21.15.67:0/3952608963 conn(0x7f4d08072da0 msgr2=0x7f4d080d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:36.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.625+0000 7f4d0d8c0640 1 -- 172.21.15.67:0/3952608963 shutdown_connections 2024-03-24T00:41:36.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.625+0000 7f4d0d8c0640 1 -- 172.21.15.67:0/3952608963 wait complete. 2024-03-24T00:41:36.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.626+0000 7f4d0d8c0640 1 Processor -- start 2024-03-24T00:41:36.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.626+0000 7f4d0d8c0640 1 -- start start 2024-03-24T00:41:36.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.627+0000 7f4d0d8c0640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4d080b5a60 0x7f4d081f4210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:36.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.627+0000 7f4d0d8c0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4d0815dea0 0x7f4d080ad9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:36.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.627+0000 7f4d0d8c0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d0815e280 0x7f4d080adee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:36.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.627+0000 7f4d0d8c0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d0806c7c0 con 0x7f4d0815e280 2024-03-24T00:41:36.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.627+0000 7f4d0d8c0640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d0806c640 con 0x7f4d0815dea0 2024-03-24T00:41:36.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.627+0000 7f4d0d8c0640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d0806c940 con 0x7f4d080b5a60 2024-03-24T00:41:36.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.627+0000 7f4d077fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d0815e280 0x7f4d080adee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:36.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.627+0000 7f4d077fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d0815e280 0x7f4d080adee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43730/0 (socket says 172.21.15.67:43730) 2024-03-24T00:41:36.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.627+0000 7f4d077fe640 1 -- 172.21.15.67:0/1427027227 learned_addr learned my addr 172.21.15.67:0/1427027227 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:36.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.627+0000 7f4d067fc640 1 --2- 172.21.15.67:0/1427027227 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4d0815dea0 0x7f4d080ad9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:36.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.627+0000 7f4d06ffd640 1 --2- 172.21.15.67:0/1427027227 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4d080b5a60 0x7f4d081f4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:36.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.627+0000 7f4d077fe640 1 -- 172.21.15.67:0/1427027227 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4d080b5a60 msgr2=0x7f4d081f4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:36.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.627+0000 7f4d077fe640 1 --2- 172.21.15.67:0/1427027227 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4d080b5a60 0x7f4d081f4210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:36.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.627+0000 7f4d077fe640 1 -- 172.21.15.67:0/1427027227 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4d0815dea0 msgr2=0x7f4d080ad9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:36.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.627+0000 7f4d077fe640 1 --2- 172.21.15.67:0/1427027227 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4d0815dea0 0x7f4d080ad9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:36.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.627+0000 7f4d077fe640 1 -- 172.21.15.67:0/1427027227 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d080ccf80 con 0x7f4d0815e280 2024-03-24T00:41:36.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.627+0000 7f4d06ffd640 1 --2- 172.21.15.67:0/1427027227 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4d080b5a60 0x7f4d081f4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:41:36.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.628+0000 7f4d067fc640 1 --2- 172.21.15.67:0/1427027227 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4d0815dea0 0x7f4d080ad9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:41:36.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.628+0000 7f4d077fe640 1 --2- 172.21.15.67:0/1427027227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d0815e280 0x7f4d080adee0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f4d0002b8a0 tx=0x7f4d000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:36.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.628+0000 7f4cfffff640 1 -- 172.21.15.67:0/1427027227 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d00032860 con 0x7f4d0815e280 2024-03-24T00:41:36.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.628+0000 7f4cfffff640 1 -- 172.21.15.67:0/1427027227 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d00032a20 con 0x7f4d0815e280 2024-03-24T00:41:36.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.628+0000 7f4d0d8c0640 1 -- 172.21.15.67:0/1427027227 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d080cd260 con 0x7f4d0815e280 2024-03-24T00:41:36.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.628+0000 7f4cfffff640 1 -- 172.21.15.67:0/1427027227 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d00032e10 con 0x7f4d0815e280 2024-03-24T00:41:36.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.628+0000 7f4d0d8c0640 1 -- 172.21.15.67:0/1427027227 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d080cd820 con 0x7f4d0815e280 2024-03-24T00:41:36.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.629+0000 7f4d0d8c0640 1 -- 172.21.15.67:0/1427027227 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d080cda60 con 0x7f4d0815e280 2024-03-24T00:41:36.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.629+0000 7f4cfffff640 1 -- 172.21.15.67:0/1427027227 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4d000453f0 con 0x7f4d0815e280 2024-03-24T00:41:36.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.630+0000 7f4cfffff640 1 --2- 172.21.15.67:0/1427027227 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4ce806bbd0 0x7f4ce806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:36.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.630+0000 7f4cfffff640 1 -- 172.21.15.67:0/1427027227 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f4d000ac9e0 con 0x7f4d0815e280 2024-03-24T00:41:36.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.630+0000 7f4d06ffd640 1 --2- 172.21.15.67:0/1427027227 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4ce806bbd0 0x7f4ce806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:36.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.631+0000 7f4d06ffd640 1 --2- 172.21.15.67:0/1427027227 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4ce806bbd0 0x7f4ce806e090 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f4d080b4de0 tx=0x7f4cf8008530 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:36.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.634+0000 7f4cfffff640 1 -- 172.21.15.67:0/1427027227 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4d080cda60 con 0x7f4d0815e280 2024-03-24T00:41:36.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.818+0000 7f4d0d8c0640 1 -- 172.21.15.67:0/1427027227 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f4d0804f400 con 0x7f4d0815e280 2024-03-24T00:41:36.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.823+0000 7f4cfffff640 1 -- 172.21.15.67:0/1427027227 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4d00040070 con 0x7f4d0815e280 2024-03-24T00:41:36.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.826+0000 7f4d0d8c0640 1 -- 172.21.15.67:0/1427027227 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4ce806bbd0 msgr2=0x7f4ce806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:36.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.826+0000 7f4d0d8c0640 1 --2- 172.21.15.67:0/1427027227 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4ce806bbd0 0x7f4ce806e090 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f4d080b4de0 tx=0x7f4cf8008530 comp rx=0 tx=0).stop 2024-03-24T00:41:36.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.826+0000 7f4d0d8c0640 1 -- 172.21.15.67:0/1427027227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d0815e280 msgr2=0x7f4d080adee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:36.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.826+0000 7f4d0d8c0640 1 --2- 172.21.15.67:0/1427027227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d0815e280 0x7f4d080adee0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f4d0002b8a0 tx=0x7f4d000046c0 comp rx=0 tx=0).stop 2024-03-24T00:41:36.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.826+0000 7f4d0d8c0640 1 -- 172.21.15.67:0/1427027227 shutdown_connections 2024-03-24T00:41:36.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.826+0000 7f4d0d8c0640 1 --2- 172.21.15.67:0/1427027227 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4ce806bbd0 0x7f4ce806e090 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:36.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.826+0000 7f4d0d8c0640 1 --2- 172.21.15.67:0/1427027227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d0815e280 0x7f4d080adee0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:36.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.826+0000 7f4d0d8c0640 1 --2- 172.21.15.67:0/1427027227 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4d0815dea0 0x7f4d080ad9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:36.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.826+0000 7f4d0d8c0640 1 --2- 172.21.15.67:0/1427027227 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4d080b5a60 0x7f4d081f4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:36.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.826+0000 7f4d0d8c0640 1 -- 172.21.15.67:0/1427027227 >> 172.21.15.67:0/1427027227 conn(0x7f4d08072da0 msgr2=0x7f4d080d31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:36.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.826+0000 7f4d0d8c0640 1 -- 172.21.15.67:0/1427027227 shutdown_connections 2024-03-24T00:41:36.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:36.826+0000 7f4d0d8c0640 1 -- 172.21.15.67:0/1427027227 wait complete. 2024-03-24T00:41:36.837 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-24T00:41:36.838 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:41:36.882 DEBUG:teuthology.orchestra.run.smithi072:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:41:36.928 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-24T00:41:36.928 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:41:36.928 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:41:36.942 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056375,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15771 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T00:41:36.942 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:41:36.942 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:41:36.959 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17590 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248808k,nr_inodes=812202,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:41:36.959 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:41:36.959 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:41:36.996 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056375,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15771 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T00:41:36.997 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:41:36.997 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:41:37.016 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17590 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248808k,nr_inodes=812202,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:41:37.016 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-24T00:41:37.016 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:41:37.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.137+0000 7f3bbb697640 1 Processor -- start 2024-03-24T00:41:37.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.137+0000 7f3bbb697640 1 -- start start 2024-03-24T00:41:37.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.138+0000 7f3bbb697640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3bb40ccab0 0x7f3bb40cce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:37.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.138+0000 7f3bbb697640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3bb40701b0 0x7f3bb4162320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:37.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.138+0000 7f3bbb697640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bb40cd3d0 con 0x7f3bb40b5900 2024-03-24T00:41:37.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.138+0000 7f3bbb697640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bb40b5e10 con 0x7f3bb40701b0 2024-03-24T00:41:37.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.138+0000 7f3bbb697640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bb406f6d0 con 0x7f3bb40ccab0 2024-03-24T00:41:37.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.138+0000 7f3bb9c34640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3bb40701b0 0x7f3bb4162320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:37.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.138+0000 7f3bb8c32640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3bb40b5900 0x7f3bb4161c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58362/0 (socket says 172.21.15.67:58362) 2024-03-24T00:41:37.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.138+0000 7f3bb8c32640 1 -- 172.21.15.67:0/4275534563 learned_addr learned my addr 172.21.15.67:0/4275534563 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:41:37.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.138+0000 7f3bb9433640 1 --2- 172.21.15.67:0/4275534563 >> v2:172.21.15.72:3300/0 conn(0x7f3bb40ccab0 0x7f3bb40cce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:37.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.138+0000 7f3bb9c34640 1 -- 172.21.15.67:0/4275534563 >> v2:172.21.15.72:3300/0 conn(0x7f3bb40ccab0 msgr2=0x7f3bb40cce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:37.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.138+0000 7f3bb9c34640 1 --2- 172.21.15.67:0/4275534563 >> v2:172.21.15.72:3300/0 conn(0x7f3bb40ccab0 0x7f3bb40cce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:37.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.138+0000 7f3bb9c34640 1 -- 172.21.15.67:0/4275534563 >> v1:172.21.15.67:6789/0 conn(0x7f3bb40b5900 legacy=0x7f3bb4161c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:37.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.138+0000 7f3bb9c34640 1 -- 172.21.15.67:0/4275534563 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bb406f9b0 con 0x7f3bb40701b0 2024-03-24T00:41:37.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.138+0000 7f3babfff640 1 -- 172.21.15.67:0/4275534563 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3367900073 0 0) 0x7f3bb40cd3d0 con 0x7f3bb40b5900 2024-03-24T00:41:37.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.138+0000 7f3bb9433640 1 --2- 172.21.15.67:0/4275534563 >> v2:172.21.15.72:3300/0 conn(0x7f3bb40ccab0 0x7f3bb40cce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:41:37.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.139+0000 7f3bb9c34640 1 --2- 172.21.15.67:0/4275534563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bb40701b0 0x7f3bb4162320 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f3bb40b6280 tx=0x7f3bac02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d770c2ca7a495b83 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:37.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.139+0000 7f3babfff640 1 -- 172.21.15.67:0/4275534563 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bac039070 con 0x7f3bb40701b0 2024-03-24T00:41:37.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.139+0000 7f3babfff640 1 -- 172.21.15.67:0/4275534563 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bac02cc80 con 0x7f3bb40701b0 2024-03-24T00:41:37.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.139+0000 7f3babfff640 1 -- 172.21.15.67:0/4275534563 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bac034660 con 0x7f3bb40701b0 2024-03-24T00:41:37.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.140+0000 7f3bbb697640 1 -- 172.21.15.67:0/4275534563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bb40701b0 msgr2=0x7f3bb4162320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:37.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.140+0000 7f3bbb697640 1 --2- 172.21.15.67:0/4275534563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bb40701b0 0x7f3bb4162320 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f3bb40b6280 tx=0x7f3bac02bed0 comp rx=0 tx=0).stop 2024-03-24T00:41:37.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.140+0000 7f3bbb697640 1 -- 172.21.15.67:0/4275534563 shutdown_connections 2024-03-24T00:41:37.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.140+0000 7f3bbb697640 1 --2- 172.21.15.67:0/4275534563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bb40701b0 0x7f3bb4162320 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:37.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.140+0000 7f3bbb697640 1 --2- 172.21.15.67:0/4275534563 >> v2:172.21.15.72:3300/0 conn(0x7f3bb40ccab0 0x7f3bb40cce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:37.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.140+0000 7f3bbb697640 1 -- 172.21.15.67:0/4275534563 >> 172.21.15.67:0/4275534563 conn(0x7f3bb40b3c40 msgr2=0x7f3bb40b1dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:37.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.140+0000 7f3bbb697640 1 -- 172.21.15.67:0/4275534563 shutdown_connections 2024-03-24T00:41:37.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.140+0000 7f3bbb697640 1 -- 172.21.15.67:0/4275534563 wait complete. 2024-03-24T00:41:37.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.141+0000 7f3bbb697640 1 Processor -- start 2024-03-24T00:41:37.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.141+0000 7f3bbb697640 1 -- start start 2024-03-24T00:41:37.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.142+0000 7f3bbb697640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bb40701b0 0x7f3bb4158c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:37.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.142+0000 7f3bbb697640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bb40b5900 0x7f3bb40cec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:37.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.142+0000 7f3bbb697640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bb40ccab0 0x7f3bb40cf140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:37.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.142+0000 7f3bbb697640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb406f470 con 0x7f3bb40b5900 2024-03-24T00:41:37.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.142+0000 7f3bbb697640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb406f2f0 con 0x7f3bb40701b0 2024-03-24T00:41:37.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.142+0000 7f3bbb697640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb415e360 con 0x7f3bb40ccab0 2024-03-24T00:41:37.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.142+0000 7f3bb8c32640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bb40b5900 0x7f3bb40cec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:37.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.142+0000 7f3bb8c32640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bb40b5900 0x7f3bb40cec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43758/0 (socket says 172.21.15.67:43758) 2024-03-24T00:41:37.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.142+0000 7f3bb8c32640 1 -- 172.21.15.67:0/3645430284 learned_addr learned my addr 172.21.15.67:0/3645430284 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:37.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.142+0000 7f3bb9433640 1 --2- 172.21.15.67:0/3645430284 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bb40701b0 0x7f3bb4158c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:37.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.142+0000 7f3bb9c34640 1 --2- 172.21.15.67:0/3645430284 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bb40ccab0 0x7f3bb40cf140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:37.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.142+0000 7f3bb8c32640 1 -- 172.21.15.67:0/3645430284 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bb40ccab0 msgr2=0x7f3bb40cf140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:37.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.142+0000 7f3bb8c32640 1 --2- 172.21.15.67:0/3645430284 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bb40ccab0 0x7f3bb40cf140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:37.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.142+0000 7f3bb8c32640 1 -- 172.21.15.67:0/3645430284 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bb40701b0 msgr2=0x7f3bb4158c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:37.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.142+0000 7f3bb8c32640 1 --2- 172.21.15.67:0/3645430284 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bb40701b0 0x7f3bb4158c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:37.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.142+0000 7f3bb8c32640 1 -- 172.21.15.67:0/3645430284 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bb41f4c70 con 0x7f3bb40b5900 2024-03-24T00:41:37.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.142+0000 7f3bb8c32640 1 --2- 172.21.15.67:0/3645430284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bb40b5900 0x7f3bb40cec00 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f3ba400c410 tx=0x7f3ba400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:37.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.142+0000 7f3ba9ffb640 1 -- 172.21.15.67:0/3645430284 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba40074e0 con 0x7f3bb40b5900 2024-03-24T00:41:37.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.143+0000 7f3ba9ffb640 1 -- 172.21.15.67:0/3645430284 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ba40047e0 con 0x7f3bb40b5900 2024-03-24T00:41:37.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.143+0000 7f3ba9ffb640 1 -- 172.21.15.67:0/3645430284 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba4004bd0 con 0x7f3bb40b5900 2024-03-24T00:41:37.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.143+0000 7f3bbb697640 1 -- 172.21.15.67:0/3645430284 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bb40b2bf0 con 0x7f3bb40b5900 2024-03-24T00:41:37.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.143+0000 7f3bbb697640 1 -- 172.21.15.67:0/3645430284 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bb41f4ee0 con 0x7f3bb40b5900 2024-03-24T00:41:37.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.144+0000 7f3bbb697640 1 -- 172.21.15.67:0/3645430284 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bb404f400 con 0x7f3bb40b5900 2024-03-24T00:41:37.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.144+0000 7f3ba9ffb640 1 -- 172.21.15.67:0/3645430284 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3ba4011070 con 0x7f3bb40b5900 2024-03-24T00:41:37.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.144+0000 7f3ba9ffb640 1 --2- 172.21.15.67:0/3645430284 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3b9806bbd0 0x7f3b9806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:37.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.145+0000 7f3ba9ffb640 1 -- 172.21.15.67:0/3645430284 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f3ba408a470 con 0x7f3bb40b5900 2024-03-24T00:41:37.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.145+0000 7f3bb9433640 1 --2- 172.21.15.67:0/3645430284 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3b9806bbd0 0x7f3b9806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:37.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.145+0000 7f3bb9433640 1 --2- 172.21.15.67:0/3645430284 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3b9806bbd0 0x7f3b9806e090 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f3bb406c520 tx=0x7f3bb0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:37.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.149+0000 7f3ba9ffb640 1 -- 172.21.15.67:0/3645430284 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3ba400e280 con 0x7f3bb40b5900 2024-03-24T00:41:37.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.370+0000 7f3bbb697640 1 -- 172.21.15.67:0/3645430284 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3bb40cd990 con 0x7f3bb40b5900 2024-03-24T00:41:37.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.371+0000 7f3ba9ffb640 1 -- 172.21.15.67:0/3645430284 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+3699 (secure 0 0 0) 0x7f3ba4059160 con 0x7f3bb40b5900 2024-03-24T00:41:37.372 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:41:37.372 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":28,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5360,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3312887486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":3312887486},{"type":"v1","addr":"172.21.15.67:6835","nonce":3312887486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5821,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/4171866637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":4171866637},{"type":"v1","addr":"172.21.15.72:6836","nonce":4171866637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5824,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/4106412452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":4106412452},{"type":"v1","addr":"172.21.15.72:6837","nonce":4106412452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":6279,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2063358316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2063358316},{"type":"v1","addr":"172.21.15.67:6834","nonce":2063358316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[]} 2024-03-24T00:41:37.372 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 28 2024-03-24T00:41:37.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.373+0000 7f3bbb697640 1 -- 172.21.15.67:0/3645430284 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3b9806bbd0 msgr2=0x7f3b9806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:37.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.373+0000 7f3bbb697640 1 --2- 172.21.15.67:0/3645430284 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3b9806bbd0 0x7f3b9806e090 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f3bb406c520 tx=0x7f3bb0008040 comp rx=0 tx=0).stop 2024-03-24T00:41:37.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.373+0000 7f3bbb697640 1 -- 172.21.15.67:0/3645430284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bb40b5900 msgr2=0x7f3bb40cec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:37.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.373+0000 7f3bbb697640 1 --2- 172.21.15.67:0/3645430284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bb40b5900 0x7f3bb40cec00 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f3ba400c410 tx=0x7f3ba400c8d0 comp rx=0 tx=0).stop 2024-03-24T00:41:37.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.373+0000 7f3bbb697640 1 -- 172.21.15.67:0/3645430284 shutdown_connections 2024-03-24T00:41:37.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.373+0000 7f3bbb697640 1 --2- 172.21.15.67:0/3645430284 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3b9806bbd0 0x7f3b9806e090 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:37.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.373+0000 7f3bbb697640 1 --2- 172.21.15.67:0/3645430284 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bb40ccab0 0x7f3bb40cf140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:37.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.373+0000 7f3bbb697640 1 --2- 172.21.15.67:0/3645430284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bb40b5900 0x7f3bb40cec00 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:37.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.373+0000 7f3bbb697640 1 --2- 172.21.15.67:0/3645430284 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bb40701b0 0x7f3bb4158c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:37.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.373+0000 7f3bbb697640 1 -- 172.21.15.67:0/3645430284 >> 172.21.15.67:0/3645430284 conn(0x7f3bb40b3c40 msgr2=0x7f3bb40cc0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:37.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.373+0000 7f3bbb697640 1 -- 172.21.15.67:0/3645430284 shutdown_connections 2024-03-24T00:41:37.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.373+0000 7f3bbb697640 1 -- 172.21.15.67:0/3645430284 wait complete. 2024-03-24T00:41:37.384 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-24T00:41:37.384 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-24T00:41:37.385 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-24T00:41:37.385 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:41:37.385 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-24T00:41:37.385 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-24T00:41:37.385 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-24T00:41:37.385 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:41:37.385 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-24T00:41:37.385 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-24T00:41:37.386 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-24T00:41:37.386 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:41:37.386 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-24T00:41:37.386 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-24T00:41:37.386 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-24T00:41:37.386 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:41:37.395 INFO:tasks.ceph.mds.a:Stopped 2024-03-24T00:41:37.395 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-24T00:41:37.395 INFO:tasks.ceph.mds.c:Stopped 2024-03-24T00:41:37.396 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-24T00:41:37.397 INFO:tasks.ceph.mds.b:Stopped 2024-03-24T00:41:37.397 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-24T00:41:37.398 INFO:tasks.ceph.mds.d:Stopped 2024-03-24T00:41:37.398 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-24T00:41:37.438 INFO:tasks.ceph.mds.a:Started 2024-03-24T00:41:37.438 DEBUG:teuthology.parallel:result is None 2024-03-24T00:41:37.439 INFO:tasks.ceph.mds.c:Started 2024-03-24T00:41:37.439 DEBUG:teuthology.parallel:result is None 2024-03-24T00:41:37.440 INFO:tasks.ceph.mds.b:Started 2024-03-24T00:41:37.440 DEBUG:teuthology.parallel:result is None 2024-03-24T00:41:37.441 INFO:tasks.ceph.mds.d:Started 2024-03-24T00:41:37.441 DEBUG:teuthology.parallel:result is None 2024-03-24T00:41:37.441 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-24T00:41:37.514 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-03-24T00:41:37.514 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-03-24T00:41:37.525 INFO:tasks.ceph.mds.d.smithi072.stdout:starting mds.d at 2024-03-24T00:41:37.525 INFO:tasks.ceph.mds.b.smithi072.stdout:starting mds.b at 2024-03-24T00:41:37.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.547+0000 7ff445109640 1 Processor -- start 2024-03-24T00:41:37.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.547+0000 7ff445109640 1 -- start start 2024-03-24T00:41:37.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.548+0000 7ff445109640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff44015e0f0 0x7ff4400b95b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:37.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.548+0000 7ff445109640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff4400b9af0 0x7ff4400d32d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:37.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.548+0000 7ff445109640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4400d2230 con 0x7ff4400d3810 2024-03-24T00:41:37.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.548+0000 7ff445109640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4400d2510 con 0x7ff44015e0f0 2024-03-24T00:41:37.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.548+0000 7ff43f59e640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff4400d3810 0x7ff4400b6dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58384/0 (socket says 172.21.15.67:58384) 2024-03-24T00:41:37.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.548+0000 7ff43f59e640 1 -- 172.21.15.67:0/4103360027 learned_addr learned my addr 172.21.15.67:0/4103360027 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:41:37.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.548+0000 7ff445109640 1 -- 172.21.15.67:0/4103360027 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4400d27f0 con 0x7ff4400b9af0 2024-03-24T00:41:37.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.548+0000 7ff43ed9d640 1 --2- 172.21.15.67:0/4103360027 >> v2:172.21.15.72:3300/0 conn(0x7ff44015e0f0 0x7ff4400b95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:37.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.548+0000 7ff43e59c640 1 --2- 172.21.15.67:0/4103360027 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff4400b9af0 0x7ff4400d32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:37.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.548+0000 7ff43dd9b640 1 -- 172.21.15.67:0/4103360027 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2357506442 0 0) 0x7ff4400d2230 con 0x7ff4400d3810 2024-03-24T00:41:37.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.548+0000 7ff43dd9b640 1 -- 172.21.15.67:0/4103360027 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff428003520 con 0x7ff4400d3810 2024-03-24T00:41:37.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.548+0000 7ff43ed9d640 1 -- 172.21.15.67:0/4103360027 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff4400b9af0 msgr2=0x7ff4400d32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:37.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.548+0000 7ff43ed9d640 1 --2- 172.21.15.67:0/4103360027 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff4400b9af0 0x7ff4400d32d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:37.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.548+0000 7ff43ed9d640 1 -- 172.21.15.67:0/4103360027 >> v1:172.21.15.67:6789/0 conn(0x7ff4400d3810 legacy=0x7ff4400b6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:37.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.548+0000 7ff43ed9d640 1 -- 172.21.15.67:0/4103360027 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4400d2ad0 con 0x7ff44015e0f0 2024-03-24T00:41:37.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.549+0000 7ff43ed9d640 1 --2- 172.21.15.67:0/4103360027 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff44015e0f0 0x7ff4400b95b0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ff4300097a0 tx=0x7ff43002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d313a9b6138a999c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:37.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.549+0000 7ff43dd9b640 1 -- 172.21.15.67:0/4103360027 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff430035020 con 0x7ff44015e0f0 2024-03-24T00:41:37.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.549+0000 7ff43dd9b640 1 -- 172.21.15.67:0/4103360027 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff430031830 con 0x7ff44015e0f0 2024-03-24T00:41:37.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.549+0000 7ff43dd9b640 1 -- 172.21.15.67:0/4103360027 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff430031be0 con 0x7ff44015e0f0 2024-03-24T00:41:37.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.550+0000 7ff445109640 1 -- 172.21.15.67:0/4103360027 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff44015e0f0 msgr2=0x7ff4400b95b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:37.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.550+0000 7ff445109640 1 --2- 172.21.15.67:0/4103360027 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff44015e0f0 0x7ff4400b95b0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ff4300097a0 tx=0x7ff43002b8d0 comp rx=0 tx=0).stop 2024-03-24T00:41:37.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.550+0000 7ff445109640 1 -- 172.21.15.67:0/4103360027 shutdown_connections 2024-03-24T00:41:37.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.550+0000 7ff445109640 1 --2- 172.21.15.67:0/4103360027 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff4400b9af0 0x7ff4400d32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:37.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.550+0000 7ff445109640 1 --2- 172.21.15.67:0/4103360027 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff44015e0f0 0x7ff4400b95b0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:37.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.550+0000 7ff445109640 1 -- 172.21.15.67:0/4103360027 >> 172.21.15.67:0/4103360027 conn(0x7ff440072b60 msgr2=0x7ff440072f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:37.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.550+0000 7ff445109640 1 -- 172.21.15.67:0/4103360027 shutdown_connections 2024-03-24T00:41:37.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.550+0000 7ff445109640 1 -- 172.21.15.67:0/4103360027 wait complete. 2024-03-24T00:41:37.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.552+0000 7ff445109640 1 Processor -- start 2024-03-24T00:41:37.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.552+0000 7ff445109640 1 -- start start 2024-03-24T00:41:37.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.552+0000 7ff445109640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff4400b9af0 0x7ff440161170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:37.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.552+0000 7ff445109640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff4400d3810 0x7ff44015fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:37.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.552+0000 7ff445109640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff44015e0f0 0x7ff440151130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:37.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.552+0000 7ff445109640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4400b3350 con 0x7ff44015e0f0 2024-03-24T00:41:37.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.552+0000 7ff445109640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4400b31d0 con 0x7ff4400b9af0 2024-03-24T00:41:37.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.552+0000 7ff445109640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4400b74d0 con 0x7ff4400d3810 2024-03-24T00:41:37.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.553+0000 7ff43f59e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff44015e0f0 0x7ff440151130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:37.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.553+0000 7ff43f59e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff44015e0f0 0x7ff440151130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43808/0 (socket says 172.21.15.67:43808) 2024-03-24T00:41:37.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.553+0000 7ff43f59e640 1 -- 172.21.15.67:0/2611965850 learned_addr learned my addr 172.21.15.67:0/2611965850 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:37.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.553+0000 7ff43ed9d640 1 --2- 172.21.15.67:0/2611965850 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff4400b9af0 0x7ff440161170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:37.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.553+0000 7ff43e59c640 1 --2- 172.21.15.67:0/2611965850 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff4400d3810 0x7ff44015fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:37.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.553+0000 7ff43f59e640 1 -- 172.21.15.67:0/2611965850 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff4400d3810 msgr2=0x7ff44015fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:37.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.553+0000 7ff43f59e640 1 --2- 172.21.15.67:0/2611965850 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff4400d3810 0x7ff44015fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:37.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.553+0000 7ff43f59e640 1 -- 172.21.15.67:0/2611965850 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff4400b9af0 msgr2=0x7ff440161170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:37.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.553+0000 7ff43f59e640 1 --2- 172.21.15.67:0/2611965850 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff4400b9af0 0x7ff440161170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:37.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.553+0000 7ff43f59e640 1 -- 172.21.15.67:0/2611965850 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff44015ab90 con 0x7ff44015e0f0 2024-03-24T00:41:37.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.553+0000 7ff43ed9d640 1 --2- 172.21.15.67:0/2611965850 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff4400b9af0 0x7ff440161170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:41:37.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.553+0000 7ff43f59e640 1 --2- 172.21.15.67:0/2611965850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff44015e0f0 0x7ff440151130 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7ff43800cb10 tx=0x7ff43800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:37.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.553+0000 7ff4277fe640 1 -- 172.21.15.67:0/2611965850 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff438017070 con 0x7ff44015e0f0 2024-03-24T00:41:37.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.553+0000 7ff4277fe640 1 -- 172.21.15.67:0/2611965850 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff43800ac70 con 0x7ff44015e0f0 2024-03-24T00:41:37.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.553+0000 7ff445109640 1 -- 172.21.15.67:0/2611965850 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4401600d0 con 0x7ff44015e0f0 2024-03-24T00:41:37.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.553+0000 7ff4277fe640 1 -- 172.21.15.67:0/2611965850 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff43800ae60 con 0x7ff44015e0f0 2024-03-24T00:41:37.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.553+0000 7ff445109640 1 -- 172.21.15.67:0/2611965850 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff440160690 con 0x7ff44015e0f0 2024-03-24T00:41:37.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.554+0000 7ff445109640 1 -- 172.21.15.67:0/2611965850 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff44004f400 con 0x7ff44015e0f0 2024-03-24T00:41:37.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.555+0000 7ff4277fe640 1 -- 172.21.15.67:0/2611965850 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff43801a070 con 0x7ff44015e0f0 2024-03-24T00:41:37.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.555+0000 7ff4277fe640 1 --2- 172.21.15.67:0/2611965850 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff42c06bbd0 0x7ff42c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:37.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.555+0000 7ff4277fe640 1 -- 172.21.15.67:0/2611965850 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5335+0+0 (secure 0 0 0) 0x7ff43808a6c0 con 0x7ff44015e0f0 2024-03-24T00:41:37.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.555+0000 7ff43ed9d640 1 --2- 172.21.15.67:0/2611965850 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff42c06bbd0 0x7ff42c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:37.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.556+0000 7ff43ed9d640 1 --2- 172.21.15.67:0/2611965850 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff42c06bbd0 0x7ff42c06e090 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ff430036000 tx=0x7ff43003c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:37.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.560+0000 7ff4277fe640 1 -- 172.21.15.67:0/2611965850 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff438058c50 con 0x7ff44015e0f0 2024-03-24T00:41:37.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:37.756+0000 7ff445109640 1 -- 172.21.15.67:0/2611965850 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7ff4400c7c30 con 0x7ff44015e0f0 2024-03-24T00:41:38.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.852+0000 7ff4277fe640 1 -- 172.21.15.67:0/2611965850 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v58) v1 ==== 97+0+0 (secure 0 0 0) 0x7ff43805ccf0 con 0x7ff44015e0f0 2024-03-24T00:41:38.853 INFO:teuthology.orchestra.run.smithi067.stderr: removed all blocklist entries 2024-03-24T00:41:38.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.855+0000 7ff445109640 1 -- 172.21.15.67:0/2611965850 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff42c06bbd0 msgr2=0x7ff42c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:38.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.855+0000 7ff445109640 1 --2- 172.21.15.67:0/2611965850 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff42c06bbd0 0x7ff42c06e090 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ff430036000 tx=0x7ff43003c040 comp rx=0 tx=0).stop 2024-03-24T00:41:38.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.855+0000 7ff445109640 1 -- 172.21.15.67:0/2611965850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff44015e0f0 msgr2=0x7ff440151130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:38.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.855+0000 7ff445109640 1 --2- 172.21.15.67:0/2611965850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff44015e0f0 0x7ff440151130 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7ff43800cb10 tx=0x7ff43800a060 comp rx=0 tx=0).stop 2024-03-24T00:41:38.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.855+0000 7ff445109640 1 -- 172.21.15.67:0/2611965850 shutdown_connections 2024-03-24T00:41:38.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.855+0000 7ff445109640 1 --2- 172.21.15.67:0/2611965850 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff42c06bbd0 0x7ff42c06e090 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:38.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.855+0000 7ff445109640 1 --2- 172.21.15.67:0/2611965850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff44015e0f0 0x7ff440151130 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:38.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.855+0000 7ff445109640 1 --2- 172.21.15.67:0/2611965850 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff4400d3810 0x7ff44015fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:38.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.855+0000 7ff445109640 1 --2- 172.21.15.67:0/2611965850 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff4400b9af0 0x7ff440161170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:38.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.855+0000 7ff445109640 1 -- 172.21.15.67:0/2611965850 >> 172.21.15.67:0/2611965850 conn(0x7ff440072b60 msgr2=0x7ff4400d0b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:38.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.855+0000 7ff445109640 1 -- 172.21.15.67:0/2611965850 shutdown_connections 2024-03-24T00:41:38.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.855+0000 7ff445109640 1 -- 172.21.15.67:0/2611965850 wait complete. 2024-03-24T00:41:38.868 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-24T00:41:38.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.971+0000 7f4718027640 1 Processor -- start 2024-03-24T00:41:38.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.971+0000 7f4718027640 1 -- start start 2024-03-24T00:41:38.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.972+0000 7f4718027640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f471015dec0 0x7f47100ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:38.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.972+0000 7f4718027640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f471015e2a0 0x7f4710162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:38.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.972+0000 7f4718027640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47100cf640 con 0x7f47100d07e0 2024-03-24T00:41:38.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.972+0000 7f4718027640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47100cee30 con 0x7f471015dec0 2024-03-24T00:41:38.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.972+0000 7f4718027640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47100d2120 con 0x7f471015e2a0 2024-03-24T00:41:38.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.972+0000 7f47165c4640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f47100d07e0 0x7f47100d0be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58392/0 (socket says 172.21.15.67:58392) 2024-03-24T00:41:38.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.972+0000 7f4715dc3640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f471015dec0 0x7f47100ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:38.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.972+0000 7f47165c4640 1 -- 172.21.15.67:0/2692971779 learned_addr learned my addr 172.21.15.67:0/2692971779 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:41:38.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.973+0000 7f47155c2640 1 --2- 172.21.15.67:0/2692971779 >> v2:172.21.15.72:3300/0 conn(0x7f471015e2a0 0x7f4710162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:38.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.973+0000 7f4715dc3640 1 -- 172.21.15.67:0/2692971779 >> v2:172.21.15.72:3300/0 conn(0x7f471015e2a0 msgr2=0x7f4710162630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:38.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.973+0000 7f4715dc3640 1 --2- 172.21.15.67:0/2692971779 >> v2:172.21.15.72:3300/0 conn(0x7f471015e2a0 0x7f4710162630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:38.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.973+0000 7f4715dc3640 1 -- 172.21.15.67:0/2692971779 >> v1:172.21.15.67:6789/0 conn(0x7f47100d07e0 legacy=0x7f47100d0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:38.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.973+0000 7f4715dc3640 1 -- 172.21.15.67:0/2692971779 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47100cf080 con 0x7f471015dec0 2024-03-24T00:41:38.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.973+0000 7f4714dc1640 1 -- 172.21.15.67:0/2692971779 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2883059880 0 0) 0x7f4700001490 con 0x7f47100d07e0 2024-03-24T00:41:38.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.973+0000 7f47155c2640 1 --2- 172.21.15.67:0/2692971779 >> v2:172.21.15.72:3300/0 conn(0x7f471015e2a0 0x7f4710162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:41:38.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.973+0000 7f4715dc3640 1 --2- 172.21.15.67:0/2692971779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f471015dec0 0x7f47100ce630 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f47100cede0 tx=0x7f470802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eac4da6398abce36 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:38.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.974+0000 7f4714dc1640 1 -- 172.21.15.67:0/2692971779 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47080315a0 con 0x7f471015dec0 2024-03-24T00:41:38.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.974+0000 7f4714dc1640 1 -- 172.21.15.67:0/2692971779 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4708037070 con 0x7f471015dec0 2024-03-24T00:41:38.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.974+0000 7f4714dc1640 1 -- 172.21.15.67:0/2692971779 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4708031960 con 0x7f471015dec0 2024-03-24T00:41:38.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.974+0000 7f4718027640 1 -- 172.21.15.67:0/2692971779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f471015dec0 msgr2=0x7f47100ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:38.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.974+0000 7f4718027640 1 --2- 172.21.15.67:0/2692971779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f471015dec0 0x7f47100ce630 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f47100cede0 tx=0x7f470802b840 comp rx=0 tx=0).stop 2024-03-24T00:41:38.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.974+0000 7f4718027640 1 -- 172.21.15.67:0/2692971779 shutdown_connections 2024-03-24T00:41:38.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.974+0000 7f4718027640 1 --2- 172.21.15.67:0/2692971779 >> v2:172.21.15.72:3300/0 conn(0x7f471015e2a0 0x7f4710162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:38.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.974+0000 7f4718027640 1 --2- 172.21.15.67:0/2692971779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f471015dec0 0x7f47100ce630 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:38.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.974+0000 7f4718027640 1 -- 172.21.15.67:0/2692971779 >> 172.21.15.67:0/2692971779 conn(0x7f471006c810 msgr2=0x7f471006cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:38.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.974+0000 7f4718027640 1 -- 172.21.15.67:0/2692971779 shutdown_connections 2024-03-24T00:41:38.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.974+0000 7f4718027640 1 -- 172.21.15.67:0/2692971779 wait complete. 2024-03-24T00:41:38.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.975+0000 7f4718027640 1 Processor -- start 2024-03-24T00:41:38.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.975+0000 7f4718027640 1 -- start start 2024-03-24T00:41:38.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.976+0000 7f4718027640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47100d07e0 0x7f47101f3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:38.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.976+0000 7f4718027640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f471015dec0 0x7f4710156d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:38.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.976+0000 7f4718027640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f471015e2a0 0x7f47101572d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:38.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.976+0000 7f4718027640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47100d2730 con 0x7f47100d07e0 2024-03-24T00:41:38.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.976+0000 7f4718027640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47100d25b0 con 0x7f471015e2a0 2024-03-24T00:41:38.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.976+0000 7f4718027640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47100d28b0 con 0x7f471015dec0 2024-03-24T00:41:38.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.976+0000 7f4715dc3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47100d07e0 0x7f47101f3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:38.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.976+0000 7f4715dc3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47100d07e0 0x7f47101f3870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43824/0 (socket says 172.21.15.67:43824) 2024-03-24T00:41:38.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.976+0000 7f4715dc3640 1 -- 172.21.15.67:0/2528390547 learned_addr learned my addr 172.21.15.67:0/2528390547 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:38.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.976+0000 7f4715dc3640 1 -- 172.21.15.67:0/2528390547 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f471015dec0 msgr2=0x7f4710156d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:38.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.976+0000 7f4715dc3640 1 --2- 172.21.15.67:0/2528390547 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f471015dec0 0x7f4710156d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:38.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.976+0000 7f4715dc3640 1 -- 172.21.15.67:0/2528390547 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f471015e2a0 msgr2=0x7f47101572d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:38.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.976+0000 7f4715dc3640 1 --2- 172.21.15.67:0/2528390547 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f471015e2a0 0x7f47101572d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:38.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.976+0000 7f4715dc3640 1 -- 172.21.15.67:0/2528390547 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47101f3db0 con 0x7f47100d07e0 2024-03-24T00:41:38.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.977+0000 7f4715dc3640 1 --2- 172.21.15.67:0/2528390547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47100d07e0 0x7f47101f3870 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f4708000c00 tx=0x7f47080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:38.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.977+0000 7f470e7fc640 1 -- 172.21.15.67:0/2528390547 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4708004230 con 0x7f47100d07e0 2024-03-24T00:41:38.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.977+0000 7f470e7fc640 1 -- 172.21.15.67:0/2528390547 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47080043d0 con 0x7f47100d07e0 2024-03-24T00:41:38.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.977+0000 7f4718027640 1 -- 172.21.15.67:0/2528390547 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4710159c60 con 0x7f47100d07e0 2024-03-24T00:41:38.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.977+0000 7f470e7fc640 1 -- 172.21.15.67:0/2528390547 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4708032880 con 0x7f47100d07e0 2024-03-24T00:41:38.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.977+0000 7f4718027640 1 -- 172.21.15.67:0/2528390547 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47100cba60 con 0x7f47100d07e0 2024-03-24T00:41:38.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.978+0000 7f4718027640 1 -- 172.21.15.67:0/2528390547 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47100cb850 con 0x7f47100d07e0 2024-03-24T00:41:38.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.978+0000 7f470e7fc640 1 -- 172.21.15.67:0/2528390547 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4708032a20 con 0x7f47100d07e0 2024-03-24T00:41:38.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.978+0000 7f470e7fc640 1 --2- 172.21.15.67:0/2528390547 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f46fc06bbd0 0x7f46fc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:38.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.979+0000 7f470e7fc640 1 -- 172.21.15.67:0/2528390547 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f470802f030 con 0x7f47100d07e0 2024-03-24T00:41:38.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.979+0000 7f47155c2640 1 --2- 172.21.15.67:0/2528390547 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f46fc06bbd0 0x7f46fc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:38.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.979+0000 7f47155c2640 1 --2- 172.21.15.67:0/2528390547 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f46fc06bbd0 0x7f46fc06e090 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f4704006fd0 tx=0x7f4704008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:38.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:38.983+0000 7f470e7fc640 1 -- 172.21.15.67:0/2528390547 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4708041070 con 0x7f47100d07e0 2024-03-24T00:41:39.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.232+0000 7f4718027640 1 -- 172.21.15.67:0/2528390547 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f471004f400 con 0x7f47100d07e0 2024-03-24T00:41:39.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.233+0000 7f470e7fc640 1 -- 172.21.15.67:0/2528390547 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f4708044340 con 0x7f47100d07e0 2024-03-24T00:41:39.234 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:41:39.234 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:41:39.234 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_dump": [ 2024-03-24T00:41:39.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:41:39.234 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.a", 2024-03-24T00:41:39.234 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9l4UENDhAASKUDtqcbSQ7wADj7uo4eyQ==", 2024-03-24T00:41:39.234 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:41:39.234 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:41:39.234 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-03-24T00:41:39.234 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-03-24T00:41:39.234 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:41:39.234 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:41:39.234 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:41:39.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:41:39.235 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.b", 2024-03-24T00:41:39.235 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9luhJWGhAAsS6PFHV2XdF9htjIjzdrdA==", 2024-03-24T00:41:39.235 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:41:39.235 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:41:39.235 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-03-24T00:41:39.235 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-03-24T00:41:39.235 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:41:39.235 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:41:39.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:41:39.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:41:39.235 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.c", 2024-03-24T00:41:39.235 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9l8mRjFRAAQ73yexNddpcyp8ayTcRKCg==", 2024-03-24T00:41:39.235 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:41:39.235 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:41:39.236 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-03-24T00:41:39.236 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-03-24T00:41:39.236 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:41:39.236 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:41:39.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:41:39.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:41:39.236 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.d", 2024-03-24T00:41:39.236 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9lcJyvIRAA2tpDb6qxn1MtljjEno4qPA==", 2024-03-24T00:41:39.236 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:41:39.236 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:41:39.236 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-03-24T00:41:39.236 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-03-24T00:41:39.236 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:41:39.236 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:41:39.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:41:39.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:41:39.237 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.0", 2024-03-24T00:41:39.237 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDgdf9lV/NUBRAAyJNajFAnNABJErnBlB8TEA==", 2024-03-24T00:41:39.237 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:41:39.237 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:41:39.237 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:41:39.237 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:41:39.237 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:41:39.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:41:39.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:41:39.237 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.1", 2024-03-24T00:41:39.237 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDjdf9l0J6lAxAA9RInt/X22jqzthE0vOiiFg==", 2024-03-24T00:41:39.237 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:41:39.237 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:41:39.237 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:41:39.237 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:41:39.238 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:41:39.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:41:39.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:41:39.238 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.2", 2024-03-24T00:41:39.238 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDmdf9lSmrEARAAwmtSQRNePsn4WIZODnjhfQ==", 2024-03-24T00:41:39.238 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:41:39.238 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:41:39.238 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:41:39.238 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:41:39.238 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:41:39.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:41:39.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:41:39.238 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.3", 2024-03-24T00:41:39.238 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDpdf9lm59bABAAgJ5AV774lw2BgkSCgaweyA==", 2024-03-24T00:41:39.238 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:41:39.239 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:41:39.239 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:41:39.239 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:41:39.239 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:41:39.239 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:41:39.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:41:39.239 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.4", 2024-03-24T00:41:39.239 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDvdf9lCo+pCRAANHkL0pPxxok6IsBCLAFCbg==", 2024-03-24T00:41:39.239 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:41:39.239 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:41:39.239 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:41:39.239 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:41:39.239 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:41:39.239 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:41:39.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:41:39.240 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.5", 2024-03-24T00:41:39.240 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDydf9lyGEdCBAAcuszPsthGlLvVryE7+D9iQ==", 2024-03-24T00:41:39.240 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:41:39.240 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:41:39.240 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:41:39.240 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:41:39.240 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:41:39.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:41:39.240 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:41:39.240 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.6", 2024-03-24T00:41:39.240 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQD1df9ljuZqBhAACt2OPNkjKOD3rK7g2mVBBA==", 2024-03-24T00:41:39.240 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:41:39.240 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:41:39.240 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:41:39.240 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:41:39.240 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:41:39.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:41:39.241 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:41:39.241 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.7", 2024-03-24T00:41:39.241 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQD4df9lYKBfBRAAy/cWFF6pvGDjXZHAuBYw7g==", 2024-03-24T00:41:39.241 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:41:39.241 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:41:39.241 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:41:39.241 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:41:39.241 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:41:39.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:41:39.241 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:41:39.241 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.0", 2024-03-24T00:41:39.241 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9lY4nvJRAA6HWOZDWA+SSvJVmyjG+51g==", 2024-03-24T00:41:39.241 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:41:39.241 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:41:39.242 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow r", 2024-03-24T00:41:39.242 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-24T00:41:39.242 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:41:39.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:41:39.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:41:39.242 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.1", 2024-03-24T00:41:39.242 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9lonsXLxAAaltVMW3n0qHX8u9X8EkSmg==", 2024-03-24T00:41:39.242 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:41:39.242 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:41:39.242 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-03-24T00:41:39.242 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-03-24T00:41:39.242 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-03-24T00:41:39.242 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:41:39.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:41:39.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:41:39.243 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.2", 2024-03-24T00:41:39.243 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9lLEm2KxAAkT65mhLrTirlDnOUyvBY7g==", 2024-03-24T00:41:39.243 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:41:39.243 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:41:39.243 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-03-24T00:41:39.243 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-03-24T00:41:39.243 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-03-24T00:41:39.243 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:41:39.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:41:39.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:41:39.243 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.3", 2024-03-24T00:41:39.243 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9l8njZNBAAaAktZpX2SPcJR9ymPm9ebw==", 2024-03-24T00:41:39.243 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:41:39.243 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:41:39.243 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-03-24T00:41:39.243 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-03-24T00:41:39.244 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-03-24T00:41:39.244 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:41:39.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:41:39.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:41:39.244 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.admin", 2024-03-24T00:41:39.244 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDbdf9lDTIwLRAAMajqhM/KfUNF/SjVv9OJDw==", 2024-03-24T00:41:39.244 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:41:39.244 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-03-24T00:41:39.244 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-03-24T00:41:39.244 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-03-24T00:41:39.244 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:41:39.244 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:41:39.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:41:39.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:41:39.244 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.x", 2024-03-24T00:41:39.244 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9lGwKvChAAxl+fRnjbAQ7dYJ727QFUgw==", 2024-03-24T00:41:39.245 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:41:39.245 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-03-24T00:41:39.245 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-03-24T00:41:39.245 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:41:39.245 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:41:39.245 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:41:39.245 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:41:39.245 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.y", 2024-03-24T00:41:39.245 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9lSvufBhAAoQ9L1GjLkstjrAw2lqlryA==", 2024-03-24T00:41:39.245 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:41:39.245 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-03-24T00:41:39.245 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-03-24T00:41:39.245 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:41:39.245 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:41:39.245 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:41:39.246 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-24T00:41:39.246 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:41:39.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.234+0000 7f4718027640 1 -- 172.21.15.67:0/2528390547 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f46fc06bbd0 msgr2=0x7f46fc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:39.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.234+0000 7f4718027640 1 --2- 172.21.15.67:0/2528390547 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f46fc06bbd0 0x7f46fc06e090 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f4704006fd0 tx=0x7f4704008040 comp rx=0 tx=0).stop 2024-03-24T00:41:39.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.234+0000 7f4718027640 1 -- 172.21.15.67:0/2528390547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47100d07e0 msgr2=0x7f47101f3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:39.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.234+0000 7f4718027640 1 --2- 172.21.15.67:0/2528390547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47100d07e0 0x7f47101f3870 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f4708000c00 tx=0x7f47080046c0 comp rx=0 tx=0).stop 2024-03-24T00:41:39.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.234+0000 7f4718027640 1 -- 172.21.15.67:0/2528390547 shutdown_connections 2024-03-24T00:41:39.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.234+0000 7f4718027640 1 --2- 172.21.15.67:0/2528390547 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f46fc06bbd0 0x7f46fc06e090 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:39.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.234+0000 7f4718027640 1 --2- 172.21.15.67:0/2528390547 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f471015e2a0 0x7f47101572d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:39.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.234+0000 7f4718027640 1 --2- 172.21.15.67:0/2528390547 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f471015dec0 0x7f4710156d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:39.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.234+0000 7f4718027640 1 --2- 172.21.15.67:0/2528390547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47100d07e0 0x7f47101f3870 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:39.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.234+0000 7f4718027640 1 -- 172.21.15.67:0/2528390547 >> 172.21.15.67:0/2528390547 conn(0x7f471006c810 msgr2=0x7f471015e6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:39.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.235+0000 7f4718027640 1 -- 172.21.15.67:0/2528390547 shutdown_connections 2024-03-24T00:41:39.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.235+0000 7f4718027640 1 -- 172.21.15.67:0/2528390547 wait complete. 2024-03-24T00:41:39.248 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:41:39.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.349+0000 7f455d505640 1 Processor -- start 2024-03-24T00:41:39.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.349+0000 7f455d505640 1 -- start start 2024-03-24T00:41:39.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.350+0000 7f455d505640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f455815dec0 0x7f45580ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:39.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.350+0000 7f455d505640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f455815e2a0 0x7f4558162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:39.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.350+0000 7f455d505640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45580cf640 con 0x7f45580d07e0 2024-03-24T00:41:39.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.350+0000 7f455d505640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45580cee30 con 0x7f455815e2a0 2024-03-24T00:41:39.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.350+0000 7f455d505640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45580d2120 con 0x7f455815dec0 2024-03-24T00:41:39.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.350+0000 7f454ffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f455815e2a0 0x7f4558162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:39.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.350+0000 7f454ffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f455815e2a0 0x7f4558162630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43838/0 (socket says 172.21.15.67:43838) 2024-03-24T00:41:39.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.350+0000 7f454ffff640 1 -- 172.21.15.67:0/774048315 learned_addr learned my addr 172.21.15.67:0/774048315 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:39.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.350+0000 7f4556ffd640 1 --2- 172.21.15.67:0/774048315 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f455815dec0 0x7f45580ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:39.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.350+0000 7f454ffff640 1 -- 172.21.15.67:0/774048315 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f455815dec0 msgr2=0x7f45580ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:39.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.350+0000 7f454ffff640 1 --2- 172.21.15.67:0/774048315 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f455815dec0 0x7f45580ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:39.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.350+0000 7f454ffff640 1 -- 172.21.15.67:0/774048315 >> v1:172.21.15.72:6789/0 conn(0x7f45580d07e0 legacy=0x7f45580b5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:39.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.350+0000 7f454ffff640 1 -- 172.21.15.67:0/774048315 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45580cf080 con 0x7f455815e2a0 2024-03-24T00:41:39.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.351+0000 7f454ffff640 1 --2- 172.21.15.67:0/774048315 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f455815e2a0 0x7f4558162630 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f45580cefb0 tx=0x7f454002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88207cedf9543df9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:39.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.351+0000 7f45567fc640 1 -- 172.21.15.67:0/774048315 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4540035020 con 0x7f455815e2a0 2024-03-24T00:41:39.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.351+0000 7f45567fc640 1 -- 172.21.15.67:0/774048315 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4540031640 con 0x7f455815e2a0 2024-03-24T00:41:39.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.351+0000 7f45567fc640 1 -- 172.21.15.67:0/774048315 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f454002d040 con 0x7f455815e2a0 2024-03-24T00:41:39.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.352+0000 7f455d505640 1 -- 172.21.15.67:0/774048315 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f455815e2a0 msgr2=0x7f4558162630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:39.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.352+0000 7f455d505640 1 --2- 172.21.15.67:0/774048315 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f455815e2a0 0x7f4558162630 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f45580cefb0 tx=0x7f454002b8f0 comp rx=0 tx=0).stop 2024-03-24T00:41:39.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.352+0000 7f455d505640 1 -- 172.21.15.67:0/774048315 shutdown_connections 2024-03-24T00:41:39.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.352+0000 7f455d505640 1 --2- 172.21.15.67:0/774048315 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f455815e2a0 0x7f4558162630 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:39.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.352+0000 7f455d505640 1 --2- 172.21.15.67:0/774048315 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f455815dec0 0x7f45580ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:39.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.352+0000 7f455d505640 1 -- 172.21.15.67:0/774048315 >> 172.21.15.67:0/774048315 conn(0x7f455806c810 msgr2=0x7f455806cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:39.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.352+0000 7f455d505640 1 -- 172.21.15.67:0/774048315 shutdown_connections 2024-03-24T00:41:39.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.352+0000 7f455d505640 1 -- 172.21.15.67:0/774048315 wait complete. 2024-03-24T00:41:39.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.353+0000 7f455d505640 1 Processor -- start 2024-03-24T00:41:39.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.353+0000 7f455d505640 1 -- start start 2024-03-24T00:41:39.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.354+0000 7f455d505640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f45580d07e0 0x7f45580cd7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:39.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.354+0000 7f455d505640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f455815dec0 0x7f45580ac7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:39.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.354+0000 7f455d505640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f455815e2a0 0x7f45580accf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:39.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.354+0000 7f455d505640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45580d3740 con 0x7f455815dec0 2024-03-24T00:41:39.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.354+0000 7f455d505640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45580d35c0 con 0x7f455815e2a0 2024-03-24T00:41:39.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.354+0000 7f455d505640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45580d38c0 con 0x7f45580d07e0 2024-03-24T00:41:39.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.354+0000 7f454ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f455815dec0 0x7f45580ac7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:39.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.354+0000 7f454ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f455815dec0 0x7f45580ac7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43840/0 (socket says 172.21.15.67:43840) 2024-03-24T00:41:39.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.354+0000 7f454ffff640 1 -- 172.21.15.67:0/1919945630 learned_addr learned my addr 172.21.15.67:0/1919945630 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:39.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.354+0000 7f4556ffd640 1 --2- 172.21.15.67:0/1919945630 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f45580d07e0 0x7f45580cd7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:39.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.354+0000 7f45577fe640 1 --2- 172.21.15.67:0/1919945630 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f455815e2a0 0x7f45580accf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:39.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.354+0000 7f454ffff640 1 -- 172.21.15.67:0/1919945630 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f45580d07e0 msgr2=0x7f45580cd7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:39.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.354+0000 7f454ffff640 1 --2- 172.21.15.67:0/1919945630 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f45580d07e0 0x7f45580cd7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:39.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.354+0000 7f454ffff640 1 -- 172.21.15.67:0/1919945630 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f455815e2a0 msgr2=0x7f45580accf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:39.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.354+0000 7f454ffff640 1 --2- 172.21.15.67:0/1919945630 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f455815e2a0 0x7f45580accf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:39.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.354+0000 7f454ffff640 1 -- 172.21.15.67:0/1919945630 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45580cbbe0 con 0x7f455815dec0 2024-03-24T00:41:39.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.354+0000 7f45577fe640 1 --2- 172.21.15.67:0/1919945630 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f455815e2a0 0x7f45580accf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:41:39.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.354+0000 7f454ffff640 1 --2- 172.21.15.67:0/1919945630 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f455815dec0 0x7f45580ac7b0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f4540000c00 tx=0x7f45400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:39.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.354+0000 7f454f7fe640 1 -- 172.21.15.67:0/1919945630 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4540040070 con 0x7f455815dec0 2024-03-24T00:41:39.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.354+0000 7f454f7fe640 1 -- 172.21.15.67:0/1919945630 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45400399f0 con 0x7f455815dec0 2024-03-24T00:41:39.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.354+0000 7f454f7fe640 1 -- 172.21.15.67:0/1919945630 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4540035040 con 0x7f455815dec0 2024-03-24T00:41:39.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.354+0000 7f455d505640 1 -- 172.21.15.67:0/1919945630 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45580cbec0 con 0x7f455815dec0 2024-03-24T00:41:39.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.355+0000 7f455d505640 1 -- 172.21.15.67:0/1919945630 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45580cc480 con 0x7f455815dec0 2024-03-24T00:41:39.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.355+0000 7f455d505640 1 -- 172.21.15.67:0/1919945630 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f455804f400 con 0x7f455815dec0 2024-03-24T00:41:39.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.356+0000 7f454f7fe640 1 -- 172.21.15.67:0/1919945630 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4540038020 con 0x7f455815dec0 2024-03-24T00:41:39.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.356+0000 7f454f7fe640 1 --2- 172.21.15.67:0/1919945630 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f453806bbd0 0x7f453806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:39.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.356+0000 7f454f7fe640 1 -- 172.21.15.67:0/1919945630 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f45400ac0d0 con 0x7f455815dec0 2024-03-24T00:41:39.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.356+0000 7f4556ffd640 1 --2- 172.21.15.67:0/1919945630 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f453806bbd0 0x7f453806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:39.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.357+0000 7f4556ffd640 1 --2- 172.21.15.67:0/1919945630 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f453806bbd0 0x7f453806e090 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f4548009ad0 tx=0x7f4548008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:39.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.361+0000 7f454f7fe640 1 -- 172.21.15.67:0/1919945630 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f454007ae10 con 0x7f455815dec0 2024-03-24T00:41:39.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.546+0000 7f455d505640 1 -- 172.21.15.67:0/1919945630 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f45580d1aa0 con 0x7f455815dec0 2024-03-24T00:41:39.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.546+0000 7f454f7fe640 1 -- 172.21.15.67:0/1919945630 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+13104 (secure 0 0 0) 0x7f45400a8340 con 0x7f455815dec0 2024-03-24T00:41:39.547 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:41:39.547 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":58,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:41:38.851395+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:41:39.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.548+0000 7f455d505640 1 -- 172.21.15.67:0/1919945630 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f453806bbd0 msgr2=0x7f453806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:39.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.548+0000 7f455d505640 1 --2- 172.21.15.67:0/1919945630 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f453806bbd0 0x7f453806e090 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f4548009ad0 tx=0x7f4548008040 comp rx=0 tx=0).stop 2024-03-24T00:41:39.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.548+0000 7f455d505640 1 -- 172.21.15.67:0/1919945630 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f455815dec0 msgr2=0x7f45580ac7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:39.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.548+0000 7f455d505640 1 --2- 172.21.15.67:0/1919945630 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f455815dec0 0x7f45580ac7b0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f4540000c00 tx=0x7f45400046c0 comp rx=0 tx=0).stop 2024-03-24T00:41:39.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.548+0000 7f455d505640 1 -- 172.21.15.67:0/1919945630 shutdown_connections 2024-03-24T00:41:39.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.548+0000 7f455d505640 1 --2- 172.21.15.67:0/1919945630 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f453806bbd0 0x7f453806e090 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:39.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.548+0000 7f455d505640 1 --2- 172.21.15.67:0/1919945630 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f455815e2a0 0x7f45580accf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:39.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.548+0000 7f455d505640 1 --2- 172.21.15.67:0/1919945630 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f455815dec0 0x7f45580ac7b0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:39.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.548+0000 7f455d505640 1 --2- 172.21.15.67:0/1919945630 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f45580d07e0 0x7f45580cd7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:39.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.548+0000 7f455d505640 1 -- 172.21.15.67:0/1919945630 >> 172.21.15.67:0/1919945630 conn(0x7f455806c810 msgr2=0x7f45580b86d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:39.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.548+0000 7f455d505640 1 -- 172.21.15.67:0/1919945630 shutdown_connections 2024-03-24T00:41:39.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.549+0000 7f455d505640 1 -- 172.21.15.67:0/1919945630 wait complete. 2024-03-24T00:41:39.560 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:38:39.364950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:41:39.561 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:41:39.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.664+0000 7fcd9dcc6640 1 Processor -- start 2024-03-24T00:41:39.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.664+0000 7fcd9dcc6640 1 -- start start 2024-03-24T00:41:39.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.664+0000 7fcd9dcc6640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcd98099d00 0x7fcd9809a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:39.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.664+0000 7fcd9dcc6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcd980b49b0 0x7fcd980d0e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:39.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.664+0000 7fcd9dcc6640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fcd980d1360 0x7fcd9815d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:39.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.664+0000 7fcd9dcc6640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd980cc1b0 con 0x7fcd98099d00 2024-03-24T00:41:39.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.664+0000 7fcd9dcc6640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd9809a620 con 0x7fcd980d1360 2024-03-24T00:41:39.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.664+0000 7fcd9dcc6640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd980cc400 con 0x7fcd980b49b0 2024-03-24T00:41:39.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.665+0000 7fcd977fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcd98099d00 0x7fcd9809a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:39.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.665+0000 7fcd977fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcd98099d00 0x7fcd9809a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43854/0 (socket says 172.21.15.67:43854) 2024-03-24T00:41:39.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.665+0000 7fcd977fe640 1 -- 172.21.15.67:0/317877783 learned_addr learned my addr 172.21.15.67:0/317877783 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:39.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.665+0000 7fcd97fff640 1 --2- 172.21.15.67:0/317877783 >> v2:172.21.15.72:3300/0 conn(0x7fcd980d1360 0x7fcd9815d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:39.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.665+0000 7fcd96ffd640 1 --2- 172.21.15.67:0/317877783 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcd980b49b0 0x7fcd980d0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:39.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.665+0000 7fcd977fe640 1 -- 172.21.15.67:0/317877783 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcd980b49b0 msgr2=0x7fcd980d0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:39.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.665+0000 7fcd977fe640 1 --2- 172.21.15.67:0/317877783 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcd980b49b0 0x7fcd980d0e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:39.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.665+0000 7fcd977fe640 1 -- 172.21.15.67:0/317877783 >> v2:172.21.15.72:3300/0 conn(0x7fcd980d1360 msgr2=0x7fcd9815d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:39.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.665+0000 7fcd977fe640 1 --2- 172.21.15.67:0/317877783 >> v2:172.21.15.72:3300/0 conn(0x7fcd980d1360 0x7fcd9815d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:39.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.665+0000 7fcd977fe640 1 -- 172.21.15.67:0/317877783 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd980cc6e0 con 0x7fcd98099d00 2024-03-24T00:41:39.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.666+0000 7fcd977fe640 1 --2- 172.21.15.67:0/317877783 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd98099d00 0x7fcd9809a0e0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fcd9806d400 tx=0x7fcd8802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef8cb2d63e32fbb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:39.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.666+0000 7fcd967fc640 1 -- 172.21.15.67:0/317877783 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd88035020 con 0x7fcd98099d00 2024-03-24T00:41:39.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.666+0000 7fcd967fc640 1 -- 172.21.15.67:0/317877783 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd88031870 con 0x7fcd98099d00 2024-03-24T00:41:39.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.666+0000 7fcd967fc640 1 -- 172.21.15.67:0/317877783 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd88031c40 con 0x7fcd98099d00 2024-03-24T00:41:39.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.666+0000 7fcd9dcc6640 1 -- 172.21.15.67:0/317877783 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd98099d00 msgr2=0x7fcd9809a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:39.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.666+0000 7fcd9dcc6640 1 --2- 172.21.15.67:0/317877783 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd98099d00 0x7fcd9809a0e0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fcd9806d400 tx=0x7fcd8802b910 comp rx=0 tx=0).stop 2024-03-24T00:41:39.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.666+0000 7fcd9dcc6640 1 -- 172.21.15.67:0/317877783 shutdown_connections 2024-03-24T00:41:39.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.666+0000 7fcd9dcc6640 1 --2- 172.21.15.67:0/317877783 >> v2:172.21.15.72:3300/0 conn(0x7fcd980d1360 0x7fcd9815d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:39.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.666+0000 7fcd9dcc6640 1 --2- 172.21.15.67:0/317877783 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcd980b49b0 0x7fcd980d0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:39.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.666+0000 7fcd9dcc6640 1 --2- 172.21.15.67:0/317877783 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd98099d00 0x7fcd9809a0e0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:39.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.666+0000 7fcd9dcc6640 1 -- 172.21.15.67:0/317877783 >> 172.21.15.67:0/317877783 conn(0x7fcd980b3650 msgr2=0x7fcd980b3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:39.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.666+0000 7fcd9dcc6640 1 -- 172.21.15.67:0/317877783 shutdown_connections 2024-03-24T00:41:39.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.666+0000 7fcd9dcc6640 1 -- 172.21.15.67:0/317877783 wait complete. 2024-03-24T00:41:39.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.668+0000 7fcd9dcc6640 1 Processor -- start 2024-03-24T00:41:39.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.668+0000 7fcd9dcc6640 1 -- start start 2024-03-24T00:41:39.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.668+0000 7fcd9dcc6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcd98099d00 0x7fcd9815dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:39.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.668+0000 7fcd9dcc6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcd980b49b0 0x7fcd980b5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:39.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.668+0000 7fcd9dcc6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd980d1360 0x7fcd980b6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:39.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.668+0000 7fcd9dcc6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd980d01c0 con 0x7fcd980d1360 2024-03-24T00:41:39.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.668+0000 7fcd9dcc6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd980ccca0 con 0x7fcd980b49b0 2024-03-24T00:41:39.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.668+0000 7fcd9dcc6640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd980b32a0 con 0x7fcd98099d00 2024-03-24T00:41:39.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.669+0000 7fcd97fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd980d1360 0x7fcd980b6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:39.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.669+0000 7fcd97fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd980d1360 0x7fcd980b6530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43868/0 (socket says 172.21.15.67:43868) 2024-03-24T00:41:39.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.669+0000 7fcd97fff640 1 -- 172.21.15.67:0/2052871191 learned_addr learned my addr 172.21.15.67:0/2052871191 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:39.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.669+0000 7fcd96ffd640 1 --2- 172.21.15.67:0/2052871191 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcd980b49b0 0x7fcd980b5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:39.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.669+0000 7fcd977fe640 1 --2- 172.21.15.67:0/2052871191 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcd98099d00 0x7fcd9815dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:39.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.669+0000 7fcd97fff640 1 -- 172.21.15.67:0/2052871191 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcd98099d00 msgr2=0x7fcd9815dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:39.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.669+0000 7fcd97fff640 1 --2- 172.21.15.67:0/2052871191 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcd98099d00 0x7fcd9815dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:39.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.669+0000 7fcd97fff640 1 -- 172.21.15.67:0/2052871191 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcd980b49b0 msgr2=0x7fcd980b5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:39.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.669+0000 7fcd97fff640 1 --2- 172.21.15.67:0/2052871191 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcd980b49b0 0x7fcd980b5ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:39.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.669+0000 7fcd97fff640 1 -- 172.21.15.67:0/2052871191 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd980c9e20 con 0x7fcd980d1360 2024-03-24T00:41:39.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.669+0000 7fcd977fe640 1 --2- 172.21.15.67:0/2052871191 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcd98099d00 0x7fcd9815dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:41:39.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.669+0000 7fcd97fff640 1 --2- 172.21.15.67:0/2052871191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd980d1360 0x7fcd980b6530 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fcd90007e60 tx=0x7fcd9000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:39.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.669+0000 7fcd7ffff640 1 -- 172.21.15.67:0/2052871191 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd90014420 con 0x7fcd980d1360 2024-03-24T00:41:39.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.669+0000 7fcd7ffff640 1 -- 172.21.15.67:0/2052871191 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd90019070 con 0x7fcd980d1360 2024-03-24T00:41:39.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.669+0000 7fcd9dcc6640 1 -- 172.21.15.67:0/2052871191 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd980ca100 con 0x7fcd980d1360 2024-03-24T00:41:39.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.669+0000 7fcd7ffff640 1 -- 172.21.15.67:0/2052871191 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd90014820 con 0x7fcd980d1360 2024-03-24T00:41:39.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.669+0000 7fcd9dcc6640 1 -- 172.21.15.67:0/2052871191 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd980ca6c0 con 0x7fcd980d1360 2024-03-24T00:41:39.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.670+0000 7fcd9dcc6640 1 -- 172.21.15.67:0/2052871191 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd9804f400 con 0x7fcd980d1360 2024-03-24T00:41:39.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.671+0000 7fcd7ffff640 1 -- 172.21.15.67:0/2052871191 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fcd900149c0 con 0x7fcd980d1360 2024-03-24T00:41:39.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.671+0000 7fcd7ffff640 1 --2- 172.21.15.67:0/2052871191 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcd8406bc00 0x7fcd8406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:39.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.671+0000 7fcd7ffff640 1 -- 172.21.15.67:0/2052871191 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fcd9008b030 con 0x7fcd980d1360 2024-03-24T00:41:39.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.671+0000 7fcd977fe640 1 --2- 172.21.15.67:0/2052871191 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcd8406bc00 0x7fcd8406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:39.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.672+0000 7fcd977fe640 1 --2- 172.21.15.67:0/2052871191 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcd8406bc00 0x7fcd8406e0c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fcd88036000 tx=0x7fcd8803c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:39.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.675+0000 7fcd7ffff640 1 -- 172.21.15.67:0/2052871191 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fcd90059ce0 con 0x7fcd980d1360 2024-03-24T00:41:39.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.856+0000 7fcd9dcc6640 1 -- 172.21.15.67:0/2052871191 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcd980718f0 con 0x7fcd980d1360 2024-03-24T00:41:39.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.857+0000 7fcd7ffff640 1 -- 172.21.15.67:0/2052871191 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v58) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcd900595e0 con 0x7fcd980d1360 2024-03-24T00:41:39.858 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-24T00:41:39.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.859+0000 7fcd9dcc6640 1 -- 172.21.15.67:0/2052871191 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcd8406bc00 msgr2=0x7fcd8406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:39.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.859+0000 7fcd9dcc6640 1 --2- 172.21.15.67:0/2052871191 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcd8406bc00 0x7fcd8406e0c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fcd88036000 tx=0x7fcd8803c040 comp rx=0 tx=0).stop 2024-03-24T00:41:39.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.859+0000 7fcd9dcc6640 1 -- 172.21.15.67:0/2052871191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd980d1360 msgr2=0x7fcd980b6530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:39.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.859+0000 7fcd9dcc6640 1 --2- 172.21.15.67:0/2052871191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd980d1360 0x7fcd980b6530 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fcd90007e60 tx=0x7fcd9000e4b0 comp rx=0 tx=0).stop 2024-03-24T00:41:39.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.859+0000 7fcd9dcc6640 1 -- 172.21.15.67:0/2052871191 shutdown_connections 2024-03-24T00:41:39.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.859+0000 7fcd9dcc6640 1 --2- 172.21.15.67:0/2052871191 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcd8406bc00 0x7fcd8406e0c0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:39.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.859+0000 7fcd9dcc6640 1 --2- 172.21.15.67:0/2052871191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd980d1360 0x7fcd980b6530 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:39.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.859+0000 7fcd9dcc6640 1 --2- 172.21.15.67:0/2052871191 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcd980b49b0 0x7fcd980b5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:39.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.859+0000 7fcd9dcc6640 1 --2- 172.21.15.67:0/2052871191 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcd98099d00 0x7fcd9815dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:39.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.859+0000 7fcd9dcc6640 1 -- 172.21.15.67:0/2052871191 >> 172.21.15.67:0/2052871191 conn(0x7fcd980b3650 msgr2=0x7fcd98070720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:39.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.859+0000 7fcd9dcc6640 1 -- 172.21.15.67:0/2052871191 shutdown_connections 2024-03-24T00:41:39.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.859+0000 7fcd9dcc6640 1 -- 172.21.15.67:0/2052871191 wait complete. 2024-03-24T00:41:39.871 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-24T00:41:39.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.971+0000 7f3d2933b640 1 Processor -- start 2024-03-24T00:41:39.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.972+0000 7f3d2933b640 1 -- start start 2024-03-24T00:41:39.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.972+0000 7f3d2933b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3d2415dec0 0x7f3d240ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:39.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.972+0000 7f3d2933b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3d2415e2a0 0x7f3d24162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:39.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.972+0000 7f3d2933b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d240cf640 con 0x7f3d240d07e0 2024-03-24T00:41:39.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.972+0000 7f3d2933b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d240cee30 con 0x7f3d2415dec0 2024-03-24T00:41:39.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.972+0000 7f3d2933b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d240d2120 con 0x7f3d2415e2a0 2024-03-24T00:41:39.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.972+0000 7f3d237fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3d240d07e0 0x7f3d240b5850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58398/0 (socket says 172.21.15.67:58398) 2024-03-24T00:41:39.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.972+0000 7f3d237fe640 1 -- 172.21.15.67:0/1882535688 learned_addr learned my addr 172.21.15.67:0/1882535688 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:41:39.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.973+0000 7f3d22ffd640 1 --2- 172.21.15.67:0/1882535688 >> v2:172.21.15.72:3300/0 conn(0x7f3d2415dec0 0x7f3d240ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:39.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.973+0000 7f3d1bfff640 1 --2- 172.21.15.67:0/1882535688 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3d2415e2a0 0x7f3d24162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:39.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.973+0000 7f3d227fc640 1 -- 172.21.15.67:0/1882535688 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4215768358 0 0) 0x7f3d1c001490 con 0x7f3d240d07e0 2024-03-24T00:41:39.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.973+0000 7f3d227fc640 1 -- 172.21.15.67:0/1882535688 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d10003520 con 0x7f3d240d07e0 2024-03-24T00:41:39.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.973+0000 7f3d22ffd640 1 -- 172.21.15.67:0/1882535688 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3d2415e2a0 msgr2=0x7f3d24162630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:39.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.973+0000 7f3d22ffd640 1 --2- 172.21.15.67:0/1882535688 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3d2415e2a0 0x7f3d24162630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:39.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.973+0000 7f3d22ffd640 1 -- 172.21.15.67:0/1882535688 >> v1:172.21.15.67:6789/0 conn(0x7f3d240d07e0 legacy=0x7f3d240b5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:39.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.973+0000 7f3d22ffd640 1 -- 172.21.15.67:0/1882535688 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d240cf080 con 0x7f3d2415dec0 2024-03-24T00:41:39.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.974+0000 7f3d22ffd640 1 --2- 172.21.15.67:0/1882535688 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d2415dec0 0x7f3d240ce630 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f3d240cefb0 tx=0x7f3d1402b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1642629c67a6ef39 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:39.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.974+0000 7f3d227fc640 1 -- 172.21.15.67:0/1882535688 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d14031770 con 0x7f3d2415dec0 2024-03-24T00:41:39.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.974+0000 7f3d227fc640 1 -- 172.21.15.67:0/1882535688 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d14037070 con 0x7f3d2415dec0 2024-03-24T00:41:39.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.974+0000 7f3d227fc640 1 -- 172.21.15.67:0/1882535688 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d14031b10 con 0x7f3d2415dec0 2024-03-24T00:41:39.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.974+0000 7f3d2933b640 1 -- 172.21.15.67:0/1882535688 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d2415dec0 msgr2=0x7f3d240ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:39.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.974+0000 7f3d2933b640 1 --2- 172.21.15.67:0/1882535688 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d2415dec0 0x7f3d240ce630 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f3d240cefb0 tx=0x7f3d1402b810 comp rx=0 tx=0).stop 2024-03-24T00:41:39.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.974+0000 7f3d2933b640 1 -- 172.21.15.67:0/1882535688 shutdown_connections 2024-03-24T00:41:39.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.975+0000 7f3d2933b640 1 --2- 172.21.15.67:0/1882535688 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3d2415e2a0 0x7f3d24162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:39.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.975+0000 7f3d2933b640 1 --2- 172.21.15.67:0/1882535688 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d2415dec0 0x7f3d240ce630 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:39.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.975+0000 7f3d2933b640 1 -- 172.21.15.67:0/1882535688 >> 172.21.15.67:0/1882535688 conn(0x7f3d2406c810 msgr2=0x7f3d2406cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:39.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.975+0000 7f3d2933b640 1 -- 172.21.15.67:0/1882535688 shutdown_connections 2024-03-24T00:41:39.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.975+0000 7f3d2933b640 1 -- 172.21.15.67:0/1882535688 wait complete. 2024-03-24T00:41:39.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.976+0000 7f3d2933b640 1 Processor -- start 2024-03-24T00:41:39.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.976+0000 7f3d2933b640 1 -- start start 2024-03-24T00:41:39.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.976+0000 7f3d2933b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d240d07e0 0x7f3d240cbce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:39.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.976+0000 7f3d2933b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d2415dec0 0x7f3d240cdc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:39.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.976+0000 7f3d2933b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3d2415e2a0 0x7f3d240b4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:39.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.976+0000 7f3d2933b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d240d3740 con 0x7f3d2415dec0 2024-03-24T00:41:39.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.976+0000 7f3d2933b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d240d35c0 con 0x7f3d240d07e0 2024-03-24T00:41:39.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.976+0000 7f3d2933b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d240d38c0 con 0x7f3d2415e2a0 2024-03-24T00:41:39.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.976+0000 7f3d1bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d2415dec0 0x7f3d240cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:39.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.976+0000 7f3d1bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d2415dec0 0x7f3d240cdc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43876/0 (socket says 172.21.15.67:43876) 2024-03-24T00:41:39.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.976+0000 7f3d1bfff640 1 -- 172.21.15.67:0/1162339736 learned_addr learned my addr 172.21.15.67:0/1162339736 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:39.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.977+0000 7f3d22ffd640 1 --2- 172.21.15.67:0/1162339736 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d240d07e0 0x7f3d240cbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:39.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.977+0000 7f3d237fe640 1 --2- 172.21.15.67:0/1162339736 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3d2415e2a0 0x7f3d240b4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:39.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.977+0000 7f3d1bfff640 1 -- 172.21.15.67:0/1162339736 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3d2415e2a0 msgr2=0x7f3d240b4220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:39.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.977+0000 7f3d1bfff640 1 --2- 172.21.15.67:0/1162339736 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3d2415e2a0 0x7f3d240b4220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:39.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.977+0000 7f3d1bfff640 1 -- 172.21.15.67:0/1162339736 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d240d07e0 msgr2=0x7f3d240cbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:39.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.977+0000 7f3d1bfff640 1 --2- 172.21.15.67:0/1162339736 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d240d07e0 0x7f3d240cbce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:39.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.977+0000 7f3d1bfff640 1 -- 172.21.15.67:0/1162339736 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d240ce180 con 0x7f3d2415dec0 2024-03-24T00:41:39.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.977+0000 7f3d1bfff640 1 --2- 172.21.15.67:0/1162339736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d2415dec0 0x7f3d240cdc40 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c00e840 tx=0x7f3d0c00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:39.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.977+0000 7f3d1b7fe640 1 -- 172.21.15.67:0/1162339736 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d0c017070 con 0x7f3d2415dec0 2024-03-24T00:41:39.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.977+0000 7f3d1b7fe640 1 -- 172.21.15.67:0/1162339736 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d0c00c940 con 0x7f3d2415dec0 2024-03-24T00:41:39.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.977+0000 7f3d2933b640 1 -- 172.21.15.67:0/1162339736 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d240b4c50 con 0x7f3d2415dec0 2024-03-24T00:41:39.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.977+0000 7f3d1b7fe640 1 -- 172.21.15.67:0/1162339736 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d0c00cd10 con 0x7f3d2415dec0 2024-03-24T00:41:39.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.977+0000 7f3d2933b640 1 -- 172.21.15.67:0/1162339736 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d24157210 con 0x7f3d2415dec0 2024-03-24T00:41:39.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.978+0000 7f3d2933b640 1 -- 172.21.15.67:0/1162339736 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d2404f400 con 0x7f3d2415dec0 2024-03-24T00:41:39.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.979+0000 7f3d1b7fe640 1 -- 172.21.15.67:0/1162339736 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3d0c01a070 con 0x7f3d2415dec0 2024-03-24T00:41:39.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.979+0000 7f3d1b7fe640 1 --2- 172.21.15.67:0/1162339736 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3d0406bc00 0x7f3d0406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:39.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.979+0000 7f3d1b7fe640 1 -- 172.21.15.67:0/1162339736 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f3d0c08a9a0 con 0x7f3d2415dec0 2024-03-24T00:41:39.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.979+0000 7f3d22ffd640 1 --2- 172.21.15.67:0/1162339736 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3d0406bc00 0x7f3d0406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:39.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.980+0000 7f3d22ffd640 1 --2- 172.21.15.67:0/1162339736 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3d0406bc00 0x7f3d0406e0c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f3d14036000 tx=0x7f3d1403d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:39.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:39.983+0000 7f3d1b7fe640 1 -- 172.21.15.67:0/1162339736 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3d0c059090 con 0x7f3d2415dec0 2024-03-24T00:41:40.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.167+0000 7f3d2933b640 1 -- 172.21.15.67:0/1162339736 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f3d240ce410 con 0x7f3d2415dec0 2024-03-24T00:41:40.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.168+0000 7f3d1b7fe640 1 -- 172.21.15.67:0/1162339736 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v58) v1 ==== 84+0+65 (secure 0 0 0) 0x7f3d0c05d130 con 0x7f3d2415dec0 2024-03-24T00:41:40.169 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:41:40.169 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-03-24T00:41:40.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:41:40.169 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2024-03-24T00:41:40.169 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2024-03-24T00:41:40.169 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:41:40.169 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-03-24T00:41:40.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.169+0000 7f3d2933b640 1 -- 172.21.15.67:0/1162339736 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3d0406bc00 msgr2=0x7f3d0406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:40.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.169+0000 7f3d2933b640 1 --2- 172.21.15.67:0/1162339736 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3d0406bc00 0x7f3d0406e0c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f3d14036000 tx=0x7f3d1403d040 comp rx=0 tx=0).stop 2024-03-24T00:41:40.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.169+0000 7f3d2933b640 1 -- 172.21.15.67:0/1162339736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d2415dec0 msgr2=0x7f3d240cdc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:40.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.169+0000 7f3d2933b640 1 --2- 172.21.15.67:0/1162339736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d2415dec0 0x7f3d240cdc40 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c00e840 tx=0x7f3d0c00ed00 comp rx=0 tx=0).stop 2024-03-24T00:41:40.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.169+0000 7f3d2933b640 1 -- 172.21.15.67:0/1162339736 shutdown_connections 2024-03-24T00:41:40.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.169+0000 7f3d2933b640 1 --2- 172.21.15.67:0/1162339736 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3d0406bc00 0x7f3d0406e0c0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:40.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.169+0000 7f3d2933b640 1 --2- 172.21.15.67:0/1162339736 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3d2415e2a0 0x7f3d240b4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:40.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.169+0000 7f3d2933b640 1 --2- 172.21.15.67:0/1162339736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d2415dec0 0x7f3d240cdc40 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:40.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.169+0000 7f3d2933b640 1 --2- 172.21.15.67:0/1162339736 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d240d07e0 0x7f3d240cbce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:40.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.169+0000 7f3d2933b640 1 -- 172.21.15.67:0/1162339736 >> 172.21.15.67:0/1162339736 conn(0x7f3d2406c810 msgr2=0x7f3d24161030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:40.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.170+0000 7f3d2933b640 1 -- 172.21.15.67:0/1162339736 shutdown_connections 2024-03-24T00:41:40.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.170+0000 7f3d2933b640 1 -- 172.21.15.67:0/1162339736 wait complete. 2024-03-24T00:41:40.181 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-24T00:41:40.181 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-24T00:41:40.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.286+0000 7f175011b640 1 Processor -- start 2024-03-24T00:41:40.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.286+0000 7f175011b640 1 -- start start 2024-03-24T00:41:40.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.286+0000 7f175011b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f17480cd230 0x7f17480d0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:40.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.286+0000 7f175011b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f17480cdb50 0x7f174806b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:40.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.286+0000 7f175011b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17480703c0 con 0x7f17480d00b0 2024-03-24T00:41:40.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.286+0000 7f175011b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1748070170 con 0x7f17480cdb50 2024-03-24T00:41:40.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.286+0000 7f175011b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17480706a0 con 0x7f17480cd230 2024-03-24T00:41:40.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.287+0000 7f174e6b8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f17480cdb50 0x7f174806b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:40.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.287+0000 7f174deb7640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f17480d00b0 0x7f1748161b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58402/0 (socket says 172.21.15.67:58402) 2024-03-24T00:41:40.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.287+0000 7f174deb7640 1 -- 172.21.15.67:0/3107026575 learned_addr learned my addr 172.21.15.67:0/3107026575 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:41:40.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.287+0000 7f174d6b6640 1 --2- 172.21.15.67:0/3107026575 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f17480cd230 0x7f17480d0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:40.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.287+0000 7f174e6b8640 1 -- 172.21.15.67:0/3107026575 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f17480cd230 msgr2=0x7f17480d0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:40.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.287+0000 7f174e6b8640 1 --2- 172.21.15.67:0/3107026575 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f17480cd230 0x7f17480d0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:40.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.287+0000 7f174e6b8640 1 -- 172.21.15.67:0/3107026575 >> v1:172.21.15.67:6789/0 conn(0x7f17480d00b0 legacy=0x7f1748161b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:40.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.287+0000 7f174e6b8640 1 -- 172.21.15.67:0/3107026575 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1748070980 con 0x7f17480cdb50 2024-03-24T00:41:40.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.288+0000 7f174e6b8640 1 --2- 172.21.15.67:0/3107026575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17480cdb50 0x7f174806b040 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f1740009d90 tx=0x7f174002be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2475767aa597e68 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:40.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.288+0000 7f174ceb5640 1 -- 172.21.15.67:0/3107026575 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1740039070 con 0x7f17480cdb50 2024-03-24T00:41:40.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.288+0000 7f174ceb5640 1 -- 172.21.15.67:0/3107026575 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f174002cbc0 con 0x7f17480cdb50 2024-03-24T00:41:40.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.288+0000 7f174ceb5640 1 -- 172.21.15.67:0/3107026575 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1740034440 con 0x7f17480cdb50 2024-03-24T00:41:40.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.288+0000 7f175011b640 1 -- 172.21.15.67:0/3107026575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17480cdb50 msgr2=0x7f174806b040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:40.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.288+0000 7f175011b640 1 --2- 172.21.15.67:0/3107026575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17480cdb50 0x7f174806b040 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f1740009d90 tx=0x7f174002be10 comp rx=0 tx=0).stop 2024-03-24T00:41:40.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.288+0000 7f175011b640 1 -- 172.21.15.67:0/3107026575 shutdown_connections 2024-03-24T00:41:40.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.288+0000 7f175011b640 1 --2- 172.21.15.67:0/3107026575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17480cdb50 0x7f174806b040 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:40.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.288+0000 7f175011b640 1 --2- 172.21.15.67:0/3107026575 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f17480cd230 0x7f17480d0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:40.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.288+0000 7f175011b640 1 -- 172.21.15.67:0/3107026575 >> 172.21.15.67:0/3107026575 conn(0x7f1748099a30 msgr2=0x7f1748099e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:40.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.288+0000 7f175011b640 1 -- 172.21.15.67:0/3107026575 shutdown_connections 2024-03-24T00:41:40.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.288+0000 7f175011b640 1 -- 172.21.15.67:0/3107026575 wait complete. 2024-03-24T00:41:40.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.290+0000 7f175011b640 1 Processor -- start 2024-03-24T00:41:40.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.290+0000 7f175011b640 1 -- start start 2024-03-24T00:41:40.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.290+0000 7f175011b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17480cd230 0x7f17481e78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:40.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.290+0000 7f175011b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17480cdb50 0x7f17480b2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:40.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.290+0000 7f175011b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f17480d00b0 0x7f1748149d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:40.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.290+0000 7f175011b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17480ce160 con 0x7f17480cd230 2024-03-24T00:41:40.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.290+0000 7f175011b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17480cdfe0 con 0x7f17480cdb50 2024-03-24T00:41:40.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.290+0000 7f175011b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f174806b580 con 0x7f17480d00b0 2024-03-24T00:41:40.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.290+0000 7f174deb7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17480cd230 0x7f17481e78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:40.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.290+0000 7f174deb7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17480cd230 0x7f17481e78f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43892/0 (socket says 172.21.15.67:43892) 2024-03-24T00:41:40.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.290+0000 7f174deb7640 1 -- 172.21.15.67:0/4208524705 learned_addr learned my addr 172.21.15.67:0/4208524705 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:40.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.291+0000 7f174d6b6640 1 --2- 172.21.15.67:0/4208524705 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17480cdb50 0x7f17480b2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:40.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.291+0000 7f174e6b8640 1 --2- 172.21.15.67:0/4208524705 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f17480d00b0 0x7f1748149d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:40.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.291+0000 7f174deb7640 1 -- 172.21.15.67:0/4208524705 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f17480d00b0 msgr2=0x7f1748149d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:40.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.291+0000 7f174deb7640 1 --2- 172.21.15.67:0/4208524705 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f17480d00b0 0x7f1748149d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:40.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.291+0000 7f174deb7640 1 -- 172.21.15.67:0/4208524705 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17480cdb50 msgr2=0x7f17480b2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:40.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.291+0000 7f174deb7640 1 --2- 172.21.15.67:0/4208524705 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17480cdb50 0x7f17480b2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:40.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.291+0000 7f174deb7640 1 -- 172.21.15.67:0/4208524705 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17480b1b60 con 0x7f17480cd230 2024-03-24T00:41:40.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.291+0000 7f174deb7640 1 --2- 172.21.15.67:0/4208524705 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17480cd230 0x7f17481e78f0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f174400c400 tx=0x7f174400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:40.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.291+0000 7f173e7fc640 1 -- 172.21.15.67:0/4208524705 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1744017070 con 0x7f17480cd230 2024-03-24T00:41:40.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.291+0000 7f173e7fc640 1 -- 172.21.15.67:0/4208524705 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1744005ce0 con 0x7f17480cd230 2024-03-24T00:41:40.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.291+0000 7f175011b640 1 -- 172.21.15.67:0/4208524705 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17480b1e40 con 0x7f17480cd230 2024-03-24T00:41:40.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.291+0000 7f173e7fc640 1 -- 172.21.15.67:0/4208524705 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17440072f0 con 0x7f17480cd230 2024-03-24T00:41:40.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.291+0000 7f175011b640 1 -- 172.21.15.67:0/4208524705 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17480b2400 con 0x7f17480cd230 2024-03-24T00:41:40.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.292+0000 7f175011b640 1 -- 172.21.15.67:0/4208524705 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f174804f400 con 0x7f17480cd230 2024-03-24T00:41:40.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.293+0000 7f173e7fc640 1 -- 172.21.15.67:0/4208524705 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f174401a070 con 0x7f17480cd230 2024-03-24T00:41:40.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.293+0000 7f173e7fc640 1 --2- 172.21.15.67:0/4208524705 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f172c06bbd0 0x7f172c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:40.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.293+0000 7f173e7fc640 1 -- 172.21.15.67:0/4208524705 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f1744089c30 con 0x7f17480cd230 2024-03-24T00:41:40.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.293+0000 7f174d6b6640 1 --2- 172.21.15.67:0/4208524705 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f172c06bbd0 0x7f172c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:40.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.294+0000 7f174d6b6640 1 --2- 172.21.15.67:0/4208524705 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f172c06bbd0 0x7f172c06e090 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f1748099ef0 tx=0x7f1738008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:40.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.297+0000 7f173e7fc640 1 -- 172.21.15.67:0/4208524705 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1744058290 con 0x7f17480cd230 2024-03-24T00:41:40.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.482+0000 7f175011b640 1 -- 172.21.15.67:0/4208524705 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f17480cd700 con 0x7f17480cd230 2024-03-24T00:41:40.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.859+0000 7f173e7fc640 1 -- 172.21.15.67:0/4208524705 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v59) v1 ==== 138+0+0 (secure 0 0 0) 0x7f174405c330 con 0x7f17480cd230 2024-03-24T00:41:40.860 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' created 2024-03-24T00:41:40.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.861+0000 7f175011b640 1 -- 172.21.15.67:0/4208524705 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f172c06bbd0 msgr2=0x7f172c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:40.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.861+0000 7f175011b640 1 --2- 172.21.15.67:0/4208524705 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f172c06bbd0 0x7f172c06e090 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f1748099ef0 tx=0x7f1738008040 comp rx=0 tx=0).stop 2024-03-24T00:41:40.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.861+0000 7f175011b640 1 -- 172.21.15.67:0/4208524705 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17480cd230 msgr2=0x7f17481e78f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:40.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.861+0000 7f175011b640 1 --2- 172.21.15.67:0/4208524705 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17480cd230 0x7f17481e78f0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f174400c400 tx=0x7f174400c8c0 comp rx=0 tx=0).stop 2024-03-24T00:41:40.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.861+0000 7f175011b640 1 -- 172.21.15.67:0/4208524705 shutdown_connections 2024-03-24T00:41:40.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.861+0000 7f175011b640 1 --2- 172.21.15.67:0/4208524705 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f172c06bbd0 0x7f172c06e090 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:40.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.861+0000 7f175011b640 1 --2- 172.21.15.67:0/4208524705 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f17480d00b0 0x7f1748149d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:40.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.861+0000 7f175011b640 1 --2- 172.21.15.67:0/4208524705 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17480cdb50 0x7f17480b2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:40.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.861+0000 7f175011b640 1 --2- 172.21.15.67:0/4208524705 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17480cd230 0x7f17481e78f0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:40.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.861+0000 7f175011b640 1 -- 172.21.15.67:0/4208524705 >> 172.21.15.67:0/4208524705 conn(0x7f1748099a30 msgr2=0x7f1748073cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:40.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.861+0000 7f175011b640 1 -- 172.21.15.67:0/4208524705 shutdown_connections 2024-03-24T00:41:40.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.861+0000 7f175011b640 1 -- 172.21.15.67:0/4208524705 wait complete. 2024-03-24T00:41:40.876 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-24T00:41:40.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.985+0000 7efc83559640 1 Processor -- start 2024-03-24T00:41:40.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.985+0000 7efc83559640 1 -- start start 2024-03-24T00:41:40.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.986+0000 7efc83559640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efc7c15df60 0x7efc7c0d0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:40.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.986+0000 7efc83559640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efc7c15e340 0x7efc7c0d0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:40.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.986+0000 7efc83559640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc7c162880 con 0x7efc7c161c30 2024-03-24T00:41:40.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.986+0000 7efc83559640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc7c0d25f0 con 0x7efc7c15e340 2024-03-24T00:41:40.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.986+0000 7efc83559640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c162b60 con 0x7efc7c15df60 2024-03-24T00:41:40.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.986+0000 7efc80af4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efc7c15e340 0x7efc7c0d0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:40.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.986+0000 7efc80af4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efc7c15e340 0x7efc7c0d0bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43902/0 (socket says 172.21.15.67:43902) 2024-03-24T00:41:40.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.986+0000 7efc80af4640 1 -- 172.21.15.67:0/2803663752 learned_addr learned my addr 172.21.15.67:0/2803663752 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:40.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.987+0000 7efc812f5640 1 --2- 172.21.15.67:0/2803663752 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efc7c15df60 0x7efc7c0d0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:40.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.987+0000 7efc80af4640 1 -- 172.21.15.67:0/2803663752 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efc7c15df60 msgr2=0x7efc7c0d0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:40.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.987+0000 7efc80af4640 1 --2- 172.21.15.67:0/2803663752 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efc7c15df60 0x7efc7c0d0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:40.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.987+0000 7efc80af4640 1 -- 172.21.15.67:0/2803663752 >> v1:172.21.15.72:6789/0 conn(0x7efc7c161c30 legacy=0x7efc7c1620c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:40.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.987+0000 7efc80af4640 1 -- 172.21.15.67:0/2803663752 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc7c0d1fa0 con 0x7efc7c15e340 2024-03-24T00:41:40.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.987+0000 7efc80af4640 1 --2- 172.21.15.67:0/2803663752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc7c15e340 0x7efc7c0d0bb0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7efc70009870 tx=0x7efc7002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d015efaf600348c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:40.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.988+0000 7efc7bfff640 1 -- 172.21.15.67:0/2803663752 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc70035020 con 0x7efc7c15e340 2024-03-24T00:41:40.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.988+0000 7efc7bfff640 1 -- 172.21.15.67:0/2803663752 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc70031640 con 0x7efc7c15e340 2024-03-24T00:41:40.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.988+0000 7efc7bfff640 1 -- 172.21.15.67:0/2803663752 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc7002d040 con 0x7efc7c15e340 2024-03-24T00:41:40.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.988+0000 7efc83559640 1 -- 172.21.15.67:0/2803663752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc7c15e340 msgr2=0x7efc7c0d0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:40.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.988+0000 7efc83559640 1 --2- 172.21.15.67:0/2803663752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc7c15e340 0x7efc7c0d0bb0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7efc70009870 tx=0x7efc7002b8f0 comp rx=0 tx=0).stop 2024-03-24T00:41:40.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.988+0000 7efc83559640 1 -- 172.21.15.67:0/2803663752 shutdown_connections 2024-03-24T00:41:40.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.988+0000 7efc83559640 1 --2- 172.21.15.67:0/2803663752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc7c15e340 0x7efc7c0d0bb0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:40.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.988+0000 7efc83559640 1 --2- 172.21.15.67:0/2803663752 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efc7c15df60 0x7efc7c0d0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:40.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.988+0000 7efc83559640 1 -- 172.21.15.67:0/2803663752 >> 172.21.15.67:0/2803663752 conn(0x7efc7c06c860 msgr2=0x7efc7c06cc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:40.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.988+0000 7efc83559640 1 -- 172.21.15.67:0/2803663752 shutdown_connections 2024-03-24T00:41:40.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.988+0000 7efc83559640 1 -- 172.21.15.67:0/2803663752 wait complete. 2024-03-24T00:41:40.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.989+0000 7efc83559640 1 Processor -- start 2024-03-24T00:41:40.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.989+0000 7efc83559640 1 -- start start 2024-03-24T00:41:40.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.990+0000 7efc83559640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc7c15df60 0x7efc7c154b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:40.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.990+0000 7efc83559640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc7c161c30 0x7efc7c157920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:40.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.990+0000 7efc83559640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efc7c1550c0 0x7efc7c157e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:40.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.990+0000 7efc83559640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c0b7100 con 0x7efc7c161c30 2024-03-24T00:41:40.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.990+0000 7efc83559640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c0b6f80 con 0x7efc7c15df60 2024-03-24T00:41:40.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.990+0000 7efc83559640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c0d2a80 con 0x7efc7c1550c0 2024-03-24T00:41:40.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.990+0000 7efc80af4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc7c161c30 0x7efc7c157920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:40.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.990+0000 7efc80af4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc7c161c30 0x7efc7c157920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43908/0 (socket says 172.21.15.67:43908) 2024-03-24T00:41:40.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.990+0000 7efc80af4640 1 -- 172.21.15.67:0/1265365813 learned_addr learned my addr 172.21.15.67:0/1265365813 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:40.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.990+0000 7efc81af6640 1 --2- 172.21.15.67:0/1265365813 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efc7c1550c0 0x7efc7c157e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:40.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.990+0000 7efc812f5640 1 --2- 172.21.15.67:0/1265365813 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc7c15df60 0x7efc7c154b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:40.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.990+0000 7efc80af4640 1 -- 172.21.15.67:0/1265365813 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efc7c1550c0 msgr2=0x7efc7c157e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:40.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.990+0000 7efc80af4640 1 --2- 172.21.15.67:0/1265365813 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efc7c1550c0 0x7efc7c157e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:40.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.990+0000 7efc80af4640 1 -- 172.21.15.67:0/1265365813 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc7c15df60 msgr2=0x7efc7c154b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:40.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.990+0000 7efc80af4640 1 --2- 172.21.15.67:0/1265365813 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc7c15df60 0x7efc7c154b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:40.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.990+0000 7efc80af4640 1 -- 172.21.15.67:0/1265365813 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc7c158490 con 0x7efc7c161c30 2024-03-24T00:41:40.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.991+0000 7efc80af4640 1 --2- 172.21.15.67:0/1265365813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc7c161c30 0x7efc7c157920 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7efc7002b930 tx=0x7efc700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:40.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.991+0000 7efc79ffb640 1 -- 172.21.15.67:0/1265365813 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc70040070 con 0x7efc7c161c30 2024-03-24T00:41:40.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.991+0000 7efc79ffb640 1 -- 172.21.15.67:0/1265365813 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc70039bf0 con 0x7efc7c161c30 2024-03-24T00:41:40.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.991+0000 7efc83559640 1 -- 172.21.15.67:0/1265365813 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc7c152f50 con 0x7efc7c161c30 2024-03-24T00:41:40.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.991+0000 7efc79ffb640 1 -- 172.21.15.67:0/1265365813 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc70035040 con 0x7efc7c161c30 2024-03-24T00:41:40.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.991+0000 7efc83559640 1 -- 172.21.15.67:0/1265365813 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc7c158c50 con 0x7efc7c161c30 2024-03-24T00:41:40.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.992+0000 7efc83559640 1 -- 172.21.15.67:0/1265365813 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc7c04f400 con 0x7efc7c161c30 2024-03-24T00:41:40.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.992+0000 7efc79ffb640 1 -- 172.21.15.67:0/1265365813 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7efc7004b060 con 0x7efc7c161c30 2024-03-24T00:41:40.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.993+0000 7efc79ffb640 1 --2- 172.21.15.67:0/1265365813 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7efc6806bc00 0x7efc6806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:40.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.993+0000 7efc79ffb640 1 -- 172.21.15.67:0/1265365813 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5619+0+0 (secure 0 0 0) 0x7efc700ac5a0 con 0x7efc7c161c30 2024-03-24T00:41:40.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.993+0000 7efc812f5640 1 --2- 172.21.15.67:0/1265365813 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7efc6806bc00 0x7efc6806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:40.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.994+0000 7efc812f5640 1 --2- 172.21.15.67:0/1265365813 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7efc6806bc00 0x7efc6806e0c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7efc6c008ef0 tx=0x7efc6c008440 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:40.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:40.998+0000 7efc79ffb640 1 -- 172.21.15.67:0/1265365813 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7efc7002f300 con 0x7efc7c161c30 2024-03-24T00:41:41.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:41.180+0000 7efc83559640 1 -- 172.21.15.67:0/1265365813 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7efc7c153750 con 0x7efc7c161c30 2024-03-24T00:41:41.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:41.866+0000 7efc79ffb640 1 -- 172.21.15.67:0/1265365813 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v60) v1 ==== 170+0+0 (secure 0 0 0) 0x7efc7007b0e0 con 0x7efc7c161c30 2024-03-24T00:41:41.867 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' created 2024-03-24T00:41:41.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:41.868+0000 7efc83559640 1 -- 172.21.15.67:0/1265365813 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7efc6806bc00 msgr2=0x7efc6806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:41.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:41.868+0000 7efc83559640 1 --2- 172.21.15.67:0/1265365813 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7efc6806bc00 0x7efc6806e0c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7efc6c008ef0 tx=0x7efc6c008440 comp rx=0 tx=0).stop 2024-03-24T00:41:41.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:41.869+0000 7efc83559640 1 -- 172.21.15.67:0/1265365813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc7c161c30 msgr2=0x7efc7c157920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:41.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:41.869+0000 7efc83559640 1 --2- 172.21.15.67:0/1265365813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc7c161c30 0x7efc7c157920 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7efc7002b930 tx=0x7efc700046c0 comp rx=0 tx=0).stop 2024-03-24T00:41:41.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:41.870+0000 7efc83559640 1 -- 172.21.15.67:0/1265365813 shutdown_connections 2024-03-24T00:41:41.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:41.870+0000 7efc83559640 1 --2- 172.21.15.67:0/1265365813 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7efc6806bc00 0x7efc6806e0c0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:41.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:41.870+0000 7efc83559640 1 --2- 172.21.15.67:0/1265365813 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efc7c1550c0 0x7efc7c157e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:41.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:41.870+0000 7efc83559640 1 --2- 172.21.15.67:0/1265365813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc7c161c30 0x7efc7c157920 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:41.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:41.870+0000 7efc83559640 1 --2- 172.21.15.67:0/1265365813 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc7c15df60 0x7efc7c154b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:41.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:41.870+0000 7efc83559640 1 -- 172.21.15.67:0/1265365813 >> 172.21.15.67:0/1265365813 conn(0x7efc7c06c860 msgr2=0x7efc7c15e850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:41.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:41.871+0000 7efc83559640 1 -- 172.21.15.67:0/1265365813 shutdown_connections 2024-03-24T00:41:41.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:41.871+0000 7efc83559640 1 -- 172.21.15.67:0/1265365813 wait complete. 2024-03-24T00:41:41.897 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-24T00:41:42.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.006+0000 7fd2b8b60640 1 Processor -- start 2024-03-24T00:41:42.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.006+0000 7fd2b8b60640 1 -- start start 2024-03-24T00:41:42.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.007+0000 7fd2b8b60640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd2b40b4a00 0x7fd2b40b3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:42.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.007+0000 7fd2b8b60640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd2b406ec60 0x7fd2b406f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:42.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.007+0000 7fd2b8b60640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2b40b2ab0 con 0x7fd2b4144670 2024-03-24T00:41:42.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.007+0000 7fd2b8b60640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2b40b2d90 con 0x7fd2b406ec60 2024-03-24T00:41:42.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.007+0000 7fd2b8b60640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b40b3070 con 0x7fd2b40b4a00 2024-03-24T00:41:42.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.007+0000 7fd2b2d9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd2b406ec60 0x7fd2b406f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:42.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.007+0000 7fd2b1d9b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd2b40b4a00 0x7fd2b40b3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:42.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.007+0000 7fd2b259c640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fd2b4144670 0x7fd2b4072500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:49672/0 (socket says 172.21.15.67:49672) 2024-03-24T00:41:42.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.007+0000 7fd2b2d9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd2b406ec60 0x7fd2b406f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:54122/0 (socket says 172.21.15.67:54122) 2024-03-24T00:41:42.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.007+0000 7fd2b1d9b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd2b40b4a00 0x7fd2b40b3870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.67:47584/0 (socket says 172.21.15.67:47584) 2024-03-24T00:41:42.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.007+0000 7fd2b259c640 1 -- 172.21.15.67:0/158035116 learned_addr learned my addr 172.21.15.67:0/158035116 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:41:42.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.008+0000 7fd2b159a640 1 -- 172.21.15.67:0/158035116 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3912882891 0 0) 0x7fd2b40b2ab0 con 0x7fd2b4144670 2024-03-24T00:41:42.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.008+0000 7fd2b159a640 1 -- 172.21.15.67:0/158035116 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2a0005730 con 0x7fd2b4144670 2024-03-24T00:41:42.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.008+0000 7fd2b2d9d640 1 -- 172.21.15.67:0/158035116 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd2b40b4a00 msgr2=0x7fd2b40b3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:42.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.008+0000 7fd2b2d9d640 1 --2- 172.21.15.67:0/158035116 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd2b40b4a00 0x7fd2b40b3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:42.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.008+0000 7fd2b2d9d640 1 -- 172.21.15.67:0/158035116 >> v1:172.21.15.72:6789/0 conn(0x7fd2b4144670 legacy=0x7fd2b4072500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:42.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.008+0000 7fd2b2d9d640 1 -- 172.21.15.67:0/158035116 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b40b3350 con 0x7fd2b406ec60 2024-03-24T00:41:42.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.008+0000 7fd2b1d9b640 1 --2- 172.21.15.67:0/158035116 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd2b40b4a00 0x7fd2b40b3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:41:42.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.009+0000 7fd2b2d9d640 1 --2- 172.21.15.67:0/158035116 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd2b406ec60 0x7fd2b406f0f0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac009d90 tx=0x7fd2ac02bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49aca780a5a3510 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:42.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.009+0000 7fd2b159a640 1 -- 172.21.15.67:0/158035116 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2ac039070 con 0x7fd2b406ec60 2024-03-24T00:41:42.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.009+0000 7fd2b159a640 1 -- 172.21.15.67:0/158035116 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2ac02cc50 con 0x7fd2b406ec60 2024-03-24T00:41:42.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.009+0000 7fd2b159a640 1 -- 172.21.15.67:0/158035116 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2ac034440 con 0x7fd2b406ec60 2024-03-24T00:41:42.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.009+0000 7fd2b8b60640 1 -- 172.21.15.67:0/158035116 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd2b406ec60 msgr2=0x7fd2b406f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:42.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.009+0000 7fd2b8b60640 1 --2- 172.21.15.67:0/158035116 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd2b406ec60 0x7fd2b406f0f0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac009d90 tx=0x7fd2ac02bea0 comp rx=0 tx=0).stop 2024-03-24T00:41:42.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.009+0000 7fd2b8b60640 1 -- 172.21.15.67:0/158035116 shutdown_connections 2024-03-24T00:41:42.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.009+0000 7fd2b8b60640 1 --2- 172.21.15.67:0/158035116 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd2b406ec60 0x7fd2b406f0f0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:42.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.009+0000 7fd2b8b60640 1 --2- 172.21.15.67:0/158035116 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd2b40b4a00 0x7fd2b40b3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:42.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.009+0000 7fd2b8b60640 1 -- 172.21.15.67:0/158035116 >> 172.21.15.67:0/158035116 conn(0x7fd2b40cf9e0 msgr2=0x7fd2b40cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:42.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.009+0000 7fd2b8b60640 1 -- 172.21.15.67:0/158035116 shutdown_connections 2024-03-24T00:41:42.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.009+0000 7fd2b8b60640 1 -- 172.21.15.67:0/158035116 wait complete. 2024-03-24T00:41:42.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.011+0000 7fd2b8b60640 1 Processor -- start 2024-03-24T00:41:42.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.011+0000 7fd2b8b60640 1 -- start start 2024-03-24T00:41:42.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.011+0000 7fd2b8b60640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd2b406ec60 0x7fd2b412eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:42.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.011+0000 7fd2b8b60640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2b40b4a00 0x7fd2b40aed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:42.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.011+0000 7fd2b8b60640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd2b4144670 0x7fd2b40ad0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:42.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.011+0000 7fd2b8b60640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b40cdfb0 con 0x7fd2b40b4a00 2024-03-24T00:41:42.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.011+0000 7fd2b8b60640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b40cde30 con 0x7fd2b406ec60 2024-03-24T00:41:42.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.011+0000 7fd2b8b60640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b40ce130 con 0x7fd2b4144670 2024-03-24T00:41:42.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.012+0000 7fd2b1d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2b40b4a00 0x7fd2b40aed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:42.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.012+0000 7fd2b1d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2b40b4a00 0x7fd2b40aed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43922/0 (socket says 172.21.15.67:43922) 2024-03-24T00:41:42.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.012+0000 7fd2b1d9b640 1 -- 172.21.15.67:0/2501463988 learned_addr learned my addr 172.21.15.67:0/2501463988 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:42.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.012+0000 7fd2b2d9d640 1 --2- 172.21.15.67:0/2501463988 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd2b4144670 0x7fd2b40ad0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:42.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.012+0000 7fd2b259c640 1 --2- 172.21.15.67:0/2501463988 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd2b406ec60 0x7fd2b412eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:42.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.012+0000 7fd2b1d9b640 1 -- 172.21.15.67:0/2501463988 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd2b4144670 msgr2=0x7fd2b40ad0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:42.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.012+0000 7fd2b1d9b640 1 --2- 172.21.15.67:0/2501463988 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd2b4144670 0x7fd2b40ad0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:42.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.012+0000 7fd2b1d9b640 1 -- 172.21.15.67:0/2501463988 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd2b406ec60 msgr2=0x7fd2b412eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:42.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.012+0000 7fd2b1d9b640 1 --2- 172.21.15.67:0/2501463988 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd2b406ec60 0x7fd2b412eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:42.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.012+0000 7fd2b1d9b640 1 -- 172.21.15.67:0/2501463988 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b40af290 con 0x7fd2b40b4a00 2024-03-24T00:41:42.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.012+0000 7fd2b1d9b640 1 --2- 172.21.15.67:0/2501463988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2b40b4a00 0x7fd2b40aed50 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fd29c00ec10 tx=0x7fd29c00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:42.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.012+0000 7fd2aaffd640 1 -- 172.21.15.67:0/2501463988 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29c017070 con 0x7fd2b40b4a00 2024-03-24T00:41:42.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.012+0000 7fd2aaffd640 1 -- 172.21.15.67:0/2501463988 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd29c00cbd0 con 0x7fd2b40b4a00 2024-03-24T00:41:42.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.012+0000 7fd2aaffd640 1 -- 172.21.15.67:0/2501463988 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29c013470 con 0x7fd2b40b4a00 2024-03-24T00:41:42.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.012+0000 7fd2b8b60640 1 -- 172.21.15.67:0/2501463988 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2b41f3060 con 0x7fd2b40b4a00 2024-03-24T00:41:42.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.012+0000 7fd2b8b60640 1 -- 172.21.15.67:0/2501463988 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2b40a9580 con 0x7fd2b40b4a00 2024-03-24T00:41:42.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.013+0000 7fd2b8b60640 1 -- 172.21.15.67:0/2501463988 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2b40cf350 con 0x7fd2b40b4a00 2024-03-24T00:41:42.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.014+0000 7fd2aaffd640 1 -- 172.21.15.67:0/2501463988 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd29c01a070 con 0x7fd2b40b4a00 2024-03-24T00:41:42.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.014+0000 7fd2aaffd640 1 --2- 172.21.15.67:0/2501463988 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd29406bbd0 0x7fd29406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:42.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.014+0000 7fd2aaffd640 1 -- 172.21.15.67:0/2501463988 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6001+0+0 (secure 0 0 0) 0x7fd29c08aa30 con 0x7fd2b40b4a00 2024-03-24T00:41:42.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.014+0000 7fd2b259c640 1 --2- 172.21.15.67:0/2501463988 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd29406bbd0 0x7fd29406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:42.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.015+0000 7fd2b259c640 1 --2- 172.21.15.67:0/2501463988 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd29406bbd0 0x7fd29406e090 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fd2a4005880 tx=0x7fd2a4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:42.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.019+0000 7fd2aaffd640 1 -- 172.21.15.67:0/2501463988 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd2b40cf350 con 0x7fd2b40b4a00 2024-03-24T00:41:42.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.240+0000 7fd2b8b60640 1 -- 172.21.15.67:0/2501463988 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fd2b404f400 con 0x7fd2b40b4a00 2024-03-24T00:41:42.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.867+0000 7fd2aaffd640 1 -- 172.21.15.67:0/2501463988 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 9 and data pool 10 v30) v1 ==== 173+0+0 (secure 0 0 0) 0x7fd29c058da0 con 0x7fd2b40b4a00 2024-03-24T00:41:42.868 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 9 and data pool 10 2024-03-24T00:41:42.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.871+0000 7fd2b8b60640 1 -- 172.21.15.67:0/2501463988 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd29406bbd0 msgr2=0x7fd29406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:42.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.871+0000 7fd2b8b60640 1 --2- 172.21.15.67:0/2501463988 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd29406bbd0 0x7fd29406e090 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fd2a4005880 tx=0x7fd2a4009040 comp rx=0 tx=0).stop 2024-03-24T00:41:42.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.871+0000 7fd2b8b60640 1 -- 172.21.15.67:0/2501463988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2b40b4a00 msgr2=0x7fd2b40aed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:42.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.871+0000 7fd2b8b60640 1 --2- 172.21.15.67:0/2501463988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2b40b4a00 0x7fd2b40aed50 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fd29c00ec10 tx=0x7fd29c00c520 comp rx=0 tx=0).stop 2024-03-24T00:41:42.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.872+0000 7fd2b8b60640 1 -- 172.21.15.67:0/2501463988 shutdown_connections 2024-03-24T00:41:42.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.872+0000 7fd2b8b60640 1 --2- 172.21.15.67:0/2501463988 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd29406bbd0 0x7fd29406e090 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:42.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.872+0000 7fd2b8b60640 1 --2- 172.21.15.67:0/2501463988 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd2b4144670 0x7fd2b40ad0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:42.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.872+0000 7fd2b8b60640 1 --2- 172.21.15.67:0/2501463988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2b40b4a00 0x7fd2b40aed50 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:42.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.872+0000 7fd2b8b60640 1 --2- 172.21.15.67:0/2501463988 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd2b406ec60 0x7fd2b412eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:42.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.872+0000 7fd2b8b60640 1 -- 172.21.15.67:0/2501463988 >> 172.21.15.67:0/2501463988 conn(0x7fd2b40cf9e0 msgr2=0x7fd2b40cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:42.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.872+0000 7fd2b8b60640 1 -- 172.21.15.67:0/2501463988 shutdown_connections 2024-03-24T00:41:42.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:42.873+0000 7fd2b8b60640 1 -- 172.21.15.67:0/2501463988 wait complete. 2024-03-24T00:41:42.901 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:41:43.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.005+0000 7f59d6109640 1 Processor -- start 2024-03-24T00:41:43.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.006+0000 7f59d6109640 1 -- start start 2024-03-24T00:41:43.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.006+0000 7f59d6109640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f59d00cfc70 0x7f59d00ce220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:43.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.006+0000 7f59d6109640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f59d00ce760 0x7f59d00b1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:43.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.006+0000 7f59d6109640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59d00ccfe0 con 0x7f59d015bf70 2024-03-24T00:41:43.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.006+0000 7f59d6109640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59d00ccda0 con 0x7f59d00cfc70 2024-03-24T00:41:43.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.006+0000 7f59d6109640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59d00cd2c0 con 0x7f59d00ce760 2024-03-24T00:41:43.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.006+0000 7f59cf7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f59d015bf70 0x7f59d00cc690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58406/0 (socket says 172.21.15.67:58406) 2024-03-24T00:41:43.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.006+0000 7f59cf7fe640 1 -- 172.21.15.67:0/1778689523 learned_addr learned my addr 172.21.15.67:0/1778689523 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:41:43.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.007+0000 7f59ceffd640 1 --2- 172.21.15.67:0/1778689523 >> v2:172.21.15.72:3300/0 conn(0x7f59d00cfc70 0x7f59d00ce220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:43.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.007+0000 7f59cffff640 1 --2- 172.21.15.67:0/1778689523 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f59d00ce760 0x7f59d00b1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:43.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.007+0000 7f59ce7fc640 1 -- 172.21.15.67:0/1778689523 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3075199649 0 0) 0x7f59d00ccfe0 con 0x7f59d015bf70 2024-03-24T00:41:43.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.007+0000 7f59ce7fc640 1 -- 172.21.15.67:0/1778689523 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59b8003520 con 0x7f59d015bf70 2024-03-24T00:41:43.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.007+0000 7f59ceffd640 1 -- 172.21.15.67:0/1778689523 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f59d00ce760 msgr2=0x7f59d00b1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:43.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.007+0000 7f59ceffd640 1 --2- 172.21.15.67:0/1778689523 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f59d00ce760 0x7f59d00b1cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:43.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.007+0000 7f59ceffd640 1 -- 172.21.15.67:0/1778689523 >> v1:172.21.15.67:6789/0 conn(0x7f59d015bf70 legacy=0x7f59d00cc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:43.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.007+0000 7f59ceffd640 1 -- 172.21.15.67:0/1778689523 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d00cd5a0 con 0x7f59d00cfc70 2024-03-24T00:41:43.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.007+0000 7f59cffff640 1 --2- 172.21.15.67:0/1778689523 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f59d00ce760 0x7f59d00b1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:41:43.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.008+0000 7f59ceffd640 1 --2- 172.21.15.67:0/1778689523 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59d00cfc70 0x7f59d00ce220 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f59c40097a0 tx=0x7f59c402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a4db2f99d0c695f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:43.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.008+0000 7f59ce7fc640 1 -- 172.21.15.67:0/1778689523 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c4035020 con 0x7f59d00cfc70 2024-03-24T00:41:43.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.008+0000 7f59ce7fc640 1 -- 172.21.15.67:0/1778689523 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59c4031830 con 0x7f59d00cfc70 2024-03-24T00:41:43.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.008+0000 7f59ce7fc640 1 -- 172.21.15.67:0/1778689523 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c4031be0 con 0x7f59d00cfc70 2024-03-24T00:41:43.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.008+0000 7f59d6109640 1 -- 172.21.15.67:0/1778689523 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59d00cfc70 msgr2=0x7f59d00ce220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:43.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.008+0000 7f59d6109640 1 --2- 172.21.15.67:0/1778689523 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59d00cfc70 0x7f59d00ce220 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f59c40097a0 tx=0x7f59c402b8d0 comp rx=0 tx=0).stop 2024-03-24T00:41:43.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.008+0000 7f59d6109640 1 -- 172.21.15.67:0/1778689523 shutdown_connections 2024-03-24T00:41:43.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.008+0000 7f59d6109640 1 --2- 172.21.15.67:0/1778689523 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f59d00ce760 0x7f59d00b1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:43.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.008+0000 7f59d6109640 1 --2- 172.21.15.67:0/1778689523 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59d00cfc70 0x7f59d00ce220 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:43.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.008+0000 7f59d6109640 1 -- 172.21.15.67:0/1778689523 >> 172.21.15.67:0/1778689523 conn(0x7f59d00d1bf0 msgr2=0x7f59d00d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:43.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.008+0000 7f59d6109640 1 -- 172.21.15.67:0/1778689523 shutdown_connections 2024-03-24T00:41:43.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.008+0000 7f59d6109640 1 -- 172.21.15.67:0/1778689523 wait complete. 2024-03-24T00:41:43.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.010+0000 7f59d6109640 1 Processor -- start 2024-03-24T00:41:43.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.010+0000 7f59d6109640 1 -- start start 2024-03-24T00:41:43.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.010+0000 7f59d6109640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f59d00ce760 0x7f59d0154340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:43.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.010+0000 7f59d6109640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59d00cfc70 0x7f59d00b6980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:43.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.010+0000 7f59d6109640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59d015bf70 0x7f59d00b6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:43.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.010+0000 7f59d6109640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d00b2210 con 0x7f59d015bf70 2024-03-24T00:41:43.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.010+0000 7f59d6109640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d015c350 con 0x7f59d00cfc70 2024-03-24T00:41:43.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.010+0000 7f59d6109640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59d00b2390 con 0x7f59d00ce760 2024-03-24T00:41:43.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.011+0000 7f59cffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59d015bf70 0x7f59d00b6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:43.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.011+0000 7f59cffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59d015bf70 0x7f59d00b6ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43932/0 (socket says 172.21.15.67:43932) 2024-03-24T00:41:43.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.011+0000 7f59cffff640 1 -- 172.21.15.67:0/1813612221 learned_addr learned my addr 172.21.15.67:0/1813612221 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:43.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.011+0000 7f59ceffd640 1 --2- 172.21.15.67:0/1813612221 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59d00cfc70 0x7f59d00b6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:43.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.011+0000 7f59cf7fe640 1 --2- 172.21.15.67:0/1813612221 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f59d00ce760 0x7f59d0154340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:43.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.011+0000 7f59cffff640 1 -- 172.21.15.67:0/1813612221 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f59d00ce760 msgr2=0x7f59d0154340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:43.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.011+0000 7f59cffff640 1 --2- 172.21.15.67:0/1813612221 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f59d00ce760 0x7f59d0154340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:43.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.011+0000 7f59cffff640 1 -- 172.21.15.67:0/1813612221 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59d00cfc70 msgr2=0x7f59d00b6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:43.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.011+0000 7f59cffff640 1 --2- 172.21.15.67:0/1813612221 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59d00cfc70 0x7f59d00b6980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:43.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.011+0000 7f59cffff640 1 -- 172.21.15.67:0/1813612221 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d00cac80 con 0x7f59d015bf70 2024-03-24T00:41:43.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.011+0000 7f59cffff640 1 --2- 172.21.15.67:0/1813612221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59d015bf70 0x7f59d00b6ec0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f59c800efc0 tx=0x7f59c8009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:43.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.011+0000 7f59b7fff640 1 -- 172.21.15.67:0/1813612221 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c8018030 con 0x7f59d015bf70 2024-03-24T00:41:43.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.011+0000 7f59b7fff640 1 -- 172.21.15.67:0/1813612221 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59c800cde0 con 0x7f59d015bf70 2024-03-24T00:41:43.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.011+0000 7f59d6109640 1 -- 172.21.15.67:0/1813612221 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59d00caf20 con 0x7f59d015bf70 2024-03-24T00:41:43.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.011+0000 7f59b7fff640 1 -- 172.21.15.67:0/1813612221 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c8003bb0 con 0x7f59d015bf70 2024-03-24T00:41:43.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.011+0000 7f59d6109640 1 -- 172.21.15.67:0/1813612221 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59d00cb4e0 con 0x7f59d015bf70 2024-03-24T00:41:43.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.012+0000 7f59d6109640 1 -- 172.21.15.67:0/1813612221 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59d0071600 con 0x7f59d015bf70 2024-03-24T00:41:43.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.013+0000 7f59b7fff640 1 -- 172.21.15.67:0/1813612221 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f59c8005d10 con 0x7f59d015bf70 2024-03-24T00:41:43.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.013+0000 7f59b7fff640 1 --2- 172.21.15.67:0/1813612221 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f59bc06bbd0 0x7f59bc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:43.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.013+0000 7f59b7fff640 1 -- 172.21.15.67:0/1813612221 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f59c808b4f0 con 0x7f59d015bf70 2024-03-24T00:41:43.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.013+0000 7f59cf7fe640 1 --2- 172.21.15.67:0/1813612221 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f59bc06bbd0 0x7f59bc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:43.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.014+0000 7f59cf7fe640 1 --2- 172.21.15.67:0/1813612221 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f59bc06bbd0 0x7f59bc06e090 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f59c0005880 tx=0x7f59c0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:43.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.018+0000 7f59b7fff640 1 -- 172.21.15.67:0/1813612221 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f59d0071600 con 0x7f59d015bf70 2024-03-24T00:41:43.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.201+0000 7f59d6109640 1 -- 172.21.15.67:0/1813612221 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f59d00b2f80 con 0x7f59d015bf70 2024-03-24T00:41:43.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.202+0000 7f59b7fff640 1 -- 172.21.15.67:0/1813612221 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v61) v1 ==== 74+0+16837 (secure 0 0 0) 0x7f59c8022070 con 0x7f59d015bf70 2024-03-24T00:41:43.203 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:41:43.203 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":61,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:41:42.241896+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:41:40.502194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-03-24T00:41:41.200073+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:41:43.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.204+0000 7f59d6109640 1 -- 172.21.15.67:0/1813612221 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f59bc06bbd0 msgr2=0x7f59bc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:43.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.204+0000 7f59d6109640 1 --2- 172.21.15.67:0/1813612221 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f59bc06bbd0 0x7f59bc06e090 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f59c0005880 tx=0x7f59c0009040 comp rx=0 tx=0).stop 2024-03-24T00:41:43.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.204+0000 7f59d6109640 1 -- 172.21.15.67:0/1813612221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59d015bf70 msgr2=0x7f59d00b6ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:43.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.204+0000 7f59d6109640 1 --2- 172.21.15.67:0/1813612221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59d015bf70 0x7f59d00b6ec0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f59c800efc0 tx=0x7f59c8009f90 comp rx=0 tx=0).stop 2024-03-24T00:41:43.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.204+0000 7f59d6109640 1 -- 172.21.15.67:0/1813612221 shutdown_connections 2024-03-24T00:41:43.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.204+0000 7f59d6109640 1 --2- 172.21.15.67:0/1813612221 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f59bc06bbd0 0x7f59bc06e090 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:43.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.204+0000 7f59d6109640 1 --2- 172.21.15.67:0/1813612221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59d015bf70 0x7f59d00b6ec0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:43.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.204+0000 7f59d6109640 1 --2- 172.21.15.67:0/1813612221 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59d00cfc70 0x7f59d00b6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:43.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.204+0000 7f59d6109640 1 --2- 172.21.15.67:0/1813612221 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f59d00ce760 0x7f59d0154340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:43.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.204+0000 7f59d6109640 1 -- 172.21.15.67:0/1813612221 >> 172.21.15.67:0/1813612221 conn(0x7f59d00d1bf0 msgr2=0x7f59d0070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:43.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.204+0000 7f59d6109640 1 -- 172.21.15.67:0/1813612221 shutdown_connections 2024-03-24T00:41:43.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.204+0000 7f59d6109640 1 -- 172.21.15.67:0/1813612221 wait complete. 2024-03-24T00:41:43.216 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:41:43.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.316+0000 7f037b5ab640 1 Processor -- start 2024-03-24T00:41:43.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.316+0000 7f037b5ab640 1 -- start start 2024-03-24T00:41:43.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.317+0000 7f037b5ab640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f037415dec0 0x7f03740ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:43.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.317+0000 7f037b5ab640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f037415e2a0 0x7f0374162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:43.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.317+0000 7f037b5ab640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03740cf640 con 0x7f03740d07e0 2024-03-24T00:41:43.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.317+0000 7f037b5ab640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03740cee30 con 0x7f037415dec0 2024-03-24T00:41:43.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.317+0000 7f037b5ab640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03740d2120 con 0x7f037415e2a0 2024-03-24T00:41:43.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.317+0000 7f0379347640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f037415dec0 0x7f03740ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:43.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.317+0000 7f0379347640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f037415dec0 0x7f03740ce630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43946/0 (socket says 172.21.15.67:43946) 2024-03-24T00:41:43.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.317+0000 7f0379347640 1 -- 172.21.15.67:0/2940831603 learned_addr learned my addr 172.21.15.67:0/2940831603 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:43.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.317+0000 7f0378b46640 1 --2- 172.21.15.67:0/2940831603 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f037415e2a0 0x7f0374162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:43.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.318+0000 7f0379347640 1 -- 172.21.15.67:0/2940831603 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f037415e2a0 msgr2=0x7f0374162630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:43.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.318+0000 7f0379347640 1 --2- 172.21.15.67:0/2940831603 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f037415e2a0 0x7f0374162630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:43.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.318+0000 7f0379347640 1 -- 172.21.15.67:0/2940831603 >> v1:172.21.15.72:6789/0 conn(0x7f03740d07e0 legacy=0x7f03740b5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:43.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.318+0000 7f0379347640 1 -- 172.21.15.67:0/2940831603 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03740cf080 con 0x7f037415dec0 2024-03-24T00:41:43.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.318+0000 7f0378b46640 1 --2- 172.21.15.67:0/2940831603 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f037415e2a0 0x7f0374162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:41:43.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.318+0000 7f0379347640 1 --2- 172.21.15.67:0/2940831603 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f037415dec0 0x7f03740ce630 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f03740cefb0 tx=0x7f036402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a22e5f409c283db1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:43.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.319+0000 7f0373fff640 1 -- 172.21.15.67:0/2940831603 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f036402ca50 con 0x7f037415dec0 2024-03-24T00:41:43.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.319+0000 7f0373fff640 1 -- 172.21.15.67:0/2940831603 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f036402cbf0 con 0x7f037415dec0 2024-03-24T00:41:43.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.319+0000 7f0373fff640 1 -- 172.21.15.67:0/2940831603 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03640343f0 con 0x7f037415dec0 2024-03-24T00:41:43.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.319+0000 7f037b5ab640 1 -- 172.21.15.67:0/2940831603 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f037415dec0 msgr2=0x7f03740ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:43.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.319+0000 7f037b5ab640 1 --2- 172.21.15.67:0/2940831603 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f037415dec0 0x7f03740ce630 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f03740cefb0 tx=0x7f036402beb0 comp rx=0 tx=0).stop 2024-03-24T00:41:43.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.319+0000 7f037b5ab640 1 -- 172.21.15.67:0/2940831603 shutdown_connections 2024-03-24T00:41:43.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.319+0000 7f037b5ab640 1 --2- 172.21.15.67:0/2940831603 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f037415e2a0 0x7f0374162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:43.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.319+0000 7f037b5ab640 1 --2- 172.21.15.67:0/2940831603 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f037415dec0 0x7f03740ce630 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:43.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.319+0000 7f037b5ab640 1 -- 172.21.15.67:0/2940831603 >> 172.21.15.67:0/2940831603 conn(0x7f037406c810 msgr2=0x7f037406cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:43.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.319+0000 7f037b5ab640 1 -- 172.21.15.67:0/2940831603 shutdown_connections 2024-03-24T00:41:43.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.319+0000 7f037b5ab640 1 -- 172.21.15.67:0/2940831603 wait complete. 2024-03-24T00:41:43.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.320+0000 7f037b5ab640 1 Processor -- start 2024-03-24T00:41:43.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.320+0000 7f037b5ab640 1 -- start start 2024-03-24T00:41:43.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.321+0000 7f037b5ab640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03740d07e0 0x7f0374159310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:43.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.321+0000 7f037b5ab640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f037415dec0 0x7f03740caea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:43.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.321+0000 7f037b5ab640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f037415e2a0 0x7f03740cb3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:43.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.321+0000 7f037b5ab640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03740d3740 con 0x7f037415e2a0 2024-03-24T00:41:43.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.321+0000 7f037b5ab640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03740d35c0 con 0x7f03740d07e0 2024-03-24T00:41:43.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.321+0000 7f037b5ab640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03740d38c0 con 0x7f037415dec0 2024-03-24T00:41:43.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.321+0000 7f0379b48640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f037415e2a0 0x7f03740cb3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:43.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.321+0000 7f0379b48640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f037415e2a0 0x7f03740cb3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43954/0 (socket says 172.21.15.67:43954) 2024-03-24T00:41:43.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.321+0000 7f0379b48640 1 -- 172.21.15.67:0/1753347152 learned_addr learned my addr 172.21.15.67:0/1753347152 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:43.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.321+0000 7f0379347640 1 --2- 172.21.15.67:0/1753347152 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03740d07e0 0x7f0374159310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:43.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.321+0000 7f0378b46640 1 --2- 172.21.15.67:0/1753347152 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f037415dec0 0x7f03740caea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:43.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.322+0000 7f0379b48640 1 -- 172.21.15.67:0/1753347152 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f037415dec0 msgr2=0x7f03740caea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:43.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.322+0000 7f0379b48640 1 --2- 172.21.15.67:0/1753347152 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f037415dec0 0x7f03740caea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:43.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.322+0000 7f0379b48640 1 -- 172.21.15.67:0/1753347152 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03740d07e0 msgr2=0x7f0374159310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:43.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.322+0000 7f0379b48640 1 --2- 172.21.15.67:0/1753347152 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03740d07e0 0x7f0374159310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:43.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.322+0000 7f0379b48640 1 -- 172.21.15.67:0/1753347152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0374158270 con 0x7f037415e2a0 2024-03-24T00:41:43.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.322+0000 7f0379347640 1 --2- 172.21.15.67:0/1753347152 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03740d07e0 0x7f0374159310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:41:43.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.322+0000 7f0379b48640 1 --2- 172.21.15.67:0/1753347152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f037415e2a0 0x7f03740cb3e0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f036c00c400 tx=0x7f036c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:43.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.322+0000 7f0371ffb640 1 -- 172.21.15.67:0/1753347152 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f036c017070 con 0x7f037415e2a0 2024-03-24T00:41:43.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.322+0000 7f0371ffb640 1 -- 172.21.15.67:0/1753347152 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f036c0052d0 con 0x7f037415e2a0 2024-03-24T00:41:43.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.322+0000 7f037b5ab640 1 -- 172.21.15.67:0/1753347152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0374158550 con 0x7f037415e2a0 2024-03-24T00:41:43.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.322+0000 7f0371ffb640 1 -- 172.21.15.67:0/1753347152 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f036c003810 con 0x7f037415e2a0 2024-03-24T00:41:43.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.322+0000 7f037b5ab640 1 -- 172.21.15.67:0/1753347152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0374158b10 con 0x7f037415e2a0 2024-03-24T00:41:43.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.323+0000 7f037b5ab640 1 -- 172.21.15.67:0/1753347152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0374068970 con 0x7f037415e2a0 2024-03-24T00:41:43.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.323+0000 7f0371ffb640 1 -- 172.21.15.67:0/1753347152 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f036c01a070 con 0x7f037415e2a0 2024-03-24T00:41:43.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.324+0000 7f0371ffb640 1 --2- 172.21.15.67:0/1753347152 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f036006bbd0 0x7f036006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:43.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.324+0000 7f0371ffb640 1 -- 172.21.15.67:0/1753347152 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f036c089fb0 con 0x7f037415e2a0 2024-03-24T00:41:43.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.324+0000 7f0379347640 1 --2- 172.21.15.67:0/1753347152 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f036006bbd0 0x7f036006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:43.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.324+0000 7f0379347640 1 --2- 172.21.15.67:0/1753347152 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f036006bbd0 0x7f036006e090 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f0364035000 tx=0x7f0364030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:43.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.329+0000 7f0371ffb640 1 -- 172.21.15.67:0/1753347152 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0374068970 con 0x7f037415e2a0 2024-03-24T00:41:43.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.512+0000 7f037b5ab640 1 -- 172.21.15.67:0/1753347152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f037404f400 con 0x7f037415e2a0 2024-03-24T00:41:43.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.514+0000 7f0371ffb640 1 -- 172.21.15.67:0/1753347152 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v61) v1 ==== 74+0+16837 (secure 0 0 0) 0x7f03740cf640 con 0x7f037415e2a0 2024-03-24T00:41:43.514 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:41:43.515 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":61,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:41:42.241896+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:41:40.502194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-03-24T00:41:41.200073+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:41:43.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.516+0000 7f037b5ab640 1 -- 172.21.15.67:0/1753347152 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f036006bbd0 msgr2=0x7f036006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:43.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.516+0000 7f037b5ab640 1 --2- 172.21.15.67:0/1753347152 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f036006bbd0 0x7f036006e090 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f0364035000 tx=0x7f0364030000 comp rx=0 tx=0).stop 2024-03-24T00:41:43.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.516+0000 7f037b5ab640 1 -- 172.21.15.67:0/1753347152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f037415e2a0 msgr2=0x7f03740cb3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:43.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.516+0000 7f037b5ab640 1 --2- 172.21.15.67:0/1753347152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f037415e2a0 0x7f03740cb3e0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f036c00c400 tx=0x7f036c00c8c0 comp rx=0 tx=0).stop 2024-03-24T00:41:43.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.516+0000 7f037b5ab640 1 -- 172.21.15.67:0/1753347152 shutdown_connections 2024-03-24T00:41:43.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.516+0000 7f037b5ab640 1 --2- 172.21.15.67:0/1753347152 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f036006bbd0 0x7f036006e090 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:43.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.516+0000 7f037b5ab640 1 --2- 172.21.15.67:0/1753347152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f037415e2a0 0x7f03740cb3e0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:43.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.516+0000 7f037b5ab640 1 --2- 172.21.15.67:0/1753347152 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f037415dec0 0x7f03740caea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:43.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.516+0000 7f037b5ab640 1 --2- 172.21.15.67:0/1753347152 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03740d07e0 0x7f0374159310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:43.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.516+0000 7f037b5ab640 1 -- 172.21.15.67:0/1753347152 >> 172.21.15.67:0/1753347152 conn(0x7f037406c810 msgr2=0x7f037415cf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:43.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.516+0000 7f037b5ab640 1 -- 172.21.15.67:0/1753347152 shutdown_connections 2024-03-24T00:41:43.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.516+0000 7f037b5ab640 1 -- 172.21.15.67:0/1753347152 wait complete. 2024-03-24T00:41:43.527 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-24T00:41:43.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.631+0000 7f1a5b2f8640 1 Processor -- start 2024-03-24T00:41:43.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.631+0000 7f1a5b2f8640 1 -- start start 2024-03-24T00:41:43.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.632+0000 7f1a5b2f8640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1a54144670 0x7f1a540b3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:43.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.632+0000 7f1a5b2f8640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1a5406ec60 0x7f1a5406f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:43.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.632+0000 7f1a5b2f8640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a540b2ab0 con 0x7f1a540b4a00 2024-03-24T00:41:43.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.632+0000 7f1a5b2f8640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a540b2d90 con 0x7f1a5406ec60 2024-03-24T00:41:43.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.632+0000 7f1a5b2f8640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a540b3070 con 0x7f1a54144670 2024-03-24T00:41:43.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.632+0000 7f1a59895640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1a5406ec60 0x7f1a5406f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:43.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.632+0000 7f1a59094640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1a54144670 0x7f1a540b3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:43.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.632+0000 7f1a59895640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1a5406ec60 0x7f1a5406f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:41614/0 (socket says 172.21.15.67:41614) 2024-03-24T00:41:43.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.632+0000 7f1a59895640 1 -- 172.21.15.67:0/979653574 learned_addr learned my addr 172.21.15.67:0/979653574 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:43.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.632+0000 7f1a59895640 1 -- 172.21.15.67:0/979653574 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1a54144670 msgr2=0x7f1a540b3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:43.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.632+0000 7f1a59895640 1 --2- 172.21.15.67:0/979653574 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1a54144670 0x7f1a540b3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:43.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.633+0000 7f1a59895640 1 -- 172.21.15.67:0/979653574 >> v1:172.21.15.72:6789/0 conn(0x7f1a540b4a00 legacy=0x7f1a54072500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:43.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.633+0000 7f1a59895640 1 -- 172.21.15.67:0/979653574 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a540b3350 con 0x7f1a5406ec60 2024-03-24T00:41:43.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.633+0000 7f1a59094640 1 --2- 172.21.15.67:0/979653574 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1a54144670 0x7f1a540b3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:41:43.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.633+0000 7f1a59895640 1 --2- 172.21.15.67:0/979653574 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1a5406ec60 0x7f1a5406f0f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c00d150 tx=0x7f1a4c02e160 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4bf1728b8169d352 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:43.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.633+0000 7f1a4bfff640 1 -- 172.21.15.67:0/979653574 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4c00b0d0 con 0x7f1a5406ec60 2024-03-24T00:41:43.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.633+0000 7f1a4bfff640 1 -- 172.21.15.67:0/979653574 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a4c00b270 con 0x7f1a5406ec60 2024-03-24T00:41:43.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.633+0000 7f1a4bfff640 1 -- 172.21.15.67:0/979653574 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4c009600 con 0x7f1a5406ec60 2024-03-24T00:41:43.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.634+0000 7f1a5b2f8640 1 -- 172.21.15.67:0/979653574 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1a5406ec60 msgr2=0x7f1a5406f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:43.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.634+0000 7f1a5b2f8640 1 --2- 172.21.15.67:0/979653574 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1a5406ec60 0x7f1a5406f0f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c00d150 tx=0x7f1a4c02e160 comp rx=0 tx=0).stop 2024-03-24T00:41:43.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.634+0000 7f1a5b2f8640 1 -- 172.21.15.67:0/979653574 shutdown_connections 2024-03-24T00:41:43.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.634+0000 7f1a5b2f8640 1 --2- 172.21.15.67:0/979653574 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1a5406ec60 0x7f1a5406f0f0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:43.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.634+0000 7f1a5b2f8640 1 --2- 172.21.15.67:0/979653574 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1a54144670 0x7f1a540b3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:43.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.634+0000 7f1a5b2f8640 1 -- 172.21.15.67:0/979653574 >> 172.21.15.67:0/979653574 conn(0x7f1a540cf9e0 msgr2=0x7f1a540cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:43.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.634+0000 7f1a5b2f8640 1 -- 172.21.15.67:0/979653574 shutdown_connections 2024-03-24T00:41:43.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.634+0000 7f1a5b2f8640 1 -- 172.21.15.67:0/979653574 wait complete. 2024-03-24T00:41:43.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.635+0000 7f1a5b2f8640 1 Processor -- start 2024-03-24T00:41:43.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.635+0000 7f1a5b2f8640 1 -- start start 2024-03-24T00:41:43.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.636+0000 7f1a5b2f8640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1a5406ec60 0x7f1a541f34e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:43.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.636+0000 7f1a5b2f8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1a540b4a00 0x7f1a540aad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:43.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.636+0000 7f1a5b2f8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a54144670 0x7f1a541ef4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:43.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.636+0000 7f1a5b2f8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a540cdfb0 con 0x7f1a54144670 2024-03-24T00:41:43.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.636+0000 7f1a5b2f8640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a540cde30 con 0x7f1a540b4a00 2024-03-24T00:41:43.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.636+0000 7f1a5b2f8640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a540ce130 con 0x7f1a5406ec60 2024-03-24T00:41:43.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.636+0000 7f1a59895640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a54144670 0x7f1a541ef4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:43.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.636+0000 7f1a59895640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a54144670 0x7f1a541ef4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54566/0 (socket says 172.21.15.67:54566) 2024-03-24T00:41:43.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.636+0000 7f1a59895640 1 -- 172.21.15.67:0/3827238414 learned_addr learned my addr 172.21.15.67:0/3827238414 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:43.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.636+0000 7f1a59094640 1 --2- 172.21.15.67:0/3827238414 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1a5406ec60 0x7f1a541f34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:43.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.636+0000 7f1a58893640 1 --2- 172.21.15.67:0/3827238414 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1a540b4a00 0x7f1a540aad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:43.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.637+0000 7f1a59895640 1 -- 172.21.15.67:0/3827238414 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1a5406ec60 msgr2=0x7f1a541f34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:43.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.637+0000 7f1a59895640 1 --2- 172.21.15.67:0/3827238414 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1a5406ec60 0x7f1a541f34e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:43.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.637+0000 7f1a59895640 1 -- 172.21.15.67:0/3827238414 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1a540b4a00 msgr2=0x7f1a540aad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:43.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.637+0000 7f1a59895640 1 --2- 172.21.15.67:0/3827238414 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1a540b4a00 0x7f1a540aad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:43.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.637+0000 7f1a59895640 1 -- 172.21.15.67:0/3827238414 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a541eff60 con 0x7f1a54144670 2024-03-24T00:41:43.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.637+0000 7f1a58893640 1 --2- 172.21.15.67:0/3827238414 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1a540b4a00 0x7f1a540aad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:41:43.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.637+0000 7f1a59895640 1 --2- 172.21.15.67:0/3827238414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a54144670 0x7f1a541ef4a0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c00d150 tx=0x7f1a4c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:43.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.637+0000 7f1a49ffb640 1 -- 172.21.15.67:0/3827238414 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4c045020 con 0x7f1a54144670 2024-03-24T00:41:43.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.637+0000 7f1a49ffb640 1 -- 172.21.15.67:0/3827238414 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a4c009e30 con 0x7f1a54144670 2024-03-24T00:41:43.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.637+0000 7f1a5b2f8640 1 -- 172.21.15.67:0/3827238414 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a541f0240 con 0x7f1a54144670 2024-03-24T00:41:43.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.637+0000 7f1a49ffb640 1 -- 172.21.15.67:0/3827238414 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4c00b0d0 con 0x7f1a54144670 2024-03-24T00:41:43.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.637+0000 7f1a5b2f8640 1 -- 172.21.15.67:0/3827238414 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a541f2fc0 con 0x7f1a54144670 2024-03-24T00:41:43.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.638+0000 7f1a5b2f8640 1 -- 172.21.15.67:0/3827238414 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a5404f400 con 0x7f1a54144670 2024-03-24T00:41:43.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.639+0000 7f1a49ffb640 1 -- 172.21.15.67:0/3827238414 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1a4c048020 con 0x7f1a54144670 2024-03-24T00:41:43.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.639+0000 7f1a49ffb640 1 --2- 172.21.15.67:0/3827238414 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1a3806bbd0 0x7f1a3806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:43.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.639+0000 7f1a49ffb640 1 -- 172.21.15.67:0/3827238414 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1a4c0af620 con 0x7f1a54144670 2024-03-24T00:41:43.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.639+0000 7f1a59094640 1 --2- 172.21.15.67:0/3827238414 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1a3806bbd0 0x7f1a3806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:43.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.640+0000 7f1a59094640 1 --2- 172.21.15.67:0/3827238414 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1a3806bbd0 0x7f1a3806e090 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f1a5406f500 tx=0x7f1a50009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:43.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.643+0000 7f1a49ffb640 1 -- 172.21.15.67:0/3827238414 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1a4c04f030 con 0x7f1a54144670 2024-03-24T00:41:43.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.864+0000 7f1a5b2f8640 1 -- 172.21.15.67:0/3827238414 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f1a540cf550 con 0x7f1a54144670 2024-03-24T00:41:43.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.882+0000 7f1a49ffb640 1 -- 172.21.15.67:0/3827238414 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v31) v1 ==== 118+0+0 (secure 0 0 0) 0x7f1a4c07df80 con 0x7f1a54144670 2024-03-24T00:41:43.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.884+0000 7f1a5b2f8640 1 -- 172.21.15.67:0/3827238414 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1a3806bbd0 msgr2=0x7f1a3806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:43.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.884+0000 7f1a5b2f8640 1 --2- 172.21.15.67:0/3827238414 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1a3806bbd0 0x7f1a3806e090 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f1a5406f500 tx=0x7f1a50009190 comp rx=0 tx=0).stop 2024-03-24T00:41:43.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.884+0000 7f1a5b2f8640 1 -- 172.21.15.67:0/3827238414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a54144670 msgr2=0x7f1a541ef4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:43.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.884+0000 7f1a5b2f8640 1 --2- 172.21.15.67:0/3827238414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a54144670 0x7f1a541ef4a0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c00d150 tx=0x7f1a4c004a40 comp rx=0 tx=0).stop 2024-03-24T00:41:43.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.885+0000 7f1a5b2f8640 1 -- 172.21.15.67:0/3827238414 shutdown_connections 2024-03-24T00:41:43.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.885+0000 7f1a5b2f8640 1 --2- 172.21.15.67:0/3827238414 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1a3806bbd0 0x7f1a3806e090 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:43.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.885+0000 7f1a5b2f8640 1 --2- 172.21.15.67:0/3827238414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a54144670 0x7f1a541ef4a0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:43.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.885+0000 7f1a5b2f8640 1 --2- 172.21.15.67:0/3827238414 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1a540b4a00 0x7f1a540aad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:43.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.885+0000 7f1a5b2f8640 1 --2- 172.21.15.67:0/3827238414 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1a5406ec60 0x7f1a541f34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:43.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.885+0000 7f1a5b2f8640 1 -- 172.21.15.67:0/3827238414 >> 172.21.15.67:0/3827238414 conn(0x7f1a540cf9e0 msgr2=0x7f1a54071ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:43.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.885+0000 7f1a5b2f8640 1 -- 172.21.15.67:0/3827238414 shutdown_connections 2024-03-24T00:41:43.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:43.885+0000 7f1a5b2f8640 1 -- 172.21.15.67:0/3827238414 wait complete. 2024-03-24T00:41:43.896 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-24T00:41:43.896 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-24T00:41:44.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.000+0000 7f868fe75640 1 Processor -- start 2024-03-24T00:41:44.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.000+0000 7f868fe75640 1 -- start start 2024-03-24T00:41:44.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.001+0000 7f868fe75640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f868815bee0 0x7f8688099a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:44.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.001+0000 7f868fe75640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8688160430 0x7f868806b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:44.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.001+0000 7f868fe75640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f86880ccc50 0x7f86880cd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:44.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.001+0000 7f868fe75640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86880cd5f0 con 0x7f8688160430 2024-03-24T00:41:44.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.001+0000 7f868fe75640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86880cf180 con 0x7f86880ccc50 2024-03-24T00:41:44.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.001+0000 7f868fe75640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86880af730 con 0x7f868815bee0 2024-03-24T00:41:44.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.001+0000 7f868d410640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8688160430 0x7f868806b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:44.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.001+0000 7f868d410640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8688160430 0x7f868806b040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54576/0 (socket says 172.21.15.67:54576) 2024-03-24T00:41:44.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.001+0000 7f868d410640 1 -- 172.21.15.67:0/2259645967 learned_addr learned my addr 172.21.15.67:0/2259645967 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:44.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.001+0000 7f868dc11640 1 --2- 172.21.15.67:0/2259645967 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f868815bee0 0x7f8688099a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:44.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.001+0000 7f868e412640 1 --2- 172.21.15.67:0/2259645967 >> v2:172.21.15.72:3300/0 conn(0x7f86880ccc50 0x7f86880cd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:44.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.001+0000 7f868d410640 1 -- 172.21.15.67:0/2259645967 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f868815bee0 msgr2=0x7f8688099a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:44.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.001+0000 7f868d410640 1 --2- 172.21.15.67:0/2259645967 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f868815bee0 0x7f8688099a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:44.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.001+0000 7f868d410640 1 -- 172.21.15.67:0/2259645967 >> v2:172.21.15.72:3300/0 conn(0x7f86880ccc50 msgr2=0x7f86880cd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:44.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.001+0000 7f868d410640 1 --2- 172.21.15.67:0/2259645967 >> v2:172.21.15.72:3300/0 conn(0x7f86880ccc50 0x7f86880cd0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:44.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.001+0000 7f868d410640 1 -- 172.21.15.67:0/2259645967 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86880b2880 con 0x7f8688160430 2024-03-24T00:41:44.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.002+0000 7f868d410640 1 --2- 172.21.15.67:0/2259645967 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8688160430 0x7f868806b040 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f86880b6250 tx=0x7f867802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ada4e51492e674bb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:44.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.002+0000 7f868cc0f640 1 -- 172.21.15.67:0/2259645967 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8678035020 con 0x7f8688160430 2024-03-24T00:41:44.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.002+0000 7f868cc0f640 1 -- 172.21.15.67:0/2259645967 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8678031870 con 0x7f8688160430 2024-03-24T00:41:44.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.002+0000 7f868cc0f640 1 -- 172.21.15.67:0/2259645967 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8678031c40 con 0x7f8688160430 2024-03-24T00:41:44.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.004+0000 7f868fe75640 1 -- 172.21.15.67:0/2259645967 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8688160430 msgr2=0x7f868806b040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:44.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.004+0000 7f868fe75640 1 --2- 172.21.15.67:0/2259645967 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8688160430 0x7f868806b040 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f86880b6250 tx=0x7f867802b910 comp rx=0 tx=0).stop 2024-03-24T00:41:44.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.004+0000 7f868fe75640 1 -- 172.21.15.67:0/2259645967 shutdown_connections 2024-03-24T00:41:44.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.004+0000 7f868fe75640 1 --2- 172.21.15.67:0/2259645967 >> v2:172.21.15.72:3300/0 conn(0x7f86880ccc50 0x7f86880cd0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:44.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.004+0000 7f868fe75640 1 --2- 172.21.15.67:0/2259645967 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8688160430 0x7f868806b040 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:44.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.004+0000 7f868fe75640 1 --2- 172.21.15.67:0/2259645967 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f868815bee0 0x7f8688099a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:44.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.004+0000 7f868fe75640 1 -- 172.21.15.67:0/2259645967 >> 172.21.15.67:0/2259645967 conn(0x7f86880b3c40 msgr2=0x7f86880b1dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:44.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.004+0000 7f868fe75640 1 -- 172.21.15.67:0/2259645967 shutdown_connections 2024-03-24T00:41:44.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.004+0000 7f868fe75640 1 -- 172.21.15.67:0/2259645967 wait complete. 2024-03-24T00:41:44.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.004+0000 7f868fe75640 1 Processor -- start 2024-03-24T00:41:44.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.004+0000 7f868fe75640 1 -- start start 2024-03-24T00:41:44.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.006+0000 7f868fe75640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86880ccc50 0x7f86880d07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:44.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.006+0000 7f868fe75640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f868815bee0 0x7f86880c7340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:44.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.006+0000 7f868fe75640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8688160430 0x7f86880c7880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:44.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.006+0000 7f868fe75640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86880ceda0 con 0x7f86880ccc50 2024-03-24T00:41:44.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.006+0000 7f868fe75640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f868815c2c0 con 0x7f868815bee0 2024-03-24T00:41:44.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.006+0000 7f868fe75640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f868806b580 con 0x7f8688160430 2024-03-24T00:41:44.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.006+0000 7f868dc11640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86880ccc50 0x7f86880d07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:44.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.006+0000 7f868dc11640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86880ccc50 0x7f86880d07c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54580/0 (socket says 172.21.15.67:54580) 2024-03-24T00:41:44.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.006+0000 7f868dc11640 1 -- 172.21.15.67:0/3596877175 learned_addr learned my addr 172.21.15.67:0/3596877175 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:44.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.006+0000 7f868d410640 1 --2- 172.21.15.67:0/3596877175 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f868815bee0 0x7f86880c7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:44.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.006+0000 7f868e412640 1 --2- 172.21.15.67:0/3596877175 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8688160430 0x7f86880c7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:44.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.006+0000 7f868dc11640 1 -- 172.21.15.67:0/3596877175 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8688160430 msgr2=0x7f86880c7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:44.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.006+0000 7f868dc11640 1 --2- 172.21.15.67:0/3596877175 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8688160430 0x7f86880c7880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:44.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.006+0000 7f868dc11640 1 -- 172.21.15.67:0/3596877175 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f868815bee0 msgr2=0x7f86880c7340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:44.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.006+0000 7f868dc11640 1 --2- 172.21.15.67:0/3596877175 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f868815bee0 0x7f86880c7340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:44.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.006+0000 7f868dc11640 1 -- 172.21.15.67:0/3596877175 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86880c7ee0 con 0x7f86880ccc50 2024-03-24T00:41:44.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.006+0000 7f868d410640 1 --2- 172.21.15.67:0/3596877175 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f868815bee0 0x7f86880c7340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:41:44.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.006+0000 7f868dc11640 1 --2- 172.21.15.67:0/3596877175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86880ccc50 0x7f86880d07c0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f868400d800 tx=0x7f868400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:44.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.006+0000 7f867e7fc640 1 -- 172.21.15.67:0/3596877175 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868400b6c0 con 0x7f86880ccc50 2024-03-24T00:41:44.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.006+0000 7f867e7fc640 1 -- 172.21.15.67:0/3596877175 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8684019070 con 0x7f86880ccc50 2024-03-24T00:41:44.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.006+0000 7f867e7fc640 1 -- 172.21.15.67:0/3596877175 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868400ba80 con 0x7f86880ccc50 2024-03-24T00:41:44.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.006+0000 7f868fe75640 1 -- 172.21.15.67:0/3596877175 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86880c81c0 con 0x7f86880ccc50 2024-03-24T00:41:44.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.006+0000 7f868fe75640 1 -- 172.21.15.67:0/3596877175 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86880c8780 con 0x7f86880ccc50 2024-03-24T00:41:44.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.007+0000 7f868fe75640 1 -- 172.21.15.67:0/3596877175 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f868804f400 con 0x7f86880ccc50 2024-03-24T00:41:44.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.007+0000 7f867e7fc640 1 -- 172.21.15.67:0/3596877175 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f868400bc20 con 0x7f86880ccc50 2024-03-24T00:41:44.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.008+0000 7f867e7fc640 1 --2- 172.21.15.67:0/3596877175 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f866c06bbd0 0x7f866c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:44.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.008+0000 7f867e7fc640 1 -- 172.21.15.67:0/3596877175 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f868408b180 con 0x7f86880ccc50 2024-03-24T00:41:44.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.008+0000 7f868d410640 1 --2- 172.21.15.67:0/3596877175 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f866c06bbd0 0x7f866c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:44.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.009+0000 7f868d410640 1 --2- 172.21.15.67:0/3596877175 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f866c06bbd0 0x7f866c06e090 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f8678036000 tx=0x7f867803c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:44.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.012+0000 7f867e7fc640 1 -- 172.21.15.67:0/3596877175 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8684059a50 con 0x7f86880ccc50 2024-03-24T00:41:44.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.236+0000 7f868fe75640 1 -- 172.21.15.67:0/3596877175 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f86880cda40 con 0x7f86880ccc50 2024-03-24T00:41:44.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.887+0000 7f867e7fc640 1 -- 172.21.15.67:0/3596877175 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v32) v1 ==== 122+0+0 (secure 0 0 0) 0x7f8684059350 con 0x7f86880ccc50 2024-03-24T00:41:44.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.889+0000 7f868fe75640 1 -- 172.21.15.67:0/3596877175 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f866c06bbd0 msgr2=0x7f866c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:44.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.889+0000 7f868fe75640 1 --2- 172.21.15.67:0/3596877175 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f866c06bbd0 0x7f866c06e090 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f8678036000 tx=0x7f867803c040 comp rx=0 tx=0).stop 2024-03-24T00:41:44.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.889+0000 7f868fe75640 1 -- 172.21.15.67:0/3596877175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86880ccc50 msgr2=0x7f86880d07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:44.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.889+0000 7f868fe75640 1 --2- 172.21.15.67:0/3596877175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86880ccc50 0x7f86880d07c0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f868400d800 tx=0x7f868400dcc0 comp rx=0 tx=0).stop 2024-03-24T00:41:44.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.889+0000 7f868fe75640 1 -- 172.21.15.67:0/3596877175 shutdown_connections 2024-03-24T00:41:44.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.889+0000 7f868fe75640 1 --2- 172.21.15.67:0/3596877175 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f866c06bbd0 0x7f866c06e090 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:44.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.889+0000 7f868fe75640 1 --2- 172.21.15.67:0/3596877175 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8688160430 0x7f86880c7880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:44.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.889+0000 7f868fe75640 1 --2- 172.21.15.67:0/3596877175 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f868815bee0 0x7f86880c7340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:44.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.889+0000 7f868fe75640 1 --2- 172.21.15.67:0/3596877175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86880ccc50 0x7f86880d07c0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:44.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.889+0000 7f868fe75640 1 -- 172.21.15.67:0/3596877175 >> 172.21.15.67:0/3596877175 conn(0x7f86880b3c40 msgr2=0x7f868815b720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:44.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.889+0000 7f868fe75640 1 -- 172.21.15.67:0/3596877175 shutdown_connections 2024-03-24T00:41:44.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:44.889+0000 7f868fe75640 1 -- 172.21.15.67:0/3596877175 wait complete. 2024-03-24T00:41:44.905 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-24T00:41:44.906 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:41:45.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.007+0000 7f827307e640 1 Processor -- start 2024-03-24T00:41:45.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.007+0000 7f827307e640 1 -- start start 2024-03-24T00:41:45.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.008+0000 7f827307e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f826c0cfc70 0x7f826c0ce220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:45.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.008+0000 7f827307e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f826c0ccfe0 con 0x7f826c0ce760 2024-03-24T00:41:45.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.008+0000 7f827307e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f826c0cd2c0 con 0x7f826c15bf70 2024-03-24T00:41:45.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.008+0000 7f827307e640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f826c0ccda0 con 0x7f826c0cfc70 2024-03-24T00:41:45.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.008+0000 7f826bfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f826c0cfc70 0x7f826c0ce220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:45.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.008+0000 7f827161b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f826c0ce760 0x7f826c0b1cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56326/0 (socket says 172.21.15.67:56326) 2024-03-24T00:41:45.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.008+0000 7f827161b640 1 -- 172.21.15.67:0/2900177058 learned_addr learned my addr 172.21.15.67:0/2900177058 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:41:45.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.008+0000 7f826bfff640 1 -- 172.21.15.67:0/2900177058 >> v1:172.21.15.72:6789/0 conn(0x7f826c15bf70 legacy=0x7f826c0cc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:45.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.008+0000 7f826bfff640 1 -- 172.21.15.67:0/2900177058 >> v1:172.21.15.67:6789/0 conn(0x7f826c0ce760 legacy=0x7f826c0b1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:45.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.008+0000 7f826bfff640 1 -- 172.21.15.67:0/2900177058 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f826c0cd5a0 con 0x7f826c0cfc70 2024-03-24T00:41:45.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.008+0000 7f826b7fe640 1 -- 172.21.15.67:0/2900177058 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2060367401 0 0) 0x7f826c0ccfe0 con 0x7f826c0ce760 2024-03-24T00:41:45.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.009+0000 7f826bfff640 1 --2- 172.21.15.67:0/2900177058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f826c0cfc70 0x7f826c0ce220 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f82580097a0 tx=0x7f825802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20cf33a08350b2d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:45.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.009+0000 7f826b7fe640 1 -- 172.21.15.67:0/2900177058 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8258034070 con 0x7f826c0cfc70 2024-03-24T00:41:45.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.009+0000 7f826b7fe640 1 -- 172.21.15.67:0/2900177058 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8258030610 con 0x7f826c0cfc70 2024-03-24T00:41:45.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.009+0000 7f826b7fe640 1 -- 172.21.15.67:0/2900177058 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82580309e0 con 0x7f826c0cfc70 2024-03-24T00:41:45.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.010+0000 7f827307e640 1 -- 172.21.15.67:0/2900177058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f826c0cfc70 msgr2=0x7f826c0ce220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:45.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.010+0000 7f827307e640 1 --2- 172.21.15.67:0/2900177058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f826c0cfc70 0x7f826c0ce220 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f82580097a0 tx=0x7f825802b960 comp rx=0 tx=0).stop 2024-03-24T00:41:45.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.010+0000 7f827307e640 1 -- 172.21.15.67:0/2900177058 shutdown_connections 2024-03-24T00:41:45.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.010+0000 7f827307e640 1 --2- 172.21.15.67:0/2900177058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f826c0cfc70 0x7f826c0ce220 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:45.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.010+0000 7f827307e640 1 -- 172.21.15.67:0/2900177058 >> 172.21.15.67:0/2900177058 conn(0x7f826c0d1bf0 msgr2=0x7f826c0d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:45.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.010+0000 7f827307e640 1 -- 172.21.15.67:0/2900177058 shutdown_connections 2024-03-24T00:41:45.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.010+0000 7f827307e640 1 -- 172.21.15.67:0/2900177058 wait complete. 2024-03-24T00:41:45.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.012+0000 7f827307e640 1 Processor -- start 2024-03-24T00:41:45.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.012+0000 7f827307e640 1 -- start start 2024-03-24T00:41:45.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.013+0000 7f827307e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f826c0ce760 0x7f826c15cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:45.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.013+0000 7f827307e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f826c0cfc70 0x7f826c14ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:45.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.013+0000 7f827307e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f826c15bf70 0x7f826c0cacd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:45.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.013+0000 7f827307e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f826c0b23e0 con 0x7f826c15bf70 2024-03-24T00:41:45.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.013+0000 7f827307e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f826c15c350 con 0x7f826c0cfc70 2024-03-24T00:41:45.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.013+0000 7f827307e640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f826c0b2560 con 0x7f826c0ce760 2024-03-24T00:41:45.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.013+0000 7f827161b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f826c15bf70 0x7f826c0cacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:45.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.013+0000 7f827161b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f826c15bf70 0x7f826c0cacd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54596/0 (socket says 172.21.15.67:54596) 2024-03-24T00:41:45.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.013+0000 7f827161b640 1 -- 172.21.15.67:0/2192741928 learned_addr learned my addr 172.21.15.67:0/2192741928 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:45.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.013+0000 7f826bfff640 1 --2- 172.21.15.67:0/2192741928 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f826c0cfc70 0x7f826c14ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:45.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.013+0000 7f8270e1a640 1 --2- 172.21.15.67:0/2192741928 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f826c0ce760 0x7f826c15cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:45.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.013+0000 7f827161b640 1 -- 172.21.15.67:0/2192741928 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f826c0ce760 msgr2=0x7f826c15cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:45.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.013+0000 7f827161b640 1 --2- 172.21.15.67:0/2192741928 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f826c0ce760 0x7f826c15cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:45.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.013+0000 7f827161b640 1 -- 172.21.15.67:0/2192741928 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f826c0cfc70 msgr2=0x7f826c14ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:45.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.013+0000 7f827161b640 1 --2- 172.21.15.67:0/2192741928 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f826c0cfc70 0x7f826c14ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:45.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.013+0000 7f827161b640 1 -- 172.21.15.67:0/2192741928 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f826c15d420 con 0x7f826c15bf70 2024-03-24T00:41:45.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.013+0000 7f827161b640 1 --2- 172.21.15.67:0/2192741928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f826c15bf70 0x7f826c0cacd0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f826400c7d0 tx=0x7f826400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:45.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.013+0000 7f82697fa640 1 -- 172.21.15.67:0/2192741928 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8264018070 con 0x7f826c15bf70 2024-03-24T00:41:45.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.014+0000 7f82697fa640 1 -- 172.21.15.67:0/2192741928 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f826400a5e0 con 0x7f826c15bf70 2024-03-24T00:41:45.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.014+0000 7f827307e640 1 -- 172.21.15.67:0/2192741928 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f826c15e4a0 con 0x7f826c15bf70 2024-03-24T00:41:45.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.014+0000 7f82697fa640 1 -- 172.21.15.67:0/2192741928 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f826400aa10 con 0x7f826c15bf70 2024-03-24T00:41:45.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.014+0000 7f827307e640 1 -- 172.21.15.67:0/2192741928 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f826c15ed40 con 0x7f826c15bf70 2024-03-24T00:41:45.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.015+0000 7f827307e640 1 -- 172.21.15.67:0/2192741928 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f826c04f400 con 0x7f826c15bf70 2024-03-24T00:41:45.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.015+0000 7f82697fa640 1 -- 172.21.15.67:0/2192741928 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f826401b070 con 0x7f826c15bf70 2024-03-24T00:41:45.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.016+0000 7f82697fa640 1 --2- 172.21.15.67:0/2192741928 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f825006bbd0 0x7f825006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:45.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.016+0000 7f82697fa640 1 -- 172.21.15.67:0/2192741928 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f826408a100 con 0x7f826c15bf70 2024-03-24T00:41:45.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.016+0000 7f8270e1a640 1 --2- 172.21.15.67:0/2192741928 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f825006bbd0 0x7f825006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:45.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.016+0000 7f8270e1a640 1 --2- 172.21.15.67:0/2192741928 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f825006bbd0 0x7f825006e090 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f8260000f60 tx=0x7f8260009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:45.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.020+0000 7f82697fa640 1 -- 172.21.15.67:0/2192741928 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8264058410 con 0x7f826c15bf70 2024-03-24T00:41:45.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.241+0000 7f827307e640 1 -- 172.21.15.67:0/2192741928 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f826c0d0600 con 0x7f826c15bf70 2024-03-24T00:41:45.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.242+0000 7f82697fa640 1 -- 172.21.15.67:0/2192741928 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+4975 (secure 0 0 0) 0x7f826405c4b0 con 0x7f826c15bf70 2024-03-24T00:41:45.242 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:41:45.242 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":32,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5627,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2941726873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2941726873},{"type":"v1","addr":"172.21.15.72:6837","nonce":2941726873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":6166,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/1957901519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":1957901519},{"type":"v1","addr":"172.21.15.72:6836","nonce":1957901519}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":6357,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/2843754917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2843754917},{"type":"v1","addr":"172.21.15.67:6833","nonce":2843754917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:41:42.242211+0000","modified":"2024-03-24T00:41:44.237560+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6360},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6360":{"gid":6360,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/451962260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":451962260},{"type":"v1","addr":"172.21.15.67:6835","nonce":451962260}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6360,"qdb_cluster":[6360]},"id":4}]} 2024-03-24T00:41:45.242 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 32 2024-03-24T00:41:45.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.244+0000 7f827307e640 1 -- 172.21.15.67:0/2192741928 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f825006bbd0 msgr2=0x7f825006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:45.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.244+0000 7f827307e640 1 --2- 172.21.15.67:0/2192741928 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f825006bbd0 0x7f825006e090 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f8260000f60 tx=0x7f8260009040 comp rx=0 tx=0).stop 2024-03-24T00:41:45.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.244+0000 7f827307e640 1 -- 172.21.15.67:0/2192741928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f826c15bf70 msgr2=0x7f826c0cacd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:45.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.244+0000 7f827307e640 1 --2- 172.21.15.67:0/2192741928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f826c15bf70 0x7f826c0cacd0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f826400c7d0 tx=0x7f826400cc90 comp rx=0 tx=0).stop 2024-03-24T00:41:45.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.244+0000 7f827307e640 1 -- 172.21.15.67:0/2192741928 shutdown_connections 2024-03-24T00:41:45.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.244+0000 7f827307e640 1 --2- 172.21.15.67:0/2192741928 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f825006bbd0 0x7f825006e090 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:45.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.244+0000 7f827307e640 1 --2- 172.21.15.67:0/2192741928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f826c15bf70 0x7f826c0cacd0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:45.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.244+0000 7f827307e640 1 --2- 172.21.15.67:0/2192741928 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f826c0cfc70 0x7f826c14ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:45.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.244+0000 7f827307e640 1 --2- 172.21.15.67:0/2192741928 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f826c0ce760 0x7f826c15cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:45.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.244+0000 7f827307e640 1 -- 172.21.15.67:0/2192741928 >> 172.21.15.67:0/2192741928 conn(0x7f826c0d1bf0 msgr2=0x7f826c0b05d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:45.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.244+0000 7f827307e640 1 -- 172.21.15.67:0/2192741928 shutdown_connections 2024-03-24T00:41:45.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.244+0000 7f827307e640 1 -- 172.21.15.67:0/2192741928 wait complete. 2024-03-24T00:41:45.255 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:41:45.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.356+0000 7fc7d4f52640 1 Processor -- start 2024-03-24T00:41:45.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.356+0000 7fc7d4f52640 1 -- start start 2024-03-24T00:41:45.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.357+0000 7fc7d4f52640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc7d015e2a0 0x7fc7d00ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:45.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.357+0000 7fc7d4f52640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc7d00d0ef0 0x7fc7d0162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:45.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.357+0000 7fc7d4f52640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7d00cf640 con 0x7fc7d015dec0 2024-03-24T00:41:45.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.357+0000 7fc7d4f52640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7d00cee30 con 0x7fc7d015e2a0 2024-03-24T00:41:45.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.357+0000 7fc7d4f52640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d00d2120 con 0x7fc7d00d0ef0 2024-03-24T00:41:45.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.358+0000 7fc7cdd9b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc7d015e2a0 0x7fc7d00ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:45.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.358+0000 7fc7ce59c640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fc7d015dec0 0x7fc7d00d07e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:52198/0 (socket says 172.21.15.67:52198) 2024-03-24T00:41:45.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.358+0000 7fc7ce59c640 1 -- 172.21.15.67:0/1771247966 learned_addr learned my addr 172.21.15.67:0/1771247966 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:41:45.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.358+0000 7fc7ced9d640 1 --2- 172.21.15.67:0/1771247966 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc7d00d0ef0 0x7fc7d0162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:45.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.358+0000 7fc7cd59a640 1 -- 172.21.15.67:0/1771247966 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2295005603 0 0) 0x7fc7c0001110 con 0x7fc7d015dec0 2024-03-24T00:41:45.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.358+0000 7fc7cd59a640 1 -- 172.21.15.67:0/1771247966 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7b8003520 con 0x7fc7d015dec0 2024-03-24T00:41:45.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.358+0000 7fc7ced9d640 1 -- 172.21.15.67:0/1771247966 >> v2:172.21.15.72:3300/0 conn(0x7fc7d015e2a0 msgr2=0x7fc7d00ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:45.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.358+0000 7fc7ced9d640 1 --2- 172.21.15.67:0/1771247966 >> v2:172.21.15.72:3300/0 conn(0x7fc7d015e2a0 0x7fc7d00ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:45.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.358+0000 7fc7ced9d640 1 -- 172.21.15.67:0/1771247966 >> v1:172.21.15.72:6789/0 conn(0x7fc7d015dec0 legacy=0x7fc7d00d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:45.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.358+0000 7fc7ced9d640 1 -- 172.21.15.67:0/1771247966 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7d00cf080 con 0x7fc7d00d0ef0 2024-03-24T00:41:45.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.359+0000 7fc7ced9d640 1 --2- 172.21.15.67:0/1771247966 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc7d00d0ef0 0x7fc7d0162630 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc7c8009d90 tx=0x7fc7c802be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1297e3c09973abde server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:45.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.359+0000 7fc7cd59a640 1 -- 172.21.15.67:0/1771247966 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c8039070 con 0x7fc7d00d0ef0 2024-03-24T00:41:45.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.359+0000 7fc7cd59a640 1 -- 172.21.15.67:0/1771247966 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7c802cbc0 con 0x7fc7d00d0ef0 2024-03-24T00:41:45.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.359+0000 7fc7cd59a640 1 -- 172.21.15.67:0/1771247966 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c8034440 con 0x7fc7d00d0ef0 2024-03-24T00:41:45.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.360+0000 7fc7d4f52640 1 -- 172.21.15.67:0/1771247966 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc7d00d0ef0 msgr2=0x7fc7d0162630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:45.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.360+0000 7fc7d4f52640 1 --2- 172.21.15.67:0/1771247966 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc7d00d0ef0 0x7fc7d0162630 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc7c8009d90 tx=0x7fc7c802be10 comp rx=0 tx=0).stop 2024-03-24T00:41:45.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.360+0000 7fc7d4f52640 1 -- 172.21.15.67:0/1771247966 shutdown_connections 2024-03-24T00:41:45.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.360+0000 7fc7d4f52640 1 --2- 172.21.15.67:0/1771247966 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc7d00d0ef0 0x7fc7d0162630 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:45.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.360+0000 7fc7d4f52640 1 --2- 172.21.15.67:0/1771247966 >> v2:172.21.15.72:3300/0 conn(0x7fc7d015e2a0 0x7fc7d00ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:45.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.360+0000 7fc7d4f52640 1 -- 172.21.15.67:0/1771247966 >> 172.21.15.67:0/1771247966 conn(0x7fc7d006c810 msgr2=0x7fc7d006cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:45.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.360+0000 7fc7d4f52640 1 -- 172.21.15.67:0/1771247966 shutdown_connections 2024-03-24T00:41:45.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.360+0000 7fc7d4f52640 1 -- 172.21.15.67:0/1771247966 wait complete. 2024-03-24T00:41:45.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.361+0000 7fc7d4f52640 1 Processor -- start 2024-03-24T00:41:45.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.361+0000 7fc7d4f52640 1 -- start start 2024-03-24T00:41:45.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.362+0000 7fc7d4f52640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc7d00d0ef0 0x7fc7d00b3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:45.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.362+0000 7fc7d4f52640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc7d015dec0 0x7fc7d01562d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:45.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.362+0000 7fc7d4f52640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7d015e2a0 0x7fc7d01f7970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:45.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.362+0000 7fc7d4f52640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d00d2730 con 0x7fc7d015e2a0 2024-03-24T00:41:45.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.362+0000 7fc7d4f52640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d00d25b0 con 0x7fc7d00d0ef0 2024-03-24T00:41:45.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.362+0000 7fc7d4f52640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d00d28b0 con 0x7fc7d015dec0 2024-03-24T00:41:45.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.362+0000 7fc7ced9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7d015e2a0 0x7fc7d01f7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:45.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.362+0000 7fc7ced9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7d015e2a0 0x7fc7d01f7970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54598/0 (socket says 172.21.15.67:54598) 2024-03-24T00:41:45.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.362+0000 7fc7ced9d640 1 -- 172.21.15.67:0/3227872635 learned_addr learned my addr 172.21.15.67:0/3227872635 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:45.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.362+0000 7fc7ce59c640 1 --2- 172.21.15.67:0/3227872635 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc7d00d0ef0 0x7fc7d00b3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:45.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.362+0000 7fc7cdd9b640 1 --2- 172.21.15.67:0/3227872635 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc7d015dec0 0x7fc7d01562d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:45.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.362+0000 7fc7ced9d640 1 -- 172.21.15.67:0/3227872635 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc7d015dec0 msgr2=0x7fc7d01562d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:45.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.362+0000 7fc7ced9d640 1 --2- 172.21.15.67:0/3227872635 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc7d015dec0 0x7fc7d01562d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:45.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.362+0000 7fc7ced9d640 1 -- 172.21.15.67:0/3227872635 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc7d00d0ef0 msgr2=0x7fc7d00b3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:45.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.362+0000 7fc7ced9d640 1 --2- 172.21.15.67:0/3227872635 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc7d00d0ef0 0x7fc7d00b3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:45.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.362+0000 7fc7ced9d640 1 -- 172.21.15.67:0/3227872635 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7d00b4290 con 0x7fc7d015e2a0 2024-03-24T00:41:45.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.363+0000 7fc7ced9d640 1 --2- 172.21.15.67:0/3227872635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7d015e2a0 0x7fc7d01f7970 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fc7c8009d90 tx=0x7fc7c8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:45.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.363+0000 7fc7beffd640 1 -- 172.21.15.67:0/3227872635 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c8039050 con 0x7fc7d015e2a0 2024-03-24T00:41:45.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.363+0000 7fc7beffd640 1 -- 172.21.15.67:0/3227872635 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7c8045070 con 0x7fc7d015e2a0 2024-03-24T00:41:45.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.363+0000 7fc7d4f52640 1 -- 172.21.15.67:0/3227872635 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7d00cbff0 con 0x7fc7d015e2a0 2024-03-24T00:41:45.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.363+0000 7fc7beffd640 1 -- 172.21.15.67:0/3227872635 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c8009450 con 0x7fc7d015e2a0 2024-03-24T00:41:45.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.363+0000 7fc7d4f52640 1 -- 172.21.15.67:0/3227872635 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7d00cc830 con 0x7fc7d015e2a0 2024-03-24T00:41:45.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.364+0000 7fc7d4f52640 1 -- 172.21.15.67:0/3227872635 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7d004f400 con 0x7fc7d015e2a0 2024-03-24T00:41:45.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.364+0000 7fc7beffd640 1 -- 172.21.15.67:0/3227872635 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc7c803e850 con 0x7fc7d015e2a0 2024-03-24T00:41:45.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.365+0000 7fc7beffd640 1 --2- 172.21.15.67:0/3227872635 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc7b006bbd0 0x7fc7b006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:45.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.365+0000 7fc7beffd640 1 -- 172.21.15.67:0/3227872635 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc7c80ada10 con 0x7fc7d015e2a0 2024-03-24T00:41:45.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.365+0000 7fc7ce59c640 1 --2- 172.21.15.67:0/3227872635 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc7b006bbd0 0x7fc7b006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:45.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.365+0000 7fc7ce59c640 1 --2- 172.21.15.67:0/3227872635 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc7b006bbd0 0x7fc7b006e090 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fc7c0005880 tx=0x7fc7c0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:45.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.369+0000 7fc7beffd640 1 -- 172.21.15.67:0/3227872635 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc7d00cbff0 con 0x7fc7d015e2a0 2024-03-24T00:41:45.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.605+0000 7fc7d4f52640 1 -- 172.21.15.67:0/3227872635 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc7d00d1aa0 con 0x7fc7d015e2a0 2024-03-24T00:41:45.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.606+0000 7fc7beffd640 1 -- 172.21.15.67:0/3227872635 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v62) v1 ==== 74+0+16824 (secure 0 0 0) 0x7fc7c807c490 con 0x7fc7d015e2a0 2024-03-24T00:41:45.606 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:41:45.606 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":62,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:41:43.868916+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:41:40.502194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-03-24T00:41:41.200073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:41:45.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.608+0000 7fc7d4f52640 1 -- 172.21.15.67:0/3227872635 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc7b006bbd0 msgr2=0x7fc7b006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:45.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.608+0000 7fc7d4f52640 1 --2- 172.21.15.67:0/3227872635 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc7b006bbd0 0x7fc7b006e090 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fc7c0005880 tx=0x7fc7c0009040 comp rx=0 tx=0).stop 2024-03-24T00:41:45.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.608+0000 7fc7d4f52640 1 -- 172.21.15.67:0/3227872635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7d015e2a0 msgr2=0x7fc7d01f7970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:45.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.608+0000 7fc7d4f52640 1 --2- 172.21.15.67:0/3227872635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7d015e2a0 0x7fc7d01f7970 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fc7c8009d90 tx=0x7fc7c8004a40 comp rx=0 tx=0).stop 2024-03-24T00:41:45.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.608+0000 7fc7d4f52640 1 -- 172.21.15.67:0/3227872635 shutdown_connections 2024-03-24T00:41:45.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.608+0000 7fc7d4f52640 1 --2- 172.21.15.67:0/3227872635 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc7b006bbd0 0x7fc7b006e090 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:45.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.608+0000 7fc7d4f52640 1 --2- 172.21.15.67:0/3227872635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7d015e2a0 0x7fc7d01f7970 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:45.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.608+0000 7fc7d4f52640 1 --2- 172.21.15.67:0/3227872635 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc7d015dec0 0x7fc7d01562d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:45.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.608+0000 7fc7d4f52640 1 --2- 172.21.15.67:0/3227872635 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc7d00d0ef0 0x7fc7d00b3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:45.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.608+0000 7fc7d4f52640 1 -- 172.21.15.67:0/3227872635 >> 172.21.15.67:0/3227872635 conn(0x7fc7d006c810 msgr2=0x7fc7d0161960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:45.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.608+0000 7fc7d4f52640 1 -- 172.21.15.67:0/3227872635 shutdown_connections 2024-03-24T00:41:45.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.608+0000 7fc7d4f52640 1 -- 172.21.15.67:0/3227872635 wait complete. 2024-03-24T00:41:45.619 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-24T00:41:45.619 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:41:45.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.720+0000 7f08a16ff640 1 Processor -- start 2024-03-24T00:41:45.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.720+0000 7f08a16ff640 1 -- start start 2024-03-24T00:41:45.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.721+0000 7f08a16ff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f089c15dec0 0x7f089c0ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:45.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.721+0000 7f08a16ff640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f089c15e2a0 0x7f089c162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:45.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.721+0000 7f08a16ff640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f089c0cf640 con 0x7f089c0d07e0 2024-03-24T00:41:45.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.721+0000 7f08a16ff640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f089c0cee30 con 0x7f089c15dec0 2024-03-24T00:41:45.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.721+0000 7f08a16ff640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f089c0d2120 con 0x7f089c15e2a0 2024-03-24T00:41:45.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.721+0000 7f089affd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f089c15dec0 0x7f089c0ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:45.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.721+0000 7f089b7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f089c0d07e0 0x7f089c0b5850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56334/0 (socket says 172.21.15.67:56334) 2024-03-24T00:41:45.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.721+0000 7f089b7fe640 1 -- 172.21.15.67:0/2759251319 learned_addr learned my addr 172.21.15.67:0/2759251319 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:41:45.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.722+0000 7f089a7fc640 1 --2- 172.21.15.67:0/2759251319 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f089c15e2a0 0x7f089c162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:45.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.722+0000 7f089affd640 1 -- 172.21.15.67:0/2759251319 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f089c15e2a0 msgr2=0x7f089c162630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:45.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.722+0000 7f089affd640 1 --2- 172.21.15.67:0/2759251319 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f089c15e2a0 0x7f089c162630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:45.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.722+0000 7f089affd640 1 -- 172.21.15.67:0/2759251319 >> v1:172.21.15.67:6789/0 conn(0x7f089c0d07e0 legacy=0x7f089c0b5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:45.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.722+0000 7f0899ffb640 1 -- 172.21.15.67:0/2759251319 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 242129799 0 0) 0x7f0894001110 con 0x7f089c0d07e0 2024-03-24T00:41:45.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.722+0000 7f089affd640 1 -- 172.21.15.67:0/2759251319 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f089c0cf080 con 0x7f089c15dec0 2024-03-24T00:41:45.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.722+0000 7f089a7fc640 1 --2- 172.21.15.67:0/2759251319 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f089c15e2a0 0x7f089c162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:41:45.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.722+0000 7f089affd640 1 --2- 172.21.15.67:0/2759251319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f089c15dec0 0x7f089c0ce630 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f089c0cefb0 tx=0x7f088c02be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e5b38c9546eddb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:45.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.723+0000 7f0899ffb640 1 -- 172.21.15.67:0/2759251319 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088c039070 con 0x7f089c15dec0 2024-03-24T00:41:45.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.723+0000 7f0899ffb640 1 -- 172.21.15.67:0/2759251319 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f088c02cbb0 con 0x7f089c15dec0 2024-03-24T00:41:45.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.723+0000 7f0899ffb640 1 -- 172.21.15.67:0/2759251319 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088c034440 con 0x7f089c15dec0 2024-03-24T00:41:45.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.723+0000 7f08a16ff640 1 -- 172.21.15.67:0/2759251319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f089c15dec0 msgr2=0x7f089c0ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:45.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.723+0000 7f08a16ff640 1 --2- 172.21.15.67:0/2759251319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f089c15dec0 0x7f089c0ce630 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f089c0cefb0 tx=0x7f088c02be00 comp rx=0 tx=0).stop 2024-03-24T00:41:45.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.723+0000 7f08a16ff640 1 -- 172.21.15.67:0/2759251319 shutdown_connections 2024-03-24T00:41:45.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.723+0000 7f08a16ff640 1 --2- 172.21.15.67:0/2759251319 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f089c15e2a0 0x7f089c162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:45.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.723+0000 7f08a16ff640 1 --2- 172.21.15.67:0/2759251319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f089c15dec0 0x7f089c0ce630 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:45.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.723+0000 7f08a16ff640 1 -- 172.21.15.67:0/2759251319 >> 172.21.15.67:0/2759251319 conn(0x7f089c06c810 msgr2=0x7f089c06cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:45.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.723+0000 7f08a16ff640 1 -- 172.21.15.67:0/2759251319 shutdown_connections 2024-03-24T00:41:45.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.723+0000 7f08a16ff640 1 -- 172.21.15.67:0/2759251319 wait complete. 2024-03-24T00:41:45.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.725+0000 7f08a16ff640 1 Processor -- start 2024-03-24T00:41:45.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.725+0000 7f08a16ff640 1 -- start start 2024-03-24T00:41:45.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.726+0000 7f08a16ff640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f089c0d07e0 0x7f089c154650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:45.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.726+0000 7f08a16ff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f089c15dec0 0x7f089c1f80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:45.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.726+0000 7f08a16ff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f089c15e2a0 0x7f089c1f3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:45.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.726+0000 7f08a16ff640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f089c0d3740 con 0x7f089c15e2a0 2024-03-24T00:41:45.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.726+0000 7f08a16ff640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f089c0d35c0 con 0x7f089c15dec0 2024-03-24T00:41:45.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.726+0000 7f08a16ff640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f089c0d38c0 con 0x7f089c0d07e0 2024-03-24T00:41:45.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.726+0000 7f089b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f089c15e2a0 0x7f089c1f3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:45.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.726+0000 7f089b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f089c15e2a0 0x7f089c1f3a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54608/0 (socket says 172.21.15.67:54608) 2024-03-24T00:41:45.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.726+0000 7f089b7fe640 1 -- 172.21.15.67:0/1285068959 learned_addr learned my addr 172.21.15.67:0/1285068959 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:45.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.726+0000 7f089b7fe640 1 -- 172.21.15.67:0/1285068959 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f089c0d07e0 msgr2=0x7f089c154650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:45.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.726+0000 7f089b7fe640 1 --2- 172.21.15.67:0/1285068959 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f089c0d07e0 0x7f089c154650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:45.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.726+0000 7f089b7fe640 1 -- 172.21.15.67:0/1285068959 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f089c15dec0 msgr2=0x7f089c1f80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:45.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.726+0000 7f089b7fe640 1 --2- 172.21.15.67:0/1285068959 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f089c15dec0 0x7f089c1f80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:45.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.726+0000 7f089b7fe640 1 -- 172.21.15.67:0/1285068959 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f089c152ba0 con 0x7f089c15e2a0 2024-03-24T00:41:45.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.727+0000 7f089b7fe640 1 --2- 172.21.15.67:0/1285068959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f089c15e2a0 0x7f089c1f3a30 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f089400c780 tx=0x7f089400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:45.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.727+0000 7f08837fe640 1 -- 172.21.15.67:0/1285068959 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0894018070 con 0x7f089c15e2a0 2024-03-24T00:41:45.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.727+0000 7f08837fe640 1 -- 172.21.15.67:0/1285068959 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08940070d0 con 0x7f089c15e2a0 2024-03-24T00:41:45.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.727+0000 7f08a16ff640 1 -- 172.21.15.67:0/1285068959 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f089c152e80 con 0x7f089c15e2a0 2024-03-24T00:41:45.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.727+0000 7f08837fe640 1 -- 172.21.15.67:0/1285068959 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08940074c0 con 0x7f089c15e2a0 2024-03-24T00:41:45.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.727+0000 7f08a16ff640 1 -- 172.21.15.67:0/1285068959 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f089c153440 con 0x7f089c15e2a0 2024-03-24T00:41:45.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.728+0000 7f08a16ff640 1 -- 172.21.15.67:0/1285068959 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f089c153640 con 0x7f089c15e2a0 2024-03-24T00:41:45.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.728+0000 7f08837fe640 1 -- 172.21.15.67:0/1285068959 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f089401b070 con 0x7f089c15e2a0 2024-03-24T00:41:45.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.728+0000 7f08837fe640 1 --2- 172.21.15.67:0/1285068959 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f088806bbd0 0x7f088806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:45.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.729+0000 7f08837fe640 1 -- 172.21.15.67:0/1285068959 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0894089e90 con 0x7f089c15e2a0 2024-03-24T00:41:45.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.729+0000 7f089affd640 1 --2- 172.21.15.67:0/1285068959 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f088806bbd0 0x7f088806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:45.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.729+0000 7f089affd640 1 --2- 172.21.15.67:0/1285068959 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f088806bbd0 0x7f088806e090 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f088c036000 tx=0x7f088c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:45.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.733+0000 7f08837fe640 1 -- 172.21.15.67:0/1285068959 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f089c153640 con 0x7f089c15e2a0 2024-03-24T00:41:45.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.913+0000 7f08a16ff640 1 -- 172.21.15.67:0/1285068959 --> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f089c04f400 con 0x7f088806bbd0 2024-03-24T00:41:45.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.919+0000 7f08837fe640 1 -- 172.21.15.67:0/1285068959 <== mgr.4105 v2:172.21.15.72:6832/2768292274 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252016 (secure 0 0 0) 0x7f089c04f400 con 0x7f088806bbd0 2024-03-24T00:41:45.922 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:41:45.923 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-24T00:41:45.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.922+0000 7f08a16ff640 1 -- 172.21.15.67:0/1285068959 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f088806bbd0 msgr2=0x7f088806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:45.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.922+0000 7f08a16ff640 1 --2- 172.21.15.67:0/1285068959 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f088806bbd0 0x7f088806e090 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f088c036000 tx=0x7f088c02e000 comp rx=0 tx=0).stop 2024-03-24T00:41:45.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.922+0000 7f08a16ff640 1 -- 172.21.15.67:0/1285068959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f089c15e2a0 msgr2=0x7f089c1f3a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:45.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.922+0000 7f08a16ff640 1 --2- 172.21.15.67:0/1285068959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f089c15e2a0 0x7f089c1f3a30 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f089400c780 tx=0x7f089400cc40 comp rx=0 tx=0).stop 2024-03-24T00:41:45.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.922+0000 7f08a16ff640 1 -- 172.21.15.67:0/1285068959 shutdown_connections 2024-03-24T00:41:45.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.922+0000 7f08a16ff640 1 --2- 172.21.15.67:0/1285068959 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f088806bbd0 0x7f088806e090 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:45.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.922+0000 7f08a16ff640 1 --2- 172.21.15.67:0/1285068959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f089c15e2a0 0x7f089c1f3a30 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:45.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.922+0000 7f08a16ff640 1 --2- 172.21.15.67:0/1285068959 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f089c15dec0 0x7f089c1f80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:45.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.922+0000 7f08a16ff640 1 --2- 172.21.15.67:0/1285068959 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f089c0d07e0 0x7f089c154650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:45.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.922+0000 7f08a16ff640 1 -- 172.21.15.67:0/1285068959 >> 172.21.15.67:0/1285068959 conn(0x7f089c06c810 msgr2=0x7f089c160600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:45.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.922+0000 7f08a16ff640 1 -- 172.21.15.67:0/1285068959 shutdown_connections 2024-03-24T00:41:45.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:45.922+0000 7f08a16ff640 1 -- 172.21.15.67:0/1285068959 wait complete. 2024-03-24T00:41:45.933 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":148,"stamp":"2024-03-24T00:41:45.360565+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":227328,"kb_used_data":6996,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754747392,"statfs":{"total":773094113280,"available":772861329408,"internally_reserved":0,"allocated":7163904,"data_stored":3844432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":225496625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.496596"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.872644+0000","last_change":"2024-03-24T00:41:42.889299+0000","last_active":"2024-03-24T00:41:43.872644+0000","last_peered":"2024-03-24T00:41:43.872644+0000","last_clean":"2024-03-24T00:41:43.872644+0000","last_became_active":"2024-03-24T00:41:42.889042+0000","last_became_peered":"2024-03-24T00:41:42.889042+0000","last_unstale":"2024-03-24T00:41:43.872644+0000","last_undegraded":"2024-03-24T00:41:43.872644+0000","last_fullsized":"2024-03-24T00:41:43.872644+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:35:28.937204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.872608+0000","last_change":"2024-03-24T00:41:42.885362+0000","last_active":"2024-03-24T00:41:43.872608+0000","last_peered":"2024-03-24T00:41:43.872608+0000","last_clean":"2024-03-24T00:41:43.872608+0000","last_became_active":"2024-03-24T00:41:42.885070+0000","last_became_peered":"2024-03-24T00:41:42.885070+0000","last_unstale":"2024-03-24T00:41:43.872608+0000","last_undegraded":"2024-03-24T00:41:43.872608+0000","last_fullsized":"2024-03-24T00:41:43.872608+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:17:59.247440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224669+0000","last_change":"2024-03-24T00:41:42.899008+0000","last_active":"2024-03-24T00:41:44.224669+0000","last_peered":"2024-03-24T00:41:44.224669+0000","last_clean":"2024-03-24T00:41:44.224669+0000","last_became_active":"2024-03-24T00:41:42.897536+0000","last_became_peered":"2024-03-24T00:41:42.897536+0000","last_unstale":"2024-03-24T00:41:44.224669+0000","last_undegraded":"2024-03-24T00:41:44.224669+0000","last_fullsized":"2024-03-24T00:41:44.224669+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:15:29.815450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.873763+0000","last_change":"2024-03-24T00:41:42.903187+0000","last_active":"2024-03-24T00:41:43.873763+0000","last_peered":"2024-03-24T00:41:43.873763+0000","last_clean":"2024-03-24T00:41:43.873763+0000","last_became_active":"2024-03-24T00:41:42.902819+0000","last_became_peered":"2024-03-24T00:41:42.902819+0000","last_unstale":"2024-03-24T00:41:43.873763+0000","last_undegraded":"2024-03-24T00:41:43.873763+0000","last_fullsized":"2024-03-24T00:41:43.873763+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:39:22.851317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224884+0000","last_change":"2024-03-24T00:41:42.890806+0000","last_active":"2024-03-24T00:41:44.224884+0000","last_peered":"2024-03-24T00:41:44.224884+0000","last_clean":"2024-03-24T00:41:44.224884+0000","last_became_active":"2024-03-24T00:41:42.890526+0000","last_became_peered":"2024-03-24T00:41:42.890526+0000","last_unstale":"2024-03-24T00:41:44.224884+0000","last_undegraded":"2024-03-24T00:41:44.224884+0000","last_fullsized":"2024-03-24T00:41:44.224884+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:27:49.707905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.225005+0000","last_change":"2024-03-24T00:41:42.896565+0000","last_active":"2024-03-24T00:41:44.225005+0000","last_peered":"2024-03-24T00:41:44.225005+0000","last_clean":"2024-03-24T00:41:44.225005+0000","last_became_active":"2024-03-24T00:41:42.896238+0000","last_became_peered":"2024-03-24T00:41:42.896238+0000","last_unstale":"2024-03-24T00:41:44.225005+0000","last_undegraded":"2024-03-24T00:41:44.225005+0000","last_fullsized":"2024-03-24T00:41:44.225005+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:16:24.700063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224537+0000","last_change":"2024-03-24T00:41:42.892632+0000","last_active":"2024-03-24T00:41:44.224537+0000","last_peered":"2024-03-24T00:41:44.224537+0000","last_clean":"2024-03-24T00:41:44.224537+0000","last_became_active":"2024-03-24T00:41:42.892349+0000","last_became_peered":"2024-03-24T00:41:42.892349+0000","last_unstale":"2024-03-24T00:41:44.224537+0000","last_undegraded":"2024-03-24T00:41:44.224537+0000","last_fullsized":"2024-03-24T00:41:44.224537+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:58:39.460940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.873292+0000","last_change":"2024-03-24T00:41:42.891893+0000","last_active":"2024-03-24T00:41:43.873292+0000","last_peered":"2024-03-24T00:41:43.873292+0000","last_clean":"2024-03-24T00:41:43.873292+0000","last_became_active":"2024-03-24T00:41:42.891635+0000","last_became_peered":"2024-03-24T00:41:42.891635+0000","last_unstale":"2024-03-24T00:41:43.873292+0000","last_undegraded":"2024-03-24T00:41:43.873292+0000","last_fullsized":"2024-03-24T00:41:43.873292+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:16:24.682756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.225223+0000","last_change":"2024-03-24T00:41:42.899252+0000","last_active":"2024-03-24T00:41:44.225223+0000","last_peered":"2024-03-24T00:41:44.225223+0000","last_clean":"2024-03-24T00:41:44.225223+0000","last_became_active":"2024-03-24T00:41:42.896380+0000","last_became_peered":"2024-03-24T00:41:42.896380+0000","last_unstale":"2024-03-24T00:41:44.225223+0000","last_undegraded":"2024-03-24T00:41:44.225223+0000","last_fullsized":"2024-03-24T00:41:44.225223+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:36:18.952935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.225096+0000","last_change":"2024-03-24T00:41:42.900879+0000","last_active":"2024-03-24T00:41:44.225096+0000","last_peered":"2024-03-24T00:41:44.225096+0000","last_clean":"2024-03-24T00:41:44.225096+0000","last_became_active":"2024-03-24T00:41:42.900633+0000","last_became_peered":"2024-03-24T00:41:42.900633+0000","last_unstale":"2024-03-24T00:41:44.225096+0000","last_undegraded":"2024-03-24T00:41:44.225096+0000","last_fullsized":"2024-03-24T00:41:44.225096+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:13:10.806112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.873566+0000","last_change":"2024-03-24T00:41:42.895957+0000","last_active":"2024-03-24T00:41:43.873566+0000","last_peered":"2024-03-24T00:41:43.873566+0000","last_clean":"2024-03-24T00:41:43.873566+0000","last_became_active":"2024-03-24T00:41:42.895681+0000","last_became_peered":"2024-03-24T00:41:42.895681+0000","last_unstale":"2024-03-24T00:41:43.873566+0000","last_undegraded":"2024-03-24T00:41:43.873566+0000","last_fullsized":"2024-03-24T00:41:43.873566+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:08:03.605976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.873768+0000","last_change":"2024-03-24T00:41:42.892526+0000","last_active":"2024-03-24T00:41:43.873768+0000","last_peered":"2024-03-24T00:41:43.873768+0000","last_clean":"2024-03-24T00:41:43.873768+0000","last_became_active":"2024-03-24T00:41:42.892275+0000","last_became_peered":"2024-03-24T00:41:42.892275+0000","last_unstale":"2024-03-24T00:41:43.873768+0000","last_undegraded":"2024-03-24T00:41:43.873768+0000","last_fullsized":"2024-03-24T00:41:43.873768+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:11:52.973533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.873439+0000","last_change":"2024-03-24T00:41:42.888506+0000","last_active":"2024-03-24T00:41:43.873439+0000","last_peered":"2024-03-24T00:41:43.873439+0000","last_clean":"2024-03-24T00:41:43.873439+0000","last_became_active":"2024-03-24T00:41:42.888237+0000","last_became_peered":"2024-03-24T00:41:42.888237+0000","last_unstale":"2024-03-24T00:41:43.873439+0000","last_undegraded":"2024-03-24T00:41:43.873439+0000","last_fullsized":"2024-03-24T00:41:43.873439+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:25:45.126263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.225188+0000","last_change":"2024-03-24T00:41:42.897079+0000","last_active":"2024-03-24T00:41:44.225188+0000","last_peered":"2024-03-24T00:41:44.225188+0000","last_clean":"2024-03-24T00:41:44.225188+0000","last_became_active":"2024-03-24T00:41:42.896802+0000","last_became_peered":"2024-03-24T00:41:42.896802+0000","last_unstale":"2024-03-24T00:41:44.225188+0000","last_undegraded":"2024-03-24T00:41:44.225188+0000","last_fullsized":"2024-03-24T00:41:44.225188+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:11:52.973533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224752+0000","last_change":"2024-03-24T00:41:42.901555+0000","last_active":"2024-03-24T00:41:44.224752+0000","last_peered":"2024-03-24T00:41:44.224752+0000","last_clean":"2024-03-24T00:41:44.224752+0000","last_became_active":"2024-03-24T00:41:42.901255+0000","last_became_peered":"2024-03-24T00:41:42.901255+0000","last_unstale":"2024-03-24T00:41:44.224752+0000","last_undegraded":"2024-03-24T00:41:44.224752+0000","last_fullsized":"2024-03-24T00:41:44.224752+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:16:24.682756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224171+0000","last_change":"2024-03-24T00:41:42.889173+0000","last_active":"2024-03-24T00:41:44.224171+0000","last_peered":"2024-03-24T00:41:44.224171+0000","last_clean":"2024-03-24T00:41:44.224171+0000","last_became_active":"2024-03-24T00:41:42.888526+0000","last_became_peered":"2024-03-24T00:41:42.888526+0000","last_unstale":"2024-03-24T00:41:44.224171+0000","last_undegraded":"2024-03-24T00:41:44.224171+0000","last_fullsized":"2024-03-24T00:41:44.224171+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:24:29.207484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.872670+0000","last_change":"2024-03-24T00:41:42.889982+0000","last_active":"2024-03-24T00:41:43.872670+0000","last_peered":"2024-03-24T00:41:43.872670+0000","last_clean":"2024-03-24T00:41:43.872670+0000","last_became_active":"2024-03-24T00:41:42.889739+0000","last_became_peered":"2024-03-24T00:41:42.889739+0000","last_unstale":"2024-03-24T00:41:43.872670+0000","last_undegraded":"2024-03-24T00:41:43.872670+0000","last_fullsized":"2024-03-24T00:41:43.872670+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:42:04.651799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.874755+0000","last_change":"2024-03-24T00:41:42.893123+0000","last_active":"2024-03-24T00:41:43.874755+0000","last_peered":"2024-03-24T00:41:43.874755+0000","last_clean":"2024-03-24T00:41:43.874755+0000","last_became_active":"2024-03-24T00:41:42.892462+0000","last_became_peered":"2024-03-24T00:41:42.892462+0000","last_unstale":"2024-03-24T00:41:43.874755+0000","last_undegraded":"2024-03-24T00:41:43.874755+0000","last_fullsized":"2024-03-24T00:41:43.874755+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:52:00.925507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.225141+0000","last_change":"2024-03-24T00:41:42.889408+0000","last_active":"2024-03-24T00:41:44.225141+0000","last_peered":"2024-03-24T00:41:44.225141+0000","last_clean":"2024-03-24T00:41:44.225141+0000","last_became_active":"2024-03-24T00:41:42.889153+0000","last_became_peered":"2024-03-24T00:41:42.889153+0000","last_unstale":"2024-03-24T00:41:44.225141+0000","last_undegraded":"2024-03-24T00:41:44.225141+0000","last_fullsized":"2024-03-24T00:41:44.225141+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:21:47.607783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224594+0000","last_change":"2024-03-24T00:41:42.891147+0000","last_active":"2024-03-24T00:41:44.224594+0000","last_peered":"2024-03-24T00:41:44.224594+0000","last_clean":"2024-03-24T00:41:44.224594+0000","last_became_active":"2024-03-24T00:41:42.890867+0000","last_became_peered":"2024-03-24T00:41:42.890867+0000","last_unstale":"2024-03-24T00:41:44.224594+0000","last_undegraded":"2024-03-24T00:41:44.224594+0000","last_fullsized":"2024-03-24T00:41:44.224594+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:51:44.721189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.873589+0000","last_change":"2024-03-24T00:41:42.881781+0000","last_active":"2024-03-24T00:41:43.873589+0000","last_peered":"2024-03-24T00:41:43.873589+0000","last_clean":"2024-03-24T00:41:43.873589+0000","last_became_active":"2024-03-24T00:41:42.881489+0000","last_became_peered":"2024-03-24T00:41:42.881489+0000","last_unstale":"2024-03-24T00:41:43.873589+0000","last_undegraded":"2024-03-24T00:41:43.873589+0000","last_fullsized":"2024-03-24T00:41:43.873589+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:13:10.806112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224523+0000","last_change":"2024-03-24T00:41:42.895887+0000","last_active":"2024-03-24T00:41:44.224523+0000","last_peered":"2024-03-24T00:41:44.224523+0000","last_clean":"2024-03-24T00:41:44.224523+0000","last_became_active":"2024-03-24T00:41:42.895495+0000","last_became_peered":"2024-03-24T00:41:42.895495+0000","last_unstale":"2024-03-24T00:41:44.224523+0000","last_undegraded":"2024-03-24T00:41:44.224523+0000","last_fullsized":"2024-03-24T00:41:44.224523+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:16:38.392703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224484+0000","last_change":"2024-03-24T00:41:42.891355+0000","last_active":"2024-03-24T00:41:44.224484+0000","last_peered":"2024-03-24T00:41:44.224484+0000","last_clean":"2024-03-24T00:41:44.224484+0000","last_became_active":"2024-03-24T00:41:42.891050+0000","last_became_peered":"2024-03-24T00:41:42.891050+0000","last_unstale":"2024-03-24T00:41:44.224484+0000","last_undegraded":"2024-03-24T00:41:44.224484+0000","last_fullsized":"2024-03-24T00:41:44.224484+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:35:15.831146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224762+0000","last_change":"2024-03-24T00:41:42.897116+0000","last_active":"2024-03-24T00:41:44.224762+0000","last_peered":"2024-03-24T00:41:44.224762+0000","last_clean":"2024-03-24T00:41:44.224762+0000","last_became_active":"2024-03-24T00:41:42.896840+0000","last_became_peered":"2024-03-24T00:41:42.896840+0000","last_unstale":"2024-03-24T00:41:44.224762+0000","last_undegraded":"2024-03-24T00:41:44.224762+0000","last_fullsized":"2024-03-24T00:41:44.224762+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:25:45.126263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.873720+0000","last_change":"2024-03-24T00:41:42.890448+0000","last_active":"2024-03-24T00:41:43.873720+0000","last_peered":"2024-03-24T00:41:43.873720+0000","last_clean":"2024-03-24T00:41:43.873720+0000","last_became_active":"2024-03-24T00:41:42.890144+0000","last_became_peered":"2024-03-24T00:41:42.890144+0000","last_unstale":"2024-03-24T00:41:43.873720+0000","last_undegraded":"2024-03-24T00:41:43.873720+0000","last_fullsized":"2024-03-24T00:41:43.873720+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:16:31.545531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.873914+0000","last_change":"2024-03-24T00:41:42.891823+0000","last_active":"2024-03-24T00:41:43.873914+0000","last_peered":"2024-03-24T00:41:43.873914+0000","last_clean":"2024-03-24T00:41:43.873914+0000","last_became_active":"2024-03-24T00:41:42.890920+0000","last_became_peered":"2024-03-24T00:41:42.890920+0000","last_unstale":"2024-03-24T00:41:43.873914+0000","last_undegraded":"2024-03-24T00:41:43.873914+0000","last_fullsized":"2024-03-24T00:41:43.873914+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:36:18.952935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.874550+0000","last_change":"2024-03-24T00:41:42.889763+0000","last_active":"2024-03-24T00:41:43.874550+0000","last_peered":"2024-03-24T00:41:43.874550+0000","last_clean":"2024-03-24T00:41:43.874550+0000","last_became_active":"2024-03-24T00:41:42.889482+0000","last_became_peered":"2024-03-24T00:41:42.889482+0000","last_unstale":"2024-03-24T00:41:43.874550+0000","last_undegraded":"2024-03-24T00:41:43.874550+0000","last_fullsized":"2024-03-24T00:41:43.874550+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:04:50.007458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.872556+0000","last_change":"2024-03-24T00:41:42.890444+0000","last_active":"2024-03-24T00:41:43.872556+0000","last_peered":"2024-03-24T00:41:43.872556+0000","last_clean":"2024-03-24T00:41:43.872556+0000","last_became_active":"2024-03-24T00:41:42.890145+0000","last_became_peered":"2024-03-24T00:41:42.890145+0000","last_unstale":"2024-03-24T00:41:43.872556+0000","last_undegraded":"2024-03-24T00:41:43.872556+0000","last_fullsized":"2024-03-24T00:41:43.872556+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:11:20.487065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224559+0000","last_change":"2024-03-24T00:41:42.899439+0000","last_active":"2024-03-24T00:41:44.224559+0000","last_peered":"2024-03-24T00:41:44.224559+0000","last_clean":"2024-03-24T00:41:44.224559+0000","last_became_active":"2024-03-24T00:41:42.897464+0000","last_became_peered":"2024-03-24T00:41:42.897464+0000","last_unstale":"2024-03-24T00:41:44.224559+0000","last_undegraded":"2024-03-24T00:41:44.224559+0000","last_fullsized":"2024-03-24T00:41:44.224559+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:15:31.777583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.873951+0000","last_change":"2024-03-24T00:41:42.891591+0000","last_active":"2024-03-24T00:41:43.873951+0000","last_peered":"2024-03-24T00:41:43.873951+0000","last_clean":"2024-03-24T00:41:43.873951+0000","last_became_active":"2024-03-24T00:41:42.891297+0000","last_became_peered":"2024-03-24T00:41:42.891297+0000","last_unstale":"2024-03-24T00:41:43.873951+0000","last_undegraded":"2024-03-24T00:41:43.873951+0000","last_fullsized":"2024-03-24T00:41:43.873951+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:59:25.755135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.874397+0000","last_change":"2024-03-24T00:41:42.888895+0000","last_active":"2024-03-24T00:41:43.874397+0000","last_peered":"2024-03-24T00:41:43.874397+0000","last_clean":"2024-03-24T00:41:43.874397+0000","last_became_active":"2024-03-24T00:41:42.888648+0000","last_became_peered":"2024-03-24T00:41:42.888648+0000","last_unstale":"2024-03-24T00:41:43.874397+0000","last_undegraded":"2024-03-24T00:41:43.874397+0000","last_fullsized":"2024-03-24T00:41:43.874397+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:41:48.723391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224389+0000","last_change":"2024-03-24T00:41:42.891152+0000","last_active":"2024-03-24T00:41:44.224389+0000","last_peered":"2024-03-24T00:41:44.224389+0000","last_clean":"2024-03-24T00:41:44.224389+0000","last_became_active":"2024-03-24T00:41:42.890890+0000","last_became_peered":"2024-03-24T00:41:42.890890+0000","last_unstale":"2024-03-24T00:41:44.224389+0000","last_undegraded":"2024-03-24T00:41:44.224389+0000","last_fullsized":"2024-03-24T00:41:44.224389+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:36:18.952935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224680+0000","last_change":"2024-03-24T00:41:42.896417+0000","last_active":"2024-03-24T00:41:44.224680+0000","last_peered":"2024-03-24T00:41:44.224680+0000","last_clean":"2024-03-24T00:41:44.224680+0000","last_became_active":"2024-03-24T00:41:42.895327+0000","last_became_peered":"2024-03-24T00:41:42.895327+0000","last_unstale":"2024-03-24T00:41:44.224680+0000","last_undegraded":"2024-03-24T00:41:44.224680+0000","last_fullsized":"2024-03-24T00:41:44.224680+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:39:22.851317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"61'4","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.872805+0000","last_change":"2024-03-24T00:41:41.884745+0000","last_active":"2024-03-24T00:41:43.872805+0000","last_peered":"2024-03-24T00:41:43.872805+0000","last_clean":"2024-03-24T00:41:43.872805+0000","last_became_active":"2024-03-24T00:41:41.884498+0000","last_became_peered":"2024-03-24T00:41:41.884498+0000","last_unstale":"2024-03-24T00:41:43.872805+0000","last_undegraded":"2024-03-24T00:41:43.872805+0000","last_fullsized":"2024-03-24T00:41:43.872805+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:24:35.066275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.874817+0000","last_change":"2024-03-24T00:41:42.894240+0000","last_active":"2024-03-24T00:41:43.874817+0000","last_peered":"2024-03-24T00:41:43.874817+0000","last_clean":"2024-03-24T00:41:43.874817+0000","last_became_active":"2024-03-24T00:41:42.893817+0000","last_became_peered":"2024-03-24T00:41:42.893817+0000","last_unstale":"2024-03-24T00:41:43.874817+0000","last_undegraded":"2024-03-24T00:41:43.874817+0000","last_fullsized":"2024-03-24T00:41:43.874817+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:32:00.196741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"61'1","reported_seq":15,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224940+0000","last_change":"2024-03-24T00:41:41.884801+0000","last_active":"2024-03-24T00:41:44.224940+0000","last_peered":"2024-03-24T00:41:44.224940+0000","last_clean":"2024-03-24T00:41:44.224940+0000","last_became_active":"2024-03-24T00:41:41.884477+0000","last_became_peered":"2024-03-24T00:41:41.884477+0000","last_unstale":"2024-03-24T00:41:44.224940+0000","last_undegraded":"2024-03-24T00:41:44.224940+0000","last_fullsized":"2024-03-24T00:41:44.224940+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:15:28.811466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224772+0000","last_change":"2024-03-24T00:41:42.893799+0000","last_active":"2024-03-24T00:41:44.224772+0000","last_peered":"2024-03-24T00:41:44.224772+0000","last_clean":"2024-03-24T00:41:44.224772+0000","last_became_active":"2024-03-24T00:41:42.893524+0000","last_became_peered":"2024-03-24T00:41:42.893524+0000","last_unstale":"2024-03-24T00:41:44.224772+0000","last_undegraded":"2024-03-24T00:41:44.224772+0000","last_fullsized":"2024-03-24T00:41:44.224772+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:22:12.038404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"61'2","reported_seq":16,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224421+0000","last_change":"2024-03-24T00:41:41.885417+0000","last_active":"2024-03-24T00:41:44.224421+0000","last_peered":"2024-03-24T00:41:44.224421+0000","last_clean":"2024-03-24T00:41:44.224421+0000","last_became_active":"2024-03-24T00:41:41.885125+0000","last_became_peered":"2024-03-24T00:41:41.885125+0000","last_unstale":"2024-03-24T00:41:44.224421+0000","last_undegraded":"2024-03-24T00:41:44.224421+0000","last_fullsized":"2024-03-24T00:41:44.224421+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:35:30.712724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.225120+0000","last_change":"2024-03-24T00:41:42.903357+0000","last_active":"2024-03-24T00:41:44.225120+0000","last_peered":"2024-03-24T00:41:44.225120+0000","last_clean":"2024-03-24T00:41:44.225120+0000","last_became_active":"2024-03-24T00:41:42.902782+0000","last_became_peered":"2024-03-24T00:41:42.902782+0000","last_unstale":"2024-03-24T00:41:44.225120+0000","last_undegraded":"2024-03-24T00:41:44.225120+0000","last_fullsized":"2024-03-24T00:41:44.225120+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:16:31.545531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"61'2","reported_seq":16,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.225196+0000","last_change":"2024-03-24T00:41:41.880628+0000","last_active":"2024-03-24T00:41:44.225196+0000","last_peered":"2024-03-24T00:41:44.225196+0000","last_clean":"2024-03-24T00:41:44.225196+0000","last_became_active":"2024-03-24T00:41:41.880334+0000","last_became_peered":"2024-03-24T00:41:41.880334+0000","last_unstale":"2024-03-24T00:41:44.225196+0000","last_undegraded":"2024-03-24T00:41:44.225196+0000","last_fullsized":"2024-03-24T00:41:44.225196+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:59:22.376959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224302+0000","last_change":"2024-03-24T00:41:42.889436+0000","last_active":"2024-03-24T00:41:44.224302+0000","last_peered":"2024-03-24T00:41:44.224302+0000","last_clean":"2024-03-24T00:41:44.224302+0000","last_became_active":"2024-03-24T00:41:42.888826+0000","last_became_peered":"2024-03-24T00:41:42.888826+0000","last_unstale":"2024-03-24T00:41:44.224302+0000","last_undegraded":"2024-03-24T00:41:44.224302+0000","last_fullsized":"2024-03-24T00:41:44.224302+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:05:53.078237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"61'1","reported_seq":15,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224232+0000","last_change":"2024-03-24T00:41:41.881319+0000","last_active":"2024-03-24T00:41:44.224232+0000","last_peered":"2024-03-24T00:41:44.224232+0000","last_clean":"2024-03-24T00:41:44.224232+0000","last_became_active":"2024-03-24T00:41:41.880598+0000","last_became_peered":"2024-03-24T00:41:41.880598+0000","last_unstale":"2024-03-24T00:41:44.224232+0000","last_undegraded":"2024-03-24T00:41:44.224232+0000","last_fullsized":"2024-03-24T00:41:44.224232+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:59:24.751151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.872289+0000","last_change":"2024-03-24T00:41:42.893623+0000","last_active":"2024-03-24T00:41:43.872289+0000","last_peered":"2024-03-24T00:41:43.872289+0000","last_clean":"2024-03-24T00:41:43.872289+0000","last_became_active":"2024-03-24T00:41:42.893372+0000","last_became_peered":"2024-03-24T00:41:42.893372+0000","last_unstale":"2024-03-24T00:41:43.872289+0000","last_undegraded":"2024-03-24T00:41:43.872289+0000","last_fullsized":"2024-03-24T00:41:43.872289+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:31:37.065977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"61'1","reported_seq":15,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224141+0000","last_change":"2024-03-24T00:41:41.882741+0000","last_active":"2024-03-24T00:41:44.224141+0000","last_peered":"2024-03-24T00:41:44.224141+0000","last_clean":"2024-03-24T00:41:44.224141+0000","last_became_active":"2024-03-24T00:41:41.882453+0000","last_became_peered":"2024-03-24T00:41:41.882453+0000","last_unstale":"2024-03-24T00:41:44.224141+0000","last_undegraded":"2024-03-24T00:41:44.224141+0000","last_fullsized":"2024-03-24T00:41:44.224141+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:16:30.541548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224961+0000","last_change":"2024-03-24T00:41:42.885195+0000","last_active":"2024-03-24T00:41:44.224961+0000","last_peered":"2024-03-24T00:41:44.224961+0000","last_clean":"2024-03-24T00:41:44.224961+0000","last_became_active":"2024-03-24T00:41:42.884880+0000","last_became_peered":"2024-03-24T00:41:42.884880+0000","last_unstale":"2024-03-24T00:41:44.224961+0000","last_undegraded":"2024-03-24T00:41:44.224961+0000","last_fullsized":"2024-03-24T00:41:44.224961+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:24:36.070259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":14,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224342+0000","last_change":"2024-03-24T00:41:41.891261+0000","last_active":"2024-03-24T00:41:44.224342+0000","last_peered":"2024-03-24T00:41:44.224342+0000","last_clean":"2024-03-24T00:41:44.224342+0000","last_became_active":"2024-03-24T00:41:41.891006+0000","last_became_peered":"2024-03-24T00:41:41.891006+0000","last_unstale":"2024-03-24T00:41:44.224342+0000","last_undegraded":"2024-03-24T00:41:44.224342+0000","last_fullsized":"2024-03-24T00:41:44.224342+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:25:44.122279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.225131+0000","last_change":"2024-03-24T00:41:42.891260+0000","last_active":"2024-03-24T00:41:44.225131+0000","last_peered":"2024-03-24T00:41:44.225131+0000","last_clean":"2024-03-24T00:41:44.225131+0000","last_became_active":"2024-03-24T00:41:42.890972+0000","last_became_peered":"2024-03-24T00:41:42.890972+0000","last_unstale":"2024-03-24T00:41:44.225131+0000","last_undegraded":"2024-03-24T00:41:44.225131+0000","last_fullsized":"2024-03-24T00:41:44.225131+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:00:07.007109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":14,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224162+0000","last_change":"2024-03-24T00:41:41.889322+0000","last_active":"2024-03-24T00:41:44.224162+0000","last_peered":"2024-03-24T00:41:44.224162+0000","last_clean":"2024-03-24T00:41:44.224162+0000","last_became_active":"2024-03-24T00:41:41.889045+0000","last_became_peered":"2024-03-24T00:41:41.889045+0000","last_unstale":"2024-03-24T00:41:44.224162+0000","last_undegraded":"2024-03-24T00:41:44.224162+0000","last_fullsized":"2024-03-24T00:41:44.224162+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:15:07.171930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.872627+0000","last_change":"2024-03-24T00:41:42.894710+0000","last_active":"2024-03-24T00:41:43.872627+0000","last_peered":"2024-03-24T00:41:43.872627+0000","last_clean":"2024-03-24T00:41:43.872627+0000","last_became_active":"2024-03-24T00:41:42.894449+0000","last_became_peered":"2024-03-24T00:41:42.894449+0000","last_unstale":"2024-03-24T00:41:43.872627+0000","last_undegraded":"2024-03-24T00:41:43.872627+0000","last_fullsized":"2024-03-24T00:41:43.872627+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:07:39.772752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":14,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.225223+0000","last_change":"2024-03-24T00:41:41.883456+0000","last_active":"2024-03-24T00:41:44.225223+0000","last_peered":"2024-03-24T00:41:44.225223+0000","last_clean":"2024-03-24T00:41:44.225223+0000","last_became_active":"2024-03-24T00:41:41.883173+0000","last_became_peered":"2024-03-24T00:41:41.883173+0000","last_unstale":"2024-03-24T00:41:44.225223+0000","last_undegraded":"2024-03-24T00:41:44.225223+0000","last_fullsized":"2024-03-24T00:41:44.225223+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:31:36.061994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224560+0000","last_change":"2024-03-24T00:41:42.877552+0000","last_active":"2024-03-24T00:41:44.224560+0000","last_peered":"2024-03-24T00:41:44.224560+0000","last_clean":"2024-03-24T00:41:44.224560+0000","last_became_active":"2024-03-24T00:41:42.877236+0000","last_became_peered":"2024-03-24T00:41:42.877236+0000","last_unstale":"2024-03-24T00:41:44.224560+0000","last_undegraded":"2024-03-24T00:41:44.224560+0000","last_fullsized":"2024-03-24T00:41:44.224560+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:55:15.958279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":14,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.874864+0000","last_change":"2024-03-24T00:41:41.888178+0000","last_active":"2024-03-24T00:41:43.874864+0000","last_peered":"2024-03-24T00:41:43.874864+0000","last_clean":"2024-03-24T00:41:43.874864+0000","last_became_active":"2024-03-24T00:41:41.887657+0000","last_became_peered":"2024-03-24T00:41:41.887657+0000","last_unstale":"2024-03-24T00:41:43.874864+0000","last_undegraded":"2024-03-24T00:41:43.874864+0000","last_fullsized":"2024-03-24T00:41:43.874864+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:19:05.072568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224890+0000","last_change":"2024-03-24T00:41:42.887646+0000","last_active":"2024-03-24T00:41:44.224890+0000","last_peered":"2024-03-24T00:41:44.224890+0000","last_clean":"2024-03-24T00:41:44.224890+0000","last_became_active":"2024-03-24T00:41:42.887366+0000","last_became_peered":"2024-03-24T00:41:42.887366+0000","last_unstale":"2024-03-24T00:41:44.224890+0000","last_undegraded":"2024-03-24T00:41:44.224890+0000","last_fullsized":"2024-03-24T00:41:44.224890+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:04:50.007458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"61'3","reported_seq":17,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224643+0000","last_change":"2024-03-24T00:41:41.884985+0000","last_active":"2024-03-24T00:41:44.224643+0000","last_peered":"2024-03-24T00:41:44.224643+0000","last_clean":"2024-03-24T00:41:44.224643+0000","last_became_active":"2024-03-24T00:41:41.883972+0000","last_became_peered":"2024-03-24T00:41:41.883972+0000","last_unstale":"2024-03-24T00:41:44.224643+0000","last_undegraded":"2024-03-24T00:41:44.224643+0000","last_fullsized":"2024-03-24T00:41:44.224643+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:07:24.703574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.874613+0000","last_change":"2024-03-24T00:41:42.891388+0000","last_active":"2024-03-24T00:41:43.874613+0000","last_peered":"2024-03-24T00:41:43.874613+0000","last_clean":"2024-03-24T00:41:43.874613+0000","last_became_active":"2024-03-24T00:41:42.891132+0000","last_became_peered":"2024-03-24T00:41:42.891132+0000","last_unstale":"2024-03-24T00:41:43.874613+0000","last_undegraded":"2024-03-24T00:41:43.874613+0000","last_fullsized":"2024-03-24T00:41:43.874613+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:24:36.070259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"61'6","reported_seq":20,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.872670+0000","last_change":"2024-03-24T00:41:41.890234+0000","last_active":"2024-03-24T00:41:43.872670+0000","last_peered":"2024-03-24T00:41:43.872670+0000","last_clean":"2024-03-24T00:41:43.872670+0000","last_became_active":"2024-03-24T00:41:41.889928+0000","last_became_peered":"2024-03-24T00:41:41.889928+0000","last_unstale":"2024-03-24T00:41:43.872670+0000","last_undegraded":"2024-03-24T00:41:43.872670+0000","last_fullsized":"2024-03-24T00:41:43.872670+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:04:49.003475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.873456+0000","last_change":"2024-03-24T00:41:42.896893+0000","last_active":"2024-03-24T00:41:43.873456+0000","last_peered":"2024-03-24T00:41:43.873456+0000","last_clean":"2024-03-24T00:41:43.873456+0000","last_became_active":"2024-03-24T00:41:42.896616+0000","last_became_peered":"2024-03-24T00:41:42.896616+0000","last_unstale":"2024-03-24T00:41:43.873456+0000","last_undegraded":"2024-03-24T00:41:43.873456+0000","last_fullsized":"2024-03-24T00:41:43.873456+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:42:34.765808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"61'2","reported_seq":16,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224506+0000","last_change":"2024-03-24T00:41:41.891160+0000","last_active":"2024-03-24T00:41:44.224506+0000","last_peered":"2024-03-24T00:41:44.224506+0000","last_clean":"2024-03-24T00:41:44.224506+0000","last_became_active":"2024-03-24T00:41:41.890904+0000","last_became_peered":"2024-03-24T00:41:41.890904+0000","last_unstale":"2024-03-24T00:41:44.224506+0000","last_undegraded":"2024-03-24T00:41:44.224506+0000","last_fullsized":"2024-03-24T00:41:44.224506+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:27:12.596556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.874215+0000","last_change":"2024-03-24T00:41:42.882382+0000","last_active":"2024-03-24T00:41:43.874215+0000","last_peered":"2024-03-24T00:41:43.874215+0000","last_clean":"2024-03-24T00:41:43.874215+0000","last_became_active":"2024-03-24T00:41:42.882105+0000","last_became_peered":"2024-03-24T00:41:42.882105+0000","last_unstale":"2024-03-24T00:41:43.874215+0000","last_undegraded":"2024-03-24T00:41:43.874215+0000","last_fullsized":"2024-03-24T00:41:43.874215+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:16:24.700063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":14,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.874072+0000","last_change":"2024-03-24T00:41:41.882297+0000","last_active":"2024-03-24T00:41:43.874072+0000","last_peered":"2024-03-24T00:41:43.874072+0000","last_clean":"2024-03-24T00:41:43.874072+0000","last_became_active":"2024-03-24T00:41:41.882002+0000","last_became_peered":"2024-03-24T00:41:41.882002+0000","last_unstale":"2024-03-24T00:41:43.874072+0000","last_undegraded":"2024-03-24T00:41:43.874072+0000","last_fullsized":"2024-03-24T00:41:43.874072+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:31:36.061994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224268+0000","last_change":"2024-03-24T00:41:42.888607+0000","last_active":"2024-03-24T00:41:44.224268+0000","last_peered":"2024-03-24T00:41:44.224268+0000","last_clean":"2024-03-24T00:41:44.224268+0000","last_became_active":"2024-03-24T00:41:42.888351+0000","last_became_peered":"2024-03-24T00:41:42.888351+0000","last_unstale":"2024-03-24T00:41:44.224268+0000","last_undegraded":"2024-03-24T00:41:44.224268+0000","last_fullsized":"2024-03-24T00:41:44.224268+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:42:34.765808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":14,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224924+0000","last_change":"2024-03-24T00:41:41.888499+0000","last_active":"2024-03-24T00:41:44.224924+0000","last_peered":"2024-03-24T00:41:44.224924+0000","last_clean":"2024-03-24T00:41:44.224924+0000","last_became_active":"2024-03-24T00:41:41.888212+0000","last_became_peered":"2024-03-24T00:41:41.888212+0000","last_unstale":"2024-03-24T00:41:44.224924+0000","last_undegraded":"2024-03-24T00:41:44.224924+0000","last_fullsized":"2024-03-24T00:41:44.224924+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:33:27.374739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.872462+0000","last_change":"2024-03-24T00:41:42.900253+0000","last_active":"2024-03-24T00:41:43.872462+0000","last_peered":"2024-03-24T00:41:43.872462+0000","last_clean":"2024-03-24T00:41:43.872462+0000","last_became_active":"2024-03-24T00:41:42.899899+0000","last_became_peered":"2024-03-24T00:41:42.899899+0000","last_unstale":"2024-03-24T00:41:43.872462+0000","last_undegraded":"2024-03-24T00:41:43.872462+0000","last_fullsized":"2024-03-24T00:41:43.872462+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:02:34.690937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"61'8","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.873577+0000","last_change":"2024-03-24T00:41:41.886878+0000","last_active":"2024-03-24T00:41:43.873577+0000","last_peered":"2024-03-24T00:41:43.873577+0000","last_clean":"2024-03-24T00:41:43.873577+0000","last_became_active":"2024-03-24T00:41:41.886581+0000","last_became_peered":"2024-03-24T00:41:41.886581+0000","last_unstale":"2024-03-24T00:41:43.873577+0000","last_undegraded":"2024-03-24T00:41:43.873577+0000","last_fullsized":"2024-03-24T00:41:43.873577+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:45:35.636179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224626+0000","last_change":"2024-03-24T00:41:42.895413+0000","last_active":"2024-03-24T00:41:44.224626+0000","last_peered":"2024-03-24T00:41:44.224626+0000","last_clean":"2024-03-24T00:41:44.224626+0000","last_became_active":"2024-03-24T00:41:42.895137+0000","last_became_peered":"2024-03-24T00:41:42.895137+0000","last_unstale":"2024-03-24T00:41:44.224626+0000","last_undegraded":"2024-03-24T00:41:44.224626+0000","last_fullsized":"2024-03-24T00:41:44.224626+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:59:23.380943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"61'1","reported_seq":15,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.873836+0000","last_change":"2024-03-24T00:41:41.885287+0000","last_active":"2024-03-24T00:41:43.873836+0000","last_peered":"2024-03-24T00:41:43.873836+0000","last_clean":"2024-03-24T00:41:43.873836+0000","last_became_active":"2024-03-24T00:41:41.884901+0000","last_became_peered":"2024-03-24T00:41:41.884901+0000","last_unstale":"2024-03-24T00:41:43.873836+0000","last_undegraded":"2024-03-24T00:41:43.873836+0000","last_fullsized":"2024-03-24T00:41:43.873836+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:02:33.686954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.225273+0000","last_change":"2024-03-24T00:41:42.890724+0000","last_active":"2024-03-24T00:41:44.225273+0000","last_peered":"2024-03-24T00:41:44.225273+0000","last_clean":"2024-03-24T00:41:44.225273+0000","last_became_active":"2024-03-24T00:41:42.890461+0000","last_became_peered":"2024-03-24T00:41:42.890461+0000","last_unstale":"2024-03-24T00:41:44.225273+0000","last_undegraded":"2024-03-24T00:41:44.225273+0000","last_fullsized":"2024-03-24T00:41:44.225273+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:28:13.579193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"61'2","reported_seq":16,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.225090+0000","last_change":"2024-03-24T00:41:41.888640+0000","last_active":"2024-03-24T00:41:44.225090+0000","last_peered":"2024-03-24T00:41:44.225090+0000","last_clean":"2024-03-24T00:41:44.225090+0000","last_became_active":"2024-03-24T00:41:41.888316+0000","last_became_peered":"2024-03-24T00:41:41.888316+0000","last_unstale":"2024-03-24T00:41:44.225090+0000","last_undegraded":"2024-03-24T00:41:44.225090+0000","last_fullsized":"2024-03-24T00:41:44.225090+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:25:09.471037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.874757+0000","last_change":"2024-03-24T00:41:42.893352+0000","last_active":"2024-03-24T00:41:43.874757+0000","last_peered":"2024-03-24T00:41:43.874757+0000","last_clean":"2024-03-24T00:41:43.874757+0000","last_became_active":"2024-03-24T00:41:42.892808+0000","last_became_peered":"2024-03-24T00:41:42.892808+0000","last_unstale":"2024-03-24T00:41:43.874757+0000","last_undegraded":"2024-03-24T00:41:43.874757+0000","last_fullsized":"2024-03-24T00:41:43.874757+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:21:47.607783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"61'1","reported_seq":15,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224948+0000","last_change":"2024-03-24T00:41:41.887827+0000","last_active":"2024-03-24T00:41:44.224948+0000","last_peered":"2024-03-24T00:41:44.224948+0000","last_clean":"2024-03-24T00:41:44.224948+0000","last_became_active":"2024-03-24T00:41:41.887507+0000","last_became_peered":"2024-03-24T00:41:41.887507+0000","last_unstale":"2024-03-24T00:41:44.224948+0000","last_undegraded":"2024-03-24T00:41:44.224948+0000","last_fullsized":"2024-03-24T00:41:44.224948+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:49:35.492413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.225067+0000","last_change":"2024-03-24T00:41:42.898105+0000","last_active":"2024-03-24T00:41:44.225067+0000","last_peered":"2024-03-24T00:41:44.225067+0000","last_clean":"2024-03-24T00:41:44.225067+0000","last_became_active":"2024-03-24T00:41:42.897801+0000","last_became_peered":"2024-03-24T00:41:42.897801+0000","last_unstale":"2024-03-24T00:41:44.225067+0000","last_undegraded":"2024-03-24T00:41:44.225067+0000","last_fullsized":"2024-03-24T00:41:44.225067+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:52:00.925507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":14,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224471+0000","last_change":"2024-03-24T00:41:41.888772+0000","last_active":"2024-03-24T00:41:44.224471+0000","last_peered":"2024-03-24T00:41:44.224471+0000","last_clean":"2024-03-24T00:41:44.224471+0000","last_became_active":"2024-03-24T00:41:41.888403+0000","last_became_peered":"2024-03-24T00:41:41.888403+0000","last_unstale":"2024-03-24T00:41:44.224471+0000","last_undegraded":"2024-03-24T00:41:44.224471+0000","last_fullsized":"2024-03-24T00:41:44.224471+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:11:51.969549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224532+0000","last_change":"2024-03-24T00:41:42.891846+0000","last_active":"2024-03-24T00:41:44.224532+0000","last_peered":"2024-03-24T00:41:44.224532+0000","last_clean":"2024-03-24T00:41:44.224532+0000","last_became_active":"2024-03-24T00:41:42.891593+0000","last_became_peered":"2024-03-24T00:41:42.891593+0000","last_unstale":"2024-03-24T00:41:44.224532+0000","last_undegraded":"2024-03-24T00:41:44.224532+0000","last_fullsized":"2024-03-24T00:41:44.224532+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:59:25.755135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"61'2","reported_seq":16,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224141+0000","last_change":"2024-03-24T00:41:41.890217+0000","last_active":"2024-03-24T00:41:44.224141+0000","last_peered":"2024-03-24T00:41:44.224141+0000","last_clean":"2024-03-24T00:41:44.224141+0000","last_became_active":"2024-03-24T00:41:41.889876+0000","last_became_peered":"2024-03-24T00:41:41.889876+0000","last_unstale":"2024-03-24T00:41:44.224141+0000","last_undegraded":"2024-03-24T00:41:44.224141+0000","last_fullsized":"2024-03-24T00:41:44.224141+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:57:54.901758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.874324+0000","last_change":"2024-03-24T00:41:42.890665+0000","last_active":"2024-03-24T00:41:43.874324+0000","last_peered":"2024-03-24T00:41:43.874324+0000","last_clean":"2024-03-24T00:41:43.874324+0000","last_became_active":"2024-03-24T00:41:42.890404+0000","last_became_peered":"2024-03-24T00:41:42.890404+0000","last_unstale":"2024-03-24T00:41:43.874324+0000","last_undegraded":"2024-03-24T00:41:43.874324+0000","last_fullsized":"2024-03-24T00:41:43.874324+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:49:44.091221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":14,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.874180+0000","last_change":"2024-03-24T00:41:41.879431+0000","last_active":"2024-03-24T00:41:43.874180+0000","last_peered":"2024-03-24T00:41:43.874180+0000","last_clean":"2024-03-24T00:41:43.874180+0000","last_became_active":"2024-03-24T00:41:41.879169+0000","last_became_peered":"2024-03-24T00:41:41.879169+0000","last_unstale":"2024-03-24T00:41:43.874180+0000","last_undegraded":"2024-03-24T00:41:43.874180+0000","last_fullsized":"2024-03-24T00:41:43.874180+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:07:24.703574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224560+0000","last_change":"2024-03-24T00:41:42.884629+0000","last_active":"2024-03-24T00:41:44.224560+0000","last_peered":"2024-03-24T00:41:44.224560+0000","last_clean":"2024-03-24T00:41:44.224560+0000","last_became_active":"2024-03-24T00:41:42.884284+0000","last_became_peered":"2024-03-24T00:41:42.884284+0000","last_unstale":"2024-03-24T00:41:44.224560+0000","last_undegraded":"2024-03-24T00:41:44.224560+0000","last_fullsized":"2024-03-24T00:41:44.224560+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:08:03.605976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":14,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224645+0000","last_change":"2024-03-24T00:41:41.886088+0000","last_active":"2024-03-24T00:41:44.224645+0000","last_peered":"2024-03-24T00:41:44.224645+0000","last_clean":"2024-03-24T00:41:44.224645+0000","last_became_active":"2024-03-24T00:41:41.885794+0000","last_became_peered":"2024-03-24T00:41:41.885794+0000","last_unstale":"2024-03-24T00:41:44.224645+0000","last_undegraded":"2024-03-24T00:41:44.224645+0000","last_fullsized":"2024-03-24T00:41:44.224645+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:29:01.056209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.225163+0000","last_change":"2024-03-24T00:41:42.892702+0000","last_active":"2024-03-24T00:41:44.225163+0000","last_peered":"2024-03-24T00:41:44.225163+0000","last_clean":"2024-03-24T00:41:44.225163+0000","last_became_active":"2024-03-24T00:41:42.892416+0000","last_became_peered":"2024-03-24T00:41:42.892416+0000","last_unstale":"2024-03-24T00:41:44.225163+0000","last_undegraded":"2024-03-24T00:41:44.225163+0000","last_fullsized":"2024-03-24T00:41:44.225163+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:49:44.091221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"61'2","reported_seq":16,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.874058+0000","last_change":"2024-03-24T00:41:41.884561+0000","last_active":"2024-03-24T00:41:43.874058+0000","last_peered":"2024-03-24T00:41:43.874058+0000","last_clean":"2024-03-24T00:41:43.874058+0000","last_became_active":"2024-03-24T00:41:41.884274+0000","last_became_peered":"2024-03-24T00:41:41.884274+0000","last_unstale":"2024-03-24T00:41:43.874058+0000","last_undegraded":"2024-03-24T00:41:43.874058+0000","last_fullsized":"2024-03-24T00:41:43.874058+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:11:19.483081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224915+0000","last_change":"2024-03-24T00:41:42.886841+0000","last_active":"2024-03-24T00:41:44.224915+0000","last_peered":"2024-03-24T00:41:44.224915+0000","last_clean":"2024-03-24T00:41:44.224915+0000","last_became_active":"2024-03-24T00:41:42.886567+0000","last_became_peered":"2024-03-24T00:41:42.886567+0000","last_unstale":"2024-03-24T00:41:44.224915+0000","last_undegraded":"2024-03-24T00:41:44.224915+0000","last_fullsized":"2024-03-24T00:41:44.224915+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:17:54.514321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":14,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.874199+0000","last_change":"2024-03-24T00:41:41.881300+0000","last_active":"2024-03-24T00:41:43.874199+0000","last_peered":"2024-03-24T00:41:43.874199+0000","last_clean":"2024-03-24T00:41:43.874199+0000","last_became_active":"2024-03-24T00:41:41.880759+0000","last_became_peered":"2024-03-24T00:41:41.880759+0000","last_unstale":"2024-03-24T00:41:43.874199+0000","last_undegraded":"2024-03-24T00:41:43.874199+0000","last_fullsized":"2024-03-24T00:41:43.874199+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:25:09.471037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.874119+0000","last_change":"2024-03-24T00:41:42.881823+0000","last_active":"2024-03-24T00:41:43.874119+0000","last_peered":"2024-03-24T00:41:43.874119+0000","last_clean":"2024-03-24T00:41:43.874119+0000","last_became_active":"2024-03-24T00:41:42.881550+0000","last_became_peered":"2024-03-24T00:41:42.881550+0000","last_unstale":"2024-03-24T00:41:43.874119+0000","last_undegraded":"2024-03-24T00:41:43.874119+0000","last_fullsized":"2024-03-24T00:41:43.874119+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:22:12.038404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":14,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224511+0000","last_change":"2024-03-24T00:41:41.886155+0000","last_active":"2024-03-24T00:41:44.224511+0000","last_peered":"2024-03-24T00:41:44.224511+0000","last_clean":"2024-03-24T00:41:44.224511+0000","last_became_active":"2024-03-24T00:41:41.885881+0000","last_became_peered":"2024-03-24T00:41:41.885881+0000","last_unstale":"2024-03-24T00:41:44.224511+0000","last_undegraded":"2024-03-24T00:41:44.224511+0000","last_fullsized":"2024-03-24T00:41:44.224511+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:15:30.773599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.225208+0000","last_change":"2024-03-24T00:41:42.894371+0000","last_active":"2024-03-24T00:41:44.225208+0000","last_peered":"2024-03-24T00:41:44.225208+0000","last_clean":"2024-03-24T00:41:44.225208+0000","last_became_active":"2024-03-24T00:41:42.894094+0000","last_became_peered":"2024-03-24T00:41:42.894094+0000","last_unstale":"2024-03-24T00:41:44.225208+0000","last_undegraded":"2024-03-24T00:41:44.225208+0000","last_fullsized":"2024-03-24T00:41:44.225208+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:32:00.196741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":14,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.872726+0000","last_change":"2024-03-24T00:41:41.890230+0000","last_active":"2024-03-24T00:41:43.872726+0000","last_peered":"2024-03-24T00:41:43.872726+0000","last_clean":"2024-03-24T00:41:43.872726+0000","last_became_active":"2024-03-24T00:41:41.889973+0000","last_became_peered":"2024-03-24T00:41:41.889973+0000","last_unstale":"2024-03-24T00:41:43.872726+0000","last_undegraded":"2024-03-24T00:41:43.872726+0000","last_fullsized":"2024-03-24T00:41:43.872726+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:51:59.921523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224739+0000","last_change":"2024-03-24T00:41:42.883352+0000","last_active":"2024-03-24T00:41:44.224739+0000","last_peered":"2024-03-24T00:41:44.224739+0000","last_clean":"2024-03-24T00:41:44.224739+0000","last_became_active":"2024-03-24T00:41:42.883073+0000","last_became_peered":"2024-03-24T00:41:42.883073+0000","last_unstale":"2024-03-24T00:41:44.224739+0000","last_undegraded":"2024-03-24T00:41:44.224739+0000","last_fullsized":"2024-03-24T00:41:44.224739+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:41:48.723391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":14,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224807+0000","last_change":"2024-03-24T00:41:41.886176+0000","last_active":"2024-03-24T00:41:44.224807+0000","last_peered":"2024-03-24T00:41:44.224807+0000","last_clean":"2024-03-24T00:41:44.224807+0000","last_became_active":"2024-03-24T00:41:41.885907+0000","last_became_peered":"2024-03-24T00:41:41.885907+0000","last_unstale":"2024-03-24T00:41:44.224807+0000","last_undegraded":"2024-03-24T00:41:44.224807+0000","last_fullsized":"2024-03-24T00:41:44.224807+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:58:09.746715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.874453+0000","last_change":"2024-03-24T00:41:42.896835+0000","last_active":"2024-03-24T00:41:43.874453+0000","last_peered":"2024-03-24T00:41:43.874453+0000","last_clean":"2024-03-24T00:41:43.874453+0000","last_became_active":"2024-03-24T00:41:42.896239+0000","last_became_peered":"2024-03-24T00:41:42.896239+0000","last_unstale":"2024-03-24T00:41:43.874453+0000","last_undegraded":"2024-03-24T00:41:43.874453+0000","last_fullsized":"2024-03-24T00:41:43.874453+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:27:49.707905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":14,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224241+0000","last_change":"2024-03-24T00:41:41.883441+0000","last_active":"2024-03-24T00:41:44.224241+0000","last_peered":"2024-03-24T00:41:44.224241+0000","last_clean":"2024-03-24T00:41:44.224241+0000","last_became_active":"2024-03-24T00:41:41.882544+0000","last_became_peered":"2024-03-24T00:41:41.882544+0000","last_unstale":"2024-03-24T00:41:44.224241+0000","last_undegraded":"2024-03-24T00:41:44.224241+0000","last_fullsized":"2024-03-24T00:41:44.224241+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:36:17.948951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.872262+0000","last_change":"2024-03-24T00:41:42.892776+0000","last_active":"2024-03-24T00:41:43.872262+0000","last_peered":"2024-03-24T00:41:43.872262+0000","last_clean":"2024-03-24T00:41:43.872262+0000","last_became_active":"2024-03-24T00:41:42.890453+0000","last_became_peered":"2024-03-24T00:41:42.890453+0000","last_unstale":"2024-03-24T00:41:43.872262+0000","last_undegraded":"2024-03-24T00:41:43.872262+0000","last_fullsized":"2024-03-24T00:41:43.872262+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:07:25.707558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":14,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224436+0000","last_change":"2024-03-24T00:41:41.890638+0000","last_active":"2024-03-24T00:41:44.224436+0000","last_peered":"2024-03-24T00:41:44.224436+0000","last_clean":"2024-03-24T00:41:44.224436+0000","last_became_active":"2024-03-24T00:41:41.890358+0000","last_became_peered":"2024-03-24T00:41:41.890358+0000","last_unstale":"2024-03-24T00:41:44.224436+0000","last_undegraded":"2024-03-24T00:41:44.224436+0000","last_fullsized":"2024-03-24T00:41:44.224436+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:13:09.802128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224681+0000","last_change":"2024-03-24T00:41:42.889493+0000","last_active":"2024-03-24T00:41:44.224681+0000","last_peered":"2024-03-24T00:41:44.224681+0000","last_clean":"2024-03-24T00:41:44.224681+0000","last_became_active":"2024-03-24T00:41:42.889192+0000","last_became_peered":"2024-03-24T00:41:42.889192+0000","last_unstale":"2024-03-24T00:41:44.224681+0000","last_undegraded":"2024-03-24T00:41:44.224681+0000","last_fullsized":"2024-03-24T00:41:44.224681+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:49:05.987117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":14,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.872260+0000","last_change":"2024-03-24T00:41:41.891095+0000","last_active":"2024-03-24T00:41:43.872260+0000","last_peered":"2024-03-24T00:41:43.872260+0000","last_clean":"2024-03-24T00:41:43.872260+0000","last_became_active":"2024-03-24T00:41:41.890825+0000","last_became_peered":"2024-03-24T00:41:41.890825+0000","last_unstale":"2024-03-24T00:41:43.872260+0000","last_undegraded":"2024-03-24T00:41:43.872260+0000","last_fullsized":"2024-03-24T00:41:43.872260+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:27:48.703921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.225412+0000","last_change":"2024-03-24T00:41:42.899591+0000","last_active":"2024-03-24T00:41:44.225412+0000","last_peered":"2024-03-24T00:41:44.225412+0000","last_clean":"2024-03-24T00:41:44.225412+0000","last_became_active":"2024-03-24T00:41:42.899309+0000","last_became_peered":"2024-03-24T00:41:42.899309+0000","last_unstale":"2024-03-24T00:41:44.225412+0000","last_undegraded":"2024-03-24T00:41:44.225412+0000","last_fullsized":"2024-03-24T00:41:44.225412+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:28:46.600783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":14,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.223876+0000","last_change":"2024-03-24T00:41:41.879753+0000","last_active":"2024-03-24T00:41:44.223876+0000","last_peered":"2024-03-24T00:41:44.223876+0000","last_clean":"2024-03-24T00:41:44.223876+0000","last_became_active":"2024-03-24T00:41:41.879424+0000","last_became_peered":"2024-03-24T00:41:41.879424+0000","last_unstale":"2024-03-24T00:41:44.223876+0000","last_undegraded":"2024-03-24T00:41:44.223876+0000","last_fullsized":"2024-03-24T00:41:44.223876+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:16:23.678772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":247,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224820+0000","last_change":"2024-03-24T00:38:40.953871+0000","last_active":"2024-03-24T00:41:44.224820+0000","last_peered":"2024-03-24T00:41:44.224820+0000","last_clean":"2024-03-24T00:41:44.224820+0000","last_became_active":"2024-03-24T00:38:40.952926+0000","last_became_peered":"2024-03-24T00:38:40.952926+0000","last_unstale":"2024-03-24T00:41:44.224820+0000","last_undegraded":"2024-03-24T00:41:44.224820+0000","last_fullsized":"2024-03-24T00:41:44.224820+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_clean_scrub_stamp":"2024-03-24T00:38:39.722194+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:05.397257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574890,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28336,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343504,"statfs":{"total":96636764160,"available":96607748096,"internally_reserved":0,"allocated":1347584,"data_stored":922667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574889,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27744,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344096,"statfs":{"total":96636764160,"available":96608354304,"internally_reserved":0,"allocated":741376,"data_stored":331557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574889,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27744,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344096,"statfs":{"total":96636764160,"available":96608354304,"internally_reserved":0,"allocated":741376,"data_stored":331545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574888,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27752,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344088,"statfs":{"total":96636764160,"available":96608346112,"internally_reserved":0,"allocated":749568,"data_stored":333135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574888,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27740,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344100,"statfs":{"total":96636764160,"available":96608358400,"internally_reserved":0,"allocated":737280,"data_stored":331523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574889,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27756,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344084,"statfs":{"total":96636764160,"available":96608342016,"internally_reserved":0,"allocated":749568,"data_stored":338215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574889,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27768,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344072,"statfs":{"total":96636764160,"available":96608329728,"internally_reserved":0,"allocated":765952,"data_stored":333899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574888,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32488,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339352,"statfs":{"total":96636764160,"available":96603496448,"internally_reserved":0,"allocated":1331200,"data_stored":921891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:41:45.937 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:41:46.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.076+0000 7fdcce137640 1 Processor -- start 2024-03-24T00:41:46.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.076+0000 7fdcce137640 1 -- start start 2024-03-24T00:41:46.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.076+0000 7fdcce137640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fdcc815dec0 0x7fdcc80ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:46.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.076+0000 7fdcce137640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdcc80d0ef0 0x7fdcc8162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:46.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.076+0000 7fdcce137640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc80cf640 con 0x7fdcc815e2a0 2024-03-24T00:41:46.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.076+0000 7fdcce137640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcc80cee30 con 0x7fdcc815dec0 2024-03-24T00:41:46.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.076+0000 7fdcce137640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc80d2120 con 0x7fdcc80d0ef0 2024-03-24T00:41:46.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.077+0000 7fdcc7fff640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdcc80d0ef0 0x7fdcc8162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:46.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.077+0000 7fdcc77fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fdcc815dec0 0x7fdcc80ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:46.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.077+0000 7fdcc7fff640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdcc80d0ef0 0x7fdcc8162630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.67:37316/0 (socket says 172.21.15.67:37316) 2024-03-24T00:41:46.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.077+0000 7fdcbffff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fdcc815e2a0 0x7fdcc80d07e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:52214/0 (socket says 172.21.15.67:52214) 2024-03-24T00:41:46.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.077+0000 7fdcc7fff640 1 -- 172.21.15.67:0/3889674431 learned_addr learned my addr 172.21.15.67:0/3889674431 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:46.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.077+0000 7fdcc7fff640 1 -- 172.21.15.67:0/3889674431 >> v2:172.21.15.72:3300/0 conn(0x7fdcc815dec0 msgr2=0x7fdcc80ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:46.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.077+0000 7fdcc7fff640 1 --2- 172.21.15.67:0/3889674431 >> v2:172.21.15.72:3300/0 conn(0x7fdcc815dec0 0x7fdcc80ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:46.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.077+0000 7fdcc7fff640 1 -- 172.21.15.67:0/3889674431 >> v1:172.21.15.72:6789/0 conn(0x7fdcc815e2a0 legacy=0x7fdcc80d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:46.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.077+0000 7fdcc7fff640 1 -- 172.21.15.67:0/3889674431 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc80cf080 con 0x7fdcc80d0ef0 2024-03-24T00:41:46.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.077+0000 7fdcc77fe640 1 --2- 172.21.15.67:0/3889674431 >> v2:172.21.15.72:3300/0 conn(0x7fdcc815dec0 0x7fdcc80ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:41:46.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.078+0000 7fdcc7fff640 1 --2- 172.21.15.67:0/3889674431 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdcc80d0ef0 0x7fdcc8162630 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fdcc0009e50 tx=0x7fdcc002bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d17e3b97a6ff27ab server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:46.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.078+0000 7fdcc6ffd640 1 -- 172.21.15.67:0/3889674431 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc002ca70 con 0x7fdcc80d0ef0 2024-03-24T00:41:46.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.078+0000 7fdcc6ffd640 1 -- 172.21.15.67:0/3889674431 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcc002cc10 con 0x7fdcc80d0ef0 2024-03-24T00:41:46.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.078+0000 7fdcc6ffd640 1 -- 172.21.15.67:0/3889674431 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc00343f0 con 0x7fdcc80d0ef0 2024-03-24T00:41:46.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.079+0000 7fdcce137640 1 -- 172.21.15.67:0/3889674431 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdcc80d0ef0 msgr2=0x7fdcc8162630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:46.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.079+0000 7fdcce137640 1 --2- 172.21.15.67:0/3889674431 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdcc80d0ef0 0x7fdcc8162630 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fdcc0009e50 tx=0x7fdcc002bed0 comp rx=0 tx=0).stop 2024-03-24T00:41:46.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.079+0000 7fdcce137640 1 -- 172.21.15.67:0/3889674431 shutdown_connections 2024-03-24T00:41:46.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.079+0000 7fdcce137640 1 --2- 172.21.15.67:0/3889674431 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdcc80d0ef0 0x7fdcc8162630 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:46.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.079+0000 7fdcce137640 1 --2- 172.21.15.67:0/3889674431 >> v2:172.21.15.72:3300/0 conn(0x7fdcc815dec0 0x7fdcc80ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:46.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.079+0000 7fdcce137640 1 -- 172.21.15.67:0/3889674431 >> 172.21.15.67:0/3889674431 conn(0x7fdcc806c810 msgr2=0x7fdcc806cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:46.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.079+0000 7fdcce137640 1 -- 172.21.15.67:0/3889674431 shutdown_connections 2024-03-24T00:41:46.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.079+0000 7fdcce137640 1 -- 172.21.15.67:0/3889674431 wait complete. 2024-03-24T00:41:46.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.080+0000 7fdcce137640 1 Processor -- start 2024-03-24T00:41:46.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.080+0000 7fdcce137640 1 -- start start 2024-03-24T00:41:46.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.080+0000 7fdcce137640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcc80d0ef0 0x7fdcc815efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:46.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.080+0000 7fdcce137640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcc815dec0 0x7fdcc81f3e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:46.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.080+0000 7fdcce137640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdcc815e2a0 0x7fdcc80b5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:46.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.080+0000 7fdcce137640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc80d2730 con 0x7fdcc815dec0 2024-03-24T00:41:46.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.080+0000 7fdcce137640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc80d25b0 con 0x7fdcc80d0ef0 2024-03-24T00:41:46.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.080+0000 7fdcce137640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc80d28b0 con 0x7fdcc815e2a0 2024-03-24T00:41:46.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.081+0000 7fdcbffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcc815dec0 0x7fdcc81f3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:46.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.081+0000 7fdcbffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcc815dec0 0x7fdcc81f3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54616/0 (socket says 172.21.15.67:54616) 2024-03-24T00:41:46.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.081+0000 7fdcbffff640 1 -- 172.21.15.67:0/3012321649 learned_addr learned my addr 172.21.15.67:0/3012321649 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:46.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.081+0000 7fdcc7fff640 1 --2- 172.21.15.67:0/3012321649 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdcc815e2a0 0x7fdcc80b5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:46.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.081+0000 7fdcc77fe640 1 --2- 172.21.15.67:0/3012321649 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcc80d0ef0 0x7fdcc815efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:46.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.081+0000 7fdcc7fff640 1 -- 172.21.15.67:0/3012321649 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcc80d0ef0 msgr2=0x7fdcc815efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:46.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.081+0000 7fdcc7fff640 1 --2- 172.21.15.67:0/3012321649 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcc80d0ef0 0x7fdcc815efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:46.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.081+0000 7fdcc7fff640 1 -- 172.21.15.67:0/3012321649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcc815dec0 msgr2=0x7fdcc81f3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:46.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.081+0000 7fdcc7fff640 1 --2- 172.21.15.67:0/3012321649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcc815dec0 0x7fdcc81f3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:46.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.081+0000 7fdcc7fff640 1 -- 172.21.15.67:0/3012321649 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc80aabc0 con 0x7fdcc815e2a0 2024-03-24T00:41:46.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.081+0000 7fdcbffff640 1 --2- 172.21.15.67:0/3012321649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcc815dec0 0x7fdcc81f3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:41:46.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.081+0000 7fdcc7fff640 1 --2- 172.21.15.67:0/3012321649 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdcc815e2a0 0x7fdcc80b5c50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdcc0032000 tx=0x7fdcc0004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:46.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.081+0000 7fdcc4ff9640 1 -- 172.21.15.67:0/3012321649 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc0043020 con 0x7fdcc815e2a0 2024-03-24T00:41:46.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.081+0000 7fdcc4ff9640 1 -- 172.21.15.67:0/3012321649 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcc003e450 con 0x7fdcc815e2a0 2024-03-24T00:41:46.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.081+0000 7fdcce137640 1 -- 172.21.15.67:0/3012321649 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcc80aae80 con 0x7fdcc815e2a0 2024-03-24T00:41:46.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.082+0000 7fdcc4ff9640 1 -- 172.21.15.67:0/3012321649 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc0009510 con 0x7fdcc815e2a0 2024-03-24T00:41:46.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.082+0000 7fdcce137640 1 -- 172.21.15.67:0/3012321649 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcc80ab440 con 0x7fdcc815e2a0 2024-03-24T00:41:46.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.082+0000 7fdcce137640 1 -- 172.21.15.67:0/3012321649 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcc804f400 con 0x7fdcc815e2a0 2024-03-24T00:41:46.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.083+0000 7fdcc4ff9640 1 -- 172.21.15.67:0/3012321649 <== mon.2 v2:172.21.15.72:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fdcc0046020 con 0x7fdcc815e2a0 2024-03-24T00:41:46.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.083+0000 7fdcc4ff9640 1 --2- 172.21.15.67:0/3012321649 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdca806bbd0 0x7fdca806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:46.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.083+0000 7fdcc4ff9640 1 -- 172.21.15.67:0/3012321649 <== mon.2 v2:172.21.15.72:3301/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fdcc00ad120 con 0x7fdcc815e2a0 2024-03-24T00:41:46.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.084+0000 7fdcc77fe640 1 --2- 172.21.15.67:0/3012321649 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdca806bbd0 0x7fdca806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:46.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.084+0000 7fdcc77fe640 1 --2- 172.21.15.67:0/3012321649 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdca806bbd0 0x7fdca806e090 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fdcb8009560 tx=0x7fdcb8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:46.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.088+0000 7fdcc4ff9640 1 -- 172.21.15.67:0/3012321649 <== mon.2 v2:172.21.15.72:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fdcc004d030 con 0x7fdcc815e2a0 2024-03-24T00:41:46.195 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:41:46.197 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:41:46.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.272+0000 7fdcce137640 1 -- 172.21.15.67:0/3012321649 --> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdcc815f4e0 con 0x7fdca806bbd0 2024-03-24T00:41:46.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.279+0000 7fdcc4ff9640 1 -- 172.21.15.67:0/3012321649 <== mgr.4105 v2:172.21.15.72:6832/2768292274 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252016 (secure 0 0 0) 0x7fdcb8011a60 con 0x7fdca806bbd0 2024-03-24T00:41:46.281 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:41:46.282 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-24T00:41:46.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.282+0000 7fdcce137640 1 -- 172.21.15.67:0/3012321649 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdca806bbd0 msgr2=0x7fdca806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:46.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.282+0000 7fdcce137640 1 --2- 172.21.15.67:0/3012321649 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdca806bbd0 0x7fdca806e090 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fdcb8009560 tx=0x7fdcb8009190 comp rx=0 tx=0).stop 2024-03-24T00:41:46.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.282+0000 7fdcce137640 1 -- 172.21.15.67:0/3012321649 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdcc815e2a0 msgr2=0x7fdcc80b5c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:46.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.282+0000 7fdcce137640 1 --2- 172.21.15.67:0/3012321649 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdcc815e2a0 0x7fdcc80b5c50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdcc0032000 tx=0x7fdcc0004a40 comp rx=0 tx=0).stop 2024-03-24T00:41:46.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.282+0000 7fdcce137640 1 -- 172.21.15.67:0/3012321649 shutdown_connections 2024-03-24T00:41:46.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.282+0000 7fdcce137640 1 --2- 172.21.15.67:0/3012321649 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdca806bbd0 0x7fdca806e090 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:46.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.282+0000 7fdcce137640 1 --2- 172.21.15.67:0/3012321649 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdcc815e2a0 0x7fdcc80b5c50 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:46.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.282+0000 7fdcce137640 1 --2- 172.21.15.67:0/3012321649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcc815dec0 0x7fdcc81f3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:46.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.282+0000 7fdcce137640 1 --2- 172.21.15.67:0/3012321649 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcc80d0ef0 0x7fdcc815efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:46.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.282+0000 7fdcce137640 1 -- 172.21.15.67:0/3012321649 >> 172.21.15.67:0/3012321649 conn(0x7fdcc806c810 msgr2=0x7fdcc80cceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:46.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.282+0000 7fdcce137640 1 -- 172.21.15.67:0/3012321649 shutdown_connections 2024-03-24T00:41:46.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.282+0000 7fdcce137640 1 -- 172.21.15.67:0/3012321649 wait complete. 2024-03-24T00:41:46.292 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":148,"stamp":"2024-03-24T00:41:45.360565+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":227328,"kb_used_data":6996,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754747392,"statfs":{"total":773094113280,"available":772861329408,"internally_reserved":0,"allocated":7163904,"data_stored":3844432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":225496625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.496596"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.872644+0000","last_change":"2024-03-24T00:41:42.889299+0000","last_active":"2024-03-24T00:41:43.872644+0000","last_peered":"2024-03-24T00:41:43.872644+0000","last_clean":"2024-03-24T00:41:43.872644+0000","last_became_active":"2024-03-24T00:41:42.889042+0000","last_became_peered":"2024-03-24T00:41:42.889042+0000","last_unstale":"2024-03-24T00:41:43.872644+0000","last_undegraded":"2024-03-24T00:41:43.872644+0000","last_fullsized":"2024-03-24T00:41:43.872644+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:35:28.937204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.872608+0000","last_change":"2024-03-24T00:41:42.885362+0000","last_active":"2024-03-24T00:41:43.872608+0000","last_peered":"2024-03-24T00:41:43.872608+0000","last_clean":"2024-03-24T00:41:43.872608+0000","last_became_active":"2024-03-24T00:41:42.885070+0000","last_became_peered":"2024-03-24T00:41:42.885070+0000","last_unstale":"2024-03-24T00:41:43.872608+0000","last_undegraded":"2024-03-24T00:41:43.872608+0000","last_fullsized":"2024-03-24T00:41:43.872608+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:17:59.247440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224669+0000","last_change":"2024-03-24T00:41:42.899008+0000","last_active":"2024-03-24T00:41:44.224669+0000","last_peered":"2024-03-24T00:41:44.224669+0000","last_clean":"2024-03-24T00:41:44.224669+0000","last_became_active":"2024-03-24T00:41:42.897536+0000","last_became_peered":"2024-03-24T00:41:42.897536+0000","last_unstale":"2024-03-24T00:41:44.224669+0000","last_undegraded":"2024-03-24T00:41:44.224669+0000","last_fullsized":"2024-03-24T00:41:44.224669+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:15:29.815450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.873763+0000","last_change":"2024-03-24T00:41:42.903187+0000","last_active":"2024-03-24T00:41:43.873763+0000","last_peered":"2024-03-24T00:41:43.873763+0000","last_clean":"2024-03-24T00:41:43.873763+0000","last_became_active":"2024-03-24T00:41:42.902819+0000","last_became_peered":"2024-03-24T00:41:42.902819+0000","last_unstale":"2024-03-24T00:41:43.873763+0000","last_undegraded":"2024-03-24T00:41:43.873763+0000","last_fullsized":"2024-03-24T00:41:43.873763+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:39:22.851317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224884+0000","last_change":"2024-03-24T00:41:42.890806+0000","last_active":"2024-03-24T00:41:44.224884+0000","last_peered":"2024-03-24T00:41:44.224884+0000","last_clean":"2024-03-24T00:41:44.224884+0000","last_became_active":"2024-03-24T00:41:42.890526+0000","last_became_peered":"2024-03-24T00:41:42.890526+0000","last_unstale":"2024-03-24T00:41:44.224884+0000","last_undegraded":"2024-03-24T00:41:44.224884+0000","last_fullsized":"2024-03-24T00:41:44.224884+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:27:49.707905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.225005+0000","last_change":"2024-03-24T00:41:42.896565+0000","last_active":"2024-03-24T00:41:44.225005+0000","last_peered":"2024-03-24T00:41:44.225005+0000","last_clean":"2024-03-24T00:41:44.225005+0000","last_became_active":"2024-03-24T00:41:42.896238+0000","last_became_peered":"2024-03-24T00:41:42.896238+0000","last_unstale":"2024-03-24T00:41:44.225005+0000","last_undegraded":"2024-03-24T00:41:44.225005+0000","last_fullsized":"2024-03-24T00:41:44.225005+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:16:24.700063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224537+0000","last_change":"2024-03-24T00:41:42.892632+0000","last_active":"2024-03-24T00:41:44.224537+0000","last_peered":"2024-03-24T00:41:44.224537+0000","last_clean":"2024-03-24T00:41:44.224537+0000","last_became_active":"2024-03-24T00:41:42.892349+0000","last_became_peered":"2024-03-24T00:41:42.892349+0000","last_unstale":"2024-03-24T00:41:44.224537+0000","last_undegraded":"2024-03-24T00:41:44.224537+0000","last_fullsized":"2024-03-24T00:41:44.224537+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:58:39.460940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.873292+0000","last_change":"2024-03-24T00:41:42.891893+0000","last_active":"2024-03-24T00:41:43.873292+0000","last_peered":"2024-03-24T00:41:43.873292+0000","last_clean":"2024-03-24T00:41:43.873292+0000","last_became_active":"2024-03-24T00:41:42.891635+0000","last_became_peered":"2024-03-24T00:41:42.891635+0000","last_unstale":"2024-03-24T00:41:43.873292+0000","last_undegraded":"2024-03-24T00:41:43.873292+0000","last_fullsized":"2024-03-24T00:41:43.873292+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:16:24.682756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.225223+0000","last_change":"2024-03-24T00:41:42.899252+0000","last_active":"2024-03-24T00:41:44.225223+0000","last_peered":"2024-03-24T00:41:44.225223+0000","last_clean":"2024-03-24T00:41:44.225223+0000","last_became_active":"2024-03-24T00:41:42.896380+0000","last_became_peered":"2024-03-24T00:41:42.896380+0000","last_unstale":"2024-03-24T00:41:44.225223+0000","last_undegraded":"2024-03-24T00:41:44.225223+0000","last_fullsized":"2024-03-24T00:41:44.225223+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:36:18.952935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.225096+0000","last_change":"2024-03-24T00:41:42.900879+0000","last_active":"2024-03-24T00:41:44.225096+0000","last_peered":"2024-03-24T00:41:44.225096+0000","last_clean":"2024-03-24T00:41:44.225096+0000","last_became_active":"2024-03-24T00:41:42.900633+0000","last_became_peered":"2024-03-24T00:41:42.900633+0000","last_unstale":"2024-03-24T00:41:44.225096+0000","last_undegraded":"2024-03-24T00:41:44.225096+0000","last_fullsized":"2024-03-24T00:41:44.225096+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:13:10.806112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.873566+0000","last_change":"2024-03-24T00:41:42.895957+0000","last_active":"2024-03-24T00:41:43.873566+0000","last_peered":"2024-03-24T00:41:43.873566+0000","last_clean":"2024-03-24T00:41:43.873566+0000","last_became_active":"2024-03-24T00:41:42.895681+0000","last_became_peered":"2024-03-24T00:41:42.895681+0000","last_unstale":"2024-03-24T00:41:43.873566+0000","last_undegraded":"2024-03-24T00:41:43.873566+0000","last_fullsized":"2024-03-24T00:41:43.873566+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:08:03.605976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.873768+0000","last_change":"2024-03-24T00:41:42.892526+0000","last_active":"2024-03-24T00:41:43.873768+0000","last_peered":"2024-03-24T00:41:43.873768+0000","last_clean":"2024-03-24T00:41:43.873768+0000","last_became_active":"2024-03-24T00:41:42.892275+0000","last_became_peered":"2024-03-24T00:41:42.892275+0000","last_unstale":"2024-03-24T00:41:43.873768+0000","last_undegraded":"2024-03-24T00:41:43.873768+0000","last_fullsized":"2024-03-24T00:41:43.873768+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:11:52.973533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.873439+0000","last_change":"2024-03-24T00:41:42.888506+0000","last_active":"2024-03-24T00:41:43.873439+0000","last_peered":"2024-03-24T00:41:43.873439+0000","last_clean":"2024-03-24T00:41:43.873439+0000","last_became_active":"2024-03-24T00:41:42.888237+0000","last_became_peered":"2024-03-24T00:41:42.888237+0000","last_unstale":"2024-03-24T00:41:43.873439+0000","last_undegraded":"2024-03-24T00:41:43.873439+0000","last_fullsized":"2024-03-24T00:41:43.873439+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:25:45.126263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.225188+0000","last_change":"2024-03-24T00:41:42.897079+0000","last_active":"2024-03-24T00:41:44.225188+0000","last_peered":"2024-03-24T00:41:44.225188+0000","last_clean":"2024-03-24T00:41:44.225188+0000","last_became_active":"2024-03-24T00:41:42.896802+0000","last_became_peered":"2024-03-24T00:41:42.896802+0000","last_unstale":"2024-03-24T00:41:44.225188+0000","last_undegraded":"2024-03-24T00:41:44.225188+0000","last_fullsized":"2024-03-24T00:41:44.225188+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:11:52.973533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224752+0000","last_change":"2024-03-24T00:41:42.901555+0000","last_active":"2024-03-24T00:41:44.224752+0000","last_peered":"2024-03-24T00:41:44.224752+0000","last_clean":"2024-03-24T00:41:44.224752+0000","last_became_active":"2024-03-24T00:41:42.901255+0000","last_became_peered":"2024-03-24T00:41:42.901255+0000","last_unstale":"2024-03-24T00:41:44.224752+0000","last_undegraded":"2024-03-24T00:41:44.224752+0000","last_fullsized":"2024-03-24T00:41:44.224752+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:16:24.682756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224171+0000","last_change":"2024-03-24T00:41:42.889173+0000","last_active":"2024-03-24T00:41:44.224171+0000","last_peered":"2024-03-24T00:41:44.224171+0000","last_clean":"2024-03-24T00:41:44.224171+0000","last_became_active":"2024-03-24T00:41:42.888526+0000","last_became_peered":"2024-03-24T00:41:42.888526+0000","last_unstale":"2024-03-24T00:41:44.224171+0000","last_undegraded":"2024-03-24T00:41:44.224171+0000","last_fullsized":"2024-03-24T00:41:44.224171+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:24:29.207484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.872670+0000","last_change":"2024-03-24T00:41:42.889982+0000","last_active":"2024-03-24T00:41:43.872670+0000","last_peered":"2024-03-24T00:41:43.872670+0000","last_clean":"2024-03-24T00:41:43.872670+0000","last_became_active":"2024-03-24T00:41:42.889739+0000","last_became_peered":"2024-03-24T00:41:42.889739+0000","last_unstale":"2024-03-24T00:41:43.872670+0000","last_undegraded":"2024-03-24T00:41:43.872670+0000","last_fullsized":"2024-03-24T00:41:43.872670+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:42:04.651799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.874755+0000","last_change":"2024-03-24T00:41:42.893123+0000","last_active":"2024-03-24T00:41:43.874755+0000","last_peered":"2024-03-24T00:41:43.874755+0000","last_clean":"2024-03-24T00:41:43.874755+0000","last_became_active":"2024-03-24T00:41:42.892462+0000","last_became_peered":"2024-03-24T00:41:42.892462+0000","last_unstale":"2024-03-24T00:41:43.874755+0000","last_undegraded":"2024-03-24T00:41:43.874755+0000","last_fullsized":"2024-03-24T00:41:43.874755+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:52:00.925507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.225141+0000","last_change":"2024-03-24T00:41:42.889408+0000","last_active":"2024-03-24T00:41:44.225141+0000","last_peered":"2024-03-24T00:41:44.225141+0000","last_clean":"2024-03-24T00:41:44.225141+0000","last_became_active":"2024-03-24T00:41:42.889153+0000","last_became_peered":"2024-03-24T00:41:42.889153+0000","last_unstale":"2024-03-24T00:41:44.225141+0000","last_undegraded":"2024-03-24T00:41:44.225141+0000","last_fullsized":"2024-03-24T00:41:44.225141+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:21:47.607783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224594+0000","last_change":"2024-03-24T00:41:42.891147+0000","last_active":"2024-03-24T00:41:44.224594+0000","last_peered":"2024-03-24T00:41:44.224594+0000","last_clean":"2024-03-24T00:41:44.224594+0000","last_became_active":"2024-03-24T00:41:42.890867+0000","last_became_peered":"2024-03-24T00:41:42.890867+0000","last_unstale":"2024-03-24T00:41:44.224594+0000","last_undegraded":"2024-03-24T00:41:44.224594+0000","last_fullsized":"2024-03-24T00:41:44.224594+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:51:44.721189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.873589+0000","last_change":"2024-03-24T00:41:42.881781+0000","last_active":"2024-03-24T00:41:43.873589+0000","last_peered":"2024-03-24T00:41:43.873589+0000","last_clean":"2024-03-24T00:41:43.873589+0000","last_became_active":"2024-03-24T00:41:42.881489+0000","last_became_peered":"2024-03-24T00:41:42.881489+0000","last_unstale":"2024-03-24T00:41:43.873589+0000","last_undegraded":"2024-03-24T00:41:43.873589+0000","last_fullsized":"2024-03-24T00:41:43.873589+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:13:10.806112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224523+0000","last_change":"2024-03-24T00:41:42.895887+0000","last_active":"2024-03-24T00:41:44.224523+0000","last_peered":"2024-03-24T00:41:44.224523+0000","last_clean":"2024-03-24T00:41:44.224523+0000","last_became_active":"2024-03-24T00:41:42.895495+0000","last_became_peered":"2024-03-24T00:41:42.895495+0000","last_unstale":"2024-03-24T00:41:44.224523+0000","last_undegraded":"2024-03-24T00:41:44.224523+0000","last_fullsized":"2024-03-24T00:41:44.224523+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:16:38.392703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224484+0000","last_change":"2024-03-24T00:41:42.891355+0000","last_active":"2024-03-24T00:41:44.224484+0000","last_peered":"2024-03-24T00:41:44.224484+0000","last_clean":"2024-03-24T00:41:44.224484+0000","last_became_active":"2024-03-24T00:41:42.891050+0000","last_became_peered":"2024-03-24T00:41:42.891050+0000","last_unstale":"2024-03-24T00:41:44.224484+0000","last_undegraded":"2024-03-24T00:41:44.224484+0000","last_fullsized":"2024-03-24T00:41:44.224484+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:35:15.831146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224762+0000","last_change":"2024-03-24T00:41:42.897116+0000","last_active":"2024-03-24T00:41:44.224762+0000","last_peered":"2024-03-24T00:41:44.224762+0000","last_clean":"2024-03-24T00:41:44.224762+0000","last_became_active":"2024-03-24T00:41:42.896840+0000","last_became_peered":"2024-03-24T00:41:42.896840+0000","last_unstale":"2024-03-24T00:41:44.224762+0000","last_undegraded":"2024-03-24T00:41:44.224762+0000","last_fullsized":"2024-03-24T00:41:44.224762+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:25:45.126263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.873720+0000","last_change":"2024-03-24T00:41:42.890448+0000","last_active":"2024-03-24T00:41:43.873720+0000","last_peered":"2024-03-24T00:41:43.873720+0000","last_clean":"2024-03-24T00:41:43.873720+0000","last_became_active":"2024-03-24T00:41:42.890144+0000","last_became_peered":"2024-03-24T00:41:42.890144+0000","last_unstale":"2024-03-24T00:41:43.873720+0000","last_undegraded":"2024-03-24T00:41:43.873720+0000","last_fullsized":"2024-03-24T00:41:43.873720+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:16:31.545531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.873914+0000","last_change":"2024-03-24T00:41:42.891823+0000","last_active":"2024-03-24T00:41:43.873914+0000","last_peered":"2024-03-24T00:41:43.873914+0000","last_clean":"2024-03-24T00:41:43.873914+0000","last_became_active":"2024-03-24T00:41:42.890920+0000","last_became_peered":"2024-03-24T00:41:42.890920+0000","last_unstale":"2024-03-24T00:41:43.873914+0000","last_undegraded":"2024-03-24T00:41:43.873914+0000","last_fullsized":"2024-03-24T00:41:43.873914+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:36:18.952935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.874550+0000","last_change":"2024-03-24T00:41:42.889763+0000","last_active":"2024-03-24T00:41:43.874550+0000","last_peered":"2024-03-24T00:41:43.874550+0000","last_clean":"2024-03-24T00:41:43.874550+0000","last_became_active":"2024-03-24T00:41:42.889482+0000","last_became_peered":"2024-03-24T00:41:42.889482+0000","last_unstale":"2024-03-24T00:41:43.874550+0000","last_undegraded":"2024-03-24T00:41:43.874550+0000","last_fullsized":"2024-03-24T00:41:43.874550+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:04:50.007458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.872556+0000","last_change":"2024-03-24T00:41:42.890444+0000","last_active":"2024-03-24T00:41:43.872556+0000","last_peered":"2024-03-24T00:41:43.872556+0000","last_clean":"2024-03-24T00:41:43.872556+0000","last_became_active":"2024-03-24T00:41:42.890145+0000","last_became_peered":"2024-03-24T00:41:42.890145+0000","last_unstale":"2024-03-24T00:41:43.872556+0000","last_undegraded":"2024-03-24T00:41:43.872556+0000","last_fullsized":"2024-03-24T00:41:43.872556+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:11:20.487065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224559+0000","last_change":"2024-03-24T00:41:42.899439+0000","last_active":"2024-03-24T00:41:44.224559+0000","last_peered":"2024-03-24T00:41:44.224559+0000","last_clean":"2024-03-24T00:41:44.224559+0000","last_became_active":"2024-03-24T00:41:42.897464+0000","last_became_peered":"2024-03-24T00:41:42.897464+0000","last_unstale":"2024-03-24T00:41:44.224559+0000","last_undegraded":"2024-03-24T00:41:44.224559+0000","last_fullsized":"2024-03-24T00:41:44.224559+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:15:31.777583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.873951+0000","last_change":"2024-03-24T00:41:42.891591+0000","last_active":"2024-03-24T00:41:43.873951+0000","last_peered":"2024-03-24T00:41:43.873951+0000","last_clean":"2024-03-24T00:41:43.873951+0000","last_became_active":"2024-03-24T00:41:42.891297+0000","last_became_peered":"2024-03-24T00:41:42.891297+0000","last_unstale":"2024-03-24T00:41:43.873951+0000","last_undegraded":"2024-03-24T00:41:43.873951+0000","last_fullsized":"2024-03-24T00:41:43.873951+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:59:25.755135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.874397+0000","last_change":"2024-03-24T00:41:42.888895+0000","last_active":"2024-03-24T00:41:43.874397+0000","last_peered":"2024-03-24T00:41:43.874397+0000","last_clean":"2024-03-24T00:41:43.874397+0000","last_became_active":"2024-03-24T00:41:42.888648+0000","last_became_peered":"2024-03-24T00:41:42.888648+0000","last_unstale":"2024-03-24T00:41:43.874397+0000","last_undegraded":"2024-03-24T00:41:43.874397+0000","last_fullsized":"2024-03-24T00:41:43.874397+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:41:48.723391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224389+0000","last_change":"2024-03-24T00:41:42.891152+0000","last_active":"2024-03-24T00:41:44.224389+0000","last_peered":"2024-03-24T00:41:44.224389+0000","last_clean":"2024-03-24T00:41:44.224389+0000","last_became_active":"2024-03-24T00:41:42.890890+0000","last_became_peered":"2024-03-24T00:41:42.890890+0000","last_unstale":"2024-03-24T00:41:44.224389+0000","last_undegraded":"2024-03-24T00:41:44.224389+0000","last_fullsized":"2024-03-24T00:41:44.224389+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:36:18.952935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224680+0000","last_change":"2024-03-24T00:41:42.896417+0000","last_active":"2024-03-24T00:41:44.224680+0000","last_peered":"2024-03-24T00:41:44.224680+0000","last_clean":"2024-03-24T00:41:44.224680+0000","last_became_active":"2024-03-24T00:41:42.895327+0000","last_became_peered":"2024-03-24T00:41:42.895327+0000","last_unstale":"2024-03-24T00:41:44.224680+0000","last_undegraded":"2024-03-24T00:41:44.224680+0000","last_fullsized":"2024-03-24T00:41:44.224680+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:39:22.851317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"61'4","reported_seq":18,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.872805+0000","last_change":"2024-03-24T00:41:41.884745+0000","last_active":"2024-03-24T00:41:43.872805+0000","last_peered":"2024-03-24T00:41:43.872805+0000","last_clean":"2024-03-24T00:41:43.872805+0000","last_became_active":"2024-03-24T00:41:41.884498+0000","last_became_peered":"2024-03-24T00:41:41.884498+0000","last_unstale":"2024-03-24T00:41:43.872805+0000","last_undegraded":"2024-03-24T00:41:43.872805+0000","last_fullsized":"2024-03-24T00:41:43.872805+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:24:35.066275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.874817+0000","last_change":"2024-03-24T00:41:42.894240+0000","last_active":"2024-03-24T00:41:43.874817+0000","last_peered":"2024-03-24T00:41:43.874817+0000","last_clean":"2024-03-24T00:41:43.874817+0000","last_became_active":"2024-03-24T00:41:42.893817+0000","last_became_peered":"2024-03-24T00:41:42.893817+0000","last_unstale":"2024-03-24T00:41:43.874817+0000","last_undegraded":"2024-03-24T00:41:43.874817+0000","last_fullsized":"2024-03-24T00:41:43.874817+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:32:00.196741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"61'1","reported_seq":15,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224940+0000","last_change":"2024-03-24T00:41:41.884801+0000","last_active":"2024-03-24T00:41:44.224940+0000","last_peered":"2024-03-24T00:41:44.224940+0000","last_clean":"2024-03-24T00:41:44.224940+0000","last_became_active":"2024-03-24T00:41:41.884477+0000","last_became_peered":"2024-03-24T00:41:41.884477+0000","last_unstale":"2024-03-24T00:41:44.224940+0000","last_undegraded":"2024-03-24T00:41:44.224940+0000","last_fullsized":"2024-03-24T00:41:44.224940+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:15:28.811466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224772+0000","last_change":"2024-03-24T00:41:42.893799+0000","last_active":"2024-03-24T00:41:44.224772+0000","last_peered":"2024-03-24T00:41:44.224772+0000","last_clean":"2024-03-24T00:41:44.224772+0000","last_became_active":"2024-03-24T00:41:42.893524+0000","last_became_peered":"2024-03-24T00:41:42.893524+0000","last_unstale":"2024-03-24T00:41:44.224772+0000","last_undegraded":"2024-03-24T00:41:44.224772+0000","last_fullsized":"2024-03-24T00:41:44.224772+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:22:12.038404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"61'2","reported_seq":16,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224421+0000","last_change":"2024-03-24T00:41:41.885417+0000","last_active":"2024-03-24T00:41:44.224421+0000","last_peered":"2024-03-24T00:41:44.224421+0000","last_clean":"2024-03-24T00:41:44.224421+0000","last_became_active":"2024-03-24T00:41:41.885125+0000","last_became_peered":"2024-03-24T00:41:41.885125+0000","last_unstale":"2024-03-24T00:41:44.224421+0000","last_undegraded":"2024-03-24T00:41:44.224421+0000","last_fullsized":"2024-03-24T00:41:44.224421+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:35:30.712724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.225120+0000","last_change":"2024-03-24T00:41:42.903357+0000","last_active":"2024-03-24T00:41:44.225120+0000","last_peered":"2024-03-24T00:41:44.225120+0000","last_clean":"2024-03-24T00:41:44.225120+0000","last_became_active":"2024-03-24T00:41:42.902782+0000","last_became_peered":"2024-03-24T00:41:42.902782+0000","last_unstale":"2024-03-24T00:41:44.225120+0000","last_undegraded":"2024-03-24T00:41:44.225120+0000","last_fullsized":"2024-03-24T00:41:44.225120+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:16:31.545531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"61'2","reported_seq":16,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.225196+0000","last_change":"2024-03-24T00:41:41.880628+0000","last_active":"2024-03-24T00:41:44.225196+0000","last_peered":"2024-03-24T00:41:44.225196+0000","last_clean":"2024-03-24T00:41:44.225196+0000","last_became_active":"2024-03-24T00:41:41.880334+0000","last_became_peered":"2024-03-24T00:41:41.880334+0000","last_unstale":"2024-03-24T00:41:44.225196+0000","last_undegraded":"2024-03-24T00:41:44.225196+0000","last_fullsized":"2024-03-24T00:41:44.225196+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:59:22.376959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224302+0000","last_change":"2024-03-24T00:41:42.889436+0000","last_active":"2024-03-24T00:41:44.224302+0000","last_peered":"2024-03-24T00:41:44.224302+0000","last_clean":"2024-03-24T00:41:44.224302+0000","last_became_active":"2024-03-24T00:41:42.888826+0000","last_became_peered":"2024-03-24T00:41:42.888826+0000","last_unstale":"2024-03-24T00:41:44.224302+0000","last_undegraded":"2024-03-24T00:41:44.224302+0000","last_fullsized":"2024-03-24T00:41:44.224302+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:05:53.078237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"61'1","reported_seq":15,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224232+0000","last_change":"2024-03-24T00:41:41.881319+0000","last_active":"2024-03-24T00:41:44.224232+0000","last_peered":"2024-03-24T00:41:44.224232+0000","last_clean":"2024-03-24T00:41:44.224232+0000","last_became_active":"2024-03-24T00:41:41.880598+0000","last_became_peered":"2024-03-24T00:41:41.880598+0000","last_unstale":"2024-03-24T00:41:44.224232+0000","last_undegraded":"2024-03-24T00:41:44.224232+0000","last_fullsized":"2024-03-24T00:41:44.224232+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:59:24.751151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.872289+0000","last_change":"2024-03-24T00:41:42.893623+0000","last_active":"2024-03-24T00:41:43.872289+0000","last_peered":"2024-03-24T00:41:43.872289+0000","last_clean":"2024-03-24T00:41:43.872289+0000","last_became_active":"2024-03-24T00:41:42.893372+0000","last_became_peered":"2024-03-24T00:41:42.893372+0000","last_unstale":"2024-03-24T00:41:43.872289+0000","last_undegraded":"2024-03-24T00:41:43.872289+0000","last_fullsized":"2024-03-24T00:41:43.872289+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:31:37.065977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"61'1","reported_seq":15,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224141+0000","last_change":"2024-03-24T00:41:41.882741+0000","last_active":"2024-03-24T00:41:44.224141+0000","last_peered":"2024-03-24T00:41:44.224141+0000","last_clean":"2024-03-24T00:41:44.224141+0000","last_became_active":"2024-03-24T00:41:41.882453+0000","last_became_peered":"2024-03-24T00:41:41.882453+0000","last_unstale":"2024-03-24T00:41:44.224141+0000","last_undegraded":"2024-03-24T00:41:44.224141+0000","last_fullsized":"2024-03-24T00:41:44.224141+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:16:30.541548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224961+0000","last_change":"2024-03-24T00:41:42.885195+0000","last_active":"2024-03-24T00:41:44.224961+0000","last_peered":"2024-03-24T00:41:44.224961+0000","last_clean":"2024-03-24T00:41:44.224961+0000","last_became_active":"2024-03-24T00:41:42.884880+0000","last_became_peered":"2024-03-24T00:41:42.884880+0000","last_unstale":"2024-03-24T00:41:44.224961+0000","last_undegraded":"2024-03-24T00:41:44.224961+0000","last_fullsized":"2024-03-24T00:41:44.224961+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:24:36.070259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":14,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224342+0000","last_change":"2024-03-24T00:41:41.891261+0000","last_active":"2024-03-24T00:41:44.224342+0000","last_peered":"2024-03-24T00:41:44.224342+0000","last_clean":"2024-03-24T00:41:44.224342+0000","last_became_active":"2024-03-24T00:41:41.891006+0000","last_became_peered":"2024-03-24T00:41:41.891006+0000","last_unstale":"2024-03-24T00:41:44.224342+0000","last_undegraded":"2024-03-24T00:41:44.224342+0000","last_fullsized":"2024-03-24T00:41:44.224342+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:25:44.122279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.225131+0000","last_change":"2024-03-24T00:41:42.891260+0000","last_active":"2024-03-24T00:41:44.225131+0000","last_peered":"2024-03-24T00:41:44.225131+0000","last_clean":"2024-03-24T00:41:44.225131+0000","last_became_active":"2024-03-24T00:41:42.890972+0000","last_became_peered":"2024-03-24T00:41:42.890972+0000","last_unstale":"2024-03-24T00:41:44.225131+0000","last_undegraded":"2024-03-24T00:41:44.225131+0000","last_fullsized":"2024-03-24T00:41:44.225131+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:00:07.007109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":14,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224162+0000","last_change":"2024-03-24T00:41:41.889322+0000","last_active":"2024-03-24T00:41:44.224162+0000","last_peered":"2024-03-24T00:41:44.224162+0000","last_clean":"2024-03-24T00:41:44.224162+0000","last_became_active":"2024-03-24T00:41:41.889045+0000","last_became_peered":"2024-03-24T00:41:41.889045+0000","last_unstale":"2024-03-24T00:41:44.224162+0000","last_undegraded":"2024-03-24T00:41:44.224162+0000","last_fullsized":"2024-03-24T00:41:44.224162+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:15:07.171930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.872627+0000","last_change":"2024-03-24T00:41:42.894710+0000","last_active":"2024-03-24T00:41:43.872627+0000","last_peered":"2024-03-24T00:41:43.872627+0000","last_clean":"2024-03-24T00:41:43.872627+0000","last_became_active":"2024-03-24T00:41:42.894449+0000","last_became_peered":"2024-03-24T00:41:42.894449+0000","last_unstale":"2024-03-24T00:41:43.872627+0000","last_undegraded":"2024-03-24T00:41:43.872627+0000","last_fullsized":"2024-03-24T00:41:43.872627+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:07:39.772752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":14,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.225223+0000","last_change":"2024-03-24T00:41:41.883456+0000","last_active":"2024-03-24T00:41:44.225223+0000","last_peered":"2024-03-24T00:41:44.225223+0000","last_clean":"2024-03-24T00:41:44.225223+0000","last_became_active":"2024-03-24T00:41:41.883173+0000","last_became_peered":"2024-03-24T00:41:41.883173+0000","last_unstale":"2024-03-24T00:41:44.225223+0000","last_undegraded":"2024-03-24T00:41:44.225223+0000","last_fullsized":"2024-03-24T00:41:44.225223+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:31:36.061994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224560+0000","last_change":"2024-03-24T00:41:42.877552+0000","last_active":"2024-03-24T00:41:44.224560+0000","last_peered":"2024-03-24T00:41:44.224560+0000","last_clean":"2024-03-24T00:41:44.224560+0000","last_became_active":"2024-03-24T00:41:42.877236+0000","last_became_peered":"2024-03-24T00:41:42.877236+0000","last_unstale":"2024-03-24T00:41:44.224560+0000","last_undegraded":"2024-03-24T00:41:44.224560+0000","last_fullsized":"2024-03-24T00:41:44.224560+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:55:15.958279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":14,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.874864+0000","last_change":"2024-03-24T00:41:41.888178+0000","last_active":"2024-03-24T00:41:43.874864+0000","last_peered":"2024-03-24T00:41:43.874864+0000","last_clean":"2024-03-24T00:41:43.874864+0000","last_became_active":"2024-03-24T00:41:41.887657+0000","last_became_peered":"2024-03-24T00:41:41.887657+0000","last_unstale":"2024-03-24T00:41:43.874864+0000","last_undegraded":"2024-03-24T00:41:43.874864+0000","last_fullsized":"2024-03-24T00:41:43.874864+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:19:05.072568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224890+0000","last_change":"2024-03-24T00:41:42.887646+0000","last_active":"2024-03-24T00:41:44.224890+0000","last_peered":"2024-03-24T00:41:44.224890+0000","last_clean":"2024-03-24T00:41:44.224890+0000","last_became_active":"2024-03-24T00:41:42.887366+0000","last_became_peered":"2024-03-24T00:41:42.887366+0000","last_unstale":"2024-03-24T00:41:44.224890+0000","last_undegraded":"2024-03-24T00:41:44.224890+0000","last_fullsized":"2024-03-24T00:41:44.224890+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:04:50.007458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"61'3","reported_seq":17,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224643+0000","last_change":"2024-03-24T00:41:41.884985+0000","last_active":"2024-03-24T00:41:44.224643+0000","last_peered":"2024-03-24T00:41:44.224643+0000","last_clean":"2024-03-24T00:41:44.224643+0000","last_became_active":"2024-03-24T00:41:41.883972+0000","last_became_peered":"2024-03-24T00:41:41.883972+0000","last_unstale":"2024-03-24T00:41:44.224643+0000","last_undegraded":"2024-03-24T00:41:44.224643+0000","last_fullsized":"2024-03-24T00:41:44.224643+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:07:24.703574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.874613+0000","last_change":"2024-03-24T00:41:42.891388+0000","last_active":"2024-03-24T00:41:43.874613+0000","last_peered":"2024-03-24T00:41:43.874613+0000","last_clean":"2024-03-24T00:41:43.874613+0000","last_became_active":"2024-03-24T00:41:42.891132+0000","last_became_peered":"2024-03-24T00:41:42.891132+0000","last_unstale":"2024-03-24T00:41:43.874613+0000","last_undegraded":"2024-03-24T00:41:43.874613+0000","last_fullsized":"2024-03-24T00:41:43.874613+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:24:36.070259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"61'6","reported_seq":20,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.872670+0000","last_change":"2024-03-24T00:41:41.890234+0000","last_active":"2024-03-24T00:41:43.872670+0000","last_peered":"2024-03-24T00:41:43.872670+0000","last_clean":"2024-03-24T00:41:43.872670+0000","last_became_active":"2024-03-24T00:41:41.889928+0000","last_became_peered":"2024-03-24T00:41:41.889928+0000","last_unstale":"2024-03-24T00:41:43.872670+0000","last_undegraded":"2024-03-24T00:41:43.872670+0000","last_fullsized":"2024-03-24T00:41:43.872670+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:04:49.003475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.873456+0000","last_change":"2024-03-24T00:41:42.896893+0000","last_active":"2024-03-24T00:41:43.873456+0000","last_peered":"2024-03-24T00:41:43.873456+0000","last_clean":"2024-03-24T00:41:43.873456+0000","last_became_active":"2024-03-24T00:41:42.896616+0000","last_became_peered":"2024-03-24T00:41:42.896616+0000","last_unstale":"2024-03-24T00:41:43.873456+0000","last_undegraded":"2024-03-24T00:41:43.873456+0000","last_fullsized":"2024-03-24T00:41:43.873456+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:42:34.765808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"61'2","reported_seq":16,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224506+0000","last_change":"2024-03-24T00:41:41.891160+0000","last_active":"2024-03-24T00:41:44.224506+0000","last_peered":"2024-03-24T00:41:44.224506+0000","last_clean":"2024-03-24T00:41:44.224506+0000","last_became_active":"2024-03-24T00:41:41.890904+0000","last_became_peered":"2024-03-24T00:41:41.890904+0000","last_unstale":"2024-03-24T00:41:44.224506+0000","last_undegraded":"2024-03-24T00:41:44.224506+0000","last_fullsized":"2024-03-24T00:41:44.224506+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:27:12.596556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.874215+0000","last_change":"2024-03-24T00:41:42.882382+0000","last_active":"2024-03-24T00:41:43.874215+0000","last_peered":"2024-03-24T00:41:43.874215+0000","last_clean":"2024-03-24T00:41:43.874215+0000","last_became_active":"2024-03-24T00:41:42.882105+0000","last_became_peered":"2024-03-24T00:41:42.882105+0000","last_unstale":"2024-03-24T00:41:43.874215+0000","last_undegraded":"2024-03-24T00:41:43.874215+0000","last_fullsized":"2024-03-24T00:41:43.874215+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:16:24.700063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":14,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.874072+0000","last_change":"2024-03-24T00:41:41.882297+0000","last_active":"2024-03-24T00:41:43.874072+0000","last_peered":"2024-03-24T00:41:43.874072+0000","last_clean":"2024-03-24T00:41:43.874072+0000","last_became_active":"2024-03-24T00:41:41.882002+0000","last_became_peered":"2024-03-24T00:41:41.882002+0000","last_unstale":"2024-03-24T00:41:43.874072+0000","last_undegraded":"2024-03-24T00:41:43.874072+0000","last_fullsized":"2024-03-24T00:41:43.874072+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:31:36.061994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224268+0000","last_change":"2024-03-24T00:41:42.888607+0000","last_active":"2024-03-24T00:41:44.224268+0000","last_peered":"2024-03-24T00:41:44.224268+0000","last_clean":"2024-03-24T00:41:44.224268+0000","last_became_active":"2024-03-24T00:41:42.888351+0000","last_became_peered":"2024-03-24T00:41:42.888351+0000","last_unstale":"2024-03-24T00:41:44.224268+0000","last_undegraded":"2024-03-24T00:41:44.224268+0000","last_fullsized":"2024-03-24T00:41:44.224268+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:42:34.765808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":14,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224924+0000","last_change":"2024-03-24T00:41:41.888499+0000","last_active":"2024-03-24T00:41:44.224924+0000","last_peered":"2024-03-24T00:41:44.224924+0000","last_clean":"2024-03-24T00:41:44.224924+0000","last_became_active":"2024-03-24T00:41:41.888212+0000","last_became_peered":"2024-03-24T00:41:41.888212+0000","last_unstale":"2024-03-24T00:41:44.224924+0000","last_undegraded":"2024-03-24T00:41:44.224924+0000","last_fullsized":"2024-03-24T00:41:44.224924+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:33:27.374739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.872462+0000","last_change":"2024-03-24T00:41:42.900253+0000","last_active":"2024-03-24T00:41:43.872462+0000","last_peered":"2024-03-24T00:41:43.872462+0000","last_clean":"2024-03-24T00:41:43.872462+0000","last_became_active":"2024-03-24T00:41:42.899899+0000","last_became_peered":"2024-03-24T00:41:42.899899+0000","last_unstale":"2024-03-24T00:41:43.872462+0000","last_undegraded":"2024-03-24T00:41:43.872462+0000","last_fullsized":"2024-03-24T00:41:43.872462+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:02:34.690937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"61'8","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.873577+0000","last_change":"2024-03-24T00:41:41.886878+0000","last_active":"2024-03-24T00:41:43.873577+0000","last_peered":"2024-03-24T00:41:43.873577+0000","last_clean":"2024-03-24T00:41:43.873577+0000","last_became_active":"2024-03-24T00:41:41.886581+0000","last_became_peered":"2024-03-24T00:41:41.886581+0000","last_unstale":"2024-03-24T00:41:43.873577+0000","last_undegraded":"2024-03-24T00:41:43.873577+0000","last_fullsized":"2024-03-24T00:41:43.873577+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:45:35.636179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224626+0000","last_change":"2024-03-24T00:41:42.895413+0000","last_active":"2024-03-24T00:41:44.224626+0000","last_peered":"2024-03-24T00:41:44.224626+0000","last_clean":"2024-03-24T00:41:44.224626+0000","last_became_active":"2024-03-24T00:41:42.895137+0000","last_became_peered":"2024-03-24T00:41:42.895137+0000","last_unstale":"2024-03-24T00:41:44.224626+0000","last_undegraded":"2024-03-24T00:41:44.224626+0000","last_fullsized":"2024-03-24T00:41:44.224626+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:59:23.380943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"61'1","reported_seq":15,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.873836+0000","last_change":"2024-03-24T00:41:41.885287+0000","last_active":"2024-03-24T00:41:43.873836+0000","last_peered":"2024-03-24T00:41:43.873836+0000","last_clean":"2024-03-24T00:41:43.873836+0000","last_became_active":"2024-03-24T00:41:41.884901+0000","last_became_peered":"2024-03-24T00:41:41.884901+0000","last_unstale":"2024-03-24T00:41:43.873836+0000","last_undegraded":"2024-03-24T00:41:43.873836+0000","last_fullsized":"2024-03-24T00:41:43.873836+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:02:33.686954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.225273+0000","last_change":"2024-03-24T00:41:42.890724+0000","last_active":"2024-03-24T00:41:44.225273+0000","last_peered":"2024-03-24T00:41:44.225273+0000","last_clean":"2024-03-24T00:41:44.225273+0000","last_became_active":"2024-03-24T00:41:42.890461+0000","last_became_peered":"2024-03-24T00:41:42.890461+0000","last_unstale":"2024-03-24T00:41:44.225273+0000","last_undegraded":"2024-03-24T00:41:44.225273+0000","last_fullsized":"2024-03-24T00:41:44.225273+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:28:13.579193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"61'2","reported_seq":16,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.225090+0000","last_change":"2024-03-24T00:41:41.888640+0000","last_active":"2024-03-24T00:41:44.225090+0000","last_peered":"2024-03-24T00:41:44.225090+0000","last_clean":"2024-03-24T00:41:44.225090+0000","last_became_active":"2024-03-24T00:41:41.888316+0000","last_became_peered":"2024-03-24T00:41:41.888316+0000","last_unstale":"2024-03-24T00:41:44.225090+0000","last_undegraded":"2024-03-24T00:41:44.225090+0000","last_fullsized":"2024-03-24T00:41:44.225090+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:25:09.471037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.874757+0000","last_change":"2024-03-24T00:41:42.893352+0000","last_active":"2024-03-24T00:41:43.874757+0000","last_peered":"2024-03-24T00:41:43.874757+0000","last_clean":"2024-03-24T00:41:43.874757+0000","last_became_active":"2024-03-24T00:41:42.892808+0000","last_became_peered":"2024-03-24T00:41:42.892808+0000","last_unstale":"2024-03-24T00:41:43.874757+0000","last_undegraded":"2024-03-24T00:41:43.874757+0000","last_fullsized":"2024-03-24T00:41:43.874757+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:21:47.607783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"61'1","reported_seq":15,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224948+0000","last_change":"2024-03-24T00:41:41.887827+0000","last_active":"2024-03-24T00:41:44.224948+0000","last_peered":"2024-03-24T00:41:44.224948+0000","last_clean":"2024-03-24T00:41:44.224948+0000","last_became_active":"2024-03-24T00:41:41.887507+0000","last_became_peered":"2024-03-24T00:41:41.887507+0000","last_unstale":"2024-03-24T00:41:44.224948+0000","last_undegraded":"2024-03-24T00:41:44.224948+0000","last_fullsized":"2024-03-24T00:41:44.224948+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:49:35.492413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.225067+0000","last_change":"2024-03-24T00:41:42.898105+0000","last_active":"2024-03-24T00:41:44.225067+0000","last_peered":"2024-03-24T00:41:44.225067+0000","last_clean":"2024-03-24T00:41:44.225067+0000","last_became_active":"2024-03-24T00:41:42.897801+0000","last_became_peered":"2024-03-24T00:41:42.897801+0000","last_unstale":"2024-03-24T00:41:44.225067+0000","last_undegraded":"2024-03-24T00:41:44.225067+0000","last_fullsized":"2024-03-24T00:41:44.225067+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:52:00.925507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":14,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224471+0000","last_change":"2024-03-24T00:41:41.888772+0000","last_active":"2024-03-24T00:41:44.224471+0000","last_peered":"2024-03-24T00:41:44.224471+0000","last_clean":"2024-03-24T00:41:44.224471+0000","last_became_active":"2024-03-24T00:41:41.888403+0000","last_became_peered":"2024-03-24T00:41:41.888403+0000","last_unstale":"2024-03-24T00:41:44.224471+0000","last_undegraded":"2024-03-24T00:41:44.224471+0000","last_fullsized":"2024-03-24T00:41:44.224471+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:11:51.969549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224532+0000","last_change":"2024-03-24T00:41:42.891846+0000","last_active":"2024-03-24T00:41:44.224532+0000","last_peered":"2024-03-24T00:41:44.224532+0000","last_clean":"2024-03-24T00:41:44.224532+0000","last_became_active":"2024-03-24T00:41:42.891593+0000","last_became_peered":"2024-03-24T00:41:42.891593+0000","last_unstale":"2024-03-24T00:41:44.224532+0000","last_undegraded":"2024-03-24T00:41:44.224532+0000","last_fullsized":"2024-03-24T00:41:44.224532+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:59:25.755135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"61'2","reported_seq":16,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224141+0000","last_change":"2024-03-24T00:41:41.890217+0000","last_active":"2024-03-24T00:41:44.224141+0000","last_peered":"2024-03-24T00:41:44.224141+0000","last_clean":"2024-03-24T00:41:44.224141+0000","last_became_active":"2024-03-24T00:41:41.889876+0000","last_became_peered":"2024-03-24T00:41:41.889876+0000","last_unstale":"2024-03-24T00:41:44.224141+0000","last_undegraded":"2024-03-24T00:41:44.224141+0000","last_fullsized":"2024-03-24T00:41:44.224141+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:57:54.901758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.874324+0000","last_change":"2024-03-24T00:41:42.890665+0000","last_active":"2024-03-24T00:41:43.874324+0000","last_peered":"2024-03-24T00:41:43.874324+0000","last_clean":"2024-03-24T00:41:43.874324+0000","last_became_active":"2024-03-24T00:41:42.890404+0000","last_became_peered":"2024-03-24T00:41:42.890404+0000","last_unstale":"2024-03-24T00:41:43.874324+0000","last_undegraded":"2024-03-24T00:41:43.874324+0000","last_fullsized":"2024-03-24T00:41:43.874324+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:49:44.091221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":14,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.874180+0000","last_change":"2024-03-24T00:41:41.879431+0000","last_active":"2024-03-24T00:41:43.874180+0000","last_peered":"2024-03-24T00:41:43.874180+0000","last_clean":"2024-03-24T00:41:43.874180+0000","last_became_active":"2024-03-24T00:41:41.879169+0000","last_became_peered":"2024-03-24T00:41:41.879169+0000","last_unstale":"2024-03-24T00:41:43.874180+0000","last_undegraded":"2024-03-24T00:41:43.874180+0000","last_fullsized":"2024-03-24T00:41:43.874180+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:07:24.703574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224560+0000","last_change":"2024-03-24T00:41:42.884629+0000","last_active":"2024-03-24T00:41:44.224560+0000","last_peered":"2024-03-24T00:41:44.224560+0000","last_clean":"2024-03-24T00:41:44.224560+0000","last_became_active":"2024-03-24T00:41:42.884284+0000","last_became_peered":"2024-03-24T00:41:42.884284+0000","last_unstale":"2024-03-24T00:41:44.224560+0000","last_undegraded":"2024-03-24T00:41:44.224560+0000","last_fullsized":"2024-03-24T00:41:44.224560+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:08:03.605976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":14,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224645+0000","last_change":"2024-03-24T00:41:41.886088+0000","last_active":"2024-03-24T00:41:44.224645+0000","last_peered":"2024-03-24T00:41:44.224645+0000","last_clean":"2024-03-24T00:41:44.224645+0000","last_became_active":"2024-03-24T00:41:41.885794+0000","last_became_peered":"2024-03-24T00:41:41.885794+0000","last_unstale":"2024-03-24T00:41:44.224645+0000","last_undegraded":"2024-03-24T00:41:44.224645+0000","last_fullsized":"2024-03-24T00:41:44.224645+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:29:01.056209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.225163+0000","last_change":"2024-03-24T00:41:42.892702+0000","last_active":"2024-03-24T00:41:44.225163+0000","last_peered":"2024-03-24T00:41:44.225163+0000","last_clean":"2024-03-24T00:41:44.225163+0000","last_became_active":"2024-03-24T00:41:42.892416+0000","last_became_peered":"2024-03-24T00:41:42.892416+0000","last_unstale":"2024-03-24T00:41:44.225163+0000","last_undegraded":"2024-03-24T00:41:44.225163+0000","last_fullsized":"2024-03-24T00:41:44.225163+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:49:44.091221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"61'2","reported_seq":16,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.874058+0000","last_change":"2024-03-24T00:41:41.884561+0000","last_active":"2024-03-24T00:41:43.874058+0000","last_peered":"2024-03-24T00:41:43.874058+0000","last_clean":"2024-03-24T00:41:43.874058+0000","last_became_active":"2024-03-24T00:41:41.884274+0000","last_became_peered":"2024-03-24T00:41:41.884274+0000","last_unstale":"2024-03-24T00:41:43.874058+0000","last_undegraded":"2024-03-24T00:41:43.874058+0000","last_fullsized":"2024-03-24T00:41:43.874058+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:11:19.483081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224915+0000","last_change":"2024-03-24T00:41:42.886841+0000","last_active":"2024-03-24T00:41:44.224915+0000","last_peered":"2024-03-24T00:41:44.224915+0000","last_clean":"2024-03-24T00:41:44.224915+0000","last_became_active":"2024-03-24T00:41:42.886567+0000","last_became_peered":"2024-03-24T00:41:42.886567+0000","last_unstale":"2024-03-24T00:41:44.224915+0000","last_undegraded":"2024-03-24T00:41:44.224915+0000","last_fullsized":"2024-03-24T00:41:44.224915+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:17:54.514321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":14,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.874199+0000","last_change":"2024-03-24T00:41:41.881300+0000","last_active":"2024-03-24T00:41:43.874199+0000","last_peered":"2024-03-24T00:41:43.874199+0000","last_clean":"2024-03-24T00:41:43.874199+0000","last_became_active":"2024-03-24T00:41:41.880759+0000","last_became_peered":"2024-03-24T00:41:41.880759+0000","last_unstale":"2024-03-24T00:41:43.874199+0000","last_undegraded":"2024-03-24T00:41:43.874199+0000","last_fullsized":"2024-03-24T00:41:43.874199+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:25:09.471037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.874119+0000","last_change":"2024-03-24T00:41:42.881823+0000","last_active":"2024-03-24T00:41:43.874119+0000","last_peered":"2024-03-24T00:41:43.874119+0000","last_clean":"2024-03-24T00:41:43.874119+0000","last_became_active":"2024-03-24T00:41:42.881550+0000","last_became_peered":"2024-03-24T00:41:42.881550+0000","last_unstale":"2024-03-24T00:41:43.874119+0000","last_undegraded":"2024-03-24T00:41:43.874119+0000","last_fullsized":"2024-03-24T00:41:43.874119+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:22:12.038404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":14,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224511+0000","last_change":"2024-03-24T00:41:41.886155+0000","last_active":"2024-03-24T00:41:44.224511+0000","last_peered":"2024-03-24T00:41:44.224511+0000","last_clean":"2024-03-24T00:41:44.224511+0000","last_became_active":"2024-03-24T00:41:41.885881+0000","last_became_peered":"2024-03-24T00:41:41.885881+0000","last_unstale":"2024-03-24T00:41:44.224511+0000","last_undegraded":"2024-03-24T00:41:44.224511+0000","last_fullsized":"2024-03-24T00:41:44.224511+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:15:30.773599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.225208+0000","last_change":"2024-03-24T00:41:42.894371+0000","last_active":"2024-03-24T00:41:44.225208+0000","last_peered":"2024-03-24T00:41:44.225208+0000","last_clean":"2024-03-24T00:41:44.225208+0000","last_became_active":"2024-03-24T00:41:42.894094+0000","last_became_peered":"2024-03-24T00:41:42.894094+0000","last_unstale":"2024-03-24T00:41:44.225208+0000","last_undegraded":"2024-03-24T00:41:44.225208+0000","last_fullsized":"2024-03-24T00:41:44.225208+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:32:00.196741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":14,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.872726+0000","last_change":"2024-03-24T00:41:41.890230+0000","last_active":"2024-03-24T00:41:43.872726+0000","last_peered":"2024-03-24T00:41:43.872726+0000","last_clean":"2024-03-24T00:41:43.872726+0000","last_became_active":"2024-03-24T00:41:41.889973+0000","last_became_peered":"2024-03-24T00:41:41.889973+0000","last_unstale":"2024-03-24T00:41:43.872726+0000","last_undegraded":"2024-03-24T00:41:43.872726+0000","last_fullsized":"2024-03-24T00:41:43.872726+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:51:59.921523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224739+0000","last_change":"2024-03-24T00:41:42.883352+0000","last_active":"2024-03-24T00:41:44.224739+0000","last_peered":"2024-03-24T00:41:44.224739+0000","last_clean":"2024-03-24T00:41:44.224739+0000","last_became_active":"2024-03-24T00:41:42.883073+0000","last_became_peered":"2024-03-24T00:41:42.883073+0000","last_unstale":"2024-03-24T00:41:44.224739+0000","last_undegraded":"2024-03-24T00:41:44.224739+0000","last_fullsized":"2024-03-24T00:41:44.224739+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:41:48.723391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":14,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224807+0000","last_change":"2024-03-24T00:41:41.886176+0000","last_active":"2024-03-24T00:41:44.224807+0000","last_peered":"2024-03-24T00:41:44.224807+0000","last_clean":"2024-03-24T00:41:44.224807+0000","last_became_active":"2024-03-24T00:41:41.885907+0000","last_became_peered":"2024-03-24T00:41:41.885907+0000","last_unstale":"2024-03-24T00:41:44.224807+0000","last_undegraded":"2024-03-24T00:41:44.224807+0000","last_fullsized":"2024-03-24T00:41:44.224807+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:58:09.746715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.874453+0000","last_change":"2024-03-24T00:41:42.896835+0000","last_active":"2024-03-24T00:41:43.874453+0000","last_peered":"2024-03-24T00:41:43.874453+0000","last_clean":"2024-03-24T00:41:43.874453+0000","last_became_active":"2024-03-24T00:41:42.896239+0000","last_became_peered":"2024-03-24T00:41:42.896239+0000","last_unstale":"2024-03-24T00:41:43.874453+0000","last_undegraded":"2024-03-24T00:41:43.874453+0000","last_fullsized":"2024-03-24T00:41:43.874453+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:27:49.707905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":14,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224241+0000","last_change":"2024-03-24T00:41:41.883441+0000","last_active":"2024-03-24T00:41:44.224241+0000","last_peered":"2024-03-24T00:41:44.224241+0000","last_clean":"2024-03-24T00:41:44.224241+0000","last_became_active":"2024-03-24T00:41:41.882544+0000","last_became_peered":"2024-03-24T00:41:41.882544+0000","last_unstale":"2024-03-24T00:41:44.224241+0000","last_undegraded":"2024-03-24T00:41:44.224241+0000","last_fullsized":"2024-03-24T00:41:44.224241+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:36:17.948951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.872262+0000","last_change":"2024-03-24T00:41:42.892776+0000","last_active":"2024-03-24T00:41:43.872262+0000","last_peered":"2024-03-24T00:41:43.872262+0000","last_clean":"2024-03-24T00:41:43.872262+0000","last_became_active":"2024-03-24T00:41:42.890453+0000","last_became_peered":"2024-03-24T00:41:42.890453+0000","last_unstale":"2024-03-24T00:41:43.872262+0000","last_undegraded":"2024-03-24T00:41:43.872262+0000","last_fullsized":"2024-03-24T00:41:43.872262+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:07:25.707558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":14,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224436+0000","last_change":"2024-03-24T00:41:41.890638+0000","last_active":"2024-03-24T00:41:44.224436+0000","last_peered":"2024-03-24T00:41:44.224436+0000","last_clean":"2024-03-24T00:41:44.224436+0000","last_became_active":"2024-03-24T00:41:41.890358+0000","last_became_peered":"2024-03-24T00:41:41.890358+0000","last_unstale":"2024-03-24T00:41:44.224436+0000","last_undegraded":"2024-03-24T00:41:44.224436+0000","last_fullsized":"2024-03-24T00:41:44.224436+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:13:09.802128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224681+0000","last_change":"2024-03-24T00:41:42.889493+0000","last_active":"2024-03-24T00:41:44.224681+0000","last_peered":"2024-03-24T00:41:44.224681+0000","last_clean":"2024-03-24T00:41:44.224681+0000","last_became_active":"2024-03-24T00:41:42.889192+0000","last_became_peered":"2024-03-24T00:41:42.889192+0000","last_unstale":"2024-03-24T00:41:44.224681+0000","last_undegraded":"2024-03-24T00:41:44.224681+0000","last_fullsized":"2024-03-24T00:41:44.224681+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:49:05.987117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":14,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:43.872260+0000","last_change":"2024-03-24T00:41:41.891095+0000","last_active":"2024-03-24T00:41:43.872260+0000","last_peered":"2024-03-24T00:41:43.872260+0000","last_clean":"2024-03-24T00:41:43.872260+0000","last_became_active":"2024-03-24T00:41:41.890825+0000","last_became_peered":"2024-03-24T00:41:41.890825+0000","last_unstale":"2024-03-24T00:41:43.872260+0000","last_undegraded":"2024-03-24T00:41:43.872260+0000","last_fullsized":"2024-03-24T00:41:43.872260+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:27:48.703921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.225412+0000","last_change":"2024-03-24T00:41:42.899591+0000","last_active":"2024-03-24T00:41:44.225412+0000","last_peered":"2024-03-24T00:41:44.225412+0000","last_clean":"2024-03-24T00:41:44.225412+0000","last_became_active":"2024-03-24T00:41:42.899309+0000","last_became_peered":"2024-03-24T00:41:42.899309+0000","last_unstale":"2024-03-24T00:41:44.225412+0000","last_undegraded":"2024-03-24T00:41:44.225412+0000","last_fullsized":"2024-03-24T00:41:44.225412+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:41.860635+0000","last_clean_scrub_stamp":"2024-03-24T00:41:41.860635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:28:46.600783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":14,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.223876+0000","last_change":"2024-03-24T00:41:41.879753+0000","last_active":"2024-03-24T00:41:44.223876+0000","last_peered":"2024-03-24T00:41:44.223876+0000","last_clean":"2024-03-24T00:41:44.223876+0000","last_became_active":"2024-03-24T00:41:41.879424+0000","last_became_peered":"2024-03-24T00:41:41.879424+0000","last_unstale":"2024-03-24T00:41:44.223876+0000","last_undegraded":"2024-03-24T00:41:44.223876+0000","last_fullsized":"2024-03-24T00:41:44.223876+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:41:40.856652+0000","last_clean_scrub_stamp":"2024-03-24T00:41:40.856652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:16:23.678772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":247,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-24T00:41:44.224820+0000","last_change":"2024-03-24T00:38:40.953871+0000","last_active":"2024-03-24T00:41:44.224820+0000","last_peered":"2024-03-24T00:41:44.224820+0000","last_clean":"2024-03-24T00:41:44.224820+0000","last_became_active":"2024-03-24T00:38:40.952926+0000","last_became_peered":"2024-03-24T00:38:40.952926+0000","last_unstale":"2024-03-24T00:41:44.224820+0000","last_undegraded":"2024-03-24T00:41:44.224820+0000","last_fullsized":"2024-03-24T00:41:44.224820+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_clean_scrub_stamp":"2024-03-24T00:38:39.722194+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:05.397257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574890,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28336,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343504,"statfs":{"total":96636764160,"available":96607748096,"internally_reserved":0,"allocated":1347584,"data_stored":922667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574889,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27744,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344096,"statfs":{"total":96636764160,"available":96608354304,"internally_reserved":0,"allocated":741376,"data_stored":331557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574889,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27744,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344096,"statfs":{"total":96636764160,"available":96608354304,"internally_reserved":0,"allocated":741376,"data_stored":331545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574888,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27752,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344088,"statfs":{"total":96636764160,"available":96608346112,"internally_reserved":0,"allocated":749568,"data_stored":333135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574888,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27740,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344100,"statfs":{"total":96636764160,"available":96608358400,"internally_reserved":0,"allocated":737280,"data_stored":331523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574889,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27756,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344084,"statfs":{"total":96636764160,"available":96608342016,"internally_reserved":0,"allocated":749568,"data_stored":338215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574889,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27768,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344072,"statfs":{"total":96636764160,"available":96608329728,"internally_reserved":0,"allocated":765952,"data_stored":333899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574888,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32488,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339352,"statfs":{"total":96636764160,"available":96603496448,"internally_reserved":0,"allocated":1331200,"data_stored":921891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:41:46.296 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-24T00:41:46.296 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-24T00:41:46.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.441+0000 7fb649a7e640 1 Processor -- start 2024-03-24T00:41:46.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.441+0000 7fb649a7e640 1 -- start start 2024-03-24T00:41:46.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.442+0000 7fb649a7e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb64415e170 0x7fb6440d21c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:46.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.442+0000 7fb649a7e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb6440cf2d0 0x7fb6440b6f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:46.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.442+0000 7fb649a7e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6440d2af0 con 0x7fb6440b9b50 2024-03-24T00:41:46.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.442+0000 7fb649a7e640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6440d2dd0 con 0x7fb6440cf2d0 2024-03-24T00:41:46.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.442+0000 7fb649a7e640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6440d30b0 con 0x7fb64415e170 2024-03-24T00:41:46.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.442+0000 7fb643fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb6440cf2d0 0x7fb6440b6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:46.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.442+0000 7fb643fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb6440cf2d0 0x7fb6440b6f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54620/0 (socket says 172.21.15.67:54620) 2024-03-24T00:41:46.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.442+0000 7fb643fff640 1 -- 172.21.15.67:0/599581903 learned_addr learned my addr 172.21.15.67:0/599581903 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:46.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.442+0000 7fb6437fe640 1 --2- 172.21.15.67:0/599581903 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb64415e170 0x7fb6440d21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:46.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.442+0000 7fb643fff640 1 -- 172.21.15.67:0/599581903 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb64415e170 msgr2=0x7fb6440d21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:46.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.442+0000 7fb643fff640 1 --2- 172.21.15.67:0/599581903 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb64415e170 0x7fb6440d21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:46.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.442+0000 7fb643fff640 1 -- 172.21.15.67:0/599581903 >> v1:172.21.15.72:6789/0 conn(0x7fb6440b9b50 legacy=0x7fb6440cebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:46.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.442+0000 7fb643fff640 1 -- 172.21.15.67:0/599581903 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6440d3390 con 0x7fb6440cf2d0 2024-03-24T00:41:46.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.443+0000 7fb643fff640 1 --2- 172.21.15.67:0/599581903 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6440cf2d0 0x7fb6440b6f20 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fb63c009e50 tx=0x7fb63c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fbb7b393c407dbc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:46.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.443+0000 7fb6427fc640 1 -- 172.21.15.67:0/599581903 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb63c02ca70 con 0x7fb6440cf2d0 2024-03-24T00:41:46.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.443+0000 7fb6427fc640 1 -- 172.21.15.67:0/599581903 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb63c02cc10 con 0x7fb6440cf2d0 2024-03-24T00:41:46.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.443+0000 7fb6427fc640 1 -- 172.21.15.67:0/599581903 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb63c0343f0 con 0x7fb6440cf2d0 2024-03-24T00:41:46.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.444+0000 7fb649a7e640 1 -- 172.21.15.67:0/599581903 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6440cf2d0 msgr2=0x7fb6440b6f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:46.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.444+0000 7fb649a7e640 1 --2- 172.21.15.67:0/599581903 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6440cf2d0 0x7fb6440b6f20 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fb63c009e50 tx=0x7fb63c02bed0 comp rx=0 tx=0).stop 2024-03-24T00:41:46.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.444+0000 7fb649a7e640 1 -- 172.21.15.67:0/599581903 shutdown_connections 2024-03-24T00:41:46.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.444+0000 7fb649a7e640 1 --2- 172.21.15.67:0/599581903 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6440cf2d0 0x7fb6440b6f20 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:46.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.444+0000 7fb649a7e640 1 --2- 172.21.15.67:0/599581903 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb64415e170 0x7fb6440d21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:46.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.444+0000 7fb649a7e640 1 -- 172.21.15.67:0/599581903 >> 172.21.15.67:0/599581903 conn(0x7fb64406c810 msgr2=0x7fb64406cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:46.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.444+0000 7fb649a7e640 1 -- 172.21.15.67:0/599581903 shutdown_connections 2024-03-24T00:41:46.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.444+0000 7fb649a7e640 1 -- 172.21.15.67:0/599581903 wait complete. 2024-03-24T00:41:46.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.445+0000 7fb649a7e640 1 Processor -- start 2024-03-24T00:41:46.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.445+0000 7fb649a7e640 1 -- start start 2024-03-24T00:41:46.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.446+0000 7fb649a7e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6440b9b50 0x7fb6440ce7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:46.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.446+0000 7fb649a7e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb6440cf2d0 0x7fb64415e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:46.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.446+0000 7fb649a7e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb64415e170 0x7fb64415ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:46.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.446+0000 7fb649a7e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6440d2880 con 0x7fb6440b9b50 2024-03-24T00:41:46.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.446+0000 7fb649a7e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6440d2700 con 0x7fb6440cf2d0 2024-03-24T00:41:46.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.446+0000 7fb649a7e640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6440b7460 con 0x7fb64415e170 2024-03-24T00:41:46.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.446+0000 7fb6437fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6440b9b50 0x7fb6440ce7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:46.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.446+0000 7fb6437fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6440b9b50 0x7fb6440ce7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54630/0 (socket says 172.21.15.67:54630) 2024-03-24T00:41:46.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.446+0000 7fb6437fe640 1 -- 172.21.15.67:0/2621217581 learned_addr learned my addr 172.21.15.67:0/2621217581 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:46.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.446+0000 7fb6437fe640 1 -- 172.21.15.67:0/2621217581 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb64415e170 msgr2=0x7fb64415ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:46.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.446+0000 7fb643fff640 1 --2- 172.21.15.67:0/2621217581 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb64415e170 0x7fb64415ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:46.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.446+0000 7fb6437fe640 1 --2- 172.21.15.67:0/2621217581 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb64415e170 0x7fb64415ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:46.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.446+0000 7fb642ffd640 1 --2- 172.21.15.67:0/2621217581 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb6440cf2d0 0x7fb64415e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:46.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.446+0000 7fb6437fe640 1 -- 172.21.15.67:0/2621217581 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb6440cf2d0 msgr2=0x7fb64415e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:46.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.446+0000 7fb6437fe640 1 --2- 172.21.15.67:0/2621217581 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb6440cf2d0 0x7fb64415e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:46.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.446+0000 7fb6437fe640 1 -- 172.21.15.67:0/2621217581 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6441507a0 con 0x7fb6440b9b50 2024-03-24T00:41:46.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.446+0000 7fb642ffd640 1 --2- 172.21.15.67:0/2621217581 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb6440cf2d0 0x7fb64415e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:41:46.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.446+0000 7fb6437fe640 1 --2- 172.21.15.67:0/2621217581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6440b9b50 0x7fb6440ce7d0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fb62c00d5d0 tx=0x7fb62c00da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:46.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.446+0000 7fb63bfff640 1 -- 172.21.15.67:0/2621217581 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb62c0125f0 con 0x7fb6440b9b50 2024-03-24T00:41:46.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.447+0000 7fb63bfff640 1 -- 172.21.15.67:0/2621217581 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb62c019070 con 0x7fb6440b9b50 2024-03-24T00:41:46.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.447+0000 7fb649a7e640 1 -- 172.21.15.67:0/2621217581 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6440ced10 con 0x7fb6440b9b50 2024-03-24T00:41:46.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.447+0000 7fb63bfff640 1 -- 172.21.15.67:0/2621217581 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb62c0129f0 con 0x7fb6440b9b50 2024-03-24T00:41:46.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.447+0000 7fb649a7e640 1 -- 172.21.15.67:0/2621217581 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6440cb860 con 0x7fb6440b9b50 2024-03-24T00:41:46.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.448+0000 7fb649a7e640 1 -- 172.21.15.67:0/2621217581 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb64404f400 con 0x7fb6440b9b50 2024-03-24T00:41:46.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.448+0000 7fb63bfff640 1 -- 172.21.15.67:0/2621217581 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb62c012b90 con 0x7fb6440b9b50 2024-03-24T00:41:46.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.449+0000 7fb63bfff640 1 --2- 172.21.15.67:0/2621217581 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb62406bc00 0x7fb62406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:46.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.449+0000 7fb63bfff640 1 -- 172.21.15.67:0/2621217581 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb62c08b010 con 0x7fb6440b9b50 2024-03-24T00:41:46.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.449+0000 7fb642ffd640 1 --2- 172.21.15.67:0/2621217581 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb62406bc00 0x7fb62406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:46.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.449+0000 7fb642ffd640 1 --2- 172.21.15.67:0/2621217581 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb62406bc00 0x7fb62406e0c0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fb6440c8780 tx=0x7fb634009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:46.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.453+0000 7fb63bfff640 1 -- 172.21.15.67:0/2621217581 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb62c059970 con 0x7fb6440b9b50 2024-03-24T00:41:46.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.789+0000 7fb649a7e640 1 -- 172.21.15.67:0/2621217581 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fb6440d0a20 con 0x7fb6440b9b50 2024-03-24T00:41:46.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.791+0000 7fb63bfff640 1 -- 172.21.15.67:0/2621217581 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) v1 ==== 190+0+147 (secure 0 0 0) 0x7fb62c059270 con 0x7fb6440b9b50 2024-03-24T00:41:46.790 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-03-24T00:41:46.791 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQDcdf9lY4nvJRAA6HWOZDWA+SSvJVmyjG+51g== 2024-03-24T00:41:46.791 INFO:teuthology.orchestra.run.smithi067.stdout: caps mds = "allow" 2024-03-24T00:41:46.791 INFO:teuthology.orchestra.run.smithi067.stdout: caps mon = "allow r" 2024-03-24T00:41:46.791 INFO:teuthology.orchestra.run.smithi067.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-24T00:41:46.791 INFO:teuthology.orchestra.run.smithi067.stderr:updated caps for client.0 2024-03-24T00:41:46.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.793+0000 7fb649a7e640 1 -- 172.21.15.67:0/2621217581 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb62406bc00 msgr2=0x7fb62406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:46.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.793+0000 7fb649a7e640 1 --2- 172.21.15.67:0/2621217581 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb62406bc00 0x7fb62406e0c0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fb6440c8780 tx=0x7fb634009040 comp rx=0 tx=0).stop 2024-03-24T00:41:46.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.793+0000 7fb649a7e640 1 -- 172.21.15.67:0/2621217581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6440b9b50 msgr2=0x7fb6440ce7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:46.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.793+0000 7fb649a7e640 1 --2- 172.21.15.67:0/2621217581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6440b9b50 0x7fb6440ce7d0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fb62c00d5d0 tx=0x7fb62c00da90 comp rx=0 tx=0).stop 2024-03-24T00:41:46.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.793+0000 7fb649a7e640 1 -- 172.21.15.67:0/2621217581 shutdown_connections 2024-03-24T00:41:46.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.793+0000 7fb649a7e640 1 --2- 172.21.15.67:0/2621217581 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb62406bc00 0x7fb62406e0c0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:46.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.793+0000 7fb649a7e640 1 --2- 172.21.15.67:0/2621217581 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb64415e170 0x7fb64415ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:46.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.793+0000 7fb649a7e640 1 --2- 172.21.15.67:0/2621217581 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb6440cf2d0 0x7fb64415e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:46.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.793+0000 7fb649a7e640 1 --2- 172.21.15.67:0/2621217581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6440b9b50 0x7fb6440ce7d0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:46.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.793+0000 7fb649a7e640 1 -- 172.21.15.67:0/2621217581 >> 172.21.15.67:0/2621217581 conn(0x7fb64406c810 msgr2=0x7fb6440b8a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:46.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.793+0000 7fb649a7e640 1 -- 172.21.15.67:0/2621217581 shutdown_connections 2024-03-24T00:41:46.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.793+0000 7fb649a7e640 1 -- 172.21.15.67:0/2621217581 wait complete. 2024-03-24T00:41:46.804 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:41:46.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.906+0000 7f66b4b60640 1 Processor -- start 2024-03-24T00:41:46.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.906+0000 7f66b4b60640 1 -- start start 2024-03-24T00:41:46.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.907+0000 7f66b4b60640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f66b01445a0 0x7f66b00cee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:46.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.907+0000 7f66b4b60640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f66b00b48d0 0x7f66b0072550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:46.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.907+0000 7f66b4b60640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66b00b4d30 con 0x7f66b006ec40 2024-03-24T00:41:46.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.907+0000 7f66b4b60640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66b00ce990 con 0x7f66b00b48d0 2024-03-24T00:41:46.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.907+0000 7f66b4b60640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66b00cddc0 con 0x7f66b01445a0 2024-03-24T00:41:46.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.907+0000 7f66add9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f66b00b48d0 0x7f66b0072550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:46.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.907+0000 7f66add9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f66b00b48d0 0x7f66b0072550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54632/0 (socket says 172.21.15.67:54632) 2024-03-24T00:41:46.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.907+0000 7f66add9b640 1 -- 172.21.15.67:0/2215723590 learned_addr learned my addr 172.21.15.67:0/2215723590 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:46.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.907+0000 7f66ae59c640 1 --2- 172.21.15.67:0/2215723590 >> v2:172.21.15.72:3300/0 conn(0x7f66b01445a0 0x7f66b00cee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:46.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.907+0000 7f66add9b640 1 -- 172.21.15.67:0/2215723590 >> v2:172.21.15.72:3300/0 conn(0x7f66b01445a0 msgr2=0x7f66b00cee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:46.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.907+0000 7f66add9b640 1 --2- 172.21.15.67:0/2215723590 >> v2:172.21.15.72:3300/0 conn(0x7f66b01445a0 0x7f66b00cee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:46.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.907+0000 7f66add9b640 1 -- 172.21.15.67:0/2215723590 >> v1:172.21.15.72:6789/0 conn(0x7f66b006ec40 legacy=0x7f66b00b2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:46.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.907+0000 7f66add9b640 1 -- 172.21.15.67:0/2215723590 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66b00ce060 con 0x7f66b00b48d0 2024-03-24T00:41:46.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.908+0000 7f66add9b640 1 --2- 172.21.15.67:0/2215723590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66b00b48d0 0x7f66b0072550 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f66b006d3f0 tx=0x7f669802e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b8a96d3013f488c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:46.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.908+0000 7f66ad59a640 1 -- 172.21.15.67:0/2215723590 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f669803b070 con 0x7f66b00b48d0 2024-03-24T00:41:46.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.908+0000 7f66ad59a640 1 -- 172.21.15.67:0/2215723590 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6698036440 con 0x7f66b00b48d0 2024-03-24T00:41:46.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.908+0000 7f66ad59a640 1 -- 172.21.15.67:0/2215723590 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6698036810 con 0x7f66b00b48d0 2024-03-24T00:41:46.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.909+0000 7f66b4b60640 1 -- 172.21.15.67:0/2215723590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66b00b48d0 msgr2=0x7f66b0072550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:46.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.909+0000 7f66b4b60640 1 --2- 172.21.15.67:0/2215723590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66b00b48d0 0x7f66b0072550 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f66b006d3f0 tx=0x7f669802e0e0 comp rx=0 tx=0).stop 2024-03-24T00:41:46.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.909+0000 7f66b4b60640 1 -- 172.21.15.67:0/2215723590 shutdown_connections 2024-03-24T00:41:46.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.909+0000 7f66b4b60640 1 --2- 172.21.15.67:0/2215723590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66b00b48d0 0x7f66b0072550 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:46.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.909+0000 7f66b4b60640 1 --2- 172.21.15.67:0/2215723590 >> v2:172.21.15.72:3300/0 conn(0x7f66b01445a0 0x7f66b00cee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:46.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.909+0000 7f66b4b60640 1 -- 172.21.15.67:0/2215723590 >> 172.21.15.67:0/2215723590 conn(0x7f66b00cf9e0 msgr2=0x7f66b00cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:46.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.909+0000 7f66b4b60640 1 -- 172.21.15.67:0/2215723590 shutdown_connections 2024-03-24T00:41:46.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.909+0000 7f66b4b60640 1 -- 172.21.15.67:0/2215723590 wait complete. 2024-03-24T00:41:46.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.911+0000 7f66b4b60640 1 Processor -- start 2024-03-24T00:41:46.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.911+0000 7f66b4b60640 1 -- start start 2024-03-24T00:41:46.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.912+0000 7f66b4b60640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f66b006ec40 0x7f66b0157810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:46.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.912+0000 7f66b4b60640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66b00b48d0 0x7f66b01f0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:46.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.912+0000 7f66b4b60640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66b01445a0 0x7f66b01405d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:46.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.912+0000 7f66b4b60640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b006f0d0 con 0x7f66b00b48d0 2024-03-24T00:41:46.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.912+0000 7f66b4b60640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b00ce620 con 0x7f66b01445a0 2024-03-24T00:41:46.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.912+0000 7f66b4b60640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66b006f250 con 0x7f66b006ec40 2024-03-24T00:41:46.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.912+0000 7f66add9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66b00b48d0 0x7f66b01f0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:46.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.912+0000 7f66add9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66b00b48d0 0x7f66b01f0850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54638/0 (socket says 172.21.15.67:54638) 2024-03-24T00:41:46.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.912+0000 7f66add9b640 1 -- 172.21.15.67:0/1792814219 learned_addr learned my addr 172.21.15.67:0/1792814219 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:46.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.912+0000 7f66ae59c640 1 --2- 172.21.15.67:0/1792814219 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f66b006ec40 0x7f66b0157810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:46.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.912+0000 7f66aed9d640 1 --2- 172.21.15.67:0/1792814219 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66b01445a0 0x7f66b01405d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:46.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.913+0000 7f66add9b640 1 -- 172.21.15.67:0/1792814219 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f66b006ec40 msgr2=0x7f66b0157810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:46.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.913+0000 7f66add9b640 1 --2- 172.21.15.67:0/1792814219 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f66b006ec40 0x7f66b0157810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:46.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.913+0000 7f66add9b640 1 -- 172.21.15.67:0/1792814219 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66b01445a0 msgr2=0x7f66b01405d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:46.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.913+0000 7f66add9b640 1 --2- 172.21.15.67:0/1792814219 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66b01445a0 0x7f66b01405d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:46.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.913+0000 7f66add9b640 1 -- 172.21.15.67:0/1792814219 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66b013ae40 con 0x7f66b00b48d0 2024-03-24T00:41:46.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.913+0000 7f66aed9d640 1 --2- 172.21.15.67:0/1792814219 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66b01445a0 0x7f66b01405d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:41:46.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.913+0000 7f66add9b640 1 --2- 172.21.15.67:0/1792814219 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66b00b48d0 0x7f66b01f0850 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f669802e120 tx=0x7f66980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:46.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.913+0000 7f66a6ffd640 1 -- 172.21.15.67:0/1792814219 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6698045020 con 0x7f66b00b48d0 2024-03-24T00:41:46.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.913+0000 7f66a6ffd640 1 -- 172.21.15.67:0/1792814219 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66980346c0 con 0x7f66b00b48d0 2024-03-24T00:41:46.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.913+0000 7f66a6ffd640 1 -- 172.21.15.67:0/1792814219 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f669803b040 con 0x7f66b00b48d0 2024-03-24T00:41:46.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.913+0000 7f66b4b60640 1 -- 172.21.15.67:0/1792814219 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66b0157d50 con 0x7f66b00b48d0 2024-03-24T00:41:46.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.913+0000 7f66b4b60640 1 -- 172.21.15.67:0/1792814219 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66b00aefc0 con 0x7f66b00b48d0 2024-03-24T00:41:46.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.914+0000 7f66b4b60640 1 -- 172.21.15.67:0/1792814219 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66b004f400 con 0x7f66b00b48d0 2024-03-24T00:41:46.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.915+0000 7f66a6ffd640 1 -- 172.21.15.67:0/1792814219 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6698048020 con 0x7f66b00b48d0 2024-03-24T00:41:46.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.915+0000 7f66a6ffd640 1 --2- 172.21.15.67:0/1792814219 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f669006bbd0 0x7f669006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:46.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.915+0000 7f66a6ffd640 1 -- 172.21.15.67:0/1792814219 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f66980af630 con 0x7f66b00b48d0 2024-03-24T00:41:46.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.916+0000 7f66ae59c640 1 --2- 172.21.15.67:0/1792814219 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f669006bbd0 0x7f669006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:46.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.916+0000 7f66ae59c640 1 --2- 172.21.15.67:0/1792814219 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f669006bbd0 0x7f669006e090 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f66a00099c0 tx=0x7f66a0003990 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:46.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:46.920+0000 7f66a6ffd640 1 -- 172.21.15.67:0/1792814219 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f669804f030 con 0x7f66b00b48d0 2024-03-24T00:41:47.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.143+0000 7f66b4b60640 1 -- 172.21.15.67:0/1792814219 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66b00cd6c0 con 0x7f66b00b48d0 2024-03-24T00:41:47.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.144+0000 7f66a6ffd640 1 -- 172.21.15.67:0/1792814219 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+4975 (secure 0 0 0) 0x7f66b00b4d30 con 0x7f66b00b48d0 2024-03-24T00:41:47.143 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:41:47.144 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":32,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5627,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2941726873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2941726873},{"type":"v1","addr":"172.21.15.72:6837","nonce":2941726873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":6166,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/1957901519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":1957901519},{"type":"v1","addr":"172.21.15.72:6836","nonce":1957901519}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":6357,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/2843754917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2843754917},{"type":"v1","addr":"172.21.15.67:6833","nonce":2843754917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:41:42.242211+0000","modified":"2024-03-24T00:41:44.237560+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6360},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6360":{"gid":6360,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/451962260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":451962260},{"type":"v1","addr":"172.21.15.67:6835","nonce":451962260}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6360,"qdb_cluster":[6360]},"id":4}]} 2024-03-24T00:41:47.144 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 32 2024-03-24T00:41:47.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.146+0000 7f66b4b60640 1 -- 172.21.15.67:0/1792814219 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f669006bbd0 msgr2=0x7f669006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:47.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.146+0000 7f66b4b60640 1 --2- 172.21.15.67:0/1792814219 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f669006bbd0 0x7f669006e090 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f66a00099c0 tx=0x7f66a0003990 comp rx=0 tx=0).stop 2024-03-24T00:41:47.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.146+0000 7f66b4b60640 1 -- 172.21.15.67:0/1792814219 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66b00b48d0 msgr2=0x7f66b01f0850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:47.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.146+0000 7f66b4b60640 1 --2- 172.21.15.67:0/1792814219 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66b00b48d0 0x7f66b01f0850 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f669802e120 tx=0x7f66980046c0 comp rx=0 tx=0).stop 2024-03-24T00:41:47.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.146+0000 7f66b4b60640 1 -- 172.21.15.67:0/1792814219 shutdown_connections 2024-03-24T00:41:47.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.146+0000 7f66b4b60640 1 --2- 172.21.15.67:0/1792814219 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f669006bbd0 0x7f669006e090 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:47.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.146+0000 7f66b4b60640 1 --2- 172.21.15.67:0/1792814219 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66b01445a0 0x7f66b01405d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:47.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.146+0000 7f66b4b60640 1 --2- 172.21.15.67:0/1792814219 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66b00b48d0 0x7f66b01f0850 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:47.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.146+0000 7f66b4b60640 1 --2- 172.21.15.67:0/1792814219 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f66b006ec40 0x7f66b0157810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:47.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.146+0000 7f66b4b60640 1 -- 172.21.15.67:0/1792814219 >> 172.21.15.67:0/1792814219 conn(0x7f66b00cf9e0 msgr2=0x7f66b00af580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:47.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.146+0000 7f66b4b60640 1 -- 172.21.15.67:0/1792814219 shutdown_connections 2024-03-24T00:41:47.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.146+0000 7f66b4b60640 1 -- 172.21.15.67:0/1792814219 wait complete. 2024-03-24T00:41:47.559 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 32, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:41:42.242211+0000', 'modified': '2024-03-24T00:41:44.237560+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6360}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6360': {'gid': 6360, 'name': 'a', 'rank': 0, 'incarnation': 30, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.67:6835/451962260', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6834', 'nonce': 451962260}, {'type': 'v1', 'addr': '172.21.15.67:6835', 'nonce': 451962260}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [10], 'metadata_pool': 9, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6360, 'qdb_cluster': [6360]} 2024-03-24T00:41:47.559 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-24T00:41:47.559 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-24T00:41:47.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.662+0000 7f1b8f685640 1 Processor -- start 2024-03-24T00:41:47.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.662+0000 7f1b8f685640 1 -- start start 2024-03-24T00:41:47.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.663+0000 7f1b8f685640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1b8815e0f0 0x7f1b880b95b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:47.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.663+0000 7f1b8f685640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b880d2230 con 0x7f1b880b9af0 2024-03-24T00:41:47.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.663+0000 7f1b8f685640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b880d2510 con 0x7f1b880b6dc0 2024-03-24T00:41:47.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.663+0000 7f1b8f685640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b880d27f0 con 0x7f1b8815e0f0 2024-03-24T00:41:47.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.663+0000 7f1b8cc20640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1b880b9af0 0x7f1b880d32d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56340/0 (socket says 172.21.15.67:56340) 2024-03-24T00:41:47.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.663+0000 7f1b8cc20640 1 -- 172.21.15.67:0/1433438704 learned_addr learned my addr 172.21.15.67:0/1433438704 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:41:47.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.663+0000 7f1b7ffff640 1 -- 172.21.15.67:0/1433438704 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4188248567 0 0) 0x7f1b880d2230 con 0x7f1b880b9af0 2024-03-24T00:41:47.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.663+0000 7f1b8d421640 1 --2- 172.21.15.67:0/1433438704 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1b8815e0f0 0x7f1b880b95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:47.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.663+0000 7f1b7ffff640 1 -- 172.21.15.67:0/1433438704 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b74003520 con 0x7f1b880b9af0 2024-03-24T00:41:47.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.663+0000 7f1b7ffff640 1 -- 172.21.15.67:0/1433438704 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1686342140 0 0) 0x7f1b74003520 con 0x7f1b880b9af0 2024-03-24T00:41:47.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.663+0000 7f1b7ffff640 1 -- 172.21.15.67:0/1433438704 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1b8815e0f0 msgr2=0x7f1b880b95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:47.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.663+0000 7f1b7ffff640 1 --2- 172.21.15.67:0/1433438704 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1b8815e0f0 0x7f1b880b95b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:47.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.663+0000 7f1b7ffff640 1 -- 172.21.15.67:0/1433438704 >> v1:172.21.15.72:6789/0 conn(0x7f1b880b6dc0 legacy=0x7f1b880d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:47.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.663+0000 7f1b7ffff640 1 -- 172.21.15.67:0/1433438704 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b880d2ad0 con 0x7f1b880b9af0 2024-03-24T00:41:47.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.663+0000 7f1b7ffff640 1 -- 172.21.15.67:0/1433438704 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f1b780017f0 con 0x7f1b880b9af0 2024-03-24T00:41:47.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.663+0000 7f1b7ffff640 1 -- 172.21.15.67:0/1433438704 >> v1:172.21.15.67:6789/0 conn(0x7f1b880b9af0 legacy=0x7f1b880d32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:47.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.663+0000 7f1b7ffff640 1 --2- 172.21.15.67:0/1433438704 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b740039e0 0x7f1b74003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:47.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.663+0000 7f1b7ffff640 1 --2- 172.21.15.67:0/1433438704 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1b8815e0f0 0x7f1b74004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:47.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.663+0000 7f1b7ffff640 1 --2- 172.21.15.67:0/1433438704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b74004d40 0x7f1b740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:47.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.663+0000 7f1b7ffff640 1 -- 172.21.15.67:0/1433438704 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b880d27f0 con 0x7f1b74004d40 2024-03-24T00:41:47.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.663+0000 7f1b7ffff640 1 -- 172.21.15.67:0/1433438704 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7400ba70 con 0x7f1b740039e0 2024-03-24T00:41:47.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.663+0000 7f1b7ffff640 1 -- 172.21.15.67:0/1433438704 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7400bc70 con 0x7f1b8815e0f0 2024-03-24T00:41:47.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.664+0000 7f1b8d421640 1 --2- 172.21.15.67:0/1433438704 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b740039e0 0x7f1b74003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:47.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.664+0000 7f1b8dc22640 1 --2- 172.21.15.67:0/1433438704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b74004d40 0x7f1b740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:47.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.664+0000 7f1b8cc20640 1 --2- 172.21.15.67:0/1433438704 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1b8815e0f0 0x7f1b74004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:47.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.664+0000 7f1b8d421640 1 -- 172.21.15.67:0/1433438704 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1b8815e0f0 msgr2=0x7f1b74004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:47.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.664+0000 7f1b8d421640 1 --2- 172.21.15.67:0/1433438704 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1b8815e0f0 0x7f1b74004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:47.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.664+0000 7f1b8d421640 1 -- 172.21.15.67:0/1433438704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b74004d40 msgr2=0x7f1b740051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:47.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.664+0000 7f1b8d421640 1 --2- 172.21.15.67:0/1433438704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b74004d40 0x7f1b740051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:47.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.664+0000 7f1b8d421640 1 -- 172.21.15.67:0/1433438704 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b74003520 con 0x7f1b740039e0 2024-03-24T00:41:47.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.664+0000 7f1b8dc22640 1 --2- 172.21.15.67:0/1433438704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b74004d40 0x7f1b740051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:41:47.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.665+0000 7f1b8d421640 1 --2- 172.21.15.67:0/1433438704 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b740039e0 0x7f1b74003e00 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f1b84009a90 tx=0x7f1b74005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b62a27263a5f787 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:47.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.665+0000 7f1b7ffff640 1 -- 172.21.15.67:0/1433438704 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b84039070 con 0x7f1b740039e0 2024-03-24T00:41:47.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.665+0000 7f1b7ffff640 1 -- 172.21.15.67:0/1433438704 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b8402c680 con 0x7f1b740039e0 2024-03-24T00:41:47.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.665+0000 7f1b7ffff640 1 -- 172.21.15.67:0/1433438704 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b84009690 con 0x7f1b740039e0 2024-03-24T00:41:47.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.665+0000 7f1b8f685640 1 -- 172.21.15.67:0/1433438704 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b740039e0 msgr2=0x7f1b74003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:47.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.665+0000 7f1b8f685640 1 --2- 172.21.15.67:0/1433438704 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b740039e0 0x7f1b74003e00 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f1b84009a90 tx=0x7f1b74005830 comp rx=0 tx=0).stop 2024-03-24T00:41:47.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.665+0000 7f1b8dc22640 1 -- 172.21.15.67:0/1433438704 reap_dead start 2024-03-24T00:41:47.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.665+0000 7f1b8f685640 1 -- 172.21.15.67:0/1433438704 shutdown_connections 2024-03-24T00:41:47.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.665+0000 7f1b8f685640 1 -- 172.21.15.67:0/1433438704 >> 172.21.15.67:0/1433438704 conn(0x7f1b88072b60 msgr2=0x7f1b88072f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:47.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.666+0000 7f1b8f685640 1 -- 172.21.15.67:0/1433438704 shutdown_connections 2024-03-24T00:41:47.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.666+0000 7f1b8f685640 1 -- 172.21.15.67:0/1433438704 wait complete. 2024-03-24T00:41:47.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.667+0000 7f1b8f685640 1 Processor -- start 2024-03-24T00:41:47.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.667+0000 7f1b8f685640 1 -- start start 2024-03-24T00:41:47.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.667+0000 7f1b8f685640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1b740039e0 0x7f1b881574b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:47.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.667+0000 7f1b8f685640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b881f4ae0 0x7f1b880b81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:47.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.667+0000 7f1b8f685640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b880b8730 0x7f1b8815e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:47.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.667+0000 7f1b8f685640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b880b3350 con 0x7f1b880b8730 2024-03-24T00:41:47.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.667+0000 7f1b8f685640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b880b31d0 con 0x7f1b881f4ae0 2024-03-24T00:41:47.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.667+0000 7f1b8f685640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b880d39e0 con 0x7f1b740039e0 2024-03-24T00:41:47.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.668+0000 7f1b8dc22640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b880b8730 0x7f1b8815e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:47.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.668+0000 7f1b8dc22640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b880b8730 0x7f1b8815e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54656/0 (socket says 172.21.15.67:54656) 2024-03-24T00:41:47.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.668+0000 7f1b8dc22640 1 -- 172.21.15.67:0/583128188 learned_addr learned my addr 172.21.15.67:0/583128188 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:47.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.668+0000 7f1b8d421640 1 --2- 172.21.15.67:0/583128188 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1b740039e0 0x7f1b881574b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:47.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.668+0000 7f1b8cc20640 1 --2- 172.21.15.67:0/583128188 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b881f4ae0 0x7f1b880b81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:47.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.668+0000 7f1b8dc22640 1 -- 172.21.15.67:0/583128188 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1b740039e0 msgr2=0x7f1b881574b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:47.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.668+0000 7f1b8dc22640 1 --2- 172.21.15.67:0/583128188 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1b740039e0 0x7f1b881574b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:47.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.668+0000 7f1b8dc22640 1 -- 172.21.15.67:0/583128188 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b881f4ae0 msgr2=0x7f1b880b81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:47.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.668+0000 7f1b8dc22640 1 --2- 172.21.15.67:0/583128188 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b881f4ae0 0x7f1b880b81f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:47.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.668+0000 7f1b8dc22640 1 -- 172.21.15.67:0/583128188 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b881579f0 con 0x7f1b880b8730 2024-03-24T00:41:47.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.668+0000 7f1b8dc22640 1 --2- 172.21.15.67:0/583128188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b880b8730 0x7f1b8815e540 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f1b880d1bb0 tx=0x7f1b8000ff70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:47.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.668+0000 7f1b7dffb640 1 -- 172.21.15.67:0/583128188 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b80001a70 con 0x7f1b880b8730 2024-03-24T00:41:47.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.668+0000 7f1b7dffb640 1 -- 172.21.15.67:0/583128188 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b80018070 con 0x7f1b880b8730 2024-03-24T00:41:47.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.668+0000 7f1b7dffb640 1 -- 172.21.15.67:0/583128188 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b80003da0 con 0x7f1b880b8730 2024-03-24T00:41:47.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.668+0000 7f1b8f685640 1 -- 172.21.15.67:0/583128188 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b8815f270 con 0x7f1b880b8730 2024-03-24T00:41:47.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.668+0000 7f1b8f685640 1 -- 172.21.15.67:0/583128188 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b8815fda0 con 0x7f1b880b8730 2024-03-24T00:41:47.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.670+0000 7f1b7dffb640 1 -- 172.21.15.67:0/583128188 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1b8000d770 con 0x7f1b880b8730 2024-03-24T00:41:47.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.670+0000 7f1b7dffb640 1 --2- 172.21.15.67:0/583128188 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1b6c06bbd0 0x7f1b6c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:47.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.670+0000 7f1b7dffb640 1 -- 172.21.15.67:0/583128188 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1b8008a950 con 0x7f1b880b8730 2024-03-24T00:41:47.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.670+0000 7f1b8d421640 1 --2- 172.21.15.67:0/583128188 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1b6c06bbd0 0x7f1b6c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:47.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.671+0000 7f1b8d421640 1 --2- 172.21.15.67:0/583128188 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1b6c06bbd0 0x7f1b6c06e090 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f1b84036000 tx=0x7f1b84030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:47.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.671+0000 7f1b92274740 1 Processor -- start 2024-03-24T00:41:47.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.671+0000 7f1b92274740 1 -- start start 2024-03-24T00:41:47.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.672+0000 7f1b92274740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560fdd9dc40 0x5560fddc4770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:47.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.672+0000 7f1b92274740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5560fddc4cb0 0x5560fdd9cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:47.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.672+0000 7f1b92274740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5560fdd9e020 0x5560fdd9d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:47.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.672+0000 7f1b92274740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560fdd01c20 con 0x5560fdd9dc40 2024-03-24T00:41:47.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.672+0000 7f1b92274740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560fdcaaa40 con 0x5560fddc4cb0 2024-03-24T00:41:47.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.672+0000 7f1b92274740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5560fdcfdcc0 con 0x5560fdd9e020 2024-03-24T00:41:47.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.672+0000 7f1b8d421640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560fdd9dc40 0x5560fddc4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:47.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.672+0000 7f1b8d421640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560fdd9dc40 0x5560fddc4770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54660/0 (socket says 172.21.15.67:54660) 2024-03-24T00:41:47.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.672+0000 7f1b8d421640 1 -- 172.21.15.67:0/1345292796 learned_addr learned my addr 172.21.15.67:0/1345292796 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:47.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.672+0000 7f1b8dc22640 1 --2- 172.21.15.67:0/1345292796 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5560fdd9e020 0x5560fdd9d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:47.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.672+0000 7f1b8cc20640 1 --2- 172.21.15.67:0/1345292796 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5560fddc4cb0 0x5560fdd9cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:47.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.672+0000 7f1b8d421640 1 -- 172.21.15.67:0/1345292796 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5560fdd9e020 msgr2=0x5560fdd9d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:47.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.672+0000 7f1b8d421640 1 --2- 172.21.15.67:0/1345292796 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5560fdd9e020 0x5560fdd9d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:47.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.672+0000 7f1b8d421640 1 -- 172.21.15.67:0/1345292796 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5560fddc4cb0 msgr2=0x5560fdd9cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:47.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.672+0000 7f1b8d421640 1 --2- 172.21.15.67:0/1345292796 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5560fddc4cb0 0x5560fdd9cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:47.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.672+0000 7f1b8d421640 1 -- 172.21.15.67:0/1345292796 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5560fdcccf90 con 0x5560fdd9dc40 2024-03-24T00:41:47.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.672+0000 7f1b8dc22640 1 --2- 172.21.15.67:0/1345292796 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5560fdd9e020 0x5560fdd9d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:41:47.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.672+0000 7f1b8cc20640 1 --2- 172.21.15.67:0/1345292796 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5560fddc4cb0 0x5560fdd9cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:41:47.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.673+0000 7f1b8d421640 1 --2- 172.21.15.67:0/1345292796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560fdd9dc40 0x5560fddc4770 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f1b84037040 tx=0x7f1b8402e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a33aaa963445d9d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:47.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.673+0000 7f1b6affd640 1 -- 172.21.15.67:0/1345292796 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b84043040 con 0x5560fdd9dc40 2024-03-24T00:41:47.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.673+0000 7f1b6affd640 1 -- 172.21.15.67:0/1345292796 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b8403a040 con 0x5560fdd9dc40 2024-03-24T00:41:47.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.673+0000 7f1b6affd640 1 -- 172.21.15.67:0/1345292796 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b84039050 con 0x5560fdd9dc40 2024-03-24T00:41:47.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.673+0000 7f1b92274740 1 -- 172.21.15.67:0/1345292796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560fdd9dc40 msgr2=0x5560fddc4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:47.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.673+0000 7f1b92274740 1 --2- 172.21.15.67:0/1345292796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560fdd9dc40 0x5560fddc4770 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f1b84037040 tx=0x7f1b8402e000 comp rx=0 tx=0).stop 2024-03-24T00:41:47.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.673+0000 7f1b92274740 1 -- 172.21.15.67:0/1345292796 shutdown_connections 2024-03-24T00:41:47.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.673+0000 7f1b92274740 1 --2- 172.21.15.67:0/1345292796 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5560fdd9e020 0x5560fdd9d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:47.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.673+0000 7f1b92274740 1 --2- 172.21.15.67:0/1345292796 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5560fddc4cb0 0x5560fdd9cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:47.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.673+0000 7f1b92274740 1 --2- 172.21.15.67:0/1345292796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560fdd9dc40 0x5560fddc4770 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:47.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.673+0000 7f1b92274740 1 -- 172.21.15.67:0/1345292796 >> 172.21.15.67:0/1345292796 conn(0x5560fdd64a60 msgr2=0x5560fdde2960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:47.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.673+0000 7f1b92274740 1 -- 172.21.15.67:0/1345292796 shutdown_connections 2024-03-24T00:41:47.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.673+0000 7f1b92274740 1 -- 172.21.15.67:0/1345292796 wait complete. 2024-03-24T00:41:47.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.674+0000 7f1b92274740 1 -- start start 2024-03-24T00:41:47.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.674+0000 7f1b92274740 1 Processor -- start 2024-03-24T00:41:47.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.675+0000 7f1b92274740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5560fdd9dc40 0x5560fdde6930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:47.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.675+0000 7f1b92274740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560fdd9e020 0x5560fddcb7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:47.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.675+0000 7f1b92274740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5560fddf6040 0x5560fdde1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:47.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.675+0000 7f1b92274740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560fddc42e0 con 0x5560fdd9e020 2024-03-24T00:41:47.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.675+0000 7f1b92274740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560fdcbde90 con 0x5560fdd9dc40 2024-03-24T00:41:47.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.675+0000 7f1b92274740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5560fddbd060 con 0x5560fddf6040 2024-03-24T00:41:47.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.675+0000 7f1b92274740 10 client.0 _ll_register_callbacks cb 0x5560fda15ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:41:47.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.675+0000 7f1b8cc20640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560fdd9e020 0x5560fddcb7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:47.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.675+0000 7f1b8cc20640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560fdd9e020 0x5560fddcb7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54676/0 (socket says 172.21.15.67:54676) 2024-03-24T00:41:47.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.675+0000 7f1b8cc20640 1 -- 172.21.15.67:0/1600665426 learned_addr learned my addr 172.21.15.67:0/1600665426 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:47.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.675+0000 7f1b8dc22640 1 --2- 172.21.15.67:0/1600665426 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5560fddf6040 0x5560fdde1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:47.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.675+0000 7f1b8d421640 1 --2- 172.21.15.67:0/1600665426 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5560fdd9dc40 0x5560fdde6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:47.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.675+0000 7f1b8cc20640 1 -- 172.21.15.67:0/1600665426 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5560fddf6040 msgr2=0x5560fdde1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:47.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.675+0000 7f1b8cc20640 1 --2- 172.21.15.67:0/1600665426 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5560fddf6040 0x5560fdde1c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:47.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.675+0000 7f1b8cc20640 1 -- 172.21.15.67:0/1600665426 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5560fdd9dc40 msgr2=0x5560fdde6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:47.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.675+0000 7f1b8cc20640 1 --2- 172.21.15.67:0/1600665426 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5560fdd9dc40 0x5560fdde6930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:47.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.675+0000 7f1b8cc20640 1 -- 172.21.15.67:0/1600665426 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5560fdcca9d0 con 0x5560fdd9e020 2024-03-24T00:41:47.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.676+0000 7f1b8cc20640 1 --2- 172.21.15.67:0/1600665426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560fdd9e020 0x5560fddcb7f0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f1b78000ef0 tx=0x7f1b78005b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d35b9d0279d71383 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:47.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.676+0000 7f1b63fff640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:41:47.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.676+0000 7f1b63fff640 1 -- 172.21.15.67:0/1600665426 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b7801a070 con 0x5560fdd9e020 2024-03-24T00:41:47.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.676+0000 7f1b63fff640 1 -- 172.21.15.67:0/1600665426 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(62..62 src has 1..62) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1b78014ad0 con 0x5560fdd9e020 2024-03-24T00:41:47.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.676+0000 7f1b92274740 1 -- 172.21.15.67:0/1600665426 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1b8403a040 con 0x5560fdd9e020 2024-03-24T00:41:47.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.676+0000 7f1b63fff640 1 -- 172.21.15.67:0/1600665426 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1b78025050 con 0x5560fdd9e020 2024-03-24T00:41:47.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.676+0000 7f1b92274740 10 client.6492 fetch_fsmap learned FSMap version 32 2024-03-24T00:41:47.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.676+0000 7f1b92274740 1 -- 172.21.15.67:0/1600665426 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=2+}) v3 -- 0x5560fda162d0 con 0x5560fdd9e020 2024-03-24T00:41:47.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.676+0000 7f1b63fff640 1 -- 172.21.15.67:0/1600665426 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b8403a040 con 0x5560fdd9e020 2024-03-24T00:41:47.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.676+0000 7f1b63fff640 1 -- 172.21.15.67:0/1600665426 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 32) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f1b78028020 con 0x5560fdd9e020 2024-03-24T00:41:47.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.676+0000 7f1b92274740 10 client.6492 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T00:41:47.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.676+0000 7f1b92274740 10 client.6492 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-24T00:41:47.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.676+0000 7f1b92274740 20 client.6492 populate_metadata read hostname 'smithi067' 2024-03-24T00:41:47.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.676+0000 7f1b92274740 1 --2- 172.21.15.67:0/1600665426 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x5560fdde03f0 0x5560fddc9370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:47.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.676+0000 7f1b92274740 4 client.6492 mds_command: new command op to 6360 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:41:47.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.676+0000 7f1b92274740 1 -- 172.21.15.67:0/1600665426 --> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5560fdd0f160 con 0x5560fdde03f0 2024-03-24T00:41:47.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.676+0000 7f1b8d421640 1 --2- 172.21.15.67:0/1600665426 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x5560fdde03f0 0x5560fddc9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:47.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.677+0000 7f1b8d421640 1 --2- 172.21.15.67:0/1600665426 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x5560fdde03f0 0x5560fddc9370 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d42468d9f44d7c7 server_cookie=18efb2545d8b2528 in_seq=0 out_seq=0 2024-03-24T00:41:47.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.677+0000 7f1b63fff640 10 client.6492 ms_handle_connect on v2:172.21.15.67:6834/451962260 2024-03-24T00:41:47.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.677+0000 7f1b63fff640 1 -- 172.21.15.67:0/1600665426 <== mds.0 v2:172.21.15.67:6834/451962260 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5560fdd0f160 con 0x5560fdde03f0 2024-03-24T00:41:47.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.677+0000 7f1b63fff640 10 client.6492 handle_command_reply: tid=0 2024-03-24T00:41:47.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.677+0000 7f1b92274740 1 client.6492 shutdown 2024-03-24T00:41:47.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.677+0000 7f1b92274740 1 -- 172.21.15.67:0/1600665426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560fdd9e020 msgr2=0x5560fddcb7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:47.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.677+0000 7f1b92274740 1 --2- 172.21.15.67:0/1600665426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560fdd9e020 0x5560fddcb7f0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f1b78000ef0 tx=0x7f1b78005b70 comp rx=0 tx=0).stop 2024-03-24T00:41:47.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.677+0000 7f1b92274740 1 -- 172.21.15.67:0/1600665426 shutdown_connections 2024-03-24T00:41:47.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.677+0000 7f1b92274740 1 --2- 172.21.15.67:0/1600665426 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x5560fdde03f0 0x5560fddc9370 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:47.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.678+0000 7f1b92274740 1 --2- 172.21.15.67:0/1600665426 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5560fddf6040 0x5560fdde1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:47.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.678+0000 7f1b92274740 1 --2- 172.21.15.67:0/1600665426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560fdd9e020 0x5560fddcb7f0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:47.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.678+0000 7f1b63fff640 0 client.6492 ms_handle_reset on v2:172.21.15.67:6834/451962260 2024-03-24T00:41:47.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.678+0000 7f1b92274740 1 --2- 172.21.15.67:0/1600665426 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5560fdd9dc40 0x5560fdde6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:47.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.678+0000 7f1b92274740 1 -- 172.21.15.67:0/1600665426 >> 172.21.15.67:0/1600665426 conn(0x5560fdd64a60 msgr2=0x5560fddc7d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:47.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.678+0000 7f1b92274740 1 -- 172.21.15.67:0/1600665426 shutdown_connections 2024-03-24T00:41:47.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.678+0000 7f1b92274740 1 -- 172.21.15.67:0/1600665426 wait complete. 2024-03-24T00:41:47.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.678+0000 7f1b92274740 20 client.6492 trim_cache size 0 max 16384 2024-03-24T00:41:47.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.695+0000 7f1b92274740 1 Processor -- start 2024-03-24T00:41:47.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.695+0000 7f1b92274740 1 -- start start 2024-03-24T00:41:47.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.695+0000 7f1b92274740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5560fdd9dc40 0x5560fddbbc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:47.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.695+0000 7f1b92274740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5560fdd9e020 0x5560fddc3880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:47.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.695+0000 7f1b92274740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560fddf6040 0x5560fdddef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:47.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.695+0000 7f1b92274740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b88200380 con 0x5560fddf6040 2024-03-24T00:41:47.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.695+0000 7f1b92274740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b88161240 con 0x5560fdd9dc40 2024-03-24T00:41:47.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.695+0000 7f1b92274740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b881ffd80 con 0x5560fdd9e020 2024-03-24T00:41:47.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.696+0000 7f1b8dc22640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560fddf6040 0x5560fdddef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:47.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.696+0000 7f1b8dc22640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560fddf6040 0x5560fdddef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54688/0 (socket says 172.21.15.67:54688) 2024-03-24T00:41:47.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.696+0000 7f1b8dc22640 1 -- 172.21.15.67:0/2445408815 learned_addr learned my addr 172.21.15.67:0/2445408815 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:47.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.696+0000 7f1b8d421640 1 --2- 172.21.15.67:0/2445408815 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5560fdd9dc40 0x5560fddbbc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:47.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.696+0000 7f1b8cc20640 1 --2- 172.21.15.67:0/2445408815 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5560fdd9e020 0x5560fddc3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:47.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.696+0000 7f1b8dc22640 1 -- 172.21.15.67:0/2445408815 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5560fdd9e020 msgr2=0x5560fddc3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:47.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.696+0000 7f1b8dc22640 1 --2- 172.21.15.67:0/2445408815 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5560fdd9e020 0x5560fddc3880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:47.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.696+0000 7f1b8dc22640 1 -- 172.21.15.67:0/2445408815 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5560fdd9dc40 msgr2=0x5560fddbbc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:47.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.696+0000 7f1b8dc22640 1 --2- 172.21.15.67:0/2445408815 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5560fdd9dc40 0x5560fddbbc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:47.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.696+0000 7f1b8dc22640 1 -- 172.21.15.67:0/2445408815 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5560fdde1ff0 con 0x5560fddf6040 2024-03-24T00:41:47.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.696+0000 7f1b8dc22640 1 --2- 172.21.15.67:0/2445408815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560fddf6040 0x5560fdddef50 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f1b8000d520 tx=0x7f1b8005b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe09141a13ed35e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:47.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.696+0000 7f1b6a7fc640 1 -- 172.21.15.67:0/2445408815 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b80061350 con 0x5560fddf6040 2024-03-24T00:41:47.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.696+0000 7f1b6a7fc640 1 -- 172.21.15.67:0/2445408815 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b80087070 con 0x5560fddf6040 2024-03-24T00:41:47.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.696+0000 7f1b6a7fc640 1 -- 172.21.15.67:0/2445408815 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b80055360 con 0x5560fddf6040 2024-03-24T00:41:47.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.697+0000 7f1b92274740 1 -- 172.21.15.67:0/2445408815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560fddf6040 msgr2=0x5560fdddef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:47.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.697+0000 7f1b92274740 1 --2- 172.21.15.67:0/2445408815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560fddf6040 0x5560fdddef50 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f1b8000d520 tx=0x7f1b8005b040 comp rx=0 tx=0).stop 2024-03-24T00:41:47.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.697+0000 7f1b92274740 1 -- 172.21.15.67:0/2445408815 shutdown_connections 2024-03-24T00:41:47.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.697+0000 7f1b92274740 1 --2- 172.21.15.67:0/2445408815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560fddf6040 0x5560fdddef50 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:47.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.697+0000 7f1b92274740 1 --2- 172.21.15.67:0/2445408815 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5560fdd9e020 0x5560fddc3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:47.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.697+0000 7f1b92274740 1 --2- 172.21.15.67:0/2445408815 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5560fdd9dc40 0x5560fddbbc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:47.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.697+0000 7f1b92274740 1 -- 172.21.15.67:0/2445408815 >> 172.21.15.67:0/2445408815 conn(0x5560fdd64a60 msgr2=0x5560fdd98a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:47.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.697+0000 7f1b92274740 1 -- 172.21.15.67:0/2445408815 shutdown_connections 2024-03-24T00:41:47.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.697+0000 7f1b92274740 1 -- 172.21.15.67:0/2445408815 wait complete. 2024-03-24T00:41:47.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.697+0000 7f1b92274740 1 -- start start 2024-03-24T00:41:47.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.697+0000 7f1b92274740 1 Processor -- start 2024-03-24T00:41:47.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.699+0000 7f1b92274740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5560fdd9dc40 0x5560fde51b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:47.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.699+0000 7f1b92274740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5560fdd9e020 0x5560fddc94a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:47.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.699+0000 7f1b92274740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560fddf6040 0x5560fddd8c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:47.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.699+0000 7f1b92274740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560fdde0270 con 0x5560fddf6040 2024-03-24T00:41:47.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.699+0000 7f1b92274740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560fddca2a0 con 0x5560fdd9dc40 2024-03-24T00:41:47.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.699+0000 7f1b92274740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5560fdde0810 con 0x5560fdd9e020 2024-03-24T00:41:47.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.699+0000 7f1b92274740 10 client.0 _ll_register_callbacks cb 0x7f1b78007c80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:41:47.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.699+0000 7f1b8dc22640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560fddf6040 0x5560fddd8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:47.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.699+0000 7f1b8dc22640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560fddf6040 0x5560fddd8c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54694/0 (socket says 172.21.15.67:54694) 2024-03-24T00:41:47.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.699+0000 7f1b8dc22640 1 -- 172.21.15.67:0/2769896223 learned_addr learned my addr 172.21.15.67:0/2769896223 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:47.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.699+0000 7f1b8cc20640 1 --2- 172.21.15.67:0/2769896223 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5560fdd9e020 0x5560fddc94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:47.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.699+0000 7f1b8d421640 1 --2- 172.21.15.67:0/2769896223 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5560fdd9dc40 0x5560fde51b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:47.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.699+0000 7f1b8dc22640 1 -- 172.21.15.67:0/2769896223 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5560fdd9e020 msgr2=0x5560fddc94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:47.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.699+0000 7f1b8dc22640 1 --2- 172.21.15.67:0/2769896223 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5560fdd9e020 0x5560fddc94a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:47.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.699+0000 7f1b8dc22640 1 -- 172.21.15.67:0/2769896223 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5560fdd9dc40 msgr2=0x5560fde51b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:47.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.699+0000 7f1b8dc22640 1 --2- 172.21.15.67:0/2769896223 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5560fdd9dc40 0x5560fde51b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:47.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.699+0000 7f1b8dc22640 1 -- 172.21.15.67:0/2769896223 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5560fdae06b0 con 0x5560fddf6040 2024-03-24T00:41:47.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.699+0000 7f1b8dc22640 1 --2- 172.21.15.67:0/2769896223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560fddf6040 0x5560fddd8c80 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f1b80058c80 tx=0x7f1b80088c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a29a900398621f7e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:47.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.699+0000 7f1b63fff640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:41:47.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.700+0000 7f1b63fff640 1 -- 172.21.15.67:0/2769896223 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b80058340 con 0x5560fddf6040 2024-03-24T00:41:47.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.700+0000 7f1b63fff640 1 -- 172.21.15.67:0/2769896223 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(62..62 src has 1..62) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1b80054a40 con 0x5560fddf6040 2024-03-24T00:41:47.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.700+0000 7f1b92274740 1 -- 172.21.15.67:0/2769896223 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1b80087070 con 0x5560fddf6040 2024-03-24T00:41:47.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.700+0000 7f1b63fff640 1 -- 172.21.15.67:0/2769896223 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1b80054c40 con 0x5560fddf6040 2024-03-24T00:41:47.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.700+0000 7f1b92274740 10 client.6498 fetch_fsmap learned FSMap version 32 2024-03-24T00:41:47.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.700+0000 7f1b92274740 1 -- 172.21.15.67:0/2769896223 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=2+}) v3 -- 0x5560fddc9ee0 con 0x5560fddf6040 2024-03-24T00:41:47.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.700+0000 7f1b63fff640 1 -- 172.21.15.67:0/2769896223 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b80087070 con 0x5560fddf6040 2024-03-24T00:41:47.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.700+0000 7f1b63fff640 1 -- 172.21.15.67:0/2769896223 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 32) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f1b80057bb0 con 0x5560fddf6040 2024-03-24T00:41:47.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.700+0000 7f1b92274740 10 client.6498 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T00:41:47.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.700+0000 7f1b92274740 10 client.6498 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-24T00:41:47.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.700+0000 7f1b92274740 20 client.6498 populate_metadata read hostname 'smithi067' 2024-03-24T00:41:47.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.700+0000 7f1b92274740 1 --2- 172.21.15.67:0/2769896223 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x5560fdddef50 0x5560fdddf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:47.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.700+0000 7f1b92274740 4 client.6498 mds_command: new command op to 6360 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:41:47.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.700+0000 7f1b92274740 1 -- 172.21.15.67:0/2769896223 --> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5560fdd094b0 con 0x5560fdddef50 2024-03-24T00:41:47.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.700+0000 7f1b8d421640 1 --2- 172.21.15.67:0/2769896223 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x5560fdddef50 0x5560fdddf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:47.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.701+0000 7f1b63fff640 10 client.6498 ms_handle_connect on v2:172.21.15.67:6834/451962260 2024-03-24T00:41:47.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.701+0000 7f1b8d421640 1 --2- 172.21.15.67:0/2769896223 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x5560fdddef50 0x5560fdddf330 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9cae628306d3ef20 server_cookie=dc99d4237c08ef88 in_seq=0 out_seq=0 2024-03-24T00:41:47.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.701+0000 7f1b63fff640 1 -- 172.21.15.67:0/2769896223 <== mds.0 v2:172.21.15.67:6834/451962260 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x5560fdd094b0 con 0x5560fdddef50 2024-03-24T00:41:47.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.701+0000 7f1b63fff640 10 client.6498 handle_command_reply: tid=0 2024-03-24T00:41:47.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.701+0000 7f1b92274740 1 client.6498 shutdown 2024-03-24T00:41:47.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.701+0000 7f1b92274740 1 -- 172.21.15.67:0/2769896223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560fddf6040 msgr2=0x5560fddd8c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:47.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.701+0000 7f1b92274740 1 --2- 172.21.15.67:0/2769896223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560fddf6040 0x5560fddd8c80 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f1b80058c80 tx=0x7f1b80088c50 comp rx=0 tx=0).stop 2024-03-24T00:41:47.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.701+0000 7f1b92274740 1 -- 172.21.15.67:0/2769896223 shutdown_connections 2024-03-24T00:41:47.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.701+0000 7f1b92274740 1 --2- 172.21.15.67:0/2769896223 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x5560fdddef50 0x5560fdddf330 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:47.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.701+0000 7f1b92274740 1 --2- 172.21.15.67:0/2769896223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560fddf6040 0x5560fddd8c80 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:47.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.701+0000 7f1b92274740 1 --2- 172.21.15.67:0/2769896223 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5560fdd9e020 0x5560fddc94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:47.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.701+0000 7f1b63fff640 0 client.6498 ms_handle_reset on v2:172.21.15.67:6834/451962260 2024-03-24T00:41:47.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.701+0000 7f1b92274740 1 --2- 172.21.15.67:0/2769896223 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5560fdd9dc40 0x5560fde51b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:47.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.701+0000 7f1b92274740 1 -- 172.21.15.67:0/2769896223 >> 172.21.15.67:0/2769896223 conn(0x5560fdd64a60 msgr2=0x5560fddc3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:47.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.701+0000 7f1b92274740 1 -- 172.21.15.67:0/2769896223 shutdown_connections 2024-03-24T00:41:47.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.701+0000 7f1b92274740 1 -- 172.21.15.67:0/2769896223 wait complete. 2024-03-24T00:41:47.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.701+0000 7f1b92274740 20 client.6498 trim_cache size 0 max 16384 2024-03-24T00:41:47.707 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:41:47.707 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "de1d0178-808e-49ca-a986-e297e523b976", 2024-03-24T00:41:47.707 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-03-24T00:41:47.707 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 6360, 2024-03-24T00:41:47.707 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-03-24T00:41:47.707 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-03-24T00:41:47.707 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-03-24T00:41:47.707 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 4.8319411040000002, 2024-03-24T00:41:47.707 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 32, 2024-03-24T00:41:47.708 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 62, 2024-03-24T00:41:47.708 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 62, 2024-03-24T00:41:47.708 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 10.184136029999999 2024-03-24T00:41:47.708 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:41:47.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.702+0000 7f1b8f685640 1 -- 172.21.15.67:0/583128188 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1b6c06bbd0 msgr2=0x7f1b6c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:47.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.702+0000 7f1b8f685640 1 --2- 172.21.15.67:0/583128188 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1b6c06bbd0 0x7f1b6c06e090 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f1b84036000 tx=0x7f1b84030000 comp rx=0 tx=0).stop 2024-03-24T00:41:47.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.702+0000 7f1b8f685640 1 -- 172.21.15.67:0/583128188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b880b8730 msgr2=0x7f1b8815e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:47.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.702+0000 7f1b8f685640 1 --2- 172.21.15.67:0/583128188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b880b8730 0x7f1b8815e540 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f1b880d1bb0 tx=0x7f1b8000ff70 comp rx=0 tx=0).stop 2024-03-24T00:41:47.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.702+0000 7f1b8f685640 1 -- 172.21.15.67:0/583128188 shutdown_connections 2024-03-24T00:41:47.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.702+0000 7f1b8f685640 1 --2- 172.21.15.67:0/583128188 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1b6c06bbd0 0x7f1b6c06e090 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:47.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.702+0000 7f1b8f685640 1 --2- 172.21.15.67:0/583128188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b880b8730 0x7f1b8815e540 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:47.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.702+0000 7f1b8f685640 1 --2- 172.21.15.67:0/583128188 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b881f4ae0 0x7f1b880b81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:47.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.702+0000 7f1b8f685640 1 --2- 172.21.15.67:0/583128188 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1b740039e0 0x7f1b881574b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:47.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.702+0000 7f1b8f685640 1 -- 172.21.15.67:0/583128188 >> 172.21.15.67:0/583128188 conn(0x7f1b88072b60 msgr2=0x7f1b880c94b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:47.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.702+0000 7f1b8f685640 1 -- 172.21.15.67:0/583128188 shutdown_connections 2024-03-24T00:41:47.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.702+0000 7f1b8f685640 1 -- 172.21.15.67:0/583128188 wait complete. 2024-03-24T00:41:47.714 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T00:41:47.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.816+0000 7fd970a7e640 1 Processor -- start 2024-03-24T00:41:47.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.816+0000 7fd970a7e640 1 -- start start 2024-03-24T00:41:47.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.816+0000 7fd970a7e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd96c15dec0 0x7fd96c0ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:47.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.816+0000 7fd970a7e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd96c0d0ef0 0x7fd96c162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:47.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.816+0000 7fd970a7e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd96c0cf640 con 0x7fd96c15e2a0 2024-03-24T00:41:47.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.816+0000 7fd970a7e640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd96c0cee30 con 0x7fd96c15dec0 2024-03-24T00:41:47.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.816+0000 7fd970a7e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd96c0d2120 con 0x7fd96c0d0ef0 2024-03-24T00:41:47.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.817+0000 7fd96a59c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd96c15dec0 0x7fd96c0ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:47.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.817+0000 7fd96a59c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd96c15dec0 0x7fd96c0ce630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54700/0 (socket says 172.21.15.67:54700) 2024-03-24T00:41:47.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.817+0000 7fd96a59c640 1 -- 172.21.15.67:0/136271689 learned_addr learned my addr 172.21.15.67:0/136271689 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:47.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.817+0000 7fd96ad9d640 1 --2- 172.21.15.67:0/136271689 >> v2:172.21.15.72:3300/0 conn(0x7fd96c0d0ef0 0x7fd96c162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:47.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.817+0000 7fd96a59c640 1 -- 172.21.15.67:0/136271689 >> v2:172.21.15.72:3300/0 conn(0x7fd96c0d0ef0 msgr2=0x7fd96c162630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:47.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.817+0000 7fd96a59c640 1 --2- 172.21.15.67:0/136271689 >> v2:172.21.15.72:3300/0 conn(0x7fd96c0d0ef0 0x7fd96c162630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:47.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.817+0000 7fd96a59c640 1 -- 172.21.15.67:0/136271689 >> v1:172.21.15.72:6789/0 conn(0x7fd96c15e2a0 legacy=0x7fd96c0d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:47.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.817+0000 7fd96a59c640 1 -- 172.21.15.67:0/136271689 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd96c0cf080 con 0x7fd96c15dec0 2024-03-24T00:41:47.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.818+0000 7fd96a59c640 1 --2- 172.21.15.67:0/136271689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd96c15dec0 0x7fd96c0ce630 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fd96c0cae20 tx=0x7fd95c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aa1e1a389434753 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:47.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.818+0000 7fd969d9b640 1 -- 172.21.15.67:0/136271689 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd95c035020 con 0x7fd96c15dec0 2024-03-24T00:41:47.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.818+0000 7fd969d9b640 1 -- 172.21.15.67:0/136271689 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd95c031870 con 0x7fd96c15dec0 2024-03-24T00:41:47.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.818+0000 7fd969d9b640 1 -- 172.21.15.67:0/136271689 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd95c031c40 con 0x7fd96c15dec0 2024-03-24T00:41:47.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.818+0000 7fd970a7e640 1 -- 172.21.15.67:0/136271689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd96c15dec0 msgr2=0x7fd96c0ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:47.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.818+0000 7fd970a7e640 1 --2- 172.21.15.67:0/136271689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd96c15dec0 0x7fd96c0ce630 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fd96c0cae20 tx=0x7fd95c02b910 comp rx=0 tx=0).stop 2024-03-24T00:41:47.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.818+0000 7fd970a7e640 1 -- 172.21.15.67:0/136271689 shutdown_connections 2024-03-24T00:41:47.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.818+0000 7fd970a7e640 1 --2- 172.21.15.67:0/136271689 >> v2:172.21.15.72:3300/0 conn(0x7fd96c0d0ef0 0x7fd96c162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:47.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.818+0000 7fd970a7e640 1 --2- 172.21.15.67:0/136271689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd96c15dec0 0x7fd96c0ce630 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:47.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.818+0000 7fd970a7e640 1 -- 172.21.15.67:0/136271689 >> 172.21.15.67:0/136271689 conn(0x7fd96c06c810 msgr2=0x7fd96c06cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:47.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.818+0000 7fd970a7e640 1 -- 172.21.15.67:0/136271689 shutdown_connections 2024-03-24T00:41:47.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.819+0000 7fd970a7e640 1 -- 172.21.15.67:0/136271689 wait complete. 2024-03-24T00:41:47.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.820+0000 7fd970a7e640 1 Processor -- start 2024-03-24T00:41:47.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.820+0000 7fd970a7e640 1 -- start start 2024-03-24T00:41:47.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.821+0000 7fd970a7e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd96c0d0ef0 0x7fd96c1f38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:47.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.821+0000 7fd970a7e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd96c15dec0 0x7fd96c0afd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:47.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.821+0000 7fd970a7e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd96c15e2a0 0x7fd96c15c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:47.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.822+0000 7fd970a7e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96c0d2730 con 0x7fd96c15e2a0 2024-03-24T00:41:47.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.822+0000 7fd970a7e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96c0d25b0 con 0x7fd96c0d0ef0 2024-03-24T00:41:47.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.822+0000 7fd970a7e640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd96c0d28b0 con 0x7fd96c15dec0 2024-03-24T00:41:47.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.822+0000 7fd96ad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd96c15e2a0 0x7fd96c15c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:47.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.822+0000 7fd96ad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd96c15e2a0 0x7fd96c15c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54710/0 (socket says 172.21.15.67:54710) 2024-03-24T00:41:47.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.822+0000 7fd96ad9d640 1 -- 172.21.15.67:0/1540392275 learned_addr learned my addr 172.21.15.67:0/1540392275 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:47.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.822+0000 7fd96a59c640 1 --2- 172.21.15.67:0/1540392275 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd96c0d0ef0 0x7fd96c1f38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:47.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.822+0000 7fd961d9b640 1 --2- 172.21.15.67:0/1540392275 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd96c15dec0 0x7fd96c0afd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:47.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.822+0000 7fd96ad9d640 1 -- 172.21.15.67:0/1540392275 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd96c15dec0 msgr2=0x7fd96c0afd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:47.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.822+0000 7fd96ad9d640 1 --2- 172.21.15.67:0/1540392275 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd96c15dec0 0x7fd96c0afd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:47.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.822+0000 7fd96ad9d640 1 -- 172.21.15.67:0/1540392275 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd96c0d0ef0 msgr2=0x7fd96c1f38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:47.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.822+0000 7fd96ad9d640 1 --2- 172.21.15.67:0/1540392275 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd96c0d0ef0 0x7fd96c1f38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:47.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.822+0000 7fd96ad9d640 1 -- 172.21.15.67:0/1540392275 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd96c1f59b0 con 0x7fd96c15e2a0 2024-03-24T00:41:47.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.822+0000 7fd961d9b640 1 --2- 172.21.15.67:0/1540392275 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd96c15dec0 0x7fd96c0afd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:41:47.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.822+0000 7fd96a59c640 1 --2- 172.21.15.67:0/1540392275 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd96c0d0ef0 0x7fd96c1f38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:41:47.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.822+0000 7fd96ad9d640 1 --2- 172.21.15.67:0/1540392275 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd96c15e2a0 0x7fd96c15c840 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fd964009dd0 tx=0x7fd96400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:47.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.822+0000 7fd9637fe640 1 -- 172.21.15.67:0/1540392275 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd964018020 con 0x7fd96c15e2a0 2024-03-24T00:41:47.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.822+0000 7fd9637fe640 1 -- 172.21.15.67:0/1540392275 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd964014480 con 0x7fd96c15e2a0 2024-03-24T00:41:47.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.822+0000 7fd970a7e640 1 -- 172.21.15.67:0/1540392275 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd96c1f5f40 con 0x7fd96c15e2a0 2024-03-24T00:41:47.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.822+0000 7fd9637fe640 1 -- 172.21.15.67:0/1540392275 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd964014870 con 0x7fd96c15e2a0 2024-03-24T00:41:47.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.822+0000 7fd970a7e640 1 -- 172.21.15.67:0/1540392275 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd96c1f6500 con 0x7fd96c15e2a0 2024-03-24T00:41:47.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.824+0000 7fd970a7e640 1 -- 172.21.15.67:0/1540392275 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd96c04f400 con 0x7fd96c15e2a0 2024-03-24T00:41:47.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.824+0000 7fd9637fe640 1 -- 172.21.15.67:0/1540392275 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd964005d10 con 0x7fd96c15e2a0 2024-03-24T00:41:47.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.824+0000 7fd9637fe640 1 --2- 172.21.15.67:0/1540392275 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd94c06bbd0 0x7fd94c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:47.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.825+0000 7fd9637fe640 1 -- 172.21.15.67:0/1540392275 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd96408ba80 con 0x7fd96c15e2a0 2024-03-24T00:41:47.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.825+0000 7fd96a59c640 1 --2- 172.21.15.67:0/1540392275 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd94c06bbd0 0x7fd94c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:47.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.825+0000 7fd96a59c640 1 --2- 172.21.15.67:0/1540392275 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd94c06bbd0 0x7fd94c06e090 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fd95c036000 tx=0x7fd95c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:47.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:47.829+0000 7fd9637fe640 1 -- 172.21.15.67:0/1540392275 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd964022030 con 0x7fd96c15e2a0 2024-03-24T00:41:48.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:48.052+0000 7fd970a7e640 1 -- 172.21.15.67:0/1540392275 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fd96c06cea0 con 0x7fd96c15e2a0 2024-03-24T00:41:48.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:48.052+0000 7fd9637fe640 1 -- 172.21.15.67:0/1540392275 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v32) v1 ==== 53+0+73 (secure 0 0 0) 0x7fd964010040 con 0x7fd96c15e2a0 2024-03-24T00:41:48.052 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-24T00:41:48.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:48.054+0000 7fd970a7e640 1 -- 172.21.15.67:0/1540392275 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd94c06bbd0 msgr2=0x7fd94c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:48.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:48.054+0000 7fd970a7e640 1 --2- 172.21.15.67:0/1540392275 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd94c06bbd0 0x7fd94c06e090 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fd95c036000 tx=0x7fd95c008030 comp rx=0 tx=0).stop 2024-03-24T00:41:48.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:48.054+0000 7fd970a7e640 1 -- 172.21.15.67:0/1540392275 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd96c15e2a0 msgr2=0x7fd96c15c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:48.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:48.054+0000 7fd970a7e640 1 --2- 172.21.15.67:0/1540392275 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd96c15e2a0 0x7fd96c15c840 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fd964009dd0 tx=0x7fd96400c490 comp rx=0 tx=0).stop 2024-03-24T00:41:48.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:48.054+0000 7fd970a7e640 1 -- 172.21.15.67:0/1540392275 shutdown_connections 2024-03-24T00:41:48.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:48.054+0000 7fd970a7e640 1 --2- 172.21.15.67:0/1540392275 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd94c06bbd0 0x7fd94c06e090 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:48.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:48.054+0000 7fd970a7e640 1 --2- 172.21.15.67:0/1540392275 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd96c15e2a0 0x7fd96c15c840 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:48.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:48.054+0000 7fd970a7e640 1 --2- 172.21.15.67:0/1540392275 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd96c15dec0 0x7fd96c0afd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:48.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:48.054+0000 7fd970a7e640 1 --2- 172.21.15.67:0/1540392275 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd96c0d0ef0 0x7fd96c1f38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:48.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:48.054+0000 7fd970a7e640 1 -- 172.21.15.67:0/1540392275 >> 172.21.15.67:0/1540392275 conn(0x7fd96c06c810 msgr2=0x7fd96c161b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:48.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:48.054+0000 7fd970a7e640 1 -- 172.21.15.67:0/1540392275 shutdown_connections 2024-03-24T00:41:48.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:48.054+0000 7fd970a7e640 1 -- 172.21.15.67:0/1540392275 wait complete. 2024-03-24T00:41:48.065 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-24T00:41:48.065 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T00:41:48.065 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-03-24T00:41:48.065 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-24T00:41:48.065 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T00:41:48.065 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-24T00:41:48.065 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-24T00:41:48.065 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T00:41:48.066 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:41:48.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:41:48.066 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-03-24T00:41:48.081 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T00:41:48.081 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T00:41:48.081 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T00:41:48.081 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:41:48.081 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-03-24T00:41:48.081 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:41:48.082 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:41:48.082 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-03-24T00:41:48.082 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-03-24T00:41:48.082 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:41:48.082 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-03-24T00:41:48.082 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-03-24T00:41:48.082 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:41:48.082 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-03-24T00:41:48.082 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T00:41:48.082 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-03-24T00:41:48.082 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-24T00:41:48.082 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 42405sec preferred_lft 42405sec 2024-03-24T00:41:48.082 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-03-24T00:41:48.083 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:41:48.083 INFO:teuthology.orchestra.run.smithi067.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T00:41:48.083 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 96:cf:26:c8:bf:62 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:41:48.083 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-24T00:41:48.083 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:41:48.083 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::7c6a:5fff:feaa:d41/64 scope link 2024-03-24T00:41:48.083 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:41:48.083 INFO:teuthology.orchestra.run.smithi067.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T00:41:48.083 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ea:22:4b:d5:48:3d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:41:48.083 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::e822:4bff:fed5:483d/64 scope link 2024-03-24T00:41:48.083 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:41:48.083 INFO:teuthology.orchestra.run.smithi067.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T00:41:48.083 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 96:cf:26:c8:bf:62 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:41:48.083 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::94cf:26ff:fec8:bf62/64 scope link 2024-03-24T00:41:48.084 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:41:48.084 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:41:48.084 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T00:41:48.146 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-03-24T00:41:48.148 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:41:48.148 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-03-24T00:41:48.205 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-03-24T00:41:48.205 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T00:41:48.206 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-24T00:41:48.206 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-24T00:41:48.206 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T00:41:48.206 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:41:48.206 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-03-24T00:41:48.206 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-03-24T00:41:48.206 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-24T00:41:48.206 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-24T00:41:48.206 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-24T00:41:48.207 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-03-24T00:41:48.299 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:41:48.299 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-03-24T00:41:48.313 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-24T00:41:48.313 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-24T00:41:48.314 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:41:48.314 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:41:48.314 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-03-24T00:41:48.383 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:41:48.383 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-24T00:41:48.437 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:41:48.438 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:41:48.438 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:41:48.491 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-24T00:41:48.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:41:48.491 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:41:48.512 INFO:tasks.ceph.mds.a.smithi067.stderr:2024-03-24T00:41:48.513+0000 7f214e7bd640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T00:41:48.555 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:41:48.555 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T00:41:48.556 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-03-24T00:41:48.563 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:41:48.564 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T00:41:48.564 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-03-24T00:41:48.564 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.72:3300,172.21.15.72:3301" 2024-03-24T00:41:48.564 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-24T00:41:48.569 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:41:48.569 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T00:41:48.569 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-03-24T00:41:48.569 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.72:3300,172.21.15.72:3301" 2024-03-24T00:41:48.569 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-24T00:41:48.569 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:6789,172.21.15.72:6789,172.21.15.72:6790" 2024-03-24T00:41:48.569 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.67:6789,172.21.15.72:6789,172.21.15.72:6790:/ 2024-03-24T00:41:48.569 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=de1d0178-808e-49ca-a986-e297e523b976" will pass to kernel 2024-03-24T00:41:48.570 INFO:teuthology.orchestra.run.smithi067.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-24T00:41:48.570 INFO:teuthology.orchestra.run.smithi067.stdout: You just mounted a file system that supports labels which does not 2024-03-24T00:41:48.570 INFO:teuthology.orchestra.run.smithi067.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-24T00:41:48.570 INFO:teuthology.orchestra.run.smithi067.stdout: applications will generate AVC messages and not be allowed access to 2024-03-24T00:41:48.570 INFO:teuthology.orchestra.run.smithi067.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-24T00:41:48.571 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-24T00:41:48.571 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:41:48.571 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:41:48.638 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:41:48.638 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T00:41:48.698 INFO:teuthology.orchestra.run.smithi067.stdout:client5732 2024-03-24T00:41:48.698 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:41:48.698 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:41:48.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:41:48.754 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir dir_test) 2024-03-24T00:41:48.812 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:41:48.812 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch dir_test/file) 2024-03-24T00:41:48.869 DEBUG:teuthology.orchestra.run.smithi067:> stdin-killer -- python3 -c ' 2024-03-24T00:41:48.870 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-03-24T00:41:48.870 DEBUG:teuthology.orchestra.run.smithi067:> import stat 2024-03-24T00:41:48.870 DEBUG:teuthology.orchestra.run.smithi067:> 2024-03-24T00:41:48.870 DEBUG:teuthology.orchestra.run.smithi067:> print(os.stat("/home/ubuntu/cephtest/mnt.0/dir_test/file").st_ino) 2024-03-24T00:41:48.870 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-03-24T00:41:48.974 INFO:teuthology.orchestra.run.smithi067.stdout:1099511627777 2024-03-24T00:41:48.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:41:48.983 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:41:49.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.126+0000 7ff0c9537640 1 Processor -- start 2024-03-24T00:41:49.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.126+0000 7ff0c9537640 1 -- start start 2024-03-24T00:41:49.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.127+0000 7ff0c9537640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff0c406cee0 0x7ff0c4099d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:49.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.127+0000 7ff0c9537640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff0c409a2b0 0x7ff0c40cfd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:49.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.127+0000 7ff0c9537640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0c40d1930 con 0x7ff0c40d1550 2024-03-24T00:41:49.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.127+0000 7ff0c9537640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0c406ca50 con 0x7ff0c409a2b0 2024-03-24T00:41:49.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.127+0000 7ff0c9537640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c40b1f60 con 0x7ff0c406cee0 2024-03-24T00:41:49.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.127+0000 7ff0c27fc640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff0c406cee0 0x7ff0c4099d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:49.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.127+0000 7ff0c37fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff0c409a2b0 0x7ff0c40cfd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:49.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.127+0000 7ff0c2ffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7ff0c40d1550 0x7ff0c40b2780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:52244/0 (socket says 172.21.15.67:52244) 2024-03-24T00:41:49.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.127+0000 7ff0c2ffd640 1 -- 172.21.15.67:0/2187089606 learned_addr learned my addr 172.21.15.67:0/2187089606 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:41:49.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.127+0000 7ff0c37fe640 1 -- 172.21.15.67:0/2187089606 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff0c406cee0 msgr2=0x7ff0c4099d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:49.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.127+0000 7ff0c37fe640 1 --2- 172.21.15.67:0/2187089606 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff0c406cee0 0x7ff0c4099d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.128+0000 7ff0c37fe640 1 -- 172.21.15.67:0/2187089606 >> v1:172.21.15.72:6789/0 conn(0x7ff0c40d1550 legacy=0x7ff0c40b2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:49.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.128+0000 7ff0c37fe640 1 -- 172.21.15.67:0/2187089606 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c40b2240 con 0x7ff0c409a2b0 2024-03-24T00:41:49.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.128+0000 7ff0c1ffb640 1 -- 172.21.15.67:0/2187089606 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4183845020 0 0) 0x7ff0c40d1930 con 0x7ff0c40d1550 2024-03-24T00:41:49.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.128+0000 7ff0c27fc640 1 --2- 172.21.15.67:0/2187089606 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff0c406cee0 0x7ff0c4099d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:41:49.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.128+0000 7ff0c37fe640 1 --2- 172.21.15.67:0/2187089606 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff0c409a2b0 0x7ff0c40cfd60 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7ff0b40097a0 tx=0x7ff0b402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a693eb1dd8181c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:49.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.128+0000 7ff0c1ffb640 1 -- 172.21.15.67:0/2187089606 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b4035020 con 0x7ff0c409a2b0 2024-03-24T00:41:49.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.129+0000 7ff0c1ffb640 1 -- 172.21.15.67:0/2187089606 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0b4031830 con 0x7ff0c409a2b0 2024-03-24T00:41:49.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.129+0000 7ff0c1ffb640 1 -- 172.21.15.67:0/2187089606 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b4031c00 con 0x7ff0c409a2b0 2024-03-24T00:41:49.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.129+0000 7ff0c9537640 1 -- 172.21.15.67:0/2187089606 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff0c409a2b0 msgr2=0x7ff0c40cfd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:49.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.129+0000 7ff0c9537640 1 --2- 172.21.15.67:0/2187089606 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff0c409a2b0 0x7ff0c40cfd60 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7ff0b40097a0 tx=0x7ff0b402b8d0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.129+0000 7ff0c9537640 1 -- 172.21.15.67:0/2187089606 shutdown_connections 2024-03-24T00:41:49.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.129+0000 7ff0c9537640 1 --2- 172.21.15.67:0/2187089606 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff0c409a2b0 0x7ff0c40cfd60 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.129+0000 7ff0c9537640 1 --2- 172.21.15.67:0/2187089606 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff0c406cee0 0x7ff0c4099d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.129+0000 7ff0c9537640 1 -- 172.21.15.67:0/2187089606 >> 172.21.15.67:0/2187089606 conn(0x7ff0c40b36e0 msgr2=0x7ff0c40b3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:49.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.129+0000 7ff0c9537640 1 -- 172.21.15.67:0/2187089606 shutdown_connections 2024-03-24T00:41:49.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.129+0000 7ff0c9537640 1 -- 172.21.15.67:0/2187089606 wait complete. 2024-03-24T00:41:49.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.131+0000 7ff0c9537640 1 Processor -- start 2024-03-24T00:41:49.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.131+0000 7ff0c9537640 1 -- start start 2024-03-24T00:41:49.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.132+0000 7ff0c9537640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff0c406cee0 0x7ff0c415b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:49.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.132+0000 7ff0c9537640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0c409a2b0 0x7ff0c41e86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:49.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.132+0000 7ff0c9537640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff0c40d1550 0x7ff0c41f8320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:49.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.132+0000 7ff0c9537640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c406c6e0 con 0x7ff0c409a2b0 2024-03-24T00:41:49.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.132+0000 7ff0c9537640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c406c400 con 0x7ff0c406cee0 2024-03-24T00:41:49.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.132+0000 7ff0c9537640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c406d360 con 0x7ff0c40d1550 2024-03-24T00:41:49.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.132+0000 7ff0c27fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0c409a2b0 0x7ff0c41e86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:49.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.132+0000 7ff0c27fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0c409a2b0 0x7ff0c41e86f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54722/0 (socket says 172.21.15.67:54722) 2024-03-24T00:41:49.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.132+0000 7ff0c27fc640 1 -- 172.21.15.67:0/4079848774 learned_addr learned my addr 172.21.15.67:0/4079848774 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:49.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.132+0000 7ff0c37fe640 1 --2- 172.21.15.67:0/4079848774 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff0c40d1550 0x7ff0c41f8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:49.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.132+0000 7ff0c2ffd640 1 --2- 172.21.15.67:0/4079848774 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff0c406cee0 0x7ff0c415b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:49.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.132+0000 7ff0c27fc640 1 -- 172.21.15.67:0/4079848774 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff0c40d1550 msgr2=0x7ff0c41f8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:49.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.132+0000 7ff0c27fc640 1 --2- 172.21.15.67:0/4079848774 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff0c40d1550 0x7ff0c41f8320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.132+0000 7ff0c27fc640 1 -- 172.21.15.67:0/4079848774 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff0c406cee0 msgr2=0x7ff0c415b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:49.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.132+0000 7ff0c27fc640 1 --2- 172.21.15.67:0/4079848774 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff0c406cee0 0x7ff0c415b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.133+0000 7ff0c27fc640 1 -- 172.21.15.67:0/4079848774 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c40b68f0 con 0x7ff0c409a2b0 2024-03-24T00:41:49.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.133+0000 7ff0c2ffd640 1 --2- 172.21.15.67:0/4079848774 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff0c406cee0 0x7ff0c415b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:41:49.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.133+0000 7ff0c27fc640 1 --2- 172.21.15.67:0/4079848774 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0c409a2b0 0x7ff0c41e86f0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac00ec10 tx=0x7ff0ac00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:49.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.133+0000 7ff0bb7fe640 1 -- 172.21.15.67:0/4079848774 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0ac017070 con 0x7ff0c409a2b0 2024-03-24T00:41:49.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.133+0000 7ff0c9537640 1 -- 172.21.15.67:0/4079848774 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0c41f8e90 con 0x7ff0c409a2b0 2024-03-24T00:41:49.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.133+0000 7ff0bb7fe640 1 -- 172.21.15.67:0/4079848774 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0ac00cc30 con 0x7ff0c409a2b0 2024-03-24T00:41:49.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.133+0000 7ff0bb7fe640 1 -- 172.21.15.67:0/4079848774 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0ac013470 con 0x7ff0c409a2b0 2024-03-24T00:41:49.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.133+0000 7ff0c9537640 1 -- 172.21.15.67:0/4079848774 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0c41e7ef0 con 0x7ff0c409a2b0 2024-03-24T00:41:49.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.134+0000 7ff0c9537640 1 -- 172.21.15.67:0/4079848774 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0c404f400 con 0x7ff0c409a2b0 2024-03-24T00:41:49.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.134+0000 7ff0bb7fe640 1 -- 172.21.15.67:0/4079848774 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff0ac01a070 con 0x7ff0c409a2b0 2024-03-24T00:41:49.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.135+0000 7ff0bb7fe640 1 --2- 172.21.15.67:0/4079848774 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff0a406bbd0 0x7ff0a406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:49.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.135+0000 7ff0bb7fe640 1 -- 172.21.15.67:0/4079848774 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff0ac08af70 con 0x7ff0c409a2b0 2024-03-24T00:41:49.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.135+0000 7ff0c2ffd640 1 --2- 172.21.15.67:0/4079848774 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff0a406bbd0 0x7ff0a406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:49.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.136+0000 7ff0c2ffd640 1 --2- 172.21.15.67:0/4079848774 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff0a406bbd0 0x7ff0a406e090 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc007fd0 tx=0x7ff0bc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:49.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.139+0000 7ff0bb7fe640 1 -- 172.21.15.67:0/4079848774 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff0ac059850 con 0x7ff0c409a2b0 2024-03-24T00:41:49.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.359+0000 7ff0c9537640 1 -- 172.21.15.67:0/4079848774 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0c40b7a70 con 0x7ff0c409a2b0 2024-03-24T00:41:49.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.359+0000 7ff0bb7fe640 1 -- 172.21.15.67:0/4079848774 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+4975 (secure 0 0 0) 0x7ff0ac059150 con 0x7ff0c409a2b0 2024-03-24T00:41:49.359 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:41:49.359 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":32,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5627,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2941726873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2941726873},{"type":"v1","addr":"172.21.15.72:6837","nonce":2941726873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":6166,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/1957901519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":1957901519},{"type":"v1","addr":"172.21.15.72:6836","nonce":1957901519}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":6357,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/2843754917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2843754917},{"type":"v1","addr":"172.21.15.67:6833","nonce":2843754917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:41:42.242211+0000","modified":"2024-03-24T00:41:44.237560+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6360},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6360":{"gid":6360,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/451962260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":451962260},{"type":"v1","addr":"172.21.15.67:6835","nonce":451962260}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6360,"qdb_cluster":[6360]},"id":4}]} 2024-03-24T00:41:49.359 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 32 2024-03-24T00:41:49.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.361+0000 7ff0c9537640 1 -- 172.21.15.67:0/4079848774 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff0a406bbd0 msgr2=0x7ff0a406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:49.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.361+0000 7ff0c9537640 1 --2- 172.21.15.67:0/4079848774 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff0a406bbd0 0x7ff0a406e090 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc007fd0 tx=0x7ff0bc009040 comp rx=0 tx=0).stop 2024-03-24T00:41:49.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.361+0000 7ff0c9537640 1 -- 172.21.15.67:0/4079848774 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0c409a2b0 msgr2=0x7ff0c41e86f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:49.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.361+0000 7ff0c9537640 1 --2- 172.21.15.67:0/4079848774 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0c409a2b0 0x7ff0c41e86f0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac00ec10 tx=0x7ff0ac00c520 comp rx=0 tx=0).stop 2024-03-24T00:41:49.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.361+0000 7ff0c9537640 1 -- 172.21.15.67:0/4079848774 shutdown_connections 2024-03-24T00:41:49.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.361+0000 7ff0c9537640 1 --2- 172.21.15.67:0/4079848774 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff0a406bbd0 0x7ff0a406e090 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.361+0000 7ff0c9537640 1 --2- 172.21.15.67:0/4079848774 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff0c40d1550 0x7ff0c41f8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.361+0000 7ff0c9537640 1 --2- 172.21.15.67:0/4079848774 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0c409a2b0 0x7ff0c41e86f0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.361+0000 7ff0c9537640 1 --2- 172.21.15.67:0/4079848774 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff0c406cee0 0x7ff0c415b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.361+0000 7ff0c9537640 1 -- 172.21.15.67:0/4079848774 >> 172.21.15.67:0/4079848774 conn(0x7ff0c40b36e0 msgr2=0x7ff0c40b2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:49.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.361+0000 7ff0c9537640 1 -- 172.21.15.67:0/4079848774 shutdown_connections 2024-03-24T00:41:49.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.361+0000 7ff0c9537640 1 -- 172.21.15.67:0/4079848774 wait complete. 2024-03-24T00:41:49.372 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-24T00:41:49.499 INFO:teuthology.orchestra.run.smithi067.stdout:{"message":"","return_code":0} 2024-03-24T00:41:49.500 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-24T00:41:49.500 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-24T00:41:49.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.542+0000 7ff64f44bf40 1 Processor -- start 2024-03-24T00:41:49.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.542+0000 7ff64f44bf40 1 -- start start 2024-03-24T00:41:49.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.542+0000 7ff64f44bf40 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff634003ab0 0x557056b950d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:49.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.542+0000 7ff64f44bf40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557056ae7c30 0x557056b77550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:49.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.542+0000 7ff64f44bf40 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x557056b01f60 0x557056abbce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:49.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.542+0000 7ff64f44bf40 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x557056a650d0 con 0x7ff634003ab0 2024-03-24T00:41:49.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.542+0000 7ff64f44bf40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x557056a69030 con 0x557056ae7c30 2024-03-24T00:41:49.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.542+0000 7ff64f44bf40 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x557056aed680 con 0x557056b01f60 2024-03-24T00:41:49.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.543+0000 7ff64c7a5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff634003ab0 0x557056b950d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:49.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.543+0000 7ff64c7a5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff634003ab0 0x557056b950d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54730/0 (socket says 172.21.15.67:54730) 2024-03-24T00:41:49.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.543+0000 7ff64c7a5640 1 -- 172.21.15.67:0/2110512600 learned_addr learned my addr 172.21.15.67:0/2110512600 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:49.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.543+0000 7ff64cfa6640 1 --2- 172.21.15.67:0/2110512600 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x557056b01f60 0x557056abbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:49.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.543+0000 7ff64bfa4640 1 --2- 172.21.15.67:0/2110512600 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557056ae7c30 0x557056b77550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:49.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.543+0000 7ff64c7a5640 1 -- 172.21.15.67:0/2110512600 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x557056b01f60 msgr2=0x557056abbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:49.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.543+0000 7ff64c7a5640 1 --2- 172.21.15.67:0/2110512600 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x557056b01f60 0x557056abbce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.543+0000 7ff64c7a5640 1 -- 172.21.15.67:0/2110512600 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557056ae7c30 msgr2=0x557056b77550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:49.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.543+0000 7ff64c7a5640 1 --2- 172.21.15.67:0/2110512600 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557056ae7c30 0x557056b77550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.543+0000 7ff64c7a5640 1 -- 172.21.15.67:0/2110512600 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557056b77cd0 con 0x7ff634003ab0 2024-03-24T00:41:49.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.543+0000 7ff64c7a5640 1 --2- 172.21.15.67:0/2110512600 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff634003ab0 0x557056b950d0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7ff63c0097b0 tx=0x7ff63c005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9281a902b2fa8ab server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:49.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.543+0000 7ff64afa2640 1 -- 172.21.15.67:0/2110512600 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff63c007830 con 0x7ff634003ab0 2024-03-24T00:41:49.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.543+0000 7ff64afa2640 1 -- 172.21.15.67:0/2110512600 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff63c0079d0 con 0x7ff634003ab0 2024-03-24T00:41:49.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.543+0000 7ff64afa2640 1 -- 172.21.15.67:0/2110512600 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff63c007d80 con 0x7ff634003ab0 2024-03-24T00:41:49.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.544+0000 7ff64f44bf40 1 -- 172.21.15.67:0/2110512600 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff634003ab0 msgr2=0x557056b950d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:49.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.544+0000 7ff64f44bf40 1 --2- 172.21.15.67:0/2110512600 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff634003ab0 0x557056b950d0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7ff63c0097b0 tx=0x7ff63c005a00 comp rx=0 tx=0).stop 2024-03-24T00:41:49.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.544+0000 7ff64f44bf40 1 -- 172.21.15.67:0/2110512600 shutdown_connections 2024-03-24T00:41:49.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.544+0000 7ff64f44bf40 1 --2- 172.21.15.67:0/2110512600 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x557056b01f60 0x557056abbce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.544+0000 7ff64f44bf40 1 --2- 172.21.15.67:0/2110512600 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557056ae7c30 0x557056b77550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.544+0000 7ff64f44bf40 1 --2- 172.21.15.67:0/2110512600 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff634003ab0 0x557056b950d0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.544+0000 7ff64f44bf40 1 -- 172.21.15.67:0/2110512600 >> 172.21.15.67:0/2110512600 conn(0x557056aeaba0 msgr2=0x557056ae9680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:49.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.544+0000 7ff64f44bf40 1 -- 172.21.15.67:0/2110512600 shutdown_connections 2024-03-24T00:41:49.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.544+0000 7ff64f44bf40 1 -- 172.21.15.67:0/2110512600 wait complete. 2024-03-24T00:41:49.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.544+0000 7ff64f44bf40 1 Processor -- start 2024-03-24T00:41:49.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.544+0000 7ff64f44bf40 1 -- start start 2024-03-24T00:41:49.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.545+0000 7ff64f44bf40 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557056ae7c30 0x557056b8e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:49.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.545+0000 7ff64f44bf40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557056b01f60 0x557056b75a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:49.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.545+0000 7ff64f44bf40 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff634003ab0 0x557056b75f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:49.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.545+0000 7ff64f44bf40 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x557056b78250 con 0x557056ae7c30 2024-03-24T00:41:49.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.545+0000 7ff64f44bf40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x557056b77a90 con 0x557056b01f60 2024-03-24T00:41:49.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.545+0000 7ff64f44bf40 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x557056b77f70 con 0x7ff634003ab0 2024-03-24T00:41:49.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.545+0000 7ff64c7a5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557056ae7c30 0x557056b8e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:49.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.545+0000 7ff64c7a5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557056ae7c30 0x557056b8e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54734/0 (socket says 172.21.15.67:54734) 2024-03-24T00:41:49.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.545+0000 7ff64c7a5640 1 -- 172.21.15.67:0/3357857117 learned_addr learned my addr 172.21.15.67:0/3357857117 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:49.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.545+0000 7ff64cfa6640 1 --2- 172.21.15.67:0/3357857117 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff634003ab0 0x557056b75f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:49.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.545+0000 7ff64bfa4640 1 --2- 172.21.15.67:0/3357857117 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557056b01f60 0x557056b75a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:49.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.545+0000 7ff64c7a5640 1 -- 172.21.15.67:0/3357857117 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff634003ab0 msgr2=0x557056b75f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:49.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.545+0000 7ff64c7a5640 1 --2- 172.21.15.67:0/3357857117 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff634003ab0 0x557056b75f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.545+0000 7ff64c7a5640 1 -- 172.21.15.67:0/3357857117 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557056b01f60 msgr2=0x557056b75a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:49.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.545+0000 7ff64c7a5640 1 --2- 172.21.15.67:0/3357857117 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557056b01f60 0x557056b75a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.545+0000 7ff64c7a5640 1 -- 172.21.15.67:0/3357857117 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557056b8e830 con 0x557056ae7c30 2024-03-24T00:41:49.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.546+0000 7ff64c7a5640 1 --2- 172.21.15.67:0/3357857117 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557056ae7c30 0x557056b8e2f0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7ff63c009d30 tx=0x7ff63c002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:49.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.546+0000 7ff649fa0640 1 -- 172.21.15.67:0/3357857117 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff63c015840 con 0x557056ae7c30 2024-03-24T00:41:49.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.546+0000 7ff649fa0640 1 -- 172.21.15.67:0/3357857117 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff63c024070 con 0x557056ae7c30 2024-03-24T00:41:49.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.546+0000 7ff64f44bf40 1 -- 172.21.15.67:0/3357857117 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x557056c1fc40 con 0x557056ae7c30 2024-03-24T00:41:49.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.546+0000 7ff649fa0640 1 -- 172.21.15.67:0/3357857117 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff63c015be0 con 0x557056ae7c30 2024-03-24T00:41:49.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.546+0000 7ff64f44bf40 1 -- 172.21.15.67:0/3357857117 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557056c20200 con 0x557056ae7c30 2024-03-24T00:41:49.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.547+0000 7ff649fa0640 1 -- 172.21.15.67:0/3357857117 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff63c01d040 con 0x557056ae7c30 2024-03-24T00:41:49.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.548+0000 7ff649fa0640 1 --2- 172.21.15.67:0/3357857117 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff63806a5e0 0x7ff63806caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:49.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.548+0000 7ff649fa0640 1 -- 172.21.15.67:0/3357857117 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff63c08b4d0 con 0x557056ae7c30 2024-03-24T00:41:49.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.548+0000 7ff64f44bf40 1 --2- 172.21.15.67:0/3357857117 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x557056b97170 0x557056c2d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:49.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.548+0000 7ff64f44bf40 1 -- 172.21.15.67:0/3357857117 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:1 10.7 10:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62) v9 -- 0x557056c2d5e0 con 0x557056b97170 2024-03-24T00:41:49.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.548+0000 7ff64bfa4640 1 --2- 172.21.15.67:0/3357857117 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff63806a5e0 0x7ff63806caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:49.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.548+0000 7ff64cfa6640 1 --2- 172.21.15.67:0/3357857117 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x557056b97170 0x557056c2d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:49.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.549+0000 7ff64cfa6640 1 --2- 172.21.15.67:0/3357857117 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x557056b97170 0x557056c2d0a0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:49.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.549+0000 7ff64bfa4640 1 --2- 172.21.15.67:0/3357857117 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff63806a5e0 0x7ff63806caa0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x557056abc9a0 tx=0x7ff6400031d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:49.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.549+0000 7ff64cfa6640 1 -- 172.21.15.67:0/3357857117 <== osd.6 v2:172.21.15.72:6816/1725206562 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (94) out=94b] v0'0 uv1 ondisk = 0) v8 ==== 164+0+94 (crc 0 0 0) 0x7ff6440017f0 con 0x557056b97170 2024-03-24T00:41:49.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.549+0000 7ff64f44bf40 1 -- 172.21.15.67:0/3357857117 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x557056b97170 msgr2=0x557056c2d0a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:49.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.549+0000 7ff64f44bf40 1 --2- 172.21.15.67:0/3357857117 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x557056b97170 0x557056c2d0a0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.549+0000 7ff64f44bf40 1 -- 172.21.15.67:0/3357857117 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff63806a5e0 msgr2=0x7ff63806caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:49.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.549+0000 7ff64f44bf40 1 --2- 172.21.15.67:0/3357857117 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff63806a5e0 0x7ff63806caa0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x557056abc9a0 tx=0x7ff6400031d0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.549+0000 7ff64f44bf40 1 -- 172.21.15.67:0/3357857117 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557056ae7c30 msgr2=0x557056b8e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:49.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.549+0000 7ff64f44bf40 1 --2- 172.21.15.67:0/3357857117 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557056ae7c30 0x557056b8e2f0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7ff63c009d30 tx=0x7ff63c002fb0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.549+0000 7ff64cfa6640 1 -- 172.21.15.67:0/3357857117 reap_dead start 2024-03-24T00:41:49.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.549+0000 7ff64f44bf40 1 -- 172.21.15.67:0/3357857117 shutdown_connections 2024-03-24T00:41:49.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.549+0000 7ff64f44bf40 1 -- 172.21.15.67:0/3357857117 >> 172.21.15.67:0/3357857117 conn(0x557056aeaba0 msgr2=0x557056b04390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:49.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.549+0000 7ff64f44bf40 1 -- 172.21.15.67:0/3357857117 shutdown_connections 2024-03-24T00:41:49.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.549+0000 7ff64f44bf40 1 -- 172.21.15.67:0/3357857117 wait complete. 2024-03-24T00:41:49.555 INFO:teuthology.orchestra.run.smithi067.stdout:Xfiledir_test 2024-03-24T00:41:49.555 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:41:49.555 DEBUG:teuthology.orchestra.run.smithi067:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-24T00:41:49.640 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:41:49.640 INFO:teuthology.orchestra.run.smithi067.stdout: "ino": 1099511627777, 2024-03-24T00:41:49.640 INFO:teuthology.orchestra.run.smithi067.stdout: "ancestors": [ 2024-03-24T00:41:49.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:41:49.640 INFO:teuthology.orchestra.run.smithi067.stdout: "dirino": 1099511627776, 2024-03-24T00:41:49.640 INFO:teuthology.orchestra.run.smithi067.stdout: "dname": "file", 2024-03-24T00:41:49.640 INFO:teuthology.orchestra.run.smithi067.stdout: "version": 2 2024-03-24T00:41:49.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:41:49.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:41:49.641 INFO:teuthology.orchestra.run.smithi067.stdout: "dirino": 1, 2024-03-24T00:41:49.641 INFO:teuthology.orchestra.run.smithi067.stdout: "dname": "dir_test", 2024-03-24T00:41:49.641 INFO:teuthology.orchestra.run.smithi067.stdout: "version": 4 2024-03-24T00:41:49.641 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:41:49.641 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-24T00:41:49.641 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 10, 2024-03-24T00:41:49.641 INFO:teuthology.orchestra.run.smithi067.stdout: "old_pools": [] 2024-03-24T00:41:49.641 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:41:49.641 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:41:49.643 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data setxattr 10000000001.00000000 parent 'The backtrace is corrupted' 2024-03-24T00:41:49.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.725+0000 7efcf4612f40 1 Processor -- start 2024-03-24T00:41:49.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.725+0000 7efcf4612f40 1 -- start start 2024-03-24T00:41:49.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.726+0000 7efcf4612f40 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cee92b03d0 0x55cee9297ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:49.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.726+0000 7efcf4612f40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55cee933acd0 0x55cee933c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:49.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.726+0000 7efcf4612f40 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55cee933d7f0 0x55cee93404c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:49.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.726+0000 7efcf4612f40 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cee933eab0 con 0x55cee92b03d0 2024-03-24T00:41:49.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.726+0000 7efcf4612f40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cee92b0840 con 0x55cee933acd0 2024-03-24T00:41:49.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.726+0000 7efcf4612f40 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cee933dfd0 con 0x55cee933d7f0 2024-03-24T00:41:49.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.726+0000 7efcf14e6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cee92b03d0 0x55cee9297ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:49.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.726+0000 7efcf14e6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cee92b03d0 0x55cee9297ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54746/0 (socket says 172.21.15.67:54746) 2024-03-24T00:41:49.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.726+0000 7efcf14e6640 1 -- 172.21.15.67:0/3433123179 learned_addr learned my addr 172.21.15.67:0/3433123179 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:49.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.726+0000 7efcf1ce7640 1 --2- 172.21.15.67:0/3433123179 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55cee933d7f0 0x55cee93404c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:49.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.726+0000 7efcf0ce5640 1 --2- 172.21.15.67:0/3433123179 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55cee933acd0 0x55cee933c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:49.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.726+0000 7efcf14e6640 1 -- 172.21.15.67:0/3433123179 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55cee933d7f0 msgr2=0x55cee93404c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:49.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.726+0000 7efcf14e6640 1 --2- 172.21.15.67:0/3433123179 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55cee933d7f0 0x55cee93404c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.726+0000 7efcf14e6640 1 -- 172.21.15.67:0/3433123179 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55cee933acd0 msgr2=0x55cee933c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:49.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.726+0000 7efcf14e6640 1 --2- 172.21.15.67:0/3433123179 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55cee933acd0 0x55cee933c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.726+0000 7efcf14e6640 1 -- 172.21.15.67:0/3433123179 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cee91aa720 con 0x55cee92b03d0 2024-03-24T00:41:49.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.726+0000 7efcf1ce7640 1 --2- 172.21.15.67:0/3433123179 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55cee933d7f0 0x55cee93404c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:41:49.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.726+0000 7efcf0ce5640 1 --2- 172.21.15.67:0/3433123179 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55cee933acd0 0x55cee933c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:41:49.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.727+0000 7efcf14e6640 1 --2- 172.21.15.67:0/3433123179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cee92b03d0 0x55cee9297ee0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7efce4009890 tx=0x7efce4005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b17812513e17c34a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:49.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.727+0000 7efce37fe640 1 -- 172.21.15.67:0/3433123179 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce4008930 con 0x55cee92b03d0 2024-03-24T00:41:49.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.727+0000 7efce37fe640 1 -- 172.21.15.67:0/3433123179 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efce4018070 con 0x55cee92b03d0 2024-03-24T00:41:49.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.727+0000 7efce37fe640 1 -- 172.21.15.67:0/3433123179 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce4008d10 con 0x55cee92b03d0 2024-03-24T00:41:49.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.727+0000 7efcf4612f40 1 -- 172.21.15.67:0/3433123179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cee92b03d0 msgr2=0x55cee9297ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:49.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.727+0000 7efcf4612f40 1 --2- 172.21.15.67:0/3433123179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cee92b03d0 0x55cee9297ee0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7efce4009890 tx=0x7efce4005a00 comp rx=0 tx=0).stop 2024-03-24T00:41:49.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.727+0000 7efcf4612f40 1 -- 172.21.15.67:0/3433123179 shutdown_connections 2024-03-24T00:41:49.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.727+0000 7efcf4612f40 1 --2- 172.21.15.67:0/3433123179 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55cee933d7f0 0x55cee93404c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.727+0000 7efcf4612f40 1 --2- 172.21.15.67:0/3433123179 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55cee933acd0 0x55cee933c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.727+0000 7efcf4612f40 1 --2- 172.21.15.67:0/3433123179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cee92b03d0 0x55cee9297ee0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.727+0000 7efcf4612f40 1 -- 172.21.15.67:0/3433123179 >> 172.21.15.67:0/3433123179 conn(0x55cee9293600 msgr2=0x55cee9296d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:49.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.727+0000 7efcf4612f40 1 -- 172.21.15.67:0/3433123179 shutdown_connections 2024-03-24T00:41:49.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.727+0000 7efcf4612f40 1 -- 172.21.15.67:0/3433123179 wait complete. 2024-03-24T00:41:49.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.728+0000 7efcf4612f40 1 Processor -- start 2024-03-24T00:41:49.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.728+0000 7efcf4612f40 1 -- start start 2024-03-24T00:41:49.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.729+0000 7efcf4612f40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55cee92b03d0 0x55cee9297eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:49.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.729+0000 7efcf4612f40 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cee933acd0 0x55cee9333bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:49.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.730+0000 7efcf4612f40 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55cee933d7f0 0x55cee9340600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:49.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.730+0000 7efcf4612f40 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cee933f930 con 0x55cee933acd0 2024-03-24T00:41:49.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.730+0000 7efcf4612f40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cee9298420 con 0x55cee92b03d0 2024-03-24T00:41:49.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.730+0000 7efcf4612f40 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cee9338880 con 0x55cee933d7f0 2024-03-24T00:41:49.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.730+0000 7efcf0ce5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cee933acd0 0x55cee9333bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:49.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.730+0000 7efcf0ce5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cee933acd0 0x55cee9333bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54748/0 (socket says 172.21.15.67:54748) 2024-03-24T00:41:49.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.730+0000 7efcf0ce5640 1 -- 172.21.15.67:0/2519454483 learned_addr learned my addr 172.21.15.67:0/2519454483 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:49.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.730+0000 7efcf14e6640 1 --2- 172.21.15.67:0/2519454483 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55cee92b03d0 0x55cee9297eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:49.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.730+0000 7efcf1ce7640 1 --2- 172.21.15.67:0/2519454483 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55cee933d7f0 0x55cee9340600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:49.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.730+0000 7efcf0ce5640 1 -- 172.21.15.67:0/2519454483 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55cee933d7f0 msgr2=0x55cee9340600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:49.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.730+0000 7efcf0ce5640 1 --2- 172.21.15.67:0/2519454483 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55cee933d7f0 0x55cee9340600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.730+0000 7efcf0ce5640 1 -- 172.21.15.67:0/2519454483 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55cee92b03d0 msgr2=0x55cee9297eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:49.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.730+0000 7efcf0ce5640 1 --2- 172.21.15.67:0/2519454483 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55cee92b03d0 0x55cee9297eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.730+0000 7efcf0ce5640 1 -- 172.21.15.67:0/2519454483 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cee9338320 con 0x55cee933acd0 2024-03-24T00:41:49.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.730+0000 7efcf1ce7640 1 --2- 172.21.15.67:0/2519454483 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55cee933d7f0 0x55cee9340600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:41:49.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.730+0000 7efcf0ce5640 1 --2- 172.21.15.67:0/2519454483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cee933acd0 0x55cee9333bd0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7efce8009f50 tx=0x7efce80084c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:49.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.730+0000 7efce27fc640 1 -- 172.21.15.67:0/2519454483 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce8008e00 con 0x55cee933acd0 2024-03-24T00:41:49.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.730+0000 7efce27fc640 1 -- 172.21.15.67:0/2519454483 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efce80143f0 con 0x55cee933acd0 2024-03-24T00:41:49.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.730+0000 7efcf4612f40 1 -- 172.21.15.67:0/2519454483 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55cee9337da0 con 0x55cee933acd0 2024-03-24T00:41:49.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.730+0000 7efce27fc640 1 -- 172.21.15.67:0/2519454483 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce80147e0 con 0x55cee933acd0 2024-03-24T00:41:49.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.730+0000 7efcf4612f40 1 -- 172.21.15.67:0/2519454483 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cee92adfa0 con 0x55cee933acd0 2024-03-24T00:41:49.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.732+0000 7efce27fc640 1 -- 172.21.15.67:0/2519454483 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7efce801b030 con 0x55cee933acd0 2024-03-24T00:41:49.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.732+0000 7efce27fc640 1 --2- 172.21.15.67:0/2519454483 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7efcd406a5b0 0x7efcd406ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:49.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.732+0000 7efce27fc640 1 -- 172.21.15.67:0/2519454483 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6101+0+0 (secure 0 0 0) 0x7efce8089290 con 0x55cee933acd0 2024-03-24T00:41:49.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.732+0000 7efcf4612f40 1 --2- 172.21.15.67:0/2519454483 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x55cee9339d40 0x55cee93e2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:49.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.732+0000 7efcf14e6640 1 --2- 172.21.15.67:0/2519454483 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7efcd406a5b0 0x7efcd406ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:49.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.732+0000 7efcf4612f40 1 -- 172.21.15.67:0/2519454483 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:1 10.7 10:e2e07930:::10000000001.00000000:head [setxattr parent (26) in=32b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62) v9 -- 0x55cee92ada30 con 0x55cee9339d40 2024-03-24T00:41:49.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.732+0000 7efcf1ce7640 1 --2- 172.21.15.67:0/2519454483 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x55cee9339d40 0x55cee93e2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:49.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.733+0000 7efcf14e6640 1 --2- 172.21.15.67:0/2519454483 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7efcd406a5b0 0x7efcd406ca70 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7efce4000c00 tx=0x7efce4012000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:49.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.733+0000 7efcf1ce7640 1 --2- 172.21.15.67:0/2519454483 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x55cee9339d40 0x55cee93e2ad0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:49.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.737+0000 7efcf1ce7640 1 -- 172.21.15.67:0/2519454483 <== osd.6 v2:172.21.15.72:6816/1725206562 1 ==== osd_op_reply(1 10000000001.00000000 [setxattr (26)] v62'2 uv2 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7efcec052720 con 0x55cee9339d40 2024-03-24T00:41:49.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.737+0000 7efcf4612f40 1 -- 172.21.15.67:0/2519454483 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x55cee9339d40 msgr2=0x55cee93e2ad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:49.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.737+0000 7efcf4612f40 1 --2- 172.21.15.67:0/2519454483 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x55cee9339d40 0x55cee93e2ad0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.737+0000 7efcf4612f40 1 -- 172.21.15.67:0/2519454483 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7efcd406a5b0 msgr2=0x7efcd406ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:49.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.737+0000 7efcf4612f40 1 --2- 172.21.15.67:0/2519454483 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7efcd406a5b0 0x7efcd406ca70 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7efce4000c00 tx=0x7efce4012000 comp rx=0 tx=0).stop 2024-03-24T00:41:49.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.737+0000 7efcf4612f40 1 -- 172.21.15.67:0/2519454483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cee933acd0 msgr2=0x55cee9333bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:49.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.737+0000 7efcf4612f40 1 --2- 172.21.15.67:0/2519454483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cee933acd0 0x55cee9333bd0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7efce8009f50 tx=0x7efce80084c0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.737+0000 7efcf1ce7640 1 -- 172.21.15.67:0/2519454483 reap_dead start 2024-03-24T00:41:49.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.737+0000 7efcf4612f40 1 -- 172.21.15.67:0/2519454483 shutdown_connections 2024-03-24T00:41:49.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.737+0000 7efcf4612f40 1 -- 172.21.15.67:0/2519454483 >> 172.21.15.67:0/2519454483 conn(0x55cee9293600 msgr2=0x55cee933c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:49.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.737+0000 7efcf4612f40 1 -- 172.21.15.67:0/2519454483 shutdown_connections 2024-03-24T00:41:49.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.737+0000 7efcf4612f40 1 -- 172.21.15.67:0/2519454483 wait complete. 2024-03-24T00:41:49.739 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:41:49.739 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.4:0 scrub start / recursive 2024-03-24T00:41:49.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.888+0000 7f5c24797640 1 Processor -- start 2024-03-24T00:41:49.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.888+0000 7f5c24797640 1 -- start start 2024-03-24T00:41:49.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.889+0000 7f5c24797640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5c1c15bed0 0x7f5c1c0725b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:49.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.889+0000 7f5c24797640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5c1c06ec40 0x7f5c1c06f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:49.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.889+0000 7f5c24797640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c1c0b4b40 con 0x7f5c1c1589a0 2024-03-24T00:41:49.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.889+0000 7f5c24797640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c1c072af0 con 0x7f5c1c06ec40 2024-03-24T00:41:49.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.889+0000 7f5c24797640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c1c06f5d0 con 0x7f5c1c15bed0 2024-03-24T00:41:49.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.889+0000 7f5c22d34640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5c1c06ec40 0x7f5c1c06f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:49.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.889+0000 7f5c22d34640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5c1c06ec40 0x7f5c1c06f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54752/0 (socket says 172.21.15.67:54752) 2024-03-24T00:41:49.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.889+0000 7f5c22d34640 1 -- 172.21.15.67:0/1685816287 learned_addr learned my addr 172.21.15.67:0/1685816287 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:49.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.889+0000 7f5c21d32640 1 --2- 172.21.15.67:0/1685816287 >> v2:172.21.15.72:3300/0 conn(0x7f5c1c15bed0 0x7f5c1c0725b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:49.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.889+0000 7f5c22d34640 1 -- 172.21.15.67:0/1685816287 >> v2:172.21.15.72:3300/0 conn(0x7f5c1c15bed0 msgr2=0x7f5c1c0725b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:49.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.889+0000 7f5c22d34640 1 --2- 172.21.15.67:0/1685816287 >> v2:172.21.15.72:3300/0 conn(0x7f5c1c15bed0 0x7f5c1c0725b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.889+0000 7f5c22d34640 1 -- 172.21.15.67:0/1685816287 >> v1:172.21.15.72:6789/0 conn(0x7f5c1c1589a0 legacy=0x7f5c1c0cc390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:49.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.889+0000 7f5c22d34640 1 -- 172.21.15.67:0/1685816287 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c1c0b4e20 con 0x7f5c1c06ec40 2024-03-24T00:41:49.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.890+0000 7f5c22d34640 1 --2- 172.21.15.67:0/1685816287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c1c06ec40 0x7f5c1c06f090 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c0b5100 tx=0x7f5c1402e180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e8d0753df857174 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:49.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.890+0000 7f5c21531640 1 -- 172.21.15.67:0/1685816287 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c1400b0d0 con 0x7f5c1c06ec40 2024-03-24T00:41:49.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.890+0000 7f5c21531640 1 -- 172.21.15.67:0/1685816287 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c1400b270 con 0x7f5c1c06ec40 2024-03-24T00:41:49.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.890+0000 7f5c21531640 1 -- 172.21.15.67:0/1685816287 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c14009600 con 0x7f5c1c06ec40 2024-03-24T00:41:49.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.891+0000 7f5c24797640 1 -- 172.21.15.67:0/1685816287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c1c06ec40 msgr2=0x7f5c1c06f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:49.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.891+0000 7f5c24797640 1 --2- 172.21.15.67:0/1685816287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c1c06ec40 0x7f5c1c06f090 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c0b5100 tx=0x7f5c1402e180 comp rx=0 tx=0).stop 2024-03-24T00:41:49.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.891+0000 7f5c24797640 1 -- 172.21.15.67:0/1685816287 shutdown_connections 2024-03-24T00:41:49.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.891+0000 7f5c24797640 1 --2- 172.21.15.67:0/1685816287 >> v2:172.21.15.72:3300/0 conn(0x7f5c1c15bed0 0x7f5c1c0725b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.891+0000 7f5c24797640 1 --2- 172.21.15.67:0/1685816287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c1c06ec40 0x7f5c1c06f090 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.891+0000 7f5c24797640 1 -- 172.21.15.67:0/1685816287 >> 172.21.15.67:0/1685816287 conn(0x7f5c1c0cf9e0 msgr2=0x7f5c1c0cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:49.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.891+0000 7f5c24797640 1 -- 172.21.15.67:0/1685816287 shutdown_connections 2024-03-24T00:41:49.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.891+0000 7f5c24797640 1 -- 172.21.15.67:0/1685816287 wait complete. 2024-03-24T00:41:49.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.892+0000 7f5c24797640 1 Processor -- start 2024-03-24T00:41:49.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.892+0000 7f5c24797640 1 -- start start 2024-03-24T00:41:49.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.893+0000 7f5c24797640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c1c06ec40 0x7f5c1c1f3de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:49.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.893+0000 7f5c24797640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5c1c1589a0 0x7f5c1c154cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:49.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.893+0000 7f5c24797640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c1c15bed0 0x7f5c1c15c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:49.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.893+0000 7f5c24797640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1c071db0 con 0x7f5c1c06ec40 2024-03-24T00:41:49.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.893+0000 7f5c24797640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1c0b48f0 con 0x7f5c1c15bed0 2024-03-24T00:41:49.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.893+0000 7f5c24797640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1c0cddc0 con 0x7f5c1c1589a0 2024-03-24T00:41:49.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.893+0000 7f5c22533640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c1c06ec40 0x7f5c1c1f3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:49.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.893+0000 7f5c22533640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c1c06ec40 0x7f5c1c1f3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54758/0 (socket says 172.21.15.67:54758) 2024-03-24T00:41:49.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.893+0000 7f5c22533640 1 -- 172.21.15.67:0/357851681 learned_addr learned my addr 172.21.15.67:0/357851681 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:49.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.893+0000 7f5c21d32640 1 --2- 172.21.15.67:0/357851681 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5c1c1589a0 0x7f5c1c154cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:49.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.893+0000 7f5c22d34640 1 --2- 172.21.15.67:0/357851681 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c1c15bed0 0x7f5c1c15c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:49.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.893+0000 7f5c22533640 1 -- 172.21.15.67:0/357851681 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5c1c1589a0 msgr2=0x7f5c1c154cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:49.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.893+0000 7f5c22533640 1 --2- 172.21.15.67:0/357851681 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5c1c1589a0 0x7f5c1c154cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.893+0000 7f5c22533640 1 -- 172.21.15.67:0/357851681 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c1c15bed0 msgr2=0x7f5c1c15c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:49.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.893+0000 7f5c22533640 1 --2- 172.21.15.67:0/357851681 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c1c15bed0 0x7f5c1c15c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.893+0000 7f5c22533640 1 -- 172.21.15.67:0/357851681 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c1c0ae0f0 con 0x7f5c1c06ec40 2024-03-24T00:41:49.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.893+0000 7f5c22533640 1 --2- 172.21.15.67:0/357851681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c1c06ec40 0x7f5c1c1f3de0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f5c1800c400 tx=0x7f5c1800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:49.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.893+0000 7f5c12ffd640 1 -- 172.21.15.67:0/357851681 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c180052d0 con 0x7f5c1c06ec40 2024-03-24T00:41:49.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.894+0000 7f5c12ffd640 1 -- 172.21.15.67:0/357851681 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c18005490 con 0x7f5c1c06ec40 2024-03-24T00:41:49.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.894+0000 7f5c24797640 1 -- 172.21.15.67:0/357851681 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c1c153820 con 0x7f5c1c06ec40 2024-03-24T00:41:49.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.894+0000 7f5c12ffd640 1 -- 172.21.15.67:0/357851681 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c18007260 con 0x7f5c1c06ec40 2024-03-24T00:41:49.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.894+0000 7f5c24797640 1 -- 172.21.15.67:0/357851681 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c1c153de0 con 0x7f5c1c06ec40 2024-03-24T00:41:49.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.895+0000 7f5c12ffd640 1 -- 172.21.15.67:0/357851681 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5c18007400 con 0x7f5c1c06ec40 2024-03-24T00:41:49.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.896+0000 7f5c12ffd640 1 --2- 172.21.15.67:0/357851681 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5c0006bc00 0x7f5c0006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:49.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.896+0000 7f5c12ffd640 1 -- 172.21.15.67:0/357851681 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5c1808a030 con 0x7f5c1c06ec40 2024-03-24T00:41:49.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.896+0000 7f5c21d32640 1 --2- 172.21.15.67:0/357851681 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5c0006bc00 0x7f5c0006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:49.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.896+0000 7f5c21d32640 1 --2- 172.21.15.67:0/357851681 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5c0006bc00 0x7f5c0006e0c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c14bc00 tx=0x7f5c0c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:49.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.897+0000 7f5c27387740 1 Processor -- start 2024-03-24T00:41:49.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.897+0000 7f5c27387740 1 -- start start 2024-03-24T00:41:49.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.897+0000 7f5c27387740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563a7116e940 0x563a71141b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:49.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.897+0000 7f5c27387740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563a7116ed20 0x563a711716d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:49.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.898+0000 7f5c27387740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a711420b0 0x563a71171c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:49.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.898+0000 7f5c27387740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a710ac070 con 0x563a711420b0 2024-03-24T00:41:49.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.898+0000 7f5c27387740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a71054bf0 con 0x563a7116ed20 2024-03-24T00:41:49.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.898+0000 7f5c27387740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x563a710a8110 con 0x563a7116e940 2024-03-24T00:41:49.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.898+0000 7f5c22d34640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a711420b0 0x563a71171c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:49.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.898+0000 7f5c22d34640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a711420b0 0x563a71171c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54766/0 (socket says 172.21.15.67:54766) 2024-03-24T00:41:49.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.898+0000 7f5c22d34640 1 -- 172.21.15.67:0/2094001719 learned_addr learned my addr 172.21.15.67:0/2094001719 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:49.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.898+0000 7f5c22533640 1 --2- 172.21.15.67:0/2094001719 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563a7116e940 0x563a71141b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:49.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.898+0000 7f5c21d32640 1 --2- 172.21.15.67:0/2094001719 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563a7116ed20 0x563a711716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:49.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.898+0000 7f5c22d34640 1 -- 172.21.15.67:0/2094001719 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563a7116e940 msgr2=0x563a71141b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:49.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.898+0000 7f5c22d34640 1 --2- 172.21.15.67:0/2094001719 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563a7116e940 0x563a71141b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.898+0000 7f5c22d34640 1 -- 172.21.15.67:0/2094001719 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563a7116ed20 msgr2=0x563a711716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:49.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.898+0000 7f5c22d34640 1 --2- 172.21.15.67:0/2094001719 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563a7116ed20 0x563a711716d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.898+0000 7f5c22d34640 1 -- 172.21.15.67:0/2094001719 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563a70e8c7b0 con 0x563a711420b0 2024-03-24T00:41:49.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.898+0000 7f5c22d34640 1 --2- 172.21.15.67:0/2094001719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a711420b0 0x563a71171c10 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f5c1400d240 tx=0x7f5c14030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=433fb32a8919d55d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:49.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.898+0000 7f5bfffff640 1 -- 172.21.15.67:0/2094001719 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c1400b2d0 con 0x563a711420b0 2024-03-24T00:41:49.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.898+0000 7f5bfffff640 1 -- 172.21.15.67:0/2094001719 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c1400b470 con 0x563a711420b0 2024-03-24T00:41:49.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.898+0000 7f5bfffff640 1 -- 172.21.15.67:0/2094001719 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c14040660 con 0x563a711420b0 2024-03-24T00:41:49.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.899+0000 7f5c27387740 1 -- 172.21.15.67:0/2094001719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a711420b0 msgr2=0x563a71171c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:49.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.899+0000 7f5c27387740 1 --2- 172.21.15.67:0/2094001719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a711420b0 0x563a71171c10 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f5c1400d240 tx=0x7f5c14030000 comp rx=0 tx=0).stop 2024-03-24T00:41:49.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.899+0000 7f5c27387740 1 -- 172.21.15.67:0/2094001719 shutdown_connections 2024-03-24T00:41:49.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.899+0000 7f5c27387740 1 --2- 172.21.15.67:0/2094001719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a711420b0 0x563a71171c10 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.899+0000 7f5c27387740 1 --2- 172.21.15.67:0/2094001719 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563a7116ed20 0x563a711716d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.899+0000 7f5c27387740 1 --2- 172.21.15.67:0/2094001719 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563a7116e940 0x563a71141b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.899+0000 7f5c27387740 1 -- 172.21.15.67:0/2094001719 >> 172.21.15.67:0/2094001719 conn(0x563a71143de0 msgr2=0x563a7118c6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:49.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.899+0000 7f5c27387740 1 -- 172.21.15.67:0/2094001719 shutdown_connections 2024-03-24T00:41:49.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.899+0000 7f5c27387740 1 -- 172.21.15.67:0/2094001719 wait complete. 2024-03-24T00:41:49.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.900+0000 7f5c27387740 1 -- start start 2024-03-24T00:41:49.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.900+0000 7f5c27387740 1 Processor -- start 2024-03-24T00:41:49.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.901+0000 7f5c27387740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563a7116e940 0x563a71167bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:49.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.901+0000 7f5c27387740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563a7116ed20 0x563a7118c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:49.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.901+0000 7f5c27387740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a7119c9d0 0x563a7118c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:49.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.901+0000 7f5c27387740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a711461f0 con 0x563a7119c9d0 2024-03-24T00:41:49.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.901+0000 7f5c27387740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a71068040 con 0x563a7116ed20 2024-03-24T00:41:49.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.901+0000 7f5c27387740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x563a71148a30 con 0x563a7116e940 2024-03-24T00:41:49.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.901+0000 7f5c27387740 10 client.0 _ll_register_callbacks cb 0x563a71053df0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:41:49.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.901+0000 7f5c22d34640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a7119c9d0 0x563a7118c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:49.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.901+0000 7f5c22d34640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a7119c9d0 0x563a7118c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54782/0 (socket says 172.21.15.67:54782) 2024-03-24T00:41:49.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.901+0000 7f5c22d34640 1 -- 172.21.15.67:0/1683459295 learned_addr learned my addr 172.21.15.67:0/1683459295 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:49.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.901+0000 7f5c21d32640 1 --2- 172.21.15.67:0/1683459295 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563a7116ed20 0x563a7118c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:49.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.901+0000 7f5c22d34640 1 -- 172.21.15.67:0/1683459295 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563a7116e940 msgr2=0x563a71167bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:49.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.901+0000 7f5c22d34640 1 --2- 172.21.15.67:0/1683459295 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563a7116e940 0x563a71167bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.901+0000 7f5c22d34640 1 -- 172.21.15.67:0/1683459295 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563a7116ed20 msgr2=0x563a7118c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:49.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.901+0000 7f5c22d34640 1 --2- 172.21.15.67:0/1683459295 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563a7116ed20 0x563a7118c390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.901+0000 7f5c22d34640 1 -- 172.21.15.67:0/1683459295 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563a70eecf40 con 0x563a7119c9d0 2024-03-24T00:41:49.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.901+0000 7f5c22d34640 1 --2- 172.21.15.67:0/1683459295 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a7119c9d0 0x563a7118c8d0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f5c14041000 tx=0x7f5c14004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e81c68dca1e2e576 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:49.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.901+0000 7f5bfd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:41:49.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.901+0000 7f5bfd7fa640 1 -- 172.21.15.67:0/1683459295 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c1404f070 con 0x563a7119c9d0 2024-03-24T00:41:49.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.901+0000 7f5bfd7fa640 1 -- 172.21.15.67:0/1683459295 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(62..62 src has 1..62) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5c14040be0 con 0x563a7119c9d0 2024-03-24T00:41:49.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.901+0000 7f5c27387740 1 -- 172.21.15.67:0/1683459295 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5c1400b470 con 0x563a7119c9d0 2024-03-24T00:41:49.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.902+0000 7f5bfd7fa640 1 -- 172.21.15.67:0/1683459295 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5c1405a050 con 0x563a7119c9d0 2024-03-24T00:41:49.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.902+0000 7f5c27387740 10 client.6537 fetch_fsmap learned FSMap version 32 2024-03-24T00:41:49.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.902+0000 7f5c27387740 1 -- 172.21.15.67:0/1683459295 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=2+}) v3 -- 0x563a70d800a0 con 0x563a7119c9d0 2024-03-24T00:41:49.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.904+0000 7f5bfd7fa640 1 -- 172.21.15.67:0/1683459295 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c1400b470 con 0x563a7119c9d0 2024-03-24T00:41:49.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.905+0000 7f5bfd7fa640 1 -- 172.21.15.67:0/1683459295 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 32) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f5c1405d020 con 0x563a7119c9d0 2024-03-24T00:41:49.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.905+0000 7f5c27387740 10 client.6537 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T00:41:49.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.905+0000 7f5c27387740 10 client.6537 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.a 2024-03-24T00:41:49.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.905+0000 7f5c27387740 20 client.6537 populate_metadata read hostname 'smithi067' 2024-03-24T00:41:49.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.905+0000 7f5c27387740 1 --2- 172.21.15.67:0/1683459295 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x563a711a0490 0x563a711a0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:49.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.905+0000 7f5c27387740 4 client.6537 mds_command: new command op to 6360 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:41:49.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.905+0000 7f5c27387740 1 -- 172.21.15.67:0/1683459295 --> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563a710b95b0 con 0x563a711a0490 2024-03-24T00:41:49.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.905+0000 7f5c22533640 1 --2- 172.21.15.67:0/1683459295 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x563a711a0490 0x563a711a0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:49.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.905+0000 7f5bfd7fa640 10 client.6537 ms_handle_connect on v2:172.21.15.67:6834/451962260 2024-03-24T00:41:49.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.905+0000 7f5c22533640 1 --2- 172.21.15.67:0/1683459295 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x563a711a0490 0x563a711a0890 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84b6a21151bbdf96 server_cookie=4639657b2b86952f in_seq=0 out_seq=0 2024-03-24T00:41:49.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.906+0000 7f5bfd7fa640 1 -- 172.21.15.67:0/1683459295 <== mds.0 v2:172.21.15.67:6834/451962260 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563a710b95b0 con 0x563a711a0490 2024-03-24T00:41:49.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.906+0000 7f5bfd7fa640 10 client.6537 handle_command_reply: tid=0 2024-03-24T00:41:49.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.906+0000 7f5c27387740 1 client.6537 shutdown 2024-03-24T00:41:49.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.906+0000 7f5c27387740 1 -- 172.21.15.67:0/1683459295 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a7119c9d0 msgr2=0x563a7118c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:49.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.906+0000 7f5c27387740 1 --2- 172.21.15.67:0/1683459295 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a7119c9d0 0x563a7118c8d0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f5c14041000 tx=0x7f5c14004ab0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.906+0000 7f5c27387740 1 -- 172.21.15.67:0/1683459295 shutdown_connections 2024-03-24T00:41:49.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.906+0000 7f5c27387740 1 --2- 172.21.15.67:0/1683459295 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x563a711a0490 0x563a711a0890 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.906+0000 7f5c27387740 1 --2- 172.21.15.67:0/1683459295 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a7119c9d0 0x563a7118c8d0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.906+0000 7f5c27387740 1 --2- 172.21.15.67:0/1683459295 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563a7116ed20 0x563a7118c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.906+0000 7f5bfd7fa640 0 client.6537 ms_handle_reset on v2:172.21.15.67:6834/451962260 2024-03-24T00:41:49.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.906+0000 7f5c27387740 1 --2- 172.21.15.67:0/1683459295 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563a7116e940 0x563a71167bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.906+0000 7f5c27387740 1 -- 172.21.15.67:0/1683459295 >> 172.21.15.67:0/1683459295 conn(0x563a711420b0 msgr2=0x563a7118fbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:49.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.906+0000 7f5c27387740 1 -- 172.21.15.67:0/1683459295 shutdown_connections 2024-03-24T00:41:49.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.906+0000 7f5c27387740 1 -- 172.21.15.67:0/1683459295 wait complete. 2024-03-24T00:41:49.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.906+0000 7f5c27387740 20 client.6537 trim_cache size 0 max 16384 2024-03-24T00:41:49.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.923+0000 7f5c27387740 1 Processor -- start 2024-03-24T00:41:49.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.923+0000 7f5c27387740 1 -- start start 2024-03-24T00:41:49.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.924+0000 7f5c27387740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a7116e940 0x563a711a0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:49.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.924+0000 7f5c27387740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563a7116ed20 0x563a7116f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:49.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.924+0000 7f5c27387740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563a7119c9d0 0x563a71174c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:49.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.924+0000 7f5c27387740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1c2007a0 con 0x563a7116e940 2024-03-24T00:41:49.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.924+0000 7f5c27387740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1c1ff920 con 0x563a7119c9d0 2024-03-24T00:41:49.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.924+0000 7f5c27387740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1c159b10 con 0x563a7116ed20 2024-03-24T00:41:49.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.924+0000 7f5c22533640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a7116e940 0x563a711a0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:49.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.924+0000 7f5c22533640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a7116e940 0x563a711a0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54790/0 (socket says 172.21.15.67:54790) 2024-03-24T00:41:49.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.924+0000 7f5c22533640 1 -- 172.21.15.67:0/3150501446 learned_addr learned my addr 172.21.15.67:0/3150501446 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:49.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.924+0000 7f5c22533640 1 -- 172.21.15.67:0/3150501446 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563a7116ed20 msgr2=0x563a7116f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:49.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.924+0000 7f5c22533640 1 --2- 172.21.15.67:0/3150501446 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563a7116ed20 0x563a7116f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.924+0000 7f5c22533640 1 -- 172.21.15.67:0/3150501446 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563a7119c9d0 msgr2=0x563a71174c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:49.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.924+0000 7f5c22533640 1 --2- 172.21.15.67:0/3150501446 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563a7119c9d0 0x563a71174c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.924+0000 7f5c22533640 1 -- 172.21.15.67:0/3150501446 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563a71147960 con 0x563a7116e940 2024-03-24T00:41:49.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.924+0000 7f5c22533640 1 --2- 172.21.15.67:0/3150501446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a7116e940 0x563a711a0870 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f5c18054830 tx=0x7f5c1800a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc3e23d7f65156d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:49.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.924+0000 7f5bff7fe640 1 -- 172.21.15.67:0/3150501446 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c18056470 con 0x563a7116e940 2024-03-24T00:41:49.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.924+0000 7f5bff7fe640 1 -- 172.21.15.67:0/3150501446 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c180566d0 con 0x563a7116e940 2024-03-24T00:41:49.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.924+0000 7f5bff7fe640 1 -- 172.21.15.67:0/3150501446 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c1805c0b0 con 0x563a7116e940 2024-03-24T00:41:49.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.925+0000 7f5c27387740 1 -- 172.21.15.67:0/3150501446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a7116e940 msgr2=0x563a711a0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:49.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.925+0000 7f5c27387740 1 --2- 172.21.15.67:0/3150501446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a7116e940 0x563a711a0870 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f5c18054830 tx=0x7f5c1800a000 comp rx=0 tx=0).stop 2024-03-24T00:41:49.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.925+0000 7f5c27387740 1 -- 172.21.15.67:0/3150501446 shutdown_connections 2024-03-24T00:41:49.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.925+0000 7f5c27387740 1 --2- 172.21.15.67:0/3150501446 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563a7119c9d0 0x563a71174c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.925+0000 7f5c27387740 1 --2- 172.21.15.67:0/3150501446 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563a7116ed20 0x563a7116f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.925+0000 7f5c27387740 1 --2- 172.21.15.67:0/3150501446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a7116e940 0x563a711a0870 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.925+0000 7f5c27387740 1 -- 172.21.15.67:0/3150501446 >> 172.21.15.67:0/3150501446 conn(0x563a711420b0 msgr2=0x563a7113cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:49.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.925+0000 7f5c27387740 1 -- 172.21.15.67:0/3150501446 shutdown_connections 2024-03-24T00:41:49.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.925+0000 7f5c27387740 1 -- 172.21.15.67:0/3150501446 wait complete. 2024-03-24T00:41:49.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.926+0000 7f5c27387740 1 -- start start 2024-03-24T00:41:49.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.926+0000 7f5c27387740 1 Processor -- start 2024-03-24T00:41:49.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.926+0000 7f5c27387740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a7116e940 0x563a7116d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:49.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.926+0000 7f5c27387740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563a7116ed20 0x563a7116d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:49.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.926+0000 7f5c27387740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563a7119c9d0 0x563a71146370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:49.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.926+0000 7f5c27387740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a710a8320 con 0x563a7116e940 2024-03-24T00:41:49.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.926+0000 7f5c27387740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a71174aa0 con 0x563a7119c9d0 2024-03-24T00:41:49.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.927+0000 7f5c27387740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x563a7119aa90 con 0x563a7116ed20 2024-03-24T00:41:49.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.927+0000 7f5c27387740 10 client.0 _ll_register_callbacks cb 0x7f5c0c0097d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:41:49.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.927+0000 7f5c22533640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a7116e940 0x563a7116d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:49.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.927+0000 7f5c22533640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a7116e940 0x563a7116d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54794/0 (socket says 172.21.15.67:54794) 2024-03-24T00:41:49.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.927+0000 7f5c22533640 1 -- 172.21.15.67:0/2439338494 learned_addr learned my addr 172.21.15.67:0/2439338494 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:49.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.927+0000 7f5c21d32640 1 --2- 172.21.15.67:0/2439338494 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563a7116ed20 0x563a7116d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:49.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.927+0000 7f5c22d34640 1 --2- 172.21.15.67:0/2439338494 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563a7119c9d0 0x563a71146370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:49.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.927+0000 7f5c22533640 1 -- 172.21.15.67:0/2439338494 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563a7116ed20 msgr2=0x563a7116d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:49.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.927+0000 7f5c22533640 1 --2- 172.21.15.67:0/2439338494 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563a7116ed20 0x563a7116d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.927+0000 7f5c22533640 1 -- 172.21.15.67:0/2439338494 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563a7119c9d0 msgr2=0x563a71146370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:49.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.927+0000 7f5c22533640 1 --2- 172.21.15.67:0/2439338494 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563a7119c9d0 0x563a71146370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.927+0000 7f5c22533640 1 -- 172.21.15.67:0/2439338494 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563a711fc960 con 0x563a7116e940 2024-03-24T00:41:49.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.927+0000 7f5c22d34640 1 --2- 172.21.15.67:0/2439338494 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563a7119c9d0 0x563a71146370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:41:49.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.927+0000 7f5c22533640 1 --2- 172.21.15.67:0/2439338494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a7116e940 0x563a7116d080 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f5c180580d0 tx=0x7f5c1806e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e7d99db360a3d4a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:49.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.927+0000 7f5bfd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:41:49.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.927+0000 7f5bfd7fa640 1 -- 172.21.15.67:0/2439338494 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c1805c0b0 con 0x563a7116e940 2024-03-24T00:41:49.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.927+0000 7f5bfd7fa640 1 -- 172.21.15.67:0/2439338494 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(62..62 src has 1..62) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5c18071c80 con 0x563a7116e940 2024-03-24T00:41:49.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.927+0000 7f5c27387740 1 -- 172.21.15.67:0/2439338494 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5c180566d0 con 0x563a7116e940 2024-03-24T00:41:49.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.927+0000 7f5bfd7fa640 1 -- 172.21.15.67:0/2439338494 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5c1805c250 con 0x563a7116e940 2024-03-24T00:41:49.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.927+0000 7f5c27387740 10 client.6543 fetch_fsmap learned FSMap version 32 2024-03-24T00:41:49.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.927+0000 7f5c27387740 1 -- 172.21.15.67:0/2439338494 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=2+}) v3 -- 0x563a7116e3b0 con 0x563a7116e940 2024-03-24T00:41:49.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.928+0000 7f5bfd7fa640 1 -- 172.21.15.67:0/2439338494 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c180566d0 con 0x563a7116e940 2024-03-24T00:41:49.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.928+0000 7f5bfd7fa640 1 -- 172.21.15.67:0/2439338494 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 32) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f5c18057820 con 0x563a7116e940 2024-03-24T00:41:49.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.928+0000 7f5c27387740 10 client.6543 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T00:41:49.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.928+0000 7f5c27387740 10 client.6543 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.a 2024-03-24T00:41:49.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.928+0000 7f5c27387740 20 client.6543 populate_metadata read hostname 'smithi067' 2024-03-24T00:41:49.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.928+0000 7f5c27387740 1 --2- 172.21.15.67:0/2439338494 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x563a711739f0 0x563a711468b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:49.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.928+0000 7f5c27387740 4 client.6543 mds_command: new command op to 6360 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["recursive"]}] 2024-03-24T00:41:49.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.928+0000 7f5c27387740 1 -- 172.21.15.67:0/2439338494 --> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["recursive"]}) v1 -- 0x563a710b3900 con 0x563a711739f0 2024-03-24T00:41:49.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.928+0000 7f5c21d32640 1 --2- 172.21.15.67:0/2439338494 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x563a711739f0 0x563a711468b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:49.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.928+0000 7f5c21d32640 1 --2- 172.21.15.67:0/2439338494 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x563a711739f0 0x563a711468b0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96d1cda574e38896 server_cookie=3841d3f4f27e60d3 in_seq=0 out_seq=0 2024-03-24T00:41:49.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.928+0000 7f5bfd7fa640 10 client.6543 ms_handle_connect on v2:172.21.15.67:6834/451962260 2024-03-24T00:41:49.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.929+0000 7f5bfd7fa640 1 -- 172.21.15.67:0/2439338494 <== mds.0 v2:172.21.15.67:6834/451962260 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x563a710b3900 con 0x563a711739f0 2024-03-24T00:41:49.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.929+0000 7f5bfd7fa640 10 client.6543 handle_command_reply: tid=0 2024-03-24T00:41:49.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.929+0000 7f5c27387740 1 client.6543 shutdown 2024-03-24T00:41:49.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.929+0000 7f5c27387740 1 -- 172.21.15.67:0/2439338494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a7116e940 msgr2=0x563a7116d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:49.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.929+0000 7f5c27387740 1 --2- 172.21.15.67:0/2439338494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a7116e940 0x563a7116d080 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f5c180580d0 tx=0x7f5c1806e040 comp rx=0 tx=0).stop 2024-03-24T00:41:49.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.929+0000 7f5c27387740 1 -- 172.21.15.67:0/2439338494 shutdown_connections 2024-03-24T00:41:49.934 INFO:tasks.ceph.mds.a.smithi067.stderr:2024-03-24T00:41:49.929+0000 7f214a7b5640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [2,head] /dir_test/file auth v2 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={5732=0-4194304@1} caps={5732=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=5732 | request=0 caps=1 dirtyparent=0 dirty=0 authpin=1 scrubqueue=0 0x55fd9feebb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(10)0x10000000001:[<0x10000000000/file v2>,<0x1/dir_test v4>]//[]","error_str":"failed to decode on-disk backtrace (26 bytes)!"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-24T00:41:49.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.929+0000 7f5c27387740 1 --2- 172.21.15.67:0/2439338494 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x563a711739f0 0x563a711468b0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.929+0000 7f5c27387740 1 --2- 172.21.15.67:0/2439338494 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563a7119c9d0 0x563a71146370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.929+0000 7f5c27387740 1 --2- 172.21.15.67:0/2439338494 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563a7116ed20 0x563a7116d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.929+0000 7f5c27387740 1 --2- 172.21.15.67:0/2439338494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a7116e940 0x563a7116d080 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.929+0000 7f5c27387740 1 -- 172.21.15.67:0/2439338494 >> 172.21.15.67:0/2439338494 conn(0x563a711420b0 msgr2=0x563a7113cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:49.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.929+0000 7f5bfd7fa640 0 client.6543 ms_handle_reset on v2:172.21.15.67:6834/451962260 2024-03-24T00:41:49.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.929+0000 7f5c27387740 1 -- 172.21.15.67:0/2439338494 shutdown_connections 2024-03-24T00:41:49.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.929+0000 7f5c27387740 1 -- 172.21.15.67:0/2439338494 wait complete. 2024-03-24T00:41:49.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.929+0000 7f5c27387740 20 client.6543 trim_cache size 0 max 16384 2024-03-24T00:41:49.936 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:41:49.936 INFO:teuthology.orchestra.run.smithi067.stdout: "return_code": 0, 2024-03-24T00:41:49.936 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_tag": "591fb9d4-a27b-49f1-a588-2d3188d744b2", 2024-03-24T00:41:49.936 INFO:teuthology.orchestra.run.smithi067.stdout: "mode": "asynchronous" 2024-03-24T00:41:49.936 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:41:49.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.930+0000 7f5c24797640 1 -- 172.21.15.67:0/357851681 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5c0006bc00 msgr2=0x7f5c0006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:49.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.930+0000 7f5c24797640 1 --2- 172.21.15.67:0/357851681 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5c0006bc00 0x7f5c0006e0c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c14bc00 tx=0x7f5c0c009190 comp rx=0 tx=0).stop 2024-03-24T00:41:49.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.930+0000 7f5c24797640 1 -- 172.21.15.67:0/357851681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c1c06ec40 msgr2=0x7f5c1c1f3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:49.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.930+0000 7f5c24797640 1 --2- 172.21.15.67:0/357851681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c1c06ec40 0x7f5c1c1f3de0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f5c1800c400 tx=0x7f5c1800c8c0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.930+0000 7f5c24797640 1 -- 172.21.15.67:0/357851681 shutdown_connections 2024-03-24T00:41:49.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.930+0000 7f5c24797640 1 --2- 172.21.15.67:0/357851681 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5c0006bc00 0x7f5c0006e0c0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.930+0000 7f5c24797640 1 --2- 172.21.15.67:0/357851681 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c1c15bed0 0x7f5c1c15c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.930+0000 7f5c24797640 1 --2- 172.21.15.67:0/357851681 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5c1c1589a0 0x7f5c1c154cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.930+0000 7f5c24797640 1 --2- 172.21.15.67:0/357851681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c1c06ec40 0x7f5c1c1f3de0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:49.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.930+0000 7f5c24797640 1 -- 172.21.15.67:0/357851681 >> 172.21.15.67:0/357851681 conn(0x7f5c1c0cf9e0 msgr2=0x7f5c1c15ee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:49.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.930+0000 7f5c24797640 1 -- 172.21.15.67:0/357851681 shutdown_connections 2024-03-24T00:41:49.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:49.930+0000 7f5c24797640 1 -- 172.21.15.67:0/357851681 wait complete. 2024-03-24T00:41:49.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:41:49.944 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.4:0 scrub status 2024-03-24T00:41:50.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.047+0000 7f05ddb87640 1 Processor -- start 2024-03-24T00:41:50.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.047+0000 7f05ddb87640 1 -- start start 2024-03-24T00:41:50.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.048+0000 7f05ddb87640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f05d815de90 0x7f05d815e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:50.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.048+0000 7f05ddb87640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f05d80af4e0 0x7f05d80cfd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:50.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.048+0000 7f05ddb87640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05d80d1630 con 0x7f05d80cd2e0 2024-03-24T00:41:50.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.048+0000 7f05ddb87640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05d80d1910 con 0x7f05d80af4e0 2024-03-24T00:41:50.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.048+0000 7f05ddb87640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05d80ad9f0 con 0x7f05d815de90 2024-03-24T00:41:50.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.048+0000 7f05d7fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f05d80af4e0 0x7f05d80cfd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:50.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.048+0000 7f05d77fe640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f05d815de90 0x7f05d815e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:50.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.048+0000 7f05d6ffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f05d80cd2e0 0x7f05d806b040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:52270/0 (socket says 172.21.15.67:52270) 2024-03-24T00:41:50.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.048+0000 7f05d6ffd640 1 -- 172.21.15.67:0/4061681032 learned_addr learned my addr 172.21.15.67:0/4061681032 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:41:50.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.049+0000 7f05d7fff640 1 -- 172.21.15.67:0/4061681032 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f05d815de90 msgr2=0x7f05d815e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:50.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.049+0000 7f05d7fff640 1 --2- 172.21.15.67:0/4061681032 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f05d815de90 0x7f05d815e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:50.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.049+0000 7f05d7fff640 1 -- 172.21.15.67:0/4061681032 >> v1:172.21.15.72:6789/0 conn(0x7f05d80cd2e0 legacy=0x7f05d806b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:50.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.049+0000 7f05d67fc640 1 -- 172.21.15.67:0/4061681032 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1330742226 0 0) 0x7f05d80d1630 con 0x7f05d80cd2e0 2024-03-24T00:41:50.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.049+0000 7f05d7fff640 1 -- 172.21.15.67:0/4061681032 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05d80adcd0 con 0x7f05d80af4e0 2024-03-24T00:41:50.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.049+0000 7f05d77fe640 1 --2- 172.21.15.67:0/4061681032 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f05d815de90 0x7f05d815e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:41:50.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.050+0000 7f05d7fff640 1 --2- 172.21.15.67:0/4061681032 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f05d80af4e0 0x7f05d80cfd30 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f05d00097a0 tx=0x7f05d002b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4056eaf8135e1c48 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:50.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.050+0000 7f05d67fc640 1 -- 172.21.15.67:0/4061681032 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d0031780 con 0x7f05d80af4e0 2024-03-24T00:41:50.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.050+0000 7f05d67fc640 1 -- 172.21.15.67:0/4061681032 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05d0037070 con 0x7f05d80af4e0 2024-03-24T00:41:50.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.050+0000 7f05d67fc640 1 -- 172.21.15.67:0/4061681032 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d0031b40 con 0x7f05d80af4e0 2024-03-24T00:41:50.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.050+0000 7f05ddb87640 1 -- 172.21.15.67:0/4061681032 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f05d80af4e0 msgr2=0x7f05d80cfd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:50.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.050+0000 7f05ddb87640 1 --2- 172.21.15.67:0/4061681032 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f05d80af4e0 0x7f05d80cfd30 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f05d00097a0 tx=0x7f05d002b820 comp rx=0 tx=0).stop 2024-03-24T00:41:50.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.050+0000 7f05ddb87640 1 -- 172.21.15.67:0/4061681032 shutdown_connections 2024-03-24T00:41:50.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.050+0000 7f05ddb87640 1 --2- 172.21.15.67:0/4061681032 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f05d80af4e0 0x7f05d80cfd30 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:50.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.050+0000 7f05ddb87640 1 --2- 172.21.15.67:0/4061681032 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f05d815de90 0x7f05d815e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:50.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.050+0000 7f05ddb87640 1 -- 172.21.15.67:0/4061681032 >> 172.21.15.67:0/4061681032 conn(0x7f05d8099a30 msgr2=0x7f05d8099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:50.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.050+0000 7f05ddb87640 1 -- 172.21.15.67:0/4061681032 shutdown_connections 2024-03-24T00:41:50.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.050+0000 7f05ddb87640 1 -- 172.21.15.67:0/4061681032 wait complete. 2024-03-24T00:41:50.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.051+0000 7f05ddb87640 1 Processor -- start 2024-03-24T00:41:50.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.052+0000 7f05ddb87640 1 -- start start 2024-03-24T00:41:50.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.052+0000 7f05ddb87640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f05d80af4e0 0x7f05d80700f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:50.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.052+0000 7f05ddb87640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f05d80cd2e0 0x7f05d8152230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:50.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.052+0000 7f05ddb87640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05d815de90 0x7f05d8152770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:50.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.052+0000 7f05ddb87640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d806b8d0 con 0x7f05d815de90 2024-03-24T00:41:50.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.052+0000 7f05ddb87640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d806b750 con 0x7f05d80af4e0 2024-03-24T00:41:50.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.052+0000 7f05ddb87640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05d80d0270 con 0x7f05d80cd2e0 2024-03-24T00:41:50.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.052+0000 7f05d7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05d815de90 0x7f05d8152770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:50.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.052+0000 7f05d7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05d815de90 0x7f05d8152770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54802/0 (socket says 172.21.15.67:54802) 2024-03-24T00:41:50.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.052+0000 7f05d7fff640 1 -- 172.21.15.67:0/3277396182 learned_addr learned my addr 172.21.15.67:0/3277396182 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:50.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.052+0000 7f05d6ffd640 1 --2- 172.21.15.67:0/3277396182 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f05d80cd2e0 0x7f05d8152230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:50.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.052+0000 7f05d77fe640 1 --2- 172.21.15.67:0/3277396182 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f05d80af4e0 0x7f05d80700f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:50.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.053+0000 7f05d7fff640 1 -- 172.21.15.67:0/3277396182 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f05d80cd2e0 msgr2=0x7f05d8152230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:50.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.053+0000 7f05d7fff640 1 --2- 172.21.15.67:0/3277396182 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f05d80cd2e0 0x7f05d8152230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:50.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.053+0000 7f05d7fff640 1 -- 172.21.15.67:0/3277396182 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f05d80af4e0 msgr2=0x7f05d80700f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:50.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.053+0000 7f05d7fff640 1 --2- 172.21.15.67:0/3277396182 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f05d80af4e0 0x7f05d80700f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:50.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.053+0000 7f05d7fff640 1 -- 172.21.15.67:0/3277396182 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05d80c7280 con 0x7f05d815de90 2024-03-24T00:41:50.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.053+0000 7f05d7fff640 1 --2- 172.21.15.67:0/3277396182 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05d815de90 0x7f05d8152770 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f05d0034040 tx=0x7f05d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:50.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.053+0000 7f05bffff640 1 -- 172.21.15.67:0/3277396182 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d0004230 con 0x7f05d815de90 2024-03-24T00:41:50.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.053+0000 7f05bffff640 1 -- 172.21.15.67:0/3277396182 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05d0031ec0 con 0x7f05d815de90 2024-03-24T00:41:50.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.053+0000 7f05ddb87640 1 -- 172.21.15.67:0/3277396182 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05d80c8d20 con 0x7f05d815de90 2024-03-24T00:41:50.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.053+0000 7f05bffff640 1 -- 172.21.15.67:0/3277396182 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d0003610 con 0x7f05d815de90 2024-03-24T00:41:50.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.053+0000 7f05ddb87640 1 -- 172.21.15.67:0/3277396182 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05d8070630 con 0x7f05d815de90 2024-03-24T00:41:50.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.055+0000 7f05bffff640 1 -- 172.21.15.67:0/3277396182 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f05d00037b0 con 0x7f05d815de90 2024-03-24T00:41:50.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.055+0000 7f05bffff640 1 --2- 172.21.15.67:0/3277396182 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f05c406bc00 0x7f05c406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:50.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.055+0000 7f05bffff640 1 -- 172.21.15.67:0/3277396182 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f05d00ac900 con 0x7f05d815de90 2024-03-24T00:41:50.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.055+0000 7f05d77fe640 1 --2- 172.21.15.67:0/3277396182 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f05c406bc00 0x7f05c406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:50.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.056+0000 7f05d77fe640 1 --2- 172.21.15.67:0/3277396182 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f05c406bc00 0x7f05c406e0c0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f05c8009bd0 tx=0x7f05c800b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:50.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.056+0000 7f05e0773740 1 Processor -- start 2024-03-24T00:41:50.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.056+0000 7f05e0773740 1 -- start start 2024-03-24T00:41:50.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.057+0000 7f05e0773740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x564e4aa03e50 0x564e4a9d6df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:50.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.057+0000 7f05e0773740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564e4aa04230 0x564e4aa06be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:50.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.057+0000 7f05e0773740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564e4a9d7330 0x564e4aa07120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:50.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.057+0000 7f05e0773740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e4a9413d0 con 0x564e4aa04230 2024-03-24T00:41:50.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.057+0000 7f05e0773740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e4a8e9ad0 con 0x564e4a9d7330 2024-03-24T00:41:50.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.057+0000 7f05e0773740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x564e4a93d470 con 0x564e4aa03e50 2024-03-24T00:41:50.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.057+0000 7f05d6ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564e4aa04230 0x564e4aa06be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:50.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.057+0000 7f05d6ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564e4aa04230 0x564e4aa06be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54810/0 (socket says 172.21.15.67:54810) 2024-03-24T00:41:50.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.057+0000 7f05d6ffd640 1 -- 172.21.15.67:0/934578270 learned_addr learned my addr 172.21.15.67:0/934578270 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:50.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.057+0000 7f05d77fe640 1 --2- 172.21.15.67:0/934578270 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x564e4aa03e50 0x564e4a9d6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:50.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.057+0000 7f05d7fff640 1 --2- 172.21.15.67:0/934578270 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564e4a9d7330 0x564e4aa07120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:50.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.057+0000 7f05d6ffd640 1 -- 172.21.15.67:0/934578270 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x564e4aa03e50 msgr2=0x564e4a9d6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:50.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.057+0000 7f05d6ffd640 1 --2- 172.21.15.67:0/934578270 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x564e4aa03e50 0x564e4a9d6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:50.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.057+0000 7f05d6ffd640 1 -- 172.21.15.67:0/934578270 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564e4a9d7330 msgr2=0x564e4aa07120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:50.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.057+0000 7f05d6ffd640 1 --2- 172.21.15.67:0/934578270 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564e4a9d7330 0x564e4aa07120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:50.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.057+0000 7f05d6ffd640 1 -- 172.21.15.67:0/934578270 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564e4a71f7d0 con 0x564e4aa04230 2024-03-24T00:41:50.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.058+0000 7f05d6ffd640 1 --2- 172.21.15.67:0/934578270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564e4aa04230 0x564e4aa06be0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f05cc007670 tx=0x7f05cc004bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75ba69636bd22244 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:50.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.058+0000 7f05bd7fa640 1 -- 172.21.15.67:0/934578270 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05cc01a070 con 0x564e4aa04230 2024-03-24T00:41:50.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.058+0000 7f05bd7fa640 1 -- 172.21.15.67:0/934578270 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05cc001850 con 0x564e4aa04230 2024-03-24T00:41:50.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.058+0000 7f05bd7fa640 1 -- 172.21.15.67:0/934578270 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05cc001c40 con 0x564e4aa04230 2024-03-24T00:41:50.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.059+0000 7f05e0773740 1 -- 172.21.15.67:0/934578270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564e4aa04230 msgr2=0x564e4aa06be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:50.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.059+0000 7f05e0773740 1 --2- 172.21.15.67:0/934578270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564e4aa04230 0x564e4aa06be0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f05cc007670 tx=0x7f05cc004bb0 comp rx=0 tx=0).stop 2024-03-24T00:41:50.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.059+0000 7f05e0773740 1 -- 172.21.15.67:0/934578270 shutdown_connections 2024-03-24T00:41:50.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.059+0000 7f05e0773740 1 --2- 172.21.15.67:0/934578270 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564e4a9d7330 0x564e4aa07120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:50.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.059+0000 7f05e0773740 1 --2- 172.21.15.67:0/934578270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564e4aa04230 0x564e4aa06be0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:50.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.059+0000 7f05e0773740 1 --2- 172.21.15.67:0/934578270 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x564e4aa03e50 0x564e4a9d6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:50.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.059+0000 7f05e0773740 1 -- 172.21.15.67:0/934578270 >> 172.21.15.67:0/934578270 conn(0x564e4a9d9060 msgr2=0x564e4aa21be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:50.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.059+0000 7f05e0773740 1 -- 172.21.15.67:0/934578270 shutdown_connections 2024-03-24T00:41:50.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.059+0000 7f05e0773740 1 -- 172.21.15.67:0/934578270 wait complete. 2024-03-24T00:41:50.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.060+0000 7f05e0773740 1 -- start start 2024-03-24T00:41:50.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.060+0000 7f05e0773740 1 Processor -- start 2024-03-24T00:41:50.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.061+0000 7f05e0773740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564e4aa03e50 0x564e4a9f6e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:50.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.061+0000 7f05e0773740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564e4aa04230 0x564e4aa08790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:50.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.061+0000 7f05e0773740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x564e4aa08cd0 0x564e4aa0d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:50.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.061+0000 7f05e0773740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e4a9db700 con 0x564e4aa03e50 2024-03-24T00:41:50.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.061+0000 7f05e0773740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e4a8fd060 con 0x564e4aa04230 2024-03-24T00:41:50.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.061+0000 7f05e0773740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x564e4a9ddf40 con 0x564e4aa08cd0 2024-03-24T00:41:50.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.061+0000 7f05e0773740 10 client.0 _ll_register_callbacks cb 0x564e4a8df6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:41:50.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.061+0000 7f05d77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564e4aa03e50 0x564e4a9f6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:50.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.061+0000 7f05d77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564e4aa03e50 0x564e4a9f6e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54822/0 (socket says 172.21.15.67:54822) 2024-03-24T00:41:50.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.061+0000 7f05d77fe640 1 -- 172.21.15.67:0/1187581641 learned_addr learned my addr 172.21.15.67:0/1187581641 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:50.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.061+0000 7f05d7fff640 1 --2- 172.21.15.67:0/1187581641 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x564e4aa08cd0 0x564e4aa0d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:50.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.061+0000 7f05d6ffd640 1 --2- 172.21.15.67:0/1187581641 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564e4aa04230 0x564e4aa08790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:50.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.061+0000 7f05d77fe640 1 -- 172.21.15.67:0/1187581641 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x564e4aa08cd0 msgr2=0x564e4aa0d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:50.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.061+0000 7f05d77fe640 1 --2- 172.21.15.67:0/1187581641 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x564e4aa08cd0 0x564e4aa0d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:50.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.061+0000 7f05d77fe640 1 -- 172.21.15.67:0/1187581641 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564e4aa04230 msgr2=0x564e4aa08790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:50.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.061+0000 7f05d77fe640 1 --2- 172.21.15.67:0/1187581641 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564e4aa04230 0x564e4aa08790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:50.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.061+0000 7f05d77fe640 1 -- 172.21.15.67:0/1187581641 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564e4a6553f0 con 0x564e4aa03e50 2024-03-24T00:41:50.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.061+0000 7f05d6ffd640 1 --2- 172.21.15.67:0/1187581641 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564e4aa04230 0x564e4aa08790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:41:50.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.062+0000 7f05d77fe640 1 --2- 172.21.15.67:0/1187581641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564e4aa03e50 0x564e4a9f6e30 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f05c80045f0 tx=0x7f05c8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a14e36a1a6a4d3a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:50.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.062+0000 7f05b67fc640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:41:50.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.062+0000 7f05b67fc640 1 -- 172.21.15.67:0/1187581641 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c8027070 con 0x564e4aa03e50 2024-03-24T00:41:50.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.062+0000 7f05b67fc640 1 -- 172.21.15.67:0/1187581641 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(62..62 src has 1..62) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f05c8023420 con 0x564e4aa03e50 2024-03-24T00:41:50.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.062+0000 7f05e0773740 1 -- 172.21.15.67:0/1187581641 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f05cc001850 con 0x564e4aa03e50 2024-03-24T00:41:50.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.062+0000 7f05b67fc640 1 -- 172.21.15.67:0/1187581641 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f05c8032050 con 0x564e4aa03e50 2024-03-24T00:41:50.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.062+0000 7f05e0773740 10 client.6552 fetch_fsmap learned FSMap version 32 2024-03-24T00:41:50.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.062+0000 7f05e0773740 1 -- 172.21.15.67:0/1187581641 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=2+}) v3 -- 0x564e4a6150f0 con 0x564e4aa03e50 2024-03-24T00:41:50.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.062+0000 7f05b67fc640 1 -- 172.21.15.67:0/1187581641 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05cc001850 con 0x564e4aa03e50 2024-03-24T00:41:50.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.062+0000 7f05b67fc640 1 -- 172.21.15.67:0/1187581641 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 32) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f05c8035020 con 0x564e4aa03e50 2024-03-24T00:41:50.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.062+0000 7f05e0773740 10 client.6552 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T00:41:50.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.062+0000 7f05e0773740 10 client.6552 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.a 2024-03-24T00:41:50.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.062+0000 7f05e0773740 20 client.6552 populate_metadata read hostname 'smithi067' 2024-03-24T00:41:50.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.062+0000 7f05e0773740 1 --2- 172.21.15.67:0/1187581641 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x564e4aa3c5b0 0x564e4aa3e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:50.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.062+0000 7f05e0773740 4 client.6552 mds_command: new command op to 6360 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:41:50.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.062+0000 7f05e0773740 1 -- 172.21.15.67:0/1187581641 --> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564e4a94e910 con 0x564e4aa3c5b0 2024-03-24T00:41:50.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.062+0000 7f05d6ffd640 1 --2- 172.21.15.67:0/1187581641 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x564e4aa3c5b0 0x564e4aa3e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:50.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.063+0000 7f05d6ffd640 1 --2- 172.21.15.67:0/1187581641 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x564e4aa3c5b0 0x564e4aa3e9c0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bab29dfa5e7ba69b server_cookie=28014f603bffbe15 in_seq=0 out_seq=0 2024-03-24T00:41:50.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.063+0000 7f05b67fc640 10 client.6552 ms_handle_connect on v2:172.21.15.67:6834/451962260 2024-03-24T00:41:50.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.063+0000 7f05b67fc640 1 -- 172.21.15.67:0/1187581641 <== mds.0 v2:172.21.15.67:6834/451962260 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564e4a94e910 con 0x564e4aa3c5b0 2024-03-24T00:41:50.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.063+0000 7f05b67fc640 10 client.6552 handle_command_reply: tid=0 2024-03-24T00:41:50.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.063+0000 7f05e0773740 1 client.6552 shutdown 2024-03-24T00:41:50.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.063+0000 7f05e0773740 1 -- 172.21.15.67:0/1187581641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564e4aa03e50 msgr2=0x564e4a9f6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:50.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.063+0000 7f05e0773740 1 --2- 172.21.15.67:0/1187581641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564e4aa03e50 0x564e4a9f6e30 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f05c80045f0 tx=0x7f05c8004a40 comp rx=0 tx=0).stop 2024-03-24T00:41:50.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.063+0000 7f05e0773740 1 -- 172.21.15.67:0/1187581641 shutdown_connections 2024-03-24T00:41:50.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.063+0000 7f05e0773740 1 --2- 172.21.15.67:0/1187581641 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x564e4aa3c5b0 0x564e4aa3e9c0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:50.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.063+0000 7f05e0773740 1 --2- 172.21.15.67:0/1187581641 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x564e4aa08cd0 0x564e4aa0d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:50.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.063+0000 7f05e0773740 1 --2- 172.21.15.67:0/1187581641 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564e4aa04230 0x564e4aa08790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:50.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.063+0000 7f05b67fc640 0 client.6552 ms_handle_reset on v2:172.21.15.67:6834/451962260 2024-03-24T00:41:50.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.063+0000 7f05e0773740 1 --2- 172.21.15.67:0/1187581641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564e4aa03e50 0x564e4a9f6e30 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:50.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.063+0000 7f05e0773740 1 -- 172.21.15.67:0/1187581641 >> 172.21.15.67:0/1187581641 conn(0x564e4a9d7330 msgr2=0x564e4a9dadd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:50.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.063+0000 7f05e0773740 1 -- 172.21.15.67:0/1187581641 shutdown_connections 2024-03-24T00:41:50.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.064+0000 7f05e0773740 1 -- 172.21.15.67:0/1187581641 wait complete. 2024-03-24T00:41:50.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.064+0000 7f05e0773740 20 client.6552 trim_cache size 0 max 16384 2024-03-24T00:41:50.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.080+0000 7f05e0773740 1 Processor -- start 2024-03-24T00:41:50.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.080+0000 7f05e0773740 1 -- start start 2024-03-24T00:41:50.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.080+0000 7f05e0773740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x564e4aa03e50 0x564e4aa0d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:50.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.080+0000 7f05e0773740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564e4aa04230 0x564e4aa0d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:50.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.080+0000 7f05e0773740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564e4aa08cd0 0x564e4aa3a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:50.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.080+0000 7f05e0773740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d82007b0 con 0x564e4aa04230 2024-03-24T00:41:50.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.080+0000 7f05e0773740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d81ff8d0 con 0x564e4aa08cd0 2024-03-24T00:41:50.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.080+0000 7f05e0773740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05d80ca4b0 con 0x564e4aa03e50 2024-03-24T00:41:50.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.081+0000 7f05d6ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564e4aa04230 0x564e4aa0d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:50.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.081+0000 7f05d6ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564e4aa04230 0x564e4aa0d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54824/0 (socket says 172.21.15.67:54824) 2024-03-24T00:41:50.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.081+0000 7f05d6ffd640 1 -- 172.21.15.67:0/1937538715 learned_addr learned my addr 172.21.15.67:0/1937538715 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:50.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.081+0000 7f05d6ffd640 1 -- 172.21.15.67:0/1937538715 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x564e4aa03e50 msgr2=0x564e4aa0d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:50.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.081+0000 7f05d6ffd640 1 --2- 172.21.15.67:0/1937538715 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x564e4aa03e50 0x564e4aa0d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:50.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.081+0000 7f05d6ffd640 1 -- 172.21.15.67:0/1937538715 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564e4aa08cd0 msgr2=0x564e4aa3a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:50.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.081+0000 7f05d6ffd640 1 --2- 172.21.15.67:0/1937538715 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564e4aa08cd0 0x564e4aa3a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:50.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.081+0000 7f05d6ffd640 1 -- 172.21.15.67:0/1937538715 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564e4aa0a210 con 0x564e4aa04230 2024-03-24T00:41:50.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.081+0000 7f05d6ffd640 1 --2- 172.21.15.67:0/1937538715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564e4aa04230 0x564e4aa0d590 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f05cc007430 tx=0x7f05cc00c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d72d4945df996d71 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:50.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.081+0000 7f05bcff9640 1 -- 172.21.15.67:0/1937538715 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05cc01a040 con 0x564e4aa04230 2024-03-24T00:41:50.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.081+0000 7f05bcff9640 1 -- 172.21.15.67:0/1937538715 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05cc01b040 con 0x564e4aa04230 2024-03-24T00:41:50.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.081+0000 7f05bcff9640 1 -- 172.21.15.67:0/1937538715 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05cc026840 con 0x564e4aa04230 2024-03-24T00:41:50.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.082+0000 7f05e0773740 1 -- 172.21.15.67:0/1937538715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564e4aa04230 msgr2=0x564e4aa0d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:50.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.082+0000 7f05e0773740 1 --2- 172.21.15.67:0/1937538715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564e4aa04230 0x564e4aa0d590 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f05cc007430 tx=0x7f05cc00c000 comp rx=0 tx=0).stop 2024-03-24T00:41:50.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.082+0000 7f05e0773740 1 -- 172.21.15.67:0/1937538715 shutdown_connections 2024-03-24T00:41:50.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.082+0000 7f05e0773740 1 --2- 172.21.15.67:0/1937538715 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564e4aa08cd0 0x564e4aa3a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:50.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.082+0000 7f05e0773740 1 --2- 172.21.15.67:0/1937538715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564e4aa04230 0x564e4aa0d590 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:50.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.082+0000 7f05e0773740 1 --2- 172.21.15.67:0/1937538715 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x564e4aa03e50 0x564e4aa0d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:50.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.082+0000 7f05e0773740 1 -- 172.21.15.67:0/1937538715 >> 172.21.15.67:0/1937538715 conn(0x564e4a9d7330 msgr2=0x564e4aa0e3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:50.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.082+0000 7f05e0773740 1 -- 172.21.15.67:0/1937538715 shutdown_connections 2024-03-24T00:41:50.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.082+0000 7f05e0773740 1 -- 172.21.15.67:0/1937538715 wait complete. 2024-03-24T00:41:50.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.082+0000 7f05e0773740 1 -- start start 2024-03-24T00:41:50.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.082+0000 7f05e0773740 1 Processor -- start 2024-03-24T00:41:50.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.083+0000 7f05e0773740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564e4aa03e50 0x564e4a9da3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:50.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.083+0000 7f05e0773740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x564e4aa04230 0x564e4aa02da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:50.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.083+0000 7f05e0773740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564e4aa08cd0 0x564e4aa032e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:50.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.083+0000 7f05e0773740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d81ff3f0 con 0x564e4aa08cd0 2024-03-24T00:41:50.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.083+0000 7f05e0773740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e4aa09fd0 con 0x564e4aa03e50 2024-03-24T00:41:50.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.083+0000 7f05e0773740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x564e4aa39860 con 0x564e4aa04230 2024-03-24T00:41:50.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.083+0000 7f05e0773740 10 client.0 _ll_register_callbacks cb 0x7f05c8003990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:41:50.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.083+0000 7f05d7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564e4aa08cd0 0x564e4aa032e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:50.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.083+0000 7f05d7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564e4aa08cd0 0x564e4aa032e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54826/0 (socket says 172.21.15.67:54826) 2024-03-24T00:41:50.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.083+0000 7f05d7fff640 1 -- 172.21.15.67:0/1602258377 learned_addr learned my addr 172.21.15.67:0/1602258377 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:41:50.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.083+0000 7f05d6ffd640 1 --2- 172.21.15.67:0/1602258377 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x564e4aa04230 0x564e4aa02da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:50.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.083+0000 7f05d77fe640 1 --2- 172.21.15.67:0/1602258377 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564e4aa03e50 0x564e4a9da3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:50.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.084+0000 7f05d7fff640 1 -- 172.21.15.67:0/1602258377 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x564e4aa04230 msgr2=0x564e4aa02da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:50.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.084+0000 7f05d7fff640 1 --2- 172.21.15.67:0/1602258377 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x564e4aa04230 0x564e4aa02da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:50.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.084+0000 7f05d7fff640 1 -- 172.21.15.67:0/1602258377 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564e4aa03e50 msgr2=0x564e4a9da3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:41:50.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.084+0000 7f05d7fff640 1 --2- 172.21.15.67:0/1602258377 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564e4aa03e50 0x564e4a9da3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:50.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.084+0000 7f05d7fff640 1 -- 172.21.15.67:0/1602258377 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564e4aa0d9b0 con 0x564e4aa08cd0 2024-03-24T00:41:50.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.084+0000 7f05d7fff640 1 --2- 172.21.15.67:0/1602258377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564e4aa08cd0 0x564e4aa032e0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f05d002be30 tx=0x7f05d007a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b370bde30fc4d1de server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:41:50.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.084+0000 7f05b67fc640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:41:50.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.084+0000 7f05b67fc640 1 -- 172.21.15.67:0/1602258377 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d0083100 con 0x564e4aa08cd0 2024-03-24T00:41:50.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.084+0000 7f05b67fc640 1 -- 172.21.15.67:0/1602258377 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(62..62 src has 1..62) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f05d007a0f0 con 0x564e4aa08cd0 2024-03-24T00:41:50.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.084+0000 7f05e0773740 1 -- 172.21.15.67:0/1602258377 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f05cc01b040 con 0x564e4aa08cd0 2024-03-24T00:41:50.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.084+0000 7f05b67fc640 1 -- 172.21.15.67:0/1602258377 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f05d0083390 con 0x564e4aa08cd0 2024-03-24T00:41:50.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.084+0000 7f05e0773740 10 client.6558 fetch_fsmap learned FSMap version 32 2024-03-24T00:41:50.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.084+0000 7f05e0773740 1 -- 172.21.15.67:0/1602258377 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=2+}) v3 -- 0x564e4aa440b0 con 0x564e4aa08cd0 2024-03-24T00:41:50.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.084+0000 7f05b67fc640 1 -- 172.21.15.67:0/1602258377 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05cc01b040 con 0x564e4aa08cd0 2024-03-24T00:41:50.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.084+0000 7f05b67fc640 1 -- 172.21.15.67:0/1602258377 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 32) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f05d00769f0 con 0x564e4aa08cd0 2024-03-24T00:41:50.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.084+0000 7f05e0773740 10 client.6558 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T00:41:50.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.084+0000 7f05e0773740 10 client.6558 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.a 2024-03-24T00:41:50.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.084+0000 7f05e0773740 20 client.6558 populate_metadata read hostname 'smithi067' 2024-03-24T00:41:50.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.084+0000 7f05e0773740 1 --2- 172.21.15.67:0/1602258377 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x564e4aa799b0 0x564e4a9dbf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:41:50.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.084+0000 7f05e0773740 4 client.6558 mds_command: new command op to 6360 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:41:50.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.084+0000 7f05e0773740 1 -- 172.21.15.67:0/1602258377 --> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x564e4a948c60 con 0x564e4aa799b0 2024-03-24T00:41:50.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.085+0000 7f05d77fe640 1 --2- 172.21.15.67:0/1602258377 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x564e4aa799b0 0x564e4a9dbf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:41:50.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.085+0000 7f05b67fc640 10 client.6558 ms_handle_connect on v2:172.21.15.67:6834/451962260 2024-03-24T00:41:50.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.085+0000 7f05d77fe640 1 --2- 172.21.15.67:0/1602258377 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x564e4aa799b0 0x564e4a9dbf30 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=debc2db2adcb43b4 server_cookie=8702d8a27bb01c81 in_seq=0 out_seq=0 2024-03-24T00:41:50.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.085+0000 7f05b67fc640 1 -- 172.21.15.67:0/1602258377 <== mds.0 v2:172.21.15.67:6834/451962260 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+253 (crc 0 0 0) 0x564e4a948c60 con 0x564e4aa799b0 2024-03-24T00:41:50.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.085+0000 7f05b67fc640 10 client.6558 handle_command_reply: tid=0 2024-03-24T00:41:50.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.085+0000 7f05e0773740 1 client.6558 shutdown 2024-03-24T00:41:50.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.085+0000 7f05e0773740 1 -- 172.21.15.67:0/1602258377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564e4aa08cd0 msgr2=0x564e4aa032e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:50.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.085+0000 7f05e0773740 1 --2- 172.21.15.67:0/1602258377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564e4aa08cd0 0x564e4aa032e0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f05d002be30 tx=0x7f05d007a6c0 comp rx=0 tx=0).stop 2024-03-24T00:41:50.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.085+0000 7f05e0773740 1 -- 172.21.15.67:0/1602258377 shutdown_connections 2024-03-24T00:41:50.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.085+0000 7f05e0773740 1 --2- 172.21.15.67:0/1602258377 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x564e4aa799b0 0x564e4a9dbf30 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:50.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.085+0000 7f05e0773740 1 --2- 172.21.15.67:0/1602258377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564e4aa08cd0 0x564e4aa032e0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:50.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.085+0000 7f05e0773740 1 --2- 172.21.15.67:0/1602258377 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x564e4aa04230 0x564e4aa02da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:50.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.085+0000 7f05b67fc640 0 client.6558 ms_handle_reset on v2:172.21.15.67:6834/451962260 2024-03-24T00:41:50.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.085+0000 7f05e0773740 1 --2- 172.21.15.67:0/1602258377 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564e4aa03e50 0x564e4a9da3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:50.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.085+0000 7f05e0773740 1 -- 172.21.15.67:0/1602258377 >> 172.21.15.67:0/1602258377 conn(0x564e4a9d7330 msgr2=0x564e4aa0b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:50.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.085+0000 7f05e0773740 1 -- 172.21.15.67:0/1602258377 shutdown_connections 2024-03-24T00:41:50.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.085+0000 7f05e0773740 1 -- 172.21.15.67:0/1602258377 wait complete. 2024-03-24T00:41:50.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.085+0000 7f05e0773740 20 client.6558 trim_cache size 0 max 16384 2024-03-24T00:41:50.091 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:41:50.092 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T00:41:50.092 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": { 2024-03-24T00:41:50.092 INFO:teuthology.orchestra.run.smithi067.stdout: "591fb9d4-a27b-49f1-a588-2d3188d744b2": { 2024-03-24T00:41:50.092 INFO:teuthology.orchestra.run.smithi067.stdout: "path": "/", 2024-03-24T00:41:50.092 INFO:teuthology.orchestra.run.smithi067.stdout: "tag": "591fb9d4-a27b-49f1-a588-2d3188d744b2", 2024-03-24T00:41:50.092 INFO:teuthology.orchestra.run.smithi067.stdout: "options": "recursive" 2024-03-24T00:41:50.092 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:41:50.092 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:41:50.092 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:41:50.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.086+0000 7f05ddb87640 1 -- 172.21.15.67:0/3277396182 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f05c406bc00 msgr2=0x7f05c406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:50.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.086+0000 7f05ddb87640 1 --2- 172.21.15.67:0/3277396182 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f05c406bc00 0x7f05c406e0c0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f05c8009bd0 tx=0x7f05c800b040 comp rx=0 tx=0).stop 2024-03-24T00:41:50.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.086+0000 7f05ddb87640 1 -- 172.21.15.67:0/3277396182 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05d815de90 msgr2=0x7f05d8152770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:41:50.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.086+0000 7f05ddb87640 1 --2- 172.21.15.67:0/3277396182 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05d815de90 0x7f05d8152770 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f05d0034040 tx=0x7f05d00046c0 comp rx=0 tx=0).stop 2024-03-24T00:41:50.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.086+0000 7f05ddb87640 1 -- 172.21.15.67:0/3277396182 shutdown_connections 2024-03-24T00:41:50.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.086+0000 7f05ddb87640 1 --2- 172.21.15.67:0/3277396182 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f05c406bc00 0x7f05c406e0c0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:50.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.086+0000 7f05ddb87640 1 --2- 172.21.15.67:0/3277396182 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05d815de90 0x7f05d8152770 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:50.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.086+0000 7f05ddb87640 1 --2- 172.21.15.67:0/3277396182 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f05d80cd2e0 0x7f05d8152230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:50.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.086+0000 7f05ddb87640 1 --2- 172.21.15.67:0/3277396182 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f05d80af4e0 0x7f05d80700f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:41:50.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.086+0000 7f05ddb87640 1 -- 172.21.15.67:0/3277396182 >> 172.21.15.67:0/3277396182 conn(0x7f05d8099a30 msgr2=0x7f05d815a250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:41:50.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.086+0000 7f05ddb87640 1 -- 172.21.15.67:0/3277396182 shutdown_connections 2024-03-24T00:41:50.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:41:50.086+0000 7f05ddb87640 1 -- 172.21.15.67:0/3277396182 wait complete. 2024-03-24T00:41:50.097 INFO:tasks.cephfs.filesystem:scrub status for tag:591fb9d4-a27b-49f1-a588-2d3188d744b2 - {'path': '/', 'tag': '591fb9d4-a27b-49f1-a588-2d3188d744b2', 'options': 'recursive'} 2024-03-24T00:41:54.915 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-24T00:41:54.915+0000 7fbfd2216640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-03-24T00:42:16.256 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:42:16.258 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:42:20.097 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:42:20.098 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.4:0 scrub status 2024-03-24T00:42:20.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.200+0000 7f533d562640 1 Processor -- start 2024-03-24T00:42:20.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.200+0000 7f533d562640 1 -- start start 2024-03-24T00:42:20.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.201+0000 7f533d562640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f53380b4a00 0x7f53380b3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.201+0000 7f533d562640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f533806ec60 0x7f533806f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.201+0000 7f533d562640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53380b2ab0 con 0x7f5338144670 2024-03-24T00:42:20.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.201+0000 7f533d562640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53380b2d90 con 0x7f533806ec60 2024-03-24T00:42:20.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.201+0000 7f533d562640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53380b3070 con 0x7f53380b4a00 2024-03-24T00:42:20.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.201+0000 7f53377fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f533806ec60 0x7f533806f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.201+0000 7f5336ffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5338144670 0x7f5338072500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33836/0 (socket says 172.21.15.67:33836) 2024-03-24T00:42:20.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.201+0000 7f5336ffd640 1 -- 172.21.15.67:0/899407733 learned_addr learned my addr 172.21.15.67:0/899407733 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:42:20.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.201+0000 7f53367fc640 1 --2- 172.21.15.67:0/899407733 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f53380b4a00 0x7f53380b3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.201+0000 7f53377fe640 1 -- 172.21.15.67:0/899407733 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f53380b4a00 msgr2=0x7f53380b3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:20.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.201+0000 7f53377fe640 1 --2- 172.21.15.67:0/899407733 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f53380b4a00 0x7f53380b3870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.201+0000 7f53377fe640 1 -- 172.21.15.67:0/899407733 >> v1:172.21.15.67:6789/0 conn(0x7f5338144670 legacy=0x7f5338072500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:20.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.201+0000 7f53377fe640 1 -- 172.21.15.67:0/899407733 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53380b3350 con 0x7f533806ec60 2024-03-24T00:42:20.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.201+0000 7f5335ffb640 1 -- 172.21.15.67:0/899407733 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2778480434 0 0) 0x7f53380b2ab0 con 0x7f5338144670 2024-03-24T00:42:20.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.202+0000 7f53377fe640 1 --2- 172.21.15.67:0/899407733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f533806ec60 0x7f533806f0f0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f5330009d90 tx=0x7f533002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd13f17b7c59630c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:20.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.202+0000 7f5335ffb640 1 -- 172.21.15.67:0/899407733 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5330039070 con 0x7f533806ec60 2024-03-24T00:42:20.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.202+0000 7f5335ffb640 1 -- 172.21.15.67:0/899407733 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f533002cc50 con 0x7f533806ec60 2024-03-24T00:42:20.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.202+0000 7f5335ffb640 1 -- 172.21.15.67:0/899407733 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5330034440 con 0x7f533806ec60 2024-03-24T00:42:20.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.202+0000 7f533d562640 1 -- 172.21.15.67:0/899407733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f533806ec60 msgr2=0x7f533806f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:20.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.202+0000 7f533d562640 1 --2- 172.21.15.67:0/899407733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f533806ec60 0x7f533806f0f0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f5330009d90 tx=0x7f533002bea0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.203+0000 7f533d562640 1 -- 172.21.15.67:0/899407733 shutdown_connections 2024-03-24T00:42:20.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.203+0000 7f533d562640 1 --2- 172.21.15.67:0/899407733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f533806ec60 0x7f533806f0f0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.203+0000 7f533d562640 1 --2- 172.21.15.67:0/899407733 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f53380b4a00 0x7f53380b3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.203+0000 7f533d562640 1 -- 172.21.15.67:0/899407733 >> 172.21.15.67:0/899407733 conn(0x7f53380cf9e0 msgr2=0x7f53380cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:20.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.203+0000 7f533d562640 1 -- 172.21.15.67:0/899407733 shutdown_connections 2024-03-24T00:42:20.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.203+0000 7f533d562640 1 -- 172.21.15.67:0/899407733 wait complete. 2024-03-24T00:42:20.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.204+0000 7f533d562640 1 Processor -- start 2024-03-24T00:42:20.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.204+0000 7f533d562640 1 -- start start 2024-03-24T00:42:20.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.205+0000 7f533d562640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f533806ec60 0x7f53381f78b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.205+0000 7f533d562640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f53380b4a00 0x7f533815b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.205+0000 7f533d562640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5338144670 0x7f53380b0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.205+0000 7f533d562640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53380cdfb0 con 0x7f533806ec60 2024-03-24T00:42:20.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.205+0000 7f533d562640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53380cde30 con 0x7f5338144670 2024-03-24T00:42:20.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.205+0000 7f533d562640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53380ce130 con 0x7f53380b4a00 2024-03-24T00:42:20.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.205+0000 7f5336ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f533806ec60 0x7f53381f78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.205+0000 7f5336ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f533806ec60 0x7f53381f78b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42660/0 (socket says 172.21.15.67:42660) 2024-03-24T00:42:20.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.205+0000 7f5336ffd640 1 -- 172.21.15.67:0/118756353 learned_addr learned my addr 172.21.15.67:0/118756353 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:20.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.205+0000 7f53367fc640 1 --2- 172.21.15.67:0/118756353 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f53380b4a00 0x7f533815b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.205+0000 7f53377fe640 1 --2- 172.21.15.67:0/118756353 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5338144670 0x7f53380b0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.205+0000 7f5336ffd640 1 -- 172.21.15.67:0/118756353 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f53380b4a00 msgr2=0x7f533815b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:20.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.205+0000 7f5336ffd640 1 --2- 172.21.15.67:0/118756353 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f53380b4a00 0x7f533815b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.205+0000 7f5336ffd640 1 -- 172.21.15.67:0/118756353 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5338144670 msgr2=0x7f53380b0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:20.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.205+0000 7f5336ffd640 1 --2- 172.21.15.67:0/118756353 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5338144670 0x7f53380b0850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.205+0000 7f5336ffd640 1 -- 172.21.15.67:0/118756353 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5338141780 con 0x7f533806ec60 2024-03-24T00:42:20.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.205+0000 7f53367fc640 1 --2- 172.21.15.67:0/118756353 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f53380b4a00 0x7f533815b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:42:20.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.205+0000 7f53377fe640 1 --2- 172.21.15.67:0/118756353 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5338144670 0x7f53380b0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:42:20.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.205+0000 7f5336ffd640 1 --2- 172.21.15.67:0/118756353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f533806ec60 0x7f53381f78b0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f532800c430 tx=0x7f532800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:20.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.205+0000 7f531f7fe640 1 -- 172.21.15.67:0/118756353 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53280074e0 con 0x7f533806ec60 2024-03-24T00:42:20.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.205+0000 7f531f7fe640 1 -- 172.21.15.67:0/118756353 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53280047e0 con 0x7f533806ec60 2024-03-24T00:42:20.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.205+0000 7f531f7fe640 1 -- 172.21.15.67:0/118756353 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5328004bd0 con 0x7f533806ec60 2024-03-24T00:42:20.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.205+0000 7f533d562640 1 -- 172.21.15.67:0/118756353 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53380ac3c0 con 0x7f533806ec60 2024-03-24T00:42:20.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.205+0000 7f533d562640 1 -- 172.21.15.67:0/118756353 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53380ac980 con 0x7f533806ec60 2024-03-24T00:42:20.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.207+0000 7f531f7fe640 1 -- 172.21.15.67:0/118756353 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5328011070 con 0x7f533806ec60 2024-03-24T00:42:20.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.207+0000 7f531f7fe640 1 --2- 172.21.15.67:0/118756353 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f532406bbd0 0x7f532406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.207+0000 7f531f7fe640 1 -- 172.21.15.67:0/118756353 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f532808a940 con 0x7f533806ec60 2024-03-24T00:42:20.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.208+0000 7f53367fc640 1 --2- 172.21.15.67:0/118756353 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f532406bbd0 0x7f532406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.208+0000 7f53367fc640 1 --2- 172.21.15.67:0/118756353 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f532406bbd0 0x7f532406e090 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f53380b1530 tx=0x7f532c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:20.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.209+0000 7f534022f740 1 Processor -- start 2024-03-24T00:42:20.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.209+0000 7f534022f740 1 -- start start 2024-03-24T00:42:20.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.210+0000 7f534022f740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563974860e50 0x563974833df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.210+0000 7f534022f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563974861230 0x563974863be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.210+0000 7f534022f740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563974834330 0x563974864120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.210+0000 7f534022f740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x56397479e3d0 con 0x563974861230 2024-03-24T00:42:20.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.210+0000 7f534022f740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x563974746ad0 con 0x563974834330 2024-03-24T00:42:20.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.210+0000 7f534022f740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x56397479a470 con 0x563974860e50 2024-03-24T00:42:20.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.211+0000 7f53367fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563974861230 0x563974863be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.211+0000 7f53367fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563974861230 0x563974863be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42668/0 (socket says 172.21.15.67:42668) 2024-03-24T00:42:20.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.211+0000 7f53367fc640 1 -- 172.21.15.67:0/2813898423 learned_addr learned my addr 172.21.15.67:0/2813898423 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:20.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.211+0000 7f5336ffd640 1 --2- 172.21.15.67:0/2813898423 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563974860e50 0x563974833df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.211+0000 7f53377fe640 1 --2- 172.21.15.67:0/2813898423 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563974834330 0x563974864120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.211+0000 7f53367fc640 1 -- 172.21.15.67:0/2813898423 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563974860e50 msgr2=0x563974833df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:20.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.211+0000 7f53367fc640 1 --2- 172.21.15.67:0/2813898423 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563974860e50 0x563974833df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.211+0000 7f53367fc640 1 -- 172.21.15.67:0/2813898423 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563974834330 msgr2=0x563974864120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:20.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.211+0000 7f53367fc640 1 --2- 172.21.15.67:0/2813898423 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563974834330 0x563974864120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.211+0000 7f53367fc640 1 -- 172.21.15.67:0/2813898423 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56397457c7d0 con 0x563974861230 2024-03-24T00:42:20.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.211+0000 7f53367fc640 1 --2- 172.21.15.67:0/2813898423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563974861230 0x563974863be0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f532c00db60 tx=0x7f532c010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=424cc6a773da2be0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:20.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.211+0000 7f531cff9640 1 -- 172.21.15.67:0/2813898423 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f532c019050 con 0x563974861230 2024-03-24T00:42:20.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.211+0000 7f531cff9640 1 -- 172.21.15.67:0/2813898423 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f532c023070 con 0x563974861230 2024-03-24T00:42:20.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.211+0000 7f531cff9640 1 -- 172.21.15.67:0/2813898423 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f532c012c90 con 0x563974861230 2024-03-24T00:42:20.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.212+0000 7f534022f740 1 -- 172.21.15.67:0/2813898423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563974861230 msgr2=0x563974863be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:20.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.212+0000 7f534022f740 1 --2- 172.21.15.67:0/2813898423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563974861230 0x563974863be0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f532c00db60 tx=0x7f532c010000 comp rx=0 tx=0).stop 2024-03-24T00:42:20.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.212+0000 7f534022f740 1 -- 172.21.15.67:0/2813898423 shutdown_connections 2024-03-24T00:42:20.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.212+0000 7f534022f740 1 --2- 172.21.15.67:0/2813898423 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563974834330 0x563974864120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.212+0000 7f534022f740 1 --2- 172.21.15.67:0/2813898423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563974861230 0x563974863be0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.212+0000 7f534022f740 1 --2- 172.21.15.67:0/2813898423 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563974860e50 0x563974833df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.212+0000 7f534022f740 1 -- 172.21.15.67:0/2813898423 >> 172.21.15.67:0/2813898423 conn(0x563974836060 msgr2=0x56397487ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:20.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.212+0000 7f534022f740 1 -- 172.21.15.67:0/2813898423 shutdown_connections 2024-03-24T00:42:20.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.212+0000 7f534022f740 1 -- 172.21.15.67:0/2813898423 wait complete. 2024-03-24T00:42:20.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.213+0000 7f534022f740 1 -- start start 2024-03-24T00:42:20.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.213+0000 7f534022f740 1 Processor -- start 2024-03-24T00:42:20.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.213+0000 7f534022f740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563974860e50 0x563974862fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.213+0000 7f534022f740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563974861230 0x563974865200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.213+0000 7f534022f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563974865740 0x563974865ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.213+0000 7f534022f740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x563974838700 con 0x563974865740 2024-03-24T00:42:20.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.213+0000 7f534022f740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x56397475a060 con 0x563974861230 2024-03-24T00:42:20.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.213+0000 7f534022f740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x56397483af40 con 0x563974860e50 2024-03-24T00:42:20.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.213+0000 7f534022f740 10 client.0 _ll_register_callbacks cb 0x56397473c6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:42:20.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.214+0000 7f53377fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563974865740 0x563974865ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.214+0000 7f53377fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563974865740 0x563974865ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42672/0 (socket says 172.21.15.67:42672) 2024-03-24T00:42:20.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.214+0000 7f53377fe640 1 -- 172.21.15.67:0/968083295 learned_addr learned my addr 172.21.15.67:0/968083295 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:20.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.214+0000 7f53367fc640 1 --2- 172.21.15.67:0/968083295 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563974861230 0x563974865200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.214+0000 7f5336ffd640 1 --2- 172.21.15.67:0/968083295 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563974860e50 0x563974862fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.214+0000 7f53377fe640 1 -- 172.21.15.67:0/968083295 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563974860e50 msgr2=0x563974862fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:20.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.214+0000 7f53377fe640 1 --2- 172.21.15.67:0/968083295 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563974860e50 0x563974862fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.214+0000 7f53377fe640 1 -- 172.21.15.67:0/968083295 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563974861230 msgr2=0x563974865200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:20.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.214+0000 7f53377fe640 1 --2- 172.21.15.67:0/968083295 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563974861230 0x563974865200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.214+0000 7f53377fe640 1 -- 172.21.15.67:0/968083295 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5639744b23f0 con 0x563974865740 2024-03-24T00:42:20.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.214+0000 7f53377fe640 1 --2- 172.21.15.67:0/968083295 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563974865740 0x563974865ba0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f5330037040 tx=0x7f5330004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8d751f3bd30a963 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:20.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.214+0000 7f5315ffb640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:42:20.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.214+0000 7f5315ffb640 1 -- 172.21.15.67:0/968083295 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5330046070 con 0x563974865740 2024-03-24T00:42:20.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.214+0000 7f5315ffb640 1 -- 172.21.15.67:0/968083295 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(62..62 src has 1..62) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5330040670 con 0x563974865740 2024-03-24T00:42:20.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.214+0000 7f534022f740 1 -- 172.21.15.67:0/968083295 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f532c023070 con 0x563974865740 2024-03-24T00:42:20.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.214+0000 7f5315ffb640 1 -- 172.21.15.67:0/968083295 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5330050050 con 0x563974865740 2024-03-24T00:42:20.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.214+0000 7f534022f740 10 client.6573 fetch_fsmap learned FSMap version 33 2024-03-24T00:42:20.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.214+0000 7f534022f740 1 -- 172.21.15.67:0/968083295 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x5639744720f0 con 0x563974865740 2024-03-24T00:42:20.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.215+0000 7f5315ffb640 1 -- 172.21.15.67:0/968083295 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f532c023070 con 0x563974865740 2024-03-24T00:42:20.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.215+0000 7f5315ffb640 1 -- 172.21.15.67:0/968083295 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 33) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f5330053020 con 0x563974865740 2024-03-24T00:42:20.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.215+0000 7f534022f740 10 client.6573 fetch_fsmap finished waiting for FSMap version 33 2024-03-24T00:42:20.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.215+0000 7f534022f740 10 client.6573 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.a 2024-03-24T00:42:20.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.215+0000 7f534022f740 20 client.6573 populate_metadata read hostname 'smithi067' 2024-03-24T00:42:20.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.215+0000 7f534022f740 1 --2- 172.21.15.67:0/968083295 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x5639748994a0 0x56397489b890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.215+0000 7f534022f740 4 client.6573 mds_command: new command op to 6360 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:42:20.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.215+0000 7f534022f740 1 -- 172.21.15.67:0/968083295 --> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5639747ab910 con 0x5639748994a0 2024-03-24T00:42:20.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.215+0000 7f5336ffd640 1 --2- 172.21.15.67:0/968083295 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x5639748994a0 0x56397489b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.215+0000 7f5336ffd640 1 --2- 172.21.15.67:0/968083295 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x5639748994a0 0x56397489b890 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2bd066f84ef2f4 server_cookie=1fa70b94a136fa31 in_seq=0 out_seq=0 2024-03-24T00:42:20.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.215+0000 7f5315ffb640 10 client.6573 ms_handle_connect on v2:172.21.15.67:6834/451962260 2024-03-24T00:42:20.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.216+0000 7f5315ffb640 1 -- 172.21.15.67:0/968083295 <== mds.0 v2:172.21.15.67:6834/451962260 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5639747ab910 con 0x5639748994a0 2024-03-24T00:42:20.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.216+0000 7f5315ffb640 10 client.6573 handle_command_reply: tid=0 2024-03-24T00:42:20.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.216+0000 7f534022f740 1 client.6573 shutdown 2024-03-24T00:42:20.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.216+0000 7f534022f740 1 -- 172.21.15.67:0/968083295 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563974865740 msgr2=0x563974865ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:20.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.216+0000 7f534022f740 1 --2- 172.21.15.67:0/968083295 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563974865740 0x563974865ba0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f5330037040 tx=0x7f5330004ab0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.216+0000 7f534022f740 1 -- 172.21.15.67:0/968083295 shutdown_connections 2024-03-24T00:42:20.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.216+0000 7f534022f740 1 --2- 172.21.15.67:0/968083295 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x5639748994a0 0x56397489b890 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.216+0000 7f534022f740 1 --2- 172.21.15.67:0/968083295 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563974865740 0x563974865ba0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.216+0000 7f534022f740 1 --2- 172.21.15.67:0/968083295 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563974861230 0x563974865200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.216+0000 7f5315ffb640 0 client.6573 ms_handle_reset on v2:172.21.15.67:6834/451962260 2024-03-24T00:42:20.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.216+0000 7f534022f740 1 --2- 172.21.15.67:0/968083295 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563974860e50 0x563974862fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.216+0000 7f534022f740 1 -- 172.21.15.67:0/968083295 >> 172.21.15.67:0/968083295 conn(0x563974834330 msgr2=0x563974837d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:20.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.216+0000 7f534022f740 1 -- 172.21.15.67:0/968083295 shutdown_connections 2024-03-24T00:42:20.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.216+0000 7f534022f740 1 -- 172.21.15.67:0/968083295 wait complete. 2024-03-24T00:42:20.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.216+0000 7f534022f740 20 client.6573 trim_cache size 0 max 16384 2024-03-24T00:42:20.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.247+0000 7f534022f740 1 Processor -- start 2024-03-24T00:42:20.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.247+0000 7f534022f740 1 -- start start 2024-03-24T00:42:20.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.248+0000 7f534022f740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563974860e50 0x56397485fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.248+0000 7f534022f740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563974861230 0x563974860360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.248+0000 7f534022f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563974865740 0x5639748a0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.248+0000 7f534022f740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5338200790 con 0x563974865740 2024-03-24T00:42:20.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.248+0000 7f534022f740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53381ff980 con 0x563974860e50 2024-03-24T00:42:20.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.248+0000 7f534022f740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53380ae180 con 0x563974861230 2024-03-24T00:42:20.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.248+0000 7f53377fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563974865740 0x5639748a0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.248+0000 7f53377fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563974865740 0x5639748a0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42676/0 (socket says 172.21.15.67:42676) 2024-03-24T00:42:20.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.248+0000 7f53377fe640 1 -- 172.21.15.67:0/1018017162 learned_addr learned my addr 172.21.15.67:0/1018017162 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:20.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.248+0000 7f5336ffd640 1 --2- 172.21.15.67:0/1018017162 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563974860e50 0x56397485fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.248+0000 7f53367fc640 1 --2- 172.21.15.67:0/1018017162 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563974861230 0x563974860360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.248+0000 7f53377fe640 1 -- 172.21.15.67:0/1018017162 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563974861230 msgr2=0x563974860360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:20.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.248+0000 7f53377fe640 1 --2- 172.21.15.67:0/1018017162 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563974861230 0x563974860360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.248+0000 7f53377fe640 1 -- 172.21.15.67:0/1018017162 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563974860e50 msgr2=0x56397485fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:20.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.248+0000 7f53377fe640 1 --2- 172.21.15.67:0/1018017162 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563974860e50 0x56397485fe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.248+0000 7f53377fe640 1 -- 172.21.15.67:0/1018017162 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5639747a3650 con 0x563974865740 2024-03-24T00:42:20.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.249+0000 7f53377fe640 1 --2- 172.21.15.67:0/1018017162 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563974865740 0x5639748a0470 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f533000c070 tx=0x7f5330040000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d282728f9a9853 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:20.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.249+0000 7f5317fff640 1 -- 172.21.15.67:0/1018017162 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5330046070 con 0x563974865740 2024-03-24T00:42:20.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.249+0000 7f5317fff640 1 -- 172.21.15.67:0/1018017162 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5330040670 con 0x563974865740 2024-03-24T00:42:20.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.249+0000 7f5317fff640 1 -- 172.21.15.67:0/1018017162 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5330039040 con 0x563974865740 2024-03-24T00:42:20.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.249+0000 7f534022f740 1 -- 172.21.15.67:0/1018017162 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563974865740 msgr2=0x5639748a0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:20.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.249+0000 7f534022f740 1 --2- 172.21.15.67:0/1018017162 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563974865740 0x5639748a0470 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f533000c070 tx=0x7f5330040000 comp rx=0 tx=0).stop 2024-03-24T00:42:20.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.249+0000 7f534022f740 1 -- 172.21.15.67:0/1018017162 shutdown_connections 2024-03-24T00:42:20.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.249+0000 7f534022f740 1 --2- 172.21.15.67:0/1018017162 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563974865740 0x5639748a0470 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.249+0000 7f534022f740 1 --2- 172.21.15.67:0/1018017162 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563974861230 0x563974860360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.249+0000 7f534022f740 1 --2- 172.21.15.67:0/1018017162 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563974860e50 0x56397485fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.249+0000 7f534022f740 1 -- 172.21.15.67:0/1018017162 >> 172.21.15.67:0/1018017162 conn(0x563974834330 msgr2=0x563974863920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:20.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.249+0000 7f534022f740 1 -- 172.21.15.67:0/1018017162 shutdown_connections 2024-03-24T00:42:20.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.249+0000 7f534022f740 1 -- 172.21.15.67:0/1018017162 wait complete. 2024-03-24T00:42:20.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.250+0000 7f534022f740 1 -- start start 2024-03-24T00:42:20.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.250+0000 7f534022f740 1 Processor -- start 2024-03-24T00:42:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.251+0000 7f534022f740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563974860e50 0x56397489b890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.251+0000 7f534022f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563974861230 0x5639748e7970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.251+0000 7f534022f740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563974865740 0x56397485a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.251+0000 7f534022f740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x563974867da0 con 0x563974861230 2024-03-24T00:42:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.251+0000 7f534022f740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53381ff380 con 0x563974860e50 2024-03-24T00:42:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.251+0000 7f534022f740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5639747a0870 con 0x563974865740 2024-03-24T00:42:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.251+0000 7f534022f740 10 client.0 _ll_register_callbacks cb 0x7f5330009c90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:42:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.251+0000 7f53367fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563974861230 0x5639748e7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.251+0000 7f53367fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563974861230 0x5639748e7970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42692/0 (socket says 172.21.15.67:42692) 2024-03-24T00:42:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.251+0000 7f53367fc640 1 -- 172.21.15.67:0/2526465393 learned_addr learned my addr 172.21.15.67:0/2526465393 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.251+0000 7f5336ffd640 1 --2- 172.21.15.67:0/2526465393 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563974860e50 0x56397489b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.251+0000 7f53377fe640 1 --2- 172.21.15.67:0/2526465393 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563974865740 0x56397485a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.251+0000 7f53367fc640 1 -- 172.21.15.67:0/2526465393 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563974865740 msgr2=0x56397485a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.251+0000 7f53367fc640 1 --2- 172.21.15.67:0/2526465393 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563974865740 0x56397485a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.251+0000 7f53367fc640 1 -- 172.21.15.67:0/2526465393 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563974860e50 msgr2=0x56397489b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:20.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.251+0000 7f53367fc640 1 --2- 172.21.15.67:0/2526465393 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563974860e50 0x56397489b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.251+0000 7f53367fc640 1 -- 172.21.15.67:0/2526465393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563974897570 con 0x563974861230 2024-03-24T00:42:20.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.251+0000 7f53367fc640 1 --2- 172.21.15.67:0/2526465393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563974861230 0x5639748e7970 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f532c00db30 tx=0x7f532c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c471504dc4dc38e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:20.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.251+0000 7f5315ffb640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:42:20.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.251+0000 7f5315ffb640 1 -- 172.21.15.67:0/2526465393 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f532c037060 con 0x563974861230 2024-03-24T00:42:20.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.252+0000 7f5315ffb640 1 -- 172.21.15.67:0/2526465393 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(62..62 src has 1..62) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f532c01e650 con 0x563974861230 2024-03-24T00:42:20.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.252+0000 7f534022f740 1 -- 172.21.15.67:0/2526465393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5330040670 con 0x563974861230 2024-03-24T00:42:20.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.252+0000 7f5315ffb640 1 -- 172.21.15.67:0/2526465393 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f532c019020 con 0x563974861230 2024-03-24T00:42:20.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.252+0000 7f534022f740 10 client.6579 fetch_fsmap learned FSMap version 33 2024-03-24T00:42:20.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.252+0000 7f534022f740 1 -- 172.21.15.67:0/2526465393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x563974865b20 con 0x563974861230 2024-03-24T00:42:20.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.252+0000 7f5315ffb640 1 -- 172.21.15.67:0/2526465393 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5330040670 con 0x563974861230 2024-03-24T00:42:20.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.252+0000 7f5315ffb640 1 -- 172.21.15.67:0/2526465393 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 33) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f532c040080 con 0x563974861230 2024-03-24T00:42:20.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.252+0000 7f534022f740 10 client.6579 fetch_fsmap finished waiting for FSMap version 33 2024-03-24T00:42:20.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.252+0000 7f534022f740 10 client.6579 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.a 2024-03-24T00:42:20.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.252+0000 7f534022f740 20 client.6579 populate_metadata read hostname 'smithi067' 2024-03-24T00:42:20.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.252+0000 7f534022f740 1 --2- 172.21.15.67:0/2526465393 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x563974880df0 0x56397485aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.252+0000 7f534022f740 4 client.6579 mds_command: new command op to 6360 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:42:20.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.252+0000 7f534022f740 1 -- 172.21.15.67:0/2526465393 --> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5639747a5c60 con 0x563974880df0 2024-03-24T00:42:20.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.252+0000 7f5336ffd640 1 --2- 172.21.15.67:0/2526465393 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x563974880df0 0x56397485aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.252+0000 7f5315ffb640 10 client.6579 ms_handle_connect on v2:172.21.15.67:6834/451962260 2024-03-24T00:42:20.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.252+0000 7f5336ffd640 1 --2- 172.21.15.67:0/2526465393 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x563974880df0 0x56397485aab0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc0d23959505e00a server_cookie=c02a77f447abde22 in_seq=0 out_seq=0 2024-03-24T00:42:20.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.253+0000 7f5315ffb640 1 -- 172.21.15.67:0/2526465393 <== mds.0 v2:172.21.15.67:6834/451962260 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5639747a5c60 con 0x563974880df0 2024-03-24T00:42:20.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.253+0000 7f5315ffb640 10 client.6579 handle_command_reply: tid=0 2024-03-24T00:42:20.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.253+0000 7f534022f740 1 client.6579 shutdown 2024-03-24T00:42:20.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.253+0000 7f534022f740 1 -- 172.21.15.67:0/2526465393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563974861230 msgr2=0x5639748e7970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:20.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.253+0000 7f534022f740 1 --2- 172.21.15.67:0/2526465393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563974861230 0x5639748e7970 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f532c00db30 tx=0x7f532c004730 comp rx=0 tx=0).stop 2024-03-24T00:42:20.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.253+0000 7f534022f740 1 -- 172.21.15.67:0/2526465393 shutdown_connections 2024-03-24T00:42:20.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.253+0000 7f534022f740 1 --2- 172.21.15.67:0/2526465393 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x563974880df0 0x56397485aab0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.253+0000 7f534022f740 1 --2- 172.21.15.67:0/2526465393 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563974865740 0x56397485a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.253+0000 7f534022f740 1 --2- 172.21.15.67:0/2526465393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563974861230 0x5639748e7970 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.253+0000 7f5315ffb640 0 client.6579 ms_handle_reset on v2:172.21.15.67:6834/451962260 2024-03-24T00:42:20.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.253+0000 7f534022f740 1 --2- 172.21.15.67:0/2526465393 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563974860e50 0x56397489b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.253+0000 7f534022f740 1 -- 172.21.15.67:0/2526465393 >> 172.21.15.67:0/2526465393 conn(0x563974834330 msgr2=0x563974833510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:20.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.253+0000 7f534022f740 1 -- 172.21.15.67:0/2526465393 shutdown_connections 2024-03-24T00:42:20.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.253+0000 7f534022f740 1 -- 172.21.15.67:0/2526465393 wait complete. 2024-03-24T00:42:20.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.253+0000 7f534022f740 20 client.6579 trim_cache size 0 max 16384 2024-03-24T00:42:20.259 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:42:20.259 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "no active scrubs running", 2024-03-24T00:42:20.259 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": {} 2024-03-24T00:42:20.259 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:42:20.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.253+0000 7f533d562640 1 -- 172.21.15.67:0/118756353 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f532406bbd0 msgr2=0x7f532406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:20.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.253+0000 7f533d562640 1 --2- 172.21.15.67:0/118756353 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f532406bbd0 0x7f532406e090 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f53380b1530 tx=0x7f532c008040 comp rx=0 tx=0).stop 2024-03-24T00:42:20.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.253+0000 7f533d562640 1 -- 172.21.15.67:0/118756353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f533806ec60 msgr2=0x7f53381f78b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:20.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.253+0000 7f533d562640 1 --2- 172.21.15.67:0/118756353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f533806ec60 0x7f53381f78b0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f532800c430 tx=0x7f532800c8f0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.253+0000 7f533d562640 1 -- 172.21.15.67:0/118756353 shutdown_connections 2024-03-24T00:42:20.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.253+0000 7f533d562640 1 --2- 172.21.15.67:0/118756353 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f532406bbd0 0x7f532406e090 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.253+0000 7f533d562640 1 --2- 172.21.15.67:0/118756353 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5338144670 0x7f53380b0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.253+0000 7f533d562640 1 --2- 172.21.15.67:0/118756353 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f53380b4a00 0x7f533815b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.253+0000 7f533d562640 1 --2- 172.21.15.67:0/118756353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f533806ec60 0x7f53381f78b0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.253+0000 7f533d562640 1 -- 172.21.15.67:0/118756353 >> 172.21.15.67:0/118756353 conn(0x7f53380cf9e0 msgr2=0x7f53380cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:20.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.253+0000 7f533d562640 1 -- 172.21.15.67:0/118756353 shutdown_connections 2024-03-24T00:42:20.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.253+0000 7f533d562640 1 -- 172.21.15.67:0/118756353 wait complete. 2024-03-24T00:42:20.265 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T00:42:20.265 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:42:20.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.365+0000 7f49d52f1640 1 Processor -- start 2024-03-24T00:42:20.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.365+0000 7f49d52f1640 1 -- start start 2024-03-24T00:42:20.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.366+0000 7f49d52f1640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f49d015e1b0 0x7f49d00cfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.366+0000 7f49d52f1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f49d009a410 0x7f49d00cc880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.366+0000 7f49d52f1640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49d00d0fc0 con 0x7f49d00b43a0 2024-03-24T00:42:20.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.366+0000 7f49d52f1640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49d015e590 con 0x7f49d015e1b0 2024-03-24T00:42:20.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.366+0000 7f49d52f1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49d00d1820 con 0x7f49d009a410 2024-03-24T00:42:20.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.366+0000 7f49c7fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f49d00b43a0 0x7f49d0099d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33838/0 (socket says 172.21.15.67:33838) 2024-03-24T00:42:20.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.366+0000 7f49c7fff640 1 -- 172.21.15.67:0/3750169454 learned_addr learned my addr 172.21.15.67:0/3750169454 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:42:20.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.366+0000 7f49ceffd640 1 --2- 172.21.15.67:0/3750169454 >> v2:172.21.15.67:3300/0 conn(0x7f49d015e1b0 0x7f49d00cfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.366+0000 7f49cf7fe640 1 --2- 172.21.15.67:0/3750169454 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f49d009a410 0x7f49d00cc880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.366+0000 7f49ce7fc640 1 -- 172.21.15.67:0/3750169454 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3133890752 0 0) 0x7f49d00d0fc0 con 0x7f49d00b43a0 2024-03-24T00:42:20.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.366+0000 7f49ce7fc640 1 -- 172.21.15.67:0/3750169454 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49bc003520 con 0x7f49d00b43a0 2024-03-24T00:42:20.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.366+0000 7f49ceffd640 1 -- 172.21.15.67:0/3750169454 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f49d009a410 msgr2=0x7f49d00cc880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:20.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.366+0000 7f49ceffd640 1 --2- 172.21.15.67:0/3750169454 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f49d009a410 0x7f49d00cc880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.366+0000 7f49ceffd640 1 -- 172.21.15.67:0/3750169454 >> v1:172.21.15.67:6789/0 conn(0x7f49d00b43a0 legacy=0x7f49d0099d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:20.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.366+0000 7f49ceffd640 1 -- 172.21.15.67:0/3750169454 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49d00d1260 con 0x7f49d015e1b0 2024-03-24T00:42:20.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.367+0000 7f49ceffd640 1 --2- 172.21.15.67:0/3750169454 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49d015e1b0 0x7f49d00cfe20 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f49c00097a0 tx=0x7f49c002b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=268bbe0426a1a3fe server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:20.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.367+0000 7f49ce7fc640 1 -- 172.21.15.67:0/3750169454 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c0038070 con 0x7f49d015e1b0 2024-03-24T00:42:20.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.367+0000 7f49ce7fc640 1 -- 172.21.15.67:0/3750169454 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c0031700 con 0x7f49d015e1b0 2024-03-24T00:42:20.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.367+0000 7f49ce7fc640 1 -- 172.21.15.67:0/3750169454 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c0031ab0 con 0x7f49d015e1b0 2024-03-24T00:42:20.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.368+0000 7f49d52f1640 1 -- 172.21.15.67:0/3750169454 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49d015e1b0 msgr2=0x7f49d00cfe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:20.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.368+0000 7f49d52f1640 1 --2- 172.21.15.67:0/3750169454 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49d015e1b0 0x7f49d00cfe20 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f49c00097a0 tx=0x7f49c002b7a0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.368+0000 7f49d52f1640 1 -- 172.21.15.67:0/3750169454 shutdown_connections 2024-03-24T00:42:20.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.368+0000 7f49d52f1640 1 --2- 172.21.15.67:0/3750169454 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f49d009a410 0x7f49d00cc880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.368+0000 7f49d52f1640 1 --2- 172.21.15.67:0/3750169454 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49d015e1b0 0x7f49d00cfe20 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.368+0000 7f49d52f1640 1 -- 172.21.15.67:0/3750169454 >> 172.21.15.67:0/3750169454 conn(0x7f49d00b36d0 msgr2=0x7f49d00b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:20.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.368+0000 7f49d52f1640 1 -- 172.21.15.67:0/3750169454 shutdown_connections 2024-03-24T00:42:20.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.368+0000 7f49d52f1640 1 -- 172.21.15.67:0/3750169454 wait complete. 2024-03-24T00:42:20.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.369+0000 7f49d52f1640 1 Processor -- start 2024-03-24T00:42:20.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.369+0000 7f49d52f1640 1 -- start start 2024-03-24T00:42:20.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.370+0000 7f49d52f1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49d009a410 0x7f49d014f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.370+0000 7f49d52f1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f49d00b43a0 0x7f49d00b06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.370+0000 7f49d52f1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f49d015e1b0 0x7f49d00ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.370+0000 7f49d52f1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d009a9a0 con 0x7f49d009a410 2024-03-24T00:42:20.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.370+0000 7f49d52f1640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d00cc360 con 0x7f49d015e1b0 2024-03-24T00:42:20.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.370+0000 7f49d52f1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49d00ccdc0 con 0x7f49d00b43a0 2024-03-24T00:42:20.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.370+0000 7f49ceffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49d009a410 0x7f49d014f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.370+0000 7f49ceffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49d009a410 0x7f49d014f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42718/0 (socket says 172.21.15.67:42718) 2024-03-24T00:42:20.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.370+0000 7f49ceffd640 1 -- 172.21.15.67:0/1139882740 learned_addr learned my addr 172.21.15.67:0/1139882740 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:20.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.370+0000 7f49cf7fe640 1 --2- 172.21.15.67:0/1139882740 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f49d015e1b0 0x7f49d00ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.370+0000 7f49c7fff640 1 --2- 172.21.15.67:0/1139882740 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f49d00b43a0 0x7f49d00b06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.370+0000 7f49ceffd640 1 -- 172.21.15.67:0/1139882740 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f49d00b43a0 msgr2=0x7f49d00b06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:20.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.370+0000 7f49ceffd640 1 --2- 172.21.15.67:0/1139882740 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f49d00b43a0 0x7f49d00b06f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.370+0000 7f49ceffd640 1 -- 172.21.15.67:0/1139882740 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f49d015e1b0 msgr2=0x7f49d00ae8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:20.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.370+0000 7f49ceffd640 1 --2- 172.21.15.67:0/1139882740 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f49d015e1b0 0x7f49d00ae8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.370+0000 7f49ceffd640 1 -- 172.21.15.67:0/1139882740 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49d014fce0 con 0x7f49d009a410 2024-03-24T00:42:20.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.370+0000 7f49cf7fe640 1 --2- 172.21.15.67:0/1139882740 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f49d015e1b0 0x7f49d00ae8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:42:20.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.370+0000 7f49ceffd640 1 --2- 172.21.15.67:0/1139882740 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49d009a410 0x7f49d014f370 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f49c002b7e0 tx=0x7f49c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:20.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.370+0000 7f49c77fe640 1 -- 172.21.15.67:0/1139882740 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c0042070 con 0x7f49d009a410 2024-03-24T00:42:20.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.370+0000 7f49c77fe640 1 -- 172.21.15.67:0/1139882740 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c003d870 con 0x7f49d009a410 2024-03-24T00:42:20.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.370+0000 7f49d52f1640 1 -- 172.21.15.67:0/1139882740 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49d014ffc0 con 0x7f49d009a410 2024-03-24T00:42:20.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.370+0000 7f49c77fe640 1 -- 172.21.15.67:0/1139882740 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c0038040 con 0x7f49d009a410 2024-03-24T00:42:20.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.371+0000 7f49d52f1640 1 -- 172.21.15.67:0/1139882740 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49d0150580 con 0x7f49d009a410 2024-03-24T00:42:20.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.372+0000 7f49d52f1640 1 -- 172.21.15.67:0/1139882740 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49d004f400 con 0x7f49d009a410 2024-03-24T00:42:20.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.372+0000 7f49c77fe640 1 -- 172.21.15.67:0/1139882740 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f49c0045070 con 0x7f49d009a410 2024-03-24T00:42:20.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.372+0000 7f49c77fe640 1 --2- 172.21.15.67:0/1139882740 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f49b006bbd0 0x7f49b006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.373+0000 7f49c77fe640 1 -- 172.21.15.67:0/1139882740 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f49c00ac640 con 0x7f49d009a410 2024-03-24T00:42:20.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.373+0000 7f49c7fff640 1 --2- 172.21.15.67:0/1139882740 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f49b006bbd0 0x7f49b006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.373+0000 7f49c7fff640 1 --2- 172.21.15.67:0/1139882740 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f49b006bbd0 0x7f49b006e090 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f49d00708b0 tx=0x7f49b8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:20.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.377+0000 7f49c77fe640 1 -- 172.21.15.67:0/1139882740 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f49c0030070 con 0x7f49d009a410 2024-03-24T00:42:20.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.598+0000 7f49d52f1640 1 -- 172.21.15.67:0/1139882740 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49d006c5a0 con 0x7f49d009a410 2024-03-24T00:42:20.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.598+0000 7f49c77fe640 1 -- 172.21.15.67:0/1139882740 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4975 (secure 0 0 0) 0x7f49c007b150 con 0x7f49d009a410 2024-03-24T00:42:20.599 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:42:20.599 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5627,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2941726873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2941726873},{"type":"v1","addr":"172.21.15.72:6837","nonce":2941726873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":6166,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/1957901519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":1957901519},{"type":"v1","addr":"172.21.15.72:6836","nonce":1957901519}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":6357,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/2843754917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2843754917},{"type":"v1","addr":"172.21.15.67:6833","nonce":2843754917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:41:42.242211+0000","modified":"2024-03-24T00:41:54.909011+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6360},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6360":{"gid":6360,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":6,"addr":"172.21.15.67:6835/451962260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":451962260},{"type":"v1","addr":"172.21.15.67:6835","nonce":451962260}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6360,"qdb_cluster":[6360]},"id":4}]} 2024-03-24T00:42:20.599 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 33 2024-03-24T00:42:20.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.600+0000 7f49d52f1640 1 -- 172.21.15.67:0/1139882740 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f49b006bbd0 msgr2=0x7f49b006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:20.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.600+0000 7f49d52f1640 1 --2- 172.21.15.67:0/1139882740 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f49b006bbd0 0x7f49b006e090 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f49d00708b0 tx=0x7f49b8009040 comp rx=0 tx=0).stop 2024-03-24T00:42:20.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.600+0000 7f49d52f1640 1 -- 172.21.15.67:0/1139882740 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49d009a410 msgr2=0x7f49d014f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:20.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.600+0000 7f49d52f1640 1 --2- 172.21.15.67:0/1139882740 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49d009a410 0x7f49d014f370 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f49c002b7e0 tx=0x7f49c00046c0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.600+0000 7f49d52f1640 1 -- 172.21.15.67:0/1139882740 shutdown_connections 2024-03-24T00:42:20.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.600+0000 7f49d52f1640 1 --2- 172.21.15.67:0/1139882740 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f49b006bbd0 0x7f49b006e090 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.600+0000 7f49d52f1640 1 --2- 172.21.15.67:0/1139882740 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f49d015e1b0 0x7f49d00ae8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.600+0000 7f49d52f1640 1 --2- 172.21.15.67:0/1139882740 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f49d00b43a0 0x7f49d00b06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.600+0000 7f49d52f1640 1 --2- 172.21.15.67:0/1139882740 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49d009a410 0x7f49d014f370 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.600+0000 7f49d52f1640 1 -- 172.21.15.67:0/1139882740 >> 172.21.15.67:0/1139882740 conn(0x7f49d00b36d0 msgr2=0x7f49d006d580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:20.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.600+0000 7f49d52f1640 1 -- 172.21.15.67:0/1139882740 shutdown_connections 2024-03-24T00:42:20.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.600+0000 7f49d52f1640 1 -- 172.21.15.67:0/1139882740 wait complete. 2024-03-24T00:42:20.612 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a damage ls --format=json-pretty 2024-03-24T00:42:20.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.715+0000 7f8c3afc6640 1 Processor -- start 2024-03-24T00:42:20.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.715+0000 7f8c3afc6640 1 -- start start 2024-03-24T00:42:20.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.716+0000 7f8c3afc6640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8c340d3ae0 0x7f8c340b5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.716+0000 7f8c3afc6640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c340d2250 con 0x7f8c3415dea0 2024-03-24T00:42:20.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.716+0000 7f8c3afc6640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c340d2530 con 0x7f8c3415e280 2024-03-24T00:42:20.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.716+0000 7f8c3afc6640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c340d2810 con 0x7f8c340d3ae0 2024-03-24T00:42:20.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.716+0000 7f8c33fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8c340d3ae0 0x7f8c340b5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.716+0000 7f8c38d62640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8c3415dea0 0x7f8c340d33d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33846/0 (socket says 172.21.15.67:33846) 2024-03-24T00:42:20.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.716+0000 7f8c38d62640 1 -- 172.21.15.67:0/655450568 learned_addr learned my addr 172.21.15.67:0/655450568 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:42:20.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.716+0000 7f8c337fe640 1 -- 172.21.15.67:0/655450568 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1969003985 0 0) 0x7f8c340d2250 con 0x7f8c3415dea0 2024-03-24T00:42:20.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.717+0000 7f8c337fe640 1 -- 172.21.15.67:0/655450568 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c24003520 con 0x7f8c3415dea0 2024-03-24T00:42:20.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.717+0000 7f8c33fff640 1 -- 172.21.15.67:0/655450568 >> v1:172.21.15.72:6789/0 conn(0x7f8c3415e280 legacy=0x7f8c340b70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:20.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.717+0000 7f8c33fff640 1 -- 172.21.15.67:0/655450568 >> v1:172.21.15.67:6789/0 conn(0x7f8c3415dea0 legacy=0x7f8c340d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:20.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.717+0000 7f8c33fff640 1 -- 172.21.15.67:0/655450568 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c340d2af0 con 0x7f8c340d3ae0 2024-03-24T00:42:20.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.718+0000 7f8c33fff640 1 --2- 172.21.15.67:0/655450568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c340d3ae0 0x7f8c340b5a60 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f8c200097d0 tx=0x7f8c2002b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d99a5a2ae2c58120 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:20.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.718+0000 7f8c337fe640 1 -- 172.21.15.67:0/655450568 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c20031780 con 0x7f8c340d3ae0 2024-03-24T00:42:20.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.718+0000 7f8c337fe640 1 -- 172.21.15.67:0/655450568 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c20037070 con 0x7f8c340d3ae0 2024-03-24T00:42:20.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.718+0000 7f8c337fe640 1 -- 172.21.15.67:0/655450568 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c20031b40 con 0x7f8c340d3ae0 2024-03-24T00:42:20.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.718+0000 7f8c3afc6640 1 -- 172.21.15.67:0/655450568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c340d3ae0 msgr2=0x7f8c340b5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:20.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.718+0000 7f8c3afc6640 1 --2- 172.21.15.67:0/655450568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c340d3ae0 0x7f8c340b5a60 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f8c200097d0 tx=0x7f8c2002b820 comp rx=0 tx=0).stop 2024-03-24T00:42:20.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.718+0000 7f8c3afc6640 1 -- 172.21.15.67:0/655450568 shutdown_connections 2024-03-24T00:42:20.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.718+0000 7f8c3afc6640 1 --2- 172.21.15.67:0/655450568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c340d3ae0 0x7f8c340b5a60 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.718+0000 7f8c3afc6640 1 -- 172.21.15.67:0/655450568 >> 172.21.15.67:0/655450568 conn(0x7f8c34072da0 msgr2=0x7f8c340d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:20.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.719+0000 7f8c3afc6640 1 -- 172.21.15.67:0/655450568 shutdown_connections 2024-03-24T00:42:20.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.719+0000 7f8c3afc6640 1 -- 172.21.15.67:0/655450568 wait complete. 2024-03-24T00:42:20.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.720+0000 7f8c3afc6640 1 Processor -- start 2024-03-24T00:42:20.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.720+0000 7f8c3afc6640 1 -- start start 2024-03-24T00:42:20.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.722+0000 7f8c3afc6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8c340d3ae0 0x7f8c3414c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.722+0000 7f8c3afc6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c3415dea0 0x7f8c340cc640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.722+0000 7f8c3afc6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8c3415e280 0x7f8c341602f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.722+0000 7f8c3afc6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c340b6370 con 0x7f8c3415dea0 2024-03-24T00:42:20.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.722+0000 7f8c3afc6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c340b61f0 con 0x7f8c3415e280 2024-03-24T00:42:20.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.722+0000 7f8c3afc6640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c340b7800 con 0x7f8c340d3ae0 2024-03-24T00:42:20.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.722+0000 7f8c33fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c3415dea0 0x7f8c340cc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.722+0000 7f8c33fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c3415dea0 0x7f8c340cc640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42730/0 (socket says 172.21.15.67:42730) 2024-03-24T00:42:20.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.722+0000 7f8c33fff640 1 -- 172.21.15.67:0/226230736 learned_addr learned my addr 172.21.15.67:0/226230736 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:20.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.722+0000 7f8c38d62640 1 --2- 172.21.15.67:0/226230736 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8c340d3ae0 0x7f8c3414c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.722+0000 7f8c33fff640 1 -- 172.21.15.67:0/226230736 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8c340d3ae0 msgr2=0x7f8c3414c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:20.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.722+0000 7f8c39563640 1 --2- 172.21.15.67:0/226230736 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8c3415e280 0x7f8c341602f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.722+0000 7f8c33fff640 1 --2- 172.21.15.67:0/226230736 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8c340d3ae0 0x7f8c3414c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.722+0000 7f8c33fff640 1 -- 172.21.15.67:0/226230736 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8c3415e280 msgr2=0x7f8c341602f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:20.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.722+0000 7f8c33fff640 1 --2- 172.21.15.67:0/226230736 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8c3415e280 0x7f8c341602f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.722+0000 7f8c33fff640 1 -- 172.21.15.67:0/226230736 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c340cd7a0 con 0x7f8c3415dea0 2024-03-24T00:42:20.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.722+0000 7f8c33fff640 1 --2- 172.21.15.67:0/226230736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c3415dea0 0x7f8c340cc640 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f8c200097d0 tx=0x7f8c200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:20.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.722+0000 7f8c317fa640 1 -- 172.21.15.67:0/226230736 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c20032610 con 0x7f8c3415dea0 2024-03-24T00:42:20.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.722+0000 7f8c317fa640 1 -- 172.21.15.67:0/226230736 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c200327b0 con 0x7f8c3415dea0 2024-03-24T00:42:20.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.722+0000 7f8c3afc6640 1 -- 172.21.15.67:0/226230736 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c340cbb60 con 0x7f8c3415dea0 2024-03-24T00:42:20.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.722+0000 7f8c317fa640 1 -- 172.21.15.67:0/226230736 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c20032ba0 con 0x7f8c3415dea0 2024-03-24T00:42:20.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.723+0000 7f8c3afc6640 1 -- 172.21.15.67:0/226230736 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c340cda80 con 0x7f8c3415dea0 2024-03-24T00:42:20.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.724+0000 7f8c317fa640 1 -- 172.21.15.67:0/226230736 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8c2004c060 con 0x7f8c3415dea0 2024-03-24T00:42:20.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.724+0000 7f8c317fa640 1 --2- 172.21.15.67:0/226230736 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8c1806bbd0 0x7f8c1806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.725+0000 7f8c317fa640 1 -- 172.21.15.67:0/226230736 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8c200ac260 con 0x7f8c3415dea0 2024-03-24T00:42:20.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.725+0000 7f8c38d62640 1 --2- 172.21.15.67:0/226230736 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8c1806bbd0 0x7f8c1806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.725+0000 7f8c38d62640 1 --2- 172.21.15.67:0/226230736 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8c1806bbd0 0x7f8c1806e090 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f8c28000f60 tx=0x7f8c28009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:20.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.726+0000 7f8c3dba8740 1 Processor -- start 2024-03-24T00:42:20.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.726+0000 7f8c3dba8740 1 -- start start 2024-03-24T00:42:20.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.726+0000 7f8c3dba8740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a4d620d0 0x5581a4d35300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.726+0000 7f8c3dba8740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5581a4d624b0 0x5581a4d64e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.726+0000 7f8c3dba8740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5581a4d35840 0x5581a4d653a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.726+0000 7f8c3dba8740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581a4c9f1f0 con 0x5581a4d620d0 2024-03-24T00:42:20.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.726+0000 7f8c3dba8740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581a4c3da80 con 0x5581a4d35840 2024-03-24T00:42:20.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.726+0000 7f8c3dba8740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5581a4c9b290 con 0x5581a4d624b0 2024-03-24T00:42:20.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.727+0000 7f8c38d62640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a4d620d0 0x5581a4d35300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.727+0000 7f8c38d62640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a4d620d0 0x5581a4d35300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42732/0 (socket says 172.21.15.67:42732) 2024-03-24T00:42:20.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.727+0000 7f8c38d62640 1 -- 172.21.15.67:0/3641573579 learned_addr learned my addr 172.21.15.67:0/3641573579 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:20.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.727+0000 7f8c33fff640 1 --2- 172.21.15.67:0/3641573579 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5581a4d624b0 0x5581a4d64e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.727+0000 7f8c39563640 1 --2- 172.21.15.67:0/3641573579 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5581a4d35840 0x5581a4d653a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.727+0000 7f8c38d62640 1 -- 172.21.15.67:0/3641573579 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5581a4d624b0 msgr2=0x5581a4d64e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:20.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.727+0000 7f8c38d62640 1 --2- 172.21.15.67:0/3641573579 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5581a4d624b0 0x5581a4d64e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.727+0000 7f8c38d62640 1 -- 172.21.15.67:0/3641573579 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5581a4d35840 msgr2=0x5581a4d653a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:20.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.727+0000 7f8c38d62640 1 --2- 172.21.15.67:0/3641573579 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5581a4d35840 0x5581a4d653a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.727+0000 7f8c38d62640 1 -- 172.21.15.67:0/3641573579 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5581a4a7f7c0 con 0x5581a4d620d0 2024-03-24T00:42:20.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.727+0000 7f8c39563640 1 --2- 172.21.15.67:0/3641573579 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5581a4d35840 0x5581a4d653a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:42:20.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.727+0000 7f8c38d62640 1 --2- 172.21.15.67:0/3641573579 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a4d620d0 0x5581a4d35300 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f8c28005cd0 tx=0x7f8c28010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8f9d13e4e4c427f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:20.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.727+0000 7f8c1e7fc640 1 -- 172.21.15.67:0/3641573579 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c2800da10 con 0x5581a4d620d0 2024-03-24T00:42:20.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.727+0000 7f8c1e7fc640 1 -- 172.21.15.67:0/3641573579 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c2800dbb0 con 0x5581a4d620d0 2024-03-24T00:42:20.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.727+0000 7f8c1e7fc640 1 -- 172.21.15.67:0/3641573579 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c28020450 con 0x5581a4d620d0 2024-03-24T00:42:20.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.728+0000 7f8c3dba8740 1 -- 172.21.15.67:0/3641573579 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a4d620d0 msgr2=0x5581a4d35300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:20.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.728+0000 7f8c3dba8740 1 --2- 172.21.15.67:0/3641573579 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a4d620d0 0x5581a4d35300 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f8c28005cd0 tx=0x7f8c28010000 comp rx=0 tx=0).stop 2024-03-24T00:42:20.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.728+0000 7f8c3dba8740 1 -- 172.21.15.67:0/3641573579 shutdown_connections 2024-03-24T00:42:20.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.728+0000 7f8c3dba8740 1 --2- 172.21.15.67:0/3641573579 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5581a4d35840 0x5581a4d653a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.728+0000 7f8c3dba8740 1 --2- 172.21.15.67:0/3641573579 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5581a4d624b0 0x5581a4d64e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.728+0000 7f8c3dba8740 1 --2- 172.21.15.67:0/3641573579 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a4d620d0 0x5581a4d35300 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.728+0000 7f8c3dba8740 1 -- 172.21.15.67:0/3641573579 >> 172.21.15.67:0/3641573579 conn(0x5581a4d37570 msgr2=0x5581a4d7fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:20.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.728+0000 7f8c3dba8740 1 -- 172.21.15.67:0/3641573579 shutdown_connections 2024-03-24T00:42:20.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.728+0000 7f8c3dba8740 1 -- 172.21.15.67:0/3641573579 wait complete. 2024-03-24T00:42:20.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.729+0000 7f8c3dba8740 1 -- start start 2024-03-24T00:42:20.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.729+0000 7f8c3dba8740 1 Processor -- start 2024-03-24T00:42:20.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.730+0000 7f8c3dba8740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a4d620d0 0x5581a4d67940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.730+0000 7f8c3dba8740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5581a4d624b0 0x5581a4d76980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.730+0000 7f8c3dba8740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5581a4d76ec0 0x5581a4d7a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.730+0000 7f8c3dba8740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581a4d39880 con 0x5581a4d620d0 2024-03-24T00:42:20.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.730+0000 7f8c3dba8740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581a4c3b410 con 0x5581a4d624b0 2024-03-24T00:42:20.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.730+0000 7f8c3dba8740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5581a4d38c70 con 0x5581a4d76ec0 2024-03-24T00:42:20.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.730+0000 7f8c3dba8740 10 client.0 _ll_register_callbacks cb 0x5581a4c63d70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:42:20.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.730+0000 7f8c38d62640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a4d620d0 0x5581a4d67940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.730+0000 7f8c38d62640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a4d620d0 0x5581a4d67940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42734/0 (socket says 172.21.15.67:42734) 2024-03-24T00:42:20.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.730+0000 7f8c38d62640 1 -- 172.21.15.67:0/2556363076 learned_addr learned my addr 172.21.15.67:0/2556363076 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:20.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.730+0000 7f8c33fff640 1 --2- 172.21.15.67:0/2556363076 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5581a4d624b0 0x5581a4d76980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.730+0000 7f8c39563640 1 --2- 172.21.15.67:0/2556363076 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5581a4d76ec0 0x5581a4d7a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.731+0000 7f8c38d62640 1 -- 172.21.15.67:0/2556363076 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5581a4d76ec0 msgr2=0x5581a4d7a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:20.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.731+0000 7f8c38d62640 1 --2- 172.21.15.67:0/2556363076 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5581a4d76ec0 0x5581a4d7a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.731+0000 7f8c38d62640 1 -- 172.21.15.67:0/2556363076 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5581a4d624b0 msgr2=0x5581a4d76980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:20.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.731+0000 7f8c38d62640 1 --2- 172.21.15.67:0/2556363076 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5581a4d624b0 0x5581a4d76980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.731+0000 7f8c38d62640 1 -- 172.21.15.67:0/2556363076 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5581a4adff50 con 0x5581a4d620d0 2024-03-24T00:42:20.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.731+0000 7f8c38d62640 1 --2- 172.21.15.67:0/2556363076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a4d620d0 0x5581a4d67940 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f8c2800d860 tx=0x7f8c28012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=956daa965000b7d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.731+0000 7f8c137fe640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:42:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.731+0000 7f8c137fe640 1 -- 172.21.15.67:0/2556363076 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c28005440 con 0x5581a4d620d0 2024-03-24T00:42:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.731+0000 7f8c137fe640 1 -- 172.21.15.67:0/2556363076 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(62..62 src has 1..62) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8c280055e0 con 0x5581a4d620d0 2024-03-24T00:42:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.731+0000 7f8c3dba8740 1 -- 172.21.15.67:0/2556363076 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8c2800dbb0 con 0x5581a4d620d0 2024-03-24T00:42:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.731+0000 7f8c137fe640 1 -- 172.21.15.67:0/2556363076 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8c28036050 con 0x5581a4d620d0 2024-03-24T00:42:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.731+0000 7f8c3dba8740 10 client.6603 fetch_fsmap learned FSMap version 33 2024-03-24T00:42:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.731+0000 7f8c3dba8740 1 -- 172.21.15.67:0/2556363076 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x5581a49730b0 con 0x5581a4d620d0 2024-03-24T00:42:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.731+0000 7f8c137fe640 1 -- 172.21.15.67:0/2556363076 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c2800dbb0 con 0x5581a4d620d0 2024-03-24T00:42:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.731+0000 7f8c137fe640 1 -- 172.21.15.67:0/2556363076 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 33) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f8c28039020 con 0x5581a4d620d0 2024-03-24T00:42:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.732+0000 7f8c3dba8740 10 client.6603 fetch_fsmap finished waiting for FSMap version 33 2024-03-24T00:42:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.732+0000 7f8c3dba8740 10 client.6603 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-24T00:42:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.732+0000 7f8c3dba8740 20 client.6603 populate_metadata read hostname 'smithi067' 2024-03-24T00:42:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.732+0000 7f8c3dba8740 1 --2- 172.21.15.67:0/2556363076 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x5581a4d9b990 0x5581a4d9dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.732+0000 7f8c3dba8740 4 client.6603 mds_command: new command op to 6360 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:42:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.732+0000 7f8c3dba8740 1 -- 172.21.15.67:0/2556363076 --> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5581a4cac730 con 0x5581a4d9b990 2024-03-24T00:42:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.732+0000 7f8c33fff640 1 --2- 172.21.15.67:0/2556363076 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x5581a4d9b990 0x5581a4d9dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.732+0000 7f8c137fe640 10 client.6603 ms_handle_connect on v2:172.21.15.67:6834/451962260 2024-03-24T00:42:20.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.732+0000 7f8c33fff640 1 --2- 172.21.15.67:0/2556363076 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x5581a4d9b990 0x5581a4d9dd80 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f302be972656280d server_cookie=b25e1eeb390a36ba in_seq=0 out_seq=0 2024-03-24T00:42:20.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.733+0000 7f8c137fe640 1 -- 172.21.15.67:0/2556363076 <== mds.0 v2:172.21.15.67:6834/451962260 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5581a4cac730 con 0x5581a4d9b990 2024-03-24T00:42:20.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.733+0000 7f8c137fe640 10 client.6603 handle_command_reply: tid=0 2024-03-24T00:42:20.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.733+0000 7f8c3dba8740 1 client.6603 shutdown 2024-03-24T00:42:20.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.733+0000 7f8c3dba8740 1 -- 172.21.15.67:0/2556363076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a4d620d0 msgr2=0x5581a4d67940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:20.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.733+0000 7f8c3dba8740 1 --2- 172.21.15.67:0/2556363076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a4d620d0 0x5581a4d67940 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f8c2800d860 tx=0x7f8c28012000 comp rx=0 tx=0).stop 2024-03-24T00:42:20.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.733+0000 7f8c3dba8740 1 -- 172.21.15.67:0/2556363076 shutdown_connections 2024-03-24T00:42:20.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.733+0000 7f8c3dba8740 1 --2- 172.21.15.67:0/2556363076 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x5581a4d9b990 0x5581a4d9dd80 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.733+0000 7f8c3dba8740 1 --2- 172.21.15.67:0/2556363076 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5581a4d76ec0 0x5581a4d7a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.733+0000 7f8c3dba8740 1 --2- 172.21.15.67:0/2556363076 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5581a4d624b0 0x5581a4d76980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.733+0000 7f8c137fe640 0 client.6603 ms_handle_reset on v2:172.21.15.67:6834/451962260 2024-03-24T00:42:20.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.733+0000 7f8c3dba8740 1 --2- 172.21.15.67:0/2556363076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a4d620d0 0x5581a4d67940 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.733+0000 7f8c3dba8740 1 -- 172.21.15.67:0/2556363076 >> 172.21.15.67:0/2556363076 conn(0x5581a4d35840 msgr2=0x5581a4d64920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:20.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.733+0000 7f8c3dba8740 1 -- 172.21.15.67:0/2556363076 shutdown_connections 2024-03-24T00:42:20.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.733+0000 7f8c3dba8740 1 -- 172.21.15.67:0/2556363076 wait complete. 2024-03-24T00:42:20.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.733+0000 7f8c3dba8740 20 client.6603 trim_cache size 0 max 16384 2024-03-24T00:42:20.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.746+0000 7f8c3dba8740 1 Processor -- start 2024-03-24T00:42:20.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.746+0000 7f8c3dba8740 1 -- start start 2024-03-24T00:42:20.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.746+0000 7f8c3dba8740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5581a4d620d0 0x5581a4d38bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.746+0000 7f8c3dba8740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5581a4d624b0 0x5581a4d39110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.746+0000 7f8c3dba8740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a4d76ec0 0x5581a4d65e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.746+0000 7f8c3dba8740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c341ff980 con 0x5581a4d76ec0 2024-03-24T00:42:20.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.746+0000 7f8c3dba8740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c3415f910 con 0x5581a4d620d0 2024-03-24T00:42:20.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.746+0000 7f8c3dba8740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c341ff380 con 0x5581a4d624b0 2024-03-24T00:42:20.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.747+0000 7f8c39563640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a4d76ec0 0x5581a4d65e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.747+0000 7f8c39563640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a4d76ec0 0x5581a4d65e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42740/0 (socket says 172.21.15.67:42740) 2024-03-24T00:42:20.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.747+0000 7f8c39563640 1 -- 172.21.15.67:0/2479987732 learned_addr learned my addr 172.21.15.67:0/2479987732 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:20.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.747+0000 7f8c38d62640 1 --2- 172.21.15.67:0/2479987732 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5581a4d620d0 0x5581a4d38bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.747+0000 7f8c33fff640 1 --2- 172.21.15.67:0/2479987732 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5581a4d624b0 0x5581a4d39110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.747+0000 7f8c39563640 1 -- 172.21.15.67:0/2479987732 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5581a4d624b0 msgr2=0x5581a4d39110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:20.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.747+0000 7f8c39563640 1 --2- 172.21.15.67:0/2479987732 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5581a4d624b0 0x5581a4d39110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.747+0000 7f8c39563640 1 -- 172.21.15.67:0/2479987732 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5581a4d620d0 msgr2=0x5581a4d38bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:20.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.747+0000 7f8c39563640 1 --2- 172.21.15.67:0/2479987732 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5581a4d620d0 0x5581a4d38bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.747+0000 7f8c39563640 1 -- 172.21.15.67:0/2479987732 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5581a4d76cf0 con 0x5581a4d76ec0 2024-03-24T00:42:20.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.747+0000 7f8c39563640 1 --2- 172.21.15.67:0/2479987732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a4d76ec0 0x5581a4d65e50 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c007d20 tx=0x7f8c2c0107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=946e4e0f59d6e2c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:20.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.747+0000 7f8c1dffb640 1 -- 172.21.15.67:0/2479987732 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c2c007e10 con 0x5581a4d76ec0 2024-03-24T00:42:20.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.747+0000 7f8c1dffb640 1 -- 172.21.15.67:0/2479987732 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c2c011650 con 0x5581a4d76ec0 2024-03-24T00:42:20.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.747+0000 7f8c1dffb640 1 -- 172.21.15.67:0/2479987732 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c2c011a80 con 0x5581a4d76ec0 2024-03-24T00:42:20.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.748+0000 7f8c3dba8740 1 -- 172.21.15.67:0/2479987732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a4d76ec0 msgr2=0x5581a4d65e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:20.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.748+0000 7f8c3dba8740 1 --2- 172.21.15.67:0/2479987732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a4d76ec0 0x5581a4d65e50 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c007d20 tx=0x7f8c2c0107a0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.748+0000 7f8c3dba8740 1 -- 172.21.15.67:0/2479987732 shutdown_connections 2024-03-24T00:42:20.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.748+0000 7f8c3dba8740 1 --2- 172.21.15.67:0/2479987732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a4d76ec0 0x5581a4d65e50 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.748+0000 7f8c3dba8740 1 --2- 172.21.15.67:0/2479987732 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5581a4d624b0 0x5581a4d39110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.748+0000 7f8c3dba8740 1 --2- 172.21.15.67:0/2479987732 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5581a4d620d0 0x5581a4d38bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.748+0000 7f8c3dba8740 1 -- 172.21.15.67:0/2479987732 >> 172.21.15.67:0/2479987732 conn(0x5581a4d35840 msgr2=0x5581a4d68920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:20.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.748+0000 7f8c3dba8740 1 -- 172.21.15.67:0/2479987732 shutdown_connections 2024-03-24T00:42:20.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.748+0000 7f8c3dba8740 1 -- 172.21.15.67:0/2479987732 wait complete. 2024-03-24T00:42:20.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.748+0000 7f8c3dba8740 1 -- start start 2024-03-24T00:42:20.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.749+0000 7f8c3dba8740 1 Processor -- start 2024-03-24T00:42:20.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.749+0000 7f8c3dba8740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5581a4d620d0 0x5581a4da2a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.749+0000 7f8c3dba8740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5581a4d624b0 0x5581a4d945f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.749+0000 7f8c3dba8740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a4d76ec0 0x5581a4d3a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.749+0000 7f8c3dba8740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581a4d784c0 con 0x5581a4d76ec0 2024-03-24T00:42:20.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.749+0000 7f8c3dba8740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581a4d39880 con 0x5581a4d624b0 2024-03-24T00:42:20.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.749+0000 7f8c3dba8740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5581a4d93830 con 0x5581a4d620d0 2024-03-24T00:42:20.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.749+0000 7f8c3dba8740 10 client.0 _ll_register_callbacks cb 0x7f8c28005ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:42:20.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.749+0000 7f8c39563640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a4d76ec0 0x5581a4d3a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.749+0000 7f8c39563640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a4d76ec0 0x5581a4d3a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42750/0 (socket says 172.21.15.67:42750) 2024-03-24T00:42:20.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.749+0000 7f8c39563640 1 -- 172.21.15.67:0/3793270182 learned_addr learned my addr 172.21.15.67:0/3793270182 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:20.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.749+0000 7f8c38d62640 1 --2- 172.21.15.67:0/3793270182 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5581a4d620d0 0x5581a4da2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.749+0000 7f8c33fff640 1 --2- 172.21.15.67:0/3793270182 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5581a4d624b0 0x5581a4d945f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.750+0000 7f8c39563640 1 -- 172.21.15.67:0/3793270182 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5581a4d620d0 msgr2=0x5581a4da2a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:20.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.750+0000 7f8c39563640 1 --2- 172.21.15.67:0/3793270182 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5581a4d620d0 0x5581a4da2a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.750+0000 7f8c39563640 1 -- 172.21.15.67:0/3793270182 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5581a4d624b0 msgr2=0x5581a4d945f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:20.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.750+0000 7f8c39563640 1 --2- 172.21.15.67:0/3793270182 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5581a4d624b0 0x5581a4d945f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.750+0000 7f8c39563640 1 -- 172.21.15.67:0/3793270182 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5581a4d94b30 con 0x5581a4d76ec0 2024-03-24T00:42:20.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.750+0000 7f8c39563640 1 --2- 172.21.15.67:0/3793270182 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a4d76ec0 0x5581a4d3a0f0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c010ca0 tx=0x7f8c2c004a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d9b40caa1090c1d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:20.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.750+0000 7f8c137fe640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:42:20.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.750+0000 7f8c137fe640 1 -- 172.21.15.67:0/3793270182 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c2c028030 con 0x5581a4d76ec0 2024-03-24T00:42:20.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.750+0000 7f8c137fe640 1 -- 172.21.15.67:0/3793270182 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(62..62 src has 1..62) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8c2c01f040 con 0x5581a4d76ec0 2024-03-24T00:42:20.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.750+0000 7f8c3dba8740 1 -- 172.21.15.67:0/3793270182 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8c2c011650 con 0x5581a4d76ec0 2024-03-24T00:42:20.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.750+0000 7f8c137fe640 1 -- 172.21.15.67:0/3793270182 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8c2c031050 con 0x5581a4d76ec0 2024-03-24T00:42:20.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.750+0000 7f8c3dba8740 10 client.6609 fetch_fsmap learned FSMap version 33 2024-03-24T00:42:20.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.750+0000 7f8c3dba8740 1 -- 172.21.15.67:0/3793270182 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x5581a4defdf0 con 0x5581a4d76ec0 2024-03-24T00:42:20.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.750+0000 7f8c137fe640 1 -- 172.21.15.67:0/3793270182 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c2c011650 con 0x5581a4d76ec0 2024-03-24T00:42:20.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.750+0000 7f8c137fe640 1 -- 172.21.15.67:0/3793270182 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 33) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f8c2c034020 con 0x5581a4d76ec0 2024-03-24T00:42:20.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.751+0000 7f8c3dba8740 10 client.6609 fetch_fsmap finished waiting for FSMap version 33 2024-03-24T00:42:20.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.751+0000 7f8c3dba8740 10 client.6609 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-24T00:42:20.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.751+0000 7f8c3dba8740 20 client.6609 populate_metadata read hostname 'smithi067' 2024-03-24T00:42:20.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.751+0000 7f8c3dba8740 1 --2- 172.21.15.67:0/3793270182 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x5581a4dee740 0x5581a4deeb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.751+0000 7f8c3dba8740 4 client.6609 mds_command: new command op to 6360 tid=0 multi_id=0 [{"prefix": "damage ls", "format": "json-pretty"}] 2024-03-24T00:42:20.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.751+0000 7f8c3dba8740 1 -- 172.21.15.67:0/3793270182 --> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) v1 -- 0x5581a4ca6a80 con 0x5581a4dee740 2024-03-24T00:42:20.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.751+0000 7f8c38d62640 1 --2- 172.21.15.67:0/3793270182 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x5581a4dee740 0x5581a4deeb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.751+0000 7f8c137fe640 10 client.6609 ms_handle_connect on v2:172.21.15.67:6834/451962260 2024-03-24T00:42:20.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.751+0000 7f8c38d62640 1 --2- 172.21.15.67:0/3793270182 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x5581a4dee740 0x5581a4deeb20 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=99486a295d445545 server_cookie=e28c2d9e37e0c430 in_seq=0 out_seq=0 2024-03-24T00:42:20.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.751+0000 7f8c137fe640 1 -- 172.21.15.67:0/3793270182 <== mds.0 v2:172.21.15.67:6834/451962260 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+141 (crc 0 0 0) 0x5581a4ca6a80 con 0x5581a4dee740 2024-03-24T00:42:20.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.751+0000 7f8c137fe640 10 client.6609 handle_command_reply: tid=0 2024-03-24T00:42:20.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.751+0000 7f8c3dba8740 1 client.6609 shutdown 2024-03-24T00:42:20.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.751+0000 7f8c3dba8740 1 -- 172.21.15.67:0/3793270182 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a4d76ec0 msgr2=0x5581a4d3a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:20.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.751+0000 7f8c3dba8740 1 --2- 172.21.15.67:0/3793270182 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a4d76ec0 0x5581a4d3a0f0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c010ca0 tx=0x7f8c2c004a00 comp rx=0 tx=0).stop 2024-03-24T00:42:20.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.751+0000 7f8c3dba8740 1 -- 172.21.15.67:0/3793270182 shutdown_connections 2024-03-24T00:42:20.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.751+0000 7f8c3dba8740 1 --2- 172.21.15.67:0/3793270182 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x5581a4dee740 0x5581a4deeb20 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.751+0000 7f8c3dba8740 1 --2- 172.21.15.67:0/3793270182 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a4d76ec0 0x5581a4d3a0f0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.751+0000 7f8c3dba8740 1 --2- 172.21.15.67:0/3793270182 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5581a4d624b0 0x5581a4d945f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.751+0000 7f8c3dba8740 1 --2- 172.21.15.67:0/3793270182 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5581a4d620d0 0x5581a4da2a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.751+0000 7f8c137fe640 0 client.6609 ms_handle_reset on v2:172.21.15.67:6834/451962260 2024-03-24T00:42:20.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.751+0000 7f8c3dba8740 1 -- 172.21.15.67:0/3793270182 >> 172.21.15.67:0/3793270182 conn(0x5581a4d35840 msgr2=0x5581a4d669e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:20.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.752+0000 7f8c3dba8740 1 -- 172.21.15.67:0/3793270182 shutdown_connections 2024-03-24T00:42:20.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.752+0000 7f8c3dba8740 1 -- 172.21.15.67:0/3793270182 wait complete. 2024-03-24T00:42:20.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.752+0000 7f8c3dba8740 20 client.6609 trim_cache size 0 max 16384 2024-03-24T00:42:20.760 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:42:20.760 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-03-24T00:42:20.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:42:20.760 INFO:teuthology.orchestra.run.smithi067.stdout: "damage_type": "backtrace", 2024-03-24T00:42:20.760 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 2297774589, 2024-03-24T00:42:20.760 INFO:teuthology.orchestra.run.smithi067.stdout: "ino": 1099511627777, 2024-03-24T00:42:20.760 INFO:teuthology.orchestra.run.smithi067.stdout: "path": "/dir_test/file" 2024-03-24T00:42:20.760 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:42:20.760 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-03-24T00:42:20.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.752+0000 7f8c3afc6640 1 -- 172.21.15.67:0/226230736 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8c1806bbd0 msgr2=0x7f8c1806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:20.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.752+0000 7f8c3afc6640 1 --2- 172.21.15.67:0/226230736 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8c1806bbd0 0x7f8c1806e090 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f8c28000f60 tx=0x7f8c28009040 comp rx=0 tx=0).stop 2024-03-24T00:42:20.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.752+0000 7f8c3afc6640 1 -- 172.21.15.67:0/226230736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c3415dea0 msgr2=0x7f8c340cc640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:20.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.752+0000 7f8c3afc6640 1 --2- 172.21.15.67:0/226230736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c3415dea0 0x7f8c340cc640 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f8c200097d0 tx=0x7f8c200046c0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.752+0000 7f8c3afc6640 1 -- 172.21.15.67:0/226230736 shutdown_connections 2024-03-24T00:42:20.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.752+0000 7f8c3afc6640 1 --2- 172.21.15.67:0/226230736 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8c1806bbd0 0x7f8c1806e090 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.752+0000 7f8c3afc6640 1 --2- 172.21.15.67:0/226230736 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8c3415e280 0x7f8c341602f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.752+0000 7f8c3afc6640 1 --2- 172.21.15.67:0/226230736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c3415dea0 0x7f8c340cc640 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.752+0000 7f8c3afc6640 1 --2- 172.21.15.67:0/226230736 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8c340d3ae0 0x7f8c3414c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.752+0000 7f8c3afc6640 1 -- 172.21.15.67:0/226230736 >> 172.21.15.67:0/226230736 conn(0x7f8c34072da0 msgr2=0x7f8c340b39e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:20.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.752+0000 7f8c3afc6640 1 -- 172.21.15.67:0/226230736 shutdown_connections 2024-03-24T00:42:20.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.752+0000 7f8c3afc6640 1 -- 172.21.15.67:0/226230736 wait complete. 2024-03-24T00:42:20.764 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-24T00:42:20.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.864+0000 7f07a8dbc640 1 Processor -- start 2024-03-24T00:42:20.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.864+0000 7f07a8dbc640 1 -- start start 2024-03-24T00:42:20.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.865+0000 7f07a8dbc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f07a415dec0 0x7f07a40ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.865+0000 7f07a8dbc640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f07a415e2a0 0x7f07a4162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.865+0000 7f07a8dbc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f07a40d07e0 0x7f07a40d0c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.865+0000 7f07a8dbc640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07a40d1190 con 0x7f07a415dec0 2024-03-24T00:42:20.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.865+0000 7f07a8dbc640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07a40cee30 con 0x7f07a40d07e0 2024-03-24T00:42:20.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.865+0000 7f07a8dbc640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07a40d2120 con 0x7f07a415e2a0 2024-03-24T00:42:20.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.865+0000 7f07a259c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f07a415dec0 0x7f07a40ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.865+0000 7f07a259c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f07a415dec0 0x7f07a40ce630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42754/0 (socket says 172.21.15.67:42754) 2024-03-24T00:42:20.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.865+0000 7f07a259c640 1 -- 172.21.15.67:0/3773736539 learned_addr learned my addr 172.21.15.67:0/3773736539 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:20.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.865+0000 7f079bfff640 1 --2- 172.21.15.67:0/3773736539 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f07a415e2a0 0x7f07a4162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.865+0000 7f07a2d9d640 1 --2- 172.21.15.67:0/3773736539 >> v2:172.21.15.72:3300/0 conn(0x7f07a40d07e0 0x7f07a40d0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.865+0000 7f07a259c640 1 -- 172.21.15.67:0/3773736539 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f07a415e2a0 msgr2=0x7f07a4162630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:20.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.866+0000 7f07a259c640 1 --2- 172.21.15.67:0/3773736539 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f07a415e2a0 0x7f07a4162630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.866+0000 7f07a259c640 1 -- 172.21.15.67:0/3773736539 >> v2:172.21.15.72:3300/0 conn(0x7f07a40d07e0 msgr2=0x7f07a40d0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:20.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.866+0000 7f07a259c640 1 --2- 172.21.15.67:0/3773736539 >> v2:172.21.15.72:3300/0 conn(0x7f07a40d07e0 0x7f07a40d0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.866+0000 7f07a259c640 1 -- 172.21.15.67:0/3773736539 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a40cf640 con 0x7f07a415dec0 2024-03-24T00:42:20.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.866+0000 7f07a2d9d640 1 --2- 172.21.15.67:0/3773736539 >> v2:172.21.15.72:3300/0 conn(0x7f07a40d07e0 0x7f07a40d0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:42:20.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.867+0000 7f07a259c640 1 --2- 172.21.15.67:0/3773736539 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07a415dec0 0x7f07a40ce630 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f07a40d1a50 tx=0x7f079402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ea96a960e84341c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:20.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.867+0000 7f07a1d9b640 1 -- 172.21.15.67:0/3773736539 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f079402e070 con 0x7f07a415dec0 2024-03-24T00:42:20.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.867+0000 7f07a1d9b640 1 -- 172.21.15.67:0/3773736539 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0794037070 con 0x7f07a415dec0 2024-03-24T00:42:20.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.867+0000 7f07a1d9b640 1 -- 172.21.15.67:0/3773736539 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0794031a00 con 0x7f07a415dec0 2024-03-24T00:42:20.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.867+0000 7f07a8dbc640 1 -- 172.21.15.67:0/3773736539 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07a415dec0 msgr2=0x7f07a40ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:20.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.867+0000 7f07a8dbc640 1 --2- 172.21.15.67:0/3773736539 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07a415dec0 0x7f07a40ce630 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f07a40d1a50 tx=0x7f079402b880 comp rx=0 tx=0).stop 2024-03-24T00:42:20.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.867+0000 7f07a8dbc640 1 -- 172.21.15.67:0/3773736539 shutdown_connections 2024-03-24T00:42:20.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.867+0000 7f07a8dbc640 1 --2- 172.21.15.67:0/3773736539 >> v2:172.21.15.72:3300/0 conn(0x7f07a40d07e0 0x7f07a40d0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.867+0000 7f07a8dbc640 1 --2- 172.21.15.67:0/3773736539 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f07a415e2a0 0x7f07a4162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.867+0000 7f07a8dbc640 1 --2- 172.21.15.67:0/3773736539 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07a415dec0 0x7f07a40ce630 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.867+0000 7f07a8dbc640 1 -- 172.21.15.67:0/3773736539 >> 172.21.15.67:0/3773736539 conn(0x7f07a406c810 msgr2=0x7f07a406cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:20.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.867+0000 7f07a8dbc640 1 -- 172.21.15.67:0/3773736539 shutdown_connections 2024-03-24T00:42:20.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.867+0000 7f07a8dbc640 1 -- 172.21.15.67:0/3773736539 wait complete. 2024-03-24T00:42:20.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.869+0000 7f07a8dbc640 1 Processor -- start 2024-03-24T00:42:20.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.869+0000 7f07a8dbc640 1 -- start start 2024-03-24T00:42:20.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.869+0000 7f07a8dbc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07a40d07e0 0x7f07a415c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.869+0000 7f07a8dbc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07a415dec0 0x7f07a41e7de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.869+0000 7f07a8dbc640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f07a415e2a0 0x7f07a41f4210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.869+0000 7f07a8dbc640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a40b5850 con 0x7f07a415dec0 2024-03-24T00:42:20.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.869+0000 7f07a8dbc640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a40cf360 con 0x7f07a40d07e0 2024-03-24T00:42:20.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.869+0000 7f07a8dbc640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07a40b59d0 con 0x7f07a415e2a0 2024-03-24T00:42:20.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.870+0000 7f079bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07a415dec0 0x7f07a41e7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.870+0000 7f079bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07a415dec0 0x7f07a41e7de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42756/0 (socket says 172.21.15.67:42756) 2024-03-24T00:42:20.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.870+0000 7f079bfff640 1 -- 172.21.15.67:0/1636213724 learned_addr learned my addr 172.21.15.67:0/1636213724 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:20.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.870+0000 7f07a259c640 1 --2- 172.21.15.67:0/1636213724 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07a40d07e0 0x7f07a415c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.870+0000 7f07a2d9d640 1 --2- 172.21.15.67:0/1636213724 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f07a415e2a0 0x7f07a41f4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.870+0000 7f079bfff640 1 -- 172.21.15.67:0/1636213724 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f07a415e2a0 msgr2=0x7f07a41f4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:20.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.870+0000 7f079bfff640 1 --2- 172.21.15.67:0/1636213724 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f07a415e2a0 0x7f07a41f4210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.870+0000 7f079bfff640 1 -- 172.21.15.67:0/1636213724 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07a40d07e0 msgr2=0x7f07a415c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:20.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.870+0000 7f079bfff640 1 --2- 172.21.15.67:0/1636213724 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07a40d07e0 0x7f07a415c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:20.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.870+0000 7f079bfff640 1 -- 172.21.15.67:0/1636213724 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a415cd10 con 0x7f07a415dec0 2024-03-24T00:42:20.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.870+0000 7f07a259c640 1 --2- 172.21.15.67:0/1636213724 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07a40d07e0 0x7f07a415c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:42:20.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.870+0000 7f079bfff640 1 --2- 172.21.15.67:0/1636213724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07a415dec0 0x7f07a41e7de0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f078c00e840 tx=0x7f078c00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:20.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.870+0000 7f079affd640 1 -- 172.21.15.67:0/1636213724 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f078c017070 con 0x7f07a415dec0 2024-03-24T00:42:20.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.870+0000 7f079affd640 1 -- 172.21.15.67:0/1636213724 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f078c00c940 con 0x7f07a415dec0 2024-03-24T00:42:20.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.870+0000 7f07a8dbc640 1 -- 172.21.15.67:0/1636213724 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07a41e87c0 con 0x7f07a415dec0 2024-03-24T00:42:20.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.870+0000 7f079affd640 1 -- 172.21.15.67:0/1636213724 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f078c00cd10 con 0x7f07a415dec0 2024-03-24T00:42:20.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.870+0000 7f07a8dbc640 1 -- 172.21.15.67:0/1636213724 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07a41e8d80 con 0x7f07a415dec0 2024-03-24T00:42:20.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.871+0000 7f07a8dbc640 1 -- 172.21.15.67:0/1636213724 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07a404f400 con 0x7f07a415dec0 2024-03-24T00:42:20.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.872+0000 7f079affd640 1 -- 172.21.15.67:0/1636213724 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f078c01a070 con 0x7f07a415dec0 2024-03-24T00:42:20.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.872+0000 7f079affd640 1 --2- 172.21.15.67:0/1636213724 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f078406bbd0 0x7f078406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:20.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.872+0000 7f079affd640 1 -- 172.21.15.67:0/1636213724 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f078c08aa40 con 0x7f07a415dec0 2024-03-24T00:42:20.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.873+0000 7f07a259c640 1 --2- 172.21.15.67:0/1636213724 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f078406bbd0 0x7f078406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:20.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.873+0000 7f07a259c640 1 --2- 172.21.15.67:0/1636213724 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f078406bbd0 0x7f078406e090 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f0794036000 tx=0x7f079403d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:20.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:20.877+0000 7f079affd640 1 -- 172.21.15.67:0/1636213724 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f078c0593a0 con 0x7f07a415dec0 2024-03-24T00:42:21.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.160+0000 7f07a8dbc640 1 -- 172.21.15.67:0/1636213724 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f07a40b4f20 con 0x7f07a415dec0 2024-03-24T00:42:21.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.161+0000 7f079affd640 1 -- 172.21.15.67:0/1636213724 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+164 (secure 0 0 0) 0x7f078c058ca0 con 0x7f07a415dec0 2024-03-24T00:42:21.162 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:42:21.163 INFO:teuthology.orchestra.run.smithi067.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2024-03-24T00:42:21.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.163+0000 7f07a8dbc640 1 -- 172.21.15.67:0/1636213724 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f078406bbd0 msgr2=0x7f078406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:21.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.163+0000 7f07a8dbc640 1 --2- 172.21.15.67:0/1636213724 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f078406bbd0 0x7f078406e090 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f0794036000 tx=0x7f079403d040 comp rx=0 tx=0).stop 2024-03-24T00:42:21.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.163+0000 7f07a8dbc640 1 -- 172.21.15.67:0/1636213724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07a415dec0 msgr2=0x7f07a41e7de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:21.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.163+0000 7f07a8dbc640 1 --2- 172.21.15.67:0/1636213724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07a415dec0 0x7f07a41e7de0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f078c00e840 tx=0x7f078c00ed00 comp rx=0 tx=0).stop 2024-03-24T00:42:21.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.163+0000 7f07a8dbc640 1 -- 172.21.15.67:0/1636213724 shutdown_connections 2024-03-24T00:42:21.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.163+0000 7f07a8dbc640 1 --2- 172.21.15.67:0/1636213724 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f078406bbd0 0x7f078406e090 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.163+0000 7f07a8dbc640 1 --2- 172.21.15.67:0/1636213724 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f07a415e2a0 0x7f07a41f4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.163+0000 7f07a8dbc640 1 --2- 172.21.15.67:0/1636213724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07a415dec0 0x7f07a41e7de0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.163+0000 7f07a8dbc640 1 --2- 172.21.15.67:0/1636213724 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07a40d07e0 0x7f07a415c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.163+0000 7f07a8dbc640 1 -- 172.21.15.67:0/1636213724 >> 172.21.15.67:0/1636213724 conn(0x7f07a406c810 msgr2=0x7f07a40b5e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:21.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.163+0000 7f07a8dbc640 1 -- 172.21.15.67:0/1636213724 shutdown_connections 2024-03-24T00:42:21.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.163+0000 7f07a8dbc640 1 -- 172.21.15.67:0/1636213724 wait complete. 2024-03-24T00:42:21.175 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-03-24T00:42:21.175 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:42:21.175 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.4:0 scrub start / repair,recursive,force 2024-03-24T00:42:21.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.280+0000 7f4c330dc640 1 Processor -- start 2024-03-24T00:42:21.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.280+0000 7f4c330dc640 1 -- start start 2024-03-24T00:42:21.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.281+0000 7f4c330dc640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4c2c0cf160 0x7f4c2c0b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:21.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.281+0000 7f4c330dc640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c2c0d19a0 con 0x7f4c2c15f3c0 2024-03-24T00:42:21.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.281+0000 7f4c330dc640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c2c0d13e0 con 0x7f4c2c0b3b80 2024-03-24T00:42:21.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.281+0000 7f4c330dc640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c0d16c0 con 0x7f4c2c0cf160 2024-03-24T00:42:21.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.281+0000 7f4c30e78640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4c2c15f3c0 0x7f4c2c0ccb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33848/0 (socket says 172.21.15.67:33848) 2024-03-24T00:42:21.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.281+0000 7f4c30e78640 1 -- 172.21.15.67:0/431975329 learned_addr learned my addr 172.21.15.67:0/431975329 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:42:21.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.281+0000 7f4c2bfff640 1 --2- 172.21.15.67:0/431975329 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4c2c0cf160 0x7f4c2c0b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:21.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.281+0000 7f4c2b7fe640 1 -- 172.21.15.67:0/431975329 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3672142755 0 0) 0x7f4c2c0d19a0 con 0x7f4c2c15f3c0 2024-03-24T00:42:21.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.281+0000 7f4c2b7fe640 1 -- 172.21.15.67:0/431975329 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c1c003520 con 0x7f4c2c15f3c0 2024-03-24T00:42:21.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.281+0000 7f4c2b7fe640 1 -- 172.21.15.67:0/431975329 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2185106437 0 0) 0x7f4c2c0d13e0 con 0x7f4c2c0b3b80 2024-03-24T00:42:21.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.281+0000 7f4c2b7fe640 1 -- 172.21.15.67:0/431975329 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c2c0d19a0 con 0x7f4c2c0b3b80 2024-03-24T00:42:21.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.281+0000 7f4c2b7fe640 1 -- 172.21.15.67:0/431975329 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1249788940 0 0) 0x7f4c1c003520 con 0x7f4c2c15f3c0 2024-03-24T00:42:21.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.281+0000 7f4c2b7fe640 1 -- 172.21.15.67:0/431975329 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4c2c0cf160 msgr2=0x7f4c2c0b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:21.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.281+0000 7f4c2b7fe640 1 --2- 172.21.15.67:0/431975329 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4c2c0cf160 0x7f4c2c0b3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.281+0000 7f4c2b7fe640 1 -- 172.21.15.67:0/431975329 >> v1:172.21.15.72:6789/0 conn(0x7f4c2c0b3b80 legacy=0x7f4c2c0c5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:21.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.281+0000 7f4c2b7fe640 1 -- 172.21.15.67:0/431975329 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c2c0c57b0 con 0x7f4c2c15f3c0 2024-03-24T00:42:21.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.281+0000 7f4c2b7fe640 1 -- 172.21.15.67:0/431975329 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f4c20001670 con 0x7f4c2c15f3c0 2024-03-24T00:42:21.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.281+0000 7f4c2b7fe640 1 -- 172.21.15.67:0/431975329 >> v1:172.21.15.67:6789/0 conn(0x7f4c2c15f3c0 legacy=0x7f4c2c0ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:21.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.281+0000 7f4c2b7fe640 1 --2- 172.21.15.67:0/431975329 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c1c003b70 0x7f4c1c003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:21.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.281+0000 7f4c2b7fe640 1 --2- 172.21.15.67:0/431975329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c1c004800 0x7f4c1c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:21.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.281+0000 7f4c2b7fe640 1 --2- 172.21.15.67:0/431975329 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4c2c0cf160 0x7f4c1c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:21.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.281+0000 7f4c2b7fe640 1 -- 172.21.15.67:0/431975329 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c0d16c0 con 0x7f4c1c004800 2024-03-24T00:42:21.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.281+0000 7f4c2b7fe640 1 -- 172.21.15.67:0/431975329 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c1c00ba70 con 0x7f4c1c003b70 2024-03-24T00:42:21.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.281+0000 7f4c2b7fe640 1 -- 172.21.15.67:0/431975329 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c1c00bc40 con 0x7f4c2c0cf160 2024-03-24T00:42:21.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.282+0000 7f4c2bfff640 1 --2- 172.21.15.67:0/431975329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c1c004800 0x7f4c1c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:21.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.282+0000 7f4c31679640 1 --2- 172.21.15.67:0/431975329 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4c2c0cf160 0x7f4c1c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:21.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.282+0000 7f4c30e78640 1 --2- 172.21.15.67:0/431975329 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c1c003b70 0x7f4c1c003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:21.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.282+0000 7f4c2bfff640 1 -- 172.21.15.67:0/431975329 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4c2c0cf160 msgr2=0x7f4c1c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:21.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.282+0000 7f4c2bfff640 1 --2- 172.21.15.67:0/431975329 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4c2c0cf160 0x7f4c1c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.282+0000 7f4c2bfff640 1 -- 172.21.15.67:0/431975329 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c1c003b70 msgr2=0x7f4c1c003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:21.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.282+0000 7f4c2bfff640 1 --2- 172.21.15.67:0/431975329 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c1c003b70 0x7f4c1c003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.282+0000 7f4c2bfff640 1 -- 172.21.15.67:0/431975329 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c1c003520 con 0x7f4c1c004800 2024-03-24T00:42:21.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.283+0000 7f4c2bfff640 1 --2- 172.21.15.67:0/431975329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c1c004800 0x7f4c1c004c80 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f4c18009ac0 tx=0x7f4c1c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a44409444c6b2f57 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:21.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.283+0000 7f4c2b7fe640 1 -- 172.21.15.67:0/431975329 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c18039070 con 0x7f4c1c004800 2024-03-24T00:42:21.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.283+0000 7f4c2b7fe640 1 -- 172.21.15.67:0/431975329 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c1802c660 con 0x7f4c1c004800 2024-03-24T00:42:21.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.283+0000 7f4c2b7fe640 1 -- 172.21.15.67:0/431975329 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c18009690 con 0x7f4c1c004800 2024-03-24T00:42:21.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.283+0000 7f4c330dc640 1 -- 172.21.15.67:0/431975329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c1c004800 msgr2=0x7f4c1c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:21.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.283+0000 7f4c330dc640 1 --2- 172.21.15.67:0/431975329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c1c004800 0x7f4c1c004c80 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f4c18009ac0 tx=0x7f4c1c005830 comp rx=0 tx=0).stop 2024-03-24T00:42:21.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.284+0000 7f4c31679640 1 -- 172.21.15.67:0/431975329 reap_dead start 2024-03-24T00:42:21.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.284+0000 7f4c330dc640 1 -- 172.21.15.67:0/431975329 shutdown_connections 2024-03-24T00:42:21.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.284+0000 7f4c330dc640 1 -- 172.21.15.67:0/431975329 >> 172.21.15.67:0/431975329 conn(0x7f4c2c0ce640 msgr2=0x7f4c2c0cea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:21.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.284+0000 7f4c330dc640 1 -- 172.21.15.67:0/431975329 shutdown_connections 2024-03-24T00:42:21.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.284+0000 7f4c330dc640 1 -- 172.21.15.67:0/431975329 wait complete. 2024-03-24T00:42:21.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.285+0000 7f4c330dc640 1 Processor -- start 2024-03-24T00:42:21.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.285+0000 7f4c330dc640 1 -- start start 2024-03-24T00:42:21.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.286+0000 7f4c330dc640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4c1c004800 0x7f4c2c1ef000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:21.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.286+0000 7f4c330dc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c2c0c8c90 0x7f4c2c1f2540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:21.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.286+0000 7f4c330dc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c2c0b5830 0x7f4c2c0b5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:21.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.286+0000 7f4c330dc640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c0b37c0 con 0x7f4c2c0c8c90 2024-03-24T00:42:21.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.286+0000 7f4c330dc640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c0b3640 con 0x7f4c2c0b5830 2024-03-24T00:42:21.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.286+0000 7f4c330dc640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c0b3940 con 0x7f4c1c004800 2024-03-24T00:42:21.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.286+0000 7f4c2bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c2c0c8c90 0x7f4c2c1f2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:21.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.286+0000 7f4c2bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c2c0c8c90 0x7f4c2c1f2540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42776/0 (socket says 172.21.15.67:42776) 2024-03-24T00:42:21.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.286+0000 7f4c2bfff640 1 -- 172.21.15.67:0/296653048 learned_addr learned my addr 172.21.15.67:0/296653048 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:21.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.286+0000 7f4c30e78640 1 --2- 172.21.15.67:0/296653048 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4c1c004800 0x7f4c2c1ef000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:21.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.286+0000 7f4c31679640 1 --2- 172.21.15.67:0/296653048 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c2c0b5830 0x7f4c2c0b5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:21.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.286+0000 7f4c2bfff640 1 -- 172.21.15.67:0/296653048 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4c1c004800 msgr2=0x7f4c2c1ef000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:21.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.286+0000 7f4c2bfff640 1 --2- 172.21.15.67:0/296653048 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4c1c004800 0x7f4c2c1ef000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.286+0000 7f4c2bfff640 1 -- 172.21.15.67:0/296653048 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c2c0b5830 msgr2=0x7f4c2c0b5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:21.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.286+0000 7f4c2bfff640 1 --2- 172.21.15.67:0/296653048 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c2c0b5830 0x7f4c2c0b5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.286+0000 7f4c2bfff640 1 -- 172.21.15.67:0/296653048 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c2c1f2a80 con 0x7f4c2c0c8c90 2024-03-24T00:42:21.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.286+0000 7f4c2bfff640 1 --2- 172.21.15.67:0/296653048 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c2c0c8c90 0x7f4c2c1f2540 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f4c18009ac0 tx=0x7f4c18004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:21.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.286+0000 7f4c297fa640 1 -- 172.21.15.67:0/296653048 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c18039050 con 0x7f4c2c0c8c90 2024-03-24T00:42:21.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.286+0000 7f4c297fa640 1 -- 172.21.15.67:0/296653048 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c18045070 con 0x7f4c2c0c8c90 2024-03-24T00:42:21.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.286+0000 7f4c330dc640 1 -- 172.21.15.67:0/296653048 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c2c0c9190 con 0x7f4c2c0c8c90 2024-03-24T00:42:21.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.286+0000 7f4c297fa640 1 -- 172.21.15.67:0/296653048 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c18003830 con 0x7f4c2c0c8c90 2024-03-24T00:42:21.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.286+0000 7f4c330dc640 1 -- 172.21.15.67:0/296653048 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c2c0c9750 con 0x7f4c2c0c8c90 2024-03-24T00:42:21.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.288+0000 7f4c297fa640 1 -- 172.21.15.67:0/296653048 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4c180039d0 con 0x7f4c2c0c8c90 2024-03-24T00:42:21.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.288+0000 7f4c297fa640 1 --2- 172.21.15.67:0/296653048 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4c1006bbd0 0x7f4c1006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:21.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.288+0000 7f4c297fa640 1 -- 172.21.15.67:0/296653048 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4c180ac650 con 0x7f4c2c0c8c90 2024-03-24T00:42:21.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.289+0000 7f4c30e78640 1 --2- 172.21.15.67:0/296653048 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4c1006bbd0 0x7f4c1006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:21.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.289+0000 7f4c30e78640 1 --2- 172.21.15.67:0/296653048 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4c1006bbd0 0x7f4c1006e090 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f4c20001030 tx=0x7f4c20009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:21.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.290+0000 7f4c35da7740 1 Processor -- start 2024-03-24T00:42:21.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.290+0000 7f4c35da7740 1 -- start start 2024-03-24T00:42:21.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.291+0000 7f4c35da7740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5635b32e74a0 0x5635b32ba440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:21.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.291+0000 7f4c35da7740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5635b32e7880 0x5635b32ea230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:21.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.291+0000 7f4c35da7740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5635b32ba980 0x5635b32ea770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:21.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.291+0000 7f4c35da7740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635b32241d0 con 0x5635b32ba980 2024-03-24T00:42:21.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.291+0000 7f4c35da7740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635b31ccdc0 con 0x5635b32e7880 2024-03-24T00:42:21.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.291+0000 7f4c35da7740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5635b3220270 con 0x5635b32e74a0 2024-03-24T00:42:21.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.291+0000 7f4c31679640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5635b32ba980 0x5635b32ea770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:21.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.291+0000 7f4c31679640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5635b32ba980 0x5635b32ea770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42790/0 (socket says 172.21.15.67:42790) 2024-03-24T00:42:21.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.291+0000 7f4c31679640 1 -- 172.21.15.67:0/1626935080 learned_addr learned my addr 172.21.15.67:0/1626935080 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:21.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.291+0000 7f4c30e78640 1 --2- 172.21.15.67:0/1626935080 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5635b32e74a0 0x5635b32ba440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:21.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.291+0000 7f4c2bfff640 1 --2- 172.21.15.67:0/1626935080 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5635b32e7880 0x5635b32ea230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:21.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.292+0000 7f4c31679640 1 -- 172.21.15.67:0/1626935080 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5635b32e74a0 msgr2=0x5635b32ba440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:21.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.292+0000 7f4c31679640 1 --2- 172.21.15.67:0/1626935080 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5635b32e74a0 0x5635b32ba440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.292+0000 7f4c31679640 1 -- 172.21.15.67:0/1626935080 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5635b32e7880 msgr2=0x5635b32ea230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:21.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.292+0000 7f4c31679640 1 --2- 172.21.15.67:0/1626935080 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5635b32e7880 0x5635b32ea230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.292+0000 7f4c31679640 1 -- 172.21.15.67:0/1626935080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5635b3004940 con 0x5635b32ba980 2024-03-24T00:42:21.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.292+0000 7f4c31679640 1 --2- 172.21.15.67:0/1626935080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5635b32ba980 0x5635b32ea770 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c072780 tx=0x7f4c24001ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df0473e53e9b6617 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:21.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.292+0000 7f4c167fc640 1 -- 172.21.15.67:0/1626935080 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c2401a070 con 0x5635b32ba980 2024-03-24T00:42:21.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.292+0000 7f4c167fc640 1 -- 172.21.15.67:0/1626935080 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c24005520 con 0x5635b32ba980 2024-03-24T00:42:21.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.292+0000 7f4c167fc640 1 -- 172.21.15.67:0/1626935080 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c24015430 con 0x5635b32ba980 2024-03-24T00:42:21.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.293+0000 7f4c35da7740 1 -- 172.21.15.67:0/1626935080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5635b32ba980 msgr2=0x5635b32ea770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:21.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.293+0000 7f4c35da7740 1 --2- 172.21.15.67:0/1626935080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5635b32ba980 0x5635b32ea770 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c072780 tx=0x7f4c24001ea0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.293+0000 7f4c35da7740 1 -- 172.21.15.67:0/1626935080 shutdown_connections 2024-03-24T00:42:21.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.293+0000 7f4c35da7740 1 --2- 172.21.15.67:0/1626935080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5635b32ba980 0x5635b32ea770 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.293+0000 7f4c35da7740 1 --2- 172.21.15.67:0/1626935080 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5635b32e7880 0x5635b32ea230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.293+0000 7f4c35da7740 1 --2- 172.21.15.67:0/1626935080 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5635b32e74a0 0x5635b32ba440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.293+0000 7f4c35da7740 1 -- 172.21.15.67:0/1626935080 >> 172.21.15.67:0/1626935080 conn(0x5635b32bc6b0 msgr2=0x5635b3305230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:21.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.293+0000 7f4c35da7740 1 -- 172.21.15.67:0/1626935080 shutdown_connections 2024-03-24T00:42:21.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.293+0000 7f4c35da7740 1 -- 172.21.15.67:0/1626935080 wait complete. 2024-03-24T00:42:21.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.293+0000 7f4c35da7740 1 -- start start 2024-03-24T00:42:21.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.294+0000 7f4c35da7740 1 Processor -- start 2024-03-24T00:42:21.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.294+0000 7f4c35da7740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5635b32e74a0 0x5635b32ec5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:21.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.294+0000 7f4c35da7740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5635b32e7880 0x5635b3305850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:21.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.294+0000 7f4c35da7740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5635b32eb700 0x5635b32da150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:21.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.294+0000 7f4c35da7740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635b32bed50 con 0x5635b32eb700 2024-03-24T00:42:21.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.294+0000 7f4c35da7740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635b31e0220 con 0x5635b32e74a0 2024-03-24T00:42:21.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.294+0000 7f4c35da7740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5635b32c1590 con 0x5635b32e7880 2024-03-24T00:42:21.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.294+0000 7f4c35da7740 10 client.0 _ll_register_callbacks cb 0x5635b321c0a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:42:21.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.294+0000 7f4c31679640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5635b32eb700 0x5635b32da150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:21.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.294+0000 7f4c31679640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5635b32eb700 0x5635b32da150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42798/0 (socket says 172.21.15.67:42798) 2024-03-24T00:42:21.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.294+0000 7f4c31679640 1 -- 172.21.15.67:0/2209842212 learned_addr learned my addr 172.21.15.67:0/2209842212 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:21.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.294+0000 7f4c30e78640 1 --2- 172.21.15.67:0/2209842212 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5635b32e74a0 0x5635b32ec5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:21.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.294+0000 7f4c2bfff640 1 --2- 172.21.15.67:0/2209842212 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5635b32e7880 0x5635b3305850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:21.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.295+0000 7f4c31679640 1 -- 172.21.15.67:0/2209842212 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5635b32e7880 msgr2=0x5635b3305850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:21.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.295+0000 7f4c31679640 1 --2- 172.21.15.67:0/2209842212 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5635b32e7880 0x5635b3305850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.295+0000 7f4c31679640 1 -- 172.21.15.67:0/2209842212 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5635b32e74a0 msgr2=0x5635b32ec5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:21.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.295+0000 7f4c31679640 1 --2- 172.21.15.67:0/2209842212 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5635b32e74a0 0x5635b32ec5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.295+0000 7f4c31679640 1 -- 172.21.15.67:0/2209842212 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5635b30650d0 con 0x5635b32eb700 2024-03-24T00:42:21.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.295+0000 7f4c31679640 1 --2- 172.21.15.67:0/2209842212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5635b32eb700 0x5635b32da150 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f4c24005900 tx=0x7f4c24001650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94842af0902fdee7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:21.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.295+0000 7f4c0b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:42:21.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.295+0000 7f4c0b7fe640 1 -- 172.21.15.67:0/2209842212 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c2401a040 con 0x5635b32eb700 2024-03-24T00:42:21.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.295+0000 7f4c0b7fe640 1 -- 172.21.15.67:0/2209842212 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(62..62 src has 1..62) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4c24007040 con 0x5635b32eb700 2024-03-24T00:42:21.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.295+0000 7f4c35da7740 1 -- 172.21.15.67:0/2209842212 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4c24005520 con 0x5635b32eb700 2024-03-24T00:42:21.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.295+0000 7f4c0b7fe640 1 -- 172.21.15.67:0/2209842212 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4c2402a050 con 0x5635b32eb700 2024-03-24T00:42:21.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.295+0000 7f4c35da7740 10 client.6627 fetch_fsmap learned FSMap version 33 2024-03-24T00:42:21.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.295+0000 7f4c35da7740 1 -- 172.21.15.67:0/2209842212 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x5635b2ef8230 con 0x5635b32eb700 2024-03-24T00:42:21.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.295+0000 7f4c0b7fe640 1 -- 172.21.15.67:0/2209842212 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c24005520 con 0x5635b32eb700 2024-03-24T00:42:21.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.295+0000 7f4c0b7fe640 1 -- 172.21.15.67:0/2209842212 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 33) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f4c2402d020 con 0x5635b32eb700 2024-03-24T00:42:21.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.295+0000 7f4c35da7740 10 client.6627 fetch_fsmap finished waiting for FSMap version 33 2024-03-24T00:42:21.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.295+0000 7f4c35da7740 10 client.6627 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.a 2024-03-24T00:42:21.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.295+0000 7f4c35da7740 20 client.6627 populate_metadata read hostname 'smithi067' 2024-03-24T00:42:21.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.295+0000 7f4c35da7740 1 --2- 172.21.15.67:0/2209842212 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x5635b3318e40 0x5635b3319250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:21.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.295+0000 7f4c35da7740 4 client.6627 mds_command: new command op to 6360 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:42:21.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.295+0000 7f4c35da7740 1 -- 172.21.15.67:0/2209842212 --> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5635b3231710 con 0x5635b3318e40 2024-03-24T00:42:21.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.296+0000 7f4c30e78640 1 --2- 172.21.15.67:0/2209842212 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x5635b3318e40 0x5635b3319250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:21.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.296+0000 7f4c0b7fe640 10 client.6627 ms_handle_connect on v2:172.21.15.67:6834/451962260 2024-03-24T00:42:21.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.296+0000 7f4c30e78640 1 --2- 172.21.15.67:0/2209842212 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x5635b3318e40 0x5635b3319250 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f6bb80a745156e3 server_cookie=128b42289f1c2de1 in_seq=0 out_seq=0 2024-03-24T00:42:21.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.296+0000 7f4c0b7fe640 1 -- 172.21.15.67:0/2209842212 <== mds.0 v2:172.21.15.67:6834/451962260 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5635b3231710 con 0x5635b3318e40 2024-03-24T00:42:21.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.296+0000 7f4c0b7fe640 10 client.6627 handle_command_reply: tid=0 2024-03-24T00:42:21.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.296+0000 7f4c35da7740 1 client.6627 shutdown 2024-03-24T00:42:21.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.297+0000 7f4c35da7740 1 -- 172.21.15.67:0/2209842212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5635b32eb700 msgr2=0x5635b32da150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:21.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.297+0000 7f4c35da7740 1 --2- 172.21.15.67:0/2209842212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5635b32eb700 0x5635b32da150 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f4c24005900 tx=0x7f4c24001650 comp rx=0 tx=0).stop 2024-03-24T00:42:21.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.297+0000 7f4c35da7740 1 -- 172.21.15.67:0/2209842212 shutdown_connections 2024-03-24T00:42:21.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.297+0000 7f4c35da7740 1 --2- 172.21.15.67:0/2209842212 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x5635b3318e40 0x5635b3319250 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.297+0000 7f4c35da7740 1 --2- 172.21.15.67:0/2209842212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5635b32eb700 0x5635b32da150 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.297+0000 7f4c35da7740 1 --2- 172.21.15.67:0/2209842212 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5635b32e7880 0x5635b3305850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.297+0000 7f4c35da7740 1 --2- 172.21.15.67:0/2209842212 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5635b32e74a0 0x5635b32ec5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.297+0000 7f4c0b7fe640 0 client.6627 ms_handle_reset on v2:172.21.15.67:6834/451962260 2024-03-24T00:42:21.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.297+0000 7f4c35da7740 1 -- 172.21.15.67:0/2209842212 >> 172.21.15.67:0/2209842212 conn(0x5635b32ba980 msgr2=0x5635b32c0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:21.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.297+0000 7f4c35da7740 1 -- 172.21.15.67:0/2209842212 shutdown_connections 2024-03-24T00:42:21.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.297+0000 7f4c35da7740 1 -- 172.21.15.67:0/2209842212 wait complete. 2024-03-24T00:42:21.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.297+0000 7f4c35da7740 20 client.6627 trim_cache size 0 max 16384 2024-03-24T00:42:21.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.313+0000 7f4c35da7740 1 Processor -- start 2024-03-24T00:42:21.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.313+0000 7f4c35da7740 1 -- start start 2024-03-24T00:42:21.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.314+0000 7f4c35da7740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5635b32e74a0 0x5635b3308610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:21.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.314+0000 7f4c35da7740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5635b32e7880 0x5635b33063f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:21.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.314+0000 7f4c35da7740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5635b32eb700 0x5635b3306930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:21.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.314+0000 7f4c35da7740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635b32bba80 con 0x5635b32e7880 2024-03-24T00:42:21.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.314+0000 7f4c35da7740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c1fee10 con 0x5635b32eb700 2024-03-24T00:42:21.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.314+0000 7f4c35da7740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c1fdf50 con 0x5635b32e74a0 2024-03-24T00:42:21.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.314+0000 7f4c2bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5635b32e7880 0x5635b33063f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:21.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.314+0000 7f4c2bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5635b32e7880 0x5635b33063f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42802/0 (socket says 172.21.15.67:42802) 2024-03-24T00:42:21.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.314+0000 7f4c2bfff640 1 -- 172.21.15.67:0/567719712 learned_addr learned my addr 172.21.15.67:0/567719712 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:21.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.314+0000 7f4c2bfff640 1 -- 172.21.15.67:0/567719712 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5635b32e74a0 msgr2=0x5635b3308610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:21.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.314+0000 7f4c2bfff640 1 --2- 172.21.15.67:0/567719712 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5635b32e74a0 0x5635b3308610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.314+0000 7f4c2bfff640 1 -- 172.21.15.67:0/567719712 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5635b32eb700 msgr2=0x5635b3306930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:21.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.314+0000 7f4c2bfff640 1 --2- 172.21.15.67:0/567719712 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5635b32eb700 0x5635b3306930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.314+0000 7f4c2bfff640 1 -- 172.21.15.67:0/567719712 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5635b33095f0 con 0x5635b32e7880 2024-03-24T00:42:21.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.314+0000 7f4c2bfff640 1 --2- 172.21.15.67:0/567719712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5635b32e7880 0x5635b33063f0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f4c180aaa20 tx=0x7f4c18009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37824903b706e365 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:21.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.314+0000 7f4c15ffb640 1 -- 172.21.15.67:0/567719712 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c1807aa00 con 0x5635b32e7880 2024-03-24T00:42:21.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.314+0000 7f4c15ffb640 1 -- 172.21.15.67:0/567719712 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c180a9020 con 0x5635b32e7880 2024-03-24T00:42:21.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.314+0000 7f4c15ffb640 1 -- 172.21.15.67:0/567719712 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c1807eb50 con 0x5635b32e7880 2024-03-24T00:42:21.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.315+0000 7f4c35da7740 1 -- 172.21.15.67:0/567719712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5635b32e7880 msgr2=0x5635b33063f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:21.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.315+0000 7f4c35da7740 1 --2- 172.21.15.67:0/567719712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5635b32e7880 0x5635b33063f0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f4c180aaa20 tx=0x7f4c18009190 comp rx=0 tx=0).stop 2024-03-24T00:42:21.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.315+0000 7f4c35da7740 1 -- 172.21.15.67:0/567719712 shutdown_connections 2024-03-24T00:42:21.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.315+0000 7f4c35da7740 1 --2- 172.21.15.67:0/567719712 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5635b32eb700 0x5635b3306930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.315+0000 7f4c35da7740 1 --2- 172.21.15.67:0/567719712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5635b32e7880 0x5635b33063f0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.315+0000 7f4c35da7740 1 --2- 172.21.15.67:0/567719712 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5635b32e74a0 0x5635b3308610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.315+0000 7f4c35da7740 1 -- 172.21.15.67:0/567719712 >> 172.21.15.67:0/567719712 conn(0x5635b32ba980 msgr2=0x5635b32e9d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:21.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.315+0000 7f4c35da7740 1 -- 172.21.15.67:0/567719712 shutdown_connections 2024-03-24T00:42:21.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.315+0000 7f4c35da7740 1 -- 172.21.15.67:0/567719712 wait complete. 2024-03-24T00:42:21.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.315+0000 7f4c35da7740 1 -- start start 2024-03-24T00:42:21.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.316+0000 7f4c35da7740 1 Processor -- start 2024-03-24T00:42:21.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.316+0000 7f4c35da7740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5635b32e74a0 0x5635b3314c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:21.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.316+0000 7f4c35da7740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5635b32e7880 0x5635b3308610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:21.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.316+0000 7f4c35da7740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5635b32eb700 0x5635b3308b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:21.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.316+0000 7f4c35da7740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c1fd950 con 0x5635b32eb700 2024-03-24T00:42:21.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.316+0000 7f4c35da7740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c1fd420 con 0x5635b32e74a0 2024-03-24T00:42:21.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.316+0000 7f4c35da7740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5635b32bba80 con 0x5635b32e7880 2024-03-24T00:42:21.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.317+0000 7f4c35da7740 10 client.0 _ll_register_callbacks cb 0x7f4c24018070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:42:21.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.317+0000 7f4c31679640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5635b32eb700 0x5635b3308b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:21.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.317+0000 7f4c31679640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5635b32eb700 0x5635b3308b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42814/0 (socket says 172.21.15.67:42814) 2024-03-24T00:42:21.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.317+0000 7f4c31679640 1 -- 172.21.15.67:0/2939045143 learned_addr learned my addr 172.21.15.67:0/2939045143 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:21.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.317+0000 7f4c2bfff640 1 --2- 172.21.15.67:0/2939045143 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5635b32e7880 0x5635b3308610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:21.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.317+0000 7f4c30e78640 1 --2- 172.21.15.67:0/2939045143 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5635b32e74a0 0x5635b3314c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:21.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.317+0000 7f4c31679640 1 -- 172.21.15.67:0/2939045143 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5635b32e7880 msgr2=0x5635b3308610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:21.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.317+0000 7f4c31679640 1 --2- 172.21.15.67:0/2939045143 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5635b32e7880 0x5635b3308610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.317+0000 7f4c31679640 1 -- 172.21.15.67:0/2939045143 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5635b32e74a0 msgr2=0x5635b3314c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:21.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.317+0000 7f4c31679640 1 --2- 172.21.15.67:0/2939045143 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5635b32e74a0 0x5635b3314c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.317+0000 7f4c31679640 1 -- 172.21.15.67:0/2939045143 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5635b3315170 con 0x5635b32eb700 2024-03-24T00:42:21.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.317+0000 7f4c31679640 1 --2- 172.21.15.67:0/2939045143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5635b32eb700 0x5635b3308b50 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f4c24003ca0 tx=0x7f4c240017d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=378de9a6ad96bb8f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:21.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.317+0000 7f4c0b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:42:21.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.317+0000 7f4c0b7fe640 1 -- 172.21.15.67:0/2939045143 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c2401a040 con 0x5635b32eb700 2024-03-24T00:42:21.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.317+0000 7f4c0b7fe640 1 -- 172.21.15.67:0/2939045143 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(62..62 src has 1..62) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4c24024a10 con 0x5635b32eb700 2024-03-24T00:42:21.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.317+0000 7f4c35da7740 1 -- 172.21.15.67:0/2939045143 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4c180a9020 con 0x5635b32eb700 2024-03-24T00:42:21.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.317+0000 7f4c0b7fe640 1 -- 172.21.15.67:0/2939045143 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4c24024c10 con 0x5635b32eb700 2024-03-24T00:42:21.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.317+0000 7f4c35da7740 10 client.6633 fetch_fsmap learned FSMap version 33 2024-03-24T00:42:21.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.317+0000 7f4c35da7740 1 -- 172.21.15.67:0/2939045143 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x5635b3375e90 con 0x5635b32eb700 2024-03-24T00:42:21.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.318+0000 7f4c0b7fe640 1 -- 172.21.15.67:0/2939045143 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c180a9020 con 0x5635b32eb700 2024-03-24T00:42:21.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.318+0000 7f4c0b7fe640 1 -- 172.21.15.67:0/2939045143 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 33) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f4c2402d070 con 0x5635b32eb700 2024-03-24T00:42:21.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.318+0000 7f4c35da7740 10 client.6633 fetch_fsmap finished waiting for FSMap version 33 2024-03-24T00:42:21.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.318+0000 7f4c35da7740 10 client.6633 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.a 2024-03-24T00:42:21.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.318+0000 7f4c35da7740 20 client.6633 populate_metadata read hostname 'smithi067' 2024-03-24T00:42:21.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.318+0000 7f4c35da7740 1 --2- 172.21.15.67:0/2939045143 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x5635b3313890 0x5635b3313ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:21.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.318+0000 7f4c35da7740 4 client.6633 mds_command: new command op to 6360 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive", "force"]}] 2024-03-24T00:42:21.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.318+0000 7f4c35da7740 1 -- 172.21.15.67:0/2939045143 --> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive", "force"]}) v1 -- 0x5635b32efd10 con 0x5635b3313890 2024-03-24T00:42:21.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.318+0000 7f4c30e78640 1 --2- 172.21.15.67:0/2939045143 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x5635b3313890 0x5635b3313ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:21.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.318+0000 7f4c0b7fe640 10 client.6633 ms_handle_connect on v2:172.21.15.67:6834/451962260 2024-03-24T00:42:21.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.318+0000 7f4c30e78640 1 --2- 172.21.15.67:0/2939045143 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x5635b3313890 0x5635b3313ca0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0b05e36b333d145 server_cookie=81d0ae7ecf9f7cc4 in_seq=0 out_seq=0 2024-03-24T00:42:21.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.319+0000 7f4c0b7fe640 1 -- 172.21.15.67:0/2939045143 <== mds.0 v2:172.21.15.67:6834/451962260 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5635b32efd10 con 0x5635b3313890 2024-03-24T00:42:21.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.319+0000 7f4c0b7fe640 10 client.6633 handle_command_reply: tid=0 2024-03-24T00:42:21.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.319+0000 7f4c35da7740 1 client.6633 shutdown 2024-03-24T00:42:21.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.319+0000 7f4c35da7740 1 -- 172.21.15.67:0/2939045143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5635b32eb700 msgr2=0x5635b3308b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:21.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.319+0000 7f4c35da7740 1 --2- 172.21.15.67:0/2939045143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5635b32eb700 0x5635b3308b50 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f4c24003ca0 tx=0x7f4c240017d0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.319+0000 7f4c35da7740 1 -- 172.21.15.67:0/2939045143 shutdown_connections 2024-03-24T00:42:21.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.319+0000 7f4c35da7740 1 --2- 172.21.15.67:0/2939045143 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x5635b3313890 0x5635b3313ca0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.319+0000 7f4c35da7740 1 --2- 172.21.15.67:0/2939045143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5635b32eb700 0x5635b3308b50 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.319+0000 7f4c35da7740 1 --2- 172.21.15.67:0/2939045143 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5635b32e7880 0x5635b3308610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.319+0000 7f4c35da7740 1 --2- 172.21.15.67:0/2939045143 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5635b32e74a0 0x5635b3314c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.319+0000 7f4c35da7740 1 -- 172.21.15.67:0/2939045143 >> 172.21.15.67:0/2939045143 conn(0x5635b32ba980 msgr2=0x5635b32b5760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:21.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.319+0000 7f4c0b7fe640 0 client.6633 ms_handle_reset on v2:172.21.15.67:6834/451962260 2024-03-24T00:42:21.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.319+0000 7f4c35da7740 1 -- 172.21.15.67:0/2939045143 shutdown_connections 2024-03-24T00:42:21.327 INFO:tasks.ceph.mds.a.smithi067.stderr:2024-03-24T00:42:21.319+0000 7f214a7b5640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [2,head] /dir_test/file auth v4 ap=1 DIRTYPARENT s=0 n(v0 rc2024-03-24T00:41:48.869203+0000 1=1+0) (iversion lock) caps={5732=pAsLsXsFs/-@2},l=5732(-1) | ptrwaiter=0 request=0 lock=0 caps=1 dirtyparent=1 dirty=1 authpin=1 scrubqueue=0 0x55fd9feebb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(10)0x10000000001:[<0x10000000000/file v4>,<0x1/dir_test v7>]//[]","error_str":"failed to decode on-disk backtrace (26 bytes)!"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-24T00:42:21.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.319+0000 7f4c35da7740 1 -- 172.21.15.67:0/2939045143 wait complete. 2024-03-24T00:42:21.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.320+0000 7f4c35da7740 20 client.6633 trim_cache size 0 max 16384 2024-03-24T00:42:21.328 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:42:21.328 INFO:teuthology.orchestra.run.smithi067.stdout: "return_code": 0, 2024-03-24T00:42:21.328 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_tag": "86b2c8ff-f197-4ffd-882e-59c3a43a026b", 2024-03-24T00:42:21.328 INFO:teuthology.orchestra.run.smithi067.stdout: "mode": "asynchronous" 2024-03-24T00:42:21.328 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:42:21.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.320+0000 7f4c330dc640 1 -- 172.21.15.67:0/296653048 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4c1006bbd0 msgr2=0x7f4c1006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:21.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.320+0000 7f4c330dc640 1 --2- 172.21.15.67:0/296653048 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4c1006bbd0 0x7f4c1006e090 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f4c20001030 tx=0x7f4c20009040 comp rx=0 tx=0).stop 2024-03-24T00:42:21.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.320+0000 7f4c330dc640 1 -- 172.21.15.67:0/296653048 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c2c0c8c90 msgr2=0x7f4c2c1f2540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:21.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.320+0000 7f4c330dc640 1 --2- 172.21.15.67:0/296653048 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c2c0c8c90 0x7f4c2c1f2540 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f4c18009ac0 tx=0x7f4c18004730 comp rx=0 tx=0).stop 2024-03-24T00:42:21.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.320+0000 7f4c330dc640 1 -- 172.21.15.67:0/296653048 shutdown_connections 2024-03-24T00:42:21.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.320+0000 7f4c330dc640 1 --2- 172.21.15.67:0/296653048 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4c1006bbd0 0x7f4c1006e090 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.320+0000 7f4c330dc640 1 --2- 172.21.15.67:0/296653048 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c2c0b5830 0x7f4c2c0b5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.320+0000 7f4c330dc640 1 --2- 172.21.15.67:0/296653048 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c2c0c8c90 0x7f4c2c1f2540 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.320+0000 7f4c330dc640 1 --2- 172.21.15.67:0/296653048 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4c1c004800 0x7f4c2c1ef000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.320+0000 7f4c330dc640 1 -- 172.21.15.67:0/296653048 >> 172.21.15.67:0/296653048 conn(0x7f4c2c0ce640 msgr2=0x7f4c2c0c7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:21.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.320+0000 7f4c330dc640 1 -- 172.21.15.67:0/296653048 shutdown_connections 2024-03-24T00:42:21.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.320+0000 7f4c330dc640 1 -- 172.21.15.67:0/296653048 wait complete. 2024-03-24T00:42:21.335 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:42:21.335 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.4:0 scrub status 2024-03-24T00:42:21.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.438+0000 7f6b87dd3640 1 Processor -- start 2024-03-24T00:42:21.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.438+0000 7f6b87dd3640 1 -- start start 2024-03-24T00:42:21.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.439+0000 7f6b87dd3640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6b800cc900 0x7f6b800ccce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:21.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.439+0000 7f6b87dd3640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b800d1880 con 0x7f6b800b14f0 2024-03-24T00:42:21.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.439+0000 7f6b87dd3640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b800cd220 con 0x7f6b8006b200 2024-03-24T00:42:21.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.439+0000 7f6b87dd3640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b800cd7e0 con 0x7f6b800cc900 2024-03-24T00:42:21.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.439+0000 7f6b86370640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6b800b14f0 0x7f6b800d1170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33854/0 (socket says 172.21.15.67:33854) 2024-03-24T00:42:21.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.439+0000 7f6b86370640 1 -- 172.21.15.67:0/410230483 learned_addr learned my addr 172.21.15.67:0/410230483 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:42:21.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.439+0000 7f6b85b6f640 1 --2- 172.21.15.67:0/410230483 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6b800cc900 0x7f6b800ccce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:21.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.439+0000 7f6b84b6d640 1 -- 172.21.15.67:0/410230483 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1164070813 0 0) 0x7f6b800d1880 con 0x7f6b800b14f0 2024-03-24T00:42:21.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.439+0000 7f6b84b6d640 1 -- 172.21.15.67:0/410230483 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b6c003520 con 0x7f6b800b14f0 2024-03-24T00:42:21.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.439+0000 7f6b84b6d640 1 -- 172.21.15.67:0/410230483 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1981302712 0 0) 0x7f6b800cd220 con 0x7f6b8006b200 2024-03-24T00:42:21.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.439+0000 7f6b84b6d640 1 -- 172.21.15.67:0/410230483 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b800d1880 con 0x7f6b8006b200 2024-03-24T00:42:21.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.439+0000 7f6b84b6d640 1 -- 172.21.15.67:0/410230483 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3822326650 0 0) 0x7f6b6c003520 con 0x7f6b800b14f0 2024-03-24T00:42:21.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.439+0000 7f6b85b6f640 1 -- 172.21.15.67:0/410230483 >> v1:172.21.15.72:6789/0 conn(0x7f6b8006b200 legacy=0x7f6b800d0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:21.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.439+0000 7f6b85b6f640 1 -- 172.21.15.67:0/410230483 >> v1:172.21.15.67:6789/0 conn(0x7f6b800b14f0 legacy=0x7f6b800d1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:21.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.439+0000 7f6b85b6f640 1 -- 172.21.15.67:0/410230483 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b800cdac0 con 0x7f6b800cc900 2024-03-24T00:42:21.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.440+0000 7f6b85b6f640 1 --2- 172.21.15.67:0/410230483 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6b800cc900 0x7f6b800ccce0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c0097d0 tx=0x7f6b7c02b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb7e75068873889a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:21.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.440+0000 7f6b84b6d640 1 -- 172.21.15.67:0/410230483 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c037070 con 0x7f6b800cc900 2024-03-24T00:42:21.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.440+0000 7f6b84b6d640 1 -- 172.21.15.67:0/410230483 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b7c030510 con 0x7f6b800cc900 2024-03-24T00:42:21.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.440+0000 7f6b84b6d640 1 -- 172.21.15.67:0/410230483 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b780019f0 con 0x7f6b800cc900 2024-03-24T00:42:21.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.441+0000 7f6b87dd3640 1 -- 172.21.15.67:0/410230483 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6b800cc900 msgr2=0x7f6b800ccce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:21.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.441+0000 7f6b87dd3640 1 --2- 172.21.15.67:0/410230483 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6b800cc900 0x7f6b800ccce0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c0097d0 tx=0x7f6b7c02b770 comp rx=0 tx=0).stop 2024-03-24T00:42:21.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.441+0000 7f6b87dd3640 1 -- 172.21.15.67:0/410230483 shutdown_connections 2024-03-24T00:42:21.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.441+0000 7f6b87dd3640 1 --2- 172.21.15.67:0/410230483 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6b800cc900 0x7f6b800ccce0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.441+0000 7f6b87dd3640 1 -- 172.21.15.67:0/410230483 >> 172.21.15.67:0/410230483 conn(0x7f6b80099a30 msgr2=0x7f6b80099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:21.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.441+0000 7f6b87dd3640 1 -- 172.21.15.67:0/410230483 shutdown_connections 2024-03-24T00:42:21.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.441+0000 7f6b87dd3640 1 -- 172.21.15.67:0/410230483 wait complete. 2024-03-24T00:42:21.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.442+0000 7f6b87dd3640 1 Processor -- start 2024-03-24T00:42:21.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.442+0000 7f6b87dd3640 1 -- start start 2024-03-24T00:42:21.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.443+0000 7f6b87dd3640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6b8006b200 0x7f6b800ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:21.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.443+0000 7f6b87dd3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b800b14f0 0x7f6b801420d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:21.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.443+0000 7f6b87dd3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b800cc900 0x7f6b800c9fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:21.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.443+0000 7f6b87dd3640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b800ce360 con 0x7f6b800b14f0 2024-03-24T00:42:21.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.443+0000 7f6b87dd3640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b800ce080 con 0x7f6b800cc900 2024-03-24T00:42:21.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.443+0000 7f6b87dd3640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b800cd4d0 con 0x7f6b8006b200 2024-03-24T00:42:21.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.443+0000 7f6b8536e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b800b14f0 0x7f6b801420d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:21.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.443+0000 7f6b8536e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b800b14f0 0x7f6b801420d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42828/0 (socket says 172.21.15.67:42828) 2024-03-24T00:42:21.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.443+0000 7f6b8536e640 1 -- 172.21.15.67:0/1557754319 learned_addr learned my addr 172.21.15.67:0/1557754319 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:21.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.443+0000 7f6b86370640 1 --2- 172.21.15.67:0/1557754319 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b800cc900 0x7f6b800c9fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:21.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.443+0000 7f6b85b6f640 1 --2- 172.21.15.67:0/1557754319 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6b8006b200 0x7f6b800ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:21.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.443+0000 7f6b8536e640 1 -- 172.21.15.67:0/1557754319 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6b8006b200 msgr2=0x7f6b800ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:21.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.443+0000 7f6b8536e640 1 --2- 172.21.15.67:0/1557754319 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6b8006b200 0x7f6b800ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.443+0000 7f6b8536e640 1 -- 172.21.15.67:0/1557754319 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b800cc900 msgr2=0x7f6b800c9fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:21.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.443+0000 7f6b8536e640 1 --2- 172.21.15.67:0/1557754319 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b800cc900 0x7f6b800c9fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.443+0000 7f6b8536e640 1 -- 172.21.15.67:0/1557754319 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b800ced50 con 0x7f6b800b14f0 2024-03-24T00:42:21.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.443+0000 7f6b8536e640 1 --2- 172.21.15.67:0/1557754319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b800b14f0 0x7f6b801420d0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f6b7000c430 tx=0x7f6b7000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:21.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.444+0000 7f6b767fc640 1 -- 172.21.15.67:0/1557754319 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b70017070 con 0x7f6b800b14f0 2024-03-24T00:42:21.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.444+0000 7f6b767fc640 1 -- 172.21.15.67:0/1557754319 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b700052f0 con 0x7f6b800b14f0 2024-03-24T00:42:21.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.444+0000 7f6b87dd3640 1 -- 172.21.15.67:0/1557754319 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b800cf030 con 0x7f6b800b14f0 2024-03-24T00:42:21.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.444+0000 7f6b767fc640 1 -- 172.21.15.67:0/1557754319 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b70003830 con 0x7f6b800b14f0 2024-03-24T00:42:21.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.444+0000 7f6b87dd3640 1 -- 172.21.15.67:0/1557754319 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b800cf5f0 con 0x7f6b800b14f0 2024-03-24T00:42:21.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.445+0000 7f6b767fc640 1 -- 172.21.15.67:0/1557754319 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6b7001a070 con 0x7f6b800b14f0 2024-03-24T00:42:21.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.446+0000 7f6b767fc640 1 --2- 172.21.15.67:0/1557754319 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6b6406bc00 0x7f6b6406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:21.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.446+0000 7f6b767fc640 1 -- 172.21.15.67:0/1557754319 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6b70089ef0 con 0x7f6b800b14f0 2024-03-24T00:42:21.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.446+0000 7f6b85b6f640 1 --2- 172.21.15.67:0/1557754319 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6b6406bc00 0x7f6b6406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:21.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.447+0000 7f6b8a9b6740 1 Processor -- start 2024-03-24T00:42:21.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.447+0000 7f6b85b6f640 1 --2- 172.21.15.67:0/1557754319 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6b6406bc00 0x7f6b6406e0c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c034000 tx=0x7f6b7c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:21.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.447+0000 7f6b8a9b6740 1 -- start start 2024-03-24T00:42:21.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.447+0000 7f6b8a9b6740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e5a95efe50 0x55e5a95c2df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:21.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.447+0000 7f6b8a9b6740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e5a95f0230 0x55e5a95f2be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:21.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.447+0000 7f6b8a9b6740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e5a95c3330 0x55e5a95f3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:21.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.447+0000 7f6b8a9b6740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5a952d3d0 con 0x55e5a95c3330 2024-03-24T00:42:21.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.447+0000 7f6b8a9b6740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5a94d5ad0 con 0x55e5a95efe50 2024-03-24T00:42:21.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.447+0000 7f6b8a9b6740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e5a9529470 con 0x55e5a95f0230 2024-03-24T00:42:21.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.447+0000 7f6b86370640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e5a95c3330 0x55e5a95f3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:21.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.447+0000 7f6b86370640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e5a95c3330 0x55e5a95f3120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42842/0 (socket says 172.21.15.67:42842) 2024-03-24T00:42:21.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.447+0000 7f6b86370640 1 -- 172.21.15.67:0/1274722947 learned_addr learned my addr 172.21.15.67:0/1274722947 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:21.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.447+0000 7f6b85b6f640 1 --2- 172.21.15.67:0/1274722947 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e5a95efe50 0x55e5a95c2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:21.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.447+0000 7f6b8536e640 1 --2- 172.21.15.67:0/1274722947 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e5a95f0230 0x55e5a95f2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:21.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.448+0000 7f6b86370640 1 -- 172.21.15.67:0/1274722947 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e5a95f0230 msgr2=0x55e5a95f2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:21.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.448+0000 7f6b86370640 1 --2- 172.21.15.67:0/1274722947 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e5a95f0230 0x55e5a95f2be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.448+0000 7f6b86370640 1 -- 172.21.15.67:0/1274722947 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e5a95efe50 msgr2=0x55e5a95c2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:21.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.448+0000 7f6b86370640 1 --2- 172.21.15.67:0/1274722947 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e5a95efe50 0x55e5a95c2df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.448+0000 7f6b86370640 1 -- 172.21.15.67:0/1274722947 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e5a930b7d0 con 0x55e5a95c3330 2024-03-24T00:42:21.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.448+0000 7f6b86370640 1 --2- 172.21.15.67:0/1274722947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e5a95c3330 0x55e5a95f3120 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f6b7800f300 tx=0x7f6b7800f7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95537beb628db94a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:21.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.448+0000 7f6b637fe640 1 -- 172.21.15.67:0/1274722947 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b780072e0 con 0x55e5a95c3330 2024-03-24T00:42:21.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.448+0000 7f6b637fe640 1 -- 172.21.15.67:0/1274722947 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b78018070 con 0x55e5a95c3330 2024-03-24T00:42:21.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.448+0000 7f6b637fe640 1 -- 172.21.15.67:0/1274722947 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b78007680 con 0x55e5a95c3330 2024-03-24T00:42:21.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.449+0000 7f6b8a9b6740 1 -- 172.21.15.67:0/1274722947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e5a95c3330 msgr2=0x55e5a95f3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:21.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.449+0000 7f6b8a9b6740 1 --2- 172.21.15.67:0/1274722947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e5a95c3330 0x55e5a95f3120 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f6b7800f300 tx=0x7f6b7800f7c0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.449+0000 7f6b8a9b6740 1 -- 172.21.15.67:0/1274722947 shutdown_connections 2024-03-24T00:42:21.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.449+0000 7f6b8a9b6740 1 --2- 172.21.15.67:0/1274722947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e5a95c3330 0x55e5a95f3120 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.449+0000 7f6b8a9b6740 1 --2- 172.21.15.67:0/1274722947 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e5a95f0230 0x55e5a95f2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.449+0000 7f6b8a9b6740 1 --2- 172.21.15.67:0/1274722947 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e5a95efe50 0x55e5a95c2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.449+0000 7f6b8a9b6740 1 -- 172.21.15.67:0/1274722947 >> 172.21.15.67:0/1274722947 conn(0x55e5a95c5060 msgr2=0x55e5a960dbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:21.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.449+0000 7f6b8a9b6740 1 -- 172.21.15.67:0/1274722947 shutdown_connections 2024-03-24T00:42:21.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.449+0000 7f6b8a9b6740 1 -- 172.21.15.67:0/1274722947 wait complete. 2024-03-24T00:42:21.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.449+0000 7f6b8a9b6740 1 -- start start 2024-03-24T00:42:21.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.449+0000 7f6b8a9b6740 1 Processor -- start 2024-03-24T00:42:21.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.450+0000 7f6b8a9b6740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e5a95efe50 0x55e5a95f5080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:21.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.450+0000 7f6b8a9b6740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e5a95f0230 0x55e5a95faac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:21.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.450+0000 7f6b8a9b6740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e5a95ddd90 0x55e5a95fb000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:21.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.450+0000 7f6b8a9b6740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5a95c7700 con 0x55e5a95efe50 2024-03-24T00:42:21.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.450+0000 7f6b8a9b6740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5a94e9060 con 0x55e5a95f0230 2024-03-24T00:42:21.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.450+0000 7f6b8a9b6740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e5a95c9f40 con 0x55e5a95ddd90 2024-03-24T00:42:21.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.450+0000 7f6b8a9b6740 10 client.0 _ll_register_callbacks cb 0x55e5a94cb6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:42:21.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.450+0000 7f6b85b6f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e5a95efe50 0x55e5a95f5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:21.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.450+0000 7f6b85b6f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e5a95efe50 0x55e5a95f5080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42852/0 (socket says 172.21.15.67:42852) 2024-03-24T00:42:21.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.450+0000 7f6b85b6f640 1 -- 172.21.15.67:0/450366709 learned_addr learned my addr 172.21.15.67:0/450366709 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:21.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.450+0000 7f6b8536e640 1 --2- 172.21.15.67:0/450366709 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e5a95f0230 0x55e5a95faac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:21.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.450+0000 7f6b86370640 1 --2- 172.21.15.67:0/450366709 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e5a95ddd90 0x55e5a95fb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:21.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.451+0000 7f6b85b6f640 1 -- 172.21.15.67:0/450366709 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e5a95ddd90 msgr2=0x55e5a95fb000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:21.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.451+0000 7f6b85b6f640 1 --2- 172.21.15.67:0/450366709 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e5a95ddd90 0x55e5a95fb000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.451+0000 7f6b85b6f640 1 -- 172.21.15.67:0/450366709 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e5a95f0230 msgr2=0x55e5a95faac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:21.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.451+0000 7f6b85b6f640 1 --2- 172.21.15.67:0/450366709 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e5a95f0230 0x55e5a95faac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.451+0000 7f6b85b6f640 1 -- 172.21.15.67:0/450366709 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e5a92413f0 con 0x55e5a95efe50 2024-03-24T00:42:21.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.451+0000 7f6b85b6f640 1 --2- 172.21.15.67:0/450366709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e5a95efe50 0x55e5a95f5080 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c035040 tx=0x7f6b7c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9a72e8e97291be7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:21.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.451+0000 7f6b60ff9640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:42:21.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.451+0000 7f6b60ff9640 1 -- 172.21.15.67:0/450366709 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c04f020 con 0x55e5a95efe50 2024-03-24T00:42:21.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.451+0000 7f6b60ff9640 1 -- 172.21.15.67:0/450366709 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(62..62 src has 1..62) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6b7c030510 con 0x55e5a95efe50 2024-03-24T00:42:21.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.451+0000 7f6b8a9b6740 1 -- 172.21.15.67:0/450366709 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6b78018070 con 0x55e5a95efe50 2024-03-24T00:42:21.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.451+0000 7f6b60ff9640 1 -- 172.21.15.67:0/450366709 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6b7c059050 con 0x55e5a95efe50 2024-03-24T00:42:21.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.451+0000 7f6b8a9b6740 10 client.6645 fetch_fsmap learned FSMap version 33 2024-03-24T00:42:21.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.451+0000 7f6b8a9b6740 1 -- 172.21.15.67:0/450366709 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x55e5a92010f0 con 0x55e5a95efe50 2024-03-24T00:42:21.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.451+0000 7f6b60ff9640 1 -- 172.21.15.67:0/450366709 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b78018070 con 0x55e5a95efe50 2024-03-24T00:42:21.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.451+0000 7f6b60ff9640 1 -- 172.21.15.67:0/450366709 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 33) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f6b7c05c020 con 0x55e5a95efe50 2024-03-24T00:42:21.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.452+0000 7f6b8a9b6740 10 client.6645 fetch_fsmap finished waiting for FSMap version 33 2024-03-24T00:42:21.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.452+0000 7f6b8a9b6740 10 client.6645 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.a 2024-03-24T00:42:21.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.452+0000 7f6b8a9b6740 20 client.6645 populate_metadata read hostname 'smithi067' 2024-03-24T00:42:21.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.452+0000 7f6b8a9b6740 1 --2- 172.21.15.67:0/450366709 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x55e5a9621580 0x55e5a96219c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:21.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.452+0000 7f6b8a9b6740 4 client.6645 mds_command: new command op to 6360 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:42:21.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.452+0000 7f6b8a9b6740 1 -- 172.21.15.67:0/450366709 --> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e5a953a910 con 0x55e5a9621580 2024-03-24T00:42:21.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.452+0000 7f6b8536e640 1 --2- 172.21.15.67:0/450366709 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x55e5a9621580 0x55e5a96219c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:21.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.452+0000 7f6b8536e640 1 --2- 172.21.15.67:0/450366709 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x55e5a9621580 0x55e5a96219c0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=87ca8fce68269406 server_cookie=60280eb74b1d90c6 in_seq=0 out_seq=0 2024-03-24T00:42:21.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.452+0000 7f6b60ff9640 10 client.6645 ms_handle_connect on v2:172.21.15.67:6834/451962260 2024-03-24T00:42:21.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.453+0000 7f6b60ff9640 1 -- 172.21.15.67:0/450366709 <== mds.0 v2:172.21.15.67:6834/451962260 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e5a953a910 con 0x55e5a9621580 2024-03-24T00:42:21.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.453+0000 7f6b60ff9640 10 client.6645 handle_command_reply: tid=0 2024-03-24T00:42:21.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.453+0000 7f6b8a9b6740 1 client.6645 shutdown 2024-03-24T00:42:21.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.453+0000 7f6b8a9b6740 1 -- 172.21.15.67:0/450366709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e5a95efe50 msgr2=0x55e5a95f5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:21.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.453+0000 7f6b8a9b6740 1 --2- 172.21.15.67:0/450366709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e5a95efe50 0x55e5a95f5080 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c035040 tx=0x7f6b7c004730 comp rx=0 tx=0).stop 2024-03-24T00:42:21.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.453+0000 7f6b8a9b6740 1 -- 172.21.15.67:0/450366709 shutdown_connections 2024-03-24T00:42:21.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.453+0000 7f6b8a9b6740 1 --2- 172.21.15.67:0/450366709 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x55e5a9621580 0x55e5a96219c0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.453+0000 7f6b8a9b6740 1 --2- 172.21.15.67:0/450366709 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e5a95ddd90 0x55e5a95fb000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.453+0000 7f6b8a9b6740 1 --2- 172.21.15.67:0/450366709 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e5a95f0230 0x55e5a95faac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.453+0000 7f6b60ff9640 0 client.6645 ms_handle_reset on v2:172.21.15.67:6834/451962260 2024-03-24T00:42:21.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.453+0000 7f6b8a9b6740 1 --2- 172.21.15.67:0/450366709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e5a95efe50 0x55e5a95f5080 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.453+0000 7f6b8a9b6740 1 -- 172.21.15.67:0/450366709 >> 172.21.15.67:0/450366709 conn(0x55e5a95c3330 msgr2=0x55e5a95c6d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:21.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.453+0000 7f6b8a9b6740 1 -- 172.21.15.67:0/450366709 shutdown_connections 2024-03-24T00:42:21.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.453+0000 7f6b8a9b6740 1 -- 172.21.15.67:0/450366709 wait complete. 2024-03-24T00:42:21.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.453+0000 7f6b8a9b6740 20 client.6645 trim_cache size 0 max 16384 2024-03-24T00:42:21.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.469+0000 7f6b8a9b6740 1 Processor -- start 2024-03-24T00:42:21.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.470+0000 7f6b8a9b6740 1 -- start start 2024-03-24T00:42:21.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.470+0000 7f6b8a9b6740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e5a95ddd90 0x55e5a95ef240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:21.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.470+0000 7f6b8a9b6740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e5a95efe50 0x55e5a95f5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:21.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.470+0000 7f6b8a9b6740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e5a95f0230 0x55e5a95c7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:21.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.470+0000 7f6b8a9b6740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b80200860 con 0x55e5a95ddd90 2024-03-24T00:42:21.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.470+0000 7f6b8a9b6740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b801ffa20 con 0x55e5a95efe50 2024-03-24T00:42:21.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.470+0000 7f6b8a9b6740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b800c67a0 con 0x55e5a95f0230 2024-03-24T00:42:21.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.470+0000 7f6b85b6f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e5a95ddd90 0x55e5a95ef240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:21.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.470+0000 7f6b85b6f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e5a95ddd90 0x55e5a95ef240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42860/0 (socket says 172.21.15.67:42860) 2024-03-24T00:42:21.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.470+0000 7f6b85b6f640 1 -- 172.21.15.67:0/3755199714 learned_addr learned my addr 172.21.15.67:0/3755199714 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:21.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.471+0000 7f6b85b6f640 1 -- 172.21.15.67:0/3755199714 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e5a95f0230 msgr2=0x55e5a95c7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:21.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.471+0000 7f6b85b6f640 1 --2- 172.21.15.67:0/3755199714 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e5a95f0230 0x55e5a95c7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.471+0000 7f6b85b6f640 1 -- 172.21.15.67:0/3755199714 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e5a95efe50 msgr2=0x55e5a95f5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:21.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.471+0000 7f6b85b6f640 1 --2- 172.21.15.67:0/3755199714 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e5a95efe50 0x55e5a95f5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.471+0000 7f6b85b6f640 1 -- 172.21.15.67:0/3755199714 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e5a95c9c00 con 0x55e5a95ddd90 2024-03-24T00:42:21.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.471+0000 7f6b85b6f640 1 --2- 172.21.15.67:0/3755199714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e5a95ddd90 0x55e5a95ef240 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c030720 tx=0x7f6b7c049000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ad9cb70d6115783 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:21.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.471+0000 7f6b62ffd640 1 -- 172.21.15.67:0/3755199714 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c04f040 con 0x55e5a95ddd90 2024-03-24T00:42:21.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.471+0000 7f6b62ffd640 1 -- 172.21.15.67:0/3755199714 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b7c030c30 con 0x55e5a95ddd90 2024-03-24T00:42:21.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.471+0000 7f6b62ffd640 1 -- 172.21.15.67:0/3755199714 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c037040 con 0x55e5a95ddd90 2024-03-24T00:42:21.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.471+0000 7f6b8a9b6740 1 -- 172.21.15.67:0/3755199714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e5a95ddd90 msgr2=0x55e5a95ef240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:21.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.471+0000 7f6b8a9b6740 1 --2- 172.21.15.67:0/3755199714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e5a95ddd90 0x55e5a95ef240 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c030720 tx=0x7f6b7c049000 comp rx=0 tx=0).stop 2024-03-24T00:42:21.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.471+0000 7f6b8a9b6740 1 -- 172.21.15.67:0/3755199714 shutdown_connections 2024-03-24T00:42:21.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.471+0000 7f6b8a9b6740 1 --2- 172.21.15.67:0/3755199714 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e5a95f0230 0x55e5a95c7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.471+0000 7f6b8a9b6740 1 --2- 172.21.15.67:0/3755199714 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e5a95efe50 0x55e5a95f5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.471+0000 7f6b8a9b6740 1 --2- 172.21.15.67:0/3755199714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e5a95ddd90 0x55e5a95ef240 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.471+0000 7f6b8a9b6740 1 -- 172.21.15.67:0/3755199714 >> 172.21.15.67:0/3755199714 conn(0x55e5a95c3330 msgr2=0x55e5a95f9520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:21.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.472+0000 7f6b8a9b6740 1 -- 172.21.15.67:0/3755199714 shutdown_connections 2024-03-24T00:42:21.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.472+0000 7f6b8a9b6740 1 -- 172.21.15.67:0/3755199714 wait complete. 2024-03-24T00:42:21.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.472+0000 7f6b8a9b6740 1 -- start start 2024-03-24T00:42:21.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.472+0000 7f6b8a9b6740 1 Processor -- start 2024-03-24T00:42:21.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.473+0000 7f6b8a9b6740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e5a95ddd90 0x55e5a961cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:21.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.473+0000 7f6b8a9b6740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e5a95efe50 0x55e5a965fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:21.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.473+0000 7f6b8a9b6740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e5a95f0230 0x55e5a960cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:21.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.473+0000 7f6b8a9b6740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5a95f5760 con 0x55e5a95efe50 2024-03-24T00:42:21.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.473+0000 7f6b8a9b6740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b801ff4a0 con 0x55e5a95ddd90 2024-03-24T00:42:21.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.473+0000 7f6b8a9b6740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e5a9610db0 con 0x55e5a95f0230 2024-03-24T00:42:21.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.473+0000 7f6b8a9b6740 10 client.0 _ll_register_callbacks cb 0x7f6b6c003880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:42:21.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.473+0000 7f6b8536e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e5a95efe50 0x55e5a965fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:21.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.473+0000 7f6b8536e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e5a95efe50 0x55e5a965fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42868/0 (socket says 172.21.15.67:42868) 2024-03-24T00:42:21.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.473+0000 7f6b8536e640 1 -- 172.21.15.67:0/1533263550 learned_addr learned my addr 172.21.15.67:0/1533263550 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:21.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.473+0000 7f6b86370640 1 --2- 172.21.15.67:0/1533263550 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e5a95f0230 0x55e5a960cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:21.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.473+0000 7f6b85b6f640 1 --2- 172.21.15.67:0/1533263550 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e5a95ddd90 0x55e5a961cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:21.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.473+0000 7f6b8536e640 1 -- 172.21.15.67:0/1533263550 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e5a95f0230 msgr2=0x55e5a960cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:21.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.473+0000 7f6b8536e640 1 --2- 172.21.15.67:0/1533263550 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e5a95f0230 0x55e5a960cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.473+0000 7f6b8536e640 1 -- 172.21.15.67:0/1533263550 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e5a95ddd90 msgr2=0x55e5a961cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:21.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.473+0000 7f6b8536e640 1 --2- 172.21.15.67:0/1533263550 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e5a95ddd90 0x55e5a961cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.473+0000 7f6b8536e640 1 -- 172.21.15.67:0/1533263550 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e5a960db80 con 0x55e5a95efe50 2024-03-24T00:42:21.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.474+0000 7f6b8536e640 1 --2- 172.21.15.67:0/1533263550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e5a95efe50 0x55e5a965fe80 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f6b7000cdf0 tx=0x7f6b7000cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4960c51bbad75e50 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:21.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.474+0000 7f6b60ff9640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:42:21.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.474+0000 7f6b60ff9640 1 -- 172.21.15.67:0/1533263550 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b700576e0 con 0x55e5a95efe50 2024-03-24T00:42:21.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.474+0000 7f6b60ff9640 1 -- 172.21.15.67:0/1533263550 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(62..62 src has 1..62) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6b70057880 con 0x55e5a95efe50 2024-03-24T00:42:21.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.474+0000 7f6b8a9b6740 1 -- 172.21.15.67:0/1533263550 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6b7c030c30 con 0x55e5a95efe50 2024-03-24T00:42:21.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.474+0000 7f6b60ff9640 1 -- 172.21.15.67:0/1533263550 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6b70077050 con 0x55e5a95efe50 2024-03-24T00:42:21.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.474+0000 7f6b8a9b6740 10 client.6651 fetch_fsmap learned FSMap version 33 2024-03-24T00:42:21.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.474+0000 7f6b8a9b6740 1 -- 172.21.15.67:0/1533263550 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x55e5a9663190 con 0x55e5a95efe50 2024-03-24T00:42:21.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.474+0000 7f6b60ff9640 1 -- 172.21.15.67:0/1533263550 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b7c030c30 con 0x55e5a95efe50 2024-03-24T00:42:21.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.474+0000 7f6b60ff9640 1 -- 172.21.15.67:0/1533263550 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 33) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f6b7005e2c0 con 0x55e5a95efe50 2024-03-24T00:42:21.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.474+0000 7f6b8a9b6740 10 client.6651 fetch_fsmap finished waiting for FSMap version 33 2024-03-24T00:42:21.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.474+0000 7f6b8a9b6740 10 client.6651 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.a 2024-03-24T00:42:21.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.474+0000 7f6b8a9b6740 20 client.6651 populate_metadata read hostname 'smithi067' 2024-03-24T00:42:21.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.474+0000 7f6b8a9b6740 1 --2- 172.21.15.67:0/1533263550 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x55e5a9621580 0x55e5a9662650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:21.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.474+0000 7f6b8a9b6740 4 client.6651 mds_command: new command op to 6360 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:42:21.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.474+0000 7f6b8a9b6740 1 -- 172.21.15.67:0/1533263550 --> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55e5a95fc540 con 0x55e5a9621580 2024-03-24T00:42:21.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.474+0000 7f6b85b6f640 1 --2- 172.21.15.67:0/1533263550 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x55e5a9621580 0x55e5a9662650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:21.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.475+0000 7f6b60ff9640 10 client.6651 ms_handle_connect on v2:172.21.15.67:6834/451962260 2024-03-24T00:42:21.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.475+0000 7f6b85b6f640 1 --2- 172.21.15.67:0/1533263550 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x55e5a9621580 0x55e5a9662650 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=379c3b28056f4de7 server_cookie=b1ba3665a9afc2d in_seq=0 out_seq=0 2024-03-24T00:42:21.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.475+0000 7f6b60ff9640 1 -- 172.21.15.67:0/1533263550 <== mds.0 v2:172.21.15.67:6834/451962260 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+266 (crc 0 0 0) 0x55e5a95fc540 con 0x55e5a9621580 2024-03-24T00:42:21.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.475+0000 7f6b60ff9640 10 client.6651 handle_command_reply: tid=0 2024-03-24T00:42:21.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.475+0000 7f6b8a9b6740 1 client.6651 shutdown 2024-03-24T00:42:21.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.475+0000 7f6b8a9b6740 1 -- 172.21.15.67:0/1533263550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e5a95efe50 msgr2=0x55e5a965fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:21.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.475+0000 7f6b8a9b6740 1 --2- 172.21.15.67:0/1533263550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e5a95efe50 0x55e5a965fe80 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f6b7000cdf0 tx=0x7f6b7000cf20 comp rx=0 tx=0).stop 2024-03-24T00:42:21.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.475+0000 7f6b8a9b6740 1 -- 172.21.15.67:0/1533263550 shutdown_connections 2024-03-24T00:42:21.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.475+0000 7f6b8a9b6740 1 --2- 172.21.15.67:0/1533263550 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x55e5a9621580 0x55e5a9662650 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.475+0000 7f6b8a9b6740 1 --2- 172.21.15.67:0/1533263550 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e5a95f0230 0x55e5a960cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.475+0000 7f6b8a9b6740 1 --2- 172.21.15.67:0/1533263550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e5a95efe50 0x55e5a965fe80 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.475+0000 7f6b60ff9640 0 client.6651 ms_handle_reset on v2:172.21.15.67:6834/451962260 2024-03-24T00:42:21.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.475+0000 7f6b8a9b6740 1 --2- 172.21.15.67:0/1533263550 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e5a95ddd90 0x55e5a961cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.475+0000 7f6b8a9b6740 1 -- 172.21.15.67:0/1533263550 >> 172.21.15.67:0/1533263550 conn(0x55e5a95c3330 msgr2=0x55e5a95f9520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:21.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.475+0000 7f6b8a9b6740 1 -- 172.21.15.67:0/1533263550 shutdown_connections 2024-03-24T00:42:21.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.475+0000 7f6b8a9b6740 1 -- 172.21.15.67:0/1533263550 wait complete. 2024-03-24T00:42:21.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.475+0000 7f6b8a9b6740 20 client.6651 trim_cache size 0 max 16384 2024-03-24T00:42:21.482 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:42:21.482 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T00:42:21.482 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": { 2024-03-24T00:42:21.482 INFO:teuthology.orchestra.run.smithi067.stdout: "86b2c8ff-f197-4ffd-882e-59c3a43a026b": { 2024-03-24T00:42:21.482 INFO:teuthology.orchestra.run.smithi067.stdout: "path": "/", 2024-03-24T00:42:21.483 INFO:teuthology.orchestra.run.smithi067.stdout: "tag": "86b2c8ff-f197-4ffd-882e-59c3a43a026b", 2024-03-24T00:42:21.483 INFO:teuthology.orchestra.run.smithi067.stdout: "options": "recursive,repair,force" 2024-03-24T00:42:21.483 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:42:21.483 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:42:21.483 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:42:21.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.476+0000 7f6b87dd3640 1 -- 172.21.15.67:0/1557754319 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6b6406bc00 msgr2=0x7f6b6406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:21.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.476+0000 7f6b87dd3640 1 --2- 172.21.15.67:0/1557754319 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6b6406bc00 0x7f6b6406e0c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c034000 tx=0x7f6b7c008030 comp rx=0 tx=0).stop 2024-03-24T00:42:21.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.476+0000 7f6b87dd3640 1 -- 172.21.15.67:0/1557754319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b800b14f0 msgr2=0x7f6b801420d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:21.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.476+0000 7f6b87dd3640 1 --2- 172.21.15.67:0/1557754319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b800b14f0 0x7f6b801420d0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f6b7000c430 tx=0x7f6b7000c8f0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.476+0000 7f6b87dd3640 1 -- 172.21.15.67:0/1557754319 shutdown_connections 2024-03-24T00:42:21.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.476+0000 7f6b87dd3640 1 --2- 172.21.15.67:0/1557754319 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6b6406bc00 0x7f6b6406e0c0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.476+0000 7f6b87dd3640 1 --2- 172.21.15.67:0/1557754319 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b800cc900 0x7f6b800c9fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.476+0000 7f6b87dd3640 1 --2- 172.21.15.67:0/1557754319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b800b14f0 0x7f6b801420d0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.476+0000 7f6b87dd3640 1 --2- 172.21.15.67:0/1557754319 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6b8006b200 0x7f6b800ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:21.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.476+0000 7f6b87dd3640 1 -- 172.21.15.67:0/1557754319 >> 172.21.15.67:0/1557754319 conn(0x7f6b80099a30 msgr2=0x7f6b80142940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:21.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.476+0000 7f6b87dd3640 1 -- 172.21.15.67:0/1557754319 shutdown_connections 2024-03-24T00:42:21.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:21.476+0000 7f6b87dd3640 1 -- 172.21.15.67:0/1557754319 wait complete. 2024-03-24T00:42:21.487 INFO:tasks.cephfs.filesystem:scrub status for tag:86b2c8ff-f197-4ffd-882e-59c3a43a026b - {'path': '/', 'tag': '86b2c8ff-f197-4ffd-882e-59c3a43a026b', 'options': 'recursive,repair,force'} 2024-03-24T00:42:46.321 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:42:46.324 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:42:51.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:42:51.489 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.4:0 scrub status 2024-03-24T00:42:51.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.591+0000 7fdf6a08a640 1 Processor -- start 2024-03-24T00:42:51.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.591+0000 7fdf6a08a640 1 -- start start 2024-03-24T00:42:51.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.592+0000 7fdf6a08a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fdf64144670 0x7fdf640b3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:51.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.592+0000 7fdf6a08a640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf640b2ab0 con 0x7fdf6406ec60 2024-03-24T00:42:51.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.592+0000 7fdf6a08a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf640b2d90 con 0x7fdf640b4a00 2024-03-24T00:42:51.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.592+0000 7fdf6a08a640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf640b3070 con 0x7fdf64144670 2024-03-24T00:42:51.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.592+0000 7fdf63fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fdf6406ec60 0x7fdf6406f080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53944/0 (socket says 172.21.15.67:53944) 2024-03-24T00:42:51.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.592+0000 7fdf63fff640 1 -- 172.21.15.67:0/3502551403 learned_addr learned my addr 172.21.15.67:0/3502551403 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:42:51.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.592+0000 7fdf637fe640 1 --2- 172.21.15.67:0/3502551403 >> v2:172.21.15.72:3300/0 conn(0x7fdf64144670 0x7fdf640b3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:51.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.593+0000 7fdf627fc640 1 -- 172.21.15.67:0/3502551403 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 135857299 0 0) 0x7fdf640b2ab0 con 0x7fdf6406ec60 2024-03-24T00:42:51.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.593+0000 7fdf627fc640 1 -- 172.21.15.67:0/3502551403 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf50005730 con 0x7fdf6406ec60 2024-03-24T00:42:51.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.593+0000 7fdf627fc640 1 -- 172.21.15.67:0/3502551403 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3002755214 0 0) 0x7fdf50005730 con 0x7fdf6406ec60 2024-03-24T00:42:51.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.593+0000 7fdf627fc640 1 -- 172.21.15.67:0/3502551403 >> v2:172.21.15.72:3300/0 conn(0x7fdf64144670 msgr2=0x7fdf640b3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:51.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.593+0000 7fdf627fc640 1 --2- 172.21.15.67:0/3502551403 >> v2:172.21.15.72:3300/0 conn(0x7fdf64144670 0x7fdf640b3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:51.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.593+0000 7fdf627fc640 1 -- 172.21.15.67:0/3502551403 >> v1:172.21.15.72:6789/0 conn(0x7fdf640b4a00 legacy=0x7fdf64072500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:51.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.593+0000 7fdf627fc640 1 -- 172.21.15.67:0/3502551403 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf640b3350 con 0x7fdf6406ec60 2024-03-24T00:42:51.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.593+0000 7fdf627fc640 1 -- 172.21.15.67:0/3502551403 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7fdf5c0019f0 con 0x7fdf6406ec60 2024-03-24T00:42:51.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.593+0000 7fdf627fc640 1 -- 172.21.15.67:0/3502551403 >> v1:172.21.15.67:6789/0 conn(0x7fdf6406ec60 legacy=0x7fdf6406f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:51.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.593+0000 7fdf627fc640 1 --2- 172.21.15.67:0/3502551403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf50005bc0 0x7fdf50005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:51.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.593+0000 7fdf627fc640 1 --2- 172.21.15.67:0/3502551403 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf500065c0 0x7fdf50006a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:51.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.593+0000 7fdf627fc640 1 --2- 172.21.15.67:0/3502551403 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdf50007290 0x7fdf5000dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:51.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.593+0000 7fdf627fc640 1 -- 172.21.15.67:0/3502551403 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf640b3070 con 0x7fdf50005bc0 2024-03-24T00:42:51.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.593+0000 7fdf627fc640 1 -- 172.21.15.67:0/3502551403 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5000e1f0 con 0x7fdf500065c0 2024-03-24T00:42:51.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.593+0000 7fdf627fc640 1 -- 172.21.15.67:0/3502551403 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5000e3f0 con 0x7fdf50007290 2024-03-24T00:42:51.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.593+0000 7fdf637fe640 1 --2- 172.21.15.67:0/3502551403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf50005bc0 0x7fdf50005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:51.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.593+0000 7fdf62ffd640 1 --2- 172.21.15.67:0/3502551403 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf500065c0 0x7fdf50006a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:51.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.593+0000 7fdf63fff640 1 --2- 172.21.15.67:0/3502551403 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdf50007290 0x7fdf5000dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:51.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.593+0000 7fdf637fe640 1 -- 172.21.15.67:0/3502551403 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdf50007290 msgr2=0x7fdf5000dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:51.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.593+0000 7fdf637fe640 1 --2- 172.21.15.67:0/3502551403 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdf50007290 0x7fdf5000dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:51.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.593+0000 7fdf637fe640 1 -- 172.21.15.67:0/3502551403 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf500065c0 msgr2=0x7fdf50006a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:51.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.593+0000 7fdf637fe640 1 --2- 172.21.15.67:0/3502551403 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf500065c0 0x7fdf50006a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:51.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.593+0000 7fdf637fe640 1 -- 172.21.15.67:0/3502551403 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf50005730 con 0x7fdf50005bc0 2024-03-24T00:42:51.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.593+0000 7fdf63fff640 1 -- 172.21.15.67:0/3502551403 reap_dead start 2024-03-24T00:42:51.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.595+0000 7fdf637fe640 1 --2- 172.21.15.67:0/3502551403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf50005bc0 0x7fdf50005fa0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c009ac0 tx=0x7fdf50007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0bf1d2ce3303f17 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:51.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.595+0000 7fdf627fc640 1 -- 172.21.15.67:0/3502551403 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf4c039070 con 0x7fdf50005bc0 2024-03-24T00:42:51.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.595+0000 7fdf627fc640 1 -- 172.21.15.67:0/3502551403 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf4c02c6d0 con 0x7fdf50005bc0 2024-03-24T00:42:51.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.595+0000 7fdf627fc640 1 -- 172.21.15.67:0/3502551403 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf4c009690 con 0x7fdf50005bc0 2024-03-24T00:42:51.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.595+0000 7fdf6a08a640 1 -- 172.21.15.67:0/3502551403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf50005bc0 msgr2=0x7fdf50005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:51.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.595+0000 7fdf6a08a640 1 --2- 172.21.15.67:0/3502551403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf50005bc0 0x7fdf50005fa0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c009ac0 tx=0x7fdf50007a40 comp rx=0 tx=0).stop 2024-03-24T00:42:51.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.595+0000 7fdf6a08a640 1 -- 172.21.15.67:0/3502551403 shutdown_connections 2024-03-24T00:42:51.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.595+0000 7fdf6a08a640 1 --2- 172.21.15.67:0/3502551403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf50005bc0 0x7fdf50005fa0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:51.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.595+0000 7fdf6a08a640 1 -- 172.21.15.67:0/3502551403 >> 172.21.15.67:0/3502551403 conn(0x7fdf640cf9e0 msgr2=0x7fdf640cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:51.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.595+0000 7fdf6a08a640 1 -- 172.21.15.67:0/3502551403 shutdown_connections 2024-03-24T00:42:51.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.595+0000 7fdf6a08a640 1 -- 172.21.15.67:0/3502551403 wait complete. 2024-03-24T00:42:51.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.596+0000 7fdf6a08a640 1 Processor -- start 2024-03-24T00:42:51.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.597+0000 7fdf6a08a640 1 -- start start 2024-03-24T00:42:51.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.597+0000 7fdf6a08a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf50005bc0 0x7fdf6413d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:51.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.597+0000 7fdf6a08a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf641f68f0 0x7fdf641f6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:51.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.597+0000 7fdf6a08a640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdf6413fe60 0x7fdf641402e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:51.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.597+0000 7fdf6a08a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf640b39f0 con 0x7fdf50005bc0 2024-03-24T00:42:51.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.597+0000 7fdf6a08a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf640b3870 con 0x7fdf641f68f0 2024-03-24T00:42:51.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.597+0000 7fdf6a08a640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf640b3b70 con 0x7fdf6413fe60 2024-03-24T00:42:51.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.597+0000 7fdf637fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf50005bc0 0x7fdf6413d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:51.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.597+0000 7fdf637fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf50005bc0 0x7fdf6413d520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47968/0 (socket says 172.21.15.67:47968) 2024-03-24T00:42:51.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.597+0000 7fdf637fe640 1 -- 172.21.15.67:0/1587760173 learned_addr learned my addr 172.21.15.67:0/1587760173 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:51.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.598+0000 7fdf63fff640 1 --2- 172.21.15.67:0/1587760173 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdf6413fe60 0x7fdf641402e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:51.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.598+0000 7fdf62ffd640 1 --2- 172.21.15.67:0/1587760173 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf641f68f0 0x7fdf641f6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:51.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.598+0000 7fdf637fe640 1 -- 172.21.15.67:0/1587760173 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdf6413fe60 msgr2=0x7fdf641402e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:51.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.598+0000 7fdf637fe640 1 --2- 172.21.15.67:0/1587760173 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdf6413fe60 0x7fdf641402e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:51.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.598+0000 7fdf637fe640 1 -- 172.21.15.67:0/1587760173 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf641f68f0 msgr2=0x7fdf641f6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:51.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.598+0000 7fdf637fe640 1 --2- 172.21.15.67:0/1587760173 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf641f68f0 0x7fdf641f6d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:51.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.598+0000 7fdf637fe640 1 -- 172.21.15.67:0/1587760173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf6413de20 con 0x7fdf50005bc0 2024-03-24T00:42:51.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.598+0000 7fdf62ffd640 1 --2- 172.21.15.67:0/1587760173 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf641f68f0 0x7fdf641f6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:42:51.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.598+0000 7fdf637fe640 1 --2- 172.21.15.67:0/1587760173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf50005bc0 0x7fdf6413d520 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c02c340 tx=0x7fdf4c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:51.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.598+0000 7fdf5bfff640 1 -- 172.21.15.67:0/1587760173 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf4c042020 con 0x7fdf50005bc0 2024-03-24T00:42:51.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.598+0000 7fdf5bfff640 1 -- 172.21.15.67:0/1587760173 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf4c03a040 con 0x7fdf50005bc0 2024-03-24T00:42:51.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.598+0000 7fdf6a08a640 1 -- 172.21.15.67:0/1587760173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf6413e100 con 0x7fdf50005bc0 2024-03-24T00:42:51.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.598+0000 7fdf5bfff640 1 -- 172.21.15.67:0/1587760173 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf4c039070 con 0x7fdf50005bc0 2024-03-24T00:42:51.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.598+0000 7fdf6a08a640 1 -- 172.21.15.67:0/1587760173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf6413e6c0 con 0x7fdf50005bc0 2024-03-24T00:42:51.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.600+0000 7fdf5bfff640 1 -- 172.21.15.67:0/1587760173 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fdf4c045020 con 0x7fdf50005bc0 2024-03-24T00:42:51.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.600+0000 7fdf5bfff640 1 --2- 172.21.15.67:0/1587760173 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdf4406bbd0 0x7fdf4406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:51.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.600+0000 7fdf5bfff640 1 -- 172.21.15.67:0/1587760173 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6246+0+0 (secure 0 0 0) 0x7fdf4c0ac310 con 0x7fdf50005bc0 2024-03-24T00:42:51.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.600+0000 7fdf62ffd640 1 --2- 172.21.15.67:0/1587760173 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdf4406bbd0 0x7fdf4406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:51.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.601+0000 7fdf62ffd640 1 --2- 172.21.15.67:0/1587760173 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdf4406bbd0 0x7fdf4406e090 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fdf6413f3c0 tx=0x7fdf54009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:51.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.601+0000 7fdf6cc70740 1 Processor -- start 2024-03-24T00:42:51.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.601+0000 7fdf6cc70740 1 -- start start 2024-03-24T00:42:51.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.602+0000 7fdf6cc70740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bf939c1e50 0x55bf93994df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:51.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.602+0000 7fdf6cc70740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55bf939c2230 0x55bf939c4be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:51.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.602+0000 7fdf6cc70740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bf93995330 0x55bf939c5120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:51.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.602+0000 7fdf6cc70740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf938ff3d0 con 0x55bf939c1e50 2024-03-24T00:42:51.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.602+0000 7fdf6cc70740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf938a7ad0 con 0x55bf93995330 2024-03-24T00:42:51.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.602+0000 7fdf6cc70740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bf938fb470 con 0x55bf939c2230 2024-03-24T00:42:51.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.602+0000 7fdf637fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bf939c1e50 0x55bf93994df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:51.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.602+0000 7fdf637fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bf939c1e50 0x55bf93994df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47970/0 (socket says 172.21.15.67:47970) 2024-03-24T00:42:51.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.602+0000 7fdf637fe640 1 -- 172.21.15.67:0/3363898795 learned_addr learned my addr 172.21.15.67:0/3363898795 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:51.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.602+0000 7fdf62ffd640 1 --2- 172.21.15.67:0/3363898795 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55bf939c2230 0x55bf939c4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:51.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.602+0000 7fdf63fff640 1 --2- 172.21.15.67:0/3363898795 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bf93995330 0x55bf939c5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:51.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.602+0000 7fdf637fe640 1 -- 172.21.15.67:0/3363898795 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55bf939c2230 msgr2=0x55bf939c4be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:51.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.602+0000 7fdf637fe640 1 --2- 172.21.15.67:0/3363898795 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55bf939c2230 0x55bf939c4be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:51.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.602+0000 7fdf637fe640 1 -- 172.21.15.67:0/3363898795 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bf93995330 msgr2=0x55bf939c5120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:51.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.602+0000 7fdf637fe640 1 --2- 172.21.15.67:0/3363898795 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bf93995330 0x55bf939c5120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:51.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.602+0000 7fdf637fe640 1 -- 172.21.15.67:0/3363898795 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bf936dd7d0 con 0x55bf939c1e50 2024-03-24T00:42:51.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.602+0000 7fdf637fe640 1 --2- 172.21.15.67:0/3363898795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bf939c1e50 0x55bf93994df0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c0827e0 tx=0x7fdf4c07a500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69da35094238a46c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:51.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.603+0000 7fdf597fa640 1 -- 172.21.15.67:0/3363898795 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf4c009730 con 0x55bf939c1e50 2024-03-24T00:42:51.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.603+0000 7fdf597fa640 1 -- 172.21.15.67:0/3363898795 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf4c07ad00 con 0x55bf939c1e50 2024-03-24T00:42:51.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.603+0000 7fdf597fa640 1 -- 172.21.15.67:0/3363898795 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf4c07a600 con 0x55bf939c1e50 2024-03-24T00:42:51.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.603+0000 7fdf6cc70740 1 -- 172.21.15.67:0/3363898795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bf939c1e50 msgr2=0x55bf93994df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:51.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.603+0000 7fdf6cc70740 1 --2- 172.21.15.67:0/3363898795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bf939c1e50 0x55bf93994df0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c0827e0 tx=0x7fdf4c07a500 comp rx=0 tx=0).stop 2024-03-24T00:42:51.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.603+0000 7fdf6cc70740 1 -- 172.21.15.67:0/3363898795 shutdown_connections 2024-03-24T00:42:51.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.603+0000 7fdf6cc70740 1 --2- 172.21.15.67:0/3363898795 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bf93995330 0x55bf939c5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:51.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.603+0000 7fdf6cc70740 1 --2- 172.21.15.67:0/3363898795 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55bf939c2230 0x55bf939c4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:51.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.603+0000 7fdf6cc70740 1 --2- 172.21.15.67:0/3363898795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bf939c1e50 0x55bf93994df0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:51.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.603+0000 7fdf6cc70740 1 -- 172.21.15.67:0/3363898795 >> 172.21.15.67:0/3363898795 conn(0x55bf93997060 msgr2=0x55bf939dfbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:51.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.603+0000 7fdf6cc70740 1 -- 172.21.15.67:0/3363898795 shutdown_connections 2024-03-24T00:42:51.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.603+0000 7fdf6cc70740 1 -- 172.21.15.67:0/3363898795 wait complete. 2024-03-24T00:42:51.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.604+0000 7fdf6cc70740 1 -- start start 2024-03-24T00:42:51.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.604+0000 7fdf6cc70740 1 Processor -- start 2024-03-24T00:42:51.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.605+0000 7fdf6cc70740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bf939c1e50 0x55bf939c8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:51.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.605+0000 7fdf6cc70740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bf939c2230 0x55bf939afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:51.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.605+0000 7fdf6cc70740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55bf939affd0 0x55bf939c95c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:51.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.605+0000 7fdf6cc70740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf938fb470 con 0x55bf939c2230 2024-03-24T00:42:51.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.605+0000 7fdf6cc70740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf938bb060 con 0x55bf939c1e50 2024-03-24T00:42:51.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.605+0000 7fdf6cc70740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bf938ff3d0 con 0x55bf939affd0 2024-03-24T00:42:51.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.605+0000 7fdf6cc70740 10 client.0 _ll_register_callbacks cb 0x55bf9389d6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:42:51.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.605+0000 7fdf62ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bf939c2230 0x55bf939afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:51.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.605+0000 7fdf62ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bf939c2230 0x55bf939afa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47972/0 (socket says 172.21.15.67:47972) 2024-03-24T00:42:51.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.605+0000 7fdf62ffd640 1 -- 172.21.15.67:0/3335188374 learned_addr learned my addr 172.21.15.67:0/3335188374 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:51.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.605+0000 7fdf637fe640 1 --2- 172.21.15.67:0/3335188374 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bf939c1e50 0x55bf939c8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:51.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.605+0000 7fdf63fff640 1 --2- 172.21.15.67:0/3335188374 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55bf939affd0 0x55bf939c95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:51.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.605+0000 7fdf62ffd640 1 -- 172.21.15.67:0/3335188374 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55bf939affd0 msgr2=0x55bf939c95c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:51.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.605+0000 7fdf62ffd640 1 --2- 172.21.15.67:0/3335188374 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55bf939affd0 0x55bf939c95c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:51.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.605+0000 7fdf62ffd640 1 -- 172.21.15.67:0/3335188374 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bf939c1e50 msgr2=0x55bf939c8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:51.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.605+0000 7fdf62ffd640 1 --2- 172.21.15.67:0/3335188374 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bf939c1e50 0x55bf939c8120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:51.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.605+0000 7fdf62ffd640 1 -- 172.21.15.67:0/3335188374 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bf936133f0 con 0x55bf939c2230 2024-03-24T00:42:51.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.605+0000 7fdf637fe640 1 --2- 172.21.15.67:0/3335188374 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bf939c1e50 0x55bf939c8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:42:51.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.605+0000 7fdf62ffd640 1 --2- 172.21.15.67:0/3335188374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bf939c2230 0x55bf939afa90 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fdf54015720 tx=0x7fdf540012b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e2bd4f126a48981 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:51.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.605+0000 7fdf427fc640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:42:51.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.606+0000 7fdf427fc640 1 -- 172.21.15.67:0/3335188374 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf540147e0 con 0x55bf939c2230 2024-03-24T00:42:51.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.606+0000 7fdf427fc640 1 -- 172.21.15.67:0/3335188374 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(64..64 src has 1..64) v4 ==== 6246+0+0 (secure 0 0 0) 0x7fdf54014980 con 0x55bf939c2230 2024-03-24T00:42:51.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.606+0000 7fdf6cc70740 1 -- 172.21.15.67:0/3335188374 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdf4c07ad00 con 0x55bf939c2230 2024-03-24T00:42:51.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.606+0000 7fdf427fc640 1 -- 172.21.15.67:0/3335188374 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdf54030050 con 0x55bf939c2230 2024-03-24T00:42:51.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.606+0000 7fdf6cc70740 10 client.6663 fetch_fsmap learned FSMap version 34 2024-03-24T00:42:51.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.606+0000 7fdf6cc70740 1 -- 172.21.15.67:0/3335188374 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=2+}) v3 -- 0x55bf935d30f0 con 0x55bf939c2230 2024-03-24T00:42:51.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.606+0000 7fdf427fc640 1 -- 172.21.15.67:0/3335188374 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf4c07ad00 con 0x55bf939c2230 2024-03-24T00:42:51.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.606+0000 7fdf427fc640 1 -- 172.21.15.67:0/3335188374 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 34) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fdf54033020 con 0x55bf939c2230 2024-03-24T00:42:51.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.606+0000 7fdf6cc70740 10 client.6663 fetch_fsmap finished waiting for FSMap version 34 2024-03-24T00:42:51.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.606+0000 7fdf6cc70740 10 client.6663 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.a 2024-03-24T00:42:51.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.606+0000 7fdf6cc70740 20 client.6663 populate_metadata read hostname 'smithi067' 2024-03-24T00:42:51.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.606+0000 7fdf6cc70740 1 --2- 172.21.15.67:0/3335188374 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x55bf939c9b00 0x55bf939c3850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:51.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.606+0000 7fdf6cc70740 4 client.6663 mds_command: new command op to 6360 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:42:51.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.606+0000 7fdf6cc70740 1 -- 172.21.15.67:0/3335188374 --> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bf9390c910 con 0x55bf939c9b00 2024-03-24T00:42:51.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.606+0000 7fdf637fe640 1 --2- 172.21.15.67:0/3335188374 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x55bf939c9b00 0x55bf939c3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:51.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.607+0000 7fdf427fc640 10 client.6663 ms_handle_connect on v2:172.21.15.67:6834/451962260 2024-03-24T00:42:51.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.607+0000 7fdf637fe640 1 --2- 172.21.15.67:0/3335188374 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x55bf939c9b00 0x55bf939c3850 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca838d2cdb48363a server_cookie=333c615e354838ac in_seq=0 out_seq=0 2024-03-24T00:42:51.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.607+0000 7fdf427fc640 1 -- 172.21.15.67:0/3335188374 <== mds.0 v2:172.21.15.67:6834/451962260 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55bf9390c910 con 0x55bf939c9b00 2024-03-24T00:42:51.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.607+0000 7fdf427fc640 10 client.6663 handle_command_reply: tid=0 2024-03-24T00:42:51.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.607+0000 7fdf6cc70740 1 client.6663 shutdown 2024-03-24T00:42:51.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.607+0000 7fdf6cc70740 1 -- 172.21.15.67:0/3335188374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bf939c2230 msgr2=0x55bf939afa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:51.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.607+0000 7fdf6cc70740 1 --2- 172.21.15.67:0/3335188374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bf939c2230 0x55bf939afa90 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fdf54015720 tx=0x7fdf540012b0 comp rx=0 tx=0).stop 2024-03-24T00:42:51.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.607+0000 7fdf6cc70740 1 -- 172.21.15.67:0/3335188374 shutdown_connections 2024-03-24T00:42:51.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.607+0000 7fdf6cc70740 1 --2- 172.21.15.67:0/3335188374 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x55bf939c9b00 0x55bf939c3850 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:51.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.607+0000 7fdf6cc70740 1 --2- 172.21.15.67:0/3335188374 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55bf939affd0 0x55bf939c95c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:51.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.607+0000 7fdf6cc70740 1 --2- 172.21.15.67:0/3335188374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bf939c2230 0x55bf939afa90 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:51.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.607+0000 7fdf427fc640 0 client.6663 ms_handle_reset on v2:172.21.15.67:6834/451962260 2024-03-24T00:42:51.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.607+0000 7fdf6cc70740 1 --2- 172.21.15.67:0/3335188374 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bf939c1e50 0x55bf939c8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:51.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.607+0000 7fdf6cc70740 1 -- 172.21.15.67:0/3335188374 >> 172.21.15.67:0/3335188374 conn(0x55bf93995330 msgr2=0x55bf93998fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:51.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.607+0000 7fdf6cc70740 1 -- 172.21.15.67:0/3335188374 shutdown_connections 2024-03-24T00:42:51.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.607+0000 7fdf6cc70740 1 -- 172.21.15.67:0/3335188374 wait complete. 2024-03-24T00:42:51.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.607+0000 7fdf6cc70740 20 client.6663 trim_cache size 0 max 16384 2024-03-24T00:42:51.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.623+0000 7fdf6cc70740 1 Processor -- start 2024-03-24T00:42:51.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.623+0000 7fdf6cc70740 1 -- start start 2024-03-24T00:42:51.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.624+0000 7fdf6cc70740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bf939c1e50 0x55bf939c3850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:51.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.624+0000 7fdf6cc70740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55bf939c2230 0x55bf939e2030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:51.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.624+0000 7fdf6cc70740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bf939e2570 0x55bf939bc8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:51.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.624+0000 7fdf6cc70740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf64201150 con 0x55bf939c1e50 2024-03-24T00:42:51.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.624+0000 7fdf6cc70740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf64200210 con 0x55bf939e2570 2024-03-24T00:42:51.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.624+0000 7fdf6cc70740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf64141b50 con 0x55bf939c2230 2024-03-24T00:42:51.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.624+0000 7fdf637fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bf939c1e50 0x55bf939c3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:51.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.624+0000 7fdf637fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bf939c1e50 0x55bf939c3850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47976/0 (socket says 172.21.15.67:47976) 2024-03-24T00:42:51.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.624+0000 7fdf637fe640 1 -- 172.21.15.67:0/4069433619 learned_addr learned my addr 172.21.15.67:0/4069433619 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:51.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.624+0000 7fdf62ffd640 1 --2- 172.21.15.67:0/4069433619 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55bf939c2230 0x55bf939e2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:51.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.625+0000 7fdf63fff640 1 --2- 172.21.15.67:0/4069433619 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bf939e2570 0x55bf939bc8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:51.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.625+0000 7fdf637fe640 1 -- 172.21.15.67:0/4069433619 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55bf939c2230 msgr2=0x55bf939e2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:51.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.625+0000 7fdf637fe640 1 --2- 172.21.15.67:0/4069433619 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55bf939c2230 0x55bf939e2030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:51.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.625+0000 7fdf637fe640 1 -- 172.21.15.67:0/4069433619 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bf939e2570 msgr2=0x55bf939bc8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:51.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.625+0000 7fdf637fe640 1 --2- 172.21.15.67:0/4069433619 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bf939e2570 0x55bf939bc8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:51.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.625+0000 7fdf637fe640 1 -- 172.21.15.67:0/4069433619 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bf939e03a0 con 0x55bf939c1e50 2024-03-24T00:42:51.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.625+0000 7fdf637fe640 1 --2- 172.21.15.67:0/4069433619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bf939c1e50 0x55bf939c3850 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c076f80 tx=0x7fdf4c008000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71b19ba0bffc1c68 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:51.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.625+0000 7fdf58ff9640 1 -- 172.21.15.67:0/4069433619 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf4c087070 con 0x55bf939c1e50 2024-03-24T00:42:51.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.625+0000 7fdf58ff9640 1 -- 172.21.15.67:0/4069433619 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf4c07e750 con 0x55bf939c1e50 2024-03-24T00:42:51.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.625+0000 7fdf58ff9640 1 -- 172.21.15.67:0/4069433619 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf4c082d10 con 0x55bf939c1e50 2024-03-24T00:42:51.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.626+0000 7fdf6cc70740 1 -- 172.21.15.67:0/4069433619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bf939c1e50 msgr2=0x55bf939c3850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:51.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.626+0000 7fdf6cc70740 1 --2- 172.21.15.67:0/4069433619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bf939c1e50 0x55bf939c3850 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c076f80 tx=0x7fdf4c008000 comp rx=0 tx=0).stop 2024-03-24T00:42:51.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.626+0000 7fdf6cc70740 1 -- 172.21.15.67:0/4069433619 shutdown_connections 2024-03-24T00:42:51.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.626+0000 7fdf6cc70740 1 --2- 172.21.15.67:0/4069433619 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bf939e2570 0x55bf939bc8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:51.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.626+0000 7fdf6cc70740 1 --2- 172.21.15.67:0/4069433619 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55bf939c2230 0x55bf939e2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:51.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.626+0000 7fdf6cc70740 1 --2- 172.21.15.67:0/4069433619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bf939c1e50 0x55bf939c3850 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:51.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.626+0000 7fdf6cc70740 1 -- 172.21.15.67:0/4069433619 >> 172.21.15.67:0/4069433619 conn(0x55bf93995330 msgr2=0x55bf939c16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:51.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.626+0000 7fdf6cc70740 1 -- 172.21.15.67:0/4069433619 shutdown_connections 2024-03-24T00:42:51.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.626+0000 7fdf6cc70740 1 -- 172.21.15.67:0/4069433619 wait complete. 2024-03-24T00:42:51.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.627+0000 7fdf6cc70740 1 -- start start 2024-03-24T00:42:51.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.627+0000 7fdf6cc70740 1 Processor -- start 2024-03-24T00:42:51.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.628+0000 7fdf6cc70740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bf939c1e50 0x55bf93a4fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:51.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.628+0000 7fdf6cc70740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55bf939c2230 0x55bf939ae6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:51.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.628+0000 7fdf6cc70740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bf939e2570 0x55bf939aec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:51.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.628+0000 7fdf6cc70740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf938a7ad0 con 0x55bf939c1e50 2024-03-24T00:42:51.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.628+0000 7fdf6cc70740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf641ffc50 con 0x55bf939e2570 2024-03-24T00:42:51.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.628+0000 7fdf6cc70740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bf939af4c0 con 0x55bf939c2230 2024-03-24T00:42:51.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.628+0000 7fdf6cc70740 10 client.0 _ll_register_callbacks cb 0x7fdf540053e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:42:51.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.628+0000 7fdf637fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bf939c1e50 0x55bf93a4fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:51.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.628+0000 7fdf637fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bf939c1e50 0x55bf93a4fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47980/0 (socket says 172.21.15.67:47980) 2024-03-24T00:42:51.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.628+0000 7fdf637fe640 1 -- 172.21.15.67:0/1602258885 learned_addr learned my addr 172.21.15.67:0/1602258885 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:51.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.628+0000 7fdf62ffd640 1 --2- 172.21.15.67:0/1602258885 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55bf939c2230 0x55bf939ae6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:51.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.628+0000 7fdf63fff640 1 --2- 172.21.15.67:0/1602258885 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bf939e2570 0x55bf939aec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:51.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.628+0000 7fdf637fe640 1 -- 172.21.15.67:0/1602258885 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55bf939c2230 msgr2=0x55bf939ae6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:51.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.628+0000 7fdf637fe640 1 --2- 172.21.15.67:0/1602258885 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55bf939c2230 0x55bf939ae6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:51.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.628+0000 7fdf637fe640 1 -- 172.21.15.67:0/1602258885 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bf939e2570 msgr2=0x55bf939aec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:51.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.628+0000 7fdf637fe640 1 --2- 172.21.15.67:0/1602258885 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bf939e2570 0x55bf939aec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:51.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.628+0000 7fdf637fe640 1 -- 172.21.15.67:0/1602258885 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bf939bff90 con 0x55bf939c1e50 2024-03-24T00:42:51.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.629+0000 7fdf637fe640 1 --2- 172.21.15.67:0/1602258885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bf939c1e50 0x55bf93a4fa60 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c006f80 tx=0x7fdf4c009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f204c367e5627e73 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:51.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.629+0000 7fdf427fc640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:42:51.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.629+0000 7fdf427fc640 1 -- 172.21.15.67:0/1602258885 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf4c087070 con 0x55bf939c1e50 2024-03-24T00:42:51.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.629+0000 7fdf427fc640 1 -- 172.21.15.67:0/1602258885 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(64..64 src has 1..64) v4 ==== 6246+0+0 (secure 0 0 0) 0x7fdf4c078900 con 0x55bf939c1e50 2024-03-24T00:42:51.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.629+0000 7fdf6cc70740 1 -- 172.21.15.67:0/1602258885 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdf4c07e750 con 0x55bf939c1e50 2024-03-24T00:42:51.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.629+0000 7fdf427fc640 1 -- 172.21.15.67:0/1602258885 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdf4c07ad00 con 0x55bf939c1e50 2024-03-24T00:42:51.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.629+0000 7fdf6cc70740 10 client.6669 fetch_fsmap learned FSMap version 34 2024-03-24T00:42:51.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.629+0000 7fdf6cc70740 1 -- 172.21.15.67:0/1602258885 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=2+}) v3 -- 0x55bf939bc820 con 0x55bf939c1e50 2024-03-24T00:42:51.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.629+0000 7fdf427fc640 1 -- 172.21.15.67:0/1602258885 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf4c07e750 con 0x55bf939c1e50 2024-03-24T00:42:51.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.629+0000 7fdf427fc640 1 -- 172.21.15.67:0/1602258885 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 34) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fdf4c08a080 con 0x55bf939c1e50 2024-03-24T00:42:51.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.629+0000 7fdf6cc70740 10 client.6669 fetch_fsmap finished waiting for FSMap version 34 2024-03-24T00:42:51.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.629+0000 7fdf6cc70740 10 client.6669 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.a 2024-03-24T00:42:51.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.629+0000 7fdf6cc70740 20 client.6669 populate_metadata read hostname 'smithi067' 2024-03-24T00:42:51.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.629+0000 7fdf6cc70740 1 --2- 172.21.15.67:0/1602258885 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x55bf93993ca0 0x55bf939c5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:51.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.629+0000 7fdf6cc70740 4 client.6669 mds_command: new command op to 6360 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:42:51.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.629+0000 7fdf6cc70740 1 -- 172.21.15.67:0/1602258885 --> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55bf93906c60 con 0x55bf93993ca0 2024-03-24T00:42:51.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.629+0000 7fdf62ffd640 1 --2- 172.21.15.67:0/1602258885 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x55bf93993ca0 0x55bf939c5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:51.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.630+0000 7fdf427fc640 10 client.6669 ms_handle_connect on v2:172.21.15.67:6834/451962260 2024-03-24T00:42:51.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.630+0000 7fdf62ffd640 1 --2- 172.21.15.67:0/1602258885 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x55bf93993ca0 0x55bf939c5ff0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19e95ad466ef5b28 server_cookie=5196027bd8a38621 in_seq=0 out_seq=0 2024-03-24T00:42:51.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.630+0000 7fdf427fc640 1 -- 172.21.15.67:0/1602258885 <== mds.0 v2:172.21.15.67:6834/451962260 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55bf93906c60 con 0x55bf93993ca0 2024-03-24T00:42:51.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.630+0000 7fdf427fc640 10 client.6669 handle_command_reply: tid=0 2024-03-24T00:42:51.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.630+0000 7fdf6cc70740 1 client.6669 shutdown 2024-03-24T00:42:51.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.630+0000 7fdf6cc70740 1 -- 172.21.15.67:0/1602258885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bf939c1e50 msgr2=0x55bf93a4fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:51.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.630+0000 7fdf6cc70740 1 --2- 172.21.15.67:0/1602258885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bf939c1e50 0x55bf93a4fa60 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c006f80 tx=0x7fdf4c009190 comp rx=0 tx=0).stop 2024-03-24T00:42:51.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.630+0000 7fdf6cc70740 1 -- 172.21.15.67:0/1602258885 shutdown_connections 2024-03-24T00:42:51.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.630+0000 7fdf6cc70740 1 --2- 172.21.15.67:0/1602258885 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x55bf93993ca0 0x55bf939c5ff0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:51.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.630+0000 7fdf6cc70740 1 --2- 172.21.15.67:0/1602258885 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bf939e2570 0x55bf939aec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:51.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.630+0000 7fdf6cc70740 1 --2- 172.21.15.67:0/1602258885 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55bf939c2230 0x55bf939ae6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:51.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.630+0000 7fdf427fc640 0 client.6669 ms_handle_reset on v2:172.21.15.67:6834/451962260 2024-03-24T00:42:51.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.630+0000 7fdf6cc70740 1 --2- 172.21.15.67:0/1602258885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bf939c1e50 0x55bf93a4fa60 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:51.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.630+0000 7fdf6cc70740 1 -- 172.21.15.67:0/1602258885 >> 172.21.15.67:0/1602258885 conn(0x55bf93995330 msgr2=0x55bf939c44c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:51.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.630+0000 7fdf6cc70740 1 -- 172.21.15.67:0/1602258885 shutdown_connections 2024-03-24T00:42:51.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.630+0000 7fdf6cc70740 1 -- 172.21.15.67:0/1602258885 wait complete. 2024-03-24T00:42:51.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.630+0000 7fdf6cc70740 20 client.6669 trim_cache size 0 max 16384 2024-03-24T00:42:51.636 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:42:51.636 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "no active scrubs running", 2024-03-24T00:42:51.636 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": {} 2024-03-24T00:42:51.636 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:42:51.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.631+0000 7fdf6a08a640 1 -- 172.21.15.67:0/1587760173 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdf4406bbd0 msgr2=0x7fdf4406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:51.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.631+0000 7fdf6a08a640 1 --2- 172.21.15.67:0/1587760173 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdf4406bbd0 0x7fdf4406e090 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fdf6413f3c0 tx=0x7fdf54009040 comp rx=0 tx=0).stop 2024-03-24T00:42:51.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.631+0000 7fdf6a08a640 1 -- 172.21.15.67:0/1587760173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf50005bc0 msgr2=0x7fdf6413d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:51.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.631+0000 7fdf6a08a640 1 --2- 172.21.15.67:0/1587760173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf50005bc0 0x7fdf6413d520 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c02c340 tx=0x7fdf4c004730 comp rx=0 tx=0).stop 2024-03-24T00:42:51.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.631+0000 7fdf6a08a640 1 -- 172.21.15.67:0/1587760173 shutdown_connections 2024-03-24T00:42:51.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.631+0000 7fdf6a08a640 1 --2- 172.21.15.67:0/1587760173 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdf4406bbd0 0x7fdf4406e090 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:51.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.631+0000 7fdf6a08a640 1 --2- 172.21.15.67:0/1587760173 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdf6413fe60 0x7fdf641402e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:51.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.631+0000 7fdf6a08a640 1 --2- 172.21.15.67:0/1587760173 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf641f68f0 0x7fdf641f6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:51.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.631+0000 7fdf6a08a640 1 --2- 172.21.15.67:0/1587760173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf50005bc0 0x7fdf6413d520 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:51.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.631+0000 7fdf6a08a640 1 -- 172.21.15.67:0/1587760173 >> 172.21.15.67:0/1587760173 conn(0x7fdf640cf9e0 msgr2=0x7fdf640cbc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:51.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.631+0000 7fdf6a08a640 1 -- 172.21.15.67:0/1587760173 shutdown_connections 2024-03-24T00:42:51.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.631+0000 7fdf6a08a640 1 -- 172.21.15.67:0/1587760173 wait complete. 2024-03-24T00:42:51.642 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T00:42:51.642 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:42:51.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.743+0000 7f6d57462640 1 Processor -- start 2024-03-24T00:42:51.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.743+0000 7f6d57462640 1 -- start start 2024-03-24T00:42:51.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.744+0000 7f6d57462640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f6d501445a0 0x7f6d500cee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:51.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.744+0000 7f6d57462640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6d500b48d0 0x7f6d50072550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:51.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.744+0000 7f6d57462640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6d5006ec40 0x7f6d5006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:51.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.744+0000 7f6d57462640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d500ce990 con 0x7f6d5006ec40 2024-03-24T00:42:51.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.744+0000 7f6d57462640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d500b4d30 con 0x7f6d501445a0 2024-03-24T00:42:51.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.744+0000 7f6d57462640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d500cddc0 con 0x7f6d500b48d0 2024-03-24T00:42:51.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.744+0000 7f6d559ff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6d5006ec40 0x7f6d5006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:51.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.744+0000 7f6d559ff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6d5006ec40 0x7f6d5006f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47994/0 (socket says 172.21.15.67:47994) 2024-03-24T00:42:51.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.744+0000 7f6d559ff640 1 -- 172.21.15.67:0/3993208273 learned_addr learned my addr 172.21.15.67:0/3993208273 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:51.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.744+0000 7f6d549fd640 1 --2- 172.21.15.67:0/3993208273 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6d500b48d0 0x7f6d50072550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:51.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.744+0000 7f6d551fe640 1 --2- 172.21.15.67:0/3993208273 >> v2:172.21.15.72:3300/0 conn(0x7f6d501445a0 0x7f6d500cee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:51.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.744+0000 7f6d559ff640 1 -- 172.21.15.67:0/3993208273 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6d500b48d0 msgr2=0x7f6d50072550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:51.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.744+0000 7f6d559ff640 1 --2- 172.21.15.67:0/3993208273 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6d500b48d0 0x7f6d50072550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:51.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.744+0000 7f6d559ff640 1 -- 172.21.15.67:0/3993208273 >> v2:172.21.15.72:3300/0 conn(0x7f6d501445a0 msgr2=0x7f6d500cee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:51.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.744+0000 7f6d559ff640 1 --2- 172.21.15.67:0/3993208273 >> v2:172.21.15.72:3300/0 conn(0x7f6d501445a0 0x7f6d500cee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:51.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.744+0000 7f6d559ff640 1 -- 172.21.15.67:0/3993208273 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d500ce060 con 0x7f6d5006ec40 2024-03-24T00:42:51.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.745+0000 7f6d559ff640 1 --2- 172.21.15.67:0/3993208273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d5006ec40 0x7f6d5006f0d0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f6d5006d3f0 tx=0x7f6d4802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75b65d34586a8faa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:51.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.745+0000 7f6d43fff640 1 -- 172.21.15.67:0/3993208273 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d4802ca70 con 0x7f6d5006ec40 2024-03-24T00:42:51.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.745+0000 7f6d43fff640 1 -- 172.21.15.67:0/3993208273 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d4802cc10 con 0x7f6d5006ec40 2024-03-24T00:42:51.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.745+0000 7f6d43fff640 1 -- 172.21.15.67:0/3993208273 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d480343f0 con 0x7f6d5006ec40 2024-03-24T00:42:51.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.746+0000 7f6d57462640 1 -- 172.21.15.67:0/3993208273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d5006ec40 msgr2=0x7f6d5006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:51.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.746+0000 7f6d57462640 1 --2- 172.21.15.67:0/3993208273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d5006ec40 0x7f6d5006f0d0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f6d5006d3f0 tx=0x7f6d4802bed0 comp rx=0 tx=0).stop 2024-03-24T00:42:51.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.746+0000 7f6d57462640 1 -- 172.21.15.67:0/3993208273 shutdown_connections 2024-03-24T00:42:51.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.746+0000 7f6d57462640 1 --2- 172.21.15.67:0/3993208273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d5006ec40 0x7f6d5006f0d0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:51.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.746+0000 7f6d57462640 1 --2- 172.21.15.67:0/3993208273 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6d500b48d0 0x7f6d50072550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:51.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.746+0000 7f6d57462640 1 --2- 172.21.15.67:0/3993208273 >> v2:172.21.15.72:3300/0 conn(0x7f6d501445a0 0x7f6d500cee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:51.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.746+0000 7f6d57462640 1 -- 172.21.15.67:0/3993208273 >> 172.21.15.67:0/3993208273 conn(0x7f6d500cf9e0 msgr2=0x7f6d500cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:51.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.746+0000 7f6d57462640 1 -- 172.21.15.67:0/3993208273 shutdown_connections 2024-03-24T00:42:51.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.746+0000 7f6d57462640 1 -- 172.21.15.67:0/3993208273 wait complete. 2024-03-24T00:42:51.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.747+0000 7f6d57462640 1 Processor -- start 2024-03-24T00:42:51.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.747+0000 7f6d57462640 1 -- start start 2024-03-24T00:42:51.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.748+0000 7f6d57462640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6d5006ec40 0x7f6d5014fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:51.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.748+0000 7f6d57462640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d500b48d0 0x7f6d50140390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:51.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.748+0000 7f6d57462640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d501445a0 0x7f6d50138200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:51.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.748+0000 7f6d57462640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d500b2a10 con 0x7f6d501445a0 2024-03-24T00:42:51.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.748+0000 7f6d57462640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d500ce620 con 0x7f6d500b48d0 2024-03-24T00:42:51.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.748+0000 7f6d57462640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d500b2b90 con 0x7f6d5006ec40 2024-03-24T00:42:51.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.748+0000 7f6d559ff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d501445a0 0x7f6d50138200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:51.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.748+0000 7f6d559ff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d501445a0 0x7f6d50138200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48000/0 (socket says 172.21.15.67:48000) 2024-03-24T00:42:51.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.748+0000 7f6d559ff640 1 -- 172.21.15.67:0/3628422559 learned_addr learned my addr 172.21.15.67:0/3628422559 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:51.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.748+0000 7f6d551fe640 1 --2- 172.21.15.67:0/3628422559 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6d5006ec40 0x7f6d5014fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:51.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.748+0000 7f6d549fd640 1 --2- 172.21.15.67:0/3628422559 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d500b48d0 0x7f6d50140390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:51.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.749+0000 7f6d559ff640 1 -- 172.21.15.67:0/3628422559 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6d5006ec40 msgr2=0x7f6d5014fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:51.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.749+0000 7f6d559ff640 1 --2- 172.21.15.67:0/3628422559 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6d5006ec40 0x7f6d5014fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:51.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.749+0000 7f6d559ff640 1 -- 172.21.15.67:0/3628422559 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d500b48d0 msgr2=0x7f6d50140390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:51.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.749+0000 7f6d559ff640 1 --2- 172.21.15.67:0/3628422559 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d500b48d0 0x7f6d50140390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:51.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.749+0000 7f6d559ff640 1 -- 172.21.15.67:0/3628422559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d50142210 con 0x7f6d501445a0 2024-03-24T00:42:51.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.749+0000 7f6d559ff640 1 --2- 172.21.15.67:0/3628422559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d501445a0 0x7f6d50138200 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f6d48036040 tx=0x7f6d48004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:51.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.749+0000 7f6d41ffb640 1 -- 172.21.15.67:0/3628422559 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d48043020 con 0x7f6d501445a0 2024-03-24T00:42:51.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.749+0000 7f6d41ffb640 1 -- 172.21.15.67:0/3628422559 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d4803e450 con 0x7f6d501445a0 2024-03-24T00:42:51.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.749+0000 7f6d57462640 1 -- 172.21.15.67:0/3628422559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d501424f0 con 0x7f6d501445a0 2024-03-24T00:42:51.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.749+0000 7f6d41ffb640 1 -- 172.21.15.67:0/3628422559 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d48009510 con 0x7f6d501445a0 2024-03-24T00:42:51.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.749+0000 7f6d57462640 1 -- 172.21.15.67:0/3628422559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d50142ab0 con 0x7f6d501445a0 2024-03-24T00:42:51.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.750+0000 7f6d57462640 1 -- 172.21.15.67:0/3628422559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d5004f400 con 0x7f6d501445a0 2024-03-24T00:42:51.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.751+0000 7f6d41ffb640 1 -- 172.21.15.67:0/3628422559 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6d48005d10 con 0x7f6d501445a0 2024-03-24T00:42:51.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.751+0000 7f6d41ffb640 1 --2- 172.21.15.67:0/3628422559 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6d3406bc00 0x7f6d3406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:51.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.751+0000 7f6d41ffb640 1 -- 172.21.15.67:0/3628422559 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6246+0+0 (secure 0 0 0) 0x7f6d480ad660 con 0x7f6d501445a0 2024-03-24T00:42:51.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.751+0000 7f6d551fe640 1 --2- 172.21.15.67:0/3628422559 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6d3406bc00 0x7f6d3406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:51.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.752+0000 7f6d551fe640 1 --2- 172.21.15.67:0/3628422559 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6d3406bc00 0x7f6d3406e0c0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c0079a0 tx=0x7f6d4c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:51.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:51.756+0000 7f6d41ffb640 1 -- 172.21.15.67:0/3628422559 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6d4807ba00 con 0x7f6d501445a0 2024-03-24T00:42:52.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.003+0000 7f6d57462640 1 -- 172.21.15.67:0/3628422559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d50072c20 con 0x7f6d501445a0 2024-03-24T00:42:52.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.004+0000 7f6d41ffb640 1 -- 172.21.15.67:0/3628422559 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4976 (secure 0 0 0) 0x7f6d4807faa0 con 0x7f6d501445a0 2024-03-24T00:42:52.005 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:42:52.005 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5627,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2941726873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2941726873},{"type":"v1","addr":"172.21.15.72:6837","nonce":2941726873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":6166,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/1957901519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":1957901519},{"type":"v1","addr":"172.21.15.72:6836","nonce":1957901519}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":6357,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/2843754917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2843754917},{"type":"v1","addr":"172.21.15.67:6833","nonce":2843754917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:41:42.242211+0000","modified":"2024-03-24T00:42:22.910300+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6360},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6360":{"gid":6360,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":13,"addr":"172.21.15.67:6835/451962260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":451962260},{"type":"v1","addr":"172.21.15.67:6835","nonce":451962260}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6360,"qdb_cluster":[6360]},"id":4}]} 2024-03-24T00:42:52.005 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 34 2024-03-24T00:42:52.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.007+0000 7f6d57462640 1 -- 172.21.15.67:0/3628422559 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6d3406bc00 msgr2=0x7f6d3406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:52.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.007+0000 7f6d57462640 1 --2- 172.21.15.67:0/3628422559 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6d3406bc00 0x7f6d3406e0c0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c0079a0 tx=0x7f6d4c008040 comp rx=0 tx=0).stop 2024-03-24T00:42:52.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.007+0000 7f6d57462640 1 -- 172.21.15.67:0/3628422559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d501445a0 msgr2=0x7f6d50138200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:52.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.007+0000 7f6d57462640 1 --2- 172.21.15.67:0/3628422559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d501445a0 0x7f6d50138200 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f6d48036040 tx=0x7f6d48004a40 comp rx=0 tx=0).stop 2024-03-24T00:42:52.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.007+0000 7f6d57462640 1 -- 172.21.15.67:0/3628422559 shutdown_connections 2024-03-24T00:42:52.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.007+0000 7f6d57462640 1 --2- 172.21.15.67:0/3628422559 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6d3406bc00 0x7f6d3406e0c0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:52.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.007+0000 7f6d57462640 1 --2- 172.21.15.67:0/3628422559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d501445a0 0x7f6d50138200 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:52.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.007+0000 7f6d57462640 1 --2- 172.21.15.67:0/3628422559 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d500b48d0 0x7f6d50140390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:52.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.007+0000 7f6d57462640 1 --2- 172.21.15.67:0/3628422559 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6d5006ec40 0x7f6d5014fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:52.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.007+0000 7f6d57462640 1 -- 172.21.15.67:0/3628422559 >> 172.21.15.67:0/3628422559 conn(0x7f6d500cf9e0 msgr2=0x7f6d500b3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:52.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.007+0000 7f6d57462640 1 -- 172.21.15.67:0/3628422559 shutdown_connections 2024-03-24T00:42:52.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.007+0000 7f6d57462640 1 -- 172.21.15.67:0/3628422559 wait complete. 2024-03-24T00:42:52.020 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a damage ls --format=json-pretty 2024-03-24T00:42:52.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.126+0000 7fe4d8cde640 1 Processor -- start 2024-03-24T00:42:52.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.126+0000 7fe4d8cde640 1 -- start start 2024-03-24T00:42:52.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.127+0000 7fe4d8cde640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe4d415dea0 0x7fe4d40d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:52.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.127+0000 7fe4d8cde640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe4d40b6170 0x7fe4d40d3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:52.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.127+0000 7fe4d8cde640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4d40d2250 con 0x7fe4d415e280 2024-03-24T00:42:52.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.127+0000 7fe4d8cde640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4d40d2530 con 0x7fe4d40b6170 2024-03-24T00:42:52.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.127+0000 7fe4d8cde640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d40d2810 con 0x7fe4d415dea0 2024-03-24T00:42:52.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.127+0000 7fe4d2d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe4d40b6170 0x7fe4d40d3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:52.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.127+0000 7fe4d2d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe4d40b6170 0x7fe4d40d3910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48014/0 (socket says 172.21.15.67:48014) 2024-03-24T00:42:52.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.127+0000 7fe4d2d9d640 1 -- 172.21.15.67:0/3728306597 learned_addr learned my addr 172.21.15.67:0/3728306597 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:52.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.127+0000 7fe4d2d9d640 1 -- 172.21.15.67:0/3728306597 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe4d415dea0 msgr2=0x7fe4d40d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:52.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.127+0000 7fe4d259c640 1 --2- 172.21.15.67:0/3728306597 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe4d415dea0 0x7fe4d40d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:52.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.127+0000 7fe4d2d9d640 1 --2- 172.21.15.67:0/3728306597 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe4d415dea0 0x7fe4d40d33d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:52.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.127+0000 7fe4d2d9d640 1 -- 172.21.15.67:0/3728306597 >> v1:172.21.15.72:6789/0 conn(0x7fe4d415e280 legacy=0x7fe4d40b5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:52.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.127+0000 7fe4d2d9d640 1 -- 172.21.15.67:0/3728306597 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4d40d2af0 con 0x7fe4d40b6170 2024-03-24T00:42:52.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.128+0000 7fe4d2d9d640 1 --2- 172.21.15.67:0/3728306597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4d40b6170 0x7fe4d40d3910 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc009e50 tx=0x7fe4cc02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f80ac18f344f961 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:52.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.128+0000 7fe4d159a640 1 -- 172.21.15.67:0/3728306597 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4cc02c9e0 con 0x7fe4d40b6170 2024-03-24T00:42:52.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.128+0000 7fe4d159a640 1 -- 172.21.15.67:0/3728306597 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4cc02cb80 con 0x7fe4d40b6170 2024-03-24T00:42:52.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.128+0000 7fe4d159a640 1 -- 172.21.15.67:0/3728306597 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4cc0343f0 con 0x7fe4d40b6170 2024-03-24T00:42:52.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.129+0000 7fe4d8cde640 1 -- 172.21.15.67:0/3728306597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4d40b6170 msgr2=0x7fe4d40d3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:52.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.129+0000 7fe4d8cde640 1 --2- 172.21.15.67:0/3728306597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4d40b6170 0x7fe4d40d3910 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc009e50 tx=0x7fe4cc02be40 comp rx=0 tx=0).stop 2024-03-24T00:42:52.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.129+0000 7fe4d8cde640 1 -- 172.21.15.67:0/3728306597 shutdown_connections 2024-03-24T00:42:52.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.129+0000 7fe4d8cde640 1 --2- 172.21.15.67:0/3728306597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4d40b6170 0x7fe4d40d3910 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:52.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.129+0000 7fe4d8cde640 1 --2- 172.21.15.67:0/3728306597 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe4d415dea0 0x7fe4d40d33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:52.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.129+0000 7fe4d8cde640 1 -- 172.21.15.67:0/3728306597 >> 172.21.15.67:0/3728306597 conn(0x7fe4d4072da0 msgr2=0x7fe4d40d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:52.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.129+0000 7fe4d8cde640 1 -- 172.21.15.67:0/3728306597 shutdown_connections 2024-03-24T00:42:52.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.129+0000 7fe4d8cde640 1 -- 172.21.15.67:0/3728306597 wait complete. 2024-03-24T00:42:52.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.130+0000 7fe4d8cde640 1 Processor -- start 2024-03-24T00:42:52.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.130+0000 7fe4d8cde640 1 -- start start 2024-03-24T00:42:52.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.131+0000 7fe4d8cde640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe4d40b6170 0x7fe4d41612a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:52.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.131+0000 7fe4d8cde640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4d415dea0 0x7fe4d414c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:52.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.131+0000 7fe4d8cde640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe4d415e280 0x7fe4d4157720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:52.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.131+0000 7fe4d8cde640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d40b7480 con 0x7fe4d415dea0 2024-03-24T00:42:52.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.131+0000 7fe4d8cde640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d40b7300 con 0x7fe4d40b6170 2024-03-24T00:42:52.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.131+0000 7fe4d8cde640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d40b7600 con 0x7fe4d415e280 2024-03-24T00:42:52.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.131+0000 7fe4d1d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4d415dea0 0x7fe4d414c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:52.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.131+0000 7fe4d1d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4d415dea0 0x7fe4d414c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48022/0 (socket says 172.21.15.67:48022) 2024-03-24T00:42:52.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.131+0000 7fe4d1d9b640 1 -- 172.21.15.67:0/2531487338 learned_addr learned my addr 172.21.15.67:0/2531487338 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:52.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.131+0000 7fe4d259c640 1 --2- 172.21.15.67:0/2531487338 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe4d40b6170 0x7fe4d41612a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:52.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.131+0000 7fe4d2d9d640 1 --2- 172.21.15.67:0/2531487338 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe4d415e280 0x7fe4d4157720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:52.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.131+0000 7fe4d1d9b640 1 -- 172.21.15.67:0/2531487338 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe4d415e280 msgr2=0x7fe4d4157720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:52.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.131+0000 7fe4d1d9b640 1 --2- 172.21.15.67:0/2531487338 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe4d415e280 0x7fe4d4157720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:52.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.131+0000 7fe4d1d9b640 1 -- 172.21.15.67:0/2531487338 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe4d40b6170 msgr2=0x7fe4d41612a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:52.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.131+0000 7fe4d1d9b640 1 --2- 172.21.15.67:0/2531487338 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe4d40b6170 0x7fe4d41612a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:52.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.131+0000 7fe4d1d9b640 1 -- 172.21.15.67:0/2531487338 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4d41566a0 con 0x7fe4d415dea0 2024-03-24T00:42:52.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.131+0000 7fe4d259c640 1 --2- 172.21.15.67:0/2531487338 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe4d40b6170 0x7fe4d41612a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:42:52.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.131+0000 7fe4d2d9d640 1 --2- 172.21.15.67:0/2531487338 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe4d415e280 0x7fe4d4157720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:42:52.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.132+0000 7fe4d1d9b640 1 --2- 172.21.15.67:0/2531487338 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4d415dea0 0x7fe4d414c550 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fe4c800b3f0 tx=0x7fe4c800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:52.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.132+0000 7fe4c2ffd640 1 -- 172.21.15.67:0/2531487338 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c8015020 con 0x7fe4d415dea0 2024-03-24T00:42:52.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.132+0000 7fe4c2ffd640 1 -- 172.21.15.67:0/2531487338 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4c8011830 con 0x7fe4d415dea0 2024-03-24T00:42:52.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.132+0000 7fe4d8cde640 1 -- 172.21.15.67:0/2531487338 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4d4156980 con 0x7fe4d415dea0 2024-03-24T00:42:52.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.132+0000 7fe4c2ffd640 1 -- 172.21.15.67:0/2531487338 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c8011c20 con 0x7fe4d415dea0 2024-03-24T00:42:52.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.132+0000 7fe4d8cde640 1 -- 172.21.15.67:0/2531487338 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4d4156f40 con 0x7fe4d415dea0 2024-03-24T00:42:52.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.134+0000 7fe4c2ffd640 1 -- 172.21.15.67:0/2531487338 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe4c8018020 con 0x7fe4d415dea0 2024-03-24T00:42:52.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.134+0000 7fe4c2ffd640 1 --2- 172.21.15.67:0/2531487338 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe4b406bbd0 0x7fe4b406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:52.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.134+0000 7fe4c2ffd640 1 -- 172.21.15.67:0/2531487338 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6246+0+0 (secure 0 0 0) 0x7fe4c808ac40 con 0x7fe4d415dea0 2024-03-24T00:42:52.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.134+0000 7fe4d259c640 1 --2- 172.21.15.67:0/2531487338 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe4b406bbd0 0x7fe4b406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:52.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.135+0000 7fe4db99c740 1 Processor -- start 2024-03-24T00:42:52.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.135+0000 7fe4d259c640 1 --2- 172.21.15.67:0/2531487338 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe4b406bbd0 0x7fe4b406e090 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fe4c4006fd0 tx=0x7fe4c4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:52.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.135+0000 7fe4db99c740 1 -- start start 2024-03-24T00:42:52.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.136+0000 7fe4db99c740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x557adad850d0 0x557adad58300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:52.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.136+0000 7fe4db99c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557adad854b0 0x557adad87e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:52.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.136+0000 7fe4db99c740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557adad58840 0x557adad883a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:52.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.136+0000 7fe4db99c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x557adacc21f0 con 0x557adad854b0 2024-03-24T00:42:52.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.136+0000 7fe4db99c740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x557adac60a80 con 0x557adad58840 2024-03-24T00:42:52.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.136+0000 7fe4db99c740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x557adacbe290 con 0x557adad850d0 2024-03-24T00:42:52.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.136+0000 7fe4d1d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557adad854b0 0x557adad87e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:52.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.136+0000 7fe4d1d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557adad854b0 0x557adad87e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48032/0 (socket says 172.21.15.67:48032) 2024-03-24T00:42:52.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.136+0000 7fe4d1d9b640 1 -- 172.21.15.67:0/2205891766 learned_addr learned my addr 172.21.15.67:0/2205891766 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:52.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.136+0000 7fe4d259c640 1 --2- 172.21.15.67:0/2205891766 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x557adad850d0 0x557adad58300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:52.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.136+0000 7fe4d2d9d640 1 --2- 172.21.15.67:0/2205891766 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557adad58840 0x557adad883a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:52.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.136+0000 7fe4d1d9b640 1 -- 172.21.15.67:0/2205891766 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x557adad850d0 msgr2=0x557adad58300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:52.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.136+0000 7fe4d1d9b640 1 --2- 172.21.15.67:0/2205891766 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x557adad850d0 0x557adad58300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:52.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.136+0000 7fe4d1d9b640 1 -- 172.21.15.67:0/2205891766 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557adad58840 msgr2=0x557adad883a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:52.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.136+0000 7fe4d1d9b640 1 --2- 172.21.15.67:0/2205891766 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557adad58840 0x557adad883a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:52.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.136+0000 7fe4d1d9b640 1 -- 172.21.15.67:0/2205891766 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557adaaa27c0 con 0x557adad854b0 2024-03-24T00:42:52.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.136+0000 7fe4d1d9b640 1 --2- 172.21.15.67:0/2205891766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557adad854b0 0x557adad87e60 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fe4c8088f50 tx=0x7fe4c80596e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa25858f6233c5b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:52.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.136+0000 7fe4b3fff640 1 -- 172.21.15.67:0/2205891766 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c8056f40 con 0x557adad854b0 2024-03-24T00:42:52.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.136+0000 7fe4b3fff640 1 -- 172.21.15.67:0/2205891766 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4c8087070 con 0x557adad854b0 2024-03-24T00:42:52.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.137+0000 7fe4b3fff640 1 -- 172.21.15.67:0/2205891766 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c80571a0 con 0x557adad854b0 2024-03-24T00:42:52.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.137+0000 7fe4db99c740 1 -- 172.21.15.67:0/2205891766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557adad854b0 msgr2=0x557adad87e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:52.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.137+0000 7fe4db99c740 1 --2- 172.21.15.67:0/2205891766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557adad854b0 0x557adad87e60 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fe4c8088f50 tx=0x7fe4c80596e0 comp rx=0 tx=0).stop 2024-03-24T00:42:52.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.137+0000 7fe4db99c740 1 -- 172.21.15.67:0/2205891766 shutdown_connections 2024-03-24T00:42:52.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.137+0000 7fe4db99c740 1 --2- 172.21.15.67:0/2205891766 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557adad58840 0x557adad883a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:52.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.137+0000 7fe4db99c740 1 --2- 172.21.15.67:0/2205891766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557adad854b0 0x557adad87e60 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:52.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.137+0000 7fe4db99c740 1 --2- 172.21.15.67:0/2205891766 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x557adad850d0 0x557adad58300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:52.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.137+0000 7fe4db99c740 1 -- 172.21.15.67:0/2205891766 >> 172.21.15.67:0/2205891766 conn(0x557adad5a570 msgr2=0x557adada2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:52.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.138+0000 7fe4db99c740 1 -- 172.21.15.67:0/2205891766 shutdown_connections 2024-03-24T00:42:52.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.138+0000 7fe4db99c740 1 -- 172.21.15.67:0/2205891766 wait complete. 2024-03-24T00:42:52.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.138+0000 7fe4db99c740 1 -- start start 2024-03-24T00:42:52.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.138+0000 7fe4db99c740 1 Processor -- start 2024-03-24T00:42:52.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.139+0000 7fe4db99c740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557adad850d0 0x557adad6f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:52.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.139+0000 7fe4db99c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557adad854b0 0x557adad87170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:52.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.139+0000 7fe4db99c740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x557adad876b0 0x557adad7cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:52.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.139+0000 7fe4db99c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x557adad5c880 con 0x557adad854b0 2024-03-24T00:42:52.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.139+0000 7fe4db99c740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x557adac5e410 con 0x557adad850d0 2024-03-24T00:42:52.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.139+0000 7fe4db99c740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x557adad5bc70 con 0x557adad876b0 2024-03-24T00:42:52.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.139+0000 7fe4d1d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557adad854b0 0x557adad87170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:52.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.139+0000 7fe4d1d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557adad854b0 0x557adad87170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48046/0 (socket says 172.21.15.67:48046) 2024-03-24T00:42:52.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.139+0000 7fe4d1d9b640 1 -- 172.21.15.67:0/2714724922 learned_addr learned my addr 172.21.15.67:0/2714724922 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:52.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.139+0000 7fe4d2d9d640 1 --2- 172.21.15.67:0/2714724922 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x557adad876b0 0x557adad7cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:52.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.139+0000 7fe4d259c640 1 --2- 172.21.15.67:0/2714724922 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557adad850d0 0x557adad6f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:52.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.139+0000 7fe4db99c740 10 client.0 _ll_register_callbacks cb 0x557adac86d70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:42:52.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.139+0000 7fe4d1d9b640 1 -- 172.21.15.67:0/2714724922 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x557adad876b0 msgr2=0x557adad7cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:52.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.139+0000 7fe4d1d9b640 1 --2- 172.21.15.67:0/2714724922 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x557adad876b0 0x557adad7cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:52.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.139+0000 7fe4d1d9b640 1 -- 172.21.15.67:0/2714724922 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557adad850d0 msgr2=0x557adad6f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:52.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.139+0000 7fe4d1d9b640 1 --2- 172.21.15.67:0/2714724922 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557adad850d0 0x557adad6f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:52.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.139+0000 7fe4d1d9b640 1 -- 172.21.15.67:0/2714724922 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557adab02f50 con 0x557adad854b0 2024-03-24T00:42:52.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.139+0000 7fe4d259c640 1 --2- 172.21.15.67:0/2714724922 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557adad850d0 0x557adad6f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:42:52.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.140+0000 7fe4d1d9b640 1 --2- 172.21.15.67:0/2714724922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557adad854b0 0x557adad87170 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fe4c800b3c0 tx=0x7fe4c801c070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49c648064a2d3601 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:52.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.140+0000 7fe4b17fa640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:42:52.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.140+0000 7fe4b17fa640 1 -- 172.21.15.67:0/2714724922 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c80571a0 con 0x557adad854b0 2024-03-24T00:42:52.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.140+0000 7fe4b17fa640 1 -- 172.21.15.67:0/2714724922 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(64..64 src has 1..64) v4 ==== 6246+0+0 (secure 0 0 0) 0x7fe4c8055510 con 0x557adad854b0 2024-03-24T00:42:52.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.140+0000 7fe4db99c740 1 -- 172.21.15.67:0/2714724922 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe4c8087070 con 0x557adad854b0 2024-03-24T00:42:52.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.140+0000 7fe4b17fa640 1 -- 172.21.15.67:0/2714724922 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe4c8061120 con 0x557adad854b0 2024-03-24T00:42:52.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.140+0000 7fe4db99c740 10 client.6687 fetch_fsmap learned FSMap version 34 2024-03-24T00:42:52.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.140+0000 7fe4db99c740 1 -- 172.21.15.67:0/2714724922 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=2+}) v3 -- 0x557ada9960b0 con 0x557adad854b0 2024-03-24T00:42:52.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.140+0000 7fe4b17fa640 1 -- 172.21.15.67:0/2714724922 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4c8087070 con 0x557adad854b0 2024-03-24T00:42:52.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.140+0000 7fe4b17fa640 1 -- 172.21.15.67:0/2714724922 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 34) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fe4c80582f0 con 0x557adad854b0 2024-03-24T00:42:52.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.140+0000 7fe4db99c740 10 client.6687 fetch_fsmap finished waiting for FSMap version 34 2024-03-24T00:42:52.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.140+0000 7fe4db99c740 10 client.6687 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-24T00:42:52.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.140+0000 7fe4db99c740 20 client.6687 populate_metadata read hostname 'smithi067' 2024-03-24T00:42:52.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.140+0000 7fe4db99c740 1 --2- 172.21.15.67:0/2714724922 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x557adad7b1e0 0x557adad8a700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:52.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.140+0000 7fe4db99c740 4 client.6687 mds_command: new command op to 6360 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:42:52.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.140+0000 7fe4db99c740 1 -- 172.21.15.67:0/2714724922 --> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557adaccf730 con 0x557adad7b1e0 2024-03-24T00:42:52.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.140+0000 7fe4d259c640 1 --2- 172.21.15.67:0/2714724922 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x557adad7b1e0 0x557adad8a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:52.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.141+0000 7fe4d259c640 1 --2- 172.21.15.67:0/2714724922 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x557adad7b1e0 0x557adad8a700 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e61d66f71033e693 server_cookie=378c594ee045fa1d in_seq=0 out_seq=0 2024-03-24T00:42:52.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.141+0000 7fe4b17fa640 10 client.6687 ms_handle_connect on v2:172.21.15.67:6834/451962260 2024-03-24T00:42:52.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.142+0000 7fe4b17fa640 1 -- 172.21.15.67:0/2714724922 <== mds.0 v2:172.21.15.67:6834/451962260 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557adaccf730 con 0x557adad7b1e0 2024-03-24T00:42:52.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.142+0000 7fe4b17fa640 10 client.6687 handle_command_reply: tid=0 2024-03-24T00:42:52.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.142+0000 7fe4db99c740 1 client.6687 shutdown 2024-03-24T00:42:52.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.142+0000 7fe4db99c740 1 -- 172.21.15.67:0/2714724922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557adad854b0 msgr2=0x557adad87170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:52.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.142+0000 7fe4db99c740 1 --2- 172.21.15.67:0/2714724922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557adad854b0 0x557adad87170 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fe4c800b3c0 tx=0x7fe4c801c070 comp rx=0 tx=0).stop 2024-03-24T00:42:52.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.142+0000 7fe4db99c740 1 -- 172.21.15.67:0/2714724922 shutdown_connections 2024-03-24T00:42:52.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.142+0000 7fe4db99c740 1 --2- 172.21.15.67:0/2714724922 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x557adad7b1e0 0x557adad8a700 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:52.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.142+0000 7fe4db99c740 1 --2- 172.21.15.67:0/2714724922 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x557adad876b0 0x557adad7cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:52.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.142+0000 7fe4db99c740 1 --2- 172.21.15.67:0/2714724922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557adad854b0 0x557adad87170 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:52.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.142+0000 7fe4b17fa640 0 client.6687 ms_handle_reset on v2:172.21.15.67:6834/451962260 2024-03-24T00:42:52.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.142+0000 7fe4db99c740 1 --2- 172.21.15.67:0/2714724922 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557adad850d0 0x557adad6f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:52.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.142+0000 7fe4db99c740 1 -- 172.21.15.67:0/2714724922 >> 172.21.15.67:0/2714724922 conn(0x557adad58840 msgr2=0x557adada6370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:52.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.142+0000 7fe4db99c740 1 -- 172.21.15.67:0/2714724922 shutdown_connections 2024-03-24T00:42:52.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.142+0000 7fe4db99c740 1 -- 172.21.15.67:0/2714724922 wait complete. 2024-03-24T00:42:52.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.142+0000 7fe4db99c740 20 client.6687 trim_cache size 0 max 16384 2024-03-24T00:42:52.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.166+0000 7fe4db99c740 1 Processor -- start 2024-03-24T00:42:52.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.166+0000 7fe4db99c740 1 -- start start 2024-03-24T00:42:52.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.167+0000 7fe4db99c740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557adad850d0 0x557adada5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:52.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.167+0000 7fe4db99c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557adad854b0 0x557adad539d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:52.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.167+0000 7fe4db99c740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x557adad876b0 0x557adae090b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:52.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.167+0000 7fe4db99c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d41ffca0 con 0x557adad854b0 2024-03-24T00:42:52.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.167+0000 7fe4db99c740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d41ff120 con 0x557adad850d0 2024-03-24T00:42:52.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.167+0000 7fe4db99c740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d41ff6a0 con 0x557adad876b0 2024-03-24T00:42:52.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.168+0000 7fe4d1d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557adad854b0 0x557adad539d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:52.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.168+0000 7fe4d1d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557adad854b0 0x557adad539d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48062/0 (socket says 172.21.15.67:48062) 2024-03-24T00:42:52.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.168+0000 7fe4d1d9b640 1 -- 172.21.15.67:0/3871146850 learned_addr learned my addr 172.21.15.67:0/3871146850 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:52.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.168+0000 7fe4d2d9d640 1 --2- 172.21.15.67:0/3871146850 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x557adad876b0 0x557adae090b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:52.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.168+0000 7fe4d259c640 1 --2- 172.21.15.67:0/3871146850 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557adad850d0 0x557adada5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:52.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.168+0000 7fe4d1d9b640 1 -- 172.21.15.67:0/3871146850 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x557adad876b0 msgr2=0x557adae090b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:52.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.168+0000 7fe4d1d9b640 1 --2- 172.21.15.67:0/3871146850 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x557adad876b0 0x557adae090b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:52.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.168+0000 7fe4d1d9b640 1 -- 172.21.15.67:0/3871146850 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557adad850d0 msgr2=0x557adada5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:52.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.168+0000 7fe4d1d9b640 1 --2- 172.21.15.67:0/3871146850 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557adad850d0 0x557adada5960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:52.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.168+0000 7fe4d1d9b640 1 -- 172.21.15.67:0/3871146850 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557adad8a210 con 0x557adad854b0 2024-03-24T00:42:52.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.168+0000 7fe4d1d9b640 1 --2- 172.21.15.67:0/3871146850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557adad854b0 0x557adad539d0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fe4c8061080 tx=0x7fe4c8072040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c9669653b69c6d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:52.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.168+0000 7fe4b37fe640 1 -- 172.21.15.67:0/3871146850 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c8055140 con 0x557adad854b0 2024-03-24T00:42:52.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.168+0000 7fe4b37fe640 1 -- 172.21.15.67:0/3871146850 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4c8087070 con 0x557adad854b0 2024-03-24T00:42:52.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.168+0000 7fe4b37fe640 1 -- 172.21.15.67:0/3871146850 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c80172c0 con 0x557adad854b0 2024-03-24T00:42:52.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.169+0000 7fe4db99c740 1 -- 172.21.15.67:0/3871146850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557adad854b0 msgr2=0x557adad539d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:52.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.169+0000 7fe4db99c740 1 --2- 172.21.15.67:0/3871146850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557adad854b0 0x557adad539d0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fe4c8061080 tx=0x7fe4c8072040 comp rx=0 tx=0).stop 2024-03-24T00:42:52.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.169+0000 7fe4db99c740 1 -- 172.21.15.67:0/3871146850 shutdown_connections 2024-03-24T00:42:52.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.169+0000 7fe4db99c740 1 --2- 172.21.15.67:0/3871146850 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x557adad876b0 0x557adae090b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:52.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.169+0000 7fe4db99c740 1 --2- 172.21.15.67:0/3871146850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557adad854b0 0x557adad539d0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:52.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.169+0000 7fe4db99c740 1 --2- 172.21.15.67:0/3871146850 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557adad850d0 0x557adada5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:52.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.169+0000 7fe4db99c740 1 -- 172.21.15.67:0/3871146850 >> 172.21.15.67:0/3871146850 conn(0x557adad58840 msgr2=0x557adad5f090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:52.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.169+0000 7fe4db99c740 1 -- 172.21.15.67:0/3871146850 shutdown_connections 2024-03-24T00:42:52.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.169+0000 7fe4db99c740 1 -- 172.21.15.67:0/3871146850 wait complete. 2024-03-24T00:42:52.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.170+0000 7fe4db99c740 1 -- start start 2024-03-24T00:42:52.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.170+0000 7fe4db99c740 1 Processor -- start 2024-03-24T00:42:52.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.171+0000 7fe4db99c740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x557adad850d0 0x557adad7fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:52.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.171+0000 7fe4db99c740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557adad854b0 0x557adad8aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:52.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.171+0000 7fe4db99c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557adad876b0 0x557adadf3990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:52.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.171+0000 7fe4db99c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x557adad5c880 con 0x557adad876b0 2024-03-24T00:42:52.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.171+0000 7fe4db99c740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x557adad5bc70 con 0x557adad854b0 2024-03-24T00:42:52.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.171+0000 7fe4db99c740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x557adac5e410 con 0x557adad850d0 2024-03-24T00:42:52.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.171+0000 7fe4db99c740 10 client.0 _ll_register_callbacks cb 0x7fe4d408d140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:42:52.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.171+0000 7fe4d2d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557adad876b0 0x557adadf3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:52.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.172+0000 7fe4d2d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557adad876b0 0x557adadf3990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48070/0 (socket says 172.21.15.67:48070) 2024-03-24T00:42:52.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.172+0000 7fe4d2d9d640 1 -- 172.21.15.67:0/3943030544 learned_addr learned my addr 172.21.15.67:0/3943030544 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:52.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.172+0000 7fe4d259c640 1 --2- 172.21.15.67:0/3943030544 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x557adad850d0 0x557adad7fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:52.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.172+0000 7fe4d1d9b640 1 --2- 172.21.15.67:0/3943030544 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557adad854b0 0x557adad8aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:52.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.172+0000 7fe4d2d9d640 1 -- 172.21.15.67:0/3943030544 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x557adad850d0 msgr2=0x557adad7fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:52.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.172+0000 7fe4d2d9d640 1 --2- 172.21.15.67:0/3943030544 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x557adad850d0 0x557adad7fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:52.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.172+0000 7fe4d2d9d640 1 -- 172.21.15.67:0/3943030544 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557adad854b0 msgr2=0x557adad8aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:52.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.172+0000 7fe4d2d9d640 1 --2- 172.21.15.67:0/3943030544 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557adad854b0 0x557adad8aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:52.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.172+0000 7fe4d2d9d640 1 -- 172.21.15.67:0/3943030544 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557adae0b2c0 con 0x557adad876b0 2024-03-24T00:42:52.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.172+0000 7fe4d1d9b640 1 --2- 172.21.15.67:0/3943030544 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557adad854b0 0x557adad8aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:42:52.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.172+0000 7fe4d2d9d640 1 --2- 172.21.15.67:0/3943030544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557adad876b0 0x557adadf3990 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc036040 tx=0x7fe4cc02e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b65dfe955074dff8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:52.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.172+0000 7fe4b17fa640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:42:52.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.172+0000 7fe4b17fa640 1 -- 172.21.15.67:0/3943030544 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4cc04e060 con 0x557adad876b0 2024-03-24T00:42:52.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.172+0000 7fe4b17fa640 1 -- 172.21.15.67:0/3943030544 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(64..64 src has 1..64) v4 ==== 6246+0+0 (secure 0 0 0) 0x7fe4cc03fb50 con 0x557adad876b0 2024-03-24T00:42:52.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.172+0000 7fe4db99c740 1 -- 172.21.15.67:0/3943030544 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe4c8087070 con 0x557adad876b0 2024-03-24T00:42:52.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.172+0000 7fe4b17fa640 1 -- 172.21.15.67:0/3943030544 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe4cc054050 con 0x557adad876b0 2024-03-24T00:42:52.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.172+0000 7fe4db99c740 10 client.6693 fetch_fsmap learned FSMap version 34 2024-03-24T00:42:52.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.172+0000 7fe4db99c740 1 -- 172.21.15.67:0/3943030544 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=2+}) v3 -- 0x557adad578d0 con 0x557adad876b0 2024-03-24T00:42:52.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.172+0000 7fe4b17fa640 1 -- 172.21.15.67:0/3943030544 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4c8087070 con 0x557adad876b0 2024-03-24T00:42:52.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.173+0000 7fe4b17fa640 1 -- 172.21.15.67:0/3943030544 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 34) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fe4cc057020 con 0x557adad876b0 2024-03-24T00:42:52.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.173+0000 7fe4db99c740 10 client.6693 fetch_fsmap finished waiting for FSMap version 34 2024-03-24T00:42:52.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.173+0000 7fe4db99c740 10 client.6693 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-24T00:42:52.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.173+0000 7fe4db99c740 20 client.6693 populate_metadata read hostname 'smithi067' 2024-03-24T00:42:52.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.173+0000 7fe4db99c740 1 --2- 172.21.15.67:0/3943030544 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x557adad5db20 0x557adadbc420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:52.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.173+0000 7fe4db99c740 4 client.6693 mds_command: new command op to 6360 tid=0 multi_id=0 [{"prefix": "damage ls", "format": "json-pretty"}] 2024-03-24T00:42:52.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.173+0000 7fe4db99c740 1 -- 172.21.15.67:0/3943030544 --> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) v1 -- 0x557adacc9a80 con 0x557adad5db20 2024-03-24T00:42:52.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.173+0000 7fe4d259c640 1 --2- 172.21.15.67:0/3943030544 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x557adad5db20 0x557adadbc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:52.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.173+0000 7fe4b17fa640 10 client.6693 ms_handle_connect on v2:172.21.15.67:6834/451962260 2024-03-24T00:42:52.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.173+0000 7fe4d259c640 1 --2- 172.21.15.67:0/3943030544 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x557adad5db20 0x557adadbc420 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7db40d6f7942de server_cookie=24226d758ff76cd4 in_seq=0 out_seq=0 2024-03-24T00:42:52.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.173+0000 7fe4b17fa640 1 -- 172.21.15.67:0/3943030544 <== mds.0 v2:172.21.15.67:6834/451962260 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x557adacc9a80 con 0x557adad5db20 2024-03-24T00:42:52.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.173+0000 7fe4b17fa640 10 client.6693 handle_command_reply: tid=0 2024-03-24T00:42:52.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.173+0000 7fe4db99c740 1 client.6693 shutdown 2024-03-24T00:42:52.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.173+0000 7fe4db99c740 1 -- 172.21.15.67:0/3943030544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557adad876b0 msgr2=0x557adadf3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:52.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.173+0000 7fe4db99c740 1 --2- 172.21.15.67:0/3943030544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557adad876b0 0x557adadf3990 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc036040 tx=0x7fe4cc02e070 comp rx=0 tx=0).stop 2024-03-24T00:42:52.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.174+0000 7fe4db99c740 1 -- 172.21.15.67:0/3943030544 shutdown_connections 2024-03-24T00:42:52.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.174+0000 7fe4db99c740 1 --2- 172.21.15.67:0/3943030544 >> [v2:172.21.15.67:6834/451962260,v1:172.21.15.67:6835/451962260] conn(0x557adad5db20 0x557adadbc420 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:52.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.174+0000 7fe4db99c740 1 --2- 172.21.15.67:0/3943030544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557adad876b0 0x557adadf3990 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:52.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.174+0000 7fe4db99c740 1 --2- 172.21.15.67:0/3943030544 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557adad854b0 0x557adad8aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:52.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.174+0000 7fe4b17fa640 0 client.6693 ms_handle_reset on v2:172.21.15.67:6834/451962260 2024-03-24T00:42:52.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.174+0000 7fe4db99c740 1 --2- 172.21.15.67:0/3943030544 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x557adad850d0 0x557adad7fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:52.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.174+0000 7fe4db99c740 1 -- 172.21.15.67:0/3943030544 >> 172.21.15.67:0/3943030544 conn(0x557adad58840 msgr2=0x557adad884a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:52.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.174+0000 7fe4db99c740 1 -- 172.21.15.67:0/3943030544 shutdown_connections 2024-03-24T00:42:52.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.174+0000 7fe4db99c740 1 -- 172.21.15.67:0/3943030544 wait complete. 2024-03-24T00:42:52.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.174+0000 7fe4db99c740 20 client.6693 trim_cache size 0 max 16384 2024-03-24T00:42:52.180 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:42:52.180 INFO:teuthology.orchestra.run.smithi067.stdout:[] 2024-03-24T00:42:52.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.174+0000 7fe4d8cde640 1 -- 172.21.15.67:0/2531487338 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe4b406bbd0 msgr2=0x7fe4b406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:52.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.174+0000 7fe4d8cde640 1 --2- 172.21.15.67:0/2531487338 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe4b406bbd0 0x7fe4b406e090 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fe4c4006fd0 tx=0x7fe4c4008040 comp rx=0 tx=0).stop 2024-03-24T00:42:52.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.174+0000 7fe4d8cde640 1 -- 172.21.15.67:0/2531487338 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4d415dea0 msgr2=0x7fe4d414c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:52.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.174+0000 7fe4d8cde640 1 --2- 172.21.15.67:0/2531487338 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4d415dea0 0x7fe4d414c550 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fe4c800b3f0 tx=0x7fe4c800b8b0 comp rx=0 tx=0).stop 2024-03-24T00:42:52.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.174+0000 7fe4d8cde640 1 -- 172.21.15.67:0/2531487338 shutdown_connections 2024-03-24T00:42:52.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.174+0000 7fe4d8cde640 1 --2- 172.21.15.67:0/2531487338 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe4b406bbd0 0x7fe4b406e090 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:52.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.174+0000 7fe4d8cde640 1 --2- 172.21.15.67:0/2531487338 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe4d415e280 0x7fe4d4157720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:52.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.174+0000 7fe4d8cde640 1 --2- 172.21.15.67:0/2531487338 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4d415dea0 0x7fe4d414c550 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:52.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.174+0000 7fe4d8cde640 1 --2- 172.21.15.67:0/2531487338 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe4d40b6170 0x7fe4d41612a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:52.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.174+0000 7fe4d8cde640 1 -- 172.21.15.67:0/2531487338 >> 172.21.15.67:0/2531487338 conn(0x7fe4d4072da0 msgr2=0x7fe4d40b39e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:52.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.174+0000 7fe4d8cde640 1 -- 172.21.15.67:0/2531487338 shutdown_connections 2024-03-24T00:42:52.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.174+0000 7fe4d8cde640 1 -- 172.21.15.67:0/2531487338 wait complete. 2024-03-24T00:42:52.188 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-24T00:42:52.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.288+0000 7fc0bd868640 1 Processor -- start 2024-03-24T00:42:52.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.288+0000 7fc0bd868640 1 -- start start 2024-03-24T00:42:52.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.289+0000 7fc0bd868640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc0b80d0b90 0x7fc0b80b01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:52.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.289+0000 7fc0bd868640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0b80b24f0 con 0x7fc0b815ae70 2024-03-24T00:42:52.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.289+0000 7fc0bd868640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0b80b1730 con 0x7fc0b80cf240 2024-03-24T00:42:52.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.289+0000 7fc0bd868640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0b80b1a10 con 0x7fc0b80d0b90 2024-03-24T00:42:52.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.289+0000 7fc0b6ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc0b80d0b90 0x7fc0b80b01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:52.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.289+0000 7fc0b67fc640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc0b815ae70 0x7fc0b80d0f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53956/0 (socket says 172.21.15.67:53956) 2024-03-24T00:42:52.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.289+0000 7fc0b67fc640 1 -- 172.21.15.67:0/3837086444 learned_addr learned my addr 172.21.15.67:0/3837086444 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:42:52.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.289+0000 7fc0b6ffd640 1 -- 172.21.15.67:0/3837086444 >> v1:172.21.15.72:6789/0 conn(0x7fc0b80cf240 legacy=0x7fc0b80ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:52.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.289+0000 7fc0b6ffd640 1 -- 172.21.15.67:0/3837086444 >> v1:172.21.15.67:6789/0 conn(0x7fc0b815ae70 legacy=0x7fc0b80d0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:52.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.289+0000 7fc0b6ffd640 1 -- 172.21.15.67:0/3837086444 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0b80b1cf0 con 0x7fc0b80d0b90 2024-03-24T00:42:52.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.290+0000 7fc0b6ffd640 1 --2- 172.21.15.67:0/3837086444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0b80d0b90 0x7fc0b80b01d0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fc0a8009d90 tx=0x7fc0a802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4278952ceb5f0fa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:52.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.291+0000 7fc0b5ffb640 1 -- 172.21.15.67:0/3837086444 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a8039070 con 0x7fc0b80d0b90 2024-03-24T00:42:52.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.291+0000 7fc0b5ffb640 1 -- 172.21.15.67:0/3837086444 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0a802cd00 con 0x7fc0b80d0b90 2024-03-24T00:42:52.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.291+0000 7fc0b5ffb640 1 -- 172.21.15.67:0/3837086444 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a8034660 con 0x7fc0b80d0b90 2024-03-24T00:42:52.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.291+0000 7fc0bd868640 1 -- 172.21.15.67:0/3837086444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0b80d0b90 msgr2=0x7fc0b80b01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:52.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.291+0000 7fc0bd868640 1 --2- 172.21.15.67:0/3837086444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0b80d0b90 0x7fc0b80b01d0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fc0a8009d90 tx=0x7fc0a802bf50 comp rx=0 tx=0).stop 2024-03-24T00:42:52.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.291+0000 7fc0bd868640 1 -- 172.21.15.67:0/3837086444 shutdown_connections 2024-03-24T00:42:52.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.291+0000 7fc0bd868640 1 --2- 172.21.15.67:0/3837086444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0b80d0b90 0x7fc0b80b01d0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:52.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.291+0000 7fc0bd868640 1 -- 172.21.15.67:0/3837086444 >> 172.21.15.67:0/3837086444 conn(0x7fc0b8099a30 msgr2=0x7fc0b8099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:52.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.291+0000 7fc0bd868640 1 -- 172.21.15.67:0/3837086444 shutdown_connections 2024-03-24T00:42:52.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.291+0000 7fc0bd868640 1 -- 172.21.15.67:0/3837086444 wait complete. 2024-03-24T00:42:52.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.293+0000 7fc0bd868640 1 Processor -- start 2024-03-24T00:42:52.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.293+0000 7fc0bd868640 1 -- start start 2024-03-24T00:42:52.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.294+0000 7fc0bd868640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc0b80cf240 0x7fc0b806efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:52.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.294+0000 7fc0bd868640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc0b80d0b90 0x7fc0b80c7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:52.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.294+0000 7fc0bd868640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0b815ae70 0x7fc0b80c8530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:52.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.294+0000 7fc0bd868640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b80cf9e0 con 0x7fc0b815ae70 2024-03-24T00:42:52.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.294+0000 7fc0bd868640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b80cf6d0 con 0x7fc0b80cf240 2024-03-24T00:42:52.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.294+0000 7fc0bd868640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b80adfe0 con 0x7fc0b80d0b90 2024-03-24T00:42:52.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.295+0000 7fc0b77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0b815ae70 0x7fc0b80c8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:52.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.295+0000 7fc0b77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0b815ae70 0x7fc0b80c8530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48090/0 (socket says 172.21.15.67:48090) 2024-03-24T00:42:52.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.295+0000 7fc0b77fe640 1 -- 172.21.15.67:0/3739951135 learned_addr learned my addr 172.21.15.67:0/3739951135 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:52.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.295+0000 7fc0b6ffd640 1 --2- 172.21.15.67:0/3739951135 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc0b80cf240 0x7fc0b806efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:52.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.295+0000 7fc0b67fc640 1 --2- 172.21.15.67:0/3739951135 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc0b80d0b90 0x7fc0b80c7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:52.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.295+0000 7fc0b77fe640 1 -- 172.21.15.67:0/3739951135 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc0b80d0b90 msgr2=0x7fc0b80c7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:52.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.295+0000 7fc0b77fe640 1 --2- 172.21.15.67:0/3739951135 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc0b80d0b90 0x7fc0b80c7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:52.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.295+0000 7fc0b77fe640 1 -- 172.21.15.67:0/3739951135 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc0b80cf240 msgr2=0x7fc0b806efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:52.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.295+0000 7fc0b77fe640 1 --2- 172.21.15.67:0/3739951135 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc0b80cf240 0x7fc0b806efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:52.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.295+0000 7fc0b77fe640 1 -- 172.21.15.67:0/3739951135 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0b80cb030 con 0x7fc0b815ae70 2024-03-24T00:42:52.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.295+0000 7fc0b77fe640 1 --2- 172.21.15.67:0/3739951135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0b815ae70 0x7fc0b80c8530 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fc0b0001eb0 tx=0x7fc0b000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:52.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.295+0000 7fc0affff640 1 -- 172.21.15.67:0/3739951135 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b0017070 con 0x7fc0b815ae70 2024-03-24T00:42:52.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.295+0000 7fc0affff640 1 -- 172.21.15.67:0/3739951135 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0b00052d0 con 0x7fc0b815ae70 2024-03-24T00:42:52.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.295+0000 7fc0bd868640 1 -- 172.21.15.67:0/3739951135 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0b80cad50 con 0x7fc0b815ae70 2024-03-24T00:42:52.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.295+0000 7fc0affff640 1 -- 172.21.15.67:0/3739951135 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b0003810 con 0x7fc0b815ae70 2024-03-24T00:42:52.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.295+0000 7fc0bd868640 1 -- 172.21.15.67:0/3739951135 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0b80b4680 con 0x7fc0b815ae70 2024-03-24T00:42:52.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.296+0000 7fc0bd868640 1 -- 172.21.15.67:0/3739951135 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0b80d0f70 con 0x7fc0b815ae70 2024-03-24T00:42:52.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.297+0000 7fc0affff640 1 -- 172.21.15.67:0/3739951135 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc0b001a070 con 0x7fc0b815ae70 2024-03-24T00:42:52.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.297+0000 7fc0affff640 1 --2- 172.21.15.67:0/3739951135 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc09806bbd0 0x7fc09806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:52.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.297+0000 7fc0affff640 1 -- 172.21.15.67:0/3739951135 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6246+0+0 (secure 0 0 0) 0x7fc0b008a030 con 0x7fc0b815ae70 2024-03-24T00:42:52.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.298+0000 7fc0b6ffd640 1 --2- 172.21.15.67:0/3739951135 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc09806bbd0 0x7fc09806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:52.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.298+0000 7fc0b6ffd640 1 --2- 172.21.15.67:0/3739951135 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc09806bbd0 0x7fc09806e090 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fc0a8036000 tx=0x7fc0a8030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:52.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.302+0000 7fc0affff640 1 -- 172.21.15.67:0/3739951135 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc0b80d0f70 con 0x7fc0b815ae70 2024-03-24T00:42:52.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.521+0000 7fc0bd868640 1 -- 172.21.15.67:0/3739951135 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc0b804f400 con 0x7fc0b815ae70 2024-03-24T00:42:52.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.522+0000 7fc0affff640 1 -- 172.21.15.67:0/3739951135 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fc0b0058910 con 0x7fc0b815ae70 2024-03-24T00:42:52.522 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:42:52.523 INFO:teuthology.orchestra.run.smithi067.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-24T00:42:52.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.523+0000 7fc0bd868640 1 -- 172.21.15.67:0/3739951135 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc09806bbd0 msgr2=0x7fc09806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:52.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.523+0000 7fc0bd868640 1 --2- 172.21.15.67:0/3739951135 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc09806bbd0 0x7fc09806e090 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fc0a8036000 tx=0x7fc0a8030000 comp rx=0 tx=0).stop 2024-03-24T00:42:52.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.523+0000 7fc0bd868640 1 -- 172.21.15.67:0/3739951135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0b815ae70 msgr2=0x7fc0b80c8530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:52.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.523+0000 7fc0bd868640 1 --2- 172.21.15.67:0/3739951135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0b815ae70 0x7fc0b80c8530 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fc0b0001eb0 tx=0x7fc0b000c890 comp rx=0 tx=0).stop 2024-03-24T00:42:52.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.523+0000 7fc0bd868640 1 -- 172.21.15.67:0/3739951135 shutdown_connections 2024-03-24T00:42:52.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.523+0000 7fc0bd868640 1 --2- 172.21.15.67:0/3739951135 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc09806bbd0 0x7fc09806e090 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:52.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.523+0000 7fc0bd868640 1 --2- 172.21.15.67:0/3739951135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0b815ae70 0x7fc0b80c8530 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:52.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.523+0000 7fc0bd868640 1 --2- 172.21.15.67:0/3739951135 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc0b80d0b90 0x7fc0b80c7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:52.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.523+0000 7fc0bd868640 1 --2- 172.21.15.67:0/3739951135 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc0b80cf240 0x7fc0b806efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:52.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.523+0000 7fc0bd868640 1 -- 172.21.15.67:0/3739951135 >> 172.21.15.67:0/3739951135 conn(0x7fc0b8099a30 msgr2=0x7fc0b815a110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:52.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.524+0000 7fc0bd868640 1 -- 172.21.15.67:0/3739951135 shutdown_connections 2024-03-24T00:42:52.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.524+0000 7fc0bd868640 1 -- 172.21.15.67:0/3739951135 wait complete. 2024-03-24T00:42:52.535 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-03-24T00:42:52.535 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-24T00:42:52.535 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:42:52.579 DEBUG:teuthology.orchestra.run.smithi072:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:42:52.630 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-24T00:42:52.630 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:42:52.630 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:42:52.645 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:42:52.645 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:42:52.699 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:42:52.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.840+0000 7f92c764b640 1 Processor -- start 2024-03-24T00:42:52.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.840+0000 7f92c764b640 1 -- start start 2024-03-24T00:42:52.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.841+0000 7f92c764b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f92c009a410 0x7f92c00cfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:52.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.841+0000 7f92c764b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f92c00b43a0 0x7f92c00cc880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:52.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.841+0000 7f92c764b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92c00d0fc0 con 0x7f92c015e1b0 2024-03-24T00:42:52.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.841+0000 7f92c764b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92c015e590 con 0x7f92c00b43a0 2024-03-24T00:42:52.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.841+0000 7f92c764b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92c00d1820 con 0x7f92c009a410 2024-03-24T00:42:52.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.841+0000 7f92c5be8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f92c00b43a0 0x7f92c00cc880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:52.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.841+0000 7f92c53e7640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f92c015e1b0 0x7f92c0099d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53970/0 (socket says 172.21.15.67:53970) 2024-03-24T00:42:52.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.841+0000 7f92c53e7640 1 -- 172.21.15.67:0/3393250679 learned_addr learned my addr 172.21.15.67:0/3393250679 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:42:52.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.841+0000 7f92c4be6640 1 --2- 172.21.15.67:0/3393250679 >> v2:172.21.15.72:3300/0 conn(0x7f92c009a410 0x7f92c00cfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:52.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.842+0000 7f92c5be8640 1 -- 172.21.15.67:0/3393250679 >> v2:172.21.15.72:3300/0 conn(0x7f92c009a410 msgr2=0x7f92c00cfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:52.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.842+0000 7f92c5be8640 1 --2- 172.21.15.67:0/3393250679 >> v2:172.21.15.72:3300/0 conn(0x7f92c009a410 0x7f92c00cfe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:52.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.842+0000 7f92c5be8640 1 -- 172.21.15.67:0/3393250679 >> v1:172.21.15.67:6789/0 conn(0x7f92c015e1b0 legacy=0x7f92c0099d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:52.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.842+0000 7f92c5be8640 1 -- 172.21.15.67:0/3393250679 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c00d1260 con 0x7f92c00b43a0 2024-03-24T00:42:52.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.842+0000 7f92b3fff640 1 -- 172.21.15.67:0/3393250679 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3056997416 0 0) 0x7f92c00d0fc0 con 0x7f92c015e1b0 2024-03-24T00:42:52.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.842+0000 7f92c5be8640 1 --2- 172.21.15.67:0/3393250679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92c00b43a0 0x7f92c00cc880 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f92c00c8c30 tx=0x7f92b802bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9b0602d6e39787 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:52.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.843+0000 7f92b3fff640 1 -- 172.21.15.67:0/3393250679 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b8039070 con 0x7f92c00b43a0 2024-03-24T00:42:52.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.843+0000 7f92b3fff640 1 -- 172.21.15.67:0/3393250679 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92b802cb60 con 0x7f92c00b43a0 2024-03-24T00:42:52.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.843+0000 7f92b3fff640 1 -- 172.21.15.67:0/3393250679 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b8034440 con 0x7f92c00b43a0 2024-03-24T00:42:52.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.843+0000 7f92c764b640 1 -- 172.21.15.67:0/3393250679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92c00b43a0 msgr2=0x7f92c00cc880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:52.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.843+0000 7f92c764b640 1 --2- 172.21.15.67:0/3393250679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92c00b43a0 0x7f92c00cc880 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f92c00c8c30 tx=0x7f92b802bdb0 comp rx=0 tx=0).stop 2024-03-24T00:42:52.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.843+0000 7f92c764b640 1 -- 172.21.15.67:0/3393250679 shutdown_connections 2024-03-24T00:42:52.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.843+0000 7f92c764b640 1 --2- 172.21.15.67:0/3393250679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92c00b43a0 0x7f92c00cc880 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:52.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.843+0000 7f92c764b640 1 --2- 172.21.15.67:0/3393250679 >> v2:172.21.15.72:3300/0 conn(0x7f92c009a410 0x7f92c00cfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:52.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.843+0000 7f92c764b640 1 -- 172.21.15.67:0/3393250679 >> 172.21.15.67:0/3393250679 conn(0x7f92c00b36d0 msgr2=0x7f92c00b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:52.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.843+0000 7f92c764b640 1 -- 172.21.15.67:0/3393250679 shutdown_connections 2024-03-24T00:42:52.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.843+0000 7f92c764b640 1 -- 172.21.15.67:0/3393250679 wait complete. 2024-03-24T00:42:52.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.844+0000 7f92c764b640 1 Processor -- start 2024-03-24T00:42:52.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.844+0000 7f92c764b640 1 -- start start 2024-03-24T00:42:52.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.845+0000 7f92c764b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f92c009a410 0x7f92c0150d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:52.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.845+0000 7f92c764b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f92c00b43a0 0x7f92c01512a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:52.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.845+0000 7f92c764b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92c015e1b0 0x7f92c00b6af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:52.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.845+0000 7f92c764b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c009a9a0 con 0x7f92c015e1b0 2024-03-24T00:42:52.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.845+0000 7f92c764b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c00cc360 con 0x7f92c00b43a0 2024-03-24T00:42:52.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.845+0000 7f92c764b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92c00ccdc0 con 0x7f92c009a410 2024-03-24T00:42:52.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.845+0000 7f92c5be8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92c015e1b0 0x7f92c00b6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:52.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.845+0000 7f92c5be8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92c015e1b0 0x7f92c00b6af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48108/0 (socket says 172.21.15.67:48108) 2024-03-24T00:42:52.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.845+0000 7f92c5be8640 1 -- 172.21.15.67:0/3231157838 learned_addr learned my addr 172.21.15.67:0/3231157838 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:52.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.845+0000 7f92c4be6640 1 --2- 172.21.15.67:0/3231157838 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f92c00b43a0 0x7f92c01512a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:52.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.845+0000 7f92c53e7640 1 --2- 172.21.15.67:0/3231157838 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f92c009a410 0x7f92c0150d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:52.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.846+0000 7f92c5be8640 1 -- 172.21.15.67:0/3231157838 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f92c009a410 msgr2=0x7f92c0150d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:52.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.846+0000 7f92c5be8640 1 --2- 172.21.15.67:0/3231157838 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f92c009a410 0x7f92c0150d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:52.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.846+0000 7f92c5be8640 1 -- 172.21.15.67:0/3231157838 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f92c00b43a0 msgr2=0x7f92c01512a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:52.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.846+0000 7f92c5be8640 1 --2- 172.21.15.67:0/3231157838 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f92c00b43a0 0x7f92c01512a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:52.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.846+0000 7f92c5be8640 1 -- 172.21.15.67:0/3231157838 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c01519e0 con 0x7f92c015e1b0 2024-03-24T00:42:52.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.846+0000 7f92c5be8640 1 --2- 172.21.15.67:0/3231157838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92c015e1b0 0x7f92c00b6af0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f92b802c3c0 tx=0x7f92b8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:52.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.846+0000 7f92b1ffb640 1 -- 172.21.15.67:0/3231157838 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b8039050 con 0x7f92c015e1b0 2024-03-24T00:42:52.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.846+0000 7f92b1ffb640 1 -- 172.21.15.67:0/3231157838 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92b803a040 con 0x7f92c015e1b0 2024-03-24T00:42:52.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.846+0000 7f92c764b640 1 -- 172.21.15.67:0/3231157838 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92c00b4820 con 0x7f92c015e1b0 2024-03-24T00:42:52.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.846+0000 7f92b1ffb640 1 -- 172.21.15.67:0/3231157838 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b803e610 con 0x7f92c015e1b0 2024-03-24T00:42:52.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.846+0000 7f92c764b640 1 -- 172.21.15.67:0/3231157838 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92c00b4de0 con 0x7f92c015e1b0 2024-03-24T00:42:52.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.847+0000 7f92c764b640 1 -- 172.21.15.67:0/3231157838 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92c004f400 con 0x7f92c015e1b0 2024-03-24T00:42:52.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.848+0000 7f92b1ffb640 1 -- 172.21.15.67:0/3231157838 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f92b8047020 con 0x7f92c015e1b0 2024-03-24T00:42:52.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.848+0000 7f92b1ffb640 1 --2- 172.21.15.67:0/3231157838 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f92a406bbd0 0x7f92a406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:52.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.848+0000 7f92b1ffb640 1 -- 172.21.15.67:0/3231157838 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6246+0+0 (secure 0 0 0) 0x7f92b80acf60 con 0x7f92c015e1b0 2024-03-24T00:42:52.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.848+0000 7f92c53e7640 1 --2- 172.21.15.67:0/3231157838 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f92a406bbd0 0x7f92a406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:52.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.849+0000 7f92c53e7640 1 --2- 172.21.15.67:0/3231157838 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f92a406bbd0 0x7f92a406e090 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f92b4000f60 tx=0x7f92b4007c00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:52.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:52.853+0000 7f92b1ffb640 1 -- 172.21.15.67:0/3231157838 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f92b804e020 con 0x7f92c015e1b0 2024-03-24T00:42:53.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.035+0000 7f92c764b640 1 -- 172.21.15.67:0/3231157838 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f92c00cf8f0 con 0x7f92c015e1b0 2024-03-24T00:42:53.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.036+0000 7f92b1ffb640 1 -- 172.21.15.67:0/3231157838 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) v1 ==== 74+0+17064 (secure 0 0 0) 0x7f92b807b8c0 con 0x7f92c015e1b0 2024-03-24T00:42:53.037 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:42:53.037 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":64,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:42:33.971493+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:41:40.502194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-03-24T00:41:41.200073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"9.7","mappings":[{"from":7,"to":2}]},{"pgid":"10.17","mappings":[{"from":4,"to":5}]},{"pgid":"10.1d","mappings":[{"from":4,"to":3}]},{"pgid":"10.2e","mappings":[{"from":4,"to":3}]},{"pgid":"10.3a","mappings":[{"from":7,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:42:53.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.038+0000 7f92c764b640 1 -- 172.21.15.67:0/3231157838 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f92a406bbd0 msgr2=0x7f92a406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:53.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.038+0000 7f92c764b640 1 --2- 172.21.15.67:0/3231157838 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f92a406bbd0 0x7f92a406e090 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f92b4000f60 tx=0x7f92b4007c00 comp rx=0 tx=0).stop 2024-03-24T00:42:53.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.038+0000 7f92c764b640 1 -- 172.21.15.67:0/3231157838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92c015e1b0 msgr2=0x7f92c00b6af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:53.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.038+0000 7f92c764b640 1 --2- 172.21.15.67:0/3231157838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92c015e1b0 0x7f92c00b6af0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f92b802c3c0 tx=0x7f92b8004a40 comp rx=0 tx=0).stop 2024-03-24T00:42:53.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.038+0000 7f92c764b640 1 -- 172.21.15.67:0/3231157838 shutdown_connections 2024-03-24T00:42:53.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.038+0000 7f92c764b640 1 --2- 172.21.15.67:0/3231157838 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f92a406bbd0 0x7f92a406e090 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:53.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.038+0000 7f92c764b640 1 --2- 172.21.15.67:0/3231157838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92c015e1b0 0x7f92c00b6af0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:53.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.038+0000 7f92c764b640 1 --2- 172.21.15.67:0/3231157838 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f92c00b43a0 0x7f92c01512a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:53.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.038+0000 7f92c764b640 1 --2- 172.21.15.67:0/3231157838 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f92c009a410 0x7f92c0150d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:53.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.039+0000 7f92c764b640 1 -- 172.21.15.67:0/3231157838 >> 172.21.15.67:0/3231157838 conn(0x7f92c00b36d0 msgr2=0x7f92c015c240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:53.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.039+0000 7f92c764b640 1 -- 172.21.15.67:0/3231157838 shutdown_connections 2024-03-24T00:42:53.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.039+0000 7f92c764b640 1 -- 172.21.15.67:0/3231157838 wait complete. 2024-03-24T00:42:53.053 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:38:39.364950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:41:40.502194+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:41:41.200073+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:42:53.054 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:42:53.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.157+0000 7f65a1cde640 1 Processor -- start 2024-03-24T00:42:53.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.157+0000 7f65a1cde640 1 -- start start 2024-03-24T00:42:53.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.158+0000 7f65a1cde640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f659c0b6170 0x7f659c168c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:53.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.158+0000 7f65a1cde640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f659c0d2250 con 0x7f659c15dea0 2024-03-24T00:42:53.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.158+0000 7f65a1cde640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f659c0d2530 con 0x7f659c0d3ae0 2024-03-24T00:42:53.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.158+0000 7f65a1cde640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f659c0d2810 con 0x7f659c0b6170 2024-03-24T00:42:53.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.158+0000 7f659b7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f659c15dea0 0x7f659c0d33d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53976/0 (socket says 172.21.15.67:53976) 2024-03-24T00:42:53.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.158+0000 7f659b7fe640 1 -- 172.21.15.67:0/815023266 learned_addr learned my addr 172.21.15.67:0/815023266 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:42:53.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.158+0000 7f659a7fc640 1 -- 172.21.15.67:0/815023266 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 251637525 0 0) 0x7f659c0d2250 con 0x7f659c15dea0 2024-03-24T00:42:53.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.158+0000 7f659bfff640 1 --2- 172.21.15.67:0/815023266 >> v2:172.21.15.72:3300/0 conn(0x7f659c0b6170 0x7f659c168c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:53.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.158+0000 7f659a7fc640 1 -- 172.21.15.67:0/815023266 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6588003520 con 0x7f659c15dea0 2024-03-24T00:42:53.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.159+0000 7f659a7fc640 1 -- 172.21.15.67:0/815023266 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 404840505 0 0) 0x7f6588003520 con 0x7f659c15dea0 2024-03-24T00:42:53.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.159+0000 7f659a7fc640 1 -- 172.21.15.67:0/815023266 >> v2:172.21.15.72:3300/0 conn(0x7f659c0b6170 msgr2=0x7f659c168c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:53.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.159+0000 7f659a7fc640 1 --2- 172.21.15.67:0/815023266 >> v2:172.21.15.72:3300/0 conn(0x7f659c0b6170 0x7f659c168c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:53.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.159+0000 7f659a7fc640 1 -- 172.21.15.67:0/815023266 >> v1:172.21.15.72:6789/0 conn(0x7f659c0d3ae0 legacy=0x7f659c0b5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:53.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.159+0000 7f659a7fc640 1 -- 172.21.15.67:0/815023266 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f659c0d2af0 con 0x7f659c15dea0 2024-03-24T00:42:53.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.159+0000 7f659a7fc640 1 -- 172.21.15.67:0/815023266 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f658c0019f0 con 0x7f659c15dea0 2024-03-24T00:42:53.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.159+0000 7f659a7fc640 1 -- 172.21.15.67:0/815023266 >> v1:172.21.15.67:6789/0 conn(0x7f659c15dea0 legacy=0x7f659c0d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:53.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.159+0000 7f659a7fc640 1 --2- 172.21.15.67:0/815023266 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f65880039b0 0x7f6588003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:53.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.159+0000 7f659a7fc640 1 --2- 172.21.15.67:0/815023266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6588004800 0x7f6588004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:53.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.159+0000 7f659a7fc640 1 --2- 172.21.15.67:0/815023266 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f65880051c0 0x7f658800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:53.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.159+0000 7f659a7fc640 1 -- 172.21.15.67:0/815023266 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f659c0d2810 con 0x7f6588004800 2024-03-24T00:42:53.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.159+0000 7f659a7fc640 1 -- 172.21.15.67:0/815023266 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f658800c160 con 0x7f65880039b0 2024-03-24T00:42:53.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.159+0000 7f659a7fc640 1 -- 172.21.15.67:0/815023266 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f658800c360 con 0x7f65880051c0 2024-03-24T00:42:53.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.159+0000 7f659affd640 1 --2- 172.21.15.67:0/815023266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6588004800 0x7f6588004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:53.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.159+0000 7f659b7fe640 1 --2- 172.21.15.67:0/815023266 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f65880039b0 0x7f6588003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:53.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.159+0000 7f659bfff640 1 --2- 172.21.15.67:0/815023266 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f65880051c0 0x7f658800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:53.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.159+0000 7f659affd640 1 -- 172.21.15.67:0/815023266 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f65880051c0 msgr2=0x7f658800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:53.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.159+0000 7f659affd640 1 --2- 172.21.15.67:0/815023266 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f65880051c0 0x7f658800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:53.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.159+0000 7f659affd640 1 -- 172.21.15.67:0/815023266 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f65880039b0 msgr2=0x7f6588003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:53.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.159+0000 7f659affd640 1 --2- 172.21.15.67:0/815023266 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f65880039b0 0x7f6588003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:53.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.159+0000 7f659affd640 1 -- 172.21.15.67:0/815023266 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6588003520 con 0x7f6588004800 2024-03-24T00:42:53.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.159+0000 7f659bfff640 1 -- 172.21.15.67:0/815023266 reap_dead start 2024-03-24T00:42:53.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.160+0000 7f659affd640 1 --2- 172.21.15.67:0/815023266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6588004800 0x7f6588004c80 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f6584001750 tx=0x7f6588005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd96fbccd082240b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:53.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.161+0000 7f659a7fc640 1 -- 172.21.15.67:0/815023266 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6584001370 con 0x7f6588004800 2024-03-24T00:42:53.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.161+0000 7f659a7fc640 1 -- 172.21.15.67:0/815023266 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65840052d0 con 0x7f6588004800 2024-03-24T00:42:53.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.161+0000 7f659a7fc640 1 -- 172.21.15.67:0/815023266 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6584003830 con 0x7f6588004800 2024-03-24T00:42:53.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.161+0000 7f65a1cde640 1 -- 172.21.15.67:0/815023266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6588004800 msgr2=0x7f6588004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:53.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.161+0000 7f65a1cde640 1 --2- 172.21.15.67:0/815023266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6588004800 0x7f6588004c80 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f6584001750 tx=0x7f6588005830 comp rx=0 tx=0).stop 2024-03-24T00:42:53.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.161+0000 7f65a1cde640 1 -- 172.21.15.67:0/815023266 shutdown_connections 2024-03-24T00:42:53.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.161+0000 7f65a1cde640 1 --2- 172.21.15.67:0/815023266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6588004800 0x7f6588004c80 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:53.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.161+0000 7f65a1cde640 1 -- 172.21.15.67:0/815023266 >> 172.21.15.67:0/815023266 conn(0x7f659c072da0 msgr2=0x7f659c0d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:53.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.161+0000 7f65a1cde640 1 -- 172.21.15.67:0/815023266 shutdown_connections 2024-03-24T00:42:53.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.161+0000 7f65a1cde640 1 -- 172.21.15.67:0/815023266 wait complete. 2024-03-24T00:42:53.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.162+0000 7f65a1cde640 1 Processor -- start 2024-03-24T00:42:53.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.162+0000 7f65a1cde640 1 -- start start 2024-03-24T00:42:53.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.163+0000 7f65a1cde640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6588004800 0x7f659c15f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:53.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.163+0000 7f65a1cde640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f659c1562b0 0x7f659c159b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:53.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.163+0000 7f65a1cde640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f659c1f6470 0x7f659c1f68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:53.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.163+0000 7f65a1cde640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f659c0b5a60 con 0x7f659c1f6470 2024-03-24T00:42:53.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.163+0000 7f65a1cde640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f659c15e490 con 0x7f659c1562b0 2024-03-24T00:42:53.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.163+0000 7f65a1cde640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f659c0b5be0 con 0x7f6588004800 2024-03-24T00:42:53.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.163+0000 7f659bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f659c1f6470 0x7f659c1f68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:53.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.163+0000 7f659bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f659c1f6470 0x7f659c1f68f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48138/0 (socket says 172.21.15.67:48138) 2024-03-24T00:42:53.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.163+0000 7f659bfff640 1 -- 172.21.15.67:0/2032217580 learned_addr learned my addr 172.21.15.67:0/2032217580 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:53.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.163+0000 7f659b7fe640 1 --2- 172.21.15.67:0/2032217580 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6588004800 0x7f659c15f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:53.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.163+0000 7f659affd640 1 --2- 172.21.15.67:0/2032217580 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f659c1562b0 0x7f659c159b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:53.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.164+0000 7f659bfff640 1 -- 172.21.15.67:0/2032217580 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6588004800 msgr2=0x7f659c15f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:53.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.164+0000 7f659bfff640 1 --2- 172.21.15.67:0/2032217580 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6588004800 0x7f659c15f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:53.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.164+0000 7f659bfff640 1 -- 172.21.15.67:0/2032217580 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f659c1562b0 msgr2=0x7f659c159b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:53.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.164+0000 7f659bfff640 1 --2- 172.21.15.67:0/2032217580 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f659c1562b0 0x7f659c159b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:53.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.164+0000 7f659bfff640 1 -- 172.21.15.67:0/2032217580 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f659c15f870 con 0x7f659c1f6470 2024-03-24T00:42:53.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.164+0000 7f659affd640 1 --2- 172.21.15.67:0/2032217580 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f659c1562b0 0x7f659c159b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:42:53.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.164+0000 7f659bfff640 1 --2- 172.21.15.67:0/2032217580 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f659c1f6470 0x7f659c1f68f0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f659c0d0500 tx=0x7f6594011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:53.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.164+0000 7f6593fff640 1 -- 172.21.15.67:0/2032217580 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6594019020 con 0x7f659c1f6470 2024-03-24T00:42:53.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.164+0000 7f6593fff640 1 -- 172.21.15.67:0/2032217580 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6594003610 con 0x7f659c1f6470 2024-03-24T00:42:53.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.164+0000 7f65a1cde640 1 -- 172.21.15.67:0/2032217580 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f659c15fb50 con 0x7f659c1f6470 2024-03-24T00:42:53.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.164+0000 7f6593fff640 1 -- 172.21.15.67:0/2032217580 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6594003a40 con 0x7f659c1f6470 2024-03-24T00:42:53.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.164+0000 7f65a1cde640 1 -- 172.21.15.67:0/2032217580 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f659c160110 con 0x7f659c1f6470 2024-03-24T00:42:53.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.165+0000 7f65a1cde640 1 -- 172.21.15.67:0/2032217580 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f659c04f400 con 0x7f659c1f6470 2024-03-24T00:42:53.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.166+0000 7f6593fff640 1 -- 172.21.15.67:0/2032217580 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f659401c020 con 0x7f659c1f6470 2024-03-24T00:42:53.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.166+0000 7f6593fff640 1 --2- 172.21.15.67:0/2032217580 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f657c06bbd0 0x7f657c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:53.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.166+0000 7f6593fff640 1 -- 172.21.15.67:0/2032217580 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6246+0+0 (secure 0 0 0) 0x7f659408b110 con 0x7f659c1f6470 2024-03-24T00:42:53.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.166+0000 7f659b7fe640 1 --2- 172.21.15.67:0/2032217580 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f657c06bbd0 0x7f657c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:53.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.167+0000 7f659b7fe640 1 --2- 172.21.15.67:0/2032217580 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f657c06bbd0 0x7f657c06e090 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f658c001370 tx=0x7f658c006040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:53.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.170+0000 7f6593fff640 1 -- 172.21.15.67:0/2032217580 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6594023020 con 0x7f659c1f6470 2024-03-24T00:42:53.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.350+0000 7f65a1cde640 1 -- 172.21.15.67:0/2032217580 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f659c0b4f90 con 0x7f659c1f6470 2024-03-24T00:42:53.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.351+0000 7f6593fff640 1 -- 172.21.15.67:0/2032217580 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v64) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6594059a70 con 0x7f659c1f6470 2024-03-24T00:42:53.351 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-24T00:42:53.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.354+0000 7f65a1cde640 1 -- 172.21.15.67:0/2032217580 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f657c06bbd0 msgr2=0x7f657c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:53.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.354+0000 7f65a1cde640 1 --2- 172.21.15.67:0/2032217580 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f657c06bbd0 0x7f657c06e090 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f658c001370 tx=0x7f658c006040 comp rx=0 tx=0).stop 2024-03-24T00:42:53.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.354+0000 7f65a1cde640 1 -- 172.21.15.67:0/2032217580 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f659c1f6470 msgr2=0x7f659c1f68f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:53.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.354+0000 7f65a1cde640 1 --2- 172.21.15.67:0/2032217580 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f659c1f6470 0x7f659c1f68f0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f659c0d0500 tx=0x7f6594011920 comp rx=0 tx=0).stop 2024-03-24T00:42:53.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.354+0000 7f65a1cde640 1 -- 172.21.15.67:0/2032217580 shutdown_connections 2024-03-24T00:42:53.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.354+0000 7f65a1cde640 1 --2- 172.21.15.67:0/2032217580 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f657c06bbd0 0x7f657c06e090 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:53.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.354+0000 7f65a1cde640 1 --2- 172.21.15.67:0/2032217580 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f659c1f6470 0x7f659c1f68f0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:53.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.354+0000 7f65a1cde640 1 --2- 172.21.15.67:0/2032217580 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f659c1562b0 0x7f659c159b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:53.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.354+0000 7f65a1cde640 1 --2- 172.21.15.67:0/2032217580 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6588004800 0x7f659c15f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:53.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.354+0000 7f65a1cde640 1 -- 172.21.15.67:0/2032217580 >> 172.21.15.67:0/2032217580 conn(0x7f659c072da0 msgr2=0x7f659c0d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:53.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.354+0000 7f65a1cde640 1 -- 172.21.15.67:0/2032217580 shutdown_connections 2024-03-24T00:42:53.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.354+0000 7f65a1cde640 1 -- 172.21.15.67:0/2032217580 wait complete. 2024-03-24T00:42:53.366 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:42:53.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.470+0000 7fe9cefcd640 1 Processor -- start 2024-03-24T00:42:53.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.470+0000 7fe9cefcd640 1 -- start start 2024-03-24T00:42:53.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.471+0000 7fe9cefcd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe9c815df00 0x7fe9c815ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:53.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.471+0000 7fe9cefcd640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe9c80cead0 0x7fe9c806c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:53.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.471+0000 7fe9cefcd640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c815e2e0 con 0x7fe9c815e580 2024-03-24T00:42:53.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.471+0000 7fe9cefcd640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9c80b1880 con 0x7fe9c815df00 2024-03-24T00:42:53.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.471+0000 7fe9cefcd640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c80b1b60 con 0x7fe9c80cead0 2024-03-24T00:42:53.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.471+0000 7fe9ccd69640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe9c815df00 0x7fe9c815ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:53.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.471+0000 7fe9ccd69640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe9c815df00 0x7fe9c815ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48146/0 (socket says 172.21.15.67:48146) 2024-03-24T00:42:53.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.471+0000 7fe9ccd69640 1 -- 172.21.15.67:0/1327917680 learned_addr learned my addr 172.21.15.67:0/1327917680 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:53.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.471+0000 7fe9cd56a640 1 --2- 172.21.15.67:0/1327917680 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe9c80cead0 0x7fe9c806c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:53.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.471+0000 7fe9ccd69640 1 -- 172.21.15.67:0/1327917680 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe9c80cead0 msgr2=0x7fe9c806c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:53.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.471+0000 7fe9ccd69640 1 --2- 172.21.15.67:0/1327917680 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe9c80cead0 0x7fe9c806c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:53.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.471+0000 7fe9ccd69640 1 -- 172.21.15.67:0/1327917680 >> v1:172.21.15.72:6789/0 conn(0x7fe9c815e580 legacy=0x7fe9c80ce3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:53.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.471+0000 7fe9ccd69640 1 -- 172.21.15.67:0/1327917680 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c80b1e40 con 0x7fe9c815df00 2024-03-24T00:42:53.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.472+0000 7fe9ccd69640 1 --2- 172.21.15.67:0/1327917680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9c815df00 0x7fe9c815ed70 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0009870 tx=0x7fe9b002b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=808bc45c2de425fe server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:53.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.472+0000 7fe9c77fe640 1 -- 172.21.15.67:0/1327917680 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b0038070 con 0x7fe9c815df00 2024-03-24T00:42:53.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.472+0000 7fe9c77fe640 1 -- 172.21.15.67:0/1327917680 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9b0031740 con 0x7fe9c815df00 2024-03-24T00:42:53.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.472+0000 7fe9c77fe640 1 -- 172.21.15.67:0/1327917680 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b002f050 con 0x7fe9c815df00 2024-03-24T00:42:53.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.473+0000 7fe9cefcd640 1 -- 172.21.15.67:0/1327917680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9c815df00 msgr2=0x7fe9c815ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:53.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.473+0000 7fe9cefcd640 1 --2- 172.21.15.67:0/1327917680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9c815df00 0x7fe9c815ed70 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0009870 tx=0x7fe9b002b7e0 comp rx=0 tx=0).stop 2024-03-24T00:42:53.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.473+0000 7fe9cefcd640 1 -- 172.21.15.67:0/1327917680 shutdown_connections 2024-03-24T00:42:53.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.473+0000 7fe9cefcd640 1 --2- 172.21.15.67:0/1327917680 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe9c80cead0 0x7fe9c806c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:53.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.473+0000 7fe9cefcd640 1 --2- 172.21.15.67:0/1327917680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9c815df00 0x7fe9c815ed70 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:53.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.473+0000 7fe9cefcd640 1 -- 172.21.15.67:0/1327917680 >> 172.21.15.67:0/1327917680 conn(0x7fe9c808b180 msgr2=0x7fe9c808b590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:53.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.473+0000 7fe9cefcd640 1 -- 172.21.15.67:0/1327917680 shutdown_connections 2024-03-24T00:42:53.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.473+0000 7fe9cefcd640 1 -- 172.21.15.67:0/1327917680 wait complete. 2024-03-24T00:42:53.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.475+0000 7fe9cefcd640 1 Processor -- start 2024-03-24T00:42:53.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.475+0000 7fe9cefcd640 1 -- start start 2024-03-24T00:42:53.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.476+0000 7fe9cefcd640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe9c80cead0 0x7fe9c814ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:53.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.476+0000 7fe9cefcd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe9c815df00 0x7fe9c80af240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:53.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.476+0000 7fe9cefcd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9c815e580 0x7fe9c80d03b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:53.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.476+0000 7fe9cefcd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c815f2b0 con 0x7fe9c815e580 2024-03-24T00:42:53.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.476+0000 7fe9cefcd640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c80b2400 con 0x7fe9c815df00 2024-03-24T00:42:53.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.476+0000 7fe9cefcd640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c815f430 con 0x7fe9c80cead0 2024-03-24T00:42:53.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.476+0000 7fe9cd56a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9c815e580 0x7fe9c80d03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:53.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.476+0000 7fe9cd56a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9c815e580 0x7fe9c80d03b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48162/0 (socket says 172.21.15.67:48162) 2024-03-24T00:42:53.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.476+0000 7fe9cd56a640 1 -- 172.21.15.67:0/1956892538 learned_addr learned my addr 172.21.15.67:0/1956892538 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:53.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.476+0000 7fe9ccd69640 1 --2- 172.21.15.67:0/1956892538 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe9c80cead0 0x7fe9c814ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:53.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.476+0000 7fe9c7fff640 1 --2- 172.21.15.67:0/1956892538 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe9c815df00 0x7fe9c80af240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:53.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.476+0000 7fe9cd56a640 1 -- 172.21.15.67:0/1956892538 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe9c80cead0 msgr2=0x7fe9c814ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:53.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.476+0000 7fe9cd56a640 1 --2- 172.21.15.67:0/1956892538 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe9c80cead0 0x7fe9c814ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:53.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.476+0000 7fe9cd56a640 1 -- 172.21.15.67:0/1956892538 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe9c815df00 msgr2=0x7fe9c80af240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:53.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.476+0000 7fe9cd56a640 1 --2- 172.21.15.67:0/1956892538 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe9c815df00 0x7fe9c80af240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:53.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.476+0000 7fe9cd56a640 1 -- 172.21.15.67:0/1956892538 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c814f3c0 con 0x7fe9c815e580 2024-03-24T00:42:53.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.477+0000 7fe9cd56a640 1 --2- 172.21.15.67:0/1956892538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9c815e580 0x7fe9c80d03b0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fe9c000cd70 tx=0x7fe9c000a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:53.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.477+0000 7fe9c57fa640 1 -- 172.21.15.67:0/1956892538 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c0003b90 con 0x7fe9c815e580 2024-03-24T00:42:53.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.477+0000 7fe9c57fa640 1 -- 172.21.15.67:0/1956892538 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9c0003d30 con 0x7fe9c815e580 2024-03-24T00:42:53.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.477+0000 7fe9cefcd640 1 -- 172.21.15.67:0/1956892538 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9c80afa90 con 0x7fe9c815e580 2024-03-24T00:42:53.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.477+0000 7fe9c57fa640 1 -- 172.21.15.67:0/1956892538 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c0004240 con 0x7fe9c815e580 2024-03-24T00:42:53.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.477+0000 7fe9cefcd640 1 -- 172.21.15.67:0/1956892538 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9c80cd500 con 0x7fe9c815e580 2024-03-24T00:42:53.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.478+0000 7fe9cefcd640 1 -- 172.21.15.67:0/1956892538 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9c804f400 con 0x7fe9c815e580 2024-03-24T00:42:53.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.478+0000 7fe9c57fa640 1 -- 172.21.15.67:0/1956892538 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe9c00043e0 con 0x7fe9c815e580 2024-03-24T00:42:53.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.479+0000 7fe9c57fa640 1 --2- 172.21.15.67:0/1956892538 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe9a806bc00 0x7fe9a806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:53.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.479+0000 7fe9c57fa640 1 -- 172.21.15.67:0/1956892538 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6246+0+0 (secure 0 0 0) 0x7fe9c008b520 con 0x7fe9c815e580 2024-03-24T00:42:53.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.479+0000 7fe9ccd69640 1 --2- 172.21.15.67:0/1956892538 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe9a806bc00 0x7fe9a806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:53.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.480+0000 7fe9ccd69640 1 --2- 172.21.15.67:0/1956892538 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe9a806bc00 0x7fe9a806e0c0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0035000 tx=0x7fe9b0008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:53.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.484+0000 7fe9c57fa640 1 -- 172.21.15.67:0/1956892538 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe9c0059830 con 0x7fe9c815e580 2024-03-24T00:42:53.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.742+0000 7fe9cefcd640 1 -- 172.21.15.67:0/1956892538 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe9c806d710 con 0x7fe9c815e580 2024-03-24T00:42:53.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.743+0000 7fe9c57fa640 1 -- 172.21.15.67:0/1956892538 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v64) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe9c005d8d0 con 0x7fe9c815e580 2024-03-24T00:42:53.744 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-24T00:42:53.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.745+0000 7fe9cefcd640 1 -- 172.21.15.67:0/1956892538 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe9a806bc00 msgr2=0x7fe9a806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:53.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.745+0000 7fe9cefcd640 1 --2- 172.21.15.67:0/1956892538 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe9a806bc00 0x7fe9a806e0c0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0035000 tx=0x7fe9b0008030 comp rx=0 tx=0).stop 2024-03-24T00:42:53.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.745+0000 7fe9cefcd640 1 -- 172.21.15.67:0/1956892538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9c815e580 msgr2=0x7fe9c80d03b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:53.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.745+0000 7fe9cefcd640 1 --2- 172.21.15.67:0/1956892538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9c815e580 0x7fe9c80d03b0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fe9c000cd70 tx=0x7fe9c000a660 comp rx=0 tx=0).stop 2024-03-24T00:42:53.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.746+0000 7fe9cefcd640 1 -- 172.21.15.67:0/1956892538 shutdown_connections 2024-03-24T00:42:53.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.746+0000 7fe9cefcd640 1 --2- 172.21.15.67:0/1956892538 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe9a806bc00 0x7fe9a806e0c0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:53.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.746+0000 7fe9cefcd640 1 --2- 172.21.15.67:0/1956892538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9c815e580 0x7fe9c80d03b0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:53.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.746+0000 7fe9cefcd640 1 --2- 172.21.15.67:0/1956892538 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe9c815df00 0x7fe9c80af240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:53.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.746+0000 7fe9cefcd640 1 --2- 172.21.15.67:0/1956892538 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe9c80cead0 0x7fe9c814ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:53.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.746+0000 7fe9cefcd640 1 -- 172.21.15.67:0/1956892538 >> 172.21.15.67:0/1956892538 conn(0x7fe9c808b180 msgr2=0x7fe9c80ce3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:53.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.746+0000 7fe9cefcd640 1 -- 172.21.15.67:0/1956892538 shutdown_connections 2024-03-24T00:42:53.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.746+0000 7fe9cefcd640 1 -- 172.21.15.67:0/1956892538 wait complete. 2024-03-24T00:42:53.760 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:42:53.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.864+0000 7f0f82759640 1 Processor -- start 2024-03-24T00:42:53.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.864+0000 7f0f82759640 1 -- start start 2024-03-24T00:42:53.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.865+0000 7f0f82759640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f0f7c15dea0 0x7f0f7c0d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:53.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.865+0000 7f0f82759640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c0b6170 0x7f0f7c0d3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:53.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.865+0000 7f0f82759640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f7c0d2250 con 0x7f0f7c15e280 2024-03-24T00:42:53.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.865+0000 7f0f82759640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f7c0d2530 con 0x7f0f7c0b6170 2024-03-24T00:42:53.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.865+0000 7f0f82759640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f7c0d2810 con 0x7f0f7c15dea0 2024-03-24T00:42:53.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.865+0000 7f0f80cf6640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c0b6170 0x7f0f7c0d3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:53.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.865+0000 7f0f80cf6640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c0b6170 0x7f0f7c0d3910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42736/0 (socket says 172.21.15.67:42736) 2024-03-24T00:42:53.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.865+0000 7f0f80cf6640 1 -- 172.21.15.67:0/4062571923 learned_addr learned my addr 172.21.15.67:0/4062571923 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:53.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.866+0000 7f0f7bfff640 1 --2- 172.21.15.67:0/4062571923 >> v2:172.21.15.72:3300/0 conn(0x7f0f7c15dea0 0x7f0f7c0d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:53.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.866+0000 7f0f80cf6640 1 -- 172.21.15.67:0/4062571923 >> v2:172.21.15.72:3300/0 conn(0x7f0f7c15dea0 msgr2=0x7f0f7c0d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:53.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.866+0000 7f0f80cf6640 1 --2- 172.21.15.67:0/4062571923 >> v2:172.21.15.72:3300/0 conn(0x7f0f7c15dea0 0x7f0f7c0d33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:53.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.866+0000 7f0f80cf6640 1 -- 172.21.15.67:0/4062571923 >> v1:172.21.15.72:6789/0 conn(0x7f0f7c15e280 legacy=0x7f0f7c0b5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:53.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.866+0000 7f0f80cf6640 1 -- 172.21.15.67:0/4062571923 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f7c0d2af0 con 0x7f0f7c0b6170 2024-03-24T00:42:53.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.866+0000 7f0f7bfff640 1 --2- 172.21.15.67:0/4062571923 >> v2:172.21.15.72:3300/0 conn(0x7f0f7c15dea0 0x7f0f7c0d33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:42:53.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.866+0000 7f0f80cf6640 1 --2- 172.21.15.67:0/4062571923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f7c0b6170 0x7f0f7c0d3910 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f0f74009e50 tx=0x7f0f7402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9471eb5f34624beb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:53.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.866+0000 7f0f7affd640 1 -- 172.21.15.67:0/4062571923 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f7402c9f0 con 0x7f0f7c0b6170 2024-03-24T00:42:53.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.866+0000 7f0f7affd640 1 -- 172.21.15.67:0/4062571923 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f7402cb90 con 0x7f0f7c0b6170 2024-03-24T00:42:53.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.867+0000 7f0f7affd640 1 -- 172.21.15.67:0/4062571923 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f740343f0 con 0x7f0f7c0b6170 2024-03-24T00:42:53.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.867+0000 7f0f82759640 1 -- 172.21.15.67:0/4062571923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f7c0b6170 msgr2=0x7f0f7c0d3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:53.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.867+0000 7f0f82759640 1 --2- 172.21.15.67:0/4062571923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f7c0b6170 0x7f0f7c0d3910 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f0f74009e50 tx=0x7f0f7402be50 comp rx=0 tx=0).stop 2024-03-24T00:42:53.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.867+0000 7f0f82759640 1 -- 172.21.15.67:0/4062571923 shutdown_connections 2024-03-24T00:42:53.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.867+0000 7f0f82759640 1 --2- 172.21.15.67:0/4062571923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f7c0b6170 0x7f0f7c0d3910 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:53.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.867+0000 7f0f82759640 1 --2- 172.21.15.67:0/4062571923 >> v2:172.21.15.72:3300/0 conn(0x7f0f7c15dea0 0x7f0f7c0d33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:53.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.867+0000 7f0f82759640 1 -- 172.21.15.67:0/4062571923 >> 172.21.15.67:0/4062571923 conn(0x7f0f7c072da0 msgr2=0x7f0f7c0d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:53.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.867+0000 7f0f82759640 1 -- 172.21.15.67:0/4062571923 shutdown_connections 2024-03-24T00:42:53.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.867+0000 7f0f82759640 1 -- 172.21.15.67:0/4062571923 wait complete. 2024-03-24T00:42:53.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.868+0000 7f0f82759640 1 Processor -- start 2024-03-24T00:42:53.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.868+0000 7f0f82759640 1 -- start start 2024-03-24T00:42:53.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.869+0000 7f0f82759640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f7c0b6170 0x7f0f7c157590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:53.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.869+0000 7f0f82759640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0f7c15dea0 0x7f0f7c1f3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:53.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.869+0000 7f0f82759640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0f7c15e280 0x7f0f7c0cc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:53.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.869+0000 7f0f82759640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7c0b7480 con 0x7f0f7c0b6170 2024-03-24T00:42:53.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.869+0000 7f0f82759640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7c0b7300 con 0x7f0f7c15e280 2024-03-24T00:42:53.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.869+0000 7f0f82759640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7c0b7600 con 0x7f0f7c15dea0 2024-03-24T00:42:53.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.869+0000 7f0f7bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f7c0b6170 0x7f0f7c157590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:53.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.869+0000 7f0f7bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f7c0b6170 0x7f0f7c157590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42738/0 (socket says 172.21.15.67:42738) 2024-03-24T00:42:53.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.869+0000 7f0f7bfff640 1 -- 172.21.15.67:0/3284948763 learned_addr learned my addr 172.21.15.67:0/3284948763 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:53.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.869+0000 7f0f80cf6640 1 --2- 172.21.15.67:0/3284948763 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0f7c15e280 0x7f0f7c0cc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:53.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.869+0000 7f0f7b7fe640 1 --2- 172.21.15.67:0/3284948763 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0f7c15dea0 0x7f0f7c1f3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:53.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.870+0000 7f0f7bfff640 1 -- 172.21.15.67:0/3284948763 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0f7c15dea0 msgr2=0x7f0f7c1f3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:53.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.870+0000 7f0f7bfff640 1 --2- 172.21.15.67:0/3284948763 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0f7c15dea0 0x7f0f7c1f3ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:53.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.870+0000 7f0f7bfff640 1 -- 172.21.15.67:0/3284948763 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0f7c15e280 msgr2=0x7f0f7c0cc0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:53.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.870+0000 7f0f7bfff640 1 --2- 172.21.15.67:0/3284948763 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0f7c15e280 0x7f0f7c0cc0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:53.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.870+0000 7f0f7bfff640 1 -- 172.21.15.67:0/3284948763 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f7c0cb300 con 0x7f0f7c0b6170 2024-03-24T00:42:53.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.870+0000 7f0f80cf6640 1 --2- 172.21.15.67:0/3284948763 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0f7c15e280 0x7f0f7c0cc0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:42:53.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.870+0000 7f0f7b7fe640 1 --2- 172.21.15.67:0/3284948763 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0f7c15dea0 0x7f0f7c1f3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:42:53.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.870+0000 7f0f7bfff640 1 --2- 172.21.15.67:0/3284948763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f7c0b6170 0x7f0f7c157590 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f0f7000c820 tx=0x7f0f7000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:53.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.870+0000 7f0f78ff9640 1 -- 172.21.15.67:0/3284948763 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f70015070 con 0x7f0f7c0b6170 2024-03-24T00:42:53.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.870+0000 7f0f78ff9640 1 -- 172.21.15.67:0/3284948763 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f70007a30 con 0x7f0f7c0b6170 2024-03-24T00:42:53.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.870+0000 7f0f82759640 1 -- 172.21.15.67:0/3284948763 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f7c0cb5e0 con 0x7f0f7c0b6170 2024-03-24T00:42:53.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.870+0000 7f0f78ff9640 1 -- 172.21.15.67:0/3284948763 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f70007e00 con 0x7f0f7c0b6170 2024-03-24T00:42:53.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.870+0000 7f0f82759640 1 -- 172.21.15.67:0/3284948763 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f7c0cbba0 con 0x7f0f7c0b6170 2024-03-24T00:42:53.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.871+0000 7f0f82759640 1 -- 172.21.15.67:0/3284948763 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f7c04f400 con 0x7f0f7c0b6170 2024-03-24T00:42:53.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.871+0000 7f0f78ff9640 1 -- 172.21.15.67:0/3284948763 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0f70018070 con 0x7f0f7c0b6170 2024-03-24T00:42:53.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.872+0000 7f0f78ff9640 1 --2- 172.21.15.67:0/3284948763 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0f6806bbd0 0x7f0f6806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:53.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.872+0000 7f0f78ff9640 1 -- 172.21.15.67:0/3284948763 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6246+0+0 (secure 0 0 0) 0x7f0f7008aca0 con 0x7f0f7c0b6170 2024-03-24T00:42:53.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.872+0000 7f0f7b7fe640 1 --2- 172.21.15.67:0/3284948763 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0f6806bbd0 0x7f0f6806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:53.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.873+0000 7f0f7b7fe640 1 --2- 172.21.15.67:0/3284948763 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0f6806bbd0 0x7f0f6806e090 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c006fd0 tx=0x7f0f6c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:53.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:53.877+0000 7f0f78ff9640 1 -- 172.21.15.67:0/3284948763 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0f70059570 con 0x7f0f7c0b6170 2024-03-24T00:42:54.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.084+0000 7f0f82759640 1 -- 172.21.15.67:0/3284948763 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0f7c0cbd70 con 0x7f0f7c0b6170 2024-03-24T00:42:54.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.084+0000 7f0f78ff9640 1 -- 172.21.15.67:0/3284948763 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v64) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0f70058e70 con 0x7f0f7c0b6170 2024-03-24T00:42:54.087 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-03-24T00:42:54.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.087+0000 7f0f82759640 1 -- 172.21.15.67:0/3284948763 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0f6806bbd0 msgr2=0x7f0f6806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:54.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.087+0000 7f0f82759640 1 --2- 172.21.15.67:0/3284948763 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0f6806bbd0 0x7f0f6806e090 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c006fd0 tx=0x7f0f6c008040 comp rx=0 tx=0).stop 2024-03-24T00:42:54.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.087+0000 7f0f82759640 1 -- 172.21.15.67:0/3284948763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f7c0b6170 msgr2=0x7f0f7c157590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:54.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.087+0000 7f0f82759640 1 --2- 172.21.15.67:0/3284948763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f7c0b6170 0x7f0f7c157590 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f0f7000c820 tx=0x7f0f7000cc20 comp rx=0 tx=0).stop 2024-03-24T00:42:54.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.087+0000 7f0f82759640 1 -- 172.21.15.67:0/3284948763 shutdown_connections 2024-03-24T00:42:54.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.087+0000 7f0f82759640 1 --2- 172.21.15.67:0/3284948763 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0f6806bbd0 0x7f0f6806e090 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:54.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.087+0000 7f0f82759640 1 --2- 172.21.15.67:0/3284948763 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0f7c15e280 0x7f0f7c0cc0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:54.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.087+0000 7f0f82759640 1 --2- 172.21.15.67:0/3284948763 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0f7c15dea0 0x7f0f7c1f3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:54.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.087+0000 7f0f82759640 1 --2- 172.21.15.67:0/3284948763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f7c0b6170 0x7f0f7c157590 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:54.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.087+0000 7f0f82759640 1 -- 172.21.15.67:0/3284948763 >> 172.21.15.67:0/3284948763 conn(0x7f0f7c072da0 msgr2=0x7f0f7c0b8a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:54.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.087+0000 7f0f82759640 1 -- 172.21.15.67:0/3284948763 shutdown_connections 2024-03-24T00:42:54.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.088+0000 7f0f82759640 1 -- 172.21.15.67:0/3284948763 wait complete. 2024-03-24T00:42:54.099 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T00:42:54.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.200+0000 7f21c5916640 1 Processor -- start 2024-03-24T00:42:54.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.200+0000 7f21c5916640 1 -- start start 2024-03-24T00:42:54.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.201+0000 7f21c5916640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f21c006de60 0x7f21c009ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:54.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.201+0000 7f21c5916640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21c00b29b0 con 0x7f21c00b1eb0 2024-03-24T00:42:54.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.201+0000 7f21c5916640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21c00b23f0 con 0x7f21c009a410 2024-03-24T00:42:54.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.201+0000 7f21c5916640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21c00cc9e0 con 0x7f21c006de60 2024-03-24T00:42:54.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.201+0000 7f21beffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f21c006de60 0x7f21c009ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:54.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.201+0000 7f21be7fc640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f21c00b1eb0 0x7f21c0099d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60900/0 (socket says 172.21.15.67:60900) 2024-03-24T00:42:54.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.201+0000 7f21be7fc640 1 -- 172.21.15.67:0/3374789425 learned_addr learned my addr 172.21.15.67:0/3374789425 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:42:54.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.201+0000 7f21beffd640 1 -- 172.21.15.67:0/3374789425 >> v1:172.21.15.72:6789/0 conn(0x7f21c009a410 legacy=0x7f21c00cd410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:54.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.201+0000 7f21bdffb640 1 -- 172.21.15.67:0/3374789425 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2787394793 0 0) 0x7f21c00b29b0 con 0x7f21c00b1eb0 2024-03-24T00:42:54.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.201+0000 7f21beffd640 1 -- 172.21.15.67:0/3374789425 >> v1:172.21.15.67:6789/0 conn(0x7f21c00b1eb0 legacy=0x7f21c0099d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:54.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.201+0000 7f21beffd640 1 -- 172.21.15.67:0/3374789425 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21c00b26d0 con 0x7f21c006de60 2024-03-24T00:42:54.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.202+0000 7f21beffd640 1 --2- 172.21.15.67:0/3374789425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21c006de60 0x7f21c009ae80 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f21a80097a0 tx=0x7f21a802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b10d6d700ce4b52d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:54.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.202+0000 7f21bdffb640 1 -- 172.21.15.67:0/3374789425 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a8034070 con 0x7f21c006de60 2024-03-24T00:42:54.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.202+0000 7f21bdffb640 1 -- 172.21.15.67:0/3374789425 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21a8030610 con 0x7f21c006de60 2024-03-24T00:42:54.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.202+0000 7f21bdffb640 1 -- 172.21.15.67:0/3374789425 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a80309e0 con 0x7f21c006de60 2024-03-24T00:42:54.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.203+0000 7f21c5916640 1 -- 172.21.15.67:0/3374789425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21c006de60 msgr2=0x7f21c009ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:54.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.203+0000 7f21c5916640 1 --2- 172.21.15.67:0/3374789425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21c006de60 0x7f21c009ae80 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f21a80097a0 tx=0x7f21a802b960 comp rx=0 tx=0).stop 2024-03-24T00:42:54.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.203+0000 7f21c5916640 1 -- 172.21.15.67:0/3374789425 shutdown_connections 2024-03-24T00:42:54.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.203+0000 7f21c5916640 1 --2- 172.21.15.67:0/3374789425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21c006de60 0x7f21c009ae80 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:54.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.203+0000 7f21c5916640 1 -- 172.21.15.67:0/3374789425 >> 172.21.15.67:0/3374789425 conn(0x7f21c00b3650 msgr2=0x7f21c00b3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:54.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.203+0000 7f21c5916640 1 -- 172.21.15.67:0/3374789425 shutdown_connections 2024-03-24T00:42:54.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.203+0000 7f21c5916640 1 -- 172.21.15.67:0/3374789425 wait complete. 2024-03-24T00:42:54.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.204+0000 7f21c5916640 1 Processor -- start 2024-03-24T00:42:54.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.204+0000 7f21c5916640 1 -- start start 2024-03-24T00:42:54.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.205+0000 7f21c5916640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21c006de60 0x7f21c0158380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:54.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.205+0000 7f21c5916640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f21c009a410 0x7f21c01f6e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:54.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.205+0000 7f21c5916640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f21c00b1eb0 0x7f21c00b0270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:54.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.205+0000 7f21c5916640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21c009b3c0 con 0x7f21c006de60 2024-03-24T00:42:54.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.205+0000 7f21c5916640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21c00ccf80 con 0x7f21c00b1eb0 2024-03-24T00:42:54.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.205+0000 7f21c5916640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21c006c150 con 0x7f21c009a410 2024-03-24T00:42:54.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.205+0000 7f21beffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21c006de60 0x7f21c0158380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:54.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.205+0000 7f21beffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21c006de60 0x7f21c0158380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42764/0 (socket says 172.21.15.67:42764) 2024-03-24T00:42:54.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.205+0000 7f21beffd640 1 -- 172.21.15.67:0/3330102812 learned_addr learned my addr 172.21.15.67:0/3330102812 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:54.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.205+0000 7f21beffd640 1 -- 172.21.15.67:0/3330102812 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f21c009a410 msgr2=0x7f21c01f6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:54.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.205+0000 7f21beffd640 1 --2- 172.21.15.67:0/3330102812 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f21c009a410 0x7f21c01f6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:54.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.205+0000 7f21beffd640 1 -- 172.21.15.67:0/3330102812 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f21c00b1eb0 msgr2=0x7f21c00b0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:54.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.205+0000 7f21beffd640 1 --2- 172.21.15.67:0/3330102812 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f21c00b1eb0 0x7f21c00b0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:54.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.205+0000 7f21beffd640 1 -- 172.21.15.67:0/3330102812 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21c01f7970 con 0x7f21c006de60 2024-03-24T00:42:54.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.206+0000 7f21beffd640 1 --2- 172.21.15.67:0/3330102812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21c006de60 0x7f21c0158380 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f21a8000c00 tx=0x7f21a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:54.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.206+0000 7f21b7fff640 1 -- 172.21.15.67:0/3330102812 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a8047070 con 0x7f21c006de60 2024-03-24T00:42:54.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.206+0000 7f21b7fff640 1 -- 172.21.15.67:0/3330102812 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21a8004030 con 0x7f21c006de60 2024-03-24T00:42:54.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.206+0000 7f21c5916640 1 -- 172.21.15.67:0/3330102812 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21c01f7c50 con 0x7f21c006de60 2024-03-24T00:42:54.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.206+0000 7f21b7fff640 1 -- 172.21.15.67:0/3330102812 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a8034040 con 0x7f21c006de60 2024-03-24T00:42:54.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.206+0000 7f21c5916640 1 -- 172.21.15.67:0/3330102812 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21c00cab90 con 0x7f21c006de60 2024-03-24T00:42:54.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.207+0000 7f21c5916640 1 -- 172.21.15.67:0/3330102812 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21c004f400 con 0x7f21c006de60 2024-03-24T00:42:54.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.207+0000 7f21b7fff640 1 -- 172.21.15.67:0/3330102812 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f21a804a070 con 0x7f21c006de60 2024-03-24T00:42:54.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.208+0000 7f21b7fff640 1 --2- 172.21.15.67:0/3330102812 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f21a006bc00 0x7f21a006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:54.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.208+0000 7f21b7fff640 1 -- 172.21.15.67:0/3330102812 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6246+0+0 (secure 0 0 0) 0x7f21a80ac600 con 0x7f21c006de60 2024-03-24T00:42:54.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.208+0000 7f21be7fc640 1 --2- 172.21.15.67:0/3330102812 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f21a006bc00 0x7f21a006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:54.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.208+0000 7f21be7fc640 1 --2- 172.21.15.67:0/3330102812 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f21a006bc00 0x7f21a006e0c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f21b0007fd0 tx=0x7f21b0007900 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:54.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.212+0000 7f21b7fff640 1 -- 172.21.15.67:0/3330102812 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f21a807af60 con 0x7f21c006de60 2024-03-24T00:42:54.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.407+0000 7f21c5916640 1 -- 172.21.15.67:0/3330102812 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f21c00c5690 con 0x7f21c006de60 2024-03-24T00:42:54.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.407+0000 7f21b7fff640 1 -- 172.21.15.67:0/3330102812 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v64) v1 ==== 80+0+0 (secure 0 0 0) 0x7f21a807a860 con 0x7f21c006de60 2024-03-24T00:42:54.408 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-03-24T00:42:54.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.409+0000 7f21c5916640 1 -- 172.21.15.67:0/3330102812 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f21a006bc00 msgr2=0x7f21a006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:54.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.409+0000 7f21c5916640 1 --2- 172.21.15.67:0/3330102812 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f21a006bc00 0x7f21a006e0c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f21b0007fd0 tx=0x7f21b0007900 comp rx=0 tx=0).stop 2024-03-24T00:42:54.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.409+0000 7f21c5916640 1 -- 172.21.15.67:0/3330102812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21c006de60 msgr2=0x7f21c0158380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:54.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.409+0000 7f21c5916640 1 --2- 172.21.15.67:0/3330102812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21c006de60 0x7f21c0158380 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f21a8000c00 tx=0x7f21a80046c0 comp rx=0 tx=0).stop 2024-03-24T00:42:54.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.409+0000 7f21c5916640 1 -- 172.21.15.67:0/3330102812 shutdown_connections 2024-03-24T00:42:54.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.409+0000 7f21c5916640 1 --2- 172.21.15.67:0/3330102812 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f21a006bc00 0x7f21a006e0c0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:54.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.409+0000 7f21c5916640 1 --2- 172.21.15.67:0/3330102812 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f21c00b1eb0 0x7f21c00b0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:54.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.409+0000 7f21c5916640 1 --2- 172.21.15.67:0/3330102812 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f21c009a410 0x7f21c01f6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:54.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.409+0000 7f21c5916640 1 --2- 172.21.15.67:0/3330102812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21c006de60 0x7f21c0158380 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:54.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.409+0000 7f21c5916640 1 -- 172.21.15.67:0/3330102812 >> 172.21.15.67:0/3330102812 conn(0x7f21c00b3650 msgr2=0x7f21c006fdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:54.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.409+0000 7f21c5916640 1 -- 172.21.15.67:0/3330102812 shutdown_connections 2024-03-24T00:42:54.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.409+0000 7f21c5916640 1 -- 172.21.15.67:0/3330102812 wait complete. 2024-03-24T00:42:54.421 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-24T00:42:54.421 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:42:54.421 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-24T00:42:54.463 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-03-24T00:42:54.463 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:42:54.463 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-24T00:42:54.519 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:42:54.519 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:42:54.536 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17590 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248808k,nr_inodes=812202,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:42:54.536 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:42:54.536 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:42:54.574 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056375,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15771 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T00:42:54.574 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:42:54.574 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:42:54.592 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17590 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248808k,nr_inodes=812202,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:42:54.592 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:42:54.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.715+0000 7fc4fa636640 1 Processor -- start 2024-03-24T00:42:54.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.715+0000 7fc4fa636640 1 -- start start 2024-03-24T00:42:54.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.716+0000 7fc4fa636640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc4f415dec0 0x7fc4f40ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:54.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.716+0000 7fc4fa636640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc4f415e2a0 0x7fc4f4162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:54.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.716+0000 7fc4fa636640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4f40cf640 con 0x7fc4f40d07e0 2024-03-24T00:42:54.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.716+0000 7fc4fa636640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4f40cee30 con 0x7fc4f415e2a0 2024-03-24T00:42:54.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.716+0000 7fc4fa636640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f40d2120 con 0x7fc4f415dec0 2024-03-24T00:42:54.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.716+0000 7fc4f8bd3640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc4f40d07e0 0x7fc4f40b5850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60912/0 (socket says 172.21.15.67:60912) 2024-03-24T00:42:54.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.716+0000 7fc4f8bd3640 1 -- 172.21.15.67:0/578667314 learned_addr learned my addr 172.21.15.67:0/578667314 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:42:54.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.716+0000 7fc4f37fe640 1 --2- 172.21.15.67:0/578667314 >> v2:172.21.15.72:3300/0 conn(0x7fc4f415e2a0 0x7fc4f4162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:54.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.716+0000 7fc4f3fff640 1 --2- 172.21.15.67:0/578667314 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc4f415dec0 0x7fc4f40ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:54.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.716+0000 7fc4f2ffd640 1 -- 172.21.15.67:0/578667314 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 65254986 0 0) 0x7fc4ec001490 con 0x7fc4f40d07e0 2024-03-24T00:42:54.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.716+0000 7fc4f2ffd640 1 -- 172.21.15.67:0/578667314 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4e0003520 con 0x7fc4f40d07e0 2024-03-24T00:42:54.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.716+0000 7fc4f2ffd640 1 -- 172.21.15.67:0/578667314 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3962681855 0 0) 0x7fc4e0003520 con 0x7fc4f40d07e0 2024-03-24T00:42:54.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.716+0000 7fc4f2ffd640 1 -- 172.21.15.67:0/578667314 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc4f415dec0 msgr2=0x7fc4f40ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:54.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.716+0000 7fc4f2ffd640 1 --2- 172.21.15.67:0/578667314 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc4f415dec0 0x7fc4f40ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:54.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.716+0000 7fc4f2ffd640 1 -- 172.21.15.67:0/578667314 >> v2:172.21.15.72:3300/0 conn(0x7fc4f415e2a0 msgr2=0x7fc4f4162630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:54.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.716+0000 7fc4f2ffd640 1 --2- 172.21.15.67:0/578667314 >> v2:172.21.15.72:3300/0 conn(0x7fc4f415e2a0 0x7fc4f4162630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:54.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.716+0000 7fc4f2ffd640 1 -- 172.21.15.67:0/578667314 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4f40cf080 con 0x7fc4f40d07e0 2024-03-24T00:42:54.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.716+0000 7fc4f2ffd640 1 -- 172.21.15.67:0/578667314 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7fc4ec001bc0 con 0x7fc4f40d07e0 2024-03-24T00:42:54.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.716+0000 7fc4f2ffd640 1 -- 172.21.15.67:0/578667314 >> v1:172.21.15.67:6789/0 conn(0x7fc4f40d07e0 legacy=0x7fc4f40b5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:54.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.716+0000 7fc4f2ffd640 1 --2- 172.21.15.67:0/578667314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4e00039b0 0x7fc4e0003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:54.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.716+0000 7fc4f2ffd640 1 --2- 172.21.15.67:0/578667314 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc4f415dec0 0x7fc4e0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:54.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.716+0000 7fc4f2ffd640 1 --2- 172.21.15.67:0/578667314 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc4e0004d40 0x7fc4e00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:54.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.716+0000 7fc4f2ffd640 1 -- 172.21.15.67:0/578667314 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f40cee30 con 0x7fc4e00039b0 2024-03-24T00:42:54.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.716+0000 7fc4f2ffd640 1 -- 172.21.15.67:0/578667314 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f40d2120 con 0x7fc4e0004d40 2024-03-24T00:42:54.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.716+0000 7fc4f2ffd640 1 -- 172.21.15.67:0/578667314 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e000b9c0 con 0x7fc4f415dec0 2024-03-24T00:42:54.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.717+0000 7fc4f3fff640 1 --2- 172.21.15.67:0/578667314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4e00039b0 0x7fc4e0003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:54.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.717+0000 7fc4f37fe640 1 --2- 172.21.15.67:0/578667314 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc4f415dec0 0x7fc4e0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:54.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.717+0000 7fc4f8bd3640 1 --2- 172.21.15.67:0/578667314 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc4e0004d40 0x7fc4e00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:54.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.717+0000 7fc4f3fff640 1 -- 172.21.15.67:0/578667314 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc4f415dec0 msgr2=0x7fc4e0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:54.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.717+0000 7fc4f3fff640 1 --2- 172.21.15.67:0/578667314 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc4f415dec0 0x7fc4e0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:54.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.717+0000 7fc4f3fff640 1 -- 172.21.15.67:0/578667314 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc4e0004d40 msgr2=0x7fc4e00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:54.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.717+0000 7fc4f3fff640 1 --2- 172.21.15.67:0/578667314 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc4e0004d40 0x7fc4e00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:54.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.717+0000 7fc4f3fff640 1 -- 172.21.15.67:0/578667314 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4e0003520 con 0x7fc4e00039b0 2024-03-24T00:42:54.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.717+0000 7fc4f37fe640 1 --2- 172.21.15.67:0/578667314 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc4f415dec0 0x7fc4e0004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:42:54.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.718+0000 7fc4f3fff640 1 --2- 172.21.15.67:0/578667314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4e00039b0 0x7fc4e0003df0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc009ac0 tx=0x7fc4e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5abba879c17684ef server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:54.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.718+0000 7fc4f2ffd640 1 -- 172.21.15.67:0/578667314 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4dc039070 con 0x7fc4e00039b0 2024-03-24T00:42:54.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.718+0000 7fc4f2ffd640 1 -- 172.21.15.67:0/578667314 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4dc02c6b0 con 0x7fc4e00039b0 2024-03-24T00:42:54.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.718+0000 7fc4f2ffd640 1 -- 172.21.15.67:0/578667314 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4dc009690 con 0x7fc4e00039b0 2024-03-24T00:42:54.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.718+0000 7fc4fa636640 1 -- 172.21.15.67:0/578667314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4e00039b0 msgr2=0x7fc4e0003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:54.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.718+0000 7fc4fa636640 1 --2- 172.21.15.67:0/578667314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4e00039b0 0x7fc4e0003df0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc009ac0 tx=0x7fc4e0005830 comp rx=0 tx=0).stop 2024-03-24T00:42:54.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.718+0000 7fc4f8bd3640 1 -- 172.21.15.67:0/578667314 reap_dead start 2024-03-24T00:42:54.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.718+0000 7fc4fa636640 1 -- 172.21.15.67:0/578667314 shutdown_connections 2024-03-24T00:42:54.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.718+0000 7fc4fa636640 1 -- 172.21.15.67:0/578667314 >> 172.21.15.67:0/578667314 conn(0x7fc4f406c810 msgr2=0x7fc4f406cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:54.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.718+0000 7fc4fa636640 1 -- 172.21.15.67:0/578667314 shutdown_connections 2024-03-24T00:42:54.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.718+0000 7fc4fa636640 1 -- 172.21.15.67:0/578667314 wait complete. 2024-03-24T00:42:54.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.719+0000 7fc4fa636640 1 Processor -- start 2024-03-24T00:42:54.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.720+0000 7fc4fa636640 1 -- start start 2024-03-24T00:42:54.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.720+0000 7fc4fa636640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc4e00039b0 0x7fc4f40cbef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:54.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.720+0000 7fc4fa636640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4f4158fe0 0x7fc4f40b3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:54.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.720+0000 7fc4fa636640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc4f40b3d50 0x7fc4f415c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:54.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.720+0000 7fc4fa636640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f40d3740 con 0x7fc4f4158fe0 2024-03-24T00:42:54.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.720+0000 7fc4fa636640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f40d35c0 con 0x7fc4e00039b0 2024-03-24T00:42:54.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.720+0000 7fc4fa636640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f40d38c0 con 0x7fc4f40b3d50 2024-03-24T00:42:54.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.720+0000 7fc4f37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4f4158fe0 0x7fc4f40b3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:54.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.720+0000 7fc4f37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4f4158fe0 0x7fc4f40b3810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42778/0 (socket says 172.21.15.67:42778) 2024-03-24T00:42:54.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.720+0000 7fc4f37fe640 1 -- 172.21.15.67:0/1417833037 learned_addr learned my addr 172.21.15.67:0/1417833037 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:54.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.720+0000 7fc4f3fff640 1 --2- 172.21.15.67:0/1417833037 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc4e00039b0 0x7fc4f40cbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:54.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.721+0000 7fc4f8bd3640 1 --2- 172.21.15.67:0/1417833037 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc4f40b3d50 0x7fc4f415c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:54.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.721+0000 7fc4f37fe640 1 -- 172.21.15.67:0/1417833037 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc4f40b3d50 msgr2=0x7fc4f415c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:54.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.721+0000 7fc4f37fe640 1 --2- 172.21.15.67:0/1417833037 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc4f40b3d50 0x7fc4f415c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:54.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.721+0000 7fc4f37fe640 1 -- 172.21.15.67:0/1417833037 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc4e00039b0 msgr2=0x7fc4f40cbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:54.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.721+0000 7fc4f37fe640 1 --2- 172.21.15.67:0/1417833037 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc4e00039b0 0x7fc4f40cbef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:54.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.721+0000 7fc4f37fe640 1 -- 172.21.15.67:0/1417833037 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4f40b43b0 con 0x7fc4f4158fe0 2024-03-24T00:42:54.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.721+0000 7fc4f37fe640 1 --2- 172.21.15.67:0/1417833037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4f4158fe0 0x7fc4f40b3810 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4011ca0 tx=0x7fc4e400f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:54.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.721+0000 7fc4f0ff9640 1 -- 172.21.15.67:0/1417833037 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e4011cd0 con 0x7fc4f4158fe0 2024-03-24T00:42:54.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.721+0000 7fc4f0ff9640 1 -- 172.21.15.67:0/1417833037 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4e4011e70 con 0x7fc4f4158fe0 2024-03-24T00:42:54.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.721+0000 7fc4fa636640 1 -- 172.21.15.67:0/1417833037 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4f40b4690 con 0x7fc4f4158fe0 2024-03-24T00:42:54.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.721+0000 7fc4f0ff9640 1 -- 172.21.15.67:0/1417833037 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e4004220 con 0x7fc4f4158fe0 2024-03-24T00:42:54.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.721+0000 7fc4fa636640 1 -- 172.21.15.67:0/1417833037 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4f40b5440 con 0x7fc4f4158fe0 2024-03-24T00:42:54.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.722+0000 7fc4fa636640 1 -- 172.21.15.67:0/1417833037 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4f40b4130 con 0x7fc4f4158fe0 2024-03-24T00:42:54.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.723+0000 7fc4f0ff9640 1 -- 172.21.15.67:0/1417833037 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc4e4012040 con 0x7fc4f4158fe0 2024-03-24T00:42:54.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.723+0000 7fc4f0ff9640 1 --2- 172.21.15.67:0/1417833037 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc4d406bc00 0x7fc4d406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:54.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.723+0000 7fc4f0ff9640 1 -- 172.21.15.67:0/1417833037 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6246+0+0 (secure 0 0 0) 0x7fc4e408b030 con 0x7fc4f4158fe0 2024-03-24T00:42:54.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.723+0000 7fc4f3fff640 1 --2- 172.21.15.67:0/1417833037 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc4d406bc00 0x7fc4d406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:54.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.724+0000 7fc4f3fff640 1 --2- 172.21.15.67:0/1417833037 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc4d406bc00 0x7fc4d406e0c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc036000 tx=0x7fc4dc030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:54.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:54.732+0000 7fc4f0ff9640 1 -- 172.21.15.67:0/1417833037 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc4f40b4130 con 0x7fc4f4158fe0 2024-03-24T00:42:55.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.064+0000 7fc4fa636640 1 -- 172.21.15.67:0/1417833037 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc4f404f400 con 0x7fc4f4158fe0 2024-03-24T00:42:55.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.065+0000 7fc4f0ff9640 1 -- 172.21.15.67:0/1417833037 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4976 (secure 0 0 0) 0x7fc4e4019030 con 0x7fc4f4158fe0 2024-03-24T00:42:55.067 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:42:55.067 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5627,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2941726873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2941726873},{"type":"v1","addr":"172.21.15.72:6837","nonce":2941726873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":6166,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/1957901519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":1957901519},{"type":"v1","addr":"172.21.15.72:6836","nonce":1957901519}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":6357,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/2843754917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2843754917},{"type":"v1","addr":"172.21.15.67:6833","nonce":2843754917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:41:42.242211+0000","modified":"2024-03-24T00:42:22.910300+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6360},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6360":{"gid":6360,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":13,"addr":"172.21.15.67:6835/451962260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":451962260},{"type":"v1","addr":"172.21.15.67:6835","nonce":451962260}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6360,"qdb_cluster":[6360]},"id":4}]} 2024-03-24T00:42:55.067 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 34 2024-03-24T00:42:55.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.068+0000 7fc4fa636640 1 -- 172.21.15.67:0/1417833037 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc4d406bc00 msgr2=0x7fc4d406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:55.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.068+0000 7fc4fa636640 1 --2- 172.21.15.67:0/1417833037 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc4d406bc00 0x7fc4d406e0c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc036000 tx=0x7fc4dc030000 comp rx=0 tx=0).stop 2024-03-24T00:42:55.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.068+0000 7fc4fa636640 1 -- 172.21.15.67:0/1417833037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4f4158fe0 msgr2=0x7fc4f40b3810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:55.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.068+0000 7fc4fa636640 1 --2- 172.21.15.67:0/1417833037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4f4158fe0 0x7fc4f40b3810 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4011ca0 tx=0x7fc4e400f610 comp rx=0 tx=0).stop 2024-03-24T00:42:55.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.068+0000 7fc4fa636640 1 -- 172.21.15.67:0/1417833037 shutdown_connections 2024-03-24T00:42:55.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.068+0000 7fc4fa636640 1 --2- 172.21.15.67:0/1417833037 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc4d406bc00 0x7fc4d406e0c0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:55.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.068+0000 7fc4fa636640 1 --2- 172.21.15.67:0/1417833037 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc4f40b3d50 0x7fc4f415c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:55.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.068+0000 7fc4fa636640 1 --2- 172.21.15.67:0/1417833037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4f4158fe0 0x7fc4f40b3810 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:55.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.068+0000 7fc4fa636640 1 --2- 172.21.15.67:0/1417833037 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc4e00039b0 0x7fc4f40cbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:55.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.068+0000 7fc4fa636640 1 -- 172.21.15.67:0/1417833037 >> 172.21.15.67:0/1417833037 conn(0x7fc4f406c810 msgr2=0x7fc4f4160670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:55.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.068+0000 7fc4fa636640 1 -- 172.21.15.67:0/1417833037 shutdown_connections 2024-03-24T00:42:55.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.068+0000 7fc4fa636640 1 -- 172.21.15.67:0/1417833037 wait complete. 2024-03-24T00:42:55.080 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:42:55.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.180+0000 7fd94ef5b640 1 Processor -- start 2024-03-24T00:42:55.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.180+0000 7fd94ef5b640 1 -- start start 2024-03-24T00:42:55.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.181+0000 7fd94ef5b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd94815dfb0 0x7fd9480cfe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:55.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.181+0000 7fd94ef5b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd94815e390 0x7fd94806b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:55.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.181+0000 7fd94ef5b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9480d07d0 con 0x7fd9480cded0 2024-03-24T00:42:55.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.181+0000 7fd94ef5b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9480d0ab0 con 0x7fd94815dfb0 2024-03-24T00:42:55.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.181+0000 7fd94ef5b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9480d0d90 con 0x7fd94815e390 2024-03-24T00:42:55.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.181+0000 7fd94ccf7640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd94815dfb0 0x7fd9480cfe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:55.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.181+0000 7fd94ccf7640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd94815dfb0 0x7fd9480cfe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42792/0 (socket says 172.21.15.67:42792) 2024-03-24T00:42:55.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.181+0000 7fd94ccf7640 1 -- 172.21.15.67:0/870022359 learned_addr learned my addr 172.21.15.67:0/870022359 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:55.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.181+0000 7fd947fff640 1 --2- 172.21.15.67:0/870022359 >> v2:172.21.15.72:3300/0 conn(0x7fd94815e390 0x7fd94806b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:55.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.181+0000 7fd94ccf7640 1 -- 172.21.15.67:0/870022359 >> v2:172.21.15.72:3300/0 conn(0x7fd94815e390 msgr2=0x7fd94806b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:55.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.181+0000 7fd94ccf7640 1 --2- 172.21.15.67:0/870022359 >> v2:172.21.15.72:3300/0 conn(0x7fd94815e390 0x7fd94806b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:55.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.181+0000 7fd94ccf7640 1 -- 172.21.15.67:0/870022359 >> v1:172.21.15.72:6789/0 conn(0x7fd9480cded0 legacy=0x7fd948161d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:55.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.181+0000 7fd94ccf7640 1 -- 172.21.15.67:0/870022359 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9480d1070 con 0x7fd94815dfb0 2024-03-24T00:42:55.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.182+0000 7fd94ccf7640 1 --2- 172.21.15.67:0/870022359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd94815dfb0 0x7fd9480cfe80 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fd9480b35f0 tx=0x7fd93c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f63ddf645736cf45 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:55.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.182+0000 7fd9477fe640 1 -- 172.21.15.67:0/870022359 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd93c034070 con 0x7fd94815dfb0 2024-03-24T00:42:55.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.182+0000 7fd9477fe640 1 -- 172.21.15.67:0/870022359 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd93c030860 con 0x7fd94815dfb0 2024-03-24T00:42:55.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.182+0000 7fd9477fe640 1 -- 172.21.15.67:0/870022359 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd93c030c30 con 0x7fd94815dfb0 2024-03-24T00:42:55.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.183+0000 7fd94ef5b640 1 -- 172.21.15.67:0/870022359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd94815dfb0 msgr2=0x7fd9480cfe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:55.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.183+0000 7fd94ef5b640 1 --2- 172.21.15.67:0/870022359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd94815dfb0 0x7fd9480cfe80 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fd9480b35f0 tx=0x7fd93c02b9a0 comp rx=0 tx=0).stop 2024-03-24T00:42:55.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.183+0000 7fd94ef5b640 1 -- 172.21.15.67:0/870022359 shutdown_connections 2024-03-24T00:42:55.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.183+0000 7fd94ef5b640 1 --2- 172.21.15.67:0/870022359 >> v2:172.21.15.72:3300/0 conn(0x7fd94815e390 0x7fd94806b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:55.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.183+0000 7fd94ef5b640 1 --2- 172.21.15.67:0/870022359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd94815dfb0 0x7fd9480cfe80 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:55.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.183+0000 7fd94ef5b640 1 -- 172.21.15.67:0/870022359 >> 172.21.15.67:0/870022359 conn(0x7fd948099a30 msgr2=0x7fd948099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:55.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.183+0000 7fd94ef5b640 1 -- 172.21.15.67:0/870022359 shutdown_connections 2024-03-24T00:42:55.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.183+0000 7fd94ef5b640 1 -- 172.21.15.67:0/870022359 wait complete. 2024-03-24T00:42:55.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.184+0000 7fd94ef5b640 1 Processor -- start 2024-03-24T00:42:55.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.184+0000 7fd94ef5b640 1 -- start start 2024-03-24T00:42:55.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.185+0000 7fd94ef5b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd9480cded0 0x7fd94814ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:55.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.185+0000 7fd94ef5b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd94815dfb0 0x7fd94814df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:55.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.185+0000 7fd94ef5b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd94815e390 0x7fd948159c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:55.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.185+0000 7fd94ef5b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9480d1870 con 0x7fd94815e390 2024-03-24T00:42:55.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.185+0000 7fd94ef5b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9480cc130 con 0x7fd94815dfb0 2024-03-24T00:42:55.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.185+0000 7fd94ef5b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9480d03c0 con 0x7fd9480cded0 2024-03-24T00:42:55.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.185+0000 7fd94d4f8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd94815e390 0x7fd948159c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:55.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.185+0000 7fd94d4f8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd94815e390 0x7fd948159c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42804/0 (socket says 172.21.15.67:42804) 2024-03-24T00:42:55.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.185+0000 7fd94d4f8640 1 -- 172.21.15.67:0/3483780912 learned_addr learned my addr 172.21.15.67:0/3483780912 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:55.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.185+0000 7fd947fff640 1 --2- 172.21.15.67:0/3483780912 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd94815dfb0 0x7fd94814df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:55.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.185+0000 7fd94ccf7640 1 --2- 172.21.15.67:0/3483780912 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd9480cded0 0x7fd94814ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:55.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.185+0000 7fd94d4f8640 1 -- 172.21.15.67:0/3483780912 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd9480cded0 msgr2=0x7fd94814ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:55.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.185+0000 7fd94d4f8640 1 --2- 172.21.15.67:0/3483780912 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd9480cded0 0x7fd94814ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:55.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.185+0000 7fd94d4f8640 1 -- 172.21.15.67:0/3483780912 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd94815dfb0 msgr2=0x7fd94814df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:55.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.185+0000 7fd94d4f8640 1 --2- 172.21.15.67:0/3483780912 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd94815dfb0 0x7fd94814df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:55.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.185+0000 7fd94d4f8640 1 -- 172.21.15.67:0/3483780912 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd94815a1c0 con 0x7fd94815e390 2024-03-24T00:42:55.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.185+0000 7fd94d4f8640 1 --2- 172.21.15.67:0/3483780912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd94815e390 0x7fd948159c80 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fd94000c7a0 tx=0x7fd94000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:55.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.185+0000 7fd9457fa640 1 -- 172.21.15.67:0/3483780912 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd940003b70 con 0x7fd94815e390 2024-03-24T00:42:55.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.185+0000 7fd9457fa640 1 -- 172.21.15.67:0/3483780912 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd940003d30 con 0x7fd94815e390 2024-03-24T00:42:55.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.185+0000 7fd94ef5b640 1 -- 172.21.15.67:0/3483780912 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd94815a440 con 0x7fd94815e390 2024-03-24T00:42:55.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.185+0000 7fd9457fa640 1 -- 172.21.15.67:0/3483780912 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd94000aa90 con 0x7fd94815e390 2024-03-24T00:42:55.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.186+0000 7fd94ef5b640 1 -- 172.21.15.67:0/3483780912 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd94815aa00 con 0x7fd94815e390 2024-03-24T00:42:55.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.186+0000 7fd94ef5b640 1 -- 172.21.15.67:0/3483780912 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd94804f400 con 0x7fd94815e390 2024-03-24T00:42:55.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.187+0000 7fd9457fa640 1 -- 172.21.15.67:0/3483780912 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd94000ac30 con 0x7fd94815e390 2024-03-24T00:42:55.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.188+0000 7fd9457fa640 1 --2- 172.21.15.67:0/3483780912 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd92c06bc00 0x7fd92c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:55.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.188+0000 7fd9457fa640 1 -- 172.21.15.67:0/3483780912 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6246+0+0 (secure 0 0 0) 0x7fd94008abe0 con 0x7fd94815e390 2024-03-24T00:42:55.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.188+0000 7fd94ccf7640 1 --2- 172.21.15.67:0/3483780912 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd92c06bc00 0x7fd92c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:55.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.188+0000 7fd94ccf7640 1 --2- 172.21.15.67:0/3483780912 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd92c06bc00 0x7fd92c06e0c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fd93c036000 tx=0x7fd93c03b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:55.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.192+0000 7fd9457fa640 1 -- 172.21.15.67:0/3483780912 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd94815a1c0 con 0x7fd94815e390 2024-03-24T00:42:55.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.373+0000 7fd94ef5b640 1 -- 172.21.15.67:0/3483780912 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd9480b6a50 con 0x7fd94815e390 2024-03-24T00:42:55.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.374+0000 7fd9457fa640 1 -- 172.21.15.67:0/3483780912 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) v1 ==== 74+0+17064 (secure 0 0 0) 0x7fd94005de20 con 0x7fd94815e390 2024-03-24T00:42:55.375 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:42:55.376 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":64,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:42:33.971493+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:41:40.502194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-03-24T00:41:41.200073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"9.7","mappings":[{"from":7,"to":2}]},{"pgid":"10.17","mappings":[{"from":4,"to":5}]},{"pgid":"10.1d","mappings":[{"from":4,"to":3}]},{"pgid":"10.2e","mappings":[{"from":4,"to":3}]},{"pgid":"10.3a","mappings":[{"from":7,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:42:55.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.376+0000 7fd94ef5b640 1 -- 172.21.15.67:0/3483780912 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd92c06bc00 msgr2=0x7fd92c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:55.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.376+0000 7fd94ef5b640 1 --2- 172.21.15.67:0/3483780912 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd92c06bc00 0x7fd92c06e0c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fd93c036000 tx=0x7fd93c03b040 comp rx=0 tx=0).stop 2024-03-24T00:42:55.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.376+0000 7fd94ef5b640 1 -- 172.21.15.67:0/3483780912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd94815e390 msgr2=0x7fd948159c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:55.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.376+0000 7fd94ef5b640 1 --2- 172.21.15.67:0/3483780912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd94815e390 0x7fd948159c80 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fd94000c7a0 tx=0x7fd94000cc60 comp rx=0 tx=0).stop 2024-03-24T00:42:55.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.377+0000 7fd94ef5b640 1 -- 172.21.15.67:0/3483780912 shutdown_connections 2024-03-24T00:42:55.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.377+0000 7fd94ef5b640 1 --2- 172.21.15.67:0/3483780912 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd92c06bc00 0x7fd92c06e0c0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:55.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.377+0000 7fd94ef5b640 1 --2- 172.21.15.67:0/3483780912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd94815e390 0x7fd948159c80 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:55.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.377+0000 7fd94ef5b640 1 --2- 172.21.15.67:0/3483780912 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd94815dfb0 0x7fd94814df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:55.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.377+0000 7fd94ef5b640 1 --2- 172.21.15.67:0/3483780912 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd9480cded0 0x7fd94814ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:55.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.377+0000 7fd94ef5b640 1 -- 172.21.15.67:0/3483780912 >> 172.21.15.67:0/3483780912 conn(0x7fd948099a30 msgr2=0x7fd9480b36f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:55.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.377+0000 7fd94ef5b640 1 -- 172.21.15.67:0/3483780912 shutdown_connections 2024-03-24T00:42:55.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.377+0000 7fd94ef5b640 1 -- 172.21.15.67:0/3483780912 wait complete. 2024-03-24T00:42:55.388 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:38:39.364950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:41:40.502194+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:41:41.200073+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:42:55.389 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:42:55.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.495+0000 7fd9ac15b640 1 Processor -- start 2024-03-24T00:42:55.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.495+0000 7fd9ac15b640 1 -- start start 2024-03-24T00:42:55.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.496+0000 7fd9ac15b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd9a40d3ae0 0x7fd9a40b5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:55.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.496+0000 7fd9ac15b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9a40d2250 con 0x7fd9a415e280 2024-03-24T00:42:55.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.496+0000 7fd9ac15b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9a40d2530 con 0x7fd9a415dea0 2024-03-24T00:42:55.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.496+0000 7fd9ac15b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9a40d2810 con 0x7fd9a40d3ae0 2024-03-24T00:42:55.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.496+0000 7fd9aa6f8640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd9a415e280 0x7fd9a40b70f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60926/0 (socket says 172.21.15.67:60926) 2024-03-24T00:42:55.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.496+0000 7fd9aa6f8640 1 -- 172.21.15.67:0/1416318009 learned_addr learned my addr 172.21.15.67:0/1416318009 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:42:55.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.496+0000 7fd9a8ef5640 1 -- 172.21.15.67:0/1416318009 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2866027012 0 0) 0x7fd9a40d2250 con 0x7fd9a415e280 2024-03-24T00:42:55.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.496+0000 7fd9a8ef5640 1 -- 172.21.15.67:0/1416318009 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd990003520 con 0x7fd9a415e280 2024-03-24T00:42:55.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.496+0000 7fd9a96f6640 1 --2- 172.21.15.67:0/1416318009 >> v2:172.21.15.72:3300/0 conn(0x7fd9a40d3ae0 0x7fd9a40b5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:55.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.496+0000 7fd9a8ef5640 1 -- 172.21.15.67:0/1416318009 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3302740541 0 0) 0x7fd990003520 con 0x7fd9a415e280 2024-03-24T00:42:55.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.496+0000 7fd9a8ef5640 1 -- 172.21.15.67:0/1416318009 >> v2:172.21.15.72:3300/0 conn(0x7fd9a40d3ae0 msgr2=0x7fd9a40b5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:55.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.496+0000 7fd9a8ef5640 1 --2- 172.21.15.67:0/1416318009 >> v2:172.21.15.72:3300/0 conn(0x7fd9a40d3ae0 0x7fd9a40b5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:55.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.496+0000 7fd9a8ef5640 1 -- 172.21.15.67:0/1416318009 >> v1:172.21.15.72:6789/0 conn(0x7fd9a415dea0 legacy=0x7fd9a40d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:55.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.496+0000 7fd9a8ef5640 1 -- 172.21.15.67:0/1416318009 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a40d2af0 con 0x7fd9a415e280 2024-03-24T00:42:55.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.496+0000 7fd9a8ef5640 1 -- 172.21.15.67:0/1416318009 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7fd99c001670 con 0x7fd9a415e280 2024-03-24T00:42:55.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.497+0000 7fd9a8ef5640 1 -- 172.21.15.67:0/1416318009 >> v1:172.21.15.67:6789/0 conn(0x7fd9a415e280 legacy=0x7fd9a40b70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:55.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.497+0000 7fd9a8ef5640 1 --2- 172.21.15.67:0/1416318009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd990003a30 0x7fd990003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:55.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.497+0000 7fd9a8ef5640 1 --2- 172.21.15.67:0/1416318009 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd990004800 0x7fd990004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:55.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.497+0000 7fd9a8ef5640 1 --2- 172.21.15.67:0/1416318009 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd990005230 0x7fd99000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:55.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.497+0000 7fd9a8ef5640 1 -- 172.21.15.67:0/1416318009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a40d2810 con 0x7fd990003a30 2024-03-24T00:42:55.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.497+0000 7fd9a8ef5640 1 -- 172.21.15.67:0/1416318009 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd99000c190 con 0x7fd990005230 2024-03-24T00:42:55.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.497+0000 7fd9a8ef5640 1 -- 172.21.15.67:0/1416318009 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd99000c390 con 0x7fd990004800 2024-03-24T00:42:55.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.497+0000 7fd9a9ef7640 1 --2- 172.21.15.67:0/1416318009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd990003a30 0x7fd990003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:55.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.497+0000 7fd9aa6f8640 1 --2- 172.21.15.67:0/1416318009 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd990005230 0x7fd99000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:55.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.497+0000 7fd9a96f6640 1 --2- 172.21.15.67:0/1416318009 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd990004800 0x7fd990004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:55.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.497+0000 7fd9a9ef7640 1 -- 172.21.15.67:0/1416318009 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd990004800 msgr2=0x7fd990004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:55.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.497+0000 7fd9a9ef7640 1 --2- 172.21.15.67:0/1416318009 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd990004800 0x7fd990004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:55.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.497+0000 7fd9a9ef7640 1 -- 172.21.15.67:0/1416318009 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd990005230 msgr2=0x7fd99000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:55.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.497+0000 7fd9a9ef7640 1 --2- 172.21.15.67:0/1416318009 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd990005230 0x7fd99000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:55.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.497+0000 7fd9a9ef7640 1 -- 172.21.15.67:0/1416318009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd990003520 con 0x7fd990003a30 2024-03-24T00:42:55.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.497+0000 7fd9a96f6640 1 --2- 172.21.15.67:0/1416318009 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd990004800 0x7fd990004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:42:55.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.497+0000 7fd9aa6f8640 1 --2- 172.21.15.67:0/1416318009 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd990005230 0x7fd99000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:42:55.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.497+0000 7fd9aa6f8640 1 -- 172.21.15.67:0/1416318009 reap_dead start 2024-03-24T00:42:55.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.498+0000 7fd9a9ef7640 1 --2- 172.21.15.67:0/1416318009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd990003a30 0x7fd990003e50 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fd9a0001e50 tx=0x7fd990005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e333cb292e09c61 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:55.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.498+0000 7fd9a8ef5640 1 -- 172.21.15.67:0/1416318009 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a0001a70 con 0x7fd990003a30 2024-03-24T00:42:55.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.498+0000 7fd9a8ef5640 1 -- 172.21.15.67:0/1416318009 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9a0003b70 con 0x7fd990003a30 2024-03-24T00:42:55.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.498+0000 7fd9a8ef5640 1 -- 172.21.15.67:0/1416318009 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a00050d0 con 0x7fd990003a30 2024-03-24T00:42:55.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.498+0000 7fd9ac15b640 1 -- 172.21.15.67:0/1416318009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd990003a30 msgr2=0x7fd990003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:55.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.498+0000 7fd9ac15b640 1 --2- 172.21.15.67:0/1416318009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd990003a30 0x7fd990003e50 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fd9a0001e50 tx=0x7fd990005830 comp rx=0 tx=0).stop 2024-03-24T00:42:55.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.498+0000 7fd9ac15b640 1 -- 172.21.15.67:0/1416318009 shutdown_connections 2024-03-24T00:42:55.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.498+0000 7fd9ac15b640 1 --2- 172.21.15.67:0/1416318009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd990003a30 0x7fd990003e50 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:55.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.498+0000 7fd9ac15b640 1 -- 172.21.15.67:0/1416318009 >> 172.21.15.67:0/1416318009 conn(0x7fd9a4072da0 msgr2=0x7fd9a40d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:55.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.499+0000 7fd9ac15b640 1 -- 172.21.15.67:0/1416318009 shutdown_connections 2024-03-24T00:42:55.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.499+0000 7fd9ac15b640 1 -- 172.21.15.67:0/1416318009 wait complete. 2024-03-24T00:42:55.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.500+0000 7fd9ac15b640 1 Processor -- start 2024-03-24T00:42:55.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.500+0000 7fd9ac15b640 1 -- start start 2024-03-24T00:42:55.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.500+0000 7fd9ac15b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd990003a30 0x7fd9a40b7d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:55.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.500+0000 7fd9ac15b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9a41f7d40 0x7fd9a41f81c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:55.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.500+0000 7fd9ac15b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd9a40aeaa0 0x7fd9a40aef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:55.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.500+0000 7fd9ac15b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a40d3760 con 0x7fd9a41f7d40 2024-03-24T00:42:55.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.500+0000 7fd9ac15b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a40d35e0 con 0x7fd990003a30 2024-03-24T00:42:55.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.500+0000 7fd9ac15b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a40d38e0 con 0x7fd9a40aeaa0 2024-03-24T00:42:55.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.501+0000 7fd9a96f6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9a41f7d40 0x7fd9a41f81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:55.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.501+0000 7fd9a96f6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9a41f7d40 0x7fd9a41f81c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42818/0 (socket says 172.21.15.67:42818) 2024-03-24T00:42:55.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.501+0000 7fd9a96f6640 1 -- 172.21.15.67:0/478165506 learned_addr learned my addr 172.21.15.67:0/478165506 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:55.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.501+0000 7fd9a9ef7640 1 --2- 172.21.15.67:0/478165506 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd990003a30 0x7fd9a40b7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:55.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.501+0000 7fd9aa6f8640 1 --2- 172.21.15.67:0/478165506 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd9a40aeaa0 0x7fd9a40aef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:55.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.501+0000 7fd9a96f6640 1 -- 172.21.15.67:0/478165506 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd9a40aeaa0 msgr2=0x7fd9a40aef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:55.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.501+0000 7fd9a96f6640 1 --2- 172.21.15.67:0/478165506 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd9a40aeaa0 0x7fd9a40aef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:55.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.501+0000 7fd9a96f6640 1 -- 172.21.15.67:0/478165506 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd990003a30 msgr2=0x7fd9a40b7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:55.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.501+0000 7fd9a96f6640 1 --2- 172.21.15.67:0/478165506 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd990003a30 0x7fd9a40b7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:55.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.501+0000 7fd9a96f6640 1 -- 172.21.15.67:0/478165506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a4150f70 con 0x7fd9a41f7d40 2024-03-24T00:42:55.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.501+0000 7fd9a96f6640 1 --2- 172.21.15.67:0/478165506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9a41f7d40 0x7fd9a41f81c0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fd9940114e0 tx=0x7fd9940119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:55.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.501+0000 7fd99a7fc640 1 -- 172.21.15.67:0/478165506 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd994015870 con 0x7fd9a41f7d40 2024-03-24T00:42:55.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.501+0000 7fd99a7fc640 1 -- 172.21.15.67:0/478165506 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd994015a10 con 0x7fd9a41f7d40 2024-03-24T00:42:55.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.501+0000 7fd9ac15b640 1 -- 172.21.15.67:0/478165506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9a4151250 con 0x7fd9a41f7d40 2024-03-24T00:42:55.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.501+0000 7fd99a7fc640 1 -- 172.21.15.67:0/478165506 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd994015e00 con 0x7fd9a41f7d40 2024-03-24T00:42:55.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.501+0000 7fd9ac15b640 1 -- 172.21.15.67:0/478165506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9a4151810 con 0x7fd9a41f7d40 2024-03-24T00:42:55.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.502+0000 7fd9ac15b640 1 -- 172.21.15.67:0/478165506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9a404f400 con 0x7fd9a41f7d40 2024-03-24T00:42:55.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.503+0000 7fd99a7fc640 1 -- 172.21.15.67:0/478165506 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd99401d400 con 0x7fd9a41f7d40 2024-03-24T00:42:55.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.503+0000 7fd99a7fc640 1 --2- 172.21.15.67:0/478165506 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd98806bc00 0x7fd98806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:55.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.503+0000 7fd99a7fc640 1 -- 172.21.15.67:0/478165506 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6246+0+0 (secure 0 0 0) 0x7fd99408b310 con 0x7fd9a41f7d40 2024-03-24T00:42:55.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.503+0000 7fd9a9ef7640 1 --2- 172.21.15.67:0/478165506 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd98806bc00 0x7fd98806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:55.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.504+0000 7fd9a9ef7640 1 --2- 172.21.15.67:0/478165506 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd98806bc00 0x7fd98806e0c0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fd9a0034000 tx=0x7fd9a003b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:55.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.507+0000 7fd99a7fc640 1 -- 172.21.15.67:0/478165506 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd994019030 con 0x7fd9a41f7d40 2024-03-24T00:42:55.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.690+0000 7fd9ac15b640 1 -- 172.21.15.67:0/478165506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd9a40b7820 con 0x7fd9a41f7d40 2024-03-24T00:42:55.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.690+0000 7fd99a7fc640 1 -- 172.21.15.67:0/478165506 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v64) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd994059580 con 0x7fd9a41f7d40 2024-03-24T00:42:55.691 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-24T00:42:55.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.692+0000 7fd9ac15b640 1 -- 172.21.15.67:0/478165506 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd98806bc00 msgr2=0x7fd98806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:55.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.692+0000 7fd9ac15b640 1 --2- 172.21.15.67:0/478165506 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd98806bc00 0x7fd98806e0c0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fd9a0034000 tx=0x7fd9a003b040 comp rx=0 tx=0).stop 2024-03-24T00:42:55.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.692+0000 7fd9ac15b640 1 -- 172.21.15.67:0/478165506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9a41f7d40 msgr2=0x7fd9a41f81c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:55.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.692+0000 7fd9ac15b640 1 --2- 172.21.15.67:0/478165506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9a41f7d40 0x7fd9a41f81c0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fd9940114e0 tx=0x7fd9940119a0 comp rx=0 tx=0).stop 2024-03-24T00:42:55.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.692+0000 7fd9ac15b640 1 -- 172.21.15.67:0/478165506 shutdown_connections 2024-03-24T00:42:55.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.692+0000 7fd9ac15b640 1 --2- 172.21.15.67:0/478165506 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd98806bc00 0x7fd98806e0c0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:55.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.692+0000 7fd9ac15b640 1 --2- 172.21.15.67:0/478165506 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd9a40aeaa0 0x7fd9a40aef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:55.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.692+0000 7fd9ac15b640 1 --2- 172.21.15.67:0/478165506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9a41f7d40 0x7fd9a41f81c0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:55.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.692+0000 7fd9ac15b640 1 --2- 172.21.15.67:0/478165506 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd990003a30 0x7fd9a40b7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:55.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.692+0000 7fd9ac15b640 1 -- 172.21.15.67:0/478165506 >> 172.21.15.67:0/478165506 conn(0x7fd9a4072da0 msgr2=0x7fd9a40d0a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:55.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.692+0000 7fd9ac15b640 1 -- 172.21.15.67:0/478165506 shutdown_connections 2024-03-24T00:42:55.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.692+0000 7fd9ac15b640 1 -- 172.21.15.67:0/478165506 wait complete. 2024-03-24T00:42:55.704 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:42:55.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.810+0000 7f1534c4b640 1 Processor -- start 2024-03-24T00:42:55.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.810+0000 7f1534c4b640 1 -- start start 2024-03-24T00:42:55.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.811+0000 7f1534c4b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f153015dea0 0x7f15300d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:55.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.811+0000 7f1534c4b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15300d2250 con 0x7f153015e280 2024-03-24T00:42:55.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.811+0000 7f1534c4b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15300d2530 con 0x7f15300b6170 2024-03-24T00:42:55.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.811+0000 7f1534c4b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15300d2810 con 0x7f153015dea0 2024-03-24T00:42:55.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.811+0000 7f152e59c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f153015dea0 0x7f15300d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:55.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.811+0000 7f152dd9b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f153015e280 0x7f15300b5a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60940/0 (socket says 172.21.15.67:60940) 2024-03-24T00:42:55.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.811+0000 7f152dd9b640 1 -- 172.21.15.67:0/2956664947 learned_addr learned my addr 172.21.15.67:0/2956664947 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:42:55.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.812+0000 7f152d59a640 1 -- 172.21.15.67:0/2956664947 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1830257109 0 0) 0x7f15300d2250 con 0x7f153015e280 2024-03-24T00:42:55.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.812+0000 7f152d59a640 1 -- 172.21.15.67:0/2956664947 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1518003520 con 0x7f153015e280 2024-03-24T00:42:55.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.812+0000 7f152e59c640 1 -- 172.21.15.67:0/2956664947 >> v1:172.21.15.72:6789/0 conn(0x7f15300b6170 legacy=0x7f15300b70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:55.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.812+0000 7f152e59c640 1 -- 172.21.15.67:0/2956664947 >> v1:172.21.15.67:6789/0 conn(0x7f153015e280 legacy=0x7f15300b5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:55.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.812+0000 7f152e59c640 1 -- 172.21.15.67:0/2956664947 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15300d2af0 con 0x7f153015dea0 2024-03-24T00:42:55.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.812+0000 7f152e59c640 1 --2- 172.21.15.67:0/2956664947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f153015dea0 0x7f15300d33d0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f15200097d0 tx=0x7f152002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5687df250a77cf0d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:55.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.813+0000 7f152d59a640 1 -- 172.21.15.67:0/2956664947 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1520035020 con 0x7f153015dea0 2024-03-24T00:42:55.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.813+0000 7f152d59a640 1 -- 172.21.15.67:0/2956664947 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1520031830 con 0x7f153015dea0 2024-03-24T00:42:55.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.813+0000 7f152d59a640 1 -- 172.21.15.67:0/2956664947 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1520031be0 con 0x7f153015dea0 2024-03-24T00:42:55.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.813+0000 7f1534c4b640 1 -- 172.21.15.67:0/2956664947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f153015dea0 msgr2=0x7f15300d33d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:55.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.813+0000 7f1534c4b640 1 --2- 172.21.15.67:0/2956664947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f153015dea0 0x7f15300d33d0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f15200097d0 tx=0x7f152002b8d0 comp rx=0 tx=0).stop 2024-03-24T00:42:55.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.813+0000 7f1534c4b640 1 -- 172.21.15.67:0/2956664947 shutdown_connections 2024-03-24T00:42:55.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.813+0000 7f1534c4b640 1 --2- 172.21.15.67:0/2956664947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f153015dea0 0x7f15300d33d0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:55.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.813+0000 7f1534c4b640 1 -- 172.21.15.67:0/2956664947 >> 172.21.15.67:0/2956664947 conn(0x7f1530072da0 msgr2=0x7f15300d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:55.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.813+0000 7f1534c4b640 1 -- 172.21.15.67:0/2956664947 shutdown_connections 2024-03-24T00:42:55.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.813+0000 7f1534c4b640 1 -- 172.21.15.67:0/2956664947 wait complete. 2024-03-24T00:42:55.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.814+0000 7f1534c4b640 1 Processor -- start 2024-03-24T00:42:55.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.815+0000 7f1534c4b640 1 -- start start 2024-03-24T00:42:55.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.815+0000 7f1534c4b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f15300b6170 0x7f153015b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:55.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.815+0000 7f1534c4b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f153015dea0 0x7f15301f47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:55.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.815+0000 7f1534c4b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f153015e280 0x7f1530160750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:55.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.815+0000 7f1534c4b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15300d3ca0 con 0x7f153015e280 2024-03-24T00:42:55.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.815+0000 7f1534c4b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15300d3b20 con 0x7f15300b6170 2024-03-24T00:42:55.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.815+0000 7f1534c4b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15300b7800 con 0x7f153015dea0 2024-03-24T00:42:55.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.815+0000 7f152ed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f153015e280 0x7f1530160750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:55.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.815+0000 7f152ed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f153015e280 0x7f1530160750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42840/0 (socket says 172.21.15.67:42840) 2024-03-24T00:42:55.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.815+0000 7f152ed9d640 1 -- 172.21.15.67:0/2887598044 learned_addr learned my addr 172.21.15.67:0/2887598044 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:55.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.815+0000 7f152e59c640 1 --2- 172.21.15.67:0/2887598044 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f15300b6170 0x7f153015b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:55.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.816+0000 7f152dd9b640 1 --2- 172.21.15.67:0/2887598044 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f153015dea0 0x7f15301f47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:55.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.816+0000 7f152ed9d640 1 -- 172.21.15.67:0/2887598044 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f153015dea0 msgr2=0x7f15301f47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:55.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.816+0000 7f152ed9d640 1 --2- 172.21.15.67:0/2887598044 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f153015dea0 0x7f15301f47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:55.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.816+0000 7f152ed9d640 1 -- 172.21.15.67:0/2887598044 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f15300b6170 msgr2=0x7f153015b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:55.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.816+0000 7f152ed9d640 1 --2- 172.21.15.67:0/2887598044 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f15300b6170 0x7f153015b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:55.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.816+0000 7f152ed9d640 1 -- 172.21.15.67:0/2887598044 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15300cc7f0 con 0x7f153015e280 2024-03-24T00:42:55.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.816+0000 7f152dd9b640 1 --2- 172.21.15.67:0/2887598044 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f153015dea0 0x7f15301f47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:42:55.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.816+0000 7f152ed9d640 1 --2- 172.21.15.67:0/2887598044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f153015e280 0x7f1530160750 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f152800c730 tx=0x7f152800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:55.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.816+0000 7f151effd640 1 -- 172.21.15.67:0/2887598044 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1528003b70 con 0x7f153015e280 2024-03-24T00:42:55.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.816+0000 7f151effd640 1 -- 172.21.15.67:0/2887598044 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1528003d30 con 0x7f153015e280 2024-03-24T00:42:55.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.816+0000 7f151effd640 1 -- 172.21.15.67:0/2887598044 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1528005310 con 0x7f153015e280 2024-03-24T00:42:55.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.816+0000 7f1534c4b640 1 -- 172.21.15.67:0/2887598044 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15300ccad0 con 0x7f153015e280 2024-03-24T00:42:55.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.816+0000 7f1534c4b640 1 -- 172.21.15.67:0/2887598044 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15300cd090 con 0x7f153015e280 2024-03-24T00:42:55.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.817+0000 7f151cff9640 1 -- 172.21.15.67:0/2887598044 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1508005290 con 0x7f153015e280 2024-03-24T00:42:55.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.818+0000 7f151effd640 1 -- 172.21.15.67:0/2887598044 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f15280056c0 con 0x7f153015e280 2024-03-24T00:42:55.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.818+0000 7f151effd640 1 --2- 172.21.15.67:0/2887598044 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f151006bbd0 0x7f151006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:55.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.818+0000 7f151effd640 1 -- 172.21.15.67:0/2887598044 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6246+0+0 (secure 0 0 0) 0x7f152808ac60 con 0x7f153015e280 2024-03-24T00:42:55.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.818+0000 7f152e59c640 1 --2- 172.21.15.67:0/2887598044 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f151006bbd0 0x7f151006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:55.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.819+0000 7f152e59c640 1 --2- 172.21.15.67:0/2887598044 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f151006bbd0 0x7f151006e090 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f1520036000 tx=0x7f1520008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:55.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:55.822+0000 7f151effd640 1 -- 172.21.15.67:0/2887598044 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f15280595c0 con 0x7f153015e280 2024-03-24T00:42:56.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.003+0000 7f151cff9640 1 -- 172.21.15.67:0/2887598044 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1508005850 con 0x7f153015e280 2024-03-24T00:42:56.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.003+0000 7f151effd640 1 -- 172.21.15.67:0/2887598044 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v64) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1528058ec0 con 0x7f153015e280 2024-03-24T00:42:56.004 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-24T00:42:56.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.005+0000 7f151cff9640 1 -- 172.21.15.67:0/2887598044 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f151006bbd0 msgr2=0x7f151006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:56.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.005+0000 7f151cff9640 1 --2- 172.21.15.67:0/2887598044 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f151006bbd0 0x7f151006e090 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f1520036000 tx=0x7f1520008030 comp rx=0 tx=0).stop 2024-03-24T00:42:56.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.005+0000 7f151cff9640 1 -- 172.21.15.67:0/2887598044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f153015e280 msgr2=0x7f1530160750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:56.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.005+0000 7f151cff9640 1 --2- 172.21.15.67:0/2887598044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f153015e280 0x7f1530160750 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f152800c730 tx=0x7f152800cbf0 comp rx=0 tx=0).stop 2024-03-24T00:42:56.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.005+0000 7f151cff9640 1 -- 172.21.15.67:0/2887598044 shutdown_connections 2024-03-24T00:42:56.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.005+0000 7f151cff9640 1 --2- 172.21.15.67:0/2887598044 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f151006bbd0 0x7f151006e090 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:56.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.005+0000 7f151cff9640 1 --2- 172.21.15.67:0/2887598044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f153015e280 0x7f1530160750 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:56.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.005+0000 7f151cff9640 1 --2- 172.21.15.67:0/2887598044 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f153015dea0 0x7f15301f47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:56.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.005+0000 7f151cff9640 1 --2- 172.21.15.67:0/2887598044 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f15300b6170 0x7f153015b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:56.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.005+0000 7f151cff9640 1 -- 172.21.15.67:0/2887598044 >> 172.21.15.67:0/2887598044 conn(0x7f1530072da0 msgr2=0x7f15300cea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:56.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.005+0000 7f151cff9640 1 -- 172.21.15.67:0/2887598044 shutdown_connections 2024-03-24T00:42:56.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.005+0000 7f151cff9640 1 -- 172.21.15.67:0/2887598044 wait complete. 2024-03-24T00:42:56.017 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:42:56.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.120+0000 7f795d836640 1 Processor -- start 2024-03-24T00:42:56.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.120+0000 7f795d836640 1 -- start start 2024-03-24T00:42:56.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.121+0000 7f795d836640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f79580d18b0 0x7f795815c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:56.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.121+0000 7f795d836640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f795815c7c0 con 0x7f79580d1c90 2024-03-24T00:42:56.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.121+0000 7f795d836640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f795815caa0 con 0x7f7958160430 2024-03-24T00:42:56.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.121+0000 7f795d836640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f795815cd80 con 0x7f79580d18b0 2024-03-24T00:42:56.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.121+0000 7f7956ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f79580d18b0 0x7f795815c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:56.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.121+0000 7f794ffff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f79580d1c90 0x7f795815fd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60952/0 (socket says 172.21.15.67:60952) 2024-03-24T00:42:56.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.121+0000 7f794ffff640 1 -- 172.21.15.67:0/713732024 learned_addr learned my addr 172.21.15.67:0/713732024 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:42:56.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.122+0000 7f7956ffd640 1 -- 172.21.15.67:0/713732024 >> v1:172.21.15.72:6789/0 conn(0x7f7958160430 legacy=0x7f79580cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:56.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.122+0000 7f7956ffd640 1 -- 172.21.15.67:0/713732024 >> v1:172.21.15.67:6789/0 conn(0x7f79580d1c90 legacy=0x7f795815fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:56.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.122+0000 7f7956ffd640 1 -- 172.21.15.67:0/713732024 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f795815d060 con 0x7f79580d18b0 2024-03-24T00:42:56.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.123+0000 7f7956ffd640 1 --2- 172.21.15.67:0/713732024 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79580d18b0 0x7f795815c1b0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f79480097a0 tx=0x7f794802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca8fa2420d900d58 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:56.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.123+0000 7f79567fc640 1 -- 172.21.15.67:0/713732024 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7948035020 con 0x7f79580d18b0 2024-03-24T00:42:56.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.123+0000 7f79567fc640 1 -- 172.21.15.67:0/713732024 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7948031830 con 0x7f79580d18b0 2024-03-24T00:42:56.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.123+0000 7f79567fc640 1 -- 172.21.15.67:0/713732024 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7948031c00 con 0x7f79580d18b0 2024-03-24T00:42:56.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.123+0000 7f795d836640 1 -- 172.21.15.67:0/713732024 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79580d18b0 msgr2=0x7f795815c1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:56.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.123+0000 7f795d836640 1 --2- 172.21.15.67:0/713732024 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79580d18b0 0x7f795815c1b0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f79480097a0 tx=0x7f794802b8d0 comp rx=0 tx=0).stop 2024-03-24T00:42:56.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.123+0000 7f795d836640 1 -- 172.21.15.67:0/713732024 shutdown_connections 2024-03-24T00:42:56.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.123+0000 7f795d836640 1 --2- 172.21.15.67:0/713732024 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79580d18b0 0x7f795815c1b0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:56.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.123+0000 7f795d836640 1 -- 172.21.15.67:0/713732024 >> 172.21.15.67:0/713732024 conn(0x7f795808b1b0 msgr2=0x7f795808b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:56.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.123+0000 7f795d836640 1 -- 172.21.15.67:0/713732024 shutdown_connections 2024-03-24T00:42:56.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.123+0000 7f795d836640 1 -- 172.21.15.67:0/713732024 wait complete. 2024-03-24T00:42:56.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.124+0000 7f795d836640 1 Processor -- start 2024-03-24T00:42:56.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.124+0000 7f795d836640 1 -- start start 2024-03-24T00:42:56.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.125+0000 7f795d836640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79580d18b0 0x7f79580cb690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:56.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.125+0000 7f795d836640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79580d1c90 0x7f79580cd720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:56.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.125+0000 7f795d836640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7958160430 0x7f79580cdc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:56.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.125+0000 7f795d836640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79580d0060 con 0x7f79580d1c90 2024-03-24T00:42:56.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.125+0000 7f795d836640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79580cfee0 con 0x7f79580d18b0 2024-03-24T00:42:56.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.125+0000 7f795d836640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79580d01e0 con 0x7f7958160430 2024-03-24T00:42:56.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.125+0000 7f794ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79580d1c90 0x7f79580cd720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:56.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.125+0000 7f794ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79580d1c90 0x7f79580cd720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42856/0 (socket says 172.21.15.67:42856) 2024-03-24T00:42:56.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.125+0000 7f794ffff640 1 -- 172.21.15.67:0/3005145601 learned_addr learned my addr 172.21.15.67:0/3005145601 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:56.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.125+0000 7f7956ffd640 1 --2- 172.21.15.67:0/3005145601 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79580d18b0 0x7f79580cb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:56.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.125+0000 7f79577fe640 1 --2- 172.21.15.67:0/3005145601 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7958160430 0x7f79580cdc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:56.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.126+0000 7f794ffff640 1 -- 172.21.15.67:0/3005145601 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7958160430 msgr2=0x7f79580cdc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:56.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.126+0000 7f794ffff640 1 --2- 172.21.15.67:0/3005145601 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7958160430 0x7f79580cdc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:56.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.126+0000 7f794ffff640 1 -- 172.21.15.67:0/3005145601 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79580d18b0 msgr2=0x7f79580cb690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:56.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.126+0000 7f794ffff640 1 --2- 172.21.15.67:0/3005145601 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79580d18b0 0x7f79580cb690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:56.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.126+0000 7f794ffff640 1 -- 172.21.15.67:0/3005145601 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79580ca5f0 con 0x7f79580d1c90 2024-03-24T00:42:56.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.126+0000 7f794ffff640 1 --2- 172.21.15.67:0/3005145601 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79580d1c90 0x7f79580cd720 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f794000c500 tx=0x7f794000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:56.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.126+0000 7f7954ff9640 1 -- 172.21.15.67:0/3005145601 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7940017070 con 0x7f79580d1c90 2024-03-24T00:42:56.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.126+0000 7f7954ff9640 1 -- 172.21.15.67:0/3005145601 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7940005ce0 con 0x7f79580d1c90 2024-03-24T00:42:56.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.126+0000 7f795d836640 1 -- 172.21.15.67:0/3005145601 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79580ca8d0 con 0x7f79580d1c90 2024-03-24T00:42:56.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.126+0000 7f7954ff9640 1 -- 172.21.15.67:0/3005145601 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79400072f0 con 0x7f79580d1c90 2024-03-24T00:42:56.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.126+0000 7f795d836640 1 -- 172.21.15.67:0/3005145601 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79580cae90 con 0x7f79580d1c90 2024-03-24T00:42:56.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.127+0000 7f795d836640 1 -- 172.21.15.67:0/3005145601 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f795804f400 con 0x7f79580d1c90 2024-03-24T00:42:56.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.128+0000 7f7954ff9640 1 -- 172.21.15.67:0/3005145601 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f794001a070 con 0x7f79580d1c90 2024-03-24T00:42:56.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.128+0000 7f7954ff9640 1 --2- 172.21.15.67:0/3005145601 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f793806bbd0 0x7f793806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:56.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.129+0000 7f7954ff9640 1 -- 172.21.15.67:0/3005145601 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6246+0+0 (secure 0 0 0) 0x7f7940089ed0 con 0x7f79580d1c90 2024-03-24T00:42:56.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.129+0000 7f7956ffd640 1 --2- 172.21.15.67:0/3005145601 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f793806bbd0 0x7f793806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:56.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.129+0000 7f7956ffd640 1 --2- 172.21.15.67:0/3005145601 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f793806bbd0 0x7f793806e090 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f7948036000 tx=0x7f7948008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:56.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.133+0000 7f7954ff9640 1 -- 172.21.15.67:0/3005145601 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f79400581e0 con 0x7f79580d1c90 2024-03-24T00:42:56.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.314+0000 7f795d836640 1 -- 172.21.15.67:0/3005145601 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7958073f60 con 0x7f79580d1c90 2024-03-24T00:42:56.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.314+0000 7f7954ff9640 1 -- 172.21.15.67:0/3005145601 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v64) v1 ==== 100+0+11 (secure 0 0 0) 0x7f794005c280 con 0x7f79580d1c90 2024-03-24T00:42:56.315 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-03-24T00:42:56.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.316+0000 7f795d836640 1 -- 172.21.15.67:0/3005145601 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f793806bbd0 msgr2=0x7f793806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:56.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.316+0000 7f795d836640 1 --2- 172.21.15.67:0/3005145601 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f793806bbd0 0x7f793806e090 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f7948036000 tx=0x7f7948008030 comp rx=0 tx=0).stop 2024-03-24T00:42:56.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.316+0000 7f795d836640 1 -- 172.21.15.67:0/3005145601 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79580d1c90 msgr2=0x7f79580cd720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:56.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.316+0000 7f795d836640 1 --2- 172.21.15.67:0/3005145601 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79580d1c90 0x7f79580cd720 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f794000c500 tx=0x7f794000c9c0 comp rx=0 tx=0).stop 2024-03-24T00:42:56.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.316+0000 7f795d836640 1 -- 172.21.15.67:0/3005145601 shutdown_connections 2024-03-24T00:42:56.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.316+0000 7f795d836640 1 --2- 172.21.15.67:0/3005145601 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f793806bbd0 0x7f793806e090 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:56.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.316+0000 7f795d836640 1 --2- 172.21.15.67:0/3005145601 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7958160430 0x7f79580cdc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:56.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.316+0000 7f795d836640 1 --2- 172.21.15.67:0/3005145601 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79580d1c90 0x7f79580cd720 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:56.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.316+0000 7f795d836640 1 --2- 172.21.15.67:0/3005145601 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79580d18b0 0x7f79580cb690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:56.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.316+0000 7f795d836640 1 -- 172.21.15.67:0/3005145601 >> 172.21.15.67:0/3005145601 conn(0x7f795808b1b0 msgr2=0x7f7958151090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:56.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.316+0000 7f795d836640 1 -- 172.21.15.67:0/3005145601 shutdown_connections 2024-03-24T00:42:56.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.316+0000 7f795d836640 1 -- 172.21.15.67:0/3005145601 wait complete. 2024-03-24T00:42:56.327 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:42:56.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.432+0000 7f29a3bb2640 1 Processor -- start 2024-03-24T00:42:56.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.432+0000 7f29a3bb2640 1 -- start start 2024-03-24T00:42:56.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.434+0000 7f29a3bb2640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f299c0d0b90 0x7f299c0b01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:56.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.434+0000 7f29a3bb2640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f299c0b24f0 con 0x7f299c15ae70 2024-03-24T00:42:56.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.434+0000 7f29a3bb2640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f299c0b1730 con 0x7f299c0cf240 2024-03-24T00:42:56.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.434+0000 7f29a3bb2640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f299c0b1a10 con 0x7f299c0d0b90 2024-03-24T00:42:56.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.434+0000 7f29a194e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f299c0d0b90 0x7f299c0b01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:56.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.434+0000 7f29a114d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f299c15ae70 0x7f299c0d0f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60960/0 (socket says 172.21.15.67:60960) 2024-03-24T00:42:56.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.434+0000 7f29a114d640 1 -- 172.21.15.67:0/3661096626 learned_addr learned my addr 172.21.15.67:0/3661096626 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:42:56.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.434+0000 7f29a194e640 1 -- 172.21.15.67:0/3661096626 >> v1:172.21.15.72:6789/0 conn(0x7f299c0cf240 legacy=0x7f299c0ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:56.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.434+0000 7f29a194e640 1 -- 172.21.15.67:0/3661096626 >> v1:172.21.15.67:6789/0 conn(0x7f299c15ae70 legacy=0x7f299c0d0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:56.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.434+0000 7f29a194e640 1 -- 172.21.15.67:0/3661096626 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f299c0b1cf0 con 0x7f299c0d0b90 2024-03-24T00:42:56.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.434+0000 7f29a094c640 1 -- 172.21.15.67:0/3661096626 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 679361463 0 0) 0x7f299c0b24f0 con 0x7f299c15ae70 2024-03-24T00:42:56.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.435+0000 7f29a194e640 1 --2- 172.21.15.67:0/3661096626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f299c0d0b90 0x7f299c0b01d0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f29980097a0 tx=0x7f299802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfd235e4a45ae736 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:56.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.435+0000 7f29a094c640 1 -- 172.21.15.67:0/3661096626 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2998035020 con 0x7f299c0d0b90 2024-03-24T00:42:56.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.435+0000 7f29a094c640 1 -- 172.21.15.67:0/3661096626 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2998031830 con 0x7f299c0d0b90 2024-03-24T00:42:56.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.435+0000 7f29a094c640 1 -- 172.21.15.67:0/3661096626 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2998031c00 con 0x7f299c0d0b90 2024-03-24T00:42:56.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.436+0000 7f29a3bb2640 1 -- 172.21.15.67:0/3661096626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f299c0d0b90 msgr2=0x7f299c0b01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:56.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.436+0000 7f29a3bb2640 1 --2- 172.21.15.67:0/3661096626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f299c0d0b90 0x7f299c0b01d0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f29980097a0 tx=0x7f299802b8d0 comp rx=0 tx=0).stop 2024-03-24T00:42:56.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.436+0000 7f29a3bb2640 1 -- 172.21.15.67:0/3661096626 shutdown_connections 2024-03-24T00:42:56.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.436+0000 7f29a3bb2640 1 --2- 172.21.15.67:0/3661096626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f299c0d0b90 0x7f299c0b01d0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:56.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.436+0000 7f29a3bb2640 1 -- 172.21.15.67:0/3661096626 >> 172.21.15.67:0/3661096626 conn(0x7f299c099a30 msgr2=0x7f299c099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:56.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.436+0000 7f29a3bb2640 1 -- 172.21.15.67:0/3661096626 shutdown_connections 2024-03-24T00:42:56.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.436+0000 7f29a3bb2640 1 -- 172.21.15.67:0/3661096626 wait complete. 2024-03-24T00:42:56.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.437+0000 7f29a3bb2640 1 Processor -- start 2024-03-24T00:42:56.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.437+0000 7f29a3bb2640 1 -- start start 2024-03-24T00:42:56.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.437+0000 7f29a3bb2640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f299c0cf240 0x7f299c1598f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:56.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.437+0000 7f29a3bb2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f299c0d0b90 0x7f299c0c4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:56.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.437+0000 7f29a3bb2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f299c15ae70 0x7f299c06ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:56.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.437+0000 7f29a3bb2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f299c0cf9e0 con 0x7f299c0d0b90 2024-03-24T00:42:56.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.437+0000 7f29a3bb2640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f299c0cf6d0 con 0x7f299c15ae70 2024-03-24T00:42:56.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.437+0000 7f29a3bb2640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f299c0adfe0 con 0x7f299c0cf240 2024-03-24T00:42:56.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.438+0000 7f29a114d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f299c0d0b90 0x7f299c0c4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:56.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.438+0000 7f29a114d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f299c0d0b90 0x7f299c0c4310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42880/0 (socket says 172.21.15.67:42880) 2024-03-24T00:42:56.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.438+0000 7f29a114d640 1 -- 172.21.15.67:0/2299731234 learned_addr learned my addr 172.21.15.67:0/2299731234 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:56.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.438+0000 7f29a114d640 1 -- 172.21.15.67:0/2299731234 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f299c0cf240 msgr2=0x7f299c1598f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:56.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.438+0000 7f29a114d640 1 --2- 172.21.15.67:0/2299731234 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f299c0cf240 0x7f299c1598f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:56.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.438+0000 7f29a114d640 1 -- 172.21.15.67:0/2299731234 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f299c15ae70 msgr2=0x7f299c06ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:56.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.438+0000 7f29a114d640 1 --2- 172.21.15.67:0/2299731234 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f299c15ae70 0x7f299c06ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:56.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.438+0000 7f29a114d640 1 -- 172.21.15.67:0/2299731234 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f299c0c71a0 con 0x7f299c0d0b90 2024-03-24T00:42:56.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.438+0000 7f29a114d640 1 --2- 172.21.15.67:0/2299731234 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f299c0d0b90 0x7f299c0c4310 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f299000c430 tx=0x7f299000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:56.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.438+0000 7f298e7fc640 1 -- 172.21.15.67:0/2299731234 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29900074e0 con 0x7f299c0d0b90 2024-03-24T00:42:56.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.438+0000 7f298e7fc640 1 -- 172.21.15.67:0/2299731234 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29900047e0 con 0x7f299c0d0b90 2024-03-24T00:42:56.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.438+0000 7f298e7fc640 1 -- 172.21.15.67:0/2299731234 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2990004bd0 con 0x7f299c0d0b90 2024-03-24T00:42:56.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.438+0000 7f29a3bb2640 1 -- 172.21.15.67:0/2299731234 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f299c0c7480 con 0x7f299c0d0b90 2024-03-24T00:42:56.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.438+0000 7f29a3bb2640 1 -- 172.21.15.67:0/2299731234 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f299c0c7d30 con 0x7f299c0d0b90 2024-03-24T00:42:56.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.439+0000 7f29a3bb2640 1 -- 172.21.15.67:0/2299731234 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f299c04f400 con 0x7f299c0d0b90 2024-03-24T00:42:56.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.440+0000 7f298e7fc640 1 -- 172.21.15.67:0/2299731234 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2990011070 con 0x7f299c0d0b90 2024-03-24T00:42:56.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.440+0000 7f298e7fc640 1 --2- 172.21.15.67:0/2299731234 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f298006bc00 0x7f298006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:56.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.440+0000 7f298e7fc640 1 -- 172.21.15.67:0/2299731234 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6246+0+0 (secure 0 0 0) 0x7f299008a930 con 0x7f299c0d0b90 2024-03-24T00:42:56.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.440+0000 7f29a194e640 1 --2- 172.21.15.67:0/2299731234 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f298006bc00 0x7f298006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:56.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.441+0000 7f29a194e640 1 --2- 172.21.15.67:0/2299731234 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f298006bc00 0x7f298006e0c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f2998036000 tx=0x7f2998030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:56.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.444+0000 7f298e7fc640 1 -- 172.21.15.67:0/2299731234 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2990059200 con 0x7f299c0d0b90 2024-03-24T00:42:56.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.663+0000 7f29a3bb2640 1 -- 172.21.15.67:0/2299731234 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f299c072f90 con 0x7f299c0d0b90 2024-03-24T00:42:56.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.663+0000 7f298e7fc640 1 -- 172.21.15.67:0/2299731234 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4976 (secure 0 0 0) 0x7f2990058b00 con 0x7f299c0d0b90 2024-03-24T00:42:56.665 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:42:56.665 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5627,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2941726873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2941726873},{"type":"v1","addr":"172.21.15.72:6837","nonce":2941726873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":6166,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/1957901519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":1957901519},{"type":"v1","addr":"172.21.15.72:6836","nonce":1957901519}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":6357,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/2843754917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2843754917},{"type":"v1","addr":"172.21.15.67:6833","nonce":2843754917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:41:42.242211+0000","modified":"2024-03-24T00:42:22.910300+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6360},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6360":{"gid":6360,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":13,"addr":"172.21.15.67:6835/451962260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":451962260},{"type":"v1","addr":"172.21.15.67:6835","nonce":451962260}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6360,"qdb_cluster":[6360]},"id":4}]} 2024-03-24T00:42:56.665 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 34 2024-03-24T00:42:56.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.665+0000 7f29a3bb2640 1 -- 172.21.15.67:0/2299731234 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f298006bc00 msgr2=0x7f298006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:56.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.665+0000 7f29a3bb2640 1 --2- 172.21.15.67:0/2299731234 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f298006bc00 0x7f298006e0c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f2998036000 tx=0x7f2998030000 comp rx=0 tx=0).stop 2024-03-24T00:42:56.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.665+0000 7f29a3bb2640 1 -- 172.21.15.67:0/2299731234 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f299c0d0b90 msgr2=0x7f299c0c4310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:56.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.665+0000 7f29a3bb2640 1 --2- 172.21.15.67:0/2299731234 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f299c0d0b90 0x7f299c0c4310 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f299000c430 tx=0x7f299000c8f0 comp rx=0 tx=0).stop 2024-03-24T00:42:56.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.665+0000 7f29a3bb2640 1 -- 172.21.15.67:0/2299731234 shutdown_connections 2024-03-24T00:42:56.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.665+0000 7f29a3bb2640 1 --2- 172.21.15.67:0/2299731234 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f298006bc00 0x7f298006e0c0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:56.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.665+0000 7f29a3bb2640 1 --2- 172.21.15.67:0/2299731234 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f299c15ae70 0x7f299c06ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:56.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.665+0000 7f29a3bb2640 1 --2- 172.21.15.67:0/2299731234 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f299c0d0b90 0x7f299c0c4310 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:56.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.665+0000 7f29a3bb2640 1 --2- 172.21.15.67:0/2299731234 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f299c0cf240 0x7f299c1598f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:56.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.665+0000 7f29a3bb2640 1 -- 172.21.15.67:0/2299731234 >> 172.21.15.67:0/2299731234 conn(0x7f299c099a30 msgr2=0x7f299c15e490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:56.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.665+0000 7f29a3bb2640 1 -- 172.21.15.67:0/2299731234 shutdown_connections 2024-03-24T00:42:56.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.665+0000 7f29a3bb2640 1 -- 172.21.15.67:0/2299731234 wait complete. 2024-03-24T00:42:56.677 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:42:56.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.777+0000 7f74b2875640 1 Processor -- start 2024-03-24T00:42:56.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.777+0000 7f74b2875640 1 -- start start 2024-03-24T00:42:56.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.778+0000 7f74b2875640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f74ac0b18f0 0x7f74ac0b1d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:56.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.778+0000 7f74b2875640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f74ac0d0440 0x7f74ac0d08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:56.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.778+0000 7f74b2875640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74ac0b2590 con 0x7f74ac06ca40 2024-03-24T00:42:56.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.778+0000 7f74b2875640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74ac0b2870 con 0x7f74ac0d0440 2024-03-24T00:42:56.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.778+0000 7f74b2875640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74ac0b2b50 con 0x7f74ac0b18f0 2024-03-24T00:42:56.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.778+0000 7f74b0e12640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f74ac0d0440 0x7f74ac0d08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:56.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.778+0000 7f74b0e12640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f74ac0d0440 0x7f74ac0d08a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42894/0 (socket says 172.21.15.67:42894) 2024-03-24T00:42:56.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.779+0000 7f74b0e12640 1 -- 172.21.15.67:0/3331048562 learned_addr learned my addr 172.21.15.67:0/3331048562 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:56.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.779+0000 7f74ab7fe640 1 --2- 172.21.15.67:0/3331048562 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f74ac0b18f0 0x7f74ac0b1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:56.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.779+0000 7f74b0e12640 1 -- 172.21.15.67:0/3331048562 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f74ac0b18f0 msgr2=0x7f74ac0b1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:56.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.779+0000 7f74b0e12640 1 --2- 172.21.15.67:0/3331048562 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f74ac0b18f0 0x7f74ac0b1d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:56.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.779+0000 7f74b0e12640 1 -- 172.21.15.67:0/3331048562 >> v1:172.21.15.72:6789/0 conn(0x7f74ac06ca40 legacy=0x7f74ac06ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:56.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.779+0000 7f74b0e12640 1 -- 172.21.15.67:0/3331048562 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74ac0b2e30 con 0x7f74ac0d0440 2024-03-24T00:42:56.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.780+0000 7f74b0e12640 1 --2- 172.21.15.67:0/3331048562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74ac0d0440 0x7f74ac0d08a0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f74a4009e50 tx=0x7f74a402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfecb6a1dbe52702 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:56.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.780+0000 7f74aaffd640 1 -- 172.21.15.67:0/3331048562 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a402cb00 con 0x7f74ac0d0440 2024-03-24T00:42:56.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.780+0000 7f74aaffd640 1 -- 172.21.15.67:0/3331048562 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74a402cca0 con 0x7f74ac0d0440 2024-03-24T00:42:56.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.780+0000 7f74aaffd640 1 -- 172.21.15.67:0/3331048562 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a4034610 con 0x7f74ac0d0440 2024-03-24T00:42:56.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.780+0000 7f74b2875640 1 -- 172.21.15.67:0/3331048562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74ac0d0440 msgr2=0x7f74ac0d08a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:56.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.780+0000 7f74b2875640 1 --2- 172.21.15.67:0/3331048562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74ac0d0440 0x7f74ac0d08a0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f74a4009e50 tx=0x7f74a402bf60 comp rx=0 tx=0).stop 2024-03-24T00:42:56.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.780+0000 7f74b2875640 1 -- 172.21.15.67:0/3331048562 shutdown_connections 2024-03-24T00:42:56.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.780+0000 7f74b2875640 1 --2- 172.21.15.67:0/3331048562 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f74ac0b18f0 0x7f74ac0b1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:56.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.780+0000 7f74b2875640 1 --2- 172.21.15.67:0/3331048562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74ac0d0440 0x7f74ac0d08a0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:56.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.780+0000 7f74b2875640 1 -- 172.21.15.67:0/3331048562 >> 172.21.15.67:0/3331048562 conn(0x7f74ac0d1bf0 msgr2=0x7f74ac0d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:56.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.780+0000 7f74b2875640 1 -- 172.21.15.67:0/3331048562 shutdown_connections 2024-03-24T00:42:56.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.780+0000 7f74b2875640 1 -- 172.21.15.67:0/3331048562 wait complete. 2024-03-24T00:42:56.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.782+0000 7f74b2875640 1 Processor -- start 2024-03-24T00:42:56.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.782+0000 7f74b2875640 1 -- start start 2024-03-24T00:42:56.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.783+0000 7f74b2875640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f74ac06ca40 0x7f74ac1f7a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:56.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.784+0000 7f74b2875640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74ac0b18f0 0x7f74ac148bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:56.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.784+0000 7f74b2875640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f74ac0d0440 0x7f74ac1f39d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:56.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.784+0000 7f74b2875640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74ac15e370 con 0x7f74ac0b18f0 2024-03-24T00:42:56.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.784+0000 7f74b2875640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74ac0b2270 con 0x7f74ac06ca40 2024-03-24T00:42:56.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.784+0000 7f74b2875640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74ac15e4f0 con 0x7f74ac0d0440 2024-03-24T00:42:56.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.784+0000 7f74ab7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74ac0b18f0 0x7f74ac148bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:56.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.784+0000 7f74ab7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74ac0b18f0 0x7f74ac148bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42902/0 (socket says 172.21.15.67:42902) 2024-03-24T00:42:56.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.784+0000 7f74ab7fe640 1 -- 172.21.15.67:0/1343580825 learned_addr learned my addr 172.21.15.67:0/1343580825 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:56.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.784+0000 7f74b0e12640 1 --2- 172.21.15.67:0/1343580825 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f74ac0d0440 0x7f74ac1f39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:56.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.784+0000 7f74abfff640 1 --2- 172.21.15.67:0/1343580825 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f74ac06ca40 0x7f74ac1f7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:56.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.784+0000 7f74ab7fe640 1 -- 172.21.15.67:0/1343580825 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f74ac0d0440 msgr2=0x7f74ac1f39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:56.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.784+0000 7f74ab7fe640 1 --2- 172.21.15.67:0/1343580825 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f74ac0d0440 0x7f74ac1f39d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:56.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.784+0000 7f74ab7fe640 1 -- 172.21.15.67:0/1343580825 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f74ac06ca40 msgr2=0x7f74ac1f7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:56.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.784+0000 7f74ab7fe640 1 --2- 172.21.15.67:0/1343580825 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f74ac06ca40 0x7f74ac1f7a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:56.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.784+0000 7f74ab7fe640 1 -- 172.21.15.67:0/1343580825 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74ac160560 con 0x7f74ac0b18f0 2024-03-24T00:42:56.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.784+0000 7f74abfff640 1 --2- 172.21.15.67:0/1343580825 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f74ac06ca40 0x7f74ac1f7a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:42:56.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.784+0000 7f74ab7fe640 1 --2- 172.21.15.67:0/1343580825 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74ac0b18f0 0x7f74ac148bd0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f74a000e9e0 tx=0x7f74a000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:56.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.784+0000 7f74a8ff9640 1 -- 172.21.15.67:0/1343580825 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a0017070 con 0x7f74ac0b18f0 2024-03-24T00:42:56.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.784+0000 7f74a8ff9640 1 -- 172.21.15.67:0/1343580825 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74a000c9d0 con 0x7f74ac0b18f0 2024-03-24T00:42:56.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.784+0000 7f74b2875640 1 -- 172.21.15.67:0/1343580825 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74ac1607e0 con 0x7f74ac0b18f0 2024-03-24T00:42:56.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.784+0000 7f74a8ff9640 1 -- 172.21.15.67:0/1343580825 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a000cdc0 con 0x7f74ac0b18f0 2024-03-24T00:42:56.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.784+0000 7f74b2875640 1 -- 172.21.15.67:0/1343580825 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74ac1495c0 con 0x7f74ac0b18f0 2024-03-24T00:42:56.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.785+0000 7f74b2875640 1 -- 172.21.15.67:0/1343580825 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74ac04f400 con 0x7f74ac0b18f0 2024-03-24T00:42:56.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.786+0000 7f74a8ff9640 1 -- 172.21.15.67:0/1343580825 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f74a001a070 con 0x7f74ac0b18f0 2024-03-24T00:42:56.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.786+0000 7f74a8ff9640 1 --2- 172.21.15.67:0/1343580825 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f749806bbd0 0x7f749806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:56.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.787+0000 7f74a8ff9640 1 -- 172.21.15.67:0/1343580825 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6246+0+0 (secure 0 0 0) 0x7f74a008ad20 con 0x7f74ac0b18f0 2024-03-24T00:42:56.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.787+0000 7f74abfff640 1 --2- 172.21.15.67:0/1343580825 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f749806bbd0 0x7f749806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:56.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.787+0000 7f74abfff640 1 --2- 172.21.15.67:0/1343580825 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f749806bbd0 0x7f749806e090 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f74ac0c8ac0 tx=0x7f749c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:56.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:56.791+0000 7f74a8ff9640 1 -- 172.21.15.67:0/1343580825 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f74ac160560 con 0x7f74ac0b18f0 2024-03-24T00:42:57.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.014+0000 7f74b2875640 1 -- 172.21.15.67:0/1343580825 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f74ac08bbf0 con 0x7f74ac0b18f0 2024-03-24T00:42:57.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.015+0000 7f74a8ff9640 1 -- 172.21.15.67:0/1343580825 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) v1 ==== 74+0+17064 (secure 0 0 0) 0x7f74a00595f0 con 0x7f74ac0b18f0 2024-03-24T00:42:57.016 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:42:57.017 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":64,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:42:33.971493+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:41:40.502194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-03-24T00:41:41.200073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"9.7","mappings":[{"from":7,"to":2}]},{"pgid":"10.17","mappings":[{"from":4,"to":5}]},{"pgid":"10.1d","mappings":[{"from":4,"to":3}]},{"pgid":"10.2e","mappings":[{"from":4,"to":3}]},{"pgid":"10.3a","mappings":[{"from":7,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:42:57.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.018+0000 7f74b2875640 1 -- 172.21.15.67:0/1343580825 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f749806bbd0 msgr2=0x7f749806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:57.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.018+0000 7f74b2875640 1 --2- 172.21.15.67:0/1343580825 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f749806bbd0 0x7f749806e090 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f74ac0c8ac0 tx=0x7f749c009040 comp rx=0 tx=0).stop 2024-03-24T00:42:57.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.018+0000 7f74b2875640 1 -- 172.21.15.67:0/1343580825 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74ac0b18f0 msgr2=0x7f74ac148bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:57.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.018+0000 7f74b2875640 1 --2- 172.21.15.67:0/1343580825 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74ac0b18f0 0x7f74ac148bd0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f74a000e9e0 tx=0x7f74a000eea0 comp rx=0 tx=0).stop 2024-03-24T00:42:57.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.018+0000 7f74b2875640 1 -- 172.21.15.67:0/1343580825 shutdown_connections 2024-03-24T00:42:57.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.018+0000 7f74b2875640 1 --2- 172.21.15.67:0/1343580825 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f749806bbd0 0x7f749806e090 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:57.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.018+0000 7f74b2875640 1 --2- 172.21.15.67:0/1343580825 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f74ac0d0440 0x7f74ac1f39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:57.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.018+0000 7f74b2875640 1 --2- 172.21.15.67:0/1343580825 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74ac0b18f0 0x7f74ac148bd0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:57.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.018+0000 7f74b2875640 1 --2- 172.21.15.67:0/1343580825 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f74ac06ca40 0x7f74ac1f7a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:57.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.018+0000 7f74b2875640 1 -- 172.21.15.67:0/1343580825 >> 172.21.15.67:0/1343580825 conn(0x7f74ac0d1bf0 msgr2=0x7f74ac15d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:57.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.018+0000 7f74b2875640 1 -- 172.21.15.67:0/1343580825 shutdown_connections 2024-03-24T00:42:57.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.018+0000 7f74b2875640 1 -- 172.21.15.67:0/1343580825 wait complete. 2024-03-24T00:42:57.033 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-24T00:42:57.034 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:42:57.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.172+0000 7fd0b057e640 1 Processor -- start 2024-03-24T00:42:57.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.172+0000 7fd0b057e640 1 -- start start 2024-03-24T00:42:57.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.173+0000 7fd0b057e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd0a815e2a0 0x7fd0a80ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:57.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.173+0000 7fd0b057e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0a80cf640 con 0x7fd0a80d0ef0 2024-03-24T00:42:57.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.173+0000 7fd0b057e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0a80cf080 con 0x7fd0a815dec0 2024-03-24T00:42:57.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.173+0000 7fd0b057e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0a80cee30 con 0x7fd0a815e2a0 2024-03-24T00:42:57.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.174+0000 7fd0aeb1b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd0a80d0ef0 0x7fd0a80d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60976/0 (socket says 172.21.15.67:60976) 2024-03-24T00:42:57.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.174+0000 7fd0aeb1b640 1 -- 172.21.15.67:0/3654700484 learned_addr learned my addr 172.21.15.67:0/3654700484 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:42:57.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.174+0000 7fd0adb19640 1 --2- 172.21.15.67:0/3654700484 >> v2:172.21.15.72:3300/0 conn(0x7fd0a815e2a0 0x7fd0a80ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:57.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.174+0000 7fd0ad318640 1 -- 172.21.15.67:0/3654700484 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1318053532 0 0) 0x7fd0a4001490 con 0x7fd0a80d0ef0 2024-03-24T00:42:57.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.174+0000 7fd0ad318640 1 -- 172.21.15.67:0/3654700484 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd094003520 con 0x7fd0a80d0ef0 2024-03-24T00:42:57.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.174+0000 7fd0ad318640 1 -- 172.21.15.67:0/3654700484 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3628466670 0 0) 0x7fd0a80cf080 con 0x7fd0a815dec0 2024-03-24T00:42:57.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.174+0000 7fd0ad318640 1 -- 172.21.15.67:0/3654700484 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0a4001490 con 0x7fd0a815dec0 2024-03-24T00:42:57.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.174+0000 7fd0ad318640 1 -- 172.21.15.67:0/3654700484 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3009122299 0 0) 0x7fd094003520 con 0x7fd0a80d0ef0 2024-03-24T00:42:57.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.174+0000 7fd0ad318640 1 -- 172.21.15.67:0/3654700484 >> v2:172.21.15.72:3300/0 conn(0x7fd0a815e2a0 msgr2=0x7fd0a80ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:57.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.174+0000 7fd0ad318640 1 --2- 172.21.15.67:0/3654700484 >> v2:172.21.15.72:3300/0 conn(0x7fd0a815e2a0 0x7fd0a80ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:57.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.174+0000 7fd0ad318640 1 -- 172.21.15.67:0/3654700484 >> v1:172.21.15.72:6789/0 conn(0x7fd0a815dec0 legacy=0x7fd0a80d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:57.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.174+0000 7fd0ad318640 1 -- 172.21.15.67:0/3654700484 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0a80cf360 con 0x7fd0a80d0ef0 2024-03-24T00:42:57.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.174+0000 7fd0ad318640 1 -- 172.21.15.67:0/3654700484 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7fd0a4001bc0 con 0x7fd0a80d0ef0 2024-03-24T00:42:57.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.174+0000 7fd0ad318640 1 -- 172.21.15.67:0/3654700484 >> v1:172.21.15.67:6789/0 conn(0x7fd0a80d0ef0 legacy=0x7fd0a80d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:57.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.174+0000 7fd0ad318640 1 --2- 172.21.15.67:0/3654700484 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd094003b40 0x7fd094003f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:57.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.174+0000 7fd0ad318640 1 --2- 172.21.15.67:0/3654700484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd094004800 0x7fd094004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:57.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.174+0000 7fd0ad318640 1 --2- 172.21.15.67:0/3654700484 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0940051c0 0x7fd09400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:57.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.174+0000 7fd0ad318640 1 -- 172.21.15.67:0/3654700484 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a80cee30 con 0x7fd094004800 2024-03-24T00:42:57.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.174+0000 7fd0ad318640 1 -- 172.21.15.67:0/3654700484 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd09400c160 con 0x7fd0940051c0 2024-03-24T00:42:57.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.174+0000 7fd0ad318640 1 -- 172.21.15.67:0/3654700484 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd09400c360 con 0x7fd094003b40 2024-03-24T00:42:57.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.174+0000 7fd0adb19640 1 --2- 172.21.15.67:0/3654700484 >> v2:172.21.15.72:3300/0 conn(0x7fd0a815e2a0 0x7fd0a80ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:42:57.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.174+0000 7fd0adb19640 1 --2- 172.21.15.67:0/3654700484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd094004800 0x7fd094004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:57.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.174+0000 7fd0aeb1b640 1 --2- 172.21.15.67:0/3654700484 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0940051c0 0x7fd09400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:57.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.174+0000 7fd0ae31a640 1 --2- 172.21.15.67:0/3654700484 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd094003b40 0x7fd094003f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:57.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.175+0000 7fd0adb19640 1 -- 172.21.15.67:0/3654700484 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd094003b40 msgr2=0x7fd094003f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:57.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.175+0000 7fd0adb19640 1 --2- 172.21.15.67:0/3654700484 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd094003b40 0x7fd094003f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:57.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.175+0000 7fd0adb19640 1 -- 172.21.15.67:0/3654700484 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0940051c0 msgr2=0x7fd09400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:57.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.175+0000 7fd0adb19640 1 --2- 172.21.15.67:0/3654700484 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0940051c0 0x7fd09400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:57.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.175+0000 7fd0adb19640 1 -- 172.21.15.67:0/3654700484 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd094003520 con 0x7fd094004800 2024-03-24T00:42:57.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.175+0000 7fd0aeb1b640 1 -- 172.21.15.67:0/3654700484 reap_dead start 2024-03-24T00:42:57.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.175+0000 7fd0ae31a640 1 --2- 172.21.15.67:0/3654700484 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd094003b40 0x7fd094003f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:42:57.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.175+0000 7fd0adb19640 1 --2- 172.21.15.67:0/3654700484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd094004800 0x7fd094004c80 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fd098009c40 tx=0x7fd094005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e2987729851713f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:57.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.175+0000 7fd0ad318640 1 -- 172.21.15.67:0/3654700484 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd098009840 con 0x7fd094004800 2024-03-24T00:42:57.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.176+0000 7fd0ad318640 1 -- 172.21.15.67:0/3654700484 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd098004020 con 0x7fd094004800 2024-03-24T00:42:57.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.176+0000 7fd0ad318640 1 -- 172.21.15.67:0/3654700484 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd098004450 con 0x7fd094004800 2024-03-24T00:42:57.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.176+0000 7fd0b057e640 1 -- 172.21.15.67:0/3654700484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd094004800 msgr2=0x7fd094004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:57.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.176+0000 7fd0b057e640 1 --2- 172.21.15.67:0/3654700484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd094004800 0x7fd094004c80 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fd098009c40 tx=0x7fd094005830 comp rx=0 tx=0).stop 2024-03-24T00:42:57.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.176+0000 7fd0b057e640 1 -- 172.21.15.67:0/3654700484 shutdown_connections 2024-03-24T00:42:57.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.176+0000 7fd0b057e640 1 --2- 172.21.15.67:0/3654700484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd094004800 0x7fd094004c80 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:57.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.176+0000 7fd0b057e640 1 -- 172.21.15.67:0/3654700484 >> 172.21.15.67:0/3654700484 conn(0x7fd0a806c810 msgr2=0x7fd0a806cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:57.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.176+0000 7fd0b057e640 1 -- 172.21.15.67:0/3654700484 shutdown_connections 2024-03-24T00:42:57.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.176+0000 7fd0b057e640 1 -- 172.21.15.67:0/3654700484 wait complete. 2024-03-24T00:42:57.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.177+0000 7fd0b057e640 1 Processor -- start 2024-03-24T00:42:57.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.178+0000 7fd0b057e640 1 -- start start 2024-03-24T00:42:57.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.178+0000 7fd0b057e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd094004800 0x7fd0a8158f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:57.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.178+0000 7fd0b057e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0a80b42f0 0x7fd0a81f7500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:57.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.178+0000 7fd0b057e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0a81f7a40 0x7fd0a80cca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:57.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.178+0000 7fd0b057e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a80ce9c0 con 0x7fd0a81f7a40 2024-03-24T00:42:57.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.178+0000 7fd0b057e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a80ce840 con 0x7fd0a80b42f0 2024-03-24T00:42:57.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.178+0000 7fd0b057e640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a80d2120 con 0x7fd094004800 2024-03-24T00:42:57.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.178+0000 7fd0aeb1b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0a81f7a40 0x7fd0a80cca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:57.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.178+0000 7fd0aeb1b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0a81f7a40 0x7fd0a80cca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42920/0 (socket says 172.21.15.67:42920) 2024-03-24T00:42:57.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.178+0000 7fd0aeb1b640 1 -- 172.21.15.67:0/3017395023 learned_addr learned my addr 172.21.15.67:0/3017395023 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:57.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.178+0000 7fd0ae31a640 1 --2- 172.21.15.67:0/3017395023 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd094004800 0x7fd0a8158f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:57.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.178+0000 7fd0adb19640 1 --2- 172.21.15.67:0/3017395023 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0a80b42f0 0x7fd0a81f7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:57.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.179+0000 7fd0aeb1b640 1 -- 172.21.15.67:0/3017395023 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd094004800 msgr2=0x7fd0a8158f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:57.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.179+0000 7fd0aeb1b640 1 --2- 172.21.15.67:0/3017395023 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd094004800 0x7fd0a8158f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:57.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.179+0000 7fd0aeb1b640 1 -- 172.21.15.67:0/3017395023 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0a80b42f0 msgr2=0x7fd0a81f7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:57.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.179+0000 7fd0aeb1b640 1 --2- 172.21.15.67:0/3017395023 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0a80b42f0 0x7fd0a81f7500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:57.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.179+0000 7fd0aeb1b640 1 -- 172.21.15.67:0/3017395023 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0a80b50c0 con 0x7fd0a81f7a40 2024-03-24T00:42:57.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.179+0000 7fd0aeb1b640 1 --2- 172.21.15.67:0/3017395023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0a81f7a40 0x7fd0a80cca40 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fd0a80b6dd0 tx=0x7fd0a400af60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:57.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.179+0000 7fd09effd640 1 -- 172.21.15.67:0/3017395023 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a40078c0 con 0x7fd0a81f7a40 2024-03-24T00:42:57.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.179+0000 7fd09effd640 1 -- 172.21.15.67:0/3017395023 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0a4018070 con 0x7fd0a81f7a40 2024-03-24T00:42:57.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.179+0000 7fd0b057e640 1 -- 172.21.15.67:0/3017395023 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0a80b4820 con 0x7fd0a81f7a40 2024-03-24T00:42:57.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.179+0000 7fd09effd640 1 -- 172.21.15.67:0/3017395023 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a4013630 con 0x7fd0a81f7a40 2024-03-24T00:42:57.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.179+0000 7fd0b057e640 1 -- 172.21.15.67:0/3017395023 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0a80b4de0 con 0x7fd0a81f7a40 2024-03-24T00:42:57.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.180+0000 7fd0b057e640 1 -- 172.21.15.67:0/3017395023 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0a804f400 con 0x7fd0a81f7a40 2024-03-24T00:42:57.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.181+0000 7fd09effd640 1 -- 172.21.15.67:0/3017395023 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd0a40137d0 con 0x7fd0a81f7a40 2024-03-24T00:42:57.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.181+0000 7fd09effd640 1 --2- 172.21.15.67:0/3017395023 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd08c06bbd0 0x7fd08c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:57.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.181+0000 7fd09effd640 1 -- 172.21.15.67:0/3017395023 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6246+0+0 (secure 0 0 0) 0x7fd0a408b300 con 0x7fd0a81f7a40 2024-03-24T00:42:57.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.181+0000 7fd0ae31a640 1 --2- 172.21.15.67:0/3017395023 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd08c06bbd0 0x7fd08c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:57.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.182+0000 7fd0ae31a640 1 --2- 172.21.15.67:0/3017395023 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd08c06bbd0 0x7fd08c06e090 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fd0a0007fd0 tx=0x7fd0a0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:57.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.185+0000 7fd09effd640 1 -- 172.21.15.67:0/3017395023 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd0a80cf640 con 0x7fd0a81f7a40 2024-03-24T00:42:57.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.405+0000 7fd0b057e640 1 -- 172.21.15.67:0/3017395023 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0a80d1aa0 con 0x7fd0a81f7a40 2024-03-24T00:42:57.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.406+0000 7fd09effd640 1 -- 172.21.15.67:0/3017395023 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4976 (secure 0 0 0) 0x7fd0a4059bd0 con 0x7fd0a81f7a40 2024-03-24T00:42:57.407 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:42:57.407 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5627,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2941726873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2941726873},{"type":"v1","addr":"172.21.15.72:6837","nonce":2941726873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":6166,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/1957901519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":1957901519},{"type":"v1","addr":"172.21.15.72:6836","nonce":1957901519}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":6357,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/2843754917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2843754917},{"type":"v1","addr":"172.21.15.67:6833","nonce":2843754917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:41:42.242211+0000","modified":"2024-03-24T00:42:22.910300+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6360},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6360":{"gid":6360,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":13,"addr":"172.21.15.67:6835/451962260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":451962260},{"type":"v1","addr":"172.21.15.67:6835","nonce":451962260}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6360,"qdb_cluster":[6360]},"id":4}]} 2024-03-24T00:42:57.407 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 34 2024-03-24T00:42:57.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.407+0000 7fd0b057e640 1 -- 172.21.15.67:0/3017395023 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd08c06bbd0 msgr2=0x7fd08c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:57.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.407+0000 7fd0b057e640 1 --2- 172.21.15.67:0/3017395023 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd08c06bbd0 0x7fd08c06e090 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fd0a0007fd0 tx=0x7fd0a0009040 comp rx=0 tx=0).stop 2024-03-24T00:42:57.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.407+0000 7fd0b057e640 1 -- 172.21.15.67:0/3017395023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0a81f7a40 msgr2=0x7fd0a80cca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:57.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.407+0000 7fd0b057e640 1 --2- 172.21.15.67:0/3017395023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0a81f7a40 0x7fd0a80cca40 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fd0a80b6dd0 tx=0x7fd0a400af60 comp rx=0 tx=0).stop 2024-03-24T00:42:57.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.407+0000 7fd0b057e640 1 -- 172.21.15.67:0/3017395023 shutdown_connections 2024-03-24T00:42:57.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.407+0000 7fd0b057e640 1 --2- 172.21.15.67:0/3017395023 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd08c06bbd0 0x7fd08c06e090 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:57.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.407+0000 7fd0b057e640 1 --2- 172.21.15.67:0/3017395023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0a81f7a40 0x7fd0a80cca40 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:57.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.407+0000 7fd0b057e640 1 --2- 172.21.15.67:0/3017395023 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0a80b42f0 0x7fd0a81f7500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:57.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.407+0000 7fd0b057e640 1 --2- 172.21.15.67:0/3017395023 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd094004800 0x7fd0a8158f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:57.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.407+0000 7fd0b057e640 1 -- 172.21.15.67:0/3017395023 >> 172.21.15.67:0/3017395023 conn(0x7fd0a806c810 msgr2=0x7fd0a81602e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:57.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.408+0000 7fd0b057e640 1 -- 172.21.15.67:0/3017395023 shutdown_connections 2024-03-24T00:42:57.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.408+0000 7fd0b057e640 1 -- 172.21.15.67:0/3017395023 wait complete. 2024-03-24T00:42:57.419 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:42:57.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.520+0000 7f362c177640 1 Processor -- start 2024-03-24T00:42:57.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.520+0000 7f362c177640 1 -- start start 2024-03-24T00:42:57.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.521+0000 7f362c177640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f36241445a0 0x7f36240cee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:57.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.521+0000 7f362c177640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f36240b48d0 0x7f3624072550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:57.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.521+0000 7f362c177640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36240b4d30 con 0x7f362406ec40 2024-03-24T00:42:57.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.521+0000 7f362c177640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36240ce990 con 0x7f36240b48d0 2024-03-24T00:42:57.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.521+0000 7f362c177640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36240cddc0 con 0x7f36241445a0 2024-03-24T00:42:57.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.521+0000 7f3629712640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f36240b48d0 0x7f3624072550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:57.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.521+0000 7f3629712640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f36240b48d0 0x7f3624072550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42934/0 (socket says 172.21.15.67:42934) 2024-03-24T00:42:57.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.521+0000 7f3629712640 1 -- 172.21.15.67:0/1785895402 learned_addr learned my addr 172.21.15.67:0/1785895402 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:57.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.521+0000 7f3629f13640 1 --2- 172.21.15.67:0/1785895402 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f36241445a0 0x7f36240cee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:57.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.521+0000 7f3629712640 1 -- 172.21.15.67:0/1785895402 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f36241445a0 msgr2=0x7f36240cee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:57.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.521+0000 7f3629712640 1 --2- 172.21.15.67:0/1785895402 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f36241445a0 0x7f36240cee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:57.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.521+0000 7f3629712640 1 -- 172.21.15.67:0/1785895402 >> v1:172.21.15.72:6789/0 conn(0x7f362406ec40 legacy=0x7f36240b2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:57.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.521+0000 7f3629712640 1 -- 172.21.15.67:0/1785895402 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36240ce060 con 0x7f36240b48d0 2024-03-24T00:42:57.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.522+0000 7f3629712640 1 --2- 172.21.15.67:0/1785895402 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36240b48d0 0x7f3624072550 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f3618009870 tx=0x7f361802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df5fdd9f20222439 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:57.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.523+0000 7f3628f11640 1 -- 172.21.15.67:0/1785895402 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3618035020 con 0x7f36240b48d0 2024-03-24T00:42:57.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.523+0000 7f3628f11640 1 -- 172.21.15.67:0/1785895402 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3618031640 con 0x7f36240b48d0 2024-03-24T00:42:57.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.523+0000 7f3628f11640 1 -- 172.21.15.67:0/1785895402 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f361802d040 con 0x7f36240b48d0 2024-03-24T00:42:57.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.523+0000 7f362c177640 1 -- 172.21.15.67:0/1785895402 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36240b48d0 msgr2=0x7f3624072550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:57.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.523+0000 7f362c177640 1 --2- 172.21.15.67:0/1785895402 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36240b48d0 0x7f3624072550 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f3618009870 tx=0x7f361802b8f0 comp rx=0 tx=0).stop 2024-03-24T00:42:57.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.523+0000 7f362c177640 1 -- 172.21.15.67:0/1785895402 shutdown_connections 2024-03-24T00:42:57.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.523+0000 7f362c177640 1 --2- 172.21.15.67:0/1785895402 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36240b48d0 0x7f3624072550 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:57.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.523+0000 7f362c177640 1 --2- 172.21.15.67:0/1785895402 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f36241445a0 0x7f36240cee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:57.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.523+0000 7f362c177640 1 -- 172.21.15.67:0/1785895402 >> 172.21.15.67:0/1785895402 conn(0x7f36240cf9e0 msgr2=0x7f36240cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:57.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.523+0000 7f362c177640 1 -- 172.21.15.67:0/1785895402 shutdown_connections 2024-03-24T00:42:57.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.523+0000 7f362c177640 1 -- 172.21.15.67:0/1785895402 wait complete. 2024-03-24T00:42:57.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.524+0000 7f362c177640 1 Processor -- start 2024-03-24T00:42:57.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.524+0000 7f362c177640 1 -- start start 2024-03-24T00:42:57.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.525+0000 7f362c177640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f362406ec40 0x7f36241519d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:57.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.525+0000 7f362c177640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f36240b48d0 0x7f3624154260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:57.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.525+0000 7f362c177640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36241445a0 0x7f36241547a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:57.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.525+0000 7f362c177640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f362406f0d0 con 0x7f36241445a0 2024-03-24T00:42:57.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.525+0000 7f362c177640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36240ce620 con 0x7f36240b48d0 2024-03-24T00:42:57.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.525+0000 7f362c177640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f362406f250 con 0x7f362406ec40 2024-03-24T00:42:57.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.525+0000 7f362a714640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36241445a0 0x7f36241547a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:57.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.525+0000 7f362a714640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36241445a0 0x7f36241547a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42938/0 (socket says 172.21.15.67:42938) 2024-03-24T00:42:57.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.525+0000 7f362a714640 1 -- 172.21.15.67:0/369139703 learned_addr learned my addr 172.21.15.67:0/369139703 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:57.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.526+0000 7f3629f13640 1 --2- 172.21.15.67:0/369139703 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f362406ec40 0x7f36241519d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:57.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.526+0000 7f3629712640 1 --2- 172.21.15.67:0/369139703 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f36240b48d0 0x7f3624154260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:57.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.526+0000 7f362a714640 1 -- 172.21.15.67:0/369139703 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f362406ec40 msgr2=0x7f36241519d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:57.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.526+0000 7f362a714640 1 --2- 172.21.15.67:0/369139703 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f362406ec40 0x7f36241519d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:57.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.526+0000 7f362a714640 1 -- 172.21.15.67:0/369139703 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f36240b48d0 msgr2=0x7f3624154260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:57.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.526+0000 7f362a714640 1 --2- 172.21.15.67:0/369139703 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f36240b48d0 0x7f3624154260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:57.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.526+0000 7f362a714640 1 -- 172.21.15.67:0/369139703 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3624072070 con 0x7f36241445a0 2024-03-24T00:42:57.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.526+0000 7f362a714640 1 --2- 172.21.15.67:0/369139703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36241445a0 0x7f36241547a0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f361c00c780 tx=0x7f361c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:57.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.526+0000 7f36167fc640 1 -- 172.21.15.67:0/369139703 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f361c004b60 con 0x7f36241445a0 2024-03-24T00:42:57.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.526+0000 7f36167fc640 1 -- 172.21.15.67:0/369139703 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f361c004d00 con 0x7f36241445a0 2024-03-24T00:42:57.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.526+0000 7f362c177640 1 -- 172.21.15.67:0/369139703 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3624159a00 con 0x7f36241445a0 2024-03-24T00:42:57.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.526+0000 7f36167fc640 1 -- 172.21.15.67:0/369139703 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f361c00aa90 con 0x7f36241445a0 2024-03-24T00:42:57.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.526+0000 7f362c177640 1 -- 172.21.15.67:0/369139703 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3624072610 con 0x7f36241445a0 2024-03-24T00:42:57.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.527+0000 7f362c177640 1 -- 172.21.15.67:0/369139703 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f362404f400 con 0x7f36241445a0 2024-03-24T00:42:57.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.528+0000 7f36167fc640 1 -- 172.21.15.67:0/369139703 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f361c00ac30 con 0x7f36241445a0 2024-03-24T00:42:57.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.528+0000 7f36167fc640 1 --2- 172.21.15.67:0/369139703 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f360806bbd0 0x7f360806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:57.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.528+0000 7f36167fc640 1 -- 172.21.15.67:0/369139703 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6246+0+0 (secure 0 0 0) 0x7f361c08ae80 con 0x7f36241445a0 2024-03-24T00:42:57.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.528+0000 7f3629f13640 1 --2- 172.21.15.67:0/369139703 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f360806bbd0 0x7f360806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:57.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.529+0000 7f3629f13640 1 --2- 172.21.15.67:0/369139703 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f360806bbd0 0x7f360806e090 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f3620006fd0 tx=0x7f3620008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:57.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.532+0000 7f36167fc640 1 -- 172.21.15.67:0/369139703 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f361c020070 con 0x7f36241445a0 2024-03-24T00:42:57.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.754+0000 7f362c177640 1 -- 172.21.15.67:0/369139703 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3624072c20 con 0x7f36241445a0 2024-03-24T00:42:57.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.755+0000 7f36167fc640 1 -- 172.21.15.67:0/369139703 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4976 (secure 0 0 0) 0x7f361c0597e0 con 0x7f36241445a0 2024-03-24T00:42:57.756 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:42:57.756 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5627,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2941726873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2941726873},{"type":"v1","addr":"172.21.15.72:6837","nonce":2941726873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":6166,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/1957901519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":1957901519},{"type":"v1","addr":"172.21.15.72:6836","nonce":1957901519}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":6357,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/2843754917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2843754917},{"type":"v1","addr":"172.21.15.67:6833","nonce":2843754917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:41:42.242211+0000","modified":"2024-03-24T00:42:22.910300+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6360},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6360":{"gid":6360,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":13,"addr":"172.21.15.67:6835/451962260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":451962260},{"type":"v1","addr":"172.21.15.67:6835","nonce":451962260}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6360,"qdb_cluster":[6360]},"id":4}]} 2024-03-24T00:42:57.756 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 34 2024-03-24T00:42:57.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.757+0000 7f362c177640 1 -- 172.21.15.67:0/369139703 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f360806bbd0 msgr2=0x7f360806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:57.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.757+0000 7f362c177640 1 --2- 172.21.15.67:0/369139703 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f360806bbd0 0x7f360806e090 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f3620006fd0 tx=0x7f3620008040 comp rx=0 tx=0).stop 2024-03-24T00:42:57.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.757+0000 7f362c177640 1 -- 172.21.15.67:0/369139703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36241445a0 msgr2=0x7f36241547a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:57.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.757+0000 7f362c177640 1 --2- 172.21.15.67:0/369139703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36241445a0 0x7f36241547a0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f361c00c780 tx=0x7f361c00cc40 comp rx=0 tx=0).stop 2024-03-24T00:42:57.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.757+0000 7f362c177640 1 -- 172.21.15.67:0/369139703 shutdown_connections 2024-03-24T00:42:57.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.757+0000 7f362c177640 1 --2- 172.21.15.67:0/369139703 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f360806bbd0 0x7f360806e090 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:57.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.757+0000 7f362c177640 1 --2- 172.21.15.67:0/369139703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36241445a0 0x7f36241547a0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:57.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.757+0000 7f362c177640 1 --2- 172.21.15.67:0/369139703 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f36240b48d0 0x7f3624154260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:57.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.757+0000 7f362c177640 1 --2- 172.21.15.67:0/369139703 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f362406ec40 0x7f36241519d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:57.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.757+0000 7f362c177640 1 -- 172.21.15.67:0/369139703 >> 172.21.15.67:0/369139703 conn(0x7f36240cf9e0 msgr2=0x7f36240cee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:57.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.757+0000 7f362c177640 1 -- 172.21.15.67:0/369139703 shutdown_connections 2024-03-24T00:42:57.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.758+0000 7f362c177640 1 -- 172.21.15.67:0/369139703 wait complete. 2024-03-24T00:42:57.769 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:42:57.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.870+0000 7fee4f485640 1 Processor -- start 2024-03-24T00:42:57.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.870+0000 7fee4f485640 1 -- start start 2024-03-24T00:42:57.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.871+0000 7fee4f485640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fee480cee20 0x7fee480b48d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:57.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.871+0000 7fee4f485640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee480cddc0 con 0x7fee4806f350 2024-03-24T00:42:57.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.871+0000 7fee4f485640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee480ce060 con 0x7fee481445a0 2024-03-24T00:42:57.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.871+0000 7fee4f485640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee480ce990 con 0x7fee480cee20 2024-03-24T00:42:57.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.871+0000 7fee4ca20640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fee4806f350 0x7fee480b2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60980/0 (socket says 172.21.15.67:60980) 2024-03-24T00:42:57.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.871+0000 7fee4ca20640 1 -- 172.21.15.67:0/3114336584 learned_addr learned my addr 172.21.15.67:0/3114336584 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:42:57.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.871+0000 7fee4da22640 1 --2- 172.21.15.67:0/3114336584 >> v2:172.21.15.72:3300/0 conn(0x7fee480cee20 0x7fee480b48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:57.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.871+0000 7fee3ffff640 1 -- 172.21.15.67:0/3114336584 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3319471963 0 0) 0x7fee480cddc0 con 0x7fee4806f350 2024-03-24T00:42:57.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.871+0000 7fee3ffff640 1 -- 172.21.15.67:0/3114336584 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee34003520 con 0x7fee4806f350 2024-03-24T00:42:57.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.871+0000 7fee4da22640 1 -- 172.21.15.67:0/3114336584 >> v1:172.21.15.72:6789/0 conn(0x7fee481445a0 legacy=0x7fee4806ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:57.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.871+0000 7fee4da22640 1 -- 172.21.15.67:0/3114336584 >> v1:172.21.15.67:6789/0 conn(0x7fee4806f350 legacy=0x7fee480b2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:57.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.871+0000 7fee4da22640 1 -- 172.21.15.67:0/3114336584 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee480ce340 con 0x7fee480cee20 2024-03-24T00:42:57.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.872+0000 7fee4da22640 1 --2- 172.21.15.67:0/3114336584 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fee480cee20 0x7fee480b48d0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fee40009d90 tx=0x7fee4002bf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d787839aa587c960 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:57.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.872+0000 7fee3ffff640 1 -- 172.21.15.67:0/3114336584 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee40039070 con 0x7fee480cee20 2024-03-24T00:42:57.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.872+0000 7fee3ffff640 1 -- 172.21.15.67:0/3114336584 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee4002ccf0 con 0x7fee480cee20 2024-03-24T00:42:57.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.872+0000 7fee3ffff640 1 -- 172.21.15.67:0/3114336584 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee40034640 con 0x7fee480cee20 2024-03-24T00:42:57.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.873+0000 7fee4f485640 1 -- 172.21.15.67:0/3114336584 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fee480cee20 msgr2=0x7fee480b48d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:57.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.873+0000 7fee4f485640 1 --2- 172.21.15.67:0/3114336584 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fee480cee20 0x7fee480b48d0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fee40009d90 tx=0x7fee4002bf40 comp rx=0 tx=0).stop 2024-03-24T00:42:57.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.873+0000 7fee4f485640 1 -- 172.21.15.67:0/3114336584 shutdown_connections 2024-03-24T00:42:57.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.873+0000 7fee4f485640 1 --2- 172.21.15.67:0/3114336584 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fee480cee20 0x7fee480b48d0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:57.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.873+0000 7fee4f485640 1 -- 172.21.15.67:0/3114336584 >> 172.21.15.67:0/3114336584 conn(0x7fee480cf9e0 msgr2=0x7fee480cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:57.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.873+0000 7fee4f485640 1 -- 172.21.15.67:0/3114336584 shutdown_connections 2024-03-24T00:42:57.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.873+0000 7fee4f485640 1 -- 172.21.15.67:0/3114336584 wait complete. 2024-03-24T00:42:57.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.874+0000 7fee4f485640 1 Processor -- start 2024-03-24T00:42:57.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.874+0000 7fee4f485640 1 -- start start 2024-03-24T00:42:57.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.875+0000 7fee4f485640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fee4806f350 0x7fee4815c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:57.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.875+0000 7fee4f485640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fee480cee20 0x7fee4815bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:57.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.875+0000 7fee4f485640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee481445a0 0x7fee4815a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:57.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.875+0000 7fee4f485640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee480726d0 con 0x7fee481445a0 2024-03-24T00:42:57.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.875+0000 7fee4f485640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee48072550 con 0x7fee480cee20 2024-03-24T00:42:57.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.875+0000 7fee4f485640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee48072850 con 0x7fee4806f350 2024-03-24T00:42:57.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.875+0000 7fee4da22640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee481445a0 0x7fee4815a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:57.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.875+0000 7fee4da22640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee481445a0 0x7fee4815a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42940/0 (socket says 172.21.15.67:42940) 2024-03-24T00:42:57.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.875+0000 7fee4da22640 1 -- 172.21.15.67:0/1905707072 learned_addr learned my addr 172.21.15.67:0/1905707072 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:57.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.875+0000 7fee4d221640 1 --2- 172.21.15.67:0/1905707072 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fee4806f350 0x7fee4815c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:57.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.875+0000 7fee4ca20640 1 --2- 172.21.15.67:0/1905707072 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fee480cee20 0x7fee4815bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:57.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.875+0000 7fee4da22640 1 -- 172.21.15.67:0/1905707072 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fee4806f350 msgr2=0x7fee4815c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:57.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.875+0000 7fee4da22640 1 --2- 172.21.15.67:0/1905707072 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fee4806f350 0x7fee4815c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:57.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.875+0000 7fee4da22640 1 -- 172.21.15.67:0/1905707072 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fee480cee20 msgr2=0x7fee4815bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:57.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.875+0000 7fee4da22640 1 --2- 172.21.15.67:0/1905707072 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fee480cee20 0x7fee4815bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:57.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.875+0000 7fee4da22640 1 -- 172.21.15.67:0/1905707072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee4815cca0 con 0x7fee481445a0 2024-03-24T00:42:57.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.875+0000 7fee4d221640 1 --2- 172.21.15.67:0/1905707072 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fee4806f350 0x7fee4815c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:42:57.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.875+0000 7fee4ca20640 1 --2- 172.21.15.67:0/1905707072 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fee480cee20 0x7fee4815bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:42:57.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.875+0000 7fee4da22640 1 --2- 172.21.15.67:0/1905707072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee481445a0 0x7fee4815a7b0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fee40009d90 tx=0x7fee40004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:57.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.876+0000 7fee3dffb640 1 -- 172.21.15.67:0/1905707072 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee40044020 con 0x7fee481445a0 2024-03-24T00:42:57.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.876+0000 7fee3dffb640 1 -- 172.21.15.67:0/1905707072 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee4003a040 con 0x7fee481445a0 2024-03-24T00:42:57.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.876+0000 7fee4f485640 1 -- 172.21.15.67:0/1905707072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee4815cf80 con 0x7fee481445a0 2024-03-24T00:42:57.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.876+0000 7fee3dffb640 1 -- 172.21.15.67:0/1905707072 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee40039050 con 0x7fee481445a0 2024-03-24T00:42:57.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.876+0000 7fee4f485640 1 -- 172.21.15.67:0/1905707072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee48138510 con 0x7fee481445a0 2024-03-24T00:42:57.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.877+0000 7fee4f485640 1 -- 172.21.15.67:0/1905707072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee4804f400 con 0x7fee481445a0 2024-03-24T00:42:57.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.877+0000 7fee3dffb640 1 -- 172.21.15.67:0/1905707072 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fee40003bd0 con 0x7fee481445a0 2024-03-24T00:42:57.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.878+0000 7fee3dffb640 1 --2- 172.21.15.67:0/1905707072 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fee2c06bbd0 0x7fee2c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:57.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.878+0000 7fee3dffb640 1 -- 172.21.15.67:0/1905707072 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6246+0+0 (secure 0 0 0) 0x7fee400ad290 con 0x7fee481445a0 2024-03-24T00:42:57.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.878+0000 7fee4d221640 1 --2- 172.21.15.67:0/1905707072 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fee2c06bbd0 0x7fee2c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:57.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.879+0000 7fee4d221640 1 --2- 172.21.15.67:0/1905707072 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fee2c06bbd0 0x7fee2c06e090 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fee440015e0 tx=0x7fee44009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:57.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:57.882+0000 7fee3dffb640 1 -- 172.21.15.67:0/1905707072 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fee4004e060 con 0x7fee481445a0 2024-03-24T00:42:58.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.069+0000 7fee4f485640 1 -- 172.21.15.67:0/1905707072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee48072c20 con 0x7fee481445a0 2024-03-24T00:42:58.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.070+0000 7fee3dffb640 1 -- 172.21.15.67:0/1905707072 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) v1 ==== 74+0+17064 (secure 0 0 0) 0x7fee4007bbf0 con 0x7fee481445a0 2024-03-24T00:42:58.071 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:42:58.071 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":64,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:42:33.971493+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:41:40.502194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-03-24T00:41:41.200073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"9.7","mappings":[{"from":7,"to":2}]},{"pgid":"10.17","mappings":[{"from":4,"to":5}]},{"pgid":"10.1d","mappings":[{"from":4,"to":3}]},{"pgid":"10.2e","mappings":[{"from":4,"to":3}]},{"pgid":"10.3a","mappings":[{"from":7,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:42:58.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.073+0000 7fee4f485640 1 -- 172.21.15.67:0/1905707072 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fee2c06bbd0 msgr2=0x7fee2c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:58.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.073+0000 7fee4f485640 1 --2- 172.21.15.67:0/1905707072 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fee2c06bbd0 0x7fee2c06e090 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fee440015e0 tx=0x7fee44009040 comp rx=0 tx=0).stop 2024-03-24T00:42:58.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.073+0000 7fee4f485640 1 -- 172.21.15.67:0/1905707072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee481445a0 msgr2=0x7fee4815a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:58.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.073+0000 7fee4f485640 1 --2- 172.21.15.67:0/1905707072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee481445a0 0x7fee4815a7b0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fee40009d90 tx=0x7fee40004a40 comp rx=0 tx=0).stop 2024-03-24T00:42:58.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.073+0000 7fee4f485640 1 -- 172.21.15.67:0/1905707072 shutdown_connections 2024-03-24T00:42:58.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.073+0000 7fee4f485640 1 --2- 172.21.15.67:0/1905707072 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fee2c06bbd0 0x7fee2c06e090 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:58.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.073+0000 7fee4f485640 1 --2- 172.21.15.67:0/1905707072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee481445a0 0x7fee4815a7b0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:58.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.073+0000 7fee4f485640 1 --2- 172.21.15.67:0/1905707072 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fee480cee20 0x7fee4815bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:58.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.073+0000 7fee4f485640 1 --2- 172.21.15.67:0/1905707072 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fee4806f350 0x7fee4815c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:58.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.073+0000 7fee4f485640 1 -- 172.21.15.67:0/1905707072 >> 172.21.15.67:0/1905707072 conn(0x7fee480cf9e0 msgr2=0x7fee4815f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:58.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.073+0000 7fee4f485640 1 -- 172.21.15.67:0/1905707072 shutdown_connections 2024-03-24T00:42:58.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.073+0000 7fee4f485640 1 -- 172.21.15.67:0/1905707072 wait complete. 2024-03-24T00:42:58.084 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-24T00:42:58.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.190+0000 7f703e0f1640 1 Processor -- start 2024-03-24T00:42:58.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.190+0000 7f703e0f1640 1 -- start start 2024-03-24T00:42:58.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.190+0000 7f703e0f1640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f70380d32d0 0x7f70380d3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:58.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.190+0000 7f703e0f1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f70380b6dc0 0x7f70380b7250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:58.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.190+0000 7f703e0f1640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70380b9cc0 con 0x7f703815e0f0 2024-03-24T00:42:58.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.190+0000 7f703e0f1640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70380b7790 con 0x7f70380d32d0 2024-03-24T00:42:58.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.190+0000 7f703e0f1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70380d2230 con 0x7f70380b6dc0 2024-03-24T00:42:58.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.191+0000 7f7036ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f70380d32d0 0x7f70380d3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:58.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.191+0000 7f70377fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f703815e0f0 0x7f70380b95b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60996/0 (socket says 172.21.15.67:60996) 2024-03-24T00:42:58.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.191+0000 7f70377fe640 1 -- 172.21.15.67:0/236344671 learned_addr learned my addr 172.21.15.67:0/236344671 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:42:58.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.191+0000 7f7037fff640 1 --2- 172.21.15.67:0/236344671 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f70380b6dc0 0x7f70380b7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:58.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.191+0000 7f7036ffd640 1 -- 172.21.15.67:0/236344671 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f70380b6dc0 msgr2=0x7f70380b7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:58.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.191+0000 7f7036ffd640 1 --2- 172.21.15.67:0/236344671 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f70380b6dc0 0x7f70380b7250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:58.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.191+0000 7f7036ffd640 1 -- 172.21.15.67:0/236344671 >> v1:172.21.15.67:6789/0 conn(0x7f703815e0f0 legacy=0x7f70380b95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:58.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.191+0000 7f70367fc640 1 -- 172.21.15.67:0/236344671 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4348809 0 0) 0x7f70380b9cc0 con 0x7f703815e0f0 2024-03-24T00:42:58.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.191+0000 7f7036ffd640 1 -- 172.21.15.67:0/236344671 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70380d2510 con 0x7f70380d32d0 2024-03-24T00:42:58.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.192+0000 7f7036ffd640 1 --2- 172.21.15.67:0/236344671 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70380d32d0 0x7f70380d3740 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f702c009790 tx=0x7f702c02b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=968543be34327682 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:58.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.192+0000 7f70367fc640 1 -- 172.21.15.67:0/236344671 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f702c035020 con 0x7f70380d32d0 2024-03-24T00:42:58.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.192+0000 7f70367fc640 1 -- 172.21.15.67:0/236344671 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f702c031800 con 0x7f70380d32d0 2024-03-24T00:42:58.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.192+0000 7f70367fc640 1 -- 172.21.15.67:0/236344671 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f702c031bd0 con 0x7f70380d32d0 2024-03-24T00:42:58.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.193+0000 7f703e0f1640 1 -- 172.21.15.67:0/236344671 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70380d32d0 msgr2=0x7f70380d3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:58.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.193+0000 7f703e0f1640 1 --2- 172.21.15.67:0/236344671 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70380d32d0 0x7f70380d3740 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f702c009790 tx=0x7f702c02b8a0 comp rx=0 tx=0).stop 2024-03-24T00:42:58.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.193+0000 7f703e0f1640 1 -- 172.21.15.67:0/236344671 shutdown_connections 2024-03-24T00:42:58.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.193+0000 7f703e0f1640 1 --2- 172.21.15.67:0/236344671 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f70380b6dc0 0x7f70380b7250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:58.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.193+0000 7f703e0f1640 1 --2- 172.21.15.67:0/236344671 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70380d32d0 0x7f70380d3740 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:58.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.193+0000 7f703e0f1640 1 -- 172.21.15.67:0/236344671 >> 172.21.15.67:0/236344671 conn(0x7f7038072b60 msgr2=0x7f7038072f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:58.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.193+0000 7f703e0f1640 1 -- 172.21.15.67:0/236344671 shutdown_connections 2024-03-24T00:42:58.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.193+0000 7f703e0f1640 1 -- 172.21.15.67:0/236344671 wait complete. 2024-03-24T00:42:58.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.194+0000 7f703e0f1640 1 Processor -- start 2024-03-24T00:42:58.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.194+0000 7f703e0f1640 1 -- start start 2024-03-24T00:42:58.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.195+0000 7f703e0f1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70380b6dc0 0x7f703815ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:58.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.195+0000 7f703e0f1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f70380d32d0 0x7f703815ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:58.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.195+0000 7f703e0f1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f703815e0f0 0x7f70380cce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:58.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.195+0000 7f703e0f1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70380d2db0 con 0x7f70380b6dc0 2024-03-24T00:42:58.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.195+0000 7f703e0f1640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70380d2ad0 con 0x7f70380d32d0 2024-03-24T00:42:58.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.195+0000 7f703e0f1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70380b31d0 con 0x7f703815e0f0 2024-03-24T00:42:58.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.195+0000 7f70377fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70380b6dc0 0x7f703815ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:58.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.195+0000 7f70377fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70380b6dc0 0x7f703815ff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42956/0 (socket says 172.21.15.67:42956) 2024-03-24T00:42:58.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.195+0000 7f70377fe640 1 -- 172.21.15.67:0/3798519513 learned_addr learned my addr 172.21.15.67:0/3798519513 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:58.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.195+0000 7f7037fff640 1 --2- 172.21.15.67:0/3798519513 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f703815e0f0 0x7f70380cce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:58.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.195+0000 7f70377fe640 1 -- 172.21.15.67:0/3798519513 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f703815e0f0 msgr2=0x7f70380cce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:58.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.195+0000 7f7036ffd640 1 --2- 172.21.15.67:0/3798519513 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f70380d32d0 0x7f703815ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:58.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.195+0000 7f70377fe640 1 --2- 172.21.15.67:0/3798519513 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f703815e0f0 0x7f70380cce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:58.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.195+0000 7f70377fe640 1 -- 172.21.15.67:0/3798519513 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f70380d32d0 msgr2=0x7f703815ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:58.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.195+0000 7f70377fe640 1 --2- 172.21.15.67:0/3798519513 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f70380d32d0 0x7f703815ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:58.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.195+0000 7f70377fe640 1 -- 172.21.15.67:0/3798519513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f703815b350 con 0x7f70380b6dc0 2024-03-24T00:42:58.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.195+0000 7f7036ffd640 1 --2- 172.21.15.67:0/3798519513 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f70380d32d0 0x7f703815ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:42:58.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.195+0000 7f70377fe640 1 --2- 172.21.15.67:0/3798519513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70380b6dc0 0x7f703815ff30 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f702800c430 tx=0x7f702800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:58.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.196+0000 7f701ffff640 1 -- 172.21.15.67:0/3798519513 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70280074e0 con 0x7f70380b6dc0 2024-03-24T00:42:58.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.196+0000 7f701ffff640 1 -- 172.21.15.67:0/3798519513 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70280047e0 con 0x7f70380b6dc0 2024-03-24T00:42:58.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.196+0000 7f703e0f1640 1 -- 172.21.15.67:0/3798519513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70381606a0 con 0x7f70380b6dc0 2024-03-24T00:42:58.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.196+0000 7f701ffff640 1 -- 172.21.15.67:0/3798519513 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7028004bd0 con 0x7f70380b6dc0 2024-03-24T00:42:58.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.196+0000 7f703e0f1640 1 -- 172.21.15.67:0/3798519513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7038160c60 con 0x7f70380b6dc0 2024-03-24T00:42:58.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.197+0000 7f703e0f1640 1 -- 172.21.15.67:0/3798519513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f703804f400 con 0x7f70380b6dc0 2024-03-24T00:42:58.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.197+0000 7f701ffff640 1 -- 172.21.15.67:0/3798519513 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7028011070 con 0x7f70380b6dc0 2024-03-24T00:42:58.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.197+0000 7f701ffff640 1 --2- 172.21.15.67:0/3798519513 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f702406bbd0 0x7f702406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:58.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.198+0000 7f701ffff640 1 -- 172.21.15.67:0/3798519513 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6246+0+0 (secure 0 0 0) 0x7f702808a980 con 0x7f70380b6dc0 2024-03-24T00:42:58.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.198+0000 7f7036ffd640 1 --2- 172.21.15.67:0/3798519513 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f702406bbd0 0x7f702406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:58.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.198+0000 7f7036ffd640 1 --2- 172.21.15.67:0/3798519513 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f702406bbd0 0x7f702406e090 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f702c036000 tx=0x7f702c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:58.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.206+0000 7f701ffff640 1 -- 172.21.15.67:0/3798519513 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7028059250 con 0x7f70380b6dc0 2024-03-24T00:42:58.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.437+0000 7f703e0f1640 1 -- 172.21.15.67:0/3798519513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f70380d3a30 con 0x7f70380b6dc0 2024-03-24T00:42:58.438 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-24T00:42:58.437+0000 7fbfcfa11640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T00:42:58.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.439+0000 7f701ffff640 1 -- 172.21.15.67:0/3798519513 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v35) v1 ==== 161+0+0 (secure 0 0 0) 0x7f7028058b50 con 0x7f70380b6dc0 2024-03-24T00:42:58.440 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-24T00:42:58.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.442+0000 7f703e0f1640 1 -- 172.21.15.67:0/3798519513 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f702406bbd0 msgr2=0x7f702406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:58.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.442+0000 7f703e0f1640 1 --2- 172.21.15.67:0/3798519513 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f702406bbd0 0x7f702406e090 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f702c036000 tx=0x7f702c008030 comp rx=0 tx=0).stop 2024-03-24T00:42:58.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.442+0000 7f703e0f1640 1 -- 172.21.15.67:0/3798519513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70380b6dc0 msgr2=0x7f703815ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:58.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.442+0000 7f703e0f1640 1 --2- 172.21.15.67:0/3798519513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70380b6dc0 0x7f703815ff30 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f702800c430 tx=0x7f702800c8f0 comp rx=0 tx=0).stop 2024-03-24T00:42:58.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.442+0000 7f703e0f1640 1 -- 172.21.15.67:0/3798519513 shutdown_connections 2024-03-24T00:42:58.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.442+0000 7f703e0f1640 1 --2- 172.21.15.67:0/3798519513 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f702406bbd0 0x7f702406e090 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:58.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.442+0000 7f703e0f1640 1 --2- 172.21.15.67:0/3798519513 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f703815e0f0 0x7f70380cce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:58.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.442+0000 7f703e0f1640 1 --2- 172.21.15.67:0/3798519513 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f70380d32d0 0x7f703815ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:58.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.442+0000 7f703e0f1640 1 --2- 172.21.15.67:0/3798519513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70380b6dc0 0x7f703815ff30 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:58.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.442+0000 7f703e0f1640 1 -- 172.21.15.67:0/3798519513 >> 172.21.15.67:0/3798519513 conn(0x7f7038072b60 msgr2=0x7f70380c77b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:58.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.442+0000 7f703e0f1640 1 -- 172.21.15.67:0/3798519513 shutdown_connections 2024-03-24T00:42:58.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.442+0000 7f703e0f1640 1 -- 172.21.15.67:0/3798519513 wait complete. 2024-03-24T00:42:58.447 INFO:tasks.ceph.mds.a.smithi067.stderr: -7359> 2024-03-24T00:41:48.513+0000 7f214e7bd640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T00:42:58.449 INFO:tasks.ceph.mds.a.smithi067.stderr: -4826> 2024-03-24T00:41:49.929+0000 7f214a7b5640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [2,head] /dir_test/file auth v2 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={5732=0-4194304@1} caps={5732=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=5732 | request=0 caps=1 dirtyparent=0 dirty=0 authpin=1 scrubqueue=0 0x55fd9feebb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(10)0x10000000001:[<0x10000000000/file v2>,<0x1/dir_test v4>]//[]","error_str":"failed to decode on-disk backtrace (26 bytes)!"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-24T00:42:58.451 INFO:tasks.ceph.mds.a.smithi067.stderr: -2907> 2024-03-24T00:42:21.319+0000 7f214a7b5640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [2,head] /dir_test/file auth v4 ap=1 DIRTYPARENT s=0 n(v0 rc2024-03-24T00:41:48.869203+0000 1=1+0) (iversion lock) caps={5732=pAsLsXsFs/-@2},l=5732(-1) | ptrwaiter=0 request=0 lock=0 caps=1 dirtyparent=1 dirty=1 authpin=1 scrubqueue=0 0x55fd9feebb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(10)0x10000000001:[<0x10000000000/file v4>,<0x1/dir_test v7>]//[]","error_str":"failed to decode on-disk backtrace (26 bytes)!"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-24T00:42:58.454 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-24T00:42:58.479 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-03-24T00:42:58.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.559+0000 7f17814c6640 1 Processor -- start 2024-03-24T00:42:58.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.559+0000 7f17814c6640 1 -- start start 2024-03-24T00:42:58.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.560+0000 7f17814c6640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f177c0d22a0 0x7f177c0b1f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:58.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.560+0000 7f17814c6640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f177c0b2440 0x7f177c15d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:58.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.560+0000 7f17814c6640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f177c0b12f0 con 0x7f177c0b0f10 2024-03-24T00:42:58.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.560+0000 7f17814c6640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f177c0d09f0 con 0x7f177c0d22a0 2024-03-24T00:42:58.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.560+0000 7f17814c6640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f177c0b2c50 con 0x7f177c0b2440 2024-03-24T00:42:58.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.560+0000 7f177a7fc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f177c0d22a0 0x7f177c0b1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:58.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.560+0000 7f177a7fc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f177c0d22a0 0x7f177c0b1f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42970/0 (socket says 172.21.15.67:42970) 2024-03-24T00:42:58.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.560+0000 7f177a7fc640 1 -- 172.21.15.67:0/675034920 learned_addr learned my addr 172.21.15.67:0/675034920 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:58.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.560+0000 7f177b7fe640 1 --2- 172.21.15.67:0/675034920 >> v2:172.21.15.72:3300/0 conn(0x7f177c0b2440 0x7f177c15d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:58.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.561+0000 7f177a7fc640 1 -- 172.21.15.67:0/675034920 >> v2:172.21.15.72:3300/0 conn(0x7f177c0b2440 msgr2=0x7f177c15d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:58.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.561+0000 7f177a7fc640 1 --2- 172.21.15.67:0/675034920 >> v2:172.21.15.72:3300/0 conn(0x7f177c0b2440 0x7f177c15d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:58.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.561+0000 7f177a7fc640 1 -- 172.21.15.67:0/675034920 >> v1:172.21.15.72:6789/0 conn(0x7f177c0b0f10 legacy=0x7f177c0d1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:58.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.561+0000 7f177a7fc640 1 -- 172.21.15.67:0/675034920 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f177c0ce880 con 0x7f177c0d22a0 2024-03-24T00:42:58.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.561+0000 7f177a7fc640 1 --2- 172.21.15.67:0/675034920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f177c0d22a0 0x7f177c0b1f00 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f177c0b35b0 tx=0x7f176402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e871486ea8d33c01 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:58.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.561+0000 7f1779ffb640 1 -- 172.21.15.67:0/675034920 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1764034070 con 0x7f177c0d22a0 2024-03-24T00:42:58.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.561+0000 7f1779ffb640 1 -- 172.21.15.67:0/675034920 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1764030860 con 0x7f177c0d22a0 2024-03-24T00:42:58.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.561+0000 7f1779ffb640 1 -- 172.21.15.67:0/675034920 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1764030c30 con 0x7f177c0d22a0 2024-03-24T00:42:58.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.562+0000 7f17814c6640 1 -- 172.21.15.67:0/675034920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f177c0d22a0 msgr2=0x7f177c0b1f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:58.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.562+0000 7f17814c6640 1 --2- 172.21.15.67:0/675034920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f177c0d22a0 0x7f177c0b1f00 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f177c0b35b0 tx=0x7f176402b9a0 comp rx=0 tx=0).stop 2024-03-24T00:42:58.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.562+0000 7f17814c6640 1 -- 172.21.15.67:0/675034920 shutdown_connections 2024-03-24T00:42:58.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.562+0000 7f17814c6640 1 --2- 172.21.15.67:0/675034920 >> v2:172.21.15.72:3300/0 conn(0x7f177c0b2440 0x7f177c15d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:58.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.562+0000 7f17814c6640 1 --2- 172.21.15.67:0/675034920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f177c0d22a0 0x7f177c0b1f00 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:58.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.562+0000 7f17814c6640 1 -- 172.21.15.67:0/675034920 >> 172.21.15.67:0/675034920 conn(0x7f177c08b1b0 msgr2=0x7f177c08b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:58.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.562+0000 7f17814c6640 1 -- 172.21.15.67:0/675034920 shutdown_connections 2024-03-24T00:42:58.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.562+0000 7f17814c6640 1 -- 172.21.15.67:0/675034920 wait complete. 2024-03-24T00:42:58.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.563+0000 7f17814c6640 1 Processor -- start 2024-03-24T00:42:58.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.563+0000 7f17814c6640 1 -- start start 2024-03-24T00:42:58.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.564+0000 7f17814c6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f177c0b0f10 0x7f177c06d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:58.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.564+0000 7f17814c6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f177c0b2440 0x7f177c06db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:58.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.564+0000 7f17814c6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f177c0d22a0 0x7f177c1e7a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:58.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.564+0000 7f17814c6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f177c0ce560 con 0x7f177c0b0f10 2024-03-24T00:42:58.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.564+0000 7f17814c6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f177c0cee40 con 0x7f177c0b2440 2024-03-24T00:42:58.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.564+0000 7f17814c6640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f177c0cf360 con 0x7f177c0d22a0 2024-03-24T00:42:58.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.564+0000 7f177affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f177c0b0f10 0x7f177c06d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:58.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.564+0000 7f177affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f177c0b0f10 0x7f177c06d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42976/0 (socket says 172.21.15.67:42976) 2024-03-24T00:42:58.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.564+0000 7f177affd640 1 -- 172.21.15.67:0/1472887201 learned_addr learned my addr 172.21.15.67:0/1472887201 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:58.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.564+0000 7f177b7fe640 1 --2- 172.21.15.67:0/1472887201 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f177c0d22a0 0x7f177c1e7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:58.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.564+0000 7f177a7fc640 1 --2- 172.21.15.67:0/1472887201 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f177c0b2440 0x7f177c06db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:58.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.564+0000 7f177affd640 1 -- 172.21.15.67:0/1472887201 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f177c0d22a0 msgr2=0x7f177c1e7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:58.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.564+0000 7f177affd640 1 --2- 172.21.15.67:0/1472887201 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f177c0d22a0 0x7f177c1e7a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:58.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.564+0000 7f177affd640 1 -- 172.21.15.67:0/1472887201 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f177c0b2440 msgr2=0x7f177c06db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:58.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.564+0000 7f177affd640 1 --2- 172.21.15.67:0/1472887201 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f177c0b2440 0x7f177c06db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:58.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.564+0000 7f177affd640 1 -- 172.21.15.67:0/1472887201 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f177c0b4c70 con 0x7f177c0b0f10 2024-03-24T00:42:58.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.565+0000 7f177affd640 1 --2- 172.21.15.67:0/1472887201 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f177c0b0f10 0x7f177c06d630 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f177400c3c0 tx=0x7f177400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:58.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.565+0000 7f17737fe640 1 -- 172.21.15.67:0/1472887201 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17740052d0 con 0x7f177c0b0f10 2024-03-24T00:42:58.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.565+0000 7f17737fe640 1 -- 172.21.15.67:0/1472887201 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1774005490 con 0x7f177c0b0f10 2024-03-24T00:42:58.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.565+0000 7f17737fe640 1 -- 172.21.15.67:0/1472887201 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1774007260 con 0x7f177c0b0f10 2024-03-24T00:42:58.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.565+0000 7f17814c6640 1 -- 172.21.15.67:0/1472887201 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f177c0c72a0 con 0x7f177c0b0f10 2024-03-24T00:42:58.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.565+0000 7f17814c6640 1 -- 172.21.15.67:0/1472887201 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f177c0c7860 con 0x7f177c0b0f10 2024-03-24T00:42:58.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.566+0000 7f17814c6640 1 -- 172.21.15.67:0/1472887201 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f177c04f400 con 0x7f177c0b0f10 2024-03-24T00:42:58.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.566+0000 7f17737fe640 1 -- 172.21.15.67:0/1472887201 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1774007400 con 0x7f177c0b0f10 2024-03-24T00:42:58.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.567+0000 7f17737fe640 1 --2- 172.21.15.67:0/1472887201 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f175c06bbd0 0x7f175c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:58.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.567+0000 7f17737fe640 1 -- 172.21.15.67:0/1472887201 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6332+0+0 (secure 0 0 0) 0x7f177408a1c0 con 0x7f177c0b0f10 2024-03-24T00:42:58.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.567+0000 7f177a7fc640 1 --2- 172.21.15.67:0/1472887201 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f175c06bbd0 0x7f175c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:58.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.567+0000 7f177a7fc640 1 --2- 172.21.15.67:0/1472887201 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f175c06bbd0 0x7f175c06e090 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f1764036000 tx=0x7f176403b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:58.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.571+0000 7f17737fe640 1 -- 172.21.15.67:0/1472887201 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1774058a40 con 0x7f177c0b0f10 2024-03-24T00:42:58.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:58.792+0000 7f17814c6640 1 -- 172.21.15.67:0/1472887201 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f177c0b1810 con 0x7f177c0b0f10 2024-03-24T00:42:59.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.441+0000 7f17737fe640 1 -- 172.21.15.67:0/1472887201 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v36) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1774058340 con 0x7f177c0b0f10 2024-03-24T00:42:59.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.443+0000 7f17814c6640 1 -- 172.21.15.67:0/1472887201 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f175c06bbd0 msgr2=0x7f175c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:59.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.443+0000 7f17814c6640 1 --2- 172.21.15.67:0/1472887201 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f175c06bbd0 0x7f175c06e090 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f1764036000 tx=0x7f176403b040 comp rx=0 tx=0).stop 2024-03-24T00:42:59.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.443+0000 7f17814c6640 1 -- 172.21.15.67:0/1472887201 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f177c0b0f10 msgr2=0x7f177c06d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:59.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.443+0000 7f17814c6640 1 --2- 172.21.15.67:0/1472887201 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f177c0b0f10 0x7f177c06d630 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f177400c3c0 tx=0x7f177400c880 comp rx=0 tx=0).stop 2024-03-24T00:42:59.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.443+0000 7f17814c6640 1 -- 172.21.15.67:0/1472887201 shutdown_connections 2024-03-24T00:42:59.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.443+0000 7f17814c6640 1 --2- 172.21.15.67:0/1472887201 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f175c06bbd0 0x7f175c06e090 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:59.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.443+0000 7f17814c6640 1 --2- 172.21.15.67:0/1472887201 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f177c0d22a0 0x7f177c1e7a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:59.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.443+0000 7f17814c6640 1 --2- 172.21.15.67:0/1472887201 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f177c0b2440 0x7f177c06db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:59.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.443+0000 7f17814c6640 1 --2- 172.21.15.67:0/1472887201 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f177c0b0f10 0x7f177c06d630 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:59.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.443+0000 7f17814c6640 1 -- 172.21.15.67:0/1472887201 >> 172.21.15.67:0/1472887201 conn(0x7f177c08b1b0 msgr2=0x7f177c071560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:59.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.444+0000 7f17814c6640 1 -- 172.21.15.67:0/1472887201 shutdown_connections 2024-03-24T00:42:59.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.444+0000 7f17814c6640 1 -- 172.21.15.67:0/1472887201 wait complete. 2024-03-24T00:42:59.458 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-24T00:42:59.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.611+0000 7f73050ff640 1 Processor -- start 2024-03-24T00:42:59.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.612+0000 7f73050ff640 1 -- start start 2024-03-24T00:42:59.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.612+0000 7f73050ff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f73000d00b0 0x7f73000d0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:59.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.612+0000 7f73050ff640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f73000cd230 0x7f7300161b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:59.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.612+0000 7f73050ff640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73000703c0 con 0x7f73000cdb50 2024-03-24T00:42:59.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.612+0000 7f73050ff640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7300070170 con 0x7f73000d00b0 2024-03-24T00:42:59.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.612+0000 7f73050ff640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73000706a0 con 0x7f73000cd230 2024-03-24T00:42:59.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.613+0000 7f72f7fff640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f73000cd230 0x7f7300161b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:59.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.613+0000 7f72ff59e640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f73000cdb50 0x7f730006b040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:55930/0 (socket says 172.21.15.67:55930) 2024-03-24T00:42:59.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.613+0000 7f72ff59e640 1 -- 172.21.15.67:0/1528021499 learned_addr learned my addr 172.21.15.67:0/1528021499 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:42:59.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.613+0000 7f72fed9d640 1 --2- 172.21.15.67:0/1528021499 >> v2:172.21.15.72:3300/0 conn(0x7f73000d00b0 0x7f73000d0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:59.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.613+0000 7f72fe59c640 1 -- 172.21.15.67:0/1528021499 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4208965527 0 0) 0x7f73000703c0 con 0x7f73000cdb50 2024-03-24T00:42:59.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.613+0000 7f72fe59c640 1 -- 172.21.15.67:0/1528021499 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72ec003520 con 0x7f73000cdb50 2024-03-24T00:42:59.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.613+0000 7f72f7fff640 1 -- 172.21.15.67:0/1528021499 >> v2:172.21.15.72:3300/0 conn(0x7f73000d00b0 msgr2=0x7f73000d0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:42:59.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.613+0000 7f72f7fff640 1 --2- 172.21.15.67:0/1528021499 >> v2:172.21.15.72:3300/0 conn(0x7f73000d00b0 0x7f73000d0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:59.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.613+0000 7f72f7fff640 1 -- 172.21.15.67:0/1528021499 >> v1:172.21.15.72:6789/0 conn(0x7f73000cdb50 legacy=0x7f730006b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:59.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.613+0000 7f72f7fff640 1 -- 172.21.15.67:0/1528021499 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7300070980 con 0x7f73000cd230 2024-03-24T00:42:59.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.613+0000 7f72fed9d640 1 --2- 172.21.15.67:0/1528021499 >> v2:172.21.15.72:3300/0 conn(0x7f73000d00b0 0x7f73000d0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:42:59.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.614+0000 7f72f7fff640 1 --2- 172.21.15.67:0/1528021499 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f73000cd230 0x7f7300161b10 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f72e80097a0 tx=0x7f72e802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ae008d8894b0df4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:59.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.614+0000 7f72fe59c640 1 -- 172.21.15.67:0/1528021499 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e80315a0 con 0x7f73000cd230 2024-03-24T00:42:59.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.614+0000 7f72fe59c640 1 -- 172.21.15.67:0/1528021499 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72e8037070 con 0x7f73000cd230 2024-03-24T00:42:59.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.614+0000 7f72fe59c640 1 -- 172.21.15.67:0/1528021499 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e8031940 con 0x7f73000cd230 2024-03-24T00:42:59.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.615+0000 7f73050ff640 1 -- 172.21.15.67:0/1528021499 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f73000cd230 msgr2=0x7f7300161b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:59.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.615+0000 7f73050ff640 1 --2- 172.21.15.67:0/1528021499 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f73000cd230 0x7f7300161b10 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f72e80097a0 tx=0x7f72e802b840 comp rx=0 tx=0).stop 2024-03-24T00:42:59.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.615+0000 7f73050ff640 1 -- 172.21.15.67:0/1528021499 shutdown_connections 2024-03-24T00:42:59.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.615+0000 7f73050ff640 1 --2- 172.21.15.67:0/1528021499 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f73000cd230 0x7f7300161b10 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:59.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.615+0000 7f73050ff640 1 --2- 172.21.15.67:0/1528021499 >> v2:172.21.15.72:3300/0 conn(0x7f73000d00b0 0x7f73000d0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:59.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.615+0000 7f73050ff640 1 -- 172.21.15.67:0/1528021499 >> 172.21.15.67:0/1528021499 conn(0x7f7300099a30 msgr2=0x7f7300099e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:42:59.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.615+0000 7f73050ff640 1 -- 172.21.15.67:0/1528021499 shutdown_connections 2024-03-24T00:42:59.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.615+0000 7f73050ff640 1 -- 172.21.15.67:0/1528021499 wait complete. 2024-03-24T00:42:59.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.616+0000 7f73050ff640 1 Processor -- start 2024-03-24T00:42:59.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.616+0000 7f73050ff640 1 -- start start 2024-03-24T00:42:59.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.617+0000 7f73050ff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73000cd230 0x7f7300159f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:59.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.617+0000 7f73050ff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73000cdb50 0x7f7300156030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:59.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.617+0000 7f73050ff640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f73000d00b0 0x7f73000c9740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:59.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.617+0000 7f73050ff640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73000cdfe0 con 0x7f73000cdb50 2024-03-24T00:42:59.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.617+0000 7f73050ff640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7300162050 con 0x7f73000cd230 2024-03-24T00:42:59.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.617+0000 7f73050ff640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73000ce160 con 0x7f73000d00b0 2024-03-24T00:42:59.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.617+0000 7f72f7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73000cdb50 0x7f7300156030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:59.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.617+0000 7f72f7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73000cdb50 0x7f7300156030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42984/0 (socket says 172.21.15.67:42984) 2024-03-24T00:42:59.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.617+0000 7f72f7fff640 1 -- 172.21.15.67:0/2684276087 learned_addr learned my addr 172.21.15.67:0/2684276087 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:42:59.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.617+0000 7f72ff59e640 1 --2- 172.21.15.67:0/2684276087 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f73000d00b0 0x7f73000c9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:59.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.617+0000 7f72fed9d640 1 --2- 172.21.15.67:0/2684276087 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73000cd230 0x7f7300159f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:59.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.617+0000 7f72f7fff640 1 -- 172.21.15.67:0/2684276087 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f73000d00b0 msgr2=0x7f73000c9740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:59.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.617+0000 7f72f7fff640 1 --2- 172.21.15.67:0/2684276087 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f73000d00b0 0x7f73000c9740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:59.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.617+0000 7f72f7fff640 1 -- 172.21.15.67:0/2684276087 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73000cd230 msgr2=0x7f7300159f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:42:59.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.617+0000 7f72f7fff640 1 --2- 172.21.15.67:0/2684276087 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73000cd230 0x7f7300159f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:42:59.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.617+0000 7f72f7fff640 1 -- 172.21.15.67:0/2684276087 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7300157770 con 0x7f73000cdb50 2024-03-24T00:42:59.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.618+0000 7f72f7fff640 1 --2- 172.21.15.67:0/2684276087 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73000cdb50 0x7f7300156030 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f72e8034040 tx=0x7f72e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:59.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.618+0000 7f72f77fe640 1 -- 172.21.15.67:0/2684276087 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e8032630 con 0x7f73000cdb50 2024-03-24T00:42:59.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.618+0000 7f72f77fe640 1 -- 172.21.15.67:0/2684276087 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72e80327f0 con 0x7f73000cdb50 2024-03-24T00:42:59.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.618+0000 7f73050ff640 1 -- 172.21.15.67:0/2684276087 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73000c9e90 con 0x7f73000cdb50 2024-03-24T00:42:59.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.618+0000 7f72f77fe640 1 -- 172.21.15.67:0/2684276087 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e8032be0 con 0x7f73000cdb50 2024-03-24T00:42:59.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.618+0000 7f73050ff640 1 -- 172.21.15.67:0/2684276087 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73000b4c60 con 0x7f73000cdb50 2024-03-24T00:42:59.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.619+0000 7f73050ff640 1 -- 172.21.15.67:0/2684276087 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73000d1870 con 0x7f73000cdb50 2024-03-24T00:42:59.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.619+0000 7f72f77fe640 1 -- 172.21.15.67:0/2684276087 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f72e803b3f0 con 0x7f73000cdb50 2024-03-24T00:42:59.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.620+0000 7f72f77fe640 1 --2- 172.21.15.67:0/2684276087 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f72e006bc00 0x7f72e006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:42:59.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.620+0000 7f72f77fe640 1 -- 172.21.15.67:0/2684276087 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6332+0+0 (secure 0 0 0) 0x7f72e802f030 con 0x7f73000cdb50 2024-03-24T00:42:59.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.620+0000 7f72fed9d640 1 --2- 172.21.15.67:0/2684276087 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f72e006bc00 0x7f72e006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:42:59.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.620+0000 7f72fed9d640 1 --2- 172.21.15.67:0/2684276087 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f72e006bc00 0x7f72e006e0c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f72f0009560 tx=0x7f72f0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:42:59.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.624+0000 7f72f77fe640 1 -- 172.21.15.67:0/2684276087 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f73000d1870 con 0x7f73000cdb50 2024-03-24T00:42:59.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:42:59.805+0000 7f73050ff640 1 -- 172.21.15.67:0/2684276087 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f730004f400 con 0x7f73000cdb50 2024-03-24T00:43:00.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.442+0000 7f72f77fe640 1 -- 172.21.15.67:0/2684276087 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v66) v1 ==== 181+0+0 (secure 0 0 0) 0x7f72e8040070 con 0x7f73000cdb50 2024-03-24T00:43:00.443 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' removed 2024-03-24T00:43:00.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.445+0000 7f73050ff640 1 -- 172.21.15.67:0/2684276087 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f72e006bc00 msgr2=0x7f72e006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:00.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.445+0000 7f73050ff640 1 --2- 172.21.15.67:0/2684276087 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f72e006bc00 0x7f72e006e0c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f72f0009560 tx=0x7f72f0009190 comp rx=0 tx=0).stop 2024-03-24T00:43:00.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.445+0000 7f73050ff640 1 -- 172.21.15.67:0/2684276087 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73000cdb50 msgr2=0x7f7300156030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:00.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.445+0000 7f73050ff640 1 --2- 172.21.15.67:0/2684276087 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73000cdb50 0x7f7300156030 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f72e8034040 tx=0x7f72e80046c0 comp rx=0 tx=0).stop 2024-03-24T00:43:00.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.446+0000 7f73050ff640 1 -- 172.21.15.67:0/2684276087 shutdown_connections 2024-03-24T00:43:00.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.446+0000 7f73050ff640 1 --2- 172.21.15.67:0/2684276087 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f72e006bc00 0x7f72e006e0c0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:00.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.446+0000 7f73050ff640 1 --2- 172.21.15.67:0/2684276087 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f73000d00b0 0x7f73000c9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:00.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.446+0000 7f73050ff640 1 --2- 172.21.15.67:0/2684276087 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73000cdb50 0x7f7300156030 secure :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f72e8034040 tx=0x7f72e80046c0 comp rx=0 tx=0).stop 2024-03-24T00:43:00.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.446+0000 7f73050ff640 1 --2- 172.21.15.67:0/2684276087 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73000cd230 0x7f7300159f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:00.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.446+0000 7f73050ff640 1 -- 172.21.15.67:0/2684276087 >> 172.21.15.67:0/2684276087 conn(0x7f7300099a30 msgr2=0x7f73000af860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:00.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.446+0000 7f73050ff640 1 -- 172.21.15.67:0/2684276087 shutdown_connections 2024-03-24T00:43:00.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.446+0000 7f73050ff640 1 -- 172.21.15.67:0/2684276087 wait complete. 2024-03-24T00:43:00.461 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-24T00:43:00.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.571+0000 7f9d8c8c6640 1 Processor -- start 2024-03-24T00:43:00.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.571+0000 7f9d8c8c6640 1 -- start start 2024-03-24T00:43:00.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.571+0000 7f9d8c8c6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9d88140780 0x7f9d88072360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:00.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.571+0000 7f9d8c8c6640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9d880b4440 0x7f9d8806e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:00.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.571+0000 7f9d8c8c6640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d880b4910 con 0x7f9d880728a0 2024-03-24T00:43:00.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.571+0000 7f9d8c8c6640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d880cbf30 con 0x7f9d880b4440 2024-03-24T00:43:00.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.571+0000 7f9d8c8c6640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d88072d00 con 0x7f9d88140780 2024-03-24T00:43:00.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.572+0000 7f9d86d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9d880b4440 0x7f9d8806e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:00.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.572+0000 7f9d86d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9d880b4440 0x7f9d8806e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42990/0 (socket says 172.21.15.67:42990) 2024-03-24T00:43:00.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.572+0000 7f9d86d9d640 1 -- 172.21.15.67:0/2717552565 learned_addr learned my addr 172.21.15.67:0/2717552565 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:00.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.572+0000 7f9d8659c640 1 --2- 172.21.15.67:0/2717552565 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9d88140780 0x7f9d88072360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:00.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.572+0000 7f9d86d9d640 1 -- 172.21.15.67:0/2717552565 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9d88140780 msgr2=0x7f9d88072360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:00.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.572+0000 7f9d86d9d640 1 --2- 172.21.15.67:0/2717552565 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9d88140780 0x7f9d88072360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:00.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.572+0000 7f9d86d9d640 1 -- 172.21.15.67:0/2717552565 >> v1:172.21.15.72:6789/0 conn(0x7f9d880728a0 legacy=0x7f9d880b3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:00.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.572+0000 7f9d86d9d640 1 -- 172.21.15.67:0/2717552565 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d880b4bf0 con 0x7f9d880b4440 2024-03-24T00:43:00.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.573+0000 7f9d86d9d640 1 --2- 172.21.15.67:0/2717552565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d880b4440 0x7f9d8806e580 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f9d880af610 tx=0x7f9d8002cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25edc5d9a2fa2793 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:00.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.573+0000 7f9d85d9b640 1 -- 172.21.15.67:0/2717552565 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8002dab0 con 0x7f9d880b4440 2024-03-24T00:43:00.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.573+0000 7f9d85d9b640 1 -- 172.21.15.67:0/2717552565 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d8002dc50 con 0x7f9d880b4440 2024-03-24T00:43:00.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.573+0000 7f9d85d9b640 1 -- 172.21.15.67:0/2717552565 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d800353f0 con 0x7f9d880b4440 2024-03-24T00:43:00.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.573+0000 7f9d8c8c6640 1 -- 172.21.15.67:0/2717552565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d880b4440 msgr2=0x7f9d8806e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:00.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.573+0000 7f9d8c8c6640 1 --2- 172.21.15.67:0/2717552565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d880b4440 0x7f9d8806e580 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f9d880af610 tx=0x7f9d8002cf10 comp rx=0 tx=0).stop 2024-03-24T00:43:00.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.573+0000 7f9d8c8c6640 1 -- 172.21.15.67:0/2717552565 shutdown_connections 2024-03-24T00:43:00.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.573+0000 7f9d8c8c6640 1 --2- 172.21.15.67:0/2717552565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d880b4440 0x7f9d8806e580 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:00.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.573+0000 7f9d8c8c6640 1 --2- 172.21.15.67:0/2717552565 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9d88140780 0x7f9d88072360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:00.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.573+0000 7f9d8c8c6640 1 -- 172.21.15.67:0/2717552565 >> 172.21.15.67:0/2717552565 conn(0x7f9d880cf9e0 msgr2=0x7f9d880cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:00.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.573+0000 7f9d8c8c6640 1 -- 172.21.15.67:0/2717552565 shutdown_connections 2024-03-24T00:43:00.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.573+0000 7f9d8c8c6640 1 -- 172.21.15.67:0/2717552565 wait complete. 2024-03-24T00:43:00.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.575+0000 7f9d8c8c6640 1 Processor -- start 2024-03-24T00:43:00.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.575+0000 7f9d8c8c6640 1 -- start start 2024-03-24T00:43:00.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.575+0000 7f9d8c8c6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d880728a0 0x7f9d8815d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:00.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.575+0000 7f9d8c8c6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d880b4440 0x7f9d88139110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:00.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.575+0000 7f9d8c8c6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9d88140780 0x7f9d88139650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:00.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.575+0000 7f9d8c8c6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8806e060 con 0x7f9d880b4440 2024-03-24T00:43:00.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.575+0000 7f9d8c8c6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8806db30 con 0x7f9d880728a0 2024-03-24T00:43:00.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.575+0000 7f9d8c8c6640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d88071ad0 con 0x7f9d88140780 2024-03-24T00:43:00.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.576+0000 7f9d7ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d880b4440 0x7f9d88139110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:00.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.576+0000 7f9d7ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d880b4440 0x7f9d88139110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43006/0 (socket says 172.21.15.67:43006) 2024-03-24T00:43:00.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.576+0000 7f9d7ffff640 1 -- 172.21.15.67:0/2463390173 learned_addr learned my addr 172.21.15.67:0/2463390173 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:00.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.576+0000 7f9d8659c640 1 --2- 172.21.15.67:0/2463390173 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d880728a0 0x7f9d8815d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:00.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.576+0000 7f9d86d9d640 1 --2- 172.21.15.67:0/2463390173 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9d88140780 0x7f9d88139650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:00.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.576+0000 7f9d7ffff640 1 -- 172.21.15.67:0/2463390173 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9d88140780 msgr2=0x7f9d88139650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:00.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.576+0000 7f9d7ffff640 1 --2- 172.21.15.67:0/2463390173 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9d88140780 0x7f9d88139650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:00.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.576+0000 7f9d7ffff640 1 -- 172.21.15.67:0/2463390173 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d880728a0 msgr2=0x7f9d8815d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:00.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.576+0000 7f9d7ffff640 1 --2- 172.21.15.67:0/2463390173 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d880728a0 0x7f9d8815d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:00.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.576+0000 7f9d7ffff640 1 -- 172.21.15.67:0/2463390173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d8815a800 con 0x7f9d880b4440 2024-03-24T00:43:00.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.576+0000 7f9d7ffff640 1 --2- 172.21.15.67:0/2463390173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d880b4440 0x7f9d88139110 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f9d7000c430 tx=0x7f9d7000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:00.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.576+0000 7f9d7effd640 1 -- 172.21.15.67:0/2463390173 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d70011790 con 0x7f9d880b4440 2024-03-24T00:43:00.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.576+0000 7f9d7effd640 1 -- 172.21.15.67:0/2463390173 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d70017070 con 0x7f9d880b4440 2024-03-24T00:43:00.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.576+0000 7f9d7effd640 1 -- 172.21.15.67:0/2463390173 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d70011bb0 con 0x7f9d880b4440 2024-03-24T00:43:00.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.576+0000 7f9d8c8c6640 1 -- 172.21.15.67:0/2463390173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d8813b4b0 con 0x7f9d880b4440 2024-03-24T00:43:00.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.576+0000 7f9d8c8c6640 1 -- 172.21.15.67:0/2463390173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d8815b080 con 0x7f9d880b4440 2024-03-24T00:43:00.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.577+0000 7f9d8c8c6640 1 -- 172.21.15.67:0/2463390173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d8815b280 con 0x7f9d880b4440 2024-03-24T00:43:00.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.578+0000 7f9d7effd640 1 -- 172.21.15.67:0/2463390173 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9d7001c460 con 0x7f9d880b4440 2024-03-24T00:43:00.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.578+0000 7f9d7effd640 1 --2- 172.21.15.67:0/2463390173 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9d6806bbd0 0x7f9d6806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:00.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.578+0000 7f9d7effd640 1 -- 172.21.15.67:0/2463390173 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5865+0+0 (secure 0 0 0) 0x7f9d7008a8f0 con 0x7f9d880b4440 2024-03-24T00:43:00.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.578+0000 7f9d8659c640 1 --2- 172.21.15.67:0/2463390173 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9d6806bbd0 0x7f9d6806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:00.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.579+0000 7f9d8659c640 1 --2- 172.21.15.67:0/2463390173 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9d6806bbd0 0x7f9d6806e090 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f9d780079a0 tx=0x7f9d78008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:00.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.583+0000 7f9d7effd640 1 -- 172.21.15.67:0/2463390173 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9d8815b280 con 0x7f9d880b4440 2024-03-24T00:43:00.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:00.764+0000 7f9d8c8c6640 1 -- 172.21.15.67:0/2463390173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9d8804f400 con 0x7f9d880b4440 2024-03-24T00:43:01.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.444+0000 7f9d7effd640 1 -- 172.21.15.67:0/2463390173 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v67) v1 ==== 169+0+0 (secure 0 0 0) 0x7f9d70058c70 con 0x7f9d880b4440 2024-03-24T00:43:01.445 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' removed 2024-03-24T00:43:01.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.447+0000 7f9d8c8c6640 1 -- 172.21.15.67:0/2463390173 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9d6806bbd0 msgr2=0x7f9d6806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:01.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.447+0000 7f9d8c8c6640 1 --2- 172.21.15.67:0/2463390173 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9d6806bbd0 0x7f9d6806e090 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f9d780079a0 tx=0x7f9d78008040 comp rx=0 tx=0).stop 2024-03-24T00:43:01.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.447+0000 7f9d8c8c6640 1 -- 172.21.15.67:0/2463390173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d880b4440 msgr2=0x7f9d88139110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:01.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.447+0000 7f9d8c8c6640 1 --2- 172.21.15.67:0/2463390173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d880b4440 0x7f9d88139110 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f9d7000c430 tx=0x7f9d7000c8f0 comp rx=0 tx=0).stop 2024-03-24T00:43:01.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.448+0000 7f9d8c8c6640 1 -- 172.21.15.67:0/2463390173 shutdown_connections 2024-03-24T00:43:01.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.448+0000 7f9d8c8c6640 1 --2- 172.21.15.67:0/2463390173 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9d6806bbd0 0x7f9d6806e090 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:01.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.448+0000 7f9d8c8c6640 1 --2- 172.21.15.67:0/2463390173 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9d88140780 0x7f9d88139650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:01.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.448+0000 7f9d8c8c6640 1 --2- 172.21.15.67:0/2463390173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d880b4440 0x7f9d88139110 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:01.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.448+0000 7f9d8c8c6640 1 --2- 172.21.15.67:0/2463390173 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d880728a0 0x7f9d8815d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:01.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.448+0000 7f9d8c8c6640 1 -- 172.21.15.67:0/2463390173 >> 172.21.15.67:0/2463390173 conn(0x7f9d880cf9e0 msgr2=0x7f9d8806f260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:01.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.448+0000 7f9d8c8c6640 1 -- 172.21.15.67:0/2463390173 shutdown_connections 2024-03-24T00:43:01.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.448+0000 7f9d8c8c6640 1 -- 172.21.15.67:0/2463390173 wait complete. 2024-03-24T00:43:01.463 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-24T00:43:01.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.569+0000 7faa58af8640 1 Processor -- start 2024-03-24T00:43:01.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.569+0000 7faa58af8640 1 -- start start 2024-03-24T00:43:01.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.570+0000 7faa58af8640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7faa5406f370 0x7faa540b3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:01.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.570+0000 7faa58af8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa540b2ab0 con 0x7faa540b4a00 2024-03-24T00:43:01.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.570+0000 7faa58af8640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa540b2d90 con 0x7faa54144670 2024-03-24T00:43:01.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.570+0000 7faa58af8640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa540b3070 con 0x7faa5406f370 2024-03-24T00:43:01.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.570+0000 7faa51d9b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7faa540b4a00 0x7faa5406ec60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:32780/0 (socket says 172.21.15.67:32780) 2024-03-24T00:43:01.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.570+0000 7faa51d9b640 1 -- 172.21.15.67:0/291860431 learned_addr learned my addr 172.21.15.67:0/291860431 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:43:01.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.570+0000 7faa52d9d640 1 --2- 172.21.15.67:0/291860431 >> v2:172.21.15.72:3300/0 conn(0x7faa5406f370 0x7faa540b3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:01.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.570+0000 7faa5159a640 1 -- 172.21.15.67:0/291860431 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1581057001 0 0) 0x7faa540b2ab0 con 0x7faa540b4a00 2024-03-24T00:43:01.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.570+0000 7faa5159a640 1 -- 172.21.15.67:0/291860431 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa3c003520 con 0x7faa540b4a00 2024-03-24T00:43:01.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.570+0000 7faa5159a640 1 -- 172.21.15.67:0/291860431 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1517110820 0 0) 0x7faa3c003520 con 0x7faa540b4a00 2024-03-24T00:43:01.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.570+0000 7faa5159a640 1 -- 172.21.15.67:0/291860431 >> v2:172.21.15.72:3300/0 conn(0x7faa5406f370 msgr2=0x7faa540b3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:01.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.570+0000 7faa5159a640 1 --2- 172.21.15.67:0/291860431 >> v2:172.21.15.72:3300/0 conn(0x7faa5406f370 0x7faa540b3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:01.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.570+0000 7faa5159a640 1 -- 172.21.15.67:0/291860431 >> v1:172.21.15.72:6789/0 conn(0x7faa54144670 legacy=0x7faa54072500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:01.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.570+0000 7faa5159a640 1 -- 172.21.15.67:0/291860431 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa540b3350 con 0x7faa540b4a00 2024-03-24T00:43:01.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.570+0000 7faa5159a640 1 -- 172.21.15.67:0/291860431 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7faa48001670 con 0x7faa540b4a00 2024-03-24T00:43:01.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.570+0000 7faa5159a640 1 -- 172.21.15.67:0/291860431 >> v1:172.21.15.67:6789/0 conn(0x7faa540b4a00 legacy=0x7faa5406ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:01.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.570+0000 7faa5159a640 1 --2- 172.21.15.67:0/291860431 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa3c0039b0 0x7faa3c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:01.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.570+0000 7faa5159a640 1 --2- 172.21.15.67:0/291860431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa3c004800 0x7faa3c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:01.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.570+0000 7faa5159a640 1 --2- 172.21.15.67:0/291860431 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faa3c0051c0 0x7faa3c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:01.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.570+0000 7faa5159a640 1 -- 172.21.15.67:0/291860431 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa540b3070 con 0x7faa3c004800 2024-03-24T00:43:01.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.570+0000 7faa5159a640 1 -- 172.21.15.67:0/291860431 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa3c00c060 con 0x7faa3c0039b0 2024-03-24T00:43:01.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.570+0000 7faa5159a640 1 -- 172.21.15.67:0/291860431 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa3c00c260 con 0x7faa3c0051c0 2024-03-24T00:43:01.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.571+0000 7faa51d9b640 1 --2- 172.21.15.67:0/291860431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa3c004800 0x7faa3c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:01.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.571+0000 7faa5259c640 1 --2- 172.21.15.67:0/291860431 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa3c0039b0 0x7faa3c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:01.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.571+0000 7faa52d9d640 1 --2- 172.21.15.67:0/291860431 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faa3c0051c0 0x7faa3c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:01.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.571+0000 7faa51d9b640 1 -- 172.21.15.67:0/291860431 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faa3c0051c0 msgr2=0x7faa3c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:01.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.571+0000 7faa51d9b640 1 --2- 172.21.15.67:0/291860431 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faa3c0051c0 0x7faa3c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:01.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.571+0000 7faa51d9b640 1 -- 172.21.15.67:0/291860431 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa3c0039b0 msgr2=0x7faa3c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:01.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.571+0000 7faa51d9b640 1 --2- 172.21.15.67:0/291860431 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa3c0039b0 0x7faa3c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:01.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.571+0000 7faa51d9b640 1 -- 172.21.15.67:0/291860431 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa3c003520 con 0x7faa3c004800 2024-03-24T00:43:01.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.571+0000 7faa52d9d640 1 --2- 172.21.15.67:0/291860431 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faa3c0051c0 0x7faa3c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:43:01.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.571+0000 7faa52d9d640 1 -- 172.21.15.67:0/291860431 reap_dead start 2024-03-24T00:43:01.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.572+0000 7faa51d9b640 1 --2- 172.21.15.67:0/291860431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa3c004800 0x7faa3c004c80 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7faa4800c240 tx=0x7faa3c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bb41375a803602e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:01.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.572+0000 7faa5159a640 1 -- 172.21.15.67:0/291860431 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa48007d10 con 0x7faa3c004800 2024-03-24T00:43:01.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.572+0000 7faa5159a640 1 -- 172.21.15.67:0/291860431 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa48030070 con 0x7faa3c004800 2024-03-24T00:43:01.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.572+0000 7faa5159a640 1 -- 172.21.15.67:0/291860431 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa48005d90 con 0x7faa3c004800 2024-03-24T00:43:01.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.572+0000 7faa58af8640 1 -- 172.21.15.67:0/291860431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa3c004800 msgr2=0x7faa3c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:01.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.572+0000 7faa58af8640 1 --2- 172.21.15.67:0/291860431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa3c004800 0x7faa3c004c80 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7faa4800c240 tx=0x7faa3c005830 comp rx=0 tx=0).stop 2024-03-24T00:43:01.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.572+0000 7faa58af8640 1 -- 172.21.15.67:0/291860431 shutdown_connections 2024-03-24T00:43:01.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.572+0000 7faa58af8640 1 --2- 172.21.15.67:0/291860431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa3c004800 0x7faa3c004c80 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:01.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.572+0000 7faa58af8640 1 -- 172.21.15.67:0/291860431 >> 172.21.15.67:0/291860431 conn(0x7faa540cf9e0 msgr2=0x7faa540cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:01.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.572+0000 7faa58af8640 1 -- 172.21.15.67:0/291860431 shutdown_connections 2024-03-24T00:43:01.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.572+0000 7faa58af8640 1 -- 172.21.15.67:0/291860431 wait complete. 2024-03-24T00:43:01.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.574+0000 7faa58af8640 1 Processor -- start 2024-03-24T00:43:01.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.574+0000 7faa58af8640 1 -- start start 2024-03-24T00:43:01.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.574+0000 7faa58af8640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faa3c004800 0x7faa5413e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:01.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.574+0000 7faa58af8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa540adf20 0x7faa540ae3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:01.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.574+0000 7faa58af8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa54142220 0x7faa541426a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:01.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.574+0000 7faa58af8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa540b39f0 con 0x7faa54142220 2024-03-24T00:43:01.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.574+0000 7faa58af8640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa540b3870 con 0x7faa540adf20 2024-03-24T00:43:01.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.574+0000 7faa58af8640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa540b3b70 con 0x7faa3c004800 2024-03-24T00:43:01.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.574+0000 7faa52d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa54142220 0x7faa541426a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:01.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.574+0000 7faa52d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa54142220 0x7faa541426a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43032/0 (socket says 172.21.15.67:43032) 2024-03-24T00:43:01.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.574+0000 7faa52d9d640 1 -- 172.21.15.67:0/2590128698 learned_addr learned my addr 172.21.15.67:0/2590128698 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:01.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.575+0000 7faa5259c640 1 --2- 172.21.15.67:0/2590128698 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faa3c004800 0x7faa5413e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:01.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.575+0000 7faa51d9b640 1 --2- 172.21.15.67:0/2590128698 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa540adf20 0x7faa540ae3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:01.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.575+0000 7faa52d9d640 1 -- 172.21.15.67:0/2590128698 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faa3c004800 msgr2=0x7faa5413e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:01.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.575+0000 7faa52d9d640 1 --2- 172.21.15.67:0/2590128698 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faa3c004800 0x7faa5413e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:01.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.575+0000 7faa52d9d640 1 -- 172.21.15.67:0/2590128698 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa540adf20 msgr2=0x7faa540ae3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:01.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.575+0000 7faa52d9d640 1 --2- 172.21.15.67:0/2590128698 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa540adf20 0x7faa540ae3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:01.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.575+0000 7faa52d9d640 1 -- 172.21.15.67:0/2590128698 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa5413ea30 con 0x7faa54142220 2024-03-24T00:43:01.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.575+0000 7faa5259c640 1 --2- 172.21.15.67:0/2590128698 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faa3c004800 0x7faa5413e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:43:01.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.575+0000 7faa51d9b640 1 --2- 172.21.15.67:0/2590128698 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa540adf20 0x7faa540ae3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:43:01.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.575+0000 7faa52d9d640 1 --2- 172.21.15.67:0/2590128698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa54142220 0x7faa541426a0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7faa5406c0e0 tx=0x7faa4c010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:01.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.575+0000 7faa42ffd640 1 -- 172.21.15.67:0/2590128698 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa4c01a030 con 0x7faa54142220 2024-03-24T00:43:01.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.575+0000 7faa42ffd640 1 -- 172.21.15.67:0/2590128698 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa4c010dd0 con 0x7faa54142220 2024-03-24T00:43:01.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.575+0000 7faa58af8640 1 -- 172.21.15.67:0/2590128698 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa540a96f0 con 0x7faa54142220 2024-03-24T00:43:01.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.575+0000 7faa42ffd640 1 -- 172.21.15.67:0/2590128698 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa4c003bd0 con 0x7faa54142220 2024-03-24T00:43:01.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.575+0000 7faa58af8640 1 -- 172.21.15.67:0/2590128698 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa540aedc0 con 0x7faa54142220 2024-03-24T00:43:01.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.576+0000 7faa58af8640 1 -- 172.21.15.67:0/2590128698 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa5404f400 con 0x7faa54142220 2024-03-24T00:43:01.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.576+0000 7faa42ffd640 1 -- 172.21.15.67:0/2590128698 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7faa4c005d10 con 0x7faa54142220 2024-03-24T00:43:01.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.577+0000 7faa42ffd640 1 --2- 172.21.15.67:0/2590128698 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7faa3406bbd0 0x7faa3406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:01.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.577+0000 7faa42ffd640 1 -- 172.21.15.67:0/2590128698 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5335+0+0 (secure 0 0 0) 0x7faa4c08af70 con 0x7faa54142220 2024-03-24T00:43:01.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.577+0000 7faa5259c640 1 --2- 172.21.15.67:0/2590128698 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7faa3406bbd0 0x7faa3406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:01.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.578+0000 7faa5259c640 1 --2- 172.21.15.67:0/2590128698 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7faa3406bbd0 0x7faa3406e090 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7faa44000c00 tx=0x7faa44009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:01.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.581+0000 7faa42ffd640 1 -- 172.21.15.67:0/2590128698 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7faa4c025020 con 0x7faa54142220 2024-03-24T00:43:01.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.764+0000 7faa58af8640 1 -- 172.21.15.67:0/2590128698 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7faa5415e4c0 con 0x7faa54142220 2024-03-24T00:43:01.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.767+0000 7faa42ffd640 1 -- 172.21.15.67:0/2590128698 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v15) v1 ==== 104+0+0 (secure 0 0 0) 0x7faa5413ea30 con 0x7faa54142220 2024-03-24T00:43:01.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.769+0000 7faa58af8640 1 -- 172.21.15.67:0/2590128698 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7faa3406bbd0 msgr2=0x7faa3406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:01.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.769+0000 7faa58af8640 1 --2- 172.21.15.67:0/2590128698 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7faa3406bbd0 0x7faa3406e090 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7faa44000c00 tx=0x7faa44009040 comp rx=0 tx=0).stop 2024-03-24T00:43:01.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.769+0000 7faa58af8640 1 -- 172.21.15.67:0/2590128698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa54142220 msgr2=0x7faa541426a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:01.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.769+0000 7faa58af8640 1 --2- 172.21.15.67:0/2590128698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa54142220 0x7faa541426a0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7faa5406c0e0 tx=0x7faa4c010590 comp rx=0 tx=0).stop 2024-03-24T00:43:01.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.769+0000 7faa58af8640 1 -- 172.21.15.67:0/2590128698 shutdown_connections 2024-03-24T00:43:01.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.769+0000 7faa58af8640 1 --2- 172.21.15.67:0/2590128698 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7faa3406bbd0 0x7faa3406e090 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:01.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.769+0000 7faa58af8640 1 --2- 172.21.15.67:0/2590128698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa54142220 0x7faa541426a0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:01.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.769+0000 7faa58af8640 1 --2- 172.21.15.67:0/2590128698 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa540adf20 0x7faa540ae3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:01.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.769+0000 7faa58af8640 1 --2- 172.21.15.67:0/2590128698 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faa3c004800 0x7faa5413e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:01.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.769+0000 7faa58af8640 1 -- 172.21.15.67:0/2590128698 >> 172.21.15.67:0/2590128698 conn(0x7faa540cf9e0 msgr2=0x7faa5406e950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:01.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.769+0000 7faa58af8640 1 -- 172.21.15.67:0/2590128698 shutdown_connections 2024-03-24T00:43:01.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.769+0000 7faa58af8640 1 -- 172.21.15.67:0/2590128698 wait complete. 2024-03-24T00:43:01.780 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_backtrace_repair' 2024-03-24T00:43:01.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.881+0000 7f48d7553640 1 Processor -- start 2024-03-24T00:43:01.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.881+0000 7f48d7553640 1 -- start start 2024-03-24T00:43:01.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.881+0000 7f48d7553640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f48d015dec0 0x7f48d00ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:01.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.881+0000 7f48d7553640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f48d00d0ef0 0x7f48d0162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:01.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.881+0000 7f48d7553640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48d00cf640 con 0x7f48d015e2a0 2024-03-24T00:43:01.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.881+0000 7f48d7553640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48d00cee30 con 0x7f48d00d0ef0 2024-03-24T00:43:01.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.882+0000 7f48d7553640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48d00d2120 con 0x7f48d015dec0 2024-03-24T00:43:01.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.882+0000 7f48d5af0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f48d00d0ef0 0x7f48d0162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:01.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.882+0000 7f48d5af0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f48d00d0ef0 0x7f48d0162630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43038/0 (socket says 172.21.15.67:43038) 2024-03-24T00:43:01.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.882+0000 7f48d5af0640 1 -- 172.21.15.67:0/1268041138 learned_addr learned my addr 172.21.15.67:0/1268041138 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:01.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.882+0000 7f48d52ef640 1 --2- 172.21.15.67:0/1268041138 >> v2:172.21.15.72:3300/0 conn(0x7f48d015dec0 0x7f48d00ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:01.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.882+0000 7f48d5af0640 1 -- 172.21.15.67:0/1268041138 >> v2:172.21.15.72:3300/0 conn(0x7f48d015dec0 msgr2=0x7f48d00ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:01.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.882+0000 7f48d5af0640 1 --2- 172.21.15.67:0/1268041138 >> v2:172.21.15.72:3300/0 conn(0x7f48d015dec0 0x7f48d00ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:01.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.882+0000 7f48d5af0640 1 -- 172.21.15.67:0/1268041138 >> v1:172.21.15.72:6789/0 conn(0x7f48d015e2a0 legacy=0x7f48d00d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:01.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.882+0000 7f48d5af0640 1 -- 172.21.15.67:0/1268041138 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d00cf080 con 0x7f48d00d0ef0 2024-03-24T00:43:01.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.883+0000 7f48d5af0640 1 --2- 172.21.15.67:0/1268041138 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48d00d0ef0 0x7f48d0162630 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f48d00d14a0 tx=0x7f48c802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8aa2d520662396a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:01.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.883+0000 7f48c7fff640 1 -- 172.21.15.67:0/1268041138 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48c802ca80 con 0x7f48d00d0ef0 2024-03-24T00:43:01.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.883+0000 7f48c7fff640 1 -- 172.21.15.67:0/1268041138 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48c802cc20 con 0x7f48d00d0ef0 2024-03-24T00:43:01.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.883+0000 7f48c7fff640 1 -- 172.21.15.67:0/1268041138 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48c80343f0 con 0x7f48d00d0ef0 2024-03-24T00:43:01.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.883+0000 7f48d7553640 1 -- 172.21.15.67:0/1268041138 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48d00d0ef0 msgr2=0x7f48d0162630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:01.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.883+0000 7f48d7553640 1 --2- 172.21.15.67:0/1268041138 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48d00d0ef0 0x7f48d0162630 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f48d00d14a0 tx=0x7f48c802bee0 comp rx=0 tx=0).stop 2024-03-24T00:43:01.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.883+0000 7f48d7553640 1 -- 172.21.15.67:0/1268041138 shutdown_connections 2024-03-24T00:43:01.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.883+0000 7f48d7553640 1 --2- 172.21.15.67:0/1268041138 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48d00d0ef0 0x7f48d0162630 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:01.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.883+0000 7f48d7553640 1 --2- 172.21.15.67:0/1268041138 >> v2:172.21.15.72:3300/0 conn(0x7f48d015dec0 0x7f48d00ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:01.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.883+0000 7f48d7553640 1 -- 172.21.15.67:0/1268041138 >> 172.21.15.67:0/1268041138 conn(0x7f48d006c810 msgr2=0x7f48d006cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:01.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.884+0000 7f48d7553640 1 -- 172.21.15.67:0/1268041138 shutdown_connections 2024-03-24T00:43:01.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.884+0000 7f48d7553640 1 -- 172.21.15.67:0/1268041138 wait complete. 2024-03-24T00:43:01.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.885+0000 7f48d7553640 1 Processor -- start 2024-03-24T00:43:01.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.885+0000 7f48d7553640 1 -- start start 2024-03-24T00:43:01.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.885+0000 7f48d7553640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48d00d0ef0 0x7f48d01593e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:01.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.885+0000 7f48d7553640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f48d015dec0 0x7f48d00b5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:01.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.885+0000 7f48d7553640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f48d015e2a0 0x7f48d01f3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:01.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.885+0000 7f48d7553640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d00d2730 con 0x7f48d00d0ef0 2024-03-24T00:43:01.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.885+0000 7f48d7553640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d00d25b0 con 0x7f48d015e2a0 2024-03-24T00:43:01.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.885+0000 7f48d7553640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48d00d28b0 con 0x7f48d015dec0 2024-03-24T00:43:01.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.885+0000 7f48d52ef640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48d00d0ef0 0x7f48d01593e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:01.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.886+0000 7f48d52ef640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48d00d0ef0 0x7f48d01593e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43048/0 (socket says 172.21.15.67:43048) 2024-03-24T00:43:01.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.886+0000 7f48d52ef640 1 -- 172.21.15.67:0/265935680 learned_addr learned my addr 172.21.15.67:0/265935680 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:01.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.886+0000 7f48d4aee640 1 --2- 172.21.15.67:0/265935680 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f48d015dec0 0x7f48d00b5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:01.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.886+0000 7f48d5af0640 1 --2- 172.21.15.67:0/265935680 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f48d015e2a0 0x7f48d01f3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:01.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.886+0000 7f48d52ef640 1 -- 172.21.15.67:0/265935680 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f48d015dec0 msgr2=0x7f48d00b5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:01.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.886+0000 7f48d52ef640 1 --2- 172.21.15.67:0/265935680 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f48d015dec0 0x7f48d00b5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:01.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.886+0000 7f48d52ef640 1 -- 172.21.15.67:0/265935680 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f48d015e2a0 msgr2=0x7f48d01f3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:01.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.886+0000 7f48d52ef640 1 --2- 172.21.15.67:0/265935680 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f48d015e2a0 0x7f48d01f3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:01.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.886+0000 7f48d52ef640 1 -- 172.21.15.67:0/265935680 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d00b6000 con 0x7f48d00d0ef0 2024-03-24T00:43:01.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.886+0000 7f48d4aee640 1 --2- 172.21.15.67:0/265935680 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f48d015dec0 0x7f48d00b5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:43:01.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.886+0000 7f48d5af0640 1 --2- 172.21.15.67:0/265935680 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f48d015e2a0 0x7f48d01f3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:43:01.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.886+0000 7f48d52ef640 1 --2- 172.21.15.67:0/265935680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48d00d0ef0 0x7f48d01593e0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f48cc00e9b0 tx=0x7f48cc00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:01.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.886+0000 7f48c5ffb640 1 -- 172.21.15.67:0/265935680 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48cc017070 con 0x7f48d00d0ef0 2024-03-24T00:43:01.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.886+0000 7f48c5ffb640 1 -- 172.21.15.67:0/265935680 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48cc00c9a0 con 0x7f48d00d0ef0 2024-03-24T00:43:01.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.886+0000 7f48d7553640 1 -- 172.21.15.67:0/265935680 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48d00b62e0 con 0x7f48d00d0ef0 2024-03-24T00:43:01.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.886+0000 7f48c5ffb640 1 -- 172.21.15.67:0/265935680 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48cc00cd90 con 0x7f48d00d0ef0 2024-03-24T00:43:01.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.886+0000 7f48d7553640 1 -- 172.21.15.67:0/265935680 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48d00cdab0 con 0x7f48d00d0ef0 2024-03-24T00:43:01.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.887+0000 7f48d7553640 1 -- 172.21.15.67:0/265935680 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48d004f400 con 0x7f48d00d0ef0 2024-03-24T00:43:01.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.888+0000 7f48c5ffb640 1 -- 172.21.15.67:0/265935680 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f48cc01a070 con 0x7f48d00d0ef0 2024-03-24T00:43:01.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.888+0000 7f48c5ffb640 1 --2- 172.21.15.67:0/265935680 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f48b406bbd0 0x7f48b406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:01.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.888+0000 7f48c5ffb640 1 -- 172.21.15.67:0/265935680 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f48cc08ac50 con 0x7f48d00d0ef0 2024-03-24T00:43:01.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.888+0000 7f48d4aee640 1 --2- 172.21.15.67:0/265935680 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f48b406bbd0 0x7f48b406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:01.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.889+0000 7f48d4aee640 1 --2- 172.21.15.67:0/265935680 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f48b406bbd0 0x7f48b406e090 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f48d00ab830 tx=0x7f48c0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:01.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:01.893+0000 7f48c5ffb640 1 -- 172.21.15.67:0/265935680 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f48cc090020 con 0x7f48d00d0ef0 2024-03-24T00:43:02.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.144+0000 7f48d7553640 1 -- 172.21.15.67:0/265935680 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_backtrace_repair"]} v 0) v1 -- 0x7f48d00d1aa0 con 0x7f48d00d0ef0 2024-03-24T00:43:02.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.445+0000 7f48c5ffb640 1 -- 172.21.15.67:0/265935680 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_backtrace_repair"]}]=0 v221) v1 ==== 169+0+0 (secure 0 0 0) 0x7f48cc059940 con 0x7f48d00d0ef0 2024-03-24T00:43:02.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.448+0000 7f48d7553640 1 -- 172.21.15.67:0/265935680 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f48b406bbd0 msgr2=0x7f48b406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:02.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.448+0000 7f48d7553640 1 --2- 172.21.15.67:0/265935680 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f48b406bbd0 0x7f48b406e090 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f48d00ab830 tx=0x7f48c0009040 comp rx=0 tx=0).stop 2024-03-24T00:43:02.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.448+0000 7f48d7553640 1 -- 172.21.15.67:0/265935680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48d00d0ef0 msgr2=0x7f48d01593e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:02.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.448+0000 7f48d7553640 1 --2- 172.21.15.67:0/265935680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48d00d0ef0 0x7f48d01593e0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f48cc00e9b0 tx=0x7f48cc00ee70 comp rx=0 tx=0).stop 2024-03-24T00:43:02.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.449+0000 7f48d7553640 1 -- 172.21.15.67:0/265935680 shutdown_connections 2024-03-24T00:43:02.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.449+0000 7f48d7553640 1 --2- 172.21.15.67:0/265935680 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f48b406bbd0 0x7f48b406e090 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:02.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.449+0000 7f48d7553640 1 --2- 172.21.15.67:0/265935680 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f48d015e2a0 0x7f48d01f3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:02.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.449+0000 7f48d7553640 1 --2- 172.21.15.67:0/265935680 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f48d015dec0 0x7f48d00b5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:02.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.449+0000 7f48d7553640 1 --2- 172.21.15.67:0/265935680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48d00d0ef0 0x7f48d01593e0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:02.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.449+0000 7f48d7553640 1 -- 172.21.15.67:0/265935680 >> 172.21.15.67:0/265935680 conn(0x7f48d006c810 msgr2=0x7f48d015eae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:02.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.449+0000 7f48d7553640 1 -- 172.21.15.67:0/265935680 shutdown_connections 2024-03-24T00:43:02.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.449+0000 7f48d7553640 1 -- 172.21.15.67:0/265935680 wait complete. 2024-03-24T00:43:02.460 INFO:tasks.cephfs_test_runner:Test that the damage health status is cleared ... ok 2024-03-24T00:43:02.460 INFO:tasks.cephfs_test_runner:Starting test: test_health_status_after_dentry_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) Test that the damage health status is cleared 2024-03-24T00:43:02.461 INFO:tasks.cephfs_test_runner:test_health_status_after_dentry_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2024-03-24T00:43:02.461 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:43:02.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.562+0000 7f9199e27640 1 Processor -- start 2024-03-24T00:43:02.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.562+0000 7f9199e27640 1 -- start start 2024-03-24T00:43:02.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.563+0000 7f9199e27640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f91940d1550 0x7f919406cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:02.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.563+0000 7f9199e27640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f919409a480 0x7f91940b2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:02.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.563+0000 7f9199e27640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91940d1930 con 0x7f91940b2780 2024-03-24T00:43:02.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.563+0000 7f9199e27640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f919406d420 con 0x7f91940d1550 2024-03-24T00:43:02.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.563+0000 7f9199e27640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f919406ca50 con 0x7f919409a480 2024-03-24T00:43:02.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.563+0000 7f91937fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f91940d1550 0x7f919406cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:02.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.563+0000 7f9193fff640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f919409a480 0x7f91940b2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:02.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.563+0000 7f9192ffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f91940b2780 0x7f9194099d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:55952/0 (socket says 172.21.15.67:55952) 2024-03-24T00:43:02.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.563+0000 7f9192ffd640 1 -- 172.21.15.67:0/2563349889 learned_addr learned my addr 172.21.15.67:0/2563349889 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:43:02.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.563+0000 7f91927fc640 1 -- 172.21.15.67:0/2563349889 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4125480866 0 0) 0x7f91940d1930 con 0x7f91940b2780 2024-03-24T00:43:02.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.563+0000 7f91927fc640 1 -- 172.21.15.67:0/2563349889 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f917c003520 con 0x7f91940b2780 2024-03-24T00:43:02.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.563+0000 7f91937fe640 1 -- 172.21.15.67:0/2563349889 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f919409a480 msgr2=0x7f91940b2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:02.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.563+0000 7f91937fe640 1 --2- 172.21.15.67:0/2563349889 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f919409a480 0x7f91940b2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:02.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.563+0000 7f91937fe640 1 -- 172.21.15.67:0/2563349889 >> v1:172.21.15.72:6789/0 conn(0x7f91940b2780 legacy=0x7f9194099d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:02.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.563+0000 7f91937fe640 1 -- 172.21.15.67:0/2563349889 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91940b1f60 con 0x7f91940d1550 2024-03-24T00:43:02.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.563+0000 7f9193fff640 1 --2- 172.21.15.67:0/2563349889 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f919409a480 0x7f91940b2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:43:02.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.564+0000 7f91937fe640 1 --2- 172.21.15.67:0/2563349889 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91940d1550 0x7f919406cee0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f91840097d0 tx=0x7f918402b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e738ab37d602a3da server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:02.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.564+0000 7f91927fc640 1 -- 172.21.15.67:0/2563349889 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9184031770 con 0x7f91940d1550 2024-03-24T00:43:02.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.564+0000 7f91927fc640 1 -- 172.21.15.67:0/2563349889 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9184037070 con 0x7f91940d1550 2024-03-24T00:43:02.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.564+0000 7f91927fc640 1 -- 172.21.15.67:0/2563349889 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9184031b10 con 0x7f91940d1550 2024-03-24T00:43:02.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.565+0000 7f9199e27640 1 -- 172.21.15.67:0/2563349889 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91940d1550 msgr2=0x7f919406cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:02.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.565+0000 7f9199e27640 1 --2- 172.21.15.67:0/2563349889 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91940d1550 0x7f919406cee0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f91840097d0 tx=0x7f918402b810 comp rx=0 tx=0).stop 2024-03-24T00:43:02.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.565+0000 7f9199e27640 1 -- 172.21.15.67:0/2563349889 shutdown_connections 2024-03-24T00:43:02.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.565+0000 7f9199e27640 1 --2- 172.21.15.67:0/2563349889 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f919409a480 0x7f91940b2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:02.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.565+0000 7f9199e27640 1 --2- 172.21.15.67:0/2563349889 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91940d1550 0x7f919406cee0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:02.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.565+0000 7f9199e27640 1 -- 172.21.15.67:0/2563349889 >> 172.21.15.67:0/2563349889 conn(0x7f91940b36e0 msgr2=0x7f91940b3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:02.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.565+0000 7f9199e27640 1 -- 172.21.15.67:0/2563349889 shutdown_connections 2024-03-24T00:43:02.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.565+0000 7f9199e27640 1 -- 172.21.15.67:0/2563349889 wait complete. 2024-03-24T00:43:02.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.566+0000 7f9199e27640 1 Processor -- start 2024-03-24T00:43:02.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.566+0000 7f9199e27640 1 -- start start 2024-03-24T00:43:02.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.567+0000 7f9199e27640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f919409a480 0x7f91941f8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:02.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.567+0000 7f9199e27640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91940b2780 0x7f919415a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:02.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.567+0000 7f9199e27640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f91940d1550 0x7f91940ca730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:02.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.567+0000 7f9199e27640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f919406c400 con 0x7f919409a480 2024-03-24T00:43:02.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.567+0000 7f9199e27640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91940b2520 con 0x7f91940b2780 2024-03-24T00:43:02.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.567+0000 7f9199e27640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f919406c6e0 con 0x7f91940d1550 2024-03-24T00:43:02.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.567+0000 7f91937fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f919409a480 0x7f91941f8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:02.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.567+0000 7f91937fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f919409a480 0x7f91941f8260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43062/0 (socket says 172.21.15.67:43062) 2024-03-24T00:43:02.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.567+0000 7f91937fe640 1 -- 172.21.15.67:0/3097461334 learned_addr learned my addr 172.21.15.67:0/3097461334 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:02.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.567+0000 7f9193fff640 1 --2- 172.21.15.67:0/3097461334 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f91940d1550 0x7f91940ca730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:02.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.567+0000 7f9192ffd640 1 --2- 172.21.15.67:0/3097461334 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91940b2780 0x7f919415a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:02.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.567+0000 7f91937fe640 1 -- 172.21.15.67:0/3097461334 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f91940d1550 msgr2=0x7f91940ca730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:02.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.567+0000 7f91937fe640 1 --2- 172.21.15.67:0/3097461334 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f91940d1550 0x7f91940ca730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:02.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.567+0000 7f91937fe640 1 -- 172.21.15.67:0/3097461334 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91940b2780 msgr2=0x7f919415a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:02.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.567+0000 7f91937fe640 1 --2- 172.21.15.67:0/3097461334 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91940b2780 0x7f919415a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:02.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.567+0000 7f91937fe640 1 -- 172.21.15.67:0/3097461334 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91941f77c0 con 0x7f919409a480 2024-03-24T00:43:02.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.567+0000 7f9193fff640 1 --2- 172.21.15.67:0/3097461334 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f91940d1550 0x7f91940ca730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:43:02.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.568+0000 7f91937fe640 1 --2- 172.21.15.67:0/3097461334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f919409a480 0x7f91941f8260 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f9184034040 tx=0x7f91840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:02.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.568+0000 7f917bfff640 1 -- 172.21.15.67:0/3097461334 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9184045400 con 0x7f919409a480 2024-03-24T00:43:02.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.568+0000 7f917bfff640 1 -- 172.21.15.67:0/3097461334 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91840455a0 con 0x7f919409a480 2024-03-24T00:43:02.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.568+0000 7f9199e27640 1 -- 172.21.15.67:0/3097461334 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91940caf80 con 0x7f919409a480 2024-03-24T00:43:02.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.568+0000 7f917bfff640 1 -- 172.21.15.67:0/3097461334 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9184045950 con 0x7f919409a480 2024-03-24T00:43:02.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.568+0000 7f9199e27640 1 -- 172.21.15.67:0/3097461334 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f919415aba0 con 0x7f919409a480 2024-03-24T00:43:02.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.569+0000 7f9199e27640 1 -- 172.21.15.67:0/3097461334 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f919404f400 con 0x7f919409a480 2024-03-24T00:43:02.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.569+0000 7f917bfff640 1 -- 172.21.15.67:0/3097461334 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9184031070 con 0x7f919409a480 2024-03-24T00:43:02.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.570+0000 7f917bfff640 1 --2- 172.21.15.67:0/3097461334 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f918006bbd0 0x7f918006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:02.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.570+0000 7f917bfff640 1 -- 172.21.15.67:0/3097461334 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f91840ac150 con 0x7f919409a480 2024-03-24T00:43:02.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.570+0000 7f9192ffd640 1 --2- 172.21.15.67:0/3097461334 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f918006bbd0 0x7f918006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:02.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.570+0000 7f9192ffd640 1 --2- 172.21.15.67:0/3097461334 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f918006bbd0 0x7f918006e090 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f91940cbd50 tx=0x7f9188009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:02.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.574+0000 7f917bfff640 1 -- 172.21.15.67:0/3097461334 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9184040070 con 0x7f919409a480 2024-03-24T00:43:02.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.755+0000 7f9199e27640 1 -- 172.21.15.67:0/3097461334 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f91940b1310 con 0x7f919409a480 2024-03-24T00:43:02.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.756+0000 7f917bfff640 1 -- 172.21.15.67:0/3097461334 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) v1 ==== 74+0+13232 (secure 0 0 0) 0x7f918407adc0 con 0x7f919409a480 2024-03-24T00:43:02.757 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:43:02.757 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":67,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:43:01.444100+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6835/451962260":"2024-03-25T00:42:58.438761+0000","172.21.15.67:6834/451962260":"2024-03-25T00:42:58.438761+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:43:02.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.758+0000 7f9199e27640 1 -- 172.21.15.67:0/3097461334 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f918006bbd0 msgr2=0x7f918006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:02.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.758+0000 7f9199e27640 1 --2- 172.21.15.67:0/3097461334 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f918006bbd0 0x7f918006e090 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f91940cbd50 tx=0x7f9188009040 comp rx=0 tx=0).stop 2024-03-24T00:43:02.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.758+0000 7f9199e27640 1 -- 172.21.15.67:0/3097461334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f919409a480 msgr2=0x7f91941f8260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:02.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.758+0000 7f9199e27640 1 --2- 172.21.15.67:0/3097461334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f919409a480 0x7f91941f8260 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f9184034040 tx=0x7f91840046c0 comp rx=0 tx=0).stop 2024-03-24T00:43:02.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.758+0000 7f9199e27640 1 -- 172.21.15.67:0/3097461334 shutdown_connections 2024-03-24T00:43:02.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.758+0000 7f9199e27640 1 --2- 172.21.15.67:0/3097461334 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f918006bbd0 0x7f918006e090 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:02.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.758+0000 7f9199e27640 1 --2- 172.21.15.67:0/3097461334 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f91940d1550 0x7f91940ca730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:02.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.758+0000 7f9199e27640 1 --2- 172.21.15.67:0/3097461334 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91940b2780 0x7f919415a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:02.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.758+0000 7f9199e27640 1 --2- 172.21.15.67:0/3097461334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f919409a480 0x7f91941f8260 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:02.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.758+0000 7f9199e27640 1 -- 172.21.15.67:0/3097461334 >> 172.21.15.67:0/3097461334 conn(0x7f91940b36e0 msgr2=0x7f91940b1530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:02.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.758+0000 7f9199e27640 1 -- 172.21.15.67:0/3097461334 shutdown_connections 2024-03-24T00:43:02.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.758+0000 7f9199e27640 1 -- 172.21.15.67:0/3097461334 wait complete. 2024-03-24T00:43:02.770 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:38:39.364950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:43:02.770 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:43:02.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.874+0000 7f0eb7b10640 1 Processor -- start 2024-03-24T00:43:02.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.874+0000 7f0eb7b10640 1 -- start start 2024-03-24T00:43:02.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.875+0000 7f0eb7b10640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f0eb015de90 0x7f0eb015e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:02.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.875+0000 7f0eb7b10640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eb00d1630 con 0x7f0eb00af4e0 2024-03-24T00:43:02.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.875+0000 7f0eb7b10640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eb00ad9f0 con 0x7f0eb00cd2e0 2024-03-24T00:43:02.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.875+0000 7f0eb7b10640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0eb00d1910 con 0x7f0eb015de90 2024-03-24T00:43:02.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.875+0000 7f0eb60ad640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0eb00af4e0 0x7f0eb00cfd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:32782/0 (socket says 172.21.15.67:32782) 2024-03-24T00:43:02.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.875+0000 7f0eb60ad640 1 -- 172.21.15.67:0/279443815 learned_addr learned my addr 172.21.15.67:0/279443815 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:43:02.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.875+0000 7f0eb58ac640 1 --2- 172.21.15.67:0/279443815 >> v2:172.21.15.72:3300/0 conn(0x7f0eb015de90 0x7f0eb015e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:02.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.875+0000 7f0eb48aa640 1 -- 172.21.15.67:0/279443815 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2526144938 0 0) 0x7f0eb00d1630 con 0x7f0eb00af4e0 2024-03-24T00:43:02.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.875+0000 7f0eb48aa640 1 -- 172.21.15.67:0/279443815 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e9c003520 con 0x7f0eb00af4e0 2024-03-24T00:43:02.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.875+0000 7f0eb48aa640 1 -- 172.21.15.67:0/279443815 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3538817819 0 0) 0x7f0e9c003520 con 0x7f0eb00af4e0 2024-03-24T00:43:02.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.875+0000 7f0eb48aa640 1 -- 172.21.15.67:0/279443815 >> v2:172.21.15.72:3300/0 conn(0x7f0eb015de90 msgr2=0x7f0eb015e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:02.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.875+0000 7f0eb48aa640 1 --2- 172.21.15.67:0/279443815 >> v2:172.21.15.72:3300/0 conn(0x7f0eb015de90 0x7f0eb015e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:02.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.875+0000 7f0eb48aa640 1 -- 172.21.15.67:0/279443815 >> v1:172.21.15.72:6789/0 conn(0x7f0eb00cd2e0 legacy=0x7f0eb006b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:02.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.876+0000 7f0eb48aa640 1 -- 172.21.15.67:0/279443815 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eb00adcd0 con 0x7f0eb00af4e0 2024-03-24T00:43:02.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.876+0000 7f0eb48aa640 1 -- 172.21.15.67:0/279443815 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f0ea80019f0 con 0x7f0eb00af4e0 2024-03-24T00:43:02.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.876+0000 7f0eb48aa640 1 -- 172.21.15.67:0/279443815 >> v1:172.21.15.67:6789/0 conn(0x7f0eb00af4e0 legacy=0x7f0eb00cfd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:02.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.876+0000 7f0eb48aa640 1 --2- 172.21.15.67:0/279443815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e9c0039b0 0x7f0e9c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:02.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.876+0000 7f0eb48aa640 1 --2- 172.21.15.67:0/279443815 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0e9c004800 0x7f0e9c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:02.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.876+0000 7f0eb48aa640 1 --2- 172.21.15.67:0/279443815 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0e9c0051c0 0x7f0e9c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:02.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.876+0000 7f0eb48aa640 1 -- 172.21.15.67:0/279443815 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb00d1910 con 0x7f0e9c0039b0 2024-03-24T00:43:02.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.876+0000 7f0eb48aa640 1 -- 172.21.15.67:0/279443815 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e9c00c160 con 0x7f0e9c004800 2024-03-24T00:43:02.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.876+0000 7f0eb48aa640 1 -- 172.21.15.67:0/279443815 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e9c00c360 con 0x7f0e9c0051c0 2024-03-24T00:43:02.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.876+0000 7f0eb58ac640 1 --2- 172.21.15.67:0/279443815 >> v2:172.21.15.72:3300/0 conn(0x7f0eb015de90 0x7f0eb015e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:43:02.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.876+0000 7f0eb50ab640 1 --2- 172.21.15.67:0/279443815 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0e9c004800 0x7f0e9c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:02.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.876+0000 7f0eb58ac640 1 --2- 172.21.15.67:0/279443815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e9c0039b0 0x7f0e9c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:02.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.876+0000 7f0eb60ad640 1 --2- 172.21.15.67:0/279443815 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0e9c0051c0 0x7f0e9c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:02.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.876+0000 7f0eb58ac640 1 -- 172.21.15.67:0/279443815 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0e9c0051c0 msgr2=0x7f0e9c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:02.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.876+0000 7f0eb58ac640 1 --2- 172.21.15.67:0/279443815 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0e9c0051c0 0x7f0e9c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:02.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.876+0000 7f0eb58ac640 1 -- 172.21.15.67:0/279443815 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0e9c004800 msgr2=0x7f0e9c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:02.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.876+0000 7f0eb58ac640 1 --2- 172.21.15.67:0/279443815 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0e9c004800 0x7f0e9c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:02.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.876+0000 7f0eb58ac640 1 -- 172.21.15.67:0/279443815 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e9c003520 con 0x7f0e9c0039b0 2024-03-24T00:43:02.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.876+0000 7f0eb60ad640 1 -- 172.21.15.67:0/279443815 reap_dead start 2024-03-24T00:43:02.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.877+0000 7f0eb58ac640 1 --2- 172.21.15.67:0/279443815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e9c0039b0 0x7f0e9c003dd0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f0eac009c70 tx=0x7f0e9c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16b5ce4c991f208f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:02.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.877+0000 7f0eb48aa640 1 -- 172.21.15.67:0/279443815 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eac009840 con 0x7f0e9c0039b0 2024-03-24T00:43:02.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.877+0000 7f0eb48aa640 1 -- 172.21.15.67:0/279443815 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0eac004020 con 0x7f0e9c0039b0 2024-03-24T00:43:02.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.877+0000 7f0eb48aa640 1 -- 172.21.15.67:0/279443815 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eac004450 con 0x7f0e9c0039b0 2024-03-24T00:43:02.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.877+0000 7f0eb7b10640 1 -- 172.21.15.67:0/279443815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e9c0039b0 msgr2=0x7f0e9c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:02.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.877+0000 7f0eb7b10640 1 --2- 172.21.15.67:0/279443815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e9c0039b0 0x7f0e9c003dd0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f0eac009c70 tx=0x7f0e9c005830 comp rx=0 tx=0).stop 2024-03-24T00:43:02.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.878+0000 7f0eb7b10640 1 -- 172.21.15.67:0/279443815 shutdown_connections 2024-03-24T00:43:02.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.878+0000 7f0eb7b10640 1 --2- 172.21.15.67:0/279443815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e9c0039b0 0x7f0e9c003dd0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:02.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.878+0000 7f0eb7b10640 1 -- 172.21.15.67:0/279443815 >> 172.21.15.67:0/279443815 conn(0x7f0eb0099a30 msgr2=0x7f0eb0099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:02.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.878+0000 7f0eb7b10640 1 -- 172.21.15.67:0/279443815 shutdown_connections 2024-03-24T00:43:02.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.878+0000 7f0eb7b10640 1 -- 172.21.15.67:0/279443815 wait complete. 2024-03-24T00:43:02.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.879+0000 7f0eb7b10640 1 Processor -- start 2024-03-24T00:43:02.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.879+0000 7f0eb7b10640 1 -- start start 2024-03-24T00:43:02.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.879+0000 7f0eb7b10640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e9c0039b0 0x7f0eb00b0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:02.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.879+0000 7f0eb7b10640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0eb00b1780 0x7f0eb00b1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:02.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.879+0000 7f0eb7b10640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0eb00c5a10 0x7f0eb00c5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:02.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.879+0000 7f0eb7b10640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb015e3f0 con 0x7f0e9c0039b0 2024-03-24T00:43:02.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.879+0000 7f0eb7b10640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb015e270 con 0x7f0eb00b1780 2024-03-24T00:43:02.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.879+0000 7f0eb7b10640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb015e570 con 0x7f0eb00c5a10 2024-03-24T00:43:02.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.880+0000 7f0eb58ac640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e9c0039b0 0x7f0eb00b0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:02.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.880+0000 7f0eb58ac640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e9c0039b0 0x7f0eb00b0b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43074/0 (socket says 172.21.15.67:43074) 2024-03-24T00:43:02.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.880+0000 7f0eb58ac640 1 -- 172.21.15.67:0/2914722904 learned_addr learned my addr 172.21.15.67:0/2914722904 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:02.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.880+0000 7f0eb60ad640 1 --2- 172.21.15.67:0/2914722904 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0eb00c5a10 0x7f0eb00c5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:02.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.880+0000 7f0eb50ab640 1 --2- 172.21.15.67:0/2914722904 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0eb00b1780 0x7f0eb00b1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:02.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.880+0000 7f0eb58ac640 1 -- 172.21.15.67:0/2914722904 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0eb00c5a10 msgr2=0x7f0eb00c5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:02.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.880+0000 7f0eb58ac640 1 --2- 172.21.15.67:0/2914722904 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0eb00c5a10 0x7f0eb00c5ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:02.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.880+0000 7f0eb58ac640 1 -- 172.21.15.67:0/2914722904 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0eb00b1780 msgr2=0x7f0eb00b1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:02.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.880+0000 7f0eb58ac640 1 --2- 172.21.15.67:0/2914722904 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0eb00b1780 0x7f0eb00b1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:02.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.880+0000 7f0eb58ac640 1 -- 172.21.15.67:0/2914722904 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eb00b1c60 con 0x7f0e9c0039b0 2024-03-24T00:43:02.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.880+0000 7f0eb58ac640 1 --2- 172.21.15.67:0/2914722904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e9c0039b0 0x7f0eb00b0b00 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f0eac036040 tx=0x7f0eac004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:02.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.880+0000 7f0ea67fc640 1 -- 172.21.15.67:0/2914722904 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eac043020 con 0x7f0e9c0039b0 2024-03-24T00:43:02.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.880+0000 7f0ea67fc640 1 -- 172.21.15.67:0/2914722904 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0eac02c910 con 0x7f0e9c0039b0 2024-03-24T00:43:02.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.880+0000 7f0eb7b10640 1 -- 172.21.15.67:0/2914722904 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0eb00c86f0 con 0x7f0e9c0039b0 2024-03-24T00:43:02.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.880+0000 7f0ea67fc640 1 -- 172.21.15.67:0/2914722904 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eac02ccc0 con 0x7f0e9c0039b0 2024-03-24T00:43:02.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.880+0000 7f0eb7b10640 1 -- 172.21.15.67:0/2914722904 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0eb00c8c70 con 0x7f0e9c0039b0 2024-03-24T00:43:02.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.881+0000 7f0eb7b10640 1 -- 172.21.15.67:0/2914722904 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0eb004f400 con 0x7f0e9c0039b0 2024-03-24T00:43:02.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.882+0000 7f0ea67fc640 1 -- 172.21.15.67:0/2914722904 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0eac046020 con 0x7f0e9c0039b0 2024-03-24T00:43:02.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.882+0000 7f0ea67fc640 1 --2- 172.21.15.67:0/2914722904 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0e9406bbd0 0x7f0e9406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:02.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.882+0000 7f0ea67fc640 1 -- 172.21.15.67:0/2914722904 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f0eac0ac8c0 con 0x7f0e9c0039b0 2024-03-24T00:43:02.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.882+0000 7f0eb50ab640 1 --2- 172.21.15.67:0/2914722904 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0e9406bbd0 0x7f0e9406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:02.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.883+0000 7f0eb50ab640 1 --2- 172.21.15.67:0/2914722904 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0e9406bbd0 0x7f0e9406e090 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f0eb00c5440 tx=0x7f0ea0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:02.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:02.887+0000 7f0ea67fc640 1 -- 172.21.15.67:0/2914722904 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0eac04d030 con 0x7f0e9c0039b0 2024-03-24T00:43:03.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.069+0000 7f0eb7b10640 1 -- 172.21.15.67:0/2914722904 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0eb00cead0 con 0x7f0e9c0039b0 2024-03-24T00:43:03.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.069+0000 7f0ea67fc640 1 -- 172.21.15.67:0/2914722904 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v67) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0eac07b6d0 con 0x7f0e9c0039b0 2024-03-24T00:43:03.070 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-24T00:43:03.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.071+0000 7f0eb7b10640 1 -- 172.21.15.67:0/2914722904 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0e9406bbd0 msgr2=0x7f0e9406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:03.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.071+0000 7f0eb7b10640 1 --2- 172.21.15.67:0/2914722904 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0e9406bbd0 0x7f0e9406e090 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f0eb00c5440 tx=0x7f0ea0009040 comp rx=0 tx=0).stop 2024-03-24T00:43:03.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.071+0000 7f0eb7b10640 1 -- 172.21.15.67:0/2914722904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e9c0039b0 msgr2=0x7f0eb00b0b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:03.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.071+0000 7f0eb7b10640 1 --2- 172.21.15.67:0/2914722904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e9c0039b0 0x7f0eb00b0b00 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f0eac036040 tx=0x7f0eac004730 comp rx=0 tx=0).stop 2024-03-24T00:43:03.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.071+0000 7f0eb7b10640 1 -- 172.21.15.67:0/2914722904 shutdown_connections 2024-03-24T00:43:03.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.071+0000 7f0eb7b10640 1 --2- 172.21.15.67:0/2914722904 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0e9406bbd0 0x7f0e9406e090 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:03.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.071+0000 7f0eb7b10640 1 --2- 172.21.15.67:0/2914722904 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0eb00c5a10 0x7f0eb00c5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:03.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.071+0000 7f0eb7b10640 1 --2- 172.21.15.67:0/2914722904 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0eb00b1780 0x7f0eb00b1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:03.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.071+0000 7f0eb7b10640 1 --2- 172.21.15.67:0/2914722904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e9c0039b0 0x7f0eb00b0b00 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:03.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.071+0000 7f0eb7b10640 1 -- 172.21.15.67:0/2914722904 >> 172.21.15.67:0/2914722904 conn(0x7f0eb0099a30 msgr2=0x7f0eb006fc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:03.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.071+0000 7f0eb7b10640 1 -- 172.21.15.67:0/2914722904 shutdown_connections 2024-03-24T00:43:03.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.071+0000 7f0eb7b10640 1 -- 172.21.15.67:0/2914722904 wait complete. 2024-03-24T00:43:03.082 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dentry_repair' 2024-03-24T00:43:03.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.184+0000 7f0a3c2dc640 1 Processor -- start 2024-03-24T00:43:03.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.184+0000 7f0a3c2dc640 1 -- start start 2024-03-24T00:43:03.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.185+0000 7f0a3c2dc640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0a3415bf70 0x7f0a340cc690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:03.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.185+0000 7f0a3c2dc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f0a340ce930 0x7f0a340b1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:03.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.185+0000 7f0a3c2dc640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a340ccfe0 con 0x7f0a340cfc70 2024-03-24T00:43:03.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.185+0000 7f0a3c2dc640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a340cd2c0 con 0x7f0a340ce930 2024-03-24T00:43:03.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.185+0000 7f0a3c2dc640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a340cd5a0 con 0x7f0a3415bf70 2024-03-24T00:43:03.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.185+0000 7f0a39877640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0a340cfc70 0x7f0a340ce220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:32786/0 (socket says 172.21.15.67:32786) 2024-03-24T00:43:03.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.185+0000 7f0a39877640 1 -- 172.21.15.67:0/155944307 learned_addr learned my addr 172.21.15.67:0/155944307 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:43:03.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.185+0000 7f0a3a879640 1 --2- 172.21.15.67:0/155944307 >> v2:172.21.15.72:3300/0 conn(0x7f0a340ce930 0x7f0a340b1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:03.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.185+0000 7f0a3a078640 1 --2- 172.21.15.67:0/155944307 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0a3415bf70 0x7f0a340cc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:03.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.185+0000 7f0a39076640 1 -- 172.21.15.67:0/155944307 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2154598811 0 0) 0x7f0a340ccfe0 con 0x7f0a340cfc70 2024-03-24T00:43:03.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.185+0000 7f0a39076640 1 -- 172.21.15.67:0/155944307 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a20003520 con 0x7f0a340cfc70 2024-03-24T00:43:03.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.185+0000 7f0a39076640 1 -- 172.21.15.67:0/155944307 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2281340390 0 0) 0x7f0a20003520 con 0x7f0a340cfc70 2024-03-24T00:43:03.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.185+0000 7f0a39076640 1 -- 172.21.15.67:0/155944307 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0a3415bf70 msgr2=0x7f0a340cc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:03.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.185+0000 7f0a39076640 1 --2- 172.21.15.67:0/155944307 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0a3415bf70 0x7f0a340cc690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:03.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.185+0000 7f0a39076640 1 -- 172.21.15.67:0/155944307 >> v2:172.21.15.72:3300/0 conn(0x7f0a340ce930 msgr2=0x7f0a340b1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:03.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.185+0000 7f0a39076640 1 --2- 172.21.15.67:0/155944307 >> v2:172.21.15.72:3300/0 conn(0x7f0a340ce930 0x7f0a340b1cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:03.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.185+0000 7f0a39076640 1 -- 172.21.15.67:0/155944307 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a340cd880 con 0x7f0a340cfc70 2024-03-24T00:43:03.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.185+0000 7f0a39076640 1 -- 172.21.15.67:0/155944307 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f0a24001670 con 0x7f0a340cfc70 2024-03-24T00:43:03.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.185+0000 7f0a3a879640 1 --2- 172.21.15.67:0/155944307 >> v2:172.21.15.72:3300/0 conn(0x7f0a340ce930 0x7f0a340b1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:43:03.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.185+0000 7f0a39076640 1 -- 172.21.15.67:0/155944307 >> v1:172.21.15.67:6789/0 conn(0x7f0a340cfc70 legacy=0x7f0a340ce220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:03.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.186+0000 7f0a39076640 1 --2- 172.21.15.67:0/155944307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a200039b0 0x7f0a20003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:03.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.186+0000 7f0a39076640 1 --2- 172.21.15.67:0/155944307 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0a3415bf70 0x7f0a20004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:03.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.186+0000 7f0a39076640 1 --2- 172.21.15.67:0/155944307 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0a20004d40 0x7f0a200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:03.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.186+0000 7f0a39076640 1 -- 172.21.15.67:0/155944307 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a340cd2c0 con 0x7f0a200039b0 2024-03-24T00:43:03.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.186+0000 7f0a39076640 1 -- 172.21.15.67:0/155944307 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a340cd5a0 con 0x7f0a20004d40 2024-03-24T00:43:03.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.186+0000 7f0a39076640 1 -- 172.21.15.67:0/155944307 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a2000b9c0 con 0x7f0a3415bf70 2024-03-24T00:43:03.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.186+0000 7f0a3a078640 1 --2- 172.21.15.67:0/155944307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a200039b0 0x7f0a20003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:03.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.186+0000 7f0a39877640 1 --2- 172.21.15.67:0/155944307 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0a3415bf70 0x7f0a20004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:03.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.186+0000 7f0a3a879640 1 --2- 172.21.15.67:0/155944307 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0a20004d40 0x7f0a200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:03.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.186+0000 7f0a3a078640 1 -- 172.21.15.67:0/155944307 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0a3415bf70 msgr2=0x7f0a20004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:03.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.186+0000 7f0a3a078640 1 --2- 172.21.15.67:0/155944307 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0a3415bf70 0x7f0a20004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:03.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.186+0000 7f0a3a078640 1 -- 172.21.15.67:0/155944307 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0a20004d40 msgr2=0x7f0a200051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:03.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.186+0000 7f0a3a078640 1 --2- 172.21.15.67:0/155944307 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0a20004d40 0x7f0a200051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:03.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.186+0000 7f0a3a078640 1 -- 172.21.15.67:0/155944307 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a20003520 con 0x7f0a200039b0 2024-03-24T00:43:03.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.186+0000 7f0a39877640 1 --2- 172.21.15.67:0/155944307 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0a3415bf70 0x7f0a20004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:43:03.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.187+0000 7f0a3a078640 1 --2- 172.21.15.67:0/155944307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a200039b0 0x7f0a20003dd0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f0a30009ac0 tx=0x7f0a20005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=590fe906e2d952fa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:03.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.187+0000 7f0a39076640 1 -- 172.21.15.67:0/155944307 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a30039070 con 0x7f0a200039b0 2024-03-24T00:43:03.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.187+0000 7f0a39076640 1 -- 172.21.15.67:0/155944307 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a3002c6b0 con 0x7f0a200039b0 2024-03-24T00:43:03.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.187+0000 7f0a39076640 1 -- 172.21.15.67:0/155944307 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a30009690 con 0x7f0a200039b0 2024-03-24T00:43:03.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.187+0000 7f0a3c2dc640 1 -- 172.21.15.67:0/155944307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a200039b0 msgr2=0x7f0a20003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:03.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.187+0000 7f0a3c2dc640 1 --2- 172.21.15.67:0/155944307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a200039b0 0x7f0a20003dd0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f0a30009ac0 tx=0x7f0a20005830 comp rx=0 tx=0).stop 2024-03-24T00:43:03.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.187+0000 7f0a3a879640 1 -- 172.21.15.67:0/155944307 reap_dead start 2024-03-24T00:43:03.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.187+0000 7f0a3c2dc640 1 -- 172.21.15.67:0/155944307 shutdown_connections 2024-03-24T00:43:03.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.187+0000 7f0a3c2dc640 1 -- 172.21.15.67:0/155944307 >> 172.21.15.67:0/155944307 conn(0x7f0a340d1bf0 msgr2=0x7f0a340d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:03.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.187+0000 7f0a3c2dc640 1 -- 172.21.15.67:0/155944307 shutdown_connections 2024-03-24T00:43:03.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.187+0000 7f0a3c2dc640 1 -- 172.21.15.67:0/155944307 wait complete. 2024-03-24T00:43:03.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.189+0000 7f0a3c2dc640 1 Processor -- start 2024-03-24T00:43:03.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.189+0000 7f0a3c2dc640 1 -- start start 2024-03-24T00:43:03.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.189+0000 7f0a3c2dc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0a200039b0 0x7f0a3415e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:03.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.189+0000 7f0a3c2dc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a3414fe30 0x7f0a341f0f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:03.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.189+0000 7f0a3c2dc640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0a3414d860 0x7f0a340b69b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:03.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.189+0000 7f0a3c2dc640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a340cc810 con 0x7f0a3414fe30 2024-03-24T00:43:03.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.189+0000 7f0a3c2dc640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a340cc690 con 0x7f0a200039b0 2024-03-24T00:43:03.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.189+0000 7f0a3c2dc640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a340cc990 con 0x7f0a3414d860 2024-03-24T00:43:03.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.189+0000 7f0a39877640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a3414fe30 0x7f0a341f0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:03.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.189+0000 7f0a39877640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a3414fe30 0x7f0a341f0f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43102/0 (socket says 172.21.15.67:43102) 2024-03-24T00:43:03.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.189+0000 7f0a39877640 1 -- 172.21.15.67:0/1721389058 learned_addr learned my addr 172.21.15.67:0/1721389058 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:03.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.189+0000 7f0a3a879640 1 --2- 172.21.15.67:0/1721389058 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0a3414d860 0x7f0a340b69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:03.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.189+0000 7f0a3a078640 1 --2- 172.21.15.67:0/1721389058 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0a200039b0 0x7f0a3415e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:03.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.190+0000 7f0a39877640 1 -- 172.21.15.67:0/1721389058 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0a3414d860 msgr2=0x7f0a340b69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:03.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.190+0000 7f0a39877640 1 --2- 172.21.15.67:0/1721389058 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0a3414d860 0x7f0a340b69b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:03.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.190+0000 7f0a39877640 1 -- 172.21.15.67:0/1721389058 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0a200039b0 msgr2=0x7f0a3415e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:03.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.190+0000 7f0a39877640 1 --2- 172.21.15.67:0/1721389058 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0a200039b0 0x7f0a3415e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:03.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.190+0000 7f0a39877640 1 -- 172.21.15.67:0/1721389058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a3415e8e0 con 0x7f0a3414fe30 2024-03-24T00:43:03.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.190+0000 7f0a39877640 1 --2- 172.21.15.67:0/1721389058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a3414fe30 0x7f0a341f0f50 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f0a2400e8a0 tx=0x7f0a2400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:03.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.190+0000 7f0a2affd640 1 -- 172.21.15.67:0/1721389058 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a2400c850 con 0x7f0a3414fe30 2024-03-24T00:43:03.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.190+0000 7f0a2affd640 1 -- 172.21.15.67:0/1721389058 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a24018070 con 0x7f0a3414fe30 2024-03-24T00:43:03.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.190+0000 7f0a2affd640 1 -- 172.21.15.67:0/1721389058 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a2400cc10 con 0x7f0a3414fe30 2024-03-24T00:43:03.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.190+0000 7f0a3c2dc640 1 -- 172.21.15.67:0/1721389058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a3415ebc0 con 0x7f0a3414fe30 2024-03-24T00:43:03.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.190+0000 7f0a3c2dc640 1 -- 172.21.15.67:0/1721389058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a341f1490 con 0x7f0a3414fe30 2024-03-24T00:43:03.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.191+0000 7f0a3c2dc640 1 -- 172.21.15.67:0/1721389058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a3404f400 con 0x7f0a3414fe30 2024-03-24T00:43:03.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.192+0000 7f0a2affd640 1 -- 172.21.15.67:0/1721389058 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0a2401d460 con 0x7f0a3414fe30 2024-03-24T00:43:03.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.192+0000 7f0a2affd640 1 --2- 172.21.15.67:0/1721389058 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0a1806bbd0 0x7f0a1806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:03.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.192+0000 7f0a2affd640 1 -- 172.21.15.67:0/1721389058 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f0a2408a7b0 con 0x7f0a3414fe30 2024-03-24T00:43:03.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.192+0000 7f0a3a078640 1 --2- 172.21.15.67:0/1721389058 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0a1806bbd0 0x7f0a1806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:03.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.193+0000 7f0a3a078640 1 --2- 172.21.15.67:0/1721389058 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0a1806bbd0 0x7f0a1806e090 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f0a30036000 tx=0x7f0a30030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:03.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.196+0000 7f0a2affd640 1 -- 172.21.15.67:0/1721389058 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0a24058dc0 con 0x7f0a3414fe30 2024-03-24T00:43:03.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:03.450+0000 7f0a3c2dc640 1 -- 172.21.15.67:0/1721389058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dentry_repair"]} v 0) v1 -- 0x7f0a340d02f0 con 0x7f0a3414fe30 2024-03-24T00:43:04.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.451+0000 7f0a2affd640 1 -- 172.21.15.67:0/1721389058 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dentry_repair"]}]=0 v223) v1 ==== 169+0+0 (secure 0 0 0) 0x7f0a2405ce60 con 0x7f0a3414fe30 2024-03-24T00:43:04.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.454+0000 7f0a3c2dc640 1 -- 172.21.15.67:0/1721389058 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0a1806bbd0 msgr2=0x7f0a1806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:04.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.454+0000 7f0a3c2dc640 1 --2- 172.21.15.67:0/1721389058 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0a1806bbd0 0x7f0a1806e090 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f0a30036000 tx=0x7f0a30030000 comp rx=0 tx=0).stop 2024-03-24T00:43:04.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.454+0000 7f0a3c2dc640 1 -- 172.21.15.67:0/1721389058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a3414fe30 msgr2=0x7f0a341f0f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:04.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.454+0000 7f0a3c2dc640 1 --2- 172.21.15.67:0/1721389058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a3414fe30 0x7f0a341f0f50 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f0a2400e8a0 tx=0x7f0a2400ed60 comp rx=0 tx=0).stop 2024-03-24T00:43:04.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.454+0000 7f0a3c2dc640 1 -- 172.21.15.67:0/1721389058 shutdown_connections 2024-03-24T00:43:04.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.454+0000 7f0a3c2dc640 1 --2- 172.21.15.67:0/1721389058 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0a1806bbd0 0x7f0a1806e090 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:04.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.454+0000 7f0a3c2dc640 1 --2- 172.21.15.67:0/1721389058 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0a3414d860 0x7f0a340b69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:04.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.454+0000 7f0a3c2dc640 1 --2- 172.21.15.67:0/1721389058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a3414fe30 0x7f0a341f0f50 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:04.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.454+0000 7f0a3c2dc640 1 --2- 172.21.15.67:0/1721389058 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0a200039b0 0x7f0a3415e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:04.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.454+0000 7f0a3c2dc640 1 -- 172.21.15.67:0/1721389058 >> 172.21.15.67:0/1721389058 conn(0x7f0a340d1bf0 msgr2=0x7f0a340ce190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:04.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.454+0000 7f0a3c2dc640 1 -- 172.21.15.67:0/1721389058 shutdown_connections 2024-03-24T00:43:04.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.454+0000 7f0a3c2dc640 1 -- 172.21.15.67:0/1721389058 wait complete. 2024-03-24T00:43:04.468 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-24T00:43:04.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.577+0000 7f35c2e4b640 1 Processor -- start 2024-03-24T00:43:04.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.578+0000 7f35c2e4b640 1 -- start start 2024-03-24T00:43:04.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.578+0000 7f35c2e4b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f35bc15dea0 0x7f35bc0d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:04.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.578+0000 7f35c2e4b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f35bc0b6170 0x7f35bc0d3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:04.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.578+0000 7f35c2e4b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35bc0d2250 con 0x7f35bc15e280 2024-03-24T00:43:04.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.578+0000 7f35c2e4b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35bc0d2530 con 0x7f35bc0b6170 2024-03-24T00:43:04.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.578+0000 7f35c2e4b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35bc0d2810 con 0x7f35bc15dea0 2024-03-24T00:43:04.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.579+0000 7f35c0be7640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f35bc15dea0 0x7f35bc0d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:04.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.579+0000 7f35c13e8640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f35bc0b6170 0x7f35bc0d3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:04.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.579+0000 7f35c0be7640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f35bc15dea0 0x7f35bc0d33d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.67:53714/0 (socket says 172.21.15.67:53714) 2024-03-24T00:43:04.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.579+0000 7f35c0be7640 1 -- 172.21.15.67:0/192245242 learned_addr learned my addr 172.21.15.67:0/192245242 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:04.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.579+0000 7f35bb7fe640 1 -- 172.21.15.67:0/192245242 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 446596423 0 0) 0x7f35bc0d2250 con 0x7f35bc15e280 2024-03-24T00:43:04.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.579+0000 7f35bb7fe640 1 -- 172.21.15.67:0/192245242 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35ac003520 con 0x7f35bc15e280 2024-03-24T00:43:04.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.579+0000 7f35c0be7640 1 -- 172.21.15.67:0/192245242 >> v2:172.21.15.72:3300/0 conn(0x7f35bc0b6170 msgr2=0x7f35bc0d3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:04.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.579+0000 7f35c0be7640 1 --2- 172.21.15.67:0/192245242 >> v2:172.21.15.72:3300/0 conn(0x7f35bc0b6170 0x7f35bc0d3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:04.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.579+0000 7f35c0be7640 1 -- 172.21.15.67:0/192245242 >> v1:172.21.15.72:6789/0 conn(0x7f35bc15e280 legacy=0x7f35bc0b5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:04.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.579+0000 7f35c0be7640 1 -- 172.21.15.67:0/192245242 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35bc0d2af0 con 0x7f35bc15dea0 2024-03-24T00:43:04.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.579+0000 7f35c13e8640 1 --2- 172.21.15.67:0/192245242 >> v2:172.21.15.72:3300/0 conn(0x7f35bc0b6170 0x7f35bc0d3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:43:04.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.580+0000 7f35c0be7640 1 --2- 172.21.15.67:0/192245242 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f35bc15dea0 0x7f35bc0d33d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f35b0009870 tx=0x7f35b002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5ee47592566863e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:04.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.580+0000 7f35bb7fe640 1 -- 172.21.15.67:0/192245242 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b0035020 con 0x7f35bc15dea0 2024-03-24T00:43:04.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.580+0000 7f35bb7fe640 1 -- 172.21.15.67:0/192245242 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35b0031640 con 0x7f35bc15dea0 2024-03-24T00:43:04.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.580+0000 7f35bb7fe640 1 -- 172.21.15.67:0/192245242 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b002d040 con 0x7f35bc15dea0 2024-03-24T00:43:04.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.581+0000 7f35c2e4b640 1 -- 172.21.15.67:0/192245242 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f35bc15dea0 msgr2=0x7f35bc0d33d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:04.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.581+0000 7f35c2e4b640 1 --2- 172.21.15.67:0/192245242 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f35bc15dea0 0x7f35bc0d33d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f35b0009870 tx=0x7f35b002b8f0 comp rx=0 tx=0).stop 2024-03-24T00:43:04.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.581+0000 7f35c2e4b640 1 -- 172.21.15.67:0/192245242 shutdown_connections 2024-03-24T00:43:04.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.581+0000 7f35c2e4b640 1 --2- 172.21.15.67:0/192245242 >> v2:172.21.15.72:3300/0 conn(0x7f35bc0b6170 0x7f35bc0d3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:04.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.581+0000 7f35c2e4b640 1 --2- 172.21.15.67:0/192245242 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f35bc15dea0 0x7f35bc0d33d0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:04.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.581+0000 7f35c2e4b640 1 -- 172.21.15.67:0/192245242 >> 172.21.15.67:0/192245242 conn(0x7f35bc072da0 msgr2=0x7f35bc0d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:04.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.581+0000 7f35c2e4b640 1 -- 172.21.15.67:0/192245242 shutdown_connections 2024-03-24T00:43:04.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.581+0000 7f35c2e4b640 1 -- 172.21.15.67:0/192245242 wait complete. 2024-03-24T00:43:04.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.582+0000 7f35c2e4b640 1 Processor -- start 2024-03-24T00:43:04.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.582+0000 7f35c2e4b640 1 -- start start 2024-03-24T00:43:04.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.583+0000 7f35c2e4b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f35bc0b6170 0x7f35bc0cb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:04.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.583+0000 7f35c2e4b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f35bc15dea0 0x7f35bc153470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:04.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.583+0000 7f35c2e4b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35bc15e280 0x7f35bc15f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:04.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.583+0000 7f35c2e4b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35bc0b7480 con 0x7f35bc15e280 2024-03-24T00:43:04.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.583+0000 7f35c2e4b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35bc0b7300 con 0x7f35bc0b6170 2024-03-24T00:43:04.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.583+0000 7f35c2e4b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35bc0b7600 con 0x7f35bc15dea0 2024-03-24T00:43:04.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.583+0000 7f35c13e8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35bc15e280 0x7f35bc15f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:04.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.583+0000 7f35c13e8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35bc15e280 0x7f35bc15f330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47110/0 (socket says 172.21.15.67:47110) 2024-03-24T00:43:04.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.583+0000 7f35c13e8640 1 -- 172.21.15.67:0/2719287278 learned_addr learned my addr 172.21.15.67:0/2719287278 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:04.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.583+0000 7f35c0be7640 1 --2- 172.21.15.67:0/2719287278 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f35bc0b6170 0x7f35bc0cb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:04.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.583+0000 7f35bbfff640 1 --2- 172.21.15.67:0/2719287278 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f35bc15dea0 0x7f35bc153470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:04.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.583+0000 7f35c13e8640 1 -- 172.21.15.67:0/2719287278 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f35bc15dea0 msgr2=0x7f35bc153470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:04.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.583+0000 7f35c13e8640 1 --2- 172.21.15.67:0/2719287278 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f35bc15dea0 0x7f35bc153470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:04.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.583+0000 7f35c13e8640 1 -- 172.21.15.67:0/2719287278 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f35bc0b6170 msgr2=0x7f35bc0cb0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:04.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.583+0000 7f35c13e8640 1 --2- 172.21.15.67:0/2719287278 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f35bc0b6170 0x7f35bc0cb0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:04.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.583+0000 7f35c13e8640 1 -- 172.21.15.67:0/2719287278 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35bc155370 con 0x7f35bc15e280 2024-03-24T00:43:04.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.583+0000 7f35bbfff640 1 --2- 172.21.15.67:0/2719287278 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f35bc15dea0 0x7f35bc153470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:43:04.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.584+0000 7f35c13e8640 1 --2- 172.21.15.67:0/2719287278 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35bc15e280 0x7f35bc15f330 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f35b400efb0 tx=0x7f35b400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:04.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.584+0000 7f35b97fa640 1 -- 172.21.15.67:0/2719287278 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b4018020 con 0x7f35bc15e280 2024-03-24T00:43:04.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.584+0000 7f35b97fa640 1 -- 172.21.15.67:0/2719287278 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35b4014480 con 0x7f35bc15e280 2024-03-24T00:43:04.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.584+0000 7f35c2e4b640 1 -- 172.21.15.67:0/2719287278 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35bc15fd80 con 0x7f35bc15e280 2024-03-24T00:43:04.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.584+0000 7f35b97fa640 1 -- 172.21.15.67:0/2719287278 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b4014830 con 0x7f35bc15e280 2024-03-24T00:43:04.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.584+0000 7f35c2e4b640 1 -- 172.21.15.67:0/2719287278 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35bc160060 con 0x7f35bc15e280 2024-03-24T00:43:04.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.585+0000 7f35c2e4b640 1 -- 172.21.15.67:0/2719287278 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35bc04f400 con 0x7f35bc15e280 2024-03-24T00:43:04.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.585+0000 7f35b97fa640 1 -- 172.21.15.67:0/2719287278 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f35b4005d10 con 0x7f35bc15e280 2024-03-24T00:43:04.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.586+0000 7f35b97fa640 1 --2- 172.21.15.67:0/2719287278 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f35a006bc00 0x7f35a006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:04.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.586+0000 7f35b97fa640 1 -- 172.21.15.67:0/2719287278 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f35b408b530 con 0x7f35bc15e280 2024-03-24T00:43:04.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.586+0000 7f35c0be7640 1 --2- 172.21.15.67:0/2719287278 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f35a006bc00 0x7f35a006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:04.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.587+0000 7f35c0be7640 1 --2- 172.21.15.67:0/2719287278 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f35a006bc00 0x7f35a006e0c0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f35b0036000 tx=0x7f35b003c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:04.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.590+0000 7f35b97fa640 1 -- 172.21.15.67:0/2719287278 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f35b4022030 con 0x7f35bc15e280 2024-03-24T00:43:04.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.772+0000 7f35c2e4b640 1 -- 172.21.15.67:0/2719287278 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f35bc0b8750 con 0x7f35bc15e280 2024-03-24T00:43:04.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.777+0000 7f35b97fa640 1 -- 172.21.15.67:0/2719287278 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7f35b4010040 con 0x7f35bc15e280 2024-03-24T00:43:04.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.779+0000 7f35c2e4b640 1 -- 172.21.15.67:0/2719287278 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f35a006bc00 msgr2=0x7f35a006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:04.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.779+0000 7f35c2e4b640 1 --2- 172.21.15.67:0/2719287278 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f35a006bc00 0x7f35a006e0c0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f35b0036000 tx=0x7f35b003c040 comp rx=0 tx=0).stop 2024-03-24T00:43:04.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.779+0000 7f35c2e4b640 1 -- 172.21.15.67:0/2719287278 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35bc15e280 msgr2=0x7f35bc15f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:04.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.779+0000 7f35c2e4b640 1 --2- 172.21.15.67:0/2719287278 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35bc15e280 0x7f35bc15f330 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f35b400efb0 tx=0x7f35b400c490 comp rx=0 tx=0).stop 2024-03-24T00:43:04.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.779+0000 7f35c2e4b640 1 -- 172.21.15.67:0/2719287278 shutdown_connections 2024-03-24T00:43:04.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.779+0000 7f35c2e4b640 1 --2- 172.21.15.67:0/2719287278 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f35a006bc00 0x7f35a006e0c0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:04.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.779+0000 7f35c2e4b640 1 --2- 172.21.15.67:0/2719287278 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35bc15e280 0x7f35bc15f330 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:04.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.779+0000 7f35c2e4b640 1 --2- 172.21.15.67:0/2719287278 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f35bc15dea0 0x7f35bc153470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:04.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.779+0000 7f35c2e4b640 1 --2- 172.21.15.67:0/2719287278 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f35bc0b6170 0x7f35bc0cb0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:04.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.779+0000 7f35c2e4b640 1 -- 172.21.15.67:0/2719287278 >> 172.21.15.67:0/2719287278 conn(0x7f35bc072da0 msgr2=0x7f35bc0b8f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:04.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.779+0000 7f35c2e4b640 1 -- 172.21.15.67:0/2719287278 shutdown_connections 2024-03-24T00:43:04.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:04.779+0000 7f35c2e4b640 1 -- 172.21.15.67:0/2719287278 wait complete. 2024-03-24T00:43:04.791 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-24T00:43:04.791 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:43:04.838 DEBUG:teuthology.orchestra.run.smithi072:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:43:04.884 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-24T00:43:04.884 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:43:04.884 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:43:04.899 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056375,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15771 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T00:43:04.899 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:43:04.899 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:43:04.941 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17590 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248808k,nr_inodes=812202,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:43:04.942 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:43:04.942 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:43:04.956 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056375,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15771 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T00:43:04.956 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:43:04.956 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:43:04.999 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17590 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248808k,nr_inodes=812202,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:43:04.999 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-24T00:43:04.999 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:43:05.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.100+0000 7f4723190640 1 Processor -- start 2024-03-24T00:43:05.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.100+0000 7f4723190640 1 -- start start 2024-03-24T00:43:05.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.100+0000 7f4723190640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f471c15dec0 0x7f471c0ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:05.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.100+0000 7f4723190640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f471c15e2a0 0x7f471c162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:05.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.100+0000 7f4723190640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f471c0cf640 con 0x7f471c0d07e0 2024-03-24T00:43:05.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.100+0000 7f4723190640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f471c0cee30 con 0x7f471c15dec0 2024-03-24T00:43:05.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.100+0000 7f4723190640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f471c0d2120 con 0x7f471c15e2a0 2024-03-24T00:43:05.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.100+0000 7f4720f2c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f471c15dec0 0x7f471c0ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:05.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.100+0000 7f472172d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f471c0d07e0 0x7f471c0d0be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44872/0 (socket says 172.21.15.67:44872) 2024-03-24T00:43:05.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.101+0000 7f472172d640 1 -- 172.21.15.67:0/3422363355 learned_addr learned my addr 172.21.15.67:0/3422363355 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:43:05.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.101+0000 7f471bfff640 1 --2- 172.21.15.67:0/3422363355 >> v2:172.21.15.72:3300/0 conn(0x7f471c15e2a0 0x7f471c162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:05.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.101+0000 7f4720f2c640 1 -- 172.21.15.67:0/3422363355 >> v2:172.21.15.72:3300/0 conn(0x7f471c15e2a0 msgr2=0x7f471c162630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:05.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.101+0000 7f4720f2c640 1 --2- 172.21.15.67:0/3422363355 >> v2:172.21.15.72:3300/0 conn(0x7f471c15e2a0 0x7f471c162630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:05.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.101+0000 7f4720f2c640 1 -- 172.21.15.67:0/3422363355 >> v1:172.21.15.67:6789/0 conn(0x7f471c0d07e0 legacy=0x7f471c0d0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:05.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.101+0000 7f4720f2c640 1 -- 172.21.15.67:0/3422363355 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f471c0cf080 con 0x7f471c15dec0 2024-03-24T00:43:05.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.101+0000 7f471b7fe640 1 -- 172.21.15.67:0/3422363355 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 355558120 0 0) 0x7f4714001490 con 0x7f471c0d07e0 2024-03-24T00:43:05.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.101+0000 7f471bfff640 1 --2- 172.21.15.67:0/3422363355 >> v2:172.21.15.72:3300/0 conn(0x7f471c15e2a0 0x7f471c162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:43:05.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.102+0000 7f4720f2c640 1 --2- 172.21.15.67:0/3422363355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f471c15dec0 0x7f471c0ce630 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f471c0cede0 tx=0x7f471002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6439fb032436c6eb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:05.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.102+0000 7f471b7fe640 1 -- 172.21.15.67:0/3422363355 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47100315a0 con 0x7f471c15dec0 2024-03-24T00:43:05.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.102+0000 7f471b7fe640 1 -- 172.21.15.67:0/3422363355 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4710037070 con 0x7f471c15dec0 2024-03-24T00:43:05.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.102+0000 7f471b7fe640 1 -- 172.21.15.67:0/3422363355 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4710031960 con 0x7f471c15dec0 2024-03-24T00:43:05.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.102+0000 7f4723190640 1 -- 172.21.15.67:0/3422363355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f471c15dec0 msgr2=0x7f471c0ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:05.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.102+0000 7f4723190640 1 --2- 172.21.15.67:0/3422363355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f471c15dec0 0x7f471c0ce630 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f471c0cede0 tx=0x7f471002b840 comp rx=0 tx=0).stop 2024-03-24T00:43:05.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.102+0000 7f4723190640 1 -- 172.21.15.67:0/3422363355 shutdown_connections 2024-03-24T00:43:05.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.102+0000 7f4723190640 1 --2- 172.21.15.67:0/3422363355 >> v2:172.21.15.72:3300/0 conn(0x7f471c15e2a0 0x7f471c162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:05.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.102+0000 7f4723190640 1 --2- 172.21.15.67:0/3422363355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f471c15dec0 0x7f471c0ce630 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:05.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.102+0000 7f4723190640 1 -- 172.21.15.67:0/3422363355 >> 172.21.15.67:0/3422363355 conn(0x7f471c06c810 msgr2=0x7f471c06cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:05.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.102+0000 7f4723190640 1 -- 172.21.15.67:0/3422363355 shutdown_connections 2024-03-24T00:43:05.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.102+0000 7f4723190640 1 -- 172.21.15.67:0/3422363355 wait complete. 2024-03-24T00:43:05.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.103+0000 7f4723190640 1 Processor -- start 2024-03-24T00:43:05.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.103+0000 7f4723190640 1 -- start start 2024-03-24T00:43:05.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.104+0000 7f4723190640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f471c0d07e0 0x7f471c14f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:05.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.104+0000 7f4723190640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f471c15dec0 0x7f471c0b8a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:05.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.104+0000 7f4723190640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f471c15e2a0 0x7f471c0b8f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:05.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.104+0000 7f4723190640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f471c0d2730 con 0x7f471c15dec0 2024-03-24T00:43:05.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.104+0000 7f4723190640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f471c0d25b0 con 0x7f471c0d07e0 2024-03-24T00:43:05.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.104+0000 7f4723190640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f471c0d28b0 con 0x7f471c15e2a0 2024-03-24T00:43:05.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.104+0000 7f471bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f471c15dec0 0x7f471c0b8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:05.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.104+0000 7f471bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f471c15dec0 0x7f471c0b8a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47128/0 (socket says 172.21.15.67:47128) 2024-03-24T00:43:05.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.104+0000 7f471bfff640 1 -- 172.21.15.67:0/2698171285 learned_addr learned my addr 172.21.15.67:0/2698171285 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:05.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.105+0000 7f4720f2c640 1 --2- 172.21.15.67:0/2698171285 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f471c0d07e0 0x7f471c14f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:05.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.105+0000 7f472172d640 1 --2- 172.21.15.67:0/2698171285 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f471c15e2a0 0x7f471c0b8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:05.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.105+0000 7f471bfff640 1 -- 172.21.15.67:0/2698171285 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f471c15e2a0 msgr2=0x7f471c0b8f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:05.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.105+0000 7f471bfff640 1 --2- 172.21.15.67:0/2698171285 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f471c15e2a0 0x7f471c0b8f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:05.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.105+0000 7f471bfff640 1 -- 172.21.15.67:0/2698171285 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f471c0d07e0 msgr2=0x7f471c14f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:05.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.105+0000 7f471bfff640 1 --2- 172.21.15.67:0/2698171285 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f471c0d07e0 0x7f471c14f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:05.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.105+0000 7f471bfff640 1 -- 172.21.15.67:0/2698171285 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f471c0b4890 con 0x7f471c15dec0 2024-03-24T00:43:05.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.105+0000 7f471bfff640 1 --2- 172.21.15.67:0/2698171285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f471c15dec0 0x7f471c0b8a40 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f470800c7c0 tx=0x7f470800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:05.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.105+0000 7f47197fa640 1 -- 172.21.15.67:0/2698171285 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4708015070 con 0x7f471c15dec0 2024-03-24T00:43:05.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.105+0000 7f47197fa640 1 -- 172.21.15.67:0/2698171285 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4708007820 con 0x7f471c15dec0 2024-03-24T00:43:05.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.105+0000 7f4723190640 1 -- 172.21.15.67:0/2698171285 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f471c0b4b70 con 0x7f471c15dec0 2024-03-24T00:43:05.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.105+0000 7f47197fa640 1 -- 172.21.15.67:0/2698171285 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4708007bf0 con 0x7f471c15dec0 2024-03-24T00:43:05.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.105+0000 7f4723190640 1 -- 172.21.15.67:0/2698171285 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f471c0b5130 con 0x7f471c15dec0 2024-03-24T00:43:05.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.106+0000 7f4723190640 1 -- 172.21.15.67:0/2698171285 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f471c04f400 con 0x7f471c15dec0 2024-03-24T00:43:05.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.107+0000 7f47197fa640 1 -- 172.21.15.67:0/2698171285 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4708018070 con 0x7f471c15dec0 2024-03-24T00:43:05.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.107+0000 7f47197fa640 1 --2- 172.21.15.67:0/2698171285 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f470006bbd0 0x7f470006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:05.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.107+0000 7f47197fa640 1 -- 172.21.15.67:0/2698171285 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f470808a700 con 0x7f471c15dec0 2024-03-24T00:43:05.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.107+0000 7f4720f2c640 1 --2- 172.21.15.67:0/2698171285 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f470006bbd0 0x7f470006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:05.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.108+0000 7f4720f2c640 1 --2- 172.21.15.67:0/2698171285 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f470006bbd0 0x7f470006e090 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f4710036000 tx=0x7f471003d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:05.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.111+0000 7f47197fa640 1 -- 172.21.15.67:0/2698171285 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4708058da0 con 0x7f471c15dec0 2024-03-24T00:43:05.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.331+0000 7f4723190640 1 -- 172.21.15.67:0/2698171285 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f471c0d1aa0 con 0x7f471c15dec0 2024-03-24T00:43:05.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.332+0000 7f47197fa640 1 -- 172.21.15.67:0/2698171285 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+3699 (secure 0 0 0) 0x7f470805ce40 con 0x7f471c15dec0 2024-03-24T00:43:05.333 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:43:05.333 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":36,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5627,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2941726873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2941726873},{"type":"v1","addr":"172.21.15.72:6837","nonce":2941726873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":6166,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/1957901519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":1957901519},{"type":"v1","addr":"172.21.15.72:6836","nonce":1957901519}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":6357,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/2843754917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2843754917},{"type":"v1","addr":"172.21.15.67:6833","nonce":2843754917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":6825,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1350838876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1350838876},{"type":"v1","addr":"172.21.15.67:6835","nonce":1350838876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[]} 2024-03-24T00:43:05.333 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 36 2024-03-24T00:43:05.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.333+0000 7f4723190640 1 -- 172.21.15.67:0/2698171285 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f470006bbd0 msgr2=0x7f470006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:05.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.333+0000 7f4723190640 1 --2- 172.21.15.67:0/2698171285 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f470006bbd0 0x7f470006e090 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f4710036000 tx=0x7f471003d040 comp rx=0 tx=0).stop 2024-03-24T00:43:05.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.333+0000 7f4723190640 1 -- 172.21.15.67:0/2698171285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f471c15dec0 msgr2=0x7f471c0b8a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:05.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.333+0000 7f4723190640 1 --2- 172.21.15.67:0/2698171285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f471c15dec0 0x7f471c0b8a40 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f470800c7c0 tx=0x7f470800cc80 comp rx=0 tx=0).stop 2024-03-24T00:43:05.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.334+0000 7f4723190640 1 -- 172.21.15.67:0/2698171285 shutdown_connections 2024-03-24T00:43:05.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.334+0000 7f4723190640 1 --2- 172.21.15.67:0/2698171285 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f470006bbd0 0x7f470006e090 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:05.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.334+0000 7f4723190640 1 --2- 172.21.15.67:0/2698171285 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f471c15e2a0 0x7f471c0b8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:05.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.334+0000 7f4723190640 1 --2- 172.21.15.67:0/2698171285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f471c15dec0 0x7f471c0b8a40 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:05.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.334+0000 7f4723190640 1 --2- 172.21.15.67:0/2698171285 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f471c0d07e0 0x7f471c14f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:05.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.334+0000 7f4723190640 1 -- 172.21.15.67:0/2698171285 >> 172.21.15.67:0/2698171285 conn(0x7f471c06c810 msgr2=0x7f471c15f2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:05.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.334+0000 7f4723190640 1 -- 172.21.15.67:0/2698171285 shutdown_connections 2024-03-24T00:43:05.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.334+0000 7f4723190640 1 -- 172.21.15.67:0/2698171285 wait complete. 2024-03-24T00:43:05.345 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-24T00:43:05.345 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-24T00:43:05.345 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-24T00:43:05.345 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:43:05.346 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-24T00:43:05.346 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-24T00:43:05.346 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-24T00:43:05.346 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:43:05.346 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-24T00:43:05.346 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-24T00:43:05.346 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-24T00:43:05.347 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:43:05.347 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-24T00:43:05.347 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-24T00:43:05.347 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-24T00:43:05.347 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:43:05.355 INFO:tasks.ceph.mds.a:Stopped 2024-03-24T00:43:05.355 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-24T00:43:05.356 INFO:tasks.ceph.mds.c:Stopped 2024-03-24T00:43:05.356 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-24T00:43:05.359 INFO:tasks.ceph.mds.b:Stopped 2024-03-24T00:43:05.359 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-24T00:43:05.360 INFO:tasks.ceph.mds.d:Stopped 2024-03-24T00:43:05.360 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-24T00:43:05.398 INFO:tasks.ceph.mds.a:Started 2024-03-24T00:43:05.399 DEBUG:teuthology.parallel:result is None 2024-03-24T00:43:05.400 INFO:tasks.ceph.mds.c:Started 2024-03-24T00:43:05.400 DEBUG:teuthology.parallel:result is None 2024-03-24T00:43:05.402 INFO:tasks.ceph.mds.b:Started 2024-03-24T00:43:05.402 DEBUG:teuthology.parallel:result is None 2024-03-24T00:43:05.402 INFO:tasks.ceph.mds.d:Started 2024-03-24T00:43:05.402 DEBUG:teuthology.parallel:result is None 2024-03-24T00:43:05.403 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-24T00:43:05.475 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-03-24T00:43:05.476 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-03-24T00:43:05.486 INFO:tasks.ceph.mds.b.smithi072.stdout:starting mds.b at 2024-03-24T00:43:05.486 INFO:tasks.ceph.mds.d.smithi072.stdout:starting mds.d at 2024-03-24T00:43:05.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.508+0000 7f5216026640 1 Processor -- start 2024-03-24T00:43:05.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.508+0000 7f5216026640 1 -- start start 2024-03-24T00:43:05.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.509+0000 7f5216026640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f52100d32d0 0x7f52100d3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:05.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.509+0000 7f5216026640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f52100b6dc0 0x7f52100b7210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:05.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.509+0000 7f520f7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f521015e0f0 0x7f52100b95b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44900/0 (socket says 172.21.15.67:44900) 2024-03-24T00:43:05.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.509+0000 7f520f7fe640 1 -- 172.21.15.67:0/2917444683 learned_addr learned my addr 172.21.15.67:0/2917444683 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:43:05.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.509+0000 7f520ffff640 1 --2- 172.21.15.67:0/2917444683 >> v2:172.21.15.67:3300/0 conn(0x7f52100b6dc0 0x7f52100b7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:05.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.509+0000 7f5216026640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52100b9cc0 con 0x7f521015e0f0 2024-03-24T00:43:05.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.509+0000 7f5216026640 1 -- 172.21.15.67:0/2917444683 --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52100b7750 con 0x7f52100b6dc0 2024-03-24T00:43:05.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.510+0000 7f5216026640 1 -- 172.21.15.67:0/2917444683 --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52100d2230 con 0x7f52100d32d0 2024-03-24T00:43:05.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.510+0000 7f520effd640 1 --2- 172.21.15.67:0/2917444683 >> v2:172.21.15.72:3300/0 conn(0x7f52100d32d0 0x7f52100d3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:05.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.510+0000 7f520e7fc640 1 -- 172.21.15.67:0/2917444683 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 278903278 0 0) 0x7f52100b9cc0 con 0x7f521015e0f0 2024-03-24T00:43:05.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.510+0000 7f520e7fc640 1 -- 172.21.15.67:0/2917444683 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51f8003520 con 0x7f521015e0f0 2024-03-24T00:43:05.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.510+0000 7f520ffff640 1 -- 172.21.15.67:0/2917444683 >> v2:172.21.15.72:3300/0 conn(0x7f52100d32d0 msgr2=0x7f52100d3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:05.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.510+0000 7f520ffff640 1 --2- 172.21.15.67:0/2917444683 >> v2:172.21.15.72:3300/0 conn(0x7f52100d32d0 0x7f52100d3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:05.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.510+0000 7f520ffff640 1 -- 172.21.15.67:0/2917444683 >> v1:172.21.15.67:6789/0 conn(0x7f521015e0f0 legacy=0x7f52100b95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:05.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.510+0000 7f520ffff640 1 -- 172.21.15.67:0/2917444683 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52100d2510 con 0x7f52100b6dc0 2024-03-24T00:43:05.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.510+0000 7f520effd640 1 --2- 172.21.15.67:0/2917444683 >> v2:172.21.15.72:3300/0 conn(0x7f52100d32d0 0x7f52100d3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:43:05.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.511+0000 7f520ffff640 1 --2- 172.21.15.67:0/2917444683 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52100b6dc0 0x7f52100b7210 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f52100cacc0 tx=0x7f520802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b717d6d81feae8c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:05.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.511+0000 7f520e7fc640 1 -- 172.21.15.67:0/2917444683 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5208039070 con 0x7f52100b6dc0 2024-03-24T00:43:05.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.511+0000 7f520e7fc640 1 -- 172.21.15.67:0/2917444683 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f520802cc80 con 0x7f52100b6dc0 2024-03-24T00:43:05.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.511+0000 7f520e7fc640 1 -- 172.21.15.67:0/2917444683 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5208034660 con 0x7f52100b6dc0 2024-03-24T00:43:05.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.511+0000 7f5216026640 1 -- 172.21.15.67:0/2917444683 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52100b6dc0 msgr2=0x7f52100b7210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:05.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.511+0000 7f5216026640 1 --2- 172.21.15.67:0/2917444683 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52100b6dc0 0x7f52100b7210 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f52100cacc0 tx=0x7f520802bed0 comp rx=0 tx=0).stop 2024-03-24T00:43:05.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.511+0000 7f5216026640 1 -- 172.21.15.67:0/2917444683 shutdown_connections 2024-03-24T00:43:05.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.511+0000 7f5216026640 1 --2- 172.21.15.67:0/2917444683 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52100b6dc0 0x7f52100b7210 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:05.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.511+0000 7f5216026640 1 --2- 172.21.15.67:0/2917444683 >> v2:172.21.15.72:3300/0 conn(0x7f52100d32d0 0x7f52100d3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:05.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.511+0000 7f5216026640 1 -- 172.21.15.67:0/2917444683 >> 172.21.15.67:0/2917444683 conn(0x7f5210072b60 msgr2=0x7f5210072f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:05.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.511+0000 7f5216026640 1 -- 172.21.15.67:0/2917444683 shutdown_connections 2024-03-24T00:43:05.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.511+0000 7f5216026640 1 -- 172.21.15.67:0/2917444683 wait complete. 2024-03-24T00:43:05.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.512+0000 7f5216026640 1 Processor -- start 2024-03-24T00:43:05.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.513+0000 7f5216026640 1 -- start start 2024-03-24T00:43:05.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.513+0000 7f5216026640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f52100b6dc0 0x7f5210149a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:05.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.513+0000 7f5216026640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52100d32d0 0x7f5210146eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:05.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.513+0000 7f5216026640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f521015e0f0 0x7f5210153a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:05.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.513+0000 7f520effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52100d32d0 0x7f5210146eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:05.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.513+0000 7f520f7fe640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f52100b6dc0 0x7f5210149a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:05.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.513+0000 7f520effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52100d32d0 0x7f5210146eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47184/0 (socket says 172.21.15.67:47184) 2024-03-24T00:43:05.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.513+0000 7f520effd640 1 -- 172.21.15.67:0/1325293152 learned_addr learned my addr 172.21.15.67:0/1325293152 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:05.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.513+0000 7f5216026640 1 -- 172.21.15.67:0/1325293152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52100d2db0 con 0x7f52100d32d0 2024-03-24T00:43:05.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.513+0000 7f5216026640 1 -- 172.21.15.67:0/1325293152 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52100d2ad0 con 0x7f521015e0f0 2024-03-24T00:43:05.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.514+0000 7f5216026640 1 -- 172.21.15.67:0/1325293152 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52100b31d0 con 0x7f52100b6dc0 2024-03-24T00:43:05.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.514+0000 7f520ffff640 1 --2- 172.21.15.67:0/1325293152 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f521015e0f0 0x7f5210153a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:05.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.514+0000 7f520effd640 1 -- 172.21.15.67:0/1325293152 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f52100b6dc0 msgr2=0x7f5210149a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:05.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.514+0000 7f520effd640 1 --2- 172.21.15.67:0/1325293152 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f52100b6dc0 0x7f5210149a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:05.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.514+0000 7f520effd640 1 -- 172.21.15.67:0/1325293152 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f521015e0f0 msgr2=0x7f5210153a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:05.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.514+0000 7f520effd640 1 --2- 172.21.15.67:0/1325293152 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f521015e0f0 0x7f5210153a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:05.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.514+0000 7f520effd640 1 -- 172.21.15.67:0/1325293152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5210156c70 con 0x7f52100d32d0 2024-03-24T00:43:05.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.514+0000 7f520f7fe640 1 --2- 172.21.15.67:0/1325293152 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f52100b6dc0 0x7f5210149a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:43:05.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.514+0000 7f520effd640 1 --2- 172.21.15.67:0/1325293152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52100d32d0 0x7f5210146eb0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f520000ec10 tx=0x7f520000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:05.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.514+0000 7f51f7fff640 1 -- 172.21.15.67:0/1325293152 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f520000cc30 con 0x7f52100d32d0 2024-03-24T00:43:05.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.514+0000 7f51f7fff640 1 -- 172.21.15.67:0/1325293152 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f520000cdd0 con 0x7f52100d32d0 2024-03-24T00:43:05.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.514+0000 7f51f7fff640 1 -- 172.21.15.67:0/1325293152 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5200005bb0 con 0x7f52100d32d0 2024-03-24T00:43:05.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.514+0000 7f5216026640 1 -- 172.21.15.67:0/1325293152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52100b4910 con 0x7f52100d32d0 2024-03-24T00:43:05.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.514+0000 7f5216026640 1 -- 172.21.15.67:0/1325293152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52100b4ed0 con 0x7f52100d32d0 2024-03-24T00:43:05.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.515+0000 7f5216026640 1 -- 172.21.15.67:0/1325293152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f521004f400 con 0x7f52100d32d0 2024-03-24T00:43:05.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.516+0000 7f51f7fff640 1 -- 172.21.15.67:0/1325293152 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5200006030 con 0x7f52100d32d0 2024-03-24T00:43:05.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.516+0000 7f51f7fff640 1 --2- 172.21.15.67:0/1325293152 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f51fc06bc00 0x7f51fc06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:05.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.517+0000 7f51f7fff640 1 -- 172.21.15.67:0/1325293152 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f520008afc0 con 0x7f52100d32d0 2024-03-24T00:43:05.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.517+0000 7f520f7fe640 1 --2- 172.21.15.67:0/1325293152 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f51fc06bc00 0x7f51fc06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:05.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.517+0000 7f520f7fe640 1 --2- 172.21.15.67:0/1325293152 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f51fc06bc00 0x7f51fc06e0c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f52040011e0 tx=0x7f5204007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:05.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.520+0000 7f51f7fff640 1 -- 172.21.15.67:0/1325293152 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5200017020 con 0x7f52100d32d0 2024-03-24T00:43:05.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:05.715+0000 7f5216026640 1 -- 172.21.15.67:0/1325293152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f521006b750 con 0x7f52100d32d0 2024-03-24T00:43:06.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.798+0000 7f51f7fff640 1 -- 172.21.15.67:0/1325293152 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v72) v1 ==== 97+0+0 (secure 0 0 0) 0x7f5200059800 con 0x7f52100d32d0 2024-03-24T00:43:06.799 INFO:teuthology.orchestra.run.smithi067.stderr: removed all blocklist entries 2024-03-24T00:43:06.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.801+0000 7f5216026640 1 -- 172.21.15.67:0/1325293152 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f51fc06bc00 msgr2=0x7f51fc06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:06.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.801+0000 7f5216026640 1 --2- 172.21.15.67:0/1325293152 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f51fc06bc00 0x7f51fc06e0c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f52040011e0 tx=0x7f5204007d30 comp rx=0 tx=0).stop 2024-03-24T00:43:06.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.801+0000 7f5216026640 1 -- 172.21.15.67:0/1325293152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52100d32d0 msgr2=0x7f5210146eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:06.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.801+0000 7f5216026640 1 --2- 172.21.15.67:0/1325293152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52100d32d0 0x7f5210146eb0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f520000ec10 tx=0x7f520000c520 comp rx=0 tx=0).stop 2024-03-24T00:43:06.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.801+0000 7f5216026640 1 -- 172.21.15.67:0/1325293152 shutdown_connections 2024-03-24T00:43:06.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.801+0000 7f5216026640 1 --2- 172.21.15.67:0/1325293152 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f51fc06bc00 0x7f51fc06e0c0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:06.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.801+0000 7f5216026640 1 --2- 172.21.15.67:0/1325293152 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f521015e0f0 0x7f5210153a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:06.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.801+0000 7f5216026640 1 --2- 172.21.15.67:0/1325293152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52100d32d0 0x7f5210146eb0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:06.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.801+0000 7f5216026640 1 --2- 172.21.15.67:0/1325293152 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f52100b6dc0 0x7f5210149a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:06.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.801+0000 7f5216026640 1 -- 172.21.15.67:0/1325293152 >> 172.21.15.67:0/1325293152 conn(0x7f5210072b60 msgr2=0x7f52100ca130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:06.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.801+0000 7f5216026640 1 -- 172.21.15.67:0/1325293152 shutdown_connections 2024-03-24T00:43:06.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.801+0000 7f5216026640 1 -- 172.21.15.67:0/1325293152 wait complete. 2024-03-24T00:43:06.815 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-24T00:43:06.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.922+0000 7f214db1b640 1 Processor -- start 2024-03-24T00:43:06.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.922+0000 7f214db1b640 1 -- start start 2024-03-24T00:43:06.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.923+0000 7f214db1b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2148145640 0x7f21480ccc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:06.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.923+0000 7f214db1b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f214815fb10 con 0x7f21480cf570 2024-03-24T00:43:06.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.923+0000 7f214db1b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f214815fdf0 con 0x7f21480cd1b0 2024-03-24T00:43:06.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.923+0000 7f214db1b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21480cd610 con 0x7f2148145640 2024-03-24T00:43:06.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.923+0000 7f2147fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f21480cf570 0x7f214806dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44912/0 (socket says 172.21.15.67:44912) 2024-03-24T00:43:06.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.923+0000 7f2147fff640 1 -- 172.21.15.67:0/154870248 learned_addr learned my addr 172.21.15.67:0/154870248 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:43:06.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.923+0000 7f21477fe640 1 --2- 172.21.15.67:0/154870248 >> v2:172.21.15.72:3300/0 conn(0x7f2148145640 0x7f21480ccc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:06.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.923+0000 7f21467fc640 1 -- 172.21.15.67:0/154870248 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 781036086 0 0) 0x7f214815fb10 con 0x7f21480cf570 2024-03-24T00:43:06.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.923+0000 7f21467fc640 1 -- 172.21.15.67:0/154870248 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2130003520 con 0x7f21480cf570 2024-03-24T00:43:06.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.923+0000 7f21467fc640 1 -- 172.21.15.67:0/154870248 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 418077163 0 0) 0x7f2130003520 con 0x7f21480cf570 2024-03-24T00:43:06.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.923+0000 7f21467fc640 1 -- 172.21.15.67:0/154870248 >> v2:172.21.15.72:3300/0 conn(0x7f2148145640 msgr2=0x7f21480ccc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:06.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.924+0000 7f21467fc640 1 --2- 172.21.15.67:0/154870248 >> v2:172.21.15.72:3300/0 conn(0x7f2148145640 0x7f21480ccc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:06.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.924+0000 7f21467fc640 1 -- 172.21.15.67:0/154870248 >> v1:172.21.15.72:6789/0 conn(0x7f21480cd1b0 legacy=0x7f21480cee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:06.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.924+0000 7f21467fc640 1 -- 172.21.15.67:0/154870248 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21481600d0 con 0x7f21480cf570 2024-03-24T00:43:06.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.924+0000 7f21467fc640 1 -- 172.21.15.67:0/154870248 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f21400019f0 con 0x7f21480cf570 2024-03-24T00:43:06.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.924+0000 7f21467fc640 1 -- 172.21.15.67:0/154870248 >> v1:172.21.15.67:6789/0 conn(0x7f21480cf570 legacy=0x7f214806dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:06.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.924+0000 7f21467fc640 1 --2- 172.21.15.67:0/154870248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21300039b0 0x7f2130003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:06.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.924+0000 7f21467fc640 1 --2- 172.21.15.67:0/154870248 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2130004800 0x7f2130004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:06.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.924+0000 7f21467fc640 1 --2- 172.21.15.67:0/154870248 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f21300051c0 0x7f213000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:06.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.924+0000 7f21467fc640 1 -- 172.21.15.67:0/154870248 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f213000c030 con 0x7f21300039b0 2024-03-24T00:43:06.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.924+0000 7f21467fc640 1 -- 172.21.15.67:0/154870248 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f213000c1e0 con 0x7f21300051c0 2024-03-24T00:43:06.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.924+0000 7f21467fc640 1 -- 172.21.15.67:0/154870248 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f213000c3e0 con 0x7f2130004800 2024-03-24T00:43:06.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.924+0000 7f21477fe640 1 --2- 172.21.15.67:0/154870248 >> v2:172.21.15.72:3300/0 conn(0x7f2148145640 0x7f21480ccc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:43:06.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.924+0000 7f2147fff640 1 --2- 172.21.15.67:0/154870248 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f21300051c0 0x7f213000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:06.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.924+0000 7f21477fe640 1 --2- 172.21.15.67:0/154870248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21300039b0 0x7f2130003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:06.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.924+0000 7f2146ffd640 1 --2- 172.21.15.67:0/154870248 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2130004800 0x7f2130004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:06.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.924+0000 7f21477fe640 1 -- 172.21.15.67:0/154870248 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2130004800 msgr2=0x7f2130004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:06.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.924+0000 7f21477fe640 1 --2- 172.21.15.67:0/154870248 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2130004800 0x7f2130004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:06.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.924+0000 7f21477fe640 1 -- 172.21.15.67:0/154870248 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f21300051c0 msgr2=0x7f213000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:06.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.924+0000 7f21477fe640 1 --2- 172.21.15.67:0/154870248 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f21300051c0 0x7f213000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:06.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.924+0000 7f21477fe640 1 -- 172.21.15.67:0/154870248 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2130003520 con 0x7f21300039b0 2024-03-24T00:43:06.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.924+0000 7f2147fff640 1 -- 172.21.15.67:0/154870248 reap_dead start 2024-03-24T00:43:06.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.925+0000 7f21477fe640 1 --2- 172.21.15.67:0/154870248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21300039b0 0x7f2130003dd0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f2138009c70 tx=0x7f2130005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abcf409137f59a56 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:06.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.925+0000 7f21467fc640 1 -- 172.21.15.67:0/154870248 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2138009840 con 0x7f21300039b0 2024-03-24T00:43:06.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.925+0000 7f21467fc640 1 -- 172.21.15.67:0/154870248 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2138004020 con 0x7f21300039b0 2024-03-24T00:43:06.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.925+0000 7f21467fc640 1 -- 172.21.15.67:0/154870248 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2138004450 con 0x7f21300039b0 2024-03-24T00:43:06.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.925+0000 7f214db1b640 1 -- 172.21.15.67:0/154870248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21300039b0 msgr2=0x7f2130003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:06.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.925+0000 7f214db1b640 1 --2- 172.21.15.67:0/154870248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21300039b0 0x7f2130003dd0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f2138009c70 tx=0x7f2130005830 comp rx=0 tx=0).stop 2024-03-24T00:43:06.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.926+0000 7f214db1b640 1 -- 172.21.15.67:0/154870248 shutdown_connections 2024-03-24T00:43:06.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.926+0000 7f214db1b640 1 --2- 172.21.15.67:0/154870248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21300039b0 0x7f2130003dd0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:06.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.926+0000 7f214db1b640 1 -- 172.21.15.67:0/154870248 >> 172.21.15.67:0/154870248 conn(0x7f21480cf9e0 msgr2=0x7f21480cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:06.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.926+0000 7f214db1b640 1 -- 172.21.15.67:0/154870248 shutdown_connections 2024-03-24T00:43:06.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.926+0000 7f214db1b640 1 -- 172.21.15.67:0/154870248 wait complete. 2024-03-24T00:43:06.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.927+0000 7f214db1b640 1 Processor -- start 2024-03-24T00:43:06.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.927+0000 7f214db1b640 1 -- start start 2024-03-24T00:43:06.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.928+0000 7f214db1b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f21300039b0 0x7f2148138d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:06.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.928+0000 7f214db1b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f21481364e0 0x7f214813aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:06.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.928+0000 7f214db1b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2148137730 0x7f214813fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:06.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.928+0000 7f214db1b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2148160720 con 0x7f2148137730 2024-03-24T00:43:06.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.928+0000 7f214db1b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f214815f8c0 con 0x7f21300039b0 2024-03-24T00:43:06.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.928+0000 7f214db1b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21480ccc70 con 0x7f21481364e0 2024-03-24T00:43:06.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.928+0000 7f2147fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2148137730 0x7f214813fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:06.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.928+0000 7f2147fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2148137730 0x7f214813fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47190/0 (socket says 172.21.15.67:47190) 2024-03-24T00:43:06.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.928+0000 7f2147fff640 1 -- 172.21.15.67:0/273276007 learned_addr learned my addr 172.21.15.67:0/273276007 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:06.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.928+0000 7f21477fe640 1 --2- 172.21.15.67:0/273276007 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f21300039b0 0x7f2148138d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:06.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.928+0000 7f2146ffd640 1 --2- 172.21.15.67:0/273276007 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f21481364e0 0x7f214813aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:06.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.928+0000 7f2147fff640 1 -- 172.21.15.67:0/273276007 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f21481364e0 msgr2=0x7f214813aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:06.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.928+0000 7f2147fff640 1 --2- 172.21.15.67:0/273276007 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f21481364e0 0x7f214813aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:06.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.928+0000 7f2147fff640 1 -- 172.21.15.67:0/273276007 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f21300039b0 msgr2=0x7f2148138d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:06.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.928+0000 7f2147fff640 1 --2- 172.21.15.67:0/273276007 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f21300039b0 0x7f2148138d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:06.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.928+0000 7f2147fff640 1 -- 172.21.15.67:0/273276007 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2148138030 con 0x7f2148137730 2024-03-24T00:43:06.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.928+0000 7f2146ffd640 1 --2- 172.21.15.67:0/273276007 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f21481364e0 0x7f214813aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:43:06.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.928+0000 7f2147fff640 1 --2- 172.21.15.67:0/273276007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2148137730 0x7f214813fc70 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f2148072ae0 tx=0x7f2140007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:06.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.928+0000 7f212ffff640 1 -- 172.21.15.67:0/273276007 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2140017020 con 0x7f2148137730 2024-03-24T00:43:06.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.928+0000 7f212ffff640 1 -- 172.21.15.67:0/273276007 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f214000aac0 con 0x7f2148137730 2024-03-24T00:43:06.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.928+0000 7f214db1b640 1 -- 172.21.15.67:0/273276007 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21480b2d60 con 0x7f2148137730 2024-03-24T00:43:06.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.928+0000 7f212ffff640 1 -- 172.21.15.67:0/273276007 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21400133f0 con 0x7f2148137730 2024-03-24T00:43:06.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.929+0000 7f214db1b640 1 -- 172.21.15.67:0/273276007 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21480b3320 con 0x7f2148137730 2024-03-24T00:43:06.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.930+0000 7f214db1b640 1 -- 172.21.15.67:0/273276007 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f214804f400 con 0x7f2148137730 2024-03-24T00:43:06.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.930+0000 7f212ffff640 1 -- 172.21.15.67:0/273276007 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f214001a020 con 0x7f2148137730 2024-03-24T00:43:06.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.930+0000 7f212ffff640 1 --2- 172.21.15.67:0/273276007 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f213406bbd0 0x7f213406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:06.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.930+0000 7f212ffff640 1 -- 172.21.15.67:0/273276007 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f214008af00 con 0x7f2148137730 2024-03-24T00:43:06.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.931+0000 7f21477fe640 1 --2- 172.21.15.67:0/273276007 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f213406bbd0 0x7f213406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:06.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.931+0000 7f21477fe640 1 --2- 172.21.15.67:0/273276007 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f213406bbd0 0x7f213406e090 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f2138035000 tx=0x7f2138030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:06.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:06.935+0000 7f212ffff640 1 -- 172.21.15.67:0/273276007 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2140021020 con 0x7f2148137730 2024-03-24T00:43:07.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.186+0000 7f214db1b640 1 -- 172.21.15.67:0/273276007 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f21480b34f0 con 0x7f2148137730 2024-03-24T00:43:07.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.186+0000 7f212ffff640 1 -- 172.21.15.67:0/273276007 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f2140059bc0 con 0x7f2148137730 2024-03-24T00:43:07.187 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:43:07.187 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:43:07.187 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_dump": [ 2024-03-24T00:43:07.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:43:07.187 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.a", 2024-03-24T00:43:07.187 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9l4UENDhAASKUDtqcbSQ7wADj7uo4eyQ==", 2024-03-24T00:43:07.187 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:43:07.188 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:43:07.188 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-03-24T00:43:07.188 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-03-24T00:43:07.188 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:43:07.188 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:43:07.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:43:07.188 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:43:07.188 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.b", 2024-03-24T00:43:07.188 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9luhJWGhAAsS6PFHV2XdF9htjIjzdrdA==", 2024-03-24T00:43:07.188 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:43:07.188 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:43:07.188 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-03-24T00:43:07.188 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-03-24T00:43:07.188 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:43:07.188 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:43:07.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:43:07.188 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:43:07.189 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.c", 2024-03-24T00:43:07.189 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9l8mRjFRAAQ73yexNddpcyp8ayTcRKCg==", 2024-03-24T00:43:07.189 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:43:07.189 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:43:07.189 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-03-24T00:43:07.189 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-03-24T00:43:07.189 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:43:07.189 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:43:07.189 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:43:07.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:43:07.189 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.d", 2024-03-24T00:43:07.189 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9lcJyvIRAA2tpDb6qxn1MtljjEno4qPA==", 2024-03-24T00:43:07.189 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:43:07.189 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:43:07.189 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-03-24T00:43:07.189 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-03-24T00:43:07.190 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:43:07.190 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:43:07.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:43:07.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:43:07.190 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.0", 2024-03-24T00:43:07.190 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDgdf9lV/NUBRAAyJNajFAnNABJErnBlB8TEA==", 2024-03-24T00:43:07.190 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:43:07.190 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:43:07.191 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:43:07.191 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:43:07.191 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:43:07.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:43:07.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:43:07.191 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.1", 2024-03-24T00:43:07.191 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDjdf9l0J6lAxAA9RInt/X22jqzthE0vOiiFg==", 2024-03-24T00:43:07.191 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:43:07.191 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:43:07.191 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:43:07.191 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:43:07.191 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:43:07.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:43:07.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:43:07.191 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.2", 2024-03-24T00:43:07.191 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDmdf9lSmrEARAAwmtSQRNePsn4WIZODnjhfQ==", 2024-03-24T00:43:07.192 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:43:07.192 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:43:07.192 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:43:07.192 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:43:07.192 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:43:07.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:43:07.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:43:07.192 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.3", 2024-03-24T00:43:07.192 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDpdf9lm59bABAAgJ5AV774lw2BgkSCgaweyA==", 2024-03-24T00:43:07.192 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:43:07.192 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:43:07.192 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:43:07.192 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:43:07.192 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:43:07.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:43:07.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:43:07.193 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.4", 2024-03-24T00:43:07.193 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDvdf9lCo+pCRAANHkL0pPxxok6IsBCLAFCbg==", 2024-03-24T00:43:07.193 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:43:07.193 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:43:07.193 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:43:07.193 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:43:07.193 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:43:07.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:43:07.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:43:07.193 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.5", 2024-03-24T00:43:07.193 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDydf9lyGEdCBAAcuszPsthGlLvVryE7+D9iQ==", 2024-03-24T00:43:07.193 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:43:07.193 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:43:07.194 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:43:07.194 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:43:07.194 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:43:07.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:43:07.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:43:07.194 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.6", 2024-03-24T00:43:07.194 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQD1df9ljuZqBhAACt2OPNkjKOD3rK7g2mVBBA==", 2024-03-24T00:43:07.194 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:43:07.194 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:43:07.194 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:43:07.194 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:43:07.194 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:43:07.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:43:07.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:43:07.194 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.7", 2024-03-24T00:43:07.195 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQD4df9lYKBfBRAAy/cWFF6pvGDjXZHAuBYw7g==", 2024-03-24T00:43:07.195 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:43:07.195 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:43:07.195 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:43:07.195 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:43:07.195 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:43:07.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:43:07.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:43:07.195 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.0", 2024-03-24T00:43:07.195 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9lY4nvJRAA6HWOZDWA+SSvJVmyjG+51g==", 2024-03-24T00:43:07.195 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:43:07.195 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:43:07.195 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow r", 2024-03-24T00:43:07.195 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-24T00:43:07.195 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:43:07.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:43:07.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:43:07.196 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.1", 2024-03-24T00:43:07.196 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9lonsXLxAAaltVMW3n0qHX8u9X8EkSmg==", 2024-03-24T00:43:07.196 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:43:07.196 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:43:07.196 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-03-24T00:43:07.196 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-03-24T00:43:07.196 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-03-24T00:43:07.196 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:43:07.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:43:07.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:43:07.196 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.2", 2024-03-24T00:43:07.196 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9lLEm2KxAAkT65mhLrTirlDnOUyvBY7g==", 2024-03-24T00:43:07.196 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:43:07.196 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:43:07.196 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-03-24T00:43:07.197 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-03-24T00:43:07.197 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-03-24T00:43:07.197 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:43:07.197 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:43:07.197 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:43:07.197 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.3", 2024-03-24T00:43:07.197 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9l8njZNBAAaAktZpX2SPcJR9ymPm9ebw==", 2024-03-24T00:43:07.197 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:43:07.197 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:43:07.197 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-03-24T00:43:07.197 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-03-24T00:43:07.197 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-03-24T00:43:07.197 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:43:07.197 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:43:07.197 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:43:07.197 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.admin", 2024-03-24T00:43:07.197 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDbdf9lDTIwLRAAMajqhM/KfUNF/SjVv9OJDw==", 2024-03-24T00:43:07.198 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:43:07.198 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-03-24T00:43:07.198 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-03-24T00:43:07.198 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-03-24T00:43:07.198 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:43:07.198 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:43:07.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:43:07.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:43:07.198 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.x", 2024-03-24T00:43:07.198 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9lGwKvChAAxl+fRnjbAQ7dYJ727QFUgw==", 2024-03-24T00:43:07.198 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:43:07.198 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-03-24T00:43:07.198 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-03-24T00:43:07.198 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:43:07.198 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:43:07.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:43:07.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:43:07.199 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.y", 2024-03-24T00:43:07.199 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9lSvufBhAAoQ9L1GjLkstjrAw2lqlryA==", 2024-03-24T00:43:07.199 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:43:07.199 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-03-24T00:43:07.199 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-03-24T00:43:07.199 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:43:07.199 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:43:07.199 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:43:07.199 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-24T00:43:07.199 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:43:07.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.188+0000 7f214db1b640 1 -- 172.21.15.67:0/273276007 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f213406bbd0 msgr2=0x7f213406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:07.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.188+0000 7f214db1b640 1 --2- 172.21.15.67:0/273276007 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f213406bbd0 0x7f213406e090 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f2138035000 tx=0x7f2138030000 comp rx=0 tx=0).stop 2024-03-24T00:43:07.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.188+0000 7f214db1b640 1 -- 172.21.15.67:0/273276007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2148137730 msgr2=0x7f214813fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:07.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.188+0000 7f214db1b640 1 --2- 172.21.15.67:0/273276007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2148137730 0x7f214813fc70 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f2148072ae0 tx=0x7f2140007a90 comp rx=0 tx=0).stop 2024-03-24T00:43:07.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.188+0000 7f214db1b640 1 -- 172.21.15.67:0/273276007 shutdown_connections 2024-03-24T00:43:07.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.188+0000 7f214db1b640 1 --2- 172.21.15.67:0/273276007 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f213406bbd0 0x7f213406e090 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.188+0000 7f214db1b640 1 --2- 172.21.15.67:0/273276007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2148137730 0x7f214813fc70 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.188+0000 7f214db1b640 1 --2- 172.21.15.67:0/273276007 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f21481364e0 0x7f214813aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.188+0000 7f214db1b640 1 --2- 172.21.15.67:0/273276007 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f21300039b0 0x7f2148138d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.188+0000 7f214db1b640 1 -- 172.21.15.67:0/273276007 >> 172.21.15.67:0/273276007 conn(0x7f21480cf9e0 msgr2=0x7f21480cee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:07.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.188+0000 7f214db1b640 1 -- 172.21.15.67:0/273276007 shutdown_connections 2024-03-24T00:43:07.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.188+0000 7f214db1b640 1 -- 172.21.15.67:0/273276007 wait complete. 2024-03-24T00:43:07.202 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:43:07.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.302+0000 7fd2da612640 1 Processor -- start 2024-03-24T00:43:07.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.302+0000 7fd2da612640 1 -- start start 2024-03-24T00:43:07.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.303+0000 7fd2da612640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd2d415dec0 0x7fd2d40ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:07.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.303+0000 7fd2da612640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd2d415e2a0 0x7fd2d4162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:07.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.303+0000 7fd2da612640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd2d40d07e0 0x7fd2d40d0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:07.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.303+0000 7fd2da612640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2d40d1170 con 0x7fd2d415dec0 2024-03-24T00:43:07.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.303+0000 7fd2da612640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2d40cee30 con 0x7fd2d415e2a0 2024-03-24T00:43:07.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.303+0000 7fd2da612640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d40d2120 con 0x7fd2d40d07e0 2024-03-24T00:43:07.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.303+0000 7fd2d3fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd2d415dec0 0x7fd2d40ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:07.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.303+0000 7fd2d3fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd2d415dec0 0x7fd2d40ce630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47196/0 (socket says 172.21.15.67:47196) 2024-03-24T00:43:07.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.303+0000 7fd2d3fff640 1 -- 172.21.15.67:0/4185856192 learned_addr learned my addr 172.21.15.67:0/4185856192 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:07.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.304+0000 7fd2d8baf640 1 --2- 172.21.15.67:0/4185856192 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd2d40d07e0 0x7fd2d40d0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:07.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.304+0000 7fd2d37fe640 1 --2- 172.21.15.67:0/4185856192 >> v2:172.21.15.72:3300/0 conn(0x7fd2d415e2a0 0x7fd2d4162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:07.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.304+0000 7fd2d3fff640 1 -- 172.21.15.67:0/4185856192 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd2d40d07e0 msgr2=0x7fd2d40d0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:07.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.304+0000 7fd2d3fff640 1 --2- 172.21.15.67:0/4185856192 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd2d40d07e0 0x7fd2d40d0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.304+0000 7fd2d3fff640 1 -- 172.21.15.67:0/4185856192 >> v2:172.21.15.72:3300/0 conn(0x7fd2d415e2a0 msgr2=0x7fd2d4162630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:07.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.304+0000 7fd2d3fff640 1 --2- 172.21.15.67:0/4185856192 >> v2:172.21.15.72:3300/0 conn(0x7fd2d415e2a0 0x7fd2d4162630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.304+0000 7fd2d3fff640 1 -- 172.21.15.67:0/4185856192 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d40cf640 con 0x7fd2d415dec0 2024-03-24T00:43:07.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.304+0000 7fd2d37fe640 1 --2- 172.21.15.67:0/4185856192 >> v2:172.21.15.72:3300/0 conn(0x7fd2d415e2a0 0x7fd2d4162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:43:07.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.304+0000 7fd2d3fff640 1 --2- 172.21.15.67:0/4185856192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2d415dec0 0x7fd2d40ce630 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fd2d40cede0 tx=0x7fd2c402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce2bb218f4c7aa32 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:07.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.304+0000 7fd2d2ffd640 1 -- 172.21.15.67:0/4185856192 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c402e070 con 0x7fd2d415dec0 2024-03-24T00:43:07.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.305+0000 7fd2d2ffd640 1 -- 172.21.15.67:0/4185856192 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2c4037070 con 0x7fd2d415dec0 2024-03-24T00:43:07.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.305+0000 7fd2d2ffd640 1 -- 172.21.15.67:0/4185856192 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c4031a00 con 0x7fd2d415dec0 2024-03-24T00:43:07.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.305+0000 7fd2da612640 1 -- 172.21.15.67:0/4185856192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2d415dec0 msgr2=0x7fd2d40ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:07.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.305+0000 7fd2da612640 1 --2- 172.21.15.67:0/4185856192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2d415dec0 0x7fd2d40ce630 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fd2d40cede0 tx=0x7fd2c402b880 comp rx=0 tx=0).stop 2024-03-24T00:43:07.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.305+0000 7fd2da612640 1 -- 172.21.15.67:0/4185856192 shutdown_connections 2024-03-24T00:43:07.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.305+0000 7fd2da612640 1 --2- 172.21.15.67:0/4185856192 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd2d40d07e0 0x7fd2d40d0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.305+0000 7fd2da612640 1 --2- 172.21.15.67:0/4185856192 >> v2:172.21.15.72:3300/0 conn(0x7fd2d415e2a0 0x7fd2d4162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.305+0000 7fd2da612640 1 --2- 172.21.15.67:0/4185856192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2d415dec0 0x7fd2d40ce630 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.305+0000 7fd2da612640 1 -- 172.21.15.67:0/4185856192 >> 172.21.15.67:0/4185856192 conn(0x7fd2d406c810 msgr2=0x7fd2d406cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:07.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.305+0000 7fd2da612640 1 -- 172.21.15.67:0/4185856192 shutdown_connections 2024-03-24T00:43:07.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.305+0000 7fd2da612640 1 -- 172.21.15.67:0/4185856192 wait complete. 2024-03-24T00:43:07.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.306+0000 7fd2da612640 1 Processor -- start 2024-03-24T00:43:07.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.306+0000 7fd2da612640 1 -- start start 2024-03-24T00:43:07.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.307+0000 7fd2da612640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd2d40d07e0 0x7fd2d41f6d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:07.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.307+0000 7fd2da612640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2d415dec0 0x7fd2d40cb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:07.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.307+0000 7fd2da612640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd2d415e2a0 0x7fd2d40ccad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:07.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.307+0000 7fd2da612640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d40b5850 con 0x7fd2d415dec0 2024-03-24T00:43:07.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.307+0000 7fd2da612640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d40cf360 con 0x7fd2d415e2a0 2024-03-24T00:43:07.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.307+0000 7fd2da612640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d40b59d0 con 0x7fd2d40d07e0 2024-03-24T00:43:07.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.307+0000 7fd2d37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2d415dec0 0x7fd2d40cb8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:07.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.307+0000 7fd2d37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2d415dec0 0x7fd2d40cb8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47208/0 (socket says 172.21.15.67:47208) 2024-03-24T00:43:07.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.307+0000 7fd2d37fe640 1 -- 172.21.15.67:0/1342078556 learned_addr learned my addr 172.21.15.67:0/1342078556 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:07.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.307+0000 7fd2d3fff640 1 --2- 172.21.15.67:0/1342078556 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd2d40d07e0 0x7fd2d41f6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:07.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.307+0000 7fd2d8baf640 1 --2- 172.21.15.67:0/1342078556 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd2d415e2a0 0x7fd2d40ccad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:07.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.308+0000 7fd2d37fe640 1 -- 172.21.15.67:0/1342078556 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd2d40d07e0 msgr2=0x7fd2d41f6d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:07.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.308+0000 7fd2d37fe640 1 --2- 172.21.15.67:0/1342078556 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd2d40d07e0 0x7fd2d41f6d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.308+0000 7fd2d37fe640 1 -- 172.21.15.67:0/1342078556 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd2d415e2a0 msgr2=0x7fd2d40ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:07.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.308+0000 7fd2d37fe640 1 --2- 172.21.15.67:0/1342078556 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd2d415e2a0 0x7fd2d40ccad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.308+0000 7fd2d37fe640 1 -- 172.21.15.67:0/1342078556 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d41e73e0 con 0x7fd2d415dec0 2024-03-24T00:43:07.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.308+0000 7fd2d37fe640 1 --2- 172.21.15.67:0/1342078556 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2d415dec0 0x7fd2d40cb8b0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fd2c800c7c0 tx=0x7fd2c800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:07.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.308+0000 7fd2d0ff9640 1 -- 172.21.15.67:0/1342078556 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c8015070 con 0x7fd2d415dec0 2024-03-24T00:43:07.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.308+0000 7fd2d0ff9640 1 -- 172.21.15.67:0/1342078556 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2c8007820 con 0x7fd2d415dec0 2024-03-24T00:43:07.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.308+0000 7fd2da612640 1 -- 172.21.15.67:0/1342078556 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2d41e76c0 con 0x7fd2d415dec0 2024-03-24T00:43:07.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.308+0000 7fd2d0ff9640 1 -- 172.21.15.67:0/1342078556 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c8007bf0 con 0x7fd2d415dec0 2024-03-24T00:43:07.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.308+0000 7fd2da612640 1 -- 172.21.15.67:0/1342078556 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2d41e7c80 con 0x7fd2d415dec0 2024-03-24T00:43:07.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.309+0000 7fd2da612640 1 -- 172.21.15.67:0/1342078556 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2d404f400 con 0x7fd2d415dec0 2024-03-24T00:43:07.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.309+0000 7fd2d0ff9640 1 -- 172.21.15.67:0/1342078556 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd2c8018070 con 0x7fd2d415dec0 2024-03-24T00:43:07.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.310+0000 7fd2d0ff9640 1 --2- 172.21.15.67:0/1342078556 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd2c006bc00 0x7fd2c006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:07.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.310+0000 7fd2d0ff9640 1 -- 172.21.15.67:0/1342078556 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fd2c808a610 con 0x7fd2d415dec0 2024-03-24T00:43:07.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.310+0000 7fd2d3fff640 1 --2- 172.21.15.67:0/1342078556 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd2c006bc00 0x7fd2c006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:07.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.310+0000 7fd2d3fff640 1 --2- 172.21.15.67:0/1342078556 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd2c006bc00 0x7fd2c006e0c0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fd2c4036000 tx=0x7fd2c403d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:07.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.314+0000 7fd2d0ff9640 1 -- 172.21.15.67:0/1342078556 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd2c8058c70 con 0x7fd2d415dec0 2024-03-24T00:43:07.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.496+0000 7fd2da612640 1 -- 172.21.15.67:0/1342078556 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd2d40d1aa0 con 0x7fd2d415dec0 2024-03-24T00:43:07.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.497+0000 7fd2d0ff9640 1 -- 172.21.15.67:0/1342078556 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) v1 ==== 74+0+13105 (secure 0 0 0) 0x7fd2c805cd10 con 0x7fd2d415dec0 2024-03-24T00:43:07.498 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:43:07.498 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":72,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:43:06.797785+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:43:07.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.499+0000 7fd2da612640 1 -- 172.21.15.67:0/1342078556 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd2c006bc00 msgr2=0x7fd2c006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:07.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.499+0000 7fd2da612640 1 --2- 172.21.15.67:0/1342078556 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd2c006bc00 0x7fd2c006e0c0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fd2c4036000 tx=0x7fd2c403d040 comp rx=0 tx=0).stop 2024-03-24T00:43:07.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.499+0000 7fd2da612640 1 -- 172.21.15.67:0/1342078556 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2d415dec0 msgr2=0x7fd2d40cb8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:07.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.499+0000 7fd2da612640 1 --2- 172.21.15.67:0/1342078556 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2d415dec0 0x7fd2d40cb8b0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fd2c800c7c0 tx=0x7fd2c800cc80 comp rx=0 tx=0).stop 2024-03-24T00:43:07.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.499+0000 7fd2da612640 1 -- 172.21.15.67:0/1342078556 shutdown_connections 2024-03-24T00:43:07.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.499+0000 7fd2da612640 1 --2- 172.21.15.67:0/1342078556 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd2c006bc00 0x7fd2c006e0c0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.499+0000 7fd2da612640 1 --2- 172.21.15.67:0/1342078556 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd2d415e2a0 0x7fd2d40ccad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.499+0000 7fd2da612640 1 --2- 172.21.15.67:0/1342078556 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2d415dec0 0x7fd2d40cb8b0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.499+0000 7fd2da612640 1 --2- 172.21.15.67:0/1342078556 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd2d40d07e0 0x7fd2d41f6d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.499+0000 7fd2da612640 1 -- 172.21.15.67:0/1342078556 >> 172.21.15.67:0/1342078556 conn(0x7fd2d406c810 msgr2=0x7fd2d40b29b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:07.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.499+0000 7fd2da612640 1 -- 172.21.15.67:0/1342078556 shutdown_connections 2024-03-24T00:43:07.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.499+0000 7fd2da612640 1 -- 172.21.15.67:0/1342078556 wait complete. 2024-03-24T00:43:07.511 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:38:39.364950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:43:07.511 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:43:07.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.614+0000 7fcc79c75640 1 Processor -- start 2024-03-24T00:43:07.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.614+0000 7fcc79c75640 1 -- start start 2024-03-24T00:43:07.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.615+0000 7fcc79c75640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcc740d0d50 0x7fcc740ce610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:07.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.615+0000 7fcc79c75640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc740cf570 con 0x7fcc740d22b0 2024-03-24T00:43:07.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.615+0000 7fcc79c75640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc740cecf0 con 0x7fcc7415e130 2024-03-24T00:43:07.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.615+0000 7fcc79c75640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc740ceb50 con 0x7fcc740d0d50 2024-03-24T00:43:07.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.615+0000 7fcc72ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcc740d0d50 0x7fcc740ce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:07.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.615+0000 7fcc73fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fcc740d22b0 0x7fcc740d2700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44928/0 (socket says 172.21.15.67:44928) 2024-03-24T00:43:07.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.615+0000 7fcc73fff640 1 -- 172.21.15.67:0/1589909327 learned_addr learned my addr 172.21.15.67:0/1589909327 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:43:07.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.616+0000 7fcc72ffd640 1 -- 172.21.15.67:0/1589909327 >> v1:172.21.15.72:6789/0 conn(0x7fcc7415e130 legacy=0x7fcc740d0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:07.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.616+0000 7fcc72ffd640 1 -- 172.21.15.67:0/1589909327 >> v1:172.21.15.67:6789/0 conn(0x7fcc740d22b0 legacy=0x7fcc740d2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:07.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.616+0000 7fcc72ffd640 1 -- 172.21.15.67:0/1589909327 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc74162a60 con 0x7fcc740d0d50 2024-03-24T00:43:07.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.616+0000 7fcc727fc640 1 -- 172.21.15.67:0/1589909327 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3812781746 0 0) 0x7fcc740cf570 con 0x7fcc740d22b0 2024-03-24T00:43:07.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.616+0000 7fcc72ffd640 1 --2- 172.21.15.67:0/1589909327 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc740d0d50 0x7fcc740ce610 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fcc680097a0 tx=0x7fcc6802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ac8fe68784a5704 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:07.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.617+0000 7fcc727fc640 1 -- 172.21.15.67:0/1589909327 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc68035020 con 0x7fcc740d0d50 2024-03-24T00:43:07.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.617+0000 7fcc727fc640 1 -- 172.21.15.67:0/1589909327 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc68031830 con 0x7fcc740d0d50 2024-03-24T00:43:07.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.617+0000 7fcc727fc640 1 -- 172.21.15.67:0/1589909327 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc68031c00 con 0x7fcc740d0d50 2024-03-24T00:43:07.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.617+0000 7fcc79c75640 1 -- 172.21.15.67:0/1589909327 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc740d0d50 msgr2=0x7fcc740ce610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:07.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.617+0000 7fcc79c75640 1 --2- 172.21.15.67:0/1589909327 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc740d0d50 0x7fcc740ce610 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fcc680097a0 tx=0x7fcc6802b8d0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.617+0000 7fcc79c75640 1 -- 172.21.15.67:0/1589909327 shutdown_connections 2024-03-24T00:43:07.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.617+0000 7fcc79c75640 1 --2- 172.21.15.67:0/1589909327 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc740d0d50 0x7fcc740ce610 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.617+0000 7fcc79c75640 1 -- 172.21.15.67:0/1589909327 >> 172.21.15.67:0/1589909327 conn(0x7fcc7406c8a0 msgr2=0x7fcc7406ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:07.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.617+0000 7fcc79c75640 1 -- 172.21.15.67:0/1589909327 shutdown_connections 2024-03-24T00:43:07.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.617+0000 7fcc79c75640 1 -- 172.21.15.67:0/1589909327 wait complete. 2024-03-24T00:43:07.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.618+0000 7fcc79c75640 1 Processor -- start 2024-03-24T00:43:07.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.619+0000 7fcc79c75640 1 -- start start 2024-03-24T00:43:07.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.619+0000 7fcc79c75640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc740d0d50 0x7fcc741457d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:07.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.619+0000 7fcc79c75640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcc740d22b0 0x7fcc74160ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:07.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.619+0000 7fcc79c75640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc7415e130 0x7fcc740c7830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:07.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.619+0000 7fcc79c75640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7415e510 con 0x7fcc740d0d50 2024-03-24T00:43:07.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.619+0000 7fcc79c75640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc740cf290 con 0x7fcc7415e130 2024-03-24T00:43:07.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.619+0000 7fcc79c75640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc740b2e10 con 0x7fcc740d22b0 2024-03-24T00:43:07.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.619+0000 7fcc737fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc740d0d50 0x7fcc741457d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:07.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.619+0000 7fcc737fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc740d0d50 0x7fcc741457d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47228/0 (socket says 172.21.15.67:47228) 2024-03-24T00:43:07.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.619+0000 7fcc737fe640 1 -- 172.21.15.67:0/4245714542 learned_addr learned my addr 172.21.15.67:0/4245714542 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:07.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.619+0000 7fcc72ffd640 1 --2- 172.21.15.67:0/4245714542 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcc740d22b0 0x7fcc74160ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:07.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.620+0000 7fcc73fff640 1 --2- 172.21.15.67:0/4245714542 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc7415e130 0x7fcc740c7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:07.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.620+0000 7fcc737fe640 1 -- 172.21.15.67:0/4245714542 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcc740d22b0 msgr2=0x7fcc74160ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:07.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.620+0000 7fcc737fe640 1 --2- 172.21.15.67:0/4245714542 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcc740d22b0 0x7fcc74160ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.620+0000 7fcc737fe640 1 -- 172.21.15.67:0/4245714542 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc7415e130 msgr2=0x7fcc740c7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:07.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.620+0000 7fcc737fe640 1 --2- 172.21.15.67:0/4245714542 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc7415e130 0x7fcc740c7830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.620+0000 7fcc737fe640 1 -- 172.21.15.67:0/4245714542 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc740c9f40 con 0x7fcc740d0d50 2024-03-24T00:43:07.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.620+0000 7fcc737fe640 1 --2- 172.21.15.67:0/4245714542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc740d0d50 0x7fcc741457d0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fcc6400b3f0 tx=0x7fcc6400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:07.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.620+0000 7fcc5bfff640 1 -- 172.21.15.67:0/4245714542 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc64015020 con 0x7fcc740d0d50 2024-03-24T00:43:07.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.620+0000 7fcc5bfff640 1 -- 172.21.15.67:0/4245714542 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc64011830 con 0x7fcc740d0d50 2024-03-24T00:43:07.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.620+0000 7fcc79c75640 1 -- 172.21.15.67:0/4245714542 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc740ca220 con 0x7fcc740d0d50 2024-03-24T00:43:07.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.620+0000 7fcc5bfff640 1 -- 172.21.15.67:0/4245714542 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc64011c20 con 0x7fcc740d0d50 2024-03-24T00:43:07.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.620+0000 7fcc79c75640 1 -- 172.21.15.67:0/4245714542 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc740c7d70 con 0x7fcc740d0d50 2024-03-24T00:43:07.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.621+0000 7fcc79c75640 1 -- 172.21.15.67:0/4245714542 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc74161530 con 0x7fcc740d0d50 2024-03-24T00:43:07.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.621+0000 7fcc5bfff640 1 -- 172.21.15.67:0/4245714542 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fcc64018020 con 0x7fcc740d0d50 2024-03-24T00:43:07.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.622+0000 7fcc5bfff640 1 --2- 172.21.15.67:0/4245714542 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcc6006bc00 0x7fcc6006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:07.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.622+0000 7fcc5bfff640 1 -- 172.21.15.67:0/4245714542 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fcc6408a830 con 0x7fcc740d0d50 2024-03-24T00:43:07.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.622+0000 7fcc72ffd640 1 --2- 172.21.15.67:0/4245714542 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcc6006bc00 0x7fcc6006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:07.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.622+0000 7fcc72ffd640 1 --2- 172.21.15.67:0/4245714542 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcc6006bc00 0x7fcc6006e0c0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fcc68036000 tx=0x7fcc68008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:07.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.626+0000 7fcc5bfff640 1 -- 172.21.15.67:0/4245714542 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fcc6401f020 con 0x7fcc740d0d50 2024-03-24T00:43:07.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.808+0000 7fcc79c75640 1 -- 172.21.15.67:0/4245714542 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcc7404f400 con 0x7fcc740d0d50 2024-03-24T00:43:07.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.808+0000 7fcc5bfff640 1 -- 172.21.15.67:0/4245714542 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v72) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcc640172c0 con 0x7fcc740d0d50 2024-03-24T00:43:07.809 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-24T00:43:07.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.811+0000 7fcc79c75640 1 -- 172.21.15.67:0/4245714542 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcc6006bc00 msgr2=0x7fcc6006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:07.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.811+0000 7fcc79c75640 1 --2- 172.21.15.67:0/4245714542 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcc6006bc00 0x7fcc6006e0c0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fcc68036000 tx=0x7fcc68008030 comp rx=0 tx=0).stop 2024-03-24T00:43:07.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.811+0000 7fcc79c75640 1 -- 172.21.15.67:0/4245714542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc740d0d50 msgr2=0x7fcc741457d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:07.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.811+0000 7fcc79c75640 1 --2- 172.21.15.67:0/4245714542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc740d0d50 0x7fcc741457d0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fcc6400b3f0 tx=0x7fcc6400b8b0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.811+0000 7fcc79c75640 1 -- 172.21.15.67:0/4245714542 shutdown_connections 2024-03-24T00:43:07.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.811+0000 7fcc79c75640 1 --2- 172.21.15.67:0/4245714542 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcc6006bc00 0x7fcc6006e0c0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.811+0000 7fcc79c75640 1 --2- 172.21.15.67:0/4245714542 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc7415e130 0x7fcc740c7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.811+0000 7fcc79c75640 1 --2- 172.21.15.67:0/4245714542 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcc740d22b0 0x7fcc74160ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.811+0000 7fcc79c75640 1 --2- 172.21.15.67:0/4245714542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc740d0d50 0x7fcc741457d0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.811+0000 7fcc79c75640 1 -- 172.21.15.67:0/4245714542 >> 172.21.15.67:0/4245714542 conn(0x7fcc7406c8a0 msgr2=0x7fcc740b37f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:07.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.811+0000 7fcc79c75640 1 -- 172.21.15.67:0/4245714542 shutdown_connections 2024-03-24T00:43:07.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.811+0000 7fcc79c75640 1 -- 172.21.15.67:0/4245714542 wait complete. 2024-03-24T00:43:07.823 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-24T00:43:07.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.923+0000 7f3e31959640 1 Processor -- start 2024-03-24T00:43:07.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.923+0000 7f3e31959640 1 -- start start 2024-03-24T00:43:07.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.924+0000 7f3e31959640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3e2c0cc360 0x7f3e2c066e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:07.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.924+0000 7f3e31959640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e2c06c210 con 0x7f3e2c06ba50 2024-03-24T00:43:07.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.924+0000 7f3e31959640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e2c0b4240 con 0x7f3e2c0cd410 2024-03-24T00:43:07.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.924+0000 7f3e31959640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2c0caad0 con 0x7f3e2c0cc360 2024-03-24T00:43:07.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.924+0000 7f3e2a7fc640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3e2c06ba50 0x7f3e2c0ccd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44932/0 (socket says 172.21.15.67:44932) 2024-03-24T00:43:07.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.924+0000 7f3e2a7fc640 1 -- 172.21.15.67:0/262504080 learned_addr learned my addr 172.21.15.67:0/262504080 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:43:07.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.924+0000 7f3e29ffb640 1 -- 172.21.15.67:0/262504080 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3149977161 0 0) 0x7f3e2c06c210 con 0x7f3e2c06ba50 2024-03-24T00:43:07.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.924+0000 7f3e29ffb640 1 -- 172.21.15.67:0/262504080 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e18003520 con 0x7f3e2c06ba50 2024-03-24T00:43:07.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.924+0000 7f3e2affd640 1 --2- 172.21.15.67:0/262504080 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3e2c0cc360 0x7f3e2c066e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:07.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.925+0000 7f3e29ffb640 1 -- 172.21.15.67:0/262504080 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4183873160 0 0) 0x7f3e18003520 con 0x7f3e2c06ba50 2024-03-24T00:43:07.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.925+0000 7f3e29ffb640 1 -- 172.21.15.67:0/262504080 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3e2c0cc360 msgr2=0x7f3e2c066e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:07.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.925+0000 7f3e29ffb640 1 --2- 172.21.15.67:0/262504080 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3e2c0cc360 0x7f3e2c066e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.925+0000 7f3e29ffb640 1 -- 172.21.15.67:0/262504080 >> v1:172.21.15.72:6789/0 conn(0x7f3e2c0cd410 legacy=0x7f3e2c0b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:07.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.925+0000 7f3e29ffb640 1 -- 172.21.15.67:0/262504080 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e2c06bf30 con 0x7f3e2c06ba50 2024-03-24T00:43:07.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.925+0000 7f3e29ffb640 1 -- 172.21.15.67:0/262504080 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f3e14001670 con 0x7f3e2c06ba50 2024-03-24T00:43:07.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.925+0000 7f3e29ffb640 1 -- 172.21.15.67:0/262504080 >> v1:172.21.15.67:6789/0 conn(0x7f3e2c06ba50 legacy=0x7f3e2c0ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:07.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.925+0000 7f3e29ffb640 1 --2- 172.21.15.67:0/262504080 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3e2c0cc360 0x7f3e180039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:07.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.925+0000 7f3e29ffb640 1 --2- 172.21.15.67:0/262504080 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e18003f20 0x7f3e18004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:07.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.925+0000 7f3e29ffb640 1 --2- 172.21.15.67:0/262504080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e18004d40 0x7f3e180051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:07.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.925+0000 7f3e29ffb640 1 -- 172.21.15.67:0/262504080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e1800b9f0 con 0x7f3e18004d40 2024-03-24T00:43:07.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.925+0000 7f3e29ffb640 1 -- 172.21.15.67:0/262504080 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e1800bba0 con 0x7f3e18003f20 2024-03-24T00:43:07.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.925+0000 7f3e29ffb640 1 -- 172.21.15.67:0/262504080 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e1800bda0 con 0x7f3e2c0cc360 2024-03-24T00:43:07.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.925+0000 7f3e2b7fe640 1 --2- 172.21.15.67:0/262504080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e18004d40 0x7f3e180051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:07.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.925+0000 7f3e2affd640 1 --2- 172.21.15.67:0/262504080 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3e2c0cc360 0x7f3e180039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:07.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.925+0000 7f3e2a7fc640 1 --2- 172.21.15.67:0/262504080 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e18003f20 0x7f3e18004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:07.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.925+0000 7f3e2b7fe640 1 -- 172.21.15.67:0/262504080 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3e2c0cc360 msgr2=0x7f3e180039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:07.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.925+0000 7f3e2b7fe640 1 --2- 172.21.15.67:0/262504080 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3e2c0cc360 0x7f3e180039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.925+0000 7f3e2b7fe640 1 -- 172.21.15.67:0/262504080 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e18003f20 msgr2=0x7f3e18004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:07.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.925+0000 7f3e2b7fe640 1 --2- 172.21.15.67:0/262504080 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e18003f20 0x7f3e18004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.925+0000 7f3e2b7fe640 1 -- 172.21.15.67:0/262504080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e18003520 con 0x7f3e18004d40 2024-03-24T00:43:07.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.926+0000 7f3e2b7fe640 1 --2- 172.21.15.67:0/262504080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e18004d40 0x7f3e180051c0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c072060 tx=0x7f3e18005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db2f6f7aa1c59be0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:07.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.926+0000 7f3e29ffb640 1 -- 172.21.15.67:0/262504080 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e24001430 con 0x7f3e18004d40 2024-03-24T00:43:07.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.926+0000 7f3e29ffb640 1 -- 172.21.15.67:0/262504080 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e24032620 con 0x7f3e18004d40 2024-03-24T00:43:07.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.926+0000 7f3e29ffb640 1 -- 172.21.15.67:0/262504080 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e24032a10 con 0x7f3e18004d40 2024-03-24T00:43:07.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.927+0000 7f3e31959640 1 -- 172.21.15.67:0/262504080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e18004d40 msgr2=0x7f3e180051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:07.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.927+0000 7f3e31959640 1 --2- 172.21.15.67:0/262504080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e18004d40 0x7f3e180051c0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c072060 tx=0x7f3e18005830 comp rx=0 tx=0).stop 2024-03-24T00:43:07.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.927+0000 7f3e2b7fe640 1 -- 172.21.15.67:0/262504080 reap_dead start 2024-03-24T00:43:07.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.927+0000 7f3e31959640 1 -- 172.21.15.67:0/262504080 shutdown_connections 2024-03-24T00:43:07.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.927+0000 7f3e31959640 1 -- 172.21.15.67:0/262504080 >> 172.21.15.67:0/262504080 conn(0x7f3e2c0b5440 msgr2=0x7f3e2c0b33d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:07.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.927+0000 7f3e31959640 1 -- 172.21.15.67:0/262504080 shutdown_connections 2024-03-24T00:43:07.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.927+0000 7f3e31959640 1 -- 172.21.15.67:0/262504080 wait complete. 2024-03-24T00:43:07.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.928+0000 7f3e31959640 1 Processor -- start 2024-03-24T00:43:07.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.928+0000 7f3e31959640 1 -- start start 2024-03-24T00:43:07.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.929+0000 7f3e31959640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3e2c158c30 0x7f3e2c0c9e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:07.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.929+0000 7f3e31959640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e2c159010 0x7f3e2c0ca920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:07.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.929+0000 7f3e31959640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e2c1594c0 0x7f3e2c0cae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:07.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.929+0000 7f3e31959640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2c0cc740 con 0x7f3e2c1594c0 2024-03-24T00:43:07.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.929+0000 7f3e31959640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2c0b3f60 con 0x7f3e2c159010 2024-03-24T00:43:07.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.929+0000 7f3e31959640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2c0cc8c0 con 0x7f3e2c158c30 2024-03-24T00:43:07.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.929+0000 7f3e2b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e2c1594c0 0x7f3e2c0cae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:07.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.929+0000 7f3e2b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e2c1594c0 0x7f3e2c0cae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47234/0 (socket says 172.21.15.67:47234) 2024-03-24T00:43:07.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.929+0000 7f3e2b7fe640 1 -- 172.21.15.67:0/2411713455 learned_addr learned my addr 172.21.15.67:0/2411713455 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:07.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.929+0000 7f3e2affd640 1 --2- 172.21.15.67:0/2411713455 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3e2c158c30 0x7f3e2c0c9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:07.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.929+0000 7f3e2a7fc640 1 --2- 172.21.15.67:0/2411713455 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e2c159010 0x7f3e2c0ca920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:07.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.929+0000 7f3e2b7fe640 1 -- 172.21.15.67:0/2411713455 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3e2c158c30 msgr2=0x7f3e2c0c9e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:07.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.929+0000 7f3e2b7fe640 1 --2- 172.21.15.67:0/2411713455 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3e2c158c30 0x7f3e2c0c9e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.929+0000 7f3e2b7fe640 1 -- 172.21.15.67:0/2411713455 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e2c159010 msgr2=0x7f3e2c0ca920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:07.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.929+0000 7f3e2b7fe640 1 --2- 172.21.15.67:0/2411713455 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e2c159010 0x7f3e2c0ca920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:07.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.929+0000 7f3e2b7fe640 1 -- 172.21.15.67:0/2411713455 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e2c1f2c90 con 0x7f3e2c1594c0 2024-03-24T00:43:07.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.929+0000 7f3e2b7fe640 1 --2- 172.21.15.67:0/2411713455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e2c1594c0 0x7f3e2c0cae60 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f3e2402bd90 tx=0x7f3e24001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:07.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.929+0000 7f3e23fff640 1 -- 172.21.15.67:0/2411713455 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e24001430 con 0x7f3e2c1594c0 2024-03-24T00:43:07.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.929+0000 7f3e23fff640 1 -- 172.21.15.67:0/2411713455 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e24043070 con 0x7f3e2c1594c0 2024-03-24T00:43:07.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.929+0000 7f3e23fff640 1 -- 172.21.15.67:0/2411713455 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e24003830 con 0x7f3e2c1594c0 2024-03-24T00:43:07.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.929+0000 7f3e31959640 1 -- 172.21.15.67:0/2411713455 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e2c1f2f70 con 0x7f3e2c1594c0 2024-03-24T00:43:07.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.929+0000 7f3e31959640 1 -- 172.21.15.67:0/2411713455 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e2c1f3530 con 0x7f3e2c1594c0 2024-03-24T00:43:07.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.930+0000 7f3e31959640 1 -- 172.21.15.67:0/2411713455 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e2c1f3730 con 0x7f3e2c1594c0 2024-03-24T00:43:07.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.931+0000 7f3e23fff640 1 -- 172.21.15.67:0/2411713455 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3e240039d0 con 0x7f3e2c1594c0 2024-03-24T00:43:07.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.931+0000 7f3e23fff640 1 --2- 172.21.15.67:0/2411713455 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3e0c06bbd0 0x7f3e0c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:07.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.931+0000 7f3e23fff640 1 -- 172.21.15.67:0/2411713455 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f3e240abe40 con 0x7f3e2c1594c0 2024-03-24T00:43:07.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.931+0000 7f3e2affd640 1 --2- 172.21.15.67:0/2411713455 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3e0c06bbd0 0x7f3e0c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:07.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.932+0000 7f3e2affd640 1 --2- 172.21.15.67:0/2411713455 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3e0c06bbd0 0x7f3e0c06e090 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c0704a0 tx=0x7f3e1c00b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:07.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:07.936+0000 7f3e23fff640 1 -- 172.21.15.67:0/2411713455 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3e2c1f3730 con 0x7f3e2c1594c0 2024-03-24T00:43:08.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.119+0000 7f3e31959640 1 -- 172.21.15.67:0/2411713455 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f3e2c04f400 con 0x7f3e2c1594c0 2024-03-24T00:43:08.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.120+0000 7f3e23fff640 1 -- 172.21.15.67:0/2411713455 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v72) v1 ==== 84+0+65 (secure 0 0 0) 0x7f3e2407a4a0 con 0x7f3e2c1594c0 2024-03-24T00:43:08.121 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:43:08.121 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-03-24T00:43:08.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:43:08.121 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2024-03-24T00:43:08.121 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2024-03-24T00:43:08.121 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:43:08.121 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-03-24T00:43:08.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.122+0000 7f3e31959640 1 -- 172.21.15.67:0/2411713455 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3e0c06bbd0 msgr2=0x7f3e0c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:08.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.122+0000 7f3e31959640 1 --2- 172.21.15.67:0/2411713455 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3e0c06bbd0 0x7f3e0c06e090 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c0704a0 tx=0x7f3e1c00b040 comp rx=0 tx=0).stop 2024-03-24T00:43:08.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.122+0000 7f3e31959640 1 -- 172.21.15.67:0/2411713455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e2c1594c0 msgr2=0x7f3e2c0cae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:08.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.122+0000 7f3e31959640 1 --2- 172.21.15.67:0/2411713455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e2c1594c0 0x7f3e2c0cae60 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f3e2402bd90 tx=0x7f3e24001230 comp rx=0 tx=0).stop 2024-03-24T00:43:08.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.122+0000 7f3e31959640 1 -- 172.21.15.67:0/2411713455 shutdown_connections 2024-03-24T00:43:08.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.122+0000 7f3e31959640 1 --2- 172.21.15.67:0/2411713455 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3e0c06bbd0 0x7f3e0c06e090 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:08.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.122+0000 7f3e31959640 1 --2- 172.21.15.67:0/2411713455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e2c1594c0 0x7f3e2c0cae60 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:08.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.122+0000 7f3e31959640 1 --2- 172.21.15.67:0/2411713455 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e2c159010 0x7f3e2c0ca920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:08.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.122+0000 7f3e31959640 1 --2- 172.21.15.67:0/2411713455 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3e2c158c30 0x7f3e2c0c9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:08.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.122+0000 7f3e31959640 1 -- 172.21.15.67:0/2411713455 >> 172.21.15.67:0/2411713455 conn(0x7f3e2c0b5440 msgr2=0x7f3e2c0b1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:08.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.122+0000 7f3e31959640 1 -- 172.21.15.67:0/2411713455 shutdown_connections 2024-03-24T00:43:08.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.122+0000 7f3e31959640 1 -- 172.21.15.67:0/2411713455 wait complete. 2024-03-24T00:43:08.134 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-24T00:43:08.134 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-24T00:43:08.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.238+0000 7fe35eb1d640 1 Processor -- start 2024-03-24T00:43:08.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.238+0000 7fe35eb1d640 1 -- start start 2024-03-24T00:43:08.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.239+0000 7fe35eb1d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe35806ec40 0x7fe35806f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:08.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.239+0000 7fe35eb1d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe35815b500 0x7fe35815b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:08.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.239+0000 7fe35eb1d640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe35806f560 con 0x7fe35815e770 2024-03-24T00:43:08.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.239+0000 7fe35eb1d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe358071ad0 con 0x7fe35806ec40 2024-03-24T00:43:08.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.239+0000 7fe35eb1d640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe358071d90 con 0x7fe35815b500 2024-03-24T00:43:08.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.239+0000 7fe35c8b9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe35806ec40 0x7fe35806f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:08.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.239+0000 7fe35c8b9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe35806ec40 0x7fe35806f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47242/0 (socket says 172.21.15.67:47242) 2024-03-24T00:43:08.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.239+0000 7fe35c8b9640 1 -- 172.21.15.67:0/325140887 learned_addr learned my addr 172.21.15.67:0/325140887 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:08.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.239+0000 7fe357fff640 1 --2- 172.21.15.67:0/325140887 >> v2:172.21.15.72:3300/0 conn(0x7fe35815b500 0x7fe35815b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:08.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.239+0000 7fe35c8b9640 1 -- 172.21.15.67:0/325140887 >> v2:172.21.15.72:3300/0 conn(0x7fe35815b500 msgr2=0x7fe35815b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:08.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.239+0000 7fe35c8b9640 1 --2- 172.21.15.67:0/325140887 >> v2:172.21.15.72:3300/0 conn(0x7fe35815b500 0x7fe35815b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:08.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.239+0000 7fe35c8b9640 1 -- 172.21.15.67:0/325140887 >> v1:172.21.15.72:6789/0 conn(0x7fe35815e770 legacy=0x7fe35815eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:08.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.239+0000 7fe35c8b9640 1 -- 172.21.15.67:0/325140887 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe358072070 con 0x7fe35806ec40 2024-03-24T00:43:08.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.239+0000 7fe357fff640 1 --2- 172.21.15.67:0/325140887 >> v2:172.21.15.72:3300/0 conn(0x7fe35815b500 0x7fe35815b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:43:08.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.240+0000 7fe35c8b9640 1 --2- 172.21.15.67:0/325140887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe35806ec40 0x7fe35806f020 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fe35806fc50 tx=0x7fe34c02e170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8866813fe1c773f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:08.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.240+0000 7fe3577fe640 1 -- 172.21.15.67:0/325140887 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34c03b070 con 0x7fe35806ec40 2024-03-24T00:43:08.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.240+0000 7fe3577fe640 1 -- 172.21.15.67:0/325140887 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe34c036440 con 0x7fe35806ec40 2024-03-24T00:43:08.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.240+0000 7fe3577fe640 1 -- 172.21.15.67:0/325140887 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34c036810 con 0x7fe35806ec40 2024-03-24T00:43:08.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.241+0000 7fe35eb1d640 1 -- 172.21.15.67:0/325140887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe35806ec40 msgr2=0x7fe35806f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:08.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.241+0000 7fe35eb1d640 1 --2- 172.21.15.67:0/325140887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe35806ec40 0x7fe35806f020 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fe35806fc50 tx=0x7fe34c02e170 comp rx=0 tx=0).stop 2024-03-24T00:43:08.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.241+0000 7fe35eb1d640 1 -- 172.21.15.67:0/325140887 shutdown_connections 2024-03-24T00:43:08.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.241+0000 7fe35eb1d640 1 --2- 172.21.15.67:0/325140887 >> v2:172.21.15.72:3300/0 conn(0x7fe35815b500 0x7fe35815b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:08.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.241+0000 7fe35eb1d640 1 --2- 172.21.15.67:0/325140887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe35806ec40 0x7fe35806f020 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:08.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.241+0000 7fe35eb1d640 1 -- 172.21.15.67:0/325140887 >> 172.21.15.67:0/325140887 conn(0x7fe3580cf9e0 msgr2=0x7fe3580cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:08.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.241+0000 7fe35eb1d640 1 -- 172.21.15.67:0/325140887 shutdown_connections 2024-03-24T00:43:08.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.241+0000 7fe35eb1d640 1 -- 172.21.15.67:0/325140887 wait complete. 2024-03-24T00:43:08.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.242+0000 7fe35eb1d640 1 Processor -- start 2024-03-24T00:43:08.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.242+0000 7fe35eb1d640 1 -- start start 2024-03-24T00:43:08.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.243+0000 7fe35eb1d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe35806ec40 0x7fe35815eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:08.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.243+0000 7fe35eb1d640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe35815b500 0x7fe35815f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:08.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.243+0000 7fe35eb1d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe35815e770 0x7fe358152830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:08.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.243+0000 7fe35eb1d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35815bed0 con 0x7fe35815e770 2024-03-24T00:43:08.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.243+0000 7fe35eb1d640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe358072630 con 0x7fe35806ec40 2024-03-24T00:43:08.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.243+0000 7fe35eb1d640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe35815c050 con 0x7fe35815b500 2024-03-24T00:43:08.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.243+0000 7fe35d0ba640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe35815e770 0x7fe358152830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:08.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.243+0000 7fe35d0ba640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe35815e770 0x7fe358152830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47244/0 (socket says 172.21.15.67:47244) 2024-03-24T00:43:08.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.243+0000 7fe35d0ba640 1 -- 172.21.15.67:0/4037196230 learned_addr learned my addr 172.21.15.67:0/4037196230 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:08.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.243+0000 7fe35c8b9640 1 --2- 172.21.15.67:0/4037196230 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe35806ec40 0x7fe35815eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:08.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.243+0000 7fe357fff640 1 --2- 172.21.15.67:0/4037196230 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe35815b500 0x7fe35815f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:08.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.243+0000 7fe35d0ba640 1 -- 172.21.15.67:0/4037196230 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe35815b500 msgr2=0x7fe35815f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:08.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.243+0000 7fe35d0ba640 1 --2- 172.21.15.67:0/4037196230 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe35815b500 0x7fe35815f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:08.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.243+0000 7fe35d0ba640 1 -- 172.21.15.67:0/4037196230 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe35806ec40 msgr2=0x7fe35815eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:08.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.243+0000 7fe35d0ba640 1 --2- 172.21.15.67:0/4037196230 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe35806ec40 0x7fe35815eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:08.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.243+0000 7fe35d0ba640 1 -- 172.21.15.67:0/4037196230 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe35814e310 con 0x7fe35815e770 2024-03-24T00:43:08.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.243+0000 7fe357fff640 1 --2- 172.21.15.67:0/4037196230 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe35815b500 0x7fe35815f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:43:08.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.243+0000 7fe35d0ba640 1 --2- 172.21.15.67:0/4037196230 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe35815e770 0x7fe358152830 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fe35000c6b0 tx=0x7fe35000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:08.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.243+0000 7fe3557fa640 1 -- 172.21.15.67:0/4037196230 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe350003b70 con 0x7fe35815e770 2024-03-24T00:43:08.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.243+0000 7fe3557fa640 1 -- 172.21.15.67:0/4037196230 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe350003d30 con 0x7fe35815e770 2024-03-24T00:43:08.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.243+0000 7fe35eb1d640 1 -- 172.21.15.67:0/4037196230 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe35814e5f0 con 0x7fe35815e770 2024-03-24T00:43:08.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.243+0000 7fe3557fa640 1 -- 172.21.15.67:0/4037196230 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe350005310 con 0x7fe35815e770 2024-03-24T00:43:08.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.243+0000 7fe35eb1d640 1 -- 172.21.15.67:0/4037196230 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe35814ebb0 con 0x7fe35815e770 2024-03-24T00:43:08.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.245+0000 7fe35eb1d640 1 -- 172.21.15.67:0/4037196230 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe35804f400 con 0x7fe35815e770 2024-03-24T00:43:08.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.245+0000 7fe3557fa640 1 -- 172.21.15.67:0/4037196230 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe3500054b0 con 0x7fe35815e770 2024-03-24T00:43:08.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.245+0000 7fe3557fa640 1 --2- 172.21.15.67:0/4037196230 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe33c06bbd0 0x7fe33c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:08.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.245+0000 7fe3557fa640 1 -- 172.21.15.67:0/4037196230 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fe35008a680 con 0x7fe35815e770 2024-03-24T00:43:08.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.245+0000 7fe35c8b9640 1 --2- 172.21.15.67:0/4037196230 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe33c06bbd0 0x7fe33c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:08.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.246+0000 7fe35c8b9640 1 --2- 172.21.15.67:0/4037196230 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe33c06bbd0 0x7fe33c06e090 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fe34c038000 tx=0x7fe34c032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:08.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.250+0000 7fe3557fa640 1 -- 172.21.15.67:0/4037196230 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe350059330 con 0x7fe35815e770 2024-03-24T00:43:08.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.432+0000 7fe35eb1d640 1 -- 172.21.15.67:0/4037196230 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fe3580cf7c0 con 0x7fe35815e770 2024-03-24T00:43:08.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.802+0000 7fe3557fa640 1 -- 172.21.15.67:0/4037196230 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v73) v1 ==== 138+0+0 (secure 0 0 0) 0x7fe350058c30 con 0x7fe35815e770 2024-03-24T00:43:08.803 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' created 2024-03-24T00:43:08.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.806+0000 7fe35eb1d640 1 -- 172.21.15.67:0/4037196230 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe33c06bbd0 msgr2=0x7fe33c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:08.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.806+0000 7fe35eb1d640 1 --2- 172.21.15.67:0/4037196230 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe33c06bbd0 0x7fe33c06e090 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fe34c038000 tx=0x7fe34c032000 comp rx=0 tx=0).stop 2024-03-24T00:43:08.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.806+0000 7fe35eb1d640 1 -- 172.21.15.67:0/4037196230 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe35815e770 msgr2=0x7fe358152830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:08.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.806+0000 7fe35eb1d640 1 --2- 172.21.15.67:0/4037196230 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe35815e770 0x7fe358152830 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fe35000c6b0 tx=0x7fe35000cb70 comp rx=0 tx=0).stop 2024-03-24T00:43:08.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.806+0000 7fe35eb1d640 1 -- 172.21.15.67:0/4037196230 shutdown_connections 2024-03-24T00:43:08.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.806+0000 7fe35eb1d640 1 --2- 172.21.15.67:0/4037196230 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe33c06bbd0 0x7fe33c06e090 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:08.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.806+0000 7fe35eb1d640 1 --2- 172.21.15.67:0/4037196230 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe35815e770 0x7fe358152830 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:08.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.806+0000 7fe35eb1d640 1 --2- 172.21.15.67:0/4037196230 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe35815b500 0x7fe35815f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:08.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.806+0000 7fe35eb1d640 1 --2- 172.21.15.67:0/4037196230 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe35806ec40 0x7fe35815eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:08.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.806+0000 7fe35eb1d640 1 -- 172.21.15.67:0/4037196230 >> 172.21.15.67:0/4037196230 conn(0x7fe3580cf9e0 msgr2=0x7fe3581507a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:08.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.807+0000 7fe35eb1d640 1 -- 172.21.15.67:0/4037196230 shutdown_connections 2024-03-24T00:43:08.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.807+0000 7fe35eb1d640 1 -- 172.21.15.67:0/4037196230 wait complete. 2024-03-24T00:43:08.821 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-24T00:43:08.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.929+0000 7f2ce13a9640 1 Processor -- start 2024-03-24T00:43:08.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.929+0000 7f2ce13a9640 1 -- start start 2024-03-24T00:43:08.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.930+0000 7f2ce13a9640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2cdc15e300 0x7f2cdc0b0790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:08.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.930+0000 7f2ce13a9640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2cdc0d21b0 0x7f2cdc15e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:08.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.930+0000 7f2ce13a9640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cdc0ce390 con 0x7f2cdc0b0cd0 2024-03-24T00:43:08.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.930+0000 7f2ce13a9640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cdc0ce630 con 0x7f2cdc0d21b0 2024-03-24T00:43:08.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.930+0000 7f2ce13a9640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cdc0ce910 con 0x7f2cdc15e300 2024-03-24T00:43:08.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.930+0000 7f2cda7fc640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2cdc0b0cd0 0x7f2cdc0d1aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44942/0 (socket says 172.21.15.67:44942) 2024-03-24T00:43:08.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.930+0000 7f2cda7fc640 1 -- 172.21.15.67:0/44398572 learned_addr learned my addr 172.21.15.67:0/44398572 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:43:08.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.930+0000 7f2cdb7fe640 1 --2- 172.21.15.67:0/44398572 >> v2:172.21.15.72:3300/0 conn(0x7f2cdc0d21b0 0x7f2cdc15e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:08.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.931+0000 7f2cdaffd640 1 --2- 172.21.15.67:0/44398572 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2cdc15e300 0x7f2cdc0b0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:08.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.931+0000 7f2cd9ffb640 1 -- 172.21.15.67:0/44398572 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3695791797 0 0) 0x7f2cdc0ce390 con 0x7f2cdc0b0cd0 2024-03-24T00:43:08.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.931+0000 7f2cd9ffb640 1 -- 172.21.15.67:0/44398572 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cc8003520 con 0x7f2cdc0b0cd0 2024-03-24T00:43:08.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.931+0000 7f2cd9ffb640 1 -- 172.21.15.67:0/44398572 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3256209240 0 0) 0x7f2cc8003520 con 0x7f2cdc0b0cd0 2024-03-24T00:43:08.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.931+0000 7f2cdb7fe640 1 -- 172.21.15.67:0/44398572 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2cdc15e300 msgr2=0x7f2cdc0b0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:08.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.931+0000 7f2cdb7fe640 1 --2- 172.21.15.67:0/44398572 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2cdc15e300 0x7f2cdc0b0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:08.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.931+0000 7f2cdb7fe640 1 -- 172.21.15.67:0/44398572 >> v1:172.21.15.67:6789/0 conn(0x7f2cdc0b0cd0 legacy=0x7f2cdc0d1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:08.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.931+0000 7f2cdb7fe640 1 -- 172.21.15.67:0/44398572 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cdc0cebf0 con 0x7f2cdc0d21b0 2024-03-24T00:43:08.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.931+0000 7f2cdaffd640 1 --2- 172.21.15.67:0/44398572 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2cdc15e300 0x7f2cdc0b0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:43:08.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.932+0000 7f2cdb7fe640 1 --2- 172.21.15.67:0/44398572 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2cdc0d21b0 0x7f2cdc15e8d0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f2cd40097a0 tx=0x7f2cd402b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54f5c3b452314d6c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:08.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.932+0000 7f2cd9ffb640 1 -- 172.21.15.67:0/44398572 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd4037070 con 0x7f2cdc0d21b0 2024-03-24T00:43:08.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.932+0000 7f2cd9ffb640 1 -- 172.21.15.67:0/44398572 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cd4030510 con 0x7f2cdc0d21b0 2024-03-24T00:43:08.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.932+0000 7f2cd9ffb640 1 -- 172.21.15.67:0/44398572 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc4001690 con 0x7f2cdc0d21b0 2024-03-24T00:43:08.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.932+0000 7f2ce13a9640 1 -- 172.21.15.67:0/44398572 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2cdc0d21b0 msgr2=0x7f2cdc15e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:08.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.932+0000 7f2ce13a9640 1 --2- 172.21.15.67:0/44398572 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2cdc0d21b0 0x7f2cdc15e8d0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f2cd40097a0 tx=0x7f2cd402b710 comp rx=0 tx=0).stop 2024-03-24T00:43:08.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.932+0000 7f2ce13a9640 1 -- 172.21.15.67:0/44398572 shutdown_connections 2024-03-24T00:43:08.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.932+0000 7f2ce13a9640 1 --2- 172.21.15.67:0/44398572 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2cdc0d21b0 0x7f2cdc15e8d0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:08.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.932+0000 7f2ce13a9640 1 --2- 172.21.15.67:0/44398572 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2cdc15e300 0x7f2cdc0b0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:08.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.932+0000 7f2ce13a9640 1 -- 172.21.15.67:0/44398572 >> 172.21.15.67:0/44398572 conn(0x7f2cdc09aef0 msgr2=0x7f2cdc08b1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:08.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.933+0000 7f2ce13a9640 1 -- 172.21.15.67:0/44398572 shutdown_connections 2024-03-24T00:43:08.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.933+0000 7f2ce13a9640 1 -- 172.21.15.67:0/44398572 wait complete. 2024-03-24T00:43:08.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.934+0000 7f2ce13a9640 1 Processor -- start 2024-03-24T00:43:08.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.934+0000 7f2ce13a9640 1 -- start start 2024-03-24T00:43:08.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.935+0000 7f2ce13a9640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2cdc0b0cd0 0x7f2cdc158660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:08.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.935+0000 7f2ce13a9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cdc0d21b0 0x7f2cdc0cd190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:08.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.935+0000 7f2ce13a9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2cdc15e300 0x7f2cdc0cd6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:08.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.935+0000 7f2ce13a9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cdc0b1c70 con 0x7f2cdc0d21b0 2024-03-24T00:43:08.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.935+0000 7f2ce13a9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cdc0b1af0 con 0x7f2cdc15e300 2024-03-24T00:43:08.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.935+0000 7f2ce13a9640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cdc15ee10 con 0x7f2cdc0b0cd0 2024-03-24T00:43:08.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.935+0000 7f2cda7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cdc0d21b0 0x7f2cdc0cd190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:08.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.935+0000 7f2cda7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cdc0d21b0 0x7f2cdc0cd190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47258/0 (socket says 172.21.15.67:47258) 2024-03-24T00:43:08.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.935+0000 7f2cda7fc640 1 -- 172.21.15.67:0/1026905625 learned_addr learned my addr 172.21.15.67:0/1026905625 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:08.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.935+0000 7f2cdb7fe640 1 --2- 172.21.15.67:0/1026905625 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2cdc15e300 0x7f2cdc0cd6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:08.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.935+0000 7f2cdaffd640 1 --2- 172.21.15.67:0/1026905625 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2cdc0b0cd0 0x7f2cdc158660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:08.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.935+0000 7f2cda7fc640 1 -- 172.21.15.67:0/1026905625 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2cdc0b0cd0 msgr2=0x7f2cdc158660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:08.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.935+0000 7f2cda7fc640 1 --2- 172.21.15.67:0/1026905625 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2cdc0b0cd0 0x7f2cdc158660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:08.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.935+0000 7f2cda7fc640 1 -- 172.21.15.67:0/1026905625 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2cdc15e300 msgr2=0x7f2cdc0cd6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:08.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.935+0000 7f2cda7fc640 1 --2- 172.21.15.67:0/1026905625 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2cdc15e300 0x7f2cdc0cd6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:08.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.935+0000 7f2cda7fc640 1 -- 172.21.15.67:0/1026905625 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cdc0c8d60 con 0x7f2cdc0d21b0 2024-03-24T00:43:08.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.935+0000 7f2cda7fc640 1 --2- 172.21.15.67:0/1026905625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cdc0d21b0 0x7f2cdc0cd190 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f2cc400c780 tx=0x7f2cc400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:08.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.935+0000 7f2cd37fe640 1 -- 172.21.15.67:0/1026905625 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc4018070 con 0x7f2cdc0d21b0 2024-03-24T00:43:08.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.935+0000 7f2cd37fe640 1 -- 172.21.15.67:0/1026905625 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cc40072f0 con 0x7f2cdc0d21b0 2024-03-24T00:43:08.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.935+0000 7f2cd37fe640 1 -- 172.21.15.67:0/1026905625 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc40047e0 con 0x7f2cdc0d21b0 2024-03-24T00:43:08.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.935+0000 7f2ce13a9640 1 -- 172.21.15.67:0/1026905625 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cdc0c9040 con 0x7f2cdc0d21b0 2024-03-24T00:43:08.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.935+0000 7f2ce13a9640 1 -- 172.21.15.67:0/1026905625 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cdc158ba0 con 0x7f2cdc0d21b0 2024-03-24T00:43:08.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.936+0000 7f2ce13a9640 1 -- 172.21.15.67:0/1026905625 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cdc04f400 con 0x7f2cdc0d21b0 2024-03-24T00:43:08.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.937+0000 7f2cd37fe640 1 -- 172.21.15.67:0/1026905625 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2cc401b070 con 0x7f2cdc0d21b0 2024-03-24T00:43:08.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.937+0000 7f2cd37fe640 1 --2- 172.21.15.67:0/1026905625 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2cbc06bc00 0x7f2cbc06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:08.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.937+0000 7f2cd37fe640 1 -- 172.21.15.67:0/1026905625 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f2cc4089e90 con 0x7f2cdc0d21b0 2024-03-24T00:43:08.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.938+0000 7f2cdaffd640 1 --2- 172.21.15.67:0/1026905625 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2cbc06bc00 0x7f2cbc06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:08.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.938+0000 7f2cdaffd640 1 --2- 172.21.15.67:0/1026905625 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2cbc06bc00 0x7f2cbc06e0c0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc009fd0 tx=0x7f2ccc00b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:08.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:08.942+0000 7f2cd37fe640 1 -- 172.21.15.67:0/1026905625 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2cc40589d0 con 0x7f2cdc0d21b0 2024-03-24T00:43:09.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.126+0000 7f2ce13a9640 1 -- 172.21.15.67:0/1026905625 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f2cdc0b1600 con 0x7f2cdc0d21b0 2024-03-24T00:43:09.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.807+0000 7f2cd37fe640 1 -- 172.21.15.67:0/1026905625 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v74) v1 ==== 170+0+0 (secure 0 0 0) 0x7f2cc40582d0 con 0x7f2cdc0d21b0 2024-03-24T00:43:09.808 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' created 2024-03-24T00:43:09.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.810+0000 7f2ce13a9640 1 -- 172.21.15.67:0/1026905625 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2cbc06bc00 msgr2=0x7f2cbc06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:09.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.810+0000 7f2ce13a9640 1 --2- 172.21.15.67:0/1026905625 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2cbc06bc00 0x7f2cbc06e0c0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc009fd0 tx=0x7f2ccc00b040 comp rx=0 tx=0).stop 2024-03-24T00:43:09.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.810+0000 7f2ce13a9640 1 -- 172.21.15.67:0/1026905625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cdc0d21b0 msgr2=0x7f2cdc0cd190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:09.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.810+0000 7f2ce13a9640 1 --2- 172.21.15.67:0/1026905625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cdc0d21b0 0x7f2cdc0cd190 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f2cc400c780 tx=0x7f2cc400cc40 comp rx=0 tx=0).stop 2024-03-24T00:43:09.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.810+0000 7f2ce13a9640 1 -- 172.21.15.67:0/1026905625 shutdown_connections 2024-03-24T00:43:09.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.810+0000 7f2ce13a9640 1 --2- 172.21.15.67:0/1026905625 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2cbc06bc00 0x7f2cbc06e0c0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.810+0000 7f2ce13a9640 1 --2- 172.21.15.67:0/1026905625 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2cdc15e300 0x7f2cdc0cd6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.810+0000 7f2ce13a9640 1 --2- 172.21.15.67:0/1026905625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cdc0d21b0 0x7f2cdc0cd190 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.810+0000 7f2ce13a9640 1 --2- 172.21.15.67:0/1026905625 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2cdc0b0cd0 0x7f2cdc158660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.810+0000 7f2ce13a9640 1 -- 172.21.15.67:0/1026905625 >> 172.21.15.67:0/1026905625 conn(0x7f2cdc09aef0 msgr2=0x7f2cdc08b060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:09.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.810+0000 7f2ce13a9640 1 -- 172.21.15.67:0/1026905625 shutdown_connections 2024-03-24T00:43:09.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.811+0000 7f2ce13a9640 1 -- 172.21.15.67:0/1026905625 wait complete. 2024-03-24T00:43:09.835 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-24T00:43:09.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.939+0000 7f90c1212640 1 Processor -- start 2024-03-24T00:43:09.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.940+0000 7f90c1212640 1 -- start start 2024-03-24T00:43:09.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.940+0000 7f90c1212640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f90bc0d3ae0 0x7f90bc0b5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:09.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.940+0000 7f90c1212640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f90bc15e280 0x7f90bc0b5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:09.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.940+0000 7f90c1212640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90bc0d2250 con 0x7f90bc15dea0 2024-03-24T00:43:09.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.940+0000 7f90c1212640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90bc0d2530 con 0x7f90bc15e280 2024-03-24T00:43:09.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.940+0000 7f90c1212640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90bc0d2810 con 0x7f90bc0d3ae0 2024-03-24T00:43:09.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.940+0000 7f90bad9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f90bc15dea0 0x7f90bc0d33d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44948/0 (socket says 172.21.15.67:44948) 2024-03-24T00:43:09.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.940+0000 7f90bad9d640 1 -- 172.21.15.67:0/2822493648 learned_addr learned my addr 172.21.15.67:0/2822493648 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:43:09.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.941+0000 7f90bb59e640 1 --2- 172.21.15.67:0/2822493648 >> v2:172.21.15.72:3300/0 conn(0x7f90bc15e280 0x7f90bc0b5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:09.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.941+0000 7f90b3fff640 1 --2- 172.21.15.67:0/2822493648 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f90bc0d3ae0 0x7f90bc0b5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:09.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.941+0000 7f90ba59c640 1 -- 172.21.15.67:0/2822493648 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3108066396 0 0) 0x7f90bc0d2250 con 0x7f90bc15dea0 2024-03-24T00:43:09.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.941+0000 7f90ba59c640 1 -- 172.21.15.67:0/2822493648 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90a8003520 con 0x7f90bc15dea0 2024-03-24T00:43:09.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.941+0000 7f90ba59c640 1 -- 172.21.15.67:0/2822493648 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 468915356 0 0) 0x7f90a8003520 con 0x7f90bc15dea0 2024-03-24T00:43:09.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.941+0000 7f90ba59c640 1 -- 172.21.15.67:0/2822493648 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f90bc0d3ae0 msgr2=0x7f90bc0b5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:09.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.941+0000 7f90ba59c640 1 --2- 172.21.15.67:0/2822493648 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f90bc0d3ae0 0x7f90bc0b5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.941+0000 7f90ba59c640 1 -- 172.21.15.67:0/2822493648 >> v2:172.21.15.72:3300/0 conn(0x7f90bc15e280 msgr2=0x7f90bc0b5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:09.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.941+0000 7f90ba59c640 1 --2- 172.21.15.67:0/2822493648 >> v2:172.21.15.72:3300/0 conn(0x7f90bc15e280 0x7f90bc0b5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.941+0000 7f90ba59c640 1 -- 172.21.15.67:0/2822493648 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90bc0d2af0 con 0x7f90bc15dea0 2024-03-24T00:43:09.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.941+0000 7f90ba59c640 1 -- 172.21.15.67:0/2822493648 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f90ac001670 con 0x7f90bc15dea0 2024-03-24T00:43:09.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.941+0000 7f90bb59e640 1 --2- 172.21.15.67:0/2822493648 >> v2:172.21.15.72:3300/0 conn(0x7f90bc15e280 0x7f90bc0b5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:43:09.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.941+0000 7f90ba59c640 1 -- 172.21.15.67:0/2822493648 >> v1:172.21.15.67:6789/0 conn(0x7f90bc15dea0 legacy=0x7f90bc0d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:09.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.941+0000 7f90ba59c640 1 --2- 172.21.15.67:0/2822493648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90a80039b0 0x7f90a8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:09.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.941+0000 7f90ba59c640 1 --2- 172.21.15.67:0/2822493648 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90a8004800 0x7f90a8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:09.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.941+0000 7f90ba59c640 1 --2- 172.21.15.67:0/2822493648 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f90bc0d3ae0 0x7f90a80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:09.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.941+0000 7f90ba59c640 1 -- 172.21.15.67:0/2822493648 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90bc0d2530 con 0x7f90a80039b0 2024-03-24T00:43:09.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.941+0000 7f90ba59c640 1 -- 172.21.15.67:0/2822493648 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90bc0d2810 con 0x7f90a8004800 2024-03-24T00:43:09.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.941+0000 7f90ba59c640 1 -- 172.21.15.67:0/2822493648 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90a800b9c0 con 0x7f90bc0d3ae0 2024-03-24T00:43:09.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.941+0000 7f90bad9d640 1 --2- 172.21.15.67:0/2822493648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90a80039b0 0x7f90a8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:09.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.941+0000 7f90b3fff640 1 --2- 172.21.15.67:0/2822493648 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90a8004800 0x7f90a8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:09.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.941+0000 7f90bb59e640 1 --2- 172.21.15.67:0/2822493648 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f90bc0d3ae0 0x7f90a80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:09.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.941+0000 7f90bad9d640 1 -- 172.21.15.67:0/2822493648 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f90bc0d3ae0 msgr2=0x7f90a80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:09.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.942+0000 7f90bad9d640 1 --2- 172.21.15.67:0/2822493648 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f90bc0d3ae0 0x7f90a80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.942+0000 7f90bad9d640 1 -- 172.21.15.67:0/2822493648 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90a8004800 msgr2=0x7f90a8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:09.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.942+0000 7f90bad9d640 1 --2- 172.21.15.67:0/2822493648 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90a8004800 0x7f90a8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.942+0000 7f90bad9d640 1 -- 172.21.15.67:0/2822493648 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a8003520 con 0x7f90a80039b0 2024-03-24T00:43:09.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.942+0000 7f90bad9d640 1 --2- 172.21.15.67:0/2822493648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90a80039b0 0x7f90a8003dd0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f90ac00c270 tx=0x7f90a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=160b6ff8b99b213e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:09.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.942+0000 7f90ba59c640 1 -- 172.21.15.67:0/2822493648 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ac007d30 con 0x7f90a80039b0 2024-03-24T00:43:09.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.942+0000 7f90ba59c640 1 -- 172.21.15.67:0/2822493648 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90ac030070 con 0x7f90a80039b0 2024-03-24T00:43:09.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.942+0000 7f90ba59c640 1 -- 172.21.15.67:0/2822493648 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ac02cd80 con 0x7f90a80039b0 2024-03-24T00:43:09.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.943+0000 7f90c1212640 1 -- 172.21.15.67:0/2822493648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90a80039b0 msgr2=0x7f90a8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:09.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.943+0000 7f90c1212640 1 --2- 172.21.15.67:0/2822493648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90a80039b0 0x7f90a8003dd0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f90ac00c270 tx=0x7f90a8005830 comp rx=0 tx=0).stop 2024-03-24T00:43:09.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.943+0000 7f90bb59e640 1 -- 172.21.15.67:0/2822493648 reap_dead start 2024-03-24T00:43:09.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.943+0000 7f90c1212640 1 -- 172.21.15.67:0/2822493648 shutdown_connections 2024-03-24T00:43:09.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.943+0000 7f90c1212640 1 -- 172.21.15.67:0/2822493648 >> 172.21.15.67:0/2822493648 conn(0x7f90bc072da0 msgr2=0x7f90bc0d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:09.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.943+0000 7f90c1212640 1 -- 172.21.15.67:0/2822493648 shutdown_connections 2024-03-24T00:43:09.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.943+0000 7f90c1212640 1 -- 172.21.15.67:0/2822493648 wait complete. 2024-03-24T00:43:09.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.944+0000 7f90c1212640 1 Processor -- start 2024-03-24T00:43:09.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.944+0000 7f90c1212640 1 -- start start 2024-03-24T00:43:09.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.945+0000 7f90c1212640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90a80039b0 0x7f90bc1e8de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:09.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.945+0000 7f90c1212640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f90bc1e9320 0x7f90bc14ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:09.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.945+0000 7f90c1212640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90bc14f0f0 0x7f90bc14b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:09.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.945+0000 7f90c1212640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90bc0b70f0 con 0x7f90a80039b0 2024-03-24T00:43:09.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.945+0000 7f90c1212640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90bc0b5cb0 con 0x7f90bc14f0f0 2024-03-24T00:43:09.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.945+0000 7f90c1212640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90bc0b7270 con 0x7f90bc1e9320 2024-03-24T00:43:09.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.945+0000 7f90bad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90a80039b0 0x7f90bc1e8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:09.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.945+0000 7f90bad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90a80039b0 0x7f90bc1e8de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47280/0 (socket says 172.21.15.67:47280) 2024-03-24T00:43:09.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.945+0000 7f90bad9d640 1 -- 172.21.15.67:0/50806751 learned_addr learned my addr 172.21.15.67:0/50806751 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:09.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.945+0000 7f90bb59e640 1 --2- 172.21.15.67:0/50806751 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90bc14f0f0 0x7f90bc14b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:09.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.945+0000 7f90b3fff640 1 --2- 172.21.15.67:0/50806751 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f90bc1e9320 0x7f90bc14ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:09.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.945+0000 7f90bad9d640 1 -- 172.21.15.67:0/50806751 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f90bc1e9320 msgr2=0x7f90bc14ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:09.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.945+0000 7f90bad9d640 1 --2- 172.21.15.67:0/50806751 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f90bc1e9320 0x7f90bc14ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.945+0000 7f90bad9d640 1 -- 172.21.15.67:0/50806751 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90bc14f0f0 msgr2=0x7f90bc14b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:09.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.945+0000 7f90bad9d640 1 --2- 172.21.15.67:0/50806751 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90bc14f0f0 0x7f90bc14b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:09.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.945+0000 7f90bad9d640 1 -- 172.21.15.67:0/50806751 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90bc0b2940 con 0x7f90a80039b0 2024-03-24T00:43:09.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.946+0000 7f90bad9d640 1 --2- 172.21.15.67:0/50806751 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90a80039b0 0x7f90bc1e8de0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f90ac00c390 tx=0x7f90bc0d2cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:09.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.946+0000 7f90b37fe640 1 -- 172.21.15.67:0/50806751 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ac045070 con 0x7f90a80039b0 2024-03-24T00:43:09.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.946+0000 7f90b37fe640 1 -- 172.21.15.67:0/50806751 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90ac004bf0 con 0x7f90a80039b0 2024-03-24T00:43:09.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.946+0000 7f90c1212640 1 -- 172.21.15.67:0/50806751 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90bc0cdac0 con 0x7f90a80039b0 2024-03-24T00:43:09.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.946+0000 7f90b37fe640 1 -- 172.21.15.67:0/50806751 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ac001c70 con 0x7f90a80039b0 2024-03-24T00:43:09.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.946+0000 7f90c1212640 1 -- 172.21.15.67:0/50806751 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90bc149b00 con 0x7f90a80039b0 2024-03-24T00:43:09.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.947+0000 7f90c1212640 1 -- 172.21.15.67:0/50806751 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90bc04f400 con 0x7f90a80039b0 2024-03-24T00:43:09.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.947+0000 7f90b37fe640 1 -- 172.21.15.67:0/50806751 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f90ac007070 con 0x7f90a80039b0 2024-03-24T00:43:09.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.948+0000 7f90b37fe640 1 --2- 172.21.15.67:0/50806751 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f909c06bc00 0x7f909c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:09.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.948+0000 7f90b37fe640 1 -- 172.21.15.67:0/50806751 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f90ac0abb30 con 0x7f90a80039b0 2024-03-24T00:43:09.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.948+0000 7f90b3fff640 1 --2- 172.21.15.67:0/50806751 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f909c06bc00 0x7f909c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:09.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.948+0000 7f90b3fff640 1 --2- 172.21.15.67:0/50806751 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f909c06bc00 0x7f909c06e0c0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f90bc0b8690 tx=0x7f90a4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:09.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:09.952+0000 7f90b37fe640 1 -- 172.21.15.67:0/50806751 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f90ac07a660 con 0x7f90a80039b0 2024-03-24T00:43:10.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.175+0000 7f90c1212640 1 -- 172.21.15.67:0/50806751 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f90bc0cea00 con 0x7f90a80039b0 2024-03-24T00:43:10.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.809+0000 7f90b37fe640 1 -- 172.21.15.67:0/50806751 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 11 and data pool 12 v38) v1 ==== 174+0+0 (secure 0 0 0) 0x7f90ac079f60 con 0x7f90a80039b0 2024-03-24T00:43:10.810 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 11 and data pool 12 2024-03-24T00:43:10.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.816+0000 7f90c1212640 1 -- 172.21.15.67:0/50806751 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f909c06bc00 msgr2=0x7f909c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:10.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.816+0000 7f90c1212640 1 --2- 172.21.15.67:0/50806751 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f909c06bc00 0x7f909c06e0c0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f90bc0b8690 tx=0x7f90a4008040 comp rx=0 tx=0).stop 2024-03-24T00:43:10.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.816+0000 7f90c1212640 1 -- 172.21.15.67:0/50806751 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90a80039b0 msgr2=0x7f90bc1e8de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:10.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.816+0000 7f90c1212640 1 --2- 172.21.15.67:0/50806751 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90a80039b0 0x7f90bc1e8de0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f90ac00c390 tx=0x7f90bc0d2cc0 comp rx=0 tx=0).stop 2024-03-24T00:43:10.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.817+0000 7f90c1212640 1 -- 172.21.15.67:0/50806751 shutdown_connections 2024-03-24T00:43:10.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.817+0000 7f90c1212640 1 --2- 172.21.15.67:0/50806751 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f909c06bc00 0x7f909c06e0c0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:10.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.817+0000 7f90c1212640 1 --2- 172.21.15.67:0/50806751 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90bc14f0f0 0x7f90bc14b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:10.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.817+0000 7f90c1212640 1 --2- 172.21.15.67:0/50806751 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f90bc1e9320 0x7f90bc14ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:10.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.817+0000 7f90c1212640 1 --2- 172.21.15.67:0/50806751 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90a80039b0 0x7f90bc1e8de0 secure :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f90ac00c390 tx=0x7f90bc0d2cc0 comp rx=0 tx=0).stop 2024-03-24T00:43:10.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.817+0000 7f90c1212640 1 -- 172.21.15.67:0/50806751 >> 172.21.15.67:0/50806751 conn(0x7f90bc072da0 msgr2=0x7f90bc0b8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:10.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.819+0000 7f90c1212640 1 -- 172.21.15.67:0/50806751 shutdown_connections 2024-03-24T00:43:10.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.819+0000 7f90c1212640 1 -- 172.21.15.67:0/50806751 wait complete. 2024-03-24T00:43:10.848 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:43:10.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.953+0000 7fd9f9377640 1 Processor -- start 2024-03-24T00:43:10.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.954+0000 7fd9f9377640 1 -- start start 2024-03-24T00:43:10.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.954+0000 7fd9f9377640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd9f415b430 0x7fd9f4072360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:10.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.954+0000 7fd9f9377640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9f40b4da0 con 0x7fd9f40b49c0 2024-03-24T00:43:10.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.954+0000 7fd9f9377640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9f406db30 con 0x7fd9f40728a0 2024-03-24T00:43:10.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.954+0000 7fd9f9377640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9f4072d10 con 0x7fd9f415b430 2024-03-24T00:43:10.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.955+0000 7fd9f27fc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd9f415b430 0x7fd9f4072360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:10.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.955+0000 7fd9f2ffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd9f40b49c0 0x7fd9f406f100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44962/0 (socket says 172.21.15.67:44962) 2024-03-24T00:43:10.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.955+0000 7fd9f2ffd640 1 -- 172.21.15.67:0/2059457331 learned_addr learned my addr 172.21.15.67:0/2059457331 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:43:10.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.955+0000 7fd9f27fc640 1 -- 172.21.15.67:0/2059457331 >> v1:172.21.15.72:6789/0 conn(0x7fd9f40728a0 legacy=0x7fd9f40b2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:10.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.955+0000 7fd9f27fc640 1 -- 172.21.15.67:0/2059457331 >> v1:172.21.15.67:6789/0 conn(0x7fd9f40b49c0 legacy=0x7fd9f406f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:10.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.955+0000 7fd9f1ffb640 1 -- 172.21.15.67:0/2059457331 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3120793511 0 0) 0x7fd9f40b4da0 con 0x7fd9f40b49c0 2024-03-24T00:43:10.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.955+0000 7fd9f27fc640 1 -- 172.21.15.67:0/2059457331 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9f406de10 con 0x7fd9f415b430 2024-03-24T00:43:10.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.956+0000 7fd9f27fc640 1 --2- 172.21.15.67:0/2059457331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9f415b430 0x7fd9f4072360 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc0097a0 tx=0x7fd9dc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d4900d9d28b334d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:10.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.956+0000 7fd9f1ffb640 1 -- 172.21.15.67:0/2059457331 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc0315a0 con 0x7fd9f415b430 2024-03-24T00:43:10.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.956+0000 7fd9f1ffb640 1 -- 172.21.15.67:0/2059457331 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9dc037070 con 0x7fd9f415b430 2024-03-24T00:43:10.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.956+0000 7fd9f1ffb640 1 -- 172.21.15.67:0/2059457331 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc031960 con 0x7fd9f415b430 2024-03-24T00:43:10.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.957+0000 7fd9f9377640 1 -- 172.21.15.67:0/2059457331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9f415b430 msgr2=0x7fd9f4072360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:10.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.957+0000 7fd9f9377640 1 --2- 172.21.15.67:0/2059457331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9f415b430 0x7fd9f4072360 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc0097a0 tx=0x7fd9dc02b840 comp rx=0 tx=0).stop 2024-03-24T00:43:10.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.957+0000 7fd9f9377640 1 -- 172.21.15.67:0/2059457331 shutdown_connections 2024-03-24T00:43:10.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.957+0000 7fd9f9377640 1 --2- 172.21.15.67:0/2059457331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9f415b430 0x7fd9f4072360 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:10.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.957+0000 7fd9f9377640 1 -- 172.21.15.67:0/2059457331 >> 172.21.15.67:0/2059457331 conn(0x7fd9f40cf9e0 msgr2=0x7fd9f40cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:10.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.957+0000 7fd9f9377640 1 -- 172.21.15.67:0/2059457331 shutdown_connections 2024-03-24T00:43:10.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.957+0000 7fd9f9377640 1 -- 172.21.15.67:0/2059457331 wait complete. 2024-03-24T00:43:10.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.958+0000 7fd9f9377640 1 Processor -- start 2024-03-24T00:43:10.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.958+0000 7fd9f9377640 1 -- start start 2024-03-24T00:43:10.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.959+0000 7fd9f9377640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd9f40728a0 0x7fd9f414c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:10.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.959+0000 7fd9f9377640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9f40b49c0 0x7fd9f414cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:10.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.959+0000 7fd9f9377640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9f415b430 0x7fd9f414d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:10.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.959+0000 7fd9f9377640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f406e6b0 con 0x7fd9f40b49c0 2024-03-24T00:43:10.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.959+0000 7fd9f9377640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f406e3d0 con 0x7fd9f415b430 2024-03-24T00:43:10.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.959+0000 7fd9f9377640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f40b3d30 con 0x7fd9f40728a0 2024-03-24T00:43:10.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.959+0000 7fd9f27fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9f40b49c0 0x7fd9f414cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:10.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.959+0000 7fd9f27fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9f40b49c0 0x7fd9f414cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47296/0 (socket says 172.21.15.67:47296) 2024-03-24T00:43:10.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.959+0000 7fd9f27fc640 1 -- 172.21.15.67:0/3443589382 learned_addr learned my addr 172.21.15.67:0/3443589382 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:10.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.959+0000 7fd9f2ffd640 1 --2- 172.21.15.67:0/3443589382 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd9f40728a0 0x7fd9f414c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:10.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.959+0000 7fd9f37fe640 1 --2- 172.21.15.67:0/3443589382 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9f415b430 0x7fd9f414d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:10.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.960+0000 7fd9f27fc640 1 -- 172.21.15.67:0/3443589382 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd9f40728a0 msgr2=0x7fd9f414c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:10.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.960+0000 7fd9f27fc640 1 --2- 172.21.15.67:0/3443589382 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd9f40728a0 0x7fd9f414c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:10.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.960+0000 7fd9f27fc640 1 -- 172.21.15.67:0/3443589382 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9f415b430 msgr2=0x7fd9f414d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:10.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.960+0000 7fd9f27fc640 1 --2- 172.21.15.67:0/3443589382 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9f415b430 0x7fd9f414d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:10.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.960+0000 7fd9f27fc640 1 -- 172.21.15.67:0/3443589382 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9f414b340 con 0x7fd9f40b49c0 2024-03-24T00:43:10.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.960+0000 7fd9f37fe640 1 --2- 172.21.15.67:0/3443589382 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9f415b430 0x7fd9f414d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:43:10.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.960+0000 7fd9f27fc640 1 --2- 172.21.15.67:0/3443589382 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9f40b49c0 0x7fd9f414cd00 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc0098c0 tx=0x7fd9dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:10.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.960+0000 7fd9eb7fe640 1 -- 172.21.15.67:0/3443589382 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc049070 con 0x7fd9f40b49c0 2024-03-24T00:43:10.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.960+0000 7fd9eb7fe640 1 -- 172.21.15.67:0/3443589382 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9dc032640 con 0x7fd9f40b49c0 2024-03-24T00:43:10.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.960+0000 7fd9f9377640 1 -- 172.21.15.67:0/3443589382 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9f414b620 con 0x7fd9f40b49c0 2024-03-24T00:43:10.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.960+0000 7fd9eb7fe640 1 -- 172.21.15.67:0/3443589382 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc032a10 con 0x7fd9f40b49c0 2024-03-24T00:43:10.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.960+0000 7fd9f9377640 1 -- 172.21.15.67:0/3443589382 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9f414bbe0 con 0x7fd9f40b49c0 2024-03-24T00:43:10.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.961+0000 7fd9f9377640 1 -- 172.21.15.67:0/3443589382 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9f404f400 con 0x7fd9f40b49c0 2024-03-24T00:43:10.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.961+0000 7fd9eb7fe640 1 -- 172.21.15.67:0/3443589382 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd9dc032bb0 con 0x7fd9f40b49c0 2024-03-24T00:43:10.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.962+0000 7fd9eb7fe640 1 --2- 172.21.15.67:0/3443589382 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd9d406bbd0 0x7fd9d406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:10.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.962+0000 7fd9eb7fe640 1 -- 172.21.15.67:0/3443589382 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd9dc02f020 con 0x7fd9f40b49c0 2024-03-24T00:43:10.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.962+0000 7fd9f2ffd640 1 --2- 172.21.15.67:0/3443589382 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd9d406bbd0 0x7fd9d406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:10.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.963+0000 7fd9f2ffd640 1 --2- 172.21.15.67:0/3443589382 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd9d406bbd0 0x7fd9d406e090 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec000f60 tx=0x7fd9ec007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:10.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:10.967+0000 7fd9eb7fe640 1 -- 172.21.15.67:0/3443589382 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd9dc07a560 con 0x7fd9f40b49c0 2024-03-24T00:43:11.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.153+0000 7fd9f9377640 1 -- 172.21.15.67:0/3443589382 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd9f414e870 con 0x7fd9f40b49c0 2024-03-24T00:43:11.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.154+0000 7fd9eb7fe640 1 -- 172.21.15.67:0/3443589382 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v75) v1 ==== 74+0+16898 (secure 0 0 0) 0x7fd9dc07e600 con 0x7fd9f40b49c0 2024-03-24T00:43:11.155 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:43:11.155 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":75,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:43:10.176785+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:43:08.452726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-24T00:43:09.146431+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:43:11.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.156+0000 7fd9f9377640 1 -- 172.21.15.67:0/3443589382 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd9d406bbd0 msgr2=0x7fd9d406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:11.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.156+0000 7fd9f9377640 1 --2- 172.21.15.67:0/3443589382 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd9d406bbd0 0x7fd9d406e090 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec000f60 tx=0x7fd9ec007d50 comp rx=0 tx=0).stop 2024-03-24T00:43:11.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.156+0000 7fd9f9377640 1 -- 172.21.15.67:0/3443589382 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9f40b49c0 msgr2=0x7fd9f414cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:11.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.156+0000 7fd9f9377640 1 --2- 172.21.15.67:0/3443589382 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9f40b49c0 0x7fd9f414cd00 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc0098c0 tx=0x7fd9dc0046c0 comp rx=0 tx=0).stop 2024-03-24T00:43:11.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.156+0000 7fd9f9377640 1 -- 172.21.15.67:0/3443589382 shutdown_connections 2024-03-24T00:43:11.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.156+0000 7fd9f9377640 1 --2- 172.21.15.67:0/3443589382 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd9d406bbd0 0x7fd9d406e090 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:11.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.156+0000 7fd9f9377640 1 --2- 172.21.15.67:0/3443589382 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9f415b430 0x7fd9f414d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:11.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.156+0000 7fd9f9377640 1 --2- 172.21.15.67:0/3443589382 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9f40b49c0 0x7fd9f414cd00 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:11.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.156+0000 7fd9f9377640 1 --2- 172.21.15.67:0/3443589382 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd9f40728a0 0x7fd9f414c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:11.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.156+0000 7fd9f9377640 1 -- 172.21.15.67:0/3443589382 >> 172.21.15.67:0/3443589382 conn(0x7fd9f40cf9e0 msgr2=0x7fd9f40b27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:11.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.156+0000 7fd9f9377640 1 -- 172.21.15.67:0/3443589382 shutdown_connections 2024-03-24T00:43:11.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.156+0000 7fd9f9377640 1 -- 172.21.15.67:0/3443589382 wait complete. 2024-03-24T00:43:11.168 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:43:11.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.269+0000 7ffa52d1b640 1 Processor -- start 2024-03-24T00:43:11.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.269+0000 7ffa52d1b640 1 -- start start 2024-03-24T00:43:11.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.270+0000 7ffa52d1b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ffa4c0d0b90 0x7ffa4c0b01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:11.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.270+0000 7ffa52d1b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa4c0b24f0 con 0x7ffa4c0cf240 2024-03-24T00:43:11.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.270+0000 7ffa52d1b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa4c0b1730 con 0x7ffa4c15ae70 2024-03-24T00:43:11.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.270+0000 7ffa52d1b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa4c0b1a10 con 0x7ffa4c0d0b90 2024-03-24T00:43:11.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.270+0000 7ffa50ab7640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ffa4c0d0b90 0x7ffa4c0b01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:11.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.270+0000 7ffa512b8640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ffa4c0cf240 0x7ffa4c0ad8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44976/0 (socket says 172.21.15.67:44976) 2024-03-24T00:43:11.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.270+0000 7ffa512b8640 1 -- 172.21.15.67:0/752352356 learned_addr learned my addr 172.21.15.67:0/752352356 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:43:11.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.271+0000 7ffa50ab7640 1 -- 172.21.15.67:0/752352356 >> v1:172.21.15.72:6789/0 conn(0x7ffa4c15ae70 legacy=0x7ffa4c0d0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:11.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.271+0000 7ffa50ab7640 1 -- 172.21.15.67:0/752352356 >> v1:172.21.15.67:6789/0 conn(0x7ffa4c0cf240 legacy=0x7ffa4c0ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:11.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.271+0000 7ffa50ab7640 1 -- 172.21.15.67:0/752352356 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa4c0b1cf0 con 0x7ffa4c0d0b90 2024-03-24T00:43:11.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.271+0000 7ffa4b7fe640 1 -- 172.21.15.67:0/752352356 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 586678921 0 0) 0x7ffa4c0b24f0 con 0x7ffa4c0cf240 2024-03-24T00:43:11.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.271+0000 7ffa50ab7640 1 --2- 172.21.15.67:0/752352356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa4c0d0b90 0x7ffa4c0b01d0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7ffa44009d90 tx=0x7ffa4402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd0e93bfdbd568bd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:11.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.272+0000 7ffa4b7fe640 1 -- 172.21.15.67:0/752352356 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa44039070 con 0x7ffa4c0d0b90 2024-03-24T00:43:11.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.272+0000 7ffa4b7fe640 1 -- 172.21.15.67:0/752352356 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa4402cd00 con 0x7ffa4c0d0b90 2024-03-24T00:43:11.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.272+0000 7ffa4b7fe640 1 -- 172.21.15.67:0/752352356 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa44034660 con 0x7ffa4c0d0b90 2024-03-24T00:43:11.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.272+0000 7ffa52d1b640 1 -- 172.21.15.67:0/752352356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa4c0d0b90 msgr2=0x7ffa4c0b01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:11.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.272+0000 7ffa52d1b640 1 --2- 172.21.15.67:0/752352356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa4c0d0b90 0x7ffa4c0b01d0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7ffa44009d90 tx=0x7ffa4402bf50 comp rx=0 tx=0).stop 2024-03-24T00:43:11.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.272+0000 7ffa52d1b640 1 -- 172.21.15.67:0/752352356 shutdown_connections 2024-03-24T00:43:11.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.272+0000 7ffa52d1b640 1 --2- 172.21.15.67:0/752352356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa4c0d0b90 0x7ffa4c0b01d0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:11.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.272+0000 7ffa52d1b640 1 -- 172.21.15.67:0/752352356 >> 172.21.15.67:0/752352356 conn(0x7ffa4c099a30 msgr2=0x7ffa4c099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:11.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.272+0000 7ffa52d1b640 1 -- 172.21.15.67:0/752352356 shutdown_connections 2024-03-24T00:43:11.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.272+0000 7ffa52d1b640 1 -- 172.21.15.67:0/752352356 wait complete. 2024-03-24T00:43:11.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.273+0000 7ffa52d1b640 1 Processor -- start 2024-03-24T00:43:11.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.273+0000 7ffa52d1b640 1 -- start start 2024-03-24T00:43:11.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.274+0000 7ffa52d1b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa4c0cf240 0x7ffa4c14acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:11.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.274+0000 7ffa52d1b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ffa4c0d0b90 0x7ffa4c0c4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:11.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.274+0000 7ffa52d1b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffa4c15ae70 0x7ffa4c0c4850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:11.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.274+0000 7ffa52d1b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4c0cf9e0 con 0x7ffa4c0cf240 2024-03-24T00:43:11.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.274+0000 7ffa52d1b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4c0cf6d0 con 0x7ffa4c15ae70 2024-03-24T00:43:11.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.274+0000 7ffa52d1b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4c0adfe0 con 0x7ffa4c0d0b90 2024-03-24T00:43:11.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.274+0000 7ffa50ab7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa4c0cf240 0x7ffa4c14acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:11.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.274+0000 7ffa50ab7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa4c0cf240 0x7ffa4c14acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47316/0 (socket says 172.21.15.67:47316) 2024-03-24T00:43:11.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.274+0000 7ffa50ab7640 1 -- 172.21.15.67:0/2016453834 learned_addr learned my addr 172.21.15.67:0/2016453834 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:11.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.274+0000 7ffa512b8640 1 --2- 172.21.15.67:0/2016453834 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffa4c15ae70 0x7ffa4c0c4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:11.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.274+0000 7ffa50ab7640 1 -- 172.21.15.67:0/2016453834 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ffa4c0d0b90 msgr2=0x7ffa4c0c4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:11.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.274+0000 7ffa4bfff640 1 --2- 172.21.15.67:0/2016453834 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ffa4c0d0b90 0x7ffa4c0c4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:11.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.274+0000 7ffa50ab7640 1 --2- 172.21.15.67:0/2016453834 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ffa4c0d0b90 0x7ffa4c0c4310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:11.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.274+0000 7ffa50ab7640 1 -- 172.21.15.67:0/2016453834 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffa4c15ae70 msgr2=0x7ffa4c0c4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:11.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.274+0000 7ffa50ab7640 1 --2- 172.21.15.67:0/2016453834 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffa4c15ae70 0x7ffa4c0c4850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:11.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.275+0000 7ffa50ab7640 1 -- 172.21.15.67:0/2016453834 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa4c0b49d0 con 0x7ffa4c0cf240 2024-03-24T00:43:11.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.275+0000 7ffa50ab7640 1 --2- 172.21.15.67:0/2016453834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa4c0cf240 0x7ffa4c14acc0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7ffa44037040 tx=0x7ffa44004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:11.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.275+0000 7ffa497fa640 1 -- 172.21.15.67:0/2016453834 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa44044020 con 0x7ffa4c0cf240 2024-03-24T00:43:11.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.275+0000 7ffa497fa640 1 -- 172.21.15.67:0/2016453834 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa4403f670 con 0x7ffa4c0cf240 2024-03-24T00:43:11.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.275+0000 7ffa52d1b640 1 -- 172.21.15.67:0/2016453834 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa4c14b200 con 0x7ffa4c0cf240 2024-03-24T00:43:11.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.275+0000 7ffa497fa640 1 -- 172.21.15.67:0/2016453834 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa44039040 con 0x7ffa4c0cf240 2024-03-24T00:43:11.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.275+0000 7ffa52d1b640 1 -- 172.21.15.67:0/2016453834 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa4c0c7590 con 0x7ffa4c0cf240 2024-03-24T00:43:11.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.276+0000 7ffa52d1b640 1 -- 172.21.15.67:0/2016453834 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa4c0b4ba0 con 0x7ffa4c0cf240 2024-03-24T00:43:11.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.276+0000 7ffa497fa640 1 -- 172.21.15.67:0/2016453834 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ffa44047020 con 0x7ffa4c0cf240 2024-03-24T00:43:11.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.277+0000 7ffa497fa640 1 --2- 172.21.15.67:0/2016453834 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ffa2c06bbd0 0x7ffa2c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:11.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.277+0000 7ffa497fa640 1 -- 172.21.15.67:0/2016453834 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ffa440ad870 con 0x7ffa4c0cf240 2024-03-24T00:43:11.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.277+0000 7ffa4bfff640 1 --2- 172.21.15.67:0/2016453834 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ffa2c06bbd0 0x7ffa2c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:11.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.277+0000 7ffa4bfff640 1 --2- 172.21.15.67:0/2016453834 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ffa2c06bbd0 0x7ffa2c06e090 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c0c8270 tx=0x7ffa3c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:11.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.281+0000 7ffa497fa640 1 -- 172.21.15.67:0/2016453834 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ffa4404e030 con 0x7ffa4c0cf240 2024-03-24T00:43:11.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.464+0000 7ffa52d1b640 1 -- 172.21.15.67:0/2016453834 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa4c04f400 con 0x7ffa4c0cf240 2024-03-24T00:43:11.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.465+0000 7ffa497fa640 1 -- 172.21.15.67:0/2016453834 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v75) v1 ==== 74+0+16898 (secure 0 0 0) 0x7ffa4407c1d0 con 0x7ffa4c0cf240 2024-03-24T00:43:11.466 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:43:11.467 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":75,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:43:10.176785+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:43:08.452726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-24T00:43:09.146431+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:43:11.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.467+0000 7ffa52d1b640 1 -- 172.21.15.67:0/2016453834 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ffa2c06bbd0 msgr2=0x7ffa2c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:11.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.467+0000 7ffa52d1b640 1 --2- 172.21.15.67:0/2016453834 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ffa2c06bbd0 0x7ffa2c06e090 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c0c8270 tx=0x7ffa3c009040 comp rx=0 tx=0).stop 2024-03-24T00:43:11.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.467+0000 7ffa52d1b640 1 -- 172.21.15.67:0/2016453834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa4c0cf240 msgr2=0x7ffa4c14acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:11.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.467+0000 7ffa52d1b640 1 --2- 172.21.15.67:0/2016453834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa4c0cf240 0x7ffa4c14acc0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7ffa44037040 tx=0x7ffa44004a40 comp rx=0 tx=0).stop 2024-03-24T00:43:11.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.467+0000 7ffa52d1b640 1 -- 172.21.15.67:0/2016453834 shutdown_connections 2024-03-24T00:43:11.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.467+0000 7ffa52d1b640 1 --2- 172.21.15.67:0/2016453834 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ffa2c06bbd0 0x7ffa2c06e090 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:11.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.467+0000 7ffa52d1b640 1 --2- 172.21.15.67:0/2016453834 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffa4c15ae70 0x7ffa4c0c4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:11.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.467+0000 7ffa52d1b640 1 --2- 172.21.15.67:0/2016453834 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ffa4c0d0b90 0x7ffa4c0c4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:11.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.467+0000 7ffa52d1b640 1 --2- 172.21.15.67:0/2016453834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa4c0cf240 0x7ffa4c14acc0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:11.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.468+0000 7ffa52d1b640 1 -- 172.21.15.67:0/2016453834 >> 172.21.15.67:0/2016453834 conn(0x7ffa4c099a30 msgr2=0x7ffa4c15d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:11.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.468+0000 7ffa52d1b640 1 -- 172.21.15.67:0/2016453834 shutdown_connections 2024-03-24T00:43:11.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.468+0000 7ffa52d1b640 1 -- 172.21.15.67:0/2016453834 wait complete. 2024-03-24T00:43:11.479 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-24T00:43:11.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.583+0000 7f37cf44b640 1 Processor -- start 2024-03-24T00:43:11.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.583+0000 7f37cf44b640 1 -- start start 2024-03-24T00:43:11.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.584+0000 7f37cf44b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f37c8144670 0x7f37c80b3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:11.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.584+0000 7f37cf44b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f37c80b4a00 0x7f37c8072500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:11.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.584+0000 7f37cf44b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f37c806ec60 0x7f37c806f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:11.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.584+0000 7f37cf44b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37c80b2ab0 con 0x7f37c806ec60 2024-03-24T00:43:11.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.584+0000 7f37cf44b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37c80b2d90 con 0x7f37c80b4a00 2024-03-24T00:43:11.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.584+0000 7f37cf44b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37c80b3070 con 0x7f37c8144670 2024-03-24T00:43:11.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.584+0000 7f37cd9e8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f37c806ec60 0x7f37c806f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:11.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.584+0000 7f37cd9e8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f37c806ec60 0x7f37c806f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47328/0 (socket says 172.21.15.67:47328) 2024-03-24T00:43:11.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.584+0000 7f37cd9e8640 1 -- 172.21.15.67:0/2864127325 learned_addr learned my addr 172.21.15.67:0/2864127325 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:11.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.585+0000 7f37cc9e6640 1 --2- 172.21.15.67:0/2864127325 >> v2:172.21.15.72:3300/0 conn(0x7f37c80b4a00 0x7f37c8072500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:11.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.585+0000 7f37cd1e7640 1 --2- 172.21.15.67:0/2864127325 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f37c8144670 0x7f37c80b3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:11.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.585+0000 7f37cd9e8640 1 -- 172.21.15.67:0/2864127325 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f37c8144670 msgr2=0x7f37c80b3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:11.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.585+0000 7f37cd9e8640 1 --2- 172.21.15.67:0/2864127325 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f37c8144670 0x7f37c80b3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:11.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.585+0000 7f37cd9e8640 1 -- 172.21.15.67:0/2864127325 >> v2:172.21.15.72:3300/0 conn(0x7f37c80b4a00 msgr2=0x7f37c8072500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:11.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.585+0000 7f37cd9e8640 1 --2- 172.21.15.67:0/2864127325 >> v2:172.21.15.72:3300/0 conn(0x7f37c80b4a00 0x7f37c8072500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:11.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.585+0000 7f37cd9e8640 1 -- 172.21.15.67:0/2864127325 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c80b3350 con 0x7f37c806ec60 2024-03-24T00:43:11.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.585+0000 7f37cc9e6640 1 --2- 172.21.15.67:0/2864127325 >> v2:172.21.15.72:3300/0 conn(0x7f37c80b4a00 0x7f37c8072500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:43:11.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.585+0000 7f37cd1e7640 1 --2- 172.21.15.67:0/2864127325 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f37c8144670 0x7f37c80b3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:43:11.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.585+0000 7f37cd9e8640 1 --2- 172.21.15.67:0/2864127325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37c806ec60 0x7f37c806f0f0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f37c806daa0 tx=0x7f37b802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebfde2692242f18e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:11.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.585+0000 7f37c7fff640 1 -- 172.21.15.67:0/2864127325 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b802ca70 con 0x7f37c806ec60 2024-03-24T00:43:11.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.586+0000 7f37c7fff640 1 -- 172.21.15.67:0/2864127325 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37b802cc10 con 0x7f37c806ec60 2024-03-24T00:43:11.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.586+0000 7f37c7fff640 1 -- 172.21.15.67:0/2864127325 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b80343f0 con 0x7f37c806ec60 2024-03-24T00:43:11.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.586+0000 7f37cf44b640 1 -- 172.21.15.67:0/2864127325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37c806ec60 msgr2=0x7f37c806f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:11.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.586+0000 7f37cf44b640 1 --2- 172.21.15.67:0/2864127325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37c806ec60 0x7f37c806f0f0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f37c806daa0 tx=0x7f37b802bed0 comp rx=0 tx=0).stop 2024-03-24T00:43:11.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.586+0000 7f37cf44b640 1 -- 172.21.15.67:0/2864127325 shutdown_connections 2024-03-24T00:43:11.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.586+0000 7f37cf44b640 1 --2- 172.21.15.67:0/2864127325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37c806ec60 0x7f37c806f0f0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:11.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.586+0000 7f37cf44b640 1 --2- 172.21.15.67:0/2864127325 >> v2:172.21.15.72:3300/0 conn(0x7f37c80b4a00 0x7f37c8072500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:11.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.586+0000 7f37cf44b640 1 --2- 172.21.15.67:0/2864127325 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f37c8144670 0x7f37c80b3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:11.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.586+0000 7f37cf44b640 1 -- 172.21.15.67:0/2864127325 >> 172.21.15.67:0/2864127325 conn(0x7f37c80cf9e0 msgr2=0x7f37c80cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:11.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.586+0000 7f37cf44b640 1 -- 172.21.15.67:0/2864127325 shutdown_connections 2024-03-24T00:43:11.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.586+0000 7f37cf44b640 1 -- 172.21.15.67:0/2864127325 wait complete. 2024-03-24T00:43:11.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.587+0000 7f37cf44b640 1 Processor -- start 2024-03-24T00:43:11.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.587+0000 7f37cf44b640 1 -- start start 2024-03-24T00:43:11.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.588+0000 7f37cf44b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f37c806ec60 0x7f37c8159150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:11.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.588+0000 7f37cf44b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37c80b4a00 0x7f37c8155770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:11.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.588+0000 7f37cf44b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37c8144670 0x7f37c8141060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:11.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.588+0000 7f37cf44b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c80cde30 con 0x7f37c8144670 2024-03-24T00:43:11.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.588+0000 7f37cf44b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c8072d30 con 0x7f37c80b4a00 2024-03-24T00:43:11.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.588+0000 7f37cf44b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37c80cdfb0 con 0x7f37c806ec60 2024-03-24T00:43:11.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.588+0000 7f37cd9e8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37c8144670 0x7f37c8141060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:11.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.588+0000 7f37cd9e8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37c8144670 0x7f37c8141060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47342/0 (socket says 172.21.15.67:47342) 2024-03-24T00:43:11.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.588+0000 7f37cd9e8640 1 -- 172.21.15.67:0/4163228722 learned_addr learned my addr 172.21.15.67:0/4163228722 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:11.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.588+0000 7f37cc9e6640 1 --2- 172.21.15.67:0/4163228722 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37c80b4a00 0x7f37c8155770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:11.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.588+0000 7f37cd1e7640 1 --2- 172.21.15.67:0/4163228722 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f37c806ec60 0x7f37c8159150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:11.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.589+0000 7f37cd9e8640 1 -- 172.21.15.67:0/4163228722 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f37c806ec60 msgr2=0x7f37c8159150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:11.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.589+0000 7f37cd9e8640 1 --2- 172.21.15.67:0/4163228722 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f37c806ec60 0x7f37c8159150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:11.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.589+0000 7f37cd9e8640 1 -- 172.21.15.67:0/4163228722 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37c80b4a00 msgr2=0x7f37c8155770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:11.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.589+0000 7f37cd9e8640 1 --2- 172.21.15.67:0/4163228722 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37c80b4a00 0x7f37c8155770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:11.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.589+0000 7f37cd9e8640 1 -- 172.21.15.67:0/4163228722 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c81561c0 con 0x7f37c8144670 2024-03-24T00:43:11.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.589+0000 7f37cd9e8640 1 --2- 172.21.15.67:0/4163228722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37c8144670 0x7f37c8141060 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f37b8036040 tx=0x7f37b8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:11.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.589+0000 7f37c5ffb640 1 -- 172.21.15.67:0/4163228722 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b8043020 con 0x7f37c8144670 2024-03-24T00:43:11.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.589+0000 7f37c5ffb640 1 -- 172.21.15.67:0/4163228722 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37b803e450 con 0x7f37c8144670 2024-03-24T00:43:11.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.589+0000 7f37cf44b640 1 -- 172.21.15.67:0/4163228722 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37c81564a0 con 0x7f37c8144670 2024-03-24T00:43:11.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.589+0000 7f37c5ffb640 1 -- 172.21.15.67:0/4163228722 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b8009510 con 0x7f37c8144670 2024-03-24T00:43:11.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.589+0000 7f37cf44b640 1 -- 172.21.15.67:0/4163228722 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37c8156a60 con 0x7f37c8144670 2024-03-24T00:43:11.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.590+0000 7f37cf44b640 1 -- 172.21.15.67:0/4163228722 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37c8156e90 con 0x7f37c8144670 2024-03-24T00:43:11.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.590+0000 7f37c5ffb640 1 -- 172.21.15.67:0/4163228722 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f37b8046020 con 0x7f37c8144670 2024-03-24T00:43:11.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.591+0000 7f37c5ffb640 1 --2- 172.21.15.67:0/4163228722 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f37b406bbd0 0x7f37b406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:11.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.591+0000 7f37c5ffb640 1 -- 172.21.15.67:0/4163228722 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f37b80ad2e0 con 0x7f37c8144670 2024-03-24T00:43:11.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.591+0000 7f37cd1e7640 1 --2- 172.21.15.67:0/4163228722 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f37b406bbd0 0x7f37b406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:11.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.592+0000 7f37cd1e7640 1 --2- 172.21.15.67:0/4163228722 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f37b406bbd0 0x7f37b406e090 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f37c000aac0 tx=0x7f37c0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:11.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.595+0000 7f37c5ffb640 1 -- 172.21.15.67:0/4163228722 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f37b804d030 con 0x7f37c8144670 2024-03-24T00:43:11.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.814+0000 7f37cf44b640 1 -- 172.21.15.67:0/4163228722 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f37c80b4280 con 0x7f37c8144670 2024-03-24T00:43:11.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.826+0000 7f37c5ffb640 1 -- 172.21.15.67:0/4163228722 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v39) v1 ==== 118+0+0 (secure 0 0 0) 0x7f37b807b680 con 0x7f37c8144670 2024-03-24T00:43:11.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.829+0000 7f37cf44b640 1 -- 172.21.15.67:0/4163228722 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f37b406bbd0 msgr2=0x7f37b406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:11.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.829+0000 7f37cf44b640 1 --2- 172.21.15.67:0/4163228722 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f37b406bbd0 0x7f37b406e090 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f37c000aac0 tx=0x7f37c0008040 comp rx=0 tx=0).stop 2024-03-24T00:43:11.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.829+0000 7f37cf44b640 1 -- 172.21.15.67:0/4163228722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37c8144670 msgr2=0x7f37c8141060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:11.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.829+0000 7f37cf44b640 1 --2- 172.21.15.67:0/4163228722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37c8144670 0x7f37c8141060 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f37b8036040 tx=0x7f37b8004a40 comp rx=0 tx=0).stop 2024-03-24T00:43:11.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.829+0000 7f37cf44b640 1 -- 172.21.15.67:0/4163228722 shutdown_connections 2024-03-24T00:43:11.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.829+0000 7f37cf44b640 1 --2- 172.21.15.67:0/4163228722 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f37b406bbd0 0x7f37b406e090 secure :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f37c000aac0 tx=0x7f37c0008040 comp rx=0 tx=0).stop 2024-03-24T00:43:11.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.829+0000 7f37cf44b640 1 --2- 172.21.15.67:0/4163228722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37c8144670 0x7f37c8141060 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:11.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.829+0000 7f37cf44b640 1 --2- 172.21.15.67:0/4163228722 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37c80b4a00 0x7f37c8155770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:11.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.829+0000 7f37cf44b640 1 --2- 172.21.15.67:0/4163228722 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f37c806ec60 0x7f37c8159150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:11.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.829+0000 7f37cf44b640 1 -- 172.21.15.67:0/4163228722 >> 172.21.15.67:0/4163228722 conn(0x7f37c80cf9e0 msgr2=0x7f37c806dcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:11.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.829+0000 7f37cf44b640 1 -- 172.21.15.67:0/4163228722 shutdown_connections 2024-03-24T00:43:11.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.830+0000 7f37cf44b640 1 -- 172.21.15.67:0/4163228722 wait complete. 2024-03-24T00:43:11.842 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-24T00:43:11.842 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-24T00:43:11.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.944+0000 7f3bcf692640 1 Processor -- start 2024-03-24T00:43:11.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.945+0000 7f3bcf692640 1 -- start start 2024-03-24T00:43:11.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.946+0000 7f3bcf692640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bc815dea0 0x7f3bc80d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:11.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.946+0000 7f3bcf692640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3bc815e280 0x7f3bc80d3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:11.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.946+0000 7f3bcf692640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3bc80b5a60 0x7f3bc80b5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:11.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.946+0000 7f3bcf692640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bc80b6410 con 0x7f3bc80b5a60 2024-03-24T00:43:11.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.946+0000 7f3bcf692640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bc80d2250 con 0x7f3bc815e280 2024-03-24T00:43:11.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.946+0000 7f3bcf692640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc80d2530 con 0x7f3bc815dea0 2024-03-24T00:43:11.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.946+0000 7f3bcdc2f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3bc80b5a60 0x7f3bc80b5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:11.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.946+0000 7f3bcdc2f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3bc80b5a60 0x7f3bc80b5ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47352/0 (socket says 172.21.15.67:47352) 2024-03-24T00:43:11.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.946+0000 7f3bcdc2f640 1 -- 172.21.15.67:0/1326182673 learned_addr learned my addr 172.21.15.67:0/1326182673 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:11.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.946+0000 7f3bccc2d640 1 --2- 172.21.15.67:0/1326182673 >> v2:172.21.15.72:3300/0 conn(0x7f3bc815e280 0x7f3bc80d3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:11.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.946+0000 7f3bcd42e640 1 --2- 172.21.15.67:0/1326182673 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bc815dea0 0x7f3bc80d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:11.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.946+0000 7f3bcdc2f640 1 -- 172.21.15.67:0/1326182673 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bc815dea0 msgr2=0x7f3bc80d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:11.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.946+0000 7f3bcdc2f640 1 --2- 172.21.15.67:0/1326182673 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bc815dea0 0x7f3bc80d33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:11.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.946+0000 7f3bcdc2f640 1 -- 172.21.15.67:0/1326182673 >> v2:172.21.15.72:3300/0 conn(0x7f3bc815e280 msgr2=0x7f3bc80d3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:11.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.946+0000 7f3bcdc2f640 1 --2- 172.21.15.67:0/1326182673 >> v2:172.21.15.72:3300/0 conn(0x7f3bc815e280 0x7f3bc80d3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:11.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.946+0000 7f3bcdc2f640 1 -- 172.21.15.67:0/1326182673 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bc80d2810 con 0x7f3bc80b5a60 2024-03-24T00:43:11.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.946+0000 7f3bcd42e640 1 --2- 172.21.15.67:0/1326182673 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bc815dea0 0x7f3bc80d33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:43:11.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.946+0000 7f3bccc2d640 1 --2- 172.21.15.67:0/1326182673 >> v2:172.21.15.72:3300/0 conn(0x7f3bc815e280 0x7f3bc80d3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:43:11.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.947+0000 7f3bcdc2f640 1 --2- 172.21.15.67:0/1326182673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bc80b5a60 0x7f3bc80b5ed0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f3bc80d2460 tx=0x7f3bc402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9ae04f2f8d9371 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:11.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.947+0000 7f3bbffff640 1 -- 172.21.15.67:0/1326182673 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc402c9e0 con 0x7f3bc80b5a60 2024-03-24T00:43:11.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.947+0000 7f3bbffff640 1 -- 172.21.15.67:0/1326182673 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bc402cb80 con 0x7f3bc80b5a60 2024-03-24T00:43:11.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.947+0000 7f3bbffff640 1 -- 172.21.15.67:0/1326182673 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc40343f0 con 0x7f3bc80b5a60 2024-03-24T00:43:11.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.948+0000 7f3bcf692640 1 -- 172.21.15.67:0/1326182673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bc80b5a60 msgr2=0x7f3bc80b5ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:11.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.948+0000 7f3bcf692640 1 --2- 172.21.15.67:0/1326182673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bc80b5a60 0x7f3bc80b5ed0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f3bc80d2460 tx=0x7f3bc402be40 comp rx=0 tx=0).stop 2024-03-24T00:43:11.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.948+0000 7f3bcf692640 1 -- 172.21.15.67:0/1326182673 shutdown_connections 2024-03-24T00:43:11.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.948+0000 7f3bcf692640 1 --2- 172.21.15.67:0/1326182673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bc80b5a60 0x7f3bc80b5ed0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:11.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.948+0000 7f3bcf692640 1 --2- 172.21.15.67:0/1326182673 >> v2:172.21.15.72:3300/0 conn(0x7f3bc815e280 0x7f3bc80d3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:11.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.948+0000 7f3bcf692640 1 --2- 172.21.15.67:0/1326182673 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bc815dea0 0x7f3bc80d33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:11.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.948+0000 7f3bcf692640 1 -- 172.21.15.67:0/1326182673 >> 172.21.15.67:0/1326182673 conn(0x7f3bc8072da0 msgr2=0x7f3bc80d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:11.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.948+0000 7f3bcf692640 1 -- 172.21.15.67:0/1326182673 shutdown_connections 2024-03-24T00:43:11.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.948+0000 7f3bcf692640 1 -- 172.21.15.67:0/1326182673 wait complete. 2024-03-24T00:43:11.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.949+0000 7f3bcf692640 1 Processor -- start 2024-03-24T00:43:11.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.949+0000 7f3bcf692640 1 -- start start 2024-03-24T00:43:11.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.950+0000 7f3bcf692640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bc80b5a60 0x7f3bc80ca860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:11.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.950+0000 7f3bcf692640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bc815dea0 0x7f3bc80cb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:11.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.950+0000 7f3bcf692640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bc815e280 0x7f3bc80cd340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:11.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.950+0000 7f3bcf692640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc806c7c0 con 0x7f3bc815dea0 2024-03-24T00:43:11.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.950+0000 7f3bcf692640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc806c640 con 0x7f3bc815e280 2024-03-24T00:43:11.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.950+0000 7f3bcf692640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc806c940 con 0x7f3bc80b5a60 2024-03-24T00:43:11.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.950+0000 7f3bccc2d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bc815dea0 0x7f3bc80cb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:11.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.950+0000 7f3bccc2d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bc815dea0 0x7f3bc80cb0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47356/0 (socket says 172.21.15.67:47356) 2024-03-24T00:43:11.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.950+0000 7f3bccc2d640 1 -- 172.21.15.67:0/3553697663 learned_addr learned my addr 172.21.15.67:0/3553697663 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:11.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.950+0000 7f3bcdc2f640 1 --2- 172.21.15.67:0/3553697663 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bc815e280 0x7f3bc80cd340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:11.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.950+0000 7f3bcd42e640 1 --2- 172.21.15.67:0/3553697663 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bc80b5a60 0x7f3bc80ca860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:11.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.951+0000 7f3bccc2d640 1 -- 172.21.15.67:0/3553697663 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bc80b5a60 msgr2=0x7f3bc80ca860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:11.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.951+0000 7f3bccc2d640 1 --2- 172.21.15.67:0/3553697663 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bc80b5a60 0x7f3bc80ca860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:11.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.951+0000 7f3bccc2d640 1 -- 172.21.15.67:0/3553697663 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bc815e280 msgr2=0x7f3bc80cd340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:11.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.951+0000 7f3bccc2d640 1 --2- 172.21.15.67:0/3553697663 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bc815e280 0x7f3bc80cd340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:11.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.951+0000 7f3bccc2d640 1 -- 172.21.15.67:0/3553697663 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bc80cba90 con 0x7f3bc815dea0 2024-03-24T00:43:11.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.951+0000 7f3bcd42e640 1 --2- 172.21.15.67:0/3553697663 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bc80b5a60 0x7f3bc80ca860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:43:11.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.951+0000 7f3bccc2d640 1 --2- 172.21.15.67:0/3553697663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bc815dea0 0x7f3bc80cb0b0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f3bb800d7d0 tx=0x7f3bb800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:11.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.951+0000 7f3bbdffb640 1 -- 172.21.15.67:0/3553697663 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bb8004030 con 0x7f3bc815dea0 2024-03-24T00:43:11.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.951+0000 7f3bbdffb640 1 -- 172.21.15.67:0/3553697663 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bb80041d0 con 0x7f3bc815dea0 2024-03-24T00:43:11.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.951+0000 7f3bcf692640 1 -- 172.21.15.67:0/3553697663 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bc80cbd70 con 0x7f3bc815dea0 2024-03-24T00:43:11.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.951+0000 7f3bbdffb640 1 -- 172.21.15.67:0/3553697663 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bb80096a0 con 0x7f3bc815dea0 2024-03-24T00:43:11.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.951+0000 7f3bcf692640 1 -- 172.21.15.67:0/3553697663 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bc80cc330 con 0x7f3bc815dea0 2024-03-24T00:43:11.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.952+0000 7f3bcf692640 1 -- 172.21.15.67:0/3553697663 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bc804f400 con 0x7f3bc815dea0 2024-03-24T00:43:11.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.953+0000 7f3bbdffb640 1 -- 172.21.15.67:0/3553697663 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3bb8009840 con 0x7f3bc815dea0 2024-03-24T00:43:11.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.953+0000 7f3bbdffb640 1 --2- 172.21.15.67:0/3553697663 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3bac06bbd0 0x7f3bac06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:11.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.953+0000 7f3bbdffb640 1 -- 172.21.15.67:0/3553697663 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3bb808ae30 con 0x7f3bc815dea0 2024-03-24T00:43:11.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.954+0000 7f3bcd42e640 1 --2- 172.21.15.67:0/3553697663 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3bac06bbd0 0x7f3bac06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:11.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.954+0000 7f3bcd42e640 1 --2- 172.21.15.67:0/3553697663 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3bac06bbd0 0x7f3bac06e090 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f3bc80b6380 tx=0x7f3bc0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:11.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:11.958+0000 7f3bbdffb640 1 -- 172.21.15.67:0/3553697663 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3bb8059790 con 0x7f3bc815dea0 2024-03-24T00:43:12.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.179+0000 7f3bcf692640 1 -- 172.21.15.67:0/3553697663 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f3bc80b8a30 con 0x7f3bc815dea0 2024-03-24T00:43:12.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.827+0000 7f3bbdffb640 1 -- 172.21.15.67:0/3553697663 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v40) v1 ==== 122+0+0 (secure 0 0 0) 0x7f3bb8059090 con 0x7f3bc815dea0 2024-03-24T00:43:12.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.829+0000 7f3bcf692640 1 -- 172.21.15.67:0/3553697663 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3bac06bbd0 msgr2=0x7f3bac06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:12.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.829+0000 7f3bcf692640 1 --2- 172.21.15.67:0/3553697663 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3bac06bbd0 0x7f3bac06e090 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f3bc80b6380 tx=0x7f3bc0008040 comp rx=0 tx=0).stop 2024-03-24T00:43:12.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.829+0000 7f3bcf692640 1 -- 172.21.15.67:0/3553697663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bc815dea0 msgr2=0x7f3bc80cb0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:12.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.829+0000 7f3bcf692640 1 --2- 172.21.15.67:0/3553697663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bc815dea0 0x7f3bc80cb0b0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f3bb800d7d0 tx=0x7f3bb800dc90 comp rx=0 tx=0).stop 2024-03-24T00:43:12.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.829+0000 7f3bcf692640 1 -- 172.21.15.67:0/3553697663 shutdown_connections 2024-03-24T00:43:12.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.829+0000 7f3bcf692640 1 --2- 172.21.15.67:0/3553697663 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3bac06bbd0 0x7f3bac06e090 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:12.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.829+0000 7f3bcf692640 1 --2- 172.21.15.67:0/3553697663 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bc815e280 0x7f3bc80cd340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:12.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.829+0000 7f3bcf692640 1 --2- 172.21.15.67:0/3553697663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bc815dea0 0x7f3bc80cb0b0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:12.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.829+0000 7f3bcf692640 1 --2- 172.21.15.67:0/3553697663 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bc80b5a60 0x7f3bc80ca860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:12.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.829+0000 7f3bcf692640 1 -- 172.21.15.67:0/3553697663 >> 172.21.15.67:0/3553697663 conn(0x7f3bc8072da0 msgr2=0x7f3bc80b6950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:12.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.829+0000 7f3bcf692640 1 -- 172.21.15.67:0/3553697663 shutdown_connections 2024-03-24T00:43:12.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.830+0000 7f3bcf692640 1 -- 172.21.15.67:0/3553697663 wait complete. 2024-03-24T00:43:12.841 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-24T00:43:12.841 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:43:12.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.943+0000 7fb4c64b2640 1 Processor -- start 2024-03-24T00:43:12.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.943+0000 7fb4c64b2640 1 -- start start 2024-03-24T00:43:12.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.943+0000 7fb4c64b2640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb4c009a410 0x7fb4c00cfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:12.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.944+0000 7fb4c64b2640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb4c00b43a0 0x7fb4c00cc880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:12.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.944+0000 7fb4c64b2640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4c00d0fc0 con 0x7fb4c015e1b0 2024-03-24T00:43:12.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.944+0000 7fb4c64b2640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4c015e590 con 0x7fb4c009a410 2024-03-24T00:43:12.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.944+0000 7fb4c64b2640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4c00d1820 con 0x7fb4c00b43a0 2024-03-24T00:43:12.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.944+0000 7fb4bf7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb4c009a410 0x7fb4c00cfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:12.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.944+0000 7fb4bf7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb4c009a410 0x7fb4c00cfe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47362/0 (socket says 172.21.15.67:47362) 2024-03-24T00:43:12.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.944+0000 7fb4bf7fe640 1 -- 172.21.15.67:0/745244979 learned_addr learned my addr 172.21.15.67:0/745244979 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:12.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.944+0000 7fb4bf7fe640 1 -- 172.21.15.67:0/745244979 >> v2:172.21.15.72:3300/0 conn(0x7fb4c00b43a0 msgr2=0x7fb4c00cc880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:12.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.944+0000 7fb4c4a4f640 1 --2- 172.21.15.67:0/745244979 >> v2:172.21.15.72:3300/0 conn(0x7fb4c00b43a0 0x7fb4c00cc880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:12.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.944+0000 7fb4bf7fe640 1 --2- 172.21.15.67:0/745244979 >> v2:172.21.15.72:3300/0 conn(0x7fb4c00b43a0 0x7fb4c00cc880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:12.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.944+0000 7fb4bf7fe640 1 -- 172.21.15.67:0/745244979 >> v1:172.21.15.72:6789/0 conn(0x7fb4c015e1b0 legacy=0x7fb4c0099d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:12.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.944+0000 7fb4bf7fe640 1 -- 172.21.15.67:0/745244979 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4c00d1260 con 0x7fb4c009a410 2024-03-24T00:43:12.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.945+0000 7fb4bf7fe640 1 --2- 172.21.15.67:0/745244979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4c009a410 0x7fb4c00cfe20 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fb4c00b36a0 tx=0x7fb4b402b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d9d875c0bce8ace server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:12.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.945+0000 7fb4beffd640 1 -- 172.21.15.67:0/745244979 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b4038070 con 0x7fb4c009a410 2024-03-24T00:43:12.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.945+0000 7fb4beffd640 1 -- 172.21.15.67:0/745244979 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4b402f050 con 0x7fb4c009a410 2024-03-24T00:43:12.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.945+0000 7fb4beffd640 1 -- 172.21.15.67:0/745244979 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b4031970 con 0x7fb4c009a410 2024-03-24T00:43:12.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.945+0000 7fb4c64b2640 1 -- 172.21.15.67:0/745244979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4c009a410 msgr2=0x7fb4c00cfe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:12.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.945+0000 7fb4c64b2640 1 --2- 172.21.15.67:0/745244979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4c009a410 0x7fb4c00cfe20 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fb4c00b36a0 tx=0x7fb4b402b7f0 comp rx=0 tx=0).stop 2024-03-24T00:43:12.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.946+0000 7fb4c64b2640 1 -- 172.21.15.67:0/745244979 shutdown_connections 2024-03-24T00:43:12.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.946+0000 7fb4c64b2640 1 --2- 172.21.15.67:0/745244979 >> v2:172.21.15.72:3300/0 conn(0x7fb4c00b43a0 0x7fb4c00cc880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:12.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.946+0000 7fb4c64b2640 1 --2- 172.21.15.67:0/745244979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4c009a410 0x7fb4c00cfe20 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:12.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.946+0000 7fb4c64b2640 1 -- 172.21.15.67:0/745244979 >> 172.21.15.67:0/745244979 conn(0x7fb4c00b36d0 msgr2=0x7fb4c00b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:12.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.946+0000 7fb4c64b2640 1 -- 172.21.15.67:0/745244979 shutdown_connections 2024-03-24T00:43:12.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.946+0000 7fb4c64b2640 1 -- 172.21.15.67:0/745244979 wait complete. 2024-03-24T00:43:12.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.947+0000 7fb4c64b2640 1 Processor -- start 2024-03-24T00:43:12.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.947+0000 7fb4c64b2640 1 -- start start 2024-03-24T00:43:12.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.948+0000 7fb4c64b2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4c009a410 0x7fb4c00afd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:12.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.948+0000 7fb4c64b2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb4c00b43a0 0x7fb4c00b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:12.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.948+0000 7fb4c64b2640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb4c015e1b0 0x7fb4c00caf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:12.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.948+0000 7fb4c64b2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c009a9a0 con 0x7fb4c009a410 2024-03-24T00:43:12.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.948+0000 7fb4c64b2640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c00cc360 con 0x7fb4c00b43a0 2024-03-24T00:43:12.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.948+0000 7fb4c64b2640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c00ccdc0 con 0x7fb4c015e1b0 2024-03-24T00:43:12.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.948+0000 7fb4bffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4c009a410 0x7fb4c00afd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:12.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.948+0000 7fb4bffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4c009a410 0x7fb4c00afd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47374/0 (socket says 172.21.15.67:47374) 2024-03-24T00:43:12.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.948+0000 7fb4bffff640 1 -- 172.21.15.67:0/2587094617 learned_addr learned my addr 172.21.15.67:0/2587094617 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:12.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.948+0000 7fb4bf7fe640 1 --2- 172.21.15.67:0/2587094617 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb4c00b43a0 0x7fb4c00b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:12.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.948+0000 7fb4c4a4f640 1 --2- 172.21.15.67:0/2587094617 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb4c015e1b0 0x7fb4c00caf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:12.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.948+0000 7fb4bffff640 1 -- 172.21.15.67:0/2587094617 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb4c015e1b0 msgr2=0x7fb4c00caf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:12.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.948+0000 7fb4bffff640 1 --2- 172.21.15.67:0/2587094617 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb4c015e1b0 0x7fb4c00caf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:12.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.948+0000 7fb4bffff640 1 -- 172.21.15.67:0/2587094617 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb4c00b43a0 msgr2=0x7fb4c00b2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:12.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.948+0000 7fb4bffff640 1 --2- 172.21.15.67:0/2587094617 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb4c00b43a0 0x7fb4c00b2120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:12.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.948+0000 7fb4bffff640 1 -- 172.21.15.67:0/2587094617 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4c00b5910 con 0x7fb4c009a410 2024-03-24T00:43:12.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.948+0000 7fb4bffff640 1 --2- 172.21.15.67:0/2587094617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4c009a410 0x7fb4c00afd80 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fb4b000b3f0 tx=0x7fb4b000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:12.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.948+0000 7fb4bcff9640 1 -- 172.21.15.67:0/2587094617 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b0015020 con 0x7fb4c009a410 2024-03-24T00:43:12.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.948+0000 7fb4bcff9640 1 -- 172.21.15.67:0/2587094617 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4b0011830 con 0x7fb4c009a410 2024-03-24T00:43:12.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.948+0000 7fb4c64b2640 1 -- 172.21.15.67:0/2587094617 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4c00b4870 con 0x7fb4c009a410 2024-03-24T00:43:12.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.948+0000 7fb4bcff9640 1 -- 172.21.15.67:0/2587094617 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b0011c20 con 0x7fb4c009a410 2024-03-24T00:43:12.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.949+0000 7fb4c64b2640 1 -- 172.21.15.67:0/2587094617 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4c00b5110 con 0x7fb4c009a410 2024-03-24T00:43:12.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.950+0000 7fb4c64b2640 1 -- 172.21.15.67:0/2587094617 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4c004f400 con 0x7fb4c009a410 2024-03-24T00:43:12.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.951+0000 7fb4bcff9640 1 -- 172.21.15.67:0/2587094617 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb4b0018020 con 0x7fb4c009a410 2024-03-24T00:43:12.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.951+0000 7fb4bcff9640 1 --2- 172.21.15.67:0/2587094617 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb4ac06bbd0 0x7fb4ac06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:12.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.951+0000 7fb4bcff9640 1 -- 172.21.15.67:0/2587094617 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb4b008a7b0 con 0x7fb4c009a410 2024-03-24T00:43:12.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.951+0000 7fb4bf7fe640 1 --2- 172.21.15.67:0/2587094617 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb4ac06bbd0 0x7fb4ac06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:12.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.952+0000 7fb4bf7fe640 1 --2- 172.21.15.67:0/2587094617 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb4ac06bbd0 0x7fb4ac06e090 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fb4b4035000 tx=0x7fb4b4008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:12.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:12.955+0000 7fb4bcff9640 1 -- 172.21.15.67:0/2587094617 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb4b001f030 con 0x7fb4c009a410 2024-03-24T00:43:13.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.181+0000 7fb4c64b2640 1 -- 172.21.15.67:0/2587094617 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb4c00b2be0 con 0x7fb4c009a410 2024-03-24T00:43:13.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.181+0000 7fb4bcff9640 1 -- 172.21.15.67:0/2587094617 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+4979 (secure 0 0 0) 0x7fb4b00172c0 con 0x7fb4c009a410 2024-03-24T00:43:13.182 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:43:13.182 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":40,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5960,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1888378520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1888378520},{"type":"v1","addr":"172.21.15.67:6835","nonce":1888378520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6610,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/2539718386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2539718386},{"type":"v1","addr":"172.21.15.72:6836","nonce":2539718386}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6613,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2376370770","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2376370770},{"type":"v1","addr":"172.21.15.72:6837","nonce":2376370770}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:43:10.177088+0000","modified":"2024-03-24T00:43:12.181406+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6894},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6894":{"gid":6894,"name":"c","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6833/2392802741","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2392802741},{"type":"v1","addr":"172.21.15.67:6833","nonce":2392802741}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6894,"qdb_cluster":[6894]},"id":5}]} 2024-03-24T00:43:13.183 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 40 2024-03-24T00:43:13.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.184+0000 7fb4c64b2640 1 -- 172.21.15.67:0/2587094617 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb4ac06bbd0 msgr2=0x7fb4ac06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:13.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.184+0000 7fb4c64b2640 1 --2- 172.21.15.67:0/2587094617 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb4ac06bbd0 0x7fb4ac06e090 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fb4b4035000 tx=0x7fb4b4008030 comp rx=0 tx=0).stop 2024-03-24T00:43:13.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.184+0000 7fb4c64b2640 1 -- 172.21.15.67:0/2587094617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4c009a410 msgr2=0x7fb4c00afd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:13.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.184+0000 7fb4c64b2640 1 --2- 172.21.15.67:0/2587094617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4c009a410 0x7fb4c00afd80 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fb4b000b3f0 tx=0x7fb4b000b8b0 comp rx=0 tx=0).stop 2024-03-24T00:43:13.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.184+0000 7fb4c64b2640 1 -- 172.21.15.67:0/2587094617 shutdown_connections 2024-03-24T00:43:13.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.184+0000 7fb4c64b2640 1 --2- 172.21.15.67:0/2587094617 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb4ac06bbd0 0x7fb4ac06e090 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:13.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.184+0000 7fb4c64b2640 1 --2- 172.21.15.67:0/2587094617 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb4c015e1b0 0x7fb4c00caf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:13.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.184+0000 7fb4c64b2640 1 --2- 172.21.15.67:0/2587094617 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb4c00b43a0 0x7fb4c00b2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:13.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.184+0000 7fb4c64b2640 1 --2- 172.21.15.67:0/2587094617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4c009a410 0x7fb4c00afd80 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:13.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.184+0000 7fb4c64b2640 1 -- 172.21.15.67:0/2587094617 >> 172.21.15.67:0/2587094617 conn(0x7fb4c00b36d0 msgr2=0x7fb4c0071580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:13.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.184+0000 7fb4c64b2640 1 -- 172.21.15.67:0/2587094617 shutdown_connections 2024-03-24T00:43:13.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.184+0000 7fb4c64b2640 1 -- 172.21.15.67:0/2587094617 wait complete. 2024-03-24T00:43:13.196 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:43:13.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.296+0000 7f7b74c0c640 1 Processor -- start 2024-03-24T00:43:13.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.296+0000 7f7b74c0c640 1 -- start start 2024-03-24T00:43:13.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.297+0000 7f7b74c0c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7b7006f140 0x7f7b700ac8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:13.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.297+0000 7f7b74c0c640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7b7006f520 0x7f7b7015ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:13.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.297+0000 7f7b74c0c640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b700b6040 con 0x7f7b7015f810 2024-03-24T00:43:13.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.297+0000 7f7b74c0c640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b700acde0 con 0x7f7b7006f140 2024-03-24T00:43:13.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.297+0000 7f7b74c0c640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b700b5a80 con 0x7f7b7006f520 2024-03-24T00:43:13.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.297+0000 7f7b6ed9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7b7015f810 0x7f7b700aeb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44978/0 (socket says 172.21.15.67:44978) 2024-03-24T00:43:13.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.297+0000 7f7b6ed9d640 1 -- 172.21.15.67:0/3335108448 learned_addr learned my addr 172.21.15.67:0/3335108448 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:43:13.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.297+0000 7f7b6e59c640 1 --2- 172.21.15.67:0/3335108448 >> v2:172.21.15.72:3300/0 conn(0x7f7b7006f140 0x7f7b700ac8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:13.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.297+0000 7f7b67fff640 1 --2- 172.21.15.67:0/3335108448 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7b7006f520 0x7f7b7015ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:13.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.297+0000 7f7b6dd9b640 1 -- 172.21.15.67:0/3335108448 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3320308963 0 0) 0x7f7b68001490 con 0x7f7b7015f810 2024-03-24T00:43:13.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.297+0000 7f7b6dd9b640 1 -- 172.21.15.67:0/3335108448 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b5c003520 con 0x7f7b7015f810 2024-03-24T00:43:13.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.297+0000 7f7b6dd9b640 1 -- 172.21.15.67:0/3335108448 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1906971665 0 0) 0x7f7b5c003520 con 0x7f7b7015f810 2024-03-24T00:43:13.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.297+0000 7f7b6e59c640 1 -- 172.21.15.67:0/3335108448 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7b7006f520 msgr2=0x7f7b7015ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:13.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.298+0000 7f7b6e59c640 1 --2- 172.21.15.67:0/3335108448 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7b7006f520 0x7f7b7015ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:13.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.298+0000 7f7b6e59c640 1 -- 172.21.15.67:0/3335108448 >> v1:172.21.15.67:6789/0 conn(0x7f7b7015f810 legacy=0x7f7b700aeb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:13.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.298+0000 7f7b6e59c640 1 -- 172.21.15.67:0/3335108448 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b700b5d60 con 0x7f7b7006f140 2024-03-24T00:43:13.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.298+0000 7f7b6e59c640 1 --2- 172.21.15.67:0/3335108448 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b7006f140 0x7f7b700ac8a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f7b600097a0 tx=0x7f7b6002b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6d0b7ec2676ff2c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:13.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.299+0000 7f7b6dd9b640 1 -- 172.21.15.67:0/3335108448 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b60037070 con 0x7f7b7006f140 2024-03-24T00:43:13.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.299+0000 7f7b6dd9b640 1 -- 172.21.15.67:0/3335108448 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b60030510 con 0x7f7b7006f140 2024-03-24T00:43:13.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.299+0000 7f7b6dd9b640 1 -- 172.21.15.67:0/3335108448 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b68001bc0 con 0x7f7b7006f140 2024-03-24T00:43:13.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.299+0000 7f7b74c0c640 1 -- 172.21.15.67:0/3335108448 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b7006f140 msgr2=0x7f7b700ac8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:13.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.299+0000 7f7b74c0c640 1 --2- 172.21.15.67:0/3335108448 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b7006f140 0x7f7b700ac8a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f7b600097a0 tx=0x7f7b6002b790 comp rx=0 tx=0).stop 2024-03-24T00:43:13.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.299+0000 7f7b74c0c640 1 -- 172.21.15.67:0/3335108448 shutdown_connections 2024-03-24T00:43:13.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.299+0000 7f7b74c0c640 1 --2- 172.21.15.67:0/3335108448 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7b7006f520 0x7f7b7015ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:13.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.299+0000 7f7b74c0c640 1 --2- 172.21.15.67:0/3335108448 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b7006f140 0x7f7b700ac8a0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:13.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.299+0000 7f7b74c0c640 1 -- 172.21.15.67:0/3335108448 >> 172.21.15.67:0/3335108448 conn(0x7f7b70071ef0 msgr2=0x7f7b70072300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:13.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.299+0000 7f7b74c0c640 1 -- 172.21.15.67:0/3335108448 shutdown_connections 2024-03-24T00:43:13.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.299+0000 7f7b74c0c640 1 -- 172.21.15.67:0/3335108448 wait complete. 2024-03-24T00:43:13.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.300+0000 7f7b74c0c640 1 Processor -- start 2024-03-24T00:43:13.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.300+0000 7f7b74c0c640 1 -- start start 2024-03-24T00:43:13.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.301+0000 7f7b74c0c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b7006f140 0x7f7b700ce8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:13.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.301+0000 7f7b74c0c640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7b7006f520 0x7f7b701e66d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:13.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.301+0000 7f7b74c0c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b7015f810 0x7f7b701e6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:13.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.301+0000 7f7b74c0c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7015f3b0 con 0x7f7b7006f140 2024-03-24T00:43:13.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.301+0000 7f7b74c0c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b700ae660 con 0x7f7b7015f810 2024-03-24T00:43:13.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.301+0000 7f7b74c0c640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7006f980 con 0x7f7b7006f520 2024-03-24T00:43:13.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.301+0000 7f7b6e59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b7006f140 0x7f7b700ce8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:13.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.301+0000 7f7b6e59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b7006f140 0x7f7b700ce8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47386/0 (socket says 172.21.15.67:47386) 2024-03-24T00:43:13.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.301+0000 7f7b6e59c640 1 -- 172.21.15.67:0/251642059 learned_addr learned my addr 172.21.15.67:0/251642059 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:13.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.301+0000 7f7b6ed9d640 1 --2- 172.21.15.67:0/251642059 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b7015f810 0x7f7b701e6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:13.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.301+0000 7f7b67fff640 1 --2- 172.21.15.67:0/251642059 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7b7006f520 0x7f7b701e66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:13.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.302+0000 7f7b6e59c640 1 -- 172.21.15.67:0/251642059 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7b7006f520 msgr2=0x7f7b701e66d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:13.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.302+0000 7f7b6e59c640 1 --2- 172.21.15.67:0/251642059 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7b7006f520 0x7f7b701e66d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:13.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.302+0000 7f7b6e59c640 1 -- 172.21.15.67:0/251642059 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b7015f810 msgr2=0x7f7b701e6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:13.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.302+0000 7f7b6e59c640 1 --2- 172.21.15.67:0/251642059 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b7015f810 0x7f7b701e6c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:13.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.302+0000 7f7b6e59c640 1 -- 172.21.15.67:0/251642059 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b7015de60 con 0x7f7b7006f140 2024-03-24T00:43:13.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.302+0000 7f7b6e59c640 1 --2- 172.21.15.67:0/251642059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b7006f140 0x7f7b700ce8b0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f7b60000c00 tx=0x7f7b600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:13.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.302+0000 7f7b66ffd640 1 -- 172.21.15.67:0/251642059 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b60042070 con 0x7f7b7006f140 2024-03-24T00:43:13.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.302+0000 7f7b74c0c640 1 -- 172.21.15.67:0/251642059 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b700cf850 con 0x7f7b7006f140 2024-03-24T00:43:13.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.302+0000 7f7b66ffd640 1 -- 172.21.15.67:0/251642059 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b6003e870 con 0x7f7b7006f140 2024-03-24T00:43:13.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.302+0000 7f7b66ffd640 1 -- 172.21.15.67:0/251642059 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b60037040 con 0x7f7b7006f140 2024-03-24T00:43:13.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.302+0000 7f7b74c0c640 1 -- 172.21.15.67:0/251642059 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b701e7500 con 0x7f7b7006f140 2024-03-24T00:43:13.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.303+0000 7f7b74c0c640 1 -- 172.21.15.67:0/251642059 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b7004f400 con 0x7f7b7006f140 2024-03-24T00:43:13.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.303+0000 7f7b66ffd640 1 -- 172.21.15.67:0/251642059 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7b60045070 con 0x7f7b7006f140 2024-03-24T00:43:13.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.304+0000 7f7b66ffd640 1 --2- 172.21.15.67:0/251642059 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7b5006bbd0 0x7f7b5006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:13.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.304+0000 7f7b66ffd640 1 -- 172.21.15.67:0/251642059 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7b600ac870 con 0x7f7b7006f140 2024-03-24T00:43:13.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.304+0000 7f7b67fff640 1 --2- 172.21.15.67:0/251642059 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7b5006bbd0 0x7f7b5006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:13.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.305+0000 7f7b67fff640 1 --2- 172.21.15.67:0/251642059 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7b5006bbd0 0x7f7b5006e090 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f7b700cfab0 tx=0x7f7b58009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:13.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.308+0000 7f7b66ffd640 1 -- 172.21.15.67:0/251642059 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7b6007acb0 con 0x7f7b7006f140 2024-03-24T00:43:13.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.490+0000 7f7b74c0c640 1 -- 172.21.15.67:0/251642059 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b7006fc70 con 0x7f7b7006f140 2024-03-24T00:43:13.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.491+0000 7f7b66ffd640 1 -- 172.21.15.67:0/251642059 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+16885 (secure 0 0 0) 0x7f7b6007ed50 con 0x7f7b7006f140 2024-03-24T00:43:13.492 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:43:13.492 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":76,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:43:11.822689+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:43:08.452726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-24T00:43:09.146431+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:43:13.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.493+0000 7f7b74c0c640 1 -- 172.21.15.67:0/251642059 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7b5006bbd0 msgr2=0x7f7b5006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:13.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.493+0000 7f7b74c0c640 1 --2- 172.21.15.67:0/251642059 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7b5006bbd0 0x7f7b5006e090 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f7b700cfab0 tx=0x7f7b58009190 comp rx=0 tx=0).stop 2024-03-24T00:43:13.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.493+0000 7f7b74c0c640 1 -- 172.21.15.67:0/251642059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b7006f140 msgr2=0x7f7b700ce8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:13.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.493+0000 7f7b74c0c640 1 --2- 172.21.15.67:0/251642059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b7006f140 0x7f7b700ce8b0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f7b60000c00 tx=0x7f7b600046c0 comp rx=0 tx=0).stop 2024-03-24T00:43:13.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.493+0000 7f7b74c0c640 1 -- 172.21.15.67:0/251642059 shutdown_connections 2024-03-24T00:43:13.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.493+0000 7f7b74c0c640 1 --2- 172.21.15.67:0/251642059 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7b5006bbd0 0x7f7b5006e090 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:13.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.493+0000 7f7b74c0c640 1 --2- 172.21.15.67:0/251642059 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b7015f810 0x7f7b701e6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:13.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.493+0000 7f7b74c0c640 1 --2- 172.21.15.67:0/251642059 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7b7006f520 0x7f7b701e66d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:13.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.493+0000 7f7b74c0c640 1 --2- 172.21.15.67:0/251642059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b7006f140 0x7f7b700ce8b0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:13.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.493+0000 7f7b74c0c640 1 -- 172.21.15.67:0/251642059 >> 172.21.15.67:0/251642059 conn(0x7f7b70071ef0 msgr2=0x7f7b7015fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:13.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.493+0000 7f7b74c0c640 1 -- 172.21.15.67:0/251642059 shutdown_connections 2024-03-24T00:43:13.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.493+0000 7f7b74c0c640 1 -- 172.21.15.67:0/251642059 wait complete. 2024-03-24T00:43:13.505 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-24T00:43:13.505 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:43:13.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.606+0000 7fa2bad44640 1 Processor -- start 2024-03-24T00:43:13.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.606+0000 7fa2bad44640 1 -- start start 2024-03-24T00:43:13.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.607+0000 7fa2bad44640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa2b415dec0 0x7fa2b40ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:13.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.607+0000 7fa2bad44640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2b40cf640 con 0x7fa2b415e2a0 2024-03-24T00:43:13.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.607+0000 7fa2bad44640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2b40cf080 con 0x7fa2b40d0ef0 2024-03-24T00:43:13.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.607+0000 7fa2bad44640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2b40cee30 con 0x7fa2b415dec0 2024-03-24T00:43:13.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.607+0000 7fa2b8ae0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa2b415dec0 0x7fa2b40ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:13.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.607+0000 7fa2b3fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa2b415e2a0 0x7fa2b40d07e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37480/0 (socket says 172.21.15.67:37480) 2024-03-24T00:43:13.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.607+0000 7fa2b3fff640 1 -- 172.21.15.67:0/2002328313 learned_addr learned my addr 172.21.15.67:0/2002328313 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:43:13.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.607+0000 7fa2b8ae0640 1 -- 172.21.15.67:0/2002328313 >> v1:172.21.15.72:6789/0 conn(0x7fa2b40d0ef0 legacy=0x7fa2b40d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:13.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.607+0000 7fa2b8ae0640 1 -- 172.21.15.67:0/2002328313 >> v1:172.21.15.67:6789/0 conn(0x7fa2b415e2a0 legacy=0x7fa2b40d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:13.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.607+0000 7fa2b8ae0640 1 -- 172.21.15.67:0/2002328313 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2b40cf360 con 0x7fa2b415dec0 2024-03-24T00:43:13.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.608+0000 7fa2b8ae0640 1 --2- 172.21.15.67:0/2002328313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2b415dec0 0x7fa2b40ce630 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fa2b40cefb0 tx=0x7fa2a802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79fee063b081ce4a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:13.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.608+0000 7fa2b37fe640 1 -- 172.21.15.67:0/2002328313 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a8035020 con 0x7fa2b415dec0 2024-03-24T00:43:13.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.608+0000 7fa2b37fe640 1 -- 172.21.15.67:0/2002328313 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2a8031830 con 0x7fa2b415dec0 2024-03-24T00:43:13.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.608+0000 7fa2b37fe640 1 -- 172.21.15.67:0/2002328313 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a8031c00 con 0x7fa2b415dec0 2024-03-24T00:43:13.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.609+0000 7fa2bad44640 1 -- 172.21.15.67:0/2002328313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2b415dec0 msgr2=0x7fa2b40ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:13.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.609+0000 7fa2bad44640 1 --2- 172.21.15.67:0/2002328313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2b415dec0 0x7fa2b40ce630 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fa2b40cefb0 tx=0x7fa2a802b8d0 comp rx=0 tx=0).stop 2024-03-24T00:43:13.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.609+0000 7fa2bad44640 1 -- 172.21.15.67:0/2002328313 shutdown_connections 2024-03-24T00:43:13.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.609+0000 7fa2bad44640 1 --2- 172.21.15.67:0/2002328313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2b415dec0 0x7fa2b40ce630 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:13.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.609+0000 7fa2bad44640 1 -- 172.21.15.67:0/2002328313 >> 172.21.15.67:0/2002328313 conn(0x7fa2b406c810 msgr2=0x7fa2b406cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:13.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.609+0000 7fa2bad44640 1 -- 172.21.15.67:0/2002328313 shutdown_connections 2024-03-24T00:43:13.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.609+0000 7fa2bad44640 1 -- 172.21.15.67:0/2002328313 wait complete. 2024-03-24T00:43:13.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.610+0000 7fa2bad44640 1 Processor -- start 2024-03-24T00:43:13.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.610+0000 7fa2bad44640 1 -- start start 2024-03-24T00:43:13.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.611+0000 7fa2bad44640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2b40d0ef0 0x7fa2b40b3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:13.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.611+0000 7fa2bad44640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2b415dec0 0x7fa2b40d2820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:13.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.611+0000 7fa2bad44640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa2b415e2a0 0x7fa2b41e8250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:13.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.611+0000 7fa2bad44640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b4162a40 con 0x7fa2b415dec0 2024-03-24T00:43:13.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.611+0000 7fa2bad44640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b41628c0 con 0x7fa2b40d0ef0 2024-03-24T00:43:13.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.611+0000 7fa2bad44640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b40d2120 con 0x7fa2b415e2a0 2024-03-24T00:43:13.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.611+0000 7fa2b3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2b415dec0 0x7fa2b40d2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:13.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.611+0000 7fa2b3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2b415dec0 0x7fa2b40d2820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50314/0 (socket says 172.21.15.67:50314) 2024-03-24T00:43:13.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.611+0000 7fa2b3fff640 1 -- 172.21.15.67:0/1382216705 learned_addr learned my addr 172.21.15.67:0/1382216705 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:13.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.611+0000 7fa2b3fff640 1 -- 172.21.15.67:0/1382216705 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa2b415e2a0 msgr2=0x7fa2b41e8250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:13.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.611+0000 7fa2b3fff640 1 --2- 172.21.15.67:0/1382216705 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa2b415e2a0 0x7fa2b41e8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:13.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.611+0000 7fa2b8ae0640 1 --2- 172.21.15.67:0/1382216705 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2b40d0ef0 0x7fa2b40b3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:13.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.611+0000 7fa2b3fff640 1 -- 172.21.15.67:0/1382216705 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2b40d0ef0 msgr2=0x7fa2b40b3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:13.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.611+0000 7fa2b3fff640 1 --2- 172.21.15.67:0/1382216705 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2b40d0ef0 0x7fa2b40b3810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:13.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.611+0000 7fa2b3fff640 1 -- 172.21.15.67:0/1382216705 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2b40cbbf0 con 0x7fa2b415dec0 2024-03-24T00:43:13.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.611+0000 7fa2b3fff640 1 --2- 172.21.15.67:0/1382216705 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2b415dec0 0x7fa2b40d2820 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fa2a000c3f0 tx=0x7fa2a000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:13.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.611+0000 7fa2b17fa640 1 -- 172.21.15.67:0/1382216705 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a0017070 con 0x7fa2b415dec0 2024-03-24T00:43:13.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.611+0000 7fa2b17fa640 1 -- 172.21.15.67:0/1382216705 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2a0005ce0 con 0x7fa2b415dec0 2024-03-24T00:43:13.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.611+0000 7fa2b17fa640 1 -- 172.21.15.67:0/1382216705 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a00072d0 con 0x7fa2b415dec0 2024-03-24T00:43:13.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.611+0000 7fa2bad44640 1 -- 172.21.15.67:0/1382216705 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2b40cbed0 con 0x7fa2b415dec0 2024-03-24T00:43:13.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.611+0000 7fa2bad44640 1 -- 172.21.15.67:0/1382216705 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2b40cc490 con 0x7fa2b415dec0 2024-03-24T00:43:13.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.612+0000 7fa2bad44640 1 -- 172.21.15.67:0/1382216705 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2b404f400 con 0x7fa2b415dec0 2024-03-24T00:43:13.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.613+0000 7fa2b17fa640 1 -- 172.21.15.67:0/1382216705 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa2a001a070 con 0x7fa2b415dec0 2024-03-24T00:43:13.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.613+0000 7fa2b17fa640 1 --2- 172.21.15.67:0/1382216705 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa29806bbd0 0x7fa29806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:13.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.613+0000 7fa2b17fa640 1 -- 172.21.15.67:0/1382216705 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa2a0089ed0 con 0x7fa2b415dec0 2024-03-24T00:43:13.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.613+0000 7fa2b8ae0640 1 --2- 172.21.15.67:0/1382216705 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa29806bbd0 0x7fa29806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:13.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.614+0000 7fa2b8ae0640 1 --2- 172.21.15.67:0/1382216705 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa29806bbd0 0x7fa29806e090 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fa2a8036000 tx=0x7fa2a8008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:13.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.618+0000 7fa2b17fa640 1 -- 172.21.15.67:0/1382216705 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa2b40cf640 con 0x7fa2b415dec0 2024-03-24T00:43:13.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.796+0000 7fa2bad44640 1 -- 172.21.15.67:0/1382216705 --> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa2b40d1aa0 con 0x7fa29806bbd0 2024-03-24T00:43:13.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.802+0000 7fa2b17fa640 1 -- 172.21.15.67:0/1382216705 <== mgr.4105 v2:172.21.15.72:6832/2768292274 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240144 (secure 0 0 0) 0x7fa2b40d1aa0 con 0x7fa29806bbd0 2024-03-24T00:43:13.805 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:43:13.807 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-24T00:43:13.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.805+0000 7fa2bad44640 1 -- 172.21.15.67:0/1382216705 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa29806bbd0 msgr2=0x7fa29806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:13.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.805+0000 7fa2bad44640 1 --2- 172.21.15.67:0/1382216705 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa29806bbd0 0x7fa29806e090 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fa2a8036000 tx=0x7fa2a8008030 comp rx=0 tx=0).stop 2024-03-24T00:43:13.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.805+0000 7fa2bad44640 1 -- 172.21.15.67:0/1382216705 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2b415dec0 msgr2=0x7fa2b40d2820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:13.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.805+0000 7fa2bad44640 1 --2- 172.21.15.67:0/1382216705 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2b415dec0 0x7fa2b40d2820 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fa2a000c3f0 tx=0x7fa2a000c8b0 comp rx=0 tx=0).stop 2024-03-24T00:43:13.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.805+0000 7fa2bad44640 1 -- 172.21.15.67:0/1382216705 shutdown_connections 2024-03-24T00:43:13.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.805+0000 7fa2bad44640 1 --2- 172.21.15.67:0/1382216705 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa29806bbd0 0x7fa29806e090 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:13.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.805+0000 7fa2bad44640 1 --2- 172.21.15.67:0/1382216705 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa2b415e2a0 0x7fa2b41e8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:13.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.805+0000 7fa2bad44640 1 --2- 172.21.15.67:0/1382216705 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2b415dec0 0x7fa2b40d2820 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:13.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.805+0000 7fa2bad44640 1 --2- 172.21.15.67:0/1382216705 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2b40d0ef0 0x7fa2b40b3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:13.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.805+0000 7fa2bad44640 1 -- 172.21.15.67:0/1382216705 >> 172.21.15.67:0/1382216705 conn(0x7fa2b406c810 msgr2=0x7fa2b415f1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:13.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.805+0000 7fa2bad44640 1 -- 172.21.15.67:0/1382216705 shutdown_connections 2024-03-24T00:43:13.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.805+0000 7fa2bad44640 1 -- 172.21.15.67:0/1382216705 wait complete. 2024-03-24T00:43:13.817 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":206,"stamp":"2024-03-24T00:43:13.382294+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":8,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":228388,"kb_used_data":8052,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754746332,"statfs":{"total":773094113280,"available":772860243968,"internally_reserved":0,"allocated":8245248,"data_stored":4373173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":225496625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.577202"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.806432+0000","last_change":"2024-03-24T00:43:08.806432+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.806432+0000","last_undegraded":"2024-03-24T00:43:08.806432+0000","last_fullsized":"2024-03-24T00:43:08.806432+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.809259+0000","last_change":"2024-03-24T00:43:08.809259+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.809259+0000","last_undegraded":"2024-03-24T00:43:08.809259+0000","last_fullsized":"2024-03-24T00:43:08.809259+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.809761+0000","last_change":"2024-03-24T00:43:08.809761+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.809761+0000","last_undegraded":"2024-03-24T00:43:08.809761+0000","last_fullsized":"2024-03-24T00:43:08.809761+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.816853+0000","last_change":"2024-03-24T00:43:08.816853+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.816853+0000","last_undegraded":"2024-03-24T00:43:08.816853+0000","last_fullsized":"2024-03-24T00:43:08.816853+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.809304+0000","last_change":"2024-03-24T00:43:08.809304+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.809304+0000","last_undegraded":"2024-03-24T00:43:08.809304+0000","last_fullsized":"2024-03-24T00:43:08.809304+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.806953+0000","last_change":"2024-03-24T00:43:08.806953+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.806953+0000","last_undegraded":"2024-03-24T00:43:08.806953+0000","last_fullsized":"2024-03-24T00:43:08.806953+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.812355+0000","last_change":"2024-03-24T00:43:08.812355+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.812355+0000","last_undegraded":"2024-03-24T00:43:08.812355+0000","last_fullsized":"2024-03-24T00:43:08.812355+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.806378+0000","last_change":"2024-03-24T00:43:08.806378+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.806378+0000","last_undegraded":"2024-03-24T00:43:08.806378+0000","last_fullsized":"2024-03-24T00:43:08.806378+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.810005+0000","last_change":"2024-03-24T00:43:08.810005+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.810005+0000","last_undegraded":"2024-03-24T00:43:08.810005+0000","last_fullsized":"2024-03-24T00:43:08.810005+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.806947+0000","last_change":"2024-03-24T00:43:08.806947+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.806947+0000","last_undegraded":"2024-03-24T00:43:08.806947+0000","last_fullsized":"2024-03-24T00:43:08.806947+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.806868+0000","last_change":"2024-03-24T00:43:08.806868+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.806868+0000","last_undegraded":"2024-03-24T00:43:08.806868+0000","last_fullsized":"2024-03-24T00:43:08.806868+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.816394+0000","last_change":"2024-03-24T00:43:08.816394+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.816394+0000","last_undegraded":"2024-03-24T00:43:08.816394+0000","last_fullsized":"2024-03-24T00:43:08.816394+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.806953+0000","last_change":"2024-03-24T00:43:08.806953+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.806953+0000","last_undegraded":"2024-03-24T00:43:08.806953+0000","last_fullsized":"2024-03-24T00:43:08.806953+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.809389+0000","last_change":"2024-03-24T00:43:08.809389+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.809389+0000","last_undegraded":"2024-03-24T00:43:08.809389+0000","last_fullsized":"2024-03-24T00:43:08.809389+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.806381+0000","last_change":"2024-03-24T00:43:08.806381+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.806381+0000","last_undegraded":"2024-03-24T00:43:08.806381+0000","last_fullsized":"2024-03-24T00:43:08.806381+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.812505+0000","last_change":"2024-03-24T00:43:08.812505+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.812505+0000","last_undegraded":"2024-03-24T00:43:08.812505+0000","last_fullsized":"2024-03-24T00:43:08.812505+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.809567+0000","last_change":"2024-03-24T00:43:08.809567+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.809567+0000","last_undegraded":"2024-03-24T00:43:08.809567+0000","last_fullsized":"2024-03-24T00:43:08.809567+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.809309+0000","last_change":"2024-03-24T00:43:08.809309+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.809309+0000","last_undegraded":"2024-03-24T00:43:08.809309+0000","last_fullsized":"2024-03-24T00:43:08.809309+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.806390+0000","last_change":"2024-03-24T00:43:08.806390+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.806390+0000","last_undegraded":"2024-03-24T00:43:08.806390+0000","last_fullsized":"2024-03-24T00:43:08.806390+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.807736+0000","last_change":"2024-03-24T00:43:08.807736+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.807736+0000","last_undegraded":"2024-03-24T00:43:08.807736+0000","last_fullsized":"2024-03-24T00:43:08.807736+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.811079+0000","last_change":"2024-03-24T00:43:08.811079+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.811079+0000","last_undegraded":"2024-03-24T00:43:08.811079+0000","last_fullsized":"2024-03-24T00:43:08.811079+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.810648+0000","last_change":"2024-03-24T00:43:08.810648+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.810648+0000","last_undegraded":"2024-03-24T00:43:08.810648+0000","last_fullsized":"2024-03-24T00:43:08.810648+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.810555+0000","last_change":"2024-03-24T00:43:08.810555+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.810555+0000","last_undegraded":"2024-03-24T00:43:08.810555+0000","last_fullsized":"2024-03-24T00:43:08.810555+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.806959+0000","last_change":"2024-03-24T00:43:08.806959+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.806959+0000","last_undegraded":"2024-03-24T00:43:08.806959+0000","last_fullsized":"2024-03-24T00:43:08.806959+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.812359+0000","last_change":"2024-03-24T00:43:08.812359+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.812359+0000","last_undegraded":"2024-03-24T00:43:08.812359+0000","last_fullsized":"2024-03-24T00:43:08.812359+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.809204+0000","last_change":"2024-03-24T00:43:08.809204+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.809204+0000","last_undegraded":"2024-03-24T00:43:08.809204+0000","last_fullsized":"2024-03-24T00:43:08.809204+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.809754+0000","last_change":"2024-03-24T00:43:08.809754+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.809754+0000","last_undegraded":"2024-03-24T00:43:08.809754+0000","last_fullsized":"2024-03-24T00:43:08.809754+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.816382+0000","last_change":"2024-03-24T00:43:08.816382+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.816382+0000","last_undegraded":"2024-03-24T00:43:08.816382+0000","last_fullsized":"2024-03-24T00:43:08.816382+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.806382+0000","last_change":"2024-03-24T00:43:08.806382+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.806382+0000","last_undegraded":"2024-03-24T00:43:08.806382+0000","last_fullsized":"2024-03-24T00:43:08.806382+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.815930+0000","last_change":"2024-03-24T00:43:08.815930+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.815930+0000","last_undegraded":"2024-03-24T00:43:08.815930+0000","last_fullsized":"2024-03-24T00:43:08.815930+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.807731+0000","last_change":"2024-03-24T00:43:08.807731+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.807731+0000","last_undegraded":"2024-03-24T00:43:08.807731+0000","last_fullsized":"2024-03-24T00:43:08.807731+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.810712+0000","last_change":"2024-03-24T00:43:08.810712+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.810712+0000","last_undegraded":"2024-03-24T00:43:08.810712+0000","last_fullsized":"2024-03-24T00:43:08.810712+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":268,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-24T00:43:08.810650+0000","last_change":"2024-03-24T00:38:40.953871+0000","last_active":"2024-03-24T00:43:08.810650+0000","last_peered":"2024-03-24T00:43:08.810650+0000","last_clean":"2024-03-24T00:43:08.810650+0000","last_became_active":"2024-03-24T00:38:40.952926+0000","last_became_peered":"2024-03-24T00:38:40.952926+0000","last_unstale":"2024-03-24T00:43:08.810650+0000","last_undegraded":"2024-03-24T00:43:08.810650+0000","last_fullsized":"2024-03-24T00:43:08.810650+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_clean_scrub_stamp":"2024-03-24T00:38:39.722194+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:05.397257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28460,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343380,"statfs":{"total":96636764160,"available":96607621120,"internally_reserved":0,"allocated":1474560,"data_stored":988474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27892,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343948,"statfs":{"total":96636764160,"available":96608202752,"internally_reserved":0,"allocated":892928,"data_stored":404315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27880,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343960,"statfs":{"total":96636764160,"available":96608215040,"internally_reserved":0,"allocated":880640,"data_stored":398106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574905,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27872,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343968,"statfs":{"total":96636764160,"available":96608223232,"internally_reserved":0,"allocated":868352,"data_stored":392587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574905,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27892,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343948,"statfs":{"total":96636764160,"available":96608202752,"internally_reserved":0,"allocated":892928,"data_stored":404315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574906,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27896,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343944,"statfs":{"total":96636764160,"available":96608198656,"internally_reserved":0,"allocated":892928,"data_stored":404315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27880,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343960,"statfs":{"total":96636764160,"available":96608215040,"internally_reserved":0,"allocated":880640,"data_stored":398106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574905,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32616,"kb_used_data":1428,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339224,"statfs":{"total":96636764160,"available":96603365376,"internally_reserved":0,"allocated":1462272,"data_stored":982955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:43:13.820 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:43:13.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.958+0000 7f81b4362640 1 Processor -- start 2024-03-24T00:43:13.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.958+0000 7f81b4362640 1 -- start start 2024-03-24T00:43:13.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.959+0000 7f81b4362640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f81ac15e2a0 0x7f81ac0ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:13.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.959+0000 7f81b4362640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81ac0cf640 con 0x7f81ac0d0ef0 2024-03-24T00:43:13.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.959+0000 7f81b4362640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81ac0cf080 con 0x7f81ac15dec0 2024-03-24T00:43:13.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.959+0000 7f81b4362640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81ac0cee30 con 0x7f81ac15e2a0 2024-03-24T00:43:13.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.959+0000 7f81b28ff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f81ac0d0ef0 0x7f81ac0d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37496/0 (socket says 172.21.15.67:37496) 2024-03-24T00:43:13.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.959+0000 7f81b28ff640 1 -- 172.21.15.67:0/4256846113 learned_addr learned my addr 172.21.15.67:0/4256846113 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:43:13.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.959+0000 7f81b18fd640 1 --2- 172.21.15.67:0/4256846113 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f81ac15e2a0 0x7f81ac0ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:13.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.959+0000 7f81b10fc640 1 -- 172.21.15.67:0/4256846113 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 998468068 0 0) 0x7f81a4001490 con 0x7f81ac0d0ef0 2024-03-24T00:43:13.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.959+0000 7f81b10fc640 1 -- 172.21.15.67:0/4256846113 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8198003520 con 0x7f81ac0d0ef0 2024-03-24T00:43:13.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.959+0000 7f81b10fc640 1 -- 172.21.15.67:0/4256846113 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 290311970 0 0) 0x7f81ac0cf080 con 0x7f81ac15dec0 2024-03-24T00:43:13.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.959+0000 7f81b10fc640 1 -- 172.21.15.67:0/4256846113 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81a4001490 con 0x7f81ac15dec0 2024-03-24T00:43:13.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.959+0000 7f81b18fd640 1 -- 172.21.15.67:0/4256846113 >> v1:172.21.15.72:6789/0 conn(0x7f81ac15dec0 legacy=0x7f81ac0d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:13.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.960+0000 7f81b18fd640 1 -- 172.21.15.67:0/4256846113 >> v1:172.21.15.67:6789/0 conn(0x7f81ac0d0ef0 legacy=0x7f81ac0d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:13.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.960+0000 7f81b18fd640 1 -- 172.21.15.67:0/4256846113 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81ac0cf360 con 0x7f81ac15e2a0 2024-03-24T00:43:13.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.960+0000 7f81b18fd640 1 --2- 172.21.15.67:0/4256846113 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f81ac15e2a0 0x7f81ac0ce630 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f819c0097d0 tx=0x7f819c02b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4d21e59ca48a147 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:13.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.961+0000 7f81b10fc640 1 -- 172.21.15.67:0/4256846113 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f819c035020 con 0x7f81ac15e2a0 2024-03-24T00:43:13.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.961+0000 7f81b10fc640 1 -- 172.21.15.67:0/4256846113 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f819c031800 con 0x7f81ac15e2a0 2024-03-24T00:43:13.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.961+0000 7f81b10fc640 1 -- 172.21.15.67:0/4256846113 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f819c031bd0 con 0x7f81ac15e2a0 2024-03-24T00:43:13.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.961+0000 7f81b4362640 1 -- 172.21.15.67:0/4256846113 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f81ac15e2a0 msgr2=0x7f81ac0ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:13.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.961+0000 7f81b4362640 1 --2- 172.21.15.67:0/4256846113 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f81ac15e2a0 0x7f81ac0ce630 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f819c0097d0 tx=0x7f819c02b8a0 comp rx=0 tx=0).stop 2024-03-24T00:43:13.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.961+0000 7f81b4362640 1 -- 172.21.15.67:0/4256846113 shutdown_connections 2024-03-24T00:43:13.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.961+0000 7f81b4362640 1 --2- 172.21.15.67:0/4256846113 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f81ac15e2a0 0x7f81ac0ce630 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:13.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.961+0000 7f81b4362640 1 -- 172.21.15.67:0/4256846113 >> 172.21.15.67:0/4256846113 conn(0x7f81ac06c810 msgr2=0x7f81ac06cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:13.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.961+0000 7f81b4362640 1 -- 172.21.15.67:0/4256846113 shutdown_connections 2024-03-24T00:43:13.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.961+0000 7f81b4362640 1 -- 172.21.15.67:0/4256846113 wait complete. 2024-03-24T00:43:13.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.962+0000 7f81b4362640 1 Processor -- start 2024-03-24T00:43:13.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.962+0000 7f81b4362640 1 -- start start 2024-03-24T00:43:13.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.963+0000 7f81b4362640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f81ac0d0ef0 0x7f81ac1592a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:13.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.963+0000 7f81b4362640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81ac15dec0 0x7f81ac0b5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:13.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.963+0000 7f81b4362640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f81ac15e2a0 0x7f81ac0d24b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:13.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.963+0000 7f81b4362640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81ac162a40 con 0x7f81ac15dec0 2024-03-24T00:43:13.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.963+0000 7f81b4362640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81ac1628c0 con 0x7f81ac15e2a0 2024-03-24T00:43:13.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.963+0000 7f81b4362640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81ac0d2120 con 0x7f81ac0d0ef0 2024-03-24T00:43:13.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.963+0000 7f81b18fd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81ac15dec0 0x7f81ac0b5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:13.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.963+0000 7f81b18fd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81ac15dec0 0x7f81ac0b5fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50326/0 (socket says 172.21.15.67:50326) 2024-03-24T00:43:13.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.963+0000 7f81b18fd640 1 -- 172.21.15.67:0/888972314 learned_addr learned my addr 172.21.15.67:0/888972314 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:13.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.963+0000 7f81b28ff640 1 --2- 172.21.15.67:0/888972314 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f81ac15e2a0 0x7f81ac0d24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:13.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.963+0000 7f81b20fe640 1 --2- 172.21.15.67:0/888972314 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f81ac0d0ef0 0x7f81ac1592a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:13.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.963+0000 7f81b18fd640 1 -- 172.21.15.67:0/888972314 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f81ac0d0ef0 msgr2=0x7f81ac1592a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:13.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.963+0000 7f81b18fd640 1 --2- 172.21.15.67:0/888972314 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f81ac0d0ef0 0x7f81ac1592a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:13.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.963+0000 7f81b18fd640 1 -- 172.21.15.67:0/888972314 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f81ac15e2a0 msgr2=0x7f81ac0d24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:13.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.963+0000 7f81b18fd640 1 --2- 172.21.15.67:0/888972314 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f81ac15e2a0 0x7f81ac0d24b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:13.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.963+0000 7f81b18fd640 1 -- 172.21.15.67:0/888972314 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81ac1597e0 con 0x7f81ac15dec0 2024-03-24T00:43:13.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.964+0000 7f81b18fd640 1 --2- 172.21.15.67:0/888972314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81ac15dec0 0x7f81ac0b5fd0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f819c0098f0 tx=0x7f819c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:13.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.964+0000 7f81a2ffd640 1 -- 172.21.15.67:0/888972314 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f819c041070 con 0x7f81ac15dec0 2024-03-24T00:43:13.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.964+0000 7f81a2ffd640 1 -- 172.21.15.67:0/888972314 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f819c031e30 con 0x7f81ac15dec0 2024-03-24T00:43:13.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.964+0000 7f81b4362640 1 -- 172.21.15.67:0/888972314 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81ac1f7d10 con 0x7f81ac15dec0 2024-03-24T00:43:13.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.964+0000 7f81a2ffd640 1 -- 172.21.15.67:0/888972314 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f819c035040 con 0x7f81ac15dec0 2024-03-24T00:43:13.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.964+0000 7f81b4362640 1 -- 172.21.15.67:0/888972314 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81ac0b54f0 con 0x7f81ac15dec0 2024-03-24T00:43:13.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.965+0000 7f81b4362640 1 -- 172.21.15.67:0/888972314 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81ac04f400 con 0x7f81ac15dec0 2024-03-24T00:43:13.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.965+0000 7f81a2ffd640 1 -- 172.21.15.67:0/888972314 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f819c04c060 con 0x7f81ac15dec0 2024-03-24T00:43:13.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.966+0000 7f81a2ffd640 1 --2- 172.21.15.67:0/888972314 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f819006bbd0 0x7f819006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:13.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.966+0000 7f81a2ffd640 1 -- 172.21.15.67:0/888972314 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f819c0ac720 con 0x7f81ac15dec0 2024-03-24T00:43:13.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.966+0000 7f81b20fe640 1 --2- 172.21.15.67:0/888972314 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f819006bbd0 0x7f819006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:13.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.966+0000 7f81b20fe640 1 --2- 172.21.15.67:0/888972314 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f819006bbd0 0x7f819006e090 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f81ac0d30b0 tx=0x7f81a8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:13.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:13.970+0000 7f81a2ffd640 1 -- 172.21.15.67:0/888972314 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f819c07b000 con 0x7f81ac15dec0 2024-03-24T00:43:14.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.160+0000 7f81b4362640 1 -- 172.21.15.67:0/888972314 --> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f81ac0d1aa0 con 0x7f819006bbd0 2024-03-24T00:43:14.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.165+0000 7f81a2ffd640 1 -- 172.21.15.67:0/888972314 <== mgr.4105 v2:172.21.15.72:6832/2768292274 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240144 (secure 0 0 0) 0x7f81ac0d1aa0 con 0x7f819006bbd0 2024-03-24T00:43:14.168 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:43:14.169 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-24T00:43:14.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.168+0000 7f81b4362640 1 -- 172.21.15.67:0/888972314 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f819006bbd0 msgr2=0x7f819006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:14.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.168+0000 7f81b4362640 1 --2- 172.21.15.67:0/888972314 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f819006bbd0 0x7f819006e090 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f81ac0d30b0 tx=0x7f81a8009040 comp rx=0 tx=0).stop 2024-03-24T00:43:14.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.168+0000 7f81b4362640 1 -- 172.21.15.67:0/888972314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81ac15dec0 msgr2=0x7f81ac0b5fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:14.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.168+0000 7f81b4362640 1 --2- 172.21.15.67:0/888972314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81ac15dec0 0x7f81ac0b5fd0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f819c0098f0 tx=0x7f819c0046c0 comp rx=0 tx=0).stop 2024-03-24T00:43:14.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.168+0000 7f81b4362640 1 -- 172.21.15.67:0/888972314 shutdown_connections 2024-03-24T00:43:14.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.169+0000 7f81b4362640 1 --2- 172.21.15.67:0/888972314 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f819006bbd0 0x7f819006e090 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:14.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.169+0000 7f81b4362640 1 --2- 172.21.15.67:0/888972314 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f81ac15e2a0 0x7f81ac0d24b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:14.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.169+0000 7f81b4362640 1 --2- 172.21.15.67:0/888972314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81ac15dec0 0x7f81ac0b5fd0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:14.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.169+0000 7f81b4362640 1 --2- 172.21.15.67:0/888972314 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f81ac0d0ef0 0x7f81ac1592a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:14.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.169+0000 7f81b4362640 1 -- 172.21.15.67:0/888972314 >> 172.21.15.67:0/888972314 conn(0x7f81ac06c810 msgr2=0x7f81ac0ca010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:14.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.169+0000 7f81b4362640 1 -- 172.21.15.67:0/888972314 shutdown_connections 2024-03-24T00:43:14.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.169+0000 7f81b4362640 1 -- 172.21.15.67:0/888972314 wait complete. 2024-03-24T00:43:14.180 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":206,"stamp":"2024-03-24T00:43:13.382294+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":8,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":228388,"kb_used_data":8052,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754746332,"statfs":{"total":773094113280,"available":772860243968,"internally_reserved":0,"allocated":8245248,"data_stored":4373173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":225496625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.577202"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.806432+0000","last_change":"2024-03-24T00:43:08.806432+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.806432+0000","last_undegraded":"2024-03-24T00:43:08.806432+0000","last_fullsized":"2024-03-24T00:43:08.806432+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.809259+0000","last_change":"2024-03-24T00:43:08.809259+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.809259+0000","last_undegraded":"2024-03-24T00:43:08.809259+0000","last_fullsized":"2024-03-24T00:43:08.809259+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.809761+0000","last_change":"2024-03-24T00:43:08.809761+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.809761+0000","last_undegraded":"2024-03-24T00:43:08.809761+0000","last_fullsized":"2024-03-24T00:43:08.809761+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.816853+0000","last_change":"2024-03-24T00:43:08.816853+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.816853+0000","last_undegraded":"2024-03-24T00:43:08.816853+0000","last_fullsized":"2024-03-24T00:43:08.816853+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.809304+0000","last_change":"2024-03-24T00:43:08.809304+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.809304+0000","last_undegraded":"2024-03-24T00:43:08.809304+0000","last_fullsized":"2024-03-24T00:43:08.809304+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.806953+0000","last_change":"2024-03-24T00:43:08.806953+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.806953+0000","last_undegraded":"2024-03-24T00:43:08.806953+0000","last_fullsized":"2024-03-24T00:43:08.806953+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.812355+0000","last_change":"2024-03-24T00:43:08.812355+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.812355+0000","last_undegraded":"2024-03-24T00:43:08.812355+0000","last_fullsized":"2024-03-24T00:43:08.812355+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.806378+0000","last_change":"2024-03-24T00:43:08.806378+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.806378+0000","last_undegraded":"2024-03-24T00:43:08.806378+0000","last_fullsized":"2024-03-24T00:43:08.806378+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.810005+0000","last_change":"2024-03-24T00:43:08.810005+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.810005+0000","last_undegraded":"2024-03-24T00:43:08.810005+0000","last_fullsized":"2024-03-24T00:43:08.810005+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.806947+0000","last_change":"2024-03-24T00:43:08.806947+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.806947+0000","last_undegraded":"2024-03-24T00:43:08.806947+0000","last_fullsized":"2024-03-24T00:43:08.806947+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.806868+0000","last_change":"2024-03-24T00:43:08.806868+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.806868+0000","last_undegraded":"2024-03-24T00:43:08.806868+0000","last_fullsized":"2024-03-24T00:43:08.806868+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.816394+0000","last_change":"2024-03-24T00:43:08.816394+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.816394+0000","last_undegraded":"2024-03-24T00:43:08.816394+0000","last_fullsized":"2024-03-24T00:43:08.816394+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.806953+0000","last_change":"2024-03-24T00:43:08.806953+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.806953+0000","last_undegraded":"2024-03-24T00:43:08.806953+0000","last_fullsized":"2024-03-24T00:43:08.806953+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.809389+0000","last_change":"2024-03-24T00:43:08.809389+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.809389+0000","last_undegraded":"2024-03-24T00:43:08.809389+0000","last_fullsized":"2024-03-24T00:43:08.809389+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.806381+0000","last_change":"2024-03-24T00:43:08.806381+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.806381+0000","last_undegraded":"2024-03-24T00:43:08.806381+0000","last_fullsized":"2024-03-24T00:43:08.806381+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.812505+0000","last_change":"2024-03-24T00:43:08.812505+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.812505+0000","last_undegraded":"2024-03-24T00:43:08.812505+0000","last_fullsized":"2024-03-24T00:43:08.812505+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.809567+0000","last_change":"2024-03-24T00:43:08.809567+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.809567+0000","last_undegraded":"2024-03-24T00:43:08.809567+0000","last_fullsized":"2024-03-24T00:43:08.809567+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.809309+0000","last_change":"2024-03-24T00:43:08.809309+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.809309+0000","last_undegraded":"2024-03-24T00:43:08.809309+0000","last_fullsized":"2024-03-24T00:43:08.809309+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.806390+0000","last_change":"2024-03-24T00:43:08.806390+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.806390+0000","last_undegraded":"2024-03-24T00:43:08.806390+0000","last_fullsized":"2024-03-24T00:43:08.806390+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.807736+0000","last_change":"2024-03-24T00:43:08.807736+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.807736+0000","last_undegraded":"2024-03-24T00:43:08.807736+0000","last_fullsized":"2024-03-24T00:43:08.807736+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.811079+0000","last_change":"2024-03-24T00:43:08.811079+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.811079+0000","last_undegraded":"2024-03-24T00:43:08.811079+0000","last_fullsized":"2024-03-24T00:43:08.811079+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.810648+0000","last_change":"2024-03-24T00:43:08.810648+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.810648+0000","last_undegraded":"2024-03-24T00:43:08.810648+0000","last_fullsized":"2024-03-24T00:43:08.810648+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.810555+0000","last_change":"2024-03-24T00:43:08.810555+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.810555+0000","last_undegraded":"2024-03-24T00:43:08.810555+0000","last_fullsized":"2024-03-24T00:43:08.810555+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.806959+0000","last_change":"2024-03-24T00:43:08.806959+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.806959+0000","last_undegraded":"2024-03-24T00:43:08.806959+0000","last_fullsized":"2024-03-24T00:43:08.806959+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.812359+0000","last_change":"2024-03-24T00:43:08.812359+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.812359+0000","last_undegraded":"2024-03-24T00:43:08.812359+0000","last_fullsized":"2024-03-24T00:43:08.812359+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.809204+0000","last_change":"2024-03-24T00:43:08.809204+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.809204+0000","last_undegraded":"2024-03-24T00:43:08.809204+0000","last_fullsized":"2024-03-24T00:43:08.809204+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.809754+0000","last_change":"2024-03-24T00:43:08.809754+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.809754+0000","last_undegraded":"2024-03-24T00:43:08.809754+0000","last_fullsized":"2024-03-24T00:43:08.809754+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.816382+0000","last_change":"2024-03-24T00:43:08.816382+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.816382+0000","last_undegraded":"2024-03-24T00:43:08.816382+0000","last_fullsized":"2024-03-24T00:43:08.816382+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.806382+0000","last_change":"2024-03-24T00:43:08.806382+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.806382+0000","last_undegraded":"2024-03-24T00:43:08.806382+0000","last_fullsized":"2024-03-24T00:43:08.806382+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.815930+0000","last_change":"2024-03-24T00:43:08.815930+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.815930+0000","last_undegraded":"2024-03-24T00:43:08.815930+0000","last_fullsized":"2024-03-24T00:43:08.815930+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.807731+0000","last_change":"2024-03-24T00:43:08.807731+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.807731+0000","last_undegraded":"2024-03-24T00:43:08.807731+0000","last_fullsized":"2024-03-24T00:43:08.807731+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.810712+0000","last_change":"2024-03-24T00:43:08.810712+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.810712+0000","last_undegraded":"2024-03-24T00:43:08.810712+0000","last_fullsized":"2024-03-24T00:43:08.810712+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":268,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-24T00:43:08.810650+0000","last_change":"2024-03-24T00:38:40.953871+0000","last_active":"2024-03-24T00:43:08.810650+0000","last_peered":"2024-03-24T00:43:08.810650+0000","last_clean":"2024-03-24T00:43:08.810650+0000","last_became_active":"2024-03-24T00:38:40.952926+0000","last_became_peered":"2024-03-24T00:38:40.952926+0000","last_unstale":"2024-03-24T00:43:08.810650+0000","last_undegraded":"2024-03-24T00:43:08.810650+0000","last_fullsized":"2024-03-24T00:43:08.810650+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_clean_scrub_stamp":"2024-03-24T00:38:39.722194+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:05.397257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28460,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343380,"statfs":{"total":96636764160,"available":96607621120,"internally_reserved":0,"allocated":1474560,"data_stored":988474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27892,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343948,"statfs":{"total":96636764160,"available":96608202752,"internally_reserved":0,"allocated":892928,"data_stored":404315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27880,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343960,"statfs":{"total":96636764160,"available":96608215040,"internally_reserved":0,"allocated":880640,"data_stored":398106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574905,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27872,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343968,"statfs":{"total":96636764160,"available":96608223232,"internally_reserved":0,"allocated":868352,"data_stored":392587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574905,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27892,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343948,"statfs":{"total":96636764160,"available":96608202752,"internally_reserved":0,"allocated":892928,"data_stored":404315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574906,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27896,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343944,"statfs":{"total":96636764160,"available":96608198656,"internally_reserved":0,"allocated":892928,"data_stored":404315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27880,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343960,"statfs":{"total":96636764160,"available":96608215040,"internally_reserved":0,"allocated":880640,"data_stored":398106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574905,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32616,"kb_used_data":1428,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339224,"statfs":{"total":96636764160,"available":96603365376,"internally_reserved":0,"allocated":1462272,"data_stored":982955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:43:14.184 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:43:14.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.321+0000 7ff22efd4640 1 Processor -- start 2024-03-24T00:43:14.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.321+0000 7ff22efd4640 1 -- start start 2024-03-24T00:43:14.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.322+0000 7ff22efd4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff22815bf70 0x7ff2280cc690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:14.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.322+0000 7ff22efd4640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2280ccfe0 con 0x7ff2280ce930 2024-03-24T00:43:14.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.322+0000 7ff22efd4640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2280cd2c0 con 0x7ff2280cfc70 2024-03-24T00:43:14.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.322+0000 7ff22efd4640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2280cd5a0 con 0x7ff22815bf70 2024-03-24T00:43:14.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.322+0000 7ff22cd70640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff22815bf70 0x7ff2280cc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:14.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.322+0000 7ff22d571640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff2280ce930 0x7ff2280b1cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37502/0 (socket says 172.21.15.67:37502) 2024-03-24T00:43:14.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.322+0000 7ff22cd70640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff22815bf70 0x7ff2280cc690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50336/0 (socket says 172.21.15.67:50336) 2024-03-24T00:43:14.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.322+0000 7ff22d571640 1 -- 172.21.15.67:0/3718580090 learned_addr learned my addr 172.21.15.67:0/3718580090 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:43:14.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.323+0000 7ff22cd70640 1 -- 172.21.15.67:0/3718580090 >> v1:172.21.15.72:6789/0 conn(0x7ff2280cfc70 legacy=0x7ff2280ce220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:14.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.323+0000 7ff22cd70640 1 -- 172.21.15.67:0/3718580090 >> v1:172.21.15.67:6789/0 conn(0x7ff2280ce930 legacy=0x7ff2280b1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:14.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.323+0000 7ff22cd70640 1 -- 172.21.15.67:0/3718580090 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2280cd880 con 0x7ff22815bf70 2024-03-24T00:43:14.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.323+0000 7ff2277fe640 1 -- 172.21.15.67:0/3718580090 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 59905023 0 0) 0x7ff2280ccfe0 con 0x7ff2280ce930 2024-03-24T00:43:14.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.323+0000 7ff22cd70640 1 --2- 172.21.15.67:0/3718580090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff22815bf70 0x7ff2280cc690 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7ff21c0097a0 tx=0x7ff21c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3ef6dfa0880c65 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:14.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.323+0000 7ff2277fe640 1 -- 172.21.15.67:0/3718580090 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff21c034070 con 0x7ff22815bf70 2024-03-24T00:43:14.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.323+0000 7ff2277fe640 1 -- 172.21.15.67:0/3718580090 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff21c030610 con 0x7ff22815bf70 2024-03-24T00:43:14.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.323+0000 7ff2277fe640 1 -- 172.21.15.67:0/3718580090 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff21c0309e0 con 0x7ff22815bf70 2024-03-24T00:43:14.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.324+0000 7ff22efd4640 1 -- 172.21.15.67:0/3718580090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff22815bf70 msgr2=0x7ff2280cc690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:14.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.324+0000 7ff22efd4640 1 --2- 172.21.15.67:0/3718580090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff22815bf70 0x7ff2280cc690 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7ff21c0097a0 tx=0x7ff21c02b960 comp rx=0 tx=0).stop 2024-03-24T00:43:14.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.324+0000 7ff22efd4640 1 -- 172.21.15.67:0/3718580090 shutdown_connections 2024-03-24T00:43:14.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.324+0000 7ff22efd4640 1 --2- 172.21.15.67:0/3718580090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff22815bf70 0x7ff2280cc690 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:14.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.324+0000 7ff22efd4640 1 -- 172.21.15.67:0/3718580090 >> 172.21.15.67:0/3718580090 conn(0x7ff2280d1bf0 msgr2=0x7ff2280d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:14.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.324+0000 7ff22efd4640 1 -- 172.21.15.67:0/3718580090 shutdown_connections 2024-03-24T00:43:14.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.324+0000 7ff22efd4640 1 -- 172.21.15.67:0/3718580090 wait complete. 2024-03-24T00:43:14.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.326+0000 7ff22efd4640 1 Processor -- start 2024-03-24T00:43:14.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.326+0000 7ff22efd4640 1 -- start start 2024-03-24T00:43:14.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.327+0000 7ff22efd4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2280ce930 0x7ff228150dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:14.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.327+0000 7ff22efd4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff2280cfc70 0x7ff2280c79f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:14.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.327+0000 7ff22efd4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff22815bf70 0x7ff22815e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:14.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.327+0000 7ff22efd4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2280ccd50 con 0x7ff2280ce930 2024-03-24T00:43:14.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.327+0000 7ff22efd4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2280ccbd0 con 0x7ff22815bf70 2024-03-24T00:43:14.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.327+0000 7ff22efd4640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2280b23e0 con 0x7ff2280cfc70 2024-03-24T00:43:14.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.328+0000 7ff22cd70640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2280ce930 0x7ff228150dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:14.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.328+0000 7ff22cd70640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2280ce930 0x7ff228150dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50348/0 (socket says 172.21.15.67:50348) 2024-03-24T00:43:14.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.328+0000 7ff22cd70640 1 -- 172.21.15.67:0/2126147040 learned_addr learned my addr 172.21.15.67:0/2126147040 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:14.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.328+0000 7ff22d571640 1 --2- 172.21.15.67:0/2126147040 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff22815bf70 0x7ff22815e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:14.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.328+0000 7ff227fff640 1 --2- 172.21.15.67:0/2126147040 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff2280cfc70 0x7ff2280c79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:14.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.328+0000 7ff22cd70640 1 -- 172.21.15.67:0/2126147040 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff2280cfc70 msgr2=0x7ff2280c79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:14.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.328+0000 7ff22cd70640 1 --2- 172.21.15.67:0/2126147040 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff2280cfc70 0x7ff2280c79f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:14.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.328+0000 7ff22cd70640 1 -- 172.21.15.67:0/2126147040 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff22815bf70 msgr2=0x7ff22815e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:14.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.328+0000 7ff22cd70640 1 --2- 172.21.15.67:0/2126147040 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff22815bf70 0x7ff22815e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:14.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.328+0000 7ff22cd70640 1 -- 172.21.15.67:0/2126147040 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff22815a3e0 con 0x7ff2280ce930 2024-03-24T00:43:14.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.328+0000 7ff22d571640 1 --2- 172.21.15.67:0/2126147040 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff22815bf70 0x7ff22815e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:43:14.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.328+0000 7ff227fff640 1 --2- 172.21.15.67:0/2126147040 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff2280cfc70 0x7ff2280c79f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:43:14.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.328+0000 7ff22cd70640 1 --2- 172.21.15.67:0/2126147040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2280ce930 0x7ff228150dc0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7ff21c0097a0 tx=0x7ff21c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:14.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.328+0000 7ff2257fa640 1 -- 172.21.15.67:0/2126147040 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff21c048070 con 0x7ff2280ce930 2024-03-24T00:43:14.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.328+0000 7ff2257fa640 1 -- 172.21.15.67:0/2126147040 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff21c039990 con 0x7ff2280ce930 2024-03-24T00:43:14.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.328+0000 7ff22efd4640 1 -- 172.21.15.67:0/2126147040 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2280b7050 con 0x7ff2280ce930 2024-03-24T00:43:14.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.328+0000 7ff2257fa640 1 -- 172.21.15.67:0/2126147040 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff21c034040 con 0x7ff2280ce930 2024-03-24T00:43:14.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.328+0000 7ff22efd4640 1 -- 172.21.15.67:0/2126147040 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2280b7610 con 0x7ff2280ce930 2024-03-24T00:43:14.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.329+0000 7ff22efd4640 1 -- 172.21.15.67:0/2126147040 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff22804f400 con 0x7ff2280ce930 2024-03-24T00:43:14.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.330+0000 7ff2257fa640 1 -- 172.21.15.67:0/2126147040 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff21c04b070 con 0x7ff2280ce930 2024-03-24T00:43:14.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.330+0000 7ff2257fa640 1 --2- 172.21.15.67:0/2126147040 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff20c06bbd0 0x7ff20c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:14.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.331+0000 7ff2257fa640 1 -- 172.21.15.67:0/2126147040 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff21c0acac0 con 0x7ff2280ce930 2024-03-24T00:43:14.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.331+0000 7ff227fff640 1 --2- 172.21.15.67:0/2126147040 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff20c06bbd0 0x7ff20c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:14.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.331+0000 7ff227fff640 1 --2- 172.21.15.67:0/2126147040 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff20c06bbd0 0x7ff20c06e090 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7ff214007fd0 tx=0x7ff214009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:14.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.335+0000 7ff2257fa640 1 -- 172.21.15.67:0/2126147040 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff22815a3e0 con 0x7ff2280ce930 2024-03-24T00:43:14.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.515+0000 7ff22efd4640 1 -- 172.21.15.67:0/2126147040 --> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff2280b1fb0 con 0x7ff20c06bbd0 2024-03-24T00:43:14.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.521+0000 7ff2257fa640 1 -- 172.21.15.67:0/2126147040 <== mgr.4105 v2:172.21.15.72:6832/2768292274 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240144 (secure 0 0 0) 0x7ff2280b1fb0 con 0x7ff20c06bbd0 2024-03-24T00:43:14.524 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:43:14.525 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-24T00:43:14.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.525+0000 7ff22efd4640 1 -- 172.21.15.67:0/2126147040 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff20c06bbd0 msgr2=0x7ff20c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:14.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.525+0000 7ff22efd4640 1 --2- 172.21.15.67:0/2126147040 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff20c06bbd0 0x7ff20c06e090 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7ff214007fd0 tx=0x7ff214009040 comp rx=0 tx=0).stop 2024-03-24T00:43:14.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.525+0000 7ff22efd4640 1 -- 172.21.15.67:0/2126147040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2280ce930 msgr2=0x7ff228150dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:14.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.525+0000 7ff22efd4640 1 --2- 172.21.15.67:0/2126147040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2280ce930 0x7ff228150dc0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7ff21c0097a0 tx=0x7ff21c0046c0 comp rx=0 tx=0).stop 2024-03-24T00:43:14.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.525+0000 7ff22efd4640 1 -- 172.21.15.67:0/2126147040 shutdown_connections 2024-03-24T00:43:14.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.525+0000 7ff22efd4640 1 --2- 172.21.15.67:0/2126147040 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff20c06bbd0 0x7ff20c06e090 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:14.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.525+0000 7ff22efd4640 1 --2- 172.21.15.67:0/2126147040 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff22815bf70 0x7ff22815e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:14.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.525+0000 7ff22efd4640 1 --2- 172.21.15.67:0/2126147040 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff2280cfc70 0x7ff2280c79f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:14.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.525+0000 7ff22efd4640 1 --2- 172.21.15.67:0/2126147040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2280ce930 0x7ff228150dc0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:14.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.525+0000 7ff22efd4640 1 -- 172.21.15.67:0/2126147040 >> 172.21.15.67:0/2126147040 conn(0x7ff2280d1bf0 msgr2=0x7ff22806fce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:14.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.525+0000 7ff22efd4640 1 -- 172.21.15.67:0/2126147040 shutdown_connections 2024-03-24T00:43:14.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.525+0000 7ff22efd4640 1 -- 172.21.15.67:0/2126147040 wait complete. 2024-03-24T00:43:14.537 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":206,"stamp":"2024-03-24T00:43:13.382294+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":8,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":228388,"kb_used_data":8052,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754746332,"statfs":{"total":773094113280,"available":772860243968,"internally_reserved":0,"allocated":8245248,"data_stored":4373173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":225496625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.577202"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.806432+0000","last_change":"2024-03-24T00:43:08.806432+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.806432+0000","last_undegraded":"2024-03-24T00:43:08.806432+0000","last_fullsized":"2024-03-24T00:43:08.806432+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.809259+0000","last_change":"2024-03-24T00:43:08.809259+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.809259+0000","last_undegraded":"2024-03-24T00:43:08.809259+0000","last_fullsized":"2024-03-24T00:43:08.809259+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.809761+0000","last_change":"2024-03-24T00:43:08.809761+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.809761+0000","last_undegraded":"2024-03-24T00:43:08.809761+0000","last_fullsized":"2024-03-24T00:43:08.809761+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.816853+0000","last_change":"2024-03-24T00:43:08.816853+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.816853+0000","last_undegraded":"2024-03-24T00:43:08.816853+0000","last_fullsized":"2024-03-24T00:43:08.816853+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.809304+0000","last_change":"2024-03-24T00:43:08.809304+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.809304+0000","last_undegraded":"2024-03-24T00:43:08.809304+0000","last_fullsized":"2024-03-24T00:43:08.809304+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.806953+0000","last_change":"2024-03-24T00:43:08.806953+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.806953+0000","last_undegraded":"2024-03-24T00:43:08.806953+0000","last_fullsized":"2024-03-24T00:43:08.806953+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.812355+0000","last_change":"2024-03-24T00:43:08.812355+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.812355+0000","last_undegraded":"2024-03-24T00:43:08.812355+0000","last_fullsized":"2024-03-24T00:43:08.812355+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.806378+0000","last_change":"2024-03-24T00:43:08.806378+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.806378+0000","last_undegraded":"2024-03-24T00:43:08.806378+0000","last_fullsized":"2024-03-24T00:43:08.806378+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.810005+0000","last_change":"2024-03-24T00:43:08.810005+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.810005+0000","last_undegraded":"2024-03-24T00:43:08.810005+0000","last_fullsized":"2024-03-24T00:43:08.810005+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.806947+0000","last_change":"2024-03-24T00:43:08.806947+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.806947+0000","last_undegraded":"2024-03-24T00:43:08.806947+0000","last_fullsized":"2024-03-24T00:43:08.806947+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.806868+0000","last_change":"2024-03-24T00:43:08.806868+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.806868+0000","last_undegraded":"2024-03-24T00:43:08.806868+0000","last_fullsized":"2024-03-24T00:43:08.806868+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.816394+0000","last_change":"2024-03-24T00:43:08.816394+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.816394+0000","last_undegraded":"2024-03-24T00:43:08.816394+0000","last_fullsized":"2024-03-24T00:43:08.816394+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.806953+0000","last_change":"2024-03-24T00:43:08.806953+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.806953+0000","last_undegraded":"2024-03-24T00:43:08.806953+0000","last_fullsized":"2024-03-24T00:43:08.806953+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.809389+0000","last_change":"2024-03-24T00:43:08.809389+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.809389+0000","last_undegraded":"2024-03-24T00:43:08.809389+0000","last_fullsized":"2024-03-24T00:43:08.809389+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.806381+0000","last_change":"2024-03-24T00:43:08.806381+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.806381+0000","last_undegraded":"2024-03-24T00:43:08.806381+0000","last_fullsized":"2024-03-24T00:43:08.806381+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.812505+0000","last_change":"2024-03-24T00:43:08.812505+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.812505+0000","last_undegraded":"2024-03-24T00:43:08.812505+0000","last_fullsized":"2024-03-24T00:43:08.812505+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.809567+0000","last_change":"2024-03-24T00:43:08.809567+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.809567+0000","last_undegraded":"2024-03-24T00:43:08.809567+0000","last_fullsized":"2024-03-24T00:43:08.809567+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.809309+0000","last_change":"2024-03-24T00:43:08.809309+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.809309+0000","last_undegraded":"2024-03-24T00:43:08.809309+0000","last_fullsized":"2024-03-24T00:43:08.809309+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.806390+0000","last_change":"2024-03-24T00:43:08.806390+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.806390+0000","last_undegraded":"2024-03-24T00:43:08.806390+0000","last_fullsized":"2024-03-24T00:43:08.806390+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.807736+0000","last_change":"2024-03-24T00:43:08.807736+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.807736+0000","last_undegraded":"2024-03-24T00:43:08.807736+0000","last_fullsized":"2024-03-24T00:43:08.807736+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.811079+0000","last_change":"2024-03-24T00:43:08.811079+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.811079+0000","last_undegraded":"2024-03-24T00:43:08.811079+0000","last_fullsized":"2024-03-24T00:43:08.811079+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.810648+0000","last_change":"2024-03-24T00:43:08.810648+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.810648+0000","last_undegraded":"2024-03-24T00:43:08.810648+0000","last_fullsized":"2024-03-24T00:43:08.810648+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.810555+0000","last_change":"2024-03-24T00:43:08.810555+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.810555+0000","last_undegraded":"2024-03-24T00:43:08.810555+0000","last_fullsized":"2024-03-24T00:43:08.810555+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.806959+0000","last_change":"2024-03-24T00:43:08.806959+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.806959+0000","last_undegraded":"2024-03-24T00:43:08.806959+0000","last_fullsized":"2024-03-24T00:43:08.806959+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.812359+0000","last_change":"2024-03-24T00:43:08.812359+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.812359+0000","last_undegraded":"2024-03-24T00:43:08.812359+0000","last_fullsized":"2024-03-24T00:43:08.812359+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.809204+0000","last_change":"2024-03-24T00:43:08.809204+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.809204+0000","last_undegraded":"2024-03-24T00:43:08.809204+0000","last_fullsized":"2024-03-24T00:43:08.809204+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.809754+0000","last_change":"2024-03-24T00:43:08.809754+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.809754+0000","last_undegraded":"2024-03-24T00:43:08.809754+0000","last_fullsized":"2024-03-24T00:43:08.809754+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.816382+0000","last_change":"2024-03-24T00:43:08.816382+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.816382+0000","last_undegraded":"2024-03-24T00:43:08.816382+0000","last_fullsized":"2024-03-24T00:43:08.816382+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.806382+0000","last_change":"2024-03-24T00:43:08.806382+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.806382+0000","last_undegraded":"2024-03-24T00:43:08.806382+0000","last_fullsized":"2024-03-24T00:43:08.806382+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.815930+0000","last_change":"2024-03-24T00:43:08.815930+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.815930+0000","last_undegraded":"2024-03-24T00:43:08.815930+0000","last_fullsized":"2024-03-24T00:43:08.815930+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.807731+0000","last_change":"2024-03-24T00:43:08.807731+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.807731+0000","last_undegraded":"2024-03-24T00:43:08.807731+0000","last_fullsized":"2024-03-24T00:43:08.807731+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:43:09.804492+0000","last_change":"2024-03-24T00:43:09.804492+0000","last_active":"2024-03-24T00:43:09.804492+0000","last_peered":"2024-03-24T00:43:09.804492+0000","last_clean":"2024-03-24T00:43:09.804492+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:09.804492+0000","last_undegraded":"2024-03-24T00:43:09.804492+0000","last_fullsized":"2024-03-24T00:43:09.804492+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1,"reported_epoch":73,"state":"creating+peering","last_fresh":"2024-03-24T00:43:08.810712+0000","last_change":"2024-03-24T00:43:08.810712+0000","last_active":"2024-03-24T00:43:08.800646+0000","last_peered":"2024-03-24T00:43:08.800646+0000","last_clean":"2024-03-24T00:43:08.800646+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:43:08.810712+0000","last_undegraded":"2024-03-24T00:43:08.810712+0000","last_fullsized":"2024-03-24T00:43:08.810712+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":268,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-24T00:43:08.810650+0000","last_change":"2024-03-24T00:38:40.953871+0000","last_active":"2024-03-24T00:43:08.810650+0000","last_peered":"2024-03-24T00:43:08.810650+0000","last_clean":"2024-03-24T00:43:08.810650+0000","last_became_active":"2024-03-24T00:38:40.952926+0000","last_became_peered":"2024-03-24T00:38:40.952926+0000","last_unstale":"2024-03-24T00:43:08.810650+0000","last_undegraded":"2024-03-24T00:43:08.810650+0000","last_fullsized":"2024-03-24T00:43:08.810650+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_clean_scrub_stamp":"2024-03-24T00:38:39.722194+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:05.397257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28460,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343380,"statfs":{"total":96636764160,"available":96607621120,"internally_reserved":0,"allocated":1474560,"data_stored":988474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27892,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343948,"statfs":{"total":96636764160,"available":96608202752,"internally_reserved":0,"allocated":892928,"data_stored":404315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27880,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343960,"statfs":{"total":96636764160,"available":96608215040,"internally_reserved":0,"allocated":880640,"data_stored":398106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574905,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27872,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343968,"statfs":{"total":96636764160,"available":96608223232,"internally_reserved":0,"allocated":868352,"data_stored":392587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574905,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27892,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343948,"statfs":{"total":96636764160,"available":96608202752,"internally_reserved":0,"allocated":892928,"data_stored":404315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574906,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27896,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343944,"statfs":{"total":96636764160,"available":96608198656,"internally_reserved":0,"allocated":892928,"data_stored":404315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27880,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343960,"statfs":{"total":96636764160,"available":96608215040,"internally_reserved":0,"allocated":880640,"data_stored":398106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574905,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32616,"kb_used_data":1428,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339224,"statfs":{"total":96636764160,"available":96603365376,"internally_reserved":0,"allocated":1462272,"data_stored":982955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:43:14.540 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.3f is not active+clean 2024-03-24T00:43:14.540 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.541 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.3e is not active+clean 2024-03-24T00:43:14.541 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.541 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.3d is not active+clean 2024-03-24T00:43:14.541 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.541 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.3c is not active+clean 2024-03-24T00:43:14.541 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.541 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.3b is not active+clean 2024-03-24T00:43:14.541 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.541 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.3a is not active+clean 2024-03-24T00:43:14.541 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.3a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.541 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.39 is not active+clean 2024-03-24T00:43:14.542 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.542 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.38 is not active+clean 2024-03-24T00:43:14.542 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.542 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.37 is not active+clean 2024-03-24T00:43:14.542 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.542 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.36 is not active+clean 2024-03-24T00:43:14.542 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.542 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.35 is not active+clean 2024-03-24T00:43:14.542 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.542 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.34 is not active+clean 2024-03-24T00:43:14.542 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.543 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.33 is not active+clean 2024-03-24T00:43:14.543 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.543 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.32 is not active+clean 2024-03-24T00:43:14.543 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.543 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.31 is not active+clean 2024-03-24T00:43:14.543 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.543 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.30 is not active+clean 2024-03-24T00:43:14.543 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.543 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.2f is not active+clean 2024-03-24T00:43:14.543 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.544 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.2e is not active+clean 2024-03-24T00:43:14.544 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.544 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.2d is not active+clean 2024-03-24T00:43:14.544 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.544 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.2c is not active+clean 2024-03-24T00:43:14.544 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.2c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.544 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.2b is not active+clean 2024-03-24T00:43:14.544 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.544 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.2a is not active+clean 2024-03-24T00:43:14.544 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.544 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.29 is not active+clean 2024-03-24T00:43:14.544 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.545 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.28 is not active+clean 2024-03-24T00:43:14.545 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.545 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.27 is not active+clean 2024-03-24T00:43:14.545 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.27', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.545 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.26 is not active+clean 2024-03-24T00:43:14.545 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.545 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.25 is not active+clean 2024-03-24T00:43:14.545 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.545 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.24 is not active+clean 2024-03-24T00:43:14.545 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.545 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.23 is not active+clean 2024-03-24T00:43:14.545 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.546 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.22 is not active+clean 2024-03-24T00:43:14.546 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.546 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.21 is not active+clean 2024-03-24T00:43:14.546 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.546 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.20 is not active+clean 2024-03-24T00:43:14.546 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.546 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.18 is not active+clean 2024-03-24T00:43:14.546 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.546 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1f is not active+clean 2024-03-24T00:43:14.546 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 73, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:43:08.806432+0000', 'last_change': '2024-03-24T00:43:08.806432+0000', 'last_active': '2024-03-24T00:43:08.800646+0000', 'last_peered': '2024-03-24T00:43:08.800646+0000', 'last_clean': '2024-03-24T00:43:08.800646+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:08.806432+0000', 'last_undegraded': '2024-03-24T00:43:08.806432+0000', 'last_fullsized': '2024-03-24T00:43:08.806432+0000', 'mapping_epoch': 73, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 73, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-24T00:43:14.546 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.19 is not active+clean 2024-03-24T00:43:14.546 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.547 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1e is not active+clean 2024-03-24T00:43:14.547 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 73, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:43:08.809259+0000', 'last_change': '2024-03-24T00:43:08.809259+0000', 'last_active': '2024-03-24T00:43:08.800646+0000', 'last_peered': '2024-03-24T00:43:08.800646+0000', 'last_clean': '2024-03-24T00:43:08.800646+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:08.809259+0000', 'last_undegraded': '2024-03-24T00:43:08.809259+0000', 'last_fullsized': '2024-03-24T00:43:08.809259+0000', 'mapping_epoch': 73, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 73, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T00:43:14.547 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.1a is not active+clean 2024-03-24T00:43:14.547 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.547 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1d is not active+clean 2024-03-24T00:43:14.547 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 73, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:43:08.809761+0000', 'last_change': '2024-03-24T00:43:08.809761+0000', 'last_active': '2024-03-24T00:43:08.800646+0000', 'last_peered': '2024-03-24T00:43:08.800646+0000', 'last_clean': '2024-03-24T00:43:08.800646+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:08.809761+0000', 'last_undegraded': '2024-03-24T00:43:08.809761+0000', 'last_fullsized': '2024-03-24T00:43:08.809761+0000', 'mapping_epoch': 73, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 73, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T00:43:14.547 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.1b is not active+clean 2024-03-24T00:43:14.547 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.547 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1c is not active+clean 2024-03-24T00:43:14.547 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 73, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:43:08.816853+0000', 'last_change': '2024-03-24T00:43:08.816853+0000', 'last_active': '2024-03-24T00:43:08.800646+0000', 'last_peered': '2024-03-24T00:43:08.800646+0000', 'last_clean': '2024-03-24T00:43:08.800646+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:08.816853+0000', 'last_undegraded': '2024-03-24T00:43:08.816853+0000', 'last_fullsized': '2024-03-24T00:43:08.816853+0000', 'mapping_epoch': 73, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 73, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-24T00:43:14.547 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.1c is not active+clean 2024-03-24T00:43:14.547 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.548 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1b is not active+clean 2024-03-24T00:43:14.548 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 73, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:43:08.809304+0000', 'last_change': '2024-03-24T00:43:08.809304+0000', 'last_active': '2024-03-24T00:43:08.800646+0000', 'last_peered': '2024-03-24T00:43:08.800646+0000', 'last_clean': '2024-03-24T00:43:08.800646+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:08.809304+0000', 'last_undegraded': '2024-03-24T00:43:08.809304+0000', 'last_fullsized': '2024-03-24T00:43:08.809304+0000', 'mapping_epoch': 73, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 73, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-24T00:43:14.548 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.1d is not active+clean 2024-03-24T00:43:14.548 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.548 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1a is not active+clean 2024-03-24T00:43:14.548 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 73, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:43:08.806953+0000', 'last_change': '2024-03-24T00:43:08.806953+0000', 'last_active': '2024-03-24T00:43:08.800646+0000', 'last_peered': '2024-03-24T00:43:08.800646+0000', 'last_clean': '2024-03-24T00:43:08.800646+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:08.806953+0000', 'last_undegraded': '2024-03-24T00:43:08.806953+0000', 'last_fullsized': '2024-03-24T00:43:08.806953+0000', 'mapping_epoch': 73, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 73, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-24T00:43:14.548 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.1e is not active+clean 2024-03-24T00:43:14.548 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.548 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.19 is not active+clean 2024-03-24T00:43:14.548 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 73, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:43:08.812355+0000', 'last_change': '2024-03-24T00:43:08.812355+0000', 'last_active': '2024-03-24T00:43:08.800646+0000', 'last_peered': '2024-03-24T00:43:08.800646+0000', 'last_clean': '2024-03-24T00:43:08.800646+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:08.812355+0000', 'last_undegraded': '2024-03-24T00:43:08.812355+0000', 'last_fullsized': '2024-03-24T00:43:08.812355+0000', 'mapping_epoch': 73, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 73, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T00:43:14.548 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.1f is not active+clean 2024-03-24T00:43:14.548 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.549 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.18 is not active+clean 2024-03-24T00:43:14.549 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 73, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:43:08.806378+0000', 'last_change': '2024-03-24T00:43:08.806378+0000', 'last_active': '2024-03-24T00:43:08.800646+0000', 'last_peered': '2024-03-24T00:43:08.800646+0000', 'last_clean': '2024-03-24T00:43:08.800646+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:08.806378+0000', 'last_undegraded': '2024-03-24T00:43:08.806378+0000', 'last_fullsized': '2024-03-24T00:43:08.806378+0000', 'mapping_epoch': 73, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 73, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-24T00:43:14.549 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.10 is not active+clean 2024-03-24T00:43:14.549 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.549 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.17 is not active+clean 2024-03-24T00:43:14.549 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 73, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:43:08.810005+0000', 'last_change': '2024-03-24T00:43:08.810005+0000', 'last_active': '2024-03-24T00:43:08.800646+0000', 'last_peered': '2024-03-24T00:43:08.800646+0000', 'last_clean': '2024-03-24T00:43:08.800646+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:08.810005+0000', 'last_undegraded': '2024-03-24T00:43:08.810005+0000', 'last_fullsized': '2024-03-24T00:43:08.810005+0000', 'mapping_epoch': 73, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 73, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T00:43:14.549 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.11 is not active+clean 2024-03-24T00:43:14.549 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.549 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.16 is not active+clean 2024-03-24T00:43:14.549 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 73, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:43:08.806947+0000', 'last_change': '2024-03-24T00:43:08.806947+0000', 'last_active': '2024-03-24T00:43:08.800646+0000', 'last_peered': '2024-03-24T00:43:08.800646+0000', 'last_clean': '2024-03-24T00:43:08.800646+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:08.806947+0000', 'last_undegraded': '2024-03-24T00:43:08.806947+0000', 'last_fullsized': '2024-03-24T00:43:08.806947+0000', 'mapping_epoch': 73, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 73, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-24T00:43:14.549 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.12 is not active+clean 2024-03-24T00:43:14.549 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.550 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.15 is not active+clean 2024-03-24T00:43:14.550 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 73, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:43:08.806868+0000', 'last_change': '2024-03-24T00:43:08.806868+0000', 'last_active': '2024-03-24T00:43:08.800646+0000', 'last_peered': '2024-03-24T00:43:08.800646+0000', 'last_clean': '2024-03-24T00:43:08.800646+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:08.806868+0000', 'last_undegraded': '2024-03-24T00:43:08.806868+0000', 'last_fullsized': '2024-03-24T00:43:08.806868+0000', 'mapping_epoch': 73, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 73, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-24T00:43:14.550 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.13 is not active+clean 2024-03-24T00:43:14.550 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.550 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.14 is not active+clean 2024-03-24T00:43:14.550 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 73, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:43:08.816394+0000', 'last_change': '2024-03-24T00:43:08.816394+0000', 'last_active': '2024-03-24T00:43:08.800646+0000', 'last_peered': '2024-03-24T00:43:08.800646+0000', 'last_clean': '2024-03-24T00:43:08.800646+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:08.816394+0000', 'last_undegraded': '2024-03-24T00:43:08.816394+0000', 'last_fullsized': '2024-03-24T00:43:08.816394+0000', 'mapping_epoch': 73, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 73, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-24T00:43:14.550 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.14 is not active+clean 2024-03-24T00:43:14.550 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.550 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.13 is not active+clean 2024-03-24T00:43:14.550 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 73, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:43:08.806953+0000', 'last_change': '2024-03-24T00:43:08.806953+0000', 'last_active': '2024-03-24T00:43:08.800646+0000', 'last_peered': '2024-03-24T00:43:08.800646+0000', 'last_clean': '2024-03-24T00:43:08.800646+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:08.806953+0000', 'last_undegraded': '2024-03-24T00:43:08.806953+0000', 'last_fullsized': '2024-03-24T00:43:08.806953+0000', 'mapping_epoch': 73, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 73, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-24T00:43:14.550 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.15 is not active+clean 2024-03-24T00:43:14.551 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.551 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.12 is not active+clean 2024-03-24T00:43:14.551 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 73, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:43:08.809389+0000', 'last_change': '2024-03-24T00:43:08.809389+0000', 'last_active': '2024-03-24T00:43:08.800646+0000', 'last_peered': '2024-03-24T00:43:08.800646+0000', 'last_clean': '2024-03-24T00:43:08.800646+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:08.809389+0000', 'last_undegraded': '2024-03-24T00:43:08.809389+0000', 'last_fullsized': '2024-03-24T00:43:08.809389+0000', 'mapping_epoch': 73, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 73, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T00:43:14.551 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.16 is not active+clean 2024-03-24T00:43:14.551 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.551 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.11 is not active+clean 2024-03-24T00:43:14.551 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 73, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:43:08.806381+0000', 'last_change': '2024-03-24T00:43:08.806381+0000', 'last_active': '2024-03-24T00:43:08.800646+0000', 'last_peered': '2024-03-24T00:43:08.800646+0000', 'last_clean': '2024-03-24T00:43:08.800646+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:08.806381+0000', 'last_undegraded': '2024-03-24T00:43:08.806381+0000', 'last_fullsized': '2024-03-24T00:43:08.806381+0000', 'mapping_epoch': 73, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 73, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-24T00:43:14.551 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.17 is not active+clean 2024-03-24T00:43:14.551 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.551 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.10 is not active+clean 2024-03-24T00:43:14.551 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 73, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:43:08.812505+0000', 'last_change': '2024-03-24T00:43:08.812505+0000', 'last_active': '2024-03-24T00:43:08.800646+0000', 'last_peered': '2024-03-24T00:43:08.800646+0000', 'last_clean': '2024-03-24T00:43:08.800646+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:08.812505+0000', 'last_undegraded': '2024-03-24T00:43:08.812505+0000', 'last_fullsized': '2024-03-24T00:43:08.812505+0000', 'mapping_epoch': 73, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 73, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T00:43:14.552 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.8 is not active+clean 2024-03-24T00:43:14.552 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.552 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.f is not active+clean 2024-03-24T00:43:14.552 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 73, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:43:08.809567+0000', 'last_change': '2024-03-24T00:43:08.809567+0000', 'last_active': '2024-03-24T00:43:08.800646+0000', 'last_peered': '2024-03-24T00:43:08.800646+0000', 'last_clean': '2024-03-24T00:43:08.800646+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:08.809567+0000', 'last_undegraded': '2024-03-24T00:43:08.809567+0000', 'last_fullsized': '2024-03-24T00:43:08.809567+0000', 'mapping_epoch': 73, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 73, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T00:43:14.552 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.9 is not active+clean 2024-03-24T00:43:14.552 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.552 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.e is not active+clean 2024-03-24T00:43:14.552 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 73, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:43:08.809309+0000', 'last_change': '2024-03-24T00:43:08.809309+0000', 'last_active': '2024-03-24T00:43:08.800646+0000', 'last_peered': '2024-03-24T00:43:08.800646+0000', 'last_clean': '2024-03-24T00:43:08.800646+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:08.809309+0000', 'last_undegraded': '2024-03-24T00:43:08.809309+0000', 'last_fullsized': '2024-03-24T00:43:08.809309+0000', 'mapping_epoch': 73, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 73, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-24T00:43:14.552 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.a is not active+clean 2024-03-24T00:43:14.552 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.552 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.d is not active+clean 2024-03-24T00:43:14.552 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 73, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:43:08.806390+0000', 'last_change': '2024-03-24T00:43:08.806390+0000', 'last_active': '2024-03-24T00:43:08.800646+0000', 'last_peered': '2024-03-24T00:43:08.800646+0000', 'last_clean': '2024-03-24T00:43:08.800646+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:08.806390+0000', 'last_undegraded': '2024-03-24T00:43:08.806390+0000', 'last_fullsized': '2024-03-24T00:43:08.806390+0000', 'mapping_epoch': 73, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 73, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-24T00:43:14.553 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.b is not active+clean 2024-03-24T00:43:14.553 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.553 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.c is not active+clean 2024-03-24T00:43:14.553 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 73, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:43:08.807736+0000', 'last_change': '2024-03-24T00:43:08.807736+0000', 'last_active': '2024-03-24T00:43:08.800646+0000', 'last_peered': '2024-03-24T00:43:08.800646+0000', 'last_clean': '2024-03-24T00:43:08.800646+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:08.807736+0000', 'last_undegraded': '2024-03-24T00:43:08.807736+0000', 'last_fullsized': '2024-03-24T00:43:08.807736+0000', 'mapping_epoch': 73, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 73, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-24T00:43:14.553 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.c is not active+clean 2024-03-24T00:43:14.553 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.553 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.b is not active+clean 2024-03-24T00:43:14.553 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 73, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:43:08.811079+0000', 'last_change': '2024-03-24T00:43:08.811079+0000', 'last_active': '2024-03-24T00:43:08.800646+0000', 'last_peered': '2024-03-24T00:43:08.800646+0000', 'last_clean': '2024-03-24T00:43:08.800646+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:08.811079+0000', 'last_undegraded': '2024-03-24T00:43:08.811079+0000', 'last_fullsized': '2024-03-24T00:43:08.811079+0000', 'mapping_epoch': 73, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 73, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T00:43:14.553 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.e is not active+clean 2024-03-24T00:43:14.553 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.554 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.9 is not active+clean 2024-03-24T00:43:14.554 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 73, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:43:08.810648+0000', 'last_change': '2024-03-24T00:43:08.810648+0000', 'last_active': '2024-03-24T00:43:08.800646+0000', 'last_peered': '2024-03-24T00:43:08.800646+0000', 'last_clean': '2024-03-24T00:43:08.800646+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:08.810648+0000', 'last_undegraded': '2024-03-24T00:43:08.810648+0000', 'last_fullsized': '2024-03-24T00:43:08.810648+0000', 'mapping_epoch': 73, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 73, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T00:43:14.554 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.f is not active+clean 2024-03-24T00:43:14.554 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.554 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.8 is not active+clean 2024-03-24T00:43:14.554 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 73, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:43:08.810555+0000', 'last_change': '2024-03-24T00:43:08.810555+0000', 'last_active': '2024-03-24T00:43:08.800646+0000', 'last_peered': '2024-03-24T00:43:08.800646+0000', 'last_clean': '2024-03-24T00:43:08.800646+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:08.810555+0000', 'last_undegraded': '2024-03-24T00:43:08.810555+0000', 'last_fullsized': '2024-03-24T00:43:08.810555+0000', 'mapping_epoch': 73, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 73, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T00:43:14.554 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.0 is not active+clean 2024-03-24T00:43:14.554 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.554 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.7 is not active+clean 2024-03-24T00:43:14.554 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 73, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:43:08.806959+0000', 'last_change': '2024-03-24T00:43:08.806959+0000', 'last_active': '2024-03-24T00:43:08.800646+0000', 'last_peered': '2024-03-24T00:43:08.800646+0000', 'last_clean': '2024-03-24T00:43:08.800646+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:08.806959+0000', 'last_undegraded': '2024-03-24T00:43:08.806959+0000', 'last_fullsized': '2024-03-24T00:43:08.806959+0000', 'mapping_epoch': 73, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 73, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-24T00:43:14.554 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.1 is not active+clean 2024-03-24T00:43:14.554 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.555 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.6 is not active+clean 2024-03-24T00:43:14.555 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 73, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:43:08.812359+0000', 'last_change': '2024-03-24T00:43:08.812359+0000', 'last_active': '2024-03-24T00:43:08.800646+0000', 'last_peered': '2024-03-24T00:43:08.800646+0000', 'last_clean': '2024-03-24T00:43:08.800646+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:08.812359+0000', 'last_undegraded': '2024-03-24T00:43:08.812359+0000', 'last_fullsized': '2024-03-24T00:43:08.812359+0000', 'mapping_epoch': 73, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 73, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T00:43:14.555 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.2 is not active+clean 2024-03-24T00:43:14.555 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.555 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.5 is not active+clean 2024-03-24T00:43:14.555 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 73, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:43:08.809204+0000', 'last_change': '2024-03-24T00:43:08.809204+0000', 'last_active': '2024-03-24T00:43:08.800646+0000', 'last_peered': '2024-03-24T00:43:08.800646+0000', 'last_clean': '2024-03-24T00:43:08.800646+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:08.809204+0000', 'last_undegraded': '2024-03-24T00:43:08.809204+0000', 'last_fullsized': '2024-03-24T00:43:08.809204+0000', 'mapping_epoch': 73, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 73, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T00:43:14.555 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.3 is not active+clean 2024-03-24T00:43:14.555 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.555 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.4 is not active+clean 2024-03-24T00:43:14.555 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 73, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:43:08.809754+0000', 'last_change': '2024-03-24T00:43:08.809754+0000', 'last_active': '2024-03-24T00:43:08.800646+0000', 'last_peered': '2024-03-24T00:43:08.800646+0000', 'last_clean': '2024-03-24T00:43:08.800646+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:08.809754+0000', 'last_undegraded': '2024-03-24T00:43:08.809754+0000', 'last_fullsized': '2024-03-24T00:43:08.809754+0000', 'mapping_epoch': 73, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 73, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T00:43:14.555 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.4 is not active+clean 2024-03-24T00:43:14.555 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.556 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3 is not active+clean 2024-03-24T00:43:14.556 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 73, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:43:08.816382+0000', 'last_change': '2024-03-24T00:43:08.816382+0000', 'last_active': '2024-03-24T00:43:08.800646+0000', 'last_peered': '2024-03-24T00:43:08.800646+0000', 'last_clean': '2024-03-24T00:43:08.800646+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:08.816382+0000', 'last_undegraded': '2024-03-24T00:43:08.816382+0000', 'last_fullsized': '2024-03-24T00:43:08.816382+0000', 'mapping_epoch': 73, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 73, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-24T00:43:14.556 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.5 is not active+clean 2024-03-24T00:43:14.556 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.556 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2 is not active+clean 2024-03-24T00:43:14.556 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 73, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:43:08.806382+0000', 'last_change': '2024-03-24T00:43:08.806382+0000', 'last_active': '2024-03-24T00:43:08.800646+0000', 'last_peered': '2024-03-24T00:43:08.800646+0000', 'last_clean': '2024-03-24T00:43:08.800646+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:08.806382+0000', 'last_undegraded': '2024-03-24T00:43:08.806382+0000', 'last_fullsized': '2024-03-24T00:43:08.806382+0000', 'mapping_epoch': 73, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 73, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-24T00:43:14.556 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.6 is not active+clean 2024-03-24T00:43:14.556 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.556 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1 is not active+clean 2024-03-24T00:43:14.556 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 73, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:43:08.815930+0000', 'last_change': '2024-03-24T00:43:08.815930+0000', 'last_active': '2024-03-24T00:43:08.800646+0000', 'last_peered': '2024-03-24T00:43:08.800646+0000', 'last_clean': '2024-03-24T00:43:08.800646+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:08.815930+0000', 'last_undegraded': '2024-03-24T00:43:08.815930+0000', 'last_fullsized': '2024-03-24T00:43:08.815930+0000', 'mapping_epoch': 73, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 73, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-24T00:43:14.557 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.7 is not active+clean 2024-03-24T00:43:14.557 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.557 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.0 is not active+clean 2024-03-24T00:43:14.557 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 73, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:43:08.807731+0000', 'last_change': '2024-03-24T00:43:08.807731+0000', 'last_active': '2024-03-24T00:43:08.800646+0000', 'last_peered': '2024-03-24T00:43:08.800646+0000', 'last_clean': '2024-03-24T00:43:08.800646+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:08.807731+0000', 'last_undegraded': '2024-03-24T00:43:08.807731+0000', 'last_fullsized': '2024-03-24T00:43:08.807731+0000', 'mapping_epoch': 73, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 73, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-24T00:43:14.557 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.d is not active+clean 2024-03-24T00:43:14.557 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:43:09.804492+0000', 'last_change': '2024-03-24T00:43:09.804492+0000', 'last_active': '2024-03-24T00:43:09.804492+0000', 'last_peered': '2024-03-24T00:43:09.804492+0000', 'last_clean': '2024-03-24T00:43:09.804492+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:09.804492+0000', 'last_undegraded': '2024-03-24T00:43:09.804492+0000', 'last_fullsized': '2024-03-24T00:43:09.804492+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:09.804492+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:43:14.557 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.a is not active+clean 2024-03-24T00:43:14.557 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 73, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:43:08.810712+0000', 'last_change': '2024-03-24T00:43:08.810712+0000', 'last_active': '2024-03-24T00:43:08.800646+0000', 'last_peered': '2024-03-24T00:43:08.800646+0000', 'last_clean': '2024-03-24T00:43:08.800646+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:43:08.810712+0000', 'last_undegraded': '2024-03-24T00:43:08.810712+0000', 'last_fullsized': '2024-03-24T00:43:08.810712+0000', 'mapping_epoch': 73, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 73, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'last_clean_scrub_stamp': '2024-03-24T00:43:08.800646+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T00:43:14.558 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-24T00:43:14.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.677+0000 7f30d344b640 1 Processor -- start 2024-03-24T00:43:14.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.677+0000 7f30d344b640 1 -- start start 2024-03-24T00:43:14.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.679+0000 7f30d344b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f30cc15bf70 0x7f30cc0cc690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:14.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.679+0000 7f30d344b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f30cc0ce930 0x7f30cc0b1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:14.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.679+0000 7f30d344b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30cc0ccfe0 con 0x7f30cc0cfc70 2024-03-24T00:43:14.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.679+0000 7f30d344b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30cc0cd2c0 con 0x7f30cc15bf70 2024-03-24T00:43:14.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.679+0000 7f30d344b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30cc0cd5a0 con 0x7f30cc0ce930 2024-03-24T00:43:14.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.679+0000 7f30d09e6640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f30cc0cfc70 0x7f30cc0ce220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37514/0 (socket says 172.21.15.67:37514) 2024-03-24T00:43:14.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.679+0000 7f30d09e6640 1 -- 172.21.15.67:0/3639455124 learned_addr learned my addr 172.21.15.67:0/3639455124 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:43:14.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.679+0000 7f30c3fff640 1 -- 172.21.15.67:0/3639455124 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1732592335 0 0) 0x7f30cc0ccfe0 con 0x7f30cc0cfc70 2024-03-24T00:43:14.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.679+0000 7f30d19e8640 1 --2- 172.21.15.67:0/3639455124 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f30cc0ce930 0x7f30cc0b1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:14.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.679+0000 7f30d11e7640 1 --2- 172.21.15.67:0/3639455124 >> v2:172.21.15.72:3300/0 conn(0x7f30cc15bf70 0x7f30cc0cc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:14.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.679+0000 7f30c3fff640 1 -- 172.21.15.67:0/3639455124 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30b8003520 con 0x7f30cc0cfc70 2024-03-24T00:43:14.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.679+0000 7f30c3fff640 1 -- 172.21.15.67:0/3639455124 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1034419883 0 0) 0x7f30b8003520 con 0x7f30cc0cfc70 2024-03-24T00:43:14.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.679+0000 7f30c3fff640 1 -- 172.21.15.67:0/3639455124 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f30cc0ce930 msgr2=0x7f30cc0b1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:14.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.679+0000 7f30c3fff640 1 --2- 172.21.15.67:0/3639455124 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f30cc0ce930 0x7f30cc0b1cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:14.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.679+0000 7f30c3fff640 1 -- 172.21.15.67:0/3639455124 >> v2:172.21.15.72:3300/0 conn(0x7f30cc15bf70 msgr2=0x7f30cc0cc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:14.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.679+0000 7f30c3fff640 1 --2- 172.21.15.67:0/3639455124 >> v2:172.21.15.72:3300/0 conn(0x7f30cc15bf70 0x7f30cc0cc690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:14.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.679+0000 7f30c3fff640 1 -- 172.21.15.67:0/3639455124 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30cc0cd880 con 0x7f30cc0cfc70 2024-03-24T00:43:14.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.679+0000 7f30c3fff640 1 -- 172.21.15.67:0/3639455124 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f30bc001670 con 0x7f30cc0cfc70 2024-03-24T00:43:14.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.679+0000 7f30c3fff640 1 -- 172.21.15.67:0/3639455124 >> v1:172.21.15.67:6789/0 conn(0x7f30cc0cfc70 legacy=0x7f30cc0ce220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:14.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.679+0000 7f30c3fff640 1 --2- 172.21.15.67:0/3639455124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30b80039b0 0x7f30b8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:14.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.679+0000 7f30c3fff640 1 --2- 172.21.15.67:0/3639455124 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f30b8004800 0x7f30b8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:14.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.680+0000 7f30c3fff640 1 --2- 172.21.15.67:0/3639455124 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f30cc0ce930 0x7f30b80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:14.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.680+0000 7f30c3fff640 1 -- 172.21.15.67:0/3639455124 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30cc0cd2c0 con 0x7f30b80039b0 2024-03-24T00:43:14.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.680+0000 7f30c3fff640 1 -- 172.21.15.67:0/3639455124 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30cc0cd5a0 con 0x7f30b8004800 2024-03-24T00:43:14.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.680+0000 7f30c3fff640 1 -- 172.21.15.67:0/3639455124 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30b800b9c0 con 0x7f30cc0ce930 2024-03-24T00:43:14.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.680+0000 7f30d11e7640 1 --2- 172.21.15.67:0/3639455124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30b80039b0 0x7f30b8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:14.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.680+0000 7f30d09e6640 1 --2- 172.21.15.67:0/3639455124 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f30b8004800 0x7f30b8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:14.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.680+0000 7f30d19e8640 1 --2- 172.21.15.67:0/3639455124 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f30cc0ce930 0x7f30b80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:14.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.680+0000 7f30d11e7640 1 -- 172.21.15.67:0/3639455124 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f30cc0ce930 msgr2=0x7f30b80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:14.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.680+0000 7f30d11e7640 1 --2- 172.21.15.67:0/3639455124 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f30cc0ce930 0x7f30b80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:14.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.680+0000 7f30d11e7640 1 -- 172.21.15.67:0/3639455124 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f30b8004800 msgr2=0x7f30b8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:14.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.680+0000 7f30d11e7640 1 --2- 172.21.15.67:0/3639455124 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f30b8004800 0x7f30b8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:14.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.680+0000 7f30d11e7640 1 -- 172.21.15.67:0/3639455124 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30b8003520 con 0x7f30b80039b0 2024-03-24T00:43:14.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.681+0000 7f30d11e7640 1 --2- 172.21.15.67:0/3639455124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30b80039b0 0x7f30b8003dd0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f30c8009ac0 tx=0x7f30b8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71f4fe194eda3d02 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:14.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.681+0000 7f30c3fff640 1 -- 172.21.15.67:0/3639455124 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c8039070 con 0x7f30b80039b0 2024-03-24T00:43:14.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.681+0000 7f30c3fff640 1 -- 172.21.15.67:0/3639455124 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30c802c710 con 0x7f30b80039b0 2024-03-24T00:43:14.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.681+0000 7f30c3fff640 1 -- 172.21.15.67:0/3639455124 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c8009690 con 0x7f30b80039b0 2024-03-24T00:43:14.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.681+0000 7f30d344b640 1 -- 172.21.15.67:0/3639455124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30b80039b0 msgr2=0x7f30b8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:14.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.681+0000 7f30d344b640 1 --2- 172.21.15.67:0/3639455124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30b80039b0 0x7f30b8003dd0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f30c8009ac0 tx=0x7f30b8005830 comp rx=0 tx=0).stop 2024-03-24T00:43:14.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.681+0000 7f30d19e8640 1 -- 172.21.15.67:0/3639455124 reap_dead start 2024-03-24T00:43:14.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.681+0000 7f30d344b640 1 -- 172.21.15.67:0/3639455124 shutdown_connections 2024-03-24T00:43:14.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.681+0000 7f30d344b640 1 -- 172.21.15.67:0/3639455124 >> 172.21.15.67:0/3639455124 conn(0x7f30cc0d1bf0 msgr2=0x7f30cc0d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:14.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.682+0000 7f30d344b640 1 -- 172.21.15.67:0/3639455124 shutdown_connections 2024-03-24T00:43:14.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.682+0000 7f30d344b640 1 -- 172.21.15.67:0/3639455124 wait complete. 2024-03-24T00:43:14.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.683+0000 7f30d344b640 1 Processor -- start 2024-03-24T00:43:14.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.683+0000 7f30d344b640 1 -- start start 2024-03-24T00:43:14.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.683+0000 7f30d344b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f30b80039b0 0x7f30cc0b47e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:14.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.683+0000 7f30d344b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30cc1591a0 0x7f30cc150070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:14.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.683+0000 7f30d344b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f30cc1f1ad0 0x7f30cc1f1f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:14.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.683+0000 7f30d344b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30cc0ccd50 con 0x7f30cc1591a0 2024-03-24T00:43:14.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.683+0000 7f30d344b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30cc0ccbd0 con 0x7f30b80039b0 2024-03-24T00:43:14.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.683+0000 7f30d344b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30cc0b1cd0 con 0x7f30cc1f1ad0 2024-03-24T00:43:14.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.684+0000 7f30d09e6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30cc1591a0 0x7f30cc150070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:14.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.684+0000 7f30d09e6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30cc1591a0 0x7f30cc150070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50368/0 (socket says 172.21.15.67:50368) 2024-03-24T00:43:14.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.684+0000 7f30d09e6640 1 -- 172.21.15.67:0/3209842642 learned_addr learned my addr 172.21.15.67:0/3209842642 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:14.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.684+0000 7f30d11e7640 1 --2- 172.21.15.67:0/3209842642 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f30b80039b0 0x7f30cc0b47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:14.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.684+0000 7f30d19e8640 1 --2- 172.21.15.67:0/3209842642 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f30cc1f1ad0 0x7f30cc1f1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:14.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.684+0000 7f30d09e6640 1 -- 172.21.15.67:0/3209842642 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f30cc1f1ad0 msgr2=0x7f30cc1f1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:14.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.684+0000 7f30d09e6640 1 --2- 172.21.15.67:0/3209842642 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f30cc1f1ad0 0x7f30cc1f1f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:14.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.684+0000 7f30d09e6640 1 -- 172.21.15.67:0/3209842642 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f30b80039b0 msgr2=0x7f30cc0b47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:14.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.684+0000 7f30d09e6640 1 --2- 172.21.15.67:0/3209842642 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f30b80039b0 0x7f30cc0b47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:14.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.684+0000 7f30d09e6640 1 -- 172.21.15.67:0/3209842642 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30cc0b4e70 con 0x7f30cc1591a0 2024-03-24T00:43:14.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.684+0000 7f30d09e6640 1 --2- 172.21.15.67:0/3209842642 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30cc1591a0 0x7f30cc150070 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f30bc007370 tx=0x7f30bc007830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:14.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.684+0000 7f30c1ffb640 1 -- 172.21.15.67:0/3209842642 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30bc00f9c0 con 0x7f30cc1591a0 2024-03-24T00:43:14.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.684+0000 7f30c1ffb640 1 -- 172.21.15.67:0/3209842642 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30bc018070 con 0x7f30cc1591a0 2024-03-24T00:43:14.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.684+0000 7f30d344b640 1 -- 172.21.15.67:0/3209842642 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30cc15df00 con 0x7f30cc1591a0 2024-03-24T00:43:14.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.684+0000 7f30c1ffb640 1 -- 172.21.15.67:0/3209842642 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30bc00fda0 con 0x7f30cc1591a0 2024-03-24T00:43:14.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.685+0000 7f30d344b640 1 -- 172.21.15.67:0/3209842642 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30cc15e480 con 0x7f30cc1591a0 2024-03-24T00:43:14.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.686+0000 7f30d344b640 1 -- 172.21.15.67:0/3209842642 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30cc04f400 con 0x7f30cc1591a0 2024-03-24T00:43:14.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.686+0000 7f30c1ffb640 1 -- 172.21.15.67:0/3209842642 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f30bc01c410 con 0x7f30cc1591a0 2024-03-24T00:43:14.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.686+0000 7f30c1ffb640 1 --2- 172.21.15.67:0/3209842642 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f30b006bc00 0x7f30b006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:14.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.686+0000 7f30c1ffb640 1 -- 172.21.15.67:0/3209842642 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f30bc08a980 con 0x7f30cc1591a0 2024-03-24T00:43:14.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.686+0000 7f30d11e7640 1 --2- 172.21.15.67:0/3209842642 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f30b006bc00 0x7f30b006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:14.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.687+0000 7f30d11e7640 1 --2- 172.21.15.67:0/3209842642 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f30b006bc00 0x7f30b006e0c0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f30c8036000 tx=0x7f30c8030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:14.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:14.691+0000 7f30c1ffb640 1 -- 172.21.15.67:0/3209842642 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f30bc0592e0 con 0x7f30cc1591a0 2024-03-24T00:43:15.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.160+0000 7f30d344b640 1 -- 172.21.15.67:0/3209842642 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f30cc08ba10 con 0x7f30cc1591a0 2024-03-24T00:43:15.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.161+0000 7f30c1ffb640 1 -- 172.21.15.67:0/3209842642 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1109 (secure 0 0 0) 0x7f30bc058be0 con 0x7f30cc1591a0 2024-03-24T00:43:15.161 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:43:15.162 INFO:teuthology.orchestra.run.smithi067.stdout:{"fsid":"de1d0178-808e-49ca-a986-e297e523b976","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":288,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":76,"num_osds":8,"num_up_osds":8,"osd_up_since":1711240718,"num_in_osds":8,"osd_in_since":1711240709,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"unknown","count":64},{"state_name":"creating+peering","count":32},{"state_name":"active+clean","count":1}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":233869312,"bytes_avail":772860243968,"bytes_total":773094113280,"unknown_pgs_ratio":0.65979379415512085,"inactive_pgs_ratio":0.32989689707756042},"fsmap":{"epoch":40,"id":5,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":5,"rank":0,"name":"c","status":"up:active","gid":6894}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-03-24T00:42:55.378346+0000","services":{}},"progress_events":{}} 2024-03-24T00:43:15.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.162+0000 7f30d344b640 1 -- 172.21.15.67:0/3209842642 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f30b006bc00 msgr2=0x7f30b006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:15.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.162+0000 7f30d344b640 1 --2- 172.21.15.67:0/3209842642 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f30b006bc00 0x7f30b006e0c0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f30c8036000 tx=0x7f30c8030000 comp rx=0 tx=0).stop 2024-03-24T00:43:15.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.162+0000 7f30d344b640 1 -- 172.21.15.67:0/3209842642 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30cc1591a0 msgr2=0x7f30cc150070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:15.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.162+0000 7f30d344b640 1 --2- 172.21.15.67:0/3209842642 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30cc1591a0 0x7f30cc150070 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f30bc007370 tx=0x7f30bc007830 comp rx=0 tx=0).stop 2024-03-24T00:43:15.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.163+0000 7f30d344b640 1 -- 172.21.15.67:0/3209842642 shutdown_connections 2024-03-24T00:43:15.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.163+0000 7f30d344b640 1 --2- 172.21.15.67:0/3209842642 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f30b006bc00 0x7f30b006e0c0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:15.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.163+0000 7f30d344b640 1 --2- 172.21.15.67:0/3209842642 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f30cc1f1ad0 0x7f30cc1f1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:15.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.163+0000 7f30d344b640 1 --2- 172.21.15.67:0/3209842642 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30cc1591a0 0x7f30cc150070 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:15.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.163+0000 7f30d344b640 1 --2- 172.21.15.67:0/3209842642 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f30b80039b0 0x7f30cc0b47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:15.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.163+0000 7f30d344b640 1 -- 172.21.15.67:0/3209842642 >> 172.21.15.67:0/3209842642 conn(0x7f30cc0d1bf0 msgr2=0x7f30cc06fce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:15.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.163+0000 7f30d344b640 1 -- 172.21.15.67:0/3209842642 shutdown_connections 2024-03-24T00:43:15.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.163+0000 7f30d344b640 1 -- 172.21.15.67:0/3209842642 wait complete. 2024-03-24T00:43:15.174 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-24T00:43:15.174 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:43:15.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.276+0000 7ff3ae777640 1 Processor -- start 2024-03-24T00:43:15.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.276+0000 7ff3ae777640 1 -- start start 2024-03-24T00:43:15.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.277+0000 7ff3ae777640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff3a815dec0 0x7ff3a80ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:15.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.277+0000 7ff3ae777640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3a80cf640 con 0x7ff3a815e2a0 2024-03-24T00:43:15.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.277+0000 7ff3ae777640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3a80cf080 con 0x7ff3a80d0ef0 2024-03-24T00:43:15.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.277+0000 7ff3ae777640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a80cee30 con 0x7ff3a815dec0 2024-03-24T00:43:15.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.277+0000 7ff3a77fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff3a815e2a0 0x7ff3a80d07e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37518/0 (socket says 172.21.15.67:37518) 2024-03-24T00:43:15.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.277+0000 7ff3a77fe640 1 -- 172.21.15.67:0/704484507 learned_addr learned my addr 172.21.15.67:0/704484507 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:43:15.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.277+0000 7ff3a7fff640 1 --2- 172.21.15.67:0/704484507 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff3a815dec0 0x7ff3a80ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:15.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.278+0000 7ff3a6ffd640 1 -- 172.21.15.67:0/704484507 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2000436115 0 0) 0x7ff39c001110 con 0x7ff3a815e2a0 2024-03-24T00:43:15.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.278+0000 7ff3a6ffd640 1 -- 172.21.15.67:0/704484507 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff390003520 con 0x7ff3a815e2a0 2024-03-24T00:43:15.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.278+0000 7ff3a6ffd640 1 -- 172.21.15.67:0/704484507 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 102706638 0 0) 0x7ff3a80cf080 con 0x7ff3a80d0ef0 2024-03-24T00:43:15.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.278+0000 7ff3a6ffd640 1 -- 172.21.15.67:0/704484507 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff39c001110 con 0x7ff3a80d0ef0 2024-03-24T00:43:15.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.278+0000 7ff3a7fff640 1 -- 172.21.15.67:0/704484507 >> v1:172.21.15.72:6789/0 conn(0x7ff3a80d0ef0 legacy=0x7ff3a80d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:15.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.278+0000 7ff3a7fff640 1 -- 172.21.15.67:0/704484507 >> v1:172.21.15.67:6789/0 conn(0x7ff3a815e2a0 legacy=0x7ff3a80d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:15.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.278+0000 7ff3a6ffd640 1 -- 172.21.15.67:0/704484507 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 182359811 0 0) 0x7ff390003520 con 0x7ff3a815e2a0 2024-03-24T00:43:15.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.278+0000 7ff3a7fff640 1 -- 172.21.15.67:0/704484507 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3a80cf360 con 0x7ff3a815dec0 2024-03-24T00:43:15.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.279+0000 7ff3a7fff640 1 --2- 172.21.15.67:0/704484507 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff3a815dec0 0x7ff3a80ce630 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff3980097d0 tx=0x7ff39802b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7250006ffb8f3c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:15.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.279+0000 7ff3a6ffd640 1 -- 172.21.15.67:0/704484507 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff398037070 con 0x7ff3a815dec0 2024-03-24T00:43:15.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.279+0000 7ff3a6ffd640 1 -- 172.21.15.67:0/704484507 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff398030510 con 0x7ff3a815dec0 2024-03-24T00:43:15.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.279+0000 7ff3a6ffd640 1 -- 172.21.15.67:0/704484507 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff39c001840 con 0x7ff3a815dec0 2024-03-24T00:43:15.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.279+0000 7ff3ae777640 1 -- 172.21.15.67:0/704484507 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff3a815dec0 msgr2=0x7ff3a80ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:15.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.279+0000 7ff3ae777640 1 --2- 172.21.15.67:0/704484507 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff3a815dec0 0x7ff3a80ce630 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff3980097d0 tx=0x7ff39802b770 comp rx=0 tx=0).stop 2024-03-24T00:43:15.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.279+0000 7ff3ae777640 1 -- 172.21.15.67:0/704484507 shutdown_connections 2024-03-24T00:43:15.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.279+0000 7ff3ae777640 1 --2- 172.21.15.67:0/704484507 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff3a815dec0 0x7ff3a80ce630 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:15.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.279+0000 7ff3ae777640 1 -- 172.21.15.67:0/704484507 >> 172.21.15.67:0/704484507 conn(0x7ff3a806c810 msgr2=0x7ff3a806cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:15.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.279+0000 7ff3ae777640 1 -- 172.21.15.67:0/704484507 shutdown_connections 2024-03-24T00:43:15.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.279+0000 7ff3ae777640 1 -- 172.21.15.67:0/704484507 wait complete. 2024-03-24T00:43:15.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.280+0000 7ff3ae777640 1 Processor -- start 2024-03-24T00:43:15.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.281+0000 7ff3ae777640 1 -- start start 2024-03-24T00:43:15.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.281+0000 7ff3ae777640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3a80d0ef0 0x7ff3a81f4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:15.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.281+0000 7ff3ae777640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3a815dec0 0x7ff3a80b49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:15.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.281+0000 7ff3ae777640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff3a815e2a0 0x7ff3a80b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:15.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.281+0000 7ff3ae777640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a8162a40 con 0x7ff3a80d0ef0 2024-03-24T00:43:15.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.281+0000 7ff3ae777640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a81628c0 con 0x7ff3a815dec0 2024-03-24T00:43:15.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.281+0000 7ff3ae777640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a80d2120 con 0x7ff3a815e2a0 2024-03-24T00:43:15.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.281+0000 7ff3a7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3a80d0ef0 0x7ff3a81f4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:15.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.282+0000 7ff3a7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3a80d0ef0 0x7ff3a81f4cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50378/0 (socket says 172.21.15.67:50378) 2024-03-24T00:43:15.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.282+0000 7ff3a7fff640 1 -- 172.21.15.67:0/1248730812 learned_addr learned my addr 172.21.15.67:0/1248730812 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:15.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.282+0000 7ff3a77fe640 1 --2- 172.21.15.67:0/1248730812 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3a815dec0 0x7ff3a80b49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:15.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.282+0000 7ff3acd14640 1 --2- 172.21.15.67:0/1248730812 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff3a815e2a0 0x7ff3a80b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:15.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.282+0000 7ff3a7fff640 1 -- 172.21.15.67:0/1248730812 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff3a815e2a0 msgr2=0x7ff3a80b4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:15.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.282+0000 7ff3a7fff640 1 --2- 172.21.15.67:0/1248730812 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff3a815e2a0 0x7ff3a80b4f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:15.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.282+0000 7ff3a7fff640 1 -- 172.21.15.67:0/1248730812 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3a815dec0 msgr2=0x7ff3a80b49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:15.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.282+0000 7ff3a7fff640 1 --2- 172.21.15.67:0/1248730812 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3a815dec0 0x7ff3a80b49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:15.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.282+0000 7ff3a7fff640 1 -- 172.21.15.67:0/1248730812 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3a80caea0 con 0x7ff3a80d0ef0 2024-03-24T00:43:15.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.282+0000 7ff3a7fff640 1 --2- 172.21.15.67:0/1248730812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3a80d0ef0 0x7ff3a81f4cf0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7ff398009720 tx=0x7ff3980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:15.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.282+0000 7ff3a4ff9640 1 -- 172.21.15.67:0/1248730812 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff398042070 con 0x7ff3a80d0ef0 2024-03-24T00:43:15.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.282+0000 7ff3a4ff9640 1 -- 172.21.15.67:0/1248730812 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff398030ec0 con 0x7ff3a80d0ef0 2024-03-24T00:43:15.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.282+0000 7ff3a4ff9640 1 -- 172.21.15.67:0/1248730812 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff398037040 con 0x7ff3a80d0ef0 2024-03-24T00:43:15.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.282+0000 7ff3ae777640 1 -- 172.21.15.67:0/1248730812 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3a80cb180 con 0x7ff3a80d0ef0 2024-03-24T00:43:15.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.282+0000 7ff3ae777640 1 -- 172.21.15.67:0/1248730812 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3a80cb740 con 0x7ff3a80d0ef0 2024-03-24T00:43:15.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.283+0000 7ff3ae777640 1 -- 172.21.15.67:0/1248730812 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3a804f400 con 0x7ff3a80d0ef0 2024-03-24T00:43:15.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.284+0000 7ff3a4ff9640 1 -- 172.21.15.67:0/1248730812 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff39803a040 con 0x7ff3a80d0ef0 2024-03-24T00:43:15.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.284+0000 7ff3a4ff9640 1 --2- 172.21.15.67:0/1248730812 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff39406bbd0 0x7ff39406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:15.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.285+0000 7ff3a4ff9640 1 -- 172.21.15.67:0/1248730812 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff3980ac350 con 0x7ff3a80d0ef0 2024-03-24T00:43:15.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.285+0000 7ff3a77fe640 1 --2- 172.21.15.67:0/1248730812 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff39406bbd0 0x7ff39406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:15.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.285+0000 7ff3a77fe640 1 --2- 172.21.15.67:0/1248730812 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff39406bbd0 0x7ff39406e090 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7ff39c001bb0 tx=0x7ff39c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:15.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.289+0000 7ff3a4ff9640 1 -- 172.21.15.67:0/1248730812 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff39807acb0 con 0x7ff3a80d0ef0 2024-03-24T00:43:15.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.498+0000 7ff3ae777640 1 -- 172.21.15.67:0/1248730812 --> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff3a80d1aa0 con 0x7ff39406bbd0 2024-03-24T00:43:15.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.504+0000 7ff3a4ff9640 1 -- 172.21.15.67:0/1248730812 <== mgr.4105 v2:172.21.15.72:6832/2768292274 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252073 (secure 0 0 0) 0x7ff3a80d1aa0 con 0x7ff39406bbd0 2024-03-24T00:43:15.508 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:43:15.509 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-24T00:43:15.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.507+0000 7ff3ae777640 1 -- 172.21.15.67:0/1248730812 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff39406bbd0 msgr2=0x7ff39406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:15.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.507+0000 7ff3ae777640 1 --2- 172.21.15.67:0/1248730812 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff39406bbd0 0x7ff39406e090 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7ff39c001bb0 tx=0x7ff39c009040 comp rx=0 tx=0).stop 2024-03-24T00:43:15.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.507+0000 7ff3ae777640 1 -- 172.21.15.67:0/1248730812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3a80d0ef0 msgr2=0x7ff3a81f4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:15.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.507+0000 7ff3ae777640 1 --2- 172.21.15.67:0/1248730812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3a80d0ef0 0x7ff3a81f4cf0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7ff398009720 tx=0x7ff3980046c0 comp rx=0 tx=0).stop 2024-03-24T00:43:15.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.507+0000 7ff3ae777640 1 -- 172.21.15.67:0/1248730812 shutdown_connections 2024-03-24T00:43:15.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.507+0000 7ff3ae777640 1 --2- 172.21.15.67:0/1248730812 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff39406bbd0 0x7ff39406e090 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:15.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.507+0000 7ff3ae777640 1 --2- 172.21.15.67:0/1248730812 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff3a815e2a0 0x7ff3a80b4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:15.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.507+0000 7ff3ae777640 1 --2- 172.21.15.67:0/1248730812 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3a815dec0 0x7ff3a80b49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:15.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.507+0000 7ff3ae777640 1 --2- 172.21.15.67:0/1248730812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3a80d0ef0 0x7ff3a81f4cf0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:15.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.507+0000 7ff3ae777640 1 -- 172.21.15.67:0/1248730812 >> 172.21.15.67:0/1248730812 conn(0x7ff3a806c810 msgr2=0x7ff3a815c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:15.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.507+0000 7ff3ae777640 1 -- 172.21.15.67:0/1248730812 shutdown_connections 2024-03-24T00:43:15.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:15.507+0000 7ff3ae777640 1 -- 172.21.15.67:0/1248730812 wait complete. 2024-03-24T00:43:15.520 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":207,"stamp":"2024-03-24T00:43:15.382751+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":187,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":233000,"kb_used_data":8488,"kb_used_omap":12,"kb_used_meta":224371,"kb_avail":754741720,"statfs":{"total":773094113280,"available":772855521280,"internally_reserved":0,"allocated":8691712,"data_stored":4584284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":229756465},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001447"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.853378+0000","last_change":"2024-03-24T00:43:10.824156+0000","last_active":"2024-03-24T00:43:12.853378+0000","last_peered":"2024-03-24T00:43:12.853378+0000","last_clean":"2024-03-24T00:43:12.853378+0000","last_became_active":"2024-03-24T00:43:10.822882+0000","last_became_peered":"2024-03-24T00:43:10.822882+0000","last_unstale":"2024-03-24T00:43:12.853378+0000","last_undegraded":"2024-03-24T00:43:12.853378+0000","last_fullsized":"2024-03-24T00:43:12.853378+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:38:33.107986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.830453+0000","last_change":"2024-03-24T00:43:10.838215+0000","last_active":"2024-03-24T00:43:11.830453+0000","last_peered":"2024-03-24T00:43:11.830453+0000","last_clean":"2024-03-24T00:43:11.830453+0000","last_became_active":"2024-03-24T00:43:10.837975+0000","last_became_peered":"2024-03-24T00:43:10.837975+0000","last_unstale":"2024-03-24T00:43:11.830453+0000","last_undegraded":"2024-03-24T00:43:11.830453+0000","last_fullsized":"2024-03-24T00:43:11.830453+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:53:11.321434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.102061+0000","last_change":"2024-03-24T00:43:10.838246+0000","last_active":"2024-03-24T00:43:12.102061+0000","last_peered":"2024-03-24T00:43:12.102061+0000","last_clean":"2024-03-24T00:43:12.102061+0000","last_became_active":"2024-03-24T00:43:10.837968+0000","last_became_peered":"2024-03-24T00:43:10.837968+0000","last_unstale":"2024-03-24T00:43:12.102061+0000","last_undegraded":"2024-03-24T00:43:12.102061+0000","last_fullsized":"2024-03-24T00:43:12.102061+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:36:52.211347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.831102+0000","last_change":"2024-03-24T00:43:10.830481+0000","last_active":"2024-03-24T00:43:11.831102+0000","last_peered":"2024-03-24T00:43:11.831102+0000","last_clean":"2024-03-24T00:43:11.831102+0000","last_became_active":"2024-03-24T00:43:10.830102+0000","last_became_peered":"2024-03-24T00:43:10.830102+0000","last_unstale":"2024-03-24T00:43:11.831102+0000","last_undegraded":"2024-03-24T00:43:11.831102+0000","last_fullsized":"2024-03-24T00:43:11.831102+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:14:34.576611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.831351+0000","last_change":"2024-03-24T00:43:10.819692+0000","last_active":"2024-03-24T00:43:11.831351+0000","last_peered":"2024-03-24T00:43:11.831351+0000","last_clean":"2024-03-24T00:43:11.831351+0000","last_became_active":"2024-03-24T00:43:10.819425+0000","last_became_peered":"2024-03-24T00:43:10.819425+0000","last_unstale":"2024-03-24T00:43:11.831351+0000","last_undegraded":"2024-03-24T00:43:11.831351+0000","last_fullsized":"2024-03-24T00:43:11.831351+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:17:59.489389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.851935+0000","last_change":"2024-03-24T00:43:10.833651+0000","last_active":"2024-03-24T00:43:12.851935+0000","last_peered":"2024-03-24T00:43:12.851935+0000","last_clean":"2024-03-24T00:43:12.851935+0000","last_became_active":"2024-03-24T00:43:10.833374+0000","last_became_peered":"2024-03-24T00:43:10.833374+0000","last_unstale":"2024-03-24T00:43:12.851935+0000","last_undegraded":"2024-03-24T00:43:12.851935+0000","last_fullsized":"2024-03-24T00:43:12.851935+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:30:53.960721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.852143+0000","last_change":"2024-03-24T00:43:10.831820+0000","last_active":"2024-03-24T00:43:12.852143+0000","last_peered":"2024-03-24T00:43:12.852143+0000","last_clean":"2024-03-24T00:43:12.852143+0000","last_became_active":"2024-03-24T00:43:10.830998+0000","last_became_peered":"2024-03-24T00:43:10.830998+0000","last_unstale":"2024-03-24T00:43:12.852143+0000","last_undegraded":"2024-03-24T00:43:12.852143+0000","last_fullsized":"2024-03-24T00:43:12.852143+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:19:48.048357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.830751+0000","last_change":"2024-03-24T00:43:10.822376+0000","last_active":"2024-03-24T00:43:11.830751+0000","last_peered":"2024-03-24T00:43:11.830751+0000","last_clean":"2024-03-24T00:43:11.830751+0000","last_became_active":"2024-03-24T00:43:10.822042+0000","last_became_peered":"2024-03-24T00:43:10.822042+0000","last_unstale":"2024-03-24T00:43:11.830751+0000","last_undegraded":"2024-03-24T00:43:11.830751+0000","last_fullsized":"2024-03-24T00:43:11.830751+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:17:52.626613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.853828+0000","last_change":"2024-03-24T00:43:10.837255+0000","last_active":"2024-03-24T00:43:12.853828+0000","last_peered":"2024-03-24T00:43:12.853828+0000","last_clean":"2024-03-24T00:43:12.853828+0000","last_became_active":"2024-03-24T00:43:10.836977+0000","last_became_peered":"2024-03-24T00:43:10.836977+0000","last_unstale":"2024-03-24T00:43:12.853828+0000","last_undegraded":"2024-03-24T00:43:12.853828+0000","last_fullsized":"2024-03-24T00:43:12.853828+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:19:25.257173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.831304+0000","last_change":"2024-03-24T00:43:10.825991+0000","last_active":"2024-03-24T00:43:11.831304+0000","last_peered":"2024-03-24T00:43:11.831304+0000","last_clean":"2024-03-24T00:43:11.831304+0000","last_became_active":"2024-03-24T00:43:10.825554+0000","last_became_peered":"2024-03-24T00:43:10.825554+0000","last_unstale":"2024-03-24T00:43:11.831304+0000","last_undegraded":"2024-03-24T00:43:11.831304+0000","last_fullsized":"2024-03-24T00:43:11.831304+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:33:32.269240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.830316+0000","last_change":"2024-03-24T00:43:10.843472+0000","last_active":"2024-03-24T00:43:11.830316+0000","last_peered":"2024-03-24T00:43:11.830316+0000","last_clean":"2024-03-24T00:43:11.830316+0000","last_became_active":"2024-03-24T00:43:10.843160+0000","last_became_peered":"2024-03-24T00:43:10.843160+0000","last_unstale":"2024-03-24T00:43:11.830316+0000","last_undegraded":"2024-03-24T00:43:11.830316+0000","last_fullsized":"2024-03-24T00:43:11.830316+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:42:14.282601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.831176+0000","last_change":"2024-03-24T00:43:10.835510+0000","last_active":"2024-03-24T00:43:11.831176+0000","last_peered":"2024-03-24T00:43:11.831176+0000","last_clean":"2024-03-24T00:43:11.831176+0000","last_became_active":"2024-03-24T00:43:10.835227+0000","last_became_peered":"2024-03-24T00:43:10.835227+0000","last_unstale":"2024-03-24T00:43:11.831176+0000","last_undegraded":"2024-03-24T00:43:11.831176+0000","last_fullsized":"2024-03-24T00:43:11.831176+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:01:23.035377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.831398+0000","last_change":"2024-03-24T00:43:10.821164+0000","last_active":"2024-03-24T00:43:11.831398+0000","last_peered":"2024-03-24T00:43:11.831398+0000","last_clean":"2024-03-24T00:43:11.831398+0000","last_became_active":"2024-03-24T00:43:10.820709+0000","last_became_peered":"2024-03-24T00:43:10.820709+0000","last_unstale":"2024-03-24T00:43:11.831398+0000","last_undegraded":"2024-03-24T00:43:11.831398+0000","last_fullsized":"2024-03-24T00:43:11.831398+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:13:20.917390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.851944+0000","last_change":"2024-03-24T00:43:10.836706+0000","last_active":"2024-03-24T00:43:12.851944+0000","last_peered":"2024-03-24T00:43:12.851944+0000","last_clean":"2024-03-24T00:43:12.851944+0000","last_became_active":"2024-03-24T00:43:10.836363+0000","last_became_peered":"2024-03-24T00:43:10.836363+0000","last_unstale":"2024-03-24T00:43:12.851944+0000","last_undegraded":"2024-03-24T00:43:12.851944+0000","last_fullsized":"2024-03-24T00:43:12.851944+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:11:52.190154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.829593+0000","last_change":"2024-03-24T00:43:10.844813+0000","last_active":"2024-03-24T00:43:11.829593+0000","last_peered":"2024-03-24T00:43:11.829593+0000","last_clean":"2024-03-24T00:43:11.829593+0000","last_became_active":"2024-03-24T00:43:10.844464+0000","last_became_peered":"2024-03-24T00:43:10.844464+0000","last_unstale":"2024-03-24T00:43:11.829593+0000","last_undegraded":"2024-03-24T00:43:11.829593+0000","last_fullsized":"2024-03-24T00:43:11.829593+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:38:33.107986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.852137+0000","last_change":"2024-03-24T00:43:10.830783+0000","last_active":"2024-03-24T00:43:12.852137+0000","last_peered":"2024-03-24T00:43:12.852137+0000","last_clean":"2024-03-24T00:43:12.852137+0000","last_became_active":"2024-03-24T00:43:10.828872+0000","last_became_peered":"2024-03-24T00:43:10.828872+0000","last_unstale":"2024-03-24T00:43:12.852137+0000","last_undegraded":"2024-03-24T00:43:12.852137+0000","last_fullsized":"2024-03-24T00:43:12.852137+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:14:34.576611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.831745+0000","last_change":"2024-03-24T00:43:10.825448+0000","last_active":"2024-03-24T00:43:11.831745+0000","last_peered":"2024-03-24T00:43:11.831745+0000","last_clean":"2024-03-24T00:43:11.831745+0000","last_became_active":"2024-03-24T00:43:10.825185+0000","last_became_peered":"2024-03-24T00:43:10.825185+0000","last_unstale":"2024-03-24T00:43:11.831745+0000","last_undegraded":"2024-03-24T00:43:11.831745+0000","last_fullsized":"2024-03-24T00:43:11.831745+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:17:52.643920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.102029+0000","last_change":"2024-03-24T00:43:10.844941+0000","last_active":"2024-03-24T00:43:12.102029+0000","last_peered":"2024-03-24T00:43:12.102029+0000","last_clean":"2024-03-24T00:43:12.102029+0000","last_became_active":"2024-03-24T00:43:10.844679+0000","last_became_peered":"2024-03-24T00:43:10.844679+0000","last_unstale":"2024-03-24T00:43:12.102029+0000","last_undegraded":"2024-03-24T00:43:12.102029+0000","last_fullsized":"2024-03-24T00:43:12.102029+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:13:31.896940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.851675+0000","last_change":"2024-03-24T00:43:10.827815+0000","last_active":"2024-03-24T00:43:12.851675+0000","last_peered":"2024-03-24T00:43:12.851675+0000","last_clean":"2024-03-24T00:43:12.851675+0000","last_became_active":"2024-03-24T00:43:10.827544+0000","last_became_peered":"2024-03-24T00:43:10.827544+0000","last_unstale":"2024-03-24T00:43:12.851675+0000","last_undegraded":"2024-03-24T00:43:12.851675+0000","last_fullsized":"2024-03-24T00:43:12.851675+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:36:07.189985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.831539+0000","last_change":"2024-03-24T00:43:10.843345+0000","last_active":"2024-03-24T00:43:11.831539+0000","last_peered":"2024-03-24T00:43:11.831539+0000","last_clean":"2024-03-24T00:43:11.831539+0000","last_became_active":"2024-03-24T00:43:10.842973+0000","last_became_peered":"2024-03-24T00:43:10.842973+0000","last_unstale":"2024-03-24T00:43:11.831539+0000","last_undegraded":"2024-03-24T00:43:11.831539+0000","last_fullsized":"2024-03-24T00:43:11.831539+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:37:46.896792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.853086+0000","last_change":"2024-03-24T00:43:10.840409+0000","last_active":"2024-03-24T00:43:12.853086+0000","last_peered":"2024-03-24T00:43:12.853086+0000","last_clean":"2024-03-24T00:43:12.853086+0000","last_became_active":"2024-03-24T00:43:10.840068+0000","last_became_peered":"2024-03-24T00:43:10.840068+0000","last_unstale":"2024-03-24T00:43:12.853086+0000","last_undegraded":"2024-03-24T00:43:12.853086+0000","last_fullsized":"2024-03-24T00:43:12.853086+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:01:34.950966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.831666+0000","last_change":"2024-03-24T00:43:10.821368+0000","last_active":"2024-03-24T00:43:11.831666+0000","last_peered":"2024-03-24T00:43:11.831666+0000","last_clean":"2024-03-24T00:43:11.831666+0000","last_became_active":"2024-03-24T00:43:10.821082+0000","last_became_peered":"2024-03-24T00:43:10.821082+0000","last_unstale":"2024-03-24T00:43:11.831666+0000","last_undegraded":"2024-03-24T00:43:11.831666+0000","last_fullsized":"2024-03-24T00:43:11.831666+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:23:39.982261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.853209+0000","last_change":"2024-03-24T00:43:10.827823+0000","last_active":"2024-03-24T00:43:12.853209+0000","last_peered":"2024-03-24T00:43:12.853209+0000","last_clean":"2024-03-24T00:43:12.853209+0000","last_became_active":"2024-03-24T00:43:10.826784+0000","last_became_peered":"2024-03-24T00:43:10.826784+0000","last_unstale":"2024-03-24T00:43:12.853209+0000","last_undegraded":"2024-03-24T00:43:12.853209+0000","last_fullsized":"2024-03-24T00:43:12.853209+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:22:27.781248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.829522+0000","last_change":"2024-03-24T00:43:10.833270+0000","last_active":"2024-03-24T00:43:11.829522+0000","last_peered":"2024-03-24T00:43:11.829522+0000","last_clean":"2024-03-24T00:43:11.829522+0000","last_became_active":"2024-03-24T00:43:10.832905+0000","last_became_peered":"2024-03-24T00:43:10.832905+0000","last_unstale":"2024-03-24T00:43:11.829522+0000","last_undegraded":"2024-03-24T00:43:11.829522+0000","last_fullsized":"2024-03-24T00:43:11.829522+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:31:51.135294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.829895+0000","last_change":"2024-03-24T00:43:10.835987+0000","last_active":"2024-03-24T00:43:11.829895+0000","last_peered":"2024-03-24T00:43:11.829895+0000","last_clean":"2024-03-24T00:43:11.829895+0000","last_became_active":"2024-03-24T00:43:10.835682+0000","last_became_peered":"2024-03-24T00:43:10.835682+0000","last_unstale":"2024-03-24T00:43:11.829895+0000","last_undegraded":"2024-03-24T00:43:11.829895+0000","last_fullsized":"2024-03-24T00:43:11.829895+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:39:41.935465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.831100+0000","last_change":"2024-03-24T00:43:10.835591+0000","last_active":"2024-03-24T00:43:11.831100+0000","last_peered":"2024-03-24T00:43:11.831100+0000","last_clean":"2024-03-24T00:43:11.831100+0000","last_became_active":"2024-03-24T00:43:10.835280+0000","last_became_peered":"2024-03-24T00:43:10.835280+0000","last_unstale":"2024-03-24T00:43:11.831100+0000","last_undegraded":"2024-03-24T00:43:11.831100+0000","last_fullsized":"2024-03-24T00:43:11.831100+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:19:31.014199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.852677+0000","last_change":"2024-03-24T00:43:10.826092+0000","last_active":"2024-03-24T00:43:12.852677+0000","last_peered":"2024-03-24T00:43:12.852677+0000","last_clean":"2024-03-24T00:43:12.852677+0000","last_became_active":"2024-03-24T00:43:10.825799+0000","last_became_peered":"2024-03-24T00:43:10.825799+0000","last_unstale":"2024-03-24T00:43:12.852677+0000","last_undegraded":"2024-03-24T00:43:12.852677+0000","last_fullsized":"2024-03-24T00:43:12.852677+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:12:07.102629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.831555+0000","last_change":"2024-03-24T00:43:10.830905+0000","last_active":"2024-03-24T00:43:11.831555+0000","last_peered":"2024-03-24T00:43:11.831555+0000","last_clean":"2024-03-24T00:43:11.831555+0000","last_became_active":"2024-03-24T00:43:10.830634+0000","last_became_peered":"2024-03-24T00:43:10.830634+0000","last_unstale":"2024-03-24T00:43:11.831555+0000","last_undegraded":"2024-03-24T00:43:11.831555+0000","last_fullsized":"2024-03-24T00:43:11.831555+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:00:53.698992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.829979+0000","last_change":"2024-03-24T00:43:10.834746+0000","last_active":"2024-03-24T00:43:11.829979+0000","last_peered":"2024-03-24T00:43:11.829979+0000","last_clean":"2024-03-24T00:43:11.829979+0000","last_became_active":"2024-03-24T00:43:10.834469+0000","last_became_peered":"2024-03-24T00:43:10.834469+0000","last_unstale":"2024-03-24T00:43:11.829979+0000","last_undegraded":"2024-03-24T00:43:11.829979+0000","last_fullsized":"2024-03-24T00:43:11.829979+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:30:53.960721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.852642+0000","last_change":"2024-03-24T00:43:10.833060+0000","last_active":"2024-03-24T00:43:12.852642+0000","last_peered":"2024-03-24T00:43:12.852642+0000","last_clean":"2024-03-24T00:43:12.852642+0000","last_became_active":"2024-03-24T00:43:10.832772+0000","last_became_peered":"2024-03-24T00:43:10.832772+0000","last_unstale":"2024-03-24T00:43:12.852642+0000","last_undegraded":"2024-03-24T00:43:12.852642+0000","last_fullsized":"2024-03-24T00:43:12.852642+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:01:23.035377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.101715+0000","last_change":"2024-03-24T00:43:10.843524+0000","last_active":"2024-03-24T00:43:12.101715+0000","last_peered":"2024-03-24T00:43:12.101715+0000","last_clean":"2024-03-24T00:43:12.101715+0000","last_became_active":"2024-03-24T00:43:10.843248+0000","last_became_peered":"2024-03-24T00:43:10.843248+0000","last_unstale":"2024-03-24T00:43:12.101715+0000","last_undegraded":"2024-03-24T00:43:12.101715+0000","last_fullsized":"2024-03-24T00:43:12.101715+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:19:08.316847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.852690+0000","last_change":"2024-03-24T00:43:10.823084+0000","last_active":"2024-03-24T00:43:12.852690+0000","last_peered":"2024-03-24T00:43:12.852690+0000","last_clean":"2024-03-24T00:43:12.852690+0000","last_became_active":"2024-03-24T00:43:10.822771+0000","last_became_peered":"2024-03-24T00:43:10.822771+0000","last_unstale":"2024-03-24T00:43:12.852690+0000","last_undegraded":"2024-03-24T00:43:12.852690+0000","last_fullsized":"2024-03-24T00:43:12.852690+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:33:32.269240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.101676+0000","last_change":"2024-03-24T00:43:10.845574+0000","last_active":"2024-03-24T00:43:12.101676+0000","last_peered":"2024-03-24T00:43:12.101676+0000","last_clean":"2024-03-24T00:43:12.101676+0000","last_became_active":"2024-03-24T00:43:10.845272+0000","last_became_peered":"2024-03-24T00:43:10.845272+0000","last_unstale":"2024-03-24T00:43:12.101676+0000","last_undegraded":"2024-03-24T00:43:12.101676+0000","last_fullsized":"2024-03-24T00:43:12.101676+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:18:54.804661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"75'4","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.831805+0000","last_change":"2024-03-24T00:43:09.827745+0000","last_active":"2024-03-24T00:43:11.831805+0000","last_peered":"2024-03-24T00:43:11.831805+0000","last_clean":"2024-03-24T00:43:11.831805+0000","last_became_active":"2024-03-24T00:43:09.827489+0000","last_became_peered":"2024-03-24T00:43:09.827489+0000","last_unstale":"2024-03-24T00:43:11.831805+0000","last_undegraded":"2024-03-24T00:43:11.831805+0000","last_fullsized":"2024-03-24T00:43:11.831805+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:04:01.630948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.829604+0000","last_change":"2024-03-24T00:43:10.836869+0000","last_active":"2024-03-24T00:43:11.829604+0000","last_peered":"2024-03-24T00:43:11.829604+0000","last_clean":"2024-03-24T00:43:11.829604+0000","last_became_active":"2024-03-24T00:43:10.836608+0000","last_became_peered":"2024-03-24T00:43:10.836608+0000","last_unstale":"2024-03-24T00:43:11.829604+0000","last_undegraded":"2024-03-24T00:43:11.829604+0000","last_fullsized":"2024-03-24T00:43:11.829604+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:10:11.482157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"75'1","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.852975+0000","last_change":"2024-03-24T00:43:09.826378+0000","last_active":"2024-03-24T00:43:12.852975+0000","last_peered":"2024-03-24T00:43:12.852975+0000","last_clean":"2024-03-24T00:43:12.852975+0000","last_became_active":"2024-03-24T00:43:09.826104+0000","last_became_peered":"2024-03-24T00:43:09.826104+0000","last_unstale":"2024-03-24T00:43:12.852975+0000","last_undegraded":"2024-03-24T00:43:12.852975+0000","last_fullsized":"2024-03-24T00:43:12.852975+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:41:34.466184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.853052+0000","last_change":"2024-03-24T00:43:10.824509+0000","last_active":"2024-03-24T00:43:12.853052+0000","last_peered":"2024-03-24T00:43:12.853052+0000","last_clean":"2024-03-24T00:43:12.853052+0000","last_became_active":"2024-03-24T00:43:10.824201+0000","last_became_peered":"2024-03-24T00:43:10.824201+0000","last_unstale":"2024-03-24T00:43:12.853052+0000","last_undegraded":"2024-03-24T00:43:12.853052+0000","last_fullsized":"2024-03-24T00:43:12.853052+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:53:12.665046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"75'2","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.852742+0000","last_change":"2024-03-24T00:43:09.826382+0000","last_active":"2024-03-24T00:43:12.852742+0000","last_peered":"2024-03-24T00:43:12.852742+0000","last_clean":"2024-03-24T00:43:12.852742+0000","last_became_active":"2024-03-24T00:43:09.826150+0000","last_became_peered":"2024-03-24T00:43:09.826150+0000","last_unstale":"2024-03-24T00:43:12.852742+0000","last_undegraded":"2024-03-24T00:43:12.852742+0000","last_fullsized":"2024-03-24T00:43:12.852742+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:23:35.409741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.853322+0000","last_change":"2024-03-24T00:43:10.844623+0000","last_active":"2024-03-24T00:43:12.853322+0000","last_peered":"2024-03-24T00:43:12.853322+0000","last_clean":"2024-03-24T00:43:12.853322+0000","last_became_active":"2024-03-24T00:43:10.844319+0000","last_became_peered":"2024-03-24T00:43:10.844319+0000","last_unstale":"2024-03-24T00:43:12.853322+0000","last_undegraded":"2024-03-24T00:43:12.853322+0000","last_fullsized":"2024-03-24T00:43:12.853322+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:52:06.375447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"75'2","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.852256+0000","last_change":"2024-03-24T00:43:09.828399+0000","last_active":"2024-03-24T00:43:12.852256+0000","last_peered":"2024-03-24T00:43:12.852256+0000","last_clean":"2024-03-24T00:43:12.852256+0000","last_became_active":"2024-03-24T00:43:09.828101+0000","last_became_peered":"2024-03-24T00:43:09.828101+0000","last_unstale":"2024-03-24T00:43:12.852256+0000","last_undegraded":"2024-03-24T00:43:12.852256+0000","last_fullsized":"2024-03-24T00:43:12.852256+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:18:05.332714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.830052+0000","last_change":"2024-03-24T00:43:10.834103+0000","last_active":"2024-03-24T00:43:11.830052+0000","last_peered":"2024-03-24T00:43:11.830052+0000","last_clean":"2024-03-24T00:43:11.830052+0000","last_became_active":"2024-03-24T00:43:10.833842+0000","last_became_peered":"2024-03-24T00:43:10.833842+0000","last_unstale":"2024-03-24T00:43:11.830052+0000","last_undegraded":"2024-03-24T00:43:11.830052+0000","last_fullsized":"2024-03-24T00:43:11.830052+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:11:52.190154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"75'1","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.853485+0000","last_change":"2024-03-24T00:43:09.826758+0000","last_active":"2024-03-24T00:43:12.853485+0000","last_peered":"2024-03-24T00:43:12.853485+0000","last_clean":"2024-03-24T00:43:12.853485+0000","last_became_active":"2024-03-24T00:43:09.826495+0000","last_became_peered":"2024-03-24T00:43:09.826495+0000","last_unstale":"2024-03-24T00:43:12.853485+0000","last_undegraded":"2024-03-24T00:43:12.853485+0000","last_fullsized":"2024-03-24T00:43:12.853485+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:37:45.892946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.830325+0000","last_change":"2024-03-24T00:43:10.844044+0000","last_active":"2024-03-24T00:43:11.830325+0000","last_peered":"2024-03-24T00:43:11.830325+0000","last_clean":"2024-03-24T00:43:11.830325+0000","last_became_active":"2024-03-24T00:43:10.843779+0000","last_became_peered":"2024-03-24T00:43:10.843779+0000","last_unstale":"2024-03-24T00:43:11.830325+0000","last_undegraded":"2024-03-24T00:43:11.830325+0000","last_fullsized":"2024-03-24T00:43:11.830325+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:09:22.854673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"75'1","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.830760+0000","last_change":"2024-03-24T00:43:09.827568+0000","last_active":"2024-03-24T00:43:11.830760+0000","last_peered":"2024-03-24T00:43:11.830760+0000","last_clean":"2024-03-24T00:43:11.830760+0000","last_became_active":"2024-03-24T00:43:09.827267+0000","last_became_peered":"2024-03-24T00:43:09.827267+0000","last_unstale":"2024-03-24T00:43:11.830760+0000","last_undegraded":"2024-03-24T00:43:11.830760+0000","last_fullsized":"2024-03-24T00:43:11.830760+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:38:27.408597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.853860+0000","last_change":"2024-03-24T00:43:10.837426+0000","last_active":"2024-03-24T00:43:12.853860+0000","last_peered":"2024-03-24T00:43:12.853860+0000","last_clean":"2024-03-24T00:43:12.853860+0000","last_became_active":"2024-03-24T00:43:10.837074+0000","last_became_peered":"2024-03-24T00:43:10.837074+0000","last_unstale":"2024-03-24T00:43:12.853860+0000","last_undegraded":"2024-03-24T00:43:12.853860+0000","last_fullsized":"2024-03-24T00:43:12.853860+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:48:25.693365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.829680+0000","last_change":"2024-03-24T00:43:09.824279+0000","last_active":"2024-03-24T00:43:11.829680+0000","last_peered":"2024-03-24T00:43:11.829680+0000","last_clean":"2024-03-24T00:43:11.829680+0000","last_became_active":"2024-03-24T00:43:09.824012+0000","last_became_peered":"2024-03-24T00:43:09.824012+0000","last_unstale":"2024-03-24T00:43:11.829680+0000","last_undegraded":"2024-03-24T00:43:11.829680+0000","last_fullsized":"2024-03-24T00:43:11.829680+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:10:01.312078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.853367+0000","last_change":"2024-03-24T00:43:10.832566+0000","last_active":"2024-03-24T00:43:12.853367+0000","last_peered":"2024-03-24T00:43:12.853367+0000","last_clean":"2024-03-24T00:43:12.853367+0000","last_became_active":"2024-03-24T00:43:10.831785+0000","last_became_peered":"2024-03-24T00:43:10.831785+0000","last_unstale":"2024-03-24T00:43:12.853367+0000","last_undegraded":"2024-03-24T00:43:12.853367+0000","last_fullsized":"2024-03-24T00:43:12.853367+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:47:25.859204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.830694+0000","last_change":"2024-03-24T00:43:09.829647+0000","last_active":"2024-03-24T00:43:11.830694+0000","last_peered":"2024-03-24T00:43:11.830694+0000","last_clean":"2024-03-24T00:43:11.830694+0000","last_became_active":"2024-03-24T00:43:09.829343+0000","last_became_peered":"2024-03-24T00:43:09.829343+0000","last_unstale":"2024-03-24T00:43:11.830694+0000","last_undegraded":"2024-03-24T00:43:11.830694+0000","last_fullsized":"2024-03-24T00:43:11.830694+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:12:47.427076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.829634+0000","last_change":"2024-03-24T00:43:10.833824+0000","last_active":"2024-03-24T00:43:11.829634+0000","last_peered":"2024-03-24T00:43:11.829634+0000","last_clean":"2024-03-24T00:43:11.829634+0000","last_became_active":"2024-03-24T00:43:10.833578+0000","last_became_peered":"2024-03-24T00:43:10.833578+0000","last_unstale":"2024-03-24T00:43:11.829634+0000","last_undegraded":"2024-03-24T00:43:11.829634+0000","last_fullsized":"2024-03-24T00:43:11.829634+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:36:07.189985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.853433+0000","last_change":"2024-03-24T00:43:09.831076+0000","last_active":"2024-03-24T00:43:12.853433+0000","last_peered":"2024-03-24T00:43:12.853433+0000","last_clean":"2024-03-24T00:43:12.853433+0000","last_became_active":"2024-03-24T00:43:09.830803+0000","last_became_peered":"2024-03-24T00:43:09.830803+0000","last_unstale":"2024-03-24T00:43:12.853433+0000","last_undegraded":"2024-03-24T00:43:12.853433+0000","last_fullsized":"2024-03-24T00:43:12.853433+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:16:30.998272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.831282+0000","last_change":"2024-03-24T00:43:10.829477+0000","last_active":"2024-03-24T00:43:11.831282+0000","last_peered":"2024-03-24T00:43:11.831282+0000","last_clean":"2024-03-24T00:43:11.831282+0000","last_became_active":"2024-03-24T00:43:10.828756+0000","last_became_peered":"2024-03-24T00:43:10.828756+0000","last_unstale":"2024-03-24T00:43:11.831282+0000","last_undegraded":"2024-03-24T00:43:11.831282+0000","last_fullsized":"2024-03-24T00:43:11.831282+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:12:07.102629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.831309+0000","last_change":"2024-03-24T00:43:09.827651+0000","last_active":"2024-03-24T00:43:11.831309+0000","last_peered":"2024-03-24T00:43:11.831309+0000","last_clean":"2024-03-24T00:43:11.831309+0000","last_became_active":"2024-03-24T00:43:09.827327+0000","last_became_peered":"2024-03-24T00:43:09.827327+0000","last_unstale":"2024-03-24T00:43:11.831309+0000","last_undegraded":"2024-03-24T00:43:11.831309+0000","last_fullsized":"2024-03-24T00:43:11.831309+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:48:30.269130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.852059+0000","last_change":"2024-03-24T00:43:10.832792+0000","last_active":"2024-03-24T00:43:12.852059+0000","last_peered":"2024-03-24T00:43:12.852059+0000","last_clean":"2024-03-24T00:43:12.852059+0000","last_became_active":"2024-03-24T00:43:10.832466+0000","last_became_peered":"2024-03-24T00:43:10.832466+0000","last_unstale":"2024-03-24T00:43:12.852059+0000","last_undegraded":"2024-03-24T00:43:12.852059+0000","last_fullsized":"2024-03-24T00:43:12.852059+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:15:40.447819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"75'3","reported_seq":17,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.831743+0000","last_change":"2024-03-24T00:43:09.830357+0000","last_active":"2024-03-24T00:43:11.831743+0000","last_peered":"2024-03-24T00:43:11.831743+0000","last_clean":"2024-03-24T00:43:11.831743+0000","last_became_active":"2024-03-24T00:43:09.828901+0000","last_became_peered":"2024-03-24T00:43:09.828901+0000","last_unstale":"2024-03-24T00:43:11.831743+0000","last_undegraded":"2024-03-24T00:43:11.831743+0000","last_fullsized":"2024-03-24T00:43:11.831743+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:08:52.647569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.101827+0000","last_change":"2024-03-24T00:43:10.838245+0000","last_active":"2024-03-24T00:43:12.101827+0000","last_peered":"2024-03-24T00:43:12.101827+0000","last_clean":"2024-03-24T00:43:12.101827+0000","last_became_active":"2024-03-24T00:43:10.837966+0000","last_became_peered":"2024-03-24T00:43:10.837966+0000","last_unstale":"2024-03-24T00:43:12.101827+0000","last_undegraded":"2024-03-24T00:43:12.101827+0000","last_fullsized":"2024-03-24T00:43:12.101827+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:34:03.395954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"75'6","reported_seq":20,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.852304+0000","last_change":"2024-03-24T00:43:09.825748+0000","last_active":"2024-03-24T00:43:12.852304+0000","last_peered":"2024-03-24T00:43:12.852304+0000","last_clean":"2024-03-24T00:43:12.852304+0000","last_became_active":"2024-03-24T00:43:09.825469+0000","last_became_peered":"2024-03-24T00:43:09.825469+0000","last_unstale":"2024-03-24T00:43:12.852304+0000","last_undegraded":"2024-03-24T00:43:12.852304+0000","last_fullsized":"2024-03-24T00:43:12.852304+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:56:42.898290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.831404+0000","last_change":"2024-03-24T00:43:10.830424+0000","last_active":"2024-03-24T00:43:11.831404+0000","last_peered":"2024-03-24T00:43:11.831404+0000","last_clean":"2024-03-24T00:43:11.831404+0000","last_became_active":"2024-03-24T00:43:10.829770+0000","last_became_peered":"2024-03-24T00:43:10.829770+0000","last_unstale":"2024-03-24T00:43:11.831404+0000","last_undegraded":"2024-03-24T00:43:11.831404+0000","last_fullsized":"2024-03-24T00:43:11.831404+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:31:51.135294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"75'2","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.830942+0000","last_change":"2024-03-24T00:43:09.821932+0000","last_active":"2024-03-24T00:43:11.830942+0000","last_peered":"2024-03-24T00:43:11.830942+0000","last_clean":"2024-03-24T00:43:11.830942+0000","last_became_active":"2024-03-24T00:43:09.821665+0000","last_became_peered":"2024-03-24T00:43:09.821665+0000","last_unstale":"2024-03-24T00:43:11.830942+0000","last_undegraded":"2024-03-24T00:43:11.830942+0000","last_fullsized":"2024-03-24T00:43:11.830942+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:16:30.998272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.852293+0000","last_change":"2024-03-24T00:43:10.840466+0000","last_active":"2024-03-24T00:43:12.852293+0000","last_peered":"2024-03-24T00:43:12.852293+0000","last_clean":"2024-03-24T00:43:12.852293+0000","last_became_active":"2024-03-24T00:43:10.839609+0000","last_became_peered":"2024-03-24T00:43:10.839609+0000","last_unstale":"2024-03-24T00:43:12.852293+0000","last_undegraded":"2024-03-24T00:43:12.852293+0000","last_fullsized":"2024-03-24T00:43:12.852293+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:13:37.904410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.852706+0000","last_change":"2024-03-24T00:43:09.829536+0000","last_active":"2024-03-24T00:43:12.852706+0000","last_peered":"2024-03-24T00:43:12.852706+0000","last_clean":"2024-03-24T00:43:12.852706+0000","last_became_active":"2024-03-24T00:43:09.829260+0000","last_became_peered":"2024-03-24T00:43:09.829260+0000","last_unstale":"2024-03-24T00:43:12.852706+0000","last_undegraded":"2024-03-24T00:43:12.852706+0000","last_fullsized":"2024-03-24T00:43:12.852706+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:10:01.312078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.852601+0000","last_change":"2024-03-24T00:43:10.832446+0000","last_active":"2024-03-24T00:43:12.852601+0000","last_peered":"2024-03-24T00:43:12.852601+0000","last_clean":"2024-03-24T00:43:12.852601+0000","last_became_active":"2024-03-24T00:43:10.831852+0000","last_became_peered":"2024-03-24T00:43:10.831852+0000","last_unstale":"2024-03-24T00:43:12.852601+0000","last_undegraded":"2024-03-24T00:43:12.852601+0000","last_fullsized":"2024-03-24T00:43:12.852601+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:19:49.017409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.830885+0000","last_change":"2024-03-24T00:43:09.829607+0000","last_active":"2024-03-24T00:43:11.830885+0000","last_peered":"2024-03-24T00:43:11.830885+0000","last_clean":"2024-03-24T00:43:11.830885+0000","last_became_active":"2024-03-24T00:43:09.829347+0000","last_became_peered":"2024-03-24T00:43:09.829347+0000","last_unstale":"2024-03-24T00:43:11.830885+0000","last_undegraded":"2024-03-24T00:43:11.830885+0000","last_fullsized":"2024-03-24T00:43:11.830885+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:19:26.187451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.853402+0000","last_change":"2024-03-24T00:43:10.836785+0000","last_active":"2024-03-24T00:43:12.853402+0000","last_peered":"2024-03-24T00:43:12.853402+0000","last_clean":"2024-03-24T00:43:12.853402+0000","last_became_active":"2024-03-24T00:43:10.836516+0000","last_became_peered":"2024-03-24T00:43:10.836516+0000","last_unstale":"2024-03-24T00:43:12.853402+0000","last_undegraded":"2024-03-24T00:43:12.853402+0000","last_fullsized":"2024-03-24T00:43:12.853402+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:44:18.631971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"75'8","reported_seq":22,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.829580+0000","last_change":"2024-03-24T00:43:09.826848+0000","last_active":"2024-03-24T00:43:11.829580+0000","last_peered":"2024-03-24T00:43:11.829580+0000","last_clean":"2024-03-24T00:43:11.829580+0000","last_became_active":"2024-03-24T00:43:09.826548+0000","last_became_peered":"2024-03-24T00:43:09.826548+0000","last_unstale":"2024-03-24T00:43:11.829580+0000","last_undegraded":"2024-03-24T00:43:11.829580+0000","last_fullsized":"2024-03-24T00:43:11.829580+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:38:27.408597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.851583+0000","last_change":"2024-03-24T00:43:10.828447+0000","last_active":"2024-03-24T00:43:12.851583+0000","last_peered":"2024-03-24T00:43:12.851583+0000","last_clean":"2024-03-24T00:43:12.851583+0000","last_became_active":"2024-03-24T00:43:10.828157+0000","last_became_peered":"2024-03-24T00:43:10.828157+0000","last_unstale":"2024-03-24T00:43:12.851583+0000","last_undegraded":"2024-03-24T00:43:12.851583+0000","last_fullsized":"2024-03-24T00:43:12.851583+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:10:11.482157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"75'1","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.853534+0000","last_change":"2024-03-24T00:43:09.826478+0000","last_active":"2024-03-24T00:43:12.853534+0000","last_peered":"2024-03-24T00:43:12.853534+0000","last_clean":"2024-03-24T00:43:12.853534+0000","last_became_active":"2024-03-24T00:43:09.826185+0000","last_became_peered":"2024-03-24T00:43:09.826185+0000","last_unstale":"2024-03-24T00:43:12.853534+0000","last_undegraded":"2024-03-24T00:43:12.853534+0000","last_fullsized":"2024-03-24T00:43:12.853534+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:38:27.408597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.830934+0000","last_change":"2024-03-24T00:43:10.834127+0000","last_active":"2024-03-24T00:43:11.830934+0000","last_peered":"2024-03-24T00:43:11.830934+0000","last_clean":"2024-03-24T00:43:11.830934+0000","last_became_active":"2024-03-24T00:43:10.833870+0000","last_became_peered":"2024-03-24T00:43:10.833870+0000","last_unstale":"2024-03-24T00:43:11.830934+0000","last_undegraded":"2024-03-24T00:43:11.830934+0000","last_fullsized":"2024-03-24T00:43:11.830934+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:27:13.070120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"75'2","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.853853+0000","last_change":"2024-03-24T00:43:09.825855+0000","last_active":"2024-03-24T00:43:12.853853+0000","last_peered":"2024-03-24T00:43:12.853853+0000","last_clean":"2024-03-24T00:43:12.853853+0000","last_became_active":"2024-03-24T00:43:09.825353+0000","last_became_peered":"2024-03-24T00:43:09.825353+0000","last_unstale":"2024-03-24T00:43:12.853853+0000","last_undegraded":"2024-03-24T00:43:12.853853+0000","last_fullsized":"2024-03-24T00:43:12.853853+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:13:19.913544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.831069+0000","last_change":"2024-03-24T00:43:10.832638+0000","last_active":"2024-03-24T00:43:11.831069+0000","last_peered":"2024-03-24T00:43:11.831069+0000","last_clean":"2024-03-24T00:43:11.831069+0000","last_became_active":"2024-03-24T00:43:10.832336+0000","last_became_peered":"2024-03-24T00:43:10.832336+0000","last_unstale":"2024-03-24T00:43:11.831069+0000","last_undegraded":"2024-03-24T00:43:11.831069+0000","last_fullsized":"2024-03-24T00:43:11.831069+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:14:38.749969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"75'1","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.831775+0000","last_change":"2024-03-24T00:43:09.829560+0000","last_active":"2024-03-24T00:43:11.831775+0000","last_peered":"2024-03-24T00:43:11.831775+0000","last_clean":"2024-03-24T00:43:11.831775+0000","last_became_active":"2024-03-24T00:43:09.829292+0000","last_became_peered":"2024-03-24T00:43:09.829292+0000","last_unstale":"2024-03-24T00:43:11.831775+0000","last_undegraded":"2024-03-24T00:43:11.831775+0000","last_fullsized":"2024-03-24T00:43:11.831775+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:33:04.005989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.851841+0000","last_change":"2024-03-24T00:43:10.824843+0000","last_active":"2024-03-24T00:43:12.851841+0000","last_peered":"2024-03-24T00:43:12.851841+0000","last_clean":"2024-03-24T00:43:12.851841+0000","last_became_active":"2024-03-24T00:43:10.824568+0000","last_became_peered":"2024-03-24T00:43:10.824568+0000","last_unstale":"2024-03-24T00:43:12.851841+0000","last_undegraded":"2024-03-24T00:43:12.851841+0000","last_fullsized":"2024-03-24T00:43:12.851841+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:39:41.935465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.101757+0000","last_change":"2024-03-24T00:43:09.830842+0000","last_active":"2024-03-24T00:43:12.101757+0000","last_peered":"2024-03-24T00:43:12.101757+0000","last_clean":"2024-03-24T00:43:12.101757+0000","last_became_active":"2024-03-24T00:43:09.830463+0000","last_became_peered":"2024-03-24T00:43:09.830463+0000","last_unstale":"2024-03-24T00:43:12.101757+0000","last_undegraded":"2024-03-24T00:43:12.101757+0000","last_fullsized":"2024-03-24T00:43:12.101757+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:19:47.044511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.830186+0000","last_change":"2024-03-24T00:43:10.837658+0000","last_active":"2024-03-24T00:43:11.830186+0000","last_peered":"2024-03-24T00:43:11.830186+0000","last_clean":"2024-03-24T00:43:11.830186+0000","last_became_active":"2024-03-24T00:43:10.837357+0000","last_became_peered":"2024-03-24T00:43:10.837357+0000","last_unstale":"2024-03-24T00:43:11.830186+0000","last_undegraded":"2024-03-24T00:43:11.830186+0000","last_fullsized":"2024-03-24T00:43:11.830186+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:21:47.552203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"75'2","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.851846+0000","last_change":"2024-03-24T00:43:09.825245+0000","last_active":"2024-03-24T00:43:12.851846+0000","last_peered":"2024-03-24T00:43:12.851846+0000","last_clean":"2024-03-24T00:43:12.851846+0000","last_became_active":"2024-03-24T00:43:09.824975+0000","last_became_peered":"2024-03-24T00:43:09.824975+0000","last_unstale":"2024-03-24T00:43:12.851846+0000","last_undegraded":"2024-03-24T00:43:12.851846+0000","last_fullsized":"2024-03-24T00:43:12.851846+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:14:14.121481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.830493+0000","last_change":"2024-03-24T00:43:10.833037+0000","last_active":"2024-03-24T00:43:11.830493+0000","last_peered":"2024-03-24T00:43:11.830493+0000","last_clean":"2024-03-24T00:43:11.830493+0000","last_became_active":"2024-03-24T00:43:10.832766+0000","last_became_peered":"2024-03-24T00:43:10.832766+0000","last_unstale":"2024-03-24T00:43:11.830493+0000","last_undegraded":"2024-03-24T00:43:11.830493+0000","last_fullsized":"2024-03-24T00:43:11.830493+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:47:19.214497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.851543+0000","last_change":"2024-03-24T00:43:09.827109+0000","last_active":"2024-03-24T00:43:12.851543+0000","last_peered":"2024-03-24T00:43:12.851543+0000","last_clean":"2024-03-24T00:43:12.851543+0000","last_became_active":"2024-03-24T00:43:09.826874+0000","last_became_peered":"2024-03-24T00:43:09.826874+0000","last_unstale":"2024-03-24T00:43:12.851543+0000","last_undegraded":"2024-03-24T00:43:12.851543+0000","last_fullsized":"2024-03-24T00:43:12.851543+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:24:43.787686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.853522+0000","last_change":"2024-03-24T00:43:10.832669+0000","last_active":"2024-03-24T00:43:12.853522+0000","last_peered":"2024-03-24T00:43:12.853522+0000","last_clean":"2024-03-24T00:43:12.853522+0000","last_became_active":"2024-03-24T00:43:10.832171+0000","last_became_peered":"2024-03-24T00:43:10.832171+0000","last_unstale":"2024-03-24T00:43:12.853522+0000","last_undegraded":"2024-03-24T00:43:12.853522+0000","last_fullsized":"2024-03-24T00:43:12.853522+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:03:52.283057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.851637+0000","last_change":"2024-03-24T00:43:09.827816+0000","last_active":"2024-03-24T00:43:12.851637+0000","last_peered":"2024-03-24T00:43:12.851637+0000","last_clean":"2024-03-24T00:43:12.851637+0000","last_became_active":"2024-03-24T00:43:09.827555+0000","last_became_peered":"2024-03-24T00:43:09.827555+0000","last_unstale":"2024-03-24T00:43:12.851637+0000","last_undegraded":"2024-03-24T00:43:12.851637+0000","last_fullsized":"2024-03-24T00:43:12.851637+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:11:49.925242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.853107+0000","last_change":"2024-03-24T00:43:10.834121+0000","last_active":"2024-03-24T00:43:12.853107+0000","last_peered":"2024-03-24T00:43:12.853107+0000","last_clean":"2024-03-24T00:43:12.853107+0000","last_became_active":"2024-03-24T00:43:10.833834+0000","last_became_peered":"2024-03-24T00:43:10.833834+0000","last_unstale":"2024-03-24T00:43:12.853107+0000","last_undegraded":"2024-03-24T00:43:12.853107+0000","last_fullsized":"2024-03-24T00:43:12.853107+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:22:27.749038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"75'2","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.831008+0000","last_change":"2024-03-24T00:43:09.823656+0000","last_active":"2024-03-24T00:43:11.831008+0000","last_peered":"2024-03-24T00:43:11.831008+0000","last_clean":"2024-03-24T00:43:11.831008+0000","last_became_active":"2024-03-24T00:43:09.823109+0000","last_became_peered":"2024-03-24T00:43:09.823109+0000","last_unstale":"2024-03-24T00:43:11.831008+0000","last_undegraded":"2024-03-24T00:43:11.831008+0000","last_fullsized":"2024-03-24T00:43:11.831008+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:23:35.409741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.852156+0000","last_change":"2024-03-24T00:43:10.836822+0000","last_active":"2024-03-24T00:43:12.852156+0000","last_peered":"2024-03-24T00:43:12.852156+0000","last_clean":"2024-03-24T00:43:12.852156+0000","last_became_active":"2024-03-24T00:43:10.836557+0000","last_became_peered":"2024-03-24T00:43:10.836557+0000","last_unstale":"2024-03-24T00:43:12.852156+0000","last_undegraded":"2024-03-24T00:43:12.852156+0000","last_fullsized":"2024-03-24T00:43:12.852156+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:35:45.892922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"76'1","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.830539+0000","last_change":"2024-03-24T00:43:09.830340+0000","last_active":"2024-03-24T00:43:11.830539+0000","last_peered":"2024-03-24T00:43:11.830539+0000","last_clean":"2024-03-24T00:43:11.830539+0000","last_became_active":"2024-03-24T00:43:09.829859+0000","last_became_peered":"2024-03-24T00:43:09.829859+0000","last_unstale":"2024-03-24T00:43:11.830539+0000","last_undegraded":"2024-03-24T00:43:11.830539+0000","last_fullsized":"2024-03-24T00:43:11.830539+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:41:34.466184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.853146+0000","last_change":"2024-03-24T00:43:10.823860+0000","last_active":"2024-03-24T00:43:12.853146+0000","last_peered":"2024-03-24T00:43:12.853146+0000","last_clean":"2024-03-24T00:43:12.853146+0000","last_became_active":"2024-03-24T00:43:10.823546+0000","last_became_peered":"2024-03-24T00:43:10.823546+0000","last_unstale":"2024-03-24T00:43:12.853146+0000","last_undegraded":"2024-03-24T00:43:12.853146+0000","last_fullsized":"2024-03-24T00:43:12.853146+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:43:07.870388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.852988+0000","last_change":"2024-03-24T00:43:09.829855+0000","last_active":"2024-03-24T00:43:12.852988+0000","last_peered":"2024-03-24T00:43:12.852988+0000","last_clean":"2024-03-24T00:43:12.852988+0000","last_became_active":"2024-03-24T00:43:09.829369+0000","last_became_peered":"2024-03-24T00:43:09.829369+0000","last_unstale":"2024-03-24T00:43:12.852988+0000","last_undegraded":"2024-03-24T00:43:12.852988+0000","last_fullsized":"2024-03-24T00:43:12.852988+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:48:30.269130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.830060+0000","last_change":"2024-03-24T00:43:10.831531+0000","last_active":"2024-03-24T00:43:11.830060+0000","last_peered":"2024-03-24T00:43:11.830060+0000","last_clean":"2024-03-24T00:43:11.830060+0000","last_became_active":"2024-03-24T00:43:10.831268+0000","last_became_peered":"2024-03-24T00:43:10.831268+0000","last_unstale":"2024-03-24T00:43:11.830060+0000","last_undegraded":"2024-03-24T00:43:11.830060+0000","last_fullsized":"2024-03-24T00:43:11.830060+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:15:40.447819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.852905+0000","last_change":"2024-03-24T00:43:09.827149+0000","last_active":"2024-03-24T00:43:12.852905+0000","last_peered":"2024-03-24T00:43:12.852905+0000","last_clean":"2024-03-24T00:43:12.852905+0000","last_became_active":"2024-03-24T00:43:09.825842+0000","last_became_peered":"2024-03-24T00:43:09.825842+0000","last_unstale":"2024-03-24T00:43:12.852905+0000","last_undegraded":"2024-03-24T00:43:12.852905+0000","last_fullsized":"2024-03-24T00:43:12.852905+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:44:53.438106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.852196+0000","last_change":"2024-03-24T00:43:10.837866+0000","last_active":"2024-03-24T00:43:12.852196+0000","last_peered":"2024-03-24T00:43:12.852196+0000","last_clean":"2024-03-24T00:43:12.852196+0000","last_became_active":"2024-03-24T00:43:10.837604+0000","last_became_peered":"2024-03-24T00:43:10.837604+0000","last_unstale":"2024-03-24T00:43:12.852196+0000","last_undegraded":"2024-03-24T00:43:12.852196+0000","last_fullsized":"2024-03-24T00:43:12.852196+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:02:07.497122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.852197+0000","last_change":"2024-03-24T00:43:09.829297+0000","last_active":"2024-03-24T00:43:12.852197+0000","last_peered":"2024-03-24T00:43:12.852197+0000","last_clean":"2024-03-24T00:43:12.852197+0000","last_became_active":"2024-03-24T00:43:09.828970+0000","last_became_peered":"2024-03-24T00:43:09.828970+0000","last_unstale":"2024-03-24T00:43:12.852197+0000","last_undegraded":"2024-03-24T00:43:12.852197+0000","last_fullsized":"2024-03-24T00:43:12.852197+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:04:56.380202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.852443+0000","last_change":"2024-03-24T00:43:10.840766+0000","last_active":"2024-03-24T00:43:12.852443+0000","last_peered":"2024-03-24T00:43:12.852443+0000","last_clean":"2024-03-24T00:43:12.852443+0000","last_became_active":"2024-03-24T00:43:10.840471+0000","last_became_peered":"2024-03-24T00:43:10.840471+0000","last_unstale":"2024-03-24T00:43:12.852443+0000","last_undegraded":"2024-03-24T00:43:12.852443+0000","last_fullsized":"2024-03-24T00:43:12.852443+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:09:35.809870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.831204+0000","last_change":"2024-03-24T00:43:09.827798+0000","last_active":"2024-03-24T00:43:11.831204+0000","last_peered":"2024-03-24T00:43:11.831204+0000","last_clean":"2024-03-24T00:43:11.831204+0000","last_became_active":"2024-03-24T00:43:09.827515+0000","last_became_peered":"2024-03-24T00:43:09.827515+0000","last_unstale":"2024-03-24T00:43:11.831204+0000","last_undegraded":"2024-03-24T00:43:11.831204+0000","last_fullsized":"2024-03-24T00:43:11.831204+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:26:37.415032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.102079+0000","last_change":"2024-03-24T00:43:10.838995+0000","last_active":"2024-03-24T00:43:12.102079+0000","last_peered":"2024-03-24T00:43:12.102079+0000","last_clean":"2024-03-24T00:43:12.102079+0000","last_became_active":"2024-03-24T00:43:10.838725+0000","last_became_peered":"2024-03-24T00:43:10.838725+0000","last_unstale":"2024-03-24T00:43:12.102079+0000","last_undegraded":"2024-03-24T00:43:12.102079+0000","last_fullsized":"2024-03-24T00:43:12.102079+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:14:33.573743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.852325+0000","last_change":"2024-03-24T00:43:09.831419+0000","last_active":"2024-03-24T00:43:12.852325+0000","last_peered":"2024-03-24T00:43:12.852325+0000","last_clean":"2024-03-24T00:43:12.852325+0000","last_became_active":"2024-03-24T00:43:09.831116+0000","last_became_peered":"2024-03-24T00:43:09.831116+0000","last_unstale":"2024-03-24T00:43:12.852325+0000","last_undegraded":"2024-03-24T00:43:12.852325+0000","last_fullsized":"2024-03-24T00:43:12.852325+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:36:42.771157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.102176+0000","last_change":"2024-03-24T00:43:10.841568+0000","last_active":"2024-03-24T00:43:12.102176+0000","last_peered":"2024-03-24T00:43:12.102176+0000","last_clean":"2024-03-24T00:43:12.102176+0000","last_became_active":"2024-03-24T00:43:10.841315+0000","last_became_peered":"2024-03-24T00:43:10.841315+0000","last_unstale":"2024-03-24T00:43:12.102176+0000","last_undegraded":"2024-03-24T00:43:12.102176+0000","last_fullsized":"2024-03-24T00:43:12.102176+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:48:14.736388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.101569+0000","last_change":"2024-03-24T00:43:09.823570+0000","last_active":"2024-03-24T00:43:12.101569+0000","last_peered":"2024-03-24T00:43:12.101569+0000","last_clean":"2024-03-24T00:43:12.101569+0000","last_became_active":"2024-03-24T00:43:09.823302+0000","last_became_peered":"2024-03-24T00:43:09.823302+0000","last_unstale":"2024-03-24T00:43:12.101569+0000","last_undegraded":"2024-03-24T00:43:12.101569+0000","last_fullsized":"2024-03-24T00:43:12.101569+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:45:52.448170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.853002+0000","last_change":"2024-03-24T00:43:10.840301+0000","last_active":"2024-03-24T00:43:12.853002+0000","last_peered":"2024-03-24T00:43:12.853002+0000","last_clean":"2024-03-24T00:43:12.853002+0000","last_became_active":"2024-03-24T00:43:10.839999+0000","last_became_peered":"2024-03-24T00:43:10.839999+0000","last_unstale":"2024-03-24T00:43:12.853002+0000","last_undegraded":"2024-03-24T00:43:12.853002+0000","last_fullsized":"2024-03-24T00:43:12.853002+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:31:51.135294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.852064+0000","last_change":"2024-03-24T00:43:09.826753+0000","last_active":"2024-03-24T00:43:12.852064+0000","last_peered":"2024-03-24T00:43:12.852064+0000","last_clean":"2024-03-24T00:43:12.852064+0000","last_became_active":"2024-03-24T00:43:09.826434+0000","last_became_peered":"2024-03-24T00:43:09.826434+0000","last_unstale":"2024-03-24T00:43:12.852064+0000","last_undegraded":"2024-03-24T00:43:12.852064+0000","last_fullsized":"2024-03-24T00:43:12.852064+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:19:30.010353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":274,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.852755+0000","last_change":"2024-03-24T00:38:40.953871+0000","last_active":"2024-03-24T00:43:12.852755+0000","last_peered":"2024-03-24T00:43:12.852755+0000","last_clean":"2024-03-24T00:43:12.852755+0000","last_became_active":"2024-03-24T00:38:40.952926+0000","last_became_peered":"2024-03-24T00:38:40.952926+0000","last_unstale":"2024-03-24T00:43:12.852755+0000","last_undegraded":"2024-03-24T00:43:12.852755+0000","last_fullsized":"2024-03-24T00:43:12.852755+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_clean_scrub_stamp":"2024-03-24T00:38:39.722194+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:05.397257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574908,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32676,"kb_used_data":1492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339164,"statfs":{"total":96636764160,"available":96603303936,"internally_reserved":0,"allocated":1527808,"data_stored":1015231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574907,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27940,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343900,"statfs":{"total":96636764160,"available":96608153600,"internally_reserved":0,"allocated":942080,"data_stored":426453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574907,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27940,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343900,"statfs":{"total":96636764160,"available":96608153600,"internally_reserved":0,"allocated":933888,"data_stored":424875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574906,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27948,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343892,"statfs":{"total":96636764160,"available":96608145408,"internally_reserved":0,"allocated":950272,"data_stored":426557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574906,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27932,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343908,"statfs":{"total":96636764160,"available":96608161792,"internally_reserved":0,"allocated":933888,"data_stored":425423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574907,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27948,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343892,"statfs":{"total":96636764160,"available":96608145408,"internally_reserved":0,"allocated":942080,"data_stored":425583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574907,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27944,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343896,"statfs":{"total":96636764160,"available":96608149504,"internally_reserved":0,"allocated":937984,"data_stored":424953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574906,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32672,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":1523712,"data_stored":1015209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:43:16.387 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:43:16.389 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:43:16.463 INFO:tasks.ceph.mds.c.smithi067.stderr:2024-03-24T00:43:16.462+0000 7f4a042aa640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T00:43:18.525 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:43:18.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.627+0000 7fc200390640 1 Processor -- start 2024-03-24T00:43:18.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.627+0000 7fc200390640 1 -- start start 2024-03-24T00:43:18.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.628+0000 7fc200390640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc1f815e2a0 0x7fc1f80ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:18.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.628+0000 7fc200390640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1f80cf640 con 0x7fc1f815dec0 2024-03-24T00:43:18.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.628+0000 7fc200390640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1f80cf080 con 0x7fc1f80d0ef0 2024-03-24T00:43:18.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.628+0000 7fc200390640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1f80cee30 con 0x7fc1f815e2a0 2024-03-24T00:43:18.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.628+0000 7fc1fd92b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc1f815e2a0 0x7fc1f80ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:18.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.628+0000 7fc1fe12c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc1f815dec0 0x7fc1f80d07e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37532/0 (socket says 172.21.15.67:37532) 2024-03-24T00:43:18.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.628+0000 7fc1fe12c640 1 -- 172.21.15.67:0/3046992949 learned_addr learned my addr 172.21.15.67:0/3046992949 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:43:18.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.628+0000 7fc1fd92b640 1 -- 172.21.15.67:0/3046992949 >> v1:172.21.15.72:6789/0 conn(0x7fc1f80d0ef0 legacy=0x7fc1f80d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:18.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.628+0000 7fc1fd92b640 1 -- 172.21.15.67:0/3046992949 >> v1:172.21.15.67:6789/0 conn(0x7fc1f815dec0 legacy=0x7fc1f80d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:18.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.628+0000 7fc1fd12a640 1 -- 172.21.15.67:0/3046992949 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2722898710 0 0) 0x7fc1f4001110 con 0x7fc1f815dec0 2024-03-24T00:43:18.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.628+0000 7fc1fd92b640 1 -- 172.21.15.67:0/3046992949 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f80cf360 con 0x7fc1f815e2a0 2024-03-24T00:43:18.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.629+0000 7fc1fd92b640 1 --2- 172.21.15.67:0/3046992949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1f815e2a0 0x7fc1f80ce630 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fc1f80cefb0 tx=0x7fc1ec02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c6a7bfbf43c17 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:18.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.629+0000 7fc1fd12a640 1 -- 172.21.15.67:0/3046992949 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1ec035020 con 0x7fc1f815e2a0 2024-03-24T00:43:18.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.629+0000 7fc1fd12a640 1 -- 172.21.15.67:0/3046992949 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1ec031830 con 0x7fc1f815e2a0 2024-03-24T00:43:18.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.629+0000 7fc1fd12a640 1 -- 172.21.15.67:0/3046992949 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1ec031c00 con 0x7fc1f815e2a0 2024-03-24T00:43:18.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.630+0000 7fc200390640 1 -- 172.21.15.67:0/3046992949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1f815e2a0 msgr2=0x7fc1f80ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:18.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.630+0000 7fc200390640 1 --2- 172.21.15.67:0/3046992949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1f815e2a0 0x7fc1f80ce630 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fc1f80cefb0 tx=0x7fc1ec02b8d0 comp rx=0 tx=0).stop 2024-03-24T00:43:18.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.630+0000 7fc200390640 1 -- 172.21.15.67:0/3046992949 shutdown_connections 2024-03-24T00:43:18.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.630+0000 7fc200390640 1 --2- 172.21.15.67:0/3046992949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1f815e2a0 0x7fc1f80ce630 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:18.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.630+0000 7fc200390640 1 -- 172.21.15.67:0/3046992949 >> 172.21.15.67:0/3046992949 conn(0x7fc1f806c810 msgr2=0x7fc1f806cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:18.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.630+0000 7fc200390640 1 -- 172.21.15.67:0/3046992949 shutdown_connections 2024-03-24T00:43:18.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.630+0000 7fc200390640 1 -- 172.21.15.67:0/3046992949 wait complete. 2024-03-24T00:43:18.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.631+0000 7fc200390640 1 Processor -- start 2024-03-24T00:43:18.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.631+0000 7fc200390640 1 -- start start 2024-03-24T00:43:18.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.632+0000 7fc200390640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc1f80d0ef0 0x7fc1f80b3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:18.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.632+0000 7fc200390640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1f815dec0 0x7fc1f815b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:18.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.632+0000 7fc200390640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc1f815e2a0 0x7fc1f815bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:18.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.632+0000 7fc200390640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f8162a40 con 0x7fc1f815dec0 2024-03-24T00:43:18.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.632+0000 7fc200390640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f81628c0 con 0x7fc1f80d0ef0 2024-03-24T00:43:18.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.632+0000 7fc200390640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f80d2120 con 0x7fc1f815e2a0 2024-03-24T00:43:18.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.632+0000 7fc1fd92b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1f815dec0 0x7fc1f815b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:18.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.632+0000 7fc1fd92b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1f815dec0 0x7fc1f815b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50390/0 (socket says 172.21.15.67:50390) 2024-03-24T00:43:18.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.632+0000 7fc1fd92b640 1 -- 172.21.15.67:0/3625647067 learned_addr learned my addr 172.21.15.67:0/3625647067 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:18.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.632+0000 7fc1fe12c640 1 --2- 172.21.15.67:0/3625647067 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc1f80d0ef0 0x7fc1f80b3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:18.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.632+0000 7fc1fd92b640 1 -- 172.21.15.67:0/3625647067 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc1f815e2a0 msgr2=0x7fc1f815bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:18.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.632+0000 7fc1fd92b640 1 --2- 172.21.15.67:0/3625647067 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc1f815e2a0 0x7fc1f815bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:18.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.632+0000 7fc1fd92b640 1 -- 172.21.15.67:0/3625647067 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc1f80d0ef0 msgr2=0x7fc1f80b3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:18.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.632+0000 7fc1fd92b640 1 --2- 172.21.15.67:0/3625647067 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc1f80d0ef0 0x7fc1f80b3a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:18.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.632+0000 7fc1fd92b640 1 -- 172.21.15.67:0/3625647067 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f81448a0 con 0x7fc1f815dec0 2024-03-24T00:43:18.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.632+0000 7fc1fd92b640 1 --2- 172.21.15.67:0/3625647067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1f815dec0 0x7fc1f815b7a0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec000c00 tx=0x7fc1ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:18.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.633+0000 7fc1eaffd640 1 -- 172.21.15.67:0/3625647067 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1ec041070 con 0x7fc1f815dec0 2024-03-24T00:43:18.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.633+0000 7fc1eaffd640 1 -- 172.21.15.67:0/3625647067 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1ec004230 con 0x7fc1f815dec0 2024-03-24T00:43:18.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.633+0000 7fc200390640 1 -- 172.21.15.67:0/3625647067 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1f815c4d0 con 0x7fc1f815dec0 2024-03-24T00:43:18.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.633+0000 7fc1eaffd640 1 -- 172.21.15.67:0/3625647067 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1ec035040 con 0x7fc1f815dec0 2024-03-24T00:43:18.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.633+0000 7fc200390640 1 -- 172.21.15.67:0/3625647067 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1f815ca90 con 0x7fc1f815dec0 2024-03-24T00:43:18.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.634+0000 7fc200390640 1 -- 172.21.15.67:0/3625647067 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1f80d0120 con 0x7fc1f815dec0 2024-03-24T00:43:18.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.635+0000 7fc1eaffd640 1 -- 172.21.15.67:0/3625647067 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc1ec038020 con 0x7fc1f815dec0 2024-03-24T00:43:18.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.635+0000 7fc1eaffd640 1 --2- 172.21.15.67:0/3625647067 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc1dc06bbd0 0x7fc1dc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:18.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.635+0000 7fc1eaffd640 1 -- 172.21.15.67:0/3625647067 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc1ec0aca60 con 0x7fc1f815dec0 2024-03-24T00:43:18.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.635+0000 7fc1fe12c640 1 --2- 172.21.15.67:0/3625647067 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc1dc06bbd0 0x7fc1dc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:18.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.636+0000 7fc1fe12c640 1 --2- 172.21.15.67:0/3625647067 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc1dc06bbd0 0x7fc1dc06e090 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fc1f4000f60 tx=0x7fc1f4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:18.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.639+0000 7fc1eaffd640 1 -- 172.21.15.67:0/3625647067 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc1f80d0120 con 0x7fc1f815dec0 2024-03-24T00:43:18.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.819+0000 7fc200390640 1 -- 172.21.15.67:0/3625647067 --> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc1f804f400 con 0x7fc1dc06bbd0 2024-03-24T00:43:18.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.826+0000 7fc1eaffd640 1 -- 172.21.15.67:0/3625647067 <== mgr.4105 v2:172.21.15.72:6832/2768292274 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252073 (secure 0 0 0) 0x7fc1f804f400 con 0x7fc1dc06bbd0 2024-03-24T00:43:18.828 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:43:18.829 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-24T00:43:18.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.830+0000 7fc200390640 1 -- 172.21.15.67:0/3625647067 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc1dc06bbd0 msgr2=0x7fc1dc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:18.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.830+0000 7fc200390640 1 --2- 172.21.15.67:0/3625647067 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc1dc06bbd0 0x7fc1dc06e090 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fc1f4000f60 tx=0x7fc1f4009040 comp rx=0 tx=0).stop 2024-03-24T00:43:18.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.830+0000 7fc200390640 1 -- 172.21.15.67:0/3625647067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1f815dec0 msgr2=0x7fc1f815b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:18.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.830+0000 7fc200390640 1 --2- 172.21.15.67:0/3625647067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1f815dec0 0x7fc1f815b7a0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec000c00 tx=0x7fc1ec0046c0 comp rx=0 tx=0).stop 2024-03-24T00:43:18.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.830+0000 7fc200390640 1 -- 172.21.15.67:0/3625647067 shutdown_connections 2024-03-24T00:43:18.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.830+0000 7fc200390640 1 --2- 172.21.15.67:0/3625647067 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc1dc06bbd0 0x7fc1dc06e090 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:18.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.830+0000 7fc200390640 1 --2- 172.21.15.67:0/3625647067 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc1f815e2a0 0x7fc1f815bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:18.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.830+0000 7fc200390640 1 --2- 172.21.15.67:0/3625647067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1f815dec0 0x7fc1f815b7a0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:18.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.830+0000 7fc200390640 1 --2- 172.21.15.67:0/3625647067 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc1f80d0ef0 0x7fc1f80b3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:18.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.830+0000 7fc200390640 1 -- 172.21.15.67:0/3625647067 >> 172.21.15.67:0/3625647067 conn(0x7fc1f806c810 msgr2=0x7fc1f815e760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:18.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.830+0000 7fc200390640 1 -- 172.21.15.67:0/3625647067 shutdown_connections 2024-03-24T00:43:18.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.830+0000 7fc200390640 1 -- 172.21.15.67:0/3625647067 wait complete. 2024-03-24T00:43:18.841 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":208,"stamp":"2024-03-24T00:43:17.383486+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":187,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":233000,"kb_used_data":8488,"kb_used_omap":12,"kb_used_meta":224371,"kb_avail":754741720,"statfs":{"total":773094113280,"available":772855521280,"internally_reserved":0,"allocated":8691712,"data_stored":4584284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":229756465},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.571074"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.853378+0000","last_change":"2024-03-24T00:43:10.824156+0000","last_active":"2024-03-24T00:43:12.853378+0000","last_peered":"2024-03-24T00:43:12.853378+0000","last_clean":"2024-03-24T00:43:12.853378+0000","last_became_active":"2024-03-24T00:43:10.822882+0000","last_became_peered":"2024-03-24T00:43:10.822882+0000","last_unstale":"2024-03-24T00:43:12.853378+0000","last_undegraded":"2024-03-24T00:43:12.853378+0000","last_fullsized":"2024-03-24T00:43:12.853378+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:38:33.107986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.830453+0000","last_change":"2024-03-24T00:43:10.838215+0000","last_active":"2024-03-24T00:43:11.830453+0000","last_peered":"2024-03-24T00:43:11.830453+0000","last_clean":"2024-03-24T00:43:11.830453+0000","last_became_active":"2024-03-24T00:43:10.837975+0000","last_became_peered":"2024-03-24T00:43:10.837975+0000","last_unstale":"2024-03-24T00:43:11.830453+0000","last_undegraded":"2024-03-24T00:43:11.830453+0000","last_fullsized":"2024-03-24T00:43:11.830453+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:53:11.321434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.102061+0000","last_change":"2024-03-24T00:43:10.838246+0000","last_active":"2024-03-24T00:43:12.102061+0000","last_peered":"2024-03-24T00:43:12.102061+0000","last_clean":"2024-03-24T00:43:12.102061+0000","last_became_active":"2024-03-24T00:43:10.837968+0000","last_became_peered":"2024-03-24T00:43:10.837968+0000","last_unstale":"2024-03-24T00:43:12.102061+0000","last_undegraded":"2024-03-24T00:43:12.102061+0000","last_fullsized":"2024-03-24T00:43:12.102061+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:36:52.211347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.831102+0000","last_change":"2024-03-24T00:43:10.830481+0000","last_active":"2024-03-24T00:43:11.831102+0000","last_peered":"2024-03-24T00:43:11.831102+0000","last_clean":"2024-03-24T00:43:11.831102+0000","last_became_active":"2024-03-24T00:43:10.830102+0000","last_became_peered":"2024-03-24T00:43:10.830102+0000","last_unstale":"2024-03-24T00:43:11.831102+0000","last_undegraded":"2024-03-24T00:43:11.831102+0000","last_fullsized":"2024-03-24T00:43:11.831102+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:14:34.576611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.831351+0000","last_change":"2024-03-24T00:43:10.819692+0000","last_active":"2024-03-24T00:43:11.831351+0000","last_peered":"2024-03-24T00:43:11.831351+0000","last_clean":"2024-03-24T00:43:11.831351+0000","last_became_active":"2024-03-24T00:43:10.819425+0000","last_became_peered":"2024-03-24T00:43:10.819425+0000","last_unstale":"2024-03-24T00:43:11.831351+0000","last_undegraded":"2024-03-24T00:43:11.831351+0000","last_fullsized":"2024-03-24T00:43:11.831351+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:17:59.489389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.851935+0000","last_change":"2024-03-24T00:43:10.833651+0000","last_active":"2024-03-24T00:43:12.851935+0000","last_peered":"2024-03-24T00:43:12.851935+0000","last_clean":"2024-03-24T00:43:12.851935+0000","last_became_active":"2024-03-24T00:43:10.833374+0000","last_became_peered":"2024-03-24T00:43:10.833374+0000","last_unstale":"2024-03-24T00:43:12.851935+0000","last_undegraded":"2024-03-24T00:43:12.851935+0000","last_fullsized":"2024-03-24T00:43:12.851935+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:30:53.960721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.852143+0000","last_change":"2024-03-24T00:43:10.831820+0000","last_active":"2024-03-24T00:43:12.852143+0000","last_peered":"2024-03-24T00:43:12.852143+0000","last_clean":"2024-03-24T00:43:12.852143+0000","last_became_active":"2024-03-24T00:43:10.830998+0000","last_became_peered":"2024-03-24T00:43:10.830998+0000","last_unstale":"2024-03-24T00:43:12.852143+0000","last_undegraded":"2024-03-24T00:43:12.852143+0000","last_fullsized":"2024-03-24T00:43:12.852143+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:19:48.048357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.830751+0000","last_change":"2024-03-24T00:43:10.822376+0000","last_active":"2024-03-24T00:43:11.830751+0000","last_peered":"2024-03-24T00:43:11.830751+0000","last_clean":"2024-03-24T00:43:11.830751+0000","last_became_active":"2024-03-24T00:43:10.822042+0000","last_became_peered":"2024-03-24T00:43:10.822042+0000","last_unstale":"2024-03-24T00:43:11.830751+0000","last_undegraded":"2024-03-24T00:43:11.830751+0000","last_fullsized":"2024-03-24T00:43:11.830751+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:17:52.626613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.853828+0000","last_change":"2024-03-24T00:43:10.837255+0000","last_active":"2024-03-24T00:43:12.853828+0000","last_peered":"2024-03-24T00:43:12.853828+0000","last_clean":"2024-03-24T00:43:12.853828+0000","last_became_active":"2024-03-24T00:43:10.836977+0000","last_became_peered":"2024-03-24T00:43:10.836977+0000","last_unstale":"2024-03-24T00:43:12.853828+0000","last_undegraded":"2024-03-24T00:43:12.853828+0000","last_fullsized":"2024-03-24T00:43:12.853828+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:19:25.257173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.831304+0000","last_change":"2024-03-24T00:43:10.825991+0000","last_active":"2024-03-24T00:43:11.831304+0000","last_peered":"2024-03-24T00:43:11.831304+0000","last_clean":"2024-03-24T00:43:11.831304+0000","last_became_active":"2024-03-24T00:43:10.825554+0000","last_became_peered":"2024-03-24T00:43:10.825554+0000","last_unstale":"2024-03-24T00:43:11.831304+0000","last_undegraded":"2024-03-24T00:43:11.831304+0000","last_fullsized":"2024-03-24T00:43:11.831304+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:33:32.269240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.830316+0000","last_change":"2024-03-24T00:43:10.843472+0000","last_active":"2024-03-24T00:43:11.830316+0000","last_peered":"2024-03-24T00:43:11.830316+0000","last_clean":"2024-03-24T00:43:11.830316+0000","last_became_active":"2024-03-24T00:43:10.843160+0000","last_became_peered":"2024-03-24T00:43:10.843160+0000","last_unstale":"2024-03-24T00:43:11.830316+0000","last_undegraded":"2024-03-24T00:43:11.830316+0000","last_fullsized":"2024-03-24T00:43:11.830316+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:42:14.282601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.831176+0000","last_change":"2024-03-24T00:43:10.835510+0000","last_active":"2024-03-24T00:43:11.831176+0000","last_peered":"2024-03-24T00:43:11.831176+0000","last_clean":"2024-03-24T00:43:11.831176+0000","last_became_active":"2024-03-24T00:43:10.835227+0000","last_became_peered":"2024-03-24T00:43:10.835227+0000","last_unstale":"2024-03-24T00:43:11.831176+0000","last_undegraded":"2024-03-24T00:43:11.831176+0000","last_fullsized":"2024-03-24T00:43:11.831176+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:01:23.035377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.831398+0000","last_change":"2024-03-24T00:43:10.821164+0000","last_active":"2024-03-24T00:43:11.831398+0000","last_peered":"2024-03-24T00:43:11.831398+0000","last_clean":"2024-03-24T00:43:11.831398+0000","last_became_active":"2024-03-24T00:43:10.820709+0000","last_became_peered":"2024-03-24T00:43:10.820709+0000","last_unstale":"2024-03-24T00:43:11.831398+0000","last_undegraded":"2024-03-24T00:43:11.831398+0000","last_fullsized":"2024-03-24T00:43:11.831398+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:13:20.917390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.851944+0000","last_change":"2024-03-24T00:43:10.836706+0000","last_active":"2024-03-24T00:43:12.851944+0000","last_peered":"2024-03-24T00:43:12.851944+0000","last_clean":"2024-03-24T00:43:12.851944+0000","last_became_active":"2024-03-24T00:43:10.836363+0000","last_became_peered":"2024-03-24T00:43:10.836363+0000","last_unstale":"2024-03-24T00:43:12.851944+0000","last_undegraded":"2024-03-24T00:43:12.851944+0000","last_fullsized":"2024-03-24T00:43:12.851944+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:11:52.190154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.829593+0000","last_change":"2024-03-24T00:43:10.844813+0000","last_active":"2024-03-24T00:43:11.829593+0000","last_peered":"2024-03-24T00:43:11.829593+0000","last_clean":"2024-03-24T00:43:11.829593+0000","last_became_active":"2024-03-24T00:43:10.844464+0000","last_became_peered":"2024-03-24T00:43:10.844464+0000","last_unstale":"2024-03-24T00:43:11.829593+0000","last_undegraded":"2024-03-24T00:43:11.829593+0000","last_fullsized":"2024-03-24T00:43:11.829593+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:38:33.107986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.852137+0000","last_change":"2024-03-24T00:43:10.830783+0000","last_active":"2024-03-24T00:43:12.852137+0000","last_peered":"2024-03-24T00:43:12.852137+0000","last_clean":"2024-03-24T00:43:12.852137+0000","last_became_active":"2024-03-24T00:43:10.828872+0000","last_became_peered":"2024-03-24T00:43:10.828872+0000","last_unstale":"2024-03-24T00:43:12.852137+0000","last_undegraded":"2024-03-24T00:43:12.852137+0000","last_fullsized":"2024-03-24T00:43:12.852137+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:14:34.576611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.831745+0000","last_change":"2024-03-24T00:43:10.825448+0000","last_active":"2024-03-24T00:43:11.831745+0000","last_peered":"2024-03-24T00:43:11.831745+0000","last_clean":"2024-03-24T00:43:11.831745+0000","last_became_active":"2024-03-24T00:43:10.825185+0000","last_became_peered":"2024-03-24T00:43:10.825185+0000","last_unstale":"2024-03-24T00:43:11.831745+0000","last_undegraded":"2024-03-24T00:43:11.831745+0000","last_fullsized":"2024-03-24T00:43:11.831745+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:17:52.643920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.102029+0000","last_change":"2024-03-24T00:43:10.844941+0000","last_active":"2024-03-24T00:43:12.102029+0000","last_peered":"2024-03-24T00:43:12.102029+0000","last_clean":"2024-03-24T00:43:12.102029+0000","last_became_active":"2024-03-24T00:43:10.844679+0000","last_became_peered":"2024-03-24T00:43:10.844679+0000","last_unstale":"2024-03-24T00:43:12.102029+0000","last_undegraded":"2024-03-24T00:43:12.102029+0000","last_fullsized":"2024-03-24T00:43:12.102029+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:13:31.896940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.851675+0000","last_change":"2024-03-24T00:43:10.827815+0000","last_active":"2024-03-24T00:43:12.851675+0000","last_peered":"2024-03-24T00:43:12.851675+0000","last_clean":"2024-03-24T00:43:12.851675+0000","last_became_active":"2024-03-24T00:43:10.827544+0000","last_became_peered":"2024-03-24T00:43:10.827544+0000","last_unstale":"2024-03-24T00:43:12.851675+0000","last_undegraded":"2024-03-24T00:43:12.851675+0000","last_fullsized":"2024-03-24T00:43:12.851675+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:36:07.189985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.831539+0000","last_change":"2024-03-24T00:43:10.843345+0000","last_active":"2024-03-24T00:43:11.831539+0000","last_peered":"2024-03-24T00:43:11.831539+0000","last_clean":"2024-03-24T00:43:11.831539+0000","last_became_active":"2024-03-24T00:43:10.842973+0000","last_became_peered":"2024-03-24T00:43:10.842973+0000","last_unstale":"2024-03-24T00:43:11.831539+0000","last_undegraded":"2024-03-24T00:43:11.831539+0000","last_fullsized":"2024-03-24T00:43:11.831539+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:37:46.896792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.853086+0000","last_change":"2024-03-24T00:43:10.840409+0000","last_active":"2024-03-24T00:43:12.853086+0000","last_peered":"2024-03-24T00:43:12.853086+0000","last_clean":"2024-03-24T00:43:12.853086+0000","last_became_active":"2024-03-24T00:43:10.840068+0000","last_became_peered":"2024-03-24T00:43:10.840068+0000","last_unstale":"2024-03-24T00:43:12.853086+0000","last_undegraded":"2024-03-24T00:43:12.853086+0000","last_fullsized":"2024-03-24T00:43:12.853086+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:01:34.950966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.831666+0000","last_change":"2024-03-24T00:43:10.821368+0000","last_active":"2024-03-24T00:43:11.831666+0000","last_peered":"2024-03-24T00:43:11.831666+0000","last_clean":"2024-03-24T00:43:11.831666+0000","last_became_active":"2024-03-24T00:43:10.821082+0000","last_became_peered":"2024-03-24T00:43:10.821082+0000","last_unstale":"2024-03-24T00:43:11.831666+0000","last_undegraded":"2024-03-24T00:43:11.831666+0000","last_fullsized":"2024-03-24T00:43:11.831666+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:23:39.982261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.853209+0000","last_change":"2024-03-24T00:43:10.827823+0000","last_active":"2024-03-24T00:43:12.853209+0000","last_peered":"2024-03-24T00:43:12.853209+0000","last_clean":"2024-03-24T00:43:12.853209+0000","last_became_active":"2024-03-24T00:43:10.826784+0000","last_became_peered":"2024-03-24T00:43:10.826784+0000","last_unstale":"2024-03-24T00:43:12.853209+0000","last_undegraded":"2024-03-24T00:43:12.853209+0000","last_fullsized":"2024-03-24T00:43:12.853209+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:22:27.781248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.829522+0000","last_change":"2024-03-24T00:43:10.833270+0000","last_active":"2024-03-24T00:43:11.829522+0000","last_peered":"2024-03-24T00:43:11.829522+0000","last_clean":"2024-03-24T00:43:11.829522+0000","last_became_active":"2024-03-24T00:43:10.832905+0000","last_became_peered":"2024-03-24T00:43:10.832905+0000","last_unstale":"2024-03-24T00:43:11.829522+0000","last_undegraded":"2024-03-24T00:43:11.829522+0000","last_fullsized":"2024-03-24T00:43:11.829522+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:31:51.135294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.829895+0000","last_change":"2024-03-24T00:43:10.835987+0000","last_active":"2024-03-24T00:43:11.829895+0000","last_peered":"2024-03-24T00:43:11.829895+0000","last_clean":"2024-03-24T00:43:11.829895+0000","last_became_active":"2024-03-24T00:43:10.835682+0000","last_became_peered":"2024-03-24T00:43:10.835682+0000","last_unstale":"2024-03-24T00:43:11.829895+0000","last_undegraded":"2024-03-24T00:43:11.829895+0000","last_fullsized":"2024-03-24T00:43:11.829895+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:39:41.935465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.831100+0000","last_change":"2024-03-24T00:43:10.835591+0000","last_active":"2024-03-24T00:43:11.831100+0000","last_peered":"2024-03-24T00:43:11.831100+0000","last_clean":"2024-03-24T00:43:11.831100+0000","last_became_active":"2024-03-24T00:43:10.835280+0000","last_became_peered":"2024-03-24T00:43:10.835280+0000","last_unstale":"2024-03-24T00:43:11.831100+0000","last_undegraded":"2024-03-24T00:43:11.831100+0000","last_fullsized":"2024-03-24T00:43:11.831100+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:19:31.014199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.852677+0000","last_change":"2024-03-24T00:43:10.826092+0000","last_active":"2024-03-24T00:43:12.852677+0000","last_peered":"2024-03-24T00:43:12.852677+0000","last_clean":"2024-03-24T00:43:12.852677+0000","last_became_active":"2024-03-24T00:43:10.825799+0000","last_became_peered":"2024-03-24T00:43:10.825799+0000","last_unstale":"2024-03-24T00:43:12.852677+0000","last_undegraded":"2024-03-24T00:43:12.852677+0000","last_fullsized":"2024-03-24T00:43:12.852677+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:12:07.102629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.831555+0000","last_change":"2024-03-24T00:43:10.830905+0000","last_active":"2024-03-24T00:43:11.831555+0000","last_peered":"2024-03-24T00:43:11.831555+0000","last_clean":"2024-03-24T00:43:11.831555+0000","last_became_active":"2024-03-24T00:43:10.830634+0000","last_became_peered":"2024-03-24T00:43:10.830634+0000","last_unstale":"2024-03-24T00:43:11.831555+0000","last_undegraded":"2024-03-24T00:43:11.831555+0000","last_fullsized":"2024-03-24T00:43:11.831555+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:00:53.698992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.829979+0000","last_change":"2024-03-24T00:43:10.834746+0000","last_active":"2024-03-24T00:43:11.829979+0000","last_peered":"2024-03-24T00:43:11.829979+0000","last_clean":"2024-03-24T00:43:11.829979+0000","last_became_active":"2024-03-24T00:43:10.834469+0000","last_became_peered":"2024-03-24T00:43:10.834469+0000","last_unstale":"2024-03-24T00:43:11.829979+0000","last_undegraded":"2024-03-24T00:43:11.829979+0000","last_fullsized":"2024-03-24T00:43:11.829979+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:30:53.960721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.852642+0000","last_change":"2024-03-24T00:43:10.833060+0000","last_active":"2024-03-24T00:43:12.852642+0000","last_peered":"2024-03-24T00:43:12.852642+0000","last_clean":"2024-03-24T00:43:12.852642+0000","last_became_active":"2024-03-24T00:43:10.832772+0000","last_became_peered":"2024-03-24T00:43:10.832772+0000","last_unstale":"2024-03-24T00:43:12.852642+0000","last_undegraded":"2024-03-24T00:43:12.852642+0000","last_fullsized":"2024-03-24T00:43:12.852642+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:01:23.035377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.101715+0000","last_change":"2024-03-24T00:43:10.843524+0000","last_active":"2024-03-24T00:43:12.101715+0000","last_peered":"2024-03-24T00:43:12.101715+0000","last_clean":"2024-03-24T00:43:12.101715+0000","last_became_active":"2024-03-24T00:43:10.843248+0000","last_became_peered":"2024-03-24T00:43:10.843248+0000","last_unstale":"2024-03-24T00:43:12.101715+0000","last_undegraded":"2024-03-24T00:43:12.101715+0000","last_fullsized":"2024-03-24T00:43:12.101715+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:19:08.316847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.852690+0000","last_change":"2024-03-24T00:43:10.823084+0000","last_active":"2024-03-24T00:43:12.852690+0000","last_peered":"2024-03-24T00:43:12.852690+0000","last_clean":"2024-03-24T00:43:12.852690+0000","last_became_active":"2024-03-24T00:43:10.822771+0000","last_became_peered":"2024-03-24T00:43:10.822771+0000","last_unstale":"2024-03-24T00:43:12.852690+0000","last_undegraded":"2024-03-24T00:43:12.852690+0000","last_fullsized":"2024-03-24T00:43:12.852690+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:33:32.269240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.101676+0000","last_change":"2024-03-24T00:43:10.845574+0000","last_active":"2024-03-24T00:43:12.101676+0000","last_peered":"2024-03-24T00:43:12.101676+0000","last_clean":"2024-03-24T00:43:12.101676+0000","last_became_active":"2024-03-24T00:43:10.845272+0000","last_became_peered":"2024-03-24T00:43:10.845272+0000","last_unstale":"2024-03-24T00:43:12.101676+0000","last_undegraded":"2024-03-24T00:43:12.101676+0000","last_fullsized":"2024-03-24T00:43:12.101676+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:18:54.804661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"75'4","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.831805+0000","last_change":"2024-03-24T00:43:09.827745+0000","last_active":"2024-03-24T00:43:11.831805+0000","last_peered":"2024-03-24T00:43:11.831805+0000","last_clean":"2024-03-24T00:43:11.831805+0000","last_became_active":"2024-03-24T00:43:09.827489+0000","last_became_peered":"2024-03-24T00:43:09.827489+0000","last_unstale":"2024-03-24T00:43:11.831805+0000","last_undegraded":"2024-03-24T00:43:11.831805+0000","last_fullsized":"2024-03-24T00:43:11.831805+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:04:01.630948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.829604+0000","last_change":"2024-03-24T00:43:10.836869+0000","last_active":"2024-03-24T00:43:11.829604+0000","last_peered":"2024-03-24T00:43:11.829604+0000","last_clean":"2024-03-24T00:43:11.829604+0000","last_became_active":"2024-03-24T00:43:10.836608+0000","last_became_peered":"2024-03-24T00:43:10.836608+0000","last_unstale":"2024-03-24T00:43:11.829604+0000","last_undegraded":"2024-03-24T00:43:11.829604+0000","last_fullsized":"2024-03-24T00:43:11.829604+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:10:11.482157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"75'1","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.852975+0000","last_change":"2024-03-24T00:43:09.826378+0000","last_active":"2024-03-24T00:43:12.852975+0000","last_peered":"2024-03-24T00:43:12.852975+0000","last_clean":"2024-03-24T00:43:12.852975+0000","last_became_active":"2024-03-24T00:43:09.826104+0000","last_became_peered":"2024-03-24T00:43:09.826104+0000","last_unstale":"2024-03-24T00:43:12.852975+0000","last_undegraded":"2024-03-24T00:43:12.852975+0000","last_fullsized":"2024-03-24T00:43:12.852975+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:41:34.466184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.853052+0000","last_change":"2024-03-24T00:43:10.824509+0000","last_active":"2024-03-24T00:43:12.853052+0000","last_peered":"2024-03-24T00:43:12.853052+0000","last_clean":"2024-03-24T00:43:12.853052+0000","last_became_active":"2024-03-24T00:43:10.824201+0000","last_became_peered":"2024-03-24T00:43:10.824201+0000","last_unstale":"2024-03-24T00:43:12.853052+0000","last_undegraded":"2024-03-24T00:43:12.853052+0000","last_fullsized":"2024-03-24T00:43:12.853052+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:53:12.665046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"75'2","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.852742+0000","last_change":"2024-03-24T00:43:09.826382+0000","last_active":"2024-03-24T00:43:12.852742+0000","last_peered":"2024-03-24T00:43:12.852742+0000","last_clean":"2024-03-24T00:43:12.852742+0000","last_became_active":"2024-03-24T00:43:09.826150+0000","last_became_peered":"2024-03-24T00:43:09.826150+0000","last_unstale":"2024-03-24T00:43:12.852742+0000","last_undegraded":"2024-03-24T00:43:12.852742+0000","last_fullsized":"2024-03-24T00:43:12.852742+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:23:35.409741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.853322+0000","last_change":"2024-03-24T00:43:10.844623+0000","last_active":"2024-03-24T00:43:12.853322+0000","last_peered":"2024-03-24T00:43:12.853322+0000","last_clean":"2024-03-24T00:43:12.853322+0000","last_became_active":"2024-03-24T00:43:10.844319+0000","last_became_peered":"2024-03-24T00:43:10.844319+0000","last_unstale":"2024-03-24T00:43:12.853322+0000","last_undegraded":"2024-03-24T00:43:12.853322+0000","last_fullsized":"2024-03-24T00:43:12.853322+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:52:06.375447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"75'2","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.852256+0000","last_change":"2024-03-24T00:43:09.828399+0000","last_active":"2024-03-24T00:43:12.852256+0000","last_peered":"2024-03-24T00:43:12.852256+0000","last_clean":"2024-03-24T00:43:12.852256+0000","last_became_active":"2024-03-24T00:43:09.828101+0000","last_became_peered":"2024-03-24T00:43:09.828101+0000","last_unstale":"2024-03-24T00:43:12.852256+0000","last_undegraded":"2024-03-24T00:43:12.852256+0000","last_fullsized":"2024-03-24T00:43:12.852256+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:18:05.332714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.830052+0000","last_change":"2024-03-24T00:43:10.834103+0000","last_active":"2024-03-24T00:43:11.830052+0000","last_peered":"2024-03-24T00:43:11.830052+0000","last_clean":"2024-03-24T00:43:11.830052+0000","last_became_active":"2024-03-24T00:43:10.833842+0000","last_became_peered":"2024-03-24T00:43:10.833842+0000","last_unstale":"2024-03-24T00:43:11.830052+0000","last_undegraded":"2024-03-24T00:43:11.830052+0000","last_fullsized":"2024-03-24T00:43:11.830052+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:11:52.190154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"75'1","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.853485+0000","last_change":"2024-03-24T00:43:09.826758+0000","last_active":"2024-03-24T00:43:12.853485+0000","last_peered":"2024-03-24T00:43:12.853485+0000","last_clean":"2024-03-24T00:43:12.853485+0000","last_became_active":"2024-03-24T00:43:09.826495+0000","last_became_peered":"2024-03-24T00:43:09.826495+0000","last_unstale":"2024-03-24T00:43:12.853485+0000","last_undegraded":"2024-03-24T00:43:12.853485+0000","last_fullsized":"2024-03-24T00:43:12.853485+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:37:45.892946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.830325+0000","last_change":"2024-03-24T00:43:10.844044+0000","last_active":"2024-03-24T00:43:11.830325+0000","last_peered":"2024-03-24T00:43:11.830325+0000","last_clean":"2024-03-24T00:43:11.830325+0000","last_became_active":"2024-03-24T00:43:10.843779+0000","last_became_peered":"2024-03-24T00:43:10.843779+0000","last_unstale":"2024-03-24T00:43:11.830325+0000","last_undegraded":"2024-03-24T00:43:11.830325+0000","last_fullsized":"2024-03-24T00:43:11.830325+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:09:22.854673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"75'1","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.830760+0000","last_change":"2024-03-24T00:43:09.827568+0000","last_active":"2024-03-24T00:43:11.830760+0000","last_peered":"2024-03-24T00:43:11.830760+0000","last_clean":"2024-03-24T00:43:11.830760+0000","last_became_active":"2024-03-24T00:43:09.827267+0000","last_became_peered":"2024-03-24T00:43:09.827267+0000","last_unstale":"2024-03-24T00:43:11.830760+0000","last_undegraded":"2024-03-24T00:43:11.830760+0000","last_fullsized":"2024-03-24T00:43:11.830760+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:38:27.408597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.853860+0000","last_change":"2024-03-24T00:43:10.837426+0000","last_active":"2024-03-24T00:43:12.853860+0000","last_peered":"2024-03-24T00:43:12.853860+0000","last_clean":"2024-03-24T00:43:12.853860+0000","last_became_active":"2024-03-24T00:43:10.837074+0000","last_became_peered":"2024-03-24T00:43:10.837074+0000","last_unstale":"2024-03-24T00:43:12.853860+0000","last_undegraded":"2024-03-24T00:43:12.853860+0000","last_fullsized":"2024-03-24T00:43:12.853860+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:48:25.693365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.829680+0000","last_change":"2024-03-24T00:43:09.824279+0000","last_active":"2024-03-24T00:43:11.829680+0000","last_peered":"2024-03-24T00:43:11.829680+0000","last_clean":"2024-03-24T00:43:11.829680+0000","last_became_active":"2024-03-24T00:43:09.824012+0000","last_became_peered":"2024-03-24T00:43:09.824012+0000","last_unstale":"2024-03-24T00:43:11.829680+0000","last_undegraded":"2024-03-24T00:43:11.829680+0000","last_fullsized":"2024-03-24T00:43:11.829680+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:10:01.312078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.853367+0000","last_change":"2024-03-24T00:43:10.832566+0000","last_active":"2024-03-24T00:43:12.853367+0000","last_peered":"2024-03-24T00:43:12.853367+0000","last_clean":"2024-03-24T00:43:12.853367+0000","last_became_active":"2024-03-24T00:43:10.831785+0000","last_became_peered":"2024-03-24T00:43:10.831785+0000","last_unstale":"2024-03-24T00:43:12.853367+0000","last_undegraded":"2024-03-24T00:43:12.853367+0000","last_fullsized":"2024-03-24T00:43:12.853367+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:47:25.859204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.830694+0000","last_change":"2024-03-24T00:43:09.829647+0000","last_active":"2024-03-24T00:43:11.830694+0000","last_peered":"2024-03-24T00:43:11.830694+0000","last_clean":"2024-03-24T00:43:11.830694+0000","last_became_active":"2024-03-24T00:43:09.829343+0000","last_became_peered":"2024-03-24T00:43:09.829343+0000","last_unstale":"2024-03-24T00:43:11.830694+0000","last_undegraded":"2024-03-24T00:43:11.830694+0000","last_fullsized":"2024-03-24T00:43:11.830694+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:12:47.427076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.829634+0000","last_change":"2024-03-24T00:43:10.833824+0000","last_active":"2024-03-24T00:43:11.829634+0000","last_peered":"2024-03-24T00:43:11.829634+0000","last_clean":"2024-03-24T00:43:11.829634+0000","last_became_active":"2024-03-24T00:43:10.833578+0000","last_became_peered":"2024-03-24T00:43:10.833578+0000","last_unstale":"2024-03-24T00:43:11.829634+0000","last_undegraded":"2024-03-24T00:43:11.829634+0000","last_fullsized":"2024-03-24T00:43:11.829634+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:36:07.189985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.853433+0000","last_change":"2024-03-24T00:43:09.831076+0000","last_active":"2024-03-24T00:43:12.853433+0000","last_peered":"2024-03-24T00:43:12.853433+0000","last_clean":"2024-03-24T00:43:12.853433+0000","last_became_active":"2024-03-24T00:43:09.830803+0000","last_became_peered":"2024-03-24T00:43:09.830803+0000","last_unstale":"2024-03-24T00:43:12.853433+0000","last_undegraded":"2024-03-24T00:43:12.853433+0000","last_fullsized":"2024-03-24T00:43:12.853433+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:16:30.998272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.831282+0000","last_change":"2024-03-24T00:43:10.829477+0000","last_active":"2024-03-24T00:43:11.831282+0000","last_peered":"2024-03-24T00:43:11.831282+0000","last_clean":"2024-03-24T00:43:11.831282+0000","last_became_active":"2024-03-24T00:43:10.828756+0000","last_became_peered":"2024-03-24T00:43:10.828756+0000","last_unstale":"2024-03-24T00:43:11.831282+0000","last_undegraded":"2024-03-24T00:43:11.831282+0000","last_fullsized":"2024-03-24T00:43:11.831282+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:12:07.102629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.831309+0000","last_change":"2024-03-24T00:43:09.827651+0000","last_active":"2024-03-24T00:43:11.831309+0000","last_peered":"2024-03-24T00:43:11.831309+0000","last_clean":"2024-03-24T00:43:11.831309+0000","last_became_active":"2024-03-24T00:43:09.827327+0000","last_became_peered":"2024-03-24T00:43:09.827327+0000","last_unstale":"2024-03-24T00:43:11.831309+0000","last_undegraded":"2024-03-24T00:43:11.831309+0000","last_fullsized":"2024-03-24T00:43:11.831309+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:48:30.269130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.852059+0000","last_change":"2024-03-24T00:43:10.832792+0000","last_active":"2024-03-24T00:43:12.852059+0000","last_peered":"2024-03-24T00:43:12.852059+0000","last_clean":"2024-03-24T00:43:12.852059+0000","last_became_active":"2024-03-24T00:43:10.832466+0000","last_became_peered":"2024-03-24T00:43:10.832466+0000","last_unstale":"2024-03-24T00:43:12.852059+0000","last_undegraded":"2024-03-24T00:43:12.852059+0000","last_fullsized":"2024-03-24T00:43:12.852059+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:15:40.447819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"75'3","reported_seq":17,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.831743+0000","last_change":"2024-03-24T00:43:09.830357+0000","last_active":"2024-03-24T00:43:11.831743+0000","last_peered":"2024-03-24T00:43:11.831743+0000","last_clean":"2024-03-24T00:43:11.831743+0000","last_became_active":"2024-03-24T00:43:09.828901+0000","last_became_peered":"2024-03-24T00:43:09.828901+0000","last_unstale":"2024-03-24T00:43:11.831743+0000","last_undegraded":"2024-03-24T00:43:11.831743+0000","last_fullsized":"2024-03-24T00:43:11.831743+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:08:52.647569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.101827+0000","last_change":"2024-03-24T00:43:10.838245+0000","last_active":"2024-03-24T00:43:12.101827+0000","last_peered":"2024-03-24T00:43:12.101827+0000","last_clean":"2024-03-24T00:43:12.101827+0000","last_became_active":"2024-03-24T00:43:10.837966+0000","last_became_peered":"2024-03-24T00:43:10.837966+0000","last_unstale":"2024-03-24T00:43:12.101827+0000","last_undegraded":"2024-03-24T00:43:12.101827+0000","last_fullsized":"2024-03-24T00:43:12.101827+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:34:03.395954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"75'6","reported_seq":20,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.852304+0000","last_change":"2024-03-24T00:43:09.825748+0000","last_active":"2024-03-24T00:43:12.852304+0000","last_peered":"2024-03-24T00:43:12.852304+0000","last_clean":"2024-03-24T00:43:12.852304+0000","last_became_active":"2024-03-24T00:43:09.825469+0000","last_became_peered":"2024-03-24T00:43:09.825469+0000","last_unstale":"2024-03-24T00:43:12.852304+0000","last_undegraded":"2024-03-24T00:43:12.852304+0000","last_fullsized":"2024-03-24T00:43:12.852304+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:56:42.898290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.831404+0000","last_change":"2024-03-24T00:43:10.830424+0000","last_active":"2024-03-24T00:43:11.831404+0000","last_peered":"2024-03-24T00:43:11.831404+0000","last_clean":"2024-03-24T00:43:11.831404+0000","last_became_active":"2024-03-24T00:43:10.829770+0000","last_became_peered":"2024-03-24T00:43:10.829770+0000","last_unstale":"2024-03-24T00:43:11.831404+0000","last_undegraded":"2024-03-24T00:43:11.831404+0000","last_fullsized":"2024-03-24T00:43:11.831404+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:31:51.135294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"75'2","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.830942+0000","last_change":"2024-03-24T00:43:09.821932+0000","last_active":"2024-03-24T00:43:11.830942+0000","last_peered":"2024-03-24T00:43:11.830942+0000","last_clean":"2024-03-24T00:43:11.830942+0000","last_became_active":"2024-03-24T00:43:09.821665+0000","last_became_peered":"2024-03-24T00:43:09.821665+0000","last_unstale":"2024-03-24T00:43:11.830942+0000","last_undegraded":"2024-03-24T00:43:11.830942+0000","last_fullsized":"2024-03-24T00:43:11.830942+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:16:30.998272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.852293+0000","last_change":"2024-03-24T00:43:10.840466+0000","last_active":"2024-03-24T00:43:12.852293+0000","last_peered":"2024-03-24T00:43:12.852293+0000","last_clean":"2024-03-24T00:43:12.852293+0000","last_became_active":"2024-03-24T00:43:10.839609+0000","last_became_peered":"2024-03-24T00:43:10.839609+0000","last_unstale":"2024-03-24T00:43:12.852293+0000","last_undegraded":"2024-03-24T00:43:12.852293+0000","last_fullsized":"2024-03-24T00:43:12.852293+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:13:37.904410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.852706+0000","last_change":"2024-03-24T00:43:09.829536+0000","last_active":"2024-03-24T00:43:12.852706+0000","last_peered":"2024-03-24T00:43:12.852706+0000","last_clean":"2024-03-24T00:43:12.852706+0000","last_became_active":"2024-03-24T00:43:09.829260+0000","last_became_peered":"2024-03-24T00:43:09.829260+0000","last_unstale":"2024-03-24T00:43:12.852706+0000","last_undegraded":"2024-03-24T00:43:12.852706+0000","last_fullsized":"2024-03-24T00:43:12.852706+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:10:01.312078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.852601+0000","last_change":"2024-03-24T00:43:10.832446+0000","last_active":"2024-03-24T00:43:12.852601+0000","last_peered":"2024-03-24T00:43:12.852601+0000","last_clean":"2024-03-24T00:43:12.852601+0000","last_became_active":"2024-03-24T00:43:10.831852+0000","last_became_peered":"2024-03-24T00:43:10.831852+0000","last_unstale":"2024-03-24T00:43:12.852601+0000","last_undegraded":"2024-03-24T00:43:12.852601+0000","last_fullsized":"2024-03-24T00:43:12.852601+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:19:49.017409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.830885+0000","last_change":"2024-03-24T00:43:09.829607+0000","last_active":"2024-03-24T00:43:11.830885+0000","last_peered":"2024-03-24T00:43:11.830885+0000","last_clean":"2024-03-24T00:43:11.830885+0000","last_became_active":"2024-03-24T00:43:09.829347+0000","last_became_peered":"2024-03-24T00:43:09.829347+0000","last_unstale":"2024-03-24T00:43:11.830885+0000","last_undegraded":"2024-03-24T00:43:11.830885+0000","last_fullsized":"2024-03-24T00:43:11.830885+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:19:26.187451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.853402+0000","last_change":"2024-03-24T00:43:10.836785+0000","last_active":"2024-03-24T00:43:12.853402+0000","last_peered":"2024-03-24T00:43:12.853402+0000","last_clean":"2024-03-24T00:43:12.853402+0000","last_became_active":"2024-03-24T00:43:10.836516+0000","last_became_peered":"2024-03-24T00:43:10.836516+0000","last_unstale":"2024-03-24T00:43:12.853402+0000","last_undegraded":"2024-03-24T00:43:12.853402+0000","last_fullsized":"2024-03-24T00:43:12.853402+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:44:18.631971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"75'8","reported_seq":22,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.829580+0000","last_change":"2024-03-24T00:43:09.826848+0000","last_active":"2024-03-24T00:43:11.829580+0000","last_peered":"2024-03-24T00:43:11.829580+0000","last_clean":"2024-03-24T00:43:11.829580+0000","last_became_active":"2024-03-24T00:43:09.826548+0000","last_became_peered":"2024-03-24T00:43:09.826548+0000","last_unstale":"2024-03-24T00:43:11.829580+0000","last_undegraded":"2024-03-24T00:43:11.829580+0000","last_fullsized":"2024-03-24T00:43:11.829580+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:38:27.408597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.851583+0000","last_change":"2024-03-24T00:43:10.828447+0000","last_active":"2024-03-24T00:43:12.851583+0000","last_peered":"2024-03-24T00:43:12.851583+0000","last_clean":"2024-03-24T00:43:12.851583+0000","last_became_active":"2024-03-24T00:43:10.828157+0000","last_became_peered":"2024-03-24T00:43:10.828157+0000","last_unstale":"2024-03-24T00:43:12.851583+0000","last_undegraded":"2024-03-24T00:43:12.851583+0000","last_fullsized":"2024-03-24T00:43:12.851583+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:10:11.482157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"75'1","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.853534+0000","last_change":"2024-03-24T00:43:09.826478+0000","last_active":"2024-03-24T00:43:12.853534+0000","last_peered":"2024-03-24T00:43:12.853534+0000","last_clean":"2024-03-24T00:43:12.853534+0000","last_became_active":"2024-03-24T00:43:09.826185+0000","last_became_peered":"2024-03-24T00:43:09.826185+0000","last_unstale":"2024-03-24T00:43:12.853534+0000","last_undegraded":"2024-03-24T00:43:12.853534+0000","last_fullsized":"2024-03-24T00:43:12.853534+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:38:27.408597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.830934+0000","last_change":"2024-03-24T00:43:10.834127+0000","last_active":"2024-03-24T00:43:11.830934+0000","last_peered":"2024-03-24T00:43:11.830934+0000","last_clean":"2024-03-24T00:43:11.830934+0000","last_became_active":"2024-03-24T00:43:10.833870+0000","last_became_peered":"2024-03-24T00:43:10.833870+0000","last_unstale":"2024-03-24T00:43:11.830934+0000","last_undegraded":"2024-03-24T00:43:11.830934+0000","last_fullsized":"2024-03-24T00:43:11.830934+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:27:13.070120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"75'2","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.853853+0000","last_change":"2024-03-24T00:43:09.825855+0000","last_active":"2024-03-24T00:43:12.853853+0000","last_peered":"2024-03-24T00:43:12.853853+0000","last_clean":"2024-03-24T00:43:12.853853+0000","last_became_active":"2024-03-24T00:43:09.825353+0000","last_became_peered":"2024-03-24T00:43:09.825353+0000","last_unstale":"2024-03-24T00:43:12.853853+0000","last_undegraded":"2024-03-24T00:43:12.853853+0000","last_fullsized":"2024-03-24T00:43:12.853853+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:13:19.913544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.831069+0000","last_change":"2024-03-24T00:43:10.832638+0000","last_active":"2024-03-24T00:43:11.831069+0000","last_peered":"2024-03-24T00:43:11.831069+0000","last_clean":"2024-03-24T00:43:11.831069+0000","last_became_active":"2024-03-24T00:43:10.832336+0000","last_became_peered":"2024-03-24T00:43:10.832336+0000","last_unstale":"2024-03-24T00:43:11.831069+0000","last_undegraded":"2024-03-24T00:43:11.831069+0000","last_fullsized":"2024-03-24T00:43:11.831069+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:14:38.749969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"75'1","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.831775+0000","last_change":"2024-03-24T00:43:09.829560+0000","last_active":"2024-03-24T00:43:11.831775+0000","last_peered":"2024-03-24T00:43:11.831775+0000","last_clean":"2024-03-24T00:43:11.831775+0000","last_became_active":"2024-03-24T00:43:09.829292+0000","last_became_peered":"2024-03-24T00:43:09.829292+0000","last_unstale":"2024-03-24T00:43:11.831775+0000","last_undegraded":"2024-03-24T00:43:11.831775+0000","last_fullsized":"2024-03-24T00:43:11.831775+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:33:04.005989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.851841+0000","last_change":"2024-03-24T00:43:10.824843+0000","last_active":"2024-03-24T00:43:12.851841+0000","last_peered":"2024-03-24T00:43:12.851841+0000","last_clean":"2024-03-24T00:43:12.851841+0000","last_became_active":"2024-03-24T00:43:10.824568+0000","last_became_peered":"2024-03-24T00:43:10.824568+0000","last_unstale":"2024-03-24T00:43:12.851841+0000","last_undegraded":"2024-03-24T00:43:12.851841+0000","last_fullsized":"2024-03-24T00:43:12.851841+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:39:41.935465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.101757+0000","last_change":"2024-03-24T00:43:09.830842+0000","last_active":"2024-03-24T00:43:12.101757+0000","last_peered":"2024-03-24T00:43:12.101757+0000","last_clean":"2024-03-24T00:43:12.101757+0000","last_became_active":"2024-03-24T00:43:09.830463+0000","last_became_peered":"2024-03-24T00:43:09.830463+0000","last_unstale":"2024-03-24T00:43:12.101757+0000","last_undegraded":"2024-03-24T00:43:12.101757+0000","last_fullsized":"2024-03-24T00:43:12.101757+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:19:47.044511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.830186+0000","last_change":"2024-03-24T00:43:10.837658+0000","last_active":"2024-03-24T00:43:11.830186+0000","last_peered":"2024-03-24T00:43:11.830186+0000","last_clean":"2024-03-24T00:43:11.830186+0000","last_became_active":"2024-03-24T00:43:10.837357+0000","last_became_peered":"2024-03-24T00:43:10.837357+0000","last_unstale":"2024-03-24T00:43:11.830186+0000","last_undegraded":"2024-03-24T00:43:11.830186+0000","last_fullsized":"2024-03-24T00:43:11.830186+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:21:47.552203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"75'2","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.851846+0000","last_change":"2024-03-24T00:43:09.825245+0000","last_active":"2024-03-24T00:43:12.851846+0000","last_peered":"2024-03-24T00:43:12.851846+0000","last_clean":"2024-03-24T00:43:12.851846+0000","last_became_active":"2024-03-24T00:43:09.824975+0000","last_became_peered":"2024-03-24T00:43:09.824975+0000","last_unstale":"2024-03-24T00:43:12.851846+0000","last_undegraded":"2024-03-24T00:43:12.851846+0000","last_fullsized":"2024-03-24T00:43:12.851846+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:14:14.121481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.830493+0000","last_change":"2024-03-24T00:43:10.833037+0000","last_active":"2024-03-24T00:43:11.830493+0000","last_peered":"2024-03-24T00:43:11.830493+0000","last_clean":"2024-03-24T00:43:11.830493+0000","last_became_active":"2024-03-24T00:43:10.832766+0000","last_became_peered":"2024-03-24T00:43:10.832766+0000","last_unstale":"2024-03-24T00:43:11.830493+0000","last_undegraded":"2024-03-24T00:43:11.830493+0000","last_fullsized":"2024-03-24T00:43:11.830493+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:47:19.214497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.851543+0000","last_change":"2024-03-24T00:43:09.827109+0000","last_active":"2024-03-24T00:43:12.851543+0000","last_peered":"2024-03-24T00:43:12.851543+0000","last_clean":"2024-03-24T00:43:12.851543+0000","last_became_active":"2024-03-24T00:43:09.826874+0000","last_became_peered":"2024-03-24T00:43:09.826874+0000","last_unstale":"2024-03-24T00:43:12.851543+0000","last_undegraded":"2024-03-24T00:43:12.851543+0000","last_fullsized":"2024-03-24T00:43:12.851543+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:24:43.787686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.853522+0000","last_change":"2024-03-24T00:43:10.832669+0000","last_active":"2024-03-24T00:43:12.853522+0000","last_peered":"2024-03-24T00:43:12.853522+0000","last_clean":"2024-03-24T00:43:12.853522+0000","last_became_active":"2024-03-24T00:43:10.832171+0000","last_became_peered":"2024-03-24T00:43:10.832171+0000","last_unstale":"2024-03-24T00:43:12.853522+0000","last_undegraded":"2024-03-24T00:43:12.853522+0000","last_fullsized":"2024-03-24T00:43:12.853522+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:03:52.283057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.851637+0000","last_change":"2024-03-24T00:43:09.827816+0000","last_active":"2024-03-24T00:43:12.851637+0000","last_peered":"2024-03-24T00:43:12.851637+0000","last_clean":"2024-03-24T00:43:12.851637+0000","last_became_active":"2024-03-24T00:43:09.827555+0000","last_became_peered":"2024-03-24T00:43:09.827555+0000","last_unstale":"2024-03-24T00:43:12.851637+0000","last_undegraded":"2024-03-24T00:43:12.851637+0000","last_fullsized":"2024-03-24T00:43:12.851637+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:11:49.925242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.853107+0000","last_change":"2024-03-24T00:43:10.834121+0000","last_active":"2024-03-24T00:43:12.853107+0000","last_peered":"2024-03-24T00:43:12.853107+0000","last_clean":"2024-03-24T00:43:12.853107+0000","last_became_active":"2024-03-24T00:43:10.833834+0000","last_became_peered":"2024-03-24T00:43:10.833834+0000","last_unstale":"2024-03-24T00:43:12.853107+0000","last_undegraded":"2024-03-24T00:43:12.853107+0000","last_fullsized":"2024-03-24T00:43:12.853107+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:22:27.749038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"75'2","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.831008+0000","last_change":"2024-03-24T00:43:09.823656+0000","last_active":"2024-03-24T00:43:11.831008+0000","last_peered":"2024-03-24T00:43:11.831008+0000","last_clean":"2024-03-24T00:43:11.831008+0000","last_became_active":"2024-03-24T00:43:09.823109+0000","last_became_peered":"2024-03-24T00:43:09.823109+0000","last_unstale":"2024-03-24T00:43:11.831008+0000","last_undegraded":"2024-03-24T00:43:11.831008+0000","last_fullsized":"2024-03-24T00:43:11.831008+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:23:35.409741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.852156+0000","last_change":"2024-03-24T00:43:10.836822+0000","last_active":"2024-03-24T00:43:12.852156+0000","last_peered":"2024-03-24T00:43:12.852156+0000","last_clean":"2024-03-24T00:43:12.852156+0000","last_became_active":"2024-03-24T00:43:10.836557+0000","last_became_peered":"2024-03-24T00:43:10.836557+0000","last_unstale":"2024-03-24T00:43:12.852156+0000","last_undegraded":"2024-03-24T00:43:12.852156+0000","last_fullsized":"2024-03-24T00:43:12.852156+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:35:45.892922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"76'1","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.830539+0000","last_change":"2024-03-24T00:43:09.830340+0000","last_active":"2024-03-24T00:43:11.830539+0000","last_peered":"2024-03-24T00:43:11.830539+0000","last_clean":"2024-03-24T00:43:11.830539+0000","last_became_active":"2024-03-24T00:43:09.829859+0000","last_became_peered":"2024-03-24T00:43:09.829859+0000","last_unstale":"2024-03-24T00:43:11.830539+0000","last_undegraded":"2024-03-24T00:43:11.830539+0000","last_fullsized":"2024-03-24T00:43:11.830539+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:41:34.466184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.853146+0000","last_change":"2024-03-24T00:43:10.823860+0000","last_active":"2024-03-24T00:43:12.853146+0000","last_peered":"2024-03-24T00:43:12.853146+0000","last_clean":"2024-03-24T00:43:12.853146+0000","last_became_active":"2024-03-24T00:43:10.823546+0000","last_became_peered":"2024-03-24T00:43:10.823546+0000","last_unstale":"2024-03-24T00:43:12.853146+0000","last_undegraded":"2024-03-24T00:43:12.853146+0000","last_fullsized":"2024-03-24T00:43:12.853146+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:43:07.870388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.852988+0000","last_change":"2024-03-24T00:43:09.829855+0000","last_active":"2024-03-24T00:43:12.852988+0000","last_peered":"2024-03-24T00:43:12.852988+0000","last_clean":"2024-03-24T00:43:12.852988+0000","last_became_active":"2024-03-24T00:43:09.829369+0000","last_became_peered":"2024-03-24T00:43:09.829369+0000","last_unstale":"2024-03-24T00:43:12.852988+0000","last_undegraded":"2024-03-24T00:43:12.852988+0000","last_fullsized":"2024-03-24T00:43:12.852988+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:48:30.269130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.830060+0000","last_change":"2024-03-24T00:43:10.831531+0000","last_active":"2024-03-24T00:43:11.830060+0000","last_peered":"2024-03-24T00:43:11.830060+0000","last_clean":"2024-03-24T00:43:11.830060+0000","last_became_active":"2024-03-24T00:43:10.831268+0000","last_became_peered":"2024-03-24T00:43:10.831268+0000","last_unstale":"2024-03-24T00:43:11.830060+0000","last_undegraded":"2024-03-24T00:43:11.830060+0000","last_fullsized":"2024-03-24T00:43:11.830060+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:15:40.447819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.852905+0000","last_change":"2024-03-24T00:43:09.827149+0000","last_active":"2024-03-24T00:43:12.852905+0000","last_peered":"2024-03-24T00:43:12.852905+0000","last_clean":"2024-03-24T00:43:12.852905+0000","last_became_active":"2024-03-24T00:43:09.825842+0000","last_became_peered":"2024-03-24T00:43:09.825842+0000","last_unstale":"2024-03-24T00:43:12.852905+0000","last_undegraded":"2024-03-24T00:43:12.852905+0000","last_fullsized":"2024-03-24T00:43:12.852905+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:44:53.438106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.852196+0000","last_change":"2024-03-24T00:43:10.837866+0000","last_active":"2024-03-24T00:43:12.852196+0000","last_peered":"2024-03-24T00:43:12.852196+0000","last_clean":"2024-03-24T00:43:12.852196+0000","last_became_active":"2024-03-24T00:43:10.837604+0000","last_became_peered":"2024-03-24T00:43:10.837604+0000","last_unstale":"2024-03-24T00:43:12.852196+0000","last_undegraded":"2024-03-24T00:43:12.852196+0000","last_fullsized":"2024-03-24T00:43:12.852196+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:02:07.497122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.852197+0000","last_change":"2024-03-24T00:43:09.829297+0000","last_active":"2024-03-24T00:43:12.852197+0000","last_peered":"2024-03-24T00:43:12.852197+0000","last_clean":"2024-03-24T00:43:12.852197+0000","last_became_active":"2024-03-24T00:43:09.828970+0000","last_became_peered":"2024-03-24T00:43:09.828970+0000","last_unstale":"2024-03-24T00:43:12.852197+0000","last_undegraded":"2024-03-24T00:43:12.852197+0000","last_fullsized":"2024-03-24T00:43:12.852197+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:04:56.380202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.852443+0000","last_change":"2024-03-24T00:43:10.840766+0000","last_active":"2024-03-24T00:43:12.852443+0000","last_peered":"2024-03-24T00:43:12.852443+0000","last_clean":"2024-03-24T00:43:12.852443+0000","last_became_active":"2024-03-24T00:43:10.840471+0000","last_became_peered":"2024-03-24T00:43:10.840471+0000","last_unstale":"2024-03-24T00:43:12.852443+0000","last_undegraded":"2024-03-24T00:43:12.852443+0000","last_fullsized":"2024-03-24T00:43:12.852443+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:09:35.809870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:11.831204+0000","last_change":"2024-03-24T00:43:09.827798+0000","last_active":"2024-03-24T00:43:11.831204+0000","last_peered":"2024-03-24T00:43:11.831204+0000","last_clean":"2024-03-24T00:43:11.831204+0000","last_became_active":"2024-03-24T00:43:09.827515+0000","last_became_peered":"2024-03-24T00:43:09.827515+0000","last_unstale":"2024-03-24T00:43:11.831204+0000","last_undegraded":"2024-03-24T00:43:11.831204+0000","last_fullsized":"2024-03-24T00:43:11.831204+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:26:37.415032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.102079+0000","last_change":"2024-03-24T00:43:10.838995+0000","last_active":"2024-03-24T00:43:12.102079+0000","last_peered":"2024-03-24T00:43:12.102079+0000","last_clean":"2024-03-24T00:43:12.102079+0000","last_became_active":"2024-03-24T00:43:10.838725+0000","last_became_peered":"2024-03-24T00:43:10.838725+0000","last_unstale":"2024-03-24T00:43:12.102079+0000","last_undegraded":"2024-03-24T00:43:12.102079+0000","last_fullsized":"2024-03-24T00:43:12.102079+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:14:33.573743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.852325+0000","last_change":"2024-03-24T00:43:09.831419+0000","last_active":"2024-03-24T00:43:12.852325+0000","last_peered":"2024-03-24T00:43:12.852325+0000","last_clean":"2024-03-24T00:43:12.852325+0000","last_became_active":"2024-03-24T00:43:09.831116+0000","last_became_peered":"2024-03-24T00:43:09.831116+0000","last_unstale":"2024-03-24T00:43:12.852325+0000","last_undegraded":"2024-03-24T00:43:12.852325+0000","last_fullsized":"2024-03-24T00:43:12.852325+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:36:42.771157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.102176+0000","last_change":"2024-03-24T00:43:10.841568+0000","last_active":"2024-03-24T00:43:12.102176+0000","last_peered":"2024-03-24T00:43:12.102176+0000","last_clean":"2024-03-24T00:43:12.102176+0000","last_became_active":"2024-03-24T00:43:10.841315+0000","last_became_peered":"2024-03-24T00:43:10.841315+0000","last_unstale":"2024-03-24T00:43:12.102176+0000","last_undegraded":"2024-03-24T00:43:12.102176+0000","last_fullsized":"2024-03-24T00:43:12.102176+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:48:14.736388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.101569+0000","last_change":"2024-03-24T00:43:09.823570+0000","last_active":"2024-03-24T00:43:12.101569+0000","last_peered":"2024-03-24T00:43:12.101569+0000","last_clean":"2024-03-24T00:43:12.101569+0000","last_became_active":"2024-03-24T00:43:09.823302+0000","last_became_peered":"2024-03-24T00:43:09.823302+0000","last_unstale":"2024-03-24T00:43:12.101569+0000","last_undegraded":"2024-03-24T00:43:12.101569+0000","last_fullsized":"2024-03-24T00:43:12.101569+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:45:52.448170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.853002+0000","last_change":"2024-03-24T00:43:10.840301+0000","last_active":"2024-03-24T00:43:12.853002+0000","last_peered":"2024-03-24T00:43:12.853002+0000","last_clean":"2024-03-24T00:43:12.853002+0000","last_became_active":"2024-03-24T00:43:10.839999+0000","last_became_peered":"2024-03-24T00:43:10.839999+0000","last_unstale":"2024-03-24T00:43:12.853002+0000","last_undegraded":"2024-03-24T00:43:12.853002+0000","last_fullsized":"2024-03-24T00:43:12.853002+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:09.804492+0000","last_clean_scrub_stamp":"2024-03-24T00:43:09.804492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:31:51.135294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.852064+0000","last_change":"2024-03-24T00:43:09.826753+0000","last_active":"2024-03-24T00:43:12.852064+0000","last_peered":"2024-03-24T00:43:12.852064+0000","last_clean":"2024-03-24T00:43:12.852064+0000","last_became_active":"2024-03-24T00:43:09.826434+0000","last_became_peered":"2024-03-24T00:43:09.826434+0000","last_unstale":"2024-03-24T00:43:12.852064+0000","last_undegraded":"2024-03-24T00:43:12.852064+0000","last_fullsized":"2024-03-24T00:43:12.852064+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:43:08.800646+0000","last_clean_scrub_stamp":"2024-03-24T00:43:08.800646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:19:30.010353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":274,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-24T00:43:12.852755+0000","last_change":"2024-03-24T00:38:40.953871+0000","last_active":"2024-03-24T00:43:12.852755+0000","last_peered":"2024-03-24T00:43:12.852755+0000","last_clean":"2024-03-24T00:43:12.852755+0000","last_became_active":"2024-03-24T00:38:40.952926+0000","last_became_peered":"2024-03-24T00:38:40.952926+0000","last_unstale":"2024-03-24T00:43:12.852755+0000","last_undegraded":"2024-03-24T00:43:12.852755+0000","last_fullsized":"2024-03-24T00:43:12.852755+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_clean_scrub_stamp":"2024-03-24T00:38:39.722194+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:05.397257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574908,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32676,"kb_used_data":1492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339164,"statfs":{"total":96636764160,"available":96603303936,"internally_reserved":0,"allocated":1527808,"data_stored":1015231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574907,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27940,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343900,"statfs":{"total":96636764160,"available":96608153600,"internally_reserved":0,"allocated":942080,"data_stored":426453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574907,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27940,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343900,"statfs":{"total":96636764160,"available":96608153600,"internally_reserved":0,"allocated":933888,"data_stored":424875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574906,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27948,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343892,"statfs":{"total":96636764160,"available":96608145408,"internally_reserved":0,"allocated":950272,"data_stored":426557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574906,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27932,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343908,"statfs":{"total":96636764160,"available":96608161792,"internally_reserved":0,"allocated":933888,"data_stored":425423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574907,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27948,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343892,"statfs":{"total":96636764160,"available":96608145408,"internally_reserved":0,"allocated":942080,"data_stored":425583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574907,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27944,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343896,"statfs":{"total":96636764160,"available":96608149504,"internally_reserved":0,"allocated":937984,"data_stored":424953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574906,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32672,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":1523712,"data_stored":1015209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:43:18.845 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-24T00:43:18.845 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-24T00:43:18.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.991+0000 7fed7f39f640 1 Processor -- start 2024-03-24T00:43:18.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.991+0000 7fed7f39f640 1 -- start start 2024-03-24T00:43:18.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.992+0000 7fed7f39f640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fed780b9530 0x7fed780d0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:18.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.992+0000 7fed7f39f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed78162920 con 0x7fed7815de40 2024-03-24T00:43:18.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.992+0000 7fed7f39f640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed78162360 con 0x7fed780d0880 2024-03-24T00:43:18.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.992+0000 7fed7f39f640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed78162640 con 0x7fed780b9530 2024-03-24T00:43:18.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.992+0000 7fed7e39d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fed7815de40 0x7fed78161a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37534/0 (socket says 172.21.15.67:37534) 2024-03-24T00:43:18.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.992+0000 7fed7e39d640 1 -- 172.21.15.67:0/3090063364 learned_addr learned my addr 172.21.15.67:0/3090063364 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:43:18.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.992+0000 7fed7d39b640 1 -- 172.21.15.67:0/3090063364 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3696884266 0 0) 0x7fed78162920 con 0x7fed7815de40 2024-03-24T00:43:18.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.992+0000 7fed7d39b640 1 -- 172.21.15.67:0/3090063364 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed64003520 con 0x7fed7815de40 2024-03-24T00:43:18.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.992+0000 7fed7db9c640 1 --2- 172.21.15.67:0/3090063364 >> v2:172.21.15.72:3300/0 conn(0x7fed780b9530 0x7fed780d0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:18.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.992+0000 7fed7d39b640 1 -- 172.21.15.67:0/3090063364 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1381431377 0 0) 0x7fed64003520 con 0x7fed7815de40 2024-03-24T00:43:18.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.992+0000 7fed7d39b640 1 -- 172.21.15.67:0/3090063364 >> v2:172.21.15.72:3300/0 conn(0x7fed780b9530 msgr2=0x7fed780d0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:18.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.992+0000 7fed7d39b640 1 --2- 172.21.15.67:0/3090063364 >> v2:172.21.15.72:3300/0 conn(0x7fed780b9530 0x7fed780d0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:18.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.992+0000 7fed7d39b640 1 -- 172.21.15.67:0/3090063364 >> v1:172.21.15.72:6789/0 conn(0x7fed780d0880 legacy=0x7fed780d25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:18.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.992+0000 7fed7d39b640 1 -- 172.21.15.67:0/3090063364 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed780d1dc0 con 0x7fed7815de40 2024-03-24T00:43:18.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.992+0000 7fed7d39b640 1 -- 172.21.15.67:0/3090063364 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7fed74001670 con 0x7fed7815de40 2024-03-24T00:43:18.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.993+0000 7fed7d39b640 1 -- 172.21.15.67:0/3090063364 >> v1:172.21.15.67:6789/0 conn(0x7fed7815de40 legacy=0x7fed78161a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:18.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.993+0000 7fed7d39b640 1 --2- 172.21.15.67:0/3090063364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed640039b0 0x7fed64003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:18.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.993+0000 7fed7d39b640 1 --2- 172.21.15.67:0/3090063364 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fed64004800 0x7fed64004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:18.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.993+0000 7fed7d39b640 1 --2- 172.21.15.67:0/3090063364 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fed640051c0 0x7fed6400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:18.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.993+0000 7fed7d39b640 1 -- 172.21.15.67:0/3090063364 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed78162640 con 0x7fed640039b0 2024-03-24T00:43:18.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.993+0000 7fed7d39b640 1 -- 172.21.15.67:0/3090063364 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed6400c060 con 0x7fed64004800 2024-03-24T00:43:18.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.993+0000 7fed7d39b640 1 -- 172.21.15.67:0/3090063364 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed6400c260 con 0x7fed640051c0 2024-03-24T00:43:18.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.993+0000 7fed7e39d640 1 --2- 172.21.15.67:0/3090063364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed640039b0 0x7fed64003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:18.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.993+0000 7fed7db9c640 1 --2- 172.21.15.67:0/3090063364 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fed64004800 0x7fed64004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:18.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.993+0000 7fed7eb9e640 1 --2- 172.21.15.67:0/3090063364 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fed640051c0 0x7fed6400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:18.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.993+0000 7fed7e39d640 1 -- 172.21.15.67:0/3090063364 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fed640051c0 msgr2=0x7fed6400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:18.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.993+0000 7fed7e39d640 1 --2- 172.21.15.67:0/3090063364 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fed640051c0 0x7fed6400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:18.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.993+0000 7fed7e39d640 1 -- 172.21.15.67:0/3090063364 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fed64004800 msgr2=0x7fed64004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:18.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.993+0000 7fed7e39d640 1 --2- 172.21.15.67:0/3090063364 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fed64004800 0x7fed64004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:18.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.993+0000 7fed7eb9e640 1 -- 172.21.15.67:0/3090063364 reap_dead start 2024-03-24T00:43:18.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.993+0000 7fed7e39d640 1 -- 172.21.15.67:0/3090063364 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed64003520 con 0x7fed640039b0 2024-03-24T00:43:18.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.993+0000 7fed7db9c640 1 --2- 172.21.15.67:0/3090063364 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fed64004800 0x7fed64004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:43:18.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.994+0000 7fed7e39d640 1 --2- 172.21.15.67:0/3090063364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed640039b0 0x7fed64003dd0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fed7400c290 tx=0x7fed64005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb852bfbba042ceb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:18.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.994+0000 7fed7d39b640 1 -- 172.21.15.67:0/3090063364 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed74007d30 con 0x7fed640039b0 2024-03-24T00:43:18.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.994+0000 7fed7d39b640 1 -- 172.21.15.67:0/3090063364 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed74030070 con 0x7fed640039b0 2024-03-24T00:43:18.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.994+0000 7fed7d39b640 1 -- 172.21.15.67:0/3090063364 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed74005d30 con 0x7fed640039b0 2024-03-24T00:43:18.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.994+0000 7fed7f39f640 1 -- 172.21.15.67:0/3090063364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed640039b0 msgr2=0x7fed64003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:18.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.994+0000 7fed7f39f640 1 --2- 172.21.15.67:0/3090063364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed640039b0 0x7fed64003dd0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fed7400c290 tx=0x7fed64005830 comp rx=0 tx=0).stop 2024-03-24T00:43:18.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.995+0000 7fed7f39f640 1 -- 172.21.15.67:0/3090063364 shutdown_connections 2024-03-24T00:43:18.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.995+0000 7fed7f39f640 1 --2- 172.21.15.67:0/3090063364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed640039b0 0x7fed64003dd0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:18.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.995+0000 7fed7f39f640 1 -- 172.21.15.67:0/3090063364 >> 172.21.15.67:0/3090063364 conn(0x7fed7806c6d0 msgr2=0x7fed7806cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:18.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.995+0000 7fed7f39f640 1 -- 172.21.15.67:0/3090063364 shutdown_connections 2024-03-24T00:43:18.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.995+0000 7fed7f39f640 1 -- 172.21.15.67:0/3090063364 wait complete. 2024-03-24T00:43:19.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.996+0000 7fed7f39f640 1 Processor -- start 2024-03-24T00:43:19.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.996+0000 7fed7f39f640 1 -- start start 2024-03-24T00:43:19.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.996+0000 7fed7f39f640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fed640039b0 0x7fed781f46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:19.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.996+0000 7fed7f39f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed780cd020 0x7fed7815a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:19.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.996+0000 7fed7f39f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fed780cd4a0 0x7fed7815ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:19.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.996+0000 7fed7f39f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed7815e220 con 0x7fed780cd020 2024-03-24T00:43:19.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.996+0000 7fed7f39f640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed780d0cd0 con 0x7fed780cd4a0 2024-03-24T00:43:19.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.996+0000 7fed7f39f640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed7815e3a0 con 0x7fed640039b0 2024-03-24T00:43:19.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.997+0000 7fed7db9c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed780cd020 0x7fed7815a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:19.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.997+0000 7fed7db9c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed780cd020 0x7fed7815a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50408/0 (socket says 172.21.15.67:50408) 2024-03-24T00:43:19.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.997+0000 7fed7db9c640 1 -- 172.21.15.67:0/1007279317 learned_addr learned my addr 172.21.15.67:0/1007279317 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:19.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.997+0000 7fed7eb9e640 1 --2- 172.21.15.67:0/1007279317 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fed780cd4a0 0x7fed7815ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:19.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.997+0000 7fed7e39d640 1 --2- 172.21.15.67:0/1007279317 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fed640039b0 0x7fed781f46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:19.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.997+0000 7fed7db9c640 1 -- 172.21.15.67:0/1007279317 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fed640039b0 msgr2=0x7fed781f46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:19.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.997+0000 7fed7db9c640 1 --2- 172.21.15.67:0/1007279317 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fed640039b0 0x7fed781f46e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:19.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.997+0000 7fed7db9c640 1 -- 172.21.15.67:0/1007279317 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fed780cd4a0 msgr2=0x7fed7815ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:19.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.997+0000 7fed7db9c640 1 --2- 172.21.15.67:0/1007279317 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fed780cd4a0 0x7fed7815ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:19.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.997+0000 7fed7db9c640 1 -- 172.21.15.67:0/1007279317 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed781f4c20 con 0x7fed780cd020 2024-03-24T00:43:19.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.997+0000 7fed7eb9e640 1 --2- 172.21.15.67:0/1007279317 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fed780cd4a0 0x7fed7815ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:43:19.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.997+0000 7fed7e39d640 1 --2- 172.21.15.67:0/1007279317 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fed640039b0 0x7fed781f46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:43:19.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.997+0000 7fed7db9c640 1 --2- 172.21.15.67:0/1007279317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed780cd020 0x7fed7815a7c0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fed68006c00 tx=0x7fed6800f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:19.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.997+0000 7fed6effd640 1 -- 172.21.15.67:0/1007279317 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed68009690 con 0x7fed780cd020 2024-03-24T00:43:19.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.997+0000 7fed6effd640 1 -- 172.21.15.67:0/1007279317 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed68011c90 con 0x7fed780cd020 2024-03-24T00:43:19.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.997+0000 7fed7f39f640 1 -- 172.21.15.67:0/1007279317 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed781f4f50 con 0x7fed780cd020 2024-03-24T00:43:19.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.997+0000 7fed6effd640 1 -- 172.21.15.67:0/1007279317 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed68014440 con 0x7fed780cd020 2024-03-24T00:43:19.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.997+0000 7fed7f39f640 1 -- 172.21.15.67:0/1007279317 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed781f51f0 con 0x7fed780cd020 2024-03-24T00:43:19.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.998+0000 7fed7f39f640 1 -- 172.21.15.67:0/1007279317 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed7804f400 con 0x7fed780cd020 2024-03-24T00:43:19.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.999+0000 7fed6effd640 1 -- 172.21.15.67:0/1007279317 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fed68012040 con 0x7fed780cd020 2024-03-24T00:43:19.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.999+0000 7fed6effd640 1 --2- 172.21.15.67:0/1007279317 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fed5c06bbd0 0x7fed5c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:19.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:18.999+0000 7fed6effd640 1 -- 172.21.15.67:0/1007279317 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fed6808b080 con 0x7fed780cd020 2024-03-24T00:43:19.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.000+0000 7fed7e39d640 1 --2- 172.21.15.67:0/1007279317 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fed5c06bbd0 0x7fed5c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:19.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.000+0000 7fed7e39d640 1 --2- 172.21.15.67:0/1007279317 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fed5c06bbd0 0x7fed5c06e090 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fed74034000 tx=0x7fed74039040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:19.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.004+0000 7fed6effd640 1 -- 172.21.15.67:0/1007279317 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fed781f4c20 con 0x7fed780cd020 2024-03-24T00:43:19.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.309+0000 7fed7f39f640 1 -- 172.21.15.67:0/1007279317 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fed780b7e60 con 0x7fed780cd020 2024-03-24T00:43:19.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.311+0000 7fed6effd640 1 -- 172.21.15.67:0/1007279317 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v16) v1 ==== 190+0+147 (secure 0 0 0) 0x7fed68019030 con 0x7fed780cd020 2024-03-24T00:43:19.310 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-03-24T00:43:19.311 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQDcdf9lY4nvJRAA6HWOZDWA+SSvJVmyjG+51g== 2024-03-24T00:43:19.311 INFO:teuthology.orchestra.run.smithi067.stdout: caps mds = "allow" 2024-03-24T00:43:19.311 INFO:teuthology.orchestra.run.smithi067.stdout: caps mon = "allow r" 2024-03-24T00:43:19.311 INFO:teuthology.orchestra.run.smithi067.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-24T00:43:19.311 INFO:teuthology.orchestra.run.smithi067.stderr:updated caps for client.0 2024-03-24T00:43:19.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.313+0000 7fed7f39f640 1 -- 172.21.15.67:0/1007279317 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fed5c06bbd0 msgr2=0x7fed5c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:19.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.313+0000 7fed7f39f640 1 --2- 172.21.15.67:0/1007279317 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fed5c06bbd0 0x7fed5c06e090 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fed74034000 tx=0x7fed74039040 comp rx=0 tx=0).stop 2024-03-24T00:43:19.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.313+0000 7fed7f39f640 1 -- 172.21.15.67:0/1007279317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed780cd020 msgr2=0x7fed7815a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:19.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.313+0000 7fed7f39f640 1 --2- 172.21.15.67:0/1007279317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed780cd020 0x7fed7815a7c0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fed68006c00 tx=0x7fed6800f610 comp rx=0 tx=0).stop 2024-03-24T00:43:19.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.313+0000 7fed7f39f640 1 -- 172.21.15.67:0/1007279317 shutdown_connections 2024-03-24T00:43:19.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.313+0000 7fed7f39f640 1 --2- 172.21.15.67:0/1007279317 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fed5c06bbd0 0x7fed5c06e090 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:19.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.313+0000 7fed7f39f640 1 --2- 172.21.15.67:0/1007279317 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fed780cd4a0 0x7fed7815ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:19.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.313+0000 7fed7f39f640 1 --2- 172.21.15.67:0/1007279317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed780cd020 0x7fed7815a7c0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:19.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.313+0000 7fed7f39f640 1 --2- 172.21.15.67:0/1007279317 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fed640039b0 0x7fed781f46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:19.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.313+0000 7fed7f39f640 1 -- 172.21.15.67:0/1007279317 >> 172.21.15.67:0/1007279317 conn(0x7fed7806c6d0 msgr2=0x7fed7815bb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:19.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.313+0000 7fed7f39f640 1 -- 172.21.15.67:0/1007279317 shutdown_connections 2024-03-24T00:43:19.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.313+0000 7fed7f39f640 1 -- 172.21.15.67:0/1007279317 wait complete. 2024-03-24T00:43:19.327 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:43:19.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.428+0000 7f4a41960640 1 Processor -- start 2024-03-24T00:43:19.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.428+0000 7f4a41960640 1 -- start start 2024-03-24T00:43:19.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.429+0000 7f4a41960640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4a3c0b2e80 0x7f4a3c0b32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:19.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.429+0000 7f4a41960640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4a3c0cf4a0 0x7f4a3c0d0d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:19.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.429+0000 7f4a41960640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a3c162b40 con 0x7f4a3c0d0990 2024-03-24T00:43:19.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.429+0000 7f4a41960640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a3c0ce770 con 0x7f4a3c0cf4a0 2024-03-24T00:43:19.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.429+0000 7f4a41960640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a3c0ce9c0 con 0x7f4a3c0b2e80 2024-03-24T00:43:19.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.429+0000 7f4a3b7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4a3c0cf4a0 0x7f4a3c0d0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:19.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.429+0000 7f4a3b7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4a3c0cf4a0 0x7f4a3c0d0d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50424/0 (socket says 172.21.15.67:50424) 2024-03-24T00:43:19.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.429+0000 7f4a3b7fe640 1 -- 172.21.15.67:0/3058665457 learned_addr learned my addr 172.21.15.67:0/3058665457 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:19.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.429+0000 7f4a3a7fc640 1 --2- 172.21.15.67:0/3058665457 >> v2:172.21.15.72:3300/0 conn(0x7f4a3c0b2e80 0x7f4a3c0b32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:19.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.430+0000 7f4a3b7fe640 1 -- 172.21.15.67:0/3058665457 >> v2:172.21.15.72:3300/0 conn(0x7f4a3c0b2e80 msgr2=0x7f4a3c0b32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:19.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.430+0000 7f4a3b7fe640 1 --2- 172.21.15.67:0/3058665457 >> v2:172.21.15.72:3300/0 conn(0x7f4a3c0b2e80 0x7f4a3c0b32f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:19.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.430+0000 7f4a3b7fe640 1 -- 172.21.15.67:0/3058665457 >> v1:172.21.15.72:6789/0 conn(0x7f4a3c0d0990 legacy=0x7f4a3c15dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:19.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.430+0000 7f4a3b7fe640 1 -- 172.21.15.67:0/3058665457 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a3c0ceca0 con 0x7f4a3c0cf4a0 2024-03-24T00:43:19.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.430+0000 7f4a3a7fc640 1 --2- 172.21.15.67:0/3058665457 >> v2:172.21.15.72:3300/0 conn(0x7f4a3c0b2e80 0x7f4a3c0b32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:43:19.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.430+0000 7f4a3b7fe640 1 --2- 172.21.15.67:0/3058665457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a3c0cf4a0 0x7f4a3c0d0d70 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c067c10 tx=0x7f4a2c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71d89d30c10f4030 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:19.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.430+0000 7f4a39ffb640 1 -- 172.21.15.67:0/3058665457 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a2c035020 con 0x7f4a3c0cf4a0 2024-03-24T00:43:19.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.430+0000 7f4a39ffb640 1 -- 172.21.15.67:0/3058665457 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a2c031870 con 0x7f4a3c0cf4a0 2024-03-24T00:43:19.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.430+0000 7f4a39ffb640 1 -- 172.21.15.67:0/3058665457 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a2c031c40 con 0x7f4a3c0cf4a0 2024-03-24T00:43:19.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.431+0000 7f4a41960640 1 -- 172.21.15.67:0/3058665457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a3c0cf4a0 msgr2=0x7f4a3c0d0d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:19.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.431+0000 7f4a41960640 1 --2- 172.21.15.67:0/3058665457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a3c0cf4a0 0x7f4a3c0d0d70 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c067c10 tx=0x7f4a2c02b910 comp rx=0 tx=0).stop 2024-03-24T00:43:19.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.431+0000 7f4a41960640 1 -- 172.21.15.67:0/3058665457 shutdown_connections 2024-03-24T00:43:19.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.431+0000 7f4a41960640 1 --2- 172.21.15.67:0/3058665457 >> v2:172.21.15.72:3300/0 conn(0x7f4a3c0b2e80 0x7f4a3c0b32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:19.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.431+0000 7f4a41960640 1 --2- 172.21.15.67:0/3058665457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a3c0cf4a0 0x7f4a3c0d0d70 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:19.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.431+0000 7f4a41960640 1 -- 172.21.15.67:0/3058665457 >> 172.21.15.67:0/3058665457 conn(0x7f4a3c06c830 msgr2=0x7f4a3c06cc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:19.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.431+0000 7f4a41960640 1 -- 172.21.15.67:0/3058665457 shutdown_connections 2024-03-24T00:43:19.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.431+0000 7f4a41960640 1 -- 172.21.15.67:0/3058665457 wait complete. 2024-03-24T00:43:19.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.433+0000 7f4a41960640 1 Processor -- start 2024-03-24T00:43:19.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.433+0000 7f4a41960640 1 -- start start 2024-03-24T00:43:19.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.434+0000 7f4a41960640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a3c0b2e80 0x7f4a3c0cd580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:19.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.434+0000 7f4a41960640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a3c0cf4a0 0x7f4a3c1f34a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:19.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.434+0000 7f4a41960640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4a3c0d0990 0x7f4a3c1f39e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:19.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.434+0000 7f4a41960640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3c15e380 con 0x7f4a3c0cf4a0 2024-03-24T00:43:19.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.434+0000 7f4a41960640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3c1627c0 con 0x7f4a3c0b2e80 2024-03-24T00:43:19.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.434+0000 7f4a41960640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3c15e500 con 0x7f4a3c0d0990 2024-03-24T00:43:19.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.434+0000 7f4a3a7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a3c0cf4a0 0x7f4a3c1f34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:19.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.434+0000 7f4a3a7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a3c0cf4a0 0x7f4a3c1f34a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50440/0 (socket says 172.21.15.67:50440) 2024-03-24T00:43:19.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.434+0000 7f4a3a7fc640 1 -- 172.21.15.67:0/3365644462 learned_addr learned my addr 172.21.15.67:0/3365644462 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:19.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.435+0000 7f4a3affd640 1 --2- 172.21.15.67:0/3365644462 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a3c0b2e80 0x7f4a3c0cd580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:19.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.435+0000 7f4a3b7fe640 1 --2- 172.21.15.67:0/3365644462 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4a3c0d0990 0x7f4a3c1f39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:19.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.435+0000 7f4a3a7fc640 1 -- 172.21.15.67:0/3365644462 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4a3c0d0990 msgr2=0x7f4a3c1f39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:19.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.435+0000 7f4a3a7fc640 1 --2- 172.21.15.67:0/3365644462 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4a3c0d0990 0x7f4a3c1f39e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:19.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.435+0000 7f4a3a7fc640 1 -- 172.21.15.67:0/3365644462 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a3c0b2e80 msgr2=0x7f4a3c0cd580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:19.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.435+0000 7f4a3a7fc640 1 --2- 172.21.15.67:0/3365644462 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a3c0b2e80 0x7f4a3c0cd580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:19.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.435+0000 7f4a3a7fc640 1 -- 172.21.15.67:0/3365644462 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a3c0ca550 con 0x7f4a3c0cf4a0 2024-03-24T00:43:19.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.435+0000 7f4a3b7fe640 1 --2- 172.21.15.67:0/3365644462 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4a3c0d0990 0x7f4a3c1f39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:43:19.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.435+0000 7f4a3a7fc640 1 --2- 172.21.15.67:0/3365644462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a3c0cf4a0 0x7f4a3c1f34a0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f4a2400c7f0 tx=0x7f4a2400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:19.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.435+0000 7f4a33fff640 1 -- 172.21.15.67:0/3365644462 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a24015070 con 0x7f4a3c0cf4a0 2024-03-24T00:43:19.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.435+0000 7f4a33fff640 1 -- 172.21.15.67:0/3365644462 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a24007a30 con 0x7f4a3c0cf4a0 2024-03-24T00:43:19.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.435+0000 7f4a41960640 1 -- 172.21.15.67:0/3365644462 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a3c0ccac0 con 0x7f4a3c0cf4a0 2024-03-24T00:43:19.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.435+0000 7f4a33fff640 1 -- 172.21.15.67:0/3365644462 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a24007e00 con 0x7f4a3c0cf4a0 2024-03-24T00:43:19.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.435+0000 7f4a41960640 1 -- 172.21.15.67:0/3365644462 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a3c0cd080 con 0x7f4a3c0cf4a0 2024-03-24T00:43:19.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.436+0000 7f4a41960640 1 -- 172.21.15.67:0/3365644462 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a3c04f400 con 0x7f4a3c0cf4a0 2024-03-24T00:43:19.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.437+0000 7f4a33fff640 1 -- 172.21.15.67:0/3365644462 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4a24018070 con 0x7f4a3c0cf4a0 2024-03-24T00:43:19.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.437+0000 7f4a33fff640 1 --2- 172.21.15.67:0/3365644462 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4a1c06bbd0 0x7f4a1c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:19.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.437+0000 7f4a33fff640 1 -- 172.21.15.67:0/3365644462 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4a2408ab00 con 0x7f4a3c0cf4a0 2024-03-24T00:43:19.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.437+0000 7f4a3affd640 1 --2- 172.21.15.67:0/3365644462 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4a1c06bbd0 0x7f4a1c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:19.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.438+0000 7f4a3affd640 1 --2- 172.21.15.67:0/3365644462 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4a1c06bbd0 0x7f4a1c06e090 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f4a3400b450 tx=0x7f4a3400d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:19.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.442+0000 7f4a33fff640 1 -- 172.21.15.67:0/3365644462 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4a24058e10 con 0x7f4a3c0cf4a0 2024-03-24T00:43:19.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.662+0000 7f4a41960640 1 -- 172.21.15.67:0/3365644462 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a3c0cc8a0 con 0x7f4a3c0cf4a0 2024-03-24T00:43:19.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.663+0000 7f4a33fff640 1 -- 172.21.15.67:0/3365644462 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f4a2405ceb0 con 0x7f4a3c0cf4a0 2024-03-24T00:43:19.663 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:43:19.663 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":40,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5960,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1888378520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1888378520},{"type":"v1","addr":"172.21.15.67:6835","nonce":1888378520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6610,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/2539718386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2539718386},{"type":"v1","addr":"172.21.15.72:6836","nonce":2539718386}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6613,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2376370770","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2376370770},{"type":"v1","addr":"172.21.15.72:6837","nonce":2376370770}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:43:10.177088+0000","modified":"2024-03-24T00:43:12.181406+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6894},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6894":{"gid":6894,"name":"c","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6833/2392802741","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2392802741},{"type":"v1","addr":"172.21.15.67:6833","nonce":2392802741}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6894,"qdb_cluster":[6894]},"id":5}]} 2024-03-24T00:43:19.663 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 40 2024-03-24T00:43:19.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.665+0000 7f4a41960640 1 -- 172.21.15.67:0/3365644462 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4a1c06bbd0 msgr2=0x7f4a1c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:19.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.665+0000 7f4a41960640 1 --2- 172.21.15.67:0/3365644462 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4a1c06bbd0 0x7f4a1c06e090 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f4a3400b450 tx=0x7f4a3400d040 comp rx=0 tx=0).stop 2024-03-24T00:43:19.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.665+0000 7f4a41960640 1 -- 172.21.15.67:0/3365644462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a3c0cf4a0 msgr2=0x7f4a3c1f34a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:19.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.665+0000 7f4a41960640 1 --2- 172.21.15.67:0/3365644462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a3c0cf4a0 0x7f4a3c1f34a0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f4a2400c7f0 tx=0x7f4a2400ccb0 comp rx=0 tx=0).stop 2024-03-24T00:43:19.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.665+0000 7f4a41960640 1 -- 172.21.15.67:0/3365644462 shutdown_connections 2024-03-24T00:43:19.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.665+0000 7f4a41960640 1 --2- 172.21.15.67:0/3365644462 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4a1c06bbd0 0x7f4a1c06e090 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:19.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.665+0000 7f4a41960640 1 --2- 172.21.15.67:0/3365644462 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4a3c0d0990 0x7f4a3c1f39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:19.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.665+0000 7f4a41960640 1 --2- 172.21.15.67:0/3365644462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a3c0cf4a0 0x7f4a3c1f34a0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:19.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.665+0000 7f4a41960640 1 --2- 172.21.15.67:0/3365644462 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a3c0b2e80 0x7f4a3c0cd580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:19.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.665+0000 7f4a41960640 1 -- 172.21.15.67:0/3365644462 >> 172.21.15.67:0/3365644462 conn(0x7f4a3c06c830 msgr2=0x7f4a3c0b8460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:19.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.665+0000 7f4a41960640 1 -- 172.21.15.67:0/3365644462 shutdown_connections 2024-03-24T00:43:19.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:19.665+0000 7f4a41960640 1 -- 172.21.15.67:0/3365644462 wait complete. 2024-03-24T00:43:20.080 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 40, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:43:10.177088+0000', 'modified': '2024-03-24T00:43:12.181406+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6894}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6894': {'gid': 6894, 'name': 'c', 'rank': 0, 'incarnation': 38, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.67:6833/2392802741', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6832', 'nonce': 2392802741}, {'type': 'v1', 'addr': '172.21.15.67:6833', 'nonce': 2392802741}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6894, 'qdb_cluster': [6894]} 2024-03-24T00:43:20.080 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-24T00:43:20.081 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-24T00:43:20.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.184+0000 7f8271e85640 1 Processor -- start 2024-03-24T00:43:20.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.184+0000 7f8271e85640 1 -- start start 2024-03-24T00:43:20.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.184+0000 7f8271e85640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f826c0ce9c0 0x7f826c070300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:20.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.184+0000 7f8271e85640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f826c0d25f0 0x7f826c070840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:20.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.184+0000 7f8271e85640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f826c06fb70 con 0x7f826c15bfa0 2024-03-24T00:43:20.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.184+0000 7f8271e85640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f826c0cd270 con 0x7f826c0d25f0 2024-03-24T00:43:20.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.185+0000 7f8271e85640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f826c0d0af0 con 0x7f826c0ce9c0 2024-03-24T00:43:20.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.185+0000 7f826bfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f826c0d25f0 0x7f826c070840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:20.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.185+0000 7f826b7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f826c15bfa0 0x7f826c0ce2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37540/0 (socket says 172.21.15.67:37540) 2024-03-24T00:43:20.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.185+0000 7f826b7fe640 1 -- 172.21.15.67:0/4146014939 learned_addr learned my addr 172.21.15.67:0/4146014939 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:43:20.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.185+0000 7f826affd640 1 --2- 172.21.15.67:0/4146014939 >> v2:172.21.15.72:3300/0 conn(0x7f826c0ce9c0 0x7f826c070300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:20.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.185+0000 7f826bfff640 1 -- 172.21.15.67:0/4146014939 >> v2:172.21.15.72:3300/0 conn(0x7f826c0ce9c0 msgr2=0x7f826c070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:20.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.185+0000 7f826bfff640 1 --2- 172.21.15.67:0/4146014939 >> v2:172.21.15.72:3300/0 conn(0x7f826c0ce9c0 0x7f826c070300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.185+0000 7f826bfff640 1 -- 172.21.15.67:0/4146014939 >> v1:172.21.15.67:6789/0 conn(0x7f826c15bfa0 legacy=0x7f826c0ce2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:20.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.185+0000 7f826a7fc640 1 -- 172.21.15.67:0/4146014939 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 454621854 0 0) 0x7f826c06fb70 con 0x7f826c15bfa0 2024-03-24T00:43:20.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.185+0000 7f826bfff640 1 -- 172.21.15.67:0/4146014939 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f826c0cd4f0 con 0x7f826c0d25f0 2024-03-24T00:43:20.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.185+0000 7f826affd640 1 --2- 172.21.15.67:0/4146014939 >> v2:172.21.15.72:3300/0 conn(0x7f826c0ce9c0 0x7f826c070300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:43:20.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.186+0000 7f826bfff640 1 --2- 172.21.15.67:0/4146014939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f826c0d25f0 0x7f826c070840 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f826c0734c0 tx=0x7f826402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5686fc60162e86f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:20.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.186+0000 7f826a7fc640 1 -- 172.21.15.67:0/4146014939 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8264039070 con 0x7f826c0d25f0 2024-03-24T00:43:20.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.186+0000 7f826a7fc640 1 -- 172.21.15.67:0/4146014939 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f826402cd00 con 0x7f826c0d25f0 2024-03-24T00:43:20.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.186+0000 7f826a7fc640 1 -- 172.21.15.67:0/4146014939 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8264034660 con 0x7f826c0d25f0 2024-03-24T00:43:20.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.187+0000 7f8271e85640 1 -- 172.21.15.67:0/4146014939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f826c0d25f0 msgr2=0x7f826c070840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:20.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.187+0000 7f8271e85640 1 --2- 172.21.15.67:0/4146014939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f826c0d25f0 0x7f826c070840 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f826c0734c0 tx=0x7f826402bf50 comp rx=0 tx=0).stop 2024-03-24T00:43:20.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.187+0000 7f8271e85640 1 -- 172.21.15.67:0/4146014939 shutdown_connections 2024-03-24T00:43:20.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.187+0000 7f8271e85640 1 --2- 172.21.15.67:0/4146014939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f826c0d25f0 0x7f826c070840 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.187+0000 7f8271e85640 1 --2- 172.21.15.67:0/4146014939 >> v2:172.21.15.72:3300/0 conn(0x7f826c0ce9c0 0x7f826c070300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.187+0000 7f8271e85640 1 -- 172.21.15.67:0/4146014939 >> 172.21.15.67:0/4146014939 conn(0x7f826c0d1ad0 msgr2=0x7f826c0d1ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:20.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.187+0000 7f8271e85640 1 -- 172.21.15.67:0/4146014939 shutdown_connections 2024-03-24T00:43:20.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.187+0000 7f8271e85640 1 -- 172.21.15.67:0/4146014939 wait complete. 2024-03-24T00:43:20.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.188+0000 7f8271e85640 1 Processor -- start 2024-03-24T00:43:20.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.189+0000 7f8271e85640 1 -- start start 2024-03-24T00:43:20.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.190+0000 7f8271e85640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f826c0ce9c0 0x7f826c15c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:20.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.190+0000 7f8271e85640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f826c0d25f0 0x7f826c15ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:20.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.190+0000 7f8271e85640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f826c15bfa0 0x7f826c0cab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:20.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.190+0000 7f8271e85640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f826c15c380 con 0x7f826c0ce9c0 2024-03-24T00:43:20.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.190+0000 7f8271e85640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f826c06fde0 con 0x7f826c15bfa0 2024-03-24T00:43:20.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.190+0000 7f8271e85640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f826c0cdab0 con 0x7f826c0d25f0 2024-03-24T00:43:20.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.190+0000 7f826b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f826c0ce9c0 0x7f826c15c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:20.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.190+0000 7f826b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f826c0ce9c0 0x7f826c15c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50458/0 (socket says 172.21.15.67:50458) 2024-03-24T00:43:20.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.190+0000 7f826b7fe640 1 -- 172.21.15.67:0/3787459908 learned_addr learned my addr 172.21.15.67:0/3787459908 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:20.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.190+0000 7f826b7fe640 1 -- 172.21.15.67:0/3787459908 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f826c0d25f0 msgr2=0x7f826c15ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:20.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.190+0000 7f826b7fe640 1 --2- 172.21.15.67:0/3787459908 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f826c0d25f0 0x7f826c15ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.190+0000 7f826b7fe640 1 -- 172.21.15.67:0/3787459908 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f826c15bfa0 msgr2=0x7f826c0cab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:20.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.190+0000 7f826b7fe640 1 --2- 172.21.15.67:0/3787459908 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f826c15bfa0 0x7f826c0cab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.190+0000 7f826b7fe640 1 -- 172.21.15.67:0/3787459908 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f826c0b5100 con 0x7f826c0ce9c0 2024-03-24T00:43:20.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.190+0000 7f826b7fe640 1 --2- 172.21.15.67:0/3787459908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f826c0ce9c0 0x7f826c15c500 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f825c00c4d0 tx=0x7f825c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:20.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.191+0000 7f8253fff640 1 -- 172.21.15.67:0/3787459908 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f825c018070 con 0x7f826c0ce9c0 2024-03-24T00:43:20.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.191+0000 7f8253fff640 1 -- 172.21.15.67:0/3787459908 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f825c007500 con 0x7f826c0ce9c0 2024-03-24T00:43:20.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.191+0000 7f8271e85640 1 -- 172.21.15.67:0/3787459908 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f826c0b53e0 con 0x7f826c0ce9c0 2024-03-24T00:43:20.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.191+0000 7f8253fff640 1 -- 172.21.15.67:0/3787459908 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f825c004a20 con 0x7f826c0ce9c0 2024-03-24T00:43:20.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.191+0000 7f8271e85640 1 -- 172.21.15.67:0/3787459908 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f826c0b59a0 con 0x7f826c0ce9c0 2024-03-24T00:43:20.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.193+0000 7f8253fff640 1 -- 172.21.15.67:0/3787459908 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f825c01b070 con 0x7f826c0ce9c0 2024-03-24T00:43:20.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.193+0000 7f8253fff640 1 --2- 172.21.15.67:0/3787459908 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f825806bbd0 0x7f825806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:20.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.193+0000 7f8253fff640 1 -- 172.21.15.67:0/3787459908 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f825c089ed0 con 0x7f826c0ce9c0 2024-03-24T00:43:20.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.193+0000 7f826affd640 1 --2- 172.21.15.67:0/3787459908 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f825806bbd0 0x7f825806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:20.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.194+0000 7f8274a75740 1 Processor -- start 2024-03-24T00:43:20.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.194+0000 7f8274a75740 1 -- start start 2024-03-24T00:43:20.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.194+0000 7f826affd640 1 --2- 172.21.15.67:0/3787459908 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f825806bbd0 0x7f825806e090 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f8260006fd0 tx=0x7f8260008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:20.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.195+0000 7f8274a75740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f182f6bc40 0x55f182f92770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:20.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.195+0000 7f8274a75740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f182f92cb0 0x55f182f6ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:20.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.195+0000 7f8274a75740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f182f6c020 0x55f182f6b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:20.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.195+0000 7f8274a75740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f182ecfc20 con 0x55f182f6c020 2024-03-24T00:43:20.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.195+0000 7f8274a75740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f182e78a40 con 0x55f182f92cb0 2024-03-24T00:43:20.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.195+0000 7f8274a75740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f182ecbcc0 con 0x55f182f6bc40 2024-03-24T00:43:20.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.195+0000 7f826bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f182f6c020 0x55f182f6b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:20.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.195+0000 7f826bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f182f6c020 0x55f182f6b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50474/0 (socket says 172.21.15.67:50474) 2024-03-24T00:43:20.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.195+0000 7f826bfff640 1 -- 172.21.15.67:0/3406116773 learned_addr learned my addr 172.21.15.67:0/3406116773 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:20.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.195+0000 7f826affd640 1 --2- 172.21.15.67:0/3406116773 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f182f92cb0 0x55f182f6ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:20.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.195+0000 7f826b7fe640 1 --2- 172.21.15.67:0/3406116773 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f182f6bc40 0x55f182f92770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:20.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.195+0000 7f826bfff640 1 -- 172.21.15.67:0/3406116773 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f182f6bc40 msgr2=0x55f182f92770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:20.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.195+0000 7f826bfff640 1 --2- 172.21.15.67:0/3406116773 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f182f6bc40 0x55f182f92770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.195+0000 7f826bfff640 1 -- 172.21.15.67:0/3406116773 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f182f92cb0 msgr2=0x55f182f6ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:20.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.195+0000 7f826bfff640 1 --2- 172.21.15.67:0/3406116773 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f182f92cb0 0x55f182f6ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.195+0000 7f826bfff640 1 -- 172.21.15.67:0/3406116773 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f182e9af90 con 0x55f182f6c020 2024-03-24T00:43:20.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.195+0000 7f826affd640 1 --2- 172.21.15.67:0/3406116773 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f182f92cb0 0x55f182f6ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:43:20.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.195+0000 7f826bfff640 1 --2- 172.21.15.67:0/3406116773 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f182f6c020 0x55f182f6b1c0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f8264009e80 tx=0x7f826402e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e9c7ea92e0e1d05 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:20.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.196+0000 7f82517fa640 1 -- 172.21.15.67:0/3406116773 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8264039050 con 0x55f182f6c020 2024-03-24T00:43:20.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.196+0000 7f82517fa640 1 -- 172.21.15.67:0/3406116773 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8264045070 con 0x55f182f6c020 2024-03-24T00:43:20.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.196+0000 7f82517fa640 1 -- 172.21.15.67:0/3406116773 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8264040450 con 0x55f182f6c020 2024-03-24T00:43:20.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.196+0000 7f8274a75740 1 -- 172.21.15.67:0/3406116773 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f182f6c020 msgr2=0x55f182f6b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:20.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.196+0000 7f8274a75740 1 --2- 172.21.15.67:0/3406116773 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f182f6c020 0x55f182f6b1c0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f8264009e80 tx=0x7f826402e070 comp rx=0 tx=0).stop 2024-03-24T00:43:20.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.196+0000 7f8274a75740 1 -- 172.21.15.67:0/3406116773 shutdown_connections 2024-03-24T00:43:20.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.196+0000 7f8274a75740 1 --2- 172.21.15.67:0/3406116773 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f182f6c020 0x55f182f6b1c0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.196+0000 7f8274a75740 1 --2- 172.21.15.67:0/3406116773 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f182f92cb0 0x55f182f6ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.196+0000 7f8274a75740 1 --2- 172.21.15.67:0/3406116773 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f182f6bc40 0x55f182f92770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.196+0000 7f8274a75740 1 -- 172.21.15.67:0/3406116773 >> 172.21.15.67:0/3406116773 conn(0x55f182f32a60 msgr2=0x55f182fb0960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:20.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.196+0000 7f8274a75740 1 -- 172.21.15.67:0/3406116773 shutdown_connections 2024-03-24T00:43:20.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.196+0000 7f8274a75740 1 -- 172.21.15.67:0/3406116773 wait complete. 2024-03-24T00:43:20.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.197+0000 7f8274a75740 1 -- start start 2024-03-24T00:43:20.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.197+0000 7f8274a75740 1 Processor -- start 2024-03-24T00:43:20.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.198+0000 7f8274a75740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f182f6bc40 0x55f182f8ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:20.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.198+0000 7f8274a75740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f182f6c020 0x55f182f90780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:20.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.198+0000 7f8274a75740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f182f90cc0 0x55f182f8b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:20.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.198+0000 7f8274a75740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f182f922e0 con 0x55f182f6bc40 2024-03-24T00:43:20.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.198+0000 7f8274a75740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f182e8be90 con 0x55f182f90cc0 2024-03-24T00:43:20.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.198+0000 7f8274a75740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f182f8b060 con 0x55f182f6c020 2024-03-24T00:43:20.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.198+0000 7f8274a75740 10 client.0 _ll_register_callbacks cb 0x55f182be3ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:43:20.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.198+0000 7f826b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f182f6bc40 0x55f182f8ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:20.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.198+0000 7f826b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f182f6bc40 0x55f182f8ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50480/0 (socket says 172.21.15.67:50480) 2024-03-24T00:43:20.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.198+0000 7f826b7fe640 1 -- 172.21.15.67:0/2974078692 learned_addr learned my addr 172.21.15.67:0/2974078692 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:20.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.198+0000 7f826bfff640 1 --2- 172.21.15.67:0/2974078692 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f182f90cc0 0x55f182f8b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:20.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.198+0000 7f826b7fe640 1 -- 172.21.15.67:0/2974078692 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f182f6c020 msgr2=0x55f182f90780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:20.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.198+0000 7f826affd640 1 --2- 172.21.15.67:0/2974078692 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f182f6c020 0x55f182f90780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:20.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.198+0000 7f826b7fe640 1 --2- 172.21.15.67:0/2974078692 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f182f6c020 0x55f182f90780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.198+0000 7f826b7fe640 1 -- 172.21.15.67:0/2974078692 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f182f90cc0 msgr2=0x55f182f8b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:20.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.198+0000 7f826b7fe640 1 --2- 172.21.15.67:0/2974078692 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f182f90cc0 0x55f182f8b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.198+0000 7f826b7fe640 1 -- 172.21.15.67:0/2974078692 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f182e989d0 con 0x55f182f6bc40 2024-03-24T00:43:20.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.198+0000 7f826bfff640 1 --2- 172.21.15.67:0/2974078692 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f182f90cc0 0x55f182f8b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:43:20.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.198+0000 7f826affd640 1 --2- 172.21.15.67:0/2974078692 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f182f6c020 0x55f182f90780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:43:20.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.198+0000 7f826b7fe640 1 --2- 172.21.15.67:0/2974078692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f182f6bc40 0x55f182f8ef70 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f825c001120 tx=0x7f825c05c310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f47e8e2e6f5ab1de server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:20.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.198+0000 7f824a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:43:20.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.198+0000 7f824a7fc640 1 -- 172.21.15.67:0/2974078692 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f825c004dc0 con 0x55f182f6bc40 2024-03-24T00:43:20.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.198+0000 7f824a7fc640 1 -- 172.21.15.67:0/2974078692 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(76..76 src has 1..76) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f825c06dd50 con 0x55f182f6bc40 2024-03-24T00:43:20.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.198+0000 7f8274a75740 1 -- 172.21.15.67:0/2974078692 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8264045070 con 0x55f182f6bc40 2024-03-24T00:43:20.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.199+0000 7f824a7fc640 1 -- 172.21.15.67:0/2974078692 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f825c057610 con 0x55f182f6bc40 2024-03-24T00:43:20.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.199+0000 7f8274a75740 10 client.7083 fetch_fsmap learned FSMap version 40 2024-03-24T00:43:20.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.199+0000 7f8274a75740 1 -- 172.21.15.67:0/2974078692 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=2+}) v3 -- 0x55f182be42d0 con 0x55f182f6bc40 2024-03-24T00:43:20.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.199+0000 7f824a7fc640 1 -- 172.21.15.67:0/2974078692 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8264045070 con 0x55f182f6bc40 2024-03-24T00:43:20.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.199+0000 7f824a7fc640 1 -- 172.21.15.67:0/2974078692 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 40) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f825c0577b0 con 0x55f182f6bc40 2024-03-24T00:43:20.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.199+0000 7f8274a75740 10 client.7083 fetch_fsmap finished waiting for FSMap version 40 2024-03-24T00:43:20.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.199+0000 7f8274a75740 10 client.7083 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T00:43:20.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.199+0000 7f8274a75740 20 client.7083 populate_metadata read hostname 'smithi067' 2024-03-24T00:43:20.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.199+0000 7f8274a75740 1 --2- 172.21.15.67:0/2974078692 >> [v2:172.21.15.67:6832/2392802741,v1:172.21.15.67:6833/2392802741] conn(0x55f182f89780 0x55f182f89b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:20.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.199+0000 7f8274a75740 4 client.7083 mds_command: new command op to 6894 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:43:20.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.199+0000 7f8274a75740 1 -- 172.21.15.67:0/2974078692 --> [v2:172.21.15.67:6832/2392802741,v1:172.21.15.67:6833/2392802741] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f182edd160 con 0x55f182f89780 2024-03-24T00:43:20.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.199+0000 7f826affd640 1 --2- 172.21.15.67:0/2974078692 >> [v2:172.21.15.67:6832/2392802741,v1:172.21.15.67:6833/2392802741] conn(0x55f182f89780 0x55f182f89b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:20.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.199+0000 7f824a7fc640 10 client.7083 ms_handle_connect on v2:172.21.15.67:6832/2392802741 2024-03-24T00:43:20.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.199+0000 7f826affd640 1 --2- 172.21.15.67:0/2974078692 >> [v2:172.21.15.67:6832/2392802741,v1:172.21.15.67:6833/2392802741] conn(0x55f182f89780 0x55f182f89b60 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8cbb1e136e7d0809 server_cookie=525dbabb45a747c9 in_seq=0 out_seq=0 2024-03-24T00:43:20.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.200+0000 7f824a7fc640 1 -- 172.21.15.67:0/2974078692 <== mds.0 v2:172.21.15.67:6832/2392802741 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f182edd160 con 0x55f182f89780 2024-03-24T00:43:20.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.200+0000 7f824a7fc640 10 client.7083 handle_command_reply: tid=0 2024-03-24T00:43:20.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.200+0000 7f8274a75740 1 client.7083 shutdown 2024-03-24T00:43:20.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.200+0000 7f8274a75740 1 -- 172.21.15.67:0/2974078692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f182f6bc40 msgr2=0x55f182f8ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:20.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.200+0000 7f8274a75740 1 --2- 172.21.15.67:0/2974078692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f182f6bc40 0x55f182f8ef70 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f825c001120 tx=0x7f825c05c310 comp rx=0 tx=0).stop 2024-03-24T00:43:20.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.200+0000 7f8274a75740 1 -- 172.21.15.67:0/2974078692 shutdown_connections 2024-03-24T00:43:20.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.200+0000 7f8274a75740 1 --2- 172.21.15.67:0/2974078692 >> [v2:172.21.15.67:6832/2392802741,v1:172.21.15.67:6833/2392802741] conn(0x55f182f89780 0x55f182f89b60 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.200+0000 7f8274a75740 1 --2- 172.21.15.67:0/2974078692 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f182f90cc0 0x55f182f8b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.200+0000 7f8274a75740 1 --2- 172.21.15.67:0/2974078692 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f182f6c020 0x55f182f90780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.200+0000 7f824a7fc640 0 client.7083 ms_handle_reset on v2:172.21.15.67:6832/2392802741 2024-03-24T00:43:20.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.200+0000 7f8274a75740 1 --2- 172.21.15.67:0/2974078692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f182f6bc40 0x55f182f8ef70 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.200+0000 7f8274a75740 1 -- 172.21.15.67:0/2974078692 >> 172.21.15.67:0/2974078692 conn(0x55f182f32a60 msgr2=0x55f182f93940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:20.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.200+0000 7f8274a75740 1 -- 172.21.15.67:0/2974078692 shutdown_connections 2024-03-24T00:43:20.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.200+0000 7f8274a75740 1 -- 172.21.15.67:0/2974078692 wait complete. 2024-03-24T00:43:20.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.200+0000 7f8274a75740 20 client.7083 trim_cache size 0 max 16384 2024-03-24T00:43:20.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.233+0000 7f8274a75740 1 Processor -- start 2024-03-24T00:43:20.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.233+0000 7f8274a75740 1 -- start start 2024-03-24T00:43:20.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.235+0000 7f8274a75740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f182f6bc40 0x55f183016c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:20.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.235+0000 7f8274a75740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f182f6c020 0x55f182fa62f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:20.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.235+0000 7f8274a75740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f182f90cc0 0x55f182fa6830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:20.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.235+0000 7f8274a75740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f826c06d960 con 0x55f182f6bc40 2024-03-24T00:43:20.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.235+0000 7f8274a75740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f826c15f110 con 0x55f182f90cc0 2024-03-24T00:43:20.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.235+0000 7f8274a75740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f826c06d360 con 0x55f182f6c020 2024-03-24T00:43:20.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.235+0000 7f826b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f182f6bc40 0x55f183016c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:20.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.235+0000 7f826b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f182f6bc40 0x55f183016c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50496/0 (socket says 172.21.15.67:50496) 2024-03-24T00:43:20.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.235+0000 7f826b7fe640 1 -- 172.21.15.67:0/548822372 learned_addr learned my addr 172.21.15.67:0/548822372 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:20.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.235+0000 7f826bfff640 1 --2- 172.21.15.67:0/548822372 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f182f90cc0 0x55f182fa6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:20.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.235+0000 7f826affd640 1 --2- 172.21.15.67:0/548822372 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f182f6c020 0x55f182fa62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:20.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.235+0000 7f826b7fe640 1 -- 172.21.15.67:0/548822372 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f182f6c020 msgr2=0x55f182fa62f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:20.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.235+0000 7f826b7fe640 1 --2- 172.21.15.67:0/548822372 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f182f6c020 0x55f182fa62f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.235+0000 7f826b7fe640 1 -- 172.21.15.67:0/548822372 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f182f90cc0 msgr2=0x55f182fa6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:20.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.235+0000 7f826b7fe640 1 --2- 172.21.15.67:0/548822372 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f182f90cc0 0x55f182fa6830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.235+0000 7f826b7fe640 1 -- 172.21.15.67:0/548822372 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f182cae6b0 con 0x55f182f6bc40 2024-03-24T00:43:20.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.235+0000 7f826b7fe640 1 --2- 172.21.15.67:0/548822372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f182f6bc40 0x55f183016c90 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f825c057ec0 tx=0x7f825c05c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2b82789857a2278 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:20.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.235+0000 7f8250ff9640 1 -- 172.21.15.67:0/548822372 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f825c004bc0 con 0x55f182f6bc40 2024-03-24T00:43:20.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.235+0000 7f8250ff9640 1 -- 172.21.15.67:0/548822372 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f825c004d60 con 0x55f182f6bc40 2024-03-24T00:43:20.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.235+0000 7f8250ff9640 1 -- 172.21.15.67:0/548822372 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f825c054110 con 0x55f182f6bc40 2024-03-24T00:43:20.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.236+0000 7f8274a75740 1 -- 172.21.15.67:0/548822372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f182f6bc40 msgr2=0x55f183016c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:20.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.236+0000 7f8274a75740 1 --2- 172.21.15.67:0/548822372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f182f6bc40 0x55f183016c90 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f825c057ec0 tx=0x7f825c05c000 comp rx=0 tx=0).stop 2024-03-24T00:43:20.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.236+0000 7f8274a75740 1 -- 172.21.15.67:0/548822372 shutdown_connections 2024-03-24T00:43:20.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.236+0000 7f8274a75740 1 --2- 172.21.15.67:0/548822372 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f182f90cc0 0x55f182fa6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.236+0000 7f8274a75740 1 --2- 172.21.15.67:0/548822372 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f182f6c020 0x55f182fa62f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.236+0000 7f8274a75740 1 --2- 172.21.15.67:0/548822372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f182f6bc40 0x55f183016c90 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.236+0000 7f8274a75740 1 -- 172.21.15.67:0/548822372 >> 172.21.15.67:0/548822372 conn(0x55f182f32a60 msgr2=0x55f182fa7770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:20.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.236+0000 7f8274a75740 1 -- 172.21.15.67:0/548822372 shutdown_connections 2024-03-24T00:43:20.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.236+0000 7f8274a75740 1 -- 172.21.15.67:0/548822372 wait complete. 2024-03-24T00:43:20.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.237+0000 7f8274a75740 1 -- start start 2024-03-24T00:43:20.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.237+0000 7f8274a75740 1 Processor -- start 2024-03-24T00:43:20.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.237+0000 7f8274a75740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f182f6bc40 0x55f182f906f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:20.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.237+0000 7f8274a75740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f182f6c020 0x55f182fa6750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:20.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.237+0000 7f8274a75740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f182f90cc0 0x55f183002630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:20.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.237+0000 7f8274a75740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f182f8bca0 con 0x55f182f6bc40 2024-03-24T00:43:20.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.237+0000 7f8274a75740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f182f8e550 con 0x55f182f6c020 2024-03-24T00:43:20.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.237+0000 7f8274a75740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f182fc4170 con 0x55f182f90cc0 2024-03-24T00:43:20.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.237+0000 7f8274a75740 10 client.0 _ll_register_callbacks cb 0x7f825c057f90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:43:20.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.237+0000 7f826b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f182f6bc40 0x55f182f906f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:20.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.238+0000 7f826b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f182f6bc40 0x55f182f906f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50506/0 (socket says 172.21.15.67:50506) 2024-03-24T00:43:20.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.238+0000 7f826b7fe640 1 -- 172.21.15.67:0/3006548397 learned_addr learned my addr 172.21.15.67:0/3006548397 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:20.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.238+0000 7f826bfff640 1 --2- 172.21.15.67:0/3006548397 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f182f90cc0 0x55f183002630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:20.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.238+0000 7f826affd640 1 --2- 172.21.15.67:0/3006548397 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f182f6c020 0x55f182fa6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:20.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.238+0000 7f826b7fe640 1 -- 172.21.15.67:0/3006548397 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f182f90cc0 msgr2=0x55f183002630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:20.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.238+0000 7f826b7fe640 1 --2- 172.21.15.67:0/3006548397 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f182f90cc0 0x55f183002630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.238+0000 7f826b7fe640 1 -- 172.21.15.67:0/3006548397 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f182f6c020 msgr2=0x55f182fa6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:20.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.238+0000 7f826b7fe640 1 --2- 172.21.15.67:0/3006548397 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f182f6c020 0x55f182fa6750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.238+0000 7f826b7fe640 1 -- 172.21.15.67:0/3006548397 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f182f89db0 con 0x55f182f6bc40 2024-03-24T00:43:20.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.238+0000 7f826affd640 1 --2- 172.21.15.67:0/3006548397 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f182f6c020 0x55f182fa6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:43:20.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.238+0000 7f826bfff640 1 --2- 172.21.15.67:0/3006548397 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f182f90cc0 0x55f183002630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:43:20.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.238+0000 7f826b7fe640 1 --2- 172.21.15.67:0/3006548397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f182f6bc40 0x55f182f906f0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f825c057ec0 tx=0x7f825c0881d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f87f66a30682978 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:20.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.238+0000 7f824a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:43:20.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.238+0000 7f824a7fc640 1 -- 172.21.15.67:0/3006548397 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f825c004bc0 con 0x55f182f6bc40 2024-03-24T00:43:20.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.238+0000 7f824a7fc640 1 -- 172.21.15.67:0/3006548397 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(76..76 src has 1..76) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f825c057080 con 0x55f182f6bc40 2024-03-24T00:43:20.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.238+0000 7f8274a75740 1 -- 172.21.15.67:0/3006548397 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f825c004d60 con 0x55f182f6bc40 2024-03-24T00:43:20.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.238+0000 7f824a7fc640 1 -- 172.21.15.67:0/3006548397 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f825c065050 con 0x55f182f6bc40 2024-03-24T00:43:20.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.238+0000 7f8274a75740 10 client.7089 fetch_fsmap learned FSMap version 40 2024-03-24T00:43:20.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.238+0000 7f8274a75740 1 -- 172.21.15.67:0/3006548397 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=2+}) v3 -- 0x55f182f89db0 con 0x55f182f6bc40 2024-03-24T00:43:20.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.238+0000 7f824a7fc640 1 -- 172.21.15.67:0/3006548397 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f825c004d60 con 0x55f182f6bc40 2024-03-24T00:43:20.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.239+0000 7f824a7fc640 1 -- 172.21.15.67:0/3006548397 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 40) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f825c057280 con 0x55f182f6bc40 2024-03-24T00:43:20.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.239+0000 7f8274a75740 10 client.7089 fetch_fsmap finished waiting for FSMap version 40 2024-03-24T00:43:20.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.239+0000 7f8274a75740 10 client.7089 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T00:43:20.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.239+0000 7f8274a75740 20 client.7089 populate_metadata read hostname 'smithi067' 2024-03-24T00:43:20.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.239+0000 7f8274a75740 1 --2- 172.21.15.67:0/3006548397 >> [v2:172.21.15.67:6832/2392802741,v1:172.21.15.67:6833/2392802741] conn(0x55f183002d00 0x55f182fcb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:20.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.239+0000 7f8274a75740 4 client.7089 mds_command: new command op to 6894 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:43:20.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.239+0000 7f8274a75740 1 -- 172.21.15.67:0/3006548397 --> [v2:172.21.15.67:6832/2392802741,v1:172.21.15.67:6833/2392802741] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f182ed74b0 con 0x55f183002d00 2024-03-24T00:43:20.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.239+0000 7f826affd640 1 --2- 172.21.15.67:0/3006548397 >> [v2:172.21.15.67:6832/2392802741,v1:172.21.15.67:6833/2392802741] conn(0x55f183002d00 0x55f182fcb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:20.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.239+0000 7f824a7fc640 10 client.7089 ms_handle_connect on v2:172.21.15.67:6832/2392802741 2024-03-24T00:43:20.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.239+0000 7f826affd640 1 --2- 172.21.15.67:0/3006548397 >> [v2:172.21.15.67:6832/2392802741,v1:172.21.15.67:6833/2392802741] conn(0x55f183002d00 0x55f182fcb2d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9b3d9f7368c044 server_cookie=169d99e9af51947e in_seq=0 out_seq=0 2024-03-24T00:43:20.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.239+0000 7f824a7fc640 1 -- 172.21.15.67:0/3006548397 <== mds.0 v2:172.21.15.67:6832/2392802741 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55f182ed74b0 con 0x55f183002d00 2024-03-24T00:43:20.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.239+0000 7f824a7fc640 10 client.7089 handle_command_reply: tid=0 2024-03-24T00:43:20.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.239+0000 7f8274a75740 1 client.7089 shutdown 2024-03-24T00:43:20.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.239+0000 7f8274a75740 1 -- 172.21.15.67:0/3006548397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f182f6bc40 msgr2=0x55f182f906f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:20.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.239+0000 7f8274a75740 1 --2- 172.21.15.67:0/3006548397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f182f6bc40 0x55f182f906f0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f825c057ec0 tx=0x7f825c0881d0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.239+0000 7f8274a75740 1 -- 172.21.15.67:0/3006548397 shutdown_connections 2024-03-24T00:43:20.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.239+0000 7f8274a75740 1 --2- 172.21.15.67:0/3006548397 >> [v2:172.21.15.67:6832/2392802741,v1:172.21.15.67:6833/2392802741] conn(0x55f183002d00 0x55f182fcb2d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.239+0000 7f8274a75740 1 --2- 172.21.15.67:0/3006548397 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f182f90cc0 0x55f183002630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.239+0000 7f8274a75740 1 --2- 172.21.15.67:0/3006548397 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f182f6c020 0x55f182fa6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.239+0000 7f824a7fc640 0 client.7089 ms_handle_reset on v2:172.21.15.67:6832/2392802741 2024-03-24T00:43:20.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.239+0000 7f8274a75740 1 --2- 172.21.15.67:0/3006548397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f182f6bc40 0x55f182f906f0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.239+0000 7f8274a75740 1 -- 172.21.15.67:0/3006548397 >> 172.21.15.67:0/3006548397 conn(0x55f182f32a60 msgr2=0x55f182f914a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:20.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.239+0000 7f8274a75740 1 -- 172.21.15.67:0/3006548397 shutdown_connections 2024-03-24T00:43:20.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.239+0000 7f8274a75740 1 -- 172.21.15.67:0/3006548397 wait complete. 2024-03-24T00:43:20.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.240+0000 7f8274a75740 20 client.7089 trim_cache size 0 max 16384 2024-03-24T00:43:20.245 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:43:20.246 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "de1d0178-808e-49ca-a986-e297e523b976", 2024-03-24T00:43:20.246 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-03-24T00:43:20.246 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 6894, 2024-03-24T00:43:20.246 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-03-24T00:43:20.246 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-03-24T00:43:20.246 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-03-24T00:43:20.246 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 9.4288153369999996, 2024-03-24T00:43:20.246 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 40, 2024-03-24T00:43:20.246 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 76, 2024-03-24T00:43:20.246 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 76, 2024-03-24T00:43:20.246 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 14.762503021000001 2024-03-24T00:43:20.246 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:43:20.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.240+0000 7f8271e85640 1 -- 172.21.15.67:0/3787459908 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f825806bbd0 msgr2=0x7f825806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:20.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.240+0000 7f8271e85640 1 --2- 172.21.15.67:0/3787459908 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f825806bbd0 0x7f825806e090 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f8260006fd0 tx=0x7f8260008040 comp rx=0 tx=0).stop 2024-03-24T00:43:20.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.240+0000 7f8271e85640 1 -- 172.21.15.67:0/3787459908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f826c0ce9c0 msgr2=0x7f826c15c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:20.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.240+0000 7f8271e85640 1 --2- 172.21.15.67:0/3787459908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f826c0ce9c0 0x7f826c15c500 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f825c00c4d0 tx=0x7f825c00c990 comp rx=0 tx=0).stop 2024-03-24T00:43:20.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.240+0000 7f8271e85640 1 -- 172.21.15.67:0/3787459908 shutdown_connections 2024-03-24T00:43:20.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.240+0000 7f8271e85640 1 --2- 172.21.15.67:0/3787459908 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f825806bbd0 0x7f825806e090 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.240+0000 7f8271e85640 1 --2- 172.21.15.67:0/3787459908 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f826c15bfa0 0x7f826c0cab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.240+0000 7f8271e85640 1 --2- 172.21.15.67:0/3787459908 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f826c0d25f0 0x7f826c15ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.240+0000 7f8271e85640 1 --2- 172.21.15.67:0/3787459908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f826c0ce9c0 0x7f826c15c500 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.240+0000 7f8271e85640 1 -- 172.21.15.67:0/3787459908 >> 172.21.15.67:0/3787459908 conn(0x7f826c0d1ad0 msgr2=0x7f826c0b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:20.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.240+0000 7f8271e85640 1 -- 172.21.15.67:0/3787459908 shutdown_connections 2024-03-24T00:43:20.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.240+0000 7f8271e85640 1 -- 172.21.15.67:0/3787459908 wait complete. 2024-03-24T00:43:20.250 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T00:43:20.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.352+0000 7f9d26e61640 1 Processor -- start 2024-03-24T00:43:20.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.353+0000 7f9d26e61640 1 -- start start 2024-03-24T00:43:20.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.353+0000 7f9d26e61640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f9d200cfc70 0x7f9d200ce220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:20.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.353+0000 7f9d26e61640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9d200ce760 0x7f9d200b1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:20.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.353+0000 7f9d26e61640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d200ccfe0 con 0x7f9d2015bf70 2024-03-24T00:43:20.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.353+0000 7f9d26e61640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d200ccda0 con 0x7f9d200cfc70 2024-03-24T00:43:20.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.353+0000 7f9d26e61640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d200cd2c0 con 0x7f9d200ce760 2024-03-24T00:43:20.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.353+0000 7f9d24bfd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9d2015bf70 0x7f9d200cc690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37554/0 (socket says 172.21.15.67:37554) 2024-03-24T00:43:20.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.353+0000 7f9d24bfd640 1 -- 172.21.15.67:0/1973614827 learned_addr learned my addr 172.21.15.67:0/1973614827 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:43:20.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.354+0000 7f9d253fe640 1 --2- 172.21.15.67:0/1973614827 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9d200ce760 0x7f9d200b1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:20.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.354+0000 7f9d17fff640 1 --2- 172.21.15.67:0/1973614827 >> v2:172.21.15.72:3300/0 conn(0x7f9d200cfc70 0x7f9d200ce220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:20.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.354+0000 7f9d177fe640 1 -- 172.21.15.67:0/1973614827 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4072472197 0 0) 0x7f9d200ccfe0 con 0x7f9d2015bf70 2024-03-24T00:43:20.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.354+0000 7f9d177fe640 1 -- 172.21.15.67:0/1973614827 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d08003520 con 0x7f9d2015bf70 2024-03-24T00:43:20.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.354+0000 7f9d253fe640 1 -- 172.21.15.67:0/1973614827 >> v2:172.21.15.72:3300/0 conn(0x7f9d200cfc70 msgr2=0x7f9d200ce220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:20.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.354+0000 7f9d253fe640 1 --2- 172.21.15.67:0/1973614827 >> v2:172.21.15.72:3300/0 conn(0x7f9d200cfc70 0x7f9d200ce220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.354+0000 7f9d253fe640 1 -- 172.21.15.67:0/1973614827 >> v1:172.21.15.67:6789/0 conn(0x7f9d2015bf70 legacy=0x7f9d200cc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:20.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.354+0000 7f9d177fe640 1 -- 172.21.15.67:0/1973614827 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3942107697 0 0) 0x7f9d08003520 con 0x7f9d2015bf70 2024-03-24T00:43:20.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.354+0000 7f9d253fe640 1 -- 172.21.15.67:0/1973614827 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d200cd5a0 con 0x7f9d200ce760 2024-03-24T00:43:20.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.354+0000 7f9d17fff640 1 --2- 172.21.15.67:0/1973614827 >> v2:172.21.15.72:3300/0 conn(0x7f9d200cfc70 0x7f9d200ce220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:43:20.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.355+0000 7f9d253fe640 1 --2- 172.21.15.67:0/1973614827 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9d200ce760 0x7f9d200b1cd0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9d18009d90 tx=0x7f9d1802bd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f14bee2f9ada7bd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:20.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.355+0000 7f9d177fe640 1 -- 172.21.15.67:0/1973614827 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d18039070 con 0x7f9d200ce760 2024-03-24T00:43:20.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.355+0000 7f9d177fe640 1 -- 172.21.15.67:0/1973614827 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d1802c8a0 con 0x7f9d200ce760 2024-03-24T00:43:20.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.355+0000 7f9d177fe640 1 -- 172.21.15.67:0/1973614827 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d1c001670 con 0x7f9d200ce760 2024-03-24T00:43:20.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.356+0000 7f9d26e61640 1 -- 172.21.15.67:0/1973614827 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9d200ce760 msgr2=0x7f9d200b1cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:20.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.356+0000 7f9d26e61640 1 --2- 172.21.15.67:0/1973614827 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9d200ce760 0x7f9d200b1cd0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9d18009d90 tx=0x7f9d1802bd70 comp rx=0 tx=0).stop 2024-03-24T00:43:20.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.356+0000 7f9d26e61640 1 -- 172.21.15.67:0/1973614827 shutdown_connections 2024-03-24T00:43:20.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.356+0000 7f9d26e61640 1 --2- 172.21.15.67:0/1973614827 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9d200ce760 0x7f9d200b1cd0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.356+0000 7f9d26e61640 1 --2- 172.21.15.67:0/1973614827 >> v2:172.21.15.72:3300/0 conn(0x7f9d200cfc70 0x7f9d200ce220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.356+0000 7f9d26e61640 1 -- 172.21.15.67:0/1973614827 >> 172.21.15.67:0/1973614827 conn(0x7f9d200d1bf0 msgr2=0x7f9d200d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:20.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.356+0000 7f9d26e61640 1 -- 172.21.15.67:0/1973614827 shutdown_connections 2024-03-24T00:43:20.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.356+0000 7f9d26e61640 1 -- 172.21.15.67:0/1973614827 wait complete. 2024-03-24T00:43:20.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.357+0000 7f9d26e61640 1 Processor -- start 2024-03-24T00:43:20.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.357+0000 7f9d26e61640 1 -- start start 2024-03-24T00:43:20.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.358+0000 7f9d26e61640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d200ce760 0x7f9d2015b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:20.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.358+0000 7f9d26e61640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d200cfc70 0x7f9d2015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:20.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.358+0000 7f9d26e61640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9d2015bf70 0x7f9d201491b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:20.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.358+0000 7f9d26e61640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d200b2210 con 0x7f9d200cfc70 2024-03-24T00:43:20.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.358+0000 7f9d26e61640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d2015c350 con 0x7f9d200ce760 2024-03-24T00:43:20.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.358+0000 7f9d26e61640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d200b2390 con 0x7f9d2015bf70 2024-03-24T00:43:20.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.358+0000 7f9d17fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d200cfc70 0x7f9d2015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:20.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.358+0000 7f9d17fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d200cfc70 0x7f9d2015d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50514/0 (socket says 172.21.15.67:50514) 2024-03-24T00:43:20.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.358+0000 7f9d17fff640 1 -- 172.21.15.67:0/2098980149 learned_addr learned my addr 172.21.15.67:0/2098980149 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:20.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.358+0000 7f9d24bfd640 1 --2- 172.21.15.67:0/2098980149 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d200ce760 0x7f9d2015b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:20.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.358+0000 7f9d253fe640 1 --2- 172.21.15.67:0/2098980149 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9d2015bf70 0x7f9d201491b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:20.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.359+0000 7f9d17fff640 1 -- 172.21.15.67:0/2098980149 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9d2015bf70 msgr2=0x7f9d201491b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:20.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.359+0000 7f9d17fff640 1 --2- 172.21.15.67:0/2098980149 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9d2015bf70 0x7f9d201491b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.359+0000 7f9d17fff640 1 -- 172.21.15.67:0/2098980149 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d200ce760 msgr2=0x7f9d2015b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:20.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.359+0000 7f9d17fff640 1 --2- 172.21.15.67:0/2098980149 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d200ce760 0x7f9d2015b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.359+0000 7f9d17fff640 1 -- 172.21.15.67:0/2098980149 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d2015b5b0 con 0x7f9d200cfc70 2024-03-24T00:43:20.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.359+0000 7f9d17fff640 1 --2- 172.21.15.67:0/2098980149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d200cfc70 0x7f9d2015d6f0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f9d1000ebb0 tx=0x7f9d1000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:20.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.359+0000 7f9d157fa640 1 -- 172.21.15.67:0/2098980149 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d10017070 con 0x7f9d200cfc70 2024-03-24T00:43:20.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.359+0000 7f9d157fa640 1 -- 172.21.15.67:0/2098980149 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d1000cc30 con 0x7f9d200cfc70 2024-03-24T00:43:20.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.359+0000 7f9d26e61640 1 -- 172.21.15.67:0/2098980149 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d2015c650 con 0x7f9d200cfc70 2024-03-24T00:43:20.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.359+0000 7f9d157fa640 1 -- 172.21.15.67:0/2098980149 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d10013470 con 0x7f9d200cfc70 2024-03-24T00:43:20.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.359+0000 7f9d26e61640 1 -- 172.21.15.67:0/2098980149 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d2015cc10 con 0x7f9d200cfc70 2024-03-24T00:43:20.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.360+0000 7f9d26e61640 1 -- 172.21.15.67:0/2098980149 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d200b27c0 con 0x7f9d200cfc70 2024-03-24T00:43:20.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.360+0000 7f9d157fa640 1 -- 172.21.15.67:0/2098980149 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9d1001a070 con 0x7f9d200cfc70 2024-03-24T00:43:20.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.361+0000 7f9d157fa640 1 --2- 172.21.15.67:0/2098980149 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9d0c06bbd0 0x7f9d0c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:20.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.361+0000 7f9d157fa640 1 -- 172.21.15.67:0/2098980149 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9d1008aa00 con 0x7f9d200cfc70 2024-03-24T00:43:20.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.361+0000 7f9d24bfd640 1 --2- 172.21.15.67:0/2098980149 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9d0c06bbd0 0x7f9d0c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:20.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.362+0000 7f9d24bfd640 1 --2- 172.21.15.67:0/2098980149 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9d0c06bbd0 0x7f9d0c06e090 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c0017e0 tx=0x7f9d1c001570 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:20.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.365+0000 7f9d157fa640 1 -- 172.21.15.67:0/2098980149 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9d200b27c0 con 0x7f9d200cfc70 2024-03-24T00:43:20.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.587+0000 7f9d26e61640 1 -- 172.21.15.67:0/2098980149 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f9d2004f400 con 0x7f9d200cfc70 2024-03-24T00:43:20.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.588+0000 7f9d157fa640 1 -- 172.21.15.67:0/2098980149 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v40) v1 ==== 53+0+73 (secure 0 0 0) 0x7f9d10058da0 con 0x7f9d200cfc70 2024-03-24T00:43:20.588 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-24T00:43:20.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.590+0000 7f9d26e61640 1 -- 172.21.15.67:0/2098980149 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9d0c06bbd0 msgr2=0x7f9d0c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:20.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.590+0000 7f9d26e61640 1 --2- 172.21.15.67:0/2098980149 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9d0c06bbd0 0x7f9d0c06e090 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c0017e0 tx=0x7f9d1c001570 comp rx=0 tx=0).stop 2024-03-24T00:43:20.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.590+0000 7f9d26e61640 1 -- 172.21.15.67:0/2098980149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d200cfc70 msgr2=0x7f9d2015d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:20.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.590+0000 7f9d26e61640 1 --2- 172.21.15.67:0/2098980149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d200cfc70 0x7f9d2015d6f0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f9d1000ebb0 tx=0x7f9d1000c520 comp rx=0 tx=0).stop 2024-03-24T00:43:20.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.590+0000 7f9d26e61640 1 -- 172.21.15.67:0/2098980149 shutdown_connections 2024-03-24T00:43:20.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.590+0000 7f9d26e61640 1 --2- 172.21.15.67:0/2098980149 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9d0c06bbd0 0x7f9d0c06e090 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.590+0000 7f9d26e61640 1 --2- 172.21.15.67:0/2098980149 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9d2015bf70 0x7f9d201491b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.590+0000 7f9d26e61640 1 --2- 172.21.15.67:0/2098980149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d200cfc70 0x7f9d2015d6f0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.590+0000 7f9d26e61640 1 --2- 172.21.15.67:0/2098980149 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d200ce760 0x7f9d2015b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:20.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.590+0000 7f9d26e61640 1 -- 172.21.15.67:0/2098980149 >> 172.21.15.67:0/2098980149 conn(0x7f9d200d1bf0 msgr2=0x7f9d20070680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:20.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.590+0000 7f9d26e61640 1 -- 172.21.15.67:0/2098980149 shutdown_connections 2024-03-24T00:43:20.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:20.590+0000 7f9d26e61640 1 -- 172.21.15.67:0/2098980149 wait complete. 2024-03-24T00:43:20.600 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-24T00:43:20.600 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T00:43:20.600 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-03-24T00:43:20.600 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-24T00:43:20.601 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T00:43:20.601 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-24T00:43:20.601 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-24T00:43:20.601 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T00:43:20.601 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:43:20.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:43:20.601 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-03-24T00:43:20.616 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T00:43:20.616 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T00:43:20.616 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T00:43:20.616 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:43:20.616 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-03-24T00:43:20.616 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:43:20.616 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:43:20.616 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-03-24T00:43:20.616 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-03-24T00:43:20.616 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:43:20.616 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-03-24T00:43:20.617 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-03-24T00:43:20.617 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:43:20.617 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-03-24T00:43:20.617 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T00:43:20.617 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-03-24T00:43:20.617 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-24T00:43:20.617 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 42312sec preferred_lft 42312sec 2024-03-24T00:43:20.617 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-03-24T00:43:20.617 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:43:20.617 INFO:teuthology.orchestra.run.smithi067.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T00:43:20.617 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 96:cf:26:c8:bf:62 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:43:20.617 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-24T00:43:20.617 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:43:20.617 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::7c6a:5fff:feaa:d41/64 scope link 2024-03-24T00:43:20.618 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:43:20.618 INFO:teuthology.orchestra.run.smithi067.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T00:43:20.618 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ea:22:4b:d5:48:3d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:43:20.618 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::e822:4bff:fed5:483d/64 scope link 2024-03-24T00:43:20.618 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:43:20.618 INFO:teuthology.orchestra.run.smithi067.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T00:43:20.618 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 96:cf:26:c8:bf:62 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:43:20.618 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::94cf:26ff:fec8:bf62/64 scope link 2024-03-24T00:43:20.618 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:43:20.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:43:20.619 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T00:43:20.682 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-03-24T00:43:20.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:43:20.683 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-03-24T00:43:20.738 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-03-24T00:43:20.738 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T00:43:20.738 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-24T00:43:20.739 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-24T00:43:20.739 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T00:43:20.739 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:43:20.739 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-03-24T00:43:20.739 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-03-24T00:43:20.739 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-24T00:43:20.739 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-24T00:43:20.739 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-24T00:43:20.739 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-03-24T00:43:20.834 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:43:20.834 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-03-24T00:43:20.849 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-24T00:43:20.849 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-24T00:43:20.849 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:43:20.849 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:43:20.849 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-03-24T00:43:20.916 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:43:20.916 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-24T00:43:20.971 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:43:20.971 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:43:20.971 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:43:21.026 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-24T00:43:21.026 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:43:21.026 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:43:21.112 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:43:21.112 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T00:43:21.112 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-03-24T00:43:21.120 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:43:21.120 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T00:43:21.120 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-03-24T00:43:21.120 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.72:3300,172.21.15.72:3301" 2024-03-24T00:43:21.120 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-24T00:43:21.125 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:43:21.125 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T00:43:21.125 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-03-24T00:43:21.125 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.72:3300,172.21.15.72:3301" 2024-03-24T00:43:21.125 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-24T00:43:21.125 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:6789,172.21.15.72:6789,172.21.15.72:6790" 2024-03-24T00:43:21.126 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.67:6789,172.21.15.72:6789,172.21.15.72:6790:/ 2024-03-24T00:43:21.126 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=de1d0178-808e-49ca-a986-e297e523b976" will pass to kernel 2024-03-24T00:43:21.126 INFO:teuthology.orchestra.run.smithi067.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-24T00:43:21.126 INFO:teuthology.orchestra.run.smithi067.stdout: You just mounted a file system that supports labels which does not 2024-03-24T00:43:21.126 INFO:teuthology.orchestra.run.smithi067.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-24T00:43:21.126 INFO:teuthology.orchestra.run.smithi067.stdout: applications will generate AVC messages and not be allowed access to 2024-03-24T00:43:21.126 INFO:teuthology.orchestra.run.smithi067.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-24T00:43:21.128 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-24T00:43:21.128 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:43:21.128 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:43:21.154 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:43:21.154 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T00:43:21.214 INFO:teuthology.orchestra.run.smithi067.stdout:client7098 2024-03-24T00:43:21.214 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:43:21.214 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:43:21.269 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:43:21.269 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir subdir/) 2024-03-24T00:43:21.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:43:21.328 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch subdir/file_undamaged) 2024-03-24T00:43:21.384 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:43:21.384 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch subdir/file_to_be_damaged) 2024-03-24T00:43:21.441 DEBUG:teuthology.orchestra.run.smithi067:> stdin-killer -- python3 -c ' 2024-03-24T00:43:21.441 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-03-24T00:43:21.441 DEBUG:teuthology.orchestra.run.smithi067:> import stat 2024-03-24T00:43:21.441 DEBUG:teuthology.orchestra.run.smithi067:> 2024-03-24T00:43:21.441 DEBUG:teuthology.orchestra.run.smithi067:> print(os.stat("/home/ubuntu/cephtest/mnt.0/subdir").st_ino) 2024-03-24T00:43:21.441 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-03-24T00:43:21.547 INFO:teuthology.orchestra.run.smithi067.stdout:1099511627776 2024-03-24T00:43:21.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:43:21.556 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:43:21.556 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:43:21.612 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:43:21.612 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:43:21.668 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:43:21.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.812+0000 7fb839c14640 1 Processor -- start 2024-03-24T00:43:21.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.812+0000 7fb839c14640 1 -- start start 2024-03-24T00:43:21.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.813+0000 7fb839c14640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb83415af80 0x7fb83406c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:21.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.813+0000 7fb839c14640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb8340b3ae0 0x7fb8340af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:21.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.813+0000 7fb839c14640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb834067150 0x7fb8340afe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:21.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.813+0000 7fb839c14640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8340b4250 con 0x7fb83415af80 2024-03-24T00:43:21.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.813+0000 7fb839c14640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8340b3f70 con 0x7fb8340b3ae0 2024-03-24T00:43:21.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.813+0000 7fb839c14640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb83406baf0 con 0x7fb834067150 2024-03-24T00:43:21.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.813+0000 7fb8337fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb83415af80 0x7fb83406c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:21.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.813+0000 7fb8337fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb83415af80 0x7fb83406c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50516/0 (socket says 172.21.15.67:50516) 2024-03-24T00:43:21.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.813+0000 7fb8337fe640 1 -- 172.21.15.67:0/631358541 learned_addr learned my addr 172.21.15.67:0/631358541 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:21.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.813+0000 7fb832ffd640 1 --2- 172.21.15.67:0/631358541 >> v2:172.21.15.72:3300/0 conn(0x7fb8340b3ae0 0x7fb8340af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:21.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.813+0000 7fb833fff640 1 --2- 172.21.15.67:0/631358541 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb834067150 0x7fb8340afe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:21.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.814+0000 7fb8337fe640 1 -- 172.21.15.67:0/631358541 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb834067150 msgr2=0x7fb8340afe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:21.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.814+0000 7fb8337fe640 1 --2- 172.21.15.67:0/631358541 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb834067150 0x7fb8340afe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:21.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.814+0000 7fb8337fe640 1 -- 172.21.15.67:0/631358541 >> v2:172.21.15.72:3300/0 conn(0x7fb8340b3ae0 msgr2=0x7fb8340af8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:21.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.814+0000 7fb8337fe640 1 --2- 172.21.15.67:0/631358541 >> v2:172.21.15.72:3300/0 conn(0x7fb8340b3ae0 0x7fb8340af8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:21.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.814+0000 7fb8337fe640 1 -- 172.21.15.67:0/631358541 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb83406bdd0 con 0x7fb83415af80 2024-03-24T00:43:21.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.814+0000 7fb8337fe640 1 --2- 172.21.15.67:0/631358541 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb83415af80 0x7fb83406c5d0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fb834073170 tx=0x7fb82402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e783e44769f452c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:21.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.815+0000 7fb8327fc640 1 -- 172.21.15.67:0/631358541 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb824035020 con 0x7fb83415af80 2024-03-24T00:43:21.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.815+0000 7fb8327fc640 1 -- 172.21.15.67:0/631358541 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb824031870 con 0x7fb83415af80 2024-03-24T00:43:21.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.815+0000 7fb8327fc640 1 -- 172.21.15.67:0/631358541 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb824031c40 con 0x7fb83415af80 2024-03-24T00:43:21.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.815+0000 7fb839c14640 1 -- 172.21.15.67:0/631358541 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb83415af80 msgr2=0x7fb83406c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:21.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.815+0000 7fb839c14640 1 --2- 172.21.15.67:0/631358541 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb83415af80 0x7fb83406c5d0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fb834073170 tx=0x7fb82402b910 comp rx=0 tx=0).stop 2024-03-24T00:43:21.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.815+0000 7fb839c14640 1 -- 172.21.15.67:0/631358541 shutdown_connections 2024-03-24T00:43:21.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.815+0000 7fb839c14640 1 --2- 172.21.15.67:0/631358541 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb834067150 0x7fb8340afe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:21.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.815+0000 7fb839c14640 1 --2- 172.21.15.67:0/631358541 >> v2:172.21.15.72:3300/0 conn(0x7fb8340b3ae0 0x7fb8340af8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:21.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.815+0000 7fb839c14640 1 --2- 172.21.15.67:0/631358541 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb83415af80 0x7fb83406c5d0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:21.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.815+0000 7fb839c14640 1 -- 172.21.15.67:0/631358541 >> 172.21.15.67:0/631358541 conn(0x7fb8340b5440 msgr2=0x7fb8340b33d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:21.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.815+0000 7fb839c14640 1 -- 172.21.15.67:0/631358541 shutdown_connections 2024-03-24T00:43:21.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.815+0000 7fb839c14640 1 -- 172.21.15.67:0/631358541 wait complete. 2024-03-24T00:43:21.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.817+0000 7fb839c14640 1 Processor -- start 2024-03-24T00:43:21.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.817+0000 7fb839c14640 1 -- start start 2024-03-24T00:43:21.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.818+0000 7fb839c14640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb834067150 0x7fb834149840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:21.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.818+0000 7fb839c14640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb83415af80 0x7fb834154e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:21.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.818+0000 7fb839c14640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb834157c40 0x7fb834159250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:21.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.819+0000 7fb839c14640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb83415b360 con 0x7fb83415af80 2024-03-24T00:43:21.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.819+0000 7fb839c14640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb83406cb10 con 0x7fb834157c40 2024-03-24T00:43:21.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.819+0000 7fb839c14640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb83415b4e0 con 0x7fb834067150 2024-03-24T00:43:21.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.819+0000 7fb832ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb83415af80 0x7fb834154e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:21.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.819+0000 7fb832ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb83415af80 0x7fb834154e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50528/0 (socket says 172.21.15.67:50528) 2024-03-24T00:43:21.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.819+0000 7fb832ffd640 1 -- 172.21.15.67:0/2704238474 learned_addr learned my addr 172.21.15.67:0/2704238474 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:21.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.819+0000 7fb833fff640 1 --2- 172.21.15.67:0/2704238474 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb834157c40 0x7fb834159250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:21.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.819+0000 7fb8337fe640 1 --2- 172.21.15.67:0/2704238474 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb834067150 0x7fb834149840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:21.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.819+0000 7fb832ffd640 1 -- 172.21.15.67:0/2704238474 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb834067150 msgr2=0x7fb834149840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:21.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.819+0000 7fb832ffd640 1 --2- 172.21.15.67:0/2704238474 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb834067150 0x7fb834149840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:21.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.819+0000 7fb832ffd640 1 -- 172.21.15.67:0/2704238474 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb834157c40 msgr2=0x7fb834159250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:21.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.819+0000 7fb832ffd640 1 --2- 172.21.15.67:0/2704238474 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb834157c40 0x7fb834159250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:21.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.819+0000 7fb832ffd640 1 -- 172.21.15.67:0/2704238474 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8341581b0 con 0x7fb83415af80 2024-03-24T00:43:21.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.819+0000 7fb833fff640 1 --2- 172.21.15.67:0/2704238474 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb834157c40 0x7fb834159250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:43:21.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.819+0000 7fb832ffd640 1 --2- 172.21.15.67:0/2704238474 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb83415af80 0x7fb834154e00 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fb81c00e8a0 tx=0x7fb81c00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:21.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.819+0000 7fb82bfff640 1 -- 172.21.15.67:0/2704238474 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb81c017070 con 0x7fb83415af80 2024-03-24T00:43:21.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.820+0000 7fb82bfff640 1 -- 172.21.15.67:0/2704238474 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb81c00c940 con 0x7fb83415af80 2024-03-24T00:43:21.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.820+0000 7fb839c14640 1 -- 172.21.15.67:0/2704238474 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb834158490 con 0x7fb83415af80 2024-03-24T00:43:21.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.820+0000 7fb82bfff640 1 -- 172.21.15.67:0/2704238474 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb81c00cd30 con 0x7fb83415af80 2024-03-24T00:43:21.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.820+0000 7fb839c14640 1 -- 172.21.15.67:0/2704238474 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb834158a50 con 0x7fb83415af80 2024-03-24T00:43:21.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.821+0000 7fb839c14640 1 -- 172.21.15.67:0/2704238474 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb834158f40 con 0x7fb83415af80 2024-03-24T00:43:21.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.821+0000 7fb82bfff640 1 -- 172.21.15.67:0/2704238474 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb81c01a070 con 0x7fb83415af80 2024-03-24T00:43:21.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.822+0000 7fb82bfff640 1 --2- 172.21.15.67:0/2704238474 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb81406bc00 0x7fb81406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:21.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.822+0000 7fb82bfff640 1 -- 172.21.15.67:0/2704238474 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb81c08b190 con 0x7fb83415af80 2024-03-24T00:43:21.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.822+0000 7fb8337fe640 1 --2- 172.21.15.67:0/2704238474 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb81406bc00 0x7fb81406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:21.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.823+0000 7fb8337fe640 1 --2- 172.21.15.67:0/2704238474 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb81406bc00 0x7fb81406e0c0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fb824036000 tx=0x7fb82403c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:21.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:21.826+0000 7fb82bfff640 1 -- 172.21.15.67:0/2704238474 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb81c059bd0 con 0x7fb83415af80 2024-03-24T00:43:22.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.082+0000 7fb839c14640 1 -- 172.21.15.67:0/2704238474 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb8340b5820 con 0x7fb83415af80 2024-03-24T00:43:22.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.083+0000 7fb82bfff640 1 -- 172.21.15.67:0/2704238474 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+16885 (secure 0 0 0) 0x7fb81c0594d0 con 0x7fb83415af80 2024-03-24T00:43:22.083 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:43:22.083 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":76,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:43:11.822689+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:43:08.452726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-24T00:43:09.146431+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:43:22.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.085+0000 7fb839c14640 1 -- 172.21.15.67:0/2704238474 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb81406bc00 msgr2=0x7fb81406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:22.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.085+0000 7fb839c14640 1 --2- 172.21.15.67:0/2704238474 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb81406bc00 0x7fb81406e0c0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fb824036000 tx=0x7fb82403c040 comp rx=0 tx=0).stop 2024-03-24T00:43:22.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.085+0000 7fb839c14640 1 -- 172.21.15.67:0/2704238474 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb83415af80 msgr2=0x7fb834154e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:22.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.085+0000 7fb839c14640 1 --2- 172.21.15.67:0/2704238474 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb83415af80 0x7fb834154e00 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fb81c00e8a0 tx=0x7fb81c00ed60 comp rx=0 tx=0).stop 2024-03-24T00:43:22.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.085+0000 7fb839c14640 1 -- 172.21.15.67:0/2704238474 shutdown_connections 2024-03-24T00:43:22.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.085+0000 7fb839c14640 1 --2- 172.21.15.67:0/2704238474 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb81406bc00 0x7fb81406e0c0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:22.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.085+0000 7fb839c14640 1 --2- 172.21.15.67:0/2704238474 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb834157c40 0x7fb834159250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:22.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.085+0000 7fb839c14640 1 --2- 172.21.15.67:0/2704238474 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb83415af80 0x7fb834154e00 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:22.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.085+0000 7fb839c14640 1 --2- 172.21.15.67:0/2704238474 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb834067150 0x7fb834149840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:22.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.085+0000 7fb839c14640 1 -- 172.21.15.67:0/2704238474 >> 172.21.15.67:0/2704238474 conn(0x7fb8340b5440 msgr2=0x7fb834156ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:22.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.085+0000 7fb839c14640 1 -- 172.21.15.67:0/2704238474 shutdown_connections 2024-03-24T00:43:22.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.085+0000 7fb839c14640 1 -- 172.21.15.67:0/2704238474 wait complete. 2024-03-24T00:43:22.096 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:38:39.364950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:43:08.452726+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:43:09.146431+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:43:22.096 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:43:22.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.203+0000 7f2398a43640 1 Processor -- start 2024-03-24T00:43:22.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.203+0000 7f2398a43640 1 -- start start 2024-03-24T00:43:22.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.204+0000 7f2398a43640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2394145540 0x7f2394160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:22.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.204+0000 7f2398a43640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f239415c2a0 0x7f23940cef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:22.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.204+0000 7f2398a43640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f239406dcf0 con 0x7f2394071af0 2024-03-24T00:43:22.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.204+0000 7f2398a43640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f239406e620 con 0x7f2394145540 2024-03-24T00:43:22.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.204+0000 7f2398a43640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f239406dfd0 con 0x7f239415c2a0 2024-03-24T00:43:22.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.205+0000 7f239259c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2394145540 0x7f2394160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:22.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.205+0000 7f239259c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2394145540 0x7f2394160420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:43960/0 (socket says 172.21.15.67:43960) 2024-03-24T00:43:22.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.205+0000 7f2392d9d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f2394071af0 0x7f23940ccbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:34496/0 (socket says 172.21.15.67:34496) 2024-03-24T00:43:22.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.205+0000 7f239259c640 1 -- 172.21.15.67:0/2814109071 learned_addr learned my addr 172.21.15.67:0/2814109071 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:22.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.205+0000 7f2391d9b640 1 --2- 172.21.15.67:0/2814109071 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f239415c2a0 0x7f23940cef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:22.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.205+0000 7f239259c640 1 -- 172.21.15.67:0/2814109071 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f239415c2a0 msgr2=0x7f23940cef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:22.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.205+0000 7f239259c640 1 --2- 172.21.15.67:0/2814109071 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f239415c2a0 0x7f23940cef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:22.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.205+0000 7f239259c640 1 -- 172.21.15.67:0/2814109071 >> v1:172.21.15.72:6789/0 conn(0x7f2394071af0 legacy=0x7f23940ccbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:22.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.205+0000 7f239259c640 1 -- 172.21.15.67:0/2814109071 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f239406e2b0 con 0x7f2394145540 2024-03-24T00:43:22.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.206+0000 7f239259c640 1 --2- 172.21.15.67:0/2814109071 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2394145540 0x7f2394160420 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f2384009870 tx=0x7f238402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e87295ad53ba8b93 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:22.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.206+0000 7f239159a640 1 -- 172.21.15.67:0/2814109071 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2384034070 con 0x7f2394145540 2024-03-24T00:43:22.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.206+0000 7f239159a640 1 -- 172.21.15.67:0/2814109071 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2384030840 con 0x7f2394145540 2024-03-24T00:43:22.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.206+0000 7f239159a640 1 -- 172.21.15.67:0/2814109071 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2384030c10 con 0x7f2394145540 2024-03-24T00:43:22.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.207+0000 7f2398a43640 1 -- 172.21.15.67:0/2814109071 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2394145540 msgr2=0x7f2394160420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:22.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.207+0000 7f2398a43640 1 --2- 172.21.15.67:0/2814109071 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2394145540 0x7f2394160420 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f2384009870 tx=0x7f238402b980 comp rx=0 tx=0).stop 2024-03-24T00:43:22.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.207+0000 7f2398a43640 1 -- 172.21.15.67:0/2814109071 shutdown_connections 2024-03-24T00:43:22.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.207+0000 7f2398a43640 1 --2- 172.21.15.67:0/2814109071 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f239415c2a0 0x7f23940cef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:22.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.207+0000 7f2398a43640 1 --2- 172.21.15.67:0/2814109071 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2394145540 0x7f2394160420 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:22.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.207+0000 7f2398a43640 1 -- 172.21.15.67:0/2814109071 >> 172.21.15.67:0/2814109071 conn(0x7f23940cf9e0 msgr2=0x7f23940cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:22.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.207+0000 7f2398a43640 1 -- 172.21.15.67:0/2814109071 shutdown_connections 2024-03-24T00:43:22.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.207+0000 7f2398a43640 1 -- 172.21.15.67:0/2814109071 wait complete. 2024-03-24T00:43:22.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.208+0000 7f2398a43640 1 Processor -- start 2024-03-24T00:43:22.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.208+0000 7f2398a43640 1 -- start start 2024-03-24T00:43:22.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.209+0000 7f2398a43640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2394071af0 0x7f239413aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:22.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.209+0000 7f2398a43640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2394145540 0x7f239415b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:22.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.209+0000 7f2398a43640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f239415c2a0 0x7f2394139370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:22.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.209+0000 7f2398a43640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23940cf600 con 0x7f239415c2a0 2024-03-24T00:43:22.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.209+0000 7f2398a43640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23940cf480 con 0x7f2394145540 2024-03-24T00:43:22.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.209+0000 7f2398a43640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2394071f60 con 0x7f2394071af0 2024-03-24T00:43:22.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.209+0000 7f2392d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f239415c2a0 0x7f2394139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:22.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.209+0000 7f2392d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f239415c2a0 0x7f2394139370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50530/0 (socket says 172.21.15.67:50530) 2024-03-24T00:43:22.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.209+0000 7f2392d9d640 1 -- 172.21.15.67:0/1613380217 learned_addr learned my addr 172.21.15.67:0/1613380217 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:22.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.209+0000 7f239259c640 1 --2- 172.21.15.67:0/1613380217 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2394071af0 0x7f239413aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:22.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.209+0000 7f2391d9b640 1 --2- 172.21.15.67:0/1613380217 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2394145540 0x7f239415b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:22.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.209+0000 7f2392d9d640 1 -- 172.21.15.67:0/1613380217 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2394071af0 msgr2=0x7f239413aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:22.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.209+0000 7f2392d9d640 1 --2- 172.21.15.67:0/1613380217 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2394071af0 0x7f239413aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:22.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.209+0000 7f2392d9d640 1 -- 172.21.15.67:0/1613380217 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2394145540 msgr2=0x7f239415b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:22.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.209+0000 7f2392d9d640 1 --2- 172.21.15.67:0/1613380217 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2394145540 0x7f239415b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:22.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.209+0000 7f2392d9d640 1 -- 172.21.15.67:0/1613380217 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2394137ff0 con 0x7f239415c2a0 2024-03-24T00:43:22.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.209+0000 7f2392d9d640 1 --2- 172.21.15.67:0/1613380217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f239415c2a0 0x7f2394139370 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f238c00c780 tx=0x7f238c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:22.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.210+0000 7f2382ffd640 1 -- 172.21.15.67:0/1613380217 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f238c017070 con 0x7f239415c2a0 2024-03-24T00:43:22.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.210+0000 7f2382ffd640 1 -- 172.21.15.67:0/1613380217 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f238c00a640 con 0x7f239415c2a0 2024-03-24T00:43:22.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.210+0000 7f2398a43640 1 -- 172.21.15.67:0/1613380217 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23941382d0 con 0x7f239415c2a0 2024-03-24T00:43:22.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.210+0000 7f2382ffd640 1 -- 172.21.15.67:0/1613380217 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f238c00a9f0 con 0x7f239415c2a0 2024-03-24T00:43:22.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.210+0000 7f2398a43640 1 -- 172.21.15.67:0/1613380217 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2394138890 con 0x7f239415c2a0 2024-03-24T00:43:22.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.211+0000 7f2398a43640 1 -- 172.21.15.67:0/1613380217 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f239404f400 con 0x7f239415c2a0 2024-03-24T00:43:22.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.211+0000 7f2382ffd640 1 -- 172.21.15.67:0/1613380217 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f238c01a070 con 0x7f239415c2a0 2024-03-24T00:43:22.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.211+0000 7f2382ffd640 1 --2- 172.21.15.67:0/1613380217 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f237406bbd0 0x7f237406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:22.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.212+0000 7f2382ffd640 1 -- 172.21.15.67:0/1613380217 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f238c08a190 con 0x7f239415c2a0 2024-03-24T00:43:22.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.212+0000 7f239259c640 1 --2- 172.21.15.67:0/1613380217 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f237406bbd0 0x7f237406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:22.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.212+0000 7f239259c640 1 --2- 172.21.15.67:0/1613380217 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f237406bbd0 0x7f237406e090 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f2384036000 tx=0x7f2384008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:22.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.220+0000 7f2382ffd640 1 -- 172.21.15.67:0/1613380217 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f238c0584b0 con 0x7f239415c2a0 2024-03-24T00:43:22.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.402+0000 7f2398a43640 1 -- 172.21.15.67:0/1613380217 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f23940d0740 con 0x7f239415c2a0 2024-03-24T00:43:22.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.402+0000 7f2382ffd640 1 -- 172.21.15.67:0/1613380217 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v76) v1 ==== 93+0+10 (secure 0 0 0) 0x7f238c05c550 con 0x7f239415c2a0 2024-03-24T00:43:22.402 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-24T00:43:22.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.404+0000 7f2398a43640 1 -- 172.21.15.67:0/1613380217 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f237406bbd0 msgr2=0x7f237406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:22.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.404+0000 7f2398a43640 1 --2- 172.21.15.67:0/1613380217 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f237406bbd0 0x7f237406e090 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f2384036000 tx=0x7f2384008030 comp rx=0 tx=0).stop 2024-03-24T00:43:22.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.404+0000 7f2398a43640 1 -- 172.21.15.67:0/1613380217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f239415c2a0 msgr2=0x7f2394139370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:22.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.404+0000 7f2398a43640 1 --2- 172.21.15.67:0/1613380217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f239415c2a0 0x7f2394139370 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f238c00c780 tx=0x7f238c00cc40 comp rx=0 tx=0).stop 2024-03-24T00:43:22.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.404+0000 7f2398a43640 1 -- 172.21.15.67:0/1613380217 shutdown_connections 2024-03-24T00:43:22.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.404+0000 7f2398a43640 1 --2- 172.21.15.67:0/1613380217 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f237406bbd0 0x7f237406e090 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:22.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.404+0000 7f2398a43640 1 --2- 172.21.15.67:0/1613380217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f239415c2a0 0x7f2394139370 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:22.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.404+0000 7f2398a43640 1 --2- 172.21.15.67:0/1613380217 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2394145540 0x7f239415b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:22.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.404+0000 7f2398a43640 1 --2- 172.21.15.67:0/1613380217 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2394071af0 0x7f239413aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:22.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.404+0000 7f2398a43640 1 -- 172.21.15.67:0/1613380217 >> 172.21.15.67:0/1613380217 conn(0x7f23940cf9e0 msgr2=0x7f23940af680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:22.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.404+0000 7f2398a43640 1 -- 172.21.15.67:0/1613380217 shutdown_connections 2024-03-24T00:43:22.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.405+0000 7f2398a43640 1 -- 172.21.15.67:0/1613380217 wait complete. 2024-03-24T00:43:22.415 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:43:22.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.521+0000 7fd1ae5cd640 1 Processor -- start 2024-03-24T00:43:22.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.521+0000 7fd1ae5cd640 1 -- start start 2024-03-24T00:43:22.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.522+0000 7fd1ae5cd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd1a80d3ae0 0x7fd1a80b5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:22.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.522+0000 7fd1ae5cd640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1a80d2250 con 0x7fd1a815dea0 2024-03-24T00:43:22.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.522+0000 7fd1ae5cd640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1a80d2530 con 0x7fd1a815e280 2024-03-24T00:43:22.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.522+0000 7fd1ae5cd640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1a80d2810 con 0x7fd1a80d3ae0 2024-03-24T00:43:22.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.522+0000 7fd1a7fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd1a815dea0 0x7fd1a80d33d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37570/0 (socket says 172.21.15.67:37570) 2024-03-24T00:43:22.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.522+0000 7fd1a7fff640 1 -- 172.21.15.67:0/4291049768 learned_addr learned my addr 172.21.15.67:0/4291049768 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:43:22.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.522+0000 7fd1a77fe640 1 --2- 172.21.15.67:0/4291049768 >> v2:172.21.15.72:3300/0 conn(0x7fd1a80d3ae0 0x7fd1a80b5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:22.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.522+0000 7fd1a6ffd640 1 -- 172.21.15.67:0/4291049768 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2771075475 0 0) 0x7fd1a80d2250 con 0x7fd1a815dea0 2024-03-24T00:43:22.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.522+0000 7fd1a6ffd640 1 -- 172.21.15.67:0/4291049768 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd190003520 con 0x7fd1a815dea0 2024-03-24T00:43:22.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.523+0000 7fd1a77fe640 1 -- 172.21.15.67:0/4291049768 >> v1:172.21.15.72:6789/0 conn(0x7fd1a815e280 legacy=0x7fd1a80b70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:22.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.523+0000 7fd1a77fe640 1 -- 172.21.15.67:0/4291049768 >> v1:172.21.15.67:6789/0 conn(0x7fd1a815dea0 legacy=0x7fd1a80d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:22.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.523+0000 7fd1a77fe640 1 -- 172.21.15.67:0/4291049768 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a80d2af0 con 0x7fd1a80d3ae0 2024-03-24T00:43:22.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.523+0000 7fd1a77fe640 1 --2- 172.21.15.67:0/4291049768 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1a80d3ae0 0x7fd1a80b5a60 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fd19c0097d0 tx=0x7fd19c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef3fc52c4a588be9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:22.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.524+0000 7fd1a6ffd640 1 -- 172.21.15.67:0/4291049768 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19c035020 con 0x7fd1a80d3ae0 2024-03-24T00:43:22.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.524+0000 7fd1a6ffd640 1 -- 172.21.15.67:0/4291049768 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd19c031830 con 0x7fd1a80d3ae0 2024-03-24T00:43:22.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.524+0000 7fd1a6ffd640 1 -- 172.21.15.67:0/4291049768 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19c031c00 con 0x7fd1a80d3ae0 2024-03-24T00:43:22.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.524+0000 7fd1ae5cd640 1 -- 172.21.15.67:0/4291049768 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1a80d3ae0 msgr2=0x7fd1a80b5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:22.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.524+0000 7fd1ae5cd640 1 --2- 172.21.15.67:0/4291049768 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1a80d3ae0 0x7fd1a80b5a60 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fd19c0097d0 tx=0x7fd19c02b8d0 comp rx=0 tx=0).stop 2024-03-24T00:43:22.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.524+0000 7fd1ae5cd640 1 -- 172.21.15.67:0/4291049768 shutdown_connections 2024-03-24T00:43:22.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.524+0000 7fd1ae5cd640 1 --2- 172.21.15.67:0/4291049768 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1a80d3ae0 0x7fd1a80b5a60 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:22.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.524+0000 7fd1ae5cd640 1 -- 172.21.15.67:0/4291049768 >> 172.21.15.67:0/4291049768 conn(0x7fd1a8072da0 msgr2=0x7fd1a80d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:22.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.524+0000 7fd1ae5cd640 1 -- 172.21.15.67:0/4291049768 shutdown_connections 2024-03-24T00:43:22.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.524+0000 7fd1ae5cd640 1 -- 172.21.15.67:0/4291049768 wait complete. 2024-03-24T00:43:22.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.525+0000 7fd1ae5cd640 1 Processor -- start 2024-03-24T00:43:22.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.525+0000 7fd1ae5cd640 1 -- start start 2024-03-24T00:43:22.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.526+0000 7fd1ae5cd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1a80d3ae0 0x7fd1a80cde60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:22.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.526+0000 7fd1ae5cd640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd1a815dea0 0x7fd1a815b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:22.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.526+0000 7fd1ae5cd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1a815e280 0x7fd1a80b4b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:22.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.526+0000 7fd1ae5cd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a80b6370 con 0x7fd1a815e280 2024-03-24T00:43:22.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.526+0000 7fd1ae5cd640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a80b61f0 con 0x7fd1a80d3ae0 2024-03-24T00:43:22.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.526+0000 7fd1ae5cd640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a80b7800 con 0x7fd1a815dea0 2024-03-24T00:43:22.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.526+0000 7fd1acb6a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1a815e280 0x7fd1a80b4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:22.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.526+0000 7fd1acb6a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1a815e280 0x7fd1a80b4b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50538/0 (socket says 172.21.15.67:50538) 2024-03-24T00:43:22.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.526+0000 7fd1acb6a640 1 -- 172.21.15.67:0/1035390540 learned_addr learned my addr 172.21.15.67:0/1035390540 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:22.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.526+0000 7fd1a7fff640 1 --2- 172.21.15.67:0/1035390540 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1a80d3ae0 0x7fd1a80cde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:22.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.526+0000 7fd1a77fe640 1 --2- 172.21.15.67:0/1035390540 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd1a815dea0 0x7fd1a815b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:22.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.527+0000 7fd1acb6a640 1 -- 172.21.15.67:0/1035390540 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd1a815dea0 msgr2=0x7fd1a815b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:22.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.527+0000 7fd1acb6a640 1 --2- 172.21.15.67:0/1035390540 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd1a815dea0 0x7fd1a815b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:22.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.527+0000 7fd1acb6a640 1 -- 172.21.15.67:0/1035390540 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1a80d3ae0 msgr2=0x7fd1a80cde60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:22.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.527+0000 7fd1acb6a640 1 --2- 172.21.15.67:0/1035390540 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1a80d3ae0 0x7fd1a80cde60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:22.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.527+0000 7fd1acb6a640 1 -- 172.21.15.67:0/1035390540 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a80cd0d0 con 0x7fd1a815e280 2024-03-24T00:43:22.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.527+0000 7fd1a77fe640 1 --2- 172.21.15.67:0/1035390540 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd1a815dea0 0x7fd1a815b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:43:22.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.527+0000 7fd1acb6a640 1 --2- 172.21.15.67:0/1035390540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1a815e280 0x7fd1a80b4b40 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fd1a000c730 tx=0x7fd1a000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:22.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.527+0000 7fd1a4ff9640 1 -- 172.21.15.67:0/1035390540 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a0003b70 con 0x7fd1a815e280 2024-03-24T00:43:22.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.527+0000 7fd1a4ff9640 1 -- 172.21.15.67:0/1035390540 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1a0003d10 con 0x7fd1a815e280 2024-03-24T00:43:22.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.527+0000 7fd1a4ff9640 1 -- 172.21.15.67:0/1035390540 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a0005310 con 0x7fd1a815e280 2024-03-24T00:43:22.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.527+0000 7fd1ae5cd640 1 -- 172.21.15.67:0/1035390540 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1a80cc310 con 0x7fd1a815e280 2024-03-24T00:43:22.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.527+0000 7fd1ae5cd640 1 -- 172.21.15.67:0/1035390540 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1a80cc8d0 con 0x7fd1a815e280 2024-03-24T00:43:22.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.528+0000 7fd1ae5cd640 1 -- 172.21.15.67:0/1035390540 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1a80cffb0 con 0x7fd1a815e280 2024-03-24T00:43:22.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.529+0000 7fd1a4ff9640 1 -- 172.21.15.67:0/1035390540 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd1a00054b0 con 0x7fd1a815e280 2024-03-24T00:43:22.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.529+0000 7fd1a4ff9640 1 --2- 172.21.15.67:0/1035390540 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd19406bc00 0x7fd19406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:22.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.530+0000 7fd1a4ff9640 1 -- 172.21.15.67:0/1035390540 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd1a008a8b0 con 0x7fd1a815e280 2024-03-24T00:43:22.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.530+0000 7fd1a7fff640 1 --2- 172.21.15.67:0/1035390540 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd19406bc00 0x7fd19406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:22.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.530+0000 7fd1a7fff640 1 --2- 172.21.15.67:0/1035390540 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd19406bc00 0x7fd19406e0c0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fd1980011e0 tx=0x7fd198009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:22.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.538+0000 7fd1a4ff9640 1 -- 172.21.15.67:0/1035390540 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd1a80cffb0 con 0x7fd1a815e280 2024-03-24T00:43:22.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.773+0000 7fd1ae5cd640 1 -- 172.21.15.67:0/1035390540 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd1a804f400 con 0x7fd1a815e280 2024-03-24T00:43:22.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.774+0000 7fd1a4ff9640 1 -- 172.21.15.67:0/1035390540 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v76) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd1a00592a0 con 0x7fd1a815e280 2024-03-24T00:43:22.773 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-24T00:43:22.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.775+0000 7fd1ae5cd640 1 -- 172.21.15.67:0/1035390540 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd19406bc00 msgr2=0x7fd19406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:22.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.775+0000 7fd1ae5cd640 1 --2- 172.21.15.67:0/1035390540 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd19406bc00 0x7fd19406e0c0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fd1980011e0 tx=0x7fd198009040 comp rx=0 tx=0).stop 2024-03-24T00:43:22.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.775+0000 7fd1ae5cd640 1 -- 172.21.15.67:0/1035390540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1a815e280 msgr2=0x7fd1a80b4b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:22.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.775+0000 7fd1ae5cd640 1 --2- 172.21.15.67:0/1035390540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1a815e280 0x7fd1a80b4b40 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fd1a000c730 tx=0x7fd1a000cbf0 comp rx=0 tx=0).stop 2024-03-24T00:43:22.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.775+0000 7fd1ae5cd640 1 -- 172.21.15.67:0/1035390540 shutdown_connections 2024-03-24T00:43:22.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.776+0000 7fd1ae5cd640 1 --2- 172.21.15.67:0/1035390540 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd19406bc00 0x7fd19406e0c0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:22.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.776+0000 7fd1ae5cd640 1 --2- 172.21.15.67:0/1035390540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1a815e280 0x7fd1a80b4b40 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:22.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.776+0000 7fd1ae5cd640 1 --2- 172.21.15.67:0/1035390540 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd1a815dea0 0x7fd1a815b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:22.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.776+0000 7fd1ae5cd640 1 --2- 172.21.15.67:0/1035390540 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1a80d3ae0 0x7fd1a80cde60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:22.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.776+0000 7fd1ae5cd640 1 -- 172.21.15.67:0/1035390540 >> 172.21.15.67:0/1035390540 conn(0x7fd1a8072da0 msgr2=0x7fd1a80d3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:22.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.776+0000 7fd1ae5cd640 1 -- 172.21.15.67:0/1035390540 shutdown_connections 2024-03-24T00:43:22.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.776+0000 7fd1ae5cd640 1 -- 172.21.15.67:0/1035390540 wait complete. 2024-03-24T00:43:22.786 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:43:22.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.891+0000 7fead3b48640 1 Processor -- start 2024-03-24T00:43:22.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.891+0000 7fead3b48640 1 -- start start 2024-03-24T00:43:22.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.892+0000 7fead3b48640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7feacc0d18b0 0x7feacc15c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:22.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.892+0000 7fead3b48640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7feacc160430 0x7feacc0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:22.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.892+0000 7fead3b48640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feacc15c7c0 con 0x7feacc0d1c90 2024-03-24T00:43:22.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.892+0000 7fead3b48640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feacc15caa0 con 0x7feacc0d18b0 2024-03-24T00:43:22.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.892+0000 7fead3b48640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feacc15cd80 con 0x7feacc160430 2024-03-24T00:43:22.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.892+0000 7fead10e3640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7feacc0d1c90 0x7feacc15fd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37580/0 (socket says 172.21.15.67:37580) 2024-03-24T00:43:22.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.892+0000 7fead18e4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7feacc0d18b0 0x7feacc15c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:22.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.892+0000 7fead10e3640 1 -- 172.21.15.67:0/1734954756 learned_addr learned my addr 172.21.15.67:0/1734954756 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:43:22.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.892+0000 7fead20e5640 1 --2- 172.21.15.67:0/1734954756 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7feacc160430 0x7feacc0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:22.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.892+0000 7fead18e4640 1 -- 172.21.15.67:0/1734954756 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7feacc160430 msgr2=0x7feacc0cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:22.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.892+0000 7fead18e4640 1 --2- 172.21.15.67:0/1734954756 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7feacc160430 0x7feacc0cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:22.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.892+0000 7fead18e4640 1 -- 172.21.15.67:0/1734954756 >> v1:172.21.15.67:6789/0 conn(0x7feacc0d1c90 legacy=0x7feacc15fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:22.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.892+0000 7fead18e4640 1 -- 172.21.15.67:0/1734954756 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feacc15d060 con 0x7feacc0d18b0 2024-03-24T00:43:22.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.893+0000 7fead18e4640 1 --2- 172.21.15.67:0/1734954756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feacc0d18b0 0x7feacc15c1b0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7feac80097a0 tx=0x7feac802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=441262a715cd45b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:22.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.893+0000 7fead08e2640 1 -- 172.21.15.67:0/1734954756 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac80315a0 con 0x7feacc0d18b0 2024-03-24T00:43:22.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.893+0000 7fead08e2640 1 -- 172.21.15.67:0/1734954756 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feac8037070 con 0x7feacc0d18b0 2024-03-24T00:43:22.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.893+0000 7fead08e2640 1 -- 172.21.15.67:0/1734954756 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac8031960 con 0x7feacc0d18b0 2024-03-24T00:43:22.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.894+0000 7fead3b48640 1 -- 172.21.15.67:0/1734954756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feacc0d18b0 msgr2=0x7feacc15c1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:22.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.894+0000 7fead3b48640 1 --2- 172.21.15.67:0/1734954756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feacc0d18b0 0x7feacc15c1b0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7feac80097a0 tx=0x7feac802b840 comp rx=0 tx=0).stop 2024-03-24T00:43:22.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.894+0000 7fead3b48640 1 -- 172.21.15.67:0/1734954756 shutdown_connections 2024-03-24T00:43:22.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.894+0000 7fead3b48640 1 --2- 172.21.15.67:0/1734954756 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7feacc160430 0x7feacc0cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:22.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.894+0000 7fead3b48640 1 --2- 172.21.15.67:0/1734954756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feacc0d18b0 0x7feacc15c1b0 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:22.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.894+0000 7fead3b48640 1 -- 172.21.15.67:0/1734954756 >> 172.21.15.67:0/1734954756 conn(0x7feacc08b1b0 msgr2=0x7feacc08b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:22.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.894+0000 7fead3b48640 1 -- 172.21.15.67:0/1734954756 shutdown_connections 2024-03-24T00:43:22.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.894+0000 7fead3b48640 1 -- 172.21.15.67:0/1734954756 wait complete. 2024-03-24T00:43:22.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.895+0000 7fead3b48640 1 Processor -- start 2024-03-24T00:43:22.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.895+0000 7fead3b48640 1 -- start start 2024-03-24T00:43:22.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.896+0000 7fead3b48640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feacc0d18b0 0x7feacc0b0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:22.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.896+0000 7fead3b48640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7feacc0d1c90 0x7feacc0ce8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:22.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.896+0000 7fead3b48640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feacc160430 0x7feacc0cede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:22.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.896+0000 7fead3b48640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feacc0cfd10 con 0x7feacc160430 2024-03-24T00:43:22.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.896+0000 7fead3b48640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feacc0d2300 con 0x7feacc0d18b0 2024-03-24T00:43:22.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.896+0000 7fead3b48640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feacc0cfe90 con 0x7feacc0d1c90 2024-03-24T00:43:22.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.896+0000 7fead20e5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feacc160430 0x7feacc0cede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:22.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.896+0000 7fead20e5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feacc160430 0x7feacc0cede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50552/0 (socket says 172.21.15.67:50552) 2024-03-24T00:43:22.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.896+0000 7fead20e5640 1 -- 172.21.15.67:0/4010540231 learned_addr learned my addr 172.21.15.67:0/4010540231 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:22.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.896+0000 7fead10e3640 1 --2- 172.21.15.67:0/4010540231 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7feacc0d1c90 0x7feacc0ce8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:22.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.896+0000 7fead18e4640 1 --2- 172.21.15.67:0/4010540231 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feacc0d18b0 0x7feacc0b0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:22.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.896+0000 7fead20e5640 1 -- 172.21.15.67:0/4010540231 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7feacc0d1c90 msgr2=0x7feacc0ce8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:22.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.896+0000 7fead20e5640 1 --2- 172.21.15.67:0/4010540231 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7feacc0d1c90 0x7feacc0ce8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:22.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.896+0000 7fead20e5640 1 -- 172.21.15.67:0/4010540231 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feacc0d18b0 msgr2=0x7feacc0b0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:22.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.896+0000 7fead20e5640 1 --2- 172.21.15.67:0/4010540231 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feacc0d18b0 0x7feacc0b0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:22.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.896+0000 7fead20e5640 1 -- 172.21.15.67:0/4010540231 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feacc0c9540 con 0x7feacc160430 2024-03-24T00:43:22.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.896+0000 7fead18e4640 1 --2- 172.21.15.67:0/4010540231 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feacc0d18b0 0x7feacc0b0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:43:22.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.897+0000 7fead20e5640 1 --2- 172.21.15.67:0/4010540231 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feacc160430 0x7feacc0cede0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7feac4007e60 tx=0x7feac400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:22.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.897+0000 7feac27fc640 1 -- 172.21.15.67:0/4010540231 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac4014420 con 0x7feacc160430 2024-03-24T00:43:22.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.897+0000 7feac27fc640 1 -- 172.21.15.67:0/4010540231 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feac4019070 con 0x7feacc160430 2024-03-24T00:43:22.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.897+0000 7fead3b48640 1 -- 172.21.15.67:0/4010540231 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feacc0b14b0 con 0x7feacc160430 2024-03-24T00:43:22.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.897+0000 7feac27fc640 1 -- 172.21.15.67:0/4010540231 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac4014820 con 0x7feacc160430 2024-03-24T00:43:22.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.897+0000 7fead3b48640 1 -- 172.21.15.67:0/4010540231 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feacc0b01b0 con 0x7feacc160430 2024-03-24T00:43:22.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.898+0000 7fead3b48640 1 -- 172.21.15.67:0/4010540231 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feacc04f400 con 0x7feacc160430 2024-03-24T00:43:22.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.898+0000 7feac27fc640 1 -- 172.21.15.67:0/4010540231 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7feac40149c0 con 0x7feacc160430 2024-03-24T00:43:22.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.899+0000 7feac27fc640 1 --2- 172.21.15.67:0/4010540231 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7feab006bbd0 0x7feab006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:22.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.899+0000 7feac27fc640 1 -- 172.21.15.67:0/4010540231 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6101+0+0 (secure 0 0 0) 0x7feac408b450 con 0x7feacc160430 2024-03-24T00:43:22.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.899+0000 7fead18e4640 1 --2- 172.21.15.67:0/4010540231 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7feab006bbd0 0x7feab006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:22.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.899+0000 7fead18e4640 1 --2- 172.21.15.67:0/4010540231 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7feab006bbd0 0x7feab006e090 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7feac8036000 tx=0x7feac8008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:22.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:22.903+0000 7feac27fc640 1 -- 172.21.15.67:0/4010540231 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7feac4059db0 con 0x7feacc160430 2024-03-24T00:43:23.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.087+0000 7fead3b48640 1 -- 172.21.15.67:0/4010540231 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7feacc0b1840 con 0x7feacc160430 2024-03-24T00:43:23.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.087+0000 7feac27fc640 1 -- 172.21.15.67:0/4010540231 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v76) v1 ==== 100+0+11 (secure 0 0 0) 0x7feac40596b0 con 0x7feacc160430 2024-03-24T00:43:23.087 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-03-24T00:43:23.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.089+0000 7fead3b48640 1 -- 172.21.15.67:0/4010540231 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7feab006bbd0 msgr2=0x7feab006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:23.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.089+0000 7fead3b48640 1 --2- 172.21.15.67:0/4010540231 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7feab006bbd0 0x7feab006e090 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7feac8036000 tx=0x7feac8008030 comp rx=0 tx=0).stop 2024-03-24T00:43:23.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.089+0000 7fead3b48640 1 -- 172.21.15.67:0/4010540231 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feacc160430 msgr2=0x7feacc0cede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:23.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.089+0000 7fead3b48640 1 --2- 172.21.15.67:0/4010540231 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feacc160430 0x7feacc0cede0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7feac4007e60 tx=0x7feac400e4b0 comp rx=0 tx=0).stop 2024-03-24T00:43:23.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.089+0000 7fead3b48640 1 -- 172.21.15.67:0/4010540231 shutdown_connections 2024-03-24T00:43:23.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.089+0000 7fead3b48640 1 --2- 172.21.15.67:0/4010540231 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7feab006bbd0 0x7feab006e090 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:23.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.089+0000 7fead3b48640 1 --2- 172.21.15.67:0/4010540231 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feacc160430 0x7feacc0cede0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:23.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.089+0000 7fead3b48640 1 --2- 172.21.15.67:0/4010540231 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7feacc0d1c90 0x7feacc0ce8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:23.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.089+0000 7fead3b48640 1 --2- 172.21.15.67:0/4010540231 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feacc0d18b0 0x7feacc0b0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:23.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.089+0000 7fead3b48640 1 -- 172.21.15.67:0/4010540231 >> 172.21.15.67:0/4010540231 conn(0x7feacc08b1b0 msgr2=0x7feacc151090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:23.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.089+0000 7fead3b48640 1 -- 172.21.15.67:0/4010540231 shutdown_connections 2024-03-24T00:43:23.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.089+0000 7fead3b48640 1 -- 172.21.15.67:0/4010540231 wait complete. 2024-03-24T00:43:23.102 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T00:43:23.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.203+0000 7f191bf22640 1 Processor -- start 2024-03-24T00:43:23.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.204+0000 7f191bf22640 1 -- start start 2024-03-24T00:43:23.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.204+0000 7f191bf22640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f191415e0f0 0x7f19140b95b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:23.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.204+0000 7f191bf22640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19140d2230 con 0x7f19140b9af0 2024-03-24T00:43:23.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.204+0000 7f191bf22640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19140d2510 con 0x7f19140b6dc0 2024-03-24T00:43:23.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.204+0000 7f191bf22640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19140d27f0 con 0x7f191415e0f0 2024-03-24T00:43:23.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.204+0000 7f19194bd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f19140b9af0 0x7f19140d32d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37596/0 (socket says 172.21.15.67:37596) 2024-03-24T00:43:23.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.204+0000 7f19194bd640 1 -- 172.21.15.67:0/1178125080 learned_addr learned my addr 172.21.15.67:0/1178125080 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:43:23.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.205+0000 7f1918cbc640 1 -- 172.21.15.67:0/1178125080 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3673100391 0 0) 0x7f19140d2230 con 0x7f19140b9af0 2024-03-24T00:43:23.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.205+0000 7f1919cbe640 1 --2- 172.21.15.67:0/1178125080 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f191415e0f0 0x7f19140b95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:23.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.205+0000 7f1918cbc640 1 -- 172.21.15.67:0/1178125080 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1900003520 con 0x7f19140b9af0 2024-03-24T00:43:23.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.205+0000 7f1918cbc640 1 -- 172.21.15.67:0/1178125080 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 690731999 0 0) 0x7f1900003520 con 0x7f19140b9af0 2024-03-24T00:43:23.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.205+0000 7f1918cbc640 1 -- 172.21.15.67:0/1178125080 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f191415e0f0 msgr2=0x7f19140b95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:23.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.205+0000 7f1918cbc640 1 --2- 172.21.15.67:0/1178125080 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f191415e0f0 0x7f19140b95b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:23.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.205+0000 7f1918cbc640 1 -- 172.21.15.67:0/1178125080 >> v1:172.21.15.72:6789/0 conn(0x7f19140b6dc0 legacy=0x7f19140d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:23.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.205+0000 7f1918cbc640 1 -- 172.21.15.67:0/1178125080 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19140d2ad0 con 0x7f19140b9af0 2024-03-24T00:43:23.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.205+0000 7f1918cbc640 1 -- 172.21.15.67:0/1178125080 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f19040017f0 con 0x7f19140b9af0 2024-03-24T00:43:23.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.205+0000 7f1918cbc640 1 -- 172.21.15.67:0/1178125080 >> v1:172.21.15.67:6789/0 conn(0x7f19140b9af0 legacy=0x7f19140d32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:23.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.205+0000 7f1918cbc640 1 --2- 172.21.15.67:0/1178125080 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f191415e0f0 0x7f19000039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:23.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.205+0000 7f1918cbc640 1 --2- 172.21.15.67:0/1178125080 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1900003f60 0x7f1900004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:23.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.205+0000 7f1918cbc640 1 --2- 172.21.15.67:0/1178125080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1900004d40 0x7f19000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:23.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.205+0000 7f1918cbc640 1 -- 172.21.15.67:0/1178125080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19140d27f0 con 0x7f1900004d40 2024-03-24T00:43:23.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.205+0000 7f1918cbc640 1 -- 172.21.15.67:0/1178125080 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f190000baa0 con 0x7f1900003f60 2024-03-24T00:43:23.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.205+0000 7f1918cbc640 1 -- 172.21.15.67:0/1178125080 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f190000bca0 con 0x7f191415e0f0 2024-03-24T00:43:23.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.205+0000 7f191a4bf640 1 --2- 172.21.15.67:0/1178125080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1900004d40 0x7f19000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:23.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.205+0000 7f19194bd640 1 --2- 172.21.15.67:0/1178125080 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1900003f60 0x7f1900004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:23.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.205+0000 7f1919cbe640 1 --2- 172.21.15.67:0/1178125080 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f191415e0f0 0x7f19000039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:23.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.205+0000 7f191a4bf640 1 -- 172.21.15.67:0/1178125080 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f191415e0f0 msgr2=0x7f19000039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:23.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.205+0000 7f191a4bf640 1 --2- 172.21.15.67:0/1178125080 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f191415e0f0 0x7f19000039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:23.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.205+0000 7f191a4bf640 1 -- 172.21.15.67:0/1178125080 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1900003f60 msgr2=0x7f1900004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:23.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.205+0000 7f191a4bf640 1 --2- 172.21.15.67:0/1178125080 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1900003f60 0x7f1900004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:23.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.205+0000 7f191a4bf640 1 -- 172.21.15.67:0/1178125080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1900003520 con 0x7f1900004d40 2024-03-24T00:43:23.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.206+0000 7f191a4bf640 1 --2- 172.21.15.67:0/1178125080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1900004d40 0x7f19000051c0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f190c001e50 tx=0x7f1900005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3824adc4ff86c6d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:23.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.206+0000 7f1918cbc640 1 -- 172.21.15.67:0/1178125080 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f190c001a70 con 0x7f1900004d40 2024-03-24T00:43:23.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.206+0000 7f1918cbc640 1 -- 172.21.15.67:0/1178125080 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f190c003b70 con 0x7f1900004d40 2024-03-24T00:43:23.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.206+0000 7f1918cbc640 1 -- 172.21.15.67:0/1178125080 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f190c0050d0 con 0x7f1900004d40 2024-03-24T00:43:23.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.207+0000 7f191bf22640 1 -- 172.21.15.67:0/1178125080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1900004d40 msgr2=0x7f19000051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:23.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.207+0000 7f191bf22640 1 --2- 172.21.15.67:0/1178125080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1900004d40 0x7f19000051c0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f190c001e50 tx=0x7f1900005830 comp rx=0 tx=0).stop 2024-03-24T00:43:23.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.207+0000 7f191a4bf640 1 -- 172.21.15.67:0/1178125080 reap_dead start 2024-03-24T00:43:23.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.207+0000 7f191bf22640 1 -- 172.21.15.67:0/1178125080 shutdown_connections 2024-03-24T00:43:23.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.207+0000 7f191bf22640 1 -- 172.21.15.67:0/1178125080 >> 172.21.15.67:0/1178125080 conn(0x7f1914072b60 msgr2=0x7f1914072f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:23.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.207+0000 7f191bf22640 1 -- 172.21.15.67:0/1178125080 shutdown_connections 2024-03-24T00:43:23.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.207+0000 7f191bf22640 1 -- 172.21.15.67:0/1178125080 wait complete. 2024-03-24T00:43:23.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.208+0000 7f191bf22640 1 Processor -- start 2024-03-24T00:43:23.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.208+0000 7f191bf22640 1 -- start start 2024-03-24T00:43:23.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.209+0000 7f191bf22640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1900004d40 0x7f1914161170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:23.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.209+0000 7f191bf22640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f19141529d0 0x7f191415e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:23.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.209+0000 7f191bf22640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f191415fe10 0x7f191415f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:23.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.209+0000 7f191bf22640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19140b3350 con 0x7f191415fe10 2024-03-24T00:43:23.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.209+0000 7f191bf22640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19140b31d0 con 0x7f1900004d40 2024-03-24T00:43:23.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.209+0000 7f191bf22640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19140d39e0 con 0x7f19141529d0 2024-03-24T00:43:23.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.209+0000 7f191a4bf640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f191415fe10 0x7f191415f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:23.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.209+0000 7f191a4bf640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f191415fe10 0x7f191415f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50570/0 (socket says 172.21.15.67:50570) 2024-03-24T00:43:23.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.209+0000 7f191a4bf640 1 -- 172.21.15.67:0/2894358614 learned_addr learned my addr 172.21.15.67:0/2894358614 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:23.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.209+0000 7f1919cbe640 1 --2- 172.21.15.67:0/2894358614 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1900004d40 0x7f1914161170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:23.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.209+0000 7f19194bd640 1 --2- 172.21.15.67:0/2894358614 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f19141529d0 0x7f191415e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:23.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.209+0000 7f191a4bf640 1 -- 172.21.15.67:0/2894358614 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f19141529d0 msgr2=0x7f191415e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:23.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.209+0000 7f191a4bf640 1 --2- 172.21.15.67:0/2894358614 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f19141529d0 0x7f191415e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:23.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.209+0000 7f191a4bf640 1 -- 172.21.15.67:0/2894358614 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1900004d40 msgr2=0x7f1914161170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:23.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.209+0000 7f191a4bf640 1 --2- 172.21.15.67:0/2894358614 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1900004d40 0x7f1914161170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:23.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.209+0000 7f191a4bf640 1 -- 172.21.15.67:0/2894358614 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1914157130 con 0x7f191415fe10 2024-03-24T00:43:23.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.209+0000 7f191a4bf640 1 --2- 172.21.15.67:0/2894358614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f191415fe10 0x7f191415f250 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f190c001f70 tx=0x7f190c0014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:23.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.210+0000 7f190a7fc640 1 -- 172.21.15.67:0/2894358614 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f190c007db0 con 0x7f191415fe10 2024-03-24T00:43:23.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.210+0000 7f190a7fc640 1 -- 172.21.15.67:0/2894358614 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f190c003970 con 0x7f191415fe10 2024-03-24T00:43:23.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.210+0000 7f190a7fc640 1 -- 172.21.15.67:0/2894358614 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f190c006d90 con 0x7f191415fe10 2024-03-24T00:43:23.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.210+0000 7f191bf22640 1 -- 172.21.15.67:0/2894358614 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f191415ee00 con 0x7f191415fe10 2024-03-24T00:43:23.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.210+0000 7f191bf22640 1 -- 172.21.15.67:0/2894358614 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1914156e50 con 0x7f191415fe10 2024-03-24T00:43:23.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.211+0000 7f191bf22640 1 -- 172.21.15.67:0/2894358614 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19140b2d70 con 0x7f191415fe10 2024-03-24T00:43:23.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.211+0000 7f190a7fc640 1 -- 172.21.15.67:0/2894358614 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f190c041030 con 0x7f191415fe10 2024-03-24T00:43:23.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.211+0000 7f190a7fc640 1 --2- 172.21.15.67:0/2894358614 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f18f806bc00 0x7f18f806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:23.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.212+0000 7f190a7fc640 1 -- 172.21.15.67:0/2894358614 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f190c0abda0 con 0x7f191415fe10 2024-03-24T00:43:23.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.212+0000 7f1919cbe640 1 --2- 172.21.15.67:0/2894358614 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f18f806bc00 0x7f18f806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:23.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.212+0000 7f1919cbe640 1 --2- 172.21.15.67:0/2894358614 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f18f806bc00 0x7f18f806e0c0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f1910009b00 tx=0x7f1910008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:23.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.216+0000 7f190a7fc640 1 -- 172.21.15.67:0/2894358614 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f19140b2d70 con 0x7f191415fe10 2024-03-24T00:43:23.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.429+0000 7f191bf22640 1 -- 172.21.15.67:0/2894358614 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f191404f400 con 0x7f191415fe10 2024-03-24T00:43:23.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.430+0000 7f190a7fc640 1 -- 172.21.15.67:0/2894358614 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v76) v1 ==== 80+0+0 (secure 0 0 0) 0x7f190c033020 con 0x7f191415fe10 2024-03-24T00:43:23.430 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-03-24T00:43:23.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.432+0000 7f191bf22640 1 -- 172.21.15.67:0/2894358614 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f18f806bc00 msgr2=0x7f18f806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:23.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.432+0000 7f191bf22640 1 --2- 172.21.15.67:0/2894358614 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f18f806bc00 0x7f18f806e0c0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f1910009b00 tx=0x7f1910008040 comp rx=0 tx=0).stop 2024-03-24T00:43:23.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.432+0000 7f191bf22640 1 -- 172.21.15.67:0/2894358614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f191415fe10 msgr2=0x7f191415f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:23.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.432+0000 7f191bf22640 1 --2- 172.21.15.67:0/2894358614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f191415fe10 0x7f191415f250 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f190c001f70 tx=0x7f190c0014a0 comp rx=0 tx=0).stop 2024-03-24T00:43:23.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.432+0000 7f191bf22640 1 -- 172.21.15.67:0/2894358614 shutdown_connections 2024-03-24T00:43:23.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.432+0000 7f191bf22640 1 --2- 172.21.15.67:0/2894358614 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f18f806bc00 0x7f18f806e0c0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:23.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.432+0000 7f191bf22640 1 --2- 172.21.15.67:0/2894358614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f191415fe10 0x7f191415f250 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:23.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.432+0000 7f191bf22640 1 --2- 172.21.15.67:0/2894358614 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f19141529d0 0x7f191415e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:23.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.432+0000 7f191bf22640 1 --2- 172.21.15.67:0/2894358614 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1900004d40 0x7f1914161170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:23.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.432+0000 7f191bf22640 1 -- 172.21.15.67:0/2894358614 >> 172.21.15.67:0/2894358614 conn(0x7f1914072b60 msgr2=0x7f1914072f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:23.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.432+0000 7f191bf22640 1 -- 172.21.15.67:0/2894358614 shutdown_connections 2024-03-24T00:43:23.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.432+0000 7f191bf22640 1 -- 172.21.15.67:0/2894358614 wait complete. 2024-03-24T00:43:23.442 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-24T00:43:23.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:43:23.442 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-24T00:43:23.490 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-03-24T00:43:23.490 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:43:23.490 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-24T00:43:23.521 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:43:23.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.663+0000 7f0d4a53b640 1 Processor -- start 2024-03-24T00:43:23.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.663+0000 7f0d4a53b640 1 -- start start 2024-03-24T00:43:23.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.665+0000 7f0d4a53b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0d4415af80 0x7f0d4406c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:23.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.665+0000 7f0d4a53b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f0d440b3ae0 0x7f0d440af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:23.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.665+0000 7f0d4a53b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d440b4250 con 0x7f0d44067150 2024-03-24T00:43:23.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.665+0000 7f0d4a53b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d440b3f70 con 0x7f0d4415af80 2024-03-24T00:43:23.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.665+0000 7f0d4a53b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d4406baf0 con 0x7f0d440b3ae0 2024-03-24T00:43:23.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.665+0000 7f0d43fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0d4415af80 0x7f0d4406c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:23.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.665+0000 7f0d48ad8640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0d44067150 0x7f0d440ad240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58264/0 (socket says 172.21.15.67:58264) 2024-03-24T00:43:23.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.665+0000 7f0d48ad8640 1 -- 172.21.15.67:0/4237923173 learned_addr learned my addr 172.21.15.67:0/4237923173 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:43:23.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.665+0000 7f0d43fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0d4415af80 0x7f0d4406c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48214/0 (socket says 172.21.15.67:48214) 2024-03-24T00:43:23.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.665+0000 7f0d437fe640 1 --2- 172.21.15.67:0/4237923173 >> v2:172.21.15.72:3300/0 conn(0x7f0d440b3ae0 0x7f0d440af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:23.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.665+0000 7f0d43fff640 1 -- 172.21.15.67:0/4237923173 >> v2:172.21.15.72:3300/0 conn(0x7f0d440b3ae0 msgr2=0x7f0d440af8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:23.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.665+0000 7f0d43fff640 1 --2- 172.21.15.67:0/4237923173 >> v2:172.21.15.72:3300/0 conn(0x7f0d440b3ae0 0x7f0d440af8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:23.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.665+0000 7f0d43fff640 1 -- 172.21.15.67:0/4237923173 >> v1:172.21.15.67:6789/0 conn(0x7f0d44067150 legacy=0x7f0d440ad240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:23.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.665+0000 7f0d43fff640 1 -- 172.21.15.67:0/4237923173 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d4406bdd0 con 0x7f0d4415af80 2024-03-24T00:43:23.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.665+0000 7f0d42ffd640 1 -- 172.21.15.67:0/4237923173 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1449103643 0 0) 0x7f0d440b4250 con 0x7f0d44067150 2024-03-24T00:43:23.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.665+0000 7f0d437fe640 1 --2- 172.21.15.67:0/4237923173 >> v2:172.21.15.72:3300/0 conn(0x7f0d440b3ae0 0x7f0d440af8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:43:23.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.666+0000 7f0d43fff640 1 --2- 172.21.15.67:0/4237923173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d4415af80 0x7f0d4406c5d0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f0d44073170 tx=0x7f0d3402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77108172dd05069e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:23.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.666+0000 7f0d42ffd640 1 -- 172.21.15.67:0/4237923173 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d34035020 con 0x7f0d4415af80 2024-03-24T00:43:23.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.666+0000 7f0d42ffd640 1 -- 172.21.15.67:0/4237923173 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d34031830 con 0x7f0d4415af80 2024-03-24T00:43:23.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.666+0000 7f0d42ffd640 1 -- 172.21.15.67:0/4237923173 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d34031c00 con 0x7f0d4415af80 2024-03-24T00:43:23.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.667+0000 7f0d4a53b640 1 -- 172.21.15.67:0/4237923173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d4415af80 msgr2=0x7f0d4406c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:23.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.667+0000 7f0d4a53b640 1 --2- 172.21.15.67:0/4237923173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d4415af80 0x7f0d4406c5d0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f0d44073170 tx=0x7f0d3402b8d0 comp rx=0 tx=0).stop 2024-03-24T00:43:23.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.667+0000 7f0d4a53b640 1 -- 172.21.15.67:0/4237923173 shutdown_connections 2024-03-24T00:43:23.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.667+0000 7f0d4a53b640 1 --2- 172.21.15.67:0/4237923173 >> v2:172.21.15.72:3300/0 conn(0x7f0d440b3ae0 0x7f0d440af8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:23.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.667+0000 7f0d4a53b640 1 --2- 172.21.15.67:0/4237923173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d4415af80 0x7f0d4406c5d0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:23.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.667+0000 7f0d4a53b640 1 -- 172.21.15.67:0/4237923173 >> 172.21.15.67:0/4237923173 conn(0x7f0d440b5440 msgr2=0x7f0d440b33d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:23.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.667+0000 7f0d4a53b640 1 -- 172.21.15.67:0/4237923173 shutdown_connections 2024-03-24T00:43:23.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.667+0000 7f0d4a53b640 1 -- 172.21.15.67:0/4237923173 wait complete. 2024-03-24T00:43:23.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.668+0000 7f0d4a53b640 1 Processor -- start 2024-03-24T00:43:23.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.668+0000 7f0d4a53b640 1 -- start start 2024-03-24T00:43:23.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.669+0000 7f0d4a53b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0d44067150 0x7f0d440c5d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:23.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.669+0000 7f0d4a53b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d4415af80 0x7f0d440ae630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:23.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.669+0000 7f0d4a53b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d4414a4b0 0x7f0d4414a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:23.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.669+0000 7f0d4a53b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d4415b360 con 0x7f0d4414a4b0 2024-03-24T00:43:23.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.669+0000 7f0d4a53b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d4406cb10 con 0x7f0d4415af80 2024-03-24T00:43:23.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.669+0000 7f0d4a53b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d4415b4e0 con 0x7f0d44067150 2024-03-24T00:43:23.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.669+0000 7f0d48ad8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d4414a4b0 0x7f0d4414a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:23.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.669+0000 7f0d48ad8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d4414a4b0 0x7f0d4414a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48226/0 (socket says 172.21.15.67:48226) 2024-03-24T00:43:23.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.669+0000 7f0d48ad8640 1 -- 172.21.15.67:0/2690186979 learned_addr learned my addr 172.21.15.67:0/2690186979 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:23.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.669+0000 7f0d437fe640 1 --2- 172.21.15.67:0/2690186979 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d4415af80 0x7f0d440ae630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:23.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.669+0000 7f0d43fff640 1 --2- 172.21.15.67:0/2690186979 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0d44067150 0x7f0d440c5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:23.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.670+0000 7f0d48ad8640 1 -- 172.21.15.67:0/2690186979 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0d44067150 msgr2=0x7f0d440c5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:23.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.670+0000 7f0d48ad8640 1 --2- 172.21.15.67:0/2690186979 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0d44067150 0x7f0d440c5d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:23.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.670+0000 7f0d48ad8640 1 -- 172.21.15.67:0/2690186979 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d4415af80 msgr2=0x7f0d440ae630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:23.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.670+0000 7f0d48ad8640 1 --2- 172.21.15.67:0/2690186979 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d4415af80 0x7f0d440ae630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:23.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.670+0000 7f0d48ad8640 1 -- 172.21.15.67:0/2690186979 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d440c62d0 con 0x7f0d4414a4b0 2024-03-24T00:43:23.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.670+0000 7f0d48ad8640 1 --2- 172.21.15.67:0/2690186979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d4414a4b0 0x7f0d4414a920 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c00c7e0 tx=0x7f0d3c00cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:23.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.670+0000 7f0d40ff9640 1 -- 172.21.15.67:0/2690186979 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d3c018070 con 0x7f0d4414a4b0 2024-03-24T00:43:23.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.670+0000 7f0d40ff9640 1 -- 172.21.15.67:0/2690186979 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d3c00a640 con 0x7f0d4414a4b0 2024-03-24T00:43:23.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.670+0000 7f0d4a53b640 1 -- 172.21.15.67:0/2690186979 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d44158d70 con 0x7f0d4414a4b0 2024-03-24T00:43:23.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.670+0000 7f0d40ff9640 1 -- 172.21.15.67:0/2690186979 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d3c00aa30 con 0x7f0d4414a4b0 2024-03-24T00:43:23.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.670+0000 7f0d4a53b640 1 -- 172.21.15.67:0/2690186979 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d44159330 con 0x7f0d4414a4b0 2024-03-24T00:43:23.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.671+0000 7f0d4a53b640 1 -- 172.21.15.67:0/2690186979 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d4404f400 con 0x7f0d4414a4b0 2024-03-24T00:43:23.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.672+0000 7f0d40ff9640 1 -- 172.21.15.67:0/2690186979 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0d3c01b070 con 0x7f0d4414a4b0 2024-03-24T00:43:23.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.672+0000 7f0d40ff9640 1 --2- 172.21.15.67:0/2690186979 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0d3006bbd0 0x7f0d3006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:23.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.672+0000 7f0d40ff9640 1 -- 172.21.15.67:0/2690186979 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0d3c08a120 con 0x7f0d4414a4b0 2024-03-24T00:43:23.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.672+0000 7f0d43fff640 1 --2- 172.21.15.67:0/2690186979 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0d3006bbd0 0x7f0d3006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:23.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.673+0000 7f0d43fff640 1 --2- 172.21.15.67:0/2690186979 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0d3006bbd0 0x7f0d3006e090 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f0d34036000 tx=0x7f0d34008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:23.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.677+0000 7f0d40ff9640 1 -- 172.21.15.67:0/2690186979 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0d3c0584c0 con 0x7f0d4414a4b0 2024-03-24T00:43:23.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.897+0000 7f0d4a53b640 1 -- 172.21.15.67:0/2690186979 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d440b1db0 con 0x7f0d4414a4b0 2024-03-24T00:43:23.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.898+0000 7f0d40ff9640 1 -- 172.21.15.67:0/2690186979 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f0d3c05c560 con 0x7f0d4414a4b0 2024-03-24T00:43:23.898 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:43:23.898 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":40,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5960,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1888378520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1888378520},{"type":"v1","addr":"172.21.15.67:6835","nonce":1888378520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6610,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/2539718386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2539718386},{"type":"v1","addr":"172.21.15.72:6836","nonce":2539718386}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6613,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2376370770","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2376370770},{"type":"v1","addr":"172.21.15.72:6837","nonce":2376370770}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:43:10.177088+0000","modified":"2024-03-24T00:43:12.181406+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6894},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6894":{"gid":6894,"name":"c","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6833/2392802741","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2392802741},{"type":"v1","addr":"172.21.15.67:6833","nonce":2392802741}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6894,"qdb_cluster":[6894]},"id":5}]} 2024-03-24T00:43:23.898 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 40 2024-03-24T00:43:23.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.900+0000 7f0d4a53b640 1 -- 172.21.15.67:0/2690186979 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0d3006bbd0 msgr2=0x7f0d3006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:23.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.900+0000 7f0d4a53b640 1 --2- 172.21.15.67:0/2690186979 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0d3006bbd0 0x7f0d3006e090 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f0d34036000 tx=0x7f0d34008030 comp rx=0 tx=0).stop 2024-03-24T00:43:23.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.900+0000 7f0d4a53b640 1 -- 172.21.15.67:0/2690186979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d4414a4b0 msgr2=0x7f0d4414a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:23.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.900+0000 7f0d4a53b640 1 --2- 172.21.15.67:0/2690186979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d4414a4b0 0x7f0d4414a920 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c00c7e0 tx=0x7f0d3c00cca0 comp rx=0 tx=0).stop 2024-03-24T00:43:23.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.900+0000 7f0d4a53b640 1 -- 172.21.15.67:0/2690186979 shutdown_connections 2024-03-24T00:43:23.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.900+0000 7f0d4a53b640 1 --2- 172.21.15.67:0/2690186979 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0d3006bbd0 0x7f0d3006e090 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:23.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.900+0000 7f0d4a53b640 1 --2- 172.21.15.67:0/2690186979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d4414a4b0 0x7f0d4414a920 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:23.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.900+0000 7f0d4a53b640 1 --2- 172.21.15.67:0/2690186979 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d4415af80 0x7f0d440ae630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:23.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.900+0000 7f0d4a53b640 1 --2- 172.21.15.67:0/2690186979 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0d44067150 0x7f0d440c5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:23.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.900+0000 7f0d4a53b640 1 -- 172.21.15.67:0/2690186979 >> 172.21.15.67:0/2690186979 conn(0x7f0d440b5440 msgr2=0x7f0d440d0cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:23.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.900+0000 7f0d4a53b640 1 -- 172.21.15.67:0/2690186979 shutdown_connections 2024-03-24T00:43:23.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:23.900+0000 7f0d4a53b640 1 -- 172.21.15.67:0/2690186979 wait complete. 2024-03-24T00:43:23.911 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-24T00:43:24.039 INFO:teuthology.orchestra.run.smithi067.stdout:{"message":"","return_code":0} 2024-03-24T00:43:24.039 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-24T00:43:24.039 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-24T00:43:24.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.142+0000 7effc6081640 1 Processor -- start 2024-03-24T00:43:24.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.142+0000 7effc6081640 1 -- start start 2024-03-24T00:43:24.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.143+0000 7effc6081640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7effc015e0f0 0x7effc00b95b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:24.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.143+0000 7effc6081640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7effc00b9af0 0x7effc00d32d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:24.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.143+0000 7effc6081640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effc00d2230 con 0x7effc00d3810 2024-03-24T00:43:24.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.143+0000 7effc6081640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effc00d2510 con 0x7effc015e0f0 2024-03-24T00:43:24.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.143+0000 7effc6081640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effc00d27f0 con 0x7effc00b9af0 2024-03-24T00:43:24.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.143+0000 7effbeffd640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7effc00b9af0 0x7effc00d32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:24.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.143+0000 7effbf7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7effc015e0f0 0x7effc00b95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:24.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.143+0000 7effbffff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7effc00d3810 0x7effc00b6dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:52848/0 (socket says 172.21.15.67:52848) 2024-03-24T00:43:24.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.143+0000 7effbffff640 1 -- 172.21.15.67:0/785206946 learned_addr learned my addr 172.21.15.67:0/785206946 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:43:24.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.144+0000 7effbeffd640 1 -- 172.21.15.67:0/785206946 >> v2:172.21.15.72:3300/0 conn(0x7effc015e0f0 msgr2=0x7effc00b95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:24.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.144+0000 7effbeffd640 1 --2- 172.21.15.67:0/785206946 >> v2:172.21.15.72:3300/0 conn(0x7effc015e0f0 0x7effc00b95b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:24.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.144+0000 7effbeffd640 1 -- 172.21.15.67:0/785206946 >> v1:172.21.15.72:6789/0 conn(0x7effc00d3810 legacy=0x7effc00b6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:24.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.144+0000 7effbeffd640 1 -- 172.21.15.67:0/785206946 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc00d2ad0 con 0x7effc00b9af0 2024-03-24T00:43:24.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.144+0000 7effbe7fc640 1 -- 172.21.15.67:0/785206946 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1225734815 0 0) 0x7effc00d2230 con 0x7effc00d3810 2024-03-24T00:43:24.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.144+0000 7effbf7fe640 1 --2- 172.21.15.67:0/785206946 >> v2:172.21.15.72:3300/0 conn(0x7effc015e0f0 0x7effc00b95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:43:24.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.144+0000 7effbeffd640 1 --2- 172.21.15.67:0/785206946 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7effc00b9af0 0x7effc00d32d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7effb4009790 tx=0x7effb402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8901fc7f9ef41e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:24.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.145+0000 7effbe7fc640 1 -- 172.21.15.67:0/785206946 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb4035020 con 0x7effc00b9af0 2024-03-24T00:43:24.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.145+0000 7effbe7fc640 1 -- 172.21.15.67:0/785206946 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effb4031800 con 0x7effc00b9af0 2024-03-24T00:43:24.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.145+0000 7effbe7fc640 1 -- 172.21.15.67:0/785206946 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb4031bd0 con 0x7effc00b9af0 2024-03-24T00:43:24.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.145+0000 7effc6081640 1 -- 172.21.15.67:0/785206946 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7effc00b9af0 msgr2=0x7effc00d32d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:24.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.145+0000 7effc6081640 1 --2- 172.21.15.67:0/785206946 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7effc00b9af0 0x7effc00d32d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7effb4009790 tx=0x7effb402b8a0 comp rx=0 tx=0).stop 2024-03-24T00:43:24.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.145+0000 7effc6081640 1 -- 172.21.15.67:0/785206946 shutdown_connections 2024-03-24T00:43:24.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.145+0000 7effc6081640 1 --2- 172.21.15.67:0/785206946 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7effc00b9af0 0x7effc00d32d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:24.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.145+0000 7effc6081640 1 --2- 172.21.15.67:0/785206946 >> v2:172.21.15.72:3300/0 conn(0x7effc015e0f0 0x7effc00b95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:24.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.145+0000 7effc6081640 1 -- 172.21.15.67:0/785206946 >> 172.21.15.67:0/785206946 conn(0x7effc0072b60 msgr2=0x7effc0072f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:24.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.145+0000 7effc6081640 1 -- 172.21.15.67:0/785206946 shutdown_connections 2024-03-24T00:43:24.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.145+0000 7effc6081640 1 -- 172.21.15.67:0/785206946 wait complete. 2024-03-24T00:43:24.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.146+0000 7effc6081640 1 Processor -- start 2024-03-24T00:43:24.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.146+0000 7effc6081640 1 -- start start 2024-03-24T00:43:24.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.147+0000 7effc6081640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effc00b9af0 0x7effc00cc000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:24.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.147+0000 7effc6081640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7effc00d3810 0x7effc01f3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:24.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.147+0000 7effc6081640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7effc015e0f0 0x7effc0157290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:24.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.147+0000 7effc6081640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc00b3350 con 0x7effc00b9af0 2024-03-24T00:43:24.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.147+0000 7effc6081640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc00b31d0 con 0x7effc015e0f0 2024-03-24T00:43:24.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.147+0000 7effc6081640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effc00b74d0 con 0x7effc00d3810 2024-03-24T00:43:24.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.147+0000 7effbf7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effc00b9af0 0x7effc00cc000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:24.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.147+0000 7effbf7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effc00b9af0 0x7effc00cc000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48236/0 (socket says 172.21.15.67:48236) 2024-03-24T00:43:24.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.147+0000 7effbf7fe640 1 -- 172.21.15.67:0/2058416805 learned_addr learned my addr 172.21.15.67:0/2058416805 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:24.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.147+0000 7effbffff640 1 --2- 172.21.15.67:0/2058416805 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7effc015e0f0 0x7effc0157290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:24.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.147+0000 7effbeffd640 1 --2- 172.21.15.67:0/2058416805 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7effc00d3810 0x7effc01f3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:24.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.148+0000 7effbf7fe640 1 -- 172.21.15.67:0/2058416805 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7effc00d3810 msgr2=0x7effc01f3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:24.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.148+0000 7effbf7fe640 1 --2- 172.21.15.67:0/2058416805 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7effc00d3810 0x7effc01f3780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:24.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.148+0000 7effbf7fe640 1 -- 172.21.15.67:0/2058416805 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7effc015e0f0 msgr2=0x7effc0157290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:24.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.148+0000 7effbf7fe640 1 --2- 172.21.15.67:0/2058416805 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7effc015e0f0 0x7effc0157290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:24.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.148+0000 7effbf7fe640 1 -- 172.21.15.67:0/2058416805 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc01f3cc0 con 0x7effc00b9af0 2024-03-24T00:43:24.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.148+0000 7effbf7fe640 1 --2- 172.21.15.67:0/2058416805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effc00b9af0 0x7effc00cc000 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7effb000e960 tx=0x7effb000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:24.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.148+0000 7effa7fff640 1 -- 172.21.15.67:0/2058416805 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb0017070 con 0x7effc00b9af0 2024-03-24T00:43:24.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.148+0000 7effa7fff640 1 -- 172.21.15.67:0/2058416805 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effb000ca00 con 0x7effc00b9af0 2024-03-24T00:43:24.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.148+0000 7effc6081640 1 -- 172.21.15.67:0/2058416805 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effc0159400 con 0x7effc00b9af0 2024-03-24T00:43:24.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.148+0000 7effa7fff640 1 -- 172.21.15.67:0/2058416805 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb000cdd0 con 0x7effc00b9af0 2024-03-24T00:43:24.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.148+0000 7effc6081640 1 -- 172.21.15.67:0/2058416805 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effc01f4240 con 0x7effc00b9af0 2024-03-24T00:43:24.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.149+0000 7effc6081640 1 -- 172.21.15.67:0/2058416805 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effc004f400 con 0x7effc00b9af0 2024-03-24T00:43:24.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.149+0000 7effa7fff640 1 -- 172.21.15.67:0/2058416805 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7effb001a070 con 0x7effc00b9af0 2024-03-24T00:43:24.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.150+0000 7effa7fff640 1 --2- 172.21.15.67:0/2058416805 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7effac06bbd0 0x7effac06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:24.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.150+0000 7effa7fff640 1 -- 172.21.15.67:0/2058416805 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6101+0+0 (secure 0 0 0) 0x7effb008a9c0 con 0x7effc00b9af0 2024-03-24T00:43:24.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.150+0000 7effbeffd640 1 --2- 172.21.15.67:0/2058416805 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7effac06bbd0 0x7effac06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:24.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.150+0000 7effbeffd640 1 --2- 172.21.15.67:0/2058416805 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7effac06bbd0 0x7effac06e090 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7effb4036000 tx=0x7effb403d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:24.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.154+0000 7effa7fff640 1 -- 172.21.15.67:0/2058416805 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7effb008ce20 con 0x7effc00b9af0 2024-03-24T00:43:24.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.373+0000 7effc6081640 1 -- 172.21.15.67:0/2058416805 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7effc00b3a00 con 0x7effc00b9af0 2024-03-24T00:43:24.373 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-24T00:43:24.373+0000 7fbfcfa11640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T00:43:24.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.375+0000 7effa7fff640 1 -- 172.21.15.67:0/2058416805 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v41) v1 ==== 161+0+0 (secure 0 0 0) 0x7effb0058d60 con 0x7effc00b9af0 2024-03-24T00:43:24.374 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-24T00:43:24.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.378+0000 7effc6081640 1 -- 172.21.15.67:0/2058416805 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7effac06bbd0 msgr2=0x7effac06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:24.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.378+0000 7effc6081640 1 --2- 172.21.15.67:0/2058416805 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7effac06bbd0 0x7effac06e090 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7effb4036000 tx=0x7effb403d040 comp rx=0 tx=0).stop 2024-03-24T00:43:24.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.378+0000 7effc6081640 1 -- 172.21.15.67:0/2058416805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effc00b9af0 msgr2=0x7effc00cc000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:24.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.378+0000 7effc6081640 1 --2- 172.21.15.67:0/2058416805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effc00b9af0 0x7effc00cc000 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7effb000e960 tx=0x7effb000ee20 comp rx=0 tx=0).stop 2024-03-24T00:43:24.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.379+0000 7effc6081640 1 -- 172.21.15.67:0/2058416805 shutdown_connections 2024-03-24T00:43:24.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.379+0000 7effc6081640 1 --2- 172.21.15.67:0/2058416805 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7effac06bbd0 0x7effac06e090 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:24.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.379+0000 7effc6081640 1 --2- 172.21.15.67:0/2058416805 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7effc015e0f0 0x7effc0157290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:24.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.379+0000 7effc6081640 1 --2- 172.21.15.67:0/2058416805 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7effc00d3810 0x7effc01f3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:24.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.379+0000 7effc6081640 1 --2- 172.21.15.67:0/2058416805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effc00b9af0 0x7effc00cc000 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:24.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.379+0000 7effc6081640 1 -- 172.21.15.67:0/2058416805 >> 172.21.15.67:0/2058416805 conn(0x7effc0072b60 msgr2=0x7effc00b9100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:24.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.379+0000 7effc6081640 1 -- 172.21.15.67:0/2058416805 shutdown_connections 2024-03-24T00:43:24.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.379+0000 7effc6081640 1 -- 172.21.15.67:0/2058416805 wait complete. 2024-03-24T00:43:24.380 INFO:tasks.ceph.mds.c.smithi067.stderr: -4508> 2024-03-24T00:43:16.462+0000 7f4a042aa640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T00:43:24.390 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata setomapval 10000000000.00000000 file_to_be_damaged_head deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 2024-03-24T00:43:24.410 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-03-24T00:43:24.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.433+0000 7f42a1a37f40 1 Processor -- start 2024-03-24T00:43:24.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.433+0000 7f42a1a37f40 1 -- start start 2024-03-24T00:43:24.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.434+0000 7f42a1a37f40 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563c59a193d0 0x563c599d4550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:24.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.434+0000 7f42a1a37f40 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c59aa3cd0 0x563c599ffc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:24.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.434+0000 7f42a1a37f40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563c59aa67f0 0x563c599d31a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:24.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.434+0000 7f42a1a37f40 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c59aa7ab0 con 0x563c59aa3cd0 2024-03-24T00:43:24.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.434+0000 7f42a1a37f40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c599780d0 con 0x563c59aa67f0 2024-03-24T00:43:24.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.434+0000 7f42a1a37f40 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x563c59a197e0 con 0x563c59a193d0 2024-03-24T00:43:24.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.434+0000 7f429e444640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c59aa3cd0 0x563c599ffc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:24.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.434+0000 7f429e444640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c59aa3cd0 0x563c599ffc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48268/0 (socket says 172.21.15.67:48268) 2024-03-24T00:43:24.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.434+0000 7f429e444640 1 -- 172.21.15.67:0/3609769314 learned_addr learned my addr 172.21.15.67:0/3609769314 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:24.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.434+0000 7f429e444640 1 -- 172.21.15.67:0/3609769314 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563c59a193d0 msgr2=0x563c599d4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:24.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.434+0000 7f429e444640 1 --2- 172.21.15.67:0/3609769314 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563c59a193d0 0x563c599d4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:24.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.434+0000 7f429e444640 1 -- 172.21.15.67:0/3609769314 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563c59aa67f0 msgr2=0x563c599d31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:24.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.434+0000 7f429e444640 1 --2- 172.21.15.67:0/3609769314 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563c59aa67f0 0x563c599d31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:24.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.434+0000 7f429e444640 1 -- 172.21.15.67:0/3609769314 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563c59aa2c20 con 0x563c59aa3cd0 2024-03-24T00:43:24.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.434+0000 7f429e444640 1 --2- 172.21.15.67:0/3609769314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c59aa3cd0 0x563c599ffc60 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f429400a770 tx=0x7f429400a860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4b94c2b0f8dd297 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:24.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.434+0000 7f429d442640 1 -- 172.21.15.67:0/3609769314 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4294019070 con 0x563c59aa3cd0 2024-03-24T00:43:24.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.434+0000 7f429d442640 1 -- 172.21.15.67:0/3609769314 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4294007600 con 0x563c59aa3cd0 2024-03-24T00:43:24.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.435+0000 7f429d442640 1 -- 172.21.15.67:0/3609769314 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42940079d0 con 0x563c59aa3cd0 2024-03-24T00:43:24.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.435+0000 7f42a1a37f40 1 -- 172.21.15.67:0/3609769314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c59aa3cd0 msgr2=0x563c599ffc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:24.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.435+0000 7f42a1a37f40 1 --2- 172.21.15.67:0/3609769314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c59aa3cd0 0x563c599ffc60 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f429400a770 tx=0x7f429400a860 comp rx=0 tx=0).stop 2024-03-24T00:43:24.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.435+0000 7f42a1a37f40 1 -- 172.21.15.67:0/3609769314 shutdown_connections 2024-03-24T00:43:24.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.435+0000 7f42a1a37f40 1 --2- 172.21.15.67:0/3609769314 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563c59aa67f0 0x563c599d31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:24.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.435+0000 7f42a1a37f40 1 --2- 172.21.15.67:0/3609769314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c59aa3cd0 0x563c599ffc60 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:24.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.435+0000 7f42a1a37f40 1 --2- 172.21.15.67:0/3609769314 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563c59a193d0 0x563c599d4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:24.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.435+0000 7f42a1a37f40 1 -- 172.21.15.67:0/3609769314 >> 172.21.15.67:0/3609769314 conn(0x563c599fc600 msgr2=0x563c59a014d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:24.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.435+0000 7f42a1a37f40 1 -- 172.21.15.67:0/3609769314 shutdown_connections 2024-03-24T00:43:24.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.435+0000 7f42a1a37f40 1 -- 172.21.15.67:0/3609769314 wait complete. 2024-03-24T00:43:24.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.436+0000 7f42a1a37f40 1 Processor -- start 2024-03-24T00:43:24.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.436+0000 7f42a1a37f40 1 -- start start 2024-03-24T00:43:24.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.436+0000 7f42a1a37f40 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563c59a193d0 0x563c59a9bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:24.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.436+0000 7f42a1a37f40 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c59aa3cd0 0x563c59a9c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:24.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.436+0000 7f42a1a37f40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563c59aa67f0 0x563c59a9c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:24.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.436+0000 7f42a1a37f40 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c59aa31a0 con 0x563c59aa3cd0 2024-03-24T00:43:24.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.436+0000 7f42a1a37f40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c59aa2ec0 con 0x563c59aa67f0 2024-03-24T00:43:24.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.436+0000 7f42a1a37f40 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x563c59aa3480 con 0x563c59a193d0 2024-03-24T00:43:24.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.436+0000 7f429e444640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c59aa3cd0 0x563c59a9c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:24.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.436+0000 7f429e444640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c59aa3cd0 0x563c59a9c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48272/0 (socket says 172.21.15.67:48272) 2024-03-24T00:43:24.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.436+0000 7f429e444640 1 -- 172.21.15.67:0/618256475 learned_addr learned my addr 172.21.15.67:0/618256475 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:24.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.436+0000 7f429ec45640 1 --2- 172.21.15.67:0/618256475 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563c59a193d0 0x563c59a9bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:24.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.436+0000 7f429f446640 1 --2- 172.21.15.67:0/618256475 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563c59aa67f0 0x563c59a9c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:24.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.437+0000 7f429e444640 1 -- 172.21.15.67:0/618256475 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563c59a193d0 msgr2=0x563c59a9bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:24.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.437+0000 7f429e444640 1 --2- 172.21.15.67:0/618256475 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563c59a193d0 0x563c59a9bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:24.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.437+0000 7f429e444640 1 -- 172.21.15.67:0/618256475 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563c59aa67f0 msgr2=0x563c59a9c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:24.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.437+0000 7f429e444640 1 --2- 172.21.15.67:0/618256475 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563c59aa67f0 0x563c59a9c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:24.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.437+0000 7f429e444640 1 -- 172.21.15.67:0/618256475 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563c599d3db0 con 0x563c59aa3cd0 2024-03-24T00:43:24.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.437+0000 7f429e444640 1 --2- 172.21.15.67:0/618256475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c59aa3cd0 0x563c59a9c210 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f4294017040 tx=0x7f4294003020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:24.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.437+0000 7f4287fff640 1 -- 172.21.15.67:0/618256475 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4294019040 con 0x563c59aa3cd0 2024-03-24T00:43:24.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.437+0000 7f4287fff640 1 -- 172.21.15.67:0/618256475 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4294022070 con 0x563c59aa3cd0 2024-03-24T00:43:24.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.437+0000 7f42a1a37f40 1 -- 172.21.15.67:0/618256475 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x563c59a95c80 con 0x563c59aa3cd0 2024-03-24T00:43:24.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.437+0000 7f4287fff640 1 -- 172.21.15.67:0/618256475 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42940146b0 con 0x563c59aa3cd0 2024-03-24T00:43:24.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.437+0000 7f42a1a37f40 1 -- 172.21.15.67:0/618256475 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563c599d3640 con 0x563c59aa3cd0 2024-03-24T00:43:24.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.439+0000 7f4287fff640 1 -- 172.21.15.67:0/618256475 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4294014850 con 0x563c59aa3cd0 2024-03-24T00:43:24.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.439+0000 7f4287fff640 1 --2- 172.21.15.67:0/618256475 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f428c06a5b0 0x7f428c06ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:24.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.439+0000 7f4287fff640 1 -- 172.21.15.67:0/618256475 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f429408a3e0 con 0x563c59aa3cd0 2024-03-24T00:43:24.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.439+0000 7f429ec45640 1 --2- 172.21.15.67:0/618256475 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f428c06a5b0 0x7f428c06ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:24.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.439+0000 7f42a1a37f40 1 --2- 172.21.15.67:0/618256475 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x563c59a91680 0x563c59b4a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:24.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.439+0000 7f42a1a37f40 1 -- 172.21.15.67:0/618256475 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:1 11.10 11:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=115b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77) v9 -- 0x563c59a9cd70 con 0x563c59a91680 2024-03-24T00:43:24.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.439+0000 7f429f446640 1 --2- 172.21.15.67:0/618256475 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x563c59a91680 0x563c59b4a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:24.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.440+0000 7f429f446640 1 --2- 172.21.15.67:0/618256475 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x563c59a91680 0x563c59b4a6d0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:24.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.440+0000 7f429ec45640 1 --2- 172.21.15.67:0/618256475 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f428c06a5b0 0x7f428c06ca70 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f4290054000 tx=0x7f429004c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:24.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.443+0000 7f429f446640 1 -- 172.21.15.67:0/618256475 <== osd.0 v2:172.21.15.67:6808/2628130519 1 ==== osd_op_reply(1 10000000000.00000000 [omap-set-vals] v77'11 uv11 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7f4298001700 con 0x563c59a91680 2024-03-24T00:43:24.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.443+0000 7f42a1a37f40 1 -- 172.21.15.67:0/618256475 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x563c59a91680 msgr2=0x563c59b4a6d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:24.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.443+0000 7f42a1a37f40 1 --2- 172.21.15.67:0/618256475 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x563c59a91680 0x563c59b4a6d0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:24.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.443+0000 7f42a1a37f40 1 -- 172.21.15.67:0/618256475 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f428c06a5b0 msgr2=0x7f428c06ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:24.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.443+0000 7f42a1a37f40 1 --2- 172.21.15.67:0/618256475 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f428c06a5b0 0x7f428c06ca70 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f4290054000 tx=0x7f429004c000 comp rx=0 tx=0).stop 2024-03-24T00:43:24.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.443+0000 7f42a1a37f40 1 -- 172.21.15.67:0/618256475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c59aa3cd0 msgr2=0x563c59a9c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:24.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.443+0000 7f42a1a37f40 1 --2- 172.21.15.67:0/618256475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c59aa3cd0 0x563c59a9c210 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f4294017040 tx=0x7f4294003020 comp rx=0 tx=0).stop 2024-03-24T00:43:24.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.443+0000 7f429f446640 1 -- 172.21.15.67:0/618256475 reap_dead start 2024-03-24T00:43:24.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.443+0000 7f42a1a37f40 1 -- 172.21.15.67:0/618256475 shutdown_connections 2024-03-24T00:43:24.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.443+0000 7f42a1a37f40 1 -- 172.21.15.67:0/618256475 >> 172.21.15.67:0/618256475 conn(0x563c599fc600 msgr2=0x563c59aa40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:24.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.443+0000 7f42a1a37f40 1 -- 172.21.15.67:0/618256475 shutdown_connections 2024-03-24T00:43:24.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.443+0000 7f42a1a37f40 1 -- 172.21.15.67:0/618256475 wait complete. 2024-03-24T00:43:24.445 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-03-24T00:43:24.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.590+0000 7f14f057e640 1 Processor -- start 2024-03-24T00:43:24.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.590+0000 7f14f057e640 1 -- start start 2024-03-24T00:43:24.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.591+0000 7f14f057e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f14e80ce610 0x7f14e80d29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:24.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.591+0000 7f14f057e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14e80cf570 con 0x7f14e80d0d50 2024-03-24T00:43:24.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.591+0000 7f14f057e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14e80cecf0 con 0x7f14e815e130 2024-03-24T00:43:24.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.591+0000 7f14f057e640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14e80ceaa0 con 0x7f14e80ce610 2024-03-24T00:43:24.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.591+0000 7f14edb19640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f14e80d0d50 0x7f14e80d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58270/0 (socket says 172.21.15.67:58270) 2024-03-24T00:43:24.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.591+0000 7f14edb19640 1 -- 172.21.15.67:0/2996441952 learned_addr learned my addr 172.21.15.67:0/2996441952 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:43:24.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.591+0000 7f14eeb1b640 1 --2- 172.21.15.67:0/2996441952 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f14e80ce610 0x7f14e80d29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:24.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.591+0000 7f14ed318640 1 -- 172.21.15.67:0/2996441952 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1917882188 0 0) 0x7f14e80cf570 con 0x7f14e80d0d50 2024-03-24T00:43:24.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.591+0000 7f14ed318640 1 -- 172.21.15.67:0/2996441952 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14d0003520 con 0x7f14e80d0d50 2024-03-24T00:43:24.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.591+0000 7f14ed318640 1 -- 172.21.15.67:0/2996441952 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2772208606 0 0) 0x7f14e80cecf0 con 0x7f14e815e130 2024-03-24T00:43:24.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.591+0000 7f14ed318640 1 -- 172.21.15.67:0/2996441952 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14e80cf570 con 0x7f14e815e130 2024-03-24T00:43:24.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.592+0000 7f14ed318640 1 -- 172.21.15.67:0/2996441952 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3865887934 0 0) 0x7f14d0003520 con 0x7f14e80d0d50 2024-03-24T00:43:24.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.592+0000 7f14eeb1b640 1 -- 172.21.15.67:0/2996441952 >> v1:172.21.15.72:6789/0 conn(0x7f14e815e130 legacy=0x7f14e80d0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:24.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.592+0000 7f14eeb1b640 1 -- 172.21.15.67:0/2996441952 >> v1:172.21.15.67:6789/0 conn(0x7f14e80d0d50 legacy=0x7f14e80d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:24.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.592+0000 7f14eeb1b640 1 -- 172.21.15.67:0/2996441952 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e8162a60 con 0x7f14e80ce610 2024-03-24T00:43:24.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.592+0000 7f14eeb1b640 1 --2- 172.21.15.67:0/2996441952 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f14e80ce610 0x7f14e80d29c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f14d8009dc0 tx=0x7f14d802bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51f921deae5a007f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:24.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.592+0000 7f14ed318640 1 -- 172.21.15.67:0/2996441952 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d8039070 con 0x7f14e80ce610 2024-03-24T00:43:24.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.593+0000 7f14ed318640 1 -- 172.21.15.67:0/2996441952 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14d802c910 con 0x7f14e80ce610 2024-03-24T00:43:24.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.593+0000 7f14ed318640 1 -- 172.21.15.67:0/2996441952 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14dc001670 con 0x7f14e80ce610 2024-03-24T00:43:24.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.593+0000 7f14f057e640 1 -- 172.21.15.67:0/2996441952 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f14e80ce610 msgr2=0x7f14e80d29c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:24.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.593+0000 7f14f057e640 1 --2- 172.21.15.67:0/2996441952 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f14e80ce610 0x7f14e80d29c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f14d8009dc0 tx=0x7f14d802bde0 comp rx=0 tx=0).stop 2024-03-24T00:43:24.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.593+0000 7f14f057e640 1 -- 172.21.15.67:0/2996441952 shutdown_connections 2024-03-24T00:43:24.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.593+0000 7f14f057e640 1 --2- 172.21.15.67:0/2996441952 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f14e80ce610 0x7f14e80d29c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:24.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.593+0000 7f14f057e640 1 -- 172.21.15.67:0/2996441952 >> 172.21.15.67:0/2996441952 conn(0x7f14e806c8a0 msgr2=0x7f14e806ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:24.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.593+0000 7f14f057e640 1 -- 172.21.15.67:0/2996441952 shutdown_connections 2024-03-24T00:43:24.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.593+0000 7f14f057e640 1 -- 172.21.15.67:0/2996441952 wait complete. 2024-03-24T00:43:24.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.594+0000 7f14f057e640 1 Processor -- start 2024-03-24T00:43:24.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.594+0000 7f14f057e640 1 -- start start 2024-03-24T00:43:24.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.595+0000 7f14f057e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14e80ce610 0x7f14e8153660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:24.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.595+0000 7f14f057e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f14e80d0d50 0x7f14e8151d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:24.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.595+0000 7f14f057e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14e815e130 0x7f14e8154c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:24.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.595+0000 7f14f057e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e815e510 con 0x7f14e815e130 2024-03-24T00:43:24.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.595+0000 7f14f057e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e80cf290 con 0x7f14e80ce610 2024-03-24T00:43:24.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.595+0000 7f14f057e640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14e80b2e10 con 0x7f14e80d0d50 2024-03-24T00:43:24.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.595+0000 7f14eeb1b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14e815e130 0x7f14e8154c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:24.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.595+0000 7f14eeb1b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14e815e130 0x7f14e8154c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48276/0 (socket says 172.21.15.67:48276) 2024-03-24T00:43:24.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.595+0000 7f14eeb1b640 1 -- 172.21.15.67:0/2723043089 learned_addr learned my addr 172.21.15.67:0/2723043089 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:24.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.595+0000 7f14edb19640 1 --2- 172.21.15.67:0/2723043089 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f14e80d0d50 0x7f14e8151d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:24.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.595+0000 7f14ee31a640 1 --2- 172.21.15.67:0/2723043089 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14e80ce610 0x7f14e8153660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:24.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.595+0000 7f14eeb1b640 1 -- 172.21.15.67:0/2723043089 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f14e80d0d50 msgr2=0x7f14e8151d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:24.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.595+0000 7f14eeb1b640 1 --2- 172.21.15.67:0/2723043089 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f14e80d0d50 0x7f14e8151d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:24.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.595+0000 7f14eeb1b640 1 -- 172.21.15.67:0/2723043089 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14e80ce610 msgr2=0x7f14e8153660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:24.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.595+0000 7f14eeb1b640 1 --2- 172.21.15.67:0/2723043089 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14e80ce610 0x7f14e8153660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:24.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.595+0000 7f14eeb1b640 1 -- 172.21.15.67:0/2723043089 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e81551c0 con 0x7f14e815e130 2024-03-24T00:43:24.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.596+0000 7f14edb19640 1 --2- 172.21.15.67:0/2723043089 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f14e80d0d50 0x7f14e8151d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:43:24.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.596+0000 7f14ee31a640 1 --2- 172.21.15.67:0/2723043089 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14e80ce610 0x7f14e8153660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:43:24.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.596+0000 7f14eeb1b640 1 --2- 172.21.15.67:0/2723043089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14e815e130 0x7f14e8154c80 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f14d802c3f0 tx=0x7f14d8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:24.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.596+0000 7f14e6ffd640 1 -- 172.21.15.67:0/2723043089 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d8039050 con 0x7f14e815e130 2024-03-24T00:43:24.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.596+0000 7f14e6ffd640 1 -- 172.21.15.67:0/2723043089 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14d8045070 con 0x7f14e815e130 2024-03-24T00:43:24.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.596+0000 7f14f057e640 1 -- 172.21.15.67:0/2723043089 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14e8153c50 con 0x7f14e815e130 2024-03-24T00:43:24.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.596+0000 7f14e6ffd640 1 -- 172.21.15.67:0/2723043089 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d8009450 con 0x7f14e815e130 2024-03-24T00:43:24.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.596+0000 7f14f057e640 1 -- 172.21.15.67:0/2723043089 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14e8154210 con 0x7f14e815e130 2024-03-24T00:43:24.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.597+0000 7f14f057e640 1 -- 172.21.15.67:0/2723043089 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14e804f400 con 0x7f14e815e130 2024-03-24T00:43:24.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.597+0000 7f14e6ffd640 1 -- 172.21.15.67:0/2723043089 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f14d8004650 con 0x7f14e815e130 2024-03-24T00:43:24.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.598+0000 7f14e6ffd640 1 --2- 172.21.15.67:0/2723043089 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f14d406bbd0 0x7f14d406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:24.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.598+0000 7f14e6ffd640 1 -- 172.21.15.67:0/2723043089 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f14d80ad270 con 0x7f14e815e130 2024-03-24T00:43:24.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.598+0000 7f14ee31a640 1 --2- 172.21.15.67:0/2723043089 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f14d406bbd0 0x7f14d406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:24.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.598+0000 7f14ee31a640 1 --2- 172.21.15.67:0/2723043089 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f14d406bbd0 0x7f14d406e090 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f14e00011e0 tx=0x7f14e0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:24.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.602+0000 7f14e6ffd640 1 -- 172.21.15.67:0/2723043089 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f14d804f020 con 0x7f14e815e130 2024-03-24T00:43:24.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:24.823+0000 7f14f057e640 1 -- 172.21.15.67:0/2723043089 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) v1 -- 0x7f14e80d1cc0 con 0x7f14e815e130 2024-03-24T00:43:25.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.376+0000 7f14e6ffd640 1 -- 172.21.15.67:0/2723043089 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v42) v1 ==== 162+0+0 (secure 0 0 0) 0x7f14d807bca0 con 0x7f14e815e130 2024-03-24T00:43:25.376 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked joinable; MDS may join as newly active. 2024-03-24T00:43:25.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.379+0000 7f14f057e640 1 -- 172.21.15.67:0/2723043089 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f14d406bbd0 msgr2=0x7f14d406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:25.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.379+0000 7f14f057e640 1 --2- 172.21.15.67:0/2723043089 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f14d406bbd0 0x7f14d406e090 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f14e00011e0 tx=0x7f14e0009040 comp rx=0 tx=0).stop 2024-03-24T00:43:25.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.379+0000 7f14f057e640 1 -- 172.21.15.67:0/2723043089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14e815e130 msgr2=0x7f14e8154c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:25.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.379+0000 7f14f057e640 1 --2- 172.21.15.67:0/2723043089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14e815e130 0x7f14e8154c80 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f14d802c3f0 tx=0x7f14d8004a40 comp rx=0 tx=0).stop 2024-03-24T00:43:25.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.379+0000 7f14f057e640 1 -- 172.21.15.67:0/2723043089 shutdown_connections 2024-03-24T00:43:25.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.379+0000 7f14f057e640 1 --2- 172.21.15.67:0/2723043089 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f14d406bbd0 0x7f14d406e090 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:25.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.379+0000 7f14f057e640 1 --2- 172.21.15.67:0/2723043089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14e815e130 0x7f14e8154c80 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:25.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.379+0000 7f14f057e640 1 --2- 172.21.15.67:0/2723043089 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f14e80d0d50 0x7f14e8151d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:25.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.379+0000 7f14f057e640 1 --2- 172.21.15.67:0/2723043089 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14e80ce610 0x7f14e8153660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:25.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.379+0000 7f14f057e640 1 -- 172.21.15.67:0/2723043089 >> 172.21.15.67:0/2723043089 conn(0x7f14e806c8a0 msgr2=0x7f14e80b8a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:25.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.379+0000 7f14f057e640 1 -- 172.21.15.67:0/2723043089 shutdown_connections 2024-03-24T00:43:25.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.379+0000 7f14f057e640 1 -- 172.21.15.67:0/2723043089 wait complete. 2024-03-24T00:43:25.393 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:43:25.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.498+0000 7fa13f9c4640 1 Processor -- start 2024-03-24T00:43:25.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.498+0000 7fa13f9c4640 1 -- start start 2024-03-24T00:43:25.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.499+0000 7fa13f9c4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa1381445a0 0x7fa1380cee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:25.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.499+0000 7fa13f9c4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa1380b48d0 0x7fa138072550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:25.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.499+0000 7fa13f9c4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa13806ec40 0x7fa13806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:25.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.499+0000 7fa13f9c4640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1380ce990 con 0x7fa13806ec40 2024-03-24T00:43:25.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.499+0000 7fa13f9c4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1380b4d30 con 0x7fa1380b48d0 2024-03-24T00:43:25.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.499+0000 7fa13f9c4640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1380cddc0 con 0x7fa1381445a0 2024-03-24T00:43:25.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.499+0000 7fa13df61640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa13806ec40 0x7fa13806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:25.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.499+0000 7fa13df61640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa13806ec40 0x7fa13806f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48290/0 (socket says 172.21.15.67:48290) 2024-03-24T00:43:25.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.499+0000 7fa13df61640 1 -- 172.21.15.67:0/3116034425 learned_addr learned my addr 172.21.15.67:0/3116034425 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:25.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.499+0000 7fa13cf5f640 1 --2- 172.21.15.67:0/3116034425 >> v2:172.21.15.72:3300/0 conn(0x7fa1380b48d0 0x7fa138072550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:25.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.499+0000 7fa13d760640 1 --2- 172.21.15.67:0/3116034425 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa1381445a0 0x7fa1380cee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:25.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.499+0000 7fa13df61640 1 -- 172.21.15.67:0/3116034425 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa1381445a0 msgr2=0x7fa1380cee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:25.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.499+0000 7fa13df61640 1 --2- 172.21.15.67:0/3116034425 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa1381445a0 0x7fa1380cee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:25.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.499+0000 7fa13df61640 1 -- 172.21.15.67:0/3116034425 >> v2:172.21.15.72:3300/0 conn(0x7fa1380b48d0 msgr2=0x7fa138072550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:25.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.499+0000 7fa13df61640 1 --2- 172.21.15.67:0/3116034425 >> v2:172.21.15.72:3300/0 conn(0x7fa1380b48d0 0x7fa138072550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:25.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.499+0000 7fa13df61640 1 -- 172.21.15.67:0/3116034425 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1380ce060 con 0x7fa13806ec40 2024-03-24T00:43:25.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.499+0000 7fa13cf5f640 1 --2- 172.21.15.67:0/3116034425 >> v2:172.21.15.72:3300/0 conn(0x7fa1380b48d0 0x7fa138072550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:43:25.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.500+0000 7fa13df61640 1 --2- 172.21.15.67:0/3116034425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa13806ec40 0x7fa13806f0d0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fa13806da80 tx=0x7fa13002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1721835a55e7a10a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:25.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.500+0000 7fa12ffff640 1 -- 172.21.15.67:0/3116034425 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa13002ca70 con 0x7fa13806ec40 2024-03-24T00:43:25.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.500+0000 7fa12ffff640 1 -- 172.21.15.67:0/3116034425 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa13002cc10 con 0x7fa13806ec40 2024-03-24T00:43:25.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.500+0000 7fa12ffff640 1 -- 172.21.15.67:0/3116034425 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1300343f0 con 0x7fa13806ec40 2024-03-24T00:43:25.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.500+0000 7fa13f9c4640 1 -- 172.21.15.67:0/3116034425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa13806ec40 msgr2=0x7fa13806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:25.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.500+0000 7fa13f9c4640 1 --2- 172.21.15.67:0/3116034425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa13806ec40 0x7fa13806f0d0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fa13806da80 tx=0x7fa13002bed0 comp rx=0 tx=0).stop 2024-03-24T00:43:25.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.500+0000 7fa13f9c4640 1 -- 172.21.15.67:0/3116034425 shutdown_connections 2024-03-24T00:43:25.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.500+0000 7fa13f9c4640 1 --2- 172.21.15.67:0/3116034425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa13806ec40 0x7fa13806f0d0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:25.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.500+0000 7fa13f9c4640 1 --2- 172.21.15.67:0/3116034425 >> v2:172.21.15.72:3300/0 conn(0x7fa1380b48d0 0x7fa138072550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:25.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.500+0000 7fa13f9c4640 1 --2- 172.21.15.67:0/3116034425 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa1381445a0 0x7fa1380cee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:25.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.500+0000 7fa13f9c4640 1 -- 172.21.15.67:0/3116034425 >> 172.21.15.67:0/3116034425 conn(0x7fa1380cf9e0 msgr2=0x7fa1380cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:25.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.501+0000 7fa13f9c4640 1 -- 172.21.15.67:0/3116034425 shutdown_connections 2024-03-24T00:43:25.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.501+0000 7fa13f9c4640 1 -- 172.21.15.67:0/3116034425 wait complete. 2024-03-24T00:43:25.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.502+0000 7fa13f9c4640 1 Processor -- start 2024-03-24T00:43:25.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.502+0000 7fa13f9c4640 1 -- start start 2024-03-24T00:43:25.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.502+0000 7fa13f9c4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa13806ec40 0x7fa13813a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:25.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.502+0000 7fa13f9c4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa1380b48d0 0x7fa138142430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:25.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.502+0000 7fa13f9c4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa1381445a0 0x7fa138142970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:25.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.502+0000 7fa13f9c4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1380b2a10 con 0x7fa13806ec40 2024-03-24T00:43:25.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.502+0000 7fa13f9c4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1380ce620 con 0x7fa1381445a0 2024-03-24T00:43:25.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.502+0000 7fa13f9c4640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1380b2b90 con 0x7fa1380b48d0 2024-03-24T00:43:25.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.503+0000 7fa13d760640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa13806ec40 0x7fa13813a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:25.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.503+0000 7fa13d760640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa13806ec40 0x7fa13813a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48292/0 (socket says 172.21.15.67:48292) 2024-03-24T00:43:25.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.503+0000 7fa13d760640 1 -- 172.21.15.67:0/3496312807 learned_addr learned my addr 172.21.15.67:0/3496312807 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:25.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.503+0000 7fa13df61640 1 --2- 172.21.15.67:0/3496312807 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa1381445a0 0x7fa138142970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:25.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.503+0000 7fa13d760640 1 -- 172.21.15.67:0/3496312807 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa1380b48d0 msgr2=0x7fa138142430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:25.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.503+0000 7fa13cf5f640 1 --2- 172.21.15.67:0/3496312807 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa1380b48d0 0x7fa138142430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:25.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.503+0000 7fa13d760640 1 --2- 172.21.15.67:0/3496312807 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa1380b48d0 0x7fa138142430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:25.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.503+0000 7fa13d760640 1 -- 172.21.15.67:0/3496312807 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa1381445a0 msgr2=0x7fa138142970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:25.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.503+0000 7fa13d760640 1 --2- 172.21.15.67:0/3496312807 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa1381445a0 0x7fa138142970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:25.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.503+0000 7fa13d760640 1 -- 172.21.15.67:0/3496312807 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa13815a8a0 con 0x7fa13806ec40 2024-03-24T00:43:25.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.503+0000 7fa13d760640 1 --2- 172.21.15.67:0/3496312807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa13806ec40 0x7fa13813a620 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fa13400d620 tx=0x7fa13400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:25.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.503+0000 7fa12dffb640 1 -- 172.21.15.67:0/3496312807 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa13400b620 con 0x7fa13806ec40 2024-03-24T00:43:25.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.503+0000 7fa12dffb640 1 -- 172.21.15.67:0/3496312807 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa134019070 con 0x7fa13806ec40 2024-03-24T00:43:25.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.503+0000 7fa13f9c4640 1 -- 172.21.15.67:0/3496312807 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa138141670 con 0x7fa13806ec40 2024-03-24T00:43:25.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.503+0000 7fa12dffb640 1 -- 172.21.15.67:0/3496312807 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa13400ba00 con 0x7fa13806ec40 2024-03-24T00:43:25.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.503+0000 7fa13f9c4640 1 -- 172.21.15.67:0/3496312807 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa138141c30 con 0x7fa13806ec40 2024-03-24T00:43:25.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.504+0000 7fa13f9c4640 1 -- 172.21.15.67:0/3496312807 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa13804f400 con 0x7fa13806ec40 2024-03-24T00:43:25.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.505+0000 7fa12dffb640 1 -- 172.21.15.67:0/3496312807 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa13400bba0 con 0x7fa13806ec40 2024-03-24T00:43:25.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.505+0000 7fa12dffb640 1 --2- 172.21.15.67:0/3496312807 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa11c06bbd0 0x7fa11c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:25.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.505+0000 7fa12dffb640 1 -- 172.21.15.67:0/3496312807 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fa13408afe0 con 0x7fa13806ec40 2024-03-24T00:43:25.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.505+0000 7fa13cf5f640 1 --2- 172.21.15.67:0/3496312807 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa11c06bbd0 0x7fa11c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:25.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.506+0000 7fa13cf5f640 1 --2- 172.21.15.67:0/3496312807 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa11c06bbd0 0x7fa11c06e090 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fa128006fd0 tx=0x7fa128008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:25.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.509+0000 7fa12dffb640 1 -- 172.21.15.67:0/3496312807 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa134059980 con 0x7fa13806ec40 2024-03-24T00:43:25.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.728+0000 7fa13f9c4640 1 -- 172.21.15.67:0/3496312807 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa13815ad10 con 0x7fa13806ec40 2024-03-24T00:43:25.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.728+0000 7fa12dffb640 1 -- 172.21.15.67:0/3496312807 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+4970 (secure 0 0 0) 0x7fa134059280 con 0x7fa13806ec40 2024-03-24T00:43:25.728 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:43:25.729 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":43,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5960,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1888378520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1888378520},{"type":"v1","addr":"172.21.15.67:6835","nonce":1888378520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6610,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/2539718386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2539718386},{"type":"v1","addr":"172.21.15.72:6836","nonce":2539718386}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6613,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2376370770","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2376370770},{"type":"v1","addr":"172.21.15.72:6837","nonce":2376370770}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:43:10.177088+0000","modified":"2024-03-24T00:43:25.379001+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7149},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7149":{"gid":7149,"name":"c","rank":0,"incarnation":43,"state":"up:replay","state_seq":1,"addr":"172.21.15.67:6833/720256752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":720256752},{"type":"v1","addr":"172.21.15.67:6833","nonce":720256752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":5}]} 2024-03-24T00:43:25.729 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 43 2024-03-24T00:43:25.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.730+0000 7fa13f9c4640 1 -- 172.21.15.67:0/3496312807 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa11c06bbd0 msgr2=0x7fa11c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:25.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.730+0000 7fa13f9c4640 1 --2- 172.21.15.67:0/3496312807 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa11c06bbd0 0x7fa11c06e090 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fa128006fd0 tx=0x7fa128008040 comp rx=0 tx=0).stop 2024-03-24T00:43:25.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.730+0000 7fa13f9c4640 1 -- 172.21.15.67:0/3496312807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa13806ec40 msgr2=0x7fa13813a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:25.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.730+0000 7fa13f9c4640 1 --2- 172.21.15.67:0/3496312807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa13806ec40 0x7fa13813a620 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fa13400d620 tx=0x7fa13400dae0 comp rx=0 tx=0).stop 2024-03-24T00:43:25.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.730+0000 7fa13f9c4640 1 -- 172.21.15.67:0/3496312807 shutdown_connections 2024-03-24T00:43:25.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.730+0000 7fa13f9c4640 1 --2- 172.21.15.67:0/3496312807 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa11c06bbd0 0x7fa11c06e090 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:25.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.730+0000 7fa13f9c4640 1 --2- 172.21.15.67:0/3496312807 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa1381445a0 0x7fa138142970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:25.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.730+0000 7fa13f9c4640 1 --2- 172.21.15.67:0/3496312807 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa1380b48d0 0x7fa138142430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:25.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.730+0000 7fa13f9c4640 1 --2- 172.21.15.67:0/3496312807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa13806ec40 0x7fa13813a620 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:25.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.730+0000 7fa13f9c4640 1 -- 172.21.15.67:0/3496312807 >> 172.21.15.67:0/3496312807 conn(0x7fa1380cf9e0 msgr2=0x7fa138071ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:25.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.730+0000 7fa13f9c4640 1 -- 172.21.15.67:0/3496312807 shutdown_connections 2024-03-24T00:43:25.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:25.730+0000 7fa13f9c4640 1 -- 172.21.15.67:0/3496312807 wait complete. 2024-03-24T00:43:26.144 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 43, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:43:10.177088+0000', 'modified': '2024-03-24T00:43:25.379001+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 77, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7149}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7149': {'gid': 7149, 'name': 'c', 'rank': 0, 'incarnation': 43, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.67:6833/720256752', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6832', 'nonce': 720256752}, {'type': 'v1', 'addr': '172.21.15.67:6833', 'nonce': 720256752}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-24T00:43:26.144 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7149:up:replay 2024-03-24T00:43:27.145 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:43:27.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.260+0000 7f6049a19640 1 Processor -- start 2024-03-24T00:43:27.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.260+0000 7f6049a19640 1 -- start start 2024-03-24T00:43:27.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.260+0000 7f6049a19640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f604415e1b0 0x7f60440cfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:27.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.260+0000 7f6049a19640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f60440b43a0 0x7f6044099d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:27.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.260+0000 7f6049a19640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f604409a650 con 0x7f604409a240 2024-03-24T00:43:27.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.260+0000 7f6049a19640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f604415e590 con 0x7f60440b43a0 2024-03-24T00:43:27.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.261+0000 7f6049a19640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60440d1820 con 0x7f604415e1b0 2024-03-24T00:43:27.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.261+0000 7f60427fc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f60440b43a0 0x7f6044099d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:27.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.261+0000 7f60427fc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f60440b43a0 0x7f6044099d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48300/0 (socket says 172.21.15.67:48300) 2024-03-24T00:43:27.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.261+0000 7f60427fc640 1 -- 172.21.15.67:0/1335452763 learned_addr learned my addr 172.21.15.67:0/1335452763 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:27.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.261+0000 7f6042ffd640 1 --2- 172.21.15.67:0/1335452763 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f604415e1b0 0x7f60440cfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:27.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.261+0000 7f60427fc640 1 -- 172.21.15.67:0/1335452763 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f604415e1b0 msgr2=0x7f60440cfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:27.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.261+0000 7f60427fc640 1 --2- 172.21.15.67:0/1335452763 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f604415e1b0 0x7f60440cfe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:27.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.261+0000 7f60427fc640 1 -- 172.21.15.67:0/1335452763 >> v1:172.21.15.72:6789/0 conn(0x7f604409a240 legacy=0x7f604406c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:27.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.261+0000 7f60427fc640 1 -- 172.21.15.67:0/1335452763 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60440d0fc0 con 0x7f60440b43a0 2024-03-24T00:43:27.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.262+0000 7f60427fc640 1 --2- 172.21.15.67:0/1335452763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60440b43a0 0x7f6044099d00 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f6038009870 tx=0x7f603802b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9215e5b96f1eaa1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:27.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.262+0000 7f6041ffb640 1 -- 172.21.15.67:0/1335452763 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6038038070 con 0x7f60440b43a0 2024-03-24T00:43:27.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.262+0000 7f6041ffb640 1 -- 172.21.15.67:0/1335452763 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6038031740 con 0x7f60440b43a0 2024-03-24T00:43:27.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.262+0000 7f6041ffb640 1 -- 172.21.15.67:0/1335452763 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f603802f050 con 0x7f60440b43a0 2024-03-24T00:43:27.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.262+0000 7f6049a19640 1 -- 172.21.15.67:0/1335452763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60440b43a0 msgr2=0x7f6044099d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:27.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.262+0000 7f6049a19640 1 --2- 172.21.15.67:0/1335452763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60440b43a0 0x7f6044099d00 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f6038009870 tx=0x7f603802b7e0 comp rx=0 tx=0).stop 2024-03-24T00:43:27.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.262+0000 7f6049a19640 1 -- 172.21.15.67:0/1335452763 shutdown_connections 2024-03-24T00:43:27.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.262+0000 7f6049a19640 1 --2- 172.21.15.67:0/1335452763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60440b43a0 0x7f6044099d00 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:27.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.262+0000 7f6049a19640 1 --2- 172.21.15.67:0/1335452763 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f604415e1b0 0x7f60440cfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:27.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.262+0000 7f6049a19640 1 -- 172.21.15.67:0/1335452763 >> 172.21.15.67:0/1335452763 conn(0x7f60440b36d0 msgr2=0x7f60440b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:27.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.263+0000 7f6049a19640 1 -- 172.21.15.67:0/1335452763 shutdown_connections 2024-03-24T00:43:27.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.263+0000 7f6049a19640 1 -- 172.21.15.67:0/1335452763 wait complete. 2024-03-24T00:43:27.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.264+0000 7f6049a19640 1 Processor -- start 2024-03-24T00:43:27.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.264+0000 7f6049a19640 1 -- start start 2024-03-24T00:43:27.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.265+0000 7f6049a19640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f604409a240 0x7f604414fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:27.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.265+0000 7f6049a19640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f60440b43a0 0x7f60440b7060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:27.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.265+0000 7f6049a19640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f604415e1b0 0x7f60440cae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:27.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.265+0000 7f6049a19640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60440cc360 con 0x7f604415e1b0 2024-03-24T00:43:27.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.265+0000 7f6049a19640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60440d1540 con 0x7f604409a240 2024-03-24T00:43:27.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.265+0000 7f6049a19640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f604409a9a0 con 0x7f60440b43a0 2024-03-24T00:43:27.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.265+0000 7f60437fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f604415e1b0 0x7f60440cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:27.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.265+0000 7f60437fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f604415e1b0 0x7f60440cae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48312/0 (socket says 172.21.15.67:48312) 2024-03-24T00:43:27.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.265+0000 7f60437fe640 1 -- 172.21.15.67:0/1517861042 learned_addr learned my addr 172.21.15.67:0/1517861042 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:27.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.265+0000 7f60427fc640 1 --2- 172.21.15.67:0/1517861042 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f60440b43a0 0x7f60440b7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:27.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.265+0000 7f6042ffd640 1 --2- 172.21.15.67:0/1517861042 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f604409a240 0x7f604414fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:27.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.265+0000 7f60437fe640 1 -- 172.21.15.67:0/1517861042 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f60440b43a0 msgr2=0x7f60440b7060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:27.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.265+0000 7f60437fe640 1 --2- 172.21.15.67:0/1517861042 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f60440b43a0 0x7f60440b7060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:27.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.265+0000 7f60437fe640 1 -- 172.21.15.67:0/1517861042 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f604409a240 msgr2=0x7f604414fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:27.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.265+0000 7f60437fe640 1 --2- 172.21.15.67:0/1517861042 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f604409a240 0x7f604414fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:27.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.265+0000 7f60437fe640 1 -- 172.21.15.67:0/1517861042 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f604415e960 con 0x7f604415e1b0 2024-03-24T00:43:27.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.265+0000 7f60437fe640 1 --2- 172.21.15.67:0/1517861042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f604415e1b0 0x7f60440cae80 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f603c00b970 tx=0x7f603c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:27.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.265+0000 7f602bfff640 1 -- 172.21.15.67:0/1517861042 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f603c019070 con 0x7f604415e1b0 2024-03-24T00:43:27.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.265+0000 7f602bfff640 1 -- 172.21.15.67:0/1517861042 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f603c00cb40 con 0x7f604415e1b0 2024-03-24T00:43:27.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.265+0000 7f6049a19640 1 -- 172.21.15.67:0/1517861042 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f604415ec40 con 0x7f604415e1b0 2024-03-24T00:43:27.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.265+0000 7f602bfff640 1 -- 172.21.15.67:0/1517861042 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f603c014440 con 0x7f604415e1b0 2024-03-24T00:43:27.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.266+0000 7f6049a19640 1 -- 172.21.15.67:0/1517861042 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f604415f200 con 0x7f604415e1b0 2024-03-24T00:43:27.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.266+0000 7f6049a19640 1 -- 172.21.15.67:0/1517861042 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f604404f400 con 0x7f604415e1b0 2024-03-24T00:43:27.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.267+0000 7f602bfff640 1 -- 172.21.15.67:0/1517861042 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f603c01c070 con 0x7f604415e1b0 2024-03-24T00:43:27.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.267+0000 7f602bfff640 1 --2- 172.21.15.67:0/1517861042 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f603006bbd0 0x7f603006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:27.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.267+0000 7f602bfff640 1 -- 172.21.15.67:0/1517861042 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f603c08a8b0 con 0x7f604415e1b0 2024-03-24T00:43:27.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.267+0000 7f6042ffd640 1 --2- 172.21.15.67:0/1517861042 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f603006bbd0 0x7f603006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:27.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.268+0000 7f6042ffd640 1 --2- 172.21.15.67:0/1517861042 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f603006bbd0 0x7f603006e090 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f6034006fd0 tx=0x7f6034008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:27.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.272+0000 7f602bfff640 1 -- 172.21.15.67:0/1517861042 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f603c0591c0 con 0x7f604415e1b0 2024-03-24T00:43:27.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.495+0000 7f6049a19640 1 -- 172.21.15.67:0/1517861042 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f604406ced0 con 0x7f604415e1b0 2024-03-24T00:43:27.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.495+0000 7f602bfff640 1 -- 172.21.15.67:0/1517861042 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4970 (secure 0 0 0) 0x7f603c058ac0 con 0x7f604415e1b0 2024-03-24T00:43:27.495 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:43:27.495 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5960,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1888378520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1888378520},{"type":"v1","addr":"172.21.15.67:6835","nonce":1888378520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6610,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/2539718386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2539718386},{"type":"v1","addr":"172.21.15.72:6836","nonce":2539718386}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6613,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2376370770","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2376370770},{"type":"v1","addr":"172.21.15.72:6837","nonce":2376370770}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:43:10.177088+0000","modified":"2024-03-24T00:43:26.385931+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7149},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7149":{"gid":7149,"name":"c","rank":0,"incarnation":43,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.67:6833/720256752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":720256752},{"type":"v1","addr":"172.21.15.67:6833","nonce":720256752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":5}]} 2024-03-24T00:43:27.495 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 45 2024-03-24T00:43:27.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.497+0000 7f6049a19640 1 -- 172.21.15.67:0/1517861042 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f603006bbd0 msgr2=0x7f603006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:27.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.497+0000 7f6049a19640 1 --2- 172.21.15.67:0/1517861042 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f603006bbd0 0x7f603006e090 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f6034006fd0 tx=0x7f6034008040 comp rx=0 tx=0).stop 2024-03-24T00:43:27.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.497+0000 7f6049a19640 1 -- 172.21.15.67:0/1517861042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f604415e1b0 msgr2=0x7f60440cae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:27.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.497+0000 7f6049a19640 1 --2- 172.21.15.67:0/1517861042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f604415e1b0 0x7f60440cae80 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f603c00b970 tx=0x7f603c00be30 comp rx=0 tx=0).stop 2024-03-24T00:43:27.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.497+0000 7f6049a19640 1 -- 172.21.15.67:0/1517861042 shutdown_connections 2024-03-24T00:43:27.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.497+0000 7f6049a19640 1 --2- 172.21.15.67:0/1517861042 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f603006bbd0 0x7f603006e090 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:27.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.497+0000 7f6049a19640 1 --2- 172.21.15.67:0/1517861042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f604415e1b0 0x7f60440cae80 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:27.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.497+0000 7f6049a19640 1 --2- 172.21.15.67:0/1517861042 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f60440b43a0 0x7f60440b7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:27.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.497+0000 7f6049a19640 1 --2- 172.21.15.67:0/1517861042 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f604409a240 0x7f604414fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:27.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.497+0000 7f6049a19640 1 -- 172.21.15.67:0/1517861042 >> 172.21.15.67:0/1517861042 conn(0x7f60440b36d0 msgr2=0x7f60440703d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:27.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.497+0000 7f6049a19640 1 -- 172.21.15.67:0/1517861042 shutdown_connections 2024-03-24T00:43:27.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:27.497+0000 7f6049a19640 1 -- 172.21.15.67:0/1517861042 wait complete. 2024-03-24T00:43:27.910 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 45, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:43:10.177088+0000', 'modified': '2024-03-24T00:43:26.385931+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 77, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7149}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7149': {'gid': 7149, 'name': 'c', 'rank': 0, 'incarnation': 43, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.67:6833/720256752', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6832', 'nonce': 720256752}, {'type': 'v1', 'addr': '172.21.15.67:6833', 'nonce': 720256752}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-24T00:43:27.910 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7149:up:rejoin 2024-03-24T00:43:28.911 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:43:29.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.021+0000 7f2ef759e640 1 Processor -- start 2024-03-24T00:43:29.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.021+0000 7f2ef759e640 1 -- start start 2024-03-24T00:43:29.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.021+0000 7f2ef759e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2ef815aff0 0x7f2ef815bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:29.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.021+0000 7f2ef759e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ef815f1e0 con 0x7f2ef80d05f0 2024-03-24T00:43:29.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.021+0000 7f2ef759e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ef815f4c0 con 0x7f2ef815e790 2024-03-24T00:43:29.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.021+0000 7f2ef759e640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef815b4e0 con 0x7f2ef815aff0 2024-03-24T00:43:29.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.022+0000 7f2ef659c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2ef80d05f0 0x7f2ef80d09d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58278/0 (socket says 172.21.15.67:58278) 2024-03-24T00:43:29.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.022+0000 7f2ef659c640 1 -- 172.21.15.67:0/1825358284 learned_addr learned my addr 172.21.15.67:0/1825358284 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:43:29.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.022+0000 7f2eedd9b640 1 --2- 172.21.15.67:0/1825358284 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2ef815aff0 0x7f2ef815bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:29.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.022+0000 7f2ef5d9b640 1 -- 172.21.15.67:0/1825358284 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4038870341 0 0) 0x7f2ef815f1e0 con 0x7f2ef80d05f0 2024-03-24T00:43:29.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.022+0000 7f2ef5d9b640 1 -- 172.21.15.67:0/1825358284 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ee4005730 con 0x7f2ef80d05f0 2024-03-24T00:43:29.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.022+0000 7f2ef5d9b640 1 -- 172.21.15.67:0/1825358284 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3338567343 0 0) 0x7f2ef815f4c0 con 0x7f2ef815e790 2024-03-24T00:43:29.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.022+0000 7f2ef5d9b640 1 -- 172.21.15.67:0/1825358284 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ef815f1e0 con 0x7f2ef815e790 2024-03-24T00:43:29.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.022+0000 7f2eedd9b640 1 -- 172.21.15.67:0/1825358284 >> v1:172.21.15.72:6789/0 conn(0x7f2ef815e790 legacy=0x7f2ef80cb450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:29.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.022+0000 7f2ef5d9b640 1 -- 172.21.15.67:0/1825358284 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3058985106 0 0) 0x7f2ee4005730 con 0x7f2ef80d05f0 2024-03-24T00:43:29.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.022+0000 7f2eedd9b640 1 -- 172.21.15.67:0/1825358284 >> v1:172.21.15.67:6789/0 conn(0x7f2ef80d05f0 legacy=0x7f2ef80d09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:29.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.022+0000 7f2eedd9b640 1 -- 172.21.15.67:0/1825358284 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef815b7c0 con 0x7f2ef815aff0 2024-03-24T00:43:29.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.023+0000 7f2eedd9b640 1 --2- 172.21.15.67:0/1825358284 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2ef815aff0 0x7f2ef815bce0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2ee00097d0 tx=0x7f2ee002b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6345e95ff8a343d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:29.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.023+0000 7f2ef5d9b640 1 -- 172.21.15.67:0/1825358284 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee0037070 con 0x7f2ef815aff0 2024-03-24T00:43:29.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.023+0000 7f2ef5d9b640 1 -- 172.21.15.67:0/1825358284 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ee0030510 con 0x7f2ef815aff0 2024-03-24T00:43:29.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.023+0000 7f2ef5d9b640 1 -- 172.21.15.67:0/1825358284 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee8001670 con 0x7f2ef815aff0 2024-03-24T00:43:29.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.024+0000 7f2ef759e640 1 -- 172.21.15.67:0/1825358284 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2ef815aff0 msgr2=0x7f2ef815bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:29.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.024+0000 7f2ef759e640 1 --2- 172.21.15.67:0/1825358284 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2ef815aff0 0x7f2ef815bce0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2ee00097d0 tx=0x7f2ee002b800 comp rx=0 tx=0).stop 2024-03-24T00:43:29.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.024+0000 7f2ef759e640 1 -- 172.21.15.67:0/1825358284 shutdown_connections 2024-03-24T00:43:29.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.024+0000 7f2ef759e640 1 --2- 172.21.15.67:0/1825358284 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2ef815aff0 0x7f2ef815bce0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:29.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.024+0000 7f2ef759e640 1 -- 172.21.15.67:0/1825358284 >> 172.21.15.67:0/1825358284 conn(0x7f2ef8099a30 msgr2=0x7f2ef8099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:29.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.024+0000 7f2ef759e640 1 -- 172.21.15.67:0/1825358284 shutdown_connections 2024-03-24T00:43:29.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.024+0000 7f2ef759e640 1 -- 172.21.15.67:0/1825358284 wait complete. 2024-03-24T00:43:29.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.026+0000 7f2ef759e640 1 Processor -- start 2024-03-24T00:43:29.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.026+0000 7f2ef759e640 1 -- start start 2024-03-24T00:43:29.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.027+0000 7f2ef759e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2ef80d05f0 0x7f2ef814bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:29.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.027+0000 7f2ef759e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ef815aff0 0x7f2ef806b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:29.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.027+0000 7f2ef759e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2ef815e790 0x7f2ef806b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:29.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.027+0000 7f2ef759e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef815ed60 con 0x7f2ef815aff0 2024-03-24T00:43:29.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.027+0000 7f2ef759e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef815ebe0 con 0x7f2ef80d05f0 2024-03-24T00:43:29.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.027+0000 7f2ef759e640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef815eee0 con 0x7f2ef815e790 2024-03-24T00:43:29.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.027+0000 7f2eedd9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ef815aff0 0x7f2ef806b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:29.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.027+0000 7f2eedd9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ef815aff0 0x7f2ef806b040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48326/0 (socket says 172.21.15.67:48326) 2024-03-24T00:43:29.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.027+0000 7f2eedd9b640 1 -- 172.21.15.67:0/1556321036 learned_addr learned my addr 172.21.15.67:0/1556321036 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:29.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.028+0000 7f2ef659c640 1 --2- 172.21.15.67:0/1556321036 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2ef80d05f0 0x7f2ef814bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:29.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.028+0000 7f2ef6d9d640 1 --2- 172.21.15.67:0/1556321036 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2ef815e790 0x7f2ef806b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:29.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.028+0000 7f2eedd9b640 1 -- 172.21.15.67:0/1556321036 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2ef815e790 msgr2=0x7f2ef806b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:29.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.028+0000 7f2eedd9b640 1 --2- 172.21.15.67:0/1556321036 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2ef815e790 0x7f2ef806b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:29.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.028+0000 7f2eedd9b640 1 -- 172.21.15.67:0/1556321036 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2ef80d05f0 msgr2=0x7f2ef814bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:29.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.028+0000 7f2eedd9b640 1 --2- 172.21.15.67:0/1556321036 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2ef80d05f0 0x7f2ef814bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:29.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.028+0000 7f2eedd9b640 1 -- 172.21.15.67:0/1556321036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef814c210 con 0x7f2ef815aff0 2024-03-24T00:43:29.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.028+0000 7f2eedd9b640 1 --2- 172.21.15.67:0/1556321036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ef815aff0 0x7f2ef806b040 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f2ee0000c00 tx=0x7f2ee00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:29.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.028+0000 7f2eef7fe640 1 -- 172.21.15.67:0/1556321036 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee0037040 con 0x7f2ef815aff0 2024-03-24T00:43:29.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.028+0000 7f2eef7fe640 1 -- 172.21.15.67:0/1556321036 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ee0004250 con 0x7f2ef815aff0 2024-03-24T00:43:29.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.028+0000 7f2ef759e640 1 -- 172.21.15.67:0/1556321036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ef80c4580 con 0x7f2ef815aff0 2024-03-24T00:43:29.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.028+0000 7f2eef7fe640 1 -- 172.21.15.67:0/1556321036 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee003c670 con 0x7f2ef815aff0 2024-03-24T00:43:29.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.028+0000 7f2ef759e640 1 -- 172.21.15.67:0/1556321036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ef80c4b40 con 0x7f2ef815aff0 2024-03-24T00:43:29.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.029+0000 7f2ef759e640 1 -- 172.21.15.67:0/1556321036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ef804f400 con 0x7f2ef815aff0 2024-03-24T00:43:29.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.030+0000 7f2eef7fe640 1 -- 172.21.15.67:0/1556321036 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2ee003a070 con 0x7f2ef815aff0 2024-03-24T00:43:29.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.030+0000 7f2eef7fe640 1 --2- 172.21.15.67:0/1556321036 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2ed806bc00 0x7f2ed806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:29.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.030+0000 7f2eef7fe640 1 -- 172.21.15.67:0/1556321036 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f2ee00ac580 con 0x7f2ef815aff0 2024-03-24T00:43:29.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.030+0000 7f2ef659c640 1 --2- 172.21.15.67:0/1556321036 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2ed806bc00 0x7f2ed806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:29.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.031+0000 7f2ef659c640 1 --2- 172.21.15.67:0/1556321036 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2ed806bc00 0x7f2ed806e0c0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f2ee8000fd0 tx=0x7f2ee8001150 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:29.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.034+0000 7f2eef7fe640 1 -- 172.21.15.67:0/1556321036 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2ee0041020 con 0x7f2ef815aff0 2024-03-24T00:43:29.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.363+0000 7f2ef759e640 1 -- 172.21.15.67:0/1556321036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ef806f190 con 0x7f2ef815aff0 2024-03-24T00:43:29.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.364+0000 7f2eef7fe640 1 -- 172.21.15.67:0/1556321036 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f2ee007a7b0 con 0x7f2ef815aff0 2024-03-24T00:43:29.364 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:43:29.364 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":46,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5960,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1888378520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1888378520},{"type":"v1","addr":"172.21.15.67:6835","nonce":1888378520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6610,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/2539718386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2539718386},{"type":"v1","addr":"172.21.15.72:6836","nonce":2539718386}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6613,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2376370770","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2376370770},{"type":"v1","addr":"172.21.15.72:6837","nonce":2376370770}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:43:10.177088+0000","modified":"2024-03-24T00:43:28.389623+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7149},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7149":{"gid":7149,"name":"c","rank":0,"incarnation":43,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6833/720256752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":720256752},{"type":"v1","addr":"172.21.15.67:6833","nonce":720256752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7149,"qdb_cluster":[7149]},"id":5}]} 2024-03-24T00:43:29.364 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 46 2024-03-24T00:43:29.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.367+0000 7f2ef759e640 1 -- 172.21.15.67:0/1556321036 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2ed806bc00 msgr2=0x7f2ed806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:29.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.367+0000 7f2ef759e640 1 --2- 172.21.15.67:0/1556321036 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2ed806bc00 0x7f2ed806e0c0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f2ee8000fd0 tx=0x7f2ee8001150 comp rx=0 tx=0).stop 2024-03-24T00:43:29.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.367+0000 7f2ef759e640 1 -- 172.21.15.67:0/1556321036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ef815aff0 msgr2=0x7f2ef806b040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:29.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.367+0000 7f2ef759e640 1 --2- 172.21.15.67:0/1556321036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ef815aff0 0x7f2ef806b040 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f2ee0000c00 tx=0x7f2ee00046c0 comp rx=0 tx=0).stop 2024-03-24T00:43:29.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.367+0000 7f2ef759e640 1 -- 172.21.15.67:0/1556321036 shutdown_connections 2024-03-24T00:43:29.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.367+0000 7f2ef759e640 1 --2- 172.21.15.67:0/1556321036 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2ed806bc00 0x7f2ed806e0c0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:29.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.367+0000 7f2ef759e640 1 --2- 172.21.15.67:0/1556321036 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2ef815e790 0x7f2ef806b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:29.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.367+0000 7f2ef759e640 1 --2- 172.21.15.67:0/1556321036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ef815aff0 0x7f2ef806b040 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:29.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.367+0000 7f2ef759e640 1 --2- 172.21.15.67:0/1556321036 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2ef80d05f0 0x7f2ef814bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:29.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.367+0000 7f2ef759e640 1 -- 172.21.15.67:0/1556321036 >> 172.21.15.67:0/1556321036 conn(0x7f2ef8099a30 msgr2=0x7f2ef806f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:29.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.367+0000 7f2ef759e640 1 -- 172.21.15.67:0/1556321036 shutdown_connections 2024-03-24T00:43:29.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.367+0000 7f2ef759e640 1 -- 172.21.15.67:0/1556321036 wait complete. 2024-03-24T00:43:29.780 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 46, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:43:10.177088+0000', 'modified': '2024-03-24T00:43:28.389623+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 77, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7149}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7149': {'gid': 7149, 'name': 'c', 'rank': 0, 'incarnation': 43, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.67:6833/720256752', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6832', 'nonce': 720256752}, {'type': 'v1', 'addr': '172.21.15.67:6833', 'nonce': 720256752}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7149, 'qdb_cluster': [7149]} 2024-03-24T00:43:29.792 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-24T00:43:29.792 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-24T00:43:29.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.896+0000 7f02237f4640 1 Processor -- start 2024-03-24T00:43:29.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.896+0000 7f02237f4640 1 -- start start 2024-03-24T00:43:29.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.897+0000 7f02237f4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f021c0d07a0 0x7f021c0cc770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:29.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.897+0000 7f02237f4640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f021c0cd160 con 0x7f021c0cccb0 2024-03-24T00:43:29.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.897+0000 7f02237f4640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f021c0cd430 con 0x7f021c15c010 2024-03-24T00:43:29.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.897+0000 7f02237f4640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f021c0cfbe0 con 0x7f021c0d07a0 2024-03-24T00:43:29.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.897+0000 7f0220d8f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f021c0d07a0 0x7f021c0cc770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:29.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.897+0000 7f0221d91640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f021c0cccb0 0x7f021c0b2790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58290/0 (socket says 172.21.15.67:58290) 2024-03-24T00:43:29.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.897+0000 7f0220d8f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f021c0d07a0 0x7f021c0cc770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48340/0 (socket says 172.21.15.67:48340) 2024-03-24T00:43:29.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.897+0000 7f0221d91640 1 -- 172.21.15.67:0/688202506 learned_addr learned my addr 172.21.15.67:0/688202506 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:43:29.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.898+0000 7f0220d8f640 1 -- 172.21.15.67:0/688202506 >> v1:172.21.15.72:6789/0 conn(0x7f021c15c010 legacy=0x7f021c0d0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:29.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.898+0000 7f0220d8f640 1 -- 172.21.15.67:0/688202506 >> v1:172.21.15.67:6789/0 conn(0x7f021c0cccb0 legacy=0x7f021c0b2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:29.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.898+0000 7f0220d8f640 1 -- 172.21.15.67:0/688202506 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f021c0cd710 con 0x7f021c0d07a0 2024-03-24T00:43:29.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.898+0000 7f0213fff640 1 -- 172.21.15.67:0/688202506 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3195528434 0 0) 0x7f021c0cd160 con 0x7f021c0cccb0 2024-03-24T00:43:29.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.898+0000 7f0220d8f640 1 --2- 172.21.15.67:0/688202506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f021c0d07a0 0x7f021c0cc770 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f020c0097a0 tx=0x7f020c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35a10c48e9d7a269 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:29.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.898+0000 7f0213fff640 1 -- 172.21.15.67:0/688202506 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f020c035020 con 0x7f021c0d07a0 2024-03-24T00:43:29.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.898+0000 7f0213fff640 1 -- 172.21.15.67:0/688202506 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f020c031830 con 0x7f021c0d07a0 2024-03-24T00:43:29.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.899+0000 7f0213fff640 1 -- 172.21.15.67:0/688202506 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f020c031c00 con 0x7f021c0d07a0 2024-03-24T00:43:29.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.899+0000 7f02237f4640 1 -- 172.21.15.67:0/688202506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f021c0d07a0 msgr2=0x7f021c0cc770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:29.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.899+0000 7f02237f4640 1 --2- 172.21.15.67:0/688202506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f021c0d07a0 0x7f021c0cc770 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f020c0097a0 tx=0x7f020c02b8d0 comp rx=0 tx=0).stop 2024-03-24T00:43:29.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.899+0000 7f02237f4640 1 -- 172.21.15.67:0/688202506 shutdown_connections 2024-03-24T00:43:29.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.899+0000 7f02237f4640 1 --2- 172.21.15.67:0/688202506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f021c0d07a0 0x7f021c0cc770 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:29.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.899+0000 7f02237f4640 1 -- 172.21.15.67:0/688202506 >> 172.21.15.67:0/688202506 conn(0x7f021c09aea0 msgr2=0x7f021c08b180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:29.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.899+0000 7f02237f4640 1 -- 172.21.15.67:0/688202506 shutdown_connections 2024-03-24T00:43:29.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.899+0000 7f02237f4640 1 -- 172.21.15.67:0/688202506 wait complete. 2024-03-24T00:43:29.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.901+0000 7f02237f4640 1 Processor -- start 2024-03-24T00:43:29.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.901+0000 7f02237f4640 1 -- start start 2024-03-24T00:43:29.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.902+0000 7f02237f4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f021c0cccb0 0x7f021c156c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:29.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.902+0000 7f02237f4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f021c0d07a0 0x7f021c15cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:29.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.902+0000 7f02237f4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f021c15c010 0x7f021c06fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:29.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.902+0000 7f02237f4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f021c0b2300 con 0x7f021c15c010 2024-03-24T00:43:29.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.902+0000 7f02237f4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f021c0b2020 con 0x7f021c0cccb0 2024-03-24T00:43:29.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.902+0000 7f02237f4640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f021c0b2ea0 con 0x7f021c0d07a0 2024-03-24T00:43:29.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.902+0000 7f0221d91640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f021c15c010 0x7f021c06fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:29.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.902+0000 7f0221d91640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f021c15c010 0x7f021c06fe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48342/0 (socket says 172.21.15.67:48342) 2024-03-24T00:43:29.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.902+0000 7f0221d91640 1 -- 172.21.15.67:0/3753645417 learned_addr learned my addr 172.21.15.67:0/3753645417 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:29.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.902+0000 7f0221590640 1 --2- 172.21.15.67:0/3753645417 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f021c0cccb0 0x7f021c156c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:29.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.902+0000 7f0220d8f640 1 --2- 172.21.15.67:0/3753645417 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f021c0d07a0 0x7f021c15cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:29.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.903+0000 7f0221d91640 1 -- 172.21.15.67:0/3753645417 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f021c0d07a0 msgr2=0x7f021c15cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:29.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.903+0000 7f0221d91640 1 --2- 172.21.15.67:0/3753645417 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f021c0d07a0 0x7f021c15cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:29.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.903+0000 7f0221d91640 1 -- 172.21.15.67:0/3753645417 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f021c0cccb0 msgr2=0x7f021c156c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:29.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.903+0000 7f0221d91640 1 --2- 172.21.15.67:0/3753645417 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f021c0cccb0 0x7f021c156c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:29.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.903+0000 7f0221d91640 1 -- 172.21.15.67:0/3753645417 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f021c157190 con 0x7f021c15c010 2024-03-24T00:43:29.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.903+0000 7f0221d91640 1 --2- 172.21.15.67:0/3753645417 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f021c15c010 0x7f021c06fe40 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f021400c7a0 tx=0x7f021400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:29.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.903+0000 7f0211ffb640 1 -- 172.21.15.67:0/3753645417 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0214018070 con 0x7f021c15c010 2024-03-24T00:43:29.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.903+0000 7f0211ffb640 1 -- 172.21.15.67:0/3753645417 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f021400a640 con 0x7f021c15c010 2024-03-24T00:43:29.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.903+0000 7f02237f4640 1 -- 172.21.15.67:0/3753645417 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f021c15d840 con 0x7f021c15c010 2024-03-24T00:43:29.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.903+0000 7f0211ffb640 1 -- 172.21.15.67:0/3753645417 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f021400aab0 con 0x7f021c15c010 2024-03-24T00:43:29.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.903+0000 7f02237f4640 1 -- 172.21.15.67:0/3753645417 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f021c0cb2d0 con 0x7f021c15c010 2024-03-24T00:43:29.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.905+0000 7f0211ffb640 1 -- 172.21.15.67:0/3753645417 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f021401b070 con 0x7f021c15c010 2024-03-24T00:43:29.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.905+0000 7f0211ffb640 1 --2- 172.21.15.67:0/3753645417 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f020006bbd0 0x7f020006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:29.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.905+0000 7f0211ffb640 1 -- 172.21.15.67:0/3753645417 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f021408a130 con 0x7f021c15c010 2024-03-24T00:43:29.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.905+0000 7f0221590640 1 --2- 172.21.15.67:0/3753645417 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f020006bbd0 0x7f020006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:29.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.906+0000 7f0221590640 1 --2- 172.21.15.67:0/3753645417 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f020006bbd0 0x7f020006e090 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f0218006fd0 tx=0x7f0218009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:29.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.907+0000 7f02263dd740 1 Processor -- start 2024-03-24T00:43:29.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.907+0000 7f02263dd740 1 -- start start 2024-03-24T00:43:29.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.908+0000 7f02263dd740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55faf983dc40 0x55faf9864770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:29.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.908+0000 7f02263dd740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55faf9864cb0 0x55faf983cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:29.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.908+0000 7f02263dd740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55faf983e020 0x55faf983d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:29.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.908+0000 7f02263dd740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faf97a1c20 con 0x55faf983dc40 2024-03-24T00:43:29.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.908+0000 7f02263dd740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faf974aa40 con 0x55faf983e020 2024-03-24T00:43:29.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.908+0000 7f02263dd740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55faf979dcc0 con 0x55faf9864cb0 2024-03-24T00:43:29.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.908+0000 7f0221590640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55faf983dc40 0x55faf9864770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:29.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.908+0000 7f0221590640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55faf983dc40 0x55faf9864770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48354/0 (socket says 172.21.15.67:48354) 2024-03-24T00:43:29.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.908+0000 7f0221590640 1 -- 172.21.15.67:0/2844196588 learned_addr learned my addr 172.21.15.67:0/2844196588 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:29.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.908+0000 7f0220d8f640 1 --2- 172.21.15.67:0/2844196588 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55faf9864cb0 0x55faf983cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:29.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.908+0000 7f0221d91640 1 --2- 172.21.15.67:0/2844196588 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55faf983e020 0x55faf983d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:29.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.909+0000 7f0221590640 1 -- 172.21.15.67:0/2844196588 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55faf9864cb0 msgr2=0x55faf983cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:29.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.909+0000 7f0221590640 1 --2- 172.21.15.67:0/2844196588 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55faf9864cb0 0x55faf983cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:29.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.909+0000 7f0221590640 1 -- 172.21.15.67:0/2844196588 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55faf983e020 msgr2=0x55faf983d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:29.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.909+0000 7f0221590640 1 --2- 172.21.15.67:0/2844196588 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55faf983e020 0x55faf983d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:29.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.909+0000 7f0221590640 1 -- 172.21.15.67:0/2844196588 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55faf976cf90 con 0x55faf983dc40 2024-03-24T00:43:29.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.909+0000 7f0221590640 1 --2- 172.21.15.67:0/2844196588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55faf983dc40 0x55faf9864770 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f021800eb70 tx=0x7f0218011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=867d0dd4de3c126b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:29.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.909+0000 7f0206ffd640 1 -- 172.21.15.67:0/2844196588 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f021801b040 con 0x55faf983dc40 2024-03-24T00:43:29.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.909+0000 7f0206ffd640 1 -- 172.21.15.67:0/2844196588 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0218024070 con 0x55faf983dc40 2024-03-24T00:43:29.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.909+0000 7f0206ffd640 1 -- 172.21.15.67:0/2844196588 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0218015d00 con 0x55faf983dc40 2024-03-24T00:43:29.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.909+0000 7f02263dd740 1 -- 172.21.15.67:0/2844196588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55faf983dc40 msgr2=0x55faf9864770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:29.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.909+0000 7f02263dd740 1 --2- 172.21.15.67:0/2844196588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55faf983dc40 0x55faf9864770 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f021800eb70 tx=0x7f0218011000 comp rx=0 tx=0).stop 2024-03-24T00:43:29.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.910+0000 7f02263dd740 1 -- 172.21.15.67:0/2844196588 shutdown_connections 2024-03-24T00:43:29.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.910+0000 7f02263dd740 1 --2- 172.21.15.67:0/2844196588 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55faf983e020 0x55faf983d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:29.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.910+0000 7f02263dd740 1 --2- 172.21.15.67:0/2844196588 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55faf9864cb0 0x55faf983cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:29.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.910+0000 7f02263dd740 1 --2- 172.21.15.67:0/2844196588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55faf983dc40 0x55faf9864770 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:29.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.910+0000 7f02263dd740 1 -- 172.21.15.67:0/2844196588 >> 172.21.15.67:0/2844196588 conn(0x55faf9804a60 msgr2=0x55faf9882960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:29.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.910+0000 7f02263dd740 1 -- 172.21.15.67:0/2844196588 shutdown_connections 2024-03-24T00:43:29.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.910+0000 7f02263dd740 1 -- 172.21.15.67:0/2844196588 wait complete. 2024-03-24T00:43:29.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.910+0000 7f02263dd740 1 -- start start 2024-03-24T00:43:29.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.910+0000 7f02263dd740 1 Processor -- start 2024-03-24T00:43:29.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.911+0000 7f02263dd740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55faf983dc40 0x55faf985d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:29.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.911+0000 7f02263dd740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55faf983e020 0x55faf985dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:29.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.911+0000 7f02263dd740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55faf985b630 0x55faf9862880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:29.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.911+0000 7f02263dd740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faf98642e0 con 0x55faf985b630 2024-03-24T00:43:29.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.911+0000 7f02263dd740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faf975de90 con 0x55faf983e020 2024-03-24T00:43:29.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.911+0000 7f02263dd740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55faf985d060 con 0x55faf983dc40 2024-03-24T00:43:29.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.911+0000 7f02263dd740 10 client.0 _ll_register_callbacks cb 0x55faf94b5ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:43:29.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.911+0000 7f0221d91640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55faf985b630 0x55faf9862880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:29.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.911+0000 7f0221d91640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55faf985b630 0x55faf9862880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48370/0 (socket says 172.21.15.67:48370) 2024-03-24T00:43:29.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.911+0000 7f0221d91640 1 -- 172.21.15.67:0/465814110 learned_addr learned my addr 172.21.15.67:0/465814110 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:29.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.911+0000 7f0220d8f640 1 --2- 172.21.15.67:0/465814110 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55faf983e020 0x55faf985dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:29.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.911+0000 7f0221590640 1 --2- 172.21.15.67:0/465814110 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55faf983dc40 0x55faf985d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:29.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.912+0000 7f0221d91640 1 -- 172.21.15.67:0/465814110 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55faf983dc40 msgr2=0x55faf985d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:29.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.912+0000 7f0221d91640 1 --2- 172.21.15.67:0/465814110 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55faf983dc40 0x55faf985d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:29.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.912+0000 7f0221d91640 1 -- 172.21.15.67:0/465814110 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55faf983e020 msgr2=0x55faf985dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:29.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.912+0000 7f0221d91640 1 --2- 172.21.15.67:0/465814110 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55faf983e020 0x55faf985dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:29.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.912+0000 7f0221d91640 1 -- 172.21.15.67:0/465814110 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55faf976a9d0 con 0x55faf985b630 2024-03-24T00:43:29.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.912+0000 7f0221d91640 1 --2- 172.21.15.67:0/465814110 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55faf985b630 0x55faf9862880 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f02140605b0 tx=0x7f02140580b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d793d8fa78c85587 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:29.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.912+0000 7f01fbfff640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:43:29.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.912+0000 7f01fbfff640 1 -- 172.21.15.67:0/465814110 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0214060ae0 con 0x55faf985b630 2024-03-24T00:43:29.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.912+0000 7f01fbfff640 1 -- 172.21.15.67:0/465814110 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f021400ac50 con 0x55faf985b630 2024-03-24T00:43:29.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.912+0000 7f02263dd740 1 -- 172.21.15.67:0/465814110 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0218024070 con 0x55faf985b630 2024-03-24T00:43:29.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.912+0000 7f01fbfff640 1 -- 172.21.15.67:0/465814110 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0214088050 con 0x55faf985b630 2024-03-24T00:43:29.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.912+0000 7f02263dd740 10 client.7197 fetch_fsmap learned FSMap version 46 2024-03-24T00:43:29.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.912+0000 7f02263dd740 1 -- 172.21.15.67:0/465814110 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=2+}) v3 -- 0x55faf94b62d0 con 0x55faf985b630 2024-03-24T00:43:29.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.912+0000 7f01fbfff640 1 -- 172.21.15.67:0/465814110 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0218024070 con 0x55faf985b630 2024-03-24T00:43:29.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.912+0000 7f01fbfff640 1 -- 172.21.15.67:0/465814110 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 46) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f0214060650 con 0x55faf985b630 2024-03-24T00:43:29.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.912+0000 7f02263dd740 10 client.7197 fetch_fsmap finished waiting for FSMap version 46 2024-03-24T00:43:29.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.912+0000 7f02263dd740 10 client.7197 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T00:43:29.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.912+0000 7f02263dd740 20 client.7197 populate_metadata read hostname 'smithi067' 2024-03-24T00:43:29.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.912+0000 7f02263dd740 1 --2- 172.21.15.67:0/465814110 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x55faf9868c30 0x55faf98a0070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:29.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.912+0000 7f02263dd740 4 client.7197 mds_command: new command op to 7149 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:43:29.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.912+0000 7f02263dd740 1 -- 172.21.15.67:0/465814110 --> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55faf97af160 con 0x55faf9868c30 2024-03-24T00:43:29.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.912+0000 7f0221590640 1 --2- 172.21.15.67:0/465814110 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x55faf9868c30 0x55faf98a0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:29.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.913+0000 7f0221590640 1 --2- 172.21.15.67:0/465814110 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x55faf9868c30 0x55faf98a0070 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e314ffd162fda07a server_cookie=5d34f70e0bbd3be6 in_seq=0 out_seq=0 2024-03-24T00:43:29.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.913+0000 7f01fbfff640 10 client.7197 ms_handle_connect on v2:172.21.15.67:6832/720256752 2024-03-24T00:43:29.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.913+0000 7f01fbfff640 1 -- 172.21.15.67:0/465814110 <== mds.0 v2:172.21.15.67:6832/720256752 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55faf97af160 con 0x55faf9868c30 2024-03-24T00:43:29.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.913+0000 7f01fbfff640 10 client.7197 handle_command_reply: tid=0 2024-03-24T00:43:29.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.913+0000 7f02263dd740 1 client.7197 shutdown 2024-03-24T00:43:29.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.913+0000 7f02263dd740 1 -- 172.21.15.67:0/465814110 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55faf985b630 msgr2=0x55faf9862880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:29.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.914+0000 7f02263dd740 1 --2- 172.21.15.67:0/465814110 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55faf985b630 0x55faf9862880 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f02140605b0 tx=0x7f02140580b0 comp rx=0 tx=0).stop 2024-03-24T00:43:29.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.914+0000 7f02263dd740 1 -- 172.21.15.67:0/465814110 shutdown_connections 2024-03-24T00:43:29.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.914+0000 7f02263dd740 1 --2- 172.21.15.67:0/465814110 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55faf985b630 0x55faf9862880 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:29.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.914+0000 7f02263dd740 1 --2- 172.21.15.67:0/465814110 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x55faf9868c30 0x55faf98a0070 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:29.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.914+0000 7f02263dd740 1 --2- 172.21.15.67:0/465814110 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55faf983e020 0x55faf985dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:29.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.914+0000 7f02263dd740 1 --2- 172.21.15.67:0/465814110 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55faf983dc40 0x55faf985d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:29.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.914+0000 7f02263dd740 1 -- 172.21.15.67:0/465814110 >> 172.21.15.67:0/465814110 conn(0x55faf9804a60 msgr2=0x55faf9883300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:29.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.914+0000 7f01fbfff640 0 client.7197 ms_handle_reset on v2:172.21.15.67:6832/720256752 2024-03-24T00:43:29.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.914+0000 7f02263dd740 1 -- 172.21.15.67:0/465814110 shutdown_connections 2024-03-24T00:43:29.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.914+0000 7f02263dd740 1 -- 172.21.15.67:0/465814110 wait complete. 2024-03-24T00:43:29.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.914+0000 7f02263dd740 20 client.7197 trim_cache size 0 max 16384 2024-03-24T00:43:29.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.945+0000 7f02263dd740 1 Processor -- start 2024-03-24T00:43:29.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.945+0000 7f02263dd740 1 -- start start 2024-03-24T00:43:29.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.946+0000 7f02263dd740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55faf983dc40 0x55faf985aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:29.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.946+0000 7f02263dd740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55faf983e020 0x55faf9867870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:29.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.946+0000 7f02263dd740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55faf985b630 0x55faf9867db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:29.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.946+0000 7f02263dd740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f021c1fe7c0 con 0x55faf983e020 2024-03-24T00:43:29.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.946+0000 7f02263dd740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f021c1fd990 con 0x55faf983dc40 2024-03-24T00:43:29.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.946+0000 7f02263dd740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f021c0ca5b0 con 0x55faf985b630 2024-03-24T00:43:29.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.947+0000 7f0220d8f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55faf983e020 0x55faf9867870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:29.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.947+0000 7f0220d8f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55faf983e020 0x55faf9867870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48384/0 (socket says 172.21.15.67:48384) 2024-03-24T00:43:29.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.947+0000 7f0220d8f640 1 -- 172.21.15.67:0/1962665469 learned_addr learned my addr 172.21.15.67:0/1962665469 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:29.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.947+0000 7f0221590640 1 --2- 172.21.15.67:0/1962665469 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55faf983dc40 0x55faf985aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:29.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.947+0000 7f0221d91640 1 --2- 172.21.15.67:0/1962665469 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55faf985b630 0x55faf9867db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:29.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.947+0000 7f0220d8f640 1 -- 172.21.15.67:0/1962665469 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55faf985b630 msgr2=0x55faf9867db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:29.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.947+0000 7f0220d8f640 1 --2- 172.21.15.67:0/1962665469 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55faf985b630 0x55faf9867db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:29.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.947+0000 7f0220d8f640 1 -- 172.21.15.67:0/1962665469 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55faf983dc40 msgr2=0x55faf985aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:29.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.947+0000 7f0220d8f640 1 --2- 172.21.15.67:0/1962665469 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55faf983dc40 0x55faf985aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:29.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.947+0000 7f0220d8f640 1 -- 172.21.15.67:0/1962665469 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55faf95806b0 con 0x55faf983e020 2024-03-24T00:43:29.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.947+0000 7f0220d8f640 1 --2- 172.21.15.67:0/1962665469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55faf983e020 0x55faf9867870 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f020c02bee0 tx=0x7f020c030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=642d2a081194bf8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:29.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.947+0000 7f02067fc640 1 -- 172.21.15.67:0/1962665469 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f020c04c070 con 0x55faf983e020 2024-03-24T00:43:29.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.947+0000 7f02067fc640 1 -- 172.21.15.67:0/1962665469 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f020c031630 con 0x55faf983e020 2024-03-24T00:43:29.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.947+0000 7f02067fc640 1 -- 172.21.15.67:0/1962665469 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f020c0319e0 con 0x55faf983e020 2024-03-24T00:43:29.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.948+0000 7f02263dd740 1 -- 172.21.15.67:0/1962665469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55faf983e020 msgr2=0x55faf9867870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:29.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.948+0000 7f02263dd740 1 --2- 172.21.15.67:0/1962665469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55faf983e020 0x55faf9867870 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f020c02bee0 tx=0x7f020c030000 comp rx=0 tx=0).stop 2024-03-24T00:43:29.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.948+0000 7f02263dd740 1 -- 172.21.15.67:0/1962665469 shutdown_connections 2024-03-24T00:43:29.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.948+0000 7f02263dd740 1 --2- 172.21.15.67:0/1962665469 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55faf985b630 0x55faf9867db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:29.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.948+0000 7f02263dd740 1 --2- 172.21.15.67:0/1962665469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55faf983e020 0x55faf9867870 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:29.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.948+0000 7f02263dd740 1 --2- 172.21.15.67:0/1962665469 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55faf983dc40 0x55faf985aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:29.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.948+0000 7f02263dd740 1 -- 172.21.15.67:0/1962665469 >> 172.21.15.67:0/1962665469 conn(0x55faf9804a60 msgr2=0x55faf9862640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:29.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.948+0000 7f02263dd740 1 -- 172.21.15.67:0/1962665469 shutdown_connections 2024-03-24T00:43:29.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.948+0000 7f02263dd740 1 -- 172.21.15.67:0/1962665469 wait complete. 2024-03-24T00:43:29.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.948+0000 7f02263dd740 1 -- start start 2024-03-24T00:43:29.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.948+0000 7f02263dd740 1 Processor -- start 2024-03-24T00:43:29.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.949+0000 7f02263dd740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55faf983dc40 0x55faf98f16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:29.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.949+0000 7f02263dd740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55faf983e020 0x55faf9837670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:29.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.949+0000 7f02263dd740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55faf985b630 0x55faf9837bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:29.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.949+0000 7f02263dd740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faf9868ab0 con 0x55faf983dc40 2024-03-24T00:43:29.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.949+0000 7f02263dd740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f021c0cabb0 con 0x55faf983e020 2024-03-24T00:43:29.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.949+0000 7f02263dd740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55faf98690b0 con 0x55faf985b630 2024-03-24T00:43:29.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.949+0000 7f02263dd740 10 client.0 _ll_register_callbacks cb 0x7f02140582e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:43:29.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.949+0000 7f0221590640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55faf983dc40 0x55faf98f16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:29.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.949+0000 7f0221590640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55faf983dc40 0x55faf98f16f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48400/0 (socket says 172.21.15.67:48400) 2024-03-24T00:43:29.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.949+0000 7f0221590640 1 -- 172.21.15.67:0/615864792 learned_addr learned my addr 172.21.15.67:0/615864792 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:29.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.949+0000 7f0220d8f640 1 --2- 172.21.15.67:0/615864792 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55faf983e020 0x55faf9837670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:29.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.949+0000 7f0221d91640 1 --2- 172.21.15.67:0/615864792 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55faf985b630 0x55faf9837bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:29.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.950+0000 7f0221590640 1 -- 172.21.15.67:0/615864792 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55faf985b630 msgr2=0x55faf9837bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:29.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.950+0000 7f0221590640 1 --2- 172.21.15.67:0/615864792 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55faf985b630 0x55faf9837bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:29.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.950+0000 7f0221590640 1 -- 172.21.15.67:0/615864792 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55faf983e020 msgr2=0x55faf9837670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:29.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.950+0000 7f0221590640 1 --2- 172.21.15.67:0/615864792 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55faf983e020 0x55faf9837670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:29.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.950+0000 7f0221590640 1 -- 172.21.15.67:0/615864792 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55faf9837130 con 0x55faf983dc40 2024-03-24T00:43:29.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.950+0000 7f0220d8f640 1 --2- 172.21.15.67:0/615864792 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55faf983e020 0x55faf9837670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:43:29.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.950+0000 7f0221590640 1 --2- 172.21.15.67:0/615864792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55faf983dc40 0x55faf98f16f0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f0218021040 tx=0x7f0218016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fdd528157337674 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:29.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.950+0000 7f01fbfff640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:43:29.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.950+0000 7f01fbfff640 1 -- 172.21.15.67:0/615864792 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0218036690 con 0x55faf983dc40 2024-03-24T00:43:29.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.950+0000 7f01fbfff640 1 -- 172.21.15.67:0/615864792 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f0218036830 con 0x55faf983dc40 2024-03-24T00:43:29.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.950+0000 7f02263dd740 1 -- 172.21.15.67:0/615864792 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f020c031630 con 0x55faf983dc40 2024-03-24T00:43:29.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.950+0000 7f01fbfff640 1 -- 172.21.15.67:0/615864792 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0218045050 con 0x55faf983dc40 2024-03-24T00:43:29.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.950+0000 7f02263dd740 10 client.7203 fetch_fsmap learned FSMap version 46 2024-03-24T00:43:29.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.950+0000 7f02263dd740 1 -- 172.21.15.67:0/615864792 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=2+}) v3 -- 0x55faf98a4640 con 0x55faf983dc40 2024-03-24T00:43:29.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.950+0000 7f01fbfff640 1 -- 172.21.15.67:0/615864792 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f020c031630 con 0x55faf983dc40 2024-03-24T00:43:29.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.950+0000 7f01fbfff640 1 -- 172.21.15.67:0/615864792 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 46) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f0218048020 con 0x55faf983dc40 2024-03-24T00:43:29.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.950+0000 7f02263dd740 10 client.7203 fetch_fsmap finished waiting for FSMap version 46 2024-03-24T00:43:29.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.950+0000 7f02263dd740 10 client.7203 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T00:43:29.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.950+0000 7f02263dd740 20 client.7203 populate_metadata read hostname 'smithi067' 2024-03-24T00:43:29.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.950+0000 7f02263dd740 1 --2- 172.21.15.67:0/615864792 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x55faf9882720 0x55faf98782f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:29.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.950+0000 7f02263dd740 4 client.7203 mds_command: new command op to 7149 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:43:29.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.950+0000 7f02263dd740 1 -- 172.21.15.67:0/615864792 --> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55faf97a94b0 con 0x55faf9882720 2024-03-24T00:43:29.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.951+0000 7f0220d8f640 1 --2- 172.21.15.67:0/615864792 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x55faf9882720 0x55faf98782f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:29.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.951+0000 7f01fbfff640 10 client.7203 ms_handle_connect on v2:172.21.15.67:6832/720256752 2024-03-24T00:43:29.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.951+0000 7f0220d8f640 1 --2- 172.21.15.67:0/615864792 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x55faf9882720 0x55faf98782f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9fad067fb293834 server_cookie=ab1d7bda8ce9595f in_seq=0 out_seq=0 2024-03-24T00:43:29.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.951+0000 7f01fbfff640 1 -- 172.21.15.67:0/615864792 <== mds.0 v2:172.21.15.67:6832/720256752 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55faf97a94b0 con 0x55faf9882720 2024-03-24T00:43:29.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.951+0000 7f01fbfff640 10 client.7203 handle_command_reply: tid=0 2024-03-24T00:43:29.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.951+0000 7f02263dd740 1 client.7203 shutdown 2024-03-24T00:43:29.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.951+0000 7f02263dd740 1 -- 172.21.15.67:0/615864792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55faf983dc40 msgr2=0x55faf98f16f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:29.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.951+0000 7f02263dd740 1 --2- 172.21.15.67:0/615864792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55faf983dc40 0x55faf98f16f0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f0218021040 tx=0x7f0218016000 comp rx=0 tx=0).stop 2024-03-24T00:43:29.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.951+0000 7f02263dd740 1 -- 172.21.15.67:0/615864792 shutdown_connections 2024-03-24T00:43:29.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.951+0000 7f02263dd740 1 --2- 172.21.15.67:0/615864792 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55faf985b630 0x55faf9837bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:29.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.951+0000 7f02263dd740 1 --2- 172.21.15.67:0/615864792 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x55faf9882720 0x55faf98782f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:29.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.951+0000 7f02263dd740 1 --2- 172.21.15.67:0/615864792 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55faf983e020 0x55faf9837670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:29.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.951+0000 7f02263dd740 1 --2- 172.21.15.67:0/615864792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55faf983dc40 0x55faf98f16f0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:29.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.951+0000 7f01fbfff640 0 client.7203 ms_handle_reset on v2:172.21.15.67:6832/720256752 2024-03-24T00:43:29.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.951+0000 7f02263dd740 1 -- 172.21.15.67:0/615864792 >> 172.21.15.67:0/615864792 conn(0x55faf9804a60 msgr2=0x55faf9867cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:29.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.951+0000 7f02263dd740 1 -- 172.21.15.67:0/615864792 shutdown_connections 2024-03-24T00:43:29.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.952+0000 7f02263dd740 1 -- 172.21.15.67:0/615864792 wait complete. 2024-03-24T00:43:29.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.952+0000 7f02263dd740 20 client.7203 trim_cache size 0 max 16384 2024-03-24T00:43:29.960 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:43:29.960 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "de1d0178-808e-49ca-a986-e297e523b976", 2024-03-24T00:43:29.960 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-03-24T00:43:29.960 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 7149, 2024-03-24T00:43:29.960 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-03-24T00:43:29.960 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-03-24T00:43:29.960 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-03-24T00:43:29.960 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 4.5690793740000002, 2024-03-24T00:43:29.960 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 46, 2024-03-24T00:43:29.960 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 77, 2024-03-24T00:43:29.960 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 77, 2024-03-24T00:43:29.960 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 5.5392015780000001 2024-03-24T00:43:29.961 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:43:29.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.952+0000 7f02237f4640 1 -- 172.21.15.67:0/3753645417 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f020006bbd0 msgr2=0x7f020006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:29.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.952+0000 7f02237f4640 1 --2- 172.21.15.67:0/3753645417 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f020006bbd0 0x7f020006e090 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f0218006fd0 tx=0x7f0218009060 comp rx=0 tx=0).stop 2024-03-24T00:43:29.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.952+0000 7f02237f4640 1 -- 172.21.15.67:0/3753645417 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f021c15c010 msgr2=0x7f021c06fe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:29.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.952+0000 7f02237f4640 1 --2- 172.21.15.67:0/3753645417 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f021c15c010 0x7f021c06fe40 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f021400c7a0 tx=0x7f021400cc60 comp rx=0 tx=0).stop 2024-03-24T00:43:29.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.952+0000 7f02237f4640 1 -- 172.21.15.67:0/3753645417 shutdown_connections 2024-03-24T00:43:29.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.952+0000 7f02237f4640 1 --2- 172.21.15.67:0/3753645417 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f020006bbd0 0x7f020006e090 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:29.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.952+0000 7f02237f4640 1 --2- 172.21.15.67:0/3753645417 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f021c15c010 0x7f021c06fe40 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:29.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.952+0000 7f02237f4640 1 --2- 172.21.15.67:0/3753645417 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f021c0d07a0 0x7f021c15cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:29.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.952+0000 7f02237f4640 1 --2- 172.21.15.67:0/3753645417 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f021c0cccb0 0x7f021c156c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:29.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.952+0000 7f02237f4640 1 -- 172.21.15.67:0/3753645417 >> 172.21.15.67:0/3753645417 conn(0x7f021c09aea0 msgr2=0x7f021c0ce110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:29.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.952+0000 7f02237f4640 1 -- 172.21.15.67:0/3753645417 shutdown_connections 2024-03-24T00:43:29.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:29.952+0000 7f02237f4640 1 -- 172.21.15.67:0/3753645417 wait complete. 2024-03-24T00:43:29.964 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T00:43:30.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.102+0000 7fb4bc190640 1 Processor -- start 2024-03-24T00:43:30.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.102+0000 7fb4bc190640 1 -- start start 2024-03-24T00:43:30.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.104+0000 7fb4bc190640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb4b415dec0 0x7fb4b40ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:30.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.104+0000 7fb4bc190640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4b40cf640 con 0x7fb4b40d0ef0 2024-03-24T00:43:30.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.104+0000 7fb4bc190640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4b40cf080 con 0x7fb4b415e2a0 2024-03-24T00:43:30.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.104+0000 7fb4bc190640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4b40cee30 con 0x7fb4b415dec0 2024-03-24T00:43:30.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.104+0000 7fb4ba72d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb4b40d0ef0 0x7fb4b40d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58296/0 (socket says 172.21.15.67:58296) 2024-03-24T00:43:30.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.104+0000 7fb4ba72d640 1 -- 172.21.15.67:0/3256646208 learned_addr learned my addr 172.21.15.67:0/3256646208 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:43:30.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.104+0000 7fb4b8f2a640 1 -- 172.21.15.67:0/3256646208 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2133197038 0 0) 0x7fb4b0001490 con 0x7fb4b40d0ef0 2024-03-24T00:43:30.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.104+0000 7fb4b8f2a640 1 -- 172.21.15.67:0/3256646208 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4a0003520 con 0x7fb4b40d0ef0 2024-03-24T00:43:30.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.104+0000 7fb4b9f2c640 1 --2- 172.21.15.67:0/3256646208 >> v2:172.21.15.72:3300/0 conn(0x7fb4b415dec0 0x7fb4b40ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:30.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.104+0000 7fb4b8f2a640 1 -- 172.21.15.67:0/3256646208 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1736335319 0 0) 0x7fb4a0003520 con 0x7fb4b40d0ef0 2024-03-24T00:43:30.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.104+0000 7fb4b8f2a640 1 -- 172.21.15.67:0/3256646208 >> v2:172.21.15.72:3300/0 conn(0x7fb4b415dec0 msgr2=0x7fb4b40ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:30.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.104+0000 7fb4b8f2a640 1 --2- 172.21.15.67:0/3256646208 >> v2:172.21.15.72:3300/0 conn(0x7fb4b415dec0 0x7fb4b40ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:30.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.104+0000 7fb4b8f2a640 1 -- 172.21.15.67:0/3256646208 >> v1:172.21.15.72:6789/0 conn(0x7fb4b415e2a0 legacy=0x7fb4b40d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:30.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.104+0000 7fb4b8f2a640 1 -- 172.21.15.67:0/3256646208 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b40cf360 con 0x7fb4b40d0ef0 2024-03-24T00:43:30.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.104+0000 7fb4b8f2a640 1 -- 172.21.15.67:0/3256646208 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7fb4b0001bc0 con 0x7fb4b40d0ef0 2024-03-24T00:43:30.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.104+0000 7fb4b8f2a640 1 -- 172.21.15.67:0/3256646208 >> v1:172.21.15.67:6789/0 conn(0x7fb4b40d0ef0 legacy=0x7fb4b40d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:30.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.104+0000 7fb4b8f2a640 1 --2- 172.21.15.67:0/3256646208 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4a00039b0 0x7fb4a0003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:30.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.104+0000 7fb4b8f2a640 1 --2- 172.21.15.67:0/3256646208 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb4a0004800 0x7fb4a0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:30.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.104+0000 7fb4b8f2a640 1 --2- 172.21.15.67:0/3256646208 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb4a00052e0 0x7fb4a000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:30.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.104+0000 7fb4b8f2a640 1 -- 172.21.15.67:0/3256646208 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b40cee30 con 0x7fb4a00039b0 2024-03-24T00:43:30.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.104+0000 7fb4b8f2a640 1 -- 172.21.15.67:0/3256646208 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a000c1e0 con 0x7fb4a0004800 2024-03-24T00:43:30.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.104+0000 7fb4b8f2a640 1 -- 172.21.15.67:0/3256646208 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a000c3e0 con 0x7fb4a00052e0 2024-03-24T00:43:30.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.104+0000 7fb4b9f2c640 1 --2- 172.21.15.67:0/3256646208 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4a00039b0 0x7fb4a0003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:30.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.105+0000 7fb4ba72d640 1 --2- 172.21.15.67:0/3256646208 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb4a00052e0 0x7fb4a000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:30.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.105+0000 7fb4b972b640 1 --2- 172.21.15.67:0/3256646208 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb4a0004800 0x7fb4a0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:30.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.105+0000 7fb4b9f2c640 1 -- 172.21.15.67:0/3256646208 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb4a00052e0 msgr2=0x7fb4a000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:30.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.105+0000 7fb4b9f2c640 1 --2- 172.21.15.67:0/3256646208 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb4a00052e0 0x7fb4a000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:30.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.105+0000 7fb4b9f2c640 1 -- 172.21.15.67:0/3256646208 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb4a0004800 msgr2=0x7fb4a0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:30.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.105+0000 7fb4b9f2c640 1 --2- 172.21.15.67:0/3256646208 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb4a0004800 0x7fb4a0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:30.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.105+0000 7fb4ba72d640 1 -- 172.21.15.67:0/3256646208 reap_dead start 2024-03-24T00:43:30.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.105+0000 7fb4b9f2c640 1 -- 172.21.15.67:0/3256646208 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a0003520 con 0x7fb4a00039b0 2024-03-24T00:43:30.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.106+0000 7fb4b9f2c640 1 --2- 172.21.15.67:0/3256646208 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4a00039b0 0x7fb4a0003df0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac00ba00 tx=0x7fb4a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7e75a2cac0833e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:30.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.106+0000 7fb4b8f2a640 1 -- 172.21.15.67:0/3256646208 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac036020 con 0x7fb4a00039b0 2024-03-24T00:43:30.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.106+0000 7fb4b8f2a640 1 -- 172.21.15.67:0/3256646208 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4ac0325f0 con 0x7fb4a00039b0 2024-03-24T00:43:30.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.106+0000 7fb4b8f2a640 1 -- 172.21.15.67:0/3256646208 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac00b600 con 0x7fb4a00039b0 2024-03-24T00:43:30.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.107+0000 7fb4bc190640 1 -- 172.21.15.67:0/3256646208 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4a00039b0 msgr2=0x7fb4a0003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:30.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.107+0000 7fb4bc190640 1 --2- 172.21.15.67:0/3256646208 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4a00039b0 0x7fb4a0003df0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac00ba00 tx=0x7fb4a0005830 comp rx=0 tx=0).stop 2024-03-24T00:43:30.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.107+0000 7fb4bc190640 1 -- 172.21.15.67:0/3256646208 shutdown_connections 2024-03-24T00:43:30.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.107+0000 7fb4bc190640 1 --2- 172.21.15.67:0/3256646208 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4a00039b0 0x7fb4a0003df0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:30.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.107+0000 7fb4bc190640 1 -- 172.21.15.67:0/3256646208 >> 172.21.15.67:0/3256646208 conn(0x7fb4b406c810 msgr2=0x7fb4b406cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:30.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.107+0000 7fb4bc190640 1 -- 172.21.15.67:0/3256646208 shutdown_connections 2024-03-24T00:43:30.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.107+0000 7fb4bc190640 1 -- 172.21.15.67:0/3256646208 wait complete. 2024-03-24T00:43:30.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.109+0000 7fb4bc190640 1 Processor -- start 2024-03-24T00:43:30.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.109+0000 7fb4bc190640 1 -- start start 2024-03-24T00:43:30.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.110+0000 7fb4bc190640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb4a00039b0 0x7fb4b40b2c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:30.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.110+0000 7fb4bc190640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4b41582e0 0x7fb4b4158760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:30.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.110+0000 7fb4bc190640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb4b41f43f0 0x7fb4b41f4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:30.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.110+0000 7fb4bc190640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b40ce9c0 con 0x7fb4b41582e0 2024-03-24T00:43:30.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.110+0000 7fb4bc190640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b40ce840 con 0x7fb4a00039b0 2024-03-24T00:43:30.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.110+0000 7fb4bc190640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b40d2120 con 0x7fb4b41f43f0 2024-03-24T00:43:30.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.110+0000 7fb4b972b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4b41582e0 0x7fb4b4158760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:30.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.110+0000 7fb4b972b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4b41582e0 0x7fb4b4158760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48416/0 (socket says 172.21.15.67:48416) 2024-03-24T00:43:30.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.110+0000 7fb4b972b640 1 -- 172.21.15.67:0/2585160852 learned_addr learned my addr 172.21.15.67:0/2585160852 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:30.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.110+0000 7fb4b9f2c640 1 --2- 172.21.15.67:0/2585160852 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb4a00039b0 0x7fb4b40b2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:30.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.110+0000 7fb4ba72d640 1 --2- 172.21.15.67:0/2585160852 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb4b41f43f0 0x7fb4b41f4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:30.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.110+0000 7fb4b972b640 1 -- 172.21.15.67:0/2585160852 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb4b41f43f0 msgr2=0x7fb4b41f4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:30.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.110+0000 7fb4b972b640 1 --2- 172.21.15.67:0/2585160852 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb4b41f43f0 0x7fb4b41f4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:30.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.110+0000 7fb4b972b640 1 -- 172.21.15.67:0/2585160852 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb4a00039b0 msgr2=0x7fb4b40b2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:30.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.110+0000 7fb4b972b640 1 --2- 172.21.15.67:0/2585160852 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb4a00039b0 0x7fb4b40b2c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:30.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.110+0000 7fb4b972b640 1 -- 172.21.15.67:0/2585160852 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b4158ca0 con 0x7fb4b41582e0 2024-03-24T00:43:30.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.110+0000 7fb4ba72d640 1 --2- 172.21.15.67:0/2585160852 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb4b41f43f0 0x7fb4b41f4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:43:30.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.110+0000 7fb4b972b640 1 --2- 172.21.15.67:0/2585160852 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4b41582e0 0x7fb4b4158760 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fb4a4007810 tx=0x7fb4a400f9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:30.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.111+0000 7fb4aa7fc640 1 -- 172.21.15.67:0/2585160852 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a40052e0 con 0x7fb4b41582e0 2024-03-24T00:43:30.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.111+0000 7fb4aa7fc640 1 -- 172.21.15.67:0/2585160852 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4a4018070 con 0x7fb4b41582e0 2024-03-24T00:43:30.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.111+0000 7fb4bc190640 1 -- 172.21.15.67:0/2585160852 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4b41f5160 con 0x7fb4b41582e0 2024-03-24T00:43:30.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.111+0000 7fb4aa7fc640 1 -- 172.21.15.67:0/2585160852 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a4003830 con 0x7fb4b41582e0 2024-03-24T00:43:30.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.111+0000 7fb4bc190640 1 -- 172.21.15.67:0/2585160852 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4b41f7c10 con 0x7fb4b41582e0 2024-03-24T00:43:30.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.112+0000 7fb4bc190640 1 -- 172.21.15.67:0/2585160852 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4b404f400 con 0x7fb4b41582e0 2024-03-24T00:43:30.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.112+0000 7fb4aa7fc640 1 -- 172.21.15.67:0/2585160852 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb4a40039d0 con 0x7fb4b41582e0 2024-03-24T00:43:30.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.113+0000 7fb4aa7fc640 1 --2- 172.21.15.67:0/2585160852 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb49806bbd0 0x7fb49806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:30.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.113+0000 7fb4aa7fc640 1 -- 172.21.15.67:0/2585160852 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fb4a408a7a0 con 0x7fb4b41582e0 2024-03-24T00:43:30.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.113+0000 7fb4b9f2c640 1 --2- 172.21.15.67:0/2585160852 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb49806bbd0 0x7fb49806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:30.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.114+0000 7fb4b9f2c640 1 --2- 172.21.15.67:0/2585160852 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb49806bbd0 0x7fb49806e090 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fb4a00057e0 tx=0x7fb4ac03d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:30.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.117+0000 7fb4aa7fc640 1 -- 172.21.15.67:0/2585160852 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb4b41f5160 con 0x7fb4b41582e0 2024-03-24T00:43:30.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.348+0000 7fb4bc190640 1 -- 172.21.15.67:0/2585160852 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fb4b40d1aa0 con 0x7fb4b41582e0 2024-03-24T00:43:30.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.348+0000 7fb4aa7fc640 1 -- 172.21.15.67:0/2585160852 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v46) v1 ==== 53+0+73 (secure 0 0 0) 0x7fb4a4059140 con 0x7fb4b41582e0 2024-03-24T00:43:30.348 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-24T00:43:30.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.351+0000 7fb4bc190640 1 -- 172.21.15.67:0/2585160852 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb49806bbd0 msgr2=0x7fb49806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:30.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.351+0000 7fb4bc190640 1 --2- 172.21.15.67:0/2585160852 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb49806bbd0 0x7fb49806e090 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fb4a00057e0 tx=0x7fb4ac03d040 comp rx=0 tx=0).stop 2024-03-24T00:43:30.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.351+0000 7fb4bc190640 1 -- 172.21.15.67:0/2585160852 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4b41582e0 msgr2=0x7fb4b4158760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:30.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.351+0000 7fb4bc190640 1 --2- 172.21.15.67:0/2585160852 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4b41582e0 0x7fb4b4158760 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fb4a4007810 tx=0x7fb4a400f9d0 comp rx=0 tx=0).stop 2024-03-24T00:43:30.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.351+0000 7fb4bc190640 1 -- 172.21.15.67:0/2585160852 shutdown_connections 2024-03-24T00:43:30.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.351+0000 7fb4bc190640 1 --2- 172.21.15.67:0/2585160852 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb49806bbd0 0x7fb49806e090 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:30.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.351+0000 7fb4bc190640 1 --2- 172.21.15.67:0/2585160852 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb4b41f43f0 0x7fb4b41f4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:30.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.351+0000 7fb4bc190640 1 --2- 172.21.15.67:0/2585160852 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4b41582e0 0x7fb4b4158760 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:30.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.351+0000 7fb4bc190640 1 --2- 172.21.15.67:0/2585160852 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb4a00039b0 0x7fb4b40b2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:30.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.351+0000 7fb4bc190640 1 -- 172.21.15.67:0/2585160852 >> 172.21.15.67:0/2585160852 conn(0x7fb4b406c810 msgr2=0x7fb4b415bd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:30.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.351+0000 7fb4bc190640 1 -- 172.21.15.67:0/2585160852 shutdown_connections 2024-03-24T00:43:30.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:30.351+0000 7fb4bc190640 1 -- 172.21.15.67:0/2585160852 wait complete. 2024-03-24T00:43:30.364 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-24T00:43:30.364 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T00:43:30.365 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-03-24T00:43:30.365 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-24T00:43:30.365 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T00:43:30.365 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-24T00:43:30.365 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-24T00:43:30.365 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T00:43:30.365 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:43:30.365 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:43:30.365 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-03-24T00:43:30.381 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T00:43:30.381 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T00:43:30.381 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T00:43:30.381 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:43:30.381 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-03-24T00:43:30.381 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:43:30.381 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:43:30.381 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-03-24T00:43:30.382 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-03-24T00:43:30.382 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:43:30.382 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-03-24T00:43:30.382 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-03-24T00:43:30.382 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:43:30.382 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-03-24T00:43:30.382 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T00:43:30.382 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-03-24T00:43:30.382 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-24T00:43:30.382 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 42303sec preferred_lft 42303sec 2024-03-24T00:43:30.382 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-03-24T00:43:30.382 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:43:30.382 INFO:teuthology.orchestra.run.smithi067.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T00:43:30.383 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 96:cf:26:c8:bf:62 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:43:30.383 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-24T00:43:30.383 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:43:30.383 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::7c6a:5fff:feaa:d41/64 scope link 2024-03-24T00:43:30.383 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:43:30.383 INFO:teuthology.orchestra.run.smithi067.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T00:43:30.383 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ea:22:4b:d5:48:3d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:43:30.383 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::e822:4bff:fed5:483d/64 scope link 2024-03-24T00:43:30.383 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:43:30.383 INFO:teuthology.orchestra.run.smithi067.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T00:43:30.383 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 96:cf:26:c8:bf:62 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:43:30.383 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::94cf:26ff:fec8:bf62/64 scope link 2024-03-24T00:43:30.383 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:43:30.384 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:43:30.384 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T00:43:30.384 INFO:tasks.ceph.mds.c.smithi067.stderr:2024-03-24T00:43:30.383+0000 7f88e6548640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T00:43:30.411 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-03-24T00:43:30.413 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:43:30.413 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-03-24T00:43:30.468 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-03-24T00:43:30.468 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T00:43:30.468 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-24T00:43:30.468 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-24T00:43:30.469 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T00:43:30.469 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:43:30.469 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-03-24T00:43:30.469 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-03-24T00:43:30.469 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-24T00:43:30.469 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-24T00:43:30.469 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-24T00:43:30.469 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-03-24T00:43:30.564 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:43:30.564 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-03-24T00:43:30.578 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-24T00:43:30.578 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-24T00:43:30.579 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:43:30.579 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:43:30.579 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-03-24T00:43:30.649 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:43:30.649 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-24T00:43:30.704 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:43:30.704 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:43:30.704 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:43:30.758 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-24T00:43:30.758 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:43:30.758 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:43:30.842 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:43:30.843 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T00:43:30.843 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-03-24T00:43:30.851 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:43:30.851 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T00:43:30.851 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-03-24T00:43:30.851 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.72:3300,172.21.15.72:3301" 2024-03-24T00:43:30.851 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-24T00:43:30.856 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:43:30.857 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T00:43:30.857 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-03-24T00:43:30.857 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.72:3300,172.21.15.72:3301" 2024-03-24T00:43:30.857 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-24T00:43:30.857 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:6789,172.21.15.72:6789,172.21.15.72:6790" 2024-03-24T00:43:30.857 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.67:6789,172.21.15.72:6789,172.21.15.72:6790:/ 2024-03-24T00:43:30.857 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=de1d0178-808e-49ca-a986-e297e523b976" will pass to kernel 2024-03-24T00:43:30.857 INFO:teuthology.orchestra.run.smithi067.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-24T00:43:30.857 INFO:teuthology.orchestra.run.smithi067.stdout: You just mounted a file system that supports labels which does not 2024-03-24T00:43:30.857 INFO:teuthology.orchestra.run.smithi067.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-24T00:43:30.858 INFO:teuthology.orchestra.run.smithi067.stdout: applications will generate AVC messages and not be allowed access to 2024-03-24T00:43:30.858 INFO:teuthology.orchestra.run.smithi067.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-24T00:43:30.859 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-24T00:43:30.859 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:43:30.859 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:43:30.925 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:43:30.925 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T00:43:30.981 INFO:teuthology.orchestra.run.smithi067.stdout:client7212 2024-03-24T00:43:30.982 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:43:30.982 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:43:31.036 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:43:31.036 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls subdir/) 2024-03-24T00:43:31.090 INFO:teuthology.orchestra.run.smithi067.stdout:file_undamaged 2024-03-24T00:43:31.092 DEBUG:teuthology.orchestra.run.smithi067:> stdin-killer -- python3 -c ' 2024-03-24T00:43:31.092 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-03-24T00:43:31.092 DEBUG:teuthology.orchestra.run.smithi067:> import stat 2024-03-24T00:43:31.092 DEBUG:teuthology.orchestra.run.smithi067:> import json 2024-03-24T00:43:31.092 DEBUG:teuthology.orchestra.run.smithi067:> import sys 2024-03-24T00:43:31.092 DEBUG:teuthology.orchestra.run.smithi067:> 2024-03-24T00:43:31.092 DEBUG:teuthology.orchestra.run.smithi067:> try: 2024-03-24T00:43:31.092 DEBUG:teuthology.orchestra.run.smithi067:> s = os.stat('"'"'/home/ubuntu/cephtest/mnt.0/subdir/file_to_be_damaged'"'"') 2024-03-24T00:43:31.092 DEBUG:teuthology.orchestra.run.smithi067:> except OSError as e: 2024-03-24T00:43:31.092 DEBUG:teuthology.orchestra.run.smithi067:> sys.exit(e.errno) 2024-03-24T00:43:31.092 DEBUG:teuthology.orchestra.run.smithi067:> 2024-03-24T00:43:31.092 DEBUG:teuthology.orchestra.run.smithi067:> attrs = ["st_mode", "st_ino", "st_dev", "st_nlink", "st_uid", "st_gid", "st_size", "st_atime", "st_mtime", "st_ctime"] 2024-03-24T00:43:31.092 DEBUG:teuthology.orchestra.run.smithi067:> print(json.dumps( 2024-03-24T00:43:31.092 DEBUG:teuthology.orchestra.run.smithi067:> dict([(a, getattr(s, a)) for a in attrs]), 2024-03-24T00:43:31.093 DEBUG:teuthology.orchestra.run.smithi067:> indent=2)) 2024-03-24T00:43:31.093 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-03-24T00:43:31.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31 stdin-killer INFO: command exited with status 2: exiting normally with same code! 2024-03-24T00:43:31.225 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-03-24T00:43:31.226 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:43:31.226 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.files ./subdir) 2024-03-24T00:43:31.282 INFO:teuthology.orchestra.run.smithi067.stdout:2 2024-03-24T00:43:31.283 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:43:31.283 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:43:31.338 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:43:31.338 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:43:31.393 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:43:31.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.536+0000 7fc7c3a7e640 1 Processor -- start 2024-03-24T00:43:31.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.536+0000 7fc7c3a7e640 1 -- start start 2024-03-24T00:43:31.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.536+0000 7fc7c3a7e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc7bc15dec0 0x7fc7bc0ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:31.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.536+0000 7fc7c3a7e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc7bc15e2a0 0x7fc7bc162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:31.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.536+0000 7fc7c3a7e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7bc0cf640 con 0x7fc7bc0d07e0 2024-03-24T00:43:31.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.536+0000 7fc7c3a7e640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7bc0cee30 con 0x7fc7bc15dec0 2024-03-24T00:43:31.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.536+0000 7fc7c3a7e640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7bc0d2120 con 0x7fc7bc15e2a0 2024-03-24T00:43:31.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.537+0000 7fc7c181a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc7bc15dec0 0x7fc7bc0ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:31.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.537+0000 7fc7c181a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc7bc15dec0 0x7fc7bc0ce630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48418/0 (socket says 172.21.15.67:48418) 2024-03-24T00:43:31.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.537+0000 7fc7c181a640 1 -- 172.21.15.67:0/2892414631 learned_addr learned my addr 172.21.15.67:0/2892414631 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:31.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.537+0000 7fc7c181a640 1 -- 172.21.15.67:0/2892414631 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc7bc15e2a0 msgr2=0x7fc7bc162630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:31.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.537+0000 7fc7c181a640 1 --2- 172.21.15.67:0/2892414631 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc7bc15e2a0 0x7fc7bc162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:31.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.537+0000 7fc7c181a640 1 -- 172.21.15.67:0/2892414631 >> v1:172.21.15.72:6789/0 conn(0x7fc7bc0d07e0 legacy=0x7fc7bc0b5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:31.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.537+0000 7fc7c181a640 1 -- 172.21.15.67:0/2892414631 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7bc0cf080 con 0x7fc7bc15dec0 2024-03-24T00:43:31.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.538+0000 7fc7c181a640 1 --2- 172.21.15.67:0/2892414631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7bc15dec0 0x7fc7bc0ce630 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fc7bc0cefb0 tx=0x7fc7b802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f17b1f924ec2e1af server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:31.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.538+0000 7fc7c0818640 1 -- 172.21.15.67:0/2892414631 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b80317c0 con 0x7fc7bc15dec0 2024-03-24T00:43:31.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.538+0000 7fc7c0818640 1 -- 172.21.15.67:0/2892414631 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7b8037070 con 0x7fc7bc15dec0 2024-03-24T00:43:31.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.538+0000 7fc7c0818640 1 -- 172.21.15.67:0/2892414631 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b8031b80 con 0x7fc7bc15dec0 2024-03-24T00:43:31.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.538+0000 7fc7c3a7e640 1 -- 172.21.15.67:0/2892414631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7bc15dec0 msgr2=0x7fc7bc0ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:31.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.538+0000 7fc7c3a7e640 1 --2- 172.21.15.67:0/2892414631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7bc15dec0 0x7fc7bc0ce630 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fc7bc0cefb0 tx=0x7fc7b802b860 comp rx=0 tx=0).stop 2024-03-24T00:43:31.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.538+0000 7fc7c3a7e640 1 -- 172.21.15.67:0/2892414631 shutdown_connections 2024-03-24T00:43:31.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.538+0000 7fc7c3a7e640 1 --2- 172.21.15.67:0/2892414631 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc7bc15e2a0 0x7fc7bc162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:31.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.538+0000 7fc7c3a7e640 1 --2- 172.21.15.67:0/2892414631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7bc15dec0 0x7fc7bc0ce630 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:31.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.538+0000 7fc7c3a7e640 1 -- 172.21.15.67:0/2892414631 >> 172.21.15.67:0/2892414631 conn(0x7fc7bc06c810 msgr2=0x7fc7bc06cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:31.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.538+0000 7fc7c3a7e640 1 -- 172.21.15.67:0/2892414631 shutdown_connections 2024-03-24T00:43:31.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.539+0000 7fc7c3a7e640 1 -- 172.21.15.67:0/2892414631 wait complete. 2024-03-24T00:43:31.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.540+0000 7fc7c3a7e640 1 Processor -- start 2024-03-24T00:43:31.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.540+0000 7fc7c3a7e640 1 -- start start 2024-03-24T00:43:31.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.540+0000 7fc7c3a7e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc7bc0d07e0 0x7fc7bc1f51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:31.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.540+0000 7fc7c3a7e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc7bc15dec0 0x7fc7bc0b5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:31.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.540+0000 7fc7c3a7e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7bc15e2a0 0x7fc7bc0b6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:31.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.540+0000 7fc7c3a7e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7bc0d3740 con 0x7fc7bc15e2a0 2024-03-24T00:43:31.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.540+0000 7fc7c3a7e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7bc0d35c0 con 0x7fc7bc0d07e0 2024-03-24T00:43:31.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.540+0000 7fc7c3a7e640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7bc0d38c0 con 0x7fc7bc15dec0 2024-03-24T00:43:31.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.540+0000 7fc7c201b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7bc15e2a0 0x7fc7bc0b6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:31.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.540+0000 7fc7c201b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7bc15e2a0 0x7fc7bc0b6100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48434/0 (socket says 172.21.15.67:48434) 2024-03-24T00:43:31.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.541+0000 7fc7c201b640 1 -- 172.21.15.67:0/2602332301 learned_addr learned my addr 172.21.15.67:0/2602332301 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:31.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.541+0000 7fc7c181a640 1 --2- 172.21.15.67:0/2602332301 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc7bc0d07e0 0x7fc7bc1f51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:31.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.541+0000 7fc7c1019640 1 --2- 172.21.15.67:0/2602332301 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc7bc15dec0 0x7fc7bc0b5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:31.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.541+0000 7fc7c201b640 1 -- 172.21.15.67:0/2602332301 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc7bc15dec0 msgr2=0x7fc7bc0b5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:31.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.541+0000 7fc7c201b640 1 --2- 172.21.15.67:0/2602332301 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc7bc15dec0 0x7fc7bc0b5bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:31.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.541+0000 7fc7c201b640 1 -- 172.21.15.67:0/2602332301 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc7bc0d07e0 msgr2=0x7fc7bc1f51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:31.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.541+0000 7fc7c201b640 1 --2- 172.21.15.67:0/2602332301 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc7bc0d07e0 0x7fc7bc1f51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:31.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.541+0000 7fc7c201b640 1 -- 172.21.15.67:0/2602332301 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7bc0b6960 con 0x7fc7bc15e2a0 2024-03-24T00:43:31.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.541+0000 7fc7c181a640 1 --2- 172.21.15.67:0/2602332301 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc7bc0d07e0 0x7fc7bc1f51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:43:31.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.541+0000 7fc7c1019640 1 --2- 172.21.15.67:0/2602332301 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc7bc15dec0 0x7fc7bc0b5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:43:31.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.541+0000 7fc7c201b640 1 --2- 172.21.15.67:0/2602332301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7bc15e2a0 0x7fc7bc0b6100 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fc7b400b990 tx=0x7fc7b400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:31.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.541+0000 7fc7b27fc640 1 -- 172.21.15.67:0/2602332301 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b4019070 con 0x7fc7bc15e2a0 2024-03-24T00:43:31.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.541+0000 7fc7b27fc640 1 -- 172.21.15.67:0/2602332301 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7b400cba0 con 0x7fc7bc15e2a0 2024-03-24T00:43:31.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.541+0000 7fc7b27fc640 1 -- 172.21.15.67:0/2602332301 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b4014440 con 0x7fc7bc15e2a0 2024-03-24T00:43:31.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.541+0000 7fc7c3a7e640 1 -- 172.21.15.67:0/2602332301 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7bc154b80 con 0x7fc7bc15e2a0 2024-03-24T00:43:31.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.541+0000 7fc7c3a7e640 1 -- 172.21.15.67:0/2602332301 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7bc155140 con 0x7fc7bc15e2a0 2024-03-24T00:43:31.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.542+0000 7fc7c3a7e640 1 -- 172.21.15.67:0/2602332301 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7bc04f400 con 0x7fc7bc15e2a0 2024-03-24T00:43:31.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.543+0000 7fc7b27fc640 1 -- 172.21.15.67:0/2602332301 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc7b401c070 con 0x7fc7bc15e2a0 2024-03-24T00:43:31.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.544+0000 7fc7b27fc640 1 --2- 172.21.15.67:0/2602332301 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc7a006bbd0 0x7fc7a006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:31.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.544+0000 7fc7b27fc640 1 -- 172.21.15.67:0/2602332301 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fc7b408a930 con 0x7fc7bc15e2a0 2024-03-24T00:43:31.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.544+0000 7fc7c181a640 1 --2- 172.21.15.67:0/2602332301 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc7a006bbd0 0x7fc7a006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:31.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.544+0000 7fc7c181a640 1 --2- 172.21.15.67:0/2602332301 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc7a006bbd0 0x7fc7a006e090 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fc7b8036000 tx=0x7fc7b8008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:31.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.548+0000 7fc7b27fc640 1 -- 172.21.15.67:0/2602332301 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc7b40592d0 con 0x7fc7bc15e2a0 2024-03-24T00:43:31.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.734+0000 7fc7c3a7e640 1 -- 172.21.15.67:0/2602332301 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc7bc0b72b0 con 0x7fc7bc15e2a0 2024-03-24T00:43:31.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.735+0000 7fc7b27fc640 1 -- 172.21.15.67:0/2602332301 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+17014 (secure 0 0 0) 0x7fc7b4058bd0 con 0x7fc7bc15e2a0 2024-03-24T00:43:31.735 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:43:31.735 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":77,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:43:24.373819+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:43:08.452726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-24T00:43:09.146431+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6833/2392802741":"2024-03-25T00:43:24.373788+0000","172.21.15.67:6832/2392802741":"2024-03-25T00:43:24.373788+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:43:31.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.737+0000 7fc7c3a7e640 1 -- 172.21.15.67:0/2602332301 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc7a006bbd0 msgr2=0x7fc7a006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:31.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.737+0000 7fc7c3a7e640 1 --2- 172.21.15.67:0/2602332301 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc7a006bbd0 0x7fc7a006e090 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fc7b8036000 tx=0x7fc7b8008030 comp rx=0 tx=0).stop 2024-03-24T00:43:31.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.737+0000 7fc7c3a7e640 1 -- 172.21.15.67:0/2602332301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7bc15e2a0 msgr2=0x7fc7bc0b6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:31.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.737+0000 7fc7c3a7e640 1 --2- 172.21.15.67:0/2602332301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7bc15e2a0 0x7fc7bc0b6100 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fc7b400b990 tx=0x7fc7b400be50 comp rx=0 tx=0).stop 2024-03-24T00:43:31.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.737+0000 7fc7c3a7e640 1 -- 172.21.15.67:0/2602332301 shutdown_connections 2024-03-24T00:43:31.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.737+0000 7fc7c3a7e640 1 --2- 172.21.15.67:0/2602332301 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc7a006bbd0 0x7fc7a006e090 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:31.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.737+0000 7fc7c3a7e640 1 --2- 172.21.15.67:0/2602332301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7bc15e2a0 0x7fc7bc0b6100 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:31.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.737+0000 7fc7c3a7e640 1 --2- 172.21.15.67:0/2602332301 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc7bc15dec0 0x7fc7bc0b5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:31.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.737+0000 7fc7c3a7e640 1 --2- 172.21.15.67:0/2602332301 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc7bc0d07e0 0x7fc7bc1f51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:31.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.737+0000 7fc7c3a7e640 1 -- 172.21.15.67:0/2602332301 >> 172.21.15.67:0/2602332301 conn(0x7fc7bc06c810 msgr2=0x7fc7bc15f640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:31.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.738+0000 7fc7c3a7e640 1 -- 172.21.15.67:0/2602332301 shutdown_connections 2024-03-24T00:43:31.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.738+0000 7fc7c3a7e640 1 -- 172.21.15.67:0/2602332301 wait complete. 2024-03-24T00:43:31.748 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:38:39.364950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:43:08.452726+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:43:09.146431+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:43:31.748 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:43:31.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.854+0000 7fa6a8212640 1 Processor -- start 2024-03-24T00:43:31.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.854+0000 7fa6a8212640 1 -- start start 2024-03-24T00:43:31.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.855+0000 7fa6a8212640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa6a00d3ae0 0x7fa6a00b5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:31.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.855+0000 7fa6a8212640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa6a015e280 0x7fa6a00b5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:31.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.855+0000 7fa6a8212640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6a00d2250 con 0x7fa6a015dea0 2024-03-24T00:43:31.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.855+0000 7fa6a8212640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6a00d2530 con 0x7fa6a00d3ae0 2024-03-24T00:43:31.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.855+0000 7fa6a8212640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a00d2810 con 0x7fa6a015e280 2024-03-24T00:43:31.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.855+0000 7fa6a5fae640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa6a015dea0 0x7fa6a00d33d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58304/0 (socket says 172.21.15.67:58304) 2024-03-24T00:43:31.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.855+0000 7fa6a5fae640 1 -- 172.21.15.67:0/1160310005 learned_addr learned my addr 172.21.15.67:0/1160310005 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:43:31.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.856+0000 7fa6a57ad640 1 --2- 172.21.15.67:0/1160310005 >> v2:172.21.15.72:3300/0 conn(0x7fa6a00d3ae0 0x7fa6a00b5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:31.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.856+0000 7fa6a67af640 1 --2- 172.21.15.67:0/1160310005 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa6a015e280 0x7fa6a00b5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:31.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.856+0000 7fa6a4fac640 1 -- 172.21.15.67:0/1160310005 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1896682790 0 0) 0x7fa6a00d2250 con 0x7fa6a015dea0 2024-03-24T00:43:31.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.856+0000 7fa6a4fac640 1 -- 172.21.15.67:0/1160310005 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa68c003520 con 0x7fa6a015dea0 2024-03-24T00:43:31.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.856+0000 7fa6a67af640 1 -- 172.21.15.67:0/1160310005 >> v2:172.21.15.72:3300/0 conn(0x7fa6a00d3ae0 msgr2=0x7fa6a00b5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:31.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.856+0000 7fa6a67af640 1 --2- 172.21.15.67:0/1160310005 >> v2:172.21.15.72:3300/0 conn(0x7fa6a00d3ae0 0x7fa6a00b5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:31.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.856+0000 7fa6a67af640 1 -- 172.21.15.67:0/1160310005 >> v1:172.21.15.67:6789/0 conn(0x7fa6a015dea0 legacy=0x7fa6a00d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:31.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.856+0000 7fa6a67af640 1 -- 172.21.15.67:0/1160310005 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6a00d2af0 con 0x7fa6a015e280 2024-03-24T00:43:31.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.856+0000 7fa6a57ad640 1 --2- 172.21.15.67:0/1160310005 >> v2:172.21.15.72:3300/0 conn(0x7fa6a00d3ae0 0x7fa6a00b5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:43:31.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.857+0000 7fa6a67af640 1 --2- 172.21.15.67:0/1160310005 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa6a015e280 0x7fa6a00b5fa0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa698009dc0 tx=0x7fa69802be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8063605af70a78e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:31.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.857+0000 7fa6a4fac640 1 -- 172.21.15.67:0/1160310005 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa698039070 con 0x7fa6a015e280 2024-03-24T00:43:31.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.857+0000 7fa6a4fac640 1 -- 172.21.15.67:0/1160310005 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa69802cbb0 con 0x7fa6a015e280 2024-03-24T00:43:31.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.857+0000 7fa6a4fac640 1 -- 172.21.15.67:0/1160310005 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa698034440 con 0x7fa6a015e280 2024-03-24T00:43:31.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.857+0000 7fa6a8212640 1 -- 172.21.15.67:0/1160310005 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa6a015e280 msgr2=0x7fa6a00b5fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:31.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.857+0000 7fa6a8212640 1 --2- 172.21.15.67:0/1160310005 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa6a015e280 0x7fa6a00b5fa0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa698009dc0 tx=0x7fa69802be00 comp rx=0 tx=0).stop 2024-03-24T00:43:31.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.857+0000 7fa6a8212640 1 -- 172.21.15.67:0/1160310005 shutdown_connections 2024-03-24T00:43:31.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.857+0000 7fa6a8212640 1 --2- 172.21.15.67:0/1160310005 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa6a015e280 0x7fa6a00b5fa0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:31.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.857+0000 7fa6a8212640 1 --2- 172.21.15.67:0/1160310005 >> v2:172.21.15.72:3300/0 conn(0x7fa6a00d3ae0 0x7fa6a00b5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:31.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.857+0000 7fa6a8212640 1 -- 172.21.15.67:0/1160310005 >> 172.21.15.67:0/1160310005 conn(0x7fa6a0072da0 msgr2=0x7fa6a00d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:31.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.857+0000 7fa6a8212640 1 -- 172.21.15.67:0/1160310005 shutdown_connections 2024-03-24T00:43:31.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.858+0000 7fa6a8212640 1 -- 172.21.15.67:0/1160310005 wait complete. 2024-03-24T00:43:31.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.859+0000 7fa6a8212640 1 Processor -- start 2024-03-24T00:43:31.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.859+0000 7fa6a8212640 1 -- start start 2024-03-24T00:43:31.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.859+0000 7fa6a8212640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6a00d3ae0 0x7fa6a015a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:31.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.859+0000 7fa6a8212640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa6a015dea0 0x7fa6a00aaca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:31.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.859+0000 7fa6a8212640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa6a015e280 0x7fa6a00b7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:31.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.859+0000 7fa6a8212640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a00b7480 con 0x7fa6a00d3ae0 2024-03-24T00:43:31.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.859+0000 7fa6a8212640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a00b7300 con 0x7fa6a015e280 2024-03-24T00:43:31.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.859+0000 7fa6a8212640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a00b7600 con 0x7fa6a015dea0 2024-03-24T00:43:31.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.859+0000 7fa6a5fae640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6a00d3ae0 0x7fa6a015a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:31.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.860+0000 7fa6a5fae640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6a00d3ae0 0x7fa6a015a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48450/0 (socket says 172.21.15.67:48450) 2024-03-24T00:43:31.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.860+0000 7fa6a5fae640 1 -- 172.21.15.67:0/3474826969 learned_addr learned my addr 172.21.15.67:0/3474826969 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:31.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.860+0000 7fa6a67af640 1 --2- 172.21.15.67:0/3474826969 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa6a015e280 0x7fa6a00b7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:31.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.860+0000 7fa6a57ad640 1 --2- 172.21.15.67:0/3474826969 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa6a015dea0 0x7fa6a00aaca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:31.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.860+0000 7fa6a5fae640 1 -- 172.21.15.67:0/3474826969 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa6a015dea0 msgr2=0x7fa6a00aaca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:31.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.860+0000 7fa6a5fae640 1 --2- 172.21.15.67:0/3474826969 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa6a015dea0 0x7fa6a00aaca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:31.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.860+0000 7fa6a5fae640 1 -- 172.21.15.67:0/3474826969 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa6a015e280 msgr2=0x7fa6a00b7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:31.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.860+0000 7fa6a5fae640 1 --2- 172.21.15.67:0/3474826969 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa6a015e280 0x7fa6a00b7c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:31.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.860+0000 7fa6a5fae640 1 -- 172.21.15.67:0/3474826969 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6a00ab760 con 0x7fa6a00d3ae0 2024-03-24T00:43:31.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.860+0000 7fa6a5fae640 1 --2- 172.21.15.67:0/3474826969 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6a00d3ae0 0x7fa6a015a670 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fa69c00c430 tx=0x7fa69c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:31.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.860+0000 7fa6967fc640 1 -- 172.21.15.67:0/3474826969 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa69c017070 con 0x7fa6a00d3ae0 2024-03-24T00:43:31.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.860+0000 7fa6967fc640 1 -- 172.21.15.67:0/3474826969 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa69c005ce0 con 0x7fa6a00d3ae0 2024-03-24T00:43:31.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.860+0000 7fa6a8212640 1 -- 172.21.15.67:0/3474826969 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6a00aba40 con 0x7fa6a00d3ae0 2024-03-24T00:43:31.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.860+0000 7fa6967fc640 1 -- 172.21.15.67:0/3474826969 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa69c0072f0 con 0x7fa6a00d3ae0 2024-03-24T00:43:31.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.860+0000 7fa6a8212640 1 -- 172.21.15.67:0/3474826969 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6a00ac000 con 0x7fa6a00d3ae0 2024-03-24T00:43:31.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.861+0000 7fa6a8212640 1 -- 172.21.15.67:0/3474826969 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6a004f400 con 0x7fa6a00d3ae0 2024-03-24T00:43:31.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.862+0000 7fa6967fc640 1 -- 172.21.15.67:0/3474826969 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa69c01a070 con 0x7fa6a00d3ae0 2024-03-24T00:43:31.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.862+0000 7fa6967fc640 1 --2- 172.21.15.67:0/3474826969 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa68406bbd0 0x7fa68406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:31.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.862+0000 7fa6967fc640 1 -- 172.21.15.67:0/3474826969 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fa69c089fb0 con 0x7fa6a00d3ae0 2024-03-24T00:43:31.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.862+0000 7fa6a57ad640 1 --2- 172.21.15.67:0/3474826969 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa68406bbd0 0x7fa68406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:31.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.863+0000 7fa6a57ad640 1 --2- 172.21.15.67:0/3474826969 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa68406bbd0 0x7fa68406e090 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fa690009560 tx=0x7fa690009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:31.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:31.871+0000 7fa6967fc640 1 -- 172.21.15.67:0/3474826969 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa69c058300 con 0x7fa6a00d3ae0 2024-03-24T00:43:32.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.056+0000 7fa6a8212640 1 -- 172.21.15.67:0/3474826969 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa6a00d36b0 con 0x7fa6a00d3ae0 2024-03-24T00:43:32.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.056+0000 7fa6967fc640 1 -- 172.21.15.67:0/3474826969 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa69c05c3a0 con 0x7fa6a00d3ae0 2024-03-24T00:43:32.057 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-24T00:43:32.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.058+0000 7fa6a8212640 1 -- 172.21.15.67:0/3474826969 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa68406bbd0 msgr2=0x7fa68406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:32.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.058+0000 7fa6a8212640 1 --2- 172.21.15.67:0/3474826969 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa68406bbd0 0x7fa68406e090 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fa690009560 tx=0x7fa690009190 comp rx=0 tx=0).stop 2024-03-24T00:43:32.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.058+0000 7fa6a8212640 1 -- 172.21.15.67:0/3474826969 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6a00d3ae0 msgr2=0x7fa6a015a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:32.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.058+0000 7fa6a8212640 1 --2- 172.21.15.67:0/3474826969 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6a00d3ae0 0x7fa6a015a670 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fa69c00c430 tx=0x7fa69c00c8f0 comp rx=0 tx=0).stop 2024-03-24T00:43:32.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.058+0000 7fa6a8212640 1 -- 172.21.15.67:0/3474826969 shutdown_connections 2024-03-24T00:43:32.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.058+0000 7fa6a8212640 1 --2- 172.21.15.67:0/3474826969 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa68406bbd0 0x7fa68406e090 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:32.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.058+0000 7fa6a8212640 1 --2- 172.21.15.67:0/3474826969 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa6a015e280 0x7fa6a00b7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:32.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.058+0000 7fa6a8212640 1 --2- 172.21.15.67:0/3474826969 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa6a015dea0 0x7fa6a00aaca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:32.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.058+0000 7fa6a8212640 1 --2- 172.21.15.67:0/3474826969 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6a00d3ae0 0x7fa6a015a670 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:32.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.058+0000 7fa6a8212640 1 -- 172.21.15.67:0/3474826969 >> 172.21.15.67:0/3474826969 conn(0x7fa6a0072da0 msgr2=0x7fa6a00d12b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:32.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.058+0000 7fa6a8212640 1 -- 172.21.15.67:0/3474826969 shutdown_connections 2024-03-24T00:43:32.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.058+0000 7fa6a8212640 1 -- 172.21.15.67:0/3474826969 wait complete. 2024-03-24T00:43:32.073 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:43:32.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.178+0000 7f7773497640 1 Processor -- start 2024-03-24T00:43:32.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.178+0000 7f7773497640 1 -- start start 2024-03-24T00:43:32.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.179+0000 7f7773497640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f776c0d3ae0 0x7f776c0b5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:32.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.179+0000 7f7773497640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f776c15e280 0x7f776c0b5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:32.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.179+0000 7f7773497640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f776c0d2250 con 0x7f776c15dea0 2024-03-24T00:43:32.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.179+0000 7f7773497640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f776c0d2530 con 0x7f776c0d3ae0 2024-03-24T00:43:32.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.179+0000 7f7773497640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f776c0d2810 con 0x7f776c15e280 2024-03-24T00:43:32.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.179+0000 7f7770a32640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f776c0d3ae0 0x7f776c0b5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:32.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.179+0000 7f7771233640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f776c15dea0 0x7f776c0d33d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58320/0 (socket says 172.21.15.67:58320) 2024-03-24T00:43:32.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.179+0000 7f7771233640 1 -- 172.21.15.67:0/1987618935 learned_addr learned my addr 172.21.15.67:0/1987618935 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:43:32.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.179+0000 7f7771a34640 1 --2- 172.21.15.67:0/1987618935 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f776c15e280 0x7f776c0b5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:32.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.179+0000 7f7770a32640 1 -- 172.21.15.67:0/1987618935 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f776c15e280 msgr2=0x7f776c0b5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:32.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.179+0000 7f7770a32640 1 --2- 172.21.15.67:0/1987618935 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f776c15e280 0x7f776c0b5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:32.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.179+0000 7f7763fff640 1 -- 172.21.15.67:0/1987618935 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1073672437 0 0) 0x7f776c0d2250 con 0x7f776c15dea0 2024-03-24T00:43:32.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.179+0000 7f7770a32640 1 -- 172.21.15.67:0/1987618935 >> v1:172.21.15.67:6789/0 conn(0x7f776c15dea0 legacy=0x7f776c0d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:32.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.179+0000 7f7770a32640 1 -- 172.21.15.67:0/1987618935 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f776c0d2af0 con 0x7f776c0d3ae0 2024-03-24T00:43:32.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.180+0000 7f7770a32640 1 --2- 172.21.15.67:0/1987618935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f776c0d3ae0 0x7f776c0b5a60 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f775c0097d0 tx=0x7f775c02b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2dd4be0f1cfdd4b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:32.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.180+0000 7f7763fff640 1 -- 172.21.15.67:0/1987618935 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f775c031770 con 0x7f776c0d3ae0 2024-03-24T00:43:32.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.180+0000 7f7763fff640 1 -- 172.21.15.67:0/1987618935 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f775c037070 con 0x7f776c0d3ae0 2024-03-24T00:43:32.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.181+0000 7f7763fff640 1 -- 172.21.15.67:0/1987618935 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f775c031b30 con 0x7f776c0d3ae0 2024-03-24T00:43:32.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.181+0000 7f7773497640 1 -- 172.21.15.67:0/1987618935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f776c0d3ae0 msgr2=0x7f776c0b5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:32.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.181+0000 7f7773497640 1 --2- 172.21.15.67:0/1987618935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f776c0d3ae0 0x7f776c0b5a60 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f775c0097d0 tx=0x7f775c02b810 comp rx=0 tx=0).stop 2024-03-24T00:43:32.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.181+0000 7f7773497640 1 -- 172.21.15.67:0/1987618935 shutdown_connections 2024-03-24T00:43:32.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.181+0000 7f7773497640 1 --2- 172.21.15.67:0/1987618935 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f776c15e280 0x7f776c0b5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:32.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.181+0000 7f7773497640 1 --2- 172.21.15.67:0/1987618935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f776c0d3ae0 0x7f776c0b5a60 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:32.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.181+0000 7f7773497640 1 -- 172.21.15.67:0/1987618935 >> 172.21.15.67:0/1987618935 conn(0x7f776c072da0 msgr2=0x7f776c0d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:32.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.181+0000 7f7773497640 1 -- 172.21.15.67:0/1987618935 shutdown_connections 2024-03-24T00:43:32.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.181+0000 7f7773497640 1 -- 172.21.15.67:0/1987618935 wait complete. 2024-03-24T00:43:32.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.183+0000 7f7773497640 1 Processor -- start 2024-03-24T00:43:32.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.183+0000 7f7773497640 1 -- start start 2024-03-24T00:43:32.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.183+0000 7f7773497640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f776c0d3ae0 0x7f776c0c9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:32.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.183+0000 7f7773497640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f776c15dea0 0x7f776c15eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:32.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.183+0000 7f7773497640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f776c15e280 0x7f776c0aa990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:32.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.183+0000 7f7773497640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f776c0b7480 con 0x7f776c0d3ae0 2024-03-24T00:43:32.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.183+0000 7f7773497640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f776c0b7300 con 0x7f776c15e280 2024-03-24T00:43:32.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.183+0000 7f7773497640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f776c0b7600 con 0x7f776c15dea0 2024-03-24T00:43:32.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.184+0000 7f7771233640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f776c0d3ae0 0x7f776c0c9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:32.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.184+0000 7f7771233640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f776c0d3ae0 0x7f776c0c9bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48464/0 (socket says 172.21.15.67:48464) 2024-03-24T00:43:32.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.184+0000 7f7771233640 1 -- 172.21.15.67:0/3651571687 learned_addr learned my addr 172.21.15.67:0/3651571687 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:32.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.184+0000 7f7770a32640 1 --2- 172.21.15.67:0/3651571687 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f776c15dea0 0x7f776c15eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:32.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.184+0000 7f7771a34640 1 --2- 172.21.15.67:0/3651571687 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f776c15e280 0x7f776c0aa990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:32.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.184+0000 7f7771233640 1 -- 172.21.15.67:0/3651571687 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f776c15dea0 msgr2=0x7f776c15eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:32.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.184+0000 7f7771233640 1 --2- 172.21.15.67:0/3651571687 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f776c15dea0 0x7f776c15eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:32.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.184+0000 7f7771233640 1 -- 172.21.15.67:0/3651571687 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f776c15e280 msgr2=0x7f776c0aa990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:32.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.184+0000 7f7771233640 1 --2- 172.21.15.67:0/3651571687 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f776c15e280 0x7f776c0aa990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:32.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.184+0000 7f7771233640 1 -- 172.21.15.67:0/3651571687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f776c14afa0 con 0x7f776c0d3ae0 2024-03-24T00:43:32.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.184+0000 7f7771233640 1 --2- 172.21.15.67:0/3651571687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f776c0d3ae0 0x7f776c0c9bd0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f776800c4d0 tx=0x7f776800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:32.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.184+0000 7f7761ffb640 1 -- 172.21.15.67:0/3651571687 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7768018070 con 0x7f776c0d3ae0 2024-03-24T00:43:32.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.184+0000 7f7761ffb640 1 -- 172.21.15.67:0/3651571687 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7768007500 con 0x7f776c0d3ae0 2024-03-24T00:43:32.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.184+0000 7f7761ffb640 1 -- 172.21.15.67:0/3651571687 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7768004a20 con 0x7f776c0d3ae0 2024-03-24T00:43:32.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.184+0000 7f7773497640 1 -- 172.21.15.67:0/3651571687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f776c14b280 con 0x7f776c0d3ae0 2024-03-24T00:43:32.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.185+0000 7f7773497640 1 -- 172.21.15.67:0/3651571687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f776c14b840 con 0x7f776c0d3ae0 2024-03-24T00:43:32.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.186+0000 7f7773497640 1 -- 172.21.15.67:0/3651571687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f776c04f400 con 0x7f776c0d3ae0 2024-03-24T00:43:32.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.186+0000 7f7761ffb640 1 -- 172.21.15.67:0/3651571687 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f776801b070 con 0x7f776c0d3ae0 2024-03-24T00:43:32.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.186+0000 7f7761ffb640 1 --2- 172.21.15.67:0/3651571687 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f775006bbd0 0x7f775006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:32.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.187+0000 7f7761ffb640 1 -- 172.21.15.67:0/3651571687 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f7768089e70 con 0x7f776c0d3ae0 2024-03-24T00:43:32.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.187+0000 7f7770a32640 1 --2- 172.21.15.67:0/3651571687 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f775006bbd0 0x7f775006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:32.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.187+0000 7f7770a32640 1 --2- 172.21.15.67:0/3651571687 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f775006bbd0 0x7f775006e090 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f775c036000 tx=0x7f775c03d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:32.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.191+0000 7f7761ffb640 1 -- 172.21.15.67:0/3651571687 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f77680581c0 con 0x7f776c0d3ae0 2024-03-24T00:43:32.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.541+0000 7f7773497640 1 -- 172.21.15.67:0/3651571687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f776c0ca900 con 0x7f776c0d3ae0 2024-03-24T00:43:32.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.542+0000 7f7761ffb640 1 -- 172.21.15.67:0/3651571687 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v77) v1 ==== 104+0+11 (secure 0 0 0) 0x7f776805c260 con 0x7f776c0d3ae0 2024-03-24T00:43:32.543 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-24T00:43:32.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.544+0000 7f7773497640 1 -- 172.21.15.67:0/3651571687 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f775006bbd0 msgr2=0x7f775006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:32.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.544+0000 7f7773497640 1 --2- 172.21.15.67:0/3651571687 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f775006bbd0 0x7f775006e090 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f775c036000 tx=0x7f775c03d040 comp rx=0 tx=0).stop 2024-03-24T00:43:32.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.544+0000 7f7773497640 1 -- 172.21.15.67:0/3651571687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f776c0d3ae0 msgr2=0x7f776c0c9bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:32.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.544+0000 7f7773497640 1 --2- 172.21.15.67:0/3651571687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f776c0d3ae0 0x7f776c0c9bd0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f776800c4d0 tx=0x7f776800c990 comp rx=0 tx=0).stop 2024-03-24T00:43:32.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.544+0000 7f7773497640 1 -- 172.21.15.67:0/3651571687 shutdown_connections 2024-03-24T00:43:32.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.544+0000 7f7773497640 1 --2- 172.21.15.67:0/3651571687 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f775006bbd0 0x7f775006e090 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:32.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.544+0000 7f7773497640 1 --2- 172.21.15.67:0/3651571687 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f776c15e280 0x7f776c0aa990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:32.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.544+0000 7f7773497640 1 --2- 172.21.15.67:0/3651571687 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f776c15dea0 0x7f776c15eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:32.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.544+0000 7f7773497640 1 --2- 172.21.15.67:0/3651571687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f776c0d3ae0 0x7f776c0c9bd0 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:32.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.544+0000 7f7773497640 1 -- 172.21.15.67:0/3651571687 >> 172.21.15.67:0/3651571687 conn(0x7f776c072da0 msgr2=0x7f776c0b6950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:32.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.544+0000 7f7773497640 1 -- 172.21.15.67:0/3651571687 shutdown_connections 2024-03-24T00:43:32.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.544+0000 7f7773497640 1 -- 172.21.15.67:0/3651571687 wait complete. 2024-03-24T00:43:32.555 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:43:32.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.674+0000 7f67117b2640 1 Processor -- start 2024-03-24T00:43:32.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.674+0000 7f67117b2640 1 -- start start 2024-03-24T00:43:32.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.675+0000 7f67117b2640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f670c0b49b0 0x7f670c0d0e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:32.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.675+0000 7f67117b2640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f670c0d1360 0x7f670c15d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:32.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.675+0000 7f67117b2640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f670c0cc400 con 0x7f670c099d00 2024-03-24T00:43:32.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.675+0000 7f67117b2640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f670c0cc1b0 con 0x7f670c0d1360 2024-03-24T00:43:32.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.675+0000 7f67117b2640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f670c0cc6e0 con 0x7f670c0b49b0 2024-03-24T00:43:32.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.675+0000 7f670affd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f670c099d00 0x7f670c09a0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58330/0 (socket says 172.21.15.67:58330) 2024-03-24T00:43:32.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.675+0000 7f670affd640 1 -- 172.21.15.67:0/1340149945 learned_addr learned my addr 172.21.15.67:0/1340149945 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:43:32.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.675+0000 7f670b7fe640 1 --2- 172.21.15.67:0/1340149945 >> v2:172.21.15.67:3300/0 conn(0x7f670c0d1360 0x7f670c15d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:32.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.675+0000 7f670a7fc640 1 --2- 172.21.15.67:0/1340149945 >> v2:172.21.15.72:3300/0 conn(0x7f670c0b49b0 0x7f670c0d0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:32.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.675+0000 7f6709ffb640 1 -- 172.21.15.67:0/1340149945 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1911497107 0 0) 0x7f670c0cc400 con 0x7f670c099d00 2024-03-24T00:43:32.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.675+0000 7f6709ffb640 1 -- 172.21.15.67:0/1340149945 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66f4003520 con 0x7f670c099d00 2024-03-24T00:43:32.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.675+0000 7f670b7fe640 1 -- 172.21.15.67:0/1340149945 >> v2:172.21.15.72:3300/0 conn(0x7f670c0b49b0 msgr2=0x7f670c0d0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:32.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.675+0000 7f670b7fe640 1 --2- 172.21.15.67:0/1340149945 >> v2:172.21.15.72:3300/0 conn(0x7f670c0b49b0 0x7f670c0d0e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:32.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.675+0000 7f670b7fe640 1 -- 172.21.15.67:0/1340149945 >> v1:172.21.15.67:6789/0 conn(0x7f670c099d00 legacy=0x7f670c09a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:32.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.675+0000 7f670b7fe640 1 -- 172.21.15.67:0/1340149945 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f670c0cc9c0 con 0x7f670c0d1360 2024-03-24T00:43:32.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.676+0000 7f670b7fe640 1 --2- 172.21.15.67:0/1340149945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f670c0d1360 0x7f670c15d7d0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f670c073100 tx=0x7f670402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d38cc2002e1ce10 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:32.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.676+0000 7f6709ffb640 1 -- 172.21.15.67:0/1340149945 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6704039070 con 0x7f670c0d1360 2024-03-24T00:43:32.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.676+0000 7f6709ffb640 1 -- 172.21.15.67:0/1340149945 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f670402cc80 con 0x7f670c0d1360 2024-03-24T00:43:32.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.676+0000 7f6709ffb640 1 -- 172.21.15.67:0/1340149945 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6704034660 con 0x7f670c0d1360 2024-03-24T00:43:32.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.677+0000 7f67117b2640 1 -- 172.21.15.67:0/1340149945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f670c0d1360 msgr2=0x7f670c15d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:32.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.677+0000 7f67117b2640 1 --2- 172.21.15.67:0/1340149945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f670c0d1360 0x7f670c15d7d0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f670c073100 tx=0x7f670402bed0 comp rx=0 tx=0).stop 2024-03-24T00:43:32.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.677+0000 7f67117b2640 1 -- 172.21.15.67:0/1340149945 shutdown_connections 2024-03-24T00:43:32.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.677+0000 7f67117b2640 1 --2- 172.21.15.67:0/1340149945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f670c0d1360 0x7f670c15d7d0 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:32.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.677+0000 7f67117b2640 1 --2- 172.21.15.67:0/1340149945 >> v2:172.21.15.72:3300/0 conn(0x7f670c0b49b0 0x7f670c0d0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:32.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.677+0000 7f67117b2640 1 -- 172.21.15.67:0/1340149945 >> 172.21.15.67:0/1340149945 conn(0x7f670c0b3650 msgr2=0x7f670c0b3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:32.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.677+0000 7f67117b2640 1 -- 172.21.15.67:0/1340149945 shutdown_connections 2024-03-24T00:43:32.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.677+0000 7f67117b2640 1 -- 172.21.15.67:0/1340149945 wait complete. 2024-03-24T00:43:32.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.678+0000 7f67117b2640 1 Processor -- start 2024-03-24T00:43:32.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.678+0000 7f67117b2640 1 -- start start 2024-03-24T00:43:32.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.679+0000 7f67117b2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f670c099d00 0x7f670c0c7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:32.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.679+0000 7f67117b2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f670c0b49b0 0x7f670c0cb240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:32.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.679+0000 7f67117b2640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f670c0d1360 0x7f670c152ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:32.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.679+0000 7f67117b2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f670c0b32a0 con 0x7f670c0b49b0 2024-03-24T00:43:32.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.679+0000 7f67117b2640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f670c0d01c0 con 0x7f670c099d00 2024-03-24T00:43:32.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.679+0000 7f67117b2640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f670c0b3420 con 0x7f670c0d1360 2024-03-24T00:43:32.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.679+0000 7f670a7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f670c0b49b0 0x7f670c0cb240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:32.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.679+0000 7f670a7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f670c0b49b0 0x7f670c0cb240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48472/0 (socket says 172.21.15.67:48472) 2024-03-24T00:43:32.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.679+0000 7f670a7fc640 1 -- 172.21.15.67:0/981355373 learned_addr learned my addr 172.21.15.67:0/981355373 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:32.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.679+0000 7f670affd640 1 --2- 172.21.15.67:0/981355373 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f670c099d00 0x7f670c0c7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:32.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.679+0000 7f670b7fe640 1 --2- 172.21.15.67:0/981355373 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f670c0d1360 0x7f670c152ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:32.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.679+0000 7f670a7fc640 1 -- 172.21.15.67:0/981355373 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f670c0d1360 msgr2=0x7f670c152ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:32.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.679+0000 7f670a7fc640 1 --2- 172.21.15.67:0/981355373 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f670c0d1360 0x7f670c152ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:32.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.679+0000 7f670a7fc640 1 -- 172.21.15.67:0/981355373 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f670c099d00 msgr2=0x7f670c0c7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:32.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.679+0000 7f670a7fc640 1 --2- 172.21.15.67:0/981355373 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f670c099d00 0x7f670c0c7fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:32.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.679+0000 7f670a7fc640 1 -- 172.21.15.67:0/981355373 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f670c0cb9d0 con 0x7f670c0b49b0 2024-03-24T00:43:32.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.679+0000 7f670a7fc640 1 --2- 172.21.15.67:0/981355373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f670c0b49b0 0x7f670c0cb240 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f670000d800 tx=0x7f670000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:32.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.679+0000 7f66f37fe640 1 -- 172.21.15.67:0/981355373 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6700004020 con 0x7f670c0b49b0 2024-03-24T00:43:32.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.680+0000 7f66f37fe640 1 -- 172.21.15.67:0/981355373 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67000041c0 con 0x7f670c0b49b0 2024-03-24T00:43:32.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.680+0000 7f66f37fe640 1 -- 172.21.15.67:0/981355373 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6700004390 con 0x7f670c0b49b0 2024-03-24T00:43:32.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.680+0000 7f67117b2640 1 -- 172.21.15.67:0/981355373 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f670c0cbcb0 con 0x7f670c0b49b0 2024-03-24T00:43:32.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.680+0000 7f67117b2640 1 -- 172.21.15.67:0/981355373 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f670c0c6be0 con 0x7f670c0b49b0 2024-03-24T00:43:32.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.681+0000 7f67117b2640 1 -- 172.21.15.67:0/981355373 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f670c04f400 con 0x7f670c0b49b0 2024-03-24T00:43:32.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.682+0000 7f66f37fe640 1 -- 172.21.15.67:0/981355373 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f670001b020 con 0x7f670c0b49b0 2024-03-24T00:43:32.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.682+0000 7f66f37fe640 1 --2- 172.21.15.67:0/981355373 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f66f806bbd0 0x7f66f806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:32.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.682+0000 7f66f37fe640 1 -- 172.21.15.67:0/981355373 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f670008a9c0 con 0x7f670c0b49b0 2024-03-24T00:43:32.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.682+0000 7f670affd640 1 --2- 172.21.15.67:0/981355373 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f66f806bbd0 0x7f66f806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:32.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.683+0000 7f670affd640 1 --2- 172.21.15.67:0/981355373 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f66f806bbd0 0x7f66f806e090 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f66fc005880 tx=0x7f66fc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:32.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.687+0000 7f66f37fe640 1 -- 172.21.15.67:0/981355373 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f670008ce20 con 0x7f670c0b49b0 2024-03-24T00:43:32.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.868+0000 7f67117b2640 1 -- 172.21.15.67:0/981355373 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f670c1617d0 con 0x7f670c0b49b0 2024-03-24T00:43:32.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.868+0000 7f66f37fe640 1 -- 172.21.15.67:0/981355373 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v77) v1 ==== 100+0+11 (secure 0 0 0) 0x7f67000592d0 con 0x7f670c0b49b0 2024-03-24T00:43:32.869 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-03-24T00:43:32.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.870+0000 7f67117b2640 1 -- 172.21.15.67:0/981355373 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f66f806bbd0 msgr2=0x7f66f806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:32.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.870+0000 7f67117b2640 1 --2- 172.21.15.67:0/981355373 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f66f806bbd0 0x7f66f806e090 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f66fc005880 tx=0x7f66fc009040 comp rx=0 tx=0).stop 2024-03-24T00:43:32.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.870+0000 7f67117b2640 1 -- 172.21.15.67:0/981355373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f670c0b49b0 msgr2=0x7f670c0cb240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:32.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.870+0000 7f67117b2640 1 --2- 172.21.15.67:0/981355373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f670c0b49b0 0x7f670c0cb240 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f670000d800 tx=0x7f670000dcc0 comp rx=0 tx=0).stop 2024-03-24T00:43:32.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.870+0000 7f67117b2640 1 -- 172.21.15.67:0/981355373 shutdown_connections 2024-03-24T00:43:32.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.870+0000 7f67117b2640 1 --2- 172.21.15.67:0/981355373 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f66f806bbd0 0x7f66f806e090 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:32.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.870+0000 7f67117b2640 1 --2- 172.21.15.67:0/981355373 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f670c0d1360 0x7f670c152ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:32.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.870+0000 7f67117b2640 1 --2- 172.21.15.67:0/981355373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f670c0b49b0 0x7f670c0cb240 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:32.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.870+0000 7f67117b2640 1 --2- 172.21.15.67:0/981355373 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f670c099d00 0x7f670c0c7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:32.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.870+0000 7f67117b2640 1 -- 172.21.15.67:0/981355373 >> 172.21.15.67:0/981355373 conn(0x7f670c0b3650 msgr2=0x7f670c071270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:32.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.870+0000 7f67117b2640 1 -- 172.21.15.67:0/981355373 shutdown_connections 2024-03-24T00:43:32.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.870+0000 7f67117b2640 1 -- 172.21.15.67:0/981355373 wait complete. 2024-03-24T00:43:32.882 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T00:43:32.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.983+0000 7fa0cd7f1640 1 Processor -- start 2024-03-24T00:43:32.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.983+0000 7fa0cd7f1640 1 -- start start 2024-03-24T00:43:32.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.984+0000 7fa0cd7f1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa0c80d26d0 0x7fa0c806c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:32.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.984+0000 7fa0cd7f1640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0c80d0270 con 0x7fa0c806d230 2024-03-24T00:43:32.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.984+0000 7fa0cd7f1640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0c80d0550 con 0x7fa0c80ceb10 2024-03-24T00:43:32.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.984+0000 7fa0cd7f1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c80d0e80 con 0x7fa0c80d26d0 2024-03-24T00:43:32.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.984+0000 7fa0c77fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa0c806d230 0x7fa0c815d740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58344/0 (socket says 172.21.15.67:58344) 2024-03-24T00:43:32.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.984+0000 7fa0c77fe640 1 -- 172.21.15.67:0/1687497686 learned_addr learned my addr 172.21.15.67:0/1687497686 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:43:32.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.985+0000 7fa0c67fc640 1 --2- 172.21.15.67:0/1687497686 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa0c80d26d0 0x7fa0c806c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:32.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.985+0000 7fa0c5ffb640 1 -- 172.21.15.67:0/1687497686 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2617436854 0 0) 0x7fa0c80d0270 con 0x7fa0c806d230 2024-03-24T00:43:32.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.985+0000 7fa0c5ffb640 1 -- 172.21.15.67:0/1687497686 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0b0003520 con 0x7fa0c806d230 2024-03-24T00:43:32.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.985+0000 7fa0c5ffb640 1 -- 172.21.15.67:0/1687497686 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2616323790 0 0) 0x7fa0b0003520 con 0x7fa0c806d230 2024-03-24T00:43:32.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.985+0000 7fa0c5ffb640 1 -- 172.21.15.67:0/1687497686 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa0c80d26d0 msgr2=0x7fa0c806c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:32.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.985+0000 7fa0c5ffb640 1 --2- 172.21.15.67:0/1687497686 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa0c80d26d0 0x7fa0c806c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:32.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.985+0000 7fa0c5ffb640 1 -- 172.21.15.67:0/1687497686 >> v1:172.21.15.72:6789/0 conn(0x7fa0c80ceb10 legacy=0x7fa0c806cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:32.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.985+0000 7fa0c5ffb640 1 -- 172.21.15.67:0/1687497686 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c80d0830 con 0x7fa0c806d230 2024-03-24T00:43:32.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.985+0000 7fa0c5ffb640 1 -- 172.21.15.67:0/1687497686 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7fa0c00019f0 con 0x7fa0c806d230 2024-03-24T00:43:32.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.985+0000 7fa0c5ffb640 1 -- 172.21.15.67:0/1687497686 >> v1:172.21.15.67:6789/0 conn(0x7fa0c806d230 legacy=0x7fa0c815d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:32.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.985+0000 7fa0c5ffb640 1 --2- 172.21.15.67:0/1687497686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0b00039e0 0x7fa0b0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:32.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.985+0000 7fa0c5ffb640 1 --2- 172.21.15.67:0/1687497686 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa0b0004390 0x7fa0b0004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:32.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.985+0000 7fa0c5ffb640 1 --2- 172.21.15.67:0/1687497686 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa0c80d26d0 0x7fa0b00050c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:32.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.985+0000 7fa0c5ffb640 1 -- 172.21.15.67:0/1687497686 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c80d0e80 con 0x7fa0b00039e0 2024-03-24T00:43:32.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.985+0000 7fa0c5ffb640 1 -- 172.21.15.67:0/1687497686 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b000b9c0 con 0x7fa0b0004390 2024-03-24T00:43:32.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.985+0000 7fa0c5ffb640 1 -- 172.21.15.67:0/1687497686 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b000bbc0 con 0x7fa0c80d26d0 2024-03-24T00:43:32.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.985+0000 7fa0c6ffd640 1 --2- 172.21.15.67:0/1687497686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0b00039e0 0x7fa0b0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:32.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.985+0000 7fa0c67fc640 1 --2- 172.21.15.67:0/1687497686 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa0b0004390 0x7fa0b0004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:32.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.985+0000 7fa0c77fe640 1 --2- 172.21.15.67:0/1687497686 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa0c80d26d0 0x7fa0b00050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:32.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.985+0000 7fa0c6ffd640 1 -- 172.21.15.67:0/1687497686 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa0c80d26d0 msgr2=0x7fa0b00050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:32.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.985+0000 7fa0c6ffd640 1 --2- 172.21.15.67:0/1687497686 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa0c80d26d0 0x7fa0b00050c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:32.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.985+0000 7fa0c6ffd640 1 -- 172.21.15.67:0/1687497686 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa0b0004390 msgr2=0x7fa0b0004860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:32.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.985+0000 7fa0c6ffd640 1 --2- 172.21.15.67:0/1687497686 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa0b0004390 0x7fa0b0004860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:32.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.985+0000 7fa0c6ffd640 1 -- 172.21.15.67:0/1687497686 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b0003520 con 0x7fa0b00039e0 2024-03-24T00:43:32.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.985+0000 7fa0c77fe640 1 --2- 172.21.15.67:0/1687497686 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa0c80d26d0 0x7fa0b00050c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:43:32.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.986+0000 7fa0c6ffd640 1 --2- 172.21.15.67:0/1687497686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0b00039e0 0x7fa0b0003dc0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8001860 tx=0x7fa0b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d6fe205c3062a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:32.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.986+0000 7fa0c5ffb640 1 -- 172.21.15.67:0/1687497686 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b802f050 con 0x7fa0b00039e0 2024-03-24T00:43:32.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.986+0000 7fa0c5ffb640 1 -- 172.21.15.67:0/1687497686 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0b8033650 con 0x7fa0b00039e0 2024-03-24T00:43:32.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.986+0000 7fa0c5ffb640 1 -- 172.21.15.67:0/1687497686 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b8001430 con 0x7fa0b00039e0 2024-03-24T00:43:32.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.987+0000 7fa0cd7f1640 1 -- 172.21.15.67:0/1687497686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0b00039e0 msgr2=0x7fa0b0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:32.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.987+0000 7fa0cd7f1640 1 --2- 172.21.15.67:0/1687497686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0b00039e0 0x7fa0b0003dc0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8001860 tx=0x7fa0b0005830 comp rx=0 tx=0).stop 2024-03-24T00:43:32.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.987+0000 7fa0c77fe640 1 -- 172.21.15.67:0/1687497686 reap_dead start 2024-03-24T00:43:32.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.987+0000 7fa0cd7f1640 1 -- 172.21.15.67:0/1687497686 shutdown_connections 2024-03-24T00:43:32.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.987+0000 7fa0cd7f1640 1 -- 172.21.15.67:0/1687497686 >> 172.21.15.67:0/1687497686 conn(0x7fa0c80d1ad0 msgr2=0x7fa0c80d1ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:32.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.987+0000 7fa0cd7f1640 1 -- 172.21.15.67:0/1687497686 shutdown_connections 2024-03-24T00:43:32.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.987+0000 7fa0cd7f1640 1 -- 172.21.15.67:0/1687497686 wait complete. 2024-03-24T00:43:32.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.988+0000 7fa0cd7f1640 1 Processor -- start 2024-03-24T00:43:32.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.988+0000 7fa0cd7f1640 1 -- start start 2024-03-24T00:43:32.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.989+0000 7fa0cd7f1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa0b00039e0 0x7fa0c80c5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:32.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.989+0000 7fa0cd7f1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0c8149d40 0x7fa0c815aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:32.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.989+0000 7fa0cd7f1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa0c8149430 0x7fa0c815fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:32.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.989+0000 7fa0cd7f1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c80cf070 con 0x7fa0c8149d40 2024-03-24T00:43:32.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.989+0000 7fa0cd7f1640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c80ceef0 con 0x7fa0b00039e0 2024-03-24T00:43:32.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.989+0000 7fa0cd7f1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c80cfe00 con 0x7fa0c8149430 2024-03-24T00:43:32.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.989+0000 7fa0c67fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0c8149d40 0x7fa0c815aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:32.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.989+0000 7fa0c67fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0c8149d40 0x7fa0c815aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48480/0 (socket says 172.21.15.67:48480) 2024-03-24T00:43:32.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.989+0000 7fa0c67fc640 1 -- 172.21.15.67:0/380855786 learned_addr learned my addr 172.21.15.67:0/380855786 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:32.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.989+0000 7fa0c6ffd640 1 --2- 172.21.15.67:0/380855786 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa0b00039e0 0x7fa0c80c5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:32.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.989+0000 7fa0c77fe640 1 --2- 172.21.15.67:0/380855786 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa0c8149430 0x7fa0c815fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:32.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.989+0000 7fa0c67fc640 1 -- 172.21.15.67:0/380855786 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa0c8149430 msgr2=0x7fa0c815fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:32.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.989+0000 7fa0c67fc640 1 --2- 172.21.15.67:0/380855786 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa0c8149430 0x7fa0c815fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:32.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.989+0000 7fa0c67fc640 1 -- 172.21.15.67:0/380855786 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa0b00039e0 msgr2=0x7fa0c80c5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:32.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.989+0000 7fa0c67fc640 1 --2- 172.21.15.67:0/380855786 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa0b00039e0 0x7fa0c80c5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:32.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.989+0000 7fa0c67fc640 1 -- 172.21.15.67:0/380855786 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c81599f0 con 0x7fa0c8149d40 2024-03-24T00:43:32.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.989+0000 7fa0c77fe640 1 --2- 172.21.15.67:0/380855786 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa0c8149430 0x7fa0c815fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:43:32.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.989+0000 7fa0c67fc640 1 --2- 172.21.15.67:0/380855786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0c8149d40 0x7fa0c815aa50 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc0119d0 tx=0x7fa0bc011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:32.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.990+0000 7fa0af7fe640 1 -- 172.21.15.67:0/380855786 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc00fad0 con 0x7fa0c8149d40 2024-03-24T00:43:32.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.990+0000 7fa0af7fe640 1 -- 172.21.15.67:0/380855786 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0bc00fc70 con 0x7fa0c8149d40 2024-03-24T00:43:32.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.990+0000 7fa0af7fe640 1 -- 172.21.15.67:0/380855786 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc014440 con 0x7fa0c8149d40 2024-03-24T00:43:32.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.990+0000 7fa0cd7f1640 1 -- 172.21.15.67:0/380855786 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0c8159c90 con 0x7fa0c8149d40 2024-03-24T00:43:32.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.990+0000 7fa0cd7f1640 1 -- 172.21.15.67:0/380855786 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0c815a250 con 0x7fa0c8149d40 2024-03-24T00:43:32.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.991+0000 7fa0cd7f1640 1 -- 172.21.15.67:0/380855786 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0c804f400 con 0x7fa0c8149d40 2024-03-24T00:43:32.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.991+0000 7fa0af7fe640 1 -- 172.21.15.67:0/380855786 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa0bc012040 con 0x7fa0c8149d40 2024-03-24T00:43:32.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.991+0000 7fa0af7fe640 1 --2- 172.21.15.67:0/380855786 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa0b406bc00 0x7fa0b406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:32.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.992+0000 7fa0af7fe640 1 -- 172.21.15.67:0/380855786 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fa0bc08b080 con 0x7fa0c8149d40 2024-03-24T00:43:32.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.992+0000 7fa0c6ffd640 1 --2- 172.21.15.67:0/380855786 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa0b406bc00 0x7fa0b406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:32.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.992+0000 7fa0c6ffd640 1 --2- 172.21.15.67:0/380855786 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa0b406bc00 0x7fa0b406e0c0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8001860 tx=0x7fa0b8031000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:32.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:32.996+0000 7fa0af7fe640 1 -- 172.21.15.67:0/380855786 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa0bc019030 con 0x7fa0c8149d40 2024-03-24T00:43:33.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.203+0000 7fa0cd7f1640 1 -- 172.21.15.67:0/380855786 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fa0c80cf460 con 0x7fa0c8149d40 2024-03-24T00:43:33.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.203+0000 7fa0af7fe640 1 -- 172.21.15.67:0/380855786 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v77) v1 ==== 80+0+122 (secure 0 0 0) 0x7fa0bc059990 con 0x7fa0c8149d40 2024-03-24T00:43:33.205 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6833/2392802741 2024-03-25T00:43:24.373788+0000 2024-03-24T00:43:33.205 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6832/2392802741 2024-03-25T00:43:24.373788+0000 2024-03-24T00:43:33.205 INFO:teuthology.orchestra.run.smithi067.stderr:listed 2 entries 2024-03-24T00:43:33.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.205+0000 7fa0cd7f1640 1 -- 172.21.15.67:0/380855786 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa0b406bc00 msgr2=0x7fa0b406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:33.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.205+0000 7fa0cd7f1640 1 --2- 172.21.15.67:0/380855786 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa0b406bc00 0x7fa0b406e0c0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8001860 tx=0x7fa0b8031000 comp rx=0 tx=0).stop 2024-03-24T00:43:33.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.205+0000 7fa0cd7f1640 1 -- 172.21.15.67:0/380855786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0c8149d40 msgr2=0x7fa0c815aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:33.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.205+0000 7fa0cd7f1640 1 --2- 172.21.15.67:0/380855786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0c8149d40 0x7fa0c815aa50 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc0119d0 tx=0x7fa0bc011e90 comp rx=0 tx=0).stop 2024-03-24T00:43:33.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.205+0000 7fa0cd7f1640 1 -- 172.21.15.67:0/380855786 shutdown_connections 2024-03-24T00:43:33.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.205+0000 7fa0cd7f1640 1 --2- 172.21.15.67:0/380855786 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa0b406bc00 0x7fa0b406e0c0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.205+0000 7fa0cd7f1640 1 --2- 172.21.15.67:0/380855786 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa0c8149430 0x7fa0c815fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.205+0000 7fa0cd7f1640 1 --2- 172.21.15.67:0/380855786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0c8149d40 0x7fa0c815aa50 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.205+0000 7fa0cd7f1640 1 --2- 172.21.15.67:0/380855786 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa0b00039e0 0x7fa0c80c5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.205+0000 7fa0cd7f1640 1 -- 172.21.15.67:0/380855786 >> 172.21.15.67:0/380855786 conn(0x7fa0c80d1ad0 msgr2=0x7fa0c80ccf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:33.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.205+0000 7fa0cd7f1640 1 -- 172.21.15.67:0/380855786 shutdown_connections 2024-03-24T00:43:33.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.205+0000 7fa0cd7f1640 1 -- 172.21.15.67:0/380855786 wait complete. 2024-03-24T00:43:33.217 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-24T00:43:33.217 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:43:33.217 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-24T00:43:33.247 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-03-24T00:43:33.247 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:43:33.247 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-24T00:43:33.302 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:43:33.302 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.5:0 scrub start /subdir recursive 2024-03-24T00:43:33.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.450+0000 7ff51b909640 1 Processor -- start 2024-03-24T00:43:33.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.450+0000 7ff51b909640 1 -- start start 2024-03-24T00:43:33.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.451+0000 7ff51b909640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff51415dfc0 0x7ff5141624c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:33.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.451+0000 7ff51b909640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff5140d0d50 0x7ff5140d2b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:33.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.451+0000 7ff51b909640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff514162a00 con 0x7ff5140ced60 2024-03-24T00:43:33.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.451+0000 7ff51b909640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff51415e3a0 con 0x7ff5140d0d50 2024-03-24T00:43:33.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.451+0000 7ff51b909640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5140cf4d0 con 0x7ff51415dfc0 2024-03-24T00:43:33.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.451+0000 7ff5196a5640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff51415dfc0 0x7ff5141624c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:33.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.451+0000 7ff519ea6640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff5140d0d50 0x7ff5140d2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:33.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.451+0000 7ff5196a5640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff51415dfc0 0x7ff5141624c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.67:42922/0 (socket says 172.21.15.67:42922) 2024-03-24T00:43:33.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.451+0000 7ff5196a5640 1 -- 172.21.15.67:0/3619307537 learned_addr learned my addr 172.21.15.67:0/3619307537 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:33.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.451+0000 7ff518ea4640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7ff5140ced60 0x7ff5140d0640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:52934/0 (socket says 172.21.15.67:52934) 2024-03-24T00:43:33.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.452+0000 7ff5196a5640 1 -- 172.21.15.67:0/3619307537 >> v2:172.21.15.72:3300/0 conn(0x7ff5140d0d50 msgr2=0x7ff5140d2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:33.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.452+0000 7ff5196a5640 1 --2- 172.21.15.67:0/3619307537 >> v2:172.21.15.72:3300/0 conn(0x7ff5140d0d50 0x7ff5140d2b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.452+0000 7ff5196a5640 1 -- 172.21.15.67:0/3619307537 >> v1:172.21.15.72:6789/0 conn(0x7ff5140ced60 legacy=0x7ff5140d0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:33.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.452+0000 7ff5196a5640 1 -- 172.21.15.67:0/3619307537 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5140cf1f0 con 0x7ff51415dfc0 2024-03-24T00:43:33.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.453+0000 7ff5196a5640 1 --2- 172.21.15.67:0/3619307537 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff51415dfc0 0x7ff5141624c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff504009870 tx=0x7ff50402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9fb02a614ed7820 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:33.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.453+0000 7ff50bfff640 1 -- 172.21.15.67:0/3619307537 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff504034070 con 0x7ff51415dfc0 2024-03-24T00:43:33.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.453+0000 7ff50bfff640 1 -- 172.21.15.67:0/3619307537 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff504030840 con 0x7ff51415dfc0 2024-03-24T00:43:33.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.453+0000 7ff50bfff640 1 -- 172.21.15.67:0/3619307537 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff504030c10 con 0x7ff51415dfc0 2024-03-24T00:43:33.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.453+0000 7ff51b909640 1 -- 172.21.15.67:0/3619307537 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff51415dfc0 msgr2=0x7ff5141624c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:33.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.453+0000 7ff51b909640 1 --2- 172.21.15.67:0/3619307537 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff51415dfc0 0x7ff5141624c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff504009870 tx=0x7ff50402b980 comp rx=0 tx=0).stop 2024-03-24T00:43:33.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.453+0000 7ff51b909640 1 -- 172.21.15.67:0/3619307537 shutdown_connections 2024-03-24T00:43:33.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.453+0000 7ff51b909640 1 --2- 172.21.15.67:0/3619307537 >> v2:172.21.15.72:3300/0 conn(0x7ff5140d0d50 0x7ff5140d2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.453+0000 7ff51b909640 1 --2- 172.21.15.67:0/3619307537 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff51415dfc0 0x7ff5141624c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.453+0000 7ff51b909640 1 -- 172.21.15.67:0/3619307537 >> 172.21.15.67:0/3619307537 conn(0x7ff51406c880 msgr2=0x7ff51406cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:33.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.454+0000 7ff51b909640 1 -- 172.21.15.67:0/3619307537 shutdown_connections 2024-03-24T00:43:33.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.454+0000 7ff51b909640 1 -- 172.21.15.67:0/3619307537 wait complete. 2024-03-24T00:43:33.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.455+0000 7ff51b909640 1 Processor -- start 2024-03-24T00:43:33.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.455+0000 7ff51b909640 1 -- start start 2024-03-24T00:43:33.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.455+0000 7ff51b909640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff5140d0d50 0x7ff5140b4c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:33.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.455+0000 7ff51b909640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff51415dfc0 0x7ff5141502f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:33.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.455+0000 7ff51b909640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff5140cbcd0 0x7ff514158c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:33.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.455+0000 7ff51b909640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5140d25e0 con 0x7ff51415dfc0 2024-03-24T00:43:33.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.455+0000 7ff51b909640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5140d2300 con 0x7ff5140cbcd0 2024-03-24T00:43:33.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.455+0000 7ff51b909640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5140b6ea0 con 0x7ff5140d0d50 2024-03-24T00:43:33.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.456+0000 7ff518ea4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff51415dfc0 0x7ff5141502f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:33.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.456+0000 7ff518ea4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff51415dfc0 0x7ff5141502f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48486/0 (socket says 172.21.15.67:48486) 2024-03-24T00:43:33.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.456+0000 7ff518ea4640 1 -- 172.21.15.67:0/2901209986 learned_addr learned my addr 172.21.15.67:0/2901209986 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:33.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.456+0000 7ff5196a5640 1 --2- 172.21.15.67:0/2901209986 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff5140d0d50 0x7ff5140b4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:33.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.456+0000 7ff519ea6640 1 --2- 172.21.15.67:0/2901209986 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff5140cbcd0 0x7ff514158c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:33.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.456+0000 7ff518ea4640 1 -- 172.21.15.67:0/2901209986 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff5140d0d50 msgr2=0x7ff5140b4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:33.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.456+0000 7ff518ea4640 1 --2- 172.21.15.67:0/2901209986 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff5140d0d50 0x7ff5140b4c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.456+0000 7ff518ea4640 1 -- 172.21.15.67:0/2901209986 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff5140cbcd0 msgr2=0x7ff514158c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:33.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.456+0000 7ff518ea4640 1 --2- 172.21.15.67:0/2901209986 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff5140cbcd0 0x7ff514158c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.456+0000 7ff518ea4640 1 -- 172.21.15.67:0/2901209986 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5140b5310 con 0x7ff51415dfc0 2024-03-24T00:43:33.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.456+0000 7ff5196a5640 1 --2- 172.21.15.67:0/2901209986 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff5140d0d50 0x7ff5140b4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:43:33.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.456+0000 7ff518ea4640 1 --2- 172.21.15.67:0/2901209986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff51415dfc0 0x7ff5141502f0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7ff51000c430 tx=0x7ff51000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:33.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.456+0000 7ff509ffb640 1 -- 172.21.15.67:0/2901209986 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5100074e0 con 0x7ff51415dfc0 2024-03-24T00:43:33.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.456+0000 7ff509ffb640 1 -- 172.21.15.67:0/2901209986 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff510007680 con 0x7ff51415dfc0 2024-03-24T00:43:33.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.456+0000 7ff51b909640 1 -- 172.21.15.67:0/2901209986 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5140b55f0 con 0x7ff51415dfc0 2024-03-24T00:43:33.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.456+0000 7ff509ffb640 1 -- 172.21.15.67:0/2901209986 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff510004a00 con 0x7ff51415dfc0 2024-03-24T00:43:33.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.456+0000 7ff51b909640 1 -- 172.21.15.67:0/2901209986 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5140b5bb0 con 0x7ff51415dfc0 2024-03-24T00:43:33.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.458+0000 7ff509ffb640 1 -- 172.21.15.67:0/2901209986 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff51001a020 con 0x7ff51415dfc0 2024-03-24T00:43:33.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.458+0000 7ff509ffb640 1 --2- 172.21.15.67:0/2901209986 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff4f806bbd0 0x7ff4f806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:33.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.458+0000 7ff509ffb640 1 -- 172.21.15.67:0/2901209986 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7ff51008a240 con 0x7ff51415dfc0 2024-03-24T00:43:33.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.459+0000 7ff5196a5640 1 --2- 172.21.15.67:0/2901209986 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff4f806bbd0 0x7ff4f806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:33.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.459+0000 7ff5196a5640 1 --2- 172.21.15.67:0/2901209986 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff4f806bbd0 0x7ff4f806e090 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7ff504036000 tx=0x7ff50403b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:33.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.460+0000 7ff51e5d1740 1 Processor -- start 2024-03-24T00:43:33.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.460+0000 7ff51e5d1740 1 -- start start 2024-03-24T00:43:33.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.461+0000 7ff51e5d1740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d7c0402cd0 0x55d7c04030b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:33.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.462+0000 7ff51e5d1740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d7c03d2cf0 0x55d7c03fff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:33.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.462+0000 7ff51e5d1740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d7c03d6850 0x55d7c0400480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:33.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.462+0000 7ff51e5d1740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7c033d630 con 0x55d7c03d6850 2024-03-24T00:43:33.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.462+0000 7ff51e5d1740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7c02e5d70 con 0x55d7c03d2cf0 2024-03-24T00:43:33.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.462+0000 7ff51e5d1740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d7c03396d0 con 0x55d7c0402cd0 2024-03-24T00:43:33.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.462+0000 7ff519ea6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d7c03d6850 0x55d7c0400480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:33.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.462+0000 7ff519ea6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d7c03d6850 0x55d7c0400480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48494/0 (socket says 172.21.15.67:48494) 2024-03-24T00:43:33.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.462+0000 7ff519ea6640 1 -- 172.21.15.67:0/3877688967 learned_addr learned my addr 172.21.15.67:0/3877688967 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:33.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.462+0000 7ff518ea4640 1 --2- 172.21.15.67:0/3877688967 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d7c03d2cf0 0x55d7c03fff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:33.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.462+0000 7ff5196a5640 1 --2- 172.21.15.67:0/3877688967 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d7c0402cd0 0x55d7c04030b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:33.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.462+0000 7ff519ea6640 1 -- 172.21.15.67:0/3877688967 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d7c0402cd0 msgr2=0x55d7c04030b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:33.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.462+0000 7ff519ea6640 1 --2- 172.21.15.67:0/3877688967 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d7c0402cd0 0x55d7c04030b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.462+0000 7ff519ea6640 1 -- 172.21.15.67:0/3877688967 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d7c03d2cf0 msgr2=0x55d7c03fff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:33.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.462+0000 7ff519ea6640 1 --2- 172.21.15.67:0/3877688967 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d7c03d2cf0 0x55d7c03fff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.462+0000 7ff519ea6640 1 -- 172.21.15.67:0/3877688967 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d7c011da20 con 0x55d7c03d6850 2024-03-24T00:43:33.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.462+0000 7ff519ea6640 1 --2- 172.21.15.67:0/3877688967 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d7c03d6850 0x55d7c0400480 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7ff50c0114a0 tx=0x7ff50c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be0ec6c780005c4f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:33.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.462+0000 7ff4feffd640 1 -- 172.21.15.67:0/3877688967 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50c015840 con 0x55d7c03d6850 2024-03-24T00:43:33.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.462+0000 7ff4feffd640 1 -- 172.21.15.67:0/3877688967 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff50c0159e0 con 0x55d7c03d6850 2024-03-24T00:43:33.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.462+0000 7ff4feffd640 1 -- 172.21.15.67:0/3877688967 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50c015d90 con 0x55d7c03d6850 2024-03-24T00:43:33.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.463+0000 7ff51e5d1740 1 -- 172.21.15.67:0/3877688967 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d7c03d6850 msgr2=0x55d7c0400480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:33.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.463+0000 7ff51e5d1740 1 --2- 172.21.15.67:0/3877688967 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d7c03d6850 0x55d7c0400480 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7ff50c0114a0 tx=0x7ff50c009750 comp rx=0 tx=0).stop 2024-03-24T00:43:33.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.463+0000 7ff51e5d1740 1 -- 172.21.15.67:0/3877688967 shutdown_connections 2024-03-24T00:43:33.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.463+0000 7ff51e5d1740 1 --2- 172.21.15.67:0/3877688967 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d7c03d6850 0x55d7c0400480 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.463+0000 7ff51e5d1740 1 --2- 172.21.15.67:0/3877688967 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d7c03d2cf0 0x55d7c03fff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.463+0000 7ff51e5d1740 1 --2- 172.21.15.67:0/3877688967 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d7c0402cd0 0x55d7c04030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.463+0000 7ff51e5d1740 1 -- 172.21.15.67:0/3877688967 >> 172.21.15.67:0/3877688967 conn(0x55d7c03d4f60 msgr2=0x55d7c041dae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:33.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.463+0000 7ff51e5d1740 1 -- 172.21.15.67:0/3877688967 shutdown_connections 2024-03-24T00:43:33.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.463+0000 7ff51e5d1740 1 -- 172.21.15.67:0/3877688967 wait complete. 2024-03-24T00:43:33.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.464+0000 7ff51e5d1740 1 -- start start 2024-03-24T00:43:33.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.464+0000 7ff51e5d1740 1 Processor -- start 2024-03-24T00:43:33.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.465+0000 7ff51e5d1740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d7c03d6850 0x55d7c042ec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:33.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.465+0000 7ff51e5d1740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d7c0402cd0 0x55d7c0403df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:33.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.465+0000 7ff51e5d1740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d7c0404330 0x55d7c0409da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:33.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.465+0000 7ff51e5d1740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7c03d7a00 con 0x55d7c03d6850 2024-03-24T00:43:33.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.465+0000 7ff51e5d1740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7c02f9240 con 0x55d7c0404330 2024-03-24T00:43:33.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.465+0000 7ff51e5d1740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d7c03d6df0 con 0x55d7c0402cd0 2024-03-24T00:43:33.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.465+0000 7ff51e5d1740 10 client.0 _ll_register_callbacks cb 0x55d7c03356c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:43:33.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.465+0000 7ff5196a5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d7c03d6850 0x55d7c042ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:33.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.465+0000 7ff5196a5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d7c03d6850 0x55d7c042ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48510/0 (socket says 172.21.15.67:48510) 2024-03-24T00:43:33.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.465+0000 7ff5196a5640 1 -- 172.21.15.67:0/557785855 learned_addr learned my addr 172.21.15.67:0/557785855 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:33.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.465+0000 7ff519ea6640 1 --2- 172.21.15.67:0/557785855 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d7c0404330 0x55d7c0409da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:33.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.465+0000 7ff518ea4640 1 --2- 172.21.15.67:0/557785855 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d7c0402cd0 0x55d7c0403df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:33.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.465+0000 7ff5196a5640 1 -- 172.21.15.67:0/557785855 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d7c0402cd0 msgr2=0x55d7c0403df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:33.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.465+0000 7ff5196a5640 1 --2- 172.21.15.67:0/557785855 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d7c0402cd0 0x55d7c0403df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.465+0000 7ff5196a5640 1 -- 172.21.15.67:0/557785855 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d7c0404330 msgr2=0x55d7c0409da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:33.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.465+0000 7ff5196a5640 1 --2- 172.21.15.67:0/557785855 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d7c0404330 0x55d7c0409da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.465+0000 7ff5196a5640 1 -- 172.21.15.67:0/557785855 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d7c0011310 con 0x55d7c03d6850 2024-03-24T00:43:33.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.465+0000 7ff519ea6640 1 --2- 172.21.15.67:0/557785855 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d7c0404330 0x55d7c0409da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:43:33.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.465+0000 7ff5196a5640 1 --2- 172.21.15.67:0/557785855 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d7c03d6850 0x55d7c042ec70 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7ff504009990 tx=0x7ff504004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6151e7b40b3fc21a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:33.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.465+0000 7ff4f3fff640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:43:33.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.465+0000 7ff4f3fff640 1 -- 172.21.15.67:0/557785855 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff504034040 con 0x55d7c03d6850 2024-03-24T00:43:33.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.466+0000 7ff4f3fff640 1 -- 172.21.15.67:0/557785855 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7ff504044df0 con 0x55d7c03d6850 2024-03-24T00:43:33.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.466+0000 7ff51e5d1740 1 -- 172.21.15.67:0/557785855 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff50c0159e0 con 0x55d7c03d6850 2024-03-24T00:43:33.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.466+0000 7ff4f3fff640 1 -- 172.21.15.67:0/557785855 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff50405c050 con 0x55d7c03d6850 2024-03-24T00:43:33.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.466+0000 7ff51e5d1740 10 client.7257 fetch_fsmap learned FSMap version 46 2024-03-24T00:43:33.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.466+0000 7ff51e5d1740 1 -- 172.21.15.67:0/557785855 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=2+}) v3 -- 0x55d7c017e1b0 con 0x55d7c03d6850 2024-03-24T00:43:33.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.466+0000 7ff4f3fff640 1 -- 172.21.15.67:0/557785855 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff50c0159e0 con 0x55d7c03d6850 2024-03-24T00:43:33.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.466+0000 7ff4f3fff640 1 -- 172.21.15.67:0/557785855 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 46) v1 ==== 3314+0+0 (secure 0 0 0) 0x7ff50405f020 con 0x55d7c03d6850 2024-03-24T00:43:33.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.466+0000 7ff51e5d1740 10 client.7257 fetch_fsmap finished waiting for FSMap version 46 2024-03-24T00:43:33.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.466+0000 7ff51e5d1740 10 client.7257 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.c 2024-03-24T00:43:33.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.466+0000 7ff51e5d1740 20 client.7257 populate_metadata read hostname 'smithi067' 2024-03-24T00:43:33.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.466+0000 7ff51e5d1740 1 --2- 172.21.15.67:0/557785855 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x55d7c0431b80 0x55d7c0431f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:33.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.466+0000 7ff51e5d1740 4 client.7257 mds_command: new command op to 7149 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:43:33.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.466+0000 7ff51e5d1740 1 -- 172.21.15.67:0/557785855 --> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d7c02de770 con 0x55d7c0431b80 2024-03-24T00:43:33.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.466+0000 7ff518ea4640 1 --2- 172.21.15.67:0/557785855 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x55d7c0431b80 0x55d7c0431f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:33.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.467+0000 7ff518ea4640 1 --2- 172.21.15.67:0/557785855 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x55d7c0431b80 0x55d7c0431f60 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8e02257425fd0af server_cookie=5890a5b81e9eb247 in_seq=0 out_seq=0 2024-03-24T00:43:33.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.467+0000 7ff4f3fff640 10 client.7257 ms_handle_connect on v2:172.21.15.67:6832/720256752 2024-03-24T00:43:33.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.467+0000 7ff4f3fff640 1 -- 172.21.15.67:0/557785855 <== mds.0 v2:172.21.15.67:6832/720256752 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d7c02de770 con 0x55d7c0431b80 2024-03-24T00:43:33.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.467+0000 7ff4f3fff640 10 client.7257 handle_command_reply: tid=0 2024-03-24T00:43:33.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.467+0000 7ff51e5d1740 1 client.7257 shutdown 2024-03-24T00:43:33.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.467+0000 7ff51e5d1740 1 -- 172.21.15.67:0/557785855 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d7c03d6850 msgr2=0x55d7c042ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:33.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.467+0000 7ff51e5d1740 1 --2- 172.21.15.67:0/557785855 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d7c03d6850 0x55d7c042ec70 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7ff504009990 tx=0x7ff504004730 comp rx=0 tx=0).stop 2024-03-24T00:43:33.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.467+0000 7ff51e5d1740 1 -- 172.21.15.67:0/557785855 shutdown_connections 2024-03-24T00:43:33.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.467+0000 7ff51e5d1740 1 --2- 172.21.15.67:0/557785855 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x55d7c0431b80 0x55d7c0431f60 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.467+0000 7ff51e5d1740 1 --2- 172.21.15.67:0/557785855 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d7c0404330 0x55d7c0409da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.467+0000 7ff4f3fff640 0 client.7257 ms_handle_reset on v2:172.21.15.67:6832/720256752 2024-03-24T00:43:33.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.467+0000 7ff51e5d1740 1 --2- 172.21.15.67:0/557785855 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d7c0402cd0 0x55d7c0403df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.467+0000 7ff51e5d1740 1 --2- 172.21.15.67:0/557785855 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d7c03d6850 0x55d7c042ec70 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.467+0000 7ff51e5d1740 1 -- 172.21.15.67:0/557785855 >> 172.21.15.67:0/557785855 conn(0x55d7c03d2cf0 msgr2=0x55d7c0420ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:33.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.467+0000 7ff51e5d1740 1 -- 172.21.15.67:0/557785855 shutdown_connections 2024-03-24T00:43:33.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.467+0000 7ff51e5d1740 1 -- 172.21.15.67:0/557785855 wait complete. 2024-03-24T00:43:33.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.468+0000 7ff51e5d1740 20 client.7257 trim_cache size 0 max 16384 2024-03-24T00:43:33.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.484+0000 7ff51e5d1740 1 Processor -- start 2024-03-24T00:43:33.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.484+0000 7ff51e5d1740 1 -- start start 2024-03-24T00:43:33.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.485+0000 7ff51e5d1740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d7c03d6850 0x55d7c03fde10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:33.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.485+0000 7ff51e5d1740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d7c0402cd0 0x55d7c04035f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:33.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.485+0000 7ff51e5d1740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d7c0404330 0x55d7c0403b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:33.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.485+0000 7ff51e5d1740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff514200ad0 con 0x55d7c0404330 2024-03-24T00:43:33.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.485+0000 7ff51e5d1740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5141ffc30 con 0x55d7c0402cd0 2024-03-24T00:43:33.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.485+0000 7ff51e5d1740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5140cd530 con 0x55d7c03d6850 2024-03-24T00:43:33.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.485+0000 7ff519ea6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d7c0404330 0x55d7c0403b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:33.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.485+0000 7ff519ea6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d7c0404330 0x55d7c0403b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60262/0 (socket says 172.21.15.67:60262) 2024-03-24T00:43:33.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.485+0000 7ff519ea6640 1 -- 172.21.15.67:0/369144802 learned_addr learned my addr 172.21.15.67:0/369144802 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:33.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.485+0000 7ff519ea6640 1 -- 172.21.15.67:0/369144802 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d7c03d6850 msgr2=0x55d7c03fde10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:33.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.485+0000 7ff519ea6640 1 --2- 172.21.15.67:0/369144802 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d7c03d6850 0x55d7c03fde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.485+0000 7ff519ea6640 1 -- 172.21.15.67:0/369144802 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d7c0402cd0 msgr2=0x55d7c04035f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:33.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.485+0000 7ff518ea4640 1 --2- 172.21.15.67:0/369144802 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d7c0402cd0 0x55d7c04035f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:33.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.485+0000 7ff519ea6640 1 --2- 172.21.15.67:0/369144802 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d7c0402cd0 0x55d7c04035f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.485+0000 7ff519ea6640 1 -- 172.21.15.67:0/369144802 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d7c04322d0 con 0x55d7c0404330 2024-03-24T00:43:33.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.485+0000 7ff519ea6640 1 --2- 172.21.15.67:0/369144802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d7c0404330 0x55d7c0403b30 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7ff50c011470 tx=0x7ff50c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=653703016872115a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:33.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.485+0000 7ff4fe7fc640 1 -- 172.21.15.67:0/369144802 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50c003a10 con 0x55d7c0404330 2024-03-24T00:43:33.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.485+0000 7ff4fe7fc640 1 -- 172.21.15.67:0/369144802 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff50c003bd0 con 0x55d7c0404330 2024-03-24T00:43:33.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.485+0000 7ff4fe7fc640 1 -- 172.21.15.67:0/369144802 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50c015640 con 0x55d7c0404330 2024-03-24T00:43:33.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.486+0000 7ff51e5d1740 1 -- 172.21.15.67:0/369144802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d7c0404330 msgr2=0x55d7c0403b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:33.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.486+0000 7ff51e5d1740 1 --2- 172.21.15.67:0/369144802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d7c0404330 0x55d7c0403b30 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7ff50c011470 tx=0x7ff50c016000 comp rx=0 tx=0).stop 2024-03-24T00:43:33.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.486+0000 7ff51e5d1740 1 -- 172.21.15.67:0/369144802 shutdown_connections 2024-03-24T00:43:33.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.486+0000 7ff51e5d1740 1 --2- 172.21.15.67:0/369144802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d7c0404330 0x55d7c0403b30 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.486+0000 7ff51e5d1740 1 --2- 172.21.15.67:0/369144802 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d7c0402cd0 0x55d7c04035f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.486+0000 7ff51e5d1740 1 --2- 172.21.15.67:0/369144802 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d7c03d6850 0x55d7c03fde10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.486+0000 7ff51e5d1740 1 -- 172.21.15.67:0/369144802 >> 172.21.15.67:0/369144802 conn(0x55d7c03d2cf0 msgr2=0x55d7c0409690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:33.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.486+0000 7ff51e5d1740 1 -- 172.21.15.67:0/369144802 shutdown_connections 2024-03-24T00:43:33.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.486+0000 7ff51e5d1740 1 -- 172.21.15.67:0/369144802 wait complete. 2024-03-24T00:43:33.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.487+0000 7ff51e5d1740 1 -- start start 2024-03-24T00:43:33.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.487+0000 7ff51e5d1740 1 Processor -- start 2024-03-24T00:43:33.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.488+0000 7ff51e5d1740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d7c03d6850 0x55d7c03fc640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:33.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.488+0000 7ff51e5d1740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d7c0402cd0 0x55d7c03fcb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:33.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.488+0000 7ff51e5d1740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d7c0404330 0x55d7c048e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:33.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.488+0000 7ff51e5d1740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7c041e100 con 0x55d7c03d6850 2024-03-24T00:43:33.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.488+0000 7ff51e5d1740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7c0430d00 con 0x55d7c0404330 2024-03-24T00:43:33.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.488+0000 7ff51e5d1740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d7c0339850 con 0x55d7c0402cd0 2024-03-24T00:43:33.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.489+0000 7ff51e5d1740 10 client.0 _ll_register_callbacks cb 0x7ff504030820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:43:33.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.489+0000 7ff5196a5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d7c03d6850 0x55d7c03fc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:33.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.489+0000 7ff5196a5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d7c03d6850 0x55d7c03fc640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60274/0 (socket says 172.21.15.67:60274) 2024-03-24T00:43:33.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.489+0000 7ff5196a5640 1 -- 172.21.15.67:0/3260309273 learned_addr learned my addr 172.21.15.67:0/3260309273 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:33.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.489+0000 7ff518ea4640 1 --2- 172.21.15.67:0/3260309273 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d7c0402cd0 0x55d7c03fcb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:33.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.489+0000 7ff519ea6640 1 --2- 172.21.15.67:0/3260309273 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d7c0404330 0x55d7c048e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:33.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.489+0000 7ff5196a5640 1 -- 172.21.15.67:0/3260309273 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d7c0402cd0 msgr2=0x55d7c03fcb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:33.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.489+0000 7ff5196a5640 1 --2- 172.21.15.67:0/3260309273 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d7c0402cd0 0x55d7c03fcb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.489+0000 7ff5196a5640 1 -- 172.21.15.67:0/3260309273 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d7c0404330 msgr2=0x55d7c048e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:33.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.489+0000 7ff5196a5640 1 --2- 172.21.15.67:0/3260309273 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d7c0404330 0x55d7c048e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.489+0000 7ff5196a5640 1 -- 172.21.15.67:0/3260309273 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d7bffea490 con 0x55d7c03d6850 2024-03-24T00:43:33.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.489+0000 7ff518ea4640 1 --2- 172.21.15.67:0/3260309273 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d7c0402cd0 0x55d7c03fcb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:43:33.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.489+0000 7ff519ea6640 1 --2- 172.21.15.67:0/3260309273 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d7c0404330 0x55d7c048e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:43:33.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.489+0000 7ff5196a5640 1 --2- 172.21.15.67:0/3260309273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d7c03d6850 0x55d7c03fc640 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7ff50402bf90 tx=0x7ff5040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4907237fc5985 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:33.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.489+0000 7ff4f3fff640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:43:33.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.489+0000 7ff4f3fff640 1 -- 172.21.15.67:0/3260309273 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff504034070 con 0x55d7c03d6850 2024-03-24T00:43:33.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.489+0000 7ff4f3fff640 1 -- 172.21.15.67:0/3260309273 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7ff5040587d0 con 0x55d7c03d6850 2024-03-24T00:43:33.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.489+0000 7ff51e5d1740 1 -- 172.21.15.67:0/3260309273 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff50c003bd0 con 0x55d7c03d6850 2024-03-24T00:43:33.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.489+0000 7ff4f3fff640 1 -- 172.21.15.67:0/3260309273 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff504067050 con 0x55d7c03d6850 2024-03-24T00:43:33.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.489+0000 7ff51e5d1740 10 client.7263 fetch_fsmap learned FSMap version 46 2024-03-24T00:43:33.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.489+0000 7ff51e5d1740 1 -- 172.21.15.67:0/3260309273 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=2+}) v3 -- 0x55d7c042b100 con 0x55d7c03d6850 2024-03-24T00:43:33.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.490+0000 7ff4f3fff640 1 -- 172.21.15.67:0/3260309273 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff50c003bd0 con 0x55d7c03d6850 2024-03-24T00:43:33.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.490+0000 7ff4f3fff640 1 -- 172.21.15.67:0/3260309273 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 46) v1 ==== 3314+0+0 (secure 0 0 0) 0x7ff50406a020 con 0x55d7c03d6850 2024-03-24T00:43:33.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.490+0000 7ff51e5d1740 10 client.7263 fetch_fsmap finished waiting for FSMap version 46 2024-03-24T00:43:33.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.490+0000 7ff51e5d1740 10 client.7263 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.c 2024-03-24T00:43:33.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.490+0000 7ff51e5d1740 20 client.7263 populate_metadata read hostname 'smithi067' 2024-03-24T00:43:33.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.490+0000 7ff51e5d1740 1 --2- 172.21.15.67:0/3260309273 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x55d7c048e050 0x55d7c048ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:33.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.490+0000 7ff51e5d1740 4 client.7263 mds_command: new command op to 7149 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/subdir", "scrubops": ["recursive"]}] 2024-03-24T00:43:33.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.490+0000 7ff51e5d1740 1 -- 172.21.15.67:0/3260309273 --> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] -- command(tid 0: {"prefix": "scrub start", "path": "/subdir", "scrubops": ["recursive"]}) v1 -- 0x55d7c03d4650 con 0x55d7c048e050 2024-03-24T00:43:33.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.490+0000 7ff518ea4640 1 --2- 172.21.15.67:0/3260309273 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x55d7c048e050 0x55d7c048ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:33.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.490+0000 7ff4f3fff640 10 client.7263 ms_handle_connect on v2:172.21.15.67:6832/720256752 2024-03-24T00:43:33.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.490+0000 7ff518ea4640 1 --2- 172.21.15.67:0/3260309273 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x55d7c048e050 0x55d7c048ec80 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d6b402602d5b577 server_cookie=ad32dbc3172b0e37 in_seq=0 out_seq=0 2024-03-24T00:43:33.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.491+0000 7ff4f3fff640 1 -- 172.21.15.67:0/3260309273 <== mds.0 v2:172.21.15.67:6832/720256752 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55d7c03d4650 con 0x55d7c048e050 2024-03-24T00:43:33.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.491+0000 7ff4f3fff640 10 client.7263 handle_command_reply: tid=0 2024-03-24T00:43:33.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.491+0000 7ff51e5d1740 1 client.7263 shutdown 2024-03-24T00:43:33.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.491+0000 7ff51e5d1740 1 -- 172.21.15.67:0/3260309273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d7c03d6850 msgr2=0x55d7c03fc640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:33.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.491+0000 7ff51e5d1740 1 --2- 172.21.15.67:0/3260309273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d7c03d6850 0x55d7c03fc640 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7ff50402bf90 tx=0x7ff5040046c0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.491+0000 7ff51e5d1740 1 -- 172.21.15.67:0/3260309273 shutdown_connections 2024-03-24T00:43:33.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.491+0000 7ff51e5d1740 1 --2- 172.21.15.67:0/3260309273 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x55d7c048e050 0x55d7c048ec80 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.491+0000 7ff51e5d1740 1 --2- 172.21.15.67:0/3260309273 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d7c0404330 0x55d7c048e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.491+0000 7ff51e5d1740 1 --2- 172.21.15.67:0/3260309273 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d7c0402cd0 0x55d7c03fcb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.491+0000 7ff51e5d1740 1 --2- 172.21.15.67:0/3260309273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d7c03d6850 0x55d7c03fc640 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.491+0000 7ff51e5d1740 1 -- 172.21.15.67:0/3260309273 >> 172.21.15.67:0/3260309273 conn(0x55d7c03d2cf0 msgr2=0x55d7c04039d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:33.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.491+0000 7ff4f3fff640 0 client.7263 ms_handle_reset on v2:172.21.15.67:6832/720256752 2024-03-24T00:43:33.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.491+0000 7ff51e5d1740 1 -- 172.21.15.67:0/3260309273 shutdown_connections 2024-03-24T00:43:33.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.491+0000 7ff51e5d1740 1 -- 172.21.15.67:0/3260309273 wait complete. 2024-03-24T00:43:33.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.491+0000 7ff51e5d1740 20 client.7263 trim_cache size 0 max 16384 2024-03-24T00:43:33.497 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:43:33.497 INFO:teuthology.orchestra.run.smithi067.stdout: "return_code": 0, 2024-03-24T00:43:33.497 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_tag": "88c9b9a3-5a26-4b91-a905-6bba2a81a108", 2024-03-24T00:43:33.497 INFO:teuthology.orchestra.run.smithi067.stdout: "mode": "asynchronous" 2024-03-24T00:43:33.497 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:43:33.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.492+0000 7ff51b909640 1 -- 172.21.15.67:0/2901209986 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff4f806bbd0 msgr2=0x7ff4f806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:33.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.492+0000 7ff51b909640 1 --2- 172.21.15.67:0/2901209986 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff4f806bbd0 0x7ff4f806e090 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7ff504036000 tx=0x7ff50403b040 comp rx=0 tx=0).stop 2024-03-24T00:43:33.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.492+0000 7ff51b909640 1 -- 172.21.15.67:0/2901209986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff51415dfc0 msgr2=0x7ff5141502f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:33.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.492+0000 7ff51b909640 1 --2- 172.21.15.67:0/2901209986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff51415dfc0 0x7ff5141502f0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7ff51000c430 tx=0x7ff51000c8f0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.492+0000 7ff51b909640 1 -- 172.21.15.67:0/2901209986 shutdown_connections 2024-03-24T00:43:33.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.492+0000 7ff51b909640 1 --2- 172.21.15.67:0/2901209986 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff4f806bbd0 0x7ff4f806e090 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.492+0000 7ff51b909640 1 --2- 172.21.15.67:0/2901209986 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff5140cbcd0 0x7ff514158c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.492+0000 7ff51b909640 1 --2- 172.21.15.67:0/2901209986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff51415dfc0 0x7ff5141502f0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.492+0000 7ff51b909640 1 --2- 172.21.15.67:0/2901209986 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff5140d0d50 0x7ff5140b4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.492+0000 7ff51b909640 1 -- 172.21.15.67:0/2901209986 >> 172.21.15.67:0/2901209986 conn(0x7ff51406c880 msgr2=0x7ff514161b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:33.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.492+0000 7ff51b909640 1 -- 172.21.15.67:0/2901209986 shutdown_connections 2024-03-24T00:43:33.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.492+0000 7ff51b909640 1 -- 172.21.15.67:0/2901209986 wait complete. 2024-03-24T00:43:33.503 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:43:33.504 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.5:0 scrub status 2024-03-24T00:43:33.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.607+0000 7ff858f13640 1 Processor -- start 2024-03-24T00:43:33.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.607+0000 7ff858f13640 1 -- start start 2024-03-24T00:43:33.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.607+0000 7ff858f13640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff85415bc30 0x7ff8540b1a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:33.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.608+0000 7ff858f13640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8540b3ae0 con 0x7ff8540b34a0 2024-03-24T00:43:33.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.608+0000 7ff858f13640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8540b3dc0 con 0x7ff85415c010 2024-03-24T00:43:33.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.608+0000 7ff858f13640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8540cca60 con 0x7ff85415bc30 2024-03-24T00:43:33.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.608+0000 7ff852d9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff8540b34a0 0x7ff8540ccef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59552/0 (socket says 172.21.15.67:59552) 2024-03-24T00:43:33.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.608+0000 7ff852d9d640 1 -- 172.21.15.67:0/1454495568 learned_addr learned my addr 172.21.15.67:0/1454495568 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:43:33.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.608+0000 7ff85259c640 1 --2- 172.21.15.67:0/1454495568 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff85415bc30 0x7ff8540b1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:33.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.608+0000 7ff85159a640 1 -- 172.21.15.67:0/1454495568 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1102526014 0 0) 0x7ff8540b3ae0 con 0x7ff8540b34a0 2024-03-24T00:43:33.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.608+0000 7ff85159a640 1 -- 172.21.15.67:0/1454495568 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff83c003520 con 0x7ff8540b34a0 2024-03-24T00:43:33.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.608+0000 7ff85159a640 1 -- 172.21.15.67:0/1454495568 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1277338787 0 0) 0x7ff8540b3dc0 con 0x7ff85415c010 2024-03-24T00:43:33.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.608+0000 7ff85159a640 1 -- 172.21.15.67:0/1454495568 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8540b3ae0 con 0x7ff85415c010 2024-03-24T00:43:33.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.608+0000 7ff85259c640 1 -- 172.21.15.67:0/1454495568 >> v1:172.21.15.72:6789/0 conn(0x7ff85415c010 legacy=0x7ff8540b2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:33.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.608+0000 7ff85259c640 1 -- 172.21.15.67:0/1454495568 >> v1:172.21.15.67:6789/0 conn(0x7ff8540b34a0 legacy=0x7ff8540ccef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:33.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.608+0000 7ff85259c640 1 -- 172.21.15.67:0/1454495568 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8540b40a0 con 0x7ff85415bc30 2024-03-24T00:43:33.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.609+0000 7ff85259c640 1 --2- 172.21.15.67:0/1454495568 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff85415bc30 0x7ff8540b1a00 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff8440097d0 tx=0x7ff84402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50e305ed8da998b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:33.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.609+0000 7ff85159a640 1 -- 172.21.15.67:0/1454495568 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff844035020 con 0x7ff85415bc30 2024-03-24T00:43:33.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.609+0000 7ff85159a640 1 -- 172.21.15.67:0/1454495568 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff844031890 con 0x7ff85415bc30 2024-03-24T00:43:33.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.609+0000 7ff85159a640 1 -- 172.21.15.67:0/1454495568 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff844031c60 con 0x7ff85415bc30 2024-03-24T00:43:33.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.610+0000 7ff858f13640 1 -- 172.21.15.67:0/1454495568 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff85415bc30 msgr2=0x7ff8540b1a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:33.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.610+0000 7ff858f13640 1 --2- 172.21.15.67:0/1454495568 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff85415bc30 0x7ff8540b1a00 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff8440097d0 tx=0x7ff84402b930 comp rx=0 tx=0).stop 2024-03-24T00:43:33.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.610+0000 7ff858f13640 1 -- 172.21.15.67:0/1454495568 shutdown_connections 2024-03-24T00:43:33.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.610+0000 7ff858f13640 1 --2- 172.21.15.67:0/1454495568 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff85415bc30 0x7ff8540b1a00 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.610+0000 7ff858f13640 1 -- 172.21.15.67:0/1454495568 >> 172.21.15.67:0/1454495568 conn(0x7ff85408b180 msgr2=0x7ff85408b590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:33.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.610+0000 7ff858f13640 1 -- 172.21.15.67:0/1454495568 shutdown_connections 2024-03-24T00:43:33.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.610+0000 7ff858f13640 1 -- 172.21.15.67:0/1454495568 wait complete. 2024-03-24T00:43:33.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.611+0000 7ff858f13640 1 Processor -- start 2024-03-24T00:43:33.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.611+0000 7ff858f13640 1 -- start start 2024-03-24T00:43:33.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.612+0000 7ff858f13640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff8540b34a0 0x7ff8540aea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:33.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.612+0000 7ff858f13640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff85415bc30 0x7ff8540c4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:33.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.612+0000 7ff858f13640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff85415c010 0x7ff8540b0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:33.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.612+0000 7ff858f13640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8540b20c0 con 0x7ff85415c010 2024-03-24T00:43:33.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.612+0000 7ff858f13640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8540b1f40 con 0x7ff85415bc30 2024-03-24T00:43:33.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.612+0000 7ff858f13640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8540b2240 con 0x7ff8540b34a0 2024-03-24T00:43:33.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.612+0000 7ff852d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff85415c010 0x7ff8540b0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:33.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.612+0000 7ff852d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff85415c010 0x7ff8540b0330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60276/0 (socket says 172.21.15.67:60276) 2024-03-24T00:43:33.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.612+0000 7ff852d9d640 1 -- 172.21.15.67:0/3858381269 learned_addr learned my addr 172.21.15.67:0/3858381269 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:33.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.612+0000 7ff851d9b640 1 --2- 172.21.15.67:0/3858381269 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff85415bc30 0x7ff8540c4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:33.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.612+0000 7ff85259c640 1 --2- 172.21.15.67:0/3858381269 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff8540b34a0 0x7ff8540aea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:33.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.612+0000 7ff852d9d640 1 -- 172.21.15.67:0/3858381269 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff8540b34a0 msgr2=0x7ff8540aea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:33.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.612+0000 7ff852d9d640 1 --2- 172.21.15.67:0/3858381269 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff8540b34a0 0x7ff8540aea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.612+0000 7ff852d9d640 1 -- 172.21.15.67:0/3858381269 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff85415bc30 msgr2=0x7ff8540c4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:33.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.612+0000 7ff852d9d640 1 --2- 172.21.15.67:0/3858381269 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff85415bc30 0x7ff8540c4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.612+0000 7ff852d9d640 1 -- 172.21.15.67:0/3858381269 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8540b0bf0 con 0x7ff85415c010 2024-03-24T00:43:33.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.612+0000 7ff85259c640 1 --2- 172.21.15.67:0/3858381269 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff8540b34a0 0x7ff8540aea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:43:33.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.613+0000 7ff852d9d640 1 --2- 172.21.15.67:0/3858381269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff85415c010 0x7ff8540b0330 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7ff84c00cba0 tx=0x7ff84c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:33.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.613+0000 7ff842ffd640 1 -- 172.21.15.67:0/3858381269 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff84c017070 con 0x7ff85415c010 2024-03-24T00:43:33.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.613+0000 7ff842ffd640 1 -- 172.21.15.67:0/3858381269 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff84c00ace0 con 0x7ff85415c010 2024-03-24T00:43:33.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.613+0000 7ff858f13640 1 -- 172.21.15.67:0/3858381269 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8540b0ed0 con 0x7ff85415c010 2024-03-24T00:43:33.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.613+0000 7ff842ffd640 1 -- 172.21.15.67:0/3858381269 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff84c012600 con 0x7ff85415c010 2024-03-24T00:43:33.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.613+0000 7ff858f13640 1 -- 172.21.15.67:0/3858381269 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8540b1490 con 0x7ff85415c010 2024-03-24T00:43:33.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.614+0000 7ff842ffd640 1 -- 172.21.15.67:0/3858381269 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff84c01a070 con 0x7ff85415c010 2024-03-24T00:43:33.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.615+0000 7ff842ffd640 1 --2- 172.21.15.67:0/3858381269 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff83406bbd0 0x7ff83406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:33.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.615+0000 7ff842ffd640 1 -- 172.21.15.67:0/3858381269 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7ff84c08a9c0 con 0x7ff85415c010 2024-03-24T00:43:33.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.615+0000 7ff85259c640 1 --2- 172.21.15.67:0/3858381269 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff83406bbd0 0x7ff83406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:33.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.615+0000 7ff85259c640 1 --2- 172.21.15.67:0/3858381269 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff83406bbd0 0x7ff83406e090 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7ff844036000 tx=0x7ff844008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:33.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.616+0000 7ff85bbcf740 1 Processor -- start 2024-03-24T00:43:33.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.616+0000 7ff85bbcf740 1 -- start start 2024-03-24T00:43:33.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.617+0000 7ff85bbcf740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ffa4229e50 0x55ffa41fcdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:33.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.617+0000 7ff85bbcf740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ffa422a230 0x55ffa422cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:33.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.617+0000 7ff85bbcf740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ffa41fd330 0x55ffa422d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:33.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.617+0000 7ff85bbcf740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ffa41673d0 con 0x55ffa4229e50 2024-03-24T00:43:33.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.617+0000 7ff85bbcf740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ffa410fad0 con 0x55ffa41fd330 2024-03-24T00:43:33.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.617+0000 7ff85bbcf740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ffa4163470 con 0x55ffa422a230 2024-03-24T00:43:33.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.617+0000 7ff85259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ffa4229e50 0x55ffa41fcdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:33.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.617+0000 7ff85259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ffa4229e50 0x55ffa41fcdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60288/0 (socket says 172.21.15.67:60288) 2024-03-24T00:43:33.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.617+0000 7ff85259c640 1 -- 172.21.15.67:0/942175935 learned_addr learned my addr 172.21.15.67:0/942175935 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:33.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.617+0000 7ff852d9d640 1 --2- 172.21.15.67:0/942175935 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ffa41fd330 0x55ffa422d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:33.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.617+0000 7ff851d9b640 1 --2- 172.21.15.67:0/942175935 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ffa422a230 0x55ffa422cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:33.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.617+0000 7ff85259c640 1 -- 172.21.15.67:0/942175935 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ffa422a230 msgr2=0x55ffa422cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:33.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.617+0000 7ff85259c640 1 --2- 172.21.15.67:0/942175935 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ffa422a230 0x55ffa422cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.617+0000 7ff85259c640 1 -- 172.21.15.67:0/942175935 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ffa41fd330 msgr2=0x55ffa422d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:33.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.617+0000 7ff85259c640 1 --2- 172.21.15.67:0/942175935 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ffa41fd330 0x55ffa422d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.617+0000 7ff85259c640 1 -- 172.21.15.67:0/942175935 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ffa3f457d0 con 0x55ffa4229e50 2024-03-24T00:43:33.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.617+0000 7ff852d9d640 1 --2- 172.21.15.67:0/942175935 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ffa41fd330 0x55ffa422d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:43:33.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.617+0000 7ff851d9b640 1 --2- 172.21.15.67:0/942175935 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ffa422a230 0x55ffa422cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:43:33.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.617+0000 7ff85259c640 1 --2- 172.21.15.67:0/942175935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ffa4229e50 0x55ffa41fcdf0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7ff8440098c0 tx=0x7ff844030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bbd944ca7b0b3c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:33.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.618+0000 7ff833fff640 1 -- 172.21.15.67:0/942175935 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff84404b070 con 0x55ffa4229e50 2024-03-24T00:43:33.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.618+0000 7ff833fff640 1 -- 172.21.15.67:0/942175935 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff84403bbb0 con 0x55ffa4229e50 2024-03-24T00:43:33.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.618+0000 7ff833fff640 1 -- 172.21.15.67:0/942175935 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff844035040 con 0x55ffa4229e50 2024-03-24T00:43:33.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.618+0000 7ff85bbcf740 1 -- 172.21.15.67:0/942175935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ffa4229e50 msgr2=0x55ffa41fcdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:33.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.618+0000 7ff85bbcf740 1 --2- 172.21.15.67:0/942175935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ffa4229e50 0x55ffa41fcdf0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7ff8440098c0 tx=0x7ff844030000 comp rx=0 tx=0).stop 2024-03-24T00:43:33.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.618+0000 7ff85bbcf740 1 -- 172.21.15.67:0/942175935 shutdown_connections 2024-03-24T00:43:33.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.618+0000 7ff85bbcf740 1 --2- 172.21.15.67:0/942175935 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ffa41fd330 0x55ffa422d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.618+0000 7ff85bbcf740 1 --2- 172.21.15.67:0/942175935 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ffa422a230 0x55ffa422cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.618+0000 7ff85bbcf740 1 --2- 172.21.15.67:0/942175935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ffa4229e50 0x55ffa41fcdf0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.618+0000 7ff85bbcf740 1 -- 172.21.15.67:0/942175935 >> 172.21.15.67:0/942175935 conn(0x55ffa41ff060 msgr2=0x55ffa4247be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:33.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.618+0000 7ff85bbcf740 1 -- 172.21.15.67:0/942175935 shutdown_connections 2024-03-24T00:43:33.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.618+0000 7ff85bbcf740 1 -- 172.21.15.67:0/942175935 wait complete. 2024-03-24T00:43:33.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.619+0000 7ff85bbcf740 1 -- start start 2024-03-24T00:43:33.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.619+0000 7ff85bbcf740 1 Processor -- start 2024-03-24T00:43:33.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.620+0000 7ff85bbcf740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ffa4229e50 0x55ffa42341a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:33.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.620+0000 7ff85bbcf740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ffa422a230 0x55ffa422f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:33.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.620+0000 7ff85bbcf740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ffa4219ec0 0x55ffa42174b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:33.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.620+0000 7ff85bbcf740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ffa4201700 con 0x55ffa4219ec0 2024-03-24T00:43:33.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.620+0000 7ff85bbcf740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ffa4123060 con 0x55ffa422a230 2024-03-24T00:43:33.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.620+0000 7ff85bbcf740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ffa4203f40 con 0x55ffa4229e50 2024-03-24T00:43:33.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.620+0000 7ff85bbcf740 10 client.0 _ll_register_callbacks cb 0x55ffa41056a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:43:33.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.620+0000 7ff852d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ffa4219ec0 0x55ffa42174b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:33.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.620+0000 7ff852d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ffa4219ec0 0x55ffa42174b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60300/0 (socket says 172.21.15.67:60300) 2024-03-24T00:43:33.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.620+0000 7ff852d9d640 1 -- 172.21.15.67:0/2950703662 learned_addr learned my addr 172.21.15.67:0/2950703662 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:33.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.620+0000 7ff85259c640 1 --2- 172.21.15.67:0/2950703662 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ffa4229e50 0x55ffa42341a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:33.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.620+0000 7ff851d9b640 1 --2- 172.21.15.67:0/2950703662 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ffa422a230 0x55ffa422f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:33.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.620+0000 7ff852d9d640 1 -- 172.21.15.67:0/2950703662 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ffa4229e50 msgr2=0x55ffa42341a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:33.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.620+0000 7ff852d9d640 1 --2- 172.21.15.67:0/2950703662 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ffa4229e50 0x55ffa42341a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.620+0000 7ff852d9d640 1 -- 172.21.15.67:0/2950703662 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ffa422a230 msgr2=0x55ffa422f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:33.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.620+0000 7ff852d9d640 1 --2- 172.21.15.67:0/2950703662 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ffa422a230 0x55ffa422f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.620+0000 7ff852d9d640 1 -- 172.21.15.67:0/2950703662 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ffa3e7b3f0 con 0x55ffa4219ec0 2024-03-24T00:43:33.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.620+0000 7ff851d9b640 1 --2- 172.21.15.67:0/2950703662 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ffa422a230 0x55ffa422f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:43:33.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.621+0000 7ff852d9d640 1 --2- 172.21.15.67:0/2950703662 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ffa4219ec0 0x55ffa42174b0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7ff84c0550e0 tx=0x7ff84c058dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edc670ca4717be6c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:33.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.621+0000 7ff8317fa640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:43:33.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.621+0000 7ff8317fa640 1 -- 172.21.15.67:0/2950703662 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff84c061050 con 0x55ffa4219ec0 2024-03-24T00:43:33.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.621+0000 7ff8317fa640 1 -- 172.21.15.67:0/2950703662 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7ff84c004b60 con 0x55ffa4219ec0 2024-03-24T00:43:33.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.621+0000 7ff85bbcf740 1 -- 172.21.15.67:0/2950703662 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff84403bbb0 con 0x55ffa4219ec0 2024-03-24T00:43:33.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.621+0000 7ff8317fa640 1 -- 172.21.15.67:0/2950703662 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff84c0580d0 con 0x55ffa4219ec0 2024-03-24T00:43:33.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.621+0000 7ff85bbcf740 10 client.7275 fetch_fsmap learned FSMap version 46 2024-03-24T00:43:33.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.621+0000 7ff85bbcf740 1 -- 172.21.15.67:0/2950703662 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=2+}) v3 -- 0x55ffa3e3b0f0 con 0x55ffa4219ec0 2024-03-24T00:43:33.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.621+0000 7ff8317fa640 1 -- 172.21.15.67:0/2950703662 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff84403bbb0 con 0x55ffa4219ec0 2024-03-24T00:43:33.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.621+0000 7ff8317fa640 1 -- 172.21.15.67:0/2950703662 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 46) v1 ==== 3314+0+0 (secure 0 0 0) 0x7ff84c058270 con 0x55ffa4219ec0 2024-03-24T00:43:33.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.621+0000 7ff85bbcf740 10 client.7275 fetch_fsmap finished waiting for FSMap version 46 2024-03-24T00:43:33.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.621+0000 7ff85bbcf740 10 client.7275 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.c 2024-03-24T00:43:33.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.621+0000 7ff85bbcf740 20 client.7275 populate_metadata read hostname 'smithi067' 2024-03-24T00:43:33.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.621+0000 7ff85bbcf740 1 --2- 172.21.15.67:0/2950703662 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x55ffa4216c20 0x55ffa4257360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:33.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.621+0000 7ff85bbcf740 4 client.7275 mds_command: new command op to 7149 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:43:33.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.621+0000 7ff85bbcf740 1 -- 172.21.15.67:0/2950703662 --> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ffa4174910 con 0x55ffa4216c20 2024-03-24T00:43:33.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.621+0000 7ff85259c640 1 --2- 172.21.15.67:0/2950703662 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x55ffa4216c20 0x55ffa4257360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:33.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.622+0000 7ff8317fa640 10 client.7275 ms_handle_connect on v2:172.21.15.67:6832/720256752 2024-03-24T00:43:33.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.622+0000 7ff85259c640 1 --2- 172.21.15.67:0/2950703662 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x55ffa4216c20 0x55ffa4257360 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2055029dd8e9fe7d server_cookie=79e5ad19c3fa1ccc in_seq=0 out_seq=0 2024-03-24T00:43:33.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.622+0000 7ff8317fa640 1 -- 172.21.15.67:0/2950703662 <== mds.0 v2:172.21.15.67:6832/720256752 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ffa4174910 con 0x55ffa4216c20 2024-03-24T00:43:33.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.622+0000 7ff8317fa640 10 client.7275 handle_command_reply: tid=0 2024-03-24T00:43:33.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.622+0000 7ff85bbcf740 1 client.7275 shutdown 2024-03-24T00:43:33.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.622+0000 7ff85bbcf740 1 -- 172.21.15.67:0/2950703662 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ffa4219ec0 msgr2=0x55ffa42174b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:33.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.622+0000 7ff85bbcf740 1 --2- 172.21.15.67:0/2950703662 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ffa4219ec0 0x55ffa42174b0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7ff84c0550e0 tx=0x7ff84c058dc0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.622+0000 7ff85bbcf740 1 -- 172.21.15.67:0/2950703662 shutdown_connections 2024-03-24T00:43:33.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.622+0000 7ff85bbcf740 1 --2- 172.21.15.67:0/2950703662 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ffa4219ec0 0x55ffa42174b0 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.622+0000 7ff85bbcf740 1 --2- 172.21.15.67:0/2950703662 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x55ffa4216c20 0x55ffa4257360 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.622+0000 7ff85bbcf740 1 --2- 172.21.15.67:0/2950703662 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ffa422a230 0x55ffa422f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.622+0000 7ff85bbcf740 1 --2- 172.21.15.67:0/2950703662 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ffa4229e50 0x55ffa42341a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.622+0000 7ff8317fa640 0 client.7275 ms_handle_reset on v2:172.21.15.67:6832/720256752 2024-03-24T00:43:33.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.622+0000 7ff85bbcf740 1 -- 172.21.15.67:0/2950703662 >> 172.21.15.67:0/2950703662 conn(0x55ffa41fd330 msgr2=0x55ffa4200dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:33.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.623+0000 7ff85bbcf740 1 -- 172.21.15.67:0/2950703662 shutdown_connections 2024-03-24T00:43:33.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.623+0000 7ff85bbcf740 1 -- 172.21.15.67:0/2950703662 wait complete. 2024-03-24T00:43:33.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.623+0000 7ff85bbcf740 20 client.7275 trim_cache size 0 max 16384 2024-03-24T00:43:33.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.639+0000 7ff85bbcf740 1 Processor -- start 2024-03-24T00:43:33.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.639+0000 7ff85bbcf740 1 -- start start 2024-03-24T00:43:33.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.639+0000 7ff85bbcf740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ffa4219ec0 0x55ffa42198a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:33.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.639+0000 7ff85bbcf740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ffa4229e50 0x55ffa424b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:33.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.639+0000 7ff85bbcf740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ffa422a230 0x55ffa422c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:33.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.639+0000 7ff85bbcf740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8541fe790 con 0x55ffa422a230 2024-03-24T00:43:33.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.639+0000 7ff85bbcf740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8541fd920 con 0x55ffa4219ec0 2024-03-24T00:43:33.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.639+0000 7ff85bbcf740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8541469f0 con 0x55ffa4229e50 2024-03-24T00:43:33.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.639+0000 7ff852d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ffa422a230 0x55ffa422c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:33.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.639+0000 7ff852d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ffa422a230 0x55ffa422c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60314/0 (socket says 172.21.15.67:60314) 2024-03-24T00:43:33.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.639+0000 7ff852d9d640 1 -- 172.21.15.67:0/1966914380 learned_addr learned my addr 172.21.15.67:0/1966914380 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:33.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.640+0000 7ff85259c640 1 --2- 172.21.15.67:0/1966914380 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ffa4219ec0 0x55ffa42198a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:33.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.640+0000 7ff852d9d640 1 -- 172.21.15.67:0/1966914380 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ffa4229e50 msgr2=0x55ffa424b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:33.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.640+0000 7ff851d9b640 1 --2- 172.21.15.67:0/1966914380 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ffa4229e50 0x55ffa424b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:33.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.640+0000 7ff852d9d640 1 --2- 172.21.15.67:0/1966914380 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ffa4229e50 0x55ffa424b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.640+0000 7ff852d9d640 1 -- 172.21.15.67:0/1966914380 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ffa4219ec0 msgr2=0x55ffa42198a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:33.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.640+0000 7ff852d9d640 1 --2- 172.21.15.67:0/1966914380 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ffa4219ec0 0x55ffa42198a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.640+0000 7ff852d9d640 1 -- 172.21.15.67:0/1966914380 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ffa41f8d80 con 0x55ffa422a230 2024-03-24T00:43:33.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.640+0000 7ff85259c640 1 --2- 172.21.15.67:0/1966914380 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ffa4219ec0 0x55ffa42198a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:43:33.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.640+0000 7ff852d9d640 1 --2- 172.21.15.67:0/1966914380 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ffa422a230 0x55ffa422c620 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7ff84c058980 tx=0x7ff84c01e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a05f97df466eaf3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:33.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.640+0000 7ff8337fe640 1 -- 172.21.15.67:0/1966914380 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff84c077020 con 0x55ffa422a230 2024-03-24T00:43:33.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.640+0000 7ff8337fe640 1 -- 172.21.15.67:0/1966914380 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff84c060e50 con 0x55ffa422a230 2024-03-24T00:43:33.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.640+0000 7ff8337fe640 1 -- 172.21.15.67:0/1966914380 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff84c004b60 con 0x55ffa422a230 2024-03-24T00:43:33.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.641+0000 7ff85bbcf740 1 -- 172.21.15.67:0/1966914380 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ffa422a230 msgr2=0x55ffa422c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:33.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.641+0000 7ff85bbcf740 1 --2- 172.21.15.67:0/1966914380 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ffa422a230 0x55ffa422c620 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7ff84c058980 tx=0x7ff84c01e000 comp rx=0 tx=0).stop 2024-03-24T00:43:33.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.641+0000 7ff85bbcf740 1 -- 172.21.15.67:0/1966914380 shutdown_connections 2024-03-24T00:43:33.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.641+0000 7ff85bbcf740 1 --2- 172.21.15.67:0/1966914380 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ffa422a230 0x55ffa422c620 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.641+0000 7ff85bbcf740 1 --2- 172.21.15.67:0/1966914380 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ffa4229e50 0x55ffa424b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.641+0000 7ff85bbcf740 1 --2- 172.21.15.67:0/1966914380 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ffa4219ec0 0x55ffa42198a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.641+0000 7ff85bbcf740 1 -- 172.21.15.67:0/1966914380 >> 172.21.15.67:0/1966914380 conn(0x55ffa41fd330 msgr2=0x55ffa4233520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:33.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.641+0000 7ff85bbcf740 1 -- 172.21.15.67:0/1966914380 shutdown_connections 2024-03-24T00:43:33.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.641+0000 7ff85bbcf740 1 -- 172.21.15.67:0/1966914380 wait complete. 2024-03-24T00:43:33.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.641+0000 7ff85bbcf740 1 -- start start 2024-03-24T00:43:33.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.641+0000 7ff85bbcf740 1 Processor -- start 2024-03-24T00:43:33.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.642+0000 7ff85bbcf740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ffa4219ec0 0x55ffa429e320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:33.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.642+0000 7ff85bbcf740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ffa4229e50 0x55ffa421c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:33.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.642+0000 7ff85bbcf740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ffa422a230 0x55ffa42a8c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:33.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.642+0000 7ff85bbcf740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ffa42a8ac0 con 0x55ffa4219ec0 2024-03-24T00:43:33.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.642+0000 7ff85bbcf740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff854146ff0 con 0x55ffa4229e50 2024-03-24T00:43:33.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.642+0000 7ff85bbcf740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ffa422f710 con 0x55ffa422a230 2024-03-24T00:43:33.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.642+0000 7ff85bbcf740 10 client.0 _ll_register_callbacks cb 0x7ff84c054ff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:43:33.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.642+0000 7ff85259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ffa4219ec0 0x55ffa429e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:33.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.642+0000 7ff85259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ffa4219ec0 0x55ffa429e320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60318/0 (socket says 172.21.15.67:60318) 2024-03-24T00:43:33.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.642+0000 7ff85259c640 1 -- 172.21.15.67:0/1113009597 learned_addr learned my addr 172.21.15.67:0/1113009597 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:43:33.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.642+0000 7ff851d9b640 1 --2- 172.21.15.67:0/1113009597 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ffa4229e50 0x55ffa421c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:33.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.642+0000 7ff852d9d640 1 --2- 172.21.15.67:0/1113009597 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ffa422a230 0x55ffa42a8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:33.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.642+0000 7ff85259c640 1 -- 172.21.15.67:0/1113009597 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ffa422a230 msgr2=0x55ffa42a8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:33.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.642+0000 7ff85259c640 1 --2- 172.21.15.67:0/1113009597 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ffa422a230 0x55ffa42a8c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.642+0000 7ff85259c640 1 -- 172.21.15.67:0/1113009597 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ffa4229e50 msgr2=0x55ffa421c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:43:33.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.642+0000 7ff85259c640 1 --2- 172.21.15.67:0/1113009597 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ffa4229e50 0x55ffa421c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.642+0000 7ff85259c640 1 -- 172.21.15.67:0/1113009597 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ffa424b8b0 con 0x55ffa4219ec0 2024-03-24T00:43:33.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.642+0000 7ff851d9b640 1 --2- 172.21.15.67:0/1113009597 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ffa4229e50 0x55ffa421c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:43:33.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.643+0000 7ff85259c640 1 --2- 172.21.15.67:0/1113009597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ffa4219ec0 0x55ffa429e320 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7ff84404c000 tx=0x7ff844004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=937058a19bd6a5f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:43:33.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.643+0000 7ff8317fa640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:43:33.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.643+0000 7ff8317fa640 1 -- 172.21.15.67:0/1113009597 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff844035040 con 0x55ffa4219ec0 2024-03-24T00:43:33.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.643+0000 7ff8317fa640 1 -- 172.21.15.67:0/1113009597 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7ff84405c280 con 0x55ffa4219ec0 2024-03-24T00:43:33.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.643+0000 7ff85bbcf740 1 -- 172.21.15.67:0/1113009597 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff84c060e50 con 0x55ffa4219ec0 2024-03-24T00:43:33.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.643+0000 7ff8317fa640 1 -- 172.21.15.67:0/1113009597 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff84404b040 con 0x55ffa4219ec0 2024-03-24T00:43:33.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.643+0000 7ff85bbcf740 10 client.7281 fetch_fsmap learned FSMap version 46 2024-03-24T00:43:33.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.643+0000 7ff85bbcf740 1 -- 172.21.15.67:0/1113009597 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=2+}) v3 -- 0x55ffa42188e0 con 0x55ffa4219ec0 2024-03-24T00:43:33.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.643+0000 7ff8317fa640 1 -- 172.21.15.67:0/1113009597 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff84c060e50 con 0x55ffa4219ec0 2024-03-24T00:43:33.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.643+0000 7ff8317fa640 1 -- 172.21.15.67:0/1113009597 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 46) v1 ==== 3314+0+0 (secure 0 0 0) 0x7ff844004250 con 0x55ffa4219ec0 2024-03-24T00:43:33.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.643+0000 7ff85bbcf740 10 client.7281 fetch_fsmap finished waiting for FSMap version 46 2024-03-24T00:43:33.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.643+0000 7ff85bbcf740 10 client.7281 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.c 2024-03-24T00:43:33.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.643+0000 7ff85bbcf740 20 client.7281 populate_metadata read hostname 'smithi067' 2024-03-24T00:43:33.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.643+0000 7ff85bbcf740 1 --2- 172.21.15.67:0/1113009597 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x55ffa42a9180 0x55ffa42b7090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:43:33.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.643+0000 7ff85bbcf740 4 client.7281 mds_command: new command op to 7149 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:43:33.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.643+0000 7ff85bbcf740 1 -- 172.21.15.67:0/1113009597 --> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55ffa421d8a0 con 0x55ffa42a9180 2024-03-24T00:43:33.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.643+0000 7ff851d9b640 1 --2- 172.21.15.67:0/1113009597 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x55ffa42a9180 0x55ffa42b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:43:33.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.644+0000 7ff8317fa640 10 client.7281 ms_handle_connect on v2:172.21.15.67:6832/720256752 2024-03-24T00:43:33.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.644+0000 7ff851d9b640 1 --2- 172.21.15.67:0/1113009597 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x55ffa42a9180 0x55ffa42b7090 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=272e1b72eb6073bb server_cookie=6f798ff3edcf331f in_seq=0 out_seq=0 2024-03-24T00:43:33.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.644+0000 7ff8317fa640 1 -- 172.21.15.67:0/1113009597 <== mds.0 v2:172.21.15.67:6832/720256752 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55ffa421d8a0 con 0x55ffa42a9180 2024-03-24T00:43:33.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.644+0000 7ff8317fa640 10 client.7281 handle_command_reply: tid=0 2024-03-24T00:43:33.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.644+0000 7ff85bbcf740 1 client.7281 shutdown 2024-03-24T00:43:33.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.644+0000 7ff85bbcf740 1 -- 172.21.15.67:0/1113009597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ffa4219ec0 msgr2=0x55ffa429e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:33.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.644+0000 7ff85bbcf740 1 --2- 172.21.15.67:0/1113009597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ffa4219ec0 0x55ffa429e320 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7ff84404c000 tx=0x7ff844004730 comp rx=0 tx=0).stop 2024-03-24T00:43:33.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.644+0000 7ff85bbcf740 1 -- 172.21.15.67:0/1113009597 shutdown_connections 2024-03-24T00:43:33.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.644+0000 7ff85bbcf740 1 --2- 172.21.15.67:0/1113009597 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ffa422a230 0x55ffa42a8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.644+0000 7ff85bbcf740 1 --2- 172.21.15.67:0/1113009597 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x55ffa42a9180 0x55ffa42b7090 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.644+0000 7ff85bbcf740 1 --2- 172.21.15.67:0/1113009597 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ffa4229e50 0x55ffa421c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.644+0000 7ff85bbcf740 1 --2- 172.21.15.67:0/1113009597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ffa4219ec0 0x55ffa429e320 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.644+0000 7ff8317fa640 0 client.7281 ms_handle_reset on v2:172.21.15.67:6832/720256752 2024-03-24T00:43:33.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.644+0000 7ff85bbcf740 1 -- 172.21.15.67:0/1113009597 >> 172.21.15.67:0/1113009597 conn(0x55ffa41fd330 msgr2=0x55ffa42a7c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:33.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.644+0000 7ff85bbcf740 1 -- 172.21.15.67:0/1113009597 shutdown_connections 2024-03-24T00:43:33.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.644+0000 7ff85bbcf740 1 -- 172.21.15.67:0/1113009597 wait complete. 2024-03-24T00:43:33.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.644+0000 7ff85bbcf740 20 client.7281 trim_cache size 0 max 16384 2024-03-24T00:43:33.652 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:43:33.652 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T00:43:33.652 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": { 2024-03-24T00:43:33.652 INFO:teuthology.orchestra.run.smithi067.stdout: "88c9b9a3-5a26-4b91-a905-6bba2a81a108": { 2024-03-24T00:43:33.652 INFO:teuthology.orchestra.run.smithi067.stdout: "path": "/subdir", 2024-03-24T00:43:33.653 INFO:teuthology.orchestra.run.smithi067.stdout: "tag": "88c9b9a3-5a26-4b91-a905-6bba2a81a108", 2024-03-24T00:43:33.653 INFO:teuthology.orchestra.run.smithi067.stdout: "options": "recursive" 2024-03-24T00:43:33.653 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:43:33.653 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:43:33.653 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:43:33.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.645+0000 7ff858f13640 1 -- 172.21.15.67:0/3858381269 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff83406bbd0 msgr2=0x7ff83406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:33.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.645+0000 7ff858f13640 1 --2- 172.21.15.67:0/3858381269 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff83406bbd0 0x7ff83406e090 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7ff844036000 tx=0x7ff844008030 comp rx=0 tx=0).stop 2024-03-24T00:43:33.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.645+0000 7ff858f13640 1 -- 172.21.15.67:0/3858381269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff85415c010 msgr2=0x7ff8540b0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:43:33.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.645+0000 7ff858f13640 1 --2- 172.21.15.67:0/3858381269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff85415c010 0x7ff8540b0330 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7ff84c00cba0 tx=0x7ff84c00a430 comp rx=0 tx=0).stop 2024-03-24T00:43:33.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.645+0000 7ff858f13640 1 -- 172.21.15.67:0/3858381269 shutdown_connections 2024-03-24T00:43:33.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.645+0000 7ff858f13640 1 --2- 172.21.15.67:0/3858381269 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff83406bbd0 0x7ff83406e090 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.645+0000 7ff858f13640 1 --2- 172.21.15.67:0/3858381269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff85415c010 0x7ff8540b0330 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.645+0000 7ff858f13640 1 --2- 172.21.15.67:0/3858381269 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff85415bc30 0x7ff8540c4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.645+0000 7ff858f13640 1 --2- 172.21.15.67:0/3858381269 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff8540b34a0 0x7ff8540aea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:43:33.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.645+0000 7ff858f13640 1 -- 172.21.15.67:0/3858381269 >> 172.21.15.67:0/3858381269 conn(0x7ff85408b180 msgr2=0x7ff8540ce410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:43:33.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.645+0000 7ff858f13640 1 -- 172.21.15.67:0/3858381269 shutdown_connections 2024-03-24T00:43:33.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:43:33.645+0000 7ff858f13640 1 -- 172.21.15.67:0/3858381269 wait complete. 2024-03-24T00:43:33.656 INFO:tasks.cephfs.filesystem:scrub status for tag:88c9b9a3-5a26-4b91-a905-6bba2a81a108 - {'path': '/subdir', 'tag': '88c9b9a3-5a26-4b91-a905-6bba2a81a108', 'options': 'recursive'} 2024-03-24T00:43:35.408 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-24T00:43:35.407+0000 7fbfd2216640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-03-24T00:43:46.447 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:43:46.450 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:44:03.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:44:03.658 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.5:0 scrub status 2024-03-24T00:44:03.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.762+0000 7f2816f5b640 1 Processor -- start 2024-03-24T00:44:03.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.762+0000 7f2816f5b640 1 -- start start 2024-03-24T00:44:03.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.763+0000 7f2816f5b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f281015dea0 0x7f28100d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:03.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.763+0000 7f2816f5b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28100d2250 con 0x7f281015e280 2024-03-24T00:44:03.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.763+0000 7f2816f5b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28100d2530 con 0x7f28100b6170 2024-03-24T00:44:03.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.763+0000 7f2816f5b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28100d2810 con 0x7f281015dea0 2024-03-24T00:44:03.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.763+0000 7f280ffff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f281015e280 0x7f28100b5a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46856/0 (socket says 172.21.15.67:46856) 2024-03-24T00:44:03.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.763+0000 7f280ffff640 1 -- 172.21.15.67:0/3586770538 learned_addr learned my addr 172.21.15.67:0/3586770538 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:44:03.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.763+0000 7f2814cf7640 1 --2- 172.21.15.67:0/3586770538 >> v2:172.21.15.72:3300/0 conn(0x7f281015dea0 0x7f28100d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:03.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.763+0000 7f280f7fe640 1 -- 172.21.15.67:0/3586770538 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2656785536 0 0) 0x7f28100d2250 con 0x7f281015e280 2024-03-24T00:44:03.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.763+0000 7f280f7fe640 1 -- 172.21.15.67:0/3586770538 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2800003520 con 0x7f281015e280 2024-03-24T00:44:03.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.763+0000 7f280f7fe640 1 -- 172.21.15.67:0/3586770538 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1178413080 0 0) 0x7f28100d2530 con 0x7f28100b6170 2024-03-24T00:44:03.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.763+0000 7f280f7fe640 1 -- 172.21.15.67:0/3586770538 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28100d2250 con 0x7f28100b6170 2024-03-24T00:44:03.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.763+0000 7f280f7fe640 1 -- 172.21.15.67:0/3586770538 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3301948106 0 0) 0x7f2800003520 con 0x7f281015e280 2024-03-24T00:44:03.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.763+0000 7f280f7fe640 1 -- 172.21.15.67:0/3586770538 >> v2:172.21.15.72:3300/0 conn(0x7f281015dea0 msgr2=0x7f28100d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:03.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.763+0000 7f280f7fe640 1 --2- 172.21.15.67:0/3586770538 >> v2:172.21.15.72:3300/0 conn(0x7f281015dea0 0x7f28100d33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:03.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.763+0000 7f280f7fe640 1 -- 172.21.15.67:0/3586770538 >> v1:172.21.15.72:6789/0 conn(0x7f28100b6170 legacy=0x7f28100b70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:03.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.763+0000 7f280f7fe640 1 -- 172.21.15.67:0/3586770538 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28100d2af0 con 0x7f281015e280 2024-03-24T00:44:03.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.763+0000 7f280f7fe640 1 -- 172.21.15.67:0/3586770538 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f27fc001670 con 0x7f281015e280 2024-03-24T00:44:03.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.763+0000 7f280f7fe640 1 -- 172.21.15.67:0/3586770538 >> v1:172.21.15.67:6789/0 conn(0x7f281015e280 legacy=0x7f28100b5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:03.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.764+0000 7f280f7fe640 1 --2- 172.21.15.67:0/3586770538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2800003b90 0x7f2800003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:03.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.764+0000 7f280f7fe640 1 --2- 172.21.15.67:0/3586770538 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2800004800 0x7f2800004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:03.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.764+0000 7f280f7fe640 1 --2- 172.21.15.67:0/3586770538 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2800005210 0x7f280000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:03.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.764+0000 7f280f7fe640 1 -- 172.21.15.67:0/3586770538 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28100d2810 con 0x7f2800003b90 2024-03-24T00:44:03.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.764+0000 7f280f7fe640 1 -- 172.21.15.67:0/3586770538 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f280000c140 con 0x7f2800004800 2024-03-24T00:44:03.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.764+0000 7f280f7fe640 1 -- 172.21.15.67:0/3586770538 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f280000c340 con 0x7f2800005210 2024-03-24T00:44:03.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.764+0000 7f2814cf7640 1 --2- 172.21.15.67:0/3586770538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2800003b90 0x7f2800003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:03.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.764+0000 7f280ffff640 1 --2- 172.21.15.67:0/3586770538 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2800004800 0x7f2800004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:03.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.764+0000 7f28154f8640 1 --2- 172.21.15.67:0/3586770538 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2800005210 0x7f280000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:03.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.764+0000 7f2814cf7640 1 -- 172.21.15.67:0/3586770538 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2800005210 msgr2=0x7f280000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:03.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.764+0000 7f2814cf7640 1 --2- 172.21.15.67:0/3586770538 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2800005210 0x7f280000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:03.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.764+0000 7f2814cf7640 1 -- 172.21.15.67:0/3586770538 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2800004800 msgr2=0x7f2800004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:03.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.764+0000 7f2814cf7640 1 --2- 172.21.15.67:0/3586770538 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2800004800 0x7f2800004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:03.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.764+0000 7f2814cf7640 1 -- 172.21.15.67:0/3586770538 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2800003520 con 0x7f2800003b90 2024-03-24T00:44:03.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.764+0000 7f28154f8640 1 -- 172.21.15.67:0/3586770538 reap_dead start 2024-03-24T00:44:03.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.765+0000 7f2814cf7640 1 --2- 172.21.15.67:0/3586770538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2800003b90 0x7f2800003f90 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f2804009ac0 tx=0x7f2800005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4c3ad85e25f87f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:03.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.765+0000 7f280f7fe640 1 -- 172.21.15.67:0/3586770538 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2804039070 con 0x7f2800003b90 2024-03-24T00:44:03.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.765+0000 7f280f7fe640 1 -- 172.21.15.67:0/3586770538 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f280402c6b0 con 0x7f2800003b90 2024-03-24T00:44:03.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.765+0000 7f280f7fe640 1 -- 172.21.15.67:0/3586770538 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2804009690 con 0x7f2800003b90 2024-03-24T00:44:03.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.765+0000 7f2816f5b640 1 -- 172.21.15.67:0/3586770538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2800003b90 msgr2=0x7f2800003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:03.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.765+0000 7f2816f5b640 1 --2- 172.21.15.67:0/3586770538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2800003b90 0x7f2800003f90 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f2804009ac0 tx=0x7f2800005830 comp rx=0 tx=0).stop 2024-03-24T00:44:03.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.765+0000 7f2816f5b640 1 -- 172.21.15.67:0/3586770538 shutdown_connections 2024-03-24T00:44:03.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.765+0000 7f2816f5b640 1 --2- 172.21.15.67:0/3586770538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2800003b90 0x7f2800003f90 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:03.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.765+0000 7f2816f5b640 1 -- 172.21.15.67:0/3586770538 >> 172.21.15.67:0/3586770538 conn(0x7f2810072da0 msgr2=0x7f28100d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:03.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.765+0000 7f2816f5b640 1 -- 172.21.15.67:0/3586770538 shutdown_connections 2024-03-24T00:44:03.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.765+0000 7f2816f5b640 1 -- 172.21.15.67:0/3586770538 wait complete. 2024-03-24T00:44:03.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.767+0000 7f2816f5b640 1 Processor -- start 2024-03-24T00:44:03.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.767+0000 7f2816f5b640 1 -- start start 2024-03-24T00:44:03.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.767+0000 7f2816f5b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2800003b90 0x7f2810155490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:03.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.767+0000 7f2816f5b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2810160b10 0x7f2810160f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:03.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.767+0000 7f2816f5b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28101f3970 0x7f28101f3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:03.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.767+0000 7f2816f5b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28100b5df0 con 0x7f28101f3970 2024-03-24T00:44:03.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.767+0000 7f2816f5b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28100b5c70 con 0x7f2800003b90 2024-03-24T00:44:03.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.767+0000 7f2816f5b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28100b5f70 con 0x7f2810160b10 2024-03-24T00:44:03.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.768+0000 7f28154f8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28101f3970 0x7f28101f3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:03.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.768+0000 7f28154f8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28101f3970 0x7f28101f3df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50984/0 (socket says 172.21.15.67:50984) 2024-03-24T00:44:03.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.768+0000 7f28154f8640 1 -- 172.21.15.67:0/3636729472 learned_addr learned my addr 172.21.15.67:0/3636729472 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:03.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.768+0000 7f2814cf7640 1 --2- 172.21.15.67:0/3636729472 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2800003b90 0x7f2810155490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:03.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.768+0000 7f280ffff640 1 --2- 172.21.15.67:0/3636729472 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2810160b10 0x7f2810160f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:03.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.768+0000 7f28154f8640 1 -- 172.21.15.67:0/3636729472 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2810160b10 msgr2=0x7f2810160f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:03.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.768+0000 7f28154f8640 1 --2- 172.21.15.67:0/3636729472 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2810160b10 0x7f2810160f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:03.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.768+0000 7f28154f8640 1 -- 172.21.15.67:0/3636729472 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2800003b90 msgr2=0x7f2810155490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:03.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.768+0000 7f28154f8640 1 --2- 172.21.15.67:0/3636729472 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2800003b90 0x7f2810155490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:03.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.768+0000 7f28154f8640 1 -- 172.21.15.67:0/3636729472 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28101f6ef0 con 0x7f28101f3970 2024-03-24T00:44:03.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.768+0000 7f2814cf7640 1 --2- 172.21.15.67:0/3636729472 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2800003b90 0x7f2810155490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:44:03.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.768+0000 7f280ffff640 1 --2- 172.21.15.67:0/3636729472 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2810160b10 0x7f2810160f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:44:03.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.768+0000 7f28154f8640 1 --2- 172.21.15.67:0/3636729472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28101f3970 0x7f28101f3df0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f28100d0220 tx=0x7f280800eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:03.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.768+0000 7f280d7fa640 1 -- 172.21.15.67:0/3636729472 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2808003970 con 0x7f28101f3970 2024-03-24T00:44:03.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.768+0000 7f280d7fa640 1 -- 172.21.15.67:0/3636729472 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2808018070 con 0x7f28101f3970 2024-03-24T00:44:03.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.768+0000 7f2816f5b640 1 -- 172.21.15.67:0/3636729472 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28101f71d0 con 0x7f28101f3970 2024-03-24T00:44:03.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.768+0000 7f280d7fa640 1 -- 172.21.15.67:0/3636729472 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2808003d70 con 0x7f28101f3970 2024-03-24T00:44:03.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.768+0000 7f2816f5b640 1 -- 172.21.15.67:0/3636729472 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28101f7790 con 0x7f28101f3970 2024-03-24T00:44:03.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.770+0000 7f280d7fa640 1 -- 172.21.15.67:0/3636729472 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f28080050d0 con 0x7f28101f3970 2024-03-24T00:44:03.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.770+0000 7f280d7fa640 1 --2- 172.21.15.67:0/3636729472 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f27f406bbd0 0x7f27f406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:03.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.771+0000 7f280d7fa640 1 -- 172.21.15.67:0/3636729472 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f280808a920 con 0x7f28101f3970 2024-03-24T00:44:03.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.771+0000 7f2814cf7640 1 --2- 172.21.15.67:0/3636729472 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f27f406bbd0 0x7f27f406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:03.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.771+0000 7f2819c28740 1 Processor -- start 2024-03-24T00:44:03.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.771+0000 7f2819c28740 1 -- start start 2024-03-24T00:44:03.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.772+0000 7f2814cf7640 1 --2- 172.21.15.67:0/3636729472 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f27f406bbd0 0x7f27f406e090 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f2804036000 tx=0x7f2804030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:03.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.772+0000 7f2819c28740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55558bf3ce50 0x55558bf0fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:03.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.772+0000 7f2819c28740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55558bf3d230 0x55558bf3fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:03.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.772+0000 7f2819c28740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55558bf10330 0x55558bf40120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:03.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.772+0000 7f2819c28740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55558be7a3d0 con 0x55558bf10330 2024-03-24T00:44:03.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.772+0000 7f2819c28740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55558be22ad0 con 0x55558bf3ce50 2024-03-24T00:44:03.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.772+0000 7f2819c28740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55558be76470 con 0x55558bf3d230 2024-03-24T00:44:03.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.772+0000 7f28154f8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55558bf10330 0x55558bf40120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:03.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.772+0000 7f28154f8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55558bf10330 0x55558bf40120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50996/0 (socket says 172.21.15.67:50996) 2024-03-24T00:44:03.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.772+0000 7f28154f8640 1 -- 172.21.15.67:0/1037408317 learned_addr learned my addr 172.21.15.67:0/1037408317 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:03.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.772+0000 7f2814cf7640 1 --2- 172.21.15.67:0/1037408317 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55558bf3ce50 0x55558bf0fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:03.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.772+0000 7f280ffff640 1 --2- 172.21.15.67:0/1037408317 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55558bf3d230 0x55558bf3fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:03.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.773+0000 7f28154f8640 1 -- 172.21.15.67:0/1037408317 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55558bf3d230 msgr2=0x55558bf3fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:03.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.773+0000 7f28154f8640 1 --2- 172.21.15.67:0/1037408317 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55558bf3d230 0x55558bf3fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:03.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.773+0000 7f28154f8640 1 -- 172.21.15.67:0/1037408317 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55558bf3ce50 msgr2=0x55558bf0fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:03.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.773+0000 7f28154f8640 1 --2- 172.21.15.67:0/1037408317 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55558bf3ce50 0x55558bf0fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:03.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.773+0000 7f28154f8640 1 -- 172.21.15.67:0/1037408317 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55558bc587d0 con 0x55558bf10330 2024-03-24T00:44:03.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.773+0000 7f280ffff640 1 --2- 172.21.15.67:0/1037408317 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55558bf3d230 0x55558bf3fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:44:03.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.773+0000 7f28154f8640 1 --2- 172.21.15.67:0/1037408317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55558bf10330 0x55558bf40120 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f28080050a0 tx=0x7f2808058510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95c655bcc936e4cf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:03.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.773+0000 7f27fa7fc640 1 -- 172.21.15.67:0/1037408317 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2808059e60 con 0x55558bf10330 2024-03-24T00:44:03.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.773+0000 7f27fa7fc640 1 -- 172.21.15.67:0/1037408317 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f280805cc80 con 0x55558bf10330 2024-03-24T00:44:03.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.773+0000 7f27fa7fc640 1 -- 172.21.15.67:0/1037408317 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2808061240 con 0x55558bf10330 2024-03-24T00:44:03.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.774+0000 7f2819c28740 1 -- 172.21.15.67:0/1037408317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55558bf10330 msgr2=0x55558bf40120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:03.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.774+0000 7f2819c28740 1 --2- 172.21.15.67:0/1037408317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55558bf10330 0x55558bf40120 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f28080050a0 tx=0x7f2808058510 comp rx=0 tx=0).stop 2024-03-24T00:44:03.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.774+0000 7f2819c28740 1 -- 172.21.15.67:0/1037408317 shutdown_connections 2024-03-24T00:44:03.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.774+0000 7f2819c28740 1 --2- 172.21.15.67:0/1037408317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55558bf10330 0x55558bf40120 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:03.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.774+0000 7f2819c28740 1 --2- 172.21.15.67:0/1037408317 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55558bf3d230 0x55558bf3fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:03.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.774+0000 7f2819c28740 1 --2- 172.21.15.67:0/1037408317 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55558bf3ce50 0x55558bf0fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:03.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.774+0000 7f2819c28740 1 -- 172.21.15.67:0/1037408317 >> 172.21.15.67:0/1037408317 conn(0x55558bf12060 msgr2=0x55558bf5abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:03.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.774+0000 7f2819c28740 1 -- 172.21.15.67:0/1037408317 shutdown_connections 2024-03-24T00:44:03.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.774+0000 7f2819c28740 1 -- 172.21.15.67:0/1037408317 wait complete. 2024-03-24T00:44:03.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.774+0000 7f2819c28740 1 -- start start 2024-03-24T00:44:03.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.774+0000 7f2819c28740 1 Processor -- start 2024-03-24T00:44:03.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.775+0000 7f2819c28740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55558bf3ce50 0x55558bf6ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:03.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.775+0000 7f2819c28740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55558bf3d230 0x55558bf32890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:03.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.775+0000 7f2819c28740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55558bf34c90 0x55558bf35110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:03.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.775+0000 7f2819c28740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55558bf14700 con 0x55558bf3ce50 2024-03-24T00:44:03.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.775+0000 7f2819c28740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55558be36060 con 0x55558bf34c90 2024-03-24T00:44:03.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.775+0000 7f2819c28740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55558bf16f40 con 0x55558bf3d230 2024-03-24T00:44:03.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.775+0000 7f2819c28740 10 client.0 _ll_register_callbacks cb 0x55558be186a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:44:03.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.775+0000 7f2814cf7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55558bf3ce50 0x55558bf6ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:03.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.775+0000 7f2814cf7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55558bf3ce50 0x55558bf6ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51004/0 (socket says 172.21.15.67:51004) 2024-03-24T00:44:03.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.775+0000 7f2814cf7640 1 -- 172.21.15.67:0/1796510860 learned_addr learned my addr 172.21.15.67:0/1796510860 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:03.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.775+0000 7f280ffff640 1 --2- 172.21.15.67:0/1796510860 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55558bf3d230 0x55558bf32890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:03.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.775+0000 7f28154f8640 1 --2- 172.21.15.67:0/1796510860 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55558bf34c90 0x55558bf35110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:03.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.776+0000 7f2814cf7640 1 -- 172.21.15.67:0/1796510860 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55558bf3d230 msgr2=0x55558bf32890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:03.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.776+0000 7f2814cf7640 1 --2- 172.21.15.67:0/1796510860 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55558bf3d230 0x55558bf32890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:03.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.776+0000 7f2814cf7640 1 -- 172.21.15.67:0/1796510860 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55558bf34c90 msgr2=0x55558bf35110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:03.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.776+0000 7f2814cf7640 1 --2- 172.21.15.67:0/1796510860 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55558bf34c90 0x55558bf35110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:03.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.776+0000 7f2814cf7640 1 -- 172.21.15.67:0/1796510860 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55558bb8e3f0 con 0x55558bf3ce50 2024-03-24T00:44:03.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.776+0000 7f2814cf7640 1 --2- 172.21.15.67:0/1796510860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55558bf3ce50 0x55558bf6ed50 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f280402c320 tx=0x7f280402e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e94fcad545f9cbd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:03.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.776+0000 7f27ef7fe640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:44:03.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.776+0000 7f27ef7fe640 1 -- 172.21.15.67:0/1796510860 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2804039050 con 0x55558bf3ce50 2024-03-24T00:44:03.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.776+0000 7f27ef7fe640 1 -- 172.21.15.67:0/1796510860 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f2804050cc0 con 0x55558bf3ce50 2024-03-24T00:44:03.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.776+0000 7f2819c28740 1 -- 172.21.15.67:0/1796510860 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f280805cc80 con 0x55558bf3ce50 2024-03-24T00:44:03.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.776+0000 7f27ef7fe640 1 -- 172.21.15.67:0/1796510860 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2804005990 con 0x55558bf3ce50 2024-03-24T00:44:03.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.776+0000 7f2819c28740 10 client.7293 fetch_fsmap learned FSMap version 47 2024-03-24T00:44:03.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.776+0000 7f2819c28740 1 -- 172.21.15.67:0/1796510860 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=2+}) v3 -- 0x55558bb4e0f0 con 0x55558bf3ce50 2024-03-24T00:44:03.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.776+0000 7f27ef7fe640 1 -- 172.21.15.67:0/1796510860 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f280805cc80 con 0x55558bf3ce50 2024-03-24T00:44:03.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.776+0000 7f27ef7fe640 1 -- 172.21.15.67:0/1796510860 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 47) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f280405a080 con 0x55558bf3ce50 2024-03-24T00:44:03.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.776+0000 7f2819c28740 10 client.7293 fetch_fsmap finished waiting for FSMap version 47 2024-03-24T00:44:03.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.776+0000 7f2819c28740 10 client.7293 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.c 2024-03-24T00:44:03.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.776+0000 7f2819c28740 20 client.7293 populate_metadata read hostname 'smithi067' 2024-03-24T00:44:03.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.776+0000 7f2819c28740 1 --2- 172.21.15.67:0/1796510860 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x55558bf3e410 0x55558bf78630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:03.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.776+0000 7f2819c28740 4 client.7293 mds_command: new command op to 7149 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:44:03.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.776+0000 7f2819c28740 1 -- 172.21.15.67:0/1796510860 --> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55558be87910 con 0x55558bf3e410 2024-03-24T00:44:03.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.777+0000 7f280ffff640 1 --2- 172.21.15.67:0/1796510860 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x55558bf3e410 0x55558bf78630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:03.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.777+0000 7f27ef7fe640 10 client.7293 ms_handle_connect on v2:172.21.15.67:6832/720256752 2024-03-24T00:44:03.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.777+0000 7f280ffff640 1 --2- 172.21.15.67:0/1796510860 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x55558bf3e410 0x55558bf78630 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f9d2159ab7e2b1a server_cookie=1835483f435fbb57 in_seq=0 out_seq=0 2024-03-24T00:44:03.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.777+0000 7f27ef7fe640 1 -- 172.21.15.67:0/1796510860 <== mds.0 v2:172.21.15.67:6832/720256752 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55558be87910 con 0x55558bf3e410 2024-03-24T00:44:03.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.778+0000 7f27ef7fe640 10 client.7293 handle_command_reply: tid=0 2024-03-24T00:44:03.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.778+0000 7f2819c28740 1 client.7293 shutdown 2024-03-24T00:44:03.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.778+0000 7f2819c28740 1 -- 172.21.15.67:0/1796510860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55558bf3ce50 msgr2=0x55558bf6ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:03.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.778+0000 7f2819c28740 1 --2- 172.21.15.67:0/1796510860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55558bf3ce50 0x55558bf6ed50 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f280402c320 tx=0x7f280402e000 comp rx=0 tx=0).stop 2024-03-24T00:44:03.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.778+0000 7f2819c28740 1 -- 172.21.15.67:0/1796510860 shutdown_connections 2024-03-24T00:44:03.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.778+0000 7f2819c28740 1 --2- 172.21.15.67:0/1796510860 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x55558bf3e410 0x55558bf78630 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:03.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.778+0000 7f2819c28740 1 --2- 172.21.15.67:0/1796510860 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55558bf34c90 0x55558bf35110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:03.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.778+0000 7f2819c28740 1 --2- 172.21.15.67:0/1796510860 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55558bf3d230 0x55558bf32890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:03.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.778+0000 7f27ef7fe640 0 client.7293 ms_handle_reset on v2:172.21.15.67:6832/720256752 2024-03-24T00:44:03.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.778+0000 7f2819c28740 1 --2- 172.21.15.67:0/1796510860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55558bf3ce50 0x55558bf6ed50 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:03.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.778+0000 7f2819c28740 1 -- 172.21.15.67:0/1796510860 >> 172.21.15.67:0/1796510860 conn(0x55558bf10330 msgr2=0x55558bf13dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:03.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.778+0000 7f2819c28740 1 -- 172.21.15.67:0/1796510860 shutdown_connections 2024-03-24T00:44:03.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.778+0000 7f2819c28740 1 -- 172.21.15.67:0/1796510860 wait complete. 2024-03-24T00:44:03.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.778+0000 7f2819c28740 20 client.7293 trim_cache size 0 max 16384 2024-03-24T00:44:03.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.795+0000 7f2819c28740 1 Processor -- start 2024-03-24T00:44:03.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.795+0000 7f2819c28740 1 -- start start 2024-03-24T00:44:03.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.795+0000 7f2819c28740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55558bf34c90 0x55558bf38460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:03.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.795+0000 7f2819c28740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55558bf3ce50 0x55558bf6dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:03.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.795+0000 7f2819c28740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55558bf3d230 0x55558bf6e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:03.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.795+0000 7f2819c28740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28102011a0 con 0x55558bf34c90 2024-03-24T00:44:03.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.795+0000 7f2819c28740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2810200340 con 0x55558bf3ce50 2024-03-24T00:44:03.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.795+0000 7f2819c28740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f281015f880 con 0x55558bf3d230 2024-03-24T00:44:03.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.795+0000 7f2814cf7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55558bf34c90 0x55558bf38460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:03.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.795+0000 7f2814cf7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55558bf34c90 0x55558bf38460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51014/0 (socket says 172.21.15.67:51014) 2024-03-24T00:44:03.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.795+0000 7f2814cf7640 1 -- 172.21.15.67:0/2153326289 learned_addr learned my addr 172.21.15.67:0/2153326289 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:03.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.795+0000 7f2814cf7640 1 -- 172.21.15.67:0/2153326289 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55558bf3d230 msgr2=0x55558bf6e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:03.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.795+0000 7f2814cf7640 1 --2- 172.21.15.67:0/2153326289 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55558bf3d230 0x55558bf6e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:03.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.795+0000 7f2814cf7640 1 -- 172.21.15.67:0/2153326289 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55558bf3ce50 msgr2=0x55558bf6dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:03.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.795+0000 7f2814cf7640 1 --2- 172.21.15.67:0/2153326289 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55558bf3ce50 0x55558bf6dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:03.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.795+0000 7f2814cf7640 1 -- 172.21.15.67:0/2153326289 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55558bf38c00 con 0x55558bf34c90 2024-03-24T00:44:03.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.795+0000 7f2814cf7640 1 --2- 172.21.15.67:0/2153326289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55558bf34c90 0x55558bf38460 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f2804009770 tx=0x7f2804009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=141380d3c8900f87 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:03.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.795+0000 7f27f9ffb640 1 -- 172.21.15.67:0/2153326289 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f280405d030 con 0x55558bf34c90 2024-03-24T00:44:03.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.795+0000 7f27f9ffb640 1 -- 172.21.15.67:0/2153326289 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f280403a040 con 0x55558bf34c90 2024-03-24T00:44:03.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.796+0000 7f27f9ffb640 1 -- 172.21.15.67:0/2153326289 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2804039040 con 0x55558bf34c90 2024-03-24T00:44:03.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.796+0000 7f2819c28740 1 -- 172.21.15.67:0/2153326289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55558bf34c90 msgr2=0x55558bf38460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:03.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.796+0000 7f2819c28740 1 --2- 172.21.15.67:0/2153326289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55558bf34c90 0x55558bf38460 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f2804009770 tx=0x7f2804009190 comp rx=0 tx=0).stop 2024-03-24T00:44:03.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.796+0000 7f2819c28740 1 -- 172.21.15.67:0/2153326289 shutdown_connections 2024-03-24T00:44:03.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.796+0000 7f2819c28740 1 --2- 172.21.15.67:0/2153326289 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55558bf3d230 0x55558bf6e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:03.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.796+0000 7f2819c28740 1 --2- 172.21.15.67:0/2153326289 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55558bf3ce50 0x55558bf6dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:03.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.796+0000 7f2819c28740 1 --2- 172.21.15.67:0/2153326289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55558bf34c90 0x55558bf38460 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:03.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.796+0000 7f2819c28740 1 -- 172.21.15.67:0/2153326289 >> 172.21.15.67:0/2153326289 conn(0x55558bf10330 msgr2=0x55558bf6eb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:03.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.796+0000 7f2819c28740 1 -- 172.21.15.67:0/2153326289 shutdown_connections 2024-03-24T00:44:03.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.796+0000 7f2819c28740 1 -- 172.21.15.67:0/2153326289 wait complete. 2024-03-24T00:44:03.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.797+0000 7f2819c28740 1 -- start start 2024-03-24T00:44:03.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.797+0000 7f2819c28740 1 Processor -- start 2024-03-24T00:44:03.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.798+0000 7f2819c28740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55558bf34c90 0x55558bf5ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:03.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.798+0000 7f2819c28740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55558bf3ce50 0x55558bf3b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:03.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.798+0000 7f2819c28740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55558bf3d230 0x55558bf6dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:03.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.798+0000 7f2819c28740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55558bf5e9b0 con 0x55558bf3ce50 2024-03-24T00:44:03.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.798+0000 7f2819c28740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28101ffd40 con 0x55558bf34c90 2024-03-24T00:44:03.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.798+0000 7f2819c28740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55558bf17010 con 0x55558bf3d230 2024-03-24T00:44:03.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.798+0000 7f2819c28740 10 client.0 _ll_register_callbacks cb 0x7f28000036f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:44:03.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.798+0000 7f280ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55558bf3ce50 0x55558bf3b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:03.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.799+0000 7f280ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55558bf3ce50 0x55558bf3b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51016/0 (socket says 172.21.15.67:51016) 2024-03-24T00:44:03.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.799+0000 7f280ffff640 1 -- 172.21.15.67:0/1840926716 learned_addr learned my addr 172.21.15.67:0/1840926716 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:03.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.799+0000 7f2814cf7640 1 --2- 172.21.15.67:0/1840926716 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55558bf34c90 0x55558bf5ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:03.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.799+0000 7f28154f8640 1 --2- 172.21.15.67:0/1840926716 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55558bf3d230 0x55558bf6dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:03.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.799+0000 7f280ffff640 1 -- 172.21.15.67:0/1840926716 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55558bf3d230 msgr2=0x55558bf6dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:03.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.799+0000 7f280ffff640 1 --2- 172.21.15.67:0/1840926716 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55558bf3d230 0x55558bf6dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:03.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.799+0000 7f280ffff640 1 -- 172.21.15.67:0/1840926716 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55558bf34c90 msgr2=0x55558bf5ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:03.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.799+0000 7f280ffff640 1 --2- 172.21.15.67:0/1840926716 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55558bf34c90 0x55558bf5ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:03.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.799+0000 7f280ffff640 1 -- 172.21.15.67:0/1840926716 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55558bf31740 con 0x55558bf3ce50 2024-03-24T00:44:03.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.799+0000 7f280ffff640 1 --2- 172.21.15.67:0/1840926716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55558bf3ce50 0x55558bf3b600 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f27fc007c30 tx=0x7f27fc001d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71c2c14c8b37c28c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:03.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.799+0000 7f27ef7fe640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:44:03.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.799+0000 7f27ef7fe640 1 -- 172.21.15.67:0/1840926716 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27fc01e520 con 0x55558bf3ce50 2024-03-24T00:44:03.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.799+0000 7f27ef7fe640 1 -- 172.21.15.67:0/1840926716 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f27fc022a10 con 0x55558bf3ce50 2024-03-24T00:44:03.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.799+0000 7f2819c28740 1 -- 172.21.15.67:0/1840926716 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f280403a040 con 0x55558bf3ce50 2024-03-24T00:44:03.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.799+0000 7f27ef7fe640 1 -- 172.21.15.67:0/1840926716 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f27fc022c10 con 0x55558bf3ce50 2024-03-24T00:44:03.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.799+0000 7f2819c28740 10 client.7299 fetch_fsmap learned FSMap version 47 2024-03-24T00:44:03.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.799+0000 7f2819c28740 1 -- 172.21.15.67:0/1840926716 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=2+}) v3 -- 0x55558bf3bb40 con 0x55558bf3ce50 2024-03-24T00:44:03.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.799+0000 7f27ef7fe640 1 -- 172.21.15.67:0/1840926716 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27fc022020 con 0x55558bf3ce50 2024-03-24T00:44:03.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.800+0000 7f27ef7fe640 1 -- 172.21.15.67:0/1840926716 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 47) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f27fc0221d0 con 0x55558bf3ce50 2024-03-24T00:44:03.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.800+0000 7f2819c28740 10 client.7299 fetch_fsmap finished waiting for FSMap version 47 2024-03-24T00:44:03.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.800+0000 7f2819c28740 10 client.7299 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.c 2024-03-24T00:44:03.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.800+0000 7f2819c28740 20 client.7299 populate_metadata read hostname 'smithi067' 2024-03-24T00:44:03.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.800+0000 7f2819c28740 1 --2- 172.21.15.67:0/1840926716 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x55558bf6e2d0 0x55558bf30280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:03.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.800+0000 7f2819c28740 4 client.7299 mds_command: new command op to 7149 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:44:03.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.800+0000 7f2819c28740 1 -- 172.21.15.67:0/1840926716 --> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55558be81c60 con 0x55558bf6e2d0 2024-03-24T00:44:03.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.800+0000 7f2814cf7640 1 --2- 172.21.15.67:0/1840926716 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x55558bf6e2d0 0x55558bf30280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:03.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.800+0000 7f27ef7fe640 10 client.7299 ms_handle_connect on v2:172.21.15.67:6832/720256752 2024-03-24T00:44:03.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.800+0000 7f2814cf7640 1 --2- 172.21.15.67:0/1840926716 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x55558bf6e2d0 0x55558bf30280 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7629e9bb1aba0281 server_cookie=9bff14d89f6b48e in_seq=0 out_seq=0 2024-03-24T00:44:03.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.800+0000 7f27ef7fe640 1 -- 172.21.15.67:0/1840926716 <== mds.0 v2:172.21.15.67:6832/720256752 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55558be81c60 con 0x55558bf6e2d0 2024-03-24T00:44:03.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.800+0000 7f27ef7fe640 10 client.7299 handle_command_reply: tid=0 2024-03-24T00:44:03.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.800+0000 7f2819c28740 1 client.7299 shutdown 2024-03-24T00:44:03.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.800+0000 7f2819c28740 1 -- 172.21.15.67:0/1840926716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55558bf3ce50 msgr2=0x55558bf3b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:03.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.800+0000 7f2819c28740 1 --2- 172.21.15.67:0/1840926716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55558bf3ce50 0x55558bf3b600 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f27fc007c30 tx=0x7f27fc001d70 comp rx=0 tx=0).stop 2024-03-24T00:44:03.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.801+0000 7f2819c28740 1 -- 172.21.15.67:0/1840926716 shutdown_connections 2024-03-24T00:44:03.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.801+0000 7f2819c28740 1 --2- 172.21.15.67:0/1840926716 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55558bf3d230 0x55558bf6dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:03.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.801+0000 7f2819c28740 1 --2- 172.21.15.67:0/1840926716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55558bf3ce50 0x55558bf3b600 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:03.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.801+0000 7f2819c28740 1 --2- 172.21.15.67:0/1840926716 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x55558bf6e2d0 0x55558bf30280 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:03.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.801+0000 7f2819c28740 1 --2- 172.21.15.67:0/1840926716 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55558bf34c90 0x55558bf5ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:03.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.801+0000 7f27ef7fe640 0 client.7299 ms_handle_reset on v2:172.21.15.67:6832/720256752 2024-03-24T00:44:03.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.801+0000 7f2819c28740 1 -- 172.21.15.67:0/1840926716 >> 172.21.15.67:0/1840926716 conn(0x55558bf10330 msgr2=0x55558bf3c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:03.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.801+0000 7f2819c28740 1 -- 172.21.15.67:0/1840926716 shutdown_connections 2024-03-24T00:44:03.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.801+0000 7f2819c28740 1 -- 172.21.15.67:0/1840926716 wait complete. 2024-03-24T00:44:03.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.801+0000 7f2819c28740 20 client.7299 trim_cache size 0 max 16384 2024-03-24T00:44:03.807 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:44:03.807 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "no active scrubs running", 2024-03-24T00:44:03.807 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": {} 2024-03-24T00:44:03.807 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:44:03.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.801+0000 7f2816f5b640 1 -- 172.21.15.67:0/3636729472 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f27f406bbd0 msgr2=0x7f27f406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:03.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.801+0000 7f2816f5b640 1 --2- 172.21.15.67:0/3636729472 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f27f406bbd0 0x7f27f406e090 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f2804036000 tx=0x7f2804030000 comp rx=0 tx=0).stop 2024-03-24T00:44:03.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.801+0000 7f2816f5b640 1 -- 172.21.15.67:0/3636729472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28101f3970 msgr2=0x7f28101f3df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:03.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.801+0000 7f2816f5b640 1 --2- 172.21.15.67:0/3636729472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28101f3970 0x7f28101f3df0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f28100d0220 tx=0x7f280800eb90 comp rx=0 tx=0).stop 2024-03-24T00:44:03.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.801+0000 7f2816f5b640 1 -- 172.21.15.67:0/3636729472 shutdown_connections 2024-03-24T00:44:03.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.801+0000 7f2816f5b640 1 --2- 172.21.15.67:0/3636729472 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f27f406bbd0 0x7f27f406e090 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:03.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.801+0000 7f2816f5b640 1 --2- 172.21.15.67:0/3636729472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28101f3970 0x7f28101f3df0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:03.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.801+0000 7f2816f5b640 1 --2- 172.21.15.67:0/3636729472 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2810160b10 0x7f2810160f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:03.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.801+0000 7f2816f5b640 1 --2- 172.21.15.67:0/3636729472 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2800003b90 0x7f2810155490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:03.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.801+0000 7f2816f5b640 1 -- 172.21.15.67:0/3636729472 >> 172.21.15.67:0/3636729472 conn(0x7f2810072da0 msgr2=0x7f28100b39e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:03.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.801+0000 7f2816f5b640 1 -- 172.21.15.67:0/3636729472 shutdown_connections 2024-03-24T00:44:03.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.801+0000 7f2816f5b640 1 -- 172.21.15.67:0/3636729472 wait complete. 2024-03-24T00:44:03.813 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T00:44:03.814 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:44:03.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.915+0000 7f64c8f40640 1 Processor -- start 2024-03-24T00:44:03.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.915+0000 7f64c8f40640 1 -- start start 2024-03-24T00:44:03.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.916+0000 7f64c8f40640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f64c415dec0 0x7f64c40ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:03.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.916+0000 7f64c8f40640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f64c415e2a0 0x7f64c4162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:03.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.916+0000 7f64c8f40640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c40cf640 con 0x7f64c40d07e0 2024-03-24T00:44:03.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.916+0000 7f64c8f40640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64c40cee30 con 0x7f64c415dec0 2024-03-24T00:44:03.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.916+0000 7f64c8f40640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64c40d2120 con 0x7f64c415e2a0 2024-03-24T00:44:03.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.917+0000 7f64c2d9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f64c40d07e0 0x7f64c40b5850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46864/0 (socket says 172.21.15.67:46864) 2024-03-24T00:44:03.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.917+0000 7f64c2d9d640 1 -- 172.21.15.67:0/2260304060 learned_addr learned my addr 172.21.15.67:0/2260304060 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:44:03.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.917+0000 7f64c259c640 1 --2- 172.21.15.67:0/2260304060 >> v2:172.21.15.72:3300/0 conn(0x7f64c415dec0 0x7f64c40ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:03.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.917+0000 7f64c1d9b640 1 --2- 172.21.15.67:0/2260304060 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f64c415e2a0 0x7f64c4162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:03.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.917+0000 7f64c159a640 1 -- 172.21.15.67:0/2260304060 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3598080424 0 0) 0x7f64bc001490 con 0x7f64c40d07e0 2024-03-24T00:44:03.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.917+0000 7f64c159a640 1 -- 172.21.15.67:0/2260304060 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64b0003520 con 0x7f64c40d07e0 2024-03-24T00:44:03.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.917+0000 7f64c1d9b640 1 -- 172.21.15.67:0/2260304060 >> v2:172.21.15.72:3300/0 conn(0x7f64c415dec0 msgr2=0x7f64c40ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:03.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.917+0000 7f64c1d9b640 1 --2- 172.21.15.67:0/2260304060 >> v2:172.21.15.72:3300/0 conn(0x7f64c415dec0 0x7f64c40ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:03.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.917+0000 7f64c1d9b640 1 -- 172.21.15.67:0/2260304060 >> v1:172.21.15.67:6789/0 conn(0x7f64c40d07e0 legacy=0x7f64c40b5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:03.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.917+0000 7f64c1d9b640 1 -- 172.21.15.67:0/2260304060 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c40cf080 con 0x7f64c415e2a0 2024-03-24T00:44:03.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.918+0000 7f64c1d9b640 1 --2- 172.21.15.67:0/2260304060 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f64c415e2a0 0x7f64c4162630 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f64c40cefb0 tx=0x7f64ac02b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84ccbcbba7c4d476 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:03.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.918+0000 7f64c159a640 1 -- 172.21.15.67:0/2260304060 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64ac031770 con 0x7f64c415e2a0 2024-03-24T00:44:03.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.918+0000 7f64c159a640 1 -- 172.21.15.67:0/2260304060 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64ac037070 con 0x7f64c415e2a0 2024-03-24T00:44:03.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.918+0000 7f64c159a640 1 -- 172.21.15.67:0/2260304060 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64ac031b10 con 0x7f64c415e2a0 2024-03-24T00:44:03.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.919+0000 7f64c8f40640 1 -- 172.21.15.67:0/2260304060 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f64c415e2a0 msgr2=0x7f64c4162630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:03.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.919+0000 7f64c8f40640 1 --2- 172.21.15.67:0/2260304060 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f64c415e2a0 0x7f64c4162630 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f64c40cefb0 tx=0x7f64ac02b810 comp rx=0 tx=0).stop 2024-03-24T00:44:03.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.919+0000 7f64c8f40640 1 -- 172.21.15.67:0/2260304060 shutdown_connections 2024-03-24T00:44:03.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.919+0000 7f64c8f40640 1 --2- 172.21.15.67:0/2260304060 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f64c415e2a0 0x7f64c4162630 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:03.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.919+0000 7f64c8f40640 1 --2- 172.21.15.67:0/2260304060 >> v2:172.21.15.72:3300/0 conn(0x7f64c415dec0 0x7f64c40ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:03.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.919+0000 7f64c8f40640 1 -- 172.21.15.67:0/2260304060 >> 172.21.15.67:0/2260304060 conn(0x7f64c406c810 msgr2=0x7f64c406cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:03.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.919+0000 7f64c8f40640 1 -- 172.21.15.67:0/2260304060 shutdown_connections 2024-03-24T00:44:03.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.919+0000 7f64c8f40640 1 -- 172.21.15.67:0/2260304060 wait complete. 2024-03-24T00:44:03.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.920+0000 7f64c8f40640 1 Processor -- start 2024-03-24T00:44:03.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.920+0000 7f64c8f40640 1 -- start start 2024-03-24T00:44:03.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.921+0000 7f64c8f40640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64c40d07e0 0x7f64c414c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:03.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.921+0000 7f64c8f40640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64c415dec0 0x7f64c40b3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:03.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.921+0000 7f64c8f40640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f64c415e2a0 0x7f64c41f3b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:03.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.921+0000 7f64c8f40640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64c40d3740 con 0x7f64c415dec0 2024-03-24T00:44:03.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.921+0000 7f64c8f40640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64c40d35c0 con 0x7f64c40d07e0 2024-03-24T00:44:03.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.921+0000 7f64c8f40640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64c40d38c0 con 0x7f64c415e2a0 2024-03-24T00:44:03.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.922+0000 7f64c1d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64c415dec0 0x7f64c40b3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:03.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.922+0000 7f64c1d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64c415dec0 0x7f64c40b3810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51018/0 (socket says 172.21.15.67:51018) 2024-03-24T00:44:03.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.922+0000 7f64c1d9b640 1 -- 172.21.15.67:0/388612693 learned_addr learned my addr 172.21.15.67:0/388612693 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:03.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.922+0000 7f64c259c640 1 --2- 172.21.15.67:0/388612693 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64c40d07e0 0x7f64c414c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:03.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.922+0000 7f64c2d9d640 1 --2- 172.21.15.67:0/388612693 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f64c415e2a0 0x7f64c41f3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:03.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.922+0000 7f64c1d9b640 1 -- 172.21.15.67:0/388612693 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f64c415e2a0 msgr2=0x7f64c41f3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:03.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.922+0000 7f64c1d9b640 1 --2- 172.21.15.67:0/388612693 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f64c415e2a0 0x7f64c41f3b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:03.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.922+0000 7f64c1d9b640 1 -- 172.21.15.67:0/388612693 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64c40d07e0 msgr2=0x7f64c414c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:03.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.922+0000 7f64c1d9b640 1 --2- 172.21.15.67:0/388612693 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64c40d07e0 0x7f64c414c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:03.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.922+0000 7f64c1d9b640 1 -- 172.21.15.67:0/388612693 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c40b3f80 con 0x7f64c415dec0 2024-03-24T00:44:03.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.922+0000 7f64c1d9b640 1 --2- 172.21.15.67:0/388612693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64c415dec0 0x7f64c40b3810 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f64ac02b850 tx=0x7f64ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:03.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.922+0000 7f64baffd640 1 -- 172.21.15.67:0/388612693 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64ac031770 con 0x7f64c415dec0 2024-03-24T00:44:03.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.922+0000 7f64baffd640 1 -- 172.21.15.67:0/388612693 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64ac031910 con 0x7f64c415dec0 2024-03-24T00:44:03.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.922+0000 7f64c8f40640 1 -- 172.21.15.67:0/388612693 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64c40b4260 con 0x7f64c415dec0 2024-03-24T00:44:03.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.922+0000 7f64baffd640 1 -- 172.21.15.67:0/388612693 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64ac031ce0 con 0x7f64c415dec0 2024-03-24T00:44:03.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.922+0000 7f64c8f40640 1 -- 172.21.15.67:0/388612693 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64c40b4820 con 0x7f64c415dec0 2024-03-24T00:44:03.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.923+0000 7f64c8f40640 1 -- 172.21.15.67:0/388612693 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64c404f400 con 0x7f64c415dec0 2024-03-24T00:44:03.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.924+0000 7f64baffd640 1 -- 172.21.15.67:0/388612693 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f64ac031070 con 0x7f64c415dec0 2024-03-24T00:44:03.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.924+0000 7f64baffd640 1 --2- 172.21.15.67:0/388612693 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f64a406bbd0 0x7f64a406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:03.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.924+0000 7f64baffd640 1 -- 172.21.15.67:0/388612693 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f64ac0acc70 con 0x7f64c415dec0 2024-03-24T00:44:03.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.925+0000 7f64c259c640 1 --2- 172.21.15.67:0/388612693 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f64a406bbd0 0x7f64a406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:03.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.925+0000 7f64c259c640 1 --2- 172.21.15.67:0/388612693 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f64a406bbd0 0x7f64a406e090 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f64b4009b40 tx=0x7f64b4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:03.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:03.934+0000 7f64baffd640 1 -- 172.21.15.67:0/388612693 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f64ac07af30 con 0x7f64c415dec0 2024-03-24T00:44:04.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.154+0000 7f64c8f40640 1 -- 172.21.15.67:0/388612693 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64c40b28c0 con 0x7f64c415dec0 2024-03-24T00:44:04.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.154+0000 7f64baffd640 1 -- 172.21.15.67:0/388612693 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f64ac07efd0 con 0x7f64c415dec0 2024-03-24T00:44:04.155 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:44:04.155 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":47,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5960,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1888378520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1888378520},{"type":"v1","addr":"172.21.15.67:6835","nonce":1888378520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6610,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/2539718386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2539718386},{"type":"v1","addr":"172.21.15.72:6836","nonce":2539718386}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6613,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2376370770","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2376370770},{"type":"v1","addr":"172.21.15.72:6837","nonce":2376370770}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:43:10.177088+0000","modified":"2024-03-24T00:43:35.390271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7149},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7149":{"gid":7149,"name":"c","rank":0,"incarnation":43,"state":"up:active","state_seq":6,"addr":"172.21.15.67:6833/720256752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":720256752},{"type":"v1","addr":"172.21.15.67:6833","nonce":720256752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7149,"qdb_cluster":[7149]},"id":5}]} 2024-03-24T00:44:04.156 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 47 2024-03-24T00:44:04.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.157+0000 7f64c8f40640 1 -- 172.21.15.67:0/388612693 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f64a406bbd0 msgr2=0x7f64a406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:04.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.157+0000 7f64c8f40640 1 --2- 172.21.15.67:0/388612693 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f64a406bbd0 0x7f64a406e090 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f64b4009b40 tx=0x7f64b4009190 comp rx=0 tx=0).stop 2024-03-24T00:44:04.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.157+0000 7f64c8f40640 1 -- 172.21.15.67:0/388612693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64c415dec0 msgr2=0x7f64c40b3810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:04.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.157+0000 7f64c8f40640 1 --2- 172.21.15.67:0/388612693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64c415dec0 0x7f64c40b3810 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f64ac02b850 tx=0x7f64ac0046c0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.157+0000 7f64c8f40640 1 -- 172.21.15.67:0/388612693 shutdown_connections 2024-03-24T00:44:04.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.157+0000 7f64c8f40640 1 --2- 172.21.15.67:0/388612693 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f64a406bbd0 0x7f64a406e090 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.157+0000 7f64c8f40640 1 --2- 172.21.15.67:0/388612693 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f64c415e2a0 0x7f64c41f3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.157+0000 7f64c8f40640 1 --2- 172.21.15.67:0/388612693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64c415dec0 0x7f64c40b3810 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.157+0000 7f64c8f40640 1 --2- 172.21.15.67:0/388612693 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64c40d07e0 0x7f64c414c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.157+0000 7f64c8f40640 1 -- 172.21.15.67:0/388612693 >> 172.21.15.67:0/388612693 conn(0x7f64c406c810 msgr2=0x7f64c4156460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:04.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.157+0000 7f64c8f40640 1 -- 172.21.15.67:0/388612693 shutdown_connections 2024-03-24T00:44:04.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.157+0000 7f64c8f40640 1 -- 172.21.15.67:0/388612693 wait complete. 2024-03-24T00:44:04.169 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c damage ls --format=json-pretty 2024-03-24T00:44:04.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.275+0000 7fed13a36640 1 Processor -- start 2024-03-24T00:44:04.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.275+0000 7fed13a36640 1 -- start start 2024-03-24T00:44:04.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.276+0000 7fed13a36640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fed0c0d3ae0 0x7fed0c0b5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.276+0000 7fed13a36640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed0c0d2250 con 0x7fed0c15dea0 2024-03-24T00:44:04.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.276+0000 7fed13a36640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed0c0d2530 con 0x7fed0c15e280 2024-03-24T00:44:04.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.276+0000 7fed13a36640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed0c0d2810 con 0x7fed0c0d3ae0 2024-03-24T00:44:04.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.276+0000 7fed117d2640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fed0c15dea0 0x7fed0c0d33d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46870/0 (socket says 172.21.15.67:46870) 2024-03-24T00:44:04.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.276+0000 7fed117d2640 1 -- 172.21.15.67:0/2832970598 learned_addr learned my addr 172.21.15.67:0/2832970598 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:44:04.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.276+0000 7fed10fd1640 1 --2- 172.21.15.67:0/2832970598 >> v2:172.21.15.72:3300/0 conn(0x7fed0c0d3ae0 0x7fed0c0b5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.276+0000 7fed03fff640 1 -- 172.21.15.67:0/2832970598 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3743837446 0 0) 0x7fed0c0d2250 con 0x7fed0c15dea0 2024-03-24T00:44:04.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.276+0000 7fed03fff640 1 -- 172.21.15.67:0/2832970598 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecf8003520 con 0x7fed0c15dea0 2024-03-24T00:44:04.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.276+0000 7fed10fd1640 1 -- 172.21.15.67:0/2832970598 >> v1:172.21.15.72:6789/0 conn(0x7fed0c15e280 legacy=0x7fed0c0b70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:04.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.276+0000 7fed10fd1640 1 -- 172.21.15.67:0/2832970598 >> v1:172.21.15.67:6789/0 conn(0x7fed0c15dea0 legacy=0x7fed0c0d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:04.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.276+0000 7fed10fd1640 1 -- 172.21.15.67:0/2832970598 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed0c0d2af0 con 0x7fed0c0d3ae0 2024-03-24T00:44:04.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.277+0000 7fed10fd1640 1 --2- 172.21.15.67:0/2832970598 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fed0c0d3ae0 0x7fed0c0b5a60 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fecfc0097d0 tx=0x7fecfc02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=188e9bf9930a1d76 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:04.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.277+0000 7fed03fff640 1 -- 172.21.15.67:0/2832970598 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecfc035020 con 0x7fed0c0d3ae0 2024-03-24T00:44:04.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.277+0000 7fed03fff640 1 -- 172.21.15.67:0/2832970598 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecfc031830 con 0x7fed0c0d3ae0 2024-03-24T00:44:04.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.277+0000 7fed03fff640 1 -- 172.21.15.67:0/2832970598 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecfc031be0 con 0x7fed0c0d3ae0 2024-03-24T00:44:04.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.278+0000 7fed13a36640 1 -- 172.21.15.67:0/2832970598 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fed0c0d3ae0 msgr2=0x7fed0c0b5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:04.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.278+0000 7fed13a36640 1 --2- 172.21.15.67:0/2832970598 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fed0c0d3ae0 0x7fed0c0b5a60 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fecfc0097d0 tx=0x7fecfc02b8d0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.278+0000 7fed13a36640 1 -- 172.21.15.67:0/2832970598 shutdown_connections 2024-03-24T00:44:04.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.278+0000 7fed13a36640 1 --2- 172.21.15.67:0/2832970598 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fed0c0d3ae0 0x7fed0c0b5a60 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.278+0000 7fed13a36640 1 -- 172.21.15.67:0/2832970598 >> 172.21.15.67:0/2832970598 conn(0x7fed0c072da0 msgr2=0x7fed0c0d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:04.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.278+0000 7fed13a36640 1 -- 172.21.15.67:0/2832970598 shutdown_connections 2024-03-24T00:44:04.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.278+0000 7fed13a36640 1 -- 172.21.15.67:0/2832970598 wait complete. 2024-03-24T00:44:04.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.279+0000 7fed13a36640 1 Processor -- start 2024-03-24T00:44:04.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.279+0000 7fed13a36640 1 -- start start 2024-03-24T00:44:04.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.280+0000 7fed13a36640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fed0c0d3ae0 0x7fed0c160930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.280+0000 7fed13a36640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fed0c15dea0 0x7fed0c1601a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.280+0000 7fed13a36640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed0c15e280 0x7fed0c0cd330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.280+0000 7fed13a36640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c0b6370 con 0x7fed0c15e280 2024-03-24T00:44:04.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.280+0000 7fed13a36640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c0b61f0 con 0x7fed0c15dea0 2024-03-24T00:44:04.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.280+0000 7fed13a36640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c0b7800 con 0x7fed0c0d3ae0 2024-03-24T00:44:04.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.280+0000 7fed11fd3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed0c15e280 0x7fed0c0cd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.280+0000 7fed11fd3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed0c15e280 0x7fed0c0cd330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51032/0 (socket says 172.21.15.67:51032) 2024-03-24T00:44:04.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.280+0000 7fed11fd3640 1 -- 172.21.15.67:0/3110468285 learned_addr learned my addr 172.21.15.67:0/3110468285 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:04.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.280+0000 7fed117d2640 1 --2- 172.21.15.67:0/3110468285 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fed0c0d3ae0 0x7fed0c160930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.280+0000 7fed10fd1640 1 --2- 172.21.15.67:0/3110468285 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fed0c15dea0 0x7fed0c1601a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.280+0000 7fed11fd3640 1 -- 172.21.15.67:0/3110468285 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fed0c0d3ae0 msgr2=0x7fed0c160930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:04.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.280+0000 7fed11fd3640 1 --2- 172.21.15.67:0/3110468285 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fed0c0d3ae0 0x7fed0c160930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.280+0000 7fed11fd3640 1 -- 172.21.15.67:0/3110468285 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fed0c15dea0 msgr2=0x7fed0c1601a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:04.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.280+0000 7fed11fd3640 1 --2- 172.21.15.67:0/3110468285 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fed0c15dea0 0x7fed0c1601a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.280+0000 7fed11fd3640 1 -- 172.21.15.67:0/3110468285 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed0c159690 con 0x7fed0c15e280 2024-03-24T00:44:04.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.281+0000 7fed11fd3640 1 --2- 172.21.15.67:0/3110468285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed0c15e280 0x7fed0c0cd330 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fed0400c760 tx=0x7fed0400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:04.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.281+0000 7fed01ffb640 1 -- 172.21.15.67:0/3110468285 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed04003b70 con 0x7fed0c15e280 2024-03-24T00:44:04.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.281+0000 7fed01ffb640 1 -- 172.21.15.67:0/3110468285 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed04003d10 con 0x7fed0c15e280 2024-03-24T00:44:04.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.281+0000 7fed13a36640 1 -- 172.21.15.67:0/3110468285 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed0c159970 con 0x7fed0c15e280 2024-03-24T00:44:04.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.281+0000 7fed01ffb640 1 -- 172.21.15.67:0/3110468285 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed04005310 con 0x7fed0c15e280 2024-03-24T00:44:04.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.281+0000 7fed13a36640 1 -- 172.21.15.67:0/3110468285 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed0c159f30 con 0x7fed0c15e280 2024-03-24T00:44:04.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.283+0000 7fed01ffb640 1 -- 172.21.15.67:0/3110468285 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fed040054b0 con 0x7fed0c15e280 2024-03-24T00:44:04.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.283+0000 7fed01ffb640 1 --2- 172.21.15.67:0/3110468285 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fecf006bbd0 0x7fecf006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.283+0000 7fed01ffb640 1 -- 172.21.15.67:0/3110468285 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fed0408a800 con 0x7fed0c15e280 2024-03-24T00:44:04.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.283+0000 7fed117d2640 1 --2- 172.21.15.67:0/3110468285 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fecf006bbd0 0x7fecf006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.284+0000 7fed117d2640 1 --2- 172.21.15.67:0/3110468285 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fecf006bbd0 0x7fecf006e090 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fed080011e0 tx=0x7fed08007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:04.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.284+0000 7fed16629740 1 Processor -- start 2024-03-24T00:44:04.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.284+0000 7fed16629740 1 -- start start 2024-03-24T00:44:04.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.285+0000 7fed16629740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55de9845e0d0 0x55de98431300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.285+0000 7fed16629740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55de9845e4b0 0x55de98460e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.285+0000 7fed16629740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55de98431840 0x55de984613a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.285+0000 7fed16629740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de9839b1f0 con 0x55de9845e0d0 2024-03-24T00:44:04.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.285+0000 7fed16629740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de98339a80 con 0x55de9845e4b0 2024-03-24T00:44:04.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.285+0000 7fed16629740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55de98397290 con 0x55de98431840 2024-03-24T00:44:04.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.285+0000 7fed117d2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55de9845e0d0 0x55de98431300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.285+0000 7fed117d2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55de9845e0d0 0x55de98431300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51044/0 (socket says 172.21.15.67:51044) 2024-03-24T00:44:04.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.285+0000 7fed117d2640 1 -- 172.21.15.67:0/4072066248 learned_addr learned my addr 172.21.15.67:0/4072066248 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:04.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.285+0000 7fed11fd3640 1 --2- 172.21.15.67:0/4072066248 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55de98431840 0x55de984613a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.285+0000 7fed117d2640 1 -- 172.21.15.67:0/4072066248 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55de98431840 msgr2=0x55de984613a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:04.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.285+0000 7fed117d2640 1 --2- 172.21.15.67:0/4072066248 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55de98431840 0x55de984613a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.285+0000 7fed117d2640 1 -- 172.21.15.67:0/4072066248 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55de9845e4b0 msgr2=0x55de98460e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:04.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.285+0000 7fed10fd1640 1 --2- 172.21.15.67:0/4072066248 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55de9845e4b0 0x55de98460e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.285+0000 7fed117d2640 1 --2- 172.21.15.67:0/4072066248 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55de9845e4b0 0x55de98460e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.285+0000 7fed117d2640 1 -- 172.21.15.67:0/4072066248 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55de9817b7c0 con 0x55de9845e0d0 2024-03-24T00:44:04.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.285+0000 7fed11fd3640 1 --2- 172.21.15.67:0/4072066248 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55de98431840 0x55de984613a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:44:04.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.285+0000 7fed117d2640 1 --2- 172.21.15.67:0/4072066248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55de9845e0d0 0x55de98431300 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fed08001850 tx=0x7fed08012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53bfbf1bd439ee99 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:04.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.285+0000 7fecf6ffd640 1 -- 172.21.15.67:0/4072066248 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed08025070 con 0x55de9845e0d0 2024-03-24T00:44:04.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.285+0000 7fecf6ffd640 1 -- 172.21.15.67:0/4072066248 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed0801b050 con 0x55de9845e0d0 2024-03-24T00:44:04.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.285+0000 7fecf6ffd640 1 -- 172.21.15.67:0/4072066248 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed08014e00 con 0x55de9845e0d0 2024-03-24T00:44:04.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.286+0000 7fed16629740 1 -- 172.21.15.67:0/4072066248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55de9845e0d0 msgr2=0x55de98431300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:04.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.286+0000 7fed16629740 1 --2- 172.21.15.67:0/4072066248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55de9845e0d0 0x55de98431300 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fed08001850 tx=0x7fed08012000 comp rx=0 tx=0).stop 2024-03-24T00:44:04.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.286+0000 7fed16629740 1 -- 172.21.15.67:0/4072066248 shutdown_connections 2024-03-24T00:44:04.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.286+0000 7fed16629740 1 --2- 172.21.15.67:0/4072066248 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55de98431840 0x55de984613a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.286+0000 7fed16629740 1 --2- 172.21.15.67:0/4072066248 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55de9845e4b0 0x55de98460e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.286+0000 7fed16629740 1 --2- 172.21.15.67:0/4072066248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55de9845e0d0 0x55de98431300 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.286+0000 7fed16629740 1 -- 172.21.15.67:0/4072066248 >> 172.21.15.67:0/4072066248 conn(0x55de98433570 msgr2=0x55de9847be60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:04.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.286+0000 7fed16629740 1 -- 172.21.15.67:0/4072066248 shutdown_connections 2024-03-24T00:44:04.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.286+0000 7fed16629740 1 -- 172.21.15.67:0/4072066248 wait complete. 2024-03-24T00:44:04.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.287+0000 7fed16629740 1 -- start start 2024-03-24T00:44:04.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.287+0000 7fed16629740 1 Processor -- start 2024-03-24T00:44:04.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.288+0000 7fed16629740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55de9845e0d0 0x55de98464640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.288+0000 7fed16629740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55de9845e4b0 0x55de98450110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.288+0000 7fed16629740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55de98450650 0x55de98476490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.288+0000 7fed16629740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de98435880 con 0x55de98450650 2024-03-24T00:44:04.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.288+0000 7fed16629740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de98337410 con 0x55de9845e0d0 2024-03-24T00:44:04.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.288+0000 7fed16629740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55de98434c70 con 0x55de9845e4b0 2024-03-24T00:44:04.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.288+0000 7fed16629740 10 client.0 _ll_register_callbacks cb 0x55de9835fd70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:44:04.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.288+0000 7fed11fd3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55de98450650 0x55de98476490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.288+0000 7fed11fd3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55de98450650 0x55de98476490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51052/0 (socket says 172.21.15.67:51052) 2024-03-24T00:44:04.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.288+0000 7fed11fd3640 1 -- 172.21.15.67:0/1591469146 learned_addr learned my addr 172.21.15.67:0/1591469146 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:04.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.288+0000 7fed10fd1640 1 --2- 172.21.15.67:0/1591469146 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55de9845e4b0 0x55de98450110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.288+0000 7fed117d2640 1 --2- 172.21.15.67:0/1591469146 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55de9845e0d0 0x55de98464640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.288+0000 7fed11fd3640 1 -- 172.21.15.67:0/1591469146 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55de9845e4b0 msgr2=0x55de98450110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:04.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.288+0000 7fed11fd3640 1 --2- 172.21.15.67:0/1591469146 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55de9845e4b0 0x55de98450110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.288+0000 7fed11fd3640 1 -- 172.21.15.67:0/1591469146 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55de9845e0d0 msgr2=0x55de98464640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:04.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.288+0000 7fed11fd3640 1 --2- 172.21.15.67:0/1591469146 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55de9845e0d0 0x55de98464640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.288+0000 7fed11fd3640 1 -- 172.21.15.67:0/1591469146 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55de981dbf50 con 0x55de98450650 2024-03-24T00:44:04.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.288+0000 7fed11fd3640 1 --2- 172.21.15.67:0/1591469146 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55de98450650 0x55de98476490 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fed04054fb0 tx=0x7fed04058c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dac2916accb05bd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:04.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.288+0000 7fecebfff640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:44:04.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.288+0000 7fecebfff640 1 -- 172.21.15.67:0/1591469146 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed040551c0 con 0x55de98450650 2024-03-24T00:44:04.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.288+0000 7fecebfff640 1 -- 172.21.15.67:0/1591469146 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fed04083dd0 con 0x55de98450650 2024-03-24T00:44:04.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.288+0000 7fed16629740 1 -- 172.21.15.67:0/1591469146 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fed0801b050 con 0x55de98450650 2024-03-24T00:44:04.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.289+0000 7fecebfff640 1 -- 172.21.15.67:0/1591469146 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7fed04055360 con 0x55de98450650 2024-03-24T00:44:04.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.289+0000 7fed16629740 10 client.7317 fetch_fsmap learned FSMap version 47 2024-03-24T00:44:04.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.289+0000 7fed16629740 1 -- 172.21.15.67:0/1591469146 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=2+}) v3 -- 0x55de9806f0b0 con 0x55de98450650 2024-03-24T00:44:04.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.289+0000 7fecebfff640 1 -- 172.21.15.67:0/1591469146 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed0801b050 con 0x55de98450650 2024-03-24T00:44:04.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.289+0000 7fecebfff640 1 -- 172.21.15.67:0/1591469146 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 47) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fed04057fa0 con 0x55de98450650 2024-03-24T00:44:04.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.289+0000 7fed16629740 10 client.7317 fetch_fsmap finished waiting for FSMap version 47 2024-03-24T00:44:04.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.289+0000 7fed16629740 10 client.7317 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T00:44:04.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.289+0000 7fed16629740 20 client.7317 populate_metadata read hostname 'smithi067' 2024-03-24T00:44:04.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.289+0000 7fed16629740 1 --2- 172.21.15.67:0/1591469146 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x55de9848f960 0x55de9848fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.289+0000 7fed16629740 4 client.7317 mds_command: new command op to 7149 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:44:04.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.289+0000 7fed16629740 1 -- 172.21.15.67:0/1591469146 --> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55de983a8730 con 0x55de9848f960 2024-03-24T00:44:04.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.289+0000 7fed117d2640 1 --2- 172.21.15.67:0/1591469146 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x55de9848f960 0x55de9848fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.289+0000 7fed117d2640 1 --2- 172.21.15.67:0/1591469146 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x55de9848f960 0x55de9848fd40 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5259e439cdc701d6 server_cookie=3ff3e5ac4b086e43 in_seq=0 out_seq=0 2024-03-24T00:44:04.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.289+0000 7fecebfff640 10 client.7317 ms_handle_connect on v2:172.21.15.67:6832/720256752 2024-03-24T00:44:04.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.290+0000 7fecebfff640 1 -- 172.21.15.67:0/1591469146 <== mds.0 v2:172.21.15.67:6832/720256752 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55de983a8730 con 0x55de9848f960 2024-03-24T00:44:04.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.290+0000 7fecebfff640 10 client.7317 handle_command_reply: tid=0 2024-03-24T00:44:04.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.290+0000 7fed16629740 1 client.7317 shutdown 2024-03-24T00:44:04.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.290+0000 7fed16629740 1 -- 172.21.15.67:0/1591469146 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55de98450650 msgr2=0x55de98476490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:04.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.290+0000 7fed16629740 1 --2- 172.21.15.67:0/1591469146 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55de98450650 0x55de98476490 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fed04054fb0 tx=0x7fed04058c10 comp rx=0 tx=0).stop 2024-03-24T00:44:04.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.290+0000 7fed16629740 1 -- 172.21.15.67:0/1591469146 shutdown_connections 2024-03-24T00:44:04.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.290+0000 7fed16629740 1 --2- 172.21.15.67:0/1591469146 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55de98450650 0x55de98476490 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.290+0000 7fed16629740 1 --2- 172.21.15.67:0/1591469146 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55de9845e4b0 0x55de98450110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.290+0000 7fed16629740 1 --2- 172.21.15.67:0/1591469146 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x55de9848f960 0x55de9848fd40 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.290+0000 7fed16629740 1 --2- 172.21.15.67:0/1591469146 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55de9845e0d0 0x55de98464640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.290+0000 7fed16629740 1 -- 172.21.15.67:0/1591469146 >> 172.21.15.67:0/1591469146 conn(0x55de98431840 msgr2=0x55de98460920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:04.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.290+0000 7fecebfff640 0 client.7317 ms_handle_reset on v2:172.21.15.67:6832/720256752 2024-03-24T00:44:04.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.290+0000 7fed16629740 1 -- 172.21.15.67:0/1591469146 shutdown_connections 2024-03-24T00:44:04.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.290+0000 7fed16629740 1 -- 172.21.15.67:0/1591469146 wait complete. 2024-03-24T00:44:04.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.290+0000 7fed16629740 20 client.7317 trim_cache size 0 max 16384 2024-03-24T00:44:04.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.303+0000 7fed16629740 1 Processor -- start 2024-03-24T00:44:04.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.303+0000 7fed16629740 1 -- start start 2024-03-24T00:44:04.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.303+0000 7fed16629740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55de9845e0d0 0x55de9848b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.303+0000 7fed16629740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55de9845e4b0 0x55de984e53a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.304+0000 7fed16629740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55de98476490 0x55de9847c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.304+0000 7fed16629740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c200a00 con 0x55de9845e0d0 2024-03-24T00:44:04.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.304+0000 7fed16629740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c1ffb50 con 0x55de9845e4b0 2024-03-24T00:44:04.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.304+0000 7fed16629740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c1f4d10 con 0x55de98476490 2024-03-24T00:44:04.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.304+0000 7fed117d2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55de9845e0d0 0x55de9848b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.304+0000 7fed117d2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55de9845e0d0 0x55de9848b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51054/0 (socket says 172.21.15.67:51054) 2024-03-24T00:44:04.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.304+0000 7fed117d2640 1 -- 172.21.15.67:0/2176371229 learned_addr learned my addr 172.21.15.67:0/2176371229 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:04.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.304+0000 7fed117d2640 1 -- 172.21.15.67:0/2176371229 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55de98476490 msgr2=0x55de9847c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:04.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.304+0000 7fed117d2640 1 --2- 172.21.15.67:0/2176371229 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55de98476490 0x55de9847c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.304+0000 7fed117d2640 1 -- 172.21.15.67:0/2176371229 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55de9845e4b0 msgr2=0x55de984e53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:04.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.304+0000 7fed117d2640 1 --2- 172.21.15.67:0/2176371229 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55de9845e4b0 0x55de984e53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.304+0000 7fed117d2640 1 -- 172.21.15.67:0/2176371229 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55de9839d690 con 0x55de9845e0d0 2024-03-24T00:44:04.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.304+0000 7fed117d2640 1 --2- 172.21.15.67:0/2176371229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55de9845e0d0 0x55de9848b870 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fed08006fd0 tx=0x7fed08014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5403ae2aab31748f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:04.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.304+0000 7fecf67fc640 1 -- 172.21.15.67:0/2176371229 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed08025040 con 0x55de9845e0d0 2024-03-24T00:44:04.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.304+0000 7fecf67fc640 1 -- 172.21.15.67:0/2176371229 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed0801b030 con 0x55de9845e0d0 2024-03-24T00:44:04.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.304+0000 7fecf67fc640 1 -- 172.21.15.67:0/2176371229 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed08014e60 con 0x55de9845e0d0 2024-03-24T00:44:04.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.305+0000 7fed16629740 1 -- 172.21.15.67:0/2176371229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55de9845e0d0 msgr2=0x55de9848b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:04.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.305+0000 7fed16629740 1 --2- 172.21.15.67:0/2176371229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55de9845e0d0 0x55de9848b870 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fed08006fd0 tx=0x7fed08014000 comp rx=0 tx=0).stop 2024-03-24T00:44:04.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.305+0000 7fed16629740 1 -- 172.21.15.67:0/2176371229 shutdown_connections 2024-03-24T00:44:04.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.305+0000 7fed16629740 1 --2- 172.21.15.67:0/2176371229 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55de98476490 0x55de9847c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.305+0000 7fed16629740 1 --2- 172.21.15.67:0/2176371229 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55de9845e4b0 0x55de984e53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.305+0000 7fed16629740 1 --2- 172.21.15.67:0/2176371229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55de9845e0d0 0x55de9848b870 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.305+0000 7fed16629740 1 -- 172.21.15.67:0/2176371229 >> 172.21.15.67:0/2176371229 conn(0x55de98431840 msgr2=0x55de98464620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:04.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.305+0000 7fed16629740 1 -- 172.21.15.67:0/2176371229 shutdown_connections 2024-03-24T00:44:04.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.305+0000 7fed16629740 1 -- 172.21.15.67:0/2176371229 wait complete. 2024-03-24T00:44:04.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.305+0000 7fed16629740 1 -- start start 2024-03-24T00:44:04.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.305+0000 7fed16629740 1 Processor -- start 2024-03-24T00:44:04.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.306+0000 7fed16629740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55de9845e0d0 0x55de984e4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.306+0000 7fed16629740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55de9845e4b0 0x55de9847c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.306+0000 7fed16629740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55de98476490 0x55de9847c930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.306+0000 7fed16629740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de984641c0 con 0x55de98476490 2024-03-24T00:44:04.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.306+0000 7fed16629740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c1f5310 con 0x55de9845e4b0 2024-03-24T00:44:04.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.306+0000 7fed16629740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55de9844f230 con 0x55de9845e0d0 2024-03-24T00:44:04.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.306+0000 7fed16629740 10 client.0 _ll_register_callbacks cb 0x7fed04054df0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:44:04.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.306+0000 7fed11fd3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55de98476490 0x55de9847c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.306+0000 7fed11fd3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55de98476490 0x55de9847c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51058/0 (socket says 172.21.15.67:51058) 2024-03-24T00:44:04.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.306+0000 7fed11fd3640 1 -- 172.21.15.67:0/3039351198 learned_addr learned my addr 172.21.15.67:0/3039351198 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:04.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.307+0000 7fed117d2640 1 --2- 172.21.15.67:0/3039351198 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55de9845e0d0 0x55de984e4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.307+0000 7fed10fd1640 1 --2- 172.21.15.67:0/3039351198 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55de9845e4b0 0x55de9847c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.307+0000 7fed11fd3640 1 -- 172.21.15.67:0/3039351198 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55de9845e0d0 msgr2=0x55de984e4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:04.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.307+0000 7fed11fd3640 1 --2- 172.21.15.67:0/3039351198 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55de9845e0d0 0x55de984e4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.307+0000 7fed11fd3640 1 -- 172.21.15.67:0/3039351198 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55de9845e4b0 msgr2=0x55de9847c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:04.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.307+0000 7fed11fd3640 1 --2- 172.21.15.67:0/3039351198 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55de9845e4b0 0x55de9847c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.307+0000 7fed11fd3640 1 -- 172.21.15.67:0/3039351198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55de9848e3f0 con 0x55de98476490 2024-03-24T00:44:04.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.307+0000 7fed11fd3640 1 --2- 172.21.15.67:0/3039351198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55de98476490 0x55de9847c930 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fed0400c730 tx=0x7fed04011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=588a5d1fe5a7acab server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:04.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.307+0000 7fecebfff640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:44:04.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.307+0000 7fecebfff640 1 -- 172.21.15.67:0/3039351198 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed040584b0 con 0x55de98476490 2024-03-24T00:44:04.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.307+0000 7fecebfff640 1 -- 172.21.15.67:0/3039351198 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fed04070dc0 con 0x55de98476490 2024-03-24T00:44:04.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.307+0000 7fed16629740 1 -- 172.21.15.67:0/3039351198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fed0801b030 con 0x55de98476490 2024-03-24T00:44:04.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.307+0000 7fecebfff640 1 -- 172.21.15.67:0/3039351198 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7fed0405cbf0 con 0x55de98476490 2024-03-24T00:44:04.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.307+0000 7fed16629740 10 client.7323 fetch_fsmap learned FSMap version 47 2024-03-24T00:44:04.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.307+0000 7fed16629740 1 -- 172.21.15.67:0/3039351198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=2+}) v3 -- 0x55de9848e3f0 con 0x55de98476490 2024-03-24T00:44:04.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.307+0000 7fecebfff640 1 -- 172.21.15.67:0/3039351198 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed0801b030 con 0x55de98476490 2024-03-24T00:44:04.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.308+0000 7fecebfff640 1 -- 172.21.15.67:0/3039351198 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 47) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fed0405db60 con 0x55de98476490 2024-03-24T00:44:04.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.308+0000 7fed16629740 10 client.7323 fetch_fsmap finished waiting for FSMap version 47 2024-03-24T00:44:04.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.308+0000 7fed16629740 10 client.7323 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T00:44:04.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.308+0000 7fed16629740 20 client.7323 populate_metadata read hostname 'smithi067' 2024-03-24T00:44:04.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.308+0000 7fed16629740 1 --2- 172.21.15.67:0/3039351198 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x55de98436680 0x55de98436a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.308+0000 7fed16629740 4 client.7323 mds_command: new command op to 7149 tid=0 multi_id=0 [{"prefix": "damage ls", "format": "json-pretty"}] 2024-03-24T00:44:04.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.308+0000 7fed16629740 1 -- 172.21.15.67:0/3039351198 --> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) v1 -- 0x55de983a2a80 con 0x55de98436680 2024-03-24T00:44:04.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.308+0000 7fed117d2640 1 --2- 172.21.15.67:0/3039351198 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x55de98436680 0x55de98436a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.308+0000 7fecebfff640 10 client.7323 ms_handle_connect on v2:172.21.15.67:6832/720256752 2024-03-24T00:44:04.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.308+0000 7fed117d2640 1 --2- 172.21.15.67:0/3039351198 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x55de98436680 0x55de98436a60 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8aa2459f1c8ad6b5 server_cookie=436773606bf2c57c in_seq=0 out_seq=0 2024-03-24T00:44:04.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.308+0000 7fecebfff640 1 -- 172.21.15.67:0/3039351198 <== mds.0 v2:172.21.15.67:6832/720256752 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+237 (crc 0 0 0) 0x55de983a2a80 con 0x55de98436680 2024-03-24T00:44:04.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.308+0000 7fecebfff640 10 client.7323 handle_command_reply: tid=0 2024-03-24T00:44:04.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.308+0000 7fed16629740 1 client.7323 shutdown 2024-03-24T00:44:04.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.308+0000 7fed16629740 1 -- 172.21.15.67:0/3039351198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55de98476490 msgr2=0x55de9847c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:04.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.308+0000 7fed16629740 1 --2- 172.21.15.67:0/3039351198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55de98476490 0x55de9847c930 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fed0400c730 tx=0x7fed04011000 comp rx=0 tx=0).stop 2024-03-24T00:44:04.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.308+0000 7fed16629740 1 -- 172.21.15.67:0/3039351198 shutdown_connections 2024-03-24T00:44:04.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.308+0000 7fed16629740 1 --2- 172.21.15.67:0/3039351198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55de98476490 0x55de9847c930 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.308+0000 7fed16629740 1 --2- 172.21.15.67:0/3039351198 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x55de98436680 0x55de98436a60 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.308+0000 7fed16629740 1 --2- 172.21.15.67:0/3039351198 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55de9845e4b0 0x55de9847c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.308+0000 7fed16629740 1 --2- 172.21.15.67:0/3039351198 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55de9845e0d0 0x55de984e4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.308+0000 7fecebfff640 0 client.7323 ms_handle_reset on v2:172.21.15.67:6832/720256752 2024-03-24T00:44:04.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.308+0000 7fed16629740 1 -- 172.21.15.67:0/3039351198 >> 172.21.15.67:0/3039351198 conn(0x55de98431840 msgr2=0x55de9848b870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:04.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.309+0000 7fed16629740 1 -- 172.21.15.67:0/3039351198 shutdown_connections 2024-03-24T00:44:04.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.309+0000 7fed16629740 1 -- 172.21.15.67:0/3039351198 wait complete. 2024-03-24T00:44:04.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.309+0000 7fed16629740 20 client.7323 trim_cache size 0 max 16384 2024-03-24T00:44:04.315 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:44:04.316 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-03-24T00:44:04.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:44:04.316 INFO:teuthology.orchestra.run.smithi067.stdout: "damage_type": "dentry", 2024-03-24T00:44:04.316 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 4148307432, 2024-03-24T00:44:04.316 INFO:teuthology.orchestra.run.smithi067.stdout: "ino": 1099511627776, 2024-03-24T00:44:04.316 INFO:teuthology.orchestra.run.smithi067.stdout: "frag": "*", 2024-03-24T00:44:04.316 INFO:teuthology.orchestra.run.smithi067.stdout: "dname": "file_to_be_damaged", 2024-03-24T00:44:04.316 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_id": "head", 2024-03-24T00:44:04.316 INFO:teuthology.orchestra.run.smithi067.stdout: "path": "/subdir/file_to_be_damaged" 2024-03-24T00:44:04.316 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:44:04.316 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-03-24T00:44:04.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.309+0000 7fed13a36640 1 -- 172.21.15.67:0/3110468285 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fecf006bbd0 msgr2=0x7fecf006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:04.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.309+0000 7fed13a36640 1 --2- 172.21.15.67:0/3110468285 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fecf006bbd0 0x7fecf006e090 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fed080011e0 tx=0x7fed08007d30 comp rx=0 tx=0).stop 2024-03-24T00:44:04.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.309+0000 7fed13a36640 1 -- 172.21.15.67:0/3110468285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed0c15e280 msgr2=0x7fed0c0cd330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:04.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.309+0000 7fed13a36640 1 --2- 172.21.15.67:0/3110468285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed0c15e280 0x7fed0c0cd330 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fed0400c760 tx=0x7fed0400cc20 comp rx=0 tx=0).stop 2024-03-24T00:44:04.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.309+0000 7fed13a36640 1 -- 172.21.15.67:0/3110468285 shutdown_connections 2024-03-24T00:44:04.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.309+0000 7fed13a36640 1 --2- 172.21.15.67:0/3110468285 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fecf006bbd0 0x7fecf006e090 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.309+0000 7fed13a36640 1 --2- 172.21.15.67:0/3110468285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed0c15e280 0x7fed0c0cd330 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.309+0000 7fed13a36640 1 --2- 172.21.15.67:0/3110468285 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fed0c15dea0 0x7fed0c1601a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.309+0000 7fed13a36640 1 --2- 172.21.15.67:0/3110468285 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fed0c0d3ae0 0x7fed0c160930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.309+0000 7fed13a36640 1 -- 172.21.15.67:0/3110468285 >> 172.21.15.67:0/3110468285 conn(0x7fed0c072da0 msgr2=0x7fed0c0d09b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:04.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.309+0000 7fed13a36640 1 -- 172.21.15.67:0/3110468285 shutdown_connections 2024-03-24T00:44:04.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.309+0000 7fed13a36640 1 -- 172.21.15.67:0/3110468285 wait complete. 2024-03-24T00:44:04.324 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-24T00:44:04.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.425+0000 7f2b22790640 1 Processor -- start 2024-03-24T00:44:04.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.425+0000 7f2b22790640 1 -- start start 2024-03-24T00:44:04.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.426+0000 7f2b22790640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2b1c06ba50 0x7f2b1c0ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.426+0000 7f2b22790640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2b1c0cd240 0x7f2b1c0b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.426+0000 7f2b22790640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b1c0cd650 con 0x7f2b1c0cc360 2024-03-24T00:44:04.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.426+0000 7f2b22790640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b1c0caad0 con 0x7f2b1c06ba50 2024-03-24T00:44:04.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.426+0000 7f2b22790640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b1c06c210 con 0x7f2b1c0cd240 2024-03-24T00:44:04.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.427+0000 7f2b1b7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2b1c06ba50 0x7f2b1c0ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.427+0000 7f2b1b7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2b1c06ba50 0x7f2b1c0ccd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51066/0 (socket says 172.21.15.67:51066) 2024-03-24T00:44:04.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.427+0000 7f2b1b7fe640 1 -- 172.21.15.67:0/689139479 learned_addr learned my addr 172.21.15.67:0/689139479 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:04.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.427+0000 7f2b20d2d640 1 --2- 172.21.15.67:0/689139479 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2b1c0cd240 0x7f2b1c0b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.427+0000 7f2b1b7fe640 1 -- 172.21.15.67:0/689139479 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2b1c0cd240 msgr2=0x7f2b1c0b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:04.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.427+0000 7f2b1b7fe640 1 --2- 172.21.15.67:0/689139479 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2b1c0cd240 0x7f2b1c0b0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.427+0000 7f2b1b7fe640 1 -- 172.21.15.67:0/689139479 >> v1:172.21.15.72:6789/0 conn(0x7f2b1c0cc360 legacy=0x7f2b1c066e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:04.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.427+0000 7f2b1b7fe640 1 -- 172.21.15.67:0/689139479 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b1c0b4240 con 0x7f2b1c06ba50 2024-03-24T00:44:04.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.428+0000 7f2b1b7fe640 1 --2- 172.21.15.67:0/689139479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b1c06ba50 0x7f2b1c0ccd00 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f2b10009870 tx=0x7f2b1002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdc27a4b5ececc29 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:04.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.428+0000 7f2b1affd640 1 -- 172.21.15.67:0/689139479 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b10034070 con 0x7f2b1c06ba50 2024-03-24T00:44:04.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.428+0000 7f2b1affd640 1 -- 172.21.15.67:0/689139479 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b10030840 con 0x7f2b1c06ba50 2024-03-24T00:44:04.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.428+0000 7f2b1affd640 1 -- 172.21.15.67:0/689139479 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b10030c10 con 0x7f2b1c06ba50 2024-03-24T00:44:04.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.429+0000 7f2b22790640 1 -- 172.21.15.67:0/689139479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b1c06ba50 msgr2=0x7f2b1c0ccd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:04.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.429+0000 7f2b22790640 1 --2- 172.21.15.67:0/689139479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b1c06ba50 0x7f2b1c0ccd00 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f2b10009870 tx=0x7f2b1002b980 comp rx=0 tx=0).stop 2024-03-24T00:44:04.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.429+0000 7f2b22790640 1 -- 172.21.15.67:0/689139479 shutdown_connections 2024-03-24T00:44:04.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.429+0000 7f2b22790640 1 --2- 172.21.15.67:0/689139479 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2b1c0cd240 0x7f2b1c0b0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.429+0000 7f2b22790640 1 --2- 172.21.15.67:0/689139479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b1c06ba50 0x7f2b1c0ccd00 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.429+0000 7f2b22790640 1 -- 172.21.15.67:0/689139479 >> 172.21.15.67:0/689139479 conn(0x7f2b1c0b5440 msgr2=0x7f2b1c0b33d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:04.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.429+0000 7f2b22790640 1 -- 172.21.15.67:0/689139479 shutdown_connections 2024-03-24T00:44:04.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.429+0000 7f2b22790640 1 -- 172.21.15.67:0/689139479 wait complete. 2024-03-24T00:44:04.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.430+0000 7f2b22790640 1 Processor -- start 2024-03-24T00:44:04.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.431+0000 7f2b22790640 1 -- start start 2024-03-24T00:44:04.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.432+0000 7f2b22790640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2b1c06ba50 0x7f2b1c15c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.432+0000 7f2b22790640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b1c0cc360 0x7f2b1c14f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.432+0000 7f2b22790640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b1c0cd240 0x7f2b1c14fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.432+0000 7f2b22790640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b1c0b3f60 con 0x7f2b1c0cc360 2024-03-24T00:44:04.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.432+0000 7f2b22790640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b1c0b3c80 con 0x7f2b1c0cd240 2024-03-24T00:44:04.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.432+0000 7f2b22790640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b1c0cc740 con 0x7f2b1c06ba50 2024-03-24T00:44:04.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.432+0000 7f2b1b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b1c0cc360 0x7f2b1c14f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.432+0000 7f2b1b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b1c0cc360 0x7f2b1c14f4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51080/0 (socket says 172.21.15.67:51080) 2024-03-24T00:44:04.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.432+0000 7f2b1b7fe640 1 -- 172.21.15.67:0/3293124088 learned_addr learned my addr 172.21.15.67:0/3293124088 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:04.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.432+0000 7f2b1bfff640 1 --2- 172.21.15.67:0/3293124088 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2b1c06ba50 0x7f2b1c15c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.432+0000 7f2b20d2d640 1 --2- 172.21.15.67:0/3293124088 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b1c0cd240 0x7f2b1c14fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.432+0000 7f2b1b7fe640 1 -- 172.21.15.67:0/3293124088 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2b1c06ba50 msgr2=0x7f2b1c15c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:04.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.432+0000 7f2b1b7fe640 1 --2- 172.21.15.67:0/3293124088 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2b1c06ba50 0x7f2b1c15c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.432+0000 7f2b1b7fe640 1 -- 172.21.15.67:0/3293124088 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b1c0cd240 msgr2=0x7f2b1c14fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:04.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.432+0000 7f2b1b7fe640 1 --2- 172.21.15.67:0/3293124088 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b1c0cd240 0x7f2b1c14fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.432+0000 7f2b1b7fe640 1 -- 172.21.15.67:0/3293124088 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b1c0cf230 con 0x7f2b1c0cc360 2024-03-24T00:44:04.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.432+0000 7f2b1b7fe640 1 --2- 172.21.15.67:0/3293124088 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b1c0cc360 0x7f2b1c14f4e0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f2b10000c00 tx=0x7f2b100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:04.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.433+0000 7f2b18ff9640 1 -- 172.21.15.67:0/3293124088 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b10034040 con 0x7f2b1c0cc360 2024-03-24T00:44:04.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.433+0000 7f2b18ff9640 1 -- 172.21.15.67:0/3293124088 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b10039d90 con 0x7f2b1c0cc360 2024-03-24T00:44:04.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.433+0000 7f2b22790640 1 -- 172.21.15.67:0/3293124088 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b1c0cf4b0 con 0x7f2b1c0cc360 2024-03-24T00:44:04.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.433+0000 7f2b18ff9640 1 -- 172.21.15.67:0/3293124088 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b10043630 con 0x7f2b1c0cc360 2024-03-24T00:44:04.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.433+0000 7f2b22790640 1 -- 172.21.15.67:0/3293124088 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b1c0cfa70 con 0x7f2b1c0cc360 2024-03-24T00:44:04.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.434+0000 7f2b22790640 1 -- 172.21.15.67:0/3293124088 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b1c04f400 con 0x7f2b1c0cc360 2024-03-24T00:44:04.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.434+0000 7f2b18ff9640 1 -- 172.21.15.67:0/3293124088 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2b1004a060 con 0x7f2b1c0cc360 2024-03-24T00:44:04.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.434+0000 7f2b18ff9640 1 --2- 172.21.15.67:0/3293124088 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2b0806bc00 0x7f2b0806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.435+0000 7f2b18ff9640 1 -- 172.21.15.67:0/3293124088 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f2b100ac9b0 con 0x7f2b1c0cc360 2024-03-24T00:44:04.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.435+0000 7f2b1bfff640 1 --2- 172.21.15.67:0/3293124088 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2b0806bc00 0x7f2b0806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.435+0000 7f2b1bfff640 1 --2- 172.21.15.67:0/3293124088 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2b0806bc00 0x7f2b0806e0c0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c000f60 tx=0x7f2b0c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:04.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.439+0000 7f2b18ff9640 1 -- 172.21.15.67:0/3293124088 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2b1c0cf230 con 0x7f2b1c0cc360 2024-03-24T00:44:04.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.658+0000 7f2b22790640 1 -- 172.21.15.67:0/3293124088 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2b1c0b4840 con 0x7f2b1c0cc360 2024-03-24T00:44:04.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.658+0000 7f2b18ff9640 1 -- 172.21.15.67:0/3293124088 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+164 (secure 0 0 0) 0x7f2b1003e070 con 0x7f2b1c0cc360 2024-03-24T00:44:04.658 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:44:04.658 INFO:teuthology.orchestra.run.smithi067.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2024-03-24T00:44:04.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.660+0000 7f2b22790640 1 -- 172.21.15.67:0/3293124088 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2b0806bc00 msgr2=0x7f2b0806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:04.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.660+0000 7f2b22790640 1 --2- 172.21.15.67:0/3293124088 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2b0806bc00 0x7f2b0806e0c0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c000f60 tx=0x7f2b0c009040 comp rx=0 tx=0).stop 2024-03-24T00:44:04.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.660+0000 7f2b22790640 1 -- 172.21.15.67:0/3293124088 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b1c0cc360 msgr2=0x7f2b1c14f4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:04.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.660+0000 7f2b22790640 1 --2- 172.21.15.67:0/3293124088 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b1c0cc360 0x7f2b1c14f4e0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f2b10000c00 tx=0x7f2b100046c0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.660+0000 7f2b22790640 1 -- 172.21.15.67:0/3293124088 shutdown_connections 2024-03-24T00:44:04.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.660+0000 7f2b22790640 1 --2- 172.21.15.67:0/3293124088 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2b0806bc00 0x7f2b0806e0c0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.660+0000 7f2b22790640 1 --2- 172.21.15.67:0/3293124088 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b1c0cd240 0x7f2b1c14fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.660+0000 7f2b22790640 1 --2- 172.21.15.67:0/3293124088 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b1c0cc360 0x7f2b1c14f4e0 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.660+0000 7f2b22790640 1 --2- 172.21.15.67:0/3293124088 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2b1c06ba50 0x7f2b1c15c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.660+0000 7f2b22790640 1 -- 172.21.15.67:0/3293124088 >> 172.21.15.67:0/3293124088 conn(0x7f2b1c0b5440 msgr2=0x7f2b1c0af250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:04.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.660+0000 7f2b22790640 1 -- 172.21.15.67:0/3293124088 shutdown_connections 2024-03-24T00:44:04.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.660+0000 7f2b22790640 1 -- 172.21.15.67:0/3293124088 wait complete. 2024-03-24T00:44:04.671 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-03-24T00:44:04.671 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:44:04.671 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.5:0 scrub start /subdir repair,recursive 2024-03-24T00:44:04.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.777+0000 7f2aa42b2640 1 Processor -- start 2024-03-24T00:44:04.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.777+0000 7f2aa42b2640 1 -- start start 2024-03-24T00:44:04.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.778+0000 7f2aa42b2640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2a9c15dfc0 0x7f2a9c1624c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.778+0000 7f2aa42b2640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2a9c0d0d50 0x7f2a9c0d2b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.778+0000 7f2aa42b2640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a9c162a00 con 0x7f2a9c0ced60 2024-03-24T00:44:04.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.778+0000 7f2aa42b2640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a9c15e3a0 con 0x7f2a9c15dfc0 2024-03-24T00:44:04.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.778+0000 7f2aa42b2640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a9c0cf4d0 con 0x7f2a9c0d0d50 2024-03-24T00:44:04.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.778+0000 7f2aa204e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2a9c15dfc0 0x7f2a9c1624c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.778+0000 7f2aa184d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2a9c0ced60 0x7f2a9c0d0640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46876/0 (socket says 172.21.15.67:46876) 2024-03-24T00:44:04.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.778+0000 7f2aa184d640 1 -- 172.21.15.67:0/3115494779 learned_addr learned my addr 172.21.15.67:0/3115494779 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:44:04.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.778+0000 7f2aa284f640 1 --2- 172.21.15.67:0/3115494779 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2a9c0d0d50 0x7f2a9c0d2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.778+0000 7f2aa204e640 1 -- 172.21.15.67:0/3115494779 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2a9c0d0d50 msgr2=0x7f2a9c0d2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:04.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.778+0000 7f2aa204e640 1 --2- 172.21.15.67:0/3115494779 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2a9c0d0d50 0x7f2a9c0d2b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.778+0000 7f2aa204e640 1 -- 172.21.15.67:0/3115494779 >> v1:172.21.15.67:6789/0 conn(0x7f2a9c0ced60 legacy=0x7f2a9c0d0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:04.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.778+0000 7f2aa204e640 1 -- 172.21.15.67:0/3115494779 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a9c0cf1f0 con 0x7f2a9c15dfc0 2024-03-24T00:44:04.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.778+0000 7f2aa104c640 1 -- 172.21.15.67:0/3115494779 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 667465454 0 0) 0x7f2a9c162a00 con 0x7f2a9c0ced60 2024-03-24T00:44:04.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.778+0000 7f2aa284f640 1 --2- 172.21.15.67:0/3115494779 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2a9c0d0d50 0x7f2a9c0d2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:44:04.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.779+0000 7f2aa204e640 1 --2- 172.21.15.67:0/3115494779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a9c15dfc0 0x7f2a9c1624c0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f2a980097a0 tx=0x7f2a9802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6aec8f66f5c4deb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:04.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.779+0000 7f2aa104c640 1 -- 172.21.15.67:0/3115494779 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a98035020 con 0x7f2a9c15dfc0 2024-03-24T00:44:04.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.779+0000 7f2aa104c640 1 -- 172.21.15.67:0/3115494779 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a98031830 con 0x7f2a9c15dfc0 2024-03-24T00:44:04.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.779+0000 7f2aa104c640 1 -- 172.21.15.67:0/3115494779 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a98031c00 con 0x7f2a9c15dfc0 2024-03-24T00:44:04.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.780+0000 7f2aa42b2640 1 -- 172.21.15.67:0/3115494779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a9c15dfc0 msgr2=0x7f2a9c1624c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:04.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.780+0000 7f2aa42b2640 1 --2- 172.21.15.67:0/3115494779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a9c15dfc0 0x7f2a9c1624c0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f2a980097a0 tx=0x7f2a9802b8d0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.780+0000 7f2aa42b2640 1 -- 172.21.15.67:0/3115494779 shutdown_connections 2024-03-24T00:44:04.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.780+0000 7f2aa42b2640 1 --2- 172.21.15.67:0/3115494779 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2a9c0d0d50 0x7f2a9c0d2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.780+0000 7f2aa42b2640 1 --2- 172.21.15.67:0/3115494779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a9c15dfc0 0x7f2a9c1624c0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.780+0000 7f2aa42b2640 1 -- 172.21.15.67:0/3115494779 >> 172.21.15.67:0/3115494779 conn(0x7f2a9c06c880 msgr2=0x7f2a9c06cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:04.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.780+0000 7f2aa42b2640 1 -- 172.21.15.67:0/3115494779 shutdown_connections 2024-03-24T00:44:04.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.780+0000 7f2aa42b2640 1 -- 172.21.15.67:0/3115494779 wait complete. 2024-03-24T00:44:04.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.781+0000 7f2aa42b2640 1 Processor -- start 2024-03-24T00:44:04.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.781+0000 7f2aa42b2640 1 -- start start 2024-03-24T00:44:04.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.782+0000 7f2aa42b2640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2a9c0d0d50 0x7f2a9c0ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.782+0000 7f2aa42b2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a9c15dfc0 0x7f2a9c0b5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.782+0000 7f2aa42b2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a9c0b5fc0 0x7f2a9c0cb060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.782+0000 7f2aa42b2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a9c0d25e0 con 0x7f2a9c0b5fc0 2024-03-24T00:44:04.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.782+0000 7f2aa42b2640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a9c0d2300 con 0x7f2a9c15dfc0 2024-03-24T00:44:04.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.782+0000 7f2aa42b2640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a9c0b6ea0 con 0x7f2a9c0d0d50 2024-03-24T00:44:04.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.782+0000 7f2aa284f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a9c0b5fc0 0x7f2a9c0cb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.782+0000 7f2aa284f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a9c0b5fc0 0x7f2a9c0cb060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51098/0 (socket says 172.21.15.67:51098) 2024-03-24T00:44:04.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.782+0000 7f2aa284f640 1 -- 172.21.15.67:0/4233936449 learned_addr learned my addr 172.21.15.67:0/4233936449 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:04.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.782+0000 7f2aa204e640 1 --2- 172.21.15.67:0/4233936449 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2a9c0d0d50 0x7f2a9c0ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.782+0000 7f2aa184d640 1 --2- 172.21.15.67:0/4233936449 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a9c15dfc0 0x7f2a9c0b5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.782+0000 7f2aa284f640 1 -- 172.21.15.67:0/4233936449 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2a9c0d0d50 msgr2=0x7f2a9c0ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:04.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.782+0000 7f2aa284f640 1 --2- 172.21.15.67:0/4233936449 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2a9c0d0d50 0x7f2a9c0ccf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.782+0000 7f2aa284f640 1 -- 172.21.15.67:0/4233936449 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a9c15dfc0 msgr2=0x7f2a9c0b5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:04.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.782+0000 7f2aa284f640 1 --2- 172.21.15.67:0/4233936449 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a9c15dfc0 0x7f2a9c0b5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.782+0000 7f2aa284f640 1 -- 172.21.15.67:0/4233936449 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a9c159d50 con 0x7f2a9c0b5fc0 2024-03-24T00:44:04.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.782+0000 7f2aa284f640 1 --2- 172.21.15.67:0/4233936449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a9c0b5fc0 0x7f2a9c0cb060 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f2a9400ce80 tx=0x7f2a9400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:04.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.782+0000 7f2a92ffd640 1 -- 172.21.15.67:0/4233936449 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9401a070 con 0x7f2a9c0b5fc0 2024-03-24T00:44:04.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.782+0000 7f2a92ffd640 1 -- 172.21.15.67:0/4233936449 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a9400d880 con 0x7f2a9c0b5fc0 2024-03-24T00:44:04.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.783+0000 7f2aa42b2640 1 -- 172.21.15.67:0/4233936449 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a9c0cc0c0 con 0x7f2a9c0b5fc0 2024-03-24T00:44:04.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.783+0000 7f2a92ffd640 1 -- 172.21.15.67:0/4233936449 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9400dc70 con 0x7f2a9c0b5fc0 2024-03-24T00:44:04.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.783+0000 7f2aa42b2640 1 -- 172.21.15.67:0/4233936449 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a9c0cb5a0 con 0x7f2a9c0b5fc0 2024-03-24T00:44:04.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.784+0000 7f2a92ffd640 1 -- 172.21.15.67:0/4233936449 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2a9401d070 con 0x7f2a9c0b5fc0 2024-03-24T00:44:04.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.785+0000 7f2a92ffd640 1 --2- 172.21.15.67:0/4233936449 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2a8006bbd0 0x7f2a8006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.785+0000 7f2a92ffd640 1 -- 172.21.15.67:0/4233936449 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f2a9408b3a0 con 0x7f2a9c0b5fc0 2024-03-24T00:44:04.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.785+0000 7f2aa204e640 1 --2- 172.21.15.67:0/4233936449 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2a8006bbd0 0x7f2a8006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.785+0000 7f2aa204e640 1 --2- 172.21.15.67:0/4233936449 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2a8006bbd0 0x7f2a8006e090 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f2a98036000 tx=0x7f2a9803c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:04.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.785+0000 7f2aa6f7e740 1 Processor -- start 2024-03-24T00:44:04.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.786+0000 7f2aa6f7e740 1 -- start start 2024-03-24T00:44:04.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.786+0000 7f2aa6f7e740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5630fa6c3dd0 0x5630fa6ea900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.786+0000 7f2aa6f7e740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5630fa6eae40 0x5630fa6c2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.786+0000 7f2aa6f7e740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5630fa6c41b0 0x5630fa6c3350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.786+0000 7f2aa6f7e740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630fa628240 con 0x5630fa6c3dd0 2024-03-24T00:44:04.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.786+0000 7f2aa6f7e740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630fa5d0e50 con 0x5630fa6c41b0 2024-03-24T00:44:04.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.786+0000 7f2aa6f7e740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5630fa6242e0 con 0x5630fa6eae40 2024-03-24T00:44:04.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.786+0000 7f2aa204e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5630fa6c3dd0 0x5630fa6ea900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.786+0000 7f2aa204e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5630fa6c3dd0 0x5630fa6ea900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51114/0 (socket says 172.21.15.67:51114) 2024-03-24T00:44:04.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.786+0000 7f2aa204e640 1 -- 172.21.15.67:0/818410966 learned_addr learned my addr 172.21.15.67:0/818410966 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:04.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.787+0000 7f2aa284f640 1 --2- 172.21.15.67:0/818410966 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5630fa6c41b0 0x5630fa6c3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.787+0000 7f2aa184d640 1 --2- 172.21.15.67:0/818410966 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5630fa6eae40 0x5630fa6c2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.787+0000 7f2aa204e640 1 -- 172.21.15.67:0/818410966 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5630fa6eae40 msgr2=0x5630fa6c2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:04.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.787+0000 7f2aa204e640 1 --2- 172.21.15.67:0/818410966 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5630fa6eae40 0x5630fa6c2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.787+0000 7f2aa204e640 1 -- 172.21.15.67:0/818410966 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5630fa6c41b0 msgr2=0x5630fa6c3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:04.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.787+0000 7f2aa204e640 1 --2- 172.21.15.67:0/818410966 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5630fa6c41b0 0x5630fa6c3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.787+0000 7f2aa204e640 1 -- 172.21.15.67:0/818410966 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5630fa5c8ee0 con 0x5630fa6c3dd0 2024-03-24T00:44:04.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.787+0000 7f2aa204e640 1 --2- 172.21.15.67:0/818410966 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5630fa6c3dd0 0x5630fa6ea900 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f2a9802bee0 tx=0x7f2a98030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5047c04bec58161 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:04.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.787+0000 7f2a7ffff640 1 -- 172.21.15.67:0/818410966 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a98053070 con 0x5630fa6c3dd0 2024-03-24T00:44:04.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.787+0000 7f2a7ffff640 1 -- 172.21.15.67:0/818410966 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a98031ca0 con 0x5630fa6c3dd0 2024-03-24T00:44:04.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.787+0000 7f2a7ffff640 1 -- 172.21.15.67:0/818410966 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9804e670 con 0x5630fa6c3dd0 2024-03-24T00:44:04.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.788+0000 7f2aa6f7e740 1 -- 172.21.15.67:0/818410966 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5630fa6c3dd0 msgr2=0x5630fa6ea900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:04.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.788+0000 7f2aa6f7e740 1 --2- 172.21.15.67:0/818410966 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5630fa6c3dd0 0x5630fa6ea900 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f2a9802bee0 tx=0x7f2a98030000 comp rx=0 tx=0).stop 2024-03-24T00:44:04.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.788+0000 7f2aa6f7e740 1 -- 172.21.15.67:0/818410966 shutdown_connections 2024-03-24T00:44:04.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.788+0000 7f2aa6f7e740 1 --2- 172.21.15.67:0/818410966 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5630fa6c41b0 0x5630fa6c3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.788+0000 7f2aa6f7e740 1 --2- 172.21.15.67:0/818410966 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5630fa6eae40 0x5630fa6c2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.788+0000 7f2aa6f7e740 1 --2- 172.21.15.67:0/818410966 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5630fa6c3dd0 0x5630fa6ea900 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.788+0000 7f2aa6f7e740 1 -- 172.21.15.67:0/818410966 >> 172.21.15.67:0/818410966 conn(0x5630fa68ae00 msgr2=0x5630fa708af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:04.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.788+0000 7f2aa6f7e740 1 -- 172.21.15.67:0/818410966 shutdown_connections 2024-03-24T00:44:04.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.788+0000 7f2aa6f7e740 1 -- 172.21.15.67:0/818410966 wait complete. 2024-03-24T00:44:04.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.789+0000 7f2aa6f7e740 1 -- start start 2024-03-24T00:44:04.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.789+0000 7f2aa6f7e740 1 Processor -- start 2024-03-24T00:44:04.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.790+0000 7f2aa6f7e740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5630fa6c3dd0 0x5630fa6e3770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.790+0000 7f2aa6f7e740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5630fa6c41b0 0x5630fa6ef6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.790+0000 7f2aa6f7e740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5630fa6e89a0 0x5630fa70bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.790+0000 7f2aa6f7e740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630fa6ea470 con 0x5630fa6e89a0 2024-03-24T00:44:04.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.790+0000 7f2aa6f7e740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630fa5e42a0 con 0x5630fa6c3dd0 2024-03-24T00:44:04.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.790+0000 7f2aa6f7e740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5630fa6e31f0 con 0x5630fa6c41b0 2024-03-24T00:44:04.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.790+0000 7f2aa6f7e740 10 client.0 _ll_register_callbacks cb 0x5630fa61fe00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:44:04.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.790+0000 7f2aa284f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5630fa6e89a0 0x5630fa70bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.790+0000 7f2aa284f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5630fa6e89a0 0x5630fa70bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51120/0 (socket says 172.21.15.67:51120) 2024-03-24T00:44:04.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.790+0000 7f2aa284f640 1 -- 172.21.15.67:0/3976990438 learned_addr learned my addr 172.21.15.67:0/3976990438 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:04.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.790+0000 7f2aa204e640 1 --2- 172.21.15.67:0/3976990438 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5630fa6c3dd0 0x5630fa6e3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.790+0000 7f2aa184d640 1 --2- 172.21.15.67:0/3976990438 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5630fa6c41b0 0x5630fa6ef6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.790+0000 7f2aa284f640 1 -- 172.21.15.67:0/3976990438 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5630fa6c41b0 msgr2=0x5630fa6ef6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:04.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.790+0000 7f2aa284f640 1 --2- 172.21.15.67:0/3976990438 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5630fa6c41b0 0x5630fa6ef6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.790+0000 7f2aa284f640 1 -- 172.21.15.67:0/3976990438 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5630fa6c3dd0 msgr2=0x5630fa6e3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:04.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.790+0000 7f2aa284f640 1 --2- 172.21.15.67:0/3976990438 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5630fa6c3dd0 0x5630fa6e3770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.790+0000 7f2aa284f640 1 -- 172.21.15.67:0/3976990438 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5630fa404a50 con 0x5630fa6e89a0 2024-03-24T00:44:04.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.790+0000 7f2aa284f640 1 --2- 172.21.15.67:0/3976990438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5630fa6e89a0 0x5630fa70bf40 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f2a9400ce50 tx=0x7f2a94059660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b62bfaeb2a1fd2a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:04.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.790+0000 7f2a7d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:44:04.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.790+0000 7f2a7d7fa640 1 -- 172.21.15.67:0/3976990438 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9400de10 con 0x5630fa6e89a0 2024-03-24T00:44:04.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.790+0000 7f2a7d7fa640 1 -- 172.21.15.67:0/3976990438 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f2a94061830 con 0x5630fa6e89a0 2024-03-24T00:44:04.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.790+0000 7f2aa6f7e740 1 -- 172.21.15.67:0/3976990438 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2a98031ca0 con 0x5630fa6e89a0 2024-03-24T00:44:04.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.790+0000 7f2a7d7fa640 1 -- 172.21.15.67:0/3976990438 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2a94061a30 con 0x5630fa6e89a0 2024-03-24T00:44:04.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.790+0000 7f2aa6f7e740 10 client.7344 fetch_fsmap learned FSMap version 47 2024-03-24T00:44:04.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.791+0000 7f2aa6f7e740 1 -- 172.21.15.67:0/3976990438 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=2+}) v3 -- 0x5630fa2fc3e0 con 0x5630fa6e89a0 2024-03-24T00:44:04.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.791+0000 7f2a7d7fa640 1 -- 172.21.15.67:0/3976990438 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a98031ca0 con 0x5630fa6e89a0 2024-03-24T00:44:04.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.791+0000 7f2a7d7fa640 1 -- 172.21.15.67:0/3976990438 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 47) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f2a94058ab0 con 0x5630fa6e89a0 2024-03-24T00:44:04.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.791+0000 7f2aa6f7e740 10 client.7344 fetch_fsmap finished waiting for FSMap version 47 2024-03-24T00:44:04.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.791+0000 7f2aa6f7e740 10 client.7344 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.c 2024-03-24T00:44:04.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.791+0000 7f2aa6f7e740 20 client.7344 populate_metadata read hostname 'smithi067' 2024-03-24T00:44:04.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.791+0000 7f2aa6f7e740 1 --2- 172.21.15.67:0/3976990438 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x5630fa6f0d30 0x5630fa6f1140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.791+0000 7f2aa6f7e740 4 client.7344 mds_command: new command op to 7149 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:44:04.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.791+0000 7f2aa6f7e740 1 -- 172.21.15.67:0/3976990438 --> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5630fa5fc7b0 con 0x5630fa6f0d30 2024-03-24T00:44:04.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.791+0000 7f2aa204e640 1 --2- 172.21.15.67:0/3976990438 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x5630fa6f0d30 0x5630fa6f1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.791+0000 7f2a7d7fa640 10 client.7344 ms_handle_connect on v2:172.21.15.67:6832/720256752 2024-03-24T00:44:04.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.791+0000 7f2aa204e640 1 --2- 172.21.15.67:0/3976990438 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x5630fa6f0d30 0x5630fa6f1140 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da7d867ce41e47bd server_cookie=1232f587565fa3fd in_seq=0 out_seq=0 2024-03-24T00:44:04.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.792+0000 7f2a7d7fa640 1 -- 172.21.15.67:0/3976990438 <== mds.0 v2:172.21.15.67:6832/720256752 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5630fa5fc7b0 con 0x5630fa6f0d30 2024-03-24T00:44:04.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.792+0000 7f2a7d7fa640 10 client.7344 handle_command_reply: tid=0 2024-03-24T00:44:04.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.792+0000 7f2aa6f7e740 1 client.7344 shutdown 2024-03-24T00:44:04.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.792+0000 7f2aa6f7e740 1 -- 172.21.15.67:0/3976990438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5630fa6e89a0 msgr2=0x5630fa70bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:04.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.792+0000 7f2aa6f7e740 1 --2- 172.21.15.67:0/3976990438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5630fa6e89a0 0x5630fa70bf40 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f2a9400ce50 tx=0x7f2a94059660 comp rx=0 tx=0).stop 2024-03-24T00:44:04.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.792+0000 7f2aa6f7e740 1 -- 172.21.15.67:0/3976990438 shutdown_connections 2024-03-24T00:44:04.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.792+0000 7f2aa6f7e740 1 --2- 172.21.15.67:0/3976990438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5630fa6e89a0 0x5630fa70bf40 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.792+0000 7f2aa6f7e740 1 --2- 172.21.15.67:0/3976990438 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5630fa6c41b0 0x5630fa6ef6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.792+0000 7f2aa6f7e740 1 --2- 172.21.15.67:0/3976990438 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x5630fa6f0d30 0x5630fa6f1140 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.792+0000 7f2aa6f7e740 1 --2- 172.21.15.67:0/3976990438 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5630fa6c3dd0 0x5630fa6e3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.792+0000 7f2a7d7fa640 0 client.7344 ms_handle_reset on v2:172.21.15.67:6832/720256752 2024-03-24T00:44:04.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.792+0000 7f2aa6f7e740 1 -- 172.21.15.67:0/3976990438 >> 172.21.15.67:0/3976990438 conn(0x5630fa68ae00 msgr2=0x5630fa6e9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:04.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.792+0000 7f2aa6f7e740 1 -- 172.21.15.67:0/3976990438 shutdown_connections 2024-03-24T00:44:04.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.792+0000 7f2aa6f7e740 1 -- 172.21.15.67:0/3976990438 wait complete. 2024-03-24T00:44:04.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.792+0000 7f2aa6f7e740 20 client.7344 trim_cache size 0 max 16384 2024-03-24T00:44:04.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.808+0000 7f2aa6f7e740 1 Processor -- start 2024-03-24T00:44:04.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.808+0000 7f2aa6f7e740 1 -- start start 2024-03-24T00:44:04.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.809+0000 7f2aa6f7e740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5630fa6c3dd0 0x5630fa70b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.809+0000 7f2aa6f7e740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5630fa6c41b0 0x5630fa772720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.809+0000 7f2aa6f7e740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5630fa6e89a0 0x5630fa772c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.809+0000 7f2aa6f7e740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630fa709470 con 0x5630fa6e89a0 2024-03-24T00:44:04.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.809+0000 7f2aa6f7e740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a9c200800 con 0x5630fa6c3dd0 2024-03-24T00:44:04.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.809+0000 7f2aa6f7e740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a9c1ff9b0 con 0x5630fa6c41b0 2024-03-24T00:44:04.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.809+0000 7f2aa284f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5630fa6e89a0 0x5630fa772c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.809+0000 7f2aa284f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5630fa6e89a0 0x5630fa772c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51122/0 (socket says 172.21.15.67:51122) 2024-03-24T00:44:04.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.809+0000 7f2aa284f640 1 -- 172.21.15.67:0/2634347703 learned_addr learned my addr 172.21.15.67:0/2634347703 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:04.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.810+0000 7f2aa184d640 1 --2- 172.21.15.67:0/2634347703 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5630fa6c41b0 0x5630fa772720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.810+0000 7f2aa284f640 1 -- 172.21.15.67:0/2634347703 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5630fa6c41b0 msgr2=0x5630fa772720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:04.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.810+0000 7f2aa204e640 1 --2- 172.21.15.67:0/2634347703 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5630fa6c3dd0 0x5630fa70b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.810+0000 7f2aa284f640 1 --2- 172.21.15.67:0/2634347703 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5630fa6c41b0 0x5630fa772720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.810+0000 7f2aa284f640 1 -- 172.21.15.67:0/2634347703 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5630fa6c3dd0 msgr2=0x5630fa70b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:04.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.810+0000 7f2aa284f640 1 --2- 172.21.15.67:0/2634347703 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5630fa6c3dd0 0x5630fa70b810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.810+0000 7f2aa284f640 1 -- 172.21.15.67:0/2634347703 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5630fa6e5f80 con 0x5630fa6e89a0 2024-03-24T00:44:04.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.810+0000 7f2aa284f640 1 --2- 172.21.15.67:0/2634347703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5630fa6e89a0 0x5630fa772c60 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f2a94059360 tx=0x7f2a9405b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f45e6a9af3dbee9a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:04.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.810+0000 7f2a7f7fe640 1 -- 172.21.15.67:0/2634347703 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9400de10 con 0x5630fa6e89a0 2024-03-24T00:44:04.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.810+0000 7f2a7f7fe640 1 -- 172.21.15.67:0/2634347703 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a94061830 con 0x5630fa6e89a0 2024-03-24T00:44:04.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.810+0000 7f2a7f7fe640 1 -- 172.21.15.67:0/2634347703 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a94089350 con 0x5630fa6e89a0 2024-03-24T00:44:04.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.811+0000 7f2aa6f7e740 1 -- 172.21.15.67:0/2634347703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5630fa6e89a0 msgr2=0x5630fa772c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:04.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.811+0000 7f2aa6f7e740 1 --2- 172.21.15.67:0/2634347703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5630fa6e89a0 0x5630fa772c60 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f2a94059360 tx=0x7f2a9405b000 comp rx=0 tx=0).stop 2024-03-24T00:44:04.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.811+0000 7f2aa6f7e740 1 -- 172.21.15.67:0/2634347703 shutdown_connections 2024-03-24T00:44:04.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.811+0000 7f2aa6f7e740 1 --2- 172.21.15.67:0/2634347703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5630fa6e89a0 0x5630fa772c60 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.811+0000 7f2aa6f7e740 1 --2- 172.21.15.67:0/2634347703 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5630fa6c41b0 0x5630fa772720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.811+0000 7f2aa6f7e740 1 --2- 172.21.15.67:0/2634347703 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5630fa6c3dd0 0x5630fa70b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.811+0000 7f2aa6f7e740 1 -- 172.21.15.67:0/2634347703 >> 172.21.15.67:0/2634347703 conn(0x5630fa68ae00 msgr2=0x5630fa6b9020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:04.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.811+0000 7f2aa6f7e740 1 -- 172.21.15.67:0/2634347703 shutdown_connections 2024-03-24T00:44:04.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.811+0000 7f2aa6f7e740 1 -- 172.21.15.67:0/2634347703 wait complete. 2024-03-24T00:44:04.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.811+0000 7f2aa6f7e740 1 -- start start 2024-03-24T00:44:04.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.811+0000 7f2aa6f7e740 1 Processor -- start 2024-03-24T00:44:04.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.812+0000 7f2aa6f7e740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5630fa6c3dd0 0x5630fa760960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.812+0000 7f2aa6f7e740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5630fa6c41b0 0x5630fa6bcad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.812+0000 7f2aa6f7e740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5630fa6e89a0 0x5630fa6bd010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.812+0000 7f2aa6f7e740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a9c15b6e0 con 0x5630fa6c3dd0 2024-03-24T00:44:04.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.812+0000 7f2aa6f7e740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a9c15b0e0 con 0x5630fa6e89a0 2024-03-24T00:44:04.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.812+0000 7f2aa6f7e740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5630fa7731a0 con 0x5630fa6c41b0 2024-03-24T00:44:04.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.812+0000 7f2aa6f7e740 10 client.0 _ll_register_callbacks cb 0x7f2a8c007bf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:44:04.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.812+0000 7f2aa204e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5630fa6c3dd0 0x5630fa760960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.812+0000 7f2aa204e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5630fa6c3dd0 0x5630fa760960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51124/0 (socket says 172.21.15.67:51124) 2024-03-24T00:44:04.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.812+0000 7f2aa204e640 1 -- 172.21.15.67:0/3408920175 learned_addr learned my addr 172.21.15.67:0/3408920175 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:04.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.812+0000 7f2aa184d640 1 --2- 172.21.15.67:0/3408920175 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5630fa6c41b0 0x5630fa6bcad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.812+0000 7f2aa284f640 1 --2- 172.21.15.67:0/3408920175 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5630fa6e89a0 0x5630fa6bd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.813+0000 7f2aa204e640 1 -- 172.21.15.67:0/3408920175 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5630fa6c41b0 msgr2=0x5630fa6bcad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:04.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.813+0000 7f2aa204e640 1 --2- 172.21.15.67:0/3408920175 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5630fa6c41b0 0x5630fa6bcad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.813+0000 7f2aa204e640 1 -- 172.21.15.67:0/3408920175 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5630fa6e89a0 msgr2=0x5630fa6bd010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:04.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.813+0000 7f2aa204e640 1 --2- 172.21.15.67:0/3408920175 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5630fa6e89a0 0x5630fa6bd010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.813+0000 7f2aa204e640 1 -- 172.21.15.67:0/3408920175 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5630fa4651e0 con 0x5630fa6c3dd0 2024-03-24T00:44:04.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.813+0000 7f2aa204e640 1 --2- 172.21.15.67:0/3408920175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5630fa6c3dd0 0x5630fa760960 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f2a98034040 tx=0x7f2a98004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e57ef24b4cb22838 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:04.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.813+0000 7f2a7d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:44:04.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.813+0000 7f2a7d7fa640 1 -- 172.21.15.67:0/3408920175 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a98060060 con 0x5630fa6c3dd0 2024-03-24T00:44:04.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.813+0000 7f2a7d7fa640 1 -- 172.21.15.67:0/3408920175 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f2a98031c50 con 0x5630fa6c3dd0 2024-03-24T00:44:04.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.813+0000 7f2aa6f7e740 1 -- 172.21.15.67:0/3408920175 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2a94061830 con 0x5630fa6c3dd0 2024-03-24T00:44:04.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.813+0000 7f2a7d7fa640 1 -- 172.21.15.67:0/3408920175 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2a98066050 con 0x5630fa6c3dd0 2024-03-24T00:44:04.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.813+0000 7f2aa6f7e740 10 client.7350 fetch_fsmap learned FSMap version 47 2024-03-24T00:44:04.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.813+0000 7f2aa6f7e740 1 -- 172.21.15.67:0/3408920175 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=2+}) v3 -- 0x5630fa2d54a0 con 0x5630fa6c3dd0 2024-03-24T00:44:04.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.813+0000 7f2a7d7fa640 1 -- 172.21.15.67:0/3408920175 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a94061830 con 0x5630fa6c3dd0 2024-03-24T00:44:04.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.813+0000 7f2a7d7fa640 1 -- 172.21.15.67:0/3408920175 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 47) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f2a98069020 con 0x5630fa6c3dd0 2024-03-24T00:44:04.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.813+0000 7f2aa6f7e740 10 client.7350 fetch_fsmap finished waiting for FSMap version 47 2024-03-24T00:44:04.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.813+0000 7f2aa6f7e740 10 client.7350 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.c 2024-03-24T00:44:04.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.813+0000 7f2aa6f7e740 20 client.7350 populate_metadata read hostname 'smithi067' 2024-03-24T00:44:04.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.813+0000 7f2aa6f7e740 1 --2- 172.21.15.67:0/3408920175 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x5630fa771e40 0x5630fa7785f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.813+0000 7f2aa6f7e740 4 client.7350 mds_command: new command op to 7149 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/subdir", "scrubops": ["repair", "recursive"]}] 2024-03-24T00:44:04.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.813+0000 7f2aa6f7e740 1 -- 172.21.15.67:0/3408920175 --> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] -- command(tid 0: {"prefix": "scrub start", "path": "/subdir", "scrubops": ["repair", "recursive"]}) v1 -- 0x5630fa635780 con 0x5630fa771e40 2024-03-24T00:44:04.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.814+0000 7f2aa184d640 1 --2- 172.21.15.67:0/3408920175 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x5630fa771e40 0x5630fa7785f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.814+0000 7f2a7d7fa640 10 client.7350 ms_handle_connect on v2:172.21.15.67:6832/720256752 2024-03-24T00:44:04.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.814+0000 7f2aa184d640 1 --2- 172.21.15.67:0/3408920175 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x5630fa771e40 0x5630fa7785f0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17c29bbbf3f2925e server_cookie=fc21d5d4b237a355 in_seq=0 out_seq=0 2024-03-24T00:44:04.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.814+0000 7f2a7d7fa640 1 -- 172.21.15.67:0/3408920175 <== mds.0 v2:172.21.15.67:6832/720256752 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5630fa635780 con 0x5630fa771e40 2024-03-24T00:44:04.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.814+0000 7f2a7d7fa640 10 client.7350 handle_command_reply: tid=0 2024-03-24T00:44:04.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.814+0000 7f2aa6f7e740 1 client.7350 shutdown 2024-03-24T00:44:04.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.815+0000 7f2aa6f7e740 1 -- 172.21.15.67:0/3408920175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5630fa6c3dd0 msgr2=0x5630fa760960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:04.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.815+0000 7f2aa6f7e740 1 --2- 172.21.15.67:0/3408920175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5630fa6c3dd0 0x5630fa760960 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f2a98034040 tx=0x7f2a98004730 comp rx=0 tx=0).stop 2024-03-24T00:44:04.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.815+0000 7f2aa6f7e740 1 -- 172.21.15.67:0/3408920175 shutdown_connections 2024-03-24T00:44:04.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.815+0000 7f2aa6f7e740 1 --2- 172.21.15.67:0/3408920175 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x5630fa771e40 0x5630fa7785f0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.815+0000 7f2aa6f7e740 1 --2- 172.21.15.67:0/3408920175 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5630fa6e89a0 0x5630fa6bd010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.815+0000 7f2aa6f7e740 1 --2- 172.21.15.67:0/3408920175 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5630fa6c41b0 0x5630fa6bcad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.815+0000 7f2a7d7fa640 0 client.7350 ms_handle_reset on v2:172.21.15.67:6832/720256752 2024-03-24T00:44:04.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.815+0000 7f2aa6f7e740 1 --2- 172.21.15.67:0/3408920175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5630fa6c3dd0 0x5630fa760960 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.815+0000 7f2aa6f7e740 1 -- 172.21.15.67:0/3408920175 >> 172.21.15.67:0/3408920175 conn(0x5630fa68ae00 msgr2=0x5630fa6bdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:04.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.815+0000 7f2aa6f7e740 1 -- 172.21.15.67:0/3408920175 shutdown_connections 2024-03-24T00:44:04.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.815+0000 7f2aa6f7e740 1 -- 172.21.15.67:0/3408920175 wait complete. 2024-03-24T00:44:04.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.815+0000 7f2aa6f7e740 20 client.7350 trim_cache size 0 max 16384 2024-03-24T00:44:04.820 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:44:04.820 INFO:teuthology.orchestra.run.smithi067.stdout: "return_code": 0, 2024-03-24T00:44:04.820 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_tag": "1af79934-4dae-42ec-a8c5-e84e101232cf", 2024-03-24T00:44:04.820 INFO:teuthology.orchestra.run.smithi067.stdout: "mode": "asynchronous" 2024-03-24T00:44:04.821 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:44:04.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.815+0000 7f2aa42b2640 1 -- 172.21.15.67:0/4233936449 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2a8006bbd0 msgr2=0x7f2a8006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:04.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.815+0000 7f2aa42b2640 1 --2- 172.21.15.67:0/4233936449 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2a8006bbd0 0x7f2a8006e090 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f2a98036000 tx=0x7f2a9803c040 comp rx=0 tx=0).stop 2024-03-24T00:44:04.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.815+0000 7f2aa42b2640 1 -- 172.21.15.67:0/4233936449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a9c0b5fc0 msgr2=0x7f2a9c0cb060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:04.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.815+0000 7f2aa42b2640 1 --2- 172.21.15.67:0/4233936449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a9c0b5fc0 0x7f2a9c0cb060 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f2a9400ce80 tx=0x7f2a9400a710 comp rx=0 tx=0).stop 2024-03-24T00:44:04.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.815+0000 7f2aa42b2640 1 -- 172.21.15.67:0/4233936449 shutdown_connections 2024-03-24T00:44:04.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.815+0000 7f2aa42b2640 1 --2- 172.21.15.67:0/4233936449 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2a8006bbd0 0x7f2a8006e090 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.815+0000 7f2aa42b2640 1 --2- 172.21.15.67:0/4233936449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a9c0b5fc0 0x7f2a9c0cb060 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.815+0000 7f2aa42b2640 1 --2- 172.21.15.67:0/4233936449 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a9c15dfc0 0x7f2a9c0b5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.815+0000 7f2aa42b2640 1 --2- 172.21.15.67:0/4233936449 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2a9c0d0d50 0x7f2a9c0ccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.815+0000 7f2aa42b2640 1 -- 172.21.15.67:0/4233936449 >> 172.21.15.67:0/4233936449 conn(0x7f2a9c06c880 msgr2=0x7f2a9c0b71f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:04.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.816+0000 7f2aa42b2640 1 -- 172.21.15.67:0/4233936449 shutdown_connections 2024-03-24T00:44:04.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.816+0000 7f2aa42b2640 1 -- 172.21.15.67:0/4233936449 wait complete. 2024-03-24T00:44:04.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:44:04.826 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.5:0 scrub status 2024-03-24T00:44:04.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.930+0000 7fd53ea62640 1 Processor -- start 2024-03-24T00:44:04.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.930+0000 7fd53ea62640 1 -- start start 2024-03-24T00:44:04.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.931+0000 7fd53ea62640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd538072380 0x7fd538072810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.931+0000 7fd53ea62640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd5380cd7a0 0x7fd5380cdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.931+0000 7fd53ea62640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5380b2b40 con 0x7fd53815a500 2024-03-24T00:44:04.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.931+0000 7fd53ea62640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5380b2e20 con 0x7fd5380cd7a0 2024-03-24T00:44:04.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.931+0000 7fd53ea62640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5380b3100 con 0x7fd538072380 2024-03-24T00:44:04.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.931+0000 7fd53cfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd5380cd7a0 0x7fd5380cdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.931+0000 7fd537fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd53815a500 0x7fd53806f030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46880/0 (socket says 172.21.15.67:46880) 2024-03-24T00:44:04.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.931+0000 7fd537fff640 1 -- 172.21.15.67:0/1164583415 learned_addr learned my addr 172.21.15.67:0/1164583415 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:44:04.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.931+0000 7fd5377fe640 1 --2- 172.21.15.67:0/1164583415 >> v2:172.21.15.72:3300/0 conn(0x7fd538072380 0x7fd538072810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.931+0000 7fd53cfff640 1 -- 172.21.15.67:0/1164583415 >> v2:172.21.15.72:3300/0 conn(0x7fd538072380 msgr2=0x7fd538072810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:04.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.931+0000 7fd53cfff640 1 --2- 172.21.15.67:0/1164583415 >> v2:172.21.15.72:3300/0 conn(0x7fd538072380 0x7fd538072810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.931+0000 7fd53cfff640 1 -- 172.21.15.67:0/1164583415 >> v1:172.21.15.67:6789/0 conn(0x7fd53815a500 legacy=0x7fd53806f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:04.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.931+0000 7fd53cfff640 1 -- 172.21.15.67:0/1164583415 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5380b33e0 con 0x7fd5380cd7a0 2024-03-24T00:44:04.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.931+0000 7fd536ffd640 1 -- 172.21.15.67:0/1164583415 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3544683224 0 0) 0x7fd5380b2b40 con 0x7fd53815a500 2024-03-24T00:44:04.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.931+0000 7fd5377fe640 1 --2- 172.21.15.67:0/1164583415 >> v2:172.21.15.72:3300/0 conn(0x7fd538072380 0x7fd538072810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:44:04.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.932+0000 7fd53cfff640 1 --2- 172.21.15.67:0/1164583415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5380cd7a0 0x7fd5380cdbb0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fd538071500 tx=0x7fd53002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8141364bbbdb854 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:04.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.932+0000 7fd536ffd640 1 -- 172.21.15.67:0/1164583415 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd530039070 con 0x7fd5380cd7a0 2024-03-24T00:44:04.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.932+0000 7fd536ffd640 1 -- 172.21.15.67:0/1164583415 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd53002cbf0 con 0x7fd5380cd7a0 2024-03-24T00:44:04.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.932+0000 7fd536ffd640 1 -- 172.21.15.67:0/1164583415 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd530034440 con 0x7fd5380cd7a0 2024-03-24T00:44:04.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.933+0000 7fd53ea62640 1 -- 172.21.15.67:0/1164583415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5380cd7a0 msgr2=0x7fd5380cdbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:04.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.933+0000 7fd53ea62640 1 --2- 172.21.15.67:0/1164583415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5380cd7a0 0x7fd5380cdbb0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fd538071500 tx=0x7fd53002be40 comp rx=0 tx=0).stop 2024-03-24T00:44:04.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.933+0000 7fd53ea62640 1 -- 172.21.15.67:0/1164583415 shutdown_connections 2024-03-24T00:44:04.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.933+0000 7fd53ea62640 1 --2- 172.21.15.67:0/1164583415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5380cd7a0 0x7fd5380cdbb0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.933+0000 7fd53ea62640 1 --2- 172.21.15.67:0/1164583415 >> v2:172.21.15.72:3300/0 conn(0x7fd538072380 0x7fd538072810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.933+0000 7fd53ea62640 1 -- 172.21.15.67:0/1164583415 >> 172.21.15.67:0/1164583415 conn(0x7fd5380cf9e0 msgr2=0x7fd5380cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:04.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.933+0000 7fd53ea62640 1 -- 172.21.15.67:0/1164583415 shutdown_connections 2024-03-24T00:44:04.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.933+0000 7fd53ea62640 1 -- 172.21.15.67:0/1164583415 wait complete. 2024-03-24T00:44:04.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.935+0000 7fd53ea62640 1 Processor -- start 2024-03-24T00:44:04.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.935+0000 7fd53ea62640 1 -- start start 2024-03-24T00:44:04.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.936+0000 7fd53ea62640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd538072380 0x7fd53813e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.936+0000 7fd53ea62640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd5380cd7a0 0x7fd5381f4c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.936+0000 7fd53ea62640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd53815a500 0x7fd5380af640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.936+0000 7fd53ea62640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd53806f740 con 0x7fd538072380 2024-03-24T00:44:04.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.936+0000 7fd53ea62640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd53815a8e0 con 0x7fd53815a500 2024-03-24T00:44:04.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.936+0000 7fd53ea62640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5380d06f0 con 0x7fd5380cd7a0 2024-03-24T00:44:04.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.936+0000 7fd537fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd538072380 0x7fd53813e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.936+0000 7fd537fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd538072380 0x7fd53813e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51136/0 (socket says 172.21.15.67:51136) 2024-03-24T00:44:04.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.936+0000 7fd537fff640 1 -- 172.21.15.67:0/6134188 learned_addr learned my addr 172.21.15.67:0/6134188 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:04.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.936+0000 7fd53cfff640 1 --2- 172.21.15.67:0/6134188 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd53815a500 0x7fd5380af640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.936+0000 7fd5377fe640 1 --2- 172.21.15.67:0/6134188 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd5380cd7a0 0x7fd5381f4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.936+0000 7fd537fff640 1 -- 172.21.15.67:0/6134188 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd5380cd7a0 msgr2=0x7fd5381f4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:04.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.936+0000 7fd537fff640 1 --2- 172.21.15.67:0/6134188 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd5380cd7a0 0x7fd5381f4c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.936+0000 7fd537fff640 1 -- 172.21.15.67:0/6134188 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd53815a500 msgr2=0x7fd5380af640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:04.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.936+0000 7fd537fff640 1 --2- 172.21.15.67:0/6134188 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd53815a500 0x7fd5380af640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.937+0000 7fd537fff640 1 -- 172.21.15.67:0/6134188 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5380b0100 con 0x7fd538072380 2024-03-24T00:44:04.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.937+0000 7fd53cfff640 1 --2- 172.21.15.67:0/6134188 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd53815a500 0x7fd5380af640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:44:04.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.937+0000 7fd537fff640 1 --2- 172.21.15.67:0/6134188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd538072380 0x7fd53813e780 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fd52800c430 tx=0x7fd52800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:04.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.937+0000 7fd534ff9640 1 -- 172.21.15.67:0/6134188 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5280074e0 con 0x7fd538072380 2024-03-24T00:44:04.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.937+0000 7fd534ff9640 1 -- 172.21.15.67:0/6134188 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5280047e0 con 0x7fd538072380 2024-03-24T00:44:04.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.937+0000 7fd53ea62640 1 -- 172.21.15.67:0/6134188 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5380b03e0 con 0x7fd538072380 2024-03-24T00:44:04.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.937+0000 7fd534ff9640 1 -- 172.21.15.67:0/6134188 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd528004bd0 con 0x7fd538072380 2024-03-24T00:44:04.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.937+0000 7fd53ea62640 1 -- 172.21.15.67:0/6134188 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5381f4760 con 0x7fd538072380 2024-03-24T00:44:04.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.938+0000 7fd534ff9640 1 -- 172.21.15.67:0/6134188 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd528011070 con 0x7fd538072380 2024-03-24T00:44:04.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.939+0000 7fd534ff9640 1 --2- 172.21.15.67:0/6134188 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd52406bbd0 0x7fd52406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.939+0000 7fd534ff9640 1 -- 172.21.15.67:0/6134188 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fd52808a940 con 0x7fd538072380 2024-03-24T00:44:04.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.939+0000 7fd5377fe640 1 --2- 172.21.15.67:0/6134188 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd52406bbd0 0x7fd52406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.939+0000 7fd5377fe640 1 --2- 172.21.15.67:0/6134188 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd52406bbd0 0x7fd52406e090 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fd538159860 tx=0x7fd52c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:04.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.940+0000 7fd541645740 1 Processor -- start 2024-03-24T00:44:04.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.940+0000 7fd541645740 1 -- start start 2024-03-24T00:44:04.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.940+0000 7fd541645740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f6bb80fe50 0x55f6bb7e2df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.940+0000 7fd541645740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f6bb810230 0x55f6bb812be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.940+0000 7fd541645740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f6bb7e3330 0x55f6bb813120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.940+0000 7fd541645740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6bb74d3d0 con 0x55f6bb810230 2024-03-24T00:44:04.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.940+0000 7fd541645740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6bb6f5ad0 con 0x55f6bb80fe50 2024-03-24T00:44:04.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.940+0000 7fd541645740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f6bb749470 con 0x55f6bb7e3330 2024-03-24T00:44:04.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.941+0000 7fd5377fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f6bb810230 0x55f6bb812be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.941+0000 7fd5377fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f6bb810230 0x55f6bb812be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51144/0 (socket says 172.21.15.67:51144) 2024-03-24T00:44:04.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.941+0000 7fd5377fe640 1 -- 172.21.15.67:0/4009650480 learned_addr learned my addr 172.21.15.67:0/4009650480 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:04.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.941+0000 7fd53cfff640 1 --2- 172.21.15.67:0/4009650480 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f6bb7e3330 0x55f6bb813120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.941+0000 7fd537fff640 1 --2- 172.21.15.67:0/4009650480 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f6bb80fe50 0x55f6bb7e2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.941+0000 7fd5377fe640 1 -- 172.21.15.67:0/4009650480 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f6bb7e3330 msgr2=0x55f6bb813120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:04.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.941+0000 7fd5377fe640 1 --2- 172.21.15.67:0/4009650480 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f6bb7e3330 0x55f6bb813120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.941+0000 7fd5377fe640 1 -- 172.21.15.67:0/4009650480 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f6bb80fe50 msgr2=0x55f6bb7e2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:04.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.941+0000 7fd5377fe640 1 --2- 172.21.15.67:0/4009650480 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f6bb80fe50 0x55f6bb7e2df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.941+0000 7fd5377fe640 1 -- 172.21.15.67:0/4009650480 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f6bb52b7d0 con 0x55f6bb810230 2024-03-24T00:44:04.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.941+0000 7fd5377fe640 1 --2- 172.21.15.67:0/4009650480 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f6bb810230 0x55f6bb812be0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7fd52c010920 tx=0x7fd52c013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f30cef9e605f2ac3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:04.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.941+0000 7fd51dffb640 1 -- 172.21.15.67:0/4009650480 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd52c01c050 con 0x55f6bb810230 2024-03-24T00:44:04.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.941+0000 7fd51dffb640 1 -- 172.21.15.67:0/4009650480 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd52c01d040 con 0x55f6bb810230 2024-03-24T00:44:04.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.941+0000 7fd51dffb640 1 -- 172.21.15.67:0/4009650480 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd52c00f040 con 0x55f6bb810230 2024-03-24T00:44:04.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.942+0000 7fd541645740 1 -- 172.21.15.67:0/4009650480 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f6bb810230 msgr2=0x55f6bb812be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:04.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.942+0000 7fd541645740 1 --2- 172.21.15.67:0/4009650480 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f6bb810230 0x55f6bb812be0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7fd52c010920 tx=0x7fd52c013000 comp rx=0 tx=0).stop 2024-03-24T00:44:04.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.942+0000 7fd541645740 1 -- 172.21.15.67:0/4009650480 shutdown_connections 2024-03-24T00:44:04.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.942+0000 7fd541645740 1 --2- 172.21.15.67:0/4009650480 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f6bb7e3330 0x55f6bb813120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.942+0000 7fd541645740 1 --2- 172.21.15.67:0/4009650480 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f6bb810230 0x55f6bb812be0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.942+0000 7fd541645740 1 --2- 172.21.15.67:0/4009650480 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f6bb80fe50 0x55f6bb7e2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.942+0000 7fd541645740 1 -- 172.21.15.67:0/4009650480 >> 172.21.15.67:0/4009650480 conn(0x55f6bb7e5060 msgr2=0x55f6bb82dbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:04.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.942+0000 7fd541645740 1 -- 172.21.15.67:0/4009650480 shutdown_connections 2024-03-24T00:44:04.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.942+0000 7fd541645740 1 -- 172.21.15.67:0/4009650480 wait complete. 2024-03-24T00:44:04.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.942+0000 7fd541645740 1 -- start start 2024-03-24T00:44:04.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.943+0000 7fd541645740 1 Processor -- start 2024-03-24T00:44:04.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.944+0000 7fd541645740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f6bb80fe50 0x55f6bb83e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.944+0000 7fd541645740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f6bb810230 0x55f6bb816590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.944+0000 7fd541645740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f6bb80b960 0x55f6bb807be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.944+0000 7fd541645740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6bb7e7700 con 0x55f6bb810230 2024-03-24T00:44:04.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.944+0000 7fd541645740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6bb709060 con 0x55f6bb80b960 2024-03-24T00:44:04.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.944+0000 7fd541645740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f6bb7e9f40 con 0x55f6bb80fe50 2024-03-24T00:44:04.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.944+0000 7fd541645740 10 client.0 _ll_register_callbacks cb 0x55f6bb6eb6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:44:04.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.944+0000 7fd5377fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f6bb810230 0x55f6bb816590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.944+0000 7fd5377fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f6bb810230 0x55f6bb816590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51154/0 (socket says 172.21.15.67:51154) 2024-03-24T00:44:04.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.944+0000 7fd5377fe640 1 -- 172.21.15.67:0/1160999196 learned_addr learned my addr 172.21.15.67:0/1160999196 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:04.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.944+0000 7fd537fff640 1 --2- 172.21.15.67:0/1160999196 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f6bb80fe50 0x55f6bb83e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.944+0000 7fd53cfff640 1 --2- 172.21.15.67:0/1160999196 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f6bb80b960 0x55f6bb807be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.944+0000 7fd5377fe640 1 -- 172.21.15.67:0/1160999196 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f6bb80fe50 msgr2=0x55f6bb83e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:04.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.944+0000 7fd5377fe640 1 --2- 172.21.15.67:0/1160999196 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f6bb80fe50 0x55f6bb83e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.944+0000 7fd5377fe640 1 -- 172.21.15.67:0/1160999196 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f6bb80b960 msgr2=0x55f6bb807be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:04.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.944+0000 7fd5377fe640 1 --2- 172.21.15.67:0/1160999196 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f6bb80b960 0x55f6bb807be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.944+0000 7fd5377fe640 1 -- 172.21.15.67:0/1160999196 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f6bb4613f0 con 0x55f6bb810230 2024-03-24T00:44:04.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.944+0000 7fd53cfff640 1 --2- 172.21.15.67:0/1160999196 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f6bb80b960 0x55f6bb807be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:44:04.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.945+0000 7fd5377fe640 1 --2- 172.21.15.67:0/1160999196 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f6bb810230 0x55f6bb816590 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fd52c0108f0 tx=0x7fd52c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a93cb88ffe01c469 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:04.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.945+0000 7fd516ffd640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:44:04.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.945+0000 7fd516ffd640 1 -- 172.21.15.67:0/1160999196 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd52c02e030 con 0x55f6bb810230 2024-03-24T00:44:04.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.945+0000 7fd516ffd640 1 -- 172.21.15.67:0/1160999196 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fd52c026070 con 0x55f6bb810230 2024-03-24T00:44:04.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.945+0000 7fd541645740 1 -- 172.21.15.67:0/1160999196 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd52c01d040 con 0x55f6bb810230 2024-03-24T00:44:04.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.945+0000 7fd516ffd640 1 -- 172.21.15.67:0/1160999196 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd52c01c030 con 0x55f6bb810230 2024-03-24T00:44:04.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.945+0000 7fd541645740 10 client.7365 fetch_fsmap learned FSMap version 47 2024-03-24T00:44:04.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.945+0000 7fd541645740 1 -- 172.21.15.67:0/1160999196 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=2+}) v3 -- 0x55f6bb4210f0 con 0x55f6bb810230 2024-03-24T00:44:04.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.945+0000 7fd516ffd640 1 -- 172.21.15.67:0/1160999196 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd52c01d040 con 0x55f6bb810230 2024-03-24T00:44:04.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.945+0000 7fd516ffd640 1 -- 172.21.15.67:0/1160999196 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 47) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fd52c004260 con 0x55f6bb810230 2024-03-24T00:44:04.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.945+0000 7fd541645740 10 client.7365 fetch_fsmap finished waiting for FSMap version 47 2024-03-24T00:44:04.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.945+0000 7fd541645740 10 client.7365 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.c 2024-03-24T00:44:04.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.945+0000 7fd541645740 20 client.7365 populate_metadata read hostname 'smithi067' 2024-03-24T00:44:04.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.945+0000 7fd541645740 1 --2- 172.21.15.67:0/1160999196 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x55f6bb811190 0x55f6bb8115a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.945+0000 7fd541645740 4 client.7365 mds_command: new command op to 7149 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:44:04.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.945+0000 7fd541645740 1 -- 172.21.15.67:0/1160999196 --> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f6bb75a910 con 0x55f6bb811190 2024-03-24T00:44:04.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.945+0000 7fd537fff640 1 --2- 172.21.15.67:0/1160999196 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x55f6bb811190 0x55f6bb8115a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.946+0000 7fd516ffd640 10 client.7365 ms_handle_connect on v2:172.21.15.67:6832/720256752 2024-03-24T00:44:04.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.946+0000 7fd537fff640 1 --2- 172.21.15.67:0/1160999196 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x55f6bb811190 0x55f6bb8115a0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dbf8c8bee4a73656 server_cookie=4211c1722f208b52 in_seq=0 out_seq=0 2024-03-24T00:44:04.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.947+0000 7fd516ffd640 1 -- 172.21.15.67:0/1160999196 <== mds.0 v2:172.21.15.67:6832/720256752 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f6bb75a910 con 0x55f6bb811190 2024-03-24T00:44:04.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.947+0000 7fd516ffd640 10 client.7365 handle_command_reply: tid=0 2024-03-24T00:44:04.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.947+0000 7fd541645740 1 client.7365 shutdown 2024-03-24T00:44:04.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.947+0000 7fd541645740 1 -- 172.21.15.67:0/1160999196 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f6bb810230 msgr2=0x55f6bb816590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:04.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.947+0000 7fd541645740 1 --2- 172.21.15.67:0/1160999196 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f6bb810230 0x55f6bb816590 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fd52c0108f0 tx=0x7fd52c004730 comp rx=0 tx=0).stop 2024-03-24T00:44:04.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.947+0000 7fd541645740 1 -- 172.21.15.67:0/1160999196 shutdown_connections 2024-03-24T00:44:04.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.947+0000 7fd541645740 1 --2- 172.21.15.67:0/1160999196 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x55f6bb811190 0x55f6bb8115a0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.947+0000 7fd541645740 1 --2- 172.21.15.67:0/1160999196 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f6bb80b960 0x55f6bb807be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.947+0000 7fd541645740 1 --2- 172.21.15.67:0/1160999196 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f6bb810230 0x55f6bb816590 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.947+0000 7fd516ffd640 0 client.7365 ms_handle_reset on v2:172.21.15.67:6832/720256752 2024-03-24T00:44:04.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.947+0000 7fd541645740 1 --2- 172.21.15.67:0/1160999196 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f6bb80fe50 0x55f6bb83e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.947+0000 7fd541645740 1 -- 172.21.15.67:0/1160999196 >> 172.21.15.67:0/1160999196 conn(0x55f6bb7e3330 msgr2=0x55f6bb831060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:04.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.947+0000 7fd541645740 1 -- 172.21.15.67:0/1160999196 shutdown_connections 2024-03-24T00:44:04.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.947+0000 7fd541645740 1 -- 172.21.15.67:0/1160999196 wait complete. 2024-03-24T00:44:04.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.947+0000 7fd541645740 20 client.7365 trim_cache size 0 max 16384 2024-03-24T00:44:04.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.963+0000 7fd541645740 1 Processor -- start 2024-03-24T00:44:04.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.963+0000 7fd541645740 1 -- start start 2024-03-24T00:44:04.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.964+0000 7fd541645740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f6bb80b960 0x55f6bb83e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.964+0000 7fd541645740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f6bb80fe50 0x55f6bb83db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.964+0000 7fd541645740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f6bb810230 0x55f6bb806710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.964+0000 7fd541645740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6bb7e3bc0 con 0x55f6bb80b960 2024-03-24T00:44:04.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.964+0000 7fd541645740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd538200980 con 0x55f6bb810230 2024-03-24T00:44:04.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.964+0000 7fd541645740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5381ffbc0 con 0x55f6bb80fe50 2024-03-24T00:44:04.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.964+0000 7fd537fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f6bb80b960 0x55f6bb83e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.964+0000 7fd537fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f6bb80b960 0x55f6bb83e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51160/0 (socket says 172.21.15.67:51160) 2024-03-24T00:44:04.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.964+0000 7fd537fff640 1 -- 172.21.15.67:0/2207876478 learned_addr learned my addr 172.21.15.67:0/2207876478 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:04.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.964+0000 7fd537fff640 1 -- 172.21.15.67:0/2207876478 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f6bb80fe50 msgr2=0x55f6bb83db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:04.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.964+0000 7fd537fff640 1 --2- 172.21.15.67:0/2207876478 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f6bb80fe50 0x55f6bb83db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.964+0000 7fd537fff640 1 -- 172.21.15.67:0/2207876478 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f6bb810230 msgr2=0x55f6bb806710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:04.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.964+0000 7fd537fff640 1 --2- 172.21.15.67:0/2207876478 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f6bb810230 0x55f6bb806710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.964+0000 7fd537fff640 1 -- 172.21.15.67:0/2207876478 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f6bb80b100 con 0x55f6bb80b960 2024-03-24T00:44:04.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.964+0000 7fd537fff640 1 --2- 172.21.15.67:0/2207876478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f6bb80b960 0x55f6bb83e810 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fd528088c90 tx=0x7fd528000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c48ed999eaa7ccfa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:04.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.964+0000 7fd51d7fa640 1 -- 172.21.15.67:0/2207876478 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd528061260 con 0x55f6bb80b960 2024-03-24T00:44:04.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.964+0000 7fd51d7fa640 1 -- 172.21.15.67:0/2207876478 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd528056bf0 con 0x55f6bb80b960 2024-03-24T00:44:04.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.964+0000 7fd51d7fa640 1 -- 172.21.15.67:0/2207876478 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd528004d70 con 0x55f6bb80b960 2024-03-24T00:44:04.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.965+0000 7fd541645740 1 -- 172.21.15.67:0/2207876478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f6bb80b960 msgr2=0x55f6bb83e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:04.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.965+0000 7fd541645740 1 --2- 172.21.15.67:0/2207876478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f6bb80b960 0x55f6bb83e810 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fd528088c90 tx=0x7fd528000ed0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.965+0000 7fd541645740 1 -- 172.21.15.67:0/2207876478 shutdown_connections 2024-03-24T00:44:04.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.965+0000 7fd541645740 1 --2- 172.21.15.67:0/2207876478 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f6bb810230 0x55f6bb806710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.965+0000 7fd541645740 1 --2- 172.21.15.67:0/2207876478 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f6bb80fe50 0x55f6bb83db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.965+0000 7fd541645740 1 --2- 172.21.15.67:0/2207876478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f6bb80b960 0x55f6bb83e810 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.965+0000 7fd541645740 1 -- 172.21.15.67:0/2207876478 >> 172.21.15.67:0/2207876478 conn(0x55f6bb7e3330 msgr2=0x55f6bb7e9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:04.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.965+0000 7fd541645740 1 -- 172.21.15.67:0/2207876478 shutdown_connections 2024-03-24T00:44:04.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.965+0000 7fd541645740 1 -- 172.21.15.67:0/2207876478 wait complete. 2024-03-24T00:44:04.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.966+0000 7fd541645740 1 -- start start 2024-03-24T00:44:04.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.966+0000 7fd541645740 1 Processor -- start 2024-03-24T00:44:04.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.967+0000 7fd541645740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f6bb80b960 0x55f6bb89c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.967+0000 7fd541645740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f6bb80fe50 0x55f6bb810650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.967+0000 7fd541645740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f6bb810230 0x55f6bb7e1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.967+0000 7fd541645740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd53815d5f0 con 0x55f6bb810230 2024-03-24T00:44:04.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.967+0000 7fd541645740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd538156da0 con 0x55f6bb80fe50 2024-03-24T00:44:04.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.967+0000 7fd541645740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f6bb83e500 con 0x55f6bb80b960 2024-03-24T00:44:04.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.967+0000 7fd541645740 10 client.0 _ll_register_callbacks cb 0x7fd528007720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:44:04.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.967+0000 7fd53cfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f6bb810230 0x55f6bb7e1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.967+0000 7fd53cfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f6bb810230 0x55f6bb7e1ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51174/0 (socket says 172.21.15.67:51174) 2024-03-24T00:44:04.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.967+0000 7fd53cfff640 1 -- 172.21.15.67:0/4230482591 learned_addr learned my addr 172.21.15.67:0/4230482591 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:04.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.967+0000 7fd537fff640 1 --2- 172.21.15.67:0/4230482591 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f6bb80b960 0x55f6bb89c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.967+0000 7fd5377fe640 1 --2- 172.21.15.67:0/4230482591 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f6bb80fe50 0x55f6bb810650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.967+0000 7fd53cfff640 1 -- 172.21.15.67:0/4230482591 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f6bb80b960 msgr2=0x55f6bb89c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:04.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.967+0000 7fd53cfff640 1 --2- 172.21.15.67:0/4230482591 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f6bb80b960 0x55f6bb89c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.967+0000 7fd53cfff640 1 -- 172.21.15.67:0/4230482591 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f6bb80fe50 msgr2=0x55f6bb810650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:04.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.967+0000 7fd53cfff640 1 --2- 172.21.15.67:0/4230482591 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f6bb80fe50 0x55f6bb810650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.967+0000 7fd53cfff640 1 -- 172.21.15.67:0/4230482591 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f6bb83ea30 con 0x55f6bb810230 2024-03-24T00:44:04.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.967+0000 7fd53cfff640 1 --2- 172.21.15.67:0/4230482591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f6bb810230 0x55f6bb7e1ca0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fd53002c450 tx=0x7fd530004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99378daab78fdd23 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:04.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.967+0000 7fd516ffd640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:44:04.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.968+0000 7fd516ffd640 1 -- 172.21.15.67:0/4230482591 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd530045070 con 0x55f6bb810230 2024-03-24T00:44:04.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.968+0000 7fd516ffd640 1 -- 172.21.15.67:0/4230482591 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fd530041850 con 0x55f6bb810230 2024-03-24T00:44:04.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.968+0000 7fd541645740 1 -- 172.21.15.67:0/4230482591 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd528056bf0 con 0x55f6bb810230 2024-03-24T00:44:04.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.968+0000 7fd516ffd640 1 -- 172.21.15.67:0/4230482591 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd530052050 con 0x55f6bb810230 2024-03-24T00:44:04.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.968+0000 7fd541645740 10 client.7371 fetch_fsmap learned FSMap version 47 2024-03-24T00:44:04.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.968+0000 7fd541645740 1 -- 172.21.15.67:0/4230482591 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=2+}) v3 -- 0x55f6bb80eab0 con 0x55f6bb810230 2024-03-24T00:44:04.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.968+0000 7fd516ffd640 1 -- 172.21.15.67:0/4230482591 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd528056bf0 con 0x55f6bb810230 2024-03-24T00:44:04.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.968+0000 7fd516ffd640 1 -- 172.21.15.67:0/4230482591 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 47) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fd530055020 con 0x55f6bb810230 2024-03-24T00:44:04.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.968+0000 7fd541645740 10 client.7371 fetch_fsmap finished waiting for FSMap version 47 2024-03-24T00:44:04.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.968+0000 7fd541645740 10 client.7371 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.c 2024-03-24T00:44:04.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.968+0000 7fd541645740 20 client.7371 populate_metadata read hostname 'smithi067' 2024-03-24T00:44:04.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.968+0000 7fd541645740 1 --2- 172.21.15.67:0/4230482591 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x55f6bb7e21e0 0x55f6bb80de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:04.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.968+0000 7fd541645740 4 client.7371 mds_command: new command op to 7149 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:44:04.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.968+0000 7fd541645740 1 -- 172.21.15.67:0/4230482591 --> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55f6bb754c60 con 0x55f6bb7e21e0 2024-03-24T00:44:04.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.968+0000 7fd537fff640 1 --2- 172.21.15.67:0/4230482591 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x55f6bb7e21e0 0x55f6bb80de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:04.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.968+0000 7fd537fff640 1 --2- 172.21.15.67:0/4230482591 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x55f6bb7e21e0 0x55f6bb80de50 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4fabc314ec7b7b64 server_cookie=78b5412ea4c1a8a6 in_seq=0 out_seq=0 2024-03-24T00:44:04.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.968+0000 7fd516ffd640 10 client.7371 ms_handle_connect on v2:172.21.15.67:6832/720256752 2024-03-24T00:44:04.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.969+0000 7fd516ffd640 1 -- 172.21.15.67:0/4230482591 <== mds.0 v2:172.21.15.67:6832/720256752 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+266 (crc 0 0 0) 0x55f6bb754c60 con 0x55f6bb7e21e0 2024-03-24T00:44:04.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.969+0000 7fd516ffd640 10 client.7371 handle_command_reply: tid=0 2024-03-24T00:44:04.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.969+0000 7fd541645740 1 client.7371 shutdown 2024-03-24T00:44:04.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.969+0000 7fd541645740 1 -- 172.21.15.67:0/4230482591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f6bb810230 msgr2=0x55f6bb7e1ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:04.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.969+0000 7fd541645740 1 --2- 172.21.15.67:0/4230482591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f6bb810230 0x55f6bb7e1ca0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fd53002c450 tx=0x7fd530004a40 comp rx=0 tx=0).stop 2024-03-24T00:44:04.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.969+0000 7fd541645740 1 -- 172.21.15.67:0/4230482591 shutdown_connections 2024-03-24T00:44:04.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.969+0000 7fd541645740 1 --2- 172.21.15.67:0/4230482591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f6bb810230 0x55f6bb7e1ca0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.969+0000 7fd541645740 1 --2- 172.21.15.67:0/4230482591 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x55f6bb7e21e0 0x55f6bb80de50 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.969+0000 7fd541645740 1 --2- 172.21.15.67:0/4230482591 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f6bb80fe50 0x55f6bb810650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.969+0000 7fd541645740 1 --2- 172.21.15.67:0/4230482591 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f6bb80b960 0x55f6bb89c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.969+0000 7fd516ffd640 0 client.7371 ms_handle_reset on v2:172.21.15.67:6832/720256752 2024-03-24T00:44:04.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.969+0000 7fd541645740 1 -- 172.21.15.67:0/4230482591 >> 172.21.15.67:0/4230482591 conn(0x55f6bb7e3330 msgr2=0x55f6bb82e150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:04.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.969+0000 7fd541645740 1 -- 172.21.15.67:0/4230482591 shutdown_connections 2024-03-24T00:44:04.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.969+0000 7fd541645740 1 -- 172.21.15.67:0/4230482591 wait complete. 2024-03-24T00:44:04.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.969+0000 7fd541645740 20 client.7371 trim_cache size 0 max 16384 2024-03-24T00:44:04.974 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:44:04.974 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T00:44:04.974 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": { 2024-03-24T00:44:04.974 INFO:teuthology.orchestra.run.smithi067.stdout: "1af79934-4dae-42ec-a8c5-e84e101232cf": { 2024-03-24T00:44:04.974 INFO:teuthology.orchestra.run.smithi067.stdout: "path": "/subdir", 2024-03-24T00:44:04.974 INFO:teuthology.orchestra.run.smithi067.stdout: "tag": "1af79934-4dae-42ec-a8c5-e84e101232cf", 2024-03-24T00:44:04.975 INFO:teuthology.orchestra.run.smithi067.stdout: "options": "recursive,repair" 2024-03-24T00:44:04.975 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:44:04.975 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:44:04.975 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:44:04.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.969+0000 7fd53ea62640 1 -- 172.21.15.67:0/6134188 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd52406bbd0 msgr2=0x7fd52406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:04.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.969+0000 7fd53ea62640 1 --2- 172.21.15.67:0/6134188 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd52406bbd0 0x7fd52406e090 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fd538159860 tx=0x7fd52c008040 comp rx=0 tx=0).stop 2024-03-24T00:44:04.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.969+0000 7fd53ea62640 1 -- 172.21.15.67:0/6134188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd538072380 msgr2=0x7fd53813e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:04.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.969+0000 7fd53ea62640 1 --2- 172.21.15.67:0/6134188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd538072380 0x7fd53813e780 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fd52800c430 tx=0x7fd52800c8f0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.970+0000 7fd53ea62640 1 -- 172.21.15.67:0/6134188 shutdown_connections 2024-03-24T00:44:04.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.970+0000 7fd53ea62640 1 --2- 172.21.15.67:0/6134188 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd52406bbd0 0x7fd52406e090 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.970+0000 7fd53ea62640 1 --2- 172.21.15.67:0/6134188 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd53815a500 0x7fd5380af640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.970+0000 7fd53ea62640 1 --2- 172.21.15.67:0/6134188 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd5380cd7a0 0x7fd5381f4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.970+0000 7fd53ea62640 1 --2- 172.21.15.67:0/6134188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd538072380 0x7fd53813e780 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:04.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.970+0000 7fd53ea62640 1 -- 172.21.15.67:0/6134188 >> 172.21.15.67:0/6134188 conn(0x7fd5380cf9e0 msgr2=0x7fd53806b040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:04.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.970+0000 7fd53ea62640 1 -- 172.21.15.67:0/6134188 shutdown_connections 2024-03-24T00:44:04.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:04.970+0000 7fd53ea62640 1 -- 172.21.15.67:0/6134188 wait complete. 2024-03-24T00:44:04.980 INFO:tasks.cephfs.filesystem:scrub status for tag:1af79934-4dae-42ec-a8c5-e84e101232cf - {'path': '/subdir', 'tag': '1af79934-4dae-42ec-a8c5-e84e101232cf', 'options': 'recursive,repair'} 2024-03-24T00:44:16.506 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:44:16.508 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:44:34.981 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:44:34.981 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.5:0 scrub status 2024-03-24T00:44:35.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.084+0000 7f031f685640 1 Processor -- start 2024-03-24T00:44:35.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.085+0000 7f031f685640 1 -- start start 2024-03-24T00:44:35.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.085+0000 7f031f685640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f03180d3ae0 0x7f03180b5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.085+0000 7f031f685640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03180d2250 con 0x7f031815e280 2024-03-24T00:44:35.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.085+0000 7f031f685640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03180d2530 con 0x7f031815dea0 2024-03-24T00:44:35.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.085+0000 7f031f685640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03180d2810 con 0x7f03180d3ae0 2024-03-24T00:44:35.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.086+0000 7f031dc22640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f031815e280 0x7f03180b70f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47792/0 (socket says 172.21.15.67:47792) 2024-03-24T00:44:35.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.086+0000 7f031dc22640 1 -- 172.21.15.67:0/2811047405 learned_addr learned my addr 172.21.15.67:0/2811047405 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:44:35.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.086+0000 7f031cc20640 1 --2- 172.21.15.67:0/2811047405 >> v2:172.21.15.72:3300/0 conn(0x7f03180d3ae0 0x7f03180b5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.086+0000 7f030ffff640 1 -- 172.21.15.67:0/2811047405 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3179263097 0 0) 0x7f03180d2250 con 0x7f031815e280 2024-03-24T00:44:35.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.086+0000 7f030ffff640 1 -- 172.21.15.67:0/2811047405 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0304003520 con 0x7f031815e280 2024-03-24T00:44:35.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.086+0000 7f030ffff640 1 -- 172.21.15.67:0/2811047405 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4229495358 0 0) 0x7f0304003520 con 0x7f031815e280 2024-03-24T00:44:35.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.086+0000 7f030ffff640 1 -- 172.21.15.67:0/2811047405 >> v2:172.21.15.72:3300/0 conn(0x7f03180d3ae0 msgr2=0x7f03180b5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:35.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.086+0000 7f030ffff640 1 --2- 172.21.15.67:0/2811047405 >> v2:172.21.15.72:3300/0 conn(0x7f03180d3ae0 0x7f03180b5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.086+0000 7f030ffff640 1 -- 172.21.15.67:0/2811047405 >> v1:172.21.15.72:6789/0 conn(0x7f031815dea0 legacy=0x7f03180d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:35.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.086+0000 7f030ffff640 1 -- 172.21.15.67:0/2811047405 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03180d2af0 con 0x7f031815e280 2024-03-24T00:44:35.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.086+0000 7f030ffff640 1 -- 172.21.15.67:0/2811047405 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f03100019f0 con 0x7f031815e280 2024-03-24T00:44:35.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.086+0000 7f030ffff640 1 -- 172.21.15.67:0/2811047405 >> v1:172.21.15.67:6789/0 conn(0x7f031815e280 legacy=0x7f03180b70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:35.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.086+0000 7f030ffff640 1 --2- 172.21.15.67:0/2811047405 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03040039e0 0x7f0304003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.086+0000 7f030ffff640 1 --2- 172.21.15.67:0/2811047405 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0304004800 0x7f0304004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.086+0000 7f030ffff640 1 --2- 172.21.15.67:0/2811047405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03040051c0 0x7f030400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.086+0000 7f030ffff640 1 -- 172.21.15.67:0/2811047405 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03180d2810 con 0x7f03040051c0 2024-03-24T00:44:35.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.086+0000 7f030ffff640 1 -- 172.21.15.67:0/2811047405 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f030400c0c0 con 0x7f03040039e0 2024-03-24T00:44:35.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.086+0000 7f030ffff640 1 -- 172.21.15.67:0/2811047405 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f030400c2c0 con 0x7f0304004800 2024-03-24T00:44:35.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.086+0000 7f031dc22640 1 --2- 172.21.15.67:0/2811047405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03040051c0 0x7f030400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.086+0000 7f031d421640 1 --2- 172.21.15.67:0/2811047405 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03040039e0 0x7f0304003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.086+0000 7f031cc20640 1 --2- 172.21.15.67:0/2811047405 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0304004800 0x7f0304004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.087+0000 7f031dc22640 1 -- 172.21.15.67:0/2811047405 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0304004800 msgr2=0x7f0304004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:35.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.087+0000 7f031dc22640 1 --2- 172.21.15.67:0/2811047405 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0304004800 0x7f0304004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.087+0000 7f031dc22640 1 -- 172.21.15.67:0/2811047405 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03040039e0 msgr2=0x7f0304003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:35.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.087+0000 7f031dc22640 1 --2- 172.21.15.67:0/2811047405 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03040039e0 0x7f0304003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.087+0000 7f031dc22640 1 -- 172.21.15.67:0/2811047405 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0304003520 con 0x7f03040051c0 2024-03-24T00:44:35.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.087+0000 7f031dc22640 1 -- 172.21.15.67:0/2811047405 reap_dead start 2024-03-24T00:44:35.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.087+0000 7f031dc22640 1 --2- 172.21.15.67:0/2811047405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03040051c0 0x7f030400b9c0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f031815e710 tx=0x7f0304005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db4f71b90c2d029 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:35.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.087+0000 7f030ffff640 1 -- 172.21.15.67:0/2811047405 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f031000a410 con 0x7f03040051c0 2024-03-24T00:44:35.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.087+0000 7f030ffff640 1 -- 172.21.15.67:0/2811047405 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0310034630 con 0x7f03040051c0 2024-03-24T00:44:35.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.087+0000 7f030ffff640 1 -- 172.21.15.67:0/2811047405 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0310034a20 con 0x7f03040051c0 2024-03-24T00:44:35.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.088+0000 7f031f685640 1 -- 172.21.15.67:0/2811047405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03040051c0 msgr2=0x7f030400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:35.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.088+0000 7f031f685640 1 --2- 172.21.15.67:0/2811047405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03040051c0 0x7f030400b9c0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f031815e710 tx=0x7f0304005830 comp rx=0 tx=0).stop 2024-03-24T00:44:35.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.088+0000 7f031f685640 1 -- 172.21.15.67:0/2811047405 shutdown_connections 2024-03-24T00:44:35.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.088+0000 7f031f685640 1 --2- 172.21.15.67:0/2811047405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03040051c0 0x7f030400b9c0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.088+0000 7f031f685640 1 -- 172.21.15.67:0/2811047405 >> 172.21.15.67:0/2811047405 conn(0x7f0318072da0 msgr2=0x7f03180d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:35.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.088+0000 7f031f685640 1 -- 172.21.15.67:0/2811047405 shutdown_connections 2024-03-24T00:44:35.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.088+0000 7f031f685640 1 -- 172.21.15.67:0/2811047405 wait complete. 2024-03-24T00:44:35.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.089+0000 7f031f685640 1 Processor -- start 2024-03-24T00:44:35.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.089+0000 7f031f685640 1 -- start start 2024-03-24T00:44:35.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.090+0000 7f031f685640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f03040051c0 0x7f0318154850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.090+0000 7f031f685640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0318149fa0 0x7f0318160340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.090+0000 7f031f685640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03180ca260 0x7f03180ca6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.090+0000 7f031f685640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03180d3760 con 0x7f0318149fa0 2024-03-24T00:44:35.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.090+0000 7f031f685640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03180d35e0 con 0x7f03180ca260 2024-03-24T00:44:35.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.090+0000 7f031f685640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03180d38e0 con 0x7f03040051c0 2024-03-24T00:44:35.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.090+0000 7f031cc20640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0318149fa0 0x7f0318160340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.090+0000 7f031cc20640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0318149fa0 0x7f0318160340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55158/0 (socket says 172.21.15.67:55158) 2024-03-24T00:44:35.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.090+0000 7f031cc20640 1 -- 172.21.15.67:0/4066735089 learned_addr learned my addr 172.21.15.67:0/4066735089 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:35.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.090+0000 7f031dc22640 1 --2- 172.21.15.67:0/4066735089 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03180ca260 0x7f03180ca6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.090+0000 7f031d421640 1 --2- 172.21.15.67:0/4066735089 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f03040051c0 0x7f0318154850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.090+0000 7f031cc20640 1 -- 172.21.15.67:0/4066735089 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f03040051c0 msgr2=0x7f0318154850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:35.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.090+0000 7f031cc20640 1 --2- 172.21.15.67:0/4066735089 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f03040051c0 0x7f0318154850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.090+0000 7f031cc20640 1 -- 172.21.15.67:0/4066735089 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03180ca260 msgr2=0x7f03180ca6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:35.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.090+0000 7f031cc20640 1 --2- 172.21.15.67:0/4066735089 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03180ca260 0x7f03180ca6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.090+0000 7f031cc20640 1 -- 172.21.15.67:0/4066735089 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0318154d90 con 0x7f0318149fa0 2024-03-24T00:44:35.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.090+0000 7f031d421640 1 --2- 172.21.15.67:0/4066735089 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f03040051c0 0x7f0318154850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:44:35.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.091+0000 7f031cc20640 1 --2- 172.21.15.67:0/4066735089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0318149fa0 0x7f0318160340 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f0308011910 tx=0x7f0308011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:35.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.091+0000 7f030dffb640 1 -- 172.21.15.67:0/4066735089 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f030800faf0 con 0x7f0318149fa0 2024-03-24T00:44:35.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.091+0000 7f030dffb640 1 -- 172.21.15.67:0/4066735089 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f030800fc90 con 0x7f0318149fa0 2024-03-24T00:44:35.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.091+0000 7f031f685640 1 -- 172.21.15.67:0/4066735089 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03181557e0 con 0x7f0318149fa0 2024-03-24T00:44:35.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.091+0000 7f030dffb640 1 -- 172.21.15.67:0/4066735089 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0308014440 con 0x7f0318149fa0 2024-03-24T00:44:35.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.091+0000 7f031f685640 1 -- 172.21.15.67:0/4066735089 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0318157560 con 0x7f0318149fa0 2024-03-24T00:44:35.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.092+0000 7f030dffb640 1 -- 172.21.15.67:0/4066735089 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0308012040 con 0x7f0318149fa0 2024-03-24T00:44:35.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.093+0000 7f030dffb640 1 --2- 172.21.15.67:0/4066735089 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f02fc06bbd0 0x7f02fc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.093+0000 7f030dffb640 1 -- 172.21.15.67:0/4066735089 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f030808b1f0 con 0x7f0318149fa0 2024-03-24T00:44:35.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.093+0000 7f031d421640 1 --2- 172.21.15.67:0/4066735089 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f02fc06bbd0 0x7f02fc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.094+0000 7f031d421640 1 --2- 172.21.15.67:0/4066735089 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f02fc06bbd0 0x7f02fc06e090 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f03140035f0 tx=0x7f0314009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:35.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.094+0000 7f032227b740 1 Processor -- start 2024-03-24T00:44:35.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.094+0000 7f032227b740 1 -- start start 2024-03-24T00:44:35.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.095+0000 7f032227b740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5581a5c44e50 0x5581a5c17df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.095+0000 7f032227b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a5c45230 0x5581a5c47be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.095+0000 7f032227b740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5581a5c18330 0x5581a5c48120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.095+0000 7f032227b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581a5b823d0 con 0x5581a5c45230 2024-03-24T00:44:35.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.095+0000 7f032227b740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581a5b2aad0 con 0x5581a5c44e50 2024-03-24T00:44:35.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.095+0000 7f032227b740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5581a5b7e470 con 0x5581a5c18330 2024-03-24T00:44:35.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.095+0000 7f031cc20640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a5c45230 0x5581a5c47be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.095+0000 7f031cc20640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a5c45230 0x5581a5c47be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55170/0 (socket says 172.21.15.67:55170) 2024-03-24T00:44:35.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.095+0000 7f031cc20640 1 -- 172.21.15.67:0/2871568184 learned_addr learned my addr 172.21.15.67:0/2871568184 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:35.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.095+0000 7f031d421640 1 --2- 172.21.15.67:0/2871568184 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5581a5c44e50 0x5581a5c17df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.095+0000 7f031dc22640 1 --2- 172.21.15.67:0/2871568184 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5581a5c18330 0x5581a5c48120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.095+0000 7f031cc20640 1 -- 172.21.15.67:0/2871568184 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5581a5c18330 msgr2=0x5581a5c48120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:35.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.095+0000 7f031cc20640 1 --2- 172.21.15.67:0/2871568184 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5581a5c18330 0x5581a5c48120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.095+0000 7f031cc20640 1 -- 172.21.15.67:0/2871568184 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5581a5c44e50 msgr2=0x5581a5c17df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:35.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.095+0000 7f031cc20640 1 --2- 172.21.15.67:0/2871568184 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5581a5c44e50 0x5581a5c17df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.095+0000 7f031cc20640 1 -- 172.21.15.67:0/2871568184 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5581a59607d0 con 0x5581a5c45230 2024-03-24T00:44:35.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.095+0000 7f031dc22640 1 --2- 172.21.15.67:0/2871568184 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5581a5c18330 0x5581a5c48120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:44:35.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.095+0000 7f031d421640 1 --2- 172.21.15.67:0/2871568184 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5581a5c44e50 0x5581a5c17df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:44:35.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.095+0000 7f031cc20640 1 --2- 172.21.15.67:0/2871568184 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a5c45230 0x5581a5c47be0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f0308089570 tx=0x7f0308059680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e720413850b789b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:35.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.095+0000 7f0302ffd640 1 -- 172.21.15.67:0/2871568184 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0308057840 con 0x5581a5c45230 2024-03-24T00:44:35.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.095+0000 7f0302ffd640 1 -- 172.21.15.67:0/2871568184 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0308055bb0 con 0x5581a5c45230 2024-03-24T00:44:35.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.095+0000 7f0302ffd640 1 -- 172.21.15.67:0/2871568184 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f030805d170 con 0x5581a5c45230 2024-03-24T00:44:35.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.096+0000 7f032227b740 1 -- 172.21.15.67:0/2871568184 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a5c45230 msgr2=0x5581a5c47be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:35.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.096+0000 7f032227b740 1 --2- 172.21.15.67:0/2871568184 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a5c45230 0x5581a5c47be0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f0308089570 tx=0x7f0308059680 comp rx=0 tx=0).stop 2024-03-24T00:44:35.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.096+0000 7f032227b740 1 -- 172.21.15.67:0/2871568184 shutdown_connections 2024-03-24T00:44:35.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.096+0000 7f032227b740 1 --2- 172.21.15.67:0/2871568184 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5581a5c18330 0x5581a5c48120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.096+0000 7f032227b740 1 --2- 172.21.15.67:0/2871568184 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a5c45230 0x5581a5c47be0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.096+0000 7f032227b740 1 --2- 172.21.15.67:0/2871568184 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5581a5c44e50 0x5581a5c17df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.096+0000 7f032227b740 1 -- 172.21.15.67:0/2871568184 >> 172.21.15.67:0/2871568184 conn(0x5581a5c1a060 msgr2=0x5581a5c62be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:35.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.096+0000 7f032227b740 1 -- 172.21.15.67:0/2871568184 shutdown_connections 2024-03-24T00:44:35.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.096+0000 7f032227b740 1 -- 172.21.15.67:0/2871568184 wait complete. 2024-03-24T00:44:35.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.096+0000 7f032227b740 1 -- start start 2024-03-24T00:44:35.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.096+0000 7f032227b740 1 Processor -- start 2024-03-24T00:44:35.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.097+0000 7f032227b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a5c44e50 0x5581a5c76220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.097+0000 7f032227b740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5581a5c45230 0x5581a5c4b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.097+0000 7f032227b740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5581a5c49090 0x5581a5c76820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.097+0000 7f032227b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581a5c1c700 con 0x5581a5c44e50 2024-03-24T00:44:35.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.097+0000 7f032227b740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581a5b3e060 con 0x5581a5c49090 2024-03-24T00:44:35.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.097+0000 7f032227b740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5581a5c1ef40 con 0x5581a5c45230 2024-03-24T00:44:35.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.097+0000 7f032227b740 10 client.0 _ll_register_callbacks cb 0x5581a5b206a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:44:35.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.097+0000 7f031d421640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a5c44e50 0x5581a5c76220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.098+0000 7f031d421640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a5c44e50 0x5581a5c76220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55172/0 (socket says 172.21.15.67:55172) 2024-03-24T00:44:35.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.098+0000 7f031d421640 1 -- 172.21.15.67:0/2931563896 learned_addr learned my addr 172.21.15.67:0/2931563896 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:35.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.098+0000 7f031dc22640 1 --2- 172.21.15.67:0/2931563896 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5581a5c49090 0x5581a5c76820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.098+0000 7f031cc20640 1 --2- 172.21.15.67:0/2931563896 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5581a5c45230 0x5581a5c4b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.098+0000 7f031d421640 1 -- 172.21.15.67:0/2931563896 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5581a5c45230 msgr2=0x5581a5c4b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:35.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.098+0000 7f031d421640 1 --2- 172.21.15.67:0/2931563896 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5581a5c45230 0x5581a5c4b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.098+0000 7f031d421640 1 -- 172.21.15.67:0/2931563896 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5581a5c49090 msgr2=0x5581a5c76820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:35.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.098+0000 7f031d421640 1 --2- 172.21.15.67:0/2931563896 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5581a5c49090 0x5581a5c76820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.098+0000 7f031d421640 1 -- 172.21.15.67:0/2931563896 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5581a58963f0 con 0x5581a5c44e50 2024-03-24T00:44:35.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.098+0000 7f031d421640 1 --2- 172.21.15.67:0/2931563896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a5c44e50 0x5581a5c76220 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f0314014430 tx=0x7f03140012b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1661415dbc9057c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:35.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.098+0000 7f02f7fff640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:44:35.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.098+0000 7f02f7fff640 1 -- 172.21.15.67:0/2931563896 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0314028060 con 0x5581a5c44e50 2024-03-24T00:44:35.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.098+0000 7f02f7fff640 1 -- 172.21.15.67:0/2931563896 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f0314023c90 con 0x5581a5c44e50 2024-03-24T00:44:35.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.098+0000 7f032227b740 1 -- 172.21.15.67:0/2931563896 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0308055bb0 con 0x5581a5c44e50 2024-03-24T00:44:35.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.098+0000 7f02f7fff640 1 -- 172.21.15.67:0/2931563896 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0314007460 con 0x5581a5c44e50 2024-03-24T00:44:35.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.098+0000 7f032227b740 10 client.7383 fetch_fsmap learned FSMap version 48 2024-03-24T00:44:35.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.098+0000 7f032227b740 1 -- 172.21.15.67:0/2931563896 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=2+}) v3 -- 0x5581a58560f0 con 0x5581a5c44e50 2024-03-24T00:44:35.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.099+0000 7f02f7fff640 1 -- 172.21.15.67:0/2931563896 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0308055bb0 con 0x5581a5c44e50 2024-03-24T00:44:35.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.099+0000 7f02f7fff640 1 -- 172.21.15.67:0/2931563896 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 48) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f03140067d0 con 0x5581a5c44e50 2024-03-24T00:44:35.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.099+0000 7f032227b740 10 client.7383 fetch_fsmap finished waiting for FSMap version 48 2024-03-24T00:44:35.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.099+0000 7f032227b740 10 client.7383 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.c 2024-03-24T00:44:35.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.099+0000 7f032227b740 20 client.7383 populate_metadata read hostname 'smithi067' 2024-03-24T00:44:35.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.099+0000 7f032227b740 1 --2- 172.21.15.67:0/2931563896 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x5581a5c47130 0x5581a5c47560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.099+0000 7f032227b740 4 client.7383 mds_command: new command op to 7149 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:44:35.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.099+0000 7f032227b740 1 -- 172.21.15.67:0/2931563896 --> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5581a5b8f910 con 0x5581a5c47130 2024-03-24T00:44:35.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.099+0000 7f031cc20640 1 --2- 172.21.15.67:0/2931563896 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x5581a5c47130 0x5581a5c47560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.099+0000 7f02f7fff640 10 client.7383 ms_handle_connect on v2:172.21.15.67:6832/720256752 2024-03-24T00:44:35.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.099+0000 7f031cc20640 1 --2- 172.21.15.67:0/2931563896 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x5581a5c47130 0x5581a5c47560 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56ac0c5d9810cb07 server_cookie=9f092bf9aa2a12f1 in_seq=0 out_seq=0 2024-03-24T00:44:35.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.100+0000 7f02f7fff640 1 -- 172.21.15.67:0/2931563896 <== mds.0 v2:172.21.15.67:6832/720256752 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5581a5b8f910 con 0x5581a5c47130 2024-03-24T00:44:35.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.100+0000 7f02f7fff640 10 client.7383 handle_command_reply: tid=0 2024-03-24T00:44:35.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.100+0000 7f032227b740 1 client.7383 shutdown 2024-03-24T00:44:35.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.100+0000 7f032227b740 1 -- 172.21.15.67:0/2931563896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a5c44e50 msgr2=0x5581a5c76220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:35.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.100+0000 7f032227b740 1 --2- 172.21.15.67:0/2931563896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a5c44e50 0x5581a5c76220 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f0314014430 tx=0x7f03140012b0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.100+0000 7f032227b740 1 -- 172.21.15.67:0/2931563896 shutdown_connections 2024-03-24T00:44:35.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.100+0000 7f032227b740 1 --2- 172.21.15.67:0/2931563896 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x5581a5c47130 0x5581a5c47560 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.100+0000 7f032227b740 1 --2- 172.21.15.67:0/2931563896 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5581a5c49090 0x5581a5c76820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.100+0000 7f032227b740 1 --2- 172.21.15.67:0/2931563896 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5581a5c45230 0x5581a5c4b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.100+0000 7f032227b740 1 --2- 172.21.15.67:0/2931563896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a5c44e50 0x5581a5c76220 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.100+0000 7f032227b740 1 -- 172.21.15.67:0/2931563896 >> 172.21.15.67:0/2931563896 conn(0x5581a5c18330 msgr2=0x5581a5c1bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:35.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.100+0000 7f02f7fff640 0 client.7383 ms_handle_reset on v2:172.21.15.67:6832/720256752 2024-03-24T00:44:35.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.100+0000 7f032227b740 1 -- 172.21.15.67:0/2931563896 shutdown_connections 2024-03-24T00:44:35.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.100+0000 7f032227b740 1 -- 172.21.15.67:0/2931563896 wait complete. 2024-03-24T00:44:35.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.100+0000 7f032227b740 20 client.7383 trim_cache size 0 max 16384 2024-03-24T00:44:35.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.116+0000 7f032227b740 1 Processor -- start 2024-03-24T00:44:35.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.116+0000 7f032227b740 1 -- start start 2024-03-24T00:44:35.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.117+0000 7f032227b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a5c44e50 0x5581a5c76220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.117+0000 7f032227b740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5581a5c45230 0x5581a5c62fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.117+0000 7f032227b740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5581a5c49090 0x5581a5c63500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.117+0000 7f032227b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0318200f60 con 0x5581a5c44e50 2024-03-24T00:44:35.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.117+0000 7f032227b740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03182000e0 con 0x5581a5c49090 2024-03-24T00:44:35.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.117+0000 7f032227b740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03181ff4e0 con 0x5581a5c45230 2024-03-24T00:44:35.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.117+0000 7f031d421640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a5c44e50 0x5581a5c76220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.117+0000 7f031d421640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a5c44e50 0x5581a5c76220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55186/0 (socket says 172.21.15.67:55186) 2024-03-24T00:44:35.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.117+0000 7f031d421640 1 -- 172.21.15.67:0/1323609407 learned_addr learned my addr 172.21.15.67:0/1323609407 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:35.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.117+0000 7f031dc22640 1 --2- 172.21.15.67:0/1323609407 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5581a5c49090 0x5581a5c63500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.117+0000 7f031cc20640 1 --2- 172.21.15.67:0/1323609407 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5581a5c45230 0x5581a5c62fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.117+0000 7f031d421640 1 -- 172.21.15.67:0/1323609407 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5581a5c45230 msgr2=0x5581a5c62fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:35.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.117+0000 7f031d421640 1 --2- 172.21.15.67:0/1323609407 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5581a5c45230 0x5581a5c62fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.117+0000 7f031d421640 1 -- 172.21.15.67:0/1323609407 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5581a5c49090 msgr2=0x5581a5c63500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:35.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.117+0000 7f031d421640 1 --2- 172.21.15.67:0/1323609407 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5581a5c49090 0x5581a5c63500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.117+0000 7f031d421640 1 -- 172.21.15.67:0/1323609407 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5581a5c4bd40 con 0x5581a5c44e50 2024-03-24T00:44:35.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.117+0000 7f031dc22640 1 --2- 172.21.15.67:0/1323609407 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5581a5c49090 0x5581a5c63500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:44:35.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.117+0000 7f031d421640 1 --2- 172.21.15.67:0/1323609407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a5c44e50 0x5581a5c76220 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f0314000fd0 tx=0x7f0314021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30c303eee3632567 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:35.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.118+0000 7f03027fc640 1 -- 172.21.15.67:0/1323609407 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0314014c00 con 0x5581a5c44e50 2024-03-24T00:44:35.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.118+0000 7f03027fc640 1 -- 172.21.15.67:0/1323609407 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0314007260 con 0x5581a5c44e50 2024-03-24T00:44:35.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.118+0000 7f03027fc640 1 -- 172.21.15.67:0/1323609407 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03140067d0 con 0x5581a5c44e50 2024-03-24T00:44:35.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.118+0000 7f032227b740 1 -- 172.21.15.67:0/1323609407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a5c44e50 msgr2=0x5581a5c76220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:35.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.118+0000 7f032227b740 1 --2- 172.21.15.67:0/1323609407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a5c44e50 0x5581a5c76220 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f0314000fd0 tx=0x7f0314021000 comp rx=0 tx=0).stop 2024-03-24T00:44:35.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.118+0000 7f032227b740 1 -- 172.21.15.67:0/1323609407 shutdown_connections 2024-03-24T00:44:35.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.118+0000 7f032227b740 1 --2- 172.21.15.67:0/1323609407 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5581a5c49090 0x5581a5c63500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.118+0000 7f032227b740 1 --2- 172.21.15.67:0/1323609407 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5581a5c45230 0x5581a5c62fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.118+0000 7f032227b740 1 --2- 172.21.15.67:0/1323609407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a5c44e50 0x5581a5c76220 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.118+0000 7f032227b740 1 -- 172.21.15.67:0/1323609407 >> 172.21.15.67:0/1323609407 conn(0x5581a5c18330 msgr2=0x5581a5c47880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:35.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.118+0000 7f032227b740 1 -- 172.21.15.67:0/1323609407 shutdown_connections 2024-03-24T00:44:35.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.118+0000 7f032227b740 1 -- 172.21.15.67:0/1323609407 wait complete. 2024-03-24T00:44:35.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.119+0000 7f032227b740 1 -- start start 2024-03-24T00:44:35.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.119+0000 7f032227b740 1 Processor -- start 2024-03-24T00:44:35.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.120+0000 7f032227b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a5c44e50 0x5581a5c437e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.120+0000 7f032227b740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5581a5c45230 0x5581a5c43d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.120+0000 7f032227b740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5581a5c49090 0x5581a5cd0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.120+0000 7f032227b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03181ffae0 con 0x5581a5c44e50 2024-03-24T00:44:35.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.120+0000 7f032227b740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581a5c1ea30 con 0x5581a5c45230 2024-03-24T00:44:35.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.120+0000 7f032227b740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5581a5cd1ce0 con 0x5581a5c49090 2024-03-24T00:44:35.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.120+0000 7f032227b740 10 client.0 _ll_register_callbacks cb 0x7f0314001a90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:44:35.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.120+0000 7f031d421640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a5c44e50 0x5581a5c437e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.120+0000 7f031d421640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a5c44e50 0x5581a5c437e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55192/0 (socket says 172.21.15.67:55192) 2024-03-24T00:44:35.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.120+0000 7f031d421640 1 -- 172.21.15.67:0/3115523868 learned_addr learned my addr 172.21.15.67:0/3115523868 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:35.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.120+0000 7f031d421640 1 -- 172.21.15.67:0/3115523868 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5581a5c49090 msgr2=0x5581a5cd0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:35.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.120+0000 7f031d421640 1 --2- 172.21.15.67:0/3115523868 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5581a5c49090 0x5581a5cd0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.120+0000 7f031d421640 1 -- 172.21.15.67:0/3115523868 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5581a5c45230 msgr2=0x5581a5c43d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:35.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.120+0000 7f031d421640 1 --2- 172.21.15.67:0/3115523868 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5581a5c45230 0x5581a5c43d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.120+0000 7f031d421640 1 -- 172.21.15.67:0/3115523868 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5581a5c47250 con 0x5581a5c44e50 2024-03-24T00:44:35.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.120+0000 7f031d421640 1 --2- 172.21.15.67:0/3115523868 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a5c44e50 0x5581a5c437e0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f0314006c50 tx=0x7f0314014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c02b8d85d60860e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:35.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.120+0000 7f02f7fff640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:44:35.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.120+0000 7f02f7fff640 1 -- 172.21.15.67:0/3115523868 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0314028030 con 0x5581a5c44e50 2024-03-24T00:44:35.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.120+0000 7f02f7fff640 1 -- 172.21.15.67:0/3115523868 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f0314033040 con 0x5581a5c44e50 2024-03-24T00:44:35.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.120+0000 7f032227b740 1 -- 172.21.15.67:0/3115523868 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0314007260 con 0x5581a5c44e50 2024-03-24T00:44:35.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.120+0000 7f02f7fff640 1 -- 172.21.15.67:0/3115523868 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0314014c00 con 0x5581a5c44e50 2024-03-24T00:44:35.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.120+0000 7f032227b740 10 client.7389 fetch_fsmap learned FSMap version 48 2024-03-24T00:44:35.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.121+0000 7f032227b740 1 -- 172.21.15.67:0/3115523868 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=2+}) v3 -- 0x5581a5c17ac0 con 0x5581a5c44e50 2024-03-24T00:44:35.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.121+0000 7f02f7fff640 1 -- 172.21.15.67:0/3115523868 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0314007260 con 0x5581a5c44e50 2024-03-24T00:44:35.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.121+0000 7f02f7fff640 1 -- 172.21.15.67:0/3115523868 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 48) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f0314040070 con 0x5581a5c44e50 2024-03-24T00:44:35.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.121+0000 7f032227b740 10 client.7389 fetch_fsmap finished waiting for FSMap version 48 2024-03-24T00:44:35.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.121+0000 7f032227b740 10 client.7389 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.c 2024-03-24T00:44:35.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.121+0000 7f032227b740 20 client.7389 populate_metadata read hostname 'smithi067' 2024-03-24T00:44:35.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.121+0000 7f032227b740 1 --2- 172.21.15.67:0/3115523868 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x5581a5c1ef40 0x5581a5cd1390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.121+0000 7f032227b740 4 client.7389 mds_command: new command op to 7149 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:44:35.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.121+0000 7f032227b740 1 -- 172.21.15.67:0/3115523868 --> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5581a5b89c60 con 0x5581a5c1ef40 2024-03-24T00:44:35.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.121+0000 7f031cc20640 1 --2- 172.21.15.67:0/3115523868 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x5581a5c1ef40 0x5581a5cd1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.121+0000 7f02f7fff640 10 client.7389 ms_handle_connect on v2:172.21.15.67:6832/720256752 2024-03-24T00:44:35.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.121+0000 7f031cc20640 1 --2- 172.21.15.67:0/3115523868 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x5581a5c1ef40 0x5581a5cd1390 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9e88f37be86e0ab server_cookie=675ab21ac95cbd6b in_seq=0 out_seq=0 2024-03-24T00:44:35.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.121+0000 7f02f7fff640 1 -- 172.21.15.67:0/3115523868 <== mds.0 v2:172.21.15.67:6832/720256752 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5581a5b89c60 con 0x5581a5c1ef40 2024-03-24T00:44:35.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.121+0000 7f02f7fff640 10 client.7389 handle_command_reply: tid=0 2024-03-24T00:44:35.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.121+0000 7f032227b740 1 client.7389 shutdown 2024-03-24T00:44:35.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.121+0000 7f032227b740 1 -- 172.21.15.67:0/3115523868 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a5c44e50 msgr2=0x5581a5c437e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:35.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.122+0000 7f032227b740 1 --2- 172.21.15.67:0/3115523868 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a5c44e50 0x5581a5c437e0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f0314006c50 tx=0x7f0314014000 comp rx=0 tx=0).stop 2024-03-24T00:44:35.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.122+0000 7f032227b740 1 -- 172.21.15.67:0/3115523868 shutdown_connections 2024-03-24T00:44:35.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.122+0000 7f032227b740 1 --2- 172.21.15.67:0/3115523868 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5581a5c49090 0x5581a5cd0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.122+0000 7f032227b740 1 --2- 172.21.15.67:0/3115523868 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x5581a5c1ef40 0x5581a5cd1390 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.122+0000 7f032227b740 1 --2- 172.21.15.67:0/3115523868 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5581a5c45230 0x5581a5c43d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.122+0000 7f032227b740 1 --2- 172.21.15.67:0/3115523868 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581a5c44e50 0x5581a5c437e0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.122+0000 7f02f7fff640 0 client.7389 ms_handle_reset on v2:172.21.15.67:6832/720256752 2024-03-24T00:44:35.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.122+0000 7f032227b740 1 -- 172.21.15.67:0/3115523868 >> 172.21.15.67:0/3115523868 conn(0x5581a5c18330 msgr2=0x5581a5c632d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:35.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.122+0000 7f032227b740 1 -- 172.21.15.67:0/3115523868 shutdown_connections 2024-03-24T00:44:35.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.122+0000 7f032227b740 1 -- 172.21.15.67:0/3115523868 wait complete. 2024-03-24T00:44:35.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.122+0000 7f032227b740 20 client.7389 trim_cache size 0 max 16384 2024-03-24T00:44:35.128 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:44:35.128 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "no active scrubs running", 2024-03-24T00:44:35.129 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": {} 2024-03-24T00:44:35.129 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:44:35.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.122+0000 7f031f685640 1 -- 172.21.15.67:0/4066735089 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f02fc06bbd0 msgr2=0x7f02fc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:35.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.122+0000 7f031f685640 1 --2- 172.21.15.67:0/4066735089 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f02fc06bbd0 0x7f02fc06e090 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f03140035f0 tx=0x7f0314009040 comp rx=0 tx=0).stop 2024-03-24T00:44:35.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.122+0000 7f031f685640 1 -- 172.21.15.67:0/4066735089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0318149fa0 msgr2=0x7f0318160340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:35.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.122+0000 7f031f685640 1 --2- 172.21.15.67:0/4066735089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0318149fa0 0x7f0318160340 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f0308011910 tx=0x7f0308011dd0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.122+0000 7f031f685640 1 -- 172.21.15.67:0/4066735089 shutdown_connections 2024-03-24T00:44:35.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.122+0000 7f031f685640 1 --2- 172.21.15.67:0/4066735089 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f02fc06bbd0 0x7f02fc06e090 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.122+0000 7f031f685640 1 --2- 172.21.15.67:0/4066735089 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03180ca260 0x7f03180ca6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.122+0000 7f031f685640 1 --2- 172.21.15.67:0/4066735089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0318149fa0 0x7f0318160340 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.122+0000 7f031f685640 1 --2- 172.21.15.67:0/4066735089 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f03040051c0 0x7f0318154850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.122+0000 7f031f685640 1 -- 172.21.15.67:0/4066735089 >> 172.21.15.67:0/4066735089 conn(0x7f0318072da0 msgr2=0x7f03180b7b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:35.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.122+0000 7f031f685640 1 -- 172.21.15.67:0/4066735089 shutdown_connections 2024-03-24T00:44:35.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.122+0000 7f031f685640 1 -- 172.21.15.67:0/4066735089 wait complete. 2024-03-24T00:44:35.134 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T00:44:35.134 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:44:35.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.238+0000 7fa64ff74640 1 Processor -- start 2024-03-24T00:44:35.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.238+0000 7fa64ff74640 1 -- start start 2024-03-24T00:44:35.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.239+0000 7fa64ff74640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa64815dec0 0x7fa6480ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.239+0000 7fa64ff74640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa64815e2a0 0x7fa648162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.239+0000 7fa64ff74640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6480cf640 con 0x7fa6480d07e0 2024-03-24T00:44:35.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.239+0000 7fa64ff74640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6480cee30 con 0x7fa64815dec0 2024-03-24T00:44:35.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.239+0000 7fa64ff74640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6480d2120 con 0x7fa64815e2a0 2024-03-24T00:44:35.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.239+0000 7fa64dd10640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa64815dec0 0x7fa6480ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.239+0000 7fa64dd10640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa64815dec0 0x7fa6480ce630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55204/0 (socket says 172.21.15.67:55204) 2024-03-24T00:44:35.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.239+0000 7fa64dd10640 1 -- 172.21.15.67:0/1872877825 learned_addr learned my addr 172.21.15.67:0/1872877825 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:35.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.239+0000 7fa64dd10640 1 -- 172.21.15.67:0/1872877825 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa64815e2a0 msgr2=0x7fa648162630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:35.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.239+0000 7fa64d50f640 1 --2- 172.21.15.67:0/1872877825 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa64815e2a0 0x7fa648162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.239+0000 7fa64dd10640 1 --2- 172.21.15.67:0/1872877825 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa64815e2a0 0x7fa648162630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.239+0000 7fa64dd10640 1 -- 172.21.15.67:0/1872877825 >> v1:172.21.15.72:6789/0 conn(0x7fa6480d07e0 legacy=0x7fa6480b5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:35.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.239+0000 7fa64dd10640 1 -- 172.21.15.67:0/1872877825 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6480cf080 con 0x7fa64815dec0 2024-03-24T00:44:35.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.239+0000 7fa64d50f640 1 --2- 172.21.15.67:0/1872877825 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa64815e2a0 0x7fa648162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:44:35.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.240+0000 7fa64dd10640 1 --2- 172.21.15.67:0/1872877825 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa64815dec0 0x7fa6480ce630 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fa6480cefb0 tx=0x7fa64402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9f26d67ae135886 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:35.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.240+0000 7fa64cd0e640 1 -- 172.21.15.67:0/1872877825 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa644035020 con 0x7fa64815dec0 2024-03-24T00:44:35.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.240+0000 7fa64cd0e640 1 -- 172.21.15.67:0/1872877825 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa644031640 con 0x7fa64815dec0 2024-03-24T00:44:35.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.240+0000 7fa64cd0e640 1 -- 172.21.15.67:0/1872877825 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64402d040 con 0x7fa64815dec0 2024-03-24T00:44:35.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.241+0000 7fa64ff74640 1 -- 172.21.15.67:0/1872877825 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa64815dec0 msgr2=0x7fa6480ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:35.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.241+0000 7fa64ff74640 1 --2- 172.21.15.67:0/1872877825 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa64815dec0 0x7fa6480ce630 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fa6480cefb0 tx=0x7fa64402b8f0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.241+0000 7fa64ff74640 1 -- 172.21.15.67:0/1872877825 shutdown_connections 2024-03-24T00:44:35.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.241+0000 7fa64ff74640 1 --2- 172.21.15.67:0/1872877825 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa64815e2a0 0x7fa648162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.241+0000 7fa64ff74640 1 --2- 172.21.15.67:0/1872877825 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa64815dec0 0x7fa6480ce630 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.241+0000 7fa64ff74640 1 -- 172.21.15.67:0/1872877825 >> 172.21.15.67:0/1872877825 conn(0x7fa64806c810 msgr2=0x7fa64806cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:35.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.241+0000 7fa64ff74640 1 -- 172.21.15.67:0/1872877825 shutdown_connections 2024-03-24T00:44:35.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.241+0000 7fa64ff74640 1 -- 172.21.15.67:0/1872877825 wait complete. 2024-03-24T00:44:35.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.243+0000 7fa64ff74640 1 Processor -- start 2024-03-24T00:44:35.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.243+0000 7fa64ff74640 1 -- start start 2024-03-24T00:44:35.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.244+0000 7fa64ff74640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa6480d07e0 0x7fa648156e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.244+0000 7fa64ff74640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa64815dec0 0x7fa648159450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.244+0000 7fa64ff74640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa64815e2a0 0x7fa6480cd690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.244+0000 7fa64ff74640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6480d3740 con 0x7fa64815e2a0 2024-03-24T00:44:35.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.244+0000 7fa64ff74640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6480d35c0 con 0x7fa64815dec0 2024-03-24T00:44:35.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.244+0000 7fa64ff74640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6480d38c0 con 0x7fa6480d07e0 2024-03-24T00:44:35.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.244+0000 7fa64e511640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa64815e2a0 0x7fa6480cd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.244+0000 7fa64e511640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa64815e2a0 0x7fa6480cd690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55212/0 (socket says 172.21.15.67:55212) 2024-03-24T00:44:35.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.244+0000 7fa64e511640 1 -- 172.21.15.67:0/3892493525 learned_addr learned my addr 172.21.15.67:0/3892493525 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:35.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.244+0000 7fa64d50f640 1 --2- 172.21.15.67:0/3892493525 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa64815dec0 0x7fa648159450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.244+0000 7fa64dd10640 1 --2- 172.21.15.67:0/3892493525 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa6480d07e0 0x7fa648156e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.245+0000 7fa64e511640 1 -- 172.21.15.67:0/3892493525 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa6480d07e0 msgr2=0x7fa648156e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:35.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.245+0000 7fa64e511640 1 --2- 172.21.15.67:0/3892493525 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa6480d07e0 0x7fa648156e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.245+0000 7fa64e511640 1 -- 172.21.15.67:0/3892493525 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa64815dec0 msgr2=0x7fa648159450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:35.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.245+0000 7fa64e511640 1 --2- 172.21.15.67:0/3892493525 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa64815dec0 0x7fa648159450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.245+0000 7fa64e511640 1 -- 172.21.15.67:0/3892493525 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6481573c0 con 0x7fa64815e2a0 2024-03-24T00:44:35.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.245+0000 7fa64e511640 1 --2- 172.21.15.67:0/3892493525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa64815e2a0 0x7fa6480cd690 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fa64000c7a0 tx=0x7fa64000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:35.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.245+0000 7fa63e7fc640 1 -- 172.21.15.67:0/3892493525 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa640003b70 con 0x7fa64815e2a0 2024-03-24T00:44:35.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.245+0000 7fa63e7fc640 1 -- 172.21.15.67:0/3892493525 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa640003d30 con 0x7fa64815e2a0 2024-03-24T00:44:35.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.245+0000 7fa63e7fc640 1 -- 172.21.15.67:0/3892493525 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64000aa90 con 0x7fa64815e2a0 2024-03-24T00:44:35.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.245+0000 7fa64ff74640 1 -- 172.21.15.67:0/3892493525 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa648157660 con 0x7fa64815e2a0 2024-03-24T00:44:35.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.245+0000 7fa64ff74640 1 -- 172.21.15.67:0/3892493525 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa648157c20 con 0x7fa64815e2a0 2024-03-24T00:44:35.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.246+0000 7fa64ff74640 1 -- 172.21.15.67:0/3892493525 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa64804f400 con 0x7fa64815e2a0 2024-03-24T00:44:35.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.247+0000 7fa63e7fc640 1 -- 172.21.15.67:0/3892493525 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa64000ac30 con 0x7fa64815e2a0 2024-03-24T00:44:35.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.247+0000 7fa63e7fc640 1 --2- 172.21.15.67:0/3892493525 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa62c06bbd0 0x7fa62c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.247+0000 7fa63e7fc640 1 -- 172.21.15.67:0/3892493525 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fa64008a990 con 0x7fa64815e2a0 2024-03-24T00:44:35.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.247+0000 7fa64dd10640 1 --2- 172.21.15.67:0/3892493525 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa62c06bbd0 0x7fa62c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.248+0000 7fa64dd10640 1 --2- 172.21.15.67:0/3892493525 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa62c06bbd0 0x7fa62c06e090 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fa644036000 tx=0x7fa64403c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:35.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.252+0000 7fa63e7fc640 1 -- 172.21.15.67:0/3892493525 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa6480cf640 con 0x7fa64815e2a0 2024-03-24T00:44:35.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.472+0000 7fa64ff74640 1 -- 172.21.15.67:0/3892493525 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa64815a780 con 0x7fa64815e2a0 2024-03-24T00:44:35.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.473+0000 7fa63e7fc640 1 -- 172.21.15.67:0/3892493525 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+4978 (secure 0 0 0) 0x7fa640058ce0 con 0x7fa64815e2a0 2024-03-24T00:44:35.474 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:44:35.474 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":48,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5960,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1888378520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1888378520},{"type":"v1","addr":"172.21.15.67:6835","nonce":1888378520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6610,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/2539718386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2539718386},{"type":"v1","addr":"172.21.15.72:6836","nonce":2539718386}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6613,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2376370770","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2376370770},{"type":"v1","addr":"172.21.15.72:6837","nonce":2376370770}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:43:10.177088+0000","modified":"2024-03-24T00:44:11.389921+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7149},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7149":{"gid":7149,"name":"c","rank":0,"incarnation":43,"state":"up:active","state_seq":15,"addr":"172.21.15.67:6833/720256752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":720256752},{"type":"v1","addr":"172.21.15.67:6833","nonce":720256752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7149,"qdb_cluster":[7149]},"id":5}]} 2024-03-24T00:44:35.474 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 48 2024-03-24T00:44:35.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.475+0000 7fa64ff74640 1 -- 172.21.15.67:0/3892493525 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa62c06bbd0 msgr2=0x7fa62c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:35.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.475+0000 7fa64ff74640 1 --2- 172.21.15.67:0/3892493525 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa62c06bbd0 0x7fa62c06e090 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fa644036000 tx=0x7fa64403c040 comp rx=0 tx=0).stop 2024-03-24T00:44:35.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.475+0000 7fa64ff74640 1 -- 172.21.15.67:0/3892493525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa64815e2a0 msgr2=0x7fa6480cd690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:35.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.475+0000 7fa64ff74640 1 --2- 172.21.15.67:0/3892493525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa64815e2a0 0x7fa6480cd690 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fa64000c7a0 tx=0x7fa64000cc60 comp rx=0 tx=0).stop 2024-03-24T00:44:35.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.475+0000 7fa64ff74640 1 -- 172.21.15.67:0/3892493525 shutdown_connections 2024-03-24T00:44:35.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.475+0000 7fa64ff74640 1 --2- 172.21.15.67:0/3892493525 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa62c06bbd0 0x7fa62c06e090 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.475+0000 7fa64ff74640 1 --2- 172.21.15.67:0/3892493525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa64815e2a0 0x7fa6480cd690 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.475+0000 7fa64ff74640 1 --2- 172.21.15.67:0/3892493525 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa64815dec0 0x7fa648159450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.475+0000 7fa64ff74640 1 --2- 172.21.15.67:0/3892493525 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa6480d07e0 0x7fa648156e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.475+0000 7fa64ff74640 1 -- 172.21.15.67:0/3892493525 >> 172.21.15.67:0/3892493525 conn(0x7fa64806c810 msgr2=0x7fa64815d070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:35.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.475+0000 7fa64ff74640 1 -- 172.21.15.67:0/3892493525 shutdown_connections 2024-03-24T00:44:35.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.475+0000 7fa64ff74640 1 -- 172.21.15.67:0/3892493525 wait complete. 2024-03-24T00:44:35.487 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c damage ls --format=json-pretty 2024-03-24T00:44:35.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.590+0000 7f36969da640 1 Processor -- start 2024-03-24T00:44:35.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.590+0000 7f36969da640 1 -- start start 2024-03-24T00:44:35.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.591+0000 7f36969da640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f369015dea0 0x7f36900d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.591+0000 7f36969da640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f369015e280 0x7f36900d3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.591+0000 7f36969da640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36900d2250 con 0x7f36900b5a60 2024-03-24T00:44:35.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.591+0000 7f36969da640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36900d2530 con 0x7f369015dea0 2024-03-24T00:44:35.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.591+0000 7f36969da640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36900d2810 con 0x7f369015e280 2024-03-24T00:44:35.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.591+0000 7f368ffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f369015dea0 0x7f36900d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.591+0000 7f3694f77640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f36900b5a60 0x7f369006c640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47798/0 (socket says 172.21.15.67:47798) 2024-03-24T00:44:35.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.591+0000 7f3694f77640 1 -- 172.21.15.67:0/693221214 learned_addr learned my addr 172.21.15.67:0/693221214 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:44:35.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.592+0000 7f368f7fe640 1 --2- 172.21.15.67:0/693221214 >> v2:172.21.15.72:3300/0 conn(0x7f369015e280 0x7f36900d3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.592+0000 7f368ffff640 1 -- 172.21.15.67:0/693221214 >> v2:172.21.15.72:3300/0 conn(0x7f369015e280 msgr2=0x7f36900d3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:35.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.592+0000 7f368ffff640 1 --2- 172.21.15.67:0/693221214 >> v2:172.21.15.72:3300/0 conn(0x7f369015e280 0x7f36900d3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.592+0000 7f368ffff640 1 -- 172.21.15.67:0/693221214 >> v1:172.21.15.67:6789/0 conn(0x7f36900b5a60 legacy=0x7f369006c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:35.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.592+0000 7f368ffff640 1 -- 172.21.15.67:0/693221214 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36900d2af0 con 0x7f369015dea0 2024-03-24T00:44:35.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.592+0000 7f368effd640 1 -- 172.21.15.67:0/693221214 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3281734408 0 0) 0x7f36900d2250 con 0x7f36900b5a60 2024-03-24T00:44:35.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.592+0000 7f368f7fe640 1 --2- 172.21.15.67:0/693221214 >> v2:172.21.15.72:3300/0 conn(0x7f369015e280 0x7f36900d3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:44:35.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.592+0000 7f368ffff640 1 --2- 172.21.15.67:0/693221214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f369015dea0 0x7f36900d33d0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f36900d04d0 tx=0x7f368002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=626d70be74557e20 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:35.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.593+0000 7f368effd640 1 -- 172.21.15.67:0/693221214 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36800315a0 con 0x7f369015dea0 2024-03-24T00:44:35.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.593+0000 7f368effd640 1 -- 172.21.15.67:0/693221214 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3680037070 con 0x7f369015dea0 2024-03-24T00:44:35.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.593+0000 7f368effd640 1 -- 172.21.15.67:0/693221214 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3680031960 con 0x7f369015dea0 2024-03-24T00:44:35.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.593+0000 7f36969da640 1 -- 172.21.15.67:0/693221214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f369015dea0 msgr2=0x7f36900d33d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:35.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.593+0000 7f36969da640 1 --2- 172.21.15.67:0/693221214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f369015dea0 0x7f36900d33d0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f36900d04d0 tx=0x7f368002b840 comp rx=0 tx=0).stop 2024-03-24T00:44:35.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.593+0000 7f36969da640 1 -- 172.21.15.67:0/693221214 shutdown_connections 2024-03-24T00:44:35.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.593+0000 7f36969da640 1 --2- 172.21.15.67:0/693221214 >> v2:172.21.15.72:3300/0 conn(0x7f369015e280 0x7f36900d3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.593+0000 7f36969da640 1 --2- 172.21.15.67:0/693221214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f369015dea0 0x7f36900d33d0 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.593+0000 7f36969da640 1 -- 172.21.15.67:0/693221214 >> 172.21.15.67:0/693221214 conn(0x7f3690072da0 msgr2=0x7f36900d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:35.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.593+0000 7f36969da640 1 -- 172.21.15.67:0/693221214 shutdown_connections 2024-03-24T00:44:35.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.593+0000 7f36969da640 1 -- 172.21.15.67:0/693221214 wait complete. 2024-03-24T00:44:35.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.594+0000 7f36969da640 1 Processor -- start 2024-03-24T00:44:35.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.594+0000 7f36969da640 1 -- start start 2024-03-24T00:44:35.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.595+0000 7f36969da640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f36900b5a60 0x7f36900b3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.595+0000 7f36969da640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f369015dea0 0x7f36900cc230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.595+0000 7f36969da640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f369015e280 0x7f36900cc770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.595+0000 7f36969da640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36900b5eb0 con 0x7f369015e280 2024-03-24T00:44:35.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.595+0000 7f36969da640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36900b8310 con 0x7f369015dea0 2024-03-24T00:44:35.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.595+0000 7f36969da640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36900b6030 con 0x7f36900b5a60 2024-03-24T00:44:35.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.595+0000 7f3694f77640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f369015e280 0x7f36900cc770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.595+0000 7f3694f77640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f369015e280 0x7f36900cc770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55230/0 (socket says 172.21.15.67:55230) 2024-03-24T00:44:35.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.595+0000 7f3694f77640 1 -- 172.21.15.67:0/431319648 learned_addr learned my addr 172.21.15.67:0/431319648 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:35.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.595+0000 7f368f7fe640 1 --2- 172.21.15.67:0/431319648 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f369015dea0 0x7f36900cc230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.595+0000 7f368ffff640 1 --2- 172.21.15.67:0/431319648 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f36900b5a60 0x7f36900b3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.596+0000 7f3694f77640 1 -- 172.21.15.67:0/431319648 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f36900b5a60 msgr2=0x7f36900b3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:35.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.596+0000 7f3694f77640 1 --2- 172.21.15.67:0/431319648 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f36900b5a60 0x7f36900b3d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.596+0000 7f3694f77640 1 -- 172.21.15.67:0/431319648 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f369015dea0 msgr2=0x7f36900cc230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:35.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.596+0000 7f3694f77640 1 --2- 172.21.15.67:0/431319648 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f369015dea0 0x7f36900cc230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.596+0000 7f3694f77640 1 -- 172.21.15.67:0/431319648 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36900cceb0 con 0x7f369015e280 2024-03-24T00:44:35.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.596+0000 7f368ffff640 1 --2- 172.21.15.67:0/431319648 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f36900b5a60 0x7f36900b3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:44:35.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.596+0000 7f3694f77640 1 --2- 172.21.15.67:0/431319648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f369015e280 0x7f36900cc770 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f368800c800 tx=0x7f368800ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:35.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.596+0000 7f368cff9640 1 -- 172.21.15.67:0/431319648 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3688018070 con 0x7f369015e280 2024-03-24T00:44:35.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.596+0000 7f368cff9640 1 -- 172.21.15.67:0/431319648 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3688004d80 con 0x7f369015e280 2024-03-24T00:44:35.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.596+0000 7f36969da640 1 -- 172.21.15.67:0/431319648 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36900cd190 con 0x7f369015e280 2024-03-24T00:44:35.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.596+0000 7f368cff9640 1 -- 172.21.15.67:0/431319648 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f368800a820 con 0x7f369015e280 2024-03-24T00:44:35.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.596+0000 7f36969da640 1 -- 172.21.15.67:0/431319648 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36900ab1a0 con 0x7f369015e280 2024-03-24T00:44:35.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.598+0000 7f368cff9640 1 -- 172.21.15.67:0/431319648 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f368801b070 con 0x7f369015e280 2024-03-24T00:44:35.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.598+0000 7f368cff9640 1 --2- 172.21.15.67:0/431319648 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f367c06bbd0 0x7f367c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.598+0000 7f368cff9640 1 -- 172.21.15.67:0/431319648 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f368808a180 con 0x7f369015e280 2024-03-24T00:44:35.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.599+0000 7f368ffff640 1 --2- 172.21.15.67:0/431319648 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f367c06bbd0 0x7f367c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.599+0000 7f36995bd740 1 Processor -- start 2024-03-24T00:44:35.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.599+0000 7f36995bd740 1 -- start start 2024-03-24T00:44:35.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.599+0000 7f368ffff640 1 --2- 172.21.15.67:0/431319648 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f367c06bbd0 0x7f367c06e090 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f3680036000 tx=0x7f3680008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:35.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.599+0000 7f36995bd740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561f6b77b0d0 0x561f6b74e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.600+0000 7f36995bd740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561f6b77b4b0 0x561f6b77de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.600+0000 7f36995bd740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x561f6b74e840 0x561f6b77e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.600+0000 7f36995bd740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f6b6b81f0 con 0x561f6b77b0d0 2024-03-24T00:44:35.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.600+0000 7f36995bd740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f6b656a80 con 0x561f6b77b4b0 2024-03-24T00:44:35.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.600+0000 7f36995bd740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x561f6b6b4290 con 0x561f6b74e840 2024-03-24T00:44:35.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.600+0000 7f368ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561f6b77b0d0 0x561f6b74e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.600+0000 7f368ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561f6b77b0d0 0x561f6b74e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55236/0 (socket says 172.21.15.67:55236) 2024-03-24T00:44:35.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.600+0000 7f368ffff640 1 -- 172.21.15.67:0/3309919727 learned_addr learned my addr 172.21.15.67:0/3309919727 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:35.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.600+0000 7f368f7fe640 1 --2- 172.21.15.67:0/3309919727 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561f6b77b4b0 0x561f6b77de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.600+0000 7f3694f77640 1 --2- 172.21.15.67:0/3309919727 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x561f6b74e840 0x561f6b77e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.600+0000 7f368ffff640 1 -- 172.21.15.67:0/3309919727 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x561f6b74e840 msgr2=0x561f6b77e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:35.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.600+0000 7f368ffff640 1 --2- 172.21.15.67:0/3309919727 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x561f6b74e840 0x561f6b77e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.600+0000 7f368ffff640 1 -- 172.21.15.67:0/3309919727 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561f6b77b4b0 msgr2=0x561f6b77de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:35.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.600+0000 7f368ffff640 1 --2- 172.21.15.67:0/3309919727 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561f6b77b4b0 0x561f6b77de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.600+0000 7f368ffff640 1 -- 172.21.15.67:0/3309919727 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561f6b4987c0 con 0x561f6b77b0d0 2024-03-24T00:44:35.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.600+0000 7f368ffff640 1 --2- 172.21.15.67:0/3309919727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561f6b77b0d0 0x561f6b74e300 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f368002be50 tx=0x7f3680030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f943966fa3abc01 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:35.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.600+0000 7f3675ffb640 1 -- 172.21.15.67:0/3309919727 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3680032830 con 0x561f6b77b0d0 2024-03-24T00:44:35.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.600+0000 7f3675ffb640 1 -- 172.21.15.67:0/3309919727 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3680037040 con 0x561f6b77b0d0 2024-03-24T00:44:35.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.600+0000 7f3675ffb640 1 -- 172.21.15.67:0/3309919727 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3680032bd0 con 0x561f6b77b0d0 2024-03-24T00:44:35.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.601+0000 7f36995bd740 1 -- 172.21.15.67:0/3309919727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561f6b77b0d0 msgr2=0x561f6b74e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:35.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.601+0000 7f36995bd740 1 --2- 172.21.15.67:0/3309919727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561f6b77b0d0 0x561f6b74e300 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f368002be50 tx=0x7f3680030000 comp rx=0 tx=0).stop 2024-03-24T00:44:35.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.601+0000 7f36995bd740 1 -- 172.21.15.67:0/3309919727 shutdown_connections 2024-03-24T00:44:35.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.601+0000 7f36995bd740 1 --2- 172.21.15.67:0/3309919727 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x561f6b74e840 0x561f6b77e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.601+0000 7f36995bd740 1 --2- 172.21.15.67:0/3309919727 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561f6b77b4b0 0x561f6b77de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.601+0000 7f36995bd740 1 --2- 172.21.15.67:0/3309919727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561f6b77b0d0 0x561f6b74e300 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.601+0000 7f36995bd740 1 -- 172.21.15.67:0/3309919727 >> 172.21.15.67:0/3309919727 conn(0x561f6b750570 msgr2=0x561f6b798e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:35.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.601+0000 7f36995bd740 1 -- 172.21.15.67:0/3309919727 shutdown_connections 2024-03-24T00:44:35.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.601+0000 7f36995bd740 1 -- 172.21.15.67:0/3309919727 wait complete. 2024-03-24T00:44:35.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.601+0000 7f36995bd740 1 -- start start 2024-03-24T00:44:35.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.602+0000 7f36995bd740 1 Processor -- start 2024-03-24T00:44:35.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.602+0000 7f36995bd740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561f6b77b0d0 0x561f6b796d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.602+0000 7f36995bd740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561f6b77b4b0 0x561f6b7808b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.602+0000 7f36995bd740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x561f6b76c160 0x561f6b76c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.602+0000 7f36995bd740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f6b752880 con 0x561f6b77b4b0 2024-03-24T00:44:35.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.602+0000 7f36995bd740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f6b654410 con 0x561f6b77b0d0 2024-03-24T00:44:35.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.602+0000 7f36995bd740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x561f6b751c70 con 0x561f6b76c160 2024-03-24T00:44:35.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.603+0000 7f36995bd740 10 client.0 _ll_register_callbacks cb 0x561f6b67cd70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:44:35.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.603+0000 7f368f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561f6b77b4b0 0x561f6b7808b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.603+0000 7f368f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561f6b77b4b0 0x561f6b7808b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55246/0 (socket says 172.21.15.67:55246) 2024-03-24T00:44:35.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.603+0000 7f368f7fe640 1 -- 172.21.15.67:0/436966113 learned_addr learned my addr 172.21.15.67:0/436966113 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:35.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.603+0000 7f368ffff640 1 --2- 172.21.15.67:0/436966113 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561f6b77b0d0 0x561f6b796d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.603+0000 7f3694f77640 1 --2- 172.21.15.67:0/436966113 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x561f6b76c160 0x561f6b76c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.603+0000 7f368f7fe640 1 -- 172.21.15.67:0/436966113 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x561f6b76c160 msgr2=0x561f6b76c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:35.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.603+0000 7f368f7fe640 1 --2- 172.21.15.67:0/436966113 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x561f6b76c160 0x561f6b76c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.603+0000 7f368f7fe640 1 -- 172.21.15.67:0/436966113 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561f6b77b0d0 msgr2=0x561f6b796d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:35.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.603+0000 7f368f7fe640 1 --2- 172.21.15.67:0/436966113 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561f6b77b0d0 0x561f6b796d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.603+0000 7f368f7fe640 1 -- 172.21.15.67:0/436966113 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561f6b4f8f50 con 0x561f6b77b4b0 2024-03-24T00:44:35.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.603+0000 7f368f7fe640 1 --2- 172.21.15.67:0/436966113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561f6b77b4b0 0x561f6b7808b0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f3684010580 tx=0x7f3684010a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16f6e50ffa95bfd5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:35.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.603+0000 7f366effd640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:44:35.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.603+0000 7f366effd640 1 -- 172.21.15.67:0/436966113 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3684018070 con 0x561f6b77b4b0 2024-03-24T00:44:35.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.603+0000 7f366effd640 1 -- 172.21.15.67:0/436966113 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f3684014640 con 0x561f6b77b4b0 2024-03-24T00:44:35.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.603+0000 7f36995bd740 1 -- 172.21.15.67:0/436966113 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3680037040 con 0x561f6b77b4b0 2024-03-24T00:44:35.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.604+0000 7f366effd640 1 -- 172.21.15.67:0/436966113 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3684014860 con 0x561f6b77b4b0 2024-03-24T00:44:35.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.604+0000 7f36995bd740 10 client.7410 fetch_fsmap learned FSMap version 48 2024-03-24T00:44:35.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.604+0000 7f36995bd740 1 -- 172.21.15.67:0/436966113 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=2+}) v3 -- 0x561f6b38c0b0 con 0x561f6b77b4b0 2024-03-24T00:44:35.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.604+0000 7f366effd640 1 -- 172.21.15.67:0/436966113 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3680037040 con 0x561f6b77b4b0 2024-03-24T00:44:35.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.604+0000 7f366effd640 1 -- 172.21.15.67:0/436966113 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 48) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f3684026080 con 0x561f6b77b4b0 2024-03-24T00:44:35.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.604+0000 7f36995bd740 10 client.7410 fetch_fsmap finished waiting for FSMap version 48 2024-03-24T00:44:35.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.604+0000 7f36995bd740 10 client.7410 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T00:44:35.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.604+0000 7f36995bd740 20 client.7410 populate_metadata read hostname 'smithi067' 2024-03-24T00:44:35.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.604+0000 7f36995bd740 1 --2- 172.21.15.67:0/436966113 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x561f6b76aa70 0x561f6b76ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.604+0000 7f36995bd740 4 client.7410 mds_command: new command op to 7149 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:44:35.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.604+0000 7f36995bd740 1 -- 172.21.15.67:0/436966113 --> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561f6b6c5730 con 0x561f6b76aa70 2024-03-24T00:44:35.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.604+0000 7f368ffff640 1 --2- 172.21.15.67:0/436966113 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x561f6b76aa70 0x561f6b76ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.604+0000 7f366effd640 10 client.7410 ms_handle_connect on v2:172.21.15.67:6832/720256752 2024-03-24T00:44:35.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.604+0000 7f368ffff640 1 --2- 172.21.15.67:0/436966113 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x561f6b76aa70 0x561f6b76ae50 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=862184736c79b904 server_cookie=8c5fcb6298c68d81 in_seq=0 out_seq=0 2024-03-24T00:44:35.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.605+0000 7f366effd640 1 -- 172.21.15.67:0/436966113 <== mds.0 v2:172.21.15.67:6832/720256752 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561f6b6c5730 con 0x561f6b76aa70 2024-03-24T00:44:35.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.605+0000 7f366effd640 10 client.7410 handle_command_reply: tid=0 2024-03-24T00:44:35.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.605+0000 7f36995bd740 1 client.7410 shutdown 2024-03-24T00:44:35.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.605+0000 7f36995bd740 1 -- 172.21.15.67:0/436966113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561f6b77b4b0 msgr2=0x561f6b7808b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:35.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.605+0000 7f36995bd740 1 --2- 172.21.15.67:0/436966113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561f6b77b4b0 0x561f6b7808b0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f3684010580 tx=0x7f3684010a40 comp rx=0 tx=0).stop 2024-03-24T00:44:35.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.605+0000 7f36995bd740 1 -- 172.21.15.67:0/436966113 shutdown_connections 2024-03-24T00:44:35.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.605+0000 7f36995bd740 1 --2- 172.21.15.67:0/436966113 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x561f6b76c160 0x561f6b76c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.605+0000 7f36995bd740 1 --2- 172.21.15.67:0/436966113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561f6b77b4b0 0x561f6b7808b0 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.605+0000 7f36995bd740 1 --2- 172.21.15.67:0/436966113 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x561f6b76aa70 0x561f6b76ae50 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.605+0000 7f36995bd740 1 --2- 172.21.15.67:0/436966113 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561f6b77b0d0 0x561f6b796d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.605+0000 7f366effd640 0 client.7410 ms_handle_reset on v2:172.21.15.67:6832/720256752 2024-03-24T00:44:35.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.605+0000 7f36995bd740 1 -- 172.21.15.67:0/436966113 >> 172.21.15.67:0/436966113 conn(0x561f6b74e840 msgr2=0x561f6b792cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:35.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.605+0000 7f36995bd740 1 -- 172.21.15.67:0/436966113 shutdown_connections 2024-03-24T00:44:35.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.605+0000 7f36995bd740 1 -- 172.21.15.67:0/436966113 wait complete. 2024-03-24T00:44:35.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.605+0000 7f36995bd740 20 client.7410 trim_cache size 0 max 16384 2024-03-24T00:44:35.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.618+0000 7f36995bd740 1 Processor -- start 2024-03-24T00:44:35.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.618+0000 7f36995bd740 1 -- start start 2024-03-24T00:44:35.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.618+0000 7f36995bd740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561f6b76c160 0x561f6b795d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.618+0000 7f36995bd740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x561f6b77b0d0 0x561f6b79ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.618+0000 7f36995bd740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561f6b77b4b0 0x561f6b752020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.618+0000 7f36995bd740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36901ff860 con 0x561f6b77b4b0 2024-03-24T00:44:35.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.618+0000 7f36995bd740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36901f3bb0 con 0x561f6b76c160 2024-03-24T00:44:35.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.618+0000 7f36995bd740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36901ff260 con 0x561f6b77b0d0 2024-03-24T00:44:35.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.619+0000 7f3694f77640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561f6b77b4b0 0x561f6b752020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.619+0000 7f3694f77640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561f6b77b4b0 0x561f6b752020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55258/0 (socket says 172.21.15.67:55258) 2024-03-24T00:44:35.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.619+0000 7f3694f77640 1 -- 172.21.15.67:0/2245458457 learned_addr learned my addr 172.21.15.67:0/2245458457 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:35.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.619+0000 7f368ffff640 1 --2- 172.21.15.67:0/2245458457 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561f6b76c160 0x561f6b795d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.619+0000 7f368f7fe640 1 --2- 172.21.15.67:0/2245458457 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x561f6b77b0d0 0x561f6b79ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.619+0000 7f3694f77640 1 -- 172.21.15.67:0/2245458457 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x561f6b77b0d0 msgr2=0x561f6b79ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:35.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.619+0000 7f3694f77640 1 --2- 172.21.15.67:0/2245458457 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x561f6b77b0d0 0x561f6b79ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.619+0000 7f3694f77640 1 -- 172.21.15.67:0/2245458457 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561f6b76c160 msgr2=0x561f6b795d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:35.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.619+0000 7f3694f77640 1 --2- 172.21.15.67:0/2245458457 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561f6b76c160 0x561f6b795d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.619+0000 7f3694f77640 1 -- 172.21.15.67:0/2245458457 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561f6b798660 con 0x561f6b77b4b0 2024-03-24T00:44:35.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.619+0000 7f3694f77640 1 --2- 172.21.15.67:0/2245458457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561f6b77b4b0 0x561f6b752020 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f3688060630 tx=0x7f368808a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6501598063320ed9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:35.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.619+0000 7f36757fa640 1 -- 172.21.15.67:0/2245458457 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3688060b60 con 0x561f6b77b4b0 2024-03-24T00:44:35.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.619+0000 7f36757fa640 1 -- 172.21.15.67:0/2245458457 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3688056750 con 0x561f6b77b4b0 2024-03-24T00:44:35.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.619+0000 7f36757fa640 1 -- 172.21.15.67:0/2245458457 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f368805c140 con 0x561f6b77b4b0 2024-03-24T00:44:35.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.620+0000 7f36995bd740 1 -- 172.21.15.67:0/2245458457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561f6b77b4b0 msgr2=0x561f6b752020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:35.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.620+0000 7f36995bd740 1 --2- 172.21.15.67:0/2245458457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561f6b77b4b0 0x561f6b752020 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f3688060630 tx=0x7f368808a040 comp rx=0 tx=0).stop 2024-03-24T00:44:35.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.620+0000 7f36995bd740 1 -- 172.21.15.67:0/2245458457 shutdown_connections 2024-03-24T00:44:35.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.620+0000 7f36995bd740 1 --2- 172.21.15.67:0/2245458457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561f6b77b4b0 0x561f6b752020 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.620+0000 7f36995bd740 1 --2- 172.21.15.67:0/2245458457 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x561f6b77b0d0 0x561f6b79ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.620+0000 7f36995bd740 1 --2- 172.21.15.67:0/2245458457 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561f6b76c160 0x561f6b795d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.620+0000 7f36995bd740 1 -- 172.21.15.67:0/2245458457 >> 172.21.15.67:0/2245458457 conn(0x561f6b74e840 msgr2=0x561f6b7499d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:35.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.620+0000 7f36995bd740 1 -- 172.21.15.67:0/2245458457 shutdown_connections 2024-03-24T00:44:35.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.620+0000 7f36995bd740 1 -- 172.21.15.67:0/2245458457 wait complete. 2024-03-24T00:44:35.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.620+0000 7f36995bd740 1 -- start start 2024-03-24T00:44:35.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.620+0000 7f36995bd740 1 Processor -- start 2024-03-24T00:44:35.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.621+0000 7f36995bd740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561f6b76c160 0x561f6b77ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.621+0000 7f36995bd740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561f6b77b0d0 0x561f6b780580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.621+0000 7f36995bd740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x561f6b77b4b0 0x561f6b780ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.621+0000 7f36995bd740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f6b798cd0 con 0x561f6b76c160 2024-03-24T00:44:35.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.621+0000 7f36995bd740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f6b76a550 con 0x561f6b77b0d0 2024-03-24T00:44:35.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.621+0000 7f36995bd740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x561f6b7acef0 con 0x561f6b77b4b0 2024-03-24T00:44:35.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.621+0000 7f36995bd740 10 client.0 _ll_register_callbacks cb 0x7f368400e270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:44:35.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.621+0000 7f368ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561f6b76c160 0x561f6b77ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.621+0000 7f368ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561f6b76c160 0x561f6b77ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55272/0 (socket says 172.21.15.67:55272) 2024-03-24T00:44:35.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.621+0000 7f368ffff640 1 -- 172.21.15.67:0/785627945 learned_addr learned my addr 172.21.15.67:0/785627945 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:35.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.621+0000 7f3694f77640 1 --2- 172.21.15.67:0/785627945 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x561f6b77b4b0 0x561f6b780ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.621+0000 7f368f7fe640 1 --2- 172.21.15.67:0/785627945 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561f6b77b0d0 0x561f6b780580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.622+0000 7f368ffff640 1 -- 172.21.15.67:0/785627945 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x561f6b77b4b0 msgr2=0x561f6b780ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:35.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.622+0000 7f368ffff640 1 --2- 172.21.15.67:0/785627945 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x561f6b77b4b0 0x561f6b780ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.622+0000 7f368ffff640 1 -- 172.21.15.67:0/785627945 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561f6b77b0d0 msgr2=0x561f6b780580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:35.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.622+0000 7f368ffff640 1 --2- 172.21.15.67:0/785627945 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561f6b77b0d0 0x561f6b780580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.622+0000 7f368ffff640 1 -- 172.21.15.67:0/785627945 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561f6b795d40 con 0x561f6b76c160 2024-03-24T00:44:35.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.622+0000 7f3694f77640 1 --2- 172.21.15.67:0/785627945 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x561f6b77b4b0 0x561f6b780ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:44:35.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.622+0000 7f368f7fe640 1 --2- 172.21.15.67:0/785627945 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561f6b77b0d0 0x561f6b780580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:44:35.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.622+0000 7f368ffff640 1 --2- 172.21.15.67:0/785627945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561f6b76c160 0x561f6b77ec10 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f3680000c00 tx=0x7f368002e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5429de4ac0dbff9c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:35.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.622+0000 7f366effd640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:44:35.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.622+0000 7f366effd640 1 -- 172.21.15.67:0/785627945 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3680004470 con 0x561f6b76c160 2024-03-24T00:44:35.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.622+0000 7f366effd640 1 -- 172.21.15.67:0/785627945 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f368005b030 con 0x561f6b76c160 2024-03-24T00:44:35.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.622+0000 7f36995bd740 1 -- 172.21.15.67:0/785627945 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3688056750 con 0x561f6b76c160 2024-03-24T00:44:35.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.622+0000 7f366effd640 1 -- 172.21.15.67:0/785627945 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) v2 ==== 24+0+0 (secure 0 0 0) 0x7f368005b230 con 0x561f6b76c160 2024-03-24T00:44:35.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.622+0000 7f36995bd740 10 client.7416 fetch_fsmap learned FSMap version 48 2024-03-24T00:44:35.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.622+0000 7f36995bd740 1 -- 172.21.15.67:0/785627945 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=2+}) v3 -- 0x561f6b779e80 con 0x561f6b76c160 2024-03-24T00:44:35.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.622+0000 7f366effd640 1 -- 172.21.15.67:0/785627945 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3688056750 con 0x561f6b76c160 2024-03-24T00:44:35.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.622+0000 7f366effd640 1 -- 172.21.15.67:0/785627945 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 48) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f3680031070 con 0x561f6b76c160 2024-03-24T00:44:35.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.622+0000 7f36995bd740 10 client.7416 fetch_fsmap finished waiting for FSMap version 48 2024-03-24T00:44:35.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.622+0000 7f36995bd740 10 client.7416 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T00:44:35.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.622+0000 7f36995bd740 20 client.7416 populate_metadata read hostname 'smithi067' 2024-03-24T00:44:35.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.622+0000 7f36995bd740 1 --2- 172.21.15.67:0/785627945 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x561f6b80b960 0x561f6b79a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.622+0000 7f36995bd740 4 client.7416 mds_command: new command op to 7149 tid=0 multi_id=0 [{"prefix": "damage ls", "format": "json-pretty"}] 2024-03-24T00:44:35.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.622+0000 7f36995bd740 1 -- 172.21.15.67:0/785627945 --> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) v1 -- 0x561f6b6bfa80 con 0x561f6b80b960 2024-03-24T00:44:35.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.623+0000 7f368f7fe640 1 --2- 172.21.15.67:0/785627945 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x561f6b80b960 0x561f6b79a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.623+0000 7f366effd640 10 client.7416 ms_handle_connect on v2:172.21.15.67:6832/720256752 2024-03-24T00:44:35.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.623+0000 7f368f7fe640 1 --2- 172.21.15.67:0/785627945 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x561f6b80b960 0x561f6b79a0b0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e15d3ecbdc1a02a0 server_cookie=69d73883ba235552 in_seq=0 out_seq=0 2024-03-24T00:44:35.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.623+0000 7f366effd640 1 -- 172.21.15.67:0/785627945 <== mds.0 v2:172.21.15.67:6832/720256752 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x561f6b6bfa80 con 0x561f6b80b960 2024-03-24T00:44:35.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.623+0000 7f366effd640 10 client.7416 handle_command_reply: tid=0 2024-03-24T00:44:35.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.623+0000 7f36995bd740 1 client.7416 shutdown 2024-03-24T00:44:35.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.623+0000 7f36995bd740 1 -- 172.21.15.67:0/785627945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561f6b76c160 msgr2=0x561f6b77ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:35.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.623+0000 7f36995bd740 1 --2- 172.21.15.67:0/785627945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561f6b76c160 0x561f6b77ec10 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f3680000c00 tx=0x7f368002e000 comp rx=0 tx=0).stop 2024-03-24T00:44:35.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.623+0000 7f36995bd740 1 -- 172.21.15.67:0/785627945 shutdown_connections 2024-03-24T00:44:35.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.623+0000 7f36995bd740 1 --2- 172.21.15.67:0/785627945 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x561f6b77b4b0 0x561f6b780ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.623+0000 7f36995bd740 1 --2- 172.21.15.67:0/785627945 >> [v2:172.21.15.67:6832/720256752,v1:172.21.15.67:6833/720256752] conn(0x561f6b80b960 0x561f6b79a0b0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.623+0000 7f36995bd740 1 --2- 172.21.15.67:0/785627945 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561f6b77b0d0 0x561f6b780580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.623+0000 7f36995bd740 1 --2- 172.21.15.67:0/785627945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561f6b76c160 0x561f6b77ec10 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.623+0000 7f366effd640 0 client.7416 ms_handle_reset on v2:172.21.15.67:6832/720256752 2024-03-24T00:44:35.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.623+0000 7f36995bd740 1 -- 172.21.15.67:0/785627945 >> 172.21.15.67:0/785627945 conn(0x561f6b74e840 msgr2=0x561f6b77e220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:35.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.623+0000 7f36995bd740 1 -- 172.21.15.67:0/785627945 shutdown_connections 2024-03-24T00:44:35.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.623+0000 7f36995bd740 1 -- 172.21.15.67:0/785627945 wait complete. 2024-03-24T00:44:35.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.624+0000 7f36995bd740 20 client.7416 trim_cache size 0 max 16384 2024-03-24T00:44:35.631 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:44:35.631 INFO:teuthology.orchestra.run.smithi067.stdout:[] 2024-03-24T00:44:35.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.624+0000 7f36969da640 1 -- 172.21.15.67:0/431319648 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f367c06bbd0 msgr2=0x7f367c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:35.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.624+0000 7f36969da640 1 --2- 172.21.15.67:0/431319648 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f367c06bbd0 0x7f367c06e090 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f3680036000 tx=0x7f3680008030 comp rx=0 tx=0).stop 2024-03-24T00:44:35.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.624+0000 7f36969da640 1 -- 172.21.15.67:0/431319648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f369015e280 msgr2=0x7f36900cc770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:35.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.624+0000 7f36969da640 1 --2- 172.21.15.67:0/431319648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f369015e280 0x7f36900cc770 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f368800c800 tx=0x7f368800ccc0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.624+0000 7f36969da640 1 -- 172.21.15.67:0/431319648 shutdown_connections 2024-03-24T00:44:35.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.624+0000 7f36969da640 1 --2- 172.21.15.67:0/431319648 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f367c06bbd0 0x7f367c06e090 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.624+0000 7f36969da640 1 --2- 172.21.15.67:0/431319648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f369015e280 0x7f36900cc770 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.624+0000 7f36969da640 1 --2- 172.21.15.67:0/431319648 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f369015dea0 0x7f36900cc230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.624+0000 7f36969da640 1 --2- 172.21.15.67:0/431319648 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f36900b5a60 0x7f36900b3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.624+0000 7f36969da640 1 -- 172.21.15.67:0/431319648 >> 172.21.15.67:0/431319648 conn(0x7f3690072da0 msgr2=0x7f3690160700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:35.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.624+0000 7f36969da640 1 -- 172.21.15.67:0/431319648 shutdown_connections 2024-03-24T00:44:35.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.624+0000 7f36969da640 1 -- 172.21.15.67:0/431319648 wait complete. 2024-03-24T00:44:35.636 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-24T00:44:35.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.739+0000 7f6b149a9640 1 Processor -- start 2024-03-24T00:44:35.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.739+0000 7f6b149a9640 1 -- start start 2024-03-24T00:44:35.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.740+0000 7f6b149a9640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6b1015dec0 0x7f6b100ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.740+0000 7f6b149a9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6b1015e2a0 0x7f6b10162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.740+0000 7f6b149a9640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b100cf640 con 0x7f6b100d07e0 2024-03-24T00:44:35.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.740+0000 7f6b149a9640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b100cee30 con 0x7f6b1015e2a0 2024-03-24T00:44:35.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.740+0000 7f6b149a9640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b100d2120 con 0x7f6b1015dec0 2024-03-24T00:44:35.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.740+0000 7f6b0dd9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6b1015e2a0 0x7f6b10162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.740+0000 7f6b0dd9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6b1015e2a0 0x7f6b10162630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55274/0 (socket says 172.21.15.67:55274) 2024-03-24T00:44:35.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.740+0000 7f6b0dd9b640 1 -- 172.21.15.67:0/1069848235 learned_addr learned my addr 172.21.15.67:0/1069848235 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:35.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.740+0000 7f6b0e59c640 1 --2- 172.21.15.67:0/1069848235 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6b1015dec0 0x7f6b100ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.740+0000 7f6b0dd9b640 1 -- 172.21.15.67:0/1069848235 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6b1015dec0 msgr2=0x7f6b100ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:35.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.741+0000 7f6b0dd9b640 1 --2- 172.21.15.67:0/1069848235 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6b1015dec0 0x7f6b100ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.741+0000 7f6b0dd9b640 1 -- 172.21.15.67:0/1069848235 >> v1:172.21.15.72:6789/0 conn(0x7f6b100d07e0 legacy=0x7f6b100b5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:35.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.741+0000 7f6b0dd9b640 1 -- 172.21.15.67:0/1069848235 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b100cf080 con 0x7f6b1015e2a0 2024-03-24T00:44:35.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.741+0000 7f6b0dd9b640 1 --2- 172.21.15.67:0/1069848235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b1015e2a0 0x7f6b10162630 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f6b100cefb0 tx=0x7f6b0402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d457ff80693455d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:35.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.741+0000 7f6b0d59a640 1 -- 172.21.15.67:0/1069848235 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b04035020 con 0x7f6b1015e2a0 2024-03-24T00:44:35.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.741+0000 7f6b0d59a640 1 -- 172.21.15.67:0/1069848235 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b04031640 con 0x7f6b1015e2a0 2024-03-24T00:44:35.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.741+0000 7f6b0d59a640 1 -- 172.21.15.67:0/1069848235 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b0402d040 con 0x7f6b1015e2a0 2024-03-24T00:44:35.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.742+0000 7f6b149a9640 1 -- 172.21.15.67:0/1069848235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b1015e2a0 msgr2=0x7f6b10162630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:35.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.742+0000 7f6b149a9640 1 --2- 172.21.15.67:0/1069848235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b1015e2a0 0x7f6b10162630 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f6b100cefb0 tx=0x7f6b0402b8f0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.742+0000 7f6b149a9640 1 -- 172.21.15.67:0/1069848235 shutdown_connections 2024-03-24T00:44:35.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.742+0000 7f6b149a9640 1 --2- 172.21.15.67:0/1069848235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b1015e2a0 0x7f6b10162630 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.742+0000 7f6b149a9640 1 --2- 172.21.15.67:0/1069848235 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6b1015dec0 0x7f6b100ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.742+0000 7f6b149a9640 1 -- 172.21.15.67:0/1069848235 >> 172.21.15.67:0/1069848235 conn(0x7f6b1006c810 msgr2=0x7f6b1006cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:35.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.742+0000 7f6b149a9640 1 -- 172.21.15.67:0/1069848235 shutdown_connections 2024-03-24T00:44:35.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.742+0000 7f6b149a9640 1 -- 172.21.15.67:0/1069848235 wait complete. 2024-03-24T00:44:35.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.743+0000 7f6b149a9640 1 Processor -- start 2024-03-24T00:44:35.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.743+0000 7f6b149a9640 1 -- start start 2024-03-24T00:44:35.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.744+0000 7f6b149a9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b100d07e0 0x7f6b10157670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.744+0000 7f6b149a9640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6b1015dec0 0x7f6b100cb920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.744+0000 7f6b149a9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b1015e2a0 0x7f6b100b4820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.744+0000 7f6b149a9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b100d3740 con 0x7f6b1015e2a0 2024-03-24T00:44:35.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.744+0000 7f6b149a9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b100d35c0 con 0x7f6b100d07e0 2024-03-24T00:44:35.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.744+0000 7f6b149a9640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b100d38c0 con 0x7f6b1015dec0 2024-03-24T00:44:35.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.744+0000 7f6b0ed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b1015e2a0 0x7f6b100b4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.744+0000 7f6b0ed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b1015e2a0 0x7f6b100b4820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55282/0 (socket says 172.21.15.67:55282) 2024-03-24T00:44:35.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.744+0000 7f6b0ed9d640 1 -- 172.21.15.67:0/3760049323 learned_addr learned my addr 172.21.15.67:0/3760049323 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:35.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.744+0000 7f6b0dd9b640 1 --2- 172.21.15.67:0/3760049323 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6b1015dec0 0x7f6b100cb920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.744+0000 7f6b0e59c640 1 --2- 172.21.15.67:0/3760049323 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b100d07e0 0x7f6b10157670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.745+0000 7f6b0ed9d640 1 -- 172.21.15.67:0/3760049323 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6b1015dec0 msgr2=0x7f6b100cb920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:35.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.745+0000 7f6b0ed9d640 1 --2- 172.21.15.67:0/3760049323 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6b1015dec0 0x7f6b100cb920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.745+0000 7f6b0ed9d640 1 -- 172.21.15.67:0/3760049323 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b100d07e0 msgr2=0x7f6b10157670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:35.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.745+0000 7f6b0ed9d640 1 --2- 172.21.15.67:0/3760049323 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b100d07e0 0x7f6b10157670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.745+0000 7f6b0ed9d640 1 -- 172.21.15.67:0/3760049323 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b100b3a60 con 0x7f6b1015e2a0 2024-03-24T00:44:35.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.745+0000 7f6b0ed9d640 1 --2- 172.21.15.67:0/3760049323 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b1015e2a0 0x7f6b100b4820 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f6b0800c400 tx=0x7f6b0800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:35.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.745+0000 7f6afeffd640 1 -- 172.21.15.67:0/3760049323 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b08017070 con 0x7f6b1015e2a0 2024-03-24T00:44:35.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.745+0000 7f6afeffd640 1 -- 172.21.15.67:0/3760049323 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b080052d0 con 0x7f6b1015e2a0 2024-03-24T00:44:35.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.745+0000 7f6b149a9640 1 -- 172.21.15.67:0/3760049323 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b100b3d40 con 0x7f6b1015e2a0 2024-03-24T00:44:35.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.745+0000 7f6afeffd640 1 -- 172.21.15.67:0/3760049323 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b08003810 con 0x7f6b1015e2a0 2024-03-24T00:44:35.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.745+0000 7f6b149a9640 1 -- 172.21.15.67:0/3760049323 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b10157bb0 con 0x7f6b1015e2a0 2024-03-24T00:44:35.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.746+0000 7f6b149a9640 1 -- 172.21.15.67:0/3760049323 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b1004f400 con 0x7f6b1015e2a0 2024-03-24T00:44:35.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.746+0000 7f6afeffd640 1 -- 172.21.15.67:0/3760049323 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6b0801a070 con 0x7f6b1015e2a0 2024-03-24T00:44:35.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.747+0000 7f6afeffd640 1 --2- 172.21.15.67:0/3760049323 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6af006bc00 0x7f6af006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:35.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.747+0000 7f6afeffd640 1 -- 172.21.15.67:0/3760049323 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f6b08089fe0 con 0x7f6b1015e2a0 2024-03-24T00:44:35.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.747+0000 7f6b0e59c640 1 --2- 172.21.15.67:0/3760049323 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6af006bc00 0x7f6af006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:35.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.748+0000 7f6b0e59c640 1 --2- 172.21.15.67:0/3760049323 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6af006bc00 0x7f6af006e0c0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f6b00009a60 tx=0x7f6b00003990 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:35.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.752+0000 7f6afeffd640 1 -- 172.21.15.67:0/3760049323 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6b100cf640 con 0x7f6b1015e2a0 2024-03-24T00:44:35.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.974+0000 7f6b149a9640 1 -- 172.21.15.67:0/3760049323 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6b100d1aa0 con 0x7f6b1015e2a0 2024-03-24T00:44:35.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.975+0000 7f6afeffd640 1 -- 172.21.15.67:0/3760049323 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f6b080588f0 con 0x7f6b1015e2a0 2024-03-24T00:44:35.976 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:44:35.976 INFO:teuthology.orchestra.run.smithi067.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-24T00:44:35.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.977+0000 7f6b149a9640 1 -- 172.21.15.67:0/3760049323 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6af006bc00 msgr2=0x7f6af006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:35.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.977+0000 7f6b149a9640 1 --2- 172.21.15.67:0/3760049323 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6af006bc00 0x7f6af006e0c0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f6b00009a60 tx=0x7f6b00003990 comp rx=0 tx=0).stop 2024-03-24T00:44:35.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.977+0000 7f6b149a9640 1 -- 172.21.15.67:0/3760049323 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b1015e2a0 msgr2=0x7f6b100b4820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:35.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.977+0000 7f6b149a9640 1 --2- 172.21.15.67:0/3760049323 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b1015e2a0 0x7f6b100b4820 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f6b0800c400 tx=0x7f6b0800c8c0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.977+0000 7f6b149a9640 1 -- 172.21.15.67:0/3760049323 shutdown_connections 2024-03-24T00:44:35.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.977+0000 7f6b149a9640 1 --2- 172.21.15.67:0/3760049323 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6af006bc00 0x7f6af006e0c0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.977+0000 7f6b149a9640 1 --2- 172.21.15.67:0/3760049323 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b1015e2a0 0x7f6b100b4820 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.977+0000 7f6b149a9640 1 --2- 172.21.15.67:0/3760049323 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6b1015dec0 0x7f6b100cb920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.977+0000 7f6b149a9640 1 --2- 172.21.15.67:0/3760049323 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b100d07e0 0x7f6b10157670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:35.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.977+0000 7f6b149a9640 1 -- 172.21.15.67:0/3760049323 >> 172.21.15.67:0/3760049323 conn(0x7f6b1006c810 msgr2=0x7f6b100d4630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:35.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.977+0000 7f6b149a9640 1 -- 172.21.15.67:0/3760049323 shutdown_connections 2024-03-24T00:44:35.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:35.977+0000 7f6b149a9640 1 -- 172.21.15.67:0/3760049323 wait complete. 2024-03-24T00:44:35.992 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-03-24T00:44:35.992 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T00:44:36.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.107+0000 7f7182576640 1 Processor -- start 2024-03-24T00:44:36.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.107+0000 7f7182576640 1 -- start start 2024-03-24T00:44:36.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.109+0000 7f7182576640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f717c0b2170 0x7f717c0cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:36.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.109+0000 7f7182576640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f717c143100 0x7f717c143590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:36.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.109+0000 7f7182576640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f717c15c230 con 0x7f717c06b040 2024-03-24T00:44:36.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.109+0000 7f7182576640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f717c0ce660 con 0x7f717c143100 2024-03-24T00:44:36.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.109+0000 7f7182576640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f717c0cddc0 con 0x7f717c0b2170 2024-03-24T00:44:36.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.109+0000 7f7180b13640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f717c143100 0x7f717c143590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:36.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.109+0000 7f717bfff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f717c06b040 0x7f717c06b420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47800/0 (socket says 172.21.15.67:47800) 2024-03-24T00:44:36.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.109+0000 7f717bfff640 1 -- 172.21.15.67:0/2712939053 learned_addr learned my addr 172.21.15.67:0/2712939053 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:44:36.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.109+0000 7f717b7fe640 1 --2- 172.21.15.67:0/2712939053 >> v2:172.21.15.72:3300/0 conn(0x7f717c0b2170 0x7f717c0cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:36.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.109+0000 7f7180b13640 1 -- 172.21.15.67:0/2712939053 >> v2:172.21.15.72:3300/0 conn(0x7f717c0b2170 msgr2=0x7f717c0cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:36.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.109+0000 7f7180b13640 1 --2- 172.21.15.67:0/2712939053 >> v2:172.21.15.72:3300/0 conn(0x7f717c0b2170 0x7f717c0cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:36.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.109+0000 7f7180b13640 1 -- 172.21.15.67:0/2712939053 >> v1:172.21.15.67:6789/0 conn(0x7f717c06b040 legacy=0x7f717c06b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:36.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.109+0000 7f7180b13640 1 -- 172.21.15.67:0/2712939053 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f717c0ce0a0 con 0x7f717c143100 2024-03-24T00:44:36.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.109+0000 7f717affd640 1 -- 172.21.15.67:0/2712939053 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 868495933 0 0) 0x7f717c15c230 con 0x7f717c06b040 2024-03-24T00:44:36.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.110+0000 7f7180b13640 1 --2- 172.21.15.67:0/2712939053 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f717c143100 0x7f717c143590 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f717c0cbb80 tx=0x7f717402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73b800d71ba00217 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:36.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.110+0000 7f717affd640 1 -- 172.21.15.67:0/2712939053 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7174039070 con 0x7f717c143100 2024-03-24T00:44:36.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.110+0000 7f717affd640 1 -- 172.21.15.67:0/2712939053 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f717402cbf0 con 0x7f717c143100 2024-03-24T00:44:36.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.110+0000 7f717affd640 1 -- 172.21.15.67:0/2712939053 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7174034440 con 0x7f717c143100 2024-03-24T00:44:36.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.110+0000 7f7182576640 1 -- 172.21.15.67:0/2712939053 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f717c143100 msgr2=0x7f717c143590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:36.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.110+0000 7f7182576640 1 --2- 172.21.15.67:0/2712939053 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f717c143100 0x7f717c143590 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f717c0cbb80 tx=0x7f717402be40 comp rx=0 tx=0).stop 2024-03-24T00:44:36.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.110+0000 7f7182576640 1 -- 172.21.15.67:0/2712939053 shutdown_connections 2024-03-24T00:44:36.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.110+0000 7f7182576640 1 --2- 172.21.15.67:0/2712939053 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f717c143100 0x7f717c143590 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:36.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.110+0000 7f7182576640 1 --2- 172.21.15.67:0/2712939053 >> v2:172.21.15.72:3300/0 conn(0x7f717c0b2170 0x7f717c0cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:36.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.110+0000 7f7182576640 1 -- 172.21.15.67:0/2712939053 >> 172.21.15.67:0/2712939053 conn(0x7f717c099a30 msgr2=0x7f717c099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:36.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.111+0000 7f7182576640 1 -- 172.21.15.67:0/2712939053 shutdown_connections 2024-03-24T00:44:36.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.111+0000 7f7182576640 1 -- 172.21.15.67:0/2712939053 wait complete. 2024-03-24T00:44:36.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.112+0000 7f7182576640 1 Processor -- start 2024-03-24T00:44:36.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.113+0000 7f7182576640 1 -- start start 2024-03-24T00:44:36.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.114+0000 7f7182576640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f717c06b040 0x7f717c0c6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:36.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.114+0000 7f7182576640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f717c0b2170 0x7f717c0c7200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:36.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.114+0000 7f7182576640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f717c143100 0x7f717c06f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:36.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.114+0000 7f7182576640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f717c0cd9a0 con 0x7f717c0b2170 2024-03-24T00:44:36.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.114+0000 7f7182576640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f717c142be0 con 0x7f717c06b040 2024-03-24T00:44:36.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.114+0000 7f7182576640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f717c0cdb20 con 0x7f717c143100 2024-03-24T00:44:36.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.114+0000 7f717b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f717c0b2170 0x7f717c0c7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:36.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.114+0000 7f717b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f717c0b2170 0x7f717c0c7200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55290/0 (socket says 172.21.15.67:55290) 2024-03-24T00:44:36.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.114+0000 7f717b7fe640 1 -- 172.21.15.67:0/4135827550 learned_addr learned my addr 172.21.15.67:0/4135827550 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:36.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.114+0000 7f717bfff640 1 --2- 172.21.15.67:0/4135827550 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f717c06b040 0x7f717c0c6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:36.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.114+0000 7f7180b13640 1 --2- 172.21.15.67:0/4135827550 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f717c143100 0x7f717c06f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:36.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.114+0000 7f717b7fe640 1 -- 172.21.15.67:0/4135827550 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f717c143100 msgr2=0x7f717c06f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:36.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.114+0000 7f717b7fe640 1 --2- 172.21.15.67:0/4135827550 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f717c143100 0x7f717c06f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:36.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.114+0000 7f717b7fe640 1 -- 172.21.15.67:0/4135827550 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f717c06b040 msgr2=0x7f717c0c6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:36.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.114+0000 7f717b7fe640 1 --2- 172.21.15.67:0/4135827550 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f717c06b040 0x7f717c0c6cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:36.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.114+0000 7f717b7fe640 1 -- 172.21.15.67:0/4135827550 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f717c15c360 con 0x7f717c0b2170 2024-03-24T00:44:36.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.114+0000 7f717b7fe640 1 --2- 172.21.15.67:0/4135827550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f717c0b2170 0x7f717c0c7200 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f717000d620 tx=0x7f717000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:36.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.114+0000 7f7178ff9640 1 -- 172.21.15.67:0/4135827550 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f717000b620 con 0x7f717c0b2170 2024-03-24T00:44:36.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.114+0000 7f7178ff9640 1 -- 172.21.15.67:0/4135827550 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7170019070 con 0x7f717c0b2170 2024-03-24T00:44:36.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.114+0000 7f7182576640 1 -- 172.21.15.67:0/4135827550 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f717c15c640 con 0x7f717c0b2170 2024-03-24T00:44:36.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.114+0000 7f7178ff9640 1 -- 172.21.15.67:0/4135827550 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f717000ba00 con 0x7f717c0b2170 2024-03-24T00:44:36.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.115+0000 7f7182576640 1 -- 172.21.15.67:0/4135827550 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f717c15cc00 con 0x7f717c0b2170 2024-03-24T00:44:36.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.115+0000 7f7182576640 1 -- 172.21.15.67:0/4135827550 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f717c04f400 con 0x7f717c0b2170 2024-03-24T00:44:36.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.116+0000 7f7178ff9640 1 -- 172.21.15.67:0/4135827550 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f717000bba0 con 0x7f717c0b2170 2024-03-24T00:44:36.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.116+0000 7f7178ff9640 1 --2- 172.21.15.67:0/4135827550 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f716806bbd0 0x7f716806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:36.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.116+0000 7f7178ff9640 1 -- 172.21.15.67:0/4135827550 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f717008afe0 con 0x7f717c0b2170 2024-03-24T00:44:36.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.116+0000 7f717bfff640 1 --2- 172.21.15.67:0/4135827550 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f716806bbd0 0x7f716806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:36.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.117+0000 7f717bfff640 1 --2- 172.21.15.67:0/4135827550 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f716806bbd0 0x7f716806e090 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f716c000f60 tx=0x7f716c007c00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:36.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.121+0000 7f7178ff9640 1 -- 172.21.15.67:0/4135827550 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7170059980 con 0x7f717c0b2170 2024-03-24T00:44:36.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.342+0000 7f7182576640 1 -- 172.21.15.67:0/4135827550 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f717c09a290 con 0x7f717c0b2170 2024-03-24T00:44:36.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.342+0000 7f7178ff9640 1 -- 172.21.15.67:0/4135827550 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v48) v1 ==== 53+0+73 (secure 0 0 0) 0x7f7170059280 con 0x7f717c0b2170 2024-03-24T00:44:36.343 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-24T00:44:36.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.344+0000 7f7182576640 1 -- 172.21.15.67:0/4135827550 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f716806bbd0 msgr2=0x7f716806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:36.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.344+0000 7f7182576640 1 --2- 172.21.15.67:0/4135827550 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f716806bbd0 0x7f716806e090 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f716c000f60 tx=0x7f716c007c00 comp rx=0 tx=0).stop 2024-03-24T00:44:36.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.344+0000 7f7182576640 1 -- 172.21.15.67:0/4135827550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f717c0b2170 msgr2=0x7f717c0c7200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:36.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.344+0000 7f7182576640 1 --2- 172.21.15.67:0/4135827550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f717c0b2170 0x7f717c0c7200 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f717000d620 tx=0x7f717000dae0 comp rx=0 tx=0).stop 2024-03-24T00:44:36.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.345+0000 7f7182576640 1 -- 172.21.15.67:0/4135827550 shutdown_connections 2024-03-24T00:44:36.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.345+0000 7f7182576640 1 --2- 172.21.15.67:0/4135827550 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f716806bbd0 0x7f716806e090 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:36.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.345+0000 7f7182576640 1 --2- 172.21.15.67:0/4135827550 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f717c143100 0x7f717c06f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:36.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.345+0000 7f7182576640 1 --2- 172.21.15.67:0/4135827550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f717c0b2170 0x7f717c0c7200 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:36.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.345+0000 7f7182576640 1 --2- 172.21.15.67:0/4135827550 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f717c06b040 0x7f717c0c6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:36.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.345+0000 7f7182576640 1 -- 172.21.15.67:0/4135827550 >> 172.21.15.67:0/4135827550 conn(0x7f717c099a30 msgr2=0x7f717c144540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:36.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.345+0000 7f7182576640 1 -- 172.21.15.67:0/4135827550 shutdown_connections 2024-03-24T00:44:36.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:36.345+0000 7f7182576640 1 -- 172.21.15.67:0/4135827550 wait complete. 2024-03-24T00:44:36.356 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-24T00:44:36.356 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T00:44:36.356 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-03-24T00:44:36.356 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-24T00:44:36.356 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T00:44:36.357 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-24T00:44:36.357 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-24T00:44:36.357 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T00:44:36.357 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:44:36.357 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:44:36.357 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-03-24T00:44:36.372 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T00:44:36.372 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T00:44:36.372 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T00:44:36.373 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:44:36.373 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-03-24T00:44:36.373 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:44:36.373 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:44:36.373 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-03-24T00:44:36.373 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-03-24T00:44:36.373 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:44:36.373 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-03-24T00:44:36.373 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-03-24T00:44:36.373 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:44:36.373 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-03-24T00:44:36.373 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T00:44:36.373 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-03-24T00:44:36.373 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-24T00:44:36.374 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 42237sec preferred_lft 42237sec 2024-03-24T00:44:36.374 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-03-24T00:44:36.374 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:44:36.374 INFO:teuthology.orchestra.run.smithi067.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T00:44:36.374 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 96:cf:26:c8:bf:62 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:44:36.374 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-24T00:44:36.374 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:44:36.374 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::7c6a:5fff:feaa:d41/64 scope link 2024-03-24T00:44:36.374 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:44:36.374 INFO:teuthology.orchestra.run.smithi067.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T00:44:36.374 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ea:22:4b:d5:48:3d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:44:36.374 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::e822:4bff:fed5:483d/64 scope link 2024-03-24T00:44:36.374 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:44:36.374 INFO:teuthology.orchestra.run.smithi067.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T00:44:36.375 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 96:cf:26:c8:bf:62 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:44:36.375 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::94cf:26ff:fec8:bf62/64 scope link 2024-03-24T00:44:36.375 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:44:36.375 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:44:36.375 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T00:44:36.438 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-03-24T00:44:36.440 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:44:36.440 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-03-24T00:44:36.495 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-03-24T00:44:36.495 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T00:44:36.495 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-24T00:44:36.495 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-24T00:44:36.495 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T00:44:36.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:44:36.496 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-03-24T00:44:36.496 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-03-24T00:44:36.496 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-24T00:44:36.496 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-24T00:44:36.496 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-24T00:44:36.496 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-03-24T00:44:36.593 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:44:36.593 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-03-24T00:44:36.607 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-24T00:44:36.608 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-24T00:44:36.608 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:44:36.608 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:44:36.608 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-03-24T00:44:36.673 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:44:36.674 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-24T00:44:36.728 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:44:36.729 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:44:36.729 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:44:36.783 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-24T00:44:36.783 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:44:36.783 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:44:36.867 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:44:36.867 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T00:44:36.867 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-03-24T00:44:36.875 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:44:36.875 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T00:44:36.875 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-03-24T00:44:36.875 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.72:3300,172.21.15.72:3301" 2024-03-24T00:44:36.875 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-24T00:44:36.880 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:44:36.880 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T00:44:36.880 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-03-24T00:44:36.880 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.72:3300,172.21.15.72:3301" 2024-03-24T00:44:36.880 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-24T00:44:36.880 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:6789,172.21.15.72:6789,172.21.15.72:6790" 2024-03-24T00:44:36.880 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.67:6789,172.21.15.72:6789,172.21.15.72:6790:/ 2024-03-24T00:44:36.880 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=de1d0178-808e-49ca-a986-e297e523b976" will pass to kernel 2024-03-24T00:44:36.884 INFO:teuthology.orchestra.run.smithi067.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-24T00:44:36.884 INFO:teuthology.orchestra.run.smithi067.stdout: You just mounted a file system that supports labels which does not 2024-03-24T00:44:36.884 INFO:teuthology.orchestra.run.smithi067.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-24T00:44:36.884 INFO:teuthology.orchestra.run.smithi067.stdout: applications will generate AVC messages and not be allowed access to 2024-03-24T00:44:36.884 INFO:teuthology.orchestra.run.smithi067.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-24T00:44:36.884 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-24T00:44:36.884 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:44:36.884 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:44:36.915 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:44:36.915 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T00:44:36.971 INFO:teuthology.orchestra.run.smithi067.stdout:client7434 2024-03-24T00:44:36.971 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:44:36.971 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:44:37.026 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:44:37.027 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.files ./subdir) 2024-03-24T00:44:37.082 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-03-24T00:44:37.082 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata setomapval 10000000000.00000000 file_to_be_damaged_head deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 2024-03-24T00:44:37.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.165+0000 7ff08c435f40 1 Processor -- start 2024-03-24T00:44:37.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.165+0000 7ff08c435f40 1 -- start start 2024-03-24T00:44:37.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.165+0000 7ff08c435f40 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5555fa242800 0x5555fa26e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:37.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.165+0000 7ff08c435f40 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5555fa26dfc0 0x5555fa26fb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:37.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.165+0000 7ff08c435f40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5555fa31ef40 0x5555fa2fe910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:37.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.165+0000 7ff08c435f40 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555fa301160 con 0x5555fa242800 2024-03-24T00:44:37.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.165+0000 7ff08c435f40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555fa1eb0d0 con 0x5555fa31ef40 2024-03-24T00:44:37.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.165+0000 7ff08c435f40 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5555fa3012e0 con 0x5555fa26dfc0 2024-03-24T00:44:37.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.165+0000 7ff0897fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5555fa242800 0x5555fa26e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:37.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.165+0000 7ff0897fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5555fa242800 0x5555fa26e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55306/0 (socket says 172.21.15.67:55306) 2024-03-24T00:44:37.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.165+0000 7ff0897fc640 1 -- 172.21.15.67:0/827744994 learned_addr learned my addr 172.21.15.67:0/827744994 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:37.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.166+0000 7ff083fff640 1 --2- 172.21.15.67:0/827744994 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5555fa26dfc0 0x5555fa26fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:37.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.166+0000 7ff089ffd640 1 --2- 172.21.15.67:0/827744994 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5555fa31ef40 0x5555fa2fe910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:37.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.166+0000 7ff0897fc640 1 -- 172.21.15.67:0/827744994 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5555fa26dfc0 msgr2=0x5555fa26fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:37.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.166+0000 7ff0897fc640 1 --2- 172.21.15.67:0/827744994 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5555fa26dfc0 0x5555fa26fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:37.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.166+0000 7ff0897fc640 1 -- 172.21.15.67:0/827744994 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5555fa31ef40 msgr2=0x5555fa2fe910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:37.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.166+0000 7ff0897fc640 1 --2- 172.21.15.67:0/827744994 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5555fa31ef40 0x5555fa2fe910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:37.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.166+0000 7ff0897fc640 1 -- 172.21.15.67:0/827744994 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5555fa28dac0 con 0x5555fa242800 2024-03-24T00:44:37.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.166+0000 7ff089ffd640 1 --2- 172.21.15.67:0/827744994 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5555fa31ef40 0x5555fa2fe910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:44:37.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.166+0000 7ff0897fc640 1 --2- 172.21.15.67:0/827744994 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5555fa242800 0x5555fa26e7d0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7ff07c009790 tx=0x7ff07c009c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=992874f1f98c0c99 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:37.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.166+0000 7ff0837fe640 1 -- 172.21.15.67:0/827744994 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07c017070 con 0x5555fa242800 2024-03-24T00:44:37.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.166+0000 7ff0837fe640 1 -- 172.21.15.67:0/827744994 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff07c0090d0 con 0x5555fa242800 2024-03-24T00:44:37.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.166+0000 7ff0837fe640 1 -- 172.21.15.67:0/827744994 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07c006b60 con 0x5555fa242800 2024-03-24T00:44:37.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.166+0000 7ff08c435f40 1 -- 172.21.15.67:0/827744994 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5555fa242800 msgr2=0x5555fa26e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:37.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.166+0000 7ff08c435f40 1 --2- 172.21.15.67:0/827744994 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5555fa242800 0x5555fa26e7d0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7ff07c009790 tx=0x7ff07c009c50 comp rx=0 tx=0).stop 2024-03-24T00:44:37.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.166+0000 7ff08c435f40 1 -- 172.21.15.67:0/827744994 shutdown_connections 2024-03-24T00:44:37.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.166+0000 7ff08c435f40 1 --2- 172.21.15.67:0/827744994 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5555fa31ef40 0x5555fa2fe910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:37.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.166+0000 7ff08c435f40 1 --2- 172.21.15.67:0/827744994 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5555fa26dfc0 0x5555fa26fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:37.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.166+0000 7ff08c435f40 1 --2- 172.21.15.67:0/827744994 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5555fa242800 0x5555fa26e7d0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:37.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.166+0000 7ff08c435f40 1 -- 172.21.15.67:0/827744994 >> 172.21.15.67:0/827744994 conn(0x5555fa243d50 msgr2=0x5555fa31abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:37.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.167+0000 7ff08c435f40 1 -- 172.21.15.67:0/827744994 shutdown_connections 2024-03-24T00:44:37.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.167+0000 7ff08c435f40 1 -- 172.21.15.67:0/827744994 wait complete. 2024-03-24T00:44:37.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.167+0000 7ff08c435f40 1 Processor -- start 2024-03-24T00:44:37.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.167+0000 7ff08c435f40 1 -- start start 2024-03-24T00:44:37.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.168+0000 7ff08c435f40 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5555fa242800 0x5555fa2fb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:37.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.168+0000 7ff08c435f40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5555fa26dfc0 0x5555fa31a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:37.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.168+0000 7ff08c435f40 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5555fa31ef40 0x5555fa2f8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:37.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.168+0000 7ff08c435f40 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555fa2fd460 con 0x5555fa31ef40 2024-03-24T00:44:37.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.168+0000 7ff08c435f40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555fa244460 con 0x5555fa26dfc0 2024-03-24T00:44:37.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.168+0000 7ff08c435f40 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5555fa273eb0 con 0x5555fa242800 2024-03-24T00:44:37.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.168+0000 7ff089ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5555fa31ef40 0x5555fa2f8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:37.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.168+0000 7ff089ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5555fa31ef40 0x5555fa2f8ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55316/0 (socket says 172.21.15.67:55316) 2024-03-24T00:44:37.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.168+0000 7ff089ffd640 1 -- 172.21.15.67:0/3585434655 learned_addr learned my addr 172.21.15.67:0/3585434655 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:37.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.168+0000 7ff083fff640 1 --2- 172.21.15.67:0/3585434655 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5555fa26dfc0 0x5555fa31a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:37.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.168+0000 7ff0897fc640 1 --2- 172.21.15.67:0/3585434655 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5555fa242800 0x5555fa2fb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:37.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.168+0000 7ff089ffd640 1 -- 172.21.15.67:0/3585434655 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5555fa242800 msgr2=0x5555fa2fb8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:37.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.168+0000 7ff089ffd640 1 --2- 172.21.15.67:0/3585434655 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5555fa242800 0x5555fa2fb8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:37.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.168+0000 7ff089ffd640 1 -- 172.21.15.67:0/3585434655 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5555fa26dfc0 msgr2=0x5555fa31a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:37.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.168+0000 7ff089ffd640 1 --2- 172.21.15.67:0/3585434655 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5555fa26dfc0 0x5555fa31a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:37.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.168+0000 7ff089ffd640 1 -- 172.21.15.67:0/3585434655 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5555fa2fa850 con 0x5555fa31ef40 2024-03-24T00:44:37.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.169+0000 7ff089ffd640 1 --2- 172.21.15.67:0/3585434655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5555fa31ef40 0x5555fa2f8ca0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7ff08400a190 tx=0x7ff084003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:37.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.169+0000 7ff0827fc640 1 -- 172.21.15.67:0/3585434655 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff084061070 con 0x5555fa31ef40 2024-03-24T00:44:37.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.169+0000 7ff0827fc640 1 -- 172.21.15.67:0/3585434655 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff084052e10 con 0x5555fa31ef40 2024-03-24T00:44:37.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.169+0000 7ff0827fc640 1 -- 172.21.15.67:0/3585434655 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff08405d630 con 0x5555fa31ef40 2024-03-24T00:44:37.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.169+0000 7ff08c435f40 1 -- 172.21.15.67:0/3585434655 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5555fa2fab30 con 0x5555fa31ef40 2024-03-24T00:44:37.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.169+0000 7ff08c435f40 1 -- 172.21.15.67:0/3585434655 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5555fa2fb0f0 con 0x5555fa31ef40 2024-03-24T00:44:37.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.171+0000 7ff0827fc640 1 -- 172.21.15.67:0/3585434655 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff084050070 con 0x5555fa31ef40 2024-03-24T00:44:37.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.171+0000 7ff0827fc640 1 --2- 172.21.15.67:0/3585434655 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff06c06a5e0 0x7ff06c06caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:37.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.171+0000 7ff0827fc640 1 -- 172.21.15.67:0/3585434655 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7ff0840cb340 con 0x5555fa31ef40 2024-03-24T00:44:37.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.171+0000 7ff0897fc640 1 --2- 172.21.15.67:0/3585434655 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff06c06a5e0 0x7ff06c06caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:37.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.171+0000 7ff08c435f40 1 --2- 172.21.15.67:0/3585434655 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x5555fa31aec0 0x5555fa28b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:37.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.171+0000 7ff08c435f40 1 -- 172.21.15.67:0/3585434655 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:1 11.10 11:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=115b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77) v9 -- 0x5555fa288290 con 0x5555fa31aec0 2024-03-24T00:44:37.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.171+0000 7ff083fff640 1 --2- 172.21.15.67:0/3585434655 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x5555fa31aec0 0x5555fa28b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:37.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.172+0000 7ff0897fc640 1 --2- 172.21.15.67:0/3585434655 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff06c06a5e0 0x7ff06c06caa0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7ff07c009790 tx=0x7ff07c009f90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:37.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.172+0000 7ff083fff640 1 --2- 172.21.15.67:0/3585434655 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x5555fa31aec0 0x5555fa28b630 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:37.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.173+0000 7ff083fff640 1 -- 172.21.15.67:0/3585434655 <== osd.0 v2:172.21.15.67:6808/2628130519 1 ==== osd_op_reply(1 10000000000.00000000 [omap-set-vals] v77'12 uv12 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7ff07400c870 con 0x5555fa31aec0 2024-03-24T00:44:37.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.173+0000 7ff08c435f40 1 -- 172.21.15.67:0/3585434655 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x5555fa31aec0 msgr2=0x5555fa28b630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:37.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.173+0000 7ff08c435f40 1 --2- 172.21.15.67:0/3585434655 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x5555fa31aec0 0x5555fa28b630 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:37.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.173+0000 7ff08c435f40 1 -- 172.21.15.67:0/3585434655 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff06c06a5e0 msgr2=0x7ff06c06caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:37.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.173+0000 7ff08c435f40 1 --2- 172.21.15.67:0/3585434655 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff06c06a5e0 0x7ff06c06caa0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7ff07c009790 tx=0x7ff07c009f90 comp rx=0 tx=0).stop 2024-03-24T00:44:37.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.173+0000 7ff08c435f40 1 -- 172.21.15.67:0/3585434655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5555fa31ef40 msgr2=0x5555fa2f8ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:37.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.173+0000 7ff08c435f40 1 --2- 172.21.15.67:0/3585434655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5555fa31ef40 0x5555fa2f8ca0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7ff08400a190 tx=0x7ff084003330 comp rx=0 tx=0).stop 2024-03-24T00:44:37.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.173+0000 7ff089ffd640 1 -- 172.21.15.67:0/3585434655 reap_dead start 2024-03-24T00:44:37.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.173+0000 7ff08c435f40 1 -- 172.21.15.67:0/3585434655 shutdown_connections 2024-03-24T00:44:37.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.173+0000 7ff08c435f40 1 -- 172.21.15.67:0/3585434655 >> 172.21.15.67:0/3585434655 conn(0x5555fa243d50 msgr2=0x5555fa2fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:37.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.173+0000 7ff08c435f40 1 -- 172.21.15.67:0/3585434655 shutdown_connections 2024-03-24T00:44:37.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.173+0000 7ff08c435f40 1 -- 172.21.15.67:0/3585434655 wait complete. 2024-03-24T00:44:37.179 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-24T00:44:37.179 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:44:37.261 DEBUG:teuthology.orchestra.run.smithi072:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:44:37.312 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-24T00:44:37.312 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:44:37.312 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:44:37.327 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:44:37.327 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:44:37.382 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:44:37.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.562+0000 7f5155844640 1 Processor -- start 2024-03-24T00:44:37.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.562+0000 7f5155844640 1 -- start start 2024-03-24T00:44:37.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.563+0000 7f5155844640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f515015e2a0 0x7f51500ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:37.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.563+0000 7f5155844640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f51500d0ef0 0x7f5150162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:37.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.563+0000 7f5155844640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51500cf640 con 0x7f515015dec0 2024-03-24T00:44:37.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.563+0000 7f5155844640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51500cee30 con 0x7f51500d0ef0 2024-03-24T00:44:37.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.563+0000 7f5155844640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51500d2120 con 0x7f515015e2a0 2024-03-24T00:44:37.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.563+0000 7f514e7fc640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f515015e2a0 0x7f51500ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:37.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.563+0000 7f514e7fc640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f515015e2a0 0x7f51500ce630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.67:53316/0 (socket says 172.21.15.67:53316) 2024-03-24T00:44:37.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.563+0000 7f514effd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f515015dec0 0x7f51500d07e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:34034/0 (socket says 172.21.15.67:34034) 2024-03-24T00:44:37.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.563+0000 7f514e7fc640 1 -- 172.21.15.67:0/2679886533 learned_addr learned my addr 172.21.15.67:0/2679886533 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:37.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.563+0000 7f514f7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f51500d0ef0 0x7f5150162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:37.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.563+0000 7f514dffb640 1 -- 172.21.15.67:0/2679886533 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1919998282 0 0) 0x7f5140001110 con 0x7f515015dec0 2024-03-24T00:44:37.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.563+0000 7f514dffb640 1 -- 172.21.15.67:0/2679886533 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5138003520 con 0x7f515015dec0 2024-03-24T00:44:37.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.563+0000 7f514e7fc640 1 -- 172.21.15.67:0/2679886533 >> v2:172.21.15.72:3300/0 conn(0x7f51500d0ef0 msgr2=0x7f5150162630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:37.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.563+0000 7f514e7fc640 1 --2- 172.21.15.67:0/2679886533 >> v2:172.21.15.72:3300/0 conn(0x7f51500d0ef0 0x7f5150162630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:37.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.563+0000 7f514e7fc640 1 -- 172.21.15.67:0/2679886533 >> v1:172.21.15.72:6789/0 conn(0x7f515015dec0 legacy=0x7f51500d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:37.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.563+0000 7f514e7fc640 1 -- 172.21.15.67:0/2679886533 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51500cf080 con 0x7f515015e2a0 2024-03-24T00:44:37.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.563+0000 7f514f7fe640 1 --2- 172.21.15.67:0/2679886533 >> v2:172.21.15.72:3300/0 conn(0x7f51500d0ef0 0x7f5150162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:44:37.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.564+0000 7f514e7fc640 1 --2- 172.21.15.67:0/2679886533 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f515015e2a0 0x7f51500ce630 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5144009870 tx=0x7f514402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=599acfb39f0a888b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:37.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.564+0000 7f514dffb640 1 -- 172.21.15.67:0/2679886533 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5144035020 con 0x7f515015e2a0 2024-03-24T00:44:37.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.564+0000 7f514dffb640 1 -- 172.21.15.67:0/2679886533 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5144031640 con 0x7f515015e2a0 2024-03-24T00:44:37.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.564+0000 7f514dffb640 1 -- 172.21.15.67:0/2679886533 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f514402d040 con 0x7f515015e2a0 2024-03-24T00:44:37.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.565+0000 7f5155844640 1 -- 172.21.15.67:0/2679886533 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f515015e2a0 msgr2=0x7f51500ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:37.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.565+0000 7f5155844640 1 --2- 172.21.15.67:0/2679886533 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f515015e2a0 0x7f51500ce630 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5144009870 tx=0x7f514402b8f0 comp rx=0 tx=0).stop 2024-03-24T00:44:37.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.565+0000 7f5155844640 1 -- 172.21.15.67:0/2679886533 shutdown_connections 2024-03-24T00:44:37.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.565+0000 7f5155844640 1 --2- 172.21.15.67:0/2679886533 >> v2:172.21.15.72:3300/0 conn(0x7f51500d0ef0 0x7f5150162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:37.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.565+0000 7f5155844640 1 --2- 172.21.15.67:0/2679886533 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f515015e2a0 0x7f51500ce630 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:37.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.565+0000 7f5155844640 1 -- 172.21.15.67:0/2679886533 >> 172.21.15.67:0/2679886533 conn(0x7f515006c810 msgr2=0x7f515006cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:37.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.565+0000 7f5155844640 1 -- 172.21.15.67:0/2679886533 shutdown_connections 2024-03-24T00:44:37.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.565+0000 7f5155844640 1 -- 172.21.15.67:0/2679886533 wait complete. 2024-03-24T00:44:37.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.566+0000 7f5155844640 1 Processor -- start 2024-03-24T00:44:37.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.566+0000 7f5155844640 1 -- start start 2024-03-24T00:44:37.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.567+0000 7f5155844640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f51500d0ef0 0x7f51501f6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:37.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.567+0000 7f5155844640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f515015dec0 0x7f51500cccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:37.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.567+0000 7f5155844640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f515015e2a0 0x7f515015c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:37.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.567+0000 7f5155844640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51500d2730 con 0x7f515015e2a0 2024-03-24T00:44:37.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.567+0000 7f5155844640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51500d25b0 con 0x7f515015dec0 2024-03-24T00:44:37.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.567+0000 7f5155844640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51500d28b0 con 0x7f51500d0ef0 2024-03-24T00:44:37.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.567+0000 7f514f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f515015e2a0 0x7f515015c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:37.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.567+0000 7f514f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f515015e2a0 0x7f515015c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55332/0 (socket says 172.21.15.67:55332) 2024-03-24T00:44:37.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.567+0000 7f514f7fe640 1 -- 172.21.15.67:0/4084378613 learned_addr learned my addr 172.21.15.67:0/4084378613 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:37.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.567+0000 7f514e7fc640 1 --2- 172.21.15.67:0/4084378613 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f515015dec0 0x7f51500cccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:37.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.567+0000 7f514effd640 1 --2- 172.21.15.67:0/4084378613 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f51500d0ef0 0x7f51501f6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:37.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.567+0000 7f514f7fe640 1 -- 172.21.15.67:0/4084378613 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f51500d0ef0 msgr2=0x7f51501f6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:37.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.567+0000 7f514f7fe640 1 --2- 172.21.15.67:0/4084378613 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f51500d0ef0 0x7f51501f6d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:37.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.567+0000 7f514f7fe640 1 -- 172.21.15.67:0/4084378613 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f515015dec0 msgr2=0x7f51500cccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:37.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.567+0000 7f514f7fe640 1 --2- 172.21.15.67:0/4084378613 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f515015dec0 0x7f51500cccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:37.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.567+0000 7f514f7fe640 1 -- 172.21.15.67:0/4084378613 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51500ab680 con 0x7f515015e2a0 2024-03-24T00:44:37.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.567+0000 7f514effd640 1 --2- 172.21.15.67:0/4084378613 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f51500d0ef0 0x7f51501f6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:44:37.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.568+0000 7f514f7fe640 1 --2- 172.21.15.67:0/4084378613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f515015e2a0 0x7f515015c240 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f514800efc0 tx=0x7f514800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:37.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.568+0000 7f5137fff640 1 -- 172.21.15.67:0/4084378613 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5148018020 con 0x7f515015e2a0 2024-03-24T00:44:37.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.568+0000 7f5155844640 1 -- 172.21.15.67:0/4084378613 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51500ab960 con 0x7f515015e2a0 2024-03-24T00:44:37.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.568+0000 7f5137fff640 1 -- 172.21.15.67:0/4084378613 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5148014480 con 0x7f515015e2a0 2024-03-24T00:44:37.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.568+0000 7f5137fff640 1 -- 172.21.15.67:0/4084378613 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5148014850 con 0x7f515015e2a0 2024-03-24T00:44:37.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.568+0000 7f5155844640 1 -- 172.21.15.67:0/4084378613 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51500abf20 con 0x7f515015e2a0 2024-03-24T00:44:37.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.569+0000 7f5155844640 1 -- 172.21.15.67:0/4084378613 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f515004f400 con 0x7f515015e2a0 2024-03-24T00:44:37.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.569+0000 7f5137fff640 1 -- 172.21.15.67:0/4084378613 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5148005d10 con 0x7f515015e2a0 2024-03-24T00:44:37.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.570+0000 7f5137fff640 1 --2- 172.21.15.67:0/4084378613 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f513c06bbd0 0x7f513c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:37.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.570+0000 7f5137fff640 1 -- 172.21.15.67:0/4084378613 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f514808ba80 con 0x7f515015e2a0 2024-03-24T00:44:37.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.570+0000 7f514effd640 1 --2- 172.21.15.67:0/4084378613 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f513c06bbd0 0x7f513c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:37.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.570+0000 7f514effd640 1 --2- 172.21.15.67:0/4084378613 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f513c06bbd0 0x7f513c06e090 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f5140000f60 tx=0x7f5140009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:37.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.574+0000 7f5137fff640 1 -- 172.21.15.67:0/4084378613 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5148022030 con 0x7f515015e2a0 2024-03-24T00:44:37.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.758+0000 7f5155844640 1 -- 172.21.15.67:0/4084378613 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f51500d1aa0 con 0x7f515015e2a0 2024-03-24T00:44:37.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.759+0000 7f5137fff640 1 -- 172.21.15.67:0/4084378613 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+17014 (secure 0 0 0) 0x7f5148010040 con 0x7f515015e2a0 2024-03-24T00:44:37.760 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:44:37.760 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":77,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:43:24.373819+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:43:08.452726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-24T00:43:09.146431+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6833/2392802741":"2024-03-25T00:43:24.373788+0000","172.21.15.67:6832/2392802741":"2024-03-25T00:43:24.373788+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:44:37.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.761+0000 7f5155844640 1 -- 172.21.15.67:0/4084378613 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f513c06bbd0 msgr2=0x7f513c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:37.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.761+0000 7f5155844640 1 --2- 172.21.15.67:0/4084378613 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f513c06bbd0 0x7f513c06e090 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f5140000f60 tx=0x7f5140009040 comp rx=0 tx=0).stop 2024-03-24T00:44:37.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.761+0000 7f5155844640 1 -- 172.21.15.67:0/4084378613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f515015e2a0 msgr2=0x7f515015c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:37.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.761+0000 7f5155844640 1 --2- 172.21.15.67:0/4084378613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f515015e2a0 0x7f515015c240 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f514800efc0 tx=0x7f514800c490 comp rx=0 tx=0).stop 2024-03-24T00:44:37.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.761+0000 7f5155844640 1 -- 172.21.15.67:0/4084378613 shutdown_connections 2024-03-24T00:44:37.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.761+0000 7f5155844640 1 --2- 172.21.15.67:0/4084378613 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f513c06bbd0 0x7f513c06e090 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:37.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.761+0000 7f5155844640 1 --2- 172.21.15.67:0/4084378613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f515015e2a0 0x7f515015c240 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:37.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.761+0000 7f5155844640 1 --2- 172.21.15.67:0/4084378613 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f515015dec0 0x7f51500cccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:37.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.761+0000 7f5155844640 1 --2- 172.21.15.67:0/4084378613 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f51500d0ef0 0x7f51501f6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:37.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.761+0000 7f5155844640 1 -- 172.21.15.67:0/4084378613 >> 172.21.15.67:0/4084378613 conn(0x7f515006c810 msgr2=0x7f515015eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:37.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.761+0000 7f5155844640 1 -- 172.21.15.67:0/4084378613 shutdown_connections 2024-03-24T00:44:37.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.761+0000 7f5155844640 1 -- 172.21.15.67:0/4084378613 wait complete. 2024-03-24T00:44:37.776 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:38:39.364950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:43:08.452726+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:43:09.146431+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:44:37.777 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:44:37.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.905+0000 7f1f4c854640 1 Processor -- start 2024-03-24T00:44:37.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.905+0000 7f1f4c854640 1 -- start start 2024-03-24T00:44:37.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.906+0000 7f1f4c854640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1f480b51d0 0x7f1f480ad850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:37.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.906+0000 7f1f4c854640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f480b6150 con 0x7f1f4806eb30 2024-03-24T00:44:37.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.906+0000 7f1f4c854640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f4806ba50 con 0x7f1f4815de30 2024-03-24T00:44:37.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.906+0000 7f1f4c854640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f4806c380 con 0x7f1f480b51d0 2024-03-24T00:44:37.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.906+0000 7f1f46d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1f480b51d0 0x7f1f480ad850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:37.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.906+0000 7f1f45d9b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1f4806eb30 0x7f1f480ad140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47804/0 (socket says 172.21.15.67:47804) 2024-03-24T00:44:37.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.906+0000 7f1f45d9b640 1 -- 172.21.15.67:0/1454359385 learned_addr learned my addr 172.21.15.67:0/1454359385 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:44:37.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.907+0000 7f1f46d9d640 1 -- 172.21.15.67:0/1454359385 >> v1:172.21.15.72:6789/0 conn(0x7f1f4815de30 legacy=0x7f1f4806e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:37.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.907+0000 7f1f46d9d640 1 -- 172.21.15.67:0/1454359385 >> v1:172.21.15.67:6789/0 conn(0x7f1f4806eb30 legacy=0x7f1f480ad140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:37.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.907+0000 7f1f46d9d640 1 -- 172.21.15.67:0/1454359385 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f4806bd30 con 0x7f1f480b51d0 2024-03-24T00:44:37.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.907+0000 7f1f4559a640 1 -- 172.21.15.67:0/1454359385 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 640012659 0 0) 0x7f1f480b6150 con 0x7f1f4806eb30 2024-03-24T00:44:37.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.908+0000 7f1f46d9d640 1 --2- 172.21.15.67:0/1454359385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f480b51d0 0x7f1f480ad850 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f1f4806c500 tx=0x7f1f4002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97d03acef04b541a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:37.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.908+0000 7f1f4559a640 1 -- 172.21.15.67:0/1454359385 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f40039070 con 0x7f1f480b51d0 2024-03-24T00:44:37.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.908+0000 7f1f4559a640 1 -- 172.21.15.67:0/1454359385 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f4002cc50 con 0x7f1f480b51d0 2024-03-24T00:44:37.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.908+0000 7f1f4559a640 1 -- 172.21.15.67:0/1454359385 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f40034440 con 0x7f1f480b51d0 2024-03-24T00:44:37.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.908+0000 7f1f4c854640 1 -- 172.21.15.67:0/1454359385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f480b51d0 msgr2=0x7f1f480ad850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:37.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.908+0000 7f1f4c854640 1 --2- 172.21.15.67:0/1454359385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f480b51d0 0x7f1f480ad850 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f1f4806c500 tx=0x7f1f4002bea0 comp rx=0 tx=0).stop 2024-03-24T00:44:37.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.908+0000 7f1f4c854640 1 -- 172.21.15.67:0/1454359385 shutdown_connections 2024-03-24T00:44:37.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.908+0000 7f1f4c854640 1 --2- 172.21.15.67:0/1454359385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f480b51d0 0x7f1f480ad850 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:37.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.908+0000 7f1f4c854640 1 -- 172.21.15.67:0/1454359385 >> 172.21.15.67:0/1454359385 conn(0x7f1f48099a30 msgr2=0x7f1f480b5680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:37.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.909+0000 7f1f4c854640 1 -- 172.21.15.67:0/1454359385 shutdown_connections 2024-03-24T00:44:37.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.909+0000 7f1f4c854640 1 -- 172.21.15.67:0/1454359385 wait complete. 2024-03-24T00:44:37.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.910+0000 7f1f4c854640 1 Processor -- start 2024-03-24T00:44:37.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.911+0000 7f1f4c854640 1 -- start start 2024-03-24T00:44:37.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.912+0000 7f1f4c854640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f4806eb30 0x7f1f481f6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:37.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.912+0000 7f1f4c854640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1f480b51d0 0x7f1f481574f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:37.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.912+0000 7f1f4c854640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f4815de30 0x7f1f48157a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:37.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.912+0000 7f1f4c854640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4815e210 con 0x7f1f4815de30 2024-03-24T00:44:37.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.912+0000 7f1f4c854640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4806c810 con 0x7f1f4806eb30 2024-03-24T00:44:37.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.912+0000 7f1f4c854640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f480caef0 con 0x7f1f480b51d0 2024-03-24T00:44:37.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.912+0000 7f1f46d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f4815de30 0x7f1f48157a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:37.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.912+0000 7f1f46d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f4815de30 0x7f1f48157a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55342/0 (socket says 172.21.15.67:55342) 2024-03-24T00:44:37.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.912+0000 7f1f46d9d640 1 -- 172.21.15.67:0/3207655660 learned_addr learned my addr 172.21.15.67:0/3207655660 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:37.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.912+0000 7f1f4659c640 1 --2- 172.21.15.67:0/3207655660 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f4806eb30 0x7f1f481f6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:37.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.912+0000 7f1f45d9b640 1 --2- 172.21.15.67:0/3207655660 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1f480b51d0 0x7f1f481574f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:37.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.912+0000 7f1f46d9d640 1 -- 172.21.15.67:0/3207655660 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1f480b51d0 msgr2=0x7f1f481574f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:37.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.912+0000 7f1f46d9d640 1 --2- 172.21.15.67:0/3207655660 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1f480b51d0 0x7f1f481574f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:37.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.912+0000 7f1f46d9d640 1 -- 172.21.15.67:0/3207655660 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f4806eb30 msgr2=0x7f1f481f6ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:37.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.912+0000 7f1f46d9d640 1 --2- 172.21.15.67:0/3207655660 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f4806eb30 0x7f1f481f6ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:37.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.912+0000 7f1f46d9d640 1 -- 172.21.15.67:0/3207655660 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f481f7c00 con 0x7f1f4815de30 2024-03-24T00:44:37.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.912+0000 7f1f45d9b640 1 --2- 172.21.15.67:0/3207655660 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1f480b51d0 0x7f1f481574f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:44:37.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.912+0000 7f1f46d9d640 1 --2- 172.21.15.67:0/3207655660 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f4815de30 0x7f1f48157a30 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f1f40037040 tx=0x7f1f40004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:37.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.912+0000 7f1f36ffd640 1 -- 172.21.15.67:0/3207655660 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f40044020 con 0x7f1f4815de30 2024-03-24T00:44:37.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.913+0000 7f1f36ffd640 1 -- 172.21.15.67:0/3207655660 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f4003a040 con 0x7f1f4815de30 2024-03-24T00:44:37.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.913+0000 7f1f4c854640 1 -- 172.21.15.67:0/3207655660 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f481f3530 con 0x7f1f4815de30 2024-03-24T00:44:37.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.913+0000 7f1f36ffd640 1 -- 172.21.15.67:0/3207655660 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f40039050 con 0x7f1f4815de30 2024-03-24T00:44:37.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.913+0000 7f1f4c854640 1 -- 172.21.15.67:0/3207655660 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f481f7640 con 0x7f1f4815de30 2024-03-24T00:44:37.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.914+0000 7f1f4c854640 1 -- 172.21.15.67:0/3207655660 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f480cac50 con 0x7f1f4815de30 2024-03-24T00:44:37.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.914+0000 7f1f36ffd640 1 -- 172.21.15.67:0/3207655660 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1f4002cad0 con 0x7f1f4815de30 2024-03-24T00:44:37.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.914+0000 7f1f36ffd640 1 --2- 172.21.15.67:0/3207655660 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1f2806bbd0 0x7f1f2806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:37.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.914+0000 7f1f36ffd640 1 -- 172.21.15.67:0/3207655660 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f1f400ad1e0 con 0x7f1f4815de30 2024-03-24T00:44:37.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.915+0000 7f1f4659c640 1 --2- 172.21.15.67:0/3207655660 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1f2806bbd0 0x7f1f2806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:37.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.915+0000 7f1f4659c640 1 --2- 172.21.15.67:0/3207655660 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1f2806bbd0 0x7f1f2806e090 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f1f38000f60 tx=0x7f1f38009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:37.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:37.919+0000 7f1f36ffd640 1 -- 172.21.15.67:0/3207655660 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1f4004e020 con 0x7f1f4815de30 2024-03-24T00:44:38.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.099+0000 7f1f4c854640 1 -- 172.21.15.67:0/3207655660 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1f4804f400 con 0x7f1f4815de30 2024-03-24T00:44:38.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.099+0000 7f1f36ffd640 1 -- 172.21.15.67:0/3207655660 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1f4007bb80 con 0x7f1f4815de30 2024-03-24T00:44:38.101 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-24T00:44:38.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.101+0000 7f1f4c854640 1 -- 172.21.15.67:0/3207655660 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1f2806bbd0 msgr2=0x7f1f2806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:38.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.101+0000 7f1f4c854640 1 --2- 172.21.15.67:0/3207655660 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1f2806bbd0 0x7f1f2806e090 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f1f38000f60 tx=0x7f1f38009040 comp rx=0 tx=0).stop 2024-03-24T00:44:38.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.101+0000 7f1f4c854640 1 -- 172.21.15.67:0/3207655660 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f4815de30 msgr2=0x7f1f48157a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:38.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.101+0000 7f1f4c854640 1 --2- 172.21.15.67:0/3207655660 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f4815de30 0x7f1f48157a30 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f1f40037040 tx=0x7f1f40004a40 comp rx=0 tx=0).stop 2024-03-24T00:44:38.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.101+0000 7f1f4c854640 1 -- 172.21.15.67:0/3207655660 shutdown_connections 2024-03-24T00:44:38.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.101+0000 7f1f4c854640 1 --2- 172.21.15.67:0/3207655660 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1f2806bbd0 0x7f1f2806e090 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:38.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.101+0000 7f1f4c854640 1 --2- 172.21.15.67:0/3207655660 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f4815de30 0x7f1f48157a30 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:38.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.101+0000 7f1f4c854640 1 --2- 172.21.15.67:0/3207655660 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1f480b51d0 0x7f1f481574f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:38.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.101+0000 7f1f4c854640 1 --2- 172.21.15.67:0/3207655660 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f4806eb30 0x7f1f481f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:38.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.102+0000 7f1f4c854640 1 -- 172.21.15.67:0/3207655660 >> 172.21.15.67:0/3207655660 conn(0x7f1f48099a30 msgr2=0x7f1f480af150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:38.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.102+0000 7f1f4c854640 1 -- 172.21.15.67:0/3207655660 shutdown_connections 2024-03-24T00:44:38.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.102+0000 7f1f4c854640 1 -- 172.21.15.67:0/3207655660 wait complete. 2024-03-24T00:44:38.116 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:44:38.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.220+0000 7f9a79e36640 1 Processor -- start 2024-03-24T00:44:38.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.220+0000 7f9a79e36640 1 -- start start 2024-03-24T00:44:38.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.221+0000 7f9a79e36640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9a7415dea0 0x7f9a740d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:38.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.221+0000 7f9a79e36640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a740d2250 con 0x7f9a740b6170 2024-03-24T00:44:38.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.221+0000 7f9a79e36640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a740d2530 con 0x7f9a7415e280 2024-03-24T00:44:38.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.221+0000 7f9a79e36640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a740d2810 con 0x7f9a7415dea0 2024-03-24T00:44:38.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.221+0000 7f9a737fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9a7415dea0 0x7f9a740d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:38.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.221+0000 7f9a73fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9a740b6170 0x7f9a740b70f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47816/0 (socket says 172.21.15.67:47816) 2024-03-24T00:44:38.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.221+0000 7f9a73fff640 1 -- 172.21.15.67:0/4046428373 learned_addr learned my addr 172.21.15.67:0/4046428373 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:44:38.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.222+0000 7f9a737fe640 1 -- 172.21.15.67:0/4046428373 >> v1:172.21.15.72:6789/0 conn(0x7f9a7415e280 legacy=0x7f9a740b5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:38.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.222+0000 7f9a737fe640 1 -- 172.21.15.67:0/4046428373 >> v1:172.21.15.67:6789/0 conn(0x7f9a740b6170 legacy=0x7f9a740b70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:38.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.222+0000 7f9a727fc640 1 -- 172.21.15.67:0/4046428373 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2031409476 0 0) 0x7f9a740d2250 con 0x7f9a740b6170 2024-03-24T00:44:38.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.222+0000 7f9a737fe640 1 -- 172.21.15.67:0/4046428373 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a740d2af0 con 0x7f9a7415dea0 2024-03-24T00:44:38.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.222+0000 7f9a737fe640 1 --2- 172.21.15.67:0/4046428373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a7415dea0 0x7f9a740d33d0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f9a640097d0 tx=0x7f9a6402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4187f7c605193b0d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:38.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.223+0000 7f9a727fc640 1 -- 172.21.15.67:0/4046428373 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a64035020 con 0x7f9a7415dea0 2024-03-24T00:44:38.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.223+0000 7f9a727fc640 1 -- 172.21.15.67:0/4046428373 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a64031830 con 0x7f9a7415dea0 2024-03-24T00:44:38.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.223+0000 7f9a727fc640 1 -- 172.21.15.67:0/4046428373 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a64031c00 con 0x7f9a7415dea0 2024-03-24T00:44:38.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.223+0000 7f9a79e36640 1 -- 172.21.15.67:0/4046428373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a7415dea0 msgr2=0x7f9a740d33d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:38.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.223+0000 7f9a79e36640 1 --2- 172.21.15.67:0/4046428373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a7415dea0 0x7f9a740d33d0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f9a640097d0 tx=0x7f9a6402b8d0 comp rx=0 tx=0).stop 2024-03-24T00:44:38.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.223+0000 7f9a79e36640 1 -- 172.21.15.67:0/4046428373 shutdown_connections 2024-03-24T00:44:38.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.223+0000 7f9a79e36640 1 --2- 172.21.15.67:0/4046428373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a7415dea0 0x7f9a740d33d0 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:38.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.223+0000 7f9a79e36640 1 -- 172.21.15.67:0/4046428373 >> 172.21.15.67:0/4046428373 conn(0x7f9a74072da0 msgr2=0x7f9a740d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:38.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.223+0000 7f9a79e36640 1 -- 172.21.15.67:0/4046428373 shutdown_connections 2024-03-24T00:44:38.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.223+0000 7f9a79e36640 1 -- 172.21.15.67:0/4046428373 wait complete. 2024-03-24T00:44:38.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.224+0000 7f9a79e36640 1 Processor -- start 2024-03-24T00:44:38.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.225+0000 7f9a79e36640 1 -- start start 2024-03-24T00:44:38.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.225+0000 7f9a79e36640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9a740b6170 0x7f9a740b2270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:38.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.225+0000 7f9a79e36640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a7415dea0 0x7f9a740ad9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:38.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.225+0000 7f9a79e36640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a7415e280 0x7f9a740adf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:38.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.225+0000 7f9a79e36640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a740d3ca0 con 0x7f9a7415e280 2024-03-24T00:44:38.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.225+0000 7f9a79e36640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a740d3b20 con 0x7f9a7415dea0 2024-03-24T00:44:38.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.225+0000 7f9a79e36640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a740b7800 con 0x7f9a740b6170 2024-03-24T00:44:38.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.225+0000 7f9a73fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a7415e280 0x7f9a740adf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:38.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.225+0000 7f9a73fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a7415e280 0x7f9a740adf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55366/0 (socket says 172.21.15.67:55366) 2024-03-24T00:44:38.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.225+0000 7f9a73fff640 1 -- 172.21.15.67:0/2300227545 learned_addr learned my addr 172.21.15.67:0/2300227545 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:38.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.225+0000 7f9a737fe640 1 --2- 172.21.15.67:0/2300227545 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9a740b6170 0x7f9a740b2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:38.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.225+0000 7f9a72ffd640 1 --2- 172.21.15.67:0/2300227545 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a7415dea0 0x7f9a740ad9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:38.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.226+0000 7f9a73fff640 1 -- 172.21.15.67:0/2300227545 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9a740b6170 msgr2=0x7f9a740b2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:38.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.226+0000 7f9a73fff640 1 --2- 172.21.15.67:0/2300227545 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9a740b6170 0x7f9a740b2270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:38.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.226+0000 7f9a73fff640 1 -- 172.21.15.67:0/2300227545 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a7415dea0 msgr2=0x7f9a740ad9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:38.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.226+0000 7f9a73fff640 1 --2- 172.21.15.67:0/2300227545 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a7415dea0 0x7f9a740ad9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:38.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.226+0000 7f9a73fff640 1 -- 172.21.15.67:0/2300227545 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a740b27b0 con 0x7f9a7415e280 2024-03-24T00:44:38.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.226+0000 7f9a72ffd640 1 --2- 172.21.15.67:0/2300227545 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a7415dea0 0x7f9a740ad9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:44:38.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.226+0000 7f9a73fff640 1 --2- 172.21.15.67:0/2300227545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a7415e280 0x7f9a740adf20 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c00c800 tx=0x7f9a6c00ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:38.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.226+0000 7f9a5bfff640 1 -- 172.21.15.67:0/2300227545 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c018070 con 0x7f9a7415e280 2024-03-24T00:44:38.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.226+0000 7f9a5bfff640 1 -- 172.21.15.67:0/2300227545 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a6c004d80 con 0x7f9a7415e280 2024-03-24T00:44:38.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.226+0000 7f9a79e36640 1 -- 172.21.15.67:0/2300227545 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a740cdb50 con 0x7f9a7415e280 2024-03-24T00:44:38.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.226+0000 7f9a5bfff640 1 -- 172.21.15.67:0/2300227545 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c00a880 con 0x7f9a7415e280 2024-03-24T00:44:38.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.226+0000 7f9a79e36640 1 -- 172.21.15.67:0/2300227545 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a740cd870 con 0x7f9a7415e280 2024-03-24T00:44:38.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.227+0000 7f9a79e36640 1 -- 172.21.15.67:0/2300227545 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a7404f400 con 0x7f9a7415e280 2024-03-24T00:44:38.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.227+0000 7f9a5bfff640 1 -- 172.21.15.67:0/2300227545 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9a6c01b070 con 0x7f9a7415e280 2024-03-24T00:44:38.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.228+0000 7f9a5bfff640 1 --2- 172.21.15.67:0/2300227545 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9a6006bbd0 0x7f9a6006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:38.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.228+0000 7f9a5bfff640 1 -- 172.21.15.67:0/2300227545 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f9a6c08a0b0 con 0x7f9a7415e280 2024-03-24T00:44:38.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.228+0000 7f9a737fe640 1 --2- 172.21.15.67:0/2300227545 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9a6006bbd0 0x7f9a6006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:38.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.229+0000 7f9a737fe640 1 --2- 172.21.15.67:0/2300227545 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9a6006bbd0 0x7f9a6006e090 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f9a64036000 tx=0x7f9a64008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:38.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.233+0000 7f9a5bfff640 1 -- 172.21.15.67:0/2300227545 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9a6c0584c0 con 0x7f9a7415e280 2024-03-24T00:44:38.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.418+0000 7f9a79e36640 1 -- 172.21.15.67:0/2300227545 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9a7414dc30 con 0x7f9a7415e280 2024-03-24T00:44:38.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.419+0000 7f9a5bfff640 1 -- 172.21.15.67:0/2300227545 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v77) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9a6c05c560 con 0x7f9a7415e280 2024-03-24T00:44:38.420 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-24T00:44:38.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.421+0000 7f9a79e36640 1 -- 172.21.15.67:0/2300227545 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9a6006bbd0 msgr2=0x7f9a6006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:38.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.421+0000 7f9a79e36640 1 --2- 172.21.15.67:0/2300227545 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9a6006bbd0 0x7f9a6006e090 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f9a64036000 tx=0x7f9a64008030 comp rx=0 tx=0).stop 2024-03-24T00:44:38.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.421+0000 7f9a79e36640 1 -- 172.21.15.67:0/2300227545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a7415e280 msgr2=0x7f9a740adf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:38.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.421+0000 7f9a79e36640 1 --2- 172.21.15.67:0/2300227545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a7415e280 0x7f9a740adf20 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c00c800 tx=0x7f9a6c00ccc0 comp rx=0 tx=0).stop 2024-03-24T00:44:38.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.422+0000 7f9a79e36640 1 -- 172.21.15.67:0/2300227545 shutdown_connections 2024-03-24T00:44:38.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.422+0000 7f9a79e36640 1 --2- 172.21.15.67:0/2300227545 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9a6006bbd0 0x7f9a6006e090 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:38.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.422+0000 7f9a79e36640 1 --2- 172.21.15.67:0/2300227545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a7415e280 0x7f9a740adf20 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:38.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.422+0000 7f9a79e36640 1 --2- 172.21.15.67:0/2300227545 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a7415dea0 0x7f9a740ad9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:38.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.422+0000 7f9a79e36640 1 --2- 172.21.15.67:0/2300227545 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9a740b6170 0x7f9a740b2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:38.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.422+0000 7f9a79e36640 1 -- 172.21.15.67:0/2300227545 >> 172.21.15.67:0/2300227545 conn(0x7f9a74072da0 msgr2=0x7f9a740d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:38.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.422+0000 7f9a79e36640 1 -- 172.21.15.67:0/2300227545 shutdown_connections 2024-03-24T00:44:38.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.422+0000 7f9a79e36640 1 -- 172.21.15.67:0/2300227545 wait complete. 2024-03-24T00:44:38.433 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:44:38.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.537+0000 7f186769a640 1 Processor -- start 2024-03-24T00:44:38.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.537+0000 7f186769a640 1 -- start start 2024-03-24T00:44:38.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.538+0000 7f186769a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f186015dea0 0x7f18600d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:38.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.538+0000 7f186769a640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f18600b6170 0x7f18600d3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:38.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.538+0000 7f186769a640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18600d2250 con 0x7f186015e280 2024-03-24T00:44:38.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.538+0000 7f186769a640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18600d2530 con 0x7f186015dea0 2024-03-24T00:44:38.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.538+0000 7f186769a640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18600d2810 con 0x7f18600b6170 2024-03-24T00:44:38.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.538+0000 7f1864c35640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f186015e280 0x7f18600b5a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47822/0 (socket says 172.21.15.67:47822) 2024-03-24T00:44:38.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.538+0000 7f1864c35640 1 -- 172.21.15.67:0/2709231499 learned_addr learned my addr 172.21.15.67:0/2709231499 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:44:38.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.538+0000 7f1865c37640 1 --2- 172.21.15.67:0/2709231499 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f18600b6170 0x7f18600d3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:38.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.538+0000 7f1865436640 1 --2- 172.21.15.67:0/2709231499 >> v2:172.21.15.72:3300/0 conn(0x7f186015dea0 0x7f18600d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:38.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.538+0000 7f1857fff640 1 -- 172.21.15.67:0/2709231499 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3778059213 0 0) 0x7f18600d2250 con 0x7f186015e280 2024-03-24T00:44:38.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.538+0000 7f1857fff640 1 -- 172.21.15.67:0/2709231499 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f184c003520 con 0x7f186015e280 2024-03-24T00:44:38.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.539+0000 7f1865c37640 1 -- 172.21.15.67:0/2709231499 >> v2:172.21.15.72:3300/0 conn(0x7f186015dea0 msgr2=0x7f18600d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:38.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.539+0000 7f1865c37640 1 --2- 172.21.15.67:0/2709231499 >> v2:172.21.15.72:3300/0 conn(0x7f186015dea0 0x7f18600d33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:38.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.539+0000 7f1865c37640 1 -- 172.21.15.67:0/2709231499 >> v1:172.21.15.67:6789/0 conn(0x7f186015e280 legacy=0x7f18600b5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:38.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.539+0000 7f1865c37640 1 -- 172.21.15.67:0/2709231499 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18600d2af0 con 0x7f18600b6170 2024-03-24T00:44:38.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.539+0000 7f1865c37640 1 --2- 172.21.15.67:0/2709231499 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f18600b6170 0x7f18600d3910 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1858009d90 tx=0x7f185802be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=486246fa36bbc31c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:38.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.540+0000 7f1857fff640 1 -- 172.21.15.67:0/2709231499 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1858039070 con 0x7f18600b6170 2024-03-24T00:44:38.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.540+0000 7f1857fff640 1 -- 172.21.15.67:0/2709231499 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f185802cbc0 con 0x7f18600b6170 2024-03-24T00:44:38.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.540+0000 7f1857fff640 1 -- 172.21.15.67:0/2709231499 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1858034440 con 0x7f18600b6170 2024-03-24T00:44:38.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.540+0000 7f186769a640 1 -- 172.21.15.67:0/2709231499 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f18600b6170 msgr2=0x7f18600d3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:38.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.540+0000 7f186769a640 1 --2- 172.21.15.67:0/2709231499 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f18600b6170 0x7f18600d3910 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1858009d90 tx=0x7f185802be10 comp rx=0 tx=0).stop 2024-03-24T00:44:38.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.540+0000 7f186769a640 1 -- 172.21.15.67:0/2709231499 shutdown_connections 2024-03-24T00:44:38.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.540+0000 7f186769a640 1 --2- 172.21.15.67:0/2709231499 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f18600b6170 0x7f18600d3910 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:38.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.540+0000 7f186769a640 1 --2- 172.21.15.67:0/2709231499 >> v2:172.21.15.72:3300/0 conn(0x7f186015dea0 0x7f18600d33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:38.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.540+0000 7f186769a640 1 -- 172.21.15.67:0/2709231499 >> 172.21.15.67:0/2709231499 conn(0x7f1860072da0 msgr2=0x7f18600d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:38.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.540+0000 7f186769a640 1 -- 172.21.15.67:0/2709231499 shutdown_connections 2024-03-24T00:44:38.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.540+0000 7f186769a640 1 -- 172.21.15.67:0/2709231499 wait complete. 2024-03-24T00:44:38.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.541+0000 7f186769a640 1 Processor -- start 2024-03-24T00:44:38.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.541+0000 7f186769a640 1 -- start start 2024-03-24T00:44:38.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.542+0000 7f186769a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18600b6170 0x7f186014e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:38.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.542+0000 7f186769a640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f186015dea0 0x7f18600cd0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:38.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.542+0000 7f186769a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f186015e280 0x7f18600cd600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:38.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.542+0000 7f186769a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18600b7480 con 0x7f186015e280 2024-03-24T00:44:38.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.542+0000 7f186769a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18600b7300 con 0x7f18600b6170 2024-03-24T00:44:38.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.542+0000 7f186769a640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18600b7600 con 0x7f186015dea0 2024-03-24T00:44:38.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.542+0000 7f1865c37640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f186015e280 0x7f18600cd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:38.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.542+0000 7f1865c37640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f186015e280 0x7f18600cd600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55380/0 (socket says 172.21.15.67:55380) 2024-03-24T00:44:38.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.542+0000 7f1865c37640 1 -- 172.21.15.67:0/3083812030 learned_addr learned my addr 172.21.15.67:0/3083812030 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:38.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.542+0000 7f1865436640 1 --2- 172.21.15.67:0/3083812030 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18600b6170 0x7f186014e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:38.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.542+0000 7f1864c35640 1 --2- 172.21.15.67:0/3083812030 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f186015dea0 0x7f18600cd0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:38.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.542+0000 7f1865c37640 1 -- 172.21.15.67:0/3083812030 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f186015dea0 msgr2=0x7f18600cd0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:38.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.542+0000 7f1865c37640 1 --2- 172.21.15.67:0/3083812030 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f186015dea0 0x7f18600cd0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:38.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.542+0000 7f1865c37640 1 -- 172.21.15.67:0/3083812030 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18600b6170 msgr2=0x7f186014e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:38.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.542+0000 7f1865c37640 1 --2- 172.21.15.67:0/3083812030 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18600b6170 0x7f186014e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:38.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.542+0000 7f1865c37640 1 -- 172.21.15.67:0/3083812030 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18601508d0 con 0x7f186015e280 2024-03-24T00:44:38.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.543+0000 7f1865c37640 1 --2- 172.21.15.67:0/3083812030 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f186015e280 0x7f18600cd600 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f1858009d90 tx=0x7f1858004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:38.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.543+0000 7f1855ffb640 1 -- 172.21.15.67:0/3083812030 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1858039050 con 0x7f186015e280 2024-03-24T00:44:38.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.543+0000 7f1855ffb640 1 -- 172.21.15.67:0/3083812030 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1858045070 con 0x7f186015e280 2024-03-24T00:44:38.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.543+0000 7f186769a640 1 -- 172.21.15.67:0/3083812030 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1860150bb0 con 0x7f186015e280 2024-03-24T00:44:38.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.543+0000 7f1855ffb640 1 -- 172.21.15.67:0/3083812030 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1858009450 con 0x7f186015e280 2024-03-24T00:44:38.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.543+0000 7f186769a640 1 -- 172.21.15.67:0/3083812030 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1860151170 con 0x7f186015e280 2024-03-24T00:44:38.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.544+0000 7f186769a640 1 -- 172.21.15.67:0/3083812030 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f186004f400 con 0x7f186015e280 2024-03-24T00:44:38.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.545+0000 7f1855ffb640 1 -- 172.21.15.67:0/3083812030 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f185803e830 con 0x7f186015e280 2024-03-24T00:44:38.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.545+0000 7f1855ffb640 1 --2- 172.21.15.67:0/3083812030 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f184406bc00 0x7f184406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:38.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.545+0000 7f1855ffb640 1 -- 172.21.15.67:0/3083812030 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f18580ad820 con 0x7f186015e280 2024-03-24T00:44:38.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.545+0000 7f1865436640 1 --2- 172.21.15.67:0/3083812030 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f184406bc00 0x7f184406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:38.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.546+0000 7f1865436640 1 --2- 172.21.15.67:0/3083812030 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f184406bc00 0x7f184406e0c0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f185c009b40 tx=0x7f185c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:38.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.549+0000 7f1855ffb640 1 -- 172.21.15.67:0/3083812030 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f185807bcc0 con 0x7f186015e280 2024-03-24T00:44:38.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.737+0000 7f186769a640 1 -- 172.21.15.67:0/3083812030 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f18600b81a0 con 0x7f186015e280 2024-03-24T00:44:38.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.737+0000 7f1855ffb640 1 -- 172.21.15.67:0/3083812030 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v77) v1 ==== 100+0+11 (secure 0 0 0) 0x7f185807fd60 con 0x7f186015e280 2024-03-24T00:44:38.738 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-03-24T00:44:38.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.740+0000 7f186769a640 1 -- 172.21.15.67:0/3083812030 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f184406bc00 msgr2=0x7f184406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:38.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.740+0000 7f186769a640 1 --2- 172.21.15.67:0/3083812030 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f184406bc00 0x7f184406e0c0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f185c009b40 tx=0x7f185c009190 comp rx=0 tx=0).stop 2024-03-24T00:44:38.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.740+0000 7f186769a640 1 -- 172.21.15.67:0/3083812030 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f186015e280 msgr2=0x7f18600cd600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:38.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.740+0000 7f186769a640 1 --2- 172.21.15.67:0/3083812030 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f186015e280 0x7f18600cd600 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f1858009d90 tx=0x7f1858004a40 comp rx=0 tx=0).stop 2024-03-24T00:44:38.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.740+0000 7f186769a640 1 -- 172.21.15.67:0/3083812030 shutdown_connections 2024-03-24T00:44:38.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.740+0000 7f186769a640 1 --2- 172.21.15.67:0/3083812030 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f184406bc00 0x7f184406e0c0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:38.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.740+0000 7f186769a640 1 --2- 172.21.15.67:0/3083812030 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f186015e280 0x7f18600cd600 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:38.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.740+0000 7f186769a640 1 --2- 172.21.15.67:0/3083812030 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f186015dea0 0x7f18600cd0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:38.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.740+0000 7f186769a640 1 --2- 172.21.15.67:0/3083812030 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18600b6170 0x7f186014e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:38.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.740+0000 7f186769a640 1 -- 172.21.15.67:0/3083812030 >> 172.21.15.67:0/3083812030 conn(0x7f1860072da0 msgr2=0x7f18600d3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:38.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.740+0000 7f186769a640 1 -- 172.21.15.67:0/3083812030 shutdown_connections 2024-03-24T00:44:38.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.740+0000 7f186769a640 1 -- 172.21.15.67:0/3083812030 wait complete. 2024-03-24T00:44:38.752 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T00:44:38.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.856+0000 7efe31787640 1 Processor -- start 2024-03-24T00:44:38.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.856+0000 7efe31787640 1 -- start start 2024-03-24T00:44:38.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.857+0000 7efe31787640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7efe2c0d32d0 0x7efe2c0d3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:38.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.857+0000 7efe31787640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efe2c0b6dc0 0x7efe2c0b7250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:38.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.858+0000 7efe31787640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe2c0b9cc0 con 0x7efe2c15e0f0 2024-03-24T00:44:38.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.858+0000 7efe31787640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe2c0b7790 con 0x7efe2c0d32d0 2024-03-24T00:44:38.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.858+0000 7efe31787640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe2c0d2230 con 0x7efe2c0b6dc0 2024-03-24T00:44:38.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.858+0000 7efe2b7fe640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efe2c0b6dc0 0x7efe2c0b7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:38.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.858+0000 7efe2b7fe640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efe2c0b6dc0 0x7efe2c0b7250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.67:53364/0 (socket says 172.21.15.67:53364) 2024-03-24T00:44:38.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.858+0000 7efe2b7fe640 1 -- 172.21.15.67:0/572157801 learned_addr learned my addr 172.21.15.67:0/572157801 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:38.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.858+0000 7efe23fff640 1 --2- 172.21.15.67:0/572157801 >> v2:172.21.15.72:3300/0 conn(0x7efe2c0d32d0 0x7efe2c0d3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:38.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.858+0000 7efe2b7fe640 1 -- 172.21.15.67:0/572157801 >> v2:172.21.15.72:3300/0 conn(0x7efe2c0d32d0 msgr2=0x7efe2c0d3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:38.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.858+0000 7efe2b7fe640 1 --2- 172.21.15.67:0/572157801 >> v2:172.21.15.72:3300/0 conn(0x7efe2c0d32d0 0x7efe2c0d3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:38.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.858+0000 7efe2b7fe640 1 -- 172.21.15.67:0/572157801 >> v1:172.21.15.72:6789/0 conn(0x7efe2c15e0f0 legacy=0x7efe2c0b95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:38.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.858+0000 7efe2b7fe640 1 -- 172.21.15.67:0/572157801 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe2c0d2510 con 0x7efe2c0b6dc0 2024-03-24T00:44:38.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.859+0000 7efe2b7fe640 1 --2- 172.21.15.67:0/572157801 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efe2c0b6dc0 0x7efe2c0b7250 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7efe24009e50 tx=0x7efe2402bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=afd89494a8a5b1fd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:38.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.859+0000 7efe2a7fc640 1 -- 172.21.15.67:0/572157801 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe2402cb00 con 0x7efe2c0b6dc0 2024-03-24T00:44:38.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.859+0000 7efe2a7fc640 1 -- 172.21.15.67:0/572157801 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe2402cca0 con 0x7efe2c0b6dc0 2024-03-24T00:44:38.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.859+0000 7efe2a7fc640 1 -- 172.21.15.67:0/572157801 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe24034610 con 0x7efe2c0b6dc0 2024-03-24T00:44:38.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.860+0000 7efe31787640 1 -- 172.21.15.67:0/572157801 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efe2c0b6dc0 msgr2=0x7efe2c0b7250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:38.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.860+0000 7efe31787640 1 --2- 172.21.15.67:0/572157801 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efe2c0b6dc0 0x7efe2c0b7250 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7efe24009e50 tx=0x7efe2402bf60 comp rx=0 tx=0).stop 2024-03-24T00:44:38.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.860+0000 7efe31787640 1 -- 172.21.15.67:0/572157801 shutdown_connections 2024-03-24T00:44:38.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.860+0000 7efe31787640 1 --2- 172.21.15.67:0/572157801 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efe2c0b6dc0 0x7efe2c0b7250 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:38.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.860+0000 7efe31787640 1 --2- 172.21.15.67:0/572157801 >> v2:172.21.15.72:3300/0 conn(0x7efe2c0d32d0 0x7efe2c0d3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:38.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.860+0000 7efe31787640 1 -- 172.21.15.67:0/572157801 >> 172.21.15.67:0/572157801 conn(0x7efe2c072b60 msgr2=0x7efe2c072f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:38.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.860+0000 7efe31787640 1 -- 172.21.15.67:0/572157801 shutdown_connections 2024-03-24T00:44:38.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.860+0000 7efe31787640 1 -- 172.21.15.67:0/572157801 wait complete. 2024-03-24T00:44:38.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.861+0000 7efe31787640 1 Processor -- start 2024-03-24T00:44:38.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.861+0000 7efe31787640 1 -- start start 2024-03-24T00:44:38.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.862+0000 7efe31787640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efe2c0b6dc0 0x7efe2c1522d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:38.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.862+0000 7efe31787640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe2c0d32d0 0x7efe2c1f3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:38.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.862+0000 7efe31787640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efe2c15e0f0 0x7efe2c0c9810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:38.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.862+0000 7efe31787640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe2c0d2db0 con 0x7efe2c0d32d0 2024-03-24T00:44:38.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.862+0000 7efe31787640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe2c0d2ad0 con 0x7efe2c0b6dc0 2024-03-24T00:44:38.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.862+0000 7efe31787640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe2c0b31d0 con 0x7efe2c15e0f0 2024-03-24T00:44:38.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.862+0000 7efe23fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe2c0d32d0 0x7efe2c1f3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:38.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.862+0000 7efe23fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe2c0d32d0 0x7efe2c1f3b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55390/0 (socket says 172.21.15.67:55390) 2024-03-24T00:44:38.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.862+0000 7efe23fff640 1 -- 172.21.15.67:0/3850395437 learned_addr learned my addr 172.21.15.67:0/3850395437 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:38.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.862+0000 7efe2affd640 1 --2- 172.21.15.67:0/3850395437 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efe2c0b6dc0 0x7efe2c1522d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:38.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.862+0000 7efe2b7fe640 1 --2- 172.21.15.67:0/3850395437 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efe2c15e0f0 0x7efe2c0c9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:38.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.862+0000 7efe23fff640 1 -- 172.21.15.67:0/3850395437 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efe2c15e0f0 msgr2=0x7efe2c0c9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:38.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.862+0000 7efe23fff640 1 --2- 172.21.15.67:0/3850395437 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efe2c15e0f0 0x7efe2c0c9810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:38.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.862+0000 7efe23fff640 1 -- 172.21.15.67:0/3850395437 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efe2c0b6dc0 msgr2=0x7efe2c1522d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:38.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.862+0000 7efe23fff640 1 --2- 172.21.15.67:0/3850395437 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efe2c0b6dc0 0x7efe2c1522d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:38.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.862+0000 7efe23fff640 1 -- 172.21.15.67:0/3850395437 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe2c0b53f0 con 0x7efe2c0d32d0 2024-03-24T00:44:38.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.862+0000 7efe2b7fe640 1 --2- 172.21.15.67:0/3850395437 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efe2c15e0f0 0x7efe2c0c9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:44:38.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.862+0000 7efe23fff640 1 --2- 172.21.15.67:0/3850395437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe2c0d32d0 0x7efe2c1f3b00 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7efe1400e900 tx=0x7efe1400edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:38.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.863+0000 7efe237fe640 1 -- 172.21.15.67:0/3850395437 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe14017070 con 0x7efe2c0d32d0 2024-03-24T00:44:38.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.863+0000 7efe237fe640 1 -- 172.21.15.67:0/3850395437 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe1400ca00 con 0x7efe2c0d32d0 2024-03-24T00:44:38.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.863+0000 7efe31787640 1 -- 172.21.15.67:0/3850395437 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe2c0cafe0 con 0x7efe2c0d32d0 2024-03-24T00:44:38.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.863+0000 7efe237fe640 1 -- 172.21.15.67:0/3850395437 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe1400cdb0 con 0x7efe2c0d32d0 2024-03-24T00:44:38.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.863+0000 7efe31787640 1 -- 172.21.15.67:0/3850395437 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe2c0cb5a0 con 0x7efe2c0d32d0 2024-03-24T00:44:38.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.864+0000 7efe31787640 1 -- 172.21.15.67:0/3850395437 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe2c0cadf0 con 0x7efe2c0d32d0 2024-03-24T00:44:38.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.864+0000 7efe237fe640 1 -- 172.21.15.67:0/3850395437 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7efe1401a070 con 0x7efe2c0d32d0 2024-03-24T00:44:38.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.865+0000 7efe237fe640 1 --2- 172.21.15.67:0/3850395437 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7efe0c06bbd0 0x7efe0c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:38.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.865+0000 7efe237fe640 1 -- 172.21.15.67:0/3850395437 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7efe1408af60 con 0x7efe2c0d32d0 2024-03-24T00:44:38.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.865+0000 7efe2affd640 1 --2- 172.21.15.67:0/3850395437 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7efe0c06bbd0 0x7efe0c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:38.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.865+0000 7efe2affd640 1 --2- 172.21.15.67:0/3850395437 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7efe0c06bbd0 0x7efe0c06e090 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7efe1c000f60 tx=0x7efe1c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:38.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:38.869+0000 7efe237fe640 1 -- 172.21.15.67:0/3850395437 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7efe140592b0 con 0x7efe2c0d32d0 2024-03-24T00:44:39.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.062+0000 7efe31787640 1 -- 172.21.15.67:0/3850395437 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7efe2c04f400 con 0x7efe2c0d32d0 2024-03-24T00:44:39.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.062+0000 7efe237fe640 1 -- 172.21.15.67:0/3850395437 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v77) v1 ==== 80+0+122 (secure 0 0 0) 0x7efe1405d350 con 0x7efe2c0d32d0 2024-03-24T00:44:39.063 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6833/2392802741 2024-03-25T00:43:24.373788+0000 2024-03-24T00:44:39.064 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6832/2392802741 2024-03-25T00:43:24.373788+0000 2024-03-24T00:44:39.064 INFO:teuthology.orchestra.run.smithi067.stderr:listed 2 entries 2024-03-24T00:44:39.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.064+0000 7efe31787640 1 -- 172.21.15.67:0/3850395437 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7efe0c06bbd0 msgr2=0x7efe0c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:39.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.064+0000 7efe31787640 1 --2- 172.21.15.67:0/3850395437 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7efe0c06bbd0 0x7efe0c06e090 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7efe1c000f60 tx=0x7efe1c009040 comp rx=0 tx=0).stop 2024-03-24T00:44:39.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.064+0000 7efe31787640 1 -- 172.21.15.67:0/3850395437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe2c0d32d0 msgr2=0x7efe2c1f3b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:39.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.064+0000 7efe31787640 1 --2- 172.21.15.67:0/3850395437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe2c0d32d0 0x7efe2c1f3b00 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7efe1400e900 tx=0x7efe1400edc0 comp rx=0 tx=0).stop 2024-03-24T00:44:39.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.064+0000 7efe31787640 1 -- 172.21.15.67:0/3850395437 shutdown_connections 2024-03-24T00:44:39.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.064+0000 7efe31787640 1 --2- 172.21.15.67:0/3850395437 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7efe0c06bbd0 0x7efe0c06e090 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:39.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.064+0000 7efe31787640 1 --2- 172.21.15.67:0/3850395437 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efe2c15e0f0 0x7efe2c0c9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:39.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.064+0000 7efe31787640 1 --2- 172.21.15.67:0/3850395437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe2c0d32d0 0x7efe2c1f3b00 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:39.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.064+0000 7efe31787640 1 --2- 172.21.15.67:0/3850395437 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efe2c0b6dc0 0x7efe2c1522d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:39.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.064+0000 7efe31787640 1 -- 172.21.15.67:0/3850395437 >> 172.21.15.67:0/3850395437 conn(0x7efe2c072b60 msgr2=0x7efe2c0b9100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:39.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.064+0000 7efe31787640 1 -- 172.21.15.67:0/3850395437 shutdown_connections 2024-03-24T00:44:39.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.064+0000 7efe31787640 1 -- 172.21.15.67:0/3850395437 wait complete. 2024-03-24T00:44:39.076 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-24T00:44:39.076 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:44:39.076 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-24T00:44:39.105 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-03-24T00:44:39.105 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:44:39.105 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-24T00:44:39.160 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:44:39.161 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:44:39.179 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17590 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248808k,nr_inodes=812202,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:44:39.179 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:44:39.179 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:44:39.215 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056375,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15771 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T00:44:39.215 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:44:39.215 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:44:39.237 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17590 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248808k,nr_inodes=812202,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:44:39.237 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:44:39.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.395+0000 7f79c5b53640 1 Processor -- start 2024-03-24T00:44:39.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.395+0000 7f79c5b53640 1 -- start start 2024-03-24T00:44:39.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.396+0000 7f79c5b53640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f79c00d0180 0x7f79c00d0610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:39.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.396+0000 7f79c5b53640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79c00ced20 con 0x7f79c00cf7b0 2024-03-24T00:44:39.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.396+0000 7f79c5b53640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79c00cf000 con 0x7f79c015bcd0 2024-03-24T00:44:39.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.396+0000 7f79c5b53640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79c00d0b50 con 0x7f79c00d0180 2024-03-24T00:44:39.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.396+0000 7f79beffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f79c00cf7b0 0x7f79c00cc760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47834/0 (socket says 172.21.15.67:47834) 2024-03-24T00:44:39.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.396+0000 7f79beffd640 1 -- 172.21.15.67:0/1445717326 learned_addr learned my addr 172.21.15.67:0/1445717326 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:44:39.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.396+0000 7f79bffff640 1 --2- 172.21.15.67:0/1445717326 >> v2:172.21.15.72:3300/0 conn(0x7f79c00d0180 0x7f79c00d0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:39.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.396+0000 7f79be7fc640 1 -- 172.21.15.67:0/1445717326 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 454982388 0 0) 0x7f79c00ced20 con 0x7f79c00cf7b0 2024-03-24T00:44:39.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.396+0000 7f79be7fc640 1 -- 172.21.15.67:0/1445717326 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79ac003520 con 0x7f79c00cf7b0 2024-03-24T00:44:39.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.396+0000 7f79be7fc640 1 -- 172.21.15.67:0/1445717326 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2418876618 0 0) 0x7f79ac003520 con 0x7f79c00cf7b0 2024-03-24T00:44:39.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.396+0000 7f79be7fc640 1 -- 172.21.15.67:0/1445717326 >> v2:172.21.15.72:3300/0 conn(0x7f79c00d0180 msgr2=0x7f79c00d0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:39.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.396+0000 7f79be7fc640 1 --2- 172.21.15.67:0/1445717326 >> v2:172.21.15.72:3300/0 conn(0x7f79c00d0180 0x7f79c00d0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:39.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.396+0000 7f79be7fc640 1 -- 172.21.15.67:0/1445717326 >> v1:172.21.15.72:6789/0 conn(0x7f79c015bcd0 legacy=0x7f79c00ce540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:39.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.397+0000 7f79be7fc640 1 -- 172.21.15.67:0/1445717326 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79c00cf2e0 con 0x7f79c00cf7b0 2024-03-24T00:44:39.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.397+0000 7f79be7fc640 1 -- 172.21.15.67:0/1445717326 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f79b0001670 con 0x7f79c00cf7b0 2024-03-24T00:44:39.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.397+0000 7f79be7fc640 1 -- 172.21.15.67:0/1445717326 >> v1:172.21.15.67:6789/0 conn(0x7f79c00cf7b0 legacy=0x7f79c00cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:39.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.397+0000 7f79be7fc640 1 --2- 172.21.15.67:0/1445717326 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f79ac0039b0 0x7f79ac003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:39.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.397+0000 7f79be7fc640 1 --2- 172.21.15.67:0/1445717326 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79ac004800 0x7f79ac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:39.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.397+0000 7f79be7fc640 1 --2- 172.21.15.67:0/1445717326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79ac0051c0 0x7f79ac00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:39.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.397+0000 7f79be7fc640 1 -- 172.21.15.67:0/1445717326 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79c00d0b50 con 0x7f79ac0051c0 2024-03-24T00:44:39.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.397+0000 7f79be7fc640 1 -- 172.21.15.67:0/1445717326 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79ac00c060 con 0x7f79ac004800 2024-03-24T00:44:39.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.397+0000 7f79be7fc640 1 -- 172.21.15.67:0/1445717326 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79ac00c260 con 0x7f79ac0039b0 2024-03-24T00:44:39.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.397+0000 7f79bffff640 1 --2- 172.21.15.67:0/1445717326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79ac0051c0 0x7f79ac00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:39.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.397+0000 7f79beffd640 1 --2- 172.21.15.67:0/1445717326 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79ac004800 0x7f79ac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:39.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.397+0000 7f79bf7fe640 1 --2- 172.21.15.67:0/1445717326 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f79ac0039b0 0x7f79ac003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:39.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.397+0000 7f79bffff640 1 -- 172.21.15.67:0/1445717326 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f79ac0039b0 msgr2=0x7f79ac003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:39.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.397+0000 7f79bffff640 1 --2- 172.21.15.67:0/1445717326 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f79ac0039b0 0x7f79ac003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:39.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.397+0000 7f79bffff640 1 -- 172.21.15.67:0/1445717326 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79ac004800 msgr2=0x7f79ac004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:39.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.397+0000 7f79bffff640 1 --2- 172.21.15.67:0/1445717326 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79ac004800 0x7f79ac004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:39.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.397+0000 7f79bffff640 1 -- 172.21.15.67:0/1445717326 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79ac003520 con 0x7f79ac0051c0 2024-03-24T00:44:39.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.398+0000 7f79bffff640 1 -- 172.21.15.67:0/1445717326 reap_dead start 2024-03-24T00:44:39.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.398+0000 7f79bffff640 1 --2- 172.21.15.67:0/1445717326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79ac0051c0 0x7f79ac00b9c0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f79b8009fc0 tx=0x7f79ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9fa217aa1721874 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:39.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.398+0000 7f79be7fc640 1 -- 172.21.15.67:0/1445717326 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b8009c10 con 0x7f79ac0051c0 2024-03-24T00:44:39.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.398+0000 7f79be7fc640 1 -- 172.21.15.67:0/1445717326 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79b802cca0 con 0x7f79ac0051c0 2024-03-24T00:44:39.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.398+0000 7f79be7fc640 1 -- 172.21.15.67:0/1445717326 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b8003bd0 con 0x7f79ac0051c0 2024-03-24T00:44:39.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.398+0000 7f79c5b53640 1 -- 172.21.15.67:0/1445717326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79ac0051c0 msgr2=0x7f79ac00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:39.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.398+0000 7f79c5b53640 1 --2- 172.21.15.67:0/1445717326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79ac0051c0 0x7f79ac00b9c0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f79b8009fc0 tx=0x7f79ac005830 comp rx=0 tx=0).stop 2024-03-24T00:44:39.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.399+0000 7f79c5b53640 1 -- 172.21.15.67:0/1445717326 shutdown_connections 2024-03-24T00:44:39.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.399+0000 7f79c5b53640 1 --2- 172.21.15.67:0/1445717326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79ac0051c0 0x7f79ac00b9c0 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:39.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.399+0000 7f79c5b53640 1 -- 172.21.15.67:0/1445717326 >> 172.21.15.67:0/1445717326 conn(0x7f79c008b1b0 msgr2=0x7f79c008b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:39.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.399+0000 7f79c5b53640 1 -- 172.21.15.67:0/1445717326 shutdown_connections 2024-03-24T00:44:39.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.399+0000 7f79c5b53640 1 -- 172.21.15.67:0/1445717326 wait complete. 2024-03-24T00:44:39.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.400+0000 7f79c5b53640 1 Processor -- start 2024-03-24T00:44:39.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.400+0000 7f79c5b53640 1 -- start start 2024-03-24T00:44:39.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.401+0000 7f79c5b53640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79ac0051c0 0x7f79c01487c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:39.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.401+0000 7f79c5b53640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f79c0158fe0 0x7f79c00b7030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:39.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.401+0000 7f79c5b53640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79c0148ea0 0x7f79c0149320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:39.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.401+0000 7f79c5b53640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79c00d0610 con 0x7f79ac0051c0 2024-03-24T00:44:39.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.401+0000 7f79c5b53640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79c00d0790 con 0x7f79c0148ea0 2024-03-24T00:44:39.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.401+0000 7f79c5b53640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79c00b5f90 con 0x7f79c0158fe0 2024-03-24T00:44:39.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.401+0000 7f79bf7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79ac0051c0 0x7f79c01487c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:39.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.401+0000 7f79bf7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79ac0051c0 0x7f79c01487c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55408/0 (socket says 172.21.15.67:55408) 2024-03-24T00:44:39.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.401+0000 7f79bf7fe640 1 -- 172.21.15.67:0/2870535761 learned_addr learned my addr 172.21.15.67:0/2870535761 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:39.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.401+0000 7f79beffd640 1 --2- 172.21.15.67:0/2870535761 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f79c0158fe0 0x7f79c00b7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:39.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.401+0000 7f79bffff640 1 --2- 172.21.15.67:0/2870535761 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79c0148ea0 0x7f79c0149320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:39.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.401+0000 7f79bf7fe640 1 -- 172.21.15.67:0/2870535761 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f79c0158fe0 msgr2=0x7f79c00b7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:39.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.401+0000 7f79bf7fe640 1 --2- 172.21.15.67:0/2870535761 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f79c0158fe0 0x7f79c00b7030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:39.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.401+0000 7f79bf7fe640 1 -- 172.21.15.67:0/2870535761 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79c0148ea0 msgr2=0x7f79c0149320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:39.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.401+0000 7f79bf7fe640 1 --2- 172.21.15.67:0/2870535761 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79c0148ea0 0x7f79c0149320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:39.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.401+0000 7f79bf7fe640 1 -- 172.21.15.67:0/2870535761 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79c00b6270 con 0x7f79ac0051c0 2024-03-24T00:44:39.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.401+0000 7f79beffd640 1 --2- 172.21.15.67:0/2870535761 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f79c0158fe0 0x7f79c00b7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:44:39.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.401+0000 7f79bf7fe640 1 --2- 172.21.15.67:0/2870535761 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79ac0051c0 0x7f79c01487c0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f79a800f5f0 tx=0x7f79a800fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:39.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.401+0000 7f79b7fff640 1 -- 172.21.15.67:0/2870535761 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a80052e0 con 0x7f79ac0051c0 2024-03-24T00:44:39.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.401+0000 7f79b7fff640 1 -- 172.21.15.67:0/2870535761 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79a8018070 con 0x7f79ac0051c0 2024-03-24T00:44:39.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.401+0000 7f79c5b53640 1 -- 172.21.15.67:0/2870535761 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79c00b6550 con 0x7f79ac0051c0 2024-03-24T00:44:39.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.401+0000 7f79b7fff640 1 -- 172.21.15.67:0/2870535761 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a8003810 con 0x7f79ac0051c0 2024-03-24T00:44:39.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.401+0000 7f79c5b53640 1 -- 172.21.15.67:0/2870535761 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79c00b6b10 con 0x7f79ac0051c0 2024-03-24T00:44:39.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.402+0000 7f79c5b53640 1 -- 172.21.15.67:0/2870535761 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79c004f400 con 0x7f79ac0051c0 2024-03-24T00:44:39.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.403+0000 7f79b7fff640 1 -- 172.21.15.67:0/2870535761 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f79a80039b0 con 0x7f79ac0051c0 2024-03-24T00:44:39.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.403+0000 7f79b7fff640 1 --2- 172.21.15.67:0/2870535761 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f79a006bbd0 0x7f79a006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:39.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.403+0000 7f79b7fff640 1 -- 172.21.15.67:0/2870535761 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f79a808aac0 con 0x7f79ac0051c0 2024-03-24T00:44:39.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.403+0000 7f79beffd640 1 --2- 172.21.15.67:0/2870535761 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f79a006bbd0 0x7f79a006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:39.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.404+0000 7f79beffd640 1 --2- 172.21.15.67:0/2870535761 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f79a006bbd0 0x7f79a006e090 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f79c009fa10 tx=0x7f79b0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:39.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.408+0000 7f79b7fff640 1 -- 172.21.15.67:0/2870535761 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f79a8059350 con 0x7f79ac0051c0 2024-03-24T00:44:39.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.633+0000 7f79c5b53640 1 -- 172.21.15.67:0/2870535761 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f79c00b30e0 con 0x7f79ac0051c0 2024-03-24T00:44:39.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.634+0000 7f79b7fff640 1 -- 172.21.15.67:0/2870535761 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+4978 (secure 0 0 0) 0x7f79a8058c50 con 0x7f79ac0051c0 2024-03-24T00:44:39.635 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:44:39.635 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":48,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5960,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1888378520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1888378520},{"type":"v1","addr":"172.21.15.67:6835","nonce":1888378520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6610,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/2539718386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2539718386},{"type":"v1","addr":"172.21.15.72:6836","nonce":2539718386}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6613,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2376370770","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2376370770},{"type":"v1","addr":"172.21.15.72:6837","nonce":2376370770}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:43:10.177088+0000","modified":"2024-03-24T00:44:11.389921+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7149},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7149":{"gid":7149,"name":"c","rank":0,"incarnation":43,"state":"up:active","state_seq":15,"addr":"172.21.15.67:6833/720256752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":720256752},{"type":"v1","addr":"172.21.15.67:6833","nonce":720256752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7149,"qdb_cluster":[7149]},"id":5}]} 2024-03-24T00:44:39.635 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 48 2024-03-24T00:44:39.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.636+0000 7f79c5b53640 1 -- 172.21.15.67:0/2870535761 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f79a006bbd0 msgr2=0x7f79a006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:39.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.636+0000 7f79c5b53640 1 --2- 172.21.15.67:0/2870535761 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f79a006bbd0 0x7f79a006e090 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f79c009fa10 tx=0x7f79b0009040 comp rx=0 tx=0).stop 2024-03-24T00:44:39.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.636+0000 7f79c5b53640 1 -- 172.21.15.67:0/2870535761 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79ac0051c0 msgr2=0x7f79c01487c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:39.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.636+0000 7f79c5b53640 1 --2- 172.21.15.67:0/2870535761 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79ac0051c0 0x7f79c01487c0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f79a800f5f0 tx=0x7f79a800fab0 comp rx=0 tx=0).stop 2024-03-24T00:44:39.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.636+0000 7f79c5b53640 1 -- 172.21.15.67:0/2870535761 shutdown_connections 2024-03-24T00:44:39.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.636+0000 7f79c5b53640 1 --2- 172.21.15.67:0/2870535761 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f79a006bbd0 0x7f79a006e090 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:39.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.636+0000 7f79c5b53640 1 --2- 172.21.15.67:0/2870535761 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79c0148ea0 0x7f79c0149320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:39.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.636+0000 7f79c5b53640 1 --2- 172.21.15.67:0/2870535761 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f79c0158fe0 0x7f79c00b7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:39.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.636+0000 7f79c5b53640 1 --2- 172.21.15.67:0/2870535761 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79ac0051c0 0x7f79c01487c0 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:39.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.636+0000 7f79c5b53640 1 -- 172.21.15.67:0/2870535761 >> 172.21.15.67:0/2870535761 conn(0x7f79c008b1b0 msgr2=0x7f79c00b1f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:39.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.636+0000 7f79c5b53640 1 -- 172.21.15.67:0/2870535761 shutdown_connections 2024-03-24T00:44:39.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.636+0000 7f79c5b53640 1 -- 172.21.15.67:0/2870535761 wait complete. 2024-03-24T00:44:39.651 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:44:39.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.752+0000 7f1c26af1640 1 Processor -- start 2024-03-24T00:44:39.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.752+0000 7f1c26af1640 1 -- start start 2024-03-24T00:44:39.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.753+0000 7f1c26af1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1c2015e2a0 0x7f1c200ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:39.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.753+0000 7f1c26af1640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1c200d0ef0 0x7f1c20162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:39.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.753+0000 7f1c26af1640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c200cf640 con 0x7f1c2015dec0 2024-03-24T00:44:39.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.753+0000 7f1c26af1640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c200cee30 con 0x7f1c200d0ef0 2024-03-24T00:44:39.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.753+0000 7f1c26af1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c200d2120 con 0x7f1c2015e2a0 2024-03-24T00:44:39.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.753+0000 7f1c2488d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1c2015dec0 0x7f1c200d07e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47846/0 (socket says 172.21.15.67:47846) 2024-03-24T00:44:39.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.753+0000 7f1c2488d640 1 -- 172.21.15.67:0/429019567 learned_addr learned my addr 172.21.15.67:0/429019567 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:44:39.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.753+0000 7f1c1ffff640 1 --2- 172.21.15.67:0/429019567 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1c2015e2a0 0x7f1c200ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:39.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.753+0000 7f1c2508e640 1 --2- 172.21.15.67:0/429019567 >> v2:172.21.15.72:3300/0 conn(0x7f1c200d0ef0 0x7f1c20162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:39.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.753+0000 7f1c1f7fe640 1 -- 172.21.15.67:0/429019567 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1273392436 0 0) 0x7f1c14001110 con 0x7f1c2015dec0 2024-03-24T00:44:39.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.753+0000 7f1c1f7fe640 1 -- 172.21.15.67:0/429019567 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c10003520 con 0x7f1c2015dec0 2024-03-24T00:44:39.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.754+0000 7f1c1f7fe640 1 -- 172.21.15.67:0/429019567 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1960025688 0 0) 0x7f1c10003520 con 0x7f1c2015dec0 2024-03-24T00:44:39.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.754+0000 7f1c1ffff640 1 -- 172.21.15.67:0/429019567 >> v2:172.21.15.72:3300/0 conn(0x7f1c200d0ef0 msgr2=0x7f1c20162630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:39.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.754+0000 7f1c1ffff640 1 --2- 172.21.15.67:0/429019567 >> v2:172.21.15.72:3300/0 conn(0x7f1c200d0ef0 0x7f1c20162630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:39.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.754+0000 7f1c1ffff640 1 -- 172.21.15.67:0/429019567 >> v1:172.21.15.67:6789/0 conn(0x7f1c2015dec0 legacy=0x7f1c200d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:39.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.754+0000 7f1c1ffff640 1 -- 172.21.15.67:0/429019567 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c200cf080 con 0x7f1c2015e2a0 2024-03-24T00:44:39.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.754+0000 7f1c2508e640 1 --2- 172.21.15.67:0/429019567 >> v2:172.21.15.72:3300/0 conn(0x7f1c200d0ef0 0x7f1c20162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:44:39.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.754+0000 7f1c1ffff640 1 --2- 172.21.15.67:0/429019567 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1c2015e2a0 0x7f1c200ce630 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c0097a0 tx=0x7f1c0c02b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e922f888494e78e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:39.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.755+0000 7f1c1f7fe640 1 -- 172.21.15.67:0/429019567 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c0c037070 con 0x7f1c2015e2a0 2024-03-24T00:44:39.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.755+0000 7f1c1f7fe640 1 -- 172.21.15.67:0/429019567 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c0c030510 con 0x7f1c2015e2a0 2024-03-24T00:44:39.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.755+0000 7f1c1f7fe640 1 -- 172.21.15.67:0/429019567 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c14001840 con 0x7f1c2015e2a0 2024-03-24T00:44:39.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.755+0000 7f1c26af1640 1 -- 172.21.15.67:0/429019567 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1c2015e2a0 msgr2=0x7f1c200ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:39.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.755+0000 7f1c26af1640 1 --2- 172.21.15.67:0/429019567 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1c2015e2a0 0x7f1c200ce630 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c0097a0 tx=0x7f1c0c02b710 comp rx=0 tx=0).stop 2024-03-24T00:44:39.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.755+0000 7f1c26af1640 1 -- 172.21.15.67:0/429019567 shutdown_connections 2024-03-24T00:44:39.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.755+0000 7f1c26af1640 1 --2- 172.21.15.67:0/429019567 >> v2:172.21.15.72:3300/0 conn(0x7f1c200d0ef0 0x7f1c20162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:39.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.755+0000 7f1c26af1640 1 --2- 172.21.15.67:0/429019567 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1c2015e2a0 0x7f1c200ce630 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:39.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.755+0000 7f1c26af1640 1 -- 172.21.15.67:0/429019567 >> 172.21.15.67:0/429019567 conn(0x7f1c2006c810 msgr2=0x7f1c2006cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:39.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.755+0000 7f1c26af1640 1 -- 172.21.15.67:0/429019567 shutdown_connections 2024-03-24T00:44:39.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.755+0000 7f1c26af1640 1 -- 172.21.15.67:0/429019567 wait complete. 2024-03-24T00:44:39.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.756+0000 7f1c26af1640 1 Processor -- start 2024-03-24T00:44:39.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.756+0000 7f1c26af1640 1 -- start start 2024-03-24T00:44:39.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.757+0000 7f1c26af1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c200d0ef0 0x7f1c200ab0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:39.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.757+0000 7f1c26af1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c2015dec0 0x7f1c200b5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:39.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.757+0000 7f1c26af1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1c2015e2a0 0x7f1c200b72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:39.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.757+0000 7f1c26af1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c200d2730 con 0x7f1c2015dec0 2024-03-24T00:44:39.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.757+0000 7f1c26af1640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c200d25b0 con 0x7f1c200d0ef0 2024-03-24T00:44:39.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.757+0000 7f1c26af1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c200d28b0 con 0x7f1c2015e2a0 2024-03-24T00:44:39.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.757+0000 7f1c1ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c2015dec0 0x7f1c200b5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:39.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.757+0000 7f1c1ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c2015dec0 0x7f1c200b5990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55422/0 (socket says 172.21.15.67:55422) 2024-03-24T00:44:39.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.757+0000 7f1c1ffff640 1 -- 172.21.15.67:0/1373955746 learned_addr learned my addr 172.21.15.67:0/1373955746 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:39.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.757+0000 7f1c2488d640 1 --2- 172.21.15.67:0/1373955746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c200d0ef0 0x7f1c200ab0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:39.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.757+0000 7f1c2508e640 1 --2- 172.21.15.67:0/1373955746 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1c2015e2a0 0x7f1c200b72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:39.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.758+0000 7f1c1ffff640 1 -- 172.21.15.67:0/1373955746 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1c2015e2a0 msgr2=0x7f1c200b72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:39.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.758+0000 7f1c1ffff640 1 --2- 172.21.15.67:0/1373955746 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1c2015e2a0 0x7f1c200b72e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:39.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.758+0000 7f1c1ffff640 1 -- 172.21.15.67:0/1373955746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c200d0ef0 msgr2=0x7f1c200ab0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:39.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.758+0000 7f1c1ffff640 1 --2- 172.21.15.67:0/1373955746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c200d0ef0 0x7f1c200ab0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:39.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.758+0000 7f1c1ffff640 1 -- 172.21.15.67:0/1373955746 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c200b5ed0 con 0x7f1c2015dec0 2024-03-24T00:44:39.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.758+0000 7f1c1ffff640 1 --2- 172.21.15.67:0/1373955746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c2015dec0 0x7f1c200b5990 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c000c00 tx=0x7f1c0c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:39.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.758+0000 7f1c1d7fa640 1 -- 172.21.15.67:0/1373955746 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c0c042070 con 0x7f1c2015dec0 2024-03-24T00:44:39.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.758+0000 7f1c1d7fa640 1 -- 172.21.15.67:0/1373955746 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c0c03e870 con 0x7f1c2015dec0 2024-03-24T00:44:39.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.758+0000 7f1c26af1640 1 -- 172.21.15.67:0/1373955746 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c200ab780 con 0x7f1c2015dec0 2024-03-24T00:44:39.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.758+0000 7f1c1d7fa640 1 -- 172.21.15.67:0/1373955746 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c0c037040 con 0x7f1c2015dec0 2024-03-24T00:44:39.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.758+0000 7f1c26af1640 1 -- 172.21.15.67:0/1373955746 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c200ac020 con 0x7f1c2015dec0 2024-03-24T00:44:39.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.759+0000 7f1c26af1640 1 -- 172.21.15.67:0/1373955746 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c2004f400 con 0x7f1c2015dec0 2024-03-24T00:44:39.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.760+0000 7f1c1d7fa640 1 -- 172.21.15.67:0/1373955746 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1c0c045070 con 0x7f1c2015dec0 2024-03-24T00:44:39.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.760+0000 7f1c1d7fa640 1 --2- 172.21.15.67:0/1373955746 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1c0406bc00 0x7f1c0406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:39.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.760+0000 7f1c1d7fa640 1 -- 172.21.15.67:0/1373955746 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f1c0c0ac740 con 0x7f1c2015dec0 2024-03-24T00:44:39.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.760+0000 7f1c2488d640 1 --2- 172.21.15.67:0/1373955746 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1c0406bc00 0x7f1c0406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:39.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.761+0000 7f1c2488d640 1 --2- 172.21.15.67:0/1373955746 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1c0406bc00 0x7f1c0406e0c0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f1c14001080 tx=0x7f1c14001010 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:39.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.768+0000 7f1c1d7fa640 1 -- 172.21.15.67:0/1373955746 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1c0c07abb0 con 0x7f1c2015dec0 2024-03-24T00:44:39.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.952+0000 7f1c26af1640 1 -- 172.21.15.67:0/1373955746 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c200683b0 con 0x7f1c2015dec0 2024-03-24T00:44:39.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.953+0000 7f1c1d7fa640 1 -- 172.21.15.67:0/1373955746 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+17014 (secure 0 0 0) 0x7f1c0c07ec50 con 0x7f1c2015dec0 2024-03-24T00:44:39.954 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:44:39.954 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":77,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:43:24.373819+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:43:08.452726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-24T00:43:09.146431+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6833/2392802741":"2024-03-25T00:43:24.373788+0000","172.21.15.67:6832/2392802741":"2024-03-25T00:43:24.373788+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:44:39.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.955+0000 7f1c26af1640 1 -- 172.21.15.67:0/1373955746 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1c0406bc00 msgr2=0x7f1c0406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:39.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.955+0000 7f1c26af1640 1 --2- 172.21.15.67:0/1373955746 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1c0406bc00 0x7f1c0406e0c0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f1c14001080 tx=0x7f1c14001010 comp rx=0 tx=0).stop 2024-03-24T00:44:39.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.955+0000 7f1c26af1640 1 -- 172.21.15.67:0/1373955746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c2015dec0 msgr2=0x7f1c200b5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:39.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.955+0000 7f1c26af1640 1 --2- 172.21.15.67:0/1373955746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c2015dec0 0x7f1c200b5990 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c000c00 tx=0x7f1c0c0046c0 comp rx=0 tx=0).stop 2024-03-24T00:44:39.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.955+0000 7f1c26af1640 1 -- 172.21.15.67:0/1373955746 shutdown_connections 2024-03-24T00:44:39.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.955+0000 7f1c26af1640 1 --2- 172.21.15.67:0/1373955746 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1c0406bc00 0x7f1c0406e0c0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:39.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.955+0000 7f1c26af1640 1 --2- 172.21.15.67:0/1373955746 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1c2015e2a0 0x7f1c200b72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:39.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.955+0000 7f1c26af1640 1 --2- 172.21.15.67:0/1373955746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c2015dec0 0x7f1c200b5990 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:39.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.955+0000 7f1c26af1640 1 --2- 172.21.15.67:0/1373955746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c200d0ef0 0x7f1c200ab0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:39.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.955+0000 7f1c26af1640 1 -- 172.21.15.67:0/1373955746 >> 172.21.15.67:0/1373955746 conn(0x7f1c2006c810 msgr2=0x7f1c2015d3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:39.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.955+0000 7f1c26af1640 1 -- 172.21.15.67:0/1373955746 shutdown_connections 2024-03-24T00:44:39.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:39.955+0000 7f1c26af1640 1 -- 172.21.15.67:0/1373955746 wait complete. 2024-03-24T00:44:39.969 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:38:39.364950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:43:08.452726+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:43:09.146431+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:44:39.969 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:44:40.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.075+0000 7fb709b37640 1 Processor -- start 2024-03-24T00:44:40.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.075+0000 7fb709b37640 1 -- start start 2024-03-24T00:44:40.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.076+0000 7fb709b37640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb7040b6170 0x7fb704168c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:40.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.076+0000 7fb709b37640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7040d2250 con 0x7fb70415dea0 2024-03-24T00:44:40.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.076+0000 7fb709b37640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7040d2530 con 0x7fb7040d3ae0 2024-03-24T00:44:40.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.076+0000 7fb709b37640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7040d2810 con 0x7fb7040b6170 2024-03-24T00:44:40.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.076+0000 7fb7037fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb70415dea0 0x7fb7040d33d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47862/0 (socket says 172.21.15.67:47862) 2024-03-24T00:44:40.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.076+0000 7fb7037fe640 1 -- 172.21.15.67:0/309686127 learned_addr learned my addr 172.21.15.67:0/309686127 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:44:40.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.076+0000 7fb703fff640 1 --2- 172.21.15.67:0/309686127 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb7040b6170 0x7fb704168c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:40.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.076+0000 7fb7027fc640 1 -- 172.21.15.67:0/309686127 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2252804137 0 0) 0x7fb7040d2250 con 0x7fb70415dea0 2024-03-24T00:44:40.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.077+0000 7fb7027fc640 1 -- 172.21.15.67:0/309686127 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6f0003520 con 0x7fb70415dea0 2024-03-24T00:44:40.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.077+0000 7fb7027fc640 1 -- 172.21.15.67:0/309686127 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 863345148 0 0) 0x7fb7040d2530 con 0x7fb7040d3ae0 2024-03-24T00:44:40.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.077+0000 7fb7027fc640 1 -- 172.21.15.67:0/309686127 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7040d2250 con 0x7fb7040d3ae0 2024-03-24T00:44:40.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.077+0000 7fb703fff640 1 -- 172.21.15.67:0/309686127 >> v1:172.21.15.72:6789/0 conn(0x7fb7040d3ae0 legacy=0x7fb7040b5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:40.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.077+0000 7fb703fff640 1 -- 172.21.15.67:0/309686127 >> v1:172.21.15.67:6789/0 conn(0x7fb70415dea0 legacy=0x7fb7040d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:40.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.077+0000 7fb7027fc640 1 -- 172.21.15.67:0/309686127 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3063705055 0 0) 0x7fb6f0003520 con 0x7fb70415dea0 2024-03-24T00:44:40.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.077+0000 7fb703fff640 1 -- 172.21.15.67:0/309686127 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7040d2af0 con 0x7fb7040b6170 2024-03-24T00:44:40.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.078+0000 7fb703fff640 1 --2- 172.21.15.67:0/309686127 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb7040b6170 0x7fb704168c80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc009df0 tx=0x7fb6fc02bd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a6501ac710e7026 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:40.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.078+0000 7fb7027fc640 1 -- 172.21.15.67:0/309686127 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6fc039070 con 0x7fb7040b6170 2024-03-24T00:44:40.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.078+0000 7fb7027fc640 1 -- 172.21.15.67:0/309686127 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6fc02c880 con 0x7fb7040b6170 2024-03-24T00:44:40.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.078+0000 7fb7027fc640 1 -- 172.21.15.67:0/309686127 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6ec001670 con 0x7fb7040b6170 2024-03-24T00:44:40.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.078+0000 7fb709b37640 1 -- 172.21.15.67:0/309686127 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb7040b6170 msgr2=0x7fb704168c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:40.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.078+0000 7fb709b37640 1 --2- 172.21.15.67:0/309686127 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb7040b6170 0x7fb704168c80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc009df0 tx=0x7fb6fc02bd50 comp rx=0 tx=0).stop 2024-03-24T00:44:40.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.078+0000 7fb709b37640 1 -- 172.21.15.67:0/309686127 shutdown_connections 2024-03-24T00:44:40.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.078+0000 7fb709b37640 1 --2- 172.21.15.67:0/309686127 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb7040b6170 0x7fb704168c80 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:40.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.079+0000 7fb709b37640 1 -- 172.21.15.67:0/309686127 >> 172.21.15.67:0/309686127 conn(0x7fb704072da0 msgr2=0x7fb7040d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:40.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.079+0000 7fb709b37640 1 -- 172.21.15.67:0/309686127 shutdown_connections 2024-03-24T00:44:40.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.079+0000 7fb709b37640 1 -- 172.21.15.67:0/309686127 wait complete. 2024-03-24T00:44:40.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.080+0000 7fb709b37640 1 Processor -- start 2024-03-24T00:44:40.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.081+0000 7fb709b37640 1 -- start start 2024-03-24T00:44:40.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.082+0000 7fb709b37640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7040b6170 0x7fb7041f3b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:40.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.082+0000 7fb709b37640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb7040d3ae0 0x7fb7040cd100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:40.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.082+0000 7fb709b37640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb70415dea0 0x7fb7040cd640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:40.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.082+0000 7fb709b37640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb70415e280 con 0x7fb7040b6170 2024-03-24T00:44:40.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.082+0000 7fb709b37640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7040b8310 con 0x7fb70415dea0 2024-03-24T00:44:40.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.082+0000 7fb709b37640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb70415e400 con 0x7fb7040d3ae0 2024-03-24T00:44:40.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.082+0000 7fb7037fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7040b6170 0x7fb7041f3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:40.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.082+0000 7fb7037fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7040b6170 0x7fb7041f3b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55428/0 (socket says 172.21.15.67:55428) 2024-03-24T00:44:40.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.082+0000 7fb7037fe640 1 -- 172.21.15.67:0/4280236358 learned_addr learned my addr 172.21.15.67:0/4280236358 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:40.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.082+0000 7fb702ffd640 1 --2- 172.21.15.67:0/4280236358 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb7040d3ae0 0x7fb7040cd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:40.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.082+0000 7fb703fff640 1 --2- 172.21.15.67:0/4280236358 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb70415dea0 0x7fb7040cd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:40.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.082+0000 7fb7037fe640 1 -- 172.21.15.67:0/4280236358 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb7040d3ae0 msgr2=0x7fb7040cd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:40.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.082+0000 7fb7037fe640 1 --2- 172.21.15.67:0/4280236358 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb7040d3ae0 0x7fb7040cd100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:40.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.082+0000 7fb7037fe640 1 -- 172.21.15.67:0/4280236358 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb70415dea0 msgr2=0x7fb7040cd640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:40.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.082+0000 7fb7037fe640 1 --2- 172.21.15.67:0/4280236358 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb70415dea0 0x7fb7040cd640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:40.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.082+0000 7fb7037fe640 1 -- 172.21.15.67:0/4280236358 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb704147c60 con 0x7fb7040b6170 2024-03-24T00:44:40.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.082+0000 7fb7037fe640 1 --2- 172.21.15.67:0/4280236358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7040b6170 0x7fb7041f3b10 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec00c550 tx=0x7fb6ec00ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:40.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.083+0000 7fb6fbfff640 1 -- 172.21.15.67:0/4280236358 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6ec017070 con 0x7fb7040b6170 2024-03-24T00:44:40.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.083+0000 7fb6fbfff640 1 -- 172.21.15.67:0/4280236358 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6ec005d00 con 0x7fb7040b6170 2024-03-24T00:44:40.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.083+0000 7fb709b37640 1 -- 172.21.15.67:0/4280236358 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb70415a790 con 0x7fb7040b6170 2024-03-24T00:44:40.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.083+0000 7fb6fbfff640 1 -- 172.21.15.67:0/4280236358 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6ec0072b0 con 0x7fb7040b6170 2024-03-24T00:44:40.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.083+0000 7fb709b37640 1 -- 172.21.15.67:0/4280236358 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7040cb5e0 con 0x7fb7040b6170 2024-03-24T00:44:40.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.084+0000 7fb709b37640 1 -- 172.21.15.67:0/4280236358 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb70404f400 con 0x7fb7040b6170 2024-03-24T00:44:40.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.084+0000 7fb6fbfff640 1 -- 172.21.15.67:0/4280236358 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb6ec01a070 con 0x7fb7040b6170 2024-03-24T00:44:40.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.085+0000 7fb6fbfff640 1 --2- 172.21.15.67:0/4280236358 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb6e406bbd0 0x7fb6e406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:40.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.085+0000 7fb6fbfff640 1 -- 172.21.15.67:0/4280236358 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fb6ec08a040 con 0x7fb7040b6170 2024-03-24T00:44:40.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.085+0000 7fb702ffd640 1 --2- 172.21.15.67:0/4280236358 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb6e406bbd0 0x7fb6e406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:40.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.085+0000 7fb702ffd640 1 --2- 172.21.15.67:0/4280236358 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb6e406bbd0 0x7fb6e406e090 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fb70415b240 tx=0x7fb6f4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:40.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.089+0000 7fb6fbfff640 1 -- 172.21.15.67:0/4280236358 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb704147c60 con 0x7fb7040b6170 2024-03-24T00:44:40.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.270+0000 7fb709b37640 1 -- 172.21.15.67:0/4280236358 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb7040b3420 con 0x7fb7040b6170 2024-03-24T00:44:40.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.270+0000 7fb6fbfff640 1 -- 172.21.15.67:0/4280236358 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb6ec058390 con 0x7fb7040b6170 2024-03-24T00:44:40.271 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-24T00:44:40.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.272+0000 7fb709b37640 1 -- 172.21.15.67:0/4280236358 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb6e406bbd0 msgr2=0x7fb6e406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:40.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.272+0000 7fb709b37640 1 --2- 172.21.15.67:0/4280236358 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb6e406bbd0 0x7fb6e406e090 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fb70415b240 tx=0x7fb6f4009040 comp rx=0 tx=0).stop 2024-03-24T00:44:40.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.272+0000 7fb709b37640 1 -- 172.21.15.67:0/4280236358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7040b6170 msgr2=0x7fb7041f3b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:40.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.272+0000 7fb709b37640 1 --2- 172.21.15.67:0/4280236358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7040b6170 0x7fb7041f3b10 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec00c550 tx=0x7fb6ec00ca10 comp rx=0 tx=0).stop 2024-03-24T00:44:40.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.272+0000 7fb709b37640 1 -- 172.21.15.67:0/4280236358 shutdown_connections 2024-03-24T00:44:40.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.272+0000 7fb709b37640 1 --2- 172.21.15.67:0/4280236358 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb6e406bbd0 0x7fb6e406e090 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:40.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.272+0000 7fb709b37640 1 --2- 172.21.15.67:0/4280236358 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb70415dea0 0x7fb7040cd640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:40.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.272+0000 7fb709b37640 1 --2- 172.21.15.67:0/4280236358 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb7040d3ae0 0x7fb7040cd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:40.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.272+0000 7fb709b37640 1 --2- 172.21.15.67:0/4280236358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7040b6170 0x7fb7041f3b10 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:40.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.272+0000 7fb709b37640 1 -- 172.21.15.67:0/4280236358 >> 172.21.15.67:0/4280236358 conn(0x7fb704072da0 msgr2=0x7fb7040d0a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:40.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.272+0000 7fb709b37640 1 -- 172.21.15.67:0/4280236358 shutdown_connections 2024-03-24T00:44:40.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.272+0000 7fb709b37640 1 -- 172.21.15.67:0/4280236358 wait complete. 2024-03-24T00:44:40.284 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:44:40.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.387+0000 7f2cfb33b640 1 Processor -- start 2024-03-24T00:44:40.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.387+0000 7f2cfb33b640 1 -- start start 2024-03-24T00:44:40.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.388+0000 7f2cfb33b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2cf40b2110 0x7f2cf40b2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:40.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.388+0000 7f2cfb33b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2cf415c010 0x7f2cf40b32d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:40.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.388+0000 7f2cfb33b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cf40b3810 con 0x7f2cf415bc30 2024-03-24T00:44:40.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.388+0000 7f2cfb33b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cf40cca60 con 0x7f2cf415c010 2024-03-24T00:44:40.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.388+0000 7f2cfb33b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf40b3ae0 con 0x7f2cf40b2110 2024-03-24T00:44:40.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.388+0000 7f2cf88d6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2cf40b2110 0x7f2cf40b2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:40.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.388+0000 7f2cf98d8640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2cf415c010 0x7f2cf40b32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:40.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.388+0000 7f2cf90d7640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f2cf415bc30 0x7f2cf40b1a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:34092/0 (socket says 172.21.15.67:34092) 2024-03-24T00:44:40.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.388+0000 7f2cf88d6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2cf40b2110 0x7f2cf40b2d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.67:53422/0 (socket says 172.21.15.67:53422) 2024-03-24T00:44:40.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.388+0000 7f2cf90d7640 1 -- 172.21.15.67:0/1099792630 learned_addr learned my addr 172.21.15.67:0/1099792630 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:44:40.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.389+0000 7f2cf98d8640 1 -- 172.21.15.67:0/1099792630 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2cf40b2110 msgr2=0x7f2cf40b2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:40.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.389+0000 7f2cf98d8640 1 --2- 172.21.15.67:0/1099792630 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2cf40b2110 0x7f2cf40b2d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:40.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.389+0000 7f2cf98d8640 1 -- 172.21.15.67:0/1099792630 >> v1:172.21.15.72:6789/0 conn(0x7f2cf415bc30 legacy=0x7f2cf40b1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:40.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.389+0000 7f2cebfff640 1 -- 172.21.15.67:0/1099792630 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1816760904 0 0) 0x7f2cf40b3810 con 0x7f2cf415bc30 2024-03-24T00:44:40.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.389+0000 7f2cf98d8640 1 -- 172.21.15.67:0/1099792630 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cf40b3dc0 con 0x7f2cf415c010 2024-03-24T00:44:40.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.389+0000 7f2cf88d6640 1 --2- 172.21.15.67:0/1099792630 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2cf40b2110 0x7f2cf40b2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:44:40.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.390+0000 7f2cf98d8640 1 --2- 172.21.15.67:0/1099792630 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2cf415c010 0x7f2cf40b32d0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f2cec009d90 tx=0x7f2cec02bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86cdbefbc0363d04 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:40.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.390+0000 7f2cebfff640 1 -- 172.21.15.67:0/1099792630 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cec039070 con 0x7f2cf415c010 2024-03-24T00:44:40.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.390+0000 7f2cebfff640 1 -- 172.21.15.67:0/1099792630 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cec02cc50 con 0x7f2cf415c010 2024-03-24T00:44:40.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.390+0000 7f2cebfff640 1 -- 172.21.15.67:0/1099792630 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cec034440 con 0x7f2cf415c010 2024-03-24T00:44:40.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.390+0000 7f2cfb33b640 1 -- 172.21.15.67:0/1099792630 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2cf415c010 msgr2=0x7f2cf40b32d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:40.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.390+0000 7f2cfb33b640 1 --2- 172.21.15.67:0/1099792630 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2cf415c010 0x7f2cf40b32d0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f2cec009d90 tx=0x7f2cec02bea0 comp rx=0 tx=0).stop 2024-03-24T00:44:40.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.390+0000 7f2cfb33b640 1 -- 172.21.15.67:0/1099792630 shutdown_connections 2024-03-24T00:44:40.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.390+0000 7f2cfb33b640 1 --2- 172.21.15.67:0/1099792630 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2cf415c010 0x7f2cf40b32d0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:40.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.390+0000 7f2cfb33b640 1 --2- 172.21.15.67:0/1099792630 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2cf40b2110 0x7f2cf40b2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:40.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.390+0000 7f2cfb33b640 1 -- 172.21.15.67:0/1099792630 >> 172.21.15.67:0/1099792630 conn(0x7f2cf408b180 msgr2=0x7f2cf408b590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:40.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.390+0000 7f2cfb33b640 1 -- 172.21.15.67:0/1099792630 shutdown_connections 2024-03-24T00:44:40.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.390+0000 7f2cfb33b640 1 -- 172.21.15.67:0/1099792630 wait complete. 2024-03-24T00:44:40.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.391+0000 7f2cfb33b640 1 Processor -- start 2024-03-24T00:44:40.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.392+0000 7f2cfb33b640 1 -- start start 2024-03-24T00:44:40.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.392+0000 7f2cfb33b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cf40b2110 0x7f2cf40c6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:40.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.392+0000 7f2cfb33b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2cf415bc30 0x7f2cf40c7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:40.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.392+0000 7f2cfb33b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2cf415c010 0x7f2cf414e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:40.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.392+0000 7f2cfb33b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf40ccef0 con 0x7f2cf40b2110 2024-03-24T00:44:40.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.392+0000 7f2cfb33b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf40b4380 con 0x7f2cf415c010 2024-03-24T00:44:40.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.392+0000 7f2cfb33b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf40cd070 con 0x7f2cf415bc30 2024-03-24T00:44:40.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.392+0000 7f2cf90d7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cf40b2110 0x7f2cf40c6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:40.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.392+0000 7f2cf90d7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cf40b2110 0x7f2cf40c6ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55430/0 (socket says 172.21.15.67:55430) 2024-03-24T00:44:40.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.392+0000 7f2cf90d7640 1 -- 172.21.15.67:0/2206888036 learned_addr learned my addr 172.21.15.67:0/2206888036 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:40.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.392+0000 7f2cf98d8640 1 --2- 172.21.15.67:0/2206888036 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2cf415c010 0x7f2cf414e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:40.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.392+0000 7f2cf88d6640 1 --2- 172.21.15.67:0/2206888036 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2cf415bc30 0x7f2cf40c7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:40.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.393+0000 7f2cf90d7640 1 -- 172.21.15.67:0/2206888036 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2cf415bc30 msgr2=0x7f2cf40c7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:40.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.393+0000 7f2cf90d7640 1 --2- 172.21.15.67:0/2206888036 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2cf415bc30 0x7f2cf40c7020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:40.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.393+0000 7f2cf90d7640 1 -- 172.21.15.67:0/2206888036 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2cf415c010 msgr2=0x7f2cf414e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:40.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.393+0000 7f2cf90d7640 1 --2- 172.21.15.67:0/2206888036 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2cf415c010 0x7f2cf414e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:40.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.393+0000 7f2cf90d7640 1 -- 172.21.15.67:0/2206888036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cf40c7850 con 0x7f2cf40b2110 2024-03-24T00:44:40.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.393+0000 7f2cf98d8640 1 --2- 172.21.15.67:0/2206888036 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2cf415c010 0x7f2cf414e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:44:40.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.393+0000 7f2cf90d7640 1 --2- 172.21.15.67:0/2206888036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cf40b2110 0x7f2cf40c6ae0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f2cf000c4d0 tx=0x7f2cf000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:40.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.393+0000 7f2ce9ffb640 1 -- 172.21.15.67:0/2206888036 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf0018070 con 0x7f2cf40b2110 2024-03-24T00:44:40.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.393+0000 7f2ce9ffb640 1 -- 172.21.15.67:0/2206888036 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cf0007500 con 0x7f2cf40b2110 2024-03-24T00:44:40.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.393+0000 7f2cfb33b640 1 -- 172.21.15.67:0/2206888036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cf40c7b30 con 0x7f2cf40b2110 2024-03-24T00:44:40.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.393+0000 7f2ce9ffb640 1 -- 172.21.15.67:0/2206888036 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf0004a20 con 0x7f2cf40b2110 2024-03-24T00:44:40.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.393+0000 7f2cfb33b640 1 -- 172.21.15.67:0/2206888036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cf40c80f0 con 0x7f2cf40b2110 2024-03-24T00:44:40.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.394+0000 7f2cfb33b640 1 -- 172.21.15.67:0/2206888036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cf40ae690 con 0x7f2cf40b2110 2024-03-24T00:44:40.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.395+0000 7f2ce9ffb640 1 -- 172.21.15.67:0/2206888036 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2cf0004bc0 con 0x7f2cf40b2110 2024-03-24T00:44:40.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.395+0000 7f2ce9ffb640 1 --2- 172.21.15.67:0/2206888036 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2cd806bbd0 0x7f2cd806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:40.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.395+0000 7f2ce9ffb640 1 -- 172.21.15.67:0/2206888036 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f2cf0089ed0 con 0x7f2cf40b2110 2024-03-24T00:44:40.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.395+0000 7f2cf88d6640 1 --2- 172.21.15.67:0/2206888036 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2cd806bbd0 0x7f2cd806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:40.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.396+0000 7f2cf88d6640 1 --2- 172.21.15.67:0/2206888036 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2cd806bbd0 0x7f2cd806e090 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f2ce4009560 tx=0x7f2ce4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:40.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.404+0000 7f2ce9ffb640 1 -- 172.21.15.67:0/2206888036 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2cf40ae690 con 0x7f2cf40b2110 2024-03-24T00:44:40.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.586+0000 7f2cfb33b640 1 -- 172.21.15.67:0/2206888036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2cf404f400 con 0x7f2cf40b2110 2024-03-24T00:44:40.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.587+0000 7f2ce9ffb640 1 -- 172.21.15.67:0/2206888036 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v77) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2cf00581a0 con 0x7f2cf40b2110 2024-03-24T00:44:40.588 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-24T00:44:40.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.589+0000 7f2cfb33b640 1 -- 172.21.15.67:0/2206888036 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2cd806bbd0 msgr2=0x7f2cd806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:40.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.589+0000 7f2cfb33b640 1 --2- 172.21.15.67:0/2206888036 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2cd806bbd0 0x7f2cd806e090 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f2ce4009560 tx=0x7f2ce4009190 comp rx=0 tx=0).stop 2024-03-24T00:44:40.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.589+0000 7f2cfb33b640 1 -- 172.21.15.67:0/2206888036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cf40b2110 msgr2=0x7f2cf40c6ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:40.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.589+0000 7f2cfb33b640 1 --2- 172.21.15.67:0/2206888036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cf40b2110 0x7f2cf40c6ae0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f2cf000c4d0 tx=0x7f2cf000c990 comp rx=0 tx=0).stop 2024-03-24T00:44:40.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.589+0000 7f2cfb33b640 1 -- 172.21.15.67:0/2206888036 shutdown_connections 2024-03-24T00:44:40.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.589+0000 7f2cfb33b640 1 --2- 172.21.15.67:0/2206888036 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2cd806bbd0 0x7f2cd806e090 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:40.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.589+0000 7f2cfb33b640 1 --2- 172.21.15.67:0/2206888036 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2cf415c010 0x7f2cf414e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:40.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.589+0000 7f2cfb33b640 1 --2- 172.21.15.67:0/2206888036 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2cf415bc30 0x7f2cf40c7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:40.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.589+0000 7f2cfb33b640 1 --2- 172.21.15.67:0/2206888036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cf40b2110 0x7f2cf40c6ae0 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:40.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.589+0000 7f2cfb33b640 1 -- 172.21.15.67:0/2206888036 >> 172.21.15.67:0/2206888036 conn(0x7f2cf408b180 msgr2=0x7f2cf4070de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:40.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.589+0000 7f2cfb33b640 1 -- 172.21.15.67:0/2206888036 shutdown_connections 2024-03-24T00:44:40.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.589+0000 7f2cfb33b640 1 -- 172.21.15.67:0/2206888036 wait complete. 2024-03-24T00:44:40.601 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:44:40.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.705+0000 7f3d9bfa9640 1 Processor -- start 2024-03-24T00:44:40.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.705+0000 7f3d9bfa9640 1 -- start start 2024-03-24T00:44:40.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.706+0000 7f3d9bfa9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3d9415e3a0 0x7f3d940b2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:40.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.706+0000 7f3d9bfa9640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3d941626d0 0x7f3d940ccaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:40.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.706+0000 7f3d9bfa9640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3d940cd030 0x7f3d940b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:40.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.706+0000 7f3d9bfa9640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d940afb80 con 0x7f3d9415e3a0 2024-03-24T00:44:40.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.706+0000 7f3d9bfa9640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d9415f470 con 0x7f3d940cd030 2024-03-24T00:44:40.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.706+0000 7f3d9bfa9640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9415e860 con 0x7f3d941626d0 2024-03-24T00:44:40.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.706+0000 7f3d99d45640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3d9415e3a0 0x7f3d940b2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:40.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.706+0000 7f3d99d45640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3d9415e3a0 0x7f3d940b2890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55438/0 (socket says 172.21.15.67:55438) 2024-03-24T00:44:40.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.706+0000 7f3d99d45640 1 -- 172.21.15.67:0/3586757610 learned_addr learned my addr 172.21.15.67:0/3586757610 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:40.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.706+0000 7f3d9a546640 1 --2- 172.21.15.67:0/3586757610 >> v2:172.21.15.72:3300/0 conn(0x7f3d940cd030 0x7f3d940b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:40.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.706+0000 7f3d99544640 1 --2- 172.21.15.67:0/3586757610 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3d941626d0 0x7f3d940ccaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:40.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.707+0000 7f3d99d45640 1 -- 172.21.15.67:0/3586757610 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3d941626d0 msgr2=0x7f3d940ccaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:40.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.707+0000 7f3d99d45640 1 --2- 172.21.15.67:0/3586757610 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3d941626d0 0x7f3d940ccaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:40.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.707+0000 7f3d99d45640 1 -- 172.21.15.67:0/3586757610 >> v2:172.21.15.72:3300/0 conn(0x7f3d940cd030 msgr2=0x7f3d940b4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:40.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.707+0000 7f3d99d45640 1 --2- 172.21.15.67:0/3586757610 >> v2:172.21.15.72:3300/0 conn(0x7f3d940cd030 0x7f3d940b4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:40.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.707+0000 7f3d99d45640 1 -- 172.21.15.67:0/3586757610 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d9415eb40 con 0x7f3d9415e3a0 2024-03-24T00:44:40.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.707+0000 7f3d9a546640 1 --2- 172.21.15.67:0/3586757610 >> v2:172.21.15.72:3300/0 conn(0x7f3d940cd030 0x7f3d940b4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:44:40.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.708+0000 7f3d99d45640 1 --2- 172.21.15.67:0/3586757610 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d9415e3a0 0x7f3d940b2890 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f3d940d0220 tx=0x7f3d9002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf785b756f9a620e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:40.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.708+0000 7f3d98d43640 1 -- 172.21.15.67:0/3586757610 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d90035020 con 0x7f3d9415e3a0 2024-03-24T00:44:40.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.708+0000 7f3d98d43640 1 -- 172.21.15.67:0/3586757610 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d90031870 con 0x7f3d9415e3a0 2024-03-24T00:44:40.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.708+0000 7f3d98d43640 1 -- 172.21.15.67:0/3586757610 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d90031c40 con 0x7f3d9415e3a0 2024-03-24T00:44:40.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.708+0000 7f3d9bfa9640 1 -- 172.21.15.67:0/3586757610 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d9415e3a0 msgr2=0x7f3d940b2890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:40.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.708+0000 7f3d9bfa9640 1 --2- 172.21.15.67:0/3586757610 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d9415e3a0 0x7f3d940b2890 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f3d940d0220 tx=0x7f3d9002b910 comp rx=0 tx=0).stop 2024-03-24T00:44:40.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.708+0000 7f3d9bfa9640 1 -- 172.21.15.67:0/3586757610 shutdown_connections 2024-03-24T00:44:40.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.708+0000 7f3d9bfa9640 1 --2- 172.21.15.67:0/3586757610 >> v2:172.21.15.72:3300/0 conn(0x7f3d940cd030 0x7f3d940b4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:40.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.708+0000 7f3d9bfa9640 1 --2- 172.21.15.67:0/3586757610 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3d941626d0 0x7f3d940ccaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:40.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.708+0000 7f3d9bfa9640 1 --2- 172.21.15.67:0/3586757610 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d9415e3a0 0x7f3d940b2890 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:40.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.708+0000 7f3d9bfa9640 1 -- 172.21.15.67:0/3586757610 >> 172.21.15.67:0/3586757610 conn(0x7f3d940b3c40 msgr2=0x7f3d940b1dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:40.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.709+0000 7f3d9bfa9640 1 -- 172.21.15.67:0/3586757610 shutdown_connections 2024-03-24T00:44:40.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.709+0000 7f3d9bfa9640 1 -- 172.21.15.67:0/3586757610 wait complete. 2024-03-24T00:44:40.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.710+0000 7f3d9bfa9640 1 Processor -- start 2024-03-24T00:44:40.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.710+0000 7f3d9bfa9640 1 -- start start 2024-03-24T00:44:40.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.710+0000 7f3d9bfa9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d940cd030 0x7f3d9414b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:40.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.710+0000 7f3d9bfa9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d9415e3a0 0x7f3d940c5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:40.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.710+0000 7f3d9bfa9640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3d941626d0 0x7f3d9415c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:40.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.710+0000 7f3d9bfa9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d94162b10 con 0x7f3d940cd030 2024-03-24T00:44:40.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.710+0000 7f3d9bfa9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9415f100 con 0x7f3d9415e3a0 2024-03-24T00:44:40.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.710+0000 7f3d9bfa9640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9406b040 con 0x7f3d941626d0 2024-03-24T00:44:40.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.710+0000 7f3d99d45640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d940cd030 0x7f3d9414b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:40.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.711+0000 7f3d99d45640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d940cd030 0x7f3d9414b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55440/0 (socket says 172.21.15.67:55440) 2024-03-24T00:44:40.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.711+0000 7f3d99d45640 1 -- 172.21.15.67:0/2636849632 learned_addr learned my addr 172.21.15.67:0/2636849632 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:40.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.711+0000 7f3d99544640 1 --2- 172.21.15.67:0/2636849632 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d9415e3a0 0x7f3d940c5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:40.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.711+0000 7f3d9a546640 1 --2- 172.21.15.67:0/2636849632 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3d941626d0 0x7f3d9415c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:40.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.711+0000 7f3d99d45640 1 -- 172.21.15.67:0/2636849632 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3d941626d0 msgr2=0x7f3d9415c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:40.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.711+0000 7f3d99d45640 1 --2- 172.21.15.67:0/2636849632 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3d941626d0 0x7f3d9415c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:40.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.711+0000 7f3d99d45640 1 -- 172.21.15.67:0/2636849632 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d9415e3a0 msgr2=0x7f3d940c5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:40.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.711+0000 7f3d99d45640 1 --2- 172.21.15.67:0/2636849632 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d9415e3a0 0x7f3d940c5bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:40.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.711+0000 7f3d99d45640 1 -- 172.21.15.67:0/2636849632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d940c3e50 con 0x7f3d940cd030 2024-03-24T00:44:40.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.711+0000 7f3d99d45640 1 --2- 172.21.15.67:0/2636849632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d940cd030 0x7f3d9414b730 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f3d90000c00 tx=0x7f3d900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:40.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.711+0000 7f3d8a7fc640 1 -- 172.21.15.67:0/2636849632 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d900443f0 con 0x7f3d940cd030 2024-03-24T00:44:40.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.711+0000 7f3d8a7fc640 1 -- 172.21.15.67:0/2636849632 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d90044590 con 0x7f3d940cd030 2024-03-24T00:44:40.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.711+0000 7f3d9bfa9640 1 -- 172.21.15.67:0/2636849632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d940c4130 con 0x7f3d940cd030 2024-03-24T00:44:40.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.711+0000 7f3d8a7fc640 1 -- 172.21.15.67:0/2636849632 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d90044980 con 0x7f3d940cd030 2024-03-24T00:44:40.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.711+0000 7f3d9bfa9640 1 -- 172.21.15.67:0/2636849632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d940c46f0 con 0x7f3d940cd030 2024-03-24T00:44:40.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.712+0000 7f3d9bfa9640 1 -- 172.21.15.67:0/2636849632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d9404f400 con 0x7f3d940cd030 2024-03-24T00:44:40.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.713+0000 7f3d8a7fc640 1 -- 172.21.15.67:0/2636849632 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3d9004b060 con 0x7f3d940cd030 2024-03-24T00:44:40.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.713+0000 7f3d8a7fc640 1 --2- 172.21.15.67:0/2636849632 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3d7806bbd0 0x7f3d7806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:40.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.713+0000 7f3d8a7fc640 1 -- 172.21.15.67:0/2636849632 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f3d900acef0 con 0x7f3d940cd030 2024-03-24T00:44:40.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.713+0000 7f3d99544640 1 --2- 172.21.15.67:0/2636849632 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3d7806bbd0 0x7f3d7806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:40.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.714+0000 7f3d99544640 1 --2- 172.21.15.67:0/2636849632 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3d7806bbd0 0x7f3d7806e090 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f3d84006fd0 tx=0x7f3d84008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:40.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.717+0000 7f3d8a7fc640 1 -- 172.21.15.67:0/2636849632 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3d9003f070 con 0x7f3d940cd030 2024-03-24T00:44:40.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.902+0000 7f3d9bfa9640 1 -- 172.21.15.67:0/2636849632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3d940c6e90 con 0x7f3d940cd030 2024-03-24T00:44:40.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.903+0000 7f3d8a7fc640 1 -- 172.21.15.67:0/2636849632 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v77) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3d9007b890 con 0x7f3d940cd030 2024-03-24T00:44:40.904 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-03-24T00:44:40.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.905+0000 7f3d9bfa9640 1 -- 172.21.15.67:0/2636849632 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3d7806bbd0 msgr2=0x7f3d7806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:40.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.905+0000 7f3d9bfa9640 1 --2- 172.21.15.67:0/2636849632 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3d7806bbd0 0x7f3d7806e090 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f3d84006fd0 tx=0x7f3d84008040 comp rx=0 tx=0).stop 2024-03-24T00:44:40.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.905+0000 7f3d9bfa9640 1 -- 172.21.15.67:0/2636849632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d940cd030 msgr2=0x7f3d9414b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:40.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.905+0000 7f3d9bfa9640 1 --2- 172.21.15.67:0/2636849632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d940cd030 0x7f3d9414b730 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f3d90000c00 tx=0x7f3d900046c0 comp rx=0 tx=0).stop 2024-03-24T00:44:40.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.905+0000 7f3d9bfa9640 1 -- 172.21.15.67:0/2636849632 shutdown_connections 2024-03-24T00:44:40.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.905+0000 7f3d9bfa9640 1 --2- 172.21.15.67:0/2636849632 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3d7806bbd0 0x7f3d7806e090 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:40.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.905+0000 7f3d9bfa9640 1 --2- 172.21.15.67:0/2636849632 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3d941626d0 0x7f3d9415c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:40.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.905+0000 7f3d9bfa9640 1 --2- 172.21.15.67:0/2636849632 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d9415e3a0 0x7f3d940c5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:40.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.905+0000 7f3d9bfa9640 1 --2- 172.21.15.67:0/2636849632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d940cd030 0x7f3d9414b730 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:40.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.905+0000 7f3d9bfa9640 1 -- 172.21.15.67:0/2636849632 >> 172.21.15.67:0/2636849632 conn(0x7f3d940b3c40 msgr2=0x7f3d9406b980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:40.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.905+0000 7f3d9bfa9640 1 -- 172.21.15.67:0/2636849632 shutdown_connections 2024-03-24T00:44:40.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:40.905+0000 7f3d9bfa9640 1 -- 172.21.15.67:0/2636849632 wait complete. 2024-03-24T00:44:40.916 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:44:41.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.016+0000 7f8af44dc640 1 Processor -- start 2024-03-24T00:44:41.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.016+0000 7f8af44dc640 1 -- start start 2024-03-24T00:44:41.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.017+0000 7f8af44dc640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8aec15e2a0 0x7f8aec0ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:41.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.017+0000 7f8af44dc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8aec0d0ef0 0x7f8aec162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:41.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.017+0000 7f8af44dc640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aec0cf640 con 0x7f8aec15dec0 2024-03-24T00:44:41.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.017+0000 7f8af44dc640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8aec0cee30 con 0x7f8aec0d0ef0 2024-03-24T00:44:41.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.017+0000 7f8af44dc640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aec0d2120 con 0x7f8aec15e2a0 2024-03-24T00:44:41.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.017+0000 7f8af2a79640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8aec0d0ef0 0x7f8aec162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:41.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.017+0000 7f8af2a79640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8aec0d0ef0 0x7f8aec162630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55452/0 (socket says 172.21.15.67:55452) 2024-03-24T00:44:41.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.017+0000 7f8af2a79640 1 -- 172.21.15.67:0/1954675851 learned_addr learned my addr 172.21.15.67:0/1954675851 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:41.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.017+0000 7f8af1a77640 1 --2- 172.21.15.67:0/1954675851 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8aec15e2a0 0x7f8aec0ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:41.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.017+0000 7f8af2a79640 1 -- 172.21.15.67:0/1954675851 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8aec15e2a0 msgr2=0x7f8aec0ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:41.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.017+0000 7f8af2a79640 1 --2- 172.21.15.67:0/1954675851 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8aec15e2a0 0x7f8aec0ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:41.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.017+0000 7f8af2a79640 1 -- 172.21.15.67:0/1954675851 >> v1:172.21.15.72:6789/0 conn(0x7f8aec15dec0 legacy=0x7f8aec0d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:41.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.017+0000 7f8af2a79640 1 -- 172.21.15.67:0/1954675851 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aec0cf080 con 0x7f8aec0d0ef0 2024-03-24T00:44:41.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.017+0000 7f8af1a77640 1 --2- 172.21.15.67:0/1954675851 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8aec15e2a0 0x7f8aec0ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:44:41.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.018+0000 7f8af2a79640 1 --2- 172.21.15.67:0/1954675851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aec0d0ef0 0x7f8aec162630 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f8ae8009e50 tx=0x7f8ae802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b497bbef590aa82a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:41.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.018+0000 7f8af1276640 1 -- 172.21.15.67:0/1954675851 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ae802ca70 con 0x7f8aec0d0ef0 2024-03-24T00:44:41.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.018+0000 7f8af1276640 1 -- 172.21.15.67:0/1954675851 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ae802cc10 con 0x7f8aec0d0ef0 2024-03-24T00:44:41.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.018+0000 7f8af1276640 1 -- 172.21.15.67:0/1954675851 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ae80343f0 con 0x7f8aec0d0ef0 2024-03-24T00:44:41.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.019+0000 7f8af44dc640 1 -- 172.21.15.67:0/1954675851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aec0d0ef0 msgr2=0x7f8aec162630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:41.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.019+0000 7f8af44dc640 1 --2- 172.21.15.67:0/1954675851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aec0d0ef0 0x7f8aec162630 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f8ae8009e50 tx=0x7f8ae802bed0 comp rx=0 tx=0).stop 2024-03-24T00:44:41.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.019+0000 7f8af44dc640 1 -- 172.21.15.67:0/1954675851 shutdown_connections 2024-03-24T00:44:41.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.019+0000 7f8af44dc640 1 --2- 172.21.15.67:0/1954675851 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8aec15e2a0 0x7f8aec0ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:41.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.019+0000 7f8af44dc640 1 --2- 172.21.15.67:0/1954675851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aec0d0ef0 0x7f8aec162630 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:41.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.019+0000 7f8af44dc640 1 -- 172.21.15.67:0/1954675851 >> 172.21.15.67:0/1954675851 conn(0x7f8aec06c810 msgr2=0x7f8aec06cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:41.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.019+0000 7f8af44dc640 1 -- 172.21.15.67:0/1954675851 shutdown_connections 2024-03-24T00:44:41.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.019+0000 7f8af44dc640 1 -- 172.21.15.67:0/1954675851 wait complete. 2024-03-24T00:44:41.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.021+0000 7f8af44dc640 1 Processor -- start 2024-03-24T00:44:41.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.021+0000 7f8af44dc640 1 -- start start 2024-03-24T00:44:41.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.022+0000 7f8af44dc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aec0d0ef0 0x7f8aec1f6a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:41.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.022+0000 7f8af44dc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8aec15dec0 0x7f8aec160800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:41.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.022+0000 7f8af44dc640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8aec15e2a0 0x7f8aec160d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:41.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.022+0000 7f8af44dc640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aec0d2730 con 0x7f8aec0d0ef0 2024-03-24T00:44:41.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.022+0000 7f8af44dc640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aec0d25b0 con 0x7f8aec15dec0 2024-03-24T00:44:41.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.022+0000 7f8af44dc640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aec0d28b0 con 0x7f8aec15e2a0 2024-03-24T00:44:41.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.022+0000 7f8af2278640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aec0d0ef0 0x7f8aec1f6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:41.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.022+0000 7f8af2278640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aec0d0ef0 0x7f8aec1f6a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55458/0 (socket says 172.21.15.67:55458) 2024-03-24T00:44:41.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.022+0000 7f8af2278640 1 -- 172.21.15.67:0/4015895822 learned_addr learned my addr 172.21.15.67:0/4015895822 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:41.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.022+0000 7f8af1a77640 1 --2- 172.21.15.67:0/4015895822 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8aec15dec0 0x7f8aec160800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:41.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.022+0000 7f8af2a79640 1 --2- 172.21.15.67:0/4015895822 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8aec15e2a0 0x7f8aec160d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:41.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.023+0000 7f8af2278640 1 -- 172.21.15.67:0/4015895822 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8aec15e2a0 msgr2=0x7f8aec160d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:41.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.023+0000 7f8af2278640 1 --2- 172.21.15.67:0/4015895822 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8aec15e2a0 0x7f8aec160d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:41.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.023+0000 7f8af2278640 1 -- 172.21.15.67:0/4015895822 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8aec15dec0 msgr2=0x7f8aec160800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:41.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.023+0000 7f8af2278640 1 --2- 172.21.15.67:0/4015895822 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8aec15dec0 0x7f8aec160800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:41.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.023+0000 7f8af2278640 1 -- 172.21.15.67:0/4015895822 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aec1f6f50 con 0x7f8aec0d0ef0 2024-03-24T00:44:41.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.023+0000 7f8af1a77640 1 --2- 172.21.15.67:0/4015895822 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8aec15dec0 0x7f8aec160800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:44:41.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.023+0000 7f8af2278640 1 --2- 172.21.15.67:0/4015895822 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aec0d0ef0 0x7f8aec1f6a10 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f8ae400c400 tx=0x7f8ae400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:41.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.023+0000 7f8ae2ffd640 1 -- 172.21.15.67:0/4015895822 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ae40052d0 con 0x7f8aec0d0ef0 2024-03-24T00:44:41.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.023+0000 7f8ae2ffd640 1 -- 172.21.15.67:0/4015895822 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ae4005490 con 0x7f8aec0d0ef0 2024-03-24T00:44:41.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.023+0000 7f8af44dc640 1 -- 172.21.15.67:0/4015895822 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8aec1f3d90 con 0x7f8aec0d0ef0 2024-03-24T00:44:41.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.023+0000 7f8ae2ffd640 1 -- 172.21.15.67:0/4015895822 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ae4007260 con 0x7f8aec0d0ef0 2024-03-24T00:44:41.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.023+0000 7f8af44dc640 1 -- 172.21.15.67:0/4015895822 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aec1f4330 con 0x7f8aec0d0ef0 2024-03-24T00:44:41.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.024+0000 7f8af44dc640 1 -- 172.21.15.67:0/4015895822 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8aec04f400 con 0x7f8aec0d0ef0 2024-03-24T00:44:41.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.024+0000 7f8ae2ffd640 1 -- 172.21.15.67:0/4015895822 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8ae4007400 con 0x7f8aec0d0ef0 2024-03-24T00:44:41.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.025+0000 7f8ae2ffd640 1 --2- 172.21.15.67:0/4015895822 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8ad006bbd0 0x7f8ad006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:41.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.025+0000 7f8ae2ffd640 1 -- 172.21.15.67:0/4015895822 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f8ae408a130 con 0x7f8aec0d0ef0 2024-03-24T00:44:41.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.025+0000 7f8af1a77640 1 --2- 172.21.15.67:0/4015895822 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8ad006bbd0 0x7f8ad006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:41.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.026+0000 7f8af1a77640 1 --2- 172.21.15.67:0/4015895822 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8ad006bbd0 0x7f8ad006e090 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f8adc006fd0 tx=0x7f8adc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:41.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.029+0000 7f8ae2ffd640 1 -- 172.21.15.67:0/4015895822 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8aec0cf640 con 0x7f8aec0d0ef0 2024-03-24T00:44:41.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.250+0000 7f8af44dc640 1 -- 172.21.15.67:0/4015895822 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8aec0d1aa0 con 0x7f8aec0d0ef0 2024-03-24T00:44:41.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.251+0000 7f8ae2ffd640 1 -- 172.21.15.67:0/4015895822 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+4978 (secure 0 0 0) 0x7f8ae4058ad0 con 0x7f8aec0d0ef0 2024-03-24T00:44:41.252 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:44:41.252 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":48,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5960,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1888378520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1888378520},{"type":"v1","addr":"172.21.15.67:6835","nonce":1888378520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6610,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/2539718386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2539718386},{"type":"v1","addr":"172.21.15.72:6836","nonce":2539718386}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6613,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2376370770","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2376370770},{"type":"v1","addr":"172.21.15.72:6837","nonce":2376370770}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:43:10.177088+0000","modified":"2024-03-24T00:44:11.389921+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7149},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7149":{"gid":7149,"name":"c","rank":0,"incarnation":43,"state":"up:active","state_seq":15,"addr":"172.21.15.67:6833/720256752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":720256752},{"type":"v1","addr":"172.21.15.67:6833","nonce":720256752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7149,"qdb_cluster":[7149]},"id":5}]} 2024-03-24T00:44:41.252 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 48 2024-03-24T00:44:41.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.253+0000 7f8af44dc640 1 -- 172.21.15.67:0/4015895822 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8ad006bbd0 msgr2=0x7f8ad006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:41.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.253+0000 7f8af44dc640 1 --2- 172.21.15.67:0/4015895822 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8ad006bbd0 0x7f8ad006e090 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f8adc006fd0 tx=0x7f8adc008040 comp rx=0 tx=0).stop 2024-03-24T00:44:41.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.253+0000 7f8af44dc640 1 -- 172.21.15.67:0/4015895822 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aec0d0ef0 msgr2=0x7f8aec1f6a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:41.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.253+0000 7f8af44dc640 1 --2- 172.21.15.67:0/4015895822 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aec0d0ef0 0x7f8aec1f6a10 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f8ae400c400 tx=0x7f8ae400c8c0 comp rx=0 tx=0).stop 2024-03-24T00:44:41.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.253+0000 7f8af44dc640 1 -- 172.21.15.67:0/4015895822 shutdown_connections 2024-03-24T00:44:41.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.253+0000 7f8af44dc640 1 --2- 172.21.15.67:0/4015895822 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8ad006bbd0 0x7f8ad006e090 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:41.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.253+0000 7f8af44dc640 1 --2- 172.21.15.67:0/4015895822 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8aec15e2a0 0x7f8aec160d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:41.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.253+0000 7f8af44dc640 1 --2- 172.21.15.67:0/4015895822 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8aec15dec0 0x7f8aec160800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:41.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.253+0000 7f8af44dc640 1 --2- 172.21.15.67:0/4015895822 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aec0d0ef0 0x7f8aec1f6a10 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:41.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.253+0000 7f8af44dc640 1 -- 172.21.15.67:0/4015895822 >> 172.21.15.67:0/4015895822 conn(0x7f8aec06c810 msgr2=0x7f8aec15e950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:41.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.253+0000 7f8af44dc640 1 -- 172.21.15.67:0/4015895822 shutdown_connections 2024-03-24T00:44:41.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.253+0000 7f8af44dc640 1 -- 172.21.15.67:0/4015895822 wait complete. 2024-03-24T00:44:41.264 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:44:41.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.365+0000 7f761e4dc640 1 Processor -- start 2024-03-24T00:44:41.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.365+0000 7f761e4dc640 1 -- start start 2024-03-24T00:44:41.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.366+0000 7f761e4dc640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f761815dec0 0x7f76180ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:41.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.366+0000 7f761e4dc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f761815e2a0 0x7f7618162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:41.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.366+0000 7f761e4dc640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76180cf640 con 0x7f76180d07e0 2024-03-24T00:44:41.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.366+0000 7f761e4dc640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76180cee30 con 0x7f761815e2a0 2024-03-24T00:44:41.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.366+0000 7f761e4dc640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76180d2120 con 0x7f761815dec0 2024-03-24T00:44:41.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.366+0000 7f761ca79640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f76180d07e0 0x7f76180b5850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47872/0 (socket says 172.21.15.67:47872) 2024-03-24T00:44:41.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.366+0000 7f761ca79640 1 -- 172.21.15.67:0/2624692976 learned_addr learned my addr 172.21.15.67:0/2624692976 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:44:41.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.366+0000 7f7617fff640 1 --2- 172.21.15.67:0/2624692976 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f761815dec0 0x7f76180ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:41.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.366+0000 7f76177fe640 1 --2- 172.21.15.67:0/2624692976 >> v2:172.21.15.72:3300/0 conn(0x7f761815e2a0 0x7f7618162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:41.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.366+0000 7f7616ffd640 1 -- 172.21.15.67:0/2624692976 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1751672045 0 0) 0x7f7610001490 con 0x7f76180d07e0 2024-03-24T00:44:41.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.366+0000 7f7616ffd640 1 -- 172.21.15.67:0/2624692976 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7600003520 con 0x7f76180d07e0 2024-03-24T00:44:41.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.366+0000 7f7616ffd640 1 -- 172.21.15.67:0/2624692976 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2764790918 0 0) 0x7f7600003520 con 0x7f76180d07e0 2024-03-24T00:44:41.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.367+0000 7f7616ffd640 1 -- 172.21.15.67:0/2624692976 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f761815dec0 msgr2=0x7f76180ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:41.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.367+0000 7f7616ffd640 1 --2- 172.21.15.67:0/2624692976 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f761815dec0 0x7f76180ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:41.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.367+0000 7f7616ffd640 1 -- 172.21.15.67:0/2624692976 >> v2:172.21.15.72:3300/0 conn(0x7f761815e2a0 msgr2=0x7f7618162630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:41.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.367+0000 7f7616ffd640 1 --2- 172.21.15.67:0/2624692976 >> v2:172.21.15.72:3300/0 conn(0x7f761815e2a0 0x7f7618162630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:41.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.367+0000 7f7616ffd640 1 -- 172.21.15.67:0/2624692976 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76180cf080 con 0x7f76180d07e0 2024-03-24T00:44:41.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.367+0000 7f7616ffd640 1 -- 172.21.15.67:0/2624692976 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f7610001bc0 con 0x7f76180d07e0 2024-03-24T00:44:41.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.367+0000 7f7616ffd640 1 -- 172.21.15.67:0/2624692976 >> v1:172.21.15.67:6789/0 conn(0x7f76180d07e0 legacy=0x7f76180b5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:41.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.367+0000 7f7616ffd640 1 --2- 172.21.15.67:0/2624692976 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76000039b0 0x7f7600003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:41.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.367+0000 7f7616ffd640 1 --2- 172.21.15.67:0/2624692976 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f761815dec0 0x7f7600004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:41.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.367+0000 7f7616ffd640 1 --2- 172.21.15.67:0/2624692976 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7600004d40 0x7f76000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:41.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.367+0000 7f7616ffd640 1 -- 172.21.15.67:0/2624692976 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76180cee30 con 0x7f76000039b0 2024-03-24T00:44:41.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.367+0000 7f7616ffd640 1 -- 172.21.15.67:0/2624692976 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76180d2120 con 0x7f7600004d40 2024-03-24T00:44:41.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.367+0000 7f7616ffd640 1 -- 172.21.15.67:0/2624692976 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f760000b9c0 con 0x7f761815dec0 2024-03-24T00:44:41.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.367+0000 7f7617fff640 1 --2- 172.21.15.67:0/2624692976 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76000039b0 0x7f7600003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:41.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.367+0000 7f761ca79640 1 --2- 172.21.15.67:0/2624692976 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7600004d40 0x7f76000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:41.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.367+0000 7f76177fe640 1 --2- 172.21.15.67:0/2624692976 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f761815dec0 0x7f7600004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:41.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.367+0000 7f7617fff640 1 -- 172.21.15.67:0/2624692976 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f761815dec0 msgr2=0x7f7600004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:41.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.367+0000 7f7617fff640 1 --2- 172.21.15.67:0/2624692976 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f761815dec0 0x7f7600004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:41.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.367+0000 7f7617fff640 1 -- 172.21.15.67:0/2624692976 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7600004d40 msgr2=0x7f76000051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:41.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.367+0000 7f7617fff640 1 --2- 172.21.15.67:0/2624692976 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7600004d40 0x7f76000051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:41.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.367+0000 7f7617fff640 1 -- 172.21.15.67:0/2624692976 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7600003520 con 0x7f76000039b0 2024-03-24T00:44:41.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.367+0000 7f761ca79640 1 --2- 172.21.15.67:0/2624692976 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7600004d40 0x7f76000051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:44:41.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.368+0000 7f7617fff640 1 --2- 172.21.15.67:0/2624692976 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76000039b0 0x7f7600003df0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f7608009ac0 tx=0x7f7600005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89dde0b5e4226f92 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:41.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.368+0000 7f7616ffd640 1 -- 172.21.15.67:0/2624692976 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7608039070 con 0x7f76000039b0 2024-03-24T00:44:41.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.368+0000 7f7616ffd640 1 -- 172.21.15.67:0/2624692976 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f760802c6b0 con 0x7f76000039b0 2024-03-24T00:44:41.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.368+0000 7f7616ffd640 1 -- 172.21.15.67:0/2624692976 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7608009690 con 0x7f76000039b0 2024-03-24T00:44:41.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.368+0000 7f761e4dc640 1 -- 172.21.15.67:0/2624692976 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76000039b0 msgr2=0x7f7600003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:41.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.368+0000 7f761e4dc640 1 --2- 172.21.15.67:0/2624692976 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76000039b0 0x7f7600003df0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f7608009ac0 tx=0x7f7600005830 comp rx=0 tx=0).stop 2024-03-24T00:44:41.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.368+0000 7f761ca79640 1 -- 172.21.15.67:0/2624692976 reap_dead start 2024-03-24T00:44:41.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.368+0000 7f761e4dc640 1 -- 172.21.15.67:0/2624692976 shutdown_connections 2024-03-24T00:44:41.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.368+0000 7f761e4dc640 1 -- 172.21.15.67:0/2624692976 >> 172.21.15.67:0/2624692976 conn(0x7f761806c810 msgr2=0x7f761806cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:41.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.369+0000 7f761e4dc640 1 -- 172.21.15.67:0/2624692976 shutdown_connections 2024-03-24T00:44:41.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.369+0000 7f761e4dc640 1 -- 172.21.15.67:0/2624692976 wait complete. 2024-03-24T00:44:41.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.370+0000 7f761e4dc640 1 Processor -- start 2024-03-24T00:44:41.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.370+0000 7f761e4dc640 1 -- start start 2024-03-24T00:44:41.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.371+0000 7f761e4dc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76000039b0 0x7f76180b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:41.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.371+0000 7f761e4dc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f761814cd60 0x7f76180b55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:41.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.371+0000 7f761e4dc640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f761814d1e0 0x7f76180b5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:41.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.371+0000 7f761e4dc640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76180d3740 con 0x7f76000039b0 2024-03-24T00:44:41.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.371+0000 7f761e4dc640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76180d35c0 con 0x7f761814cd60 2024-03-24T00:44:41.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.371+0000 7f761e4dc640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76180d38c0 con 0x7f761814d1e0 2024-03-24T00:44:41.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.371+0000 7f7617fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76000039b0 0x7f76180b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:41.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.371+0000 7f7617fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76000039b0 0x7f76180b21f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55484/0 (socket says 172.21.15.67:55484) 2024-03-24T00:44:41.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.371+0000 7f7617fff640 1 -- 172.21.15.67:0/1607773842 learned_addr learned my addr 172.21.15.67:0/1607773842 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:41.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.371+0000 7f76177fe640 1 --2- 172.21.15.67:0/1607773842 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f761814cd60 0x7f76180b55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:41.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.371+0000 7f7617fff640 1 -- 172.21.15.67:0/1607773842 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f761814d1e0 msgr2=0x7f76180b5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:41.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.371+0000 7f7617fff640 1 --2- 172.21.15.67:0/1607773842 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f761814d1e0 0x7f76180b5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:41.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.371+0000 7f7617fff640 1 -- 172.21.15.67:0/1607773842 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f761814cd60 msgr2=0x7f76180b55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:41.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.371+0000 7f7617fff640 1 --2- 172.21.15.67:0/1607773842 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f761814cd60 0x7f76180b55e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:41.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.371+0000 7f7617fff640 1 -- 172.21.15.67:0/1607773842 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76180b2730 con 0x7f76000039b0 2024-03-24T00:44:41.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.371+0000 7f7617fff640 1 --2- 172.21.15.67:0/1607773842 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76000039b0 0x7f76180b21f0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f7608009be0 tx=0x7f7608004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:41.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.371+0000 7f7614ff9640 1 -- 172.21.15.67:0/1607773842 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7608039050 con 0x7f76000039b0 2024-03-24T00:44:41.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.371+0000 7f7614ff9640 1 -- 172.21.15.67:0/1607773842 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7608045070 con 0x7f76000039b0 2024-03-24T00:44:41.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.371+0000 7f761e4dc640 1 -- 172.21.15.67:0/1607773842 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76180b4540 con 0x7f76000039b0 2024-03-24T00:44:41.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.371+0000 7f7614ff9640 1 -- 172.21.15.67:0/1607773842 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7608003850 con 0x7f76000039b0 2024-03-24T00:44:41.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.371+0000 7f761e4dc640 1 -- 172.21.15.67:0/1607773842 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76180b4b00 con 0x7f76000039b0 2024-03-24T00:44:41.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.373+0000 7f761e4dc640 1 -- 172.21.15.67:0/1607773842 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f761804f400 con 0x7f76000039b0 2024-03-24T00:44:41.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.373+0000 7f7614ff9640 1 -- 172.21.15.67:0/1607773842 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f76080039f0 con 0x7f76000039b0 2024-03-24T00:44:41.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.373+0000 7f7614ff9640 1 --2- 172.21.15.67:0/1607773842 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f760406bbd0 0x7f760406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:41.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.373+0000 7f7614ff9640 1 -- 172.21.15.67:0/1607773842 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f76080ac7b0 con 0x7f76000039b0 2024-03-24T00:44:41.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.374+0000 7f76177fe640 1 --2- 172.21.15.67:0/1607773842 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f760406bbd0 0x7f760406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:41.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.374+0000 7f76177fe640 1 --2- 172.21.15.67:0/1607773842 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f760406bbd0 0x7f760406e090 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f7618159be0 tx=0x7f760c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:41.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.378+0000 7f7614ff9640 1 -- 172.21.15.67:0/1607773842 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f760804e020 con 0x7f76000039b0 2024-03-24T00:44:41.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.564+0000 7f761e4dc640 1 -- 172.21.15.67:0/1607773842 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f76180d1aa0 con 0x7f76000039b0 2024-03-24T00:44:41.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.565+0000 7f7614ff9640 1 -- 172.21.15.67:0/1607773842 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+17014 (secure 0 0 0) 0x7f760807b180 con 0x7f76000039b0 2024-03-24T00:44:41.566 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:44:41.566 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":77,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:43:24.373819+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:43:08.452726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-24T00:43:09.146431+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6833/2392802741":"2024-03-25T00:43:24.373788+0000","172.21.15.67:6832/2392802741":"2024-03-25T00:43:24.373788+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:44:41.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.567+0000 7f761e4dc640 1 -- 172.21.15.67:0/1607773842 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f760406bbd0 msgr2=0x7f760406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:41.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.567+0000 7f761e4dc640 1 --2- 172.21.15.67:0/1607773842 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f760406bbd0 0x7f760406e090 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f7618159be0 tx=0x7f760c008040 comp rx=0 tx=0).stop 2024-03-24T00:44:41.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.567+0000 7f761e4dc640 1 -- 172.21.15.67:0/1607773842 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76000039b0 msgr2=0x7f76180b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:41.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.567+0000 7f761e4dc640 1 --2- 172.21.15.67:0/1607773842 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76000039b0 0x7f76180b21f0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f7608009be0 tx=0x7f7608004730 comp rx=0 tx=0).stop 2024-03-24T00:44:41.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.567+0000 7f761e4dc640 1 -- 172.21.15.67:0/1607773842 shutdown_connections 2024-03-24T00:44:41.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.567+0000 7f761e4dc640 1 --2- 172.21.15.67:0/1607773842 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f760406bbd0 0x7f760406e090 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:41.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.567+0000 7f761e4dc640 1 --2- 172.21.15.67:0/1607773842 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f761814d1e0 0x7f76180b5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:41.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.567+0000 7f761e4dc640 1 --2- 172.21.15.67:0/1607773842 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f761814cd60 0x7f76180b55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:41.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.567+0000 7f761e4dc640 1 --2- 172.21.15.67:0/1607773842 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76000039b0 0x7f76180b21f0 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:41.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.567+0000 7f761e4dc640 1 -- 172.21.15.67:0/1607773842 >> 172.21.15.67:0/1607773842 conn(0x7f761806c810 msgr2=0x7f761815e6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:41.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.567+0000 7f761e4dc640 1 -- 172.21.15.67:0/1607773842 shutdown_connections 2024-03-24T00:44:41.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.567+0000 7f761e4dc640 1 -- 172.21.15.67:0/1607773842 wait complete. 2024-03-24T00:44:41.579 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-24T00:44:41.579 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:44:41.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.679+0000 7ffbdb93b640 1 Processor -- start 2024-03-24T00:44:41.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.679+0000 7ffbdb93b640 1 -- start start 2024-03-24T00:44:41.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.680+0000 7ffbdb93b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ffbd40b49c0 0x7ffbd415b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:41.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.680+0000 7ffbdb93b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ffbd4072a70 0x7ffbd40b2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:41.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.680+0000 7ffbdb93b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbd40b4da0 con 0x7ffbd406f100 2024-03-24T00:44:41.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.681+0000 7ffbdb93b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbd406db30 con 0x7ffbd4072a70 2024-03-24T00:44:41.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.681+0000 7ffbdb93b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbd406de10 con 0x7ffbd40b49c0 2024-03-24T00:44:41.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.681+0000 7ffbd9ed8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ffbd4072a70 0x7ffbd40b2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:41.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.681+0000 7ffbd8ed6640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ffbd406f100 0x7ffbd4072360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47874/0 (socket says 172.21.15.67:47874) 2024-03-24T00:44:41.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.681+0000 7ffbd8ed6640 1 -- 172.21.15.67:0/1187846659 learned_addr learned my addr 172.21.15.67:0/1187846659 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:44:41.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.681+0000 7ffbd96d7640 1 --2- 172.21.15.67:0/1187846659 >> v2:172.21.15.72:3300/0 conn(0x7ffbd40b49c0 0x7ffbd415b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:41.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.681+0000 7ffbd9ed8640 1 -- 172.21.15.67:0/1187846659 >> v2:172.21.15.72:3300/0 conn(0x7ffbd40b49c0 msgr2=0x7ffbd415b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:41.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.681+0000 7ffbd9ed8640 1 --2- 172.21.15.67:0/1187846659 >> v2:172.21.15.72:3300/0 conn(0x7ffbd40b49c0 0x7ffbd415b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:41.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.681+0000 7ffbd9ed8640 1 -- 172.21.15.67:0/1187846659 >> v1:172.21.15.67:6789/0 conn(0x7ffbd406f100 legacy=0x7ffbd4072360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:41.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.681+0000 7ffbd9ed8640 1 -- 172.21.15.67:0/1187846659 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbd406e0f0 con 0x7ffbd4072a70 2024-03-24T00:44:41.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.681+0000 7ffbd96d7640 1 --2- 172.21.15.67:0/1187846659 >> v2:172.21.15.72:3300/0 conn(0x7ffbd40b49c0 0x7ffbd415b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:44:41.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.682+0000 7ffbd9ed8640 1 --2- 172.21.15.67:0/1187846659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbd4072a70 0x7ffbd40b2a10 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7ffbd4072f50 tx=0x7ffbcc02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20280170c44da04d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:41.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.682+0000 7ffbcbfff640 1 -- 172.21.15.67:0/1187846659 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbcc039070 con 0x7ffbd4072a70 2024-03-24T00:44:41.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.682+0000 7ffbcbfff640 1 -- 172.21.15.67:0/1187846659 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbcc02cbf0 con 0x7ffbd4072a70 2024-03-24T00:44:41.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.682+0000 7ffbcbfff640 1 -- 172.21.15.67:0/1187846659 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbcc034440 con 0x7ffbd4072a70 2024-03-24T00:44:41.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.682+0000 7ffbdb93b640 1 -- 172.21.15.67:0/1187846659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbd4072a70 msgr2=0x7ffbd40b2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:41.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.682+0000 7ffbdb93b640 1 --2- 172.21.15.67:0/1187846659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbd4072a70 0x7ffbd40b2a10 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7ffbd4072f50 tx=0x7ffbcc02be40 comp rx=0 tx=0).stop 2024-03-24T00:44:41.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.682+0000 7ffbdb93b640 1 -- 172.21.15.67:0/1187846659 shutdown_connections 2024-03-24T00:44:41.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.682+0000 7ffbdb93b640 1 --2- 172.21.15.67:0/1187846659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbd4072a70 0x7ffbd40b2a10 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:41.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.682+0000 7ffbdb93b640 1 --2- 172.21.15.67:0/1187846659 >> v2:172.21.15.72:3300/0 conn(0x7ffbd40b49c0 0x7ffbd415b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:41.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.682+0000 7ffbdb93b640 1 -- 172.21.15.67:0/1187846659 >> 172.21.15.67:0/1187846659 conn(0x7ffbd40cf9e0 msgr2=0x7ffbd40cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:41.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.683+0000 7ffbdb93b640 1 -- 172.21.15.67:0/1187846659 shutdown_connections 2024-03-24T00:44:41.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.683+0000 7ffbdb93b640 1 -- 172.21.15.67:0/1187846659 wait complete. 2024-03-24T00:44:41.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.684+0000 7ffbdb93b640 1 Processor -- start 2024-03-24T00:44:41.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.684+0000 7ffbdb93b640 1 -- start start 2024-03-24T00:44:41.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.684+0000 7ffbdb93b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ffbd406f100 0x7ffbd4159560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:41.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.684+0000 7ffbdb93b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffbd4072a70 0x7ffbd4159aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:41.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.684+0000 7ffbdb93b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbd40b49c0 0x7ffbd40cd560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:41.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.684+0000 7ffbdb93b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd406f590 con 0x7ffbd40b49c0 2024-03-24T00:44:41.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.684+0000 7ffbdb93b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd406e6b0 con 0x7ffbd4072a70 2024-03-24T00:44:41.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.684+0000 7ffbdb93b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd40b3d30 con 0x7ffbd406f100 2024-03-24T00:44:41.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.685+0000 7ffbd9ed8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbd40b49c0 0x7ffbd40cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:41.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.685+0000 7ffbd9ed8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbd40b49c0 0x7ffbd40cd560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55508/0 (socket says 172.21.15.67:55508) 2024-03-24T00:44:41.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.685+0000 7ffbd9ed8640 1 -- 172.21.15.67:0/2582889202 learned_addr learned my addr 172.21.15.67:0/2582889202 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:41.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.685+0000 7ffbd8ed6640 1 --2- 172.21.15.67:0/2582889202 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffbd4072a70 0x7ffbd4159aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:41.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.685+0000 7ffbd96d7640 1 --2- 172.21.15.67:0/2582889202 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ffbd406f100 0x7ffbd4159560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:41.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.685+0000 7ffbd9ed8640 1 -- 172.21.15.67:0/2582889202 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ffbd406f100 msgr2=0x7ffbd4159560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:41.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.685+0000 7ffbd9ed8640 1 --2- 172.21.15.67:0/2582889202 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ffbd406f100 0x7ffbd4159560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:41.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.685+0000 7ffbd9ed8640 1 -- 172.21.15.67:0/2582889202 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffbd4072a70 msgr2=0x7ffbd4159aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:41.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.685+0000 7ffbd9ed8640 1 --2- 172.21.15.67:0/2582889202 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffbd4072a70 0x7ffbd4159aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:41.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.685+0000 7ffbd9ed8640 1 -- 172.21.15.67:0/2582889202 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbd41f2af0 con 0x7ffbd40b49c0 2024-03-24T00:44:41.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.685+0000 7ffbd96d7640 1 --2- 172.21.15.67:0/2582889202 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ffbd406f100 0x7ffbd4159560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:44:41.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.685+0000 7ffbd9ed8640 1 --2- 172.21.15.67:0/2582889202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbd40b49c0 0x7ffbd40cd560 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc009e80 tx=0x7ffbcc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:41.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.685+0000 7ffbc9ffb640 1 -- 172.21.15.67:0/2582889202 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbcc039050 con 0x7ffbd40b49c0 2024-03-24T00:44:41.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.685+0000 7ffbc9ffb640 1 -- 172.21.15.67:0/2582889202 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbcc045070 con 0x7ffbd40b49c0 2024-03-24T00:44:41.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.685+0000 7ffbdb93b640 1 -- 172.21.15.67:0/2582889202 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbd4152f10 con 0x7ffbd40b49c0 2024-03-24T00:44:41.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.685+0000 7ffbc9ffb640 1 -- 172.21.15.67:0/2582889202 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbcc03e830 con 0x7ffbd40b49c0 2024-03-24T00:44:41.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.685+0000 7ffbdb93b640 1 -- 172.21.15.67:0/2582889202 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbd41534d0 con 0x7ffbd40b49c0 2024-03-24T00:44:41.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.687+0000 7ffbdb93b640 1 -- 172.21.15.67:0/2582889202 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbd404f400 con 0x7ffbd40b49c0 2024-03-24T00:44:41.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.687+0000 7ffbc9ffb640 1 -- 172.21.15.67:0/2582889202 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ffbcc03e9d0 con 0x7ffbd40b49c0 2024-03-24T00:44:41.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.687+0000 7ffbc9ffb640 1 --2- 172.21.15.67:0/2582889202 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ffbb806bbd0 0x7ffbb806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:41.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.688+0000 7ffbc9ffb640 1 -- 172.21.15.67:0/2582889202 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7ffbcc0ad950 con 0x7ffbd40b49c0 2024-03-24T00:44:41.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.688+0000 7ffbd96d7640 1 --2- 172.21.15.67:0/2582889202 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ffbb806bbd0 0x7ffbb806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:41.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.688+0000 7ffbd96d7640 1 --2- 172.21.15.67:0/2582889202 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ffbb806bbd0 0x7ffbb806e090 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7ffbd0006fd0 tx=0x7ffbd0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:41.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.692+0000 7ffbc9ffb640 1 -- 172.21.15.67:0/2582889202 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ffbcc07c410 con 0x7ffbd40b49c0 2024-03-24T00:44:41.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.910+0000 7ffbdb93b640 1 -- 172.21.15.67:0/2582889202 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffbd40cf560 con 0x7ffbd40b49c0 2024-03-24T00:44:41.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.911+0000 7ffbc9ffb640 1 -- 172.21.15.67:0/2582889202 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+4978 (secure 0 0 0) 0x7ffbcc07bd10 con 0x7ffbd40b49c0 2024-03-24T00:44:41.912 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:44:41.912 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":48,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5960,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1888378520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1888378520},{"type":"v1","addr":"172.21.15.67:6835","nonce":1888378520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6610,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/2539718386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2539718386},{"type":"v1","addr":"172.21.15.72:6836","nonce":2539718386}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6613,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2376370770","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2376370770},{"type":"v1","addr":"172.21.15.72:6837","nonce":2376370770}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:43:10.177088+0000","modified":"2024-03-24T00:44:11.389921+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7149},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7149":{"gid":7149,"name":"c","rank":0,"incarnation":43,"state":"up:active","state_seq":15,"addr":"172.21.15.67:6833/720256752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":720256752},{"type":"v1","addr":"172.21.15.67:6833","nonce":720256752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7149,"qdb_cluster":[7149]},"id":5}]} 2024-03-24T00:44:41.912 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 48 2024-03-24T00:44:41.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.913+0000 7ffbdb93b640 1 -- 172.21.15.67:0/2582889202 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ffbb806bbd0 msgr2=0x7ffbb806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:41.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.913+0000 7ffbdb93b640 1 --2- 172.21.15.67:0/2582889202 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ffbb806bbd0 0x7ffbb806e090 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7ffbd0006fd0 tx=0x7ffbd0008040 comp rx=0 tx=0).stop 2024-03-24T00:44:41.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.913+0000 7ffbdb93b640 1 -- 172.21.15.67:0/2582889202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbd40b49c0 msgr2=0x7ffbd40cd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:41.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.913+0000 7ffbdb93b640 1 --2- 172.21.15.67:0/2582889202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbd40b49c0 0x7ffbd40cd560 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc009e80 tx=0x7ffbcc004a40 comp rx=0 tx=0).stop 2024-03-24T00:44:41.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.913+0000 7ffbdb93b640 1 -- 172.21.15.67:0/2582889202 shutdown_connections 2024-03-24T00:44:41.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.913+0000 7ffbdb93b640 1 --2- 172.21.15.67:0/2582889202 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ffbb806bbd0 0x7ffbb806e090 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:41.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.913+0000 7ffbdb93b640 1 --2- 172.21.15.67:0/2582889202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbd40b49c0 0x7ffbd40cd560 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:41.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.913+0000 7ffbdb93b640 1 --2- 172.21.15.67:0/2582889202 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffbd4072a70 0x7ffbd4159aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:41.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.913+0000 7ffbdb93b640 1 --2- 172.21.15.67:0/2582889202 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ffbd406f100 0x7ffbd4159560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:41.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.913+0000 7ffbdb93b640 1 -- 172.21.15.67:0/2582889202 >> 172.21.15.67:0/2582889202 conn(0x7ffbd40cf9e0 msgr2=0x7ffbd415bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:41.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.913+0000 7ffbdb93b640 1 -- 172.21.15.67:0/2582889202 shutdown_connections 2024-03-24T00:44:41.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:41.913+0000 7ffbdb93b640 1 -- 172.21.15.67:0/2582889202 wait complete. 2024-03-24T00:44:41.924 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:44:42.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.025+0000 7f3bf2685640 1 Processor -- start 2024-03-24T00:44:42.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.025+0000 7f3bf2685640 1 -- start start 2024-03-24T00:44:42.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.026+0000 7f3bf2685640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3bec15bf70 0x7f3bec0cc690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:42.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.026+0000 7f3bf2685640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bec0cfc70 0x7f3bec0ce220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:42.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.026+0000 7f3bf2685640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bec0ccfe0 con 0x7f3bec0ce760 2024-03-24T00:44:42.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.026+0000 7f3bf2685640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bec0cd2c0 con 0x7f3bec15bf70 2024-03-24T00:44:42.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.026+0000 7f3bf2685640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bec0cd5a0 con 0x7f3bec0cfc70 2024-03-24T00:44:42.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.026+0000 7f3beb7fe640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bec0cfc70 0x7f3bec0ce220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:42.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.026+0000 7f3bebfff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3bec15bf70 0x7f3bec0cc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:42.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.026+0000 7f3bf0c22640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f3bec0ce760 0x7f3bec08b060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:34110/0 (socket says 172.21.15.67:34110) 2024-03-24T00:44:42.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.026+0000 7f3bf0c22640 1 -- 172.21.15.67:0/2939765983 learned_addr learned my addr 172.21.15.67:0/2939765983 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:44:42.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.027+0000 7f3bebfff640 1 -- 172.21.15.67:0/2939765983 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bec0cfc70 msgr2=0x7f3bec0ce220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:42.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.027+0000 7f3bebfff640 1 --2- 172.21.15.67:0/2939765983 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bec0cfc70 0x7f3bec0ce220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:42.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.027+0000 7f3bebfff640 1 -- 172.21.15.67:0/2939765983 >> v1:172.21.15.72:6789/0 conn(0x7f3bec0ce760 legacy=0x7f3bec08b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:42.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.027+0000 7f3bebfff640 1 -- 172.21.15.67:0/2939765983 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bec0cd880 con 0x7f3bec15bf70 2024-03-24T00:44:42.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.027+0000 7f3beaffd640 1 -- 172.21.15.67:0/2939765983 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3316412665 0 0) 0x7f3bec0ccfe0 con 0x7f3bec0ce760 2024-03-24T00:44:42.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.028+0000 7f3bebfff640 1 --2- 172.21.15.67:0/2939765983 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bec15bf70 0x7f3bec0cc690 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc0097a0 tx=0x7f3bdc02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ea10ae6e3220ee7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:42.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.028+0000 7f3beaffd640 1 -- 172.21.15.67:0/2939765983 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bdc035020 con 0x7f3bec15bf70 2024-03-24T00:44:42.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.028+0000 7f3beaffd640 1 -- 172.21.15.67:0/2939765983 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bdc031830 con 0x7f3bec15bf70 2024-03-24T00:44:42.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.028+0000 7f3beaffd640 1 -- 172.21.15.67:0/2939765983 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bdc031c00 con 0x7f3bec15bf70 2024-03-24T00:44:42.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.029+0000 7f3bf2685640 1 -- 172.21.15.67:0/2939765983 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bec15bf70 msgr2=0x7f3bec0cc690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:42.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.029+0000 7f3bf2685640 1 --2- 172.21.15.67:0/2939765983 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bec15bf70 0x7f3bec0cc690 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc0097a0 tx=0x7f3bdc02b8d0 comp rx=0 tx=0).stop 2024-03-24T00:44:42.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.029+0000 7f3bf2685640 1 -- 172.21.15.67:0/2939765983 shutdown_connections 2024-03-24T00:44:42.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.029+0000 7f3bf2685640 1 --2- 172.21.15.67:0/2939765983 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bec0cfc70 0x7f3bec0ce220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:42.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.029+0000 7f3bf2685640 1 --2- 172.21.15.67:0/2939765983 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bec15bf70 0x7f3bec0cc690 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:42.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.029+0000 7f3bf2685640 1 -- 172.21.15.67:0/2939765983 >> 172.21.15.67:0/2939765983 conn(0x7f3bec0d1bf0 msgr2=0x7f3bec0d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:42.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.029+0000 7f3bf2685640 1 -- 172.21.15.67:0/2939765983 shutdown_connections 2024-03-24T00:44:42.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.029+0000 7f3bf2685640 1 -- 172.21.15.67:0/2939765983 wait complete. 2024-03-24T00:44:42.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.030+0000 7f3bf2685640 1 Processor -- start 2024-03-24T00:44:42.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.031+0000 7f3bf2685640 1 -- start start 2024-03-24T00:44:42.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.032+0000 7f3bf2685640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bec0ce760 0x7f3bec15c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:42.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.032+0000 7f3bf2685640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bec0cfc70 0x7f3bec06c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:42.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.032+0000 7f3bf2685640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bec15bf70 0x7f3bec0b67f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:42.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.032+0000 7f3bf2685640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bec0ccd50 con 0x7f3bec15bf70 2024-03-24T00:44:42.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.032+0000 7f3bf2685640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bec0ccbd0 con 0x7f3bec0cfc70 2024-03-24T00:44:42.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.032+0000 7f3bf2685640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bec0b2ce0 con 0x7f3bec0ce760 2024-03-24T00:44:42.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.032+0000 7f3bf0c22640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bec15bf70 0x7f3bec0b67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:42.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.032+0000 7f3bf0c22640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bec15bf70 0x7f3bec0b67f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55522/0 (socket says 172.21.15.67:55522) 2024-03-24T00:44:42.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.032+0000 7f3bf0c22640 1 -- 172.21.15.67:0/2639432732 learned_addr learned my addr 172.21.15.67:0/2639432732 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:42.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.032+0000 7f3beb7fe640 1 --2- 172.21.15.67:0/2639432732 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bec0cfc70 0x7f3bec06c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:42.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.032+0000 7f3bebfff640 1 --2- 172.21.15.67:0/2639432732 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bec0ce760 0x7f3bec15c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:42.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.032+0000 7f3bf0c22640 1 -- 172.21.15.67:0/2639432732 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bec0ce760 msgr2=0x7f3bec15c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:42.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.032+0000 7f3bf0c22640 1 --2- 172.21.15.67:0/2639432732 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bec0ce760 0x7f3bec15c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:42.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.032+0000 7f3bf0c22640 1 -- 172.21.15.67:0/2639432732 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bec0cfc70 msgr2=0x7f3bec06c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:42.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.032+0000 7f3bf0c22640 1 --2- 172.21.15.67:0/2639432732 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bec0cfc70 0x7f3bec06c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:42.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.032+0000 7f3bf0c22640 1 -- 172.21.15.67:0/2639432732 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bec1f4000 con 0x7f3bec15bf70 2024-03-24T00:44:42.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.032+0000 7f3bf0c22640 1 --2- 172.21.15.67:0/2639432732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bec15bf70 0x7f3bec0b67f0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f3be400c700 tx=0x7f3be400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:42.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.033+0000 7f3be8ff9640 1 -- 172.21.15.67:0/2639432732 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be4018070 con 0x7f3bec15bf70 2024-03-24T00:44:42.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.033+0000 7f3be8ff9640 1 -- 172.21.15.67:0/2639432732 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3be400ce10 con 0x7f3bec15bf70 2024-03-24T00:44:42.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.033+0000 7f3bf2685640 1 -- 172.21.15.67:0/2639432732 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bec1f42e0 con 0x7f3bec15bf70 2024-03-24T00:44:42.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.033+0000 7f3be8ff9640 1 -- 172.21.15.67:0/2639432732 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be400a860 con 0x7f3bec15bf70 2024-03-24T00:44:42.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.033+0000 7f3bf2685640 1 -- 172.21.15.67:0/2639432732 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bec0c9f20 con 0x7f3bec15bf70 2024-03-24T00:44:42.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.034+0000 7f3bf2685640 1 -- 172.21.15.67:0/2639432732 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bec0c9d10 con 0x7f3bec15bf70 2024-03-24T00:44:42.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.035+0000 7f3be8ff9640 1 -- 172.21.15.67:0/2639432732 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3be401b070 con 0x7f3bec15bf70 2024-03-24T00:44:42.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.035+0000 7f3be8ff9640 1 --2- 172.21.15.67:0/2639432732 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3bd806bbd0 0x7f3bd806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:42.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.035+0000 7f3be8ff9640 1 -- 172.21.15.67:0/2639432732 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f3be408a180 con 0x7f3bec15bf70 2024-03-24T00:44:42.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.035+0000 7f3bebfff640 1 --2- 172.21.15.67:0/2639432732 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3bd806bbd0 0x7f3bd806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:42.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.036+0000 7f3bebfff640 1 --2- 172.21.15.67:0/2639432732 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3bd806bbd0 0x7f3bd806e090 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc036000 tx=0x7f3bdc008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:42.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.040+0000 7f3be8ff9640 1 -- 172.21.15.67:0/2639432732 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3be4058b20 con 0x7f3bec15bf70 2024-03-24T00:44:42.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.259+0000 7f3bf2685640 1 -- 172.21.15.67:0/2639432732 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3bec04f400 con 0x7f3bec15bf70 2024-03-24T00:44:42.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.259+0000 7f3be8ff9640 1 -- 172.21.15.67:0/2639432732 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+4978 (secure 0 0 0) 0x7f3be4058420 con 0x7f3bec15bf70 2024-03-24T00:44:42.260 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:44:42.260 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":48,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5960,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1888378520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1888378520},{"type":"v1","addr":"172.21.15.67:6835","nonce":1888378520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6610,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/2539718386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2539718386},{"type":"v1","addr":"172.21.15.72:6836","nonce":2539718386}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6613,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2376370770","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2376370770},{"type":"v1","addr":"172.21.15.72:6837","nonce":2376370770}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:43:10.177088+0000","modified":"2024-03-24T00:44:11.389921+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7149},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7149":{"gid":7149,"name":"c","rank":0,"incarnation":43,"state":"up:active","state_seq":15,"addr":"172.21.15.67:6833/720256752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":720256752},{"type":"v1","addr":"172.21.15.67:6833","nonce":720256752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7149,"qdb_cluster":[7149]},"id":5}]} 2024-03-24T00:44:42.261 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 48 2024-03-24T00:44:42.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.261+0000 7f3bf2685640 1 -- 172.21.15.67:0/2639432732 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3bd806bbd0 msgr2=0x7f3bd806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:42.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.261+0000 7f3bf2685640 1 --2- 172.21.15.67:0/2639432732 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3bd806bbd0 0x7f3bd806e090 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc036000 tx=0x7f3bdc008030 comp rx=0 tx=0).stop 2024-03-24T00:44:42.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.261+0000 7f3bf2685640 1 -- 172.21.15.67:0/2639432732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bec15bf70 msgr2=0x7f3bec0b67f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:42.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.261+0000 7f3bf2685640 1 --2- 172.21.15.67:0/2639432732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bec15bf70 0x7f3bec0b67f0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f3be400c700 tx=0x7f3be400cbc0 comp rx=0 tx=0).stop 2024-03-24T00:44:42.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.261+0000 7f3bf2685640 1 -- 172.21.15.67:0/2639432732 shutdown_connections 2024-03-24T00:44:42.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.261+0000 7f3bf2685640 1 --2- 172.21.15.67:0/2639432732 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3bd806bbd0 0x7f3bd806e090 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:42.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.261+0000 7f3bf2685640 1 --2- 172.21.15.67:0/2639432732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bec15bf70 0x7f3bec0b67f0 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:42.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.261+0000 7f3bf2685640 1 --2- 172.21.15.67:0/2639432732 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bec0cfc70 0x7f3bec06c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:42.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.261+0000 7f3bf2685640 1 --2- 172.21.15.67:0/2639432732 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bec0ce760 0x7f3bec15c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:42.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.261+0000 7f3bf2685640 1 -- 172.21.15.67:0/2639432732 >> 172.21.15.67:0/2639432732 conn(0x7f3bec0d1bf0 msgr2=0x7f3bec08bb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:42.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.261+0000 7f3bf2685640 1 -- 172.21.15.67:0/2639432732 shutdown_connections 2024-03-24T00:44:42.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.262+0000 7f3bf2685640 1 -- 172.21.15.67:0/2639432732 wait complete. 2024-03-24T00:44:42.273 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:44:42.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.374+0000 7f240ad77640 1 Processor -- start 2024-03-24T00:44:42.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.374+0000 7f240ad77640 1 -- start start 2024-03-24T00:44:42.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.374+0000 7f240ad77640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f240415dec0 0x7f24040ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:42.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.374+0000 7f240ad77640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f240415e2a0 0x7f2404162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:42.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.374+0000 7f240ad77640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24040cf640 con 0x7f24040d07e0 2024-03-24T00:44:42.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.374+0000 7f240ad77640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24040cee30 con 0x7f240415e2a0 2024-03-24T00:44:42.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.374+0000 7f240ad77640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24040d2120 con 0x7f240415dec0 2024-03-24T00:44:42.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.375+0000 7f2403fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f240415e2a0 0x7f2404162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:42.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.375+0000 7f2409314640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f24040d07e0 0x7f24040b5850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47882/0 (socket says 172.21.15.67:47882) 2024-03-24T00:44:42.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.375+0000 7f2409314640 1 -- 172.21.15.67:0/319310538 learned_addr learned my addr 172.21.15.67:0/319310538 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:44:42.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.375+0000 7f2408b13640 1 --2- 172.21.15.67:0/319310538 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f240415dec0 0x7f24040ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:42.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.375+0000 7f2403fff640 1 -- 172.21.15.67:0/319310538 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f240415dec0 msgr2=0x7f24040ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:42.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.375+0000 7f2403fff640 1 --2- 172.21.15.67:0/319310538 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f240415dec0 0x7f24040ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:42.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.375+0000 7f2403fff640 1 -- 172.21.15.67:0/319310538 >> v1:172.21.15.67:6789/0 conn(0x7f24040d07e0 legacy=0x7f24040b5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:42.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.375+0000 7f24037fe640 1 -- 172.21.15.67:0/319310538 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3627518128 0 0) 0x7f23fc001490 con 0x7f24040d07e0 2024-03-24T00:44:42.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.375+0000 7f2403fff640 1 -- 172.21.15.67:0/319310538 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24040cf080 con 0x7f240415e2a0 2024-03-24T00:44:42.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.375+0000 7f2408b13640 1 --2- 172.21.15.67:0/319310538 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f240415dec0 0x7f24040ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:44:42.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.376+0000 7f2403fff640 1 --2- 172.21.15.67:0/319310538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f240415e2a0 0x7f2404162630 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f24040cefb0 tx=0x7f23f002b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a57281d56ba665a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:42.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.376+0000 7f24037fe640 1 -- 172.21.15.67:0/319310538 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f0031770 con 0x7f240415e2a0 2024-03-24T00:44:42.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.376+0000 7f24037fe640 1 -- 172.21.15.67:0/319310538 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23f0037070 con 0x7f240415e2a0 2024-03-24T00:44:42.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.376+0000 7f24037fe640 1 -- 172.21.15.67:0/319310538 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f0031b30 con 0x7f240415e2a0 2024-03-24T00:44:42.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.376+0000 7f240ad77640 1 -- 172.21.15.67:0/319310538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f240415e2a0 msgr2=0x7f2404162630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:42.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.376+0000 7f240ad77640 1 --2- 172.21.15.67:0/319310538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f240415e2a0 0x7f2404162630 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f24040cefb0 tx=0x7f23f002b810 comp rx=0 tx=0).stop 2024-03-24T00:44:42.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.376+0000 7f240ad77640 1 -- 172.21.15.67:0/319310538 shutdown_connections 2024-03-24T00:44:42.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.376+0000 7f240ad77640 1 --2- 172.21.15.67:0/319310538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f240415e2a0 0x7f2404162630 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:42.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.376+0000 7f240ad77640 1 --2- 172.21.15.67:0/319310538 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f240415dec0 0x7f24040ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:42.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.376+0000 7f240ad77640 1 -- 172.21.15.67:0/319310538 >> 172.21.15.67:0/319310538 conn(0x7f240406c810 msgr2=0x7f240406cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:42.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.376+0000 7f240ad77640 1 -- 172.21.15.67:0/319310538 shutdown_connections 2024-03-24T00:44:42.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.376+0000 7f240ad77640 1 -- 172.21.15.67:0/319310538 wait complete. 2024-03-24T00:44:42.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.377+0000 7f240ad77640 1 Processor -- start 2024-03-24T00:44:42.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.378+0000 7f240ad77640 1 -- start start 2024-03-24T00:44:42.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.378+0000 7f240ad77640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24040d07e0 0x7f24041f4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:42.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.378+0000 7f240ad77640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f240415dec0 0x7f24041f6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:42.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.378+0000 7f240ad77640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f240415e2a0 0x7f24040b7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:42.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.378+0000 7f240ad77640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24040d3740 con 0x7f24040d07e0 2024-03-24T00:44:42.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.378+0000 7f240ad77640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24040d35c0 con 0x7f240415dec0 2024-03-24T00:44:42.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.378+0000 7f240ad77640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24040d38c0 con 0x7f240415e2a0 2024-03-24T00:44:42.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.378+0000 7f2408b13640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24040d07e0 0x7f24041f4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:42.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.378+0000 7f2408b13640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24040d07e0 0x7f24041f4170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55546/0 (socket says 172.21.15.67:55546) 2024-03-24T00:44:42.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.379+0000 7f2408b13640 1 -- 172.21.15.67:0/2459537721 learned_addr learned my addr 172.21.15.67:0/2459537721 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:42.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.379+0000 7f2403fff640 1 --2- 172.21.15.67:0/2459537721 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f240415dec0 0x7f24041f6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:42.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.379+0000 7f2409314640 1 --2- 172.21.15.67:0/2459537721 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f240415e2a0 0x7f24040b7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:42.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.379+0000 7f2408b13640 1 -- 172.21.15.67:0/2459537721 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f240415e2a0 msgr2=0x7f24040b7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:42.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.379+0000 7f2408b13640 1 --2- 172.21.15.67:0/2459537721 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f240415e2a0 0x7f24040b7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:42.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.379+0000 7f2408b13640 1 -- 172.21.15.67:0/2459537721 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f240415dec0 msgr2=0x7f24041f6fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:42.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.379+0000 7f2408b13640 1 --2- 172.21.15.67:0/2459537721 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f240415dec0 0x7f24041f6fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:42.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.379+0000 7f2408b13640 1 -- 172.21.15.67:0/2459537721 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2404158980 con 0x7f24040d07e0 2024-03-24T00:44:42.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.379+0000 7f2408b13640 1 --2- 172.21.15.67:0/2459537721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24040d07e0 0x7f24041f4170 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f23f800c7f0 tx=0x7f23f800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:42.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.379+0000 7f24017fa640 1 -- 172.21.15.67:0/2459537721 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f8015070 con 0x7f24040d07e0 2024-03-24T00:44:42.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.379+0000 7f24017fa640 1 -- 172.21.15.67:0/2459537721 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23f8007a30 con 0x7f24040d07e0 2024-03-24T00:44:42.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.379+0000 7f240ad77640 1 -- 172.21.15.67:0/2459537721 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2404158c60 con 0x7f24040d07e0 2024-03-24T00:44:42.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.379+0000 7f24017fa640 1 -- 172.21.15.67:0/2459537721 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f8007e00 con 0x7f24040d07e0 2024-03-24T00:44:42.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.379+0000 7f240ad77640 1 -- 172.21.15.67:0/2459537721 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2404159220 con 0x7f24040d07e0 2024-03-24T00:44:42.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.380+0000 7f240ad77640 1 -- 172.21.15.67:0/2459537721 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24040cae80 con 0x7f24040d07e0 2024-03-24T00:44:42.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.381+0000 7f24017fa640 1 -- 172.21.15.67:0/2459537721 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f23f8018070 con 0x7f24040d07e0 2024-03-24T00:44:42.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.381+0000 7f24017fa640 1 --2- 172.21.15.67:0/2459537721 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f23e806bbd0 0x7f23e806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:42.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.382+0000 7f24017fa640 1 -- 172.21.15.67:0/2459537721 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f23f808ab70 con 0x7f24040d07e0 2024-03-24T00:44:42.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.382+0000 7f2403fff640 1 --2- 172.21.15.67:0/2459537721 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f23e806bbd0 0x7f23e806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:42.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.382+0000 7f2403fff640 1 --2- 172.21.15.67:0/2459537721 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f23e806bbd0 0x7f23e806e090 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f23f0036000 tx=0x7f23f003d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:42.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.390+0000 7f24017fa640 1 -- 172.21.15.67:0/2459537721 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f23f8058e30 con 0x7f24040d07e0 2024-03-24T00:44:42.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.647+0000 7f240ad77640 1 -- 172.21.15.67:0/2459537721 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2404161530 con 0x7f24040d07e0 2024-03-24T00:44:42.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.648+0000 7f24017fa640 1 -- 172.21.15.67:0/2459537721 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+17014 (secure 0 0 0) 0x7f23f805ced0 con 0x7f24040d07e0 2024-03-24T00:44:42.649 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:44:42.649 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":77,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:43:24.373819+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:43:08.452726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-24T00:43:09.146431+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6833/2392802741":"2024-03-25T00:43:24.373788+0000","172.21.15.67:6832/2392802741":"2024-03-25T00:43:24.373788+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:44:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.650+0000 7f240ad77640 1 -- 172.21.15.67:0/2459537721 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f23e806bbd0 msgr2=0x7f23e806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.650+0000 7f240ad77640 1 --2- 172.21.15.67:0/2459537721 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f23e806bbd0 0x7f23e806e090 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f23f0036000 tx=0x7f23f003d040 comp rx=0 tx=0).stop 2024-03-24T00:44:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.650+0000 7f240ad77640 1 -- 172.21.15.67:0/2459537721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24040d07e0 msgr2=0x7f24041f4170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.650+0000 7f240ad77640 1 --2- 172.21.15.67:0/2459537721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24040d07e0 0x7f24041f4170 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f23f800c7f0 tx=0x7f23f800cbf0 comp rx=0 tx=0).stop 2024-03-24T00:44:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.650+0000 7f240ad77640 1 -- 172.21.15.67:0/2459537721 shutdown_connections 2024-03-24T00:44:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.650+0000 7f240ad77640 1 --2- 172.21.15.67:0/2459537721 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f23e806bbd0 0x7f23e806e090 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.650+0000 7f240ad77640 1 --2- 172.21.15.67:0/2459537721 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f240415e2a0 0x7f24040b7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.650+0000 7f240ad77640 1 --2- 172.21.15.67:0/2459537721 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f240415dec0 0x7f24041f6fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.650+0000 7f240ad77640 1 --2- 172.21.15.67:0/2459537721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24040d07e0 0x7f24041f4170 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.650+0000 7f240ad77640 1 -- 172.21.15.67:0/2459537721 >> 172.21.15.67:0/2459537721 conn(0x7f240406c810 msgr2=0x7f24040b8760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.650+0000 7f240ad77640 1 -- 172.21.15.67:0/2459537721 shutdown_connections 2024-03-24T00:44:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.650+0000 7f240ad77640 1 -- 172.21.15.67:0/2459537721 wait complete. 2024-03-24T00:44:42.662 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-24T00:44:42.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.763+0000 7f8111987640 1 Processor -- start 2024-03-24T00:44:42.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.763+0000 7f8111987640 1 -- start start 2024-03-24T00:44:42.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.764+0000 7f8111987640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f810c0d26d0 0x7f810c06c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:42.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.764+0000 7f8111987640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f810c06d230 0x7f810c15d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:42.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.764+0000 7f8111987640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f810c0d0270 con 0x7f810c0ceb10 2024-03-24T00:44:42.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.764+0000 7f8111987640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f810c0d0e80 con 0x7f810c06d230 2024-03-24T00:44:42.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.764+0000 7f8111987640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f810c0d0550 con 0x7f810c0d26d0 2024-03-24T00:44:42.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.764+0000 7f810b7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f810c06d230 0x7f810c15d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:42.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.764+0000 7f810b7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f810c06d230 0x7f810c15d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55552/0 (socket says 172.21.15.67:55552) 2024-03-24T00:44:42.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.764+0000 7f810b7fe640 1 -- 172.21.15.67:0/626337125 learned_addr learned my addr 172.21.15.67:0/626337125 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:42.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.764+0000 7f81027fc640 1 --2- 172.21.15.67:0/626337125 >> v2:172.21.15.72:3300/0 conn(0x7f810c0d26d0 0x7f810c06c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:42.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.764+0000 7f810b7fe640 1 -- 172.21.15.67:0/626337125 >> v2:172.21.15.72:3300/0 conn(0x7f810c0d26d0 msgr2=0x7f810c06c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:42.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.764+0000 7f810b7fe640 1 --2- 172.21.15.67:0/626337125 >> v2:172.21.15.72:3300/0 conn(0x7f810c0d26d0 0x7f810c06c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:42.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.765+0000 7f810b7fe640 1 -- 172.21.15.67:0/626337125 >> v1:172.21.15.72:6789/0 conn(0x7f810c0ceb10 legacy=0x7f810c06cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:42.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.765+0000 7f810b7fe640 1 -- 172.21.15.67:0/626337125 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f810c0d0830 con 0x7f810c06d230 2024-03-24T00:44:42.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.765+0000 7f81027fc640 1 --2- 172.21.15.67:0/626337125 >> v2:172.21.15.72:3300/0 conn(0x7f810c0d26d0 0x7f810c06c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:44:42.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.766+0000 7f810b7fe640 1 --2- 172.21.15.67:0/626337125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f810c06d230 0x7f810c15d740 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f810c0b1500 tx=0x7f810402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b10987d50d4aa31 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:42.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.766+0000 7f810a7fc640 1 -- 172.21.15.67:0/626337125 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f810402cb20 con 0x7f810c06d230 2024-03-24T00:44:42.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.766+0000 7f810a7fc640 1 -- 172.21.15.67:0/626337125 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f810402ccc0 con 0x7f810c06d230 2024-03-24T00:44:42.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.766+0000 7f810a7fc640 1 -- 172.21.15.67:0/626337125 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8104034610 con 0x7f810c06d230 2024-03-24T00:44:42.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.766+0000 7f8111987640 1 -- 172.21.15.67:0/626337125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f810c06d230 msgr2=0x7f810c15d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:42.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.766+0000 7f8111987640 1 --2- 172.21.15.67:0/626337125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f810c06d230 0x7f810c15d740 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f810c0b1500 tx=0x7f810402bf80 comp rx=0 tx=0).stop 2024-03-24T00:44:42.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.766+0000 7f8111987640 1 -- 172.21.15.67:0/626337125 shutdown_connections 2024-03-24T00:44:42.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.766+0000 7f8111987640 1 --2- 172.21.15.67:0/626337125 >> v2:172.21.15.72:3300/0 conn(0x7f810c0d26d0 0x7f810c06c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:42.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.766+0000 7f8111987640 1 --2- 172.21.15.67:0/626337125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f810c06d230 0x7f810c15d740 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:42.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.766+0000 7f8111987640 1 -- 172.21.15.67:0/626337125 >> 172.21.15.67:0/626337125 conn(0x7f810c0d1ad0 msgr2=0x7f810c0d1ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:42.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.766+0000 7f8111987640 1 -- 172.21.15.67:0/626337125 shutdown_connections 2024-03-24T00:44:42.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.766+0000 7f8111987640 1 -- 172.21.15.67:0/626337125 wait complete. 2024-03-24T00:44:42.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.768+0000 7f8111987640 1 Processor -- start 2024-03-24T00:44:42.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.768+0000 7f8111987640 1 -- start start 2024-03-24T00:44:42.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.768+0000 7f8111987640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f810c06d230 0x7f810c156b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:42.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.768+0000 7f8111987640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f810c0ceb10 0x7f810c0b4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:42.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.768+0000 7f8111987640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f810c0d26d0 0x7f810c0b4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:42.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.768+0000 7f8111987640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f810c0cf070 con 0x7f810c06d230 2024-03-24T00:44:42.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.768+0000 7f8111987640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f810c0ceef0 con 0x7f810c0ceb10 2024-03-24T00:44:42.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.768+0000 7f8111987640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f810c0cfe00 con 0x7f810c0d26d0 2024-03-24T00:44:42.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.769+0000 7f810affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f810c06d230 0x7f810c156b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:42.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.769+0000 7f810affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f810c06d230 0x7f810c156b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55562/0 (socket says 172.21.15.67:55562) 2024-03-24T00:44:42.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.769+0000 7f810affd640 1 -- 172.21.15.67:0/3349776615 learned_addr learned my addr 172.21.15.67:0/3349776615 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:42.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.769+0000 7f81027fc640 1 --2- 172.21.15.67:0/3349776615 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f810c0ceb10 0x7f810c0b4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:42.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.769+0000 7f810b7fe640 1 --2- 172.21.15.67:0/3349776615 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f810c0d26d0 0x7f810c0b4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:42.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.769+0000 7f810affd640 1 -- 172.21.15.67:0/3349776615 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f810c0d26d0 msgr2=0x7f810c0b4f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:42.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.769+0000 7f810affd640 1 --2- 172.21.15.67:0/3349776615 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f810c0d26d0 0x7f810c0b4f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:42.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.769+0000 7f810affd640 1 -- 172.21.15.67:0/3349776615 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f810c0ceb10 msgr2=0x7f810c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:42.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.769+0000 7f810affd640 1 --2- 172.21.15.67:0/3349776615 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f810c0ceb10 0x7f810c0b4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:42.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.769+0000 7f810affd640 1 -- 172.21.15.67:0/3349776615 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f810c0b6a60 con 0x7f810c06d230 2024-03-24T00:44:42.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.769+0000 7f810b7fe640 1 --2- 172.21.15.67:0/3349776615 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f810c0d26d0 0x7f810c0b4f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:44:42.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.769+0000 7f810affd640 1 --2- 172.21.15.67:0/3349776615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f810c06d230 0x7f810c156b70 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f80fc00c3c0 tx=0x7f80fc00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:42.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.769+0000 7f8108ff9640 1 -- 172.21.15.67:0/3349776615 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80fc0052d0 con 0x7f810c06d230 2024-03-24T00:44:42.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.769+0000 7f8108ff9640 1 -- 172.21.15.67:0/3349776615 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80fc005490 con 0x7f810c06d230 2024-03-24T00:44:42.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.769+0000 7f8111987640 1 -- 172.21.15.67:0/3349776615 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f810c0b6d00 con 0x7f810c06d230 2024-03-24T00:44:42.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.769+0000 7f8108ff9640 1 -- 172.21.15.67:0/3349776615 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80fc007260 con 0x7f810c06d230 2024-03-24T00:44:42.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.769+0000 7f8111987640 1 -- 172.21.15.67:0/3349776615 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f810c0b6fa0 con 0x7f810c06d230 2024-03-24T00:44:42.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.770+0000 7f8111987640 1 -- 172.21.15.67:0/3349776615 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f810c04f400 con 0x7f810c06d230 2024-03-24T00:44:42.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.771+0000 7f8108ff9640 1 -- 172.21.15.67:0/3349776615 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f80fc007400 con 0x7f810c06d230 2024-03-24T00:44:42.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.771+0000 7f8108ff9640 1 --2- 172.21.15.67:0/3349776615 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f80ec06bbd0 0x7f80ec06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:42.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.771+0000 7f8108ff9640 1 -- 172.21.15.67:0/3349776615 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f80fc08a130 con 0x7f810c06d230 2024-03-24T00:44:42.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.772+0000 7f81027fc640 1 --2- 172.21.15.67:0/3349776615 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f80ec06bbd0 0x7f80ec06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:42.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.772+0000 7f81027fc640 1 --2- 172.21.15.67:0/3349776615 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f80ec06bbd0 0x7f80ec06e090 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f810c0c7f00 tx=0x7f80f4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:42.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.776+0000 7f8108ff9640 1 -- 172.21.15.67:0/3349776615 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f80fc058a40 con 0x7f810c06d230 2024-03-24T00:44:42.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.997+0000 7f8111987640 1 -- 172.21.15.67:0/3349776615 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f810c0cf460 con 0x7f810c06d230 2024-03-24T00:44:42.998 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-24T00:44:42.998+0000 7fbfcfa11640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T00:44:43.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:42.999+0000 7f8108ff9640 1 -- 172.21.15.67:0/3349776615 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v49) v1 ==== 161+0+0 (secure 0 0 0) 0x7f80fc058340 con 0x7f810c06d230 2024-03-24T00:44:43.000 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-24T00:44:43.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.003+0000 7f8111987640 1 -- 172.21.15.67:0/3349776615 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f80ec06bbd0 msgr2=0x7f80ec06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:43.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.003+0000 7f8111987640 1 --2- 172.21.15.67:0/3349776615 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f80ec06bbd0 0x7f80ec06e090 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f810c0c7f00 tx=0x7f80f4008040 comp rx=0 tx=0).stop 2024-03-24T00:44:43.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.003+0000 7f8111987640 1 -- 172.21.15.67:0/3349776615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f810c06d230 msgr2=0x7f810c156b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:43.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.003+0000 7f8111987640 1 --2- 172.21.15.67:0/3349776615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f810c06d230 0x7f810c156b70 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f80fc00c3c0 tx=0x7f80fc00c880 comp rx=0 tx=0).stop 2024-03-24T00:44:43.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.003+0000 7f8111987640 1 -- 172.21.15.67:0/3349776615 shutdown_connections 2024-03-24T00:44:43.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.003+0000 7f8111987640 1 --2- 172.21.15.67:0/3349776615 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f80ec06bbd0 0x7f80ec06e090 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:43.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.003+0000 7f8111987640 1 --2- 172.21.15.67:0/3349776615 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f810c0d26d0 0x7f810c0b4f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:43.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.003+0000 7f8111987640 1 --2- 172.21.15.67:0/3349776615 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f810c0ceb10 0x7f810c0b4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:43.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.003+0000 7f8111987640 1 --2- 172.21.15.67:0/3349776615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f810c06d230 0x7f810c156b70 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:43.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.003+0000 7f8111987640 1 -- 172.21.15.67:0/3349776615 >> 172.21.15.67:0/3349776615 conn(0x7f810c0d1ad0 msgr2=0x7f810c0cd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:43.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.003+0000 7f8111987640 1 -- 172.21.15.67:0/3349776615 shutdown_connections 2024-03-24T00:44:43.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.003+0000 7f8111987640 1 -- 172.21.15.67:0/3349776615 wait complete. 2024-03-24T00:44:43.006 INFO:tasks.ceph.mds.c.smithi067.stderr: -4024> 2024-03-24T00:43:30.383+0000 7f88e6548640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T00:44:43.016 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-24T00:44:43.034 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-03-24T00:44:43.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.119+0000 7f20157a9640 1 Processor -- start 2024-03-24T00:44:43.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.119+0000 7f20157a9640 1 -- start start 2024-03-24T00:44:43.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.120+0000 7f20157a9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f201006b750 0x7f20100b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:43.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.120+0000 7f20157a9640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f201015c220 con 0x7f20100b0c10 2024-03-24T00:44:43.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.120+0000 7f20157a9640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20100cc950 con 0x7f20100afae0 2024-03-24T00:44:43.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.120+0000 7f20157a9640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20100adfd0 con 0x7f201006b750 2024-03-24T00:44:43.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.120+0000 7f200f7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f201006b750 0x7f20100b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:43.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.120+0000 7f200f7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f201006b750 0x7f20100b1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55584/0 (socket says 172.21.15.67:55584) 2024-03-24T00:44:43.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.120+0000 7f200e7fc640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f20100b0c10 0x7f201006b040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47884/0 (socket says 172.21.15.67:47884) 2024-03-24T00:44:43.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.120+0000 7f200f7fe640 1 -- 172.21.15.67:0/3477605319 learned_addr learned my addr 172.21.15.67:0/3477605319 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:43.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.120+0000 7f200f7fe640 1 -- 172.21.15.67:0/3477605319 >> v1:172.21.15.72:6789/0 conn(0x7f20100afae0 legacy=0x7f20100ad8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:43.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.120+0000 7f200f7fe640 1 -- 172.21.15.67:0/3477605319 >> v1:172.21.15.67:6789/0 conn(0x7f20100b0c10 legacy=0x7f201006b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:43.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.120+0000 7f200f7fe640 1 -- 172.21.15.67:0/3477605319 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20100ccc30 con 0x7f201006b750 2024-03-24T00:44:43.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.120+0000 7f200dffb640 1 -- 172.21.15.67:0/3477605319 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 122973022 0 0) 0x7f201015c220 con 0x7f20100b0c10 2024-03-24T00:44:43.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.121+0000 7f200f7fe640 1 --2- 172.21.15.67:0/3477605319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f201006b750 0x7f20100b1070 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f2008009e50 tx=0x7f200802c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4be4b65f7970ea8d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:43.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.121+0000 7f200dffb640 1 -- 172.21.15.67:0/3477605319 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f200802cbb0 con 0x7f201006b750 2024-03-24T00:44:43.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.121+0000 7f200dffb640 1 -- 172.21.15.67:0/3477605319 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f200802cd50 con 0x7f201006b750 2024-03-24T00:44:43.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.121+0000 7f200dffb640 1 -- 172.21.15.67:0/3477605319 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2008034610 con 0x7f201006b750 2024-03-24T00:44:43.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.122+0000 7f20157a9640 1 -- 172.21.15.67:0/3477605319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f201006b750 msgr2=0x7f20100b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:43.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.122+0000 7f20157a9640 1 --2- 172.21.15.67:0/3477605319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f201006b750 0x7f20100b1070 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f2008009e50 tx=0x7f200802c010 comp rx=0 tx=0).stop 2024-03-24T00:44:43.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.122+0000 7f20157a9640 1 -- 172.21.15.67:0/3477605319 shutdown_connections 2024-03-24T00:44:43.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.122+0000 7f20157a9640 1 --2- 172.21.15.67:0/3477605319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f201006b750 0x7f20100b1070 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:43.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.122+0000 7f20157a9640 1 -- 172.21.15.67:0/3477605319 >> 172.21.15.67:0/3477605319 conn(0x7f2010099a30 msgr2=0x7f2010099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:43.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.122+0000 7f20157a9640 1 -- 172.21.15.67:0/3477605319 shutdown_connections 2024-03-24T00:44:43.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.122+0000 7f20157a9640 1 -- 172.21.15.67:0/3477605319 wait complete. 2024-03-24T00:44:43.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.123+0000 7f20157a9640 1 Processor -- start 2024-03-24T00:44:43.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.123+0000 7f20157a9640 1 -- start start 2024-03-24T00:44:43.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.124+0000 7f20157a9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f201006b750 0x7f2010144fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:43.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.124+0000 7f20157a9640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f20100afae0 0x7f20100c8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:43.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.124+0000 7f20157a9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20100b0c10 0x7f20100b6030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:43.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.124+0000 7f20157a9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20100cd4d0 con 0x7f201006b750 2024-03-24T00:44:43.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.124+0000 7f20157a9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20100cd1f0 con 0x7f20100b0c10 2024-03-24T00:44:43.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.124+0000 7f20157a9640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20100cc600 con 0x7f20100afae0 2024-03-24T00:44:43.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.124+0000 7f200effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f201006b750 0x7f2010144fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:43.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.124+0000 7f200effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f201006b750 0x7f2010144fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55600/0 (socket says 172.21.15.67:55600) 2024-03-24T00:44:43.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.124+0000 7f200effd640 1 -- 172.21.15.67:0/2479815672 learned_addr learned my addr 172.21.15.67:0/2479815672 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:43.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.124+0000 7f200f7fe640 1 --2- 172.21.15.67:0/2479815672 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20100b0c10 0x7f20100b6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:43.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.124+0000 7f200effd640 1 -- 172.21.15.67:0/2479815672 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f20100afae0 msgr2=0x7f20100c8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:43.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.124+0000 7f200e7fc640 1 --2- 172.21.15.67:0/2479815672 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f20100afae0 0x7f20100c8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:43.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.124+0000 7f200effd640 1 --2- 172.21.15.67:0/2479815672 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f20100afae0 0x7f20100c8c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:43.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.124+0000 7f200effd640 1 -- 172.21.15.67:0/2479815672 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20100b0c10 msgr2=0x7f20100b6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:43.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.124+0000 7f200effd640 1 --2- 172.21.15.67:0/2479815672 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20100b0c10 0x7f20100b6030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:43.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.124+0000 7f200effd640 1 -- 172.21.15.67:0/2479815672 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20100c9430 con 0x7f201006b750 2024-03-24T00:44:43.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.124+0000 7f200f7fe640 1 --2- 172.21.15.67:0/2479815672 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20100b0c10 0x7f20100b6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:44:43.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.125+0000 7f200effd640 1 --2- 172.21.15.67:0/2479815672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f201006b750 0x7f2010144fd0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f200000c450 tx=0x7f200000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:43.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.125+0000 7f20077fe640 1 -- 172.21.15.67:0/2479815672 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20000052d0 con 0x7f201006b750 2024-03-24T00:44:43.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.125+0000 7f20077fe640 1 -- 172.21.15.67:0/2479815672 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2000005470 con 0x7f201006b750 2024-03-24T00:44:43.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.125+0000 7f20157a9640 1 -- 172.21.15.67:0/2479815672 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20100c9710 con 0x7f201006b750 2024-03-24T00:44:43.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.125+0000 7f20077fe640 1 -- 172.21.15.67:0/2479815672 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2000012870 con 0x7f201006b750 2024-03-24T00:44:43.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.125+0000 7f20157a9640 1 -- 172.21.15.67:0/2479815672 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20100b5b10 con 0x7f201006b750 2024-03-24T00:44:43.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.126+0000 7f20157a9640 1 -- 172.21.15.67:0/2479815672 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f201004f400 con 0x7f201006b750 2024-03-24T00:44:43.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.126+0000 7f20077fe640 1 -- 172.21.15.67:0/2479815672 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2000012a10 con 0x7f201006b750 2024-03-24T00:44:43.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.126+0000 7f20077fe640 1 --2- 172.21.15.67:0/2479815672 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1ff006bbd0 0x7f1ff006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:43.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.127+0000 7f20077fe640 1 -- 172.21.15.67:0/2479815672 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f200008a850 con 0x7f201006b750 2024-03-24T00:44:43.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.127+0000 7f200e7fc640 1 --2- 172.21.15.67:0/2479815672 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1ff006bbd0 0x7f1ff006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:43.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.127+0000 7f200e7fc640 1 --2- 172.21.15.67:0/2479815672 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1ff006bbd0 0x7f1ff006e090 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8007fd0 tx=0x7f1ff8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:43.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.131+0000 7f20077fe640 1 -- 172.21.15.67:0/2479815672 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2000059190 con 0x7f201006b750 2024-03-24T00:44:43.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:43.350+0000 7f20157a9640 1 -- 172.21.15.67:0/2479815672 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f20100c9140 con 0x7f201006b750 2024-03-24T00:44:44.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.004+0000 7f20077fe640 1 -- 172.21.15.67:0/2479815672 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v50) v1 ==== 104+0+0 (secure 0 0 0) 0x7f2000058a90 con 0x7f201006b750 2024-03-24T00:44:44.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.006+0000 7f20157a9640 1 -- 172.21.15.67:0/2479815672 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1ff006bbd0 msgr2=0x7f1ff006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:44.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.006+0000 7f20157a9640 1 --2- 172.21.15.67:0/2479815672 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1ff006bbd0 0x7f1ff006e090 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8007fd0 tx=0x7f1ff8009040 comp rx=0 tx=0).stop 2024-03-24T00:44:44.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.006+0000 7f20157a9640 1 -- 172.21.15.67:0/2479815672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f201006b750 msgr2=0x7f2010144fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:44.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.006+0000 7f20157a9640 1 --2- 172.21.15.67:0/2479815672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f201006b750 0x7f2010144fd0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f200000c450 tx=0x7f200000c910 comp rx=0 tx=0).stop 2024-03-24T00:44:44.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.006+0000 7f20157a9640 1 -- 172.21.15.67:0/2479815672 shutdown_connections 2024-03-24T00:44:44.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.006+0000 7f20157a9640 1 --2- 172.21.15.67:0/2479815672 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1ff006bbd0 0x7f1ff006e090 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:44.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.006+0000 7f20157a9640 1 --2- 172.21.15.67:0/2479815672 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20100b0c10 0x7f20100b6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:44.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.006+0000 7f20157a9640 1 --2- 172.21.15.67:0/2479815672 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f20100afae0 0x7f20100c8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:44.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.006+0000 7f20157a9640 1 --2- 172.21.15.67:0/2479815672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f201006b750 0x7f2010144fd0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:44.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.006+0000 7f20157a9640 1 -- 172.21.15.67:0/2479815672 >> 172.21.15.67:0/2479815672 conn(0x7f2010099a30 msgr2=0x7f20100b1910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:44.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.006+0000 7f20157a9640 1 -- 172.21.15.67:0/2479815672 shutdown_connections 2024-03-24T00:44:44.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.006+0000 7f20157a9640 1 -- 172.21.15.67:0/2479815672 wait complete. 2024-03-24T00:44:44.019 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-24T00:44:44.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.167+0000 7f6790b22640 1 Processor -- start 2024-03-24T00:44:44.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.167+0000 7f6790b22640 1 -- start start 2024-03-24T00:44:44.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.168+0000 7f6790b22640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f678c15dfc0 0x7f678c1624c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:44.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.168+0000 7f6790b22640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f678c0d0d50 0x7f678c0d2b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:44.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.168+0000 7f6790b22640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f678c162a00 con 0x7f678c0ced60 2024-03-24T00:44:44.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.168+0000 7f6790b22640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f678c15e3a0 con 0x7f678c0d0d50 2024-03-24T00:44:44.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.168+0000 7f6790b22640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f678c0cf4d0 con 0x7f678c15dfc0 2024-03-24T00:44:44.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.168+0000 7f6789d9b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f678c0ced60 0x7f678c0d0640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52620/0 (socket says 172.21.15.67:52620) 2024-03-24T00:44:44.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.168+0000 7f6789d9b640 1 -- 172.21.15.67:0/4169518555 learned_addr learned my addr 172.21.15.67:0/4169518555 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:44:44.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.169+0000 7f678ad9d640 1 --2- 172.21.15.67:0/4169518555 >> v2:172.21.15.72:3300/0 conn(0x7f678c0d0d50 0x7f678c0d2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:44.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.169+0000 7f678a59c640 1 --2- 172.21.15.67:0/4169518555 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f678c15dfc0 0x7f678c1624c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:44.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.169+0000 7f678959a640 1 -- 172.21.15.67:0/4169518555 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1573473365 0 0) 0x7f678c162a00 con 0x7f678c0ced60 2024-03-24T00:44:44.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.169+0000 7f678959a640 1 -- 172.21.15.67:0/4169518555 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6778003520 con 0x7f678c0ced60 2024-03-24T00:44:44.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.169+0000 7f678959a640 1 -- 172.21.15.67:0/4169518555 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 36360540 0 0) 0x7f6778003520 con 0x7f678c0ced60 2024-03-24T00:44:44.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.169+0000 7f678959a640 1 -- 172.21.15.67:0/4169518555 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f678c15dfc0 msgr2=0x7f678c1624c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:44.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.169+0000 7f678959a640 1 --2- 172.21.15.67:0/4169518555 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f678c15dfc0 0x7f678c1624c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:44.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.169+0000 7f678959a640 1 -- 172.21.15.67:0/4169518555 >> v2:172.21.15.72:3300/0 conn(0x7f678c0d0d50 msgr2=0x7f678c0d2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:44.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.169+0000 7f678959a640 1 --2- 172.21.15.67:0/4169518555 >> v2:172.21.15.72:3300/0 conn(0x7f678c0d0d50 0x7f678c0d2b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:44.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.169+0000 7f678959a640 1 -- 172.21.15.67:0/4169518555 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f678c0cf1f0 con 0x7f678c0ced60 2024-03-24T00:44:44.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.169+0000 7f678959a640 1 -- 172.21.15.67:0/4169518555 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f6774001690 con 0x7f678c0ced60 2024-03-24T00:44:44.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.169+0000 7f678959a640 1 -- 172.21.15.67:0/4169518555 >> v1:172.21.15.67:6789/0 conn(0x7f678c0ced60 legacy=0x7f678c0d0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:44.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.169+0000 7f678959a640 1 --2- 172.21.15.67:0/4169518555 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f67780039b0 0x7f6778003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:44.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.169+0000 7f678959a640 1 --2- 172.21.15.67:0/4169518555 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f678c15dfc0 0x7f6778004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:44.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.169+0000 7f678959a640 1 --2- 172.21.15.67:0/4169518555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6778004d40 0x7f67780051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:44.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.169+0000 7f678959a640 1 -- 172.21.15.67:0/4169518555 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f678c15e3a0 con 0x7f6778004d40 2024-03-24T00:44:44.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.169+0000 7f678959a640 1 -- 172.21.15.67:0/4169518555 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f678c0cf4d0 con 0x7f67780039b0 2024-03-24T00:44:44.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.169+0000 7f678959a640 1 -- 172.21.15.67:0/4169518555 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f677800b9c0 con 0x7f678c15dfc0 2024-03-24T00:44:44.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.169+0000 7f678ad9d640 1 --2- 172.21.15.67:0/4169518555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6778004d40 0x7f67780051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:44.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.169+0000 7f678a59c640 1 --2- 172.21.15.67:0/4169518555 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f67780039b0 0x7f6778003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:44.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.169+0000 7f6789d9b640 1 --2- 172.21.15.67:0/4169518555 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f678c15dfc0 0x7f6778004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:44.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.169+0000 7f678ad9d640 1 -- 172.21.15.67:0/4169518555 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f678c15dfc0 msgr2=0x7f6778004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:44.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.169+0000 7f678ad9d640 1 --2- 172.21.15.67:0/4169518555 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f678c15dfc0 0x7f6778004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:44.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.169+0000 7f678ad9d640 1 -- 172.21.15.67:0/4169518555 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f67780039b0 msgr2=0x7f6778003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:44.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.169+0000 7f678ad9d640 1 --2- 172.21.15.67:0/4169518555 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f67780039b0 0x7f6778003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:44.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.169+0000 7f678ad9d640 1 -- 172.21.15.67:0/4169518555 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6778003520 con 0x7f6778004d40 2024-03-24T00:44:44.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.170+0000 7f678ad9d640 1 --2- 172.21.15.67:0/4169518555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6778004d40 0x7f67780051c0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f677c009fc0 tx=0x7f6778005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0e12479f6c20df4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:44.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.170+0000 7f678959a640 1 -- 172.21.15.67:0/4169518555 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f677c039070 con 0x7f6778004d40 2024-03-24T00:44:44.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.170+0000 7f678959a640 1 -- 172.21.15.67:0/4169518555 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f677c02cc60 con 0x7f6778004d40 2024-03-24T00:44:44.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.170+0000 7f678959a640 1 -- 172.21.15.67:0/4169518555 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f677c009c10 con 0x7f6778004d40 2024-03-24T00:44:44.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.171+0000 7f6790b22640 1 -- 172.21.15.67:0/4169518555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6778004d40 msgr2=0x7f67780051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:44.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.171+0000 7f6790b22640 1 --2- 172.21.15.67:0/4169518555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6778004d40 0x7f67780051c0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f677c009fc0 tx=0x7f6778005830 comp rx=0 tx=0).stop 2024-03-24T00:44:44.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.171+0000 7f678ad9d640 1 -- 172.21.15.67:0/4169518555 reap_dead start 2024-03-24T00:44:44.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.171+0000 7f6790b22640 1 -- 172.21.15.67:0/4169518555 shutdown_connections 2024-03-24T00:44:44.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.171+0000 7f6790b22640 1 -- 172.21.15.67:0/4169518555 >> 172.21.15.67:0/4169518555 conn(0x7f678c06c880 msgr2=0x7f678c06cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:44.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.171+0000 7f6790b22640 1 -- 172.21.15.67:0/4169518555 shutdown_connections 2024-03-24T00:44:44.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.171+0000 7f6790b22640 1 -- 172.21.15.67:0/4169518555 wait complete. 2024-03-24T00:44:44.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.172+0000 7f6790b22640 1 Processor -- start 2024-03-24T00:44:44.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.172+0000 7f6790b22640 1 -- start start 2024-03-24T00:44:44.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.173+0000 7f6790b22640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6778004d40 0x7f678c15a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:44.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.173+0000 7f6790b22640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f678c15aa10 0x7f678c15ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:44.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.173+0000 7f6790b22640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f678c15f190 0x7f678c15f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:44.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.173+0000 7f6790b22640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f678c0d2300 con 0x7f678c15aa10 2024-03-24T00:44:44.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.173+0000 7f6790b22640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f678c0b3230 con 0x7f678c15f190 2024-03-24T00:44:44.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.173+0000 7f6790b22640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f678c0d25e0 con 0x7f6778004d40 2024-03-24T00:44:44.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.173+0000 7f6789d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f678c15aa10 0x7f678c15ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:44.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.173+0000 7f6789d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f678c15aa10 0x7f678c15ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55880/0 (socket says 172.21.15.67:55880) 2024-03-24T00:44:44.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.173+0000 7f6789d9b640 1 -- 172.21.15.67:0/2322891195 learned_addr learned my addr 172.21.15.67:0/2322891195 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:44.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.173+0000 7f6789d9b640 1 -- 172.21.15.67:0/2322891195 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6778004d40 msgr2=0x7f678c15a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:44.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.173+0000 7f6789d9b640 1 --2- 172.21.15.67:0/2322891195 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6778004d40 0x7f678c15a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:44.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.173+0000 7f6789d9b640 1 -- 172.21.15.67:0/2322891195 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f678c15f190 msgr2=0x7f678c15f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:44.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.173+0000 7f678ad9d640 1 --2- 172.21.15.67:0/2322891195 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f678c15f190 0x7f678c15f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:44.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.173+0000 7f6789d9b640 1 --2- 172.21.15.67:0/2322891195 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f678c15f190 0x7f678c15f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:44.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.173+0000 7f6789d9b640 1 -- 172.21.15.67:0/2322891195 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f678c15af20 con 0x7f678c15aa10 2024-03-24T00:44:44.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.173+0000 7f678ad9d640 1 --2- 172.21.15.67:0/2322891195 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f678c15f190 0x7f678c15f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:44:44.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.173+0000 7f6789d9b640 1 --2- 172.21.15.67:0/2322891195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f678c15aa10 0x7f678c15ec50 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f6774007370 tx=0x7f6774007770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:44.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.174+0000 7f6782ffd640 1 -- 172.21.15.67:0/2322891195 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f677401a070 con 0x7f678c15aa10 2024-03-24T00:44:44.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.174+0000 7f6782ffd640 1 -- 172.21.15.67:0/2322891195 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f677400f960 con 0x7f678c15aa10 2024-03-24T00:44:44.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.174+0000 7f6790b22640 1 -- 172.21.15.67:0/2322891195 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f678c15b200 con 0x7f678c15aa10 2024-03-24T00:44:44.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.174+0000 7f6782ffd640 1 -- 172.21.15.67:0/2322891195 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f677400fd30 con 0x7f678c15aa10 2024-03-24T00:44:44.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.174+0000 7f6790b22640 1 -- 172.21.15.67:0/2322891195 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f678c15e730 con 0x7f678c15aa10 2024-03-24T00:44:44.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.175+0000 7f6790b22640 1 -- 172.21.15.67:0/2322891195 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f678c15e520 con 0x7f678c15aa10 2024-03-24T00:44:44.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.175+0000 7f6782ffd640 1 -- 172.21.15.67:0/2322891195 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f677401d070 con 0x7f678c15aa10 2024-03-24T00:44:44.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.176+0000 7f6782ffd640 1 --2- 172.21.15.67:0/2322891195 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f676c06bc00 0x7f676c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:44.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.176+0000 7f6782ffd640 1 -- 172.21.15.67:0/2322891195 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f677408a720 con 0x7f678c15aa10 2024-03-24T00:44:44.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.176+0000 7f678a59c640 1 --2- 172.21.15.67:0/2322891195 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f676c06bc00 0x7f676c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:44.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.176+0000 7f678a59c640 1 --2- 172.21.15.67:0/2322891195 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f676c06bc00 0x7f676c06e0c0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f678c0cddf0 tx=0x7f6784009320 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:44.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.180+0000 7f6782ffd640 1 -- 172.21.15.67:0/2322891195 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6774059090 con 0x7f678c15aa10 2024-03-24T00:44:44.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:44.364+0000 7f6790b22640 1 -- 172.21.15.67:0/2322891195 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f678c04f400 con 0x7f678c15aa10 2024-03-24T00:44:45.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.005+0000 7f6782ffd640 1 -- 172.21.15.67:0/2322891195 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v79) v1 ==== 181+0+0 (secure 0 0 0) 0x7f6774058990 con 0x7f678c15aa10 2024-03-24T00:44:45.006 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' removed 2024-03-24T00:44:45.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.008+0000 7f6790b22640 1 -- 172.21.15.67:0/2322891195 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f676c06bc00 msgr2=0x7f676c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:45.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.008+0000 7f6790b22640 1 --2- 172.21.15.67:0/2322891195 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f676c06bc00 0x7f676c06e0c0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f678c0cddf0 tx=0x7f6784009320 comp rx=0 tx=0).stop 2024-03-24T00:44:45.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.008+0000 7f6790b22640 1 -- 172.21.15.67:0/2322891195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f678c15aa10 msgr2=0x7f678c15ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:45.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.008+0000 7f6790b22640 1 --2- 172.21.15.67:0/2322891195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f678c15aa10 0x7f678c15ec50 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f6774007370 tx=0x7f6774007770 comp rx=0 tx=0).stop 2024-03-24T00:44:45.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.008+0000 7f6790b22640 1 -- 172.21.15.67:0/2322891195 shutdown_connections 2024-03-24T00:44:45.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.008+0000 7f6790b22640 1 --2- 172.21.15.67:0/2322891195 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f676c06bc00 0x7f676c06e0c0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:45.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.008+0000 7f6790b22640 1 --2- 172.21.15.67:0/2322891195 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f678c15f190 0x7f678c15f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:45.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.008+0000 7f6790b22640 1 --2- 172.21.15.67:0/2322891195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f678c15aa10 0x7f678c15ec50 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:45.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.008+0000 7f6790b22640 1 --2- 172.21.15.67:0/2322891195 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6778004d40 0x7f678c15a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:45.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.008+0000 7f6790b22640 1 -- 172.21.15.67:0/2322891195 >> 172.21.15.67:0/2322891195 conn(0x7f678c06c880 msgr2=0x7f678c0b86e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:45.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.009+0000 7f6790b22640 1 -- 172.21.15.67:0/2322891195 shutdown_connections 2024-03-24T00:44:45.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.009+0000 7f6790b22640 1 -- 172.21.15.67:0/2322891195 wait complete. 2024-03-24T00:44:45.021 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-24T00:44:45.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.131+0000 7f38008b2640 1 Processor -- start 2024-03-24T00:44:45.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.131+0000 7f38008b2640 1 -- start start 2024-03-24T00:44:45.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.132+0000 7f38008b2640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f37fc09a780 0x7f37fc0cfe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:45.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.132+0000 7f38008b2640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f37fc0b2800 0x7f37fc0b2c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:45.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.132+0000 7f38008b2640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37fc0d06d0 con 0x7f37fc06cb60 2024-03-24T00:44:45.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.132+0000 7f38008b2640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37fc0d09b0 con 0x7f37fc0b2800 2024-03-24T00:44:45.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.132+0000 7f38008b2640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37fc0d0c90 con 0x7f37fc09a780 2024-03-24T00:44:45.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.132+0000 7f37fad9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f37fc0b2800 0x7f37fc0b2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:45.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.132+0000 7f37fad9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f37fc0b2800 0x7f37fc0b2c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55888/0 (socket says 172.21.15.67:55888) 2024-03-24T00:44:45.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.132+0000 7f37fad9d640 1 -- 172.21.15.67:0/207344989 learned_addr learned my addr 172.21.15.67:0/207344989 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:45.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.132+0000 7f37f1d9b640 1 --2- 172.21.15.67:0/207344989 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f37fc09a780 0x7f37fc0cfe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:45.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.132+0000 7f37fad9d640 1 -- 172.21.15.67:0/207344989 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f37fc09a780 msgr2=0x7f37fc0cfe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:45.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.132+0000 7f37fad9d640 1 --2- 172.21.15.67:0/207344989 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f37fc09a780 0x7f37fc0cfe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:45.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.132+0000 7f37fad9d640 1 -- 172.21.15.67:0/207344989 >> v1:172.21.15.72:6789/0 conn(0x7f37fc06cb60 legacy=0x7f37fc06cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:45.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.132+0000 7f37fad9d640 1 -- 172.21.15.67:0/207344989 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37fc0d0f70 con 0x7f37fc0b2800 2024-03-24T00:44:45.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.132+0000 7f37f1d9b640 1 --2- 172.21.15.67:0/207344989 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f37fc09a780 0x7f37fc0cfe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:44:45.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.133+0000 7f37fad9d640 1 --2- 172.21.15.67:0/207344989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37fc0b2800 0x7f37fc0b2c90 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f37f4009e50 tx=0x7f37f402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d4647168cb2c856 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:45.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.133+0000 7f37f9d9b640 1 -- 172.21.15.67:0/207344989 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37f402c9e0 con 0x7f37fc0b2800 2024-03-24T00:44:45.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.133+0000 7f37f9d9b640 1 -- 172.21.15.67:0/207344989 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37f402cb80 con 0x7f37fc0b2800 2024-03-24T00:44:45.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.133+0000 7f37f9d9b640 1 -- 172.21.15.67:0/207344989 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37f40343f0 con 0x7f37fc0b2800 2024-03-24T00:44:45.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.134+0000 7f38008b2640 1 -- 172.21.15.67:0/207344989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37fc0b2800 msgr2=0x7f37fc0b2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:45.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.134+0000 7f38008b2640 1 --2- 172.21.15.67:0/207344989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37fc0b2800 0x7f37fc0b2c90 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f37f4009e50 tx=0x7f37f402be40 comp rx=0 tx=0).stop 2024-03-24T00:44:45.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.134+0000 7f38008b2640 1 -- 172.21.15.67:0/207344989 shutdown_connections 2024-03-24T00:44:45.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.134+0000 7f38008b2640 1 --2- 172.21.15.67:0/207344989 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f37fc09a780 0x7f37fc0cfe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:45.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.134+0000 7f38008b2640 1 --2- 172.21.15.67:0/207344989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37fc0b2800 0x7f37fc0b2c90 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:45.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.134+0000 7f38008b2640 1 -- 172.21.15.67:0/207344989 >> 172.21.15.67:0/207344989 conn(0x7f37fc0b36c0 msgr2=0x7f37fc0b3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:45.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.134+0000 7f38008b2640 1 -- 172.21.15.67:0/207344989 shutdown_connections 2024-03-24T00:44:45.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.134+0000 7f38008b2640 1 -- 172.21.15.67:0/207344989 wait complete. 2024-03-24T00:44:45.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.135+0000 7f38008b2640 1 Processor -- start 2024-03-24T00:44:45.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.135+0000 7f38008b2640 1 -- start start 2024-03-24T00:44:45.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.136+0000 7f38008b2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37fc06cb60 0x7f37fc0c70c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:45.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.136+0000 7f38008b2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37fc09a780 0x7f37fc0b69b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:45.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.136+0000 7f38008b2640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f37fc0b2800 0x7f37fc15ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:45.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.136+0000 7f38008b2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37fc0cd230 con 0x7f37fc06cb60 2024-03-24T00:44:45.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.136+0000 7f38008b2640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37fc0d1770 con 0x7f37fc09a780 2024-03-24T00:44:45.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.136+0000 7f38008b2640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37fc0d0350 con 0x7f37fc0b2800 2024-03-24T00:44:45.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.136+0000 7f37fa59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37fc06cb60 0x7f37fc0c70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:45.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.136+0000 7f37fa59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37fc06cb60 0x7f37fc0c70c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55892/0 (socket says 172.21.15.67:55892) 2024-03-24T00:44:45.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.136+0000 7f37fa59c640 1 -- 172.21.15.67:0/1471538878 learned_addr learned my addr 172.21.15.67:0/1471538878 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:45.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.136+0000 7f37f1d9b640 1 --2- 172.21.15.67:0/1471538878 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37fc09a780 0x7f37fc0b69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:45.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.136+0000 7f37fad9d640 1 --2- 172.21.15.67:0/1471538878 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f37fc0b2800 0x7f37fc15ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:45.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.136+0000 7f37fa59c640 1 -- 172.21.15.67:0/1471538878 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f37fc0b2800 msgr2=0x7f37fc15ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:45.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.136+0000 7f37fa59c640 1 --2- 172.21.15.67:0/1471538878 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f37fc0b2800 0x7f37fc15ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:45.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.136+0000 7f37fa59c640 1 -- 172.21.15.67:0/1471538878 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37fc09a780 msgr2=0x7f37fc0b69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:45.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.136+0000 7f37fa59c640 1 --2- 172.21.15.67:0/1471538878 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37fc09a780 0x7f37fc0b69b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:45.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.136+0000 7f37fa59c640 1 -- 172.21.15.67:0/1471538878 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37fc0c8480 con 0x7f37fc06cb60 2024-03-24T00:44:45.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.136+0000 7f37fad9d640 1 --2- 172.21.15.67:0/1471538878 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f37fc0b2800 0x7f37fc15ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:44:45.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.136+0000 7f37f1d9b640 1 --2- 172.21.15.67:0/1471538878 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37fc09a780 0x7f37fc0b69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:44:45.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.136+0000 7f37fa59c640 1 --2- 172.21.15.67:0/1471538878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37fc06cb60 0x7f37fc0c70c0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f37ec00b3f0 tx=0x7f37ec00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:45.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.137+0000 7f37f37fe640 1 -- 172.21.15.67:0/1471538878 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ec015020 con 0x7f37fc06cb60 2024-03-24T00:44:45.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.137+0000 7f37f37fe640 1 -- 172.21.15.67:0/1471538878 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37ec011830 con 0x7f37fc06cb60 2024-03-24T00:44:45.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.137+0000 7f38008b2640 1 -- 172.21.15.67:0/1471538878 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37fc1f44e0 con 0x7f37fc06cb60 2024-03-24T00:44:45.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.137+0000 7f37f37fe640 1 -- 172.21.15.67:0/1471538878 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ec011c20 con 0x7f37fc06cb60 2024-03-24T00:44:45.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.137+0000 7f38008b2640 1 -- 172.21.15.67:0/1471538878 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37fc0c68c0 con 0x7f37fc06cb60 2024-03-24T00:44:45.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.138+0000 7f38008b2640 1 -- 172.21.15.67:0/1471538878 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37fc1f48d0 con 0x7f37fc06cb60 2024-03-24T00:44:45.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.138+0000 7f37f37fe640 1 -- 172.21.15.67:0/1471538878 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f37ec018020 con 0x7f37fc06cb60 2024-03-24T00:44:45.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.139+0000 7f37f37fe640 1 --2- 172.21.15.67:0/1471538878 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f37dc06bbd0 0x7f37dc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:45.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.139+0000 7f37f37fe640 1 -- 172.21.15.67:0/1471538878 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5835+0+0 (secure 0 0 0) 0x7f37ec08a680 con 0x7f37fc06cb60 2024-03-24T00:44:45.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.139+0000 7f37f1d9b640 1 --2- 172.21.15.67:0/1471538878 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f37dc06bbd0 0x7f37dc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:45.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.139+0000 7f37f1d9b640 1 --2- 172.21.15.67:0/1471538878 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f37dc06bbd0 0x7f37dc06e090 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f37fc0b7f50 tx=0x7f37e4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:45.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.143+0000 7f37f37fe640 1 -- 172.21.15.67:0/1471538878 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f37ec01f030 con 0x7f37fc06cb60 2024-03-24T00:44:45.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:45.326+0000 7f38008b2640 1 -- 172.21.15.67:0/1471538878 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f37fc0c7600 con 0x7f37fc06cb60 2024-03-24T00:44:46.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.010+0000 7f37f37fe640 1 -- 172.21.15.67:0/1471538878 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v80) v1 ==== 169+0+0 (secure 0 0 0) 0x7f37ec0172c0 con 0x7f37fc06cb60 2024-03-24T00:44:46.011 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' removed 2024-03-24T00:44:46.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.014+0000 7f38008b2640 1 -- 172.21.15.67:0/1471538878 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f37dc06bbd0 msgr2=0x7f37dc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:46.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.014+0000 7f38008b2640 1 --2- 172.21.15.67:0/1471538878 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f37dc06bbd0 0x7f37dc06e090 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f37fc0b7f50 tx=0x7f37e4008040 comp rx=0 tx=0).stop 2024-03-24T00:44:46.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.014+0000 7f38008b2640 1 -- 172.21.15.67:0/1471538878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37fc06cb60 msgr2=0x7f37fc0c70c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:46.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.014+0000 7f38008b2640 1 --2- 172.21.15.67:0/1471538878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37fc06cb60 0x7f37fc0c70c0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f37ec00b3f0 tx=0x7f37ec00b8b0 comp rx=0 tx=0).stop 2024-03-24T00:44:46.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.014+0000 7f38008b2640 1 -- 172.21.15.67:0/1471538878 shutdown_connections 2024-03-24T00:44:46.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.014+0000 7f38008b2640 1 --2- 172.21.15.67:0/1471538878 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f37dc06bbd0 0x7f37dc06e090 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:46.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.014+0000 7f38008b2640 1 --2- 172.21.15.67:0/1471538878 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f37fc0b2800 0x7f37fc15ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:46.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.014+0000 7f38008b2640 1 --2- 172.21.15.67:0/1471538878 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37fc09a780 0x7f37fc0b69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:46.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.014+0000 7f38008b2640 1 --2- 172.21.15.67:0/1471538878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37fc06cb60 0x7f37fc0c70c0 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:46.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.014+0000 7f38008b2640 1 -- 172.21.15.67:0/1471538878 >> 172.21.15.67:0/1471538878 conn(0x7f37fc0b36c0 msgr2=0x7f37fc160d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:46.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.014+0000 7f38008b2640 1 -- 172.21.15.67:0/1471538878 shutdown_connections 2024-03-24T00:44:46.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.014+0000 7f38008b2640 1 -- 172.21.15.67:0/1471538878 wait complete. 2024-03-24T00:44:46.028 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-24T00:44:46.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.133+0000 7f9bed5b9640 1 Processor -- start 2024-03-24T00:44:46.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.133+0000 7f9bed5b9640 1 -- start start 2024-03-24T00:44:46.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.134+0000 7f9bed5b9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9be80b41e0 0x7f9be80b3620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:46.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.134+0000 7f9bed5b9640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f9be80b3b60 0x7f9be80cc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:46.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.134+0000 7f9bed5b9640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9be80cf090 con 0x7f9be815dfd0 2024-03-24T00:44:46.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.134+0000 7f9bed5b9640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9be80b3fd0 con 0x7f9be80b41e0 2024-03-24T00:44:46.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.134+0000 7f9bed5b9640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9be80cc260 con 0x7f9be80b3b60 2024-03-24T00:44:46.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.134+0000 7f9be67fc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9be80b41e0 0x7f9be80b3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:46.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.134+0000 7f9be67fc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9be80b41e0 0x7f9be80b3620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55898/0 (socket says 172.21.15.67:55898) 2024-03-24T00:44:46.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.134+0000 7f9be67fc640 1 -- 172.21.15.67:0/300548171 learned_addr learned my addr 172.21.15.67:0/300548171 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:46.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.134+0000 7f9be67fc640 1 -- 172.21.15.67:0/300548171 >> v2:172.21.15.72:3300/0 conn(0x7f9be80b3b60 msgr2=0x7f9be80cc6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:46.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.134+0000 7f9be67fc640 1 --2- 172.21.15.67:0/300548171 >> v2:172.21.15.72:3300/0 conn(0x7f9be80b3b60 0x7f9be80cc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:46.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.134+0000 7f9be67fc640 1 -- 172.21.15.67:0/300548171 >> v1:172.21.15.72:6789/0 conn(0x7f9be815dfd0 legacy=0x7f9be80cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:46.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.134+0000 7f9be67fc640 1 -- 172.21.15.67:0/300548171 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9be80d0320 con 0x7f9be80b41e0 2024-03-24T00:44:46.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.135+0000 7f9be67fc640 1 --2- 172.21.15.67:0/300548171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9be80b41e0 0x7f9be80b3620 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f9bd0009870 tx=0x7f9bd002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f5e5c171a676998 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:46.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.135+0000 7f9be5ffb640 1 -- 172.21.15.67:0/300548171 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd0034070 con 0x7f9be80b41e0 2024-03-24T00:44:46.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.135+0000 7f9be5ffb640 1 -- 172.21.15.67:0/300548171 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bd0030860 con 0x7f9be80b41e0 2024-03-24T00:44:46.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.135+0000 7f9be5ffb640 1 -- 172.21.15.67:0/300548171 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd0030c30 con 0x7f9be80b41e0 2024-03-24T00:44:46.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.136+0000 7f9bed5b9640 1 -- 172.21.15.67:0/300548171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9be80b41e0 msgr2=0x7f9be80b3620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:46.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.136+0000 7f9bed5b9640 1 --2- 172.21.15.67:0/300548171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9be80b41e0 0x7f9be80b3620 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f9bd0009870 tx=0x7f9bd002b9a0 comp rx=0 tx=0).stop 2024-03-24T00:44:46.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.136+0000 7f9bed5b9640 1 -- 172.21.15.67:0/300548171 shutdown_connections 2024-03-24T00:44:46.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.136+0000 7f9bed5b9640 1 --2- 172.21.15.67:0/300548171 >> v2:172.21.15.72:3300/0 conn(0x7f9be80b3b60 0x7f9be80cc6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:46.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.136+0000 7f9bed5b9640 1 --2- 172.21.15.67:0/300548171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9be80b41e0 0x7f9be80b3620 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:46.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.136+0000 7f9bed5b9640 1 -- 172.21.15.67:0/300548171 >> 172.21.15.67:0/300548171 conn(0x7f9be806fcf0 msgr2=0x7f9be80700d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:46.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.136+0000 7f9bed5b9640 1 -- 172.21.15.67:0/300548171 shutdown_connections 2024-03-24T00:44:46.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.136+0000 7f9bed5b9640 1 -- 172.21.15.67:0/300548171 wait complete. 2024-03-24T00:44:46.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.137+0000 7f9bed5b9640 1 Processor -- start 2024-03-24T00:44:46.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.137+0000 7f9bed5b9640 1 -- start start 2024-03-24T00:44:46.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.138+0000 7f9bed5b9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9be80b3b60 0x7f9be80cbc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:46.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.138+0000 7f9bed5b9640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9be80b41e0 0x7f9be80b57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:46.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.138+0000 7f9bed5b9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9be815dfd0 0x7f9be81f7520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:46.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.138+0000 7f9bed5b9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9be80cedb0 con 0x7f9be815dfd0 2024-03-24T00:44:46.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.138+0000 7f9bed5b9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9be80cead0 con 0x7f9be80b3b60 2024-03-24T00:44:46.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.138+0000 7f9bed5b9640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9be815e3b0 con 0x7f9be80b41e0 2024-03-24T00:44:46.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.138+0000 7f9be77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9be815dfd0 0x7f9be81f7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:46.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.138+0000 7f9be77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9be815dfd0 0x7f9be81f7520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55910/0 (socket says 172.21.15.67:55910) 2024-03-24T00:44:46.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.138+0000 7f9be77fe640 1 -- 172.21.15.67:0/3925552057 learned_addr learned my addr 172.21.15.67:0/3925552057 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:46.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.138+0000 7f9be6ffd640 1 --2- 172.21.15.67:0/3925552057 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9be80b3b60 0x7f9be80cbc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:46.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.138+0000 7f9be67fc640 1 --2- 172.21.15.67:0/3925552057 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9be80b41e0 0x7f9be80b57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:46.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.138+0000 7f9be77fe640 1 -- 172.21.15.67:0/3925552057 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9be80b41e0 msgr2=0x7f9be80b57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:46.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.138+0000 7f9be77fe640 1 --2- 172.21.15.67:0/3925552057 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9be80b41e0 0x7f9be80b57b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:46.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.138+0000 7f9be77fe640 1 -- 172.21.15.67:0/3925552057 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9be80b3b60 msgr2=0x7f9be80cbc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:46.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.138+0000 7f9be77fe640 1 --2- 172.21.15.67:0/3925552057 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9be80b3b60 0x7f9be80cbc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:46.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.138+0000 7f9be77fe640 1 -- 172.21.15.67:0/3925552057 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9be815a650 con 0x7f9be815dfd0 2024-03-24T00:44:46.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.138+0000 7f9be77fe640 1 --2- 172.21.15.67:0/3925552057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9be815dfd0 0x7f9be81f7520 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f9bd800b9a0 tx=0x7f9bd800be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:46.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.139+0000 7f9bdf7fe640 1 -- 172.21.15.67:0/3925552057 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd8019070 con 0x7f9be815dfd0 2024-03-24T00:44:46.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.139+0000 7f9bdf7fe640 1 -- 172.21.15.67:0/3925552057 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bd800cbc0 con 0x7f9be815dfd0 2024-03-24T00:44:46.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.139+0000 7f9bdf7fe640 1 -- 172.21.15.67:0/3925552057 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd8014440 con 0x7f9be815dfd0 2024-03-24T00:44:46.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.139+0000 7f9bed5b9640 1 -- 172.21.15.67:0/3925552057 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9be81f7f00 con 0x7f9be815dfd0 2024-03-24T00:44:46.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.139+0000 7f9bed5b9640 1 -- 172.21.15.67:0/3925552057 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9be815a8f0 con 0x7f9be815dfd0 2024-03-24T00:44:46.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.140+0000 7f9bed5b9640 1 -- 172.21.15.67:0/3925552057 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9be804f400 con 0x7f9be815dfd0 2024-03-24T00:44:46.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.140+0000 7f9bdf7fe640 1 -- 172.21.15.67:0/3925552057 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9bd801c070 con 0x7f9be815dfd0 2024-03-24T00:44:46.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.140+0000 7f9bdf7fe640 1 --2- 172.21.15.67:0/3925552057 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9bc806bbd0 0x7f9bc806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:46.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.141+0000 7f9bdf7fe640 1 -- 172.21.15.67:0/3925552057 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f9bd808a5d0 con 0x7f9be815dfd0 2024-03-24T00:44:46.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.141+0000 7f9be6ffd640 1 --2- 172.21.15.67:0/3925552057 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9bc806bbd0 0x7f9bc806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:46.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.141+0000 7f9be6ffd640 1 --2- 172.21.15.67:0/3925552057 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9bc806bbd0 0x7f9bc806e090 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f9be0006fd0 tx=0x7f9be0009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:46.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.145+0000 7f9bdf7fe640 1 -- 172.21.15.67:0/3925552057 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9bd8058b90 con 0x7f9be815dfd0 2024-03-24T00:44:46.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.327+0000 7f9bed5b9640 1 -- 172.21.15.67:0/3925552057 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f9be81f8420 con 0x7f9be815dfd0 2024-03-24T00:44:46.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.330+0000 7f9bdf7fe640 1 -- 172.21.15.67:0/3925552057 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v17) v1 ==== 104+0+0 (secure 0 0 0) 0x7f9bd805cc30 con 0x7f9be815dfd0 2024-03-24T00:44:46.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.332+0000 7f9bed5b9640 1 -- 172.21.15.67:0/3925552057 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9bc806bbd0 msgr2=0x7f9bc806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:46.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.332+0000 7f9bed5b9640 1 --2- 172.21.15.67:0/3925552057 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9bc806bbd0 0x7f9bc806e090 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f9be0006fd0 tx=0x7f9be0009060 comp rx=0 tx=0).stop 2024-03-24T00:44:46.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.332+0000 7f9bed5b9640 1 -- 172.21.15.67:0/3925552057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9be815dfd0 msgr2=0x7f9be81f7520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:46.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.332+0000 7f9bed5b9640 1 --2- 172.21.15.67:0/3925552057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9be815dfd0 0x7f9be81f7520 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f9bd800b9a0 tx=0x7f9bd800be60 comp rx=0 tx=0).stop 2024-03-24T00:44:46.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.332+0000 7f9bed5b9640 1 -- 172.21.15.67:0/3925552057 shutdown_connections 2024-03-24T00:44:46.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.332+0000 7f9bed5b9640 1 --2- 172.21.15.67:0/3925552057 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9bc806bbd0 0x7f9bc806e090 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:46.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.332+0000 7f9bed5b9640 1 --2- 172.21.15.67:0/3925552057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9be815dfd0 0x7f9be81f7520 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:46.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.332+0000 7f9bed5b9640 1 --2- 172.21.15.67:0/3925552057 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9be80b41e0 0x7f9be80b57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:46.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.332+0000 7f9bed5b9640 1 --2- 172.21.15.67:0/3925552057 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9be80b3b60 0x7f9be80cbc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:46.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.332+0000 7f9bed5b9640 1 -- 172.21.15.67:0/3925552057 >> 172.21.15.67:0/3925552057 conn(0x7f9be806fcf0 msgr2=0x7f9be81605e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:46.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.332+0000 7f9bed5b9640 1 -- 172.21.15.67:0/3925552057 shutdown_connections 2024-03-24T00:44:46.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.332+0000 7f9bed5b9640 1 -- 172.21.15.67:0/3925552057 wait complete. 2024-03-24T00:44:46.344 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dentry_repair' 2024-03-24T00:44:46.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.445+0000 7f1f3f33b640 1 Processor -- start 2024-03-24T00:44:46.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.445+0000 7f1f3f33b640 1 -- start start 2024-03-24T00:44:46.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.446+0000 7f1f3f33b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1f381445a0 0x7f1f380cee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:46.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.446+0000 7f1f3f33b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1f380b48d0 0x7f1f38072550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:46.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.446+0000 7f1f3f33b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1f3806ec40 0x7f1f3806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:46.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.446+0000 7f1f3f33b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f380ce990 con 0x7f1f380b48d0 2024-03-24T00:44:46.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.446+0000 7f1f3f33b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f380b4d30 con 0x7f1f381445a0 2024-03-24T00:44:46.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.446+0000 7f1f3f33b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f380cddc0 con 0x7f1f3806ec40 2024-03-24T00:44:46.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.446+0000 7f1f3c8d6640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1f380b48d0 0x7f1f38072550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:46.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.446+0000 7f1f3c8d6640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1f380b48d0 0x7f1f38072550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55926/0 (socket says 172.21.15.67:55926) 2024-03-24T00:44:46.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.446+0000 7f1f3c8d6640 1 -- 172.21.15.67:0/3639977902 learned_addr learned my addr 172.21.15.67:0/3639977902 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:46.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.446+0000 7f1f3c8d6640 1 -- 172.21.15.67:0/3639977902 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1f3806ec40 msgr2=0x7f1f3806f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:46.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.446+0000 7f1f3d8d8640 1 --2- 172.21.15.67:0/3639977902 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1f3806ec40 0x7f1f3806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:46.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.446+0000 7f1f3d0d7640 1 --2- 172.21.15.67:0/3639977902 >> v2:172.21.15.72:3300/0 conn(0x7f1f381445a0 0x7f1f380cee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:46.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.447+0000 7f1f3c8d6640 1 --2- 172.21.15.67:0/3639977902 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1f3806ec40 0x7f1f3806f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:46.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.447+0000 7f1f3c8d6640 1 -- 172.21.15.67:0/3639977902 >> v2:172.21.15.72:3300/0 conn(0x7f1f381445a0 msgr2=0x7f1f380cee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:46.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.447+0000 7f1f3c8d6640 1 --2- 172.21.15.67:0/3639977902 >> v2:172.21.15.72:3300/0 conn(0x7f1f381445a0 0x7f1f380cee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:46.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.447+0000 7f1f3c8d6640 1 -- 172.21.15.67:0/3639977902 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f380ce060 con 0x7f1f380b48d0 2024-03-24T00:44:46.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.447+0000 7f1f3d0d7640 1 --2- 172.21.15.67:0/3639977902 >> v2:172.21.15.72:3300/0 conn(0x7f1f381445a0 0x7f1f380cee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:44:46.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.447+0000 7f1f3c8d6640 1 --2- 172.21.15.67:0/3639977902 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f380b48d0 0x7f1f38072550 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f1f3806d3f0 tx=0x7f1f2802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f2a37229c6c4155 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:46.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.447+0000 7f1f2ffff640 1 -- 172.21.15.67:0/3639977902 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f28035020 con 0x7f1f380b48d0 2024-03-24T00:44:46.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.448+0000 7f1f2ffff640 1 -- 172.21.15.67:0/3639977902 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f28031870 con 0x7f1f380b48d0 2024-03-24T00:44:46.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.448+0000 7f1f2ffff640 1 -- 172.21.15.67:0/3639977902 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f28031c40 con 0x7f1f380b48d0 2024-03-24T00:44:46.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.448+0000 7f1f3f33b640 1 -- 172.21.15.67:0/3639977902 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f380b48d0 msgr2=0x7f1f38072550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:46.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.448+0000 7f1f3f33b640 1 --2- 172.21.15.67:0/3639977902 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f380b48d0 0x7f1f38072550 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f1f3806d3f0 tx=0x7f1f2802b910 comp rx=0 tx=0).stop 2024-03-24T00:44:46.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.448+0000 7f1f3f33b640 1 -- 172.21.15.67:0/3639977902 shutdown_connections 2024-03-24T00:44:46.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.448+0000 7f1f3f33b640 1 --2- 172.21.15.67:0/3639977902 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1f3806ec40 0x7f1f3806f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:46.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.448+0000 7f1f3f33b640 1 --2- 172.21.15.67:0/3639977902 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f380b48d0 0x7f1f38072550 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:46.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.448+0000 7f1f3f33b640 1 --2- 172.21.15.67:0/3639977902 >> v2:172.21.15.72:3300/0 conn(0x7f1f381445a0 0x7f1f380cee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:46.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.448+0000 7f1f3f33b640 1 -- 172.21.15.67:0/3639977902 >> 172.21.15.67:0/3639977902 conn(0x7f1f380cf9e0 msgr2=0x7f1f380cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:46.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.448+0000 7f1f3f33b640 1 -- 172.21.15.67:0/3639977902 shutdown_connections 2024-03-24T00:44:46.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.448+0000 7f1f3f33b640 1 -- 172.21.15.67:0/3639977902 wait complete. 2024-03-24T00:44:46.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.449+0000 7f1f3f33b640 1 Processor -- start 2024-03-24T00:44:46.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.449+0000 7f1f3f33b640 1 -- start start 2024-03-24T00:44:46.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.450+0000 7f1f3f33b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f3806ec40 0x7f1f381557d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:46.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.450+0000 7f1f3f33b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f380b48d0 0x7f1f38131310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:46.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.450+0000 7f1f3f33b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1f381445a0 0x7f1f38131850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:46.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.450+0000 7f1f3f33b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f380b2a10 con 0x7f1f380b48d0 2024-03-24T00:44:46.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.450+0000 7f1f3f33b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f380ce620 con 0x7f1f3806ec40 2024-03-24T00:44:46.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.450+0000 7f1f3f33b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f380b2b90 con 0x7f1f381445a0 2024-03-24T00:44:46.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.450+0000 7f1f3c8d6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f380b48d0 0x7f1f38131310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:46.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.450+0000 7f1f3c8d6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f380b48d0 0x7f1f38131310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55938/0 (socket says 172.21.15.67:55938) 2024-03-24T00:44:46.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.450+0000 7f1f3c8d6640 1 -- 172.21.15.67:0/4058484581 learned_addr learned my addr 172.21.15.67:0/4058484581 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:46.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.450+0000 7f1f3d0d7640 1 --2- 172.21.15.67:0/4058484581 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f3806ec40 0x7f1f381557d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:46.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.450+0000 7f1f3d8d8640 1 --2- 172.21.15.67:0/4058484581 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1f381445a0 0x7f1f38131850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:46.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.450+0000 7f1f3c8d6640 1 -- 172.21.15.67:0/4058484581 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1f381445a0 msgr2=0x7f1f38131850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:46.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.450+0000 7f1f3c8d6640 1 --2- 172.21.15.67:0/4058484581 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1f381445a0 0x7f1f38131850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:46.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.450+0000 7f1f3c8d6640 1 -- 172.21.15.67:0/4058484581 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f3806ec40 msgr2=0x7f1f381557d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:46.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.450+0000 7f1f3c8d6640 1 --2- 172.21.15.67:0/4058484581 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f3806ec40 0x7f1f381557d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:46.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.450+0000 7f1f3c8d6640 1 -- 172.21.15.67:0/4058484581 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f38155d10 con 0x7f1f380b48d0 2024-03-24T00:44:46.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.450+0000 7f1f3d0d7640 1 --2- 172.21.15.67:0/4058484581 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f3806ec40 0x7f1f381557d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:44:46.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.451+0000 7f1f3c8d6640 1 --2- 172.21.15.67:0/4058484581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f380b48d0 0x7f1f38131310 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f1f2802b950 tx=0x7f1f280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:46.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.451+0000 7f1f2dffb640 1 -- 172.21.15.67:0/4058484581 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f28040070 con 0x7f1f380b48d0 2024-03-24T00:44:46.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.451+0000 7f1f2dffb640 1 -- 172.21.15.67:0/4058484581 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f2803a420 con 0x7f1f380b48d0 2024-03-24T00:44:46.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.451+0000 7f1f2dffb640 1 -- 172.21.15.67:0/4058484581 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f28035040 con 0x7f1f380b48d0 2024-03-24T00:44:46.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.451+0000 7f1f3f33b640 1 -- 172.21.15.67:0/4058484581 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f3813b650 con 0x7f1f380b48d0 2024-03-24T00:44:46.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.451+0000 7f1f3f33b640 1 -- 172.21.15.67:0/4058484581 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f381321c0 con 0x7f1f380b48d0 2024-03-24T00:44:46.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.452+0000 7f1f3f33b640 1 -- 172.21.15.67:0/4058484581 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f3804f400 con 0x7f1f380b48d0 2024-03-24T00:44:46.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.453+0000 7f1f2dffb640 1 -- 172.21.15.67:0/4058484581 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1f28038020 con 0x7f1f380b48d0 2024-03-24T00:44:46.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.453+0000 7f1f2dffb640 1 --2- 172.21.15.67:0/4058484581 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1f1c06bbd0 0x7f1f1c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:46.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.453+0000 7f1f2dffb640 1 -- 172.21.15.67:0/4058484581 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f1f280ac960 con 0x7f1f380b48d0 2024-03-24T00:44:46.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.453+0000 7f1f3d0d7640 1 --2- 172.21.15.67:0/4058484581 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1f1c06bbd0 0x7f1f1c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:46.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.454+0000 7f1f3d0d7640 1 --2- 172.21.15.67:0/4058484581 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1f1c06bbd0 0x7f1f1c06e090 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f1f30006fd0 tx=0x7f1f30008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:46.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.457+0000 7f1f2dffb640 1 -- 172.21.15.67:0/4058484581 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1f2807b570 con 0x7f1f380b48d0 2024-03-24T00:44:46.573 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:44:46.575 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:44:46.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:46.709+0000 7f1f3f33b640 1 -- 172.21.15.67:0/4058484581 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dentry_repair"]} v 0) v1 -- 0x7f1f380afaa0 con 0x7f1f380b48d0 2024-03-24T00:44:47.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.011+0000 7f1f2dffb640 1 -- 172.21.15.67:0/4058484581 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dentry_repair"]}]=0 v293) v1 ==== 166+0+0 (secure 0 0 0) 0x7f1f2807ae70 con 0x7f1f380b48d0 2024-03-24T00:44:47.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.013+0000 7f1f3f33b640 1 -- 172.21.15.67:0/4058484581 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1f1c06bbd0 msgr2=0x7f1f1c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:47.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.013+0000 7f1f3f33b640 1 --2- 172.21.15.67:0/4058484581 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1f1c06bbd0 0x7f1f1c06e090 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f1f30006fd0 tx=0x7f1f30008040 comp rx=0 tx=0).stop 2024-03-24T00:44:47.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.014+0000 7f1f3f33b640 1 -- 172.21.15.67:0/4058484581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f380b48d0 msgr2=0x7f1f38131310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:47.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.014+0000 7f1f3f33b640 1 --2- 172.21.15.67:0/4058484581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f380b48d0 0x7f1f38131310 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f1f2802b950 tx=0x7f1f280046c0 comp rx=0 tx=0).stop 2024-03-24T00:44:47.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.014+0000 7f1f3f33b640 1 -- 172.21.15.67:0/4058484581 shutdown_connections 2024-03-24T00:44:47.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.014+0000 7f1f3f33b640 1 --2- 172.21.15.67:0/4058484581 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1f1c06bbd0 0x7f1f1c06e090 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:47.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.014+0000 7f1f3f33b640 1 --2- 172.21.15.67:0/4058484581 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1f381445a0 0x7f1f38131850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:47.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.014+0000 7f1f3f33b640 1 --2- 172.21.15.67:0/4058484581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f380b48d0 0x7f1f38131310 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:47.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.014+0000 7f1f3f33b640 1 --2- 172.21.15.67:0/4058484581 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f3806ec40 0x7f1f381557d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:47.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.014+0000 7f1f3f33b640 1 -- 172.21.15.67:0/4058484581 >> 172.21.15.67:0/4058484581 conn(0x7f1f380cf9e0 msgr2=0x7f1f380b0250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:47.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.014+0000 7f1f3f33b640 1 -- 172.21.15.67:0/4058484581 shutdown_connections 2024-03-24T00:44:47.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.014+0000 7f1f3f33b640 1 -- 172.21.15.67:0/4058484581 wait complete. 2024-03-24T00:44:47.025 INFO:tasks.cephfs_test_runner:Test that the damage health status is cleared ... ok 2024-03-24T00:44:47.025 INFO:tasks.cephfs_test_runner:Starting test: test_health_status_after_dirfrag_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) Test that the damage health status is cleared 2024-03-24T00:44:47.026 INFO:tasks.cephfs_test_runner:test_health_status_after_dirfrag_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2024-03-24T00:44:47.026 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:44:47.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.127+0000 7f6010c3f640 1 Processor -- start 2024-03-24T00:44:47.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.127+0000 7f6010c3f640 1 -- start start 2024-03-24T00:44:47.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.127+0000 7f6010c3f640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f600c15e390 0x7f600c06b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:47.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.127+0000 7f6010c3f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f600c0d07d0 con 0x7f600c0cded0 2024-03-24T00:44:47.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.127+0000 7f6010c3f640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f600c0d0ab0 con 0x7f600c15dfb0 2024-03-24T00:44:47.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.127+0000 7f6010c3f640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f600c0d0590 con 0x7f600c15e390 2024-03-24T00:44:47.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.128+0000 7f600ad9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f600c0cded0 0x7f600c0cb120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52632/0 (socket says 172.21.15.67:52632) 2024-03-24T00:44:47.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.128+0000 7f600ad9d640 1 -- 172.21.15.67:0/1617029106 learned_addr learned my addr 172.21.15.67:0/1617029106 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:44:47.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.128+0000 7f6003fff640 1 --2- 172.21.15.67:0/1617029106 >> v2:172.21.15.72:3300/0 conn(0x7f600c15e390 0x7f600c06b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:47.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.128+0000 7f6009d9b640 1 -- 172.21.15.67:0/1617029106 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3136528798 0 0) 0x7f600c0d07d0 con 0x7f600c0cded0 2024-03-24T00:44:47.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.128+0000 7f6009d9b640 1 -- 172.21.15.67:0/1617029106 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ff8003520 con 0x7f600c0cded0 2024-03-24T00:44:47.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.128+0000 7f6009d9b640 1 -- 172.21.15.67:0/1617029106 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2731256160 0 0) 0x7f600c0d0ab0 con 0x7f600c15dfb0 2024-03-24T00:44:47.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.128+0000 7f6009d9b640 1 -- 172.21.15.67:0/1617029106 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f600c0d07d0 con 0x7f600c15dfb0 2024-03-24T00:44:47.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.128+0000 7f6009d9b640 1 -- 172.21.15.67:0/1617029106 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4254296675 0 0) 0x7f5ff8003520 con 0x7f600c0cded0 2024-03-24T00:44:47.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.128+0000 7f6009d9b640 1 -- 172.21.15.67:0/1617029106 >> v2:172.21.15.72:3300/0 conn(0x7f600c15e390 msgr2=0x7f600c06b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:47.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.128+0000 7f6009d9b640 1 --2- 172.21.15.67:0/1617029106 >> v2:172.21.15.72:3300/0 conn(0x7f600c15e390 0x7f600c06b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:47.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.128+0000 7f6009d9b640 1 -- 172.21.15.67:0/1617029106 >> v1:172.21.15.72:6789/0 conn(0x7f600c15dfb0 legacy=0x7f600c0cfe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:47.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.128+0000 7f6009d9b640 1 -- 172.21.15.67:0/1617029106 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f600c0d0d90 con 0x7f600c0cded0 2024-03-24T00:44:47.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.128+0000 7f6009d9b640 1 -- 172.21.15.67:0/1617029106 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f60040019f0 con 0x7f600c0cded0 2024-03-24T00:44:47.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.128+0000 7f6009d9b640 1 -- 172.21.15.67:0/1617029106 >> v1:172.21.15.67:6789/0 conn(0x7f600c0cded0 legacy=0x7f600c0cb120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:47.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.128+0000 7f6009d9b640 1 --2- 172.21.15.67:0/1617029106 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5ff8003b90 0x7f5ff8003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:47.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.128+0000 7f6009d9b640 1 --2- 172.21.15.67:0/1617029106 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ff8004800 0x7f5ff8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:47.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.128+0000 7f6009d9b640 1 --2- 172.21.15.67:0/1617029106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ff80051c0 0x7f5ff800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:47.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.128+0000 7f6009d9b640 1 -- 172.21.15.67:0/1617029106 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f600c0d0590 con 0x7f5ff80051c0 2024-03-24T00:44:47.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.128+0000 7f6009d9b640 1 -- 172.21.15.67:0/1617029106 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff800c090 con 0x7f5ff8004800 2024-03-24T00:44:47.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.128+0000 7f6009d9b640 1 -- 172.21.15.67:0/1617029106 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff800c290 con 0x7f5ff8003b90 2024-03-24T00:44:47.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.128+0000 7f600ad9d640 1 --2- 172.21.15.67:0/1617029106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ff80051c0 0x7f5ff800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:47.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.128+0000 7f6003fff640 1 --2- 172.21.15.67:0/1617029106 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ff8004800 0x7f5ff8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:47.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.128+0000 7f600a59c640 1 --2- 172.21.15.67:0/1617029106 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5ff8003b90 0x7f5ff8003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:47.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.129+0000 7f600ad9d640 1 -- 172.21.15.67:0/1617029106 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5ff8003b90 msgr2=0x7f5ff8003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:47.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.129+0000 7f600ad9d640 1 --2- 172.21.15.67:0/1617029106 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5ff8003b90 0x7f5ff8003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:47.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.129+0000 7f600ad9d640 1 -- 172.21.15.67:0/1617029106 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ff8004800 msgr2=0x7f5ff8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:47.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.129+0000 7f600ad9d640 1 --2- 172.21.15.67:0/1617029106 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ff8004800 0x7f5ff8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:47.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.129+0000 7f600ad9d640 1 -- 172.21.15.67:0/1617029106 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ff8003520 con 0x7f5ff80051c0 2024-03-24T00:44:47.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.129+0000 7f600a59c640 1 --2- 172.21.15.67:0/1617029106 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5ff8003b90 0x7f5ff8003f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:44:47.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.129+0000 7f600ad9d640 1 -- 172.21.15.67:0/1617029106 reap_dead start 2024-03-24T00:44:47.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.129+0000 7f600ad9d640 1 --2- 172.21.15.67:0/1617029106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ff80051c0 0x7f5ff800b9c0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f600400a750 tx=0x7f5ff8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f140a259cfd770d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:47.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.129+0000 7f6009d9b640 1 -- 172.21.15.67:0/1617029106 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f600400a370 con 0x7f5ff80051c0 2024-03-24T00:44:47.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.129+0000 7f6009d9b640 1 -- 172.21.15.67:0/1617029106 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6004034650 con 0x7f5ff80051c0 2024-03-24T00:44:47.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.129+0000 7f6009d9b640 1 -- 172.21.15.67:0/1617029106 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6004034a40 con 0x7f5ff80051c0 2024-03-24T00:44:47.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.130+0000 7f6010c3f640 1 -- 172.21.15.67:0/1617029106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ff80051c0 msgr2=0x7f5ff800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:47.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.130+0000 7f6010c3f640 1 --2- 172.21.15.67:0/1617029106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ff80051c0 0x7f5ff800b9c0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f600400a750 tx=0x7f5ff8005830 comp rx=0 tx=0).stop 2024-03-24T00:44:47.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.130+0000 7f6010c3f640 1 -- 172.21.15.67:0/1617029106 shutdown_connections 2024-03-24T00:44:47.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.130+0000 7f6010c3f640 1 --2- 172.21.15.67:0/1617029106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ff80051c0 0x7f5ff800b9c0 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:47.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.130+0000 7f6010c3f640 1 -- 172.21.15.67:0/1617029106 >> 172.21.15.67:0/1617029106 conn(0x7f600c099a30 msgr2=0x7f600c099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:47.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.130+0000 7f6010c3f640 1 -- 172.21.15.67:0/1617029106 shutdown_connections 2024-03-24T00:44:47.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.130+0000 7f6010c3f640 1 -- 172.21.15.67:0/1617029106 wait complete. 2024-03-24T00:44:47.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.131+0000 7f6010c3f640 1 Processor -- start 2024-03-24T00:44:47.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.131+0000 7f6010c3f640 1 -- start start 2024-03-24T00:44:47.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.132+0000 7f6010c3f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ff80051c0 0x7f600c0c6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:47.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.132+0000 7f6010c3f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f600c158110 0x7f600c1f3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:47.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.132+0000 7f6010c3f640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f600c15ba40 0x7f600c15bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:47.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.132+0000 7f6010c3f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f600c0d1870 con 0x7f5ff80051c0 2024-03-24T00:44:47.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.132+0000 7f6010c3f640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f600c0d1350 con 0x7f600c158110 2024-03-24T00:44:47.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.132+0000 7f6010c3f640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f600c0cfe80 con 0x7f600c15ba40 2024-03-24T00:44:47.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.132+0000 7f600a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ff80051c0 0x7f600c0c6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:47.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.132+0000 7f600a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ff80051c0 0x7f600c0c6fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55950/0 (socket says 172.21.15.67:55950) 2024-03-24T00:44:47.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.132+0000 7f600a59c640 1 -- 172.21.15.67:0/2371347239 learned_addr learned my addr 172.21.15.67:0/2371347239 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:47.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.132+0000 7f6003fff640 1 --2- 172.21.15.67:0/2371347239 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f600c158110 0x7f600c1f3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:47.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.132+0000 7f600ad9d640 1 --2- 172.21.15.67:0/2371347239 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f600c15ba40 0x7f600c15bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:47.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.132+0000 7f600a59c640 1 -- 172.21.15.67:0/2371347239 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f600c15ba40 msgr2=0x7f600c15bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:47.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.132+0000 7f600a59c640 1 --2- 172.21.15.67:0/2371347239 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f600c15ba40 0x7f600c15bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:47.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.132+0000 7f600a59c640 1 -- 172.21.15.67:0/2371347239 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f600c158110 msgr2=0x7f600c1f3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:47.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.132+0000 7f600a59c640 1 --2- 172.21.15.67:0/2371347239 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f600c158110 0x7f600c1f3310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:47.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.132+0000 7f600a59c640 1 -- 172.21.15.67:0/2371347239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f600c159290 con 0x7f5ff80051c0 2024-03-24T00:44:47.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.132+0000 7f600ad9d640 1 --2- 172.21.15.67:0/2371347239 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f600c15ba40 0x7f600c15bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:44:47.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.132+0000 7f6003fff640 1 --2- 172.21.15.67:0/2371347239 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f600c158110 0x7f600c1f3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:44:47.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.133+0000 7f600a59c640 1 --2- 172.21.15.67:0/2371347239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ff80051c0 0x7f600c0c6fd0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc00e7b0 tx=0x7f5ffc00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:47.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.133+0000 7f6002ffd640 1 -- 172.21.15.67:0/2371347239 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc0050d0 con 0x7f5ff80051c0 2024-03-24T00:44:47.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.133+0000 7f6002ffd640 1 -- 172.21.15.67:0/2371347239 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ffc005270 con 0x7f5ff80051c0 2024-03-24T00:44:47.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.133+0000 7f6010c3f640 1 -- 172.21.15.67:0/2371347239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f600c1589f0 con 0x7f5ff80051c0 2024-03-24T00:44:47.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.133+0000 7f6002ffd640 1 -- 172.21.15.67:0/2371347239 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc005440 con 0x7f5ff80051c0 2024-03-24T00:44:47.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.133+0000 7f6010c3f640 1 -- 172.21.15.67:0/2371347239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f600c158fb0 con 0x7f5ff80051c0 2024-03-24T00:44:47.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.134+0000 7f6010c3f640 1 -- 172.21.15.67:0/2371347239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f600c04f400 con 0x7f5ff80051c0 2024-03-24T00:44:47.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.134+0000 7f6002ffd640 1 -- 172.21.15.67:0/2371347239 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5ffc01a020 con 0x7f5ff80051c0 2024-03-24T00:44:47.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.135+0000 7f6002ffd640 1 --2- 172.21.15.67:0/2371347239 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5fec06bc00 0x7f5fec06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:47.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.135+0000 7f6002ffd640 1 -- 172.21.15.67:0/2371347239 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f5ffc08a020 con 0x7f5ff80051c0 2024-03-24T00:44:47.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.135+0000 7f6003fff640 1 --2- 172.21.15.67:0/2371347239 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5fec06bc00 0x7f5fec06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:47.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.136+0000 7f6003fff640 1 --2- 172.21.15.67:0/2371347239 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5fec06bc00 0x7f5fec06e0c0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f5ff40093d0 tx=0x7f5ff4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:47.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.139+0000 7f6002ffd640 1 -- 172.21.15.67:0/2371347239 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5ffc058cf0 con 0x7f5ff80051c0 2024-03-24T00:44:47.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.330+0000 7f6010c3f640 1 -- 172.21.15.67:0/2371347239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f600c0b6a50 con 0x7f5ff80051c0 2024-03-24T00:44:47.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.331+0000 7f6002ffd640 1 -- 172.21.15.67:0/2371347239 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+13362 (secure 0 0 0) 0x7f5ffc0585f0 con 0x7f5ff80051c0 2024-03-24T00:44:47.332 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:44:47.332 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":80,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:44:46.007418+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6832/720256752":"2024-03-25T00:44:42.999299+0000","172.21.15.67:6833/2392802741":"2024-03-25T00:43:24.373788+0000","172.21.15.67:6833/720256752":"2024-03-25T00:44:42.999299+0000","172.21.15.67:6832/2392802741":"2024-03-25T00:43:24.373788+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:44:47.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.333+0000 7f6010c3f640 1 -- 172.21.15.67:0/2371347239 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5fec06bc00 msgr2=0x7f5fec06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:47.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.333+0000 7f6010c3f640 1 --2- 172.21.15.67:0/2371347239 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5fec06bc00 0x7f5fec06e0c0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f5ff40093d0 tx=0x7f5ff4008040 comp rx=0 tx=0).stop 2024-03-24T00:44:47.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.333+0000 7f6010c3f640 1 -- 172.21.15.67:0/2371347239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ff80051c0 msgr2=0x7f600c0c6fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:47.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.333+0000 7f6010c3f640 1 --2- 172.21.15.67:0/2371347239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ff80051c0 0x7f600c0c6fd0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc00e7b0 tx=0x7f5ffc00ec70 comp rx=0 tx=0).stop 2024-03-24T00:44:47.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.333+0000 7f6010c3f640 1 -- 172.21.15.67:0/2371347239 shutdown_connections 2024-03-24T00:44:47.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.333+0000 7f6010c3f640 1 --2- 172.21.15.67:0/2371347239 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5fec06bc00 0x7f5fec06e0c0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:47.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.333+0000 7f6010c3f640 1 --2- 172.21.15.67:0/2371347239 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f600c15ba40 0x7f600c15bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:47.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.333+0000 7f6010c3f640 1 --2- 172.21.15.67:0/2371347239 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f600c158110 0x7f600c1f3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:47.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.333+0000 7f6010c3f640 1 --2- 172.21.15.67:0/2371347239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ff80051c0 0x7f600c0c6fd0 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:47.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.333+0000 7f6010c3f640 1 -- 172.21.15.67:0/2371347239 >> 172.21.15.67:0/2371347239 conn(0x7f600c099a30 msgr2=0x7f600c06ec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:47.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.333+0000 7f6010c3f640 1 -- 172.21.15.67:0/2371347239 shutdown_connections 2024-03-24T00:44:47.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.333+0000 7f6010c3f640 1 -- 172.21.15.67:0/2371347239 wait complete. 2024-03-24T00:44:47.345 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:38:39.364950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:44:47.345 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:44:47.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.448+0000 7f26d1b48640 1 Processor -- start 2024-03-24T00:44:47.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.448+0000 7f26d1b48640 1 -- start start 2024-03-24T00:44:47.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.449+0000 7f26d1b48640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f26cc0d18b0 0x7f26cc15c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:47.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.449+0000 7f26d1b48640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f26cc0d1c90 0x7f26cc15fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:47.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.449+0000 7f26d1b48640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f26cc160260 0x7f26cc0b1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:47.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.449+0000 7f26d1b48640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26cc15c7c0 con 0x7f26cc0d18b0 2024-03-24T00:44:47.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.449+0000 7f26d1b48640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26cc15caa0 con 0x7f26cc160260 2024-03-24T00:44:47.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.449+0000 7f26d1b48640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26cc15cd80 con 0x7f26cc0d1c90 2024-03-24T00:44:47.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.449+0000 7f26cb7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f26cc0d18b0 0x7f26cc15c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:47.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.449+0000 7f26cb7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f26cc0d18b0 0x7f26cc15c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55960/0 (socket says 172.21.15.67:55960) 2024-03-24T00:44:47.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.449+0000 7f26cb7fe640 1 -- 172.21.15.67:0/207683100 learned_addr learned my addr 172.21.15.67:0/207683100 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:47.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.450+0000 7f26cb7fe640 1 -- 172.21.15.67:0/207683100 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f26cc0d1c90 msgr2=0x7f26cc15fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:47.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.450+0000 7f26caffd640 1 --2- 172.21.15.67:0/207683100 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f26cc0d1c90 0x7f26cc15fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:47.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.450+0000 7f26cbfff640 1 --2- 172.21.15.67:0/207683100 >> v2:172.21.15.72:3300/0 conn(0x7f26cc160260 0x7f26cc0b1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:47.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.450+0000 7f26cb7fe640 1 --2- 172.21.15.67:0/207683100 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f26cc0d1c90 0x7f26cc15fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:47.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.450+0000 7f26cb7fe640 1 -- 172.21.15.67:0/207683100 >> v2:172.21.15.72:3300/0 conn(0x7f26cc160260 msgr2=0x7f26cc0b1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:47.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.450+0000 7f26cb7fe640 1 --2- 172.21.15.67:0/207683100 >> v2:172.21.15.72:3300/0 conn(0x7f26cc160260 0x7f26cc0b1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:47.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.450+0000 7f26cb7fe640 1 -- 172.21.15.67:0/207683100 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26cc15d060 con 0x7f26cc0d18b0 2024-03-24T00:44:47.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.450+0000 7f26caffd640 1 --2- 172.21.15.67:0/207683100 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f26cc0d1c90 0x7f26cc15fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:44:47.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.450+0000 7f26cb7fe640 1 --2- 172.21.15.67:0/207683100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26cc0d18b0 0x7f26cc15c1b0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f26cc06e360 tx=0x7f26bc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aba8095a9b5ec0ed server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:47.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.451+0000 7f26ca7fc640 1 -- 172.21.15.67:0/207683100 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26bc02e070 con 0x7f26cc0d18b0 2024-03-24T00:44:47.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.451+0000 7f26ca7fc640 1 -- 172.21.15.67:0/207683100 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26bc037070 con 0x7f26cc0d18b0 2024-03-24T00:44:47.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.451+0000 7f26ca7fc640 1 -- 172.21.15.67:0/207683100 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26bc031a00 con 0x7f26cc0d18b0 2024-03-24T00:44:47.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.451+0000 7f26d1b48640 1 -- 172.21.15.67:0/207683100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26cc0d18b0 msgr2=0x7f26cc15c1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:47.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.451+0000 7f26d1b48640 1 --2- 172.21.15.67:0/207683100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26cc0d18b0 0x7f26cc15c1b0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f26cc06e360 tx=0x7f26bc02b880 comp rx=0 tx=0).stop 2024-03-24T00:44:47.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.451+0000 7f26d1b48640 1 -- 172.21.15.67:0/207683100 shutdown_connections 2024-03-24T00:44:47.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.451+0000 7f26d1b48640 1 --2- 172.21.15.67:0/207683100 >> v2:172.21.15.72:3300/0 conn(0x7f26cc160260 0x7f26cc0b1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:47.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.451+0000 7f26d1b48640 1 --2- 172.21.15.67:0/207683100 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f26cc0d1c90 0x7f26cc15fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:47.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.451+0000 7f26d1b48640 1 --2- 172.21.15.67:0/207683100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26cc0d18b0 0x7f26cc15c1b0 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:47.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.451+0000 7f26d1b48640 1 -- 172.21.15.67:0/207683100 >> 172.21.15.67:0/207683100 conn(0x7f26cc08b1b0 msgr2=0x7f26cc08b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:47.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.451+0000 7f26d1b48640 1 -- 172.21.15.67:0/207683100 shutdown_connections 2024-03-24T00:44:47.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.451+0000 7f26d1b48640 1 -- 172.21.15.67:0/207683100 wait complete. 2024-03-24T00:44:47.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.452+0000 7f26d1b48640 1 Processor -- start 2024-03-24T00:44:47.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.453+0000 7f26d1b48640 1 -- start start 2024-03-24T00:44:47.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.453+0000 7f26d1b48640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f26cc0d18b0 0x7f26cc14e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:47.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.453+0000 7f26d1b48640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26cc0d1c90 0x7f26cc06c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:47.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.453+0000 7f26d1b48640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f26cc160260 0x7f26cc0b36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:47.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.453+0000 7f26d1b48640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26cc0d2270 con 0x7f26cc0d1c90 2024-03-24T00:44:47.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.453+0000 7f26d1b48640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26cc0d20f0 con 0x7f26cc0d18b0 2024-03-24T00:44:47.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.453+0000 7f26d1b48640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26cc0b22b0 con 0x7f26cc160260 2024-03-24T00:44:47.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.453+0000 7f26caffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26cc0d1c90 0x7f26cc06c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:47.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.453+0000 7f26caffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26cc0d1c90 0x7f26cc06c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55968/0 (socket says 172.21.15.67:55968) 2024-03-24T00:44:47.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.453+0000 7f26caffd640 1 -- 172.21.15.67:0/287569720 learned_addr learned my addr 172.21.15.67:0/287569720 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:47.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.453+0000 7f26cb7fe640 1 --2- 172.21.15.67:0/287569720 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f26cc0d18b0 0x7f26cc14e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:47.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.454+0000 7f26cbfff640 1 --2- 172.21.15.67:0/287569720 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f26cc160260 0x7f26cc0b36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:47.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.454+0000 7f26caffd640 1 -- 172.21.15.67:0/287569720 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f26cc160260 msgr2=0x7f26cc0b36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:47.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.454+0000 7f26caffd640 1 --2- 172.21.15.67:0/287569720 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f26cc160260 0x7f26cc0b36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:47.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.454+0000 7f26caffd640 1 -- 172.21.15.67:0/287569720 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f26cc0d18b0 msgr2=0x7f26cc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:47.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.454+0000 7f26caffd640 1 --2- 172.21.15.67:0/287569720 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f26cc0d18b0 0x7f26cc14e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:47.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.454+0000 7f26caffd640 1 -- 172.21.15.67:0/287569720 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26cc06ca30 con 0x7f26cc0d1c90 2024-03-24T00:44:47.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.454+0000 7f26cb7fe640 1 --2- 172.21.15.67:0/287569720 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f26cc0d18b0 0x7f26cc14e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:44:47.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.454+0000 7f26caffd640 1 --2- 172.21.15.67:0/287569720 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26cc0d1c90 0x7f26cc06c150 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f26c000c820 tx=0x7f26c000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:47.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.454+0000 7f26b3fff640 1 -- 172.21.15.67:0/287569720 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c0004230 con 0x7f26cc0d1c90 2024-03-24T00:44:47.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.454+0000 7f26b3fff640 1 -- 172.21.15.67:0/287569720 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26c00043d0 con 0x7f26cc0d1c90 2024-03-24T00:44:47.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.454+0000 7f26d1b48640 1 -- 172.21.15.67:0/287569720 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26cc06cd10 con 0x7f26cc0d1c90 2024-03-24T00:44:47.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.454+0000 7f26b3fff640 1 -- 172.21.15.67:0/287569720 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c0007a70 con 0x7f26cc0d1c90 2024-03-24T00:44:47.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.454+0000 7f26d1b48640 1 -- 172.21.15.67:0/287569720 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26cc06d2d0 con 0x7f26cc0d1c90 2024-03-24T00:44:47.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.455+0000 7f26d1b48640 1 -- 172.21.15.67:0/287569720 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26cc04f400 con 0x7f26cc0d1c90 2024-03-24T00:44:47.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.455+0000 7f26b3fff640 1 -- 172.21.15.67:0/287569720 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f26c0007c10 con 0x7f26cc0d1c90 2024-03-24T00:44:47.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.456+0000 7f26b3fff640 1 --2- 172.21.15.67:0/287569720 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f26b806bbd0 0x7f26b806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:47.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.456+0000 7f26b3fff640 1 -- 172.21.15.67:0/287569720 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f26c008a9e0 con 0x7f26cc0d1c90 2024-03-24T00:44:47.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.456+0000 7f26cb7fe640 1 --2- 172.21.15.67:0/287569720 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f26b806bbd0 0x7f26b806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:47.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.457+0000 7f26cb7fe640 1 --2- 172.21.15.67:0/287569720 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f26b806bbd0 0x7f26b806e090 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f26bc036000 tx=0x7f26bc03d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:47.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.460+0000 7f26b3fff640 1 -- 172.21.15.67:0/287569720 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f26c0015020 con 0x7f26cc0d1c90 2024-03-24T00:44:47.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.642+0000 7f26d1b48640 1 -- 172.21.15.67:0/287569720 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f26cc0b1840 con 0x7f26cc0d1c90 2024-03-24T00:44:47.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.642+0000 7f26b3fff640 1 -- 172.21.15.67:0/287569720 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v80) v1 ==== 93+0+10 (secure 0 0 0) 0x7f26c00595f0 con 0x7f26cc0d1c90 2024-03-24T00:44:47.643 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-24T00:44:47.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.645+0000 7f26d1b48640 1 -- 172.21.15.67:0/287569720 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f26b806bbd0 msgr2=0x7f26b806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:47.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.645+0000 7f26d1b48640 1 --2- 172.21.15.67:0/287569720 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f26b806bbd0 0x7f26b806e090 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f26bc036000 tx=0x7f26bc03d040 comp rx=0 tx=0).stop 2024-03-24T00:44:47.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.645+0000 7f26d1b48640 1 -- 172.21.15.67:0/287569720 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26cc0d1c90 msgr2=0x7f26cc06c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:47.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.645+0000 7f26d1b48640 1 --2- 172.21.15.67:0/287569720 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26cc0d1c90 0x7f26cc06c150 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f26c000c820 tx=0x7f26c000cce0 comp rx=0 tx=0).stop 2024-03-24T00:44:47.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.645+0000 7f26d1b48640 1 -- 172.21.15.67:0/287569720 shutdown_connections 2024-03-24T00:44:47.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.645+0000 7f26d1b48640 1 --2- 172.21.15.67:0/287569720 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f26b806bbd0 0x7f26b806e090 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:47.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.645+0000 7f26d1b48640 1 --2- 172.21.15.67:0/287569720 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f26cc160260 0x7f26cc0b36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:47.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.645+0000 7f26d1b48640 1 --2- 172.21.15.67:0/287569720 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26cc0d1c90 0x7f26cc06c150 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:47.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.645+0000 7f26d1b48640 1 --2- 172.21.15.67:0/287569720 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f26cc0d18b0 0x7f26cc14e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:47.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.645+0000 7f26d1b48640 1 -- 172.21.15.67:0/287569720 >> 172.21.15.67:0/287569720 conn(0x7f26cc08b1b0 msgr2=0x7f26cc15d680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:47.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.645+0000 7f26d1b48640 1 -- 172.21.15.67:0/287569720 shutdown_connections 2024-03-24T00:44:47.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.645+0000 7f26d1b48640 1 -- 172.21.15.67:0/287569720 wait complete. 2024-03-24T00:44:47.657 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dirfrag_repair' 2024-03-24T00:44:47.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.757+0000 7f6833c52640 1 Processor -- start 2024-03-24T00:44:47.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.758+0000 7f6833c52640 1 -- start start 2024-03-24T00:44:47.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.758+0000 7f6833c52640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f682c0b49c0 0x7f682c071ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:47.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.758+0000 7f6833c52640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f682c072010 0x7f682c072480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:47.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.758+0000 7f6833c52640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f682c06db30 con 0x7f682c15b430 2024-03-24T00:44:47.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.758+0000 7f6833c52640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f682c0b4e20 con 0x7f682c072010 2024-03-24T00:44:47.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.758+0000 7f6833c52640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f682c15b810 con 0x7f682c0b49c0 2024-03-24T00:44:47.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.759+0000 7f68321ef640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f682c072010 0x7f682c072480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:47.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.759+0000 7f68321ef640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f682c072010 0x7f682c072480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55980/0 (socket says 172.21.15.67:55980) 2024-03-24T00:44:47.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.759+0000 7f68321ef640 1 -- 172.21.15.67:0/3311863544 learned_addr learned my addr 172.21.15.67:0/3311863544 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:47.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.759+0000 7f68311ed640 1 --2- 172.21.15.67:0/3311863544 >> v2:172.21.15.72:3300/0 conn(0x7f682c0b49c0 0x7f682c071ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:47.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.759+0000 7f68321ef640 1 -- 172.21.15.67:0/3311863544 >> v2:172.21.15.72:3300/0 conn(0x7f682c0b49c0 msgr2=0x7f682c071ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:47.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.759+0000 7f68321ef640 1 --2- 172.21.15.67:0/3311863544 >> v2:172.21.15.72:3300/0 conn(0x7f682c0b49c0 0x7f682c071ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:47.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.759+0000 7f68321ef640 1 -- 172.21.15.67:0/3311863544 >> v1:172.21.15.72:6789/0 conn(0x7f682c15b430 legacy=0x7f682c06f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:47.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.759+0000 7f68321ef640 1 -- 172.21.15.67:0/3311863544 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f682c06de10 con 0x7f682c072010 2024-03-24T00:44:47.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.760+0000 7f68321ef640 1 --2- 172.21.15.67:0/3311863544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f682c072010 0x7f682c072480 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f682c072f20 tx=0x7f681c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac5885349dc31ca6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:47.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.760+0000 7f68309ec640 1 -- 172.21.15.67:0/3311863544 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681c02c9f0 con 0x7f682c072010 2024-03-24T00:44:47.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.760+0000 7f68309ec640 1 -- 172.21.15.67:0/3311863544 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f681c02cb90 con 0x7f682c072010 2024-03-24T00:44:47.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.760+0000 7f68309ec640 1 -- 172.21.15.67:0/3311863544 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681c0343f0 con 0x7f682c072010 2024-03-24T00:44:47.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.760+0000 7f6833c52640 1 -- 172.21.15.67:0/3311863544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f682c072010 msgr2=0x7f682c072480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:47.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.760+0000 7f6833c52640 1 --2- 172.21.15.67:0/3311863544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f682c072010 0x7f682c072480 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f682c072f20 tx=0x7f681c02be50 comp rx=0 tx=0).stop 2024-03-24T00:44:47.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.760+0000 7f6833c52640 1 -- 172.21.15.67:0/3311863544 shutdown_connections 2024-03-24T00:44:47.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.760+0000 7f6833c52640 1 --2- 172.21.15.67:0/3311863544 >> v2:172.21.15.72:3300/0 conn(0x7f682c0b49c0 0x7f682c071ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:47.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.760+0000 7f6833c52640 1 --2- 172.21.15.67:0/3311863544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f682c072010 0x7f682c072480 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:47.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.760+0000 7f6833c52640 1 -- 172.21.15.67:0/3311863544 >> 172.21.15.67:0/3311863544 conn(0x7f682c0cf9e0 msgr2=0x7f682c0cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:47.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.760+0000 7f6833c52640 1 -- 172.21.15.67:0/3311863544 shutdown_connections 2024-03-24T00:44:47.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.761+0000 7f6833c52640 1 -- 172.21.15.67:0/3311863544 wait complete. 2024-03-24T00:44:47.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.762+0000 7f6833c52640 1 Processor -- start 2024-03-24T00:44:47.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.762+0000 7f6833c52640 1 -- start start 2024-03-24T00:44:47.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.762+0000 7f6833c52640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f682c072010 0x7f682c1f01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:47.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.762+0000 7f6833c52640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f682c0b49c0 0x7f682c1522b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:47.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.762+0000 7f6833c52640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f682c15b430 0x7f682c1527f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:47.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.762+0000 7f6833c52640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f682c06e6b0 con 0x7f682c0b49c0 2024-03-24T00:44:47.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.762+0000 7f6833c52640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f682c06e3d0 con 0x7f682c15b430 2024-03-24T00:44:47.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.762+0000 7f6833c52640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f682c0b3d30 con 0x7f682c072010 2024-03-24T00:44:47.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.762+0000 7f68311ed640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f682c0b49c0 0x7f682c1522b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:47.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.762+0000 7f68311ed640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f682c0b49c0 0x7f682c1522b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55982/0 (socket says 172.21.15.67:55982) 2024-03-24T00:44:47.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.762+0000 7f68311ed640 1 -- 172.21.15.67:0/1543813021 learned_addr learned my addr 172.21.15.67:0/1543813021 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:47.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.763+0000 7f68321ef640 1 --2- 172.21.15.67:0/1543813021 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f682c15b430 0x7f682c1527f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:47.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.763+0000 7f68319ee640 1 --2- 172.21.15.67:0/1543813021 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f682c072010 0x7f682c1f01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:47.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.763+0000 7f68311ed640 1 -- 172.21.15.67:0/1543813021 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f682c072010 msgr2=0x7f682c1f01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:47.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.763+0000 7f68311ed640 1 --2- 172.21.15.67:0/1543813021 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f682c072010 0x7f682c1f01b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:47.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.763+0000 7f68311ed640 1 -- 172.21.15.67:0/1543813021 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f682c15b430 msgr2=0x7f682c1527f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:47.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.763+0000 7f68311ed640 1 --2- 172.21.15.67:0/1543813021 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f682c15b430 0x7f682c1527f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:47.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.763+0000 7f68311ed640 1 -- 172.21.15.67:0/1543813021 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f682c155ad0 con 0x7f682c0b49c0 2024-03-24T00:44:47.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.763+0000 7f68311ed640 1 --2- 172.21.15.67:0/1543813021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f682c0b49c0 0x7f682c1522b0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f682000eac0 tx=0x7f682000ef80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:47.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.763+0000 7f682a7fc640 1 -- 172.21.15.67:0/1543813021 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6820007c10 con 0x7f682c0b49c0 2024-03-24T00:44:47.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.763+0000 7f682a7fc640 1 -- 172.21.15.67:0/1543813021 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6820007db0 con 0x7f682c0b49c0 2024-03-24T00:44:47.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.763+0000 7f6833c52640 1 -- 172.21.15.67:0/1543813021 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f682c1412b0 con 0x7f682c0b49c0 2024-03-24T00:44:47.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.763+0000 7f682a7fc640 1 -- 172.21.15.67:0/1543813021 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6820004240 con 0x7f682c0b49c0 2024-03-24T00:44:47.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.763+0000 7f6833c52640 1 -- 172.21.15.67:0/1543813021 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f682c141870 con 0x7f682c0b49c0 2024-03-24T00:44:47.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.764+0000 7f6833c52640 1 -- 172.21.15.67:0/1543813021 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f682c04f400 con 0x7f682c0b49c0 2024-03-24T00:44:47.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.765+0000 7f682a7fc640 1 -- 172.21.15.67:0/1543813021 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f682000b030 con 0x7f682c0b49c0 2024-03-24T00:44:47.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.765+0000 7f682a7fc640 1 --2- 172.21.15.67:0/1543813021 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f681806bbd0 0x7f681806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:47.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.765+0000 7f682a7fc640 1 -- 172.21.15.67:0/1543813021 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f682008cb80 con 0x7f682c0b49c0 2024-03-24T00:44:47.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.765+0000 7f68319ee640 1 --2- 172.21.15.67:0/1543813021 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f681806bbd0 0x7f681806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:47.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.766+0000 7f68319ee640 1 --2- 172.21.15.67:0/1543813021 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f681806bbd0 0x7f681806e090 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f6824006fd0 tx=0x7f6824009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:47.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:47.769+0000 7f682a7fc640 1 -- 172.21.15.67:0/1543813021 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f682005b1d0 con 0x7f682c0b49c0 2024-03-24T00:44:48.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:48.022+0000 7f6833c52640 1 -- 172.21.15.67:0/1543813021 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dirfrag_repair"]} v 0) v1 -- 0x7f682c06f350 con 0x7f682c0b49c0 2024-03-24T00:44:49.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.018+0000 7f682a7fc640 1 -- 172.21.15.67:0/1543813021 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dirfrag_repair"]}]=0 v295) v1 ==== 170+0+0 (secure 0 0 0) 0x7f682005f270 con 0x7f682c0b49c0 2024-03-24T00:44:49.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.020+0000 7f6833c52640 1 -- 172.21.15.67:0/1543813021 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f681806bbd0 msgr2=0x7f681806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:49.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.020+0000 7f6833c52640 1 --2- 172.21.15.67:0/1543813021 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f681806bbd0 0x7f681806e090 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f6824006fd0 tx=0x7f6824009060 comp rx=0 tx=0).stop 2024-03-24T00:44:49.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.020+0000 7f6833c52640 1 -- 172.21.15.67:0/1543813021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f682c0b49c0 msgr2=0x7f682c1522b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:49.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.021+0000 7f6833c52640 1 --2- 172.21.15.67:0/1543813021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f682c0b49c0 0x7f682c1522b0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f682000eac0 tx=0x7f682000ef80 comp rx=0 tx=0).stop 2024-03-24T00:44:49.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.021+0000 7f6833c52640 1 -- 172.21.15.67:0/1543813021 shutdown_connections 2024-03-24T00:44:49.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.021+0000 7f6833c52640 1 --2- 172.21.15.67:0/1543813021 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f681806bbd0 0x7f681806e090 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:49.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.021+0000 7f6833c52640 1 --2- 172.21.15.67:0/1543813021 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f682c15b430 0x7f682c1527f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:49.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.021+0000 7f6833c52640 1 --2- 172.21.15.67:0/1543813021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f682c0b49c0 0x7f682c1522b0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:49.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.021+0000 7f6833c52640 1 --2- 172.21.15.67:0/1543813021 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f682c072010 0x7f682c1f01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:49.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.021+0000 7f6833c52640 1 -- 172.21.15.67:0/1543813021 >> 172.21.15.67:0/1543813021 conn(0x7f682c0cf9e0 msgr2=0x7f682c0ce650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:49.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.021+0000 7f6833c52640 1 -- 172.21.15.67:0/1543813021 shutdown_connections 2024-03-24T00:44:49.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.021+0000 7f6833c52640 1 -- 172.21.15.67:0/1543813021 wait complete. 2024-03-24T00:44:49.035 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-24T00:44:49.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.145+0000 7f09becbf640 1 Processor -- start 2024-03-24T00:44:49.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.145+0000 7f09becbf640 1 -- start start 2024-03-24T00:44:49.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.146+0000 7f09becbf640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f09b80b6170 0x7f09b8168c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:49.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.146+0000 7f09becbf640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09b80d2250 con 0x7f09b80d3ae0 2024-03-24T00:44:49.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.146+0000 7f09becbf640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09b80d2530 con 0x7f09b815dea0 2024-03-24T00:44:49.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.146+0000 7f09becbf640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09b80d2810 con 0x7f09b80b6170 2024-03-24T00:44:49.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.146+0000 7f09affff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f09b80d3ae0 0x7f09b80b5a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52638/0 (socket says 172.21.15.67:52638) 2024-03-24T00:44:49.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.146+0000 7f09affff640 1 -- 172.21.15.67:0/3835701185 learned_addr learned my addr 172.21.15.67:0/3835701185 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:44:49.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.146+0000 7f09af7fe640 1 -- 172.21.15.67:0/3835701185 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3122334008 0 0) 0x7f09b80d2250 con 0x7f09b80d3ae0 2024-03-24T00:44:49.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.146+0000 7f09af7fe640 1 -- 172.21.15.67:0/3835701185 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09a0003520 con 0x7f09b80d3ae0 2024-03-24T00:44:49.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.146+0000 7f09bd25c640 1 --2- 172.21.15.67:0/3835701185 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f09b80b6170 0x7f09b8168c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:49.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.146+0000 7f09af7fe640 1 -- 172.21.15.67:0/3835701185 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 755771481 0 0) 0x7f09a0003520 con 0x7f09b80d3ae0 2024-03-24T00:44:49.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.146+0000 7f09af7fe640 1 -- 172.21.15.67:0/3835701185 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f09b80b6170 msgr2=0x7f09b8168c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:49.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.146+0000 7f09af7fe640 1 --2- 172.21.15.67:0/3835701185 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f09b80b6170 0x7f09b8168c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:49.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.146+0000 7f09af7fe640 1 -- 172.21.15.67:0/3835701185 >> v1:172.21.15.72:6789/0 conn(0x7f09b815dea0 legacy=0x7f09b80d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:49.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.146+0000 7f09af7fe640 1 -- 172.21.15.67:0/3835701185 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09b80d2af0 con 0x7f09b80d3ae0 2024-03-24T00:44:49.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.146+0000 7f09af7fe640 1 -- 172.21.15.67:0/3835701185 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f09a8001670 con 0x7f09b80d3ae0 2024-03-24T00:44:49.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.147+0000 7f09af7fe640 1 -- 172.21.15.67:0/3835701185 >> v1:172.21.15.67:6789/0 conn(0x7f09b80d3ae0 legacy=0x7f09b80b5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:49.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.147+0000 7f09af7fe640 1 --2- 172.21.15.67:0/3835701185 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f09b80b6170 0x7f09a00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:49.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.147+0000 7f09af7fe640 1 --2- 172.21.15.67:0/3835701185 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09a0003f60 0x7f09a0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:49.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.147+0000 7f09af7fe640 1 --2- 172.21.15.67:0/3835701185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09a0004d40 0x7f09a00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:49.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.147+0000 7f09af7fe640 1 -- 172.21.15.67:0/3835701185 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b80d2810 con 0x7f09a0004d40 2024-03-24T00:44:49.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.147+0000 7f09af7fe640 1 -- 172.21.15.67:0/3835701185 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a000bb50 con 0x7f09a0003f60 2024-03-24T00:44:49.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.147+0000 7f09af7fe640 1 -- 172.21.15.67:0/3835701185 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09a000bd50 con 0x7f09b80b6170 2024-03-24T00:44:49.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.147+0000 7f09bd25c640 1 --2- 172.21.15.67:0/3835701185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09a0004d40 0x7f09a00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:49.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.147+0000 7f09bca5b640 1 --2- 172.21.15.67:0/3835701185 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f09b80b6170 0x7f09a00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:49.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.147+0000 7f09affff640 1 --2- 172.21.15.67:0/3835701185 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09a0003f60 0x7f09a0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:49.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.147+0000 7f09bd25c640 1 -- 172.21.15.67:0/3835701185 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f09b80b6170 msgr2=0x7f09a00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:49.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.147+0000 7f09bd25c640 1 --2- 172.21.15.67:0/3835701185 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f09b80b6170 0x7f09a00039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:49.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.147+0000 7f09bd25c640 1 -- 172.21.15.67:0/3835701185 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09a0003f60 msgr2=0x7f09a0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:49.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.147+0000 7f09bd25c640 1 --2- 172.21.15.67:0/3835701185 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09a0003f60 0x7f09a0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:49.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.147+0000 7f09bd25c640 1 -- 172.21.15.67:0/3835701185 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a0003520 con 0x7f09a0004d40 2024-03-24T00:44:49.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.148+0000 7f09bd25c640 1 --2- 172.21.15.67:0/3835701185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09a0004d40 0x7f09a00051c0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f09b0009fc0 tx=0x7f09a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cca91ce632af6ee7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:49.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.148+0000 7f09af7fe640 1 -- 172.21.15.67:0/3835701185 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b0039070 con 0x7f09a0004d40 2024-03-24T00:44:49.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.148+0000 7f09af7fe640 1 -- 172.21.15.67:0/3835701185 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09b002cc20 con 0x7f09a0004d40 2024-03-24T00:44:49.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.148+0000 7f09af7fe640 1 -- 172.21.15.67:0/3835701185 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b0009c10 con 0x7f09a0004d40 2024-03-24T00:44:49.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.148+0000 7f09becbf640 1 -- 172.21.15.67:0/3835701185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09a0004d40 msgr2=0x7f09a00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:49.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.148+0000 7f09becbf640 1 --2- 172.21.15.67:0/3835701185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09a0004d40 0x7f09a00051c0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f09b0009fc0 tx=0x7f09a0005830 comp rx=0 tx=0).stop 2024-03-24T00:44:49.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.148+0000 7f09bd25c640 1 -- 172.21.15.67:0/3835701185 reap_dead start 2024-03-24T00:44:49.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.148+0000 7f09becbf640 1 -- 172.21.15.67:0/3835701185 shutdown_connections 2024-03-24T00:44:49.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.148+0000 7f09becbf640 1 -- 172.21.15.67:0/3835701185 >> 172.21.15.67:0/3835701185 conn(0x7f09b8072da0 msgr2=0x7f09b80d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:49.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.148+0000 7f09becbf640 1 -- 172.21.15.67:0/3835701185 shutdown_connections 2024-03-24T00:44:49.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.149+0000 7f09becbf640 1 -- 172.21.15.67:0/3835701185 wait complete. 2024-03-24T00:44:49.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.150+0000 7f09becbf640 1 Processor -- start 2024-03-24T00:44:49.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.150+0000 7f09becbf640 1 -- start start 2024-03-24T00:44:49.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.150+0000 7f09becbf640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09b815a650 0x7f09b80b5340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:49.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.150+0000 7f09becbf640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09b80b5880 0x7f09b80abb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:49.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.150+0000 7f09becbf640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f09b80ac0d0 0x7f09b80aa990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:49.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.150+0000 7f09becbf640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b815e280 con 0x7f09b80b5880 2024-03-24T00:44:49.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.150+0000 7f09becbf640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b80b8310 con 0x7f09b815a650 2024-03-24T00:44:49.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.150+0000 7f09becbf640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09b815e400 con 0x7f09b80ac0d0 2024-03-24T00:44:49.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.151+0000 7f09affff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09b80b5880 0x7f09b80abb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:49.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.151+0000 7f09affff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09b80b5880 0x7f09b80abb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56004/0 (socket says 172.21.15.67:56004) 2024-03-24T00:44:49.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.151+0000 7f09affff640 1 -- 172.21.15.67:0/1531959993 learned_addr learned my addr 172.21.15.67:0/1531959993 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:49.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.151+0000 7f09bd25c640 1 --2- 172.21.15.67:0/1531959993 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f09b80ac0d0 0x7f09b80aa990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:49.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.151+0000 7f09bca5b640 1 --2- 172.21.15.67:0/1531959993 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09b815a650 0x7f09b80b5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:49.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.151+0000 7f09affff640 1 -- 172.21.15.67:0/1531959993 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f09b80ac0d0 msgr2=0x7f09b80aa990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:49.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.151+0000 7f09affff640 1 --2- 172.21.15.67:0/1531959993 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f09b80ac0d0 0x7f09b80aa990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:49.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.151+0000 7f09affff640 1 -- 172.21.15.67:0/1531959993 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09b815a650 msgr2=0x7f09b80b5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:49.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.151+0000 7f09affff640 1 --2- 172.21.15.67:0/1531959993 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09b815a650 0x7f09b80b5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:49.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.151+0000 7f09affff640 1 -- 172.21.15.67:0/1531959993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09b80ac550 con 0x7f09b80b5880 2024-03-24T00:44:49.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.151+0000 7f09bca5b640 1 --2- 172.21.15.67:0/1531959993 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09b815a650 0x7f09b80b5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:44:49.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.151+0000 7f09affff640 1 --2- 172.21.15.67:0/1531959993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09b80b5880 0x7f09b80abb90 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f09a80072c0 tx=0x7f09a8007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:49.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.151+0000 7f09ad7fa640 1 -- 172.21.15.67:0/1531959993 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a800fa20 con 0x7f09b80b5880 2024-03-24T00:44:49.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.151+0000 7f09ad7fa640 1 -- 172.21.15.67:0/1531959993 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09a8018070 con 0x7f09b80b5880 2024-03-24T00:44:49.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.151+0000 7f09becbf640 1 -- 172.21.15.67:0/1531959993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09b815aa30 con 0x7f09b80b5880 2024-03-24T00:44:49.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.151+0000 7f09ad7fa640 1 -- 172.21.15.67:0/1531959993 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a800fe20 con 0x7f09b80b5880 2024-03-24T00:44:49.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.151+0000 7f09becbf640 1 -- 172.21.15.67:0/1531959993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09b80cc630 con 0x7f09b80b5880 2024-03-24T00:44:49.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.152+0000 7f09becbf640 1 -- 172.21.15.67:0/1531959993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09b804f400 con 0x7f09b80b5880 2024-03-24T00:44:49.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.153+0000 7f09ad7fa640 1 -- 172.21.15.67:0/1531959993 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f09a801d460 con 0x7f09b80b5880 2024-03-24T00:44:49.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.153+0000 7f09ad7fa640 1 --2- 172.21.15.67:0/1531959993 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f09a406bc00 0x7f09a406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:49.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.153+0000 7f09ad7fa640 1 -- 172.21.15.67:0/1531959993 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f09a808aae0 con 0x7f09b80b5880 2024-03-24T00:44:49.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.153+0000 7f09bca5b640 1 --2- 172.21.15.67:0/1531959993 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f09a406bc00 0x7f09a406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:49.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.154+0000 7f09bca5b640 1 --2- 172.21.15.67:0/1531959993 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f09a406bc00 0x7f09a406e0c0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f09b40036b0 tx=0x7f09b4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:49.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.157+0000 7f09ad7fa640 1 -- 172.21.15.67:0/1531959993 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f09a80596f0 con 0x7f09b80b5880 2024-03-24T00:44:49.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.340+0000 7f09becbf640 1 -- 172.21.15.67:0/1531959993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f09b80b2e40 con 0x7f09b80b5880 2024-03-24T00:44:49.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.343+0000 7f09ad7fa640 1 -- 172.21.15.67:0/1531959993 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v18)=0 v18) v1 ==== 122+0+0 (secure 0 0 0) 0x7f09a8058ff0 con 0x7f09b80b5880 2024-03-24T00:44:49.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.345+0000 7f09becbf640 1 -- 172.21.15.67:0/1531959993 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f09a406bc00 msgr2=0x7f09a406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:49.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.345+0000 7f09becbf640 1 --2- 172.21.15.67:0/1531959993 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f09a406bc00 0x7f09a406e0c0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f09b40036b0 tx=0x7f09b4009040 comp rx=0 tx=0).stop 2024-03-24T00:44:49.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.345+0000 7f09becbf640 1 -- 172.21.15.67:0/1531959993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09b80b5880 msgr2=0x7f09b80abb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:49.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.345+0000 7f09becbf640 1 --2- 172.21.15.67:0/1531959993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09b80b5880 0x7f09b80abb90 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f09a80072c0 tx=0x7f09a8007780 comp rx=0 tx=0).stop 2024-03-24T00:44:49.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.345+0000 7f09becbf640 1 -- 172.21.15.67:0/1531959993 shutdown_connections 2024-03-24T00:44:49.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.345+0000 7f09becbf640 1 --2- 172.21.15.67:0/1531959993 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f09a406bc00 0x7f09a406e0c0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:49.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.345+0000 7f09becbf640 1 --2- 172.21.15.67:0/1531959993 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f09b80ac0d0 0x7f09b80aa990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:49.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.345+0000 7f09becbf640 1 --2- 172.21.15.67:0/1531959993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09b80b5880 0x7f09b80abb90 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:49.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.345+0000 7f09becbf640 1 --2- 172.21.15.67:0/1531959993 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09b815a650 0x7f09b80b5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:49.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.345+0000 7f09becbf640 1 -- 172.21.15.67:0/1531959993 >> 172.21.15.67:0/1531959993 conn(0x7f09b8072da0 msgr2=0x7f09b80b7010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:49.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.345+0000 7f09becbf640 1 -- 172.21.15.67:0/1531959993 shutdown_connections 2024-03-24T00:44:49.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.346+0000 7f09becbf640 1 -- 172.21.15.67:0/1531959993 wait complete. 2024-03-24T00:44:49.357 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-24T00:44:49.357 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:44:49.396 DEBUG:teuthology.orchestra.run.smithi072:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:44:49.443 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-24T00:44:49.443 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:44:49.443 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:44:49.458 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056375,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15771 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T00:44:49.458 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:44:49.458 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:44:49.476 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17590 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248808k,nr_inodes=812202,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:44:49.476 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:44:49.476 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:44:49.513 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056375,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15771 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T00:44:49.513 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:44:49.513 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:44:49.532 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17590 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248808k,nr_inodes=812202,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:44:49.533 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-24T00:44:49.533 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:44:49.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.655+0000 7f0f7d510640 1 Processor -- start 2024-03-24T00:44:49.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.655+0000 7f0f7d510640 1 -- start start 2024-03-24T00:44:49.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.655+0000 7f0f7d510640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f0f7815e0c0 0x7f0f780d21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:49.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.655+0000 7f0f7d510640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0f780b9920 0x7f0f780ceba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:49.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.655+0000 7f0f7d510640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f780b9d80 con 0x7f0f780cf0e0 2024-03-24T00:44:49.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.655+0000 7f0f7d510640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f7815e4a0 con 0x7f0f7815e0c0 2024-03-24T00:44:49.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.655+0000 7f0f7d510640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f780b7f80 con 0x7f0f780b9920 2024-03-24T00:44:49.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.656+0000 7f0f767fc640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0f780b9920 0x7f0f780ceba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:49.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.656+0000 7f0f76ffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f0f7815e0c0 0x7f0f780d21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:49.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.656+0000 7f0f777fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f0f780cf0e0 0x7f0f780b5850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:34038/0 (socket says 172.21.15.67:34038) 2024-03-24T00:44:49.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.656+0000 7f0f777fe640 1 -- 172.21.15.67:0/1325258607 learned_addr learned my addr 172.21.15.67:0/1325258607 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:44:49.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.656+0000 7f0f75ffb640 1 -- 172.21.15.67:0/1325258607 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2231641721 0 0) 0x7f0f780b9d80 con 0x7f0f780cf0e0 2024-03-24T00:44:49.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.656+0000 7f0f75ffb640 1 -- 172.21.15.67:0/1325258607 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f60003520 con 0x7f0f780cf0e0 2024-03-24T00:44:49.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.656+0000 7f0f76ffd640 1 -- 172.21.15.67:0/1325258607 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0f780b9920 msgr2=0x7f0f780ceba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:49.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.656+0000 7f0f76ffd640 1 --2- 172.21.15.67:0/1325258607 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0f780b9920 0x7f0f780ceba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:49.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.656+0000 7f0f76ffd640 1 -- 172.21.15.67:0/1325258607 >> v1:172.21.15.72:6789/0 conn(0x7f0f780cf0e0 legacy=0x7f0f780b5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:49.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.656+0000 7f0f76ffd640 1 -- 172.21.15.67:0/1325258607 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f780d2880 con 0x7f0f7815e0c0 2024-03-24T00:44:49.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.656+0000 7f0f767fc640 1 --2- 172.21.15.67:0/1325258607 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0f780b9920 0x7f0f780ceba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:44:49.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.657+0000 7f0f76ffd640 1 --2- 172.21.15.67:0/1325258607 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0f7815e0c0 0x7f0f780d21a0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f0f680097a0 tx=0x7f0f6802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32d0d1f70bcecdd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:49.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.657+0000 7f0f75ffb640 1 -- 172.21.15.67:0/1325258607 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f68035020 con 0x7f0f7815e0c0 2024-03-24T00:44:49.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.657+0000 7f0f75ffb640 1 -- 172.21.15.67:0/1325258607 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f68031830 con 0x7f0f7815e0c0 2024-03-24T00:44:49.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.657+0000 7f0f75ffb640 1 -- 172.21.15.67:0/1325258607 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f68031be0 con 0x7f0f7815e0c0 2024-03-24T00:44:49.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.657+0000 7f0f7d510640 1 -- 172.21.15.67:0/1325258607 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0f7815e0c0 msgr2=0x7f0f780d21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:49.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.657+0000 7f0f7d510640 1 --2- 172.21.15.67:0/1325258607 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0f7815e0c0 0x7f0f780d21a0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f0f680097a0 tx=0x7f0f6802b8d0 comp rx=0 tx=0).stop 2024-03-24T00:44:49.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.658+0000 7f0f7d510640 1 -- 172.21.15.67:0/1325258607 shutdown_connections 2024-03-24T00:44:49.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.658+0000 7f0f7d510640 1 --2- 172.21.15.67:0/1325258607 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0f780b9920 0x7f0f780ceba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:49.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.658+0000 7f0f7d510640 1 --2- 172.21.15.67:0/1325258607 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0f7815e0c0 0x7f0f780d21a0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:49.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.658+0000 7f0f7d510640 1 -- 172.21.15.67:0/1325258607 >> 172.21.15.67:0/1325258607 conn(0x7f0f7806c830 msgr2=0x7f0f7806cc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:49.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.658+0000 7f0f7d510640 1 -- 172.21.15.67:0/1325258607 shutdown_connections 2024-03-24T00:44:49.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.658+0000 7f0f7d510640 1 -- 172.21.15.67:0/1325258607 wait complete. 2024-03-24T00:44:49.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.659+0000 7f0f7d510640 1 Processor -- start 2024-03-24T00:44:49.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.659+0000 7f0f7d510640 1 -- start start 2024-03-24T00:44:49.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.660+0000 7f0f7d510640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0f780b9920 0x7f0f78155bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:49.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.660+0000 7f0f7d510640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0f780cf0e0 0x7f0f781f3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:49.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.660+0000 7f0f7d510640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f7815e0c0 0x7f0f78148d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:49.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.660+0000 7f0f7d510640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f780d3120 con 0x7f0f7815e0c0 2024-03-24T00:44:49.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.660+0000 7f0f7d510640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f780d2e40 con 0x7f0f780b9920 2024-03-24T00:44:49.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.660+0000 7f0f7d510640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f780d3400 con 0x7f0f780cf0e0 2024-03-24T00:44:49.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.660+0000 7f0f777fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f7815e0c0 0x7f0f78148d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:49.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.660+0000 7f0f777fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f7815e0c0 0x7f0f78148d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56020/0 (socket says 172.21.15.67:56020) 2024-03-24T00:44:49.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.660+0000 7f0f777fe640 1 -- 172.21.15.67:0/430235435 learned_addr learned my addr 172.21.15.67:0/430235435 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:49.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.660+0000 7f0f76ffd640 1 --2- 172.21.15.67:0/430235435 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0f780b9920 0x7f0f78155bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:49.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.660+0000 7f0f767fc640 1 --2- 172.21.15.67:0/430235435 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0f780cf0e0 0x7f0f781f3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:49.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.660+0000 7f0f777fe640 1 -- 172.21.15.67:0/430235435 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0f780cf0e0 msgr2=0x7f0f781f3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:49.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.660+0000 7f0f777fe640 1 --2- 172.21.15.67:0/430235435 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0f780cf0e0 0x7f0f781f3ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:49.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.660+0000 7f0f777fe640 1 -- 172.21.15.67:0/430235435 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0f780b9920 msgr2=0x7f0f78155bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:49.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.660+0000 7f0f777fe640 1 --2- 172.21.15.67:0/430235435 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0f780b9920 0x7f0f78155bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:49.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.660+0000 7f0f777fe640 1 -- 172.21.15.67:0/430235435 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f780cccc0 con 0x7f0f7815e0c0 2024-03-24T00:44:49.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.660+0000 7f0f777fe640 1 --2- 172.21.15.67:0/430235435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f7815e0c0 0x7f0f78148d20 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f0f7000cb40 tx=0x7f0f7000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:49.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.660+0000 7f0f5f7fe640 1 -- 172.21.15.67:0/430235435 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f70017070 con 0x7f0f7815e0c0 2024-03-24T00:44:49.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.660+0000 7f0f5f7fe640 1 -- 172.21.15.67:0/430235435 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f7000aca0 con 0x7f0f7815e0c0 2024-03-24T00:44:49.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.660+0000 7f0f7d510640 1 -- 172.21.15.67:0/430235435 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f780ccfa0 con 0x7f0f7815e0c0 2024-03-24T00:44:49.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.661+0000 7f0f5f7fe640 1 -- 172.21.15.67:0/430235435 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f70012620 con 0x7f0f7815e0c0 2024-03-24T00:44:49.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.661+0000 7f0f7d510640 1 -- 172.21.15.67:0/430235435 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f780cd840 con 0x7f0f7815e0c0 2024-03-24T00:44:49.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.662+0000 7f0f7d510640 1 -- 172.21.15.67:0/430235435 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f780cd390 con 0x7f0f7815e0c0 2024-03-24T00:44:49.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.662+0000 7f0f5f7fe640 1 -- 172.21.15.67:0/430235435 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0f7001a070 con 0x7f0f7815e0c0 2024-03-24T00:44:49.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.662+0000 7f0f5f7fe640 1 --2- 172.21.15.67:0/430235435 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0f6406bbd0 0x7f0f6406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:49.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.663+0000 7f0f5f7fe640 1 -- 172.21.15.67:0/430235435 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f0f7008a6c0 con 0x7f0f7815e0c0 2024-03-24T00:44:49.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.663+0000 7f0f76ffd640 1 --2- 172.21.15.67:0/430235435 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0f6406bbd0 0x7f0f6406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:49.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.663+0000 7f0f76ffd640 1 --2- 172.21.15.67:0/430235435 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0f6406bbd0 0x7f0f6406e090 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f0f68036000 tx=0x7f0f6803c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:49.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.667+0000 7f0f5f7fe640 1 -- 172.21.15.67:0/430235435 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0f70058d10 con 0x7f0f7815e0c0 2024-03-24T00:44:49.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.886+0000 7f0f7d510640 1 -- 172.21.15.67:0/430235435 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f780cda80 con 0x7f0f7815e0c0 2024-03-24T00:44:49.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.887+0000 7f0f5f7fe640 1 -- 172.21.15.67:0/430235435 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+3699 (secure 0 0 0) 0x7f0f780cda80 con 0x7f0f7815e0c0 2024-03-24T00:44:49.888 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:44:49.888 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":50,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5960,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1888378520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1888378520},{"type":"v1","addr":"172.21.15.67:6835","nonce":1888378520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6610,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/2539718386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2539718386},{"type":"v1","addr":"172.21.15.72:6836","nonce":2539718386}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6613,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2376370770","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2376370770},{"type":"v1","addr":"172.21.15.72:6837","nonce":2376370770}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":7545,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/1475917855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1475917855},{"type":"v1","addr":"172.21.15.67:6833","nonce":1475917855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[]} 2024-03-24T00:44:49.888 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 50 2024-03-24T00:44:49.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.889+0000 7f0f7d510640 1 -- 172.21.15.67:0/430235435 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0f6406bbd0 msgr2=0x7f0f6406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:49.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.889+0000 7f0f7d510640 1 --2- 172.21.15.67:0/430235435 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0f6406bbd0 0x7f0f6406e090 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f0f68036000 tx=0x7f0f6803c040 comp rx=0 tx=0).stop 2024-03-24T00:44:49.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.889+0000 7f0f7d510640 1 -- 172.21.15.67:0/430235435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f7815e0c0 msgr2=0x7f0f78148d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:49.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.889+0000 7f0f7d510640 1 --2- 172.21.15.67:0/430235435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f7815e0c0 0x7f0f78148d20 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f0f7000cb40 tx=0x7f0f7000a060 comp rx=0 tx=0).stop 2024-03-24T00:44:49.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.889+0000 7f0f7d510640 1 -- 172.21.15.67:0/430235435 shutdown_connections 2024-03-24T00:44:49.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.889+0000 7f0f7d510640 1 --2- 172.21.15.67:0/430235435 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0f6406bbd0 0x7f0f6406e090 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:49.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.889+0000 7f0f7d510640 1 --2- 172.21.15.67:0/430235435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f7815e0c0 0x7f0f78148d20 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:49.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.889+0000 7f0f7d510640 1 --2- 172.21.15.67:0/430235435 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0f780cf0e0 0x7f0f781f3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:49.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.889+0000 7f0f7d510640 1 --2- 172.21.15.67:0/430235435 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0f780b9920 0x7f0f78155bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:49.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.889+0000 7f0f7d510640 1 -- 172.21.15.67:0/430235435 >> 172.21.15.67:0/430235435 conn(0x7f0f7806c830 msgr2=0x7f0f78160e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:49.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.889+0000 7f0f7d510640 1 -- 172.21.15.67:0/430235435 shutdown_connections 2024-03-24T00:44:49.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:49.889+0000 7f0f7d510640 1 -- 172.21.15.67:0/430235435 wait complete. 2024-03-24T00:44:49.901 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-24T00:44:49.901 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-24T00:44:49.901 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-24T00:44:49.901 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:44:49.901 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-24T00:44:49.901 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-24T00:44:49.902 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-24T00:44:49.902 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:44:49.902 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-24T00:44:49.902 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-24T00:44:49.902 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-24T00:44:49.902 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:44:49.902 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-24T00:44:49.902 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-24T00:44:49.903 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-24T00:44:49.903 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:44:49.910 INFO:tasks.ceph.mds.a:Stopped 2024-03-24T00:44:49.910 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-24T00:44:49.911 INFO:tasks.ceph.mds.c:Stopped 2024-03-24T00:44:49.911 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-24T00:44:49.914 INFO:tasks.ceph.mds.d:Stopped 2024-03-24T00:44:49.914 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-24T00:44:49.915 INFO:tasks.ceph.mds.b:Stopped 2024-03-24T00:44:49.915 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-24T00:44:49.953 INFO:tasks.ceph.mds.a:Started 2024-03-24T00:44:49.953 DEBUG:teuthology.parallel:result is None 2024-03-24T00:44:49.954 INFO:tasks.ceph.mds.c:Started 2024-03-24T00:44:49.954 DEBUG:teuthology.parallel:result is None 2024-03-24T00:44:49.957 INFO:tasks.ceph.mds.d:Started 2024-03-24T00:44:49.958 DEBUG:teuthology.parallel:result is None 2024-03-24T00:44:49.958 INFO:tasks.ceph.mds.b:Started 2024-03-24T00:44:49.958 DEBUG:teuthology.parallel:result is None 2024-03-24T00:44:49.959 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-24T00:44:50.030 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-03-24T00:44:50.030 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-03-24T00:44:50.040 INFO:tasks.ceph.mds.d.smithi072.stdout:starting mds.d at 2024-03-24T00:44:50.040 INFO:tasks.ceph.mds.b.smithi072.stdout:starting mds.b at 2024-03-24T00:44:50.041 INFO:tasks.ceph.mds.d.smithi072.stderr:2024-03-24T00:44:50.040+0000 7f51c46df640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2024-03-24T00:44:50.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.065+0000 7f4e6fc27640 1 Processor -- start 2024-03-24T00:44:50.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.065+0000 7f4e6fc27640 1 -- start start 2024-03-24T00:44:50.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.066+0000 7f4e6fc27640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4e680d32d0 0x7f4e680d3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:50.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.066+0000 7f4e6fc27640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4e680b6dc0 0x7f4e680b7210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:50.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.066+0000 7f4e6fc27640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e680b9cc0 con 0x7f4e6815e0f0 2024-03-24T00:44:50.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.066+0000 7f4e6fc27640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e680b7750 con 0x7f4e680d32d0 2024-03-24T00:44:50.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.066+0000 7f4e6fc27640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e680d2230 con 0x7f4e680b6dc0 2024-03-24T00:44:50.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.066+0000 7f4e6d1c2640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4e680d32d0 0x7f4e680d3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:50.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.066+0000 7f4e6d9c3640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4e6815e0f0 0x7f4e680b95b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52670/0 (socket says 172.21.15.67:52670) 2024-03-24T00:44:50.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.066+0000 7f4e6d9c3640 1 -- 172.21.15.67:0/350848982 learned_addr learned my addr 172.21.15.67:0/350848982 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:44:50.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.067+0000 7f4e6e1c4640 1 --2- 172.21.15.67:0/350848982 >> v2:172.21.15.72:3300/0 conn(0x7f4e680b6dc0 0x7f4e680b7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:50.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.067+0000 7f4e6d1c2640 1 -- 172.21.15.67:0/350848982 >> v2:172.21.15.72:3300/0 conn(0x7f4e680b6dc0 msgr2=0x7f4e680b7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:50.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.067+0000 7f4e6d1c2640 1 --2- 172.21.15.67:0/350848982 >> v2:172.21.15.72:3300/0 conn(0x7f4e680b6dc0 0x7f4e680b7210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:50.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.067+0000 7f4e6d1c2640 1 -- 172.21.15.67:0/350848982 >> v1:172.21.15.67:6789/0 conn(0x7f4e6815e0f0 legacy=0x7f4e680b95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:50.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.067+0000 7f4e6d1c2640 1 -- 172.21.15.67:0/350848982 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e680d2510 con 0x7f4e680d32d0 2024-03-24T00:44:50.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.067+0000 7f4e6c9c1640 1 -- 172.21.15.67:0/350848982 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4146995732 0 0) 0x7f4e680b9cc0 con 0x7f4e6815e0f0 2024-03-24T00:44:50.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.067+0000 7f4e6e1c4640 1 --2- 172.21.15.67:0/350848982 >> v2:172.21.15.72:3300/0 conn(0x7f4e680b6dc0 0x7f4e680b7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:44:50.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.067+0000 7f4e6d1c2640 1 --2- 172.21.15.67:0/350848982 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e680d32d0 0x7f4e680d3740 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f4e6808d0b0 tx=0x7f4e5802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be6987310ab23951 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:50.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.068+0000 7f4e6c9c1640 1 -- 172.21.15.67:0/350848982 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e58035020 con 0x7f4e680d32d0 2024-03-24T00:44:50.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.068+0000 7f4e6c9c1640 1 -- 172.21.15.67:0/350848982 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e58031830 con 0x7f4e680d32d0 2024-03-24T00:44:50.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.068+0000 7f4e6c9c1640 1 -- 172.21.15.67:0/350848982 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e58031c00 con 0x7f4e680d32d0 2024-03-24T00:44:50.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.068+0000 7f4e6fc27640 1 -- 172.21.15.67:0/350848982 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e680d32d0 msgr2=0x7f4e680d3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:50.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.068+0000 7f4e6fc27640 1 --2- 172.21.15.67:0/350848982 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e680d32d0 0x7f4e680d3740 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f4e6808d0b0 tx=0x7f4e5802b8d0 comp rx=0 tx=0).stop 2024-03-24T00:44:50.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.068+0000 7f4e6fc27640 1 -- 172.21.15.67:0/350848982 shutdown_connections 2024-03-24T00:44:50.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.068+0000 7f4e6fc27640 1 --2- 172.21.15.67:0/350848982 >> v2:172.21.15.72:3300/0 conn(0x7f4e680b6dc0 0x7f4e680b7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:50.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.068+0000 7f4e6fc27640 1 --2- 172.21.15.67:0/350848982 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e680d32d0 0x7f4e680d3740 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:50.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.068+0000 7f4e6fc27640 1 -- 172.21.15.67:0/350848982 >> 172.21.15.67:0/350848982 conn(0x7f4e68072b60 msgr2=0x7f4e68072f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:50.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.068+0000 7f4e6fc27640 1 -- 172.21.15.67:0/350848982 shutdown_connections 2024-03-24T00:44:50.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.068+0000 7f4e6fc27640 1 -- 172.21.15.67:0/350848982 wait complete. 2024-03-24T00:44:50.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.069+0000 7f4e6fc27640 1 Processor -- start 2024-03-24T00:44:50.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.069+0000 7f4e6fc27640 1 -- start start 2024-03-24T00:44:50.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.070+0000 7f4e6fc27640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e680b6dc0 0x7f4e681521d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:50.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.070+0000 7f4e6fc27640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e680d32d0 0x7f4e680b8650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:50.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.070+0000 7f4e6fc27640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4e6815e0f0 0x7f4e680ccd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:50.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.070+0000 7f4e6fc27640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e680d2db0 con 0x7f4e680b6dc0 2024-03-24T00:44:50.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.070+0000 7f4e6fc27640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e680d2ad0 con 0x7f4e680d32d0 2024-03-24T00:44:50.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.070+0000 7f4e6fc27640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e680b31d0 con 0x7f4e6815e0f0 2024-03-24T00:44:50.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.070+0000 7f4e6d9c3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e680b6dc0 0x7f4e681521d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:50.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.070+0000 7f4e6d9c3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e680b6dc0 0x7f4e681521d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56068/0 (socket says 172.21.15.67:56068) 2024-03-24T00:44:50.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.070+0000 7f4e6d9c3640 1 -- 172.21.15.67:0/3381639862 learned_addr learned my addr 172.21.15.67:0/3381639862 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:50.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.070+0000 7f4e6d9c3640 1 -- 172.21.15.67:0/3381639862 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4e6815e0f0 msgr2=0x7f4e680ccd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:50.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.070+0000 7f4e6e1c4640 1 --2- 172.21.15.67:0/3381639862 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4e6815e0f0 0x7f4e680ccd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:50.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.070+0000 7f4e6d1c2640 1 --2- 172.21.15.67:0/3381639862 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e680d32d0 0x7f4e680b8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:50.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.070+0000 7f4e6d9c3640 1 --2- 172.21.15.67:0/3381639862 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4e6815e0f0 0x7f4e680ccd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:50.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.070+0000 7f4e6d9c3640 1 -- 172.21.15.67:0/3381639862 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e680d32d0 msgr2=0x7f4e680b8650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:50.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.070+0000 7f4e6d9c3640 1 --2- 172.21.15.67:0/3381639862 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e680d32d0 0x7f4e680b8650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:50.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.070+0000 7f4e6d9c3640 1 -- 172.21.15.67:0/3381639862 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e681600d0 con 0x7f4e680b6dc0 2024-03-24T00:44:50.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.070+0000 7f4e6e1c4640 1 --2- 172.21.15.67:0/3381639862 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4e6815e0f0 0x7f4e680ccd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:44:50.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.070+0000 7f4e6d1c2640 1 --2- 172.21.15.67:0/3381639862 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e680d32d0 0x7f4e680b8650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:44:50.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.071+0000 7f4e6d9c3640 1 --2- 172.21.15.67:0/3381639862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e680b6dc0 0x7f4e681521d0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f4e6400c4d0 tx=0x7f4e6400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:50.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.071+0000 7f4e5e7fc640 1 -- 172.21.15.67:0/3381639862 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e64018070 con 0x7f4e680b6dc0 2024-03-24T00:44:50.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.071+0000 7f4e5e7fc640 1 -- 172.21.15.67:0/3381639862 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e64007500 con 0x7f4e680b6dc0 2024-03-24T00:44:50.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.071+0000 7f4e6fc27640 1 -- 172.21.15.67:0/3381639862 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e681603b0 con 0x7f4e680b6dc0 2024-03-24T00:44:50.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.071+0000 7f4e5e7fc640 1 -- 172.21.15.67:0/3381639862 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e64004a20 con 0x7f4e680b6dc0 2024-03-24T00:44:50.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.071+0000 7f4e6fc27640 1 -- 172.21.15.67:0/3381639862 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e68160970 con 0x7f4e680b6dc0 2024-03-24T00:44:50.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.072+0000 7f4e6fc27640 1 -- 172.21.15.67:0/3381639862 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e6804f400 con 0x7f4e680b6dc0 2024-03-24T00:44:50.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.072+0000 7f4e5e7fc640 1 -- 172.21.15.67:0/3381639862 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4e6401b070 con 0x7f4e680b6dc0 2024-03-24T00:44:50.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.073+0000 7f4e5e7fc640 1 --2- 172.21.15.67:0/3381639862 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4e4c06bbd0 0x7f4e4c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:50.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.073+0000 7f4e5e7fc640 1 -- 172.21.15.67:0/3381639862 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f4e64089bf0 con 0x7f4e680b6dc0 2024-03-24T00:44:50.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.073+0000 7f4e6d1c2640 1 --2- 172.21.15.67:0/3381639862 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4e4c06bbd0 0x7f4e4c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:50.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.073+0000 7f4e6d1c2640 1 --2- 172.21.15.67:0/3381639862 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4e4c06bbd0 0x7f4e4c06e090 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f4e58036000 tx=0x7f4e58008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:50.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.077+0000 7f4e5e7fc640 1 -- 172.21.15.67:0/3381639862 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4e64058890 con 0x7f4e680b6dc0 2024-03-24T00:44:50.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:50.277+0000 7f4e6fc27640 1 -- 172.21.15.67:0/3381639862 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f4e680b6100 con 0x7f4e680b6dc0 2024-03-24T00:44:51.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.367+0000 7f4e5e7fc640 1 -- 172.21.15.67:0/3381639862 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v85) v1 ==== 97+0+0 (secure 0 0 0) 0x7f4e64058190 con 0x7f4e680b6dc0 2024-03-24T00:44:51.367 INFO:teuthology.orchestra.run.smithi067.stderr: removed all blocklist entries 2024-03-24T00:44:51.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.370+0000 7f4e6fc27640 1 -- 172.21.15.67:0/3381639862 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4e4c06bbd0 msgr2=0x7f4e4c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:51.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.370+0000 7f4e6fc27640 1 --2- 172.21.15.67:0/3381639862 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4e4c06bbd0 0x7f4e4c06e090 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f4e58036000 tx=0x7f4e58008030 comp rx=0 tx=0).stop 2024-03-24T00:44:51.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.370+0000 7f4e6fc27640 1 -- 172.21.15.67:0/3381639862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e680b6dc0 msgr2=0x7f4e681521d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:51.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.370+0000 7f4e6fc27640 1 --2- 172.21.15.67:0/3381639862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e680b6dc0 0x7f4e681521d0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f4e6400c4d0 tx=0x7f4e6400c990 comp rx=0 tx=0).stop 2024-03-24T00:44:51.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.371+0000 7f4e6fc27640 1 -- 172.21.15.67:0/3381639862 shutdown_connections 2024-03-24T00:44:51.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.371+0000 7f4e6fc27640 1 --2- 172.21.15.67:0/3381639862 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4e4c06bbd0 0x7f4e4c06e090 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.371+0000 7f4e6fc27640 1 --2- 172.21.15.67:0/3381639862 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4e6815e0f0 0x7f4e680ccd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.371+0000 7f4e6fc27640 1 --2- 172.21.15.67:0/3381639862 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e680d32d0 0x7f4e680b8650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.371+0000 7f4e6fc27640 1 --2- 172.21.15.67:0/3381639862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e680b6dc0 0x7f4e681521d0 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.371+0000 7f4e6fc27640 1 -- 172.21.15.67:0/3381639862 >> 172.21.15.67:0/3381639862 conn(0x7f4e68072b60 msgr2=0x7f4e680d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:51.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.371+0000 7f4e6fc27640 1 -- 172.21.15.67:0/3381639862 shutdown_connections 2024-03-24T00:44:51.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.371+0000 7f4e6fc27640 1 -- 172.21.15.67:0/3381639862 wait complete. 2024-03-24T00:44:51.384 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-24T00:44:51.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.493+0000 7fe9446f1640 1 Processor -- start 2024-03-24T00:44:51.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.493+0000 7fe9446f1640 1 -- start start 2024-03-24T00:44:51.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.494+0000 7fe9446f1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe93c15f410 0x7fe93c0cf7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:51.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.494+0000 7fe9446f1640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe93c099d70 0x7fe93c0b19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:51.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.494+0000 7fe9446f1640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe93c0cdb50 con 0x7fe93c0d0940 2024-03-24T00:44:51.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.494+0000 7fe9446f1640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe93c0cd2b0 con 0x7fe93c099d70 2024-03-24T00:44:51.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.494+0000 7fe9446f1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c0cd590 con 0x7fe93c15f410 2024-03-24T00:44:51.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.494+0000 7fe942c8e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe93c099d70 0x7fe93c0b19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:51.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.494+0000 7fe942c8e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe93c099d70 0x7fe93c0b19f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56070/0 (socket says 172.21.15.67:56070) 2024-03-24T00:44:51.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.494+0000 7fe942c8e640 1 -- 172.21.15.67:0/2708214051 learned_addr learned my addr 172.21.15.67:0/2708214051 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:51.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.494+0000 7fe94248d640 1 --2- 172.21.15.67:0/2708214051 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe93c15f410 0x7fe93c0cf7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:51.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.494+0000 7fe942c8e640 1 -- 172.21.15.67:0/2708214051 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe93c15f410 msgr2=0x7fe93c0cf7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:51.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.494+0000 7fe942c8e640 1 --2- 172.21.15.67:0/2708214051 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe93c15f410 0x7fe93c0cf7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.494+0000 7fe942c8e640 1 -- 172.21.15.67:0/2708214051 >> v1:172.21.15.72:6789/0 conn(0x7fe93c0d0940 legacy=0x7fe93c0cfd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:51.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.494+0000 7fe942c8e640 1 -- 172.21.15.67:0/2708214051 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe93c0cd870 con 0x7fe93c099d70 2024-03-24T00:44:51.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.495+0000 7fe942c8e640 1 --2- 172.21.15.67:0/2708214051 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe93c099d70 0x7fe93c0b19f0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fe93c0b34a0 tx=0x7fe93402be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2d6d32020252be2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:51.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.495+0000 7fe94148b640 1 -- 172.21.15.67:0/2708214051 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe93402ca30 con 0x7fe93c099d70 2024-03-24T00:44:51.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.495+0000 7fe94148b640 1 -- 172.21.15.67:0/2708214051 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe93402cbd0 con 0x7fe93c099d70 2024-03-24T00:44:51.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.495+0000 7fe94148b640 1 -- 172.21.15.67:0/2708214051 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9340343f0 con 0x7fe93c099d70 2024-03-24T00:44:51.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.496+0000 7fe9446f1640 1 -- 172.21.15.67:0/2708214051 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe93c099d70 msgr2=0x7fe93c0b19f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:51.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.496+0000 7fe9446f1640 1 --2- 172.21.15.67:0/2708214051 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe93c099d70 0x7fe93c0b19f0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fe93c0b34a0 tx=0x7fe93402be90 comp rx=0 tx=0).stop 2024-03-24T00:44:51.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.496+0000 7fe9446f1640 1 -- 172.21.15.67:0/2708214051 shutdown_connections 2024-03-24T00:44:51.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.496+0000 7fe9446f1640 1 --2- 172.21.15.67:0/2708214051 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe93c099d70 0x7fe93c0b19f0 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.496+0000 7fe9446f1640 1 --2- 172.21.15.67:0/2708214051 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe93c15f410 0x7fe93c0cf7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.496+0000 7fe9446f1640 1 -- 172.21.15.67:0/2708214051 >> 172.21.15.67:0/2708214051 conn(0x7fe93c0b36e0 msgr2=0x7fe93c0b3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:51.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.496+0000 7fe9446f1640 1 -- 172.21.15.67:0/2708214051 shutdown_connections 2024-03-24T00:44:51.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.496+0000 7fe9446f1640 1 -- 172.21.15.67:0/2708214051 wait complete. 2024-03-24T00:44:51.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.497+0000 7fe9446f1640 1 Processor -- start 2024-03-24T00:44:51.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.497+0000 7fe9446f1640 1 -- start start 2024-03-24T00:44:51.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.498+0000 7fe9446f1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe93c099d70 0x7fe93c14e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:51.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.498+0000 7fe9446f1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe93c15f410 0x7fe93c14e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:51.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.498+0000 7fe9446f1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe93c151500 0x7fe93c0cb370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:51.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.498+0000 7fe9446f1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c15f7f0 con 0x7fe93c151500 2024-03-24T00:44:51.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.498+0000 7fe9446f1640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c0d0410 con 0x7fe93c099d70 2024-03-24T00:44:51.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.498+0000 7fe9446f1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c0aee90 con 0x7fe93c15f410 2024-03-24T00:44:51.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.498+0000 7fe942c8e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe93c151500 0x7fe93c0cb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:51.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.498+0000 7fe942c8e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe93c151500 0x7fe93c0cb370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56086/0 (socket says 172.21.15.67:56086) 2024-03-24T00:44:51.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.498+0000 7fe942c8e640 1 -- 172.21.15.67:0/2196699525 learned_addr learned my addr 172.21.15.67:0/2196699525 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:51.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.498+0000 7fe941c8c640 1 --2- 172.21.15.67:0/2196699525 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe93c15f410 0x7fe93c14e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:51.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.498+0000 7fe94248d640 1 --2- 172.21.15.67:0/2196699525 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe93c099d70 0x7fe93c14e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:51.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.498+0000 7fe942c8e640 1 -- 172.21.15.67:0/2196699525 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe93c15f410 msgr2=0x7fe93c14e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:51.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.498+0000 7fe942c8e640 1 --2- 172.21.15.67:0/2196699525 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe93c15f410 0x7fe93c14e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.498+0000 7fe942c8e640 1 -- 172.21.15.67:0/2196699525 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe93c099d70 msgr2=0x7fe93c14e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:51.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.498+0000 7fe942c8e640 1 --2- 172.21.15.67:0/2196699525 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe93c099d70 0x7fe93c14e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.498+0000 7fe942c8e640 1 -- 172.21.15.67:0/2196699525 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe93c0af170 con 0x7fe93c151500 2024-03-24T00:44:51.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.498+0000 7fe94248d640 1 --2- 172.21.15.67:0/2196699525 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe93c099d70 0x7fe93c14e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:44:51.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.498+0000 7fe942c8e640 1 --2- 172.21.15.67:0/2196699525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe93c151500 0x7fe93c0cb370 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fe934036040 tx=0x7fe934004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:51.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.498+0000 7fe932ffd640 1 -- 172.21.15.67:0/2196699525 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe934043020 con 0x7fe93c151500 2024-03-24T00:44:51.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.498+0000 7fe932ffd640 1 -- 172.21.15.67:0/2196699525 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe93403e450 con 0x7fe93c151500 2024-03-24T00:44:51.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.498+0000 7fe9446f1640 1 -- 172.21.15.67:0/2196699525 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe93c0af450 con 0x7fe93c151500 2024-03-24T00:44:51.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.498+0000 7fe932ffd640 1 -- 172.21.15.67:0/2196699525 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe93403e860 con 0x7fe93c151500 2024-03-24T00:44:51.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.499+0000 7fe9446f1640 1 -- 172.21.15.67:0/2196699525 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe93c0afa10 con 0x7fe93c151500 2024-03-24T00:44:51.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.500+0000 7fe9446f1640 1 -- 172.21.15.67:0/2196699525 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe93c0afc10 con 0x7fe93c151500 2024-03-24T00:44:51.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.500+0000 7fe932ffd640 1 -- 172.21.15.67:0/2196699525 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe934046020 con 0x7fe93c151500 2024-03-24T00:44:51.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.500+0000 7fe932ffd640 1 --2- 172.21.15.67:0/2196699525 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe92006bbd0 0x7fe92006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:51.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.500+0000 7fe932ffd640 1 -- 172.21.15.67:0/2196699525 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fe9340ad100 con 0x7fe93c151500 2024-03-24T00:44:51.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.500+0000 7fe94248d640 1 --2- 172.21.15.67:0/2196699525 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe92006bbd0 0x7fe92006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:51.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.501+0000 7fe94248d640 1 --2- 172.21.15.67:0/2196699525 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe92006bbd0 0x7fe92006e090 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fe93800a9d0 tx=0x7fe938008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:51.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.505+0000 7fe932ffd640 1 -- 172.21.15.67:0/2196699525 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe93c0afc10 con 0x7fe93c151500 2024-03-24T00:44:51.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.755+0000 7fe9446f1640 1 -- 172.21.15.67:0/2196699525 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fe93c04f400 con 0x7fe93c151500 2024-03-24T00:44:51.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.755+0000 7fe932ffd640 1 -- 172.21.15.67:0/2196699525 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fe93404d030 con 0x7fe93c151500 2024-03-24T00:44:51.755 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:44:51.756 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:44:51.756 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_dump": [ 2024-03-24T00:44:51.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:44:51.756 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.a", 2024-03-24T00:44:51.756 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9l4UENDhAASKUDtqcbSQ7wADj7uo4eyQ==", 2024-03-24T00:44:51.756 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:44:51.756 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:44:51.756 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-03-24T00:44:51.756 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-03-24T00:44:51.756 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:44:51.756 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:44:51.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:44:51.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:44:51.757 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.b", 2024-03-24T00:44:51.757 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9luhJWGhAAsS6PFHV2XdF9htjIjzdrdA==", 2024-03-24T00:44:51.757 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:44:51.757 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:44:51.757 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-03-24T00:44:51.757 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-03-24T00:44:51.757 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:44:51.757 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:44:51.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:44:51.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:44:51.757 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.c", 2024-03-24T00:44:51.757 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9l8mRjFRAAQ73yexNddpcyp8ayTcRKCg==", 2024-03-24T00:44:51.757 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:44:51.758 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:44:51.758 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-03-24T00:44:51.758 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-03-24T00:44:51.758 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:44:51.758 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:44:51.758 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:44:51.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:44:51.758 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.d", 2024-03-24T00:44:51.758 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9lcJyvIRAA2tpDb6qxn1MtljjEno4qPA==", 2024-03-24T00:44:51.758 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:44:51.758 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:44:51.758 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-03-24T00:44:51.758 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-03-24T00:44:51.758 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:44:51.759 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:44:51.759 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:44:51.759 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:44:51.759 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.0", 2024-03-24T00:44:51.759 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDgdf9lV/NUBRAAyJNajFAnNABJErnBlB8TEA==", 2024-03-24T00:44:51.759 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:44:51.759 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:44:51.759 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:44:51.759 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:44:51.759 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:44:51.759 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:44:51.759 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:44:51.759 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.1", 2024-03-24T00:44:51.760 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDjdf9l0J6lAxAA9RInt/X22jqzthE0vOiiFg==", 2024-03-24T00:44:51.760 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:44:51.760 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:44:51.760 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:44:51.760 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:44:51.760 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:44:51.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:44:51.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:44:51.760 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.2", 2024-03-24T00:44:51.760 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDmdf9lSmrEARAAwmtSQRNePsn4WIZODnjhfQ==", 2024-03-24T00:44:51.760 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:44:51.760 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:44:51.760 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:44:51.760 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:44:51.761 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:44:51.761 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:44:51.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:44:51.761 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.3", 2024-03-24T00:44:51.761 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDpdf9lm59bABAAgJ5AV774lw2BgkSCgaweyA==", 2024-03-24T00:44:51.761 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:44:51.761 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:44:51.761 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:44:51.761 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:44:51.761 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:44:51.761 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:44:51.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:44:51.761 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.4", 2024-03-24T00:44:51.761 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDvdf9lCo+pCRAANHkL0pPxxok6IsBCLAFCbg==", 2024-03-24T00:44:51.761 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:44:51.762 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:44:51.762 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:44:51.762 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:44:51.762 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:44:51.762 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:44:51.762 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:44:51.762 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.5", 2024-03-24T00:44:51.762 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDydf9lyGEdCBAAcuszPsthGlLvVryE7+D9iQ==", 2024-03-24T00:44:51.762 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:44:51.762 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:44:51.762 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:44:51.762 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:44:51.762 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:44:51.762 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:44:51.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:44:51.763 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.6", 2024-03-24T00:44:51.763 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQD1df9ljuZqBhAACt2OPNkjKOD3rK7g2mVBBA==", 2024-03-24T00:44:51.763 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:44:51.763 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:44:51.763 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:44:51.763 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:44:51.763 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:44:51.763 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:44:51.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:44:51.763 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.7", 2024-03-24T00:44:51.763 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQD4df9lYKBfBRAAy/cWFF6pvGDjXZHAuBYw7g==", 2024-03-24T00:44:51.763 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:44:51.763 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:44:51.764 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:44:51.764 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:44:51.764 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:44:51.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:44:51.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:44:51.764 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.0", 2024-03-24T00:44:51.764 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9lY4nvJRAA6HWOZDWA+SSvJVmyjG+51g==", 2024-03-24T00:44:51.764 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:44:51.764 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:44:51.764 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow r", 2024-03-24T00:44:51.764 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-24T00:44:51.764 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:44:51.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:44:51.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:44:51.764 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.1", 2024-03-24T00:44:51.765 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9lonsXLxAAaltVMW3n0qHX8u9X8EkSmg==", 2024-03-24T00:44:51.765 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:44:51.765 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:44:51.765 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-03-24T00:44:51.765 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-03-24T00:44:51.765 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-03-24T00:44:51.765 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:44:51.765 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:44:51.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:44:51.765 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.2", 2024-03-24T00:44:51.765 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9lLEm2KxAAkT65mhLrTirlDnOUyvBY7g==", 2024-03-24T00:44:51.765 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:44:51.765 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:44:51.765 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-03-24T00:44:51.765 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-03-24T00:44:51.766 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-03-24T00:44:51.766 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:44:51.766 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:44:51.766 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:44:51.766 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.3", 2024-03-24T00:44:51.766 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9l8njZNBAAaAktZpX2SPcJR9ymPm9ebw==", 2024-03-24T00:44:51.766 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:44:51.766 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:44:51.766 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-03-24T00:44:51.766 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-03-24T00:44:51.766 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-03-24T00:44:51.766 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:44:51.766 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:44:51.767 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:44:51.767 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.admin", 2024-03-24T00:44:51.767 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDbdf9lDTIwLRAAMajqhM/KfUNF/SjVv9OJDw==", 2024-03-24T00:44:51.767 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:44:51.767 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-03-24T00:44:51.767 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-03-24T00:44:51.767 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-03-24T00:44:51.767 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:44:51.767 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:44:51.767 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:44:51.767 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:44:51.767 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.x", 2024-03-24T00:44:51.767 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9lGwKvChAAxl+fRnjbAQ7dYJ727QFUgw==", 2024-03-24T00:44:51.767 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:44:51.768 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-03-24T00:44:51.768 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-03-24T00:44:51.768 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:44:51.768 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:44:51.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:44:51.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:44:51.768 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.y", 2024-03-24T00:44:51.768 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9lSvufBhAAoQ9L1GjLkstjrAw2lqlryA==", 2024-03-24T00:44:51.768 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:44:51.768 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-03-24T00:44:51.768 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-03-24T00:44:51.768 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:44:51.768 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:44:51.768 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:44:51.768 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-24T00:44:51.768 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:44:51.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.757+0000 7fe9446f1640 1 -- 172.21.15.67:0/2196699525 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe92006bbd0 msgr2=0x7fe92006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:51.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.757+0000 7fe9446f1640 1 --2- 172.21.15.67:0/2196699525 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe92006bbd0 0x7fe92006e090 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fe93800a9d0 tx=0x7fe938008040 comp rx=0 tx=0).stop 2024-03-24T00:44:51.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.757+0000 7fe9446f1640 1 -- 172.21.15.67:0/2196699525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe93c151500 msgr2=0x7fe93c0cb370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:51.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.757+0000 7fe9446f1640 1 --2- 172.21.15.67:0/2196699525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe93c151500 0x7fe93c0cb370 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fe934036040 tx=0x7fe934004a40 comp rx=0 tx=0).stop 2024-03-24T00:44:51.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.757+0000 7fe9446f1640 1 -- 172.21.15.67:0/2196699525 shutdown_connections 2024-03-24T00:44:51.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.757+0000 7fe9446f1640 1 --2- 172.21.15.67:0/2196699525 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe92006bbd0 0x7fe92006e090 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.757+0000 7fe9446f1640 1 --2- 172.21.15.67:0/2196699525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe93c151500 0x7fe93c0cb370 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.757+0000 7fe9446f1640 1 --2- 172.21.15.67:0/2196699525 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe93c15f410 0x7fe93c14e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.757+0000 7fe9446f1640 1 --2- 172.21.15.67:0/2196699525 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe93c099d70 0x7fe93c14e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.757+0000 7fe9446f1640 1 -- 172.21.15.67:0/2196699525 >> 172.21.15.67:0/2196699525 conn(0x7fe93c0b36e0 msgr2=0x7fe93c0b1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:51.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.758+0000 7fe9446f1640 1 -- 172.21.15.67:0/2196699525 shutdown_connections 2024-03-24T00:44:51.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.758+0000 7fe9446f1640 1 -- 172.21.15.67:0/2196699525 wait complete. 2024-03-24T00:44:51.771 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:44:51.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.873+0000 7fe1aedb9640 1 Processor -- start 2024-03-24T00:44:51.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.873+0000 7fe1aedb9640 1 -- start start 2024-03-24T00:44:51.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.874+0000 7fe1aedb9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe1a806f350 0x7fe1a80cee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:51.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.874+0000 7fe1aedb9640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1a80b4d40 con 0x7fe1a80b48d0 2024-03-24T00:44:51.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.874+0000 7fe1aedb9640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1a80cddc0 con 0x7fe1a81445a0 2024-03-24T00:44:51.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.874+0000 7fe1aedb9640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1a80ce990 con 0x7fe1a806f350 2024-03-24T00:44:51.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.874+0000 7fe1a7fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe1a806f350 0x7fe1a80cee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:51.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.874+0000 7fe1ad356640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe1a80b48d0 0x7fe1a80b2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52686/0 (socket says 172.21.15.67:52686) 2024-03-24T00:44:51.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.874+0000 7fe1ad356640 1 -- 172.21.15.67:0/3428688945 learned_addr learned my addr 172.21.15.67:0/3428688945 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:44:51.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.874+0000 7fe1a7fff640 1 -- 172.21.15.67:0/3428688945 >> v1:172.21.15.72:6789/0 conn(0x7fe1a81445a0 legacy=0x7fe1a806ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:51.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.874+0000 7fe1a7fff640 1 -- 172.21.15.67:0/3428688945 >> v1:172.21.15.67:6789/0 conn(0x7fe1a80b48d0 legacy=0x7fe1a80b2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:51.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.874+0000 7fe1a7fff640 1 -- 172.21.15.67:0/3428688945 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1a80ce060 con 0x7fe1a806f350 2024-03-24T00:44:51.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.875+0000 7fe1a7fff640 1 --2- 172.21.15.67:0/3428688945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1a806f350 0x7fe1a80cee20 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fe1940097a0 tx=0x7fe19402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=495fcc3b3b348bea server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:51.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.875+0000 7fe1a77fe640 1 -- 172.21.15.67:0/3428688945 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe194034070 con 0x7fe1a806f350 2024-03-24T00:44:51.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.875+0000 7fe1a77fe640 1 -- 172.21.15.67:0/3428688945 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe194030610 con 0x7fe1a806f350 2024-03-24T00:44:51.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.875+0000 7fe1a77fe640 1 -- 172.21.15.67:0/3428688945 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1940309e0 con 0x7fe1a806f350 2024-03-24T00:44:51.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.875+0000 7fe1aedb9640 1 -- 172.21.15.67:0/3428688945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1a806f350 msgr2=0x7fe1a80cee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:51.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.875+0000 7fe1aedb9640 1 --2- 172.21.15.67:0/3428688945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1a806f350 0x7fe1a80cee20 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fe1940097a0 tx=0x7fe19402b960 comp rx=0 tx=0).stop 2024-03-24T00:44:51.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.875+0000 7fe1aedb9640 1 -- 172.21.15.67:0/3428688945 shutdown_connections 2024-03-24T00:44:51.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.875+0000 7fe1aedb9640 1 --2- 172.21.15.67:0/3428688945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1a806f350 0x7fe1a80cee20 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.875+0000 7fe1aedb9640 1 -- 172.21.15.67:0/3428688945 >> 172.21.15.67:0/3428688945 conn(0x7fe1a80cf9e0 msgr2=0x7fe1a80cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:51.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.875+0000 7fe1aedb9640 1 -- 172.21.15.67:0/3428688945 shutdown_connections 2024-03-24T00:44:51.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.876+0000 7fe1aedb9640 1 -- 172.21.15.67:0/3428688945 wait complete. 2024-03-24T00:44:51.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.877+0000 7fe1aedb9640 1 Processor -- start 2024-03-24T00:44:51.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.877+0000 7fe1aedb9640 1 -- start start 2024-03-24T00:44:51.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.878+0000 7fe1aedb9640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe1a806f350 0x7fe1a8139420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:51.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.878+0000 7fe1aedb9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1a80b48d0 0x7fe1a81e7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:51.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.878+0000 7fe1aedb9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe1a81445a0 0x7fe1a813c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:51.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.878+0000 7fe1aedb9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a8072550 con 0x7fe1a80b48d0 2024-03-24T00:44:51.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.878+0000 7fe1aedb9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a80ce620 con 0x7fe1a81445a0 2024-03-24T00:44:51.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.878+0000 7fe1aedb9640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a80726d0 con 0x7fe1a806f350 2024-03-24T00:44:51.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.878+0000 7fe1a7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1a80b48d0 0x7fe1a81e7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:51.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.878+0000 7fe1a7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1a80b48d0 0x7fe1a81e7ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56096/0 (socket says 172.21.15.67:56096) 2024-03-24T00:44:51.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.878+0000 7fe1a7fff640 1 -- 172.21.15.67:0/1511906465 learned_addr learned my addr 172.21.15.67:0/1511906465 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:51.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.878+0000 7fe1ad356640 1 --2- 172.21.15.67:0/1511906465 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe1a81445a0 0x7fe1a813c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:51.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.878+0000 7fe1acb55640 1 --2- 172.21.15.67:0/1511906465 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe1a806f350 0x7fe1a8139420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:51.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.878+0000 7fe1a7fff640 1 -- 172.21.15.67:0/1511906465 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe1a806f350 msgr2=0x7fe1a8139420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:51.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.878+0000 7fe1a7fff640 1 --2- 172.21.15.67:0/1511906465 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe1a806f350 0x7fe1a8139420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.878+0000 7fe1a7fff640 1 -- 172.21.15.67:0/1511906465 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe1a81445a0 msgr2=0x7fe1a813c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:51.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.878+0000 7fe1a7fff640 1 --2- 172.21.15.67:0/1511906465 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe1a81445a0 0x7fe1a813c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:51.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.878+0000 7fe1a7fff640 1 -- 172.21.15.67:0/1511906465 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1a8139c50 con 0x7fe1a80b48d0 2024-03-24T00:44:51.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.878+0000 7fe1ad356640 1 --2- 172.21.15.67:0/1511906465 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe1a81445a0 0x7fe1a813c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:44:51.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.878+0000 7fe1a7fff640 1 --2- 172.21.15.67:0/1511906465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1a80b48d0 0x7fe1a81e7ab0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7fe19402bf70 tx=0x7fe1940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:51.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.878+0000 7fe1a57fa640 1 -- 172.21.15.67:0/1511906465 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe194034040 con 0x7fe1a80b48d0 2024-03-24T00:44:51.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.878+0000 7fe1a57fa640 1 -- 172.21.15.67:0/1511906465 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe194039a10 con 0x7fe1a80b48d0 2024-03-24T00:44:51.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.878+0000 7fe1a57fa640 1 -- 172.21.15.67:0/1511906465 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe194039e20 con 0x7fe1a80b48d0 2024-03-24T00:44:51.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.878+0000 7fe1aedb9640 1 -- 172.21.15.67:0/1511906465 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1a8139f30 con 0x7fe1a80b48d0 2024-03-24T00:44:51.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.878+0000 7fe1aedb9640 1 -- 172.21.15.67:0/1511906465 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1a813df10 con 0x7fe1a80b48d0 2024-03-24T00:44:51.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.879+0000 7fe1aedb9640 1 -- 172.21.15.67:0/1511906465 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1a804f400 con 0x7fe1a80b48d0 2024-03-24T00:44:51.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.880+0000 7fe1a57fa640 1 -- 172.21.15.67:0/1511906465 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe194037070 con 0x7fe1a80b48d0 2024-03-24T00:44:51.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.880+0000 7fe1a57fa640 1 --2- 172.21.15.67:0/1511906465 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe18c06bc00 0x7fe18c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:51.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.880+0000 7fe1a57fa640 1 -- 172.21.15.67:0/1511906465 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fe1940ac940 con 0x7fe1a80b48d0 2024-03-24T00:44:51.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.880+0000 7fe1acb55640 1 --2- 172.21.15.67:0/1511906465 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe18c06bc00 0x7fe18c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:51.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.881+0000 7fe1acb55640 1 --2- 172.21.15.67:0/1511906465 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe18c06bc00 0x7fe18c06e0c0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fe19c007fd0 tx=0x7fe19c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:51.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:51.885+0000 7fe1a57fa640 1 -- 172.21.15.67:0/1511906465 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe19407b5f0 con 0x7fe1a80b48d0 2024-03-24T00:44:52.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.069+0000 7fe1aedb9640 1 -- 172.21.15.67:0/1511906465 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe1a80cd500 con 0x7fe1a80b48d0 2024-03-24T00:44:52.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.070+0000 7fe1a57fa640 1 -- 172.21.15.67:0/1511906465 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+13105 (secure 0 0 0) 0x7fe19407aef0 con 0x7fe1a80b48d0 2024-03-24T00:44:52.070 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:44:52.070 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":85,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:44:51.364575+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:44:52.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.072+0000 7fe1aedb9640 1 -- 172.21.15.67:0/1511906465 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe18c06bc00 msgr2=0x7fe18c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:52.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.072+0000 7fe1aedb9640 1 --2- 172.21.15.67:0/1511906465 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe18c06bc00 0x7fe18c06e0c0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fe19c007fd0 tx=0x7fe19c009040 comp rx=0 tx=0).stop 2024-03-24T00:44:52.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.072+0000 7fe1aedb9640 1 -- 172.21.15.67:0/1511906465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1a80b48d0 msgr2=0x7fe1a81e7ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:52.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.072+0000 7fe1aedb9640 1 --2- 172.21.15.67:0/1511906465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1a80b48d0 0x7fe1a81e7ab0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7fe19402bf70 tx=0x7fe1940046c0 comp rx=0 tx=0).stop 2024-03-24T00:44:52.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.072+0000 7fe1aedb9640 1 -- 172.21.15.67:0/1511906465 shutdown_connections 2024-03-24T00:44:52.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.072+0000 7fe1aedb9640 1 --2- 172.21.15.67:0/1511906465 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe18c06bc00 0x7fe18c06e0c0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:52.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.072+0000 7fe1aedb9640 1 --2- 172.21.15.67:0/1511906465 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe1a81445a0 0x7fe1a813c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:52.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.072+0000 7fe1aedb9640 1 --2- 172.21.15.67:0/1511906465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1a80b48d0 0x7fe1a81e7ab0 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:52.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.072+0000 7fe1aedb9640 1 --2- 172.21.15.67:0/1511906465 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe1a806f350 0x7fe1a8139420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:52.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.072+0000 7fe1aedb9640 1 -- 172.21.15.67:0/1511906465 >> 172.21.15.67:0/1511906465 conn(0x7fe1a80cf9e0 msgr2=0x7fe1a815c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:52.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.072+0000 7fe1aedb9640 1 -- 172.21.15.67:0/1511906465 shutdown_connections 2024-03-24T00:44:52.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.072+0000 7fe1aedb9640 1 -- 172.21.15.67:0/1511906465 wait complete. 2024-03-24T00:44:52.083 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:38:39.364950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:44:52.083 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:44:52.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.188+0000 7f49fa960640 1 Processor -- start 2024-03-24T00:44:52.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.188+0000 7f49fa960640 1 -- start start 2024-03-24T00:44:52.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.189+0000 7f49fa960640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f49f40b6170 0x7f49f4168c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:52.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.189+0000 7f49fa960640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49f40d2250 con 0x7f49f40d3ae0 2024-03-24T00:44:52.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.189+0000 7f49fa960640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49f40d2530 con 0x7f49f415dea0 2024-03-24T00:44:52.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.189+0000 7f49fa960640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49f40d2810 con 0x7f49f40b6170 2024-03-24T00:44:52.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.189+0000 7f49f37fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f49f40d3ae0 0x7f49f40b5a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52688/0 (socket says 172.21.15.67:52688) 2024-03-24T00:44:52.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.189+0000 7f49f37fe640 1 -- 172.21.15.67:0/1708109070 learned_addr learned my addr 172.21.15.67:0/1708109070 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:44:52.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.189+0000 7f49f8efd640 1 --2- 172.21.15.67:0/1708109070 >> v2:172.21.15.72:3300/0 conn(0x7f49f40b6170 0x7f49f4168c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:52.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.189+0000 7f49f2ffd640 1 -- 172.21.15.67:0/1708109070 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2549626170 0 0) 0x7f49f40d2250 con 0x7f49f40d3ae0 2024-03-24T00:44:52.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.189+0000 7f49f2ffd640 1 -- 172.21.15.67:0/1708109070 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49dc003520 con 0x7f49f40d3ae0 2024-03-24T00:44:52.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.189+0000 7f49f2ffd640 1 -- 172.21.15.67:0/1708109070 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 428404722 0 0) 0x7f49dc003520 con 0x7f49f40d3ae0 2024-03-24T00:44:52.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.189+0000 7f49f2ffd640 1 -- 172.21.15.67:0/1708109070 >> v2:172.21.15.72:3300/0 conn(0x7f49f40b6170 msgr2=0x7f49f4168c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:52.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.189+0000 7f49f2ffd640 1 --2- 172.21.15.67:0/1708109070 >> v2:172.21.15.72:3300/0 conn(0x7f49f40b6170 0x7f49f4168c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:52.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.189+0000 7f49f2ffd640 1 -- 172.21.15.67:0/1708109070 >> v1:172.21.15.72:6789/0 conn(0x7f49f415dea0 legacy=0x7f49f40d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:52.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.189+0000 7f49f2ffd640 1 -- 172.21.15.67:0/1708109070 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49f40d2af0 con 0x7f49f40d3ae0 2024-03-24T00:44:52.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.189+0000 7f49f2ffd640 1 -- 172.21.15.67:0/1708109070 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f49e4001670 con 0x7f49f40d3ae0 2024-03-24T00:44:52.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.189+0000 7f49f2ffd640 1 -- 172.21.15.67:0/1708109070 >> v1:172.21.15.67:6789/0 conn(0x7f49f40d3ae0 legacy=0x7f49f40b5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:52.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.190+0000 7f49f2ffd640 1 --2- 172.21.15.67:0/1708109070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49dc0039b0 0x7f49dc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:52.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.190+0000 7f49f2ffd640 1 --2- 172.21.15.67:0/1708109070 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f49dc004800 0x7f49dc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:52.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.190+0000 7f49f2ffd640 1 --2- 172.21.15.67:0/1708109070 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f49dc0051c0 0x7f49dc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:52.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.190+0000 7f49f2ffd640 1 -- 172.21.15.67:0/1708109070 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f40d2810 con 0x7f49dc0039b0 2024-03-24T00:44:52.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.190+0000 7f49f2ffd640 1 -- 172.21.15.67:0/1708109070 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49dc00c160 con 0x7f49dc004800 2024-03-24T00:44:52.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.190+0000 7f49f2ffd640 1 -- 172.21.15.67:0/1708109070 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49dc00c360 con 0x7f49dc0051c0 2024-03-24T00:44:52.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.190+0000 7f49f3fff640 1 --2- 172.21.15.67:0/1708109070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49dc0039b0 0x7f49dc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:52.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.190+0000 7f49f37fe640 1 --2- 172.21.15.67:0/1708109070 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f49dc004800 0x7f49dc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:52.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.190+0000 7f49f8efd640 1 --2- 172.21.15.67:0/1708109070 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f49dc0051c0 0x7f49dc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:52.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.190+0000 7f49f3fff640 1 -- 172.21.15.67:0/1708109070 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f49dc0051c0 msgr2=0x7f49dc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:52.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.190+0000 7f49f3fff640 1 --2- 172.21.15.67:0/1708109070 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f49dc0051c0 0x7f49dc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:52.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.190+0000 7f49f3fff640 1 -- 172.21.15.67:0/1708109070 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f49dc004800 msgr2=0x7f49dc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:52.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.190+0000 7f49f3fff640 1 --2- 172.21.15.67:0/1708109070 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f49dc004800 0x7f49dc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:52.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.190+0000 7f49f3fff640 1 -- 172.21.15.67:0/1708109070 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49dc003520 con 0x7f49dc0039b0 2024-03-24T00:44:52.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.190+0000 7f49f37fe640 1 --2- 172.21.15.67:0/1708109070 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f49dc004800 0x7f49dc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:44:52.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.190+0000 7f49f8efd640 1 -- 172.21.15.67:0/1708109070 reap_dead start 2024-03-24T00:44:52.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.191+0000 7f49f3fff640 1 --2- 172.21.15.67:0/1708109070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49dc0039b0 0x7f49dc003dd0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f49e8001750 tx=0x7f49dc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e5864fb813c512a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:52.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.191+0000 7f49f2ffd640 1 -- 172.21.15.67:0/1708109070 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e8001370 con 0x7f49dc0039b0 2024-03-24T00:44:52.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.191+0000 7f49f2ffd640 1 -- 172.21.15.67:0/1708109070 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49e80052d0 con 0x7f49dc0039b0 2024-03-24T00:44:52.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.191+0000 7f49f2ffd640 1 -- 172.21.15.67:0/1708109070 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e8003830 con 0x7f49dc0039b0 2024-03-24T00:44:52.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.191+0000 7f49fa960640 1 -- 172.21.15.67:0/1708109070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49dc0039b0 msgr2=0x7f49dc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:52.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.191+0000 7f49fa960640 1 --2- 172.21.15.67:0/1708109070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49dc0039b0 0x7f49dc003dd0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f49e8001750 tx=0x7f49dc005830 comp rx=0 tx=0).stop 2024-03-24T00:44:52.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.191+0000 7f49fa960640 1 -- 172.21.15.67:0/1708109070 shutdown_connections 2024-03-24T00:44:52.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.191+0000 7f49fa960640 1 --2- 172.21.15.67:0/1708109070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49dc0039b0 0x7f49dc003dd0 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:52.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.191+0000 7f49fa960640 1 -- 172.21.15.67:0/1708109070 >> 172.21.15.67:0/1708109070 conn(0x7f49f4072da0 msgr2=0x7f49f40d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:52.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.191+0000 7f49fa960640 1 -- 172.21.15.67:0/1708109070 shutdown_connections 2024-03-24T00:44:52.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.191+0000 7f49fa960640 1 -- 172.21.15.67:0/1708109070 wait complete. 2024-03-24T00:44:52.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.192+0000 7f49fa960640 1 Processor -- start 2024-03-24T00:44:52.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.193+0000 7f49fa960640 1 -- start start 2024-03-24T00:44:52.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.193+0000 7f49fa960640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49dc0039b0 0x7f49f40cd010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:52.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.193+0000 7f49fa960640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f49f40b3290 0x7f49f40ab440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:52.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.193+0000 7f49fa960640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f49f4159a40 0x7f49f4159ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:52.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.193+0000 7f49fa960640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f40b5a60 con 0x7f49dc0039b0 2024-03-24T00:44:52.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.193+0000 7f49fa960640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f415e490 con 0x7f49f4159a40 2024-03-24T00:44:52.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.193+0000 7f49fa960640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49f40b5be0 con 0x7f49f40b3290 2024-03-24T00:44:52.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.193+0000 7f49f3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49dc0039b0 0x7f49f40cd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:52.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.193+0000 7f49f3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49dc0039b0 0x7f49f40cd010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56116/0 (socket says 172.21.15.67:56116) 2024-03-24T00:44:52.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.193+0000 7f49f3fff640 1 -- 172.21.15.67:0/3967537750 learned_addr learned my addr 172.21.15.67:0/3967537750 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:52.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.194+0000 7f49f8efd640 1 --2- 172.21.15.67:0/3967537750 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f49f4159a40 0x7f49f4159ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:52.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.194+0000 7f49f37fe640 1 --2- 172.21.15.67:0/3967537750 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f49f40b3290 0x7f49f40ab440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:52.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.194+0000 7f49f3fff640 1 -- 172.21.15.67:0/3967537750 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f49f40b3290 msgr2=0x7f49f40ab440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:52.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.194+0000 7f49f3fff640 1 --2- 172.21.15.67:0/3967537750 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f49f40b3290 0x7f49f40ab440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:52.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.194+0000 7f49f3fff640 1 -- 172.21.15.67:0/3967537750 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f49f4159a40 msgr2=0x7f49f4159ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:52.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.194+0000 7f49f3fff640 1 --2- 172.21.15.67:0/3967537750 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f49f4159a40 0x7f49f4159ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:52.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.194+0000 7f49f3fff640 1 -- 172.21.15.67:0/3967537750 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49f40cd910 con 0x7f49dc0039b0 2024-03-24T00:44:52.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.194+0000 7f49f3fff640 1 --2- 172.21.15.67:0/3967537750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49dc0039b0 0x7f49f40cd010 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f49e8000c00 tx=0x7f49e8001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:52.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.194+0000 7f49f0ff9640 1 -- 172.21.15.67:0/3967537750 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e8041070 con 0x7f49dc0039b0 2024-03-24T00:44:52.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.194+0000 7f49f0ff9640 1 -- 172.21.15.67:0/3967537750 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49e8005480 con 0x7f49dc0039b0 2024-03-24T00:44:52.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.194+0000 7f49f0ff9640 1 -- 172.21.15.67:0/3967537750 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e8001370 con 0x7f49dc0039b0 2024-03-24T00:44:52.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.194+0000 7f49fa960640 1 -- 172.21.15.67:0/3967537750 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49f40cdbf0 con 0x7f49dc0039b0 2024-03-24T00:44:52.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.194+0000 7f49fa960640 1 -- 172.21.15.67:0/3967537750 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49f415f880 con 0x7f49dc0039b0 2024-03-24T00:44:52.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.195+0000 7f49fa960640 1 -- 172.21.15.67:0/3967537750 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49f404f400 con 0x7f49dc0039b0 2024-03-24T00:44:52.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.195+0000 7f49f0ff9640 1 -- 172.21.15.67:0/3967537750 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f49e8044070 con 0x7f49dc0039b0 2024-03-24T00:44:52.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.196+0000 7f49f0ff9640 1 --2- 172.21.15.67:0/3967537750 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f49e006bbd0 0x7f49e006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:52.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.196+0000 7f49f0ff9640 1 -- 172.21.15.67:0/3967537750 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f49e80ac260 con 0x7f49dc0039b0 2024-03-24T00:44:52.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.196+0000 7f49f37fe640 1 --2- 172.21.15.67:0/3967537750 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f49e006bbd0 0x7f49e006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:52.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.197+0000 7f49f37fe640 1 --2- 172.21.15.67:0/3967537750 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f49e006bbd0 0x7f49e006e090 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f49f415e840 tx=0x7f49e4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:52.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.200+0000 7f49f0ff9640 1 -- 172.21.15.67:0/3967537750 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f49e807afa0 con 0x7f49dc0039b0 2024-03-24T00:44:52.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.383+0000 7f49fa960640 1 -- 172.21.15.67:0/3967537750 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f49f40b94b0 con 0x7f49dc0039b0 2024-03-24T00:44:52.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.384+0000 7f49f0ff9640 1 -- 172.21.15.67:0/3967537750 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7f49e807a8a0 con 0x7f49dc0039b0 2024-03-24T00:44:52.383 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-24T00:44:52.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.386+0000 7f49fa960640 1 -- 172.21.15.67:0/3967537750 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f49e006bbd0 msgr2=0x7f49e006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:52.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.386+0000 7f49fa960640 1 --2- 172.21.15.67:0/3967537750 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f49e006bbd0 0x7f49e006e090 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f49f415e840 tx=0x7f49e4009040 comp rx=0 tx=0).stop 2024-03-24T00:44:52.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.386+0000 7f49fa960640 1 -- 172.21.15.67:0/3967537750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49dc0039b0 msgr2=0x7f49f40cd010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:52.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.386+0000 7f49fa960640 1 --2- 172.21.15.67:0/3967537750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49dc0039b0 0x7f49f40cd010 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f49e8000c00 tx=0x7f49e8001120 comp rx=0 tx=0).stop 2024-03-24T00:44:52.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.386+0000 7f49fa960640 1 -- 172.21.15.67:0/3967537750 shutdown_connections 2024-03-24T00:44:52.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.386+0000 7f49fa960640 1 --2- 172.21.15.67:0/3967537750 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f49e006bbd0 0x7f49e006e090 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:52.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.386+0000 7f49fa960640 1 --2- 172.21.15.67:0/3967537750 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f49f4159a40 0x7f49f4159ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:52.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.386+0000 7f49fa960640 1 --2- 172.21.15.67:0/3967537750 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f49f40b3290 0x7f49f40ab440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:52.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.386+0000 7f49fa960640 1 --2- 172.21.15.67:0/3967537750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49dc0039b0 0x7f49f40cd010 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:52.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.386+0000 7f49fa960640 1 -- 172.21.15.67:0/3967537750 >> 172.21.15.67:0/3967537750 conn(0x7f49f4072da0 msgr2=0x7f49f40b88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:52.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.386+0000 7f49fa960640 1 -- 172.21.15.67:0/3967537750 shutdown_connections 2024-03-24T00:44:52.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.386+0000 7f49fa960640 1 -- 172.21.15.67:0/3967537750 wait complete. 2024-03-24T00:44:52.397 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-24T00:44:52.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.499+0000 7f63d7875640 1 Processor -- start 2024-03-24T00:44:52.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.499+0000 7f63d7875640 1 -- start start 2024-03-24T00:44:52.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.500+0000 7f63d7875640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f63d015df30 0x7f63d0162610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:52.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.500+0000 7f63d7875640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f63d015e310 0x7f63d00d0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:52.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.500+0000 7f63d7875640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f63d00d0b80 0x7f63d00b57a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:52.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.500+0000 7f63d7875640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63d0162b50 con 0x7f63d015e310 2024-03-24T00:44:52.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.500+0000 7f63d7875640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63d00d0fd0 con 0x7f63d00d0b80 2024-03-24T00:44:52.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.500+0000 7f63d7875640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63d00ce920 con 0x7f63d015df30 2024-03-24T00:44:52.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.500+0000 7f63d4e10640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f63d015e310 0x7f63d00d0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:52.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.500+0000 7f63d4e10640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f63d015e310 0x7f63d00d0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56130/0 (socket says 172.21.15.67:56130) 2024-03-24T00:44:52.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.500+0000 7f63d4e10640 1 -- 172.21.15.67:0/587307097 learned_addr learned my addr 172.21.15.67:0/587307097 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:52.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.500+0000 7f63d5611640 1 --2- 172.21.15.67:0/587307097 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f63d015df30 0x7f63d0162610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:52.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.500+0000 7f63d4e10640 1 -- 172.21.15.67:0/587307097 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f63d015df30 msgr2=0x7f63d0162610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:52.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.500+0000 7f63d5e12640 1 --2- 172.21.15.67:0/587307097 >> v2:172.21.15.72:3300/0 conn(0x7f63d00d0b80 0x7f63d00b57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:52.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.500+0000 7f63d4e10640 1 --2- 172.21.15.67:0/587307097 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f63d015df30 0x7f63d0162610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:52.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.500+0000 7f63d4e10640 1 -- 172.21.15.67:0/587307097 >> v2:172.21.15.72:3300/0 conn(0x7f63d00d0b80 msgr2=0x7f63d00b57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:52.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.500+0000 7f63d4e10640 1 --2- 172.21.15.67:0/587307097 >> v2:172.21.15.72:3300/0 conn(0x7f63d00d0b80 0x7f63d00b57a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:52.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.500+0000 7f63d4e10640 1 -- 172.21.15.67:0/587307097 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d00cef20 con 0x7f63d015e310 2024-03-24T00:44:52.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.501+0000 7f63d4e10640 1 --2- 172.21.15.67:0/587307097 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63d015e310 0x7f63d00d0640 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f63d006d7b0 tx=0x7f63c402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8126b7d8257045e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:52.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.501+0000 7f63cffff640 1 -- 172.21.15.67:0/587307097 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c40317c0 con 0x7f63d015e310 2024-03-24T00:44:52.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.501+0000 7f63cffff640 1 -- 172.21.15.67:0/587307097 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c4037070 con 0x7f63d015e310 2024-03-24T00:44:52.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.501+0000 7f63cffff640 1 -- 172.21.15.67:0/587307097 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c4031b80 con 0x7f63d015e310 2024-03-24T00:44:52.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.502+0000 7f63d7875640 1 -- 172.21.15.67:0/587307097 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63d015e310 msgr2=0x7f63d00d0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:52.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.502+0000 7f63d7875640 1 --2- 172.21.15.67:0/587307097 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63d015e310 0x7f63d00d0640 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f63d006d7b0 tx=0x7f63c402b860 comp rx=0 tx=0).stop 2024-03-24T00:44:52.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.502+0000 7f63d7875640 1 -- 172.21.15.67:0/587307097 shutdown_connections 2024-03-24T00:44:52.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.502+0000 7f63d7875640 1 --2- 172.21.15.67:0/587307097 >> v2:172.21.15.72:3300/0 conn(0x7f63d00d0b80 0x7f63d00b57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:52.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.502+0000 7f63d7875640 1 --2- 172.21.15.67:0/587307097 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63d015e310 0x7f63d00d0640 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:52.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.502+0000 7f63d7875640 1 --2- 172.21.15.67:0/587307097 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f63d015df30 0x7f63d0162610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:52.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.502+0000 7f63d7875640 1 -- 172.21.15.67:0/587307097 >> 172.21.15.67:0/587307097 conn(0x7f63d006c880 msgr2=0x7f63d006cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:52.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.502+0000 7f63d7875640 1 -- 172.21.15.67:0/587307097 shutdown_connections 2024-03-24T00:44:52.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.502+0000 7f63d7875640 1 -- 172.21.15.67:0/587307097 wait complete. 2024-03-24T00:44:52.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.503+0000 7f63d7875640 1 Processor -- start 2024-03-24T00:44:52.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.503+0000 7f63d7875640 1 -- start start 2024-03-24T00:44:52.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.504+0000 7f63d7875640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63d00d0b80 0x7f63d0159bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:52.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.504+0000 7f63d7875640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63d015df30 0x7f63d00b77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:52.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.504+0000 7f63d7875640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f63d01f38a0 0x7f63d00b7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:52.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.504+0000 7f63d7875640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d00b32f0 con 0x7f63d015df30 2024-03-24T00:44:52.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.504+0000 7f63d7875640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d00cf4e0 con 0x7f63d00d0b80 2024-03-24T00:44:52.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.504+0000 7f63d7875640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63d00ce580 con 0x7f63d01f38a0 2024-03-24T00:44:52.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.504+0000 7f63d4e10640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63d015df30 0x7f63d00b77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:52.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.504+0000 7f63d4e10640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63d015df30 0x7f63d00b77a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56132/0 (socket says 172.21.15.67:56132) 2024-03-24T00:44:52.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.504+0000 7f63d4e10640 1 -- 172.21.15.67:0/1798034651 learned_addr learned my addr 172.21.15.67:0/1798034651 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:52.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.504+0000 7f63d5e12640 1 --2- 172.21.15.67:0/1798034651 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f63d01f38a0 0x7f63d00b7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:52.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.504+0000 7f63d4e10640 1 -- 172.21.15.67:0/1798034651 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f63d01f38a0 msgr2=0x7f63d00b7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:52.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.504+0000 7f63d5611640 1 --2- 172.21.15.67:0/1798034651 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63d00d0b80 0x7f63d0159bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:52.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.504+0000 7f63d4e10640 1 --2- 172.21.15.67:0/1798034651 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f63d01f38a0 0x7f63d00b7ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:52.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.504+0000 7f63d4e10640 1 -- 172.21.15.67:0/1798034651 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63d00d0b80 msgr2=0x7f63d0159bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:52.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.504+0000 7f63d4e10640 1 --2- 172.21.15.67:0/1798034651 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63d00d0b80 0x7f63d0159bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:52.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.504+0000 7f63d4e10640 1 -- 172.21.15.67:0/1798034651 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d015a260 con 0x7f63d015df30 2024-03-24T00:44:52.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.504+0000 7f63d4e10640 1 --2- 172.21.15.67:0/1798034651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63d015df30 0x7f63d00b77a0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f63c4000c00 tx=0x7f63c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:52.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.505+0000 7f63cdffb640 1 -- 172.21.15.67:0/1798034651 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c4032860 con 0x7f63d015df30 2024-03-24T00:44:52.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.505+0000 7f63cdffb640 1 -- 172.21.15.67:0/1798034651 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c4032a00 con 0x7f63d015df30 2024-03-24T00:44:52.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.505+0000 7f63d7875640 1 -- 172.21.15.67:0/1798034651 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63d015a540 con 0x7f63d015df30 2024-03-24T00:44:52.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.505+0000 7f63cdffb640 1 -- 172.21.15.67:0/1798034651 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c4032df0 con 0x7f63d015df30 2024-03-24T00:44:52.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.505+0000 7f63d7875640 1 -- 172.21.15.67:0/1798034651 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63d015ab00 con 0x7f63d015df30 2024-03-24T00:44:52.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.506+0000 7f63d7875640 1 -- 172.21.15.67:0/1798034651 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63d004f400 con 0x7f63d015df30 2024-03-24T00:44:52.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.506+0000 7f63cdffb640 1 -- 172.21.15.67:0/1798034651 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f63c404c060 con 0x7f63d015df30 2024-03-24T00:44:52.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.506+0000 7f63cdffb640 1 --2- 172.21.15.67:0/1798034651 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f63bc06bbd0 0x7f63bc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:52.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.507+0000 7f63cdffb640 1 -- 172.21.15.67:0/1798034651 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f63c40ac7d0 con 0x7f63d015df30 2024-03-24T00:44:52.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.507+0000 7f63d5611640 1 --2- 172.21.15.67:0/1798034651 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f63bc06bbd0 0x7f63bc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:52.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.507+0000 7f63d5611640 1 --2- 172.21.15.67:0/1798034651 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f63bc06bbd0 0x7f63bc06e090 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f63c8006fd0 tx=0x7f63c8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:52.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.511+0000 7f63cdffb640 1 -- 172.21.15.67:0/1798034651 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f63c4040070 con 0x7f63d015df30 2024-03-24T00:44:52.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.693+0000 7f63d7875640 1 -- 172.21.15.67:0/1798034651 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f63d00d1a30 con 0x7f63d015df30 2024-03-24T00:44:52.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.694+0000 7f63cdffb640 1 -- 172.21.15.67:0/1798034651 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v85) v1 ==== 84+0+65 (secure 0 0 0) 0x7f63c407b400 con 0x7f63d015df30 2024-03-24T00:44:52.693 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:44:52.693 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-03-24T00:44:52.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:44:52.694 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2024-03-24T00:44:52.694 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2024-03-24T00:44:52.694 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:44:52.694 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-03-24T00:44:52.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.695+0000 7f63d7875640 1 -- 172.21.15.67:0/1798034651 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f63bc06bbd0 msgr2=0x7f63bc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:52.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.695+0000 7f63d7875640 1 --2- 172.21.15.67:0/1798034651 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f63bc06bbd0 0x7f63bc06e090 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f63c8006fd0 tx=0x7f63c8008040 comp rx=0 tx=0).stop 2024-03-24T00:44:52.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.695+0000 7f63d7875640 1 -- 172.21.15.67:0/1798034651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63d015df30 msgr2=0x7f63d00b77a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:52.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.695+0000 7f63d7875640 1 --2- 172.21.15.67:0/1798034651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63d015df30 0x7f63d00b77a0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f63c4000c00 tx=0x7f63c40046c0 comp rx=0 tx=0).stop 2024-03-24T00:44:52.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.695+0000 7f63d7875640 1 -- 172.21.15.67:0/1798034651 shutdown_connections 2024-03-24T00:44:52.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.695+0000 7f63d7875640 1 --2- 172.21.15.67:0/1798034651 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f63bc06bbd0 0x7f63bc06e090 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:52.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.695+0000 7f63d7875640 1 --2- 172.21.15.67:0/1798034651 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f63d01f38a0 0x7f63d00b7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:52.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.695+0000 7f63d7875640 1 --2- 172.21.15.67:0/1798034651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63d015df30 0x7f63d00b77a0 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:52.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.695+0000 7f63d7875640 1 --2- 172.21.15.67:0/1798034651 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63d00d0b80 0x7f63d0159bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:52.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.695+0000 7f63d7875640 1 -- 172.21.15.67:0/1798034651 >> 172.21.15.67:0/1798034651 conn(0x7f63d006c880 msgr2=0x7f63d006cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:52.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.696+0000 7f63d7875640 1 -- 172.21.15.67:0/1798034651 shutdown_connections 2024-03-24T00:44:52.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.696+0000 7f63d7875640 1 -- 172.21.15.67:0/1798034651 wait complete. 2024-03-24T00:44:52.706 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-24T00:44:52.706 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-24T00:44:52.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.812+0000 7f3716c73640 1 Processor -- start 2024-03-24T00:44:52.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.813+0000 7f3716c73640 1 -- start start 2024-03-24T00:44:52.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.813+0000 7f3716c73640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f371015dfc0 0x7f37101624c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:52.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.813+0000 7f3716c73640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f37100ced60 0x7f37100d0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:52.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.813+0000 7f3716c73640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37100d0fa0 con 0x7f37100d0b80 2024-03-24T00:44:52.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.813+0000 7f3716c73640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f371015e3a0 con 0x7f37100ced60 2024-03-24T00:44:52.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.813+0000 7f3716c73640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37100cf4d0 con 0x7f371015dfc0 2024-03-24T00:44:52.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.813+0000 7f3715210640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f37100d0b80 0x7f37100b5830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52692/0 (socket says 172.21.15.67:52692) 2024-03-24T00:44:52.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.813+0000 7f3715210640 1 -- 172.21.15.67:0/3901759981 learned_addr learned my addr 172.21.15.67:0/3901759981 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:44:52.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.814+0000 7f370f7fe640 1 -- 172.21.15.67:0/3901759981 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1821262005 0 0) 0x7f37100d0fa0 con 0x7f37100d0b80 2024-03-24T00:44:52.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.814+0000 7f370ffff640 1 --2- 172.21.15.67:0/3901759981 >> v2:172.21.15.72:3300/0 conn(0x7f37100ced60 0x7f37100d0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:52.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.814+0000 7f3714a0f640 1 --2- 172.21.15.67:0/3901759981 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f371015dfc0 0x7f37101624c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:52.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.814+0000 7f370f7fe640 1 -- 172.21.15.67:0/3901759981 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3700003520 con 0x7f37100d0b80 2024-03-24T00:44:52.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.814+0000 7f370f7fe640 1 -- 172.21.15.67:0/3901759981 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3713653391 0 0) 0x7f3700003520 con 0x7f37100d0b80 2024-03-24T00:44:52.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.814+0000 7f370f7fe640 1 -- 172.21.15.67:0/3901759981 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f371015dfc0 msgr2=0x7f37101624c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:52.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.814+0000 7f370f7fe640 1 --2- 172.21.15.67:0/3901759981 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f371015dfc0 0x7f37101624c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:52.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.814+0000 7f370f7fe640 1 -- 172.21.15.67:0/3901759981 >> v2:172.21.15.72:3300/0 conn(0x7f37100ced60 msgr2=0x7f37100d0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:52.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.814+0000 7f370f7fe640 1 --2- 172.21.15.67:0/3901759981 >> v2:172.21.15.72:3300/0 conn(0x7f37100ced60 0x7f37100d0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:52.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.814+0000 7f370f7fe640 1 -- 172.21.15.67:0/3901759981 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3710162a00 con 0x7f37100d0b80 2024-03-24T00:44:52.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.814+0000 7f370f7fe640 1 -- 172.21.15.67:0/3901759981 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f37080019f0 con 0x7f37100d0b80 2024-03-24T00:44:52.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.814+0000 7f370f7fe640 1 -- 172.21.15.67:0/3901759981 >> v1:172.21.15.67:6789/0 conn(0x7f37100d0b80 legacy=0x7f37100b5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:52.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.814+0000 7f370f7fe640 1 --2- 172.21.15.67:0/3901759981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37000039b0 0x7f3700003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:52.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.814+0000 7f370f7fe640 1 --2- 172.21.15.67:0/3901759981 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f371015dfc0 0x7f3700004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:52.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.814+0000 7f370f7fe640 1 --2- 172.21.15.67:0/3901759981 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3700004d40 0x7f37000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:52.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.814+0000 7f370f7fe640 1 -- 172.21.15.67:0/3901759981 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f371015e3a0 con 0x7f37000039b0 2024-03-24T00:44:52.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.814+0000 7f370f7fe640 1 -- 172.21.15.67:0/3901759981 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37100cf4d0 con 0x7f3700004d40 2024-03-24T00:44:52.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.814+0000 7f370f7fe640 1 -- 172.21.15.67:0/3901759981 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f370000b9c0 con 0x7f371015dfc0 2024-03-24T00:44:52.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.814+0000 7f3714a0f640 1 --2- 172.21.15.67:0/3901759981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37000039b0 0x7f3700003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:52.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.814+0000 7f370ffff640 1 --2- 172.21.15.67:0/3901759981 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f371015dfc0 0x7f3700004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:52.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.814+0000 7f3715210640 1 --2- 172.21.15.67:0/3901759981 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3700004d40 0x7f37000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:52.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.814+0000 7f3714a0f640 1 -- 172.21.15.67:0/3901759981 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f371015dfc0 msgr2=0x7f3700004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:52.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.815+0000 7f3714a0f640 1 --2- 172.21.15.67:0/3901759981 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f371015dfc0 0x7f3700004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:52.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.815+0000 7f3714a0f640 1 -- 172.21.15.67:0/3901759981 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3700004d40 msgr2=0x7f37000051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:52.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.815+0000 7f3714a0f640 1 --2- 172.21.15.67:0/3901759981 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3700004d40 0x7f37000051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:52.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.815+0000 7f3714a0f640 1 -- 172.21.15.67:0/3901759981 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3700003520 con 0x7f37000039b0 2024-03-24T00:44:52.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.815+0000 7f3715210640 1 --2- 172.21.15.67:0/3901759981 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3700004d40 0x7f37000051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:44:52.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.815+0000 7f3714a0f640 1 --2- 172.21.15.67:0/3901759981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37000039b0 0x7f3700003dd0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f3704009ac0 tx=0x7f3700005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd85f3ff1497ba13 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:52.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.815+0000 7f370f7fe640 1 -- 172.21.15.67:0/3901759981 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3704039070 con 0x7f37000039b0 2024-03-24T00:44:52.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.815+0000 7f370f7fe640 1 -- 172.21.15.67:0/3901759981 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f370402c6b0 con 0x7f37000039b0 2024-03-24T00:44:52.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.815+0000 7f370f7fe640 1 -- 172.21.15.67:0/3901759981 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3704009690 con 0x7f37000039b0 2024-03-24T00:44:52.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.816+0000 7f3716c73640 1 -- 172.21.15.67:0/3901759981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37000039b0 msgr2=0x7f3700003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:52.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.816+0000 7f3716c73640 1 --2- 172.21.15.67:0/3901759981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37000039b0 0x7f3700003dd0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f3704009ac0 tx=0x7f3700005830 comp rx=0 tx=0).stop 2024-03-24T00:44:52.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.816+0000 7f3715210640 1 -- 172.21.15.67:0/3901759981 reap_dead start 2024-03-24T00:44:52.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.816+0000 7f3716c73640 1 -- 172.21.15.67:0/3901759981 shutdown_connections 2024-03-24T00:44:52.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.816+0000 7f3716c73640 1 -- 172.21.15.67:0/3901759981 >> 172.21.15.67:0/3901759981 conn(0x7f371006c880 msgr2=0x7f371006cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:52.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.816+0000 7f3716c73640 1 -- 172.21.15.67:0/3901759981 shutdown_connections 2024-03-24T00:44:52.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.816+0000 7f3716c73640 1 -- 172.21.15.67:0/3901759981 wait complete. 2024-03-24T00:44:52.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.817+0000 7f3716c73640 1 Processor -- start 2024-03-24T00:44:52.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.817+0000 7f3716c73640 1 -- start start 2024-03-24T00:44:52.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.818+0000 7f3716c73640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37000039b0 0x7f37101500e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:52.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.818+0000 7f3716c73640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f371015f1d0 0x7f371015f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:52.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.818+0000 7f3716c73640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37101f4910 0x7f37101f4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:52.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.818+0000 7f3716c73640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37100d2300 con 0x7f37000039b0 2024-03-24T00:44:52.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.818+0000 7f3716c73640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37100d2020 con 0x7f37101f4910 2024-03-24T00:44:52.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.818+0000 7f3716c73640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37100d25e0 con 0x7f371015f1d0 2024-03-24T00:44:52.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.818+0000 7f3714a0f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37000039b0 0x7f37101500e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:52.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.818+0000 7f3714a0f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37000039b0 0x7f37101500e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56148/0 (socket says 172.21.15.67:56148) 2024-03-24T00:44:52.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.818+0000 7f3714a0f640 1 -- 172.21.15.67:0/1082176845 learned_addr learned my addr 172.21.15.67:0/1082176845 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:52.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.818+0000 7f370ffff640 1 --2- 172.21.15.67:0/1082176845 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f371015f1d0 0x7f371015f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:52.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.818+0000 7f3715210640 1 --2- 172.21.15.67:0/1082176845 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37101f4910 0x7f37101f4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:52.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.819+0000 7f3714a0f640 1 -- 172.21.15.67:0/1082176845 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f371015f1d0 msgr2=0x7f371015f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:52.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.819+0000 7f3714a0f640 1 --2- 172.21.15.67:0/1082176845 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f371015f1d0 0x7f371015f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:52.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.819+0000 7f3714a0f640 1 -- 172.21.15.67:0/1082176845 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37101f4910 msgr2=0x7f37101f4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:52.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.819+0000 7f3714a0f640 1 --2- 172.21.15.67:0/1082176845 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37101f4910 0x7f37101f4d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:52.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.819+0000 7f3714a0f640 1 -- 172.21.15.67:0/1082176845 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f371015fcc0 con 0x7f37000039b0 2024-03-24T00:44:52.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.819+0000 7f3715210640 1 --2- 172.21.15.67:0/1082176845 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37101f4910 0x7f37101f4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:44:52.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.819+0000 7f3714a0f640 1 --2- 172.21.15.67:0/1082176845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37000039b0 0x7f37101500e0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f3704000c00 tx=0x7f3704004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:52.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.819+0000 7f370d7fa640 1 -- 172.21.15.67:0/1082176845 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3704042020 con 0x7f37000039b0 2024-03-24T00:44:52.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.819+0000 7f370d7fa640 1 -- 172.21.15.67:0/1082176845 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f370403a040 con 0x7f37000039b0 2024-03-24T00:44:52.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.819+0000 7f3716c73640 1 -- 172.21.15.67:0/1082176845 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f371015ffa0 con 0x7f37000039b0 2024-03-24T00:44:52.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.819+0000 7f370d7fa640 1 -- 172.21.15.67:0/1082176845 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3704039070 con 0x7f37000039b0 2024-03-24T00:44:52.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.819+0000 7f3716c73640 1 -- 172.21.15.67:0/1082176845 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3710160560 con 0x7f37000039b0 2024-03-24T00:44:52.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.820+0000 7f3716c73640 1 -- 172.21.15.67:0/1082176845 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37100cfa80 con 0x7f37000039b0 2024-03-24T00:44:52.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.820+0000 7f370d7fa640 1 -- 172.21.15.67:0/1082176845 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3704045020 con 0x7f37000039b0 2024-03-24T00:44:52.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.821+0000 7f370d7fa640 1 --2- 172.21.15.67:0/1082176845 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f36f406bc00 0x7f36f406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:52.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.821+0000 7f370d7fa640 1 -- 172.21.15.67:0/1082176845 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f37040abc80 con 0x7f37000039b0 2024-03-24T00:44:52.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.821+0000 7f370ffff640 1 --2- 172.21.15.67:0/1082176845 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f36f406bc00 0x7f36f406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:52.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.821+0000 7f370ffff640 1 --2- 172.21.15.67:0/1082176845 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f36f406bc00 0x7f36f406e0c0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f37100b6550 tx=0x7f36fc009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:52.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:52.825+0000 7f370d7fa640 1 -- 172.21.15.67:0/1082176845 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f37100cfa80 con 0x7f37000039b0 2024-03-24T00:44:53.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.007+0000 7f3716c73640 1 -- 172.21.15.67:0/1082176845 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f371004f400 con 0x7f37000039b0 2024-03-24T00:44:53.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.370+0000 7f370d7fa640 1 -- 172.21.15.67:0/1082176845 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v86) v1 ==== 138+0+0 (secure 0 0 0) 0x7f370404c070 con 0x7f37000039b0 2024-03-24T00:44:53.370 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' created 2024-03-24T00:44:53.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.373+0000 7f3716c73640 1 -- 172.21.15.67:0/1082176845 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f36f406bc00 msgr2=0x7f36f406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:53.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.373+0000 7f3716c73640 1 --2- 172.21.15.67:0/1082176845 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f36f406bc00 0x7f36f406e0c0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f37100b6550 tx=0x7f36fc009190 comp rx=0 tx=0).stop 2024-03-24T00:44:53.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.373+0000 7f3716c73640 1 -- 172.21.15.67:0/1082176845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37000039b0 msgr2=0x7f37101500e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:53.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.373+0000 7f3716c73640 1 --2- 172.21.15.67:0/1082176845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37000039b0 0x7f37101500e0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f3704000c00 tx=0x7f3704004730 comp rx=0 tx=0).stop 2024-03-24T00:44:53.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.373+0000 7f3716c73640 1 -- 172.21.15.67:0/1082176845 shutdown_connections 2024-03-24T00:44:53.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.373+0000 7f3716c73640 1 --2- 172.21.15.67:0/1082176845 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f36f406bc00 0x7f36f406e0c0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.373+0000 7f3716c73640 1 --2- 172.21.15.67:0/1082176845 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37101f4910 0x7f37101f4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.373+0000 7f3716c73640 1 --2- 172.21.15.67:0/1082176845 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f371015f1d0 0x7f371015f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.373+0000 7f3716c73640 1 --2- 172.21.15.67:0/1082176845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37000039b0 0x7f37101500e0 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.373+0000 7f3716c73640 1 -- 172.21.15.67:0/1082176845 >> 172.21.15.67:0/1082176845 conn(0x7f371006c880 msgr2=0x7f37100b8e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:53.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.373+0000 7f3716c73640 1 -- 172.21.15.67:0/1082176845 shutdown_connections 2024-03-24T00:44:53.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.373+0000 7f3716c73640 1 -- 172.21.15.67:0/1082176845 wait complete. 2024-03-24T00:44:53.388 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-24T00:44:53.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.498+0000 7fb9cc2fe640 1 Processor -- start 2024-03-24T00:44:53.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.498+0000 7fb9cc2fe640 1 -- start start 2024-03-24T00:44:53.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.499+0000 7fb9cc2fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb9c415bd90 0x7fb9c408bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:53.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.499+0000 7fb9cc2fe640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb9c40ce600 0x7fb9c40cea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:53.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.499+0000 7fb9cc2fe640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9c40cf910 con 0x7fb9c40d06d0 2024-03-24T00:44:53.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.499+0000 7fb9cc2fe640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9c40cefb0 con 0x7fb9c415bd90 2024-03-24T00:44:53.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.499+0000 7fb9cc2fe640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c40d0b60 con 0x7fb9c40ce600 2024-03-24T00:44:53.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.499+0000 7fb9ca09a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb9c415bd90 0x7fb9c408bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:53.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.499+0000 7fb9c9899640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb9c40d06d0 0x7fb9c40b23e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55484/0 (socket says 172.21.15.67:55484) 2024-03-24T00:44:53.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.499+0000 7fb9c9899640 1 -- 172.21.15.67:0/3147287643 learned_addr learned my addr 172.21.15.67:0/3147287643 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:44:53.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.500+0000 7fb9ca89b640 1 --2- 172.21.15.67:0/3147287643 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb9c40ce600 0x7fb9c40cea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:53.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.500+0000 7fb9ca09a640 1 -- 172.21.15.67:0/3147287643 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb9c40ce600 msgr2=0x7fb9c40cea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:53.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.500+0000 7fb9ca09a640 1 --2- 172.21.15.67:0/3147287643 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb9c40ce600 0x7fb9c40cea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.500+0000 7fb9ca09a640 1 -- 172.21.15.67:0/3147287643 >> v1:172.21.15.67:6789/0 conn(0x7fb9c40d06d0 legacy=0x7fb9c40b23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:53.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.500+0000 7fb9c9098640 1 -- 172.21.15.67:0/3147287643 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 983870007 0 0) 0x7fb9c40cf910 con 0x7fb9c40d06d0 2024-03-24T00:44:53.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.500+0000 7fb9ca09a640 1 -- 172.21.15.67:0/3147287643 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c40cfbf0 con 0x7fb9c415bd90 2024-03-24T00:44:53.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.500+0000 7fb9ca09a640 1 --2- 172.21.15.67:0/3147287643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9c415bd90 0x7fb9c408bd20 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc0097d0 tx=0x7fb9bc02b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d7d168688b0bcdf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:53.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.501+0000 7fb9c9098640 1 -- 172.21.15.67:0/3147287643 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9bc031770 con 0x7fb9c415bd90 2024-03-24T00:44:53.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.501+0000 7fb9c9098640 1 -- 172.21.15.67:0/3147287643 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9bc037070 con 0x7fb9c415bd90 2024-03-24T00:44:53.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.501+0000 7fb9c9098640 1 -- 172.21.15.67:0/3147287643 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9bc031b30 con 0x7fb9c415bd90 2024-03-24T00:44:53.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.501+0000 7fb9cc2fe640 1 -- 172.21.15.67:0/3147287643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9c415bd90 msgr2=0x7fb9c408bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:53.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.501+0000 7fb9cc2fe640 1 --2- 172.21.15.67:0/3147287643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9c415bd90 0x7fb9c408bd20 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc0097d0 tx=0x7fb9bc02b810 comp rx=0 tx=0).stop 2024-03-24T00:44:53.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.501+0000 7fb9cc2fe640 1 -- 172.21.15.67:0/3147287643 shutdown_connections 2024-03-24T00:44:53.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.501+0000 7fb9cc2fe640 1 --2- 172.21.15.67:0/3147287643 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb9c40ce600 0x7fb9c40cea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.501+0000 7fb9cc2fe640 1 --2- 172.21.15.67:0/3147287643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9c415bd90 0x7fb9c408bd20 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.501+0000 7fb9cc2fe640 1 -- 172.21.15.67:0/3147287643 >> 172.21.15.67:0/3147287643 conn(0x7fb9c40d1c10 msgr2=0x7fb9c40d2020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:53.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.501+0000 7fb9cc2fe640 1 -- 172.21.15.67:0/3147287643 shutdown_connections 2024-03-24T00:44:53.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.501+0000 7fb9cc2fe640 1 -- 172.21.15.67:0/3147287643 wait complete. 2024-03-24T00:44:53.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.502+0000 7fb9cc2fe640 1 Processor -- start 2024-03-24T00:44:53.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.502+0000 7fb9cc2fe640 1 -- start start 2024-03-24T00:44:53.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.503+0000 7fb9cc2fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9c40ce600 0x7fb9c41eeac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:53.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.503+0000 7fb9cc2fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9c40d06d0 0x7fb9c40b69e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:53.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.503+0000 7fb9cc2fe640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb9c415bd90 0x7fb9c40b6f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:53.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.503+0000 7fb9cc2fe640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c40d0240 con 0x7fb9c40d06d0 2024-03-24T00:44:53.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.503+0000 7fb9cc2fe640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c40b2c10 con 0x7fb9c40ce600 2024-03-24T00:44:53.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.503+0000 7fb9cc2fe640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c415c170 con 0x7fb9c415bd90 2024-03-24T00:44:53.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.503+0000 7fb9c9899640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9c40d06d0 0x7fb9c40b69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:53.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.503+0000 7fb9c9899640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9c40d06d0 0x7fb9c40b69e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44950/0 (socket says 172.21.15.67:44950) 2024-03-24T00:44:53.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.503+0000 7fb9c9899640 1 -- 172.21.15.67:0/2615212400 learned_addr learned my addr 172.21.15.67:0/2615212400 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:53.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.503+0000 7fb9ca89b640 1 --2- 172.21.15.67:0/2615212400 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb9c415bd90 0x7fb9c40b6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:53.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.503+0000 7fb9ca09a640 1 --2- 172.21.15.67:0/2615212400 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9c40ce600 0x7fb9c41eeac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:53.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.504+0000 7fb9c9899640 1 -- 172.21.15.67:0/2615212400 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb9c415bd90 msgr2=0x7fb9c40b6f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:53.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.504+0000 7fb9c9899640 1 --2- 172.21.15.67:0/2615212400 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb9c415bd90 0x7fb9c40b6f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.504+0000 7fb9c9899640 1 -- 172.21.15.67:0/2615212400 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9c40ce600 msgr2=0x7fb9c41eeac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:53.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.504+0000 7fb9c9899640 1 --2- 172.21.15.67:0/2615212400 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9c40ce600 0x7fb9c41eeac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:53.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.504+0000 7fb9c9899640 1 -- 172.21.15.67:0/2615212400 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c40b75a0 con 0x7fb9c40d06d0 2024-03-24T00:44:53.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.504+0000 7fb9c9899640 1 --2- 172.21.15.67:0/2615212400 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9c40d06d0 0x7fb9c40b69e0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fb9b800c4a0 tx=0x7fb9b800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:53.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.504+0000 7fb9c2ffd640 1 -- 172.21.15.67:0/2615212400 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b8017070 con 0x7fb9c40d06d0 2024-03-24T00:44:53.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.504+0000 7fb9c2ffd640 1 -- 172.21.15.67:0/2615212400 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9b8005ce0 con 0x7fb9c40d06d0 2024-03-24T00:44:53.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.504+0000 7fb9cc2fe640 1 -- 172.21.15.67:0/2615212400 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9c40b7880 con 0x7fb9c40d06d0 2024-03-24T00:44:53.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.504+0000 7fb9c2ffd640 1 -- 172.21.15.67:0/2615212400 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b80072b0 con 0x7fb9c40d06d0 2024-03-24T00:44:53.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.504+0000 7fb9cc2fe640 1 -- 172.21.15.67:0/2615212400 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9c41f19d0 con 0x7fb9c40d06d0 2024-03-24T00:44:53.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.505+0000 7fb9cc2fe640 1 -- 172.21.15.67:0/2615212400 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9c40ca1e0 con 0x7fb9c40d06d0 2024-03-24T00:44:53.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.506+0000 7fb9c2ffd640 1 -- 172.21.15.67:0/2615212400 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb9b801a070 con 0x7fb9c40d06d0 2024-03-24T00:44:53.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.506+0000 7fb9c2ffd640 1 --2- 172.21.15.67:0/2615212400 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb9b006bbd0 0x7fb9b006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:53.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.506+0000 7fb9c2ffd640 1 -- 172.21.15.67:0/2615212400 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5619+0+0 (secure 0 0 0) 0x7fb9b8089da0 con 0x7fb9c40d06d0 2024-03-24T00:44:53.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.506+0000 7fb9ca09a640 1 --2- 172.21.15.67:0/2615212400 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb9b006bbd0 0x7fb9b006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:53.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.507+0000 7fb9ca09a640 1 --2- 172.21.15.67:0/2615212400 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb9b006bbd0 0x7fb9b006e090 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc036000 tx=0x7fb9bc03d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:53.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.511+0000 7fb9c2ffd640 1 -- 172.21.15.67:0/2615212400 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb9b8058210 con 0x7fb9c40d06d0 2024-03-24T00:44:53.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:53.694+0000 7fb9cc2fe640 1 -- 172.21.15.67:0/2615212400 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fb9c404f400 con 0x7fb9c40d06d0 2024-03-24T00:44:54.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.375+0000 7fb9c2ffd640 1 -- 172.21.15.67:0/2615212400 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v87) v1 ==== 170+0+0 (secure 0 0 0) 0x7fb9b805c2b0 con 0x7fb9c40d06d0 2024-03-24T00:44:54.375 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' created 2024-03-24T00:44:54.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.378+0000 7fb9cc2fe640 1 -- 172.21.15.67:0/2615212400 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb9b006bbd0 msgr2=0x7fb9b006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:54.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.378+0000 7fb9cc2fe640 1 --2- 172.21.15.67:0/2615212400 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb9b006bbd0 0x7fb9b006e090 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc036000 tx=0x7fb9bc03d040 comp rx=0 tx=0).stop 2024-03-24T00:44:54.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.379+0000 7fb9cc2fe640 1 -- 172.21.15.67:0/2615212400 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9c40d06d0 msgr2=0x7fb9c40b69e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:54.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.379+0000 7fb9cc2fe640 1 --2- 172.21.15.67:0/2615212400 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9c40d06d0 0x7fb9c40b69e0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fb9b800c4a0 tx=0x7fb9b800c960 comp rx=0 tx=0).stop 2024-03-24T00:44:54.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.380+0000 7fb9cc2fe640 1 -- 172.21.15.67:0/2615212400 shutdown_connections 2024-03-24T00:44:54.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.380+0000 7fb9cc2fe640 1 --2- 172.21.15.67:0/2615212400 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb9b006bbd0 0x7fb9b006e090 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:54.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.380+0000 7fb9cc2fe640 1 --2- 172.21.15.67:0/2615212400 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb9c415bd90 0x7fb9c40b6f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:54.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.380+0000 7fb9cc2fe640 1 --2- 172.21.15.67:0/2615212400 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9c40d06d0 0x7fb9c40b69e0 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:54.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.380+0000 7fb9cc2fe640 1 --2- 172.21.15.67:0/2615212400 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9c40ce600 0x7fb9c41eeac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:54.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.381+0000 7fb9cc2fe640 1 -- 172.21.15.67:0/2615212400 >> 172.21.15.67:0/2615212400 conn(0x7fb9c40d1c10 msgr2=0x7fb9c415d870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:54.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.390+0000 7fb9cc2fe640 1 -- 172.21.15.67:0/2615212400 shutdown_connections 2024-03-24T00:44:54.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.390+0000 7fb9cc2fe640 1 -- 172.21.15.67:0/2615212400 wait complete. 2024-03-24T00:44:54.408 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-24T00:44:54.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.516+0000 7fd9ebcc6640 1 Processor -- start 2024-03-24T00:44:54.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.516+0000 7fd9ebcc6640 1 -- start start 2024-03-24T00:44:54.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.517+0000 7fd9ebcc6640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd9e415dfd0 0x7fd9e40cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:54.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.517+0000 7fd9ebcc6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd9e40b41e0 0x7fd9e40b3620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:54.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.517+0000 7fd9ebcc6640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9e40cf090 con 0x7fd9e40b3b60 2024-03-24T00:44:54.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.517+0000 7fd9ebcc6640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9e40cc260 con 0x7fd9e415dfd0 2024-03-24T00:44:54.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.517+0000 7fd9ebcc6640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e40d0320 con 0x7fd9e40b41e0 2024-03-24T00:44:54.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.517+0000 7fd9e9a62640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd9e415dfd0 0x7fd9e40cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:54.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.517+0000 7fd9e9a62640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd9e415dfd0 0x7fd9e40cf980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44964/0 (socket says 172.21.15.67:44964) 2024-03-24T00:44:54.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.517+0000 7fd9e9a62640 1 -- 172.21.15.67:0/2965763045 learned_addr learned my addr 172.21.15.67:0/2965763045 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:54.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.517+0000 7fd9e9261640 1 --2- 172.21.15.67:0/2965763045 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd9e40b41e0 0x7fd9e40b3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:54.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.518+0000 7fd9e9a62640 1 -- 172.21.15.67:0/2965763045 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd9e40b41e0 msgr2=0x7fd9e40b3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:54.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.518+0000 7fd9e9a62640 1 --2- 172.21.15.67:0/2965763045 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd9e40b41e0 0x7fd9e40b3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:54.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.518+0000 7fd9e9a62640 1 -- 172.21.15.67:0/2965763045 >> v1:172.21.15.72:6789/0 conn(0x7fd9e40b3b60 legacy=0x7fd9e4161b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:54.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.518+0000 7fd9e9a62640 1 -- 172.21.15.67:0/2965763045 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e40d0600 con 0x7fd9e415dfd0 2024-03-24T00:44:54.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.518+0000 7fd9e9261640 1 --2- 172.21.15.67:0/2965763045 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd9e40b41e0 0x7fd9e40b3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:44:54.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.518+0000 7fd9e9a62640 1 --2- 172.21.15.67:0/2965763045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9e415dfd0 0x7fd9e40cf980 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc009870 tx=0x7fd9dc02ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=318dcd1862f188b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:54.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.518+0000 7fd9e8a60640 1 -- 172.21.15.67:0/2965763045 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc02f840 con 0x7fd9e415dfd0 2024-03-24T00:44:54.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.519+0000 7fd9e8a60640 1 -- 172.21.15.67:0/2965763045 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9dc02f9e0 con 0x7fd9e415dfd0 2024-03-24T00:44:54.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.519+0000 7fd9e8a60640 1 -- 172.21.15.67:0/2965763045 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc02fdb0 con 0x7fd9e415dfd0 2024-03-24T00:44:54.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.519+0000 7fd9ebcc6640 1 -- 172.21.15.67:0/2965763045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9e415dfd0 msgr2=0x7fd9e40cf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:54.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.519+0000 7fd9ebcc6640 1 --2- 172.21.15.67:0/2965763045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9e415dfd0 0x7fd9e40cf980 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc009870 tx=0x7fd9dc02ba20 comp rx=0 tx=0).stop 2024-03-24T00:44:54.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.519+0000 7fd9ebcc6640 1 -- 172.21.15.67:0/2965763045 shutdown_connections 2024-03-24T00:44:54.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.519+0000 7fd9ebcc6640 1 --2- 172.21.15.67:0/2965763045 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd9e40b41e0 0x7fd9e40b3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:54.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.519+0000 7fd9ebcc6640 1 --2- 172.21.15.67:0/2965763045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9e415dfd0 0x7fd9e40cf980 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:54.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.519+0000 7fd9ebcc6640 1 -- 172.21.15.67:0/2965763045 >> 172.21.15.67:0/2965763045 conn(0x7fd9e406fcf0 msgr2=0x7fd9e40700d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:54.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.519+0000 7fd9ebcc6640 1 -- 172.21.15.67:0/2965763045 shutdown_connections 2024-03-24T00:44:54.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.519+0000 7fd9ebcc6640 1 -- 172.21.15.67:0/2965763045 wait complete. 2024-03-24T00:44:54.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.520+0000 7fd9ebcc6640 1 Processor -- start 2024-03-24T00:44:54.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.520+0000 7fd9ebcc6640 1 -- start start 2024-03-24T00:44:54.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.521+0000 7fd9ebcc6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9e40b3b60 0x7fd9e4155170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:54.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.521+0000 7fd9ebcc6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd9e40b41e0 0x7fd9e41559b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:54.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.521+0000 7fd9ebcc6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9e415dfd0 0x7fd9e40ca210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:54.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.521+0000 7fd9ebcc6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e40cfec0 con 0x7fd9e415dfd0 2024-03-24T00:44:54.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.521+0000 7fd9ebcc6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e40cedb0 con 0x7fd9e40b3b60 2024-03-24T00:44:54.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.521+0000 7fd9ebcc6640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e415e3b0 con 0x7fd9e40b41e0 2024-03-24T00:44:54.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.521+0000 7fd9ea263640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9e415dfd0 0x7fd9e40ca210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:54.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.521+0000 7fd9ea263640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9e415dfd0 0x7fd9e40ca210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44978/0 (socket says 172.21.15.67:44978) 2024-03-24T00:44:54.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.521+0000 7fd9ea263640 1 -- 172.21.15.67:0/4041801680 learned_addr learned my addr 172.21.15.67:0/4041801680 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:54.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.521+0000 7fd9e9261640 1 --2- 172.21.15.67:0/4041801680 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd9e40b41e0 0x7fd9e41559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:54.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.521+0000 7fd9e9a62640 1 --2- 172.21.15.67:0/4041801680 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9e40b3b60 0x7fd9e4155170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:54.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.521+0000 7fd9ea263640 1 -- 172.21.15.67:0/4041801680 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd9e40b41e0 msgr2=0x7fd9e41559b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:54.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.521+0000 7fd9ea263640 1 --2- 172.21.15.67:0/4041801680 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd9e40b41e0 0x7fd9e41559b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:54.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.521+0000 7fd9ea263640 1 -- 172.21.15.67:0/4041801680 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9e40b3b60 msgr2=0x7fd9e4155170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:54.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.521+0000 7fd9ea263640 1 --2- 172.21.15.67:0/4041801680 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9e40b3b60 0x7fd9e4155170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:54.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.521+0000 7fd9ea263640 1 -- 172.21.15.67:0/4041801680 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e415ede0 con 0x7fd9e415dfd0 2024-03-24T00:44:54.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.522+0000 7fd9ea263640 1 --2- 172.21.15.67:0/4041801680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9e415dfd0 0x7fd9e40ca210 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fd9d400c7a0 tx=0x7fd9d400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:54.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.522+0000 7fd9e27fc640 1 -- 172.21.15.67:0/4041801680 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d4003b70 con 0x7fd9e415dfd0 2024-03-24T00:44:54.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.522+0000 7fd9e27fc640 1 -- 172.21.15.67:0/4041801680 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9d4003d30 con 0x7fd9e415dfd0 2024-03-24T00:44:54.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.522+0000 7fd9ebcc6640 1 -- 172.21.15.67:0/4041801680 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9e415f0c0 con 0x7fd9e415dfd0 2024-03-24T00:44:54.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.522+0000 7fd9e27fc640 1 -- 172.21.15.67:0/4041801680 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d400aa90 con 0x7fd9e415dfd0 2024-03-24T00:44:54.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.522+0000 7fd9ebcc6640 1 -- 172.21.15.67:0/4041801680 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9e415f680 con 0x7fd9e415dfd0 2024-03-24T00:44:54.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.523+0000 7fd9ebcc6640 1 -- 172.21.15.67:0/4041801680 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9e415f880 con 0x7fd9e415dfd0 2024-03-24T00:44:54.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.523+0000 7fd9e27fc640 1 -- 172.21.15.67:0/4041801680 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd9d400ac30 con 0x7fd9e415dfd0 2024-03-24T00:44:54.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.524+0000 7fd9e27fc640 1 --2- 172.21.15.67:0/4041801680 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd9d006bbd0 0x7fd9d006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:54.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.524+0000 7fd9e27fc640 1 -- 172.21.15.67:0/4041801680 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6001+0+0 (secure 0 0 0) 0x7fd9d408a970 con 0x7fd9e415dfd0 2024-03-24T00:44:54.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.524+0000 7fd9e9a62640 1 --2- 172.21.15.67:0/4041801680 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd9d006bbd0 0x7fd9d006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:54.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.524+0000 7fd9e9a62640 1 --2- 172.21.15.67:0/4041801680 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd9d006bbd0 0x7fd9d006e090 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc036000 tx=0x7fd9dc008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:54.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.528+0000 7fd9e27fc640 1 -- 172.21.15.67:0/4041801680 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd9d4058ce0 con 0x7fd9e415dfd0 2024-03-24T00:44:54.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:54.748+0000 7fd9ebcc6640 1 -- 172.21.15.67:0/4041801680 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fd9e404f400 con 0x7fd9e415dfd0 2024-03-24T00:44:55.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.376+0000 7fd9e27fc640 1 -- 172.21.15.67:0/4041801680 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 13 and data pool 14 v52) v1 ==== 174+0+0 (secure 0 0 0) 0x7fd9d405cd80 con 0x7fd9e415dfd0 2024-03-24T00:44:55.376 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 13 and data pool 14 2024-03-24T00:44:55.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.380+0000 7fd9ebcc6640 1 -- 172.21.15.67:0/4041801680 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd9d006bbd0 msgr2=0x7fd9d006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:55.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.380+0000 7fd9ebcc6640 1 --2- 172.21.15.67:0/4041801680 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd9d006bbd0 0x7fd9d006e090 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc036000 tx=0x7fd9dc008030 comp rx=0 tx=0).stop 2024-03-24T00:44:55.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.380+0000 7fd9ebcc6640 1 -- 172.21.15.67:0/4041801680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9e415dfd0 msgr2=0x7fd9e40ca210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:55.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.380+0000 7fd9ebcc6640 1 --2- 172.21.15.67:0/4041801680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9e415dfd0 0x7fd9e40ca210 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fd9d400c7a0 tx=0x7fd9d400cc60 comp rx=0 tx=0).stop 2024-03-24T00:44:55.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.380+0000 7fd9ebcc6640 1 -- 172.21.15.67:0/4041801680 shutdown_connections 2024-03-24T00:44:55.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.380+0000 7fd9ebcc6640 1 --2- 172.21.15.67:0/4041801680 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd9d006bbd0 0x7fd9d006e090 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:55.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.380+0000 7fd9ebcc6640 1 --2- 172.21.15.67:0/4041801680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9e415dfd0 0x7fd9e40ca210 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:55.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.380+0000 7fd9ebcc6640 1 --2- 172.21.15.67:0/4041801680 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd9e40b41e0 0x7fd9e41559b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:55.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.380+0000 7fd9ebcc6640 1 --2- 172.21.15.67:0/4041801680 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9e40b3b60 0x7fd9e4155170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:55.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.380+0000 7fd9ebcc6640 1 -- 172.21.15.67:0/4041801680 >> 172.21.15.67:0/4041801680 conn(0x7fd9e406fcf0 msgr2=0x7fd9e40b0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:55.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.381+0000 7fd9ebcc6640 1 -- 172.21.15.67:0/4041801680 shutdown_connections 2024-03-24T00:44:55.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.381+0000 7fd9ebcc6640 1 -- 172.21.15.67:0/4041801680 wait complete. 2024-03-24T00:44:55.415 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:44:55.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.518+0000 7f33e7fc4640 1 Processor -- start 2024-03-24T00:44:55.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.518+0000 7f33e7fc4640 1 -- start start 2024-03-24T00:44:55.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.519+0000 7f33e7fc4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f33e00b5ac0 0x7f33e00b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:55.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.519+0000 7f33e7fc4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f33e00b2770 0x7f33e00b5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:55.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.519+0000 7f33e7fc4640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33e00701e0 con 0x7f33e015e130 2024-03-24T00:44:55.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.519+0000 7f33e7fc4640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33e00d00b0 con 0x7f33e00b5ac0 2024-03-24T00:44:55.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.519+0000 7f33e7fc4640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33e00ce510 con 0x7f33e00b2770 2024-03-24T00:44:55.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.519+0000 7f33e555f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f33e00b5ac0 0x7f33e00b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:55.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.519+0000 7f33e5d60640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f33e015e130 0x7f33e00cde00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55494/0 (socket says 172.21.15.67:55494) 2024-03-24T00:44:55.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.519+0000 7f33e5d60640 1 -- 172.21.15.67:0/3422054630 learned_addr learned my addr 172.21.15.67:0/3422054630 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:44:55.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.519+0000 7f33e555f640 1 -- 172.21.15.67:0/3422054630 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f33e00b2770 msgr2=0x7f33e00b5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:55.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.519+0000 7f33e555f640 1 --2- 172.21.15.67:0/3422054630 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f33e00b2770 0x7f33e00b5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:55.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.519+0000 7f33e555f640 1 -- 172.21.15.67:0/3422054630 >> v1:172.21.15.67:6789/0 conn(0x7f33e015e130 legacy=0x7f33e00cde00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:55.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.519+0000 7f33e555f640 1 -- 172.21.15.67:0/3422054630 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33e006b170 con 0x7f33e00b5ac0 2024-03-24T00:44:55.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.519+0000 7f33e4d5e640 1 -- 172.21.15.67:0/3422054630 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2407757678 0 0) 0x7f33e00701e0 con 0x7f33e015e130 2024-03-24T00:44:55.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.520+0000 7f33e555f640 1 --2- 172.21.15.67:0/3422054630 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33e00b5ac0 0x7f33e00b4f20 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f33d40097a0 tx=0x7f33d402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d76927e313d21e10 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:55.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.520+0000 7f33e4d5e640 1 -- 172.21.15.67:0/3422054630 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33d40315a0 con 0x7f33e00b5ac0 2024-03-24T00:44:55.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.520+0000 7f33e4d5e640 1 -- 172.21.15.67:0/3422054630 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33d4037070 con 0x7f33e00b5ac0 2024-03-24T00:44:55.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.520+0000 7f33e4d5e640 1 -- 172.21.15.67:0/3422054630 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33d4031960 con 0x7f33e00b5ac0 2024-03-24T00:44:55.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.521+0000 7f33e7fc4640 1 -- 172.21.15.67:0/3422054630 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33e00b5ac0 msgr2=0x7f33e00b4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:55.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.521+0000 7f33e7fc4640 1 --2- 172.21.15.67:0/3422054630 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33e00b5ac0 0x7f33e00b4f20 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f33d40097a0 tx=0x7f33d402b840 comp rx=0 tx=0).stop 2024-03-24T00:44:55.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.521+0000 7f33e7fc4640 1 -- 172.21.15.67:0/3422054630 shutdown_connections 2024-03-24T00:44:55.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.521+0000 7f33e7fc4640 1 --2- 172.21.15.67:0/3422054630 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f33e00b2770 0x7f33e00b5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:55.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.521+0000 7f33e7fc4640 1 --2- 172.21.15.67:0/3422054630 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33e00b5ac0 0x7f33e00b4f20 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:55.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.521+0000 7f33e7fc4640 1 -- 172.21.15.67:0/3422054630 >> 172.21.15.67:0/3422054630 conn(0x7f33e00b3c40 msgr2=0x7f33e00b1dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:55.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.521+0000 7f33e7fc4640 1 -- 172.21.15.67:0/3422054630 shutdown_connections 2024-03-24T00:44:55.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.521+0000 7f33e7fc4640 1 -- 172.21.15.67:0/3422054630 wait complete. 2024-03-24T00:44:55.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.522+0000 7f33e7fc4640 1 Processor -- start 2024-03-24T00:44:55.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.522+0000 7f33e7fc4640 1 -- start start 2024-03-24T00:44:55.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.523+0000 7f33e7fc4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f33e00b2770 0x7f33e00ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:55.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.523+0000 7f33e7fc4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f33e00b5ac0 0x7f33e015e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:55.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.523+0000 7f33e7fc4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33e015e130 0x7f33e0152450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:55.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.523+0000 7f33e7fc4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33e006b7c0 con 0x7f33e015e130 2024-03-24T00:44:55.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.523+0000 7f33e7fc4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33e00704c0 con 0x7f33e00b2770 2024-03-24T00:44:55.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.523+0000 7f33e7fc4640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33e015e510 con 0x7f33e00b5ac0 2024-03-24T00:44:55.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.523+0000 7f33e6561640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33e015e130 0x7f33e0152450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:55.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.523+0000 7f33e6561640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33e015e130 0x7f33e0152450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44998/0 (socket says 172.21.15.67:44998) 2024-03-24T00:44:55.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.523+0000 7f33e6561640 1 -- 172.21.15.67:0/3729155368 learned_addr learned my addr 172.21.15.67:0/3729155368 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:55.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.523+0000 7f33e555f640 1 --2- 172.21.15.67:0/3729155368 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f33e00b5ac0 0x7f33e015e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:55.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.523+0000 7f33e5d60640 1 --2- 172.21.15.67:0/3729155368 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f33e00b2770 0x7f33e00ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:55.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.523+0000 7f33e6561640 1 -- 172.21.15.67:0/3729155368 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f33e00b5ac0 msgr2=0x7f33e015e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:55.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.523+0000 7f33e6561640 1 --2- 172.21.15.67:0/3729155368 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f33e00b5ac0 0x7f33e015e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:55.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.523+0000 7f33e6561640 1 -- 172.21.15.67:0/3729155368 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f33e00b2770 msgr2=0x7f33e00ccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:55.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.523+0000 7f33e6561640 1 --2- 172.21.15.67:0/3729155368 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f33e00b2770 0x7f33e00ccab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:55.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.523+0000 7f33e6561640 1 -- 172.21.15.67:0/3729155368 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33e00cd120 con 0x7f33e015e130 2024-03-24T00:44:55.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.523+0000 7f33e555f640 1 --2- 172.21.15.67:0/3729155368 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f33e00b5ac0 0x7f33e015e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:44:55.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.523+0000 7f33e5d60640 1 --2- 172.21.15.67:0/3729155368 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f33e00b2770 0x7f33e00ccab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:44:55.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.523+0000 7f33e6561640 1 --2- 172.21.15.67:0/3729155368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33e015e130 0x7f33e0152450 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f33d800b3f0 tx=0x7f33d800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:55.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.524+0000 7f33de7fc640 1 -- 172.21.15.67:0/3729155368 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33d8015020 con 0x7f33e015e130 2024-03-24T00:44:55.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.524+0000 7f33de7fc640 1 -- 172.21.15.67:0/3729155368 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33d8011850 con 0x7f33e015e130 2024-03-24T00:44:55.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.524+0000 7f33e7fc4640 1 -- 172.21.15.67:0/3729155368 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33e00c8ff0 con 0x7f33e015e130 2024-03-24T00:44:55.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.524+0000 7f33de7fc640 1 -- 172.21.15.67:0/3729155368 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33d8011c40 con 0x7f33e015e130 2024-03-24T00:44:55.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.524+0000 7f33e7fc4640 1 -- 172.21.15.67:0/3729155368 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33e00c95b0 con 0x7f33e015e130 2024-03-24T00:44:55.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.525+0000 7f33e7fc4640 1 -- 172.21.15.67:0/3729155368 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33e004f400 con 0x7f33e015e130 2024-03-24T00:44:55.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.525+0000 7f33de7fc640 1 -- 172.21.15.67:0/3729155368 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f33d8018020 con 0x7f33e015e130 2024-03-24T00:44:55.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.526+0000 7f33de7fc640 1 --2- 172.21.15.67:0/3729155368 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f33cc06bc00 0x7f33cc06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:55.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.526+0000 7f33de7fc640 1 -- 172.21.15.67:0/3729155368 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f33d808aae0 con 0x7f33e015e130 2024-03-24T00:44:55.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.526+0000 7f33e5d60640 1 --2- 172.21.15.67:0/3729155368 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f33cc06bc00 0x7f33cc06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:55.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.526+0000 7f33e5d60640 1 --2- 172.21.15.67:0/3729155368 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f33cc06bc00 0x7f33cc06e0c0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f33d0007fd0 tx=0x7f33d0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:55.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.530+0000 7f33de7fc640 1 -- 172.21.15.67:0/3729155368 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f33d801f020 con 0x7f33e015e130 2024-03-24T00:44:55.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.714+0000 7f33e7fc4640 1 -- 172.21.15.67:0/3729155368 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f33e015a790 con 0x7f33e015e130 2024-03-24T00:44:55.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.715+0000 7f33de7fc640 1 -- 172.21.15.67:0/3729155368 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+16838 (secure 0 0 0) 0x7f33d80172c0 con 0x7f33e015e130 2024-03-24T00:44:55.715 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:44:55.715 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":88,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:44:54.749345+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:44:53.025965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-03-24T00:44:53.712957+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:44:55.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.717+0000 7f33e7fc4640 1 -- 172.21.15.67:0/3729155368 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f33cc06bc00 msgr2=0x7f33cc06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:55.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.717+0000 7f33e7fc4640 1 --2- 172.21.15.67:0/3729155368 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f33cc06bc00 0x7f33cc06e0c0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f33d0007fd0 tx=0x7f33d0009040 comp rx=0 tx=0).stop 2024-03-24T00:44:55.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.717+0000 7f33e7fc4640 1 -- 172.21.15.67:0/3729155368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33e015e130 msgr2=0x7f33e0152450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:55.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.717+0000 7f33e7fc4640 1 --2- 172.21.15.67:0/3729155368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33e015e130 0x7f33e0152450 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f33d800b3f0 tx=0x7f33d800b8b0 comp rx=0 tx=0).stop 2024-03-24T00:44:55.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.717+0000 7f33e7fc4640 1 -- 172.21.15.67:0/3729155368 shutdown_connections 2024-03-24T00:44:55.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.717+0000 7f33e7fc4640 1 --2- 172.21.15.67:0/3729155368 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f33cc06bc00 0x7f33cc06e0c0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:55.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.717+0000 7f33e7fc4640 1 --2- 172.21.15.67:0/3729155368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33e015e130 0x7f33e0152450 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:55.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.717+0000 7f33e7fc4640 1 --2- 172.21.15.67:0/3729155368 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f33e00b5ac0 0x7f33e015e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:55.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.717+0000 7f33e7fc4640 1 --2- 172.21.15.67:0/3729155368 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f33e00b2770 0x7f33e00ccab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:55.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.717+0000 7f33e7fc4640 1 -- 172.21.15.67:0/3729155368 >> 172.21.15.67:0/3729155368 conn(0x7f33e00b3c40 msgr2=0x7f33e00afd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:55.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.717+0000 7f33e7fc4640 1 -- 172.21.15.67:0/3729155368 shutdown_connections 2024-03-24T00:44:55.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.717+0000 7f33e7fc4640 1 -- 172.21.15.67:0/3729155368 wait complete. 2024-03-24T00:44:55.728 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:44:55.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.830+0000 7f66a72dc640 1 Processor -- start 2024-03-24T00:44:55.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.830+0000 7f66a72dc640 1 -- start start 2024-03-24T00:44:55.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.831+0000 7f66a72dc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f66a00cfc70 0x7f66a00ce220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:55.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.831+0000 7f66a72dc640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66a00ccfe0 con 0x7f66a00ce760 2024-03-24T00:44:55.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.831+0000 7f66a72dc640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66a00cd2c0 con 0x7f66a015bf70 2024-03-24T00:44:55.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.831+0000 7f66a72dc640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66a00ccda0 con 0x7f66a00cfc70 2024-03-24T00:44:55.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.831+0000 7f66a4877640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f66a00cfc70 0x7f66a00ce220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:55.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.831+0000 7f66a4877640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f66a00cfc70 0x7f66a00ce220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45006/0 (socket says 172.21.15.67:45006) 2024-03-24T00:44:55.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.831+0000 7f66a4877640 1 -- 172.21.15.67:0/2788242434 learned_addr learned my addr 172.21.15.67:0/2788242434 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:55.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.832+0000 7f66a4877640 1 -- 172.21.15.67:0/2788242434 >> v1:172.21.15.72:6789/0 conn(0x7f66a015bf70 legacy=0x7f66a00cc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:55.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.832+0000 7f66a4877640 1 -- 172.21.15.67:0/2788242434 >> v1:172.21.15.67:6789/0 conn(0x7f66a00ce760 legacy=0x7f66a00b1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:55.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.832+0000 7f669ffff640 1 -- 172.21.15.67:0/2788242434 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3415682532 0 0) 0x7f66a00ccfe0 con 0x7f66a00ce760 2024-03-24T00:44:55.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.832+0000 7f66a4877640 1 -- 172.21.15.67:0/2788242434 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a00cd5a0 con 0x7f66a00cfc70 2024-03-24T00:44:55.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.832+0000 7f66a4877640 1 --2- 172.21.15.67:0/2788242434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66a00cfc70 0x7f66a00ce220 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f6694009870 tx=0x7f669402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3ad4086581ce3ff server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:55.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.832+0000 7f669ffff640 1 -- 172.21.15.67:0/2788242434 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6694034070 con 0x7f66a00cfc70 2024-03-24T00:44:55.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.833+0000 7f669ffff640 1 -- 172.21.15.67:0/2788242434 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6694030860 con 0x7f66a00cfc70 2024-03-24T00:44:55.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.833+0000 7f669ffff640 1 -- 172.21.15.67:0/2788242434 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6694030c30 con 0x7f66a00cfc70 2024-03-24T00:44:55.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.833+0000 7f66a72dc640 1 -- 172.21.15.67:0/2788242434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66a00cfc70 msgr2=0x7f66a00ce220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:55.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.833+0000 7f66a72dc640 1 --2- 172.21.15.67:0/2788242434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66a00cfc70 0x7f66a00ce220 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f6694009870 tx=0x7f669402b9a0 comp rx=0 tx=0).stop 2024-03-24T00:44:55.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.833+0000 7f66a72dc640 1 -- 172.21.15.67:0/2788242434 shutdown_connections 2024-03-24T00:44:55.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.833+0000 7f66a72dc640 1 --2- 172.21.15.67:0/2788242434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66a00cfc70 0x7f66a00ce220 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:55.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.833+0000 7f66a72dc640 1 -- 172.21.15.67:0/2788242434 >> 172.21.15.67:0/2788242434 conn(0x7f66a00d1bf0 msgr2=0x7f66a00d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:55.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.833+0000 7f66a72dc640 1 -- 172.21.15.67:0/2788242434 shutdown_connections 2024-03-24T00:44:55.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.833+0000 7f66a72dc640 1 -- 172.21.15.67:0/2788242434 wait complete. 2024-03-24T00:44:55.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.834+0000 7f66a72dc640 1 Processor -- start 2024-03-24T00:44:55.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.834+0000 7f66a72dc640 1 -- start start 2024-03-24T00:44:55.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.835+0000 7f66a72dc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66a00ce760 0x7f66a01ef0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:55.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.835+0000 7f66a72dc640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f66a00cfc70 0x7f66a00cacd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:55.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.835+0000 7f66a72dc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66a015bf70 0x7f66a00cb210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:55.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.835+0000 7f66a72dc640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a00b23e0 con 0x7f66a00ce760 2024-03-24T00:44:55.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.835+0000 7f66a72dc640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a015c350 con 0x7f66a015bf70 2024-03-24T00:44:55.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.835+0000 7f66a72dc640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66a00b2560 con 0x7f66a00cfc70 2024-03-24T00:44:55.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.835+0000 7f66a5078640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66a00ce760 0x7f66a01ef0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:55.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.835+0000 7f66a5078640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66a00ce760 0x7f66a01ef0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45008/0 (socket says 172.21.15.67:45008) 2024-03-24T00:44:55.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.835+0000 7f66a5078640 1 -- 172.21.15.67:0/358204011 learned_addr learned my addr 172.21.15.67:0/358204011 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:55.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.835+0000 7f66a4877640 1 --2- 172.21.15.67:0/358204011 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f66a00cfc70 0x7f66a00cacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:55.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.835+0000 7f66a5879640 1 --2- 172.21.15.67:0/358204011 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66a015bf70 0x7f66a00cb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:55.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.835+0000 7f66a5078640 1 -- 172.21.15.67:0/358204011 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f66a00cfc70 msgr2=0x7f66a00cacd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:55.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.835+0000 7f66a5078640 1 --2- 172.21.15.67:0/358204011 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f66a00cfc70 0x7f66a00cacd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:55.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.835+0000 7f66a5078640 1 -- 172.21.15.67:0/358204011 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66a015bf70 msgr2=0x7f66a00cb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:55.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.835+0000 7f66a5078640 1 --2- 172.21.15.67:0/358204011 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66a015bf70 0x7f66a00cb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:55.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.835+0000 7f66a5078640 1 -- 172.21.15.67:0/358204011 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a01f5e30 con 0x7f66a00ce760 2024-03-24T00:44:55.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.835+0000 7f66a4877640 1 --2- 172.21.15.67:0/358204011 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f66a00cfc70 0x7f66a00cacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:44:55.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.835+0000 7f66a5879640 1 --2- 172.21.15.67:0/358204011 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66a015bf70 0x7f66a00cb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:44:55.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.836+0000 7f66a5078640 1 --2- 172.21.15.67:0/358204011 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66a00ce760 0x7f66a01ef0c0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f669000ba50 tx=0x7f669000bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:55.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.836+0000 7f669dffb640 1 -- 172.21.15.67:0/358204011 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6690019070 con 0x7f66a00ce760 2024-03-24T00:44:55.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.836+0000 7f669dffb640 1 -- 172.21.15.67:0/358204011 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f669000cc20 con 0x7f66a00ce760 2024-03-24T00:44:55.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.836+0000 7f66a72dc640 1 -- 172.21.15.67:0/358204011 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66a01f1d40 con 0x7f66a00ce760 2024-03-24T00:44:55.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.836+0000 7f669dffb640 1 -- 172.21.15.67:0/358204011 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6690014680 con 0x7f66a00ce760 2024-03-24T00:44:55.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.836+0000 7f66a72dc640 1 -- 172.21.15.67:0/358204011 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66a01f2820 con 0x7f66a00ce760 2024-03-24T00:44:55.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.837+0000 7f66a72dc640 1 -- 172.21.15.67:0/358204011 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66a01f1b30 con 0x7f66a00ce760 2024-03-24T00:44:55.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.837+0000 7f669dffb640 1 -- 172.21.15.67:0/358204011 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f669001c070 con 0x7f66a00ce760 2024-03-24T00:44:55.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.838+0000 7f669dffb640 1 --2- 172.21.15.67:0/358204011 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f668c06bbd0 0x7f668c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:55.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.838+0000 7f669dffb640 1 -- 172.21.15.67:0/358204011 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f669008a900 con 0x7f66a00ce760 2024-03-24T00:44:55.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.838+0000 7f66a4877640 1 --2- 172.21.15.67:0/358204011 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f668c06bbd0 0x7f668c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:55.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.838+0000 7f66a4877640 1 --2- 172.21.15.67:0/358204011 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f668c06bbd0 0x7f668c06e090 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f6694036000 tx=0x7f669403b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:55.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:55.842+0000 7f669dffb640 1 -- 172.21.15.67:0/358204011 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6690058b80 con 0x7f66a00ce760 2024-03-24T00:44:56.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.025+0000 7f66a72dc640 1 -- 172.21.15.67:0/358204011 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f66a004f400 con 0x7f66a00ce760 2024-03-24T00:44:56.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.026+0000 7f669dffb640 1 -- 172.21.15.67:0/358204011 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+16838 (secure 0 0 0) 0x7f669005cc20 con 0x7f66a00ce760 2024-03-24T00:44:56.026 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:44:56.026 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":88,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:44:54.749345+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:44:53.025965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-03-24T00:44:53.712957+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:44:56.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.028+0000 7f66a72dc640 1 -- 172.21.15.67:0/358204011 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f668c06bbd0 msgr2=0x7f668c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:56.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.028+0000 7f66a72dc640 1 --2- 172.21.15.67:0/358204011 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f668c06bbd0 0x7f668c06e090 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f6694036000 tx=0x7f669403b040 comp rx=0 tx=0).stop 2024-03-24T00:44:56.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.028+0000 7f66a72dc640 1 -- 172.21.15.67:0/358204011 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66a00ce760 msgr2=0x7f66a01ef0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:56.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.028+0000 7f66a72dc640 1 --2- 172.21.15.67:0/358204011 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66a00ce760 0x7f66a01ef0c0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f669000ba50 tx=0x7f669000bf10 comp rx=0 tx=0).stop 2024-03-24T00:44:56.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.028+0000 7f66a72dc640 1 -- 172.21.15.67:0/358204011 shutdown_connections 2024-03-24T00:44:56.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.028+0000 7f66a72dc640 1 --2- 172.21.15.67:0/358204011 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f668c06bbd0 0x7f668c06e090 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:56.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.028+0000 7f66a72dc640 1 --2- 172.21.15.67:0/358204011 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66a015bf70 0x7f66a00cb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:56.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.028+0000 7f66a72dc640 1 --2- 172.21.15.67:0/358204011 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f66a00cfc70 0x7f66a00cacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:56.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.028+0000 7f66a72dc640 1 --2- 172.21.15.67:0/358204011 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66a00ce760 0x7f66a01ef0c0 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:56.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.028+0000 7f66a72dc640 1 -- 172.21.15.67:0/358204011 >> 172.21.15.67:0/358204011 conn(0x7f66a00d1bf0 msgr2=0x7f66a00b1070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:56.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.028+0000 7f66a72dc640 1 -- 172.21.15.67:0/358204011 shutdown_connections 2024-03-24T00:44:56.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.028+0000 7f66a72dc640 1 -- 172.21.15.67:0/358204011 wait complete. 2024-03-24T00:44:56.039 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-24T00:44:56.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.143+0000 7f4789868640 1 Processor -- start 2024-03-24T00:44:56.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.143+0000 7f4789868640 1 -- start start 2024-03-24T00:44:56.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.144+0000 7f4789868640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f478415dea0 0x7f47840d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:56.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.144+0000 7f4789868640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f47840b6170 0x7f47840d3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:56.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.144+0000 7f4789868640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47840d2250 con 0x7f478415e280 2024-03-24T00:44:56.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.144+0000 7f4789868640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47840d2530 con 0x7f47840b6170 2024-03-24T00:44:56.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.144+0000 7f4789868640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47840d2810 con 0x7f478415dea0 2024-03-24T00:44:56.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.144+0000 7f47837fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f47840b6170 0x7f47840d3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:56.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.144+0000 7f47837fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f47840b6170 0x7f47840d3910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45024/0 (socket says 172.21.15.67:45024) 2024-03-24T00:44:56.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.144+0000 7f47837fe640 1 -- 172.21.15.67:0/3300633786 learned_addr learned my addr 172.21.15.67:0/3300633786 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:56.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.144+0000 7f4782ffd640 1 --2- 172.21.15.67:0/3300633786 >> v2:172.21.15.72:3300/0 conn(0x7f478415dea0 0x7f47840d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:56.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.144+0000 7f47837fe640 1 -- 172.21.15.67:0/3300633786 >> v2:172.21.15.72:3300/0 conn(0x7f478415dea0 msgr2=0x7f47840d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:56.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.144+0000 7f47837fe640 1 --2- 172.21.15.67:0/3300633786 >> v2:172.21.15.72:3300/0 conn(0x7f478415dea0 0x7f47840d33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:56.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.144+0000 7f47837fe640 1 -- 172.21.15.67:0/3300633786 >> v1:172.21.15.72:6789/0 conn(0x7f478415e280 legacy=0x7f47840b5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:56.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.144+0000 7f47837fe640 1 -- 172.21.15.67:0/3300633786 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47840d2af0 con 0x7f47840b6170 2024-03-24T00:44:56.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.144+0000 7f4782ffd640 1 --2- 172.21.15.67:0/3300633786 >> v2:172.21.15.72:3300/0 conn(0x7f478415dea0 0x7f47840d33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:44:56.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.145+0000 7f47837fe640 1 --2- 172.21.15.67:0/3300633786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47840b6170 0x7f47840d3910 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f4774009e50 tx=0x7f477402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=564f636a12d359c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:56.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.145+0000 7f4781ffb640 1 -- 172.21.15.67:0/3300633786 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f477402ca80 con 0x7f47840b6170 2024-03-24T00:44:56.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.145+0000 7f4781ffb640 1 -- 172.21.15.67:0/3300633786 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f477402cc20 con 0x7f47840b6170 2024-03-24T00:44:56.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.145+0000 7f4781ffb640 1 -- 172.21.15.67:0/3300633786 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47740343f0 con 0x7f47840b6170 2024-03-24T00:44:56.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.146+0000 7f4789868640 1 -- 172.21.15.67:0/3300633786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47840b6170 msgr2=0x7f47840d3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:56.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.146+0000 7f4789868640 1 --2- 172.21.15.67:0/3300633786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47840b6170 0x7f47840d3910 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f4774009e50 tx=0x7f477402bee0 comp rx=0 tx=0).stop 2024-03-24T00:44:56.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.146+0000 7f4789868640 1 -- 172.21.15.67:0/3300633786 shutdown_connections 2024-03-24T00:44:56.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.146+0000 7f4789868640 1 --2- 172.21.15.67:0/3300633786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47840b6170 0x7f47840d3910 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:56.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.146+0000 7f4789868640 1 --2- 172.21.15.67:0/3300633786 >> v2:172.21.15.72:3300/0 conn(0x7f478415dea0 0x7f47840d33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:56.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.146+0000 7f4789868640 1 -- 172.21.15.67:0/3300633786 >> 172.21.15.67:0/3300633786 conn(0x7f4784072da0 msgr2=0x7f47840d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:56.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.146+0000 7f4789868640 1 -- 172.21.15.67:0/3300633786 shutdown_connections 2024-03-24T00:44:56.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.146+0000 7f4789868640 1 -- 172.21.15.67:0/3300633786 wait complete. 2024-03-24T00:44:56.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.147+0000 7f4789868640 1 Processor -- start 2024-03-24T00:44:56.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.147+0000 7f4789868640 1 -- start start 2024-03-24T00:44:56.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.148+0000 7f4789868640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f47840b6170 0x7f478415ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:56.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.148+0000 7f4789868640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f478415dea0 0x7f4784159e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:56.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.148+0000 7f4789868640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f478415e280 0x7f478415f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:56.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.148+0000 7f4789868640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47840b7480 con 0x7f478415dea0 2024-03-24T00:44:56.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.148+0000 7f4789868640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47840b7300 con 0x7f478415e280 2024-03-24T00:44:56.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.148+0000 7f4789868640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47840b7600 con 0x7f47840b6170 2024-03-24T00:44:56.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.148+0000 7f47827fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f478415dea0 0x7f4784159e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:56.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.148+0000 7f47827fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f478415dea0 0x7f4784159e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45026/0 (socket says 172.21.15.67:45026) 2024-03-24T00:44:56.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.148+0000 7f47827fc640 1 -- 172.21.15.67:0/1733302147 learned_addr learned my addr 172.21.15.67:0/1733302147 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:56.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.148+0000 7f4782ffd640 1 --2- 172.21.15.67:0/1733302147 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f47840b6170 0x7f478415ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:56.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.148+0000 7f47837fe640 1 --2- 172.21.15.67:0/1733302147 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f478415e280 0x7f478415f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:56.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.148+0000 7f47827fc640 1 -- 172.21.15.67:0/1733302147 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f47840b6170 msgr2=0x7f478415ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:56.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.148+0000 7f47827fc640 1 --2- 172.21.15.67:0/1733302147 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f47840b6170 0x7f478415ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:56.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.148+0000 7f47827fc640 1 -- 172.21.15.67:0/1733302147 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f478415e280 msgr2=0x7f478415f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:56.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.148+0000 7f47827fc640 1 --2- 172.21.15.67:0/1733302147 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f478415e280 0x7f478415f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:56.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.148+0000 7f47827fc640 1 -- 172.21.15.67:0/1733302147 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f478415f6f0 con 0x7f478415dea0 2024-03-24T00:44:56.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.148+0000 7f47827fc640 1 --2- 172.21.15.67:0/1733302147 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f478415dea0 0x7f4784159e60 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f476c00c3c0 tx=0x7f476c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:56.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.149+0000 7f477bfff640 1 -- 172.21.15.67:0/1733302147 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f476c0052d0 con 0x7f478415dea0 2024-03-24T00:44:56.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.149+0000 7f477bfff640 1 -- 172.21.15.67:0/1733302147 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f476c005490 con 0x7f478415dea0 2024-03-24T00:44:56.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.149+0000 7f4789868640 1 -- 172.21.15.67:0/1733302147 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4784161680 con 0x7f478415dea0 2024-03-24T00:44:56.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.149+0000 7f477bfff640 1 -- 172.21.15.67:0/1733302147 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f476c007260 con 0x7f478415dea0 2024-03-24T00:44:56.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.149+0000 7f4789868640 1 -- 172.21.15.67:0/1733302147 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47841608c0 con 0x7f478415dea0 2024-03-24T00:44:56.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.150+0000 7f4789868640 1 -- 172.21.15.67:0/1733302147 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f478404f400 con 0x7f478415dea0 2024-03-24T00:44:56.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.150+0000 7f477bfff640 1 -- 172.21.15.67:0/1733302147 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f476c007400 con 0x7f478415dea0 2024-03-24T00:44:56.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.150+0000 7f477bfff640 1 --2- 172.21.15.67:0/1733302147 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f476406bbd0 0x7f476406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:56.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.151+0000 7f477bfff640 1 -- 172.21.15.67:0/1733302147 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f476c08a050 con 0x7f478415dea0 2024-03-24T00:44:56.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.151+0000 7f4782ffd640 1 --2- 172.21.15.67:0/1733302147 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f476406bbd0 0x7f476406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:56.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.151+0000 7f4782ffd640 1 --2- 172.21.15.67:0/1733302147 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f476406bbd0 0x7f476406e090 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f477c006fd0 tx=0x7f477c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:56.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.155+0000 7f477bfff640 1 -- 172.21.15.67:0/1733302147 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f476c058a40 con 0x7f478415dea0 2024-03-24T00:44:56.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.376+0000 7f4789868640 1 -- 172.21.15.67:0/1733302147 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f478415da10 con 0x7f478415dea0 2024-03-24T00:44:56.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.397+0000 7f477bfff640 1 -- 172.21.15.67:0/1733302147 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v53) v1 ==== 118+0+0 (secure 0 0 0) 0x7f476c058340 con 0x7f478415dea0 2024-03-24T00:44:56.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.399+0000 7f4789868640 1 -- 172.21.15.67:0/1733302147 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f476406bbd0 msgr2=0x7f476406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:56.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.399+0000 7f4789868640 1 --2- 172.21.15.67:0/1733302147 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f476406bbd0 0x7f476406e090 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f477c006fd0 tx=0x7f477c008040 comp rx=0 tx=0).stop 2024-03-24T00:44:56.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.399+0000 7f4789868640 1 -- 172.21.15.67:0/1733302147 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f478415dea0 msgr2=0x7f4784159e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:56.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.399+0000 7f4789868640 1 --2- 172.21.15.67:0/1733302147 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f478415dea0 0x7f4784159e60 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f476c00c3c0 tx=0x7f476c00c880 comp rx=0 tx=0).stop 2024-03-24T00:44:56.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.399+0000 7f4789868640 1 -- 172.21.15.67:0/1733302147 shutdown_connections 2024-03-24T00:44:56.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.399+0000 7f4789868640 1 --2- 172.21.15.67:0/1733302147 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f476406bbd0 0x7f476406e090 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:56.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.399+0000 7f4789868640 1 --2- 172.21.15.67:0/1733302147 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f478415e280 0x7f478415f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:56.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.399+0000 7f4789868640 1 --2- 172.21.15.67:0/1733302147 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f478415dea0 0x7f4784159e60 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:56.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.399+0000 7f4789868640 1 --2- 172.21.15.67:0/1733302147 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f47840b6170 0x7f478415ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:56.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.399+0000 7f4789868640 1 -- 172.21.15.67:0/1733302147 >> 172.21.15.67:0/1733302147 conn(0x7f4784072da0 msgr2=0x7f47840b39e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:56.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.399+0000 7f4789868640 1 -- 172.21.15.67:0/1733302147 shutdown_connections 2024-03-24T00:44:56.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.399+0000 7f4789868640 1 -- 172.21.15.67:0/1733302147 wait complete. 2024-03-24T00:44:56.409 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-24T00:44:56.410 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-24T00:44:56.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.514+0000 7f1ac6673640 1 Processor -- start 2024-03-24T00:44:56.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.514+0000 7f1ac6673640 1 -- start start 2024-03-24T00:44:56.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.514+0000 7f1ac6673640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1ac015dea0 0x7f1ac00d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:56.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.514+0000 7f1ac6673640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1ac015e280 0x7f1ac00d3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:56.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.514+0000 7f1ac6673640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ac00d2250 con 0x7f1ac00b5a60 2024-03-24T00:44:56.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.514+0000 7f1ac6673640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ac00d2530 con 0x7f1ac015dea0 2024-03-24T00:44:56.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.514+0000 7f1ac6673640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac00d2810 con 0x7f1ac015e280 2024-03-24T00:44:56.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.515+0000 7f1abffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1ac015dea0 0x7f1ac00d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:56.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.515+0000 7f1abffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1ac015dea0 0x7f1ac00d33d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45040/0 (socket says 172.21.15.67:45040) 2024-03-24T00:44:56.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.515+0000 7f1abffff640 1 -- 172.21.15.67:0/2147489676 learned_addr learned my addr 172.21.15.67:0/2147489676 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:56.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.515+0000 7f1abf7fe640 1 --2- 172.21.15.67:0/2147489676 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1ac015e280 0x7f1ac00d3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:56.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.515+0000 7f1abffff640 1 -- 172.21.15.67:0/2147489676 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1ac015e280 msgr2=0x7f1ac00d3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:56.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.515+0000 7f1abffff640 1 --2- 172.21.15.67:0/2147489676 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1ac015e280 0x7f1ac00d3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:56.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.515+0000 7f1abffff640 1 -- 172.21.15.67:0/2147489676 >> v1:172.21.15.72:6789/0 conn(0x7f1ac00b5a60 legacy=0x7f1ac006c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:56.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.515+0000 7f1abffff640 1 -- 172.21.15.67:0/2147489676 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac00d2af0 con 0x7f1ac015dea0 2024-03-24T00:44:56.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.516+0000 7f1abffff640 1 --2- 172.21.15.67:0/2147489676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ac015dea0 0x7f1ac00d33d0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0009870 tx=0x7f1ab002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5914b92cdac0effb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:56.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.516+0000 7f1abeffd640 1 -- 172.21.15.67:0/2147489676 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab0035020 con 0x7f1ac015dea0 2024-03-24T00:44:56.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.516+0000 7f1abeffd640 1 -- 172.21.15.67:0/2147489676 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ab0031640 con 0x7f1ac015dea0 2024-03-24T00:44:56.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.516+0000 7f1abeffd640 1 -- 172.21.15.67:0/2147489676 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab002d040 con 0x7f1ac015dea0 2024-03-24T00:44:56.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.516+0000 7f1ac6673640 1 -- 172.21.15.67:0/2147489676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ac015dea0 msgr2=0x7f1ac00d33d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:56.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.516+0000 7f1ac6673640 1 --2- 172.21.15.67:0/2147489676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ac015dea0 0x7f1ac00d33d0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0009870 tx=0x7f1ab002b8f0 comp rx=0 tx=0).stop 2024-03-24T00:44:56.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.516+0000 7f1ac6673640 1 -- 172.21.15.67:0/2147489676 shutdown_connections 2024-03-24T00:44:56.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.516+0000 7f1ac6673640 1 --2- 172.21.15.67:0/2147489676 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1ac015e280 0x7f1ac00d3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:56.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.516+0000 7f1ac6673640 1 --2- 172.21.15.67:0/2147489676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ac015dea0 0x7f1ac00d33d0 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:56.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.516+0000 7f1ac6673640 1 -- 172.21.15.67:0/2147489676 >> 172.21.15.67:0/2147489676 conn(0x7f1ac0072da0 msgr2=0x7f1ac00d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:56.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.517+0000 7f1ac6673640 1 -- 172.21.15.67:0/2147489676 shutdown_connections 2024-03-24T00:44:56.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.517+0000 7f1ac6673640 1 -- 172.21.15.67:0/2147489676 wait complete. 2024-03-24T00:44:56.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.518+0000 7f1ac6673640 1 Processor -- start 2024-03-24T00:44:56.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.518+0000 7f1ac6673640 1 -- start start 2024-03-24T00:44:56.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.519+0000 7f1ac6673640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1ac00b5a60 0x7f1ac01e6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:56.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.519+0000 7f1ac6673640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ac015dea0 0x7f1ac01f5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:56.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.519+0000 7f1ac6673640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1ac015e280 0x7f1ac014e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:56.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.519+0000 7f1ac6673640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac00b5ed0 con 0x7f1ac015dea0 2024-03-24T00:44:56.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.519+0000 7f1ac6673640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac00b8350 con 0x7f1ac00b5a60 2024-03-24T00:44:56.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.519+0000 7f1ac6673640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac00b6050 con 0x7f1ac015e280 2024-03-24T00:44:56.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.519+0000 7f1abf7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ac015dea0 0x7f1ac01f5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:56.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.520+0000 7f1abf7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ac015dea0 0x7f1ac01f5360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45042/0 (socket says 172.21.15.67:45042) 2024-03-24T00:44:56.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.520+0000 7f1abf7fe640 1 -- 172.21.15.67:0/499996415 learned_addr learned my addr 172.21.15.67:0/499996415 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:56.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.520+0000 7f1abffff640 1 --2- 172.21.15.67:0/499996415 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1ac00b5a60 0x7f1ac01e6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:56.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.520+0000 7f1ac4c10640 1 --2- 172.21.15.67:0/499996415 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1ac015e280 0x7f1ac014e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:56.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.520+0000 7f1abf7fe640 1 -- 172.21.15.67:0/499996415 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1ac015e280 msgr2=0x7f1ac014e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:56.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.520+0000 7f1abf7fe640 1 --2- 172.21.15.67:0/499996415 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1ac015e280 0x7f1ac014e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:56.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.520+0000 7f1abf7fe640 1 -- 172.21.15.67:0/499996415 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1ac00b5a60 msgr2=0x7f1ac01e6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:56.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.520+0000 7f1abf7fe640 1 --2- 172.21.15.67:0/499996415 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1ac00b5a60 0x7f1ac01e6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:56.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.520+0000 7f1abf7fe640 1 -- 172.21.15.67:0/499996415 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac01f58a0 con 0x7f1ac015dea0 2024-03-24T00:44:56.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.520+0000 7f1abf7fe640 1 --2- 172.21.15.67:0/499996415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ac015dea0 0x7f1ac01f5360 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f1ab400d5d0 tx=0x7f1ab400da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:56.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.520+0000 7f1abcff9640 1 -- 172.21.15.67:0/499996415 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab40125f0 con 0x7f1ac015dea0 2024-03-24T00:44:56.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.520+0000 7f1abcff9640 1 -- 172.21.15.67:0/499996415 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ab4019070 con 0x7f1ac015dea0 2024-03-24T00:44:56.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.520+0000 7f1ac6673640 1 -- 172.21.15.67:0/499996415 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ac014ec10 con 0x7f1ac015dea0 2024-03-24T00:44:56.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.520+0000 7f1abcff9640 1 -- 172.21.15.67:0/499996415 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab40129f0 con 0x7f1ac015dea0 2024-03-24T00:44:56.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.520+0000 7f1ac6673640 1 -- 172.21.15.67:0/499996415 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ac01e6e40 con 0x7f1ac015dea0 2024-03-24T00:44:56.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.521+0000 7f1ac6673640 1 -- 172.21.15.67:0/499996415 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ac0068980 con 0x7f1ac015dea0 2024-03-24T00:44:56.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.522+0000 7f1abcff9640 1 -- 172.21.15.67:0/499996415 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1ab4012b90 con 0x7f1ac015dea0 2024-03-24T00:44:56.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.522+0000 7f1abcff9640 1 --2- 172.21.15.67:0/499996415 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1aac06bbd0 0x7f1aac06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:56.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.522+0000 7f1abcff9640 1 -- 172.21.15.67:0/499996415 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1ab408af40 con 0x7f1ac015dea0 2024-03-24T00:44:56.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.522+0000 7f1abffff640 1 --2- 172.21.15.67:0/499996415 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1aac06bbd0 0x7f1aac06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:56.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.523+0000 7f1abffff640 1 --2- 172.21.15.67:0/499996415 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1aac06bbd0 0x7f1aac06e090 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0036000 tx=0x7f1ab003c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:56.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.526+0000 7f1abcff9640 1 -- 172.21.15.67:0/499996415 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1ac0068980 con 0x7f1ac015dea0 2024-03-24T00:44:56.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:56.822+0000 7f1ac6673640 1 -- 172.21.15.67:0/499996415 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f1ac004f400 con 0x7f1ac015dea0 2024-03-24T00:44:57.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.400+0000 7f1abcff9640 1 -- 172.21.15.67:0/499996415 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v54) v1 ==== 122+0+0 (secure 0 0 0) 0x7f1ac01f58a0 con 0x7f1ac015dea0 2024-03-24T00:44:57.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.402+0000 7f1ac6673640 1 -- 172.21.15.67:0/499996415 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1aac06bbd0 msgr2=0x7f1aac06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:57.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.402+0000 7f1ac6673640 1 --2- 172.21.15.67:0/499996415 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1aac06bbd0 0x7f1aac06e090 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0036000 tx=0x7f1ab003c040 comp rx=0 tx=0).stop 2024-03-24T00:44:57.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.402+0000 7f1ac6673640 1 -- 172.21.15.67:0/499996415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ac015dea0 msgr2=0x7f1ac01f5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:57.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.402+0000 7f1ac6673640 1 --2- 172.21.15.67:0/499996415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ac015dea0 0x7f1ac01f5360 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f1ab400d5d0 tx=0x7f1ab400da90 comp rx=0 tx=0).stop 2024-03-24T00:44:57.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.402+0000 7f1ac6673640 1 -- 172.21.15.67:0/499996415 shutdown_connections 2024-03-24T00:44:57.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.402+0000 7f1ac6673640 1 --2- 172.21.15.67:0/499996415 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1aac06bbd0 0x7f1aac06e090 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:57.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.402+0000 7f1ac6673640 1 --2- 172.21.15.67:0/499996415 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1ac015e280 0x7f1ac014e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:57.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.402+0000 7f1ac6673640 1 --2- 172.21.15.67:0/499996415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ac015dea0 0x7f1ac01f5360 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:57.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.402+0000 7f1ac6673640 1 --2- 172.21.15.67:0/499996415 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1ac00b5a60 0x7f1ac01e6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:57.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.402+0000 7f1ac6673640 1 -- 172.21.15.67:0/499996415 >> 172.21.15.67:0/499996415 conn(0x7f1ac0072da0 msgr2=0x7f1ac00b4ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:57.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.402+0000 7f1ac6673640 1 -- 172.21.15.67:0/499996415 shutdown_connections 2024-03-24T00:44:57.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.402+0000 7f1ac6673640 1 -- 172.21.15.67:0/499996415 wait complete. 2024-03-24T00:44:57.412 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-24T00:44:57.413 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:44:57.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.515+0000 7fd2e19b2640 1 Processor -- start 2024-03-24T00:44:57.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.515+0000 7fd2e19b2640 1 -- start start 2024-03-24T00:44:57.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.516+0000 7fd2e19b2640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd2dc15dec0 0x7fd2dc0ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:57.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.516+0000 7fd2e19b2640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd2dc15e2a0 0x7fd2dc162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:57.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.516+0000 7fd2e19b2640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2dc0cf640 con 0x7fd2dc0d07e0 2024-03-24T00:44:57.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.516+0000 7fd2e19b2640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2dc0cee30 con 0x7fd2dc15e2a0 2024-03-24T00:44:57.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.516+0000 7fd2e19b2640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2dc0d2120 con 0x7fd2dc15dec0 2024-03-24T00:44:57.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.516+0000 7fd2db7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd2dc0d07e0 0x7fd2dc0b5850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55524/0 (socket says 172.21.15.67:55524) 2024-03-24T00:44:57.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.516+0000 7fd2db7fe640 1 -- 172.21.15.67:0/1203989471 learned_addr learned my addr 172.21.15.67:0/1203989471 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:44:57.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.516+0000 7fd2daffd640 1 --2- 172.21.15.67:0/1203989471 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd2dc15dec0 0x7fd2dc0ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:57.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.516+0000 7fd2d3fff640 1 --2- 172.21.15.67:0/1203989471 >> v2:172.21.15.72:3300/0 conn(0x7fd2dc15e2a0 0x7fd2dc162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:57.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.516+0000 7fd2da7fc640 1 -- 172.21.15.67:0/1203989471 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3404511290 0 0) 0x7fd2d4001490 con 0x7fd2dc0d07e0 2024-03-24T00:44:57.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.517+0000 7fd2da7fc640 1 -- 172.21.15.67:0/1203989471 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2c8003520 con 0x7fd2dc0d07e0 2024-03-24T00:44:57.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.517+0000 7fd2daffd640 1 -- 172.21.15.67:0/1203989471 >> v2:172.21.15.72:3300/0 conn(0x7fd2dc15e2a0 msgr2=0x7fd2dc162630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:57.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.517+0000 7fd2daffd640 1 --2- 172.21.15.67:0/1203989471 >> v2:172.21.15.72:3300/0 conn(0x7fd2dc15e2a0 0x7fd2dc162630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:57.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.517+0000 7fd2daffd640 1 -- 172.21.15.67:0/1203989471 >> v1:172.21.15.67:6789/0 conn(0x7fd2dc0d07e0 legacy=0x7fd2dc0b5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:57.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.517+0000 7fd2daffd640 1 -- 172.21.15.67:0/1203989471 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2dc0cf080 con 0x7fd2dc15dec0 2024-03-24T00:44:57.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.517+0000 7fd2d3fff640 1 --2- 172.21.15.67:0/1203989471 >> v2:172.21.15.72:3300/0 conn(0x7fd2dc15e2a0 0x7fd2dc162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:44:57.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.517+0000 7fd2daffd640 1 --2- 172.21.15.67:0/1203989471 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd2dc15dec0 0x7fd2dc0ce630 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd2dc0cefb0 tx=0x7fd2cc02b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14974dde6095b497 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:57.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.518+0000 7fd2da7fc640 1 -- 172.21.15.67:0/1203989471 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2cc031770 con 0x7fd2dc15dec0 2024-03-24T00:44:57.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.518+0000 7fd2da7fc640 1 -- 172.21.15.67:0/1203989471 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2cc037070 con 0x7fd2dc15dec0 2024-03-24T00:44:57.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.518+0000 7fd2da7fc640 1 -- 172.21.15.67:0/1203989471 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2cc031b30 con 0x7fd2dc15dec0 2024-03-24T00:44:57.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.518+0000 7fd2e19b2640 1 -- 172.21.15.67:0/1203989471 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd2dc15dec0 msgr2=0x7fd2dc0ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:57.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.518+0000 7fd2e19b2640 1 --2- 172.21.15.67:0/1203989471 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd2dc15dec0 0x7fd2dc0ce630 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd2dc0cefb0 tx=0x7fd2cc02b810 comp rx=0 tx=0).stop 2024-03-24T00:44:57.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.518+0000 7fd2e19b2640 1 -- 172.21.15.67:0/1203989471 shutdown_connections 2024-03-24T00:44:57.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.518+0000 7fd2e19b2640 1 --2- 172.21.15.67:0/1203989471 >> v2:172.21.15.72:3300/0 conn(0x7fd2dc15e2a0 0x7fd2dc162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:57.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.518+0000 7fd2e19b2640 1 --2- 172.21.15.67:0/1203989471 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd2dc15dec0 0x7fd2dc0ce630 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:57.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.518+0000 7fd2e19b2640 1 -- 172.21.15.67:0/1203989471 >> 172.21.15.67:0/1203989471 conn(0x7fd2dc06c810 msgr2=0x7fd2dc06cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:57.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.518+0000 7fd2e19b2640 1 -- 172.21.15.67:0/1203989471 shutdown_connections 2024-03-24T00:44:57.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.518+0000 7fd2e19b2640 1 -- 172.21.15.67:0/1203989471 wait complete. 2024-03-24T00:44:57.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.519+0000 7fd2e19b2640 1 Processor -- start 2024-03-24T00:44:57.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.519+0000 7fd2e19b2640 1 -- start start 2024-03-24T00:44:57.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.520+0000 7fd2e19b2640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd2dc0d07e0 0x7fd2dc1539d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:57.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.520+0000 7fd2e19b2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd2dc15dec0 0x7fd2dc15be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:57.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.520+0000 7fd2e19b2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2dc15e2a0 0x7fd2dc15ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:57.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.520+0000 7fd2e19b2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2dc0d3740 con 0x7fd2dc15e2a0 2024-03-24T00:44:57.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.520+0000 7fd2e19b2640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2dc0d35c0 con 0x7fd2dc15dec0 2024-03-24T00:44:57.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.520+0000 7fd2e19b2640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2dc0d38c0 con 0x7fd2dc0d07e0 2024-03-24T00:44:57.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.520+0000 7fd2db7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2dc15e2a0 0x7fd2dc15ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:57.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.520+0000 7fd2db7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2dc15e2a0 0x7fd2dc15ec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45058/0 (socket says 172.21.15.67:45058) 2024-03-24T00:44:57.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.520+0000 7fd2db7fe640 1 -- 172.21.15.67:0/3432386238 learned_addr learned my addr 172.21.15.67:0/3432386238 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:57.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.520+0000 7fd2daffd640 1 --2- 172.21.15.67:0/3432386238 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd2dc0d07e0 0x7fd2dc1539d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:57.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.520+0000 7fd2d3fff640 1 --2- 172.21.15.67:0/3432386238 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd2dc15dec0 0x7fd2dc15be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:57.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.521+0000 7fd2db7fe640 1 -- 172.21.15.67:0/3432386238 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd2dc0d07e0 msgr2=0x7fd2dc1539d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:57.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.521+0000 7fd2db7fe640 1 --2- 172.21.15.67:0/3432386238 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd2dc0d07e0 0x7fd2dc1539d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:57.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.521+0000 7fd2db7fe640 1 -- 172.21.15.67:0/3432386238 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd2dc15dec0 msgr2=0x7fd2dc15be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:57.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.521+0000 7fd2db7fe640 1 --2- 172.21.15.67:0/3432386238 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd2dc15dec0 0x7fd2dc15be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:57.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.521+0000 7fd2db7fe640 1 -- 172.21.15.67:0/3432386238 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2dc1581e0 con 0x7fd2dc15e2a0 2024-03-24T00:44:57.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.521+0000 7fd2d3fff640 1 --2- 172.21.15.67:0/3432386238 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd2dc15dec0 0x7fd2dc15be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:44:57.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.521+0000 7fd2db7fe640 1 --2- 172.21.15.67:0/3432386238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2dc15e2a0 0x7fd2dc15ec90 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7fd2d400cb50 tx=0x7fd2d400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:57.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.521+0000 7fd2d8ff9640 1 -- 172.21.15.67:0/3432386238 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d4017070 con 0x7fd2dc15e2a0 2024-03-24T00:44:57.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.521+0000 7fd2d8ff9640 1 -- 172.21.15.67:0/3432386238 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2d400ad40 con 0x7fd2dc15e2a0 2024-03-24T00:44:57.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.521+0000 7fd2e19b2640 1 -- 172.21.15.67:0/3432386238 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2dc1584c0 con 0x7fd2dc15e2a0 2024-03-24T00:44:57.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.521+0000 7fd2d8ff9640 1 -- 172.21.15.67:0/3432386238 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d4012620 con 0x7fd2dc15e2a0 2024-03-24T00:44:57.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.521+0000 7fd2e19b2640 1 -- 172.21.15.67:0/3432386238 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2dc158a80 con 0x7fd2dc15e2a0 2024-03-24T00:44:57.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.522+0000 7fd2e19b2640 1 -- 172.21.15.67:0/3432386238 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2dc04f400 con 0x7fd2dc15e2a0 2024-03-24T00:44:57.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.523+0000 7fd2d8ff9640 1 -- 172.21.15.67:0/3432386238 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd2d401a070 con 0x7fd2dc15e2a0 2024-03-24T00:44:57.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.523+0000 7fd2d8ff9640 1 --2- 172.21.15.67:0/3432386238 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd2bc06bbd0 0x7fd2bc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:57.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.523+0000 7fd2d8ff9640 1 -- 172.21.15.67:0/3432386238 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd2d408a9e0 con 0x7fd2dc15e2a0 2024-03-24T00:44:57.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.524+0000 7fd2daffd640 1 --2- 172.21.15.67:0/3432386238 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd2bc06bbd0 0x7fd2bc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:57.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.524+0000 7fd2daffd640 1 --2- 172.21.15.67:0/3432386238 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd2bc06bbd0 0x7fd2bc06e090 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc036000 tx=0x7fd2cc03d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:57.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.527+0000 7fd2d8ff9640 1 -- 172.21.15.67:0/3432386238 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd2dc0cf640 con 0x7fd2dc15e2a0 2024-03-24T00:44:57.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.858+0000 7fd2e19b2640 1 -- 172.21.15.67:0/3432386238 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd2dc0b27a0 con 0x7fd2dc15e2a0 2024-03-24T00:44:57.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.858+0000 7fd2d8ff9640 1 -- 172.21.15.67:0/3432386238 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+4979 (secure 0 0 0) 0x7fd2d4058cf0 con 0x7fd2dc15e2a0 2024-03-24T00:44:57.858 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:44:57.858 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":54,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6359,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2664460392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2664460392},{"type":"v1","addr":"172.21.15.67:6835","nonce":2664460392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6368,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/3323873506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3323873506},{"type":"v1","addr":"172.21.15.72:6837","nonce":3323873506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":7150,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/2744016217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2744016217},{"type":"v1","addr":"172.21.15.72:6836","nonce":2744016217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:44:54.749656+0000","modified":"2024-03-24T00:44:56.822686+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7617},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7617":{"gid":7617,"name":"c","rank":0,"incarnation":52,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6833/3021622863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3021622863},{"type":"v1","addr":"172.21.15.67:6833","nonce":3021622863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7617,"qdb_cluster":[7617]},"id":6}]} 2024-03-24T00:44:57.859 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 54 2024-03-24T00:44:57.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.861+0000 7fd2e19b2640 1 -- 172.21.15.67:0/3432386238 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd2bc06bbd0 msgr2=0x7fd2bc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:57.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.861+0000 7fd2e19b2640 1 --2- 172.21.15.67:0/3432386238 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd2bc06bbd0 0x7fd2bc06e090 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc036000 tx=0x7fd2cc03d040 comp rx=0 tx=0).stop 2024-03-24T00:44:57.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.861+0000 7fd2e19b2640 1 -- 172.21.15.67:0/3432386238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2dc15e2a0 msgr2=0x7fd2dc15ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:57.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.861+0000 7fd2e19b2640 1 --2- 172.21.15.67:0/3432386238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2dc15e2a0 0x7fd2dc15ec90 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7fd2d400cb50 tx=0x7fd2d400a430 comp rx=0 tx=0).stop 2024-03-24T00:44:57.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.861+0000 7fd2e19b2640 1 -- 172.21.15.67:0/3432386238 shutdown_connections 2024-03-24T00:44:57.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.861+0000 7fd2e19b2640 1 --2- 172.21.15.67:0/3432386238 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd2bc06bbd0 0x7fd2bc06e090 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:57.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.861+0000 7fd2e19b2640 1 --2- 172.21.15.67:0/3432386238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2dc15e2a0 0x7fd2dc15ec90 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:57.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.861+0000 7fd2e19b2640 1 --2- 172.21.15.67:0/3432386238 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd2dc15dec0 0x7fd2dc15be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:57.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.861+0000 7fd2e19b2640 1 --2- 172.21.15.67:0/3432386238 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd2dc0d07e0 0x7fd2dc1539d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:57.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.861+0000 7fd2e19b2640 1 -- 172.21.15.67:0/3432386238 >> 172.21.15.67:0/3432386238 conn(0x7fd2dc06c810 msgr2=0x7fd2dc0b2bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:57.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.861+0000 7fd2e19b2640 1 -- 172.21.15.67:0/3432386238 shutdown_connections 2024-03-24T00:44:57.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.861+0000 7fd2e19b2640 1 -- 172.21.15.67:0/3432386238 wait complete. 2024-03-24T00:44:57.872 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:44:57.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.974+0000 7f06ce212640 1 Processor -- start 2024-03-24T00:44:57.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.974+0000 7f06ce212640 1 -- start start 2024-03-24T00:44:57.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.974+0000 7f06ce212640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f06c815dec0 0x7f06c80ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:57.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.974+0000 7f06ce212640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f06c815e2a0 0x7f06c8162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:57.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.974+0000 7f06ce212640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06c80cf640 con 0x7f06c80d07e0 2024-03-24T00:44:57.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.974+0000 7f06ce212640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06c80cee30 con 0x7f06c815dec0 2024-03-24T00:44:57.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.974+0000 7f06ce212640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06c80d2120 con 0x7f06c815e2a0 2024-03-24T00:44:57.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.975+0000 7f06c77fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f06c815dec0 0x7f06c80ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:57.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.975+0000 7f06c77fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f06c815dec0 0x7f06c80ce630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45072/0 (socket says 172.21.15.67:45072) 2024-03-24T00:44:57.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.975+0000 7f06c77fe640 1 -- 172.21.15.67:0/619746035 learned_addr learned my addr 172.21.15.67:0/619746035 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:57.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.975+0000 7f06c6ffd640 1 --2- 172.21.15.67:0/619746035 >> v2:172.21.15.72:3300/0 conn(0x7f06c815e2a0 0x7f06c8162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:57.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.975+0000 7f06c77fe640 1 -- 172.21.15.67:0/619746035 >> v2:172.21.15.72:3300/0 conn(0x7f06c815e2a0 msgr2=0x7f06c8162630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:57.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.975+0000 7f06c77fe640 1 --2- 172.21.15.67:0/619746035 >> v2:172.21.15.72:3300/0 conn(0x7f06c815e2a0 0x7f06c8162630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:57.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.975+0000 7f06c77fe640 1 -- 172.21.15.67:0/619746035 >> v1:172.21.15.72:6789/0 conn(0x7f06c80d07e0 legacy=0x7f06c80d0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:57.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.975+0000 7f06c77fe640 1 -- 172.21.15.67:0/619746035 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06c80cf080 con 0x7f06c815dec0 2024-03-24T00:44:57.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.976+0000 7f06c77fe640 1 --2- 172.21.15.67:0/619746035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06c815dec0 0x7f06c80ce630 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f06c80cede0 tx=0x7f06b802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7432a3d9007d2729 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:57.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.976+0000 7f06c67fc640 1 -- 172.21.15.67:0/619746035 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b802c9f0 con 0x7f06c815dec0 2024-03-24T00:44:57.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.976+0000 7f06c67fc640 1 -- 172.21.15.67:0/619746035 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06b802cb90 con 0x7f06c815dec0 2024-03-24T00:44:57.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.976+0000 7f06c67fc640 1 -- 172.21.15.67:0/619746035 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b80343f0 con 0x7f06c815dec0 2024-03-24T00:44:57.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.976+0000 7f06ce212640 1 -- 172.21.15.67:0/619746035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06c815dec0 msgr2=0x7f06c80ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:57.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.976+0000 7f06ce212640 1 --2- 172.21.15.67:0/619746035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06c815dec0 0x7f06c80ce630 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f06c80cede0 tx=0x7f06b802be50 comp rx=0 tx=0).stop 2024-03-24T00:44:57.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.976+0000 7f06ce212640 1 -- 172.21.15.67:0/619746035 shutdown_connections 2024-03-24T00:44:57.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.976+0000 7f06ce212640 1 --2- 172.21.15.67:0/619746035 >> v2:172.21.15.72:3300/0 conn(0x7f06c815e2a0 0x7f06c8162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:57.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.976+0000 7f06ce212640 1 --2- 172.21.15.67:0/619746035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06c815dec0 0x7f06c80ce630 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:57.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.976+0000 7f06ce212640 1 -- 172.21.15.67:0/619746035 >> 172.21.15.67:0/619746035 conn(0x7f06c806c810 msgr2=0x7f06c806cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:57.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.976+0000 7f06ce212640 1 -- 172.21.15.67:0/619746035 shutdown_connections 2024-03-24T00:44:57.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.976+0000 7f06ce212640 1 -- 172.21.15.67:0/619746035 wait complete. 2024-03-24T00:44:57.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.977+0000 7f06ce212640 1 Processor -- start 2024-03-24T00:44:57.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.978+0000 7f06ce212640 1 -- start start 2024-03-24T00:44:57.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.978+0000 7f06ce212640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06c80d07e0 0x7f06c8156300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:57.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.978+0000 7f06ce212640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06c815dec0 0x7f06c80b6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:57.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.978+0000 7f06ce212640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f06c815e2a0 0x7f06c815ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:57.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.978+0000 7f06ce212640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c80d2730 con 0x7f06c80d07e0 2024-03-24T00:44:57.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.978+0000 7f06ce212640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c80d25b0 con 0x7f06c815dec0 2024-03-24T00:44:57.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.978+0000 7f06ce212640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06c80d28b0 con 0x7f06c815e2a0 2024-03-24T00:44:57.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.978+0000 7f06c77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06c80d07e0 0x7f06c8156300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:57.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.978+0000 7f06c77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06c80d07e0 0x7f06c8156300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45088/0 (socket says 172.21.15.67:45088) 2024-03-24T00:44:57.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.978+0000 7f06c77fe640 1 -- 172.21.15.67:0/1931051922 learned_addr learned my addr 172.21.15.67:0/1931051922 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:57.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.979+0000 7f06c7fff640 1 --2- 172.21.15.67:0/1931051922 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f06c815e2a0 0x7f06c815ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:57.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.979+0000 7f06c6ffd640 1 --2- 172.21.15.67:0/1931051922 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06c815dec0 0x7f06c80b6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:57.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.979+0000 7f06c77fe640 1 -- 172.21.15.67:0/1931051922 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f06c815e2a0 msgr2=0x7f06c815ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:57.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.979+0000 7f06c77fe640 1 --2- 172.21.15.67:0/1931051922 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f06c815e2a0 0x7f06c815ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:57.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.979+0000 7f06c77fe640 1 -- 172.21.15.67:0/1931051922 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06c815dec0 msgr2=0x7f06c80b6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:57.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.979+0000 7f06c77fe640 1 --2- 172.21.15.67:0/1931051922 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06c815dec0 0x7f06c80b6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:57.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.979+0000 7f06c77fe640 1 -- 172.21.15.67:0/1931051922 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06c8156840 con 0x7f06c80d07e0 2024-03-24T00:44:57.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.979+0000 7f06c77fe640 1 --2- 172.21.15.67:0/1931051922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06c80d07e0 0x7f06c8156300 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f06b8000f80 tx=0x7f06b8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:57.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.979+0000 7f06affff640 1 -- 172.21.15.67:0/1931051922 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b8043020 con 0x7f06c80d07e0 2024-03-24T00:44:57.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.979+0000 7f06affff640 1 -- 172.21.15.67:0/1931051922 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06b803e450 con 0x7f06c80d07e0 2024-03-24T00:44:57.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.979+0000 7f06ce212640 1 -- 172.21.15.67:0/1931051922 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06c80b46b0 con 0x7f06c80d07e0 2024-03-24T00:44:57.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.979+0000 7f06affff640 1 -- 172.21.15.67:0/1931051922 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b803e840 con 0x7f06c80d07e0 2024-03-24T00:44:57.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.979+0000 7f06ce212640 1 -- 172.21.15.67:0/1931051922 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06c80b4c70 con 0x7f06c80d07e0 2024-03-24T00:44:57.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.980+0000 7f06ce212640 1 -- 172.21.15.67:0/1931051922 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06c804f400 con 0x7f06c80d07e0 2024-03-24T00:44:57.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.981+0000 7f06affff640 1 -- 172.21.15.67:0/1931051922 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f06b8046020 con 0x7f06c80d07e0 2024-03-24T00:44:57.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.981+0000 7f06affff640 1 --2- 172.21.15.67:0/1931051922 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f06b406bc00 0x7f06b406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:57.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.981+0000 7f06affff640 1 -- 172.21.15.67:0/1931051922 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f06b80ad220 con 0x7f06c80d07e0 2024-03-24T00:44:57.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.981+0000 7f06c6ffd640 1 --2- 172.21.15.67:0/1931051922 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f06b406bc00 0x7f06b406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:57.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.982+0000 7f06c6ffd640 1 --2- 172.21.15.67:0/1931051922 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f06b406bc00 0x7f06b406e0c0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f06bc006fd0 tx=0x7f06bc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:57.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:57.985+0000 7f06affff640 1 -- 172.21.15.67:0/1931051922 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f06b804d070 con 0x7f06c80d07e0 2024-03-24T00:44:58.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.170+0000 7f06ce212640 1 -- 172.21.15.67:0/1931051922 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f06c8161100 con 0x7f06c80d07e0 2024-03-24T00:44:58.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.171+0000 7f06affff640 1 -- 172.21.15.67:0/1931051922 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+16825 (secure 0 0 0) 0x7f06c8156840 con 0x7f06c80d07e0 2024-03-24T00:44:58.171 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:44:58.171 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":89,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:44:56.377082+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:44:53.025965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-03-24T00:44:53.712957+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:44:58.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.173+0000 7f06ce212640 1 -- 172.21.15.67:0/1931051922 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f06b406bc00 msgr2=0x7f06b406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:58.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.173+0000 7f06ce212640 1 --2- 172.21.15.67:0/1931051922 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f06b406bc00 0x7f06b406e0c0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f06bc006fd0 tx=0x7f06bc008040 comp rx=0 tx=0).stop 2024-03-24T00:44:58.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.173+0000 7f06ce212640 1 -- 172.21.15.67:0/1931051922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06c80d07e0 msgr2=0x7f06c8156300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:58.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.173+0000 7f06ce212640 1 --2- 172.21.15.67:0/1931051922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06c80d07e0 0x7f06c8156300 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f06b8000f80 tx=0x7f06b8004a40 comp rx=0 tx=0).stop 2024-03-24T00:44:58.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.173+0000 7f06ce212640 1 -- 172.21.15.67:0/1931051922 shutdown_connections 2024-03-24T00:44:58.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.173+0000 7f06ce212640 1 --2- 172.21.15.67:0/1931051922 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f06b406bc00 0x7f06b406e0c0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:58.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.173+0000 7f06ce212640 1 --2- 172.21.15.67:0/1931051922 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f06c815e2a0 0x7f06c815ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:58.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.173+0000 7f06ce212640 1 --2- 172.21.15.67:0/1931051922 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06c815dec0 0x7f06c80b6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:58.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.173+0000 7f06ce212640 1 --2- 172.21.15.67:0/1931051922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06c80d07e0 0x7f06c8156300 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:58.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.173+0000 7f06ce212640 1 -- 172.21.15.67:0/1931051922 >> 172.21.15.67:0/1931051922 conn(0x7f06c806c810 msgr2=0x7f06c815bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:58.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.173+0000 7f06ce212640 1 -- 172.21.15.67:0/1931051922 shutdown_connections 2024-03-24T00:44:58.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.173+0000 7f06ce212640 1 -- 172.21.15.67:0/1931051922 wait complete. 2024-03-24T00:44:58.184 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-24T00:44:58.184 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:44:58.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.287+0000 7f3dc9ebf640 1 Processor -- start 2024-03-24T00:44:58.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.287+0000 7f3dc9ebf640 1 -- start start 2024-03-24T00:44:58.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.288+0000 7f3dc9ebf640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3dc40b15d0 0x7f3dc40cd710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:58.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.288+0000 7f3dc9ebf640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3dc415de20 0x7f3dc40cb590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:58.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.288+0000 7f3dc9ebf640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dc40b4270 con 0x7f3dc40b19b0 2024-03-24T00:44:58.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.288+0000 7f3dc9ebf640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dc40cb100 con 0x7f3dc40b15d0 2024-03-24T00:44:58.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.288+0000 7f3dc9ebf640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc40caad0 con 0x7f3dc415de20 2024-03-24T00:44:58.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.288+0000 7f3dc37fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3dc40b15d0 0x7f3dc40cd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:58.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.288+0000 7f3dc3fff640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3dc415de20 0x7f3dc40cb590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:58.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.288+0000 7f3dbaffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f3dc40b19b0 0x7f3dc415d710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:50336/0 (socket says 172.21.15.67:50336) 2024-03-24T00:44:58.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.288+0000 7f3dbaffd640 1 -- 172.21.15.67:0/574379117 learned_addr learned my addr 172.21.15.67:0/574379117 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:44:58.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.289+0000 7f3dc37fe640 1 -- 172.21.15.67:0/574379117 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3dc415de20 msgr2=0x7f3dc40cb590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:58.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.289+0000 7f3dc37fe640 1 --2- 172.21.15.67:0/574379117 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3dc415de20 0x7f3dc40cb590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:58.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.289+0000 7f3dc37fe640 1 -- 172.21.15.67:0/574379117 >> v1:172.21.15.72:6789/0 conn(0x7f3dc40b19b0 legacy=0x7f3dc415d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:58.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.289+0000 7f3dc37fe640 1 -- 172.21.15.67:0/574379117 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dc406b590 con 0x7f3dc40b15d0 2024-03-24T00:44:58.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.289+0000 7f3dc3fff640 1 --2- 172.21.15.67:0/574379117 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3dc415de20 0x7f3dc40cb590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:44:58.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.289+0000 7f3dc37fe640 1 --2- 172.21.15.67:0/574379117 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3dc40b15d0 0x7f3dc40cd710 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f3dc40cb280 tx=0x7f3db402b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10def437f76b9325 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:58.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.290+0000 7f3dc2ffd640 1 -- 172.21.15.67:0/574379117 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db4031770 con 0x7f3dc40b15d0 2024-03-24T00:44:58.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.290+0000 7f3dc2ffd640 1 -- 172.21.15.67:0/574379117 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3db4037070 con 0x7f3dc40b15d0 2024-03-24T00:44:58.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.290+0000 7f3dc2ffd640 1 -- 172.21.15.67:0/574379117 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db4031b30 con 0x7f3dc40b15d0 2024-03-24T00:44:58.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.290+0000 7f3dc9ebf640 1 -- 172.21.15.67:0/574379117 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3dc40b15d0 msgr2=0x7f3dc40cd710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:58.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.290+0000 7f3dc9ebf640 1 --2- 172.21.15.67:0/574379117 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3dc40b15d0 0x7f3dc40cd710 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f3dc40cb280 tx=0x7f3db402b810 comp rx=0 tx=0).stop 2024-03-24T00:44:58.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.290+0000 7f3dc9ebf640 1 -- 172.21.15.67:0/574379117 shutdown_connections 2024-03-24T00:44:58.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.290+0000 7f3dc9ebf640 1 --2- 172.21.15.67:0/574379117 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3dc415de20 0x7f3dc40cb590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:58.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.290+0000 7f3dc9ebf640 1 --2- 172.21.15.67:0/574379117 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3dc40b15d0 0x7f3dc40cd710 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:58.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.290+0000 7f3dc9ebf640 1 -- 172.21.15.67:0/574379117 >> 172.21.15.67:0/574379117 conn(0x7f3dc40b5440 msgr2=0x7f3dc40b33d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:58.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.290+0000 7f3dc9ebf640 1 -- 172.21.15.67:0/574379117 shutdown_connections 2024-03-24T00:44:58.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.290+0000 7f3dc9ebf640 1 -- 172.21.15.67:0/574379117 wait complete. 2024-03-24T00:44:58.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.292+0000 7f3dc9ebf640 1 Processor -- start 2024-03-24T00:44:58.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.292+0000 7f3dc9ebf640 1 -- start start 2024-03-24T00:44:58.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.293+0000 7f3dc9ebf640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3dc40b15d0 0x7f3dc40c7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:58.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.293+0000 7f3dc9ebf640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dc40b19b0 0x7f3dc413d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:58.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.293+0000 7f3dc9ebf640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3dc415de20 0x7f3dc413d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:58.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.293+0000 7f3dc9ebf640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc406b260 con 0x7f3dc40b19b0 2024-03-24T00:44:58.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.293+0000 7f3dc9ebf640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc40cad90 con 0x7f3dc40b15d0 2024-03-24T00:44:58.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.293+0000 7f3dc9ebf640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc40cdc50 con 0x7f3dc415de20 2024-03-24T00:44:58.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.293+0000 7f3dbaffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dc40b19b0 0x7f3dc413d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:58.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.293+0000 7f3dbaffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dc40b19b0 0x7f3dc413d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45102/0 (socket says 172.21.15.67:45102) 2024-03-24T00:44:58.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.293+0000 7f3dbaffd640 1 -- 172.21.15.67:0/3068332922 learned_addr learned my addr 172.21.15.67:0/3068332922 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:58.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.293+0000 7f3dc3fff640 1 --2- 172.21.15.67:0/3068332922 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3dc415de20 0x7f3dc413d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:58.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.293+0000 7f3dc37fe640 1 --2- 172.21.15.67:0/3068332922 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3dc40b15d0 0x7f3dc40c7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:58.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.293+0000 7f3dbaffd640 1 -- 172.21.15.67:0/3068332922 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3dc415de20 msgr2=0x7f3dc413d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:58.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.293+0000 7f3dbaffd640 1 --2- 172.21.15.67:0/3068332922 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3dc415de20 0x7f3dc413d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:58.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.293+0000 7f3dbaffd640 1 -- 172.21.15.67:0/3068332922 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3dc40b15d0 msgr2=0x7f3dc40c7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:58.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.293+0000 7f3dbaffd640 1 --2- 172.21.15.67:0/3068332922 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3dc40b15d0 0x7f3dc40c7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:58.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.293+0000 7f3dbaffd640 1 -- 172.21.15.67:0/3068332922 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dc40ce770 con 0x7f3dc40b19b0 2024-03-24T00:44:58.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.294+0000 7f3dbaffd640 1 --2- 172.21.15.67:0/3068332922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dc40b19b0 0x7f3dc413d1e0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f3dac00c430 tx=0x7f3dac00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:58.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.294+0000 7f3dc0ff9640 1 -- 172.21.15.67:0/3068332922 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dac017070 con 0x7f3dc40b19b0 2024-03-24T00:44:58.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.294+0000 7f3dc0ff9640 1 -- 172.21.15.67:0/3068332922 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dac005ce0 con 0x7f3dc40b19b0 2024-03-24T00:44:58.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.294+0000 7f3dc0ff9640 1 -- 172.21.15.67:0/3068332922 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dac0072f0 con 0x7f3dc40b19b0 2024-03-24T00:44:58.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.294+0000 7f3dc9ebf640 1 -- 172.21.15.67:0/3068332922 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dc40cc2a0 con 0x7f3dc40b19b0 2024-03-24T00:44:58.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.294+0000 7f3dc9ebf640 1 -- 172.21.15.67:0/3068332922 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dc40ce390 con 0x7f3dc40b19b0 2024-03-24T00:44:58.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.295+0000 7f3dc9ebf640 1 -- 172.21.15.67:0/3068332922 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dc40cbff0 con 0x7f3dc40b19b0 2024-03-24T00:44:58.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.295+0000 7f3dc0ff9640 1 -- 172.21.15.67:0/3068332922 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3dac01a070 con 0x7f3dc40b19b0 2024-03-24T00:44:58.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.296+0000 7f3dc0ff9640 1 --2- 172.21.15.67:0/3068332922 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3da406bbd0 0x7f3da406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:58.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.296+0000 7f3dc0ff9640 1 -- 172.21.15.67:0/3068332922 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3dac089e60 con 0x7f3dc40b19b0 2024-03-24T00:44:58.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.296+0000 7f3dc37fe640 1 --2- 172.21.15.67:0/3068332922 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3da406bbd0 0x7f3da406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:58.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.297+0000 7f3dc37fe640 1 --2- 172.21.15.67:0/3068332922 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3da406bbd0 0x7f3da406e090 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f3db4036000 tx=0x7f3db403d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:58.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.300+0000 7f3dc0ff9640 1 -- 172.21.15.67:0/3068332922 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3dac058200 con 0x7f3dc40b19b0 2024-03-24T00:44:58.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.481+0000 7f3dc9ebf640 1 -- 172.21.15.67:0/3068332922 --> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3dc404f400 con 0x7f3da406bbd0 2024-03-24T00:44:58.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.487+0000 7f3dc0ff9640 1 -- 172.21.15.67:0/3068332922 <== mgr.4105 v2:172.21.15.72:6832/2768292274 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240182 (secure 0 0 0) 0x7f3dc404f400 con 0x7f3da406bbd0 2024-03-24T00:44:58.490 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:44:58.491 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-24T00:44:58.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.490+0000 7f3dc9ebf640 1 -- 172.21.15.67:0/3068332922 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3da406bbd0 msgr2=0x7f3da406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:58.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.490+0000 7f3dc9ebf640 1 --2- 172.21.15.67:0/3068332922 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3da406bbd0 0x7f3da406e090 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f3db4036000 tx=0x7f3db403d040 comp rx=0 tx=0).stop 2024-03-24T00:44:58.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.490+0000 7f3dc9ebf640 1 -- 172.21.15.67:0/3068332922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dc40b19b0 msgr2=0x7f3dc413d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:58.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.490+0000 7f3dc9ebf640 1 --2- 172.21.15.67:0/3068332922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dc40b19b0 0x7f3dc413d1e0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f3dac00c430 tx=0x7f3dac00c830 comp rx=0 tx=0).stop 2024-03-24T00:44:58.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.490+0000 7f3dc9ebf640 1 -- 172.21.15.67:0/3068332922 shutdown_connections 2024-03-24T00:44:58.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.490+0000 7f3dc9ebf640 1 --2- 172.21.15.67:0/3068332922 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3da406bbd0 0x7f3da406e090 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:58.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.490+0000 7f3dc9ebf640 1 --2- 172.21.15.67:0/3068332922 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3dc415de20 0x7f3dc413d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:58.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.490+0000 7f3dc9ebf640 1 --2- 172.21.15.67:0/3068332922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dc40b19b0 0x7f3dc413d1e0 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:58.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.490+0000 7f3dc9ebf640 1 --2- 172.21.15.67:0/3068332922 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3dc40b15d0 0x7f3dc40c7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:58.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.490+0000 7f3dc9ebf640 1 -- 172.21.15.67:0/3068332922 >> 172.21.15.67:0/3068332922 conn(0x7f3dc40b5440 msgr2=0x7f3dc40b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:58.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.490+0000 7f3dc9ebf640 1 -- 172.21.15.67:0/3068332922 shutdown_connections 2024-03-24T00:44:58.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.490+0000 7f3dc9ebf640 1 -- 172.21.15.67:0/3068332922 wait complete. 2024-03-24T00:44:58.501 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":271,"stamp":"2024-03-24T00:44:57.408670+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":246408,"kb_used_data":9448,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754728312,"statfs":{"total":773094113280,"available":772841791488,"internally_reserved":0,"allocated":9674752,"data_stored":5058168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":242535985},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.036782"},"pg_stats":[{"pgid":"14.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.375232+0000","last_change":"2024-03-24T00:44:53.375232+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.375232+0000","last_undegraded":"2024-03-24T00:44:53.375232+0000","last_fullsized":"2024-03-24T00:44:53.375232+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.374673+0000","last_change":"2024-03-24T00:44:53.374673+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.374673+0000","last_undegraded":"2024-03-24T00:44:53.374673+0000","last_fullsized":"2024-03-24T00:44:53.374673+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.377713+0000","last_change":"2024-03-24T00:44:53.377713+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.377713+0000","last_undegraded":"2024-03-24T00:44:53.377713+0000","last_fullsized":"2024-03-24T00:44:53.377713+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.374345+0000","last_change":"2024-03-24T00:44:53.374345+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.374345+0000","last_undegraded":"2024-03-24T00:44:53.374345+0000","last_fullsized":"2024-03-24T00:44:53.374345+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.373750+0000","last_change":"2024-03-24T00:44:53.373750+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.373750+0000","last_undegraded":"2024-03-24T00:44:53.373750+0000","last_fullsized":"2024-03-24T00:44:53.373750+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.375689+0000","last_change":"2024-03-24T00:44:53.375689+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.375689+0000","last_undegraded":"2024-03-24T00:44:53.375689+0000","last_fullsized":"2024-03-24T00:44:53.375689+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.373823+0000","last_change":"2024-03-24T00:44:53.373823+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.373823+0000","last_undegraded":"2024-03-24T00:44:53.373823+0000","last_fullsized":"2024-03-24T00:44:53.373823+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.378054+0000","last_change":"2024-03-24T00:44:53.378054+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.378054+0000","last_undegraded":"2024-03-24T00:44:53.378054+0000","last_fullsized":"2024-03-24T00:44:53.378054+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.377903+0000","last_change":"2024-03-24T00:44:53.377903+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.377903+0000","last_undegraded":"2024-03-24T00:44:53.377903+0000","last_fullsized":"2024-03-24T00:44:53.377903+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.373843+0000","last_change":"2024-03-24T00:44:53.373843+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.373843+0000","last_undegraded":"2024-03-24T00:44:53.373843+0000","last_fullsized":"2024-03-24T00:44:53.373843+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.374785+0000","last_change":"2024-03-24T00:44:53.374785+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.374785+0000","last_undegraded":"2024-03-24T00:44:53.374785+0000","last_fullsized":"2024-03-24T00:44:53.374785+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.376808+0000","last_change":"2024-03-24T00:44:53.376808+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.376808+0000","last_undegraded":"2024-03-24T00:44:53.376808+0000","last_fullsized":"2024-03-24T00:44:53.376808+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.377208+0000","last_change":"2024-03-24T00:44:53.377208+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.377208+0000","last_undegraded":"2024-03-24T00:44:53.377208+0000","last_fullsized":"2024-03-24T00:44:53.377208+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.376814+0000","last_change":"2024-03-24T00:44:53.376814+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.376814+0000","last_undegraded":"2024-03-24T00:44:53.376814+0000","last_fullsized":"2024-03-24T00:44:53.376814+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.375656+0000","last_change":"2024-03-24T00:44:53.375656+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.375656+0000","last_undegraded":"2024-03-24T00:44:53.375656+0000","last_fullsized":"2024-03-24T00:44:53.375656+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.375797+0000","last_change":"2024-03-24T00:44:53.375797+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.375797+0000","last_undegraded":"2024-03-24T00:44:53.375797+0000","last_fullsized":"2024-03-24T00:44:53.375797+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.373960+0000","last_change":"2024-03-24T00:44:53.373960+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.373960+0000","last_undegraded":"2024-03-24T00:44:53.373960+0000","last_fullsized":"2024-03-24T00:44:53.373960+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.376955+0000","last_change":"2024-03-24T00:44:53.376955+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.376955+0000","last_undegraded":"2024-03-24T00:44:53.376955+0000","last_fullsized":"2024-03-24T00:44:53.376955+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.374336+0000","last_change":"2024-03-24T00:44:53.374336+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.374336+0000","last_undegraded":"2024-03-24T00:44:53.374336+0000","last_fullsized":"2024-03-24T00:44:53.374336+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.378153+0000","last_change":"2024-03-24T00:44:53.378153+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.378153+0000","last_undegraded":"2024-03-24T00:44:53.378153+0000","last_fullsized":"2024-03-24T00:44:53.378153+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.376525+0000","last_change":"2024-03-24T00:44:53.376525+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.376525+0000","last_undegraded":"2024-03-24T00:44:53.376525+0000","last_fullsized":"2024-03-24T00:44:53.376525+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.377062+0000","last_change":"2024-03-24T00:44:53.377062+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.377062+0000","last_undegraded":"2024-03-24T00:44:53.377062+0000","last_fullsized":"2024-03-24T00:44:53.377062+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.375988+0000","last_change":"2024-03-24T00:44:53.375988+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.375988+0000","last_undegraded":"2024-03-24T00:44:53.375988+0000","last_fullsized":"2024-03-24T00:44:53.375988+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.376770+0000","last_change":"2024-03-24T00:44:53.376770+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.376770+0000","last_undegraded":"2024-03-24T00:44:53.376770+0000","last_fullsized":"2024-03-24T00:44:53.376770+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.374244+0000","last_change":"2024-03-24T00:44:53.374244+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.374244+0000","last_undegraded":"2024-03-24T00:44:53.374244+0000","last_fullsized":"2024-03-24T00:44:53.374244+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.373845+0000","last_change":"2024-03-24T00:44:53.373845+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.373845+0000","last_undegraded":"2024-03-24T00:44:53.373845+0000","last_fullsized":"2024-03-24T00:44:53.373845+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.373752+0000","last_change":"2024-03-24T00:44:53.373752+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.373752+0000","last_undegraded":"2024-03-24T00:44:53.373752+0000","last_fullsized":"2024-03-24T00:44:53.373752+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.377232+0000","last_change":"2024-03-24T00:44:53.377232+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.377232+0000","last_undegraded":"2024-03-24T00:44:53.377232+0000","last_fullsized":"2024-03-24T00:44:53.377232+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.377878+0000","last_change":"2024-03-24T00:44:53.377878+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.377878+0000","last_undegraded":"2024-03-24T00:44:53.377878+0000","last_fullsized":"2024-03-24T00:44:53.377878+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.373915+0000","last_change":"2024-03-24T00:44:53.373915+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.373915+0000","last_undegraded":"2024-03-24T00:44:53.373915+0000","last_fullsized":"2024-03-24T00:44:53.373915+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.376719+0000","last_change":"2024-03-24T00:44:53.376719+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.376719+0000","last_undegraded":"2024-03-24T00:44:53.376719+0000","last_fullsized":"2024-03-24T00:44:53.376719+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.375668+0000","last_change":"2024-03-24T00:44:53.375668+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.375668+0000","last_undegraded":"2024-03-24T00:44:53.375668+0000","last_fullsized":"2024-03-24T00:44:53.375668+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":293,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:44:53.378811+0000","last_change":"2024-03-24T00:38:40.953871+0000","last_active":"2024-03-24T00:44:53.378811+0000","last_peered":"2024-03-24T00:44:53.378811+0000","last_clean":"2024-03-24T00:44:53.378811+0000","last_became_active":"2024-03-24T00:38:40.952926+0000","last_became_peered":"2024-03-24T00:38:40.952926+0000","last_unstale":"2024-03-24T00:44:53.378811+0000","last_undegraded":"2024-03-24T00:44:53.378811+0000","last_fullsized":"2024-03-24T00:44:53.378811+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_clean_scrub_stamp":"2024-03-24T00:38:39.722194+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:05.397257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574928,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32796,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339044,"statfs":{"total":96636764160,"available":96603181056,"internally_reserved":0,"allocated":1654784,"data_stored":1075047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574927,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28056,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343784,"statfs":{"total":96636764160,"available":96608034816,"internally_reserved":0,"allocated":1060864,"data_stored":484679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574927,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339624,"statfs":{"total":96636764160,"available":96603774976,"internally_reserved":0,"allocated":1060864,"data_stored":484679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574926,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339624,"statfs":{"total":96636764160,"available":96603774976,"internally_reserved":0,"allocated":1060864,"data_stored":484679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574926,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339624,"statfs":{"total":96636764160,"available":96603774976,"internally_reserved":0,"allocated":1060864,"data_stored":484679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574927,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28056,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343784,"statfs":{"total":96636764160,"available":96608034816,"internally_reserved":0,"allocated":1060864,"data_stored":484679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574927,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28056,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343784,"statfs":{"total":96636764160,"available":96608034816,"internally_reserved":0,"allocated":1060864,"data_stored":484679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574926,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32796,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339044,"statfs":{"total":96636764160,"available":96603181056,"internally_reserved":0,"allocated":1654784,"data_stored":1075047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:44:58.505 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:44:58.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.643+0000 7fc9e15d4640 1 Processor -- start 2024-03-24T00:44:58.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.643+0000 7fc9e15d4640 1 -- start start 2024-03-24T00:44:58.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.644+0000 7fc9e15d4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc9dc15bf70 0x7fc9dc0cc690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:58.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.644+0000 7fc9e15d4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc9dc0ce930 0x7fc9dc0b1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:58.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.644+0000 7fc9e15d4640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9dc0ccfe0 con 0x7fc9dc0cfc70 2024-03-24T00:44:58.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.644+0000 7fc9e15d4640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9dc0cd2c0 con 0x7fc9dc15bf70 2024-03-24T00:44:58.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.644+0000 7fc9e15d4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9dc0cd5a0 con 0x7fc9dc0ce930 2024-03-24T00:44:58.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.644+0000 7fc9daffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc9dc15bf70 0x7fc9dc0cc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:58.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.644+0000 7fc9d3fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc9dc0cfc70 0x7fc9dc0ce220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55532/0 (socket says 172.21.15.67:55532) 2024-03-24T00:44:58.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.644+0000 7fc9d3fff640 1 -- 172.21.15.67:0/2117645030 learned_addr learned my addr 172.21.15.67:0/2117645030 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:44:58.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.644+0000 7fc9db7fe640 1 --2- 172.21.15.67:0/2117645030 >> v2:172.21.15.72:3300/0 conn(0x7fc9dc0ce930 0x7fc9dc0b1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:58.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.644+0000 7fc9daffd640 1 -- 172.21.15.67:0/2117645030 >> v2:172.21.15.72:3300/0 conn(0x7fc9dc0ce930 msgr2=0x7fc9dc0b1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:58.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.644+0000 7fc9daffd640 1 --2- 172.21.15.67:0/2117645030 >> v2:172.21.15.72:3300/0 conn(0x7fc9dc0ce930 0x7fc9dc0b1cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:58.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.644+0000 7fc9daffd640 1 -- 172.21.15.67:0/2117645030 >> v1:172.21.15.67:6789/0 conn(0x7fc9dc0cfc70 legacy=0x7fc9dc0ce220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:58.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.644+0000 7fc9da7fc640 1 -- 172.21.15.67:0/2117645030 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2291581395 0 0) 0x7fc9dc0ccfe0 con 0x7fc9dc0cfc70 2024-03-24T00:44:58.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.644+0000 7fc9daffd640 1 -- 172.21.15.67:0/2117645030 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9dc0cd880 con 0x7fc9dc15bf70 2024-03-24T00:44:58.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.645+0000 7fc9daffd640 1 --2- 172.21.15.67:0/2117645030 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9dc15bf70 0x7fc9dc0cc690 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc0b0f80 tx=0x7fc9cc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd321aeaf4456be6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:58.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.645+0000 7fc9da7fc640 1 -- 172.21.15.67:0/2117645030 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9cc035020 con 0x7fc9dc15bf70 2024-03-24T00:44:58.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.645+0000 7fc9da7fc640 1 -- 172.21.15.67:0/2117645030 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9cc031830 con 0x7fc9dc15bf70 2024-03-24T00:44:58.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.645+0000 7fc9da7fc640 1 -- 172.21.15.67:0/2117645030 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9cc031c00 con 0x7fc9dc15bf70 2024-03-24T00:44:58.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.646+0000 7fc9e15d4640 1 -- 172.21.15.67:0/2117645030 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9dc15bf70 msgr2=0x7fc9dc0cc690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:58.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.646+0000 7fc9e15d4640 1 --2- 172.21.15.67:0/2117645030 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9dc15bf70 0x7fc9dc0cc690 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc0b0f80 tx=0x7fc9cc02b8d0 comp rx=0 tx=0).stop 2024-03-24T00:44:58.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.646+0000 7fc9e15d4640 1 -- 172.21.15.67:0/2117645030 shutdown_connections 2024-03-24T00:44:58.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.646+0000 7fc9e15d4640 1 --2- 172.21.15.67:0/2117645030 >> v2:172.21.15.72:3300/0 conn(0x7fc9dc0ce930 0x7fc9dc0b1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:58.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.646+0000 7fc9e15d4640 1 --2- 172.21.15.67:0/2117645030 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9dc15bf70 0x7fc9dc0cc690 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:58.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.646+0000 7fc9e15d4640 1 -- 172.21.15.67:0/2117645030 >> 172.21.15.67:0/2117645030 conn(0x7fc9dc0d1bf0 msgr2=0x7fc9dc0d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:58.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.646+0000 7fc9e15d4640 1 -- 172.21.15.67:0/2117645030 shutdown_connections 2024-03-24T00:44:58.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.646+0000 7fc9e15d4640 1 -- 172.21.15.67:0/2117645030 wait complete. 2024-03-24T00:44:58.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.647+0000 7fc9e15d4640 1 Processor -- start 2024-03-24T00:44:58.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.647+0000 7fc9e15d4640 1 -- start start 2024-03-24T00:44:58.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.648+0000 7fc9e15d4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc9dc0ce930 0x7fc9dc1595c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:58.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.648+0000 7fc9e15d4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9dc0cfc70 0x7fc9dc15e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:58.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.648+0000 7fc9e15d4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9dc15bf70 0x7fc9dc15ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:58.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.648+0000 7fc9e15d4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9dc0ccd50 con 0x7fc9dc0cfc70 2024-03-24T00:44:58.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.648+0000 7fc9e15d4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9dc0ccbd0 con 0x7fc9dc15bf70 2024-03-24T00:44:58.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.648+0000 7fc9e15d4640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9dc0b2210 con 0x7fc9dc0ce930 2024-03-24T00:44:58.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.648+0000 7fc9d3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9dc0cfc70 0x7fc9dc15e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:58.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.648+0000 7fc9d3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9dc0cfc70 0x7fc9dc15e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45134/0 (socket says 172.21.15.67:45134) 2024-03-24T00:44:58.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.648+0000 7fc9d3fff640 1 -- 172.21.15.67:0/4029113675 learned_addr learned my addr 172.21.15.67:0/4029113675 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:58.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.648+0000 7fc9db7fe640 1 --2- 172.21.15.67:0/4029113675 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9dc15bf70 0x7fc9dc15ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:58.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.648+0000 7fc9d3fff640 1 -- 172.21.15.67:0/4029113675 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc9dc0ce930 msgr2=0x7fc9dc1595c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:58.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.648+0000 7fc9d3fff640 1 --2- 172.21.15.67:0/4029113675 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc9dc0ce930 0x7fc9dc1595c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:58.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.648+0000 7fc9d3fff640 1 -- 172.21.15.67:0/4029113675 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9dc15bf70 msgr2=0x7fc9dc15ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:58.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.648+0000 7fc9d3fff640 1 --2- 172.21.15.67:0/4029113675 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9dc15bf70 0x7fc9dc15ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:58.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.648+0000 7fc9d3fff640 1 -- 172.21.15.67:0/4029113675 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9dc0cbe30 con 0x7fc9dc0cfc70 2024-03-24T00:44:58.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.648+0000 7fc9d3fff640 1 --2- 172.21.15.67:0/4029113675 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9dc0cfc70 0x7fc9dc15e730 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fc9c400c4b0 tx=0x7fc9c400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:58.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.649+0000 7fc9d37fe640 1 -- 172.21.15.67:0/4029113675 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9c4018070 con 0x7fc9dc0cfc70 2024-03-24T00:44:58.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.649+0000 7fc9d37fe640 1 -- 172.21.15.67:0/4029113675 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9c4007500 con 0x7fc9dc0cfc70 2024-03-24T00:44:58.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.649+0000 7fc9e15d4640 1 -- 172.21.15.67:0/4029113675 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9dc1f3ef0 con 0x7fc9dc0cfc70 2024-03-24T00:44:58.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.649+0000 7fc9d37fe640 1 -- 172.21.15.67:0/4029113675 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9c4004a20 con 0x7fc9dc0cfc70 2024-03-24T00:44:58.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.649+0000 7fc9e15d4640 1 -- 172.21.15.67:0/4029113675 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9dc1f44b0 con 0x7fc9dc0cfc70 2024-03-24T00:44:58.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.650+0000 7fc9e15d4640 1 -- 172.21.15.67:0/4029113675 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9dc04f400 con 0x7fc9dc0cfc70 2024-03-24T00:44:58.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.650+0000 7fc9d37fe640 1 -- 172.21.15.67:0/4029113675 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc9c401b070 con 0x7fc9dc0cfc70 2024-03-24T00:44:58.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.651+0000 7fc9d37fe640 1 --2- 172.21.15.67:0/4029113675 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc9bc06bbd0 0x7fc9bc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:58.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.651+0000 7fc9d37fe640 1 -- 172.21.15.67:0/4029113675 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc9c4089ec0 con 0x7fc9dc0cfc70 2024-03-24T00:44:58.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.651+0000 7fc9daffd640 1 --2- 172.21.15.67:0/4029113675 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc9bc06bbd0 0x7fc9bc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:58.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.651+0000 7fc9daffd640 1 --2- 172.21.15.67:0/4029113675 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc9bc06bbd0 0x7fc9bc06e090 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc036000 tx=0x7fc9cc030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:58.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.655+0000 7fc9d37fe640 1 -- 172.21.15.67:0/4029113675 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc9c4058970 con 0x7fc9dc0cfc70 2024-03-24T00:44:58.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.832+0000 7fc9e15d4640 1 -- 172.21.15.67:0/4029113675 --> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc9dc159b00 con 0x7fc9bc06bbd0 2024-03-24T00:44:58.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.838+0000 7fc9d37fe640 1 -- 172.21.15.67:0/4029113675 <== mgr.4105 v2:172.21.15.72:6832/2768292274 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240182 (secure 0 0 0) 0x7fc9cc03a420 con 0x7fc9bc06bbd0 2024-03-24T00:44:58.842 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:44:58.844 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-24T00:44:58.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.841+0000 7fc9e15d4640 1 -- 172.21.15.67:0/4029113675 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc9bc06bbd0 msgr2=0x7fc9bc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:58.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.841+0000 7fc9e15d4640 1 --2- 172.21.15.67:0/4029113675 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc9bc06bbd0 0x7fc9bc06e090 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc036000 tx=0x7fc9cc030000 comp rx=0 tx=0).stop 2024-03-24T00:44:58.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.841+0000 7fc9e15d4640 1 -- 172.21.15.67:0/4029113675 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9dc0cfc70 msgr2=0x7fc9dc15e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:58.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.841+0000 7fc9e15d4640 1 --2- 172.21.15.67:0/4029113675 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9dc0cfc70 0x7fc9dc15e730 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fc9c400c4b0 tx=0x7fc9c400c970 comp rx=0 tx=0).stop 2024-03-24T00:44:58.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.841+0000 7fc9e15d4640 1 -- 172.21.15.67:0/4029113675 shutdown_connections 2024-03-24T00:44:58.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.841+0000 7fc9e15d4640 1 --2- 172.21.15.67:0/4029113675 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc9bc06bbd0 0x7fc9bc06e090 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:58.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.841+0000 7fc9e15d4640 1 --2- 172.21.15.67:0/4029113675 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9dc15bf70 0x7fc9dc15ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:58.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.841+0000 7fc9e15d4640 1 --2- 172.21.15.67:0/4029113675 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9dc0cfc70 0x7fc9dc15e730 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:58.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.841+0000 7fc9e15d4640 1 --2- 172.21.15.67:0/4029113675 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc9dc0ce930 0x7fc9dc1595c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:58.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.841+0000 7fc9e15d4640 1 -- 172.21.15.67:0/4029113675 >> 172.21.15.67:0/4029113675 conn(0x7fc9dc0d1bf0 msgr2=0x7fc9dc08b060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:58.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.841+0000 7fc9e15d4640 1 -- 172.21.15.67:0/4029113675 shutdown_connections 2024-03-24T00:44:58.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:58.841+0000 7fc9e15d4640 1 -- 172.21.15.67:0/4029113675 wait complete. 2024-03-24T00:44:58.851 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":271,"stamp":"2024-03-24T00:44:57.408670+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":246408,"kb_used_data":9448,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754728312,"statfs":{"total":773094113280,"available":772841791488,"internally_reserved":0,"allocated":9674752,"data_stored":5058168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":242535985},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.036782"},"pg_stats":[{"pgid":"14.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.375232+0000","last_change":"2024-03-24T00:44:53.375232+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.375232+0000","last_undegraded":"2024-03-24T00:44:53.375232+0000","last_fullsized":"2024-03-24T00:44:53.375232+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.374673+0000","last_change":"2024-03-24T00:44:53.374673+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.374673+0000","last_undegraded":"2024-03-24T00:44:53.374673+0000","last_fullsized":"2024-03-24T00:44:53.374673+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.377713+0000","last_change":"2024-03-24T00:44:53.377713+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.377713+0000","last_undegraded":"2024-03-24T00:44:53.377713+0000","last_fullsized":"2024-03-24T00:44:53.377713+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.374345+0000","last_change":"2024-03-24T00:44:53.374345+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.374345+0000","last_undegraded":"2024-03-24T00:44:53.374345+0000","last_fullsized":"2024-03-24T00:44:53.374345+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.373750+0000","last_change":"2024-03-24T00:44:53.373750+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.373750+0000","last_undegraded":"2024-03-24T00:44:53.373750+0000","last_fullsized":"2024-03-24T00:44:53.373750+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.375689+0000","last_change":"2024-03-24T00:44:53.375689+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.375689+0000","last_undegraded":"2024-03-24T00:44:53.375689+0000","last_fullsized":"2024-03-24T00:44:53.375689+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.373823+0000","last_change":"2024-03-24T00:44:53.373823+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.373823+0000","last_undegraded":"2024-03-24T00:44:53.373823+0000","last_fullsized":"2024-03-24T00:44:53.373823+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.378054+0000","last_change":"2024-03-24T00:44:53.378054+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.378054+0000","last_undegraded":"2024-03-24T00:44:53.378054+0000","last_fullsized":"2024-03-24T00:44:53.378054+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.377903+0000","last_change":"2024-03-24T00:44:53.377903+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.377903+0000","last_undegraded":"2024-03-24T00:44:53.377903+0000","last_fullsized":"2024-03-24T00:44:53.377903+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.373843+0000","last_change":"2024-03-24T00:44:53.373843+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.373843+0000","last_undegraded":"2024-03-24T00:44:53.373843+0000","last_fullsized":"2024-03-24T00:44:53.373843+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.374785+0000","last_change":"2024-03-24T00:44:53.374785+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.374785+0000","last_undegraded":"2024-03-24T00:44:53.374785+0000","last_fullsized":"2024-03-24T00:44:53.374785+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.376808+0000","last_change":"2024-03-24T00:44:53.376808+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.376808+0000","last_undegraded":"2024-03-24T00:44:53.376808+0000","last_fullsized":"2024-03-24T00:44:53.376808+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.377208+0000","last_change":"2024-03-24T00:44:53.377208+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.377208+0000","last_undegraded":"2024-03-24T00:44:53.377208+0000","last_fullsized":"2024-03-24T00:44:53.377208+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.376814+0000","last_change":"2024-03-24T00:44:53.376814+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.376814+0000","last_undegraded":"2024-03-24T00:44:53.376814+0000","last_fullsized":"2024-03-24T00:44:53.376814+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.375656+0000","last_change":"2024-03-24T00:44:53.375656+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.375656+0000","last_undegraded":"2024-03-24T00:44:53.375656+0000","last_fullsized":"2024-03-24T00:44:53.375656+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.375797+0000","last_change":"2024-03-24T00:44:53.375797+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.375797+0000","last_undegraded":"2024-03-24T00:44:53.375797+0000","last_fullsized":"2024-03-24T00:44:53.375797+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.373960+0000","last_change":"2024-03-24T00:44:53.373960+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.373960+0000","last_undegraded":"2024-03-24T00:44:53.373960+0000","last_fullsized":"2024-03-24T00:44:53.373960+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.376955+0000","last_change":"2024-03-24T00:44:53.376955+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.376955+0000","last_undegraded":"2024-03-24T00:44:53.376955+0000","last_fullsized":"2024-03-24T00:44:53.376955+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.374336+0000","last_change":"2024-03-24T00:44:53.374336+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.374336+0000","last_undegraded":"2024-03-24T00:44:53.374336+0000","last_fullsized":"2024-03-24T00:44:53.374336+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.378153+0000","last_change":"2024-03-24T00:44:53.378153+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.378153+0000","last_undegraded":"2024-03-24T00:44:53.378153+0000","last_fullsized":"2024-03-24T00:44:53.378153+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.376525+0000","last_change":"2024-03-24T00:44:53.376525+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.376525+0000","last_undegraded":"2024-03-24T00:44:53.376525+0000","last_fullsized":"2024-03-24T00:44:53.376525+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.377062+0000","last_change":"2024-03-24T00:44:53.377062+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.377062+0000","last_undegraded":"2024-03-24T00:44:53.377062+0000","last_fullsized":"2024-03-24T00:44:53.377062+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.375988+0000","last_change":"2024-03-24T00:44:53.375988+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.375988+0000","last_undegraded":"2024-03-24T00:44:53.375988+0000","last_fullsized":"2024-03-24T00:44:53.375988+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.376770+0000","last_change":"2024-03-24T00:44:53.376770+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.376770+0000","last_undegraded":"2024-03-24T00:44:53.376770+0000","last_fullsized":"2024-03-24T00:44:53.376770+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.374244+0000","last_change":"2024-03-24T00:44:53.374244+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.374244+0000","last_undegraded":"2024-03-24T00:44:53.374244+0000","last_fullsized":"2024-03-24T00:44:53.374244+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.373845+0000","last_change":"2024-03-24T00:44:53.373845+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.373845+0000","last_undegraded":"2024-03-24T00:44:53.373845+0000","last_fullsized":"2024-03-24T00:44:53.373845+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.373752+0000","last_change":"2024-03-24T00:44:53.373752+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.373752+0000","last_undegraded":"2024-03-24T00:44:53.373752+0000","last_fullsized":"2024-03-24T00:44:53.373752+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.377232+0000","last_change":"2024-03-24T00:44:53.377232+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.377232+0000","last_undegraded":"2024-03-24T00:44:53.377232+0000","last_fullsized":"2024-03-24T00:44:53.377232+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.377878+0000","last_change":"2024-03-24T00:44:53.377878+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.377878+0000","last_undegraded":"2024-03-24T00:44:53.377878+0000","last_fullsized":"2024-03-24T00:44:53.377878+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.373915+0000","last_change":"2024-03-24T00:44:53.373915+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.373915+0000","last_undegraded":"2024-03-24T00:44:53.373915+0000","last_fullsized":"2024-03-24T00:44:53.373915+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.376719+0000","last_change":"2024-03-24T00:44:53.376719+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.376719+0000","last_undegraded":"2024-03-24T00:44:53.376719+0000","last_fullsized":"2024-03-24T00:44:53.376719+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.375668+0000","last_change":"2024-03-24T00:44:53.375668+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.375668+0000","last_undegraded":"2024-03-24T00:44:53.375668+0000","last_fullsized":"2024-03-24T00:44:53.375668+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":293,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:44:53.378811+0000","last_change":"2024-03-24T00:38:40.953871+0000","last_active":"2024-03-24T00:44:53.378811+0000","last_peered":"2024-03-24T00:44:53.378811+0000","last_clean":"2024-03-24T00:44:53.378811+0000","last_became_active":"2024-03-24T00:38:40.952926+0000","last_became_peered":"2024-03-24T00:38:40.952926+0000","last_unstale":"2024-03-24T00:44:53.378811+0000","last_undegraded":"2024-03-24T00:44:53.378811+0000","last_fullsized":"2024-03-24T00:44:53.378811+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_clean_scrub_stamp":"2024-03-24T00:38:39.722194+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:05.397257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574928,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32796,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339044,"statfs":{"total":96636764160,"available":96603181056,"internally_reserved":0,"allocated":1654784,"data_stored":1075047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574927,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28056,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343784,"statfs":{"total":96636764160,"available":96608034816,"internally_reserved":0,"allocated":1060864,"data_stored":484679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574927,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339624,"statfs":{"total":96636764160,"available":96603774976,"internally_reserved":0,"allocated":1060864,"data_stored":484679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574926,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339624,"statfs":{"total":96636764160,"available":96603774976,"internally_reserved":0,"allocated":1060864,"data_stored":484679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574926,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339624,"statfs":{"total":96636764160,"available":96603774976,"internally_reserved":0,"allocated":1060864,"data_stored":484679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574927,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28056,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343784,"statfs":{"total":96636764160,"available":96608034816,"internally_reserved":0,"allocated":1060864,"data_stored":484679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574927,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28056,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343784,"statfs":{"total":96636764160,"available":96608034816,"internally_reserved":0,"allocated":1060864,"data_stored":484679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574926,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32796,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339044,"statfs":{"total":96636764160,"available":96603181056,"internally_reserved":0,"allocated":1654784,"data_stored":1075047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:44:58.855 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:44:59.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.006+0000 7f460bef8640 1 Processor -- start 2024-03-24T00:44:59.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.007+0000 7f460bef8640 1 -- start start 2024-03-24T00:44:59.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.007+0000 7f460bef8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f460415e130 0x7f46040cde00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:59.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.007+0000 7f460bef8640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f46040b5ac0 0x7f46040b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:59.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.007+0000 7f460bef8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46040701e0 con 0x7f46040b2770 2024-03-24T00:44:59.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.007+0000 7f460bef8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46040d00b0 con 0x7f460415e130 2024-03-24T00:44:59.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.007+0000 7f460bef8640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46040704c0 con 0x7f46040b5ac0 2024-03-24T00:44:59.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.008+0000 7f4609c94640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f460415e130 0x7f46040cde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:59.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.008+0000 7f460a495640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f46040b2770 0x7f460406bc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55544/0 (socket says 172.21.15.67:55544) 2024-03-24T00:44:59.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.008+0000 7f4609c94640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f460415e130 0x7f46040cde00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45136/0 (socket says 172.21.15.67:45136) 2024-03-24T00:44:59.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.008+0000 7f460a495640 1 -- 172.21.15.67:0/2053040923 learned_addr learned my addr 172.21.15.67:0/2053040923 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:44:59.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.008+0000 7f4609493640 1 --2- 172.21.15.67:0/2053040923 >> v2:172.21.15.72:3300/0 conn(0x7f46040b5ac0 0x7f46040b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:59.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.008+0000 7f4609c94640 1 -- 172.21.15.67:0/2053040923 >> v2:172.21.15.72:3300/0 conn(0x7f46040b5ac0 msgr2=0x7f46040b4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:59.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.008+0000 7f4609c94640 1 --2- 172.21.15.67:0/2053040923 >> v2:172.21.15.72:3300/0 conn(0x7f46040b5ac0 0x7f46040b4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:59.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.008+0000 7f4609c94640 1 -- 172.21.15.67:0/2053040923 >> v1:172.21.15.67:6789/0 conn(0x7f46040b2770 legacy=0x7f460406bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:59.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.008+0000 7f4609c94640 1 -- 172.21.15.67:0/2053040923 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f460406b170 con 0x7f460415e130 2024-03-24T00:44:59.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.008+0000 7f4608c92640 1 -- 172.21.15.67:0/2053040923 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2787180017 0 0) 0x7f46040701e0 con 0x7f46040b2770 2024-03-24T00:44:59.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.008+0000 7f4609493640 1 --2- 172.21.15.67:0/2053040923 >> v2:172.21.15.72:3300/0 conn(0x7f46040b5ac0 0x7f46040b4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:44:59.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.009+0000 7f4609c94640 1 --2- 172.21.15.67:0/2053040923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f460415e130 0x7f46040cde00 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f4604070d10 tx=0x7f460002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96de2a5dcbe8bf78 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:59.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.009+0000 7f4608c92640 1 -- 172.21.15.67:0/2053040923 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46000315a0 con 0x7f460415e130 2024-03-24T00:44:59.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.009+0000 7f4608c92640 1 -- 172.21.15.67:0/2053040923 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4600037070 con 0x7f460415e130 2024-03-24T00:44:59.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.009+0000 7f4608c92640 1 -- 172.21.15.67:0/2053040923 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4600031960 con 0x7f460415e130 2024-03-24T00:44:59.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.009+0000 7f460bef8640 1 -- 172.21.15.67:0/2053040923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f460415e130 msgr2=0x7f46040cde00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:59.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.009+0000 7f460bef8640 1 --2- 172.21.15.67:0/2053040923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f460415e130 0x7f46040cde00 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f4604070d10 tx=0x7f460002b840 comp rx=0 tx=0).stop 2024-03-24T00:44:59.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.009+0000 7f460bef8640 1 -- 172.21.15.67:0/2053040923 shutdown_connections 2024-03-24T00:44:59.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.009+0000 7f460bef8640 1 --2- 172.21.15.67:0/2053040923 >> v2:172.21.15.72:3300/0 conn(0x7f46040b5ac0 0x7f46040b4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:59.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.009+0000 7f460bef8640 1 --2- 172.21.15.67:0/2053040923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f460415e130 0x7f46040cde00 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:59.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.009+0000 7f460bef8640 1 -- 172.21.15.67:0/2053040923 >> 172.21.15.67:0/2053040923 conn(0x7f46040b3c40 msgr2=0x7f46040b1dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:59.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.009+0000 7f460bef8640 1 -- 172.21.15.67:0/2053040923 shutdown_connections 2024-03-24T00:44:59.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.010+0000 7f460bef8640 1 -- 172.21.15.67:0/2053040923 wait complete. 2024-03-24T00:44:59.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.011+0000 7f460bef8640 1 Processor -- start 2024-03-24T00:44:59.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.011+0000 7f460bef8640 1 -- start start 2024-03-24T00:44:59.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.012+0000 7f460bef8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f46040b2770 0x7f46040c58a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:59.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.012+0000 7f460bef8640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f46040b5ac0 0x7f46041516c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:59.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.012+0000 7f460bef8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f460415e130 0x7f460409d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:59.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.012+0000 7f460bef8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f460415e510 con 0x7f460415e130 2024-03-24T00:44:59.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.012+0000 7f460bef8640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f460406b7c0 con 0x7f46040b2770 2024-03-24T00:44:59.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.012+0000 7f460bef8640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f460406fe00 con 0x7f46040b5ac0 2024-03-24T00:44:59.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.012+0000 7f460a495640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f460415e130 0x7f460409d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:59.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.012+0000 7f460a495640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f460415e130 0x7f460409d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45146/0 (socket says 172.21.15.67:45146) 2024-03-24T00:44:59.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.012+0000 7f460a495640 1 -- 172.21.15.67:0/1102019151 learned_addr learned my addr 172.21.15.67:0/1102019151 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:59.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.012+0000 7f4609c94640 1 --2- 172.21.15.67:0/1102019151 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f46040b2770 0x7f46040c58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:59.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.012+0000 7f4609493640 1 --2- 172.21.15.67:0/1102019151 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f46040b5ac0 0x7f46041516c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:59.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.012+0000 7f460a495640 1 -- 172.21.15.67:0/1102019151 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f46040b5ac0 msgr2=0x7f46041516c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:59.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.012+0000 7f460a495640 1 --2- 172.21.15.67:0/1102019151 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f46040b5ac0 0x7f46041516c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:59.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.012+0000 7f460a495640 1 -- 172.21.15.67:0/1102019151 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f46040b2770 msgr2=0x7f46040c58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:59.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.012+0000 7f460a495640 1 --2- 172.21.15.67:0/1102019151 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f46040b2770 0x7f46040c58a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:59.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.012+0000 7f460a495640 1 -- 172.21.15.67:0/1102019151 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46040c65f0 con 0x7f460415e130 2024-03-24T00:44:59.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.012+0000 7f460a495640 1 --2- 172.21.15.67:0/1102019151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f460415e130 0x7f460409d730 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f45fc00c7b0 tx=0x7f45fc00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:59.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.012+0000 7f45fa7fc640 1 -- 172.21.15.67:0/1102019151 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45fc018070 con 0x7f460415e130 2024-03-24T00:44:59.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.012+0000 7f45fa7fc640 1 -- 172.21.15.67:0/1102019151 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45fc00a640 con 0x7f460415e130 2024-03-24T00:44:59.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.012+0000 7f45fa7fc640 1 -- 172.21.15.67:0/1102019151 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45fc00aa70 con 0x7f460415e130 2024-03-24T00:44:59.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.013+0000 7f460bef8640 1 -- 172.21.15.67:0/1102019151 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46040c6310 con 0x7f460415e130 2024-03-24T00:44:59.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.013+0000 7f460bef8640 1 -- 172.21.15.67:0/1102019151 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46040c50a0 con 0x7f460415e130 2024-03-24T00:44:59.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.013+0000 7f460bef8640 1 -- 172.21.15.67:0/1102019151 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f460404f400 con 0x7f460415e130 2024-03-24T00:44:59.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.014+0000 7f45fa7fc640 1 -- 172.21.15.67:0/1102019151 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f45fc01b070 con 0x7f460415e130 2024-03-24T00:44:59.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.014+0000 7f45fa7fc640 1 --2- 172.21.15.67:0/1102019151 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f45e806bbd0 0x7f45e806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:59.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.014+0000 7f45fa7fc640 1 -- 172.21.15.67:0/1102019151 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f45fc08a140 con 0x7f460415e130 2024-03-24T00:44:59.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.015+0000 7f4609c94640 1 --2- 172.21.15.67:0/1102019151 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f45e806bbd0 0x7f45e806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:59.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.015+0000 7f4609c94640 1 --2- 172.21.15.67:0/1102019151 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f45e806bbd0 0x7f45e806e090 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f4600036000 tx=0x7f4600008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:59.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.019+0000 7f45fa7fc640 1 -- 172.21.15.67:0/1102019151 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f45fc058b30 con 0x7f460415e130 2024-03-24T00:44:59.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.202+0000 7f460bef8640 1 -- 172.21.15.67:0/1102019151 --> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f46040b51f0 con 0x7f45e806bbd0 2024-03-24T00:44:59.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.208+0000 7f45fa7fc640 1 -- 172.21.15.67:0/1102019151 <== mgr.4105 v2:172.21.15.72:6832/2768292274 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240182 (secure 0 0 0) 0x7f46040b51f0 con 0x7f45e806bbd0 2024-03-24T00:44:59.210 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:44:59.211 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-24T00:44:59.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.211+0000 7f460bef8640 1 -- 172.21.15.67:0/1102019151 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f45e806bbd0 msgr2=0x7f45e806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:59.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.211+0000 7f460bef8640 1 --2- 172.21.15.67:0/1102019151 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f45e806bbd0 0x7f45e806e090 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f4600036000 tx=0x7f4600008030 comp rx=0 tx=0).stop 2024-03-24T00:44:59.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.211+0000 7f460bef8640 1 -- 172.21.15.67:0/1102019151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f460415e130 msgr2=0x7f460409d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:59.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.211+0000 7f460bef8640 1 --2- 172.21.15.67:0/1102019151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f460415e130 0x7f460409d730 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f45fc00c7b0 tx=0x7f45fc00cc70 comp rx=0 tx=0).stop 2024-03-24T00:44:59.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.211+0000 7f460bef8640 1 -- 172.21.15.67:0/1102019151 shutdown_connections 2024-03-24T00:44:59.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.211+0000 7f460bef8640 1 --2- 172.21.15.67:0/1102019151 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f45e806bbd0 0x7f45e806e090 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:59.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.211+0000 7f460bef8640 1 --2- 172.21.15.67:0/1102019151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f460415e130 0x7f460409d730 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:59.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.211+0000 7f460bef8640 1 --2- 172.21.15.67:0/1102019151 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f46040b5ac0 0x7f46041516c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:59.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.211+0000 7f460bef8640 1 --2- 172.21.15.67:0/1102019151 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f46040b2770 0x7f46040c58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:59.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.211+0000 7f460bef8640 1 -- 172.21.15.67:0/1102019151 >> 172.21.15.67:0/1102019151 conn(0x7f46040b3c40 msgr2=0x7f460415d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:59.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.211+0000 7f460bef8640 1 -- 172.21.15.67:0/1102019151 shutdown_connections 2024-03-24T00:44:59.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.211+0000 7f460bef8640 1 -- 172.21.15.67:0/1102019151 wait complete. 2024-03-24T00:44:59.222 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":271,"stamp":"2024-03-24T00:44:57.408670+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":246408,"kb_used_data":9448,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754728312,"statfs":{"total":773094113280,"available":772841791488,"internally_reserved":0,"allocated":9674752,"data_stored":5058168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":242535985},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.036782"},"pg_stats":[{"pgid":"14.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.375232+0000","last_change":"2024-03-24T00:44:53.375232+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.375232+0000","last_undegraded":"2024-03-24T00:44:53.375232+0000","last_fullsized":"2024-03-24T00:44:53.375232+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.374673+0000","last_change":"2024-03-24T00:44:53.374673+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.374673+0000","last_undegraded":"2024-03-24T00:44:53.374673+0000","last_fullsized":"2024-03-24T00:44:53.374673+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.377713+0000","last_change":"2024-03-24T00:44:53.377713+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.377713+0000","last_undegraded":"2024-03-24T00:44:53.377713+0000","last_fullsized":"2024-03-24T00:44:53.377713+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.374345+0000","last_change":"2024-03-24T00:44:53.374345+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.374345+0000","last_undegraded":"2024-03-24T00:44:53.374345+0000","last_fullsized":"2024-03-24T00:44:53.374345+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.373750+0000","last_change":"2024-03-24T00:44:53.373750+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.373750+0000","last_undegraded":"2024-03-24T00:44:53.373750+0000","last_fullsized":"2024-03-24T00:44:53.373750+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.375689+0000","last_change":"2024-03-24T00:44:53.375689+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.375689+0000","last_undegraded":"2024-03-24T00:44:53.375689+0000","last_fullsized":"2024-03-24T00:44:53.375689+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.373823+0000","last_change":"2024-03-24T00:44:53.373823+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.373823+0000","last_undegraded":"2024-03-24T00:44:53.373823+0000","last_fullsized":"2024-03-24T00:44:53.373823+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.378054+0000","last_change":"2024-03-24T00:44:53.378054+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.378054+0000","last_undegraded":"2024-03-24T00:44:53.378054+0000","last_fullsized":"2024-03-24T00:44:53.378054+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.377903+0000","last_change":"2024-03-24T00:44:53.377903+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.377903+0000","last_undegraded":"2024-03-24T00:44:53.377903+0000","last_fullsized":"2024-03-24T00:44:53.377903+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.373843+0000","last_change":"2024-03-24T00:44:53.373843+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.373843+0000","last_undegraded":"2024-03-24T00:44:53.373843+0000","last_fullsized":"2024-03-24T00:44:53.373843+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.374785+0000","last_change":"2024-03-24T00:44:53.374785+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.374785+0000","last_undegraded":"2024-03-24T00:44:53.374785+0000","last_fullsized":"2024-03-24T00:44:53.374785+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.376808+0000","last_change":"2024-03-24T00:44:53.376808+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.376808+0000","last_undegraded":"2024-03-24T00:44:53.376808+0000","last_fullsized":"2024-03-24T00:44:53.376808+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.377208+0000","last_change":"2024-03-24T00:44:53.377208+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.377208+0000","last_undegraded":"2024-03-24T00:44:53.377208+0000","last_fullsized":"2024-03-24T00:44:53.377208+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.376814+0000","last_change":"2024-03-24T00:44:53.376814+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.376814+0000","last_undegraded":"2024-03-24T00:44:53.376814+0000","last_fullsized":"2024-03-24T00:44:53.376814+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.375656+0000","last_change":"2024-03-24T00:44:53.375656+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.375656+0000","last_undegraded":"2024-03-24T00:44:53.375656+0000","last_fullsized":"2024-03-24T00:44:53.375656+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.375797+0000","last_change":"2024-03-24T00:44:53.375797+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.375797+0000","last_undegraded":"2024-03-24T00:44:53.375797+0000","last_fullsized":"2024-03-24T00:44:53.375797+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.373960+0000","last_change":"2024-03-24T00:44:53.373960+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.373960+0000","last_undegraded":"2024-03-24T00:44:53.373960+0000","last_fullsized":"2024-03-24T00:44:53.373960+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.376955+0000","last_change":"2024-03-24T00:44:53.376955+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.376955+0000","last_undegraded":"2024-03-24T00:44:53.376955+0000","last_fullsized":"2024-03-24T00:44:53.376955+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.374336+0000","last_change":"2024-03-24T00:44:53.374336+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.374336+0000","last_undegraded":"2024-03-24T00:44:53.374336+0000","last_fullsized":"2024-03-24T00:44:53.374336+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.378153+0000","last_change":"2024-03-24T00:44:53.378153+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.378153+0000","last_undegraded":"2024-03-24T00:44:53.378153+0000","last_fullsized":"2024-03-24T00:44:53.378153+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.376525+0000","last_change":"2024-03-24T00:44:53.376525+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.376525+0000","last_undegraded":"2024-03-24T00:44:53.376525+0000","last_fullsized":"2024-03-24T00:44:53.376525+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.377062+0000","last_change":"2024-03-24T00:44:53.377062+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.377062+0000","last_undegraded":"2024-03-24T00:44:53.377062+0000","last_fullsized":"2024-03-24T00:44:53.377062+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.375988+0000","last_change":"2024-03-24T00:44:53.375988+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.375988+0000","last_undegraded":"2024-03-24T00:44:53.375988+0000","last_fullsized":"2024-03-24T00:44:53.375988+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.376770+0000","last_change":"2024-03-24T00:44:53.376770+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.376770+0000","last_undegraded":"2024-03-24T00:44:53.376770+0000","last_fullsized":"2024-03-24T00:44:53.376770+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.374244+0000","last_change":"2024-03-24T00:44:53.374244+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.374244+0000","last_undegraded":"2024-03-24T00:44:53.374244+0000","last_fullsized":"2024-03-24T00:44:53.374244+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.373845+0000","last_change":"2024-03-24T00:44:53.373845+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.373845+0000","last_undegraded":"2024-03-24T00:44:53.373845+0000","last_fullsized":"2024-03-24T00:44:53.373845+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.373752+0000","last_change":"2024-03-24T00:44:53.373752+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.373752+0000","last_undegraded":"2024-03-24T00:44:53.373752+0000","last_fullsized":"2024-03-24T00:44:53.373752+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.377232+0000","last_change":"2024-03-24T00:44:53.377232+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.377232+0000","last_undegraded":"2024-03-24T00:44:53.377232+0000","last_fullsized":"2024-03-24T00:44:53.377232+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.377878+0000","last_change":"2024-03-24T00:44:53.377878+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.377878+0000","last_undegraded":"2024-03-24T00:44:53.377878+0000","last_fullsized":"2024-03-24T00:44:53.377878+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.373915+0000","last_change":"2024-03-24T00:44:53.373915+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.373915+0000","last_undegraded":"2024-03-24T00:44:53.373915+0000","last_fullsized":"2024-03-24T00:44:53.373915+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.376719+0000","last_change":"2024-03-24T00:44:53.376719+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.376719+0000","last_undegraded":"2024-03-24T00:44:53.376719+0000","last_fullsized":"2024-03-24T00:44:53.376719+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:44:54.371295+0000","last_change":"2024-03-24T00:44:54.371295+0000","last_active":"2024-03-24T00:44:54.371295+0000","last_peered":"2024-03-24T00:44:54.371295+0000","last_clean":"2024-03-24T00:44:54.371295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:54.371295+0000","last_undegraded":"2024-03-24T00:44:54.371295+0000","last_fullsized":"2024-03-24T00:44:54.371295+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":1,"reported_epoch":86,"state":"creating+peering","last_fresh":"2024-03-24T00:44:53.375668+0000","last_change":"2024-03-24T00:44:53.375668+0000","last_active":"2024-03-24T00:44:53.369112+0000","last_peered":"2024-03-24T00:44:53.369112+0000","last_clean":"2024-03-24T00:44:53.369112+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:44:53.375668+0000","last_undegraded":"2024-03-24T00:44:53.375668+0000","last_fullsized":"2024-03-24T00:44:53.375668+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":293,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:44:53.378811+0000","last_change":"2024-03-24T00:38:40.953871+0000","last_active":"2024-03-24T00:44:53.378811+0000","last_peered":"2024-03-24T00:44:53.378811+0000","last_clean":"2024-03-24T00:44:53.378811+0000","last_became_active":"2024-03-24T00:38:40.952926+0000","last_became_peered":"2024-03-24T00:38:40.952926+0000","last_unstale":"2024-03-24T00:44:53.378811+0000","last_undegraded":"2024-03-24T00:44:53.378811+0000","last_fullsized":"2024-03-24T00:44:53.378811+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_clean_scrub_stamp":"2024-03-24T00:38:39.722194+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:05.397257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574928,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32796,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339044,"statfs":{"total":96636764160,"available":96603181056,"internally_reserved":0,"allocated":1654784,"data_stored":1075047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574927,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28056,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343784,"statfs":{"total":96636764160,"available":96608034816,"internally_reserved":0,"allocated":1060864,"data_stored":484679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574927,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339624,"statfs":{"total":96636764160,"available":96603774976,"internally_reserved":0,"allocated":1060864,"data_stored":484679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574926,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339624,"statfs":{"total":96636764160,"available":96603774976,"internally_reserved":0,"allocated":1060864,"data_stored":484679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574926,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339624,"statfs":{"total":96636764160,"available":96603774976,"internally_reserved":0,"allocated":1060864,"data_stored":484679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574927,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28056,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343784,"statfs":{"total":96636764160,"available":96608034816,"internally_reserved":0,"allocated":1060864,"data_stored":484679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574927,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28056,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343784,"statfs":{"total":96636764160,"available":96608034816,"internally_reserved":0,"allocated":1060864,"data_stored":484679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574926,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32796,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339044,"statfs":{"total":96636764160,"available":96603181056,"internally_reserved":0,"allocated":1654784,"data_stored":1075047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:44:59.226 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.3f is not active+clean 2024-03-24T00:44:59.226 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.226 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.3e is not active+clean 2024-03-24T00:44:59.226 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.226 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.3d is not active+clean 2024-03-24T00:44:59.226 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.226 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.3c is not active+clean 2024-03-24T00:44:59.226 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.226 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.3b is not active+clean 2024-03-24T00:44:59.226 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.227 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.3a is not active+clean 2024-03-24T00:44:59.227 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.3a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.227 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.39 is not active+clean 2024-03-24T00:44:59.227 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.227 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.38 is not active+clean 2024-03-24T00:44:59.227 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.227 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.37 is not active+clean 2024-03-24T00:44:59.227 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.227 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.36 is not active+clean 2024-03-24T00:44:59.227 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.227 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.35 is not active+clean 2024-03-24T00:44:59.227 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.228 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.34 is not active+clean 2024-03-24T00:44:59.228 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.228 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.33 is not active+clean 2024-03-24T00:44:59.228 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.228 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.32 is not active+clean 2024-03-24T00:44:59.228 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.228 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.31 is not active+clean 2024-03-24T00:44:59.228 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.229 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.30 is not active+clean 2024-03-24T00:44:59.229 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.229 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.2f is not active+clean 2024-03-24T00:44:59.229 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.229 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.2e is not active+clean 2024-03-24T00:44:59.229 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.229 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.2d is not active+clean 2024-03-24T00:44:59.229 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.229 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.2c is not active+clean 2024-03-24T00:44:59.229 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.2c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.229 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.2b is not active+clean 2024-03-24T00:44:59.229 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.230 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.2a is not active+clean 2024-03-24T00:44:59.230 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.230 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.29 is not active+clean 2024-03-24T00:44:59.230 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.230 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.28 is not active+clean 2024-03-24T00:44:59.230 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.230 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.27 is not active+clean 2024-03-24T00:44:59.230 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.27', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.230 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.26 is not active+clean 2024-03-24T00:44:59.230 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.230 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.25 is not active+clean 2024-03-24T00:44:59.231 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.231 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.24 is not active+clean 2024-03-24T00:44:59.231 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.231 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.23 is not active+clean 2024-03-24T00:44:59.231 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.231 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.22 is not active+clean 2024-03-24T00:44:59.231 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.231 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.21 is not active+clean 2024-03-24T00:44:59.231 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.231 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.20 is not active+clean 2024-03-24T00:44:59.231 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.232 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.1c is not active+clean 2024-03-24T00:44:59.232 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.232 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1f is not active+clean 2024-03-24T00:44:59.232 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 86, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:44:53.375232+0000', 'last_change': '2024-03-24T00:44:53.375232+0000', 'last_active': '2024-03-24T00:44:53.369112+0000', 'last_peered': '2024-03-24T00:44:53.369112+0000', 'last_clean': '2024-03-24T00:44:53.369112+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:53.375232+0000', 'last_undegraded': '2024-03-24T00:44:53.375232+0000', 'last_fullsized': '2024-03-24T00:44:53.375232+0000', 'mapping_epoch': 86, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 86, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-24T00:44:59.232 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.1d is not active+clean 2024-03-24T00:44:59.232 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.232 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1e is not active+clean 2024-03-24T00:44:59.232 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 86, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:44:53.374673+0000', 'last_change': '2024-03-24T00:44:53.374673+0000', 'last_active': '2024-03-24T00:44:53.369112+0000', 'last_peered': '2024-03-24T00:44:53.369112+0000', 'last_clean': '2024-03-24T00:44:53.369112+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:53.374673+0000', 'last_undegraded': '2024-03-24T00:44:53.374673+0000', 'last_fullsized': '2024-03-24T00:44:53.374673+0000', 'mapping_epoch': 86, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 86, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-24T00:44:59.232 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.1e is not active+clean 2024-03-24T00:44:59.232 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.233 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1d is not active+clean 2024-03-24T00:44:59.233 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 86, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:44:53.377713+0000', 'last_change': '2024-03-24T00:44:53.377713+0000', 'last_active': '2024-03-24T00:44:53.369112+0000', 'last_peered': '2024-03-24T00:44:53.369112+0000', 'last_clean': '2024-03-24T00:44:53.369112+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:53.377713+0000', 'last_undegraded': '2024-03-24T00:44:53.377713+0000', 'last_fullsized': '2024-03-24T00:44:53.377713+0000', 'mapping_epoch': 86, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 86, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T00:44:59.233 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.1f is not active+clean 2024-03-24T00:44:59.233 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.233 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1c is not active+clean 2024-03-24T00:44:59.233 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 86, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:44:53.374345+0000', 'last_change': '2024-03-24T00:44:53.374345+0000', 'last_active': '2024-03-24T00:44:53.369112+0000', 'last_peered': '2024-03-24T00:44:53.369112+0000', 'last_clean': '2024-03-24T00:44:53.369112+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:53.374345+0000', 'last_undegraded': '2024-03-24T00:44:53.374345+0000', 'last_fullsized': '2024-03-24T00:44:53.374345+0000', 'mapping_epoch': 86, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 86, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-24T00:44:59.233 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.18 is not active+clean 2024-03-24T00:44:59.233 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.233 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1b is not active+clean 2024-03-24T00:44:59.233 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 86, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:44:53.373750+0000', 'last_change': '2024-03-24T00:44:53.373750+0000', 'last_active': '2024-03-24T00:44:53.369112+0000', 'last_peered': '2024-03-24T00:44:53.369112+0000', 'last_clean': '2024-03-24T00:44:53.369112+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:53.373750+0000', 'last_undegraded': '2024-03-24T00:44:53.373750+0000', 'last_fullsized': '2024-03-24T00:44:53.373750+0000', 'mapping_epoch': 86, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 86, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-24T00:44:59.233 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.19 is not active+clean 2024-03-24T00:44:59.233 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.234 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1a is not active+clean 2024-03-24T00:44:59.234 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 86, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:44:53.375689+0000', 'last_change': '2024-03-24T00:44:53.375689+0000', 'last_active': '2024-03-24T00:44:53.369112+0000', 'last_peered': '2024-03-24T00:44:53.369112+0000', 'last_clean': '2024-03-24T00:44:53.369112+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:53.375689+0000', 'last_undegraded': '2024-03-24T00:44:53.375689+0000', 'last_fullsized': '2024-03-24T00:44:53.375689+0000', 'mapping_epoch': 86, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 86, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-24T00:44:59.234 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.1a is not active+clean 2024-03-24T00:44:59.234 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.234 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.19 is not active+clean 2024-03-24T00:44:59.234 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 86, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:44:53.373823+0000', 'last_change': '2024-03-24T00:44:53.373823+0000', 'last_active': '2024-03-24T00:44:53.369112+0000', 'last_peered': '2024-03-24T00:44:53.369112+0000', 'last_clean': '2024-03-24T00:44:53.369112+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:53.373823+0000', 'last_undegraded': '2024-03-24T00:44:53.373823+0000', 'last_fullsized': '2024-03-24T00:44:53.373823+0000', 'mapping_epoch': 86, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 86, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T00:44:59.234 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.1b is not active+clean 2024-03-24T00:44:59.234 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.234 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.18 is not active+clean 2024-03-24T00:44:59.234 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 86, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:44:53.378054+0000', 'last_change': '2024-03-24T00:44:53.378054+0000', 'last_active': '2024-03-24T00:44:53.369112+0000', 'last_peered': '2024-03-24T00:44:53.369112+0000', 'last_clean': '2024-03-24T00:44:53.369112+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:53.378054+0000', 'last_undegraded': '2024-03-24T00:44:53.378054+0000', 'last_fullsized': '2024-03-24T00:44:53.378054+0000', 'mapping_epoch': 86, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 86, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T00:44:59.234 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.14 is not active+clean 2024-03-24T00:44:59.234 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.235 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.17 is not active+clean 2024-03-24T00:44:59.235 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 86, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:44:53.377903+0000', 'last_change': '2024-03-24T00:44:53.377903+0000', 'last_active': '2024-03-24T00:44:53.369112+0000', 'last_peered': '2024-03-24T00:44:53.369112+0000', 'last_clean': '2024-03-24T00:44:53.369112+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:53.377903+0000', 'last_undegraded': '2024-03-24T00:44:53.377903+0000', 'last_fullsized': '2024-03-24T00:44:53.377903+0000', 'mapping_epoch': 86, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 86, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-24T00:44:59.235 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.15 is not active+clean 2024-03-24T00:44:59.235 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.235 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.16 is not active+clean 2024-03-24T00:44:59.235 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 86, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:44:53.373843+0000', 'last_change': '2024-03-24T00:44:53.373843+0000', 'last_active': '2024-03-24T00:44:53.369112+0000', 'last_peered': '2024-03-24T00:44:53.369112+0000', 'last_clean': '2024-03-24T00:44:53.369112+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:53.373843+0000', 'last_undegraded': '2024-03-24T00:44:53.373843+0000', 'last_fullsized': '2024-03-24T00:44:53.373843+0000', 'mapping_epoch': 86, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 86, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T00:44:59.235 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.16 is not active+clean 2024-03-24T00:44:59.235 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.235 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.15 is not active+clean 2024-03-24T00:44:59.235 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 86, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:44:53.374785+0000', 'last_change': '2024-03-24T00:44:53.374785+0000', 'last_active': '2024-03-24T00:44:53.369112+0000', 'last_peered': '2024-03-24T00:44:53.369112+0000', 'last_clean': '2024-03-24T00:44:53.369112+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:53.374785+0000', 'last_undegraded': '2024-03-24T00:44:53.374785+0000', 'last_fullsized': '2024-03-24T00:44:53.374785+0000', 'mapping_epoch': 86, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 86, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-24T00:44:59.235 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.17 is not active+clean 2024-03-24T00:44:59.236 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.236 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.14 is not active+clean 2024-03-24T00:44:59.236 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 86, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:44:53.376808+0000', 'last_change': '2024-03-24T00:44:53.376808+0000', 'last_active': '2024-03-24T00:44:53.369112+0000', 'last_peered': '2024-03-24T00:44:53.369112+0000', 'last_clean': '2024-03-24T00:44:53.369112+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:53.376808+0000', 'last_undegraded': '2024-03-24T00:44:53.376808+0000', 'last_fullsized': '2024-03-24T00:44:53.376808+0000', 'mapping_epoch': 86, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 86, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T00:44:59.236 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.10 is not active+clean 2024-03-24T00:44:59.236 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.236 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.13 is not active+clean 2024-03-24T00:44:59.236 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 86, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:44:53.377208+0000', 'last_change': '2024-03-24T00:44:53.377208+0000', 'last_active': '2024-03-24T00:44:53.369112+0000', 'last_peered': '2024-03-24T00:44:53.369112+0000', 'last_clean': '2024-03-24T00:44:53.369112+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:53.377208+0000', 'last_undegraded': '2024-03-24T00:44:53.377208+0000', 'last_fullsized': '2024-03-24T00:44:53.377208+0000', 'mapping_epoch': 86, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 86, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T00:44:59.236 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.11 is not active+clean 2024-03-24T00:44:59.236 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.236 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.12 is not active+clean 2024-03-24T00:44:59.236 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 86, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:44:53.376814+0000', 'last_change': '2024-03-24T00:44:53.376814+0000', 'last_active': '2024-03-24T00:44:53.369112+0000', 'last_peered': '2024-03-24T00:44:53.369112+0000', 'last_clean': '2024-03-24T00:44:53.369112+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:53.376814+0000', 'last_undegraded': '2024-03-24T00:44:53.376814+0000', 'last_fullsized': '2024-03-24T00:44:53.376814+0000', 'mapping_epoch': 86, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 86, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T00:44:59.237 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.12 is not active+clean 2024-03-24T00:44:59.237 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.237 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.11 is not active+clean 2024-03-24T00:44:59.237 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 86, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:44:53.375656+0000', 'last_change': '2024-03-24T00:44:53.375656+0000', 'last_active': '2024-03-24T00:44:53.369112+0000', 'last_peered': '2024-03-24T00:44:53.369112+0000', 'last_clean': '2024-03-24T00:44:53.369112+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:53.375656+0000', 'last_undegraded': '2024-03-24T00:44:53.375656+0000', 'last_fullsized': '2024-03-24T00:44:53.375656+0000', 'mapping_epoch': 86, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 86, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-24T00:44:59.237 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.13 is not active+clean 2024-03-24T00:44:59.237 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.237 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.10 is not active+clean 2024-03-24T00:44:59.237 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 86, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:44:53.375797+0000', 'last_change': '2024-03-24T00:44:53.375797+0000', 'last_active': '2024-03-24T00:44:53.369112+0000', 'last_peered': '2024-03-24T00:44:53.369112+0000', 'last_clean': '2024-03-24T00:44:53.369112+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:53.375797+0000', 'last_undegraded': '2024-03-24T00:44:53.375797+0000', 'last_fullsized': '2024-03-24T00:44:53.375797+0000', 'mapping_epoch': 86, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 86, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-24T00:44:59.237 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.c is not active+clean 2024-03-24T00:44:59.237 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.237 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.f is not active+clean 2024-03-24T00:44:59.238 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 86, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:44:53.373960+0000', 'last_change': '2024-03-24T00:44:53.373960+0000', 'last_active': '2024-03-24T00:44:53.369112+0000', 'last_peered': '2024-03-24T00:44:53.369112+0000', 'last_clean': '2024-03-24T00:44:53.369112+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:53.373960+0000', 'last_undegraded': '2024-03-24T00:44:53.373960+0000', 'last_fullsized': '2024-03-24T00:44:53.373960+0000', 'mapping_epoch': 86, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 86, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-24T00:44:59.238 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.d is not active+clean 2024-03-24T00:44:59.238 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.238 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.e is not active+clean 2024-03-24T00:44:59.238 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 86, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:44:53.376955+0000', 'last_change': '2024-03-24T00:44:53.376955+0000', 'last_active': '2024-03-24T00:44:53.369112+0000', 'last_peered': '2024-03-24T00:44:53.369112+0000', 'last_clean': '2024-03-24T00:44:53.369112+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:53.376955+0000', 'last_undegraded': '2024-03-24T00:44:53.376955+0000', 'last_fullsized': '2024-03-24T00:44:53.376955+0000', 'mapping_epoch': 86, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 86, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T00:44:59.238 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.e is not active+clean 2024-03-24T00:44:59.238 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.238 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.d is not active+clean 2024-03-24T00:44:59.238 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 86, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:44:53.374336+0000', 'last_change': '2024-03-24T00:44:53.374336+0000', 'last_active': '2024-03-24T00:44:53.369112+0000', 'last_peered': '2024-03-24T00:44:53.369112+0000', 'last_clean': '2024-03-24T00:44:53.369112+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:53.374336+0000', 'last_undegraded': '2024-03-24T00:44:53.374336+0000', 'last_fullsized': '2024-03-24T00:44:53.374336+0000', 'mapping_epoch': 86, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 86, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-24T00:44:59.238 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.8 is not active+clean 2024-03-24T00:44:59.238 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.239 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.b is not active+clean 2024-03-24T00:44:59.239 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 86, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:44:53.378153+0000', 'last_change': '2024-03-24T00:44:53.378153+0000', 'last_active': '2024-03-24T00:44:53.369112+0000', 'last_peered': '2024-03-24T00:44:53.369112+0000', 'last_clean': '2024-03-24T00:44:53.369112+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:53.378153+0000', 'last_undegraded': '2024-03-24T00:44:53.378153+0000', 'last_fullsized': '2024-03-24T00:44:53.378153+0000', 'mapping_epoch': 86, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 86, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-24T00:44:59.239 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.9 is not active+clean 2024-03-24T00:44:59.239 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.239 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.a is not active+clean 2024-03-24T00:44:59.239 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 86, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:44:53.376525+0000', 'last_change': '2024-03-24T00:44:53.376525+0000', 'last_active': '2024-03-24T00:44:53.369112+0000', 'last_peered': '2024-03-24T00:44:53.369112+0000', 'last_clean': '2024-03-24T00:44:53.369112+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:53.376525+0000', 'last_undegraded': '2024-03-24T00:44:53.376525+0000', 'last_fullsized': '2024-03-24T00:44:53.376525+0000', 'mapping_epoch': 86, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 86, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-24T00:44:59.239 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.a is not active+clean 2024-03-24T00:44:59.239 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.239 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.9 is not active+clean 2024-03-24T00:44:59.239 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 86, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:44:53.377062+0000', 'last_change': '2024-03-24T00:44:53.377062+0000', 'last_active': '2024-03-24T00:44:53.369112+0000', 'last_peered': '2024-03-24T00:44:53.369112+0000', 'last_clean': '2024-03-24T00:44:53.369112+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:53.377062+0000', 'last_undegraded': '2024-03-24T00:44:53.377062+0000', 'last_fullsized': '2024-03-24T00:44:53.377062+0000', 'mapping_epoch': 86, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 86, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-24T00:44:59.239 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.b is not active+clean 2024-03-24T00:44:59.239 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.240 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.8 is not active+clean 2024-03-24T00:44:59.240 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 86, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:44:53.375988+0000', 'last_change': '2024-03-24T00:44:53.375988+0000', 'last_active': '2024-03-24T00:44:53.369112+0000', 'last_peered': '2024-03-24T00:44:53.369112+0000', 'last_clean': '2024-03-24T00:44:53.369112+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:53.375988+0000', 'last_undegraded': '2024-03-24T00:44:53.375988+0000', 'last_fullsized': '2024-03-24T00:44:53.375988+0000', 'mapping_epoch': 86, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 86, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-24T00:44:59.240 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.4 is not active+clean 2024-03-24T00:44:59.240 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.240 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.7 is not active+clean 2024-03-24T00:44:59.240 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 86, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:44:53.376770+0000', 'last_change': '2024-03-24T00:44:53.376770+0000', 'last_active': '2024-03-24T00:44:53.369112+0000', 'last_peered': '2024-03-24T00:44:53.369112+0000', 'last_clean': '2024-03-24T00:44:53.369112+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:53.376770+0000', 'last_undegraded': '2024-03-24T00:44:53.376770+0000', 'last_fullsized': '2024-03-24T00:44:53.376770+0000', 'mapping_epoch': 86, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 86, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T00:44:59.240 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.5 is not active+clean 2024-03-24T00:44:59.240 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.240 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.6 is not active+clean 2024-03-24T00:44:59.240 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 86, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:44:53.374244+0000', 'last_change': '2024-03-24T00:44:53.374244+0000', 'last_active': '2024-03-24T00:44:53.369112+0000', 'last_peered': '2024-03-24T00:44:53.369112+0000', 'last_clean': '2024-03-24T00:44:53.369112+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:53.374244+0000', 'last_undegraded': '2024-03-24T00:44:53.374244+0000', 'last_fullsized': '2024-03-24T00:44:53.374244+0000', 'mapping_epoch': 86, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 86, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-24T00:44:59.240 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.6 is not active+clean 2024-03-24T00:44:59.240 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.241 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.5 is not active+clean 2024-03-24T00:44:59.241 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 86, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:44:53.373845+0000', 'last_change': '2024-03-24T00:44:53.373845+0000', 'last_active': '2024-03-24T00:44:53.369112+0000', 'last_peered': '2024-03-24T00:44:53.369112+0000', 'last_clean': '2024-03-24T00:44:53.369112+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:53.373845+0000', 'last_undegraded': '2024-03-24T00:44:53.373845+0000', 'last_fullsized': '2024-03-24T00:44:53.373845+0000', 'mapping_epoch': 86, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 86, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-24T00:44:59.241 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.7 is not active+clean 2024-03-24T00:44:59.241 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.241 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.4 is not active+clean 2024-03-24T00:44:59.241 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 86, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:44:53.373752+0000', 'last_change': '2024-03-24T00:44:53.373752+0000', 'last_active': '2024-03-24T00:44:53.369112+0000', 'last_peered': '2024-03-24T00:44:53.369112+0000', 'last_clean': '2024-03-24T00:44:53.369112+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:53.373752+0000', 'last_undegraded': '2024-03-24T00:44:53.373752+0000', 'last_fullsized': '2024-03-24T00:44:53.373752+0000', 'mapping_epoch': 86, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 86, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-24T00:44:59.241 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.0 is not active+clean 2024-03-24T00:44:59.242 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.242 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.3 is not active+clean 2024-03-24T00:44:59.242 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 86, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:44:53.377232+0000', 'last_change': '2024-03-24T00:44:53.377232+0000', 'last_active': '2024-03-24T00:44:53.369112+0000', 'last_peered': '2024-03-24T00:44:53.369112+0000', 'last_clean': '2024-03-24T00:44:53.369112+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:53.377232+0000', 'last_undegraded': '2024-03-24T00:44:53.377232+0000', 'last_fullsized': '2024-03-24T00:44:53.377232+0000', 'mapping_epoch': 86, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 86, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T00:44:59.242 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.1 is not active+clean 2024-03-24T00:44:59.242 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.242 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.2 is not active+clean 2024-03-24T00:44:59.242 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 86, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:44:53.377878+0000', 'last_change': '2024-03-24T00:44:53.377878+0000', 'last_active': '2024-03-24T00:44:53.369112+0000', 'last_peered': '2024-03-24T00:44:53.369112+0000', 'last_clean': '2024-03-24T00:44:53.369112+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:53.377878+0000', 'last_undegraded': '2024-03-24T00:44:53.377878+0000', 'last_fullsized': '2024-03-24T00:44:53.377878+0000', 'mapping_epoch': 86, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 86, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-24T00:44:59.242 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.2 is not active+clean 2024-03-24T00:44:59.242 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.242 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1 is not active+clean 2024-03-24T00:44:59.242 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 86, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:44:53.373915+0000', 'last_change': '2024-03-24T00:44:53.373915+0000', 'last_active': '2024-03-24T00:44:53.369112+0000', 'last_peered': '2024-03-24T00:44:53.369112+0000', 'last_clean': '2024-03-24T00:44:53.369112+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:53.373915+0000', 'last_undegraded': '2024-03-24T00:44:53.373915+0000', 'last_fullsized': '2024-03-24T00:44:53.373915+0000', 'mapping_epoch': 86, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 86, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-24T00:44:59.243 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.3 is not active+clean 2024-03-24T00:44:59.243 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.243 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.0 is not active+clean 2024-03-24T00:44:59.243 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 86, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:44:53.376719+0000', 'last_change': '2024-03-24T00:44:53.376719+0000', 'last_active': '2024-03-24T00:44:53.369112+0000', 'last_peered': '2024-03-24T00:44:53.369112+0000', 'last_clean': '2024-03-24T00:44:53.369112+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:53.376719+0000', 'last_undegraded': '2024-03-24T00:44:53.376719+0000', 'last_fullsized': '2024-03-24T00:44:53.376719+0000', 'mapping_epoch': 86, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 86, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T00:44:59.243 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.f is not active+clean 2024-03-24T00:44:59.243 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:44:54.371295+0000', 'last_change': '2024-03-24T00:44:54.371295+0000', 'last_active': '2024-03-24T00:44:54.371295+0000', 'last_peered': '2024-03-24T00:44:54.371295+0000', 'last_clean': '2024-03-24T00:44:54.371295+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:54.371295+0000', 'last_undegraded': '2024-03-24T00:44:54.371295+0000', 'last_fullsized': '2024-03-24T00:44:54.371295+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:54.371295+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:44:59.243 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.c is not active+clean 2024-03-24T00:44:59.243 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 86, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:44:53.375668+0000', 'last_change': '2024-03-24T00:44:53.375668+0000', 'last_active': '2024-03-24T00:44:53.369112+0000', 'last_peered': '2024-03-24T00:44:53.369112+0000', 'last_clean': '2024-03-24T00:44:53.369112+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:44:53.375668+0000', 'last_undegraded': '2024-03-24T00:44:53.375668+0000', 'last_fullsized': '2024-03-24T00:44:53.375668+0000', 'mapping_epoch': 86, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 86, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'last_clean_scrub_stamp': '2024-03-24T00:44:53.369112+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-24T00:44:59.244 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-24T00:44:59.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.364+0000 7f9ee8789640 1 Processor -- start 2024-03-24T00:44:59.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.364+0000 7f9ee8789640 1 -- start start 2024-03-24T00:44:59.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.365+0000 7f9ee8789640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f9ee00b49c0 0x7f9ee015b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:59.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.365+0000 7f9ee8789640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9ee006f100 0x7f9ee0072360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:59.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.366+0000 7f9ee8789640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ee00b4da0 con 0x7f9ee00728a0 2024-03-24T00:44:59.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.366+0000 7f9ee8789640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ee0072d10 con 0x7f9ee006f100 2024-03-24T00:44:59.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.366+0000 7f9ee8789640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ee006db30 con 0x7f9ee00b49c0 2024-03-24T00:44:59.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.366+0000 7f9ee5d24640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9ee006f100 0x7f9ee0072360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:59.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.366+0000 7f9ee5d24640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9ee006f100 0x7f9ee0072360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45154/0 (socket says 172.21.15.67:45154) 2024-03-24T00:44:59.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.366+0000 7f9ee5d24640 1 -- 172.21.15.67:0/3056701941 learned_addr learned my addr 172.21.15.67:0/3056701941 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:59.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.366+0000 7f9ee6525640 1 --2- 172.21.15.67:0/3056701941 >> v2:172.21.15.72:3300/0 conn(0x7f9ee00b49c0 0x7f9ee015b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:59.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.366+0000 7f9ee5d24640 1 -- 172.21.15.67:0/3056701941 >> v2:172.21.15.72:3300/0 conn(0x7f9ee00b49c0 msgr2=0x7f9ee015b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:59.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.366+0000 7f9ee5d24640 1 --2- 172.21.15.67:0/3056701941 >> v2:172.21.15.72:3300/0 conn(0x7f9ee00b49c0 0x7f9ee015b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:59.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.366+0000 7f9ee5d24640 1 -- 172.21.15.67:0/3056701941 >> v1:172.21.15.72:6789/0 conn(0x7f9ee00728a0 legacy=0x7f9ee015df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:59.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.366+0000 7f9ee5d24640 1 -- 172.21.15.67:0/3056701941 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ee006de10 con 0x7f9ee006f100 2024-03-24T00:44:59.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.366+0000 7f9ee6525640 1 --2- 172.21.15.67:0/3056701941 >> v2:172.21.15.72:3300/0 conn(0x7f9ee00b49c0 0x7f9ee015b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:44:59.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.367+0000 7f9ee5d24640 1 --2- 172.21.15.67:0/3056701941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ee006f100 0x7f9ee0072360 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f9ee0072f50 tx=0x7f9ed002ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c28b49a582f9249 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:59.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.367+0000 7f9ee5523640 1 -- 172.21.15.67:0/3056701941 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed0030840 con 0x7f9ee006f100 2024-03-24T00:44:59.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.367+0000 7f9ee5523640 1 -- 172.21.15.67:0/3056701941 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ed00309e0 con 0x7f9ee006f100 2024-03-24T00:44:59.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.367+0000 7f9ee5523640 1 -- 172.21.15.67:0/3056701941 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed0030db0 con 0x7f9ee006f100 2024-03-24T00:44:59.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.368+0000 7f9ee8789640 1 -- 172.21.15.67:0/3056701941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ee006f100 msgr2=0x7f9ee0072360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:59.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.368+0000 7f9ee8789640 1 --2- 172.21.15.67:0/3056701941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ee006f100 0x7f9ee0072360 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f9ee0072f50 tx=0x7f9ed002ca20 comp rx=0 tx=0).stop 2024-03-24T00:44:59.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.368+0000 7f9ee8789640 1 -- 172.21.15.67:0/3056701941 shutdown_connections 2024-03-24T00:44:59.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.368+0000 7f9ee8789640 1 --2- 172.21.15.67:0/3056701941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ee006f100 0x7f9ee0072360 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:59.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.368+0000 7f9ee8789640 1 --2- 172.21.15.67:0/3056701941 >> v2:172.21.15.72:3300/0 conn(0x7f9ee00b49c0 0x7f9ee015b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:59.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.368+0000 7f9ee8789640 1 -- 172.21.15.67:0/3056701941 >> 172.21.15.67:0/3056701941 conn(0x7f9ee00cf9e0 msgr2=0x7f9ee00cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:59.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.368+0000 7f9ee8789640 1 -- 172.21.15.67:0/3056701941 shutdown_connections 2024-03-24T00:44:59.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.368+0000 7f9ee8789640 1 -- 172.21.15.67:0/3056701941 wait complete. 2024-03-24T00:44:59.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.369+0000 7f9ee8789640 1 Processor -- start 2024-03-24T00:44:59.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.369+0000 7f9ee8789640 1 -- start start 2024-03-24T00:44:59.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.370+0000 7f9ee8789640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ee006f100 0x7f9ee00aee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:59.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.370+0000 7f9ee8789640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ee00728a0 0x7f9ee01591c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:59.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.370+0000 7f9ee8789640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9ee00b49c0 0x7f9ee0159700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:59.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.370+0000 7f9ee8789640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee006e6b0 con 0x7f9ee00728a0 2024-03-24T00:44:59.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.370+0000 7f9ee8789640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee006e3d0 con 0x7f9ee006f100 2024-03-24T00:44:59.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.370+0000 7f9ee8789640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee006f590 con 0x7f9ee00b49c0 2024-03-24T00:44:59.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.370+0000 7f9ee5d24640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ee00728a0 0x7f9ee01591c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:59.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.370+0000 7f9ee5d24640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ee00728a0 0x7f9ee01591c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45170/0 (socket says 172.21.15.67:45170) 2024-03-24T00:44:59.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.370+0000 7f9ee5d24640 1 -- 172.21.15.67:0/3017244456 learned_addr learned my addr 172.21.15.67:0/3017244456 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:59.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.370+0000 7f9ee6525640 1 --2- 172.21.15.67:0/3017244456 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ee006f100 0x7f9ee00aee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:59.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.370+0000 7f9ee6d26640 1 --2- 172.21.15.67:0/3017244456 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9ee00b49c0 0x7f9ee0159700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:59.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.371+0000 7f9ee5d24640 1 -- 172.21.15.67:0/3017244456 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9ee00b49c0 msgr2=0x7f9ee0159700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:59.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.371+0000 7f9ee5d24640 1 --2- 172.21.15.67:0/3017244456 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9ee00b49c0 0x7f9ee0159700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:59.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.371+0000 7f9ee5d24640 1 -- 172.21.15.67:0/3017244456 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ee006f100 msgr2=0x7f9ee00aee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:59.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.371+0000 7f9ee5d24640 1 --2- 172.21.15.67:0/3017244456 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ee006f100 0x7f9ee00aee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:59.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.371+0000 7f9ee5d24640 1 -- 172.21.15.67:0/3017244456 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ee0147ff0 con 0x7f9ee00728a0 2024-03-24T00:44:59.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.371+0000 7f9ee5d24640 1 --2- 172.21.15.67:0/3017244456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ee00728a0 0x7f9ee01591c0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f9ed002ca60 tx=0x7f9ed0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:59.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.371+0000 7f9ed6ffd640 1 -- 172.21.15.67:0/3017244456 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed0047070 con 0x7f9ee00728a0 2024-03-24T00:44:59.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.371+0000 7f9ed6ffd640 1 -- 172.21.15.67:0/3017244456 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ed0038b10 con 0x7f9ee00728a0 2024-03-24T00:44:59.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.371+0000 7f9ed6ffd640 1 -- 172.21.15.67:0/3017244456 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed0042410 con 0x7f9ee00728a0 2024-03-24T00:44:59.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.371+0000 7f9ee8789640 1 -- 172.21.15.67:0/3017244456 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ee01482d0 con 0x7f9ee00728a0 2024-03-24T00:44:59.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.371+0000 7f9ee8789640 1 -- 172.21.15.67:0/3017244456 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ee0148890 con 0x7f9ee00728a0 2024-03-24T00:44:59.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.372+0000 7f9ee8789640 1 -- 172.21.15.67:0/3017244456 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ee00af3a0 con 0x7f9ee00728a0 2024-03-24T00:44:59.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.373+0000 7f9ed6ffd640 1 -- 172.21.15.67:0/3017244456 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9ed002e3d0 con 0x7f9ee00728a0 2024-03-24T00:44:59.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.373+0000 7f9ed6ffd640 1 --2- 172.21.15.67:0/3017244456 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9ec406bbd0 0x7f9ec406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:59.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.373+0000 7f9ed6ffd640 1 -- 172.21.15.67:0/3017244456 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9ed00aca00 con 0x7f9ee00728a0 2024-03-24T00:44:59.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.373+0000 7f9ee6525640 1 --2- 172.21.15.67:0/3017244456 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9ec406bbd0 0x7f9ec406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:59.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.374+0000 7f9ee6525640 1 --2- 172.21.15.67:0/3017244456 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9ec406bbd0 0x7f9ec406e090 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f9edc006fd0 tx=0x7f9edc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:59.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.377+0000 7f9ed6ffd640 1 -- 172.21.15.67:0/3017244456 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9ee00af3a0 con 0x7f9ee00728a0 2024-03-24T00:44:59.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.747+0000 7f9ee8789640 1 -- 172.21.15.67:0/3017244456 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f9ee004f400 con 0x7f9ee00728a0 2024-03-24T00:44:59.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.747+0000 7f9ed6ffd640 1 -- 172.21.15.67:0/3017244456 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1110 (secure 0 0 0) 0x7f9ed0034030 con 0x7f9ee00728a0 2024-03-24T00:44:59.747 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:44:59.747 INFO:teuthology.orchestra.run.smithi067.stdout:{"fsid":"de1d0178-808e-49ca-a986-e297e523b976","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":392,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":89,"num_osds":8,"num_up_osds":8,"osd_up_since":1711240718,"num_in_osds":8,"osd_in_since":1711240709,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"unknown","count":64},{"state_name":"creating+peering","count":32},{"state_name":"active+clean","count":1}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":252321792,"bytes_avail":772841791488,"bytes_total":773094113280,"unknown_pgs_ratio":0.65979379415512085,"inactive_pgs_ratio":0.32989689707756042},"fsmap":{"epoch":54,"id":6,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":6,"rank":0,"name":"c","status":"up:active","gid":7617}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-03-24T00:44:43.405823+0000","services":{}},"progress_events":{}} 2024-03-24T00:44:59.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.750+0000 7f9ee8789640 1 -- 172.21.15.67:0/3017244456 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9ec406bbd0 msgr2=0x7f9ec406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:59.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.750+0000 7f9ee8789640 1 --2- 172.21.15.67:0/3017244456 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9ec406bbd0 0x7f9ec406e090 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f9edc006fd0 tx=0x7f9edc008040 comp rx=0 tx=0).stop 2024-03-24T00:44:59.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.750+0000 7f9ee8789640 1 -- 172.21.15.67:0/3017244456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ee00728a0 msgr2=0x7f9ee01591c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:59.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.750+0000 7f9ee8789640 1 --2- 172.21.15.67:0/3017244456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ee00728a0 0x7f9ee01591c0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f9ed002ca60 tx=0x7f9ed0004730 comp rx=0 tx=0).stop 2024-03-24T00:44:59.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.750+0000 7f9ee8789640 1 -- 172.21.15.67:0/3017244456 shutdown_connections 2024-03-24T00:44:59.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.750+0000 7f9ee8789640 1 --2- 172.21.15.67:0/3017244456 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9ec406bbd0 0x7f9ec406e090 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:59.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.750+0000 7f9ee8789640 1 --2- 172.21.15.67:0/3017244456 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9ee00b49c0 0x7f9ee0159700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:59.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.750+0000 7f9ee8789640 1 --2- 172.21.15.67:0/3017244456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ee00728a0 0x7f9ee01591c0 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:59.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.750+0000 7f9ee8789640 1 --2- 172.21.15.67:0/3017244456 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ee006f100 0x7f9ee00aee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:59.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.750+0000 7f9ee8789640 1 -- 172.21.15.67:0/3017244456 >> 172.21.15.67:0/3017244456 conn(0x7f9ee00cf9e0 msgr2=0x7f9ee015bae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:59.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.750+0000 7f9ee8789640 1 -- 172.21.15.67:0/3017244456 shutdown_connections 2024-03-24T00:44:59.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.750+0000 7f9ee8789640 1 -- 172.21.15.67:0/3017244456 wait complete. 2024-03-24T00:44:59.761 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-24T00:44:59.761 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:44:59.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.862+0000 7f85e09b3640 1 Processor -- start 2024-03-24T00:44:59.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.862+0000 7f85e09b3640 1 -- start start 2024-03-24T00:44:59.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.863+0000 7f85e09b3640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f85dc15df30 0x7f85dc162610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:59.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.863+0000 7f85e09b3640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f85dc15e310 0x7f85dc0d0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:59.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.863+0000 7f85e09b3640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85dc0ce920 con 0x7f85dc0d0b80 2024-03-24T00:44:59.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.863+0000 7f85e09b3640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85dc162b50 con 0x7f85dc15e310 2024-03-24T00:44:59.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.863+0000 7f85e09b3640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85dc0d0fd0 con 0x7f85dc15df30 2024-03-24T00:44:59.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.864+0000 7f85d9d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f85dc15e310 0x7f85dc0d0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:59.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.864+0000 7f85d9d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f85dc15e310 0x7f85dc0d0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45186/0 (socket says 172.21.15.67:45186) 2024-03-24T00:44:59.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.864+0000 7f85d9d9b640 1 -- 172.21.15.67:0/184550682 learned_addr learned my addr 172.21.15.67:0/184550682 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:59.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.864+0000 7f85da59c640 1 --2- 172.21.15.67:0/184550682 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f85dc15df30 0x7f85dc162610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:59.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.864+0000 7f85d9d9b640 1 -- 172.21.15.67:0/184550682 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f85dc15df30 msgr2=0x7f85dc162610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:44:59.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.864+0000 7f85d9d9b640 1 --2- 172.21.15.67:0/184550682 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f85dc15df30 0x7f85dc162610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:59.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.864+0000 7f85d9d9b640 1 -- 172.21.15.67:0/184550682 >> v1:172.21.15.72:6789/0 conn(0x7f85dc0d0b80 legacy=0x7f85dc0b57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:59.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.864+0000 7f85d9d9b640 1 -- 172.21.15.67:0/184550682 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85dc0cef20 con 0x7f85dc15e310 2024-03-24T00:44:59.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.865+0000 7f85d9d9b640 1 --2- 172.21.15.67:0/184550682 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85dc15e310 0x7f85dc0d0640 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f85d0009e80 tx=0x7f85d002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be587d81cdd09e04 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:59.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.865+0000 7f85d959a640 1 -- 172.21.15.67:0/184550682 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d002ca50 con 0x7f85dc15e310 2024-03-24T00:44:59.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.865+0000 7f85d959a640 1 -- 172.21.15.67:0/184550682 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85d002cbf0 con 0x7f85dc15e310 2024-03-24T00:44:59.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.865+0000 7f85d959a640 1 -- 172.21.15.67:0/184550682 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d00343f0 con 0x7f85dc15e310 2024-03-24T00:44:59.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.865+0000 7f85e09b3640 1 -- 172.21.15.67:0/184550682 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85dc15e310 msgr2=0x7f85dc0d0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:59.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.865+0000 7f85e09b3640 1 --2- 172.21.15.67:0/184550682 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85dc15e310 0x7f85dc0d0640 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f85d0009e80 tx=0x7f85d002beb0 comp rx=0 tx=0).stop 2024-03-24T00:44:59.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.865+0000 7f85e09b3640 1 -- 172.21.15.67:0/184550682 shutdown_connections 2024-03-24T00:44:59.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.865+0000 7f85e09b3640 1 --2- 172.21.15.67:0/184550682 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85dc15e310 0x7f85dc0d0640 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:59.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.865+0000 7f85e09b3640 1 --2- 172.21.15.67:0/184550682 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f85dc15df30 0x7f85dc162610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:59.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.865+0000 7f85e09b3640 1 -- 172.21.15.67:0/184550682 >> 172.21.15.67:0/184550682 conn(0x7f85dc06c880 msgr2=0x7f85dc06cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:44:59.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.865+0000 7f85e09b3640 1 -- 172.21.15.67:0/184550682 shutdown_connections 2024-03-24T00:44:59.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.865+0000 7f85e09b3640 1 -- 172.21.15.67:0/184550682 wait complete. 2024-03-24T00:44:59.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.867+0000 7f85e09b3640 1 Processor -- start 2024-03-24T00:44:59.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.867+0000 7f85e09b3640 1 -- start start 2024-03-24T00:44:59.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.869+0000 7f85e09b3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85dc0d0b80 0x7f85dc1f3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:59.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.869+0000 7f85e09b3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85dc15df30 0x7f85dc15f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:59.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.869+0000 7f85e09b3640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f85dc15f650 0x7f85dc1f4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:59.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.869+0000 7f85e09b3640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85dc0b32f0 con 0x7f85dc0d0b80 2024-03-24T00:44:59.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.869+0000 7f85e09b3640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85dc0cf4e0 con 0x7f85dc15df30 2024-03-24T00:44:59.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.869+0000 7f85e09b3640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85dc0ce580 con 0x7f85dc15f650 2024-03-24T00:44:59.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.869+0000 7f85da59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85dc0d0b80 0x7f85dc1f3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:59.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.869+0000 7f85da59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85dc0d0b80 0x7f85dc1f3770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45202/0 (socket says 172.21.15.67:45202) 2024-03-24T00:44:59.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.869+0000 7f85da59c640 1 -- 172.21.15.67:0/3674460347 learned_addr learned my addr 172.21.15.67:0/3674460347 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:44:59.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.869+0000 7f85dad9d640 1 --2- 172.21.15.67:0/3674460347 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f85dc15f650 0x7f85dc1f4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:59.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.869+0000 7f85d9d9b640 1 --2- 172.21.15.67:0/3674460347 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85dc15df30 0x7f85dc15f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:59.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.869+0000 7f85da59c640 1 -- 172.21.15.67:0/3674460347 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f85dc15f650 msgr2=0x7f85dc1f4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:59.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.869+0000 7f85da59c640 1 --2- 172.21.15.67:0/3674460347 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f85dc15f650 0x7f85dc1f4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:59.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.869+0000 7f85da59c640 1 -- 172.21.15.67:0/3674460347 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85dc15df30 msgr2=0x7f85dc15f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:44:59.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.869+0000 7f85da59c640 1 --2- 172.21.15.67:0/3674460347 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85dc15df30 0x7f85dc15f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:44:59.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.869+0000 7f85da59c640 1 -- 172.21.15.67:0/3674460347 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85dc1606c0 con 0x7f85dc0d0b80 2024-03-24T00:44:59.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.869+0000 7f85d9d9b640 1 --2- 172.21.15.67:0/3674460347 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85dc15df30 0x7f85dc15f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:44:59.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.869+0000 7f85da59c640 1 --2- 172.21.15.67:0/3674460347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85dc0d0b80 0x7f85dc1f3770 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f85d400e9b0 tx=0x7f85d400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:59.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.869+0000 7f85caffd640 1 -- 172.21.15.67:0/3674460347 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d4017070 con 0x7f85dc0d0b80 2024-03-24T00:44:59.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.870+0000 7f85caffd640 1 -- 172.21.15.67:0/3674460347 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85d400c9a0 con 0x7f85dc0d0b80 2024-03-24T00:44:59.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.870+0000 7f85e09b3640 1 -- 172.21.15.67:0/3674460347 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85dc15fb40 con 0x7f85dc0d0b80 2024-03-24T00:44:59.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.870+0000 7f85caffd640 1 -- 172.21.15.67:0/3674460347 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d400cd90 con 0x7f85dc0d0b80 2024-03-24T00:44:59.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.870+0000 7f85e09b3640 1 -- 172.21.15.67:0/3674460347 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85dc160100 con 0x7f85dc0d0b80 2024-03-24T00:44:59.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.871+0000 7f85e09b3640 1 -- 172.21.15.67:0/3674460347 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85dc04f400 con 0x7f85dc0d0b80 2024-03-24T00:44:59.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.871+0000 7f85caffd640 1 -- 172.21.15.67:0/3674460347 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f85d401a070 con 0x7f85dc0d0b80 2024-03-24T00:44:59.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.872+0000 7f85caffd640 1 --2- 172.21.15.67:0/3674460347 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f85bc06bbd0 0x7f85bc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:44:59.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.872+0000 7f85caffd640 1 -- 172.21.15.67:0/3674460347 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f85d408abc0 con 0x7f85dc0d0b80 2024-03-24T00:44:59.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.872+0000 7f85d9d9b640 1 --2- 172.21.15.67:0/3674460347 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f85bc06bbd0 0x7f85bc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:44:59.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.872+0000 7f85d9d9b640 1 --2- 172.21.15.67:0/3674460347 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f85bc06bbd0 0x7f85bc06e090 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f85d0035000 tx=0x7f85d0030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:44:59.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:44:59.876+0000 7f85caffd640 1 -- 172.21.15.67:0/3674460347 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f85d4058ed0 con 0x7f85dc0d0b80 2024-03-24T00:45:00.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:00.057+0000 7f85e09b3640 1 -- 172.21.15.67:0/3674460347 --> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f85dc1604a0 con 0x7f85bc06bbd0 2024-03-24T00:45:00.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:00.063+0000 7f85caffd640 1 -- 172.21.15.67:0/3674460347 <== mgr.4105 v2:172.21.15.72:6832/2768292274 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252075 (secure 0 0 0) 0x7f85dc1604a0 con 0x7f85bc06bbd0 2024-03-24T00:45:00.067 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:45:00.069 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-24T00:45:00.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:00.066+0000 7f85e09b3640 1 -- 172.21.15.67:0/3674460347 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f85bc06bbd0 msgr2=0x7f85bc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:00.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:00.066+0000 7f85e09b3640 1 --2- 172.21.15.67:0/3674460347 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f85bc06bbd0 0x7f85bc06e090 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f85d0035000 tx=0x7f85d0030000 comp rx=0 tx=0).stop 2024-03-24T00:45:00.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:00.066+0000 7f85e09b3640 1 -- 172.21.15.67:0/3674460347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85dc0d0b80 msgr2=0x7f85dc1f3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:00.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:00.066+0000 7f85e09b3640 1 --2- 172.21.15.67:0/3674460347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85dc0d0b80 0x7f85dc1f3770 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f85d400e9b0 tx=0x7f85d400ee70 comp rx=0 tx=0).stop 2024-03-24T00:45:00.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:00.066+0000 7f85e09b3640 1 -- 172.21.15.67:0/3674460347 shutdown_connections 2024-03-24T00:45:00.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:00.066+0000 7f85e09b3640 1 --2- 172.21.15.67:0/3674460347 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f85bc06bbd0 0x7f85bc06e090 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:00.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:00.066+0000 7f85e09b3640 1 --2- 172.21.15.67:0/3674460347 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f85dc15f650 0x7f85dc1f4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:00.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:00.066+0000 7f85e09b3640 1 --2- 172.21.15.67:0/3674460347 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85dc15df30 0x7f85dc15f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:00.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:00.066+0000 7f85e09b3640 1 --2- 172.21.15.67:0/3674460347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85dc0d0b80 0x7f85dc1f3770 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:00.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:00.066+0000 7f85e09b3640 1 -- 172.21.15.67:0/3674460347 >> 172.21.15.67:0/3674460347 conn(0x7f85dc06c880 msgr2=0x7f85dc0c7b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:00.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:00.066+0000 7f85e09b3640 1 -- 172.21.15.67:0/3674460347 shutdown_connections 2024-03-24T00:45:00.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:00.066+0000 7f85e09b3640 1 -- 172.21.15.67:0/3674460347 wait complete. 2024-03-24T00:45:00.077 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":272,"stamp":"2024-03-24T00:44:59.408998+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":184,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":259268,"kb_used_data":9800,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754715452,"statfs":{"total":773094113280,"available":772828622848,"internally_reserved":0,"allocated":10035200,"data_stored":5227196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001218"},"pg_stats":[{"pgid":"14.3f","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.514662+0000","last_change":"2024-03-24T00:44:55.387093+0000","last_active":"2024-03-24T00:44:57.514662+0000","last_peered":"2024-03-24T00:44:57.514662+0000","last_clean":"2024-03-24T00:44:57.514662+0000","last_became_active":"2024-03-24T00:44:55.386764+0000","last_became_peered":"2024-03-24T00:44:55.386764+0000","last_unstale":"2024-03-24T00:44:57.514662+0000","last_undegraded":"2024-03-24T00:44:57.514662+0000","last_fullsized":"2024-03-24T00:44:57.514662+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:17:11.603293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.385353+0000","last_change":"2024-03-24T00:44:55.393287+0000","last_active":"2024-03-24T00:44:56.385353+0000","last_peered":"2024-03-24T00:44:56.385353+0000","last_clean":"2024-03-24T00:44:56.385353+0000","last_became_active":"2024-03-24T00:44:55.393012+0000","last_became_peered":"2024-03-24T00:44:55.393012+0000","last_unstale":"2024-03-24T00:44:56.385353+0000","last_undegraded":"2024-03-24T00:44:56.385353+0000","last_fullsized":"2024-03-24T00:44:56.385353+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:45:58.413741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3d","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.514517+0000","last_change":"2024-03-24T00:44:55.390994+0000","last_active":"2024-03-24T00:44:57.514517+0000","last_peered":"2024-03-24T00:44:57.514517+0000","last_clean":"2024-03-24T00:44:57.514517+0000","last_became_active":"2024-03-24T00:44:55.390746+0000","last_became_peered":"2024-03-24T00:44:55.390746+0000","last_unstale":"2024-03-24T00:44:57.514517+0000","last_undegraded":"2024-03-24T00:44:57.514517+0000","last_fullsized":"2024-03-24T00:44:57.514517+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:19:39.501718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.777676+0000","last_change":"2024-03-24T00:44:55.396111+0000","last_active":"2024-03-24T00:44:57.777676+0000","last_peered":"2024-03-24T00:44:57.777676+0000","last_clean":"2024-03-24T00:44:57.777676+0000","last_became_active":"2024-03-24T00:44:55.395858+0000","last_became_peered":"2024-03-24T00:44:55.395858+0000","last_unstale":"2024-03-24T00:44:57.777676+0000","last_undegraded":"2024-03-24T00:44:57.777676+0000","last_fullsized":"2024-03-24T00:44:57.777676+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:19:21.001867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.386233+0000","last_change":"2024-03-24T00:44:55.392701+0000","last_active":"2024-03-24T00:44:56.386233+0000","last_peered":"2024-03-24T00:44:56.386233+0000","last_clean":"2024-03-24T00:44:56.386233+0000","last_became_active":"2024-03-24T00:44:55.392361+0000","last_became_peered":"2024-03-24T00:44:55.392361+0000","last_unstale":"2024-03-24T00:44:56.386233+0000","last_undegraded":"2024-03-24T00:44:56.386233+0000","last_fullsized":"2024-03-24T00:44:56.386233+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:16:07.455446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.382497+0000","last_change":"2024-03-24T00:44:55.411755+0000","last_active":"2024-03-24T00:44:56.382497+0000","last_peered":"2024-03-24T00:44:56.382497+0000","last_clean":"2024-03-24T00:44:56.382497+0000","last_became_active":"2024-03-24T00:44:55.411469+0000","last_became_peered":"2024-03-24T00:44:55.411469+0000","last_unstale":"2024-03-24T00:44:56.382497+0000","last_undegraded":"2024-03-24T00:44:56.382497+0000","last_fullsized":"2024-03-24T00:44:56.382497+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:50:38.597482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.729131+0000","last_change":"2024-03-24T00:44:55.398900+0000","last_active":"2024-03-24T00:44:56.729131+0000","last_peered":"2024-03-24T00:44:56.729131+0000","last_clean":"2024-03-24T00:44:56.729131+0000","last_became_active":"2024-03-24T00:44:55.398002+0000","last_became_peered":"2024-03-24T00:44:55.398002+0000","last_unstale":"2024-03-24T00:44:56.729131+0000","last_undegraded":"2024-03-24T00:44:56.729131+0000","last_fullsized":"2024-03-24T00:44:56.729131+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:39:23.097646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.777262+0000","last_change":"2024-03-24T00:44:55.407339+0000","last_active":"2024-03-24T00:44:57.777262+0000","last_peered":"2024-03-24T00:44:57.777262+0000","last_clean":"2024-03-24T00:44:57.777262+0000","last_became_active":"2024-03-24T00:44:55.407066+0000","last_became_peered":"2024-03-24T00:44:55.407066+0000","last_unstale":"2024-03-24T00:44:57.777262+0000","last_undegraded":"2024-03-24T00:44:57.777262+0000","last_fullsized":"2024-03-24T00:44:57.777262+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:33:37.494533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.386774+0000","last_change":"2024-03-24T00:44:55.396053+0000","last_active":"2024-03-24T00:44:56.386774+0000","last_peered":"2024-03-24T00:44:56.386774+0000","last_clean":"2024-03-24T00:44:56.386774+0000","last_became_active":"2024-03-24T00:44:55.395764+0000","last_became_peered":"2024-03-24T00:44:55.395764+0000","last_unstale":"2024-03-24T00:44:56.386774+0000","last_undegraded":"2024-03-24T00:44:56.386774+0000","last_fullsized":"2024-03-24T00:44:56.386774+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:18:39.887633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.777394+0000","last_change":"2024-03-24T00:44:55.407603+0000","last_active":"2024-03-24T00:44:57.777394+0000","last_peered":"2024-03-24T00:44:57.777394+0000","last_clean":"2024-03-24T00:44:57.777394+0000","last_became_active":"2024-03-24T00:44:55.407319+0000","last_became_peered":"2024-03-24T00:44:55.407319+0000","last_unstale":"2024-03-24T00:44:57.777394+0000","last_undegraded":"2024-03-24T00:44:57.777394+0000","last_fullsized":"2024-03-24T00:44:57.777394+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:03:07.602179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.386770+0000","last_change":"2024-03-24T00:44:55.404378+0000","last_active":"2024-03-24T00:44:56.386770+0000","last_peered":"2024-03-24T00:44:56.386770+0000","last_clean":"2024-03-24T00:44:56.386770+0000","last_became_active":"2024-03-24T00:44:55.404111+0000","last_became_peered":"2024-03-24T00:44:55.404111+0000","last_unstale":"2024-03-24T00:44:56.386770+0000","last_undegraded":"2024-03-24T00:44:56.386770+0000","last_fullsized":"2024-03-24T00:44:56.386770+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:16:13.352120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.776854+0000","last_change":"2024-03-24T00:44:55.398924+0000","last_active":"2024-03-24T00:44:57.776854+0000","last_peered":"2024-03-24T00:44:57.776854+0000","last_clean":"2024-03-24T00:44:57.776854+0000","last_became_active":"2024-03-24T00:44:55.398672+0000","last_became_peered":"2024-03-24T00:44:55.398672+0000","last_unstale":"2024-03-24T00:44:57.776854+0000","last_undegraded":"2024-03-24T00:44:57.776854+0000","last_fullsized":"2024-03-24T00:44:57.776854+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:26:29.358335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.777450+0000","last_change":"2024-03-24T00:44:55.393133+0000","last_active":"2024-03-24T00:44:57.777450+0000","last_peered":"2024-03-24T00:44:57.777450+0000","last_clean":"2024-03-24T00:44:57.777450+0000","last_became_active":"2024-03-24T00:44:55.392877+0000","last_became_peered":"2024-03-24T00:44:55.392877+0000","last_unstale":"2024-03-24T00:44:57.777450+0000","last_undegraded":"2024-03-24T00:44:57.777450+0000","last_fullsized":"2024-03-24T00:44:57.777450+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:46:50.358541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.386025+0000","last_change":"2024-03-24T00:44:55.398283+0000","last_active":"2024-03-24T00:44:56.386025+0000","last_peered":"2024-03-24T00:44:56.386025+0000","last_clean":"2024-03-24T00:44:56.386025+0000","last_became_active":"2024-03-24T00:44:55.398003+0000","last_became_peered":"2024-03-24T00:44:55.398003+0000","last_unstale":"2024-03-24T00:44:56.386025+0000","last_undegraded":"2024-03-24T00:44:56.386025+0000","last_fullsized":"2024-03-24T00:44:56.386025+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:56:22.901806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.777011+0000","last_change":"2024-03-24T00:44:55.398640+0000","last_active":"2024-03-24T00:44:57.777011+0000","last_peered":"2024-03-24T00:44:57.777011+0000","last_clean":"2024-03-24T00:44:57.777011+0000","last_became_active":"2024-03-24T00:44:55.398396+0000","last_became_peered":"2024-03-24T00:44:55.398396+0000","last_unstale":"2024-03-24T00:44:57.777011+0000","last_undegraded":"2024-03-24T00:44:57.777011+0000","last_fullsized":"2024-03-24T00:44:57.777011+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:50:18.227568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.382424+0000","last_change":"2024-03-24T00:44:55.407535+0000","last_active":"2024-03-24T00:44:56.382424+0000","last_peered":"2024-03-24T00:44:56.382424+0000","last_clean":"2024-03-24T00:44:56.382424+0000","last_became_active":"2024-03-24T00:44:55.407267+0000","last_became_peered":"2024-03-24T00:44:55.407267+0000","last_unstale":"2024-03-24T00:44:56.382424+0000","last_undegraded":"2024-03-24T00:44:56.382424+0000","last_fullsized":"2024-03-24T00:44:56.382424+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:36:59.898017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.382873+0000","last_change":"2024-03-24T00:44:55.398350+0000","last_active":"2024-03-24T00:44:56.382873+0000","last_peered":"2024-03-24T00:44:56.382873+0000","last_clean":"2024-03-24T00:44:56.382873+0000","last_became_active":"2024-03-24T00:44:55.396755+0000","last_became_peered":"2024-03-24T00:44:55.396755+0000","last_unstale":"2024-03-24T00:44:56.382873+0000","last_undegraded":"2024-03-24T00:44:56.382873+0000","last_fullsized":"2024-03-24T00:44:56.382873+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:17:48.749325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.514362+0000","last_change":"2024-03-24T00:44:55.385251+0000","last_active":"2024-03-24T00:44:57.514362+0000","last_peered":"2024-03-24T00:44:57.514362+0000","last_clean":"2024-03-24T00:44:57.514362+0000","last_became_active":"2024-03-24T00:44:55.384991+0000","last_became_peered":"2024-03-24T00:44:55.384991+0000","last_unstale":"2024-03-24T00:44:57.514362+0000","last_undegraded":"2024-03-24T00:44:57.514362+0000","last_fullsized":"2024-03-24T00:44:57.514362+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:45:52.622468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.386621+0000","last_change":"2024-03-24T00:44:55.398470+0000","last_active":"2024-03-24T00:44:56.386621+0000","last_peered":"2024-03-24T00:44:56.386621+0000","last_clean":"2024-03-24T00:44:56.386621+0000","last_became_active":"2024-03-24T00:44:55.397407+0000","last_became_peered":"2024-03-24T00:44:55.397407+0000","last_unstale":"2024-03-24T00:44:56.386621+0000","last_undegraded":"2024-03-24T00:44:56.386621+0000","last_fullsized":"2024-03-24T00:44:56.386621+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:00:26.656749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.777726+0000","last_change":"2024-03-24T00:44:55.397440+0000","last_active":"2024-03-24T00:44:57.777726+0000","last_peered":"2024-03-24T00:44:57.777726+0000","last_clean":"2024-03-24T00:44:57.777726+0000","last_became_active":"2024-03-24T00:44:55.397147+0000","last_became_peered":"2024-03-24T00:44:55.397147+0000","last_unstale":"2024-03-24T00:44:57.777726+0000","last_undegraded":"2024-03-24T00:44:57.777726+0000","last_fullsized":"2024-03-24T00:44:57.777726+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:59:59.398432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.386157+0000","last_change":"2024-03-24T00:44:55.390870+0000","last_active":"2024-03-24T00:44:56.386157+0000","last_peered":"2024-03-24T00:44:56.386157+0000","last_clean":"2024-03-24T00:44:56.386157+0000","last_became_active":"2024-03-24T00:44:55.390570+0000","last_became_peered":"2024-03-24T00:44:55.390570+0000","last_unstale":"2024-03-24T00:44:56.386157+0000","last_undegraded":"2024-03-24T00:44:56.386157+0000","last_fullsized":"2024-03-24T00:44:56.386157+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:18:31.095437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.386062+0000","last_change":"2024-03-24T00:44:55.397511+0000","last_active":"2024-03-24T00:44:56.386062+0000","last_peered":"2024-03-24T00:44:56.386062+0000","last_clean":"2024-03-24T00:44:56.386062+0000","last_became_active":"2024-03-24T00:44:55.397188+0000","last_became_peered":"2024-03-24T00:44:55.397188+0000","last_unstale":"2024-03-24T00:44:56.386062+0000","last_undegraded":"2024-03-24T00:44:56.386062+0000","last_fullsized":"2024-03-24T00:44:56.386062+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:13:32.135329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.406289+0000","last_change":"2024-03-24T00:44:55.399158+0000","last_active":"2024-03-24T00:44:57.406289+0000","last_peered":"2024-03-24T00:44:57.406289+0000","last_clean":"2024-03-24T00:44:57.406289+0000","last_became_active":"2024-03-24T00:44:55.398538+0000","last_became_peered":"2024-03-24T00:44:55.398538+0000","last_unstale":"2024-03-24T00:44:57.406289+0000","last_undegraded":"2024-03-24T00:44:57.406289+0000","last_fullsized":"2024-03-24T00:44:57.406289+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:18:31.095437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.514125+0000","last_change":"2024-03-24T00:44:55.388719+0000","last_active":"2024-03-24T00:44:57.514125+0000","last_peered":"2024-03-24T00:44:57.514125+0000","last_clean":"2024-03-24T00:44:57.514125+0000","last_became_active":"2024-03-24T00:44:55.387921+0000","last_became_peered":"2024-03-24T00:44:55.387921+0000","last_unstale":"2024-03-24T00:44:57.514125+0000","last_undegraded":"2024-03-24T00:44:57.514125+0000","last_fullsized":"2024-03-24T00:44:57.514125+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:39:23.097646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.385512+0000","last_change":"2024-03-24T00:44:55.392532+0000","last_active":"2024-03-24T00:44:56.385512+0000","last_peered":"2024-03-24T00:44:56.385512+0000","last_clean":"2024-03-24T00:44:56.385512+0000","last_became_active":"2024-03-24T00:44:55.392200+0000","last_became_peered":"2024-03-24T00:44:55.392200+0000","last_unstale":"2024-03-24T00:44:56.385512+0000","last_undegraded":"2024-03-24T00:44:56.385512+0000","last_fullsized":"2024-03-24T00:44:56.385512+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:38:36.247055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.386585+0000","last_change":"2024-03-24T00:44:55.397669+0000","last_active":"2024-03-24T00:44:56.386585+0000","last_peered":"2024-03-24T00:44:56.386585+0000","last_clean":"2024-03-24T00:44:56.386585+0000","last_became_active":"2024-03-24T00:44:55.397301+0000","last_became_peered":"2024-03-24T00:44:55.397301+0000","last_unstale":"2024-03-24T00:44:56.386585+0000","last_undegraded":"2024-03-24T00:44:56.386585+0000","last_fullsized":"2024-03-24T00:44:56.386585+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:44:48.580022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.777003+0000","last_change":"2024-03-24T00:44:55.398384+0000","last_active":"2024-03-24T00:44:57.777003+0000","last_peered":"2024-03-24T00:44:57.777003+0000","last_clean":"2024-03-24T00:44:57.777003+0000","last_became_active":"2024-03-24T00:44:55.398105+0000","last_became_peered":"2024-03-24T00:44:55.398105+0000","last_unstale":"2024-03-24T00:44:57.777003+0000","last_undegraded":"2024-03-24T00:44:57.777003+0000","last_fullsized":"2024-03-24T00:44:57.777003+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:21:15.581002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.777817+0000","last_change":"2024-03-24T00:44:55.405559+0000","last_active":"2024-03-24T00:44:57.777817+0000","last_peered":"2024-03-24T00:44:57.777817+0000","last_clean":"2024-03-24T00:44:57.777817+0000","last_became_active":"2024-03-24T00:44:55.405275+0000","last_became_peered":"2024-03-24T00:44:55.405275+0000","last_unstale":"2024-03-24T00:44:57.777817+0000","last_undegraded":"2024-03-24T00:44:57.777817+0000","last_fullsized":"2024-03-24T00:44:57.777817+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:28:05.467698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.729245+0000","last_change":"2024-03-24T00:44:55.398815+0000","last_active":"2024-03-24T00:44:56.729245+0000","last_peered":"2024-03-24T00:44:56.729245+0000","last_clean":"2024-03-24T00:44:56.729245+0000","last_became_active":"2024-03-24T00:44:55.398185+0000","last_became_peered":"2024-03-24T00:44:55.398185+0000","last_unstale":"2024-03-24T00:44:56.729245+0000","last_undegraded":"2024-03-24T00:44:56.729245+0000","last_fullsized":"2024-03-24T00:44:56.729245+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:38:36.247055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.386231+0000","last_change":"2024-03-24T00:44:55.400229+0000","last_active":"2024-03-24T00:44:56.386231+0000","last_peered":"2024-03-24T00:44:56.386231+0000","last_clean":"2024-03-24T00:44:56.386231+0000","last_became_active":"2024-03-24T00:44:55.399941+0000","last_became_peered":"2024-03-24T00:44:55.399941+0000","last_unstale":"2024-03-24T00:44:56.386231+0000","last_undegraded":"2024-03-24T00:44:56.386231+0000","last_fullsized":"2024-03-24T00:44:56.386231+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:18:51.669689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.384981+0000","last_change":"2024-03-24T00:44:55.395189+0000","last_active":"2024-03-24T00:44:56.384981+0000","last_peered":"2024-03-24T00:44:56.384981+0000","last_clean":"2024-03-24T00:44:56.384981+0000","last_became_active":"2024-03-24T00:44:55.394811+0000","last_became_peered":"2024-03-24T00:44:55.394811+0000","last_unstale":"2024-03-24T00:44:56.384981+0000","last_undegraded":"2024-03-24T00:44:56.384981+0000","last_fullsized":"2024-03-24T00:44:56.384981+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:05:25.799252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.382432+0000","last_change":"2024-03-24T00:44:55.405118+0000","last_active":"2024-03-24T00:44:56.382432+0000","last_peered":"2024-03-24T00:44:56.382432+0000","last_clean":"2024-03-24T00:44:56.382432+0000","last_became_active":"2024-03-24T00:44:55.404477+0000","last_became_peered":"2024-03-24T00:44:55.404477+0000","last_unstale":"2024-03-24T00:44:56.382432+0000","last_undegraded":"2024-03-24T00:44:56.382432+0000","last_fullsized":"2024-03-24T00:44:56.382432+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:04:03.451551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.382699+0000","last_change":"2024-03-24T00:44:55.402620+0000","last_active":"2024-03-24T00:44:56.382699+0000","last_peered":"2024-03-24T00:44:56.382699+0000","last_clean":"2024-03-24T00:44:56.382699+0000","last_became_active":"2024-03-24T00:44:55.401195+0000","last_became_peered":"2024-03-24T00:44:55.401195+0000","last_unstale":"2024-03-24T00:44:56.382699+0000","last_undegraded":"2024-03-24T00:44:56.382699+0000","last_fullsized":"2024-03-24T00:44:56.382699+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:44:15.022767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"88'4","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.406465+0000","last_change":"2024-03-24T00:44:54.396737+0000","last_active":"2024-03-24T00:44:57.406465+0000","last_peered":"2024-03-24T00:44:57.406465+0000","last_clean":"2024-03-24T00:44:57.406465+0000","last_became_active":"2024-03-24T00:44:54.396431+0000","last_became_peered":"2024-03-24T00:44:54.396431+0000","last_unstale":"2024-03-24T00:44:57.406465+0000","last_undegraded":"2024-03-24T00:44:57.406465+0000","last_fullsized":"2024-03-24T00:44:57.406465+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:54:54.886053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.382799+0000","last_change":"2024-03-24T00:44:55.401707+0000","last_active":"2024-03-24T00:44:56.382799+0000","last_peered":"2024-03-24T00:44:56.382799+0000","last_clean":"2024-03-24T00:44:56.382799+0000","last_became_active":"2024-03-24T00:44:55.400657+0000","last_became_peered":"2024-03-24T00:44:55.400657+0000","last_unstale":"2024-03-24T00:44:56.382799+0000","last_undegraded":"2024-03-24T00:44:56.382799+0000","last_fullsized":"2024-03-24T00:44:56.382799+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:40:43.577674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"88'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.386644+0000","last_change":"2024-03-24T00:44:54.392014+0000","last_active":"2024-03-24T00:44:56.386644+0000","last_peered":"2024-03-24T00:44:56.386644+0000","last_clean":"2024-03-24T00:44:56.386644+0000","last_became_active":"2024-03-24T00:44:54.390205+0000","last_became_peered":"2024-03-24T00:44:54.390205+0000","last_unstale":"2024-03-24T00:44:56.386644+0000","last_undegraded":"2024-03-24T00:44:56.386644+0000","last_fullsized":"2024-03-24T00:44:56.386644+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:33:34.699913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.729554+0000","last_change":"2024-03-24T00:44:55.401412+0000","last_active":"2024-03-24T00:44:56.729554+0000","last_peered":"2024-03-24T00:44:56.729554+0000","last_clean":"2024-03-24T00:44:56.729554+0000","last_became_active":"2024-03-24T00:44:55.401105+0000","last_became_peered":"2024-03-24T00:44:55.401105+0000","last_unstale":"2024-03-24T00:44:56.729554+0000","last_undegraded":"2024-03-24T00:44:56.729554+0000","last_fullsized":"2024-03-24T00:44:56.729554+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:19:39.501718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"88'2","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.729205+0000","last_change":"2024-03-24T00:44:54.390117+0000","last_active":"2024-03-24T00:44:56.729205+0000","last_peered":"2024-03-24T00:44:56.729205+0000","last_clean":"2024-03-24T00:44:56.729205+0000","last_became_active":"2024-03-24T00:44:54.389849+0000","last_became_peered":"2024-03-24T00:44:54.389849+0000","last_unstale":"2024-03-24T00:44:56.729205+0000","last_undegraded":"2024-03-24T00:44:56.729205+0000","last_fullsized":"2024-03-24T00:44:56.729205+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:13:30.737465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.383050+0000","last_change":"2024-03-24T00:44:55.398111+0000","last_active":"2024-03-24T00:44:56.383050+0000","last_peered":"2024-03-24T00:44:56.383050+0000","last_clean":"2024-03-24T00:44:56.383050+0000","last_became_active":"2024-03-24T00:44:55.397334+0000","last_became_peered":"2024-03-24T00:44:55.397334+0000","last_unstale":"2024-03-24T00:44:56.383050+0000","last_undegraded":"2024-03-24T00:44:56.383050+0000","last_fullsized":"2024-03-24T00:44:56.383050+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:33:11.855966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"88'2","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.406406+0000","last_change":"2024-03-24T00:44:54.393213+0000","last_active":"2024-03-24T00:44:57.406406+0000","last_peered":"2024-03-24T00:44:57.406406+0000","last_clean":"2024-03-24T00:44:57.406406+0000","last_became_active":"2024-03-24T00:44:54.392741+0000","last_became_peered":"2024-03-24T00:44:54.392741+0000","last_unstale":"2024-03-24T00:44:57.406406+0000","last_undegraded":"2024-03-24T00:44:57.406406+0000","last_fullsized":"2024-03-24T00:44:57.406406+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:23:31.116823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.777295+0000","last_change":"2024-03-24T00:44:55.398025+0000","last_active":"2024-03-24T00:44:57.777295+0000","last_peered":"2024-03-24T00:44:57.777295+0000","last_clean":"2024-03-24T00:44:57.777295+0000","last_became_active":"2024-03-24T00:44:55.397785+0000","last_became_peered":"2024-03-24T00:44:55.397785+0000","last_unstale":"2024-03-24T00:44:57.777295+0000","last_undegraded":"2024-03-24T00:44:57.777295+0000","last_fullsized":"2024-03-24T00:44:57.777295+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:00:18.603927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"88'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.386228+0000","last_change":"2024-03-24T00:44:54.391305+0000","last_active":"2024-03-24T00:44:56.386228+0000","last_peered":"2024-03-24T00:44:56.386228+0000","last_clean":"2024-03-24T00:44:56.386228+0000","last_became_active":"2024-03-24T00:44:54.390877+0000","last_became_peered":"2024-03-24T00:44:54.390877+0000","last_unstale":"2024-03-24T00:44:56.386228+0000","last_undegraded":"2024-03-24T00:44:56.386228+0000","last_fullsized":"2024-03-24T00:44:56.386228+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:35:15.833860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.406292+0000","last_change":"2024-03-24T00:44:55.399402+0000","last_active":"2024-03-24T00:44:57.406292+0000","last_peered":"2024-03-24T00:44:57.406292+0000","last_clean":"2024-03-24T00:44:57.406292+0000","last_became_active":"2024-03-24T00:44:55.398894+0000","last_became_peered":"2024-03-24T00:44:55.398894+0000","last_unstale":"2024-03-24T00:44:57.406292+0000","last_undegraded":"2024-03-24T00:44:57.406292+0000","last_fullsized":"2024-03-24T00:44:57.406292+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:16:07.455446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"88'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.777682+0000","last_change":"2024-03-24T00:44:54.393934+0000","last_active":"2024-03-24T00:44:57.777682+0000","last_peered":"2024-03-24T00:44:57.777682+0000","last_clean":"2024-03-24T00:44:57.777682+0000","last_became_active":"2024-03-24T00:44:54.393459+0000","last_became_peered":"2024-03-24T00:44:54.393459+0000","last_unstale":"2024-03-24T00:44:57.777682+0000","last_undegraded":"2024-03-24T00:44:57.777682+0000","last_fullsized":"2024-03-24T00:44:57.777682+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:56:21.899623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.382606+0000","last_change":"2024-03-24T00:44:55.410380+0000","last_active":"2024-03-24T00:44:56.382606+0000","last_peered":"2024-03-24T00:44:56.382606+0000","last_clean":"2024-03-24T00:44:56.382606+0000","last_became_active":"2024-03-24T00:44:55.410099+0000","last_became_peered":"2024-03-24T00:44:55.410099+0000","last_unstale":"2024-03-24T00:44:56.382606+0000","last_undegraded":"2024-03-24T00:44:56.382606+0000","last_fullsized":"2024-03-24T00:44:56.382606+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:33:49.693865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.514115+0000","last_change":"2024-03-24T00:44:54.383196+0000","last_active":"2024-03-24T00:44:57.514115+0000","last_peered":"2024-03-24T00:44:57.514115+0000","last_clean":"2024-03-24T00:44:57.514115+0000","last_became_active":"2024-03-24T00:44:54.382875+0000","last_became_peered":"2024-03-24T00:44:54.382875+0000","last_unstale":"2024-03-24T00:44:57.514115+0000","last_undegraded":"2024-03-24T00:44:57.514115+0000","last_fullsized":"2024-03-24T00:44:57.514115+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:32:13.875499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.777537+0000","last_change":"2024-03-24T00:44:55.393864+0000","last_active":"2024-03-24T00:44:57.777537+0000","last_peered":"2024-03-24T00:44:57.777537+0000","last_clean":"2024-03-24T00:44:57.777537+0000","last_became_active":"2024-03-24T00:44:55.393588+0000","last_became_peered":"2024-03-24T00:44:55.393588+0000","last_unstale":"2024-03-24T00:44:57.777537+0000","last_undegraded":"2024-03-24T00:44:57.777537+0000","last_fullsized":"2024-03-24T00:44:57.777537+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:21:21.650742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.729046+0000","last_change":"2024-03-24T00:44:54.385598+0000","last_active":"2024-03-24T00:44:56.729046+0000","last_peered":"2024-03-24T00:44:56.729046+0000","last_clean":"2024-03-24T00:44:56.729046+0000","last_became_active":"2024-03-24T00:44:54.385270+0000","last_became_peered":"2024-03-24T00:44:54.385270+0000","last_unstale":"2024-03-24T00:44:56.729046+0000","last_undegraded":"2024-03-24T00:44:56.729046+0000","last_fullsized":"2024-03-24T00:44:56.729046+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:45:33.049844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.386469+0000","last_change":"2024-03-24T00:44:55.391244+0000","last_active":"2024-03-24T00:44:56.386469+0000","last_peered":"2024-03-24T00:44:56.386469+0000","last_clean":"2024-03-24T00:44:56.386469+0000","last_became_active":"2024-03-24T00:44:55.390752+0000","last_became_peered":"2024-03-24T00:44:55.390752+0000","last_unstale":"2024-03-24T00:44:56.386469+0000","last_undegraded":"2024-03-24T00:44:56.386469+0000","last_fullsized":"2024-03-24T00:44:56.386469+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:49:36.416272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.385545+0000","last_change":"2024-03-24T00:44:54.393609+0000","last_active":"2024-03-24T00:44:56.385545+0000","last_peered":"2024-03-24T00:44:56.385545+0000","last_clean":"2024-03-24T00:44:56.385545+0000","last_became_active":"2024-03-24T00:44:54.393163+0000","last_became_peered":"2024-03-24T00:44:54.393163+0000","last_unstale":"2024-03-24T00:44:56.385545+0000","last_undegraded":"2024-03-24T00:44:56.385545+0000","last_fullsized":"2024-03-24T00:44:56.385545+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:09:59.913191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.729266+0000","last_change":"2024-03-24T00:44:55.400989+0000","last_active":"2024-03-24T00:44:56.729266+0000","last_peered":"2024-03-24T00:44:56.729266+0000","last_clean":"2024-03-24T00:44:56.729266+0000","last_became_active":"2024-03-24T00:44:55.400717+0000","last_became_peered":"2024-03-24T00:44:55.400717+0000","last_unstale":"2024-03-24T00:44:56.729266+0000","last_undegraded":"2024-03-24T00:44:56.729266+0000","last_fullsized":"2024-03-24T00:44:56.729266+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:45:52.622468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.514408+0000","last_change":"2024-03-24T00:44:54.391371+0000","last_active":"2024-03-24T00:44:57.514408+0000","last_peered":"2024-03-24T00:44:57.514408+0000","last_clean":"2024-03-24T00:44:57.514408+0000","last_became_active":"2024-03-24T00:44:54.391075+0000","last_became_peered":"2024-03-24T00:44:54.391075+0000","last_unstale":"2024-03-24T00:44:57.514408+0000","last_undegraded":"2024-03-24T00:44:57.514408+0000","last_fullsized":"2024-03-24T00:44:57.514408+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:13:30.737465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.514478+0000","last_change":"2024-03-24T00:44:55.397952+0000","last_active":"2024-03-24T00:44:57.514478+0000","last_peered":"2024-03-24T00:44:57.514478+0000","last_clean":"2024-03-24T00:44:57.514478+0000","last_became_active":"2024-03-24T00:44:55.397681+0000","last_became_peered":"2024-03-24T00:44:55.397681+0000","last_unstale":"2024-03-24T00:44:57.514478+0000","last_undegraded":"2024-03-24T00:44:57.514478+0000","last_fullsized":"2024-03-24T00:44:57.514478+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:54:55.383079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"88'3","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.386817+0000","last_change":"2024-03-24T00:44:54.395602+0000","last_active":"2024-03-24T00:44:56.386817+0000","last_peered":"2024-03-24T00:44:56.386817+0000","last_clean":"2024-03-24T00:44:56.386817+0000","last_became_active":"2024-03-24T00:44:54.395326+0000","last_became_peered":"2024-03-24T00:44:54.395326+0000","last_unstale":"2024-03-24T00:44:56.386817+0000","last_undegraded":"2024-03-24T00:44:56.386817+0000","last_fullsized":"2024-03-24T00:44:56.386817+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:40:16.672606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.777920+0000","last_change":"2024-03-24T00:44:55.405635+0000","last_active":"2024-03-24T00:44:57.777920+0000","last_peered":"2024-03-24T00:44:57.777920+0000","last_clean":"2024-03-24T00:44:57.777920+0000","last_became_active":"2024-03-24T00:44:55.405341+0000","last_became_peered":"2024-03-24T00:44:55.405341+0000","last_unstale":"2024-03-24T00:44:57.777920+0000","last_undegraded":"2024-03-24T00:44:57.777920+0000","last_fullsized":"2024-03-24T00:44:57.777920+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:45:34.052028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"88'6","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.382606+0000","last_change":"2024-03-24T00:44:54.389907+0000","last_active":"2024-03-24T00:44:56.382606+0000","last_peered":"2024-03-24T00:44:56.382606+0000","last_clean":"2024-03-24T00:44:56.382606+0000","last_became_active":"2024-03-24T00:44:54.389643+0000","last_became_peered":"2024-03-24T00:44:54.389643+0000","last_unstale":"2024-03-24T00:44:56.382606+0000","last_undegraded":"2024-03-24T00:44:56.382606+0000","last_fullsized":"2024-03-24T00:44:56.382606+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:14:44.327426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.777563+0000","last_change":"2024-03-24T00:44:55.406640+0000","last_active":"2024-03-24T00:44:57.777563+0000","last_peered":"2024-03-24T00:44:57.777563+0000","last_clean":"2024-03-24T00:44:57.777563+0000","last_became_active":"2024-03-24T00:44:55.406390+0000","last_became_peered":"2024-03-24T00:44:55.406390+0000","last_unstale":"2024-03-24T00:44:57.777563+0000","last_undegraded":"2024-03-24T00:44:57.777563+0000","last_fullsized":"2024-03-24T00:44:57.777563+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:29:00.989588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"88'2","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.729141+0000","last_change":"2024-03-24T00:44:54.384267+0000","last_active":"2024-03-24T00:44:56.729141+0000","last_peered":"2024-03-24T00:44:56.729141+0000","last_clean":"2024-03-24T00:44:56.729141+0000","last_became_active":"2024-03-24T00:44:54.383981+0000","last_became_peered":"2024-03-24T00:44:54.383981+0000","last_unstale":"2024-03-24T00:44:56.729141+0000","last_undegraded":"2024-03-24T00:44:56.729141+0000","last_fullsized":"2024-03-24T00:44:56.729141+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:32:13.875499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.386003+0000","last_change":"2024-03-24T00:44:55.393817+0000","last_active":"2024-03-24T00:44:56.386003+0000","last_peered":"2024-03-24T00:44:56.386003+0000","last_clean":"2024-03-24T00:44:56.386003+0000","last_became_active":"2024-03-24T00:44:55.393489+0000","last_became_peered":"2024-03-24T00:44:55.393489+0000","last_unstale":"2024-03-24T00:44:56.386003+0000","last_undegraded":"2024-03-24T00:44:56.386003+0000","last_fullsized":"2024-03-24T00:44:56.386003+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:26:43.751379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.382568+0000","last_change":"2024-03-24T00:44:54.396876+0000","last_active":"2024-03-24T00:44:56.382568+0000","last_peered":"2024-03-24T00:44:56.382568+0000","last_clean":"2024-03-24T00:44:56.382568+0000","last_became_active":"2024-03-24T00:44:54.396539+0000","last_became_peered":"2024-03-24T00:44:54.396539+0000","last_unstale":"2024-03-24T00:44:56.382568+0000","last_undegraded":"2024-03-24T00:44:56.382568+0000","last_fullsized":"2024-03-24T00:44:56.382568+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:05:24.797069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.776806+0000","last_change":"2024-03-24T00:44:55.408116+0000","last_active":"2024-03-24T00:44:57.776806+0000","last_peered":"2024-03-24T00:44:57.776806+0000","last_clean":"2024-03-24T00:44:57.776806+0000","last_became_active":"2024-03-24T00:44:55.407841+0000","last_became_peered":"2024-03-24T00:44:55.407841+0000","last_unstale":"2024-03-24T00:44:57.776806+0000","last_undegraded":"2024-03-24T00:44:57.776806+0000","last_fullsized":"2024-03-24T00:44:57.776806+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:13:35.495890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.777134+0000","last_change":"2024-03-24T00:44:54.393291+0000","last_active":"2024-03-24T00:44:57.777134+0000","last_peered":"2024-03-24T00:44:57.777134+0000","last_clean":"2024-03-24T00:44:57.777134+0000","last_became_active":"2024-03-24T00:44:54.392987+0000","last_became_peered":"2024-03-24T00:44:54.392987+0000","last_unstale":"2024-03-24T00:44:57.777134+0000","last_undegraded":"2024-03-24T00:44:57.777134+0000","last_fullsized":"2024-03-24T00:44:57.777134+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:13:31.133146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.777087+0000","last_change":"2024-03-24T00:44:55.402701+0000","last_active":"2024-03-24T00:44:57.777087+0000","last_peered":"2024-03-24T00:44:57.777087+0000","last_clean":"2024-03-24T00:44:57.777087+0000","last_became_active":"2024-03-24T00:44:55.402373+0000","last_became_peered":"2024-03-24T00:44:55.402373+0000","last_unstale":"2024-03-24T00:44:57.777087+0000","last_undegraded":"2024-03-24T00:44:57.777087+0000","last_fullsized":"2024-03-24T00:44:57.777087+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:16:19.143414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"88'8","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.777426+0000","last_change":"2024-03-24T00:44:54.394615+0000","last_active":"2024-03-24T00:44:57.777426+0000","last_peered":"2024-03-24T00:44:57.777426+0000","last_clean":"2024-03-24T00:44:57.777426+0000","last_became_active":"2024-03-24T00:44:54.393386+0000","last_became_peered":"2024-03-24T00:44:54.393386+0000","last_unstale":"2024-03-24T00:44:57.777426+0000","last_undegraded":"2024-03-24T00:44:57.777426+0000","last_fullsized":"2024-03-24T00:44:57.777426+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:18:30.093254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.382554+0000","last_change":"2024-03-24T00:44:55.400804+0000","last_active":"2024-03-24T00:44:56.382554+0000","last_peered":"2024-03-24T00:44:56.382554+0000","last_clean":"2024-03-24T00:44:56.382554+0000","last_became_active":"2024-03-24T00:44:55.400543+0000","last_became_peered":"2024-03-24T00:44:55.400543+0000","last_unstale":"2024-03-24T00:44:56.382554+0000","last_undegraded":"2024-03-24T00:44:56.382554+0000","last_fullsized":"2024-03-24T00:44:56.382554+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:11:26.528589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"88'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.406518+0000","last_change":"2024-03-24T00:44:54.394526+0000","last_active":"2024-03-24T00:44:57.406518+0000","last_peered":"2024-03-24T00:44:57.406518+0000","last_clean":"2024-03-24T00:44:57.406518+0000","last_became_active":"2024-03-24T00:44:54.393823+0000","last_became_peered":"2024-03-24T00:44:54.393823+0000","last_unstale":"2024-03-24T00:44:57.406518+0000","last_undegraded":"2024-03-24T00:44:57.406518+0000","last_fullsized":"2024-03-24T00:44:57.406518+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:49:02.779117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.406398+0000","last_change":"2024-03-24T00:44:55.402478+0000","last_active":"2024-03-24T00:44:57.406398+0000","last_peered":"2024-03-24T00:44:57.406398+0000","last_clean":"2024-03-24T00:44:57.406398+0000","last_became_active":"2024-03-24T00:44:55.402160+0000","last_became_peered":"2024-03-24T00:44:55.402160+0000","last_unstale":"2024-03-24T00:44:57.406398+0000","last_undegraded":"2024-03-24T00:44:57.406398+0000","last_fullsized":"2024-03-24T00:44:57.406398+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:18:51.669689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"88'2","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.729268+0000","last_change":"2024-03-24T00:44:54.389428+0000","last_active":"2024-03-24T00:44:56.729268+0000","last_peered":"2024-03-24T00:44:56.729268+0000","last_clean":"2024-03-24T00:44:56.729268+0000","last_became_active":"2024-03-24T00:44:54.389123+0000","last_became_peered":"2024-03-24T00:44:54.389123+0000","last_unstale":"2024-03-24T00:44:56.729268+0000","last_undegraded":"2024-03-24T00:44:56.729268+0000","last_fullsized":"2024-03-24T00:44:56.729268+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:18:27.095669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.386770+0000","last_change":"2024-03-24T00:44:55.396270+0000","last_active":"2024-03-24T00:44:56.386770+0000","last_peered":"2024-03-24T00:44:56.386770+0000","last_clean":"2024-03-24T00:44:56.386770+0000","last_became_active":"2024-03-24T00:44:55.395927+0000","last_became_peered":"2024-03-24T00:44:55.395927+0000","last_unstale":"2024-03-24T00:44:56.386770+0000","last_undegraded":"2024-03-24T00:44:56.386770+0000","last_fullsized":"2024-03-24T00:44:56.386770+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:25:57.625958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"88'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.386861+0000","last_change":"2024-03-24T00:44:54.391906+0000","last_active":"2024-03-24T00:44:56.386861+0000","last_peered":"2024-03-24T00:44:56.386861+0000","last_clean":"2024-03-24T00:44:56.386861+0000","last_became_active":"2024-03-24T00:44:54.391580+0000","last_became_peered":"2024-03-24T00:44:54.391580+0000","last_unstale":"2024-03-24T00:44:56.386861+0000","last_undegraded":"2024-03-24T00:44:56.386861+0000","last_fullsized":"2024-03-24T00:44:56.386861+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:11:55.046776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.382599+0000","last_change":"2024-03-24T00:44:55.403460+0000","last_active":"2024-03-24T00:44:56.382599+0000","last_peered":"2024-03-24T00:44:56.382599+0000","last_clean":"2024-03-24T00:44:56.382599+0000","last_became_active":"2024-03-24T00:44:55.403202+0000","last_became_peered":"2024-03-24T00:44:55.403202+0000","last_unstale":"2024-03-24T00:44:56.382599+0000","last_undegraded":"2024-03-24T00:44:56.382599+0000","last_fullsized":"2024-03-24T00:44:56.382599+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:19:08.478709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"88'2","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.385103+0000","last_change":"2024-03-24T00:44:54.393313+0000","last_active":"2024-03-24T00:44:56.385103+0000","last_peered":"2024-03-24T00:44:56.385103+0000","last_clean":"2024-03-24T00:44:56.385103+0000","last_became_active":"2024-03-24T00:44:54.393033+0000","last_became_peered":"2024-03-24T00:44:54.393033+0000","last_unstale":"2024-03-24T00:44:56.385103+0000","last_undegraded":"2024-03-24T00:44:56.385103+0000","last_fullsized":"2024-03-24T00:44:56.385103+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:45:33.049844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.384983+0000","last_change":"2024-03-24T00:44:55.393095+0000","last_active":"2024-03-24T00:44:56.384983+0000","last_peered":"2024-03-24T00:44:56.384983+0000","last_clean":"2024-03-24T00:44:56.384983+0000","last_became_active":"2024-03-24T00:44:55.392797+0000","last_became_peered":"2024-03-24T00:44:55.392797+0000","last_unstale":"2024-03-24T00:44:56.384983+0000","last_undegraded":"2024-03-24T00:44:56.384983+0000","last_fullsized":"2024-03-24T00:44:56.384983+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:14:45.329609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.776751+0000","last_change":"2024-03-24T00:44:54.391499+0000","last_active":"2024-03-24T00:44:57.776751+0000","last_peered":"2024-03-24T00:44:57.776751+0000","last_clean":"2024-03-24T00:44:57.776751+0000","last_became_active":"2024-03-24T00:44:54.391170+0000","last_became_peered":"2024-03-24T00:44:54.391170+0000","last_unstale":"2024-03-24T00:44:57.776751+0000","last_undegraded":"2024-03-24T00:44:57.776751+0000","last_fullsized":"2024-03-24T00:44:57.776751+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:40:11.977062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.385233+0000","last_change":"2024-03-24T00:44:55.394421+0000","last_active":"2024-03-24T00:44:56.385233+0000","last_peered":"2024-03-24T00:44:56.385233+0000","last_clean":"2024-03-24T00:44:56.385233+0000","last_became_active":"2024-03-24T00:44:55.394112+0000","last_became_peered":"2024-03-24T00:44:55.394112+0000","last_unstale":"2024-03-24T00:44:56.385233+0000","last_undegraded":"2024-03-24T00:44:56.385233+0000","last_fullsized":"2024-03-24T00:44:56.385233+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:54.202078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.385047+0000","last_change":"2024-03-24T00:44:54.399808+0000","last_active":"2024-03-24T00:44:56.385047+0000","last_peered":"2024-03-24T00:44:56.385047+0000","last_clean":"2024-03-24T00:44:56.385047+0000","last_became_active":"2024-03-24T00:44:54.399554+0000","last_became_peered":"2024-03-24T00:44:54.399554+0000","last_unstale":"2024-03-24T00:44:56.385047+0000","last_undegraded":"2024-03-24T00:44:56.385047+0000","last_fullsized":"2024-03-24T00:44:56.385047+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:59:58.396249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.382650+0000","last_change":"2024-03-24T00:44:55.411074+0000","last_active":"2024-03-24T00:44:56.382650+0000","last_peered":"2024-03-24T00:44:56.382650+0000","last_clean":"2024-03-24T00:44:56.382650+0000","last_became_active":"2024-03-24T00:44:55.410826+0000","last_became_peered":"2024-03-24T00:44:55.410826+0000","last_unstale":"2024-03-24T00:44:56.382650+0000","last_undegraded":"2024-03-24T00:44:56.382650+0000","last_fullsized":"2024-03-24T00:44:56.382650+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:12:06.209321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.776608+0000","last_change":"2024-03-24T00:44:54.389975+0000","last_active":"2024-03-24T00:44:57.776608+0000","last_peered":"2024-03-24T00:44:57.776608+0000","last_clean":"2024-03-24T00:44:57.776608+0000","last_became_active":"2024-03-24T00:44:54.389711+0000","last_became_peered":"2024-03-24T00:44:54.389711+0000","last_unstale":"2024-03-24T00:44:57.776608+0000","last_undegraded":"2024-03-24T00:44:57.776608+0000","last_fullsized":"2024-03-24T00:44:57.776608+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:11:45.880543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.406483+0000","last_change":"2024-03-24T00:44:55.396584+0000","last_active":"2024-03-24T00:44:57.406483+0000","last_peered":"2024-03-24T00:44:57.406483+0000","last_clean":"2024-03-24T00:44:57.406483+0000","last_became_active":"2024-03-24T00:44:55.396277+0000","last_became_peered":"2024-03-24T00:44:55.396277+0000","last_unstale":"2024-03-24T00:44:57.406483+0000","last_undegraded":"2024-03-24T00:44:57.406483+0000","last_fullsized":"2024-03-24T00:44:57.406483+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:49:36.416272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"88'2","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.729331+0000","last_change":"2024-03-24T00:44:54.387054+0000","last_active":"2024-03-24T00:44:56.729331+0000","last_peered":"2024-03-24T00:44:56.729331+0000","last_clean":"2024-03-24T00:44:56.729331+0000","last_became_active":"2024-03-24T00:44:54.386785+0000","last_became_peered":"2024-03-24T00:44:54.386785+0000","last_unstale":"2024-03-24T00:44:56.729331+0000","last_undegraded":"2024-03-24T00:44:56.729331+0000","last_fullsized":"2024-03-24T00:44:56.729331+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:03:15.796915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.382948+0000","last_change":"2024-03-24T00:44:55.403999+0000","last_active":"2024-03-24T00:44:56.382948+0000","last_peered":"2024-03-24T00:44:56.382948+0000","last_clean":"2024-03-24T00:44:56.382948+0000","last_became_active":"2024-03-24T00:44:55.401878+0000","last_became_peered":"2024-03-24T00:44:55.401878+0000","last_unstale":"2024-03-24T00:44:56.382948+0000","last_undegraded":"2024-03-24T00:44:56.382948+0000","last_fullsized":"2024-03-24T00:44:56.382948+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:32:37.241577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"89'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.386780+0000","last_change":"2024-03-24T00:44:54.391792+0000","last_active":"2024-03-24T00:44:56.386780+0000","last_peered":"2024-03-24T00:44:56.386780+0000","last_clean":"2024-03-24T00:44:56.386780+0000","last_became_active":"2024-03-24T00:44:54.390529+0000","last_became_peered":"2024-03-24T00:44:54.390529+0000","last_unstale":"2024-03-24T00:44:56.386780+0000","last_undegraded":"2024-03-24T00:44:56.386780+0000","last_fullsized":"2024-03-24T00:44:56.386780+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:37:50.754605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.382941+0000","last_change":"2024-03-24T00:44:55.401482+0000","last_active":"2024-03-24T00:44:56.382941+0000","last_peered":"2024-03-24T00:44:56.382941+0000","last_clean":"2024-03-24T00:44:56.382941+0000","last_became_active":"2024-03-24T00:44:55.401223+0000","last_became_peered":"2024-03-24T00:44:55.401223+0000","last_unstale":"2024-03-24T00:44:56.382941+0000","last_undegraded":"2024-03-24T00:44:56.382941+0000","last_fullsized":"2024-03-24T00:44:56.382941+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:39:32.628036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.386915+0000","last_change":"2024-03-24T00:44:54.388742+0000","last_active":"2024-03-24T00:44:56.386915+0000","last_peered":"2024-03-24T00:44:56.386915+0000","last_clean":"2024-03-24T00:44:56.386915+0000","last_became_active":"2024-03-24T00:44:54.388466+0000","last_became_peered":"2024-03-24T00:44:54.388466+0000","last_unstale":"2024-03-24T00:44:56.386915+0000","last_undegraded":"2024-03-24T00:44:56.386915+0000","last_fullsized":"2024-03-24T00:44:56.386915+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:41:25.500084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.385509+0000","last_change":"2024-03-24T00:44:55.388203+0000","last_active":"2024-03-24T00:44:56.385509+0000","last_peered":"2024-03-24T00:44:56.385509+0000","last_clean":"2024-03-24T00:44:56.385509+0000","last_became_active":"2024-03-24T00:44:55.387932+0000","last_became_peered":"2024-03-24T00:44:55.387932+0000","last_unstale":"2024-03-24T00:44:56.385509+0000","last_undegraded":"2024-03-24T00:44:56.385509+0000","last_fullsized":"2024-03-24T00:44:56.385509+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:39:23.097646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.406512+0000","last_change":"2024-03-24T00:44:54.391056+0000","last_active":"2024-03-24T00:44:57.406512+0000","last_peered":"2024-03-24T00:44:57.406512+0000","last_clean":"2024-03-24T00:44:57.406512+0000","last_became_active":"2024-03-24T00:44:54.390695+0000","last_became_peered":"2024-03-24T00:44:54.390695+0000","last_unstale":"2024-03-24T00:44:57.406512+0000","last_undegraded":"2024-03-24T00:44:57.406512+0000","last_fullsized":"2024-03-24T00:44:57.406512+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:57.847221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.776673+0000","last_change":"2024-03-24T00:44:55.407491+0000","last_active":"2024-03-24T00:44:57.776673+0000","last_peered":"2024-03-24T00:44:57.776673+0000","last_clean":"2024-03-24T00:44:57.776673+0000","last_became_active":"2024-03-24T00:44:55.407221+0000","last_became_peered":"2024-03-24T00:44:55.407221+0000","last_unstale":"2024-03-24T00:44:57.776673+0000","last_undegraded":"2024-03-24T00:44:57.776673+0000","last_fullsized":"2024-03-24T00:44:57.776673+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:15:59.692129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.382768+0000","last_change":"2024-03-24T00:44:54.397277+0000","last_active":"2024-03-24T00:44:56.382768+0000","last_peered":"2024-03-24T00:44:56.382768+0000","last_clean":"2024-03-24T00:44:56.382768+0000","last_became_active":"2024-03-24T00:44:54.397008+0000","last_became_peered":"2024-03-24T00:44:54.397008+0000","last_unstale":"2024-03-24T00:44:56.382768+0000","last_undegraded":"2024-03-24T00:44:56.382768+0000","last_fullsized":"2024-03-24T00:44:56.382768+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:53.199895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.514142+0000","last_change":"2024-03-24T00:44:55.389272+0000","last_active":"2024-03-24T00:44:57.514142+0000","last_peered":"2024-03-24T00:44:57.514142+0000","last_clean":"2024-03-24T00:44:57.514142+0000","last_became_active":"2024-03-24T00:44:55.389005+0000","last_became_peered":"2024-03-24T00:44:55.389005+0000","last_unstale":"2024-03-24T00:44:57.514142+0000","last_undegraded":"2024-03-24T00:44:57.514142+0000","last_fullsized":"2024-03-24T00:44:57.514142+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:38:36.247055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.385038+0000","last_change":"2024-03-24T00:44:54.392645+0000","last_active":"2024-03-24T00:44:56.385038+0000","last_peered":"2024-03-24T00:44:56.385038+0000","last_clean":"2024-03-24T00:44:56.385038+0000","last_became_active":"2024-03-24T00:44:54.392370+0000","last_became_peered":"2024-03-24T00:44:54.392370+0000","last_unstale":"2024-03-24T00:44:56.385038+0000","last_undegraded":"2024-03-24T00:44:56.385038+0000","last_fullsized":"2024-03-24T00:44:56.385038+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:28:04.465514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.406218+0000","last_change":"2024-03-24T00:44:55.399019+0000","last_active":"2024-03-24T00:44:57.406218+0000","last_peered":"2024-03-24T00:44:57.406218+0000","last_clean":"2024-03-24T00:44:57.406218+0000","last_became_active":"2024-03-24T00:44:55.398194+0000","last_became_peered":"2024-03-24T00:44:55.398194+0000","last_unstale":"2024-03-24T00:44:57.406218+0000","last_undegraded":"2024-03-24T00:44:57.406218+0000","last_fullsized":"2024-03-24T00:44:57.406218+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:13:32.135329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.406348+0000","last_change":"2024-03-24T00:44:54.391852+0000","last_active":"2024-03-24T00:44:57.406348+0000","last_peered":"2024-03-24T00:44:57.406348+0000","last_clean":"2024-03-24T00:44:57.406348+0000","last_became_active":"2024-03-24T00:44:54.390810+0000","last_became_peered":"2024-03-24T00:44:54.390810+0000","last_unstale":"2024-03-24T00:44:57.406348+0000","last_undegraded":"2024-03-24T00:44:57.406348+0000","last_fullsized":"2024-03-24T00:44:57.406348+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:11:06.419292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.382779+0000","last_change":"2024-03-24T00:44:55.405256+0000","last_active":"2024-03-24T00:44:56.382779+0000","last_peered":"2024-03-24T00:44:56.382779+0000","last_clean":"2024-03-24T00:44:56.382779+0000","last_became_active":"2024-03-24T00:44:55.403655+0000","last_became_peered":"2024-03-24T00:44:55.403655+0000","last_unstale":"2024-03-24T00:44:56.382779+0000","last_undegraded":"2024-03-24T00:44:56.382779+0000","last_fullsized":"2024-03-24T00:44:56.382779+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:21:10.200232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.729154+0000","last_change":"2024-03-24T00:44:54.390612+0000","last_active":"2024-03-24T00:44:56.729154+0000","last_peered":"2024-03-24T00:44:56.729154+0000","last_clean":"2024-03-24T00:44:56.729154+0000","last_became_active":"2024-03-24T00:44:54.390348+0000","last_became_peered":"2024-03-24T00:44:54.390348+0000","last_unstale":"2024-03-24T00:44:56.729154+0000","last_undegraded":"2024-03-24T00:44:56.729154+0000","last_fullsized":"2024-03-24T00:44:56.729154+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:09:59.913191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.729417+0000","last_change":"2024-03-24T00:44:55.398320+0000","last_active":"2024-03-24T00:44:56.729417+0000","last_peered":"2024-03-24T00:44:56.729417+0000","last_clean":"2024-03-24T00:44:56.729417+0000","last_became_active":"2024-03-24T00:44:55.398040+0000","last_became_peered":"2024-03-24T00:44:55.398040+0000","last_unstale":"2024-03-24T00:44:56.729417+0000","last_undegraded":"2024-03-24T00:44:56.729417+0000","last_fullsized":"2024-03-24T00:44:56.729417+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:54:55.383079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.777788+0000","last_change":"2024-03-24T00:44:54.395548+0000","last_active":"2024-03-24T00:44:57.777788+0000","last_peered":"2024-03-24T00:44:57.777788+0000","last_clean":"2024-03-24T00:44:57.777788+0000","last_became_active":"2024-03-24T00:44:54.395266+0000","last_became_peered":"2024-03-24T00:44:54.395266+0000","last_unstale":"2024-03-24T00:44:57.777788+0000","last_undegraded":"2024-03-24T00:44:57.777788+0000","last_fullsized":"2024-03-24T00:44:57.777788+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:16:06.453263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":299,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.729348+0000","last_change":"2024-03-24T00:38:40.953871+0000","last_active":"2024-03-24T00:44:56.729348+0000","last_peered":"2024-03-24T00:44:56.729348+0000","last_clean":"2024-03-24T00:44:56.729348+0000","last_became_active":"2024-03-24T00:38:40.952926+0000","last_became_peered":"2024-03-24T00:38:40.952926+0000","last_unstale":"2024-03-24T00:44:56.729348+0000","last_undegraded":"2024-03-24T00:44:56.729348+0000","last_fullsized":"2024-03-24T00:44:56.729348+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_clean_scrub_stamp":"2024-03-24T00:38:39.722194+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:05.397257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574929,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32844,"kb_used_data":1660,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338996,"statfs":{"total":96636764160,"available":96603131904,"internally_reserved":0,"allocated":1699840,"data_stored":1095629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574928,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32276,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339564,"statfs":{"total":96636764160,"available":96603713536,"internally_reserved":0,"allocated":1118208,"data_stored":507421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574928,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32256,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339584,"statfs":{"total":96636764160,"available":96603734016,"internally_reserved":0,"allocated":1097728,"data_stored":505205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574927,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32264,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339576,"statfs":{"total":96636764160,"available":96603725824,"internally_reserved":0,"allocated":1105920,"data_stored":505365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574927,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32256,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339584,"statfs":{"total":96636764160,"available":96603734016,"internally_reserved":0,"allocated":1097728,"data_stored":505205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574928,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32256,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339584,"statfs":{"total":96636764160,"available":96603734016,"internally_reserved":0,"allocated":1097728,"data_stored":505205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574928,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32260,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339580,"statfs":{"total":96636764160,"available":96603729920,"internally_reserved":0,"allocated":1101824,"data_stored":505275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574927,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32856,"kb_used_data":1676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338984,"statfs":{"total":96636764160,"available":96603119616,"internally_reserved":0,"allocated":1716224,"data_stored":1097891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:45:01.033 INFO:tasks.ceph.mds.c.smithi067.stderr:2024-03-24T00:45:01.034+0000 7fcb01199640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T00:45:03.082 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:45:03.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.218+0000 7f70e7c85640 1 Processor -- start 2024-03-24T00:45:03.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.218+0000 7f70e7c85640 1 -- start start 2024-03-24T00:45:03.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.219+0000 7f70e7c85640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f70e015dec0 0x7f70e00ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:03.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.219+0000 7f70e7c85640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f70e015e2a0 0x7f70e0162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:03.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.219+0000 7f70e7c85640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70e00cf640 con 0x7f70e00d07e0 2024-03-24T00:45:03.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.219+0000 7f70e7c85640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70e00cee30 con 0x7f70e015dec0 2024-03-24T00:45:03.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.219+0000 7f70e7c85640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70e00d2120 con 0x7f70e015e2a0 2024-03-24T00:45:03.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.219+0000 7f70e5a21640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f70e015dec0 0x7f70e00ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:03.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.219+0000 7f70e5a21640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f70e015dec0 0x7f70e00ce630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45212/0 (socket says 172.21.15.67:45212) 2024-03-24T00:45:03.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.219+0000 7f70e5a21640 1 -- 172.21.15.67:0/3452974058 learned_addr learned my addr 172.21.15.67:0/3452974058 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:03.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.220+0000 7f70e5220640 1 --2- 172.21.15.67:0/3452974058 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f70e015e2a0 0x7f70e0162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:03.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.220+0000 7f70e5a21640 1 -- 172.21.15.67:0/3452974058 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f70e015e2a0 msgr2=0x7f70e0162630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:03.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.220+0000 7f70e5a21640 1 --2- 172.21.15.67:0/3452974058 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f70e015e2a0 0x7f70e0162630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.220+0000 7f70e5a21640 1 -- 172.21.15.67:0/3452974058 >> v1:172.21.15.72:6789/0 conn(0x7f70e00d07e0 legacy=0x7f70e00b5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:03.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.220+0000 7f70e5a21640 1 -- 172.21.15.67:0/3452974058 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70e00cf080 con 0x7f70e015dec0 2024-03-24T00:45:03.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.220+0000 7f70e5220640 1 --2- 172.21.15.67:0/3452974058 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f70e015e2a0 0x7f70e0162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:45:03.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.220+0000 7f70e5a21640 1 --2- 172.21.15.67:0/3452974058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70e015dec0 0x7f70e00ce630 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f70e00cefb0 tx=0x7f70dc02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecb4db2cf875f92 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:03.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.221+0000 7f70e4a1f640 1 -- 172.21.15.67:0/3452974058 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70dc0317c0 con 0x7f70e015dec0 2024-03-24T00:45:03.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.221+0000 7f70e4a1f640 1 -- 172.21.15.67:0/3452974058 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70dc037070 con 0x7f70e015dec0 2024-03-24T00:45:03.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.221+0000 7f70e4a1f640 1 -- 172.21.15.67:0/3452974058 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70dc031b80 con 0x7f70e015dec0 2024-03-24T00:45:03.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.221+0000 7f70e7c85640 1 -- 172.21.15.67:0/3452974058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70e015dec0 msgr2=0x7f70e00ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:03.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.221+0000 7f70e7c85640 1 --2- 172.21.15.67:0/3452974058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70e015dec0 0x7f70e00ce630 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f70e00cefb0 tx=0x7f70dc02b860 comp rx=0 tx=0).stop 2024-03-24T00:45:03.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.221+0000 7f70e7c85640 1 -- 172.21.15.67:0/3452974058 shutdown_connections 2024-03-24T00:45:03.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.221+0000 7f70e7c85640 1 --2- 172.21.15.67:0/3452974058 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f70e015e2a0 0x7f70e0162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.221+0000 7f70e7c85640 1 --2- 172.21.15.67:0/3452974058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70e015dec0 0x7f70e00ce630 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.221+0000 7f70e7c85640 1 -- 172.21.15.67:0/3452974058 >> 172.21.15.67:0/3452974058 conn(0x7f70e006c810 msgr2=0x7f70e006cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:03.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.221+0000 7f70e7c85640 1 -- 172.21.15.67:0/3452974058 shutdown_connections 2024-03-24T00:45:03.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.221+0000 7f70e7c85640 1 -- 172.21.15.67:0/3452974058 wait complete. 2024-03-24T00:45:03.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.222+0000 7f70e7c85640 1 Processor -- start 2024-03-24T00:45:03.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.222+0000 7f70e7c85640 1 -- start start 2024-03-24T00:45:03.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.223+0000 7f70e7c85640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70e00d07e0 0x7f70e00abfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:03.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.223+0000 7f70e7c85640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f70e015dec0 0x7f70e015f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:03.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.223+0000 7f70e7c85640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f70e015e2a0 0x7f70e015f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:03.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.223+0000 7f70e7c85640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e00d3740 con 0x7f70e00d07e0 2024-03-24T00:45:03.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.223+0000 7f70e7c85640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e00d35c0 con 0x7f70e015e2a0 2024-03-24T00:45:03.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.223+0000 7f70e7c85640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70e00d38c0 con 0x7f70e015dec0 2024-03-24T00:45:03.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.223+0000 7f70e5a21640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70e00d07e0 0x7f70e00abfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:03.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.223+0000 7f70e5a21640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70e00d07e0 0x7f70e00abfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45220/0 (socket says 172.21.15.67:45220) 2024-03-24T00:45:03.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.223+0000 7f70e5a21640 1 -- 172.21.15.67:0/1018786595 learned_addr learned my addr 172.21.15.67:0/1018786595 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:03.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.223+0000 7f70e5220640 1 --2- 172.21.15.67:0/1018786595 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f70e015dec0 0x7f70e015f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:03.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.223+0000 7f70e6222640 1 --2- 172.21.15.67:0/1018786595 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f70e015e2a0 0x7f70e015f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:03.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.223+0000 7f70e5a21640 1 -- 172.21.15.67:0/1018786595 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f70e015dec0 msgr2=0x7f70e015f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:03.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.223+0000 7f70e5a21640 1 --2- 172.21.15.67:0/1018786595 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f70e015dec0 0x7f70e015f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.223+0000 7f70e5a21640 1 -- 172.21.15.67:0/1018786595 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f70e015e2a0 msgr2=0x7f70e015f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:03.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.223+0000 7f70e5a21640 1 --2- 172.21.15.67:0/1018786595 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f70e015e2a0 0x7f70e015f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.223+0000 7f70e5a21640 1 -- 172.21.15.67:0/1018786595 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70e00cb040 con 0x7f70e00d07e0 2024-03-24T00:45:03.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.224+0000 7f70e6222640 1 --2- 172.21.15.67:0/1018786595 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f70e015e2a0 0x7f70e015f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:45:03.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.224+0000 7f70e5a21640 1 --2- 172.21.15.67:0/1018786595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70e00d07e0 0x7f70e00abfc0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f70dc000c00 tx=0x7f70dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:03.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.224+0000 7f70d67fc640 1 -- 172.21.15.67:0/1018786595 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70dc032860 con 0x7f70e00d07e0 2024-03-24T00:45:03.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.224+0000 7f70d67fc640 1 -- 172.21.15.67:0/1018786595 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70dc032a00 con 0x7f70e00d07e0 2024-03-24T00:45:03.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.224+0000 7f70e7c85640 1 -- 172.21.15.67:0/1018786595 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70e00cb320 con 0x7f70e00d07e0 2024-03-24T00:45:03.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.224+0000 7f70d67fc640 1 -- 172.21.15.67:0/1018786595 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70dc032dd0 con 0x7f70e00d07e0 2024-03-24T00:45:03.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.224+0000 7f70e7c85640 1 -- 172.21.15.67:0/1018786595 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70e00cb8e0 con 0x7f70e00d07e0 2024-03-24T00:45:03.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.225+0000 7f70e7c85640 1 -- 172.21.15.67:0/1018786595 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70e004f400 con 0x7f70e00d07e0 2024-03-24T00:45:03.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.225+0000 7f70d67fc640 1 -- 172.21.15.67:0/1018786595 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f70dc04c060 con 0x7f70e00d07e0 2024-03-24T00:45:03.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.226+0000 7f70d67fc640 1 --2- 172.21.15.67:0/1018786595 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f70c406bbd0 0x7f70c406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:03.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.226+0000 7f70d67fc640 1 -- 172.21.15.67:0/1018786595 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f70dc0ac5e0 con 0x7f70e00d07e0 2024-03-24T00:45:03.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.226+0000 7f70e5220640 1 --2- 172.21.15.67:0/1018786595 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f70c406bbd0 0x7f70c406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:03.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.226+0000 7f70e5220640 1 --2- 172.21.15.67:0/1018786595 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f70c406bbd0 0x7f70c406e090 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f70e015e680 tx=0x7f70d0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:03.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.230+0000 7f70d67fc640 1 -- 172.21.15.67:0/1018786595 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f70dc040070 con 0x7f70e00d07e0 2024-03-24T00:45:03.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.415+0000 7f70e7c85640 1 -- 172.21.15.67:0/1018786595 --> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f70e00cbb20 con 0x7f70c406bbd0 2024-03-24T00:45:03.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.422+0000 7f70d67fc640 1 -- 172.21.15.67:0/1018786595 <== mgr.4105 v2:172.21.15.72:6832/2768292274 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252075 (secure 0 0 0) 0x7f70e00cbb20 con 0x7f70c406bbd0 2024-03-24T00:45:03.424 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:45:03.425 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-24T00:45:03.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.425+0000 7f70e7c85640 1 -- 172.21.15.67:0/1018786595 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f70c406bbd0 msgr2=0x7f70c406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:03.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.425+0000 7f70e7c85640 1 --2- 172.21.15.67:0/1018786595 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f70c406bbd0 0x7f70c406e090 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f70e015e680 tx=0x7f70d0008040 comp rx=0 tx=0).stop 2024-03-24T00:45:03.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.425+0000 7f70e7c85640 1 -- 172.21.15.67:0/1018786595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70e00d07e0 msgr2=0x7f70e00abfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:03.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.425+0000 7f70e7c85640 1 --2- 172.21.15.67:0/1018786595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70e00d07e0 0x7f70e00abfc0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f70dc000c00 tx=0x7f70dc0046c0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.425+0000 7f70e7c85640 1 -- 172.21.15.67:0/1018786595 shutdown_connections 2024-03-24T00:45:03.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.425+0000 7f70e7c85640 1 --2- 172.21.15.67:0/1018786595 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f70c406bbd0 0x7f70c406e090 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.425+0000 7f70e7c85640 1 --2- 172.21.15.67:0/1018786595 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f70e015e2a0 0x7f70e015f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.425+0000 7f70e7c85640 1 --2- 172.21.15.67:0/1018786595 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f70e015dec0 0x7f70e015f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.425+0000 7f70e7c85640 1 --2- 172.21.15.67:0/1018786595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70e00d07e0 0x7f70e00abfc0 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.425+0000 7f70e7c85640 1 -- 172.21.15.67:0/1018786595 >> 172.21.15.67:0/1018786595 conn(0x7f70e006c810 msgr2=0x7f70e00b90a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:03.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.425+0000 7f70e7c85640 1 -- 172.21.15.67:0/1018786595 shutdown_connections 2024-03-24T00:45:03.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.425+0000 7f70e7c85640 1 -- 172.21.15.67:0/1018786595 wait complete. 2024-03-24T00:45:03.435 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":274,"stamp":"2024-03-24T00:45:03.410058+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":184,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":259268,"kb_used_data":9800,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754715452,"statfs":{"total":773094113280,"available":772828622848,"internally_reserved":0,"allocated":10035200,"data_stored":5227196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.030962"},"pg_stats":[{"pgid":"14.3f","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.514662+0000","last_change":"2024-03-24T00:44:55.387093+0000","last_active":"2024-03-24T00:44:57.514662+0000","last_peered":"2024-03-24T00:44:57.514662+0000","last_clean":"2024-03-24T00:44:57.514662+0000","last_became_active":"2024-03-24T00:44:55.386764+0000","last_became_peered":"2024-03-24T00:44:55.386764+0000","last_unstale":"2024-03-24T00:44:57.514662+0000","last_undegraded":"2024-03-24T00:44:57.514662+0000","last_fullsized":"2024-03-24T00:44:57.514662+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:17:11.603293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.385353+0000","last_change":"2024-03-24T00:44:55.393287+0000","last_active":"2024-03-24T00:44:56.385353+0000","last_peered":"2024-03-24T00:44:56.385353+0000","last_clean":"2024-03-24T00:44:56.385353+0000","last_became_active":"2024-03-24T00:44:55.393012+0000","last_became_peered":"2024-03-24T00:44:55.393012+0000","last_unstale":"2024-03-24T00:44:56.385353+0000","last_undegraded":"2024-03-24T00:44:56.385353+0000","last_fullsized":"2024-03-24T00:44:56.385353+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:45:58.413741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3d","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.514517+0000","last_change":"2024-03-24T00:44:55.390994+0000","last_active":"2024-03-24T00:44:57.514517+0000","last_peered":"2024-03-24T00:44:57.514517+0000","last_clean":"2024-03-24T00:44:57.514517+0000","last_became_active":"2024-03-24T00:44:55.390746+0000","last_became_peered":"2024-03-24T00:44:55.390746+0000","last_unstale":"2024-03-24T00:44:57.514517+0000","last_undegraded":"2024-03-24T00:44:57.514517+0000","last_fullsized":"2024-03-24T00:44:57.514517+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:19:39.501718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.777676+0000","last_change":"2024-03-24T00:44:55.396111+0000","last_active":"2024-03-24T00:44:57.777676+0000","last_peered":"2024-03-24T00:44:57.777676+0000","last_clean":"2024-03-24T00:44:57.777676+0000","last_became_active":"2024-03-24T00:44:55.395858+0000","last_became_peered":"2024-03-24T00:44:55.395858+0000","last_unstale":"2024-03-24T00:44:57.777676+0000","last_undegraded":"2024-03-24T00:44:57.777676+0000","last_fullsized":"2024-03-24T00:44:57.777676+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:19:21.001867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.386233+0000","last_change":"2024-03-24T00:44:55.392701+0000","last_active":"2024-03-24T00:44:56.386233+0000","last_peered":"2024-03-24T00:44:56.386233+0000","last_clean":"2024-03-24T00:44:56.386233+0000","last_became_active":"2024-03-24T00:44:55.392361+0000","last_became_peered":"2024-03-24T00:44:55.392361+0000","last_unstale":"2024-03-24T00:44:56.386233+0000","last_undegraded":"2024-03-24T00:44:56.386233+0000","last_fullsized":"2024-03-24T00:44:56.386233+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:16:07.455446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.382497+0000","last_change":"2024-03-24T00:44:55.411755+0000","last_active":"2024-03-24T00:44:56.382497+0000","last_peered":"2024-03-24T00:44:56.382497+0000","last_clean":"2024-03-24T00:44:56.382497+0000","last_became_active":"2024-03-24T00:44:55.411469+0000","last_became_peered":"2024-03-24T00:44:55.411469+0000","last_unstale":"2024-03-24T00:44:56.382497+0000","last_undegraded":"2024-03-24T00:44:56.382497+0000","last_fullsized":"2024-03-24T00:44:56.382497+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:50:38.597482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.729131+0000","last_change":"2024-03-24T00:44:55.398900+0000","last_active":"2024-03-24T00:44:56.729131+0000","last_peered":"2024-03-24T00:44:56.729131+0000","last_clean":"2024-03-24T00:44:56.729131+0000","last_became_active":"2024-03-24T00:44:55.398002+0000","last_became_peered":"2024-03-24T00:44:55.398002+0000","last_unstale":"2024-03-24T00:44:56.729131+0000","last_undegraded":"2024-03-24T00:44:56.729131+0000","last_fullsized":"2024-03-24T00:44:56.729131+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:39:23.097646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.777262+0000","last_change":"2024-03-24T00:44:55.407339+0000","last_active":"2024-03-24T00:44:57.777262+0000","last_peered":"2024-03-24T00:44:57.777262+0000","last_clean":"2024-03-24T00:44:57.777262+0000","last_became_active":"2024-03-24T00:44:55.407066+0000","last_became_peered":"2024-03-24T00:44:55.407066+0000","last_unstale":"2024-03-24T00:44:57.777262+0000","last_undegraded":"2024-03-24T00:44:57.777262+0000","last_fullsized":"2024-03-24T00:44:57.777262+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:33:37.494533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.386774+0000","last_change":"2024-03-24T00:44:55.396053+0000","last_active":"2024-03-24T00:44:56.386774+0000","last_peered":"2024-03-24T00:44:56.386774+0000","last_clean":"2024-03-24T00:44:56.386774+0000","last_became_active":"2024-03-24T00:44:55.395764+0000","last_became_peered":"2024-03-24T00:44:55.395764+0000","last_unstale":"2024-03-24T00:44:56.386774+0000","last_undegraded":"2024-03-24T00:44:56.386774+0000","last_fullsized":"2024-03-24T00:44:56.386774+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:18:39.887633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.777394+0000","last_change":"2024-03-24T00:44:55.407603+0000","last_active":"2024-03-24T00:44:57.777394+0000","last_peered":"2024-03-24T00:44:57.777394+0000","last_clean":"2024-03-24T00:44:57.777394+0000","last_became_active":"2024-03-24T00:44:55.407319+0000","last_became_peered":"2024-03-24T00:44:55.407319+0000","last_unstale":"2024-03-24T00:44:57.777394+0000","last_undegraded":"2024-03-24T00:44:57.777394+0000","last_fullsized":"2024-03-24T00:44:57.777394+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:03:07.602179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.386770+0000","last_change":"2024-03-24T00:44:55.404378+0000","last_active":"2024-03-24T00:44:56.386770+0000","last_peered":"2024-03-24T00:44:56.386770+0000","last_clean":"2024-03-24T00:44:56.386770+0000","last_became_active":"2024-03-24T00:44:55.404111+0000","last_became_peered":"2024-03-24T00:44:55.404111+0000","last_unstale":"2024-03-24T00:44:56.386770+0000","last_undegraded":"2024-03-24T00:44:56.386770+0000","last_fullsized":"2024-03-24T00:44:56.386770+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:16:13.352120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.776854+0000","last_change":"2024-03-24T00:44:55.398924+0000","last_active":"2024-03-24T00:44:57.776854+0000","last_peered":"2024-03-24T00:44:57.776854+0000","last_clean":"2024-03-24T00:44:57.776854+0000","last_became_active":"2024-03-24T00:44:55.398672+0000","last_became_peered":"2024-03-24T00:44:55.398672+0000","last_unstale":"2024-03-24T00:44:57.776854+0000","last_undegraded":"2024-03-24T00:44:57.776854+0000","last_fullsized":"2024-03-24T00:44:57.776854+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:26:29.358335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.777450+0000","last_change":"2024-03-24T00:44:55.393133+0000","last_active":"2024-03-24T00:44:57.777450+0000","last_peered":"2024-03-24T00:44:57.777450+0000","last_clean":"2024-03-24T00:44:57.777450+0000","last_became_active":"2024-03-24T00:44:55.392877+0000","last_became_peered":"2024-03-24T00:44:55.392877+0000","last_unstale":"2024-03-24T00:44:57.777450+0000","last_undegraded":"2024-03-24T00:44:57.777450+0000","last_fullsized":"2024-03-24T00:44:57.777450+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:46:50.358541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.386025+0000","last_change":"2024-03-24T00:44:55.398283+0000","last_active":"2024-03-24T00:44:56.386025+0000","last_peered":"2024-03-24T00:44:56.386025+0000","last_clean":"2024-03-24T00:44:56.386025+0000","last_became_active":"2024-03-24T00:44:55.398003+0000","last_became_peered":"2024-03-24T00:44:55.398003+0000","last_unstale":"2024-03-24T00:44:56.386025+0000","last_undegraded":"2024-03-24T00:44:56.386025+0000","last_fullsized":"2024-03-24T00:44:56.386025+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:56:22.901806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.777011+0000","last_change":"2024-03-24T00:44:55.398640+0000","last_active":"2024-03-24T00:44:57.777011+0000","last_peered":"2024-03-24T00:44:57.777011+0000","last_clean":"2024-03-24T00:44:57.777011+0000","last_became_active":"2024-03-24T00:44:55.398396+0000","last_became_peered":"2024-03-24T00:44:55.398396+0000","last_unstale":"2024-03-24T00:44:57.777011+0000","last_undegraded":"2024-03-24T00:44:57.777011+0000","last_fullsized":"2024-03-24T00:44:57.777011+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:50:18.227568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.382424+0000","last_change":"2024-03-24T00:44:55.407535+0000","last_active":"2024-03-24T00:44:56.382424+0000","last_peered":"2024-03-24T00:44:56.382424+0000","last_clean":"2024-03-24T00:44:56.382424+0000","last_became_active":"2024-03-24T00:44:55.407267+0000","last_became_peered":"2024-03-24T00:44:55.407267+0000","last_unstale":"2024-03-24T00:44:56.382424+0000","last_undegraded":"2024-03-24T00:44:56.382424+0000","last_fullsized":"2024-03-24T00:44:56.382424+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:36:59.898017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.382873+0000","last_change":"2024-03-24T00:44:55.398350+0000","last_active":"2024-03-24T00:44:56.382873+0000","last_peered":"2024-03-24T00:44:56.382873+0000","last_clean":"2024-03-24T00:44:56.382873+0000","last_became_active":"2024-03-24T00:44:55.396755+0000","last_became_peered":"2024-03-24T00:44:55.396755+0000","last_unstale":"2024-03-24T00:44:56.382873+0000","last_undegraded":"2024-03-24T00:44:56.382873+0000","last_fullsized":"2024-03-24T00:44:56.382873+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:17:48.749325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.514362+0000","last_change":"2024-03-24T00:44:55.385251+0000","last_active":"2024-03-24T00:44:57.514362+0000","last_peered":"2024-03-24T00:44:57.514362+0000","last_clean":"2024-03-24T00:44:57.514362+0000","last_became_active":"2024-03-24T00:44:55.384991+0000","last_became_peered":"2024-03-24T00:44:55.384991+0000","last_unstale":"2024-03-24T00:44:57.514362+0000","last_undegraded":"2024-03-24T00:44:57.514362+0000","last_fullsized":"2024-03-24T00:44:57.514362+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:45:52.622468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.386621+0000","last_change":"2024-03-24T00:44:55.398470+0000","last_active":"2024-03-24T00:44:56.386621+0000","last_peered":"2024-03-24T00:44:56.386621+0000","last_clean":"2024-03-24T00:44:56.386621+0000","last_became_active":"2024-03-24T00:44:55.397407+0000","last_became_peered":"2024-03-24T00:44:55.397407+0000","last_unstale":"2024-03-24T00:44:56.386621+0000","last_undegraded":"2024-03-24T00:44:56.386621+0000","last_fullsized":"2024-03-24T00:44:56.386621+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:00:26.656749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.777726+0000","last_change":"2024-03-24T00:44:55.397440+0000","last_active":"2024-03-24T00:44:57.777726+0000","last_peered":"2024-03-24T00:44:57.777726+0000","last_clean":"2024-03-24T00:44:57.777726+0000","last_became_active":"2024-03-24T00:44:55.397147+0000","last_became_peered":"2024-03-24T00:44:55.397147+0000","last_unstale":"2024-03-24T00:44:57.777726+0000","last_undegraded":"2024-03-24T00:44:57.777726+0000","last_fullsized":"2024-03-24T00:44:57.777726+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:59:59.398432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.386157+0000","last_change":"2024-03-24T00:44:55.390870+0000","last_active":"2024-03-24T00:44:56.386157+0000","last_peered":"2024-03-24T00:44:56.386157+0000","last_clean":"2024-03-24T00:44:56.386157+0000","last_became_active":"2024-03-24T00:44:55.390570+0000","last_became_peered":"2024-03-24T00:44:55.390570+0000","last_unstale":"2024-03-24T00:44:56.386157+0000","last_undegraded":"2024-03-24T00:44:56.386157+0000","last_fullsized":"2024-03-24T00:44:56.386157+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:18:31.095437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.386062+0000","last_change":"2024-03-24T00:44:55.397511+0000","last_active":"2024-03-24T00:44:56.386062+0000","last_peered":"2024-03-24T00:44:56.386062+0000","last_clean":"2024-03-24T00:44:56.386062+0000","last_became_active":"2024-03-24T00:44:55.397188+0000","last_became_peered":"2024-03-24T00:44:55.397188+0000","last_unstale":"2024-03-24T00:44:56.386062+0000","last_undegraded":"2024-03-24T00:44:56.386062+0000","last_fullsized":"2024-03-24T00:44:56.386062+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:13:32.135329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.406289+0000","last_change":"2024-03-24T00:44:55.399158+0000","last_active":"2024-03-24T00:44:57.406289+0000","last_peered":"2024-03-24T00:44:57.406289+0000","last_clean":"2024-03-24T00:44:57.406289+0000","last_became_active":"2024-03-24T00:44:55.398538+0000","last_became_peered":"2024-03-24T00:44:55.398538+0000","last_unstale":"2024-03-24T00:44:57.406289+0000","last_undegraded":"2024-03-24T00:44:57.406289+0000","last_fullsized":"2024-03-24T00:44:57.406289+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:18:31.095437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.514125+0000","last_change":"2024-03-24T00:44:55.388719+0000","last_active":"2024-03-24T00:44:57.514125+0000","last_peered":"2024-03-24T00:44:57.514125+0000","last_clean":"2024-03-24T00:44:57.514125+0000","last_became_active":"2024-03-24T00:44:55.387921+0000","last_became_peered":"2024-03-24T00:44:55.387921+0000","last_unstale":"2024-03-24T00:44:57.514125+0000","last_undegraded":"2024-03-24T00:44:57.514125+0000","last_fullsized":"2024-03-24T00:44:57.514125+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:39:23.097646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.385512+0000","last_change":"2024-03-24T00:44:55.392532+0000","last_active":"2024-03-24T00:44:56.385512+0000","last_peered":"2024-03-24T00:44:56.385512+0000","last_clean":"2024-03-24T00:44:56.385512+0000","last_became_active":"2024-03-24T00:44:55.392200+0000","last_became_peered":"2024-03-24T00:44:55.392200+0000","last_unstale":"2024-03-24T00:44:56.385512+0000","last_undegraded":"2024-03-24T00:44:56.385512+0000","last_fullsized":"2024-03-24T00:44:56.385512+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:38:36.247055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.386585+0000","last_change":"2024-03-24T00:44:55.397669+0000","last_active":"2024-03-24T00:44:56.386585+0000","last_peered":"2024-03-24T00:44:56.386585+0000","last_clean":"2024-03-24T00:44:56.386585+0000","last_became_active":"2024-03-24T00:44:55.397301+0000","last_became_peered":"2024-03-24T00:44:55.397301+0000","last_unstale":"2024-03-24T00:44:56.386585+0000","last_undegraded":"2024-03-24T00:44:56.386585+0000","last_fullsized":"2024-03-24T00:44:56.386585+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:44:48.580022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.777003+0000","last_change":"2024-03-24T00:44:55.398384+0000","last_active":"2024-03-24T00:44:57.777003+0000","last_peered":"2024-03-24T00:44:57.777003+0000","last_clean":"2024-03-24T00:44:57.777003+0000","last_became_active":"2024-03-24T00:44:55.398105+0000","last_became_peered":"2024-03-24T00:44:55.398105+0000","last_unstale":"2024-03-24T00:44:57.777003+0000","last_undegraded":"2024-03-24T00:44:57.777003+0000","last_fullsized":"2024-03-24T00:44:57.777003+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:21:15.581002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.777817+0000","last_change":"2024-03-24T00:44:55.405559+0000","last_active":"2024-03-24T00:44:57.777817+0000","last_peered":"2024-03-24T00:44:57.777817+0000","last_clean":"2024-03-24T00:44:57.777817+0000","last_became_active":"2024-03-24T00:44:55.405275+0000","last_became_peered":"2024-03-24T00:44:55.405275+0000","last_unstale":"2024-03-24T00:44:57.777817+0000","last_undegraded":"2024-03-24T00:44:57.777817+0000","last_fullsized":"2024-03-24T00:44:57.777817+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:28:05.467698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.729245+0000","last_change":"2024-03-24T00:44:55.398815+0000","last_active":"2024-03-24T00:44:56.729245+0000","last_peered":"2024-03-24T00:44:56.729245+0000","last_clean":"2024-03-24T00:44:56.729245+0000","last_became_active":"2024-03-24T00:44:55.398185+0000","last_became_peered":"2024-03-24T00:44:55.398185+0000","last_unstale":"2024-03-24T00:44:56.729245+0000","last_undegraded":"2024-03-24T00:44:56.729245+0000","last_fullsized":"2024-03-24T00:44:56.729245+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:38:36.247055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.386231+0000","last_change":"2024-03-24T00:44:55.400229+0000","last_active":"2024-03-24T00:44:56.386231+0000","last_peered":"2024-03-24T00:44:56.386231+0000","last_clean":"2024-03-24T00:44:56.386231+0000","last_became_active":"2024-03-24T00:44:55.399941+0000","last_became_peered":"2024-03-24T00:44:55.399941+0000","last_unstale":"2024-03-24T00:44:56.386231+0000","last_undegraded":"2024-03-24T00:44:56.386231+0000","last_fullsized":"2024-03-24T00:44:56.386231+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:18:51.669689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.384981+0000","last_change":"2024-03-24T00:44:55.395189+0000","last_active":"2024-03-24T00:44:56.384981+0000","last_peered":"2024-03-24T00:44:56.384981+0000","last_clean":"2024-03-24T00:44:56.384981+0000","last_became_active":"2024-03-24T00:44:55.394811+0000","last_became_peered":"2024-03-24T00:44:55.394811+0000","last_unstale":"2024-03-24T00:44:56.384981+0000","last_undegraded":"2024-03-24T00:44:56.384981+0000","last_fullsized":"2024-03-24T00:44:56.384981+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:05:25.799252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.382432+0000","last_change":"2024-03-24T00:44:55.405118+0000","last_active":"2024-03-24T00:44:56.382432+0000","last_peered":"2024-03-24T00:44:56.382432+0000","last_clean":"2024-03-24T00:44:56.382432+0000","last_became_active":"2024-03-24T00:44:55.404477+0000","last_became_peered":"2024-03-24T00:44:55.404477+0000","last_unstale":"2024-03-24T00:44:56.382432+0000","last_undegraded":"2024-03-24T00:44:56.382432+0000","last_fullsized":"2024-03-24T00:44:56.382432+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:04:03.451551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.382699+0000","last_change":"2024-03-24T00:44:55.402620+0000","last_active":"2024-03-24T00:44:56.382699+0000","last_peered":"2024-03-24T00:44:56.382699+0000","last_clean":"2024-03-24T00:44:56.382699+0000","last_became_active":"2024-03-24T00:44:55.401195+0000","last_became_peered":"2024-03-24T00:44:55.401195+0000","last_unstale":"2024-03-24T00:44:56.382699+0000","last_undegraded":"2024-03-24T00:44:56.382699+0000","last_fullsized":"2024-03-24T00:44:56.382699+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:44:15.022767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"88'4","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.406465+0000","last_change":"2024-03-24T00:44:54.396737+0000","last_active":"2024-03-24T00:44:57.406465+0000","last_peered":"2024-03-24T00:44:57.406465+0000","last_clean":"2024-03-24T00:44:57.406465+0000","last_became_active":"2024-03-24T00:44:54.396431+0000","last_became_peered":"2024-03-24T00:44:54.396431+0000","last_unstale":"2024-03-24T00:44:57.406465+0000","last_undegraded":"2024-03-24T00:44:57.406465+0000","last_fullsized":"2024-03-24T00:44:57.406465+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:54:54.886053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.382799+0000","last_change":"2024-03-24T00:44:55.401707+0000","last_active":"2024-03-24T00:44:56.382799+0000","last_peered":"2024-03-24T00:44:56.382799+0000","last_clean":"2024-03-24T00:44:56.382799+0000","last_became_active":"2024-03-24T00:44:55.400657+0000","last_became_peered":"2024-03-24T00:44:55.400657+0000","last_unstale":"2024-03-24T00:44:56.382799+0000","last_undegraded":"2024-03-24T00:44:56.382799+0000","last_fullsized":"2024-03-24T00:44:56.382799+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:40:43.577674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"88'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.386644+0000","last_change":"2024-03-24T00:44:54.392014+0000","last_active":"2024-03-24T00:44:56.386644+0000","last_peered":"2024-03-24T00:44:56.386644+0000","last_clean":"2024-03-24T00:44:56.386644+0000","last_became_active":"2024-03-24T00:44:54.390205+0000","last_became_peered":"2024-03-24T00:44:54.390205+0000","last_unstale":"2024-03-24T00:44:56.386644+0000","last_undegraded":"2024-03-24T00:44:56.386644+0000","last_fullsized":"2024-03-24T00:44:56.386644+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:33:34.699913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.729554+0000","last_change":"2024-03-24T00:44:55.401412+0000","last_active":"2024-03-24T00:44:56.729554+0000","last_peered":"2024-03-24T00:44:56.729554+0000","last_clean":"2024-03-24T00:44:56.729554+0000","last_became_active":"2024-03-24T00:44:55.401105+0000","last_became_peered":"2024-03-24T00:44:55.401105+0000","last_unstale":"2024-03-24T00:44:56.729554+0000","last_undegraded":"2024-03-24T00:44:56.729554+0000","last_fullsized":"2024-03-24T00:44:56.729554+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:19:39.501718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"88'2","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.729205+0000","last_change":"2024-03-24T00:44:54.390117+0000","last_active":"2024-03-24T00:44:56.729205+0000","last_peered":"2024-03-24T00:44:56.729205+0000","last_clean":"2024-03-24T00:44:56.729205+0000","last_became_active":"2024-03-24T00:44:54.389849+0000","last_became_peered":"2024-03-24T00:44:54.389849+0000","last_unstale":"2024-03-24T00:44:56.729205+0000","last_undegraded":"2024-03-24T00:44:56.729205+0000","last_fullsized":"2024-03-24T00:44:56.729205+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:13:30.737465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.383050+0000","last_change":"2024-03-24T00:44:55.398111+0000","last_active":"2024-03-24T00:44:56.383050+0000","last_peered":"2024-03-24T00:44:56.383050+0000","last_clean":"2024-03-24T00:44:56.383050+0000","last_became_active":"2024-03-24T00:44:55.397334+0000","last_became_peered":"2024-03-24T00:44:55.397334+0000","last_unstale":"2024-03-24T00:44:56.383050+0000","last_undegraded":"2024-03-24T00:44:56.383050+0000","last_fullsized":"2024-03-24T00:44:56.383050+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:33:11.855966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"88'2","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.406406+0000","last_change":"2024-03-24T00:44:54.393213+0000","last_active":"2024-03-24T00:44:57.406406+0000","last_peered":"2024-03-24T00:44:57.406406+0000","last_clean":"2024-03-24T00:44:57.406406+0000","last_became_active":"2024-03-24T00:44:54.392741+0000","last_became_peered":"2024-03-24T00:44:54.392741+0000","last_unstale":"2024-03-24T00:44:57.406406+0000","last_undegraded":"2024-03-24T00:44:57.406406+0000","last_fullsized":"2024-03-24T00:44:57.406406+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:23:31.116823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.777295+0000","last_change":"2024-03-24T00:44:55.398025+0000","last_active":"2024-03-24T00:44:57.777295+0000","last_peered":"2024-03-24T00:44:57.777295+0000","last_clean":"2024-03-24T00:44:57.777295+0000","last_became_active":"2024-03-24T00:44:55.397785+0000","last_became_peered":"2024-03-24T00:44:55.397785+0000","last_unstale":"2024-03-24T00:44:57.777295+0000","last_undegraded":"2024-03-24T00:44:57.777295+0000","last_fullsized":"2024-03-24T00:44:57.777295+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:00:18.603927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"88'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.386228+0000","last_change":"2024-03-24T00:44:54.391305+0000","last_active":"2024-03-24T00:44:56.386228+0000","last_peered":"2024-03-24T00:44:56.386228+0000","last_clean":"2024-03-24T00:44:56.386228+0000","last_became_active":"2024-03-24T00:44:54.390877+0000","last_became_peered":"2024-03-24T00:44:54.390877+0000","last_unstale":"2024-03-24T00:44:56.386228+0000","last_undegraded":"2024-03-24T00:44:56.386228+0000","last_fullsized":"2024-03-24T00:44:56.386228+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:35:15.833860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.406292+0000","last_change":"2024-03-24T00:44:55.399402+0000","last_active":"2024-03-24T00:44:57.406292+0000","last_peered":"2024-03-24T00:44:57.406292+0000","last_clean":"2024-03-24T00:44:57.406292+0000","last_became_active":"2024-03-24T00:44:55.398894+0000","last_became_peered":"2024-03-24T00:44:55.398894+0000","last_unstale":"2024-03-24T00:44:57.406292+0000","last_undegraded":"2024-03-24T00:44:57.406292+0000","last_fullsized":"2024-03-24T00:44:57.406292+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:16:07.455446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"88'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.777682+0000","last_change":"2024-03-24T00:44:54.393934+0000","last_active":"2024-03-24T00:44:57.777682+0000","last_peered":"2024-03-24T00:44:57.777682+0000","last_clean":"2024-03-24T00:44:57.777682+0000","last_became_active":"2024-03-24T00:44:54.393459+0000","last_became_peered":"2024-03-24T00:44:54.393459+0000","last_unstale":"2024-03-24T00:44:57.777682+0000","last_undegraded":"2024-03-24T00:44:57.777682+0000","last_fullsized":"2024-03-24T00:44:57.777682+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:56:21.899623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.382606+0000","last_change":"2024-03-24T00:44:55.410380+0000","last_active":"2024-03-24T00:44:56.382606+0000","last_peered":"2024-03-24T00:44:56.382606+0000","last_clean":"2024-03-24T00:44:56.382606+0000","last_became_active":"2024-03-24T00:44:55.410099+0000","last_became_peered":"2024-03-24T00:44:55.410099+0000","last_unstale":"2024-03-24T00:44:56.382606+0000","last_undegraded":"2024-03-24T00:44:56.382606+0000","last_fullsized":"2024-03-24T00:44:56.382606+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:33:49.693865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.514115+0000","last_change":"2024-03-24T00:44:54.383196+0000","last_active":"2024-03-24T00:44:57.514115+0000","last_peered":"2024-03-24T00:44:57.514115+0000","last_clean":"2024-03-24T00:44:57.514115+0000","last_became_active":"2024-03-24T00:44:54.382875+0000","last_became_peered":"2024-03-24T00:44:54.382875+0000","last_unstale":"2024-03-24T00:44:57.514115+0000","last_undegraded":"2024-03-24T00:44:57.514115+0000","last_fullsized":"2024-03-24T00:44:57.514115+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:32:13.875499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.777537+0000","last_change":"2024-03-24T00:44:55.393864+0000","last_active":"2024-03-24T00:44:57.777537+0000","last_peered":"2024-03-24T00:44:57.777537+0000","last_clean":"2024-03-24T00:44:57.777537+0000","last_became_active":"2024-03-24T00:44:55.393588+0000","last_became_peered":"2024-03-24T00:44:55.393588+0000","last_unstale":"2024-03-24T00:44:57.777537+0000","last_undegraded":"2024-03-24T00:44:57.777537+0000","last_fullsized":"2024-03-24T00:44:57.777537+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:21:21.650742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.729046+0000","last_change":"2024-03-24T00:44:54.385598+0000","last_active":"2024-03-24T00:44:56.729046+0000","last_peered":"2024-03-24T00:44:56.729046+0000","last_clean":"2024-03-24T00:44:56.729046+0000","last_became_active":"2024-03-24T00:44:54.385270+0000","last_became_peered":"2024-03-24T00:44:54.385270+0000","last_unstale":"2024-03-24T00:44:56.729046+0000","last_undegraded":"2024-03-24T00:44:56.729046+0000","last_fullsized":"2024-03-24T00:44:56.729046+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:45:33.049844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.386469+0000","last_change":"2024-03-24T00:44:55.391244+0000","last_active":"2024-03-24T00:44:56.386469+0000","last_peered":"2024-03-24T00:44:56.386469+0000","last_clean":"2024-03-24T00:44:56.386469+0000","last_became_active":"2024-03-24T00:44:55.390752+0000","last_became_peered":"2024-03-24T00:44:55.390752+0000","last_unstale":"2024-03-24T00:44:56.386469+0000","last_undegraded":"2024-03-24T00:44:56.386469+0000","last_fullsized":"2024-03-24T00:44:56.386469+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:49:36.416272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.385545+0000","last_change":"2024-03-24T00:44:54.393609+0000","last_active":"2024-03-24T00:44:56.385545+0000","last_peered":"2024-03-24T00:44:56.385545+0000","last_clean":"2024-03-24T00:44:56.385545+0000","last_became_active":"2024-03-24T00:44:54.393163+0000","last_became_peered":"2024-03-24T00:44:54.393163+0000","last_unstale":"2024-03-24T00:44:56.385545+0000","last_undegraded":"2024-03-24T00:44:56.385545+0000","last_fullsized":"2024-03-24T00:44:56.385545+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:09:59.913191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.729266+0000","last_change":"2024-03-24T00:44:55.400989+0000","last_active":"2024-03-24T00:44:56.729266+0000","last_peered":"2024-03-24T00:44:56.729266+0000","last_clean":"2024-03-24T00:44:56.729266+0000","last_became_active":"2024-03-24T00:44:55.400717+0000","last_became_peered":"2024-03-24T00:44:55.400717+0000","last_unstale":"2024-03-24T00:44:56.729266+0000","last_undegraded":"2024-03-24T00:44:56.729266+0000","last_fullsized":"2024-03-24T00:44:56.729266+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:45:52.622468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.514408+0000","last_change":"2024-03-24T00:44:54.391371+0000","last_active":"2024-03-24T00:44:57.514408+0000","last_peered":"2024-03-24T00:44:57.514408+0000","last_clean":"2024-03-24T00:44:57.514408+0000","last_became_active":"2024-03-24T00:44:54.391075+0000","last_became_peered":"2024-03-24T00:44:54.391075+0000","last_unstale":"2024-03-24T00:44:57.514408+0000","last_undegraded":"2024-03-24T00:44:57.514408+0000","last_fullsized":"2024-03-24T00:44:57.514408+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:13:30.737465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.514478+0000","last_change":"2024-03-24T00:44:55.397952+0000","last_active":"2024-03-24T00:44:57.514478+0000","last_peered":"2024-03-24T00:44:57.514478+0000","last_clean":"2024-03-24T00:44:57.514478+0000","last_became_active":"2024-03-24T00:44:55.397681+0000","last_became_peered":"2024-03-24T00:44:55.397681+0000","last_unstale":"2024-03-24T00:44:57.514478+0000","last_undegraded":"2024-03-24T00:44:57.514478+0000","last_fullsized":"2024-03-24T00:44:57.514478+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:54:55.383079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"88'3","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.386817+0000","last_change":"2024-03-24T00:44:54.395602+0000","last_active":"2024-03-24T00:44:56.386817+0000","last_peered":"2024-03-24T00:44:56.386817+0000","last_clean":"2024-03-24T00:44:56.386817+0000","last_became_active":"2024-03-24T00:44:54.395326+0000","last_became_peered":"2024-03-24T00:44:54.395326+0000","last_unstale":"2024-03-24T00:44:56.386817+0000","last_undegraded":"2024-03-24T00:44:56.386817+0000","last_fullsized":"2024-03-24T00:44:56.386817+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:40:16.672606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.777920+0000","last_change":"2024-03-24T00:44:55.405635+0000","last_active":"2024-03-24T00:44:57.777920+0000","last_peered":"2024-03-24T00:44:57.777920+0000","last_clean":"2024-03-24T00:44:57.777920+0000","last_became_active":"2024-03-24T00:44:55.405341+0000","last_became_peered":"2024-03-24T00:44:55.405341+0000","last_unstale":"2024-03-24T00:44:57.777920+0000","last_undegraded":"2024-03-24T00:44:57.777920+0000","last_fullsized":"2024-03-24T00:44:57.777920+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:45:34.052028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"88'6","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.382606+0000","last_change":"2024-03-24T00:44:54.389907+0000","last_active":"2024-03-24T00:44:56.382606+0000","last_peered":"2024-03-24T00:44:56.382606+0000","last_clean":"2024-03-24T00:44:56.382606+0000","last_became_active":"2024-03-24T00:44:54.389643+0000","last_became_peered":"2024-03-24T00:44:54.389643+0000","last_unstale":"2024-03-24T00:44:56.382606+0000","last_undegraded":"2024-03-24T00:44:56.382606+0000","last_fullsized":"2024-03-24T00:44:56.382606+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:14:44.327426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.777563+0000","last_change":"2024-03-24T00:44:55.406640+0000","last_active":"2024-03-24T00:44:57.777563+0000","last_peered":"2024-03-24T00:44:57.777563+0000","last_clean":"2024-03-24T00:44:57.777563+0000","last_became_active":"2024-03-24T00:44:55.406390+0000","last_became_peered":"2024-03-24T00:44:55.406390+0000","last_unstale":"2024-03-24T00:44:57.777563+0000","last_undegraded":"2024-03-24T00:44:57.777563+0000","last_fullsized":"2024-03-24T00:44:57.777563+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:29:00.989588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"88'2","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.729141+0000","last_change":"2024-03-24T00:44:54.384267+0000","last_active":"2024-03-24T00:44:56.729141+0000","last_peered":"2024-03-24T00:44:56.729141+0000","last_clean":"2024-03-24T00:44:56.729141+0000","last_became_active":"2024-03-24T00:44:54.383981+0000","last_became_peered":"2024-03-24T00:44:54.383981+0000","last_unstale":"2024-03-24T00:44:56.729141+0000","last_undegraded":"2024-03-24T00:44:56.729141+0000","last_fullsized":"2024-03-24T00:44:56.729141+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:32:13.875499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.386003+0000","last_change":"2024-03-24T00:44:55.393817+0000","last_active":"2024-03-24T00:44:56.386003+0000","last_peered":"2024-03-24T00:44:56.386003+0000","last_clean":"2024-03-24T00:44:56.386003+0000","last_became_active":"2024-03-24T00:44:55.393489+0000","last_became_peered":"2024-03-24T00:44:55.393489+0000","last_unstale":"2024-03-24T00:44:56.386003+0000","last_undegraded":"2024-03-24T00:44:56.386003+0000","last_fullsized":"2024-03-24T00:44:56.386003+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:26:43.751379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.382568+0000","last_change":"2024-03-24T00:44:54.396876+0000","last_active":"2024-03-24T00:44:56.382568+0000","last_peered":"2024-03-24T00:44:56.382568+0000","last_clean":"2024-03-24T00:44:56.382568+0000","last_became_active":"2024-03-24T00:44:54.396539+0000","last_became_peered":"2024-03-24T00:44:54.396539+0000","last_unstale":"2024-03-24T00:44:56.382568+0000","last_undegraded":"2024-03-24T00:44:56.382568+0000","last_fullsized":"2024-03-24T00:44:56.382568+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:05:24.797069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.776806+0000","last_change":"2024-03-24T00:44:55.408116+0000","last_active":"2024-03-24T00:44:57.776806+0000","last_peered":"2024-03-24T00:44:57.776806+0000","last_clean":"2024-03-24T00:44:57.776806+0000","last_became_active":"2024-03-24T00:44:55.407841+0000","last_became_peered":"2024-03-24T00:44:55.407841+0000","last_unstale":"2024-03-24T00:44:57.776806+0000","last_undegraded":"2024-03-24T00:44:57.776806+0000","last_fullsized":"2024-03-24T00:44:57.776806+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:13:35.495890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.777134+0000","last_change":"2024-03-24T00:44:54.393291+0000","last_active":"2024-03-24T00:44:57.777134+0000","last_peered":"2024-03-24T00:44:57.777134+0000","last_clean":"2024-03-24T00:44:57.777134+0000","last_became_active":"2024-03-24T00:44:54.392987+0000","last_became_peered":"2024-03-24T00:44:54.392987+0000","last_unstale":"2024-03-24T00:44:57.777134+0000","last_undegraded":"2024-03-24T00:44:57.777134+0000","last_fullsized":"2024-03-24T00:44:57.777134+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:13:31.133146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.777087+0000","last_change":"2024-03-24T00:44:55.402701+0000","last_active":"2024-03-24T00:44:57.777087+0000","last_peered":"2024-03-24T00:44:57.777087+0000","last_clean":"2024-03-24T00:44:57.777087+0000","last_became_active":"2024-03-24T00:44:55.402373+0000","last_became_peered":"2024-03-24T00:44:55.402373+0000","last_unstale":"2024-03-24T00:44:57.777087+0000","last_undegraded":"2024-03-24T00:44:57.777087+0000","last_fullsized":"2024-03-24T00:44:57.777087+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:16:19.143414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"88'8","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.777426+0000","last_change":"2024-03-24T00:44:54.394615+0000","last_active":"2024-03-24T00:44:57.777426+0000","last_peered":"2024-03-24T00:44:57.777426+0000","last_clean":"2024-03-24T00:44:57.777426+0000","last_became_active":"2024-03-24T00:44:54.393386+0000","last_became_peered":"2024-03-24T00:44:54.393386+0000","last_unstale":"2024-03-24T00:44:57.777426+0000","last_undegraded":"2024-03-24T00:44:57.777426+0000","last_fullsized":"2024-03-24T00:44:57.777426+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:18:30.093254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.382554+0000","last_change":"2024-03-24T00:44:55.400804+0000","last_active":"2024-03-24T00:44:56.382554+0000","last_peered":"2024-03-24T00:44:56.382554+0000","last_clean":"2024-03-24T00:44:56.382554+0000","last_became_active":"2024-03-24T00:44:55.400543+0000","last_became_peered":"2024-03-24T00:44:55.400543+0000","last_unstale":"2024-03-24T00:44:56.382554+0000","last_undegraded":"2024-03-24T00:44:56.382554+0000","last_fullsized":"2024-03-24T00:44:56.382554+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:11:26.528589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"88'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.406518+0000","last_change":"2024-03-24T00:44:54.394526+0000","last_active":"2024-03-24T00:44:57.406518+0000","last_peered":"2024-03-24T00:44:57.406518+0000","last_clean":"2024-03-24T00:44:57.406518+0000","last_became_active":"2024-03-24T00:44:54.393823+0000","last_became_peered":"2024-03-24T00:44:54.393823+0000","last_unstale":"2024-03-24T00:44:57.406518+0000","last_undegraded":"2024-03-24T00:44:57.406518+0000","last_fullsized":"2024-03-24T00:44:57.406518+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:49:02.779117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.406398+0000","last_change":"2024-03-24T00:44:55.402478+0000","last_active":"2024-03-24T00:44:57.406398+0000","last_peered":"2024-03-24T00:44:57.406398+0000","last_clean":"2024-03-24T00:44:57.406398+0000","last_became_active":"2024-03-24T00:44:55.402160+0000","last_became_peered":"2024-03-24T00:44:55.402160+0000","last_unstale":"2024-03-24T00:44:57.406398+0000","last_undegraded":"2024-03-24T00:44:57.406398+0000","last_fullsized":"2024-03-24T00:44:57.406398+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:18:51.669689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"88'2","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.729268+0000","last_change":"2024-03-24T00:44:54.389428+0000","last_active":"2024-03-24T00:44:56.729268+0000","last_peered":"2024-03-24T00:44:56.729268+0000","last_clean":"2024-03-24T00:44:56.729268+0000","last_became_active":"2024-03-24T00:44:54.389123+0000","last_became_peered":"2024-03-24T00:44:54.389123+0000","last_unstale":"2024-03-24T00:44:56.729268+0000","last_undegraded":"2024-03-24T00:44:56.729268+0000","last_fullsized":"2024-03-24T00:44:56.729268+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:18:27.095669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.386770+0000","last_change":"2024-03-24T00:44:55.396270+0000","last_active":"2024-03-24T00:44:56.386770+0000","last_peered":"2024-03-24T00:44:56.386770+0000","last_clean":"2024-03-24T00:44:56.386770+0000","last_became_active":"2024-03-24T00:44:55.395927+0000","last_became_peered":"2024-03-24T00:44:55.395927+0000","last_unstale":"2024-03-24T00:44:56.386770+0000","last_undegraded":"2024-03-24T00:44:56.386770+0000","last_fullsized":"2024-03-24T00:44:56.386770+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:25:57.625958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"88'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.386861+0000","last_change":"2024-03-24T00:44:54.391906+0000","last_active":"2024-03-24T00:44:56.386861+0000","last_peered":"2024-03-24T00:44:56.386861+0000","last_clean":"2024-03-24T00:44:56.386861+0000","last_became_active":"2024-03-24T00:44:54.391580+0000","last_became_peered":"2024-03-24T00:44:54.391580+0000","last_unstale":"2024-03-24T00:44:56.386861+0000","last_undegraded":"2024-03-24T00:44:56.386861+0000","last_fullsized":"2024-03-24T00:44:56.386861+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:11:55.046776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.382599+0000","last_change":"2024-03-24T00:44:55.403460+0000","last_active":"2024-03-24T00:44:56.382599+0000","last_peered":"2024-03-24T00:44:56.382599+0000","last_clean":"2024-03-24T00:44:56.382599+0000","last_became_active":"2024-03-24T00:44:55.403202+0000","last_became_peered":"2024-03-24T00:44:55.403202+0000","last_unstale":"2024-03-24T00:44:56.382599+0000","last_undegraded":"2024-03-24T00:44:56.382599+0000","last_fullsized":"2024-03-24T00:44:56.382599+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:19:08.478709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"88'2","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.385103+0000","last_change":"2024-03-24T00:44:54.393313+0000","last_active":"2024-03-24T00:44:56.385103+0000","last_peered":"2024-03-24T00:44:56.385103+0000","last_clean":"2024-03-24T00:44:56.385103+0000","last_became_active":"2024-03-24T00:44:54.393033+0000","last_became_peered":"2024-03-24T00:44:54.393033+0000","last_unstale":"2024-03-24T00:44:56.385103+0000","last_undegraded":"2024-03-24T00:44:56.385103+0000","last_fullsized":"2024-03-24T00:44:56.385103+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:45:33.049844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.384983+0000","last_change":"2024-03-24T00:44:55.393095+0000","last_active":"2024-03-24T00:44:56.384983+0000","last_peered":"2024-03-24T00:44:56.384983+0000","last_clean":"2024-03-24T00:44:56.384983+0000","last_became_active":"2024-03-24T00:44:55.392797+0000","last_became_peered":"2024-03-24T00:44:55.392797+0000","last_unstale":"2024-03-24T00:44:56.384983+0000","last_undegraded":"2024-03-24T00:44:56.384983+0000","last_fullsized":"2024-03-24T00:44:56.384983+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:14:45.329609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.776751+0000","last_change":"2024-03-24T00:44:54.391499+0000","last_active":"2024-03-24T00:44:57.776751+0000","last_peered":"2024-03-24T00:44:57.776751+0000","last_clean":"2024-03-24T00:44:57.776751+0000","last_became_active":"2024-03-24T00:44:54.391170+0000","last_became_peered":"2024-03-24T00:44:54.391170+0000","last_unstale":"2024-03-24T00:44:57.776751+0000","last_undegraded":"2024-03-24T00:44:57.776751+0000","last_fullsized":"2024-03-24T00:44:57.776751+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:40:11.977062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.385233+0000","last_change":"2024-03-24T00:44:55.394421+0000","last_active":"2024-03-24T00:44:56.385233+0000","last_peered":"2024-03-24T00:44:56.385233+0000","last_clean":"2024-03-24T00:44:56.385233+0000","last_became_active":"2024-03-24T00:44:55.394112+0000","last_became_peered":"2024-03-24T00:44:55.394112+0000","last_unstale":"2024-03-24T00:44:56.385233+0000","last_undegraded":"2024-03-24T00:44:56.385233+0000","last_fullsized":"2024-03-24T00:44:56.385233+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:54.202078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.385047+0000","last_change":"2024-03-24T00:44:54.399808+0000","last_active":"2024-03-24T00:44:56.385047+0000","last_peered":"2024-03-24T00:44:56.385047+0000","last_clean":"2024-03-24T00:44:56.385047+0000","last_became_active":"2024-03-24T00:44:54.399554+0000","last_became_peered":"2024-03-24T00:44:54.399554+0000","last_unstale":"2024-03-24T00:44:56.385047+0000","last_undegraded":"2024-03-24T00:44:56.385047+0000","last_fullsized":"2024-03-24T00:44:56.385047+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:59:58.396249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.382650+0000","last_change":"2024-03-24T00:44:55.411074+0000","last_active":"2024-03-24T00:44:56.382650+0000","last_peered":"2024-03-24T00:44:56.382650+0000","last_clean":"2024-03-24T00:44:56.382650+0000","last_became_active":"2024-03-24T00:44:55.410826+0000","last_became_peered":"2024-03-24T00:44:55.410826+0000","last_unstale":"2024-03-24T00:44:56.382650+0000","last_undegraded":"2024-03-24T00:44:56.382650+0000","last_fullsized":"2024-03-24T00:44:56.382650+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:12:06.209321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.776608+0000","last_change":"2024-03-24T00:44:54.389975+0000","last_active":"2024-03-24T00:44:57.776608+0000","last_peered":"2024-03-24T00:44:57.776608+0000","last_clean":"2024-03-24T00:44:57.776608+0000","last_became_active":"2024-03-24T00:44:54.389711+0000","last_became_peered":"2024-03-24T00:44:54.389711+0000","last_unstale":"2024-03-24T00:44:57.776608+0000","last_undegraded":"2024-03-24T00:44:57.776608+0000","last_fullsized":"2024-03-24T00:44:57.776608+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:11:45.880543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.406483+0000","last_change":"2024-03-24T00:44:55.396584+0000","last_active":"2024-03-24T00:44:57.406483+0000","last_peered":"2024-03-24T00:44:57.406483+0000","last_clean":"2024-03-24T00:44:57.406483+0000","last_became_active":"2024-03-24T00:44:55.396277+0000","last_became_peered":"2024-03-24T00:44:55.396277+0000","last_unstale":"2024-03-24T00:44:57.406483+0000","last_undegraded":"2024-03-24T00:44:57.406483+0000","last_fullsized":"2024-03-24T00:44:57.406483+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:49:36.416272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"88'2","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.729331+0000","last_change":"2024-03-24T00:44:54.387054+0000","last_active":"2024-03-24T00:44:56.729331+0000","last_peered":"2024-03-24T00:44:56.729331+0000","last_clean":"2024-03-24T00:44:56.729331+0000","last_became_active":"2024-03-24T00:44:54.386785+0000","last_became_peered":"2024-03-24T00:44:54.386785+0000","last_unstale":"2024-03-24T00:44:56.729331+0000","last_undegraded":"2024-03-24T00:44:56.729331+0000","last_fullsized":"2024-03-24T00:44:56.729331+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:03:15.796915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.382948+0000","last_change":"2024-03-24T00:44:55.403999+0000","last_active":"2024-03-24T00:44:56.382948+0000","last_peered":"2024-03-24T00:44:56.382948+0000","last_clean":"2024-03-24T00:44:56.382948+0000","last_became_active":"2024-03-24T00:44:55.401878+0000","last_became_peered":"2024-03-24T00:44:55.401878+0000","last_unstale":"2024-03-24T00:44:56.382948+0000","last_undegraded":"2024-03-24T00:44:56.382948+0000","last_fullsized":"2024-03-24T00:44:56.382948+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:32:37.241577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"89'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.386780+0000","last_change":"2024-03-24T00:44:54.391792+0000","last_active":"2024-03-24T00:44:56.386780+0000","last_peered":"2024-03-24T00:44:56.386780+0000","last_clean":"2024-03-24T00:44:56.386780+0000","last_became_active":"2024-03-24T00:44:54.390529+0000","last_became_peered":"2024-03-24T00:44:54.390529+0000","last_unstale":"2024-03-24T00:44:56.386780+0000","last_undegraded":"2024-03-24T00:44:56.386780+0000","last_fullsized":"2024-03-24T00:44:56.386780+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:37:50.754605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.382941+0000","last_change":"2024-03-24T00:44:55.401482+0000","last_active":"2024-03-24T00:44:56.382941+0000","last_peered":"2024-03-24T00:44:56.382941+0000","last_clean":"2024-03-24T00:44:56.382941+0000","last_became_active":"2024-03-24T00:44:55.401223+0000","last_became_peered":"2024-03-24T00:44:55.401223+0000","last_unstale":"2024-03-24T00:44:56.382941+0000","last_undegraded":"2024-03-24T00:44:56.382941+0000","last_fullsized":"2024-03-24T00:44:56.382941+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:39:32.628036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.386915+0000","last_change":"2024-03-24T00:44:54.388742+0000","last_active":"2024-03-24T00:44:56.386915+0000","last_peered":"2024-03-24T00:44:56.386915+0000","last_clean":"2024-03-24T00:44:56.386915+0000","last_became_active":"2024-03-24T00:44:54.388466+0000","last_became_peered":"2024-03-24T00:44:54.388466+0000","last_unstale":"2024-03-24T00:44:56.386915+0000","last_undegraded":"2024-03-24T00:44:56.386915+0000","last_fullsized":"2024-03-24T00:44:56.386915+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:41:25.500084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.385509+0000","last_change":"2024-03-24T00:44:55.388203+0000","last_active":"2024-03-24T00:44:56.385509+0000","last_peered":"2024-03-24T00:44:56.385509+0000","last_clean":"2024-03-24T00:44:56.385509+0000","last_became_active":"2024-03-24T00:44:55.387932+0000","last_became_peered":"2024-03-24T00:44:55.387932+0000","last_unstale":"2024-03-24T00:44:56.385509+0000","last_undegraded":"2024-03-24T00:44:56.385509+0000","last_fullsized":"2024-03-24T00:44:56.385509+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:39:23.097646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.406512+0000","last_change":"2024-03-24T00:44:54.391056+0000","last_active":"2024-03-24T00:44:57.406512+0000","last_peered":"2024-03-24T00:44:57.406512+0000","last_clean":"2024-03-24T00:44:57.406512+0000","last_became_active":"2024-03-24T00:44:54.390695+0000","last_became_peered":"2024-03-24T00:44:54.390695+0000","last_unstale":"2024-03-24T00:44:57.406512+0000","last_undegraded":"2024-03-24T00:44:57.406512+0000","last_fullsized":"2024-03-24T00:44:57.406512+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:57.847221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.776673+0000","last_change":"2024-03-24T00:44:55.407491+0000","last_active":"2024-03-24T00:44:57.776673+0000","last_peered":"2024-03-24T00:44:57.776673+0000","last_clean":"2024-03-24T00:44:57.776673+0000","last_became_active":"2024-03-24T00:44:55.407221+0000","last_became_peered":"2024-03-24T00:44:55.407221+0000","last_unstale":"2024-03-24T00:44:57.776673+0000","last_undegraded":"2024-03-24T00:44:57.776673+0000","last_fullsized":"2024-03-24T00:44:57.776673+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:15:59.692129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.382768+0000","last_change":"2024-03-24T00:44:54.397277+0000","last_active":"2024-03-24T00:44:56.382768+0000","last_peered":"2024-03-24T00:44:56.382768+0000","last_clean":"2024-03-24T00:44:56.382768+0000","last_became_active":"2024-03-24T00:44:54.397008+0000","last_became_peered":"2024-03-24T00:44:54.397008+0000","last_unstale":"2024-03-24T00:44:56.382768+0000","last_undegraded":"2024-03-24T00:44:56.382768+0000","last_fullsized":"2024-03-24T00:44:56.382768+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:53.199895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.514142+0000","last_change":"2024-03-24T00:44:55.389272+0000","last_active":"2024-03-24T00:44:57.514142+0000","last_peered":"2024-03-24T00:44:57.514142+0000","last_clean":"2024-03-24T00:44:57.514142+0000","last_became_active":"2024-03-24T00:44:55.389005+0000","last_became_peered":"2024-03-24T00:44:55.389005+0000","last_unstale":"2024-03-24T00:44:57.514142+0000","last_undegraded":"2024-03-24T00:44:57.514142+0000","last_fullsized":"2024-03-24T00:44:57.514142+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:38:36.247055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.385038+0000","last_change":"2024-03-24T00:44:54.392645+0000","last_active":"2024-03-24T00:44:56.385038+0000","last_peered":"2024-03-24T00:44:56.385038+0000","last_clean":"2024-03-24T00:44:56.385038+0000","last_became_active":"2024-03-24T00:44:54.392370+0000","last_became_peered":"2024-03-24T00:44:54.392370+0000","last_unstale":"2024-03-24T00:44:56.385038+0000","last_undegraded":"2024-03-24T00:44:56.385038+0000","last_fullsized":"2024-03-24T00:44:56.385038+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:28:04.465514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.406218+0000","last_change":"2024-03-24T00:44:55.399019+0000","last_active":"2024-03-24T00:44:57.406218+0000","last_peered":"2024-03-24T00:44:57.406218+0000","last_clean":"2024-03-24T00:44:57.406218+0000","last_became_active":"2024-03-24T00:44:55.398194+0000","last_became_peered":"2024-03-24T00:44:55.398194+0000","last_unstale":"2024-03-24T00:44:57.406218+0000","last_undegraded":"2024-03-24T00:44:57.406218+0000","last_fullsized":"2024-03-24T00:44:57.406218+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:13:32.135329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.406348+0000","last_change":"2024-03-24T00:44:54.391852+0000","last_active":"2024-03-24T00:44:57.406348+0000","last_peered":"2024-03-24T00:44:57.406348+0000","last_clean":"2024-03-24T00:44:57.406348+0000","last_became_active":"2024-03-24T00:44:54.390810+0000","last_became_peered":"2024-03-24T00:44:54.390810+0000","last_unstale":"2024-03-24T00:44:57.406348+0000","last_undegraded":"2024-03-24T00:44:57.406348+0000","last_fullsized":"2024-03-24T00:44:57.406348+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:11:06.419292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.382779+0000","last_change":"2024-03-24T00:44:55.405256+0000","last_active":"2024-03-24T00:44:56.382779+0000","last_peered":"2024-03-24T00:44:56.382779+0000","last_clean":"2024-03-24T00:44:56.382779+0000","last_became_active":"2024-03-24T00:44:55.403655+0000","last_became_peered":"2024-03-24T00:44:55.403655+0000","last_unstale":"2024-03-24T00:44:56.382779+0000","last_undegraded":"2024-03-24T00:44:56.382779+0000","last_fullsized":"2024-03-24T00:44:56.382779+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:21:10.200232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.729154+0000","last_change":"2024-03-24T00:44:54.390612+0000","last_active":"2024-03-24T00:44:56.729154+0000","last_peered":"2024-03-24T00:44:56.729154+0000","last_clean":"2024-03-24T00:44:56.729154+0000","last_became_active":"2024-03-24T00:44:54.390348+0000","last_became_peered":"2024-03-24T00:44:54.390348+0000","last_unstale":"2024-03-24T00:44:56.729154+0000","last_undegraded":"2024-03-24T00:44:56.729154+0000","last_fullsized":"2024-03-24T00:44:56.729154+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:09:59.913191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.729417+0000","last_change":"2024-03-24T00:44:55.398320+0000","last_active":"2024-03-24T00:44:56.729417+0000","last_peered":"2024-03-24T00:44:56.729417+0000","last_clean":"2024-03-24T00:44:56.729417+0000","last_became_active":"2024-03-24T00:44:55.398040+0000","last_became_peered":"2024-03-24T00:44:55.398040+0000","last_unstale":"2024-03-24T00:44:56.729417+0000","last_undegraded":"2024-03-24T00:44:56.729417+0000","last_fullsized":"2024-03-24T00:44:56.729417+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:54.371295+0000","last_clean_scrub_stamp":"2024-03-24T00:44:54.371295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:54:55.383079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:57.777788+0000","last_change":"2024-03-24T00:44:54.395548+0000","last_active":"2024-03-24T00:44:57.777788+0000","last_peered":"2024-03-24T00:44:57.777788+0000","last_clean":"2024-03-24T00:44:57.777788+0000","last_became_active":"2024-03-24T00:44:54.395266+0000","last_became_peered":"2024-03-24T00:44:54.395266+0000","last_unstale":"2024-03-24T00:44:57.777788+0000","last_undegraded":"2024-03-24T00:44:57.777788+0000","last_fullsized":"2024-03-24T00:44:57.777788+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:44:53.369112+0000","last_clean_scrub_stamp":"2024-03-24T00:44:53.369112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:16:06.453263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":299,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T00:44:56.729348+0000","last_change":"2024-03-24T00:38:40.953871+0000","last_active":"2024-03-24T00:44:56.729348+0000","last_peered":"2024-03-24T00:44:56.729348+0000","last_clean":"2024-03-24T00:44:56.729348+0000","last_became_active":"2024-03-24T00:38:40.952926+0000","last_became_peered":"2024-03-24T00:38:40.952926+0000","last_unstale":"2024-03-24T00:44:56.729348+0000","last_undegraded":"2024-03-24T00:44:56.729348+0000","last_fullsized":"2024-03-24T00:44:56.729348+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_clean_scrub_stamp":"2024-03-24T00:38:39.722194+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:05.397257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574929,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32844,"kb_used_data":1660,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338996,"statfs":{"total":96636764160,"available":96603131904,"internally_reserved":0,"allocated":1699840,"data_stored":1095629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574928,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32276,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339564,"statfs":{"total":96636764160,"available":96603713536,"internally_reserved":0,"allocated":1118208,"data_stored":507421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574928,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32256,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339584,"statfs":{"total":96636764160,"available":96603734016,"internally_reserved":0,"allocated":1097728,"data_stored":505205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574927,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32264,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339576,"statfs":{"total":96636764160,"available":96603725824,"internally_reserved":0,"allocated":1105920,"data_stored":505365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574927,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32256,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339584,"statfs":{"total":96636764160,"available":96603734016,"internally_reserved":0,"allocated":1097728,"data_stored":505205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574928,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32256,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339584,"statfs":{"total":96636764160,"available":96603734016,"internally_reserved":0,"allocated":1097728,"data_stored":505205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574928,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32260,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339580,"statfs":{"total":96636764160,"available":96603729920,"internally_reserved":0,"allocated":1101824,"data_stored":505275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574927,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32856,"kb_used_data":1676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338984,"statfs":{"total":96636764160,"available":96603119616,"internally_reserved":0,"allocated":1716224,"data_stored":1097891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:45:03.473 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-24T00:45:03.474 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-24T00:45:03.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.584+0000 7fba34da9640 1 Processor -- start 2024-03-24T00:45:03.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.584+0000 7fba34da9640 1 -- start start 2024-03-24T00:45:03.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.585+0000 7fba34da9640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fba300cf2d0 0x7fba30168ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:03.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.585+0000 7fba34da9640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba300d2af0 con 0x7fba300b9b50 2024-03-24T00:45:03.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.585+0000 7fba34da9640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba300d2dd0 con 0x7fba3015e170 2024-03-24T00:45:03.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.585+0000 7fba34da9640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba300d28d0 con 0x7fba300cf2d0 2024-03-24T00:45:03.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.585+0000 7fba2dd9b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fba300b9b50 0x7fba300cebc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46144/0 (socket says 172.21.15.67:46144) 2024-03-24T00:45:03.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.585+0000 7fba2dd9b640 1 -- 172.21.15.67:0/1241077655 learned_addr learned my addr 172.21.15.67:0/1241077655 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:45:03.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.586+0000 7fba2ed9d640 1 --2- 172.21.15.67:0/1241077655 >> v2:172.21.15.72:3300/0 conn(0x7fba300cf2d0 0x7fba30168ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:03.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.586+0000 7fba2d59a640 1 -- 172.21.15.67:0/1241077655 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1139292564 0 0) 0x7fba300d2af0 con 0x7fba300b9b50 2024-03-24T00:45:03.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.586+0000 7fba2d59a640 1 -- 172.21.15.67:0/1241077655 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba1c003520 con 0x7fba300b9b50 2024-03-24T00:45:03.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.586+0000 7fba2ed9d640 1 -- 172.21.15.67:0/1241077655 >> v1:172.21.15.72:6789/0 conn(0x7fba3015e170 legacy=0x7fba300d21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:03.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.586+0000 7fba2ed9d640 1 -- 172.21.15.67:0/1241077655 >> v1:172.21.15.67:6789/0 conn(0x7fba300b9b50 legacy=0x7fba300cebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:03.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.586+0000 7fba2d59a640 1 -- 172.21.15.67:0/1241077655 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2109216763 0 0) 0x7fba1c003520 con 0x7fba300b9b50 2024-03-24T00:45:03.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.586+0000 7fba2ed9d640 1 -- 172.21.15.67:0/1241077655 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba300d30b0 con 0x7fba300cf2d0 2024-03-24T00:45:03.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.587+0000 7fba2ed9d640 1 --2- 172.21.15.67:0/1241077655 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba300cf2d0 0x7fba30168ba0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fba200097a0 tx=0x7fba2002b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb088955eec2887e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:03.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.587+0000 7fba2d59a640 1 -- 172.21.15.67:0/1241077655 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba20038070 con 0x7fba300cf2d0 2024-03-24T00:45:03.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.587+0000 7fba2d59a640 1 -- 172.21.15.67:0/1241077655 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba20031500 con 0x7fba300cf2d0 2024-03-24T00:45:03.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.587+0000 7fba2d59a640 1 -- 172.21.15.67:0/1241077655 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba18001670 con 0x7fba300cf2d0 2024-03-24T00:45:03.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.588+0000 7fba34da9640 1 -- 172.21.15.67:0/1241077655 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba300cf2d0 msgr2=0x7fba30168ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:03.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.588+0000 7fba34da9640 1 --2- 172.21.15.67:0/1241077655 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba300cf2d0 0x7fba30168ba0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fba200097a0 tx=0x7fba2002b820 comp rx=0 tx=0).stop 2024-03-24T00:45:03.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.588+0000 7fba34da9640 1 -- 172.21.15.67:0/1241077655 shutdown_connections 2024-03-24T00:45:03.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.588+0000 7fba34da9640 1 --2- 172.21.15.67:0/1241077655 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba300cf2d0 0x7fba30168ba0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.588+0000 7fba34da9640 1 -- 172.21.15.67:0/1241077655 >> 172.21.15.67:0/1241077655 conn(0x7fba3006c810 msgr2=0x7fba3006cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:03.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.588+0000 7fba34da9640 1 -- 172.21.15.67:0/1241077655 shutdown_connections 2024-03-24T00:45:03.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.588+0000 7fba34da9640 1 -- 172.21.15.67:0/1241077655 wait complete. 2024-03-24T00:45:03.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.589+0000 7fba34da9640 1 Processor -- start 2024-03-24T00:45:03.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.589+0000 7fba34da9640 1 -- start start 2024-03-24T00:45:03.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.590+0000 7fba34da9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba300b9b50 0x7fba301532f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:03.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.590+0000 7fba34da9640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fba300cf2d0 0x7fba300cc740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:03.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.590+0000 7fba34da9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba3015e170 0x7fba300caec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:03.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.590+0000 7fba34da9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba300b7f30 con 0x7fba300b9b50 2024-03-24T00:45:03.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.590+0000 7fba34da9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba300d3670 con 0x7fba3015e170 2024-03-24T00:45:03.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.590+0000 7fba34da9640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba300b80b0 con 0x7fba300cf2d0 2024-03-24T00:45:03.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.590+0000 7fba2e59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba300b9b50 0x7fba301532f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:03.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.590+0000 7fba2e59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba300b9b50 0x7fba301532f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32856/0 (socket says 172.21.15.67:32856) 2024-03-24T00:45:03.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.590+0000 7fba2e59c640 1 -- 172.21.15.67:0/1759386264 learned_addr learned my addr 172.21.15.67:0/1759386264 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:03.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.590+0000 7fba2dd9b640 1 --2- 172.21.15.67:0/1759386264 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fba300cf2d0 0x7fba300cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:03.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.590+0000 7fba2ed9d640 1 --2- 172.21.15.67:0/1759386264 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba3015e170 0x7fba300caec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:03.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.590+0000 7fba2e59c640 1 -- 172.21.15.67:0/1759386264 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fba300cf2d0 msgr2=0x7fba300cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:03.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.590+0000 7fba2e59c640 1 --2- 172.21.15.67:0/1759386264 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fba300cf2d0 0x7fba300cc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.590+0000 7fba2e59c640 1 -- 172.21.15.67:0/1759386264 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba3015e170 msgr2=0x7fba300caec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:03.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.590+0000 7fba2e59c640 1 --2- 172.21.15.67:0/1759386264 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba3015e170 0x7fba300caec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.590+0000 7fba2e59c640 1 -- 172.21.15.67:0/1759386264 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba300cb400 con 0x7fba300b9b50 2024-03-24T00:45:03.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.590+0000 7fba2e59c640 1 --2- 172.21.15.67:0/1759386264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba300b9b50 0x7fba301532f0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fba2800c760 tx=0x7fba2800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:03.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.590+0000 7fba26ffd640 1 -- 172.21.15.67:0/1759386264 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba28003b70 con 0x7fba300b9b50 2024-03-24T00:45:03.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.590+0000 7fba26ffd640 1 -- 172.21.15.67:0/1759386264 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba28003d30 con 0x7fba300b9b50 2024-03-24T00:45:03.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.591+0000 7fba34da9640 1 -- 172.21.15.67:0/1759386264 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba300cb6a0 con 0x7fba300b9b50 2024-03-24T00:45:03.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.591+0000 7fba26ffd640 1 -- 172.21.15.67:0/1759386264 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2800aa30 con 0x7fba300b9b50 2024-03-24T00:45:03.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.591+0000 7fba34da9640 1 -- 172.21.15.67:0/1759386264 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba300cbc60 con 0x7fba300b9b50 2024-03-24T00:45:03.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.591+0000 7fba34da9640 1 -- 172.21.15.67:0/1759386264 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba3004f400 con 0x7fba300b9b50 2024-03-24T00:45:03.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.592+0000 7fba26ffd640 1 -- 172.21.15.67:0/1759386264 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fba2800abd0 con 0x7fba300b9b50 2024-03-24T00:45:03.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.592+0000 7fba26ffd640 1 --2- 172.21.15.67:0/1759386264 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fba1006bbd0 0x7fba1006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:03.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.593+0000 7fba26ffd640 1 -- 172.21.15.67:0/1759386264 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fba2808aac0 con 0x7fba300b9b50 2024-03-24T00:45:03.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.593+0000 7fba2dd9b640 1 --2- 172.21.15.67:0/1759386264 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fba1006bbd0 0x7fba1006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:03.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.593+0000 7fba2dd9b640 1 --2- 172.21.15.67:0/1759386264 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fba1006bbd0 0x7fba1006e090 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fba3015d310 tx=0x7fba18009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:03.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.601+0000 7fba26ffd640 1 -- 172.21.15.67:0/1759386264 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fba28059420 con 0x7fba300b9b50 2024-03-24T00:45:03.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.852+0000 7fba34da9640 1 -- 172.21.15.67:0/1759386264 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fba3006b790 con 0x7fba300b9b50 2024-03-24T00:45:03.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.855+0000 7fba26ffd640 1 -- 172.21.15.67:0/1759386264 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) v1 ==== 190+0+147 (secure 0 0 0) 0x7fba28058d20 con 0x7fba300b9b50 2024-03-24T00:45:03.855 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-03-24T00:45:03.855 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQDcdf9lY4nvJRAA6HWOZDWA+SSvJVmyjG+51g== 2024-03-24T00:45:03.855 INFO:teuthology.orchestra.run.smithi067.stdout: caps mds = "allow" 2024-03-24T00:45:03.855 INFO:teuthology.orchestra.run.smithi067.stdout: caps mon = "allow r" 2024-03-24T00:45:03.855 INFO:teuthology.orchestra.run.smithi067.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-24T00:45:03.855 INFO:teuthology.orchestra.run.smithi067.stderr:updated caps for client.0 2024-03-24T00:45:03.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.857+0000 7fba34da9640 1 -- 172.21.15.67:0/1759386264 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fba1006bbd0 msgr2=0x7fba1006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:03.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.857+0000 7fba34da9640 1 --2- 172.21.15.67:0/1759386264 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fba1006bbd0 0x7fba1006e090 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fba3015d310 tx=0x7fba18009040 comp rx=0 tx=0).stop 2024-03-24T00:45:03.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.857+0000 7fba34da9640 1 -- 172.21.15.67:0/1759386264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba300b9b50 msgr2=0x7fba301532f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:03.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.857+0000 7fba34da9640 1 --2- 172.21.15.67:0/1759386264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba300b9b50 0x7fba301532f0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fba2800c760 tx=0x7fba2800cc20 comp rx=0 tx=0).stop 2024-03-24T00:45:03.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.857+0000 7fba34da9640 1 -- 172.21.15.67:0/1759386264 shutdown_connections 2024-03-24T00:45:03.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.857+0000 7fba34da9640 1 --2- 172.21.15.67:0/1759386264 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fba1006bbd0 0x7fba1006e090 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.857+0000 7fba34da9640 1 --2- 172.21.15.67:0/1759386264 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba3015e170 0x7fba300caec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.857+0000 7fba34da9640 1 --2- 172.21.15.67:0/1759386264 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fba300cf2d0 0x7fba300cc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.857+0000 7fba34da9640 1 --2- 172.21.15.67:0/1759386264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba300b9b50 0x7fba301532f0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.857+0000 7fba34da9640 1 -- 172.21.15.67:0/1759386264 >> 172.21.15.67:0/1759386264 conn(0x7fba3006c810 msgr2=0x7fba300ce6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:03.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.857+0000 7fba34da9640 1 -- 172.21.15.67:0/1759386264 shutdown_connections 2024-03-24T00:45:03.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.857+0000 7fba34da9640 1 -- 172.21.15.67:0/1759386264 wait complete. 2024-03-24T00:45:03.868 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:45:03.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.969+0000 7efe5427e640 1 Processor -- start 2024-03-24T00:45:03.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.969+0000 7efe5427e640 1 -- start start 2024-03-24T00:45:03.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.970+0000 7efe5427e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efe4c15dec0 0x7efe4c0ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:03.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.970+0000 7efe5427e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efe4c15e2a0 0x7efe4c162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:03.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.970+0000 7efe5427e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7efe4c0d07e0 0x7efe4c0d0c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:03.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.970+0000 7efe5427e640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe4c0d1190 con 0x7efe4c15dec0 2024-03-24T00:45:03.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.970+0000 7efe5427e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe4c0cee30 con 0x7efe4c0d07e0 2024-03-24T00:45:03.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.970+0000 7efe5427e640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe4c0d2120 con 0x7efe4c15e2a0 2024-03-24T00:45:03.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.970+0000 7efe5201a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efe4c15dec0 0x7efe4c0ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:03.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.970+0000 7efe5201a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efe4c15dec0 0x7efe4c0ce630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32872/0 (socket says 172.21.15.67:32872) 2024-03-24T00:45:03.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.970+0000 7efe5201a640 1 -- 172.21.15.67:0/858104721 learned_addr learned my addr 172.21.15.67:0/858104721 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:03.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.970+0000 7efe5281b640 1 --2- 172.21.15.67:0/858104721 >> v2:172.21.15.72:3300/0 conn(0x7efe4c0d07e0 0x7efe4c0d0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:03.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.970+0000 7efe51819640 1 --2- 172.21.15.67:0/858104721 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efe4c15e2a0 0x7efe4c162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:03.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.970+0000 7efe5201a640 1 -- 172.21.15.67:0/858104721 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efe4c15e2a0 msgr2=0x7efe4c162630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:03.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.970+0000 7efe5201a640 1 --2- 172.21.15.67:0/858104721 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efe4c15e2a0 0x7efe4c162630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.970+0000 7efe5201a640 1 -- 172.21.15.67:0/858104721 >> v2:172.21.15.72:3300/0 conn(0x7efe4c0d07e0 msgr2=0x7efe4c0d0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:03.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.970+0000 7efe5201a640 1 --2- 172.21.15.67:0/858104721 >> v2:172.21.15.72:3300/0 conn(0x7efe4c0d07e0 0x7efe4c0d0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.970+0000 7efe5201a640 1 -- 172.21.15.67:0/858104721 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe4c0cf640 con 0x7efe4c15dec0 2024-03-24T00:45:03.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.970+0000 7efe5281b640 1 --2- 172.21.15.67:0/858104721 >> v2:172.21.15.72:3300/0 conn(0x7efe4c0d07e0 0x7efe4c0d0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:45:03.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.970+0000 7efe51819640 1 --2- 172.21.15.67:0/858104721 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efe4c15e2a0 0x7efe4c162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:45:03.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.971+0000 7efe5201a640 1 --2- 172.21.15.67:0/858104721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe4c15dec0 0x7efe4c0ce630 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7efe4c0d1a50 tx=0x7efe3c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dadd81e49d9dd7a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:03.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.971+0000 7efe51018640 1 -- 172.21.15.67:0/858104721 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe3c02e070 con 0x7efe4c15dec0 2024-03-24T00:45:03.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.971+0000 7efe51018640 1 -- 172.21.15.67:0/858104721 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe3c037070 con 0x7efe4c15dec0 2024-03-24T00:45:03.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.971+0000 7efe51018640 1 -- 172.21.15.67:0/858104721 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe3c031a00 con 0x7efe4c15dec0 2024-03-24T00:45:03.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.972+0000 7efe5427e640 1 -- 172.21.15.67:0/858104721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe4c15dec0 msgr2=0x7efe4c0ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:03.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.972+0000 7efe5427e640 1 --2- 172.21.15.67:0/858104721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe4c15dec0 0x7efe4c0ce630 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7efe4c0d1a50 tx=0x7efe3c02b880 comp rx=0 tx=0).stop 2024-03-24T00:45:03.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.972+0000 7efe5427e640 1 -- 172.21.15.67:0/858104721 shutdown_connections 2024-03-24T00:45:03.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.972+0000 7efe5427e640 1 --2- 172.21.15.67:0/858104721 >> v2:172.21.15.72:3300/0 conn(0x7efe4c0d07e0 0x7efe4c0d0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.972+0000 7efe5427e640 1 --2- 172.21.15.67:0/858104721 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efe4c15e2a0 0x7efe4c162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.972+0000 7efe5427e640 1 --2- 172.21.15.67:0/858104721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe4c15dec0 0x7efe4c0ce630 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.972+0000 7efe5427e640 1 -- 172.21.15.67:0/858104721 >> 172.21.15.67:0/858104721 conn(0x7efe4c06c810 msgr2=0x7efe4c06cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:03.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.972+0000 7efe5427e640 1 -- 172.21.15.67:0/858104721 shutdown_connections 2024-03-24T00:45:03.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.972+0000 7efe5427e640 1 -- 172.21.15.67:0/858104721 wait complete. 2024-03-24T00:45:03.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.973+0000 7efe5427e640 1 Processor -- start 2024-03-24T00:45:03.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.973+0000 7efe5427e640 1 -- start start 2024-03-24T00:45:03.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.974+0000 7efe5427e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe4c0d07e0 0x7efe4c0cda70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:03.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.974+0000 7efe5427e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efe4c15dec0 0x7efe4c14e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:03.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.974+0000 7efe5427e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efe4c15e2a0 0x7efe4c0aa870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:03.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.974+0000 7efe5427e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe4c0b5850 con 0x7efe4c0d07e0 2024-03-24T00:45:03.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.974+0000 7efe5427e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe4c0cf360 con 0x7efe4c15e2a0 2024-03-24T00:45:03.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.974+0000 7efe5427e640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe4c0b59d0 con 0x7efe4c15dec0 2024-03-24T00:45:03.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.974+0000 7efe5201a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe4c0d07e0 0x7efe4c0cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:03.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.974+0000 7efe5201a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe4c0d07e0 0x7efe4c0cda70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32884/0 (socket says 172.21.15.67:32884) 2024-03-24T00:45:03.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.974+0000 7efe5201a640 1 -- 172.21.15.67:0/945648892 learned_addr learned my addr 172.21.15.67:0/945648892 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:03.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.974+0000 7efe5281b640 1 --2- 172.21.15.67:0/945648892 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efe4c15e2a0 0x7efe4c0aa870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:03.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.974+0000 7efe51819640 1 --2- 172.21.15.67:0/945648892 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efe4c15dec0 0x7efe4c14e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:03.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.974+0000 7efe5201a640 1 -- 172.21.15.67:0/945648892 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efe4c15dec0 msgr2=0x7efe4c14e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:03.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.974+0000 7efe5201a640 1 --2- 172.21.15.67:0/945648892 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efe4c15dec0 0x7efe4c14e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.974+0000 7efe5201a640 1 -- 172.21.15.67:0/945648892 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efe4c15e2a0 msgr2=0x7efe4c0aa870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:03.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.974+0000 7efe5201a640 1 --2- 172.21.15.67:0/945648892 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efe4c15e2a0 0x7efe4c0aa870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.974+0000 7efe5201a640 1 -- 172.21.15.67:0/945648892 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe4c0cc9d0 con 0x7efe4c0d07e0 2024-03-24T00:45:03.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.974+0000 7efe5201a640 1 --2- 172.21.15.67:0/945648892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe4c0d07e0 0x7efe4c0cda70 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7efe3c02b8c0 tx=0x7efe3c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:03.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.975+0000 7efe42ffd640 1 -- 172.21.15.67:0/945648892 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe3c032880 con 0x7efe4c0d07e0 2024-03-24T00:45:03.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.975+0000 7efe42ffd640 1 -- 172.21.15.67:0/945648892 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe3c032a20 con 0x7efe4c0d07e0 2024-03-24T00:45:03.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.975+0000 7efe5427e640 1 -- 172.21.15.67:0/945648892 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe4c0cccb0 con 0x7efe4c0d07e0 2024-03-24T00:45:03.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.975+0000 7efe42ffd640 1 -- 172.21.15.67:0/945648892 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe3c032e10 con 0x7efe4c0d07e0 2024-03-24T00:45:03.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.975+0000 7efe5427e640 1 -- 172.21.15.67:0/945648892 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe4c0cd270 con 0x7efe4c0d07e0 2024-03-24T00:45:03.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.976+0000 7efe5427e640 1 -- 172.21.15.67:0/945648892 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe4c04f400 con 0x7efe4c0d07e0 2024-03-24T00:45:03.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.976+0000 7efe42ffd640 1 -- 172.21.15.67:0/945648892 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7efe3c0453f0 con 0x7efe4c0d07e0 2024-03-24T00:45:03.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.976+0000 7efe42ffd640 1 --2- 172.21.15.67:0/945648892 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7efe3006bbd0 0x7efe3006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:03.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.977+0000 7efe42ffd640 1 -- 172.21.15.67:0/945648892 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6101+0+0 (secure 0 0 0) 0x7efe3c0acb80 con 0x7efe4c0d07e0 2024-03-24T00:45:03.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.977+0000 7efe51819640 1 --2- 172.21.15.67:0/945648892 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7efe3006bbd0 0x7efe3006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:03.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.977+0000 7efe51819640 1 --2- 172.21.15.67:0/945648892 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7efe3006bbd0 0x7efe3006e090 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7efe4c0ad4e0 tx=0x7efe48008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:03.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:03.981+0000 7efe42ffd640 1 -- 172.21.15.67:0/945648892 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7efe3c040070 con 0x7efe4c0d07e0 2024-03-24T00:45:04.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.199+0000 7efe5427e640 1 -- 172.21.15.67:0/945648892 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efe4c0d1aa0 con 0x7efe4c0d07e0 2024-03-24T00:45:04.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.199+0000 7efe42ffd640 1 -- 172.21.15.67:0/945648892 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+4979 (secure 0 0 0) 0x7efe4c0cf640 con 0x7efe4c0d07e0 2024-03-24T00:45:04.199 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:45:04.200 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":54,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6359,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2664460392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2664460392},{"type":"v1","addr":"172.21.15.67:6835","nonce":2664460392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6368,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/3323873506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3323873506},{"type":"v1","addr":"172.21.15.72:6837","nonce":3323873506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":7150,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/2744016217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2744016217},{"type":"v1","addr":"172.21.15.72:6836","nonce":2744016217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:44:54.749656+0000","modified":"2024-03-24T00:44:56.822686+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7617},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7617":{"gid":7617,"name":"c","rank":0,"incarnation":52,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6833/3021622863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3021622863},{"type":"v1","addr":"172.21.15.67:6833","nonce":3021622863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7617,"qdb_cluster":[7617]},"id":6}]} 2024-03-24T00:45:04.200 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 54 2024-03-24T00:45:04.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.201+0000 7efe5427e640 1 -- 172.21.15.67:0/945648892 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7efe3006bbd0 msgr2=0x7efe3006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:04.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.201+0000 7efe5427e640 1 --2- 172.21.15.67:0/945648892 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7efe3006bbd0 0x7efe3006e090 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7efe4c0ad4e0 tx=0x7efe48008040 comp rx=0 tx=0).stop 2024-03-24T00:45:04.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.201+0000 7efe5427e640 1 -- 172.21.15.67:0/945648892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe4c0d07e0 msgr2=0x7efe4c0cda70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:04.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.201+0000 7efe5427e640 1 --2- 172.21.15.67:0/945648892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe4c0d07e0 0x7efe4c0cda70 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7efe3c02b8c0 tx=0x7efe3c0046c0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.201+0000 7efe5427e640 1 -- 172.21.15.67:0/945648892 shutdown_connections 2024-03-24T00:45:04.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.201+0000 7efe5427e640 1 --2- 172.21.15.67:0/945648892 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7efe3006bbd0 0x7efe3006e090 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.201+0000 7efe5427e640 1 --2- 172.21.15.67:0/945648892 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efe4c15e2a0 0x7efe4c0aa870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.201+0000 7efe5427e640 1 --2- 172.21.15.67:0/945648892 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7efe4c15dec0 0x7efe4c14e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.201+0000 7efe5427e640 1 --2- 172.21.15.67:0/945648892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe4c0d07e0 0x7efe4c0cda70 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.201+0000 7efe5427e640 1 -- 172.21.15.67:0/945648892 >> 172.21.15.67:0/945648892 conn(0x7efe4c06c810 msgr2=0x7efe4c0b5130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:04.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.201+0000 7efe5427e640 1 -- 172.21.15.67:0/945648892 shutdown_connections 2024-03-24T00:45:04.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.201+0000 7efe5427e640 1 -- 172.21.15.67:0/945648892 wait complete. 2024-03-24T00:45:04.614 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 54, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:44:54.749656+0000', 'modified': '2024-03-24T00:44:56.822686+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7617}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7617': {'gid': 7617, 'name': 'c', 'rank': 0, 'incarnation': 52, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.67:6833/3021622863', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6832', 'nonce': 3021622863}, {'type': 'v1', 'addr': '172.21.15.67:6833', 'nonce': 3021622863}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7617, 'qdb_cluster': [7617]} 2024-03-24T00:45:04.614 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-24T00:45:04.615 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-24T00:45:04.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.717+0000 7ff61ef45640 1 Processor -- start 2024-03-24T00:45:04.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.717+0000 7ff61ef45640 1 -- start start 2024-03-24T00:45:04.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.718+0000 7ff61ef45640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff6180d32d0 0x7ff6180d3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.718+0000 7ff61ef45640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6180b9cc0 con 0x7ff6180b6dc0 2024-03-24T00:45:04.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.718+0000 7ff61ef45640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6180d2230 con 0x7ff61815e0f0 2024-03-24T00:45:04.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.718+0000 7ff61ef45640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6180d2510 con 0x7ff6180d32d0 2024-03-24T00:45:04.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.718+0000 7ff61d4e2640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff6180b6dc0 0x7ff6180d0740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46158/0 (socket says 172.21.15.67:46158) 2024-03-24T00:45:04.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.718+0000 7ff61d4e2640 1 -- 172.21.15.67:0/784538378 learned_addr learned my addr 172.21.15.67:0/784538378 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:45:04.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.719+0000 7ff617fff640 1 --2- 172.21.15.67:0/784538378 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff6180d32d0 0x7ff6180d3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.719+0000 7ff6177fe640 1 -- 172.21.15.67:0/784538378 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2736318379 0 0) 0x7ff6180b9cc0 con 0x7ff6180b6dc0 2024-03-24T00:45:04.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.719+0000 7ff6177fe640 1 -- 172.21.15.67:0/784538378 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff608003520 con 0x7ff6180b6dc0 2024-03-24T00:45:04.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.719+0000 7ff6177fe640 1 -- 172.21.15.67:0/784538378 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2901776660 0 0) 0x7ff6180d2230 con 0x7ff61815e0f0 2024-03-24T00:45:04.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.719+0000 7ff6177fe640 1 -- 172.21.15.67:0/784538378 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6180b9cc0 con 0x7ff61815e0f0 2024-03-24T00:45:04.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.719+0000 7ff617fff640 1 -- 172.21.15.67:0/784538378 >> v1:172.21.15.72:6789/0 conn(0x7ff61815e0f0 legacy=0x7ff6180b95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:04.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.719+0000 7ff617fff640 1 -- 172.21.15.67:0/784538378 >> v1:172.21.15.67:6789/0 conn(0x7ff6180b6dc0 legacy=0x7ff6180d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:04.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.719+0000 7ff617fff640 1 -- 172.21.15.67:0/784538378 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6180d27f0 con 0x7ff6180d32d0 2024-03-24T00:45:04.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.720+0000 7ff617fff640 1 --2- 172.21.15.67:0/784538378 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff6180d32d0 0x7ff6180d3710 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff6040097c0 tx=0x7ff60402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2b83d7ab0badd8b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:04.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.720+0000 7ff6177fe640 1 -- 172.21.15.67:0/784538378 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff604035020 con 0x7ff6180d32d0 2024-03-24T00:45:04.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.720+0000 7ff6177fe640 1 -- 172.21.15.67:0/784538378 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff604031890 con 0x7ff6180d32d0 2024-03-24T00:45:04.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.720+0000 7ff6177fe640 1 -- 172.21.15.67:0/784538378 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff604031c60 con 0x7ff6180d32d0 2024-03-24T00:45:04.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.721+0000 7ff61ef45640 1 -- 172.21.15.67:0/784538378 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff6180d32d0 msgr2=0x7ff6180d3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:04.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.721+0000 7ff61ef45640 1 --2- 172.21.15.67:0/784538378 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff6180d32d0 0x7ff6180d3710 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff6040097c0 tx=0x7ff60402b930 comp rx=0 tx=0).stop 2024-03-24T00:45:04.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.721+0000 7ff61ef45640 1 -- 172.21.15.67:0/784538378 shutdown_connections 2024-03-24T00:45:04.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.721+0000 7ff61ef45640 1 --2- 172.21.15.67:0/784538378 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff6180d32d0 0x7ff6180d3710 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.721+0000 7ff61ef45640 1 -- 172.21.15.67:0/784538378 >> 172.21.15.67:0/784538378 conn(0x7ff618072b60 msgr2=0x7ff618072f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:04.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.721+0000 7ff61ef45640 1 -- 172.21.15.67:0/784538378 shutdown_connections 2024-03-24T00:45:04.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.721+0000 7ff61ef45640 1 -- 172.21.15.67:0/784538378 wait complete. 2024-03-24T00:45:04.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.722+0000 7ff61ef45640 1 Processor -- start 2024-03-24T00:45:04.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.723+0000 7ff61ef45640 1 -- start start 2024-03-24T00:45:04.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.724+0000 7ff61ef45640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff6180b6dc0 0x7ff6180cc730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.724+0000 7ff61ef45640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6180d32d0 0x7ff61814f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.724+0000 7ff61ef45640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff61815e0f0 0x7ff6180cad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.724+0000 7ff61ef45640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6180b31d0 con 0x7ff6180d32d0 2024-03-24T00:45:04.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.724+0000 7ff61ef45640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6180d2db0 con 0x7ff61815e0f0 2024-03-24T00:45:04.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.724+0000 7ff61ef45640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6180b3350 con 0x7ff6180b6dc0 2024-03-24T00:45:04.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.724+0000 7ff617fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6180d32d0 0x7ff61814f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.724+0000 7ff617fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6180d32d0 0x7ff61814f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32896/0 (socket says 172.21.15.67:32896) 2024-03-24T00:45:04.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.724+0000 7ff617fff640 1 -- 172.21.15.67:0/798537769 learned_addr learned my addr 172.21.15.67:0/798537769 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:04.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.724+0000 7ff61d4e2640 1 --2- 172.21.15.67:0/798537769 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff61815e0f0 0x7ff6180cad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.724+0000 7ff61cce1640 1 --2- 172.21.15.67:0/798537769 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff6180b6dc0 0x7ff6180cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.724+0000 7ff617fff640 1 -- 172.21.15.67:0/798537769 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff6180b6dc0 msgr2=0x7ff6180cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:04.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.724+0000 7ff617fff640 1 --2- 172.21.15.67:0/798537769 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff6180b6dc0 0x7ff6180cc730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.724+0000 7ff617fff640 1 -- 172.21.15.67:0/798537769 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff61815e0f0 msgr2=0x7ff6180cad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:04.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.724+0000 7ff617fff640 1 --2- 172.21.15.67:0/798537769 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff61815e0f0 0x7ff6180cad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.724+0000 7ff617fff640 1 -- 172.21.15.67:0/798537769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6180ccc70 con 0x7ff6180d32d0 2024-03-24T00:45:04.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.724+0000 7ff617fff640 1 --2- 172.21.15.67:0/798537769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6180d32d0 0x7ff61814f0f0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7ff60402bf40 tx=0x7ff6040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:04.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.725+0000 7ff6157fa640 1 -- 172.21.15.67:0/798537769 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff604035040 con 0x7ff6180d32d0 2024-03-24T00:45:04.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.725+0000 7ff6157fa640 1 -- 172.21.15.67:0/798537769 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff604030b10 con 0x7ff6180d32d0 2024-03-24T00:45:04.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.725+0000 7ff61ef45640 1 -- 172.21.15.67:0/798537769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6180cb690 con 0x7ff6180d32d0 2024-03-24T00:45:04.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.725+0000 7ff6157fa640 1 -- 172.21.15.67:0/798537769 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff604044430 con 0x7ff6180d32d0 2024-03-24T00:45:04.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.725+0000 7ff61ef45640 1 -- 172.21.15.67:0/798537769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6180cbc50 con 0x7ff6180d32d0 2024-03-24T00:45:04.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.727+0000 7ff6157fa640 1 -- 172.21.15.67:0/798537769 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff604038020 con 0x7ff6180d32d0 2024-03-24T00:45:04.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.727+0000 7ff6157fa640 1 --2- 172.21.15.67:0/798537769 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff5fc06bc00 0x7ff5fc06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.727+0000 7ff6157fa640 1 -- 172.21.15.67:0/798537769 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff6040ad120 con 0x7ff6180d32d0 2024-03-24T00:45:04.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.727+0000 7ff61cce1640 1 --2- 172.21.15.67:0/798537769 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff5fc06bc00 0x7ff5fc06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.728+0000 7ff621c0d740 1 Processor -- start 2024-03-24T00:45:04.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.728+0000 7ff621c0d740 1 -- start start 2024-03-24T00:45:04.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.728+0000 7ff61cce1640 1 --2- 172.21.15.67:0/798537769 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff5fc06bc00 0x7ff5fc06e0c0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7ff60c0015b0 tx=0x7ff60c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:04.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.729+0000 7ff621c0d740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558edcd9ac40 0x558edcdc1770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.729+0000 7ff621c0d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558edcdc1cb0 0x558edcd99c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.729+0000 7ff621c0d740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558edcd9b020 0x558edcd9a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.729+0000 7ff621c0d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x558edccfec20 con 0x558edcdc1cb0 2024-03-24T00:45:04.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.729+0000 7ff621c0d740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x558edcca7a40 con 0x558edcd9ac40 2024-03-24T00:45:04.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.729+0000 7ff621c0d740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x558edccfacc0 con 0x558edcd9b020 2024-03-24T00:45:04.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.729+0000 7ff617fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558edcdc1cb0 0x558edcd99c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.729+0000 7ff617fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558edcdc1cb0 0x558edcd99c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32900/0 (socket says 172.21.15.67:32900) 2024-03-24T00:45:04.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.729+0000 7ff617fff640 1 -- 172.21.15.67:0/1873935450 learned_addr learned my addr 172.21.15.67:0/1873935450 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:04.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.729+0000 7ff61cce1640 1 --2- 172.21.15.67:0/1873935450 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558edcd9ac40 0x558edcdc1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.729+0000 7ff61d4e2640 1 --2- 172.21.15.67:0/1873935450 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558edcd9b020 0x558edcd9a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.729+0000 7ff617fff640 1 -- 172.21.15.67:0/1873935450 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558edcd9b020 msgr2=0x558edcd9a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:04.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.729+0000 7ff617fff640 1 --2- 172.21.15.67:0/1873935450 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558edcd9b020 0x558edcd9a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.729+0000 7ff617fff640 1 -- 172.21.15.67:0/1873935450 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558edcd9ac40 msgr2=0x558edcdc1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:04.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.729+0000 7ff617fff640 1 --2- 172.21.15.67:0/1873935450 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558edcd9ac40 0x558edcdc1770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.729+0000 7ff617fff640 1 -- 172.21.15.67:0/1873935450 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558edccc9f90 con 0x558edcdc1cb0 2024-03-24T00:45:04.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.729+0000 7ff617fff640 1 --2- 172.21.15.67:0/1873935450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558edcdc1cb0 0x558edcd99c80 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7ff604083540 tx=0x7ff60407b1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c32cf218eae3dd45 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:04.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.729+0000 7ff6027fc640 1 -- 172.21.15.67:0/1873935450 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff60407b360 con 0x558edcdc1cb0 2024-03-24T00:45:04.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.729+0000 7ff6027fc640 1 -- 172.21.15.67:0/1873935450 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff60407f4b0 con 0x558edcdc1cb0 2024-03-24T00:45:04.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.729+0000 7ff6027fc640 1 -- 172.21.15.67:0/1873935450 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff604083a70 con 0x558edcdc1cb0 2024-03-24T00:45:04.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.730+0000 7ff621c0d740 1 -- 172.21.15.67:0/1873935450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558edcdc1cb0 msgr2=0x558edcd99c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:04.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.730+0000 7ff621c0d740 1 --2- 172.21.15.67:0/1873935450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558edcdc1cb0 0x558edcd99c80 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7ff604083540 tx=0x7ff60407b1f0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.730+0000 7ff621c0d740 1 -- 172.21.15.67:0/1873935450 shutdown_connections 2024-03-24T00:45:04.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.730+0000 7ff621c0d740 1 --2- 172.21.15.67:0/1873935450 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558edcd9b020 0x558edcd9a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.730+0000 7ff621c0d740 1 --2- 172.21.15.67:0/1873935450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558edcdc1cb0 0x558edcd99c80 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.730+0000 7ff621c0d740 1 --2- 172.21.15.67:0/1873935450 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558edcd9ac40 0x558edcdc1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.730+0000 7ff621c0d740 1 -- 172.21.15.67:0/1873935450 >> 172.21.15.67:0/1873935450 conn(0x558edcd61a60 msgr2=0x558edcddf960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:04.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.730+0000 7ff621c0d740 1 -- 172.21.15.67:0/1873935450 shutdown_connections 2024-03-24T00:45:04.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.730+0000 7ff621c0d740 1 -- 172.21.15.67:0/1873935450 wait complete. 2024-03-24T00:45:04.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.730+0000 7ff621c0d740 1 -- start start 2024-03-24T00:45:04.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.731+0000 7ff621c0d740 1 Processor -- start 2024-03-24T00:45:04.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.731+0000 7ff621c0d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558edcd9ac40 0x558edcdc8a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.731+0000 7ff621c0d740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558edcd9b020 0x558edcdd7e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.731+0000 7ff621c0d740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558edcdd8370 0x558edcdc5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.731+0000 7ff621c0d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x558edcdc12e0 con 0x558edcd9ac40 2024-03-24T00:45:04.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.731+0000 7ff621c0d740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x558edccbae90 con 0x558edcdd8370 2024-03-24T00:45:04.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.731+0000 7ff621c0d740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x558edcdba060 con 0x558edcd9b020 2024-03-24T00:45:04.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.731+0000 7ff621c0d740 10 client.0 _ll_register_callbacks cb 0x558edca12ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:45:04.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.732+0000 7ff61cce1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558edcd9ac40 0x558edcdc8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.732+0000 7ff61cce1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558edcd9ac40 0x558edcdc8a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32908/0 (socket says 172.21.15.67:32908) 2024-03-24T00:45:04.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.732+0000 7ff61cce1640 1 -- 172.21.15.67:0/4206349103 learned_addr learned my addr 172.21.15.67:0/4206349103 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:04.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.732+0000 7ff617fff640 1 --2- 172.21.15.67:0/4206349103 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558edcd9b020 0x558edcdd7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.732+0000 7ff61d4e2640 1 --2- 172.21.15.67:0/4206349103 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558edcdd8370 0x558edcdc5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.732+0000 7ff61cce1640 1 -- 172.21.15.67:0/4206349103 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558edcd9b020 msgr2=0x558edcdd7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:04.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.732+0000 7ff61cce1640 1 --2- 172.21.15.67:0/4206349103 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558edcd9b020 0x558edcdd7e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.732+0000 7ff61cce1640 1 -- 172.21.15.67:0/4206349103 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558edcdd8370 msgr2=0x558edcdc5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:04.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.732+0000 7ff61cce1640 1 --2- 172.21.15.67:0/4206349103 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558edcdd8370 0x558edcdc5ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.732+0000 7ff61cce1640 1 -- 172.21.15.67:0/4206349103 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558edccc79d0 con 0x558edcd9ac40 2024-03-24T00:45:04.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.732+0000 7ff61cce1640 1 --2- 172.21.15.67:0/4206349103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558edcd9ac40 0x558edcdc8a20 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7ff60c0039d0 tx=0x7ff60c000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5afcfd5ec2ca3e64 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:04.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.732+0000 7ff5f77fe640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:45:04.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.732+0000 7ff5f77fe640 1 -- 172.21.15.67:0/4206349103 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff60c027060 con 0x558edcd9ac40 2024-03-24T00:45:04.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.732+0000 7ff5f77fe640 1 -- 172.21.15.67:0/4206349103 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff60c010040 con 0x558edcd9ac40 2024-03-24T00:45:04.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.732+0000 7ff621c0d740 1 -- 172.21.15.67:0/4206349103 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff60407f4b0 con 0x558edcd9ac40 2024-03-24T00:45:04.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.732+0000 7ff5f77fe640 1 -- 172.21.15.67:0/4206349103 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff60c00ec40 con 0x558edcd9ac40 2024-03-24T00:45:04.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.732+0000 7ff621c0d740 10 client.7794 fetch_fsmap learned FSMap version 54 2024-03-24T00:45:04.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.732+0000 7ff621c0d740 1 -- 172.21.15.67:0/4206349103 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=2+}) v3 -- 0x558edca132d0 con 0x558edcd9ac40 2024-03-24T00:45:04.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.733+0000 7ff5f77fe640 1 -- 172.21.15.67:0/4206349103 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff60407f4b0 con 0x558edcd9ac40 2024-03-24T00:45:04.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.733+0000 7ff5f77fe640 1 -- 172.21.15.67:0/4206349103 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 54) v1 ==== 3314+0+0 (secure 0 0 0) 0x7ff60c013420 con 0x558edcd9ac40 2024-03-24T00:45:04.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.733+0000 7ff621c0d740 10 client.7794 fetch_fsmap finished waiting for FSMap version 54 2024-03-24T00:45:04.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.733+0000 7ff621c0d740 10 client.7794 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T00:45:04.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.733+0000 7ff621c0d740 20 client.7794 populate_metadata read hostname 'smithi067' 2024-03-24T00:45:04.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.733+0000 7ff621c0d740 1 --2- 172.21.15.67:0/4206349103 >> [v2:172.21.15.67:6832/3021622863,v1:172.21.15.67:6833/3021622863] conn(0x558edcdefce0 0x558edcdf00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.733+0000 7ff621c0d740 4 client.7794 mds_command: new command op to 7617 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:45:04.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.733+0000 7ff621c0d740 1 -- 172.21.15.67:0/4206349103 --> [v2:172.21.15.67:6832/3021622863,v1:172.21.15.67:6833/3021622863] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558edcd0c160 con 0x558edcdefce0 2024-03-24T00:45:04.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.733+0000 7ff617fff640 1 --2- 172.21.15.67:0/4206349103 >> [v2:172.21.15.67:6832/3021622863,v1:172.21.15.67:6833/3021622863] conn(0x558edcdefce0 0x558edcdf00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.733+0000 7ff5f77fe640 10 client.7794 ms_handle_connect on v2:172.21.15.67:6832/3021622863 2024-03-24T00:45:04.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.733+0000 7ff617fff640 1 --2- 172.21.15.67:0/4206349103 >> [v2:172.21.15.67:6832/3021622863,v1:172.21.15.67:6833/3021622863] conn(0x558edcdefce0 0x558edcdf00e0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52b78eb4d7173ac4 server_cookie=20ec49ce6e12649d in_seq=0 out_seq=0 2024-03-24T00:45:04.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.734+0000 7ff5f77fe640 1 -- 172.21.15.67:0/4206349103 <== mds.0 v2:172.21.15.67:6832/3021622863 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x558edcd0c160 con 0x558edcdefce0 2024-03-24T00:45:04.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.734+0000 7ff5f77fe640 10 client.7794 handle_command_reply: tid=0 2024-03-24T00:45:04.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.734+0000 7ff621c0d740 1 client.7794 shutdown 2024-03-24T00:45:04.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.734+0000 7ff621c0d740 1 -- 172.21.15.67:0/4206349103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558edcd9ac40 msgr2=0x558edcdc8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:04.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.734+0000 7ff621c0d740 1 --2- 172.21.15.67:0/4206349103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558edcd9ac40 0x558edcdc8a20 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7ff60c0039d0 tx=0x7ff60c000ed0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.734+0000 7ff621c0d740 1 -- 172.21.15.67:0/4206349103 shutdown_connections 2024-03-24T00:45:04.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.734+0000 7ff621c0d740 1 --2- 172.21.15.67:0/4206349103 >> [v2:172.21.15.67:6832/3021622863,v1:172.21.15.67:6833/3021622863] conn(0x558edcdefce0 0x558edcdf00e0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.734+0000 7ff621c0d740 1 --2- 172.21.15.67:0/4206349103 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558edcdd8370 0x558edcdc5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.734+0000 7ff621c0d740 1 --2- 172.21.15.67:0/4206349103 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558edcd9b020 0x558edcdd7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.734+0000 7ff5f77fe640 0 client.7794 ms_handle_reset on v2:172.21.15.67:6832/3021622863 2024-03-24T00:45:04.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.734+0000 7ff621c0d740 1 --2- 172.21.15.67:0/4206349103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558edcd9ac40 0x558edcdc8a20 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.734+0000 7ff621c0d740 1 -- 172.21.15.67:0/4206349103 >> 172.21.15.67:0/4206349103 conn(0x558edcd61a60 msgr2=0x558edcdc2940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:04.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.734+0000 7ff621c0d740 1 -- 172.21.15.67:0/4206349103 shutdown_connections 2024-03-24T00:45:04.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.734+0000 7ff621c0d740 1 -- 172.21.15.67:0/4206349103 wait complete. 2024-03-24T00:45:04.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.734+0000 7ff621c0d740 20 client.7794 trim_cache size 0 max 16384 2024-03-24T00:45:04.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.751+0000 7ff621c0d740 1 Processor -- start 2024-03-24T00:45:04.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.751+0000 7ff621c0d740 1 -- start start 2024-03-24T00:45:04.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.752+0000 7ff621c0d740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558edcd61a60 0x558edcdc7550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.752+0000 7ff621c0d740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558edcd9ac40 0x558edcdee080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.752+0000 7ff621c0d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558edcd9b020 0x558edcdee5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.752+0000 7ff621c0d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6181ffa00 con 0x558edcd9b020 2024-03-24T00:45:04.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.752+0000 7ff621c0d740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff61815fe30 con 0x558edcd9ac40 2024-03-24T00:45:04.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.752+0000 7ff621c0d740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6181ff670 con 0x558edcd61a60 2024-03-24T00:45:04.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.752+0000 7ff61d4e2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558edcd9b020 0x558edcdee5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.752+0000 7ff61d4e2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558edcd9b020 0x558edcdee5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32910/0 (socket says 172.21.15.67:32910) 2024-03-24T00:45:04.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.752+0000 7ff61d4e2640 1 -- 172.21.15.67:0/4134862655 learned_addr learned my addr 172.21.15.67:0/4134862655 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:04.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.752+0000 7ff61d4e2640 1 -- 172.21.15.67:0/4134862655 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558edcd61a60 msgr2=0x558edcdc7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:04.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.752+0000 7ff61d4e2640 1 --2- 172.21.15.67:0/4134862655 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558edcd61a60 0x558edcdc7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.752+0000 7ff61d4e2640 1 -- 172.21.15.67:0/4134862655 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558edcd9ac40 msgr2=0x558edcdee080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:04.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.752+0000 7ff61d4e2640 1 --2- 172.21.15.67:0/4134862655 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558edcd9ac40 0x558edcdee080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.752+0000 7ff61d4e2640 1 -- 172.21.15.67:0/4134862655 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558edcdc2bb0 con 0x558edcd9b020 2024-03-24T00:45:04.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.752+0000 7ff61d4e2640 1 --2- 172.21.15.67:0/4134862655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558edcd9b020 0x558edcdee5c0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7ff610002170 tx=0x7ff610014710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce5b2a414cfd1f98 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:04.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.753+0000 7ff601ffb640 1 -- 172.21.15.67:0/4134862655 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61001a070 con 0x558edcd9b020 2024-03-24T00:45:04.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.753+0000 7ff601ffb640 1 -- 172.21.15.67:0/4134862655 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff610005bd0 con 0x558edcd9b020 2024-03-24T00:45:04.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.753+0000 7ff601ffb640 1 -- 172.21.15.67:0/4134862655 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6100070d0 con 0x558edcd9b020 2024-03-24T00:45:04.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.753+0000 7ff621c0d740 1 -- 172.21.15.67:0/4134862655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558edcd9b020 msgr2=0x558edcdee5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:04.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.753+0000 7ff621c0d740 1 --2- 172.21.15.67:0/4134862655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558edcd9b020 0x558edcdee5c0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7ff610002170 tx=0x7ff610014710 comp rx=0 tx=0).stop 2024-03-24T00:45:04.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.753+0000 7ff621c0d740 1 -- 172.21.15.67:0/4134862655 shutdown_connections 2024-03-24T00:45:04.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.753+0000 7ff621c0d740 1 --2- 172.21.15.67:0/4134862655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558edcd9b020 0x558edcdee5c0 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.753+0000 7ff621c0d740 1 --2- 172.21.15.67:0/4134862655 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558edcd9ac40 0x558edcdee080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.753+0000 7ff621c0d740 1 --2- 172.21.15.67:0/4134862655 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558edcd61a60 0x558edcdc7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.753+0000 7ff621c0d740 1 -- 172.21.15.67:0/4134862655 >> 172.21.15.67:0/4134862655 conn(0x558edcdd8cd0 msgr2=0x558edcdd7750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:04.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.753+0000 7ff621c0d740 1 -- 172.21.15.67:0/4134862655 shutdown_connections 2024-03-24T00:45:04.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.753+0000 7ff621c0d740 1 -- 172.21.15.67:0/4134862655 wait complete. 2024-03-24T00:45:04.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.754+0000 7ff621c0d740 1 -- start start 2024-03-24T00:45:04.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.754+0000 7ff621c0d740 1 Processor -- start 2024-03-24T00:45:04.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.755+0000 7ff621c0d740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558edcd61a60 0x558edce35bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.755+0000 7ff621c0d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558edcd9ac40 0x558edcd93c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.755+0000 7ff621c0d740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558edcd9b020 0x558edce50930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.755+0000 7ff621c0d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x558edcdc6f40 con 0x558edcd9ac40 2024-03-24T00:45:04.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.755+0000 7ff621c0d740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x558edcdd7400 con 0x558edcd9b020 2024-03-24T00:45:04.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.755+0000 7ff621c0d740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x558edcdd6690 con 0x558edcd61a60 2024-03-24T00:45:04.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.755+0000 7ff621c0d740 10 client.0 _ll_register_callbacks cb 0x7ff60c0050f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:45:04.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.755+0000 7ff617fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558edcd9ac40 0x558edcd93c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.755+0000 7ff617fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558edcd9ac40 0x558edcd93c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32924/0 (socket says 172.21.15.67:32924) 2024-03-24T00:45:04.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.755+0000 7ff617fff640 1 -- 172.21.15.67:0/2984767139 learned_addr learned my addr 172.21.15.67:0/2984767139 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:04.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.755+0000 7ff61cce1640 1 --2- 172.21.15.67:0/2984767139 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558edcd61a60 0x558edce35bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.755+0000 7ff61d4e2640 1 --2- 172.21.15.67:0/2984767139 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558edcd9b020 0x558edce50930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.755+0000 7ff617fff640 1 -- 172.21.15.67:0/2984767139 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558edcd61a60 msgr2=0x558edce35bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:04.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.755+0000 7ff617fff640 1 --2- 172.21.15.67:0/2984767139 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558edcd61a60 0x558edce35bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.755+0000 7ff617fff640 1 -- 172.21.15.67:0/2984767139 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558edcd9b020 msgr2=0x558edce50930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:04.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.755+0000 7ff617fff640 1 --2- 172.21.15.67:0/2984767139 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558edcd9b020 0x558edce50930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.755+0000 7ff617fff640 1 -- 172.21.15.67:0/2984767139 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558edcadd6b0 con 0x558edcd9ac40 2024-03-24T00:45:04.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.755+0000 7ff61cce1640 1 --2- 172.21.15.67:0/2984767139 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558edcd61a60 0x558edce35bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:45:04.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.755+0000 7ff617fff640 1 --2- 172.21.15.67:0/2984767139 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558edcd9ac40 0x558edcd93c10 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7ff604009710 tx=0x7ff60403a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afd6daf5c62d028 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:04.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.755+0000 7ff5f77fe640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:45:04.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.755+0000 7ff5f77fe640 1 -- 172.21.15.67:0/2984767139 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff60407f4b0 con 0x558edcd9ac40 2024-03-24T00:45:04.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.755+0000 7ff5f77fe640 1 -- 172.21.15.67:0/2984767139 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff6040ab040 con 0x558edcd9ac40 2024-03-24T00:45:04.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.755+0000 7ff621c0d740 1 -- 172.21.15.67:0/2984767139 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff610005bd0 con 0x558edcd9ac40 2024-03-24T00:45:04.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.755+0000 7ff5f77fe640 1 -- 172.21.15.67:0/2984767139 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff60409f050 con 0x558edcd9ac40 2024-03-24T00:45:04.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.755+0000 7ff621c0d740 10 client.7800 fetch_fsmap learned FSMap version 54 2024-03-24T00:45:04.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.755+0000 7ff621c0d740 1 -- 172.21.15.67:0/2984767139 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=2+}) v3 -- 0x558edce413c0 con 0x558edcd9ac40 2024-03-24T00:45:04.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.756+0000 7ff5f77fe640 1 -- 172.21.15.67:0/2984767139 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff610005bd0 con 0x558edcd9ac40 2024-03-24T00:45:04.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.756+0000 7ff5f77fe640 1 -- 172.21.15.67:0/2984767139 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 54) v1 ==== 3314+0+0 (secure 0 0 0) 0x7ff6040779d0 con 0x558edcd9ac40 2024-03-24T00:45:04.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.756+0000 7ff621c0d740 10 client.7800 fetch_fsmap finished waiting for FSMap version 54 2024-03-24T00:45:04.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.756+0000 7ff621c0d740 10 client.7800 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T00:45:04.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.756+0000 7ff621c0d740 20 client.7800 populate_metadata read hostname 'smithi067' 2024-03-24T00:45:04.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.756+0000 7ff621c0d740 1 --2- 172.21.15.67:0/2984767139 >> [v2:172.21.15.67:6832/3021622863,v1:172.21.15.67:6833/3021622863] conn(0x558edcdd8370 0x558edcdd52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.756+0000 7ff621c0d740 4 client.7800 mds_command: new command op to 7617 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:45:04.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.756+0000 7ff621c0d740 1 -- 172.21.15.67:0/2984767139 --> [v2:172.21.15.67:6832/3021622863,v1:172.21.15.67:6833/3021622863] -- command(tid 0: {"prefix": "status"}) v1 -- 0x558edcd064b0 con 0x558edcdd8370 2024-03-24T00:45:04.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.756+0000 7ff61cce1640 1 --2- 172.21.15.67:0/2984767139 >> [v2:172.21.15.67:6832/3021622863,v1:172.21.15.67:6833/3021622863] conn(0x558edcdd8370 0x558edcdd52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.756+0000 7ff5f77fe640 10 client.7800 ms_handle_connect on v2:172.21.15.67:6832/3021622863 2024-03-24T00:45:04.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.756+0000 7ff61cce1640 1 --2- 172.21.15.67:0/2984767139 >> [v2:172.21.15.67:6832/3021622863,v1:172.21.15.67:6833/3021622863] conn(0x558edcdd8370 0x558edcdd52f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9fc741cda647defc server_cookie=e5bb1a1226707cfc in_seq=0 out_seq=0 2024-03-24T00:45:04.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.756+0000 7ff5f77fe640 1 -- 172.21.15.67:0/2984767139 <== mds.0 v2:172.21.15.67:6832/3021622863 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x558edcd064b0 con 0x558edcdd8370 2024-03-24T00:45:04.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.756+0000 7ff5f77fe640 10 client.7800 handle_command_reply: tid=0 2024-03-24T00:45:04.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.756+0000 7ff621c0d740 1 client.7800 shutdown 2024-03-24T00:45:04.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.757+0000 7ff621c0d740 1 -- 172.21.15.67:0/2984767139 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558edcd9ac40 msgr2=0x558edcd93c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:04.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.757+0000 7ff621c0d740 1 --2- 172.21.15.67:0/2984767139 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558edcd9ac40 0x558edcd93c10 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7ff604009710 tx=0x7ff60403a000 comp rx=0 tx=0).stop 2024-03-24T00:45:04.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.757+0000 7ff621c0d740 1 -- 172.21.15.67:0/2984767139 shutdown_connections 2024-03-24T00:45:04.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.757+0000 7ff621c0d740 1 --2- 172.21.15.67:0/2984767139 >> [v2:172.21.15.67:6832/3021622863,v1:172.21.15.67:6833/3021622863] conn(0x558edcdd8370 0x558edcdd52f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.757+0000 7ff621c0d740 1 --2- 172.21.15.67:0/2984767139 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558edcd9b020 0x558edce50930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.757+0000 7ff621c0d740 1 --2- 172.21.15.67:0/2984767139 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558edcd9ac40 0x558edcd93c10 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.757+0000 7ff5f77fe640 0 client.7800 ms_handle_reset on v2:172.21.15.67:6832/3021622863 2024-03-24T00:45:04.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.757+0000 7ff621c0d740 1 --2- 172.21.15.67:0/2984767139 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558edcd61a60 0x558edce35bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.757+0000 7ff621c0d740 1 -- 172.21.15.67:0/2984767139 >> 172.21.15.67:0/2984767139 conn(0x558edcdd8cd0 msgr2=0x558edcdc7550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:04.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.757+0000 7ff621c0d740 1 -- 172.21.15.67:0/2984767139 shutdown_connections 2024-03-24T00:45:04.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.757+0000 7ff621c0d740 1 -- 172.21.15.67:0/2984767139 wait complete. 2024-03-24T00:45:04.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.757+0000 7ff621c0d740 20 client.7800 trim_cache size 0 max 16384 2024-03-24T00:45:04.766 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:45:04.766 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "de1d0178-808e-49ca-a986-e297e523b976", 2024-03-24T00:45:04.766 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-03-24T00:45:04.766 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 7617, 2024-03-24T00:45:04.766 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-03-24T00:45:04.766 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-03-24T00:45:04.766 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-03-24T00:45:04.766 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 9.3795748490000008, 2024-03-24T00:45:04.766 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 54, 2024-03-24T00:45:04.767 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 89, 2024-03-24T00:45:04.767 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 89, 2024-03-24T00:45:04.767 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 14.725104271999999 2024-03-24T00:45:04.767 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:45:04.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.757+0000 7ff61ef45640 1 -- 172.21.15.67:0/798537769 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff5fc06bc00 msgr2=0x7ff5fc06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:04.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.757+0000 7ff61ef45640 1 --2- 172.21.15.67:0/798537769 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff5fc06bc00 0x7ff5fc06e0c0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7ff60c0015b0 tx=0x7ff60c009040 comp rx=0 tx=0).stop 2024-03-24T00:45:04.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.757+0000 7ff61ef45640 1 -- 172.21.15.67:0/798537769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6180d32d0 msgr2=0x7ff61814f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:04.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.757+0000 7ff61ef45640 1 --2- 172.21.15.67:0/798537769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6180d32d0 0x7ff61814f0f0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7ff60402bf40 tx=0x7ff6040046c0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.757+0000 7ff61ef45640 1 -- 172.21.15.67:0/798537769 shutdown_connections 2024-03-24T00:45:04.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.757+0000 7ff61ef45640 1 --2- 172.21.15.67:0/798537769 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff5fc06bc00 0x7ff5fc06e0c0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.757+0000 7ff61ef45640 1 --2- 172.21.15.67:0/798537769 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff61815e0f0 0x7ff6180cad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.757+0000 7ff61ef45640 1 --2- 172.21.15.67:0/798537769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6180d32d0 0x7ff61814f0f0 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.757+0000 7ff61ef45640 1 --2- 172.21.15.67:0/798537769 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff6180b6dc0 0x7ff6180cc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.757+0000 7ff61ef45640 1 -- 172.21.15.67:0/798537769 >> 172.21.15.67:0/798537769 conn(0x7ff618072b60 msgr2=0x7ff6180c8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:04.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.758+0000 7ff61ef45640 1 -- 172.21.15.67:0/798537769 shutdown_connections 2024-03-24T00:45:04.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.758+0000 7ff61ef45640 1 -- 172.21.15.67:0/798537769 wait complete. 2024-03-24T00:45:04.769 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T00:45:04.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.870+0000 7f781a063640 1 Processor -- start 2024-03-24T00:45:04.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.870+0000 7f781a063640 1 -- start start 2024-03-24T00:45:04.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.871+0000 7f781a063640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f781415df30 0x7f7814162610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.871+0000 7f781a063640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f781415e310 0x7f78140d0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.871+0000 7f781a063640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78140ce920 con 0x7f78140d0b80 2024-03-24T00:45:04.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.871+0000 7f781a063640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7814162b50 con 0x7f781415e310 2024-03-24T00:45:04.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.871+0000 7f781a063640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78140d0fd0 con 0x7f781415df30 2024-03-24T00:45:04.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.871+0000 7f7812ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f781415e310 0x7f78140d0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.871+0000 7f7813fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f78140d0b80 0x7f78140b57a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46168/0 (socket says 172.21.15.67:46168) 2024-03-24T00:45:04.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.871+0000 7f7813fff640 1 -- 172.21.15.67:0/2087754516 learned_addr learned my addr 172.21.15.67:0/2087754516 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:45:04.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.871+0000 7f78137fe640 1 --2- 172.21.15.67:0/2087754516 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f781415df30 0x7f7814162610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.872+0000 7f7812ffd640 1 -- 172.21.15.67:0/2087754516 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f781415df30 msgr2=0x7f7814162610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:04.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.872+0000 7f7812ffd640 1 --2- 172.21.15.67:0/2087754516 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f781415df30 0x7f7814162610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.872+0000 7f7812ffd640 1 -- 172.21.15.67:0/2087754516 >> v1:172.21.15.67:6789/0 conn(0x7f78140d0b80 legacy=0x7f78140b57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:04.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.872+0000 7f7812ffd640 1 -- 172.21.15.67:0/2087754516 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78140cef20 con 0x7f781415e310 2024-03-24T00:45:04.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.872+0000 7f78127fc640 1 -- 172.21.15.67:0/2087754516 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4232715182 0 0) 0x7f78140ce920 con 0x7f78140d0b80 2024-03-24T00:45:04.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.873+0000 7f7812ffd640 1 --2- 172.21.15.67:0/2087754516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f781415e310 0x7f78140d0640 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f78080097d0 tx=0x7f780802b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85fc1af5bcadc071 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:04.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.873+0000 7f78127fc640 1 -- 172.21.15.67:0/2087754516 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7808031770 con 0x7f781415e310 2024-03-24T00:45:04.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.873+0000 7f78127fc640 1 -- 172.21.15.67:0/2087754516 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7808037070 con 0x7f781415e310 2024-03-24T00:45:04.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.873+0000 7f78127fc640 1 -- 172.21.15.67:0/2087754516 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7808031b30 con 0x7f781415e310 2024-03-24T00:45:04.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.874+0000 7f781a063640 1 -- 172.21.15.67:0/2087754516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f781415e310 msgr2=0x7f78140d0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:04.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.874+0000 7f781a063640 1 --2- 172.21.15.67:0/2087754516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f781415e310 0x7f78140d0640 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f78080097d0 tx=0x7f780802b810 comp rx=0 tx=0).stop 2024-03-24T00:45:04.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.874+0000 7f781a063640 1 -- 172.21.15.67:0/2087754516 shutdown_connections 2024-03-24T00:45:04.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.874+0000 7f781a063640 1 --2- 172.21.15.67:0/2087754516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f781415e310 0x7f78140d0640 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.874+0000 7f781a063640 1 --2- 172.21.15.67:0/2087754516 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f781415df30 0x7f7814162610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.874+0000 7f781a063640 1 -- 172.21.15.67:0/2087754516 >> 172.21.15.67:0/2087754516 conn(0x7f781406c880 msgr2=0x7f781406cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:04.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.874+0000 7f781a063640 1 -- 172.21.15.67:0/2087754516 shutdown_connections 2024-03-24T00:45:04.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.874+0000 7f781a063640 1 -- 172.21.15.67:0/2087754516 wait complete. 2024-03-24T00:45:04.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.875+0000 7f781a063640 1 Processor -- start 2024-03-24T00:45:04.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.876+0000 7f781a063640 1 -- start start 2024-03-24T00:45:04.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.877+0000 7f781a063640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78140d0b80 0x7f78140aa990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.877+0000 7f781a063640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f781415df30 0x7f78140b4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.877+0000 7f781a063640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7814152d30 0x7f781414f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.877+0000 7f781a063640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78140b32f0 con 0x7f781415df30 2024-03-24T00:45:04.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.877+0000 7f781a063640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78140cf4e0 con 0x7f78140d0b80 2024-03-24T00:45:04.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.877+0000 7f781a063640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78140ce580 con 0x7f7814152d30 2024-03-24T00:45:04.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.877+0000 7f7812ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f781415df30 0x7f78140b4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.877+0000 7f7812ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f781415df30 0x7f78140b4550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32942/0 (socket says 172.21.15.67:32942) 2024-03-24T00:45:04.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.877+0000 7f7812ffd640 1 -- 172.21.15.67:0/890469361 learned_addr learned my addr 172.21.15.67:0/890469361 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:04.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.877+0000 7f7813fff640 1 --2- 172.21.15.67:0/890469361 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7814152d30 0x7f781414f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.877+0000 7f78137fe640 1 --2- 172.21.15.67:0/890469361 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78140d0b80 0x7f78140aa990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.877+0000 7f7812ffd640 1 -- 172.21.15.67:0/890469361 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7814152d30 msgr2=0x7f781414f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:04.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.877+0000 7f7812ffd640 1 --2- 172.21.15.67:0/890469361 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7814152d30 0x7f781414f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.877+0000 7f7812ffd640 1 -- 172.21.15.67:0/890469361 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78140d0b80 msgr2=0x7f78140aa990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:04.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.877+0000 7f7812ffd640 1 --2- 172.21.15.67:0/890469361 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78140d0b80 0x7f78140aa990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.877+0000 7f7812ffd640 1 -- 172.21.15.67:0/890469361 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78140b4f10 con 0x7f781415df30 2024-03-24T00:45:04.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.878+0000 7f7812ffd640 1 --2- 172.21.15.67:0/890469361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f781415df30 0x7f78140b4550 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f7808034040 tx=0x7f78080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:04.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.878+0000 7f77fbfff640 1 -- 172.21.15.67:0/890469361 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7808032600 con 0x7f781415df30 2024-03-24T00:45:04.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.878+0000 7f77fbfff640 1 -- 172.21.15.67:0/890469361 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78080327e0 con 0x7f781415df30 2024-03-24T00:45:04.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.878+0000 7f781a063640 1 -- 172.21.15.67:0/890469361 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78140b51f0 con 0x7f781415df30 2024-03-24T00:45:04.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.878+0000 7f77fbfff640 1 -- 172.21.15.67:0/890469361 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7808032bd0 con 0x7f781415df30 2024-03-24T00:45:04.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.878+0000 7f781a063640 1 -- 172.21.15.67:0/890469361 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78140b57b0 con 0x7f781415df30 2024-03-24T00:45:04.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.879+0000 7f77fbfff640 1 -- 172.21.15.67:0/890469361 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f780804c060 con 0x7f781415df30 2024-03-24T00:45:04.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.879+0000 7f781a063640 1 -- 172.21.15.67:0/890469361 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f781404f400 con 0x7f781415df30 2024-03-24T00:45:04.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.879+0000 7f77fbfff640 1 --2- 172.21.15.67:0/890469361 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f780006bbd0 0x7f780006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.880+0000 7f77fbfff640 1 -- 172.21.15.67:0/890469361 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f78080ac430 con 0x7f781415df30 2024-03-24T00:45:04.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.880+0000 7f78137fe640 1 --2- 172.21.15.67:0/890469361 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f780006bbd0 0x7f780006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.880+0000 7f78137fe640 1 --2- 172.21.15.67:0/890469361 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f780006bbd0 0x7f780006e090 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f7804009b40 tx=0x7f7804009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:04.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:04.884+0000 7f77fbfff640 1 -- 172.21.15.67:0/890469361 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7808040070 con 0x7f781415df30 2024-03-24T00:45:05.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:05.284+0000 7f781a063640 1 -- 172.21.15.67:0/890469361 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f78140b92a0 con 0x7f781415df30 2024-03-24T00:45:05.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:05.285+0000 7f77fbfff640 1 -- 172.21.15.67:0/890469361 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v54) v1 ==== 53+0+73 (secure 0 0 0) 0x7f780807a7d0 con 0x7f781415df30 2024-03-24T00:45:05.285 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-24T00:45:05.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:05.286+0000 7f781a063640 1 -- 172.21.15.67:0/890469361 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f780006bbd0 msgr2=0x7f780006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:05.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:05.286+0000 7f781a063640 1 --2- 172.21.15.67:0/890469361 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f780006bbd0 0x7f780006e090 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f7804009b40 tx=0x7f7804009190 comp rx=0 tx=0).stop 2024-03-24T00:45:05.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:05.286+0000 7f781a063640 1 -- 172.21.15.67:0/890469361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f781415df30 msgr2=0x7f78140b4550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:05.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:05.286+0000 7f781a063640 1 --2- 172.21.15.67:0/890469361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f781415df30 0x7f78140b4550 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f7808034040 tx=0x7f78080046c0 comp rx=0 tx=0).stop 2024-03-24T00:45:05.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:05.287+0000 7f781a063640 1 -- 172.21.15.67:0/890469361 shutdown_connections 2024-03-24T00:45:05.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:05.287+0000 7f781a063640 1 --2- 172.21.15.67:0/890469361 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f780006bbd0 0x7f780006e090 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:05.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:05.287+0000 7f781a063640 1 --2- 172.21.15.67:0/890469361 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7814152d30 0x7f781414f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:05.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:05.287+0000 7f781a063640 1 --2- 172.21.15.67:0/890469361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f781415df30 0x7f78140b4550 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:05.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:05.287+0000 7f781a063640 1 --2- 172.21.15.67:0/890469361 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78140d0b80 0x7f78140aa990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:05.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:05.287+0000 7f781a063640 1 -- 172.21.15.67:0/890469361 >> 172.21.15.67:0/890469361 conn(0x7f781406c880 msgr2=0x7f7814160710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:05.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:05.287+0000 7f781a063640 1 -- 172.21.15.67:0/890469361 shutdown_connections 2024-03-24T00:45:05.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:05.287+0000 7f781a063640 1 -- 172.21.15.67:0/890469361 wait complete. 2024-03-24T00:45:05.297 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-24T00:45:05.297 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T00:45:05.297 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-03-24T00:45:05.297 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-24T00:45:05.297 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T00:45:05.297 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-24T00:45:05.298 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-24T00:45:05.298 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T00:45:05.298 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:45:05.298 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:45:05.298 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-03-24T00:45:05.313 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T00:45:05.313 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T00:45:05.313 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T00:45:05.313 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:45:05.314 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-03-24T00:45:05.314 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:45:05.314 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:45:05.314 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-03-24T00:45:05.314 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-03-24T00:45:05.314 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:45:05.314 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-03-24T00:45:05.314 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-03-24T00:45:05.314 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:45:05.314 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-03-24T00:45:05.314 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T00:45:05.315 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-03-24T00:45:05.315 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-24T00:45:05.315 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 42208sec preferred_lft 42208sec 2024-03-24T00:45:05.315 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-03-24T00:45:05.315 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:45:05.315 INFO:teuthology.orchestra.run.smithi067.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T00:45:05.315 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 96:cf:26:c8:bf:62 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:45:05.315 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-24T00:45:05.315 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:45:05.315 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::7c6a:5fff:feaa:d41/64 scope link 2024-03-24T00:45:05.315 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:45:05.315 INFO:teuthology.orchestra.run.smithi067.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T00:45:05.316 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ea:22:4b:d5:48:3d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:45:05.316 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::e822:4bff:fed5:483d/64 scope link 2024-03-24T00:45:05.316 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:45:05.316 INFO:teuthology.orchestra.run.smithi067.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T00:45:05.316 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 96:cf:26:c8:bf:62 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:45:05.316 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::94cf:26ff:fec8:bf62/64 scope link 2024-03-24T00:45:05.316 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:45:05.316 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:45:05.316 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T00:45:05.378 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-03-24T00:45:05.380 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:45:05.380 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-03-24T00:45:05.435 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-03-24T00:45:05.435 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T00:45:05.435 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-24T00:45:05.435 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-24T00:45:05.435 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T00:45:05.436 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:45:05.436 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-03-24T00:45:05.436 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-03-24T00:45:05.436 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-24T00:45:05.436 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-24T00:45:05.436 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-24T00:45:05.436 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-03-24T00:45:05.530 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:45:05.531 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-03-24T00:45:05.545 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-24T00:45:05.545 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-24T00:45:05.545 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:45:05.545 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:45:05.545 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-03-24T00:45:05.611 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:45:05.611 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-24T00:45:05.666 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:45:05.666 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:45:05.666 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:45:05.720 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-24T00:45:05.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:45:05.721 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:45:05.804 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:45:05.804 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T00:45:05.804 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-03-24T00:45:05.812 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:45:05.812 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T00:45:05.813 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-03-24T00:45:05.813 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.72:3300,172.21.15.72:3301" 2024-03-24T00:45:05.813 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-24T00:45:05.817 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:45:05.818 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T00:45:05.818 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-03-24T00:45:05.818 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.72:3300,172.21.15.72:3301" 2024-03-24T00:45:05.818 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-24T00:45:05.818 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:6789,172.21.15.72:6789,172.21.15.72:6790" 2024-03-24T00:45:05.818 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.67:6789,172.21.15.72:6789,172.21.15.72:6790:/ 2024-03-24T00:45:05.818 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=de1d0178-808e-49ca-a986-e297e523b976" will pass to kernel 2024-03-24T00:45:05.818 INFO:teuthology.orchestra.run.smithi067.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-24T00:45:05.819 INFO:teuthology.orchestra.run.smithi067.stdout: You just mounted a file system that supports labels which does not 2024-03-24T00:45:05.819 INFO:teuthology.orchestra.run.smithi067.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-24T00:45:05.819 INFO:teuthology.orchestra.run.smithi067.stdout: applications will generate AVC messages and not be allowed access to 2024-03-24T00:45:05.819 INFO:teuthology.orchestra.run.smithi067.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-24T00:45:05.820 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-24T00:45:05.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:45:05.821 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:45:05.848 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:45:05.848 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T00:45:05.908 INFO:teuthology.orchestra.run.smithi067.stdout:client6464 2024-03-24T00:45:05.908 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:45:05.908 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:45:05.964 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:45:05.964 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir dir) 2024-03-24T00:45:06.022 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:45:06.022 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch dir/file) 2024-03-24T00:45:06.078 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:45:06.079 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir testdir) 2024-03-24T00:45:06.135 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:45:06.135 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ln dir/file testdir/hardlink) 2024-03-24T00:45:06.192 DEBUG:teuthology.orchestra.run.smithi067:> stdin-killer -- python3 -c ' 2024-03-24T00:45:06.192 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-03-24T00:45:06.192 DEBUG:teuthology.orchestra.run.smithi067:> import stat 2024-03-24T00:45:06.192 DEBUG:teuthology.orchestra.run.smithi067:> 2024-03-24T00:45:06.192 DEBUG:teuthology.orchestra.run.smithi067:> print(os.stat("/home/ubuntu/cephtest/mnt.0/dir").st_ino) 2024-03-24T00:45:06.192 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-03-24T00:45:06.297 INFO:teuthology.orchestra.run.smithi067.stdout:1099511627776 2024-03-24T00:45:06.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:45:06.306 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:45:06.306 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:45:06.361 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:45:06.361 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:45:06.416 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:45:06.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.557+0000 7f9bb79f1640 1 Processor -- start 2024-03-24T00:45:06.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.557+0000 7f9bb79f1640 1 -- start start 2024-03-24T00:45:06.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.558+0000 7f9bb79f1640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f9bb006cee0 0x7f9bb0099d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:06.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.558+0000 7f9bb79f1640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9bb009a2b0 0x7f9bb00cfd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:06.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.558+0000 7f9bb79f1640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bb00d1930 con 0x7f9bb00d1550 2024-03-24T00:45:06.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.558+0000 7f9bb79f1640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bb009a700 con 0x7f9bb009a2b0 2024-03-24T00:45:06.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.558+0000 7f9bb79f1640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bb006ca50 con 0x7f9bb006cee0 2024-03-24T00:45:06.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.558+0000 7f9bb5f8e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9bb009a2b0 0x7f9bb00cfd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:06.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.558+0000 7f9bb5f8e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9bb009a2b0 0x7f9bb00cfd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32948/0 (socket says 172.21.15.67:32948) 2024-03-24T00:45:06.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.558+0000 7f9bb5f8e640 1 -- 172.21.15.67:0/198638187 learned_addr learned my addr 172.21.15.67:0/198638187 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:06.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.558+0000 7f9bb578d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9bb00d1550 0x7f9bb00b2780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46176/0 (socket says 172.21.15.67:46176) 2024-03-24T00:45:06.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.558+0000 7f9bb4f8c640 1 --2- 172.21.15.67:0/198638187 >> v2:172.21.15.72:3300/0 conn(0x7f9bb006cee0 0x7f9bb0099d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:06.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.558+0000 7f9bb5f8e640 1 -- 172.21.15.67:0/198638187 >> v2:172.21.15.72:3300/0 conn(0x7f9bb006cee0 msgr2=0x7f9bb0099d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:06.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.558+0000 7f9bb5f8e640 1 --2- 172.21.15.67:0/198638187 >> v2:172.21.15.72:3300/0 conn(0x7f9bb006cee0 0x7f9bb0099d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:06.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.558+0000 7f9bb5f8e640 1 -- 172.21.15.67:0/198638187 >> v1:172.21.15.67:6789/0 conn(0x7f9bb00d1550 legacy=0x7f9bb00b2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:06.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.558+0000 7f9bb5f8e640 1 -- 172.21.15.67:0/198638187 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bb006d390 con 0x7f9bb009a2b0 2024-03-24T00:45:06.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.558+0000 7f9ba7fff640 1 -- 172.21.15.67:0/198638187 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3513436141 0 0) 0x7f9bb00d1930 con 0x7f9bb00d1550 2024-03-24T00:45:06.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.558+0000 7f9bb4f8c640 1 --2- 172.21.15.67:0/198638187 >> v2:172.21.15.72:3300/0 conn(0x7f9bb006cee0 0x7f9bb0099d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:45:06.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.559+0000 7f9bb5f8e640 1 --2- 172.21.15.67:0/198638187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bb009a2b0 0x7f9bb00cfd60 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f9ba8009e50 tx=0x7f9ba802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5838884fee19b837 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:06.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.559+0000 7f9ba7fff640 1 -- 172.21.15.67:0/198638187 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba802cb20 con 0x7f9bb009a2b0 2024-03-24T00:45:06.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.559+0000 7f9ba7fff640 1 -- 172.21.15.67:0/198638187 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ba802ccc0 con 0x7f9bb009a2b0 2024-03-24T00:45:06.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.559+0000 7f9ba7fff640 1 -- 172.21.15.67:0/198638187 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba8034610 con 0x7f9bb009a2b0 2024-03-24T00:45:06.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.560+0000 7f9bb79f1640 1 -- 172.21.15.67:0/198638187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bb009a2b0 msgr2=0x7f9bb00cfd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:06.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.560+0000 7f9bb79f1640 1 --2- 172.21.15.67:0/198638187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bb009a2b0 0x7f9bb00cfd60 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f9ba8009e50 tx=0x7f9ba802bf80 comp rx=0 tx=0).stop 2024-03-24T00:45:06.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.560+0000 7f9bb79f1640 1 -- 172.21.15.67:0/198638187 shutdown_connections 2024-03-24T00:45:06.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.560+0000 7f9bb79f1640 1 --2- 172.21.15.67:0/198638187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bb009a2b0 0x7f9bb00cfd60 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:06.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.560+0000 7f9bb79f1640 1 --2- 172.21.15.67:0/198638187 >> v2:172.21.15.72:3300/0 conn(0x7f9bb006cee0 0x7f9bb0099d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:06.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.560+0000 7f9bb79f1640 1 -- 172.21.15.67:0/198638187 >> 172.21.15.67:0/198638187 conn(0x7f9bb00b36e0 msgr2=0x7f9bb00b3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:06.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.560+0000 7f9bb79f1640 1 -- 172.21.15.67:0/198638187 shutdown_connections 2024-03-24T00:45:06.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.560+0000 7f9bb79f1640 1 -- 172.21.15.67:0/198638187 wait complete. 2024-03-24T00:45:06.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.561+0000 7f9bb79f1640 1 Processor -- start 2024-03-24T00:45:06.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.561+0000 7f9bb79f1640 1 -- start start 2024-03-24T00:45:06.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.562+0000 7f9bb79f1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9bb006cee0 0x7f9bb00b4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:06.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.562+0000 7f9bb79f1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bb009a2b0 0x7f9bb00b71a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:06.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.562+0000 7f9bb79f1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9bb00d1550 0x7f9bb0150f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:06.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.562+0000 7f9bb79f1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb00b2520 con 0x7f9bb009a2b0 2024-03-24T00:45:06.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.562+0000 7f9bb79f1640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb00b2240 con 0x7f9bb00d1550 2024-03-24T00:45:06.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.562+0000 7f9bb79f1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb006c400 con 0x7f9bb006cee0 2024-03-24T00:45:06.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.562+0000 7f9bb4f8c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bb009a2b0 0x7f9bb00b71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:06.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.562+0000 7f9bb4f8c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bb009a2b0 0x7f9bb00b71a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32958/0 (socket says 172.21.15.67:32958) 2024-03-24T00:45:06.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.562+0000 7f9bb4f8c640 1 -- 172.21.15.67:0/3816425513 learned_addr learned my addr 172.21.15.67:0/3816425513 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:06.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.562+0000 7f9bb4f8c640 1 -- 172.21.15.67:0/3816425513 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9bb006cee0 msgr2=0x7f9bb00b4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:06.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.562+0000 7f9bb4f8c640 1 --2- 172.21.15.67:0/3816425513 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9bb006cee0 0x7f9bb00b4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:06.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.562+0000 7f9bb4f8c640 1 -- 172.21.15.67:0/3816425513 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9bb00d1550 msgr2=0x7f9bb0150f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:06.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.562+0000 7f9bb4f8c640 1 --2- 172.21.15.67:0/3816425513 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9bb00d1550 0x7f9bb0150f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:06.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.562+0000 7f9bb4f8c640 1 -- 172.21.15.67:0/3816425513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bb01518a0 con 0x7f9bb009a2b0 2024-03-24T00:45:06.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.562+0000 7f9bb4f8c640 1 --2- 172.21.15.67:0/3816425513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bb009a2b0 0x7f9bb00b71a0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f9ba000d7d0 tx=0x7f9ba000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:06.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.562+0000 7f9ba5ffb640 1 -- 172.21.15.67:0/3816425513 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba0004030 con 0x7f9bb009a2b0 2024-03-24T00:45:06.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.562+0000 7f9ba5ffb640 1 -- 172.21.15.67:0/3816425513 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ba00041d0 con 0x7f9bb009a2b0 2024-03-24T00:45:06.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.562+0000 7f9bb79f1640 1 -- 172.21.15.67:0/3816425513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bb00b76e0 con 0x7f9bb009a2b0 2024-03-24T00:45:06.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.562+0000 7f9ba5ffb640 1 -- 172.21.15.67:0/3816425513 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba000b6c0 con 0x7f9bb009a2b0 2024-03-24T00:45:06.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.563+0000 7f9bb79f1640 1 -- 172.21.15.67:0/3816425513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bb00c9ea0 con 0x7f9bb009a2b0 2024-03-24T00:45:06.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.563+0000 7f9bb79f1640 1 -- 172.21.15.67:0/3816425513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bb0161a30 con 0x7f9bb009a2b0 2024-03-24T00:45:06.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.564+0000 7f9ba5ffb640 1 -- 172.21.15.67:0/3816425513 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9ba000b860 con 0x7f9bb009a2b0 2024-03-24T00:45:06.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.564+0000 7f9ba5ffb640 1 --2- 172.21.15.67:0/3816425513 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9b9406bc00 0x7f9b9406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:06.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.564+0000 7f9ba5ffb640 1 -- 172.21.15.67:0/3816425513 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9ba008af10 con 0x7f9bb009a2b0 2024-03-24T00:45:06.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.565+0000 7f9bb578d640 1 --2- 172.21.15.67:0/3816425513 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9b9406bc00 0x7f9b9406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:06.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.565+0000 7f9bb578d640 1 --2- 172.21.15.67:0/3816425513 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9b9406bc00 0x7f9b9406e0c0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f9bac007fd0 tx=0x7f9bac007920 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:06.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.570+0000 7f9ba5ffb640 1 -- 172.21.15.67:0/3816425513 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9bb0161a30 con 0x7f9bb009a2b0 2024-03-24T00:45:06.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.751+0000 7f9bb79f1640 1 -- 172.21.15.67:0/3816425513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9bb004f400 con 0x7f9bb009a2b0 2024-03-24T00:45:06.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.753+0000 7f9ba5ffb640 1 -- 172.21.15.67:0/3816425513 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+16825 (secure 0 0 0) 0x7f9ba00597e0 con 0x7f9bb009a2b0 2024-03-24T00:45:06.754 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:45:06.754 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":89,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:44:56.377082+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:44:53.025965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-03-24T00:44:53.712957+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:45:06.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.755+0000 7f9bb79f1640 1 -- 172.21.15.67:0/3816425513 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9b9406bc00 msgr2=0x7f9b9406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:06.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.755+0000 7f9bb79f1640 1 --2- 172.21.15.67:0/3816425513 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9b9406bc00 0x7f9b9406e0c0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f9bac007fd0 tx=0x7f9bac007920 comp rx=0 tx=0).stop 2024-03-24T00:45:06.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.755+0000 7f9bb79f1640 1 -- 172.21.15.67:0/3816425513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bb009a2b0 msgr2=0x7f9bb00b71a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:06.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.755+0000 7f9bb79f1640 1 --2- 172.21.15.67:0/3816425513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bb009a2b0 0x7f9bb00b71a0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f9ba000d7d0 tx=0x7f9ba000dc90 comp rx=0 tx=0).stop 2024-03-24T00:45:06.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.755+0000 7f9bb79f1640 1 -- 172.21.15.67:0/3816425513 shutdown_connections 2024-03-24T00:45:06.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.755+0000 7f9bb79f1640 1 --2- 172.21.15.67:0/3816425513 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9b9406bc00 0x7f9b9406e0c0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:06.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.755+0000 7f9bb79f1640 1 --2- 172.21.15.67:0/3816425513 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9bb00d1550 0x7f9bb0150f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:06.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.755+0000 7f9bb79f1640 1 --2- 172.21.15.67:0/3816425513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bb009a2b0 0x7f9bb00b71a0 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:06.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.755+0000 7f9bb79f1640 1 --2- 172.21.15.67:0/3816425513 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9bb006cee0 0x7f9bb00b4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:06.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.755+0000 7f9bb79f1640 1 -- 172.21.15.67:0/3816425513 >> 172.21.15.67:0/3816425513 conn(0x7f9bb00b36e0 msgr2=0x7f9bb00ccf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:06.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.755+0000 7f9bb79f1640 1 -- 172.21.15.67:0/3816425513 shutdown_connections 2024-03-24T00:45:06.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.755+0000 7f9bb79f1640 1 -- 172.21.15.67:0/3816425513 wait complete. 2024-03-24T00:45:06.767 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:38:39.364950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:44:53.025965+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:44:53.712957+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:45:06.767 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:45:06.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.875+0000 7f5fc6e27640 1 Processor -- start 2024-03-24T00:45:06.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.875+0000 7f5fc6e27640 1 -- start start 2024-03-24T00:45:06.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.876+0000 7f5fc6e27640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5fc00b2110 0x7f5fc00b2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:06.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.876+0000 7f5fc6e27640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fc00b3ae0 con 0x7f5fc015bc30 2024-03-24T00:45:06.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.876+0000 7f5fc6e27640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fc00b3dc0 con 0x7f5fc015c010 2024-03-24T00:45:06.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.876+0000 7f5fc6e27640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fc00cca60 con 0x7f5fc00b2110 2024-03-24T00:45:06.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.876+0000 7f5fbffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5fc00b2110 0x7f5fc00b2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:06.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.876+0000 7f5fc4bc3640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5fc015bc30 0x7f5fc00b1a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46190/0 (socket says 172.21.15.67:46190) 2024-03-24T00:45:06.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.876+0000 7f5fc4bc3640 1 -- 172.21.15.67:0/3275434276 learned_addr learned my addr 172.21.15.67:0/3275434276 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:45:06.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.876+0000 7f5fbffff640 1 -- 172.21.15.67:0/3275434276 >> v1:172.21.15.72:6789/0 conn(0x7f5fc015c010 legacy=0x7f5fc00b32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:06.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.876+0000 7f5fbffff640 1 -- 172.21.15.67:0/3275434276 >> v1:172.21.15.67:6789/0 conn(0x7f5fc015bc30 legacy=0x7f5fc00b1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:06.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.876+0000 7f5fbf7fe640 1 -- 172.21.15.67:0/3275434276 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2820618756 0 0) 0x7f5fc00b3ae0 con 0x7f5fc015bc30 2024-03-24T00:45:06.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.876+0000 7f5fbffff640 1 -- 172.21.15.67:0/3275434276 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fc00b40a0 con 0x7f5fc00b2110 2024-03-24T00:45:06.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.877+0000 7f5fbffff640 1 --2- 172.21.15.67:0/3275434276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fc00b2110 0x7f5fc00b2d90 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f5fc00ccbe0 tx=0x7f5fac02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5493d56f18f4dc70 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:06.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.877+0000 7f5fbf7fe640 1 -- 172.21.15.67:0/3275434276 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fac035020 con 0x7f5fc00b2110 2024-03-24T00:45:06.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.877+0000 7f5fbf7fe640 1 -- 172.21.15.67:0/3275434276 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fac031830 con 0x7f5fc00b2110 2024-03-24T00:45:06.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.877+0000 7f5fbf7fe640 1 -- 172.21.15.67:0/3275434276 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fac031c00 con 0x7f5fc00b2110 2024-03-24T00:45:06.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.878+0000 7f5fc6e27640 1 -- 172.21.15.67:0/3275434276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fc00b2110 msgr2=0x7f5fc00b2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:06.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.878+0000 7f5fc6e27640 1 --2- 172.21.15.67:0/3275434276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fc00b2110 0x7f5fc00b2d90 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f5fc00ccbe0 tx=0x7f5fac02b8d0 comp rx=0 tx=0).stop 2024-03-24T00:45:06.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.878+0000 7f5fc6e27640 1 -- 172.21.15.67:0/3275434276 shutdown_connections 2024-03-24T00:45:06.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.878+0000 7f5fc6e27640 1 --2- 172.21.15.67:0/3275434276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fc00b2110 0x7f5fc00b2d90 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:06.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.878+0000 7f5fc6e27640 1 -- 172.21.15.67:0/3275434276 >> 172.21.15.67:0/3275434276 conn(0x7f5fc008b180 msgr2=0x7f5fc008b590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:06.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.878+0000 7f5fc6e27640 1 -- 172.21.15.67:0/3275434276 shutdown_connections 2024-03-24T00:45:06.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.878+0000 7f5fc6e27640 1 -- 172.21.15.67:0/3275434276 wait complete. 2024-03-24T00:45:06.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.880+0000 7f5fc6e27640 1 Processor -- start 2024-03-24T00:45:06.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.880+0000 7f5fc6e27640 1 -- start start 2024-03-24T00:45:06.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.881+0000 7f5fc6e27640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fc00b2110 0x7f5fc014e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:06.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.881+0000 7f5fc6e27640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5fc015bc30 0x7f5fc014e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:06.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.881+0000 7f5fc6e27640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fc015c010 0x7f5fc015d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:06.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.881+0000 7f5fc6e27640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc00ce080 con 0x7f5fc015c010 2024-03-24T00:45:06.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.881+0000 7f5fc6e27640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc00cdf00 con 0x7f5fc00b2110 2024-03-24T00:45:06.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.881+0000 7f5fc6e27640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc00ce200 con 0x7f5fc015bc30 2024-03-24T00:45:06.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.881+0000 7f5fc53c4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fc015c010 0x7f5fc015d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:06.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.881+0000 7f5fc53c4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fc015c010 0x7f5fc015d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32982/0 (socket says 172.21.15.67:32982) 2024-03-24T00:45:06.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.881+0000 7f5fc53c4640 1 -- 172.21.15.67:0/4154145518 learned_addr learned my addr 172.21.15.67:0/4154145518 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:06.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.881+0000 7f5fc4bc3640 1 --2- 172.21.15.67:0/4154145518 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fc00b2110 0x7f5fc014e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:06.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.881+0000 7f5fbffff640 1 --2- 172.21.15.67:0/4154145518 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5fc015bc30 0x7f5fc014e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:06.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.881+0000 7f5fc53c4640 1 -- 172.21.15.67:0/4154145518 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5fc015bc30 msgr2=0x7f5fc014e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:06.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.881+0000 7f5fc53c4640 1 --2- 172.21.15.67:0/4154145518 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5fc015bc30 0x7f5fc014e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:06.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.882+0000 7f5fc53c4640 1 -- 172.21.15.67:0/4154145518 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fc00b2110 msgr2=0x7f5fc014e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:06.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.882+0000 7f5fc53c4640 1 --2- 172.21.15.67:0/4154145518 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fc00b2110 0x7f5fc014e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:06.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.882+0000 7f5fc53c4640 1 -- 172.21.15.67:0/4154145518 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fc014ef00 con 0x7f5fc015c010 2024-03-24T00:45:06.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.882+0000 7f5fc53c4640 1 --2- 172.21.15.67:0/4154145518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fc015c010 0x7f5fc015d160 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f5fb800b970 tx=0x7f5fb800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:06.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.882+0000 7f5fbd7fa640 1 -- 172.21.15.67:0/4154145518 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb8019070 con 0x7f5fc015c010 2024-03-24T00:45:06.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.882+0000 7f5fbd7fa640 1 -- 172.21.15.67:0/4154145518 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fb800cb40 con 0x7f5fc015c010 2024-03-24T00:45:06.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.882+0000 7f5fc6e27640 1 -- 172.21.15.67:0/4154145518 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fc00c57d0 con 0x7f5fc015c010 2024-03-24T00:45:06.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.882+0000 7f5fbd7fa640 1 -- 172.21.15.67:0/4154145518 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb8014440 con 0x7f5fc015c010 2024-03-24T00:45:06.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.882+0000 7f5fc6e27640 1 -- 172.21.15.67:0/4154145518 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fc00ae810 con 0x7f5fc015c010 2024-03-24T00:45:06.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.883+0000 7f5fc6e27640 1 -- 172.21.15.67:0/4154145518 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fc004f400 con 0x7f5fc015c010 2024-03-24T00:45:06.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.884+0000 7f5fbd7fa640 1 -- 172.21.15.67:0/4154145518 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5fb801c070 con 0x7f5fc015c010 2024-03-24T00:45:06.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.884+0000 7f5fbd7fa640 1 --2- 172.21.15.67:0/4154145518 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5fa406bbd0 0x7f5fa406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:06.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.884+0000 7f5fbd7fa640 1 -- 172.21.15.67:0/4154145518 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5fb808a7f0 con 0x7f5fc015c010 2024-03-24T00:45:06.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.884+0000 7f5fc4bc3640 1 --2- 172.21.15.67:0/4154145518 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5fa406bbd0 0x7f5fa406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:06.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.885+0000 7f5fc4bc3640 1 --2- 172.21.15.67:0/4154145518 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5fa406bbd0 0x7f5fa406e090 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f5fb4000f60 tx=0x7f5fb4007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:06.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:06.888+0000 7f5fbd7fa640 1 -- 172.21.15.67:0/4154145518 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5fb8058a70 con 0x7f5fc015c010 2024-03-24T00:45:07.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.071+0000 7f5fc6e27640 1 -- 172.21.15.67:0/4154145518 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5fc00b17e0 con 0x7f5fc015c010 2024-03-24T00:45:07.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.071+0000 7f5fbd7fa640 1 -- 172.21.15.67:0/4154145518 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5fb805cb10 con 0x7f5fc015c010 2024-03-24T00:45:07.073 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-24T00:45:07.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.073+0000 7f5fc6e27640 1 -- 172.21.15.67:0/4154145518 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5fa406bbd0 msgr2=0x7f5fa406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:07.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.073+0000 7f5fc6e27640 1 --2- 172.21.15.67:0/4154145518 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5fa406bbd0 0x7f5fa406e090 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f5fb4000f60 tx=0x7f5fb4007d50 comp rx=0 tx=0).stop 2024-03-24T00:45:07.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.073+0000 7f5fc6e27640 1 -- 172.21.15.67:0/4154145518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fc015c010 msgr2=0x7f5fc015d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:07.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.073+0000 7f5fc6e27640 1 --2- 172.21.15.67:0/4154145518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fc015c010 0x7f5fc015d160 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f5fb800b970 tx=0x7f5fb800be30 comp rx=0 tx=0).stop 2024-03-24T00:45:07.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.073+0000 7f5fc6e27640 1 -- 172.21.15.67:0/4154145518 shutdown_connections 2024-03-24T00:45:07.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.073+0000 7f5fc6e27640 1 --2- 172.21.15.67:0/4154145518 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5fa406bbd0 0x7f5fa406e090 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:07.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.073+0000 7f5fc6e27640 1 --2- 172.21.15.67:0/4154145518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fc015c010 0x7f5fc015d160 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:07.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.073+0000 7f5fc6e27640 1 --2- 172.21.15.67:0/4154145518 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5fc015bc30 0x7f5fc014e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:07.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.073+0000 7f5fc6e27640 1 --2- 172.21.15.67:0/4154145518 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fc00b2110 0x7f5fc014e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:07.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.073+0000 7f5fc6e27640 1 -- 172.21.15.67:0/4154145518 >> 172.21.15.67:0/4154145518 conn(0x7f5fc008b180 msgr2=0x7f5fc006fed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:07.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.073+0000 7f5fc6e27640 1 -- 172.21.15.67:0/4154145518 shutdown_connections 2024-03-24T00:45:07.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.073+0000 7f5fc6e27640 1 -- 172.21.15.67:0/4154145518 wait complete. 2024-03-24T00:45:07.085 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:45:07.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.190+0000 7f6f3cd87640 1 Processor -- start 2024-03-24T00:45:07.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.190+0000 7f6f3cd87640 1 -- start start 2024-03-24T00:45:07.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.192+0000 7f6f3cd87640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6f3815deb0 0x7f6f380b9900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:07.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.192+0000 7f6f3cd87640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6f380cf0f0 0x7f6f380b6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:07.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.192+0000 7f6f3cd87640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f380d28e0 con 0x7f6f380d2100 2024-03-24T00:45:07.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.192+0000 7f6f3cd87640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f380d2bc0 con 0x7f6f3815deb0 2024-03-24T00:45:07.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.192+0000 7f6f3cd87640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f380d2ea0 con 0x7f6f380cf0f0 2024-03-24T00:45:07.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.192+0000 7f6f3659c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6f3815deb0 0x7f6f380b9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:07.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.192+0000 7f6f3659c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6f3815deb0 0x7f6f380b9900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32992/0 (socket says 172.21.15.67:32992) 2024-03-24T00:45:07.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.192+0000 7f6f3659c640 1 -- 172.21.15.67:0/1737117957 learned_addr learned my addr 172.21.15.67:0/1737117957 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:07.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.192+0000 7f6f36d9d640 1 --2- 172.21.15.67:0/1737117957 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6f380cf0f0 0x7f6f380b6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:07.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.192+0000 7f6f3659c640 1 -- 172.21.15.67:0/1737117957 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6f380cf0f0 msgr2=0x7f6f380b6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:07.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.192+0000 7f6f3659c640 1 --2- 172.21.15.67:0/1737117957 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6f380cf0f0 0x7f6f380b6dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:07.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.192+0000 7f6f3659c640 1 -- 172.21.15.67:0/1737117957 >> v1:172.21.15.72:6789/0 conn(0x7f6f380d2100 legacy=0x7f6f380ce9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:07.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.192+0000 7f6f3659c640 1 -- 172.21.15.67:0/1737117957 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f380d3180 con 0x7f6f3815deb0 2024-03-24T00:45:07.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.193+0000 7f6f3659c640 1 --2- 172.21.15.67:0/1737117957 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f3815deb0 0x7f6f380b9900 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f6f28009870 tx=0x7f6f2802ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84bdbe70774ef1b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:07.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.193+0000 7f6f3559a640 1 -- 172.21.15.67:0/1737117957 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f2802f840 con 0x7f6f3815deb0 2024-03-24T00:45:07.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.193+0000 7f6f3559a640 1 -- 172.21.15.67:0/1737117957 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f2802f9e0 con 0x7f6f3815deb0 2024-03-24T00:45:07.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.193+0000 7f6f3559a640 1 -- 172.21.15.67:0/1737117957 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f2802fdb0 con 0x7f6f3815deb0 2024-03-24T00:45:07.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.193+0000 7f6f3cd87640 1 -- 172.21.15.67:0/1737117957 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f3815deb0 msgr2=0x7f6f380b9900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:07.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.193+0000 7f6f3cd87640 1 --2- 172.21.15.67:0/1737117957 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f3815deb0 0x7f6f380b9900 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f6f28009870 tx=0x7f6f2802ba20 comp rx=0 tx=0).stop 2024-03-24T00:45:07.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.194+0000 7f6f3cd87640 1 -- 172.21.15.67:0/1737117957 shutdown_connections 2024-03-24T00:45:07.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.194+0000 7f6f3cd87640 1 --2- 172.21.15.67:0/1737117957 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6f380cf0f0 0x7f6f380b6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:07.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.194+0000 7f6f3cd87640 1 --2- 172.21.15.67:0/1737117957 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f3815deb0 0x7f6f380b9900 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:07.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.194+0000 7f6f3cd87640 1 -- 172.21.15.67:0/1737117957 >> 172.21.15.67:0/1737117957 conn(0x7f6f38072b60 msgr2=0x7f6f38072f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:07.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.194+0000 7f6f3cd87640 1 -- 172.21.15.67:0/1737117957 shutdown_connections 2024-03-24T00:45:07.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.194+0000 7f6f3cd87640 1 -- 172.21.15.67:0/1737117957 wait complete. 2024-03-24T00:45:07.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.195+0000 7f6f3cd87640 1 Processor -- start 2024-03-24T00:45:07.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.195+0000 7f6f3cd87640 1 -- start start 2024-03-24T00:45:07.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.196+0000 7f6f3cd87640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f380cf0f0 0x7f6f38141c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:07.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.196+0000 7f6f3cd87640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6f380d2100 0x7f6f380ccf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:07.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.196+0000 7f6f3cd87640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f3815deb0 0x7f6f3814f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:07.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.196+0000 7f6f3cd87640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f380d2560 con 0x7f6f380cf0f0 2024-03-24T00:45:07.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.196+0000 7f6f3cd87640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f3815e290 con 0x7f6f3815deb0 2024-03-24T00:45:07.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.196+0000 7f6f3cd87640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f380d3980 con 0x7f6f380d2100 2024-03-24T00:45:07.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.196+0000 7f6f3659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f380cf0f0 0x7f6f38141c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:07.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.196+0000 7f6f3659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f380cf0f0 0x7f6f38141c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32998/0 (socket says 172.21.15.67:32998) 2024-03-24T00:45:07.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.196+0000 7f6f3659c640 1 -- 172.21.15.67:0/168689646 learned_addr learned my addr 172.21.15.67:0/168689646 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:07.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.196+0000 7f6f3659c640 1 -- 172.21.15.67:0/168689646 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6f380d2100 msgr2=0x7f6f380ccf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:07.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.196+0000 7f6f3659c640 1 --2- 172.21.15.67:0/168689646 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6f380d2100 0x7f6f380ccf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:07.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.196+0000 7f6f3659c640 1 -- 172.21.15.67:0/168689646 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f3815deb0 msgr2=0x7f6f3814f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:07.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.196+0000 7f6f3659c640 1 --2- 172.21.15.67:0/168689646 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f3815deb0 0x7f6f3814f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:07.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.196+0000 7f6f3659c640 1 -- 172.21.15.67:0/168689646 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f380cbe80 con 0x7f6f380cf0f0 2024-03-24T00:45:07.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.196+0000 7f6f3659c640 1 --2- 172.21.15.67:0/168689646 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f380cf0f0 0x7f6f38141c90 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f6f28000c00 tx=0x7f6f28004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:07.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.196+0000 7f6f2effd640 1 -- 172.21.15.67:0/168689646 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f28037c90 con 0x7f6f380cf0f0 2024-03-24T00:45:07.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.196+0000 7f6f2effd640 1 -- 172.21.15.67:0/168689646 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f28046070 con 0x7f6f380cf0f0 2024-03-24T00:45:07.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.196+0000 7f6f3cd87640 1 -- 172.21.15.67:0/168689646 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f380cc160 con 0x7f6f380cf0f0 2024-03-24T00:45:07.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.196+0000 7f6f2effd640 1 -- 172.21.15.67:0/168689646 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f28041640 con 0x7f6f380cf0f0 2024-03-24T00:45:07.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.196+0000 7f6f3cd87640 1 -- 172.21.15.67:0/168689646 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f380cc720 con 0x7f6f380cf0f0 2024-03-24T00:45:07.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.197+0000 7f6f3cd87640 1 -- 172.21.15.67:0/168689646 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f3804f400 con 0x7f6f380cf0f0 2024-03-24T00:45:07.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.198+0000 7f6f2effd640 1 -- 172.21.15.67:0/168689646 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6f28040070 con 0x7f6f380cf0f0 2024-03-24T00:45:07.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.198+0000 7f6f2effd640 1 --2- 172.21.15.67:0/168689646 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6f1806bbd0 0x7f6f1806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:07.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.198+0000 7f6f2effd640 1 -- 172.21.15.67:0/168689646 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6f280ad250 con 0x7f6f380cf0f0 2024-03-24T00:45:07.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.198+0000 7f6f35d9b640 1 --2- 172.21.15.67:0/168689646 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6f1806bbd0 0x7f6f1806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:07.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.199+0000 7f6f35d9b640 1 --2- 172.21.15.67:0/168689646 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6f1806bbd0 0x7f6f1806e090 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f6f20000f60 tx=0x7f6f200079d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:07.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.203+0000 7f6f2effd640 1 -- 172.21.15.67:0/168689646 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6f28033030 con 0x7f6f380cf0f0 2024-03-24T00:45:07.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.383+0000 7f6f3cd87640 1 -- 172.21.15.67:0/168689646 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6f3806b790 con 0x7f6f380cf0f0 2024-03-24T00:45:07.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.383+0000 7f6f2effd640 1 -- 172.21.15.67:0/168689646 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6f2807bc40 con 0x7f6f380cf0f0 2024-03-24T00:45:07.384 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-24T00:45:07.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.385+0000 7f6f3cd87640 1 -- 172.21.15.67:0/168689646 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6f1806bbd0 msgr2=0x7f6f1806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:07.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.385+0000 7f6f3cd87640 1 --2- 172.21.15.67:0/168689646 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6f1806bbd0 0x7f6f1806e090 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f6f20000f60 tx=0x7f6f200079d0 comp rx=0 tx=0).stop 2024-03-24T00:45:07.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.385+0000 7f6f3cd87640 1 -- 172.21.15.67:0/168689646 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f380cf0f0 msgr2=0x7f6f38141c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:07.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.385+0000 7f6f3cd87640 1 --2- 172.21.15.67:0/168689646 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f380cf0f0 0x7f6f38141c90 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f6f28000c00 tx=0x7f6f28004730 comp rx=0 tx=0).stop 2024-03-24T00:45:07.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.385+0000 7f6f3cd87640 1 -- 172.21.15.67:0/168689646 shutdown_connections 2024-03-24T00:45:07.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.385+0000 7f6f3cd87640 1 --2- 172.21.15.67:0/168689646 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6f1806bbd0 0x7f6f1806e090 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:07.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.385+0000 7f6f3cd87640 1 --2- 172.21.15.67:0/168689646 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f3815deb0 0x7f6f3814f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:07.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.385+0000 7f6f3cd87640 1 --2- 172.21.15.67:0/168689646 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6f380d2100 0x7f6f380ccf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:07.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.385+0000 7f6f3cd87640 1 --2- 172.21.15.67:0/168689646 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f380cf0f0 0x7f6f38141c90 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:07.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.385+0000 7f6f3cd87640 1 -- 172.21.15.67:0/168689646 >> 172.21.15.67:0/168689646 conn(0x7f6f38072b60 msgr2=0x7f6f3815d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:07.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.385+0000 7f6f3cd87640 1 -- 172.21.15.67:0/168689646 shutdown_connections 2024-03-24T00:45:07.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.385+0000 7f6f3cd87640 1 -- 172.21.15.67:0/168689646 wait complete. 2024-03-24T00:45:07.397 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:45:07.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.501+0000 7fa5cc522640 1 Processor -- start 2024-03-24T00:45:07.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.501+0000 7fa5cc522640 1 -- start start 2024-03-24T00:45:07.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.502+0000 7fa5cc522640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa5c40d3ae0 0x7fa5c40b5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:07.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.502+0000 7fa5cc522640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa5c415e280 0x7fa5c40b5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:07.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.502+0000 7fa5cc522640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5c40d2250 con 0x7fa5c415dea0 2024-03-24T00:45:07.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.502+0000 7fa5cc522640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5c40d2530 con 0x7fa5c415e280 2024-03-24T00:45:07.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.502+0000 7fa5cc522640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c40d2810 con 0x7fa5c40d3ae0 2024-03-24T00:45:07.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.502+0000 7fa5ca2be640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa5c415dea0 0x7fa5c40d33d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46196/0 (socket says 172.21.15.67:46196) 2024-03-24T00:45:07.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.502+0000 7fa5ca2be640 1 -- 172.21.15.67:0/574840377 learned_addr learned my addr 172.21.15.67:0/574840377 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:45:07.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.502+0000 7fa5c9abd640 1 --2- 172.21.15.67:0/574840377 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa5c40d3ae0 0x7fa5c40b5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:07.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.502+0000 7fa5caabf640 1 --2- 172.21.15.67:0/574840377 >> v2:172.21.15.72:3300/0 conn(0x7fa5c415e280 0x7fa5c40b5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:07.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.502+0000 7fa5c92bc640 1 -- 172.21.15.67:0/574840377 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 145164282 0 0) 0x7fa5c40d2250 con 0x7fa5c415dea0 2024-03-24T00:45:07.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.502+0000 7fa5c92bc640 1 -- 172.21.15.67:0/574840377 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5b0003520 con 0x7fa5c415dea0 2024-03-24T00:45:07.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.503+0000 7fa5c92bc640 1 -- 172.21.15.67:0/574840377 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 532614578 0 0) 0x7fa5b0003520 con 0x7fa5c415dea0 2024-03-24T00:45:07.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.503+0000 7fa5c9abd640 1 -- 172.21.15.67:0/574840377 >> v2:172.21.15.72:3300/0 conn(0x7fa5c415e280 msgr2=0x7fa5c40b5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:07.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.503+0000 7fa5c9abd640 1 --2- 172.21.15.67:0/574840377 >> v2:172.21.15.72:3300/0 conn(0x7fa5c415e280 0x7fa5c40b5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:07.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.503+0000 7fa5c9abd640 1 -- 172.21.15.67:0/574840377 >> v1:172.21.15.67:6789/0 conn(0x7fa5c415dea0 legacy=0x7fa5c40d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:07.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.503+0000 7fa5c9abd640 1 -- 172.21.15.67:0/574840377 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c40d2af0 con 0x7fa5c40d3ae0 2024-03-24T00:45:07.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.503+0000 7fa5caabf640 1 --2- 172.21.15.67:0/574840377 >> v2:172.21.15.72:3300/0 conn(0x7fa5c415e280 0x7fa5c40b5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:45:07.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.503+0000 7fa5c9abd640 1 --2- 172.21.15.67:0/574840377 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa5c40d3ae0 0x7fa5c40b5a60 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa5c00097a0 tx=0x7fa5c002b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6abfd433050b257c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:07.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.504+0000 7fa5c92bc640 1 -- 172.21.15.67:0/574840377 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c0037070 con 0x7fa5c40d3ae0 2024-03-24T00:45:07.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.504+0000 7fa5c92bc640 1 -- 172.21.15.67:0/574840377 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5c0030510 con 0x7fa5c40d3ae0 2024-03-24T00:45:07.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.504+0000 7fa5c92bc640 1 -- 172.21.15.67:0/574840377 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b4001670 con 0x7fa5c40d3ae0 2024-03-24T00:45:07.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.504+0000 7fa5cc522640 1 -- 172.21.15.67:0/574840377 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa5c40d3ae0 msgr2=0x7fa5c40b5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:07.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.504+0000 7fa5cc522640 1 --2- 172.21.15.67:0/574840377 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa5c40d3ae0 0x7fa5c40b5a60 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa5c00097a0 tx=0x7fa5c002b710 comp rx=0 tx=0).stop 2024-03-24T00:45:07.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.504+0000 7fa5cc522640 1 -- 172.21.15.67:0/574840377 shutdown_connections 2024-03-24T00:45:07.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.504+0000 7fa5cc522640 1 --2- 172.21.15.67:0/574840377 >> v2:172.21.15.72:3300/0 conn(0x7fa5c415e280 0x7fa5c40b5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:07.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.504+0000 7fa5cc522640 1 --2- 172.21.15.67:0/574840377 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa5c40d3ae0 0x7fa5c40b5a60 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:07.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.504+0000 7fa5cc522640 1 -- 172.21.15.67:0/574840377 >> 172.21.15.67:0/574840377 conn(0x7fa5c4072da0 msgr2=0x7fa5c40d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:07.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.504+0000 7fa5cc522640 1 -- 172.21.15.67:0/574840377 shutdown_connections 2024-03-24T00:45:07.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.504+0000 7fa5cc522640 1 -- 172.21.15.67:0/574840377 wait complete. 2024-03-24T00:45:07.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.506+0000 7fa5cc522640 1 Processor -- start 2024-03-24T00:45:07.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.506+0000 7fa5cc522640 1 -- start start 2024-03-24T00:45:07.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.507+0000 7fa5cc522640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa5c40d3ae0 0x7fa5c415ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:07.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.507+0000 7fa5cc522640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5c415dea0 0x7fa5c4161260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:07.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.507+0000 7fa5cc522640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5c415e280 0x7fa5c4150540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:07.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.507+0000 7fa5cc522640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c40b74c0 con 0x7fa5c415e280 2024-03-24T00:45:07.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.507+0000 7fa5cc522640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c40b7340 con 0x7fa5c415dea0 2024-03-24T00:45:07.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.507+0000 7fa5cc522640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c40b7640 con 0x7fa5c40d3ae0 2024-03-24T00:45:07.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.507+0000 7fa5caabf640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5c415e280 0x7fa5c4150540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:07.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.507+0000 7fa5caabf640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5c415e280 0x7fa5c4150540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33010/0 (socket says 172.21.15.67:33010) 2024-03-24T00:45:07.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.507+0000 7fa5caabf640 1 -- 172.21.15.67:0/3640682414 learned_addr learned my addr 172.21.15.67:0/3640682414 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:07.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.507+0000 7fa5ca2be640 1 --2- 172.21.15.67:0/3640682414 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa5c40d3ae0 0x7fa5c415ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:07.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.507+0000 7fa5c9abd640 1 --2- 172.21.15.67:0/3640682414 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5c415dea0 0x7fa5c4161260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:07.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.508+0000 7fa5caabf640 1 -- 172.21.15.67:0/3640682414 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa5c40d3ae0 msgr2=0x7fa5c415ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:07.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.508+0000 7fa5caabf640 1 --2- 172.21.15.67:0/3640682414 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa5c40d3ae0 0x7fa5c415ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:07.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.508+0000 7fa5caabf640 1 -- 172.21.15.67:0/3640682414 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5c415dea0 msgr2=0x7fa5c4161260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:07.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.508+0000 7fa5caabf640 1 --2- 172.21.15.67:0/3640682414 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5c415dea0 0x7fa5c4161260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:07.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.508+0000 7fa5caabf640 1 -- 172.21.15.67:0/3640682414 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c4160cb0 con 0x7fa5c415e280 2024-03-24T00:45:07.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.508+0000 7fa5caabf640 1 --2- 172.21.15.67:0/3640682414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5c415e280 0x7fa5c4150540 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc00efc0 tx=0x7fa5bc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:07.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.508+0000 7fa5baffd640 1 -- 172.21.15.67:0/3640682414 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5bc018030 con 0x7fa5c415e280 2024-03-24T00:45:07.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.508+0000 7fa5baffd640 1 -- 172.21.15.67:0/3640682414 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5bc00cd30 con 0x7fa5c415e280 2024-03-24T00:45:07.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.508+0000 7fa5cc522640 1 -- 172.21.15.67:0/3640682414 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5c40abad0 con 0x7fa5c415e280 2024-03-24T00:45:07.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.508+0000 7fa5baffd640 1 -- 172.21.15.67:0/3640682414 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5bc003bb0 con 0x7fa5c415e280 2024-03-24T00:45:07.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.508+0000 7fa5cc522640 1 -- 172.21.15.67:0/3640682414 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5c4150e90 con 0x7fa5c415e280 2024-03-24T00:45:07.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.509+0000 7fa5cc522640 1 -- 172.21.15.67:0/3640682414 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5c404f400 con 0x7fa5c415e280 2024-03-24T00:45:07.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.509+0000 7fa5baffd640 1 -- 172.21.15.67:0/3640682414 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa5bc01b070 con 0x7fa5c415e280 2024-03-24T00:45:07.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.510+0000 7fa5baffd640 1 --2- 172.21.15.67:0/3640682414 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa5a806bbd0 0x7fa5a806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:07.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.510+0000 7fa5baffd640 1 -- 172.21.15.67:0/3640682414 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa5bc08b3a0 con 0x7fa5c415e280 2024-03-24T00:45:07.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.510+0000 7fa5ca2be640 1 --2- 172.21.15.67:0/3640682414 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa5a806bbd0 0x7fa5a806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:07.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.511+0000 7fa5ca2be640 1 --2- 172.21.15.67:0/3640682414 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa5a806bbd0 0x7fa5a806e090 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fa5b4001eb0 tx=0x7fa5b40016d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:07.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.514+0000 7fa5baffd640 1 -- 172.21.15.67:0/3640682414 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa5bc010040 con 0x7fa5c415e280 2024-03-24T00:45:07.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.695+0000 7fa5cc522640 1 -- 172.21.15.67:0/3640682414 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa5c40cdec0 con 0x7fa5c415e280 2024-03-24T00:45:07.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.695+0000 7fa5baffd640 1 -- 172.21.15.67:0/3640682414 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa5bc0596b0 con 0x7fa5c415e280 2024-03-24T00:45:07.696 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-03-24T00:45:07.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.697+0000 7fa5cc522640 1 -- 172.21.15.67:0/3640682414 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa5a806bbd0 msgr2=0x7fa5a806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:07.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.697+0000 7fa5cc522640 1 --2- 172.21.15.67:0/3640682414 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa5a806bbd0 0x7fa5a806e090 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fa5b4001eb0 tx=0x7fa5b40016d0 comp rx=0 tx=0).stop 2024-03-24T00:45:07.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.697+0000 7fa5cc522640 1 -- 172.21.15.67:0/3640682414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5c415e280 msgr2=0x7fa5c4150540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:07.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.697+0000 7fa5cc522640 1 --2- 172.21.15.67:0/3640682414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5c415e280 0x7fa5c4150540 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc00efc0 tx=0x7fa5bc00c490 comp rx=0 tx=0).stop 2024-03-24T00:45:07.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.697+0000 7fa5cc522640 1 -- 172.21.15.67:0/3640682414 shutdown_connections 2024-03-24T00:45:07.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.697+0000 7fa5cc522640 1 --2- 172.21.15.67:0/3640682414 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa5a806bbd0 0x7fa5a806e090 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:07.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.697+0000 7fa5cc522640 1 --2- 172.21.15.67:0/3640682414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5c415e280 0x7fa5c4150540 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:07.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.697+0000 7fa5cc522640 1 --2- 172.21.15.67:0/3640682414 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5c415dea0 0x7fa5c4161260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:07.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.697+0000 7fa5cc522640 1 --2- 172.21.15.67:0/3640682414 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa5c40d3ae0 0x7fa5c415ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:07.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.697+0000 7fa5cc522640 1 -- 172.21.15.67:0/3640682414 >> 172.21.15.67:0/3640682414 conn(0x7fa5c4072da0 msgr2=0x7fa5c40b4fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:07.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.697+0000 7fa5cc522640 1 -- 172.21.15.67:0/3640682414 shutdown_connections 2024-03-24T00:45:07.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.697+0000 7fa5cc522640 1 -- 172.21.15.67:0/3640682414 wait complete. 2024-03-24T00:45:07.709 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T00:45:07.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.826+0000 7fd63d55b640 1 Processor -- start 2024-03-24T00:45:07.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.826+0000 7fd63d55b640 1 -- start start 2024-03-24T00:45:07.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.826+0000 7fd63d55b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd638144430 0x7fd6380cef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:07.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.827+0000 7fd63d55b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd638072210 0x7fd6380b3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:07.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.827+0000 7fd63d55b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6380b2ad0 con 0x7fd6380cd830 2024-03-24T00:45:07.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.827+0000 7fd63d55b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6380b36e0 con 0x7fd638072210 2024-03-24T00:45:07.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.827+0000 7fd63d55b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6380b2db0 con 0x7fd638144430 2024-03-24T00:45:07.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.827+0000 7fd62ffff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd6380cd830 0x7fd638071b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46204/0 (socket says 172.21.15.67:46204) 2024-03-24T00:45:07.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.827+0000 7fd62ffff640 1 -- 172.21.15.67:0/279146384 learned_addr learned my addr 172.21.15.67:0/279146384 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:45:07.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.827+0000 7fd636ffd640 1 --2- 172.21.15.67:0/279146384 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd638144430 0x7fd6380cef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:07.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.827+0000 7fd6377fe640 1 --2- 172.21.15.67:0/279146384 >> v2:172.21.15.72:3300/0 conn(0x7fd638072210 0x7fd6380b3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:07.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.827+0000 7fd6367fc640 1 -- 172.21.15.67:0/279146384 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 414350350 0 0) 0x7fd6380b2ad0 con 0x7fd6380cd830 2024-03-24T00:45:07.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.827+0000 7fd6367fc640 1 -- 172.21.15.67:0/279146384 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd624003520 con 0x7fd6380cd830 2024-03-24T00:45:07.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.827+0000 7fd6367fc640 1 -- 172.21.15.67:0/279146384 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 54456454 0 0) 0x7fd624003520 con 0x7fd6380cd830 2024-03-24T00:45:07.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.827+0000 7fd6367fc640 1 -- 172.21.15.67:0/279146384 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd638144430 msgr2=0x7fd6380cef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:07.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.827+0000 7fd6367fc640 1 --2- 172.21.15.67:0/279146384 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd638144430 0x7fd6380cef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:07.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.827+0000 7fd6367fc640 1 -- 172.21.15.67:0/279146384 >> v2:172.21.15.72:3300/0 conn(0x7fd638072210 msgr2=0x7fd6380b3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:07.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.827+0000 7fd6367fc640 1 --2- 172.21.15.67:0/279146384 >> v2:172.21.15.72:3300/0 conn(0x7fd638072210 0x7fd6380b3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:07.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.827+0000 7fd6367fc640 1 -- 172.21.15.67:0/279146384 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6380b3090 con 0x7fd6380cd830 2024-03-24T00:45:07.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.827+0000 7fd6367fc640 1 -- 172.21.15.67:0/279146384 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7fd620001670 con 0x7fd6380cd830 2024-03-24T00:45:07.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.827+0000 7fd6367fc640 1 -- 172.21.15.67:0/279146384 >> v1:172.21.15.67:6789/0 conn(0x7fd6380cd830 legacy=0x7fd638071b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:07.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.827+0000 7fd6367fc640 1 --2- 172.21.15.67:0/279146384 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd6240039b0 0x7fd624003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:07.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.827+0000 7fd6367fc640 1 --2- 172.21.15.67:0/279146384 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd638144430 0x7fd624004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:07.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.827+0000 7fd6367fc640 1 --2- 172.21.15.67:0/279146384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd624004d40 0x7fd6240051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:07.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.827+0000 7fd6367fc640 1 -- 172.21.15.67:0/279146384 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6380b36e0 con 0x7fd624004d40 2024-03-24T00:45:07.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.827+0000 7fd6367fc640 1 -- 172.21.15.67:0/279146384 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6380b2db0 con 0x7fd6240039b0 2024-03-24T00:45:07.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.827+0000 7fd6367fc640 1 -- 172.21.15.67:0/279146384 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd62400b9c0 con 0x7fd638144430 2024-03-24T00:45:07.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.828+0000 7fd6377fe640 1 --2- 172.21.15.67:0/279146384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd624004d40 0x7fd6240051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:07.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.828+0000 7fd636ffd640 1 --2- 172.21.15.67:0/279146384 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd6240039b0 0x7fd624003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:07.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.828+0000 7fd62ffff640 1 --2- 172.21.15.67:0/279146384 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd638144430 0x7fd624004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:07.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.828+0000 7fd636ffd640 1 -- 172.21.15.67:0/279146384 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd638144430 msgr2=0x7fd624004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:07.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.828+0000 7fd636ffd640 1 --2- 172.21.15.67:0/279146384 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd638144430 0x7fd624004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:07.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.828+0000 7fd636ffd640 1 -- 172.21.15.67:0/279146384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd624004d40 msgr2=0x7fd6240051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:07.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.828+0000 7fd636ffd640 1 --2- 172.21.15.67:0/279146384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd624004d40 0x7fd6240051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:07.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.828+0000 7fd636ffd640 1 -- 172.21.15.67:0/279146384 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd624003520 con 0x7fd6240039b0 2024-03-24T00:45:07.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.828+0000 7fd62ffff640 1 --2- 172.21.15.67:0/279146384 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd638144430 0x7fd624004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:45:07.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.829+0000 7fd636ffd640 1 --2- 172.21.15.67:0/279146384 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd6240039b0 0x7fd624003dd0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fd628009ac0 tx=0x7fd624005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83a107f773a2469b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:07.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.829+0000 7fd6367fc640 1 -- 172.21.15.67:0/279146384 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd628039070 con 0x7fd6240039b0 2024-03-24T00:45:07.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.829+0000 7fd6367fc640 1 -- 172.21.15.67:0/279146384 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd62802c6b0 con 0x7fd6240039b0 2024-03-24T00:45:07.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.829+0000 7fd6367fc640 1 -- 172.21.15.67:0/279146384 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd628009690 con 0x7fd6240039b0 2024-03-24T00:45:07.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.829+0000 7fd63d55b640 1 -- 172.21.15.67:0/279146384 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd6240039b0 msgr2=0x7fd624003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:07.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.829+0000 7fd63d55b640 1 --2- 172.21.15.67:0/279146384 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd6240039b0 0x7fd624003dd0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fd628009ac0 tx=0x7fd624005830 comp rx=0 tx=0).stop 2024-03-24T00:45:07.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.829+0000 7fd6377fe640 1 -- 172.21.15.67:0/279146384 reap_dead start 2024-03-24T00:45:07.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.829+0000 7fd63d55b640 1 -- 172.21.15.67:0/279146384 shutdown_connections 2024-03-24T00:45:07.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.829+0000 7fd63d55b640 1 -- 172.21.15.67:0/279146384 >> 172.21.15.67:0/279146384 conn(0x7fd638067470 msgr2=0x7fd63806b040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:07.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.830+0000 7fd63d55b640 1 -- 172.21.15.67:0/279146384 shutdown_connections 2024-03-24T00:45:07.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.830+0000 7fd63d55b640 1 -- 172.21.15.67:0/279146384 wait complete. 2024-03-24T00:45:07.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.831+0000 7fd63d55b640 1 Processor -- start 2024-03-24T00:45:07.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.831+0000 7fd63d55b640 1 -- start start 2024-03-24T00:45:07.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.831+0000 7fd63d55b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd6240039b0 0x7fd638142d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:07.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.831+0000 7fd63d55b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd6381eff20 0x7fd638141d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:07.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.831+0000 7fd63d55b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd638142240 0x7fd6381f2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:07.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.831+0000 7fd63d55b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd63815b110 con 0x7fd638142240 2024-03-24T00:45:07.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.831+0000 7fd63d55b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd638144810 con 0x7fd6240039b0 2024-03-24T00:45:07.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.831+0000 7fd63d55b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd63815b290 con 0x7fd6381eff20 2024-03-24T00:45:07.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.832+0000 7fd6377fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd638142240 0x7fd6381f2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:07.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.832+0000 7fd6377fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd638142240 0x7fd6381f2ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33026/0 (socket says 172.21.15.67:33026) 2024-03-24T00:45:07.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.832+0000 7fd6377fe640 1 -- 172.21.15.67:0/4055021521 learned_addr learned my addr 172.21.15.67:0/4055021521 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:07.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.832+0000 7fd636ffd640 1 --2- 172.21.15.67:0/4055021521 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd6240039b0 0x7fd638142d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:07.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.832+0000 7fd62ffff640 1 --2- 172.21.15.67:0/4055021521 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd6381eff20 0x7fd638141d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:07.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.832+0000 7fd6377fe640 1 -- 172.21.15.67:0/4055021521 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd6381eff20 msgr2=0x7fd638141d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:07.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.832+0000 7fd6377fe640 1 --2- 172.21.15.67:0/4055021521 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd6381eff20 0x7fd638141d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:07.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.832+0000 7fd6377fe640 1 -- 172.21.15.67:0/4055021521 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd6240039b0 msgr2=0x7fd638142d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:07.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.832+0000 7fd6377fe640 1 --2- 172.21.15.67:0/4055021521 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd6240039b0 0x7fd638142d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:07.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.832+0000 7fd6377fe640 1 -- 172.21.15.67:0/4055021521 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6381f03a0 con 0x7fd638142240 2024-03-24T00:45:07.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.832+0000 7fd62ffff640 1 --2- 172.21.15.67:0/4055021521 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd6381eff20 0x7fd638141d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:45:07.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.832+0000 7fd6377fe640 1 --2- 172.21.15.67:0/4055021521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd638142240 0x7fd6381f2ac0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fd63806f770 tx=0x7fd63000f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:07.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.832+0000 7fd62f7fe640 1 -- 172.21.15.67:0/4055021521 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd620007d30 con 0x7fd638142240 2024-03-24T00:45:07.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.832+0000 7fd62f7fe640 1 -- 172.21.15.67:0/4055021521 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd63001d070 con 0x7fd638142240 2024-03-24T00:45:07.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.832+0000 7fd62f7fe640 1 -- 172.21.15.67:0/4055021521 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd630015430 con 0x7fd638142240 2024-03-24T00:45:07.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.832+0000 7fd63d55b640 1 -- 172.21.15.67:0/4055021521 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd63815db30 con 0x7fd638142240 2024-03-24T00:45:07.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.832+0000 7fd63d55b640 1 -- 172.21.15.67:0/4055021521 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd63815ddf0 con 0x7fd638142240 2024-03-24T00:45:07.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.833+0000 7fd63d55b640 1 -- 172.21.15.67:0/4055021521 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd63804f400 con 0x7fd638142240 2024-03-24T00:45:07.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.834+0000 7fd62f7fe640 1 -- 172.21.15.67:0/4055021521 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd6300155d0 con 0x7fd638142240 2024-03-24T00:45:07.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.834+0000 7fd62f7fe640 1 --2- 172.21.15.67:0/4055021521 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd61806bc00 0x7fd61806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:07.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.834+0000 7fd62f7fe640 1 -- 172.21.15.67:0/4055021521 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd63008b170 con 0x7fd638142240 2024-03-24T00:45:07.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.834+0000 7fd636ffd640 1 --2- 172.21.15.67:0/4055021521 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd61806bc00 0x7fd61806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:07.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.835+0000 7fd636ffd640 1 --2- 172.21.15.67:0/4055021521 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd61806bc00 0x7fd61806e0c0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fd628036000 tx=0x7fd628030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:07.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:07.838+0000 7fd62f7fe640 1 -- 172.21.15.67:0/4055021521 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd630010040 con 0x7fd638142240 2024-03-24T00:45:08.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.144+0000 7fd63d55b640 1 -- 172.21.15.67:0/4055021521 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd638142ac0 con 0x7fd638142240 2024-03-24T00:45:08.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.144+0000 7fd62f7fe640 1 -- 172.21.15.67:0/4055021521 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v89) v1 ==== 80+0+0 (secure 0 0 0) 0x7fd630059ad0 con 0x7fd638142240 2024-03-24T00:45:08.145 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-03-24T00:45:08.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.147+0000 7fd63d55b640 1 -- 172.21.15.67:0/4055021521 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd61806bc00 msgr2=0x7fd61806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:08.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.147+0000 7fd63d55b640 1 --2- 172.21.15.67:0/4055021521 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd61806bc00 0x7fd61806e0c0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fd628036000 tx=0x7fd628030000 comp rx=0 tx=0).stop 2024-03-24T00:45:08.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.147+0000 7fd63d55b640 1 -- 172.21.15.67:0/4055021521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd638142240 msgr2=0x7fd6381f2ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:08.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.147+0000 7fd63d55b640 1 --2- 172.21.15.67:0/4055021521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd638142240 0x7fd6381f2ac0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fd63806f770 tx=0x7fd63000f580 comp rx=0 tx=0).stop 2024-03-24T00:45:08.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.147+0000 7fd63d55b640 1 -- 172.21.15.67:0/4055021521 shutdown_connections 2024-03-24T00:45:08.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.147+0000 7fd63d55b640 1 --2- 172.21.15.67:0/4055021521 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd61806bc00 0x7fd61806e0c0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:08.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.147+0000 7fd63d55b640 1 --2- 172.21.15.67:0/4055021521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd638142240 0x7fd6381f2ac0 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:08.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.147+0000 7fd63d55b640 1 --2- 172.21.15.67:0/4055021521 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd6381eff20 0x7fd638141d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:08.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.147+0000 7fd63d55b640 1 --2- 172.21.15.67:0/4055021521 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd6240039b0 0x7fd638142d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:08.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.147+0000 7fd63d55b640 1 -- 172.21.15.67:0/4055021521 >> 172.21.15.67:0/4055021521 conn(0x7fd638067470 msgr2=0x7fd6380af740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:08.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.147+0000 7fd63d55b640 1 -- 172.21.15.67:0/4055021521 shutdown_connections 2024-03-24T00:45:08.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.147+0000 7fd63d55b640 1 -- 172.21.15.67:0/4055021521 wait complete. 2024-03-24T00:45:08.158 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-24T00:45:08.158 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:45:08.158 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-24T00:45:08.210 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-03-24T00:45:08.210 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:45:08.210 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-24T00:45:08.238 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:45:08.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.379+0000 7f32ae2d1640 1 Processor -- start 2024-03-24T00:45:08.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.379+0000 7f32ae2d1640 1 -- start start 2024-03-24T00:45:08.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.380+0000 7f32ae2d1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f32a815bf70 0x7f32a80cc690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:08.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.380+0000 7f32ae2d1640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f32a80ce930 0x7f32a80b1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:08.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.380+0000 7f32ae2d1640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32a80ccfe0 con 0x7f32a80cfc70 2024-03-24T00:45:08.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.380+0000 7f32ae2d1640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32a80cd2c0 con 0x7f32a80ce930 2024-03-24T00:45:08.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.380+0000 7f32ae2d1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32a80cd5a0 con 0x7f32a815bf70 2024-03-24T00:45:08.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.380+0000 7f32ac86e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f32a80ce930 0x7f32a80b1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:08.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.380+0000 7f32a7fff640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f32a815bf70 0x7f32a80cc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:08.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.380+0000 7f32a77fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f32a80cfc70 0x7f32a80ce220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:60154/0 (socket says 172.21.15.67:60154) 2024-03-24T00:45:08.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.380+0000 7f32a7fff640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f32a815bf70 0x7f32a80cc690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.67:59868/0 (socket says 172.21.15.67:59868) 2024-03-24T00:45:08.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.380+0000 7f32ac86e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f32a80ce930 0x7f32a80b1cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:56106/0 (socket says 172.21.15.67:56106) 2024-03-24T00:45:08.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.380+0000 7f32a7fff640 1 -- 172.21.15.67:0/2619937341 learned_addr learned my addr 172.21.15.67:0/2619937341 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:08.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.380+0000 7f32ac86e640 1 -- 172.21.15.67:0/2619937341 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f32a815bf70 msgr2=0x7f32a80cc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:08.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.380+0000 7f32ac86e640 1 --2- 172.21.15.67:0/2619937341 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f32a815bf70 0x7f32a80cc690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:08.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.380+0000 7f32ac86e640 1 -- 172.21.15.67:0/2619937341 >> v1:172.21.15.72:6789/0 conn(0x7f32a80cfc70 legacy=0x7f32a80ce220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:08.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.381+0000 7f32ac86e640 1 -- 172.21.15.67:0/2619937341 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a80cd880 con 0x7f32a80ce930 2024-03-24T00:45:08.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.381+0000 7f32ac86e640 1 --2- 172.21.15.67:0/2619937341 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32a80ce930 0x7f32a80b1cd0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f32a0009d90 tx=0x7f32a002bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=188fced8306aeeee server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:08.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.381+0000 7f32a6ffd640 1 -- 172.21.15.67:0/2619937341 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a0039070 con 0x7f32a80ce930 2024-03-24T00:45:08.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.382+0000 7f32a6ffd640 1 -- 172.21.15.67:0/2619937341 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32a002cc50 con 0x7f32a80ce930 2024-03-24T00:45:08.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.382+0000 7f32a6ffd640 1 -- 172.21.15.67:0/2619937341 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a0034440 con 0x7f32a80ce930 2024-03-24T00:45:08.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.382+0000 7f32ae2d1640 1 -- 172.21.15.67:0/2619937341 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32a80ce930 msgr2=0x7f32a80b1cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:08.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.382+0000 7f32ae2d1640 1 --2- 172.21.15.67:0/2619937341 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32a80ce930 0x7f32a80b1cd0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f32a0009d90 tx=0x7f32a002bea0 comp rx=0 tx=0).stop 2024-03-24T00:45:08.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.382+0000 7f32ae2d1640 1 -- 172.21.15.67:0/2619937341 shutdown_connections 2024-03-24T00:45:08.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.382+0000 7f32ae2d1640 1 --2- 172.21.15.67:0/2619937341 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32a80ce930 0x7f32a80b1cd0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:08.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.382+0000 7f32ae2d1640 1 --2- 172.21.15.67:0/2619937341 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f32a815bf70 0x7f32a80cc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:08.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.382+0000 7f32ae2d1640 1 -- 172.21.15.67:0/2619937341 >> 172.21.15.67:0/2619937341 conn(0x7f32a80d1bf0 msgr2=0x7f32a80d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:08.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.382+0000 7f32ae2d1640 1 -- 172.21.15.67:0/2619937341 shutdown_connections 2024-03-24T00:45:08.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.382+0000 7f32ae2d1640 1 -- 172.21.15.67:0/2619937341 wait complete. 2024-03-24T00:45:08.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.383+0000 7f32ae2d1640 1 Processor -- start 2024-03-24T00:45:08.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.383+0000 7f32ae2d1640 1 -- start start 2024-03-24T00:45:08.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.384+0000 7f32ae2d1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f32a80ce930 0x7f32a815aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:08.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.384+0000 7f32ae2d1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32a80cfc70 0x7f32a806c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:08.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.384+0000 7f32ae2d1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32a815bf70 0x7f32a815e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:08.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.384+0000 7f32ae2d1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a80ccd50 con 0x7f32a80cfc70 2024-03-24T00:45:08.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.384+0000 7f32ae2d1640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a80ccbd0 con 0x7f32a815bf70 2024-03-24T00:45:08.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.384+0000 7f32ae2d1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32a80b2210 con 0x7f32a80ce930 2024-03-24T00:45:08.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.384+0000 7f32a77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32a80cfc70 0x7f32a806c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:08.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.384+0000 7f32a77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32a80cfc70 0x7f32a806c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33038/0 (socket says 172.21.15.67:33038) 2024-03-24T00:45:08.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.384+0000 7f32a77fe640 1 -- 172.21.15.67:0/2980420360 learned_addr learned my addr 172.21.15.67:0/2980420360 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:08.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.384+0000 7f32ac86e640 1 --2- 172.21.15.67:0/2980420360 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32a815bf70 0x7f32a815e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:08.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.384+0000 7f32a7fff640 1 --2- 172.21.15.67:0/2980420360 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f32a80ce930 0x7f32a815aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:08.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.385+0000 7f32ac86e640 1 -- 172.21.15.67:0/2980420360 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f32a80ce930 msgr2=0x7f32a815aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:08.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.385+0000 7f32ac86e640 1 --2- 172.21.15.67:0/2980420360 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f32a80ce930 0x7f32a815aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:08.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.385+0000 7f32ac86e640 1 -- 172.21.15.67:0/2980420360 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32a80cfc70 msgr2=0x7f32a806c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:08.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.385+0000 7f32ac86e640 1 --2- 172.21.15.67:0/2980420360 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32a80cfc70 0x7f32a806c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:08.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.385+0000 7f32ac86e640 1 -- 172.21.15.67:0/2980420360 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a80b4ce0 con 0x7f32a815bf70 2024-03-24T00:45:08.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.385+0000 7f32ac86e640 1 --2- 172.21.15.67:0/2980420360 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32a815bf70 0x7f32a815e7e0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f32a0033000 tx=0x7f32a0004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:08.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.385+0000 7f32a4ff9640 1 -- 172.21.15.67:0/2980420360 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a0039050 con 0x7f32a815bf70 2024-03-24T00:45:08.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.385+0000 7f32a4ff9640 1 -- 172.21.15.67:0/2980420360 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32a0045070 con 0x7f32a815bf70 2024-03-24T00:45:08.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.385+0000 7f32ae2d1640 1 -- 172.21.15.67:0/2980420360 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32a80b4fc0 con 0x7f32a815bf70 2024-03-24T00:45:08.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.385+0000 7f32a4ff9640 1 -- 172.21.15.67:0/2980420360 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a0009450 con 0x7f32a815bf70 2024-03-24T00:45:08.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.385+0000 7f32ae2d1640 1 -- 172.21.15.67:0/2980420360 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32a815b3e0 con 0x7f32a815bf70 2024-03-24T00:45:08.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.386+0000 7f32ae2d1640 1 -- 172.21.15.67:0/2980420360 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32a80b5190 con 0x7f32a815bf70 2024-03-24T00:45:08.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.387+0000 7f32a4ff9640 1 -- 172.21.15.67:0/2980420360 <== mon.1 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f32a003e610 con 0x7f32a815bf70 2024-03-24T00:45:08.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.387+0000 7f32a4ff9640 1 --2- 172.21.15.67:0/2980420360 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f329406bc00 0x7f329406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:08.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.387+0000 7f32a4ff9640 1 -- 172.21.15.67:0/2980420360 <== mon.1 v2:172.21.15.72:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f32a00ad4a0 con 0x7f32a815bf70 2024-03-24T00:45:08.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.387+0000 7f32a7fff640 1 --2- 172.21.15.67:0/2980420360 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f329406bc00 0x7f329406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:08.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.388+0000 7f32a7fff640 1 --2- 172.21.15.67:0/2980420360 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f329406bc00 0x7f329406e0c0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f3298009b30 tx=0x7f3298009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:08.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.392+0000 7f32a4ff9640 1 -- 172.21.15.67:0/2980420360 <== mon.1 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f32a007b7a0 con 0x7f32a815bf70 2024-03-24T00:45:08.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.613+0000 7f32ae2d1640 1 -- 172.21.15.67:0/2980420360 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f32a804f400 con 0x7f32a815bf70 2024-03-24T00:45:08.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.614+0000 7f32a4ff9640 1 -- 172.21.15.67:0/2980420360 <== mon.1 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f32a007f840 con 0x7f32a815bf70 2024-03-24T00:45:08.615 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:45:08.615 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":54,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6359,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2664460392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2664460392},{"type":"v1","addr":"172.21.15.67:6835","nonce":2664460392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6368,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/3323873506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3323873506},{"type":"v1","addr":"172.21.15.72:6837","nonce":3323873506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":7150,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/2744016217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2744016217},{"type":"v1","addr":"172.21.15.72:6836","nonce":2744016217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:44:54.749656+0000","modified":"2024-03-24T00:44:56.822686+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7617},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7617":{"gid":7617,"name":"c","rank":0,"incarnation":52,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6833/3021622863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3021622863},{"type":"v1","addr":"172.21.15.67:6833","nonce":3021622863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7617,"qdb_cluster":[7617]},"id":6}]} 2024-03-24T00:45:08.616 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 54 2024-03-24T00:45:08.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.616+0000 7f32ae2d1640 1 -- 172.21.15.67:0/2980420360 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f329406bc00 msgr2=0x7f329406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:08.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.616+0000 7f32ae2d1640 1 --2- 172.21.15.67:0/2980420360 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f329406bc00 0x7f329406e0c0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f3298009b30 tx=0x7f3298009240 comp rx=0 tx=0).stop 2024-03-24T00:45:08.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.616+0000 7f32ae2d1640 1 -- 172.21.15.67:0/2980420360 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32a815bf70 msgr2=0x7f32a815e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:08.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.616+0000 7f32ae2d1640 1 --2- 172.21.15.67:0/2980420360 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32a815bf70 0x7f32a815e7e0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f32a0033000 tx=0x7f32a0004a40 comp rx=0 tx=0).stop 2024-03-24T00:45:08.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.616+0000 7f32ae2d1640 1 -- 172.21.15.67:0/2980420360 shutdown_connections 2024-03-24T00:45:08.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.616+0000 7f32ae2d1640 1 --2- 172.21.15.67:0/2980420360 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f329406bc00 0x7f329406e0c0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:08.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.616+0000 7f32ae2d1640 1 --2- 172.21.15.67:0/2980420360 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32a815bf70 0x7f32a815e7e0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:08.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.616+0000 7f32ae2d1640 1 --2- 172.21.15.67:0/2980420360 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32a80cfc70 0x7f32a806c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:08.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.616+0000 7f32ae2d1640 1 --2- 172.21.15.67:0/2980420360 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f32a80ce930 0x7f32a815aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:08.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.616+0000 7f32ae2d1640 1 -- 172.21.15.67:0/2980420360 >> 172.21.15.67:0/2980420360 conn(0x7f32a80d1bf0 msgr2=0x7f32a80d1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:08.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.616+0000 7f32ae2d1640 1 -- 172.21.15.67:0/2980420360 shutdown_connections 2024-03-24T00:45:08.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.616+0000 7f32ae2d1640 1 -- 172.21.15.67:0/2980420360 wait complete. 2024-03-24T00:45:08.628 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-24T00:45:08.755 INFO:teuthology.orchestra.run.smithi067.stdout:{"message":"","return_code":0} 2024-03-24T00:45:08.755 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-24T00:45:08.756 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-24T00:45:08.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.858+0000 7f0946fbc640 1 Processor -- start 2024-03-24T00:45:08.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.858+0000 7f0946fbc640 1 -- start start 2024-03-24T00:45:08.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.859+0000 7f0946fbc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f09400d07a0 0x7f09400cc770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:08.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.859+0000 7f0946fbc640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09400cd160 con 0x7f09400cccb0 2024-03-24T00:45:08.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.859+0000 7f0946fbc640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09400cd430 con 0x7f094015c010 2024-03-24T00:45:08.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.859+0000 7f0946fbc640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09400cfbe0 con 0x7f09400d07a0 2024-03-24T00:45:08.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.859+0000 7f0945559640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f09400cccb0 0x7f09400b2790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46214/0 (socket says 172.21.15.67:46214) 2024-03-24T00:45:08.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.859+0000 7f0945559640 1 -- 172.21.15.67:0/3498650608 learned_addr learned my addr 172.21.15.67:0/3498650608 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:45:08.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.859+0000 7f093ffff640 1 --2- 172.21.15.67:0/3498650608 >> v2:172.21.15.72:3300/0 conn(0x7f09400d07a0 0x7f09400cc770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:08.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.859+0000 7f093f7fe640 1 -- 172.21.15.67:0/3498650608 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1290369366 0 0) 0x7f09400cd160 con 0x7f09400cccb0 2024-03-24T00:45:08.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.859+0000 7f093f7fe640 1 -- 172.21.15.67:0/3498650608 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0930003520 con 0x7f09400cccb0 2024-03-24T00:45:08.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.860+0000 7f093f7fe640 1 -- 172.21.15.67:0/3498650608 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3226306241 0 0) 0x7f09400cd430 con 0x7f094015c010 2024-03-24T00:45:08.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.860+0000 7f093f7fe640 1 -- 172.21.15.67:0/3498650608 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09400cd160 con 0x7f094015c010 2024-03-24T00:45:08.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.860+0000 7f093f7fe640 1 -- 172.21.15.67:0/3498650608 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 543971019 0 0) 0x7f0930003520 con 0x7f09400cccb0 2024-03-24T00:45:08.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.860+0000 7f093f7fe640 1 -- 172.21.15.67:0/3498650608 >> v2:172.21.15.72:3300/0 conn(0x7f09400d07a0 msgr2=0x7f09400cc770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:08.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.860+0000 7f093f7fe640 1 --2- 172.21.15.67:0/3498650608 >> v2:172.21.15.72:3300/0 conn(0x7f09400d07a0 0x7f09400cc770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:08.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.860+0000 7f093f7fe640 1 -- 172.21.15.67:0/3498650608 >> v1:172.21.15.72:6789/0 conn(0x7f094015c010 legacy=0x7f09400d0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:08.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.860+0000 7f093f7fe640 1 -- 172.21.15.67:0/3498650608 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09400cd710 con 0x7f09400cccb0 2024-03-24T00:45:08.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.860+0000 7f093f7fe640 1 -- 172.21.15.67:0/3498650608 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f09380019f0 con 0x7f09400cccb0 2024-03-24T00:45:08.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.860+0000 7f093f7fe640 1 -- 172.21.15.67:0/3498650608 >> v1:172.21.15.67:6789/0 conn(0x7f09400cccb0 legacy=0x7f09400b2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:08.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.860+0000 7f093f7fe640 1 --2- 172.21.15.67:0/3498650608 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0930003b90 0x7f0930003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:08.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.860+0000 7f093f7fe640 1 --2- 172.21.15.67:0/3498650608 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0930004800 0x7f0930004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:08.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.860+0000 7f093f7fe640 1 --2- 172.21.15.67:0/3498650608 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f09300051c0 0x7f093000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:08.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.860+0000 7f093f7fe640 1 -- 172.21.15.67:0/3498650608 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f093000c090 con 0x7f0930003b90 2024-03-24T00:45:08.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.860+0000 7f093f7fe640 1 -- 172.21.15.67:0/3498650608 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f093000c210 con 0x7f0930004800 2024-03-24T00:45:08.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.860+0000 7f093f7fe640 1 -- 172.21.15.67:0/3498650608 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f093000c410 con 0x7f09300051c0 2024-03-24T00:45:08.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.860+0000 7f0944d58640 1 --2- 172.21.15.67:0/3498650608 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0930003b90 0x7f0930003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:08.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.860+0000 7f093ffff640 1 --2- 172.21.15.67:0/3498650608 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0930004800 0x7f0930004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:08.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.860+0000 7f0945559640 1 --2- 172.21.15.67:0/3498650608 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f09300051c0 0x7f093000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:08.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.860+0000 7f0944d58640 1 -- 172.21.15.67:0/3498650608 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f09300051c0 msgr2=0x7f093000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:08.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.860+0000 7f0944d58640 1 --2- 172.21.15.67:0/3498650608 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f09300051c0 0x7f093000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:08.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.860+0000 7f0944d58640 1 -- 172.21.15.67:0/3498650608 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0930004800 msgr2=0x7f0930004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:08.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.860+0000 7f0944d58640 1 --2- 172.21.15.67:0/3498650608 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0930004800 0x7f0930004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:08.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.860+0000 7f0944d58640 1 -- 172.21.15.67:0/3498650608 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0930003520 con 0x7f0930003b90 2024-03-24T00:45:08.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.860+0000 7f0945559640 1 --2- 172.21.15.67:0/3498650608 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f09300051c0 0x7f093000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:45:08.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.860+0000 7f0945559640 1 -- 172.21.15.67:0/3498650608 reap_dead start 2024-03-24T00:45:08.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.861+0000 7f0944d58640 1 --2- 172.21.15.67:0/3498650608 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0930003b90 0x7f0930003f90 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f09340018d0 tx=0x7f0930005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcd97800df480231 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:08.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.861+0000 7f093f7fe640 1 -- 172.21.15.67:0/3498650608 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0934037070 con 0x7f0930003b90 2024-03-24T00:45:08.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.861+0000 7f093f7fe640 1 -- 172.21.15.67:0/3498650608 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09340052d0 con 0x7f0930003b90 2024-03-24T00:45:08.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.861+0000 7f093f7fe640 1 -- 172.21.15.67:0/3498650608 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09340014f0 con 0x7f0930003b90 2024-03-24T00:45:08.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.862+0000 7f0946fbc640 1 -- 172.21.15.67:0/3498650608 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0930003b90 msgr2=0x7f0930003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:08.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.862+0000 7f0946fbc640 1 --2- 172.21.15.67:0/3498650608 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0930003b90 0x7f0930003f90 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f09340018d0 tx=0x7f0930005830 comp rx=0 tx=0).stop 2024-03-24T00:45:08.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.862+0000 7f0946fbc640 1 -- 172.21.15.67:0/3498650608 shutdown_connections 2024-03-24T00:45:08.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.862+0000 7f0946fbc640 1 --2- 172.21.15.67:0/3498650608 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0930003b90 0x7f0930003f90 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:08.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.862+0000 7f0946fbc640 1 -- 172.21.15.67:0/3498650608 >> 172.21.15.67:0/3498650608 conn(0x7f094009aea0 msgr2=0x7f094008b180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:08.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.862+0000 7f0946fbc640 1 -- 172.21.15.67:0/3498650608 shutdown_connections 2024-03-24T00:45:08.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.862+0000 7f0946fbc640 1 -- 172.21.15.67:0/3498650608 wait complete. 2024-03-24T00:45:08.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.863+0000 7f0946fbc640 1 Processor -- start 2024-03-24T00:45:08.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.863+0000 7f0946fbc640 1 -- start start 2024-03-24T00:45:08.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.864+0000 7f0946fbc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0930003b90 0x7f09401e6870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:08.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.864+0000 7f0946fbc640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f094015c3f0 0x7f094015c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:08.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.864+0000 7f0946fbc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09401484f0 0x7f0940148970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:08.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.864+0000 7f0946fbc640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09400b2300 con 0x7f0930003b90 2024-03-24T00:45:08.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.864+0000 7f0946fbc640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09400b2020 con 0x7f09401484f0 2024-03-24T00:45:08.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.864+0000 7f0946fbc640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09400cc770 con 0x7f094015c3f0 2024-03-24T00:45:08.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.864+0000 7f0944d58640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0930003b90 0x7f09401e6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:08.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.864+0000 7f0944d58640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0930003b90 0x7f09401e6870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33056/0 (socket says 172.21.15.67:33056) 2024-03-24T00:45:08.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.864+0000 7f0944d58640 1 -- 172.21.15.67:0/1323819820 learned_addr learned my addr 172.21.15.67:0/1323819820 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:08.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.864+0000 7f0944d58640 1 -- 172.21.15.67:0/1323819820 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f094015c3f0 msgr2=0x7f094015c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:08.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.864+0000 7f0944d58640 1 --2- 172.21.15.67:0/1323819820 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f094015c3f0 0x7f094015c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:08.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.864+0000 7f0944d58640 1 -- 172.21.15.67:0/1323819820 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09401484f0 msgr2=0x7f0940148970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:08.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.864+0000 7f0945559640 1 --2- 172.21.15.67:0/1323819820 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09401484f0 0x7f0940148970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:08.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.864+0000 7f0944d58640 1 --2- 172.21.15.67:0/1323819820 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09401484f0 0x7f0940148970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:08.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.864+0000 7f0944d58640 1 -- 172.21.15.67:0/1323819820 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09401e6fa0 con 0x7f0930003b90 2024-03-24T00:45:08.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.864+0000 7f0944d58640 1 --2- 172.21.15.67:0/1323819820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0930003b90 0x7f09401e6870 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f09340018d0 tx=0x7f09340012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:08.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.864+0000 7f093d7fa640 1 -- 172.21.15.67:0/1323819820 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0934042020 con 0x7f0930003b90 2024-03-24T00:45:08.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.864+0000 7f093d7fa640 1 -- 172.21.15.67:0/1323819820 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0934038040 con 0x7f0930003b90 2024-03-24T00:45:08.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.864+0000 7f0946fbc640 1 -- 172.21.15.67:0/1323819820 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09401e7280 con 0x7f0930003b90 2024-03-24T00:45:08.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.864+0000 7f093d7fa640 1 -- 172.21.15.67:0/1323819820 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0934037070 con 0x7f0930003b90 2024-03-24T00:45:08.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.864+0000 7f0946fbc640 1 -- 172.21.15.67:0/1323819820 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09401e7840 con 0x7f0930003b90 2024-03-24T00:45:08.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.866+0000 7f0946fbc640 1 -- 172.21.15.67:0/1323819820 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f094006e4a0 con 0x7f0930003b90 2024-03-24T00:45:08.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.866+0000 7f093d7fa640 1 -- 172.21.15.67:0/1323819820 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0934045020 con 0x7f0930003b90 2024-03-24T00:45:08.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.866+0000 7f093d7fa640 1 --2- 172.21.15.67:0/1323819820 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f092406bc00 0x7f092406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:08.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.867+0000 7f093d7fa640 1 -- 172.21.15.67:0/1323819820 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f09340ac210 con 0x7f0930003b90 2024-03-24T00:45:08.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.867+0000 7f093ffff640 1 --2- 172.21.15.67:0/1323819820 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f092406bc00 0x7f092406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:08.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.867+0000 7f093ffff640 1 --2- 172.21.15.67:0/1323819820 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f092406bc00 0x7f092406e0c0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f092c006e80 tx=0x7f092c006df0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:08.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:08.875+0000 7f093d7fa640 1 -- 172.21.15.67:0/1323819820 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f093404c030 con 0x7f0930003b90 2024-03-24T00:45:09.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.101+0000 7f0946fbc640 1 -- 172.21.15.67:0/1323819820 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f094004f400 con 0x7f0930003b90 2024-03-24T00:45:09.102 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-24T00:45:09.101+0000 7fbfcfa11640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T00:45:09.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.103+0000 7f093d7fa640 1 -- 172.21.15.67:0/1323819820 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v55) v1 ==== 161+0+0 (secure 0 0 0) 0x7f09340052d0 con 0x7f0930003b90 2024-03-24T00:45:09.103 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-24T00:45:09.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.105+0000 7f0946fbc640 1 -- 172.21.15.67:0/1323819820 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f092406bc00 msgr2=0x7f092406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:09.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.105+0000 7f0946fbc640 1 --2- 172.21.15.67:0/1323819820 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f092406bc00 0x7f092406e0c0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f092c006e80 tx=0x7f092c006df0 comp rx=0 tx=0).stop 2024-03-24T00:45:09.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.105+0000 7f0946fbc640 1 -- 172.21.15.67:0/1323819820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0930003b90 msgr2=0x7f09401e6870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:09.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.105+0000 7f0946fbc640 1 --2- 172.21.15.67:0/1323819820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0930003b90 0x7f09401e6870 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f09340018d0 tx=0x7f09340012d0 comp rx=0 tx=0).stop 2024-03-24T00:45:09.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.106+0000 7f0946fbc640 1 -- 172.21.15.67:0/1323819820 shutdown_connections 2024-03-24T00:45:09.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.106+0000 7f0946fbc640 1 --2- 172.21.15.67:0/1323819820 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f092406bc00 0x7f092406e0c0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:09.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.106+0000 7f0946fbc640 1 --2- 172.21.15.67:0/1323819820 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09401484f0 0x7f0940148970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:09.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.106+0000 7f0946fbc640 1 --2- 172.21.15.67:0/1323819820 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f094015c3f0 0x7f094015c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:09.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.106+0000 7f0946fbc640 1 --2- 172.21.15.67:0/1323819820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0930003b90 0x7f09401e6870 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:09.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.106+0000 7f0946fbc640 1 -- 172.21.15.67:0/1323819820 >> 172.21.15.67:0/1323819820 conn(0x7f094009aea0 msgr2=0x7f09400d2400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:09.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.106+0000 7f0946fbc640 1 -- 172.21.15.67:0/1323819820 shutdown_connections 2024-03-24T00:45:09.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.106+0000 7f0946fbc640 1 -- 172.21.15.67:0/1323819820 wait complete. 2024-03-24T00:45:09.109 INFO:tasks.ceph.mds.c.smithi067.stderr: -5212> 2024-03-24T00:45:01.034+0000 7fcb01199640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T00:45:09.120 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm 10000000000.00000000 2024-03-24T00:45:09.138 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-03-24T00:45:09.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.161+0000 7f04e7042f40 1 Processor -- start 2024-03-24T00:45:09.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.161+0000 7f04e7042f40 1 -- start start 2024-03-24T00:45:09.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.161+0000 7f04e7042f40 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04cc0022a0 0x55af2d4477f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:09.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.161+0000 7f04e7042f40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55af2d446ad0 0x55af2d446f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:09.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.161+0000 7f04e7042f40 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55af2d41b730 0x55af2d41bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:09.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.161+0000 7f04e7042f40 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af2d446650 con 0x7f04cc0022a0 2024-03-24T00:45:09.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.161+0000 7f04e7042f40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af2d3c40d0 con 0x55af2d446ad0 2024-03-24T00:45:09.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.161+0000 7f04e7042f40 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55af2d4f5780 con 0x55af2d41b730 2024-03-24T00:45:09.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.161+0000 7f04e432e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04cc0022a0 0x55af2d4477f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:09.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.161+0000 7f04e432e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04cc0022a0 0x55af2d4477f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33084/0 (socket says 172.21.15.67:33084) 2024-03-24T00:45:09.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.161+0000 7f04e432e640 1 -- 172.21.15.67:0/255219059 learned_addr learned my addr 172.21.15.67:0/255219059 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:09.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.161+0000 7f04e3b2d640 1 --2- 172.21.15.67:0/255219059 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55af2d446ad0 0x55af2d446f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:09.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.161+0000 7f04e4b2f640 1 --2- 172.21.15.67:0/255219059 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55af2d41b730 0x55af2d41bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:09.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.162+0000 7f04e432e640 1 -- 172.21.15.67:0/255219059 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55af2d41b730 msgr2=0x55af2d41bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:09.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.162+0000 7f04e432e640 1 --2- 172.21.15.67:0/255219059 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55af2d41b730 0x55af2d41bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:09.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.162+0000 7f04e432e640 1 -- 172.21.15.67:0/255219059 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55af2d446ad0 msgr2=0x55af2d446f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:09.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.162+0000 7f04e432e640 1 --2- 172.21.15.67:0/255219059 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55af2d446ad0 0x55af2d446f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:09.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.162+0000 7f04e432e640 1 -- 172.21.15.67:0/255219059 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55af2d4f72b0 con 0x7f04cc0022a0 2024-03-24T00:45:09.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.162+0000 7f04e3b2d640 1 --2- 172.21.15.67:0/255219059 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55af2d446ad0 0x55af2d446f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:45:09.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.162+0000 7f04e432e640 1 --2- 172.21.15.67:0/255219059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04cc0022a0 0x55af2d4477f0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f04d8055040 tx=0x7f04d804d070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bcef55e7162a6fe server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:09.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.162+0000 7f04e2b2b640 1 -- 172.21.15.67:0/255219059 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d8064070 con 0x7f04cc0022a0 2024-03-24T00:45:09.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.162+0000 7f04e2b2b640 1 -- 172.21.15.67:0/255219059 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04d804bd60 con 0x7f04cc0022a0 2024-03-24T00:45:09.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.162+0000 7f04e2b2b640 1 -- 172.21.15.67:0/255219059 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d80015d0 con 0x7f04cc0022a0 2024-03-24T00:45:09.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.162+0000 7f04e7042f40 1 -- 172.21.15.67:0/255219059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04cc0022a0 msgr2=0x55af2d4477f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:09.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.162+0000 7f04e7042f40 1 --2- 172.21.15.67:0/255219059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04cc0022a0 0x55af2d4477f0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f04d8055040 tx=0x7f04d804d070 comp rx=0 tx=0).stop 2024-03-24T00:45:09.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.162+0000 7f04e7042f40 1 -- 172.21.15.67:0/255219059 shutdown_connections 2024-03-24T00:45:09.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.162+0000 7f04e7042f40 1 --2- 172.21.15.67:0/255219059 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55af2d41b730 0x55af2d41bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:09.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.162+0000 7f04e7042f40 1 --2- 172.21.15.67:0/255219059 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55af2d446ad0 0x55af2d446f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:09.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.162+0000 7f04e7042f40 1 --2- 172.21.15.67:0/255219059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04cc0022a0 0x55af2d4477f0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:09.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.162+0000 7f04e7042f40 1 -- 172.21.15.67:0/255219059 >> 172.21.15.67:0/255219059 conn(0x55af2d449320 msgr2=0x55af2d44ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:09.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.162+0000 7f04e7042f40 1 -- 172.21.15.67:0/255219059 shutdown_connections 2024-03-24T00:45:09.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.162+0000 7f04e7042f40 1 -- 172.21.15.67:0/255219059 wait complete. 2024-03-24T00:45:09.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.163+0000 7f04e7042f40 1 Processor -- start 2024-03-24T00:45:09.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.163+0000 7f04e7042f40 1 -- start start 2024-03-24T00:45:09.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.164+0000 7f04e7042f40 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55af2d41b730 0x55af2d58e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:09.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.164+0000 7f04e7042f40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55af2d446ad0 0x55af2d4d46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:09.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.164+0000 7f04e7042f40 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04cc0022a0 0x55af2d4f1390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:09.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.164+0000 7f04e7042f40 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af2d41b3a0 con 0x55af2d41b730 2024-03-24T00:45:09.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.164+0000 7f04e7042f40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af2d41b0c0 con 0x55af2d446ad0 2024-03-24T00:45:09.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.164+0000 7f04e7042f40 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55af2d4f2870 con 0x7f04cc0022a0 2024-03-24T00:45:09.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.164+0000 7f04e432e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55af2d41b730 0x55af2d58e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:09.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.164+0000 7f04e432e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55af2d41b730 0x55af2d58e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33092/0 (socket says 172.21.15.67:33092) 2024-03-24T00:45:09.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.164+0000 7f04e432e640 1 -- 172.21.15.67:0/3281315466 learned_addr learned my addr 172.21.15.67:0/3281315466 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:09.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.164+0000 7f04e3b2d640 1 --2- 172.21.15.67:0/3281315466 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55af2d446ad0 0x55af2d4d46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:09.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.164+0000 7f04e4b2f640 1 --2- 172.21.15.67:0/3281315466 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04cc0022a0 0x55af2d4f1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:09.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.164+0000 7f04e432e640 1 -- 172.21.15.67:0/3281315466 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04cc0022a0 msgr2=0x55af2d4f1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:09.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.164+0000 7f04e432e640 1 --2- 172.21.15.67:0/3281315466 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f04cc0022a0 0x55af2d4f1390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:09.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.164+0000 7f04e432e640 1 -- 172.21.15.67:0/3281315466 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55af2d446ad0 msgr2=0x55af2d4d46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:09.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.164+0000 7f04e432e640 1 --2- 172.21.15.67:0/3281315466 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55af2d446ad0 0x55af2d4d46e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:09.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.164+0000 7f04e432e640 1 -- 172.21.15.67:0/3281315466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55af2d41c5c0 con 0x55af2d41b730 2024-03-24T00:45:09.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.164+0000 7f04e432e640 1 --2- 172.21.15.67:0/3281315466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55af2d41b730 0x55af2d58e230 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f04d8001050 tx=0x7f04cc004120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:09.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.164+0000 7f04e1b29640 1 -- 172.21.15.67:0/3281315466 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d8064040 con 0x55af2d41b730 2024-03-24T00:45:09.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.164+0000 7f04e1b29640 1 -- 172.21.15.67:0/3281315466 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04d8070070 con 0x55af2d41b730 2024-03-24T00:45:09.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.164+0000 7f04e7042f40 1 -- 172.21.15.67:0/3281315466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55af2d41bd20 con 0x55af2d41b730 2024-03-24T00:45:09.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.164+0000 7f04e1b29640 1 -- 172.21.15.67:0/3281315466 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d804b4e0 con 0x55af2d41b730 2024-03-24T00:45:09.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.164+0000 7f04e7042f40 1 -- 172.21.15.67:0/3281315466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55af2d41c2e0 con 0x55af2d41b730 2024-03-24T00:45:09.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.166+0000 7f04e1b29640 1 -- 172.21.15.67:0/3281315466 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f04d8051030 con 0x55af2d41b730 2024-03-24T00:45:09.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.166+0000 7f04e1b29640 1 --2- 172.21.15.67:0/3281315466 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f04d006a5b0 0x7f04d006ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:09.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.166+0000 7f04e1b29640 1 -- 172.21.15.67:0/3281315466 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f04d80cd150 con 0x55af2d41b730 2024-03-24T00:45:09.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.166+0000 7f04e3b2d640 1 --2- 172.21.15.67:0/3281315466 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f04d006a5b0 0x7f04d006ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:09.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.166+0000 7f04e7042f40 1 --2- 172.21.15.67:0/3281315466 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x55af2d4d6240 0x55af2d460340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:09.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.166+0000 7f04e7042f40 1 -- 172.21.15.67:0/3281315466 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:1 13.10 13:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e90) v9 -- 0x55af2d4d4c20 con 0x55af2d4d6240 2024-03-24T00:45:09.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.166+0000 7f04e4b2f640 1 --2- 172.21.15.67:0/3281315466 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x55af2d4d6240 0x55af2d460340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:09.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.167+0000 7f04e3b2d640 1 --2- 172.21.15.67:0/3281315466 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f04d006a5b0 0x7f04d006ca70 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x55af2d44ba20 tx=0x7f04d400c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:09.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.167+0000 7f04e4b2f640 1 --2- 172.21.15.67:0/3281315466 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x55af2d4d6240 0x55af2d460340 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:09.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.170+0000 7f04e4b2f640 1 -- 172.21.15.67:0/3281315466 <== osd.1 v2:172.21.15.67:6800/4062510404 1 ==== osd_op_reply(1 10000000000.00000000 [delete] v90'11 uv11 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7f04dc0128e0 con 0x55af2d4d6240 2024-03-24T00:45:09.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.170+0000 7f04e7042f40 1 -- 172.21.15.67:0/3281315466 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x55af2d4d6240 msgr2=0x55af2d460340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:09.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.170+0000 7f04e7042f40 1 --2- 172.21.15.67:0/3281315466 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x55af2d4d6240 0x55af2d460340 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:09.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.170+0000 7f04e7042f40 1 -- 172.21.15.67:0/3281315466 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f04d006a5b0 msgr2=0x7f04d006ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:09.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.170+0000 7f04e7042f40 1 --2- 172.21.15.67:0/3281315466 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f04d006a5b0 0x7f04d006ca70 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x55af2d44ba20 tx=0x7f04d400c040 comp rx=0 tx=0).stop 2024-03-24T00:45:09.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.170+0000 7f04e7042f40 1 -- 172.21.15.67:0/3281315466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55af2d41b730 msgr2=0x55af2d58e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:09.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.170+0000 7f04e7042f40 1 --2- 172.21.15.67:0/3281315466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55af2d41b730 0x55af2d58e230 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f04d8001050 tx=0x7f04cc004120 comp rx=0 tx=0).stop 2024-03-24T00:45:09.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.170+0000 7f04e4b2f640 1 -- 172.21.15.67:0/3281315466 reap_dead start 2024-03-24T00:45:09.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.170+0000 7f04e7042f40 1 -- 172.21.15.67:0/3281315466 shutdown_connections 2024-03-24T00:45:09.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.170+0000 7f04e7042f40 1 -- 172.21.15.67:0/3281315466 >> 172.21.15.67:0/3281315466 conn(0x55af2d449320 msgr2=0x55af2d4f38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:09.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.170+0000 7f04e7042f40 1 -- 172.21.15.67:0/3281315466 shutdown_connections 2024-03-24T00:45:09.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.170+0000 7f04e7042f40 1 -- 172.21.15.67:0/3281315466 wait complete. 2024-03-24T00:45:09.173 DEBUG:teuthology.orchestra.run.smithi067:> cephfs-journal-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 --rank cephfs:0 journal reset 2024-03-24T00:45:09.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.252+0000 7f2c5203a280 1 -- start start 2024-03-24T00:45:09.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.252+0000 7f2c5203a280 1 Processor -- start 2024-03-24T00:45:09.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.253+0000 7f2c5203a280 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5566b5c04c00 0x5566b5cac580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:09.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.253+0000 7f2c5203a280 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5566b5c05800 0x5566b5cacb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:09.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.253+0000 7f2c5203a280 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5566b5c05c00 0x5566b5cad080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:09.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.253+0000 7f2c5203a280 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566b69ca180 con 0x5566b5c05c00 2024-03-24T00:45:09.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.253+0000 7f2c5203a280 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566b69ca300 con 0x5566b5c04c00 2024-03-24T00:45:09.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.253+0000 7f2c5203a280 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5566b69ca480 con 0x5566b5c05800 2024-03-24T00:45:09.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.253+0000 7f2c50827640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5566b5c05c00 0x5566b5cad080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:09.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.253+0000 7f2c50827640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5566b5c05c00 0x5566b5cad080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33104/0 (socket says 172.21.15.67:33104) 2024-03-24T00:45:09.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.253+0000 7f2c50827640 1 -- 172.21.15.67:0/239062179 learned_addr learned my addr 172.21.15.67:0/239062179 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:09.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.253+0000 7f2c4f825640 1 --2- 172.21.15.67:0/239062179 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5566b5c05800 0x5566b5cacb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:09.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.253+0000 7f2c50026640 1 --2- 172.21.15.67:0/239062179 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5566b5c04c00 0x5566b5cac580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:09.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.253+0000 7f2c50827640 1 -- 172.21.15.67:0/239062179 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5566b5c05800 msgr2=0x5566b5cacb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:09.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.253+0000 7f2c50827640 1 --2- 172.21.15.67:0/239062179 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5566b5c05800 0x5566b5cacb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:09.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.253+0000 7f2c50827640 1 -- 172.21.15.67:0/239062179 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5566b5c04c00 msgr2=0x5566b5cac580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:09.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.253+0000 7f2c50827640 1 --2- 172.21.15.67:0/239062179 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5566b5c04c00 0x5566b5cac580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:09.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.253+0000 7f2c50827640 1 -- 172.21.15.67:0/239062179 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5566b5c3a780 con 0x5566b5c05c00 2024-03-24T00:45:09.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.253+0000 7f2c50827640 1 --2- 172.21.15.67:0/239062179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5566b5c05c00 0x5566b5cad080 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x5566b6a25380 tx=0x5566b699f5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8766f31c522bf086 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:09.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.254+0000 7f2c4d821640 1 -- 172.21.15.67:0/239062179 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5566b5c2a680 con 0x5566b5c05c00 2024-03-24T00:45:09.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.254+0000 7f2c4d821640 1 -- 172.21.15.67:0/239062179 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5566b5c30a80 con 0x5566b5c05c00 2024-03-24T00:45:09.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.254+0000 7f2c5203a280 1 -- 172.21.15.67:0/239062179 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5566b5c3a3c0 con 0x5566b5c05c00 2024-03-24T00:45:09.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.254+0000 7f2c5203a280 1 -- 172.21.15.67:0/239062179 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x5566b5c3b680 con 0x5566b5c05c00 2024-03-24T00:45:09.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.254+0000 7f2c5203a280 4 waiting for MDS map... 2024-03-24T00:45:09.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.254+0000 7f2c4d821640 1 -- 172.21.15.67:0/239062179 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5566b5c2ab60 con 0x5566b5c05c00 2024-03-24T00:45:09.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.254+0000 7f2c4d821640 1 -- 172.21.15.67:0/239062179 <== mon.0 v2:172.21.15.67:3300/0 4 ==== osd_map(90..90 src has 1..90) v4 ==== 6187+0+0 (secure 0 0 0) 0x5566b6adc600 con 0x5566b5c05c00 2024-03-24T00:45:09.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.254+0000 7f2c4d821640 1 -- 172.21.15.67:0/239062179 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 55) v1 ==== 2735+0+0 (secure 0 0 0) 0x5566b5c5f180 con 0x5566b5c05c00 2024-03-24T00:45:09.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.254+0000 7f2c5203a280 4 Got MDS map 55 2024-03-24T00:45:09.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.254+0000 7f2c5203a280 10 main: JournalTool::main 2024-03-24T00:45:09.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.254+0000 7f2c5203a280 4 main: JournalTool: connecting to RADOS... 2024-03-24T00:45:09.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.255+0000 7f2c5203a280 1 Processor -- start 2024-03-24T00:45:09.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.255+0000 7f2c5203a280 1 -- start start 2024-03-24T00:45:09.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.255+0000 7f2c5203a280 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5566b69ff400 0x5566b5cad600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:09.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.255+0000 7f2c5203a280 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5566b69ff800 0x5566b5cadb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:09.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.255+0000 7f2c5203a280 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5566b69ffc00 0x5566b5cae100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:09.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.255+0000 7f2c5203a280 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566b69ca600 con 0x5566b69ff400 2024-03-24T00:45:09.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.255+0000 7f2c5203a280 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566b69ca780 con 0x5566b69ffc00 2024-03-24T00:45:09.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.255+0000 7f2c5203a280 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5566b69ca900 con 0x5566b69ff800 2024-03-24T00:45:09.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.255+0000 7f2c4f825640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5566b69ff400 0x5566b5cad600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:09.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.255+0000 7f2c4f825640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5566b69ff400 0x5566b5cad600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33116/0 (socket says 172.21.15.67:33116) 2024-03-24T00:45:09.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.255+0000 7f2c4f825640 1 -- 172.21.15.67:0/2741759084 learned_addr learned my addr 172.21.15.67:0/2741759084 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:09.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.255+0000 7f2c50026640 1 --2- 172.21.15.67:0/2741759084 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5566b69ff800 0x5566b5cadb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:09.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.255+0000 7f2c4f825640 1 --2- 172.21.15.67:0/2741759084 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5566b69ffc00 0x5566b5cae100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:09.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.255+0000 7f2c4f825640 1 -- 172.21.15.67:0/2741759084 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5566b69ff800 msgr2=0x5566b5cadb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:09.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.255+0000 7f2c4f825640 1 --2- 172.21.15.67:0/2741759084 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5566b69ff800 0x5566b5cadb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:09.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.255+0000 7f2c4f825640 1 -- 172.21.15.67:0/2741759084 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5566b69ffc00 msgr2=0x5566b5cae100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:09.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.255+0000 7f2c4f825640 1 --2- 172.21.15.67:0/2741759084 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5566b69ffc00 0x5566b5cae100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:09.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.255+0000 7f2c4f825640 1 -- 172.21.15.67:0/2741759084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5566b5c3b4a0 con 0x5566b69ff400 2024-03-24T00:45:09.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.256+0000 7f2c4f825640 1 --2- 172.21.15.67:0/2741759084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5566b69ff400 0x5566b5cad600 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x5566b6ac4690 tx=0x5566b6ac7860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd42e7d691fd2a8a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:09.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.256+0000 7f2c4b81d640 1 -- 172.21.15.67:0/2741759084 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5566b5c2a680 con 0x5566b69ff400 2024-03-24T00:45:09.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.256+0000 7f2c4b81d640 1 -- 172.21.15.67:0/2741759084 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5566b5c30fc0 con 0x5566b69ff400 2024-03-24T00:45:09.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.256+0000 7f2c4b81d640 1 -- 172.21.15.67:0/2741759084 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5566b5c2ab60 con 0x5566b69ff400 2024-03-24T00:45:09.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.256+0000 7f2c5203a280 1 -- 172.21.15.67:0/2741759084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5566b69ff400 msgr2=0x5566b5cad600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:09.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.256+0000 7f2c5203a280 1 --2- 172.21.15.67:0/2741759084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5566b69ff400 0x5566b5cad600 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x5566b6ac4690 tx=0x5566b6ac7860 comp rx=0 tx=0).stop 2024-03-24T00:45:09.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.256+0000 7f2c5203a280 1 -- 172.21.15.67:0/2741759084 shutdown_connections 2024-03-24T00:45:09.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.256+0000 7f2c5203a280 1 --2- 172.21.15.67:0/2741759084 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5566b69ffc00 0x5566b5cae100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:09.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.256+0000 7f2c5203a280 1 --2- 172.21.15.67:0/2741759084 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5566b69ff800 0x5566b5cadb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:09.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.256+0000 7f2c5203a280 1 --2- 172.21.15.67:0/2741759084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5566b69ff400 0x5566b5cad600 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:09.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.256+0000 7f2c5203a280 1 -- 172.21.15.67:0/2741759084 >> 172.21.15.67:0/2741759084 conn(0x5566b69ff000 msgr2=0x5566b6989000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:09.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.256+0000 7f2c5203a280 1 -- 172.21.15.67:0/2741759084 shutdown_connections 2024-03-24T00:45:09.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.256+0000 7f2c5203a280 1 -- 172.21.15.67:0/2741759084 wait complete. 2024-03-24T00:45:09.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.257+0000 7f2c5203a280 1 Processor -- start 2024-03-24T00:45:09.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.257+0000 7f2c5203a280 1 -- start start 2024-03-24T00:45:09.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.257+0000 7f2c5203a280 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5566b69ff400 0x5566b5caf180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:09.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.257+0000 7f2c5203a280 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5566b69ff800 0x5566b5caec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:09.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.257+0000 7f2c5203a280 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5566b69ffc00 0x5566b5cae680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:09.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.257+0000 7f2c5203a280 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566b69cbb00 con 0x5566b69ff800 2024-03-24T00:45:09.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.257+0000 7f2c5203a280 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566b69cb980 con 0x5566b69ff400 2024-03-24T00:45:09.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.257+0000 7f2c5203a280 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5566b69cb800 con 0x5566b69ffc00 2024-03-24T00:45:09.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.257+0000 7f2c50026640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5566b69ff800 0x5566b5caec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:09.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.257+0000 7f2c50026640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5566b69ff800 0x5566b5caec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33126/0 (socket says 172.21.15.67:33126) 2024-03-24T00:45:09.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.258+0000 7f2c50026640 1 -- 172.21.15.67:0/3774722021 learned_addr learned my addr 172.21.15.67:0/3774722021 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:09.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.258+0000 7f2c4f825640 1 --2- 172.21.15.67:0/3774722021 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5566b69ff400 0x5566b5caf180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:09.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.258+0000 7f2c4f825640 1 --2- 172.21.15.67:0/3774722021 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5566b69ffc00 0x5566b5cae680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:09.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.258+0000 7f2c50026640 1 -- 172.21.15.67:0/3774722021 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5566b69ffc00 msgr2=0x5566b5cae680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:09.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.258+0000 7f2c50026640 1 --2- 172.21.15.67:0/3774722021 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5566b69ffc00 0x5566b5cae680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:09.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.258+0000 7f2c50026640 1 -- 172.21.15.67:0/3774722021 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5566b69ff400 msgr2=0x5566b5caf180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:09.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.258+0000 7f2c50026640 1 --2- 172.21.15.67:0/3774722021 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5566b69ff400 0x5566b5caf180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:09.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.258+0000 7f2c50026640 1 -- 172.21.15.67:0/3774722021 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5566b5c3a780 con 0x5566b69ff800 2024-03-24T00:45:09.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.258+0000 7f2c4f825640 1 --2- 172.21.15.67:0/3774722021 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5566b69ffc00 0x5566b5cae680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:45:09.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.258+0000 7f2c50026640 1 --2- 172.21.15.67:0/3774722021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5566b69ff800 0x5566b5caec00 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x5566b6ac5290 tx=0x5566b699f3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:09.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.258+0000 7f2c4a01a640 1 -- 172.21.15.67:0/3774722021 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5566b5c2a9c0 con 0x5566b69ff800 2024-03-24T00:45:09.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.258+0000 7f2c4a01a640 1 -- 172.21.15.67:0/3774722021 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5566b5c30e00 con 0x5566b69ff800 2024-03-24T00:45:09.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.258+0000 7f2c5203a280 1 -- 172.21.15.67:0/3774722021 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5566b5c3a3c0 con 0x5566b69ff800 2024-03-24T00:45:09.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.258+0000 7f2c4a01a640 1 -- 172.21.15.67:0/3774722021 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5566b5c2a680 con 0x5566b69ff800 2024-03-24T00:45:09.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.258+0000 7f2c5203a280 1 -- 172.21.15.67:0/3774722021 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5566b5c3b680 con 0x5566b69ff800 2024-03-24T00:45:09.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.258+0000 7f2c5203a280 4 main: JournalTool: resolving pool 13 2024-03-24T00:45:09.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.259+0000 7f2c4a01a640 1 -- 172.21.15.67:0/3774722021 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x5566b6b46c00 con 0x5566b69ff800 2024-03-24T00:45:09.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.260+0000 7f2c4a01a640 1 --2- 172.21.15.67:0/3774722021 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x5566b6b48c00 0x5566b5caf700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:09.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.260+0000 7f2c4a01a640 1 -- 172.21.15.67:0/3774722021 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6187+0+0 (secure 0 0 0) 0x5566b6b44a00 con 0x5566b69ff800 2024-03-24T00:45:09.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.260+0000 7f2c5203a280 4 main: JournalTool: creating IoCtx.. 2024-03-24T00:45:09.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.260+0000 7f2c5203a280 4 main: Executing for rank 0 2024-03-24T00:45:09.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.260+0000 7f2c4f825640 1 --2- 172.21.15.67:0/3774722021 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x5566b6b48c00 0x5566b5caf700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:09.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.260+0000 7f2c4f825640 1 --2- 172.21.15.67:0/3774722021 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x5566b6b48c00 0x5566b5caf700 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x5566b6ac4690 tx=0x5566b699cba0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:09.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.260+0000 7f2c5203a280 1 -- start start 2024-03-24T00:45:09.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.260+0000 7f2c5203a280 1 Processor -- start 2024-03-24T00:45:09.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.261+0000 7f2c5203a280 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5566b6b49400 0x5566b5cae100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:09.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.261+0000 7f2c5203a280 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5566b6b49800 0x5566b5cadb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:09.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.261+0000 7f2c5203a280 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5566b6b49c00 0x5566b5cad600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:09.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.261+0000 7f2c5203a280 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566b69cb500 con 0x5566b6b49c00 2024-03-24T00:45:09.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.261+0000 7f2c5203a280 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566b69ca600 con 0x5566b6b49800 2024-03-24T00:45:09.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.261+0000 7f2c5203a280 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5566b69ca780 con 0x5566b6b49400 2024-03-24T00:45:09.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.261+0000 7f2c4f825640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5566b6b49c00 0x5566b5cad600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:09.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.261+0000 7f2c4f825640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5566b6b49c00 0x5566b5cad600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33138/0 (socket says 172.21.15.67:33138) 2024-03-24T00:45:09.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.261+0000 7f2c4f825640 1 -- 172.21.15.67:0/1304759559 learned_addr learned my addr 172.21.15.67:0/1304759559 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:09.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.261+0000 7f2c50827640 1 --2- 172.21.15.67:0/1304759559 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5566b6b49400 0x5566b5cae100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:09.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.261+0000 7f2c50026640 1 --2- 172.21.15.67:0/1304759559 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5566b6b49800 0x5566b5cadb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:09.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.261+0000 7f2c4f825640 1 -- 172.21.15.67:0/1304759559 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5566b6b49400 msgr2=0x5566b5cae100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:09.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.261+0000 7f2c4f825640 1 --2- 172.21.15.67:0/1304759559 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5566b6b49400 0x5566b5cae100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:09.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.261+0000 7f2c4f825640 1 -- 172.21.15.67:0/1304759559 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5566b6b49800 msgr2=0x5566b5cadb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:09.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.261+0000 7f2c4f825640 1 --2- 172.21.15.67:0/1304759559 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5566b6b49800 0x5566b5cadb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:09.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.261+0000 7f2c4f825640 1 -- 172.21.15.67:0/1304759559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5566b5c3a000 con 0x5566b6b49c00 2024-03-24T00:45:09.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.261+0000 7f2c4f825640 1 --2- 172.21.15.67:0/1304759559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5566b6b49c00 0x5566b5cad600 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x5566b5f416b0 tx=0x5566b699c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dedd678237fb2851 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:09.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.262+0000 7f2c47014640 1 -- 172.21.15.67:0/1304759559 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5566b5c2a4e0 con 0x5566b6b49c00 2024-03-24T00:45:09.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.262+0000 7f2c47014640 1 -- 172.21.15.67:0/1304759559 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5566b5c301c0 con 0x5566b6b49c00 2024-03-24T00:45:09.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.262+0000 7f2c47014640 1 -- 172.21.15.67:0/1304759559 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5566b6b84d00 con 0x5566b6b49c00 2024-03-24T00:45:09.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.262+0000 7f2c5203a280 1 -- 172.21.15.67:0/1304759559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5566b5c3b4a0 con 0x5566b6b49c00 2024-03-24T00:45:09.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.262+0000 7f2c5203a280 1 -- 172.21.15.67:0/1304759559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x5566b5c3a000 con 0x5566b6b49c00 2024-03-24T00:45:09.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.262+0000 7f2c5203a280 4 waiting for MDS map... 2024-03-24T00:45:09.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.262+0000 7f2c47014640 1 -- 172.21.15.67:0/1304759559 <== mon.0 v2:172.21.15.67:3300/0 4 ==== osd_map(90..90 src has 1..90) v4 ==== 6187+0+0 (secure 0 0 0) 0x5566b5fe0000 con 0x5566b6b49c00 2024-03-24T00:45:09.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.262+0000 7f2c47014640 1 -- 172.21.15.67:0/1304759559 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 55) v1 ==== 2735+0+0 (secure 0 0 0) 0x5566b5c5f500 con 0x5566b6b49c00 2024-03-24T00:45:09.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.262+0000 7f2c5203a280 4 Got MDS map 55 2024-03-24T00:45:09.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.262+0000 7f2c5203a280 1 --2- 172.21.15.67:0/1304759559 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x5566b5fdf400 0x5566b5ff4000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:09.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.262+0000 7f2c5203a280 1 -- 172.21.15.67:0/1304759559 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:1 13.f 13:f1f69726:::400.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90) v9 -- 0x5566b5fdf800 con 0x5566b5fdf400 2024-03-24T00:45:09.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.262+0000 7f2c50827640 1 --2- 172.21.15.67:0/1304759559 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x5566b5fdf400 0x5566b5ff4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:09.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.263+0000 7f2c50827640 1 --2- 172.21.15.67:0/1304759559 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x5566b5fdf400 0x5566b5ff4000 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=35d9890be2037c01 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:09.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.265+0000 7f2c50827640 1 -- 172.21.15.67:0/1304759559 <== osd.6 v2:172.21.15.72:6816/1725206562 1 ==== osd_op_reply(1 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+22 (crc 0 0 0) 0x5566b5cfa000 con 0x5566b5fdf400 2024-03-24T00:45:09.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.265+0000 7f2c5203a280 1 --2- 172.21.15.67:0/1304759559 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x5566b5fdfc00 0x5566b5ff4580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:09.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.265+0000 7f2c5203a280 1 -- 172.21.15.67:0/1304759559 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:2 13.14 13:292cf221:::200.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90) v9 -- 0x5566b6004000 con 0x5566b5fdfc00 2024-03-24T00:45:09.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.265+0000 7f2c50026640 1 --2- 172.21.15.67:0/1304759559 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x5566b5fdfc00 0x5566b5ff4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:09.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.266+0000 7f2c50026640 1 --2- 172.21.15.67:0/1304759559 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x5566b5fdfc00 0x5566b5ff4580 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=bee35a4404ebf64c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:09.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.268+0000 7f2c50026640 1 -- 172.21.15.67:0/1304759559 <== osd.0 v2:172.21.15.67:6808/2628130519 1 ==== osd_op_reply(2 200.00000000 [read 0~90 out=90b] v0'0 uv19 ondisk = 0) v8 ==== 156+0+90 (crc 0 0 0) 0x5566b5cfa500 con 0x5566b5fdfc00 2024-03-24T00:45:09.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.268+0000 7f2c45811640 1 -- 172.21.15.67:0/1304759559 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:3 13.14 13:2e2fa760:::200.00000001:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e90) v9 -- 0x5566b6005c00 con 0x5566b5fdfc00 2024-03-24T00:45:09.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.268+0000 7f2c45811640 1 --2- 172.21.15.67:0/1304759559 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x5566b6006000 0x5566b5ff6680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:09.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.268+0000 7f2c45811640 1 -- 172.21.15.67:0/1304759559 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:4 13.1b 13:ddb4e4d7:::200.00000002:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e90) v9 -- 0x5566b6006400 con 0x5566b6006000 2024-03-24T00:45:09.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.268+0000 7f2c50026640 1 -- 172.21.15.67:0/1304759559 <== osd.0 v2:172.21.15.67:6808/2628130519 2 ==== osd_op_reply(3 200.00000001 [stat out=16b] v0'0 uv16 ondisk = 0) v8 ==== 156+0+16 (crc 0 0 0) 0x5566b5cfa500 con 0x5566b5fdfc00 2024-03-24T00:45:09.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.268+0000 7f2c50827640 1 --2- 172.21.15.67:0/1304759559 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x5566b6006000 0x5566b5ff6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:09.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.268+0000 7f2c50827640 1 --2- 172.21.15.67:0/1304759559 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x5566b6006000 0x5566b5ff6680 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=8e755484382c57d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:09.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.269+0000 7f2c50827640 1 -- 172.21.15.67:0/1304759559 <== osd.4 v2:172.21.15.72:6808/477220793 1 ==== osd_op_reply(4 200.00000002 [stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x5566b5cfa000 con 0x5566b6006000 2024-03-24T00:45:09.273 INFO:teuthology.orchestra.run.smithi067.stdout:old journal was 4212724~2030 2024-03-24T00:45:09.273 INFO:teuthology.orchestra.run.smithi067.stdout:new journal start will be 8388608 (4173854 bytes past old end) 2024-03-24T00:45:09.273 INFO:teuthology.orchestra.run.smithi067.stdout:writing journal head 2024-03-24T00:45:09.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.269+0000 7f2c5203a280 1 -- 172.21.15.67:0/1304759559 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:5 13.14 13:292cf221:::200.00000000:head [writefull 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) v9 -- 0x5566b6004400 con 0x5566b5fdfc00 2024-03-24T00:45:09.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.270+0000 7f2c50026640 1 -- 172.21.15.67:0/1304759559 <== osd.0 v2:172.21.15.67:6808/2628130519 3 ==== osd_op_reply(5 200.00000000 [writefull 0~90] v90'20 uv20 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x5566b5cfa500 con 0x5566b5fdfc00 2024-03-24T00:45:09.274 INFO:teuthology.orchestra.run.smithi067.stdout:writing EResetJournal entry 2024-03-24T00:45:09.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.270+0000 7f2c5203a280 1 -- 172.21.15.67:0/1304759559 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:6 13.1b 13:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) v9 -- 0x5566b6004800 con 0x5566b6006000 2024-03-24T00:45:09.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.270+0000 7f2c5203a280 1 --2- 172.21.15.67:0/1304759559 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x5566b6004c00 0x5566b5ff4b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:09.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.270+0000 7f2c5203a280 1 -- 172.21.15.67:0/1304759559 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:7 13.b 13:d5c7a900:::200.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) v9 -- 0x5566b6005000 con 0x5566b6004c00 2024-03-24T00:45:09.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.270+0000 7f2c5203a280 1 -- 172.21.15.67:0/1304759559 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:8 13.1f 13:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) v9 -- 0x5566b6005400 con 0x5566b5fdf400 2024-03-24T00:45:09.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.270+0000 7f2c5203a280 1 --2- 172.21.15.67:0/1304759559 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x5566b6005800 0x5566b5ff5080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:09.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.270+0000 7f2c5203a280 1 -- 172.21.15.67:0/1304759559 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:9 13.1a 13:5e41ab69:::200.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) v9 -- 0x5566b6006400 con 0x5566b6005800 2024-03-24T00:45:09.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.270+0000 7f2c5203a280 1 -- 172.21.15.67:0/1304759559 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:10 13.1e 13:7bf7e7b5:::200.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) v9 -- 0x5566b6006800 con 0x5566b6006000 2024-03-24T00:45:09.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.270+0000 7f2c5203a280 1 --2- 172.21.15.67:0/1304759559 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x5566b6006c00 0x5566b5ff5600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:09.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.270+0000 7f2c5203a280 1 -- 172.21.15.67:0/1304759559 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:11 13.1d 13:bec1251c:::200.00000007:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) v9 -- 0x5566b6007000 con 0x5566b6006c00 2024-03-24T00:45:09.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.270+0000 7f2c4f825640 1 --2- 172.21.15.67:0/1304759559 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x5566b6005800 0x5566b5ff5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:09.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.270+0000 7f2c50026640 1 --2- 172.21.15.67:0/1304759559 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x5566b6004c00 0x5566b5ff4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:09.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.270+0000 7f2c50827640 1 --2- 172.21.15.67:0/1304759559 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x5566b6006c00 0x5566b5ff5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:09.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.270+0000 7f2c4f825640 1 --2- 172.21.15.67:0/1304759559 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x5566b6005800 0x5566b5ff5080 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=657e26fce46c2ad6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:09.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.271+0000 7f2c50026640 1 --2- 172.21.15.67:0/1304759559 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x5566b6004c00 0x5566b5ff4b00 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=59702bf940eec4f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:09.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.271+0000 7f2c50827640 1 --2- 172.21.15.67:0/1304759559 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x5566b6006c00 0x5566b5ff5600 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=bec09c5a33b25284 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:09.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.271+0000 7f2c50827640 1 -- 172.21.15.67:0/1304759559 <== osd.6 v2:172.21.15.72:6816/1725206562 2 ==== osd_op_reply(8 200.00000004 [delete] v90'7 uv6 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x5566b5cfa000 con 0x5566b5fdf400 2024-03-24T00:45:09.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.272+0000 7f2c50827640 1 -- 172.21.15.67:0/1304759559 <== osd.4 v2:172.21.15.72:6808/477220793 2 ==== osd_op_reply(6 200.00000002 [delete] v90'2 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x5566b5cfa000 con 0x5566b6006000 2024-03-24T00:45:09.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.272+0000 7f2c50827640 1 -- 172.21.15.67:0/1304759559 <== osd.4 v2:172.21.15.72:6808/477220793 3 ==== osd_op_reply(10 200.00000006 [delete] v90'2 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x5566b5cfa000 con 0x5566b6006000 2024-03-24T00:45:09.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.274+0000 7f2c50026640 1 -- 172.21.15.67:0/1304759559 <== osd.5 v2:172.21.15.72:6800/288486206 1 ==== osd_op_reply(7 200.00000003 [delete] v90'4 uv3 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x5566b5cfa500 con 0x5566b6004c00 2024-03-24T00:45:09.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.274+0000 7f2c4f825640 1 -- 172.21.15.67:0/1304759559 <== osd.1 v2:172.21.15.67:6800/4062510404 1 ==== osd_op_reply(9 200.00000005 [delete] v90'2 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x5566b5cfa780 con 0x5566b6005800 2024-03-24T00:45:09.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.274+0000 7f2c45811640 1 -- 172.21.15.67:0/1304759559 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:12 13.1b 13:ddb4e4d7:::200.00000002:head [write 0~48 in=48b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) v9 -- 0x5566b6004000 con 0x5566b6006000 2024-03-24T00:45:09.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.275+0000 7f2c50827640 1 -- 172.21.15.67:0/1304759559 <== osd.7 v2:172.21.15.72:6824/2893038746 1 ==== osd_op_reply(11 200.00000007 [delete] v90'3 uv2 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x5566b5cfa000 con 0x5566b6006c00 2024-03-24T00:45:09.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.275+0000 7f2c50827640 1 -- 172.21.15.67:0/1304759559 <== osd.4 v2:172.21.15.72:6808/477220793 4 ==== osd_op_reply(12 200.00000002 [write 0~48] v90'3 uv3 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x5566b5cfa000 con 0x5566b6006000 2024-03-24T00:45:09.278 INFO:teuthology.orchestra.run.smithi067.stdout:done 2024-03-24T00:45:09.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.275+0000 7f2c5203a280 1 -- 172.21.15.67:0/1304759559 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x5566b5fdfc00 msgr2=0x5566b5ff4580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:09.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.275+0000 7f2c5203a280 1 --2- 172.21.15.67:0/1304759559 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x5566b5fdfc00 0x5566b5ff4580 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:09.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.275+0000 7f2c5203a280 1 -- 172.21.15.67:0/1304759559 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x5566b6005800 msgr2=0x5566b5ff5080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:09.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.275+0000 7f2c5203a280 1 --2- 172.21.15.67:0/1304759559 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x5566b6005800 0x5566b5ff5080 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:09.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.275+0000 7f2c5203a280 1 -- 172.21.15.67:0/1304759559 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x5566b6006000 msgr2=0x5566b5ff6680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:09.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.275+0000 7f2c5203a280 1 --2- 172.21.15.67:0/1304759559 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x5566b6006000 0x5566b5ff6680 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:09.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.275+0000 7f2c5203a280 1 -- 172.21.15.67:0/1304759559 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x5566b6004c00 msgr2=0x5566b5ff4b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:09.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.276+0000 7f2c5203a280 1 --2- 172.21.15.67:0/1304759559 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x5566b6004c00 0x5566b5ff4b00 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:09.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.276+0000 7f2c5203a280 1 -- 172.21.15.67:0/1304759559 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x5566b5fdf400 msgr2=0x5566b5ff4000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:09.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.276+0000 7f2c5203a280 1 --2- 172.21.15.67:0/1304759559 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x5566b5fdf400 0x5566b5ff4000 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:09.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.276+0000 7f2c5203a280 1 -- 172.21.15.67:0/1304759559 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x5566b6006c00 msgr2=0x5566b5ff5600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:09.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.276+0000 7f2c5203a280 1 --2- 172.21.15.67:0/1304759559 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x5566b6006c00 0x5566b5ff5600 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:09.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.276+0000 7f2c4f825640 1 -- 172.21.15.67:0/1304759559 reap_dead start 2024-03-24T00:45:09.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.276+0000 7f2c4f825640 1 -- 172.21.15.67:0/1304759559 reap_dead start 2024-03-24T00:45:09.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.276+0000 7f2c5203a280 1 -- 172.21.15.67:0/1304759559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5566b6b49c00 msgr2=0x5566b5cad600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:09.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.276+0000 7f2c5203a280 1 --2- 172.21.15.67:0/1304759559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5566b6b49c00 0x5566b5cad600 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x5566b5f416b0 tx=0x5566b699c960 comp rx=0 tx=0).stop 2024-03-24T00:45:09.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.276+0000 7f2c5203a280 1 -- 172.21.15.67:0/1304759559 shutdown_connections 2024-03-24T00:45:09.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.276+0000 7f2c5203a280 1 --2- 172.21.15.67:0/1304759559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5566b6b49c00 0x5566b5cad600 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:09.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.276+0000 7f2c5203a280 1 -- 172.21.15.67:0/1304759559 >> 172.21.15.67:0/1304759559 conn(0x5566b6b49000 msgr2=0x5566b6989800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:09.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.276+0000 7f2c5203a280 1 -- 172.21.15.67:0/1304759559 shutdown_connections 2024-03-24T00:45:09.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.276+0000 7f2c5203a280 1 -- 172.21.15.67:0/1304759559 wait complete. 2024-03-24T00:45:09.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.276+0000 7f2c5203a280 1 -- 172.21.15.67:0/3774722021 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x5566b6b48c00 msgr2=0x5566b5caf700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:09.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.276+0000 7f2c5203a280 1 --2- 172.21.15.67:0/3774722021 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x5566b6b48c00 0x5566b5caf700 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x5566b6ac4690 tx=0x5566b699cba0 comp rx=0 tx=0).stop 2024-03-24T00:45:09.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.276+0000 7f2c5203a280 1 -- 172.21.15.67:0/3774722021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5566b69ff800 msgr2=0x5566b5caec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:09.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.276+0000 7f2c5203a280 1 --2- 172.21.15.67:0/3774722021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5566b69ff800 0x5566b5caec00 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x5566b6ac5290 tx=0x5566b699f3e0 comp rx=0 tx=0).stop 2024-03-24T00:45:09.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.276+0000 7f2c5203a280 1 -- 172.21.15.67:0/3774722021 shutdown_connections 2024-03-24T00:45:09.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.276+0000 7f2c5203a280 1 --2- 172.21.15.67:0/3774722021 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x5566b6b48c00 0x5566b5caf700 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:09.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.276+0000 7f2c5203a280 1 --2- 172.21.15.67:0/3774722021 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5566b69ffc00 0x5566b5cae680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:09.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.276+0000 7f2c5203a280 1 --2- 172.21.15.67:0/3774722021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5566b69ff800 0x5566b5caec00 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:09.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.276+0000 7f2c5203a280 1 --2- 172.21.15.67:0/3774722021 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5566b69ff400 0x5566b5caf180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:09.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.276+0000 7f2c5203a280 1 -- 172.21.15.67:0/3774722021 >> 172.21.15.67:0/3774722021 conn(0x5566b69ff000 msgr2=0x5566b6989000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:09.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.276+0000 7f2c5203a280 1 -- 172.21.15.67:0/3774722021 shutdown_connections 2024-03-24T00:45:09.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.276+0000 7f2c5203a280 1 -- 172.21.15.67:0/3774722021 wait complete. 2024-03-24T00:45:09.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.276+0000 7f2c5203a280 1 -- 172.21.15.67:0/239062179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5566b5c05c00 msgr2=0x5566b5cad080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:09.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.276+0000 7f2c5203a280 1 --2- 172.21.15.67:0/239062179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5566b5c05c00 0x5566b5cad080 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x5566b6a25380 tx=0x5566b699f5c0 comp rx=0 tx=0).stop 2024-03-24T00:45:09.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.276+0000 7f2c5203a280 1 -- 172.21.15.67:0/239062179 shutdown_connections 2024-03-24T00:45:09.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.276+0000 7f2c5203a280 1 --2- 172.21.15.67:0/239062179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5566b5c05c00 0x5566b5cad080 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:09.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.276+0000 7f2c5203a280 1 --2- 172.21.15.67:0/239062179 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5566b5c05800 0x5566b5cacb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:09.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.276+0000 7f2c5203a280 1 --2- 172.21.15.67:0/239062179 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5566b5c04c00 0x5566b5cac580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:09.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.276+0000 7f2c5203a280 1 -- 172.21.15.67:0/239062179 >> 172.21.15.67:0/239062179 conn(0x5566b5c05000 msgr2=0x5566b6988800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:09.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.276+0000 7f2c5203a280 1 -- 172.21.15.67:0/239062179 shutdown_connections 2024-03-24T00:45:09.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.276+0000 7f2c5203a280 1 -- 172.21.15.67:0/239062179 wait complete. 2024-03-24T00:45:09.284 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-journal-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', '--rank', 'cephfs:0', 'journal', 'reset'] in time 0:00:00.111381, result: old journal was 4212724~2030 new journal start will be 8388608 (4173854 bytes past old end) writing journal head writing EResetJournal entry done 2024-03-24T00:45:09.285 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-03-24T00:45:09.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.425+0000 7f8211cea640 1 Processor -- start 2024-03-24T00:45:09.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.425+0000 7f8211cea640 1 -- start start 2024-03-24T00:45:09.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.426+0000 7f8211cea640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f820c099d00 0x7f820c09a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:09.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.426+0000 7f8211cea640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f820c0d1530 0x7f820c15d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:09.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.426+0000 7f8211cea640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f820c09a620 con 0x7f820c0b49b0 2024-03-24T00:45:09.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.426+0000 7f8211cea640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f820c0cc1b0 con 0x7f820c099d00 2024-03-24T00:45:09.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.426+0000 7f8211cea640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f820c0cc400 con 0x7f820c0d1530 2024-03-24T00:45:09.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.426+0000 7f820affd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f820c0b49b0 0x7f820c0d0e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46242/0 (socket says 172.21.15.67:46242) 2024-03-24T00:45:09.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.426+0000 7f820affd640 1 -- 172.21.15.67:0/1091104869 learned_addr learned my addr 172.21.15.67:0/1091104869 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:45:09.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.426+0000 7f820b7fe640 1 --2- 172.21.15.67:0/1091104869 >> v2:172.21.15.72:3300/0 conn(0x7f820c099d00 0x7f820c09a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:09.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.426+0000 7f820bfff640 1 --2- 172.21.15.67:0/1091104869 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f820c0d1530 0x7f820c15d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:09.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.426+0000 7f820a7fc640 1 -- 172.21.15.67:0/1091104869 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3925611574 0 0) 0x7f820c09a620 con 0x7f820c0b49b0 2024-03-24T00:45:09.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.426+0000 7f820a7fc640 1 -- 172.21.15.67:0/1091104869 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81f4003520 con 0x7f820c0b49b0 2024-03-24T00:45:09.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.427+0000 7f820b7fe640 1 -- 172.21.15.67:0/1091104869 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f820c0d1530 msgr2=0x7f820c15d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:09.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.427+0000 7f820b7fe640 1 --2- 172.21.15.67:0/1091104869 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f820c0d1530 0x7f820c15d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:09.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.427+0000 7f820b7fe640 1 -- 172.21.15.67:0/1091104869 >> v1:172.21.15.67:6789/0 conn(0x7f820c0b49b0 legacy=0x7f820c0d0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:09.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.427+0000 7f820b7fe640 1 -- 172.21.15.67:0/1091104869 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f820c0cc6e0 con 0x7f820c099d00 2024-03-24T00:45:09.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.427+0000 7f820bfff640 1 --2- 172.21.15.67:0/1091104869 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f820c0d1530 0x7f820c15d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:45:09.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.427+0000 7f820b7fe640 1 --2- 172.21.15.67:0/1091104869 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f820c099d00 0x7f820c09a0e0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f81fc0097a0 tx=0x7f81fc02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f107355a5c668508 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:09.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.433+0000 7f820a7fc640 1 -- 172.21.15.67:0/1091104869 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81fc035020 con 0x7f820c099d00 2024-03-24T00:45:09.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.433+0000 7f820a7fc640 1 -- 172.21.15.67:0/1091104869 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81fc031830 con 0x7f820c099d00 2024-03-24T00:45:09.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.433+0000 7f820a7fc640 1 -- 172.21.15.67:0/1091104869 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81fc031c00 con 0x7f820c099d00 2024-03-24T00:45:09.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.433+0000 7f8211cea640 1 -- 172.21.15.67:0/1091104869 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f820c099d00 msgr2=0x7f820c09a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:09.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.433+0000 7f8211cea640 1 --2- 172.21.15.67:0/1091104869 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f820c099d00 0x7f820c09a0e0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f81fc0097a0 tx=0x7f81fc02b8d0 comp rx=0 tx=0).stop 2024-03-24T00:45:09.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.433+0000 7f8211cea640 1 -- 172.21.15.67:0/1091104869 shutdown_connections 2024-03-24T00:45:09.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.433+0000 7f8211cea640 1 --2- 172.21.15.67:0/1091104869 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f820c0d1530 0x7f820c15d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:09.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.433+0000 7f8211cea640 1 --2- 172.21.15.67:0/1091104869 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f820c099d00 0x7f820c09a0e0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:09.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.433+0000 7f8211cea640 1 -- 172.21.15.67:0/1091104869 >> 172.21.15.67:0/1091104869 conn(0x7f820c0b3650 msgr2=0x7f820c0b3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:09.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.433+0000 7f8211cea640 1 -- 172.21.15.67:0/1091104869 shutdown_connections 2024-03-24T00:45:09.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.433+0000 7f8211cea640 1 -- 172.21.15.67:0/1091104869 wait complete. 2024-03-24T00:45:09.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.434+0000 7f8211cea640 1 Processor -- start 2024-03-24T00:45:09.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.435+0000 7f8211cea640 1 -- start start 2024-03-24T00:45:09.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.435+0000 7f8211cea640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f820c099d00 0x7f820c15b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:09.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.435+0000 7f8211cea640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f820c0b49b0 0x7f820c15cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:09.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.435+0000 7f8211cea640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f820c06c7c0 0x7f820c06cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:09.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.435+0000 7f8211cea640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f820c0d01c0 con 0x7f820c099d00 2024-03-24T00:45:09.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.435+0000 7f8211cea640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f820c0ccca0 con 0x7f820c06c7c0 2024-03-24T00:45:09.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.435+0000 7f8211cea640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f820c0b32a0 con 0x7f820c0b49b0 2024-03-24T00:45:09.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.435+0000 7f820b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f820c099d00 0x7f820c15b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:09.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.435+0000 7f820b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f820c099d00 0x7f820c15b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33148/0 (socket says 172.21.15.67:33148) 2024-03-24T00:45:09.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.435+0000 7f820b7fe640 1 -- 172.21.15.67:0/3244504232 learned_addr learned my addr 172.21.15.67:0/3244504232 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:09.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.436+0000 7f820affd640 1 --2- 172.21.15.67:0/3244504232 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f820c0b49b0 0x7f820c15cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:09.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.436+0000 7f820bfff640 1 --2- 172.21.15.67:0/3244504232 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f820c06c7c0 0x7f820c06cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:09.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.436+0000 7f820b7fe640 1 -- 172.21.15.67:0/3244504232 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f820c0b49b0 msgr2=0x7f820c15cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:09.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.436+0000 7f820b7fe640 1 --2- 172.21.15.67:0/3244504232 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f820c0b49b0 0x7f820c15cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:09.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.436+0000 7f820b7fe640 1 -- 172.21.15.67:0/3244504232 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f820c06c7c0 msgr2=0x7f820c06cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:09.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.436+0000 7f820b7fe640 1 --2- 172.21.15.67:0/3244504232 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f820c06c7c0 0x7f820c06cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:09.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.436+0000 7f820b7fe640 1 -- 172.21.15.67:0/3244504232 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f820c0cb990 con 0x7f820c099d00 2024-03-24T00:45:09.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.436+0000 7f820b7fe640 1 --2- 172.21.15.67:0/3244504232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f820c099d00 0x7f820c15b220 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f81fc0098c0 tx=0x7f81fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:09.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.436+0000 7f81f3fff640 1 -- 172.21.15.67:0/3244504232 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81fc035040 con 0x7f820c099d00 2024-03-24T00:45:09.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.436+0000 7f81f3fff640 1 -- 172.21.15.67:0/3244504232 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81fc03b920 con 0x7f820c099d00 2024-03-24T00:45:09.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.436+0000 7f8211cea640 1 -- 172.21.15.67:0/3244504232 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f820c0c8cd0 con 0x7f820c099d00 2024-03-24T00:45:09.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.436+0000 7f81f3fff640 1 -- 172.21.15.67:0/3244504232 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81fc03bd10 con 0x7f820c099d00 2024-03-24T00:45:09.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.436+0000 7f8211cea640 1 -- 172.21.15.67:0/3244504232 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f820c0c9290 con 0x7f820c099d00 2024-03-24T00:45:09.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.437+0000 7f8211cea640 1 -- 172.21.15.67:0/3244504232 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f820c04f400 con 0x7f820c099d00 2024-03-24T00:45:09.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.438+0000 7f81f3fff640 1 -- 172.21.15.67:0/3244504232 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f81fc038020 con 0x7f820c099d00 2024-03-24T00:45:09.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.438+0000 7f81f3fff640 1 --2- 172.21.15.67:0/3244504232 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f81f806bbd0 0x7f81f806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:09.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.438+0000 7f81f3fff640 1 -- 172.21.15.67:0/3244504232 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f81fc0acd40 con 0x7f820c099d00 2024-03-24T00:45:09.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.439+0000 7f820affd640 1 --2- 172.21.15.67:0/3244504232 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f81f806bbd0 0x7f81f806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:09.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.439+0000 7f820affd640 1 --2- 172.21.15.67:0/3244504232 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f81f806bbd0 0x7f81f806e090 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f820c09a9c0 tx=0x7f8200009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:09.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.443+0000 7f81f3fff640 1 -- 172.21.15.67:0/3244504232 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f81fc07b650 con 0x7f820c099d00 2024-03-24T00:45:09.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:09.662+0000 7f8211cea640 1 -- 172.21.15.67:0/3244504232 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) v1 -- 0x7f820c06ec70 con 0x7f820c099d00 2024-03-24T00:45:10.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.109+0000 7f81f3fff640 1 -- 172.21.15.67:0/3244504232 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v56) v1 ==== 162+0+0 (secure 0 0 0) 0x7f81fc07af50 con 0x7f820c099d00 2024-03-24T00:45:10.111 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked joinable; MDS may join as newly active. 2024-03-24T00:45:10.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.112+0000 7f8211cea640 1 -- 172.21.15.67:0/3244504232 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f81f806bbd0 msgr2=0x7f81f806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:10.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.112+0000 7f8211cea640 1 --2- 172.21.15.67:0/3244504232 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f81f806bbd0 0x7f81f806e090 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f820c09a9c0 tx=0x7f8200009040 comp rx=0 tx=0).stop 2024-03-24T00:45:10.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.112+0000 7f8211cea640 1 -- 172.21.15.67:0/3244504232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f820c099d00 msgr2=0x7f820c15b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:10.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.112+0000 7f8211cea640 1 --2- 172.21.15.67:0/3244504232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f820c099d00 0x7f820c15b220 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f81fc0098c0 tx=0x7f81fc0046c0 comp rx=0 tx=0).stop 2024-03-24T00:45:10.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.112+0000 7f8211cea640 1 -- 172.21.15.67:0/3244504232 shutdown_connections 2024-03-24T00:45:10.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.112+0000 7f8211cea640 1 --2- 172.21.15.67:0/3244504232 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f81f806bbd0 0x7f81f806e090 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:10.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.112+0000 7f8211cea640 1 --2- 172.21.15.67:0/3244504232 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f820c06c7c0 0x7f820c06cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:10.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.112+0000 7f8211cea640 1 --2- 172.21.15.67:0/3244504232 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f820c0b49b0 0x7f820c15cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:10.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.112+0000 7f8211cea640 1 --2- 172.21.15.67:0/3244504232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f820c099d00 0x7f820c15b220 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:10.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.112+0000 7f8211cea640 1 -- 172.21.15.67:0/3244504232 >> 172.21.15.67:0/3244504232 conn(0x7f820c0b3650 msgr2=0x7f820c1612a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:10.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.112+0000 7f8211cea640 1 -- 172.21.15.67:0/3244504232 shutdown_connections 2024-03-24T00:45:10.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.112+0000 7f8211cea640 1 -- 172.21.15.67:0/3244504232 wait complete. 2024-03-24T00:45:10.127 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:45:10.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.228+0000 7f5226c85640 1 Processor -- start 2024-03-24T00:45:10.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.228+0000 7f5226c85640 1 -- start start 2024-03-24T00:45:10.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.229+0000 7f5226c85640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f522015dec0 0x7f52200ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:10.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.229+0000 7f5226c85640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f522015e2a0 0x7f5220162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:10.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.229+0000 7f5226c85640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52200cf640 con 0x7f52200d07e0 2024-03-24T00:45:10.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.229+0000 7f5226c85640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52200cee30 con 0x7f522015e2a0 2024-03-24T00:45:10.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.229+0000 7f5226c85640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52200d2120 con 0x7f522015dec0 2024-03-24T00:45:10.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.229+0000 7f521ffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f522015e2a0 0x7f5220162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:10.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.229+0000 7f521ffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f522015e2a0 0x7f5220162630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33158/0 (socket says 172.21.15.67:33158) 2024-03-24T00:45:10.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.229+0000 7f521ffff640 1 -- 172.21.15.67:0/899185069 learned_addr learned my addr 172.21.15.67:0/899185069 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:10.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.229+0000 7f5224a21640 1 --2- 172.21.15.67:0/899185069 >> v2:172.21.15.72:3300/0 conn(0x7f522015dec0 0x7f52200ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:10.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.230+0000 7f521ffff640 1 -- 172.21.15.67:0/899185069 >> v2:172.21.15.72:3300/0 conn(0x7f522015dec0 msgr2=0x7f52200ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:10.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.230+0000 7f521ffff640 1 --2- 172.21.15.67:0/899185069 >> v2:172.21.15.72:3300/0 conn(0x7f522015dec0 0x7f52200ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:10.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.230+0000 7f521ffff640 1 -- 172.21.15.67:0/899185069 >> v1:172.21.15.72:6789/0 conn(0x7f52200d07e0 legacy=0x7f52200d0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:10.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.230+0000 7f521ffff640 1 -- 172.21.15.67:0/899185069 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52200cf080 con 0x7f522015e2a0 2024-03-24T00:45:10.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.230+0000 7f5224a21640 1 --2- 172.21.15.67:0/899185069 >> v2:172.21.15.72:3300/0 conn(0x7f522015dec0 0x7f52200ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:45:10.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.230+0000 7f521ffff640 1 --2- 172.21.15.67:0/899185069 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f522015e2a0 0x7f5220162630 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f52200d14a0 tx=0x7f521402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8f321fae0ae0e75 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:10.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.230+0000 7f521f7fe640 1 -- 172.21.15.67:0/899185069 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5214035020 con 0x7f522015e2a0 2024-03-24T00:45:10.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.230+0000 7f521f7fe640 1 -- 172.21.15.67:0/899185069 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5214031870 con 0x7f522015e2a0 2024-03-24T00:45:10.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.230+0000 7f521f7fe640 1 -- 172.21.15.67:0/899185069 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5214031c40 con 0x7f522015e2a0 2024-03-24T00:45:10.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.231+0000 7f5226c85640 1 -- 172.21.15.67:0/899185069 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f522015e2a0 msgr2=0x7f5220162630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:10.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.231+0000 7f5226c85640 1 --2- 172.21.15.67:0/899185069 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f522015e2a0 0x7f5220162630 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f52200d14a0 tx=0x7f521402b910 comp rx=0 tx=0).stop 2024-03-24T00:45:10.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.231+0000 7f5226c85640 1 -- 172.21.15.67:0/899185069 shutdown_connections 2024-03-24T00:45:10.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.231+0000 7f5226c85640 1 --2- 172.21.15.67:0/899185069 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f522015e2a0 0x7f5220162630 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:10.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.231+0000 7f5226c85640 1 --2- 172.21.15.67:0/899185069 >> v2:172.21.15.72:3300/0 conn(0x7f522015dec0 0x7f52200ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:10.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.231+0000 7f5226c85640 1 -- 172.21.15.67:0/899185069 >> 172.21.15.67:0/899185069 conn(0x7f522006c810 msgr2=0x7f522006cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:10.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.231+0000 7f5226c85640 1 -- 172.21.15.67:0/899185069 shutdown_connections 2024-03-24T00:45:10.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.231+0000 7f5226c85640 1 -- 172.21.15.67:0/899185069 wait complete. 2024-03-24T00:45:10.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.232+0000 7f5226c85640 1 Processor -- start 2024-03-24T00:45:10.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.232+0000 7f5226c85640 1 -- start start 2024-03-24T00:45:10.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.233+0000 7f5226c85640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52200d07e0 0x7f52200ab130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:10.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.233+0000 7f5226c85640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f522015dec0 0x7f52200b62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:10.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.233+0000 7f5226c85640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f522015e2a0 0x7f5220152ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:10.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.233+0000 7f5226c85640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52200d2730 con 0x7f52200d07e0 2024-03-24T00:45:10.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.233+0000 7f5226c85640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52200d25b0 con 0x7f522015dec0 2024-03-24T00:45:10.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.233+0000 7f5226c85640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52200d28b0 con 0x7f522015e2a0 2024-03-24T00:45:10.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.233+0000 7f5224a21640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52200d07e0 0x7f52200ab130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:10.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.233+0000 7f5224a21640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52200d07e0 0x7f52200ab130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33166/0 (socket says 172.21.15.67:33166) 2024-03-24T00:45:10.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.233+0000 7f5224a21640 1 -- 172.21.15.67:0/1008382255 learned_addr learned my addr 172.21.15.67:0/1008382255 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:10.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.233+0000 7f521ffff640 1 --2- 172.21.15.67:0/1008382255 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f522015dec0 0x7f52200b62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:10.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.233+0000 7f5225222640 1 --2- 172.21.15.67:0/1008382255 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f522015e2a0 0x7f5220152ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:10.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.233+0000 7f5224a21640 1 -- 172.21.15.67:0/1008382255 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f522015e2a0 msgr2=0x7f5220152ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:10.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.233+0000 7f5224a21640 1 --2- 172.21.15.67:0/1008382255 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f522015e2a0 0x7f5220152ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:10.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.233+0000 7f5224a21640 1 -- 172.21.15.67:0/1008382255 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f522015dec0 msgr2=0x7f52200b62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:10.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.233+0000 7f5224a21640 1 --2- 172.21.15.67:0/1008382255 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f522015dec0 0x7f52200b62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:10.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.233+0000 7f5224a21640 1 -- 172.21.15.67:0/1008382255 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52200b38b0 con 0x7f52200d07e0 2024-03-24T00:45:10.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.233+0000 7f521ffff640 1 --2- 172.21.15.67:0/1008382255 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f522015dec0 0x7f52200b62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:45:10.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.234+0000 7f5224a21640 1 --2- 172.21.15.67:0/1008382255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52200d07e0 0x7f52200ab130 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f521000e760 tx=0x7f521000ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:10.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.234+0000 7f521d7fa640 1 -- 172.21.15.67:0/1008382255 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521000f040 con 0x7f52200d07e0 2024-03-24T00:45:10.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.234+0000 7f521d7fa640 1 -- 172.21.15.67:0/1008382255 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5210004230 con 0x7f52200d07e0 2024-03-24T00:45:10.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.234+0000 7f5226c85640 1 -- 172.21.15.67:0/1008382255 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52200ab750 con 0x7f52200d07e0 2024-03-24T00:45:10.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.234+0000 7f521d7fa640 1 -- 172.21.15.67:0/1008382255 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521000ee70 con 0x7f52200d07e0 2024-03-24T00:45:10.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.234+0000 7f5226c85640 1 -- 172.21.15.67:0/1008382255 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52200abd10 con 0x7f52200d07e0 2024-03-24T00:45:10.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.235+0000 7f5226c85640 1 -- 172.21.15.67:0/1008382255 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f522004f400 con 0x7f52200d07e0 2024-03-24T00:45:10.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.235+0000 7f521d7fa640 1 -- 172.21.15.67:0/1008382255 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f521000c6d0 con 0x7f52200d07e0 2024-03-24T00:45:10.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.236+0000 7f521d7fa640 1 --2- 172.21.15.67:0/1008382255 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f520c06bc00 0x7f520c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:10.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.236+0000 7f521d7fa640 1 -- 172.21.15.67:0/1008382255 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f521008a950 con 0x7f52200d07e0 2024-03-24T00:45:10.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.236+0000 7f521ffff640 1 --2- 172.21.15.67:0/1008382255 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f520c06bc00 0x7f520c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:10.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.236+0000 7f521ffff640 1 --2- 172.21.15.67:0/1008382255 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f520c06bc00 0x7f520c06e0c0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f5214036000 tx=0x7f521403c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:10.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.240+0000 7f521d7fa640 1 -- 172.21.15.67:0/1008382255 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5210017020 con 0x7f52200d07e0 2024-03-24T00:45:10.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.460+0000 7f5226c85640 1 -- 172.21.15.67:0/1008382255 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52200b9000 con 0x7f52200d07e0 2024-03-24T00:45:10.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.461+0000 7f521d7fa640 1 -- 172.21.15.67:0/1008382255 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+4973 (secure 0 0 0) 0x7f5210059260 con 0x7f52200d07e0 2024-03-24T00:45:10.462 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:45:10.462 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":57,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6359,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2664460392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2664460392},{"type":"v1","addr":"172.21.15.67:6835","nonce":2664460392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6368,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/3323873506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3323873506},{"type":"v1","addr":"172.21.15.72:6837","nonce":3323873506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":7150,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/2744016217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2744016217},{"type":"v1","addr":"172.21.15.72:6836","nonce":2744016217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:44:54.749656+0000","modified":"2024-03-24T00:45:10.112825+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7860},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7860":{"gid":7860,"name":"c","rank":0,"incarnation":57,"state":"up:replay","state_seq":1,"addr":"172.21.15.67:6833/1336395002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1336395002},{"type":"v1","addr":"172.21.15.67:6833","nonce":1336395002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-03-24T00:45:10.462 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 57 2024-03-24T00:45:10.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.463+0000 7f5226c85640 1 -- 172.21.15.67:0/1008382255 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f520c06bc00 msgr2=0x7f520c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:10.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.463+0000 7f5226c85640 1 --2- 172.21.15.67:0/1008382255 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f520c06bc00 0x7f520c06e0c0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f5214036000 tx=0x7f521403c040 comp rx=0 tx=0).stop 2024-03-24T00:45:10.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.463+0000 7f5226c85640 1 -- 172.21.15.67:0/1008382255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52200d07e0 msgr2=0x7f52200ab130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:10.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.463+0000 7f5226c85640 1 --2- 172.21.15.67:0/1008382255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52200d07e0 0x7f52200ab130 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f521000e760 tx=0x7f521000ec20 comp rx=0 tx=0).stop 2024-03-24T00:45:10.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.463+0000 7f5226c85640 1 -- 172.21.15.67:0/1008382255 shutdown_connections 2024-03-24T00:45:10.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.463+0000 7f5226c85640 1 --2- 172.21.15.67:0/1008382255 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f520c06bc00 0x7f520c06e0c0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:10.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.463+0000 7f5226c85640 1 --2- 172.21.15.67:0/1008382255 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f522015e2a0 0x7f5220152ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:10.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.463+0000 7f5226c85640 1 --2- 172.21.15.67:0/1008382255 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f522015dec0 0x7f52200b62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:10.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.463+0000 7f5226c85640 1 --2- 172.21.15.67:0/1008382255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52200d07e0 0x7f52200ab130 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:10.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.463+0000 7f5226c85640 1 -- 172.21.15.67:0/1008382255 >> 172.21.15.67:0/1008382255 conn(0x7f522006c810 msgr2=0x7f52200ca210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:10.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.463+0000 7f5226c85640 1 -- 172.21.15.67:0/1008382255 shutdown_connections 2024-03-24T00:45:10.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:10.463+0000 7f5226c85640 1 -- 172.21.15.67:0/1008382255 wait complete. 2024-03-24T00:45:10.877 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 57, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:44:54.749656+0000', 'modified': '2024-03-24T00:45:10.112825+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 90, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7860}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7860': {'gid': 7860, 'name': 'c', 'rank': 0, 'incarnation': 57, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.67:6833/1336395002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6832', 'nonce': 1336395002}, {'type': 'v1', 'addr': '172.21.15.67:6833', 'nonce': 1336395002}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-24T00:45:10.877 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7860:up:replay 2024-03-24T00:45:11.878 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:45:11.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.991+0000 7f82f0f9d640 1 Processor -- start 2024-03-24T00:45:11.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.991+0000 7f82f0f9d640 1 -- start start 2024-03-24T00:45:11.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.992+0000 7f82f0f9d640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f82ec0d0570 0x7f82ec08bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:11.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.992+0000 7f82f0f9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f82ec15f980 0x7f82ec15fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:11.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.992+0000 7f82f0f9d640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82ec08afc0 con 0x7f82ec15bd50 2024-03-24T00:45:11.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.992+0000 7f82f0f9d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82ec15d140 con 0x7f82ec15f980 2024-03-24T00:45:11.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.992+0000 7f82f0f9d640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82ec1606f0 con 0x7f82ec0d0570 2024-03-24T00:45:11.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.992+0000 7f82ebfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f82ec15f980 0x7f82ec15fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:11.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.992+0000 7f82ebfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f82ec15f980 0x7f82ec15fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33176/0 (socket says 172.21.15.67:33176) 2024-03-24T00:45:11.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.992+0000 7f82ebfff640 1 -- 172.21.15.67:0/777848555 learned_addr learned my addr 172.21.15.67:0/777848555 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:11.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.992+0000 7f82eaffd640 1 --2- 172.21.15.67:0/777848555 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f82ec0d0570 0x7f82ec08bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:11.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.992+0000 7f82ebfff640 1 -- 172.21.15.67:0/777848555 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f82ec0d0570 msgr2=0x7f82ec08bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:11.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.992+0000 7f82ebfff640 1 --2- 172.21.15.67:0/777848555 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f82ec0d0570 0x7f82ec08bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:11.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.992+0000 7f82ebfff640 1 -- 172.21.15.67:0/777848555 >> v1:172.21.15.72:6789/0 conn(0x7f82ec15bd50 legacy=0x7f82ec0cfe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:11.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.992+0000 7f82ebfff640 1 -- 172.21.15.67:0/777848555 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82ec08b280 con 0x7f82ec15f980 2024-03-24T00:45:11.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.992+0000 7f82eaffd640 1 --2- 172.21.15.67:0/777848555 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f82ec0d0570 0x7f82ec08bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:45:11.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.993+0000 7f82ebfff640 1 --2- 172.21.15.67:0/777848555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82ec15f980 0x7f82ec15fe30 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f82e4009e50 tx=0x7f82e402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88c4444ef7b283ab server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:11.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.993+0000 7f82ea7fc640 1 -- 172.21.15.67:0/777848555 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e402ca70 con 0x7f82ec15f980 2024-03-24T00:45:11.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.993+0000 7f82ea7fc640 1 -- 172.21.15.67:0/777848555 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82e402cc10 con 0x7f82ec15f980 2024-03-24T00:45:11.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.993+0000 7f82ea7fc640 1 -- 172.21.15.67:0/777848555 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e40343f0 con 0x7f82ec15f980 2024-03-24T00:45:11.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.994+0000 7f82f0f9d640 1 -- 172.21.15.67:0/777848555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82ec15f980 msgr2=0x7f82ec15fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:11.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.994+0000 7f82f0f9d640 1 --2- 172.21.15.67:0/777848555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82ec15f980 0x7f82ec15fe30 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f82e4009e50 tx=0x7f82e402bed0 comp rx=0 tx=0).stop 2024-03-24T00:45:11.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.994+0000 7f82f0f9d640 1 -- 172.21.15.67:0/777848555 shutdown_connections 2024-03-24T00:45:11.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.994+0000 7f82f0f9d640 1 --2- 172.21.15.67:0/777848555 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f82ec0d0570 0x7f82ec08bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:11.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.994+0000 7f82f0f9d640 1 --2- 172.21.15.67:0/777848555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82ec15f980 0x7f82ec15fe30 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:11.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.994+0000 7f82f0f9d640 1 -- 172.21.15.67:0/777848555 >> 172.21.15.67:0/777848555 conn(0x7f82ec0d1ad0 msgr2=0x7f82ec0d1ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:11.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.994+0000 7f82f0f9d640 1 -- 172.21.15.67:0/777848555 shutdown_connections 2024-03-24T00:45:11.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.994+0000 7f82f0f9d640 1 -- 172.21.15.67:0/777848555 wait complete. 2024-03-24T00:45:11.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.995+0000 7f82f0f9d640 1 Processor -- start 2024-03-24T00:45:11.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.995+0000 7f82f0f9d640 1 -- start start 2024-03-24T00:45:11.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.996+0000 7f82f0f9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f82ec0d0570 0x7f82ec06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:11.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.996+0000 7f82f0f9d640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f82ec15bd50 0x7f82ec0c7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:11.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.996+0000 7f82f0f9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82ec15f980 0x7f82ec0c5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:11.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.996+0000 7f82f0f9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82ec160370 con 0x7f82ec15f980 2024-03-24T00:45:11.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.996+0000 7f82f0f9d640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82ec08b840 con 0x7f82ec0d0570 2024-03-24T00:45:11.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.996+0000 7f82f0f9d640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82ec0ce260 con 0x7f82ec15bd50 2024-03-24T00:45:11.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.996+0000 7f82ebfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82ec15f980 0x7f82ec0c5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:11.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.996+0000 7f82ebfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82ec15f980 0x7f82ec0c5750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33184/0 (socket says 172.21.15.67:33184) 2024-03-24T00:45:12.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.996+0000 7f82ebfff640 1 -- 172.21.15.67:0/3191341294 learned_addr learned my addr 172.21.15.67:0/3191341294 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:12.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.996+0000 7f82eb7fe640 1 --2- 172.21.15.67:0/3191341294 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f82ec0d0570 0x7f82ec06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:12.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.996+0000 7f82eaffd640 1 --2- 172.21.15.67:0/3191341294 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f82ec15bd50 0x7f82ec0c7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:12.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.996+0000 7f82ebfff640 1 -- 172.21.15.67:0/3191341294 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f82ec15bd50 msgr2=0x7f82ec0c7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:12.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.996+0000 7f82ebfff640 1 --2- 172.21.15.67:0/3191341294 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f82ec15bd50 0x7f82ec0c7780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:12.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.996+0000 7f82ebfff640 1 -- 172.21.15.67:0/3191341294 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f82ec0d0570 msgr2=0x7f82ec06d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:12.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.996+0000 7f82ebfff640 1 --2- 172.21.15.67:0/3191341294 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f82ec0d0570 0x7f82ec06d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:12.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.996+0000 7f82ebfff640 1 -- 172.21.15.67:0/3191341294 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82ec06db40 con 0x7f82ec15f980 2024-03-24T00:45:12.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.997+0000 7f82ebfff640 1 --2- 172.21.15.67:0/3191341294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82ec15f980 0x7f82ec0c5750 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f82e4009e50 tx=0x7f82e4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:12.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.997+0000 7f82e3fff640 1 -- 172.21.15.67:0/3191341294 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e4043020 con 0x7f82ec15f980 2024-03-24T00:45:12.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.997+0000 7f82e3fff640 1 -- 172.21.15.67:0/3191341294 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82e403e450 con 0x7f82ec15f980 2024-03-24T00:45:12.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.997+0000 7f82f0f9d640 1 -- 172.21.15.67:0/3191341294 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82ec158ab0 con 0x7f82ec15f980 2024-03-24T00:45:12.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.997+0000 7f82e3fff640 1 -- 172.21.15.67:0/3191341294 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e403e820 con 0x7f82ec15f980 2024-03-24T00:45:12.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.997+0000 7f82f0f9d640 1 -- 172.21.15.67:0/3191341294 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82ec0c82d0 con 0x7f82ec15f980 2024-03-24T00:45:12.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.998+0000 7f82f0f9d640 1 -- 172.21.15.67:0/3191341294 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82ec04f400 con 0x7f82ec15f980 2024-03-24T00:45:12.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.998+0000 7f82e3fff640 1 -- 172.21.15.67:0/3191341294 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f82e4005d10 con 0x7f82ec15f980 2024-03-24T00:45:12.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.999+0000 7f82e3fff640 1 --2- 172.21.15.67:0/3191341294 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f82cc06bbd0 0x7f82cc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:12.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.999+0000 7f82e3fff640 1 -- 172.21.15.67:0/3191341294 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f82e40ad740 con 0x7f82ec15f980 2024-03-24T00:45:12.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:11.999+0000 7f82eb7fe640 1 --2- 172.21.15.67:0/3191341294 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f82cc06bbd0 0x7f82cc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:12.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:12.000+0000 7f82eb7fe640 1 --2- 172.21.15.67:0/3191341294 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f82cc06bbd0 0x7f82cc06e090 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f82dc0017f0 tx=0x7f82dc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:12.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:12.003+0000 7f82e3fff640 1 -- 172.21.15.67:0/3191341294 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f82e407ba00 con 0x7f82ec15f980 2024-03-24T00:45:12.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:12.224+0000 7f82f0f9d640 1 -- 172.21.15.67:0/3191341294 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82ec0d2970 con 0x7f82ec15f980 2024-03-24T00:45:12.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:12.225+0000 7f82e3fff640 1 -- 172.21.15.67:0/3191341294 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+4973 (secure 0 0 0) 0x7f82e407faa0 con 0x7f82ec15f980 2024-03-24T00:45:12.226 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:45:12.226 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":59,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6359,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2664460392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2664460392},{"type":"v1","addr":"172.21.15.67:6835","nonce":2664460392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6368,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/3323873506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3323873506},{"type":"v1","addr":"172.21.15.72:6837","nonce":3323873506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":7150,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/2744016217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2744016217},{"type":"v1","addr":"172.21.15.72:6836","nonce":2744016217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:44:54.749656+0000","modified":"2024-03-24T00:45:11.118245+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7860},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7860":{"gid":7860,"name":"c","rank":0,"incarnation":57,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.67:6833/1336395002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1336395002},{"type":"v1","addr":"172.21.15.67:6833","nonce":1336395002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-03-24T00:45:12.226 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 59 2024-03-24T00:45:12.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:12.227+0000 7f82f0f9d640 1 -- 172.21.15.67:0/3191341294 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f82cc06bbd0 msgr2=0x7f82cc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:12.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:12.227+0000 7f82f0f9d640 1 --2- 172.21.15.67:0/3191341294 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f82cc06bbd0 0x7f82cc06e090 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f82dc0017f0 tx=0x7f82dc009040 comp rx=0 tx=0).stop 2024-03-24T00:45:12.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:12.227+0000 7f82f0f9d640 1 -- 172.21.15.67:0/3191341294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82ec15f980 msgr2=0x7f82ec0c5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:12.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:12.227+0000 7f82f0f9d640 1 --2- 172.21.15.67:0/3191341294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82ec15f980 0x7f82ec0c5750 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f82e4009e50 tx=0x7f82e4004a40 comp rx=0 tx=0).stop 2024-03-24T00:45:12.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:12.227+0000 7f82f0f9d640 1 -- 172.21.15.67:0/3191341294 shutdown_connections 2024-03-24T00:45:12.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:12.227+0000 7f82f0f9d640 1 --2- 172.21.15.67:0/3191341294 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f82cc06bbd0 0x7f82cc06e090 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:12.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:12.227+0000 7f82f0f9d640 1 --2- 172.21.15.67:0/3191341294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82ec15f980 0x7f82ec0c5750 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:12.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:12.227+0000 7f82f0f9d640 1 --2- 172.21.15.67:0/3191341294 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f82ec15bd50 0x7f82ec0c7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:12.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:12.227+0000 7f82f0f9d640 1 --2- 172.21.15.67:0/3191341294 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f82ec0d0570 0x7f82ec06d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:12.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:12.227+0000 7f82f0f9d640 1 -- 172.21.15.67:0/3191341294 >> 172.21.15.67:0/3191341294 conn(0x7f82ec0d1ad0 msgr2=0x7f82ec0b1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:12.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:12.227+0000 7f82f0f9d640 1 -- 172.21.15.67:0/3191341294 shutdown_connections 2024-03-24T00:45:12.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:12.227+0000 7f82f0f9d640 1 -- 172.21.15.67:0/3191341294 wait complete. 2024-03-24T00:45:12.641 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 59, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:44:54.749656+0000', 'modified': '2024-03-24T00:45:11.118245+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 90, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7860}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7860': {'gid': 7860, 'name': 'c', 'rank': 0, 'incarnation': 57, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.67:6833/1336395002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6832', 'nonce': 1336395002}, {'type': 'v1', 'addr': '172.21.15.67:6833', 'nonce': 1336395002}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-24T00:45:12.641 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7860:up:rejoin 2024-03-24T00:45:13.643 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:45:13.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.744+0000 7f8ba9e9a640 1 Processor -- start 2024-03-24T00:45:13.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.744+0000 7f8ba9e9a640 1 -- start start 2024-03-24T00:45:13.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.745+0000 7f8ba9e9a640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8ba40ce930 0x7f8ba4166b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:13.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.745+0000 7f8ba9e9a640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ba40ccfe0 con 0x7f8ba40cfc70 2024-03-24T00:45:13.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.745+0000 7f8ba9e9a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ba40cd2c0 con 0x7f8ba415bf70 2024-03-24T00:45:13.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.745+0000 7f8ba9e9a640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba40ccda0 con 0x7f8ba40ce930 2024-03-24T00:45:13.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.745+0000 7f8ba2ffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8ba40cfc70 0x7f8ba40ce220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56208/0 (socket says 172.21.15.67:56208) 2024-03-24T00:45:13.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.745+0000 7f8ba2ffd640 1 -- 172.21.15.67:0/2143769301 learned_addr learned my addr 172.21.15.67:0/2143769301 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:45:13.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.745+0000 7f8ba3fff640 1 --2- 172.21.15.67:0/2143769301 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8ba40ce930 0x7f8ba4166b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:13.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.745+0000 7f8ba27fc640 1 -- 172.21.15.67:0/2143769301 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3657452420 0 0) 0x7f8ba40ccfe0 con 0x7f8ba40cfc70 2024-03-24T00:45:13.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.745+0000 7f8ba27fc640 1 -- 172.21.15.67:0/2143769301 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b8c003520 con 0x7f8ba40cfc70 2024-03-24T00:45:13.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.745+0000 7f8ba3fff640 1 -- 172.21.15.67:0/2143769301 >> v1:172.21.15.72:6789/0 conn(0x7f8ba415bf70 legacy=0x7f8ba40cc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:13.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.745+0000 7f8ba27fc640 1 -- 172.21.15.67:0/2143769301 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1862541294 0 0) 0x7f8ba40cd2c0 con 0x7f8ba415bf70 2024-03-24T00:45:13.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.746+0000 7f8ba3fff640 1 -- 172.21.15.67:0/2143769301 >> v1:172.21.15.67:6789/0 conn(0x7f8ba40cfc70 legacy=0x7f8ba40ce220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:13.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.746+0000 7f8ba3fff640 1 -- 172.21.15.67:0/2143769301 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ba40cd5a0 con 0x7f8ba40ce930 2024-03-24T00:45:13.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.746+0000 7f8ba3fff640 1 --2- 172.21.15.67:0/2143769301 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8ba40ce930 0x7f8ba4166b10 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c0097d0 tx=0x7f8b9c02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1fd741e65af26271 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:13.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.746+0000 7f8ba27fc640 1 -- 172.21.15.67:0/2143769301 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b9c035020 con 0x7f8ba40ce930 2024-03-24T00:45:13.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.747+0000 7f8ba27fc640 1 -- 172.21.15.67:0/2143769301 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b9c031890 con 0x7f8ba40ce930 2024-03-24T00:45:13.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.747+0000 7f8ba27fc640 1 -- 172.21.15.67:0/2143769301 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b9c031c40 con 0x7f8ba40ce930 2024-03-24T00:45:13.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.747+0000 7f8ba9e9a640 1 -- 172.21.15.67:0/2143769301 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8ba40ce930 msgr2=0x7f8ba4166b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:13.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.747+0000 7f8ba9e9a640 1 --2- 172.21.15.67:0/2143769301 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8ba40ce930 0x7f8ba4166b10 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c0097d0 tx=0x7f8b9c02b930 comp rx=0 tx=0).stop 2024-03-24T00:45:13.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.747+0000 7f8ba9e9a640 1 -- 172.21.15.67:0/2143769301 shutdown_connections 2024-03-24T00:45:13.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.747+0000 7f8ba9e9a640 1 --2- 172.21.15.67:0/2143769301 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8ba40ce930 0x7f8ba4166b10 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:13.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.747+0000 7f8ba9e9a640 1 -- 172.21.15.67:0/2143769301 >> 172.21.15.67:0/2143769301 conn(0x7f8ba40d1bf0 msgr2=0x7f8ba40d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:13.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.747+0000 7f8ba9e9a640 1 -- 172.21.15.67:0/2143769301 shutdown_connections 2024-03-24T00:45:13.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.747+0000 7f8ba9e9a640 1 -- 172.21.15.67:0/2143769301 wait complete. 2024-03-24T00:45:13.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.749+0000 7f8ba9e9a640 1 Processor -- start 2024-03-24T00:45:13.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.749+0000 7f8ba9e9a640 1 -- start start 2024-03-24T00:45:13.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.750+0000 7f8ba9e9a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ba40ce930 0x7f8ba40afa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:13.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.750+0000 7f8ba9e9a640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8ba40cfc70 0x7f8ba40cb320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:13.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.750+0000 7f8ba9e9a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8ba415bf70 0x7f8ba415eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:13.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.750+0000 7f8ba9e9a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba40b2ce0 con 0x7f8ba40ce930 2024-03-24T00:45:13.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.750+0000 7f8ba9e9a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba415c350 con 0x7f8ba415bf70 2024-03-24T00:45:13.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.750+0000 7f8ba9e9a640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba40b2e60 con 0x7f8ba40cfc70 2024-03-24T00:45:13.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.750+0000 7f8ba37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ba40ce930 0x7f8ba40afa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:13.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.750+0000 7f8ba37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ba40ce930 0x7f8ba40afa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41408/0 (socket says 172.21.15.67:41408) 2024-03-24T00:45:13.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.750+0000 7f8ba37fe640 1 -- 172.21.15.67:0/224817740 learned_addr learned my addr 172.21.15.67:0/224817740 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:13.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.750+0000 7f8ba2ffd640 1 --2- 172.21.15.67:0/224817740 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8ba40cfc70 0x7f8ba40cb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:13.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.750+0000 7f8ba3fff640 1 --2- 172.21.15.67:0/224817740 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8ba415bf70 0x7f8ba415eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:13.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.751+0000 7f8ba37fe640 1 -- 172.21.15.67:0/224817740 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8ba40cfc70 msgr2=0x7f8ba40cb320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:13.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.751+0000 7f8ba37fe640 1 --2- 172.21.15.67:0/224817740 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8ba40cfc70 0x7f8ba40cb320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:13.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.751+0000 7f8ba37fe640 1 -- 172.21.15.67:0/224817740 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8ba415bf70 msgr2=0x7f8ba415eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:13.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.751+0000 7f8ba37fe640 1 --2- 172.21.15.67:0/224817740 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8ba415bf70 0x7f8ba415eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:13.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.751+0000 7f8ba37fe640 1 -- 172.21.15.67:0/224817740 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ba40c9fe0 con 0x7f8ba40ce930 2024-03-24T00:45:13.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.751+0000 7f8ba37fe640 1 --2- 172.21.15.67:0/224817740 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ba40ce930 0x7f8ba40afa70 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f8b9400c7f0 tx=0x7f8b9400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:13.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.751+0000 7f8b8bfff640 1 -- 172.21.15.67:0/224817740 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b94017070 con 0x7f8ba40ce930 2024-03-24T00:45:13.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.751+0000 7f8b8bfff640 1 -- 172.21.15.67:0/224817740 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b94003da0 con 0x7f8ba40ce930 2024-03-24T00:45:13.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.751+0000 7f8ba9e9a640 1 -- 172.21.15.67:0/224817740 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ba40ca2e0 con 0x7f8ba40ce930 2024-03-24T00:45:13.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.751+0000 7f8b8bfff640 1 -- 172.21.15.67:0/224817740 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b94005290 con 0x7f8ba40ce930 2024-03-24T00:45:13.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.751+0000 7f8ba9e9a640 1 -- 172.21.15.67:0/224817740 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ba40cab20 con 0x7f8ba40ce930 2024-03-24T00:45:13.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.752+0000 7f8ba9e9a640 1 -- 172.21.15.67:0/224817740 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ba404f400 con 0x7f8ba40ce930 2024-03-24T00:45:13.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.752+0000 7f8b8bfff640 1 -- 172.21.15.67:0/224817740 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8b9401a070 con 0x7f8ba40ce930 2024-03-24T00:45:13.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.753+0000 7f8b8bfff640 1 --2- 172.21.15.67:0/224817740 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8b9006bbd0 0x7f8b9006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:13.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.753+0000 7f8b8bfff640 1 -- 172.21.15.67:0/224817740 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f8b9408a1b0 con 0x7f8ba40ce930 2024-03-24T00:45:13.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.753+0000 7f8ba2ffd640 1 --2- 172.21.15.67:0/224817740 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8b9006bbd0 0x7f8b9006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:13.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.754+0000 7f8ba2ffd640 1 --2- 172.21.15.67:0/224817740 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8b9006bbd0 0x7f8b9006e090 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f8ba409fb00 tx=0x7f8b98009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:13.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.757+0000 7f8b8bfff640 1 -- 172.21.15.67:0/224817740 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8ba40ca2e0 con 0x7f8ba40ce930 2024-03-24T00:45:13.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.997+0000 7f8ba9e9a640 1 -- 172.21.15.67:0/224817740 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ba40c8670 con 0x7f8ba40ce930 2024-03-24T00:45:13.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:13.998+0000 7f8b8bfff640 1 -- 172.21.15.67:0/224817740 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f8b94058500 con 0x7f8ba40ce930 2024-03-24T00:45:13.999 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:45:13.999 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":60,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6359,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2664460392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2664460392},{"type":"v1","addr":"172.21.15.67:6835","nonce":2664460392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6368,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/3323873506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3323873506},{"type":"v1","addr":"172.21.15.72:6837","nonce":3323873506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":7150,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/2744016217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2744016217},{"type":"v1","addr":"172.21.15.72:6836","nonce":2744016217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:44:54.749656+0000","modified":"2024-03-24T00:45:13.120404+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7860},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7860":{"gid":7860,"name":"c","rank":0,"incarnation":57,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6833/1336395002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1336395002},{"type":"v1","addr":"172.21.15.67:6833","nonce":1336395002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7860,"qdb_cluster":[7860]},"id":6}]} 2024-03-24T00:45:13.999 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 60 2024-03-24T00:45:14.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.000+0000 7f8ba9e9a640 1 -- 172.21.15.67:0/224817740 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8b9006bbd0 msgr2=0x7f8b9006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:14.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.000+0000 7f8ba9e9a640 1 --2- 172.21.15.67:0/224817740 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8b9006bbd0 0x7f8b9006e090 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f8ba409fb00 tx=0x7f8b98009040 comp rx=0 tx=0).stop 2024-03-24T00:45:14.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.000+0000 7f8ba9e9a640 1 -- 172.21.15.67:0/224817740 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ba40ce930 msgr2=0x7f8ba40afa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:14.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.000+0000 7f8ba9e9a640 1 --2- 172.21.15.67:0/224817740 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ba40ce930 0x7f8ba40afa70 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f8b9400c7f0 tx=0x7f8b9400ccb0 comp rx=0 tx=0).stop 2024-03-24T00:45:14.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.000+0000 7f8ba9e9a640 1 -- 172.21.15.67:0/224817740 shutdown_connections 2024-03-24T00:45:14.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.000+0000 7f8ba9e9a640 1 --2- 172.21.15.67:0/224817740 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8b9006bbd0 0x7f8b9006e090 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:14.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.000+0000 7f8ba9e9a640 1 --2- 172.21.15.67:0/224817740 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8ba415bf70 0x7f8ba415eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:14.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.000+0000 7f8ba9e9a640 1 --2- 172.21.15.67:0/224817740 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8ba40cfc70 0x7f8ba40cb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:14.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.000+0000 7f8ba9e9a640 1 --2- 172.21.15.67:0/224817740 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ba40ce930 0x7f8ba40afa70 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:14.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.000+0000 7f8ba9e9a640 1 -- 172.21.15.67:0/224817740 >> 172.21.15.67:0/224817740 conn(0x7f8ba40d1bf0 msgr2=0x7f8ba4070550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:14.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.000+0000 7f8ba9e9a640 1 -- 172.21.15.67:0/224817740 shutdown_connections 2024-03-24T00:45:14.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.000+0000 7f8ba9e9a640 1 -- 172.21.15.67:0/224817740 wait complete. 2024-03-24T00:45:14.415 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 60, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:44:54.749656+0000', 'modified': '2024-03-24T00:45:13.120404+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 90, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7860}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7860': {'gid': 7860, 'name': 'c', 'rank': 0, 'incarnation': 57, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.67:6833/1336395002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6832', 'nonce': 1336395002}, {'type': 'v1', 'addr': '172.21.15.67:6833', 'nonce': 1336395002}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7860, 'qdb_cluster': [7860]} 2024-03-24T00:45:14.415 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-24T00:45:14.415 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-24T00:45:14.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.517+0000 7f3996f9d640 1 Processor -- start 2024-03-24T00:45:14.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.517+0000 7f3996f9d640 1 -- start start 2024-03-24T00:45:14.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.519+0000 7f3996f9d640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f399015bae0 0x7f39900cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:14.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.519+0000 7f3996f9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f39901603c0 0x7f39900d1840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:14.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.519+0000 7f3996f9d640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39900cf650 con 0x7f399015bec0 2024-03-24T00:45:14.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.519+0000 7f3996f9d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39900cd9d0 con 0x7f39901603c0 2024-03-24T00:45:14.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.519+0000 7f3996f9d640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39900cf910 con 0x7f399015bae0 2024-03-24T00:45:14.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.519+0000 7f399679c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f39901603c0 0x7f39900d1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:14.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.519+0000 7f399679c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f39901603c0 0x7f39900d1840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41414/0 (socket says 172.21.15.67:41414) 2024-03-24T00:45:14.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.519+0000 7f399679c640 1 -- 172.21.15.67:0/2590810613 learned_addr learned my addr 172.21.15.67:0/2590810613 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:14.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.519+0000 7f3995f9b640 1 --2- 172.21.15.67:0/2590810613 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f399015bae0 0x7f39900cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:14.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.519+0000 7f399679c640 1 -- 172.21.15.67:0/2590810613 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f399015bae0 msgr2=0x7f39900cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:14.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.519+0000 7f399679c640 1 --2- 172.21.15.67:0/2590810613 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f399015bae0 0x7f39900cd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:14.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.519+0000 7f399679c640 1 -- 172.21.15.67:0/2590810613 >> v1:172.21.15.72:6789/0 conn(0x7f399015bec0 legacy=0x7f39900d03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:14.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.519+0000 7f399679c640 1 -- 172.21.15.67:0/2590810613 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39900cfbf0 con 0x7f39901603c0 2024-03-24T00:45:14.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.520+0000 7f399679c640 1 --2- 172.21.15.67:0/2590810613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39901603c0 0x7f39900d1840 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f3988009e50 tx=0x7f398802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baae19888d6707a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:14.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.520+0000 7f3994f99640 1 -- 172.21.15.67:0/2590810613 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398802cb00 con 0x7f39901603c0 2024-03-24T00:45:14.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.520+0000 7f3994f99640 1 -- 172.21.15.67:0/2590810613 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f398802cca0 con 0x7f39901603c0 2024-03-24T00:45:14.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.520+0000 7f3994f99640 1 -- 172.21.15.67:0/2590810613 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3988034610 con 0x7f39901603c0 2024-03-24T00:45:14.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.521+0000 7f3996f9d640 1 -- 172.21.15.67:0/2590810613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39901603c0 msgr2=0x7f39900d1840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:14.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.521+0000 7f3996f9d640 1 --2- 172.21.15.67:0/2590810613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39901603c0 0x7f39900d1840 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f3988009e50 tx=0x7f398802bf60 comp rx=0 tx=0).stop 2024-03-24T00:45:14.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.521+0000 7f3996f9d640 1 -- 172.21.15.67:0/2590810613 shutdown_connections 2024-03-24T00:45:14.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.521+0000 7f3996f9d640 1 --2- 172.21.15.67:0/2590810613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39901603c0 0x7f39900d1840 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:14.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.521+0000 7f3996f9d640 1 --2- 172.21.15.67:0/2590810613 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f399015bae0 0x7f39900cd180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:14.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.521+0000 7f3996f9d640 1 -- 172.21.15.67:0/2590810613 >> 172.21.15.67:0/2590810613 conn(0x7f399008afc0 msgr2=0x7f399008b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:14.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.521+0000 7f3996f9d640 1 -- 172.21.15.67:0/2590810613 shutdown_connections 2024-03-24T00:45:14.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.521+0000 7f3996f9d640 1 -- 172.21.15.67:0/2590810613 wait complete. 2024-03-24T00:45:14.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.522+0000 7f3996f9d640 1 Processor -- start 2024-03-24T00:45:14.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.522+0000 7f3996f9d640 1 -- start start 2024-03-24T00:45:14.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.523+0000 7f3996f9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f399015bae0 0x7f39901f0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:14.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.523+0000 7f3996f9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f399015bec0 0x7f39900af620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:14.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.523+0000 7f3996f9d640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f39901603c0 0x7f39900afb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:14.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.523+0000 7f3996f9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39900d1d80 con 0x7f399015bec0 2024-03-24T00:45:14.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.523+0000 7f3996f9d640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39900cd6c0 con 0x7f399015bae0 2024-03-24T00:45:14.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.523+0000 7f3996f9d640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39900d1f00 con 0x7f39901603c0 2024-03-24T00:45:14.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.523+0000 7f399579a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f399015bec0 0x7f39900af620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:14.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.523+0000 7f399579a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f399015bec0 0x7f39900af620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41422/0 (socket says 172.21.15.67:41422) 2024-03-24T00:45:14.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.523+0000 7f399579a640 1 -- 172.21.15.67:0/3062576509 learned_addr learned my addr 172.21.15.67:0/3062576509 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:14.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.523+0000 7f3995f9b640 1 --2- 172.21.15.67:0/3062576509 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f399015bae0 0x7f39901f0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:14.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.523+0000 7f399679c640 1 --2- 172.21.15.67:0/3062576509 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f39901603c0 0x7f39900afb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:14.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.523+0000 7f399579a640 1 -- 172.21.15.67:0/3062576509 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f39901603c0 msgr2=0x7f39900afb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:14.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.523+0000 7f399579a640 1 --2- 172.21.15.67:0/3062576509 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f39901603c0 0x7f39900afb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:14.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.523+0000 7f399579a640 1 -- 172.21.15.67:0/3062576509 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f399015bae0 msgr2=0x7f39901f0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:14.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.523+0000 7f399579a640 1 --2- 172.21.15.67:0/3062576509 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f399015bae0 0x7f39901f0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:14.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.523+0000 7f399579a640 1 -- 172.21.15.67:0/3062576509 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39901f4fe0 con 0x7f399015bec0 2024-03-24T00:45:14.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.523+0000 7f3995f9b640 1 --2- 172.21.15.67:0/3062576509 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f399015bae0 0x7f39901f0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:45:14.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.523+0000 7f399679c640 1 --2- 172.21.15.67:0/3062576509 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f39901603c0 0x7f39900afb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:45:14.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.523+0000 7f399579a640 1 --2- 172.21.15.67:0/3062576509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f399015bec0 0x7f39900af620 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f398000c430 tx=0x7f398000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:14.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.524+0000 7f39867fc640 1 -- 172.21.15.67:0/3062576509 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3980011790 con 0x7f399015bec0 2024-03-24T00:45:14.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.524+0000 7f39867fc640 1 -- 172.21.15.67:0/3062576509 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3980017070 con 0x7f399015bec0 2024-03-24T00:45:14.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.524+0000 7f3996f9d640 1 -- 172.21.15.67:0/3062576509 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39901f52c0 con 0x7f399015bec0 2024-03-24T00:45:14.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.524+0000 7f39867fc640 1 -- 172.21.15.67:0/3062576509 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3980011bb0 con 0x7f399015bec0 2024-03-24T00:45:14.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.524+0000 7f3996f9d640 1 -- 172.21.15.67:0/3062576509 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39901f5880 con 0x7f399015bec0 2024-03-24T00:45:14.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.526+0000 7f39867fc640 1 -- 172.21.15.67:0/3062576509 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f398001c460 con 0x7f399015bec0 2024-03-24T00:45:14.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.526+0000 7f39867fc640 1 --2- 172.21.15.67:0/3062576509 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f397406bc00 0x7f397406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:14.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.526+0000 7f39867fc640 1 -- 172.21.15.67:0/3062576509 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f398008aa00 con 0x7f399015bec0 2024-03-24T00:45:14.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.526+0000 7f3995f9b640 1 --2- 172.21.15.67:0/3062576509 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f397406bc00 0x7f397406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:14.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.527+0000 7f399ade1740 1 Processor -- start 2024-03-24T00:45:14.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.527+0000 7f3995f9b640 1 --2- 172.21.15.67:0/3062576509 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f397406bc00 0x7f397406e0c0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f398c006fd0 tx=0x7f398c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:14.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.527+0000 7f399ade1740 1 -- start start 2024-03-24T00:45:14.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.528+0000 7f399ade1740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5576369a0db0 0x5576369c78e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:14.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.528+0000 7f399ade1740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5576369c7e20 0x55763699fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:14.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.528+0000 7f399ade1740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5576369a1190 0x5576369a0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:14.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.528+0000 7f399ade1740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x557636904c20 con 0x5576369a1190 2024-03-24T00:45:14.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.528+0000 7f399ade1740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576368ada40 con 0x5576369c7e20 2024-03-24T00:45:14.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.528+0000 7f399ade1740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x557636900cc0 con 0x5576369a0db0 2024-03-24T00:45:14.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.528+0000 7f399679c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5576369a1190 0x5576369a0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:14.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.528+0000 7f399679c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5576369a1190 0x5576369a0330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41432/0 (socket says 172.21.15.67:41432) 2024-03-24T00:45:14.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.528+0000 7f399679c640 1 -- 172.21.15.67:0/4103891016 learned_addr learned my addr 172.21.15.67:0/4103891016 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:14.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.528+0000 7f3995f9b640 1 --2- 172.21.15.67:0/4103891016 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5576369a0db0 0x5576369c78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:14.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.528+0000 7f399579a640 1 --2- 172.21.15.67:0/4103891016 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5576369c7e20 0x55763699fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:14.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.528+0000 7f399679c640 1 -- 172.21.15.67:0/4103891016 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5576369a0db0 msgr2=0x5576369c78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:14.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.528+0000 7f399679c640 1 --2- 172.21.15.67:0/4103891016 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5576369a0db0 0x5576369c78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:14.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.528+0000 7f399679c640 1 -- 172.21.15.67:0/4103891016 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5576369c7e20 msgr2=0x55763699fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:14.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.528+0000 7f399679c640 1 --2- 172.21.15.67:0/4103891016 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5576369c7e20 0x55763699fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:14.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.528+0000 7f399679c640 1 -- 172.21.15.67:0/4103891016 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5576368cff90 con 0x5576369a1190 2024-03-24T00:45:14.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.528+0000 7f399679c640 1 --2- 172.21.15.67:0/4103891016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5576369a1190 0x5576369a0330 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f3988036040 tx=0x7f398802e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9f0af92e96dc569 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:14.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.528+0000 7f39737fe640 1 -- 172.21.15.67:0/4103891016 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398802cb00 con 0x5576369a1190 2024-03-24T00:45:14.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.528+0000 7f39737fe640 1 -- 172.21.15.67:0/4103891016 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f398802ce50 con 0x5576369a1190 2024-03-24T00:45:14.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.528+0000 7f39737fe640 1 -- 172.21.15.67:0/4103891016 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398803e670 con 0x5576369a1190 2024-03-24T00:45:14.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.529+0000 7f399ade1740 1 -- 172.21.15.67:0/4103891016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5576369a1190 msgr2=0x5576369a0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:14.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.529+0000 7f399ade1740 1 --2- 172.21.15.67:0/4103891016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5576369a1190 0x5576369a0330 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f3988036040 tx=0x7f398802e000 comp rx=0 tx=0).stop 2024-03-24T00:45:14.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.529+0000 7f399ade1740 1 -- 172.21.15.67:0/4103891016 shutdown_connections 2024-03-24T00:45:14.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.529+0000 7f399ade1740 1 --2- 172.21.15.67:0/4103891016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5576369a1190 0x5576369a0330 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:14.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.529+0000 7f399ade1740 1 --2- 172.21.15.67:0/4103891016 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5576369c7e20 0x55763699fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:14.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.529+0000 7f399ade1740 1 --2- 172.21.15.67:0/4103891016 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5576369a0db0 0x5576369c78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:14.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.529+0000 7f399ade1740 1 -- 172.21.15.67:0/4103891016 >> 172.21.15.67:0/4103891016 conn(0x557636967a60 msgr2=0x5576369e5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:14.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.529+0000 7f399ade1740 1 -- 172.21.15.67:0/4103891016 shutdown_connections 2024-03-24T00:45:14.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.529+0000 7f399ade1740 1 -- 172.21.15.67:0/4103891016 wait complete. 2024-03-24T00:45:14.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.530+0000 7f399ade1740 1 -- start start 2024-03-24T00:45:14.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.530+0000 7f399ade1740 1 Processor -- start 2024-03-24T00:45:14.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.531+0000 7f399ade1740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5576369a0db0 0x5576369beea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:14.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.531+0000 7f399ade1740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5576369a1190 0x5576369c40f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:14.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.531+0000 7f399ade1740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5576369c4630 0x5576369cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:14.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.531+0000 7f399ade1740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576369c7450 con 0x5576369c4630 2024-03-24T00:45:14.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.531+0000 7f399ade1740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576368c0e90 con 0x5576369a1190 2024-03-24T00:45:14.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.531+0000 7f399ade1740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5576369c01d0 con 0x5576369a0db0 2024-03-24T00:45:14.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.531+0000 7f399ade1740 10 client.0 _ll_register_callbacks cb 0x557636618ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:45:14.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.531+0000 7f399679c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5576369c4630 0x5576369cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:14.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.531+0000 7f399679c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5576369c4630 0x5576369cdfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41444/0 (socket says 172.21.15.67:41444) 2024-03-24T00:45:14.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.531+0000 7f399679c640 1 -- 172.21.15.67:0/281677485 learned_addr learned my addr 172.21.15.67:0/281677485 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:14.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.531+0000 7f3995f9b640 1 --2- 172.21.15.67:0/281677485 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5576369a0db0 0x5576369beea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:14.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.531+0000 7f399579a640 1 --2- 172.21.15.67:0/281677485 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5576369a1190 0x5576369c40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:14.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.531+0000 7f399679c640 1 -- 172.21.15.67:0/281677485 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5576369a0db0 msgr2=0x5576369beea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:14.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.531+0000 7f399679c640 1 --2- 172.21.15.67:0/281677485 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5576369a0db0 0x5576369beea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:14.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.531+0000 7f399679c640 1 -- 172.21.15.67:0/281677485 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5576369a1190 msgr2=0x5576369c40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:14.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.531+0000 7f399679c640 1 --2- 172.21.15.67:0/281677485 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5576369a1190 0x5576369c40f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:14.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.531+0000 7f399679c640 1 -- 172.21.15.67:0/281677485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5576368cd9d0 con 0x5576369c4630 2024-03-24T00:45:14.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.531+0000 7f3995f9b640 1 --2- 172.21.15.67:0/281677485 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5576369a0db0 0x5576369beea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:45:14.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.531+0000 7f399579a640 1 --2- 172.21.15.67:0/281677485 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5576369a1190 0x5576369c40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:45:14.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.531+0000 7f399679c640 1 --2- 172.21.15.67:0/281677485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5576369c4630 0x5576369cdfb0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f398803e660 tx=0x7f3988004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43d05f77bb170b33 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:14.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.531+0000 7f3970ff9640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:45:14.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.531+0000 7f3970ff9640 1 -- 172.21.15.67:0/281677485 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3988003ba0 con 0x5576369c4630 2024-03-24T00:45:14.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.531+0000 7f3970ff9640 1 -- 172.21.15.67:0/281677485 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f3988003d40 con 0x5576369c4630 2024-03-24T00:45:14.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.531+0000 7f399ade1740 1 -- 172.21.15.67:0/281677485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f398802ce50 con 0x5576369c4630 2024-03-24T00:45:14.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.532+0000 7f3970ff9640 1 -- 172.21.15.67:0/281677485 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3988056050 con 0x5576369c4630 2024-03-24T00:45:14.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.532+0000 7f399ade1740 10 client.7917 fetch_fsmap learned FSMap version 60 2024-03-24T00:45:14.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.532+0000 7f399ade1740 1 -- 172.21.15.67:0/281677485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=2+}) v3 -- 0x5576366192d0 con 0x5576369c4630 2024-03-24T00:45:14.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.532+0000 7f3970ff9640 1 -- 172.21.15.67:0/281677485 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f398802ce50 con 0x5576369c4630 2024-03-24T00:45:14.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.532+0000 7f3970ff9640 1 -- 172.21.15.67:0/281677485 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 60) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f3988059020 con 0x5576369c4630 2024-03-24T00:45:14.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.532+0000 7f399ade1740 10 client.7917 fetch_fsmap finished waiting for FSMap version 60 2024-03-24T00:45:14.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.532+0000 7f399ade1740 10 client.7917 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T00:45:14.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.532+0000 7f399ade1740 20 client.7917 populate_metadata read hostname 'smithi067' 2024-03-24T00:45:14.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.532+0000 7f399ade1740 1 --2- 172.21.15.67:0/281677485 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x5576369c03b0 0x5576369c2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:14.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.532+0000 7f399ade1740 4 client.7917 mds_command: new command op to 7860 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:45:14.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.532+0000 7f399ade1740 1 -- 172.21.15.67:0/281677485 --> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557636912160 con 0x5576369c03b0 2024-03-24T00:45:14.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.532+0000 7f3995f9b640 1 --2- 172.21.15.67:0/281677485 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x5576369c03b0 0x5576369c2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:14.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.532+0000 7f3970ff9640 10 client.7917 ms_handle_connect on v2:172.21.15.67:6832/1336395002 2024-03-24T00:45:14.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.532+0000 7f3995f9b640 1 --2- 172.21.15.67:0/281677485 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x5576369c03b0 0x5576369c2590 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=220c08fa00132c7d server_cookie=66265b3fa7c98fe1 in_seq=0 out_seq=0 2024-03-24T00:45:14.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.533+0000 7f3970ff9640 1 -- 172.21.15.67:0/281677485 <== mds.0 v2:172.21.15.67:6832/1336395002 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557636912160 con 0x5576369c03b0 2024-03-24T00:45:14.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.533+0000 7f3970ff9640 10 client.7917 handle_command_reply: tid=0 2024-03-24T00:45:14.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.533+0000 7f399ade1740 1 client.7917 shutdown 2024-03-24T00:45:14.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.533+0000 7f399ade1740 1 -- 172.21.15.67:0/281677485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5576369c4630 msgr2=0x5576369cdfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:14.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.533+0000 7f399ade1740 1 --2- 172.21.15.67:0/281677485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5576369c4630 0x5576369cdfb0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f398803e660 tx=0x7f3988004ab0 comp rx=0 tx=0).stop 2024-03-24T00:45:14.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.533+0000 7f399ade1740 1 -- 172.21.15.67:0/281677485 shutdown_connections 2024-03-24T00:45:14.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.533+0000 7f399ade1740 1 --2- 172.21.15.67:0/281677485 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x5576369c03b0 0x5576369c2590 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:14.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.533+0000 7f399ade1740 1 --2- 172.21.15.67:0/281677485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5576369c4630 0x5576369cdfb0 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:14.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.533+0000 7f399ade1740 1 --2- 172.21.15.67:0/281677485 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5576369a1190 0x5576369c40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:14.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.533+0000 7f3970ff9640 0 client.7917 ms_handle_reset on v2:172.21.15.67:6832/1336395002 2024-03-24T00:45:14.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.533+0000 7f399ade1740 1 --2- 172.21.15.67:0/281677485 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5576369a0db0 0x5576369beea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:14.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.533+0000 7f399ade1740 1 -- 172.21.15.67:0/281677485 >> 172.21.15.67:0/281677485 conn(0x557636967a60 msgr2=0x5576369caac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:14.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.533+0000 7f399ade1740 1 -- 172.21.15.67:0/281677485 shutdown_connections 2024-03-24T00:45:14.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.533+0000 7f399ade1740 1 -- 172.21.15.67:0/281677485 wait complete. 2024-03-24T00:45:14.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.533+0000 7f399ade1740 20 client.7917 trim_cache size 0 max 16384 2024-03-24T00:45:14.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.550+0000 7f399ade1740 1 Processor -- start 2024-03-24T00:45:14.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.550+0000 7f399ade1740 1 -- start start 2024-03-24T00:45:14.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.551+0000 7f399ade1740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5576369a0db0 0x5576369e7d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:14.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.551+0000 7f399ade1740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5576369a1190 0x5576369c3a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:14.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.551+0000 7f399ade1740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5576369c4630 0x5576369c3f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:14.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.551+0000 7f399ade1740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39901fd620 con 0x5576369a0db0 2024-03-24T00:45:14.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.551+0000 7f399ade1740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39900cb190 con 0x5576369c4630 2024-03-24T00:45:14.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.551+0000 7f399ade1740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39901fd020 con 0x5576369a1190 2024-03-24T00:45:14.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.551+0000 7f3995f9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5576369a0db0 0x5576369e7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:14.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.551+0000 7f3995f9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5576369a0db0 0x5576369e7d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41448/0 (socket says 172.21.15.67:41448) 2024-03-24T00:45:14.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.551+0000 7f3995f9b640 1 -- 172.21.15.67:0/2116651047 learned_addr learned my addr 172.21.15.67:0/2116651047 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:14.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.551+0000 7f3995f9b640 1 -- 172.21.15.67:0/2116651047 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5576369a1190 msgr2=0x5576369c3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:14.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.551+0000 7f3995f9b640 1 --2- 172.21.15.67:0/2116651047 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5576369a1190 0x5576369c3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:14.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.551+0000 7f3995f9b640 1 -- 172.21.15.67:0/2116651047 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5576369c4630 msgr2=0x5576369c3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:14.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.551+0000 7f3995f9b640 1 --2- 172.21.15.67:0/2116651047 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5576369c4630 0x5576369c3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:14.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.551+0000 7f3995f9b640 1 -- 172.21.15.67:0/2116651047 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5576366e36b0 con 0x5576369a0db0 2024-03-24T00:45:14.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.552+0000 7f3995f9b640 1 --2- 172.21.15.67:0/2116651047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5576369a0db0 0x5576369e7d20 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x5576369c0cc0 tx=0x7f398c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b61706586b034a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:14.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.552+0000 7f3972ffd640 1 -- 172.21.15.67:0/2116651047 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398c02c070 con 0x5576369a0db0 2024-03-24T00:45:14.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.552+0000 7f3972ffd640 1 -- 172.21.15.67:0/2116651047 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f398c0266f0 con 0x5576369a0db0 2024-03-24T00:45:14.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.552+0000 7f3972ffd640 1 -- 172.21.15.67:0/2116651047 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398c00f040 con 0x5576369a0db0 2024-03-24T00:45:14.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.552+0000 7f399ade1740 1 -- 172.21.15.67:0/2116651047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5576369a0db0 msgr2=0x5576369e7d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:14.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.552+0000 7f399ade1740 1 --2- 172.21.15.67:0/2116651047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5576369a0db0 0x5576369e7d20 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x5576369c0cc0 tx=0x7f398c014000 comp rx=0 tx=0).stop 2024-03-24T00:45:14.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.552+0000 7f399ade1740 1 -- 172.21.15.67:0/2116651047 shutdown_connections 2024-03-24T00:45:14.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.552+0000 7f399ade1740 1 --2- 172.21.15.67:0/2116651047 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5576369c4630 0x5576369c3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:14.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.552+0000 7f399ade1740 1 --2- 172.21.15.67:0/2116651047 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5576369a1190 0x5576369c3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:14.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.552+0000 7f399ade1740 1 --2- 172.21.15.67:0/2116651047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5576369a0db0 0x5576369e7d20 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:14.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.552+0000 7f399ade1740 1 -- 172.21.15.67:0/2116651047 >> 172.21.15.67:0/2116651047 conn(0x557636967a60 msgr2=0x5576369bf8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:14.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.552+0000 7f399ade1740 1 -- 172.21.15.67:0/2116651047 shutdown_connections 2024-03-24T00:45:14.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.552+0000 7f399ade1740 1 -- 172.21.15.67:0/2116651047 wait complete. 2024-03-24T00:45:14.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.553+0000 7f399ade1740 1 -- start start 2024-03-24T00:45:14.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.553+0000 7f399ade1740 1 Processor -- start 2024-03-24T00:45:14.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.554+0000 7f399ade1740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5576369a0db0 0x557636a44410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:14.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.554+0000 7f399ade1740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5576369a1190 0x5576369c13c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:14.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.554+0000 7f399ade1740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5576369c4630 0x5576369c1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:14.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.554+0000 7f399ade1740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576369c36e0 con 0x5576369a0db0 2024-03-24T00:45:14.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.554+0000 7f399ade1740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576369c7450 con 0x5576369c4630 2024-03-24T00:45:14.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.554+0000 7f399ade1740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5576369c07e0 con 0x5576369a1190 2024-03-24T00:45:14.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.554+0000 7f399ade1740 10 client.0 _ll_register_callbacks cb 0x7f3980058af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:45:14.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.554+0000 7f3995f9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5576369a0db0 0x557636a44410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:14.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.554+0000 7f3995f9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5576369a0db0 0x557636a44410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41450/0 (socket says 172.21.15.67:41450) 2024-03-24T00:45:14.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.554+0000 7f3995f9b640 1 -- 172.21.15.67:0/2310850152 learned_addr learned my addr 172.21.15.67:0/2310850152 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:14.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.555+0000 7f399579a640 1 --2- 172.21.15.67:0/2310850152 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5576369a1190 0x5576369c13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:14.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.555+0000 7f399679c640 1 --2- 172.21.15.67:0/2310850152 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5576369c4630 0x5576369c1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:14.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.555+0000 7f3995f9b640 1 -- 172.21.15.67:0/2310850152 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5576369a1190 msgr2=0x5576369c13c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:14.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.555+0000 7f3995f9b640 1 --2- 172.21.15.67:0/2310850152 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5576369a1190 0x5576369c13c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:14.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.555+0000 7f3995f9b640 1 -- 172.21.15.67:0/2310850152 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5576369c4630 msgr2=0x5576369c1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:14.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.555+0000 7f3995f9b640 1 --2- 172.21.15.67:0/2310850152 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5576369c4630 0x5576369c1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:14.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.555+0000 7f3995f9b640 1 -- 172.21.15.67:0/2310850152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557636a44950 con 0x5576369a0db0 2024-03-24T00:45:14.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.555+0000 7f399679c640 1 --2- 172.21.15.67:0/2310850152 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5576369c4630 0x5576369c1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:45:14.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.555+0000 7f3995f9b640 1 --2- 172.21.15.67:0/2310850152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5576369a0db0 0x557636a44410 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f398c008010 tx=0x7f398c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=686b6ca827a913ea server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:14.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.555+0000 7f3970ff9640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:45:14.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.555+0000 7f3970ff9640 1 -- 172.21.15.67:0/2310850152 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398c02c040 con 0x5576369a0db0 2024-03-24T00:45:14.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.555+0000 7f3970ff9640 1 -- 172.21.15.67:0/2310850152 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f398c026aa0 con 0x5576369a0db0 2024-03-24T00:45:14.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.555+0000 7f399ade1740 1 -- 172.21.15.67:0/2310850152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f398c0266f0 con 0x5576369a0db0 2024-03-24T00:45:14.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.555+0000 7f3970ff9640 1 -- 172.21.15.67:0/2310850152 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) v2 ==== 24+0+0 (secure 0 0 0) 0x7f398c03d050 con 0x5576369a0db0 2024-03-24T00:45:14.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.555+0000 7f399ade1740 10 client.7923 fetch_fsmap learned FSMap version 60 2024-03-24T00:45:14.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.555+0000 7f399ade1740 1 -- 172.21.15.67:0/2310850152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=2+}) v3 -- 0x557636a4eb60 con 0x5576369a0db0 2024-03-24T00:45:14.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.555+0000 7f3970ff9640 1 -- 172.21.15.67:0/2310850152 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f398c0266f0 con 0x5576369a0db0 2024-03-24T00:45:14.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.556+0000 7f3970ff9640 1 -- 172.21.15.67:0/2310850152 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 60) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f398c040020 con 0x5576369a0db0 2024-03-24T00:45:14.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.556+0000 7f399ade1740 10 client.7923 fetch_fsmap finished waiting for FSMap version 60 2024-03-24T00:45:14.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.556+0000 7f399ade1740 10 client.7923 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T00:45:14.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.556+0000 7f399ade1740 20 client.7923 populate_metadata read hostname 'smithi067' 2024-03-24T00:45:14.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.556+0000 7f399ade1740 1 --2- 172.21.15.67:0/2310850152 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x557636a4b2b0 0x5576369c2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:14.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.556+0000 7f399ade1740 4 client.7923 mds_command: new command op to 7860 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:45:14.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.556+0000 7f399ade1740 1 -- 172.21.15.67:0/2310850152 --> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55763690c4b0 con 0x557636a4b2b0 2024-03-24T00:45:14.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.556+0000 7f399579a640 1 --2- 172.21.15.67:0/2310850152 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x557636a4b2b0 0x5576369c2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:14.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.556+0000 7f3970ff9640 10 client.7923 ms_handle_connect on v2:172.21.15.67:6832/1336395002 2024-03-24T00:45:14.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.556+0000 7f399579a640 1 --2- 172.21.15.67:0/2310850152 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x557636a4b2b0 0x5576369c2500 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=adafcc97aa8569a4 server_cookie=65310c285967ab9a in_seq=0 out_seq=0 2024-03-24T00:45:14.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.556+0000 7f3970ff9640 1 -- 172.21.15.67:0/2310850152 <== mds.0 v2:172.21.15.67:6832/1336395002 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55763690c4b0 con 0x557636a4b2b0 2024-03-24T00:45:14.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.556+0000 7f3970ff9640 10 client.7923 handle_command_reply: tid=0 2024-03-24T00:45:14.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.556+0000 7f399ade1740 1 client.7923 shutdown 2024-03-24T00:45:14.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.556+0000 7f399ade1740 1 -- 172.21.15.67:0/2310850152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5576369a0db0 msgr2=0x557636a44410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:14.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.556+0000 7f399ade1740 1 --2- 172.21.15.67:0/2310850152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5576369a0db0 0x557636a44410 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f398c008010 tx=0x7f398c004730 comp rx=0 tx=0).stop 2024-03-24T00:45:14.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.556+0000 7f399ade1740 1 -- 172.21.15.67:0/2310850152 shutdown_connections 2024-03-24T00:45:14.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.556+0000 7f399ade1740 1 --2- 172.21.15.67:0/2310850152 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x557636a4b2b0 0x5576369c2500 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:14.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.556+0000 7f399ade1740 1 --2- 172.21.15.67:0/2310850152 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5576369c4630 0x5576369c1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:14.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.556+0000 7f399ade1740 1 --2- 172.21.15.67:0/2310850152 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5576369a1190 0x5576369c13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:14.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.556+0000 7f3970ff9640 0 client.7923 ms_handle_reset on v2:172.21.15.67:6832/1336395002 2024-03-24T00:45:14.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.556+0000 7f399ade1740 1 --2- 172.21.15.67:0/2310850152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5576369a0db0 0x557636a44410 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:14.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.556+0000 7f399ade1740 1 -- 172.21.15.67:0/2310850152 >> 172.21.15.67:0/2310850152 conn(0x557636967a60 msgr2=0x557636a4f3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:14.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.556+0000 7f399ade1740 1 -- 172.21.15.67:0/2310850152 shutdown_connections 2024-03-24T00:45:14.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.556+0000 7f399ade1740 1 -- 172.21.15.67:0/2310850152 wait complete. 2024-03-24T00:45:14.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.557+0000 7f399ade1740 20 client.7923 trim_cache size 0 max 16384 2024-03-24T00:45:14.566 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:45:14.566 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "de1d0178-808e-49ca-a986-e297e523b976", 2024-03-24T00:45:14.566 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-03-24T00:45:14.566 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 7860, 2024-03-24T00:45:14.566 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-03-24T00:45:14.566 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-03-24T00:45:14.566 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-03-24T00:45:14.566 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 4.4409688200000002, 2024-03-24T00:45:14.566 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 60, 2024-03-24T00:45:14.566 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 90, 2024-03-24T00:45:14.567 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 90, 2024-03-24T00:45:14.567 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 5.4172486810000002 2024-03-24T00:45:14.567 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:45:14.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.557+0000 7f3996f9d640 1 -- 172.21.15.67:0/3062576509 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f397406bc00 msgr2=0x7f397406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:14.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.557+0000 7f3996f9d640 1 --2- 172.21.15.67:0/3062576509 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f397406bc00 0x7f397406e0c0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f398c006fd0 tx=0x7f398c008040 comp rx=0 tx=0).stop 2024-03-24T00:45:14.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.557+0000 7f3996f9d640 1 -- 172.21.15.67:0/3062576509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f399015bec0 msgr2=0x7f39900af620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:14.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.557+0000 7f3996f9d640 1 --2- 172.21.15.67:0/3062576509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f399015bec0 0x7f39900af620 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f398000c430 tx=0x7f398000c8f0 comp rx=0 tx=0).stop 2024-03-24T00:45:14.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.557+0000 7f3996f9d640 1 -- 172.21.15.67:0/3062576509 shutdown_connections 2024-03-24T00:45:14.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.557+0000 7f3996f9d640 1 --2- 172.21.15.67:0/3062576509 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f397406bc00 0x7f397406e0c0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:14.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.557+0000 7f3996f9d640 1 --2- 172.21.15.67:0/3062576509 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f39901603c0 0x7f39900afb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:14.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.557+0000 7f3996f9d640 1 --2- 172.21.15.67:0/3062576509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f399015bec0 0x7f39900af620 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:14.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.557+0000 7f3996f9d640 1 --2- 172.21.15.67:0/3062576509 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f399015bae0 0x7f39901f0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:14.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.557+0000 7f3996f9d640 1 -- 172.21.15.67:0/3062576509 >> 172.21.15.67:0/3062576509 conn(0x7f399008afc0 msgr2=0x7f399008b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:14.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.557+0000 7f3996f9d640 1 -- 172.21.15.67:0/3062576509 shutdown_connections 2024-03-24T00:45:14.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.557+0000 7f3996f9d640 1 -- 172.21.15.67:0/3062576509 wait complete. 2024-03-24T00:45:14.569 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T00:45:14.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.669+0000 7fbef5c27640 1 Processor -- start 2024-03-24T00:45:14.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.670+0000 7fbef5c27640 1 -- start start 2024-03-24T00:45:14.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.670+0000 7fbef5c27640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fbef015e2a0 0x7fbef00ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:14.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.670+0000 7fbef5c27640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbef00cf640 con 0x7fbef015dec0 2024-03-24T00:45:14.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.670+0000 7fbef5c27640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbef00cf080 con 0x7fbef00d0ef0 2024-03-24T00:45:14.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.670+0000 7fbef5c27640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbef00cee30 con 0x7fbef015e2a0 2024-03-24T00:45:14.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.670+0000 7fbeef7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbef015dec0 0x7fbef00d07e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56214/0 (socket says 172.21.15.67:56214) 2024-03-24T00:45:14.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.670+0000 7fbeef7fe640 1 -- 172.21.15.67:0/3036839454 learned_addr learned my addr 172.21.15.67:0/3036839454 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:45:14.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.671+0000 7fbeeeffd640 1 --2- 172.21.15.67:0/3036839454 >> v2:172.21.15.72:3300/0 conn(0x7fbef015e2a0 0x7fbef00ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:14.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.671+0000 7fbeee7fc640 1 -- 172.21.15.67:0/3036839454 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2506279683 0 0) 0x7fbee0001110 con 0x7fbef015dec0 2024-03-24T00:45:14.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.671+0000 7fbeee7fc640 1 -- 172.21.15.67:0/3036839454 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbedc003520 con 0x7fbef015dec0 2024-03-24T00:45:14.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.671+0000 7fbeeeffd640 1 -- 172.21.15.67:0/3036839454 >> v1:172.21.15.72:6789/0 conn(0x7fbef00d0ef0 legacy=0x7fbef00d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:14.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.671+0000 7fbeeeffd640 1 -- 172.21.15.67:0/3036839454 >> v1:172.21.15.67:6789/0 conn(0x7fbef015dec0 legacy=0x7fbef00d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:14.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.671+0000 7fbeee7fc640 1 -- 172.21.15.67:0/3036839454 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2521716400 0 0) 0x7fbef00cf080 con 0x7fbef00d0ef0 2024-03-24T00:45:14.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.671+0000 7fbeeeffd640 1 -- 172.21.15.67:0/3036839454 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef00cf360 con 0x7fbef015e2a0 2024-03-24T00:45:14.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.672+0000 7fbeeeffd640 1 --2- 172.21.15.67:0/3036839454 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbef015e2a0 0x7fbef00ce630 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fbef00cefb0 tx=0x7fbed802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=edb1694efe2fa3c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:14.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.672+0000 7fbeee7fc640 1 -- 172.21.15.67:0/3036839454 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed8035020 con 0x7fbef015e2a0 2024-03-24T00:45:14.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.672+0000 7fbeee7fc640 1 -- 172.21.15.67:0/3036839454 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbed8031830 con 0x7fbef015e2a0 2024-03-24T00:45:14.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.672+0000 7fbeee7fc640 1 -- 172.21.15.67:0/3036839454 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed8031c00 con 0x7fbef015e2a0 2024-03-24T00:45:14.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.673+0000 7fbef5c27640 1 -- 172.21.15.67:0/3036839454 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbef015e2a0 msgr2=0x7fbef00ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:14.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.673+0000 7fbef5c27640 1 --2- 172.21.15.67:0/3036839454 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbef015e2a0 0x7fbef00ce630 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fbef00cefb0 tx=0x7fbed802b8d0 comp rx=0 tx=0).stop 2024-03-24T00:45:14.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.673+0000 7fbef5c27640 1 -- 172.21.15.67:0/3036839454 shutdown_connections 2024-03-24T00:45:14.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.673+0000 7fbef5c27640 1 --2- 172.21.15.67:0/3036839454 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbef015e2a0 0x7fbef00ce630 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:14.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.673+0000 7fbef5c27640 1 -- 172.21.15.67:0/3036839454 >> 172.21.15.67:0/3036839454 conn(0x7fbef006c810 msgr2=0x7fbef006cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:14.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.673+0000 7fbef5c27640 1 -- 172.21.15.67:0/3036839454 shutdown_connections 2024-03-24T00:45:14.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.673+0000 7fbef5c27640 1 -- 172.21.15.67:0/3036839454 wait complete. 2024-03-24T00:45:14.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.675+0000 7fbef5c27640 1 Processor -- start 2024-03-24T00:45:14.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.675+0000 7fbef5c27640 1 -- start start 2024-03-24T00:45:14.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.676+0000 7fbef5c27640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbef00d0ef0 0x7fbef00b4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:14.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.676+0000 7fbef5c27640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbef015dec0 0x7fbef0160ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:14.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.676+0000 7fbef5c27640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbef015e2a0 0x7fbef00b5b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:14.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.676+0000 7fbef5c27640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef0162a40 con 0x7fbef015dec0 2024-03-24T00:45:14.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.676+0000 7fbef5c27640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef01628c0 con 0x7fbef00d0ef0 2024-03-24T00:45:14.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.676+0000 7fbef5c27640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbef00d2120 con 0x7fbef015e2a0 2024-03-24T00:45:14.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.676+0000 7fbeeeffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbef015dec0 0x7fbef0160ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:14.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.676+0000 7fbeeeffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbef015dec0 0x7fbef0160ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41458/0 (socket says 172.21.15.67:41458) 2024-03-24T00:45:14.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.676+0000 7fbeeeffd640 1 -- 172.21.15.67:0/1680621226 learned_addr learned my addr 172.21.15.67:0/1680621226 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:14.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.676+0000 7fbeeeffd640 1 -- 172.21.15.67:0/1680621226 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbef015e2a0 msgr2=0x7fbef00b5b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:14.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.676+0000 7fbeeffff640 1 --2- 172.21.15.67:0/1680621226 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbef015e2a0 0x7fbef00b5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:14.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.676+0000 7fbeef7fe640 1 --2- 172.21.15.67:0/1680621226 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbef00d0ef0 0x7fbef00b4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:14.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.676+0000 7fbeeeffd640 1 --2- 172.21.15.67:0/1680621226 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbef015e2a0 0x7fbef00b5b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:14.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.676+0000 7fbeeeffd640 1 -- 172.21.15.67:0/1680621226 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbef00d0ef0 msgr2=0x7fbef00b4220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:14.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.676+0000 7fbeeeffd640 1 --2- 172.21.15.67:0/1680621226 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbef00d0ef0 0x7fbef00b4220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:14.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.676+0000 7fbeeeffd640 1 -- 172.21.15.67:0/1680621226 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef00b4a70 con 0x7fbef015dec0 2024-03-24T00:45:14.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.677+0000 7fbeeeffd640 1 --2- 172.21.15.67:0/1680621226 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbef015dec0 0x7fbef0160ab0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7fbed8009890 tx=0x7fbed80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:14.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.677+0000 7fbee7fff640 1 -- 172.21.15.67:0/1680621226 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed8040070 con 0x7fbef015dec0 2024-03-24T00:45:14.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.677+0000 7fbee7fff640 1 -- 172.21.15.67:0/1680621226 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbed803a910 con 0x7fbef015dec0 2024-03-24T00:45:14.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.677+0000 7fbef5c27640 1 -- 172.21.15.67:0/1680621226 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbef00b4d50 con 0x7fbef015dec0 2024-03-24T00:45:14.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.677+0000 7fbee7fff640 1 -- 172.21.15.67:0/1680621226 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed8035040 con 0x7fbef015dec0 2024-03-24T00:45:14.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.677+0000 7fbef5c27640 1 -- 172.21.15.67:0/1680621226 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbef00b4760 con 0x7fbef015dec0 2024-03-24T00:45:14.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.678+0000 7fbef5c27640 1 -- 172.21.15.67:0/1680621226 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbef004f400 con 0x7fbef015dec0 2024-03-24T00:45:14.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.679+0000 7fbee7fff640 1 -- 172.21.15.67:0/1680621226 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fbed8038020 con 0x7fbef015dec0 2024-03-24T00:45:14.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.679+0000 7fbee7fff640 1 --2- 172.21.15.67:0/1680621226 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbed006bbd0 0x7fbed006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:14.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.679+0000 7fbee7fff640 1 -- 172.21.15.67:0/1680621226 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fbed80ac4d0 con 0x7fbef015dec0 2024-03-24T00:45:14.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.679+0000 7fbeef7fe640 1 --2- 172.21.15.67:0/1680621226 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbed006bbd0 0x7fbed006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:14.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.680+0000 7fbeef7fe640 1 --2- 172.21.15.67:0/1680621226 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbed006bbd0 0x7fbed006e090 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fbee00012b0 tx=0x7fbee0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:14.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.683+0000 7fbee7fff640 1 -- 172.21.15.67:0/1680621226 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fbed807ae70 con 0x7fbef015dec0 2024-03-24T00:45:14.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.904+0000 7fbef5c27640 1 -- 172.21.15.67:0/1680621226 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fbef00b9500 con 0x7fbef015dec0 2024-03-24T00:45:14.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.904+0000 7fbee7fff640 1 -- 172.21.15.67:0/1680621226 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v60) v1 ==== 53+0+73 (secure 0 0 0) 0x7fbed807a770 con 0x7fbef015dec0 2024-03-24T00:45:14.905 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-24T00:45:14.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.906+0000 7fbef5c27640 1 -- 172.21.15.67:0/1680621226 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbed006bbd0 msgr2=0x7fbed006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:14.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.906+0000 7fbef5c27640 1 --2- 172.21.15.67:0/1680621226 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbed006bbd0 0x7fbed006e090 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fbee00012b0 tx=0x7fbee0009040 comp rx=0 tx=0).stop 2024-03-24T00:45:14.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.906+0000 7fbef5c27640 1 -- 172.21.15.67:0/1680621226 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbef015dec0 msgr2=0x7fbef0160ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:14.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.906+0000 7fbef5c27640 1 --2- 172.21.15.67:0/1680621226 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbef015dec0 0x7fbef0160ab0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7fbed8009890 tx=0x7fbed80046c0 comp rx=0 tx=0).stop 2024-03-24T00:45:14.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.906+0000 7fbef5c27640 1 -- 172.21.15.67:0/1680621226 shutdown_connections 2024-03-24T00:45:14.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.906+0000 7fbef5c27640 1 --2- 172.21.15.67:0/1680621226 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbed006bbd0 0x7fbed006e090 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:14.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.906+0000 7fbef5c27640 1 --2- 172.21.15.67:0/1680621226 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbef015e2a0 0x7fbef00b5b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:14.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.906+0000 7fbef5c27640 1 --2- 172.21.15.67:0/1680621226 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbef015dec0 0x7fbef0160ab0 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:14.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.906+0000 7fbef5c27640 1 --2- 172.21.15.67:0/1680621226 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbef00d0ef0 0x7fbef00b4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:14.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.906+0000 7fbef5c27640 1 -- 172.21.15.67:0/1680621226 >> 172.21.15.67:0/1680621226 conn(0x7fbef006c810 msgr2=0x7fbef015e680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:14.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.906+0000 7fbef5c27640 1 -- 172.21.15.67:0/1680621226 shutdown_connections 2024-03-24T00:45:14.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:14.906+0000 7fbef5c27640 1 -- 172.21.15.67:0/1680621226 wait complete. 2024-03-24T00:45:14.918 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-24T00:45:14.918 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T00:45:14.918 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-03-24T00:45:14.918 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-24T00:45:14.918 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T00:45:14.918 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-24T00:45:14.918 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-24T00:45:14.918 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T00:45:14.919 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:45:14.919 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:45:14.919 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-03-24T00:45:14.934 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T00:45:14.935 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T00:45:14.935 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T00:45:14.935 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:45:14.935 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-03-24T00:45:14.935 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:45:14.935 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:45:14.935 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-03-24T00:45:14.935 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-03-24T00:45:14.935 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:45:14.935 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-03-24T00:45:14.935 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-03-24T00:45:14.935 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:45:14.935 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-03-24T00:45:14.935 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T00:45:14.935 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-03-24T00:45:14.936 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-24T00:45:14.936 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 42198sec preferred_lft 42198sec 2024-03-24T00:45:14.936 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-03-24T00:45:14.936 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:45:14.936 INFO:teuthology.orchestra.run.smithi067.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T00:45:14.936 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 96:cf:26:c8:bf:62 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:45:14.936 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-24T00:45:14.936 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:45:14.936 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::7c6a:5fff:feaa:d41/64 scope link 2024-03-24T00:45:14.936 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:45:14.936 INFO:teuthology.orchestra.run.smithi067.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T00:45:14.936 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ea:22:4b:d5:48:3d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:45:14.936 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::e822:4bff:fed5:483d/64 scope link 2024-03-24T00:45:14.936 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:45:14.936 INFO:teuthology.orchestra.run.smithi067.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T00:45:14.937 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 96:cf:26:c8:bf:62 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:45:14.937 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::94cf:26ff:fec8:bf62/64 scope link 2024-03-24T00:45:14.937 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:45:14.937 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:45:14.937 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T00:45:15.000 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-03-24T00:45:15.002 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:45:15.002 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-03-24T00:45:15.058 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-03-24T00:45:15.058 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T00:45:15.058 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-24T00:45:15.058 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-24T00:45:15.058 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T00:45:15.058 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:45:15.059 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-03-24T00:45:15.059 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-03-24T00:45:15.059 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-24T00:45:15.059 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-24T00:45:15.059 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-24T00:45:15.059 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-03-24T00:45:15.155 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:45:15.156 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-03-24T00:45:15.172 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-24T00:45:15.172 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-24T00:45:15.172 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:45:15.172 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:45:15.172 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-03-24T00:45:15.239 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:45:15.239 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-24T00:45:15.294 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:45:15.294 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:45:15.294 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:45:15.348 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-24T00:45:15.348 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:45:15.348 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:45:15.435 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:45:15.436 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T00:45:15.436 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-03-24T00:45:15.444 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:45:15.444 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T00:45:15.444 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-03-24T00:45:15.444 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.72:3300,172.21.15.72:3301" 2024-03-24T00:45:15.444 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-24T00:45:15.449 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:45:15.449 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T00:45:15.449 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-03-24T00:45:15.449 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.72:3300,172.21.15.72:3301" 2024-03-24T00:45:15.449 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-24T00:45:15.449 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:6789,172.21.15.72:6789,172.21.15.72:6790" 2024-03-24T00:45:15.449 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.67:6789,172.21.15.72:6789,172.21.15.72:6790:/ 2024-03-24T00:45:15.449 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=de1d0178-808e-49ca-a986-e297e523b976" will pass to kernel 2024-03-24T00:45:15.450 INFO:teuthology.orchestra.run.smithi067.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-24T00:45:15.450 INFO:teuthology.orchestra.run.smithi067.stdout: You just mounted a file system that supports labels which does not 2024-03-24T00:45:15.450 INFO:teuthology.orchestra.run.smithi067.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-24T00:45:15.450 INFO:teuthology.orchestra.run.smithi067.stdout: applications will generate AVC messages and not be allowed access to 2024-03-24T00:45:15.450 INFO:teuthology.orchestra.run.smithi067.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-24T00:45:15.452 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-24T00:45:15.452 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:45:15.452 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:45:15.483 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:45:15.483 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T00:45:15.539 INFO:teuthology.orchestra.run.smithi067.stdout:client7932 2024-03-24T00:45:15.540 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:45:15.540 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:45:15.595 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:45:15.595 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat testdir/hardlink) 2024-03-24T00:45:15.652 INFO:tasks.ceph.mds.c.smithi067.stderr:2024-03-24T00:45:15.651+0000 7fdf3050a640 -1 log_channel(cluster) log [ERR] : dir 0x10000000000 object missing on disk; some files may be lost (/dir) 2024-03-24T00:45:15.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:45:15.653 INFO:teuthology.orchestra.run.smithi067.stderr:stat: cannot statx 'testdir/hardlink': Input/output error 2024-03-24T00:45:15.653 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:45:15.654 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.6:0 scrub start /dir recursive 2024-03-24T00:45:15.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.801+0000 7f5470c7e640 1 Processor -- start 2024-03-24T00:45:15.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.801+0000 7f5470c7e640 1 -- start start 2024-03-24T00:45:15.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.802+0000 7f5470c7e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f546c15dfc0 0x7f546c1624c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:15.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.802+0000 7f5470c7e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f546c0d0d50 0x7f546c0d2b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:15.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.802+0000 7f5470c7e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f546c162a00 con 0x7f546c0ced60 2024-03-24T00:45:15.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.802+0000 7f5470c7e640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f546c15e3a0 con 0x7f546c15dfc0 2024-03-24T00:45:15.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.802+0000 7f5470c7e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f546c0cf4d0 con 0x7f546c0d0d50 2024-03-24T00:45:15.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.803+0000 7f546a59c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f546c15dfc0 0x7f546c1624c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:15.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.803+0000 7f5469d9b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f546c0ced60 0x7f546c0d0640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56230/0 (socket says 172.21.15.67:56230) 2024-03-24T00:45:15.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.803+0000 7f5469d9b640 1 -- 172.21.15.67:0/1584315859 learned_addr learned my addr 172.21.15.67:0/1584315859 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:45:15.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.803+0000 7f546ad9d640 1 --2- 172.21.15.67:0/1584315859 >> v2:172.21.15.72:3300/0 conn(0x7f546c0d0d50 0x7f546c0d2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:15.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.803+0000 7f546a59c640 1 -- 172.21.15.67:0/1584315859 >> v2:172.21.15.72:3300/0 conn(0x7f546c0d0d50 msgr2=0x7f546c0d2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:15.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.803+0000 7f546a59c640 1 --2- 172.21.15.67:0/1584315859 >> v2:172.21.15.72:3300/0 conn(0x7f546c0d0d50 0x7f546c0d2b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.803+0000 7f546a59c640 1 -- 172.21.15.67:0/1584315859 >> v1:172.21.15.67:6789/0 conn(0x7f546c0ced60 legacy=0x7f546c0d0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:15.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.803+0000 7f546a59c640 1 -- 172.21.15.67:0/1584315859 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f546c0cf1f0 con 0x7f546c15dfc0 2024-03-24T00:45:15.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.804+0000 7f546a59c640 1 --2- 172.21.15.67:0/1584315859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f546c15dfc0 0x7f546c1624c0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f546c0d0250 tx=0x7f545c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18acd3729229118f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:15.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.804+0000 7f546959a640 1 -- 172.21.15.67:0/1584315859 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f545c035020 con 0x7f546c15dfc0 2024-03-24T00:45:15.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.804+0000 7f546959a640 1 -- 172.21.15.67:0/1584315859 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f545c031830 con 0x7f546c15dfc0 2024-03-24T00:45:15.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.804+0000 7f546959a640 1 -- 172.21.15.67:0/1584315859 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f545c031c00 con 0x7f546c15dfc0 2024-03-24T00:45:15.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.805+0000 7f5470c7e640 1 -- 172.21.15.67:0/1584315859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f546c15dfc0 msgr2=0x7f546c1624c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:15.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.805+0000 7f5470c7e640 1 --2- 172.21.15.67:0/1584315859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f546c15dfc0 0x7f546c1624c0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f546c0d0250 tx=0x7f545c02b8d0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.805+0000 7f5470c7e640 1 -- 172.21.15.67:0/1584315859 shutdown_connections 2024-03-24T00:45:15.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.805+0000 7f5470c7e640 1 --2- 172.21.15.67:0/1584315859 >> v2:172.21.15.72:3300/0 conn(0x7f546c0d0d50 0x7f546c0d2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.805+0000 7f5470c7e640 1 --2- 172.21.15.67:0/1584315859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f546c15dfc0 0x7f546c1624c0 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.805+0000 7f5470c7e640 1 -- 172.21.15.67:0/1584315859 >> 172.21.15.67:0/1584315859 conn(0x7f546c06c880 msgr2=0x7f546c06cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:15.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.805+0000 7f5470c7e640 1 -- 172.21.15.67:0/1584315859 shutdown_connections 2024-03-24T00:45:15.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.805+0000 7f5470c7e640 1 -- 172.21.15.67:0/1584315859 wait complete. 2024-03-24T00:45:15.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.806+0000 7f5470c7e640 1 Processor -- start 2024-03-24T00:45:15.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.806+0000 7f5470c7e640 1 -- start start 2024-03-24T00:45:15.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.807+0000 7f5470c7e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f546c0d0d50 0x7f546c1f4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:15.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.807+0000 7f5470c7e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f546c15dfc0 0x7f546c0cd710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:15.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.807+0000 7f5470c7e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f546c1f7cb0 0x7f546c0cb7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:15.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.807+0000 7f5470c7e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f546c0d25e0 con 0x7f546c1f7cb0 2024-03-24T00:45:15.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.807+0000 7f5470c7e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f546c0d2300 con 0x7f546c15dfc0 2024-03-24T00:45:15.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.807+0000 7f5470c7e640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f546c0b6ea0 con 0x7f546c0d0d50 2024-03-24T00:45:15.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.807+0000 7f546ad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f546c1f7cb0 0x7f546c0cb7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:15.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.807+0000 7f546ad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f546c1f7cb0 0x7f546c0cb7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41476/0 (socket says 172.21.15.67:41476) 2024-03-24T00:45:15.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.807+0000 7f546ad9d640 1 -- 172.21.15.67:0/3933287827 learned_addr learned my addr 172.21.15.67:0/3933287827 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:15.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.807+0000 7f546a59c640 1 --2- 172.21.15.67:0/3933287827 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f546c0d0d50 0x7f546c1f4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:15.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.807+0000 7f5469d9b640 1 --2- 172.21.15.67:0/3933287827 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f546c15dfc0 0x7f546c0cd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:15.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.807+0000 7f546ad9d640 1 -- 172.21.15.67:0/3933287827 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f546c0d0d50 msgr2=0x7f546c1f4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:15.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.807+0000 7f546ad9d640 1 --2- 172.21.15.67:0/3933287827 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f546c0d0d50 0x7f546c1f4090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.807+0000 7f546ad9d640 1 -- 172.21.15.67:0/3933287827 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f546c15dfc0 msgr2=0x7f546c0cd710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:15.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.807+0000 7f546ad9d640 1 --2- 172.21.15.67:0/3933287827 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f546c15dfc0 0x7f546c0cd710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.807+0000 7f546ad9d640 1 -- 172.21.15.67:0/3933287827 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f546c1f8140 con 0x7f546c1f7cb0 2024-03-24T00:45:15.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.807+0000 7f546ad9d640 1 --2- 172.21.15.67:0/3933287827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f546c1f7cb0 0x7f546c0cb7e0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f5464007de0 tx=0x7f546400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:15.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.808+0000 7f5462ffd640 1 -- 172.21.15.67:0/3933287827 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5464014620 con 0x7f546c1f7cb0 2024-03-24T00:45:15.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.808+0000 7f5462ffd640 1 -- 172.21.15.67:0/3933287827 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5464019070 con 0x7f546c1f7cb0 2024-03-24T00:45:15.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.808+0000 7f5470c7e640 1 -- 172.21.15.67:0/3933287827 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f546c1f8420 con 0x7f546c1f7cb0 2024-03-24T00:45:15.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.808+0000 7f5462ffd640 1 -- 172.21.15.67:0/3933287827 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5464014a20 con 0x7f546c1f7cb0 2024-03-24T00:45:15.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.808+0000 7f5470c7e640 1 -- 172.21.15.67:0/3933287827 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f546c1f89e0 con 0x7f546c1f7cb0 2024-03-24T00:45:15.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.809+0000 7f5462ffd640 1 -- 172.21.15.67:0/3933287827 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5464014bc0 con 0x7f546c1f7cb0 2024-03-24T00:45:15.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.810+0000 7f5462ffd640 1 --2- 172.21.15.67:0/3933287827 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f544c06bbd0 0x7f544c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:15.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.810+0000 7f5462ffd640 1 -- 172.21.15.67:0/3933287827 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f546408b5f0 con 0x7f546c1f7cb0 2024-03-24T00:45:15.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.810+0000 7f546a59c640 1 --2- 172.21.15.67:0/3933287827 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f544c06bbd0 0x7f544c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:15.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.810+0000 7f546a59c640 1 --2- 172.21.15.67:0/3933287827 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f544c06bbd0 0x7f544c06e090 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f545c036000 tx=0x7f545c03c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:15.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.811+0000 7f5473869740 1 Processor -- start 2024-03-24T00:45:15.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.812+0000 7f5473869740 1 -- start start 2024-03-24T00:45:15.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.813+0000 7f5473869740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5556d5299940 0x5556d526cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:15.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.813+0000 7f5473869740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556d5299d20 0x5556d529c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:15.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.813+0000 7f5473869740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5556d526d0b0 0x5556d529cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:15.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.813+0000 7f5473869740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556d51d7070 con 0x5556d5299d20 2024-03-24T00:45:15.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.813+0000 7f5473869740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556d517fbf0 con 0x5556d5299940 2024-03-24T00:45:15.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.813+0000 7f5473869740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5556d51d3110 con 0x5556d526d0b0 2024-03-24T00:45:15.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.813+0000 7f5469d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556d5299d20 0x5556d529c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:15.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.813+0000 7f5469d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556d5299d20 0x5556d529c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41482/0 (socket says 172.21.15.67:41482) 2024-03-24T00:45:15.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.813+0000 7f5469d9b640 1 -- 172.21.15.67:0/1131040034 learned_addr learned my addr 172.21.15.67:0/1131040034 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:15.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.813+0000 7f546a59c640 1 --2- 172.21.15.67:0/1131040034 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5556d5299940 0x5556d526cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:15.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.813+0000 7f546ad9d640 1 --2- 172.21.15.67:0/1131040034 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5556d526d0b0 0x5556d529cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:15.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.813+0000 7f5469d9b640 1 -- 172.21.15.67:0/1131040034 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5556d526d0b0 msgr2=0x5556d529cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:15.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.813+0000 7f5469d9b640 1 --2- 172.21.15.67:0/1131040034 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5556d526d0b0 0x5556d529cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.813+0000 7f5469d9b640 1 -- 172.21.15.67:0/1131040034 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5556d5299940 msgr2=0x5556d526cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:15.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.813+0000 7f5469d9b640 1 --2- 172.21.15.67:0/1131040034 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5556d5299940 0x5556d526cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.813+0000 7f5469d9b640 1 -- 172.21.15.67:0/1131040034 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5556d4fb77b0 con 0x5556d5299d20 2024-03-24T00:45:15.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.813+0000 7f5469d9b640 1 --2- 172.21.15.67:0/1131040034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556d5299d20 0x5556d529c6d0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f5454001530 tx=0x7f54540019f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b58f4356cae1c393 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:15.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.813+0000 7f544bfff640 1 -- 172.21.15.67:0/1131040034 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5454019070 con 0x5556d5299d20 2024-03-24T00:45:15.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.814+0000 7f544bfff640 1 -- 172.21.15.67:0/1131040034 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5454005870 con 0x5556d5299d20 2024-03-24T00:45:15.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.814+0000 7f544bfff640 1 -- 172.21.15.67:0/1131040034 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5454005c20 con 0x5556d5299d20 2024-03-24T00:45:15.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.814+0000 7f5473869740 1 -- 172.21.15.67:0/1131040034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556d5299d20 msgr2=0x5556d529c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:15.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.814+0000 7f5473869740 1 --2- 172.21.15.67:0/1131040034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556d5299d20 0x5556d529c6d0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f5454001530 tx=0x7f54540019f0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.814+0000 7f5473869740 1 -- 172.21.15.67:0/1131040034 shutdown_connections 2024-03-24T00:45:15.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.814+0000 7f5473869740 1 --2- 172.21.15.67:0/1131040034 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5556d526d0b0 0x5556d529cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.814+0000 7f5473869740 1 --2- 172.21.15.67:0/1131040034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556d5299d20 0x5556d529c6d0 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.814+0000 7f5473869740 1 --2- 172.21.15.67:0/1131040034 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5556d5299940 0x5556d526cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.814+0000 7f5473869740 1 -- 172.21.15.67:0/1131040034 >> 172.21.15.67:0/1131040034 conn(0x5556d526ede0 msgr2=0x5556d52b76d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:15.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.814+0000 7f5473869740 1 -- 172.21.15.67:0/1131040034 shutdown_connections 2024-03-24T00:45:15.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.814+0000 7f5473869740 1 -- 172.21.15.67:0/1131040034 wait complete. 2024-03-24T00:45:15.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.815+0000 7f5473869740 1 -- start start 2024-03-24T00:45:15.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.815+0000 7f5473869740 1 Processor -- start 2024-03-24T00:45:15.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.816+0000 7f5473869740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556d5299940 0x5556d52b7d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:15.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.816+0000 7f5473869740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5556d5299d20 0x5556d529b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:15.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.816+0000 7f5473869740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5556d52b5360 0x5556d52b5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:15.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.816+0000 7f5473869740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556d52711f0 con 0x5556d5299940 2024-03-24T00:45:15.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.816+0000 7f5473869740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556d5193040 con 0x5556d52b5360 2024-03-24T00:45:15.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.816+0000 7f5473869740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5556d5273a30 con 0x5556d5299d20 2024-03-24T00:45:15.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.816+0000 7f5473869740 10 client.0 _ll_register_callbacks cb 0x5556d517edf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:45:15.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.816+0000 7f546a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556d5299940 0x5556d52b7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:15.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.816+0000 7f546a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556d5299940 0x5556d52b7d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41488/0 (socket says 172.21.15.67:41488) 2024-03-24T00:45:15.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.817+0000 7f546a59c640 1 -- 172.21.15.67:0/1562459349 learned_addr learned my addr 172.21.15.67:0/1562459349 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:15.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.817+0000 7f5469d9b640 1 --2- 172.21.15.67:0/1562459349 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5556d5299d20 0x5556d529b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:15.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.817+0000 7f546ad9d640 1 --2- 172.21.15.67:0/1562459349 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5556d52b5360 0x5556d52b5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:15.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.817+0000 7f546a59c640 1 -- 172.21.15.67:0/1562459349 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5556d5299d20 msgr2=0x5556d529b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:15.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.817+0000 7f546a59c640 1 --2- 172.21.15.67:0/1562459349 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5556d5299d20 0x5556d529b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.817+0000 7f546a59c640 1 -- 172.21.15.67:0/1562459349 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5556d52b5360 msgr2=0x5556d52b5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:15.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.817+0000 7f546a59c640 1 --2- 172.21.15.67:0/1562459349 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5556d52b5360 0x5556d52b5ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.817+0000 7f546a59c640 1 -- 172.21.15.67:0/1562459349 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5556d5017f40 con 0x5556d5299940 2024-03-24T00:45:15.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.817+0000 7f546ad9d640 1 --2- 172.21.15.67:0/1562459349 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5556d52b5360 0x5556d52b5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:45:15.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.817+0000 7f546a59c640 1 --2- 172.21.15.67:0/1562459349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556d5299940 0x5556d52b7d80 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f545c034040 tx=0x7f545c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=812e405b824fbd00 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:15.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.817+0000 7f54497fa640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:45:15.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.817+0000 7f54497fa640 1 -- 172.21.15.67:0/1562459349 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f545c054060 con 0x5556d5299940 2024-03-24T00:45:15.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.817+0000 7f5473869740 1 -- 172.21.15.67:0/1562459349 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5454005870 con 0x5556d5299940 2024-03-24T00:45:15.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.817+0000 7f54497fa640 1 -- 172.21.15.67:0/1562459349 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f545c047b80 con 0x5556d5299940 2024-03-24T00:45:15.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.817+0000 7f54497fa640 1 -- 172.21.15.67:0/1562459349 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) v2 ==== 24+0+0 (secure 0 0 0) 0x7f545c035040 con 0x5556d5299940 2024-03-24T00:45:15.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.817+0000 7f5473869740 10 client.7947 fetch_fsmap learned FSMap version 60 2024-03-24T00:45:15.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.817+0000 7f5473869740 1 -- 172.21.15.67:0/1562459349 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=2+}) v3 -- 0x5556d4eab0a0 con 0x5556d5299940 2024-03-24T00:45:15.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.817+0000 7f54497fa640 1 -- 172.21.15.67:0/1562459349 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5454005870 con 0x5556d5299940 2024-03-24T00:45:15.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.818+0000 7f54497fa640 1 -- 172.21.15.67:0/1562459349 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 60) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f545c04e030 con 0x5556d5299940 2024-03-24T00:45:15.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.818+0000 7f5473869740 10 client.7947 fetch_fsmap finished waiting for FSMap version 60 2024-03-24T00:45:15.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.818+0000 7f5473869740 10 client.7947 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-03-24T00:45:15.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.818+0000 7f5473869740 20 client.7947 populate_metadata read hostname 'smithi067' 2024-03-24T00:45:15.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.818+0000 7f5473869740 1 --2- 172.21.15.67:0/1562459349 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x5556d52cb440 0x5556d52cb820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:15.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.818+0000 7f5473869740 4 client.7947 mds_command: new command op to 7860 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:45:15.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.818+0000 7f5473869740 1 -- 172.21.15.67:0/1562459349 --> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5556d51e45b0 con 0x5556d52cb440 2024-03-24T00:45:15.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.818+0000 7f5469d9b640 1 --2- 172.21.15.67:0/1562459349 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x5556d52cb440 0x5556d52cb820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:15.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.818+0000 7f54497fa640 10 client.7947 ms_handle_connect on v2:172.21.15.67:6832/1336395002 2024-03-24T00:45:15.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.818+0000 7f5469d9b640 1 --2- 172.21.15.67:0/1562459349 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x5556d52cb440 0x5556d52cb820 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb2b290130786c1d server_cookie=45505b0ca7d304cf in_seq=0 out_seq=0 2024-03-24T00:45:15.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.819+0000 7f54497fa640 1 -- 172.21.15.67:0/1562459349 <== mds.0 v2:172.21.15.67:6832/1336395002 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5556d51e45b0 con 0x5556d52cb440 2024-03-24T00:45:15.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.819+0000 7f54497fa640 10 client.7947 handle_command_reply: tid=0 2024-03-24T00:45:15.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.819+0000 7f5473869740 1 client.7947 shutdown 2024-03-24T00:45:15.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.819+0000 7f5473869740 1 -- 172.21.15.67:0/1562459349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556d5299940 msgr2=0x5556d52b7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:15.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.819+0000 7f5473869740 1 --2- 172.21.15.67:0/1562459349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556d5299940 0x5556d52b7d80 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f545c034040 tx=0x7f545c004730 comp rx=0 tx=0).stop 2024-03-24T00:45:15.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.819+0000 7f5473869740 1 -- 172.21.15.67:0/1562459349 shutdown_connections 2024-03-24T00:45:15.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.819+0000 7f5473869740 1 --2- 172.21.15.67:0/1562459349 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x5556d52cb440 0x5556d52cb820 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.819+0000 7f5473869740 1 --2- 172.21.15.67:0/1562459349 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5556d52b5360 0x5556d52b5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.819+0000 7f5473869740 1 --2- 172.21.15.67:0/1562459349 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5556d5299d20 0x5556d529b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.819+0000 7f54497fa640 0 client.7947 ms_handle_reset on v2:172.21.15.67:6832/1336395002 2024-03-24T00:45:15.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.819+0000 7f5473869740 1 --2- 172.21.15.67:0/1562459349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556d5299940 0x5556d52b7d80 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.819+0000 7f5473869740 1 -- 172.21.15.67:0/1562459349 >> 172.21.15.67:0/1562459349 conn(0x5556d526d0b0 msgr2=0x5556d52708c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:15.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.819+0000 7f5473869740 1 -- 172.21.15.67:0/1562459349 shutdown_connections 2024-03-24T00:45:15.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.819+0000 7f5473869740 1 -- 172.21.15.67:0/1562459349 wait complete. 2024-03-24T00:45:15.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.819+0000 7f5473869740 20 client.7947 trim_cache size 0 max 16384 2024-03-24T00:45:15.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.835+0000 7f5473869740 1 Processor -- start 2024-03-24T00:45:15.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.835+0000 7f5473869740 1 -- start start 2024-03-24T00:45:15.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.836+0000 7f5473869740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5556d5299940 0x5556d531d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:15.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.836+0000 7f5473869740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556d5299d20 0x5556d52c9d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:15.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.836+0000 7f5473869740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5556d52b5360 0x5556d52b70c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:15.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.836+0000 7f5473869740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f546c200aa0 con 0x5556d5299d20 2024-03-24T00:45:15.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.836+0000 7f5473869740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f546c160a10 con 0x5556d5299940 2024-03-24T00:45:15.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.836+0000 7f5473869740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f546c15f0c0 con 0x5556d52b5360 2024-03-24T00:45:15.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.836+0000 7f5469d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556d5299d20 0x5556d52c9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:15.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.836+0000 7f5469d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556d5299d20 0x5556d52c9d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41490/0 (socket says 172.21.15.67:41490) 2024-03-24T00:45:15.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.836+0000 7f5469d9b640 1 -- 172.21.15.67:0/1038752960 learned_addr learned my addr 172.21.15.67:0/1038752960 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:15.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.836+0000 7f5469d9b640 1 -- 172.21.15.67:0/1038752960 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5556d52b5360 msgr2=0x5556d52b70c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:15.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.836+0000 7f5469d9b640 1 --2- 172.21.15.67:0/1038752960 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5556d52b5360 0x5556d52b70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.836+0000 7f5469d9b640 1 -- 172.21.15.67:0/1038752960 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5556d5299940 msgr2=0x5556d531d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:15.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.836+0000 7f5469d9b640 1 --2- 172.21.15.67:0/1038752960 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5556d5299940 0x5556d531d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.836+0000 7f5469d9b640 1 -- 172.21.15.67:0/1038752960 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5556d52b81d0 con 0x5556d5299d20 2024-03-24T00:45:15.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.837+0000 7f5469d9b640 1 --2- 172.21.15.67:0/1038752960 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556d5299d20 0x5556d52c9d70 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f5454001500 tx=0x7f545400a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=433f4e57556475b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:15.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.837+0000 7f544b7fe640 1 -- 172.21.15.67:0/1038752960 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f545402a070 con 0x5556d5299d20 2024-03-24T00:45:15.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.837+0000 7f544b7fe640 1 -- 172.21.15.67:0/1038752960 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f545401a040 con 0x5556d5299d20 2024-03-24T00:45:15.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.837+0000 7f544b7fe640 1 -- 172.21.15.67:0/1038752960 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5454019040 con 0x5556d5299d20 2024-03-24T00:45:15.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.837+0000 7f5473869740 1 -- 172.21.15.67:0/1038752960 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556d5299d20 msgr2=0x5556d52c9d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:15.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.837+0000 7f5473869740 1 --2- 172.21.15.67:0/1038752960 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556d5299d20 0x5556d52c9d70 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f5454001500 tx=0x7f545400a000 comp rx=0 tx=0).stop 2024-03-24T00:45:15.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.837+0000 7f5473869740 1 -- 172.21.15.67:0/1038752960 shutdown_connections 2024-03-24T00:45:15.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.837+0000 7f5473869740 1 --2- 172.21.15.67:0/1038752960 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5556d52b5360 0x5556d52b70c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.837+0000 7f5473869740 1 --2- 172.21.15.67:0/1038752960 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556d5299d20 0x5556d52c9d70 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.837+0000 7f5473869740 1 --2- 172.21.15.67:0/1038752960 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5556d5299940 0x5556d531d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.837+0000 7f5473869740 1 -- 172.21.15.67:0/1038752960 >> 172.21.15.67:0/1038752960 conn(0x5556d526d0b0 msgr2=0x5556d52ca5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:15.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.838+0000 7f5473869740 1 -- 172.21.15.67:0/1038752960 shutdown_connections 2024-03-24T00:45:15.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.838+0000 7f5473869740 1 -- 172.21.15.67:0/1038752960 wait complete. 2024-03-24T00:45:15.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.838+0000 7f5473869740 1 -- start start 2024-03-24T00:45:15.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.838+0000 7f5473869740 1 Processor -- start 2024-03-24T00:45:15.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.839+0000 7f5473869740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5556d5299940 0x5556d5314db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:15.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.839+0000 7f5473869740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5556d5299d20 0x5556d53189d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:15.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.839+0000 7f5473869740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556d52b5360 0x5556d5318f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:15.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.839+0000 7f5473869740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f546c160410 con 0x5556d52b5360 2024-03-24T00:45:15.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.839+0000 7f5473869740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556d5299750 con 0x5556d5299d20 2024-03-24T00:45:15.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.839+0000 7f5473869740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5556d52d9a80 con 0x5556d5299940 2024-03-24T00:45:15.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.839+0000 7f5473869740 10 client.0 _ll_register_callbacks cb 0x7f545c009620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:45:15.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.839+0000 7f546ad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556d52b5360 0x5556d5318f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:15.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.839+0000 7f546ad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556d52b5360 0x5556d5318f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41496/0 (socket says 172.21.15.67:41496) 2024-03-24T00:45:15.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.839+0000 7f546ad9d640 1 -- 172.21.15.67:0/952129721 learned_addr learned my addr 172.21.15.67:0/952129721 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:15.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.839+0000 7f546a59c640 1 --2- 172.21.15.67:0/952129721 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5556d5299940 0x5556d5314db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:15.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.839+0000 7f5469d9b640 1 --2- 172.21.15.67:0/952129721 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5556d5299d20 0x5556d53189d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:15.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.839+0000 7f546ad9d640 1 -- 172.21.15.67:0/952129721 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5556d5299940 msgr2=0x5556d5314db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:15.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.839+0000 7f546ad9d640 1 --2- 172.21.15.67:0/952129721 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5556d5299940 0x5556d5314db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.839+0000 7f546ad9d640 1 -- 172.21.15.67:0/952129721 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5556d5299d20 msgr2=0x5556d53189d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:15.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.839+0000 7f546ad9d640 1 --2- 172.21.15.67:0/952129721 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5556d5299d20 0x5556d53189d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.839+0000 7f546ad9d640 1 -- 172.21.15.67:0/952129721 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5556d529c0f0 con 0x5556d52b5360 2024-03-24T00:45:15.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.840+0000 7f546ad9d640 1 --2- 172.21.15.67:0/952129721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556d52b5360 0x5556d5318f10 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f54640070b0 tx=0x7f546405a050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1682beb1698b618 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:15.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.840+0000 7f54497fa640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:45:15.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.840+0000 7f54497fa640 1 -- 172.21.15.67:0/952129721 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5464061f20 con 0x5556d52b5360 2024-03-24T00:45:15.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.840+0000 7f54497fa640 1 -- 172.21.15.67:0/952129721 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f5464059a70 con 0x5556d52b5360 2024-03-24T00:45:15.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.840+0000 7f5473869740 1 -- 172.21.15.67:0/952129721 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f545401a040 con 0x5556d52b5360 2024-03-24T00:45:15.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.840+0000 7f54497fa640 1 -- 172.21.15.67:0/952129721 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5464059c70 con 0x5556d52b5360 2024-03-24T00:45:15.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.840+0000 7f5473869740 10 client.7953 fetch_fsmap learned FSMap version 60 2024-03-24T00:45:15.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.840+0000 7f5473869740 1 -- 172.21.15.67:0/952129721 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=2+}) v3 -- 0x5556d53161c0 con 0x5556d52b5360 2024-03-24T00:45:15.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.840+0000 7f54497fa640 1 -- 172.21.15.67:0/952129721 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f545401a040 con 0x5556d52b5360 2024-03-24T00:45:15.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.840+0000 7f54497fa640 1 -- 172.21.15.67:0/952129721 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 60) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f5464061a90 con 0x5556d52b5360 2024-03-24T00:45:15.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.840+0000 7f5473869740 10 client.7953 fetch_fsmap finished waiting for FSMap version 60 2024-03-24T00:45:15.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.840+0000 7f5473869740 10 client.7953 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-03-24T00:45:15.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.840+0000 7f5473869740 20 client.7953 populate_metadata read hostname 'smithi067' 2024-03-24T00:45:15.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.840+0000 7f5473869740 1 --2- 172.21.15.67:0/952129721 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x5556d5315900 0x5556d5298240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:15.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.840+0000 7f5473869740 4 client.7953 mds_command: new command op to 7860 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/dir", "scrubops": ["recursive"]}] 2024-03-24T00:45:15.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.840+0000 7f5473869740 1 -- 172.21.15.67:0/952129721 --> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] -- command(tid 0: {"prefix": "scrub start", "path": "/dir", "scrubops": ["recursive"]}) v1 -- 0x5556d51de900 con 0x5556d5315900 2024-03-24T00:45:15.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.840+0000 7f546a59c640 1 --2- 172.21.15.67:0/952129721 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x5556d5315900 0x5556d5298240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:15.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.841+0000 7f54497fa640 10 client.7953 ms_handle_connect on v2:172.21.15.67:6832/1336395002 2024-03-24T00:45:15.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.841+0000 7f546a59c640 1 --2- 172.21.15.67:0/952129721 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x5556d5315900 0x5556d5298240 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2d46f504661d7ee server_cookie=3a08280ff2dc8373 in_seq=0 out_seq=0 2024-03-24T00:45:15.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.841+0000 7f54497fa640 1 -- 172.21.15.67:0/952129721 <== mds.0 v2:172.21.15.67:6832/1336395002 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5556d51de900 con 0x5556d5315900 2024-03-24T00:45:15.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.841+0000 7f54497fa640 10 client.7953 handle_command_reply: tid=0 2024-03-24T00:45:15.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.841+0000 7f5473869740 1 client.7953 shutdown 2024-03-24T00:45:15.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.841+0000 7f5473869740 1 -- 172.21.15.67:0/952129721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556d52b5360 msgr2=0x5556d5318f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:15.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.841+0000 7f5473869740 1 --2- 172.21.15.67:0/952129721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556d52b5360 0x5556d5318f10 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f54640070b0 tx=0x7f546405a050 comp rx=0 tx=0).stop 2024-03-24T00:45:15.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.841+0000 7f5473869740 1 -- 172.21.15.67:0/952129721 shutdown_connections 2024-03-24T00:45:15.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.841+0000 7f5473869740 1 --2- 172.21.15.67:0/952129721 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x5556d5315900 0x5556d5298240 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.841+0000 7f5473869740 1 --2- 172.21.15.67:0/952129721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556d52b5360 0x5556d5318f10 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.841+0000 7f5473869740 1 --2- 172.21.15.67:0/952129721 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5556d5299d20 0x5556d53189d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.841+0000 7f54497fa640 0 client.7953 ms_handle_reset on v2:172.21.15.67:6832/1336395002 2024-03-24T00:45:15.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.841+0000 7f5473869740 1 --2- 172.21.15.67:0/952129721 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5556d5299940 0x5556d5314db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.841+0000 7f5473869740 1 -- 172.21.15.67:0/952129721 >> 172.21.15.67:0/952129721 conn(0x5556d526d0b0 msgr2=0x5556d526c350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:15.848 INFO:tasks.ceph.mds.c.smithi067.stderr:2024-03-24T00:45:15.841+0000 7fdf3050a640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000000 [...2,head] /dir/ auth v15 ap=1 f(v0 m2024-03-24T00:45:06.076992+0000 1=1+0) n(v0 rc2024-03-24T00:45:06.190990+0000 2=1+1) (iversion lock) | dirfrag=0 authpin=1 scrubqueue=0 0x56393232bb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(13)0x10000000000:[<0x1/dir v15>]//[]","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f(v0 m2024-03-24T00:45:06.076992+0000 1=1+0)","memory_value.rstat":"n(v0 rc2024-03-24T00:45:06.190990+0000 2=1+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":-2} 2024-03-24T00:45:15.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.841+0000 7f5473869740 1 -- 172.21.15.67:0/952129721 shutdown_connections 2024-03-24T00:45:15.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.841+0000 7f5473869740 1 -- 172.21.15.67:0/952129721 wait complete. 2024-03-24T00:45:15.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.842+0000 7f5473869740 20 client.7953 trim_cache size 0 max 16384 2024-03-24T00:45:15.849 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:45:15.849 INFO:teuthology.orchestra.run.smithi067.stdout: "return_code": 0, 2024-03-24T00:45:15.849 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_tag": "12e04979-fc27-4372-b278-1dd4e563caef", 2024-03-24T00:45:15.849 INFO:teuthology.orchestra.run.smithi067.stdout: "mode": "asynchronous" 2024-03-24T00:45:15.849 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:45:15.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.842+0000 7f5470c7e640 1 -- 172.21.15.67:0/3933287827 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f544c06bbd0 msgr2=0x7f544c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:15.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.842+0000 7f5470c7e640 1 --2- 172.21.15.67:0/3933287827 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f544c06bbd0 0x7f544c06e090 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f545c036000 tx=0x7f545c03c040 comp rx=0 tx=0).stop 2024-03-24T00:45:15.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.842+0000 7f5470c7e640 1 -- 172.21.15.67:0/3933287827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f546c1f7cb0 msgr2=0x7f546c0cb7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:15.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.842+0000 7f5470c7e640 1 --2- 172.21.15.67:0/3933287827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f546c1f7cb0 0x7f546c0cb7e0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f5464007de0 tx=0x7f546400e4b0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.842+0000 7f5470c7e640 1 -- 172.21.15.67:0/3933287827 shutdown_connections 2024-03-24T00:45:15.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.842+0000 7f5470c7e640 1 --2- 172.21.15.67:0/3933287827 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f544c06bbd0 0x7f544c06e090 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.842+0000 7f5470c7e640 1 --2- 172.21.15.67:0/3933287827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f546c1f7cb0 0x7f546c0cb7e0 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.842+0000 7f5470c7e640 1 --2- 172.21.15.67:0/3933287827 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f546c15dfc0 0x7f546c0cd710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.842+0000 7f5470c7e640 1 --2- 172.21.15.67:0/3933287827 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f546c0d0d50 0x7f546c1f4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.842+0000 7f5470c7e640 1 -- 172.21.15.67:0/3933287827 >> 172.21.15.67:0/3933287827 conn(0x7f546c06c880 msgr2=0x7f546c0c7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:15.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.842+0000 7f5470c7e640 1 -- 172.21.15.67:0/3933287827 shutdown_connections 2024-03-24T00:45:15.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.842+0000 7f5470c7e640 1 -- 172.21.15.67:0/3933287827 wait complete. 2024-03-24T00:45:15.853 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:45:15.853 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.6:0 scrub status 2024-03-24T00:45:15.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.955+0000 7fc143a54640 1 Processor -- start 2024-03-24T00:45:15.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.956+0000 7fc143a54640 1 -- start start 2024-03-24T00:45:15.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.956+0000 7fc143a54640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc13c15dea0 0x7fc13c0d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:15.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.956+0000 7fc143a54640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc13c15e280 0x7fc13c0d3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:15.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.956+0000 7fc143a54640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc13c0d2250 con 0x7fc13c0b5a60 2024-03-24T00:45:15.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.956+0000 7fc143a54640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc13c0d2530 con 0x7fc13c15e280 2024-03-24T00:45:15.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.956+0000 7fc143a54640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc13c0d2810 con 0x7fc13c15dea0 2024-03-24T00:45:15.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.957+0000 7fc140fef640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc13c15e280 0x7fc13c0d3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:15.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.957+0000 7fc141ff1640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc13c0b5a60 0x7fc13c06c640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56232/0 (socket says 172.21.15.67:56232) 2024-03-24T00:45:15.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.957+0000 7fc140fef640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc13c15e280 0x7fc13c0d3910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41500/0 (socket says 172.21.15.67:41500) 2024-03-24T00:45:15.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.957+0000 7fc141ff1640 1 -- 172.21.15.67:0/3195305891 learned_addr learned my addr 172.21.15.67:0/3195305891 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:45:15.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.957+0000 7fc1417f0640 1 --2- 172.21.15.67:0/3195305891 >> v2:172.21.15.72:3300/0 conn(0x7fc13c15dea0 0x7fc13c0d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:15.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.957+0000 7fc140fef640 1 -- 172.21.15.67:0/3195305891 >> v2:172.21.15.72:3300/0 conn(0x7fc13c15dea0 msgr2=0x7fc13c0d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:15.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.957+0000 7fc140fef640 1 --2- 172.21.15.67:0/3195305891 >> v2:172.21.15.72:3300/0 conn(0x7fc13c15dea0 0x7fc13c0d33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.957+0000 7fc140fef640 1 -- 172.21.15.67:0/3195305891 >> v1:172.21.15.67:6789/0 conn(0x7fc13c0b5a60 legacy=0x7fc13c06c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:15.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.957+0000 7fc140fef640 1 -- 172.21.15.67:0/3195305891 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc13c0d2af0 con 0x7fc13c15e280 2024-03-24T00:45:15.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.957+0000 7fc133fff640 1 -- 172.21.15.67:0/3195305891 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2059950731 0 0) 0x7fc13c0d2250 con 0x7fc13c0b5a60 2024-03-24T00:45:15.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.957+0000 7fc1417f0640 1 --2- 172.21.15.67:0/3195305891 >> v2:172.21.15.72:3300/0 conn(0x7fc13c15dea0 0x7fc13c0d33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:45:15.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.958+0000 7fc140fef640 1 --2- 172.21.15.67:0/3195305891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc13c15e280 0x7fc13c0d3910 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fc13c0d4c30 tx=0x7fc12c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18db60b3f1e66b2e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:15.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.958+0000 7fc133fff640 1 -- 172.21.15.67:0/3195305891 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12c0315a0 con 0x7fc13c15e280 2024-03-24T00:45:15.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.958+0000 7fc133fff640 1 -- 172.21.15.67:0/3195305891 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc12c037070 con 0x7fc13c15e280 2024-03-24T00:45:15.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.958+0000 7fc133fff640 1 -- 172.21.15.67:0/3195305891 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12c031960 con 0x7fc13c15e280 2024-03-24T00:45:15.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.959+0000 7fc143a54640 1 -- 172.21.15.67:0/3195305891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc13c15e280 msgr2=0x7fc13c0d3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:15.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.959+0000 7fc143a54640 1 --2- 172.21.15.67:0/3195305891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc13c15e280 0x7fc13c0d3910 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fc13c0d4c30 tx=0x7fc12c02b840 comp rx=0 tx=0).stop 2024-03-24T00:45:15.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.959+0000 7fc143a54640 1 -- 172.21.15.67:0/3195305891 shutdown_connections 2024-03-24T00:45:15.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.959+0000 7fc143a54640 1 --2- 172.21.15.67:0/3195305891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc13c15e280 0x7fc13c0d3910 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.959+0000 7fc143a54640 1 --2- 172.21.15.67:0/3195305891 >> v2:172.21.15.72:3300/0 conn(0x7fc13c15dea0 0x7fc13c0d33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.959+0000 7fc143a54640 1 -- 172.21.15.67:0/3195305891 >> 172.21.15.67:0/3195305891 conn(0x7fc13c072da0 msgr2=0x7fc13c0d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:15.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.959+0000 7fc143a54640 1 -- 172.21.15.67:0/3195305891 shutdown_connections 2024-03-24T00:45:15.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.959+0000 7fc143a54640 1 -- 172.21.15.67:0/3195305891 wait complete. 2024-03-24T00:45:15.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.960+0000 7fc143a54640 1 Processor -- start 2024-03-24T00:45:15.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.960+0000 7fc143a54640 1 -- start start 2024-03-24T00:45:15.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.961+0000 7fc143a54640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc13c0b5a60 0x7fc13c0ad2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:15.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.961+0000 7fc143a54640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc13c15dea0 0x7fc13c1f37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:15.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.961+0000 7fc143a54640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc13c15e280 0x7fc13c0ab8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:15.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.961+0000 7fc143a54640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c0b5eb0 con 0x7fc13c0b5a60 2024-03-24T00:45:15.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.961+0000 7fc143a54640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c0b8310 con 0x7fc13c15e280 2024-03-24T00:45:15.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.961+0000 7fc143a54640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c0b6030 con 0x7fc13c15dea0 2024-03-24T00:45:15.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.961+0000 7fc1417f0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc13c0b5a60 0x7fc13c0ad2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:15.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.961+0000 7fc1417f0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc13c0b5a60 0x7fc13c0ad2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41512/0 (socket says 172.21.15.67:41512) 2024-03-24T00:45:15.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.961+0000 7fc1417f0640 1 -- 172.21.15.67:0/2032943076 learned_addr learned my addr 172.21.15.67:0/2032943076 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:15.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.961+0000 7fc1417f0640 1 -- 172.21.15.67:0/2032943076 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc13c15dea0 msgr2=0x7fc13c1f37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:15.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.961+0000 7fc141ff1640 1 --2- 172.21.15.67:0/2032943076 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc13c15e280 0x7fc13c0ab8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:15.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.961+0000 7fc140fef640 1 --2- 172.21.15.67:0/2032943076 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc13c15dea0 0x7fc13c1f37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:15.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.961+0000 7fc1417f0640 1 --2- 172.21.15.67:0/2032943076 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc13c15dea0 0x7fc13c1f37c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.961+0000 7fc1417f0640 1 -- 172.21.15.67:0/2032943076 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc13c15e280 msgr2=0x7fc13c0ab8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:15.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.961+0000 7fc1417f0640 1 --2- 172.21.15.67:0/2032943076 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc13c15e280 0x7fc13c0ab8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.961+0000 7fc1417f0640 1 -- 172.21.15.67:0/2032943076 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc13c0cc2a0 con 0x7fc13c0b5a60 2024-03-24T00:45:15.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.961+0000 7fc141ff1640 1 --2- 172.21.15.67:0/2032943076 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc13c15e280 0x7fc13c0ab8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:45:15.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.961+0000 7fc1417f0640 1 --2- 172.21.15.67:0/2032943076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc13c0b5a60 0x7fc13c0ad2e0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7fc13800e7e0 tx=0x7fc13800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:15.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.962+0000 7fc131ffb640 1 -- 172.21.15.67:0/2032943076 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc13800f040 con 0x7fc13c0b5a60 2024-03-24T00:45:15.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.962+0000 7fc131ffb640 1 -- 172.21.15.67:0/2032943076 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc138004030 con 0x7fc13c0b5a60 2024-03-24T00:45:15.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.962+0000 7fc131ffb640 1 -- 172.21.15.67:0/2032943076 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc138004400 con 0x7fc13c0b5a60 2024-03-24T00:45:15.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.962+0000 7fc143a54640 1 -- 172.21.15.67:0/2032943076 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc13c0cc580 con 0x7fc13c0b5a60 2024-03-24T00:45:15.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.962+0000 7fc143a54640 1 -- 172.21.15.67:0/2032943076 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc13c0ccb40 con 0x7fc13c0b5a60 2024-03-24T00:45:15.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.963+0000 7fc131ffb640 1 -- 172.21.15.67:0/2032943076 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc13800c6d0 con 0x7fc13c0b5a60 2024-03-24T00:45:15.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.963+0000 7fc131ffb640 1 --2- 172.21.15.67:0/2032943076 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc12006bbd0 0x7fc12006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:15.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.964+0000 7fc131ffb640 1 -- 172.21.15.67:0/2032943076 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fc13808a9b0 con 0x7fc13c0b5a60 2024-03-24T00:45:15.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.964+0000 7fc140fef640 1 --2- 172.21.15.67:0/2032943076 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc12006bbd0 0x7fc12006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:15.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.964+0000 7fc140fef640 1 --2- 172.21.15.67:0/2032943076 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc12006bbd0 0x7fc12006e090 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fc12c036000 tx=0x7fc12c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:15.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.964+0000 7fc14663d740 1 Processor -- start 2024-03-24T00:45:15.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.964+0000 7fc14663d740 1 -- start start 2024-03-24T00:45:15.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.965+0000 7fc14663d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55db50813e50 0x55db507e6df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:15.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.965+0000 7fc14663d740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55db50814230 0x55db50816be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:15.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.965+0000 7fc14663d740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55db507e7330 0x55db50817120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:15.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.965+0000 7fc14663d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db507513d0 con 0x55db50813e50 2024-03-24T00:45:15.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.965+0000 7fc14663d740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db506f9ad0 con 0x55db507e7330 2024-03-24T00:45:15.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.965+0000 7fc14663d740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55db5074d470 con 0x55db50814230 2024-03-24T00:45:15.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.965+0000 7fc1417f0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55db50813e50 0x55db507e6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:15.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.965+0000 7fc1417f0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55db50813e50 0x55db507e6df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41518/0 (socket says 172.21.15.67:41518) 2024-03-24T00:45:15.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.965+0000 7fc1417f0640 1 -- 172.21.15.67:0/3134213060 learned_addr learned my addr 172.21.15.67:0/3134213060 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:15.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.965+0000 7fc140fef640 1 --2- 172.21.15.67:0/3134213060 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55db50814230 0x55db50816be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:15.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.965+0000 7fc141ff1640 1 --2- 172.21.15.67:0/3134213060 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55db507e7330 0x55db50817120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:15.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.966+0000 7fc1417f0640 1 -- 172.21.15.67:0/3134213060 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55db50814230 msgr2=0x55db50816be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:15.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.966+0000 7fc1417f0640 1 --2- 172.21.15.67:0/3134213060 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55db50814230 0x55db50816be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.966+0000 7fc1417f0640 1 -- 172.21.15.67:0/3134213060 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55db507e7330 msgr2=0x55db50817120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:15.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.966+0000 7fc1417f0640 1 --2- 172.21.15.67:0/3134213060 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55db507e7330 0x55db50817120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.966+0000 7fc1417f0640 1 -- 172.21.15.67:0/3134213060 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55db5052f7d0 con 0x55db50813e50 2024-03-24T00:45:15.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.966+0000 7fc1417f0640 1 --2- 172.21.15.67:0/3134213060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55db50813e50 0x55db507e6df0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fc138088d00 tx=0x7fc1380586e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e252c88e48925d55 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:15.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.966+0000 7fc11effd640 1 -- 172.21.15.67:0/3134213060 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc138020040 con 0x55db50813e50 2024-03-24T00:45:15.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.966+0000 7fc11effd640 1 -- 172.21.15.67:0/3134213060 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc138056f50 con 0x55db50813e50 2024-03-24T00:45:15.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.966+0000 7fc11effd640 1 -- 172.21.15.67:0/3134213060 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1380580a0 con 0x55db50813e50 2024-03-24T00:45:15.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.967+0000 7fc14663d740 1 -- 172.21.15.67:0/3134213060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55db50813e50 msgr2=0x55db507e6df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:15.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.967+0000 7fc14663d740 1 --2- 172.21.15.67:0/3134213060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55db50813e50 0x55db507e6df0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fc138088d00 tx=0x7fc1380586e0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.967+0000 7fc14663d740 1 -- 172.21.15.67:0/3134213060 shutdown_connections 2024-03-24T00:45:15.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.967+0000 7fc14663d740 1 --2- 172.21.15.67:0/3134213060 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55db507e7330 0x55db50817120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.967+0000 7fc14663d740 1 --2- 172.21.15.67:0/3134213060 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55db50814230 0x55db50816be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.967+0000 7fc14663d740 1 --2- 172.21.15.67:0/3134213060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55db50813e50 0x55db507e6df0 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.967+0000 7fc14663d740 1 -- 172.21.15.67:0/3134213060 >> 172.21.15.67:0/3134213060 conn(0x55db507e9060 msgr2=0x55db50831be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:15.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.967+0000 7fc14663d740 1 -- 172.21.15.67:0/3134213060 shutdown_connections 2024-03-24T00:45:15.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.967+0000 7fc14663d740 1 -- 172.21.15.67:0/3134213060 wait complete. 2024-03-24T00:45:15.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.967+0000 7fc14663d740 1 -- start start 2024-03-24T00:45:15.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.967+0000 7fc14663d740 1 Processor -- start 2024-03-24T00:45:15.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.968+0000 7fc14663d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55db50813e50 0x55db50818d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:15.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.968+0000 7fc14663d740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55db50814230 0x55db5081d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:15.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.968+0000 7fc14663d740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55db508192c0 0x55db5081dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:15.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.968+0000 7fc14663d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db507eb700 con 0x55db50813e50 2024-03-24T00:45:15.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.968+0000 7fc14663d740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db5070d060 con 0x55db508192c0 2024-03-24T00:45:15.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.968+0000 7fc14663d740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55db507edf40 con 0x55db50814230 2024-03-24T00:45:15.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.968+0000 7fc14663d740 10 client.0 _ll_register_callbacks cb 0x55db506ef6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:45:15.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.968+0000 7fc1417f0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55db50813e50 0x55db50818d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:15.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.968+0000 7fc1417f0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55db50813e50 0x55db50818d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41520/0 (socket says 172.21.15.67:41520) 2024-03-24T00:45:15.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.968+0000 7fc1417f0640 1 -- 172.21.15.67:0/3784650121 learned_addr learned my addr 172.21.15.67:0/3784650121 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:15.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.968+0000 7fc140fef640 1 --2- 172.21.15.67:0/3784650121 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55db50814230 0x55db5081d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:15.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.968+0000 7fc141ff1640 1 --2- 172.21.15.67:0/3784650121 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55db508192c0 0x55db5081dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:15.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.969+0000 7fc1417f0640 1 -- 172.21.15.67:0/3784650121 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55db50814230 msgr2=0x55db5081d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:15.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.969+0000 7fc1417f0640 1 --2- 172.21.15.67:0/3784650121 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55db50814230 0x55db5081d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.969+0000 7fc1417f0640 1 -- 172.21.15.67:0/3784650121 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55db508192c0 msgr2=0x55db5081dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:15.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.969+0000 7fc1417f0640 1 --2- 172.21.15.67:0/3784650121 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55db508192c0 0x55db5081dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.969+0000 7fc1417f0640 1 -- 172.21.15.67:0/3784650121 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55db504653f0 con 0x55db50813e50 2024-03-24T00:45:15.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.969+0000 7fc1417f0640 1 --2- 172.21.15.67:0/3784650121 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55db50813e50 0x55db50818d80 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7fc138088de0 tx=0x7fc138006d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=967d88cfbd4e6209 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:15.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.969+0000 7fc117fff640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:45:15.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.969+0000 7fc117fff640 1 -- 172.21.15.67:0/3784650121 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc138020040 con 0x55db50813e50 2024-03-24T00:45:15.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.969+0000 7fc117fff640 1 -- 172.21.15.67:0/3784650121 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fc1380549a0 con 0x55db50813e50 2024-03-24T00:45:15.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.969+0000 7fc14663d740 1 -- 172.21.15.67:0/3784650121 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc138056f50 con 0x55db50813e50 2024-03-24T00:45:15.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.969+0000 7fc117fff640 1 -- 172.21.15.67:0/3784650121 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc138054ba0 con 0x55db50813e50 2024-03-24T00:45:15.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.969+0000 7fc14663d740 10 client.7968 fetch_fsmap learned FSMap version 60 2024-03-24T00:45:15.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.969+0000 7fc14663d740 1 -- 172.21.15.67:0/3784650121 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=2+}) v3 -- 0x55db504250f0 con 0x55db50813e50 2024-03-24T00:45:15.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.969+0000 7fc117fff640 1 -- 172.21.15.67:0/3784650121 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc138056f50 con 0x55db50813e50 2024-03-24T00:45:15.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.969+0000 7fc117fff640 1 -- 172.21.15.67:0/3784650121 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 60) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fc138012050 con 0x55db50813e50 2024-03-24T00:45:15.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.969+0000 7fc14663d740 10 client.7968 fetch_fsmap finished waiting for FSMap version 60 2024-03-24T00:45:15.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.969+0000 7fc14663d740 10 client.7968 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-03-24T00:45:15.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.970+0000 7fc14663d740 20 client.7968 populate_metadata read hostname 'smithi067' 2024-03-24T00:45:15.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.970+0000 7fc14663d740 1 --2- 172.21.15.67:0/3784650121 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x55db5081c5b0 0x55db5081c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:15.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.970+0000 7fc14663d740 4 client.7968 mds_command: new command op to 7860 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:45:15.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.970+0000 7fc14663d740 1 -- 172.21.15.67:0/3784650121 --> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55db5075e910 con 0x55db5081c5b0 2024-03-24T00:45:15.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.970+0000 7fc140fef640 1 --2- 172.21.15.67:0/3784650121 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x55db5081c5b0 0x55db5081c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:15.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.970+0000 7fc117fff640 10 client.7968 ms_handle_connect on v2:172.21.15.67:6832/1336395002 2024-03-24T00:45:15.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.970+0000 7fc140fef640 1 --2- 172.21.15.67:0/3784650121 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x55db5081c5b0 0x55db5081c9c0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a937f5828cd179fd server_cookie=5bd5c679f05d3210 in_seq=0 out_seq=0 2024-03-24T00:45:15.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.971+0000 7fc117fff640 1 -- 172.21.15.67:0/3784650121 <== mds.0 v2:172.21.15.67:6832/1336395002 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55db5075e910 con 0x55db5081c5b0 2024-03-24T00:45:15.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.971+0000 7fc117fff640 10 client.7968 handle_command_reply: tid=0 2024-03-24T00:45:15.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.971+0000 7fc14663d740 1 client.7968 shutdown 2024-03-24T00:45:15.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.971+0000 7fc14663d740 1 -- 172.21.15.67:0/3784650121 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55db50813e50 msgr2=0x55db50818d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:15.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.971+0000 7fc14663d740 1 --2- 172.21.15.67:0/3784650121 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55db50813e50 0x55db50818d80 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7fc138088de0 tx=0x7fc138006d40 comp rx=0 tx=0).stop 2024-03-24T00:45:15.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.971+0000 7fc14663d740 1 -- 172.21.15.67:0/3784650121 shutdown_connections 2024-03-24T00:45:15.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.971+0000 7fc14663d740 1 --2- 172.21.15.67:0/3784650121 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x55db5081c5b0 0x55db5081c9c0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.971+0000 7fc14663d740 1 --2- 172.21.15.67:0/3784650121 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55db508192c0 0x55db5081dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.971+0000 7fc14663d740 1 --2- 172.21.15.67:0/3784650121 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55db50814230 0x55db5081d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.971+0000 7fc14663d740 1 --2- 172.21.15.67:0/3784650121 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55db50813e50 0x55db50818d80 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.971+0000 7fc14663d740 1 -- 172.21.15.67:0/3784650121 >> 172.21.15.67:0/3784650121 conn(0x55db507e7330 msgr2=0x55db507ead00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:15.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.971+0000 7fc117fff640 0 client.7968 ms_handle_reset on v2:172.21.15.67:6832/1336395002 2024-03-24T00:45:15.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.971+0000 7fc14663d740 1 -- 172.21.15.67:0/3784650121 shutdown_connections 2024-03-24T00:45:15.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.971+0000 7fc14663d740 1 -- 172.21.15.67:0/3784650121 wait complete. 2024-03-24T00:45:15.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.971+0000 7fc14663d740 20 client.7968 trim_cache size 0 max 16384 2024-03-24T00:45:15.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.987+0000 7fc14663d740 1 Processor -- start 2024-03-24T00:45:15.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.987+0000 7fc14663d740 1 -- start start 2024-03-24T00:45:15.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.988+0000 7fc14663d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55db50813e50 0x55db5081f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:15.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.988+0000 7fc14663d740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55db50814230 0x55db50831e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:15.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.988+0000 7fc14663d740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55db508192c0 0x55db5089a610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:15.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.988+0000 7fc14663d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c2006d0 con 0x55db50813e50 2024-03-24T00:45:15.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.988+0000 7fc14663d740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c1ff860 con 0x55db50814230 2024-03-24T00:45:15.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.988+0000 7fc14663d740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c1f7970 con 0x55db508192c0 2024-03-24T00:45:15.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.988+0000 7fc1417f0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55db50813e50 0x55db5081f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:15.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.988+0000 7fc1417f0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55db50813e50 0x55db5081f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41526/0 (socket says 172.21.15.67:41526) 2024-03-24T00:45:15.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.988+0000 7fc1417f0640 1 -- 172.21.15.67:0/1171008688 learned_addr learned my addr 172.21.15.67:0/1171008688 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:15.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.988+0000 7fc141ff1640 1 --2- 172.21.15.67:0/1171008688 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55db508192c0 0x55db5089a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:15.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.988+0000 7fc140fef640 1 --2- 172.21.15.67:0/1171008688 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55db50814230 0x55db50831e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:15.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.988+0000 7fc1417f0640 1 -- 172.21.15.67:0/1171008688 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55db508192c0 msgr2=0x55db5089a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:15.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.988+0000 7fc1417f0640 1 --2- 172.21.15.67:0/1171008688 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55db508192c0 0x55db5089a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.988+0000 7fc1417f0640 1 -- 172.21.15.67:0/1171008688 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55db50814230 msgr2=0x55db50831e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:15.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.988+0000 7fc1417f0640 1 --2- 172.21.15.67:0/1171008688 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55db50814230 0x55db50831e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.988+0000 7fc1417f0640 1 -- 172.21.15.67:0/1171008688 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55db507eb530 con 0x55db50813e50 2024-03-24T00:45:15.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.988+0000 7fc140fef640 1 --2- 172.21.15.67:0/1171008688 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55db50814230 0x55db50831e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:45:15.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.988+0000 7fc141ff1640 1 --2- 172.21.15.67:0/1171008688 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55db508192c0 0x55db5089a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:45:15.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.988+0000 7fc1417f0640 1 --2- 172.21.15.67:0/1171008688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55db50813e50 0x55db5081f0f0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7fc13800efd0 tx=0x7fc13806e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=707f8542460f485d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:15.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.989+0000 7fc11e7fc640 1 -- 172.21.15.67:0/1171008688 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc138020070 con 0x55db50813e50 2024-03-24T00:45:15.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.989+0000 7fc11e7fc640 1 -- 172.21.15.67:0/1171008688 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc138056f50 con 0x55db50813e50 2024-03-24T00:45:15.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.989+0000 7fc11e7fc640 1 -- 172.21.15.67:0/1171008688 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc138057da0 con 0x55db50813e50 2024-03-24T00:45:15.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.989+0000 7fc14663d740 1 -- 172.21.15.67:0/1171008688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55db50813e50 msgr2=0x55db5081f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:15.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.989+0000 7fc14663d740 1 --2- 172.21.15.67:0/1171008688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55db50813e50 0x55db5081f0f0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7fc13800efd0 tx=0x7fc13806e040 comp rx=0 tx=0).stop 2024-03-24T00:45:15.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.989+0000 7fc14663d740 1 -- 172.21.15.67:0/1171008688 shutdown_connections 2024-03-24T00:45:15.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.989+0000 7fc14663d740 1 --2- 172.21.15.67:0/1171008688 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55db508192c0 0x55db5089a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.989+0000 7fc14663d740 1 --2- 172.21.15.67:0/1171008688 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55db50814230 0x55db50831e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.989+0000 7fc14663d740 1 --2- 172.21.15.67:0/1171008688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55db50813e50 0x55db5081f0f0 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.989+0000 7fc14663d740 1 -- 172.21.15.67:0/1171008688 >> 172.21.15.67:0/1171008688 conn(0x55db507e7330 msgr2=0x55db50812750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:15.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.989+0000 7fc14663d740 1 -- 172.21.15.67:0/1171008688 shutdown_connections 2024-03-24T00:45:15.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.989+0000 7fc14663d740 1 -- 172.21.15.67:0/1171008688 wait complete. 2024-03-24T00:45:15.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.990+0000 7fc14663d740 1 -- start start 2024-03-24T00:45:15.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.990+0000 7fc14663d740 1 Processor -- start 2024-03-24T00:45:15.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.991+0000 7fc14663d740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55db50813e50 0x55db5081c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:15.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.991+0000 7fc14663d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55db50814230 0x55db508a03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:15.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.991+0000 7fc14663d740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55db508192c0 0x55db507ea820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:15.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.991+0000 7fc14663d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db5081f760 con 0x55db50814230 2024-03-24T00:45:15.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.991+0000 7fc14663d740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c1ff260 con 0x55db50813e50 2024-03-24T00:45:15.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.991+0000 7fc14663d740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55db508a0f70 con 0x55db508192c0 2024-03-24T00:45:15.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.991+0000 7fc14663d740 10 client.0 _ll_register_callbacks cb 0x7fc13806f070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:45:15.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.991+0000 7fc140fef640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55db50814230 0x55db508a03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:15.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.991+0000 7fc140fef640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55db50814230 0x55db508a03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41532/0 (socket says 172.21.15.67:41532) 2024-03-24T00:45:15.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.991+0000 7fc140fef640 1 -- 172.21.15.67:0/1407339040 learned_addr learned my addr 172.21.15.67:0/1407339040 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:15.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.991+0000 7fc141ff1640 1 --2- 172.21.15.67:0/1407339040 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55db508192c0 0x55db507ea820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:15.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.991+0000 7fc1417f0640 1 --2- 172.21.15.67:0/1407339040 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55db50813e50 0x55db5081c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:15.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.991+0000 7fc140fef640 1 -- 172.21.15.67:0/1407339040 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55db508192c0 msgr2=0x55db507ea820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:15.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.991+0000 7fc140fef640 1 --2- 172.21.15.67:0/1407339040 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55db508192c0 0x55db507ea820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.991+0000 7fc140fef640 1 -- 172.21.15.67:0/1407339040 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55db50813e50 msgr2=0x55db5081c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:15.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.991+0000 7fc140fef640 1 --2- 172.21.15.67:0/1407339040 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55db50813e50 0x55db5081c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.991+0000 7fc140fef640 1 -- 172.21.15.67:0/1407339040 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55db5081c1f0 con 0x55db50814230 2024-03-24T00:45:15.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.991+0000 7fc1417f0640 1 --2- 172.21.15.67:0/1407339040 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55db50813e50 0x55db5081c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:45:15.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.991+0000 7fc141ff1640 1 --2- 172.21.15.67:0/1407339040 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55db508192c0 0x55db507ea820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:45:15.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.991+0000 7fc140fef640 1 --2- 172.21.15.67:0/1407339040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55db50814230 0x55db508a03e0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fc12c00b0b0 tx=0x7fc12c02e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20583a3bff4e2dd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:15.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.991+0000 7fc117fff640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:45:15.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.991+0000 7fc117fff640 1 -- 172.21.15.67:0/1407339040 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12c031740 con 0x55db50814230 2024-03-24T00:45:15.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.991+0000 7fc117fff640 1 -- 172.21.15.67:0/1407339040 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fc12c0318e0 con 0x55db50814230 2024-03-24T00:45:15.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.991+0000 7fc14663d740 1 -- 172.21.15.67:0/1407339040 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc138056f50 con 0x55db50814230 2024-03-24T00:45:15.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.992+0000 7fc117fff640 1 -- 172.21.15.67:0/1407339040 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc12c064050 con 0x55db50814230 2024-03-24T00:45:15.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.992+0000 7fc14663d740 10 client.7974 fetch_fsmap learned FSMap version 60 2024-03-24T00:45:15.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.992+0000 7fc14663d740 1 -- 172.21.15.67:0/1407339040 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=2+}) v3 -- 0x55db50897230 con 0x55db50814230 2024-03-24T00:45:15.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.992+0000 7fc117fff640 1 -- 172.21.15.67:0/1407339040 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc138056f50 con 0x55db50814230 2024-03-24T00:45:15.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.992+0000 7fc117fff640 1 -- 172.21.15.67:0/1407339040 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 60) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fc12c067020 con 0x55db50814230 2024-03-24T00:45:15.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.992+0000 7fc14663d740 10 client.7974 fetch_fsmap finished waiting for FSMap version 60 2024-03-24T00:45:15.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.992+0000 7fc14663d740 10 client.7974 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-03-24T00:45:15.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.992+0000 7fc14663d740 20 client.7974 populate_metadata read hostname 'smithi067' 2024-03-24T00:45:15.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.992+0000 7fc14663d740 1 --2- 172.21.15.67:0/1407339040 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x55db50835250 0x55db507ead60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:16.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.992+0000 7fc14663d740 4 client.7974 mds_command: new command op to 7860 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:45:16.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.992+0000 7fc14663d740 1 -- 172.21.15.67:0/1407339040 --> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55db50758c60 con 0x55db50835250 2024-03-24T00:45:16.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.992+0000 7fc1417f0640 1 --2- 172.21.15.67:0/1407339040 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x55db50835250 0x55db507ead60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:16.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.992+0000 7fc117fff640 10 client.7974 ms_handle_connect on v2:172.21.15.67:6832/1336395002 2024-03-24T00:45:16.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.992+0000 7fc1417f0640 1 --2- 172.21.15.67:0/1407339040 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x55db50835250 0x55db507ead60 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42f665a9f1406ac7 server_cookie=a198ab9955f3982c in_seq=0 out_seq=0 2024-03-24T00:45:16.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.993+0000 7fc117fff640 1 -- 172.21.15.67:0/1407339040 <== mds.0 v2:172.21.15.67:6832/1336395002 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+256 (crc 0 0 0) 0x55db50758c60 con 0x55db50835250 2024-03-24T00:45:16.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.993+0000 7fc117fff640 10 client.7974 handle_command_reply: tid=0 2024-03-24T00:45:16.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.993+0000 7fc14663d740 1 client.7974 shutdown 2024-03-24T00:45:16.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.993+0000 7fc14663d740 1 -- 172.21.15.67:0/1407339040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55db50814230 msgr2=0x55db508a03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:16.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.993+0000 7fc14663d740 1 --2- 172.21.15.67:0/1407339040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55db50814230 0x55db508a03e0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fc12c00b0b0 tx=0x7fc12c02e000 comp rx=0 tx=0).stop 2024-03-24T00:45:16.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.993+0000 7fc14663d740 1 -- 172.21.15.67:0/1407339040 shutdown_connections 2024-03-24T00:45:16.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.993+0000 7fc14663d740 1 --2- 172.21.15.67:0/1407339040 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x55db50835250 0x55db507ead60 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:16.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.993+0000 7fc14663d740 1 --2- 172.21.15.67:0/1407339040 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55db508192c0 0x55db507ea820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:16.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.993+0000 7fc14663d740 1 --2- 172.21.15.67:0/1407339040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55db50814230 0x55db508a03e0 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:16.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.993+0000 7fc14663d740 1 --2- 172.21.15.67:0/1407339040 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55db50813e50 0x55db5081c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:16.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.993+0000 7fc117fff640 0 client.7974 ms_handle_reset on v2:172.21.15.67:6832/1336395002 2024-03-24T00:45:16.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.993+0000 7fc14663d740 1 -- 172.21.15.67:0/1407339040 >> 172.21.15.67:0/1407339040 conn(0x55db507e7330 msgr2=0x55db50812750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:16.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.993+0000 7fc14663d740 1 -- 172.21.15.67:0/1407339040 shutdown_connections 2024-03-24T00:45:16.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.993+0000 7fc14663d740 1 -- 172.21.15.67:0/1407339040 wait complete. 2024-03-24T00:45:16.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.993+0000 7fc14663d740 20 client.7974 trim_cache size 0 max 16384 2024-03-24T00:45:16.001 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:45:16.001 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T00:45:16.002 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": { 2024-03-24T00:45:16.002 INFO:teuthology.orchestra.run.smithi067.stdout: "12e04979-fc27-4372-b278-1dd4e563caef": { 2024-03-24T00:45:16.002 INFO:teuthology.orchestra.run.smithi067.stdout: "path": "/dir", 2024-03-24T00:45:16.002 INFO:teuthology.orchestra.run.smithi067.stdout: "tag": "12e04979-fc27-4372-b278-1dd4e563caef", 2024-03-24T00:45:16.002 INFO:teuthology.orchestra.run.smithi067.stdout: "options": "recursive" 2024-03-24T00:45:16.002 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:45:16.002 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:45:16.002 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:45:16.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.994+0000 7fc143a54640 1 -- 172.21.15.67:0/2032943076 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc12006bbd0 msgr2=0x7fc12006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:16.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.994+0000 7fc143a54640 1 --2- 172.21.15.67:0/2032943076 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc12006bbd0 0x7fc12006e090 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fc12c036000 tx=0x7fc12c008030 comp rx=0 tx=0).stop 2024-03-24T00:45:16.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.994+0000 7fc143a54640 1 -- 172.21.15.67:0/2032943076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc13c0b5a60 msgr2=0x7fc13c0ad2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:16.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.994+0000 7fc143a54640 1 --2- 172.21.15.67:0/2032943076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc13c0b5a60 0x7fc13c0ad2e0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7fc13800e7e0 tx=0x7fc13800eca0 comp rx=0 tx=0).stop 2024-03-24T00:45:16.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.994+0000 7fc143a54640 1 -- 172.21.15.67:0/2032943076 shutdown_connections 2024-03-24T00:45:16.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.994+0000 7fc143a54640 1 --2- 172.21.15.67:0/2032943076 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc12006bbd0 0x7fc12006e090 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:16.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.994+0000 7fc143a54640 1 --2- 172.21.15.67:0/2032943076 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc13c15e280 0x7fc13c0ab8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:16.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.994+0000 7fc143a54640 1 --2- 172.21.15.67:0/2032943076 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc13c15dea0 0x7fc13c1f37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:16.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.994+0000 7fc143a54640 1 --2- 172.21.15.67:0/2032943076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc13c0b5a60 0x7fc13c0ad2e0 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:16.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.994+0000 7fc143a54640 1 -- 172.21.15.67:0/2032943076 >> 172.21.15.67:0/2032943076 conn(0x7fc13c072da0 msgr2=0x7fc13c0b8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:16.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.994+0000 7fc143a54640 1 -- 172.21.15.67:0/2032943076 shutdown_connections 2024-03-24T00:45:16.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:15.994+0000 7fc143a54640 1 -- 172.21.15.67:0/2032943076 wait complete. 2024-03-24T00:45:16.005 INFO:tasks.cephfs.filesystem:scrub status for tag:12e04979-fc27-4372-b278-1dd4e563caef - {'path': '/dir', 'tag': '12e04979-fc27-4372-b278-1dd4e563caef', 'options': 'recursive'} 2024-03-24T00:45:16.116 INFO:tasks.ceph.mds.c.smithi067.stderr:2024-03-24T00:45:16.115+0000 7fdf34512640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T00:45:16.646 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:45:16.648 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:45:20.465 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-24T00:45:20.464+0000 7fbfd2216640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-03-24T00:45:46.005 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:45:46.006 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.6:0 scrub status 2024-03-24T00:45:46.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.112+0000 7f20a4ef8640 1 Processor -- start 2024-03-24T00:45:46.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.112+0000 7f20a4ef8640 1 -- start start 2024-03-24T00:45:46.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.113+0000 7f20a4ef8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f20a015bc30 0x7f20a00b1a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.113+0000 7f20a4ef8640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f20a00b34a0 0x7f20a00b1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.113+0000 7f20a4ef8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20a00b3ae0 con 0x7f20a015c010 2024-03-24T00:45:46.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.113+0000 7f20a4ef8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20a00cca60 con 0x7f20a015bc30 2024-03-24T00:45:46.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.113+0000 7f20a4ef8640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20a00b3dc0 con 0x7f20a00b34a0 2024-03-24T00:45:46.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.113+0000 7f209e59c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f20a015bc30 0x7f20a00b1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.113+0000 7f209e59c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f20a015bc30 0x7f20a00b1a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37318/0 (socket says 172.21.15.67:37318) 2024-03-24T00:45:46.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.113+0000 7f2095d9b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f20a015c010 0x7f20a00b2d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47612/0 (socket says 172.21.15.67:47612) 2024-03-24T00:45:46.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.113+0000 7f209e59c640 1 -- 172.21.15.67:0/2932586379 learned_addr learned my addr 172.21.15.67:0/2932586379 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:46.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.114+0000 7f209ed9d640 1 --2- 172.21.15.67:0/2932586379 >> v2:172.21.15.72:3300/0 conn(0x7f20a00b34a0 0x7f20a00b1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.114+0000 7f209e59c640 1 -- 172.21.15.67:0/2932586379 >> v2:172.21.15.72:3300/0 conn(0x7f20a00b34a0 msgr2=0x7f20a00b1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:46.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.114+0000 7f209e59c640 1 --2- 172.21.15.67:0/2932586379 >> v2:172.21.15.72:3300/0 conn(0x7f20a00b34a0 0x7f20a00b1f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.114+0000 7f209e59c640 1 -- 172.21.15.67:0/2932586379 >> v1:172.21.15.67:6789/0 conn(0x7f20a015c010 legacy=0x7f20a00b2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:46.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.114+0000 7f209dd9b640 1 -- 172.21.15.67:0/2932586379 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3462677299 0 0) 0x7f20a00b3ae0 con 0x7f20a015c010 2024-03-24T00:45:46.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.114+0000 7f209e59c640 1 -- 172.21.15.67:0/2932586379 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a00b40a0 con 0x7f20a015bc30 2024-03-24T00:45:46.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.114+0000 7f209e59c640 1 --2- 172.21.15.67:0/2932586379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20a015bc30 0x7f20a00b1a00 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f20a008bca0 tx=0x7f209002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61c73ca4b7a16197 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:46.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.115+0000 7f209dd9b640 1 -- 172.21.15.67:0/2932586379 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2090034070 con 0x7f20a015bc30 2024-03-24T00:45:46.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.115+0000 7f209dd9b640 1 -- 172.21.15.67:0/2932586379 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2090030860 con 0x7f20a015bc30 2024-03-24T00:45:46.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.115+0000 7f209dd9b640 1 -- 172.21.15.67:0/2932586379 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2090030c30 con 0x7f20a015bc30 2024-03-24T00:45:46.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.115+0000 7f20a4ef8640 1 -- 172.21.15.67:0/2932586379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20a015bc30 msgr2=0x7f20a00b1a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:46.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.115+0000 7f20a4ef8640 1 --2- 172.21.15.67:0/2932586379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20a015bc30 0x7f20a00b1a00 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f20a008bca0 tx=0x7f209002b9a0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.115+0000 7f20a4ef8640 1 -- 172.21.15.67:0/2932586379 shutdown_connections 2024-03-24T00:45:46.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.115+0000 7f20a4ef8640 1 --2- 172.21.15.67:0/2932586379 >> v2:172.21.15.72:3300/0 conn(0x7f20a00b34a0 0x7f20a00b1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.115+0000 7f20a4ef8640 1 --2- 172.21.15.67:0/2932586379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20a015bc30 0x7f20a00b1a00 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.115+0000 7f20a4ef8640 1 -- 172.21.15.67:0/2932586379 >> 172.21.15.67:0/2932586379 conn(0x7f20a008b180 msgr2=0x7f20a008b590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:46.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.115+0000 7f20a4ef8640 1 -- 172.21.15.67:0/2932586379 shutdown_connections 2024-03-24T00:45:46.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.115+0000 7f20a4ef8640 1 -- 172.21.15.67:0/2932586379 wait complete. 2024-03-24T00:45:46.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.117+0000 7f20a4ef8640 1 Processor -- start 2024-03-24T00:45:46.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.117+0000 7f20a4ef8640 1 -- start start 2024-03-24T00:45:46.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.117+0000 7f20a4ef8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20a00b34a0 0x7f20a015ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.117+0000 7f20a4ef8640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f20a015bc30 0x7f20a00c46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.117+0000 7f20a4ef8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20a015c010 0x7f20a01f3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.117+0000 7f20a4ef8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a00cd070 con 0x7f20a00b34a0 2024-03-24T00:45:46.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.117+0000 7f20a4ef8640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a00ccef0 con 0x7f20a015c010 2024-03-24T00:45:46.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.117+0000 7f20a4ef8640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20a00cd1f0 con 0x7f20a015bc30 2024-03-24T00:45:46.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.117+0000 7f209e59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20a00b34a0 0x7f20a015ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.117+0000 7f209e59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20a00b34a0 0x7f20a015ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37324/0 (socket says 172.21.15.67:37324) 2024-03-24T00:45:46.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.117+0000 7f209e59c640 1 -- 172.21.15.67:0/329950872 learned_addr learned my addr 172.21.15.67:0/329950872 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:46.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.118+0000 7f2095d9b640 1 --2- 172.21.15.67:0/329950872 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f20a015bc30 0x7f20a00c46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.118+0000 7f209ed9d640 1 --2- 172.21.15.67:0/329950872 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20a015c010 0x7f20a01f3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.118+0000 7f209e59c640 1 -- 172.21.15.67:0/329950872 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f20a015bc30 msgr2=0x7f20a00c46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:46.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.118+0000 7f209e59c640 1 --2- 172.21.15.67:0/329950872 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f20a015bc30 0x7f20a00c46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.118+0000 7f209e59c640 1 -- 172.21.15.67:0/329950872 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20a015c010 msgr2=0x7f20a01f3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:46.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.118+0000 7f209e59c640 1 --2- 172.21.15.67:0/329950872 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20a015c010 0x7f20a01f3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.118+0000 7f209e59c640 1 -- 172.21.15.67:0/329950872 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a015eb40 con 0x7f20a00b34a0 2024-03-24T00:45:46.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.118+0000 7f2095d9b640 1 --2- 172.21.15.67:0/329950872 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f20a015bc30 0x7f20a00c46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:45:46.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.118+0000 7f209e59c640 1 --2- 172.21.15.67:0/329950872 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20a00b34a0 0x7f20a015ab90 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f209002b9e0 tx=0x7f2090004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:46.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.118+0000 7f20977fe640 1 -- 172.21.15.67:0/329950872 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2090048070 con 0x7f20a00b34a0 2024-03-24T00:45:46.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.118+0000 7f20977fe640 1 -- 172.21.15.67:0/329950872 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2090030660 con 0x7f20a00b34a0 2024-03-24T00:45:46.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.118+0000 7f20977fe640 1 -- 172.21.15.67:0/329950872 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2090034040 con 0x7f20a00b34a0 2024-03-24T00:45:46.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.118+0000 7f20a4ef8640 1 -- 172.21.15.67:0/329950872 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20a015ee20 con 0x7f20a00b34a0 2024-03-24T00:45:46.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.118+0000 7f20a4ef8640 1 -- 172.21.15.67:0/329950872 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20a01f3a30 con 0x7f20a00b34a0 2024-03-24T00:45:46.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.120+0000 7f20977fe640 1 -- 172.21.15.67:0/329950872 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2090037070 con 0x7f20a00b34a0 2024-03-24T00:45:46.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.120+0000 7f20977fe640 1 --2- 172.21.15.67:0/329950872 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f208006bbd0 0x7f208006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.120+0000 7f20977fe640 1 -- 172.21.15.67:0/329950872 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 6303+0+0 (secure 0 0 0) 0x7f20900acb00 con 0x7f20a00b34a0 2024-03-24T00:45:46.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.120+0000 7f2095d9b640 1 --2- 172.21.15.67:0/329950872 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f208006bbd0 0x7f208006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.121+0000 7f2095d9b640 1 --2- 172.21.15.67:0/329950872 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f208006bbd0 0x7f208006e090 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f20a00b6f50 tx=0x7f2088007ca0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:46.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.122+0000 7f20a7bbe740 1 Processor -- start 2024-03-24T00:45:46.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.122+0000 7f20a7bbe740 1 -- start start 2024-03-24T00:45:46.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.123+0000 7f20a7bbe740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558a3dd16e50 0x558a3dce9df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.123+0000 7f20a7bbe740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558a3dd17230 0x558a3dd19be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.123+0000 7f20a7bbe740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558a3dcea330 0x558a3dd1a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.123+0000 7f20a7bbe740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a3dc543d0 con 0x558a3dd16e50 2024-03-24T00:45:46.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.123+0000 7f20a7bbe740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a3dbfcad0 con 0x558a3dcea330 2024-03-24T00:45:46.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.123+0000 7f20a7bbe740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x558a3dc50470 con 0x558a3dd17230 2024-03-24T00:45:46.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.123+0000 7f209e59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558a3dd16e50 0x558a3dce9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.123+0000 7f209e59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558a3dd16e50 0x558a3dce9df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37340/0 (socket says 172.21.15.67:37340) 2024-03-24T00:45:46.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.123+0000 7f209e59c640 1 -- 172.21.15.67:0/1620658155 learned_addr learned my addr 172.21.15.67:0/1620658155 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:46.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.123+0000 7f209ed9d640 1 --2- 172.21.15.67:0/1620658155 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558a3dcea330 0x558a3dd1a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.123+0000 7f2095d9b640 1 --2- 172.21.15.67:0/1620658155 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558a3dd17230 0x558a3dd19be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.124+0000 7f209e59c640 1 -- 172.21.15.67:0/1620658155 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558a3dd17230 msgr2=0x558a3dd19be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:46.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.124+0000 7f209e59c640 1 --2- 172.21.15.67:0/1620658155 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558a3dd17230 0x558a3dd19be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.124+0000 7f209e59c640 1 -- 172.21.15.67:0/1620658155 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558a3dcea330 msgr2=0x558a3dd1a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:46.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.124+0000 7f209e59c640 1 --2- 172.21.15.67:0/1620658155 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558a3dcea330 0x558a3dd1a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.124+0000 7f209e59c640 1 -- 172.21.15.67:0/1620658155 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558a3da327d0 con 0x558a3dd16e50 2024-03-24T00:45:46.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.124+0000 7f209e59c640 1 --2- 172.21.15.67:0/1620658155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558a3dd16e50 0x558a3dce9df0 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f20900aade0 tx=0x7f20900770f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad28b329e17506f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:46.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.124+0000 7f207ffff640 1 -- 172.21.15.67:0/1620658155 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2090083560 con 0x558a3dd16e50 2024-03-24T00:45:46.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.124+0000 7f207ffff640 1 -- 172.21.15.67:0/1620658155 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f209007d070 con 0x558a3dd16e50 2024-03-24T00:45:46.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.124+0000 7f207ffff640 1 -- 172.21.15.67:0/1620658155 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2090079150 con 0x558a3dd16e50 2024-03-24T00:45:46.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.125+0000 7f20a7bbe740 1 -- 172.21.15.67:0/1620658155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558a3dd16e50 msgr2=0x558a3dce9df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:46.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.125+0000 7f20a7bbe740 1 --2- 172.21.15.67:0/1620658155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558a3dd16e50 0x558a3dce9df0 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f20900aade0 tx=0x7f20900770f0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.125+0000 7f20a7bbe740 1 -- 172.21.15.67:0/1620658155 shutdown_connections 2024-03-24T00:45:46.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.125+0000 7f20a7bbe740 1 --2- 172.21.15.67:0/1620658155 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558a3dcea330 0x558a3dd1a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.125+0000 7f20a7bbe740 1 --2- 172.21.15.67:0/1620658155 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558a3dd17230 0x558a3dd19be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.125+0000 7f20a7bbe740 1 --2- 172.21.15.67:0/1620658155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558a3dd16e50 0x558a3dce9df0 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.125+0000 7f20a7bbe740 1 -- 172.21.15.67:0/1620658155 >> 172.21.15.67:0/1620658155 conn(0x558a3dcec060 msgr2=0x558a3dd34be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:46.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.125+0000 7f20a7bbe740 1 -- 172.21.15.67:0/1620658155 shutdown_connections 2024-03-24T00:45:46.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.125+0000 7f20a7bbe740 1 -- 172.21.15.67:0/1620658155 wait complete. 2024-03-24T00:45:46.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.126+0000 7f20a7bbe740 1 -- start start 2024-03-24T00:45:46.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.126+0000 7f20a7bbe740 1 Processor -- start 2024-03-24T00:45:46.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.126+0000 7f20a7bbe740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558a3dd16e50 0x558a3dd206b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.126+0000 7f20a7bbe740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558a3dd17230 0x558a3dd1ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.126+0000 7f20a7bbe740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558a3dd44c40 0x558a3dd34be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.126+0000 7f20a7bbe740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a3dc543d0 con 0x558a3dd17230 2024-03-24T00:45:46.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.126+0000 7f20a7bbe740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a3dc10060 con 0x558a3dd44c40 2024-03-24T00:45:46.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.126+0000 7f20a7bbe740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x558a3dcee700 con 0x558a3dd16e50 2024-03-24T00:45:46.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.127+0000 7f20a7bbe740 10 client.0 _ll_register_callbacks cb 0x558a3dbf26a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:45:46.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.127+0000 7f2095d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558a3dd17230 0x558a3dd1ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.127+0000 7f2095d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558a3dd17230 0x558a3dd1ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37356/0 (socket says 172.21.15.67:37356) 2024-03-24T00:45:46.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.127+0000 7f2095d9b640 1 -- 172.21.15.67:0/924602165 learned_addr learned my addr 172.21.15.67:0/924602165 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:46.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.127+0000 7f209ed9d640 1 --2- 172.21.15.67:0/924602165 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558a3dd44c40 0x558a3dd34be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.127+0000 7f209e59c640 1 --2- 172.21.15.67:0/924602165 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558a3dd16e50 0x558a3dd206b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.127+0000 7f2095d9b640 1 -- 172.21.15.67:0/924602165 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558a3dd16e50 msgr2=0x558a3dd206b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:46.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.127+0000 7f2095d9b640 1 --2- 172.21.15.67:0/924602165 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558a3dd16e50 0x558a3dd206b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.127+0000 7f2095d9b640 1 -- 172.21.15.67:0/924602165 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558a3dd44c40 msgr2=0x558a3dd34be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:46.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.127+0000 7f2095d9b640 1 --2- 172.21.15.67:0/924602165 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558a3dd44c40 0x558a3dd34be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.127+0000 7f2095d9b640 1 -- 172.21.15.67:0/924602165 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558a3d9683f0 con 0x558a3dd17230 2024-03-24T00:45:46.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.127+0000 7f209ed9d640 1 --2- 172.21.15.67:0/924602165 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558a3dd44c40 0x558a3dd34be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:45:46.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.127+0000 7f209e59c640 1 --2- 172.21.15.67:0/924602165 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558a3dd16e50 0x558a3dd206b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:45:46.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.127+0000 7f2095d9b640 1 --2- 172.21.15.67:0/924602165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558a3dd17230 0x558a3dd1ef90 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f2088001d50 tx=0x7f2088000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c480e81bc40c8480 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:46.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.127+0000 7f207d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:45:46.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.127+0000 7f207d7fa640 1 -- 172.21.15.67:0/924602165 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2088014e40 con 0x558a3dd17230 2024-03-24T00:45:46.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.127+0000 7f207d7fa640 1 -- 172.21.15.67:0/924602165 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 6303+0+0 (secure 0 0 0) 0x7f2088010040 con 0x558a3dd17230 2024-03-24T00:45:46.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.127+0000 7f20a7bbe740 1 -- 172.21.15.67:0/924602165 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f209007d070 con 0x558a3dd17230 2024-03-24T00:45:46.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.127+0000 7f207d7fa640 1 -- 172.21.15.67:0/924602165 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) v2 ==== 24+0+0 (secure 0 0 0) 0x7f208802e050 con 0x558a3dd17230 2024-03-24T00:45:46.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.127+0000 7f20a7bbe740 10 client.7989 fetch_fsmap learned FSMap version 61 2024-03-24T00:45:46.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.127+0000 7f20a7bbe740 1 -- 172.21.15.67:0/924602165 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=2+}) v3 -- 0x558a3d9280f0 con 0x558a3dd17230 2024-03-24T00:45:46.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.128+0000 7f207d7fa640 1 -- 172.21.15.67:0/924602165 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f209007d070 con 0x558a3dd17230 2024-03-24T00:45:46.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.128+0000 7f207d7fa640 1 -- 172.21.15.67:0/924602165 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 61) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f2088031020 con 0x558a3dd17230 2024-03-24T00:45:46.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.128+0000 7f20a7bbe740 10 client.7989 fetch_fsmap finished waiting for FSMap version 61 2024-03-24T00:45:46.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.128+0000 7f20a7bbe740 10 client.7989 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-03-24T00:45:46.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.128+0000 7f20a7bbe740 20 client.7989 populate_metadata read hostname 'smithi067' 2024-03-24T00:45:46.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.128+0000 7f20a7bbe740 1 --2- 172.21.15.67:0/924602165 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x558a3dd1bbc0 0x558a3dd1c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.128+0000 7f20a7bbe740 4 client.7989 mds_command: new command op to 7860 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:45:46.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.128+0000 7f20a7bbe740 1 -- 172.21.15.67:0/924602165 --> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558a3dc61910 con 0x558a3dd1bbc0 2024-03-24T00:45:46.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.128+0000 7f209e59c640 1 --2- 172.21.15.67:0/924602165 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x558a3dd1bbc0 0x558a3dd1c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.128+0000 7f207d7fa640 10 client.7989 ms_handle_connect on v2:172.21.15.67:6832/1336395002 2024-03-24T00:45:46.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.128+0000 7f209e59c640 1 --2- 172.21.15.67:0/924602165 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x558a3dd1bbc0 0x558a3dd1c030 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49c6200835aaac95 server_cookie=550c5745663b286d in_seq=0 out_seq=0 2024-03-24T00:45:46.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.129+0000 7f207d7fa640 1 -- 172.21.15.67:0/924602165 <== mds.0 v2:172.21.15.67:6832/1336395002 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x558a3dc61910 con 0x558a3dd1bbc0 2024-03-24T00:45:46.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.129+0000 7f207d7fa640 10 client.7989 handle_command_reply: tid=0 2024-03-24T00:45:46.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.129+0000 7f20a7bbe740 1 client.7989 shutdown 2024-03-24T00:45:46.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.129+0000 7f20a7bbe740 1 -- 172.21.15.67:0/924602165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558a3dd17230 msgr2=0x558a3dd1ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:46.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.129+0000 7f20a7bbe740 1 --2- 172.21.15.67:0/924602165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558a3dd17230 0x558a3dd1ef90 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f2088001d50 tx=0x7f2088000ed0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.129+0000 7f20a7bbe740 1 -- 172.21.15.67:0/924602165 shutdown_connections 2024-03-24T00:45:46.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.129+0000 7f20a7bbe740 1 --2- 172.21.15.67:0/924602165 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x558a3dd1bbc0 0x558a3dd1c030 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.129+0000 7f20a7bbe740 1 --2- 172.21.15.67:0/924602165 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558a3dd44c40 0x558a3dd34be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.129+0000 7f20a7bbe740 1 --2- 172.21.15.67:0/924602165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558a3dd17230 0x558a3dd1ef90 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.129+0000 7f207d7fa640 0 client.7989 ms_handle_reset on v2:172.21.15.67:6832/1336395002 2024-03-24T00:45:46.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.129+0000 7f20a7bbe740 1 --2- 172.21.15.67:0/924602165 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558a3dd16e50 0x558a3dd206b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.129+0000 7f20a7bbe740 1 -- 172.21.15.67:0/924602165 >> 172.21.15.67:0/924602165 conn(0x558a3dcea330 msgr2=0x558a3dcf0c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:46.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.129+0000 7f20a7bbe740 1 -- 172.21.15.67:0/924602165 shutdown_connections 2024-03-24T00:45:46.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.129+0000 7f20a7bbe740 1 -- 172.21.15.67:0/924602165 wait complete. 2024-03-24T00:45:46.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.129+0000 7f20a7bbe740 20 client.7989 trim_cache size 0 max 16384 2024-03-24T00:45:46.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.145+0000 7f20a7bbe740 1 Processor -- start 2024-03-24T00:45:46.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.145+0000 7f20a7bbe740 1 -- start start 2024-03-24T00:45:46.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.146+0000 7f20a7bbe740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558a3dd16e50 0x558a3dd380f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.146+0000 7f20a7bbe740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558a3dd17230 0x558a3dd38630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.146+0000 7f20a7bbe740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558a3dd44c40 0x558a3dd19620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.146+0000 7f20a7bbe740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a0200930 con 0x558a3dd16e50 2024-03-24T00:45:46.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.146+0000 7f20a7bbe740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a01ffab0 con 0x558a3dd44c40 2024-03-24T00:45:46.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.146+0000 7f20a7bbe740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20a01f7b50 con 0x558a3dd17230 2024-03-24T00:45:46.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.146+0000 7f209e59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558a3dd16e50 0x558a3dd380f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.146+0000 7f209e59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558a3dd16e50 0x558a3dd380f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37372/0 (socket says 172.21.15.67:37372) 2024-03-24T00:45:46.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.146+0000 7f209e59c640 1 -- 172.21.15.67:0/1069045328 learned_addr learned my addr 172.21.15.67:0/1069045328 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:46.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.146+0000 7f2095d9b640 1 --2- 172.21.15.67:0/1069045328 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558a3dd17230 0x558a3dd38630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.146+0000 7f209ed9d640 1 --2- 172.21.15.67:0/1069045328 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558a3dd44c40 0x558a3dd19620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.146+0000 7f209e59c640 1 -- 172.21.15.67:0/1069045328 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558a3dd17230 msgr2=0x558a3dd38630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:46.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.146+0000 7f209e59c640 1 --2- 172.21.15.67:0/1069045328 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558a3dd17230 0x558a3dd38630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.146+0000 7f209e59c640 1 -- 172.21.15.67:0/1069045328 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558a3dd44c40 msgr2=0x558a3dd19620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:46.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.146+0000 7f209e59c640 1 --2- 172.21.15.67:0/1069045328 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558a3dd44c40 0x558a3dd19620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.146+0000 7f209e59c640 1 -- 172.21.15.67:0/1069045328 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558a3dd21250 con 0x558a3dd16e50 2024-03-24T00:45:46.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.146+0000 7f209ed9d640 1 --2- 172.21.15.67:0/1069045328 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558a3dd44c40 0x558a3dd19620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:45:46.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.146+0000 7f209e59c640 1 --2- 172.21.15.67:0/1069045328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558a3dd16e50 0x558a3dd380f0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f20900aaec0 tx=0x7f2090044000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=274d633ff62cb34 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:46.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.147+0000 7f207f7fe640 1 -- 172.21.15.67:0/1069045328 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2090078ef0 con 0x558a3dd16e50 2024-03-24T00:45:46.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.147+0000 7f207f7fe640 1 -- 172.21.15.67:0/1069045328 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2090079150 con 0x558a3dd16e50 2024-03-24T00:45:46.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.147+0000 7f207f7fe640 1 -- 172.21.15.67:0/1069045328 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20900810b0 con 0x558a3dd16e50 2024-03-24T00:45:46.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.147+0000 7f20a7bbe740 1 -- 172.21.15.67:0/1069045328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558a3dd16e50 msgr2=0x558a3dd380f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:46.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.147+0000 7f20a7bbe740 1 --2- 172.21.15.67:0/1069045328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558a3dd16e50 0x558a3dd380f0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f20900aaec0 tx=0x7f2090044000 comp rx=0 tx=0).stop 2024-03-24T00:45:46.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.147+0000 7f20a7bbe740 1 -- 172.21.15.67:0/1069045328 shutdown_connections 2024-03-24T00:45:46.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.147+0000 7f20a7bbe740 1 --2- 172.21.15.67:0/1069045328 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558a3dd44c40 0x558a3dd19620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.147+0000 7f20a7bbe740 1 --2- 172.21.15.67:0/1069045328 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558a3dd17230 0x558a3dd38630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.147+0000 7f20a7bbe740 1 --2- 172.21.15.67:0/1069045328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558a3dd16e50 0x558a3dd380f0 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.147+0000 7f20a7bbe740 1 -- 172.21.15.67:0/1069045328 >> 172.21.15.67:0/1069045328 conn(0x558a3dcea330 msgr2=0x558a3dd349b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:46.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.147+0000 7f20a7bbe740 1 -- 172.21.15.67:0/1069045328 shutdown_connections 2024-03-24T00:45:46.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.147+0000 7f20a7bbe740 1 -- 172.21.15.67:0/1069045328 wait complete. 2024-03-24T00:45:46.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.148+0000 7f20a7bbe740 1 -- start start 2024-03-24T00:45:46.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.148+0000 7f20a7bbe740 1 Processor -- start 2024-03-24T00:45:46.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.150+0000 7f20a7bbe740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558a3dd16e50 0x558a3dd21430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.150+0000 7f20a7bbe740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558a3dd17230 0x558a3dd48760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.150+0000 7f20a7bbe740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558a3dd44c40 0x558a3dd48ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.150+0000 7f20a7bbe740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a3dceb960 con 0x558a3dd16e50 2024-03-24T00:45:46.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.150+0000 7f20a7bbe740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a01ff4b0 con 0x558a3dd44c40 2024-03-24T00:45:46.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.150+0000 7f20a7bbe740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x558a3dd1a780 con 0x558a3dd17230 2024-03-24T00:45:46.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.150+0000 7f20a7bbe740 10 client.0 _ll_register_callbacks cb 0x7f2088005d80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:45:46.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.150+0000 7f209e59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558a3dd16e50 0x558a3dd21430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.150+0000 7f209e59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558a3dd16e50 0x558a3dd21430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37376/0 (socket says 172.21.15.67:37376) 2024-03-24T00:45:46.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.150+0000 7f209e59c640 1 -- 172.21.15.67:0/2561682641 learned_addr learned my addr 172.21.15.67:0/2561682641 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:46.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.150+0000 7f209ed9d640 1 --2- 172.21.15.67:0/2561682641 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558a3dd44c40 0x558a3dd48ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.150+0000 7f2095d9b640 1 --2- 172.21.15.67:0/2561682641 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558a3dd17230 0x558a3dd48760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.150+0000 7f209e59c640 1 -- 172.21.15.67:0/2561682641 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558a3dd17230 msgr2=0x558a3dd48760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:46.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.150+0000 7f209e59c640 1 --2- 172.21.15.67:0/2561682641 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558a3dd17230 0x558a3dd48760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.150+0000 7f209e59c640 1 -- 172.21.15.67:0/2561682641 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558a3dd44c40 msgr2=0x558a3dd48ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:46.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.150+0000 7f209e59c640 1 --2- 172.21.15.67:0/2561682641 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558a3dd44c40 0x558a3dd48ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.150+0000 7f209e59c640 1 -- 172.21.15.67:0/2561682641 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558a3dd165b0 con 0x558a3dd16e50 2024-03-24T00:45:46.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.150+0000 7f209e59c640 1 --2- 172.21.15.67:0/2561682641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558a3dd16e50 0x558a3dd21430 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f209007ab50 tx=0x7f20900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d15c5b9ec79a4d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:46.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.150+0000 7f207d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:45:46.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.150+0000 7f207d7fa640 1 -- 172.21.15.67:0/2561682641 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2090083560 con 0x558a3dd16e50 2024-03-24T00:45:46.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.150+0000 7f207d7fa640 1 -- 172.21.15.67:0/2561682641 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 6303+0+0 (secure 0 0 0) 0x7f20900774c0 con 0x558a3dd16e50 2024-03-24T00:45:46.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.150+0000 7f20a7bbe740 1 -- 172.21.15.67:0/2561682641 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2090079150 con 0x558a3dd16e50 2024-03-24T00:45:46.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.150+0000 7f207d7fa640 1 -- 172.21.15.67:0/2561682641 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) v2 ==== 24+0+0 (secure 0 0 0) 0x7f20900810b0 con 0x558a3dd16e50 2024-03-24T00:45:46.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.150+0000 7f20a7bbe740 10 client.7995 fetch_fsmap learned FSMap version 61 2024-03-24T00:45:46.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.150+0000 7f20a7bbe740 1 -- 172.21.15.67:0/2561682641 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=2+}) v3 -- 0x558a3dd16160 con 0x558a3dd16e50 2024-03-24T00:45:46.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.151+0000 7f207d7fa640 1 -- 172.21.15.67:0/2561682641 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2090079150 con 0x558a3dd16e50 2024-03-24T00:45:46.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.151+0000 7f207d7fa640 1 -- 172.21.15.67:0/2561682641 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 61) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f209007a2a0 con 0x558a3dd16e50 2024-03-24T00:45:46.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.151+0000 7f20a7bbe740 10 client.7995 fetch_fsmap finished waiting for FSMap version 61 2024-03-24T00:45:46.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.151+0000 7f20a7bbe740 10 client.7995 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-03-24T00:45:46.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.151+0000 7f20a7bbe740 20 client.7995 populate_metadata read hostname 'smithi067' 2024-03-24T00:45:46.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.151+0000 7f20a7bbe740 1 --2- 172.21.15.67:0/2561682641 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x558a3dd49310 0x558a3dd437e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.151+0000 7f20a7bbe740 4 client.7995 mds_command: new command op to 7860 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:45:46.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.151+0000 7f20a7bbe740 1 -- 172.21.15.67:0/2561682641 --> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x558a3dc5bc60 con 0x558a3dd49310 2024-03-24T00:45:46.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.151+0000 7f2095d9b640 1 --2- 172.21.15.67:0/2561682641 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x558a3dd49310 0x558a3dd437e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.151+0000 7f207d7fa640 10 client.7995 ms_handle_connect on v2:172.21.15.67:6832/1336395002 2024-03-24T00:45:46.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.151+0000 7f2095d9b640 1 --2- 172.21.15.67:0/2561682641 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x558a3dd49310 0x558a3dd437e0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f42abf0d7422eefe server_cookie=5679d18c5572d76a in_seq=0 out_seq=0 2024-03-24T00:45:46.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.151+0000 7f207d7fa640 1 -- 172.21.15.67:0/2561682641 <== mds.0 v2:172.21.15.67:6832/1336395002 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x558a3dc5bc60 con 0x558a3dd49310 2024-03-24T00:45:46.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.151+0000 7f207d7fa640 10 client.7995 handle_command_reply: tid=0 2024-03-24T00:45:46.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.151+0000 7f20a7bbe740 1 client.7995 shutdown 2024-03-24T00:45:46.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.152+0000 7f20a7bbe740 1 -- 172.21.15.67:0/2561682641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558a3dd16e50 msgr2=0x558a3dd21430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:46.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.152+0000 7f20a7bbe740 1 --2- 172.21.15.67:0/2561682641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558a3dd16e50 0x558a3dd21430 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f209007ab50 tx=0x7f20900046c0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.152+0000 7f20a7bbe740 1 -- 172.21.15.67:0/2561682641 shutdown_connections 2024-03-24T00:45:46.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.152+0000 7f20a7bbe740 1 --2- 172.21.15.67:0/2561682641 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x558a3dd49310 0x558a3dd437e0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.152+0000 7f20a7bbe740 1 --2- 172.21.15.67:0/2561682641 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558a3dd44c40 0x558a3dd48ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.152+0000 7f20a7bbe740 1 --2- 172.21.15.67:0/2561682641 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558a3dd17230 0x558a3dd48760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.152+0000 7f207d7fa640 0 client.7995 ms_handle_reset on v2:172.21.15.67:6832/1336395002 2024-03-24T00:45:46.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.152+0000 7f20a7bbe740 1 --2- 172.21.15.67:0/2561682641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558a3dd16e50 0x558a3dd21430 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.152+0000 7f20a7bbe740 1 -- 172.21.15.67:0/2561682641 >> 172.21.15.67:0/2561682641 conn(0x558a3dcea330 msgr2=0x558a3dd37020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:46.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.152+0000 7f20a7bbe740 1 -- 172.21.15.67:0/2561682641 shutdown_connections 2024-03-24T00:45:46.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.152+0000 7f20a7bbe740 1 -- 172.21.15.67:0/2561682641 wait complete. 2024-03-24T00:45:46.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.152+0000 7f20a7bbe740 20 client.7995 trim_cache size 0 max 16384 2024-03-24T00:45:46.158 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:45:46.159 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "no active scrubs running", 2024-03-24T00:45:46.159 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": {} 2024-03-24T00:45:46.159 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:45:46.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.152+0000 7f20a4ef8640 1 -- 172.21.15.67:0/329950872 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f208006bbd0 msgr2=0x7f208006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:46.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.152+0000 7f20a4ef8640 1 --2- 172.21.15.67:0/329950872 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f208006bbd0 0x7f208006e090 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f20a00b6f50 tx=0x7f2088007ca0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.152+0000 7f20a4ef8640 1 -- 172.21.15.67:0/329950872 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20a00b34a0 msgr2=0x7f20a015ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:46.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.152+0000 7f20a4ef8640 1 --2- 172.21.15.67:0/329950872 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20a00b34a0 0x7f20a015ab90 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f209002b9e0 tx=0x7f2090004730 comp rx=0 tx=0).stop 2024-03-24T00:45:46.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.152+0000 7f20a4ef8640 1 -- 172.21.15.67:0/329950872 shutdown_connections 2024-03-24T00:45:46.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.152+0000 7f20a4ef8640 1 --2- 172.21.15.67:0/329950872 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f208006bbd0 0x7f208006e090 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.152+0000 7f20a4ef8640 1 --2- 172.21.15.67:0/329950872 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20a015c010 0x7f20a01f3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.152+0000 7f20a4ef8640 1 --2- 172.21.15.67:0/329950872 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f20a015bc30 0x7f20a00c46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.152+0000 7f20a4ef8640 1 --2- 172.21.15.67:0/329950872 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20a00b34a0 0x7f20a015ab90 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.152+0000 7f20a4ef8640 1 -- 172.21.15.67:0/329950872 >> 172.21.15.67:0/329950872 conn(0x7f20a008b180 msgr2=0x7f20a00701d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:46.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.152+0000 7f20a4ef8640 1 -- 172.21.15.67:0/329950872 shutdown_connections 2024-03-24T00:45:46.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.152+0000 7f20a4ef8640 1 -- 172.21.15.67:0/329950872 wait complete. 2024-03-24T00:45:46.163 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T00:45:46.163 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:45:46.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.265+0000 7f54c66f7640 1 Processor -- start 2024-03-24T00:45:46.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.265+0000 7f54c66f7640 1 -- start start 2024-03-24T00:45:46.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.265+0000 7f54c66f7640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f54c015dec0 0x7f54c00ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.265+0000 7f54c66f7640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f54c015e2a0 0x7f54c0162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.265+0000 7f54c66f7640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f54c00d07e0 0x7f54c00d0c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.265+0000 7f54c66f7640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54c00d1190 con 0x7f54c015e2a0 2024-03-24T00:45:46.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.265+0000 7f54c66f7640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54c00cee30 con 0x7f54c00d07e0 2024-03-24T00:45:46.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.265+0000 7f54c66f7640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54c00d2120 con 0x7f54c015dec0 2024-03-24T00:45:46.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.265+0000 7f54bf7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f54c015e2a0 0x7f54c0162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.266+0000 7f54bf7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f54c015e2a0 0x7f54c0162630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37380/0 (socket says 172.21.15.67:37380) 2024-03-24T00:45:46.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.266+0000 7f54bf7fe640 1 -- 172.21.15.67:0/3359635562 learned_addr learned my addr 172.21.15.67:0/3359635562 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:46.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.266+0000 7f54bffff640 1 --2- 172.21.15.67:0/3359635562 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f54c015dec0 0x7f54c00ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.266+0000 7f54c4c94640 1 --2- 172.21.15.67:0/3359635562 >> v2:172.21.15.72:3300/0 conn(0x7f54c00d07e0 0x7f54c00d0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.266+0000 7f54bf7fe640 1 -- 172.21.15.67:0/3359635562 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f54c015dec0 msgr2=0x7f54c00ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:46.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.266+0000 7f54bf7fe640 1 --2- 172.21.15.67:0/3359635562 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f54c015dec0 0x7f54c00ce630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.266+0000 7f54bf7fe640 1 -- 172.21.15.67:0/3359635562 >> v2:172.21.15.72:3300/0 conn(0x7f54c00d07e0 msgr2=0x7f54c00d0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:46.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.266+0000 7f54bf7fe640 1 --2- 172.21.15.67:0/3359635562 >> v2:172.21.15.72:3300/0 conn(0x7f54c00d07e0 0x7f54c00d0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.266+0000 7f54bf7fe640 1 -- 172.21.15.67:0/3359635562 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54c00cf640 con 0x7f54c015e2a0 2024-03-24T00:45:46.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.267+0000 7f54bf7fe640 1 --2- 172.21.15.67:0/3359635562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54c015e2a0 0x7f54c0162630 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f54c00d1a50 tx=0x7f54b402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14ce9dfc3e92377c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:46.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.267+0000 7f54beffd640 1 -- 172.21.15.67:0/3359635562 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b402e070 con 0x7f54c015e2a0 2024-03-24T00:45:46.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.267+0000 7f54beffd640 1 -- 172.21.15.67:0/3359635562 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54b4037070 con 0x7f54c015e2a0 2024-03-24T00:45:46.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.267+0000 7f54beffd640 1 -- 172.21.15.67:0/3359635562 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b4031a00 con 0x7f54c015e2a0 2024-03-24T00:45:46.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.267+0000 7f54c66f7640 1 -- 172.21.15.67:0/3359635562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54c015e2a0 msgr2=0x7f54c0162630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:46.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.267+0000 7f54c66f7640 1 --2- 172.21.15.67:0/3359635562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54c015e2a0 0x7f54c0162630 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f54c00d1a50 tx=0x7f54b402b880 comp rx=0 tx=0).stop 2024-03-24T00:45:46.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.267+0000 7f54c66f7640 1 -- 172.21.15.67:0/3359635562 shutdown_connections 2024-03-24T00:45:46.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.267+0000 7f54c66f7640 1 --2- 172.21.15.67:0/3359635562 >> v2:172.21.15.72:3300/0 conn(0x7f54c00d07e0 0x7f54c00d0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.267+0000 7f54c66f7640 1 --2- 172.21.15.67:0/3359635562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54c015e2a0 0x7f54c0162630 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.267+0000 7f54c66f7640 1 --2- 172.21.15.67:0/3359635562 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f54c015dec0 0x7f54c00ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.267+0000 7f54c66f7640 1 -- 172.21.15.67:0/3359635562 >> 172.21.15.67:0/3359635562 conn(0x7f54c006c810 msgr2=0x7f54c006cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:46.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.267+0000 7f54c66f7640 1 -- 172.21.15.67:0/3359635562 shutdown_connections 2024-03-24T00:45:46.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.267+0000 7f54c66f7640 1 -- 172.21.15.67:0/3359635562 wait complete. 2024-03-24T00:45:46.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.268+0000 7f54c66f7640 1 Processor -- start 2024-03-24T00:45:46.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.269+0000 7f54c66f7640 1 -- start start 2024-03-24T00:45:46.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.269+0000 7f54c66f7640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f54c00d07e0 0x7f54c00c97e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.269+0000 7f54c66f7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54c015dec0 0x7f54c014b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.269+0000 7f54c66f7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54c015e2a0 0x7f54c01e7240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.269+0000 7f54c66f7640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c00b5850 con 0x7f54c015dec0 2024-03-24T00:45:46.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.269+0000 7f54c66f7640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c00cf360 con 0x7f54c015e2a0 2024-03-24T00:45:46.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.269+0000 7f54c66f7640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54c00b59d0 con 0x7f54c00d07e0 2024-03-24T00:45:46.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.269+0000 7f54bf7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54c015dec0 0x7f54c014b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.269+0000 7f54bf7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54c015dec0 0x7f54c014b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37390/0 (socket says 172.21.15.67:37390) 2024-03-24T00:45:46.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.269+0000 7f54bf7fe640 1 -- 172.21.15.67:0/511227717 learned_addr learned my addr 172.21.15.67:0/511227717 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:46.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.269+0000 7f54bffff640 1 --2- 172.21.15.67:0/511227717 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f54c00d07e0 0x7f54c00c97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.269+0000 7f54c4c94640 1 --2- 172.21.15.67:0/511227717 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54c015e2a0 0x7f54c01e7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.270+0000 7f54bf7fe640 1 -- 172.21.15.67:0/511227717 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f54c00d07e0 msgr2=0x7f54c00c97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:46.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.270+0000 7f54bf7fe640 1 --2- 172.21.15.67:0/511227717 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f54c00d07e0 0x7f54c00c97e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.270+0000 7f54bf7fe640 1 -- 172.21.15.67:0/511227717 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54c015e2a0 msgr2=0x7f54c01e7240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:46.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.270+0000 7f54bf7fe640 1 --2- 172.21.15.67:0/511227717 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54c015e2a0 0x7f54c01e7240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.270+0000 7f54bf7fe640 1 -- 172.21.15.67:0/511227717 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54c014bc90 con 0x7f54c015dec0 2024-03-24T00:45:46.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.270+0000 7f54bf7fe640 1 --2- 172.21.15.67:0/511227717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54c015dec0 0x7f54c014b3c0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f54b4000c00 tx=0x7f54b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:46.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.270+0000 7f54bcff9640 1 -- 172.21.15.67:0/511227717 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b4032880 con 0x7f54c015dec0 2024-03-24T00:45:46.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.270+0000 7f54bcff9640 1 -- 172.21.15.67:0/511227717 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54b4032a20 con 0x7f54c015dec0 2024-03-24T00:45:46.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.270+0000 7f54c66f7640 1 -- 172.21.15.67:0/511227717 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54c014bf70 con 0x7f54c015dec0 2024-03-24T00:45:46.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.270+0000 7f54bcff9640 1 -- 172.21.15.67:0/511227717 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b4032e10 con 0x7f54c015dec0 2024-03-24T00:45:46.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.270+0000 7f54c66f7640 1 -- 172.21.15.67:0/511227717 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54c014c530 con 0x7f54c015dec0 2024-03-24T00:45:46.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.271+0000 7f54c66f7640 1 -- 172.21.15.67:0/511227717 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54c004f400 con 0x7f54c015dec0 2024-03-24T00:45:46.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.272+0000 7f54bcff9640 1 -- 172.21.15.67:0/511227717 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f54b40453f0 con 0x7f54c015dec0 2024-03-24T00:45:46.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.272+0000 7f54bcff9640 1 --2- 172.21.15.67:0/511227717 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f54ac06bbd0 0x7f54ac06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.272+0000 7f54bcff9640 1 -- 172.21.15.67:0/511227717 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 6303+0+0 (secure 0 0 0) 0x7f54b40acd20 con 0x7f54c015dec0 2024-03-24T00:45:46.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.272+0000 7f54bffff640 1 --2- 172.21.15.67:0/511227717 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f54ac06bbd0 0x7f54ac06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.273+0000 7f54bffff640 1 --2- 172.21.15.67:0/511227717 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f54ac06bbd0 0x7f54ac06e090 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f54b000aa40 tx=0x7f54b0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:46.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.281+0000 7f54bcff9640 1 -- 172.21.15.67:0/511227717 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f54b4040070 con 0x7f54c015dec0 2024-03-24T00:45:46.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.500+0000 7f54c66f7640 1 -- 172.21.15.67:0/511227717 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54c00b7e60 con 0x7f54c015dec0 2024-03-24T00:45:46.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.501+0000 7f54bcff9640 1 -- 172.21.15.67:0/511227717 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f54c00cf640 con 0x7f54c015dec0 2024-03-24T00:45:46.501 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:45:46.501 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":61,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6359,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2664460392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2664460392},{"type":"v1","addr":"172.21.15.67:6835","nonce":2664460392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6368,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/3323873506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3323873506},{"type":"v1","addr":"172.21.15.72:6837","nonce":3323873506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":7150,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/2744016217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2744016217},{"type":"v1","addr":"172.21.15.72:6836","nonce":2744016217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:44:54.749656+0000","modified":"2024-03-24T00:45:20.120495+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7860},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7860":{"gid":7860,"name":"c","rank":0,"incarnation":57,"state":"up:active","state_seq":6,"addr":"172.21.15.67:6833/1336395002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1336395002},{"type":"v1","addr":"172.21.15.67:6833","nonce":1336395002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7860,"qdb_cluster":[7860]},"id":6}]} 2024-03-24T00:45:46.501 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 61 2024-03-24T00:45:46.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.503+0000 7f54c66f7640 1 -- 172.21.15.67:0/511227717 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f54ac06bbd0 msgr2=0x7f54ac06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:46.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.503+0000 7f54c66f7640 1 --2- 172.21.15.67:0/511227717 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f54ac06bbd0 0x7f54ac06e090 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f54b000aa40 tx=0x7f54b0008040 comp rx=0 tx=0).stop 2024-03-24T00:45:46.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.503+0000 7f54c66f7640 1 -- 172.21.15.67:0/511227717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54c015dec0 msgr2=0x7f54c014b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:46.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.503+0000 7f54c66f7640 1 --2- 172.21.15.67:0/511227717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54c015dec0 0x7f54c014b3c0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f54b4000c00 tx=0x7f54b40046c0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.503+0000 7f54c66f7640 1 -- 172.21.15.67:0/511227717 shutdown_connections 2024-03-24T00:45:46.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.503+0000 7f54c66f7640 1 --2- 172.21.15.67:0/511227717 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f54ac06bbd0 0x7f54ac06e090 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.503+0000 7f54c66f7640 1 --2- 172.21.15.67:0/511227717 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54c015e2a0 0x7f54c01e7240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.503+0000 7f54c66f7640 1 --2- 172.21.15.67:0/511227717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54c015dec0 0x7f54c014b3c0 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.503+0000 7f54c66f7640 1 --2- 172.21.15.67:0/511227717 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f54c00d07e0 0x7f54c00c97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.503+0000 7f54c66f7640 1 -- 172.21.15.67:0/511227717 >> 172.21.15.67:0/511227717 conn(0x7f54c006c810 msgr2=0x7f54c00b7190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:46.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.503+0000 7f54c66f7640 1 -- 172.21.15.67:0/511227717 shutdown_connections 2024-03-24T00:45:46.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.503+0000 7f54c66f7640 1 -- 172.21.15.67:0/511227717 wait complete. 2024-03-24T00:45:46.518 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c damage ls --format=json-pretty 2024-03-24T00:45:46.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.621+0000 7fb8123e6640 1 Processor -- start 2024-03-24T00:45:46.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.621+0000 7fb8123e6640 1 -- start start 2024-03-24T00:45:46.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.622+0000 7fb8123e6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb80c15c1b0 0x7fb80c15fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.622+0000 7fb8123e6640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb80c160260 0x7fb80c0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.622+0000 7fb8123e6640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb80c15c7c0 con 0x7fb80c0d18b0 2024-03-24T00:45:46.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.622+0000 7fb8123e6640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb80c15caa0 con 0x7fb80c160260 2024-03-24T00:45:46.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.622+0000 7fb8123e6640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb80c15cd80 con 0x7fb80c15c1b0 2024-03-24T00:45:46.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.622+0000 7fb80bfff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb80c0d18b0 0x7fb80c0d1c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47614/0 (socket says 172.21.15.67:47614) 2024-03-24T00:45:46.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.622+0000 7fb80bfff640 1 -- 172.21.15.67:0/2655167260 learned_addr learned my addr 172.21.15.67:0/2655167260 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:45:46.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.622+0000 7fb80b7fe640 1 --2- 172.21.15.67:0/2655167260 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb80c15c1b0 0x7fb80c15fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.622+0000 7fb80affd640 1 -- 172.21.15.67:0/2655167260 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3891403168 0 0) 0x7fb80c15c7c0 con 0x7fb80c0d18b0 2024-03-24T00:45:46.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.622+0000 7fb810983640 1 --2- 172.21.15.67:0/2655167260 >> v2:172.21.15.72:3300/0 conn(0x7fb80c160260 0x7fb80c0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.622+0000 7fb80affd640 1 -- 172.21.15.67:0/2655167260 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7f4003520 con 0x7fb80c0d18b0 2024-03-24T00:45:46.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.622+0000 7fb80affd640 1 -- 172.21.15.67:0/2655167260 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1554534834 0 0) 0x7fb7f4003520 con 0x7fb80c0d18b0 2024-03-24T00:45:46.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.622+0000 7fb80affd640 1 -- 172.21.15.67:0/2655167260 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb80c15c1b0 msgr2=0x7fb80c15fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:46.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.622+0000 7fb80affd640 1 --2- 172.21.15.67:0/2655167260 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb80c15c1b0 0x7fb80c15fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.622+0000 7fb80affd640 1 -- 172.21.15.67:0/2655167260 >> v2:172.21.15.72:3300/0 conn(0x7fb80c160260 msgr2=0x7fb80c0cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:46.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.622+0000 7fb80affd640 1 --2- 172.21.15.67:0/2655167260 >> v2:172.21.15.72:3300/0 conn(0x7fb80c160260 0x7fb80c0cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.622+0000 7fb80affd640 1 -- 172.21.15.67:0/2655167260 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb80c15d060 con 0x7fb80c0d18b0 2024-03-24T00:45:46.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.622+0000 7fb80affd640 1 -- 172.21.15.67:0/2655167260 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7fb7fc001670 con 0x7fb80c0d18b0 2024-03-24T00:45:46.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.623+0000 7fb80affd640 1 -- 172.21.15.67:0/2655167260 >> v1:172.21.15.67:6789/0 conn(0x7fb80c0d18b0 legacy=0x7fb80c0d1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:46.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.623+0000 7fb80affd640 1 --2- 172.21.15.67:0/2655167260 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7f40039b0 0x7fb7f4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.623+0000 7fb80affd640 1 --2- 172.21.15.67:0/2655167260 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7f4004800 0x7fb7f4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.623+0000 7fb80affd640 1 --2- 172.21.15.67:0/2655167260 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb80c15c1b0 0x7fb7f40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.623+0000 7fb80affd640 1 -- 172.21.15.67:0/2655167260 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb80c15caa0 con 0x7fb7f40039b0 2024-03-24T00:45:46.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.623+0000 7fb80affd640 1 -- 172.21.15.67:0/2655167260 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb80c15cd80 con 0x7fb7f4004800 2024-03-24T00:45:46.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.623+0000 7fb80affd640 1 -- 172.21.15.67:0/2655167260 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f400b9c0 con 0x7fb80c15c1b0 2024-03-24T00:45:46.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.623+0000 7fb80bfff640 1 --2- 172.21.15.67:0/2655167260 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7f40039b0 0x7fb7f4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.623+0000 7fb80b7fe640 1 --2- 172.21.15.67:0/2655167260 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7f4004800 0x7fb7f4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.623+0000 7fb810983640 1 --2- 172.21.15.67:0/2655167260 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb80c15c1b0 0x7fb7f40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.623+0000 7fb80bfff640 1 -- 172.21.15.67:0/2655167260 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb80c15c1b0 msgr2=0x7fb7f40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:46.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.623+0000 7fb80bfff640 1 --2- 172.21.15.67:0/2655167260 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb80c15c1b0 0x7fb7f40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.623+0000 7fb80bfff640 1 -- 172.21.15.67:0/2655167260 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7f4004800 msgr2=0x7fb7f4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:46.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.623+0000 7fb80bfff640 1 --2- 172.21.15.67:0/2655167260 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7f4004800 0x7fb7f4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.623+0000 7fb80bfff640 1 -- 172.21.15.67:0/2655167260 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f4003520 con 0x7fb7f40039b0 2024-03-24T00:45:46.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.624+0000 7fb80bfff640 1 --2- 172.21.15.67:0/2655167260 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7f40039b0 0x7fb7f4003dd0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc00c240 tx=0x7fb7f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b94bc0c35392f78 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:46.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.624+0000 7fb80affd640 1 -- 172.21.15.67:0/2655167260 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7fc007d50 con 0x7fb7f40039b0 2024-03-24T00:45:46.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.624+0000 7fb80affd640 1 -- 172.21.15.67:0/2655167260 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7fc030070 con 0x7fb7f40039b0 2024-03-24T00:45:46.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.624+0000 7fb80affd640 1 -- 172.21.15.67:0/2655167260 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7fc02cd80 con 0x7fb7f40039b0 2024-03-24T00:45:46.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.624+0000 7fb8123e6640 1 -- 172.21.15.67:0/2655167260 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7f40039b0 msgr2=0x7fb7f4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:46.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.624+0000 7fb8123e6640 1 --2- 172.21.15.67:0/2655167260 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7f40039b0 0x7fb7f4003dd0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc00c240 tx=0x7fb7f4005830 comp rx=0 tx=0).stop 2024-03-24T00:45:46.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.624+0000 7fb810983640 1 -- 172.21.15.67:0/2655167260 reap_dead start 2024-03-24T00:45:46.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.624+0000 7fb8123e6640 1 -- 172.21.15.67:0/2655167260 shutdown_connections 2024-03-24T00:45:46.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.624+0000 7fb8123e6640 1 -- 172.21.15.67:0/2655167260 >> 172.21.15.67:0/2655167260 conn(0x7fb80c08b1b0 msgr2=0x7fb80c08b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:46.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.624+0000 7fb8123e6640 1 -- 172.21.15.67:0/2655167260 shutdown_connections 2024-03-24T00:45:46.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.624+0000 7fb8123e6640 1 -- 172.21.15.67:0/2655167260 wait complete. 2024-03-24T00:45:46.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.626+0000 7fb8123e6640 1 Processor -- start 2024-03-24T00:45:46.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.626+0000 7fb8123e6640 1 -- start start 2024-03-24T00:45:46.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.628+0000 7fb8123e6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7f40039b0 0x7fb80c14cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.628+0000 7fb8123e6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb80c14d870 0x7fb80c14dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.628+0000 7fb8123e6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb80c14fb70 0x7fb80c14fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.628+0000 7fb8123e6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb80c15fea0 con 0x7fb7f40039b0 2024-03-24T00:45:46.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.628+0000 7fb8123e6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb80c15fd20 con 0x7fb80c14fb70 2024-03-24T00:45:46.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.628+0000 7fb8123e6640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb80c160020 con 0x7fb80c14d870 2024-03-24T00:45:46.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.628+0000 7fb80bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7f40039b0 0x7fb80c14cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.628+0000 7fb80bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7f40039b0 0x7fb80c14cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37412/0 (socket says 172.21.15.67:37412) 2024-03-24T00:45:46.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.628+0000 7fb80bfff640 1 -- 172.21.15.67:0/3045025165 learned_addr learned my addr 172.21.15.67:0/3045025165 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:46.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.628+0000 7fb80b7fe640 1 --2- 172.21.15.67:0/3045025165 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb80c14d870 0x7fb80c14dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.628+0000 7fb810983640 1 --2- 172.21.15.67:0/3045025165 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb80c14fb70 0x7fb80c14fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.628+0000 7fb80bfff640 1 -- 172.21.15.67:0/3045025165 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb80c14d870 msgr2=0x7fb80c14dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:46.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.628+0000 7fb80bfff640 1 --2- 172.21.15.67:0/3045025165 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb80c14d870 0x7fb80c14dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.628+0000 7fb80bfff640 1 -- 172.21.15.67:0/3045025165 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb80c14fb70 msgr2=0x7fb80c14fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:46.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.628+0000 7fb80bfff640 1 --2- 172.21.15.67:0/3045025165 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb80c14fb70 0x7fb80c14fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.628+0000 7fb80bfff640 1 -- 172.21.15.67:0/3045025165 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb80c0ca040 con 0x7fb7f40039b0 2024-03-24T00:45:46.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.628+0000 7fb810983640 1 --2- 172.21.15.67:0/3045025165 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb80c14fb70 0x7fb80c14fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:45:46.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.628+0000 7fb80bfff640 1 --2- 172.21.15.67:0/3045025165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7f40039b0 0x7fb80c14cf20 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc00c240 tx=0x7fb7fc001ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:46.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.628+0000 7fb808ff9640 1 -- 172.21.15.67:0/3045025165 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7fc001c90 con 0x7fb7f40039b0 2024-03-24T00:45:46.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.628+0000 7fb808ff9640 1 -- 172.21.15.67:0/3045025165 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7fc004bf0 con 0x7fb7f40039b0 2024-03-24T00:45:46.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.629+0000 7fb8123e6640 1 -- 172.21.15.67:0/3045025165 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb80c14d460 con 0x7fb7f40039b0 2024-03-24T00:45:46.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.629+0000 7fb808ff9640 1 -- 172.21.15.67:0/3045025165 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7fc007070 con 0x7fb7f40039b0 2024-03-24T00:45:46.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.629+0000 7fb8123e6640 1 -- 172.21.15.67:0/3045025165 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb80c0ca2b0 con 0x7fb7f40039b0 2024-03-24T00:45:46.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.630+0000 7fb808ff9640 1 -- 172.21.15.67:0/3045025165 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb7fc007210 con 0x7fb7f40039b0 2024-03-24T00:45:46.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.631+0000 7fb808ff9640 1 --2- 172.21.15.67:0/3045025165 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb7f806bbd0 0x7fb7f806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.631+0000 7fb808ff9640 1 -- 172.21.15.67:0/3045025165 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 6303+0+0 (secure 0 0 0) 0x7fb7fc0abfa0 con 0x7fb7f40039b0 2024-03-24T00:45:46.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.631+0000 7fb80b7fe640 1 --2- 172.21.15.67:0/3045025165 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb7f806bbd0 0x7fb7f806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.631+0000 7fb814fcf740 1 Processor -- start 2024-03-24T00:45:46.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.631+0000 7fb80b7fe640 1 --2- 172.21.15.67:0/3045025165 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb7f806bbd0 0x7fb7f806e090 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fb80c0c95a0 tx=0x7fb800008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:46.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.631+0000 7fb814fcf740 1 -- start start 2024-03-24T00:45:46.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.632+0000 7fb814fcf740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56243129fc50 0x56243129d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.632+0000 7fb814fcf740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56243127f490 0x562431257180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.632+0000 7fb814fcf740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5624312576c0 0x562431281e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.632+0000 7fb814fcf740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624311bc1f0 con 0x56243127f490 2024-03-24T00:45:46.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.632+0000 7fb814fcf740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x56243115aa80 con 0x56243129fc50 2024-03-24T00:45:46.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.632+0000 7fb814fcf740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5624311b8290 con 0x5624312576c0 2024-03-24T00:45:46.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.632+0000 7fb80b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56243127f490 0x562431257180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.632+0000 7fb80b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56243127f490 0x562431257180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37424/0 (socket says 172.21.15.67:37424) 2024-03-24T00:45:46.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.632+0000 7fb80b7fe640 1 -- 172.21.15.67:0/155805439 learned_addr learned my addr 172.21.15.67:0/155805439 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:46.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.632+0000 7fb810983640 1 --2- 172.21.15.67:0/155805439 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5624312576c0 0x562431281e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.632+0000 7fb80bfff640 1 --2- 172.21.15.67:0/155805439 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56243129fc50 0x56243129d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.632+0000 7fb80b7fe640 1 -- 172.21.15.67:0/155805439 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5624312576c0 msgr2=0x562431281e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:46.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.632+0000 7fb80b7fe640 1 --2- 172.21.15.67:0/155805439 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5624312576c0 0x562431281e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.632+0000 7fb80b7fe640 1 -- 172.21.15.67:0/155805439 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56243129fc50 msgr2=0x56243129d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:46.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.632+0000 7fb80b7fe640 1 --2- 172.21.15.67:0/155805439 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56243129fc50 0x56243129d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.633+0000 7fb80b7fe640 1 -- 172.21.15.67:0/155805439 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562430f9c7c0 con 0x56243127f490 2024-03-24T00:45:46.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.633+0000 7fb80b7fe640 1 --2- 172.21.15.67:0/155805439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56243127f490 0x562431257180 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fb800014e10 tx=0x7fb800017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=996f37f4c3caf4c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:46.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.633+0000 7fb7f1ffb640 1 -- 172.21.15.67:0/155805439 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8000096d0 con 0x56243127f490 2024-03-24T00:45:46.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.633+0000 7fb7f1ffb640 1 -- 172.21.15.67:0/155805439 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb800029070 con 0x56243127f490 2024-03-24T00:45:46.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.633+0000 7fb7f1ffb640 1 -- 172.21.15.67:0/155805439 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb800003da0 con 0x56243127f490 2024-03-24T00:45:46.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.633+0000 7fb814fcf740 1 -- 172.21.15.67:0/155805439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56243127f490 msgr2=0x562431257180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:46.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.633+0000 7fb814fcf740 1 --2- 172.21.15.67:0/155805439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56243127f490 0x562431257180 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fb800014e10 tx=0x7fb800017000 comp rx=0 tx=0).stop 2024-03-24T00:45:46.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.633+0000 7fb814fcf740 1 -- 172.21.15.67:0/155805439 shutdown_connections 2024-03-24T00:45:46.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.633+0000 7fb814fcf740 1 --2- 172.21.15.67:0/155805439 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5624312576c0 0x562431281e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.633+0000 7fb814fcf740 1 --2- 172.21.15.67:0/155805439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56243127f490 0x562431257180 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.633+0000 7fb814fcf740 1 --2- 172.21.15.67:0/155805439 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56243129fc50 0x56243129d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.633+0000 7fb814fcf740 1 -- 172.21.15.67:0/155805439 >> 172.21.15.67:0/155805439 conn(0x562431254570 msgr2=0x56243129ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:46.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.633+0000 7fb814fcf740 1 -- 172.21.15.67:0/155805439 shutdown_connections 2024-03-24T00:45:46.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.633+0000 7fb814fcf740 1 -- 172.21.15.67:0/155805439 wait complete. 2024-03-24T00:45:46.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.634+0000 7fb814fcf740 1 -- start start 2024-03-24T00:45:46.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.634+0000 7fb814fcf740 1 Processor -- start 2024-03-24T00:45:46.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.635+0000 7fb814fcf740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56243127f490 0x562431298aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.635+0000 7fb814fcf740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56243129fc50 0x562431298fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.635+0000 7fb814fcf740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x562431293f10 0x56243129acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.635+0000 7fb814fcf740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624312565e0 con 0x56243129fc50 2024-03-24T00:45:46.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.635+0000 7fb814fcf740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x562431158410 con 0x562431293f10 2024-03-24T00:45:46.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.635+0000 7fb814fcf740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x562431252ba0 con 0x56243127f490 2024-03-24T00:45:46.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.635+0000 7fb814fcf740 10 client.0 _ll_register_callbacks cb 0x562431180d70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:45:46.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.635+0000 7fb80b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56243129fc50 0x562431298fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.635+0000 7fb80b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56243129fc50 0x562431298fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37438/0 (socket says 172.21.15.67:37438) 2024-03-24T00:45:46.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.635+0000 7fb80b7fe640 1 -- 172.21.15.67:0/2961982058 learned_addr learned my addr 172.21.15.67:0/2961982058 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:46.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.635+0000 7fb810983640 1 --2- 172.21.15.67:0/2961982058 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x562431293f10 0x56243129acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.635+0000 7fb80bfff640 1 --2- 172.21.15.67:0/2961982058 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56243127f490 0x562431298aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.636+0000 7fb80b7fe640 1 -- 172.21.15.67:0/2961982058 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56243127f490 msgr2=0x562431298aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:46.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.636+0000 7fb80b7fe640 1 --2- 172.21.15.67:0/2961982058 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56243127f490 0x562431298aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.636+0000 7fb80b7fe640 1 -- 172.21.15.67:0/2961982058 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x562431293f10 msgr2=0x56243129acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:46.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.636+0000 7fb80b7fe640 1 --2- 172.21.15.67:0/2961982058 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x562431293f10 0x56243129acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.636+0000 7fb80b7fe640 1 -- 172.21.15.67:0/2961982058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562430ffcf50 con 0x56243129fc50 2024-03-24T00:45:46.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.636+0000 7fb80bfff640 1 --2- 172.21.15.67:0/2961982058 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56243127f490 0x562431298aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:45:46.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.636+0000 7fb810983640 1 --2- 172.21.15.67:0/2961982058 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x562431293f10 0x56243129acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:45:46.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.636+0000 7fb80b7fe640 1 --2- 172.21.15.67:0/2961982058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56243129fc50 0x562431298fe0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7fb800026040 tx=0x7fb800004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59c50b9c47e7d36b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:46.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.636+0000 7fb7eaffd640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:45:46.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.636+0000 7fb7eaffd640 1 -- 172.21.15.67:0/2961982058 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb800006040 con 0x56243129fc50 2024-03-24T00:45:46.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.636+0000 7fb7eaffd640 1 -- 172.21.15.67:0/2961982058 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 6303+0+0 (secure 0 0 0) 0x7fb8000061e0 con 0x56243129fc50 2024-03-24T00:45:46.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.636+0000 7fb814fcf740 1 -- 172.21.15.67:0/2961982058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb800029070 con 0x56243129fc50 2024-03-24T00:45:46.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.636+0000 7fb7eaffd640 1 -- 172.21.15.67:0/2961982058 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb80003e050 con 0x56243129fc50 2024-03-24T00:45:46.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.636+0000 7fb814fcf740 10 client.8013 fetch_fsmap learned FSMap version 61 2024-03-24T00:45:46.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.636+0000 7fb814fcf740 1 -- 172.21.15.67:0/2961982058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=2+}) v3 -- 0x562430e900b0 con 0x56243129fc50 2024-03-24T00:45:46.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.636+0000 7fb7eaffd640 1 -- 172.21.15.67:0/2961982058 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb800029070 con 0x56243129fc50 2024-03-24T00:45:46.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.636+0000 7fb7eaffd640 1 -- 172.21.15.67:0/2961982058 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 61) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fb800041020 con 0x56243129fc50 2024-03-24T00:45:46.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.636+0000 7fb814fcf740 10 client.8013 fetch_fsmap finished waiting for FSMap version 61 2024-03-24T00:45:46.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.636+0000 7fb814fcf740 10 client.8013 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T00:45:46.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.636+0000 7fb814fcf740 20 client.8013 populate_metadata read hostname 'smithi067' 2024-03-24T00:45:46.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.636+0000 7fb814fcf740 1 --2- 172.21.15.67:0/2961982058 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x5624312bca40 0x5624312bee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.636+0000 7fb814fcf740 4 client.8013 mds_command: new command op to 7860 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:45:46.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.636+0000 7fb814fcf740 1 -- 172.21.15.67:0/2961982058 --> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5624311c9730 con 0x5624312bca40 2024-03-24T00:45:46.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.637+0000 7fb80bfff640 1 --2- 172.21.15.67:0/2961982058 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x5624312bca40 0x5624312bee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.637+0000 7fb80bfff640 1 --2- 172.21.15.67:0/2961982058 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x5624312bca40 0x5624312bee30 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=812ad8b24348097d server_cookie=a44353e06f64d614 in_seq=0 out_seq=0 2024-03-24T00:45:46.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.637+0000 7fb7eaffd640 10 client.8013 ms_handle_connect on v2:172.21.15.67:6832/1336395002 2024-03-24T00:45:46.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.637+0000 7fb7eaffd640 1 -- 172.21.15.67:0/2961982058 <== mds.0 v2:172.21.15.67:6832/1336395002 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5624311c9730 con 0x5624312bca40 2024-03-24T00:45:46.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.637+0000 7fb7eaffd640 10 client.8013 handle_command_reply: tid=0 2024-03-24T00:45:46.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.637+0000 7fb814fcf740 1 client.8013 shutdown 2024-03-24T00:45:46.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.638+0000 7fb814fcf740 1 -- 172.21.15.67:0/2961982058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56243129fc50 msgr2=0x562431298fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:46.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.638+0000 7fb814fcf740 1 --2- 172.21.15.67:0/2961982058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56243129fc50 0x562431298fe0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7fb800026040 tx=0x7fb800004730 comp rx=0 tx=0).stop 2024-03-24T00:45:46.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.638+0000 7fb814fcf740 1 -- 172.21.15.67:0/2961982058 shutdown_connections 2024-03-24T00:45:46.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.638+0000 7fb814fcf740 1 --2- 172.21.15.67:0/2961982058 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x5624312bca40 0x5624312bee30 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.638+0000 7fb814fcf740 1 --2- 172.21.15.67:0/2961982058 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x562431293f10 0x56243129acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.638+0000 7fb814fcf740 1 --2- 172.21.15.67:0/2961982058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56243129fc50 0x562431298fe0 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.638+0000 7fb7eaffd640 0 client.8013 ms_handle_reset on v2:172.21.15.67:6832/1336395002 2024-03-24T00:45:46.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.638+0000 7fb814fcf740 1 --2- 172.21.15.67:0/2961982058 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56243127f490 0x562431298aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.638+0000 7fb814fcf740 1 -- 172.21.15.67:0/2961982058 >> 172.21.15.67:0/2961982058 conn(0x5624312576c0 msgr2=0x5624312512b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:46.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.638+0000 7fb814fcf740 1 -- 172.21.15.67:0/2961982058 shutdown_connections 2024-03-24T00:45:46.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.638+0000 7fb814fcf740 1 -- 172.21.15.67:0/2961982058 wait complete. 2024-03-24T00:45:46.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.638+0000 7fb814fcf740 20 client.8013 trim_cache size 0 max 16384 2024-03-24T00:45:46.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.650+0000 7fb814fcf740 1 Processor -- start 2024-03-24T00:45:46.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.650+0000 7fb814fcf740 1 -- start start 2024-03-24T00:45:46.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.651+0000 7fb814fcf740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56243127f490 0x562431298de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.651+0000 7fb814fcf740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x562431293f10 0x562431251a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.651+0000 7fb814fcf740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56243129fc50 0x56243129be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.651+0000 7fb814fcf740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb80c06dc30 con 0x56243129fc50 2024-03-24T00:45:46.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.651+0000 7fb814fcf740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb80c0cb9b0 con 0x562431293f10 2024-03-24T00:45:46.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.651+0000 7fb814fcf740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb80c06d610 con 0x56243127f490 2024-03-24T00:45:46.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.651+0000 7fb810983640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56243129fc50 0x56243129be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.651+0000 7fb810983640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56243129fc50 0x56243129be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37448/0 (socket says 172.21.15.67:37448) 2024-03-24T00:45:46.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.651+0000 7fb810983640 1 -- 172.21.15.67:0/4207648580 learned_addr learned my addr 172.21.15.67:0/4207648580 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:46.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.651+0000 7fb80bfff640 1 --2- 172.21.15.67:0/4207648580 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56243127f490 0x562431298de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.651+0000 7fb80b7fe640 1 --2- 172.21.15.67:0/4207648580 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x562431293f10 0x562431251a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.652+0000 7fb810983640 1 -- 172.21.15.67:0/4207648580 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56243127f490 msgr2=0x562431298de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:46.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.652+0000 7fb810983640 1 --2- 172.21.15.67:0/4207648580 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56243127f490 0x562431298de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.652+0000 7fb810983640 1 -- 172.21.15.67:0/4207648580 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x562431293f10 msgr2=0x562431251a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:46.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.652+0000 7fb810983640 1 --2- 172.21.15.67:0/4207648580 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x562431293f10 0x562431251a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.652+0000 7fb810983640 1 -- 172.21.15.67:0/4207648580 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56243129b040 con 0x56243129fc50 2024-03-24T00:45:46.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.652+0000 7fb810983640 1 --2- 172.21.15.67:0/4207648580 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56243129fc50 0x56243129be20 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7fb804003c60 tx=0x7fb804009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5857d5d67c02efa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:46.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.652+0000 7fb7f17fa640 1 -- 172.21.15.67:0/4207648580 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8040047a0 con 0x56243129fc50 2024-03-24T00:45:46.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.652+0000 7fb7f17fa640 1 -- 172.21.15.67:0/4207648580 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb804015640 con 0x56243129fc50 2024-03-24T00:45:46.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.652+0000 7fb7f17fa640 1 -- 172.21.15.67:0/4207648580 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb804015a10 con 0x56243129fc50 2024-03-24T00:45:46.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.652+0000 7fb814fcf740 1 -- 172.21.15.67:0/4207648580 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56243129fc50 msgr2=0x56243129be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:46.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.652+0000 7fb814fcf740 1 --2- 172.21.15.67:0/4207648580 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56243129fc50 0x56243129be20 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7fb804003c60 tx=0x7fb804009750 comp rx=0 tx=0).stop 2024-03-24T00:45:46.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.652+0000 7fb814fcf740 1 -- 172.21.15.67:0/4207648580 shutdown_connections 2024-03-24T00:45:46.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.652+0000 7fb814fcf740 1 --2- 172.21.15.67:0/4207648580 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56243129fc50 0x56243129be20 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.652+0000 7fb814fcf740 1 --2- 172.21.15.67:0/4207648580 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x562431293f10 0x562431251a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.652+0000 7fb814fcf740 1 --2- 172.21.15.67:0/4207648580 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56243127f490 0x562431298de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.652+0000 7fb814fcf740 1 -- 172.21.15.67:0/4207648580 >> 172.21.15.67:0/4207648580 conn(0x5624312576c0 msgr2=0x56243124da40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:46.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.652+0000 7fb814fcf740 1 -- 172.21.15.67:0/4207648580 shutdown_connections 2024-03-24T00:45:46.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.653+0000 7fb814fcf740 1 -- 172.21.15.67:0/4207648580 wait complete. 2024-03-24T00:45:46.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.653+0000 7fb814fcf740 1 -- start start 2024-03-24T00:45:46.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.654+0000 7fb814fcf740 1 Processor -- start 2024-03-24T00:45:46.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.655+0000 7fb814fcf740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56243127f490 0x562431295920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.655+0000 7fb814fcf740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x562431293f10 0x562431295e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.655+0000 7fb814fcf740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56243129fc50 0x5624312c2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.655+0000 7fb814fcf740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624312957a0 con 0x56243127f490 2024-03-24T00:45:46.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.655+0000 7fb814fcf740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x56243129ba40 con 0x562431293f10 2024-03-24T00:45:46.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.655+0000 7fb814fcf740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x562431298530 con 0x56243129fc50 2024-03-24T00:45:46.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.655+0000 7fb814fcf740 10 client.0 _ll_register_callbacks cb 0x7fb7fc076460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:45:46.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.655+0000 7fb80bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56243127f490 0x562431295920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.655+0000 7fb80bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56243127f490 0x562431295920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37462/0 (socket says 172.21.15.67:37462) 2024-03-24T00:45:46.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.655+0000 7fb80bfff640 1 -- 172.21.15.67:0/1996153633 learned_addr learned my addr 172.21.15.67:0/1996153633 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:46.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.655+0000 7fb810983640 1 --2- 172.21.15.67:0/1996153633 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56243129fc50 0x5624312c2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.655+0000 7fb80b7fe640 1 --2- 172.21.15.67:0/1996153633 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x562431293f10 0x562431295e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.655+0000 7fb80bfff640 1 -- 172.21.15.67:0/1996153633 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56243129fc50 msgr2=0x5624312c2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:46.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.655+0000 7fb80bfff640 1 --2- 172.21.15.67:0/1996153633 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56243129fc50 0x5624312c2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.655+0000 7fb80bfff640 1 -- 172.21.15.67:0/1996153633 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x562431293f10 msgr2=0x562431295e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:46.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.655+0000 7fb80bfff640 1 --2- 172.21.15.67:0/1996153633 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x562431293f10 0x562431295e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.655+0000 7fb80bfff640 1 -- 172.21.15.67:0/1996153633 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5624313102b0 con 0x56243127f490 2024-03-24T00:45:46.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.655+0000 7fb80bfff640 1 --2- 172.21.15.67:0/1996153633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56243127f490 0x562431295920 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc07a0d0 tx=0x7fb7fc076510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7106baa09d73d013 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:46.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.655+0000 7fb7eaffd640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:45:46.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.655+0000 7fb7eaffd640 1 -- 172.21.15.67:0/1996153633 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7fc0767b0 con 0x56243127f490 2024-03-24T00:45:46.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.655+0000 7fb7eaffd640 1 -- 172.21.15.67:0/1996153633 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 6303+0+0 (secure 0 0 0) 0x7fb7fc0823c0 con 0x56243127f490 2024-03-24T00:45:46.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.655+0000 7fb814fcf740 1 -- 172.21.15.67:0/1996153633 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb804015640 con 0x56243127f490 2024-03-24T00:45:46.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.656+0000 7fb7eaffd640 1 -- 172.21.15.67:0/1996153633 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb7fc076950 con 0x56243127f490 2024-03-24T00:45:46.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.656+0000 7fb814fcf740 10 client.8019 fetch_fsmap learned FSMap version 61 2024-03-24T00:45:46.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.656+0000 7fb814fcf740 1 -- 172.21.15.67:0/1996153633 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=2+}) v3 -- 0x5624312c3d00 con 0x56243127f490 2024-03-24T00:45:46.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.656+0000 7fb7eaffd640 1 -- 172.21.15.67:0/1996153633 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb804015640 con 0x56243127f490 2024-03-24T00:45:46.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.656+0000 7fb7eaffd640 1 -- 172.21.15.67:0/1996153633 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 61) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fb7fc048030 con 0x56243127f490 2024-03-24T00:45:46.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.656+0000 7fb814fcf740 10 client.8019 fetch_fsmap finished waiting for FSMap version 61 2024-03-24T00:45:46.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.656+0000 7fb814fcf740 10 client.8019 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T00:45:46.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.656+0000 7fb814fcf740 20 client.8019 populate_metadata read hostname 'smithi067' 2024-03-24T00:45:46.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.656+0000 7fb814fcf740 1 --2- 172.21.15.67:0/1996153633 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x5624312c3390 0x562431292be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.656+0000 7fb814fcf740 4 client.8019 mds_command: new command op to 7860 tid=0 multi_id=0 [{"prefix": "damage ls", "format": "json-pretty"}] 2024-03-24T00:45:46.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.656+0000 7fb814fcf740 1 -- 172.21.15.67:0/1996153633 --> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) v1 -- 0x5624311c3a80 con 0x5624312c3390 2024-03-24T00:45:46.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.656+0000 7fb80b7fe640 1 --2- 172.21.15.67:0/1996153633 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x5624312c3390 0x562431292be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.656+0000 7fb80b7fe640 1 --2- 172.21.15.67:0/1996153633 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x5624312c3390 0x562431292be0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1fa39f19dfb6ad9f server_cookie=9de557ab002a45ef in_seq=0 out_seq=0 2024-03-24T00:45:46.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.656+0000 7fb7eaffd640 10 client.8019 ms_handle_connect on v2:172.21.15.67:6832/1336395002 2024-03-24T00:45:46.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.656+0000 7fb7eaffd640 1 -- 172.21.15.67:0/1996153633 <== mds.0 v2:172.21.15.67:6832/1336395002 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+420 (crc 0 0 0) 0x5624311c3a80 con 0x5624312c3390 2024-03-24T00:45:46.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.656+0000 7fb7eaffd640 10 client.8019 handle_command_reply: tid=0 2024-03-24T00:45:46.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.657+0000 7fb814fcf740 1 client.8019 shutdown 2024-03-24T00:45:46.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.657+0000 7fb814fcf740 1 -- 172.21.15.67:0/1996153633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56243127f490 msgr2=0x562431295920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:46.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.657+0000 7fb814fcf740 1 --2- 172.21.15.67:0/1996153633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56243127f490 0x562431295920 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc07a0d0 tx=0x7fb7fc076510 comp rx=0 tx=0).stop 2024-03-24T00:45:46.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.657+0000 7fb814fcf740 1 -- 172.21.15.67:0/1996153633 shutdown_connections 2024-03-24T00:45:46.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.657+0000 7fb814fcf740 1 --2- 172.21.15.67:0/1996153633 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x5624312c3390 0x562431292be0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.657+0000 7fb814fcf740 1 --2- 172.21.15.67:0/1996153633 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56243129fc50 0x5624312c2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.657+0000 7fb7eaffd640 0 client.8019 ms_handle_reset on v2:172.21.15.67:6832/1336395002 2024-03-24T00:45:46.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.657+0000 7fb814fcf740 1 --2- 172.21.15.67:0/1996153633 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x562431293f10 0x562431295e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.657+0000 7fb814fcf740 1 --2- 172.21.15.67:0/1996153633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56243127f490 0x562431295920 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.657+0000 7fb814fcf740 1 -- 172.21.15.67:0/1996153633 >> 172.21.15.67:0/1996153633 conn(0x5624312576c0 msgr2=0x562431298dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:46.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.657+0000 7fb814fcf740 1 -- 172.21.15.67:0/1996153633 shutdown_connections 2024-03-24T00:45:46.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.657+0000 7fb814fcf740 1 -- 172.21.15.67:0/1996153633 wait complete. 2024-03-24T00:45:46.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.657+0000 7fb814fcf740 20 client.8019 trim_cache size 0 max 16384 2024-03-24T00:45:46.662 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:45:46.662 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-03-24T00:45:46.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:45:46.662 INFO:teuthology.orchestra.run.smithi067.stdout: "damage_type": "backtrace", 2024-03-24T00:45:46.662 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 1644331677, 2024-03-24T00:45:46.663 INFO:teuthology.orchestra.run.smithi067.stdout: "ino": 1099511627777, 2024-03-24T00:45:46.663 INFO:teuthology.orchestra.run.smithi067.stdout: "path": "/testdir/hardlink" 2024-03-24T00:45:46.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:45:46.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:45:46.663 INFO:teuthology.orchestra.run.smithi067.stdout: "damage_type": "backtrace", 2024-03-24T00:45:46.663 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 1928141924, 2024-03-24T00:45:46.663 INFO:teuthology.orchestra.run.smithi067.stdout: "ino": 1099511627776, 2024-03-24T00:45:46.663 INFO:teuthology.orchestra.run.smithi067.stdout: "path": "/dir" 2024-03-24T00:45:46.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:45:46.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:45:46.663 INFO:teuthology.orchestra.run.smithi067.stdout: "damage_type": "dir_frag", 2024-03-24T00:45:46.663 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 2106693715, 2024-03-24T00:45:46.663 INFO:teuthology.orchestra.run.smithi067.stdout: "ino": 1099511627776, 2024-03-24T00:45:46.663 INFO:teuthology.orchestra.run.smithi067.stdout: "frag": "*", 2024-03-24T00:45:46.664 INFO:teuthology.orchestra.run.smithi067.stdout: "path": "/dir" 2024-03-24T00:45:46.664 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:45:46.664 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-03-24T00:45:46.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.657+0000 7fb8123e6640 1 -- 172.21.15.67:0/3045025165 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb7f806bbd0 msgr2=0x7fb7f806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:46.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.657+0000 7fb8123e6640 1 --2- 172.21.15.67:0/3045025165 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb7f806bbd0 0x7fb7f806e090 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fb80c0c95a0 tx=0x7fb800008040 comp rx=0 tx=0).stop 2024-03-24T00:45:46.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.657+0000 7fb8123e6640 1 -- 172.21.15.67:0/3045025165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7f40039b0 msgr2=0x7fb80c14cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:46.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.657+0000 7fb8123e6640 1 --2- 172.21.15.67:0/3045025165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7f40039b0 0x7fb80c14cf20 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc00c240 tx=0x7fb7fc001ab0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.657+0000 7fb8123e6640 1 -- 172.21.15.67:0/3045025165 shutdown_connections 2024-03-24T00:45:46.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.657+0000 7fb8123e6640 1 --2- 172.21.15.67:0/3045025165 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb7f806bbd0 0x7fb7f806e090 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.657+0000 7fb8123e6640 1 --2- 172.21.15.67:0/3045025165 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb80c14fb70 0x7fb80c14fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.657+0000 7fb8123e6640 1 --2- 172.21.15.67:0/3045025165 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb80c14d870 0x7fb80c14dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.657+0000 7fb8123e6640 1 --2- 172.21.15.67:0/3045025165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7f40039b0 0x7fb80c14cf20 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.657+0000 7fb8123e6640 1 -- 172.21.15.67:0/3045025165 >> 172.21.15.67:0/3045025165 conn(0x7fb80c08b1b0 msgr2=0x7fb80c15d580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:46.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.658+0000 7fb8123e6640 1 -- 172.21.15.67:0/3045025165 shutdown_connections 2024-03-24T00:45:46.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.658+0000 7fb8123e6640 1 -- 172.21.15.67:0/3045025165 wait complete. 2024-03-24T00:45:46.668 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-24T00:45:46.724 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:45:46.726 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:45:46.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.769+0000 7fe996d10640 1 Processor -- start 2024-03-24T00:45:46.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.769+0000 7fe996d10640 1 -- start start 2024-03-24T00:45:46.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.770+0000 7fe996d10640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe9900cfe80 0x7fe9900cd670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.770+0000 7fe996d10640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe99006b040 0x7fe99006b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.770+0000 7fe996d10640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe990161d10 con 0x7fe9900b21f0 2024-03-24T00:45:46.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.770+0000 7fe996d10640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9900b25d0 con 0x7fe9900cfe80 2024-03-24T00:45:46.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.770+0000 7fe996d10640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9900d1a50 con 0x7fe99006b040 2024-03-24T00:45:46.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.770+0000 7fe994aac640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe9900b21f0 0x7fe9901624d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47618/0 (socket says 172.21.15.67:47618) 2024-03-24T00:45:46.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.770+0000 7fe994aac640 1 -- 172.21.15.67:0/3286324106 learned_addr learned my addr 172.21.15.67:0/3286324106 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:45:46.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.771+0000 7fe9952ad640 1 --2- 172.21.15.67:0/3286324106 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe99006b040 0x7fe99006b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.771+0000 7fe98f7fe640 1 -- 172.21.15.67:0/3286324106 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1552966944 0 0) 0x7fe990161d10 con 0x7fe9900b21f0 2024-03-24T00:45:46.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.771+0000 7fe98ffff640 1 --2- 172.21.15.67:0/3286324106 >> v2:172.21.15.72:3300/0 conn(0x7fe9900cfe80 0x7fe9900cd670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.771+0000 7fe98f7fe640 1 -- 172.21.15.67:0/3286324106 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe980003520 con 0x7fe9900b21f0 2024-03-24T00:45:46.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.771+0000 7fe98f7fe640 1 -- 172.21.15.67:0/3286324106 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1866424102 0 0) 0x7fe980003520 con 0x7fe9900b21f0 2024-03-24T00:45:46.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.771+0000 7fe98f7fe640 1 -- 172.21.15.67:0/3286324106 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe99006b040 msgr2=0x7fe99006b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:46.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.771+0000 7fe98f7fe640 1 --2- 172.21.15.67:0/3286324106 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe99006b040 0x7fe99006b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.771+0000 7fe98f7fe640 1 -- 172.21.15.67:0/3286324106 >> v2:172.21.15.72:3300/0 conn(0x7fe9900cfe80 msgr2=0x7fe9900cd670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:46.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.771+0000 7fe98f7fe640 1 --2- 172.21.15.67:0/3286324106 >> v2:172.21.15.72:3300/0 conn(0x7fe9900cfe80 0x7fe9900cd670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.771+0000 7fe98f7fe640 1 -- 172.21.15.67:0/3286324106 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9900d0480 con 0x7fe9900b21f0 2024-03-24T00:45:46.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.771+0000 7fe98f7fe640 1 -- 172.21.15.67:0/3286324106 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7fe984001670 con 0x7fe9900b21f0 2024-03-24T00:45:46.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.771+0000 7fe98f7fe640 1 -- 172.21.15.67:0/3286324106 >> v1:172.21.15.67:6789/0 conn(0x7fe9900b21f0 legacy=0x7fe9901624d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:46.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.771+0000 7fe98f7fe640 1 --2- 172.21.15.67:0/3286324106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9800039b0 0x7fe980003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.771+0000 7fe98f7fe640 1 --2- 172.21.15.67:0/3286324106 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe980004800 0x7fe980004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.771+0000 7fe98f7fe640 1 --2- 172.21.15.67:0/3286324106 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe99006b040 0x7fe9800051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.771+0000 7fe98f7fe640 1 -- 172.21.15.67:0/3286324106 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9900b25d0 con 0x7fe9800039b0 2024-03-24T00:45:46.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.771+0000 7fe98f7fe640 1 -- 172.21.15.67:0/3286324106 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9900d1a50 con 0x7fe980004800 2024-03-24T00:45:46.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.771+0000 7fe98f7fe640 1 -- 172.21.15.67:0/3286324106 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe98000b9f0 con 0x7fe99006b040 2024-03-24T00:45:46.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.771+0000 7fe994aac640 1 --2- 172.21.15.67:0/3286324106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9800039b0 0x7fe980003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.771+0000 7fe98ffff640 1 --2- 172.21.15.67:0/3286324106 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe980004800 0x7fe980004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.771+0000 7fe9952ad640 1 --2- 172.21.15.67:0/3286324106 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe99006b040 0x7fe9800051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.772+0000 7fe994aac640 1 -- 172.21.15.67:0/3286324106 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe99006b040 msgr2=0x7fe9800051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:46.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.772+0000 7fe994aac640 1 --2- 172.21.15.67:0/3286324106 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe99006b040 0x7fe9800051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.772+0000 7fe994aac640 1 -- 172.21.15.67:0/3286324106 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe980004800 msgr2=0x7fe980004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:46.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.772+0000 7fe994aac640 1 --2- 172.21.15.67:0/3286324106 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe980004800 0x7fe980004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.772+0000 7fe994aac640 1 -- 172.21.15.67:0/3286324106 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe980003520 con 0x7fe9800039b0 2024-03-24T00:45:46.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.772+0000 7fe994aac640 1 --2- 172.21.15.67:0/3286324106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9800039b0 0x7fe980003dd0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7fe984001050 tx=0x7fe980005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd36d57a811ad84f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:46.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.772+0000 7fe98f7fe640 1 -- 172.21.15.67:0/3286324106 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe984007d70 con 0x7fe9800039b0 2024-03-24T00:45:46.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.772+0000 7fe98f7fe640 1 -- 172.21.15.67:0/3286324106 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe984030070 con 0x7fe9800039b0 2024-03-24T00:45:46.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.772+0000 7fe98f7fe640 1 -- 172.21.15.67:0/3286324106 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe98402cd70 con 0x7fe9800039b0 2024-03-24T00:45:46.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.773+0000 7fe996d10640 1 -- 172.21.15.67:0/3286324106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9800039b0 msgr2=0x7fe980003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:46.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.773+0000 7fe996d10640 1 --2- 172.21.15.67:0/3286324106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9800039b0 0x7fe980003dd0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7fe984001050 tx=0x7fe980005830 comp rx=0 tx=0).stop 2024-03-24T00:45:46.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.773+0000 7fe9952ad640 1 -- 172.21.15.67:0/3286324106 reap_dead start 2024-03-24T00:45:46.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.773+0000 7fe996d10640 1 -- 172.21.15.67:0/3286324106 shutdown_connections 2024-03-24T00:45:46.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.773+0000 7fe996d10640 1 -- 172.21.15.67:0/3286324106 >> 172.21.15.67:0/3286324106 conn(0x7fe990099a30 msgr2=0x7fe990099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:46.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.773+0000 7fe996d10640 1 -- 172.21.15.67:0/3286324106 shutdown_connections 2024-03-24T00:45:46.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.773+0000 7fe996d10640 1 -- 172.21.15.67:0/3286324106 wait complete. 2024-03-24T00:45:46.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.775+0000 7fe996d10640 1 Processor -- start 2024-03-24T00:45:46.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.775+0000 7fe996d10640 1 -- start start 2024-03-24T00:45:46.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.776+0000 7fe996d10640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9800039b0 0x7fe9900c4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.776+0000 7fe996d10640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe99015aee0 0x7fe9900c51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.776+0000 7fe996d10640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe9900c6c00 0x7fe9900c7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.776+0000 7fe996d10640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9900ce1a0 con 0x7fe9800039b0 2024-03-24T00:45:46.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.776+0000 7fe996d10640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe990161fb0 con 0x7fe99015aee0 2024-03-24T00:45:46.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.776+0000 7fe996d10640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe99006b4d0 con 0x7fe9900c6c00 2024-03-24T00:45:46.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.777+0000 7fe994aac640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9800039b0 0x7fe9900c4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.777+0000 7fe994aac640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9800039b0 0x7fe9900c4790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37474/0 (socket says 172.21.15.67:37474) 2024-03-24T00:45:46.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.777+0000 7fe994aac640 1 -- 172.21.15.67:0/3998076348 learned_addr learned my addr 172.21.15.67:0/3998076348 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:46.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.777+0000 7fe98ffff640 1 --2- 172.21.15.67:0/3998076348 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe99015aee0 0x7fe9900c51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.777+0000 7fe9952ad640 1 --2- 172.21.15.67:0/3998076348 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe9900c6c00 0x7fe9900c7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.777+0000 7fe994aac640 1 -- 172.21.15.67:0/3998076348 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe9900c6c00 msgr2=0x7fe9900c7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:46.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.777+0000 7fe994aac640 1 --2- 172.21.15.67:0/3998076348 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe9900c6c00 0x7fe9900c7080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.777+0000 7fe994aac640 1 -- 172.21.15.67:0/3998076348 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe99015aee0 msgr2=0x7fe9900c51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:46.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.777+0000 7fe994aac640 1 --2- 172.21.15.67:0/3998076348 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe99015aee0 0x7fe9900c51d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:46.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.777+0000 7fe994aac640 1 -- 172.21.15.67:0/3998076348 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe99015b360 con 0x7fe9800039b0 2024-03-24T00:45:46.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.777+0000 7fe994aac640 1 --2- 172.21.15.67:0/3998076348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9800039b0 0x7fe9900c4790 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fe98402c5e0 tx=0x7fe984000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:46.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.777+0000 7fe98d7fa640 1 -- 172.21.15.67:0/3998076348 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe984045070 con 0x7fe9800039b0 2024-03-24T00:45:46.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.777+0000 7fe98d7fa640 1 -- 172.21.15.67:0/3998076348 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe984004bf0 con 0x7fe9800039b0 2024-03-24T00:45:46.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.777+0000 7fe996d10640 1 -- 172.21.15.67:0/3998076348 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe99015bcf0 con 0x7fe9800039b0 2024-03-24T00:45:46.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.777+0000 7fe98d7fa640 1 -- 172.21.15.67:0/3998076348 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe984007d50 con 0x7fe9800039b0 2024-03-24T00:45:46.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.777+0000 7fe996d10640 1 -- 172.21.15.67:0/3998076348 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe99015c2b0 con 0x7fe9800039b0 2024-03-24T00:45:46.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.778+0000 7fe996d10640 1 -- 172.21.15.67:0/3998076348 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe99004f400 con 0x7fe9800039b0 2024-03-24T00:45:46.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.779+0000 7fe98d7fa640 1 -- 172.21.15.67:0/3998076348 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe984007070 con 0x7fe9800039b0 2024-03-24T00:45:46.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.779+0000 7fe98d7fa640 1 --2- 172.21.15.67:0/3998076348 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe97406bbd0 0x7fe97406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:46.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.779+0000 7fe98d7fa640 1 -- 172.21.15.67:0/3998076348 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 6303+0+0 (secure 0 0 0) 0x7fe9840abe10 con 0x7fe9800039b0 2024-03-24T00:45:46.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.779+0000 7fe98ffff640 1 --2- 172.21.15.67:0/3998076348 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe97406bbd0 0x7fe97406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:46.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.780+0000 7fe98ffff640 1 --2- 172.21.15.67:0/3998076348 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe97406bbd0 0x7fe97406e090 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fe9901f7290 tx=0x7fe97c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:46.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:46.783+0000 7fe98d7fa640 1 -- 172.21.15.67:0/3998076348 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe98407a140 con 0x7fe9800039b0 2024-03-24T00:45:47.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.003+0000 7fe996d10640 1 -- 172.21.15.67:0/3998076348 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe990072f90 con 0x7fe9800039b0 2024-03-24T00:45:47.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.003+0000 7fe98d7fa640 1 -- 172.21.15.67:0/3998076348 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+164 (secure 0 0 0) 0x7fe98407e1e0 con 0x7fe9800039b0 2024-03-24T00:45:47.003 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:45:47.003 INFO:teuthology.orchestra.run.smithi067.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2024-03-24T00:45:47.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.006+0000 7fe996d10640 1 -- 172.21.15.67:0/3998076348 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe97406bbd0 msgr2=0x7fe97406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:47.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.006+0000 7fe996d10640 1 --2- 172.21.15.67:0/3998076348 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe97406bbd0 0x7fe97406e090 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fe9901f7290 tx=0x7fe97c008040 comp rx=0 tx=0).stop 2024-03-24T00:45:47.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.006+0000 7fe996d10640 1 -- 172.21.15.67:0/3998076348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9800039b0 msgr2=0x7fe9900c4790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:47.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.006+0000 7fe996d10640 1 --2- 172.21.15.67:0/3998076348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9800039b0 0x7fe9900c4790 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fe98402c5e0 tx=0x7fe984000fe0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.006+0000 7fe996d10640 1 -- 172.21.15.67:0/3998076348 shutdown_connections 2024-03-24T00:45:47.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.006+0000 7fe996d10640 1 --2- 172.21.15.67:0/3998076348 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe97406bbd0 0x7fe97406e090 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.006+0000 7fe996d10640 1 --2- 172.21.15.67:0/3998076348 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe9900c6c00 0x7fe9900c7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.006+0000 7fe996d10640 1 --2- 172.21.15.67:0/3998076348 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe99015aee0 0x7fe9900c51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.006+0000 7fe996d10640 1 --2- 172.21.15.67:0/3998076348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9800039b0 0x7fe9900c4790 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.006+0000 7fe996d10640 1 -- 172.21.15.67:0/3998076348 >> 172.21.15.67:0/3998076348 conn(0x7fe990099a30 msgr2=0x7fe9900cd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:47.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.006+0000 7fe996d10640 1 -- 172.21.15.67:0/3998076348 shutdown_connections 2024-03-24T00:45:47.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.006+0000 7fe996d10640 1 -- 172.21.15.67:0/3998076348 wait complete. 2024-03-24T00:45:47.017 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-03-24T00:45:47.018 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:45:47.018 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.6:0 scrub start /dir recursive,repair 2024-03-24T00:45:47.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.123+0000 7f5514081640 1 Processor -- start 2024-03-24T00:45:47.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.123+0000 7f5514081640 1 -- start start 2024-03-24T00:45:47.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.124+0000 7f5514081640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f550c0ced60 0x7f550c0d0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:47.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.124+0000 7f5514081640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f550c0cf1f0 con 0x7f550c0d0b80 2024-03-24T00:45:47.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.124+0000 7f5514081640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f550c0d2020 con 0x7f550c15dfc0 2024-03-24T00:45:47.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.124+0000 7f5514081640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f550c15e3a0 con 0x7f550c0ced60 2024-03-24T00:45:47.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.124+0000 7f551261e640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f550c0d0b80 0x7f550c0d2b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47632/0 (socket says 172.21.15.67:47632) 2024-03-24T00:45:47.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.124+0000 7f551261e640 1 -- 172.21.15.67:0/3708269168 learned_addr learned my addr 172.21.15.67:0/3708269168 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:45:47.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.124+0000 7f551161c640 1 --2- 172.21.15.67:0/3708269168 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f550c0ced60 0x7f550c0d0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:47.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.125+0000 7f5510e1b640 1 -- 172.21.15.67:0/3708269168 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3046282986 0 0) 0x7f550c0cf1f0 con 0x7f550c0d0b80 2024-03-24T00:45:47.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.125+0000 7f5510e1b640 1 -- 172.21.15.67:0/3708269168 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54f8003520 con 0x7f550c0d0b80 2024-03-24T00:45:47.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.125+0000 7f551161c640 1 -- 172.21.15.67:0/3708269168 >> v1:172.21.15.72:6789/0 conn(0x7f550c15dfc0 legacy=0x7f550c1624c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:47.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.125+0000 7f551161c640 1 -- 172.21.15.67:0/3708269168 >> v1:172.21.15.67:6789/0 conn(0x7f550c0d0b80 legacy=0x7f550c0d2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:47.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.125+0000 7f5510e1b640 1 -- 172.21.15.67:0/3708269168 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2743196194 0 0) 0x7f550c0d2020 con 0x7f550c15dfc0 2024-03-24T00:45:47.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.125+0000 7f551161c640 1 -- 172.21.15.67:0/3708269168 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f550c0d2300 con 0x7f550c0ced60 2024-03-24T00:45:47.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.126+0000 7f551161c640 1 --2- 172.21.15.67:0/3708269168 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f550c0ced60 0x7f550c0d0640 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f54fc0097d0 tx=0x7f54fc02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7f1d4dc2bd7cb3c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:47.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.126+0000 7f5510e1b640 1 -- 172.21.15.67:0/3708269168 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54fc035020 con 0x7f550c0ced60 2024-03-24T00:45:47.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.126+0000 7f5510e1b640 1 -- 172.21.15.67:0/3708269168 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54fc031890 con 0x7f550c0ced60 2024-03-24T00:45:47.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.126+0000 7f5510e1b640 1 -- 172.21.15.67:0/3708269168 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54fc031c60 con 0x7f550c0ced60 2024-03-24T00:45:47.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.127+0000 7f5514081640 1 -- 172.21.15.67:0/3708269168 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f550c0ced60 msgr2=0x7f550c0d0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:47.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.127+0000 7f5514081640 1 --2- 172.21.15.67:0/3708269168 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f550c0ced60 0x7f550c0d0640 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f54fc0097d0 tx=0x7f54fc02b930 comp rx=0 tx=0).stop 2024-03-24T00:45:47.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.127+0000 7f5514081640 1 -- 172.21.15.67:0/3708269168 shutdown_connections 2024-03-24T00:45:47.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.127+0000 7f5514081640 1 --2- 172.21.15.67:0/3708269168 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f550c0ced60 0x7f550c0d0640 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.127+0000 7f5514081640 1 -- 172.21.15.67:0/3708269168 >> 172.21.15.67:0/3708269168 conn(0x7f550c06c880 msgr2=0x7f550c06cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:47.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.127+0000 7f5514081640 1 -- 172.21.15.67:0/3708269168 shutdown_connections 2024-03-24T00:45:47.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.127+0000 7f5514081640 1 -- 172.21.15.67:0/3708269168 wait complete. 2024-03-24T00:45:47.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.129+0000 7f5514081640 1 Processor -- start 2024-03-24T00:45:47.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.129+0000 7f5514081640 1 -- start start 2024-03-24T00:45:47.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.130+0000 7f5514081640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f550c0d0b80 0x7f550c15c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:47.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.130+0000 7f5514081640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f550c15dfc0 0x7f550c0cdce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:47.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.130+0000 7f5514081640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f550c15fa00 0x7f550c1e7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:47.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.130+0000 7f5514081640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f550c0b31f0 con 0x7f550c0d0b80 2024-03-24T00:45:47.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.130+0000 7f5514081640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f550c0cf4d0 con 0x7f550c15dfc0 2024-03-24T00:45:47.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.130+0000 7f5514081640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f550c0d0fd0 con 0x7f550c15fa00 2024-03-24T00:45:47.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.130+0000 7f5511e1d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f550c0d0b80 0x7f550c15c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:47.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.130+0000 7f5511e1d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f550c0d0b80 0x7f550c15c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37490/0 (socket says 172.21.15.67:37490) 2024-03-24T00:45:47.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.130+0000 7f5511e1d640 1 -- 172.21.15.67:0/323300989 learned_addr learned my addr 172.21.15.67:0/323300989 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:47.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.130+0000 7f551261e640 1 --2- 172.21.15.67:0/323300989 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f550c15fa00 0x7f550c1e7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:47.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.130+0000 7f551161c640 1 --2- 172.21.15.67:0/323300989 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f550c15dfc0 0x7f550c0cdce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:47.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.130+0000 7f5511e1d640 1 -- 172.21.15.67:0/323300989 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f550c15fa00 msgr2=0x7f550c1e7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:47.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.130+0000 7f5511e1d640 1 --2- 172.21.15.67:0/323300989 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f550c15fa00 0x7f550c1e7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.130+0000 7f5511e1d640 1 -- 172.21.15.67:0/323300989 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f550c15dfc0 msgr2=0x7f550c0cdce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:47.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.130+0000 7f5511e1d640 1 --2- 172.21.15.67:0/323300989 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f550c15dfc0 0x7f550c0cdce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.130+0000 7f5511e1d640 1 -- 172.21.15.67:0/323300989 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f550c15fec0 con 0x7f550c0d0b80 2024-03-24T00:45:47.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.130+0000 7f551161c640 1 --2- 172.21.15.67:0/323300989 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f550c15dfc0 0x7f550c0cdce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:45:47.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.131+0000 7f5511e1d640 1 --2- 172.21.15.67:0/323300989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f550c0d0b80 0x7f550c15c540 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f550800c3c0 tx=0x7f550800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:47.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.131+0000 7f55027fc640 1 -- 172.21.15.67:0/323300989 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5508017070 con 0x7f550c0d0b80 2024-03-24T00:45:47.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.131+0000 7f5514081640 1 -- 172.21.15.67:0/323300989 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f550c1601a0 con 0x7f550c0d0b80 2024-03-24T00:45:47.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.131+0000 7f55027fc640 1 -- 172.21.15.67:0/323300989 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55080035f0 con 0x7f550c0d0b80 2024-03-24T00:45:47.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.131+0000 7f55027fc640 1 -- 172.21.15.67:0/323300989 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55080039e0 con 0x7f550c0d0b80 2024-03-24T00:45:47.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.131+0000 7f5514081640 1 -- 172.21.15.67:0/323300989 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f550c160a40 con 0x7f550c0d0b80 2024-03-24T00:45:47.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.133+0000 7f55027fc640 1 -- 172.21.15.67:0/323300989 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f550801a070 con 0x7f550c0d0b80 2024-03-24T00:45:47.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.133+0000 7f55027fc640 1 --2- 172.21.15.67:0/323300989 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f54f006bbd0 0x7f54f006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:47.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.133+0000 7f55027fc640 1 -- 172.21.15.67:0/323300989 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 6303+0+0 (secure 0 0 0) 0x7f5508089f60 con 0x7f550c0d0b80 2024-03-24T00:45:47.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.133+0000 7f551161c640 1 --2- 172.21.15.67:0/323300989 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f54f006bbd0 0x7f54f006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:47.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.134+0000 7f5516c6b740 1 Processor -- start 2024-03-24T00:45:47.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.134+0000 7f551161c640 1 --2- 172.21.15.67:0/323300989 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f54f006bbd0 0x7f54f006e090 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f54fc036000 tx=0x7f54fc008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:47.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.134+0000 7f5516c6b740 1 -- start start 2024-03-24T00:45:47.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.135+0000 7f5516c6b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f9ed98e60 0x560f9edbf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:47.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.135+0000 7f5516c6b740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560f9edbfed0 0x560f9ed97ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:47.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.135+0000 7f5516c6b740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x560f9ed99240 0x560f9ed983e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:47.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.135+0000 7f5516c6b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f9ecd7c90 con 0x560f9ed98e60 2024-03-24T00:45:47.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.135+0000 7f5516c6b740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f9ecfd070 con 0x560f9edbfed0 2024-03-24T00:45:47.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.135+0000 7f5516c6b740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x560f9eca5c80 con 0x560f9ed99240 2024-03-24T00:45:47.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.135+0000 7f5511e1d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f9ed98e60 0x560f9edbf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:47.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.135+0000 7f5511e1d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f9ed98e60 0x560f9edbf990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37492/0 (socket says 172.21.15.67:37492) 2024-03-24T00:45:47.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.135+0000 7f5511e1d640 1 -- 172.21.15.67:0/3654908722 learned_addr learned my addr 172.21.15.67:0/3654908722 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:47.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.135+0000 7f551261e640 1 --2- 172.21.15.67:0/3654908722 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x560f9ed99240 0x560f9ed983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:47.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.135+0000 7f551161c640 1 --2- 172.21.15.67:0/3654908722 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560f9edbfed0 0x560f9ed97ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:47.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.135+0000 7f5511e1d640 1 -- 172.21.15.67:0/3654908722 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x560f9ed99240 msgr2=0x560f9ed983e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:47.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.135+0000 7f5511e1d640 1 --2- 172.21.15.67:0/3654908722 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x560f9ed99240 0x560f9ed983e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.135+0000 7f5511e1d640 1 -- 172.21.15.67:0/3654908722 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560f9edbfed0 msgr2=0x560f9ed97ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:47.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.135+0000 7f5511e1d640 1 --2- 172.21.15.67:0/3654908722 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560f9edbfed0 0x560f9ed97ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.135+0000 7f5511e1d640 1 -- 172.21.15.67:0/3654908722 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560f9ec9dd10 con 0x560f9ed98e60 2024-03-24T00:45:47.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.135+0000 7f551161c640 1 --2- 172.21.15.67:0/3654908722 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560f9edbfed0 0x560f9ed97ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:45:47.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.135+0000 7f5511e1d640 1 --2- 172.21.15.67:0/3654908722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f9ed98e60 0x560f9edbf990 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f55080602f0 tx=0x7f550805c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8bd123206fa9952 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:47.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.135+0000 7f54ef7fe640 1 -- 172.21.15.67:0/3654908722 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55080564c0 con 0x560f9ed98e60 2024-03-24T00:45:47.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.136+0000 7f54ef7fe640 1 -- 172.21.15.67:0/3654908722 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f550805e050 con 0x560f9ed98e60 2024-03-24T00:45:47.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.136+0000 7f54ef7fe640 1 -- 172.21.15.67:0/3654908722 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5508003b80 con 0x560f9ed98e60 2024-03-24T00:45:47.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.136+0000 7f5516c6b740 1 -- 172.21.15.67:0/3654908722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f9ed98e60 msgr2=0x560f9edbf990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:47.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.136+0000 7f5516c6b740 1 --2- 172.21.15.67:0/3654908722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f9ed98e60 0x560f9edbf990 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f55080602f0 tx=0x7f550805c260 comp rx=0 tx=0).stop 2024-03-24T00:45:47.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.136+0000 7f5516c6b740 1 -- 172.21.15.67:0/3654908722 shutdown_connections 2024-03-24T00:45:47.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.136+0000 7f5516c6b740 1 --2- 172.21.15.67:0/3654908722 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x560f9ed99240 0x560f9ed983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.136+0000 7f5516c6b740 1 --2- 172.21.15.67:0/3654908722 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560f9edbfed0 0x560f9ed97ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.136+0000 7f5516c6b740 1 --2- 172.21.15.67:0/3654908722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f9ed98e60 0x560f9edbf990 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.136+0000 7f5516c6b740 1 -- 172.21.15.67:0/3654908722 >> 172.21.15.67:0/3654908722 conn(0x560f9ed601a0 msgr2=0x560f9edddb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:47.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.136+0000 7f5516c6b740 1 -- 172.21.15.67:0/3654908722 shutdown_connections 2024-03-24T00:45:47.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.136+0000 7f5516c6b740 1 -- 172.21.15.67:0/3654908722 wait complete. 2024-03-24T00:45:47.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.137+0000 7f5516c6b740 1 -- start start 2024-03-24T00:45:47.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.137+0000 7f5516c6b740 1 Processor -- start 2024-03-24T00:45:47.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.138+0000 7f5516c6b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f9ed98e60 0x560f9edabf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:47.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.138+0000 7f5516c6b740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560f9ed99240 0x560f9edda0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:47.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.138+0000 7f5516c6b740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x560f9edda600 0x560f9eddbd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:47.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.138+0000 7f5516c6b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f9ecb90d0 con 0x560f9ed98e60 2024-03-24T00:45:47.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.138+0000 7f5516c6b740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f9ecf9110 con 0x560f9ed99240 2024-03-24T00:45:47.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.138+0000 7f5516c6b740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x560f9edbf500 con 0x560f9edda600 2024-03-24T00:45:47.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.138+0000 7f5516c6b740 10 client.0 _ll_register_callbacks cb 0x560f9ecf4e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:45:47.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.138+0000 7f5511e1d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f9ed98e60 0x560f9edabf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:47.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.138+0000 7f5511e1d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f9ed98e60 0x560f9edabf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37500/0 (socket says 172.21.15.67:37500) 2024-03-24T00:45:47.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.138+0000 7f5511e1d640 1 -- 172.21.15.67:0/1936541957 learned_addr learned my addr 172.21.15.67:0/1936541957 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:47.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.138+0000 7f551161c640 1 --2- 172.21.15.67:0/1936541957 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560f9ed99240 0x560f9edda0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:47.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.138+0000 7f551261e640 1 --2- 172.21.15.67:0/1936541957 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x560f9edda600 0x560f9eddbd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:47.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.138+0000 7f5511e1d640 1 -- 172.21.15.67:0/1936541957 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x560f9edda600 msgr2=0x560f9eddbd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:47.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.138+0000 7f5511e1d640 1 --2- 172.21.15.67:0/1936541957 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x560f9edda600 0x560f9eddbd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.138+0000 7f5511e1d640 1 -- 172.21.15.67:0/1936541957 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560f9ed99240 msgr2=0x560f9edda0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:47.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.138+0000 7f5511e1d640 1 --2- 172.21.15.67:0/1936541957 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560f9ed99240 0x560f9edda0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.138+0000 7f5511e1d640 1 -- 172.21.15.67:0/1936541957 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560f9ead97e0 con 0x560f9ed98e60 2024-03-24T00:45:47.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.138+0000 7f551261e640 1 --2- 172.21.15.67:0/1936541957 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x560f9edda600 0x560f9eddbd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:45:47.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.138+0000 7f5511e1d640 1 --2- 172.21.15.67:0/1936541957 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f9ed98e60 0x560f9edabf00 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f5508007fd0 tx=0x7f550801e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=477c5100c763fb91 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:47.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.138+0000 7f54ecff9640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:45:47.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.138+0000 7f54ecff9640 1 -- 172.21.15.67:0/1936541957 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5508003d80 con 0x560f9ed98e60 2024-03-24T00:45:47.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.138+0000 7f54ecff9640 1 -- 172.21.15.67:0/1936541957 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 6303+0+0 (secure 0 0 0) 0x7f5508057610 con 0x560f9ed98e60 2024-03-24T00:45:47.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.139+0000 7f5516c6b740 1 -- 172.21.15.67:0/1936541957 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f550805e050 con 0x560f9ed98e60 2024-03-24T00:45:47.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.139+0000 7f54ecff9640 1 -- 172.21.15.67:0/1936541957 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5508057810 con 0x560f9ed98e60 2024-03-24T00:45:47.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.139+0000 7f5516c6b740 10 client.8037 fetch_fsmap learned FSMap version 61 2024-03-24T00:45:47.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.139+0000 7f5516c6b740 1 -- 172.21.15.67:0/1936541957 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=2+}) v3 -- 0x560f9eb39f70 con 0x560f9ed98e60 2024-03-24T00:45:47.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.139+0000 7f54ecff9640 1 -- 172.21.15.67:0/1936541957 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f550805e050 con 0x560f9ed98e60 2024-03-24T00:45:47.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.139+0000 7f54ecff9640 1 -- 172.21.15.67:0/1936541957 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 61) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f550805e200 con 0x560f9ed98e60 2024-03-24T00:45:47.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.139+0000 7f5516c6b740 10 client.8037 fetch_fsmap finished waiting for FSMap version 61 2024-03-24T00:45:47.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.139+0000 7f5516c6b740 10 client.8037 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-03-24T00:45:47.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.139+0000 7f5516c6b740 20 client.8037 populate_metadata read hostname 'smithi067' 2024-03-24T00:45:47.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.139+0000 7f5516c6b740 1 --2- 172.21.15.67:0/1936541957 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x560f9edc6bf0 0x560f9edc3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:47.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.139+0000 7f5516c6b740 4 client.8037 mds_command: new command op to 7860 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:45:47.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.139+0000 7f5516c6b740 1 -- 172.21.15.67:0/1936541957 --> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560f9ed0a5b0 con 0x560f9edc6bf0 2024-03-24T00:45:47.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.139+0000 7f551161c640 1 --2- 172.21.15.67:0/1936541957 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x560f9edc6bf0 0x560f9edc3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:47.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.139+0000 7f54ecff9640 10 client.8037 ms_handle_connect on v2:172.21.15.67:6832/1336395002 2024-03-24T00:45:47.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.139+0000 7f551161c640 1 --2- 172.21.15.67:0/1936541957 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x560f9edc6bf0 0x560f9edc3cd0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43bd87c591449bac server_cookie=835ca5c5619e0c3b in_seq=0 out_seq=0 2024-03-24T00:45:47.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.140+0000 7f54ecff9640 1 -- 172.21.15.67:0/1936541957 <== mds.0 v2:172.21.15.67:6832/1336395002 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560f9ed0a5b0 con 0x560f9edc6bf0 2024-03-24T00:45:47.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.140+0000 7f54ecff9640 10 client.8037 handle_command_reply: tid=0 2024-03-24T00:45:47.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.140+0000 7f5516c6b740 1 client.8037 shutdown 2024-03-24T00:45:47.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.140+0000 7f5516c6b740 1 -- 172.21.15.67:0/1936541957 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f9ed98e60 msgr2=0x560f9edabf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:47.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.140+0000 7f5516c6b740 1 --2- 172.21.15.67:0/1936541957 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f9ed98e60 0x560f9edabf00 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f5508007fd0 tx=0x7f550801e040 comp rx=0 tx=0).stop 2024-03-24T00:45:47.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.140+0000 7f5516c6b740 1 -- 172.21.15.67:0/1936541957 shutdown_connections 2024-03-24T00:45:47.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.140+0000 7f5516c6b740 1 --2- 172.21.15.67:0/1936541957 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x560f9edc6bf0 0x560f9edc3cd0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.140+0000 7f5516c6b740 1 --2- 172.21.15.67:0/1936541957 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x560f9edda600 0x560f9eddbd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.140+0000 7f5516c6b740 1 --2- 172.21.15.67:0/1936541957 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560f9ed99240 0x560f9edda0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.140+0000 7f54ecff9640 0 client.8037 ms_handle_reset on v2:172.21.15.67:6832/1336395002 2024-03-24T00:45:47.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.140+0000 7f5516c6b740 1 --2- 172.21.15.67:0/1936541957 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f9ed98e60 0x560f9edabf00 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.140+0000 7f5516c6b740 1 -- 172.21.15.67:0/1936541957 >> 172.21.15.67:0/1936541957 conn(0x560f9ed601a0 msgr2=0x560f9edc3270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:47.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.140+0000 7f5516c6b740 1 -- 172.21.15.67:0/1936541957 shutdown_connections 2024-03-24T00:45:47.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.140+0000 7f5516c6b740 1 -- 172.21.15.67:0/1936541957 wait complete. 2024-03-24T00:45:47.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.140+0000 7f5516c6b740 20 client.8037 trim_cache size 0 max 16384 2024-03-24T00:45:47.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.159+0000 7f5516c6b740 1 Processor -- start 2024-03-24T00:45:47.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.159+0000 7f5516c6b740 1 -- start start 2024-03-24T00:45:47.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.160+0000 7f5516c6b740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x560f9ed98e60 0x560f9edabe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:47.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.160+0000 7f5516c6b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f9ed99240 0x560f9eddaed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:47.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.160+0000 7f5516c6b740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560f9edda600 0x560f9ed91ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:47.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.160+0000 7f5516c6b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f550c200820 con 0x560f9ed99240 2024-03-24T00:45:47.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.160+0000 7f5516c6b740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f550c1ff9e0 con 0x560f9edda600 2024-03-24T00:45:47.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.160+0000 7f5516c6b740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f550c15f270 con 0x560f9ed98e60 2024-03-24T00:45:47.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.161+0000 7f551161c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f9ed99240 0x560f9eddaed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:47.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.161+0000 7f551161c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f9ed99240 0x560f9eddaed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37506/0 (socket says 172.21.15.67:37506) 2024-03-24T00:45:47.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.161+0000 7f551161c640 1 -- 172.21.15.67:0/3880786347 learned_addr learned my addr 172.21.15.67:0/3880786347 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:47.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.161+0000 7f5511e1d640 1 --2- 172.21.15.67:0/3880786347 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x560f9ed98e60 0x560f9edabe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:47.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.161+0000 7f551261e640 1 --2- 172.21.15.67:0/3880786347 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560f9edda600 0x560f9ed91ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:47.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.161+0000 7f551161c640 1 -- 172.21.15.67:0/3880786347 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x560f9ed98e60 msgr2=0x560f9edabe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:47.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.161+0000 7f551161c640 1 --2- 172.21.15.67:0/3880786347 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x560f9ed98e60 0x560f9edabe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.161+0000 7f551161c640 1 -- 172.21.15.67:0/3880786347 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560f9edda600 msgr2=0x560f9ed91ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:47.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.161+0000 7f551161c640 1 --2- 172.21.15.67:0/3880786347 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560f9edda600 0x560f9ed91ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.161+0000 7f551161c640 1 -- 172.21.15.67:0/3880786347 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560f9e9d1170 con 0x560f9ed99240 2024-03-24T00:45:47.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.161+0000 7f5511e1d640 1 --2- 172.21.15.67:0/3880786347 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x560f9ed98e60 0x560f9edabe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:45:47.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.161+0000 7f551161c640 1 --2- 172.21.15.67:0/3880786347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f9ed99240 0x560f9eddaed0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f54fc031bc0 tx=0x7f54fc030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8844c90596c24369 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:47.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.161+0000 7f54eeffd640 1 -- 172.21.15.67:0/3880786347 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54fc05a030 con 0x560f9ed99240 2024-03-24T00:45:47.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.161+0000 7f54eeffd640 1 -- 172.21.15.67:0/3880786347 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54fc0561a0 con 0x560f9ed99240 2024-03-24T00:45:47.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.161+0000 7f54eeffd640 1 -- 172.21.15.67:0/3880786347 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54fc035040 con 0x560f9ed99240 2024-03-24T00:45:47.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.162+0000 7f5516c6b740 1 -- 172.21.15.67:0/3880786347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f9ed99240 msgr2=0x560f9eddaed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:47.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.162+0000 7f5516c6b740 1 --2- 172.21.15.67:0/3880786347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f9ed99240 0x560f9eddaed0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f54fc031bc0 tx=0x7f54fc030000 comp rx=0 tx=0).stop 2024-03-24T00:45:47.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.162+0000 7f5516c6b740 1 -- 172.21.15.67:0/3880786347 shutdown_connections 2024-03-24T00:45:47.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.162+0000 7f5516c6b740 1 --2- 172.21.15.67:0/3880786347 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560f9edda600 0x560f9ed91ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.162+0000 7f5516c6b740 1 --2- 172.21.15.67:0/3880786347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f9ed99240 0x560f9eddaed0 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.162+0000 7f5516c6b740 1 --2- 172.21.15.67:0/3880786347 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x560f9ed98e60 0x560f9edabe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.162+0000 7f5516c6b740 1 -- 172.21.15.67:0/3880786347 >> 172.21.15.67:0/3880786347 conn(0x560f9ed601a0 msgr2=0x560f9edc28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:47.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.162+0000 7f5516c6b740 1 -- 172.21.15.67:0/3880786347 shutdown_connections 2024-03-24T00:45:47.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.162+0000 7f5516c6b740 1 -- 172.21.15.67:0/3880786347 wait complete. 2024-03-24T00:45:47.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.162+0000 7f5516c6b740 1 -- start start 2024-03-24T00:45:47.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.162+0000 7f5516c6b740 1 Processor -- start 2024-03-24T00:45:47.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.163+0000 7f5516c6b740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560f9ed98e60 0x560f9edefcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:47.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.163+0000 7f5516c6b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f9ed99240 0x560f9ee44410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:47.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.163+0000 7f5516c6b740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x560f9edda600 0x560f9edd8030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:47.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.163+0000 7f5516c6b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f550c1f4bf0 con 0x560f9ed99240 2024-03-24T00:45:47.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.163+0000 7f5516c6b740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f9ed0a5b0 con 0x560f9ed98e60 2024-03-24T00:45:47.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.163+0000 7f5516c6b740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x560f9edaadc0 con 0x560f9edda600 2024-03-24T00:45:47.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.163+0000 7f5516c6b740 10 client.0 _ll_register_callbacks cb 0x7f5504006cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:45:47.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.163+0000 7f551161c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f9ed99240 0x560f9ee44410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:47.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.163+0000 7f551161c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f9ed99240 0x560f9ee44410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37522/0 (socket says 172.21.15.67:37522) 2024-03-24T00:45:47.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.163+0000 7f551161c640 1 -- 172.21.15.67:0/2158942331 learned_addr learned my addr 172.21.15.67:0/2158942331 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:47.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.164+0000 7f551261e640 1 --2- 172.21.15.67:0/2158942331 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x560f9edda600 0x560f9edd8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:47.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.164+0000 7f5511e1d640 1 --2- 172.21.15.67:0/2158942331 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560f9ed98e60 0x560f9edefcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:47.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.164+0000 7f551161c640 1 -- 172.21.15.67:0/2158942331 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x560f9edda600 msgr2=0x560f9edd8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:47.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.164+0000 7f551161c640 1 --2- 172.21.15.67:0/2158942331 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x560f9edda600 0x560f9edd8030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.164+0000 7f551161c640 1 -- 172.21.15.67:0/2158942331 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560f9ed98e60 msgr2=0x560f9edefcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:47.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.164+0000 7f551161c640 1 --2- 172.21.15.67:0/2158942331 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560f9ed98e60 0x560f9edefcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.164+0000 7f551161c640 1 -- 172.21.15.67:0/2158942331 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560f9edf0700 con 0x560f9ed99240 2024-03-24T00:45:47.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.164+0000 7f551261e640 1 --2- 172.21.15.67:0/2158942331 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x560f9edda600 0x560f9edd8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:45:47.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.164+0000 7f551161c640 1 --2- 172.21.15.67:0/2158942331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f9ed99240 0x560f9ee44410 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f54fc0098f0 tx=0x7f54fc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18fb1124b2593ef9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:47.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.164+0000 7f54ecff9640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:45:47.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.164+0000 7f54ecff9640 1 -- 172.21.15.67:0/2158942331 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54fc064060 con 0x560f9ed99240 2024-03-24T00:45:47.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.164+0000 7f5516c6b740 1 -- 172.21.15.67:0/2158942331 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f54fc0561a0 con 0x560f9ed99240 2024-03-24T00:45:47.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.164+0000 7f54ecff9640 1 -- 172.21.15.67:0/2158942331 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 6303+0+0 (secure 0 0 0) 0x7f54fc0567f0 con 0x560f9ed99240 2024-03-24T00:45:47.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.164+0000 7f54ecff9640 1 -- 172.21.15.67:0/2158942331 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) v2 ==== 24+0+0 (secure 0 0 0) 0x7f54fc06a050 con 0x560f9ed99240 2024-03-24T00:45:47.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.164+0000 7f5516c6b740 10 client.8043 fetch_fsmap learned FSMap version 61 2024-03-24T00:45:47.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.164+0000 7f5516c6b740 1 -- 172.21.15.67:0/2158942331 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=2+}) v3 -- 0x560f9edac040 con 0x560f9ed99240 2024-03-24T00:45:47.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.164+0000 7f54ecff9640 1 -- 172.21.15.67:0/2158942331 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54fc0561a0 con 0x560f9ed99240 2024-03-24T00:45:47.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.164+0000 7f54ecff9640 1 -- 172.21.15.67:0/2158942331 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 61) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f54fc06d020 con 0x560f9ed99240 2024-03-24T00:45:47.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.165+0000 7f5516c6b740 10 client.8043 fetch_fsmap finished waiting for FSMap version 61 2024-03-24T00:45:47.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.165+0000 7f5516c6b740 10 client.8043 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-03-24T00:45:47.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.165+0000 7f5516c6b740 20 client.8043 populate_metadata read hostname 'smithi067' 2024-03-24T00:45:47.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.165+0000 7f5516c6b740 1 --2- 172.21.15.67:0/2158942331 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x560f9edd8570 0x560f9edc3d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:47.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.165+0000 7f5516c6b740 4 client.8043 mds_command: new command op to 7860 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/dir", "scrubops": ["recursive", "repair"]}] 2024-03-24T00:45:47.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.165+0000 7f5516c6b740 1 -- 172.21.15.67:0/2158942331 --> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] -- command(tid 0: {"prefix": "scrub start", "path": "/dir", "scrubops": ["recursive", "repair"]}) v1 -- 0x560f9ed04900 con 0x560f9edd8570 2024-03-24T00:45:47.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.165+0000 7f5511e1d640 1 --2- 172.21.15.67:0/2158942331 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x560f9edd8570 0x560f9edc3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:47.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.165+0000 7f54ecff9640 10 client.8043 ms_handle_connect on v2:172.21.15.67:6832/1336395002 2024-03-24T00:45:47.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.165+0000 7f5511e1d640 1 --2- 172.21.15.67:0/2158942331 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x560f9edd8570 0x560f9edc3d90 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad9baa1f57100ae0 server_cookie=6e80b9204ff2de7d in_seq=0 out_seq=0 2024-03-24T00:45:47.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.165+0000 7f54ecff9640 1 -- 172.21.15.67:0/2158942331 <== mds.0 v2:172.21.15.67:6832/1336395002 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x560f9ed04900 con 0x560f9edd8570 2024-03-24T00:45:47.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.165+0000 7f54ecff9640 10 client.8043 handle_command_reply: tid=0 2024-03-24T00:45:47.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.165+0000 7f5516c6b740 1 client.8043 shutdown 2024-03-24T00:45:47.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.166+0000 7f5516c6b740 1 -- 172.21.15.67:0/2158942331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f9ed99240 msgr2=0x560f9ee44410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:47.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.166+0000 7f5516c6b740 1 --2- 172.21.15.67:0/2158942331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f9ed99240 0x560f9ee44410 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f54fc0098f0 tx=0x7f54fc004730 comp rx=0 tx=0).stop 2024-03-24T00:45:47.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.166+0000 7f5516c6b740 1 -- 172.21.15.67:0/2158942331 shutdown_connections 2024-03-24T00:45:47.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.166+0000 7f5516c6b740 1 --2- 172.21.15.67:0/2158942331 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x560f9edd8570 0x560f9edc3d90 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.166+0000 7f5516c6b740 1 --2- 172.21.15.67:0/2158942331 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x560f9edda600 0x560f9edd8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.166+0000 7f5516c6b740 1 --2- 172.21.15.67:0/2158942331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f9ed99240 0x560f9ee44410 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.166+0000 7f54ecff9640 0 client.8043 ms_handle_reset on v2:172.21.15.67:6832/1336395002 2024-03-24T00:45:47.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.166+0000 7f5516c6b740 1 --2- 172.21.15.67:0/2158942331 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560f9ed98e60 0x560f9edefcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.166+0000 7f5516c6b740 1 -- 172.21.15.67:0/2158942331 >> 172.21.15.67:0/2158942331 conn(0x560f9ed601a0 msgr2=0x560f9edc06f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:47.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.166+0000 7f5516c6b740 1 -- 172.21.15.67:0/2158942331 shutdown_connections 2024-03-24T00:45:47.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.166+0000 7f5516c6b740 1 -- 172.21.15.67:0/2158942331 wait complete. 2024-03-24T00:45:47.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.166+0000 7f5516c6b740 20 client.8043 trim_cache size 0 max 16384 2024-03-24T00:45:47.172 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:45:47.172 INFO:teuthology.orchestra.run.smithi067.stdout: "return_code": 0, 2024-03-24T00:45:47.172 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_tag": "09f0c7ed-470c-4990-bf9e-af15c4326158", 2024-03-24T00:45:47.172 INFO:teuthology.orchestra.run.smithi067.stdout: "mode": "asynchronous" 2024-03-24T00:45:47.172 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:45:47.173 INFO:tasks.ceph.mds.c.smithi067.stderr:2024-03-24T00:45:47.166+0000 7fdf3050a640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000000 [...2,head] /dir/ auth v15 pv18 ap=3 DIRTYPARENT f(v0 m2024-03-24T00:45:06.076992+0000 1=1+0)->f(v1 m2024-03-24T00:45:06.076992+0000) n(v0 rc2024-03-24T00:45:06.190990+0000 2=1+1)->n(v0 rc2024-03-24T00:45:06.190990+0000 2=1+1) (inest lock dirty) (ifile lock->sync w=1 flushing) (iversion lock) | dirtyscattered=2 lock=1 dirfrag=1 dirtyrstat=1 dirtyparent=1 waiter=1 authpin=1 scrubqueue=0 0x56393232bb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(13)0x10000000000:[<0x1/dir v15>]//[]","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f(v0 m2024-03-24T00:45:06.076992+0000 1=1+0)","memory_value.rstat":"n(v0 rc2024-03-24T00:45:06.190990+0000 2=1+1)","error_str":"freshly-calculated rstats don't match existing ones (will be fixed)"},"return_code":-2} 2024-03-24T00:45:47.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.166+0000 7f5514081640 1 -- 172.21.15.67:0/323300989 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f54f006bbd0 msgr2=0x7f54f006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:47.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.166+0000 7f5514081640 1 --2- 172.21.15.67:0/323300989 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f54f006bbd0 0x7f54f006e090 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f54fc036000 tx=0x7f54fc008030 comp rx=0 tx=0).stop 2024-03-24T00:45:47.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.166+0000 7f5514081640 1 -- 172.21.15.67:0/323300989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f550c0d0b80 msgr2=0x7f550c15c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:47.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.166+0000 7f5514081640 1 --2- 172.21.15.67:0/323300989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f550c0d0b80 0x7f550c15c540 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f550800c3c0 tx=0x7f550800c880 comp rx=0 tx=0).stop 2024-03-24T00:45:47.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.167+0000 7f5514081640 1 -- 172.21.15.67:0/323300989 shutdown_connections 2024-03-24T00:45:47.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.167+0000 7f5514081640 1 --2- 172.21.15.67:0/323300989 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f54f006bbd0 0x7f54f006e090 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.167+0000 7f5514081640 1 --2- 172.21.15.67:0/323300989 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f550c15fa00 0x7f550c1e7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.167+0000 7f5514081640 1 --2- 172.21.15.67:0/323300989 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f550c15dfc0 0x7f550c0cdce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.167+0000 7f5514081640 1 --2- 172.21.15.67:0/323300989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f550c0d0b80 0x7f550c15c540 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.167+0000 7f5514081640 1 -- 172.21.15.67:0/323300989 >> 172.21.15.67:0/323300989 conn(0x7f550c06c880 msgr2=0x7f550c15d390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:47.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.167+0000 7f5514081640 1 -- 172.21.15.67:0/323300989 shutdown_connections 2024-03-24T00:45:47.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.167+0000 7f5514081640 1 -- 172.21.15.67:0/323300989 wait complete. 2024-03-24T00:45:47.177 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:45:47.177 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.6:0 scrub status 2024-03-24T00:45:47.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.281+0000 7fd47b8e8640 1 Processor -- start 2024-03-24T00:45:47.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.281+0000 7fd47b8e8640 1 -- start start 2024-03-24T00:45:47.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.282+0000 7fd47b8e8640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd47415dea0 0x7fd4740d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:47.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.282+0000 7fd47b8e8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd47415e280 0x7fd4740d3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:47.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.282+0000 7fd47b8e8640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd4740b5a60 0x7fd4740b5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:47.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.282+0000 7fd47b8e8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4740b6410 con 0x7fd47415e280 2024-03-24T00:45:47.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.282+0000 7fd47b8e8640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4740d2250 con 0x7fd4740b5a60 2024-03-24T00:45:47.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.282+0000 7fd47b8e8640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4740d2530 con 0x7fd47415dea0 2024-03-24T00:45:47.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.282+0000 7fd478e83640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd47415e280 0x7fd4740d3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:47.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.282+0000 7fd478e83640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd47415e280 0x7fd4740d3910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37526/0 (socket says 172.21.15.67:37526) 2024-03-24T00:45:47.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.282+0000 7fd478e83640 1 -- 172.21.15.67:0/2387904327 learned_addr learned my addr 172.21.15.67:0/2387904327 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:47.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.282+0000 7fd479684640 1 --2- 172.21.15.67:0/2387904327 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd47415dea0 0x7fd4740d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:47.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.282+0000 7fd479e85640 1 --2- 172.21.15.67:0/2387904327 >> v2:172.21.15.72:3300/0 conn(0x7fd4740b5a60 0x7fd4740b5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:47.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.283+0000 7fd478e83640 1 -- 172.21.15.67:0/2387904327 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd47415dea0 msgr2=0x7fd4740d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:47.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.283+0000 7fd478e83640 1 --2- 172.21.15.67:0/2387904327 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd47415dea0 0x7fd4740d33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.283+0000 7fd478e83640 1 -- 172.21.15.67:0/2387904327 >> v2:172.21.15.72:3300/0 conn(0x7fd4740b5a60 msgr2=0x7fd4740b5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:47.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.283+0000 7fd478e83640 1 --2- 172.21.15.67:0/2387904327 >> v2:172.21.15.72:3300/0 conn(0x7fd4740b5a60 0x7fd4740b5ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.283+0000 7fd478e83640 1 -- 172.21.15.67:0/2387904327 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4740d2810 con 0x7fd47415e280 2024-03-24T00:45:47.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.284+0000 7fd478e83640 1 --2- 172.21.15.67:0/2387904327 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd47415e280 0x7fd4740d3910 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fd4740d2460 tx=0x7fd46402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=657745485cef8f22 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:47.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.284+0000 7fd46bfff640 1 -- 172.21.15.67:0/2387904327 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4640317c0 con 0x7fd47415e280 2024-03-24T00:45:47.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.284+0000 7fd46bfff640 1 -- 172.21.15.67:0/2387904327 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd464037070 con 0x7fd47415e280 2024-03-24T00:45:47.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.284+0000 7fd46bfff640 1 -- 172.21.15.67:0/2387904327 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd464031b80 con 0x7fd47415e280 2024-03-24T00:45:47.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.285+0000 7fd47b8e8640 1 -- 172.21.15.67:0/2387904327 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd47415e280 msgr2=0x7fd4740d3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:47.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.285+0000 7fd47b8e8640 1 --2- 172.21.15.67:0/2387904327 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd47415e280 0x7fd4740d3910 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fd4740d2460 tx=0x7fd46402b860 comp rx=0 tx=0).stop 2024-03-24T00:45:47.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.285+0000 7fd47b8e8640 1 -- 172.21.15.67:0/2387904327 shutdown_connections 2024-03-24T00:45:47.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.285+0000 7fd47b8e8640 1 --2- 172.21.15.67:0/2387904327 >> v2:172.21.15.72:3300/0 conn(0x7fd4740b5a60 0x7fd4740b5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.285+0000 7fd47b8e8640 1 --2- 172.21.15.67:0/2387904327 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd47415e280 0x7fd4740d3910 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.285+0000 7fd47b8e8640 1 --2- 172.21.15.67:0/2387904327 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd47415dea0 0x7fd4740d33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.285+0000 7fd47b8e8640 1 -- 172.21.15.67:0/2387904327 >> 172.21.15.67:0/2387904327 conn(0x7fd474072da0 msgr2=0x7fd4740d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:47.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.285+0000 7fd47b8e8640 1 -- 172.21.15.67:0/2387904327 shutdown_connections 2024-03-24T00:45:47.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.285+0000 7fd47b8e8640 1 -- 172.21.15.67:0/2387904327 wait complete. 2024-03-24T00:45:47.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.286+0000 7fd47b8e8640 1 Processor -- start 2024-03-24T00:45:47.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.287+0000 7fd47b8e8640 1 -- start start 2024-03-24T00:45:47.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.288+0000 7fd47b8e8640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd4740b5a60 0x7fd4740aca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:47.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.288+0000 7fd47b8e8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd47415dea0 0x7fd474150440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:47.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.288+0000 7fd47b8e8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd47415e280 0x7fd4740ad160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:47.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.288+0000 7fd47b8e8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd47406c7c0 con 0x7fd47415e280 2024-03-24T00:45:47.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.288+0000 7fd47b8e8640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd47406c640 con 0x7fd47415dea0 2024-03-24T00:45:47.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.288+0000 7fd47b8e8640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd47406c940 con 0x7fd4740b5a60 2024-03-24T00:45:47.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.288+0000 7fd479e85640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd47415e280 0x7fd4740ad160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:47.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.288+0000 7fd479e85640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd47415e280 0x7fd4740ad160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37538/0 (socket says 172.21.15.67:37538) 2024-03-24T00:45:47.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.288+0000 7fd479e85640 1 -- 172.21.15.67:0/2232672500 learned_addr learned my addr 172.21.15.67:0/2232672500 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:47.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.288+0000 7fd478e83640 1 --2- 172.21.15.67:0/2232672500 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd47415dea0 0x7fd474150440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:47.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.288+0000 7fd479684640 1 --2- 172.21.15.67:0/2232672500 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd4740b5a60 0x7fd4740aca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:47.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.288+0000 7fd479e85640 1 -- 172.21.15.67:0/2232672500 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd4740b5a60 msgr2=0x7fd4740aca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:47.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.288+0000 7fd479e85640 1 --2- 172.21.15.67:0/2232672500 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd4740b5a60 0x7fd4740aca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.288+0000 7fd479e85640 1 -- 172.21.15.67:0/2232672500 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd47415dea0 msgr2=0x7fd474150440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:47.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.288+0000 7fd479e85640 1 --2- 172.21.15.67:0/2232672500 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd47415dea0 0x7fd474150440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.288+0000 7fd479e85640 1 -- 172.21.15.67:0/2232672500 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd474150980 con 0x7fd47415e280 2024-03-24T00:45:47.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.288+0000 7fd479e85640 1 --2- 172.21.15.67:0/2232672500 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd47415e280 0x7fd4740ad160 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fd46c00eed0 tx=0x7fd46c00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:47.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.288+0000 7fd469ffb640 1 -- 172.21.15.67:0/2232672500 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd46c018020 con 0x7fd47415e280 2024-03-24T00:45:47.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.289+0000 7fd469ffb640 1 -- 172.21.15.67:0/2232672500 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd46c00bd10 con 0x7fd47415e280 2024-03-24T00:45:47.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.289+0000 7fd47b8e8640 1 -- 172.21.15.67:0/2232672500 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4740ab9a0 con 0x7fd47415e280 2024-03-24T00:45:47.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.289+0000 7fd469ffb640 1 -- 172.21.15.67:0/2232672500 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd46c014640 con 0x7fd47415e280 2024-03-24T00:45:47.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.289+0000 7fd47b8e8640 1 -- 172.21.15.67:0/2232672500 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4740abf60 con 0x7fd47415e280 2024-03-24T00:45:47.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.290+0000 7fd469ffb640 1 -- 172.21.15.67:0/2232672500 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd46c005d10 con 0x7fd47415e280 2024-03-24T00:45:47.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.291+0000 7fd469ffb640 1 --2- 172.21.15.67:0/2232672500 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd45806bc00 0x7fd45806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:47.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.291+0000 7fd469ffb640 1 -- 172.21.15.67:0/2232672500 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 6303+0+0 (secure 0 0 0) 0x7fd46c08b240 con 0x7fd47415e280 2024-03-24T00:45:47.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.291+0000 7fd479684640 1 --2- 172.21.15.67:0/2232672500 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd45806bc00 0x7fd45806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:47.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.291+0000 7fd479684640 1 --2- 172.21.15.67:0/2232672500 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd45806bc00 0x7fd45806e0c0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fd470009a10 tx=0x7fd470008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:47.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.292+0000 7fd47e5ab740 1 Processor -- start 2024-03-24T00:45:47.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.292+0000 7fd47e5ab740 1 -- start start 2024-03-24T00:45:47.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.292+0000 7fd47e5ab740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c357879e50 0x55c35784cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:47.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.292+0000 7fd47e5ab740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55c35787a230 0x55c35787cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:47.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.292+0000 7fd47e5ab740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c35784d330 0x55c35787d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:47.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.292+0000 7fd47e5ab740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3577b73d0 con 0x55c35784d330 2024-03-24T00:45:47.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.292+0000 7fd47e5ab740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c35775fad0 con 0x55c357879e50 2024-03-24T00:45:47.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.292+0000 7fd47e5ab740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c3577b3470 con 0x55c35787a230 2024-03-24T00:45:47.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.292+0000 7fd479e85640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c35784d330 0x55c35787d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:47.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.292+0000 7fd479e85640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c35784d330 0x55c35787d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37546/0 (socket says 172.21.15.67:37546) 2024-03-24T00:45:47.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.292+0000 7fd479e85640 1 -- 172.21.15.67:0/1244630734 learned_addr learned my addr 172.21.15.67:0/1244630734 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:47.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.292+0000 7fd479684640 1 --2- 172.21.15.67:0/1244630734 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c357879e50 0x55c35784cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:47.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.292+0000 7fd478e83640 1 --2- 172.21.15.67:0/1244630734 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55c35787a230 0x55c35787cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:47.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.293+0000 7fd479e85640 1 -- 172.21.15.67:0/1244630734 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55c35787a230 msgr2=0x55c35787cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:47.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.293+0000 7fd479e85640 1 --2- 172.21.15.67:0/1244630734 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55c35787a230 0x55c35787cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.293+0000 7fd479e85640 1 -- 172.21.15.67:0/1244630734 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c357879e50 msgr2=0x55c35784cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:47.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.293+0000 7fd479e85640 1 --2- 172.21.15.67:0/1244630734 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c357879e50 0x55c35784cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.293+0000 7fd479e85640 1 -- 172.21.15.67:0/1244630734 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c3575957d0 con 0x55c35784d330 2024-03-24T00:45:47.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.293+0000 7fd479e85640 1 --2- 172.21.15.67:0/1244630734 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c35784d330 0x55c35787d120 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fd46c0615b0 tx=0x7fd46c059ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=657c9ed91b50a612 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:47.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.293+0000 7fd45effd640 1 -- 172.21.15.67:0/1244630734 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd46c087070 con 0x55c35784d330 2024-03-24T00:45:47.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.293+0000 7fd45effd640 1 -- 172.21.15.67:0/1244630734 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd46c05d520 con 0x55c35784d330 2024-03-24T00:45:47.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.293+0000 7fd45effd640 1 -- 172.21.15.67:0/1244630734 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd46c071070 con 0x55c35784d330 2024-03-24T00:45:47.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.293+0000 7fd47e5ab740 1 -- 172.21.15.67:0/1244630734 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c35784d330 msgr2=0x55c35787d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:47.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.293+0000 7fd47e5ab740 1 --2- 172.21.15.67:0/1244630734 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c35784d330 0x55c35787d120 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fd46c0615b0 tx=0x7fd46c059ad0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.294+0000 7fd47e5ab740 1 -- 172.21.15.67:0/1244630734 shutdown_connections 2024-03-24T00:45:47.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.294+0000 7fd47e5ab740 1 --2- 172.21.15.67:0/1244630734 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c35784d330 0x55c35787d120 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.294+0000 7fd47e5ab740 1 --2- 172.21.15.67:0/1244630734 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55c35787a230 0x55c35787cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.294+0000 7fd47e5ab740 1 --2- 172.21.15.67:0/1244630734 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c357879e50 0x55c35784cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.294+0000 7fd47e5ab740 1 -- 172.21.15.67:0/1244630734 >> 172.21.15.67:0/1244630734 conn(0x55c35784f060 msgr2=0x55c357897be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:47.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.294+0000 7fd47e5ab740 1 -- 172.21.15.67:0/1244630734 shutdown_connections 2024-03-24T00:45:47.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.294+0000 7fd47e5ab740 1 -- 172.21.15.67:0/1244630734 wait complete. 2024-03-24T00:45:47.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.294+0000 7fd47e5ab740 1 -- start start 2024-03-24T00:45:47.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.294+0000 7fd47e5ab740 1 Processor -- start 2024-03-24T00:45:47.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.296+0000 7fd47e5ab740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c357879e50 0x55c357884030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:47.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.296+0000 7fd47e5ab740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55c35787a230 0x55c35787e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:47.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.296+0000 7fd47e5ab740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c35787e810 0x55c3578ab8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:47.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.296+0000 7fd47e5ab740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c357851700 con 0x55c35787e810 2024-03-24T00:45:47.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.296+0000 7fd47e5ab740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c357773060 con 0x55c357879e50 2024-03-24T00:45:47.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.296+0000 7fd47e5ab740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c357853f40 con 0x55c35787a230 2024-03-24T00:45:47.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.296+0000 7fd47e5ab740 10 client.0 _ll_register_callbacks cb 0x55c3577556a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:45:47.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.296+0000 7fd479e85640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c35787e810 0x55c3578ab8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:47.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.296+0000 7fd479e85640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c35787e810 0x55c3578ab8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37550/0 (socket says 172.21.15.67:37550) 2024-03-24T00:45:47.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.296+0000 7fd479e85640 1 -- 172.21.15.67:0/1097567590 learned_addr learned my addr 172.21.15.67:0/1097567590 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:47.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.296+0000 7fd478e83640 1 --2- 172.21.15.67:0/1097567590 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55c35787a230 0x55c35787e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:47.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.296+0000 7fd479684640 1 --2- 172.21.15.67:0/1097567590 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c357879e50 0x55c357884030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:47.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.296+0000 7fd479e85640 1 -- 172.21.15.67:0/1097567590 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55c35787a230 msgr2=0x55c35787e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:47.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.296+0000 7fd479e85640 1 --2- 172.21.15.67:0/1097567590 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55c35787a230 0x55c35787e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.296+0000 7fd479e85640 1 -- 172.21.15.67:0/1097567590 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c357879e50 msgr2=0x55c357884030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:47.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.296+0000 7fd479e85640 1 --2- 172.21.15.67:0/1097567590 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c357879e50 0x55c357884030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.296+0000 7fd479e85640 1 -- 172.21.15.67:0/1097567590 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c3574cb3f0 con 0x55c35787e810 2024-03-24T00:45:47.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.296+0000 7fd479e85640 1 --2- 172.21.15.67:0/1097567590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c35787e810 0x55c3578ab8b0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7fd46c059590 tx=0x7fd46c01f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a00b888a2a43166 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:47.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.296+0000 7fd453fff640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:45:47.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.296+0000 7fd453fff640 1 -- 172.21.15.67:0/1097567590 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd46c087030 con 0x55c35787e810 2024-03-24T00:45:47.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.296+0000 7fd453fff640 1 -- 172.21.15.67:0/1097567590 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 6303+0+0 (secure 0 0 0) 0x7fd46c078300 con 0x55c35787e810 2024-03-24T00:45:47.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.296+0000 7fd47e5ab740 1 -- 172.21.15.67:0/1097567590 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd46c05d520 con 0x55c35787e810 2024-03-24T00:45:47.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.296+0000 7fd453fff640 1 -- 172.21.15.67:0/1097567590 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd46c07e050 con 0x55c35787e810 2024-03-24T00:45:47.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.296+0000 7fd47e5ab740 10 client.8055 fetch_fsmap learned FSMap version 61 2024-03-24T00:45:47.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.297+0000 7fd47e5ab740 1 -- 172.21.15.67:0/1097567590 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=2+}) v3 -- 0x55c35748b0f0 con 0x55c35787e810 2024-03-24T00:45:47.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.297+0000 7fd453fff640 1 -- 172.21.15.67:0/1097567590 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd46c05d520 con 0x55c35787e810 2024-03-24T00:45:47.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.297+0000 7fd453fff640 1 -- 172.21.15.67:0/1097567590 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 61) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fd46c081020 con 0x55c35787e810 2024-03-24T00:45:47.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.297+0000 7fd47e5ab740 10 client.8055 fetch_fsmap finished waiting for FSMap version 61 2024-03-24T00:45:47.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.297+0000 7fd47e5ab740 10 client.8055 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-03-24T00:45:47.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.297+0000 7fd47e5ab740 20 client.8055 populate_metadata read hostname 'smithi067' 2024-03-24T00:45:47.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.297+0000 7fd47e5ab740 1 --2- 172.21.15.67:0/1097567590 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x55c3578ac2c0 0x55c357869780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:47.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.297+0000 7fd47e5ab740 4 client.8055 mds_command: new command op to 7860 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:45:47.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.297+0000 7fd47e5ab740 1 -- 172.21.15.67:0/1097567590 --> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c3577c4910 con 0x55c3578ac2c0 2024-03-24T00:45:47.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.297+0000 7fd479684640 1 --2- 172.21.15.67:0/1097567590 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x55c3578ac2c0 0x55c357869780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:47.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.297+0000 7fd479684640 1 --2- 172.21.15.67:0/1097567590 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x55c3578ac2c0 0x55c357869780 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed52f23e0bf5d46 server_cookie=f3048ebf94610d44 in_seq=0 out_seq=0 2024-03-24T00:45:47.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.297+0000 7fd453fff640 10 client.8055 ms_handle_connect on v2:172.21.15.67:6832/1336395002 2024-03-24T00:45:47.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.298+0000 7fd453fff640 1 -- 172.21.15.67:0/1097567590 <== mds.0 v2:172.21.15.67:6832/1336395002 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c3577c4910 con 0x55c3578ac2c0 2024-03-24T00:45:47.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.298+0000 7fd453fff640 10 client.8055 handle_command_reply: tid=0 2024-03-24T00:45:47.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.298+0000 7fd47e5ab740 1 client.8055 shutdown 2024-03-24T00:45:47.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.298+0000 7fd47e5ab740 1 -- 172.21.15.67:0/1097567590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c35787e810 msgr2=0x55c3578ab8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:47.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.298+0000 7fd47e5ab740 1 --2- 172.21.15.67:0/1097567590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c35787e810 0x55c3578ab8b0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7fd46c059590 tx=0x7fd46c01f040 comp rx=0 tx=0).stop 2024-03-24T00:45:47.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.298+0000 7fd47e5ab740 1 -- 172.21.15.67:0/1097567590 shutdown_connections 2024-03-24T00:45:47.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.298+0000 7fd47e5ab740 1 --2- 172.21.15.67:0/1097567590 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x55c3578ac2c0 0x55c357869780 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.298+0000 7fd47e5ab740 1 --2- 172.21.15.67:0/1097567590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c35787e810 0x55c3578ab8b0 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.298+0000 7fd47e5ab740 1 --2- 172.21.15.67:0/1097567590 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55c35787a230 0x55c35787e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.298+0000 7fd453fff640 0 client.8055 ms_handle_reset on v2:172.21.15.67:6832/1336395002 2024-03-24T00:45:47.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.298+0000 7fd47e5ab740 1 --2- 172.21.15.67:0/1097567590 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c357879e50 0x55c357884030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.298+0000 7fd47e5ab740 1 -- 172.21.15.67:0/1097567590 >> 172.21.15.67:0/1097567590 conn(0x55c35784d330 msgr2=0x55c35789b060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:47.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.298+0000 7fd47e5ab740 1 -- 172.21.15.67:0/1097567590 shutdown_connections 2024-03-24T00:45:47.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.298+0000 7fd47e5ab740 1 -- 172.21.15.67:0/1097567590 wait complete. 2024-03-24T00:45:47.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.298+0000 7fd47e5ab740 20 client.8055 trim_cache size 0 max 16384 2024-03-24T00:45:47.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.315+0000 7fd47e5ab740 1 Processor -- start 2024-03-24T00:45:47.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.315+0000 7fd47e5ab740 1 -- start start 2024-03-24T00:45:47.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.316+0000 7fd47e5ab740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c357879e50 0x55c35787e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:47.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.316+0000 7fd47e5ab740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c35787a230 0x55c35789b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:47.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.316+0000 7fd47e5ab740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55c35787e810 0x55c35789b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:47.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.316+0000 7fd47e5ab740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd474200a30 con 0x55c35787a230 2024-03-24T00:45:47.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.316+0000 7fd47e5ab740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4741ffbc0 con 0x55c357879e50 2024-03-24T00:45:47.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.316+0000 7fd47e5ab740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4741600a0 con 0x55c35787e810 2024-03-24T00:45:47.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.316+0000 7fd478e83640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c35787a230 0x55c35789b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:47.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.316+0000 7fd478e83640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c35787a230 0x55c35789b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37564/0 (socket says 172.21.15.67:37564) 2024-03-24T00:45:47.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.316+0000 7fd478e83640 1 -- 172.21.15.67:0/297848034 learned_addr learned my addr 172.21.15.67:0/297848034 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:47.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.316+0000 7fd479e85640 1 --2- 172.21.15.67:0/297848034 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55c35787e810 0x55c35789b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:47.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.316+0000 7fd479684640 1 --2- 172.21.15.67:0/297848034 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c357879e50 0x55c35787e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:47.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.316+0000 7fd478e83640 1 -- 172.21.15.67:0/297848034 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55c35787e810 msgr2=0x55c35789b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:47.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.316+0000 7fd478e83640 1 --2- 172.21.15.67:0/297848034 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55c35787e810 0x55c35789b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.316+0000 7fd478e83640 1 -- 172.21.15.67:0/297848034 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c357879e50 msgr2=0x55c35787e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:47.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.316+0000 7fd478e83640 1 --2- 172.21.15.67:0/297848034 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c357879e50 0x55c35787e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.316+0000 7fd478e83640 1 -- 172.21.15.67:0/297848034 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c3578794f0 con 0x55c35787a230 2024-03-24T00:45:47.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.316+0000 7fd479e85640 1 --2- 172.21.15.67:0/297848034 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55c35787e810 0x55c35789b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:45:47.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.316+0000 7fd479684640 1 --2- 172.21.15.67:0/297848034 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c357879e50 0x55c35787e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:45:47.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.316+0000 7fd478e83640 1 --2- 172.21.15.67:0/297848034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c35787a230 0x55c35789b200 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fd464000c00 tx=0x7fd464030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8594386df5baee5c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:47.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.316+0000 7fd45e7fc640 1 -- 172.21.15.67:0/297848034 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd46404b070 con 0x55c35787a230 2024-03-24T00:45:47.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.316+0000 7fd45e7fc640 1 -- 172.21.15.67:0/297848034 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd464037040 con 0x55c35787a230 2024-03-24T00:45:47.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.316+0000 7fd45e7fc640 1 -- 172.21.15.67:0/297848034 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4640319e0 con 0x55c35787a230 2024-03-24T00:45:47.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.317+0000 7fd47e5ab740 1 -- 172.21.15.67:0/297848034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c35787a230 msgr2=0x55c35789b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:47.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.317+0000 7fd47e5ab740 1 --2- 172.21.15.67:0/297848034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c35787a230 0x55c35789b200 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fd464000c00 tx=0x7fd464030000 comp rx=0 tx=0).stop 2024-03-24T00:45:47.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.317+0000 7fd47e5ab740 1 -- 172.21.15.67:0/297848034 shutdown_connections 2024-03-24T00:45:47.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.317+0000 7fd47e5ab740 1 --2- 172.21.15.67:0/297848034 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55c35787e810 0x55c35789b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.317+0000 7fd47e5ab740 1 --2- 172.21.15.67:0/297848034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c35787a230 0x55c35789b200 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.317+0000 7fd47e5ab740 1 --2- 172.21.15.67:0/297848034 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c357879e50 0x55c35787e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.317+0000 7fd47e5ab740 1 -- 172.21.15.67:0/297848034 >> 172.21.15.67:0/297848034 conn(0x55c35784d330 msgr2=0x55c35787c940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:47.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.317+0000 7fd47e5ab740 1 -- 172.21.15.67:0/297848034 shutdown_connections 2024-03-24T00:45:47.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.317+0000 7fd47e5ab740 1 -- 172.21.15.67:0/297848034 wait complete. 2024-03-24T00:45:47.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.318+0000 7fd47e5ab740 1 -- start start 2024-03-24T00:45:47.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.318+0000 7fd47e5ab740 1 Processor -- start 2024-03-24T00:45:47.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.319+0000 7fd47e5ab740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55c357879e50 0x55c357906350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:47.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.319+0000 7fd47e5ab740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c35787a230 0x55c35789adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:47.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.320+0000 7fd47e5ab740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c35787e810 0x55c35789b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:47.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.320+0000 7fd47e5ab740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3578a8160 con 0x55c35787a230 2024-03-24T00:45:47.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.320+0000 7fd47e5ab740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4741ff660 con 0x55c35787e810 2024-03-24T00:45:47.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.320+0000 7fd47e5ab740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c3577b9870 con 0x55c357879e50 2024-03-24T00:45:47.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.320+0000 7fd47e5ab740 10 client.0 _ll_register_callbacks cb 0x7fd46c085070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:45:47.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.320+0000 7fd478e83640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c35787a230 0x55c35789adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:47.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.320+0000 7fd478e83640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c35787a230 0x55c35789adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37572/0 (socket says 172.21.15.67:37572) 2024-03-24T00:45:47.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.320+0000 7fd478e83640 1 -- 172.21.15.67:0/2495580620 learned_addr learned my addr 172.21.15.67:0/2495580620 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:45:47.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.320+0000 7fd479e85640 1 --2- 172.21.15.67:0/2495580620 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c35787e810 0x55c35789b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:47.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.320+0000 7fd479684640 1 --2- 172.21.15.67:0/2495580620 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55c357879e50 0x55c357906350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:47.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.320+0000 7fd478e83640 1 -- 172.21.15.67:0/2495580620 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55c357879e50 msgr2=0x55c357906350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:47.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.320+0000 7fd478e83640 1 --2- 172.21.15.67:0/2495580620 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55c357879e50 0x55c357906350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.320+0000 7fd478e83640 1 -- 172.21.15.67:0/2495580620 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c35787e810 msgr2=0x55c35789b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:47.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.320+0000 7fd478e83640 1 --2- 172.21.15.67:0/2495580620 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c35787e810 0x55c35789b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.320+0000 7fd478e83640 1 -- 172.21.15.67:0/2495580620 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c3578792c0 con 0x55c35787a230 2024-03-24T00:45:47.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.320+0000 7fd479684640 1 --2- 172.21.15.67:0/2495580620 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55c357879e50 0x55c357906350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:45:47.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.320+0000 7fd479e85640 1 --2- 172.21.15.67:0/2495580620 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c35787e810 0x55c35789b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:45:47.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.320+0000 7fd478e83640 1 --2- 172.21.15.67:0/2495580620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c35787a230 0x55c35789adb0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7fd464009930 tx=0x7fd464004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dc55347b4a85b6a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:47.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.320+0000 7fd453fff640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:45:47.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.320+0000 7fd453fff640 1 -- 172.21.15.67:0/2495580620 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd464054060 con 0x55c35787a230 2024-03-24T00:45:47.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.320+0000 7fd453fff640 1 -- 172.21.15.67:0/2495580620 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 6303+0+0 (secure 0 0 0) 0x7fd464032450 con 0x55c35787a230 2024-03-24T00:45:47.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.320+0000 7fd47e5ab740 1 -- 172.21.15.67:0/2495580620 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd464037040 con 0x55c35787a230 2024-03-24T00:45:47.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.321+0000 7fd453fff640 1 -- 172.21.15.67:0/2495580620 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd46404b040 con 0x55c35787a230 2024-03-24T00:45:47.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.321+0000 7fd47e5ab740 10 client.8061 fetch_fsmap learned FSMap version 61 2024-03-24T00:45:47.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.321+0000 7fd47e5ab740 1 -- 172.21.15.67:0/2495580620 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=2+}) v3 -- 0x55c357876710 con 0x55c35787a230 2024-03-24T00:45:47.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.321+0000 7fd453fff640 1 -- 172.21.15.67:0/2495580620 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd464037040 con 0x55c35787a230 2024-03-24T00:45:47.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.321+0000 7fd453fff640 1 -- 172.21.15.67:0/2495580620 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 61) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fd46404e070 con 0x55c35787a230 2024-03-24T00:45:47.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.321+0000 7fd47e5ab740 10 client.8061 fetch_fsmap finished waiting for FSMap version 61 2024-03-24T00:45:47.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.321+0000 7fd47e5ab740 10 client.8061 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-03-24T00:45:47.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.321+0000 7fd47e5ab740 20 client.8061 populate_metadata read hostname 'smithi067' 2024-03-24T00:45:47.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.321+0000 7fd47e5ab740 1 --2- 172.21.15.67:0/2495580620 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x55c35787e190 0x55c35784bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:47.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.321+0000 7fd47e5ab740 4 client.8061 mds_command: new command op to 7860 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:45:47.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.321+0000 7fd47e5ab740 1 -- 172.21.15.67:0/2495580620 --> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55c3577bec60 con 0x55c35787e190 2024-03-24T00:45:47.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.321+0000 7fd479684640 1 --2- 172.21.15.67:0/2495580620 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x55c35787e190 0x55c35784bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:47.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.321+0000 7fd453fff640 10 client.8061 ms_handle_connect on v2:172.21.15.67:6832/1336395002 2024-03-24T00:45:47.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.321+0000 7fd479684640 1 --2- 172.21.15.67:0/2495580620 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x55c35787e190 0x55c35784bca0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6cd88bf146367e8e server_cookie=753c84bcdb6cd622 in_seq=0 out_seq=0 2024-03-24T00:45:47.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.321+0000 7fd453fff640 1 -- 172.21.15.67:0/2495580620 <== mds.0 v2:172.21.15.67:6832/1336395002 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+263 (crc 0 0 0) 0x55c3577bec60 con 0x55c35787e190 2024-03-24T00:45:47.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.321+0000 7fd453fff640 10 client.8061 handle_command_reply: tid=0 2024-03-24T00:45:47.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.321+0000 7fd47e5ab740 1 client.8061 shutdown 2024-03-24T00:45:47.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.322+0000 7fd47e5ab740 1 -- 172.21.15.67:0/2495580620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c35787a230 msgr2=0x55c35789adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:47.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.322+0000 7fd47e5ab740 1 --2- 172.21.15.67:0/2495580620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c35787a230 0x55c35789adb0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7fd464009930 tx=0x7fd464004730 comp rx=0 tx=0).stop 2024-03-24T00:45:47.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.322+0000 7fd47e5ab740 1 -- 172.21.15.67:0/2495580620 shutdown_connections 2024-03-24T00:45:47.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.322+0000 7fd47e5ab740 1 --2- 172.21.15.67:0/2495580620 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x55c35787e190 0x55c35784bca0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.322+0000 7fd47e5ab740 1 --2- 172.21.15.67:0/2495580620 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c35787e810 0x55c35789b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.322+0000 7fd47e5ab740 1 --2- 172.21.15.67:0/2495580620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c35787a230 0x55c35789adb0 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.322+0000 7fd453fff640 0 client.8061 ms_handle_reset on v2:172.21.15.67:6832/1336395002 2024-03-24T00:45:47.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.322+0000 7fd47e5ab740 1 --2- 172.21.15.67:0/2495580620 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55c357879e50 0x55c357906350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.322+0000 7fd47e5ab740 1 -- 172.21.15.67:0/2495580620 >> 172.21.15.67:0/2495580620 conn(0x55c35784d330 msgr2=0x55c357850b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:47.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.322+0000 7fd47e5ab740 1 -- 172.21.15.67:0/2495580620 shutdown_connections 2024-03-24T00:45:47.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.322+0000 7fd47e5ab740 1 -- 172.21.15.67:0/2495580620 wait complete. 2024-03-24T00:45:47.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.322+0000 7fd47e5ab740 20 client.8061 trim_cache size 0 max 16384 2024-03-24T00:45:47.328 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:45:47.328 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T00:45:47.328 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": { 2024-03-24T00:45:47.328 INFO:teuthology.orchestra.run.smithi067.stdout: "09f0c7ed-470c-4990-bf9e-af15c4326158": { 2024-03-24T00:45:47.328 INFO:teuthology.orchestra.run.smithi067.stdout: "path": "/dir", 2024-03-24T00:45:47.328 INFO:teuthology.orchestra.run.smithi067.stdout: "tag": "09f0c7ed-470c-4990-bf9e-af15c4326158", 2024-03-24T00:45:47.328 INFO:teuthology.orchestra.run.smithi067.stdout: "options": "recursive,repair" 2024-03-24T00:45:47.328 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:45:47.328 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:45:47.328 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:45:47.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.322+0000 7fd47b8e8640 1 -- 172.21.15.67:0/2232672500 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd45806bc00 msgr2=0x7fd45806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:47.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.322+0000 7fd47b8e8640 1 --2- 172.21.15.67:0/2232672500 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd45806bc00 0x7fd45806e0c0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fd470009a10 tx=0x7fd470008040 comp rx=0 tx=0).stop 2024-03-24T00:45:47.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.322+0000 7fd47b8e8640 1 -- 172.21.15.67:0/2232672500 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd47415e280 msgr2=0x7fd4740ad160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:47.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.322+0000 7fd47b8e8640 1 --2- 172.21.15.67:0/2232672500 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd47415e280 0x7fd4740ad160 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fd46c00eed0 tx=0x7fd46c00b480 comp rx=0 tx=0).stop 2024-03-24T00:45:47.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.322+0000 7fd47b8e8640 1 -- 172.21.15.67:0/2232672500 shutdown_connections 2024-03-24T00:45:47.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.322+0000 7fd47b8e8640 1 --2- 172.21.15.67:0/2232672500 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd45806bc00 0x7fd45806e0c0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.323+0000 7fd47b8e8640 1 --2- 172.21.15.67:0/2232672500 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd47415e280 0x7fd4740ad160 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.323+0000 7fd47b8e8640 1 --2- 172.21.15.67:0/2232672500 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd47415dea0 0x7fd474150440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.323+0000 7fd47b8e8640 1 --2- 172.21.15.67:0/2232672500 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd4740b5a60 0x7fd4740aca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:47.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.323+0000 7fd47b8e8640 1 -- 172.21.15.67:0/2232672500 >> 172.21.15.67:0/2232672500 conn(0x7fd474072da0 msgr2=0x7fd4740d3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:47.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.323+0000 7fd47b8e8640 1 -- 172.21.15.67:0/2232672500 shutdown_connections 2024-03-24T00:45:47.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:45:47.323+0000 7fd47b8e8640 1 -- 172.21.15.67:0/2232672500 wait complete. 2024-03-24T00:45:47.333 INFO:tasks.cephfs.filesystem:scrub status for tag:09f0c7ed-470c-4990-bf9e-af15c4326158 - {'path': '/dir', 'tag': '09f0c7ed-470c-4990-bf9e-af15c4326158', 'options': 'recursive,repair'} 2024-03-24T00:46:16.790 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:46:16.793 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:46:17.334 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:46:17.334 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.6:0 scrub status 2024-03-24T00:46:17.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.437+0000 7fd7a01df640 1 Processor -- start 2024-03-24T00:46:17.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.437+0000 7fd7a01df640 1 -- start start 2024-03-24T00:46:17.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.438+0000 7fd7a01df640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd79815dea0 0x7fd7980d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:17.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.438+0000 7fd7a01df640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd7980b6170 0x7fd7980d3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:17.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.438+0000 7fd7a01df640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7980d2250 con 0x7fd79815e280 2024-03-24T00:46:17.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.438+0000 7fd7a01df640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7980d2530 con 0x7fd7980b6170 2024-03-24T00:46:17.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.438+0000 7fd7a01df640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7980d2810 con 0x7fd79815dea0 2024-03-24T00:46:17.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.438+0000 7fd79d77a640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd79815e280 0x7fd7980b5a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59410/0 (socket says 172.21.15.67:59410) 2024-03-24T00:46:17.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.438+0000 7fd79d77a640 1 -- 172.21.15.67:0/4075568297 learned_addr learned my addr 172.21.15.67:0/4075568297 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:46:17.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.438+0000 7fd79e77c640 1 --2- 172.21.15.67:0/4075568297 >> v2:172.21.15.67:3300/0 conn(0x7fd7980b6170 0x7fd7980d3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:17.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.438+0000 7fd79df7b640 1 --2- 172.21.15.67:0/4075568297 >> v2:172.21.15.72:3300/0 conn(0x7fd79815dea0 0x7fd7980d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:17.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.438+0000 7fd79cf79640 1 -- 172.21.15.67:0/4075568297 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1293773029 0 0) 0x7fd7980d2250 con 0x7fd79815e280 2024-03-24T00:46:17.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.438+0000 7fd79cf79640 1 -- 172.21.15.67:0/4075568297 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd784003520 con 0x7fd79815e280 2024-03-24T00:46:17.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.438+0000 7fd79e77c640 1 -- 172.21.15.67:0/4075568297 >> v2:172.21.15.72:3300/0 conn(0x7fd79815dea0 msgr2=0x7fd7980d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:17.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.438+0000 7fd79e77c640 1 --2- 172.21.15.67:0/4075568297 >> v2:172.21.15.72:3300/0 conn(0x7fd79815dea0 0x7fd7980d33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.438+0000 7fd79e77c640 1 -- 172.21.15.67:0/4075568297 >> v1:172.21.15.67:6789/0 conn(0x7fd79815e280 legacy=0x7fd7980b5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:17.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.438+0000 7fd79e77c640 1 -- 172.21.15.67:0/4075568297 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7980d2af0 con 0x7fd7980b6170 2024-03-24T00:46:17.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.439+0000 7fd79e77c640 1 --2- 172.21.15.67:0/4075568297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7980b6170 0x7fd7980d3910 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fd790009d90 tx=0x7fd79002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83f8dd7c7f490e67 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:17.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.440+0000 7fd79cf79640 1 -- 172.21.15.67:0/4075568297 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd790039070 con 0x7fd7980b6170 2024-03-24T00:46:17.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.440+0000 7fd79cf79640 1 -- 172.21.15.67:0/4075568297 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd79002cbd0 con 0x7fd7980b6170 2024-03-24T00:46:17.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.440+0000 7fd79cf79640 1 -- 172.21.15.67:0/4075568297 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd790034440 con 0x7fd7980b6170 2024-03-24T00:46:17.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.440+0000 7fd7a01df640 1 -- 172.21.15.67:0/4075568297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7980b6170 msgr2=0x7fd7980d3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:17.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.440+0000 7fd7a01df640 1 --2- 172.21.15.67:0/4075568297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7980b6170 0x7fd7980d3910 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fd790009d90 tx=0x7fd79002be20 comp rx=0 tx=0).stop 2024-03-24T00:46:17.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.440+0000 7fd7a01df640 1 -- 172.21.15.67:0/4075568297 shutdown_connections 2024-03-24T00:46:17.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.440+0000 7fd7a01df640 1 --2- 172.21.15.67:0/4075568297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7980b6170 0x7fd7980d3910 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.440+0000 7fd7a01df640 1 --2- 172.21.15.67:0/4075568297 >> v2:172.21.15.72:3300/0 conn(0x7fd79815dea0 0x7fd7980d33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.440+0000 7fd7a01df640 1 -- 172.21.15.67:0/4075568297 >> 172.21.15.67:0/4075568297 conn(0x7fd798072da0 msgr2=0x7fd7980d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:17.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.440+0000 7fd7a01df640 1 -- 172.21.15.67:0/4075568297 shutdown_connections 2024-03-24T00:46:17.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.440+0000 7fd7a01df640 1 -- 172.21.15.67:0/4075568297 wait complete. 2024-03-24T00:46:17.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.442+0000 7fd7a01df640 1 Processor -- start 2024-03-24T00:46:17.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.442+0000 7fd7a01df640 1 -- start start 2024-03-24T00:46:17.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.443+0000 7fd7a01df640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd7980b6170 0x7fd798161280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:17.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.443+0000 7fd7a01df640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd79815dea0 0x7fd7980aa990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:17.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.443+0000 7fd7a01df640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd79815e280 0x7fd7981f2d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:17.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.443+0000 7fd7a01df640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7980b7480 con 0x7fd79815e280 2024-03-24T00:46:17.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.443+0000 7fd7a01df640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7980b7300 con 0x7fd79815dea0 2024-03-24T00:46:17.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.443+0000 7fd7a01df640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7980b7600 con 0x7fd7980b6170 2024-03-24T00:46:17.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.443+0000 7fd79e77c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd79815e280 0x7fd7981f2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:17.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.443+0000 7fd79e77c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd79815e280 0x7fd7981f2d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55126/0 (socket says 172.21.15.67:55126) 2024-03-24T00:46:17.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.443+0000 7fd79e77c640 1 -- 172.21.15.67:0/247570518 learned_addr learned my addr 172.21.15.67:0/247570518 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:17.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.443+0000 7fd79d77a640 1 --2- 172.21.15.67:0/247570518 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd79815dea0 0x7fd7980aa990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:17.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.443+0000 7fd79df7b640 1 --2- 172.21.15.67:0/247570518 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd7980b6170 0x7fd798161280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:17.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.444+0000 7fd79e77c640 1 -- 172.21.15.67:0/247570518 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd7980b6170 msgr2=0x7fd798161280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:17.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.444+0000 7fd79e77c640 1 --2- 172.21.15.67:0/247570518 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd7980b6170 0x7fd798161280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.444+0000 7fd79e77c640 1 -- 172.21.15.67:0/247570518 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd79815dea0 msgr2=0x7fd7980aa990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:17.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.444+0000 7fd79e77c640 1 --2- 172.21.15.67:0/247570518 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd79815dea0 0x7fd7980aa990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.444+0000 7fd79e77c640 1 -- 172.21.15.67:0/247570518 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7980ab100 con 0x7fd79815e280 2024-03-24T00:46:17.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.444+0000 7fd79e77c640 1 --2- 172.21.15.67:0/247570518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd79815e280 0x7fd7981f2d00 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fd790009d90 tx=0x7fd790004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:17.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.444+0000 7fd78e7fc640 1 -- 172.21.15.67:0/247570518 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd790039050 con 0x7fd79815e280 2024-03-24T00:46:17.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.444+0000 7fd78e7fc640 1 -- 172.21.15.67:0/247570518 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd790045070 con 0x7fd79815e280 2024-03-24T00:46:17.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.444+0000 7fd78e7fc640 1 -- 172.21.15.67:0/247570518 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd790009450 con 0x7fd79815e280 2024-03-24T00:46:17.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.444+0000 7fd7a01df640 1 -- 172.21.15.67:0/247570518 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7980ab3e0 con 0x7fd79815e280 2024-03-24T00:46:17.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.444+0000 7fd7a01df640 1 -- 172.21.15.67:0/247570518 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7980ab9a0 con 0x7fd79815e280 2024-03-24T00:46:17.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.445+0000 7fd78e7fc640 1 -- 172.21.15.67:0/247570518 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd79003e850 con 0x7fd79815e280 2024-03-24T00:46:17.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.446+0000 7fd78e7fc640 1 --2- 172.21.15.67:0/247570518 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd77c06bbd0 0x7fd77c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:17.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.446+0000 7fd78e7fc640 1 -- 172.21.15.67:0/247570518 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 6303+0+0 (secure 0 0 0) 0x7fd7900ad7e0 con 0x7fd79815e280 2024-03-24T00:46:17.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.446+0000 7fd79df7b640 1 --2- 172.21.15.67:0/247570518 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd77c06bbd0 0x7fd77c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:17.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.447+0000 7fd79df7b640 1 --2- 172.21.15.67:0/247570518 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd77c06bbd0 0x7fd77c06e090 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fd794009b40 tx=0x7fd794009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:17.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.447+0000 7fd7a2dc7740 1 Processor -- start 2024-03-24T00:46:17.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.447+0000 7fd7a2dc7740 1 -- start start 2024-03-24T00:46:17.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.448+0000 7fd7a2dc7740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f3bbee1e50 0x55f3bbeb4df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:17.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.448+0000 7fd7a2dc7740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f3bbee2230 0x55f3bbee4be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:17.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.448+0000 7fd7a2dc7740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f3bbeb5330 0x55f3bbee5120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:17.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.448+0000 7fd7a2dc7740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3bbe1f3d0 con 0x55f3bbee2230 2024-03-24T00:46:17.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.448+0000 7fd7a2dc7740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3bbdc7ad0 con 0x55f3bbeb5330 2024-03-24T00:46:17.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.448+0000 7fd7a2dc7740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f3bbe1b470 con 0x55f3bbee1e50 2024-03-24T00:46:17.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.448+0000 7fd79d77a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f3bbee2230 0x55f3bbee4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:17.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.448+0000 7fd79d77a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f3bbee2230 0x55f3bbee4be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55136/0 (socket says 172.21.15.67:55136) 2024-03-24T00:46:17.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.448+0000 7fd79d77a640 1 -- 172.21.15.67:0/2226180633 learned_addr learned my addr 172.21.15.67:0/2226180633 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:17.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.448+0000 7fd79df7b640 1 --2- 172.21.15.67:0/2226180633 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f3bbee1e50 0x55f3bbeb4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:17.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.448+0000 7fd79e77c640 1 --2- 172.21.15.67:0/2226180633 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f3bbeb5330 0x55f3bbee5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:17.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.448+0000 7fd79d77a640 1 -- 172.21.15.67:0/2226180633 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f3bbee1e50 msgr2=0x55f3bbeb4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:17.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.448+0000 7fd79d77a640 1 --2- 172.21.15.67:0/2226180633 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f3bbee1e50 0x55f3bbeb4df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.448+0000 7fd79d77a640 1 -- 172.21.15.67:0/2226180633 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f3bbeb5330 msgr2=0x55f3bbee5120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:17.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.448+0000 7fd79d77a640 1 --2- 172.21.15.67:0/2226180633 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f3bbeb5330 0x55f3bbee5120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.448+0000 7fd79d77a640 1 -- 172.21.15.67:0/2226180633 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f3bbbfd7d0 con 0x55f3bbee2230 2024-03-24T00:46:17.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.448+0000 7fd79e77c640 1 --2- 172.21.15.67:0/2226180633 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f3bbeb5330 0x55f3bbee5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:46:17.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.448+0000 7fd79d77a640 1 --2- 172.21.15.67:0/2226180633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f3bbee2230 0x55f3bbee4be0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7fd788000f60 tx=0x7fd788000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eee2b91a9245d855 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:17.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.449+0000 7fd77b7fe640 1 -- 172.21.15.67:0/2226180633 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd788005ac0 con 0x55f3bbee2230 2024-03-24T00:46:17.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.449+0000 7fd77b7fe640 1 -- 172.21.15.67:0/2226180633 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd788018070 con 0x55f3bbee2230 2024-03-24T00:46:17.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.449+0000 7fd77b7fe640 1 -- 172.21.15.67:0/2226180633 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd788005e60 con 0x55f3bbee2230 2024-03-24T00:46:17.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.449+0000 7fd7a2dc7740 1 -- 172.21.15.67:0/2226180633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f3bbee2230 msgr2=0x55f3bbee4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:17.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.449+0000 7fd7a2dc7740 1 --2- 172.21.15.67:0/2226180633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f3bbee2230 0x55f3bbee4be0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7fd788000f60 tx=0x7fd788000bc0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.449+0000 7fd7a2dc7740 1 -- 172.21.15.67:0/2226180633 shutdown_connections 2024-03-24T00:46:17.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.449+0000 7fd7a2dc7740 1 --2- 172.21.15.67:0/2226180633 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f3bbeb5330 0x55f3bbee5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.449+0000 7fd7a2dc7740 1 --2- 172.21.15.67:0/2226180633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f3bbee2230 0x55f3bbee4be0 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.449+0000 7fd7a2dc7740 1 --2- 172.21.15.67:0/2226180633 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f3bbee1e50 0x55f3bbeb4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.449+0000 7fd7a2dc7740 1 -- 172.21.15.67:0/2226180633 >> 172.21.15.67:0/2226180633 conn(0x55f3bbeb7060 msgr2=0x55f3bbeffbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:17.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.449+0000 7fd7a2dc7740 1 -- 172.21.15.67:0/2226180633 shutdown_connections 2024-03-24T00:46:17.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.449+0000 7fd7a2dc7740 1 -- 172.21.15.67:0/2226180633 wait complete. 2024-03-24T00:46:17.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.450+0000 7fd7a2dc7740 1 -- start start 2024-03-24T00:46:17.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.450+0000 7fd7a2dc7740 1 Processor -- start 2024-03-24T00:46:17.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.451+0000 7fd7a2dc7740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f3bbee1e50 0x55f3bbee9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:17.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.451+0000 7fd7a2dc7740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f3bbee2230 0x55f3bbeeb780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:17.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.451+0000 7fd7a2dc7740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f3bbee8be0 0x55f3bbedf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:17.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.451+0000 7fd7a2dc7740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3bbeb9700 con 0x55f3bbee2230 2024-03-24T00:46:17.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.451+0000 7fd7a2dc7740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3bbddb060 con 0x55f3bbee1e50 2024-03-24T00:46:17.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.451+0000 7fd7a2dc7740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f3bbebbf40 con 0x55f3bbee8be0 2024-03-24T00:46:17.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.451+0000 7fd7a2dc7740 10 client.0 _ll_register_callbacks cb 0x55f3bbdbd6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:46:17.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.451+0000 7fd79d77a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f3bbee2230 0x55f3bbeeb780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:17.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.451+0000 7fd79d77a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f3bbee2230 0x55f3bbeeb780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55152/0 (socket says 172.21.15.67:55152) 2024-03-24T00:46:17.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.451+0000 7fd79d77a640 1 -- 172.21.15.67:0/275367935 learned_addr learned my addr 172.21.15.67:0/275367935 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:17.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.451+0000 7fd79e77c640 1 --2- 172.21.15.67:0/275367935 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f3bbee8be0 0x55f3bbedf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:17.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.451+0000 7fd79df7b640 1 --2- 172.21.15.67:0/275367935 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f3bbee1e50 0x55f3bbee9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:17.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.451+0000 7fd79d77a640 1 -- 172.21.15.67:0/275367935 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f3bbee8be0 msgr2=0x55f3bbedf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:17.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.451+0000 7fd79d77a640 1 --2- 172.21.15.67:0/275367935 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f3bbee8be0 0x55f3bbedf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.451+0000 7fd79d77a640 1 -- 172.21.15.67:0/275367935 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f3bbee1e50 msgr2=0x55f3bbee9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:17.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.451+0000 7fd79d77a640 1 --2- 172.21.15.67:0/275367935 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f3bbee1e50 0x55f3bbee9ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.451+0000 7fd79d77a640 1 -- 172.21.15.67:0/275367935 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f3bbb333f0 con 0x55f3bbee2230 2024-03-24T00:46:17.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.451+0000 7fd79d77a640 1 --2- 172.21.15.67:0/275367935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f3bbee2230 0x55f3bbeeb780 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7fd788009010 tx=0x7fd7880010a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dd4446bfe60d488 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:17.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.452+0000 7fd778ff9640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:46:17.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.452+0000 7fd778ff9640 1 -- 172.21.15.67:0/275367935 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7880072d0 con 0x55f3bbee2230 2024-03-24T00:46:17.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.452+0000 7fd778ff9640 1 -- 172.21.15.67:0/275367935 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 6303+0+0 (secure 0 0 0) 0x7fd788028d30 con 0x55f3bbee2230 2024-03-24T00:46:17.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.452+0000 7fd7a2dc7740 1 -- 172.21.15.67:0/275367935 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd788018070 con 0x55f3bbee2230 2024-03-24T00:46:17.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.452+0000 7fd778ff9640 1 -- 172.21.15.67:0/275367935 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd78802f050 con 0x55f3bbee2230 2024-03-24T00:46:17.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.452+0000 7fd7a2dc7740 10 client.8076 fetch_fsmap learned FSMap version 61 2024-03-24T00:46:17.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.452+0000 7fd7a2dc7740 1 -- 172.21.15.67:0/275367935 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=2+}) v3 -- 0x55f3bbaf30f0 con 0x55f3bbee2230 2024-03-24T00:46:17.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.452+0000 7fd778ff9640 1 -- 172.21.15.67:0/275367935 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd788018070 con 0x55f3bbee2230 2024-03-24T00:46:17.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.452+0000 7fd778ff9640 1 -- 172.21.15.67:0/275367935 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 61) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fd788032020 con 0x55f3bbee2230 2024-03-24T00:46:17.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.452+0000 7fd7a2dc7740 10 client.8076 fetch_fsmap finished waiting for FSMap version 61 2024-03-24T00:46:17.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.452+0000 7fd7a2dc7740 10 client.8076 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-03-24T00:46:17.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.452+0000 7fd7a2dc7740 20 client.8076 populate_metadata read hostname 'smithi067' 2024-03-24T00:46:17.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.452+0000 7fd7a2dc7740 1 --2- 172.21.15.67:0/275367935 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x55f3bbf0fb40 0x55f3bbee6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:17.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.452+0000 7fd7a2dc7740 4 client.8076 mds_command: new command op to 7860 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:46:17.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.452+0000 7fd7a2dc7740 1 -- 172.21.15.67:0/275367935 --> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f3bbe2c910 con 0x55f3bbf0fb40 2024-03-24T00:46:17.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.452+0000 7fd79df7b640 1 --2- 172.21.15.67:0/275367935 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x55f3bbf0fb40 0x55f3bbee6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:17.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.453+0000 7fd778ff9640 10 client.8076 ms_handle_connect on v2:172.21.15.67:6832/1336395002 2024-03-24T00:46:17.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.453+0000 7fd79df7b640 1 --2- 172.21.15.67:0/275367935 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x55f3bbf0fb40 0x55f3bbee6cc0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=850a26c1a6c68c71 server_cookie=56f74eb6b0c5c5b4 in_seq=0 out_seq=0 2024-03-24T00:46:17.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.453+0000 7fd778ff9640 1 -- 172.21.15.67:0/275367935 <== mds.0 v2:172.21.15.67:6832/1336395002 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f3bbe2c910 con 0x55f3bbf0fb40 2024-03-24T00:46:17.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.453+0000 7fd778ff9640 10 client.8076 handle_command_reply: tid=0 2024-03-24T00:46:17.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.453+0000 7fd7a2dc7740 1 client.8076 shutdown 2024-03-24T00:46:17.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.454+0000 7fd7a2dc7740 1 -- 172.21.15.67:0/275367935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f3bbee2230 msgr2=0x55f3bbeeb780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:17.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.454+0000 7fd7a2dc7740 1 --2- 172.21.15.67:0/275367935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f3bbee2230 0x55f3bbeeb780 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7fd788009010 tx=0x7fd7880010a0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.454+0000 7fd7a2dc7740 1 -- 172.21.15.67:0/275367935 shutdown_connections 2024-03-24T00:46:17.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.454+0000 7fd7a2dc7740 1 --2- 172.21.15.67:0/275367935 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x55f3bbf0fb40 0x55f3bbee6cc0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.454+0000 7fd7a2dc7740 1 --2- 172.21.15.67:0/275367935 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f3bbee8be0 0x55f3bbedf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.454+0000 7fd7a2dc7740 1 --2- 172.21.15.67:0/275367935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f3bbee2230 0x55f3bbeeb780 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.454+0000 7fd778ff9640 0 client.8076 ms_handle_reset on v2:172.21.15.67:6832/1336395002 2024-03-24T00:46:17.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.454+0000 7fd7a2dc7740 1 --2- 172.21.15.67:0/275367935 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f3bbee1e50 0x55f3bbee9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.454+0000 7fd7a2dc7740 1 -- 172.21.15.67:0/275367935 >> 172.21.15.67:0/275367935 conn(0x55f3bbeb5330 msgr2=0x55f3bbeb8dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:17.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.454+0000 7fd7a2dc7740 1 -- 172.21.15.67:0/275367935 shutdown_connections 2024-03-24T00:46:17.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.454+0000 7fd7a2dc7740 1 -- 172.21.15.67:0/275367935 wait complete. 2024-03-24T00:46:17.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.454+0000 7fd7a2dc7740 20 client.8076 trim_cache size 0 max 16384 2024-03-24T00:46:17.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.470+0000 7fd7a2dc7740 1 Processor -- start 2024-03-24T00:46:17.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.470+0000 7fd7a2dc7740 1 -- start start 2024-03-24T00:46:17.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.471+0000 7fd7a2dc7740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f3bbee1e50 0x55f3bbee7290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:17.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.471+0000 7fd7a2dc7740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f3bbee2230 0x55f3bbee77d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:17.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.471+0000 7fd7a2dc7740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f3bbee8be0 0x55f3bbeb9a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:17.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.471+0000 7fd7a2dc7740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd798200930 con 0x55f3bbee8be0 2024-03-24T00:46:17.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.471+0000 7fd7a2dc7740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7981ffa30 con 0x55f3bbee1e50 2024-03-24T00:46:17.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.471+0000 7fd7a2dc7740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7981fee60 con 0x55f3bbee2230 2024-03-24T00:46:17.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.471+0000 7fd79e77c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f3bbee8be0 0x55f3bbeb9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:17.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.471+0000 7fd79e77c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f3bbee8be0 0x55f3bbeb9a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55162/0 (socket says 172.21.15.67:55162) 2024-03-24T00:46:17.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.471+0000 7fd79e77c640 1 -- 172.21.15.67:0/3729417446 learned_addr learned my addr 172.21.15.67:0/3729417446 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:17.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.471+0000 7fd79d77a640 1 --2- 172.21.15.67:0/3729417446 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f3bbee2230 0x55f3bbee77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:17.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.471+0000 7fd79df7b640 1 --2- 172.21.15.67:0/3729417446 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f3bbee1e50 0x55f3bbee7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:17.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.471+0000 7fd79e77c640 1 -- 172.21.15.67:0/3729417446 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f3bbee2230 msgr2=0x55f3bbee77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:17.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.471+0000 7fd79e77c640 1 --2- 172.21.15.67:0/3729417446 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f3bbee2230 0x55f3bbee77d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.471+0000 7fd79e77c640 1 -- 172.21.15.67:0/3729417446 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f3bbee1e50 msgr2=0x55f3bbee7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:17.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.471+0000 7fd79e77c640 1 --2- 172.21.15.67:0/3729417446 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f3bbee1e50 0x55f3bbee7290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.471+0000 7fd79e77c640 1 -- 172.21.15.67:0/3729417446 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f3bbeea260 con 0x55f3bbee8be0 2024-03-24T00:46:17.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.471+0000 7fd79d77a640 1 --2- 172.21.15.67:0/3729417446 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f3bbee2230 0x55f3bbee77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:46:17.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.471+0000 7fd79e77c640 1 --2- 172.21.15.67:0/3729417446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f3bbee8be0 0x55f3bbeb9a90 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fd790078040 tx=0x7fd7900ad040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b417f4fbd14c8ac4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:17.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.471+0000 7fd77affd640 1 -- 172.21.15.67:0/3729417446 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd79007b540 con 0x55f3bbee8be0 2024-03-24T00:46:17.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.471+0000 7fd77affd640 1 -- 172.21.15.67:0/3729417446 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd79003a040 con 0x55f3bbee8be0 2024-03-24T00:46:17.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.471+0000 7fd77affd640 1 -- 172.21.15.67:0/3729417446 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd79002cbd0 con 0x55f3bbee8be0 2024-03-24T00:46:17.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.472+0000 7fd7a2dc7740 1 -- 172.21.15.67:0/3729417446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f3bbee8be0 msgr2=0x55f3bbeb9a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:17.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.472+0000 7fd7a2dc7740 1 --2- 172.21.15.67:0/3729417446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f3bbee8be0 0x55f3bbeb9a90 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fd790078040 tx=0x7fd7900ad040 comp rx=0 tx=0).stop 2024-03-24T00:46:17.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.472+0000 7fd7a2dc7740 1 -- 172.21.15.67:0/3729417446 shutdown_connections 2024-03-24T00:46:17.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.472+0000 7fd7a2dc7740 1 --2- 172.21.15.67:0/3729417446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f3bbee8be0 0x55f3bbeb9a90 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.472+0000 7fd7a2dc7740 1 --2- 172.21.15.67:0/3729417446 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f3bbee2230 0x55f3bbee77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.472+0000 7fd7a2dc7740 1 --2- 172.21.15.67:0/3729417446 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f3bbee1e50 0x55f3bbee7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.472+0000 7fd7a2dc7740 1 -- 172.21.15.67:0/3729417446 >> 172.21.15.67:0/3729417446 conn(0x55f3bbeb5330 msgr2=0x55f3bbee6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:17.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.472+0000 7fd7a2dc7740 1 -- 172.21.15.67:0/3729417446 shutdown_connections 2024-03-24T00:46:17.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.472+0000 7fd7a2dc7740 1 -- 172.21.15.67:0/3729417446 wait complete. 2024-03-24T00:46:17.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.473+0000 7fd7a2dc7740 1 -- start start 2024-03-24T00:46:17.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.473+0000 7fd7a2dc7740 1 Processor -- start 2024-03-24T00:46:17.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.474+0000 7fd7a2dc7740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f3bbee1e50 0x55f3bbee7290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:17.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.474+0000 7fd7a2dc7740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f3bbee2230 0x55f3bbee77d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:17.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.474+0000 7fd7a2dc7740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f3bbee8be0 0x55f3bbeb9d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:17.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.474+0000 7fd7a2dc7740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7981ff430 con 0x55f3bbee2230 2024-03-24T00:46:17.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.474+0000 7fd7a2dc7740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3bbf02fe0 con 0x55f3bbee1e50 2024-03-24T00:46:17.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.474+0000 7fd7a2dc7740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f3bbeb43a0 con 0x55f3bbee8be0 2024-03-24T00:46:17.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.474+0000 7fd7a2dc7740 10 client.0 _ll_register_callbacks cb 0x7fd7940095d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:46:17.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.474+0000 7fd79d77a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f3bbee2230 0x55f3bbee77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:17.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.474+0000 7fd79d77a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f3bbee2230 0x55f3bbee77d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55176/0 (socket says 172.21.15.67:55176) 2024-03-24T00:46:17.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.474+0000 7fd79d77a640 1 -- 172.21.15.67:0/3329733747 learned_addr learned my addr 172.21.15.67:0/3329733747 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:17.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.474+0000 7fd79df7b640 1 --2- 172.21.15.67:0/3329733747 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f3bbee1e50 0x55f3bbee7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:17.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.474+0000 7fd79e77c640 1 --2- 172.21.15.67:0/3329733747 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f3bbee8be0 0x55f3bbeb9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:17.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.474+0000 7fd79d77a640 1 -- 172.21.15.67:0/3329733747 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f3bbee8be0 msgr2=0x55f3bbeb9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:17.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.474+0000 7fd79d77a640 1 --2- 172.21.15.67:0/3329733747 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f3bbee8be0 0x55f3bbeb9d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.474+0000 7fd79d77a640 1 -- 172.21.15.67:0/3329733747 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f3bbee1e50 msgr2=0x55f3bbee7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:17.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.474+0000 7fd79d77a640 1 --2- 172.21.15.67:0/3329733747 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f3bbee1e50 0x55f3bbee7290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.474+0000 7fd79d77a640 1 -- 172.21.15.67:0/3329733747 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f3bbedbed0 con 0x55f3bbee2230 2024-03-24T00:46:17.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.474+0000 7fd79d77a640 1 --2- 172.21.15.67:0/3329733747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f3bbee2230 0x55f3bbee77d0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7fd788009010 tx=0x7fd788009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53d0e30b9e91e065 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:17.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.474+0000 7fd778ff9640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:46:17.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.474+0000 7fd778ff9640 1 -- 172.21.15.67:0/3329733747 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd788004bf0 con 0x55f3bbee2230 2024-03-24T00:46:17.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.474+0000 7fd778ff9640 1 -- 172.21.15.67:0/3329733747 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 6303+0+0 (secure 0 0 0) 0x7fd788004d90 con 0x55f3bbee2230 2024-03-24T00:46:17.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.474+0000 7fd7a2dc7740 1 -- 172.21.15.67:0/3329733747 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd79003a040 con 0x55f3bbee2230 2024-03-24T00:46:17.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.474+0000 7fd778ff9640 1 -- 172.21.15.67:0/3329733747 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd788007740 con 0x55f3bbee2230 2024-03-24T00:46:17.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.475+0000 7fd7a2dc7740 10 client.8082 fetch_fsmap learned FSMap version 61 2024-03-24T00:46:17.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.475+0000 7fd7a2dc7740 1 -- 172.21.15.67:0/3329733747 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=2+}) v3 -- 0x55f3bbeb3ca0 con 0x55f3bbee2230 2024-03-24T00:46:17.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.475+0000 7fd778ff9640 1 -- 172.21.15.67:0/3329733747 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd79003a040 con 0x55f3bbee2230 2024-03-24T00:46:17.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.475+0000 7fd778ff9640 1 -- 172.21.15.67:0/3329733747 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 61) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fd788032040 con 0x55f3bbee2230 2024-03-24T00:46:17.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.475+0000 7fd7a2dc7740 10 client.8082 fetch_fsmap finished waiting for FSMap version 61 2024-03-24T00:46:17.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.475+0000 7fd7a2dc7740 10 client.8082 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-03-24T00:46:17.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.475+0000 7fd7a2dc7740 20 client.8082 populate_metadata read hostname 'smithi067' 2024-03-24T00:46:17.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.475+0000 7fd7a2dc7740 1 --2- 172.21.15.67:0/3329733747 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x55f3bbf6dd60 0x55f3bbeba250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:17.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.475+0000 7fd7a2dc7740 4 client.8082 mds_command: new command op to 7860 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:46:17.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.475+0000 7fd7a2dc7740 1 -- 172.21.15.67:0/3329733747 --> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55f3bbe26c60 con 0x55f3bbf6dd60 2024-03-24T00:46:17.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.475+0000 7fd79df7b640 1 --2- 172.21.15.67:0/3329733747 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x55f3bbf6dd60 0x55f3bbeba250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:17.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.475+0000 7fd778ff9640 10 client.8082 ms_handle_connect on v2:172.21.15.67:6832/1336395002 2024-03-24T00:46:17.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.475+0000 7fd79df7b640 1 --2- 172.21.15.67:0/3329733747 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x55f3bbf6dd60 0x55f3bbeba250 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66b1643ef594e8cb server_cookie=487b4cdc0bfa1c8a in_seq=0 out_seq=0 2024-03-24T00:46:17.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.475+0000 7fd778ff9640 1 -- 172.21.15.67:0/3329733747 <== mds.0 v2:172.21.15.67:6832/1336395002 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55f3bbe26c60 con 0x55f3bbf6dd60 2024-03-24T00:46:17.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.475+0000 7fd778ff9640 10 client.8082 handle_command_reply: tid=0 2024-03-24T00:46:17.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.476+0000 7fd7a2dc7740 1 client.8082 shutdown 2024-03-24T00:46:17.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.476+0000 7fd7a2dc7740 1 -- 172.21.15.67:0/3329733747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f3bbee2230 msgr2=0x55f3bbee77d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:17.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.476+0000 7fd7a2dc7740 1 --2- 172.21.15.67:0/3329733747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f3bbee2230 0x55f3bbee77d0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7fd788009010 tx=0x7fd788009040 comp rx=0 tx=0).stop 2024-03-24T00:46:17.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.476+0000 7fd7a2dc7740 1 -- 172.21.15.67:0/3329733747 shutdown_connections 2024-03-24T00:46:17.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.476+0000 7fd7a2dc7740 1 --2- 172.21.15.67:0/3329733747 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x55f3bbf6dd60 0x55f3bbeba250 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.476+0000 7fd7a2dc7740 1 --2- 172.21.15.67:0/3329733747 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55f3bbee8be0 0x55f3bbeb9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.476+0000 7fd7a2dc7740 1 --2- 172.21.15.67:0/3329733747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f3bbee2230 0x55f3bbee77d0 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.476+0000 7fd778ff9640 0 client.8082 ms_handle_reset on v2:172.21.15.67:6832/1336395002 2024-03-24T00:46:17.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.476+0000 7fd7a2dc7740 1 --2- 172.21.15.67:0/3329733747 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f3bbee1e50 0x55f3bbee7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.476+0000 7fd7a2dc7740 1 -- 172.21.15.67:0/3329733747 >> 172.21.15.67:0/3329733747 conn(0x55f3bbeb5330 msgr2=0x55f3bbeea920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:17.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.476+0000 7fd7a2dc7740 1 -- 172.21.15.67:0/3329733747 shutdown_connections 2024-03-24T00:46:17.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.476+0000 7fd7a2dc7740 1 -- 172.21.15.67:0/3329733747 wait complete. 2024-03-24T00:46:17.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.476+0000 7fd7a2dc7740 20 client.8082 trim_cache size 0 max 16384 2024-03-24T00:46:17.483 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:46:17.483 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "no active scrubs running", 2024-03-24T00:46:17.483 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": {} 2024-03-24T00:46:17.483 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:46:17.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.476+0000 7fd7a01df640 1 -- 172.21.15.67:0/247570518 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd77c06bbd0 msgr2=0x7fd77c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:17.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.476+0000 7fd7a01df640 1 --2- 172.21.15.67:0/247570518 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd77c06bbd0 0x7fd77c06e090 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fd794009b40 tx=0x7fd794009190 comp rx=0 tx=0).stop 2024-03-24T00:46:17.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.476+0000 7fd7a01df640 1 -- 172.21.15.67:0/247570518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd79815e280 msgr2=0x7fd7981f2d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:17.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.476+0000 7fd7a01df640 1 --2- 172.21.15.67:0/247570518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd79815e280 0x7fd7981f2d00 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fd790009d90 tx=0x7fd790004a40 comp rx=0 tx=0).stop 2024-03-24T00:46:17.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.477+0000 7fd7a01df640 1 -- 172.21.15.67:0/247570518 shutdown_connections 2024-03-24T00:46:17.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.477+0000 7fd7a01df640 1 --2- 172.21.15.67:0/247570518 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd77c06bbd0 0x7fd77c06e090 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.477+0000 7fd7a01df640 1 --2- 172.21.15.67:0/247570518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd79815e280 0x7fd7981f2d00 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.477+0000 7fd7a01df640 1 --2- 172.21.15.67:0/247570518 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd79815dea0 0x7fd7980aa990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.477+0000 7fd7a01df640 1 --2- 172.21.15.67:0/247570518 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd7980b6170 0x7fd798161280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.477+0000 7fd7a01df640 1 -- 172.21.15.67:0/247570518 >> 172.21.15.67:0/247570518 conn(0x7fd798072da0 msgr2=0x7fd7980b7db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:17.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.477+0000 7fd7a01df640 1 -- 172.21.15.67:0/247570518 shutdown_connections 2024-03-24T00:46:17.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.477+0000 7fd7a01df640 1 -- 172.21.15.67:0/247570518 wait complete. 2024-03-24T00:46:17.488 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T00:46:17.489 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:46:17.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.590+0000 7f89f053b640 1 Processor -- start 2024-03-24T00:46:17.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.590+0000 7f89f053b640 1 -- start start 2024-03-24T00:46:17.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.591+0000 7f89f053b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f89e815dec0 0x7f89e80ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:17.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.591+0000 7f89f053b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e80cf640 con 0x7f89e815e2a0 2024-03-24T00:46:17.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.591+0000 7f89f053b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e80cf080 con 0x7f89e80d0ef0 2024-03-24T00:46:17.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.591+0000 7f89f053b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89e80cee30 con 0x7f89e815dec0 2024-03-24T00:46:17.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.591+0000 7f89ee2d7640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f89e815dec0 0x7f89e80ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:17.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.591+0000 7f89edad6640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f89e815e2a0 0x7f89e80d07e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59416/0 (socket says 172.21.15.67:59416) 2024-03-24T00:46:17.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.591+0000 7f89edad6640 1 -- 172.21.15.67:0/949736513 learned_addr learned my addr 172.21.15.67:0/949736513 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:46:17.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.591+0000 7f89ee2d7640 1 -- 172.21.15.67:0/949736513 >> v1:172.21.15.72:6789/0 conn(0x7f89e80d0ef0 legacy=0x7f89e80d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:17.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.591+0000 7f89ee2d7640 1 -- 172.21.15.67:0/949736513 >> v1:172.21.15.67:6789/0 conn(0x7f89e815e2a0 legacy=0x7f89e80d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:17.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.591+0000 7f89ee2d7640 1 -- 172.21.15.67:0/949736513 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e80cf360 con 0x7f89e815dec0 2024-03-24T00:46:17.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.591+0000 7f89ed2d5640 1 -- 172.21.15.67:0/949736513 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 80723098 0 0) 0x7f89d8001110 con 0x7f89e815e2a0 2024-03-24T00:46:17.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.592+0000 7f89ee2d7640 1 --2- 172.21.15.67:0/949736513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89e815dec0 0x7f89e80ce630 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f89e80cefb0 tx=0x7f89e402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f8f20b9fd205e71 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:17.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.592+0000 7f89ed2d5640 1 -- 172.21.15.67:0/949736513 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e40315a0 con 0x7f89e815dec0 2024-03-24T00:46:17.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.592+0000 7f89ed2d5640 1 -- 172.21.15.67:0/949736513 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89e4037070 con 0x7f89e815dec0 2024-03-24T00:46:17.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.592+0000 7f89ed2d5640 1 -- 172.21.15.67:0/949736513 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e4031960 con 0x7f89e815dec0 2024-03-24T00:46:17.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.592+0000 7f89f053b640 1 -- 172.21.15.67:0/949736513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89e815dec0 msgr2=0x7f89e80ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:17.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.592+0000 7f89f053b640 1 --2- 172.21.15.67:0/949736513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89e815dec0 0x7f89e80ce630 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f89e80cefb0 tx=0x7f89e402b840 comp rx=0 tx=0).stop 2024-03-24T00:46:17.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.593+0000 7f89f053b640 1 -- 172.21.15.67:0/949736513 shutdown_connections 2024-03-24T00:46:17.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.593+0000 7f89f053b640 1 --2- 172.21.15.67:0/949736513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89e815dec0 0x7f89e80ce630 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.593+0000 7f89f053b640 1 -- 172.21.15.67:0/949736513 >> 172.21.15.67:0/949736513 conn(0x7f89e806c810 msgr2=0x7f89e806cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:17.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.593+0000 7f89f053b640 1 -- 172.21.15.67:0/949736513 shutdown_connections 2024-03-24T00:46:17.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.593+0000 7f89f053b640 1 -- 172.21.15.67:0/949736513 wait complete. 2024-03-24T00:46:17.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.594+0000 7f89f053b640 1 Processor -- start 2024-03-24T00:46:17.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.594+0000 7f89f053b640 1 -- start start 2024-03-24T00:46:17.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.594+0000 7f89f053b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89e80d0ef0 0x7f89e8158f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:17.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.594+0000 7f89f053b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89e815dec0 0x7f89e80b5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:17.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.595+0000 7f89f053b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f89e815e2a0 0x7f89e815abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:17.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.595+0000 7f89f053b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e8162a40 con 0x7f89e815dec0 2024-03-24T00:46:17.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.595+0000 7f89f053b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e81628c0 con 0x7f89e80d0ef0 2024-03-24T00:46:17.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.595+0000 7f89f053b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89e80d2120 con 0x7f89e815e2a0 2024-03-24T00:46:17.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.595+0000 7f89edad6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89e815dec0 0x7f89e80b5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:17.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.595+0000 7f89edad6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89e815dec0 0x7f89e80b5660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55192/0 (socket says 172.21.15.67:55192) 2024-03-24T00:46:17.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.595+0000 7f89edad6640 1 -- 172.21.15.67:0/2373623267 learned_addr learned my addr 172.21.15.67:0/2373623267 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:17.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.595+0000 7f89edad6640 1 -- 172.21.15.67:0/2373623267 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f89e815e2a0 msgr2=0x7f89e815abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:17.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.595+0000 7f89edad6640 1 --2- 172.21.15.67:0/2373623267 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f89e815e2a0 0x7f89e815abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.595+0000 7f89edad6640 1 -- 172.21.15.67:0/2373623267 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89e80d0ef0 msgr2=0x7f89e8158f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:17.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.595+0000 7f89edad6640 1 --2- 172.21.15.67:0/2373623267 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89e80d0ef0 0x7f89e8158f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.595+0000 7f89edad6640 1 -- 172.21.15.67:0/2373623267 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e81e89e0 con 0x7f89e815dec0 2024-03-24T00:46:17.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.595+0000 7f89edad6640 1 --2- 172.21.15.67:0/2373623267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89e815dec0 0x7f89e80b5660 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f89d800c410 tx=0x7f89d800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:17.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.595+0000 7f89deffd640 1 -- 172.21.15.67:0/2373623267 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d80074e0 con 0x7f89e815dec0 2024-03-24T00:46:17.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.595+0000 7f89deffd640 1 -- 172.21.15.67:0/2373623267 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89d80047e0 con 0x7f89e815dec0 2024-03-24T00:46:17.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.595+0000 7f89f053b640 1 -- 172.21.15.67:0/2373623267 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89e81e7e60 con 0x7f89e815dec0 2024-03-24T00:46:17.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.595+0000 7f89deffd640 1 -- 172.21.15.67:0/2373623267 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d8004bd0 con 0x7f89e815dec0 2024-03-24T00:46:17.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.595+0000 7f89f053b640 1 -- 172.21.15.67:0/2373623267 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89e81e8420 con 0x7f89e815dec0 2024-03-24T00:46:17.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.596+0000 7f89f053b640 1 -- 172.21.15.67:0/2373623267 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89e804f400 con 0x7f89e815dec0 2024-03-24T00:46:17.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.597+0000 7f89deffd640 1 -- 172.21.15.67:0/2373623267 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f89d8011070 con 0x7f89e815dec0 2024-03-24T00:46:17.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.597+0000 7f89deffd640 1 --2- 172.21.15.67:0/2373623267 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f89cc06bbd0 0x7f89cc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:17.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.597+0000 7f89deffd640 1 -- 172.21.15.67:0/2373623267 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 6303+0+0 (secure 0 0 0) 0x7f89d808a980 con 0x7f89e815dec0 2024-03-24T00:46:17.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.598+0000 7f89ee2d7640 1 --2- 172.21.15.67:0/2373623267 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f89cc06bbd0 0x7f89cc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:17.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.598+0000 7f89ee2d7640 1 --2- 172.21.15.67:0/2373623267 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f89cc06bbd0 0x7f89cc06e090 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f89e4036000 tx=0x7f89e4030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:17.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.606+0000 7f89deffd640 1 -- 172.21.15.67:0/2373623267 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f89e80cf640 con 0x7f89e815dec0 2024-03-24T00:46:17.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.826+0000 7f89f053b640 1 -- 172.21.15.67:0/2373623267 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89e80d1aa0 con 0x7f89e815dec0 2024-03-24T00:46:17.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.827+0000 7f89deffd640 1 -- 172.21.15.67:0/2373623267 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f89d800e280 con 0x7f89e815dec0 2024-03-24T00:46:17.828 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:46:17.828 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":61,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6359,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2664460392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2664460392},{"type":"v1","addr":"172.21.15.67:6835","nonce":2664460392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6368,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/3323873506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3323873506},{"type":"v1","addr":"172.21.15.72:6837","nonce":3323873506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":7150,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/2744016217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2744016217},{"type":"v1","addr":"172.21.15.72:6836","nonce":2744016217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:44:54.749656+0000","modified":"2024-03-24T00:45:20.120495+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7860},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7860":{"gid":7860,"name":"c","rank":0,"incarnation":57,"state":"up:active","state_seq":6,"addr":"172.21.15.67:6833/1336395002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1336395002},{"type":"v1","addr":"172.21.15.67:6833","nonce":1336395002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7860,"qdb_cluster":[7860]},"id":6}]} 2024-03-24T00:46:17.828 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 61 2024-03-24T00:46:17.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.829+0000 7f89f053b640 1 -- 172.21.15.67:0/2373623267 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f89cc06bbd0 msgr2=0x7f89cc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:17.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.829+0000 7f89f053b640 1 --2- 172.21.15.67:0/2373623267 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f89cc06bbd0 0x7f89cc06e090 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f89e4036000 tx=0x7f89e4030000 comp rx=0 tx=0).stop 2024-03-24T00:46:17.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.829+0000 7f89f053b640 1 -- 172.21.15.67:0/2373623267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89e815dec0 msgr2=0x7f89e80b5660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:17.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.829+0000 7f89f053b640 1 --2- 172.21.15.67:0/2373623267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89e815dec0 0x7f89e80b5660 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f89d800c410 tx=0x7f89d800c8d0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.829+0000 7f89f053b640 1 -- 172.21.15.67:0/2373623267 shutdown_connections 2024-03-24T00:46:17.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.829+0000 7f89f053b640 1 --2- 172.21.15.67:0/2373623267 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f89cc06bbd0 0x7f89cc06e090 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.829+0000 7f89f053b640 1 --2- 172.21.15.67:0/2373623267 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f89e815e2a0 0x7f89e815abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.829+0000 7f89f053b640 1 --2- 172.21.15.67:0/2373623267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89e815dec0 0x7f89e80b5660 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.829+0000 7f89f053b640 1 --2- 172.21.15.67:0/2373623267 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89e80d0ef0 0x7f89e8158f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.829+0000 7f89f053b640 1 -- 172.21.15.67:0/2373623267 >> 172.21.15.67:0/2373623267 conn(0x7f89e806c810 msgr2=0x7f89e8160d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:17.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.829+0000 7f89f053b640 1 -- 172.21.15.67:0/2373623267 shutdown_connections 2024-03-24T00:46:17.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.829+0000 7f89f053b640 1 -- 172.21.15.67:0/2373623267 wait complete. 2024-03-24T00:46:17.841 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c damage ls --format=json-pretty 2024-03-24T00:46:17.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.946+0000 7f73dc3b9640 1 Processor -- start 2024-03-24T00:46:17.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.946+0000 7f73dc3b9640 1 -- start start 2024-03-24T00:46:17.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.947+0000 7f73dc3b9640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f73d415bc30 0x7f73d40b1a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:17.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.947+0000 7f73dc3b9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f73d40b34a0 0x7f73d40b1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:17.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.947+0000 7f73dc3b9640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73d40b3ae0 con 0x7f73d415c010 2024-03-24T00:46:17.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.947+0000 7f73dc3b9640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73d40cca60 con 0x7f73d40b34a0 2024-03-24T00:46:17.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.947+0000 7f73dc3b9640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73d40b3dc0 con 0x7f73d415bc30 2024-03-24T00:46:17.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.947+0000 7f73da956640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f73d40b34a0 0x7f73d40b1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:17.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.947+0000 7f73da956640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f73d40b34a0 0x7f73d40b1f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55208/0 (socket says 172.21.15.67:55208) 2024-03-24T00:46:17.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.947+0000 7f73da956640 1 -- 172.21.15.67:0/3033062295 learned_addr learned my addr 172.21.15.67:0/3033062295 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:17.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.947+0000 7f73da155640 1 --2- 172.21.15.67:0/3033062295 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f73d415bc30 0x7f73d40b1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:17.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.947+0000 7f73da956640 1 -- 172.21.15.67:0/3033062295 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f73d415bc30 msgr2=0x7f73d40b1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:17.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.947+0000 7f73da956640 1 --2- 172.21.15.67:0/3033062295 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f73d415bc30 0x7f73d40b1a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.947+0000 7f73da956640 1 -- 172.21.15.67:0/3033062295 >> v1:172.21.15.72:6789/0 conn(0x7f73d415c010 legacy=0x7f73d40b2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:17.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.947+0000 7f73da956640 1 -- 172.21.15.67:0/3033062295 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73d40b40a0 con 0x7f73d40b34a0 2024-03-24T00:46:17.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.947+0000 7f73da155640 1 --2- 172.21.15.67:0/3033062295 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f73d415bc30 0x7f73d40b1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:46:17.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.948+0000 7f73da956640 1 --2- 172.21.15.67:0/3033062295 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73d40b34a0 0x7f73d40b1f40 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f73cc009e50 tx=0x7f73cc02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbe56cec37125662 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:17.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.948+0000 7f73d9153640 1 -- 172.21.15.67:0/3033062295 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73cc02cb00 con 0x7f73d40b34a0 2024-03-24T00:46:17.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.948+0000 7f73d9153640 1 -- 172.21.15.67:0/3033062295 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73cc02cca0 con 0x7f73d40b34a0 2024-03-24T00:46:17.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.948+0000 7f73d9153640 1 -- 172.21.15.67:0/3033062295 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73cc034610 con 0x7f73d40b34a0 2024-03-24T00:46:17.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.949+0000 7f73dc3b9640 1 -- 172.21.15.67:0/3033062295 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73d40b34a0 msgr2=0x7f73d40b1f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:17.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.949+0000 7f73dc3b9640 1 --2- 172.21.15.67:0/3033062295 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73d40b34a0 0x7f73d40b1f40 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f73cc009e50 tx=0x7f73cc02bf60 comp rx=0 tx=0).stop 2024-03-24T00:46:17.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.949+0000 7f73dc3b9640 1 -- 172.21.15.67:0/3033062295 shutdown_connections 2024-03-24T00:46:17.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.949+0000 7f73dc3b9640 1 --2- 172.21.15.67:0/3033062295 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73d40b34a0 0x7f73d40b1f40 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.949+0000 7f73dc3b9640 1 --2- 172.21.15.67:0/3033062295 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f73d415bc30 0x7f73d40b1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.949+0000 7f73dc3b9640 1 -- 172.21.15.67:0/3033062295 >> 172.21.15.67:0/3033062295 conn(0x7f73d408b180 msgr2=0x7f73d408b590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:17.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.949+0000 7f73dc3b9640 1 -- 172.21.15.67:0/3033062295 shutdown_connections 2024-03-24T00:46:17.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.949+0000 7f73dc3b9640 1 -- 172.21.15.67:0/3033062295 wait complete. 2024-03-24T00:46:17.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.950+0000 7f73dc3b9640 1 Processor -- start 2024-03-24T00:46:17.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.950+0000 7f73dc3b9640 1 -- start start 2024-03-24T00:46:17.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.951+0000 7f73dc3b9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73d40b34a0 0x7f73d41f7f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:17.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.951+0000 7f73dc3b9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73d415bc30 0x7f73d41f3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:17.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.951+0000 7f73dc3b9640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f73d415c010 0x7f73d40c4480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:17.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.951+0000 7f73dc3b9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d40cd070 con 0x7f73d415bc30 2024-03-24T00:46:17.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.951+0000 7f73dc3b9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d40ccef0 con 0x7f73d40b34a0 2024-03-24T00:46:17.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.951+0000 7f73dc3b9640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73d40cd1f0 con 0x7f73d415c010 2024-03-24T00:46:17.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.951+0000 7f73d9954640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73d415bc30 0x7f73d41f3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:17.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.951+0000 7f73d9954640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73d415bc30 0x7f73d41f3f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55216/0 (socket says 172.21.15.67:55216) 2024-03-24T00:46:17.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.951+0000 7f73d9954640 1 -- 172.21.15.67:0/252221934 learned_addr learned my addr 172.21.15.67:0/252221934 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:17.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.951+0000 7f73da956640 1 --2- 172.21.15.67:0/252221934 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f73d415c010 0x7f73d40c4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:17.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.951+0000 7f73da155640 1 --2- 172.21.15.67:0/252221934 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73d40b34a0 0x7f73d41f7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:17.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.951+0000 7f73d9954640 1 -- 172.21.15.67:0/252221934 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f73d415c010 msgr2=0x7f73d40c4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:17.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.951+0000 7f73d9954640 1 --2- 172.21.15.67:0/252221934 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f73d415c010 0x7f73d40c4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.951+0000 7f73d9954640 1 -- 172.21.15.67:0/252221934 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73d40b34a0 msgr2=0x7f73d41f7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:17.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.951+0000 7f73d9954640 1 --2- 172.21.15.67:0/252221934 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73d40b34a0 0x7f73d41f7f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.951+0000 7f73d9954640 1 -- 172.21.15.67:0/252221934 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73d41f4830 con 0x7f73d415bc30 2024-03-24T00:46:17.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.952+0000 7f73d9954640 1 --2- 172.21.15.67:0/252221934 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73d415bc30 0x7f73d41f3f70 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f73c400c3c0 tx=0x7f73c400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:17.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.952+0000 7f73caffd640 1 -- 172.21.15.67:0/252221934 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c40052d0 con 0x7f73d415bc30 2024-03-24T00:46:17.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.952+0000 7f73caffd640 1 -- 172.21.15.67:0/252221934 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73c4005490 con 0x7f73d415bc30 2024-03-24T00:46:17.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.952+0000 7f73dc3b9640 1 -- 172.21.15.67:0/252221934 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73d41f4b10 con 0x7f73d415bc30 2024-03-24T00:46:17.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.952+0000 7f73caffd640 1 -- 172.21.15.67:0/252221934 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c4007260 con 0x7f73d415bc30 2024-03-24T00:46:17.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.952+0000 7f73dc3b9640 1 -- 172.21.15.67:0/252221934 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73d406caf0 con 0x7f73d415bc30 2024-03-24T00:46:17.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.954+0000 7f73caffd640 1 -- 172.21.15.67:0/252221934 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f73c4007400 con 0x7f73d415bc30 2024-03-24T00:46:17.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.955+0000 7f73caffd640 1 --2- 172.21.15.67:0/252221934 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f73b806bbd0 0x7f73b806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:17.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.955+0000 7f73caffd640 1 -- 172.21.15.67:0/252221934 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 6303+0+0 (secure 0 0 0) 0x7f73c408a1e0 con 0x7f73d415bc30 2024-03-24T00:46:17.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.955+0000 7f73defa4740 1 Processor -- start 2024-03-24T00:46:17.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.955+0000 7f73da155640 1 --2- 172.21.15.67:0/252221934 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f73b806bbd0 0x7f73b806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:17.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.955+0000 7f73defa4740 1 -- start start 2024-03-24T00:46:17.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.956+0000 7f73da155640 1 --2- 172.21.15.67:0/252221934 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f73b806bbd0 0x7f73b806e090 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f73d0006fd0 tx=0x7f73d0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:17.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.956+0000 7f73defa4740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b740c7d0d0 0x55b740c50300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:17.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.956+0000 7f73defa4740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b740c7d4b0 0x55b740c7fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:17.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.956+0000 7f73defa4740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b740c50840 0x55b740c803a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:17.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.956+0000 7f73defa4740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b740bba1f0 con 0x55b740c50840 2024-03-24T00:46:17.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.956+0000 7f73defa4740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b740b58a80 con 0x55b740c7d0d0 2024-03-24T00:46:17.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.956+0000 7f73defa4740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b740bb6290 con 0x55b740c7d4b0 2024-03-24T00:46:17.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.956+0000 7f73da956640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b740c50840 0x55b740c803a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:17.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.956+0000 7f73da956640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b740c50840 0x55b740c803a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55230/0 (socket says 172.21.15.67:55230) 2024-03-24T00:46:17.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.956+0000 7f73da956640 1 -- 172.21.15.67:0/3257337347 learned_addr learned my addr 172.21.15.67:0/3257337347 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:17.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.956+0000 7f73da155640 1 --2- 172.21.15.67:0/3257337347 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b740c7d0d0 0x55b740c50300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:17.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.956+0000 7f73d9954640 1 --2- 172.21.15.67:0/3257337347 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b740c7d4b0 0x55b740c7fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:17.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.957+0000 7f73da956640 1 -- 172.21.15.67:0/3257337347 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b740c7d4b0 msgr2=0x55b740c7fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:17.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.957+0000 7f73da956640 1 --2- 172.21.15.67:0/3257337347 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b740c7d4b0 0x55b740c7fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.957+0000 7f73da956640 1 -- 172.21.15.67:0/3257337347 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b740c7d0d0 msgr2=0x55b740c50300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:17.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.957+0000 7f73da956640 1 --2- 172.21.15.67:0/3257337347 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b740c7d0d0 0x55b740c50300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.957+0000 7f73da956640 1 -- 172.21.15.67:0/3257337347 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b74099a7c0 con 0x55b740c50840 2024-03-24T00:46:17.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.957+0000 7f73da155640 1 --2- 172.21.15.67:0/3257337347 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b740c7d0d0 0x55b740c50300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:46:17.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.957+0000 7f73d9954640 1 --2- 172.21.15.67:0/3257337347 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b740c7d4b0 0x55b740c7fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:46:17.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.957+0000 7f73da956640 1 --2- 172.21.15.67:0/3257337347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b740c50840 0x55b740c803a0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f73cc009f70 tx=0x7f73cc02e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66d6de2fadd039f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:17.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.957+0000 7f73b7fff640 1 -- 172.21.15.67:0/3257337347 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73cc02cb00 con 0x55b740c50840 2024-03-24T00:46:17.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.957+0000 7f73b7fff640 1 -- 172.21.15.67:0/3257337347 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73cc02ce50 con 0x55b740c50840 2024-03-24T00:46:17.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.957+0000 7f73b7fff640 1 -- 172.21.15.67:0/3257337347 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73cc03e670 con 0x55b740c50840 2024-03-24T00:46:17.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.958+0000 7f73defa4740 1 -- 172.21.15.67:0/3257337347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b740c50840 msgr2=0x55b740c803a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:17.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.958+0000 7f73defa4740 1 --2- 172.21.15.67:0/3257337347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b740c50840 0x55b740c803a0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f73cc009f70 tx=0x7f73cc02e000 comp rx=0 tx=0).stop 2024-03-24T00:46:17.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.958+0000 7f73defa4740 1 -- 172.21.15.67:0/3257337347 shutdown_connections 2024-03-24T00:46:17.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.958+0000 7f73defa4740 1 --2- 172.21.15.67:0/3257337347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b740c50840 0x55b740c803a0 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.958+0000 7f73defa4740 1 --2- 172.21.15.67:0/3257337347 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b740c7d4b0 0x55b740c7fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.958+0000 7f73defa4740 1 --2- 172.21.15.67:0/3257337347 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b740c7d0d0 0x55b740c50300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.958+0000 7f73defa4740 1 -- 172.21.15.67:0/3257337347 >> 172.21.15.67:0/3257337347 conn(0x55b740c52570 msgr2=0x55b740c9ae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:17.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.958+0000 7f73defa4740 1 -- 172.21.15.67:0/3257337347 shutdown_connections 2024-03-24T00:46:17.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.958+0000 7f73defa4740 1 -- 172.21.15.67:0/3257337347 wait complete. 2024-03-24T00:46:17.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.958+0000 7f73defa4740 1 -- start start 2024-03-24T00:46:17.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.958+0000 7f73defa4740 1 Processor -- start 2024-03-24T00:46:17.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.959+0000 7f73defa4740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b740c7d0d0 0x55b740c9d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:17.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.959+0000 7f73defa4740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b740c7d4b0 0x55b740c9b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:17.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.959+0000 7f73defa4740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b740c74a60 0x55b740c74ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:17.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.959+0000 7f73defa4740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b740c54880 con 0x55b740c7d4b0 2024-03-24T00:46:17.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.959+0000 7f73defa4740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b740b56410 con 0x55b740c74a60 2024-03-24T00:46:17.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.959+0000 7f73defa4740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b740c53c70 con 0x55b740c7d0d0 2024-03-24T00:46:17.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.959+0000 7f73defa4740 10 client.0 _ll_register_callbacks cb 0x55b740b7ed70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:46:17.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.959+0000 7f73d9954640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b740c7d4b0 0x55b740c9b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:17.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.959+0000 7f73d9954640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b740c7d4b0 0x55b740c9b4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55234/0 (socket says 172.21.15.67:55234) 2024-03-24T00:46:17.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.959+0000 7f73d9954640 1 -- 172.21.15.67:0/619790611 learned_addr learned my addr 172.21.15.67:0/619790611 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:17.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.960+0000 7f73da956640 1 --2- 172.21.15.67:0/619790611 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b740c74a60 0x55b740c74ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:17.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.960+0000 7f73da155640 1 --2- 172.21.15.67:0/619790611 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b740c7d0d0 0x55b740c9d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:17.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.960+0000 7f73d9954640 1 -- 172.21.15.67:0/619790611 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b740c7d0d0 msgr2=0x55b740c9d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:17.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.960+0000 7f73d9954640 1 --2- 172.21.15.67:0/619790611 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b740c7d0d0 0x55b740c9d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.960+0000 7f73d9954640 1 -- 172.21.15.67:0/619790611 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b740c74a60 msgr2=0x55b740c74ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:17.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.960+0000 7f73d9954640 1 --2- 172.21.15.67:0/619790611 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b740c74a60 0x55b740c74ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.960+0000 7f73d9954640 1 -- 172.21.15.67:0/619790611 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b7409faf50 con 0x55b740c7d4b0 2024-03-24T00:46:17.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.960+0000 7f73da956640 1 --2- 172.21.15.67:0/619790611 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b740c74a60 0x55b740c74ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:46:17.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.960+0000 7f73da155640 1 --2- 172.21.15.67:0/619790611 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b740c7d0d0 0x55b740c9d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:46:17.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.960+0000 7f73d9954640 1 --2- 172.21.15.67:0/619790611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b740c7d4b0 0x55b740c9b4a0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f73c4058310 tx=0x7f73c405c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5dce37f1a4959a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:17.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.960+0000 7f73b57fa640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:46:17.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.960+0000 7f73b57fa640 1 -- 172.21.15.67:0/619790611 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c405c150 con 0x55b740c7d4b0 2024-03-24T00:46:17.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.960+0000 7f73b57fa640 1 -- 172.21.15.67:0/619790611 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 6303+0+0 (secure 0 0 0) 0x7f73c40549f0 con 0x55b740c7d4b0 2024-03-24T00:46:17.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.960+0000 7f73defa4740 1 -- 172.21.15.67:0/619790611 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f73cc02ce50 con 0x55b740c7d4b0 2024-03-24T00:46:17.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.960+0000 7f73b57fa640 1 -- 172.21.15.67:0/619790611 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) v2 ==== 24+0+0 (secure 0 0 0) 0x7f73c4088050 con 0x55b740c7d4b0 2024-03-24T00:46:17.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.960+0000 7f73defa4740 10 client.8103 fetch_fsmap learned FSMap version 61 2024-03-24T00:46:17.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.960+0000 7f73defa4740 1 -- 172.21.15.67:0/619790611 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=2+}) v3 -- 0x55b74088e0b0 con 0x55b740c7d4b0 2024-03-24T00:46:17.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.960+0000 7f73b57fa640 1 -- 172.21.15.67:0/619790611 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73cc02ce50 con 0x55b740c7d4b0 2024-03-24T00:46:17.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.960+0000 7f73b57fa640 1 -- 172.21.15.67:0/619790611 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 61) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f73c40577d0 con 0x55b740c7d4b0 2024-03-24T00:46:17.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.960+0000 7f73defa4740 10 client.8103 fetch_fsmap finished waiting for FSMap version 61 2024-03-24T00:46:17.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.960+0000 7f73defa4740 10 client.8103 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T00:46:17.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.960+0000 7f73defa4740 20 client.8103 populate_metadata read hostname 'smithi067' 2024-03-24T00:46:17.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.960+0000 7f73defa4740 1 --2- 172.21.15.67:0/619790611 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x55b740c81820 0x55b740c81c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:17.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.961+0000 7f73defa4740 4 client.8103 mds_command: new command op to 7860 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:46:17.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.961+0000 7f73defa4740 1 -- 172.21.15.67:0/619790611 --> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b740bc7730 con 0x55b740c81820 2024-03-24T00:46:17.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.961+0000 7f73da155640 1 --2- 172.21.15.67:0/619790611 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x55b740c81820 0x55b740c81c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:17.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.961+0000 7f73da155640 1 --2- 172.21.15.67:0/619790611 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x55b740c81820 0x55b740c81c00 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dbd4a692dcb8d179 server_cookie=719fcbfa1d93a10f in_seq=0 out_seq=0 2024-03-24T00:46:17.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.961+0000 7f73b57fa640 10 client.8103 ms_handle_connect on v2:172.21.15.67:6832/1336395002 2024-03-24T00:46:17.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.961+0000 7f73b57fa640 1 -- 172.21.15.67:0/619790611 <== mds.0 v2:172.21.15.67:6832/1336395002 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b740bc7730 con 0x55b740c81820 2024-03-24T00:46:17.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.961+0000 7f73b57fa640 10 client.8103 handle_command_reply: tid=0 2024-03-24T00:46:17.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.961+0000 7f73defa4740 1 client.8103 shutdown 2024-03-24T00:46:17.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.962+0000 7f73defa4740 1 -- 172.21.15.67:0/619790611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b740c7d4b0 msgr2=0x55b740c9b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:17.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.962+0000 7f73defa4740 1 --2- 172.21.15.67:0/619790611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b740c7d4b0 0x55b740c9b4a0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f73c4058310 tx=0x7f73c405c4d0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.962+0000 7f73defa4740 1 -- 172.21.15.67:0/619790611 shutdown_connections 2024-03-24T00:46:17.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.962+0000 7f73defa4740 1 --2- 172.21.15.67:0/619790611 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x55b740c81820 0x55b740c81c00 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.962+0000 7f73defa4740 1 --2- 172.21.15.67:0/619790611 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b740c74a60 0x55b740c74ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.962+0000 7f73b57fa640 0 client.8103 ms_handle_reset on v2:172.21.15.67:6832/1336395002 2024-03-24T00:46:17.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.962+0000 7f73defa4740 1 --2- 172.21.15.67:0/619790611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b740c7d4b0 0x55b740c9b4a0 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.962+0000 7f73defa4740 1 --2- 172.21.15.67:0/619790611 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b740c7d0d0 0x55b740c9d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.962+0000 7f73defa4740 1 -- 172.21.15.67:0/619790611 >> 172.21.15.67:0/619790611 conn(0x55b740c50840 msgr2=0x55b740c9c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:17.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.962+0000 7f73defa4740 1 -- 172.21.15.67:0/619790611 shutdown_connections 2024-03-24T00:46:17.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.962+0000 7f73defa4740 1 -- 172.21.15.67:0/619790611 wait complete. 2024-03-24T00:46:17.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.962+0000 7f73defa4740 20 client.8103 trim_cache size 0 max 16384 2024-03-24T00:46:17.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.975+0000 7f73defa4740 1 Processor -- start 2024-03-24T00:46:17.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.975+0000 7f73defa4740 1 -- start start 2024-03-24T00:46:17.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.975+0000 7f73defa4740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b740c74a60 0x55b740c81c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:17.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.975+0000 7f73defa4740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b740c7d0d0 0x55b740c82140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:17.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.975+0000 7f73defa4740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b740c7d4b0 0x55b740c73990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:17.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.975+0000 7f73defa4740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d4200cb0 con 0x55b740c7d0d0 2024-03-24T00:46:17.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.975+0000 7f73defa4740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d41ffde0 con 0x55b740c7d4b0 2024-03-24T00:46:17.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.975+0000 7f73defa4740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73d40c9680 con 0x55b740c74a60 2024-03-24T00:46:17.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.976+0000 7f73d9954640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b740c7d0d0 0x55b740c82140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:17.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.976+0000 7f73d9954640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b740c7d0d0 0x55b740c82140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55244/0 (socket says 172.21.15.67:55244) 2024-03-24T00:46:17.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.976+0000 7f73d9954640 1 -- 172.21.15.67:0/1207972835 learned_addr learned my addr 172.21.15.67:0/1207972835 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:17.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.976+0000 7f73da956640 1 --2- 172.21.15.67:0/1207972835 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b740c7d4b0 0x55b740c73990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:17.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.976+0000 7f73da155640 1 --2- 172.21.15.67:0/1207972835 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b740c74a60 0x55b740c81c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:17.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.976+0000 7f73d9954640 1 -- 172.21.15.67:0/1207972835 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b740c74a60 msgr2=0x55b740c81c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:17.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.976+0000 7f73d9954640 1 --2- 172.21.15.67:0/1207972835 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b740c74a60 0x55b740c81c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.976+0000 7f73d9954640 1 -- 172.21.15.67:0/1207972835 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b740c7d4b0 msgr2=0x55b740c73990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:17.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.976+0000 7f73d9954640 1 --2- 172.21.15.67:0/1207972835 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b740c7d4b0 0x55b740c73990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.976+0000 7f73d9954640 1 -- 172.21.15.67:0/1207972835 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b740c52db0 con 0x55b740c7d0d0 2024-03-24T00:46:17.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.976+0000 7f73da956640 1 --2- 172.21.15.67:0/1207972835 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b740c7d4b0 0x55b740c73990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:46:17.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.976+0000 7f73da155640 1 --2- 172.21.15.67:0/1207972835 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b740c74a60 0x55b740c81c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:46:17.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.976+0000 7f73d9954640 1 --2- 172.21.15.67:0/1207972835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b740c7d0d0 0x55b740c82140 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f73c40568b0 tx=0x7f73c405c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a99590e48e638ab1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:17.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.976+0000 7f73b77fe640 1 -- 172.21.15.67:0/1207972835 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c4070170 con 0x55b740c7d0d0 2024-03-24T00:46:17.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.976+0000 7f73b77fe640 1 -- 172.21.15.67:0/1207972835 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73c40577d0 con 0x55b740c7d0d0 2024-03-24T00:46:17.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.976+0000 7f73b77fe640 1 -- 172.21.15.67:0/1207972835 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c40549f0 con 0x55b740c7d0d0 2024-03-24T00:46:17.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.977+0000 7f73defa4740 1 -- 172.21.15.67:0/1207972835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b740c7d0d0 msgr2=0x55b740c82140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:17.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.977+0000 7f73defa4740 1 --2- 172.21.15.67:0/1207972835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b740c7d0d0 0x55b740c82140 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f73c40568b0 tx=0x7f73c405c000 comp rx=0 tx=0).stop 2024-03-24T00:46:17.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.977+0000 7f73defa4740 1 -- 172.21.15.67:0/1207972835 shutdown_connections 2024-03-24T00:46:17.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.977+0000 7f73defa4740 1 --2- 172.21.15.67:0/1207972835 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b740c7d4b0 0x55b740c73990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.977+0000 7f73defa4740 1 --2- 172.21.15.67:0/1207972835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b740c7d0d0 0x55b740c82140 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.977+0000 7f73defa4740 1 --2- 172.21.15.67:0/1207972835 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b740c74a60 0x55b740c81c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.977+0000 7f73defa4740 1 -- 172.21.15.67:0/1207972835 >> 172.21.15.67:0/1207972835 conn(0x55b740c50840 msgr2=0x55b740c7b5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:17.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.977+0000 7f73defa4740 1 -- 172.21.15.67:0/1207972835 shutdown_connections 2024-03-24T00:46:17.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.977+0000 7f73defa4740 1 -- 172.21.15.67:0/1207972835 wait complete. 2024-03-24T00:46:17.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.977+0000 7f73defa4740 1 -- start start 2024-03-24T00:46:17.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.977+0000 7f73defa4740 1 Processor -- start 2024-03-24T00:46:17.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.978+0000 7f73defa4740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b740c74a60 0x55b740c7e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:17.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.978+0000 7f73defa4740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b740c7d0d0 0x55b740c759b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:17.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.978+0000 7f73defa4740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b740c7d4b0 0x55b740c75ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:17.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.978+0000 7f73defa4740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b740c827f0 con 0x55b740c7d0d0 2024-03-24T00:46:17.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.978+0000 7f73defa4740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d41ff7e0 con 0x55b740c7d4b0 2024-03-24T00:46:17.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.978+0000 7f73defa4740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b740c7f170 con 0x55b740c74a60 2024-03-24T00:46:17.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.978+0000 7f73defa4740 10 client.0 _ll_register_callbacks cb 0x7f73c4058150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:46:17.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.978+0000 7f73d9954640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b740c7d0d0 0x55b740c759b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:17.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.978+0000 7f73d9954640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b740c7d0d0 0x55b740c759b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55252/0 (socket says 172.21.15.67:55252) 2024-03-24T00:46:17.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.978+0000 7f73d9954640 1 -- 172.21.15.67:0/1416030925 learned_addr learned my addr 172.21.15.67:0/1416030925 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:17.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.979+0000 7f73da155640 1 --2- 172.21.15.67:0/1416030925 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b740c74a60 0x55b740c7e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:17.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.979+0000 7f73da956640 1 --2- 172.21.15.67:0/1416030925 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b740c7d4b0 0x55b740c75ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:17.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.979+0000 7f73d9954640 1 -- 172.21.15.67:0/1416030925 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b740c74a60 msgr2=0x55b740c7e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:17.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.979+0000 7f73d9954640 1 --2- 172.21.15.67:0/1416030925 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b740c74a60 0x55b740c7e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.979+0000 7f73d9954640 1 -- 172.21.15.67:0/1416030925 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b740c7d4b0 msgr2=0x55b740c75ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:17.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.979+0000 7f73d9954640 1 --2- 172.21.15.67:0/1416030925 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b740c7d4b0 0x55b740c75ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.979+0000 7f73d9954640 1 -- 172.21.15.67:0/1416030925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b740c83000 con 0x55b740c7d0d0 2024-03-24T00:46:17.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.979+0000 7f73da956640 1 --2- 172.21.15.67:0/1416030925 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b740c7d4b0 0x55b740c75ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:46:17.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.979+0000 7f73d9954640 1 --2- 172.21.15.67:0/1416030925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b740c7d0d0 0x55b740c759b0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f73c4057c70 tx=0x7f73c4088390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92d3ff30ece03951 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:17.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.979+0000 7f73b57fa640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:46:17.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.979+0000 7f73b57fa640 1 -- 172.21.15.67:0/1416030925 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c4057ce0 con 0x55b740c7d0d0 2024-03-24T00:46:17.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.979+0000 7f73b57fa640 1 -- 172.21.15.67:0/1416030925 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 6303+0+0 (secure 0 0 0) 0x7f73c40540d0 con 0x55b740c7d0d0 2024-03-24T00:46:17.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.979+0000 7f73defa4740 1 -- 172.21.15.67:0/1416030925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f73c40577d0 con 0x55b740c7d0d0 2024-03-24T00:46:17.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.979+0000 7f73b57fa640 1 -- 172.21.15.67:0/1416030925 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) v2 ==== 24+0+0 (secure 0 0 0) 0x7f73c40542d0 con 0x55b740c7d0d0 2024-03-24T00:46:17.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.979+0000 7f73defa4740 10 client.8109 fetch_fsmap learned FSMap version 61 2024-03-24T00:46:17.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.979+0000 7f73defa4740 1 -- 172.21.15.67:0/1416030925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=2+}) v3 -- 0x55b740c83000 con 0x55b740c7d0d0 2024-03-24T00:46:17.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.979+0000 7f73b57fa640 1 -- 172.21.15.67:0/1416030925 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73c40577d0 con 0x55b740c7d0d0 2024-03-24T00:46:17.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.979+0000 7f73b57fa640 1 -- 172.21.15.67:0/1416030925 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 61) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f73c4057240 con 0x55b740c7d0d0 2024-03-24T00:46:17.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.979+0000 7f73defa4740 10 client.8109 fetch_fsmap finished waiting for FSMap version 61 2024-03-24T00:46:17.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.980+0000 7f73defa4740 10 client.8109 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T00:46:17.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.980+0000 7f73defa4740 20 client.8109 populate_metadata read hostname 'smithi067' 2024-03-24T00:46:17.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.980+0000 7f73defa4740 1 --2- 172.21.15.67:0/1416030925 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x55b740d0a8e0 0x55b740d0acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:17.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.980+0000 7f73defa4740 4 client.8109 mds_command: new command op to 7860 tid=0 multi_id=0 [{"prefix": "damage ls", "format": "json-pretty"}] 2024-03-24T00:46:17.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.980+0000 7f73defa4740 1 -- 172.21.15.67:0/1416030925 --> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) v1 -- 0x55b740bc1a80 con 0x55b740d0a8e0 2024-03-24T00:46:17.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.980+0000 7f73da155640 1 --2- 172.21.15.67:0/1416030925 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x55b740d0a8e0 0x55b740d0acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:17.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.980+0000 7f73b57fa640 10 client.8109 ms_handle_connect on v2:172.21.15.67:6832/1336395002 2024-03-24T00:46:17.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.980+0000 7f73da155640 1 --2- 172.21.15.67:0/1416030925 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x55b740d0a8e0 0x55b740d0acc0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8bb1e0e2678143cf server_cookie=4dbf3eaf5660991 in_seq=0 out_seq=0 2024-03-24T00:46:17.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.980+0000 7f73b57fa640 1 -- 172.21.15.67:0/1416030925 <== mds.0 v2:172.21.15.67:6832/1336395002 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+144 (crc 0 0 0) 0x55b740bc1a80 con 0x55b740d0a8e0 2024-03-24T00:46:17.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.980+0000 7f73b57fa640 10 client.8109 handle_command_reply: tid=0 2024-03-24T00:46:17.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.980+0000 7f73defa4740 1 client.8109 shutdown 2024-03-24T00:46:17.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.980+0000 7f73defa4740 1 -- 172.21.15.67:0/1416030925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b740c7d0d0 msgr2=0x55b740c759b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:17.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.980+0000 7f73defa4740 1 --2- 172.21.15.67:0/1416030925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b740c7d0d0 0x55b740c759b0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f73c4057c70 tx=0x7f73c4088390 comp rx=0 tx=0).stop 2024-03-24T00:46:17.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.980+0000 7f73defa4740 1 -- 172.21.15.67:0/1416030925 shutdown_connections 2024-03-24T00:46:17.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.980+0000 7f73defa4740 1 --2- 172.21.15.67:0/1416030925 >> [v2:172.21.15.67:6832/1336395002,v1:172.21.15.67:6833/1336395002] conn(0x55b740d0a8e0 0x55b740d0acc0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.980+0000 7f73defa4740 1 --2- 172.21.15.67:0/1416030925 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b740c7d4b0 0x55b740c75ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.980+0000 7f73b57fa640 0 client.8109 ms_handle_reset on v2:172.21.15.67:6832/1336395002 2024-03-24T00:46:17.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.980+0000 7f73defa4740 1 --2- 172.21.15.67:0/1416030925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b740c7d0d0 0x55b740c759b0 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.980+0000 7f73defa4740 1 --2- 172.21.15.67:0/1416030925 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b740c74a60 0x55b740c7e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.980+0000 7f73defa4740 1 -- 172.21.15.67:0/1416030925 >> 172.21.15.67:0/1416030925 conn(0x55b740c50840 msgr2=0x55b740c81c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:17.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.980+0000 7f73defa4740 1 -- 172.21.15.67:0/1416030925 shutdown_connections 2024-03-24T00:46:17.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.981+0000 7f73defa4740 1 -- 172.21.15.67:0/1416030925 wait complete. 2024-03-24T00:46:17.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.981+0000 7f73defa4740 20 client.8109 trim_cache size 0 max 16384 2024-03-24T00:46:17.988 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:46:17.988 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-03-24T00:46:17.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:46:17.989 INFO:teuthology.orchestra.run.smithi067.stdout: "damage_type": "backtrace", 2024-03-24T00:46:17.989 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 1644331677, 2024-03-24T00:46:17.989 INFO:teuthology.orchestra.run.smithi067.stdout: "ino": 1099511627777, 2024-03-24T00:46:17.989 INFO:teuthology.orchestra.run.smithi067.stdout: "path": "/testdir/hardlink" 2024-03-24T00:46:17.989 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:46:17.989 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-03-24T00:46:17.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.981+0000 7f73dc3b9640 1 -- 172.21.15.67:0/252221934 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f73b806bbd0 msgr2=0x7f73b806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:17.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.981+0000 7f73dc3b9640 1 --2- 172.21.15.67:0/252221934 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f73b806bbd0 0x7f73b806e090 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f73d0006fd0 tx=0x7f73d0008040 comp rx=0 tx=0).stop 2024-03-24T00:46:17.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.981+0000 7f73dc3b9640 1 -- 172.21.15.67:0/252221934 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73d415bc30 msgr2=0x7f73d41f3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:17.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.981+0000 7f73dc3b9640 1 --2- 172.21.15.67:0/252221934 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73d415bc30 0x7f73d41f3f70 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f73c400c3c0 tx=0x7f73c400c880 comp rx=0 tx=0).stop 2024-03-24T00:46:17.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.981+0000 7f73dc3b9640 1 -- 172.21.15.67:0/252221934 shutdown_connections 2024-03-24T00:46:17.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.981+0000 7f73dc3b9640 1 --2- 172.21.15.67:0/252221934 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f73b806bbd0 0x7f73b806e090 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.981+0000 7f73dc3b9640 1 --2- 172.21.15.67:0/252221934 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f73d415c010 0x7f73d40c4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.981+0000 7f73dc3b9640 1 --2- 172.21.15.67:0/252221934 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73d415bc30 0x7f73d41f3f70 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.981+0000 7f73dc3b9640 1 --2- 172.21.15.67:0/252221934 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73d40b34a0 0x7f73d41f7f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:17.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.981+0000 7f73dc3b9640 1 -- 172.21.15.67:0/252221934 >> 172.21.15.67:0/252221934 conn(0x7f73d408b180 msgr2=0x7f73d406fc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:17.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.981+0000 7f73dc3b9640 1 -- 172.21.15.67:0/252221934 shutdown_connections 2024-03-24T00:46:17.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:17.981+0000 7f73dc3b9640 1 -- 172.21.15.67:0/252221934 wait complete. 2024-03-24T00:46:17.994 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:46:17.994 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:46:18.009 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:46:18.009 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:46:18.064 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:46:18.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.204+0000 7fde9d97e640 1 Processor -- start 2024-03-24T00:46:18.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.204+0000 7fde9d97e640 1 -- start start 2024-03-24T00:46:18.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.205+0000 7fde9d97e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fde9815df30 0x7fde98162610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:18.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.205+0000 7fde9d97e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fde9815e310 0x7fde980d0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:18.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.205+0000 7fde9d97e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fde980d0b80 0x7fde980b57a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:18.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.205+0000 7fde9d97e640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde98162b50 con 0x7fde980d0b80 2024-03-24T00:46:18.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.205+0000 7fde9d97e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde980d0fd0 con 0x7fde9815e310 2024-03-24T00:46:18.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.205+0000 7fde9d97e640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde980ce920 con 0x7fde9815df30 2024-03-24T00:46:18.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.205+0000 7fde977fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fde980d0b80 0x7fde980b57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:18.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.205+0000 7fde977fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fde980d0b80 0x7fde980b57a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55256/0 (socket says 172.21.15.67:55256) 2024-03-24T00:46:18.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.205+0000 7fde977fe640 1 -- 172.21.15.67:0/3128630550 learned_addr learned my addr 172.21.15.67:0/3128630550 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:18.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.205+0000 7fde96ffd640 1 --2- 172.21.15.67:0/3128630550 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fde9815df30 0x7fde98162610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:18.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.205+0000 7fde8ffff640 1 --2- 172.21.15.67:0/3128630550 >> v2:172.21.15.72:3300/0 conn(0x7fde9815e310 0x7fde980d0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:18.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.205+0000 7fde977fe640 1 -- 172.21.15.67:0/3128630550 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fde9815df30 msgr2=0x7fde98162610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:18.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.205+0000 7fde977fe640 1 --2- 172.21.15.67:0/3128630550 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fde9815df30 0x7fde98162610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:18.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.205+0000 7fde977fe640 1 -- 172.21.15.67:0/3128630550 >> v2:172.21.15.72:3300/0 conn(0x7fde9815e310 msgr2=0x7fde980d0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:18.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.205+0000 7fde977fe640 1 --2- 172.21.15.67:0/3128630550 >> v2:172.21.15.72:3300/0 conn(0x7fde9815e310 0x7fde980d0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:18.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.205+0000 7fde977fe640 1 -- 172.21.15.67:0/3128630550 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde980cef20 con 0x7fde980d0b80 2024-03-24T00:46:18.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.205+0000 7fde8ffff640 1 --2- 172.21.15.67:0/3128630550 >> v2:172.21.15.72:3300/0 conn(0x7fde9815e310 0x7fde980d0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:46:18.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.206+0000 7fde977fe640 1 --2- 172.21.15.67:0/3128630550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde980d0b80 0x7fde980b57a0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7fde980d4d80 tx=0x7fde9002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=434579347f1c3247 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:18.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.206+0000 7fde967fc640 1 -- 172.21.15.67:0/3128630550 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde9002c9e0 con 0x7fde980d0b80 2024-03-24T00:46:18.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.206+0000 7fde967fc640 1 -- 172.21.15.67:0/3128630550 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde9002cb80 con 0x7fde980d0b80 2024-03-24T00:46:18.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.206+0000 7fde967fc640 1 -- 172.21.15.67:0/3128630550 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde900343f0 con 0x7fde980d0b80 2024-03-24T00:46:18.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.207+0000 7fde9d97e640 1 -- 172.21.15.67:0/3128630550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde980d0b80 msgr2=0x7fde980b57a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:18.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.207+0000 7fde9d97e640 1 --2- 172.21.15.67:0/3128630550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde980d0b80 0x7fde980b57a0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7fde980d4d80 tx=0x7fde9002be40 comp rx=0 tx=0).stop 2024-03-24T00:46:18.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.207+0000 7fde9d97e640 1 -- 172.21.15.67:0/3128630550 shutdown_connections 2024-03-24T00:46:18.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.207+0000 7fde9d97e640 1 --2- 172.21.15.67:0/3128630550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde980d0b80 0x7fde980b57a0 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:18.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.207+0000 7fde9d97e640 1 --2- 172.21.15.67:0/3128630550 >> v2:172.21.15.72:3300/0 conn(0x7fde9815e310 0x7fde980d0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:18.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.207+0000 7fde9d97e640 1 --2- 172.21.15.67:0/3128630550 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fde9815df30 0x7fde98162610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:18.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.207+0000 7fde9d97e640 1 -- 172.21.15.67:0/3128630550 >> 172.21.15.67:0/3128630550 conn(0x7fde9806c880 msgr2=0x7fde9806cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:18.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.207+0000 7fde9d97e640 1 -- 172.21.15.67:0/3128630550 shutdown_connections 2024-03-24T00:46:18.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.207+0000 7fde9d97e640 1 -- 172.21.15.67:0/3128630550 wait complete. 2024-03-24T00:46:18.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.209+0000 7fde9d97e640 1 Processor -- start 2024-03-24T00:46:18.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.209+0000 7fde9d97e640 1 -- start start 2024-03-24T00:46:18.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.210+0000 7fde9d97e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fde980d0b80 0x7fde98150260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:18.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.210+0000 7fde9d97e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde9815df30 0x7fde980ccda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:18.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.210+0000 7fde9d97e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde980b6140 0x7fde980b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:18.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.210+0000 7fde9d97e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde980b32f0 con 0x7fde9815df30 2024-03-24T00:46:18.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.210+0000 7fde9d97e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde980cf4e0 con 0x7fde980b6140 2024-03-24T00:46:18.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.210+0000 7fde9d97e640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde980ce580 con 0x7fde980d0b80 2024-03-24T00:46:18.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.210+0000 7fde8ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde9815df30 0x7fde980ccda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:18.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.210+0000 7fde8ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde9815df30 0x7fde980ccda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55270/0 (socket says 172.21.15.67:55270) 2024-03-24T00:46:18.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.210+0000 7fde8ffff640 1 -- 172.21.15.67:0/918895764 learned_addr learned my addr 172.21.15.67:0/918895764 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:18.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.210+0000 7fde96ffd640 1 --2- 172.21.15.67:0/918895764 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fde980d0b80 0x7fde98150260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:18.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.210+0000 7fde977fe640 1 --2- 172.21.15.67:0/918895764 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde980b6140 0x7fde980b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:18.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.211+0000 7fde8ffff640 1 -- 172.21.15.67:0/918895764 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fde980d0b80 msgr2=0x7fde98150260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:18.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.211+0000 7fde8ffff640 1 --2- 172.21.15.67:0/918895764 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fde980d0b80 0x7fde98150260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:18.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.211+0000 7fde8ffff640 1 -- 172.21.15.67:0/918895764 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde980b6140 msgr2=0x7fde980b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:18.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.211+0000 7fde8ffff640 1 --2- 172.21.15.67:0/918895764 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde980b6140 0x7fde980b4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:18.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.211+0000 7fde8ffff640 1 -- 172.21.15.67:0/918895764 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde980cd6d0 con 0x7fde9815df30 2024-03-24T00:46:18.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.211+0000 7fde8ffff640 1 --2- 172.21.15.67:0/918895764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde9815df30 0x7fde980ccda0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7fde8000d7d0 tx=0x7fde8000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:18.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.211+0000 7fde94ff9640 1 -- 172.21.15.67:0/918895764 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde800096a0 con 0x7fde9815df30 2024-03-24T00:46:18.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.211+0000 7fde94ff9640 1 -- 172.21.15.67:0/918895764 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde80019070 con 0x7fde9815df30 2024-03-24T00:46:18.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.211+0000 7fde9d97e640 1 -- 172.21.15.67:0/918895764 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde980cd9b0 con 0x7fde9815df30 2024-03-24T00:46:18.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.211+0000 7fde94ff9640 1 -- 172.21.15.67:0/918895764 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde80009a80 con 0x7fde9815df30 2024-03-24T00:46:18.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.211+0000 7fde9d97e640 1 -- 172.21.15.67:0/918895764 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde980cdf70 con 0x7fde9815df30 2024-03-24T00:46:18.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.212+0000 7fde9d97e640 1 -- 172.21.15.67:0/918895764 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde9804f400 con 0x7fde9815df30 2024-03-24T00:46:18.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.212+0000 7fde94ff9640 1 -- 172.21.15.67:0/918895764 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fde80009c20 con 0x7fde9815df30 2024-03-24T00:46:18.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.213+0000 7fde94ff9640 1 --2- 172.21.15.67:0/918895764 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fde7806bbd0 0x7fde7806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:18.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.213+0000 7fde94ff9640 1 -- 172.21.15.67:0/918895764 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 6303+0+0 (secure 0 0 0) 0x7fde8008af90 con 0x7fde9815df30 2024-03-24T00:46:18.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.213+0000 7fde96ffd640 1 --2- 172.21.15.67:0/918895764 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fde7806bbd0 0x7fde7806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:18.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.214+0000 7fde96ffd640 1 --2- 172.21.15.67:0/918895764 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fde7806bbd0 0x7fde7806e090 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fde88006fd0 tx=0x7fde88008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:18.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.217+0000 7fde94ff9640 1 -- 172.21.15.67:0/918895764 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fde800591e0 con 0x7fde9815df30 2024-03-24T00:46:18.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.400+0000 7fde9d97e640 1 -- 172.21.15.67:0/918895764 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fde980d1a30 con 0x7fde9815df30 2024-03-24T00:46:18.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.402+0000 7fde94ff9640 1 -- 172.21.15.67:0/918895764 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+17148 (secure 0 0 0) 0x7fde8005d280 con 0x7fde9815df30 2024-03-24T00:46:18.403 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:46:18.403 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":92,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:45:34.493790+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:44:53.025965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-03-24T00:44:53.712957+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.32","mappings":[{"from":1,"to":7}]},{"pgid":"14.36","mappings":[{"from":1,"to":5}]},{"pgid":"14.38","mappings":[{"from":1,"to":2}]},{"pgid":"14.3a","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6833/3021622863":"2024-03-25T00:45:09.102827+0000","172.21.15.67:6832/3021622863":"2024-03-25T00:45:09.102827+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:46:18.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.404+0000 7fde9d97e640 1 -- 172.21.15.67:0/918895764 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fde7806bbd0 msgr2=0x7fde7806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:18.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.404+0000 7fde9d97e640 1 --2- 172.21.15.67:0/918895764 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fde7806bbd0 0x7fde7806e090 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fde88006fd0 tx=0x7fde88008040 comp rx=0 tx=0).stop 2024-03-24T00:46:18.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.404+0000 7fde9d97e640 1 -- 172.21.15.67:0/918895764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde9815df30 msgr2=0x7fde980ccda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:18.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.404+0000 7fde9d97e640 1 --2- 172.21.15.67:0/918895764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde9815df30 0x7fde980ccda0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7fde8000d7d0 tx=0x7fde8000dc90 comp rx=0 tx=0).stop 2024-03-24T00:46:18.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.404+0000 7fde9d97e640 1 -- 172.21.15.67:0/918895764 shutdown_connections 2024-03-24T00:46:18.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.404+0000 7fde9d97e640 1 --2- 172.21.15.67:0/918895764 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fde7806bbd0 0x7fde7806e090 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:18.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.404+0000 7fde9d97e640 1 --2- 172.21.15.67:0/918895764 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde980b6140 0x7fde980b4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:18.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.404+0000 7fde9d97e640 1 --2- 172.21.15.67:0/918895764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde9815df30 0x7fde980ccda0 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:18.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.404+0000 7fde9d97e640 1 --2- 172.21.15.67:0/918895764 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fde980d0b80 0x7fde98150260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:18.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.404+0000 7fde9d97e640 1 -- 172.21.15.67:0/918895764 >> 172.21.15.67:0/918895764 conn(0x7fde9806c880 msgr2=0x7fde980c7930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:18.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.404+0000 7fde9d97e640 1 -- 172.21.15.67:0/918895764 shutdown_connections 2024-03-24T00:46:18.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.404+0000 7fde9d97e640 1 -- 172.21.15.67:0/918895764 wait complete. 2024-03-24T00:46:18.416 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:38:39.364950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:44:53.025965+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:44:53.712957+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:46:18.416 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:46:18.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.519+0000 7fe9c7419640 1 Processor -- start 2024-03-24T00:46:18.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.520+0000 7fe9c7419640 1 -- start start 2024-03-24T00:46:18.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.521+0000 7fe9c7419640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe9c015c1b0 0x7fe9c015fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:18.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.521+0000 7fe9c7419640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe9c0160260 0x7fe9c00cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:18.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.521+0000 7fe9c7419640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c015c7c0 con 0x7fe9c00d18b0 2024-03-24T00:46:18.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.521+0000 7fe9c7419640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9c015caa0 con 0x7fe9c015c1b0 2024-03-24T00:46:18.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.521+0000 7fe9c7419640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9c015cd80 con 0x7fe9c0160260 2024-03-24T00:46:18.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.521+0000 7fe9c49b4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe9c015c1b0 0x7fe9c015fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:18.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.521+0000 7fe9c51b5640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe9c00d18b0 0x7fe9c00d1c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59430/0 (socket says 172.21.15.67:59430) 2024-03-24T00:46:18.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.521+0000 7fe9c51b5640 1 -- 172.21.15.67:0/1608274345 learned_addr learned my addr 172.21.15.67:0/1608274345 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:46:18.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.521+0000 7fe9c49b4640 1 -- 172.21.15.67:0/1608274345 >> v2:172.21.15.72:3300/0 conn(0x7fe9c0160260 msgr2=0x7fe9c00cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:18.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.521+0000 7fe9c59b6640 1 --2- 172.21.15.67:0/1608274345 >> v2:172.21.15.72:3300/0 conn(0x7fe9c0160260 0x7fe9c00cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:18.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.521+0000 7fe9c49b4640 1 --2- 172.21.15.67:0/1608274345 >> v2:172.21.15.72:3300/0 conn(0x7fe9c0160260 0x7fe9c00cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:18.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.521+0000 7fe9b7fff640 1 -- 172.21.15.67:0/1608274345 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1525510551 0 0) 0x7fe9c015c7c0 con 0x7fe9c00d18b0 2024-03-24T00:46:18.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.521+0000 7fe9c49b4640 1 -- 172.21.15.67:0/1608274345 >> v1:172.21.15.67:6789/0 conn(0x7fe9c00d18b0 legacy=0x7fe9c00d1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:18.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.521+0000 7fe9c49b4640 1 -- 172.21.15.67:0/1608274345 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c015d060 con 0x7fe9c015c1b0 2024-03-24T00:46:18.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.521+0000 7fe9c59b6640 1 --2- 172.21.15.67:0/1608274345 >> v2:172.21.15.72:3300/0 conn(0x7fe9c0160260 0x7fe9c00cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:46:18.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.522+0000 7fe9c49b4640 1 --2- 172.21.15.67:0/1608274345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9c015c1b0 0x7fe9c015fd20 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7fe9c00b4790 tx=0x7fe9bc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99da88d94d3544bb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:18.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.522+0000 7fe9b7fff640 1 -- 172.21.15.67:0/1608274345 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc0315a0 con 0x7fe9c015c1b0 2024-03-24T00:46:18.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.522+0000 7fe9b7fff640 1 -- 172.21.15.67:0/1608274345 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9bc037070 con 0x7fe9c015c1b0 2024-03-24T00:46:18.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.522+0000 7fe9b7fff640 1 -- 172.21.15.67:0/1608274345 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc031960 con 0x7fe9c015c1b0 2024-03-24T00:46:18.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.523+0000 7fe9c7419640 1 -- 172.21.15.67:0/1608274345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9c015c1b0 msgr2=0x7fe9c015fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:18.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.523+0000 7fe9c7419640 1 --2- 172.21.15.67:0/1608274345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9c015c1b0 0x7fe9c015fd20 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7fe9c00b4790 tx=0x7fe9bc02b840 comp rx=0 tx=0).stop 2024-03-24T00:46:18.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.523+0000 7fe9c7419640 1 -- 172.21.15.67:0/1608274345 shutdown_connections 2024-03-24T00:46:18.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.523+0000 7fe9c7419640 1 --2- 172.21.15.67:0/1608274345 >> v2:172.21.15.72:3300/0 conn(0x7fe9c0160260 0x7fe9c00cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:18.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.523+0000 7fe9c7419640 1 --2- 172.21.15.67:0/1608274345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9c015c1b0 0x7fe9c015fd20 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:18.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.523+0000 7fe9c7419640 1 -- 172.21.15.67:0/1608274345 >> 172.21.15.67:0/1608274345 conn(0x7fe9c008b1b0 msgr2=0x7fe9c008b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:18.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.523+0000 7fe9c7419640 1 -- 172.21.15.67:0/1608274345 shutdown_connections 2024-03-24T00:46:18.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.523+0000 7fe9c7419640 1 -- 172.21.15.67:0/1608274345 wait complete. 2024-03-24T00:46:18.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.524+0000 7fe9c7419640 1 Processor -- start 2024-03-24T00:46:18.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.524+0000 7fe9c7419640 1 -- start start 2024-03-24T00:46:18.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.525+0000 7fe9c7419640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe9c00d18b0 0x7fe9c00af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:18.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.525+0000 7fe9c7419640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe9c015c1b0 0x7fe9c00c9d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:18.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.525+0000 7fe9c7419640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9c0160260 0x7fe9c015e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:18.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.525+0000 7fe9c7419640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c00cfe90 con 0x7fe9c0160260 2024-03-24T00:46:18.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.525+0000 7fe9c7419640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c00cfd10 con 0x7fe9c00d18b0 2024-03-24T00:46:18.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.525+0000 7fe9c7419640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c00d0010 con 0x7fe9c015c1b0 2024-03-24T00:46:18.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.525+0000 7fe9c59b6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9c0160260 0x7fe9c015e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:18.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.525+0000 7fe9c59b6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9c0160260 0x7fe9c015e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55284/0 (socket says 172.21.15.67:55284) 2024-03-24T00:46:18.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.525+0000 7fe9c59b6640 1 -- 172.21.15.67:0/2358149846 learned_addr learned my addr 172.21.15.67:0/2358149846 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:18.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.525+0000 7fe9c59b6640 1 -- 172.21.15.67:0/2358149846 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe9c015c1b0 msgr2=0x7fe9c00c9d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:18.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.525+0000 7fe9c59b6640 1 --2- 172.21.15.67:0/2358149846 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe9c015c1b0 0x7fe9c00c9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:18.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.525+0000 7fe9c59b6640 1 -- 172.21.15.67:0/2358149846 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe9c00d18b0 msgr2=0x7fe9c00af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:18.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.525+0000 7fe9c59b6640 1 --2- 172.21.15.67:0/2358149846 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe9c00d18b0 0x7fe9c00af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:18.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.525+0000 7fe9c59b6640 1 -- 172.21.15.67:0/2358149846 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c00cbab0 con 0x7fe9c0160260 2024-03-24T00:46:18.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.525+0000 7fe9c59b6640 1 --2- 172.21.15.67:0/2358149846 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9c0160260 0x7fe9c015e420 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7fe9b800cd50 tx=0x7fe9b800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:18.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.525+0000 7fe9b5ffb640 1 -- 172.21.15.67:0/2358149846 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b800f680 con 0x7fe9c0160260 2024-03-24T00:46:18.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.525+0000 7fe9b5ffb640 1 -- 172.21.15.67:0/2358149846 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9b800f840 con 0x7fe9c0160260 2024-03-24T00:46:18.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.525+0000 7fe9c7419640 1 -- 172.21.15.67:0/2358149846 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9c00caa10 con 0x7fe9c0160260 2024-03-24T00:46:18.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.525+0000 7fe9b5ffb640 1 -- 172.21.15.67:0/2358149846 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b800fc30 con 0x7fe9c0160260 2024-03-24T00:46:18.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.526+0000 7fe9c7419640 1 -- 172.21.15.67:0/2358149846 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9c00cafd0 con 0x7fe9c0160260 2024-03-24T00:46:18.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.527+0000 7fe9c7419640 1 -- 172.21.15.67:0/2358149846 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9c0071ae0 con 0x7fe9c0160260 2024-03-24T00:46:18.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.527+0000 7fe9b5ffb640 1 -- 172.21.15.67:0/2358149846 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe9b80213f0 con 0x7fe9c0160260 2024-03-24T00:46:18.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.528+0000 7fe9b5ffb640 1 --2- 172.21.15.67:0/2358149846 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe9a406bbd0 0x7fe9a406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:18.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.528+0000 7fe9b5ffb640 1 -- 172.21.15.67:0/2358149846 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 6303+0+0 (secure 0 0 0) 0x7fe9b808ba60 con 0x7fe9c0160260 2024-03-24T00:46:18.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.528+0000 7fe9c51b5640 1 --2- 172.21.15.67:0/2358149846 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe9a406bbd0 0x7fe9a406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:18.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.528+0000 7fe9c51b5640 1 --2- 172.21.15.67:0/2358149846 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe9a406bbd0 0x7fe9a406e090 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0007fd0 tx=0x7fe9b0007920 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:18.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.532+0000 7fe9b5ffb640 1 -- 172.21.15.67:0/2358149846 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe9c0071ae0 con 0x7fe9c0160260 2024-03-24T00:46:18.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.769+0000 7fe9c7419640 1 -- 172.21.15.67:0/2358149846 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe9c00b34d0 con 0x7fe9c0160260 2024-03-24T00:46:18.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.769+0000 7fe9b5ffb640 1 -- 172.21.15.67:0/2358149846 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe9b8014030 con 0x7fe9c0160260 2024-03-24T00:46:18.770 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-24T00:46:18.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.771+0000 7fe9c7419640 1 -- 172.21.15.67:0/2358149846 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe9a406bbd0 msgr2=0x7fe9a406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:18.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.771+0000 7fe9c7419640 1 --2- 172.21.15.67:0/2358149846 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe9a406bbd0 0x7fe9a406e090 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0007fd0 tx=0x7fe9b0007920 comp rx=0 tx=0).stop 2024-03-24T00:46:18.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.771+0000 7fe9c7419640 1 -- 172.21.15.67:0/2358149846 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9c0160260 msgr2=0x7fe9c015e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:18.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.771+0000 7fe9c7419640 1 --2- 172.21.15.67:0/2358149846 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9c0160260 0x7fe9c015e420 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7fe9b800cd50 tx=0x7fe9b800a650 comp rx=0 tx=0).stop 2024-03-24T00:46:18.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.771+0000 7fe9c7419640 1 -- 172.21.15.67:0/2358149846 shutdown_connections 2024-03-24T00:46:18.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.771+0000 7fe9c7419640 1 --2- 172.21.15.67:0/2358149846 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe9a406bbd0 0x7fe9a406e090 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:18.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.771+0000 7fe9c7419640 1 --2- 172.21.15.67:0/2358149846 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9c0160260 0x7fe9c015e420 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:18.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.771+0000 7fe9c7419640 1 --2- 172.21.15.67:0/2358149846 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe9c015c1b0 0x7fe9c00c9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:18.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.771+0000 7fe9c7419640 1 --2- 172.21.15.67:0/2358149846 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe9c00d18b0 0x7fe9c00af6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:18.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.771+0000 7fe9c7419640 1 -- 172.21.15.67:0/2358149846 >> 172.21.15.67:0/2358149846 conn(0x7fe9c008b1b0 msgr2=0x7fe9c00709e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:18.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.771+0000 7fe9c7419640 1 -- 172.21.15.67:0/2358149846 shutdown_connections 2024-03-24T00:46:18.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.771+0000 7fe9c7419640 1 -- 172.21.15.67:0/2358149846 wait complete. 2024-03-24T00:46:18.782 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:46:18.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.885+0000 7f466d881640 1 Processor -- start 2024-03-24T00:46:18.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.885+0000 7f466d881640 1 -- start start 2024-03-24T00:46:18.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.887+0000 7f466d881640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f46680d18b0 0x7f466815c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:18.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.887+0000 7f466d881640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f46680d1c90 0x7f466815fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:18.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.887+0000 7f466d881640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f466815c7c0 con 0x7f4668160260 2024-03-24T00:46:18.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.887+0000 7f466d881640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f466815caa0 con 0x7f46680d18b0 2024-03-24T00:46:18.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.887+0000 7f466d881640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f466815cd80 con 0x7f46680d1c90 2024-03-24T00:46:18.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.887+0000 7f46677fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4668160260 0x7f46680b1d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59438/0 (socket says 172.21.15.67:59438) 2024-03-24T00:46:18.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.887+0000 7f4666ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f46680d18b0 0x7f466815c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:18.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.887+0000 7f46677fe640 1 -- 172.21.15.67:0/72360309 learned_addr learned my addr 172.21.15.67:0/72360309 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:46:18.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.887+0000 7f465ffff640 1 --2- 172.21.15.67:0/72360309 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f46680d1c90 0x7f466815fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:18.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.887+0000 7f4666ffd640 1 -- 172.21.15.67:0/72360309 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f46680d1c90 msgr2=0x7f466815fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:18.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.887+0000 7f4666ffd640 1 --2- 172.21.15.67:0/72360309 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f46680d1c90 0x7f466815fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:18.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.887+0000 7f4666ffd640 1 -- 172.21.15.67:0/72360309 >> v1:172.21.15.67:6789/0 conn(0x7f4668160260 legacy=0x7f46680b1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:18.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.887+0000 7f4666ffd640 1 -- 172.21.15.67:0/72360309 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f466815d060 con 0x7f46680d18b0 2024-03-24T00:46:18.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.887+0000 7f46667fc640 1 -- 172.21.15.67:0/72360309 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1032129563 0 0) 0x7f466815c7c0 con 0x7f4668160260 2024-03-24T00:46:18.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.888+0000 7f4666ffd640 1 --2- 172.21.15.67:0/72360309 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46680d18b0 0x7f466815c1b0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f46580097a0 tx=0x7f465802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad02f521064d7d32 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:18.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.888+0000 7f46667fc640 1 -- 172.21.15.67:0/72360309 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46580315a0 con 0x7f46680d18b0 2024-03-24T00:46:18.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.888+0000 7f46667fc640 1 -- 172.21.15.67:0/72360309 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4658037070 con 0x7f46680d18b0 2024-03-24T00:46:18.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.888+0000 7f46667fc640 1 -- 172.21.15.67:0/72360309 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4658031960 con 0x7f46680d18b0 2024-03-24T00:46:18.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.889+0000 7f466d881640 1 -- 172.21.15.67:0/72360309 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46680d18b0 msgr2=0x7f466815c1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:18.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.889+0000 7f466d881640 1 --2- 172.21.15.67:0/72360309 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46680d18b0 0x7f466815c1b0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f46580097a0 tx=0x7f465802b840 comp rx=0 tx=0).stop 2024-03-24T00:46:18.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.889+0000 7f466d881640 1 -- 172.21.15.67:0/72360309 shutdown_connections 2024-03-24T00:46:18.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.889+0000 7f466d881640 1 --2- 172.21.15.67:0/72360309 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f46680d1c90 0x7f466815fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:18.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.889+0000 7f466d881640 1 --2- 172.21.15.67:0/72360309 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46680d18b0 0x7f466815c1b0 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:18.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.889+0000 7f466d881640 1 -- 172.21.15.67:0/72360309 >> 172.21.15.67:0/72360309 conn(0x7f466808b1b0 msgr2=0x7f466808b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:18.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.889+0000 7f466d881640 1 -- 172.21.15.67:0/72360309 shutdown_connections 2024-03-24T00:46:18.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.889+0000 7f466d881640 1 -- 172.21.15.67:0/72360309 wait complete. 2024-03-24T00:46:18.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.890+0000 7f466d881640 1 Processor -- start 2024-03-24T00:46:18.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.890+0000 7f466d881640 1 -- start start 2024-03-24T00:46:18.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.891+0000 7f466d881640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46680d18b0 0x7f4668157e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:18.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.891+0000 7f466d881640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f46680d1c90 0x7f466814e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:18.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.891+0000 7f466d881640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4668160260 0x7f46680cab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:18.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.891+0000 7f466d881640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46680d2270 con 0x7f46680d18b0 2024-03-24T00:46:18.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.891+0000 7f466d881640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46680d20f0 con 0x7f46680d1c90 2024-03-24T00:46:18.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.891+0000 7f466d881640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46680d07e0 con 0x7f4668160260 2024-03-24T00:46:18.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.891+0000 7f4666ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46680d18b0 0x7f4668157e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:18.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.891+0000 7f4666ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46680d18b0 0x7f4668157e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55308/0 (socket says 172.21.15.67:55308) 2024-03-24T00:46:18.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.891+0000 7f4666ffd640 1 -- 172.21.15.67:0/134100110 learned_addr learned my addr 172.21.15.67:0/134100110 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:18.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.891+0000 7f46677fe640 1 --2- 172.21.15.67:0/134100110 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4668160260 0x7f46680cab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:18.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.891+0000 7f465ffff640 1 --2- 172.21.15.67:0/134100110 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f46680d1c90 0x7f466814e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:18.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.891+0000 7f4666ffd640 1 -- 172.21.15.67:0/134100110 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4668160260 msgr2=0x7f46680cab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:18.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.891+0000 7f4666ffd640 1 --2- 172.21.15.67:0/134100110 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4668160260 0x7f46680cab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:18.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.891+0000 7f4666ffd640 1 -- 172.21.15.67:0/134100110 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f46680d1c90 msgr2=0x7f466814e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:18.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.891+0000 7f4666ffd640 1 --2- 172.21.15.67:0/134100110 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f46680d1c90 0x7f466814e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:18.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.891+0000 7f4666ffd640 1 -- 172.21.15.67:0/134100110 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46681f3c80 con 0x7f46680d18b0 2024-03-24T00:46:18.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.891+0000 7f465ffff640 1 --2- 172.21.15.67:0/134100110 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f46680d1c90 0x7f466814e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:46:18.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.891+0000 7f46677fe640 1 --2- 172.21.15.67:0/134100110 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4668160260 0x7f46680cab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:46:18.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.892+0000 7f4666ffd640 1 --2- 172.21.15.67:0/134100110 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46680d18b0 0x7f4668157e80 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f4658034040 tx=0x7f46580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:18.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.892+0000 7f4664ff9640 1 -- 172.21.15.67:0/134100110 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4658032840 con 0x7f46680d18b0 2024-03-24T00:46:18.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.892+0000 7f4664ff9640 1 -- 172.21.15.67:0/134100110 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46580329e0 con 0x7f46680d18b0 2024-03-24T00:46:18.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.892+0000 7f4664ff9640 1 -- 172.21.15.67:0/134100110 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4658032dd0 con 0x7f46680d18b0 2024-03-24T00:46:18.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.892+0000 7f466d881640 1 -- 172.21.15.67:0/134100110 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46680cb390 con 0x7f46680d18b0 2024-03-24T00:46:18.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.892+0000 7f466d881640 1 -- 172.21.15.67:0/134100110 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46680cb950 con 0x7f46680d18b0 2024-03-24T00:46:18.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.893+0000 7f466d881640 1 -- 172.21.15.67:0/134100110 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4668158410 con 0x7f46680d18b0 2024-03-24T00:46:18.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.893+0000 7f4664ff9640 1 -- 172.21.15.67:0/134100110 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f465804e470 con 0x7f46680d18b0 2024-03-24T00:46:18.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.894+0000 7f4664ff9640 1 --2- 172.21.15.67:0/134100110 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f464806bc00 0x7f464806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:18.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.894+0000 7f4664ff9640 1 -- 172.21.15.67:0/134100110 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 6303+0+0 (secure 0 0 0) 0x7f465802f030 con 0x7f46680d18b0 2024-03-24T00:46:18.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.894+0000 7f465ffff640 1 --2- 172.21.15.67:0/134100110 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f464806bc00 0x7f464806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:18.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.895+0000 7f465ffff640 1 --2- 172.21.15.67:0/134100110 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f464806bc00 0x7f464806e0c0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f4668157480 tx=0x7f4650008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:18.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:18.898+0000 7f4664ff9640 1 -- 172.21.15.67:0/134100110 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f46681f3c80 con 0x7f46680d18b0 2024-03-24T00:46:19.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.217+0000 7f466d881640 1 -- 172.21.15.67:0/134100110 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f466804f400 con 0x7f46680d18b0 2024-03-24T00:46:19.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.217+0000 7f4664ff9640 1 -- 172.21.15.67:0/134100110 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4658040070 con 0x7f46680d18b0 2024-03-24T00:46:19.218 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-24T00:46:19.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.219+0000 7f466d881640 1 -- 172.21.15.67:0/134100110 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f464806bc00 msgr2=0x7f464806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:19.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.219+0000 7f466d881640 1 --2- 172.21.15.67:0/134100110 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f464806bc00 0x7f464806e0c0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f4668157480 tx=0x7f4650008040 comp rx=0 tx=0).stop 2024-03-24T00:46:19.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.219+0000 7f466d881640 1 -- 172.21.15.67:0/134100110 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46680d18b0 msgr2=0x7f4668157e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:19.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.219+0000 7f466d881640 1 --2- 172.21.15.67:0/134100110 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46680d18b0 0x7f4668157e80 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f4658034040 tx=0x7f46580046c0 comp rx=0 tx=0).stop 2024-03-24T00:46:19.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.219+0000 7f466d881640 1 -- 172.21.15.67:0/134100110 shutdown_connections 2024-03-24T00:46:19.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.219+0000 7f466d881640 1 --2- 172.21.15.67:0/134100110 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f464806bc00 0x7f464806e0c0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:19.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.219+0000 7f466d881640 1 --2- 172.21.15.67:0/134100110 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4668160260 0x7f46680cab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:19.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.219+0000 7f466d881640 1 --2- 172.21.15.67:0/134100110 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f46680d1c90 0x7f466814e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:19.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.219+0000 7f466d881640 1 --2- 172.21.15.67:0/134100110 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46680d18b0 0x7f4668157e80 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:19.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.219+0000 7f466d881640 1 -- 172.21.15.67:0/134100110 >> 172.21.15.67:0/134100110 conn(0x7f466808b1b0 msgr2=0x7f4668070960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:19.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.219+0000 7f466d881640 1 -- 172.21.15.67:0/134100110 shutdown_connections 2024-03-24T00:46:19.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.219+0000 7f466d881640 1 -- 172.21.15.67:0/134100110 wait complete. 2024-03-24T00:46:19.231 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:46:19.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.334+0000 7fd070abf640 1 Processor -- start 2024-03-24T00:46:19.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.334+0000 7fd070abf640 1 -- start start 2024-03-24T00:46:19.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.335+0000 7fd070abf640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd06c15dea0 0x7fd06c0d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:19.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.335+0000 7fd070abf640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd06c15e280 0x7fd06c0d3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:19.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.335+0000 7fd070abf640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd06c0b5a60 0x7fd06c0b5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:19.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.335+0000 7fd070abf640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd06c0b63f0 con 0x7fd06c15dea0 2024-03-24T00:46:19.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.335+0000 7fd070abf640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd06c0d2250 con 0x7fd06c15e280 2024-03-24T00:46:19.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.335+0000 7fd070abf640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd06c0d2530 con 0x7fd06c0b5a60 2024-03-24T00:46:19.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.335+0000 7fd06a59c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd06c15dea0 0x7fd06c0d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:19.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.336+0000 7fd06a59c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd06c15dea0 0x7fd06c0d33d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55324/0 (socket says 172.21.15.67:55324) 2024-03-24T00:46:19.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.336+0000 7fd06a59c640 1 -- 172.21.15.67:0/3848136589 learned_addr learned my addr 172.21.15.67:0/3848136589 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:19.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.336+0000 7fd069d9b640 1 --2- 172.21.15.67:0/3848136589 >> v2:172.21.15.72:3300/0 conn(0x7fd06c15e280 0x7fd06c0d3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:19.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.336+0000 7fd06ad9d640 1 --2- 172.21.15.67:0/3848136589 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd06c0b5a60 0x7fd06c0b5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:19.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.336+0000 7fd06a59c640 1 -- 172.21.15.67:0/3848136589 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd06c0b5a60 msgr2=0x7fd06c0b5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:19.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.336+0000 7fd06a59c640 1 --2- 172.21.15.67:0/3848136589 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd06c0b5a60 0x7fd06c0b5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:19.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.336+0000 7fd06a59c640 1 -- 172.21.15.67:0/3848136589 >> v2:172.21.15.72:3300/0 conn(0x7fd06c15e280 msgr2=0x7fd06c0d3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:19.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.336+0000 7fd06a59c640 1 --2- 172.21.15.67:0/3848136589 >> v2:172.21.15.72:3300/0 conn(0x7fd06c15e280 0x7fd06c0d3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:19.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.336+0000 7fd06a59c640 1 -- 172.21.15.67:0/3848136589 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd06c0d2810 con 0x7fd06c15dea0 2024-03-24T00:46:19.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.336+0000 7fd069d9b640 1 --2- 172.21.15.67:0/3848136589 >> v2:172.21.15.72:3300/0 conn(0x7fd06c15e280 0x7fd06c0d3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:46:19.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.336+0000 7fd06a59c640 1 --2- 172.21.15.67:0/3848136589 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd06c15dea0 0x7fd06c0d33d0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7fd06c0d04d0 tx=0x7fd06002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fefacecb0c55c102 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:19.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.337+0000 7fd06959a640 1 -- 172.21.15.67:0/3848136589 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd06002e070 con 0x7fd06c15dea0 2024-03-24T00:46:19.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.337+0000 7fd06959a640 1 -- 172.21.15.67:0/3848136589 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd060037070 con 0x7fd06c15dea0 2024-03-24T00:46:19.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.337+0000 7fd06959a640 1 -- 172.21.15.67:0/3848136589 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd060031a00 con 0x7fd06c15dea0 2024-03-24T00:46:19.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.337+0000 7fd070abf640 1 -- 172.21.15.67:0/3848136589 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd06c15dea0 msgr2=0x7fd06c0d33d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:19.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.337+0000 7fd070abf640 1 --2- 172.21.15.67:0/3848136589 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd06c15dea0 0x7fd06c0d33d0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7fd06c0d04d0 tx=0x7fd06002b880 comp rx=0 tx=0).stop 2024-03-24T00:46:19.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.337+0000 7fd070abf640 1 -- 172.21.15.67:0/3848136589 shutdown_connections 2024-03-24T00:46:19.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.337+0000 7fd070abf640 1 --2- 172.21.15.67:0/3848136589 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd06c0b5a60 0x7fd06c0b5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:19.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.337+0000 7fd070abf640 1 --2- 172.21.15.67:0/3848136589 >> v2:172.21.15.72:3300/0 conn(0x7fd06c15e280 0x7fd06c0d3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:19.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.337+0000 7fd070abf640 1 --2- 172.21.15.67:0/3848136589 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd06c15dea0 0x7fd06c0d33d0 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:19.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.337+0000 7fd070abf640 1 -- 172.21.15.67:0/3848136589 >> 172.21.15.67:0/3848136589 conn(0x7fd06c072da0 msgr2=0x7fd06c0d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:19.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.337+0000 7fd070abf640 1 -- 172.21.15.67:0/3848136589 shutdown_connections 2024-03-24T00:46:19.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.337+0000 7fd070abf640 1 -- 172.21.15.67:0/3848136589 wait complete. 2024-03-24T00:46:19.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.339+0000 7fd070abf640 1 Processor -- start 2024-03-24T00:46:19.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.339+0000 7fd070abf640 1 -- start start 2024-03-24T00:46:19.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.339+0000 7fd070abf640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd06c0b5a60 0x7fd06c14f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:19.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.339+0000 7fd070abf640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd06c15dea0 0x7fd06c0cd870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:19.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.339+0000 7fd070abf640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd06c15e280 0x7fd06c0ce200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:19.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.339+0000 7fd070abf640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd06c06c7c0 con 0x7fd06c0b5a60 2024-03-24T00:46:19.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.339+0000 7fd070abf640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd06c06c640 con 0x7fd06c15dea0 2024-03-24T00:46:19.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.339+0000 7fd070abf640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd06c06c940 con 0x7fd06c15e280 2024-03-24T00:46:19.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.339+0000 7fd06a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd06c0b5a60 0x7fd06c14f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:19.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.340+0000 7fd06a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd06c0b5a60 0x7fd06c14f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55338/0 (socket says 172.21.15.67:55338) 2024-03-24T00:46:19.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.340+0000 7fd06a59c640 1 -- 172.21.15.67:0/4112485287 learned_addr learned my addr 172.21.15.67:0/4112485287 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:19.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.340+0000 7fd069d9b640 1 --2- 172.21.15.67:0/4112485287 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd06c15dea0 0x7fd06c0cd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:19.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.340+0000 7fd06ad9d640 1 --2- 172.21.15.67:0/4112485287 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd06c15e280 0x7fd06c0ce200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:19.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.340+0000 7fd06a59c640 1 -- 172.21.15.67:0/4112485287 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd06c15e280 msgr2=0x7fd06c0ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:19.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.340+0000 7fd06a59c640 1 --2- 172.21.15.67:0/4112485287 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd06c15e280 0x7fd06c0ce200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:19.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.340+0000 7fd06a59c640 1 -- 172.21.15.67:0/4112485287 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd06c15dea0 msgr2=0x7fd06c0cd870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:19.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.340+0000 7fd06a59c640 1 --2- 172.21.15.67:0/4112485287 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd06c15dea0 0x7fd06c0cd870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:19.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.340+0000 7fd06a59c640 1 -- 172.21.15.67:0/4112485287 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd06c1f80b0 con 0x7fd06c0b5a60 2024-03-24T00:46:19.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.340+0000 7fd06a59c640 1 --2- 172.21.15.67:0/4112485287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd06c0b5a60 0x7fd06c14f530 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7fd06002b8c0 tx=0x7fd0600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:19.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.340+0000 7fd05affd640 1 -- 172.21.15.67:0/4112485287 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd060032880 con 0x7fd06c0b5a60 2024-03-24T00:46:19.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.340+0000 7fd05affd640 1 -- 172.21.15.67:0/4112485287 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd060032a40 con 0x7fd06c0b5a60 2024-03-24T00:46:19.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.340+0000 7fd05affd640 1 -- 172.21.15.67:0/4112485287 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd060032e30 con 0x7fd06c0b5a60 2024-03-24T00:46:19.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.340+0000 7fd070abf640 1 -- 172.21.15.67:0/4112485287 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd06c1f7330 con 0x7fd06c0b5a60 2024-03-24T00:46:19.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.340+0000 7fd070abf640 1 -- 172.21.15.67:0/4112485287 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd06c1f78b0 con 0x7fd06c0b5a60 2024-03-24T00:46:19.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.341+0000 7fd070abf640 1 -- 172.21.15.67:0/4112485287 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd06c04f400 con 0x7fd06c0b5a60 2024-03-24T00:46:19.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.342+0000 7fd05affd640 1 -- 172.21.15.67:0/4112485287 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd06004c060 con 0x7fd06c0b5a60 2024-03-24T00:46:19.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.342+0000 7fd05affd640 1 --2- 172.21.15.67:0/4112485287 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd04c06bbd0 0x7fd04c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:19.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.342+0000 7fd05affd640 1 -- 172.21.15.67:0/4112485287 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 6303+0+0 (secure 0 0 0) 0x7fd0600acad0 con 0x7fd06c0b5a60 2024-03-24T00:46:19.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.343+0000 7fd069d9b640 1 --2- 172.21.15.67:0/4112485287 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd04c06bbd0 0x7fd04c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:19.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.343+0000 7fd069d9b640 1 --2- 172.21.15.67:0/4112485287 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd04c06bbd0 0x7fd04c06e090 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fd06c0ac030 tx=0x7fd05c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:19.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.346+0000 7fd05affd640 1 -- 172.21.15.67:0/4112485287 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd0600ace20 con 0x7fd06c0b5a60 2024-03-24T00:46:19.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.586+0000 7fd070abf640 1 -- 172.21.15.67:0/4112485287 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd06c161630 con 0x7fd06c0b5a60 2024-03-24T00:46:19.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.586+0000 7fd05affd640 1 -- 172.21.15.67:0/4112485287 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd060040070 con 0x7fd06c0b5a60 2024-03-24T00:46:19.587 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-03-24T00:46:19.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.588+0000 7fd070abf640 1 -- 172.21.15.67:0/4112485287 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd04c06bbd0 msgr2=0x7fd04c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:19.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.588+0000 7fd070abf640 1 --2- 172.21.15.67:0/4112485287 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd04c06bbd0 0x7fd04c06e090 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fd06c0ac030 tx=0x7fd05c008040 comp rx=0 tx=0).stop 2024-03-24T00:46:19.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.588+0000 7fd070abf640 1 -- 172.21.15.67:0/4112485287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd06c0b5a60 msgr2=0x7fd06c14f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:19.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.588+0000 7fd070abf640 1 --2- 172.21.15.67:0/4112485287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd06c0b5a60 0x7fd06c14f530 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7fd06002b8c0 tx=0x7fd0600046c0 comp rx=0 tx=0).stop 2024-03-24T00:46:19.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.588+0000 7fd070abf640 1 -- 172.21.15.67:0/4112485287 shutdown_connections 2024-03-24T00:46:19.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.588+0000 7fd070abf640 1 --2- 172.21.15.67:0/4112485287 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd04c06bbd0 0x7fd04c06e090 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:19.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.588+0000 7fd070abf640 1 --2- 172.21.15.67:0/4112485287 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd06c15e280 0x7fd06c0ce200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:19.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.588+0000 7fd070abf640 1 --2- 172.21.15.67:0/4112485287 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd06c15dea0 0x7fd06c0cd870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:19.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.588+0000 7fd070abf640 1 --2- 172.21.15.67:0/4112485287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd06c0b5a60 0x7fd06c14f530 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:19.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.588+0000 7fd070abf640 1 -- 172.21.15.67:0/4112485287 >> 172.21.15.67:0/4112485287 conn(0x7fd06c072da0 msgr2=0x7fd06c0b76c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:19.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.588+0000 7fd070abf640 1 -- 172.21.15.67:0/4112485287 shutdown_connections 2024-03-24T00:46:19.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.588+0000 7fd070abf640 1 -- 172.21.15.67:0/4112485287 wait complete. 2024-03-24T00:46:19.600 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T00:46:19.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.702+0000 7ff2fdd3b640 1 Processor -- start 2024-03-24T00:46:19.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.702+0000 7ff2fdd3b640 1 -- start start 2024-03-24T00:46:19.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.704+0000 7ff2fdd3b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff2f80b37a0 0x7ff2f8166d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:19.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.704+0000 7ff2fdd3b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2f80cd430 con 0x7ff2f80d07a0 2024-03-24T00:46:19.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.704+0000 7ff2fdd3b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2f80cd710 con 0x7ff2f815c010 2024-03-24T00:46:19.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.704+0000 7ff2fdd3b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f80cfbe0 con 0x7ff2f80b37a0 2024-03-24T00:46:19.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.704+0000 7ff2f6ffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff2f80d07a0 0x7ff2f80cc770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59442/0 (socket says 172.21.15.67:59442) 2024-03-24T00:46:19.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.704+0000 7ff2f6ffd640 1 -- 172.21.15.67:0/1320309456 learned_addr learned my addr 172.21.15.67:0/1320309456 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:46:19.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.704+0000 7ff2f7fff640 1 --2- 172.21.15.67:0/1320309456 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff2f80b37a0 0x7ff2f8166d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:19.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.704+0000 7ff2f67fc640 1 -- 172.21.15.67:0/1320309456 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3954014285 0 0) 0x7ff2f80cd430 con 0x7ff2f80d07a0 2024-03-24T00:46:19.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.704+0000 7ff2f67fc640 1 -- 172.21.15.67:0/1320309456 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2e0003520 con 0x7ff2f80d07a0 2024-03-24T00:46:19.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.704+0000 7ff2f67fc640 1 -- 172.21.15.67:0/1320309456 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1515435232 0 0) 0x7ff2e0003520 con 0x7ff2f80d07a0 2024-03-24T00:46:19.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.704+0000 7ff2f67fc640 1 -- 172.21.15.67:0/1320309456 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff2f80b37a0 msgr2=0x7ff2f8166d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:19.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.704+0000 7ff2f67fc640 1 --2- 172.21.15.67:0/1320309456 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff2f80b37a0 0x7ff2f8166d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:19.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.704+0000 7ff2f67fc640 1 -- 172.21.15.67:0/1320309456 >> v1:172.21.15.72:6789/0 conn(0x7ff2f815c010 legacy=0x7ff2f80d0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:19.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.704+0000 7ff2f67fc640 1 -- 172.21.15.67:0/1320309456 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f80cd9f0 con 0x7ff2f80d07a0 2024-03-24T00:46:19.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.704+0000 7ff2f67fc640 1 -- 172.21.15.67:0/1320309456 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7ff2ec001670 con 0x7ff2f80d07a0 2024-03-24T00:46:19.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.704+0000 7ff2f67fc640 1 -- 172.21.15.67:0/1320309456 >> v1:172.21.15.67:6789/0 conn(0x7ff2f80d07a0 legacy=0x7ff2f80cc770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:19.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.704+0000 7ff2f67fc640 1 --2- 172.21.15.67:0/1320309456 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2e00039e0 0x7ff2e0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:19.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.704+0000 7ff2f67fc640 1 --2- 172.21.15.67:0/1320309456 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff2f80b37a0 0x7ff2e0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:19.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.704+0000 7ff2f67fc640 1 --2- 172.21.15.67:0/1320309456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2e0004d40 0x7ff2e00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:19.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.704+0000 7ff2f67fc640 1 -- 172.21.15.67:0/1320309456 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e000b9c0 con 0x7ff2e0004d40 2024-03-24T00:46:19.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.704+0000 7ff2f67fc640 1 -- 172.21.15.67:0/1320309456 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e000bb70 con 0x7ff2e00039e0 2024-03-24T00:46:19.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.704+0000 7ff2f67fc640 1 -- 172.21.15.67:0/1320309456 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e000bd70 con 0x7ff2f80b37a0 2024-03-24T00:46:19.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.705+0000 7ff2f7fff640 1 --2- 172.21.15.67:0/1320309456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2e0004d40 0x7ff2e00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:19.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.705+0000 7ff2f6ffd640 1 --2- 172.21.15.67:0/1320309456 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff2f80b37a0 0x7ff2e0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:19.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.705+0000 7ff2f77fe640 1 --2- 172.21.15.67:0/1320309456 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2e00039e0 0x7ff2e0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:19.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.705+0000 7ff2f7fff640 1 -- 172.21.15.67:0/1320309456 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff2f80b37a0 msgr2=0x7ff2e0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:19.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.705+0000 7ff2f7fff640 1 --2- 172.21.15.67:0/1320309456 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff2f80b37a0 0x7ff2e0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:19.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.705+0000 7ff2f7fff640 1 -- 172.21.15.67:0/1320309456 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2e00039e0 msgr2=0x7ff2e0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:19.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.705+0000 7ff2f7fff640 1 --2- 172.21.15.67:0/1320309456 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2e00039e0 0x7ff2e0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:19.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.705+0000 7ff2f7fff640 1 -- 172.21.15.67:0/1320309456 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2e0003520 con 0x7ff2e0004d40 2024-03-24T00:46:19.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.705+0000 7ff2f77fe640 1 --2- 172.21.15.67:0/1320309456 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2e00039e0 0x7ff2e0003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:46:19.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.706+0000 7ff2f7fff640 1 --2- 172.21.15.67:0/1320309456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2e0004d40 0x7ff2e00051c0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7ff2f0009f90 tx=0x7ff2e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ee70586269cd453 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:19.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.706+0000 7ff2f67fc640 1 -- 172.21.15.67:0/1320309456 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f0039070 con 0x7ff2e0004d40 2024-03-24T00:46:19.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.706+0000 7ff2f67fc640 1 -- 172.21.15.67:0/1320309456 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2f002cbd0 con 0x7ff2e0004d40 2024-03-24T00:46:19.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.706+0000 7ff2f67fc640 1 -- 172.21.15.67:0/1320309456 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f0009c10 con 0x7ff2e0004d40 2024-03-24T00:46:19.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.706+0000 7ff2fdd3b640 1 -- 172.21.15.67:0/1320309456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2e0004d40 msgr2=0x7ff2e00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:19.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.706+0000 7ff2fdd3b640 1 --2- 172.21.15.67:0/1320309456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2e0004d40 0x7ff2e00051c0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7ff2f0009f90 tx=0x7ff2e0005830 comp rx=0 tx=0).stop 2024-03-24T00:46:19.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.706+0000 7ff2f7fff640 1 -- 172.21.15.67:0/1320309456 reap_dead start 2024-03-24T00:46:19.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.706+0000 7ff2fdd3b640 1 -- 172.21.15.67:0/1320309456 shutdown_connections 2024-03-24T00:46:19.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.706+0000 7ff2fdd3b640 1 -- 172.21.15.67:0/1320309456 >> 172.21.15.67:0/1320309456 conn(0x7ff2f809aea0 msgr2=0x7ff2f808b180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:19.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.706+0000 7ff2fdd3b640 1 -- 172.21.15.67:0/1320309456 shutdown_connections 2024-03-24T00:46:19.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.706+0000 7ff2fdd3b640 1 -- 172.21.15.67:0/1320309456 wait complete. 2024-03-24T00:46:19.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.708+0000 7ff2fdd3b640 1 Processor -- start 2024-03-24T00:46:19.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.708+0000 7ff2fdd3b640 1 -- start start 2024-03-24T00:46:19.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.708+0000 7ff2fdd3b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2e0004d40 0x7ff2f8153980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:19.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.708+0000 7ff2fdd3b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff2f80b0f20 0x7ff2f815d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:19.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.709+0000 7ff2fdd3b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2f80b13a0 0x7ff2f815df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:19.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.709+0000 7ff2fdd3b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f80cd210 con 0x7ff2e0004d40 2024-03-24T00:46:19.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.709+0000 7ff2fdd3b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f80cd090 con 0x7ff2f80b13a0 2024-03-24T00:46:19.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.709+0000 7ff2fdd3b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f80b2020 con 0x7ff2f80b0f20 2024-03-24T00:46:19.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.709+0000 7ff2f77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2e0004d40 0x7ff2f8153980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:19.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.709+0000 7ff2f77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2e0004d40 0x7ff2f8153980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55356/0 (socket says 172.21.15.67:55356) 2024-03-24T00:46:19.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.709+0000 7ff2f77fe640 1 -- 172.21.15.67:0/209553930 learned_addr learned my addr 172.21.15.67:0/209553930 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:19.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.709+0000 7ff2f6ffd640 1 --2- 172.21.15.67:0/209553930 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff2f80b0f20 0x7ff2f815d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:19.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.709+0000 7ff2f7fff640 1 --2- 172.21.15.67:0/209553930 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2f80b13a0 0x7ff2f815df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:19.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.709+0000 7ff2f77fe640 1 -- 172.21.15.67:0/209553930 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff2f80b0f20 msgr2=0x7ff2f815d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:19.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.709+0000 7ff2f77fe640 1 --2- 172.21.15.67:0/209553930 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff2f80b0f20 0x7ff2f815d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:19.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.709+0000 7ff2f77fe640 1 -- 172.21.15.67:0/209553930 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2f80b13a0 msgr2=0x7ff2f815df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:19.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.709+0000 7ff2f77fe640 1 --2- 172.21.15.67:0/209553930 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2f80b13a0 0x7ff2f815df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:19.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.709+0000 7ff2f77fe640 1 -- 172.21.15.67:0/209553930 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f815c620 con 0x7ff2e0004d40 2024-03-24T00:46:19.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.709+0000 7ff2f6ffd640 1 --2- 172.21.15.67:0/209553930 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff2f80b0f20 0x7ff2f815d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:46:19.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.709+0000 7ff2f7fff640 1 --2- 172.21.15.67:0/209553930 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2f80b13a0 0x7ff2f815df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:46:19.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.709+0000 7ff2f77fe640 1 --2- 172.21.15.67:0/209553930 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2e0004d40 0x7ff2f8153980 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7ff2e800e800 tx=0x7ff2e800ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:19.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.709+0000 7ff2dffff640 1 -- 172.21.15.67:0/209553930 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e80050d0 con 0x7ff2e0004d40 2024-03-24T00:46:19.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.709+0000 7ff2dffff640 1 -- 172.21.15.67:0/209553930 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2e8018070 con 0x7ff2e0004d40 2024-03-24T00:46:19.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.709+0000 7ff2fdd3b640 1 -- 172.21.15.67:0/209553930 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2f815c900 con 0x7ff2e0004d40 2024-03-24T00:46:19.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.709+0000 7ff2dffff640 1 -- 172.21.15.67:0/209553930 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e800c660 con 0x7ff2e0004d40 2024-03-24T00:46:19.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.709+0000 7ff2fdd3b640 1 -- 172.21.15.67:0/209553930 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2f815cec0 con 0x7ff2e0004d40 2024-03-24T00:46:19.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.710+0000 7ff2fdd3b640 1 -- 172.21.15.67:0/209553930 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2f806e4a0 con 0x7ff2e0004d40 2024-03-24T00:46:19.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.711+0000 7ff2dffff640 1 -- 172.21.15.67:0/209553930 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff2e800c800 con 0x7ff2e0004d40 2024-03-24T00:46:19.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.711+0000 7ff2dffff640 1 --2- 172.21.15.67:0/209553930 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff2e406bbd0 0x7ff2e406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:19.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.711+0000 7ff2dffff640 1 -- 172.21.15.67:0/209553930 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 6303+0+0 (secure 0 0 0) 0x7ff2e808aa70 con 0x7ff2e0004d40 2024-03-24T00:46:19.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.711+0000 7ff2f6ffd640 1 --2- 172.21.15.67:0/209553930 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff2e406bbd0 0x7ff2e406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:19.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.712+0000 7ff2f6ffd640 1 --2- 172.21.15.67:0/209553930 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff2e406bbd0 0x7ff2e406e090 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7ff2f8154410 tx=0x7ff2ec009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:19.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.716+0000 7ff2dffff640 1 -- 172.21.15.67:0/209553930 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff2e8059310 con 0x7ff2e0004d40 2024-03-24T00:46:19.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.912+0000 7ff2fdd3b640 1 -- 172.21.15.67:0/209553930 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff2f80b18b0 con 0x7ff2e0004d40 2024-03-24T00:46:19.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.913+0000 7ff2dffff640 1 -- 172.21.15.67:0/209553930 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v92) v1 ==== 80+0+122 (secure 0 0 0) 0x7ff2e801be40 con 0x7ff2e0004d40 2024-03-24T00:46:19.914 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6833/3021622863 2024-03-25T00:45:09.102827+0000 2024-03-24T00:46:19.914 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6832/3021622863 2024-03-25T00:45:09.102827+0000 2024-03-24T00:46:19.914 INFO:teuthology.orchestra.run.smithi067.stderr:listed 2 entries 2024-03-24T00:46:19.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.915+0000 7ff2fdd3b640 1 -- 172.21.15.67:0/209553930 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff2e406bbd0 msgr2=0x7ff2e406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:19.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.915+0000 7ff2fdd3b640 1 --2- 172.21.15.67:0/209553930 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff2e406bbd0 0x7ff2e406e090 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7ff2f8154410 tx=0x7ff2ec009040 comp rx=0 tx=0).stop 2024-03-24T00:46:19.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.915+0000 7ff2fdd3b640 1 -- 172.21.15.67:0/209553930 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2e0004d40 msgr2=0x7ff2f8153980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:19.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.915+0000 7ff2fdd3b640 1 --2- 172.21.15.67:0/209553930 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2e0004d40 0x7ff2f8153980 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7ff2e800e800 tx=0x7ff2e800ecc0 comp rx=0 tx=0).stop 2024-03-24T00:46:19.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.915+0000 7ff2fdd3b640 1 -- 172.21.15.67:0/209553930 shutdown_connections 2024-03-24T00:46:19.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.915+0000 7ff2fdd3b640 1 --2- 172.21.15.67:0/209553930 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff2e406bbd0 0x7ff2e406e090 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:19.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.915+0000 7ff2fdd3b640 1 --2- 172.21.15.67:0/209553930 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2f80b13a0 0x7ff2f815df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:19.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.915+0000 7ff2fdd3b640 1 --2- 172.21.15.67:0/209553930 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff2f80b0f20 0x7ff2f815d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:19.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.915+0000 7ff2fdd3b640 1 --2- 172.21.15.67:0/209553930 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2e0004d40 0x7ff2f8153980 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:19.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.915+0000 7ff2fdd3b640 1 -- 172.21.15.67:0/209553930 >> 172.21.15.67:0/209553930 conn(0x7ff2f809aea0 msgr2=0x7ff2f8070e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:19.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.915+0000 7ff2fdd3b640 1 -- 172.21.15.67:0/209553930 shutdown_connections 2024-03-24T00:46:19.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:19.915+0000 7ff2fdd3b640 1 -- 172.21.15.67:0/209553930 wait complete. 2024-03-24T00:46:19.926 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-24T00:46:19.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:46:19.926 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-24T00:46:19.956 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-03-24T00:46:19.956 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:46:19.957 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-24T00:46:20.012 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:46:20.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.152+0000 7f518e48c640 1 Processor -- start 2024-03-24T00:46:20.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.152+0000 7f518e48c640 1 -- start start 2024-03-24T00:46:20.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.153+0000 7f518e48c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f518806ba50 0x7f51880ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:20.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.153+0000 7f518e48c640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f51880cd240 0x7f51880b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:20.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.153+0000 7f518e48c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51880cd650 con 0x7f51880cc360 2024-03-24T00:46:20.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.153+0000 7f518e48c640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51880caad0 con 0x7f518806ba50 2024-03-24T00:46:20.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.153+0000 7f518e48c640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f518806c210 con 0x7f51880cd240 2024-03-24T00:46:20.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.154+0000 7f51877fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f518806ba50 0x7f51880ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:20.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.154+0000 7f5187fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f51880cc360 0x7f5188066e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:39406/0 (socket says 172.21.15.67:39406) 2024-03-24T00:46:20.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.154+0000 7f51877fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f518806ba50 0x7f51880ccd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:56234/0 (socket says 172.21.15.67:56234) 2024-03-24T00:46:20.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.154+0000 7f5187fff640 1 -- 172.21.15.67:0/4145392346 learned_addr learned my addr 172.21.15.67:0/4145392346 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:46:20.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.154+0000 7f518ca29640 1 --2- 172.21.15.67:0/4145392346 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f51880cd240 0x7f51880b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:20.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.154+0000 7f5186ffd640 1 -- 172.21.15.67:0/4145392346 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3081908048 0 0) 0x7f5178001110 con 0x7f51880cc360 2024-03-24T00:46:20.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.154+0000 7f5186ffd640 1 -- 172.21.15.67:0/4145392346 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5170003520 con 0x7f51880cc360 2024-03-24T00:46:20.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.154+0000 7f51877fe640 1 -- 172.21.15.67:0/4145392346 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f51880cd240 msgr2=0x7f51880b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:20.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.154+0000 7f51877fe640 1 --2- 172.21.15.67:0/4145392346 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f51880cd240 0x7f51880b0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:20.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.154+0000 7f51877fe640 1 -- 172.21.15.67:0/4145392346 >> v1:172.21.15.72:6789/0 conn(0x7f51880cc360 legacy=0x7f5188066e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:20.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.154+0000 7f51877fe640 1 -- 172.21.15.67:0/4145392346 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51880b4240 con 0x7f518806ba50 2024-03-24T00:46:20.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.154+0000 7f518ca29640 1 --2- 172.21.15.67:0/4145392346 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f51880cd240 0x7f51880b0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:46:20.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.155+0000 7f51877fe640 1 --2- 172.21.15.67:0/4145392346 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f518806ba50 0x7f51880ccd00 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f517c0097a0 tx=0x7f517c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13624a13e651fb51 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:20.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.155+0000 7f5186ffd640 1 -- 172.21.15.67:0/4145392346 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f517c035020 con 0x7f518806ba50 2024-03-24T00:46:20.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.155+0000 7f5186ffd640 1 -- 172.21.15.67:0/4145392346 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f517c031830 con 0x7f518806ba50 2024-03-24T00:46:20.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.155+0000 7f5186ffd640 1 -- 172.21.15.67:0/4145392346 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f517c031be0 con 0x7f518806ba50 2024-03-24T00:46:20.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.156+0000 7f518e48c640 1 -- 172.21.15.67:0/4145392346 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f518806ba50 msgr2=0x7f51880ccd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:20.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.156+0000 7f518e48c640 1 --2- 172.21.15.67:0/4145392346 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f518806ba50 0x7f51880ccd00 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f517c0097a0 tx=0x7f517c02b8d0 comp rx=0 tx=0).stop 2024-03-24T00:46:20.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.156+0000 7f518e48c640 1 -- 172.21.15.67:0/4145392346 shutdown_connections 2024-03-24T00:46:20.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.156+0000 7f518e48c640 1 --2- 172.21.15.67:0/4145392346 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f51880cd240 0x7f51880b0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:20.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.156+0000 7f518e48c640 1 --2- 172.21.15.67:0/4145392346 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f518806ba50 0x7f51880ccd00 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:20.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.156+0000 7f518e48c640 1 -- 172.21.15.67:0/4145392346 >> 172.21.15.67:0/4145392346 conn(0x7f51880b5440 msgr2=0x7f51880b33d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:20.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.156+0000 7f518e48c640 1 -- 172.21.15.67:0/4145392346 shutdown_connections 2024-03-24T00:46:20.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.156+0000 7f518e48c640 1 -- 172.21.15.67:0/4145392346 wait complete. 2024-03-24T00:46:20.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.157+0000 7f518e48c640 1 Processor -- start 2024-03-24T00:46:20.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.157+0000 7f518e48c640 1 -- start start 2024-03-24T00:46:20.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.158+0000 7f518e48c640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f518806ba50 0x7f51880ca020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:20.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.158+0000 7f518e48c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51880cc360 0x7f51880c85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:20.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.158+0000 7f518e48c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51880cd240 0x7f51881f1380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:20.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.158+0000 7f518e48c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51880b3f60 con 0x7f51880cc360 2024-03-24T00:46:20.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.158+0000 7f518e48c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51880b3c80 con 0x7f51880cd240 2024-03-24T00:46:20.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.158+0000 7f518e48c640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51880cc740 con 0x7f518806ba50 2024-03-24T00:46:20.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.158+0000 7f51877fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51880cc360 0x7f51880c85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:20.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.158+0000 7f51877fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51880cc360 0x7f51880c85a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55360/0 (socket says 172.21.15.67:55360) 2024-03-24T00:46:20.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.158+0000 7f51877fe640 1 -- 172.21.15.67:0/2055869201 learned_addr learned my addr 172.21.15.67:0/2055869201 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:20.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.158+0000 7f5187fff640 1 --2- 172.21.15.67:0/2055869201 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f518806ba50 0x7f51880ca020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:20.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.158+0000 7f518ca29640 1 --2- 172.21.15.67:0/2055869201 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51880cd240 0x7f51881f1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:20.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.158+0000 7f51877fe640 1 -- 172.21.15.67:0/2055869201 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f518806ba50 msgr2=0x7f51880ca020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:20.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.158+0000 7f51877fe640 1 --2- 172.21.15.67:0/2055869201 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f518806ba50 0x7f51880ca020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:20.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.158+0000 7f51877fe640 1 -- 172.21.15.67:0/2055869201 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51880cd240 msgr2=0x7f51881f1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:20.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.158+0000 7f51877fe640 1 --2- 172.21.15.67:0/2055869201 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51880cd240 0x7f51881f1380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:20.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.158+0000 7f51877fe640 1 -- 172.21.15.67:0/2055869201 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51880c8ca0 con 0x7f51880cc360 2024-03-24T00:46:20.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.158+0000 7f51877fe640 1 --2- 172.21.15.67:0/2055869201 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51880cc360 0x7f51880c85a0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f517c000c00 tx=0x7f517c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:20.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.159+0000 7f5184ff9640 1 -- 172.21.15.67:0/2055869201 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f517c035050 con 0x7f51880cc360 2024-03-24T00:46:20.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.159+0000 7f5184ff9640 1 -- 172.21.15.67:0/2055869201 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f517c004450 con 0x7f51880cc360 2024-03-24T00:46:20.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.159+0000 7f518e48c640 1 -- 172.21.15.67:0/2055869201 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51880c8f80 con 0x7f51880cc360 2024-03-24T00:46:20.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.159+0000 7f5184ff9640 1 -- 172.21.15.67:0/2055869201 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f517c003810 con 0x7f51880cc360 2024-03-24T00:46:20.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.159+0000 7f518e48c640 1 -- 172.21.15.67:0/2055869201 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51880c9540 con 0x7f51880cc360 2024-03-24T00:46:20.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.160+0000 7f518e48c640 1 -- 172.21.15.67:0/2055869201 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f518804f400 con 0x7f51880cc360 2024-03-24T00:46:20.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.160+0000 7f5184ff9640 1 -- 172.21.15.67:0/2055869201 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f517c038020 con 0x7f51880cc360 2024-03-24T00:46:20.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.161+0000 7f5184ff9640 1 --2- 172.21.15.67:0/2055869201 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f517406bbd0 0x7f517406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:20.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.161+0000 7f5184ff9640 1 -- 172.21.15.67:0/2055869201 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 6303+0+0 (secure 0 0 0) 0x7f517c0ac940 con 0x7f51880cc360 2024-03-24T00:46:20.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.161+0000 7f5187fff640 1 --2- 172.21.15.67:0/2055869201 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f517406bbd0 0x7f517406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:20.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.161+0000 7f5187fff640 1 --2- 172.21.15.67:0/2055869201 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f517406bbd0 0x7f517406e090 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f51780013d0 tx=0x7f5178009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:20.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.165+0000 7f5184ff9640 1 -- 172.21.15.67:0/2055869201 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f517c07b1e0 con 0x7f51880cc360 2024-03-24T00:46:20.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.387+0000 7f518e48c640 1 -- 172.21.15.67:0/2055869201 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51880c9780 con 0x7f51880cc360 2024-03-24T00:46:20.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.387+0000 7f5184ff9640 1 -- 172.21.15.67:0/2055869201 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f517c07aae0 con 0x7f51880cc360 2024-03-24T00:46:20.388 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:46:20.389 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":61,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6359,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2664460392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2664460392},{"type":"v1","addr":"172.21.15.67:6835","nonce":2664460392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6368,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/3323873506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3323873506},{"type":"v1","addr":"172.21.15.72:6837","nonce":3323873506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":7150,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/2744016217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2744016217},{"type":"v1","addr":"172.21.15.72:6836","nonce":2744016217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:44:54.749656+0000","modified":"2024-03-24T00:45:20.120495+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7860},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7860":{"gid":7860,"name":"c","rank":0,"incarnation":57,"state":"up:active","state_seq":6,"addr":"172.21.15.67:6833/1336395002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1336395002},{"type":"v1","addr":"172.21.15.67:6833","nonce":1336395002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7860,"qdb_cluster":[7860]},"id":6}]} 2024-03-24T00:46:20.389 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 61 2024-03-24T00:46:20.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.389+0000 7f518e48c640 1 -- 172.21.15.67:0/2055869201 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f517406bbd0 msgr2=0x7f517406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:20.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.389+0000 7f518e48c640 1 --2- 172.21.15.67:0/2055869201 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f517406bbd0 0x7f517406e090 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f51780013d0 tx=0x7f5178009040 comp rx=0 tx=0).stop 2024-03-24T00:46:20.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.389+0000 7f518e48c640 1 -- 172.21.15.67:0/2055869201 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51880cc360 msgr2=0x7f51880c85a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:20.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.389+0000 7f518e48c640 1 --2- 172.21.15.67:0/2055869201 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51880cc360 0x7f51880c85a0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f517c000c00 tx=0x7f517c0046c0 comp rx=0 tx=0).stop 2024-03-24T00:46:20.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.390+0000 7f518e48c640 1 -- 172.21.15.67:0/2055869201 shutdown_connections 2024-03-24T00:46:20.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.390+0000 7f518e48c640 1 --2- 172.21.15.67:0/2055869201 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f517406bbd0 0x7f517406e090 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:20.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.390+0000 7f518e48c640 1 --2- 172.21.15.67:0/2055869201 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51880cd240 0x7f51881f1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:20.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.390+0000 7f518e48c640 1 --2- 172.21.15.67:0/2055869201 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51880cc360 0x7f51880c85a0 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:20.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.390+0000 7f518e48c640 1 --2- 172.21.15.67:0/2055869201 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f518806ba50 0x7f51880ca020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:20.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.390+0000 7f518e48c640 1 -- 172.21.15.67:0/2055869201 >> 172.21.15.67:0/2055869201 conn(0x7f51880b5440 msgr2=0x7f518815b850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:20.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.390+0000 7f518e48c640 1 -- 172.21.15.67:0/2055869201 shutdown_connections 2024-03-24T00:46:20.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.390+0000 7f518e48c640 1 -- 172.21.15.67:0/2055869201 wait complete. 2024-03-24T00:46:20.401 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-24T00:46:20.529 INFO:teuthology.orchestra.run.smithi067.stdout:{"message":"","return_code":0} 2024-03-24T00:46:20.529 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-24T00:46:20.530 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-24T00:46:20.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.635+0000 7f84e195b640 1 Processor -- start 2024-03-24T00:46:20.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.636+0000 7f84e195b640 1 -- start start 2024-03-24T00:46:20.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.636+0000 7f84e195b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f84dc15c010 0x7f84dc0d0090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:20.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.636+0000 7f84e195b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f84dc0d05d0 0x7f84dc0cc770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:20.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.636+0000 7f84e195b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f84dc0cccb0 0x7f84dc0ce200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:20.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.636+0000 7f84e195b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84dc0d0a40 con 0x7f84dc15c010 2024-03-24T00:46:20.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.636+0000 7f84e195b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84dc0cfbe0 con 0x7f84dc0d05d0 2024-03-24T00:46:20.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.636+0000 7f84e195b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84dc0b2020 con 0x7f84dc0cccb0 2024-03-24T00:46:20.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.636+0000 7f84daffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f84dc15c010 0x7f84dc0d0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:20.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.636+0000 7f84daffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f84dc15c010 0x7f84dc0d0090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55370/0 (socket says 172.21.15.67:55370) 2024-03-24T00:46:20.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.636+0000 7f84daffd640 1 -- 172.21.15.67:0/1056018325 learned_addr learned my addr 172.21.15.67:0/1056018325 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:20.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.637+0000 7f84da7fc640 1 --2- 172.21.15.67:0/1056018325 >> v2:172.21.15.72:3300/0 conn(0x7f84dc0d05d0 0x7f84dc0cc770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:20.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.637+0000 7f84db7fe640 1 --2- 172.21.15.67:0/1056018325 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f84dc0cccb0 0x7f84dc0ce200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:20.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.637+0000 7f84daffd640 1 -- 172.21.15.67:0/1056018325 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f84dc0cccb0 msgr2=0x7f84dc0ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:20.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.637+0000 7f84daffd640 1 --2- 172.21.15.67:0/1056018325 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f84dc0cccb0 0x7f84dc0ce200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:20.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.637+0000 7f84daffd640 1 -- 172.21.15.67:0/1056018325 >> v2:172.21.15.72:3300/0 conn(0x7f84dc0d05d0 msgr2=0x7f84dc0cc770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:20.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.637+0000 7f84daffd640 1 --2- 172.21.15.67:0/1056018325 >> v2:172.21.15.72:3300/0 conn(0x7f84dc0d05d0 0x7f84dc0cc770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:20.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.637+0000 7f84daffd640 1 -- 172.21.15.67:0/1056018325 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84dc0cd430 con 0x7f84dc15c010 2024-03-24T00:46:20.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.637+0000 7f84daffd640 1 --2- 172.21.15.67:0/1056018325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84dc15c010 0x7f84dc0d0090 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f84dc0c8b30 tx=0x7f84d402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=660823482869bbb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:20.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.638+0000 7f84d9ffb640 1 -- 172.21.15.67:0/1056018325 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d4035020 con 0x7f84dc15c010 2024-03-24T00:46:20.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.638+0000 7f84d9ffb640 1 -- 172.21.15.67:0/1056018325 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84d4031870 con 0x7f84dc15c010 2024-03-24T00:46:20.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.638+0000 7f84d9ffb640 1 -- 172.21.15.67:0/1056018325 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d4031c40 con 0x7f84dc15c010 2024-03-24T00:46:20.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.638+0000 7f84e195b640 1 -- 172.21.15.67:0/1056018325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84dc15c010 msgr2=0x7f84dc0d0090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:20.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.638+0000 7f84e195b640 1 --2- 172.21.15.67:0/1056018325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84dc15c010 0x7f84dc0d0090 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f84dc0c8b30 tx=0x7f84d402b910 comp rx=0 tx=0).stop 2024-03-24T00:46:20.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.638+0000 7f84e195b640 1 -- 172.21.15.67:0/1056018325 shutdown_connections 2024-03-24T00:46:20.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.638+0000 7f84e195b640 1 --2- 172.21.15.67:0/1056018325 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f84dc0cccb0 0x7f84dc0ce200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:20.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.638+0000 7f84e195b640 1 --2- 172.21.15.67:0/1056018325 >> v2:172.21.15.72:3300/0 conn(0x7f84dc0d05d0 0x7f84dc0cc770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:20.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.638+0000 7f84e195b640 1 --2- 172.21.15.67:0/1056018325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84dc15c010 0x7f84dc0d0090 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:20.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.638+0000 7f84e195b640 1 -- 172.21.15.67:0/1056018325 >> 172.21.15.67:0/1056018325 conn(0x7f84dc09aea0 msgr2=0x7f84dc08b180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:20.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.638+0000 7f84e195b640 1 -- 172.21.15.67:0/1056018325 shutdown_connections 2024-03-24T00:46:20.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.638+0000 7f84e195b640 1 -- 172.21.15.67:0/1056018325 wait complete. 2024-03-24T00:46:20.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.640+0000 7f84e195b640 1 Processor -- start 2024-03-24T00:46:20.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.640+0000 7f84e195b640 1 -- start start 2024-03-24T00:46:20.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.641+0000 7f84e195b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84dc0cccb0 0x7f84dc148040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:20.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.641+0000 7f84e195b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f84dc0d05d0 0x7f84dc15edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:20.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.641+0000 7f84e195b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f84dc15c010 0x7f84dc15a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:20.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.641+0000 7f84e195b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84dc0cd9f0 con 0x7f84dc0cccb0 2024-03-24T00:46:20.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.641+0000 7f84e195b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84dc0b2300 con 0x7f84dc0d05d0 2024-03-24T00:46:20.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.641+0000 7f84e195b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84dc0cd140 con 0x7f84dc15c010 2024-03-24T00:46:20.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.641+0000 7f84daffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84dc0cccb0 0x7f84dc148040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:20.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.641+0000 7f84daffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84dc0cccb0 0x7f84dc148040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55386/0 (socket says 172.21.15.67:55386) 2024-03-24T00:46:20.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.641+0000 7f84daffd640 1 -- 172.21.15.67:0/2945068706 learned_addr learned my addr 172.21.15.67:0/2945068706 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:20.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.641+0000 7f84db7fe640 1 --2- 172.21.15.67:0/2945068706 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f84dc15c010 0x7f84dc15a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:20.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.641+0000 7f84da7fc640 1 --2- 172.21.15.67:0/2945068706 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f84dc0d05d0 0x7f84dc15edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:20.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.641+0000 7f84daffd640 1 -- 172.21.15.67:0/2945068706 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f84dc15c010 msgr2=0x7f84dc15a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:20.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.641+0000 7f84daffd640 1 --2- 172.21.15.67:0/2945068706 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f84dc15c010 0x7f84dc15a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:20.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.641+0000 7f84daffd640 1 -- 172.21.15.67:0/2945068706 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f84dc0d05d0 msgr2=0x7f84dc15edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:20.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.641+0000 7f84daffd640 1 --2- 172.21.15.67:0/2945068706 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f84dc0d05d0 0x7f84dc15edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:20.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.641+0000 7f84daffd640 1 -- 172.21.15.67:0/2945068706 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84dc148b40 con 0x7f84dc0cccb0 2024-03-24T00:46:20.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.641+0000 7f84da7fc640 1 --2- 172.21.15.67:0/2945068706 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f84dc0d05d0 0x7f84dc15edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:46:20.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.641+0000 7f84daffd640 1 --2- 172.21.15.67:0/2945068706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84dc0cccb0 0x7f84dc148040 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f84d4000c00 tx=0x7f84d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:20.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.641+0000 7f84c3fff640 1 -- 172.21.15.67:0/2945068706 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d4035040 con 0x7f84dc0cccb0 2024-03-24T00:46:20.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.641+0000 7f84c3fff640 1 -- 172.21.15.67:0/2945068706 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84d4030d00 con 0x7f84dc0cccb0 2024-03-24T00:46:20.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.641+0000 7f84e195b640 1 -- 172.21.15.67:0/2945068706 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84dc148e20 con 0x7f84dc0cccb0 2024-03-24T00:46:20.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.641+0000 7f84c3fff640 1 -- 172.21.15.67:0/2945068706 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d40456a0 con 0x7f84dc0cccb0 2024-03-24T00:46:20.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.642+0000 7f84e195b640 1 -- 172.21.15.67:0/2945068706 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84dc1493e0 con 0x7f84dc0cccb0 2024-03-24T00:46:20.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.642+0000 7f84e195b640 1 -- 172.21.15.67:0/2945068706 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84dc06e4a0 con 0x7f84dc0cccb0 2024-03-24T00:46:20.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.643+0000 7f84c3fff640 1 -- 172.21.15.67:0/2945068706 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f84d4038020 con 0x7f84dc0cccb0 2024-03-24T00:46:20.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.643+0000 7f84c3fff640 1 --2- 172.21.15.67:0/2945068706 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f84c806bbd0 0x7f84c806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:20.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.643+0000 7f84c3fff640 1 -- 172.21.15.67:0/2945068706 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 6303+0+0 (secure 0 0 0) 0x7f84d40acf20 con 0x7f84dc0cccb0 2024-03-24T00:46:20.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.643+0000 7f84da7fc640 1 --2- 172.21.15.67:0/2945068706 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f84c806bbd0 0x7f84c806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:20.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.644+0000 7f84da7fc640 1 --2- 172.21.15.67:0/2945068706 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f84c806bbd0 0x7f84c806e090 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f84dc14b8f0 tx=0x7f84d00039d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:20.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.648+0000 7f84c3fff640 1 -- 172.21.15.67:0/2945068706 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f84d407b850 con 0x7f84dc0cccb0 2024-03-24T00:46:20.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.876+0000 7f84e195b640 1 -- 172.21.15.67:0/2945068706 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f84dc0cfe70 con 0x7f84dc0cccb0 2024-03-24T00:46:20.877 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-24T00:46:20.876+0000 7fbfcfa11640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T00:46:20.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.880+0000 7f84c3fff640 1 -- 172.21.15.67:0/2945068706 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v62) v1 ==== 161+0+0 (secure 0 0 0) 0x7f84d407b150 con 0x7f84dc0cccb0 2024-03-24T00:46:20.880 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-24T00:46:20.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.883+0000 7f84e195b640 1 -- 172.21.15.67:0/2945068706 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f84c806bbd0 msgr2=0x7f84c806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:20.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.883+0000 7f84e195b640 1 --2- 172.21.15.67:0/2945068706 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f84c806bbd0 0x7f84c806e090 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f84dc14b8f0 tx=0x7f84d00039d0 comp rx=0 tx=0).stop 2024-03-24T00:46:20.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.883+0000 7f84e195b640 1 -- 172.21.15.67:0/2945068706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84dc0cccb0 msgr2=0x7f84dc148040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:20.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.883+0000 7f84e195b640 1 --2- 172.21.15.67:0/2945068706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84dc0cccb0 0x7f84dc148040 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f84d4000c00 tx=0x7f84d40046c0 comp rx=0 tx=0).stop 2024-03-24T00:46:20.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.883+0000 7f84e195b640 1 -- 172.21.15.67:0/2945068706 shutdown_connections 2024-03-24T00:46:20.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.883+0000 7f84e195b640 1 --2- 172.21.15.67:0/2945068706 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f84c806bbd0 0x7f84c806e090 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:20.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.883+0000 7f84e195b640 1 --2- 172.21.15.67:0/2945068706 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f84dc15c010 0x7f84dc15a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:20.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.883+0000 7f84e195b640 1 --2- 172.21.15.67:0/2945068706 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f84dc0d05d0 0x7f84dc15edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:20.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.883+0000 7f84e195b640 1 --2- 172.21.15.67:0/2945068706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84dc0cccb0 0x7f84dc148040 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:20.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.883+0000 7f84e195b640 1 -- 172.21.15.67:0/2945068706 >> 172.21.15.67:0/2945068706 conn(0x7f84dc09aea0 msgr2=0x7f84dc0b2790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:20.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.883+0000 7f84e195b640 1 -- 172.21.15.67:0/2945068706 shutdown_connections 2024-03-24T00:46:20.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.883+0000 7f84e195b640 1 -- 172.21.15.67:0/2945068706 wait complete. 2024-03-24T00:46:20.887 INFO:tasks.ceph.mds.c.smithi067.stderr: -4164> 2024-03-24T00:45:15.651+0000 7fdf3050a640 -1 log_channel(cluster) log [ERR] : dir 0x10000000000 object missing on disk; some files may be lost (/dir) 2024-03-24T00:46:20.890 INFO:tasks.ceph.mds.c.smithi067.stderr: -4002> 2024-03-24T00:45:15.841+0000 7fdf3050a640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000000 [...2,head] /dir/ auth v15 ap=1 f(v0 m2024-03-24T00:45:06.076992+0000 1=1+0) n(v0 rc2024-03-24T00:45:06.190990+0000 2=1+1) (iversion lock) | dirfrag=0 authpin=1 scrubqueue=0 0x56393232bb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(13)0x10000000000:[<0x1/dir v15>]//[]","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f(v0 m2024-03-24T00:45:06.076992+0000 1=1+0)","memory_value.rstat":"n(v0 rc2024-03-24T00:45:06.190990+0000 2=1+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":-2} 2024-03-24T00:46:20.890 INFO:tasks.ceph.mds.c.smithi067.stderr: -3951> 2024-03-24T00:45:16.115+0000 7fdf34512640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T00:46:20.893 INFO:tasks.ceph.mds.c.smithi067.stderr: -2243> 2024-03-24T00:45:47.166+0000 7fdf3050a640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000000 [...2,head] /dir/ auth v15 pv18 ap=3 DIRTYPARENT f(v0 m2024-03-24T00:45:06.076992+0000 1=1+0)->f(v1 m2024-03-24T00:45:06.076992+0000) n(v0 rc2024-03-24T00:45:06.190990+0000 2=1+1)->n(v0 rc2024-03-24T00:45:06.190990+0000 2=1+1) (inest lock dirty) (ifile lock->sync w=1 flushing) (iversion lock) | dirtyscattered=2 lock=1 dirfrag=1 dirtyrstat=1 dirtyparent=1 waiter=1 authpin=1 scrubqueue=0 0x56393232bb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(13)0x10000000000:[<0x1/dir v15>]//[]","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f(v0 m2024-03-24T00:45:06.076992+0000 1=1+0)","memory_value.rstat":"n(v0 rc2024-03-24T00:45:06.190990+0000 2=1+1)","error_str":"freshly-calculated rstats don't match existing ones (will be fixed)"},"return_code":-2} 2024-03-24T00:46:20.896 DEBUG:teuthology.orchestra.run.smithi067:> cephfs-data-scan --debug-mds=20 --debug-ms=1 --debug-objecter=1 scan_extents cephfs_data 2024-03-24T00:46:20.916 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-03-24T00:46:20.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.935+0000 7ff6d5a2a740 1 -- start start 2024-03-24T00:46:20.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.935+0000 7ff6d5a2a740 1 Processor -- start 2024-03-24T00:46:20.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.935+0000 7ff6d5a2a740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x555b4e696c00 0x555b4e740580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:20.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.935+0000 7ff6d5a2a740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x555b4e697800 0x555b4e740b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:20.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.935+0000 7ff6d5a2a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555b4e697c00 0x555b4e741080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:20.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.935+0000 7ff6d5a2a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b4f47a600 con 0x555b4e697c00 2024-03-24T00:46:20.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.935+0000 7ff6d5a2a740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b4f47a480 con 0x555b4e697800 2024-03-24T00:46:20.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.935+0000 7ff6d5a2a740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x555b4f47a300 con 0x555b4e696c00 2024-03-24T00:46:20.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.936+0000 7ff6d3b0f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555b4e697c00 0x555b4e741080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:20.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.936+0000 7ff6d3b0f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555b4e697c00 0x555b4e741080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55410/0 (socket says 172.21.15.67:55410) 2024-03-24T00:46:20.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.936+0000 7ff6d3b0f640 1 -- 172.21.15.67:0/4186184975 learned_addr learned my addr 172.21.15.67:0/4186184975 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:20.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.936+0000 7ff6d2b0d640 1 --2- 172.21.15.67:0/4186184975 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x555b4e697800 0x555b4e740b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:20.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.936+0000 7ff6d330e640 1 --2- 172.21.15.67:0/4186184975 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x555b4e696c00 0x555b4e740580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:20.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.936+0000 7ff6d3b0f640 1 -- 172.21.15.67:0/4186184975 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x555b4e696c00 msgr2=0x555b4e740580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:20.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.936+0000 7ff6d3b0f640 1 --2- 172.21.15.67:0/4186184975 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x555b4e696c00 0x555b4e740580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:20.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.936+0000 7ff6d3b0f640 1 -- 172.21.15.67:0/4186184975 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x555b4e697800 msgr2=0x555b4e740b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:20.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.936+0000 7ff6d3b0f640 1 --2- 172.21.15.67:0/4186184975 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x555b4e697800 0x555b4e740b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:20.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.936+0000 7ff6d3b0f640 1 -- 172.21.15.67:0/4186184975 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555b4e6c3680 con 0x555b4e697c00 2024-03-24T00:46:20.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.936+0000 7ff6d3b0f640 1 --2- 172.21.15.67:0/4186184975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555b4e697c00 0x555b4e741080 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x555b4f4b78c0 tx=0x555b4f436060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4574e623e58ffe9a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:20.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.936+0000 7ff6d0b09640 1 -- 172.21.15.67:0/4186184975 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x555b4e6bc680 con 0x555b4e697c00 2024-03-24T00:46:20.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.936+0000 7ff6d0b09640 1 -- 172.21.15.67:0/4186184975 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x555b4e6e0c40 con 0x555b4e697c00 2024-03-24T00:46:20.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.936+0000 7ff6d0b09640 1 -- 172.21.15.67:0/4186184975 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x555b4e6bcb60 con 0x555b4e697c00 2024-03-24T00:46:20.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.936+0000 7ff6d5a2a740 1 -- 172.21.15.67:0/4186184975 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555b4e6c34a0 con 0x555b4e697c00 2024-03-24T00:46:20.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.936+0000 7ff6d5a2a740 1 -- 172.21.15.67:0/4186184975 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x555b4e6c3a40 con 0x555b4e697c00 2024-03-24T00:46:20.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.936+0000 7ff6d5a2a740 4 waiting for MDS map... 2024-03-24T00:46:20.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.936+0000 7ff6d0b09640 1 -- 172.21.15.67:0/4186184975 <== mon.0 v2:172.21.15.67:3300/0 4 ==== osd_map(93..93 src has 1..93) v4 ==== 6389+0+0 (secure 0 0 0) 0x555b4f574600 con 0x555b4e697c00 2024-03-24T00:46:20.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.936+0000 7ff6d0b09640 1 -- 172.21.15.67:0/4186184975 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 62) v1 ==== 2735+0+0 (secure 0 0 0) 0x555b4e6ef180 con 0x555b4e697c00 2024-03-24T00:46:20.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.936+0000 7ff6d5a2a740 4 Got MDS map 62 2024-03-24T00:46:20.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.937+0000 7ff6d5a2a740 4 datascan.main: Using metadata pool output 2024-03-24T00:46:20.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.937+0000 7ff6d5a2a740 4 datascan.main: connecting to RADOS... 2024-03-24T00:46:20.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.937+0000 7ff6d5a2a740 1 Processor -- start 2024-03-24T00:46:20.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.937+0000 7ff6d5a2a740 1 -- start start 2024-03-24T00:46:20.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.938+0000 7ff6d5a2a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555b4f56d400 0x555b4e741600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:20.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.938+0000 7ff6d5a2a740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x555b4f56d800 0x555b4e741b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:20.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.938+0000 7ff6d5a2a740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x555b4f56dc00 0x555b4e742100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:20.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.938+0000 7ff6d5a2a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b4f47a180 con 0x555b4f56d400 2024-03-24T00:46:20.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.938+0000 7ff6d5a2a740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b4e6f6480 con 0x555b4f56d800 2024-03-24T00:46:20.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.938+0000 7ff6d5a2a740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x555b4e6f6900 con 0x555b4f56dc00 2024-03-24T00:46:20.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.938+0000 7ff6d2b0d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555b4f56d400 0x555b4e741600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:20.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.938+0000 7ff6d2b0d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555b4f56d400 0x555b4e741600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55426/0 (socket says 172.21.15.67:55426) 2024-03-24T00:46:20.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.938+0000 7ff6d2b0d640 1 -- 172.21.15.67:0/3202378236 learned_addr learned my addr 172.21.15.67:0/3202378236 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:20.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.938+0000 7ff6d330e640 1 --2- 172.21.15.67:0/3202378236 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x555b4f56d800 0x555b4e741b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:20.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.938+0000 7ff6d2b0d640 1 --2- 172.21.15.67:0/3202378236 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x555b4f56dc00 0x555b4e742100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:20.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.938+0000 7ff6d2b0d640 1 -- 172.21.15.67:0/3202378236 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x555b4f56dc00 msgr2=0x555b4e742100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:20.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.938+0000 7ff6d2b0d640 1 --2- 172.21.15.67:0/3202378236 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x555b4f56dc00 0x555b4e742100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:20.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.938+0000 7ff6d2b0d640 1 -- 172.21.15.67:0/3202378236 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x555b4f56d800 msgr2=0x555b4e741b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:20.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.938+0000 7ff6d2b0d640 1 --2- 172.21.15.67:0/3202378236 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x555b4f56d800 0x555b4e741b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:20.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.938+0000 7ff6d2b0d640 1 -- 172.21.15.67:0/3202378236 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555b4e6c3860 con 0x555b4f56d400 2024-03-24T00:46:20.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.938+0000 7ff6d330e640 1 --2- 172.21.15.67:0/3202378236 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x555b4f56d800 0x555b4e741b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:46:20.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.938+0000 7ff6d2b0d640 1 --2- 172.21.15.67:0/3202378236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555b4f56d400 0x555b4e741600 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x555b4f49ccc0 tx=0x555b4f432ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdfe0337071576af server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:20.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.938+0000 7ff6ceb05640 1 -- 172.21.15.67:0/3202378236 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x555b4e6bc9c0 con 0x555b4f56d400 2024-03-24T00:46:20.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.938+0000 7ff6ceb05640 1 -- 172.21.15.67:0/3202378236 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x555b4e6e0a80 con 0x555b4f56d400 2024-03-24T00:46:20.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.938+0000 7ff6ceb05640 1 -- 172.21.15.67:0/3202378236 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x555b4e6bc680 con 0x555b4f56d400 2024-03-24T00:46:20.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.939+0000 7ff6d5a2a740 1 -- 172.21.15.67:0/3202378236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555b4f56d400 msgr2=0x555b4e741600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:20.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.939+0000 7ff6d5a2a740 1 --2- 172.21.15.67:0/3202378236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555b4f56d400 0x555b4e741600 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x555b4f49ccc0 tx=0x555b4f432ae0 comp rx=0 tx=0).stop 2024-03-24T00:46:20.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.939+0000 7ff6d5a2a740 1 -- 172.21.15.67:0/3202378236 shutdown_connections 2024-03-24T00:46:20.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.939+0000 7ff6d5a2a740 1 --2- 172.21.15.67:0/3202378236 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x555b4f56dc00 0x555b4e742100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:20.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.939+0000 7ff6d5a2a740 1 --2- 172.21.15.67:0/3202378236 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x555b4f56d800 0x555b4e741b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:20.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.939+0000 7ff6d5a2a740 1 --2- 172.21.15.67:0/3202378236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555b4f56d400 0x555b4e741600 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:20.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.939+0000 7ff6d5a2a740 1 -- 172.21.15.67:0/3202378236 >> 172.21.15.67:0/3202378236 conn(0x555b4f56d000 msgr2=0x555b4f41c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:20.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.939+0000 7ff6d5a2a740 1 -- 172.21.15.67:0/3202378236 shutdown_connections 2024-03-24T00:46:20.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.939+0000 7ff6d5a2a740 1 -- 172.21.15.67:0/3202378236 wait complete. 2024-03-24T00:46:20.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.939+0000 7ff6d5a2a740 1 Processor -- start 2024-03-24T00:46:20.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.939+0000 7ff6d5a2a740 1 -- start start 2024-03-24T00:46:20.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.940+0000 7ff6d5a2a740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x555b4f56d400 0x555b4e743180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:20.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.940+0000 7ff6d5a2a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555b4f56d800 0x555b4e742c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:20.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.940+0000 7ff6d5a2a740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x555b4f56dc00 0x555b4e742680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:20.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.940+0000 7ff6d5a2a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b4f47b200 con 0x555b4f56d800 2024-03-24T00:46:20.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.940+0000 7ff6d5a2a740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b4f47a180 con 0x555b4f56dc00 2024-03-24T00:46:20.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.940+0000 7ff6d5a2a740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x555b4f47b380 con 0x555b4f56d400 2024-03-24T00:46:20.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.940+0000 7ff6d330e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555b4f56d800 0x555b4e742c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:20.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.940+0000 7ff6d330e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555b4f56d800 0x555b4e742c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55440/0 (socket says 172.21.15.67:55440) 2024-03-24T00:46:20.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.940+0000 7ff6d330e640 1 -- 172.21.15.67:0/3299235368 learned_addr learned my addr 172.21.15.67:0/3299235368 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:20.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.940+0000 7ff6d2b0d640 1 --2- 172.21.15.67:0/3299235368 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x555b4f56d400 0x555b4e743180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:20.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.940+0000 7ff6d2b0d640 1 --2- 172.21.15.67:0/3299235368 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x555b4f56dc00 0x555b4e742680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:20.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.940+0000 7ff6d330e640 1 -- 172.21.15.67:0/3299235368 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x555b4f56d400 msgr2=0x555b4e743180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:20.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.940+0000 7ff6d330e640 1 --2- 172.21.15.67:0/3299235368 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x555b4f56d400 0x555b4e743180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:20.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.940+0000 7ff6d330e640 1 -- 172.21.15.67:0/3299235368 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x555b4f56dc00 msgr2=0x555b4e742680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:20.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.940+0000 7ff6d330e640 1 --2- 172.21.15.67:0/3299235368 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x555b4f56dc00 0x555b4e742680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:20.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.940+0000 7ff6d330e640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555b4e6c3680 con 0x555b4f56d800 2024-03-24T00:46:20.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.940+0000 7ff6d2b0d640 1 --2- 172.21.15.67:0/3299235368 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x555b4f56d400 0x555b4e743180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:46:20.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.941+0000 7ff6d330e640 1 --2- 172.21.15.67:0/3299235368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555b4f56d800 0x555b4e742c00 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x555b4f49da10 tx=0x555b4f579800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:20.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.941+0000 7ff6cd302640 1 -- 172.21.15.67:0/3299235368 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x555b4e6bc9c0 con 0x555b4f56d800 2024-03-24T00:46:20.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.941+0000 7ff6cd302640 1 -- 172.21.15.67:0/3299235368 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x555b4e6e0540 con 0x555b4f56d800 2024-03-24T00:46:20.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.941+0000 7ff6d5a2a740 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x555b4e6c34a0 con 0x555b4f56d800 2024-03-24T00:46:20.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.941+0000 7ff6cd302640 1 -- 172.21.15.67:0/3299235368 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x555b4e6bc680 con 0x555b4f56d800 2024-03-24T00:46:20.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.941+0000 7ff6d5a2a740 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555b4e6c32c0 con 0x555b4f56d800 2024-03-24T00:46:20.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.941+0000 7ff6d5a2a740 4 datascan.init: resolving metadata pool 13 2024-03-24T00:46:20.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.942+0000 7ff6cd302640 1 -- 172.21.15.67:0/3299235368 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x555b4f5e0c00 con 0x555b4f56d800 2024-03-24T00:46:20.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.942+0000 7ff6cd302640 1 --2- 172.21.15.67:0/3299235368 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x555b4f5ecc00 0x555b4e743700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:20.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.942+0000 7ff6cd302640 1 -- 172.21.15.67:0/3299235368 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6389+0+0 (secure 0 0 0) 0x555b4f5eea00 con 0x555b4f56d800 2024-03-24T00:46:20.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.942+0000 7ff6d5a2a740 4 datascan.init: found metadata pool 'cephfs_metadata' 2024-03-24T00:46:20.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.942+0000 7ff6d5a2a740 4 datascan.main: data pool 'cephfs_data' has ID 14 2024-03-24T00:46:20.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.942+0000 7ff6d5a2a740 4 datascan.main: opening data pool 'cephfs_data' 2024-03-24T00:46:20.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.942+0000 7ff6d5a2a740 1 --2- 172.21.15.67:0/3299235368 >> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] conn(0x555b4f5ed000 0x555b4e742100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:20.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.942+0000 7ff6d5a2a740 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] -- osd_op(unknown.0.0:1 14.0 14:00000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4f5ed400 con 0x555b4f5ed000 2024-03-24T00:46:20.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.943+0000 7ff6d2b0d640 1 --2- 172.21.15.67:0/3299235368 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x555b4f5ecc00 0x555b4e743700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:20.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.943+0000 7ff6d2b0d640 1 --2- 172.21.15.67:0/3299235368 >> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] conn(0x555b4f5ed000 0x555b4e742100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:20.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.943+0000 7ff6d2b0d640 1 --2- 172.21.15.67:0/3299235368 >> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] conn(0x555b4f5ed000 0x555b4e742100 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:20.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.943+0000 7ff6d2b0d640 1 --2- 172.21.15.67:0/3299235368 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x555b4f5ecc00 0x555b4e743700 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x555b4f49ced0 tx=0x555b4f436ae0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:20.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.945+0000 7ff6d2b0d640 1 -- 172.21.15.67:0/3299235368 <== osd.2 v2:172.21.15.67:6817/1268432561 1 ==== osd_op_reply(1 [pgnls start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e000 con 0x555b4f5ed000 2024-03-24T00:46:20.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.945+0000 7ff6ce304640 1 --2- 172.21.15.67:0/3299235368 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x555b4ea24000 0x555b4ea2a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:20.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.945+0000 7ff6ce304640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:2 14.20 14:04000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea24400 con 0x555b4ea24000 2024-03-24T00:46:20.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.945+0000 7ff6d3b0f640 1 --2- 172.21.15.67:0/3299235368 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x555b4ea24000 0x555b4ea2a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:20.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.945+0000 7ff6d3b0f640 1 --2- 172.21.15.67:0/3299235368 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x555b4ea24000 0x555b4ea2a000 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:20.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.945+0000 7ff6d3b0f640 1 -- 172.21.15.67:0/3299235368 <== osd.0 v2:172.21.15.67:6808/2628130519 1 ==== osd_op_reply(2 [pgnls start_epoch 93 out=49b] v89'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e500 con 0x555b4ea24000 2024-03-24T00:46:20.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.946+0000 7ff6ceb05640 1 --2- 172.21.15.67:0/3299235368 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x555b4ea24c00 0x555b4ea2a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:20.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.946+0000 7ff6ceb05640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:3 14.10 14:08000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea25000 con 0x555b4ea24c00 2024-03-24T00:46:20.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.946+0000 7ff6d330e640 1 --2- 172.21.15.67:0/3299235368 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x555b4ea24c00 0x555b4ea2a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:20.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.946+0000 7ff6d330e640 1 --2- 172.21.15.67:0/3299235368 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x555b4ea24c00 0x555b4ea2a580 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:20.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.948+0000 7ff6d330e640 1 -- 172.21.15.67:0/3299235368 <== osd.1 v2:172.21.15.67:6800/4062510404 1 ==== osd_op_reply(3 [pgnls start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e780 con 0x555b4ea24c00 2024-03-24T00:46:20.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.948+0000 7ff6ce304640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:4 14.30 14:0c000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea24800 con 0x555b4ea24000 2024-03-24T00:46:20.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.948+0000 7ff6d3b0f640 1 -- 172.21.15.67:0/3299235368 <== osd.0 v2:172.21.15.67:6808/2628130519 2 ==== osd_op_reply(4 [pgnls start_epoch 93 out=49b] v89'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e500 con 0x555b4ea24000 2024-03-24T00:46:20.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.948+0000 7ff6ceb05640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:5 14.8 14:10000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea25400 con 0x555b4ea24000 2024-03-24T00:46:20.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.948+0000 7ff6d3b0f640 1 -- 172.21.15.67:0/3299235368 <== osd.0 v2:172.21.15.67:6808/2628130519 3 ==== osd_op_reply(5 [pgnls start_epoch 93 out=49b] v89'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e500 con 0x555b4ea24000 2024-03-24T00:46:20.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.948+0000 7ff6ce304640 1 --2- 172.21.15.67:0/3299235368 >> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] conn(0x555b4ea25800 0x555b4ea2ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:20.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.948+0000 7ff6ce304640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:6 14.28 14:14000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea25c00 con 0x555b4ea25800 2024-03-24T00:46:20.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.948+0000 7ff6d2b0d640 1 --2- 172.21.15.67:0/3299235368 >> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] conn(0x555b4ea25800 0x555b4ea2ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:20.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.949+0000 7ff6d2b0d640 1 --2- 172.21.15.67:0/3299235368 >> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] conn(0x555b4ea25800 0x555b4ea2ab00 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:20.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.950+0000 7ff6d2b0d640 1 -- 172.21.15.67:0/3299235368 <== osd.3 v2:172.21.15.67:6816/3303489481 1 ==== osd_op_reply(6 [pgnls start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e000 con 0x555b4ea25800 2024-03-24T00:46:20.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.950+0000 7ff6ceb05640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:7 14.18 14:18000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea3c000 con 0x555b4ea24c00 2024-03-24T00:46:20.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.951+0000 7ff6d330e640 1 -- 172.21.15.67:0/3299235368 <== osd.1 v2:172.21.15.67:6800/4062510404 2 ==== osd_op_reply(7 [pgnls start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e780 con 0x555b4ea24c00 2024-03-24T00:46:20.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.951+0000 7ff6ce304640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] -- osd_op(unknown.0.0:8 14.38 14:1c000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea3cc00 con 0x555b4f5ed000 2024-03-24T00:46:20.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.951+0000 7ff6d2b0d640 1 -- 172.21.15.67:0/3299235368 <== osd.2 v2:172.21.15.67:6817/1268432561 2 ==== osd_op_reply(8 [pgnls start_epoch 93 out=49b] v89'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e000 con 0x555b4f5ed000 2024-03-24T00:46:20.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.951+0000 7ff6ceb05640 1 --2- 172.21.15.67:0/3299235368 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x555b4ea3c400 0x555b4ea2b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:20.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.951+0000 7ff6ceb05640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:9 14.4 14:20000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea3c800 con 0x555b4ea3c400 2024-03-24T00:46:20.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.951+0000 7ff6d3b0f640 1 --2- 172.21.15.67:0/3299235368 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x555b4ea3c400 0x555b4ea2b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:20.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.952+0000 7ff6d3b0f640 1 --2- 172.21.15.67:0/3299235368 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x555b4ea3c400 0x555b4ea2b600 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:20.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.954+0000 7ff6d3b0f640 1 -- 172.21.15.67:0/3299235368 <== osd.6 v2:172.21.15.72:6816/1725206562 1 ==== osd_op_reply(9 [pgnls start_epoch 93 out=49b] v89'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e500 con 0x555b4ea3c400 2024-03-24T00:46:20.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.954+0000 7ff6ce304640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:10 14.24 14:24000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea3d000 con 0x555b4ea24c00 2024-03-24T00:46:20.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.954+0000 7ff6d330e640 1 -- 172.21.15.67:0/3299235368 <== osd.1 v2:172.21.15.67:6800/4062510404 3 ==== osd_op_reply(10 [pgnls start_epoch 93 out=49b] v89'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e780 con 0x555b4ea24c00 2024-03-24T00:46:20.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.954+0000 7ff6ceb05640 1 --2- 172.21.15.67:0/3299235368 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x555b4ea24400 0x555b4ea2bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:20.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.954+0000 7ff6ceb05640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:11 14.14 14:28000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea24800 con 0x555b4ea24400 2024-03-24T00:46:20.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.954+0000 7ff6d330e640 1 --2- 172.21.15.67:0/3299235368 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x555b4ea24400 0x555b4ea2bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:20.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.955+0000 7ff6d330e640 1 --2- 172.21.15.67:0/3299235368 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x555b4ea24400 0x555b4ea2bb80 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:20.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.955+0000 7ff6d330e640 1 -- 172.21.15.67:0/3299235368 <== osd.4 v2:172.21.15.72:6808/477220793 1 ==== osd_op_reply(11 [pgnls start_epoch 93 out=49b] v89'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e780 con 0x555b4ea24400 2024-03-24T00:46:20.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.955+0000 7ff6ce304640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] -- osd_op(unknown.0.0:12 14.34 14:2c000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea3d400 con 0x555b4f5ed000 2024-03-24T00:46:20.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.955+0000 7ff6d2b0d640 1 -- 172.21.15.67:0/3299235368 <== osd.2 v2:172.21.15.67:6817/1268432561 3 ==== osd_op_reply(12 [pgnls start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e000 con 0x555b4f5ed000 2024-03-24T00:46:20.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.955+0000 7ff6ceb05640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:13 14.c 14:30000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea25400 con 0x555b4ea24000 2024-03-24T00:46:20.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.955+0000 7ff6d3b0f640 1 -- 172.21.15.67:0/3299235368 <== osd.0 v2:172.21.15.67:6808/2628130519 4 ==== osd_op_reply(13 [pgnls start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e500 con 0x555b4ea24000 2024-03-24T00:46:20.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.955+0000 7ff6ce304640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:14 14.2c 14:34000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea3d800 con 0x555b4ea24c00 2024-03-24T00:46:20.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.956+0000 7ff6d330e640 1 -- 172.21.15.67:0/3299235368 <== osd.1 v2:172.21.15.67:6800/4062510404 4 ==== osd_op_reply(14 [pgnls start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e780 con 0x555b4ea24c00 2024-03-24T00:46:20.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.956+0000 7ff6ceb05640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:15 14.1c 14:38000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4f56c800 con 0x555b4ea24000 2024-03-24T00:46:20.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.956+0000 7ff6d3b0f640 1 -- 172.21.15.67:0/3299235368 <== osd.0 v2:172.21.15.67:6808/2628130519 5 ==== osd_op_reply(15 [pgnls start_epoch 93 out=49b] v89'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e500 con 0x555b4ea24000 2024-03-24T00:46:20.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.956+0000 7ff6ce304640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:16 14.3c 14:3c000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea25c00 con 0x555b4ea24c00 2024-03-24T00:46:20.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.956+0000 7ff6d330e640 1 -- 172.21.15.67:0/3299235368 <== osd.1 v2:172.21.15.67:6800/4062510404 5 ==== osd_op_reply(16 [pgnls start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e780 con 0x555b4ea24c00 2024-03-24T00:46:20.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.956+0000 7ff6ceb05640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:17 14.2 14:40000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea3cc00 con 0x555b4ea3c400 2024-03-24T00:46:20.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.956+0000 7ff6d3b0f640 1 -- 172.21.15.67:0/3299235368 <== osd.6 v2:172.21.15.72:6816/1725206562 2 ==== osd_op_reply(17 [pgnls start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e500 con 0x555b4ea3c400 2024-03-24T00:46:20.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.956+0000 7ff6ce304640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:18 14.22 14:44000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea25000 con 0x555b4ea24400 2024-03-24T00:46:20.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.957+0000 7ff6d330e640 1 -- 172.21.15.67:0/3299235368 <== osd.4 v2:172.21.15.72:6808/477220793 2 ==== osd_op_reply(18 [pgnls start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e780 con 0x555b4ea24400 2024-03-24T00:46:20.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.957+0000 7ff6ceb05640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] -- osd_op(unknown.0.0:19 14.12 14:48000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea3d400 con 0x555b4f5ed000 2024-03-24T00:46:20.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.957+0000 7ff6d2b0d640 1 -- 172.21.15.67:0/3299235368 <== osd.2 v2:172.21.15.67:6817/1268432561 4 ==== osd_op_reply(19 [pgnls start_epoch 93 out=49b] v89'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e000 con 0x555b4f5ed000 2024-03-24T00:46:20.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.957+0000 7ff6ce304640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:20 14.32 14:4c000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea24800 con 0x555b4ea24400 2024-03-24T00:46:20.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.957+0000 7ff6d330e640 1 -- 172.21.15.67:0/3299235368 <== osd.4 v2:172.21.15.72:6808/477220793 3 ==== osd_op_reply(20 [pgnls start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e780 con 0x555b4ea24400 2024-03-24T00:46:20.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.957+0000 7ff6ceb05640 1 --2- 172.21.15.67:0/3299235368 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x555b4ea3c000 0x555b4ea2c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:20.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.957+0000 7ff6ceb05640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:21 14.a 14:50000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea3d000 con 0x555b4ea3c000 2024-03-24T00:46:20.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.958+0000 7ff6d2b0d640 1 --2- 172.21.15.67:0/3299235368 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x555b4ea3c000 0x555b4ea2c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:20.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.958+0000 7ff6d2b0d640 1 --2- 172.21.15.67:0/3299235368 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x555b4ea3c000 0x555b4ea2c100 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:20.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.958+0000 7ff6d2b0d640 1 -- 172.21.15.67:0/3299235368 <== osd.5 v2:172.21.15.72:6800/288486206 1 ==== osd_op_reply(21 [pgnls start_epoch 93 out=49b] v89'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e000 con 0x555b4ea3c000 2024-03-24T00:46:20.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.958+0000 7ff6ce304640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:22 14.2a 14:54000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4f5ed400 con 0x555b4ea24400 2024-03-24T00:46:20.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.958+0000 7ff6d330e640 1 -- 172.21.15.67:0/3299235368 <== osd.4 v2:172.21.15.72:6808/477220793 4 ==== osd_op_reply(22 [pgnls start_epoch 93 out=49b] v89'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e780 con 0x555b4ea24400 2024-03-24T00:46:20.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.958+0000 7ff6ceb05640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:23 14.1a 14:58000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea3c800 con 0x555b4ea24000 2024-03-24T00:46:20.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.959+0000 7ff6d3b0f640 1 -- 172.21.15.67:0/3299235368 <== osd.0 v2:172.21.15.67:6808/2628130519 6 ==== osd_op_reply(23 [pgnls start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e500 con 0x555b4ea24000 2024-03-24T00:46:20.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.959+0000 7ff6ce304640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:24 14.3a 14:5c000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4f47d400 con 0x555b4ea24000 2024-03-24T00:46:20.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.959+0000 7ff6d3b0f640 1 -- 172.21.15.67:0/3299235368 <== osd.0 v2:172.21.15.67:6808/2628130519 7 ==== osd_op_reply(24 [pgnls start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e500 con 0x555b4ea24000 2024-03-24T00:46:20.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.959+0000 7ff6ceb05640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:25 14.6 14:60000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea3d800 con 0x555b4ea24000 2024-03-24T00:46:20.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.959+0000 7ff6d3b0f640 1 -- 172.21.15.67:0/3299235368 <== osd.0 v2:172.21.15.67:6808/2628130519 8 ==== osd_op_reply(25 [pgnls start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e500 con 0x555b4ea24000 2024-03-24T00:46:20.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.959+0000 7ff6ce304640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:26 14.26 14:64000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea3cc00 con 0x555b4ea24400 2024-03-24T00:46:20.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.959+0000 7ff6d330e640 1 -- 172.21.15.67:0/3299235368 <== osd.4 v2:172.21.15.72:6808/477220793 5 ==== osd_op_reply(26 [pgnls start_epoch 93 out=49b] v89'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e780 con 0x555b4ea24400 2024-03-24T00:46:20.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.959+0000 7ff6ceb05640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:27 14.16 14:68000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea3dc00 con 0x555b4ea25800 2024-03-24T00:46:20.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.960+0000 7ff6d2b0d640 1 -- 172.21.15.67:0/3299235368 <== osd.3 v2:172.21.15.67:6816/3303489481 2 ==== osd_op_reply(27 [pgnls start_epoch 93 out=49b] v89'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e000 con 0x555b4ea25800 2024-03-24T00:46:20.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.960+0000 7ff6ce304640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] -- osd_op(unknown.0.0:28 14.36 14:6c000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea3c800 con 0x555b4f5ed000 2024-03-24T00:46:20.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.960+0000 7ff6d2b0d640 1 -- 172.21.15.67:0/3299235368 <== osd.2 v2:172.21.15.67:6817/1268432561 5 ==== osd_op_reply(28 [pgnls start_epoch 93 out=49b] v89'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e000 con 0x555b4f5ed000 2024-03-24T00:46:20.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.960+0000 7ff6ceb05640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:29 14.e 14:70000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4f5ed400 con 0x555b4ea24400 2024-03-24T00:46:20.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.960+0000 7ff6d330e640 1 -- 172.21.15.67:0/3299235368 <== osd.4 v2:172.21.15.72:6808/477220793 6 ==== osd_op_reply(29 [pgnls start_epoch 93 out=49b] v89'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e780 con 0x555b4ea24400 2024-03-24T00:46:20.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.960+0000 7ff6ce304640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:30 14.2e 14:74000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea3d800 con 0x555b4ea25800 2024-03-24T00:46:20.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.961+0000 7ff6d2b0d640 1 -- 172.21.15.67:0/3299235368 <== osd.3 v2:172.21.15.67:6816/3303489481 3 ==== osd_op_reply(30 [pgnls start_epoch 93 out=49b] v89'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e000 con 0x555b4ea25800 2024-03-24T00:46:20.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.961+0000 7ff6ceb05640 1 --2- 172.21.15.67:0/3299235368 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x555b4ea25c00 0x555b4ea2c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:20.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.961+0000 7ff6ceb05640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:31 14.1e 14:78000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea25000 con 0x555b4ea25c00 2024-03-24T00:46:20.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.961+0000 7ff6d3b0f640 1 --2- 172.21.15.67:0/3299235368 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x555b4ea25c00 0x555b4ea2c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:20.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.961+0000 7ff6d3b0f640 1 --2- 172.21.15.67:0/3299235368 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x555b4ea25c00 0x555b4ea2c680 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:20.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.963+0000 7ff6d3b0f640 1 -- 172.21.15.67:0/3299235368 <== osd.7 v2:172.21.15.72:6824/2893038746 1 ==== osd_op_reply(31 [pgnls start_epoch 93 out=49b] v89'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e500 con 0x555b4ea25c00 2024-03-24T00:46:20.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.963+0000 7ff6ce304640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:32 14.3e 14:7c000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4f47d400 con 0x555b4ea3c000 2024-03-24T00:46:20.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.964+0000 7ff6d2b0d640 1 -- 172.21.15.67:0/3299235368 <== osd.5 v2:172.21.15.72:6800/288486206 2 ==== osd_op_reply(32 [pgnls start_epoch 93 out=49b] v89'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e000 con 0x555b4ea3c000 2024-03-24T00:46:20.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.964+0000 7ff6ceb05640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:33 14.1 14:80000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea24800 con 0x555b4ea25800 2024-03-24T00:46:20.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.964+0000 7ff6d2b0d640 1 -- 172.21.15.67:0/3299235368 <== osd.3 v2:172.21.15.67:6816/3303489481 4 ==== osd_op_reply(33 [pgnls start_epoch 93 out=49b] v89'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e000 con 0x555b4ea25800 2024-03-24T00:46:20.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.964+0000 7ff6ce304640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:34 14.21 14:84000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4f56c800 con 0x555b4ea3c000 2024-03-24T00:46:20.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.964+0000 7ff6d2b0d640 1 -- 172.21.15.67:0/3299235368 <== osd.5 v2:172.21.15.72:6800/288486206 3 ==== osd_op_reply(34 [pgnls start_epoch 93 out=49b] v89'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e000 con 0x555b4ea3c000 2024-03-24T00:46:20.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.964+0000 7ff6ceb05640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:35 14.11 14:88000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4f47d400 con 0x555b4ea24400 2024-03-24T00:46:20.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.965+0000 7ff6d330e640 1 -- 172.21.15.67:0/3299235368 <== osd.4 v2:172.21.15.72:6808/477220793 7 ==== osd_op_reply(35 [pgnls start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e780 con 0x555b4ea24400 2024-03-24T00:46:20.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.965+0000 7ff6ce304640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:36 14.31 14:8c000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea25400 con 0x555b4ea24c00 2024-03-24T00:46:20.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.965+0000 7ff6d330e640 1 -- 172.21.15.67:0/3299235368 <== osd.1 v2:172.21.15.67:6800/4062510404 6 ==== osd_op_reply(36 [pgnls start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e780 con 0x555b4ea24c00 2024-03-24T00:46:20.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.965+0000 7ff6ceb05640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:37 14.9 14:90000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea3d400 con 0x555b4ea3c000 2024-03-24T00:46:20.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.965+0000 7ff6d2b0d640 1 -- 172.21.15.67:0/3299235368 <== osd.5 v2:172.21.15.72:6800/288486206 4 ==== osd_op_reply(37 [pgnls start_epoch 93 out=49b] v89'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e000 con 0x555b4ea3c000 2024-03-24T00:46:20.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.965+0000 7ff6ce304640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:38 14.29 14:94000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea56000 con 0x555b4ea3c400 2024-03-24T00:46:20.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.965+0000 7ff6d3b0f640 1 -- 172.21.15.67:0/3299235368 <== osd.6 v2:172.21.15.72:6816/1725206562 3 ==== osd_op_reply(38 [pgnls start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e500 con 0x555b4ea3c400 2024-03-24T00:46:20.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.965+0000 7ff6ceb05640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:39 14.19 14:98000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea3d000 con 0x555b4ea3c400 2024-03-24T00:46:20.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.966+0000 7ff6d3b0f640 1 -- 172.21.15.67:0/3299235368 <== osd.6 v2:172.21.15.72:6816/1725206562 4 ==== osd_op_reply(39 [pgnls start_epoch 93 out=49b] v89'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e500 con 0x555b4ea3c400 2024-03-24T00:46:20.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.966+0000 7ff6ce304640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:40 14.39 14:9c000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea56400 con 0x555b4ea25c00 2024-03-24T00:46:20.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.966+0000 7ff6d3b0f640 1 -- 172.21.15.67:0/3299235368 <== osd.7 v2:172.21.15.72:6824/2893038746 2 ==== osd_op_reply(40 [pgnls start_epoch 93 out=49b] v89'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e500 con 0x555b4ea25c00 2024-03-24T00:46:20.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.966+0000 7ff6ceb05640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:41 14.5 14:a0000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea3dc00 con 0x555b4ea24000 2024-03-24T00:46:20.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.966+0000 7ff6d3b0f640 1 -- 172.21.15.67:0/3299235368 <== osd.0 v2:172.21.15.67:6808/2628130519 9 ==== osd_op_reply(41 [pgnls start_epoch 93 out=49b] v89'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e500 con 0x555b4ea24000 2024-03-24T00:46:20.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.966+0000 7ff6ce304640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] -- osd_op(unknown.0.0:42 14.25 14:a4000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea56800 con 0x555b4f5ed000 2024-03-24T00:46:20.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.966+0000 7ff6d2b0d640 1 -- 172.21.15.67:0/3299235368 <== osd.2 v2:172.21.15.67:6817/1268432561 6 ==== osd_op_reply(42 [pgnls start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e000 con 0x555b4f5ed000 2024-03-24T00:46:20.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.967+0000 7ff6ceb05640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:43 14.15 14:a8000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea3c800 con 0x555b4ea25c00 2024-03-24T00:46:20.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.967+0000 7ff6d3b0f640 1 -- 172.21.15.67:0/3299235368 <== osd.7 v2:172.21.15.72:6824/2893038746 3 ==== osd_op_reply(43 [pgnls start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e500 con 0x555b4ea25c00 2024-03-24T00:46:20.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.967+0000 7ff6ce304640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:44 14.35 14:ac000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea56c00 con 0x555b4ea24400 2024-03-24T00:46:20.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.967+0000 7ff6d330e640 1 -- 172.21.15.67:0/3299235368 <== osd.4 v2:172.21.15.72:6808/477220793 8 ==== osd_op_reply(44 [pgnls start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e780 con 0x555b4ea24400 2024-03-24T00:46:20.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.967+0000 7ff6ceb05640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:45 14.d 14:b0000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea3d800 con 0x555b4ea3c400 2024-03-24T00:46:20.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.967+0000 7ff6d3b0f640 1 -- 172.21.15.67:0/3299235368 <== osd.6 v2:172.21.15.72:6816/1725206562 5 ==== osd_op_reply(45 [pgnls start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e500 con 0x555b4ea3c400 2024-03-24T00:46:20.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.967+0000 7ff6ce304640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:46 14.2d 14:b4000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea57000 con 0x555b4ea24400 2024-03-24T00:46:20.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.968+0000 7ff6d330e640 1 -- 172.21.15.67:0/3299235368 <== osd.4 v2:172.21.15.72:6808/477220793 9 ==== osd_op_reply(46 [pgnls start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e780 con 0x555b4ea24400 2024-03-24T00:46:20.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.968+0000 7ff6ceb05640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:47 14.1d 14:b8000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea3cc00 con 0x555b4ea24000 2024-03-24T00:46:20.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.968+0000 7ff6d3b0f640 1 -- 172.21.15.67:0/3299235368 <== osd.0 v2:172.21.15.67:6808/2628130519 10 ==== osd_op_reply(47 [pgnls start_epoch 93 out=49b] v89'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e500 con 0x555b4ea24000 2024-03-24T00:46:20.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.968+0000 7ff6ce304640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:48 14.3d 14:bc000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea57400 con 0x555b4ea25800 2024-03-24T00:46:20.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.968+0000 7ff6d2b0d640 1 -- 172.21.15.67:0/3299235368 <== osd.3 v2:172.21.15.67:6816/3303489481 5 ==== osd_op_reply(48 [pgnls start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e000 con 0x555b4ea25800 2024-03-24T00:46:20.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.968+0000 7ff6ceb05640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:49 14.3 14:c0000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea3d000 con 0x555b4ea24000 2024-03-24T00:46:20.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.968+0000 7ff6d3b0f640 1 -- 172.21.15.67:0/3299235368 <== osd.0 v2:172.21.15.67:6808/2628130519 11 ==== osd_op_reply(49 [pgnls start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e500 con 0x555b4ea24000 2024-03-24T00:46:20.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.969+0000 7ff6ce304640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:50 14.23 14:c4000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea57800 con 0x555b4ea25c00 2024-03-24T00:46:20.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.969+0000 7ff6d3b0f640 1 -- 172.21.15.67:0/3299235368 <== osd.7 v2:172.21.15.72:6824/2893038746 4 ==== osd_op_reply(50 [pgnls start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e500 con 0x555b4ea25c00 2024-03-24T00:46:20.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.969+0000 7ff6ceb05640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] -- osd_op(unknown.0.0:51 14.13 14:c8000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea3dc00 con 0x555b4f5ed000 2024-03-24T00:46:20.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.969+0000 7ff6d2b0d640 1 -- 172.21.15.67:0/3299235368 <== osd.2 v2:172.21.15.67:6817/1268432561 7 ==== osd_op_reply(51 [pgnls start_epoch 93 out=49b] v89'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e000 con 0x555b4f5ed000 2024-03-24T00:46:20.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.969+0000 7ff6ce304640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:52 14.33 14:cc000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea56c00 con 0x555b4ea24c00 2024-03-24T00:46:20.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.969+0000 7ff6d330e640 1 -- 172.21.15.67:0/3299235368 <== osd.1 v2:172.21.15.67:6800/4062510404 7 ==== osd_op_reply(52 [pgnls start_epoch 93 out=49b] v89'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e780 con 0x555b4ea24c00 2024-03-24T00:46:20.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.970+0000 7ff6ceb05640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:53 14.b 14:d0000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea3c800 con 0x555b4ea24000 2024-03-24T00:46:20.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.970+0000 7ff6d3b0f640 1 -- 172.21.15.67:0/3299235368 <== osd.0 v2:172.21.15.67:6808/2628130519 12 ==== osd_op_reply(53 [pgnls start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e500 con 0x555b4ea24000 2024-03-24T00:46:20.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.970+0000 7ff6ce304640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:54 14.2b 14:d4000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea57000 con 0x555b4ea24400 2024-03-24T00:46:20.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.970+0000 7ff6d330e640 1 -- 172.21.15.67:0/3299235368 <== osd.4 v2:172.21.15.72:6808/477220793 10 ==== osd_op_reply(54 [pgnls start_epoch 93 out=49b] v89'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e780 con 0x555b4ea24400 2024-03-24T00:46:20.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.970+0000 7ff6ceb05640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:55 14.1b 14:d8000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea25400 con 0x555b4ea24c00 2024-03-24T00:46:20.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.971+0000 7ff6d330e640 1 -- 172.21.15.67:0/3299235368 <== osd.1 v2:172.21.15.67:6800/4062510404 8 ==== osd_op_reply(55 [pgnls start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e780 con 0x555b4ea24c00 2024-03-24T00:46:20.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.971+0000 7ff6ce304640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:56 14.3b 14:dc000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea56000 con 0x555b4ea24400 2024-03-24T00:46:20.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.971+0000 7ff6d330e640 1 -- 172.21.15.67:0/3299235368 <== osd.4 v2:172.21.15.72:6808/477220793 11 ==== osd_op_reply(56 [pgnls start_epoch 93 out=49b] v89'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e780 con 0x555b4ea24400 2024-03-24T00:46:20.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.971+0000 7ff6ceb05640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:57 14.7 14:e0000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea25000 con 0x555b4ea3c000 2024-03-24T00:46:20.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.971+0000 7ff6d2b0d640 1 -- 172.21.15.67:0/3299235368 <== osd.5 v2:172.21.15.72:6800/288486206 5 ==== osd_op_reply(57 [pgnls start_epoch 93 out=81b] v89'6 uv6 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x555b4e80e000 con 0x555b4ea3c000 2024-03-24T00:46:20.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.971+0000 7ff6d5a2a740 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:58 14.7 14:e2e07930:::10000000001.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4f5ed800 con 0x555b4ea3c000 2024-03-24T00:46:20.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.972+0000 7ff6d2b0d640 1 -- 172.21.15.67:0/3299235368 <== osd.5 v2:172.21.15.72:6800/288486206 6 ==== osd_op_reply(58 10000000001.00000000 [stat out=16b] v0'0 uv6 ondisk = 0) v8 ==== 164+0+16 (crc 0 0 0) 0x555b4e80e000 con 0x555b4ea3c000 2024-03-24T00:46:20.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.972+0000 7ff6d5a2a740 10 datascan.operator(): handling object 1099511627777.0 2024-03-24T00:46:20.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.972+0000 7ff6d5a2a740 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:59 14.7 14:e2e07930:::10000000001.00000000:head [call cephfs.accumulate_inode_metadata in=112b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4f5edc00 con 0x555b4ea3c000 2024-03-24T00:46:20.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.973+0000 7ff6d2b0d640 1 -- 172.21.15.67:0/3299235368 <== osd.5 v2:172.21.15.72:6800/288486206 7 ==== osd_op_reply(59 10000000001.00000000 [call] v93'7 uv7 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x555b4e80e000 con 0x555b4ea3c000 2024-03-24T00:46:20.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.973+0000 7ff6d5a2a740 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:60 14.27 14:e4000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea24800 con 0x555b4ea3c000 2024-03-24T00:46:20.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.974+0000 7ff6d2b0d640 1 -- 172.21.15.67:0/3299235368 <== osd.5 v2:172.21.15.72:6800/288486206 8 ==== osd_op_reply(60 [pgnls start_epoch 93 out=49b] v89'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e000 con 0x555b4ea3c000 2024-03-24T00:46:20.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.974+0000 7ff6ceb05640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:61 14.17 14:e8000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4f47d400 con 0x555b4ea24c00 2024-03-24T00:46:20.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.974+0000 7ff6d330e640 1 -- 172.21.15.67:0/3299235368 <== osd.1 v2:172.21.15.67:6800/4062510404 9 ==== osd_op_reply(61 [pgnls start_epoch 93 out=49b] v89'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e780 con 0x555b4ea24c00 2024-03-24T00:46:20.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.974+0000 7ff6ce304640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:62 14.37 14:ec000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea56400 con 0x555b4ea24400 2024-03-24T00:46:20.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.974+0000 7ff6d330e640 1 -- 172.21.15.67:0/3299235368 <== osd.4 v2:172.21.15.72:6808/477220793 12 ==== osd_op_reply(62 [pgnls start_epoch 93 out=49b] v89'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e780 con 0x555b4ea24400 2024-03-24T00:46:20.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.974+0000 7ff6ceb05640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:63 14.f 14:f0000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4f5ed400 con 0x555b4ea25c00 2024-03-24T00:46:20.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.975+0000 7ff6d3b0f640 1 -- 172.21.15.67:0/3299235368 <== osd.7 v2:172.21.15.72:6824/2893038746 5 ==== osd_op_reply(63 [pgnls start_epoch 93 out=49b] v89'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e500 con 0x555b4ea25c00 2024-03-24T00:46:20.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.975+0000 7ff6ce304640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:64 14.2f 14:f4000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea57c00 con 0x555b4ea24000 2024-03-24T00:46:20.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.975+0000 7ff6d3b0f640 1 -- 172.21.15.67:0/3299235368 <== osd.0 v2:172.21.15.67:6808/2628130519 13 ==== osd_op_reply(64 [pgnls start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e500 con 0x555b4ea24000 2024-03-24T00:46:20.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.975+0000 7ff6ceb05640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:65 14.1f 14:f8000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea25000 con 0x555b4ea24000 2024-03-24T00:46:20.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.975+0000 7ff6d3b0f640 1 -- 172.21.15.67:0/3299235368 <== osd.0 v2:172.21.15.67:6808/2628130519 14 ==== osd_op_reply(65 [pgnls start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e500 con 0x555b4ea24000 2024-03-24T00:46:20.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.975+0000 7ff6ce304640 1 -- 172.21.15.67:0/3299235368 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:66 14.3f 14:fc000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x555b4ea3d800 con 0x555b4ea25800 2024-03-24T00:46:20.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.975+0000 7ff6d2b0d640 1 -- 172.21.15.67:0/3299235368 <== osd.3 v2:172.21.15.67:6816/3303489481 6 ==== osd_op_reply(66 [pgnls start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555b4e80e000 con 0x555b4ea25800 2024-03-24T00:46:20.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.976+0000 7ff6d5a2a740 1 -- 172.21.15.67:0/3299235368 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x555b4ea24000 msgr2=0x555b4ea2a000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:20.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.976+0000 7ff6d5a2a740 1 --2- 172.21.15.67:0/3299235368 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x555b4ea24000 0x555b4ea2a000 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:20.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.976+0000 7ff6d5a2a740 1 -- 172.21.15.67:0/3299235368 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x555b4ea24c00 msgr2=0x555b4ea2a580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:20.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.976+0000 7ff6d5a2a740 1 --2- 172.21.15.67:0/3299235368 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x555b4ea24c00 0x555b4ea2a580 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:20.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.976+0000 7ff6d5a2a740 1 -- 172.21.15.67:0/3299235368 >> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] conn(0x555b4f5ed000 msgr2=0x555b4e742100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:20.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.976+0000 7ff6d5a2a740 1 --2- 172.21.15.67:0/3299235368 >> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] conn(0x555b4f5ed000 0x555b4e742100 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:20.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.976+0000 7ff6d5a2a740 1 -- 172.21.15.67:0/3299235368 >> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] conn(0x555b4ea25800 msgr2=0x555b4ea2ab00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:20.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.976+0000 7ff6d5a2a740 1 --2- 172.21.15.67:0/3299235368 >> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] conn(0x555b4ea25800 0x555b4ea2ab00 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:20.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.976+0000 7ff6d5a2a740 1 -- 172.21.15.67:0/3299235368 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x555b4ea24400 msgr2=0x555b4ea2bb80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:20.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.976+0000 7ff6d5a2a740 1 --2- 172.21.15.67:0/3299235368 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x555b4ea24400 0x555b4ea2bb80 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:20.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.976+0000 7ff6d5a2a740 1 -- 172.21.15.67:0/3299235368 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x555b4ea3c000 msgr2=0x555b4ea2c100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:20.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.976+0000 7ff6d5a2a740 1 --2- 172.21.15.67:0/3299235368 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x555b4ea3c000 0x555b4ea2c100 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:20.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.976+0000 7ff6d5a2a740 1 -- 172.21.15.67:0/3299235368 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x555b4ea3c400 msgr2=0x555b4ea2b600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:20.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.976+0000 7ff6d330e640 1 -- 172.21.15.67:0/3299235368 reap_dead start 2024-03-24T00:46:20.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.976+0000 7ff6d5a2a740 1 --2- 172.21.15.67:0/3299235368 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x555b4ea3c400 0x555b4ea2b600 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:20.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.976+0000 7ff6d330e640 1 -- 172.21.15.67:0/3299235368 reap_dead start 2024-03-24T00:46:20.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.976+0000 7ff6d330e640 1 -- 172.21.15.67:0/3299235368 reap_dead start 2024-03-24T00:46:20.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.976+0000 7ff6d5a2a740 1 -- 172.21.15.67:0/3299235368 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x555b4ea25c00 msgr2=0x555b4ea2c680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:20.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.976+0000 7ff6d5a2a740 1 --2- 172.21.15.67:0/3299235368 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x555b4ea25c00 0x555b4ea2c680 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:20.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.976+0000 7ff6d5a2a740 1 -- 172.21.15.67:0/3299235368 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x555b4f5ecc00 msgr2=0x555b4e743700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:20.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.976+0000 7ff6d5a2a740 1 --2- 172.21.15.67:0/3299235368 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x555b4f5ecc00 0x555b4e743700 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x555b4f49ced0 tx=0x555b4f436ae0 comp rx=0 tx=0).stop 2024-03-24T00:46:20.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.976+0000 7ff6d5a2a740 1 -- 172.21.15.67:0/3299235368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555b4f56d800 msgr2=0x555b4e742c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:20.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.976+0000 7ff6d5a2a740 1 --2- 172.21.15.67:0/3299235368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555b4f56d800 0x555b4e742c00 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x555b4f49da10 tx=0x555b4f579800 comp rx=0 tx=0).stop 2024-03-24T00:46:20.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.976+0000 7ff6d5a2a740 1 -- 172.21.15.67:0/3299235368 shutdown_connections 2024-03-24T00:46:20.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.976+0000 7ff6d5a2a740 1 --2- 172.21.15.67:0/3299235368 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x555b4ea3c400 0x555b4ea2b600 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:20.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.976+0000 7ff6d5a2a740 1 --2- 172.21.15.67:0/3299235368 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x555b4f5ecc00 0x555b4e743700 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:20.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.976+0000 7ff6d5a2a740 1 --2- 172.21.15.67:0/3299235368 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x555b4ea25c00 0x555b4ea2c680 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:20.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.976+0000 7ff6d5a2a740 1 --2- 172.21.15.67:0/3299235368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555b4f56d800 0x555b4e742c00 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:20.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.976+0000 7ff6d5a2a740 1 -- 172.21.15.67:0/3299235368 >> 172.21.15.67:0/3299235368 conn(0x555b4f56d000 msgr2=0x555b4f41c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:20.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.976+0000 7ff6d5a2a740 1 -- 172.21.15.67:0/3299235368 shutdown_connections 2024-03-24T00:46:20.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.976+0000 7ff6d5a2a740 1 -- 172.21.15.67:0/3299235368 wait complete. 2024-03-24T00:46:20.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.976+0000 7ff6d5a2a740 1 -- 172.21.15.67:0/4186184975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555b4e697c00 msgr2=0x555b4e741080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:20.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.976+0000 7ff6d5a2a740 1 --2- 172.21.15.67:0/4186184975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555b4e697c00 0x555b4e741080 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x555b4f4b78c0 tx=0x555b4f436060 comp rx=0 tx=0).stop 2024-03-24T00:46:20.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.976+0000 7ff6d5a2a740 1 -- 172.21.15.67:0/4186184975 shutdown_connections 2024-03-24T00:46:20.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.976+0000 7ff6d5a2a740 1 --2- 172.21.15.67:0/4186184975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555b4e697c00 0x555b4e741080 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:20.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.976+0000 7ff6d5a2a740 1 --2- 172.21.15.67:0/4186184975 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x555b4e697800 0x555b4e740b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:20.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.976+0000 7ff6d5a2a740 1 --2- 172.21.15.67:0/4186184975 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x555b4e696c00 0x555b4e740580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:20.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.976+0000 7ff6d5a2a740 1 -- 172.21.15.67:0/4186184975 >> 172.21.15.67:0/4186184975 conn(0x555b4e697000 msgr2=0x555b4f41d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:20.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.976+0000 7ff6d5a2a740 1 -- 172.21.15.67:0/4186184975 shutdown_connections 2024-03-24T00:46:20.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:20.976+0000 7ff6d5a2a740 1 -- 172.21.15.67:0/4186184975 wait complete. 2024-03-24T00:46:20.986 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-data-scan', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'scan_extents', 'cephfs_data'] in time 0:00:00.089412, result: 2024-03-24T00:46:20.986 DEBUG:teuthology.orchestra.run.smithi067:> cephfs-data-scan --debug-mds=20 --debug-ms=1 --debug-objecter=1 scan_inodes cephfs_data 2024-03-24T00:46:21.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.059+0000 7f7570c14740 1 -- start start 2024-03-24T00:46:21.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.059+0000 7f7570c14740 1 Processor -- start 2024-03-24T00:46:21.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.060+0000 7f7570c14740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56157fa04800 0x56157ecd9b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:21.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.060+0000 7f7570c14740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56157ec2fc00 0x56157ecd8b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:21.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.060+0000 7f7570c14740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56157fa05c00 0x56157ecd9600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:21.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.060+0000 7f7570c14740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x56157fa0c000 con 0x56157ec2fc00 2024-03-24T00:46:21.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.060+0000 7f7570c14740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x56157fa0c180 con 0x56157fa05c00 2024-03-24T00:46:21.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.060+0000 7f7570c14740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x56157fa0c300 con 0x56157fa04800 2024-03-24T00:46:21.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.060+0000 7f756e27a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56157ec2fc00 0x56157ecd8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:21.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.060+0000 7f756e27a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56157ec2fc00 0x56157ecd8b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55448/0 (socket says 172.21.15.67:55448) 2024-03-24T00:46:21.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.060+0000 7f756e27a640 1 -- 172.21.15.67:0/4047905513 learned_addr learned my addr 172.21.15.67:0/4047905513 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:21.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.060+0000 7f756ea7b640 1 --2- 172.21.15.67:0/4047905513 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56157fa04800 0x56157ecd9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:21.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.060+0000 7f756f27c640 1 --2- 172.21.15.67:0/4047905513 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56157fa05c00 0x56157ecd9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:21.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.060+0000 7f756e27a640 1 -- 172.21.15.67:0/4047905513 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56157fa04800 msgr2=0x56157ecd9b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:21.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.060+0000 7f756e27a640 1 --2- 172.21.15.67:0/4047905513 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56157fa04800 0x56157ecd9b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.060+0000 7f756e27a640 1 -- 172.21.15.67:0/4047905513 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56157fa05c00 msgr2=0x56157ecd9600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:21.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.060+0000 7f756e27a640 1 --2- 172.21.15.67:0/4047905513 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56157fa05c00 0x56157ecd9600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.060+0000 7f756e27a640 1 -- 172.21.15.67:0/4047905513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56157ec5b680 con 0x56157ec2fc00 2024-03-24T00:46:21.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.061+0000 7f756e27a640 1 --2- 172.21.15.67:0/4047905513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56157ec2fc00 0x56157ecd8b00 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x56157fa61920 tx=0x56157f9ce000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc6c75e797b2ed98 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:21.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.061+0000 7f756c276640 1 -- 172.21.15.67:0/4047905513 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x56157ec54d00 con 0x56157ec2fc00 2024-03-24T00:46:21.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.061+0000 7f756c276640 1 -- 172.21.15.67:0/4047905513 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56157ec781c0 con 0x56157ec2fc00 2024-03-24T00:46:21.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.061+0000 7f756c276640 1 -- 172.21.15.67:0/4047905513 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x56157ec549c0 con 0x56157ec2fc00 2024-03-24T00:46:21.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.061+0000 7f7570c14740 1 -- 172.21.15.67:0/4047905513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56157ec5b2c0 con 0x56157ec2fc00 2024-03-24T00:46:21.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.061+0000 7f7570c14740 1 -- 172.21.15.67:0/4047905513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x56157ec5ba40 con 0x56157ec2fc00 2024-03-24T00:46:21.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.061+0000 7f7570c14740 4 waiting for MDS map... 2024-03-24T00:46:21.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.061+0000 7f756c276640 1 -- 172.21.15.67:0/4047905513 <== mon.0 v2:172.21.15.67:3300/0 4 ==== osd_map(93..93 src has 1..93) v4 ==== 6389+0+0 (secure 0 0 0) 0x56157fa48600 con 0x56157ec2fc00 2024-03-24T00:46:21.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.061+0000 7f756c276640 1 -- 172.21.15.67:0/4047905513 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 62) v1 ==== 2735+0+0 (secure 0 0 0) 0x56157ec87180 con 0x56157ec2fc00 2024-03-24T00:46:21.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.061+0000 7f7570c14740 4 Got MDS map 62 2024-03-24T00:46:21.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.061+0000 7f7570c14740 4 datascan.main: Using metadata pool output 2024-03-24T00:46:21.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.061+0000 7f7570c14740 4 datascan.main: connecting to RADOS... 2024-03-24T00:46:21.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.062+0000 7f7570c14740 1 Processor -- start 2024-03-24T00:46:21.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.062+0000 7f7570c14740 1 -- start start 2024-03-24T00:46:21.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.062+0000 7f7570c14740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56157fa04000 0x56157ecd9080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:21.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.062+0000 7f7570c14740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56157fb07800 0x56157ecda100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:21.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.062+0000 7f7570c14740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56157fb07c00 0x56157ecda680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:21.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.062+0000 7f7570c14740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x56157fa0c480 con 0x56157fb07c00 2024-03-24T00:46:21.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.062+0000 7f7570c14740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x56157fa0c600 con 0x56157fb07800 2024-03-24T00:46:21.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.062+0000 7f7570c14740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x56157fa0c780 con 0x56157fa04000 2024-03-24T00:46:21.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.062+0000 7f756e27a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56157fb07c00 0x56157ecda680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:21.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.062+0000 7f756e27a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56157fb07c00 0x56157ecda680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55452/0 (socket says 172.21.15.67:55452) 2024-03-24T00:46:21.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.062+0000 7f756e27a640 1 -- 172.21.15.67:0/1786243249 learned_addr learned my addr 172.21.15.67:0/1786243249 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:21.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.062+0000 7f756ea7b640 1 --2- 172.21.15.67:0/1786243249 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56157fb07800 0x56157ecda100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:21.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.062+0000 7f756f27c640 1 --2- 172.21.15.67:0/1786243249 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56157fa04000 0x56157ecd9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:21.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.062+0000 7f756e27a640 1 -- 172.21.15.67:0/1786243249 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56157fa04000 msgr2=0x56157ecd9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:21.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.062+0000 7f756e27a640 1 --2- 172.21.15.67:0/1786243249 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56157fa04000 0x56157ecd9080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.062+0000 7f756e27a640 1 -- 172.21.15.67:0/1786243249 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56157fb07800 msgr2=0x56157ecda100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:21.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.062+0000 7f756e27a640 1 --2- 172.21.15.67:0/1786243249 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56157fb07800 0x56157ecda100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.062+0000 7f756e27a640 1 -- 172.21.15.67:0/1786243249 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56157ec5a3c0 con 0x56157fb07c00 2024-03-24T00:46:21.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.063+0000 7f756e27a640 1 --2- 172.21.15.67:0/1786243249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56157fb07c00 0x56157ecda680 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x56157fb20db0 tx=0x56157f9ca6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e177a62366a287e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:21.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.063+0000 7f756a272640 1 -- 172.21.15.67:0/1786243249 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x56157ec544e0 con 0x56157fb07c00 2024-03-24T00:46:21.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.063+0000 7f756a272640 1 -- 172.21.15.67:0/1786243249 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56157ec78a80 con 0x56157fb07c00 2024-03-24T00:46:21.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.063+0000 7f756a272640 1 -- 172.21.15.67:0/1786243249 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x56157ec54d00 con 0x56157fb07c00 2024-03-24T00:46:21.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.063+0000 7f7570c14740 1 -- 172.21.15.67:0/1786243249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56157fb07c00 msgr2=0x56157ecda680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:21.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.063+0000 7f7570c14740 1 --2- 172.21.15.67:0/1786243249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56157fb07c00 0x56157ecda680 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x56157fb20db0 tx=0x56157f9ca6c0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.063+0000 7f7570c14740 1 -- 172.21.15.67:0/1786243249 shutdown_connections 2024-03-24T00:46:21.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.063+0000 7f7570c14740 1 --2- 172.21.15.67:0/1786243249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56157fb07c00 0x56157ecda680 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.063+0000 7f7570c14740 1 --2- 172.21.15.67:0/1786243249 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56157fb07800 0x56157ecda100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.063+0000 7f7570c14740 1 --2- 172.21.15.67:0/1786243249 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56157fa04000 0x56157ecd9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.063+0000 7f7570c14740 1 -- 172.21.15.67:0/1786243249 >> 172.21.15.67:0/1786243249 conn(0x56157fa04400 msgr2=0x56157f9b7800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:21.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.063+0000 7f7570c14740 1 -- 172.21.15.67:0/1786243249 shutdown_connections 2024-03-24T00:46:21.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.063+0000 7f7570c14740 1 -- 172.21.15.67:0/1786243249 wait complete. 2024-03-24T00:46:21.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.064+0000 7f7570c14740 1 Processor -- start 2024-03-24T00:46:21.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.064+0000 7f7570c14740 1 -- start start 2024-03-24T00:46:21.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.064+0000 7f7570c14740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56157fa04000 0x56157ecdb700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:21.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.064+0000 7f7570c14740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56157fb07800 0x56157ecdb180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:21.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.064+0000 7f7570c14740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56157fb07c00 0x56157ecdac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:21.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.064+0000 7f7570c14740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x56157fa0de00 con 0x56157fb07800 2024-03-24T00:46:21.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.064+0000 7f7570c14740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x56157fa0dc80 con 0x56157fa04000 2024-03-24T00:46:21.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.064+0000 7f7570c14740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x56157fa0db00 con 0x56157fb07c00 2024-03-24T00:46:21.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.064+0000 7f756ea7b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56157fb07800 0x56157ecdb180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:21.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.064+0000 7f756ea7b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56157fb07800 0x56157ecdb180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55458/0 (socket says 172.21.15.67:55458) 2024-03-24T00:46:21.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.064+0000 7f756ea7b640 1 -- 172.21.15.67:0/312922798 learned_addr learned my addr 172.21.15.67:0/312922798 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:21.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.064+0000 7f756f27c640 1 --2- 172.21.15.67:0/312922798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56157fa04000 0x56157ecdb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:21.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.064+0000 7f756e27a640 1 --2- 172.21.15.67:0/312922798 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56157fb07c00 0x56157ecdac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:21.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.065+0000 7f756ea7b640 1 -- 172.21.15.67:0/312922798 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56157fb07c00 msgr2=0x56157ecdac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:21.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.065+0000 7f756ea7b640 1 --2- 172.21.15.67:0/312922798 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56157fb07c00 0x56157ecdac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.065+0000 7f756ea7b640 1 -- 172.21.15.67:0/312922798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56157fa04000 msgr2=0x56157ecdb700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:21.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.065+0000 7f756ea7b640 1 --2- 172.21.15.67:0/312922798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56157fa04000 0x56157ecdb700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.065+0000 7f756ea7b640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56157ec5b680 con 0x56157fb07800 2024-03-24T00:46:21.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.065+0000 7f756f27c640 1 --2- 172.21.15.67:0/312922798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56157fa04000 0x56157ecdb700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:46:21.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.065+0000 7f756ea7b640 1 --2- 172.21.15.67:0/312922798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56157fb07800 0x56157ecdb180 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x56157fb21b90 tx=0x56157fb23a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:21.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.065+0000 7f7568a6f640 1 -- 172.21.15.67:0/312922798 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x56157ec544e0 con 0x56157fb07800 2024-03-24T00:46:21.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.065+0000 7f7568a6f640 1 -- 172.21.15.67:0/312922798 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56157ec79340 con 0x56157fb07800 2024-03-24T00:46:21.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.065+0000 7f7570c14740 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x56157ec5b2c0 con 0x56157fb07800 2024-03-24T00:46:21.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.065+0000 7f7568a6f640 1 -- 172.21.15.67:0/312922798 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x56157ec54d00 con 0x56157fb07800 2024-03-24T00:46:21.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.065+0000 7f7570c14740 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56157ec5ab40 con 0x56157fb07800 2024-03-24T00:46:21.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.065+0000 7f7570c14740 4 datascan.init: resolving metadata pool 13 2024-03-24T00:46:21.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.066+0000 7f7568a6f640 1 -- 172.21.15.67:0/312922798 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x56157fb74c00 con 0x56157fb07800 2024-03-24T00:46:21.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.066+0000 7f7568a6f640 1 --2- 172.21.15.67:0/312922798 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x56157fb78c00 0x56157efb0000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:21.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.066+0000 7f7568a6f640 1 -- 172.21.15.67:0/312922798 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6389+0+0 (secure 0 0 0) 0x56157fb7aa00 con 0x56157fb07800 2024-03-24T00:46:21.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.067+0000 7f7570c14740 4 datascan.init: found metadata pool 'cephfs_metadata' 2024-03-24T00:46:21.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.067+0000 7f7570c14740 4 datascan.main: data pool 'cephfs_data' has ID 14 2024-03-24T00:46:21.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.067+0000 7f7570c14740 4 datascan.main: opening data pool 'cephfs_data' 2024-03-24T00:46:21.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.067+0000 7f756f27c640 1 --2- 172.21.15.67:0/312922798 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x56157fb78c00 0x56157efb0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:21.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.067+0000 7f7570c14740 1 --2- 172.21.15.67:0/312922798 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x56157fb79000 0x56157ecda680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:21.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.067+0000 7f7570c14740 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:1 13.14 13:287034c4:::1.00000000.inode:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157fb79400 con 0x56157fb79000 2024-03-24T00:46:21.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.067+0000 7f756e27a640 1 --2- 172.21.15.67:0/312922798 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x56157fb79000 0x56157ecda680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:21.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.067+0000 7f756e27a640 1 --2- 172.21.15.67:0/312922798 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x56157fb79000 0x56157ecda680 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:21.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.067+0000 7f756f27c640 1 --2- 172.21.15.67:0/312922798 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x56157fb78c00 0x56157efb0000 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x56157f9cd140 tx=0x56157f9caa80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:21.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.067+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.0 v2:172.21.15.67:6808/2628130519 1 ==== osd_op_reply(1 1.00000000.inode [stat out=16b] v0'0 uv24 ondisk = 0) v8 ==== 160+0+16 (crc 0 0 0) 0x56157eda6000 con 0x56157fb79000 2024-03-24T00:46:21.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.067+0000 7f7570c14740 1 --2- 172.21.15.67:0/312922798 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x56157fb79800 0x56157ecda100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:21.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.067+0000 7f7570c14740 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:2 13.1f 13:fe07bba1:::100.00000000.inode:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157fb79c00 con 0x56157fb79800 2024-03-24T00:46:21.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.068+0000 7f756f27c640 1 --2- 172.21.15.67:0/312922798 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x56157fb79800 0x56157ecda100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:21.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.068+0000 7f756f27c640 1 --2- 172.21.15.67:0/312922798 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x56157fb79800 0x56157ecda100 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:21.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.068+0000 7f756f27c640 1 -- 172.21.15.67:0/312922798 <== osd.6 v2:172.21.15.72:6816/1725206562 1 ==== osd_op_reply(2 100.00000000.inode [stat out=16b] v0'0 uv3 ondisk = 0) v8 ==== 162+0+16 (crc 0 0 0) 0x56157eda6500 con 0x56157fb79800 2024-03-24T00:46:21.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.068+0000 7f7570c14740 1 --2- 172.21.15.67:0/312922798 >> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] conn(0x56157efc4000 0x56157ecd9080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:21.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.068+0000 7f7570c14740 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] -- osd_op(unknown.0.0:3 14.0 14:00000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efc4400 con 0x56157efc4000 2024-03-24T00:46:21.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.068+0000 7f756ea7b640 1 --2- 172.21.15.67:0/312922798 >> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] conn(0x56157efc4000 0x56157ecd9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:21.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.068+0000 7f756ea7b640 1 --2- 172.21.15.67:0/312922798 >> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] conn(0x56157efc4000 0x56157ecd9080 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:21.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.069+0000 7f756ea7b640 1 -- 172.21.15.67:0/312922798 <== osd.2 v2:172.21.15.67:6817/1268432561 1 ==== osd_op_reply(3 [pgnls-filter start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6780 con 0x56157efc4000 2024-03-24T00:46:21.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.069+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:4 14.20 14:04000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efc5c00 con 0x56157fb79000 2024-03-24T00:46:21.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.069+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.0 v2:172.21.15.67:6808/2628130519 2 ==== osd_op_reply(4 [pgnls-filter start_epoch 93 out=49b] v89'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157fb79000 2024-03-24T00:46:21.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.069+0000 7f756a272640 1 --2- 172.21.15.67:0/312922798 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x56157efca000 0x56157efb0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:21.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.069+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:5 14.10 14:08000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efca400 con 0x56157efca000 2024-03-24T00:46:21.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.069+0000 7f756e27a640 1 --2- 172.21.15.67:0/312922798 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x56157efca000 0x56157efb0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:21.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.070+0000 7f756e27a640 1 --2- 172.21.15.67:0/312922798 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x56157efca000 0x56157efb0580 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:21.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.070+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.1 v2:172.21.15.67:6800/4062510404 1 ==== osd_op_reply(5 [pgnls-filter start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157efca000 2024-03-24T00:46:21.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.070+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:6 14.30 14:0c000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157fb06c00 con 0x56157fb79000 2024-03-24T00:46:21.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.070+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.0 v2:172.21.15.67:6808/2628130519 3 ==== osd_op_reply(6 [pgnls-filter start_epoch 93 out=49b] v89'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157fb79000 2024-03-24T00:46:21.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.070+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:7 14.8 14:10000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efca800 con 0x56157fb79000 2024-03-24T00:46:21.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.070+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.0 v2:172.21.15.67:6808/2628130519 4 ==== osd_op_reply(7 [pgnls-filter start_epoch 93 out=49b] v89'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157fb79000 2024-03-24T00:46:21.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.070+0000 7f7569a71640 1 --2- 172.21.15.67:0/312922798 >> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] conn(0x56157fb07000 0x56157efb0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:21.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.070+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:8 14.28 14:14000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efca400 con 0x56157fb07000 2024-03-24T00:46:21.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.070+0000 7f756f27c640 1 --2- 172.21.15.67:0/312922798 >> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] conn(0x56157fb07000 0x56157efb0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:21.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.071+0000 7f756f27c640 1 --2- 172.21.15.67:0/312922798 >> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] conn(0x56157fb07000 0x56157efb0b00 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:21.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.071+0000 7f756f27c640 1 -- 172.21.15.67:0/312922798 <== osd.3 v2:172.21.15.67:6816/3303489481 1 ==== osd_op_reply(8 [pgnls-filter start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6500 con 0x56157fb07000 2024-03-24T00:46:21.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.071+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:9 14.18 14:18000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efcac00 con 0x56157efca000 2024-03-24T00:46:21.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.071+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.1 v2:172.21.15.67:6800/4062510404 2 ==== osd_op_reply(9 [pgnls-filter start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157efca000 2024-03-24T00:46:21.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.071+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] -- osd_op(unknown.0.0:10 14.38 14:1c000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efc5c00 con 0x56157efc4000 2024-03-24T00:46:21.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.071+0000 7f756ea7b640 1 -- 172.21.15.67:0/312922798 <== osd.2 v2:172.21.15.67:6817/1268432561 2 ==== osd_op_reply(10 [pgnls-filter start_epoch 93 out=49b] v89'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6780 con 0x56157efc4000 2024-03-24T00:46:21.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.071+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:11 14.4 14:20000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efcb000 con 0x56157fb79800 2024-03-24T00:46:21.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.072+0000 7f756f27c640 1 -- 172.21.15.67:0/312922798 <== osd.6 v2:172.21.15.72:6816/1725206562 2 ==== osd_op_reply(11 [pgnls-filter start_epoch 93 out=49b] v89'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6500 con 0x56157fb79800 2024-03-24T00:46:21.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.072+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:12 14.24 14:24000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157fb79400 con 0x56157efca000 2024-03-24T00:46:21.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.072+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.1 v2:172.21.15.67:6800/4062510404 3 ==== osd_op_reply(12 [pgnls-filter start_epoch 93 out=49b] v89'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157efca000 2024-03-24T00:46:21.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.072+0000 7f756a272640 1 --2- 172.21.15.67:0/312922798 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x56157efcb400 0x56157efb1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:21.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.072+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:13 14.14 14:28000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efcb800 con 0x56157efcb400 2024-03-24T00:46:21.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.072+0000 7f756ea7b640 1 --2- 172.21.15.67:0/312922798 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x56157efcb400 0x56157efb1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:21.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.073+0000 7f756ea7b640 1 --2- 172.21.15.67:0/312922798 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x56157efcb400 0x56157efb1080 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:21.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.073+0000 7f756ea7b640 1 -- 172.21.15.67:0/312922798 <== osd.4 v2:172.21.15.72:6808/477220793 1 ==== osd_op_reply(13 [pgnls-filter start_epoch 93 out=49b] v89'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6780 con 0x56157efcb400 2024-03-24T00:46:21.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.073+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] -- osd_op(unknown.0.0:14 14.34 14:2c000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efec000 con 0x56157efc4000 2024-03-24T00:46:21.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.073+0000 7f756ea7b640 1 -- 172.21.15.67:0/312922798 <== osd.2 v2:172.21.15.67:6817/1268432561 3 ==== osd_op_reply(14 [pgnls-filter start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6780 con 0x56157efc4000 2024-03-24T00:46:21.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.073+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:15 14.c 14:30000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efcbc00 con 0x56157fb79000 2024-03-24T00:46:21.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.073+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.0 v2:172.21.15.67:6808/2628130519 5 ==== osd_op_reply(15 [pgnls-filter start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157fb79000 2024-03-24T00:46:21.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.073+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:16 14.2c 14:34000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efec400 con 0x56157efca000 2024-03-24T00:46:21.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.074+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.1 v2:172.21.15.67:6800/4062510404 4 ==== osd_op_reply(16 [pgnls-filter start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157efca000 2024-03-24T00:46:21.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.074+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:17 14.1c 14:38000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efca800 con 0x56157fb79000 2024-03-24T00:46:21.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.074+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.0 v2:172.21.15.67:6808/2628130519 6 ==== osd_op_reply(17 [pgnls-filter start_epoch 93 out=49b] v89'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157fb79000 2024-03-24T00:46:21.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.074+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:18 14.3c 14:3c000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efec800 con 0x56157efca000 2024-03-24T00:46:21.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.074+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.1 v2:172.21.15.67:6800/4062510404 5 ==== osd_op_reply(18 [pgnls-filter start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157efca000 2024-03-24T00:46:21.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.074+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:19 14.2 14:40000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efc4400 con 0x56157fb79800 2024-03-24T00:46:21.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.074+0000 7f756f27c640 1 -- 172.21.15.67:0/312922798 <== osd.6 v2:172.21.15.72:6816/1725206562 3 ==== osd_op_reply(19 [pgnls-filter start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6500 con 0x56157fb79800 2024-03-24T00:46:21.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.074+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:20 14.22 14:44000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efecc00 con 0x56157efcb400 2024-03-24T00:46:21.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.075+0000 7f756ea7b640 1 -- 172.21.15.67:0/312922798 <== osd.4 v2:172.21.15.72:6808/477220793 2 ==== osd_op_reply(20 [pgnls-filter start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6780 con 0x56157efcb400 2024-03-24T00:46:21.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.075+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] -- osd_op(unknown.0.0:21 14.12 14:48000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efc5c00 con 0x56157efc4000 2024-03-24T00:46:21.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.075+0000 7f756ea7b640 1 -- 172.21.15.67:0/312922798 <== osd.2 v2:172.21.15.67:6817/1268432561 4 ==== osd_op_reply(21 [pgnls-filter start_epoch 93 out=49b] v89'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6780 con 0x56157efc4000 2024-03-24T00:46:21.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.075+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:22 14.32 14:4c000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efec400 con 0x56157efcb400 2024-03-24T00:46:21.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.075+0000 7f756ea7b640 1 -- 172.21.15.67:0/312922798 <== osd.4 v2:172.21.15.72:6808/477220793 3 ==== osd_op_reply(22 [pgnls-filter start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6780 con 0x56157efcb400 2024-03-24T00:46:21.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.075+0000 7f756a272640 1 --2- 172.21.15.67:0/312922798 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x56157fb79400 0x56157efb1600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:21.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.075+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:23 14.a 14:50000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157fb06c00 con 0x56157fb79400 2024-03-24T00:46:21.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.075+0000 7f756e27a640 1 --2- 172.21.15.67:0/312922798 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x56157fb79400 0x56157efb1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:21.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.076+0000 7f756e27a640 1 --2- 172.21.15.67:0/312922798 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x56157fb79400 0x56157efb1600 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:21.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.076+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.5 v2:172.21.15.72:6800/288486206 1 ==== osd_op_reply(23 [pgnls-filter start_epoch 93 out=49b] v89'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157fb79400 2024-03-24T00:46:21.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.076+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:24 14.2a 14:54000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efec000 con 0x56157efcb400 2024-03-24T00:46:21.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.076+0000 7f756ea7b640 1 -- 172.21.15.67:0/312922798 <== osd.4 v2:172.21.15.72:6808/477220793 4 ==== osd_op_reply(24 [pgnls-filter start_epoch 93 out=49b] v89'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6780 con 0x56157efcb400 2024-03-24T00:46:21.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.076+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:25 14.1a 14:58000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efed000 con 0x56157fb79000 2024-03-24T00:46:21.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.077+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.0 v2:172.21.15.67:6808/2628130519 7 ==== osd_op_reply(25 [pgnls-filter start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157fb79000 2024-03-24T00:46:21.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.077+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:26 14.3a 14:5c000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efcac00 con 0x56157fb79000 2024-03-24T00:46:21.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.077+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.0 v2:172.21.15.67:6808/2628130519 8 ==== osd_op_reply(26 [pgnls-filter start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157fb79000 2024-03-24T00:46:21.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.077+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:27 14.6 14:60000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efed400 con 0x56157fb79000 2024-03-24T00:46:21.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.077+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.0 v2:172.21.15.67:6808/2628130519 9 ==== osd_op_reply(27 [pgnls-filter start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157fb79000 2024-03-24T00:46:21.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.077+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:28 14.26 14:64000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efcbc00 con 0x56157efcb400 2024-03-24T00:46:21.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.077+0000 7f756ea7b640 1 -- 172.21.15.67:0/312922798 <== osd.4 v2:172.21.15.72:6808/477220793 5 ==== osd_op_reply(28 [pgnls-filter start_epoch 93 out=49b] v89'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6780 con 0x56157efcb400 2024-03-24T00:46:21.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.077+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:29 14.16 14:68000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efed800 con 0x56157fb07000 2024-03-24T00:46:21.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.078+0000 7f756f27c640 1 -- 172.21.15.67:0/312922798 <== osd.3 v2:172.21.15.67:6816/3303489481 2 ==== osd_op_reply(29 [pgnls-filter start_epoch 93 out=49b] v89'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6500 con 0x56157fb07000 2024-03-24T00:46:21.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.078+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] -- osd_op(unknown.0.0:30 14.36 14:6c000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efcb800 con 0x56157efc4000 2024-03-24T00:46:21.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.078+0000 7f756ea7b640 1 -- 172.21.15.67:0/312922798 <== osd.2 v2:172.21.15.67:6817/1268432561 5 ==== osd_op_reply(30 [pgnls-filter start_epoch 93 out=49b] v89'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6780 con 0x56157efc4000 2024-03-24T00:46:21.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.078+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:31 14.e 14:70000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efedc00 con 0x56157efcb400 2024-03-24T00:46:21.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.078+0000 7f756ea7b640 1 -- 172.21.15.67:0/312922798 <== osd.4 v2:172.21.15.72:6808/477220793 6 ==== osd_op_reply(31 [pgnls-filter start_epoch 93 out=49b] v89'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6780 con 0x56157efcb400 2024-03-24T00:46:21.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.078+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:32 14.2e 14:74000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efcbc00 con 0x56157fb07000 2024-03-24T00:46:21.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.079+0000 7f756f27c640 1 -- 172.21.15.67:0/312922798 <== osd.3 v2:172.21.15.67:6816/3303489481 3 ==== osd_op_reply(32 [pgnls-filter start_epoch 93 out=49b] v89'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6500 con 0x56157fb07000 2024-03-24T00:46:21.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.079+0000 7f756a272640 1 --2- 172.21.15.67:0/312922798 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x56157efca400 0x56157efb1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:21.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.079+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:33 14.1e 14:78000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efcb000 con 0x56157efca400 2024-03-24T00:46:21.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.079+0000 7f756f27c640 1 --2- 172.21.15.67:0/312922798 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x56157efca400 0x56157efb1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:21.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.079+0000 7f756f27c640 1 --2- 172.21.15.67:0/312922798 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x56157efca400 0x56157efb1b80 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:21.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.079+0000 7f756f27c640 1 -- 172.21.15.67:0/312922798 <== osd.7 v2:172.21.15.72:6824/2893038746 1 ==== osd_op_reply(33 [pgnls-filter start_epoch 93 out=49b] v89'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6500 con 0x56157efca400 2024-03-24T00:46:21.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.079+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:34 14.3e 14:7c000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efecc00 con 0x56157fb79400 2024-03-24T00:46:21.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.080+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.5 v2:172.21.15.72:6800/288486206 2 ==== osd_op_reply(34 [pgnls-filter start_epoch 93 out=49b] v89'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157fb79400 2024-03-24T00:46:21.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.080+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:35 14.1 14:80000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efcbc00 con 0x56157fb07000 2024-03-24T00:46:21.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.080+0000 7f756f27c640 1 -- 172.21.15.67:0/312922798 <== osd.3 v2:172.21.15.67:6816/3303489481 4 ==== osd_op_reply(35 [pgnls-filter start_epoch 93 out=49b] v89'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6500 con 0x56157fb07000 2024-03-24T00:46:21.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.080+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:36 14.21 14:84000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efec400 con 0x56157fb79400 2024-03-24T00:46:21.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.080+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.5 v2:172.21.15.72:6800/288486206 3 ==== osd_op_reply(36 [pgnls-filter start_epoch 93 out=49b] v89'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157fb79400 2024-03-24T00:46:21.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.080+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:37 14.11 14:88000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efed800 con 0x56157efcb400 2024-03-24T00:46:21.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.081+0000 7f756ea7b640 1 -- 172.21.15.67:0/312922798 <== osd.4 v2:172.21.15.72:6808/477220793 7 ==== osd_op_reply(37 [pgnls-filter start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6780 con 0x56157efcb400 2024-03-24T00:46:21.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.081+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:38 14.31 14:8c000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efec000 con 0x56157efca000 2024-03-24T00:46:21.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.081+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.1 v2:172.21.15.67:6800/4062510404 6 ==== osd_op_reply(38 [pgnls-filter start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157efca000 2024-03-24T00:46:21.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.081+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:39 14.9 14:90000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efc4400 con 0x56157fb79400 2024-03-24T00:46:21.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.081+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.5 v2:172.21.15.72:6800/288486206 4 ==== osd_op_reply(39 [pgnls-filter start_epoch 93 out=49b] v89'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157fb79400 2024-03-24T00:46:21.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.081+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:40 14.29 14:94000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efc5c00 con 0x56157fb79800 2024-03-24T00:46:21.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.081+0000 7f756f27c640 1 -- 172.21.15.67:0/312922798 <== osd.6 v2:172.21.15.72:6816/1725206562 4 ==== osd_op_reply(40 [pgnls-filter start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6500 con 0x56157fb79800 2024-03-24T00:46:21.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.082+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:41 14.19 14:98000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157fb79c00 con 0x56157fb79800 2024-03-24T00:46:21.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.082+0000 7f756f27c640 1 -- 172.21.15.67:0/312922798 <== osd.6 v2:172.21.15.72:6816/1725206562 5 ==== osd_op_reply(41 [pgnls-filter start_epoch 93 out=49b] v89'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6500 con 0x56157fb79800 2024-03-24T00:46:21.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.082+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:42 14.39 14:9c000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efec800 con 0x56157efca400 2024-03-24T00:46:21.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.082+0000 7f756f27c640 1 -- 172.21.15.67:0/312922798 <== osd.7 v2:172.21.15.72:6824/2893038746 2 ==== osd_op_reply(42 [pgnls-filter start_epoch 93 out=49b] v89'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6500 con 0x56157efca400 2024-03-24T00:46:21.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.082+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:43 14.5 14:a0000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157eff6000 con 0x56157fb79000 2024-03-24T00:46:21.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.082+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.0 v2:172.21.15.67:6808/2628130519 10 ==== osd_op_reply(43 [pgnls-filter start_epoch 93 out=49b] v89'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157fb79000 2024-03-24T00:46:21.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.082+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] -- osd_op(unknown.0.0:44 14.25 14:a4000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efed000 con 0x56157efc4000 2024-03-24T00:46:21.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.083+0000 7f756ea7b640 1 -- 172.21.15.67:0/312922798 <== osd.2 v2:172.21.15.67:6817/1268432561 6 ==== osd_op_reply(44 [pgnls-filter start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6780 con 0x56157efc4000 2024-03-24T00:46:21.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.083+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:45 14.15 14:a8000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157eff6400 con 0x56157efca400 2024-03-24T00:46:21.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.083+0000 7f756f27c640 1 -- 172.21.15.67:0/312922798 <== osd.7 v2:172.21.15.72:6824/2893038746 3 ==== osd_op_reply(45 [pgnls-filter start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6500 con 0x56157efca400 2024-03-24T00:46:21.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.083+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:46 14.35 14:ac000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efed400 con 0x56157efcb400 2024-03-24T00:46:21.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.083+0000 7f756ea7b640 1 -- 172.21.15.67:0/312922798 <== osd.4 v2:172.21.15.72:6808/477220793 8 ==== osd_op_reply(46 [pgnls-filter start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6780 con 0x56157efcb400 2024-03-24T00:46:21.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.083+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:47 14.d 14:b0000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157eff6800 con 0x56157fb79800 2024-03-24T00:46:21.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.084+0000 7f756f27c640 1 -- 172.21.15.67:0/312922798 <== osd.6 v2:172.21.15.72:6816/1725206562 6 ==== osd_op_reply(47 [pgnls-filter start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6500 con 0x56157fb79800 2024-03-24T00:46:21.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.084+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:48 14.2d 14:b4000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efecc00 con 0x56157efcb400 2024-03-24T00:46:21.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.084+0000 7f756ea7b640 1 -- 172.21.15.67:0/312922798 <== osd.4 v2:172.21.15.72:6808/477220793 9 ==== osd_op_reply(48 [pgnls-filter start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6780 con 0x56157efcb400 2024-03-24T00:46:21.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.084+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:49 14.1d 14:b8000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157eff6c00 con 0x56157fb79000 2024-03-24T00:46:21.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.084+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.0 v2:172.21.15.67:6808/2628130519 11 ==== osd_op_reply(49 [pgnls-filter start_epoch 93 out=49b] v89'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157fb79000 2024-03-24T00:46:21.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.084+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:50 14.3d 14:bc000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efca800 con 0x56157fb07000 2024-03-24T00:46:21.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.084+0000 7f756f27c640 1 -- 172.21.15.67:0/312922798 <== osd.3 v2:172.21.15.67:6816/3303489481 5 ==== osd_op_reply(50 [pgnls-filter start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6500 con 0x56157fb07000 2024-03-24T00:46:21.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.084+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:51 14.3 14:c0000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157eff7000 con 0x56157fb79000 2024-03-24T00:46:21.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.085+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.0 v2:172.21.15.67:6808/2628130519 12 ==== osd_op_reply(51 [pgnls-filter start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157fb79000 2024-03-24T00:46:21.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.085+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:52 14.23 14:c4000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efcac00 con 0x56157efca400 2024-03-24T00:46:21.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.085+0000 7f756f27c640 1 -- 172.21.15.67:0/312922798 <== osd.7 v2:172.21.15.72:6824/2893038746 4 ==== osd_op_reply(52 [pgnls-filter start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6500 con 0x56157efca400 2024-03-24T00:46:21.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.085+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] -- osd_op(unknown.0.0:53 14.13 14:c8000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157eff7400 con 0x56157efc4000 2024-03-24T00:46:21.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.085+0000 7f756ea7b640 1 -- 172.21.15.67:0/312922798 <== osd.2 v2:172.21.15.67:6817/1268432561 7 ==== osd_op_reply(53 [pgnls-filter start_epoch 93 out=49b] v89'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6780 con 0x56157efc4000 2024-03-24T00:46:21.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.085+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:54 14.33 14:cc000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157fb06c00 con 0x56157efca000 2024-03-24T00:46:21.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.086+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.1 v2:172.21.15.67:6800/4062510404 7 ==== osd_op_reply(54 [pgnls-filter start_epoch 93 out=49b] v89'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157efca000 2024-03-24T00:46:21.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.086+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:55 14.b 14:d0000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157eff7800 con 0x56157fb79000 2024-03-24T00:46:21.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.086+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.0 v2:172.21.15.67:6808/2628130519 13 ==== osd_op_reply(55 [pgnls-filter start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157fb79000 2024-03-24T00:46:21.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.086+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:56 14.2b 14:d4000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157eff6000 con 0x56157efcb400 2024-03-24T00:46:21.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.086+0000 7f756ea7b640 1 -- 172.21.15.67:0/312922798 <== osd.4 v2:172.21.15.72:6808/477220793 10 ==== osd_op_reply(56 [pgnls-filter start_epoch 93 out=49b] v89'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6780 con 0x56157efcb400 2024-03-24T00:46:21.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.086+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:57 14.1b 14:d8000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157eff7c00 con 0x56157efca000 2024-03-24T00:46:21.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.086+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.1 v2:172.21.15.67:6800/4062510404 8 ==== osd_op_reply(57 [pgnls-filter start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157efca000 2024-03-24T00:46:21.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.087+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:58 14.3b 14:dc000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157fb79c00 con 0x56157efcb400 2024-03-24T00:46:21.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.087+0000 7f756ea7b640 1 -- 172.21.15.67:0/312922798 <== osd.4 v2:172.21.15.72:6808/477220793 11 ==== osd_op_reply(58 [pgnls-filter start_epoch 93 out=49b] v89'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6780 con 0x56157efcb400 2024-03-24T00:46:21.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.087+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:59 14.7 14:e0000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157eff6c00 con 0x56157fb79400 2024-03-24T00:46:21.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.087+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.5 v2:172.21.15.72:6800/288486206 5 ==== osd_op_reply(59 [pgnls-filter start_epoch 93 out=81b] v93'7 uv7 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x56157eda6000 con 0x56157fb79400 2024-03-24T00:46:21.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.087+0000 7f7570c14740 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] -- osd_op(unknown.0.0:60 14.0 14:00000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efc4800 con 0x56157efc4000 2024-03-24T00:46:21.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.088+0000 7f756ea7b640 1 -- 172.21.15.67:0/312922798 <== osd.2 v2:172.21.15.67:6817/1268432561 8 ==== osd_op_reply(60 [pgnls-filter start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6780 con 0x56157efc4000 2024-03-24T00:46:21.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.088+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:61 14.20 14:04000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157eff7000 con 0x56157fb79000 2024-03-24T00:46:21.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.088+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.0 v2:172.21.15.67:6808/2628130519 14 ==== osd_op_reply(61 [pgnls-filter start_epoch 93 out=49b] v89'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157fb79000 2024-03-24T00:46:21.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.088+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:62 14.10 14:08000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efc4400 con 0x56157efca000 2024-03-24T00:46:21.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.088+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.1 v2:172.21.15.67:6800/4062510404 9 ==== osd_op_reply(62 [pgnls-filter start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157efca000 2024-03-24T00:46:21.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.088+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:63 14.30 14:0c000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157eff7800 con 0x56157fb79000 2024-03-24T00:46:21.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.088+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.0 v2:172.21.15.67:6808/2628130519 15 ==== osd_op_reply(63 [pgnls-filter start_epoch 93 out=49b] v89'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157fb79000 2024-03-24T00:46:21.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.088+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:64 14.8 14:10000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efc5c00 con 0x56157fb79000 2024-03-24T00:46:21.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.088+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.0 v2:172.21.15.67:6808/2628130519 16 ==== osd_op_reply(64 [pgnls-filter start_epoch 93 out=49b] v89'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157fb79000 2024-03-24T00:46:21.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.089+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:65 14.28 14:14000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157eff6400 con 0x56157fb07000 2024-03-24T00:46:21.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.089+0000 7f756f27c640 1 -- 172.21.15.67:0/312922798 <== osd.3 v2:172.21.15.67:6816/3303489481 6 ==== osd_op_reply(65 [pgnls-filter start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6500 con 0x56157fb07000 2024-03-24T00:46:21.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.089+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:66 14.18 14:18000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efcb800 con 0x56157efca000 2024-03-24T00:46:21.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.089+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.1 v2:172.21.15.67:6800/4062510404 10 ==== osd_op_reply(66 [pgnls-filter start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157efca000 2024-03-24T00:46:21.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.089+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] -- osd_op(unknown.0.0:67 14.38 14:1c000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157eff6800 con 0x56157efc4000 2024-03-24T00:46:21.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.089+0000 7f756ea7b640 1 -- 172.21.15.67:0/312922798 <== osd.2 v2:172.21.15.67:6817/1268432561 9 ==== osd_op_reply(67 [pgnls-filter start_epoch 93 out=49b] v89'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6780 con 0x56157efc4000 2024-03-24T00:46:21.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.089+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:68 14.4 14:20000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efcb000 con 0x56157fb79800 2024-03-24T00:46:21.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.090+0000 7f756f27c640 1 -- 172.21.15.67:0/312922798 <== osd.6 v2:172.21.15.72:6816/1725206562 7 ==== osd_op_reply(68 [pgnls-filter start_epoch 93 out=49b] v89'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6500 con 0x56157fb79800 2024-03-24T00:46:21.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.090+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:69 14.24 14:24000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efec000 con 0x56157efca000 2024-03-24T00:46:21.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.090+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.1 v2:172.21.15.67:6800/4062510404 11 ==== osd_op_reply(69 [pgnls-filter start_epoch 93 out=49b] v89'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157efca000 2024-03-24T00:46:21.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.090+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:70 14.14 14:28000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efcbc00 con 0x56157efcb400 2024-03-24T00:46:21.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.091+0000 7f756ea7b640 1 -- 172.21.15.67:0/312922798 <== osd.4 v2:172.21.15.72:6808/477220793 12 ==== osd_op_reply(70 [pgnls-filter start_epoch 93 out=49b] v89'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6780 con 0x56157efcb400 2024-03-24T00:46:21.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.091+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] -- osd_op(unknown.0.0:71 14.34 14:2c000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efec400 con 0x56157efc4000 2024-03-24T00:46:21.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.091+0000 7f756ea7b640 1 -- 172.21.15.67:0/312922798 <== osd.2 v2:172.21.15.67:6817/1268432561 10 ==== osd_op_reply(71 [pgnls-filter start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6780 con 0x56157efc4000 2024-03-24T00:46:21.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.091+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:72 14.c 14:30000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157fb06c00 con 0x56157fb79000 2024-03-24T00:46:21.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.091+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.0 v2:172.21.15.67:6808/2628130519 17 ==== osd_op_reply(72 [pgnls-filter start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157fb79000 2024-03-24T00:46:21.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.092+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:73 14.2c 14:34000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efec800 con 0x56157efca000 2024-03-24T00:46:21.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.092+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.1 v2:172.21.15.67:6800/4062510404 12 ==== osd_op_reply(73 [pgnls-filter start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157efca000 2024-03-24T00:46:21.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.092+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:74 14.1c 14:38000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157eff7c00 con 0x56157fb79000 2024-03-24T00:46:21.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.092+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.0 v2:172.21.15.67:6808/2628130519 18 ==== osd_op_reply(74 [pgnls-filter start_epoch 93 out=49b] v89'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157fb79000 2024-03-24T00:46:21.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.092+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:75 14.3c 14:3c000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efecc00 con 0x56157efca000 2024-03-24T00:46:21.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.092+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.1 v2:172.21.15.67:6800/4062510404 13 ==== osd_op_reply(75 [pgnls-filter start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157efca000 2024-03-24T00:46:21.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.092+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:76 14.2 14:40000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157eff6c00 con 0x56157fb79800 2024-03-24T00:46:21.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.092+0000 7f756f27c640 1 -- 172.21.15.67:0/312922798 <== osd.6 v2:172.21.15.72:6816/1725206562 8 ==== osd_op_reply(76 [pgnls-filter start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6500 con 0x56157fb79800 2024-03-24T00:46:21.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.093+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:77 14.22 14:44000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efed000 con 0x56157efcb400 2024-03-24T00:46:21.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.093+0000 7f756ea7b640 1 -- 172.21.15.67:0/312922798 <== osd.4 v2:172.21.15.72:6808/477220793 13 ==== osd_op_reply(77 [pgnls-filter start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6780 con 0x56157efcb400 2024-03-24T00:46:21.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.093+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] -- osd_op(unknown.0.0:78 14.12 14:48000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157eff7000 con 0x56157efc4000 2024-03-24T00:46:21.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.093+0000 7f756ea7b640 1 -- 172.21.15.67:0/312922798 <== osd.2 v2:172.21.15.67:6817/1268432561 11 ==== osd_op_reply(78 [pgnls-filter start_epoch 93 out=49b] v89'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6780 con 0x56157efc4000 2024-03-24T00:46:21.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.093+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:79 14.32 14:4c000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efed400 con 0x56157efcb400 2024-03-24T00:46:21.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.093+0000 7f756ea7b640 1 -- 172.21.15.67:0/312922798 <== osd.4 v2:172.21.15.72:6808/477220793 14 ==== osd_op_reply(79 [pgnls-filter start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6780 con 0x56157efcb400 2024-03-24T00:46:21.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.093+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:80 14.a 14:50000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157eff7800 con 0x56157fb79400 2024-03-24T00:46:21.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.094+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.5 v2:172.21.15.72:6800/288486206 6 ==== osd_op_reply(80 [pgnls-filter start_epoch 93 out=49b] v89'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157fb79400 2024-03-24T00:46:21.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.094+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:81 14.2a 14:54000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efed800 con 0x56157efcb400 2024-03-24T00:46:21.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.094+0000 7f756ea7b640 1 -- 172.21.15.67:0/312922798 <== osd.4 v2:172.21.15.72:6808/477220793 15 ==== osd_op_reply(81 [pgnls-filter start_epoch 93 out=49b] v89'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6780 con 0x56157efcb400 2024-03-24T00:46:21.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.094+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:82 14.1a 14:58000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efc4400 con 0x56157fb79000 2024-03-24T00:46:21.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.094+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.0 v2:172.21.15.67:6808/2628130519 19 ==== osd_op_reply(82 [pgnls-filter start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157fb79000 2024-03-24T00:46:21.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.094+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:83 14.3a 14:5c000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efedc00 con 0x56157fb79000 2024-03-24T00:46:21.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.094+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.0 v2:172.21.15.67:6808/2628130519 20 ==== osd_op_reply(83 [pgnls-filter start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157fb79000 2024-03-24T00:46:21.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.094+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:84 14.6 14:60000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efc5c00 con 0x56157fb79000 2024-03-24T00:46:21.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.095+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.0 v2:172.21.15.67:6808/2628130519 21 ==== osd_op_reply(84 [pgnls-filter start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157fb79000 2024-03-24T00:46:21.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.095+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:85 14.26 14:64000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efec000 con 0x56157efcb400 2024-03-24T00:46:21.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.095+0000 7f756ea7b640 1 -- 172.21.15.67:0/312922798 <== osd.4 v2:172.21.15.72:6808/477220793 16 ==== osd_op_reply(85 [pgnls-filter start_epoch 93 out=49b] v89'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6780 con 0x56157efcb400 2024-03-24T00:46:21.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.095+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:86 14.16 14:68000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efc4800 con 0x56157fb07000 2024-03-24T00:46:21.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.095+0000 7f756f27c640 1 -- 172.21.15.67:0/312922798 <== osd.3 v2:172.21.15.67:6816/3303489481 7 ==== osd_op_reply(86 [pgnls-filter start_epoch 93 out=49b] v89'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6500 con 0x56157fb07000 2024-03-24T00:46:21.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.095+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] -- osd_op(unknown.0.0:87 14.36 14:6c000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efec400 con 0x56157efc4000 2024-03-24T00:46:21.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.095+0000 7f756ea7b640 1 -- 172.21.15.67:0/312922798 <== osd.2 v2:172.21.15.67:6817/1268432561 12 ==== osd_op_reply(87 [pgnls-filter start_epoch 93 out=49b] v89'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6780 con 0x56157efc4000 2024-03-24T00:46:21.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.095+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:88 14.e 14:70000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157fb79c00 con 0x56157efcb400 2024-03-24T00:46:21.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.096+0000 7f756ea7b640 1 -- 172.21.15.67:0/312922798 <== osd.4 v2:172.21.15.72:6808/477220793 17 ==== osd_op_reply(88 [pgnls-filter start_epoch 93 out=49b] v89'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6780 con 0x56157efcb400 2024-03-24T00:46:21.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.096+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:89 14.2e 14:74000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efcb800 con 0x56157fb07000 2024-03-24T00:46:21.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.096+0000 7f756f27c640 1 -- 172.21.15.67:0/312922798 <== osd.3 v2:172.21.15.67:6816/3303489481 8 ==== osd_op_reply(89 [pgnls-filter start_epoch 93 out=49b] v89'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6500 con 0x56157fb07000 2024-03-24T00:46:21.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.096+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:90 14.1e 14:78000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157fb06c00 con 0x56157efca400 2024-03-24T00:46:21.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.096+0000 7f756f27c640 1 -- 172.21.15.67:0/312922798 <== osd.7 v2:172.21.15.72:6824/2893038746 5 ==== osd_op_reply(90 [pgnls-filter start_epoch 93 out=49b] v89'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6500 con 0x56157efca400 2024-03-24T00:46:21.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.096+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:91 14.3e 14:7c000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efcbc00 con 0x56157fb79400 2024-03-24T00:46:21.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.097+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.5 v2:172.21.15.72:6800/288486206 7 ==== osd_op_reply(91 [pgnls-filter start_epoch 93 out=49b] v89'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157fb79400 2024-03-24T00:46:21.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.097+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:92 14.1 14:80000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157fb06c00 con 0x56157fb07000 2024-03-24T00:46:21.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.097+0000 7f756f27c640 1 -- 172.21.15.67:0/312922798 <== osd.3 v2:172.21.15.67:6816/3303489481 9 ==== osd_op_reply(92 [pgnls-filter start_epoch 93 out=49b] v89'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6500 con 0x56157fb07000 2024-03-24T00:46:21.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.097+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:93 14.21 14:84000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157eff7400 con 0x56157fb79400 2024-03-24T00:46:21.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.097+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.5 v2:172.21.15.72:6800/288486206 8 ==== osd_op_reply(93 [pgnls-filter start_epoch 93 out=49b] v89'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157fb79400 2024-03-24T00:46:21.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.097+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:94 14.11 14:88000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efca800 con 0x56157efcb400 2024-03-24T00:46:21.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.097+0000 7f756ea7b640 1 -- 172.21.15.67:0/312922798 <== osd.4 v2:172.21.15.72:6808/477220793 18 ==== osd_op_reply(94 [pgnls-filter start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6780 con 0x56157efcb400 2024-03-24T00:46:21.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.097+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:95 14.31 14:8c000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157eff6000 con 0x56157efca000 2024-03-24T00:46:21.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.098+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.1 v2:172.21.15.67:6800/4062510404 14 ==== osd_op_reply(95 [pgnls-filter start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157efca000 2024-03-24T00:46:21.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.098+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:96 14.9 14:90000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efcac00 con 0x56157fb79400 2024-03-24T00:46:21.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.098+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.5 v2:172.21.15.72:6800/288486206 9 ==== osd_op_reply(96 [pgnls-filter start_epoch 93 out=49b] v89'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157fb79400 2024-03-24T00:46:21.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.098+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:97 14.29 14:94000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157eff6800 con 0x56157fb79800 2024-03-24T00:46:21.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.098+0000 7f756f27c640 1 -- 172.21.15.67:0/312922798 <== osd.6 v2:172.21.15.72:6816/1725206562 9 ==== osd_op_reply(97 [pgnls-filter start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6500 con 0x56157fb79800 2024-03-24T00:46:21.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.098+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:98 14.19 14:98000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efcb000 con 0x56157fb79800 2024-03-24T00:46:21.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.099+0000 7f756f27c640 1 -- 172.21.15.67:0/312922798 <== osd.6 v2:172.21.15.72:6816/1725206562 10 ==== osd_op_reply(98 [pgnls-filter start_epoch 93 out=49b] v89'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6500 con 0x56157fb79800 2024-03-24T00:46:21.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.099+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:99 14.39 14:9c000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efcbc00 con 0x56157efca400 2024-03-24T00:46:21.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.099+0000 7f756f27c640 1 -- 172.21.15.67:0/312922798 <== osd.7 v2:172.21.15.72:6824/2893038746 6 ==== osd_op_reply(99 [pgnls-filter start_epoch 93 out=49b] v89'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6500 con 0x56157efca400 2024-03-24T00:46:21.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.099+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:100 14.5 14:a0000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efcb800 con 0x56157fb79000 2024-03-24T00:46:21.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.099+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.0 v2:172.21.15.67:6808/2628130519 22 ==== osd_op_reply(100 [pgnls-filter start_epoch 93 out=49b] v89'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157fb79000 2024-03-24T00:46:21.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.099+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] -- osd_op(unknown.0.0:101 14.25 14:a4000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efca800 con 0x56157efc4000 2024-03-24T00:46:21.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.099+0000 7f756ea7b640 1 -- 172.21.15.67:0/312922798 <== osd.2 v2:172.21.15.67:6817/1268432561 13 ==== osd_op_reply(101 [pgnls-filter start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6780 con 0x56157efc4000 2024-03-24T00:46:21.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.099+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:102 14.15 14:a8000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efc4800 con 0x56157efca400 2024-03-24T00:46:21.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.100+0000 7f756f27c640 1 -- 172.21.15.67:0/312922798 <== osd.7 v2:172.21.15.72:6824/2893038746 7 ==== osd_op_reply(102 [pgnls-filter start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6500 con 0x56157efca400 2024-03-24T00:46:21.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.100+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:103 14.35 14:ac000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157fb79c00 con 0x56157efcb400 2024-03-24T00:46:21.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.100+0000 7f756ea7b640 1 -- 172.21.15.67:0/312922798 <== osd.4 v2:172.21.15.72:6808/477220793 19 ==== osd_op_reply(103 [pgnls-filter start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6780 con 0x56157efcb400 2024-03-24T00:46:21.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.100+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:104 14.d 14:b0000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157eff6400 con 0x56157fb79800 2024-03-24T00:46:21.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.100+0000 7f756f27c640 1 -- 172.21.15.67:0/312922798 <== osd.6 v2:172.21.15.72:6816/1725206562 11 ==== osd_op_reply(104 [pgnls-filter start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6500 con 0x56157fb79800 2024-03-24T00:46:21.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.100+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:105 14.2d 14:b4000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157eff7c00 con 0x56157efcb400 2024-03-24T00:46:21.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.101+0000 7f756ea7b640 1 -- 172.21.15.67:0/312922798 <== osd.4 v2:172.21.15.72:6808/477220793 20 ==== osd_op_reply(105 [pgnls-filter start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6780 con 0x56157efcb400 2024-03-24T00:46:21.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.101+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:106 14.1d 14:b8000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157eff6c00 con 0x56157fb79000 2024-03-24T00:46:21.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.101+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.0 v2:172.21.15.67:6808/2628130519 23 ==== osd_op_reply(106 [pgnls-filter start_epoch 93 out=49b] v89'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157fb79000 2024-03-24T00:46:21.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.101+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:107 14.3d 14:bc000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157eff7800 con 0x56157fb07000 2024-03-24T00:46:21.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.101+0000 7f756f27c640 1 -- 172.21.15.67:0/312922798 <== osd.3 v2:172.21.15.67:6816/3303489481 10 ==== osd_op_reply(107 [pgnls-filter start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6500 con 0x56157fb07000 2024-03-24T00:46:21.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.101+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:108 14.3 14:c0000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efec000 con 0x56157fb79000 2024-03-24T00:46:21.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.101+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.0 v2:172.21.15.67:6808/2628130519 24 ==== osd_op_reply(108 [pgnls-filter start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157fb79000 2024-03-24T00:46:21.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.101+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:109 14.23 14:c4000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157eff7400 con 0x56157efca400 2024-03-24T00:46:21.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.102+0000 7f756f27c640 1 -- 172.21.15.67:0/312922798 <== osd.7 v2:172.21.15.72:6824/2893038746 8 ==== osd_op_reply(109 [pgnls-filter start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6500 con 0x56157efca400 2024-03-24T00:46:21.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.102+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] -- osd_op(unknown.0.0:110 14.13 14:c8000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efec400 con 0x56157efc4000 2024-03-24T00:46:21.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.102+0000 7f756ea7b640 1 -- 172.21.15.67:0/312922798 <== osd.2 v2:172.21.15.67:6817/1268432561 14 ==== osd_op_reply(110 [pgnls-filter start_epoch 93 out=49b] v89'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6780 con 0x56157efc4000 2024-03-24T00:46:21.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.102+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:111 14.33 14:cc000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157eff6000 con 0x56157efca000 2024-03-24T00:46:21.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.102+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.1 v2:172.21.15.67:6800/4062510404 15 ==== osd_op_reply(111 [pgnls-filter start_epoch 93 out=49b] v89'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157efca000 2024-03-24T00:46:21.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.102+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:112 14.b 14:d0000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efec800 con 0x56157fb79000 2024-03-24T00:46:21.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.102+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.0 v2:172.21.15.67:6808/2628130519 25 ==== osd_op_reply(112 [pgnls-filter start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157fb79000 2024-03-24T00:46:21.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.102+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:113 14.2b 14:d4000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157eff7000 con 0x56157efcb400 2024-03-24T00:46:21.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.103+0000 7f756ea7b640 1 -- 172.21.15.67:0/312922798 <== osd.4 v2:172.21.15.72:6808/477220793 21 ==== osd_op_reply(113 [pgnls-filter start_epoch 93 out=49b] v89'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6780 con 0x56157efcb400 2024-03-24T00:46:21.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.103+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:114 14.1b 14:d8000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efecc00 con 0x56157efca000 2024-03-24T00:46:21.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.103+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.1 v2:172.21.15.67:6800/4062510404 16 ==== osd_op_reply(114 [pgnls-filter start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157efca000 2024-03-24T00:46:21.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.103+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:115 14.3b 14:dc000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efc4400 con 0x56157efcb400 2024-03-24T00:46:21.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.103+0000 7f756ea7b640 1 -- 172.21.15.67:0/312922798 <== osd.4 v2:172.21.15.72:6808/477220793 22 ==== osd_op_reply(115 [pgnls-filter start_epoch 93 out=49b] v89'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6780 con 0x56157efcb400 2024-03-24T00:46:21.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.103+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:116 14.7 14:e0000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efed000 con 0x56157fb79400 2024-03-24T00:46:21.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.104+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.5 v2:172.21.15.72:6800/288486206 10 ==== osd_op_reply(116 [pgnls-filter start_epoch 93 out=81b] v93'7 uv7 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x56157eda6000 con 0x56157fb79400 2024-03-24T00:46:21.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.104+0000 7f7570c14740 20 datascan.forall_objects: OSD matched oid 10000000001.00000000 2024-03-24T00:46:21.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.104+0000 7f7570c14740 10 datascan.operator(): handling object 10000000001.0 2024-03-24T00:46:21.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.104+0000 7f7570c14740 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:117 14.7 14:e2e07930:::10000000001.00000000:head [getxattr scan_ceiling in=12b,getxattr scan_max_size in=13b,getxattr scan_max_mtime in=14b,getxattr scan_pool_id in=12b,getxattr parent in=6b,getxattr layout in=6b,getxattr symlink in=7b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efc5000 con 0x56157fb79400 2024-03-24T00:46:21.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.104+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.5 v2:172.21.15.72:6800/288486206 11 ==== osd_op_reply(117 10000000001.00000000 [getxattr (22) out=22b,getxattr (8) out=8b,getxattr (8) out=8b,getxattr,getxattr (89) out=89b,getxattr (30) out=30b,getxattr] v0'0 uv7 ondisk = 0) v8 ==== 416+0+157 (crc 0 0 0) 0x56157eda6000 con 0x56157fb79400 2024-03-24T00:46:21.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.104+0000 7f7570c14740 10 datascan.inject_with_backtrace: inode: 0x0x10000000001 2024-03-24T00:46:21.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.104+0000 7f7570c14740 10 datascan.inject_with_backtrace: backptr: 0x0x10000000000/file 2024-03-24T00:46:21.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.104+0000 7f7570c14740 20 datascan.get_frag_of: dirino=0x10000000000 target_dname=file 2024-03-24T00:46:21.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.104+0000 7f7570c14740 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:118 13.10 13:0cf6ad0f:::10000000000.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efc5400 con 0x56157efca000 2024-03-24T00:46:21.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.104+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.1 v2:172.21.15.67:6800/4062510404 17 ==== osd_op_reply(118 10000000000.00000000 [getxattr (59) out=59b] v0'0 uv17 ondisk = 0) v8 ==== 164+0+59 (crc 0 0 0) 0x56157eda6000 con 0x56157efca000 2024-03-24T00:46:21.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.104+0000 7f7570c14740 20 datascan.get_frag_of: got backtrace parent 0x1/dir 2024-03-24T00:46:21.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.104+0000 7f7570c14740 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:119 13.1f 13:ff5b34d6:::1.00000000:head [omap-get-vals-by-keys in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efc5800 con 0x56157fb79800 2024-03-24T00:46:21.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.105+0000 7f756f27c640 1 -- 172.21.15.67:0/312922798 <== osd.6 v2:172.21.15.72:6816/1725206562 12 ==== osd_op_reply(119 1.00000000 [omap-get-vals-by-keys out=581b] v0'0 uv9 ondisk = 0) v8 ==== 154+0+581 (crc 0 0 0) 0x56157eda6500 con 0x56157fb79800 2024-03-24T00:46:21.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.105+0000 7f7570c14740 20 datascan.read_dentry: oid=1.00000000 dname=dir frag=*, r=0 2024-03-24T00:46:21.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.105+0000 7f7570c14740 20 datascan.get_frag_of: fast path, fragtree is fragtree_t() 2024-03-24T00:46:21.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.105+0000 7f7570c14740 20 datascan.get_frag_of: frag is * 2024-03-24T00:46:21.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.105+0000 7f7570c14740 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:120 13.10 13:0cf6ad0f:::10000000000.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157eff6800 con 0x56157efca000 2024-03-24T00:46:21.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.105+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.1 v2:172.21.15.67:6800/4062510404 18 ==== osd_op_reply(120 10000000000.00000000 [omap-get-header out=274b] v0'0 uv17 ondisk = 0) v8 ==== 164+0+274 (crc 0 0 0) 0x56157eda6000 con 0x56157efca000 2024-03-24T00:46:21.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.105+0000 7f7570c14740 10 datascan.find_or_create_dirfrag: read_version = 17 2024-03-24T00:46:21.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.105+0000 7f7570c14740 20 datascan.find_or_create_dirfrag: Dirfrag already exists: 0x0x10000000000 * 2024-03-24T00:46:21.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.105+0000 7f7570c14740 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:121 13.10 13:0cf6ad0f:::10000000000.00000000:head [omap-get-vals-by-keys in=17b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157eff6400 con 0x56157efca000 2024-03-24T00:46:21.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.106+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.1 v2:172.21.15.67:6800/4062510404 19 ==== osd_op_reply(121 10000000000.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv17 ondisk = 0) v8 ==== 164+0+4 (crc 0 0 0) 0x56157eda6000 con 0x56157efca000 2024-03-24T00:46:21.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.106+0000 7f7570c14740 20 datascan.read_dentry: oid=10000000000.00000000 dname=file frag=*, r=0 2024-03-24T00:46:21.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.106+0000 7f7570c14740 20 datascan.read_dentry: file_head not found in result 2024-03-24T00:46:21.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.106+0000 7f7570c14740 10 datascan.inject_with_backtrace: Linking inode 0x0x10000000001 at 0x0x10000000000/file with size=0 bytes 2024-03-24T00:46:21.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.106+0000 7f7570c14740 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:122 13.10 13:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=505b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157eff7800 con 0x56157efca000 2024-03-24T00:46:21.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.107+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.1 v2:172.21.15.67:6800/4062510404 20 ==== osd_op_reply(122 10000000000.00000000 [omap-set-vals] v93'18 uv18 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x56157eda6000 con 0x56157efca000 2024-03-24T00:46:21.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.107+0000 7f7570c14740 20 datascan.inject_linkage: Injected dentry 0x0x10000000000/file pointing to 0x0x10000000001 2024-03-24T00:46:21.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.107+0000 7f7570c14740 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:123 14.27 14:e4000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efcb000 con 0x56157fb79400 2024-03-24T00:46:21.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.107+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.5 v2:172.21.15.72:6800/288486206 12 ==== osd_op_reply(123 [pgnls-filter start_epoch 93 out=49b] v89'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157fb79400 2024-03-24T00:46:21.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.107+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:124 14.17 14:e8000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efed400 con 0x56157efca000 2024-03-24T00:46:21.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.107+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.1 v2:172.21.15.67:6800/4062510404 21 ==== osd_op_reply(124 [pgnls-filter start_epoch 93 out=49b] v89'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157efca000 2024-03-24T00:46:21.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.107+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:125 14.37 14:ec000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efc5c00 con 0x56157efcb400 2024-03-24T00:46:21.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.108+0000 7f756ea7b640 1 -- 172.21.15.67:0/312922798 <== osd.4 v2:172.21.15.72:6808/477220793 23 ==== osd_op_reply(125 [pgnls-filter start_epoch 93 out=49b] v89'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6780 con 0x56157efcb400 2024-03-24T00:46:21.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.108+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:126 14.f 14:f0000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efed800 con 0x56157efca400 2024-03-24T00:46:21.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.108+0000 7f756f27c640 1 -- 172.21.15.67:0/312922798 <== osd.7 v2:172.21.15.72:6824/2893038746 9 ==== osd_op_reply(126 [pgnls-filter start_epoch 93 out=49b] v89'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6500 con 0x56157efca400 2024-03-24T00:46:21.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.108+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:127 14.2f 14:f4000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efc4800 con 0x56157fb79000 2024-03-24T00:46:21.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.108+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.0 v2:172.21.15.67:6808/2628130519 26 ==== osd_op_reply(127 [pgnls-filter start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157fb79000 2024-03-24T00:46:21.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.108+0000 7f7569a71640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:128 14.1f 14:f8000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efedc00 con 0x56157fb79000 2024-03-24T00:46:21.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.109+0000 7f756e27a640 1 -- 172.21.15.67:0/312922798 <== osd.0 v2:172.21.15.67:6808/2628130519 27 ==== osd_op_reply(128 [pgnls-filter start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6000 con 0x56157fb79000 2024-03-24T00:46:21.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.109+0000 7f756a272640 1 -- 172.21.15.67:0/312922798 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:129 14.3f 14:fc000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x56157efc5800 con 0x56157fb07000 2024-03-24T00:46:21.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.109+0000 7f756f27c640 1 -- 172.21.15.67:0/312922798 <== osd.3 v2:172.21.15.67:6816/3303489481 11 ==== osd_op_reply(129 [pgnls-filter start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56157eda6500 con 0x56157fb07000 2024-03-24T00:46:21.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.109+0000 7f7570c14740 1 -- 172.21.15.67:0/312922798 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x56157fb79000 msgr2=0x56157ecda680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:21.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.109+0000 7f7570c14740 1 --2- 172.21.15.67:0/312922798 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x56157fb79000 0x56157ecda680 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.109+0000 7f7570c14740 1 -- 172.21.15.67:0/312922798 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x56157efca000 msgr2=0x56157efb0580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:21.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.109+0000 7f7570c14740 1 --2- 172.21.15.67:0/312922798 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x56157efca000 0x56157efb0580 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.109+0000 7f7570c14740 1 -- 172.21.15.67:0/312922798 >> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] conn(0x56157efc4000 msgr2=0x56157ecd9080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:21.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.109+0000 7f7570c14740 1 --2- 172.21.15.67:0/312922798 >> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] conn(0x56157efc4000 0x56157ecd9080 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.109+0000 7f7570c14740 1 -- 172.21.15.67:0/312922798 >> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] conn(0x56157fb07000 msgr2=0x56157efb0b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:21.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.109+0000 7f7570c14740 1 --2- 172.21.15.67:0/312922798 >> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] conn(0x56157fb07000 0x56157efb0b00 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.109+0000 7f756ea7b640 1 -- 172.21.15.67:0/312922798 reap_dead start 2024-03-24T00:46:21.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.109+0000 7f7570c14740 1 -- 172.21.15.67:0/312922798 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x56157efcb400 msgr2=0x56157efb1080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:21.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.109+0000 7f7570c14740 1 --2- 172.21.15.67:0/312922798 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x56157efcb400 0x56157efb1080 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.109+0000 7f7570c14740 1 -- 172.21.15.67:0/312922798 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x56157fb79400 msgr2=0x56157efb1600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:21.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.109+0000 7f7570c14740 1 --2- 172.21.15.67:0/312922798 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x56157fb79400 0x56157efb1600 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.109+0000 7f7570c14740 1 -- 172.21.15.67:0/312922798 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x56157fb79800 msgr2=0x56157ecda100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:21.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.109+0000 7f7570c14740 1 --2- 172.21.15.67:0/312922798 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x56157fb79800 0x56157ecda100 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.109+0000 7f7570c14740 1 -- 172.21.15.67:0/312922798 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x56157efca400 msgr2=0x56157efb1b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:21.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.109+0000 7f7570c14740 1 --2- 172.21.15.67:0/312922798 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x56157efca400 0x56157efb1b80 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.109+0000 7f756ea7b640 1 -- 172.21.15.67:0/312922798 reap_dead start 2024-03-24T00:46:21.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.109+0000 7f7570c14740 1 -- 172.21.15.67:0/312922798 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x56157fb78c00 msgr2=0x56157efb0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:21.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.109+0000 7f7570c14740 1 --2- 172.21.15.67:0/312922798 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x56157fb78c00 0x56157efb0000 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x56157f9cd140 tx=0x56157f9caa80 comp rx=0 tx=0).stop 2024-03-24T00:46:21.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.109+0000 7f7570c14740 1 -- 172.21.15.67:0/312922798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56157fb07800 msgr2=0x56157ecdb180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:21.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.109+0000 7f7570c14740 1 --2- 172.21.15.67:0/312922798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56157fb07800 0x56157ecdb180 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x56157fb21b90 tx=0x56157fb23a40 comp rx=0 tx=0).stop 2024-03-24T00:46:21.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.109+0000 7f7570c14740 1 -- 172.21.15.67:0/312922798 shutdown_connections 2024-03-24T00:46:21.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.109+0000 7f7570c14740 1 --2- 172.21.15.67:0/312922798 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x56157fb78c00 0x56157efb0000 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.109+0000 7f7570c14740 1 --2- 172.21.15.67:0/312922798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56157fb07800 0x56157ecdb180 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.109+0000 7f7570c14740 1 -- 172.21.15.67:0/312922798 >> 172.21.15.67:0/312922798 conn(0x56157fa04400 msgr2=0x56157f9b7800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:21.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.109+0000 7f7570c14740 1 -- 172.21.15.67:0/312922798 shutdown_connections 2024-03-24T00:46:21.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.109+0000 7f7570c14740 1 -- 172.21.15.67:0/312922798 wait complete. 2024-03-24T00:46:21.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.110+0000 7f7570c14740 1 -- 172.21.15.67:0/4047905513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56157ec2fc00 msgr2=0x56157ecd8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:21.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.110+0000 7f7570c14740 1 --2- 172.21.15.67:0/4047905513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56157ec2fc00 0x56157ecd8b00 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x56157fa61920 tx=0x56157f9ce000 comp rx=0 tx=0).stop 2024-03-24T00:46:21.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.110+0000 7f7570c14740 1 -- 172.21.15.67:0/4047905513 shutdown_connections 2024-03-24T00:46:21.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.110+0000 7f7570c14740 1 --2- 172.21.15.67:0/4047905513 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56157fa05c00 0x56157ecd9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.110+0000 7f7570c14740 1 --2- 172.21.15.67:0/4047905513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56157ec2fc00 0x56157ecd8b00 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.110+0000 7f7570c14740 1 --2- 172.21.15.67:0/4047905513 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56157fa04800 0x56157ecd9b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.110+0000 7f7570c14740 1 -- 172.21.15.67:0/4047905513 >> 172.21.15.67:0/4047905513 conn(0x56157ec2f000 msgr2=0x56157f9b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:21.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.110+0000 7f7570c14740 1 -- 172.21.15.67:0/4047905513 shutdown_connections 2024-03-24T00:46:21.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.110+0000 7f7570c14740 1 -- 172.21.15.67:0/4047905513 wait complete. 2024-03-24T00:46:21.137 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-data-scan', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'scan_inodes', 'cephfs_data'] in time 0:00:00.150854, result: 2024-03-24T00:46:21.137 DEBUG:teuthology.orchestra.run.smithi067:> cephfs-data-scan --debug-mds=20 --debug-ms=1 --debug-objecter=1 scan_links 2024-03-24T00:46:21.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.213+0000 7f7ca842a740 1 -- start start 2024-03-24T00:46:21.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.213+0000 7f7ca842a740 1 Processor -- start 2024-03-24T00:46:21.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.213+0000 7f7ca842a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ef3e7aec00 0x55ef3e858580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:21.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.213+0000 7f7ca842a740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ef3e7af800 0x55ef3e858b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:21.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.213+0000 7f7ca842a740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ef3e7afc00 0x55ef3e859080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:21.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.213+0000 7f7ca842a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef3f66c180 con 0x55ef3e7aec00 2024-03-24T00:46:21.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.213+0000 7f7ca842a740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef3f66c300 con 0x55ef3e7af800 2024-03-24T00:46:21.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.213+0000 7f7ca842a740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ef3f66c480 con 0x55ef3e7afc00 2024-03-24T00:46:21.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.213+0000 7f7ca5e98640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ef3e7aec00 0x55ef3e858580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:21.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.213+0000 7f7ca5e98640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ef3e7aec00 0x55ef3e858580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55482/0 (socket says 172.21.15.67:55482) 2024-03-24T00:46:21.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.213+0000 7f7ca5e98640 1 -- 172.21.15.67:0/3191124449 learned_addr learned my addr 172.21.15.67:0/3191124449 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:21.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.213+0000 7f7ca6699640 1 --2- 172.21.15.67:0/3191124449 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ef3e7afc00 0x55ef3e859080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:21.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.213+0000 7f7ca5697640 1 --2- 172.21.15.67:0/3191124449 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ef3e7af800 0x55ef3e858b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:21.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.214+0000 7f7ca5e98640 1 -- 172.21.15.67:0/3191124449 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ef3e7afc00 msgr2=0x55ef3e859080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:21.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.214+0000 7f7ca5e98640 1 --2- 172.21.15.67:0/3191124449 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ef3e7afc00 0x55ef3e859080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.214+0000 7f7ca5e98640 1 -- 172.21.15.67:0/3191124449 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ef3e7af800 msgr2=0x55ef3e858b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:21.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.214+0000 7f7ca5e98640 1 --2- 172.21.15.67:0/3191124449 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ef3e7af800 0x55ef3e858b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.214+0000 7f7ca5e98640 1 -- 172.21.15.67:0/3191124449 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ef3e7db680 con 0x55ef3e7aec00 2024-03-24T00:46:21.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.214+0000 7f7ca5e98640 1 --2- 172.21.15.67:0/3191124449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ef3e7aec00 0x55ef3e858580 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x55ef3f5c9170 tx=0x55ef3f54a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfe0bba596ff540f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:21.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.214+0000 7f7ca3693640 1 -- 172.21.15.67:0/3191124449 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55ef3e7d49c0 con 0x55ef3e7aec00 2024-03-24T00:46:21.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.214+0000 7f7ca3693640 1 -- 172.21.15.67:0/3191124449 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55ef3e7f8e00 con 0x55ef3e7aec00 2024-03-24T00:46:21.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.214+0000 7f7ca3693640 1 -- 172.21.15.67:0/3191124449 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55ef3e7d4680 con 0x55ef3e7aec00 2024-03-24T00:46:21.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.214+0000 7f7ca842a740 1 -- 172.21.15.67:0/3191124449 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ef3e7db4a0 con 0x55ef3e7aec00 2024-03-24T00:46:21.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.214+0000 7f7ca842a740 1 -- 172.21.15.67:0/3191124449 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x55ef3e7dba40 con 0x55ef3e7aec00 2024-03-24T00:46:21.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.214+0000 7f7ca842a740 4 waiting for MDS map... 2024-03-24T00:46:21.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.214+0000 7f7ca3693640 1 -- 172.21.15.67:0/3191124449 <== mon.0 v2:172.21.15.67:3300/0 4 ==== osd_map(93..93 src has 1..93) v4 ==== 6389+0+0 (secure 0 0 0) 0x55ef3f68a200 con 0x55ef3e7aec00 2024-03-24T00:46:21.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.214+0000 7f7ca3693640 1 -- 172.21.15.67:0/3191124449 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 62) v1 ==== 2735+0+0 (secure 0 0 0) 0x55ef3e807180 con 0x55ef3e7aec00 2024-03-24T00:46:21.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.214+0000 7f7ca842a740 4 Got MDS map 62 2024-03-24T00:46:21.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.215+0000 7f7ca842a740 4 datascan.main: Using metadata pool output 2024-03-24T00:46:21.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.215+0000 7f7ca842a740 4 datascan.main: connecting to RADOS... 2024-03-24T00:46:21.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.215+0000 7f7ca842a740 1 Processor -- start 2024-03-24T00:46:21.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.215+0000 7f7ca842a740 1 -- start start 2024-03-24T00:46:21.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.215+0000 7f7ca842a740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ef3f6d4000 0x55ef3e859600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:21.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.216+0000 7f7ca842a740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ef3f6d4400 0x55ef3e859b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:21.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.216+0000 7f7ca842a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ef3f6d4800 0x55ef3e85a100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:21.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.216+0000 7f7ca842a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef3f66c600 con 0x55ef3f6d4800 2024-03-24T00:46:21.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.216+0000 7f7ca842a740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef3f66c780 con 0x55ef3f6d4400 2024-03-24T00:46:21.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.216+0000 7f7ca842a740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ef3f66c900 con 0x55ef3f6d4000 2024-03-24T00:46:21.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.216+0000 7f7ca5697640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ef3f6d4800 0x55ef3e85a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:21.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.216+0000 7f7ca5697640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ef3f6d4800 0x55ef3e85a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55498/0 (socket says 172.21.15.67:55498) 2024-03-24T00:46:21.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.216+0000 7f7ca5697640 1 -- 172.21.15.67:0/824266536 learned_addr learned my addr 172.21.15.67:0/824266536 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:21.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.216+0000 7f7ca6699640 1 --2- 172.21.15.67:0/824266536 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ef3f6d4000 0x55ef3e859600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:21.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.216+0000 7f7ca5e98640 1 --2- 172.21.15.67:0/824266536 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ef3f6d4400 0x55ef3e859b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:21.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.216+0000 7f7ca5697640 1 -- 172.21.15.67:0/824266536 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ef3f6d4000 msgr2=0x55ef3e859600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:21.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.216+0000 7f7ca5697640 1 --2- 172.21.15.67:0/824266536 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ef3f6d4000 0x55ef3e859600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.216+0000 7f7ca5697640 1 -- 172.21.15.67:0/824266536 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ef3f6d4400 msgr2=0x55ef3e859b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:21.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.216+0000 7f7ca5697640 1 --2- 172.21.15.67:0/824266536 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ef3f6d4400 0x55ef3e859b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.216+0000 7f7ca5697640 1 -- 172.21.15.67:0/824266536 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ef3e7db860 con 0x55ef3f6d4800 2024-03-24T00:46:21.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.216+0000 7f7ca5e98640 1 --2- 172.21.15.67:0/824266536 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ef3f6d4400 0x55ef3e859b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:46:21.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.216+0000 7f7ca5697640 1 --2- 172.21.15.67:0/824266536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ef3f6d4800 0x55ef3e85a100 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x55ef3f5b07e0 tx=0x55ef3f675800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2362c06fc33b723 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:21.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.216+0000 7f7ca168f640 1 -- 172.21.15.67:0/824266536 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55ef3e7d49c0 con 0x55ef3f6d4800 2024-03-24T00:46:21.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.216+0000 7f7ca168f640 1 -- 172.21.15.67:0/824266536 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55ef3e7f8a80 con 0x55ef3f6d4800 2024-03-24T00:46:21.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.216+0000 7f7ca168f640 1 -- 172.21.15.67:0/824266536 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55ef3e7d4680 con 0x55ef3f6d4800 2024-03-24T00:46:21.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.217+0000 7f7ca842a740 1 -- 172.21.15.67:0/824266536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ef3f6d4800 msgr2=0x55ef3e85a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:21.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.217+0000 7f7ca842a740 1 --2- 172.21.15.67:0/824266536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ef3f6d4800 0x55ef3e85a100 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x55ef3f5b07e0 tx=0x55ef3f675800 comp rx=0 tx=0).stop 2024-03-24T00:46:21.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.217+0000 7f7ca842a740 1 -- 172.21.15.67:0/824266536 shutdown_connections 2024-03-24T00:46:21.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.217+0000 7f7ca842a740 1 --2- 172.21.15.67:0/824266536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ef3f6d4800 0x55ef3e85a100 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.217+0000 7f7ca842a740 1 --2- 172.21.15.67:0/824266536 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ef3f6d4400 0x55ef3e859b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.217+0000 7f7ca842a740 1 --2- 172.21.15.67:0/824266536 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ef3f6d4000 0x55ef3e859600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.217+0000 7f7ca842a740 1 -- 172.21.15.67:0/824266536 >> 172.21.15.67:0/824266536 conn(0x55ef3f595c00 msgr2=0x55ef3f537000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:21.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.217+0000 7f7ca842a740 1 -- 172.21.15.67:0/824266536 shutdown_connections 2024-03-24T00:46:21.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.217+0000 7f7ca842a740 1 -- 172.21.15.67:0/824266536 wait complete. 2024-03-24T00:46:21.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.217+0000 7f7ca842a740 1 Processor -- start 2024-03-24T00:46:21.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.217+0000 7f7ca842a740 1 -- start start 2024-03-24T00:46:21.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.218+0000 7f7ca842a740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ef3f6d4000 0x55ef3e85b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:21.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.218+0000 7f7ca842a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ef3f6d4400 0x55ef3e85ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:21.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.218+0000 7f7ca842a740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ef3f6d4800 0x55ef3e85a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:21.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.218+0000 7f7ca842a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef3f66dc80 con 0x55ef3f6d4400 2024-03-24T00:46:21.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.218+0000 7f7ca842a740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef3f66db00 con 0x55ef3f6d4000 2024-03-24T00:46:21.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.218+0000 7f7ca842a740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ef3f66d980 con 0x55ef3f6d4800 2024-03-24T00:46:21.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.218+0000 7f7ca5e98640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ef3f6d4400 0x55ef3e85ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:21.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.218+0000 7f7ca5e98640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ef3f6d4400 0x55ef3e85ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55512/0 (socket says 172.21.15.67:55512) 2024-03-24T00:46:21.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.218+0000 7f7ca5e98640 1 -- 172.21.15.67:0/2570979052 learned_addr learned my addr 172.21.15.67:0/2570979052 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:21.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.218+0000 7f7ca6699640 1 --2- 172.21.15.67:0/2570979052 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ef3f6d4000 0x55ef3e85b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:21.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.218+0000 7f7ca5697640 1 --2- 172.21.15.67:0/2570979052 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ef3f6d4800 0x55ef3e85a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:21.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.218+0000 7f7ca5e98640 1 -- 172.21.15.67:0/2570979052 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ef3f6d4800 msgr2=0x55ef3e85a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:21.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.218+0000 7f7ca5e98640 1 --2- 172.21.15.67:0/2570979052 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ef3f6d4800 0x55ef3e85a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.218+0000 7f7ca5e98640 1 -- 172.21.15.67:0/2570979052 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ef3f6d4000 msgr2=0x55ef3e85b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:21.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.218+0000 7f7ca5e98640 1 --2- 172.21.15.67:0/2570979052 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ef3f6d4000 0x55ef3e85b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.218+0000 7f7ca5e98640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ef3e7db2c0 con 0x55ef3f6d4400 2024-03-24T00:46:21.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.218+0000 7f7ca5e98640 1 --2- 172.21.15.67:0/2570979052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ef3f6d4400 0x55ef3e85ac00 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x55ef3f5c9dd0 tx=0x55ef3e864ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:21.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.218+0000 7f7c9fe8c640 1 -- 172.21.15.67:0/2570979052 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55ef3e7d4b60 con 0x55ef3f6d4400 2024-03-24T00:46:21.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.218+0000 7f7c9fe8c640 1 -- 172.21.15.67:0/2570979052 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55ef3e7f8c40 con 0x55ef3f6d4400 2024-03-24T00:46:21.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.218+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55ef3e7db0e0 con 0x55ef3f6d4400 2024-03-24T00:46:21.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.218+0000 7f7c9fe8c640 1 -- 172.21.15.67:0/2570979052 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55ef3e7d4820 con 0x55ef3f6d4400 2024-03-24T00:46:21.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.218+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ef3e7da3c0 con 0x55ef3f6d4400 2024-03-24T00:46:21.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.219+0000 7f7ca842a740 4 datascan.init: resolving metadata pool 13 2024-03-24T00:46:21.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.220+0000 7f7c9fe8c640 1 -- 172.21.15.67:0/2570979052 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x55ef3f6e4c00 con 0x55ef3f6d4400 2024-03-24T00:46:21.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.220+0000 7f7c9fe8c640 1 --2- 172.21.15.67:0/2570979052 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x55ef3f6ff000 0x55ef3e85b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:21.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.220+0000 7f7c9fe8c640 1 -- 172.21.15.67:0/2570979052 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6389+0+0 (secure 0 0 0) 0x55ef3f6db200 con 0x55ef3f6d4400 2024-03-24T00:46:21.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.220+0000 7f7ca842a740 4 datascan.init: found metadata pool 'cephfs_metadata' 2024-03-24T00:46:21.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.220+0000 7f7ca842a740 4 datascan.main: resolving metadata pool 13 2024-03-24T00:46:21.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.220+0000 7f7ca842a740 1 --2- 172.21.15.67:0/2570979052 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x55ef3f6d4c00 0x55ef3e85a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:21.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.220+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:1 13.0 13:00000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3f6d5000 con 0x55ef3f6d4c00 2024-03-24T00:46:21.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.220+0000 7f7ca6699640 1 --2- 172.21.15.67:0/2570979052 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x55ef3f6ff000 0x55ef3e85b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:21.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.220+0000 7f7ca5697640 1 --2- 172.21.15.67:0/2570979052 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x55ef3f6d4c00 0x55ef3e85a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:21.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.221+0000 7f7ca6699640 1 --2- 172.21.15.67:0/2570979052 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x55ef3f6ff000 0x55ef3e85b700 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x55ef3f54f020 tx=0x55ef3eabef00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:21.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.221+0000 7f7ca5697640 1 --2- 172.21.15.67:0/2570979052 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x55ef3f6d4c00 0x55ef3e85a100 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:21.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.221+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 <== osd.7 v2:172.21.15.72:6824/2893038746 1 ==== osd_op_reply(1 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55ef3f546000 con 0x55ef3f6d4c00 2024-03-24T00:46:21.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.221+0000 7f7ca5697640 1 --2- 172.21.15.67:0/2570979052 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x55ef3f6d5c00 0x55ef3eb3c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:21.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.221+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:2 13.10 13:08000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3f6d5400 con 0x55ef3f6d5c00 2024-03-24T00:46:21.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.221+0000 7f7ca6699640 1 --2- 172.21.15.67:0/2570979052 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x55ef3f6d5c00 0x55ef3eb3c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:21.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.222+0000 7f7ca6699640 1 --2- 172.21.15.67:0/2570979052 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x55ef3f6d5c00 0x55ef3eb3c000 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:21.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.222+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 <== osd.1 v2:172.21.15.67:6800/4062510404 1 ==== osd_op_reply(2 [pgnls start_epoch 93 out=177b] v93'18 uv18 ondisk = 1) v8 ==== 144+0+177 (crc 0 0 0) 0x55ef3f546500 con 0x55ef3f6d5c00 2024-03-24T00:46:21.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.222+0000 7f7ca6699640 1 --2- 172.21.15.67:0/2570979052 >> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] conn(0x55ef3f6d5400 0x55ef3eb3c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:21.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.222+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] -- osd_op(unknown.0.0:3 13.8 13:10000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3f6ffc00 con 0x55ef3f6d5400 2024-03-24T00:46:21.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.222+0000 7f7ca5e98640 1 --2- 172.21.15.67:0/2570979052 >> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] conn(0x55ef3f6d5400 0x55ef3eb3c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:21.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.222+0000 7f7ca5e98640 1 --2- 172.21.15.67:0/2570979052 >> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] conn(0x55ef3f6d5400 0x55ef3eb3c580 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:21.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.223+0000 7f7ca5e98640 1 -- 172.21.15.67:0/2570979052 <== osd.2 v2:172.21.15.67:6817/1268432561 1 ==== osd_op_reply(3 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55ef3f546780 con 0x55ef3f6d5400 2024-03-24T00:46:21.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.223+0000 7f7ca5e98640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:4 13.18 13:18000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3f6ffc00 con 0x55ef3f6d4c00 2024-03-24T00:46:21.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.223+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 <== osd.7 v2:172.21.15.72:6824/2893038746 2 ==== osd_op_reply(4 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55ef3f546000 con 0x55ef3f6d4c00 2024-03-24T00:46:21.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.223+0000 7f7ca5697640 1 --2- 172.21.15.67:0/2570979052 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x55ef3f6ffc00 0x55ef3eb3cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:21.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.223+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:5 13.4 13:20000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3f6d5800 con 0x55ef3f6ffc00 2024-03-24T00:46:21.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.223+0000 7f7ca5697640 1 --2- 172.21.15.67:0/2570979052 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x55ef3f6ffc00 0x55ef3eb3cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:21.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.223+0000 7f7ca5697640 1 --2- 172.21.15.67:0/2570979052 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x55ef3f6ffc00 0x55ef3eb3cb00 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:21.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.224+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 <== osd.6 v2:172.21.15.72:6816/1725206562 1 ==== osd_op_reply(5 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55ef3f546000 con 0x55ef3f6ffc00 2024-03-24T00:46:21.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.224+0000 7f7ca5697640 1 --2- 172.21.15.67:0/2570979052 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x55ef3f6d5800 0x55ef3eb3d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:21.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.224+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:6 13.14 13:28000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3f6d5000 con 0x55ef3f6d5800 2024-03-24T00:46:21.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.224+0000 7f7ca6699640 1 --2- 172.21.15.67:0/2570979052 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x55ef3f6d5800 0x55ef3eb3d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:21.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.224+0000 7f7ca6699640 1 --2- 172.21.15.67:0/2570979052 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x55ef3f6d5800 0x55ef3eb3d080 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:21.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.224+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 <== osd.0 v2:172.21.15.67:6808/2628130519 1 ==== osd_op_reply(6 [pgnls start_epoch 93 out=149b] v92'27 uv27 ondisk = 1) v8 ==== 144+0+149 (crc 0 0 0) 0x55ef3f546500 con 0x55ef3f6d5800 2024-03-24T00:46:21.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.224+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:7 13.c 13:30000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3f6d5000 con 0x55ef3f6d5c00 2024-03-24T00:46:21.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.225+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 <== osd.1 v2:172.21.15.67:6800/4062510404 2 ==== osd_op_reply(7 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55ef3f546500 con 0x55ef3f6d5c00 2024-03-24T00:46:21.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.225+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:8 13.1c 13:38000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3f6d5000 con 0x55ef3f6ffc00 2024-03-24T00:46:21.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.225+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 <== osd.6 v2:172.21.15.72:6816/1725206562 2 ==== osd_op_reply(8 [pgnls start_epoch 93 out=73b] v92'3 uv3 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x55ef3f546000 con 0x55ef3f6ffc00 2024-03-24T00:46:21.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.225+0000 7f7ca5697640 1 --2- 172.21.15.67:0/2570979052 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x55ef3f6d5000 0x55ef3eb3d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:21.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.225+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:9 13.2 13:40000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb52000 con 0x55ef3f6d5000 2024-03-24T00:46:21.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.225+0000 7f7ca5e98640 1 --2- 172.21.15.67:0/2570979052 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x55ef3f6d5000 0x55ef3eb3d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:21.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.225+0000 7f7ca5e98640 1 --2- 172.21.15.67:0/2570979052 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x55ef3f6d5000 0x55ef3eb3d600 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:21.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.226+0000 7f7ca5e98640 1 -- 172.21.15.67:0/2570979052 <== osd.5 v2:172.21.15.72:6800/288486206 1 ==== osd_op_reply(9 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55ef3f546780 con 0x55ef3f6d5000 2024-03-24T00:46:21.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.226+0000 7f7ca5e98640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:10 13.12 13:48000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb52000 con 0x55ef3f6d5800 2024-03-24T00:46:21.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.226+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 <== osd.0 v2:172.21.15.67:6808/2628130519 2 ==== osd_op_reply(10 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55ef3f546500 con 0x55ef3f6d5800 2024-03-24T00:46:21.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.226+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] -- osd_op(unknown.0.0:11 13.a 13:50000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb52000 con 0x55ef3f6d5400 2024-03-24T00:46:21.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.226+0000 7f7ca5e98640 1 -- 172.21.15.67:0/2570979052 <== osd.2 v2:172.21.15.67:6817/1268432561 2 ==== osd_op_reply(11 [pgnls start_epoch 93 out=81b] v92'3 uv3 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x55ef3f546780 con 0x55ef3f6d5400 2024-03-24T00:46:21.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.226+0000 7f7ca5e98640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:12 13.1a 13:58000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb52000 con 0x55ef3f6d5c00 2024-03-24T00:46:21.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.227+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 <== osd.1 v2:172.21.15.67:6800/4062510404 3 ==== osd_op_reply(12 [pgnls start_epoch 93 out=49b] v90'3 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55ef3f546500 con 0x55ef3f6d5c00 2024-03-24T00:46:21.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.227+0000 7f7ca6699640 1 --2- 172.21.15.67:0/2570979052 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x55ef3eb52000 0x55ef3eb3e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:21.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.227+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:13 13.6 13:60000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb53400 con 0x55ef3eb52000 2024-03-24T00:46:21.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.227+0000 7f7ca5697640 1 --2- 172.21.15.67:0/2570979052 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x55ef3eb52000 0x55ef3eb3e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:21.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.227+0000 7f7ca5697640 1 --2- 172.21.15.67:0/2570979052 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x55ef3eb52000 0x55ef3eb3e680 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:21.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.227+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 <== osd.4 v2:172.21.15.72:6808/477220793 1 ==== osd_op_reply(13 [pgnls start_epoch 93 out=77b] v92'8 uv8 ondisk = 1) v8 ==== 144+0+77 (crc 0 0 0) 0x55ef3f546000 con 0x55ef3eb52000 2024-03-24T00:46:21.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.227+0000 7f7ca5697640 1 --2- 172.21.15.67:0/2570979052 >> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] conn(0x55ef3eb53400 0x55ef3eb3db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:21.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.227+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:14 13.16 13:68000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb52400 con 0x55ef3eb53400 2024-03-24T00:46:21.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.227+0000 7f7ca6699640 1 --2- 172.21.15.67:0/2570979052 >> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] conn(0x55ef3eb53400 0x55ef3eb3db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:21.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.228+0000 7f7ca6699640 1 --2- 172.21.15.67:0/2570979052 >> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] conn(0x55ef3eb53400 0x55ef3eb3db80 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:21.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.228+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 <== osd.3 v2:172.21.15.67:6816/3303489481 1 ==== osd_op_reply(14 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55ef3f546500 con 0x55ef3eb53400 2024-03-24T00:46:21.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.228+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:15 13.e 13:70000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb52400 con 0x55ef3f6d4c00 2024-03-24T00:46:21.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.228+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 <== osd.7 v2:172.21.15.72:6824/2893038746 3 ==== osd_op_reply(15 [pgnls start_epoch 93 out=73b] v92'3 uv3 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x55ef3f546000 con 0x55ef3f6d4c00 2024-03-24T00:46:21.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.228+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:16 13.1e 13:78000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb52400 con 0x55ef3eb52000 2024-03-24T00:46:21.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.229+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 <== osd.4 v2:172.21.15.72:6808/477220793 2 ==== osd_op_reply(16 [pgnls start_epoch 93 out=49b] v90'3 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55ef3f546000 con 0x55ef3eb52000 2024-03-24T00:46:21.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.229+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:17 13.1 13:80000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb52400 con 0x55ef3f6ffc00 2024-03-24T00:46:21.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.229+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 <== osd.6 v2:172.21.15.72:6816/1725206562 3 ==== osd_op_reply(17 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55ef3f546000 con 0x55ef3f6ffc00 2024-03-24T00:46:21.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.229+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:18 13.11 13:88000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb52400 con 0x55ef3f6d5c00 2024-03-24T00:46:21.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.229+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 <== osd.1 v2:172.21.15.67:6800/4062510404 4 ==== osd_op_reply(18 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55ef3f546500 con 0x55ef3f6d5c00 2024-03-24T00:46:21.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.229+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:19 13.9 13:90000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb52400 con 0x55ef3f6d5000 2024-03-24T00:46:21.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.229+0000 7f7ca5e98640 1 -- 172.21.15.67:0/2570979052 <== osd.5 v2:172.21.15.72:6800/288486206 2 ==== osd_op_reply(19 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55ef3f546780 con 0x55ef3f6d5000 2024-03-24T00:46:21.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.229+0000 7f7ca5e98640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:20 13.19 13:98000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb52400 con 0x55ef3eb53400 2024-03-24T00:46:21.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.230+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 <== osd.3 v2:172.21.15.67:6816/3303489481 2 ==== osd_op_reply(20 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55ef3f546500 con 0x55ef3eb53400 2024-03-24T00:46:21.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.230+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:21 13.5 13:a0000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb52400 con 0x55ef3eb52000 2024-03-24T00:46:21.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.230+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 <== osd.4 v2:172.21.15.72:6808/477220793 3 ==== osd_op_reply(21 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55ef3f546000 con 0x55ef3eb52000 2024-03-24T00:46:21.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.230+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:22 13.15 13:a8000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb52400 con 0x55ef3eb52000 2024-03-24T00:46:21.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.230+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 <== osd.4 v2:172.21.15.72:6808/477220793 4 ==== osd_op_reply(22 [pgnls start_epoch 93 out=97b] v92'4 uv4 ondisk = 1) v8 ==== 144+0+97 (crc 0 0 0) 0x55ef3f546000 con 0x55ef3eb52000 2024-03-24T00:46:21.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.230+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:23 13.d 13:b0000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb52400 con 0x55ef3eb52000 2024-03-24T00:46:21.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.230+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 <== osd.4 v2:172.21.15.72:6808/477220793 5 ==== osd_op_reply(23 [pgnls start_epoch 93 out=74b] v88'1 uv1 ondisk = 1) v8 ==== 144+0+74 (crc 0 0 0) 0x55ef3f546000 con 0x55ef3eb52000 2024-03-24T00:46:21.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.230+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:24 13.1d 13:b8000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb52400 con 0x55ef3f6d4c00 2024-03-24T00:46:21.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.231+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 <== osd.7 v2:172.21.15.72:6824/2893038746 4 ==== osd_op_reply(24 [pgnls start_epoch 93 out=73b] v92'5 uv5 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x55ef3f546000 con 0x55ef3f6d4c00 2024-03-24T00:46:21.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.231+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:25 13.3 13:c0000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb52400 con 0x55ef3f6d5800 2024-03-24T00:46:21.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.231+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 <== osd.0 v2:172.21.15.67:6808/2628130519 3 ==== osd_op_reply(25 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55ef3f546500 con 0x55ef3f6d5800 2024-03-24T00:46:21.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.231+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:26 13.13 13:c8000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb52400 con 0x55ef3f6d4c00 2024-03-24T00:46:21.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.231+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 <== osd.7 v2:172.21.15.72:6824/2893038746 5 ==== osd_op_reply(26 [pgnls start_epoch 93 out=98b] v92'4 uv4 ondisk = 1) v8 ==== 144+0+98 (crc 0 0 0) 0x55ef3f546000 con 0x55ef3f6d4c00 2024-03-24T00:46:21.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.231+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:27 13.b 13:d0000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb52400 con 0x55ef3f6d5000 2024-03-24T00:46:21.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.232+0000 7f7ca5e98640 1 -- 172.21.15.67:0/2570979052 <== osd.5 v2:172.21.15.72:6800/288486206 3 ==== osd_op_reply(27 [pgnls start_epoch 93 out=76b] v92'6 uv6 ondisk = 1) v8 ==== 144+0+76 (crc 0 0 0) 0x55ef3f546780 con 0x55ef3f6d5000 2024-03-24T00:46:21.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.232+0000 7f7ca5e98640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:28 13.1b 13:d8000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb52400 con 0x55ef3eb52000 2024-03-24T00:46:21.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.232+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 <== osd.4 v2:172.21.15.72:6808/477220793 6 ==== osd_op_reply(28 [pgnls start_epoch 93 out=73b] v92'10 uv10 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x55ef3f546000 con 0x55ef3eb52000 2024-03-24T00:46:21.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.232+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:29 13.7 13:e0000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb52400 con 0x55ef3f6d4c00 2024-03-24T00:46:21.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.232+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 <== osd.7 v2:172.21.15.72:6824/2893038746 6 ==== osd_op_reply(29 [pgnls start_epoch 93 out=73b] v92'3 uv3 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x55ef3f546000 con 0x55ef3f6d4c00 2024-03-24T00:46:21.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.232+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:30 13.17 13:e8000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb52400 con 0x55ef3f6d5000 2024-03-24T00:46:21.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.233+0000 7f7ca5e98640 1 -- 172.21.15.67:0/2570979052 <== osd.5 v2:172.21.15.72:6800/288486206 4 ==== osd_op_reply(30 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55ef3f546780 con 0x55ef3f6d5000 2024-03-24T00:46:21.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.233+0000 7f7ca5e98640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:31 13.f 13:f0000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb52400 con 0x55ef3f6ffc00 2024-03-24T00:46:21.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.233+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 <== osd.6 v2:172.21.15.72:6816/1725206562 4 ==== osd_op_reply(31 [pgnls start_epoch 93 out=73b] v88'1 uv1 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x55ef3f546000 con 0x55ef3f6ffc00 2024-03-24T00:46:21.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.233+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:32 13.1f 13:f8000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb52400 con 0x55ef3f6ffc00 2024-03-24T00:46:21.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.233+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 <== osd.6 v2:172.21.15.72:6816/1725206562 5 ==== osd_op_reply(32 [pgnls start_epoch 93 out=101b] v92'9 uv9 ondisk = 1) v8 ==== 144+0+101 (crc 0 0 0) 0x55ef3f546000 con 0x55ef3f6ffc00 2024-03-24T00:46:21.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.233+0000 7f7ca842a740 10 datascan.scan_links: step 1: handling object 601.00000000 2024-03-24T00:46:21.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.233+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:33 13.10 13:09401a81:::601.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb60000 con 0x55ef3f6d5c00 2024-03-24T00:46:21.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.234+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 <== osd.1 v2:172.21.15.67:6800/4062510404 5 ==== osd_op_reply(33 601.00000000 [omap-get-vals out=5b] v0'0 uv13 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55ef3f546500 con 0x55ef3f6d5c00 2024-03-24T00:46:21.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.234+0000 7f7ca842a740 10 datascan.scan_links: step 1: handling object 602.00000000 2024-03-24T00:46:21.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.234+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:34 13.10 13:0bd6d154:::602.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb60400 con 0x55ef3f6d5c00 2024-03-24T00:46:21.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.234+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 <== osd.1 v2:172.21.15.67:6800/4062510404 6 ==== osd_op_reply(34 602.00000000 [omap-get-vals out=5b] v0'0 uv14 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55ef3f546500 con 0x55ef3f6d5c00 2024-03-24T00:46:21.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.234+0000 7f7ca842a740 10 datascan.scan_links: step 1: handling object 10000000000.00000000 2024-03-24T00:46:21.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.234+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:35 13.10 13:0cf6ad0f:::10000000000.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb60800 con 0x55ef3f6d5c00 2024-03-24T00:46:21.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.234+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 <== osd.1 v2:172.21.15.67:6800/4062510404 7 ==== osd_op_reply(35 10000000000.00000000 [omap-get-vals out=506b] v0'0 uv18 ondisk = 0) v8 ==== 164+0+506 (crc 0 0 0) 0x55ef3f546500 con 0x55ef3f6d5c00 2024-03-24T00:46:21.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.234+0000 7f7ca842a740 20 datascan.scan_links: injected ino detected 2024-03-24T00:46:21.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.234+0000 7f7ca842a740 10 datascan.scan_links: step 1: handling object 600.00000000 2024-03-24T00:46:21.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.234+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:36 13.10 13:0d82a743:::600.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb60c00 con 0x55ef3f6d5c00 2024-03-24T00:46:21.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.235+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 <== osd.1 v2:172.21.15.67:6800/4062510404 8 ==== osd_op_reply(36 600.00000000 [omap-get-vals out=5b] v0'0 uv12 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55ef3f546500 con 0x55ef3f6d5c00 2024-03-24T00:46:21.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.235+0000 7f7ca842a740 10 datascan.scan_links: step 1: handling object 603.00000000 2024-03-24T00:46:21.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.235+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:37 13.10 13:0d89b25e:::603.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb61000 con 0x55ef3f6d5c00 2024-03-24T00:46:21.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.235+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 <== osd.1 v2:172.21.15.67:6800/4062510404 9 ==== osd_op_reply(37 603.00000000 [omap-get-vals out=5b] v0'0 uv15 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55ef3f546500 con 0x55ef3f6d5c00 2024-03-24T00:46:21.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.235+0000 7f7ca842a740 10 datascan.scan_links: step 1: handling object 1.00000000.inode 2024-03-24T00:46:21.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.235+0000 7f7ca842a740 10 datascan.scan_links: Not a dirfrag: '1.00000000.inode' 2024-03-24T00:46:21.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.235+0000 7f7ca842a740 10 datascan.scan_links: step 1: handling object 200.00000000 2024-03-24T00:46:21.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.235+0000 7f7ca842a740 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000000' 2024-03-24T00:46:21.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.235+0000 7f7ca842a740 10 datascan.scan_links: step 1: handling object 200.00000001 2024-03-24T00:46:21.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.235+0000 7f7ca842a740 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000001' 2024-03-24T00:46:21.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.235+0000 7f7ca842a740 10 datascan.scan_links: step 1: handling object 606.00000000 2024-03-24T00:46:21.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.235+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:38 13.14 13:2f55791f:::606.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb61400 con 0x55ef3f6d5800 2024-03-24T00:46:21.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.235+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 <== osd.0 v2:172.21.15.67:6808/2628130519 4 ==== osd_op_reply(38 606.00000000 [omap-get-vals out=5b] v0'0 uv25 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55ef3f546500 con 0x55ef3f6d5800 2024-03-24T00:46:21.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.235+0000 7f7ca842a740 10 datascan.scan_links: step 1: handling object 607.00000000 2024-03-24T00:46:21.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.235+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:39 13.1c 13:3ed09add:::607.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb60000 con 0x55ef3f6ffc00 2024-03-24T00:46:21.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.236+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 <== osd.6 v2:172.21.15.72:6816/1725206562 6 ==== osd_op_reply(39 607.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55ef3f546000 con 0x55ef3f6ffc00 2024-03-24T00:46:21.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.236+0000 7f7ca842a740 10 datascan.scan_links: step 1: handling object 100000001f6.00000000 2024-03-24T00:46:21.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.236+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] -- osd_op(unknown.0.0:40 13.a 13:57fe7535:::100000001f6.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb60400 con 0x55ef3f6d5400 2024-03-24T00:46:21.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.236+0000 7f7ca5e98640 1 -- 172.21.15.67:0/2570979052 <== osd.2 v2:172.21.15.67:6817/1268432561 3 ==== osd_op_reply(40 100000001f6.00000000 [omap-get-vals out=54b] v0'0 uv3 ondisk = 0) v8 ==== 164+0+54 (crc 0 0 0) 0x55ef3f546780 con 0x55ef3f6d5400 2024-03-24T00:46:21.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.236+0000 7f7ca842a740 10 datascan.scan_links: step 1: handling object mds0_openfiles.0 2024-03-24T00:46:21.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.236+0000 7f7ca842a740 10 datascan.scan_links: Not a dirfrag: 'mds0_openfiles.0' 2024-03-24T00:46:21.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.236+0000 7f7ca842a740 10 datascan.scan_links: step 1: handling object 608.00000000 2024-03-24T00:46:21.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.236+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:41 13.e 13:717a0223:::608.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb60800 con 0x55ef3f6d4c00 2024-03-24T00:46:21.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.236+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 <== osd.7 v2:172.21.15.72:6824/2893038746 7 ==== osd_op_reply(41 608.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55ef3f546000 con 0x55ef3f6d4c00 2024-03-24T00:46:21.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.236+0000 7f7ca842a740 10 datascan.scan_links: step 1: handling object 604.00000000 2024-03-24T00:46:21.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.236+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:42 13.15 13:a93a17c2:::604.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb61800 con 0x55ef3eb52000 2024-03-24T00:46:21.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.237+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 <== osd.4 v2:172.21.15.72:6808/477220793 7 ==== osd_op_reply(42 604.00000000 [omap-get-vals out=5b] v0'0 uv4 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55ef3f546000 con 0x55ef3eb52000 2024-03-24T00:46:21.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.237+0000 7f7ca842a740 10 datascan.scan_links: step 1: handling object 500.00000000 2024-03-24T00:46:21.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.237+0000 7f7ca842a740 10 datascan.scan_links: Not a dirfrag (invalid ino): '500.00000000' 2024-03-24T00:46:21.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.237+0000 7f7ca842a740 10 datascan.scan_links: step 1: handling object mds_snaptable 2024-03-24T00:46:21.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.237+0000 7f7ca842a740 10 datascan.scan_links: Not a dirfrag: 'mds_snaptable' 2024-03-24T00:46:21.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.237+0000 7f7ca842a740 10 datascan.scan_links: step 1: handling object 605.00000000 2024-03-24T00:46:21.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.237+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:43 13.1d 13:b871830b:::605.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb61c00 con 0x55ef3f6d4c00 2024-03-24T00:46:21.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.237+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 <== osd.7 v2:172.21.15.72:6824/2893038746 8 ==== osd_op_reply(43 605.00000000 [omap-get-vals out=5b] v0'0 uv5 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55ef3f546000 con 0x55ef3f6d4c00 2024-03-24T00:46:21.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.237+0000 7f7ca842a740 10 datascan.scan_links: step 1: handling object mds0_inotable 2024-03-24T00:46:21.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.237+0000 7f7ca842a740 10 datascan.scan_links: Not a dirfrag: 'mds0_inotable' 2024-03-24T00:46:21.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.237+0000 7f7ca842a740 10 datascan.scan_links: step 1: handling object 100.00000000 2024-03-24T00:46:21.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.237+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:44 13.13 13:cd5a64a3:::100.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb53c00 con 0x55ef3f6d4c00 2024-03-24T00:46:21.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.237+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 <== osd.7 v2:172.21.15.72:6824/2893038746 9 ==== osd_op_reply(44 100.00000000 [omap-get-vals out=5195b] v0'0 uv4 ondisk = 0) v8 ==== 156+0+5195 (crc 0 0 0) 0x55ef3f546000 con 0x55ef3f6d4c00 2024-03-24T00:46:21.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.237+0000 7f7ca842a740 10 datascan.scan_links: step 1: handling object mds0_sessionmap 2024-03-24T00:46:21.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.237+0000 7f7ca842a740 10 datascan.scan_links: Not a dirfrag: 'mds0_sessionmap' 2024-03-24T00:46:21.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.237+0000 7f7ca842a740 10 datascan.scan_links: step 1: handling object 200.00000002 2024-03-24T00:46:21.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.237+0000 7f7ca842a740 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000002' 2024-03-24T00:46:21.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.237+0000 7f7ca842a740 10 datascan.scan_links: step 1: handling object 609.00000000 2024-03-24T00:46:21.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.237+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:45 13.7 13:e0b41d06:::609.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb53800 con 0x55ef3f6d4c00 2024-03-24T00:46:21.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.238+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 <== osd.7 v2:172.21.15.72:6824/2893038746 10 ==== osd_op_reply(45 609.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55ef3f546000 con 0x55ef3f6d4c00 2024-03-24T00:46:21.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.238+0000 7f7ca842a740 10 datascan.scan_links: step 1: handling object 400.00000000 2024-03-24T00:46:21.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.238+0000 7f7ca842a740 10 datascan.scan_links: Not a dirfrag (invalid ino): '400.00000000' 2024-03-24T00:46:21.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.238+0000 7f7ca842a740 10 datascan.scan_links: step 1: handling object 100.00000000.inode 2024-03-24T00:46:21.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.238+0000 7f7ca842a740 10 datascan.scan_links: Not a dirfrag: '100.00000000.inode' 2024-03-24T00:46:21.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.238+0000 7f7ca842a740 10 datascan.scan_links: step 1: handling object 1.00000000 2024-03-24T00:46:21.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.238+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:46 13.1f 13:ff5b34d6:::1.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb60000 con 0x55ef3f6ffc00 2024-03-24T00:46:21.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.238+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 <== osd.6 v2:172.21.15.72:6816/1725206562 7 ==== osd_op_reply(46 1.00000000 [omap-get-vals out=1163b] v0'0 uv9 ondisk = 0) v8 ==== 154+0+1163 (crc 0 0 0) 0x55ef3f546000 con 0x55ef3f6ffc00 2024-03-24T00:46:21.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.238+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:47 13.0 13:00000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb60800 con 0x55ef3f6d4c00 2024-03-24T00:46:21.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.238+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 <== osd.7 v2:172.21.15.72:6824/2893038746 11 ==== osd_op_reply(47 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55ef3f546000 con 0x55ef3f6d4c00 2024-03-24T00:46:21.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.238+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:48 13.10 13:08000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb60800 con 0x55ef3f6d5c00 2024-03-24T00:46:21.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.239+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 <== osd.1 v2:172.21.15.67:6800/4062510404 10 ==== osd_op_reply(48 [pgnls start_epoch 93 out=177b] v93'18 uv18 ondisk = 1) v8 ==== 144+0+177 (crc 0 0 0) 0x55ef3f546500 con 0x55ef3f6d5c00 2024-03-24T00:46:21.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.239+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] -- osd_op(unknown.0.0:49 13.8 13:10000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb60800 con 0x55ef3f6d5400 2024-03-24T00:46:21.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.239+0000 7f7ca5e98640 1 -- 172.21.15.67:0/2570979052 <== osd.2 v2:172.21.15.67:6817/1268432561 4 ==== osd_op_reply(49 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55ef3f546780 con 0x55ef3f6d5400 2024-03-24T00:46:21.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.239+0000 7f7ca5e98640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:50 13.18 13:18000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb60800 con 0x55ef3f6d4c00 2024-03-24T00:46:21.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.239+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 <== osd.7 v2:172.21.15.72:6824/2893038746 12 ==== osd_op_reply(50 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55ef3f546000 con 0x55ef3f6d4c00 2024-03-24T00:46:21.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.239+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:51 13.4 13:20000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb60800 con 0x55ef3f6ffc00 2024-03-24T00:46:21.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.239+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 <== osd.6 v2:172.21.15.72:6816/1725206562 8 ==== osd_op_reply(51 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55ef3f546000 con 0x55ef3f6ffc00 2024-03-24T00:46:21.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.239+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:52 13.14 13:28000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb60800 con 0x55ef3f6d5800 2024-03-24T00:46:21.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.240+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 <== osd.0 v2:172.21.15.67:6808/2628130519 5 ==== osd_op_reply(52 [pgnls start_epoch 93 out=149b] v92'27 uv27 ondisk = 1) v8 ==== 144+0+149 (crc 0 0 0) 0x55ef3f546500 con 0x55ef3f6d5800 2024-03-24T00:46:21.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.240+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:53 13.c 13:30000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb60800 con 0x55ef3f6d5c00 2024-03-24T00:46:21.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.240+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 <== osd.1 v2:172.21.15.67:6800/4062510404 11 ==== osd_op_reply(53 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55ef3f546500 con 0x55ef3f6d5c00 2024-03-24T00:46:21.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.240+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:54 13.1c 13:38000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb60800 con 0x55ef3f6ffc00 2024-03-24T00:46:21.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.240+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 <== osd.6 v2:172.21.15.72:6816/1725206562 9 ==== osd_op_reply(54 [pgnls start_epoch 93 out=73b] v92'3 uv3 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x55ef3f546000 con 0x55ef3f6ffc00 2024-03-24T00:46:21.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.240+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:55 13.2 13:40000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb60800 con 0x55ef3f6d5000 2024-03-24T00:46:21.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.241+0000 7f7ca5e98640 1 -- 172.21.15.67:0/2570979052 <== osd.5 v2:172.21.15.72:6800/288486206 5 ==== osd_op_reply(55 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55ef3f546780 con 0x55ef3f6d5000 2024-03-24T00:46:21.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.241+0000 7f7ca5e98640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:56 13.12 13:48000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb60800 con 0x55ef3f6d5800 2024-03-24T00:46:21.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.241+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 <== osd.0 v2:172.21.15.67:6808/2628130519 6 ==== osd_op_reply(56 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55ef3f546500 con 0x55ef3f6d5800 2024-03-24T00:46:21.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.241+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] -- osd_op(unknown.0.0:57 13.a 13:50000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb60800 con 0x55ef3f6d5400 2024-03-24T00:46:21.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.241+0000 7f7ca5e98640 1 -- 172.21.15.67:0/2570979052 <== osd.2 v2:172.21.15.67:6817/1268432561 5 ==== osd_op_reply(57 [pgnls start_epoch 93 out=81b] v92'3 uv3 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x55ef3f546780 con 0x55ef3f6d5400 2024-03-24T00:46:21.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.241+0000 7f7ca5e98640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:58 13.1a 13:58000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb60800 con 0x55ef3f6d5c00 2024-03-24T00:46:21.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.242+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 <== osd.1 v2:172.21.15.67:6800/4062510404 12 ==== osd_op_reply(58 [pgnls start_epoch 93 out=49b] v90'3 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55ef3f546500 con 0x55ef3f6d5c00 2024-03-24T00:46:21.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.242+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:59 13.6 13:60000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb60800 con 0x55ef3eb52000 2024-03-24T00:46:21.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.242+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 <== osd.4 v2:172.21.15.72:6808/477220793 8 ==== osd_op_reply(59 [pgnls start_epoch 93 out=77b] v92'8 uv8 ondisk = 1) v8 ==== 144+0+77 (crc 0 0 0) 0x55ef3f546000 con 0x55ef3eb52000 2024-03-24T00:46:21.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.242+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:60 13.16 13:68000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb60800 con 0x55ef3eb53400 2024-03-24T00:46:21.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.242+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 <== osd.3 v2:172.21.15.67:6816/3303489481 3 ==== osd_op_reply(60 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55ef3f546500 con 0x55ef3eb53400 2024-03-24T00:46:21.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.242+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:61 13.e 13:70000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb60800 con 0x55ef3f6d4c00 2024-03-24T00:46:21.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.242+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 <== osd.7 v2:172.21.15.72:6824/2893038746 13 ==== osd_op_reply(61 [pgnls start_epoch 93 out=73b] v92'3 uv3 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x55ef3f546000 con 0x55ef3f6d4c00 2024-03-24T00:46:21.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.242+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:62 13.1e 13:78000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb60800 con 0x55ef3eb52000 2024-03-24T00:46:21.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.243+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 <== osd.4 v2:172.21.15.72:6808/477220793 9 ==== osd_op_reply(62 [pgnls start_epoch 93 out=49b] v90'3 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55ef3f546000 con 0x55ef3eb52000 2024-03-24T00:46:21.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.243+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:63 13.1 13:80000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb60800 con 0x55ef3f6ffc00 2024-03-24T00:46:21.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.243+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 <== osd.6 v2:172.21.15.72:6816/1725206562 10 ==== osd_op_reply(63 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55ef3f546000 con 0x55ef3f6ffc00 2024-03-24T00:46:21.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.243+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:64 13.11 13:88000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb60800 con 0x55ef3f6d5c00 2024-03-24T00:46:21.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.243+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 <== osd.1 v2:172.21.15.67:6800/4062510404 13 ==== osd_op_reply(64 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55ef3f546500 con 0x55ef3f6d5c00 2024-03-24T00:46:21.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.243+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:65 13.9 13:90000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb60800 con 0x55ef3f6d5000 2024-03-24T00:46:21.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.244+0000 7f7ca5e98640 1 -- 172.21.15.67:0/2570979052 <== osd.5 v2:172.21.15.72:6800/288486206 6 ==== osd_op_reply(65 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55ef3f546780 con 0x55ef3f6d5000 2024-03-24T00:46:21.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.244+0000 7f7ca5e98640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:66 13.19 13:98000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb60800 con 0x55ef3eb53400 2024-03-24T00:46:21.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.244+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 <== osd.3 v2:172.21.15.67:6816/3303489481 4 ==== osd_op_reply(66 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55ef3f546500 con 0x55ef3eb53400 2024-03-24T00:46:21.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.244+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:67 13.5 13:a0000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb60800 con 0x55ef3eb52000 2024-03-24T00:46:21.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.244+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 <== osd.4 v2:172.21.15.72:6808/477220793 10 ==== osd_op_reply(67 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55ef3f546000 con 0x55ef3eb52000 2024-03-24T00:46:21.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.244+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:68 13.15 13:a8000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb60800 con 0x55ef3eb52000 2024-03-24T00:46:21.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.245+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 <== osd.4 v2:172.21.15.72:6808/477220793 11 ==== osd_op_reply(68 [pgnls start_epoch 93 out=97b] v92'4 uv4 ondisk = 1) v8 ==== 144+0+97 (crc 0 0 0) 0x55ef3f546000 con 0x55ef3eb52000 2024-03-24T00:46:21.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.245+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:69 13.d 13:b0000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb60800 con 0x55ef3eb52000 2024-03-24T00:46:21.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.245+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 <== osd.4 v2:172.21.15.72:6808/477220793 12 ==== osd_op_reply(69 [pgnls start_epoch 93 out=74b] v88'1 uv1 ondisk = 1) v8 ==== 144+0+74 (crc 0 0 0) 0x55ef3f546000 con 0x55ef3eb52000 2024-03-24T00:46:21.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.245+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:70 13.1d 13:b8000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb60800 con 0x55ef3f6d4c00 2024-03-24T00:46:21.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.245+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 <== osd.7 v2:172.21.15.72:6824/2893038746 14 ==== osd_op_reply(70 [pgnls start_epoch 93 out=73b] v92'5 uv5 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x55ef3f546000 con 0x55ef3f6d4c00 2024-03-24T00:46:21.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.245+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:71 13.3 13:c0000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb60800 con 0x55ef3f6d5800 2024-03-24T00:46:21.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.245+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 <== osd.0 v2:172.21.15.67:6808/2628130519 7 ==== osd_op_reply(71 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55ef3f546500 con 0x55ef3f6d5800 2024-03-24T00:46:21.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.245+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:72 13.13 13:c8000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb60800 con 0x55ef3f6d4c00 2024-03-24T00:46:21.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.245+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 <== osd.7 v2:172.21.15.72:6824/2893038746 15 ==== osd_op_reply(72 [pgnls start_epoch 93 out=98b] v92'4 uv4 ondisk = 1) v8 ==== 144+0+98 (crc 0 0 0) 0x55ef3f546000 con 0x55ef3f6d4c00 2024-03-24T00:46:21.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.246+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:73 13.b 13:d0000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb60800 con 0x55ef3f6d5000 2024-03-24T00:46:21.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.246+0000 7f7ca5e98640 1 -- 172.21.15.67:0/2570979052 <== osd.5 v2:172.21.15.72:6800/288486206 7 ==== osd_op_reply(73 [pgnls start_epoch 93 out=76b] v92'6 uv6 ondisk = 1) v8 ==== 144+0+76 (crc 0 0 0) 0x55ef3f546780 con 0x55ef3f6d5000 2024-03-24T00:46:21.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.246+0000 7f7ca5e98640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:74 13.1b 13:d8000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb60800 con 0x55ef3eb52000 2024-03-24T00:46:21.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.246+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 <== osd.4 v2:172.21.15.72:6808/477220793 13 ==== osd_op_reply(74 [pgnls start_epoch 93 out=73b] v92'10 uv10 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x55ef3f546000 con 0x55ef3eb52000 2024-03-24T00:46:21.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.246+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:75 13.7 13:e0000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb60800 con 0x55ef3f6d4c00 2024-03-24T00:46:21.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.246+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 <== osd.7 v2:172.21.15.72:6824/2893038746 16 ==== osd_op_reply(75 [pgnls start_epoch 93 out=73b] v92'3 uv3 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x55ef3f546000 con 0x55ef3f6d4c00 2024-03-24T00:46:21.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.246+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:76 13.17 13:e8000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb60800 con 0x55ef3f6d5000 2024-03-24T00:46:21.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.247+0000 7f7ca5e98640 1 -- 172.21.15.67:0/2570979052 <== osd.5 v2:172.21.15.72:6800/288486206 8 ==== osd_op_reply(76 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55ef3f546780 con 0x55ef3f6d5000 2024-03-24T00:46:21.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.247+0000 7f7ca5e98640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:77 13.f 13:f0000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb60800 con 0x55ef3f6ffc00 2024-03-24T00:46:21.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.247+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 <== osd.6 v2:172.21.15.72:6816/1725206562 11 ==== osd_op_reply(77 [pgnls start_epoch 93 out=73b] v88'1 uv1 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x55ef3f546000 con 0x55ef3f6ffc00 2024-03-24T00:46:21.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.247+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:78 13.1f 13:f8000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb60800 con 0x55ef3f6ffc00 2024-03-24T00:46:21.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.248+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 <== osd.6 v2:172.21.15.72:6816/1725206562 12 ==== osd_op_reply(78 [pgnls start_epoch 93 out=101b] v92'9 uv9 ondisk = 1) v8 ==== 144+0+101 (crc 0 0 0) 0x55ef3f546000 con 0x55ef3f6ffc00 2024-03-24T00:46:21.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.248+0000 7f7ca842a740 10 datascan.scan_links: step 2: handling object 601.00000000 2024-03-24T00:46:21.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.248+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:79 13.10 13:09401a81:::601.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb61800 con 0x55ef3f6d5c00 2024-03-24T00:46:21.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.248+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 <== osd.1 v2:172.21.15.67:6800/4062510404 14 ==== osd_op_reply(79 601.00000000 [omap-get-vals out=5b] v0'0 uv13 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55ef3f546500 con 0x55ef3f6d5c00 2024-03-24T00:46:21.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.248+0000 7f7ca842a740 10 datascan.scan_links: step 2: handling object 602.00000000 2024-03-24T00:46:21.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.248+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:80 13.10 13:0bd6d154:::602.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb53000 con 0x55ef3f6d5c00 2024-03-24T00:46:21.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.248+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 <== osd.1 v2:172.21.15.67:6800/4062510404 15 ==== osd_op_reply(80 602.00000000 [omap-get-vals out=5b] v0'0 uv14 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55ef3f546500 con 0x55ef3f6d5c00 2024-03-24T00:46:21.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.248+0000 7f7ca842a740 10 datascan.scan_links: step 2: handling object 10000000000.00000000 2024-03-24T00:46:21.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.248+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:81 13.10 13:0cf6ad0f:::10000000000.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb52c00 con 0x55ef3f6d5c00 2024-03-24T00:46:21.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.249+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 <== osd.1 v2:172.21.15.67:6800/4062510404 16 ==== osd_op_reply(81 10000000000.00000000 [omap-get-vals out=506b] v0'0 uv18 ondisk = 0) v8 ==== 164+0+506 (crc 0 0 0) 0x55ef3f546500 con 0x55ef3f6d5c00 2024-03-24T00:46:21.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.249+0000 7f7ca842a740 20 datascan.scan_links: injected ino detected 2024-03-24T00:46:21.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.249+0000 7f7ca842a740 -1 datascan.scan_links: Bad nlink on 0x10000000001 expected 2 has 1 2024-03-24T00:46:21.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.249+0000 7f7ca842a740 20 datascan.scan_links: adding 0x10000000001 for future processing to fix dnfirst 2024-03-24T00:46:21.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.249+0000 7f7ca842a740 10 datascan.scan_links: step 2: handling object 600.00000000 2024-03-24T00:46:21.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.249+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:82 13.10 13:0d82a743:::600.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb52800 con 0x55ef3f6d5c00 2024-03-24T00:46:21.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.249+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 <== osd.1 v2:172.21.15.67:6800/4062510404 17 ==== osd_op_reply(82 600.00000000 [omap-get-vals out=5b] v0'0 uv12 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55ef3f546500 con 0x55ef3f6d5c00 2024-03-24T00:46:21.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.249+0000 7f7ca842a740 10 datascan.scan_links: step 2: handling object 603.00000000 2024-03-24T00:46:21.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.249+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:83 13.10 13:0d89b25e:::603.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb52400 con 0x55ef3f6d5c00 2024-03-24T00:46:21.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.249+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 <== osd.1 v2:172.21.15.67:6800/4062510404 18 ==== osd_op_reply(83 603.00000000 [omap-get-vals out=5b] v0'0 uv15 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55ef3f546500 con 0x55ef3f6d5c00 2024-03-24T00:46:21.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.249+0000 7f7ca842a740 10 datascan.scan_links: step 2: handling object 1.00000000.inode 2024-03-24T00:46:21.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.249+0000 7f7ca842a740 10 datascan.scan_links: Not a dirfrag: '1.00000000.inode' 2024-03-24T00:46:21.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.249+0000 7f7ca842a740 10 datascan.scan_links: step 2: handling object 200.00000000 2024-03-24T00:46:21.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.249+0000 7f7ca842a740 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000000' 2024-03-24T00:46:21.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.249+0000 7f7ca842a740 10 datascan.scan_links: step 2: handling object 200.00000001 2024-03-24T00:46:21.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.249+0000 7f7ca842a740 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000001' 2024-03-24T00:46:21.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.249+0000 7f7ca842a740 10 datascan.scan_links: step 2: handling object 606.00000000 2024-03-24T00:46:21.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.249+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:84 13.14 13:2f55791f:::606.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb53000 con 0x55ef3f6d5800 2024-03-24T00:46:21.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.249+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 <== osd.0 v2:172.21.15.67:6808/2628130519 8 ==== osd_op_reply(84 606.00000000 [omap-get-vals out=5b] v0'0 uv25 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55ef3f546500 con 0x55ef3f6d5800 2024-03-24T00:46:21.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.249+0000 7f7ca842a740 10 datascan.scan_links: step 2: handling object 607.00000000 2024-03-24T00:46:21.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.249+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:85 13.1c 13:3ed09add:::607.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb52c00 con 0x55ef3f6ffc00 2024-03-24T00:46:21.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.250+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 <== osd.6 v2:172.21.15.72:6816/1725206562 13 ==== osd_op_reply(85 607.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55ef3f546000 con 0x55ef3f6ffc00 2024-03-24T00:46:21.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.250+0000 7f7ca842a740 10 datascan.scan_links: step 2: handling object 100000001f6.00000000 2024-03-24T00:46:21.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.250+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] -- osd_op(unknown.0.0:86 13.a 13:57fe7535:::100000001f6.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb60c00 con 0x55ef3f6d5400 2024-03-24T00:46:21.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.250+0000 7f7ca5e98640 1 -- 172.21.15.67:0/2570979052 <== osd.2 v2:172.21.15.67:6817/1268432561 6 ==== osd_op_reply(86 100000001f6.00000000 [omap-get-vals out=54b] v0'0 uv3 ondisk = 0) v8 ==== 164+0+54 (crc 0 0 0) 0x55ef3f546780 con 0x55ef3f6d5400 2024-03-24T00:46:21.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.250+0000 7f7ca842a740 10 datascan.scan_links: step 2: handling object mds0_openfiles.0 2024-03-24T00:46:21.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.250+0000 7f7ca842a740 10 datascan.scan_links: Not a dirfrag: 'mds0_openfiles.0' 2024-03-24T00:46:21.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.250+0000 7f7ca842a740 10 datascan.scan_links: step 2: handling object 608.00000000 2024-03-24T00:46:21.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.250+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:87 13.e 13:717a0223:::608.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb61000 con 0x55ef3f6d4c00 2024-03-24T00:46:21.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.251+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 <== osd.7 v2:172.21.15.72:6824/2893038746 17 ==== osd_op_reply(87 608.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55ef3f546000 con 0x55ef3f6d4c00 2024-03-24T00:46:21.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.251+0000 7f7ca842a740 10 datascan.scan_links: step 2: handling object 604.00000000 2024-03-24T00:46:21.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.251+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:88 13.15 13:a93a17c2:::604.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb61400 con 0x55ef3eb52000 2024-03-24T00:46:21.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.251+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 <== osd.4 v2:172.21.15.72:6808/477220793 14 ==== osd_op_reply(88 604.00000000 [omap-get-vals out=5b] v0'0 uv4 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55ef3f546000 con 0x55ef3eb52000 2024-03-24T00:46:21.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.251+0000 7f7ca842a740 10 datascan.scan_links: step 2: handling object 500.00000000 2024-03-24T00:46:21.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.251+0000 7f7ca842a740 10 datascan.scan_links: Not a dirfrag (invalid ino): '500.00000000' 2024-03-24T00:46:21.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.251+0000 7f7ca842a740 10 datascan.scan_links: step 2: handling object mds_snaptable 2024-03-24T00:46:21.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.251+0000 7f7ca842a740 10 datascan.scan_links: Not a dirfrag: 'mds_snaptable' 2024-03-24T00:46:21.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.251+0000 7f7ca842a740 10 datascan.scan_links: step 2: handling object 605.00000000 2024-03-24T00:46:21.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.251+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:89 13.1d 13:b871830b:::605.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb61800 con 0x55ef3f6d4c00 2024-03-24T00:46:21.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.251+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 <== osd.7 v2:172.21.15.72:6824/2893038746 18 ==== osd_op_reply(89 605.00000000 [omap-get-vals out=5b] v0'0 uv5 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55ef3f546000 con 0x55ef3f6d4c00 2024-03-24T00:46:21.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.251+0000 7f7ca842a740 10 datascan.scan_links: step 2: handling object mds0_inotable 2024-03-24T00:46:21.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.251+0000 7f7ca842a740 10 datascan.scan_links: Not a dirfrag: 'mds0_inotable' 2024-03-24T00:46:21.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.251+0000 7f7ca842a740 10 datascan.scan_links: step 2: handling object 100.00000000 2024-03-24T00:46:21.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.251+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:90 13.13 13:cd5a64a3:::100.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb61c00 con 0x55ef3f6d4c00 2024-03-24T00:46:21.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.252+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 <== osd.7 v2:172.21.15.72:6824/2893038746 19 ==== osd_op_reply(90 100.00000000 [omap-get-vals out=5195b] v0'0 uv4 ondisk = 0) v8 ==== 156+0+5195 (crc 0 0 0) 0x55ef3f546000 con 0x55ef3f6d4c00 2024-03-24T00:46:21.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.252+0000 7f7ca842a740 10 datascan.scan_links: step 2: handling object mds0_sessionmap 2024-03-24T00:46:21.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.252+0000 7f7ca842a740 10 datascan.scan_links: Not a dirfrag: 'mds0_sessionmap' 2024-03-24T00:46:21.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.252+0000 7f7ca842a740 10 datascan.scan_links: step 2: handling object 200.00000002 2024-03-24T00:46:21.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.252+0000 7f7ca842a740 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000002' 2024-03-24T00:46:21.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.252+0000 7f7ca842a740 10 datascan.scan_links: step 2: handling object 609.00000000 2024-03-24T00:46:21.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.252+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:91 13.7 13:e0b41d06:::609.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb60000 con 0x55ef3f6d4c00 2024-03-24T00:46:21.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.252+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 <== osd.7 v2:172.21.15.72:6824/2893038746 20 ==== osd_op_reply(91 609.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55ef3f546000 con 0x55ef3f6d4c00 2024-03-24T00:46:21.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.252+0000 7f7ca842a740 10 datascan.scan_links: step 2: handling object 400.00000000 2024-03-24T00:46:21.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.252+0000 7f7ca842a740 10 datascan.scan_links: Not a dirfrag (invalid ino): '400.00000000' 2024-03-24T00:46:21.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.252+0000 7f7ca842a740 10 datascan.scan_links: step 2: handling object 100.00000000.inode 2024-03-24T00:46:21.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.252+0000 7f7ca842a740 10 datascan.scan_links: Not a dirfrag: '100.00000000.inode' 2024-03-24T00:46:21.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.252+0000 7f7ca842a740 10 datascan.scan_links: step 2: handling object 1.00000000 2024-03-24T00:46:21.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.252+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:92 13.1f 13:ff5b34d6:::1.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb60800 con 0x55ef3f6ffc00 2024-03-24T00:46:21.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.252+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 <== osd.6 v2:172.21.15.72:6816/1725206562 14 ==== osd_op_reply(92 1.00000000 [omap-get-vals out=1163b] v0'0 uv9 ondisk = 0) v8 ==== 154+0+1163 (crc 0 0 0) 0x55ef3f546000 con 0x55ef3f6ffc00 2024-03-24T00:46:21.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.253+0000 7f7ca842a740 10 datascan.scan_links: processing 0 dup_primaries, 1 remote_links 2024-03-24T00:46:21.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.253+0000 7f7ca842a740 10 datascan.scan_links: removing dup dentries from 0 objects 2024-03-24T00:46:21.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.253+0000 7f7ca842a740 10 datascan.scan_links: processing 1 bad_nlink_inos 2024-03-24T00:46:21.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.253+0000 7f7ca842a740 10 datascan.scan_links: handling bad_nlink_ino 0x10000000001 2024-03-24T00:46:21.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.253+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:93 13.10 13:0cf6ad0f:::10000000000.00000000:head [omap-get-vals-by-keys in=17b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb61000 con 0x55ef3f6d5c00 2024-03-24T00:46:21.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.253+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 <== osd.1 v2:172.21.15.67:6800/4062510404 19 ==== osd_op_reply(93 10000000000.00000000 [omap-get-vals-by-keys out=505b] v0'0 uv18 ondisk = 0) v8 ==== 164+0+505 (crc 0 0 0) 0x55ef3f546500 con 0x55ef3f6d5c00 2024-03-24T00:46:21.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.253+0000 7f7ca842a740 20 datascan.read_dentry: oid=10000000000.00000000 dname=file frag=*, r=0 2024-03-24T00:46:21.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.253+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:94 13.10 13:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=505b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb61400 con 0x55ef3f6d5c00 2024-03-24T00:46:21.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.256+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 <== osd.1 v2:172.21.15.67:6800/4062510404 20 ==== osd_op_reply(94 10000000000.00000000 [omap-set-vals] v93'19 uv19 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55ef3f546500 con 0x55ef3f6d5c00 2024-03-24T00:46:21.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.256+0000 7f7ca842a740 20 datascan.inject_linkage: Injected dentry 0x0x10000000000/file pointing to 0x0x10000000001 2024-03-24T00:46:21.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.256+0000 7f7ca842a740 10 datascan.scan_links: processing 1 injected_inos 2024-03-24T00:46:21.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.256+0000 7f7ca842a740 10 datascan.scan_links: handling injected_ino 0x10000000001 2024-03-24T00:46:21.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.256+0000 7f7ca842a740 20 datascan.scan_links: fixing linkage (dnfirst) of 0x10000000000:file 2024-03-24T00:46:21.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.256+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:95 13.10 13:0cf6ad0f:::10000000000.00000000:head [omap-get-vals-by-keys in=17b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb53c00 con 0x55ef3f6d5c00 2024-03-24T00:46:21.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.257+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 <== osd.1 v2:172.21.15.67:6800/4062510404 21 ==== osd_op_reply(95 10000000000.00000000 [omap-get-vals-by-keys out=505b] v0'0 uv19 ondisk = 0) v8 ==== 164+0+505 (crc 0 0 0) 0x55ef3f546500 con 0x55ef3f6d5c00 2024-03-24T00:46:21.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.257+0000 7f7ca842a740 20 datascan.read_dentry: oid=10000000000.00000000 dname=file frag=*, r=0 2024-03-24T00:46:21.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.257+0000 7f7ca842a740 20 datascan.scan_links: first is now 2 2024-03-24T00:46:21.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.257+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:96 13.10 13:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=505b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb53800 con 0x55ef3f6d5c00 2024-03-24T00:46:21.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.258+0000 7f7ca6699640 1 -- 172.21.15.67:0/2570979052 <== osd.1 v2:172.21.15.67:6800/4062510404 22 ==== osd_op_reply(96 10000000000.00000000 [omap-set-vals] v93'20 uv20 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55ef3f546500 con 0x55ef3f6d5c00 2024-03-24T00:46:21.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.258+0000 7f7ca842a740 20 datascan.inject_linkage: Injected dentry 0x0x10000000000/file pointing to 0x0x10000000001 2024-03-24T00:46:21.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.258+0000 7f7ca842a740 10 datascan.scan_links: updating inotable 2024-03-24T00:46:21.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.258+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:97 13.13 13:c91d4a1d:::mds0_inotable:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb52c00 con 0x55ef3f6d4c00 2024-03-24T00:46:21.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.258+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 <== osd.7 v2:172.21.15.72:6824/2893038746 21 ==== osd_op_reply(97 mds0_inotable [read 0~50 out=50b] v0'0 uv3 ondisk = 0) v8 ==== 157+0+50 (crc 0 0 0) 0x55ef3f546000 con 0x55ef3f6d4c00 2024-03-24T00:46:21.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.258+0000 7f7ca842a740 10 mds.0.inotable: skip_inos was [0x10000000002~0x1f4,0x100000001f7~0xfffffffe09] 2024-03-24T00:46:21.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.258+0000 7f7ca842a740 10 mds.0.inotable: skip_inos now [0x100000001f7~0xfffffffe09] 2024-03-24T00:46:21.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.258+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:98 13.13 13:c91d4a1d:::mds0_inotable:head [writefull 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb52800 con 0x55ef3f6d4c00 2024-03-24T00:46:21.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.259+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 <== osd.7 v2:172.21.15.72:6824/2893038746 22 ==== osd_op_reply(98 mds0_inotable [writefull 0~34] v93'5 uv5 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x55ef3f546000 con 0x55ef3f6d4c00 2024-03-24T00:46:21.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.259+0000 7f7ca842a740 10 datascan.scan_links: updating snaptable 2024-03-24T00:46:21.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.259+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:99 13.d 13:b50e409b:::mds_snaptable:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v9 -- 0x55ef3eb52400 con 0x55ef3eb52000 2024-03-24T00:46:21.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.260+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 <== osd.4 v2:172.21.15.72:6808/477220793 15 ==== osd_op_reply(99 mds_snaptable [read 0~70 out=70b] v0'0 uv1 ondisk = 0) v8 ==== 157+0+70 (crc 0 0 0) 0x55ef3f546000 con 0x55ef3eb52000 2024-03-24T00:46:21.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.260+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x55ef3f6d5800 msgr2=0x55ef3eb3d080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:21.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.260+0000 7f7ca842a740 1 --2- 172.21.15.67:0/2570979052 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x55ef3f6d5800 0x55ef3eb3d080 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.260+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x55ef3f6d5c00 msgr2=0x55ef3eb3c000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:21.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.260+0000 7f7ca842a740 1 --2- 172.21.15.67:0/2570979052 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x55ef3f6d5c00 0x55ef3eb3c000 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.260+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 >> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] conn(0x55ef3f6d5400 msgr2=0x55ef3eb3c580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:21.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.260+0000 7f7ca842a740 1 --2- 172.21.15.67:0/2570979052 >> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] conn(0x55ef3f6d5400 0x55ef3eb3c580 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.260+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 >> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] conn(0x55ef3eb53400 msgr2=0x55ef3eb3db80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:21.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.260+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 reap_dead start 2024-03-24T00:46:21.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.260+0000 7f7ca842a740 1 --2- 172.21.15.67:0/2570979052 >> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] conn(0x55ef3eb53400 0x55ef3eb3db80 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.260+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x55ef3eb52000 msgr2=0x55ef3eb3e680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:21.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.260+0000 7f7ca842a740 1 --2- 172.21.15.67:0/2570979052 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x55ef3eb52000 0x55ef3eb3e680 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.260+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x55ef3f6d5000 msgr2=0x55ef3eb3d600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:21.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.260+0000 7f7ca842a740 1 --2- 172.21.15.67:0/2570979052 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x55ef3f6d5000 0x55ef3eb3d600 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.260+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x55ef3f6ffc00 msgr2=0x55ef3eb3cb00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:21.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.260+0000 7f7ca842a740 1 --2- 172.21.15.67:0/2570979052 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x55ef3f6ffc00 0x55ef3eb3cb00 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.260+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x55ef3f6d4c00 msgr2=0x55ef3e85a100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:21.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.260+0000 7f7ca842a740 1 --2- 172.21.15.67:0/2570979052 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x55ef3f6d4c00 0x55ef3e85a100 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.260+0000 7f7ca5697640 1 -- 172.21.15.67:0/2570979052 reap_dead start 2024-03-24T00:46:21.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.260+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x55ef3f6ff000 msgr2=0x55ef3e85b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:21.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.260+0000 7f7ca842a740 1 --2- 172.21.15.67:0/2570979052 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x55ef3f6ff000 0x55ef3e85b700 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x55ef3f54f020 tx=0x55ef3eabef00 comp rx=0 tx=0).stop 2024-03-24T00:46:21.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.260+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ef3f6d4400 msgr2=0x55ef3e85ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:21.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.260+0000 7f7ca842a740 1 --2- 172.21.15.67:0/2570979052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ef3f6d4400 0x55ef3e85ac00 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x55ef3f5c9dd0 tx=0x55ef3e864ae0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.260+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 shutdown_connections 2024-03-24T00:46:21.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.260+0000 7f7ca842a740 1 --2- 172.21.15.67:0/2570979052 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x55ef3f6ff000 0x55ef3e85b700 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.260+0000 7f7ca842a740 1 --2- 172.21.15.67:0/2570979052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ef3f6d4400 0x55ef3e85ac00 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.260+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 >> 172.21.15.67:0/2570979052 conn(0x55ef3f595c00 msgr2=0x55ef3f537000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:21.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.260+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 shutdown_connections 2024-03-24T00:46:21.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.260+0000 7f7ca842a740 1 -- 172.21.15.67:0/2570979052 wait complete. 2024-03-24T00:46:21.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.260+0000 7f7ca842a740 1 -- 172.21.15.67:0/3191124449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ef3e7aec00 msgr2=0x55ef3e858580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:21.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.260+0000 7f7ca842a740 1 --2- 172.21.15.67:0/3191124449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ef3e7aec00 0x55ef3e858580 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x55ef3f5c9170 tx=0x55ef3f54a6c0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.260+0000 7f7ca842a740 1 -- 172.21.15.67:0/3191124449 shutdown_connections 2024-03-24T00:46:21.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.260+0000 7f7ca842a740 1 --2- 172.21.15.67:0/3191124449 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ef3e7afc00 0x55ef3e859080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.260+0000 7f7ca842a740 1 --2- 172.21.15.67:0/3191124449 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ef3e7af800 0x55ef3e858b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.260+0000 7f7ca842a740 1 --2- 172.21.15.67:0/3191124449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ef3e7aec00 0x55ef3e858580 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.260+0000 7f7ca842a740 1 -- 172.21.15.67:0/3191124449 >> 172.21.15.67:0/3191124449 conn(0x55ef3e7af000 msgr2=0x55ef3f536800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:21.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.260+0000 7f7ca842a740 1 -- 172.21.15.67:0/3191124449 shutdown_connections 2024-03-24T00:46:21.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.261+0000 7f7ca842a740 1 -- 172.21.15.67:0/3191124449 wait complete. 2024-03-24T00:46:21.281 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-data-scan', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'scan_links'] in time 0:00:00.144190, result: 2024-03-24T00:46:21.281 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-03-24T00:46:21.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.421+0000 7f178d9f3640 1 Processor -- start 2024-03-24T00:46:21.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.421+0000 7f178d9f3640 1 -- start start 2024-03-24T00:46:21.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.422+0000 7f178d9f3640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1788099d00 0x7f178809a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:21.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.422+0000 7f178d9f3640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f17880b49b0 0x7f17880d0e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:21.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.422+0000 7f178d9f3640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f178809a620 con 0x7f17880d1360 2024-03-24T00:46:21.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.422+0000 7f178d9f3640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17880cc1b0 con 0x7f17880b49b0 2024-03-24T00:46:21.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.422+0000 7f178d9f3640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17880cc400 con 0x7f1788099d00 2024-03-24T00:46:21.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.422+0000 7f17867fc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f17880b49b0 0x7f17880d0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:21.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.422+0000 7f17867fc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f17880b49b0 0x7f17880d0e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55518/0 (socket says 172.21.15.67:55518) 2024-03-24T00:46:21.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.422+0000 7f17867fc640 1 -- 172.21.15.67:0/793880910 learned_addr learned my addr 172.21.15.67:0/793880910 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:21.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.422+0000 7f1786ffd640 1 --2- 172.21.15.67:0/793880910 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1788099d00 0x7f178809a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:21.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.422+0000 7f17867fc640 1 -- 172.21.15.67:0/793880910 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1788099d00 msgr2=0x7f178809a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:21.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.422+0000 7f17867fc640 1 --2- 172.21.15.67:0/793880910 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1788099d00 0x7f178809a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.422+0000 7f17867fc640 1 -- 172.21.15.67:0/793880910 >> v1:172.21.15.72:6789/0 conn(0x7f17880d1360 legacy=0x7f178815d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:21.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.422+0000 7f17867fc640 1 -- 172.21.15.67:0/793880910 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17880cc6e0 con 0x7f17880b49b0 2024-03-24T00:46:21.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.423+0000 7f17867fc640 1 --2- 172.21.15.67:0/793880910 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17880b49b0 0x7f17880d0e20 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f1770009870 tx=0x7f177002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc50057361ef82d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:21.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.423+0000 7f1785ffb640 1 -- 172.21.15.67:0/793880910 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1770034070 con 0x7f17880b49b0 2024-03-24T00:46:21.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.423+0000 7f1785ffb640 1 -- 172.21.15.67:0/793880910 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1770030840 con 0x7f17880b49b0 2024-03-24T00:46:21.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.423+0000 7f1785ffb640 1 -- 172.21.15.67:0/793880910 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1770030c10 con 0x7f17880b49b0 2024-03-24T00:46:21.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.424+0000 7f178d9f3640 1 -- 172.21.15.67:0/793880910 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17880b49b0 msgr2=0x7f17880d0e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:21.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.424+0000 7f178d9f3640 1 --2- 172.21.15.67:0/793880910 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17880b49b0 0x7f17880d0e20 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f1770009870 tx=0x7f177002b980 comp rx=0 tx=0).stop 2024-03-24T00:46:21.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.424+0000 7f178d9f3640 1 -- 172.21.15.67:0/793880910 shutdown_connections 2024-03-24T00:46:21.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.424+0000 7f178d9f3640 1 --2- 172.21.15.67:0/793880910 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17880b49b0 0x7f17880d0e20 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.424+0000 7f178d9f3640 1 --2- 172.21.15.67:0/793880910 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1788099d00 0x7f178809a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.424+0000 7f178d9f3640 1 -- 172.21.15.67:0/793880910 >> 172.21.15.67:0/793880910 conn(0x7f17880b3650 msgr2=0x7f17880b3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:21.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.424+0000 7f178d9f3640 1 -- 172.21.15.67:0/793880910 shutdown_connections 2024-03-24T00:46:21.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.424+0000 7f178d9f3640 1 -- 172.21.15.67:0/793880910 wait complete. 2024-03-24T00:46:21.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.425+0000 7f178d9f3640 1 Processor -- start 2024-03-24T00:46:21.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.425+0000 7f178d9f3640 1 -- start start 2024-03-24T00:46:21.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.426+0000 7f178d9f3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1788099d00 0x7f1788159420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:21.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.426+0000 7f178d9f3640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f17880b49b0 0x7f17881f3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:21.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.426+0000 7f178d9f3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17880d1360 0x7f1788070610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:21.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.426+0000 7f178d9f3640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17880d01c0 con 0x7f1788099d00 2024-03-24T00:46:21.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.426+0000 7f178d9f3640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17880ccca0 con 0x7f17880d1360 2024-03-24T00:46:21.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.426+0000 7f178d9f3640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17880b32a0 con 0x7f17880b49b0 2024-03-24T00:46:21.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.426+0000 7f1786ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1788099d00 0x7f1788159420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:21.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.426+0000 7f1786ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1788099d00 0x7f1788159420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55522/0 (socket says 172.21.15.67:55522) 2024-03-24T00:46:21.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.426+0000 7f1786ffd640 1 -- 172.21.15.67:0/2210685926 learned_addr learned my addr 172.21.15.67:0/2210685926 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:21.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.426+0000 7f17867fc640 1 --2- 172.21.15.67:0/2210685926 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f17880b49b0 0x7f17881f3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:21.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.426+0000 7f1786ffd640 1 -- 172.21.15.67:0/2210685926 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f17880b49b0 msgr2=0x7f17881f3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:21.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.426+0000 7f1786ffd640 1 --2- 172.21.15.67:0/2210685926 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f17880b49b0 0x7f17881f3800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.426+0000 7f1786ffd640 1 -- 172.21.15.67:0/2210685926 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17880d1360 msgr2=0x7f1788070610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:21.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.426+0000 7f17877fe640 1 --2- 172.21.15.67:0/2210685926 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17880d1360 0x7f1788070610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:21.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.426+0000 7f1786ffd640 1 --2- 172.21.15.67:0/2210685926 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17880d1360 0x7f1788070610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.426+0000 7f1786ffd640 1 -- 172.21.15.67:0/2210685926 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17881f42f0 con 0x7f1788099d00 2024-03-24T00:46:21.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.426+0000 7f1786ffd640 1 --2- 172.21.15.67:0/2210685926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1788099d00 0x7f1788159420 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f178000d800 tx=0x7f178000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:21.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.427+0000 7f177ffff640 1 -- 172.21.15.67:0/2210685926 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f178000b6c0 con 0x7f1788099d00 2024-03-24T00:46:21.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.427+0000 7f177ffff640 1 -- 172.21.15.67:0/2210685926 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1780019070 con 0x7f1788099d00 2024-03-24T00:46:21.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.427+0000 7f178d9f3640 1 -- 172.21.15.67:0/2210685926 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17881f45d0 con 0x7f1788099d00 2024-03-24T00:46:21.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.427+0000 7f177ffff640 1 -- 172.21.15.67:0/2210685926 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f178000ba80 con 0x7f1788099d00 2024-03-24T00:46:21.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.427+0000 7f178d9f3640 1 -- 172.21.15.67:0/2210685926 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17880ca7e0 con 0x7f1788099d00 2024-03-24T00:46:21.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.428+0000 7f178d9f3640 1 -- 172.21.15.67:0/2210685926 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f178804f400 con 0x7f1788099d00 2024-03-24T00:46:21.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.428+0000 7f177ffff640 1 -- 172.21.15.67:0/2210685926 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f178000bc20 con 0x7f1788099d00 2024-03-24T00:46:21.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.428+0000 7f177ffff640 1 --2- 172.21.15.67:0/2210685926 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f176806bbd0 0x7f176806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:21.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.428+0000 7f177ffff640 1 -- 172.21.15.67:0/2210685926 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f178008b210 con 0x7f1788099d00 2024-03-24T00:46:21.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.429+0000 7f17867fc640 1 --2- 172.21.15.67:0/2210685926 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f176806bbd0 0x7f176806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:21.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.429+0000 7f17867fc640 1 --2- 172.21.15.67:0/2210685926 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f176806bbd0 0x7f176806e090 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f1770036000 tx=0x7f177003b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:21.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.433+0000 7f177ffff640 1 -- 172.21.15.67:0/2210685926 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1780059a50 con 0x7f1788099d00 2024-03-24T00:46:21.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.654+0000 7f178d9f3640 1 -- 172.21.15.67:0/2210685926 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) v1 -- 0x7f1788159960 con 0x7f1788099d00 2024-03-24T00:46:21.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.844+0000 7f177ffff640 1 -- 172.21.15.67:0/2210685926 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v63) v1 ==== 162+0+0 (secure 0 0 0) 0x7f1780059350 con 0x7f1788099d00 2024-03-24T00:46:21.845 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked joinable; MDS may join as newly active. 2024-03-24T00:46:21.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.846+0000 7f178d9f3640 1 -- 172.21.15.67:0/2210685926 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f176806bbd0 msgr2=0x7f176806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:21.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.846+0000 7f178d9f3640 1 --2- 172.21.15.67:0/2210685926 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f176806bbd0 0x7f176806e090 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f1770036000 tx=0x7f177003b040 comp rx=0 tx=0).stop 2024-03-24T00:46:21.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.846+0000 7f178d9f3640 1 -- 172.21.15.67:0/2210685926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1788099d00 msgr2=0x7f1788159420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:21.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.846+0000 7f178d9f3640 1 --2- 172.21.15.67:0/2210685926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1788099d00 0x7f1788159420 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f178000d800 tx=0x7f178000dcc0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.846+0000 7f178d9f3640 1 -- 172.21.15.67:0/2210685926 shutdown_connections 2024-03-24T00:46:21.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.846+0000 7f178d9f3640 1 --2- 172.21.15.67:0/2210685926 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f176806bbd0 0x7f176806e090 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.846+0000 7f178d9f3640 1 --2- 172.21.15.67:0/2210685926 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17880d1360 0x7f1788070610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.846+0000 7f178d9f3640 1 --2- 172.21.15.67:0/2210685926 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f17880b49b0 0x7f17881f3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.846+0000 7f178d9f3640 1 --2- 172.21.15.67:0/2210685926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1788099d00 0x7f1788159420 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.846+0000 7f178d9f3640 1 -- 172.21.15.67:0/2210685926 >> 172.21.15.67:0/2210685926 conn(0x7f17880b3650 msgr2=0x7f17880d0520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:21.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.847+0000 7f178d9f3640 1 -- 172.21.15.67:0/2210685926 shutdown_connections 2024-03-24T00:46:21.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.847+0000 7f178d9f3640 1 -- 172.21.15.67:0/2210685926 wait complete. 2024-03-24T00:46:21.860 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:46:21.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.960+0000 7f2265c36640 1 Processor -- start 2024-03-24T00:46:21.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.960+0000 7f2265c36640 1 -- start start 2024-03-24T00:46:21.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.961+0000 7f2265c36640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f226015bd60 0x7f22600d01c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:21.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.961+0000 7f2265c36640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22600cdaa0 con 0x7f226009a410 2024-03-24T00:46:21.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.961+0000 7f2265c36640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22600d08c0 con 0x7f226015c140 2024-03-24T00:46:21.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.961+0000 7f2265c36640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22600c58d0 con 0x7f226015bd60 2024-03-24T00:46:21.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.961+0000 7f225f7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f226015bd60 0x7f22600d01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:21.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.961+0000 7f225ffff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f226009a410 0x7f2260165a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59490/0 (socket says 172.21.15.67:59490) 2024-03-24T00:46:21.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.961+0000 7f225f7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f226015bd60 0x7f22600d01c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55538/0 (socket says 172.21.15.67:55538) 2024-03-24T00:46:21.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.961+0000 7f225ffff640 1 -- 172.21.15.67:0/3912372605 learned_addr learned my addr 172.21.15.67:0/3912372605 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:46:21.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.961+0000 7f225f7fe640 1 -- 172.21.15.67:0/3912372605 >> v1:172.21.15.72:6789/0 conn(0x7f226015c140 legacy=0x7f2260099d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:21.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.961+0000 7f225f7fe640 1 -- 172.21.15.67:0/3912372605 >> v1:172.21.15.67:6789/0 conn(0x7f226009a410 legacy=0x7f2260165a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:21.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.961+0000 7f225f7fe640 1 -- 172.21.15.67:0/3912372605 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22600c5bb0 con 0x7f226015bd60 2024-03-24T00:46:21.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.961+0000 7f225e7fc640 1 -- 172.21.15.67:0/3912372605 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2687494452 0 0) 0x7f22600cdaa0 con 0x7f226009a410 2024-03-24T00:46:21.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.962+0000 7f225f7fe640 1 --2- 172.21.15.67:0/3912372605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f226015bd60 0x7f22600d01c0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f22500097a0 tx=0x7f225002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86b5d9347eb8fc66 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:21.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.962+0000 7f225e7fc640 1 -- 172.21.15.67:0/3912372605 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2250035020 con 0x7f226015bd60 2024-03-24T00:46:21.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.962+0000 7f225e7fc640 1 -- 172.21.15.67:0/3912372605 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2250031830 con 0x7f226015bd60 2024-03-24T00:46:21.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.962+0000 7f225e7fc640 1 -- 172.21.15.67:0/3912372605 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2250031c00 con 0x7f226015bd60 2024-03-24T00:46:21.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.963+0000 7f2265c36640 1 -- 172.21.15.67:0/3912372605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f226015bd60 msgr2=0x7f22600d01c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:21.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.963+0000 7f2265c36640 1 --2- 172.21.15.67:0/3912372605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f226015bd60 0x7f22600d01c0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f22500097a0 tx=0x7f225002b8d0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.963+0000 7f2265c36640 1 -- 172.21.15.67:0/3912372605 shutdown_connections 2024-03-24T00:46:21.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.963+0000 7f2265c36640 1 --2- 172.21.15.67:0/3912372605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f226015bd60 0x7f22600d01c0 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.963+0000 7f2265c36640 1 -- 172.21.15.67:0/3912372605 >> 172.21.15.67:0/3912372605 conn(0x7f22600b36d0 msgr2=0x7f22600b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:21.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.963+0000 7f2265c36640 1 -- 172.21.15.67:0/3912372605 shutdown_connections 2024-03-24T00:46:21.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.963+0000 7f2265c36640 1 -- 172.21.15.67:0/3912372605 wait complete. 2024-03-24T00:46:21.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.964+0000 7f2265c36640 1 Processor -- start 2024-03-24T00:46:21.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.964+0000 7f2265c36640 1 -- start start 2024-03-24T00:46:21.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.965+0000 7f2265c36640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f226009a410 0x7f22601f18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:21.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.965+0000 7f2265c36640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f226015bd60 0x7f2260159590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:21.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.965+0000 7f2265c36640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f226015c140 0x7f22600c9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:21.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.965+0000 7f2265c36640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22600c8540 con 0x7f226015c140 2024-03-24T00:46:21.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.965+0000 7f2265c36640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22600c83c0 con 0x7f226009a410 2024-03-24T00:46:21.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.965+0000 7f2265c36640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f226009ad40 con 0x7f226015bd60 2024-03-24T00:46:21.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.965+0000 7f225ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f226015c140 0x7f22600c9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:21.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.965+0000 7f225ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f226015c140 0x7f22600c9cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55554/0 (socket says 172.21.15.67:55554) 2024-03-24T00:46:21.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.965+0000 7f225ffff640 1 -- 172.21.15.67:0/1461969513 learned_addr learned my addr 172.21.15.67:0/1461969513 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:21.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.965+0000 7f225f7fe640 1 --2- 172.21.15.67:0/1461969513 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f226009a410 0x7f22601f18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:21.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.965+0000 7f225effd640 1 --2- 172.21.15.67:0/1461969513 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f226015bd60 0x7f2260159590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:21.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.965+0000 7f225ffff640 1 -- 172.21.15.67:0/1461969513 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f226015bd60 msgr2=0x7f2260159590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:21.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.965+0000 7f225ffff640 1 --2- 172.21.15.67:0/1461969513 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f226015bd60 0x7f2260159590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.965+0000 7f225ffff640 1 -- 172.21.15.67:0/1461969513 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f226009a410 msgr2=0x7f22601f18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:21.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.965+0000 7f225ffff640 1 --2- 172.21.15.67:0/1461969513 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f226009a410 0x7f22601f18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.965+0000 7f225ffff640 1 -- 172.21.15.67:0/1461969513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22601f2190 con 0x7f226015c140 2024-03-24T00:46:21.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.965+0000 7f225effd640 1 --2- 172.21.15.67:0/1461969513 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f226015bd60 0x7f2260159590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:46:21.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.965+0000 7f225ffff640 1 --2- 172.21.15.67:0/1461969513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f226015c140 0x7f22600c9cc0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f225800c7a0 tx=0x7f225800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:21.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.966+0000 7f2247fff640 1 -- 172.21.15.67:0/1461969513 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2258018070 con 0x7f226015c140 2024-03-24T00:46:21.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.966+0000 7f2247fff640 1 -- 172.21.15.67:0/1461969513 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f225800a640 con 0x7f226015c140 2024-03-24T00:46:21.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.966+0000 7f2247fff640 1 -- 172.21.15.67:0/1461969513 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f225800aab0 con 0x7f226015c140 2024-03-24T00:46:21.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.966+0000 7f2265c36640 1 -- 172.21.15.67:0/1461969513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22601f2750 con 0x7f226015c140 2024-03-24T00:46:21.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.966+0000 7f2265c36640 1 -- 172.21.15.67:0/1461969513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22600b4fb0 con 0x7f226015c140 2024-03-24T00:46:21.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.967+0000 7f2265c36640 1 -- 172.21.15.67:0/1461969513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f226004f400 con 0x7f226015c140 2024-03-24T00:46:21.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.967+0000 7f2247fff640 1 -- 172.21.15.67:0/1461969513 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f225801b070 con 0x7f226015c140 2024-03-24T00:46:21.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.967+0000 7f2247fff640 1 --2- 172.21.15.67:0/1461969513 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f224c06bbd0 0x7f224c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:21.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.967+0000 7f2247fff640 1 -- 172.21.15.67:0/1461969513 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f225808a290 con 0x7f226015c140 2024-03-24T00:46:21.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.968+0000 7f225f7fe640 1 --2- 172.21.15.67:0/1461969513 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f224c06bbd0 0x7f224c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:21.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.968+0000 7f225f7fe640 1 --2- 172.21.15.67:0/1461969513 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f224c06bbd0 0x7f224c06e090 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f2250036000 tx=0x7f225003e040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:21.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:21.972+0000 7f2247fff640 1 -- 172.21.15.67:0/1461969513 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2258058480 con 0x7f226015c140 2024-03-24T00:46:22.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:22.190+0000 7f2265c36640 1 -- 172.21.15.67:0/1461969513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f22600b5180 con 0x7f226015c140 2024-03-24T00:46:22.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:22.191+0000 7f2247fff640 1 -- 172.21.15.67:0/1461969513 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+4973 (secure 0 0 0) 0x7f225805c520 con 0x7f226015c140 2024-03-24T00:46:22.192 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:46:22.192 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":63,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6359,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2664460392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2664460392},{"type":"v1","addr":"172.21.15.67:6835","nonce":2664460392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6368,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/3323873506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3323873506},{"type":"v1","addr":"172.21.15.72:6837","nonce":3323873506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":7150,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/2744016217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2744016217},{"type":"v1","addr":"172.21.15.72:6836","nonce":2744016217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:44:54.749656+0000","modified":"2024-03-24T00:46:21.844455+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8163},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8163":{"gid":8163,"name":"c","rank":0,"incarnation":63,"state":"up:replay","state_seq":1,"addr":"172.21.15.67:6833/3311719072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3311719072},{"type":"v1","addr":"172.21.15.67:6833","nonce":3311719072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-03-24T00:46:22.192 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 63 2024-03-24T00:46:22.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:22.193+0000 7f2265c36640 1 -- 172.21.15.67:0/1461969513 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f224c06bbd0 msgr2=0x7f224c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:22.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:22.193+0000 7f2265c36640 1 --2- 172.21.15.67:0/1461969513 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f224c06bbd0 0x7f224c06e090 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f2250036000 tx=0x7f225003e040 comp rx=0 tx=0).stop 2024-03-24T00:46:22.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:22.193+0000 7f2265c36640 1 -- 172.21.15.67:0/1461969513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f226015c140 msgr2=0x7f22600c9cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:22.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:22.193+0000 7f2265c36640 1 --2- 172.21.15.67:0/1461969513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f226015c140 0x7f22600c9cc0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f225800c7a0 tx=0x7f225800cc60 comp rx=0 tx=0).stop 2024-03-24T00:46:22.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:22.193+0000 7f2265c36640 1 -- 172.21.15.67:0/1461969513 shutdown_connections 2024-03-24T00:46:22.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:22.193+0000 7f2265c36640 1 --2- 172.21.15.67:0/1461969513 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f224c06bbd0 0x7f224c06e090 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:22.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:22.193+0000 7f2265c36640 1 --2- 172.21.15.67:0/1461969513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f226015c140 0x7f22600c9cc0 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:22.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:22.193+0000 7f2265c36640 1 --2- 172.21.15.67:0/1461969513 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f226015bd60 0x7f2260159590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:22.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:22.193+0000 7f2265c36640 1 --2- 172.21.15.67:0/1461969513 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f226009a410 0x7f22601f18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:22.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:22.193+0000 7f2265c36640 1 -- 172.21.15.67:0/1461969513 >> 172.21.15.67:0/1461969513 conn(0x7f22600b36d0 msgr2=0x7f226015e560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:22.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:22.193+0000 7f2265c36640 1 -- 172.21.15.67:0/1461969513 shutdown_connections 2024-03-24T00:46:22.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:22.193+0000 7f2265c36640 1 -- 172.21.15.67:0/1461969513 wait complete. 2024-03-24T00:46:22.606 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 63, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:44:54.749656+0000', 'modified': '2024-03-24T00:46:21.844455+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 93, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8163}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8163': {'gid': 8163, 'name': 'c', 'rank': 0, 'incarnation': 63, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.67:6833/3311719072', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6832', 'nonce': 3311719072}, {'type': 'v1', 'addr': '172.21.15.67:6833', 'nonce': 3311719072}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-24T00:46:22.606 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8163:up:replay 2024-03-24T00:46:23.608 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:46:23.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.720+0000 7fc50770c640 1 Processor -- start 2024-03-24T00:46:23.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.720+0000 7fc50770c640 1 -- start start 2024-03-24T00:46:23.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.721+0000 7fc50770c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc50015dec0 0x7fc5000ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:23.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.721+0000 7fc50770c640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5000cf640 con 0x7fc5000d0ef0 2024-03-24T00:46:23.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.721+0000 7fc50770c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5000cf080 con 0x7fc50015e2a0 2024-03-24T00:46:23.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.721+0000 7fc50770c640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5000cee30 con 0x7fc50015dec0 2024-03-24T00:46:23.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.721+0000 7fc5054a8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc50015dec0 0x7fc5000ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:23.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.721+0000 7fc505ca9640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc5000d0ef0 0x7fc5000d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35002/0 (socket says 172.21.15.67:35002) 2024-03-24T00:46:23.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.721+0000 7fc505ca9640 1 -- 172.21.15.67:0/4249327213 learned_addr learned my addr 172.21.15.67:0/4249327213 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:46:23.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.721+0000 7fc5054a8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc50015dec0 0x7fc5000ce630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56176/0 (socket says 172.21.15.67:56176) 2024-03-24T00:46:23.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.722+0000 7fc5054a8640 1 -- 172.21.15.67:0/4249327213 >> v1:172.21.15.72:6789/0 conn(0x7fc50015e2a0 legacy=0x7fc5000d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:23.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.722+0000 7fc5054a8640 1 -- 172.21.15.67:0/4249327213 >> v1:172.21.15.67:6789/0 conn(0x7fc5000d0ef0 legacy=0x7fc5000d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:23.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.722+0000 7fc4f7fff640 1 -- 172.21.15.67:0/4249327213 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3106478592 0 0) 0x7fc4fc001490 con 0x7fc5000d0ef0 2024-03-24T00:46:23.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.722+0000 7fc5054a8640 1 -- 172.21.15.67:0/4249327213 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5000cf360 con 0x7fc50015dec0 2024-03-24T00:46:23.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.722+0000 7fc5054a8640 1 --2- 172.21.15.67:0/4249327213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc50015dec0 0x7fc5000ce630 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fc5000cefb0 tx=0x7fc4f802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bd8f098afd54c73 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:23.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.723+0000 7fc4f7fff640 1 -- 172.21.15.67:0/4249327213 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4f8035020 con 0x7fc50015dec0 2024-03-24T00:46:23.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.723+0000 7fc4f7fff640 1 -- 172.21.15.67:0/4249327213 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4f8031830 con 0x7fc50015dec0 2024-03-24T00:46:23.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.723+0000 7fc4f7fff640 1 -- 172.21.15.67:0/4249327213 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4f8031c00 con 0x7fc50015dec0 2024-03-24T00:46:23.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.723+0000 7fc50770c640 1 -- 172.21.15.67:0/4249327213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc50015dec0 msgr2=0x7fc5000ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:23.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.723+0000 7fc50770c640 1 --2- 172.21.15.67:0/4249327213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc50015dec0 0x7fc5000ce630 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fc5000cefb0 tx=0x7fc4f802b8d0 comp rx=0 tx=0).stop 2024-03-24T00:46:23.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.723+0000 7fc50770c640 1 -- 172.21.15.67:0/4249327213 shutdown_connections 2024-03-24T00:46:23.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.723+0000 7fc50770c640 1 --2- 172.21.15.67:0/4249327213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc50015dec0 0x7fc5000ce630 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:23.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.723+0000 7fc50770c640 1 -- 172.21.15.67:0/4249327213 >> 172.21.15.67:0/4249327213 conn(0x7fc50006c810 msgr2=0x7fc50006cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:23.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.723+0000 7fc50770c640 1 -- 172.21.15.67:0/4249327213 shutdown_connections 2024-03-24T00:46:23.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.723+0000 7fc50770c640 1 -- 172.21.15.67:0/4249327213 wait complete. 2024-03-24T00:46:23.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.724+0000 7fc50770c640 1 Processor -- start 2024-03-24T00:46:23.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.724+0000 7fc50770c640 1 -- start start 2024-03-24T00:46:23.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.725+0000 7fc50770c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5000d0ef0 0x7fc5000ad2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:23.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.725+0000 7fc50770c640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc50015dec0 0x7fc5000ad800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:23.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.725+0000 7fc50770c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc50015e2a0 0x7fc50014edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:23.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.725+0000 7fc50770c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc500162a40 con 0x7fc5000d0ef0 2024-03-24T00:46:23.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.725+0000 7fc50770c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5001628c0 con 0x7fc50015e2a0 2024-03-24T00:46:23.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.725+0000 7fc50770c640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5000d2120 con 0x7fc50015dec0 2024-03-24T00:46:23.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.725+0000 7fc5054a8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5000d0ef0 0x7fc5000ad2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:23.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.725+0000 7fc5054a8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5000d0ef0 0x7fc5000ad2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56188/0 (socket says 172.21.15.67:56188) 2024-03-24T00:46:23.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.725+0000 7fc5054a8640 1 -- 172.21.15.67:0/1720783876 learned_addr learned my addr 172.21.15.67:0/1720783876 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:23.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.725+0000 7fc505ca9640 1 --2- 172.21.15.67:0/1720783876 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc50015e2a0 0x7fc50014edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:23.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.725+0000 7fc504ca7640 1 --2- 172.21.15.67:0/1720783876 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc50015dec0 0x7fc5000ad800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:23.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.726+0000 7fc5054a8640 1 -- 172.21.15.67:0/1720783876 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc50015dec0 msgr2=0x7fc5000ad800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:23.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.726+0000 7fc5054a8640 1 --2- 172.21.15.67:0/1720783876 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc50015dec0 0x7fc5000ad800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:23.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.726+0000 7fc5054a8640 1 -- 172.21.15.67:0/1720783876 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc50015e2a0 msgr2=0x7fc50014edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:23.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.726+0000 7fc5054a8640 1 --2- 172.21.15.67:0/1720783876 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc50015e2a0 0x7fc50014edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:23.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.726+0000 7fc5054a8640 1 -- 172.21.15.67:0/1720783876 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5000b5780 con 0x7fc5000d0ef0 2024-03-24T00:46:23.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.726+0000 7fc504ca7640 1 --2- 172.21.15.67:0/1720783876 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc50015dec0 0x7fc5000ad800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:46:23.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.726+0000 7fc5054a8640 1 --2- 172.21.15.67:0/1720783876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5000d0ef0 0x7fc5000ad2c0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7fc4f8000c00 tx=0x7fc4f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:23.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.726+0000 7fc4f5ffb640 1 -- 172.21.15.67:0/1720783876 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4f8035050 con 0x7fc5000d0ef0 2024-03-24T00:46:23.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.726+0000 7fc4f5ffb640 1 -- 172.21.15.67:0/1720783876 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4f8004230 con 0x7fc5000d0ef0 2024-03-24T00:46:23.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.726+0000 7fc50770c640 1 -- 172.21.15.67:0/1720783876 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc50014ce00 con 0x7fc5000d0ef0 2024-03-24T00:46:23.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.726+0000 7fc4f5ffb640 1 -- 172.21.15.67:0/1720783876 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4f803b740 con 0x7fc5000d0ef0 2024-03-24T00:46:23.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.726+0000 7fc50770c640 1 -- 172.21.15.67:0/1720783876 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc50014d3c0 con 0x7fc5000d0ef0 2024-03-24T00:46:23.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.727+0000 7fc50770c640 1 -- 172.21.15.67:0/1720783876 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc50004f400 con 0x7fc5000d0ef0 2024-03-24T00:46:23.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.728+0000 7fc4f5ffb640 1 -- 172.21.15.67:0/1720783876 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc4f8038020 con 0x7fc5000d0ef0 2024-03-24T00:46:23.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.728+0000 7fc4f5ffb640 1 --2- 172.21.15.67:0/1720783876 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc4e406bc00 0x7fc4e406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:23.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.728+0000 7fc4f5ffb640 1 -- 172.21.15.67:0/1720783876 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6389+0+0 (secure 0 0 0) 0x7fc4f80acad0 con 0x7fc5000d0ef0 2024-03-24T00:46:23.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.728+0000 7fc504ca7640 1 --2- 172.21.15.67:0/1720783876 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc4e406bc00 0x7fc4e406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:23.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.729+0000 7fc504ca7640 1 --2- 172.21.15.67:0/1720783876 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc4e406bc00 0x7fc4e406e0c0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fc4f0007fd0 tx=0x7fc4f0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:23.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.732+0000 7fc4f5ffb640 1 -- 172.21.15.67:0/1720783876 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc4f8040070 con 0x7fc5000d0ef0 2024-03-24T00:46:23.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.956+0000 7fc50770c640 1 -- 172.21.15.67:0/1720783876 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5000d1aa0 con 0x7fc5000d0ef0 2024-03-24T00:46:23.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.957+0000 7fc4f5ffb640 1 -- 172.21.15.67:0/1720783876 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+4973 (secure 0 0 0) 0x7fc4f807b310 con 0x7fc5000d0ef0 2024-03-24T00:46:23.958 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:46:23.958 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":65,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6359,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2664460392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2664460392},{"type":"v1","addr":"172.21.15.67:6835","nonce":2664460392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6368,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/3323873506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3323873506},{"type":"v1","addr":"172.21.15.72:6837","nonce":3323873506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":7150,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/2744016217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2744016217},{"type":"v1","addr":"172.21.15.72:6836","nonce":2744016217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:44:54.749656+0000","modified":"2024-03-24T00:46:22.849195+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8163},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8163":{"gid":8163,"name":"c","rank":0,"incarnation":63,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.67:6833/3311719072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3311719072},{"type":"v1","addr":"172.21.15.67:6833","nonce":3311719072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-03-24T00:46:23.958 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 65 2024-03-24T00:46:23.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.959+0000 7fc50770c640 1 -- 172.21.15.67:0/1720783876 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc4e406bc00 msgr2=0x7fc4e406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:23.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.959+0000 7fc50770c640 1 --2- 172.21.15.67:0/1720783876 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc4e406bc00 0x7fc4e406e0c0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fc4f0007fd0 tx=0x7fc4f0009040 comp rx=0 tx=0).stop 2024-03-24T00:46:23.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.959+0000 7fc50770c640 1 -- 172.21.15.67:0/1720783876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5000d0ef0 msgr2=0x7fc5000ad2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:23.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.959+0000 7fc50770c640 1 --2- 172.21.15.67:0/1720783876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5000d0ef0 0x7fc5000ad2c0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7fc4f8000c00 tx=0x7fc4f80046c0 comp rx=0 tx=0).stop 2024-03-24T00:46:23.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.959+0000 7fc50770c640 1 -- 172.21.15.67:0/1720783876 shutdown_connections 2024-03-24T00:46:23.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.959+0000 7fc50770c640 1 --2- 172.21.15.67:0/1720783876 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc4e406bc00 0x7fc4e406e0c0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:23.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.959+0000 7fc50770c640 1 --2- 172.21.15.67:0/1720783876 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc50015e2a0 0x7fc50014edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:23.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.959+0000 7fc50770c640 1 --2- 172.21.15.67:0/1720783876 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc50015dec0 0x7fc5000ad800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:23.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.959+0000 7fc50770c640 1 --2- 172.21.15.67:0/1720783876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5000d0ef0 0x7fc5000ad2c0 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:23.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.959+0000 7fc50770c640 1 -- 172.21.15.67:0/1720783876 >> 172.21.15.67:0/1720783876 conn(0x7fc50006c810 msgr2=0x7fc5000ce4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:23.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.959+0000 7fc50770c640 1 -- 172.21.15.67:0/1720783876 shutdown_connections 2024-03-24T00:46:23.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:23.959+0000 7fc50770c640 1 -- 172.21.15.67:0/1720783876 wait complete. 2024-03-24T00:46:24.373 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 65, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:44:54.749656+0000', 'modified': '2024-03-24T00:46:22.849195+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 93, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8163}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8163': {'gid': 8163, 'name': 'c', 'rank': 0, 'incarnation': 63, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.67:6833/3311719072', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6832', 'nonce': 3311719072}, {'type': 'v1', 'addr': '172.21.15.67:6833', 'nonce': 3311719072}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-24T00:46:24.373 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8163:up:rejoin 2024-03-24T00:46:25.374 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:46:25.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.476+0000 7f85b1661640 1 Processor -- start 2024-03-24T00:46:25.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.476+0000 7f85b1661640 1 -- start start 2024-03-24T00:46:25.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.477+0000 7f85b1661640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f85ac0ce6b0 0x7f85ac06ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:25.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.477+0000 7f85b1661640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85ac0b1cb0 con 0x7f85ac0d05a0 2024-03-24T00:46:25.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.477+0000 7f85b1661640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85ac0b1f90 con 0x7f85ac1616d0 2024-03-24T00:46:25.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.477+0000 7f85b1661640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85ac0b0440 con 0x7f85ac0ce6b0 2024-03-24T00:46:25.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.477+0000 7f85a3fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f85ac0d05a0 0x7f85ac161b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35006/0 (socket says 172.21.15.67:35006) 2024-03-24T00:46:25.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.477+0000 7f85a3fff640 1 -- 172.21.15.67:0/3650524540 learned_addr learned my addr 172.21.15.67:0/3650524540 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:46:25.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.477+0000 7f85aa7fc640 1 -- 172.21.15.67:0/3650524540 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 34223782 0 0) 0x7f85ac0b1cb0 con 0x7f85ac0d05a0 2024-03-24T00:46:25.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.478+0000 7f85aa7fc640 1 -- 172.21.15.67:0/3650524540 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8598003520 con 0x7f85ac0d05a0 2024-03-24T00:46:25.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.478+0000 7f85ab7fe640 1 --2- 172.21.15.67:0/3650524540 >> v2:172.21.15.72:3300/0 conn(0x7f85ac0ce6b0 0x7f85ac06ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:25.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.478+0000 7f85aa7fc640 1 -- 172.21.15.67:0/3650524540 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1422783915 0 0) 0x7f8598003520 con 0x7f85ac0d05a0 2024-03-24T00:46:25.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.478+0000 7f85aa7fc640 1 -- 172.21.15.67:0/3650524540 >> v2:172.21.15.72:3300/0 conn(0x7f85ac0ce6b0 msgr2=0x7f85ac06ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:25.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.478+0000 7f85aa7fc640 1 --2- 172.21.15.67:0/3650524540 >> v2:172.21.15.72:3300/0 conn(0x7f85ac0ce6b0 0x7f85ac06ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:25.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.478+0000 7f85aa7fc640 1 -- 172.21.15.67:0/3650524540 >> v1:172.21.15.72:6789/0 conn(0x7f85ac1616d0 legacy=0x7f85ac0cfe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:25.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.478+0000 7f85aa7fc640 1 -- 172.21.15.67:0/3650524540 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85ac0b2270 con 0x7f85ac0d05a0 2024-03-24T00:46:25.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.478+0000 7f85aa7fc640 1 -- 172.21.15.67:0/3650524540 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f8594001670 con 0x7f85ac0d05a0 2024-03-24T00:46:25.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.478+0000 7f85aa7fc640 1 -- 172.21.15.67:0/3650524540 >> v1:172.21.15.67:6789/0 conn(0x7f85ac0d05a0 legacy=0x7f85ac161b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:25.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.478+0000 7f85aa7fc640 1 --2- 172.21.15.67:0/3650524540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85980039b0 0x7f8598003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:25.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.478+0000 7f85aa7fc640 1 --2- 172.21.15.67:0/3650524540 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8598004800 0x7f8598004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:25.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.478+0000 7f85aa7fc640 1 --2- 172.21.15.67:0/3650524540 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85980051c0 0x7f859800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:25.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.478+0000 7f85aa7fc640 1 -- 172.21.15.67:0/3650524540 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85ac0b0440 con 0x7f85980039b0 2024-03-24T00:46:25.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.478+0000 7f85aa7fc640 1 -- 172.21.15.67:0/3650524540 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f859800c160 con 0x7f85980051c0 2024-03-24T00:46:25.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.478+0000 7f85aa7fc640 1 -- 172.21.15.67:0/3650524540 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f859800c360 con 0x7f8598004800 2024-03-24T00:46:25.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.478+0000 7f85aaffd640 1 --2- 172.21.15.67:0/3650524540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85980039b0 0x7f8598003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:25.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.478+0000 7f85ab7fe640 1 --2- 172.21.15.67:0/3650524540 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85980051c0 0x7f859800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:25.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.478+0000 7f85a3fff640 1 --2- 172.21.15.67:0/3650524540 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8598004800 0x7f8598004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:25.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.478+0000 7f85aaffd640 1 -- 172.21.15.67:0/3650524540 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8598004800 msgr2=0x7f8598004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:25.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.478+0000 7f85aaffd640 1 --2- 172.21.15.67:0/3650524540 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8598004800 0x7f8598004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:25.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.478+0000 7f85aaffd640 1 -- 172.21.15.67:0/3650524540 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85980051c0 msgr2=0x7f859800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:25.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.478+0000 7f85aaffd640 1 --2- 172.21.15.67:0/3650524540 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85980051c0 0x7f859800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:25.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.478+0000 7f85aaffd640 1 -- 172.21.15.67:0/3650524540 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8598003520 con 0x7f85980039b0 2024-03-24T00:46:25.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.478+0000 7f85ab7fe640 1 --2- 172.21.15.67:0/3650524540 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85980051c0 0x7f859800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:46:25.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.478+0000 7f85a3fff640 1 --2- 172.21.15.67:0/3650524540 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8598004800 0x7f8598004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:46:25.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.478+0000 7f85ab7fe640 1 -- 172.21.15.67:0/3650524540 reap_dead start 2024-03-24T00:46:25.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.479+0000 7f85aaffd640 1 --2- 172.21.15.67:0/3650524540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85980039b0 0x7f8598003dd0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f859c001750 tx=0x7f8598005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a62190f532e3fd6b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:25.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.479+0000 7f85aa7fc640 1 -- 172.21.15.67:0/3650524540 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f859c001370 con 0x7f85980039b0 2024-03-24T00:46:25.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.479+0000 7f85aa7fc640 1 -- 172.21.15.67:0/3650524540 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f859c0052d0 con 0x7f85980039b0 2024-03-24T00:46:25.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.479+0000 7f85aa7fc640 1 -- 172.21.15.67:0/3650524540 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f859c003830 con 0x7f85980039b0 2024-03-24T00:46:25.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.480+0000 7f85b1661640 1 -- 172.21.15.67:0/3650524540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85980039b0 msgr2=0x7f8598003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:25.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.480+0000 7f85b1661640 1 --2- 172.21.15.67:0/3650524540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85980039b0 0x7f8598003dd0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f859c001750 tx=0x7f8598005830 comp rx=0 tx=0).stop 2024-03-24T00:46:25.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.480+0000 7f85b1661640 1 -- 172.21.15.67:0/3650524540 shutdown_connections 2024-03-24T00:46:25.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.480+0000 7f85b1661640 1 --2- 172.21.15.67:0/3650524540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85980039b0 0x7f8598003dd0 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:25.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.480+0000 7f85b1661640 1 -- 172.21.15.67:0/3650524540 >> 172.21.15.67:0/3650524540 conn(0x7f85ac0d0ae0 msgr2=0x7f85ac0d0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:25.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.480+0000 7f85b1661640 1 -- 172.21.15.67:0/3650524540 shutdown_connections 2024-03-24T00:46:25.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.480+0000 7f85b1661640 1 -- 172.21.15.67:0/3650524540 wait complete. 2024-03-24T00:46:25.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.481+0000 7f85b1661640 1 Processor -- start 2024-03-24T00:46:25.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.481+0000 7f85b1661640 1 -- start start 2024-03-24T00:46:25.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.482+0000 7f85b1661640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85980039b0 0x7f85ac0c7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:25.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.482+0000 7f85b1661640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85ac1f3bb0 0x7f85ac0b5e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:25.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.482+0000 7f85b1661640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f85ac1f4030 0x7f85ac0b6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:25.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.482+0000 7f85b1661640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85ac06ba50 con 0x7f85ac1f3bb0 2024-03-24T00:46:25.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.482+0000 7f85b1661640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85ac0b0970 con 0x7f85980039b0 2024-03-24T00:46:25.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.482+0000 7f85b1661640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85ac06bbd0 con 0x7f85ac1f4030 2024-03-24T00:46:25.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.482+0000 7f85a3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85ac1f3bb0 0x7f85ac0b5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:25.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.482+0000 7f85a3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85ac1f3bb0 0x7f85ac0b5e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56204/0 (socket says 172.21.15.67:56204) 2024-03-24T00:46:25.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.482+0000 7f85a3fff640 1 -- 172.21.15.67:0/296117193 learned_addr learned my addr 172.21.15.67:0/296117193 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:25.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.482+0000 7f85aaffd640 1 --2- 172.21.15.67:0/296117193 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85980039b0 0x7f85ac0c7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:25.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.482+0000 7f85a3fff640 1 -- 172.21.15.67:0/296117193 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f85ac1f4030 msgr2=0x7f85ac0b6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:25.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.482+0000 7f85ab7fe640 1 --2- 172.21.15.67:0/296117193 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f85ac1f4030 0x7f85ac0b6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:25.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.482+0000 7f85a3fff640 1 --2- 172.21.15.67:0/296117193 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f85ac1f4030 0x7f85ac0b6350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:25.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.482+0000 7f85a3fff640 1 -- 172.21.15.67:0/296117193 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85980039b0 msgr2=0x7f85ac0c7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:25.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.482+0000 7f85a3fff640 1 --2- 172.21.15.67:0/296117193 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85980039b0 0x7f85ac0c7150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:25.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.482+0000 7f85a3fff640 1 -- 172.21.15.67:0/296117193 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85ac0c5f10 con 0x7f85ac1f3bb0 2024-03-24T00:46:25.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.482+0000 7f85a3fff640 1 --2- 172.21.15.67:0/296117193 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85ac1f3bb0 0x7f85ac0b5e10 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f859400e780 tx=0x7f859400ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:25.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.483+0000 7f85a37fe640 1 -- 172.21.15.67:0/296117193 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f859400c8d0 con 0x7f85ac1f3bb0 2024-03-24T00:46:25.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.483+0000 7f85a37fe640 1 -- 172.21.15.67:0/296117193 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8594018070 con 0x7f85ac1f3bb0 2024-03-24T00:46:25.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.483+0000 7f85b1661640 1 -- 172.21.15.67:0/296117193 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85ac0c5150 con 0x7f85ac1f3bb0 2024-03-24T00:46:25.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.483+0000 7f85a37fe640 1 -- 172.21.15.67:0/296117193 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f859400cc90 con 0x7f85ac1f3bb0 2024-03-24T00:46:25.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.483+0000 7f85b1661640 1 -- 172.21.15.67:0/296117193 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85ac0c5710 con 0x7f85ac1f3bb0 2024-03-24T00:46:25.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.484+0000 7f85b1661640 1 -- 172.21.15.67:0/296117193 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85ac04f400 con 0x7f85ac1f3bb0 2024-03-24T00:46:25.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.484+0000 7f85a37fe640 1 -- 172.21.15.67:0/296117193 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f859401d460 con 0x7f85ac1f3bb0 2024-03-24T00:46:25.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.484+0000 7f85a37fe640 1 --2- 172.21.15.67:0/296117193 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f858c06bbd0 0x7f858c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:25.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.485+0000 7f85a37fe640 1 -- 172.21.15.67:0/296117193 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f859408ac10 con 0x7f85ac1f3bb0 2024-03-24T00:46:25.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.485+0000 7f85aaffd640 1 --2- 172.21.15.67:0/296117193 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f858c06bbd0 0x7f858c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:25.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.485+0000 7f85aaffd640 1 --2- 172.21.15.67:0/296117193 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f858c06bbd0 0x7f858c06e090 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f859c033000 tx=0x7f859c009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:25.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.489+0000 7f85a37fe640 1 -- 172.21.15.67:0/296117193 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f85940594e0 con 0x7f85ac1f3bb0 2024-03-24T00:46:25.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.758+0000 7f85b1661640 1 -- 172.21.15.67:0/296117193 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85ac0c4330 con 0x7f85ac1f3bb0 2024-03-24T00:46:25.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.759+0000 7f85a37fe640 1 -- 172.21.15.67:0/296117193 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f8594058de0 con 0x7f85ac1f3bb0 2024-03-24T00:46:25.759 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:46:25.759 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":66,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6359,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2664460392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2664460392},{"type":"v1","addr":"172.21.15.67:6835","nonce":2664460392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6368,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/3323873506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3323873506},{"type":"v1","addr":"172.21.15.72:6837","nonce":3323873506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":7150,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/2744016217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2744016217},{"type":"v1","addr":"172.21.15.72:6836","nonce":2744016217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:44:54.749656+0000","modified":"2024-03-24T00:46:24.852760+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8163},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8163":{"gid":8163,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6833/3311719072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3311719072},{"type":"v1","addr":"172.21.15.67:6833","nonce":3311719072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8163,"qdb_cluster":[8163]},"id":6}]} 2024-03-24T00:46:25.759 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 66 2024-03-24T00:46:25.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.760+0000 7f85b1661640 1 -- 172.21.15.67:0/296117193 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f858c06bbd0 msgr2=0x7f858c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:25.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.760+0000 7f85b1661640 1 --2- 172.21.15.67:0/296117193 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f858c06bbd0 0x7f858c06e090 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f859c033000 tx=0x7f859c009030 comp rx=0 tx=0).stop 2024-03-24T00:46:25.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.760+0000 7f85b1661640 1 -- 172.21.15.67:0/296117193 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85ac1f3bb0 msgr2=0x7f85ac0b5e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:25.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.760+0000 7f85b1661640 1 --2- 172.21.15.67:0/296117193 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85ac1f3bb0 0x7f85ac0b5e10 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f859400e780 tx=0x7f859400ec40 comp rx=0 tx=0).stop 2024-03-24T00:46:25.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.760+0000 7f85b1661640 1 -- 172.21.15.67:0/296117193 shutdown_connections 2024-03-24T00:46:25.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.760+0000 7f85b1661640 1 --2- 172.21.15.67:0/296117193 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f858c06bbd0 0x7f858c06e090 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:25.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.760+0000 7f85b1661640 1 --2- 172.21.15.67:0/296117193 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f85ac1f4030 0x7f85ac0b6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:25.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.760+0000 7f85b1661640 1 --2- 172.21.15.67:0/296117193 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85ac1f3bb0 0x7f85ac0b5e10 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:25.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.761+0000 7f85b1661640 1 --2- 172.21.15.67:0/296117193 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85980039b0 0x7f85ac0c7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:25.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.761+0000 7f85b1661640 1 -- 172.21.15.67:0/296117193 >> 172.21.15.67:0/296117193 conn(0x7f85ac0d0ae0 msgr2=0x7f85ac0b2970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:25.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.761+0000 7f85b1661640 1 -- 172.21.15.67:0/296117193 shutdown_connections 2024-03-24T00:46:25.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:25.761+0000 7f85b1661640 1 -- 172.21.15.67:0/296117193 wait complete. 2024-03-24T00:46:26.174 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 66, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:44:54.749656+0000', 'modified': '2024-03-24T00:46:24.852760+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 93, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8163}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8163': {'gid': 8163, 'name': 'c', 'rank': 0, 'incarnation': 63, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.67:6833/3311719072', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6832', 'nonce': 3311719072}, {'type': 'v1', 'addr': '172.21.15.67:6833', 'nonce': 3311719072}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8163, 'qdb_cluster': [8163]} 2024-03-24T00:46:26.174 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-24T00:46:26.174 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-24T00:46:26.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.277+0000 7fe1afb10640 1 Processor -- start 2024-03-24T00:46:26.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.277+0000 7fe1afb10640 1 -- start start 2024-03-24T00:46:26.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.278+0000 7fe1afb10640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe1a815e0f0 0x7fe1a80b95b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:26.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.278+0000 7fe1afb10640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe1a80b6dc0 0x7fe1a80b71c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:26.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.278+0000 7fe1afb10640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1a80b7700 con 0x7fe1a80b9af0 2024-03-24T00:46:26.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.278+0000 7fe1afb10640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1a80d2230 con 0x7fe1a80b6dc0 2024-03-24T00:46:26.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.278+0000 7fe1afb10640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1a80d2510 con 0x7fe1a815e0f0 2024-03-24T00:46:26.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.278+0000 7fe1ae0ad640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe1a80b6dc0 0x7fe1a80b71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:26.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.278+0000 7fe1ae0ad640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe1a80b6dc0 0x7fe1a80b71c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56212/0 (socket says 172.21.15.67:56212) 2024-03-24T00:46:26.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.278+0000 7fe1ae0ad640 1 -- 172.21.15.67:0/576740848 learned_addr learned my addr 172.21.15.67:0/576740848 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:26.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.279+0000 7fe1ad8ac640 1 --2- 172.21.15.67:0/576740848 >> v2:172.21.15.72:3300/0 conn(0x7fe1a815e0f0 0x7fe1a80b95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:26.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.279+0000 7fe1ae0ad640 1 -- 172.21.15.67:0/576740848 >> v2:172.21.15.72:3300/0 conn(0x7fe1a815e0f0 msgr2=0x7fe1a80b95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:26.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.279+0000 7fe1ae0ad640 1 --2- 172.21.15.67:0/576740848 >> v2:172.21.15.72:3300/0 conn(0x7fe1a815e0f0 0x7fe1a80b95b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:26.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.279+0000 7fe1ae0ad640 1 -- 172.21.15.67:0/576740848 >> v1:172.21.15.72:6789/0 conn(0x7fe1a80b9af0 legacy=0x7fe1a80d32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:26.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.279+0000 7fe1ae0ad640 1 -- 172.21.15.67:0/576740848 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1a80d27f0 con 0x7fe1a80b6dc0 2024-03-24T00:46:26.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.279+0000 7fe1ae0ad640 1 --2- 172.21.15.67:0/576740848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1a80b6dc0 0x7fe1a80b71c0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7fe1a808cc00 tx=0x7fe1a002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca13a0a7f4e5c855 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:26.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.280+0000 7fe1ac8aa640 1 -- 172.21.15.67:0/576740848 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a002cb20 con 0x7fe1a80b6dc0 2024-03-24T00:46:26.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.280+0000 7fe1ac8aa640 1 -- 172.21.15.67:0/576740848 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1a002ccc0 con 0x7fe1a80b6dc0 2024-03-24T00:46:26.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.280+0000 7fe1ac8aa640 1 -- 172.21.15.67:0/576740848 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a0034610 con 0x7fe1a80b6dc0 2024-03-24T00:46:26.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.280+0000 7fe1afb10640 1 -- 172.21.15.67:0/576740848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1a80b6dc0 msgr2=0x7fe1a80b71c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:26.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.280+0000 7fe1afb10640 1 --2- 172.21.15.67:0/576740848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1a80b6dc0 0x7fe1a80b71c0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7fe1a808cc00 tx=0x7fe1a002bf80 comp rx=0 tx=0).stop 2024-03-24T00:46:26.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.280+0000 7fe1afb10640 1 -- 172.21.15.67:0/576740848 shutdown_connections 2024-03-24T00:46:26.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.280+0000 7fe1afb10640 1 --2- 172.21.15.67:0/576740848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1a80b6dc0 0x7fe1a80b71c0 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:26.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.280+0000 7fe1afb10640 1 --2- 172.21.15.67:0/576740848 >> v2:172.21.15.72:3300/0 conn(0x7fe1a815e0f0 0x7fe1a80b95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:26.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.280+0000 7fe1afb10640 1 -- 172.21.15.67:0/576740848 >> 172.21.15.67:0/576740848 conn(0x7fe1a8072b60 msgr2=0x7fe1a8072f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:26.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.280+0000 7fe1afb10640 1 -- 172.21.15.67:0/576740848 shutdown_connections 2024-03-24T00:46:26.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.280+0000 7fe1afb10640 1 -- 172.21.15.67:0/576740848 wait complete. 2024-03-24T00:46:26.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.281+0000 7fe1afb10640 1 Processor -- start 2024-03-24T00:46:26.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.281+0000 7fe1afb10640 1 -- start start 2024-03-24T00:46:26.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.282+0000 7fe1afb10640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1a80b6dc0 0x7fe1a815a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:26.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.282+0000 7fe1afb10640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe1a80b9af0 0x7fe1a81575b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:26.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.282+0000 7fe1afb10640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe1a815e0f0 0x7fe1a80b1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:26.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.282+0000 7fe1afb10640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a80b31d0 con 0x7fe1a80b6dc0 2024-03-24T00:46:26.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.282+0000 7fe1afb10640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a80d2db0 con 0x7fe1a80b9af0 2024-03-24T00:46:26.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.282+0000 7fe1afb10640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a80b3350 con 0x7fe1a815e0f0 2024-03-24T00:46:26.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.282+0000 7fe1ad8ac640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1a80b6dc0 0x7fe1a815a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:26.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.282+0000 7fe1ad8ac640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1a80b6dc0 0x7fe1a815a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56220/0 (socket says 172.21.15.67:56220) 2024-03-24T00:46:26.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.282+0000 7fe1ad8ac640 1 -- 172.21.15.67:0/1498089267 learned_addr learned my addr 172.21.15.67:0/1498089267 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:26.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.282+0000 7fe1ae0ad640 1 --2- 172.21.15.67:0/1498089267 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe1a815e0f0 0x7fe1a80b1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:26.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.282+0000 7fe1ad8ac640 1 -- 172.21.15.67:0/1498089267 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe1a815e0f0 msgr2=0x7fe1a80b1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:26.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.282+0000 7fe1ad0ab640 1 --2- 172.21.15.67:0/1498089267 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe1a80b9af0 0x7fe1a81575b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:26.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.283+0000 7fe1ad8ac640 1 --2- 172.21.15.67:0/1498089267 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe1a815e0f0 0x7fe1a80b1c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:26.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.283+0000 7fe1ad8ac640 1 -- 172.21.15.67:0/1498089267 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe1a80b9af0 msgr2=0x7fe1a81575b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:26.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.283+0000 7fe1ad8ac640 1 --2- 172.21.15.67:0/1498089267 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe1a80b9af0 0x7fe1a81575b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:26.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.283+0000 7fe1ad8ac640 1 -- 172.21.15.67:0/1498089267 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1a815a9f0 con 0x7fe1a80b6dc0 2024-03-24T00:46:26.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.283+0000 7fe1ad8ac640 1 --2- 172.21.15.67:0/1498089267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1a80b6dc0 0x7fe1a815a4b0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7fe1a400d5f0 tx=0x7fe1a400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:26.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.283+0000 7fe19e7fc640 1 -- 172.21.15.67:0/1498089267 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a40125f0 con 0x7fe1a80b6dc0 2024-03-24T00:46:26.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.283+0000 7fe19e7fc640 1 -- 172.21.15.67:0/1498089267 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1a4019070 con 0x7fe1a80b6dc0 2024-03-24T00:46:26.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.283+0000 7fe19e7fc640 1 -- 172.21.15.67:0/1498089267 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a40129d0 con 0x7fe1a80b6dc0 2024-03-24T00:46:26.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.283+0000 7fe1afb10640 1 -- 172.21.15.67:0/1498089267 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1a8157af0 con 0x7fe1a80b6dc0 2024-03-24T00:46:26.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.283+0000 7fe1afb10640 1 -- 172.21.15.67:0/1498089267 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1a81f4210 con 0x7fe1a80b6dc0 2024-03-24T00:46:26.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.285+0000 7fe19e7fc640 1 -- 172.21.15.67:0/1498089267 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe1a4012b70 con 0x7fe1a80b6dc0 2024-03-24T00:46:26.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.285+0000 7fe19e7fc640 1 --2- 172.21.15.67:0/1498089267 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe18c06bbd0 0x7fe18c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:26.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.285+0000 7fe19e7fc640 1 -- 172.21.15.67:0/1498089267 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6389+0+0 (secure 0 0 0) 0x7fe1a408b0c0 con 0x7fe1a80b6dc0 2024-03-24T00:46:26.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.286+0000 7fe1ad0ab640 1 --2- 172.21.15.67:0/1498089267 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe18c06bbd0 0x7fe18c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:26.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.286+0000 7fe1b27d9740 1 Processor -- start 2024-03-24T00:46:26.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.286+0000 7fe1b27d9740 1 -- start start 2024-03-24T00:46:26.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.286+0000 7fe1ad0ab640 1 --2- 172.21.15.67:0/1498089267 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe18c06bbd0 0x7fe18c06e090 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fe1a80ce490 tx=0x7fe198009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:26.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.287+0000 7fe1b27d9740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5636bc4e5c40 0x5636bc50c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:26.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.287+0000 7fe1b27d9740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636bc50ccb0 0x5636bc4e4c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:26.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.287+0000 7fe1b27d9740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5636bc4e6020 0x5636bc4e51c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:26.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.287+0000 7fe1b27d9740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636bc449c20 con 0x5636bc50ccb0 2024-03-24T00:46:26.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.287+0000 7fe1b27d9740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636bc3f2a40 con 0x5636bc4e6020 2024-03-24T00:46:26.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.287+0000 7fe1b27d9740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5636bc445cc0 con 0x5636bc4e5c40 2024-03-24T00:46:26.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.287+0000 7fe1ad0ab640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636bc50ccb0 0x5636bc4e4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:26.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.287+0000 7fe1ad0ab640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636bc50ccb0 0x5636bc4e4c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56226/0 (socket says 172.21.15.67:56226) 2024-03-24T00:46:26.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.287+0000 7fe1ad0ab640 1 -- 172.21.15.67:0/3675023770 learned_addr learned my addr 172.21.15.67:0/3675023770 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:26.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.287+0000 7fe1ae0ad640 1 --2- 172.21.15.67:0/3675023770 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5636bc4e6020 0x5636bc4e51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:26.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.287+0000 7fe1ad8ac640 1 --2- 172.21.15.67:0/3675023770 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5636bc4e5c40 0x5636bc50c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:26.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.287+0000 7fe1ad0ab640 1 -- 172.21.15.67:0/3675023770 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5636bc4e5c40 msgr2=0x5636bc50c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:26.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.287+0000 7fe1ad0ab640 1 --2- 172.21.15.67:0/3675023770 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5636bc4e5c40 0x5636bc50c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:26.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.287+0000 7fe1ad0ab640 1 -- 172.21.15.67:0/3675023770 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5636bc4e6020 msgr2=0x5636bc4e51c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:26.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.287+0000 7fe1ad0ab640 1 --2- 172.21.15.67:0/3675023770 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5636bc4e6020 0x5636bc4e51c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:26.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.287+0000 7fe1ad0ab640 1 -- 172.21.15.67:0/3675023770 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5636bc414f90 con 0x5636bc50ccb0 2024-03-24T00:46:26.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.287+0000 7fe1ad0ab640 1 --2- 172.21.15.67:0/3675023770 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636bc50ccb0 0x5636bc4e4c80 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7fe1980147a0 tx=0x7fe198010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b017c0828e632f0a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:26.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.287+0000 7fe18b7fe640 1 -- 172.21.15.67:0/3675023770 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe198024070 con 0x5636bc50ccb0 2024-03-24T00:46:26.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.287+0000 7fe18b7fe640 1 -- 172.21.15.67:0/3675023770 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe19800dbd0 con 0x5636bc50ccb0 2024-03-24T00:46:26.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.287+0000 7fe18b7fe640 1 -- 172.21.15.67:0/3675023770 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe19801f400 con 0x5636bc50ccb0 2024-03-24T00:46:26.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.288+0000 7fe1b27d9740 1 -- 172.21.15.67:0/3675023770 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636bc50ccb0 msgr2=0x5636bc4e4c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:26.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.288+0000 7fe1b27d9740 1 --2- 172.21.15.67:0/3675023770 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636bc50ccb0 0x5636bc4e4c80 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7fe1980147a0 tx=0x7fe198010000 comp rx=0 tx=0).stop 2024-03-24T00:46:26.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.288+0000 7fe1b27d9740 1 -- 172.21.15.67:0/3675023770 shutdown_connections 2024-03-24T00:46:26.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.288+0000 7fe1b27d9740 1 --2- 172.21.15.67:0/3675023770 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5636bc4e6020 0x5636bc4e51c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:26.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.288+0000 7fe1b27d9740 1 --2- 172.21.15.67:0/3675023770 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636bc50ccb0 0x5636bc4e4c80 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:26.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.288+0000 7fe1b27d9740 1 --2- 172.21.15.67:0/3675023770 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5636bc4e5c40 0x5636bc50c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:26.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.288+0000 7fe1b27d9740 1 -- 172.21.15.67:0/3675023770 >> 172.21.15.67:0/3675023770 conn(0x5636bc4aca60 msgr2=0x5636bc52a960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:26.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.288+0000 7fe1b27d9740 1 -- 172.21.15.67:0/3675023770 shutdown_connections 2024-03-24T00:46:26.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.288+0000 7fe1b27d9740 1 -- 172.21.15.67:0/3675023770 wait complete. 2024-03-24T00:46:26.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.289+0000 7fe1b27d9740 1 -- start start 2024-03-24T00:46:26.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.289+0000 7fe1b27d9740 1 Processor -- start 2024-03-24T00:46:26.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.290+0000 7fe1b27d9740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636bc4e5c40 0x5636bc53daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:26.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.290+0000 7fe1b27d9740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5636bc4e6020 0x5636bc528cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:26.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.290+0000 7fe1b27d9740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5636bc512fe0 0x5636bc513460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:26.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.290+0000 7fe1b27d9740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636bc50c2e0 con 0x5636bc4e5c40 2024-03-24T00:46:26.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.290+0000 7fe1b27d9740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636bc405e90 con 0x5636bc4e6020 2024-03-24T00:46:26.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.290+0000 7fe1b27d9740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5636bc505060 con 0x5636bc512fe0 2024-03-24T00:46:26.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.290+0000 7fe1b27d9740 10 client.0 _ll_register_callbacks cb 0x5636bc15dac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:46:26.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.290+0000 7fe1ad8ac640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636bc4e5c40 0x5636bc53daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:26.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.290+0000 7fe1ad8ac640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636bc4e5c40 0x5636bc53daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56238/0 (socket says 172.21.15.67:56238) 2024-03-24T00:46:26.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.290+0000 7fe1ad8ac640 1 -- 172.21.15.67:0/1059991803 learned_addr learned my addr 172.21.15.67:0/1059991803 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:26.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.290+0000 7fe1ad0ab640 1 --2- 172.21.15.67:0/1059991803 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5636bc4e6020 0x5636bc528cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:26.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.290+0000 7fe1ae0ad640 1 --2- 172.21.15.67:0/1059991803 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5636bc512fe0 0x5636bc513460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:26.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.290+0000 7fe1ad8ac640 1 -- 172.21.15.67:0/1059991803 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5636bc512fe0 msgr2=0x5636bc513460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:26.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.290+0000 7fe1ad8ac640 1 --2- 172.21.15.67:0/1059991803 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5636bc512fe0 0x5636bc513460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:26.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.290+0000 7fe1ad8ac640 1 -- 172.21.15.67:0/1059991803 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5636bc4e6020 msgr2=0x5636bc528cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:26.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.290+0000 7fe1ad8ac640 1 --2- 172.21.15.67:0/1059991803 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5636bc4e6020 0x5636bc528cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:26.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.290+0000 7fe1ad8ac640 1 -- 172.21.15.67:0/1059991803 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5636bc4129d0 con 0x5636bc4e5c40 2024-03-24T00:46:26.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.290+0000 7fe1ae0ad640 1 --2- 172.21.15.67:0/1059991803 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5636bc512fe0 0x5636bc513460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:46:26.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.290+0000 7fe1ad8ac640 1 --2- 172.21.15.67:0/1059991803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636bc4e5c40 0x5636bc53daa0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7fe1a4061470 tx=0x7fe1a4059ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b5c27f1b114d015 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:26.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.290+0000 7fe188ff9640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:46:26.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.291+0000 7fe188ff9640 1 -- 172.21.15.67:0/1059991803 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a4057590 con 0x5636bc4e5c40 2024-03-24T00:46:26.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.291+0000 7fe188ff9640 1 -- 172.21.15.67:0/1059991803 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 6389+0+0 (secure 0 0 0) 0x7fe1a4015d90 con 0x5636bc4e5c40 2024-03-24T00:46:26.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.291+0000 7fe1b27d9740 1 -- 172.21.15.67:0/1059991803 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe19800dbd0 con 0x5636bc4e5c40 2024-03-24T00:46:26.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.291+0000 7fe188ff9640 1 -- 172.21.15.67:0/1059991803 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe1a4055900 con 0x5636bc4e5c40 2024-03-24T00:46:26.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.291+0000 7fe1b27d9740 10 client.8247 fetch_fsmap learned FSMap version 66 2024-03-24T00:46:26.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.291+0000 7fe1b27d9740 1 -- 172.21.15.67:0/1059991803 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=2+}) v3 -- 0x5636bc15e2d0 con 0x5636bc4e5c40 2024-03-24T00:46:26.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.291+0000 7fe188ff9640 1 -- 172.21.15.67:0/1059991803 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe19800dbd0 con 0x5636bc4e5c40 2024-03-24T00:46:26.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.291+0000 7fe188ff9640 1 -- 172.21.15.67:0/1059991803 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 66) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fe1a4013070 con 0x5636bc4e5c40 2024-03-24T00:46:26.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.291+0000 7fe1b27d9740 10 client.8247 fetch_fsmap finished waiting for FSMap version 66 2024-03-24T00:46:26.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.291+0000 7fe1b27d9740 10 client.8247 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T00:46:26.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.291+0000 7fe1b27d9740 20 client.8247 populate_metadata read hostname 'smithi067' 2024-03-24T00:46:26.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.291+0000 7fe1b27d9740 1 --2- 172.21.15.67:0/1059991803 >> [v2:172.21.15.67:6832/3311719072,v1:172.21.15.67:6833/3311719072] conn(0x5636bc529be0 0x5636bc529fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:26.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.291+0000 7fe1b27d9740 4 client.8247 mds_command: new command op to 8163 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:46:26.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.291+0000 7fe1b27d9740 1 -- 172.21.15.67:0/1059991803 --> [v2:172.21.15.67:6832/3311719072,v1:172.21.15.67:6833/3311719072] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5636bc457160 con 0x5636bc529be0 2024-03-24T00:46:26.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.291+0000 7fe1ad0ab640 1 --2- 172.21.15.67:0/1059991803 >> [v2:172.21.15.67:6832/3311719072,v1:172.21.15.67:6833/3311719072] conn(0x5636bc529be0 0x5636bc529fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:26.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.292+0000 7fe188ff9640 10 client.8247 ms_handle_connect on v2:172.21.15.67:6832/3311719072 2024-03-24T00:46:26.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.292+0000 7fe1ad0ab640 1 --2- 172.21.15.67:0/1059991803 >> [v2:172.21.15.67:6832/3311719072,v1:172.21.15.67:6833/3311719072] conn(0x5636bc529be0 0x5636bc529fe0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c36924734a254c4 server_cookie=1c098ffbff71f141 in_seq=0 out_seq=0 2024-03-24T00:46:26.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.292+0000 7fe188ff9640 1 -- 172.21.15.67:0/1059991803 <== mds.0 v2:172.21.15.67:6832/3311719072 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5636bc457160 con 0x5636bc529be0 2024-03-24T00:46:26.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.292+0000 7fe188ff9640 10 client.8247 handle_command_reply: tid=0 2024-03-24T00:46:26.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.292+0000 7fe1b27d9740 1 client.8247 shutdown 2024-03-24T00:46:26.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.292+0000 7fe1b27d9740 1 -- 172.21.15.67:0/1059991803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636bc4e5c40 msgr2=0x5636bc53daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:26.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.292+0000 7fe1b27d9740 1 --2- 172.21.15.67:0/1059991803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636bc4e5c40 0x5636bc53daa0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7fe1a4061470 tx=0x7fe1a4059ad0 comp rx=0 tx=0).stop 2024-03-24T00:46:26.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.292+0000 7fe1b27d9740 1 -- 172.21.15.67:0/1059991803 shutdown_connections 2024-03-24T00:46:26.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.292+0000 7fe1b27d9740 1 --2- 172.21.15.67:0/1059991803 >> [v2:172.21.15.67:6832/3311719072,v1:172.21.15.67:6833/3311719072] conn(0x5636bc529be0 0x5636bc529fe0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:26.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.292+0000 7fe1b27d9740 1 --2- 172.21.15.67:0/1059991803 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5636bc512fe0 0x5636bc513460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:26.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.292+0000 7fe1b27d9740 1 --2- 172.21.15.67:0/1059991803 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5636bc4e6020 0x5636bc528cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:26.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.292+0000 7fe1b27d9740 1 --2- 172.21.15.67:0/1059991803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636bc4e5c40 0x5636bc53daa0 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:26.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.292+0000 7fe188ff9640 0 client.8247 ms_handle_reset on v2:172.21.15.67:6832/3311719072 2024-03-24T00:46:26.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.292+0000 7fe1b27d9740 1 -- 172.21.15.67:0/1059991803 >> 172.21.15.67:0/1059991803 conn(0x5636bc4aca60 msgr2=0x5636bc4e36d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:26.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.292+0000 7fe1b27d9740 1 -- 172.21.15.67:0/1059991803 shutdown_connections 2024-03-24T00:46:26.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.293+0000 7fe1b27d9740 1 -- 172.21.15.67:0/1059991803 wait complete. 2024-03-24T00:46:26.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.293+0000 7fe1b27d9740 20 client.8247 trim_cache size 0 max 16384 2024-03-24T00:46:26.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.323+0000 7fe1b27d9740 1 Processor -- start 2024-03-24T00:46:26.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.323+0000 7fe1b27d9740 1 -- start start 2024-03-24T00:46:26.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.324+0000 7fe1b27d9740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636bc4e5c40 0x5636bc591c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:26.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.324+0000 7fe1b27d9740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5636bc4e6020 0x5636bc528250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:26.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.324+0000 7fe1b27d9740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5636bc512fe0 0x5636bc528790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:26.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.324+0000 7fe1b27d9740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a81ffcb0 con 0x5636bc4e5c40 2024-03-24T00:46:26.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.324+0000 7fe1b27d9740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a80cbef0 con 0x5636bc512fe0 2024-03-24T00:46:26.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.324+0000 7fe1b27d9740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a81ff690 con 0x5636bc4e6020 2024-03-24T00:46:26.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.324+0000 7fe1ad8ac640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636bc4e5c40 0x5636bc591c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:26.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.324+0000 7fe1ad8ac640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636bc4e5c40 0x5636bc591c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56252/0 (socket says 172.21.15.67:56252) 2024-03-24T00:46:26.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.324+0000 7fe1ad8ac640 1 -- 172.21.15.67:0/15118297 learned_addr learned my addr 172.21.15.67:0/15118297 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:26.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.324+0000 7fe1ae0ad640 1 --2- 172.21.15.67:0/15118297 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5636bc512fe0 0x5636bc528790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:26.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.324+0000 7fe1ad0ab640 1 --2- 172.21.15.67:0/15118297 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5636bc4e6020 0x5636bc528250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:26.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.324+0000 7fe1ad8ac640 1 -- 172.21.15.67:0/15118297 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5636bc4e6020 msgr2=0x5636bc528250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:26.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.324+0000 7fe1ad8ac640 1 --2- 172.21.15.67:0/15118297 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5636bc4e6020 0x5636bc528250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:26.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.324+0000 7fe1ad8ac640 1 -- 172.21.15.67:0/15118297 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5636bc512fe0 msgr2=0x5636bc528790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:26.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.324+0000 7fe1ad8ac640 1 --2- 172.21.15.67:0/15118297 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5636bc512fe0 0x5636bc528790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:26.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.324+0000 7fe1ad8ac640 1 -- 172.21.15.67:0/15118297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5636bc2286b0 con 0x5636bc4e5c40 2024-03-24T00:46:26.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.324+0000 7fe1ad0ab640 1 --2- 172.21.15.67:0/15118297 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5636bc4e6020 0x5636bc528250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:46:26.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.324+0000 7fe1ae0ad640 1 --2- 172.21.15.67:0/15118297 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5636bc512fe0 0x5636bc528790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:46:26.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.324+0000 7fe1ad8ac640 1 --2- 172.21.15.67:0/15118297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636bc4e5c40 0x5636bc591c60 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fe1a4059140 tx=0x7fe1a405c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1931319ef49d1074 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:26.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.324+0000 7fe18affd640 1 -- 172.21.15.67:0/15118297 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a4055aa0 con 0x5636bc4e5c40 2024-03-24T00:46:26.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.324+0000 7fe18affd640 1 -- 172.21.15.67:0/15118297 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1a4057330 con 0x5636bc4e5c40 2024-03-24T00:46:26.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.324+0000 7fe18affd640 1 -- 172.21.15.67:0/15118297 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a4078140 con 0x5636bc4e5c40 2024-03-24T00:46:26.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.325+0000 7fe1b27d9740 1 -- 172.21.15.67:0/15118297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636bc4e5c40 msgr2=0x5636bc591c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:26.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.325+0000 7fe1b27d9740 1 --2- 172.21.15.67:0/15118297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636bc4e5c40 0x5636bc591c60 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fe1a4059140 tx=0x7fe1a405c000 comp rx=0 tx=0).stop 2024-03-24T00:46:26.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.325+0000 7fe1b27d9740 1 -- 172.21.15.67:0/15118297 shutdown_connections 2024-03-24T00:46:26.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.325+0000 7fe1b27d9740 1 --2- 172.21.15.67:0/15118297 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5636bc512fe0 0x5636bc528790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:26.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.325+0000 7fe1b27d9740 1 --2- 172.21.15.67:0/15118297 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5636bc4e6020 0x5636bc528250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:26.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.325+0000 7fe1b27d9740 1 --2- 172.21.15.67:0/15118297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636bc4e5c40 0x5636bc591c60 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:26.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.325+0000 7fe1b27d9740 1 -- 172.21.15.67:0/15118297 >> 172.21.15.67:0/15118297 conn(0x5636bc4aca60 msgr2=0x5636bc503290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:26.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.325+0000 7fe1b27d9740 1 -- 172.21.15.67:0/15118297 shutdown_connections 2024-03-24T00:46:26.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.325+0000 7fe1b27d9740 1 -- 172.21.15.67:0/15118297 wait complete. 2024-03-24T00:46:26.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.326+0000 7fe1b27d9740 1 -- start start 2024-03-24T00:46:26.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.326+0000 7fe1b27d9740 1 Processor -- start 2024-03-24T00:46:26.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.327+0000 7fe1b27d9740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5636bc4e5c40 0x5636bc5950b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:26.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.327+0000 7fe1b27d9740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636bc4e6020 0x5636bc5037a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:26.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.327+0000 7fe1b27d9740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5636bc512fe0 0x5636bc5202f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:26.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.327+0000 7fe1b27d9740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636bc529230 con 0x5636bc4e6020 2024-03-24T00:46:26.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.327+0000 7fe1b27d9740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636bc5111a0 con 0x5636bc512fe0 2024-03-24T00:46:26.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.327+0000 7fe1b27d9740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5636bc511da0 con 0x5636bc4e5c40 2024-03-24T00:46:26.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.327+0000 7fe1b27d9740 10 client.0 _ll_register_callbacks cb 0x7fe1a4006d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:46:26.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.327+0000 7fe1ad0ab640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636bc4e6020 0x5636bc5037a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:26.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.327+0000 7fe1ad0ab640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636bc4e6020 0x5636bc5037a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56262/0 (socket says 172.21.15.67:56262) 2024-03-24T00:46:26.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.327+0000 7fe1ad0ab640 1 -- 172.21.15.67:0/3978978006 learned_addr learned my addr 172.21.15.67:0/3978978006 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:26.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.327+0000 7fe1ae0ad640 1 --2- 172.21.15.67:0/3978978006 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5636bc512fe0 0x5636bc5202f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:26.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.327+0000 7fe1ad8ac640 1 --2- 172.21.15.67:0/3978978006 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5636bc4e5c40 0x5636bc5950b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:26.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.327+0000 7fe1ad0ab640 1 -- 172.21.15.67:0/3978978006 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5636bc4e5c40 msgr2=0x5636bc5950b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:26.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.327+0000 7fe1ad0ab640 1 --2- 172.21.15.67:0/3978978006 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5636bc4e5c40 0x5636bc5950b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:26.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.327+0000 7fe1ad0ab640 1 -- 172.21.15.67:0/3978978006 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5636bc512fe0 msgr2=0x5636bc5202f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:26.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.327+0000 7fe1ad0ab640 1 --2- 172.21.15.67:0/3978978006 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5636bc512fe0 0x5636bc5202f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:26.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.327+0000 7fe1ad0ab640 1 -- 172.21.15.67:0/3978978006 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5636bc594640 con 0x5636bc4e6020 2024-03-24T00:46:26.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.327+0000 7fe1ae0ad640 1 --2- 172.21.15.67:0/3978978006 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5636bc512fe0 0x5636bc5202f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:46:26.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.327+0000 7fe1ad8ac640 1 --2- 172.21.15.67:0/3978978006 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5636bc4e5c40 0x5636bc5950b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:46:26.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.327+0000 7fe1ad0ab640 1 --2- 172.21.15.67:0/3978978006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636bc4e6020 0x5636bc5037a0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7fe198017040 tx=0x7fe198000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df7f86c45cd6f80b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:26.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.327+0000 7fe188ff9640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:46:26.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.327+0000 7fe188ff9640 1 -- 172.21.15.67:0/3978978006 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe198033060 con 0x5636bc4e6020 2024-03-24T00:46:26.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.327+0000 7fe188ff9640 1 -- 172.21.15.67:0/3978978006 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 6389+0+0 (secure 0 0 0) 0x7fe19802e160 con 0x5636bc4e6020 2024-03-24T00:46:26.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.327+0000 7fe1b27d9740 1 -- 172.21.15.67:0/3978978006 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe1a4057330 con 0x5636bc4e6020 2024-03-24T00:46:26.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.328+0000 7fe188ff9640 1 -- 172.21.15.67:0/3978978006 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe198039050 con 0x5636bc4e6020 2024-03-24T00:46:26.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.328+0000 7fe1b27d9740 10 client.8253 fetch_fsmap learned FSMap version 66 2024-03-24T00:46:26.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.328+0000 7fe1b27d9740 1 -- 172.21.15.67:0/3978978006 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=2+}) v3 -- 0x5636bc5914f0 con 0x5636bc4e6020 2024-03-24T00:46:26.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.328+0000 7fe188ff9640 1 -- 172.21.15.67:0/3978978006 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1a4057330 con 0x5636bc4e6020 2024-03-24T00:46:26.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.328+0000 7fe188ff9640 1 -- 172.21.15.67:0/3978978006 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 66) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fe19803c020 con 0x5636bc4e6020 2024-03-24T00:46:26.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.328+0000 7fe1b27d9740 10 client.8253 fetch_fsmap finished waiting for FSMap version 66 2024-03-24T00:46:26.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.328+0000 7fe1b27d9740 10 client.8253 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T00:46:26.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.328+0000 7fe1b27d9740 20 client.8253 populate_metadata read hostname 'smithi067' 2024-03-24T00:46:26.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.328+0000 7fe1b27d9740 1 --2- 172.21.15.67:0/3978978006 >> [v2:172.21.15.67:6832/3311719072,v1:172.21.15.67:6833/3311719072] conn(0x5636bc4ded40 0x5636bc52d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:26.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.328+0000 7fe1b27d9740 4 client.8253 mds_command: new command op to 8163 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:46:26.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.328+0000 7fe1b27d9740 1 -- 172.21.15.67:0/3978978006 --> [v2:172.21.15.67:6832/3311719072,v1:172.21.15.67:6833/3311719072] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5636bc4514b0 con 0x5636bc4ded40 2024-03-24T00:46:26.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.328+0000 7fe1ad8ac640 1 --2- 172.21.15.67:0/3978978006 >> [v2:172.21.15.67:6832/3311719072,v1:172.21.15.67:6833/3311719072] conn(0x5636bc4ded40 0x5636bc52d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:26.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.328+0000 7fe188ff9640 10 client.8253 ms_handle_connect on v2:172.21.15.67:6832/3311719072 2024-03-24T00:46:26.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.328+0000 7fe1ad8ac640 1 --2- 172.21.15.67:0/3978978006 >> [v2:172.21.15.67:6832/3311719072,v1:172.21.15.67:6833/3311719072] conn(0x5636bc4ded40 0x5636bc52d2a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4417c5c2d912fe2a server_cookie=830d160e1413dc04 in_seq=0 out_seq=0 2024-03-24T00:46:26.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.329+0000 7fe188ff9640 1 -- 172.21.15.67:0/3978978006 <== mds.0 v2:172.21.15.67:6832/3311719072 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x5636bc4514b0 con 0x5636bc4ded40 2024-03-24T00:46:26.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.329+0000 7fe188ff9640 10 client.8253 handle_command_reply: tid=0 2024-03-24T00:46:26.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.329+0000 7fe1b27d9740 1 client.8253 shutdown 2024-03-24T00:46:26.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.329+0000 7fe1b27d9740 1 -- 172.21.15.67:0/3978978006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636bc4e6020 msgr2=0x5636bc5037a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:26.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.329+0000 7fe1b27d9740 1 --2- 172.21.15.67:0/3978978006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636bc4e6020 0x5636bc5037a0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7fe198017040 tx=0x7fe198000bc0 comp rx=0 tx=0).stop 2024-03-24T00:46:26.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.329+0000 7fe1b27d9740 1 -- 172.21.15.67:0/3978978006 shutdown_connections 2024-03-24T00:46:26.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.329+0000 7fe1b27d9740 1 --2- 172.21.15.67:0/3978978006 >> [v2:172.21.15.67:6832/3311719072,v1:172.21.15.67:6833/3311719072] conn(0x5636bc4ded40 0x5636bc52d2a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:26.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.329+0000 7fe1b27d9740 1 --2- 172.21.15.67:0/3978978006 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5636bc512fe0 0x5636bc5202f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:26.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.329+0000 7fe1b27d9740 1 --2- 172.21.15.67:0/3978978006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636bc4e6020 0x5636bc5037a0 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:26.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.329+0000 7fe188ff9640 0 client.8253 ms_handle_reset on v2:172.21.15.67:6832/3311719072 2024-03-24T00:46:26.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.329+0000 7fe1b27d9740 1 --2- 172.21.15.67:0/3978978006 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5636bc4e5c40 0x5636bc5950b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:26.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.329+0000 7fe1b27d9740 1 -- 172.21.15.67:0/3978978006 >> 172.21.15.67:0/3978978006 conn(0x5636bc4aca60 msgr2=0x5636bc529fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:26.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.329+0000 7fe1b27d9740 1 -- 172.21.15.67:0/3978978006 shutdown_connections 2024-03-24T00:46:26.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.329+0000 7fe1b27d9740 1 -- 172.21.15.67:0/3978978006 wait complete. 2024-03-24T00:46:26.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.329+0000 7fe1b27d9740 20 client.8253 trim_cache size 0 max 16384 2024-03-24T00:46:26.340 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:46:26.341 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "de1d0178-808e-49ca-a986-e297e523b976", 2024-03-24T00:46:26.341 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-03-24T00:46:26.341 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 8163, 2024-03-24T00:46:26.341 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-03-24T00:46:26.341 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-03-24T00:46:26.341 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-03-24T00:46:26.341 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 4.4828028450000001, 2024-03-24T00:46:26.341 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 66, 2024-03-24T00:46:26.341 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 93, 2024-03-24T00:46:26.341 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 93, 2024-03-24T00:46:26.341 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 5.4109875340000002 2024-03-24T00:46:26.341 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:46:26.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.330+0000 7fe1afb10640 1 -- 172.21.15.67:0/1498089267 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe18c06bbd0 msgr2=0x7fe18c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:26.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.330+0000 7fe1afb10640 1 --2- 172.21.15.67:0/1498089267 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe18c06bbd0 0x7fe18c06e090 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fe1a80ce490 tx=0x7fe198009040 comp rx=0 tx=0).stop 2024-03-24T00:46:26.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.330+0000 7fe1afb10640 1 -- 172.21.15.67:0/1498089267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1a80b6dc0 msgr2=0x7fe1a815a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:26.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.330+0000 7fe1afb10640 1 --2- 172.21.15.67:0/1498089267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1a80b6dc0 0x7fe1a815a4b0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7fe1a400d5f0 tx=0x7fe1a400dab0 comp rx=0 tx=0).stop 2024-03-24T00:46:26.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.330+0000 7fe1afb10640 1 -- 172.21.15.67:0/1498089267 shutdown_connections 2024-03-24T00:46:26.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.330+0000 7fe1afb10640 1 --2- 172.21.15.67:0/1498089267 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe18c06bbd0 0x7fe18c06e090 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:26.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.330+0000 7fe1afb10640 1 --2- 172.21.15.67:0/1498089267 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe1a815e0f0 0x7fe1a80b1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:26.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.330+0000 7fe1afb10640 1 --2- 172.21.15.67:0/1498089267 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe1a80b9af0 0x7fe1a81575b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:26.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.330+0000 7fe1afb10640 1 --2- 172.21.15.67:0/1498089267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1a80b6dc0 0x7fe1a815a4b0 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:26.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.330+0000 7fe1afb10640 1 -- 172.21.15.67:0/1498089267 >> 172.21.15.67:0/1498089267 conn(0x7fe1a8072b60 msgr2=0x7fe1a80d1920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:26.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.330+0000 7fe1afb10640 1 -- 172.21.15.67:0/1498089267 shutdown_connections 2024-03-24T00:46:26.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.330+0000 7fe1afb10640 1 -- 172.21.15.67:0/1498089267 wait complete. 2024-03-24T00:46:26.344 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T00:46:26.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.445+0000 7f844480b640 1 Processor -- start 2024-03-24T00:46:26.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.445+0000 7f844480b640 1 -- start start 2024-03-24T00:46:26.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.446+0000 7f844480b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f844015dec0 0x7f84400ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:26.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.446+0000 7f844480b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84400cf640 con 0x7f844015e2a0 2024-03-24T00:46:26.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.446+0000 7f844480b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84400cf080 con 0x7f84400d0ef0 2024-03-24T00:46:26.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.446+0000 7f844480b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84400cee30 con 0x7f844015dec0 2024-03-24T00:46:26.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.446+0000 7f843e59c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f844015dec0 0x7f84400ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:26.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.446+0000 7f843dd9b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f844015e2a0 0x7f84400d07e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35012/0 (socket says 172.21.15.67:35012) 2024-03-24T00:46:26.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.446+0000 7f843e59c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f844015dec0 0x7f84400ce630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56276/0 (socket says 172.21.15.67:56276) 2024-03-24T00:46:26.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.446+0000 7f843dd9b640 1 -- 172.21.15.67:0/2826789058 learned_addr learned my addr 172.21.15.67:0/2826789058 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:46:26.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.447+0000 7f843e59c640 1 -- 172.21.15.67:0/2826789058 >> v1:172.21.15.72:6789/0 conn(0x7f84400d0ef0 legacy=0x7f84400d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:26.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.447+0000 7f843e59c640 1 -- 172.21.15.67:0/2826789058 >> v1:172.21.15.67:6789/0 conn(0x7f844015e2a0 legacy=0x7f84400d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:26.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.447+0000 7f843e59c640 1 -- 172.21.15.67:0/2826789058 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84400cf360 con 0x7f844015dec0 2024-03-24T00:46:26.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.447+0000 7f843d59a640 1 -- 172.21.15.67:0/2826789058 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2384007656 0 0) 0x7f8430001110 con 0x7f844015e2a0 2024-03-24T00:46:26.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.447+0000 7f843e59c640 1 --2- 172.21.15.67:0/2826789058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f844015dec0 0x7f84400ce630 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f84400cefb0 tx=0x7f842802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b77b5141243ccad5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:26.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.448+0000 7f843d59a640 1 -- 172.21.15.67:0/2826789058 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84280315a0 con 0x7f844015dec0 2024-03-24T00:46:26.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.448+0000 7f843d59a640 1 -- 172.21.15.67:0/2826789058 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8428037070 con 0x7f844015dec0 2024-03-24T00:46:26.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.448+0000 7f843d59a640 1 -- 172.21.15.67:0/2826789058 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8428031960 con 0x7f844015dec0 2024-03-24T00:46:26.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.448+0000 7f844480b640 1 -- 172.21.15.67:0/2826789058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f844015dec0 msgr2=0x7f84400ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:26.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.448+0000 7f844480b640 1 --2- 172.21.15.67:0/2826789058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f844015dec0 0x7f84400ce630 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f84400cefb0 tx=0x7f842802b840 comp rx=0 tx=0).stop 2024-03-24T00:46:26.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.448+0000 7f844480b640 1 -- 172.21.15.67:0/2826789058 shutdown_connections 2024-03-24T00:46:26.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.448+0000 7f844480b640 1 --2- 172.21.15.67:0/2826789058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f844015dec0 0x7f84400ce630 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:26.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.448+0000 7f844480b640 1 -- 172.21.15.67:0/2826789058 >> 172.21.15.67:0/2826789058 conn(0x7f844006c810 msgr2=0x7f844006cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:26.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.448+0000 7f844480b640 1 -- 172.21.15.67:0/2826789058 shutdown_connections 2024-03-24T00:46:26.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.448+0000 7f844480b640 1 -- 172.21.15.67:0/2826789058 wait complete. 2024-03-24T00:46:26.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.450+0000 7f844480b640 1 Processor -- start 2024-03-24T00:46:26.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.450+0000 7f844480b640 1 -- start start 2024-03-24T00:46:26.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.451+0000 7f844480b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f84400d0ef0 0x7f8440158f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:26.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.451+0000 7f844480b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f844015dec0 0x7f84400b5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:26.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.451+0000 7f844480b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f844015e2a0 0x7f844015abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:26.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.451+0000 7f844480b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8440162a40 con 0x7f844015dec0 2024-03-24T00:46:26.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.451+0000 7f844480b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84401628c0 con 0x7f844015e2a0 2024-03-24T00:46:26.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.451+0000 7f844480b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84400d2120 con 0x7f84400d0ef0 2024-03-24T00:46:26.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.451+0000 7f843dd9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f844015dec0 0x7f84400b5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:26.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.452+0000 7f843dd9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f844015dec0 0x7f84400b5660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56288/0 (socket says 172.21.15.67:56288) 2024-03-24T00:46:26.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.452+0000 7f843dd9b640 1 -- 172.21.15.67:0/1317276434 learned_addr learned my addr 172.21.15.67:0/1317276434 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:26.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.452+0000 7f843e59c640 1 --2- 172.21.15.67:0/1317276434 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f84400d0ef0 0x7f8440158f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:26.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.452+0000 7f843ed9d640 1 --2- 172.21.15.67:0/1317276434 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f844015e2a0 0x7f844015abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:26.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.452+0000 7f843dd9b640 1 -- 172.21.15.67:0/1317276434 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f84400d0ef0 msgr2=0x7f8440158f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:26.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.452+0000 7f843dd9b640 1 --2- 172.21.15.67:0/1317276434 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f84400d0ef0 0x7f8440158f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:26.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.452+0000 7f843dd9b640 1 -- 172.21.15.67:0/1317276434 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f844015e2a0 msgr2=0x7f844015abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:26.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.452+0000 7f843dd9b640 1 --2- 172.21.15.67:0/1317276434 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f844015e2a0 0x7f844015abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:26.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.452+0000 7f843dd9b640 1 -- 172.21.15.67:0/1317276434 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84401e89e0 con 0x7f844015dec0 2024-03-24T00:46:26.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.452+0000 7f843e59c640 1 --2- 172.21.15.67:0/1317276434 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f84400d0ef0 0x7f8440158f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:46:26.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.452+0000 7f843dd9b640 1 --2- 172.21.15.67:0/1317276434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f844015dec0 0x7f84400b5660 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f843000c410 tx=0x7f843000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:26.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.452+0000 7f8436ffd640 1 -- 172.21.15.67:0/1317276434 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84300074e0 con 0x7f844015dec0 2024-03-24T00:46:26.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.452+0000 7f8436ffd640 1 -- 172.21.15.67:0/1317276434 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84300047e0 con 0x7f844015dec0 2024-03-24T00:46:26.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.452+0000 7f844480b640 1 -- 172.21.15.67:0/1317276434 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84401e7e60 con 0x7f844015dec0 2024-03-24T00:46:26.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.452+0000 7f8436ffd640 1 -- 172.21.15.67:0/1317276434 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8430004bd0 con 0x7f844015dec0 2024-03-24T00:46:26.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.452+0000 7f844480b640 1 -- 172.21.15.67:0/1317276434 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84401e8420 con 0x7f844015dec0 2024-03-24T00:46:26.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.453+0000 7f844480b640 1 -- 172.21.15.67:0/1317276434 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f844004f400 con 0x7f844015dec0 2024-03-24T00:46:26.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.454+0000 7f8436ffd640 1 -- 172.21.15.67:0/1317276434 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8430011070 con 0x7f844015dec0 2024-03-24T00:46:26.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.454+0000 7f8436ffd640 1 --2- 172.21.15.67:0/1317276434 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f842006bbd0 0x7f842006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:26.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.454+0000 7f8436ffd640 1 -- 172.21.15.67:0/1317276434 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f843008a9e0 con 0x7f844015dec0 2024-03-24T00:46:26.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.454+0000 7f843e59c640 1 --2- 172.21.15.67:0/1317276434 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f842006bbd0 0x7f842006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:26.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.455+0000 7f843e59c640 1 --2- 172.21.15.67:0/1317276434 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f842006bbd0 0x7f842006e090 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f8428036000 tx=0x7f8428008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:26.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.458+0000 7f8436ffd640 1 -- 172.21.15.67:0/1317276434 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f84400cf640 con 0x7f844015dec0 2024-03-24T00:46:26.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.730+0000 7f844480b640 1 -- 172.21.15.67:0/1317276434 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f84400d1aa0 con 0x7f844015dec0 2024-03-24T00:46:26.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.730+0000 7f8436ffd640 1 -- 172.21.15.67:0/1317276434 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v66) v1 ==== 53+0+73 (secure 0 0 0) 0x7f843000e280 con 0x7f844015dec0 2024-03-24T00:46:26.731 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-24T00:46:26.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.732+0000 7f844480b640 1 -- 172.21.15.67:0/1317276434 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f842006bbd0 msgr2=0x7f842006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:26.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.732+0000 7f844480b640 1 --2- 172.21.15.67:0/1317276434 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f842006bbd0 0x7f842006e090 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f8428036000 tx=0x7f8428008030 comp rx=0 tx=0).stop 2024-03-24T00:46:26.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.732+0000 7f844480b640 1 -- 172.21.15.67:0/1317276434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f844015dec0 msgr2=0x7f84400b5660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:26.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.732+0000 7f844480b640 1 --2- 172.21.15.67:0/1317276434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f844015dec0 0x7f84400b5660 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f843000c410 tx=0x7f843000c8d0 comp rx=0 tx=0).stop 2024-03-24T00:46:26.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.732+0000 7f844480b640 1 -- 172.21.15.67:0/1317276434 shutdown_connections 2024-03-24T00:46:26.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.732+0000 7f844480b640 1 --2- 172.21.15.67:0/1317276434 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f842006bbd0 0x7f842006e090 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:26.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.732+0000 7f844480b640 1 --2- 172.21.15.67:0/1317276434 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f844015e2a0 0x7f844015abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:26.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.732+0000 7f844480b640 1 --2- 172.21.15.67:0/1317276434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f844015dec0 0x7f84400b5660 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:26.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.732+0000 7f844480b640 1 --2- 172.21.15.67:0/1317276434 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f84400d0ef0 0x7f8440158f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:26.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.732+0000 7f844480b640 1 -- 172.21.15.67:0/1317276434 >> 172.21.15.67:0/1317276434 conn(0x7f844006c810 msgr2=0x7f8440160d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:26.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.732+0000 7f844480b640 1 -- 172.21.15.67:0/1317276434 shutdown_connections 2024-03-24T00:46:26.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:26.732+0000 7f844480b640 1 -- 172.21.15.67:0/1317276434 wait complete. 2024-03-24T00:46:26.743 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-24T00:46:26.743 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T00:46:26.743 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-03-24T00:46:26.743 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-24T00:46:26.743 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T00:46:26.743 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-24T00:46:26.743 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-24T00:46:26.743 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T00:46:26.743 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:46:26.743 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:46:26.744 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-03-24T00:46:26.758 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T00:46:26.758 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T00:46:26.758 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T00:46:26.758 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:46:26.758 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-03-24T00:46:26.759 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:46:26.759 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:46:26.759 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-03-24T00:46:26.759 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-03-24T00:46:26.759 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:46:26.759 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-03-24T00:46:26.759 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-03-24T00:46:26.759 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:46:26.759 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-03-24T00:46:26.759 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T00:46:26.759 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-03-24T00:46:26.759 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-24T00:46:26.759 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 42126sec preferred_lft 42126sec 2024-03-24T00:46:26.759 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-03-24T00:46:26.759 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:46:26.760 INFO:teuthology.orchestra.run.smithi067.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T00:46:26.760 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 96:cf:26:c8:bf:62 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:46:26.760 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-24T00:46:26.760 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:46:26.760 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::7c6a:5fff:feaa:d41/64 scope link 2024-03-24T00:46:26.760 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:46:26.760 INFO:teuthology.orchestra.run.smithi067.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T00:46:26.760 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ea:22:4b:d5:48:3d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:46:26.760 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::e822:4bff:fed5:483d/64 scope link 2024-03-24T00:46:26.760 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:46:26.760 INFO:teuthology.orchestra.run.smithi067.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T00:46:26.760 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 96:cf:26:c8:bf:62 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:46:26.760 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::94cf:26ff:fec8:bf62/64 scope link 2024-03-24T00:46:26.760 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:46:26.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:46:26.761 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T00:46:26.826 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-03-24T00:46:26.828 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:46:26.828 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-03-24T00:46:26.851 INFO:tasks.ceph.mds.c.smithi067.stderr:2024-03-24T00:46:26.851+0000 7f97dcee7640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T00:46:26.865 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-03-24T00:46:26.865 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T00:46:26.865 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-24T00:46:26.866 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-24T00:46:26.866 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T00:46:26.866 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:46:26.866 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-03-24T00:46:26.866 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-03-24T00:46:26.866 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-24T00:46:26.866 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-24T00:46:26.866 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-24T00:46:26.866 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-03-24T00:46:26.960 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:46:26.960 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-03-24T00:46:26.974 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-24T00:46:26.974 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-24T00:46:26.974 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:46:26.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:46:26.975 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-03-24T00:46:27.040 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:46:27.041 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-24T00:46:27.095 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:46:27.095 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:46:27.095 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:46:27.152 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-24T00:46:27.152 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:46:27.152 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:46:27.236 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:46:27.236 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T00:46:27.236 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-03-24T00:46:27.244 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:46:27.244 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T00:46:27.244 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-03-24T00:46:27.244 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.72:3300,172.21.15.72:3301" 2024-03-24T00:46:27.244 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-24T00:46:27.249 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:46:27.249 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T00:46:27.249 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-03-24T00:46:27.249 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.72:3300,172.21.15.72:3301" 2024-03-24T00:46:27.249 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-24T00:46:27.249 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:6789,172.21.15.72:6789,172.21.15.72:6790" 2024-03-24T00:46:27.249 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.67:6789,172.21.15.72:6789,172.21.15.72:6790:/ 2024-03-24T00:46:27.250 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=de1d0178-808e-49ca-a986-e297e523b976" will pass to kernel 2024-03-24T00:46:27.253 INFO:teuthology.orchestra.run.smithi067.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-24T00:46:27.253 INFO:teuthology.orchestra.run.smithi067.stdout: You just mounted a file system that supports labels which does not 2024-03-24T00:46:27.253 INFO:teuthology.orchestra.run.smithi067.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-24T00:46:27.253 INFO:teuthology.orchestra.run.smithi067.stdout: applications will generate AVC messages and not be allowed access to 2024-03-24T00:46:27.253 INFO:teuthology.orchestra.run.smithi067.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-24T00:46:27.253 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-24T00:46:27.253 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:46:27.253 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:46:27.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:46:27.323 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T00:46:27.379 INFO:teuthology.orchestra.run.smithi067.stdout:client8265 2024-03-24T00:46:27.379 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:46:27.379 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:46:27.435 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:46:27.435 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.6:0 scrub start /dir recursive,repair 2024-03-24T00:46:27.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.580+0000 7fe810937640 1 Processor -- start 2024-03-24T00:46:27.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.580+0000 7fe810937640 1 -- start start 2024-03-24T00:46:27.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.582+0000 7fe810937640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe80c15f630 0x7fe80c099d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:27.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.582+0000 7fe810937640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe80c0b1070 0x7fe80c09a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:27.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.582+0000 7fe810937640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe80c0cdb50 con 0x7fe80c0b0ab0 2024-03-24T00:46:27.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.582+0000 7fe810937640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe80c09b090 con 0x7fe80c0b1070 2024-03-24T00:46:27.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.582+0000 7fe810937640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe80c09ad20 con 0x7fe80c15f630 2024-03-24T00:46:27.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.582+0000 7fe80ad9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe80c0b1070 0x7fe80c09a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:27.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.582+0000 7fe80ad9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe80c0b1070 0x7fe80c09a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56300/0 (socket says 172.21.15.67:56300) 2024-03-24T00:46:27.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.582+0000 7fe80ad9d640 1 -- 172.21.15.67:0/3516935407 learned_addr learned my addr 172.21.15.67:0/3516935407 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:27.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.582+0000 7fe80a59c640 1 --2- 172.21.15.67:0/3516935407 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe80c15f630 0x7fe80c099d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:27.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.582+0000 7fe80ad9d640 1 -- 172.21.15.67:0/3516935407 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe80c15f630 msgr2=0x7fe80c099d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:27.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.582+0000 7fe80ad9d640 1 --2- 172.21.15.67:0/3516935407 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe80c15f630 0x7fe80c099d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.582+0000 7fe80ad9d640 1 -- 172.21.15.67:0/3516935407 >> v1:172.21.15.72:6789/0 conn(0x7fe80c0b0ab0 legacy=0x7fe80c0cc860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:27.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.582+0000 7fe80ad9d640 1 -- 172.21.15.67:0/3516935407 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe80c15fa10 con 0x7fe80c0b1070 2024-03-24T00:46:27.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.583+0000 7fe80ad9d640 1 --2- 172.21.15.67:0/3516935407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe80c0b1070 0x7fe80c09a240 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7fe804009e50 tx=0x7fe80402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5d14564c503d7f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:27.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.583+0000 7fe809d9b640 1 -- 172.21.15.67:0/3516935407 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe80402cb00 con 0x7fe80c0b1070 2024-03-24T00:46:27.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.583+0000 7fe809d9b640 1 -- 172.21.15.67:0/3516935407 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe80402cca0 con 0x7fe80c0b1070 2024-03-24T00:46:27.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.583+0000 7fe809d9b640 1 -- 172.21.15.67:0/3516935407 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe804034610 con 0x7fe80c0b1070 2024-03-24T00:46:27.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.583+0000 7fe810937640 1 -- 172.21.15.67:0/3516935407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe80c0b1070 msgr2=0x7fe80c09a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:27.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.583+0000 7fe810937640 1 --2- 172.21.15.67:0/3516935407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe80c0b1070 0x7fe80c09a240 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7fe804009e50 tx=0x7fe80402bf60 comp rx=0 tx=0).stop 2024-03-24T00:46:27.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.584+0000 7fe810937640 1 -- 172.21.15.67:0/3516935407 shutdown_connections 2024-03-24T00:46:27.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.584+0000 7fe810937640 1 --2- 172.21.15.67:0/3516935407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe80c0b1070 0x7fe80c09a240 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.584+0000 7fe810937640 1 --2- 172.21.15.67:0/3516935407 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe80c15f630 0x7fe80c099d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.584+0000 7fe810937640 1 -- 172.21.15.67:0/3516935407 >> 172.21.15.67:0/3516935407 conn(0x7fe80c0b36d0 msgr2=0x7fe80c0b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:27.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.584+0000 7fe810937640 1 -- 172.21.15.67:0/3516935407 shutdown_connections 2024-03-24T00:46:27.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.584+0000 7fe810937640 1 -- 172.21.15.67:0/3516935407 wait complete. 2024-03-24T00:46:27.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.585+0000 7fe810937640 1 Processor -- start 2024-03-24T00:46:27.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.585+0000 7fe810937640 1 -- start start 2024-03-24T00:46:27.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.586+0000 7fe810937640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe80c0b1070 0x7fe80c0c78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:27.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.586+0000 7fe810937640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe80c15f630 0x7fe80c0c96a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:27.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.586+0000 7fe810937640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe80c06db00 0x7fe80c15b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:27.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.586+0000 7fe810937640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe80c0cf9c0 con 0x7fe80c15f630 2024-03-24T00:46:27.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.586+0000 7fe810937640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe80c0b2910 con 0x7fe80c06db00 2024-03-24T00:46:27.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.586+0000 7fe810937640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe80c0cfb40 con 0x7fe80c0b1070 2024-03-24T00:46:27.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.586+0000 7fe803fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe80c15f630 0x7fe80c0c96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:27.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.586+0000 7fe803fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe80c15f630 0x7fe80c0c96a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56312/0 (socket says 172.21.15.67:56312) 2024-03-24T00:46:27.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.586+0000 7fe803fff640 1 -- 172.21.15.67:0/645346194 learned_addr learned my addr 172.21.15.67:0/645346194 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:27.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.586+0000 7fe80a59c640 1 --2- 172.21.15.67:0/645346194 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe80c0b1070 0x7fe80c0c78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:27.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.586+0000 7fe80ad9d640 1 --2- 172.21.15.67:0/645346194 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe80c06db00 0x7fe80c15b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:27.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.586+0000 7fe803fff640 1 -- 172.21.15.67:0/645346194 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe80c0b1070 msgr2=0x7fe80c0c78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:27.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.586+0000 7fe803fff640 1 --2- 172.21.15.67:0/645346194 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe80c0b1070 0x7fe80c0c78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.586+0000 7fe803fff640 1 -- 172.21.15.67:0/645346194 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe80c06db00 msgr2=0x7fe80c15b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:27.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.586+0000 7fe803fff640 1 --2- 172.21.15.67:0/645346194 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe80c06db00 0x7fe80c15b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.586+0000 7fe803fff640 1 -- 172.21.15.67:0/645346194 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe80c0c7fe0 con 0x7fe80c15f630 2024-03-24T00:46:27.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.586+0000 7fe80a59c640 1 --2- 172.21.15.67:0/645346194 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe80c0b1070 0x7fe80c0c78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:46:27.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.586+0000 7fe80ad9d640 1 --2- 172.21.15.67:0/645346194 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe80c06db00 0x7fe80c15b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:46:27.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.586+0000 7fe803fff640 1 --2- 172.21.15.67:0/645346194 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe80c15f630 0x7fe80c0c96a0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7fe7f400c430 tx=0x7fe7f400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:27.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.586+0000 7fe802ffd640 1 -- 172.21.15.67:0/645346194 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f4011790 con 0x7fe80c15f630 2024-03-24T00:46:27.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.586+0000 7fe802ffd640 1 -- 172.21.15.67:0/645346194 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7f4017070 con 0x7fe80c15f630 2024-03-24T00:46:27.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.586+0000 7fe810937640 1 -- 172.21.15.67:0/645346194 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe80c0c82c0 con 0x7fe80c15f630 2024-03-24T00:46:27.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.586+0000 7fe802ffd640 1 -- 172.21.15.67:0/645346194 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f4011bb0 con 0x7fe80c15f630 2024-03-24T00:46:27.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.586+0000 7fe810937640 1 -- 172.21.15.67:0/645346194 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe80c0c9be0 con 0x7fe80c15f630 2024-03-24T00:46:27.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.588+0000 7fe802ffd640 1 -- 172.21.15.67:0/645346194 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe7f401c460 con 0x7fe80c15f630 2024-03-24T00:46:27.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.588+0000 7fe802ffd640 1 --2- 172.21.15.67:0/645346194 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe7ec06bbd0 0x7fe7ec06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:27.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.589+0000 7fe802ffd640 1 -- 172.21.15.67:0/645346194 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6389+0+0 (secure 0 0 0) 0x7fe7f408ab70 con 0x7fe80c15f630 2024-03-24T00:46:27.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.589+0000 7fe80a59c640 1 --2- 172.21.15.67:0/645346194 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe7ec06bbd0 0x7fe7ec06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:27.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.589+0000 7fe80a59c640 1 --2- 172.21.15.67:0/645346194 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe7ec06bbd0 0x7fe7ec06e090 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc006fd0 tx=0x7fe7fc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:27.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.590+0000 7fe8135fc740 1 Processor -- start 2024-03-24T00:46:27.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.590+0000 7fe8135fc740 1 -- start start 2024-03-24T00:46:27.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.591+0000 7fe8135fc740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55eae1fbee60 0x55eae1fe5990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:27.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.591+0000 7fe8135fc740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55eae1fe5ed0 0x55eae1fbdea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:27.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.591+0000 7fe8135fc740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eae1fbf240 0x55eae1fbe3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:27.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.591+0000 7fe8135fc740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eae1efdc90 con 0x55eae1fbf240 2024-03-24T00:46:27.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.591+0000 7fe8135fc740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eae1f23070 con 0x55eae1fe5ed0 2024-03-24T00:46:27.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.591+0000 7fe8135fc740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55eae1ecbc80 con 0x55eae1fbee60 2024-03-24T00:46:27.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.592+0000 7fe80ad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eae1fbf240 0x55eae1fbe3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:27.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.592+0000 7fe80ad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eae1fbf240 0x55eae1fbe3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56320/0 (socket says 172.21.15.67:56320) 2024-03-24T00:46:27.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.592+0000 7fe80ad9d640 1 -- 172.21.15.67:0/3615233733 learned_addr learned my addr 172.21.15.67:0/3615233733 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:27.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.592+0000 7fe80a59c640 1 --2- 172.21.15.67:0/3615233733 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55eae1fbee60 0x55eae1fe5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:27.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.592+0000 7fe803fff640 1 --2- 172.21.15.67:0/3615233733 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55eae1fe5ed0 0x55eae1fbdea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:27.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.592+0000 7fe80ad9d640 1 -- 172.21.15.67:0/3615233733 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55eae1fbee60 msgr2=0x55eae1fe5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:27.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.592+0000 7fe80ad9d640 1 --2- 172.21.15.67:0/3615233733 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55eae1fbee60 0x55eae1fe5990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.592+0000 7fe80ad9d640 1 -- 172.21.15.67:0/3615233733 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55eae1fe5ed0 msgr2=0x55eae1fbdea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:27.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.592+0000 7fe80ad9d640 1 --2- 172.21.15.67:0/3615233733 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55eae1fe5ed0 0x55eae1fbdea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.592+0000 7fe80ad9d640 1 -- 172.21.15.67:0/3615233733 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eae1ec3d10 con 0x55eae1fbf240 2024-03-24T00:46:27.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.592+0000 7fe80ad9d640 1 --2- 172.21.15.67:0/3615233733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eae1fbf240 0x55eae1fbe3e0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7fe8040093e0 tx=0x7fe80402e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=166370875b543649 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:27.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.592+0000 7fe7ebfff640 1 -- 172.21.15.67:0/3615233733 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe80402cb00 con 0x55eae1fbf240 2024-03-24T00:46:27.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.592+0000 7fe7ebfff640 1 -- 172.21.15.67:0/3615233733 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe80402ce50 con 0x55eae1fbf240 2024-03-24T00:46:27.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.592+0000 7fe7ebfff640 1 -- 172.21.15.67:0/3615233733 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe80403e630 con 0x55eae1fbf240 2024-03-24T00:46:27.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.593+0000 7fe8135fc740 1 -- 172.21.15.67:0/3615233733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eae1fbf240 msgr2=0x55eae1fbe3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:27.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.593+0000 7fe8135fc740 1 --2- 172.21.15.67:0/3615233733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eae1fbf240 0x55eae1fbe3e0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7fe8040093e0 tx=0x7fe80402e000 comp rx=0 tx=0).stop 2024-03-24T00:46:27.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.593+0000 7fe8135fc740 1 -- 172.21.15.67:0/3615233733 shutdown_connections 2024-03-24T00:46:27.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.593+0000 7fe8135fc740 1 --2- 172.21.15.67:0/3615233733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eae1fbf240 0x55eae1fbe3e0 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.593+0000 7fe8135fc740 1 --2- 172.21.15.67:0/3615233733 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55eae1fe5ed0 0x55eae1fbdea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.593+0000 7fe8135fc740 1 --2- 172.21.15.67:0/3615233733 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55eae1fbee60 0x55eae1fe5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.593+0000 7fe8135fc740 1 -- 172.21.15.67:0/3615233733 >> 172.21.15.67:0/3615233733 conn(0x55eae1f861a0 msgr2=0x55eae2003b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:27.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.593+0000 7fe8135fc740 1 -- 172.21.15.67:0/3615233733 shutdown_connections 2024-03-24T00:46:27.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.593+0000 7fe8135fc740 1 -- 172.21.15.67:0/3615233733 wait complete. 2024-03-24T00:46:27.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.594+0000 7fe8135fc740 1 -- start start 2024-03-24T00:46:27.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.594+0000 7fe8135fc740 1 Processor -- start 2024-03-24T00:46:27.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.596+0000 7fe8135fc740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55eae1fbee60 0x55eae1fd21c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:27.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.596+0000 7fe8135fc740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eae1fbf240 0x55eae1ffb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:27.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.596+0000 7fe8135fc740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55eae1ffbd40 0x55eae20003c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:27.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.596+0000 7fe8135fc740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eae1edf0d0 con 0x55eae1fbf240 2024-03-24T00:46:27.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.596+0000 7fe8135fc740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eae1f1f110 con 0x55eae1ffbd40 2024-03-24T00:46:27.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.596+0000 7fe8135fc740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55eae1fe5500 con 0x55eae1fbee60 2024-03-24T00:46:27.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.596+0000 7fe8135fc740 10 client.0 _ll_register_callbacks cb 0x55eae1f1ae60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:46:27.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.596+0000 7fe803fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eae1fbf240 0x55eae1ffb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:27.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.596+0000 7fe803fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eae1fbf240 0x55eae1ffb800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56326/0 (socket says 172.21.15.67:56326) 2024-03-24T00:46:27.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.596+0000 7fe803fff640 1 -- 172.21.15.67:0/2887464592 learned_addr learned my addr 172.21.15.67:0/2887464592 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:27.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.596+0000 7fe80ad9d640 1 --2- 172.21.15.67:0/2887464592 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55eae1ffbd40 0x55eae20003c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:27.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.596+0000 7fe80a59c640 1 --2- 172.21.15.67:0/2887464592 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55eae1fbee60 0x55eae1fd21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:27.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.596+0000 7fe803fff640 1 -- 172.21.15.67:0/2887464592 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55eae1fbee60 msgr2=0x55eae1fd21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:27.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.596+0000 7fe803fff640 1 --2- 172.21.15.67:0/2887464592 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55eae1fbee60 0x55eae1fd21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.596+0000 7fe803fff640 1 -- 172.21.15.67:0/2887464592 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55eae1ffbd40 msgr2=0x55eae20003c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:27.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.596+0000 7fe803fff640 1 --2- 172.21.15.67:0/2887464592 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55eae1ffbd40 0x55eae20003c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.596+0000 7fe803fff640 1 -- 172.21.15.67:0/2887464592 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55eae1cff7e0 con 0x55eae1fbf240 2024-03-24T00:46:27.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.596+0000 7fe803fff640 1 --2- 172.21.15.67:0/2887464592 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eae1fbf240 0x55eae1ffb800 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7fe7f400cdf0 tx=0x7fe7f4058e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c001b97662847bb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:27.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.596+0000 7fe7e97fa640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:46:27.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.596+0000 7fe7e97fa640 1 -- 172.21.15.67:0/2887464592 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f4070e60 con 0x55eae1fbf240 2024-03-24T00:46:27.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.596+0000 7fe7e97fa640 1 -- 172.21.15.67:0/2887464592 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 6389+0+0 (secure 0 0 0) 0x7fe7f4011d50 con 0x55eae1fbf240 2024-03-24T00:46:27.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.596+0000 7fe8135fc740 1 -- 172.21.15.67:0/2887464592 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe80402ce50 con 0x55eae1fbf240 2024-03-24T00:46:27.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.596+0000 7fe7e97fa640 1 -- 172.21.15.67:0/2887464592 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe7f4077050 con 0x55eae1fbf240 2024-03-24T00:46:27.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.596+0000 7fe8135fc740 10 client.8277 fetch_fsmap learned FSMap version 66 2024-03-24T00:46:27.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.596+0000 7fe8135fc740 1 -- 172.21.15.67:0/2887464592 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=2+}) v3 -- 0x55eae1d5ff70 con 0x55eae1fbf240 2024-03-24T00:46:27.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.597+0000 7fe7e97fa640 1 -- 172.21.15.67:0/2887464592 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe80402ce50 con 0x55eae1fbf240 2024-03-24T00:46:27.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.597+0000 7fe7e97fa640 1 -- 172.21.15.67:0/2887464592 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 66) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fe7f40553b0 con 0x55eae1fbf240 2024-03-24T00:46:27.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.597+0000 7fe8135fc740 10 client.8277 fetch_fsmap finished waiting for FSMap version 66 2024-03-24T00:46:27.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.597+0000 7fe8135fc740 10 client.8277 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-03-24T00:46:27.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.597+0000 7fe8135fc740 20 client.8277 populate_metadata read hostname 'smithi067' 2024-03-24T00:46:27.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.597+0000 7fe8135fc740 1 --2- 172.21.15.67:0/2887464592 >> [v2:172.21.15.67:6832/3311719072,v1:172.21.15.67:6833/3311719072] conn(0x55eae2017210 0x55eae2017630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:27.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.597+0000 7fe8135fc740 4 client.8277 mds_command: new command op to 8163 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:46:27.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.597+0000 7fe8135fc740 1 -- 172.21.15.67:0/2887464592 --> [v2:172.21.15.67:6832/3311719072,v1:172.21.15.67:6833/3311719072] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55eae1f305b0 con 0x55eae2017210 2024-03-24T00:46:27.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.597+0000 7fe80a59c640 1 --2- 172.21.15.67:0/2887464592 >> [v2:172.21.15.67:6832/3311719072,v1:172.21.15.67:6833/3311719072] conn(0x55eae2017210 0x55eae2017630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:27.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.597+0000 7fe80a59c640 1 --2- 172.21.15.67:0/2887464592 >> [v2:172.21.15.67:6832/3311719072,v1:172.21.15.67:6833/3311719072] conn(0x55eae2017210 0x55eae2017630 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c4d6e954df9dcfe server_cookie=bc2b55be116a12c6 in_seq=0 out_seq=0 2024-03-24T00:46:27.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.597+0000 7fe7e97fa640 10 client.8277 ms_handle_connect on v2:172.21.15.67:6832/3311719072 2024-03-24T00:46:27.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.598+0000 7fe7e97fa640 1 -- 172.21.15.67:0/2887464592 <== mds.0 v2:172.21.15.67:6832/3311719072 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55eae1f305b0 con 0x55eae2017210 2024-03-24T00:46:27.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.598+0000 7fe7e97fa640 10 client.8277 handle_command_reply: tid=0 2024-03-24T00:46:27.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.598+0000 7fe8135fc740 1 client.8277 shutdown 2024-03-24T00:46:27.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.598+0000 7fe8135fc740 1 -- 172.21.15.67:0/2887464592 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eae1fbf240 msgr2=0x55eae1ffb800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:27.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.598+0000 7fe8135fc740 1 --2- 172.21.15.67:0/2887464592 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eae1fbf240 0x55eae1ffb800 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7fe7f400cdf0 tx=0x7fe7f4058e00 comp rx=0 tx=0).stop 2024-03-24T00:46:27.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.598+0000 7fe8135fc740 1 -- 172.21.15.67:0/2887464592 shutdown_connections 2024-03-24T00:46:27.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.598+0000 7fe8135fc740 1 --2- 172.21.15.67:0/2887464592 >> [v2:172.21.15.67:6832/3311719072,v1:172.21.15.67:6833/3311719072] conn(0x55eae2017210 0x55eae2017630 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.598+0000 7fe8135fc740 1 --2- 172.21.15.67:0/2887464592 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55eae1ffbd40 0x55eae20003c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.598+0000 7fe8135fc740 1 --2- 172.21.15.67:0/2887464592 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eae1fbf240 0x55eae1ffb800 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.598+0000 7fe8135fc740 1 --2- 172.21.15.67:0/2887464592 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55eae1fbee60 0x55eae1fd21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.598+0000 7fe7e97fa640 0 client.8277 ms_handle_reset on v2:172.21.15.67:6832/3311719072 2024-03-24T00:46:27.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.598+0000 7fe8135fc740 1 -- 172.21.15.67:0/2887464592 >> 172.21.15.67:0/2887464592 conn(0x55eae1f861a0 msgr2=0x55eae1fe6b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:27.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.598+0000 7fe8135fc740 1 -- 172.21.15.67:0/2887464592 shutdown_connections 2024-03-24T00:46:27.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.598+0000 7fe8135fc740 1 -- 172.21.15.67:0/2887464592 wait complete. 2024-03-24T00:46:27.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.598+0000 7fe8135fc740 20 client.8277 trim_cache size 0 max 16384 2024-03-24T00:46:27.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.614+0000 7fe8135fc740 1 Processor -- start 2024-03-24T00:46:27.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.614+0000 7fe8135fc740 1 -- start start 2024-03-24T00:46:27.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.615+0000 7fe8135fc740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eae1fbee60 0x55eae1ffe5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:27.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.615+0000 7fe8135fc740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55eae1fbf240 0x55eae1fb7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:27.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.615+0000 7fe8135fc740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55eae1ffbd40 0x55eae1fb81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:27.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.615+0000 7fe8135fc740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe80c200c10 con 0x55eae1fbee60 2024-03-24T00:46:27.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.615+0000 7fe8135fc740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe80c1ffde0 con 0x55eae1fbf240 2024-03-24T00:46:27.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.615+0000 7fe8135fc740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe80c0c6070 con 0x55eae1ffbd40 2024-03-24T00:46:27.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.615+0000 7fe80a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eae1fbee60 0x55eae1ffe5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:27.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.615+0000 7fe80a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eae1fbee60 0x55eae1ffe5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56338/0 (socket says 172.21.15.67:56338) 2024-03-24T00:46:27.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.615+0000 7fe80a59c640 1 -- 172.21.15.67:0/66207628 learned_addr learned my addr 172.21.15.67:0/66207628 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:27.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.615+0000 7fe80a59c640 1 -- 172.21.15.67:0/66207628 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55eae1ffbd40 msgr2=0x55eae1fb81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:27.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.615+0000 7fe80a59c640 1 --2- 172.21.15.67:0/66207628 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55eae1ffbd40 0x55eae1fb81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.615+0000 7fe803fff640 1 --2- 172.21.15.67:0/66207628 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55eae1fbf240 0x55eae1fb7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:27.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.615+0000 7fe80a59c640 1 -- 172.21.15.67:0/66207628 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55eae1fbf240 msgr2=0x55eae1fb7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:27.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.615+0000 7fe80a59c640 1 --2- 172.21.15.67:0/66207628 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55eae1fbf240 0x55eae1fb7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.616+0000 7fe80a59c640 1 -- 172.21.15.67:0/66207628 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eae1bf7170 con 0x55eae1fbee60 2024-03-24T00:46:27.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.616+0000 7fe80a59c640 1 --2- 172.21.15.67:0/66207628 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eae1fbee60 0x55eae1ffe5b0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc00f5b0 tx=0x7fe7fc018040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a6e4aeff43bcc52 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:27.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.616+0000 7fe7eb7fe640 1 -- 172.21.15.67:0/66207628 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7fc02f070 con 0x55eae1fbee60 2024-03-24T00:46:27.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.616+0000 7fe7eb7fe640 1 -- 172.21.15.67:0/66207628 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7fc01c040 con 0x55eae1fbee60 2024-03-24T00:46:27.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.616+0000 7fe7eb7fe640 1 -- 172.21.15.67:0/66207628 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7fc01b040 con 0x55eae1fbee60 2024-03-24T00:46:27.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.616+0000 7fe8135fc740 1 -- 172.21.15.67:0/66207628 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eae1fbee60 msgr2=0x55eae1ffe5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:27.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.616+0000 7fe8135fc740 1 --2- 172.21.15.67:0/66207628 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eae1fbee60 0x55eae1ffe5b0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc00f5b0 tx=0x7fe7fc018040 comp rx=0 tx=0).stop 2024-03-24T00:46:27.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.617+0000 7fe8135fc740 1 -- 172.21.15.67:0/66207628 shutdown_connections 2024-03-24T00:46:27.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.617+0000 7fe8135fc740 1 --2- 172.21.15.67:0/66207628 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55eae1ffbd40 0x55eae1fb81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.617+0000 7fe8135fc740 1 --2- 172.21.15.67:0/66207628 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55eae1fbf240 0x55eae1fb7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.617+0000 7fe8135fc740 1 --2- 172.21.15.67:0/66207628 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eae1fbee60 0x55eae1ffe5b0 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.617+0000 7fe8135fc740 1 -- 172.21.15.67:0/66207628 >> 172.21.15.67:0/66207628 conn(0x55eae1f861a0 msgr2=0x55eae1fd1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:27.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.617+0000 7fe8135fc740 1 -- 172.21.15.67:0/66207628 shutdown_connections 2024-03-24T00:46:27.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.617+0000 7fe8135fc740 1 -- 172.21.15.67:0/66207628 wait complete. 2024-03-24T00:46:27.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.617+0000 7fe8135fc740 1 -- start start 2024-03-24T00:46:27.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.617+0000 7fe8135fc740 1 Processor -- start 2024-03-24T00:46:27.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.619+0000 7fe8135fc740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eae1fbee60 0x55eae2073700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:27.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.619+0000 7fe8135fc740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55eae1fbf240 0x55eae2077c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:27.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.619+0000 7fe8135fc740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55eae1ffbd40 0x55eae1fe3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:27.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.619+0000 7fe8135fc740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe80c0c6670 con 0x55eae1fbee60 2024-03-24T00:46:27.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.619+0000 7fe8135fc740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eae2002280 con 0x55eae1fbf240 2024-03-24T00:46:27.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.619+0000 7fe8135fc740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55eae1fd1370 con 0x55eae1ffbd40 2024-03-24T00:46:27.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.619+0000 7fe8135fc740 10 client.0 _ll_register_callbacks cb 0x7fe7f400c380 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:46:27.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.619+0000 7fe80a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eae1fbee60 0x55eae2073700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:27.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.619+0000 7fe80a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eae1fbee60 0x55eae2073700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56350/0 (socket says 172.21.15.67:56350) 2024-03-24T00:46:27.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.619+0000 7fe80a59c640 1 -- 172.21.15.67:0/2858523176 learned_addr learned my addr 172.21.15.67:0/2858523176 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:27.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.619+0000 7fe80a59c640 1 -- 172.21.15.67:0/2858523176 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55eae1ffbd40 msgr2=0x55eae1fe3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:27.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.619+0000 7fe80a59c640 1 --2- 172.21.15.67:0/2858523176 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55eae1ffbd40 0x55eae1fe3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.619+0000 7fe80a59c640 1 -- 172.21.15.67:0/2858523176 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55eae1fbf240 msgr2=0x55eae2077c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:27.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.619+0000 7fe803fff640 1 --2- 172.21.15.67:0/2858523176 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55eae1fbf240 0x55eae2077c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:27.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.619+0000 7fe80a59c640 1 --2- 172.21.15.67:0/2858523176 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55eae1fbf240 0x55eae2077c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.619+0000 7fe80a59c640 1 -- 172.21.15.67:0/2858523176 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55eae1fe4730 con 0x55eae1fbee60 2024-03-24T00:46:27.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.619+0000 7fe80a59c640 1 --2- 172.21.15.67:0/2858523176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eae1fbee60 0x55eae2073700 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc00b390 tx=0x7fe7fc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbf4bdd1d1e4c6d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:27.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.619+0000 7fe7e97fa640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:46:27.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.620+0000 7fe7e97fa640 1 -- 172.21.15.67:0/2858523176 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7fc02f070 con 0x55eae1fbee60 2024-03-24T00:46:27.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.620+0000 7fe7e97fa640 1 -- 172.21.15.67:0/2858523176 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 6389+0+0 (secure 0 0 0) 0x7fe7fc005b90 con 0x55eae1fbee60 2024-03-24T00:46:27.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.620+0000 7fe8135fc740 1 -- 172.21.15.67:0/2858523176 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe7fc01c040 con 0x55eae1fbee60 2024-03-24T00:46:27.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.620+0000 7fe7e97fa640 1 -- 172.21.15.67:0/2858523176 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe7fc03c050 con 0x55eae1fbee60 2024-03-24T00:46:27.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.620+0000 7fe8135fc740 10 client.8283 fetch_fsmap learned FSMap version 66 2024-03-24T00:46:27.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.620+0000 7fe8135fc740 1 -- 172.21.15.67:0/2858523176 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=2+}) v3 -- 0x55eae201d9e0 con 0x55eae1fbee60 2024-03-24T00:46:27.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.620+0000 7fe7e97fa640 1 -- 172.21.15.67:0/2858523176 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7fc01c040 con 0x55eae1fbee60 2024-03-24T00:46:27.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.620+0000 7fe7e97fa640 1 -- 172.21.15.67:0/2858523176 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 66) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fe7fc03f020 con 0x55eae1fbee60 2024-03-24T00:46:27.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.620+0000 7fe8135fc740 10 client.8283 fetch_fsmap finished waiting for FSMap version 66 2024-03-24T00:46:27.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.620+0000 7fe8135fc740 10 client.8283 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-03-24T00:46:27.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.620+0000 7fe8135fc740 20 client.8283 populate_metadata read hostname 'smithi067' 2024-03-24T00:46:27.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.620+0000 7fe8135fc740 1 --2- 172.21.15.67:0/2858523176 >> [v2:172.21.15.67:6832/3311719072,v1:172.21.15.67:6833/3311719072] conn(0x55eae1fd0710 0x55eae1fd0af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:27.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.620+0000 7fe8135fc740 4 client.8283 mds_command: new command op to 8163 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/dir", "scrubops": ["recursive", "repair"]}] 2024-03-24T00:46:27.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.620+0000 7fe8135fc740 1 -- 172.21.15.67:0/2858523176 --> [v2:172.21.15.67:6832/3311719072,v1:172.21.15.67:6833/3311719072] -- command(tid 0: {"prefix": "scrub start", "path": "/dir", "scrubops": ["recursive", "repair"]}) v1 -- 0x55eae1f2a900 con 0x55eae1fd0710 2024-03-24T00:46:27.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.620+0000 7fe803fff640 1 --2- 172.21.15.67:0/2858523176 >> [v2:172.21.15.67:6832/3311719072,v1:172.21.15.67:6833/3311719072] conn(0x55eae1fd0710 0x55eae1fd0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:27.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.621+0000 7fe7e97fa640 10 client.8283 ms_handle_connect on v2:172.21.15.67:6832/3311719072 2024-03-24T00:46:27.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.621+0000 7fe803fff640 1 --2- 172.21.15.67:0/2858523176 >> [v2:172.21.15.67:6832/3311719072,v1:172.21.15.67:6833/3311719072] conn(0x55eae1fd0710 0x55eae1fd0af0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76ae1d7d676d80d server_cookie=ea926438379d62ef in_seq=0 out_seq=0 2024-03-24T00:46:27.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.621+0000 7fe7e97fa640 1 -- 172.21.15.67:0/2858523176 <== mds.0 v2:172.21.15.67:6832/3311719072 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55eae1f2a900 con 0x55eae1fd0710 2024-03-24T00:46:27.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.621+0000 7fe7e97fa640 10 client.8283 handle_command_reply: tid=0 2024-03-24T00:46:27.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.621+0000 7fe8135fc740 1 client.8283 shutdown 2024-03-24T00:46:27.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.621+0000 7fe8135fc740 1 -- 172.21.15.67:0/2858523176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eae1fbee60 msgr2=0x55eae2073700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:27.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.621+0000 7fe8135fc740 1 --2- 172.21.15.67:0/2858523176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eae1fbee60 0x55eae2073700 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc00b390 tx=0x7fe7fc004730 comp rx=0 tx=0).stop 2024-03-24T00:46:27.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.621+0000 7fe8135fc740 1 -- 172.21.15.67:0/2858523176 shutdown_connections 2024-03-24T00:46:27.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.621+0000 7fe8135fc740 1 --2- 172.21.15.67:0/2858523176 >> [v2:172.21.15.67:6832/3311719072,v1:172.21.15.67:6833/3311719072] conn(0x55eae1fd0710 0x55eae1fd0af0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.621+0000 7fe8135fc740 1 --2- 172.21.15.67:0/2858523176 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55eae1ffbd40 0x55eae1fe3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.621+0000 7fe8135fc740 1 --2- 172.21.15.67:0/2858523176 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55eae1fbf240 0x55eae2077c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.621+0000 7fe7e97fa640 0 client.8283 ms_handle_reset on v2:172.21.15.67:6832/3311719072 2024-03-24T00:46:27.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.621+0000 7fe8135fc740 1 --2- 172.21.15.67:0/2858523176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eae1fbee60 0x55eae2073700 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.621+0000 7fe8135fc740 1 -- 172.21.15.67:0/2858523176 >> 172.21.15.67:0/2858523176 conn(0x55eae1f861a0 msgr2=0x55eae1fd1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:27.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.621+0000 7fe8135fc740 1 -- 172.21.15.67:0/2858523176 shutdown_connections 2024-03-24T00:46:27.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.621+0000 7fe8135fc740 1 -- 172.21.15.67:0/2858523176 wait complete. 2024-03-24T00:46:27.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.621+0000 7fe8135fc740 20 client.8283 trim_cache size 0 max 16384 2024-03-24T00:46:27.626 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:46:27.626 INFO:teuthology.orchestra.run.smithi067.stdout: "return_code": 0, 2024-03-24T00:46:27.626 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_tag": "4a999f56-60aa-4a25-bf45-1c9c537f5d9c", 2024-03-24T00:46:27.626 INFO:teuthology.orchestra.run.smithi067.stdout: "mode": "asynchronous" 2024-03-24T00:46:27.626 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:46:27.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.622+0000 7fe810937640 1 -- 172.21.15.67:0/645346194 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe7ec06bbd0 msgr2=0x7fe7ec06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:27.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.622+0000 7fe810937640 1 --2- 172.21.15.67:0/645346194 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe7ec06bbd0 0x7fe7ec06e090 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc006fd0 tx=0x7fe7fc008040 comp rx=0 tx=0).stop 2024-03-24T00:46:27.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.622+0000 7fe810937640 1 -- 172.21.15.67:0/645346194 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe80c15f630 msgr2=0x7fe80c0c96a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:27.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.622+0000 7fe810937640 1 --2- 172.21.15.67:0/645346194 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe80c15f630 0x7fe80c0c96a0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7fe7f400c430 tx=0x7fe7f400c8f0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.622+0000 7fe810937640 1 -- 172.21.15.67:0/645346194 shutdown_connections 2024-03-24T00:46:27.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.622+0000 7fe810937640 1 --2- 172.21.15.67:0/645346194 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe7ec06bbd0 0x7fe7ec06e090 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.622+0000 7fe810937640 1 --2- 172.21.15.67:0/645346194 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe80c06db00 0x7fe80c15b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.622+0000 7fe810937640 1 --2- 172.21.15.67:0/645346194 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe80c15f630 0x7fe80c0c96a0 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.622+0000 7fe810937640 1 --2- 172.21.15.67:0/645346194 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe80c0b1070 0x7fe80c0c78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.622+0000 7fe810937640 1 -- 172.21.15.67:0/645346194 >> 172.21.15.67:0/645346194 conn(0x7fe80c0b36d0 msgr2=0x7fe80c0b1540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:27.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.622+0000 7fe810937640 1 -- 172.21.15.67:0/645346194 shutdown_connections 2024-03-24T00:46:27.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.622+0000 7fe810937640 1 -- 172.21.15.67:0/645346194 wait complete. 2024-03-24T00:46:27.628 INFO:tasks.ceph.mds.c.smithi067.stderr:2024-03-24T00:46:27.623+0000 7f97d8edf640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [...2,head] /dir/file auth v1 ap=1 DIRTYPARENT s=0 nl=2 n() (iversion lock) | dirtyparent=1 authpin=1 scrubqueue=0 0x55e4b9e62100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(14)0x10000000001:[<0x10000000000/file v8>,<0x1/dir v15>]//[]","memoryvalue":"(14)0x10000000001:[<0x10000000000/file v1>,<0x1/dir v22>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-24T00:46:27.633 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:46:27.633 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.6:0 scrub status 2024-03-24T00:46:27.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.737+0000 7f4816337640 1 Processor -- start 2024-03-24T00:46:27.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.737+0000 7f4816337640 1 -- start start 2024-03-24T00:46:27.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.738+0000 7f4816337640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f48100d0470 0x7f48100b5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:27.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.738+0000 7f4816337640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f48100b5fa0 0x7f4810165e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:27.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.738+0000 7f4816337640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f481006c7c0 con 0x7f48101623c0 2024-03-24T00:46:27.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.738+0000 7f4816337640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48100d08d0 con 0x7f48100b5fa0 2024-03-24T00:46:27.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.738+0000 7f4816337640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48100b63f0 con 0x7f48100d0470 2024-03-24T00:46:27.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.738+0000 7f48148d4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f48100b5fa0 0x7f4810165e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:27.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.738+0000 7f480ffff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f48101623c0 0x7f48100ce630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35016/0 (socket says 172.21.15.67:35016) 2024-03-24T00:46:27.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.738+0000 7f480ffff640 1 -- 172.21.15.67:0/1090166041 learned_addr learned my addr 172.21.15.67:0/1090166041 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:46:27.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.738+0000 7f4807fff640 1 --2- 172.21.15.67:0/1090166041 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f48100d0470 0x7f48100b5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:27.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.738+0000 7f48148d4640 1 -- 172.21.15.67:0/1090166041 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f48100d0470 msgr2=0x7f48100b5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:27.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.738+0000 7f48148d4640 1 --2- 172.21.15.67:0/1090166041 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f48100d0470 0x7f48100b5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.738+0000 7f48148d4640 1 -- 172.21.15.67:0/1090166041 >> v1:172.21.15.67:6789/0 conn(0x7f48101623c0 legacy=0x7f48100ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:27.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.738+0000 7f48148d4640 1 -- 172.21.15.67:0/1090166041 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f481006caa0 con 0x7f48100b5fa0 2024-03-24T00:46:27.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.738+0000 7f480f7fe640 1 -- 172.21.15.67:0/1090166041 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3021635769 0 0) 0x7f481006c7c0 con 0x7f48101623c0 2024-03-24T00:46:27.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.740+0000 7f48148d4640 1 --2- 172.21.15.67:0/1090166041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48100b5fa0 0x7f4810165e90 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f4808009d90 tx=0x7f480802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfe126a16ddd4551 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:27.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.740+0000 7f480f7fe640 1 -- 172.21.15.67:0/1090166041 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4808039070 con 0x7f48100b5fa0 2024-03-24T00:46:27.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.740+0000 7f480f7fe640 1 -- 172.21.15.67:0/1090166041 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f480802cc50 con 0x7f48100b5fa0 2024-03-24T00:46:27.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.740+0000 7f480f7fe640 1 -- 172.21.15.67:0/1090166041 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4808034440 con 0x7f48100b5fa0 2024-03-24T00:46:27.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.740+0000 7f4816337640 1 -- 172.21.15.67:0/1090166041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48100b5fa0 msgr2=0x7f4810165e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:27.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.740+0000 7f4816337640 1 --2- 172.21.15.67:0/1090166041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48100b5fa0 0x7f4810165e90 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f4808009d90 tx=0x7f480802bea0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.740+0000 7f4816337640 1 -- 172.21.15.67:0/1090166041 shutdown_connections 2024-03-24T00:46:27.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.740+0000 7f4816337640 1 --2- 172.21.15.67:0/1090166041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48100b5fa0 0x7f4810165e90 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.740+0000 7f4816337640 1 --2- 172.21.15.67:0/1090166041 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f48100d0470 0x7f48100b5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.740+0000 7f4816337640 1 -- 172.21.15.67:0/1090166041 >> 172.21.15.67:0/1090166041 conn(0x7f4810072da0 msgr2=0x7f48100d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:27.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.740+0000 7f4816337640 1 -- 172.21.15.67:0/1090166041 shutdown_connections 2024-03-24T00:46:27.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.740+0000 7f4816337640 1 -- 172.21.15.67:0/1090166041 wait complete. 2024-03-24T00:46:27.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.742+0000 7f4816337640 1 Processor -- start 2024-03-24T00:46:27.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.742+0000 7f4816337640 1 -- start start 2024-03-24T00:46:27.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.743+0000 7f4816337640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48100b5fa0 0x7f4810155360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:27.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.743+0000 7f4816337640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f48100d0470 0x7f48100c9fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:27.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.743+0000 7f4816337640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f48101623c0 0x7f4810163c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:27.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.743+0000 7f4816337640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f481006d060 con 0x7f48100b5fa0 2024-03-24T00:46:27.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.743+0000 7f4816337640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48100cedf0 con 0x7f48101623c0 2024-03-24T00:46:27.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.743+0000 7f4816337640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48100b6e30 con 0x7f48100d0470 2024-03-24T00:46:27.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.743+0000 7f480ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48100b5fa0 0x7f4810155360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:27.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.743+0000 7f480ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48100b5fa0 0x7f4810155360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56362/0 (socket says 172.21.15.67:56362) 2024-03-24T00:46:27.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.743+0000 7f480ffff640 1 -- 172.21.15.67:0/2934861857 learned_addr learned my addr 172.21.15.67:0/2934861857 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:27.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.744+0000 7f4807fff640 1 --2- 172.21.15.67:0/2934861857 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f48100d0470 0x7f48100c9fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:27.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.744+0000 7f48148d4640 1 --2- 172.21.15.67:0/2934861857 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f48101623c0 0x7f4810163c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:27.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.744+0000 7f480ffff640 1 -- 172.21.15.67:0/2934861857 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f48100d0470 msgr2=0x7f48100c9fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:27.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.744+0000 7f480ffff640 1 --2- 172.21.15.67:0/2934861857 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f48100d0470 0x7f48100c9fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.744+0000 7f480ffff640 1 -- 172.21.15.67:0/2934861857 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f48101623c0 msgr2=0x7f4810163c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:27.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.744+0000 7f480ffff640 1 --2- 172.21.15.67:0/2934861857 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f48101623c0 0x7f4810163c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.744+0000 7f480ffff640 1 -- 172.21.15.67:0/2934861857 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48100c9210 con 0x7f48100b5fa0 2024-03-24T00:46:27.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.744+0000 7f480ffff640 1 --2- 172.21.15.67:0/2934861857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48100b5fa0 0x7f4810155360 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f480000c430 tx=0x7f480000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:27.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.744+0000 7f480d7fa640 1 -- 172.21.15.67:0/2934861857 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48000074e0 con 0x7f48100b5fa0 2024-03-24T00:46:27.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.744+0000 7f480d7fa640 1 -- 172.21.15.67:0/2934861857 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48000047e0 con 0x7f48100b5fa0 2024-03-24T00:46:27.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.744+0000 7f4816337640 1 -- 172.21.15.67:0/2934861857 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48100c94f0 con 0x7f48100b5fa0 2024-03-24T00:46:27.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.744+0000 7f480d7fa640 1 -- 172.21.15.67:0/2934861857 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4800004bd0 con 0x7f48100b5fa0 2024-03-24T00:46:27.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.744+0000 7f4816337640 1 -- 172.21.15.67:0/2934861857 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48101558a0 con 0x7f48100b5fa0 2024-03-24T00:46:27.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.746+0000 7f480d7fa640 1 -- 172.21.15.67:0/2934861857 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4800011070 con 0x7f48100b5fa0 2024-03-24T00:46:27.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.746+0000 7f480d7fa640 1 --2- 172.21.15.67:0/2934861857 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f47f006bc00 0x7f47f006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:27.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.746+0000 7f480d7fa640 1 -- 172.21.15.67:0/2934861857 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f480008a9c0 con 0x7f48100b5fa0 2024-03-24T00:46:27.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.747+0000 7f4807fff640 1 --2- 172.21.15.67:0/2934861857 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f47f006bc00 0x7f47f006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:27.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.747+0000 7f4807fff640 1 --2- 172.21.15.67:0/2934861857 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f47f006bc00 0x7f47f006e0c0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f481015ec80 tx=0x7f47f8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:27.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.747+0000 7f4818ffc740 1 Processor -- start 2024-03-24T00:46:27.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.747+0000 7f4818ffc740 1 -- start start 2024-03-24T00:46:27.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.748+0000 7f4818ffc740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2571ce50 0x559b256efdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:27.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.748+0000 7f4818ffc740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559b2571d230 0x559b2571fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:27.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.748+0000 7f4818ffc740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559b256f0330 0x559b25720120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:27.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.748+0000 7f4818ffc740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b2565a3d0 con 0x559b2571ce50 2024-03-24T00:46:27.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.748+0000 7f4818ffc740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b25602ad0 con 0x559b2571d230 2024-03-24T00:46:27.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.748+0000 7f4818ffc740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x559b25656470 con 0x559b256f0330 2024-03-24T00:46:27.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.748+0000 7f480ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2571ce50 0x559b256efdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:27.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.748+0000 7f480ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2571ce50 0x559b256efdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56376/0 (socket says 172.21.15.67:56376) 2024-03-24T00:46:27.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.748+0000 7f480ffff640 1 -- 172.21.15.67:0/4025841274 learned_addr learned my addr 172.21.15.67:0/4025841274 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:27.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.748+0000 7f4807fff640 1 --2- 172.21.15.67:0/4025841274 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559b2571d230 0x559b2571fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:27.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.748+0000 7f480ffff640 1 -- 172.21.15.67:0/4025841274 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559b256f0330 msgr2=0x559b25720120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:27.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.748+0000 7f480ffff640 1 --2- 172.21.15.67:0/4025841274 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559b256f0330 0x559b25720120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.748+0000 7f480ffff640 1 -- 172.21.15.67:0/4025841274 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559b2571d230 msgr2=0x559b2571fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:27.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.748+0000 7f480ffff640 1 --2- 172.21.15.67:0/4025841274 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559b2571d230 0x559b2571fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.748+0000 7f480ffff640 1 -- 172.21.15.67:0/4025841274 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559b254387d0 con 0x559b2571ce50 2024-03-24T00:46:27.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.749+0000 7f480ffff640 1 --2- 172.21.15.67:0/4025841274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2571ce50 0x559b256efdf0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f4800001120 tx=0x7f4800058770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91c091872c248712 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:27.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.749+0000 7f4805ffb640 1 -- 172.21.15.67:0/4025841274 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4800061210 con 0x559b2571ce50 2024-03-24T00:46:27.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.749+0000 7f4805ffb640 1 -- 172.21.15.67:0/4025841274 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f480006ce50 con 0x559b2571ce50 2024-03-24T00:46:27.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.749+0000 7f4805ffb640 1 -- 172.21.15.67:0/4025841274 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4800004d70 con 0x559b2571ce50 2024-03-24T00:46:27.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.749+0000 7f4818ffc740 1 -- 172.21.15.67:0/4025841274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2571ce50 msgr2=0x559b256efdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:27.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.749+0000 7f4818ffc740 1 --2- 172.21.15.67:0/4025841274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2571ce50 0x559b256efdf0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f4800001120 tx=0x7f4800058770 comp rx=0 tx=0).stop 2024-03-24T00:46:27.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.749+0000 7f4818ffc740 1 -- 172.21.15.67:0/4025841274 shutdown_connections 2024-03-24T00:46:27.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.749+0000 7f4818ffc740 1 --2- 172.21.15.67:0/4025841274 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559b256f0330 0x559b25720120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.749+0000 7f4818ffc740 1 --2- 172.21.15.67:0/4025841274 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559b2571d230 0x559b2571fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.749+0000 7f4818ffc740 1 --2- 172.21.15.67:0/4025841274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2571ce50 0x559b256efdf0 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.749+0000 7f4818ffc740 1 -- 172.21.15.67:0/4025841274 >> 172.21.15.67:0/4025841274 conn(0x559b256f2060 msgr2=0x559b2573abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:27.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.749+0000 7f4818ffc740 1 -- 172.21.15.67:0/4025841274 shutdown_connections 2024-03-24T00:46:27.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.749+0000 7f4818ffc740 1 -- 172.21.15.67:0/4025841274 wait complete. 2024-03-24T00:46:27.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.750+0000 7f4818ffc740 1 -- start start 2024-03-24T00:46:27.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.750+0000 7f4818ffc740 1 Processor -- start 2024-03-24T00:46:27.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.751+0000 7f4818ffc740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2571ce50 0x559b25715410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:27.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.751+0000 7f4818ffc740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559b2571d230 0x559b2571e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:27.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.751+0000 7f4818ffc740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559b2571ecd0 0x559b257231b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:27.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.751+0000 7f4818ffc740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b256f4700 con 0x559b2571ce50 2024-03-24T00:46:27.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.751+0000 7f4818ffc740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b25616060 con 0x559b2571d230 2024-03-24T00:46:27.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.751+0000 7f4818ffc740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x559b256f6f40 con 0x559b2571ecd0 2024-03-24T00:46:27.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.751+0000 7f4818ffc740 10 client.0 _ll_register_callbacks cb 0x559b255f86a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:46:27.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.751+0000 7f480ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2571ce50 0x559b25715410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:27.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.751+0000 7f480ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2571ce50 0x559b25715410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56392/0 (socket says 172.21.15.67:56392) 2024-03-24T00:46:27.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.751+0000 7f480ffff640 1 -- 172.21.15.67:0/3511510506 learned_addr learned my addr 172.21.15.67:0/3511510506 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:27.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.751+0000 7f4807fff640 1 --2- 172.21.15.67:0/3511510506 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559b2571d230 0x559b2571e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:27.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.751+0000 7f48148d4640 1 --2- 172.21.15.67:0/3511510506 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559b2571ecd0 0x559b257231b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:27.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.751+0000 7f480ffff640 1 -- 172.21.15.67:0/3511510506 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559b2571ecd0 msgr2=0x559b257231b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:27.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.751+0000 7f480ffff640 1 --2- 172.21.15.67:0/3511510506 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559b2571ecd0 0x559b257231b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.751+0000 7f480ffff640 1 -- 172.21.15.67:0/3511510506 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559b2571d230 msgr2=0x559b2571e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:27.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.751+0000 7f480ffff640 1 --2- 172.21.15.67:0/3511510506 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559b2571d230 0x559b2571e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.751+0000 7f480ffff640 1 -- 172.21.15.67:0/3511510506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559b2536e3f0 con 0x559b2571ce50 2024-03-24T00:46:27.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.751+0000 7f4807fff640 1 --2- 172.21.15.67:0/3511510506 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559b2571d230 0x559b2571e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:46:27.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.751+0000 7f480ffff640 1 --2- 172.21.15.67:0/3511510506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2571ce50 0x559b25715410 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f4800000f60 tx=0x7f4800000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fa7bb94bb7b4342 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:27.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.751+0000 7f47eeffd640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:46:27.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.752+0000 7f47eeffd640 1 -- 172.21.15.67:0/3511510506 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4800061210 con 0x559b2571ce50 2024-03-24T00:46:27.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.752+0000 7f47eeffd640 1 -- 172.21.15.67:0/3511510506 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f480001ad60 con 0x559b2571ce50 2024-03-24T00:46:27.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.752+0000 7f4818ffc740 1 -- 172.21.15.67:0/3511510506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f480006ce50 con 0x559b2571ce50 2024-03-24T00:46:27.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.752+0000 7f47eeffd640 1 -- 172.21.15.67:0/3511510506 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) v2 ==== 24+0+0 (secure 0 0 0) 0x7f480005db70 con 0x559b2571ce50 2024-03-24T00:46:27.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.752+0000 7f4818ffc740 10 client.8298 fetch_fsmap learned FSMap version 66 2024-03-24T00:46:27.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.752+0000 7f4818ffc740 1 -- 172.21.15.67:0/3511510506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=2+}) v3 -- 0x559b2532e0f0 con 0x559b2571ce50 2024-03-24T00:46:27.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.752+0000 7f47eeffd640 1 -- 172.21.15.67:0/3511510506 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f480006ce50 con 0x559b2571ce50 2024-03-24T00:46:27.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.752+0000 7f47eeffd640 1 -- 172.21.15.67:0/3511510506 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 66) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f4800057f50 con 0x559b2571ce50 2024-03-24T00:46:27.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.752+0000 7f4818ffc740 10 client.8298 fetch_fsmap finished waiting for FSMap version 66 2024-03-24T00:46:27.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.752+0000 7f4818ffc740 10 client.8298 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-03-24T00:46:27.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.752+0000 7f4818ffc740 20 client.8298 populate_metadata read hostname 'smithi067' 2024-03-24T00:46:27.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.752+0000 7f4818ffc740 1 --2- 172.21.15.67:0/3511510506 >> [v2:172.21.15.67:6832/3311719072,v1:172.21.15.67:6833/3311719072] conn(0x559b25739be0 0x559b25739fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:27.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.752+0000 7f4818ffc740 4 client.8298 mds_command: new command op to 8163 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:46:27.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.752+0000 7f4818ffc740 1 -- 172.21.15.67:0/3511510506 --> [v2:172.21.15.67:6832/3311719072,v1:172.21.15.67:6833/3311719072] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559b25667910 con 0x559b25739be0 2024-03-24T00:46:27.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.752+0000 7f4807fff640 1 --2- 172.21.15.67:0/3511510506 >> [v2:172.21.15.67:6832/3311719072,v1:172.21.15.67:6833/3311719072] conn(0x559b25739be0 0x559b25739fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:27.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.753+0000 7f4807fff640 1 --2- 172.21.15.67:0/3511510506 >> [v2:172.21.15.67:6832/3311719072,v1:172.21.15.67:6833/3311719072] conn(0x559b25739be0 0x559b25739fc0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=662057f7925ec50b server_cookie=1aa52f5dbafd2828 in_seq=0 out_seq=0 2024-03-24T00:46:27.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.753+0000 7f47eeffd640 10 client.8298 ms_handle_connect on v2:172.21.15.67:6832/3311719072 2024-03-24T00:46:27.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.753+0000 7f47eeffd640 1 -- 172.21.15.67:0/3511510506 <== mds.0 v2:172.21.15.67:6832/3311719072 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x559b25667910 con 0x559b25739be0 2024-03-24T00:46:27.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.753+0000 7f47eeffd640 10 client.8298 handle_command_reply: tid=0 2024-03-24T00:46:27.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.753+0000 7f4818ffc740 1 client.8298 shutdown 2024-03-24T00:46:27.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.753+0000 7f4818ffc740 1 -- 172.21.15.67:0/3511510506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2571ce50 msgr2=0x559b25715410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:27.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.753+0000 7f4818ffc740 1 --2- 172.21.15.67:0/3511510506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2571ce50 0x559b25715410 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f4800000f60 tx=0x7f4800000ed0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.753+0000 7f4818ffc740 1 -- 172.21.15.67:0/3511510506 shutdown_connections 2024-03-24T00:46:27.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.753+0000 7f4818ffc740 1 --2- 172.21.15.67:0/3511510506 >> [v2:172.21.15.67:6832/3311719072,v1:172.21.15.67:6833/3311719072] conn(0x559b25739be0 0x559b25739fc0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.753+0000 7f4818ffc740 1 --2- 172.21.15.67:0/3511510506 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559b2571ecd0 0x559b257231b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.753+0000 7f47eeffd640 0 client.8298 ms_handle_reset on v2:172.21.15.67:6832/3311719072 2024-03-24T00:46:27.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.753+0000 7f4818ffc740 1 --2- 172.21.15.67:0/3511510506 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559b2571d230 0x559b2571e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.753+0000 7f4818ffc740 1 --2- 172.21.15.67:0/3511510506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2571ce50 0x559b25715410 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.753+0000 7f4818ffc740 1 -- 172.21.15.67:0/3511510506 >> 172.21.15.67:0/3511510506 conn(0x559b256f0330 msgr2=0x559b256f3d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:27.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.753+0000 7f4818ffc740 1 -- 172.21.15.67:0/3511510506 shutdown_connections 2024-03-24T00:46:27.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.753+0000 7f4818ffc740 1 -- 172.21.15.67:0/3511510506 wait complete. 2024-03-24T00:46:27.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.754+0000 7f4818ffc740 20 client.8298 trim_cache size 0 max 16384 2024-03-24T00:46:27.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.785+0000 7f4818ffc740 1 Processor -- start 2024-03-24T00:46:27.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.785+0000 7f4818ffc740 1 -- start start 2024-03-24T00:46:27.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.786+0000 7f4818ffc740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559b2571ce50 0x559b256f3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:27.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.786+0000 7f4818ffc740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2571d230 0x559b25717230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:27.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.786+0000 7f4818ffc740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559b2571ecd0 0x559b25717770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:27.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.786+0000 7f4818ffc740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4810204f30 con 0x559b2571d230 2024-03-24T00:46:27.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.786+0000 7f4818ffc740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4810204090 con 0x559b2571ecd0 2024-03-24T00:46:27.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.786+0000 7f4818ffc740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48100c8360 con 0x559b2571ce50 2024-03-24T00:46:27.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.786+0000 7f4807fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2571d230 0x559b25717230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:27.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.786+0000 7f4807fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2571d230 0x559b25717230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56406/0 (socket says 172.21.15.67:56406) 2024-03-24T00:46:27.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.786+0000 7f4807fff640 1 -- 172.21.15.67:0/749326040 learned_addr learned my addr 172.21.15.67:0/749326040 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:27.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.787+0000 7f4807fff640 1 -- 172.21.15.67:0/749326040 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559b2571ce50 msgr2=0x559b256f3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:27.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.787+0000 7f4807fff640 1 --2- 172.21.15.67:0/749326040 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559b2571ce50 0x559b256f3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.787+0000 7f4807fff640 1 -- 172.21.15.67:0/749326040 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559b2571ecd0 msgr2=0x559b25717770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:27.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.787+0000 7f48148d4640 1 --2- 172.21.15.67:0/749326040 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559b2571ecd0 0x559b25717770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:27.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.787+0000 7f4807fff640 1 --2- 172.21.15.67:0/749326040 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559b2571ecd0 0x559b25717770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.787+0000 7f4807fff640 1 -- 172.21.15.67:0/749326040 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559b25723520 con 0x559b2571d230 2024-03-24T00:46:27.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.787+0000 7f48148d4640 1 --2- 172.21.15.67:0/749326040 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559b2571ecd0 0x559b25717770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:46:27.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.787+0000 7f4807fff640 1 --2- 172.21.15.67:0/749326040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2571d230 0x559b25717230 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f47f8006fd0 tx=0x7f47f8014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2682a9b1fed0a1e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:27.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.787+0000 7f48057fa640 1 -- 172.21.15.67:0/749326040 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f802d070 con 0x559b2571d230 2024-03-24T00:46:27.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.787+0000 7f48057fa640 1 -- 172.21.15.67:0/749326040 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47f8027d70 con 0x559b2571d230 2024-03-24T00:46:27.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.787+0000 7f48057fa640 1 -- 172.21.15.67:0/749326040 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f800f040 con 0x559b2571d230 2024-03-24T00:46:27.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.787+0000 7f4818ffc740 1 -- 172.21.15.67:0/749326040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2571d230 msgr2=0x559b25717230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:27.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.787+0000 7f4818ffc740 1 --2- 172.21.15.67:0/749326040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2571d230 0x559b25717230 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f47f8006fd0 tx=0x7f47f8014000 comp rx=0 tx=0).stop 2024-03-24T00:46:27.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.787+0000 7f4818ffc740 1 -- 172.21.15.67:0/749326040 shutdown_connections 2024-03-24T00:46:27.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.787+0000 7f4818ffc740 1 --2- 172.21.15.67:0/749326040 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559b2571ecd0 0x559b25717770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.787+0000 7f4818ffc740 1 --2- 172.21.15.67:0/749326040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2571d230 0x559b25717230 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.787+0000 7f4818ffc740 1 --2- 172.21.15.67:0/749326040 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559b2571ce50 0x559b256f3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.787+0000 7f4818ffc740 1 -- 172.21.15.67:0/749326040 >> 172.21.15.67:0/749326040 conn(0x559b256f0330 msgr2=0x559b256eb390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:27.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.788+0000 7f4818ffc740 1 -- 172.21.15.67:0/749326040 shutdown_connections 2024-03-24T00:46:27.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.788+0000 7f4818ffc740 1 -- 172.21.15.67:0/749326040 wait complete. 2024-03-24T00:46:27.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.788+0000 7f4818ffc740 1 -- start start 2024-03-24T00:46:27.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.788+0000 7f4818ffc740 1 Processor -- start 2024-03-24T00:46:27.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.789+0000 7f4818ffc740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559b2571ce50 0x559b256ef0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:27.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.789+0000 7f4818ffc740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559b2571d230 0x559b257ac3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:27.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.789+0000 7f4818ffc740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2571ecd0 0x559b257ac8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:27.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.789+0000 7f4818ffc740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4810203af0 con 0x559b2571ecd0 2024-03-24T00:46:27.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.789+0000 7f4818ffc740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b2573aa60 con 0x559b2571d230 2024-03-24T00:46:27.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.789+0000 7f4818ffc740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x559b2571b570 con 0x559b2571ce50 2024-03-24T00:46:27.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.789+0000 7f4818ffc740 10 client.0 _ll_register_callbacks cb 0x7f480006f070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:46:27.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.789+0000 7f48148d4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2571ecd0 0x559b257ac8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:27.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.789+0000 7f48148d4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2571ecd0 0x559b257ac8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56412/0 (socket says 172.21.15.67:56412) 2024-03-24T00:46:27.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.789+0000 7f48148d4640 1 -- 172.21.15.67:0/118535062 learned_addr learned my addr 172.21.15.67:0/118535062 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:27.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.789+0000 7f4807fff640 1 --2- 172.21.15.67:0/118535062 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559b2571d230 0x559b257ac3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:27.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.789+0000 7f480ffff640 1 --2- 172.21.15.67:0/118535062 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559b2571ce50 0x559b256ef0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:27.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.789+0000 7f48148d4640 1 -- 172.21.15.67:0/118535062 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559b2571ce50 msgr2=0x559b256ef0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:27.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.789+0000 7f48148d4640 1 --2- 172.21.15.67:0/118535062 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559b2571ce50 0x559b256ef0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.789+0000 7f48148d4640 1 -- 172.21.15.67:0/118535062 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559b2571d230 msgr2=0x559b257ac3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:27.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.789+0000 7f48148d4640 1 --2- 172.21.15.67:0/118535062 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559b2571d230 0x559b257ac3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.789+0000 7f48148d4640 1 -- 172.21.15.67:0/118535062 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559b257ace30 con 0x559b2571ecd0 2024-03-24T00:46:27.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.790+0000 7f48148d4640 1 --2- 172.21.15.67:0/118535062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2571ecd0 0x559b257ac8f0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f4808009eb0 tx=0x7f4808004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94bf6bff739b5a7a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:27.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.790+0000 7f47eeffd640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:46:27.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.790+0000 7f47eeffd640 1 -- 172.21.15.67:0/118535062 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4808039040 con 0x559b2571ecd0 2024-03-24T00:46:27.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.790+0000 7f47eeffd640 1 -- 172.21.15.67:0/118535062 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f48080346d0 con 0x559b2571ecd0 2024-03-24T00:46:27.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.790+0000 7f4818ffc740 1 -- 172.21.15.67:0/118535062 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f47f8027d70 con 0x559b2571ecd0 2024-03-24T00:46:27.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.790+0000 7f47eeffd640 1 -- 172.21.15.67:0/118535062 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4808003990 con 0x559b2571ecd0 2024-03-24T00:46:27.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.790+0000 7f4818ffc740 10 client.8304 fetch_fsmap learned FSMap version 66 2024-03-24T00:46:27.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.790+0000 7f4818ffc740 1 -- 172.21.15.67:0/118535062 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=2+}) v3 -- 0x559b256ef920 con 0x559b2571ecd0 2024-03-24T00:46:27.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.790+0000 7f47eeffd640 1 -- 172.21.15.67:0/118535062 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47f8027d70 con 0x559b2571ecd0 2024-03-24T00:46:27.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.790+0000 7f47eeffd640 1 -- 172.21.15.67:0/118535062 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 66) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f4808053080 con 0x559b2571ecd0 2024-03-24T00:46:27.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.790+0000 7f4818ffc740 10 client.8304 fetch_fsmap finished waiting for FSMap version 66 2024-03-24T00:46:27.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.790+0000 7f4818ffc740 10 client.8304 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-03-24T00:46:27.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.790+0000 7f4818ffc740 20 client.8304 populate_metadata read hostname 'smithi067' 2024-03-24T00:46:27.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.790+0000 7f4818ffc740 1 --2- 172.21.15.67:0/118535062 >> [v2:172.21.15.67:6832/3311719072,v1:172.21.15.67:6833/3311719072] conn(0x559b2571dcb0 0x559b257210e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:27.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.790+0000 7f4818ffc740 4 client.8304 mds_command: new command op to 8163 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:46:27.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.790+0000 7f4818ffc740 1 -- 172.21.15.67:0/118535062 --> [v2:172.21.15.67:6832/3311719072,v1:172.21.15.67:6833/3311719072] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x559b25661c60 con 0x559b2571dcb0 2024-03-24T00:46:27.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.790+0000 7f480ffff640 1 --2- 172.21.15.67:0/118535062 >> [v2:172.21.15.67:6832/3311719072,v1:172.21.15.67:6833/3311719072] conn(0x559b2571dcb0 0x559b257210e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:27.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.791+0000 7f47eeffd640 10 client.8304 ms_handle_connect on v2:172.21.15.67:6832/3311719072 2024-03-24T00:46:27.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.791+0000 7f480ffff640 1 --2- 172.21.15.67:0/118535062 >> [v2:172.21.15.67:6832/3311719072,v1:172.21.15.67:6833/3311719072] conn(0x559b2571dcb0 0x559b257210e0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a44a186e67ee0f81 server_cookie=328844c4d6956714 in_seq=0 out_seq=0 2024-03-24T00:46:27.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.791+0000 7f47eeffd640 1 -- 172.21.15.67:0/118535062 <== mds.0 v2:172.21.15.67:6832/3311719072 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+263 (crc 0 0 0) 0x559b25661c60 con 0x559b2571dcb0 2024-03-24T00:46:27.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.791+0000 7f47eeffd640 10 client.8304 handle_command_reply: tid=0 2024-03-24T00:46:27.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.791+0000 7f4818ffc740 1 client.8304 shutdown 2024-03-24T00:46:27.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.791+0000 7f4818ffc740 1 -- 172.21.15.67:0/118535062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2571ecd0 msgr2=0x559b257ac8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:27.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.791+0000 7f4818ffc740 1 --2- 172.21.15.67:0/118535062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2571ecd0 0x559b257ac8f0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f4808009eb0 tx=0x7f4808004a40 comp rx=0 tx=0).stop 2024-03-24T00:46:27.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.791+0000 7f4818ffc740 1 -- 172.21.15.67:0/118535062 shutdown_connections 2024-03-24T00:46:27.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.791+0000 7f4818ffc740 1 --2- 172.21.15.67:0/118535062 >> [v2:172.21.15.67:6832/3311719072,v1:172.21.15.67:6833/3311719072] conn(0x559b2571dcb0 0x559b257210e0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.791+0000 7f4818ffc740 1 --2- 172.21.15.67:0/118535062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2571ecd0 0x559b257ac8f0 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.791+0000 7f4818ffc740 1 --2- 172.21.15.67:0/118535062 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559b2571d230 0x559b257ac3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.791+0000 7f47eeffd640 0 client.8304 ms_handle_reset on v2:172.21.15.67:6832/3311719072 2024-03-24T00:46:27.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.791+0000 7f4818ffc740 1 --2- 172.21.15.67:0/118535062 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559b2571ce50 0x559b256ef0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.791+0000 7f4818ffc740 1 -- 172.21.15.67:0/118535062 >> 172.21.15.67:0/118535062 conn(0x559b256f0330 msgr2=0x559b2573e060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:27.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.791+0000 7f4818ffc740 1 -- 172.21.15.67:0/118535062 shutdown_connections 2024-03-24T00:46:27.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.791+0000 7f4818ffc740 1 -- 172.21.15.67:0/118535062 wait complete. 2024-03-24T00:46:27.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.791+0000 7f4818ffc740 20 client.8304 trim_cache size 0 max 16384 2024-03-24T00:46:27.797 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:46:27.797 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T00:46:27.797 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": { 2024-03-24T00:46:27.797 INFO:teuthology.orchestra.run.smithi067.stdout: "4a999f56-60aa-4a25-bf45-1c9c537f5d9c": { 2024-03-24T00:46:27.797 INFO:teuthology.orchestra.run.smithi067.stdout: "path": "/dir", 2024-03-24T00:46:27.798 INFO:teuthology.orchestra.run.smithi067.stdout: "tag": "4a999f56-60aa-4a25-bf45-1c9c537f5d9c", 2024-03-24T00:46:27.798 INFO:teuthology.orchestra.run.smithi067.stdout: "options": "recursive,repair" 2024-03-24T00:46:27.798 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:46:27.798 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:46:27.798 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:46:27.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.792+0000 7f4816337640 1 -- 172.21.15.67:0/2934861857 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f47f006bc00 msgr2=0x7f47f006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:27.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.792+0000 7f4816337640 1 --2- 172.21.15.67:0/2934861857 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f47f006bc00 0x7f47f006e0c0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f481015ec80 tx=0x7f47f8008040 comp rx=0 tx=0).stop 2024-03-24T00:46:27.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.792+0000 7f4816337640 1 -- 172.21.15.67:0/2934861857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48100b5fa0 msgr2=0x7f4810155360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:27.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.792+0000 7f4816337640 1 --2- 172.21.15.67:0/2934861857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48100b5fa0 0x7f4810155360 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f480000c430 tx=0x7f480000c8f0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.792+0000 7f4816337640 1 -- 172.21.15.67:0/2934861857 shutdown_connections 2024-03-24T00:46:27.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.792+0000 7f4816337640 1 --2- 172.21.15.67:0/2934861857 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f47f006bc00 0x7f47f006e0c0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.792+0000 7f4816337640 1 --2- 172.21.15.67:0/2934861857 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f48101623c0 0x7f4810163c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.792+0000 7f4816337640 1 --2- 172.21.15.67:0/2934861857 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f48100d0470 0x7f48100c9fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.792+0000 7f4816337640 1 --2- 172.21.15.67:0/2934861857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48100b5fa0 0x7f4810155360 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.792+0000 7f4816337640 1 -- 172.21.15.67:0/2934861857 >> 172.21.15.67:0/2934861857 conn(0x7f4810072da0 msgr2=0x7f48100b85d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:27.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.792+0000 7f4816337640 1 -- 172.21.15.67:0/2934861857 shutdown_connections 2024-03-24T00:46:27.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:27.792+0000 7f4816337640 1 -- 172.21.15.67:0/2934861857 wait complete. 2024-03-24T00:46:27.802 INFO:tasks.cephfs.filesystem:scrub status for tag:4a999f56-60aa-4a25-bf45-1c9c537f5d9c - {'path': '/dir', 'tag': '4a999f56-60aa-4a25-bf45-1c9c537f5d9c', 'options': 'recursive,repair'} 2024-03-24T00:46:46.864 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:46:46.867 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:46:57.803 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:46:57.804 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.6:0 scrub status 2024-03-24T00:46:57.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.945+0000 7f6807190640 1 Processor -- start 2024-03-24T00:46:57.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.945+0000 7f6807190640 1 -- start start 2024-03-24T00:46:57.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.946+0000 7f6807190640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f680015a500 0x7f680006f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:57.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.946+0000 7f6807190640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6800072a90 0x7f68000cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:57.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.946+0000 7f6807190640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68000b2b40 con 0x7f680006f570 2024-03-24T00:46:57.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.946+0000 7f6807190640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68000b2e20 con 0x7f680015a500 2024-03-24T00:46:57.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.946+0000 7f6807190640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68000b3100 con 0x7f6800072a90 2024-03-24T00:46:57.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.946+0000 7f680572d640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6800072a90 0x7f68000cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:57.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.946+0000 7f6804f2c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f680015a500 0x7f680006f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:57.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.946+0000 7f67fffff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f680006f570 0x7f6800072380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:54594/0 (socket says 172.21.15.67:54594) 2024-03-24T00:46:57.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.946+0000 7f67fffff640 1 -- 172.21.15.67:0/4266180796 learned_addr learned my addr 172.21.15.67:0/4266180796 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:46:57.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.947+0000 7f680572d640 1 -- 172.21.15.67:0/4266180796 >> v2:172.21.15.72:3300/0 conn(0x7f680015a500 msgr2=0x7f680006f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:57.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.947+0000 7f680572d640 1 --2- 172.21.15.67:0/4266180796 >> v2:172.21.15.72:3300/0 conn(0x7f680015a500 0x7f680006f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:57.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.947+0000 7f680572d640 1 -- 172.21.15.67:0/4266180796 >> v1:172.21.15.72:6789/0 conn(0x7f680006f570 legacy=0x7f6800072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:57.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.947+0000 7f680572d640 1 -- 172.21.15.67:0/4266180796 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68000b33e0 con 0x7f6800072a90 2024-03-24T00:46:57.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.947+0000 7f67ff7fe640 1 -- 172.21.15.67:0/4266180796 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 644906574 0 0) 0x7f68000b2b40 con 0x7f680006f570 2024-03-24T00:46:57.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.947+0000 7f6804f2c640 1 --2- 172.21.15.67:0/4266180796 >> v2:172.21.15.72:3300/0 conn(0x7f680015a500 0x7f680006f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:46:57.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.947+0000 7f680572d640 1 --2- 172.21.15.67:0/4266180796 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6800072a90 0x7f68000cd7a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f67f8009d90 tx=0x7f67f802be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9082704c852dfb63 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:57.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.948+0000 7f67ff7fe640 1 -- 172.21.15.67:0/4266180796 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f8039070 con 0x7f6800072a90 2024-03-24T00:46:57.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.948+0000 7f67ff7fe640 1 -- 172.21.15.67:0/4266180796 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67f802cbc0 con 0x7f6800072a90 2024-03-24T00:46:57.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.948+0000 7f67ff7fe640 1 -- 172.21.15.67:0/4266180796 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f8034440 con 0x7f6800072a90 2024-03-24T00:46:57.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.948+0000 7f6807190640 1 -- 172.21.15.67:0/4266180796 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6800072a90 msgr2=0x7f68000cd7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:57.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.948+0000 7f6807190640 1 --2- 172.21.15.67:0/4266180796 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6800072a90 0x7f68000cd7a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f67f8009d90 tx=0x7f67f802be10 comp rx=0 tx=0).stop 2024-03-24T00:46:57.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.948+0000 7f6807190640 1 -- 172.21.15.67:0/4266180796 shutdown_connections 2024-03-24T00:46:57.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.948+0000 7f6807190640 1 --2- 172.21.15.67:0/4266180796 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6800072a90 0x7f68000cd7a0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:57.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.948+0000 7f6807190640 1 --2- 172.21.15.67:0/4266180796 >> v2:172.21.15.72:3300/0 conn(0x7f680015a500 0x7f680006f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:57.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.948+0000 7f6807190640 1 -- 172.21.15.67:0/4266180796 >> 172.21.15.67:0/4266180796 conn(0x7f68000cf9e0 msgr2=0x7f68000cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:57.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.948+0000 7f6807190640 1 -- 172.21.15.67:0/4266180796 shutdown_connections 2024-03-24T00:46:57.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.948+0000 7f6807190640 1 -- 172.21.15.67:0/4266180796 wait complete. 2024-03-24T00:46:57.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.950+0000 7f6807190640 1 Processor -- start 2024-03-24T00:46:57.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.950+0000 7f6807190640 1 -- start start 2024-03-24T00:46:57.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.952+0000 7f6807190640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f680006f570 0x7f6800156280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:57.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.952+0000 7f6807190640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6800072a90 0x7f6800157360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:57.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.952+0000 7f6807190640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f680015a500 0x7f68001f3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:57.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.952+0000 7f6807190640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68000d06f0 con 0x7f680006f570 2024-03-24T00:46:57.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.952+0000 7f6807190640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f680015a8e0 con 0x7f680015a500 2024-03-24T00:46:57.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.952+0000 7f6807190640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68000d0870 con 0x7f6800072a90 2024-03-24T00:46:57.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.952+0000 7f6804f2c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f680006f570 0x7f6800156280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:57.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.952+0000 7f6804f2c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f680006f570 0x7f6800156280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50644/0 (socket says 172.21.15.67:50644) 2024-03-24T00:46:57.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.952+0000 7f6804f2c640 1 -- 172.21.15.67:0/2698095042 learned_addr learned my addr 172.21.15.67:0/2698095042 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:57.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.952+0000 7f680572d640 1 --2- 172.21.15.67:0/2698095042 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f680015a500 0x7f68001f3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:57.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.952+0000 7f67fffff640 1 --2- 172.21.15.67:0/2698095042 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6800072a90 0x7f6800157360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:57.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.952+0000 7f6804f2c640 1 -- 172.21.15.67:0/2698095042 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6800072a90 msgr2=0x7f6800157360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:57.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.952+0000 7f6804f2c640 1 --2- 172.21.15.67:0/2698095042 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6800072a90 0x7f6800157360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:57.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.952+0000 7f6804f2c640 1 -- 172.21.15.67:0/2698095042 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f680015a500 msgr2=0x7f68001f3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:57.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.952+0000 7f6804f2c640 1 --2- 172.21.15.67:0/2698095042 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f680015a500 0x7f68001f3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:57.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.952+0000 7f6804f2c640 1 -- 172.21.15.67:0/2698095042 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68001f4380 con 0x7f680006f570 2024-03-24T00:46:57.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.952+0000 7f67fffff640 1 --2- 172.21.15.67:0/2698095042 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6800072a90 0x7f6800157360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:46:57.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.952+0000 7f6804f2c640 1 --2- 172.21.15.67:0/2698095042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f680006f570 0x7f6800156280 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f67f400e860 tx=0x7f67f400ed20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:57.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.952+0000 7f67fd7fa640 1 -- 172.21.15.67:0/2698095042 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f4016070 con 0x7f680006f570 2024-03-24T00:46:57.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.952+0000 7f67fd7fa640 1 -- 172.21.15.67:0/2698095042 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67f400cc40 con 0x7f680006f570 2024-03-24T00:46:57.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.952+0000 7f6807190640 1 -- 172.21.15.67:0/2698095042 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68001f4620 con 0x7f680006f570 2024-03-24T00:46:57.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.952+0000 7f67fd7fa640 1 -- 172.21.15.67:0/2698095042 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f400b040 con 0x7f680006f570 2024-03-24T00:46:57.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.952+0000 7f6807190640 1 -- 172.21.15.67:0/2698095042 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68001f4be0 con 0x7f680006f570 2024-03-24T00:46:57.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.954+0000 7f67fd7fa640 1 -- 172.21.15.67:0/2698095042 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f67f4019070 con 0x7f680006f570 2024-03-24T00:46:57.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.955+0000 7f67fd7fa640 1 --2- 172.21.15.67:0/2698095042 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f67e406bbd0 0x7f67e406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:57.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.955+0000 7f67fd7fa640 1 -- 172.21.15.67:0/2698095042 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f67f408ac20 con 0x7f680006f570 2024-03-24T00:46:57.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.955+0000 7f67fffff640 1 --2- 172.21.15.67:0/2698095042 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f67e406bbd0 0x7f67e406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:57.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.955+0000 7f6809d78740 1 Processor -- start 2024-03-24T00:46:57.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.955+0000 7f6809d78740 1 -- start start 2024-03-24T00:46:57.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.956+0000 7f67fffff640 1 --2- 172.21.15.67:0/2698095042 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f67e406bbd0 0x7f67e406e090 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f6800155510 tx=0x7f67ec009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:57.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.956+0000 7f6809d78740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5646215f7e50 0x5646215cadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:57.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.956+0000 7f6809d78740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5646215f8230 0x5646215fabe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:57.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.956+0000 7f6809d78740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5646215cb330 0x5646215fb120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:57.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.956+0000 7f6809d78740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646215353d0 con 0x5646215cb330 2024-03-24T00:46:57.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.956+0000 7f6809d78740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646214ddad0 con 0x5646215f8230 2024-03-24T00:46:57.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.956+0000 7f6809d78740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x564621531470 con 0x5646215f7e50 2024-03-24T00:46:57.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.956+0000 7f680572d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5646215cb330 0x5646215fb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:57.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.956+0000 7f680572d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5646215cb330 0x5646215fb120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50650/0 (socket says 172.21.15.67:50650) 2024-03-24T00:46:57.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.956+0000 7f680572d640 1 -- 172.21.15.67:0/985226334 learned_addr learned my addr 172.21.15.67:0/985226334 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:57.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.956+0000 7f67fffff640 1 --2- 172.21.15.67:0/985226334 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5646215f8230 0x5646215fabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:57.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.956+0000 7f6804f2c640 1 --2- 172.21.15.67:0/985226334 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5646215f7e50 0x5646215cadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:57.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.956+0000 7f680572d640 1 -- 172.21.15.67:0/985226334 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5646215f7e50 msgr2=0x5646215cadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:57.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.956+0000 7f680572d640 1 --2- 172.21.15.67:0/985226334 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5646215f7e50 0x5646215cadf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:57.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.956+0000 7f680572d640 1 -- 172.21.15.67:0/985226334 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5646215f8230 msgr2=0x5646215fabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:57.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.956+0000 7f680572d640 1 --2- 172.21.15.67:0/985226334 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5646215f8230 0x5646215fabe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:57.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.956+0000 7f680572d640 1 -- 172.21.15.67:0/985226334 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5646213137d0 con 0x5646215cb330 2024-03-24T00:46:57.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.956+0000 7f6804f2c640 1 --2- 172.21.15.67:0/985226334 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5646215f7e50 0x5646215cadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:46:57.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.957+0000 7f680572d640 1 --2- 172.21.15.67:0/985226334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5646215cb330 0x5646215fb120 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f67f802c420 tx=0x7f67f802e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb102427552e7d95 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:57.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.957+0000 7f67ea7fc640 1 -- 172.21.15.67:0/985226334 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f8039040 con 0x5646215cb330 2024-03-24T00:46:57.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.957+0000 7f67ea7fc640 1 -- 172.21.15.67:0/985226334 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67f8044070 con 0x5646215cb330 2024-03-24T00:46:57.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.957+0000 7f67ea7fc640 1 -- 172.21.15.67:0/985226334 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f8003bd0 con 0x5646215cb330 2024-03-24T00:46:57.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.957+0000 7f6809d78740 1 -- 172.21.15.67:0/985226334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5646215cb330 msgr2=0x5646215fb120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:57.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.957+0000 7f6809d78740 1 --2- 172.21.15.67:0/985226334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5646215cb330 0x5646215fb120 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f67f802c420 tx=0x7f67f802e000 comp rx=0 tx=0).stop 2024-03-24T00:46:57.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.957+0000 7f6809d78740 1 -- 172.21.15.67:0/985226334 shutdown_connections 2024-03-24T00:46:57.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.957+0000 7f6809d78740 1 --2- 172.21.15.67:0/985226334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5646215cb330 0x5646215fb120 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:57.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.957+0000 7f6809d78740 1 --2- 172.21.15.67:0/985226334 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5646215f8230 0x5646215fabe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:57.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.957+0000 7f6809d78740 1 --2- 172.21.15.67:0/985226334 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5646215f7e50 0x5646215cadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:57.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.957+0000 7f6809d78740 1 -- 172.21.15.67:0/985226334 >> 172.21.15.67:0/985226334 conn(0x5646215cd060 msgr2=0x564621615be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:57.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.957+0000 7f6809d78740 1 -- 172.21.15.67:0/985226334 shutdown_connections 2024-03-24T00:46:57.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.957+0000 7f6809d78740 1 -- 172.21.15.67:0/985226334 wait complete. 2024-03-24T00:46:57.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.958+0000 7f6809d78740 1 -- start start 2024-03-24T00:46:57.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.958+0000 7f6809d78740 1 Processor -- start 2024-03-24T00:46:57.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.959+0000 7f6809d78740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5646215f7e50 0x5646215df0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:57.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.959+0000 7f6809d78740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5646215f8230 0x5646215f6050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:57.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.959+0000 7f6809d78740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5646215f4040 0x5646215f44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:57.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.959+0000 7f6809d78740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646215cf700 con 0x5646215f4040 2024-03-24T00:46:57.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.959+0000 7f6809d78740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646214f1060 con 0x5646215f8230 2024-03-24T00:46:57.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.959+0000 7f6809d78740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5646215d1f40 con 0x5646215f7e50 2024-03-24T00:46:57.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.959+0000 7f6809d78740 10 client.0 _ll_register_callbacks cb 0x5646214d36a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:46:57.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.959+0000 7f680572d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5646215f4040 0x5646215f44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:57.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.959+0000 7f680572d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5646215f4040 0x5646215f44c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50654/0 (socket says 172.21.15.67:50654) 2024-03-24T00:46:57.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.959+0000 7f680572d640 1 -- 172.21.15.67:0/2106784971 learned_addr learned my addr 172.21.15.67:0/2106784971 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:57.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.959+0000 7f6804f2c640 1 --2- 172.21.15.67:0/2106784971 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5646215f7e50 0x5646215df0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:57.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.959+0000 7f67fffff640 1 --2- 172.21.15.67:0/2106784971 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5646215f8230 0x5646215f6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:57.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.959+0000 7f680572d640 1 -- 172.21.15.67:0/2106784971 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5646215f7e50 msgr2=0x5646215df0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:57.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.959+0000 7f680572d640 1 --2- 172.21.15.67:0/2106784971 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5646215f7e50 0x5646215df0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:57.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.959+0000 7f680572d640 1 -- 172.21.15.67:0/2106784971 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5646215f8230 msgr2=0x5646215f6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:57.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.959+0000 7f680572d640 1 --2- 172.21.15.67:0/2106784971 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5646215f8230 0x5646215f6050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:57.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.959+0000 7f680572d640 1 -- 172.21.15.67:0/2106784971 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5646212493f0 con 0x5646215f4040 2024-03-24T00:46:57.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.959+0000 7f680572d640 1 --2- 172.21.15.67:0/2106784971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5646215f4040 0x5646215f44c0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f67f8009d90 tx=0x7f67f8004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df718eecb2fa0d32 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:57.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.959+0000 7f67df7fe640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:46:57.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.960+0000 7f67df7fe640 1 -- 172.21.15.67:0/2106784971 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f8004040 con 0x5646215f4040 2024-03-24T00:46:57.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.960+0000 7f67df7fe640 1 -- 172.21.15.67:0/2106784971 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f67f80041e0 con 0x5646215f4040 2024-03-24T00:46:57.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.960+0000 7f6809d78740 1 -- 172.21.15.67:0/2106784971 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f67f8044070 con 0x5646215f4040 2024-03-24T00:46:57.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.960+0000 7f67df7fe640 1 -- 172.21.15.67:0/2106784971 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) v2 ==== 24+0+0 (secure 0 0 0) 0x7f67f805a050 con 0x5646215f4040 2024-03-24T00:46:57.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.960+0000 7f6809d78740 10 client.8313 fetch_fsmap learned FSMap version 66 2024-03-24T00:46:57.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.960+0000 7f6809d78740 1 -- 172.21.15.67:0/2106784971 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=2+}) v3 -- 0x5646212090f0 con 0x5646215f4040 2024-03-24T00:46:57.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.960+0000 7f67df7fe640 1 -- 172.21.15.67:0/2106784971 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67f8044070 con 0x5646215f4040 2024-03-24T00:46:57.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.960+0000 7f67df7fe640 1 -- 172.21.15.67:0/2106784971 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 66) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f67f805d020 con 0x5646215f4040 2024-03-24T00:46:57.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.960+0000 7f6809d78740 10 client.8313 fetch_fsmap finished waiting for FSMap version 66 2024-03-24T00:46:57.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.960+0000 7f6809d78740 10 client.8313 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-03-24T00:46:57.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.960+0000 7f6809d78740 20 client.8313 populate_metadata read hostname 'smithi067' 2024-03-24T00:46:57.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.960+0000 7f6809d78740 1 --2- 172.21.15.67:0/2106784971 >> [v2:172.21.15.67:6832/3311719072,v1:172.21.15.67:6833/3311719072] conn(0x564621629840 0x564621629c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:57.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.960+0000 7f6809d78740 4 client.8313 mds_command: new command op to 8163 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:46:57.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.960+0000 7f6809d78740 1 -- 172.21.15.67:0/2106784971 --> [v2:172.21.15.67:6832/3311719072,v1:172.21.15.67:6833/3311719072] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564621542910 con 0x564621629840 2024-03-24T00:46:57.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.960+0000 7f6804f2c640 1 --2- 172.21.15.67:0/2106784971 >> [v2:172.21.15.67:6832/3311719072,v1:172.21.15.67:6833/3311719072] conn(0x564621629840 0x564621629c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:57.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.961+0000 7f67df7fe640 10 client.8313 ms_handle_connect on v2:172.21.15.67:6832/3311719072 2024-03-24T00:46:57.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.961+0000 7f6804f2c640 1 --2- 172.21.15.67:0/2106784971 >> [v2:172.21.15.67:6832/3311719072,v1:172.21.15.67:6833/3311719072] conn(0x564621629840 0x564621629c50 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad23e04b682878d server_cookie=73700a7633bf6869 in_seq=0 out_seq=0 2024-03-24T00:46:57.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.961+0000 7f67df7fe640 1 -- 172.21.15.67:0/2106784971 <== mds.0 v2:172.21.15.67:6832/3311719072 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564621542910 con 0x564621629840 2024-03-24T00:46:57.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.961+0000 7f67df7fe640 10 client.8313 handle_command_reply: tid=0 2024-03-24T00:46:57.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.961+0000 7f6809d78740 1 client.8313 shutdown 2024-03-24T00:46:57.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.961+0000 7f6809d78740 1 -- 172.21.15.67:0/2106784971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5646215f4040 msgr2=0x5646215f44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:57.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.961+0000 7f6809d78740 1 --2- 172.21.15.67:0/2106784971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5646215f4040 0x5646215f44c0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f67f8009d90 tx=0x7f67f8004ab0 comp rx=0 tx=0).stop 2024-03-24T00:46:57.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.961+0000 7f6809d78740 1 -- 172.21.15.67:0/2106784971 shutdown_connections 2024-03-24T00:46:57.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.961+0000 7f6809d78740 1 --2- 172.21.15.67:0/2106784971 >> [v2:172.21.15.67:6832/3311719072,v1:172.21.15.67:6833/3311719072] conn(0x564621629840 0x564621629c50 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:57.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.961+0000 7f6809d78740 1 --2- 172.21.15.67:0/2106784971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5646215f4040 0x5646215f44c0 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:57.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.961+0000 7f6809d78740 1 --2- 172.21.15.67:0/2106784971 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5646215f8230 0x5646215f6050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:57.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.961+0000 7f67df7fe640 0 client.8313 ms_handle_reset on v2:172.21.15.67:6832/3311719072 2024-03-24T00:46:57.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.961+0000 7f6809d78740 1 --2- 172.21.15.67:0/2106784971 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5646215f7e50 0x5646215df0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:57.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.961+0000 7f6809d78740 1 -- 172.21.15.67:0/2106784971 >> 172.21.15.67:0/2106784971 conn(0x5646215cb330 msgr2=0x564621619060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:57.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.961+0000 7f6809d78740 1 -- 172.21.15.67:0/2106784971 shutdown_connections 2024-03-24T00:46:57.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.961+0000 7f6809d78740 1 -- 172.21.15.67:0/2106784971 wait complete. 2024-03-24T00:46:57.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.962+0000 7f6809d78740 20 client.8313 trim_cache size 0 max 16384 2024-03-24T00:46:57.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.978+0000 7f6809d78740 1 Processor -- start 2024-03-24T00:46:57.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.978+0000 7f6809d78740 1 -- start start 2024-03-24T00:46:57.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.978+0000 7f6809d78740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5646215f4040 0x564621618db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:57.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.978+0000 7f6809d78740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5646215f7e50 0x5646216192f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:57.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.978+0000 7f6809d78740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5646215f8230 0x5646215e0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:57.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.978+0000 7f6809d78740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6800200c80 con 0x5646215f4040 2024-03-24T00:46:57.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.978+0000 7f6809d78740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68001ffd70 con 0x5646215f8230 2024-03-24T00:46:57.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.978+0000 7f6809d78740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68001f7e10 con 0x5646215f7e50 2024-03-24T00:46:57.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.978+0000 7f6804f2c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5646215f4040 0x564621618db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:57.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.978+0000 7f6804f2c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5646215f4040 0x564621618db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50666/0 (socket says 172.21.15.67:50666) 2024-03-24T00:46:57.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.978+0000 7f6804f2c640 1 -- 172.21.15.67:0/1059728798 learned_addr learned my addr 172.21.15.67:0/1059728798 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:57.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.979+0000 7f6804f2c640 1 -- 172.21.15.67:0/1059728798 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5646215f7e50 msgr2=0x5646216192f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:57.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.979+0000 7f6804f2c640 1 --2- 172.21.15.67:0/1059728798 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5646215f7e50 0x5646216192f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:57.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.979+0000 7f6804f2c640 1 -- 172.21.15.67:0/1059728798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5646215f8230 msgr2=0x5646215e0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:57.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.979+0000 7f6804f2c640 1 --2- 172.21.15.67:0/1059728798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5646215f8230 0x5646215e0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:57.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.979+0000 7f6804f2c640 1 -- 172.21.15.67:0/1059728798 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5646215f5b30 con 0x5646215f4040 2024-03-24T00:46:57.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.979+0000 7f6804f2c640 1 --2- 172.21.15.67:0/1059728798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5646215f4040 0x564621618db0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f67f40551c0 tx=0x7f67f405f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50995499c29e9bca server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:57.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.979+0000 7f67e9ffb640 1 -- 172.21.15.67:0/1059728798 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f4075070 con 0x5646215f4040 2024-03-24T00:46:57.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.979+0000 7f67e9ffb640 1 -- 172.21.15.67:0/1059728798 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67f4086070 con 0x5646215f4040 2024-03-24T00:46:57.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.979+0000 7f67e9ffb640 1 -- 172.21.15.67:0/1059728798 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f4020040 con 0x5646215f4040 2024-03-24T00:46:57.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.979+0000 7f6809d78740 1 -- 172.21.15.67:0/1059728798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5646215f4040 msgr2=0x564621618db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:57.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.979+0000 7f6809d78740 1 --2- 172.21.15.67:0/1059728798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5646215f4040 0x564621618db0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f67f40551c0 tx=0x7f67f405f040 comp rx=0 tx=0).stop 2024-03-24T00:46:57.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.979+0000 7f6809d78740 1 -- 172.21.15.67:0/1059728798 shutdown_connections 2024-03-24T00:46:57.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.979+0000 7f6809d78740 1 --2- 172.21.15.67:0/1059728798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5646215f8230 0x5646215e0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:57.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.980+0000 7f6809d78740 1 --2- 172.21.15.67:0/1059728798 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5646215f7e50 0x5646216192f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:57.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.980+0000 7f6809d78740 1 --2- 172.21.15.67:0/1059728798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5646215f4040 0x564621618db0 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:57.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.980+0000 7f6809d78740 1 -- 172.21.15.67:0/1059728798 >> 172.21.15.67:0/1059728798 conn(0x5646215cb330 msgr2=0x5646215f6050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:57.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.980+0000 7f6809d78740 1 -- 172.21.15.67:0/1059728798 shutdown_connections 2024-03-24T00:46:57.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.980+0000 7f6809d78740 1 -- 172.21.15.67:0/1059728798 wait complete. 2024-03-24T00:46:57.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.980+0000 7f6809d78740 1 -- start start 2024-03-24T00:46:57.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.980+0000 7f6809d78740 1 Processor -- start 2024-03-24T00:46:57.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.981+0000 7f6809d78740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5646215f4040 0x5646215fc470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:57.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.981+0000 7f6809d78740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5646215f7e50 0x5646215e0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:57.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.981+0000 7f6809d78740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5646215f8230 0x5646215e1440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:57.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.981+0000 7f6809d78740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646215cc7c0 con 0x5646215f4040 2024-03-24T00:46:57.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.981+0000 7f6809d78740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68001ff7a0 con 0x5646215f7e50 2024-03-24T00:46:57.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.981+0000 7f6809d78740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5646215cc0c0 con 0x5646215f8230 2024-03-24T00:46:57.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.981+0000 7f6809d78740 10 client.0 _ll_register_callbacks cb 0x7f67f400a170 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:46:57.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.981+0000 7f6804f2c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5646215f4040 0x5646215fc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:57.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.981+0000 7f6804f2c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5646215f4040 0x5646215fc470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50678/0 (socket says 172.21.15.67:50678) 2024-03-24T00:46:57.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.981+0000 7f6804f2c640 1 -- 172.21.15.67:0/111019292 learned_addr learned my addr 172.21.15.67:0/111019292 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:57.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.981+0000 7f680572d640 1 --2- 172.21.15.67:0/111019292 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5646215f8230 0x5646215e1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:57.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.981+0000 7f67fffff640 1 --2- 172.21.15.67:0/111019292 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5646215f7e50 0x5646215e0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:57.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.981+0000 7f6804f2c640 1 -- 172.21.15.67:0/111019292 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5646215f8230 msgr2=0x5646215e1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:57.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.981+0000 7f6804f2c640 1 --2- 172.21.15.67:0/111019292 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5646215f8230 0x5646215e1440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:57.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.981+0000 7f6804f2c640 1 -- 172.21.15.67:0/111019292 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5646215f7e50 msgr2=0x5646215e0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:57.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.981+0000 7f6804f2c640 1 --2- 172.21.15.67:0/111019292 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5646215f7e50 0x5646215e0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:57.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.981+0000 7f6804f2c640 1 -- 172.21.15.67:0/111019292 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5646215f4560 con 0x5646215f4040 2024-03-24T00:46:57.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.982+0000 7f6804f2c640 1 --2- 172.21.15.67:0/111019292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5646215f4040 0x5646215fc470 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f67f4088ef0 tx=0x7f67f8031000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6724cd5225b601bf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:57.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.982+0000 7f67df7fe640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:46:57.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.982+0000 7f67df7fe640 1 -- 172.21.15.67:0/111019292 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f407c060 con 0x5646215f4040 2024-03-24T00:46:57.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.982+0000 7f67df7fe640 1 -- 172.21.15.67:0/111019292 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f67f40553d0 con 0x5646215f4040 2024-03-24T00:46:57.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.982+0000 7f6809d78740 1 -- 172.21.15.67:0/111019292 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f67f4086070 con 0x5646215f4040 2024-03-24T00:46:57.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.982+0000 7f67df7fe640 1 -- 172.21.15.67:0/111019292 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) v2 ==== 24+0+0 (secure 0 0 0) 0x7f67f4082050 con 0x5646215f4040 2024-03-24T00:46:57.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.982+0000 7f6809d78740 10 client.8319 fetch_fsmap learned FSMap version 66 2024-03-24T00:46:57.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.982+0000 7f6809d78740 1 -- 172.21.15.67:0/111019292 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=2+}) v3 -- 0x5646215fbb60 con 0x5646215f4040 2024-03-24T00:46:57.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.982+0000 7f67df7fe640 1 -- 172.21.15.67:0/111019292 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67f4086070 con 0x5646215f4040 2024-03-24T00:46:57.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.982+0000 7f67df7fe640 1 -- 172.21.15.67:0/111019292 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 66) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f67f40581b0 con 0x5646215f4040 2024-03-24T00:46:57.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.982+0000 7f6809d78740 10 client.8319 fetch_fsmap finished waiting for FSMap version 66 2024-03-24T00:46:57.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.982+0000 7f6809d78740 10 client.8319 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-03-24T00:46:57.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.982+0000 7f6809d78740 20 client.8319 populate_metadata read hostname 'smithi067' 2024-03-24T00:46:57.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.982+0000 7f6809d78740 1 --2- 172.21.15.67:0/111019292 >> [v2:172.21.15.67:6832/3311719072,v1:172.21.15.67:6833/3311719072] conn(0x5646215fc9b0 0x5646215c9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:57.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.982+0000 7f6809d78740 4 client.8319 mds_command: new command op to 8163 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:46:57.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.982+0000 7f6809d78740 1 -- 172.21.15.67:0/111019292 --> [v2:172.21.15.67:6832/3311719072,v1:172.21.15.67:6833/3311719072] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56462153cc60 con 0x5646215fc9b0 2024-03-24T00:46:57.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.982+0000 7f67fffff640 1 --2- 172.21.15.67:0/111019292 >> [v2:172.21.15.67:6832/3311719072,v1:172.21.15.67:6833/3311719072] conn(0x5646215fc9b0 0x5646215c9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:57.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.983+0000 7f67df7fe640 10 client.8319 ms_handle_connect on v2:172.21.15.67:6832/3311719072 2024-03-24T00:46:57.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.983+0000 7f67fffff640 1 --2- 172.21.15.67:0/111019292 >> [v2:172.21.15.67:6832/3311719072,v1:172.21.15.67:6833/3311719072] conn(0x5646215fc9b0 0x5646215c9ca0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d3497eefaf4f059 server_cookie=d4bada771bd17618 in_seq=0 out_seq=0 2024-03-24T00:46:57.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.983+0000 7f67df7fe640 1 -- 172.21.15.67:0/111019292 <== mds.0 v2:172.21.15.67:6832/3311719072 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x56462153cc60 con 0x5646215fc9b0 2024-03-24T00:46:57.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.983+0000 7f67df7fe640 10 client.8319 handle_command_reply: tid=0 2024-03-24T00:46:57.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.983+0000 7f6809d78740 1 client.8319 shutdown 2024-03-24T00:46:57.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.983+0000 7f6809d78740 1 -- 172.21.15.67:0/111019292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5646215f4040 msgr2=0x5646215fc470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:57.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.983+0000 7f6809d78740 1 --2- 172.21.15.67:0/111019292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5646215f4040 0x5646215fc470 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f67f4088ef0 tx=0x7f67f8031000 comp rx=0 tx=0).stop 2024-03-24T00:46:57.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.983+0000 7f6809d78740 1 -- 172.21.15.67:0/111019292 shutdown_connections 2024-03-24T00:46:57.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.983+0000 7f6809d78740 1 --2- 172.21.15.67:0/111019292 >> [v2:172.21.15.67:6832/3311719072,v1:172.21.15.67:6833/3311719072] conn(0x5646215fc9b0 0x5646215c9ca0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:57.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.983+0000 7f6809d78740 1 --2- 172.21.15.67:0/111019292 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5646215f8230 0x5646215e1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:57.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.983+0000 7f6809d78740 1 --2- 172.21.15.67:0/111019292 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5646215f7e50 0x5646215e0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:57.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.983+0000 7f67df7fe640 0 client.8319 ms_handle_reset on v2:172.21.15.67:6832/3311719072 2024-03-24T00:46:57.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.983+0000 7f6809d78740 1 --2- 172.21.15.67:0/111019292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5646215f4040 0x5646215fc470 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:57.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.983+0000 7f6809d78740 1 -- 172.21.15.67:0/111019292 >> 172.21.15.67:0/111019292 conn(0x5646215cb330 msgr2=0x5646215f0cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:57.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.983+0000 7f6809d78740 1 -- 172.21.15.67:0/111019292 shutdown_connections 2024-03-24T00:46:57.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.983+0000 7f6809d78740 1 -- 172.21.15.67:0/111019292 wait complete. 2024-03-24T00:46:57.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.983+0000 7f6809d78740 20 client.8319 trim_cache size 0 max 16384 2024-03-24T00:46:57.990 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:46:57.990 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "no active scrubs running", 2024-03-24T00:46:57.990 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": {} 2024-03-24T00:46:57.990 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:46:57.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.984+0000 7f6807190640 1 -- 172.21.15.67:0/2698095042 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f67e406bbd0 msgr2=0x7f67e406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:57.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.984+0000 7f6807190640 1 --2- 172.21.15.67:0/2698095042 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f67e406bbd0 0x7f67e406e090 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f6800155510 tx=0x7f67ec009040 comp rx=0 tx=0).stop 2024-03-24T00:46:57.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.984+0000 7f6807190640 1 -- 172.21.15.67:0/2698095042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f680006f570 msgr2=0x7f6800156280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:57.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.984+0000 7f6807190640 1 --2- 172.21.15.67:0/2698095042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f680006f570 0x7f6800156280 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f67f400e860 tx=0x7f67f400ed20 comp rx=0 tx=0).stop 2024-03-24T00:46:57.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.984+0000 7f6807190640 1 -- 172.21.15.67:0/2698095042 shutdown_connections 2024-03-24T00:46:57.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.984+0000 7f6807190640 1 --2- 172.21.15.67:0/2698095042 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f67e406bbd0 0x7f67e406e090 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:57.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.984+0000 7f6807190640 1 --2- 172.21.15.67:0/2698095042 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f680015a500 0x7f68001f3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:57.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.984+0000 7f6807190640 1 --2- 172.21.15.67:0/2698095042 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6800072a90 0x7f6800157360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:57.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.984+0000 7f6807190640 1 --2- 172.21.15.67:0/2698095042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f680006f570 0x7f6800156280 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:57.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.984+0000 7f6807190640 1 -- 172.21.15.67:0/2698095042 >> 172.21.15.67:0/2698095042 conn(0x7f68000cf9e0 msgr2=0x7f68001476c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:57.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.984+0000 7f6807190640 1 -- 172.21.15.67:0/2698095042 shutdown_connections 2024-03-24T00:46:57.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:57.984+0000 7f6807190640 1 -- 172.21.15.67:0/2698095042 wait complete. 2024-03-24T00:46:57.995 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T00:46:57.996 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:46:58.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.096+0000 7fc3f464b640 1 Processor -- start 2024-03-24T00:46:58.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.096+0000 7fc3f464b640 1 -- start start 2024-03-24T00:46:58.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.097+0000 7fc3f464b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc3ec0cfc70 0x7fc3ec0ce220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:58.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.097+0000 7fc3f464b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc3ec0ce760 0x7fc3ec0b1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:58.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.097+0000 7fc3f464b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3ec0ccfe0 con 0x7fc3ec15bf70 2024-03-24T00:46:58.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.097+0000 7fc3f464b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3ec0ccda0 con 0x7fc3ec0ce760 2024-03-24T00:46:58.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.097+0000 7fc3f464b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ec0cd2c0 con 0x7fc3ec0cfc70 2024-03-24T00:46:58.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.097+0000 7fc3f2be8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc3ec0ce760 0x7fc3ec0b1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:58.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.097+0000 7fc3f23e7640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc3ec15bf70 0x7fc3ec0cc690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44726/0 (socket says 172.21.15.67:44726) 2024-03-24T00:46:58.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.097+0000 7fc3f2be8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc3ec0ce760 0x7fc3ec0b1cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50688/0 (socket says 172.21.15.67:50688) 2024-03-24T00:46:58.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.097+0000 7fc3f23e7640 1 -- 172.21.15.67:0/3534970501 learned_addr learned my addr 172.21.15.67:0/3534970501 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:46:58.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.098+0000 7fc3f2be8640 1 -- 172.21.15.67:0/3534970501 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc3ec0cfc70 msgr2=0x7fc3ec0ce220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:58.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.098+0000 7fc3f2be8640 1 --2- 172.21.15.67:0/3534970501 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc3ec0cfc70 0x7fc3ec0ce220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.098+0000 7fc3f2be8640 1 -- 172.21.15.67:0/3534970501 >> v1:172.21.15.67:6789/0 conn(0x7fc3ec15bf70 legacy=0x7fc3ec0cc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:58.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.098+0000 7fc3f13e5640 1 -- 172.21.15.67:0/3534970501 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2636520614 0 0) 0x7fc3ec0ccfe0 con 0x7fc3ec15bf70 2024-03-24T00:46:58.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.098+0000 7fc3f2be8640 1 -- 172.21.15.67:0/3534970501 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3ec0cd5a0 con 0x7fc3ec0ce760 2024-03-24T00:46:58.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.099+0000 7fc3f2be8640 1 --2- 172.21.15.67:0/3534970501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3ec0ce760 0x7fc3ec0b1cd0 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7fc3e4009d90 tx=0x7fc3e402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c2b8c376d5bead server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:58.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.099+0000 7fc3f13e5640 1 -- 172.21.15.67:0/3534970501 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e4039070 con 0x7fc3ec0ce760 2024-03-24T00:46:58.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.099+0000 7fc3f13e5640 1 -- 172.21.15.67:0/3534970501 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3e402cc50 con 0x7fc3ec0ce760 2024-03-24T00:46:58.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.099+0000 7fc3f13e5640 1 -- 172.21.15.67:0/3534970501 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e4034440 con 0x7fc3ec0ce760 2024-03-24T00:46:58.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.099+0000 7fc3f464b640 1 -- 172.21.15.67:0/3534970501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3ec0ce760 msgr2=0x7fc3ec0b1cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:58.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.099+0000 7fc3f464b640 1 --2- 172.21.15.67:0/3534970501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3ec0ce760 0x7fc3ec0b1cd0 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7fc3e4009d90 tx=0x7fc3e402bea0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.099+0000 7fc3f464b640 1 -- 172.21.15.67:0/3534970501 shutdown_connections 2024-03-24T00:46:58.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.099+0000 7fc3f464b640 1 --2- 172.21.15.67:0/3534970501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3ec0ce760 0x7fc3ec0b1cd0 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.099+0000 7fc3f464b640 1 --2- 172.21.15.67:0/3534970501 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc3ec0cfc70 0x7fc3ec0ce220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.099+0000 7fc3f464b640 1 -- 172.21.15.67:0/3534970501 >> 172.21.15.67:0/3534970501 conn(0x7fc3ec0d1bf0 msgr2=0x7fc3ec0d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:58.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.099+0000 7fc3f464b640 1 -- 172.21.15.67:0/3534970501 shutdown_connections 2024-03-24T00:46:58.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.099+0000 7fc3f464b640 1 -- 172.21.15.67:0/3534970501 wait complete. 2024-03-24T00:46:58.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.101+0000 7fc3f464b640 1 Processor -- start 2024-03-24T00:46:58.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.101+0000 7fc3f464b640 1 -- start start 2024-03-24T00:46:58.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.101+0000 7fc3f464b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3ec0ce760 0x7fc3ec0cb9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:58.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.101+0000 7fc3f464b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3ec0cfc70 0x7fc3ec0c98f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:58.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.101+0000 7fc3f464b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc3ec15bf70 0x7fc3ec0b5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:58.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.101+0000 7fc3f464b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ec0b2210 con 0x7fc3ec0cfc70 2024-03-24T00:46:58.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.101+0000 7fc3f464b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ec15c350 con 0x7fc3ec0ce760 2024-03-24T00:46:58.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.101+0000 7fc3f464b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ec0b2390 con 0x7fc3ec15bf70 2024-03-24T00:46:58.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.101+0000 7fc3f1be6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3ec0cfc70 0x7fc3ec0c98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:58.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.102+0000 7fc3f1be6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3ec0cfc70 0x7fc3ec0c98f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50692/0 (socket says 172.21.15.67:50692) 2024-03-24T00:46:58.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.102+0000 7fc3f1be6640 1 -- 172.21.15.67:0/2723777791 learned_addr learned my addr 172.21.15.67:0/2723777791 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:58.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.102+0000 7fc3f23e7640 1 --2- 172.21.15.67:0/2723777791 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3ec0ce760 0x7fc3ec0cb9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:58.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.102+0000 7fc3f2be8640 1 --2- 172.21.15.67:0/2723777791 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc3ec15bf70 0x7fc3ec0b5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:58.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.102+0000 7fc3f1be6640 1 -- 172.21.15.67:0/2723777791 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc3ec15bf70 msgr2=0x7fc3ec0b5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:58.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.102+0000 7fc3f1be6640 1 --2- 172.21.15.67:0/2723777791 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc3ec15bf70 0x7fc3ec0b5f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.102+0000 7fc3f1be6640 1 -- 172.21.15.67:0/2723777791 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3ec0ce760 msgr2=0x7fc3ec0cb9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:58.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.102+0000 7fc3f1be6640 1 --2- 172.21.15.67:0/2723777791 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3ec0ce760 0x7fc3ec0cb9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.102+0000 7fc3f1be6640 1 -- 172.21.15.67:0/2723777791 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3ec06c360 con 0x7fc3ec0cfc70 2024-03-24T00:46:58.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.102+0000 7fc3f23e7640 1 --2- 172.21.15.67:0/2723777791 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3ec0ce760 0x7fc3ec0cb9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:46:58.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.102+0000 7fc3f1be6640 1 --2- 172.21.15.67:0/2723777791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3ec0cfc70 0x7fc3ec0c98f0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc00b4a0 tx=0x7fc3dc00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:58.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.102+0000 7fc3e2ffd640 1 -- 172.21.15.67:0/2723777791 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc014070 con 0x7fc3ec0cfc70 2024-03-24T00:46:58.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.102+0000 7fc3e2ffd640 1 -- 172.21.15.67:0/2723777791 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3dc010610 con 0x7fc3ec0cfc70 2024-03-24T00:46:58.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.102+0000 7fc3e2ffd640 1 -- 172.21.15.67:0/2723777791 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc010a00 con 0x7fc3ec0cfc70 2024-03-24T00:46:58.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.102+0000 7fc3f464b640 1 -- 172.21.15.67:0/2723777791 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3ec06c640 con 0x7fc3ec0cfc70 2024-03-24T00:46:58.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.102+0000 7fc3f464b640 1 -- 172.21.15.67:0/2723777791 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3ec06cc00 con 0x7fc3ec0cfc70 2024-03-24T00:46:58.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.103+0000 7fc3f464b640 1 -- 172.21.15.67:0/2723777791 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3ec04f400 con 0x7fc3ec0cfc70 2024-03-24T00:46:58.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.104+0000 7fc3e2ffd640 1 -- 172.21.15.67:0/2723777791 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc3dc017070 con 0x7fc3ec0cfc70 2024-03-24T00:46:58.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.104+0000 7fc3e2ffd640 1 --2- 172.21.15.67:0/2723777791 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc3d006bbd0 0x7fc3d006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:58.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.104+0000 7fc3e2ffd640 1 -- 172.21.15.67:0/2723777791 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6389+0+0 (secure 0 0 0) 0x7fc3dc08a810 con 0x7fc3ec0cfc70 2024-03-24T00:46:58.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.104+0000 7fc3f23e7640 1 --2- 172.21.15.67:0/2723777791 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc3d006bbd0 0x7fc3d006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:58.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.105+0000 7fc3f23e7640 1 --2- 172.21.15.67:0/2723777791 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc3d006bbd0 0x7fc3d006e090 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fc3e8000f60 tx=0x7fc3e8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:58.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.113+0000 7fc3e2ffd640 1 -- 172.21.15.67:0/2723777791 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc3dc058fd0 con 0x7fc3ec0cfc70 2024-03-24T00:46:58.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.334+0000 7fc3f464b640 1 -- 172.21.15.67:0/2723777791 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc3ec06cdd0 con 0x7fc3ec0cfc70 2024-03-24T00:46:58.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.334+0000 7fc3e2ffd640 1 -- 172.21.15.67:0/2723777791 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4980 (secure 0 0 0) 0x7fc3dc0588d0 con 0x7fc3ec0cfc70 2024-03-24T00:46:58.336 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:46:58.336 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":66,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6359,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2664460392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2664460392},{"type":"v1","addr":"172.21.15.67:6835","nonce":2664460392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6368,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/3323873506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3323873506},{"type":"v1","addr":"172.21.15.72:6837","nonce":3323873506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":7150,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/2744016217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2744016217},{"type":"v1","addr":"172.21.15.72:6836","nonce":2744016217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:44:54.749656+0000","modified":"2024-03-24T00:46:24.852760+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8163},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8163":{"gid":8163,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6833/3311719072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3311719072},{"type":"v1","addr":"172.21.15.67:6833","nonce":3311719072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8163,"qdb_cluster":[8163]},"id":6}]} 2024-03-24T00:46:58.336 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 66 2024-03-24T00:46:58.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.336+0000 7fc3f464b640 1 -- 172.21.15.67:0/2723777791 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc3d006bbd0 msgr2=0x7fc3d006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:58.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.336+0000 7fc3f464b640 1 --2- 172.21.15.67:0/2723777791 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc3d006bbd0 0x7fc3d006e090 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fc3e8000f60 tx=0x7fc3e8009040 comp rx=0 tx=0).stop 2024-03-24T00:46:58.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.336+0000 7fc3f464b640 1 -- 172.21.15.67:0/2723777791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3ec0cfc70 msgr2=0x7fc3ec0c98f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:58.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.336+0000 7fc3f464b640 1 --2- 172.21.15.67:0/2723777791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3ec0cfc70 0x7fc3ec0c98f0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc00b4a0 tx=0x7fc3dc00b960 comp rx=0 tx=0).stop 2024-03-24T00:46:58.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.336+0000 7fc3f464b640 1 -- 172.21.15.67:0/2723777791 shutdown_connections 2024-03-24T00:46:58.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.336+0000 7fc3f464b640 1 --2- 172.21.15.67:0/2723777791 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc3d006bbd0 0x7fc3d006e090 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.336+0000 7fc3f464b640 1 --2- 172.21.15.67:0/2723777791 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc3ec15bf70 0x7fc3ec0b5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.336+0000 7fc3f464b640 1 --2- 172.21.15.67:0/2723777791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3ec0cfc70 0x7fc3ec0c98f0 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.336+0000 7fc3f464b640 1 --2- 172.21.15.67:0/2723777791 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3ec0ce760 0x7fc3ec0cb9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.336+0000 7fc3f464b640 1 -- 172.21.15.67:0/2723777791 >> 172.21.15.67:0/2723777791 conn(0x7fc3ec0d1bf0 msgr2=0x7fc3ec0b2850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:58.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.337+0000 7fc3f464b640 1 -- 172.21.15.67:0/2723777791 shutdown_connections 2024-03-24T00:46:58.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.337+0000 7fc3f464b640 1 -- 172.21.15.67:0/2723777791 wait complete. 2024-03-24T00:46:58.349 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c damage ls --format=json-pretty 2024-03-24T00:46:58.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.451+0000 7fcc91ac6640 1 Processor -- start 2024-03-24T00:46:58.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.452+0000 7fcc91ac6640 1 -- start start 2024-03-24T00:46:58.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.453+0000 7fcc91ac6640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcc8c15dea0 0x7fcc8c0d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:58.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.453+0000 7fcc91ac6640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fcc8c0b6170 0x7fcc8c0d3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:58.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.453+0000 7fcc91ac6640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc8c0d2250 con 0x7fcc8c15e280 2024-03-24T00:46:58.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.453+0000 7fcc91ac6640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc8c0d2530 con 0x7fcc8c15dea0 2024-03-24T00:46:58.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.453+0000 7fcc91ac6640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc8c0d2810 con 0x7fcc8c0b6170 2024-03-24T00:46:58.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.453+0000 7fcc8b7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcc8c15dea0 0x7fcc8c0d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:58.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.453+0000 7fcc8affd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fcc8c15e280 0x7fcc8c0b5a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44736/0 (socket says 172.21.15.67:44736) 2024-03-24T00:46:58.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.453+0000 7fcc8affd640 1 -- 172.21.15.67:0/963556056 learned_addr learned my addr 172.21.15.67:0/963556056 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:46:58.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.453+0000 7fcc8b7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcc8c15dea0 0x7fcc8c0d33d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50704/0 (socket says 172.21.15.67:50704) 2024-03-24T00:46:58.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.453+0000 7fcc8bfff640 1 --2- 172.21.15.67:0/963556056 >> v2:172.21.15.72:3300/0 conn(0x7fcc8c0b6170 0x7fcc8c0d3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:58.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.453+0000 7fcc8b7fe640 1 -- 172.21.15.67:0/963556056 >> v2:172.21.15.72:3300/0 conn(0x7fcc8c0b6170 msgr2=0x7fcc8c0d3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:58.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.453+0000 7fcc8b7fe640 1 --2- 172.21.15.67:0/963556056 >> v2:172.21.15.72:3300/0 conn(0x7fcc8c0b6170 0x7fcc8c0d3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.453+0000 7fcc8b7fe640 1 -- 172.21.15.67:0/963556056 >> v1:172.21.15.67:6789/0 conn(0x7fcc8c15e280 legacy=0x7fcc8c0b5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:58.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.453+0000 7fcc8b7fe640 1 -- 172.21.15.67:0/963556056 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc8c0d2af0 con 0x7fcc8c15dea0 2024-03-24T00:46:58.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.453+0000 7fcc8a7fc640 1 -- 172.21.15.67:0/963556056 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2921752752 0 0) 0x7fcc8c0d2250 con 0x7fcc8c15e280 2024-03-24T00:46:58.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.454+0000 7fcc8b7fe640 1 --2- 172.21.15.67:0/963556056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc8c15dea0 0x7fcc8c0d33d0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c009d90 tx=0x7fcc7c02be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad03b309e7b29d8d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:58.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.454+0000 7fcc8a7fc640 1 -- 172.21.15.67:0/963556056 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc7c039070 con 0x7fcc8c15dea0 2024-03-24T00:46:58.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.454+0000 7fcc8a7fc640 1 -- 172.21.15.67:0/963556056 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc7c02cbd0 con 0x7fcc8c15dea0 2024-03-24T00:46:58.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.454+0000 7fcc8a7fc640 1 -- 172.21.15.67:0/963556056 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc7c034440 con 0x7fcc8c15dea0 2024-03-24T00:46:58.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.455+0000 7fcc91ac6640 1 -- 172.21.15.67:0/963556056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc8c15dea0 msgr2=0x7fcc8c0d33d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:58.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.455+0000 7fcc91ac6640 1 --2- 172.21.15.67:0/963556056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc8c15dea0 0x7fcc8c0d33d0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c009d90 tx=0x7fcc7c02be20 comp rx=0 tx=0).stop 2024-03-24T00:46:58.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.455+0000 7fcc91ac6640 1 -- 172.21.15.67:0/963556056 shutdown_connections 2024-03-24T00:46:58.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.455+0000 7fcc91ac6640 1 --2- 172.21.15.67:0/963556056 >> v2:172.21.15.72:3300/0 conn(0x7fcc8c0b6170 0x7fcc8c0d3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.455+0000 7fcc91ac6640 1 --2- 172.21.15.67:0/963556056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc8c15dea0 0x7fcc8c0d33d0 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.455+0000 7fcc91ac6640 1 -- 172.21.15.67:0/963556056 >> 172.21.15.67:0/963556056 conn(0x7fcc8c072da0 msgr2=0x7fcc8c0d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:58.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.455+0000 7fcc91ac6640 1 -- 172.21.15.67:0/963556056 shutdown_connections 2024-03-24T00:46:58.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.455+0000 7fcc91ac6640 1 -- 172.21.15.67:0/963556056 wait complete. 2024-03-24T00:46:58.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.457+0000 7fcc91ac6640 1 Processor -- start 2024-03-24T00:46:58.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.457+0000 7fcc91ac6640 1 -- start start 2024-03-24T00:46:58.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.458+0000 7fcc91ac6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcc8c0b6170 0x7fcc8c0cc100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:58.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.458+0000 7fcc91ac6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc8c15dea0 0x7fcc8c0b2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:58.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.458+0000 7fcc91ac6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc8c15e280 0x7fcc8c1f3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:58.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.458+0000 7fcc91ac6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8c0b7480 con 0x7fcc8c15dea0 2024-03-24T00:46:58.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.458+0000 7fcc91ac6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8c0b7300 con 0x7fcc8c15e280 2024-03-24T00:46:58.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.458+0000 7fcc91ac6640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8c0b7600 con 0x7fcc8c0b6170 2024-03-24T00:46:58.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.458+0000 7fcc8affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc8c15dea0 0x7fcc8c0b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:58.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.458+0000 7fcc8affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc8c15dea0 0x7fcc8c0b2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50714/0 (socket says 172.21.15.67:50714) 2024-03-24T00:46:58.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.458+0000 7fcc8affd640 1 -- 172.21.15.67:0/2688607188 learned_addr learned my addr 172.21.15.67:0/2688607188 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:58.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.458+0000 7fcc8b7fe640 1 --2- 172.21.15.67:0/2688607188 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcc8c0b6170 0x7fcc8c0cc100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:58.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.458+0000 7fcc8bfff640 1 --2- 172.21.15.67:0/2688607188 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc8c15e280 0x7fcc8c1f3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:58.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.458+0000 7fcc8affd640 1 -- 172.21.15.67:0/2688607188 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcc8c0b6170 msgr2=0x7fcc8c0cc100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:58.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.458+0000 7fcc8affd640 1 --2- 172.21.15.67:0/2688607188 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcc8c0b6170 0x7fcc8c0cc100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.458+0000 7fcc8affd640 1 -- 172.21.15.67:0/2688607188 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc8c15e280 msgr2=0x7fcc8c1f3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:58.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.458+0000 7fcc8affd640 1 --2- 172.21.15.67:0/2688607188 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc8c15e280 0x7fcc8c1f3800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.458+0000 7fcc8affd640 1 -- 172.21.15.67:0/2688607188 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc8c0cdb10 con 0x7fcc8c15dea0 2024-03-24T00:46:58.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.459+0000 7fcc8affd640 1 --2- 172.21.15.67:0/2688607188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc8c15dea0 0x7fcc8c0b2590 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7fcc8000c430 tx=0x7fcc8000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:58.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.459+0000 7fcc73fff640 1 -- 172.21.15.67:0/2688607188 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc800074e0 con 0x7fcc8c15dea0 2024-03-24T00:46:58.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.459+0000 7fcc73fff640 1 -- 172.21.15.67:0/2688607188 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc800047e0 con 0x7fcc8c15dea0 2024-03-24T00:46:58.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.459+0000 7fcc91ac6640 1 -- 172.21.15.67:0/2688607188 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc8c0ccad0 con 0x7fcc8c15dea0 2024-03-24T00:46:58.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.459+0000 7fcc73fff640 1 -- 172.21.15.67:0/2688607188 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc80004bd0 con 0x7fcc8c15dea0 2024-03-24T00:46:58.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.459+0000 7fcc91ac6640 1 -- 172.21.15.67:0/2688607188 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc8c0cd310 con 0x7fcc8c15dea0 2024-03-24T00:46:58.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.460+0000 7fcc73fff640 1 -- 172.21.15.67:0/2688607188 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fcc80011070 con 0x7fcc8c15dea0 2024-03-24T00:46:58.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.461+0000 7fcc73fff640 1 --2- 172.21.15.67:0/2688607188 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcc7806bbd0 0x7fcc7806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:58.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.461+0000 7fcc73fff640 1 -- 172.21.15.67:0/2688607188 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6389+0+0 (secure 0 0 0) 0x7fcc8008a840 con 0x7fcc8c15dea0 2024-03-24T00:46:58.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.461+0000 7fcc8b7fe640 1 --2- 172.21.15.67:0/2688607188 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcc7806bbd0 0x7fcc7806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:58.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.462+0000 7fcc8b7fe640 1 --2- 172.21.15.67:0/2688607188 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcc7806bbd0 0x7fcc7806e090 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c036000 tx=0x7fcc7c030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:58.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.462+0000 7fcc9478d740 1 Processor -- start 2024-03-24T00:46:58.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.462+0000 7fcc9478d740 1 -- start start 2024-03-24T00:46:58.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.462+0000 7fcc9478d740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556089a3b0d0 0x556089a0e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:58.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.462+0000 7fcc9478d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556089a3b4b0 0x556089a3de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:58.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.462+0000 7fcc9478d740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556089a0e840 0x556089a3e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:58.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.462+0000 7fcc9478d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560899781f0 con 0x556089a3b4b0 2024-03-24T00:46:58.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.462+0000 7fcc9478d740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x556089916a80 con 0x556089a3b0d0 2024-03-24T00:46:58.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.462+0000 7fcc9478d740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x556089974290 con 0x556089a0e840 2024-03-24T00:46:58.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.463+0000 7fcc8affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556089a3b4b0 0x556089a3de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:58.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.463+0000 7fcc8affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556089a3b4b0 0x556089a3de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50728/0 (socket says 172.21.15.67:50728) 2024-03-24T00:46:58.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.463+0000 7fcc8affd640 1 -- 172.21.15.67:0/2272073778 learned_addr learned my addr 172.21.15.67:0/2272073778 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:58.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.463+0000 7fcc8b7fe640 1 --2- 172.21.15.67:0/2272073778 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556089a3b0d0 0x556089a0e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:58.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.463+0000 7fcc8bfff640 1 --2- 172.21.15.67:0/2272073778 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556089a0e840 0x556089a3e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:58.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.463+0000 7fcc8affd640 1 -- 172.21.15.67:0/2272073778 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556089a0e840 msgr2=0x556089a3e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:58.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.463+0000 7fcc8affd640 1 --2- 172.21.15.67:0/2272073778 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556089a0e840 0x556089a3e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.463+0000 7fcc8affd640 1 -- 172.21.15.67:0/2272073778 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556089a3b0d0 msgr2=0x556089a0e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:58.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.463+0000 7fcc8affd640 1 --2- 172.21.15.67:0/2272073778 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556089a3b0d0 0x556089a0e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.463+0000 7fcc8affd640 1 -- 172.21.15.67:0/2272073778 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5560897587c0 con 0x556089a3b4b0 2024-03-24T00:46:58.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.463+0000 7fcc8affd640 1 --2- 172.21.15.67:0/2272073778 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556089a3b4b0 0x556089a3de60 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fcc80088a60 tx=0x7fcc800077a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e24597e504f013b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:58.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.463+0000 7fcc717fa640 1 -- 172.21.15.67:0/2272073778 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc80056d30 con 0x556089a3b4b0 2024-03-24T00:46:58.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.463+0000 7fcc717fa640 1 -- 172.21.15.67:0/2272073778 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc80004d70 con 0x556089a3b4b0 2024-03-24T00:46:58.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.463+0000 7fcc717fa640 1 -- 172.21.15.67:0/2272073778 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc80057e80 con 0x556089a3b4b0 2024-03-24T00:46:58.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.464+0000 7fcc9478d740 1 -- 172.21.15.67:0/2272073778 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556089a3b4b0 msgr2=0x556089a3de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:58.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.464+0000 7fcc9478d740 1 --2- 172.21.15.67:0/2272073778 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556089a3b4b0 0x556089a3de60 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fcc80088a60 tx=0x7fcc800077a0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.464+0000 7fcc9478d740 1 -- 172.21.15.67:0/2272073778 shutdown_connections 2024-03-24T00:46:58.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.464+0000 7fcc9478d740 1 --2- 172.21.15.67:0/2272073778 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556089a0e840 0x556089a3e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.464+0000 7fcc9478d740 1 --2- 172.21.15.67:0/2272073778 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556089a3b4b0 0x556089a3de60 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.464+0000 7fcc9478d740 1 --2- 172.21.15.67:0/2272073778 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556089a3b0d0 0x556089a0e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.464+0000 7fcc9478d740 1 -- 172.21.15.67:0/2272073778 >> 172.21.15.67:0/2272073778 conn(0x556089a10570 msgr2=0x556089a58e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:58.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.464+0000 7fcc9478d740 1 -- 172.21.15.67:0/2272073778 shutdown_connections 2024-03-24T00:46:58.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.464+0000 7fcc9478d740 1 -- 172.21.15.67:0/2272073778 wait complete. 2024-03-24T00:46:58.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.465+0000 7fcc9478d740 1 -- start start 2024-03-24T00:46:58.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.465+0000 7fcc9478d740 1 Processor -- start 2024-03-24T00:46:58.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.465+0000 7fcc9478d740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556089a3b0d0 0x556089a53490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:58.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.465+0000 7fcc9478d740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556089a3b4b0 0x556089a539d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:58.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.465+0000 7fcc9478d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556089a29130 0x556089a2a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:58.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.465+0000 7fcc9478d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x556089a12880 con 0x556089a29130 2024-03-24T00:46:58.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.465+0000 7fcc9478d740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x556089914410 con 0x556089a3b0d0 2024-03-24T00:46:58.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.465+0000 7fcc9478d740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x556089a11c70 con 0x556089a3b4b0 2024-03-24T00:46:58.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.465+0000 7fcc9478d740 10 client.0 _ll_register_callbacks cb 0x55608993cd70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:46:58.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.465+0000 7fcc8bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556089a29130 0x556089a2a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:58.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.466+0000 7fcc8bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556089a29130 0x556089a2a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50732/0 (socket says 172.21.15.67:50732) 2024-03-24T00:46:58.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.466+0000 7fcc8bfff640 1 -- 172.21.15.67:0/4102083973 learned_addr learned my addr 172.21.15.67:0/4102083973 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:58.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.466+0000 7fcc8affd640 1 --2- 172.21.15.67:0/4102083973 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556089a3b4b0 0x556089a539d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:58.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.466+0000 7fcc8b7fe640 1 --2- 172.21.15.67:0/4102083973 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556089a3b0d0 0x556089a53490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:58.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.466+0000 7fcc8bfff640 1 -- 172.21.15.67:0/4102083973 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556089a3b4b0 msgr2=0x556089a539d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:58.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.466+0000 7fcc8bfff640 1 --2- 172.21.15.67:0/4102083973 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556089a3b4b0 0x556089a539d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.466+0000 7fcc8bfff640 1 -- 172.21.15.67:0/4102083973 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556089a3b0d0 msgr2=0x556089a53490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:58.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.466+0000 7fcc8bfff640 1 --2- 172.21.15.67:0/4102083973 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556089a3b0d0 0x556089a53490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.466+0000 7fcc8bfff640 1 -- 172.21.15.67:0/4102083973 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5560897b8f50 con 0x556089a29130 2024-03-24T00:46:58.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.466+0000 7fcc8bfff640 1 --2- 172.21.15.67:0/4102083973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556089a29130 0x556089a2a7e0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7fcc84013650 tx=0x7fcc84013a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b4113528db62e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:58.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.466+0000 7fcc6a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:46:58.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.466+0000 7fcc6a7fc640 1 -- 172.21.15.67:0/4102083973 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc84004300 con 0x556089a29130 2024-03-24T00:46:58.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.466+0000 7fcc6a7fc640 1 -- 172.21.15.67:0/4102083973 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 6389+0+0 (secure 0 0 0) 0x7fcc8401ed60 con 0x556089a29130 2024-03-24T00:46:58.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.466+0000 7fcc9478d740 1 -- 172.21.15.67:0/4102083973 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcc80004d70 con 0x556089a29130 2024-03-24T00:46:58.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.466+0000 7fcc6a7fc640 1 -- 172.21.15.67:0/4102083973 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcc840044a0 con 0x556089a29130 2024-03-24T00:46:58.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.466+0000 7fcc9478d740 10 client.8343 fetch_fsmap learned FSMap version 66 2024-03-24T00:46:58.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.466+0000 7fcc9478d740 1 -- 172.21.15.67:0/4102083973 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=2+}) v3 -- 0x55608964c0b0 con 0x556089a29130 2024-03-24T00:46:58.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.466+0000 7fcc6a7fc640 1 -- 172.21.15.67:0/4102083973 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc80004d70 con 0x556089a29130 2024-03-24T00:46:58.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.467+0000 7fcc6a7fc640 1 -- 172.21.15.67:0/4102083973 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 66) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fcc84003610 con 0x556089a29130 2024-03-24T00:46:58.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.467+0000 7fcc9478d740 10 client.8343 fetch_fsmap finished waiting for FSMap version 66 2024-03-24T00:46:58.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.467+0000 7fcc9478d740 10 client.8343 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T00:46:58.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.467+0000 7fcc9478d740 20 client.8343 populate_metadata read hostname 'smithi067' 2024-03-24T00:46:58.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.467+0000 7fcc9478d740 1 --2- 172.21.15.67:0/4102083973 >> [v2:172.21.15.67:6832/3311719072,v1:172.21.15.67:6833/3311719072] conn(0x556089a2bc90 0x556089a2c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:58.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.467+0000 7fcc9478d740 4 client.8343 mds_command: new command op to 8163 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:46:58.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.467+0000 7fcc9478d740 1 -- 172.21.15.67:0/4102083973 --> [v2:172.21.15.67:6832/3311719072,v1:172.21.15.67:6833/3311719072] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556089985730 con 0x556089a2bc90 2024-03-24T00:46:58.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.467+0000 7fcc8b7fe640 1 --2- 172.21.15.67:0/4102083973 >> [v2:172.21.15.67:6832/3311719072,v1:172.21.15.67:6833/3311719072] conn(0x556089a2bc90 0x556089a2c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:58.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.467+0000 7fcc8b7fe640 1 --2- 172.21.15.67:0/4102083973 >> [v2:172.21.15.67:6832/3311719072,v1:172.21.15.67:6833/3311719072] conn(0x556089a2bc90 0x556089a2c070 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7911b8d9146f7 server_cookie=6c8fe95f0606ca10 in_seq=0 out_seq=0 2024-03-24T00:46:58.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.467+0000 7fcc6a7fc640 10 client.8343 ms_handle_connect on v2:172.21.15.67:6832/3311719072 2024-03-24T00:46:58.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.468+0000 7fcc6a7fc640 1 -- 172.21.15.67:0/4102083973 <== mds.0 v2:172.21.15.67:6832/3311719072 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x556089985730 con 0x556089a2bc90 2024-03-24T00:46:58.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.468+0000 7fcc6a7fc640 10 client.8343 handle_command_reply: tid=0 2024-03-24T00:46:58.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.468+0000 7fcc9478d740 1 client.8343 shutdown 2024-03-24T00:46:58.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.468+0000 7fcc9478d740 1 -- 172.21.15.67:0/4102083973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556089a29130 msgr2=0x556089a2a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:58.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.468+0000 7fcc9478d740 1 --2- 172.21.15.67:0/4102083973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556089a29130 0x556089a2a7e0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7fcc84013650 tx=0x7fcc84013a50 comp rx=0 tx=0).stop 2024-03-24T00:46:58.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.468+0000 7fcc9478d740 1 -- 172.21.15.67:0/4102083973 shutdown_connections 2024-03-24T00:46:58.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.468+0000 7fcc9478d740 1 --2- 172.21.15.67:0/4102083973 >> [v2:172.21.15.67:6832/3311719072,v1:172.21.15.67:6833/3311719072] conn(0x556089a2bc90 0x556089a2c070 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.468+0000 7fcc9478d740 1 --2- 172.21.15.67:0/4102083973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556089a29130 0x556089a2a7e0 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.468+0000 7fcc9478d740 1 --2- 172.21.15.67:0/4102083973 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556089a3b4b0 0x556089a539d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.468+0000 7fcc6a7fc640 0 client.8343 ms_handle_reset on v2:172.21.15.67:6832/3311719072 2024-03-24T00:46:58.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.468+0000 7fcc9478d740 1 --2- 172.21.15.67:0/4102083973 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556089a3b0d0 0x556089a53490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.468+0000 7fcc9478d740 1 -- 172.21.15.67:0/4102083973 >> 172.21.15.67:0/4102083973 conn(0x556089a0e840 msgr2=0x556089a5c2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:58.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.468+0000 7fcc9478d740 1 -- 172.21.15.67:0/4102083973 shutdown_connections 2024-03-24T00:46:58.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.468+0000 7fcc9478d740 1 -- 172.21.15.67:0/4102083973 wait complete. 2024-03-24T00:46:58.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.468+0000 7fcc9478d740 20 client.8343 trim_cache size 0 max 16384 2024-03-24T00:46:58.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.486+0000 7fcc9478d740 1 Processor -- start 2024-03-24T00:46:58.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.486+0000 7fcc9478d740 1 -- start start 2024-03-24T00:46:58.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.487+0000 7fcc9478d740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556089a29130 0x556089a2a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:58.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.487+0000 7fcc9478d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556089a3b0d0 0x556089a11bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:58.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.487+0000 7fcc9478d740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556089a3b4b0 0x556089a12110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:58.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.487+0000 7fcc9478d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8c1ffbf0 con 0x556089a3b0d0 2024-03-24T00:46:58.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.487+0000 7fcc9478d740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8c1ff020 con 0x556089a29130 2024-03-24T00:46:58.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.487+0000 7fcc9478d740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8c1ff5f0 con 0x556089a3b4b0 2024-03-24T00:46:58.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.487+0000 7fcc8affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556089a3b0d0 0x556089a11bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:58.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.487+0000 7fcc8affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556089a3b0d0 0x556089a11bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50748/0 (socket says 172.21.15.67:50748) 2024-03-24T00:46:58.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.487+0000 7fcc8affd640 1 -- 172.21.15.67:0/3914714552 learned_addr learned my addr 172.21.15.67:0/3914714552 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:58.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.487+0000 7fcc8b7fe640 1 --2- 172.21.15.67:0/3914714552 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556089a29130 0x556089a2a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:58.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.487+0000 7fcc8bfff640 1 --2- 172.21.15.67:0/3914714552 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556089a3b4b0 0x556089a12110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:58.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.487+0000 7fcc8affd640 1 -- 172.21.15.67:0/3914714552 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556089a3b4b0 msgr2=0x556089a12110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:58.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.487+0000 7fcc8affd640 1 --2- 172.21.15.67:0/3914714552 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556089a3b4b0 0x556089a12110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.487+0000 7fcc8affd640 1 -- 172.21.15.67:0/3914714552 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556089a29130 msgr2=0x556089a2a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:58.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.487+0000 7fcc8affd640 1 --2- 172.21.15.67:0/3914714552 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556089a29130 0x556089a2a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.487+0000 7fcc8affd640 1 -- 172.21.15.67:0/3914714552 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556089a0dc80 con 0x556089a3b0d0 2024-03-24T00:46:58.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.487+0000 7fcc8b7fe640 1 --2- 172.21.15.67:0/3914714552 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556089a29130 0x556089a2a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:46:58.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.487+0000 7fcc8bfff640 1 --2- 172.21.15.67:0/3914714552 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556089a3b4b0 0x556089a12110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:46:58.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.487+0000 7fcc8affd640 1 --2- 172.21.15.67:0/3914714552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556089a3b0d0 0x556089a11bd0 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7fcc80058730 tx=0x7fcc8008a070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efdecc0158cf851e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:58.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.487+0000 7fcc70ff9640 1 -- 172.21.15.67:0/3914714552 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc80075070 con 0x556089a3b0d0 2024-03-24T00:46:58.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.487+0000 7fcc70ff9640 1 -- 172.21.15.67:0/3914714552 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc80056d30 con 0x556089a3b0d0 2024-03-24T00:46:58.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.487+0000 7fcc70ff9640 1 -- 172.21.15.67:0/3914714552 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc80054cd0 con 0x556089a3b0d0 2024-03-24T00:46:58.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.488+0000 7fcc9478d740 1 -- 172.21.15.67:0/3914714552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556089a3b0d0 msgr2=0x556089a11bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:58.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.488+0000 7fcc9478d740 1 --2- 172.21.15.67:0/3914714552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556089a3b0d0 0x556089a11bd0 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7fcc80058730 tx=0x7fcc8008a070 comp rx=0 tx=0).stop 2024-03-24T00:46:58.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.488+0000 7fcc9478d740 1 -- 172.21.15.67:0/3914714552 shutdown_connections 2024-03-24T00:46:58.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.488+0000 7fcc9478d740 1 --2- 172.21.15.67:0/3914714552 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556089a3b4b0 0x556089a12110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.488+0000 7fcc9478d740 1 --2- 172.21.15.67:0/3914714552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556089a3b0d0 0x556089a11bd0 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.488+0000 7fcc9478d740 1 --2- 172.21.15.67:0/3914714552 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556089a29130 0x556089a2a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.488+0000 7fcc9478d740 1 -- 172.21.15.67:0/3914714552 >> 172.21.15.67:0/3914714552 conn(0x556089a0e840 msgr2=0x556089a15090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:58.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.488+0000 7fcc9478d740 1 -- 172.21.15.67:0/3914714552 shutdown_connections 2024-03-24T00:46:58.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.488+0000 7fcc9478d740 1 -- 172.21.15.67:0/3914714552 wait complete. 2024-03-24T00:46:58.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.489+0000 7fcc9478d740 1 -- start start 2024-03-24T00:46:58.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.489+0000 7fcc9478d740 1 Processor -- start 2024-03-24T00:46:58.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.489+0000 7fcc9478d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556089a29130 0x556089a5b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:58.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.489+0000 7fcc9478d740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556089a3b0d0 0x556089a3a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:58.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.489+0000 7fcc9478d740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556089a3b4b0 0x556089a4f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:58.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.489+0000 7fcc9478d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x556089a2b290 con 0x556089a29130 2024-03-24T00:46:58.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.489+0000 7fcc9478d740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x556089a51140 con 0x556089a3b4b0 2024-03-24T00:46:58.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.489+0000 7fcc9478d740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x556089a41520 con 0x556089a3b0d0 2024-03-24T00:46:58.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.489+0000 7fcc9478d740 10 client.0 _ll_register_callbacks cb 0x7fcc840075f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:46:58.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.489+0000 7fcc8b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556089a29130 0x556089a5b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:58.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.489+0000 7fcc8b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556089a29130 0x556089a5b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50758/0 (socket says 172.21.15.67:50758) 2024-03-24T00:46:58.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.489+0000 7fcc8b7fe640 1 -- 172.21.15.67:0/4003166731 learned_addr learned my addr 172.21.15.67:0/4003166731 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:58.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.490+0000 7fcc8affd640 1 --2- 172.21.15.67:0/4003166731 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556089a3b0d0 0x556089a3a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:58.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.490+0000 7fcc8bfff640 1 --2- 172.21.15.67:0/4003166731 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556089a3b4b0 0x556089a4f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:58.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.490+0000 7fcc8b7fe640 1 -- 172.21.15.67:0/4003166731 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556089a3b0d0 msgr2=0x556089a3a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:58.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.490+0000 7fcc8b7fe640 1 --2- 172.21.15.67:0/4003166731 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556089a3b0d0 0x556089a3a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.490+0000 7fcc8b7fe640 1 -- 172.21.15.67:0/4003166731 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556089a3b4b0 msgr2=0x556089a4f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:58.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.490+0000 7fcc8b7fe640 1 --2- 172.21.15.67:0/4003166731 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556089a3b4b0 0x556089a4f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.490+0000 7fcc8b7fe640 1 -- 172.21.15.67:0/4003166731 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556089a09c20 con 0x556089a29130 2024-03-24T00:46:58.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.490+0000 7fcc8bfff640 1 --2- 172.21.15.67:0/4003166731 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556089a3b4b0 0x556089a4f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:46:58.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.490+0000 7fcc8b7fe640 1 --2- 172.21.15.67:0/4003166731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556089a29130 0x556089a5b2d0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c037040 tx=0x7fcc7c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=209a4d006201e6bd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:58.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.490+0000 7fcc6a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:46:58.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.490+0000 7fcc6a7fc640 1 -- 172.21.15.67:0/4003166731 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc7c039040 con 0x556089a29130 2024-03-24T00:46:58.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.490+0000 7fcc6a7fc640 1 -- 172.21.15.67:0/4003166731 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 6389+0+0 (secure 0 0 0) 0x7fcc7c04f410 con 0x556089a29130 2024-03-24T00:46:58.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.490+0000 7fcc9478d740 1 -- 172.21.15.67:0/4003166731 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcc80056d30 con 0x556089a29130 2024-03-24T00:46:58.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.490+0000 7fcc6a7fc640 1 -- 172.21.15.67:0/4003166731 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcc7c04f610 con 0x556089a29130 2024-03-24T00:46:58.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.490+0000 7fcc9478d740 10 client.8349 fetch_fsmap learned FSMap version 66 2024-03-24T00:46:58.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.490+0000 7fcc9478d740 1 -- 172.21.15.67:0/4003166731 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=2+}) v3 -- 0x556089a3ab90 con 0x556089a29130 2024-03-24T00:46:58.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.490+0000 7fcc6a7fc640 1 -- 172.21.15.67:0/4003166731 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc80056d30 con 0x556089a29130 2024-03-24T00:46:58.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.490+0000 7fcc6a7fc640 1 -- 172.21.15.67:0/4003166731 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 66) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fcc7c060050 con 0x556089a29130 2024-03-24T00:46:58.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.491+0000 7fcc9478d740 10 client.8349 fetch_fsmap finished waiting for FSMap version 66 2024-03-24T00:46:58.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.491+0000 7fcc9478d740 10 client.8349 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T00:46:58.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.491+0000 7fcc9478d740 20 client.8349 populate_metadata read hostname 'smithi067' 2024-03-24T00:46:58.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.491+0000 7fcc9478d740 1 --2- 172.21.15.67:0/4003166731 >> [v2:172.21.15.67:6832/3311719072,v1:172.21.15.67:6833/3311719072] conn(0x556089acaef0 0x556089acb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:58.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.491+0000 7fcc9478d740 4 client.8349 mds_command: new command op to 8163 tid=0 multi_id=0 [{"prefix": "damage ls", "format": "json-pretty"}] 2024-03-24T00:46:58.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.491+0000 7fcc9478d740 1 -- 172.21.15.67:0/4003166731 --> [v2:172.21.15.67:6832/3311719072,v1:172.21.15.67:6833/3311719072] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) v1 -- 0x556089a546b0 con 0x556089acaef0 2024-03-24T00:46:58.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.491+0000 7fcc8affd640 1 --2- 172.21.15.67:0/4003166731 >> [v2:172.21.15.67:6832/3311719072,v1:172.21.15.67:6833/3311719072] conn(0x556089acaef0 0x556089acb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:58.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.491+0000 7fcc6a7fc640 10 client.8349 ms_handle_connect on v2:172.21.15.67:6832/3311719072 2024-03-24T00:46:58.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.491+0000 7fcc8affd640 1 --2- 172.21.15.67:0/4003166731 >> [v2:172.21.15.67:6832/3311719072,v1:172.21.15.67:6833/3311719072] conn(0x556089acaef0 0x556089acb2d0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3549653df63d98b server_cookie=2d848d06fc7c0b03 in_seq=0 out_seq=0 2024-03-24T00:46:58.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.491+0000 7fcc6a7fc640 1 -- 172.21.15.67:0/4003166731 <== mds.0 v2:172.21.15.67:6832/3311719072 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x556089a546b0 con 0x556089acaef0 2024-03-24T00:46:58.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.491+0000 7fcc6a7fc640 10 client.8349 handle_command_reply: tid=0 2024-03-24T00:46:58.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.491+0000 7fcc9478d740 1 client.8349 shutdown 2024-03-24T00:46:58.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.491+0000 7fcc9478d740 1 -- 172.21.15.67:0/4003166731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556089a29130 msgr2=0x556089a5b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:58.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.491+0000 7fcc9478d740 1 --2- 172.21.15.67:0/4003166731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556089a29130 0x556089a5b2d0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c037040 tx=0x7fcc7c004ab0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.491+0000 7fcc9478d740 1 -- 172.21.15.67:0/4003166731 shutdown_connections 2024-03-24T00:46:58.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.491+0000 7fcc9478d740 1 --2- 172.21.15.67:0/4003166731 >> [v2:172.21.15.67:6832/3311719072,v1:172.21.15.67:6833/3311719072] conn(0x556089acaef0 0x556089acb2d0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.491+0000 7fcc9478d740 1 --2- 172.21.15.67:0/4003166731 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556089a3b4b0 0x556089a4f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.491+0000 7fcc9478d740 1 --2- 172.21.15.67:0/4003166731 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556089a3b0d0 0x556089a3a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.491+0000 7fcc6a7fc640 0 client.8349 ms_handle_reset on v2:172.21.15.67:6832/3311719072 2024-03-24T00:46:58.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.491+0000 7fcc9478d740 1 --2- 172.21.15.67:0/4003166731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556089a29130 0x556089a5b2d0 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.491+0000 7fcc9478d740 1 -- 172.21.15.67:0/4003166731 >> 172.21.15.67:0/4003166731 conn(0x556089a0e840 msgr2=0x556089a5c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:58.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.492+0000 7fcc9478d740 1 -- 172.21.15.67:0/4003166731 shutdown_connections 2024-03-24T00:46:58.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.492+0000 7fcc9478d740 1 -- 172.21.15.67:0/4003166731 wait complete. 2024-03-24T00:46:58.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.492+0000 7fcc9478d740 20 client.8349 trim_cache size 0 max 16384 2024-03-24T00:46:58.499 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:46:58.499 INFO:teuthology.orchestra.run.smithi067.stdout:[] 2024-03-24T00:46:58.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.492+0000 7fcc91ac6640 1 -- 172.21.15.67:0/2688607188 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcc7806bbd0 msgr2=0x7fcc7806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:58.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.492+0000 7fcc91ac6640 1 --2- 172.21.15.67:0/2688607188 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcc7806bbd0 0x7fcc7806e090 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c036000 tx=0x7fcc7c030000 comp rx=0 tx=0).stop 2024-03-24T00:46:58.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.492+0000 7fcc91ac6640 1 -- 172.21.15.67:0/2688607188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc8c15dea0 msgr2=0x7fcc8c0b2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:58.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.492+0000 7fcc91ac6640 1 --2- 172.21.15.67:0/2688607188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc8c15dea0 0x7fcc8c0b2590 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7fcc8000c430 tx=0x7fcc8000c8f0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.492+0000 7fcc91ac6640 1 -- 172.21.15.67:0/2688607188 shutdown_connections 2024-03-24T00:46:58.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.492+0000 7fcc91ac6640 1 --2- 172.21.15.67:0/2688607188 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcc7806bbd0 0x7fcc7806e090 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.492+0000 7fcc91ac6640 1 --2- 172.21.15.67:0/2688607188 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc8c15e280 0x7fcc8c1f3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.492+0000 7fcc91ac6640 1 --2- 172.21.15.67:0/2688607188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc8c15dea0 0x7fcc8c0b2590 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.492+0000 7fcc91ac6640 1 --2- 172.21.15.67:0/2688607188 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcc8c0b6170 0x7fcc8c0cc100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.492+0000 7fcc91ac6640 1 -- 172.21.15.67:0/2688607188 >> 172.21.15.67:0/2688607188 conn(0x7fcc8c072da0 msgr2=0x7fcc8c0d31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:58.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.492+0000 7fcc91ac6640 1 -- 172.21.15.67:0/2688607188 shutdown_connections 2024-03-24T00:46:58.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.492+0000 7fcc91ac6640 1 -- 172.21.15.67:0/2688607188 wait complete. 2024-03-24T00:46:58.503 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-24T00:46:58.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.602+0000 7f770d948640 1 Processor -- start 2024-03-24T00:46:58.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.602+0000 7f770d948640 1 -- start start 2024-03-24T00:46:58.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.603+0000 7f770d948640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f770815e2a0 0x7f77080ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:58.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.603+0000 7f770d948640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f77080d0ef0 0x7f7708162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:58.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.603+0000 7f770d948640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77080cf640 con 0x7f770815dec0 2024-03-24T00:46:58.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.603+0000 7f770d948640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77080cee30 con 0x7f77080d0ef0 2024-03-24T00:46:58.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.603+0000 7f770d948640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77080d2120 con 0x7f770815e2a0 2024-03-24T00:46:58.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.603+0000 7f77077fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f77080d0ef0 0x7f7708162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:58.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.603+0000 7f77067fc640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f770815e2a0 0x7f77080ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:58.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.603+0000 7f7706ffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f770815dec0 0x7f77080d07e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:54598/0 (socket says 172.21.15.67:54598) 2024-03-24T00:46:58.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.603+0000 7f77077fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f77080d0ef0 0x7f7708162630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:50106/0 (socket says 172.21.15.67:50106) 2024-03-24T00:46:58.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.603+0000 7f77077fe640 1 -- 172.21.15.67:0/1051730527 learned_addr learned my addr 172.21.15.67:0/1051730527 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:58.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.604+0000 7f7705ffb640 1 -- 172.21.15.67:0/1051730527 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3710545120 0 0) 0x7f76f8001110 con 0x7f770815dec0 2024-03-24T00:46:58.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.604+0000 7f77067fc640 1 -- 172.21.15.67:0/1051730527 >> v2:172.21.15.72:3300/0 conn(0x7f77080d0ef0 msgr2=0x7f7708162630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:58.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.604+0000 7f77067fc640 1 --2- 172.21.15.67:0/1051730527 >> v2:172.21.15.72:3300/0 conn(0x7f77080d0ef0 0x7f7708162630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.604+0000 7f77067fc640 1 -- 172.21.15.67:0/1051730527 >> v1:172.21.15.72:6789/0 conn(0x7f770815dec0 legacy=0x7f77080d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:58.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.604+0000 7f77067fc640 1 -- 172.21.15.67:0/1051730527 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77080cf080 con 0x7f770815e2a0 2024-03-24T00:46:58.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.604+0000 7f77077fe640 1 --2- 172.21.15.67:0/1051730527 >> v2:172.21.15.72:3300/0 conn(0x7f77080d0ef0 0x7f7708162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:46:58.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.605+0000 7f77067fc640 1 --2- 172.21.15.67:0/1051730527 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f770815e2a0 0x7f77080ce630 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f76fc0097a0 tx=0x7f76fc02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3e6335f0cd7e04e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:58.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.605+0000 7f7705ffb640 1 -- 172.21.15.67:0/1051730527 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76fc0315a0 con 0x7f770815e2a0 2024-03-24T00:46:58.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.605+0000 7f7705ffb640 1 -- 172.21.15.67:0/1051730527 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76fc037070 con 0x7f770815e2a0 2024-03-24T00:46:58.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.605+0000 7f7705ffb640 1 -- 172.21.15.67:0/1051730527 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76fc031960 con 0x7f770815e2a0 2024-03-24T00:46:58.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.605+0000 7f770d948640 1 -- 172.21.15.67:0/1051730527 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f770815e2a0 msgr2=0x7f77080ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:58.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.605+0000 7f770d948640 1 --2- 172.21.15.67:0/1051730527 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f770815e2a0 0x7f77080ce630 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f76fc0097a0 tx=0x7f76fc02b840 comp rx=0 tx=0).stop 2024-03-24T00:46:58.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.605+0000 7f770d948640 1 -- 172.21.15.67:0/1051730527 shutdown_connections 2024-03-24T00:46:58.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.605+0000 7f770d948640 1 --2- 172.21.15.67:0/1051730527 >> v2:172.21.15.72:3300/0 conn(0x7f77080d0ef0 0x7f7708162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.605+0000 7f770d948640 1 --2- 172.21.15.67:0/1051730527 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f770815e2a0 0x7f77080ce630 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.605+0000 7f770d948640 1 -- 172.21.15.67:0/1051730527 >> 172.21.15.67:0/1051730527 conn(0x7f770806c810 msgr2=0x7f770806cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:58.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.605+0000 7f770d948640 1 -- 172.21.15.67:0/1051730527 shutdown_connections 2024-03-24T00:46:58.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.605+0000 7f770d948640 1 -- 172.21.15.67:0/1051730527 wait complete. 2024-03-24T00:46:58.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.606+0000 7f770d948640 1 Processor -- start 2024-03-24T00:46:58.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.607+0000 7f770d948640 1 -- start start 2024-03-24T00:46:58.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.607+0000 7f770d948640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f77080d0ef0 0x7f770815d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:58.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.607+0000 7f770d948640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f770815dec0 0x7f77080ad9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:58.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.607+0000 7f770d948640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f770815e2a0 0x7f77081f7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:58.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.607+0000 7f770d948640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77080d2730 con 0x7f770815e2a0 2024-03-24T00:46:58.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.607+0000 7f770d948640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77080d25b0 con 0x7f77080d0ef0 2024-03-24T00:46:58.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.607+0000 7f770d948640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77080d28b0 con 0x7f770815dec0 2024-03-24T00:46:58.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.607+0000 7f77077fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f770815e2a0 0x7f77081f7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:58.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.607+0000 7f77077fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f770815e2a0 0x7f77081f7780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50762/0 (socket says 172.21.15.67:50762) 2024-03-24T00:46:58.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.607+0000 7f77077fe640 1 -- 172.21.15.67:0/4114018601 learned_addr learned my addr 172.21.15.67:0/4114018601 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:58.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.607+0000 7f7706ffd640 1 --2- 172.21.15.67:0/4114018601 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f77080d0ef0 0x7f770815d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:58.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.607+0000 7f77067fc640 1 --2- 172.21.15.67:0/4114018601 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f770815dec0 0x7f77080ad9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:58.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.608+0000 7f77077fe640 1 -- 172.21.15.67:0/4114018601 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f770815dec0 msgr2=0x7f77080ad9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:58.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.608+0000 7f77077fe640 1 --2- 172.21.15.67:0/4114018601 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f770815dec0 0x7f77080ad9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.608+0000 7f77077fe640 1 -- 172.21.15.67:0/4114018601 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f77080d0ef0 msgr2=0x7f770815d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:58.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.608+0000 7f77077fe640 1 --2- 172.21.15.67:0/4114018601 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f77080d0ef0 0x7f770815d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.608+0000 7f77077fe640 1 -- 172.21.15.67:0/4114018601 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77080ab0f0 con 0x7f770815e2a0 2024-03-24T00:46:58.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.608+0000 7f77077fe640 1 --2- 172.21.15.67:0/4114018601 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f770815e2a0 0x7f77081f7780 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f770000ef90 tx=0x7f770000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:58.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.608+0000 7f76effff640 1 -- 172.21.15.67:0/4114018601 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f770000cde0 con 0x7f770815e2a0 2024-03-24T00:46:58.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.608+0000 7f76effff640 1 -- 172.21.15.67:0/4114018601 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7700003990 con 0x7f770815e2a0 2024-03-24T00:46:58.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.608+0000 7f770d948640 1 -- 172.21.15.67:0/4114018601 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77080ab3d0 con 0x7f770815e2a0 2024-03-24T00:46:58.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.608+0000 7f76effff640 1 -- 172.21.15.67:0/4114018601 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7700003d80 con 0x7f770815e2a0 2024-03-24T00:46:58.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.608+0000 7f770d948640 1 -- 172.21.15.67:0/4114018601 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77080ab990 con 0x7f770815e2a0 2024-03-24T00:46:58.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.609+0000 7f770d948640 1 -- 172.21.15.67:0/4114018601 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f770804f400 con 0x7f770815e2a0 2024-03-24T00:46:58.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.610+0000 7f76effff640 1 -- 172.21.15.67:0/4114018601 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7700012050 con 0x7f770815e2a0 2024-03-24T00:46:58.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.610+0000 7f76effff640 1 --2- 172.21.15.67:0/4114018601 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f76f406bbd0 0x7f76f406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:58.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.610+0000 7f76effff640 1 -- 172.21.15.67:0/4114018601 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f770008aff0 con 0x7f770815e2a0 2024-03-24T00:46:58.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.610+0000 7f7706ffd640 1 --2- 172.21.15.67:0/4114018601 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f76f406bbd0 0x7f76f406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:58.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.611+0000 7f7706ffd640 1 --2- 172.21.15.67:0/4114018601 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f76f406bbd0 0x7f76f406e090 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f76f8000f60 tx=0x7f76f8007b50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:58.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.619+0000 7f76effff640 1 -- 172.21.15.67:0/4114018601 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7700010040 con 0x7f770815e2a0 2024-03-24T00:46:58.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.844+0000 7f770d948640 1 -- 172.21.15.67:0/4114018601 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f77080b8c50 con 0x7f770815e2a0 2024-03-24T00:46:58.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.845+0000 7f76effff640 1 -- 172.21.15.67:0/4114018601 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f7700059270 con 0x7f770815e2a0 2024-03-24T00:46:58.846 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:46:58.846 INFO:teuthology.orchestra.run.smithi067.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-24T00:46:58.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.847+0000 7f770d948640 1 -- 172.21.15.67:0/4114018601 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f76f406bbd0 msgr2=0x7f76f406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:58.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.847+0000 7f770d948640 1 --2- 172.21.15.67:0/4114018601 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f76f406bbd0 0x7f76f406e090 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f76f8000f60 tx=0x7f76f8007b50 comp rx=0 tx=0).stop 2024-03-24T00:46:58.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.847+0000 7f770d948640 1 -- 172.21.15.67:0/4114018601 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f770815e2a0 msgr2=0x7f77081f7780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:58.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.847+0000 7f770d948640 1 --2- 172.21.15.67:0/4114018601 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f770815e2a0 0x7f77081f7780 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f770000ef90 tx=0x7f770000c490 comp rx=0 tx=0).stop 2024-03-24T00:46:58.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.847+0000 7f770d948640 1 -- 172.21.15.67:0/4114018601 shutdown_connections 2024-03-24T00:46:58.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.847+0000 7f770d948640 1 --2- 172.21.15.67:0/4114018601 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f76f406bbd0 0x7f76f406e090 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.847+0000 7f770d948640 1 --2- 172.21.15.67:0/4114018601 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f770815e2a0 0x7f77081f7780 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.847+0000 7f770d948640 1 --2- 172.21.15.67:0/4114018601 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f770815dec0 0x7f77080ad9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.847+0000 7f770d948640 1 --2- 172.21.15.67:0/4114018601 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f77080d0ef0 0x7f770815d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:58.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.847+0000 7f770d948640 1 -- 172.21.15.67:0/4114018601 >> 172.21.15.67:0/4114018601 conn(0x7f770806c810 msgr2=0x7f77080b7a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:58.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.847+0000 7f770d948640 1 -- 172.21.15.67:0/4114018601 shutdown_connections 2024-03-24T00:46:58.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:58.847+0000 7f770d948640 1 -- 172.21.15.67:0/4114018601 wait complete. 2024-03-24T00:46:58.862 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-03-24T00:46:58.862 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-24T00:46:58.862 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:46:58.902 DEBUG:teuthology.orchestra.run.smithi072:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:46:58.959 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-24T00:46:58.959 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:46:58.959 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:46:58.974 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:46:58.974 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:46:59.029 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:46:59.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.169+0000 7fcf4713b640 1 Processor -- start 2024-03-24T00:46:59.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.169+0000 7fcf4713b640 1 -- start start 2024-03-24T00:46:59.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.170+0000 7fcf4713b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fcf4015dec0 0x7fcf400ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:59.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.170+0000 7fcf4713b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcf4015e2a0 0x7fcf40162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:59.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.170+0000 7fcf4713b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf400cf640 con 0x7fcf400d07e0 2024-03-24T00:46:59.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.170+0000 7fcf4713b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf400cee30 con 0x7fcf4015e2a0 2024-03-24T00:46:59.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.170+0000 7fcf4713b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf400d2120 con 0x7fcf4015dec0 2024-03-24T00:46:59.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.170+0000 7fcf3ffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcf4015e2a0 0x7fcf40162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:59.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.170+0000 7fcf456d8640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fcf400d07e0 0x7fcf400d0be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44750/0 (socket says 172.21.15.67:44750) 2024-03-24T00:46:59.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.170+0000 7fcf456d8640 1 -- 172.21.15.67:0/1608794400 learned_addr learned my addr 172.21.15.67:0/1608794400 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:46:59.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.170+0000 7fcf44ed7640 1 --2- 172.21.15.67:0/1608794400 >> v2:172.21.15.72:3300/0 conn(0x7fcf4015dec0 0x7fcf400ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:59.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.170+0000 7fcf3ffff640 1 -- 172.21.15.67:0/1608794400 >> v2:172.21.15.72:3300/0 conn(0x7fcf4015dec0 msgr2=0x7fcf400ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:59.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.170+0000 7fcf3ffff640 1 --2- 172.21.15.67:0/1608794400 >> v2:172.21.15.72:3300/0 conn(0x7fcf4015dec0 0x7fcf400ce630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:59.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.170+0000 7fcf3ffff640 1 -- 172.21.15.67:0/1608794400 >> v1:172.21.15.67:6789/0 conn(0x7fcf400d07e0 legacy=0x7fcf400d0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:59.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.170+0000 7fcf3ffff640 1 -- 172.21.15.67:0/1608794400 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf400cf080 con 0x7fcf4015e2a0 2024-03-24T00:46:59.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.171+0000 7fcf3ffff640 1 --2- 172.21.15.67:0/1608794400 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf4015e2a0 0x7fcf40162630 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7fcf400d14a0 tx=0x7fcf2c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8f0c3622997b376 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:59.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.171+0000 7fcf3f7fe640 1 -- 172.21.15.67:0/1608794400 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf2c0315a0 con 0x7fcf4015e2a0 2024-03-24T00:46:59.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.171+0000 7fcf3f7fe640 1 -- 172.21.15.67:0/1608794400 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf2c037070 con 0x7fcf4015e2a0 2024-03-24T00:46:59.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.171+0000 7fcf3f7fe640 1 -- 172.21.15.67:0/1608794400 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf2c031960 con 0x7fcf4015e2a0 2024-03-24T00:46:59.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.172+0000 7fcf4713b640 1 -- 172.21.15.67:0/1608794400 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf4015e2a0 msgr2=0x7fcf40162630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:59.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.172+0000 7fcf4713b640 1 --2- 172.21.15.67:0/1608794400 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf4015e2a0 0x7fcf40162630 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7fcf400d14a0 tx=0x7fcf2c02b840 comp rx=0 tx=0).stop 2024-03-24T00:46:59.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.172+0000 7fcf4713b640 1 -- 172.21.15.67:0/1608794400 shutdown_connections 2024-03-24T00:46:59.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.172+0000 7fcf4713b640 1 --2- 172.21.15.67:0/1608794400 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf4015e2a0 0x7fcf40162630 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:59.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.172+0000 7fcf4713b640 1 --2- 172.21.15.67:0/1608794400 >> v2:172.21.15.72:3300/0 conn(0x7fcf4015dec0 0x7fcf400ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:59.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.172+0000 7fcf4713b640 1 -- 172.21.15.67:0/1608794400 >> 172.21.15.67:0/1608794400 conn(0x7fcf4006c810 msgr2=0x7fcf4006cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:59.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.172+0000 7fcf4713b640 1 -- 172.21.15.67:0/1608794400 shutdown_connections 2024-03-24T00:46:59.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.172+0000 7fcf4713b640 1 -- 172.21.15.67:0/1608794400 wait complete. 2024-03-24T00:46:59.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.173+0000 7fcf4713b640 1 Processor -- start 2024-03-24T00:46:59.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.173+0000 7fcf4713b640 1 -- start start 2024-03-24T00:46:59.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.174+0000 7fcf4713b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf400d07e0 0x7fcf40158cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:59.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.174+0000 7fcf4713b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcf4015dec0 0x7fcf400b3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:59.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.174+0000 7fcf4713b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcf4015e2a0 0x7fcf400b3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:59.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.174+0000 7fcf4713b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf400d2730 con 0x7fcf400d07e0 2024-03-24T00:46:59.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.174+0000 7fcf4713b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf400d25b0 con 0x7fcf4015e2a0 2024-03-24T00:46:59.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.174+0000 7fcf4713b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf400d28b0 con 0x7fcf4015dec0 2024-03-24T00:46:59.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.174+0000 7fcf44ed7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf400d07e0 0x7fcf40158cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:59.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.174+0000 7fcf44ed7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf400d07e0 0x7fcf40158cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50786/0 (socket says 172.21.15.67:50786) 2024-03-24T00:46:59.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.174+0000 7fcf44ed7640 1 -- 172.21.15.67:0/3057873342 learned_addr learned my addr 172.21.15.67:0/3057873342 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:59.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.174+0000 7fcf44ed7640 1 -- 172.21.15.67:0/3057873342 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcf4015dec0 msgr2=0x7fcf400b3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:59.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.174+0000 7fcf3ffff640 1 --2- 172.21.15.67:0/3057873342 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcf4015dec0 0x7fcf400b3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:59.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.174+0000 7fcf456d8640 1 --2- 172.21.15.67:0/3057873342 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcf4015e2a0 0x7fcf400b3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:59.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.174+0000 7fcf44ed7640 1 --2- 172.21.15.67:0/3057873342 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcf4015dec0 0x7fcf400b3810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:59.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.174+0000 7fcf44ed7640 1 -- 172.21.15.67:0/3057873342 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcf4015e2a0 msgr2=0x7fcf400b3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:59.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.174+0000 7fcf44ed7640 1 --2- 172.21.15.67:0/3057873342 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcf4015e2a0 0x7fcf400b3d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:59.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.174+0000 7fcf44ed7640 1 -- 172.21.15.67:0/3057873342 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf40159200 con 0x7fcf400d07e0 2024-03-24T00:46:59.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.175+0000 7fcf44ed7640 1 --2- 172.21.15.67:0/3057873342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf400d07e0 0x7fcf40158cc0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7fcf3400c7f0 tx=0x7fcf3400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:59.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.175+0000 7fcf3d7fa640 1 -- 172.21.15.67:0/3057873342 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf34015070 con 0x7fcf400d07e0 2024-03-24T00:46:59.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.175+0000 7fcf3d7fa640 1 -- 172.21.15.67:0/3057873342 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf34004240 con 0x7fcf400d07e0 2024-03-24T00:46:59.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.175+0000 7fcf4713b640 1 -- 172.21.15.67:0/3057873342 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf400b65e0 con 0x7fcf400d07e0 2024-03-24T00:46:59.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.175+0000 7fcf3d7fa640 1 -- 172.21.15.67:0/3057873342 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf34007620 con 0x7fcf400d07e0 2024-03-24T00:46:59.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.175+0000 7fcf4713b640 1 -- 172.21.15.67:0/3057873342 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf400cbaa0 con 0x7fcf400d07e0 2024-03-24T00:46:59.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.176+0000 7fcf4713b640 1 -- 172.21.15.67:0/3057873342 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf401527d0 con 0x7fcf400d07e0 2024-03-24T00:46:59.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.177+0000 7fcf3d7fa640 1 -- 172.21.15.67:0/3057873342 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fcf340079d0 con 0x7fcf400d07e0 2024-03-24T00:46:59.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.177+0000 7fcf3d7fa640 1 --2- 172.21.15.67:0/3057873342 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcf2406bbd0 0x7fcf2406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:59.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.177+0000 7fcf3d7fa640 1 -- 172.21.15.67:0/3057873342 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6389+0+0 (secure 0 0 0) 0x7fcf3408ada0 con 0x7fcf400d07e0 2024-03-24T00:46:59.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.177+0000 7fcf3ffff640 1 --2- 172.21.15.67:0/3057873342 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcf2406bbd0 0x7fcf2406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:59.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.178+0000 7fcf3ffff640 1 --2- 172.21.15.67:0/3057873342 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcf2406bbd0 0x7fcf2406e090 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c036000 tx=0x7fcf2c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:59.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.181+0000 7fcf3d7fa640 1 -- 172.21.15.67:0/3057873342 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fcf340595e0 con 0x7fcf400d07e0 2024-03-24T00:46:59.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.365+0000 7fcf4713b640 1 -- 172.21.15.67:0/3057873342 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcf400d1aa0 con 0x7fcf400d07e0 2024-03-24T00:46:59.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.366+0000 7fcf3d7fa640 1 -- 172.21.15.67:0/3057873342 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+17278 (secure 0 0 0) 0x7fcf34058ee0 con 0x7fcf400d07e0 2024-03-24T00:46:59.367 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:46:59.367 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":93,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:46:20.877901+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:44:53.025965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-03-24T00:44:53.712957+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.32","mappings":[{"from":1,"to":7}]},{"pgid":"14.36","mappings":[{"from":1,"to":5}]},{"pgid":"14.38","mappings":[{"from":1,"to":2}]},{"pgid":"14.3a","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6833/1336395002":"2024-03-25T00:46:20.877877+0000","172.21.15.67:6832/1336395002":"2024-03-25T00:46:20.877877+0000","172.21.15.67:6833/3021622863":"2024-03-25T00:45:09.102827+0000","172.21.15.67:6832/3021622863":"2024-03-25T00:45:09.102827+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:46:59.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.368+0000 7fcf4713b640 1 -- 172.21.15.67:0/3057873342 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcf2406bbd0 msgr2=0x7fcf2406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:59.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.368+0000 7fcf4713b640 1 --2- 172.21.15.67:0/3057873342 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcf2406bbd0 0x7fcf2406e090 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c036000 tx=0x7fcf2c008030 comp rx=0 tx=0).stop 2024-03-24T00:46:59.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.368+0000 7fcf4713b640 1 -- 172.21.15.67:0/3057873342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf400d07e0 msgr2=0x7fcf40158cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:59.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.368+0000 7fcf4713b640 1 --2- 172.21.15.67:0/3057873342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf400d07e0 0x7fcf40158cc0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7fcf3400c7f0 tx=0x7fcf3400ccb0 comp rx=0 tx=0).stop 2024-03-24T00:46:59.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.368+0000 7fcf4713b640 1 -- 172.21.15.67:0/3057873342 shutdown_connections 2024-03-24T00:46:59.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.368+0000 7fcf4713b640 1 --2- 172.21.15.67:0/3057873342 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcf2406bbd0 0x7fcf2406e090 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:59.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.368+0000 7fcf4713b640 1 --2- 172.21.15.67:0/3057873342 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcf4015e2a0 0x7fcf400b3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:59.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.368+0000 7fcf4713b640 1 --2- 172.21.15.67:0/3057873342 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcf4015dec0 0x7fcf400b3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:59.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.368+0000 7fcf4713b640 1 --2- 172.21.15.67:0/3057873342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf400d07e0 0x7fcf40158cc0 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:59.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.368+0000 7fcf4713b640 1 -- 172.21.15.67:0/3057873342 >> 172.21.15.67:0/3057873342 conn(0x7fcf4006c810 msgr2=0x7fcf4015d400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:59.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.368+0000 7fcf4713b640 1 -- 172.21.15.67:0/3057873342 shutdown_connections 2024-03-24T00:46:59.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.368+0000 7fcf4713b640 1 -- 172.21.15.67:0/3057873342 wait complete. 2024-03-24T00:46:59.380 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:38:39.364950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:44:53.025965+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:44:53.712957+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:46:59.381 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:46:59.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.484+0000 7f49e27b9640 1 Processor -- start 2024-03-24T00:46:59.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.484+0000 7f49e27b9640 1 -- start start 2024-03-24T00:46:59.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.485+0000 7f49e27b9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f49dc15dea0 0x7f49dc0d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:59.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.485+0000 7f49e27b9640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f49dc15e280 0x7f49dc0d3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:59.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.485+0000 7f49e27b9640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49dc0d2250 con 0x7f49dc0b5a60 2024-03-24T00:46:59.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.485+0000 7f49e27b9640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49dc0d2530 con 0x7f49dc15dea0 2024-03-24T00:46:59.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.485+0000 7f49e27b9640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49dc0d2810 con 0x7f49dc15e280 2024-03-24T00:46:59.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.485+0000 7f49dbfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f49dc15dea0 0x7f49dc0d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:59.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.485+0000 7f49e0d56640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f49dc0b5a60 0x7f49dc06c640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44762/0 (socket says 172.21.15.67:44762) 2024-03-24T00:46:59.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.485+0000 7f49e0d56640 1 -- 172.21.15.67:0/2601940326 learned_addr learned my addr 172.21.15.67:0/2601940326 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:46:59.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.486+0000 7f49db7fe640 1 --2- 172.21.15.67:0/2601940326 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f49dc15e280 0x7f49dc0d3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:59.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.486+0000 7f49dbfff640 1 -- 172.21.15.67:0/2601940326 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f49dc15e280 msgr2=0x7f49dc0d3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:59.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.486+0000 7f49dbfff640 1 --2- 172.21.15.67:0/2601940326 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f49dc15e280 0x7f49dc0d3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:59.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.486+0000 7f49dbfff640 1 -- 172.21.15.67:0/2601940326 >> v1:172.21.15.67:6789/0 conn(0x7f49dc0b5a60 legacy=0x7f49dc06c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:59.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.486+0000 7f49dbfff640 1 -- 172.21.15.67:0/2601940326 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49dc0d2af0 con 0x7f49dc15dea0 2024-03-24T00:46:59.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.486+0000 7f49daffd640 1 -- 172.21.15.67:0/2601940326 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3443505761 0 0) 0x7f49dc0d2250 con 0x7f49dc0b5a60 2024-03-24T00:46:59.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.486+0000 7f49db7fe640 1 --2- 172.21.15.67:0/2601940326 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f49dc15e280 0x7f49dc0d3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:46:59.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.486+0000 7f49dbfff640 1 --2- 172.21.15.67:0/2601940326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49dc15dea0 0x7f49dc0d33d0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f49cc0097a0 tx=0x7f49cc02b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39b3772a03dbaea0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:59.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.487+0000 7f49daffd640 1 -- 172.21.15.67:0/2601940326 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49cc031780 con 0x7f49dc15dea0 2024-03-24T00:46:59.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.487+0000 7f49daffd640 1 -- 172.21.15.67:0/2601940326 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49cc037070 con 0x7f49dc15dea0 2024-03-24T00:46:59.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.487+0000 7f49daffd640 1 -- 172.21.15.67:0/2601940326 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49cc031b40 con 0x7f49dc15dea0 2024-03-24T00:46:59.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.487+0000 7f49e27b9640 1 -- 172.21.15.67:0/2601940326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49dc15dea0 msgr2=0x7f49dc0d33d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:59.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.487+0000 7f49e27b9640 1 --2- 172.21.15.67:0/2601940326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49dc15dea0 0x7f49dc0d33d0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f49cc0097a0 tx=0x7f49cc02b820 comp rx=0 tx=0).stop 2024-03-24T00:46:59.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.487+0000 7f49e27b9640 1 -- 172.21.15.67:0/2601940326 shutdown_connections 2024-03-24T00:46:59.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.487+0000 7f49e27b9640 1 --2- 172.21.15.67:0/2601940326 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f49dc15e280 0x7f49dc0d3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:59.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.487+0000 7f49e27b9640 1 --2- 172.21.15.67:0/2601940326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49dc15dea0 0x7f49dc0d33d0 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:59.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.487+0000 7f49e27b9640 1 -- 172.21.15.67:0/2601940326 >> 172.21.15.67:0/2601940326 conn(0x7f49dc072da0 msgr2=0x7f49dc0d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:59.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.487+0000 7f49e27b9640 1 -- 172.21.15.67:0/2601940326 shutdown_connections 2024-03-24T00:46:59.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.487+0000 7f49e27b9640 1 -- 172.21.15.67:0/2601940326 wait complete. 2024-03-24T00:46:59.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.488+0000 7f49e27b9640 1 Processor -- start 2024-03-24T00:46:59.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.489+0000 7f49e27b9640 1 -- start start 2024-03-24T00:46:59.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.489+0000 7f49e27b9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49dc0b5a60 0x7f49dc14f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:59.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.489+0000 7f49e27b9640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f49dc15dea0 0x7f49dc0cd730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:59.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.489+0000 7f49e27b9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f49dc15e280 0x7f49dc0cdc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:59.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.489+0000 7f49e27b9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49dc0b5ed0 con 0x7f49dc0b5a60 2024-03-24T00:46:59.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.489+0000 7f49e27b9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49dc0b8350 con 0x7f49dc15e280 2024-03-24T00:46:59.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.489+0000 7f49e27b9640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49dc0b6050 con 0x7f49dc15dea0 2024-03-24T00:46:59.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.489+0000 7f49dbfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49dc0b5a60 0x7f49dc14f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:59.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.489+0000 7f49dbfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49dc0b5a60 0x7f49dc14f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50800/0 (socket says 172.21.15.67:50800) 2024-03-24T00:46:59.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.489+0000 7f49dbfff640 1 -- 172.21.15.67:0/1725345427 learned_addr learned my addr 172.21.15.67:0/1725345427 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:59.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.490+0000 7f49db7fe640 1 --2- 172.21.15.67:0/1725345427 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f49dc15dea0 0x7f49dc0cd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:59.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.490+0000 7f49e0d56640 1 --2- 172.21.15.67:0/1725345427 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f49dc15e280 0x7f49dc0cdc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:59.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.490+0000 7f49dbfff640 1 -- 172.21.15.67:0/1725345427 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f49dc15dea0 msgr2=0x7f49dc0cd730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:59.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.490+0000 7f49dbfff640 1 --2- 172.21.15.67:0/1725345427 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f49dc15dea0 0x7f49dc0cd730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:59.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.490+0000 7f49dbfff640 1 -- 172.21.15.67:0/1725345427 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f49dc15e280 msgr2=0x7f49dc0cdc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:59.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.490+0000 7f49dbfff640 1 --2- 172.21.15.67:0/1725345427 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f49dc15e280 0x7f49dc0cdc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:59.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.490+0000 7f49dbfff640 1 -- 172.21.15.67:0/1725345427 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49dc14f970 con 0x7f49dc0b5a60 2024-03-24T00:46:59.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.490+0000 7f49e0d56640 1 --2- 172.21.15.67:0/1725345427 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f49dc15e280 0x7f49dc0cdc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:46:59.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.490+0000 7f49dbfff640 1 --2- 172.21.15.67:0/1725345427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49dc0b5a60 0x7f49dc14f430 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f49cc0097a0 tx=0x7f49cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:59.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.490+0000 7f49d8ff9640 1 -- 172.21.15.67:0/1725345427 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49cc004230 con 0x7f49dc0b5a60 2024-03-24T00:46:59.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.490+0000 7f49d8ff9640 1 -- 172.21.15.67:0/1725345427 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49cc0043d0 con 0x7f49dc0b5a60 2024-03-24T00:46:59.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.490+0000 7f49e27b9640 1 -- 172.21.15.67:0/1725345427 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49dc0ac270 con 0x7f49dc0b5a60 2024-03-24T00:46:59.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.490+0000 7f49d8ff9640 1 -- 172.21.15.67:0/1725345427 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49cc003850 con 0x7f49dc0b5a60 2024-03-24T00:46:59.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.490+0000 7f49e27b9640 1 -- 172.21.15.67:0/1725345427 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49dc0abcb0 con 0x7f49dc0b5a60 2024-03-24T00:46:59.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.491+0000 7f49e27b9640 1 -- 172.21.15.67:0/1725345427 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49dc04f400 con 0x7f49dc0b5a60 2024-03-24T00:46:59.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.492+0000 7f49d8ff9640 1 -- 172.21.15.67:0/1725345427 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f49cc0039f0 con 0x7f49dc0b5a60 2024-03-24T00:46:59.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.492+0000 7f49d8ff9640 1 --2- 172.21.15.67:0/1725345427 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f49c806bbd0 0x7f49c806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:59.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.492+0000 7f49d8ff9640 1 -- 172.21.15.67:0/1725345427 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f49cc0acb60 con 0x7f49dc0b5a60 2024-03-24T00:46:59.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.492+0000 7f49db7fe640 1 --2- 172.21.15.67:0/1725345427 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f49c806bbd0 0x7f49c806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:59.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.493+0000 7f49db7fe640 1 --2- 172.21.15.67:0/1725345427 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f49c806bbd0 0x7f49c806e090 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f49dc0cd5f0 tx=0x7f49d0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:59.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.497+0000 7f49d8ff9640 1 -- 172.21.15.67:0/1725345427 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f49dc14f970 con 0x7f49dc0b5a60 2024-03-24T00:46:59.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.679+0000 7f49e27b9640 1 -- 172.21.15.67:0/1725345427 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f49dc0ceb60 con 0x7f49dc0b5a60 2024-03-24T00:46:59.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.679+0000 7f49d8ff9640 1 -- 172.21.15.67:0/1725345427 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7f49cc041070 con 0x7f49dc0b5a60 2024-03-24T00:46:59.681 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-24T00:46:59.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.681+0000 7f49e27b9640 1 -- 172.21.15.67:0/1725345427 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f49c806bbd0 msgr2=0x7f49c806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:59.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.681+0000 7f49e27b9640 1 --2- 172.21.15.67:0/1725345427 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f49c806bbd0 0x7f49c806e090 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f49dc0cd5f0 tx=0x7f49d0008040 comp rx=0 tx=0).stop 2024-03-24T00:46:59.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.681+0000 7f49e27b9640 1 -- 172.21.15.67:0/1725345427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49dc0b5a60 msgr2=0x7f49dc14f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:59.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.681+0000 7f49e27b9640 1 --2- 172.21.15.67:0/1725345427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49dc0b5a60 0x7f49dc14f430 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f49cc0097a0 tx=0x7f49cc0046c0 comp rx=0 tx=0).stop 2024-03-24T00:46:59.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.681+0000 7f49e27b9640 1 -- 172.21.15.67:0/1725345427 shutdown_connections 2024-03-24T00:46:59.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.681+0000 7f49e27b9640 1 --2- 172.21.15.67:0/1725345427 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f49c806bbd0 0x7f49c806e090 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:59.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.681+0000 7f49e27b9640 1 --2- 172.21.15.67:0/1725345427 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f49dc15e280 0x7f49dc0cdc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:59.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.681+0000 7f49e27b9640 1 --2- 172.21.15.67:0/1725345427 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f49dc15dea0 0x7f49dc0cd730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:59.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.681+0000 7f49e27b9640 1 --2- 172.21.15.67:0/1725345427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49dc0b5a60 0x7f49dc14f430 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:59.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.681+0000 7f49e27b9640 1 -- 172.21.15.67:0/1725345427 >> 172.21.15.67:0/1725345427 conn(0x7f49dc072da0 msgr2=0x7f49dc0ca8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:59.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.681+0000 7f49e27b9640 1 -- 172.21.15.67:0/1725345427 shutdown_connections 2024-03-24T00:46:59.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.681+0000 7f49e27b9640 1 -- 172.21.15.67:0/1725345427 wait complete. 2024-03-24T00:46:59.697 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:46:59.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.800+0000 7f68ea468640 1 Processor -- start 2024-03-24T00:46:59.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.800+0000 7f68ea468640 1 -- start start 2024-03-24T00:46:59.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.801+0000 7f68ea468640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f68e415dea0 0x7f68e40d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:59.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.801+0000 7f68ea468640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f68e415e280 0x7f68e40d3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:59.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.801+0000 7f68ea468640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f68e40b5a60 0x7f68e40b5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:59.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.801+0000 7f68ea468640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68e40b6410 con 0x7f68e415e280 2024-03-24T00:46:59.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.801+0000 7f68ea468640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68e40d2250 con 0x7f68e40b5a60 2024-03-24T00:46:59.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.801+0000 7f68ea468640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68e40d2530 con 0x7f68e415dea0 2024-03-24T00:46:59.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.801+0000 7f68e37fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f68e415e280 0x7f68e40d3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:59.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.801+0000 7f68e37fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f68e415e280 0x7f68e40d3910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50816/0 (socket says 172.21.15.67:50816) 2024-03-24T00:46:59.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.801+0000 7f68e37fe640 1 -- 172.21.15.67:0/1594606954 learned_addr learned my addr 172.21.15.67:0/1594606954 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:59.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.801+0000 7f68e8a05640 1 --2- 172.21.15.67:0/1594606954 >> v2:172.21.15.72:3300/0 conn(0x7f68e40b5a60 0x7f68e40b5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:59.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.801+0000 7f68e3fff640 1 --2- 172.21.15.67:0/1594606954 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f68e415dea0 0x7f68e40d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:59.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.802+0000 7f68e37fe640 1 -- 172.21.15.67:0/1594606954 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f68e415dea0 msgr2=0x7f68e40d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:59.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.802+0000 7f68e37fe640 1 --2- 172.21.15.67:0/1594606954 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f68e415dea0 0x7f68e40d33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:59.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.802+0000 7f68e37fe640 1 -- 172.21.15.67:0/1594606954 >> v2:172.21.15.72:3300/0 conn(0x7f68e40b5a60 msgr2=0x7f68e40b5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:46:59.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.802+0000 7f68e37fe640 1 --2- 172.21.15.67:0/1594606954 >> v2:172.21.15.72:3300/0 conn(0x7f68e40b5a60 0x7f68e40b5ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:59.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.802+0000 7f68e37fe640 1 -- 172.21.15.67:0/1594606954 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68e40d2810 con 0x7f68e415e280 2024-03-24T00:46:59.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.802+0000 7f68e8a05640 1 --2- 172.21.15.67:0/1594606954 >> v2:172.21.15.72:3300/0 conn(0x7f68e40b5a60 0x7f68e40b5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:46:59.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.802+0000 7f68e37fe640 1 --2- 172.21.15.67:0/1594606954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68e415e280 0x7f68e40d3910 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f68e40d2460 tx=0x7f68d802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e306408c52da0779 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:59.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.802+0000 7f68e2ffd640 1 -- 172.21.15.67:0/1594606954 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d80317c0 con 0x7f68e415e280 2024-03-24T00:46:59.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.802+0000 7f68e2ffd640 1 -- 172.21.15.67:0/1594606954 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68d8037070 con 0x7f68e415e280 2024-03-24T00:46:59.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.802+0000 7f68e2ffd640 1 -- 172.21.15.67:0/1594606954 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d8031b80 con 0x7f68e415e280 2024-03-24T00:46:59.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.803+0000 7f68ea468640 1 -- 172.21.15.67:0/1594606954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68e415e280 msgr2=0x7f68e40d3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:59.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.803+0000 7f68ea468640 1 --2- 172.21.15.67:0/1594606954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68e415e280 0x7f68e40d3910 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f68e40d2460 tx=0x7f68d802b860 comp rx=0 tx=0).stop 2024-03-24T00:46:59.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.803+0000 7f68ea468640 1 -- 172.21.15.67:0/1594606954 shutdown_connections 2024-03-24T00:46:59.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.803+0000 7f68ea468640 1 --2- 172.21.15.67:0/1594606954 >> v2:172.21.15.72:3300/0 conn(0x7f68e40b5a60 0x7f68e40b5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:59.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.803+0000 7f68ea468640 1 --2- 172.21.15.67:0/1594606954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68e415e280 0x7f68e40d3910 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:59.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.803+0000 7f68ea468640 1 --2- 172.21.15.67:0/1594606954 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f68e415dea0 0x7f68e40d33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:59.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.803+0000 7f68ea468640 1 -- 172.21.15.67:0/1594606954 >> 172.21.15.67:0/1594606954 conn(0x7f68e4072da0 msgr2=0x7f68e40d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:46:59.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.803+0000 7f68ea468640 1 -- 172.21.15.67:0/1594606954 shutdown_connections 2024-03-24T00:46:59.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.803+0000 7f68ea468640 1 -- 172.21.15.67:0/1594606954 wait complete. 2024-03-24T00:46:59.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.804+0000 7f68ea468640 1 Processor -- start 2024-03-24T00:46:59.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.804+0000 7f68ea468640 1 -- start start 2024-03-24T00:46:59.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.805+0000 7f68ea468640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f68e40b5a60 0x7f68e4154c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:59.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.805+0000 7f68ea468640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f68e415dea0 0x7f68e4156400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:59.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.805+0000 7f68ea468640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68e415e280 0x7f68e415fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:59.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.805+0000 7f68ea468640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e406c7c0 con 0x7f68e415e280 2024-03-24T00:46:59.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.805+0000 7f68ea468640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e406c640 con 0x7f68e40b5a60 2024-03-24T00:46:59.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.805+0000 7f68ea468640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68e406c940 con 0x7f68e415dea0 2024-03-24T00:46:59.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.805+0000 7f68e8a05640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68e415e280 0x7f68e415fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:59.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.805+0000 7f68e8a05640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68e415e280 0x7f68e415fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50832/0 (socket says 172.21.15.67:50832) 2024-03-24T00:46:59.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.805+0000 7f68e8a05640 1 -- 172.21.15.67:0/3737480306 learned_addr learned my addr 172.21.15.67:0/3737480306 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:46:59.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.805+0000 7f68e3fff640 1 --2- 172.21.15.67:0/3737480306 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f68e40b5a60 0x7f68e4154c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:59.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.805+0000 7f68e37fe640 1 --2- 172.21.15.67:0/3737480306 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f68e415dea0 0x7f68e4156400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:59.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.805+0000 7f68e8a05640 1 -- 172.21.15.67:0/3737480306 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f68e415dea0 msgr2=0x7f68e4156400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:59.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.805+0000 7f68e8a05640 1 --2- 172.21.15.67:0/3737480306 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f68e415dea0 0x7f68e4156400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:59.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.805+0000 7f68e8a05640 1 -- 172.21.15.67:0/3737480306 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f68e40b5a60 msgr2=0x7f68e4154c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:46:59.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.805+0000 7f68e8a05640 1 --2- 172.21.15.67:0/3737480306 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f68e40b5a60 0x7f68e4154c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:59.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.805+0000 7f68e8a05640 1 -- 172.21.15.67:0/3737480306 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68e4156940 con 0x7f68e415e280 2024-03-24T00:46:59.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.805+0000 7f68e3fff640 1 --2- 172.21.15.67:0/3737480306 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f68e40b5a60 0x7f68e4154c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:46:59.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.806+0000 7f68e8a05640 1 --2- 172.21.15.67:0/3737480306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68e415e280 0x7f68e415fce0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f68dc007d60 tx=0x7f68dc00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:59.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.806+0000 7f68e0ff9640 1 -- 172.21.15.67:0/3737480306 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68dc01a070 con 0x7f68e415e280 2024-03-24T00:46:59.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.806+0000 7f68e0ff9640 1 -- 172.21.15.67:0/3737480306 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68dc003990 con 0x7f68e415e280 2024-03-24T00:46:59.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.806+0000 7f68e0ff9640 1 -- 172.21.15.67:0/3737480306 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68dc003d80 con 0x7f68e415e280 2024-03-24T00:46:59.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.806+0000 7f68ea468640 1 -- 172.21.15.67:0/3737480306 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68e4155a60 con 0x7f68e415e280 2024-03-24T00:46:59.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.806+0000 7f68ea468640 1 -- 172.21.15.67:0/3737480306 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68e415ef20 con 0x7f68e415e280 2024-03-24T00:46:59.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.807+0000 7f68ea468640 1 -- 172.21.15.67:0/3737480306 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68e404f400 con 0x7f68e415e280 2024-03-24T00:46:59.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.807+0000 7f68e0ff9640 1 -- 172.21.15.67:0/3737480306 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f68dc01d070 con 0x7f68e415e280 2024-03-24T00:46:59.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.808+0000 7f68e0ff9640 1 --2- 172.21.15.67:0/3737480306 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f68d006bbd0 0x7f68d006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:59.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.808+0000 7f68e0ff9640 1 -- 172.21.15.67:0/3737480306 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f68dc08aeb0 con 0x7f68e415e280 2024-03-24T00:46:59.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.808+0000 7f68e3fff640 1 --2- 172.21.15.67:0/3737480306 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f68d006bbd0 0x7f68d006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:46:59.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.808+0000 7f68e3fff640 1 --2- 172.21.15.67:0/3737480306 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f68d006bbd0 0x7f68d006e090 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f68d4009b90 tx=0x7f68d4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:46:59.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.816+0000 7f68e0ff9640 1 -- 172.21.15.67:0/3737480306 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f68dc010070 con 0x7f68e415e280 2024-03-24T00:47:00.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.999+0000 7f68ea468640 1 -- 172.21.15.67:0/3737480306 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f68e406b990 con 0x7f68e415e280 2024-03-24T00:47:00.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:46:59.999+0000 7f68e0ff9640 1 -- 172.21.15.67:0/3737480306 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) v1 ==== 104+0+11 (secure 0 0 0) 0x7f68dc0590a0 con 0x7f68e415e280 2024-03-24T00:47:00.000 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-24T00:47:00.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.001+0000 7f68ea468640 1 -- 172.21.15.67:0/3737480306 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f68d006bbd0 msgr2=0x7f68d006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:00.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.001+0000 7f68ea468640 1 --2- 172.21.15.67:0/3737480306 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f68d006bbd0 0x7f68d006e090 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f68d4009b90 tx=0x7f68d4008040 comp rx=0 tx=0).stop 2024-03-24T00:47:00.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.001+0000 7f68ea468640 1 -- 172.21.15.67:0/3737480306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68e415e280 msgr2=0x7f68e415fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:00.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.001+0000 7f68ea468640 1 --2- 172.21.15.67:0/3737480306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68e415e280 0x7f68e415fce0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f68dc007d60 tx=0x7f68dc00d4a0 comp rx=0 tx=0).stop 2024-03-24T00:47:00.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.001+0000 7f68ea468640 1 -- 172.21.15.67:0/3737480306 shutdown_connections 2024-03-24T00:47:00.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.001+0000 7f68ea468640 1 --2- 172.21.15.67:0/3737480306 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f68d006bbd0 0x7f68d006e090 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:00.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.001+0000 7f68ea468640 1 --2- 172.21.15.67:0/3737480306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68e415e280 0x7f68e415fce0 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:00.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.001+0000 7f68ea468640 1 --2- 172.21.15.67:0/3737480306 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f68e415dea0 0x7f68e4156400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:00.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.001+0000 7f68ea468640 1 --2- 172.21.15.67:0/3737480306 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f68e40b5a60 0x7f68e4154c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:00.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.001+0000 7f68ea468640 1 -- 172.21.15.67:0/3737480306 >> 172.21.15.67:0/3737480306 conn(0x7f68e4072da0 msgr2=0x7f68e40d3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:00.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.002+0000 7f68ea468640 1 -- 172.21.15.67:0/3737480306 shutdown_connections 2024-03-24T00:47:00.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.002+0000 7f68ea468640 1 -- 172.21.15.67:0/3737480306 wait complete. 2024-03-24T00:47:00.013 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:47:00.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.118+0000 7f6cfb1f4640 1 Processor -- start 2024-03-24T00:47:00.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.118+0000 7f6cfb1f4640 1 -- start start 2024-03-24T00:47:00.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.119+0000 7f6cfb1f4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6cf415dea0 0x7f6cf40d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:00.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.119+0000 7f6cfb1f4640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cf40d2250 con 0x7f6cf415e280 2024-03-24T00:47:00.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.119+0000 7f6cfb1f4640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cf40d2530 con 0x7f6cf40b6170 2024-03-24T00:47:00.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.119+0000 7f6cfb1f4640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cf40d2810 con 0x7f6cf415dea0 2024-03-24T00:47:00.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.119+0000 7f6cf8f90640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6cf415dea0 0x7f6cf40d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:00.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.119+0000 7f6cebfff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6cf415e280 0x7f6cf40b5a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44768/0 (socket says 172.21.15.67:44768) 2024-03-24T00:47:00.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.119+0000 7f6cebfff640 1 -- 172.21.15.67:0/3516881861 learned_addr learned my addr 172.21.15.67:0/3516881861 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:47:00.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.119+0000 7f6cf8f90640 1 -- 172.21.15.67:0/3516881861 >> v1:172.21.15.72:6789/0 conn(0x7f6cf40b6170 legacy=0x7f6cf40b70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:00.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.119+0000 7f6cf8f90640 1 -- 172.21.15.67:0/3516881861 >> v1:172.21.15.67:6789/0 conn(0x7f6cf415e280 legacy=0x7f6cf40b5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:00.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.119+0000 7f6cf8f90640 1 -- 172.21.15.67:0/3516881861 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cf40d2af0 con 0x7f6cf415dea0 2024-03-24T00:47:00.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.119+0000 7f6ceb7fe640 1 -- 172.21.15.67:0/3516881861 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3632736666 0 0) 0x7f6cf40d2250 con 0x7f6cf415e280 2024-03-24T00:47:00.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.120+0000 7f6cf8f90640 1 --2- 172.21.15.67:0/3516881861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf415dea0 0x7f6cf40d33d0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f6cf00097d0 tx=0x7f6cf002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4be78951ec33fc29 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:00.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.120+0000 7f6ceb7fe640 1 -- 172.21.15.67:0/3516881861 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf0035020 con 0x7f6cf415dea0 2024-03-24T00:47:00.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.120+0000 7f6ceb7fe640 1 -- 172.21.15.67:0/3516881861 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cf0031830 con 0x7f6cf415dea0 2024-03-24T00:47:00.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.120+0000 7f6ceb7fe640 1 -- 172.21.15.67:0/3516881861 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf0031c00 con 0x7f6cf415dea0 2024-03-24T00:47:00.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.121+0000 7f6cfb1f4640 1 -- 172.21.15.67:0/3516881861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf415dea0 msgr2=0x7f6cf40d33d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:00.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.121+0000 7f6cfb1f4640 1 --2- 172.21.15.67:0/3516881861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf415dea0 0x7f6cf40d33d0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f6cf00097d0 tx=0x7f6cf002b8d0 comp rx=0 tx=0).stop 2024-03-24T00:47:00.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.121+0000 7f6cfb1f4640 1 -- 172.21.15.67:0/3516881861 shutdown_connections 2024-03-24T00:47:00.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.121+0000 7f6cfb1f4640 1 --2- 172.21.15.67:0/3516881861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf415dea0 0x7f6cf40d33d0 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:00.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.121+0000 7f6cfb1f4640 1 -- 172.21.15.67:0/3516881861 >> 172.21.15.67:0/3516881861 conn(0x7f6cf4072da0 msgr2=0x7f6cf40d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:00.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.121+0000 7f6cfb1f4640 1 -- 172.21.15.67:0/3516881861 shutdown_connections 2024-03-24T00:47:00.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.121+0000 7f6cfb1f4640 1 -- 172.21.15.67:0/3516881861 wait complete. 2024-03-24T00:47:00.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.122+0000 7f6cfb1f4640 1 Processor -- start 2024-03-24T00:47:00.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.122+0000 7f6cfb1f4640 1 -- start start 2024-03-24T00:47:00.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.123+0000 7f6cfb1f4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6cf40b6170 0x7f6cf415b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:00.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.123+0000 7f6cfb1f4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6cf415dea0 0x7f6cf4144aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:00.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.123+0000 7f6cfb1f4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf415e280 0x7f6cf4144fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:00.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.123+0000 7f6cfb1f4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf40d3ca0 con 0x7f6cf415e280 2024-03-24T00:47:00.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.123+0000 7f6cfb1f4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf40d3b20 con 0x7f6cf415dea0 2024-03-24T00:47:00.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.123+0000 7f6cfb1f4640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf40b7800 con 0x7f6cf40b6170 2024-03-24T00:47:00.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.123+0000 7f6cf9791640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf415e280 0x7f6cf4144fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:00.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.123+0000 7f6cf9791640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf415e280 0x7f6cf4144fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50858/0 (socket says 172.21.15.67:50858) 2024-03-24T00:47:00.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.123+0000 7f6cf9791640 1 -- 172.21.15.67:0/791762685 learned_addr learned my addr 172.21.15.67:0/791762685 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:00.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.123+0000 7f6cebfff640 1 --2- 172.21.15.67:0/791762685 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6cf415dea0 0x7f6cf4144aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:00.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.123+0000 7f6cf8f90640 1 --2- 172.21.15.67:0/791762685 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6cf40b6170 0x7f6cf415b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:00.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.123+0000 7f6cf9791640 1 -- 172.21.15.67:0/791762685 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6cf40b6170 msgr2=0x7f6cf415b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:00.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.123+0000 7f6cf9791640 1 --2- 172.21.15.67:0/791762685 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6cf40b6170 0x7f6cf415b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:00.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.123+0000 7f6cf9791640 1 -- 172.21.15.67:0/791762685 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6cf415dea0 msgr2=0x7f6cf4144aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:00.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.123+0000 7f6cf9791640 1 --2- 172.21.15.67:0/791762685 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6cf415dea0 0x7f6cf4144aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:00.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.123+0000 7f6cf9791640 1 -- 172.21.15.67:0/791762685 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cf415a610 con 0x7f6cf415e280 2024-03-24T00:47:00.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.123+0000 7f6cebfff640 1 --2- 172.21.15.67:0/791762685 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6cf415dea0 0x7f6cf4144aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:47:00.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.124+0000 7f6cf9791640 1 --2- 172.21.15.67:0/791762685 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf415e280 0x7f6cf4144fe0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f6cec00c760 tx=0x7f6cec00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:00.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.124+0000 7f6ce97fa640 1 -- 172.21.15.67:0/791762685 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cec003b70 con 0x7f6cf415e280 2024-03-24T00:47:00.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.124+0000 7f6ce97fa640 1 -- 172.21.15.67:0/791762685 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cec003d30 con 0x7f6cf415e280 2024-03-24T00:47:00.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.124+0000 7f6cfb1f4640 1 -- 172.21.15.67:0/791762685 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cf4159850 con 0x7f6cf415e280 2024-03-24T00:47:00.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.124+0000 7f6ce97fa640 1 -- 172.21.15.67:0/791762685 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cec00aa90 con 0x7f6cf415e280 2024-03-24T00:47:00.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.124+0000 7f6cfb1f4640 1 -- 172.21.15.67:0/791762685 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cf4159e10 con 0x7f6cf415e280 2024-03-24T00:47:00.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.125+0000 7f6cfb1f4640 1 -- 172.21.15.67:0/791762685 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cf415a140 con 0x7f6cf415e280 2024-03-24T00:47:00.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.125+0000 7f6ce97fa640 1 -- 172.21.15.67:0/791762685 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6cec00ac30 con 0x7f6cf415e280 2024-03-24T00:47:00.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.126+0000 7f6ce97fa640 1 --2- 172.21.15.67:0/791762685 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6ce006bbd0 0x7f6ce006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:00.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.126+0000 7f6ce97fa640 1 -- 172.21.15.67:0/791762685 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f6cec08aac0 con 0x7f6cf415e280 2024-03-24T00:47:00.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.126+0000 7f6cf8f90640 1 --2- 172.21.15.67:0/791762685 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6ce006bbd0 0x7f6ce006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:00.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.127+0000 7f6cf8f90640 1 --2- 172.21.15.67:0/791762685 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6ce006bbd0 0x7f6ce006e090 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f6cf0036000 tx=0x7f6cf0008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:00.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.130+0000 7f6ce97fa640 1 -- 172.21.15.67:0/791762685 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6cec059390 con 0x7f6cf415e280 2024-03-24T00:47:00.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.311+0000 7f6cfb1f4640 1 -- 172.21.15.67:0/791762685 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6cf404f400 con 0x7f6cf415e280 2024-03-24T00:47:00.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.311+0000 7f6ce97fa640 1 -- 172.21.15.67:0/791762685 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6cec058c90 con 0x7f6cf415e280 2024-03-24T00:47:00.312 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-03-24T00:47:00.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.313+0000 7f6cfb1f4640 1 -- 172.21.15.67:0/791762685 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6ce006bbd0 msgr2=0x7f6ce006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:00.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.313+0000 7f6cfb1f4640 1 --2- 172.21.15.67:0/791762685 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6ce006bbd0 0x7f6ce006e090 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f6cf0036000 tx=0x7f6cf0008030 comp rx=0 tx=0).stop 2024-03-24T00:47:00.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.313+0000 7f6cfb1f4640 1 -- 172.21.15.67:0/791762685 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf415e280 msgr2=0x7f6cf4144fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:00.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.313+0000 7f6cfb1f4640 1 --2- 172.21.15.67:0/791762685 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf415e280 0x7f6cf4144fe0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f6cec00c760 tx=0x7f6cec00cc20 comp rx=0 tx=0).stop 2024-03-24T00:47:00.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.313+0000 7f6cfb1f4640 1 -- 172.21.15.67:0/791762685 shutdown_connections 2024-03-24T00:47:00.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.313+0000 7f6cfb1f4640 1 --2- 172.21.15.67:0/791762685 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6ce006bbd0 0x7f6ce006e090 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:00.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.313+0000 7f6cfb1f4640 1 --2- 172.21.15.67:0/791762685 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf415e280 0x7f6cf4144fe0 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:00.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.313+0000 7f6cfb1f4640 1 --2- 172.21.15.67:0/791762685 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6cf415dea0 0x7f6cf4144aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:00.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.313+0000 7f6cfb1f4640 1 --2- 172.21.15.67:0/791762685 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6cf40b6170 0x7f6cf415b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:00.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.313+0000 7f6cfb1f4640 1 -- 172.21.15.67:0/791762685 >> 172.21.15.67:0/791762685 conn(0x7f6cf4072da0 msgr2=0x7f6cf40b5620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:00.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.313+0000 7f6cfb1f4640 1 -- 172.21.15.67:0/791762685 shutdown_connections 2024-03-24T00:47:00.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.313+0000 7f6cfb1f4640 1 -- 172.21.15.67:0/791762685 wait complete. 2024-03-24T00:47:00.325 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T00:47:00.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.426+0000 7f1cdb412640 1 Processor -- start 2024-03-24T00:47:00.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.426+0000 7f1cdb412640 1 -- start start 2024-03-24T00:47:00.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.427+0000 7f1cdb412640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1cd40d07a0 0x7f1cd40cc770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:00.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.427+0000 7f1cdb412640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1cd40cccb0 0x7f1cd40b2790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:00.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.427+0000 7f1cdb412640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cd40cd160 con 0x7f1cd415c010 2024-03-24T00:47:00.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.427+0000 7f1cdb412640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cd40cfbe0 con 0x7f1cd40cccb0 2024-03-24T00:47:00.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.427+0000 7f1cdb412640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd40b2020 con 0x7f1cd40d07a0 2024-03-24T00:47:00.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.427+0000 7f1cd91ae640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1cd415c010 0x7f1cd40d0090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44774/0 (socket says 172.21.15.67:44774) 2024-03-24T00:47:00.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.427+0000 7f1cd91ae640 1 -- 172.21.15.67:0/2297725366 learned_addr learned my addr 172.21.15.67:0/2297725366 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:47:00.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.427+0000 7f1cd99af640 1 --2- 172.21.15.67:0/2297725366 >> v2:172.21.15.72:3300/0 conn(0x7f1cd40cccb0 0x7f1cd40b2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:00.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.427+0000 7f1cd89ad640 1 --2- 172.21.15.67:0/2297725366 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1cd40d07a0 0x7f1cd40cc770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:00.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.427+0000 7f1ccbfff640 1 -- 172.21.15.67:0/2297725366 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3101862229 0 0) 0x7f1cd40cd160 con 0x7f1cd415c010 2024-03-24T00:47:00.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.427+0000 7f1ccbfff640 1 -- 172.21.15.67:0/2297725366 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cc0003520 con 0x7f1cd415c010 2024-03-24T00:47:00.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.427+0000 7f1ccbfff640 1 -- 172.21.15.67:0/2297725366 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2649622045 0 0) 0x7f1cc0003520 con 0x7f1cd415c010 2024-03-24T00:47:00.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.427+0000 7f1cd99af640 1 -- 172.21.15.67:0/2297725366 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1cd40d07a0 msgr2=0x7f1cd40cc770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:00.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.427+0000 7f1cd99af640 1 --2- 172.21.15.67:0/2297725366 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1cd40d07a0 0x7f1cd40cc770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:00.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.428+0000 7f1cd99af640 1 -- 172.21.15.67:0/2297725366 >> v1:172.21.15.67:6789/0 conn(0x7f1cd415c010 legacy=0x7f1cd40d0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:00.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.428+0000 7f1cd99af640 1 -- 172.21.15.67:0/2297725366 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cd40cd430 con 0x7f1cd40cccb0 2024-03-24T00:47:00.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.428+0000 7f1cd89ad640 1 --2- 172.21.15.67:0/2297725366 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1cd40d07a0 0x7f1cd40cc770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:47:00.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.428+0000 7f1cd99af640 1 --2- 172.21.15.67:0/2297725366 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1cd40cccb0 0x7f1cd40b2790 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc009d90 tx=0x7f1ccc02bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a92913f0eb28c4d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:00.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.429+0000 7f1ccbfff640 1 -- 172.21.15.67:0/2297725366 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc039070 con 0x7f1cd40cccb0 2024-03-24T00:47:00.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.429+0000 7f1ccbfff640 1 -- 172.21.15.67:0/2297725366 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ccc02c8a0 con 0x7f1cd40cccb0 2024-03-24T00:47:00.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.429+0000 7f1ccbfff640 1 -- 172.21.15.67:0/2297725366 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cd0001670 con 0x7f1cd40cccb0 2024-03-24T00:47:00.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.429+0000 7f1cdb412640 1 -- 172.21.15.67:0/2297725366 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1cd40cccb0 msgr2=0x7f1cd40b2790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:00.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.429+0000 7f1cdb412640 1 --2- 172.21.15.67:0/2297725366 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1cd40cccb0 0x7f1cd40b2790 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc009d90 tx=0x7f1ccc02bd70 comp rx=0 tx=0).stop 2024-03-24T00:47:00.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.429+0000 7f1cdb412640 1 -- 172.21.15.67:0/2297725366 shutdown_connections 2024-03-24T00:47:00.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.429+0000 7f1cdb412640 1 --2- 172.21.15.67:0/2297725366 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1cd40cccb0 0x7f1cd40b2790 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:00.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.429+0000 7f1cdb412640 1 --2- 172.21.15.67:0/2297725366 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1cd40d07a0 0x7f1cd40cc770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:00.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.429+0000 7f1cdb412640 1 -- 172.21.15.67:0/2297725366 >> 172.21.15.67:0/2297725366 conn(0x7f1cd409aea0 msgr2=0x7f1cd408b180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:00.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.429+0000 7f1cdb412640 1 -- 172.21.15.67:0/2297725366 shutdown_connections 2024-03-24T00:47:00.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.429+0000 7f1cdb412640 1 -- 172.21.15.67:0/2297725366 wait complete. 2024-03-24T00:47:00.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.430+0000 7f1cdb412640 1 Processor -- start 2024-03-24T00:47:00.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.430+0000 7f1cdb412640 1 -- start start 2024-03-24T00:47:00.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.431+0000 7f1cdb412640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1cd40cccb0 0x7f1cd415c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:00.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.431+0000 7f1cdb412640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1cd40d07a0 0x7f1cd40b66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:00.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.431+0000 7f1cdb412640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1cd415c010 0x7f1cd40b6c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:00.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.431+0000 7f1cdb412640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd40cd9f0 con 0x7f1cd415c010 2024-03-24T00:47:00.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.431+0000 7f1cdb412640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd40b2300 con 0x7f1cd40d07a0 2024-03-24T00:47:00.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.431+0000 7f1cdb412640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd40b2cd0 con 0x7f1cd40cccb0 2024-03-24T00:47:00.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.431+0000 7f1cd99af640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1cd415c010 0x7f1cd40b6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:00.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.431+0000 7f1cd99af640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1cd415c010 0x7f1cd40b6c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50874/0 (socket says 172.21.15.67:50874) 2024-03-24T00:47:00.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.431+0000 7f1cd99af640 1 -- 172.21.15.67:0/4253993320 learned_addr learned my addr 172.21.15.67:0/4253993320 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:00.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.431+0000 7f1cd91ae640 1 --2- 172.21.15.67:0/4253993320 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1cd40cccb0 0x7f1cd415c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:00.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.431+0000 7f1cd89ad640 1 --2- 172.21.15.67:0/4253993320 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1cd40d07a0 0x7f1cd40b66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:00.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.431+0000 7f1cd99af640 1 -- 172.21.15.67:0/4253993320 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1cd40cccb0 msgr2=0x7f1cd415c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:00.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.431+0000 7f1cd99af640 1 --2- 172.21.15.67:0/4253993320 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1cd40cccb0 0x7f1cd415c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:00.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.431+0000 7f1cd99af640 1 -- 172.21.15.67:0/4253993320 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1cd40d07a0 msgr2=0x7f1cd40b66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:00.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.431+0000 7f1cd99af640 1 --2- 172.21.15.67:0/4253993320 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1cd40d07a0 0x7f1cd40b66e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:00.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.431+0000 7f1cd99af640 1 -- 172.21.15.67:0/4253993320 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cd40c5680 con 0x7f1cd415c010 2024-03-24T00:47:00.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.432+0000 7f1cd99af640 1 --2- 172.21.15.67:0/4253993320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1cd415c010 0x7f1cd40b6c20 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc009d90 tx=0x7f1ccc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:00.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.432+0000 7f1cc9ffb640 1 -- 172.21.15.67:0/4253993320 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc039050 con 0x7f1cd415c010 2024-03-24T00:47:00.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.432+0000 7f1cc9ffb640 1 -- 172.21.15.67:0/4253993320 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ccc03a040 con 0x7f1cd415c010 2024-03-24T00:47:00.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.432+0000 7f1cdb412640 1 -- 172.21.15.67:0/4253993320 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cd40b7160 con 0x7f1cd415c010 2024-03-24T00:47:00.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.432+0000 7f1cc9ffb640 1 -- 172.21.15.67:0/4253993320 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc009450 con 0x7f1cd415c010 2024-03-24T00:47:00.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.432+0000 7f1cdb412640 1 -- 172.21.15.67:0/4253993320 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cd40c5eb0 con 0x7f1cd415c010 2024-03-24T00:47:00.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.433+0000 7f1cdb412640 1 -- 172.21.15.67:0/4253993320 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cd406e4a0 con 0x7f1cd415c010 2024-03-24T00:47:00.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.434+0000 7f1cc9ffb640 1 -- 172.21.15.67:0/4253993320 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1ccc047020 con 0x7f1cd415c010 2024-03-24T00:47:00.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.434+0000 7f1cc9ffb640 1 --2- 172.21.15.67:0/4253993320 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1cb806bbd0 0x7f1cb806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:00.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.434+0000 7f1cc9ffb640 1 -- 172.21.15.67:0/4253993320 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f1ccc0aca70 con 0x7f1cd415c010 2024-03-24T00:47:00.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.434+0000 7f1cd91ae640 1 --2- 172.21.15.67:0/4253993320 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1cb806bbd0 0x7f1cb806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:00.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.435+0000 7f1cd91ae640 1 --2- 172.21.15.67:0/4253993320 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1cb806bbd0 0x7f1cb806e090 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f1cd0001320 tx=0x7f1cd00012b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:00.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.439+0000 7f1cc9ffb640 1 -- 172.21.15.67:0/4253993320 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1ccc04e020 con 0x7f1cd415c010 2024-03-24T00:47:00.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.672+0000 7f1cdb412640 1 -- 172.21.15.67:0/4253993320 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f1cd404f400 con 0x7f1cd415c010 2024-03-24T00:47:00.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.673+0000 7f1cc9ffb640 1 -- 172.21.15.67:0/4253993320 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 4 entries v93) v1 ==== 80+0+244 (secure 0 0 0) 0x7f1ccc07ac60 con 0x7f1cd415c010 2024-03-24T00:47:00.674 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6833/1336395002 2024-03-25T00:46:20.877877+0000 2024-03-24T00:47:00.674 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6832/1336395002 2024-03-25T00:46:20.877877+0000 2024-03-24T00:47:00.674 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6833/3021622863 2024-03-25T00:45:09.102827+0000 2024-03-24T00:47:00.674 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6832/3021622863 2024-03-25T00:45:09.102827+0000 2024-03-24T00:47:00.674 INFO:teuthology.orchestra.run.smithi067.stderr:listed 4 entries 2024-03-24T00:47:00.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.675+0000 7f1cdb412640 1 -- 172.21.15.67:0/4253993320 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1cb806bbd0 msgr2=0x7f1cb806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:00.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.675+0000 7f1cdb412640 1 --2- 172.21.15.67:0/4253993320 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1cb806bbd0 0x7f1cb806e090 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f1cd0001320 tx=0x7f1cd00012b0 comp rx=0 tx=0).stop 2024-03-24T00:47:00.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.675+0000 7f1cdb412640 1 -- 172.21.15.67:0/4253993320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1cd415c010 msgr2=0x7f1cd40b6c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:00.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.675+0000 7f1cdb412640 1 --2- 172.21.15.67:0/4253993320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1cd415c010 0x7f1cd40b6c20 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc009d90 tx=0x7f1ccc004a40 comp rx=0 tx=0).stop 2024-03-24T00:47:00.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.676+0000 7f1cdb412640 1 -- 172.21.15.67:0/4253993320 shutdown_connections 2024-03-24T00:47:00.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.676+0000 7f1cdb412640 1 --2- 172.21.15.67:0/4253993320 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1cb806bbd0 0x7f1cb806e090 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:00.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.676+0000 7f1cdb412640 1 --2- 172.21.15.67:0/4253993320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1cd415c010 0x7f1cd40b6c20 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:00.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.676+0000 7f1cdb412640 1 --2- 172.21.15.67:0/4253993320 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1cd40d07a0 0x7f1cd40b66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:00.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.676+0000 7f1cdb412640 1 --2- 172.21.15.67:0/4253993320 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1cd40cccb0 0x7f1cd415c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:00.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.676+0000 7f1cdb412640 1 -- 172.21.15.67:0/4253993320 >> 172.21.15.67:0/4253993320 conn(0x7f1cd409aea0 msgr2=0x7f1cd40ceb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:00.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.676+0000 7f1cdb412640 1 -- 172.21.15.67:0/4253993320 shutdown_connections 2024-03-24T00:47:00.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.676+0000 7f1cdb412640 1 -- 172.21.15.67:0/4253993320 wait complete. 2024-03-24T00:47:00.688 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-24T00:47:00.688 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:47:00.688 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-24T00:47:00.718 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-03-24T00:47:00.718 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:47:00.718 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-24T00:47:00.773 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:47:00.773 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:47:00.791 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17590 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248808k,nr_inodes=812202,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:47:00.792 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:47:00.792 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:47:00.828 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056375,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15771 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T00:47:00.828 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:47:00.828 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:47:00.850 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17590 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248808k,nr_inodes=812202,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:47:00.850 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:47:00.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.969+0000 7ff3d26e8640 1 Processor -- start 2024-03-24T00:47:00.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.969+0000 7ff3d26e8640 1 -- start start 2024-03-24T00:47:00.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.970+0000 7ff3d26e8640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff3cc15e2a0 0x7ff3cc0ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:00.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.970+0000 7ff3d26e8640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff3cc0d0ef0 0x7ff3cc162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:00.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.970+0000 7ff3d26e8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3cc0cf640 con 0x7ff3cc15dec0 2024-03-24T00:47:00.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.970+0000 7ff3d26e8640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3cc0cee30 con 0x7ff3cc15e2a0 2024-03-24T00:47:00.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.970+0000 7ff3d26e8640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc0d2120 con 0x7ff3cc0d0ef0 2024-03-24T00:47:00.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.970+0000 7ff3cbfff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff3cc15dec0 0x7ff3cc0d07e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44788/0 (socket says 172.21.15.67:44788) 2024-03-24T00:47:00.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.970+0000 7ff3cbfff640 1 -- 172.21.15.67:0/181743012 learned_addr learned my addr 172.21.15.67:0/181743012 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:47:00.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.970+0000 7ff3d0c85640 1 --2- 172.21.15.67:0/181743012 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff3cc0d0ef0 0x7ff3cc162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:00.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.970+0000 7ff3cb7fe640 1 --2- 172.21.15.67:0/181743012 >> v2:172.21.15.72:3300/0 conn(0x7ff3cc15e2a0 0x7ff3cc0ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:00.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.970+0000 7ff3caffd640 1 -- 172.21.15.67:0/181743012 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2741185695 0 0) 0x7ff3bc001110 con 0x7ff3cc15dec0 2024-03-24T00:47:00.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.970+0000 7ff3caffd640 1 -- 172.21.15.67:0/181743012 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3b4003520 con 0x7ff3cc15dec0 2024-03-24T00:47:00.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.971+0000 7ff3caffd640 1 -- 172.21.15.67:0/181743012 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1766127944 0 0) 0x7ff3b4003520 con 0x7ff3cc15dec0 2024-03-24T00:47:00.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.971+0000 7ff3caffd640 1 -- 172.21.15.67:0/181743012 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff3cc0d0ef0 msgr2=0x7ff3cc162630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:00.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.971+0000 7ff3caffd640 1 --2- 172.21.15.67:0/181743012 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff3cc0d0ef0 0x7ff3cc162630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:00.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.971+0000 7ff3caffd640 1 -- 172.21.15.67:0/181743012 >> v2:172.21.15.72:3300/0 conn(0x7ff3cc15e2a0 msgr2=0x7ff3cc0ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:00.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.971+0000 7ff3caffd640 1 --2- 172.21.15.67:0/181743012 >> v2:172.21.15.72:3300/0 conn(0x7ff3cc15e2a0 0x7ff3cc0ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:00.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.971+0000 7ff3caffd640 1 -- 172.21.15.67:0/181743012 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3cc0cf080 con 0x7ff3cc15dec0 2024-03-24T00:47:00.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.971+0000 7ff3caffd640 1 -- 172.21.15.67:0/181743012 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7ff3bc001840 con 0x7ff3cc15dec0 2024-03-24T00:47:00.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.971+0000 7ff3d0c85640 1 --2- 172.21.15.67:0/181743012 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff3cc0d0ef0 0x7ff3cc162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:47:00.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.971+0000 7ff3cb7fe640 1 --2- 172.21.15.67:0/181743012 >> v2:172.21.15.72:3300/0 conn(0x7ff3cc15e2a0 0x7ff3cc0ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:47:00.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.971+0000 7ff3caffd640 1 -- 172.21.15.67:0/181743012 >> v1:172.21.15.67:6789/0 conn(0x7ff3cc15dec0 legacy=0x7ff3cc0d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:00.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.971+0000 7ff3caffd640 1 --2- 172.21.15.67:0/181743012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3b40039b0 0x7ff3b4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:00.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.971+0000 7ff3caffd640 1 --2- 172.21.15.67:0/181743012 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3b4004800 0x7ff3b4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:00.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.971+0000 7ff3caffd640 1 --2- 172.21.15.67:0/181743012 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff3cc0d0ef0 0x7ff3b40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:00.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.971+0000 7ff3caffd640 1 -- 172.21.15.67:0/181743012 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc0cee30 con 0x7ff3b40039b0 2024-03-24T00:47:00.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.971+0000 7ff3caffd640 1 -- 172.21.15.67:0/181743012 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc0d2120 con 0x7ff3b4004800 2024-03-24T00:47:00.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.971+0000 7ff3caffd640 1 -- 172.21.15.67:0/181743012 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b400b9f0 con 0x7ff3cc0d0ef0 2024-03-24T00:47:00.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.971+0000 7ff3cbfff640 1 --2- 172.21.15.67:0/181743012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3b40039b0 0x7ff3b4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:00.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.971+0000 7ff3cb7fe640 1 --2- 172.21.15.67:0/181743012 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3b4004800 0x7ff3b4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:00.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.971+0000 7ff3d0c85640 1 --2- 172.21.15.67:0/181743012 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff3cc0d0ef0 0x7ff3b40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:00.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.971+0000 7ff3cbfff640 1 -- 172.21.15.67:0/181743012 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff3cc0d0ef0 msgr2=0x7ff3b40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:00.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.971+0000 7ff3cbfff640 1 --2- 172.21.15.67:0/181743012 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff3cc0d0ef0 0x7ff3b40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:00.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.971+0000 7ff3cbfff640 1 -- 172.21.15.67:0/181743012 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3b4004800 msgr2=0x7ff3b4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:00.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.971+0000 7ff3cbfff640 1 --2- 172.21.15.67:0/181743012 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3b4004800 0x7ff3b4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:00.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.971+0000 7ff3cbfff640 1 -- 172.21.15.67:0/181743012 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b4003520 con 0x7ff3b40039b0 2024-03-24T00:47:00.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.972+0000 7ff3cbfff640 1 --2- 172.21.15.67:0/181743012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3b40039b0 0x7ff3b4003dd0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc00c3e0 tx=0x7ff3b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2f23a716df0622d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:00.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.972+0000 7ff3caffd640 1 -- 172.21.15.67:0/181743012 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3bc00c0e0 con 0x7ff3b40039b0 2024-03-24T00:47:00.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.972+0000 7ff3caffd640 1 -- 172.21.15.67:0/181743012 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3bc030070 con 0x7ff3b40039b0 2024-03-24T00:47:00.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.972+0000 7ff3caffd640 1 -- 172.21.15.67:0/181743012 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3bc0058d0 con 0x7ff3b40039b0 2024-03-24T00:47:00.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.973+0000 7ff3d26e8640 1 -- 172.21.15.67:0/181743012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3b40039b0 msgr2=0x7ff3b4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:00.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.973+0000 7ff3d26e8640 1 --2- 172.21.15.67:0/181743012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3b40039b0 0x7ff3b4003dd0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc00c3e0 tx=0x7ff3b4005830 comp rx=0 tx=0).stop 2024-03-24T00:47:00.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.973+0000 7ff3d0c85640 1 -- 172.21.15.67:0/181743012 reap_dead start 2024-03-24T00:47:00.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.973+0000 7ff3d26e8640 1 -- 172.21.15.67:0/181743012 shutdown_connections 2024-03-24T00:47:00.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.973+0000 7ff3d26e8640 1 -- 172.21.15.67:0/181743012 >> 172.21.15.67:0/181743012 conn(0x7ff3cc06c810 msgr2=0x7ff3cc06cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:00.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.973+0000 7ff3d26e8640 1 -- 172.21.15.67:0/181743012 shutdown_connections 2024-03-24T00:47:00.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.973+0000 7ff3d26e8640 1 -- 172.21.15.67:0/181743012 wait complete. 2024-03-24T00:47:00.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.974+0000 7ff3d26e8640 1 Processor -- start 2024-03-24T00:47:00.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.974+0000 7ff3d26e8640 1 -- start start 2024-03-24T00:47:00.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.975+0000 7ff3d26e8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3b40039b0 0x7ff3cc152330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:00.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.975+0000 7ff3d26e8640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff3cc0b4db0 0x7ff3cc0b80e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:00.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.975+0000 7ff3d26e8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3cc0b5230 0x7ff3cc0b8620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:00.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.975+0000 7ff3d26e8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc1627b0 con 0x7ff3cc0b5230 2024-03-24T00:47:00.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.975+0000 7ff3d26e8640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc162630 con 0x7ff3b40039b0 2024-03-24T00:47:00.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.975+0000 7ff3d26e8640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc162930 con 0x7ff3cc0b4db0 2024-03-24T00:47:00.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.975+0000 7ff3d0c85640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3cc0b5230 0x7ff3cc0b8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:00.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.975+0000 7ff3d0c85640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3cc0b5230 0x7ff3cc0b8620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50892/0 (socket says 172.21.15.67:50892) 2024-03-24T00:47:00.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.975+0000 7ff3d0c85640 1 -- 172.21.15.67:0/1203624388 learned_addr learned my addr 172.21.15.67:0/1203624388 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:00.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.975+0000 7ff3cbfff640 1 --2- 172.21.15.67:0/1203624388 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3b40039b0 0x7ff3cc152330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:00.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.975+0000 7ff3cb7fe640 1 --2- 172.21.15.67:0/1203624388 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff3cc0b4db0 0x7ff3cc0b80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:00.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.975+0000 7ff3d0c85640 1 -- 172.21.15.67:0/1203624388 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff3cc0b4db0 msgr2=0x7ff3cc0b80e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:00.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.975+0000 7ff3d0c85640 1 --2- 172.21.15.67:0/1203624388 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff3cc0b4db0 0x7ff3cc0b80e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:00.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.975+0000 7ff3d0c85640 1 -- 172.21.15.67:0/1203624388 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3b40039b0 msgr2=0x7ff3cc152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:00.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.975+0000 7ff3d0c85640 1 --2- 172.21.15.67:0/1203624388 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3b40039b0 0x7ff3cc152330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:00.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.975+0000 7ff3d0c85640 1 -- 172.21.15.67:0/1203624388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3cc0b5800 con 0x7ff3cc0b5230 2024-03-24T00:47:00.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.975+0000 7ff3cb7fe640 1 --2- 172.21.15.67:0/1203624388 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff3cc0b4db0 0x7ff3cc0b80e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:47:00.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.975+0000 7ff3cbfff640 1 --2- 172.21.15.67:0/1203624388 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3b40039b0 0x7ff3cc152330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:47:00.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.975+0000 7ff3d0c85640 1 --2- 172.21.15.67:0/1203624388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3cc0b5230 0x7ff3cc0b8620 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc0b37e0 tx=0x7ff3c4009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:00.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.976+0000 7ff3c8ff9640 1 -- 172.21.15.67:0/1203624388 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c400fb70 con 0x7ff3cc0b5230 2024-03-24T00:47:00.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.976+0000 7ff3c8ff9640 1 -- 172.21.15.67:0/1203624388 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3c400fd50 con 0x7ff3cc0b5230 2024-03-24T00:47:00.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.976+0000 7ff3d26e8640 1 -- 172.21.15.67:0/1203624388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3cc0b5ae0 con 0x7ff3cc0b5230 2024-03-24T00:47:00.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.976+0000 7ff3c8ff9640 1 -- 172.21.15.67:0/1203624388 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c4003bd0 con 0x7ff3cc0b5230 2024-03-24T00:47:00.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.976+0000 7ff3d26e8640 1 -- 172.21.15.67:0/1203624388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3cc0b60a0 con 0x7ff3cc0b5230 2024-03-24T00:47:00.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.977+0000 7ff3d26e8640 1 -- 172.21.15.67:0/1203624388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3cc04f400 con 0x7ff3cc0b5230 2024-03-24T00:47:00.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.977+0000 7ff3c8ff9640 1 -- 172.21.15.67:0/1203624388 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff3c401f020 con 0x7ff3cc0b5230 2024-03-24T00:47:00.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.978+0000 7ff3c8ff9640 1 --2- 172.21.15.67:0/1203624388 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff3b806bc00 0x7ff3b806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:00.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.978+0000 7ff3c8ff9640 1 -- 172.21.15.67:0/1203624388 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6389+0+0 (secure 0 0 0) 0x7ff3c408aff0 con 0x7ff3cc0b5230 2024-03-24T00:47:00.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.978+0000 7ff3cbfff640 1 --2- 172.21.15.67:0/1203624388 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff3b806bc00 0x7ff3b806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:00.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.979+0000 7ff3cbfff640 1 --2- 172.21.15.67:0/1203624388 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff3b806bc00 0x7ff3b806e0c0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc00c3b0 tx=0x7ff3bc007000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:00.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:00.982+0000 7ff3c8ff9640 1 -- 172.21.15.67:0/1203624388 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff3c4010040 con 0x7ff3cc0b5230 2024-03-24T00:47:01.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.201+0000 7ff3d26e8640 1 -- 172.21.15.67:0/1203624388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff3cc0008d0 con 0x7ff3cc0b5230 2024-03-24T00:47:01.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.202+0000 7ff3c8ff9640 1 -- 172.21.15.67:0/1203624388 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4980 (secure 0 0 0) 0x7ff3c4059830 con 0x7ff3cc0b5230 2024-03-24T00:47:01.203 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:47:01.203 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":66,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6359,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2664460392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2664460392},{"type":"v1","addr":"172.21.15.67:6835","nonce":2664460392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6368,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/3323873506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3323873506},{"type":"v1","addr":"172.21.15.72:6837","nonce":3323873506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":7150,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/2744016217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2744016217},{"type":"v1","addr":"172.21.15.72:6836","nonce":2744016217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:44:54.749656+0000","modified":"2024-03-24T00:46:24.852760+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8163},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8163":{"gid":8163,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6833/3311719072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3311719072},{"type":"v1","addr":"172.21.15.67:6833","nonce":3311719072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8163,"qdb_cluster":[8163]},"id":6}]} 2024-03-24T00:47:01.203 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 66 2024-03-24T00:47:01.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.204+0000 7ff3d26e8640 1 -- 172.21.15.67:0/1203624388 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff3b806bc00 msgr2=0x7ff3b806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:01.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.204+0000 7ff3d26e8640 1 --2- 172.21.15.67:0/1203624388 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff3b806bc00 0x7ff3b806e0c0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc00c3b0 tx=0x7ff3bc007000 comp rx=0 tx=0).stop 2024-03-24T00:47:01.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.204+0000 7ff3d26e8640 1 -- 172.21.15.67:0/1203624388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3cc0b5230 msgr2=0x7ff3cc0b8620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:01.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.204+0000 7ff3d26e8640 1 --2- 172.21.15.67:0/1203624388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3cc0b5230 0x7ff3cc0b8620 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc0b37e0 tx=0x7ff3c4009f90 comp rx=0 tx=0).stop 2024-03-24T00:47:01.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.204+0000 7ff3d26e8640 1 -- 172.21.15.67:0/1203624388 shutdown_connections 2024-03-24T00:47:01.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.204+0000 7ff3d26e8640 1 --2- 172.21.15.67:0/1203624388 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff3b806bc00 0x7ff3b806e0c0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:01.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.204+0000 7ff3d26e8640 1 --2- 172.21.15.67:0/1203624388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3cc0b5230 0x7ff3cc0b8620 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:01.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.204+0000 7ff3d26e8640 1 --2- 172.21.15.67:0/1203624388 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff3cc0b4db0 0x7ff3cc0b80e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:01.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.204+0000 7ff3d26e8640 1 --2- 172.21.15.67:0/1203624388 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3b40039b0 0x7ff3cc152330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:01.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.204+0000 7ff3d26e8640 1 -- 172.21.15.67:0/1203624388 >> 172.21.15.67:0/1203624388 conn(0x7ff3cc06c810 msgr2=0x7ff3cc15e740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:01.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.204+0000 7ff3d26e8640 1 -- 172.21.15.67:0/1203624388 shutdown_connections 2024-03-24T00:47:01.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.204+0000 7ff3d26e8640 1 -- 172.21.15.67:0/1203624388 wait complete. 2024-03-24T00:47:01.216 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:47:01.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.317+0000 7f61722d1640 1 Processor -- start 2024-03-24T00:47:01.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.317+0000 7f61722d1640 1 -- start start 2024-03-24T00:47:01.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.317+0000 7f61722d1640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f616c15e2a0 0x7f616c0ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:01.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.317+0000 7f61722d1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f616c0d0ef0 0x7f616c162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:01.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.317+0000 7f61722d1640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f616c0cf640 con 0x7f616c15dec0 2024-03-24T00:47:01.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.317+0000 7f61722d1640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f616c0cee30 con 0x7f616c15e2a0 2024-03-24T00:47:01.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.317+0000 7f61722d1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f616c0d2120 con 0x7f616c0d0ef0 2024-03-24T00:47:01.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.317+0000 7f616bfff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f616c15dec0 0x7f616c0d07e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44792/0 (socket says 172.21.15.67:44792) 2024-03-24T00:47:01.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.317+0000 7f616bfff640 1 -- 172.21.15.67:0/2630874449 learned_addr learned my addr 172.21.15.67:0/2630874449 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:47:01.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.318+0000 7f617086e640 1 --2- 172.21.15.67:0/2630874449 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f616c0d0ef0 0x7f616c162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:01.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.318+0000 7f616b7fe640 1 --2- 172.21.15.67:0/2630874449 >> v2:172.21.15.72:3300/0 conn(0x7f616c15e2a0 0x7f616c0ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:01.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.318+0000 7f616affd640 1 -- 172.21.15.67:0/2630874449 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2265567823 0 0) 0x7f6160001110 con 0x7f616c15dec0 2024-03-24T00:47:01.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.318+0000 7f616affd640 1 -- 172.21.15.67:0/2630874449 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6154003520 con 0x7f616c15dec0 2024-03-24T00:47:01.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.318+0000 7f616b7fe640 1 -- 172.21.15.67:0/2630874449 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f616c0d0ef0 msgr2=0x7f616c162630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:01.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.318+0000 7f616b7fe640 1 --2- 172.21.15.67:0/2630874449 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f616c0d0ef0 0x7f616c162630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:01.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.318+0000 7f616b7fe640 1 -- 172.21.15.67:0/2630874449 >> v1:172.21.15.67:6789/0 conn(0x7f616c15dec0 legacy=0x7f616c0d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:01.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.318+0000 7f616b7fe640 1 -- 172.21.15.67:0/2630874449 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f616c0cf080 con 0x7f616c15e2a0 2024-03-24T00:47:01.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.318+0000 7f617086e640 1 --2- 172.21.15.67:0/2630874449 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f616c0d0ef0 0x7f616c162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:47:01.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.319+0000 7f616b7fe640 1 --2- 172.21.15.67:0/2630874449 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f616c15e2a0 0x7f616c0ce630 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f615c0097a0 tx=0x7f615c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94b2632cc4531614 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:01.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.319+0000 7f616affd640 1 -- 172.21.15.67:0/2630874449 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615c0315a0 con 0x7f616c15e2a0 2024-03-24T00:47:01.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.319+0000 7f616affd640 1 -- 172.21.15.67:0/2630874449 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f615c037070 con 0x7f616c15e2a0 2024-03-24T00:47:01.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.319+0000 7f616affd640 1 -- 172.21.15.67:0/2630874449 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615c031940 con 0x7f616c15e2a0 2024-03-24T00:47:01.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.319+0000 7f61722d1640 1 -- 172.21.15.67:0/2630874449 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f616c15e2a0 msgr2=0x7f616c0ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:01.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.319+0000 7f61722d1640 1 --2- 172.21.15.67:0/2630874449 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f616c15e2a0 0x7f616c0ce630 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f615c0097a0 tx=0x7f615c02b840 comp rx=0 tx=0).stop 2024-03-24T00:47:01.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.320+0000 7f61722d1640 1 -- 172.21.15.67:0/2630874449 shutdown_connections 2024-03-24T00:47:01.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.320+0000 7f61722d1640 1 --2- 172.21.15.67:0/2630874449 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f616c0d0ef0 0x7f616c162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:01.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.320+0000 7f61722d1640 1 --2- 172.21.15.67:0/2630874449 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f616c15e2a0 0x7f616c0ce630 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:01.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.320+0000 7f61722d1640 1 -- 172.21.15.67:0/2630874449 >> 172.21.15.67:0/2630874449 conn(0x7f616c06c810 msgr2=0x7f616c06cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:01.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.320+0000 7f61722d1640 1 -- 172.21.15.67:0/2630874449 shutdown_connections 2024-03-24T00:47:01.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.320+0000 7f61722d1640 1 -- 172.21.15.67:0/2630874449 wait complete. 2024-03-24T00:47:01.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.321+0000 7f61722d1640 1 Processor -- start 2024-03-24T00:47:01.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.321+0000 7f61722d1640 1 -- start start 2024-03-24T00:47:01.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.322+0000 7f61722d1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f616c0d0ef0 0x7f616c149e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:01.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.322+0000 7f61722d1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f616c15dec0 0x7f616c160260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:01.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.322+0000 7f61722d1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f616c15e2a0 0x7f616c158230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:01.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.322+0000 7f61722d1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616c0d2730 con 0x7f616c15dec0 2024-03-24T00:47:01.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.322+0000 7f61722d1640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616c0d25b0 con 0x7f616c0d0ef0 2024-03-24T00:47:01.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.322+0000 7f61722d1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f616c0d28b0 con 0x7f616c15e2a0 2024-03-24T00:47:01.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.322+0000 7f616b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f616c15dec0 0x7f616c160260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:01.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.322+0000 7f616b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f616c15dec0 0x7f616c160260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50906/0 (socket says 172.21.15.67:50906) 2024-03-24T00:47:01.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.322+0000 7f616b7fe640 1 -- 172.21.15.67:0/3059110957 learned_addr learned my addr 172.21.15.67:0/3059110957 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:01.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.322+0000 7f616bfff640 1 --2- 172.21.15.67:0/3059110957 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f616c0d0ef0 0x7f616c149e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:01.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.322+0000 7f617086e640 1 --2- 172.21.15.67:0/3059110957 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f616c15e2a0 0x7f616c158230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:01.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.322+0000 7f616b7fe640 1 -- 172.21.15.67:0/3059110957 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f616c15e2a0 msgr2=0x7f616c158230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:01.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.322+0000 7f616b7fe640 1 --2- 172.21.15.67:0/3059110957 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f616c15e2a0 0x7f616c158230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:01.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.322+0000 7f616b7fe640 1 -- 172.21.15.67:0/3059110957 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f616c0d0ef0 msgr2=0x7f616c149e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:01.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.322+0000 7f616b7fe640 1 --2- 172.21.15.67:0/3059110957 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f616c0d0ef0 0x7f616c149e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:01.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.322+0000 7f616b7fe640 1 -- 172.21.15.67:0/3059110957 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f616c160c40 con 0x7f616c15dec0 2024-03-24T00:47:01.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.322+0000 7f616b7fe640 1 --2- 172.21.15.67:0/3059110957 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f616c15dec0 0x7f616c160260 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f615c02b880 tx=0x7f615c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:01.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.322+0000 7f6168ff9640 1 -- 172.21.15.67:0/3059110957 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615c032630 con 0x7f616c15dec0 2024-03-24T00:47:01.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.322+0000 7f6168ff9640 1 -- 172.21.15.67:0/3059110957 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f615c0327f0 con 0x7f616c15dec0 2024-03-24T00:47:01.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.322+0000 7f61722d1640 1 -- 172.21.15.67:0/3059110957 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f616c160f20 con 0x7f616c15dec0 2024-03-24T00:47:01.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.322+0000 7f6168ff9640 1 -- 172.21.15.67:0/3059110957 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615c032be0 con 0x7f616c15dec0 2024-03-24T00:47:01.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.322+0000 7f61722d1640 1 -- 172.21.15.67:0/3059110957 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f616c0b47d0 con 0x7f616c15dec0 2024-03-24T00:47:01.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.324+0000 7f61722d1640 1 -- 172.21.15.67:0/3059110957 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f616c04f400 con 0x7f616c15dec0 2024-03-24T00:47:01.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.324+0000 7f6168ff9640 1 -- 172.21.15.67:0/3059110957 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f615c03b3f0 con 0x7f616c15dec0 2024-03-24T00:47:01.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.324+0000 7f6168ff9640 1 --2- 172.21.15.67:0/3059110957 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f615806bc00 0x7f615806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:01.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.324+0000 7f6168ff9640 1 -- 172.21.15.67:0/3059110957 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f615c02f030 con 0x7f616c15dec0 2024-03-24T00:47:01.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.324+0000 7f616bfff640 1 --2- 172.21.15.67:0/3059110957 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f615806bc00 0x7f615806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:01.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.325+0000 7f616bfff640 1 --2- 172.21.15.67:0/3059110957 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f615806bc00 0x7f615806e0c0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f6160005880 tx=0x7f6160009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:01.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.329+0000 7f6168ff9640 1 -- 172.21.15.67:0/3059110957 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f615c040070 con 0x7f616c15dec0 2024-03-24T00:47:01.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.673+0000 7f61722d1640 1 -- 172.21.15.67:0/3059110957 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f616c0d1aa0 con 0x7f616c15dec0 2024-03-24T00:47:01.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.674+0000 7f6168ff9640 1 -- 172.21.15.67:0/3059110957 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+17278 (secure 0 0 0) 0x7f615c07b250 con 0x7f616c15dec0 2024-03-24T00:47:01.675 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:47:01.675 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":93,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:46:20.877901+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:44:53.025965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-03-24T00:44:53.712957+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.32","mappings":[{"from":1,"to":7}]},{"pgid":"14.36","mappings":[{"from":1,"to":5}]},{"pgid":"14.38","mappings":[{"from":1,"to":2}]},{"pgid":"14.3a","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6833/1336395002":"2024-03-25T00:46:20.877877+0000","172.21.15.67:6832/1336395002":"2024-03-25T00:46:20.877877+0000","172.21.15.67:6833/3021622863":"2024-03-25T00:45:09.102827+0000","172.21.15.67:6832/3021622863":"2024-03-25T00:45:09.102827+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:47:01.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.676+0000 7f61722d1640 1 -- 172.21.15.67:0/3059110957 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f615806bc00 msgr2=0x7f615806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:01.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.676+0000 7f61722d1640 1 --2- 172.21.15.67:0/3059110957 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f615806bc00 0x7f615806e0c0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f6160005880 tx=0x7f6160009040 comp rx=0 tx=0).stop 2024-03-24T00:47:01.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.676+0000 7f61722d1640 1 -- 172.21.15.67:0/3059110957 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f616c15dec0 msgr2=0x7f616c160260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:01.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.676+0000 7f61722d1640 1 --2- 172.21.15.67:0/3059110957 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f616c15dec0 0x7f616c160260 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f615c02b880 tx=0x7f615c0046c0 comp rx=0 tx=0).stop 2024-03-24T00:47:01.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.676+0000 7f61722d1640 1 -- 172.21.15.67:0/3059110957 shutdown_connections 2024-03-24T00:47:01.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.676+0000 7f61722d1640 1 --2- 172.21.15.67:0/3059110957 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f615806bc00 0x7f615806e0c0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:01.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.676+0000 7f61722d1640 1 --2- 172.21.15.67:0/3059110957 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f616c15e2a0 0x7f616c158230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:01.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.676+0000 7f61722d1640 1 --2- 172.21.15.67:0/3059110957 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f616c15dec0 0x7f616c160260 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:01.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.676+0000 7f61722d1640 1 --2- 172.21.15.67:0/3059110957 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f616c0d0ef0 0x7f616c149e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:01.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.676+0000 7f61722d1640 1 -- 172.21.15.67:0/3059110957 >> 172.21.15.67:0/3059110957 conn(0x7f616c06c810 msgr2=0x7f616c15ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:01.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.676+0000 7f61722d1640 1 -- 172.21.15.67:0/3059110957 shutdown_connections 2024-03-24T00:47:01.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.676+0000 7f61722d1640 1 -- 172.21.15.67:0/3059110957 wait complete. 2024-03-24T00:47:01.688 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:38:39.364950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:44:53.025965+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:44:53.712957+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:47:01.688 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:47:01.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.791+0000 7fb31d8e8640 1 Processor -- start 2024-03-24T00:47:01.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.791+0000 7fb31d8e8640 1 -- start start 2024-03-24T00:47:01.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.792+0000 7fb31d8e8640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb31815dea0 0x7fb3180d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:01.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.792+0000 7fb31d8e8640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb31815e280 0x7fb3180d3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:01.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.792+0000 7fb31d8e8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3180d2250 con 0x7fb3180b5a60 2024-03-24T00:47:01.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.792+0000 7fb31d8e8640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3180d2530 con 0x7fb31815dea0 2024-03-24T00:47:01.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.792+0000 7fb31d8e8640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3180d2810 con 0x7fb31815e280 2024-03-24T00:47:01.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.792+0000 7fb3177fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb3180b5a60 0x7fb31806c640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44806/0 (socket says 172.21.15.67:44806) 2024-03-24T00:47:01.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.792+0000 7fb3177fe640 1 -- 172.21.15.67:0/1172593113 learned_addr learned my addr 172.21.15.67:0/1172593113 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:47:01.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.792+0000 7fb316ffd640 1 --2- 172.21.15.67:0/1172593113 >> v2:172.21.15.72:3300/0 conn(0x7fb31815dea0 0x7fb3180d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:01.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.792+0000 7fb30e7fc640 1 --2- 172.21.15.67:0/1172593113 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb31815e280 0x7fb3180d3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:01.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.792+0000 7fb3167fc640 1 -- 172.21.15.67:0/1172593113 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3243580353 0 0) 0x7fb3180d2250 con 0x7fb3180b5a60 2024-03-24T00:47:01.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.793+0000 7fb3167fc640 1 -- 172.21.15.67:0/1172593113 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb304003520 con 0x7fb3180b5a60 2024-03-24T00:47:01.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.793+0000 7fb3167fc640 1 -- 172.21.15.67:0/1172593113 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3997285175 0 0) 0x7fb304003520 con 0x7fb3180b5a60 2024-03-24T00:47:01.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.793+0000 7fb3167fc640 1 -- 172.21.15.67:0/1172593113 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb31815e280 msgr2=0x7fb3180d3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:01.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.793+0000 7fb3167fc640 1 --2- 172.21.15.67:0/1172593113 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb31815e280 0x7fb3180d3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:01.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.793+0000 7fb3167fc640 1 -- 172.21.15.67:0/1172593113 >> v2:172.21.15.72:3300/0 conn(0x7fb31815dea0 msgr2=0x7fb3180d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:01.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.793+0000 7fb3167fc640 1 --2- 172.21.15.67:0/1172593113 >> v2:172.21.15.72:3300/0 conn(0x7fb31815dea0 0x7fb3180d33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:01.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.793+0000 7fb3167fc640 1 -- 172.21.15.67:0/1172593113 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3180d2af0 con 0x7fb3180b5a60 2024-03-24T00:47:01.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.793+0000 7fb3167fc640 1 -- 172.21.15.67:0/1172593113 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7fb3100019f0 con 0x7fb3180b5a60 2024-03-24T00:47:01.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.793+0000 7fb3167fc640 1 -- 172.21.15.67:0/1172593113 >> v1:172.21.15.67:6789/0 conn(0x7fb3180b5a60 legacy=0x7fb31806c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:01.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.793+0000 7fb3167fc640 1 --2- 172.21.15.67:0/1172593113 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3040039b0 0x7fb304003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:01.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.793+0000 7fb3167fc640 1 --2- 172.21.15.67:0/1172593113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb304004800 0x7fb304004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:01.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.793+0000 7fb3167fc640 1 --2- 172.21.15.67:0/1172593113 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb31815e280 0x7fb3040051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:01.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.793+0000 7fb3167fc640 1 -- 172.21.15.67:0/1172593113 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3180d2530 con 0x7fb304004800 2024-03-24T00:47:01.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.793+0000 7fb3167fc640 1 -- 172.21.15.67:0/1172593113 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3180d2810 con 0x7fb3040039b0 2024-03-24T00:47:01.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.793+0000 7fb3167fc640 1 -- 172.21.15.67:0/1172593113 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb30400b9f0 con 0x7fb31815e280 2024-03-24T00:47:01.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.793+0000 7fb30e7fc640 1 --2- 172.21.15.67:0/1172593113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb304004800 0x7fb304004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:01.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.793+0000 7fb3177fe640 1 --2- 172.21.15.67:0/1172593113 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb31815e280 0x7fb3040051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:01.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.793+0000 7fb316ffd640 1 --2- 172.21.15.67:0/1172593113 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3040039b0 0x7fb304003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:01.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.793+0000 7fb30e7fc640 1 -- 172.21.15.67:0/1172593113 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb31815e280 msgr2=0x7fb3040051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:01.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.793+0000 7fb30e7fc640 1 --2- 172.21.15.67:0/1172593113 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb31815e280 0x7fb3040051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:01.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.793+0000 7fb30e7fc640 1 -- 172.21.15.67:0/1172593113 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3040039b0 msgr2=0x7fb304003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:01.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.793+0000 7fb30e7fc640 1 --2- 172.21.15.67:0/1172593113 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3040039b0 0x7fb304003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:01.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.793+0000 7fb30e7fc640 1 -- 172.21.15.67:0/1172593113 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb304003520 con 0x7fb304004800 2024-03-24T00:47:01.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.793+0000 7fb3177fe640 1 --2- 172.21.15.67:0/1172593113 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb31815e280 0x7fb3040051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:47:01.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.793+0000 7fb316ffd640 1 --2- 172.21.15.67:0/1172593113 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3040039b0 0x7fb304003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:47:01.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.794+0000 7fb30e7fc640 1 --2- 172.21.15.67:0/1172593113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb304004800 0x7fb304004c80 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7fb300009ac0 tx=0x7fb304005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4a2f16618a4c68e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:01.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.794+0000 7fb3167fc640 1 -- 172.21.15.67:0/1172593113 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb300039070 con 0x7fb304004800 2024-03-24T00:47:01.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.794+0000 7fb3167fc640 1 -- 172.21.15.67:0/1172593113 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb30002c6b0 con 0x7fb304004800 2024-03-24T00:47:01.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.794+0000 7fb3167fc640 1 -- 172.21.15.67:0/1172593113 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb300009690 con 0x7fb304004800 2024-03-24T00:47:01.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.795+0000 7fb31d8e8640 1 -- 172.21.15.67:0/1172593113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb304004800 msgr2=0x7fb304004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:01.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.795+0000 7fb31d8e8640 1 --2- 172.21.15.67:0/1172593113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb304004800 0x7fb304004c80 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7fb300009ac0 tx=0x7fb304005830 comp rx=0 tx=0).stop 2024-03-24T00:47:01.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.795+0000 7fb3177fe640 1 -- 172.21.15.67:0/1172593113 reap_dead start 2024-03-24T00:47:01.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.795+0000 7fb31d8e8640 1 -- 172.21.15.67:0/1172593113 shutdown_connections 2024-03-24T00:47:01.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.795+0000 7fb31d8e8640 1 -- 172.21.15.67:0/1172593113 >> 172.21.15.67:0/1172593113 conn(0x7fb318072da0 msgr2=0x7fb3180d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:01.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.795+0000 7fb31d8e8640 1 -- 172.21.15.67:0/1172593113 shutdown_connections 2024-03-24T00:47:01.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.795+0000 7fb31d8e8640 1 -- 172.21.15.67:0/1172593113 wait complete. 2024-03-24T00:47:01.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.796+0000 7fb31d8e8640 1 Processor -- start 2024-03-24T00:47:01.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.796+0000 7fb31d8e8640 1 -- start start 2024-03-24T00:47:01.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.797+0000 7fb31d8e8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb304004800 0x7fb318157c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:01.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.797+0000 7fb31d8e8640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb318160fd0 0x7fb31815abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:01.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.797+0000 7fb31d8e8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3181587f0 0x7fb31815b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:01.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.797+0000 7fb31d8e8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3180d3910 con 0x7fb3181587f0 2024-03-24T00:47:01.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.797+0000 7fb31d8e8640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3180b8390 con 0x7fb304004800 2024-03-24T00:47:01.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.797+0000 7fb31d8e8640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3180d3a90 con 0x7fb318160fd0 2024-03-24T00:47:01.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.797+0000 7fb3177fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3181587f0 0x7fb31815b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:01.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.797+0000 7fb3177fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3181587f0 0x7fb31815b120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50926/0 (socket says 172.21.15.67:50926) 2024-03-24T00:47:01.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.797+0000 7fb3177fe640 1 -- 172.21.15.67:0/1471511714 learned_addr learned my addr 172.21.15.67:0/1471511714 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:01.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.797+0000 7fb30e7fc640 1 --2- 172.21.15.67:0/1471511714 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb318160fd0 0x7fb31815abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:01.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.797+0000 7fb316ffd640 1 --2- 172.21.15.67:0/1471511714 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb304004800 0x7fb318157c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:01.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.797+0000 7fb3177fe640 1 -- 172.21.15.67:0/1471511714 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb318160fd0 msgr2=0x7fb31815abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:01.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.797+0000 7fb3177fe640 1 --2- 172.21.15.67:0/1471511714 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb318160fd0 0x7fb31815abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:01.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.797+0000 7fb3177fe640 1 -- 172.21.15.67:0/1471511714 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb304004800 msgr2=0x7fb318157c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:01.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.797+0000 7fb3177fe640 1 --2- 172.21.15.67:0/1471511714 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb304004800 0x7fb318157c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:01.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.797+0000 7fb3177fe640 1 -- 172.21.15.67:0/1471511714 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3181585b0 con 0x7fb3181587f0 2024-03-24T00:47:01.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.797+0000 7fb3177fe640 1 --2- 172.21.15.67:0/1471511714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3181587f0 0x7fb31815b120 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7fb310000f80 tx=0x7fb31000e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:01.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.798+0000 7fb314ff9640 1 -- 172.21.15.67:0/1471511714 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb31000eb90 con 0x7fb3181587f0 2024-03-24T00:47:01.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.798+0000 7fb314ff9640 1 -- 172.21.15.67:0/1471511714 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb310018070 con 0x7fb3181587f0 2024-03-24T00:47:01.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.798+0000 7fb31d8e8640 1 -- 172.21.15.67:0/1471511714 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb318160210 con 0x7fb3181587f0 2024-03-24T00:47:01.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.798+0000 7fb314ff9640 1 -- 172.21.15.67:0/1471511714 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb310013420 con 0x7fb3181587f0 2024-03-24T00:47:01.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.798+0000 7fb31d8e8640 1 -- 172.21.15.67:0/1471511714 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3181607d0 con 0x7fb3181587f0 2024-03-24T00:47:01.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.799+0000 7fb31d8e8640 1 -- 172.21.15.67:0/1471511714 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb31804f400 con 0x7fb3181587f0 2024-03-24T00:47:01.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.799+0000 7fb314ff9640 1 -- 172.21.15.67:0/1471511714 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb3100135c0 con 0x7fb3181587f0 2024-03-24T00:47:01.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.800+0000 7fb314ff9640 1 --2- 172.21.15.67:0/1471511714 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb2f806bbd0 0x7fb2f806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:01.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.800+0000 7fb314ff9640 1 -- 172.21.15.67:0/1471511714 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6389+0+0 (secure 0 0 0) 0x7fb31008b070 con 0x7fb3181587f0 2024-03-24T00:47:01.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.800+0000 7fb316ffd640 1 --2- 172.21.15.67:0/1471511714 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb2f806bbd0 0x7fb2f806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:01.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.800+0000 7fb316ffd640 1 --2- 172.21.15.67:0/1471511714 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb2f806bbd0 0x7fb2f806e090 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fb308009a30 tx=0x7fb308008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:01.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.804+0000 7fb314ff9640 1 -- 172.21.15.67:0/1471511714 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb3100598b0 con 0x7fb3181587f0 2024-03-24T00:47:01.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.988+0000 7fb31d8e8640 1 -- 172.21.15.67:0/1471511714 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb3180b4340 con 0x7fb3181587f0 2024-03-24T00:47:01.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.988+0000 7fb314ff9640 1 -- 172.21.15.67:0/1471511714 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb3100591b0 con 0x7fb3181587f0 2024-03-24T00:47:01.989 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-24T00:47:01.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.990+0000 7fb31d8e8640 1 -- 172.21.15.67:0/1471511714 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb2f806bbd0 msgr2=0x7fb2f806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:01.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.990+0000 7fb31d8e8640 1 --2- 172.21.15.67:0/1471511714 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb2f806bbd0 0x7fb2f806e090 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fb308009a30 tx=0x7fb308008040 comp rx=0 tx=0).stop 2024-03-24T00:47:01.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.990+0000 7fb31d8e8640 1 -- 172.21.15.67:0/1471511714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3181587f0 msgr2=0x7fb31815b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:01.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.990+0000 7fb31d8e8640 1 --2- 172.21.15.67:0/1471511714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3181587f0 0x7fb31815b120 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7fb310000f80 tx=0x7fb31000e530 comp rx=0 tx=0).stop 2024-03-24T00:47:01.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.990+0000 7fb31d8e8640 1 -- 172.21.15.67:0/1471511714 shutdown_connections 2024-03-24T00:47:01.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.990+0000 7fb31d8e8640 1 --2- 172.21.15.67:0/1471511714 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb2f806bbd0 0x7fb2f806e090 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:01.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.990+0000 7fb31d8e8640 1 --2- 172.21.15.67:0/1471511714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3181587f0 0x7fb31815b120 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:01.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.990+0000 7fb31d8e8640 1 --2- 172.21.15.67:0/1471511714 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb318160fd0 0x7fb31815abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:01.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.990+0000 7fb31d8e8640 1 --2- 172.21.15.67:0/1471511714 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb304004800 0x7fb318157c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:01.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.990+0000 7fb31d8e8640 1 -- 172.21.15.67:0/1471511714 >> 172.21.15.67:0/1471511714 conn(0x7fb318072da0 msgr2=0x7fb3180ce410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:01.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.991+0000 7fb31d8e8640 1 -- 172.21.15.67:0/1471511714 shutdown_connections 2024-03-24T00:47:01.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:01.991+0000 7fb31d8e8640 1 -- 172.21.15.67:0/1471511714 wait complete. 2024-03-24T00:47:02.002 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:47:02.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.105+0000 7f80e7989640 1 Processor -- start 2024-03-24T00:47:02.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.105+0000 7f80e7989640 1 -- start start 2024-03-24T00:47:02.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.106+0000 7f80e7989640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f80e015e580 0x7f80e00ce3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:02.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.106+0000 7f80e7989640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f80e00b2920 0x7f80e00ce900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:02.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.106+0000 7f80e7989640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80e015e2e0 con 0x7f80e015df00 2024-03-24T00:47:02.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.106+0000 7f80e7989640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80e00b1880 con 0x7f80e00b2920 2024-03-24T00:47:02.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.106+0000 7f80e7989640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80e00b1b60 con 0x7f80e015e580 2024-03-24T00:47:02.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.106+0000 7f80e5f26640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f80e00b2920 0x7f80e00ce900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:02.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.106+0000 7f80e5725640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f80e015df00 0x7f80e015ed70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44808/0 (socket says 172.21.15.67:44808) 2024-03-24T00:47:02.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.106+0000 7f80e5725640 1 -- 172.21.15.67:0/3224888249 learned_addr learned my addr 172.21.15.67:0/3224888249 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:47:02.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.106+0000 7f80e4f24640 1 --2- 172.21.15.67:0/3224888249 >> v2:172.21.15.72:3300/0 conn(0x7f80e015e580 0x7f80e00ce3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:02.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.106+0000 7f80e5f26640 1 -- 172.21.15.67:0/3224888249 >> v2:172.21.15.72:3300/0 conn(0x7f80e015e580 msgr2=0x7f80e00ce3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:02.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.106+0000 7f80e5f26640 1 --2- 172.21.15.67:0/3224888249 >> v2:172.21.15.72:3300/0 conn(0x7f80e015e580 0x7f80e00ce3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:02.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.106+0000 7f80e5f26640 1 -- 172.21.15.67:0/3224888249 >> v1:172.21.15.67:6789/0 conn(0x7f80e015df00 legacy=0x7f80e015ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:02.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.106+0000 7f80e5f26640 1 -- 172.21.15.67:0/3224888249 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80e00b1e40 con 0x7f80e00b2920 2024-03-24T00:47:02.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.106+0000 7f80dffff640 1 -- 172.21.15.67:0/3224888249 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2736020807 0 0) 0x7f80e015e2e0 con 0x7f80e015df00 2024-03-24T00:47:02.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.106+0000 7f80e4f24640 1 --2- 172.21.15.67:0/3224888249 >> v2:172.21.15.72:3300/0 conn(0x7f80e015e580 0x7f80e00ce3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:47:02.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.107+0000 7f80e5f26640 1 --2- 172.21.15.67:0/3224888249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80e00b2920 0x7f80e00ce900 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f80e015e960 tx=0x7f80d802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4822a87e966fbb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:02.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.107+0000 7f80dffff640 1 -- 172.21.15.67:0/3224888249 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d8039070 con 0x7f80e00b2920 2024-03-24T00:47:02.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.107+0000 7f80dffff640 1 -- 172.21.15.67:0/3224888249 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80d802cbf0 con 0x7f80e00b2920 2024-03-24T00:47:02.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.107+0000 7f80dffff640 1 -- 172.21.15.67:0/3224888249 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d8034440 con 0x7f80e00b2920 2024-03-24T00:47:02.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.108+0000 7f80e7989640 1 -- 172.21.15.67:0/3224888249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80e00b2920 msgr2=0x7f80e00ce900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:02.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.108+0000 7f80e7989640 1 --2- 172.21.15.67:0/3224888249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80e00b2920 0x7f80e00ce900 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f80e015e960 tx=0x7f80d802be40 comp rx=0 tx=0).stop 2024-03-24T00:47:02.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.108+0000 7f80e7989640 1 -- 172.21.15.67:0/3224888249 shutdown_connections 2024-03-24T00:47:02.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.108+0000 7f80e7989640 1 --2- 172.21.15.67:0/3224888249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80e00b2920 0x7f80e00ce900 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:02.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.108+0000 7f80e7989640 1 --2- 172.21.15.67:0/3224888249 >> v2:172.21.15.72:3300/0 conn(0x7f80e015e580 0x7f80e00ce3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:02.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.108+0000 7f80e7989640 1 -- 172.21.15.67:0/3224888249 >> 172.21.15.67:0/3224888249 conn(0x7f80e008b180 msgr2=0x7f80e008b590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:02.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.108+0000 7f80e7989640 1 -- 172.21.15.67:0/3224888249 shutdown_connections 2024-03-24T00:47:02.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.108+0000 7f80e7989640 1 -- 172.21.15.67:0/3224888249 wait complete. 2024-03-24T00:47:02.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.110+0000 7f80e7989640 1 Processor -- start 2024-03-24T00:47:02.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.110+0000 7f80e7989640 1 -- start start 2024-03-24T00:47:02.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.111+0000 7f80e7989640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80e00b2920 0x7f80e015a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:02.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.111+0000 7f80e7989640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f80e015df00 0x7f80e015f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:02.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.111+0000 7f80e7989640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f80e015e580 0x7f80e00ce3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:02.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.111+0000 7f80e7989640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80e006c360 con 0x7f80e00b2920 2024-03-24T00:47:02.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.111+0000 7f80e7989640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80e00b2400 con 0x7f80e015df00 2024-03-24T00:47:02.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.111+0000 7f80e7989640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80e006c4e0 con 0x7f80e015e580 2024-03-24T00:47:02.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.111+0000 7f80e5725640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80e00b2920 0x7f80e015a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:02.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.111+0000 7f80e5725640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80e00b2920 0x7f80e015a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50958/0 (socket says 172.21.15.67:50958) 2024-03-24T00:47:02.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.111+0000 7f80e5725640 1 -- 172.21.15.67:0/2875344007 learned_addr learned my addr 172.21.15.67:0/2875344007 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:02.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.111+0000 7f80e4f24640 1 --2- 172.21.15.67:0/2875344007 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f80e015df00 0x7f80e015f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:02.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.111+0000 7f80e5f26640 1 --2- 172.21.15.67:0/2875344007 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f80e015e580 0x7f80e00ce3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:02.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.112+0000 7f80e5725640 1 -- 172.21.15.67:0/2875344007 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f80e015e580 msgr2=0x7f80e00ce3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:02.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.112+0000 7f80e5725640 1 --2- 172.21.15.67:0/2875344007 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f80e015e580 0x7f80e00ce3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:02.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.112+0000 7f80e5725640 1 -- 172.21.15.67:0/2875344007 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f80e015df00 msgr2=0x7f80e015f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:02.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.112+0000 7f80e5725640 1 --2- 172.21.15.67:0/2875344007 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f80e015df00 0x7f80e015f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:02.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.112+0000 7f80e5725640 1 -- 172.21.15.67:0/2875344007 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80e015fe70 con 0x7f80e00b2920 2024-03-24T00:47:02.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.112+0000 7f80e5f26640 1 --2- 172.21.15.67:0/2875344007 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f80e015e580 0x7f80e00ce3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:47:02.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.112+0000 7f80e4f24640 1 --2- 172.21.15.67:0/2875344007 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f80e015df00 0x7f80e015f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:47:02.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.112+0000 7f80e5725640 1 --2- 172.21.15.67:0/2875344007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80e00b2920 0x7f80e015a9f0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f80d000c4d0 tx=0x7f80d000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:02.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.112+0000 7f80ddffb640 1 -- 172.21.15.67:0/2875344007 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d0018070 con 0x7f80e00b2920 2024-03-24T00:47:02.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.112+0000 7f80ddffb640 1 -- 172.21.15.67:0/2875344007 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80d0007500 con 0x7f80e00b2920 2024-03-24T00:47:02.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.112+0000 7f80e7989640 1 -- 172.21.15.67:0/2875344007 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80e0160150 con 0x7f80e00b2920 2024-03-24T00:47:02.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.112+0000 7f80ddffb640 1 -- 172.21.15.67:0/2875344007 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d0004a20 con 0x7f80e00b2920 2024-03-24T00:47:02.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.112+0000 7f80e7989640 1 -- 172.21.15.67:0/2875344007 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80e0160710 con 0x7f80e00b2920 2024-03-24T00:47:02.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.113+0000 7f80e7989640 1 -- 172.21.15.67:0/2875344007 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80e004f400 con 0x7f80e00b2920 2024-03-24T00:47:02.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.114+0000 7f80ddffb640 1 -- 172.21.15.67:0/2875344007 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f80d001b070 con 0x7f80e00b2920 2024-03-24T00:47:02.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.114+0000 7f80ddffb640 1 --2- 172.21.15.67:0/2875344007 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f80cc06bc00 0x7f80cc06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:02.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.114+0000 7f80ddffb640 1 -- 172.21.15.67:0/2875344007 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f80d0089f50 con 0x7f80e00b2920 2024-03-24T00:47:02.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.114+0000 7f80e4f24640 1 --2- 172.21.15.67:0/2875344007 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f80cc06bc00 0x7f80cc06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:02.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.115+0000 7f80e4f24640 1 --2- 172.21.15.67:0/2875344007 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f80cc06bc00 0x7f80cc06e0c0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f80e015af30 tx=0x7f80d4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:02.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.119+0000 7f80ddffb640 1 -- 172.21.15.67:0/2875344007 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f80d0058820 con 0x7f80e00b2920 2024-03-24T00:47:02.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.347+0000 7f80e7989640 1 -- 172.21.15.67:0/2875344007 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f80e00b0d30 con 0x7f80e00b2920 2024-03-24T00:47:02.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.348+0000 7f80ddffb640 1 -- 172.21.15.67:0/2875344007 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) v1 ==== 104+0+11 (secure 0 0 0) 0x7f80d0058120 con 0x7f80e00b2920 2024-03-24T00:47:02.349 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-24T00:47:02.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.350+0000 7f80e7989640 1 -- 172.21.15.67:0/2875344007 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f80cc06bc00 msgr2=0x7f80cc06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:02.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.350+0000 7f80e7989640 1 --2- 172.21.15.67:0/2875344007 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f80cc06bc00 0x7f80cc06e0c0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f80e015af30 tx=0x7f80d4008040 comp rx=0 tx=0).stop 2024-03-24T00:47:02.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.350+0000 7f80e7989640 1 -- 172.21.15.67:0/2875344007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80e00b2920 msgr2=0x7f80e015a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:02.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.350+0000 7f80e7989640 1 --2- 172.21.15.67:0/2875344007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80e00b2920 0x7f80e015a9f0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f80d000c4d0 tx=0x7f80d000c990 comp rx=0 tx=0).stop 2024-03-24T00:47:02.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.350+0000 7f80e7989640 1 -- 172.21.15.67:0/2875344007 shutdown_connections 2024-03-24T00:47:02.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.350+0000 7f80e7989640 1 --2- 172.21.15.67:0/2875344007 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f80cc06bc00 0x7f80cc06e0c0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:02.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.350+0000 7f80e7989640 1 --2- 172.21.15.67:0/2875344007 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f80e015e580 0x7f80e00ce3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:02.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.350+0000 7f80e7989640 1 --2- 172.21.15.67:0/2875344007 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f80e015df00 0x7f80e015f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:02.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.350+0000 7f80e7989640 1 --2- 172.21.15.67:0/2875344007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80e00b2920 0x7f80e015a9f0 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:02.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.350+0000 7f80e7989640 1 -- 172.21.15.67:0/2875344007 >> 172.21.15.67:0/2875344007 conn(0x7f80e008b180 msgr2=0x7f80e015bb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:02.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.350+0000 7f80e7989640 1 -- 172.21.15.67:0/2875344007 shutdown_connections 2024-03-24T00:47:02.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.350+0000 7f80e7989640 1 -- 172.21.15.67:0/2875344007 wait complete. 2024-03-24T00:47:02.361 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:47:02.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.465+0000 7fbb8f436640 1 Processor -- start 2024-03-24T00:47:02.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.465+0000 7fbb8f436640 1 -- start start 2024-03-24T00:47:02.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.466+0000 7fbb8f436640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbb8815dea0 0x7fbb880d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:02.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.466+0000 7fbb8f436640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbb8815e280 0x7fbb880d3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:02.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.466+0000 7fbb8f436640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fbb880b5a60 0x7fbb880b5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:02.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.466+0000 7fbb8f436640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb880b6410 con 0x7fbb8815dea0 2024-03-24T00:47:02.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.466+0000 7fbb8f436640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb880d2250 con 0x7fbb880b5a60 2024-03-24T00:47:02.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.466+0000 7fbb8f436640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb880d2530 con 0x7fbb8815e280 2024-03-24T00:47:02.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.466+0000 7fbb8d1d2640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbb8815dea0 0x7fbb880d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:02.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.466+0000 7fbb8d1d2640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbb8815dea0 0x7fbb880d33d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50970/0 (socket says 172.21.15.67:50970) 2024-03-24T00:47:02.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.466+0000 7fbb8d1d2640 1 -- 172.21.15.67:0/173509997 learned_addr learned my addr 172.21.15.67:0/173509997 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:02.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.467+0000 7fbb8c9d1640 1 --2- 172.21.15.67:0/173509997 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbb8815e280 0x7fbb880d3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:02.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.467+0000 7fbb8d9d3640 1 --2- 172.21.15.67:0/173509997 >> v2:172.21.15.72:3300/0 conn(0x7fbb880b5a60 0x7fbb880b5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:02.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.467+0000 7fbb8d1d2640 1 -- 172.21.15.67:0/173509997 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbb8815e280 msgr2=0x7fbb880d3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:02.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.467+0000 7fbb8d1d2640 1 --2- 172.21.15.67:0/173509997 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbb8815e280 0x7fbb880d3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:02.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.467+0000 7fbb8d1d2640 1 -- 172.21.15.67:0/173509997 >> v2:172.21.15.72:3300/0 conn(0x7fbb880b5a60 msgr2=0x7fbb880b5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:02.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.467+0000 7fbb8d1d2640 1 --2- 172.21.15.67:0/173509997 >> v2:172.21.15.72:3300/0 conn(0x7fbb880b5a60 0x7fbb880b5ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:02.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.467+0000 7fbb8d1d2640 1 -- 172.21.15.67:0/173509997 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb880d2810 con 0x7fbb8815dea0 2024-03-24T00:47:02.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.467+0000 7fbb8c9d1640 1 --2- 172.21.15.67:0/173509997 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbb8815e280 0x7fbb880d3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:47:02.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.467+0000 7fbb8d9d3640 1 --2- 172.21.15.67:0/173509997 >> v2:172.21.15.72:3300/0 conn(0x7fbb880b5a60 0x7fbb880b5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:47:02.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.467+0000 7fbb8d1d2640 1 --2- 172.21.15.67:0/173509997 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb8815dea0 0x7fbb880d33d0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7fbb880d2460 tx=0x7fbb8402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99dbc744f082524f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:02.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.468+0000 7fbb7ffff640 1 -- 172.21.15.67:0/173509997 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb840317c0 con 0x7fbb8815dea0 2024-03-24T00:47:02.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.468+0000 7fbb7ffff640 1 -- 172.21.15.67:0/173509997 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb84037070 con 0x7fbb8815dea0 2024-03-24T00:47:02.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.468+0000 7fbb7ffff640 1 -- 172.21.15.67:0/173509997 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb84031b80 con 0x7fbb8815dea0 2024-03-24T00:47:02.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.468+0000 7fbb8f436640 1 -- 172.21.15.67:0/173509997 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb8815dea0 msgr2=0x7fbb880d33d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:02.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.468+0000 7fbb8f436640 1 --2- 172.21.15.67:0/173509997 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb8815dea0 0x7fbb880d33d0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7fbb880d2460 tx=0x7fbb8402b860 comp rx=0 tx=0).stop 2024-03-24T00:47:02.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.468+0000 7fbb8f436640 1 -- 172.21.15.67:0/173509997 shutdown_connections 2024-03-24T00:47:02.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.468+0000 7fbb8f436640 1 --2- 172.21.15.67:0/173509997 >> v2:172.21.15.72:3300/0 conn(0x7fbb880b5a60 0x7fbb880b5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:02.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.468+0000 7fbb8f436640 1 --2- 172.21.15.67:0/173509997 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbb8815e280 0x7fbb880d3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:02.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.468+0000 7fbb8f436640 1 --2- 172.21.15.67:0/173509997 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb8815dea0 0x7fbb880d33d0 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:02.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.468+0000 7fbb8f436640 1 -- 172.21.15.67:0/173509997 >> 172.21.15.67:0/173509997 conn(0x7fbb88072da0 msgr2=0x7fbb880d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:02.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.468+0000 7fbb8f436640 1 -- 172.21.15.67:0/173509997 shutdown_connections 2024-03-24T00:47:02.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.468+0000 7fbb8f436640 1 -- 172.21.15.67:0/173509997 wait complete. 2024-03-24T00:47:02.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.470+0000 7fbb8f436640 1 Processor -- start 2024-03-24T00:47:02.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.470+0000 7fbb8f436640 1 -- start start 2024-03-24T00:47:02.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.471+0000 7fbb8f436640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbb880b5a60 0x7fbb88160b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:02.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.471+0000 7fbb8f436640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbb8815dea0 0x7fbb88154030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:02.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.471+0000 7fbb8f436640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb8815e280 0x7fbb88154570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:02.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.471+0000 7fbb8f436640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb8806c7c0 con 0x7fbb8815e280 2024-03-24T00:47:02.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.471+0000 7fbb8f436640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb8806c640 con 0x7fbb880b5a60 2024-03-24T00:47:02.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.471+0000 7fbb8f436640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb8806c940 con 0x7fbb8815dea0 2024-03-24T00:47:02.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.471+0000 7fbb8d9d3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb8815e280 0x7fbb88154570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:02.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.471+0000 7fbb8d9d3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb8815e280 0x7fbb88154570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50972/0 (socket says 172.21.15.67:50972) 2024-03-24T00:47:02.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.471+0000 7fbb8d9d3640 1 -- 172.21.15.67:0/3599254772 learned_addr learned my addr 172.21.15.67:0/3599254772 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:02.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.471+0000 7fbb8c9d1640 1 --2- 172.21.15.67:0/3599254772 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbb8815dea0 0x7fbb88154030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:02.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.471+0000 7fbb8d1d2640 1 --2- 172.21.15.67:0/3599254772 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbb880b5a60 0x7fbb88160b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:02.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.471+0000 7fbb8d9d3640 1 -- 172.21.15.67:0/3599254772 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbb8815dea0 msgr2=0x7fbb88154030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:02.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.471+0000 7fbb8d9d3640 1 --2- 172.21.15.67:0/3599254772 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbb8815dea0 0x7fbb88154030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:02.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.471+0000 7fbb8d9d3640 1 -- 172.21.15.67:0/3599254772 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbb880b5a60 msgr2=0x7fbb88160b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:02.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.471+0000 7fbb8d9d3640 1 --2- 172.21.15.67:0/3599254772 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbb880b5a60 0x7fbb88160b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:02.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.471+0000 7fbb8d9d3640 1 -- 172.21.15.67:0/3599254772 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb881f4a70 con 0x7fbb8815e280 2024-03-24T00:47:02.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.471+0000 7fbb8c9d1640 1 --2- 172.21.15.67:0/3599254772 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbb8815dea0 0x7fbb88154030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:47:02.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.471+0000 7fbb8d1d2640 1 --2- 172.21.15.67:0/3599254772 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbb880b5a60 0x7fbb88160b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:47:02.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.472+0000 7fbb8d9d3640 1 --2- 172.21.15.67:0/3599254772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb8815e280 0x7fbb88154570 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7fbb80007d90 tx=0x7fbb8000d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:02.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.472+0000 7fbb7dffb640 1 -- 172.21.15.67:0/3599254772 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb8001a070 con 0x7fbb8815e280 2024-03-24T00:47:02.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.472+0000 7fbb7dffb640 1 -- 172.21.15.67:0/3599254772 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb80003990 con 0x7fbb8815e280 2024-03-24T00:47:02.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.472+0000 7fbb8f436640 1 -- 172.21.15.67:0/3599254772 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb881f4d50 con 0x7fbb8815e280 2024-03-24T00:47:02.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.472+0000 7fbb7dffb640 1 -- 172.21.15.67:0/3599254772 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb80003d80 con 0x7fbb8815e280 2024-03-24T00:47:02.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.472+0000 7fbb8f436640 1 -- 172.21.15.67:0/3599254772 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb881f5310 con 0x7fbb8815e280 2024-03-24T00:47:02.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.473+0000 7fbb8f436640 1 -- 172.21.15.67:0/3599254772 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb8804f400 con 0x7fbb8815e280 2024-03-24T00:47:02.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.473+0000 7fbb7dffb640 1 -- 172.21.15.67:0/3599254772 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fbb8001d070 con 0x7fbb8815e280 2024-03-24T00:47:02.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.474+0000 7fbb7dffb640 1 --2- 172.21.15.67:0/3599254772 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbb6c06bbd0 0x7fbb6c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:02.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.474+0000 7fbb7dffb640 1 -- 172.21.15.67:0/3599254772 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6389+0+0 (secure 0 0 0) 0x7fbb8008ae90 con 0x7fbb8815e280 2024-03-24T00:47:02.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.474+0000 7fbb8d1d2640 1 --2- 172.21.15.67:0/3599254772 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbb6c06bbd0 0x7fbb6c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:02.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.474+0000 7fbb8d1d2640 1 --2- 172.21.15.67:0/3599254772 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbb6c06bbd0 0x7fbb6c06e090 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fbb84036000 tx=0x7fbb8403d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:02.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.478+0000 7fbb7dffb640 1 -- 172.21.15.67:0/3599254772 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fbb80010070 con 0x7fbb8815e280 2024-03-24T00:47:02.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.660+0000 7fbb8f436640 1 -- 172.21.15.67:0/3599254772 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbb880b8fe0 con 0x7fbb8815e280 2024-03-24T00:47:02.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.660+0000 7fbb7dffb640 1 -- 172.21.15.67:0/3599254772 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbb80059760 con 0x7fbb8815e280 2024-03-24T00:47:02.661 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-03-24T00:47:02.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.662+0000 7fbb8f436640 1 -- 172.21.15.67:0/3599254772 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbb6c06bbd0 msgr2=0x7fbb6c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:02.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.662+0000 7fbb8f436640 1 --2- 172.21.15.67:0/3599254772 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbb6c06bbd0 0x7fbb6c06e090 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fbb84036000 tx=0x7fbb8403d040 comp rx=0 tx=0).stop 2024-03-24T00:47:02.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.662+0000 7fbb8f436640 1 -- 172.21.15.67:0/3599254772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb8815e280 msgr2=0x7fbb88154570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:02.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.662+0000 7fbb8f436640 1 --2- 172.21.15.67:0/3599254772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb8815e280 0x7fbb88154570 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7fbb80007d90 tx=0x7fbb8000d4a0 comp rx=0 tx=0).stop 2024-03-24T00:47:02.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.663+0000 7fbb8f436640 1 -- 172.21.15.67:0/3599254772 shutdown_connections 2024-03-24T00:47:02.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.663+0000 7fbb8f436640 1 --2- 172.21.15.67:0/3599254772 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbb6c06bbd0 0x7fbb6c06e090 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:02.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.663+0000 7fbb8f436640 1 --2- 172.21.15.67:0/3599254772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb8815e280 0x7fbb88154570 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:02.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.663+0000 7fbb8f436640 1 --2- 172.21.15.67:0/3599254772 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbb8815dea0 0x7fbb88154030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:02.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.663+0000 7fbb8f436640 1 --2- 172.21.15.67:0/3599254772 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbb880b5a60 0x7fbb88160b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:02.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.663+0000 7fbb8f436640 1 -- 172.21.15.67:0/3599254772 >> 172.21.15.67:0/3599254772 conn(0x7fbb88072da0 msgr2=0x7fbb880d31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:02.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.663+0000 7fbb8f436640 1 -- 172.21.15.67:0/3599254772 shutdown_connections 2024-03-24T00:47:02.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.663+0000 7fbb8f436640 1 -- 172.21.15.67:0/3599254772 wait complete. 2024-03-24T00:47:02.674 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:47:02.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.774+0000 7f0067a4d640 1 Processor -- start 2024-03-24T00:47:02.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.775+0000 7f0067a4d640 1 -- start start 2024-03-24T00:47:02.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.775+0000 7f0067a4d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f006015dec0 0x7f00600ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:02.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.775+0000 7f0067a4d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f006015e2a0 0x7f0060162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:02.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.775+0000 7f0067a4d640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f00600d07e0 0x7f00600d0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:02.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.775+0000 7f0067a4d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00600d1170 con 0x7f006015e2a0 2024-03-24T00:47:02.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.775+0000 7f0067a4d640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00600cee30 con 0x7f006015dec0 2024-03-24T00:47:02.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.775+0000 7f0067a4d640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00600d2120 con 0x7f00600d07e0 2024-03-24T00:47:02.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.775+0000 7f0064fe8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f006015e2a0 0x7f0060162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:02.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.776+0000 7f0064fe8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f006015e2a0 0x7f0060162630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50982/0 (socket says 172.21.15.67:50982) 2024-03-24T00:47:02.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.776+0000 7f0064fe8640 1 -- 172.21.15.67:0/3060797222 learned_addr learned my addr 172.21.15.67:0/3060797222 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:02.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.776+0000 7f0065fea640 1 --2- 172.21.15.67:0/3060797222 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f00600d07e0 0x7f00600d0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:02.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.776+0000 7f00657e9640 1 --2- 172.21.15.67:0/3060797222 >> v2:172.21.15.72:3300/0 conn(0x7f006015dec0 0x7f00600ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:02.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.776+0000 7f0064fe8640 1 -- 172.21.15.67:0/3060797222 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f00600d07e0 msgr2=0x7f00600d0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:02.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.776+0000 7f0064fe8640 1 --2- 172.21.15.67:0/3060797222 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f00600d07e0 0x7f00600d0c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:02.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.776+0000 7f0064fe8640 1 -- 172.21.15.67:0/3060797222 >> v2:172.21.15.72:3300/0 conn(0x7f006015dec0 msgr2=0x7f00600ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:02.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.776+0000 7f0064fe8640 1 --2- 172.21.15.67:0/3060797222 >> v2:172.21.15.72:3300/0 conn(0x7f006015dec0 0x7f00600ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:02.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.776+0000 7f0064fe8640 1 -- 172.21.15.67:0/3060797222 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00600cf640 con 0x7f006015e2a0 2024-03-24T00:47:02.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.776+0000 7f00657e9640 1 --2- 172.21.15.67:0/3060797222 >> v2:172.21.15.72:3300/0 conn(0x7f006015dec0 0x7f00600ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:47:02.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.777+0000 7f0064fe8640 1 --2- 172.21.15.67:0/3060797222 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f006015e2a0 0x7f0060162630 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f00600d14a0 tx=0x7f005002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66f9198e3c16beea server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:02.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.777+0000 7f0057fff640 1 -- 172.21.15.67:0/3060797222 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f005002e070 con 0x7f006015e2a0 2024-03-24T00:47:02.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.777+0000 7f0057fff640 1 -- 172.21.15.67:0/3060797222 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0050037070 con 0x7f006015e2a0 2024-03-24T00:47:02.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.777+0000 7f0057fff640 1 -- 172.21.15.67:0/3060797222 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0050031a00 con 0x7f006015e2a0 2024-03-24T00:47:02.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.777+0000 7f0067a4d640 1 -- 172.21.15.67:0/3060797222 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f006015e2a0 msgr2=0x7f0060162630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:02.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.777+0000 7f0067a4d640 1 --2- 172.21.15.67:0/3060797222 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f006015e2a0 0x7f0060162630 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f00600d14a0 tx=0x7f005002b880 comp rx=0 tx=0).stop 2024-03-24T00:47:02.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.777+0000 7f0067a4d640 1 -- 172.21.15.67:0/3060797222 shutdown_connections 2024-03-24T00:47:02.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.777+0000 7f0067a4d640 1 --2- 172.21.15.67:0/3060797222 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f00600d07e0 0x7f00600d0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:02.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.777+0000 7f0067a4d640 1 --2- 172.21.15.67:0/3060797222 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f006015e2a0 0x7f0060162630 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:02.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.777+0000 7f0067a4d640 1 --2- 172.21.15.67:0/3060797222 >> v2:172.21.15.72:3300/0 conn(0x7f006015dec0 0x7f00600ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:02.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.777+0000 7f0067a4d640 1 -- 172.21.15.67:0/3060797222 >> 172.21.15.67:0/3060797222 conn(0x7f006006c810 msgr2=0x7f006006cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:02.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.777+0000 7f0067a4d640 1 -- 172.21.15.67:0/3060797222 shutdown_connections 2024-03-24T00:47:02.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.777+0000 7f0067a4d640 1 -- 172.21.15.67:0/3060797222 wait complete. 2024-03-24T00:47:02.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.779+0000 7f0067a4d640 1 Processor -- start 2024-03-24T00:47:02.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.779+0000 7f0067a4d640 1 -- start start 2024-03-24T00:47:02.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.780+0000 7f0067a4d640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f00600d07e0 0x7f0060158b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:02.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.780+0000 7f0067a4d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f006015dec0 0x7f0060152020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:02.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.780+0000 7f0067a4d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f006015e2a0 0x7f0060160260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:02.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.780+0000 7f0067a4d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00600b5850 con 0x7f006015dec0 2024-03-24T00:47:02.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.780+0000 7f0067a4d640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00600cf360 con 0x7f006015e2a0 2024-03-24T00:47:02.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.780+0000 7f0067a4d640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00600b59d0 con 0x7f00600d07e0 2024-03-24T00:47:02.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.780+0000 7f0064fe8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f006015dec0 0x7f0060152020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:02.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.781+0000 7f0064fe8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f006015dec0 0x7f0060152020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50994/0 (socket says 172.21.15.67:50994) 2024-03-24T00:47:02.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.781+0000 7f0064fe8640 1 -- 172.21.15.67:0/3533642062 learned_addr learned my addr 172.21.15.67:0/3533642062 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:02.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.781+0000 7f00657e9640 1 --2- 172.21.15.67:0/3533642062 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f00600d07e0 0x7f0060158b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:02.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.781+0000 7f0065fea640 1 --2- 172.21.15.67:0/3533642062 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f006015e2a0 0x7f0060160260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:02.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.781+0000 7f0064fe8640 1 -- 172.21.15.67:0/3533642062 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f00600d07e0 msgr2=0x7f0060158b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:02.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.781+0000 7f0064fe8640 1 --2- 172.21.15.67:0/3533642062 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f00600d07e0 0x7f0060158b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:02.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.781+0000 7f0064fe8640 1 -- 172.21.15.67:0/3533642062 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f006015e2a0 msgr2=0x7f0060160260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:02.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.781+0000 7f0064fe8640 1 --2- 172.21.15.67:0/3533642062 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f006015e2a0 0x7f0060160260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:02.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.781+0000 7f0064fe8640 1 -- 172.21.15.67:0/3533642062 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00600cb950 con 0x7f006015dec0 2024-03-24T00:47:02.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.781+0000 7f0065fea640 1 --2- 172.21.15.67:0/3533642062 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f006015e2a0 0x7f0060160260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:47:02.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.781+0000 7f0064fe8640 1 --2- 172.21.15.67:0/3533642062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f006015dec0 0x7f0060152020 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f005002b8c0 tx=0x7f00500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:02.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.781+0000 7f0055ffb640 1 -- 172.21.15.67:0/3533642062 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0050032880 con 0x7f006015dec0 2024-03-24T00:47:02.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.781+0000 7f0055ffb640 1 -- 172.21.15.67:0/3533642062 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0050032a20 con 0x7f006015dec0 2024-03-24T00:47:02.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.781+0000 7f0067a4d640 1 -- 172.21.15.67:0/3533642062 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00600cbc30 con 0x7f006015dec0 2024-03-24T00:47:02.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.781+0000 7f0055ffb640 1 -- 172.21.15.67:0/3533642062 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0050032e10 con 0x7f006015dec0 2024-03-24T00:47:02.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.781+0000 7f0067a4d640 1 -- 172.21.15.67:0/3533642062 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00600cc1f0 con 0x7f006015dec0 2024-03-24T00:47:02.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.783+0000 7f0055ffb640 1 -- 172.21.15.67:0/3533642062 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f005004c060 con 0x7f006015dec0 2024-03-24T00:47:02.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.783+0000 7f0067a4d640 1 -- 172.21.15.67:0/3533642062 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f006004f400 con 0x7f006015dec0 2024-03-24T00:47:02.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.783+0000 7f0055ffb640 1 --2- 172.21.15.67:0/3533642062 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f004406bbd0 0x7f004406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:02.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.783+0000 7f0055ffb640 1 -- 172.21.15.67:0/3533642062 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f00500ac860 con 0x7f006015dec0 2024-03-24T00:47:02.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.784+0000 7f00657e9640 1 --2- 172.21.15.67:0/3533642062 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f004406bbd0 0x7f004406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:02.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.784+0000 7f00657e9640 1 --2- 172.21.15.67:0/3533642062 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f004406bbd0 0x7f004406e090 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f005c00aae0 tx=0x7f005c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:02.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:02.788+0000 7f0055ffb640 1 -- 172.21.15.67:0/3533642062 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0050040070 con 0x7f006015dec0 2024-03-24T00:47:03.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.047+0000 7f0067a4d640 1 -- 172.21.15.67:0/3533642062 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00600d1aa0 con 0x7f006015dec0 2024-03-24T00:47:03.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.048+0000 7f0055ffb640 1 -- 172.21.15.67:0/3533642062 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f00600cf640 con 0x7f006015dec0 2024-03-24T00:47:03.049 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:47:03.049 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":66,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6359,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2664460392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2664460392},{"type":"v1","addr":"172.21.15.67:6835","nonce":2664460392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6368,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/3323873506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3323873506},{"type":"v1","addr":"172.21.15.72:6837","nonce":3323873506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":7150,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/2744016217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2744016217},{"type":"v1","addr":"172.21.15.72:6836","nonce":2744016217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:44:54.749656+0000","modified":"2024-03-24T00:46:24.852760+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8163},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8163":{"gid":8163,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6833/3311719072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3311719072},{"type":"v1","addr":"172.21.15.67:6833","nonce":3311719072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8163,"qdb_cluster":[8163]},"id":6}]} 2024-03-24T00:47:03.049 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 66 2024-03-24T00:47:03.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.050+0000 7f0067a4d640 1 -- 172.21.15.67:0/3533642062 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f004406bbd0 msgr2=0x7f004406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:03.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.050+0000 7f0067a4d640 1 --2- 172.21.15.67:0/3533642062 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f004406bbd0 0x7f004406e090 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f005c00aae0 tx=0x7f005c008040 comp rx=0 tx=0).stop 2024-03-24T00:47:03.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.050+0000 7f0067a4d640 1 -- 172.21.15.67:0/3533642062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f006015dec0 msgr2=0x7f0060152020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:03.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.050+0000 7f0067a4d640 1 --2- 172.21.15.67:0/3533642062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f006015dec0 0x7f0060152020 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f005002b8c0 tx=0x7f00500046c0 comp rx=0 tx=0).stop 2024-03-24T00:47:03.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.050+0000 7f0067a4d640 1 -- 172.21.15.67:0/3533642062 shutdown_connections 2024-03-24T00:47:03.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.050+0000 7f0067a4d640 1 --2- 172.21.15.67:0/3533642062 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f004406bbd0 0x7f004406e090 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:03.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.050+0000 7f0067a4d640 1 --2- 172.21.15.67:0/3533642062 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f006015e2a0 0x7f0060160260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:03.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.050+0000 7f0067a4d640 1 --2- 172.21.15.67:0/3533642062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f006015dec0 0x7f0060152020 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:03.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.050+0000 7f0067a4d640 1 --2- 172.21.15.67:0/3533642062 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f00600d07e0 0x7f0060158b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:03.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.050+0000 7f0067a4d640 1 -- 172.21.15.67:0/3533642062 >> 172.21.15.67:0/3533642062 conn(0x7f006006c810 msgr2=0x7f0060161b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:03.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.050+0000 7f0067a4d640 1 -- 172.21.15.67:0/3533642062 shutdown_connections 2024-03-24T00:47:03.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.050+0000 7f0067a4d640 1 -- 172.21.15.67:0/3533642062 wait complete. 2024-03-24T00:47:03.062 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:47:03.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.163+0000 7fd8c9bb3640 1 Processor -- start 2024-03-24T00:47:03.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.163+0000 7fd8c9bb3640 1 -- start start 2024-03-24T00:47:03.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.163+0000 7fd8c9bb3640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd8c415e2a0 0x7fd8c40ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:03.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.163+0000 7fd8c9bb3640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd8c40d0ef0 0x7fd8c4162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:03.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.163+0000 7fd8c9bb3640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8c40cf640 con 0x7fd8c415dec0 2024-03-24T00:47:03.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.163+0000 7fd8c9bb3640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8c40cee30 con 0x7fd8c415e2a0 2024-03-24T00:47:03.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.163+0000 7fd8c9bb3640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8c40d2120 con 0x7fd8c40d0ef0 2024-03-24T00:47:03.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.164+0000 7fd8c2ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd8c415e2a0 0x7fd8c40ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:03.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.164+0000 7fd8c37fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd8c415dec0 0x7fd8c40d07e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44814/0 (socket says 172.21.15.67:44814) 2024-03-24T00:47:03.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.164+0000 7fd8c37fe640 1 -- 172.21.15.67:0/2944752303 learned_addr learned my addr 172.21.15.67:0/2944752303 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:47:03.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.164+0000 7fd8c3fff640 1 --2- 172.21.15.67:0/2944752303 >> v2:172.21.15.72:3300/0 conn(0x7fd8c40d0ef0 0x7fd8c4162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:03.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.164+0000 7fd8c2ffd640 1 -- 172.21.15.67:0/2944752303 >> v2:172.21.15.72:3300/0 conn(0x7fd8c40d0ef0 msgr2=0x7fd8c4162630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:03.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.164+0000 7fd8c2ffd640 1 --2- 172.21.15.67:0/2944752303 >> v2:172.21.15.72:3300/0 conn(0x7fd8c40d0ef0 0x7fd8c4162630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:03.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.164+0000 7fd8c2ffd640 1 -- 172.21.15.67:0/2944752303 >> v1:172.21.15.67:6789/0 conn(0x7fd8c415dec0 legacy=0x7fd8c40d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:03.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.164+0000 7fd8c2ffd640 1 -- 172.21.15.67:0/2944752303 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8c40cf080 con 0x7fd8c415e2a0 2024-03-24T00:47:03.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.164+0000 7fd8c27fc640 1 -- 172.21.15.67:0/2944752303 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3820702128 0 0) 0x7fd8b4001110 con 0x7fd8c415dec0 2024-03-24T00:47:03.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.164+0000 7fd8c3fff640 1 --2- 172.21.15.67:0/2944752303 >> v2:172.21.15.72:3300/0 conn(0x7fd8c40d0ef0 0x7fd8c4162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:47:03.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.165+0000 7fd8c2ffd640 1 --2- 172.21.15.67:0/2944752303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8c415e2a0 0x7fd8c40ce630 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7fd8c40cae20 tx=0x7fd8b802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=777eda00743b0876 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:03.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.165+0000 7fd8c27fc640 1 -- 172.21.15.67:0/2944752303 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b80315a0 con 0x7fd8c415e2a0 2024-03-24T00:47:03.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.165+0000 7fd8c27fc640 1 -- 172.21.15.67:0/2944752303 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8b8037070 con 0x7fd8c415e2a0 2024-03-24T00:47:03.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.165+0000 7fd8c27fc640 1 -- 172.21.15.67:0/2944752303 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b8031960 con 0x7fd8c415e2a0 2024-03-24T00:47:03.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.165+0000 7fd8c9bb3640 1 -- 172.21.15.67:0/2944752303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8c415e2a0 msgr2=0x7fd8c40ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:03.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.165+0000 7fd8c9bb3640 1 --2- 172.21.15.67:0/2944752303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8c415e2a0 0x7fd8c40ce630 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7fd8c40cae20 tx=0x7fd8b802b840 comp rx=0 tx=0).stop 2024-03-24T00:47:03.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.165+0000 7fd8c9bb3640 1 -- 172.21.15.67:0/2944752303 shutdown_connections 2024-03-24T00:47:03.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.165+0000 7fd8c9bb3640 1 --2- 172.21.15.67:0/2944752303 >> v2:172.21.15.72:3300/0 conn(0x7fd8c40d0ef0 0x7fd8c4162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:03.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.165+0000 7fd8c9bb3640 1 --2- 172.21.15.67:0/2944752303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8c415e2a0 0x7fd8c40ce630 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:03.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.165+0000 7fd8c9bb3640 1 -- 172.21.15.67:0/2944752303 >> 172.21.15.67:0/2944752303 conn(0x7fd8c406c810 msgr2=0x7fd8c406cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:03.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.165+0000 7fd8c9bb3640 1 -- 172.21.15.67:0/2944752303 shutdown_connections 2024-03-24T00:47:03.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.165+0000 7fd8c9bb3640 1 -- 172.21.15.67:0/2944752303 wait complete. 2024-03-24T00:47:03.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.166+0000 7fd8c9bb3640 1 Processor -- start 2024-03-24T00:47:03.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.167+0000 7fd8c9bb3640 1 -- start start 2024-03-24T00:47:03.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.167+0000 7fd8c9bb3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8c40d0ef0 0x7fd8c40ab820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:03.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.167+0000 7fd8c9bb3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd8c415dec0 0x7fd8c41f3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:03.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.167+0000 7fd8c9bb3640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd8c415e2a0 0x7fd8c41f3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:03.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.167+0000 7fd8c9bb3640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c40d2730 con 0x7fd8c40d0ef0 2024-03-24T00:47:03.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.167+0000 7fd8c9bb3640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c40d25b0 con 0x7fd8c415dec0 2024-03-24T00:47:03.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.167+0000 7fd8c9bb3640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c40d28b0 con 0x7fd8c415e2a0 2024-03-24T00:47:03.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.167+0000 7fd8c37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8c40d0ef0 0x7fd8c40ab820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:03.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.167+0000 7fd8c37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8c40d0ef0 0x7fd8c40ab820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51014/0 (socket says 172.21.15.67:51014) 2024-03-24T00:47:03.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.167+0000 7fd8c37fe640 1 -- 172.21.15.67:0/866818585 learned_addr learned my addr 172.21.15.67:0/866818585 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:03.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.168+0000 7fd8c37fe640 1 -- 172.21.15.67:0/866818585 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd8c415e2a0 msgr2=0x7fd8c41f3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:03.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.168+0000 7fd8c37fe640 1 --2- 172.21.15.67:0/866818585 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd8c415e2a0 0x7fd8c41f3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:03.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.168+0000 7fd8c37fe640 1 -- 172.21.15.67:0/866818585 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd8c415dec0 msgr2=0x7fd8c41f3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:03.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.168+0000 7fd8c37fe640 1 --2- 172.21.15.67:0/866818585 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd8c415dec0 0x7fd8c41f3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:03.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.168+0000 7fd8c37fe640 1 -- 172.21.15.67:0/866818585 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8c40ad7f0 con 0x7fd8c40d0ef0 2024-03-24T00:47:03.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.168+0000 7fd8c37fe640 1 --2- 172.21.15.67:0/866818585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8c40d0ef0 0x7fd8c40ab820 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7fd8b400c780 tx=0x7fd8b400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:03.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.168+0000 7fd8abfff640 1 -- 172.21.15.67:0/866818585 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b4018070 con 0x7fd8c40d0ef0 2024-03-24T00:47:03.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.168+0000 7fd8abfff640 1 -- 172.21.15.67:0/866818585 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8b40072d0 con 0x7fd8c40d0ef0 2024-03-24T00:47:03.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.168+0000 7fd8c9bb3640 1 -- 172.21.15.67:0/866818585 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8c40abd60 con 0x7fd8c40d0ef0 2024-03-24T00:47:03.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.168+0000 7fd8abfff640 1 -- 172.21.15.67:0/866818585 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b40047e0 con 0x7fd8c40d0ef0 2024-03-24T00:47:03.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.168+0000 7fd8c9bb3640 1 -- 172.21.15.67:0/866818585 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8c40acd10 con 0x7fd8c40d0ef0 2024-03-24T00:47:03.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.169+0000 7fd8c9bb3640 1 -- 172.21.15.67:0/866818585 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8c404f400 con 0x7fd8c40d0ef0 2024-03-24T00:47:03.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.170+0000 7fd8abfff640 1 -- 172.21.15.67:0/866818585 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd8b401b070 con 0x7fd8c40d0ef0 2024-03-24T00:47:03.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.170+0000 7fd8abfff640 1 --2- 172.21.15.67:0/866818585 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd8b006bbd0 0x7fd8b006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:03.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.170+0000 7fd8abfff640 1 -- 172.21.15.67:0/866818585 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6389+0+0 (secure 0 0 0) 0x7fd8b408a050 con 0x7fd8c40d0ef0 2024-03-24T00:47:03.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.170+0000 7fd8c2ffd640 1 --2- 172.21.15.67:0/866818585 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd8b006bbd0 0x7fd8b006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:03.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.171+0000 7fd8c2ffd640 1 --2- 172.21.15.67:0/866818585 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd8b006bbd0 0x7fd8b006e090 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8036000 tx=0x7fd8b8030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:03.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.174+0000 7fd8abfff640 1 -- 172.21.15.67:0/866818585 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd8c40cf640 con 0x7fd8c40d0ef0 2024-03-24T00:47:03.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.454+0000 7fd8c9bb3640 1 -- 172.21.15.67:0/866818585 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd8c40d1aa0 con 0x7fd8c40d0ef0 2024-03-24T00:47:03.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.455+0000 7fd8abfff640 1 -- 172.21.15.67:0/866818585 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+17278 (secure 0 0 0) 0x7fd8b4058890 con 0x7fd8c40d0ef0 2024-03-24T00:47:03.456 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:47:03.456 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":93,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:46:20.877901+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:44:53.025965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-03-24T00:44:53.712957+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.32","mappings":[{"from":1,"to":7}]},{"pgid":"14.36","mappings":[{"from":1,"to":5}]},{"pgid":"14.38","mappings":[{"from":1,"to":2}]},{"pgid":"14.3a","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6833/1336395002":"2024-03-25T00:46:20.877877+0000","172.21.15.67:6832/1336395002":"2024-03-25T00:46:20.877877+0000","172.21.15.67:6833/3021622863":"2024-03-25T00:45:09.102827+0000","172.21.15.67:6832/3021622863":"2024-03-25T00:45:09.102827+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:47:03.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.458+0000 7fd8c9bb3640 1 -- 172.21.15.67:0/866818585 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd8b006bbd0 msgr2=0x7fd8b006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:03.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.458+0000 7fd8c9bb3640 1 --2- 172.21.15.67:0/866818585 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd8b006bbd0 0x7fd8b006e090 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8036000 tx=0x7fd8b8030000 comp rx=0 tx=0).stop 2024-03-24T00:47:03.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.458+0000 7fd8c9bb3640 1 -- 172.21.15.67:0/866818585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8c40d0ef0 msgr2=0x7fd8c40ab820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:03.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.458+0000 7fd8c9bb3640 1 --2- 172.21.15.67:0/866818585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8c40d0ef0 0x7fd8c40ab820 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7fd8b400c780 tx=0x7fd8b400cc40 comp rx=0 tx=0).stop 2024-03-24T00:47:03.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.458+0000 7fd8c9bb3640 1 -- 172.21.15.67:0/866818585 shutdown_connections 2024-03-24T00:47:03.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.458+0000 7fd8c9bb3640 1 --2- 172.21.15.67:0/866818585 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd8b006bbd0 0x7fd8b006e090 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:03.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.458+0000 7fd8c9bb3640 1 --2- 172.21.15.67:0/866818585 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd8c415e2a0 0x7fd8c41f3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:03.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.458+0000 7fd8c9bb3640 1 --2- 172.21.15.67:0/866818585 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd8c415dec0 0x7fd8c41f3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:03.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.458+0000 7fd8c9bb3640 1 --2- 172.21.15.67:0/866818585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8c40d0ef0 0x7fd8c40ab820 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:03.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.458+0000 7fd8c9bb3640 1 -- 172.21.15.67:0/866818585 >> 172.21.15.67:0/866818585 conn(0x7fd8c406c810 msgr2=0x7fd8c415d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:03.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.458+0000 7fd8c9bb3640 1 -- 172.21.15.67:0/866818585 shutdown_connections 2024-03-24T00:47:03.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.458+0000 7fd8c9bb3640 1 -- 172.21.15.67:0/866818585 wait complete. 2024-03-24T00:47:03.474 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-24T00:47:03.474 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:47:03.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.577+0000 7fc354cf1640 1 Processor -- start 2024-03-24T00:47:03.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.577+0000 7fc354cf1640 1 -- start start 2024-03-24T00:47:03.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.577+0000 7fc354cf1640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc35015dfe0 0x7fc3500ceb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:03.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.577+0000 7fc354cf1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc3500d3670 0x7fc3500cf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:03.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.577+0000 7fc354cf1640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc350162120 con 0x7fc350161bb0 2024-03-24T00:47:03.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.577+0000 7fc354cf1640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3500d21d0 con 0x7fc35015dfe0 2024-03-24T00:47:03.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.577+0000 7fc354cf1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc350162400 con 0x7fc3500d3670 2024-03-24T00:47:03.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.578+0000 7fc347fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc350161bb0 0x7fc3500b57c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46766/0 (socket says 172.21.15.67:46766) 2024-03-24T00:47:03.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.578+0000 7fc347fff640 1 -- 172.21.15.67:0/4203795345 learned_addr learned my addr 172.21.15.67:0/4203795345 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:47:03.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.578+0000 7fc34e59c640 1 --2- 172.21.15.67:0/4203795345 >> v2:172.21.15.72:3300/0 conn(0x7fc35015dfe0 0x7fc3500ceb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:03.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.578+0000 7fc34ed9d640 1 --2- 172.21.15.67:0/4203795345 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc3500d3670 0x7fc3500cf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:03.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.578+0000 7fc34dd9b640 1 -- 172.21.15.67:0/4203795345 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3133362825 0 0) 0x7fc350162120 con 0x7fc350161bb0 2024-03-24T00:47:03.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.578+0000 7fc34dd9b640 1 -- 172.21.15.67:0/4203795345 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc33c003520 con 0x7fc350161bb0 2024-03-24T00:47:03.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.578+0000 7fc34dd9b640 1 -- 172.21.15.67:0/4203795345 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3701550569 0 0) 0x7fc33c003520 con 0x7fc350161bb0 2024-03-24T00:47:03.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.578+0000 7fc34dd9b640 1 -- 172.21.15.67:0/4203795345 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc3500d3670 msgr2=0x7fc3500cf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:03.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.578+0000 7fc34dd9b640 1 --2- 172.21.15.67:0/4203795345 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc3500d3670 0x7fc3500cf050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:03.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.578+0000 7fc34dd9b640 1 -- 172.21.15.67:0/4203795345 >> v2:172.21.15.72:3300/0 conn(0x7fc35015dfe0 msgr2=0x7fc3500ceb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:03.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.578+0000 7fc34dd9b640 1 --2- 172.21.15.67:0/4203795345 >> v2:172.21.15.72:3300/0 conn(0x7fc35015dfe0 0x7fc3500ceb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:03.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.578+0000 7fc34dd9b640 1 -- 172.21.15.67:0/4203795345 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3501626e0 con 0x7fc350161bb0 2024-03-24T00:47:03.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.578+0000 7fc34dd9b640 1 -- 172.21.15.67:0/4203795345 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7fc338001670 con 0x7fc350161bb0 2024-03-24T00:47:03.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.578+0000 7fc34dd9b640 1 -- 172.21.15.67:0/4203795345 >> v1:172.21.15.67:6789/0 conn(0x7fc350161bb0 legacy=0x7fc3500b57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:03.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.578+0000 7fc34dd9b640 1 --2- 172.21.15.67:0/4203795345 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc3500d3670 0x7fc33c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:03.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.578+0000 7fc34dd9b640 1 --2- 172.21.15.67:0/4203795345 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc33c003f30 0x7fc33c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:03.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.578+0000 7fc34dd9b640 1 --2- 172.21.15.67:0/4203795345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc33c004d40 0x7fc33c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:03.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.578+0000 7fc34dd9b640 1 -- 172.21.15.67:0/4203795345 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3500d21d0 con 0x7fc33c004d40 2024-03-24T00:47:03.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.578+0000 7fc34dd9b640 1 -- 172.21.15.67:0/4203795345 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc350162400 con 0x7fc33c003f30 2024-03-24T00:47:03.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.578+0000 7fc34dd9b640 1 -- 172.21.15.67:0/4203795345 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc33c00b9c0 con 0x7fc3500d3670 2024-03-24T00:47:03.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.579+0000 7fc34ed9d640 1 --2- 172.21.15.67:0/4203795345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc33c004d40 0x7fc33c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:03.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.579+0000 7fc34e59c640 1 --2- 172.21.15.67:0/4203795345 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc3500d3670 0x7fc33c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:03.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.579+0000 7fc347fff640 1 --2- 172.21.15.67:0/4203795345 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc33c003f30 0x7fc33c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:03.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.579+0000 7fc34ed9d640 1 -- 172.21.15.67:0/4203795345 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc3500d3670 msgr2=0x7fc33c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:03.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.579+0000 7fc34ed9d640 1 --2- 172.21.15.67:0/4203795345 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc3500d3670 0x7fc33c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:03.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.579+0000 7fc34ed9d640 1 -- 172.21.15.67:0/4203795345 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc33c003f30 msgr2=0x7fc33c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:03.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.579+0000 7fc34ed9d640 1 --2- 172.21.15.67:0/4203795345 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc33c003f30 0x7fc33c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:03.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.579+0000 7fc34ed9d640 1 -- 172.21.15.67:0/4203795345 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc33c003520 con 0x7fc33c004d40 2024-03-24T00:47:03.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.580+0000 7fc34ed9d640 1 --2- 172.21.15.67:0/4203795345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc33c004d40 0x7fc33c0051c0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7fc348009f90 tx=0x7fc33c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=320a8699d9eefdb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:03.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.580+0000 7fc34dd9b640 1 -- 172.21.15.67:0/4203795345 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc348039070 con 0x7fc33c004d40 2024-03-24T00:47:03.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.580+0000 7fc34dd9b640 1 -- 172.21.15.67:0/4203795345 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc34802cbc0 con 0x7fc33c004d40 2024-03-24T00:47:03.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.580+0000 7fc34dd9b640 1 -- 172.21.15.67:0/4203795345 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc348009c10 con 0x7fc33c004d40 2024-03-24T00:47:03.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.580+0000 7fc354cf1640 1 -- 172.21.15.67:0/4203795345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc33c004d40 msgr2=0x7fc33c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:03.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.580+0000 7fc354cf1640 1 --2- 172.21.15.67:0/4203795345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc33c004d40 0x7fc33c0051c0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7fc348009f90 tx=0x7fc33c005830 comp rx=0 tx=0).stop 2024-03-24T00:47:03.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.580+0000 7fc34ed9d640 1 -- 172.21.15.67:0/4203795345 reap_dead start 2024-03-24T00:47:03.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.580+0000 7fc354cf1640 1 -- 172.21.15.67:0/4203795345 shutdown_connections 2024-03-24T00:47:03.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.580+0000 7fc354cf1640 1 -- 172.21.15.67:0/4203795345 >> 172.21.15.67:0/4203795345 conn(0x7fc35006c890 msgr2=0x7fc35006cc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:03.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.580+0000 7fc354cf1640 1 -- 172.21.15.67:0/4203795345 shutdown_connections 2024-03-24T00:47:03.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.580+0000 7fc354cf1640 1 -- 172.21.15.67:0/4203795345 wait complete. 2024-03-24T00:47:03.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.582+0000 7fc354cf1640 1 Processor -- start 2024-03-24T00:47:03.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.582+0000 7fc354cf1640 1 -- start start 2024-03-24T00:47:03.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.583+0000 7fc354cf1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc35015b120 0x7fc3501f4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:03.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.583+0000 7fc354cf1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3500b5b10 0x7fc3500b5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:03.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.583+0000 7fc354cf1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc350160550 0x7fc3501609d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:03.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.583+0000 7fc354cf1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3500d1120 con 0x7fc350160550 2024-03-24T00:47:03.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.583+0000 7fc354cf1640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc35015e3c0 con 0x7fc3500b5b10 2024-03-24T00:47:03.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.583+0000 7fc354cf1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3500d12a0 con 0x7fc35015b120 2024-03-24T00:47:03.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.583+0000 7fc34ed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc350160550 0x7fc3501609d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:03.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.583+0000 7fc34ed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc350160550 0x7fc3501609d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39232/0 (socket says 172.21.15.67:39232) 2024-03-24T00:47:03.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.583+0000 7fc34ed9d640 1 -- 172.21.15.67:0/2315598491 learned_addr learned my addr 172.21.15.67:0/2315598491 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:03.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.583+0000 7fc347fff640 1 --2- 172.21.15.67:0/2315598491 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3500b5b10 0x7fc3500b5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:03.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.583+0000 7fc34e59c640 1 --2- 172.21.15.67:0/2315598491 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc35015b120 0x7fc3501f4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:03.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.583+0000 7fc34ed9d640 1 -- 172.21.15.67:0/2315598491 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc35015b120 msgr2=0x7fc3501f4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:03.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.583+0000 7fc34ed9d640 1 --2- 172.21.15.67:0/2315598491 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc35015b120 0x7fc3501f4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:03.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.583+0000 7fc34ed9d640 1 -- 172.21.15.67:0/2315598491 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3500b5b10 msgr2=0x7fc3500b5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:03.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.583+0000 7fc34ed9d640 1 --2- 172.21.15.67:0/2315598491 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3500b5b10 0x7fc3500b5180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:03.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.583+0000 7fc34ed9d640 1 -- 172.21.15.67:0/2315598491 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3500cc4d0 con 0x7fc350160550 2024-03-24T00:47:03.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.583+0000 7fc34ed9d640 1 --2- 172.21.15.67:0/2315598491 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc350160550 0x7fc3501609d0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7fc34800e230 tx=0x7fc348009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:03.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.583+0000 7fc346ffd640 1 -- 172.21.15.67:0/2315598491 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc348039030 con 0x7fc350160550 2024-03-24T00:47:03.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.583+0000 7fc346ffd640 1 -- 172.21.15.67:0/2315598491 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc34803a040 con 0x7fc350160550 2024-03-24T00:47:03.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.583+0000 7fc354cf1640 1 -- 172.21.15.67:0/2315598491 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3500cc7b0 con 0x7fc350160550 2024-03-24T00:47:03.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.583+0000 7fc346ffd640 1 -- 172.21.15.67:0/2315598491 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc348009c10 con 0x7fc350160550 2024-03-24T00:47:03.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.583+0000 7fc354cf1640 1 -- 172.21.15.67:0/2315598491 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3500b56c0 con 0x7fc350160550 2024-03-24T00:47:03.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.584+0000 7fc354cf1640 1 -- 172.21.15.67:0/2315598491 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc35006b180 con 0x7fc350160550 2024-03-24T00:47:03.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.585+0000 7fc346ffd640 1 -- 172.21.15.67:0/2315598491 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc348045020 con 0x7fc350160550 2024-03-24T00:47:03.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.586+0000 7fc346ffd640 1 --2- 172.21.15.67:0/2315598491 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc33006bc00 0x7fc33006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:03.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.586+0000 7fc346ffd640 1 -- 172.21.15.67:0/2315598491 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6389+0+0 (secure 0 0 0) 0x7fc3480accf0 con 0x7fc350160550 2024-03-24T00:47:03.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.586+0000 7fc34e59c640 1 --2- 172.21.15.67:0/2315598491 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc33006bc00 0x7fc33006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:03.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.587+0000 7fc34e59c640 1 --2- 172.21.15.67:0/2315598491 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc33006bc00 0x7fc33006e0c0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fc3500cdff0 tx=0x7fc340008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:03.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.590+0000 7fc346ffd640 1 -- 172.21.15.67:0/2315598491 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc3480b1070 con 0x7fc350160550 2024-03-24T00:47:03.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.811+0000 7fc354cf1640 1 -- 172.21.15.67:0/2315598491 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc3500cccd0 con 0x7fc350160550 2024-03-24T00:47:03.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.812+0000 7fc346ffd640 1 -- 172.21.15.67:0/2315598491 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4980 (secure 0 0 0) 0x7fc3500cccd0 con 0x7fc350160550 2024-03-24T00:47:03.813 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:47:03.813 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":66,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6359,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2664460392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2664460392},{"type":"v1","addr":"172.21.15.67:6835","nonce":2664460392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6368,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/3323873506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3323873506},{"type":"v1","addr":"172.21.15.72:6837","nonce":3323873506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":7150,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/2744016217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2744016217},{"type":"v1","addr":"172.21.15.72:6836","nonce":2744016217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:44:54.749656+0000","modified":"2024-03-24T00:46:24.852760+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8163},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8163":{"gid":8163,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6833/3311719072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3311719072},{"type":"v1","addr":"172.21.15.67:6833","nonce":3311719072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8163,"qdb_cluster":[8163]},"id":6}]} 2024-03-24T00:47:03.813 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 66 2024-03-24T00:47:03.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.814+0000 7fc354cf1640 1 -- 172.21.15.67:0/2315598491 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc33006bc00 msgr2=0x7fc33006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:03.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.814+0000 7fc354cf1640 1 --2- 172.21.15.67:0/2315598491 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc33006bc00 0x7fc33006e0c0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fc3500cdff0 tx=0x7fc340008040 comp rx=0 tx=0).stop 2024-03-24T00:47:03.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.814+0000 7fc354cf1640 1 -- 172.21.15.67:0/2315598491 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc350160550 msgr2=0x7fc3501609d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:03.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.814+0000 7fc354cf1640 1 --2- 172.21.15.67:0/2315598491 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc350160550 0x7fc3501609d0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7fc34800e230 tx=0x7fc348009750 comp rx=0 tx=0).stop 2024-03-24T00:47:03.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.814+0000 7fc354cf1640 1 -- 172.21.15.67:0/2315598491 shutdown_connections 2024-03-24T00:47:03.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.814+0000 7fc354cf1640 1 --2- 172.21.15.67:0/2315598491 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc33006bc00 0x7fc33006e0c0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:03.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.814+0000 7fc354cf1640 1 --2- 172.21.15.67:0/2315598491 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc350160550 0x7fc3501609d0 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:03.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.814+0000 7fc354cf1640 1 --2- 172.21.15.67:0/2315598491 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3500b5b10 0x7fc3500b5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:03.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.814+0000 7fc354cf1640 1 --2- 172.21.15.67:0/2315598491 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc35015b120 0x7fc3501f4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:03.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.814+0000 7fc354cf1640 1 -- 172.21.15.67:0/2315598491 >> 172.21.15.67:0/2315598491 conn(0x7fc35006c890 msgr2=0x7fc3500c7c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:03.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.814+0000 7fc354cf1640 1 -- 172.21.15.67:0/2315598491 shutdown_connections 2024-03-24T00:47:03.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.814+0000 7fc354cf1640 1 -- 172.21.15.67:0/2315598491 wait complete. 2024-03-24T00:47:03.825 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:47:03.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.927+0000 7ff1ede81640 1 Processor -- start 2024-03-24T00:47:03.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.927+0000 7ff1ede81640 1 -- start start 2024-03-24T00:47:03.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.928+0000 7ff1ede81640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff1e815dfe0 0x7ff1e80ceb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:03.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.928+0000 7ff1ede81640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff1e8161bb0 0x7ff1e80cf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:03.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.928+0000 7ff1ede81640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1e8162120 con 0x7ff1e80b96d0 2024-03-24T00:47:03.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.928+0000 7ff1ede81640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1e80d21d0 con 0x7ff1e815dfe0 2024-03-24T00:47:03.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.928+0000 7ff1ede81640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e8162400 con 0x7ff1e8161bb0 2024-03-24T00:47:03.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.928+0000 7ff1e77fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff1e815dfe0 0x7ff1e80ceb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:03.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.928+0000 7ff1e77fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff1e815dfe0 0x7ff1e80ceb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39248/0 (socket says 172.21.15.67:39248) 2024-03-24T00:47:03.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.928+0000 7ff1e77fe640 1 -- 172.21.15.67:0/2834017700 learned_addr learned my addr 172.21.15.67:0/2834017700 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:03.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.928+0000 7ff1e7fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff1e80b96d0 0x7ff1e80b57c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46782/0 (socket says 172.21.15.67:46782) 2024-03-24T00:47:03.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.928+0000 7ff1e6ffd640 1 --2- 172.21.15.67:0/2834017700 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff1e8161bb0 0x7ff1e80cf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:03.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.928+0000 7ff1e77fe640 1 -- 172.21.15.67:0/2834017700 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff1e8161bb0 msgr2=0x7ff1e80cf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:03.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.928+0000 7ff1e77fe640 1 --2- 172.21.15.67:0/2834017700 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff1e8161bb0 0x7ff1e80cf050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:03.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.928+0000 7ff1e77fe640 1 -- 172.21.15.67:0/2834017700 >> v1:172.21.15.67:6789/0 conn(0x7ff1e80b96d0 legacy=0x7ff1e80b57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:03.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.928+0000 7ff1e67fc640 1 -- 172.21.15.67:0/2834017700 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 562724817 0 0) 0x7ff1e8162120 con 0x7ff1e80b96d0 2024-03-24T00:47:03.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.928+0000 7ff1e77fe640 1 -- 172.21.15.67:0/2834017700 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1e81626e0 con 0x7ff1e815dfe0 2024-03-24T00:47:03.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.928+0000 7ff1e6ffd640 1 --2- 172.21.15.67:0/2834017700 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff1e8161bb0 0x7ff1e80cf050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:47:03.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.929+0000 7ff1e77fe640 1 --2- 172.21.15.67:0/2834017700 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1e815dfe0 0x7ff1e80ceb10 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7ff1d0009870 tx=0x7ff1d002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=592261155b9f72be server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:03.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.929+0000 7ff1e67fc640 1 -- 172.21.15.67:0/2834017700 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d0034070 con 0x7ff1e815dfe0 2024-03-24T00:47:03.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.929+0000 7ff1e67fc640 1 -- 172.21.15.67:0/2834017700 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1d0030840 con 0x7ff1e815dfe0 2024-03-24T00:47:03.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.929+0000 7ff1e67fc640 1 -- 172.21.15.67:0/2834017700 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d0030c10 con 0x7ff1e815dfe0 2024-03-24T00:47:03.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.930+0000 7ff1ede81640 1 -- 172.21.15.67:0/2834017700 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1e815dfe0 msgr2=0x7ff1e80ceb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:03.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.930+0000 7ff1ede81640 1 --2- 172.21.15.67:0/2834017700 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1e815dfe0 0x7ff1e80ceb10 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7ff1d0009870 tx=0x7ff1d002b980 comp rx=0 tx=0).stop 2024-03-24T00:47:03.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.930+0000 7ff1ede81640 1 -- 172.21.15.67:0/2834017700 shutdown_connections 2024-03-24T00:47:03.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.930+0000 7ff1ede81640 1 --2- 172.21.15.67:0/2834017700 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff1e8161bb0 0x7ff1e80cf050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:03.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.930+0000 7ff1ede81640 1 --2- 172.21.15.67:0/2834017700 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1e815dfe0 0x7ff1e80ceb10 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:03.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.930+0000 7ff1ede81640 1 -- 172.21.15.67:0/2834017700 >> 172.21.15.67:0/2834017700 conn(0x7ff1e806c890 msgr2=0x7ff1e806cc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:03.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.930+0000 7ff1ede81640 1 -- 172.21.15.67:0/2834017700 shutdown_connections 2024-03-24T00:47:03.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.930+0000 7ff1ede81640 1 -- 172.21.15.67:0/2834017700 wait complete. 2024-03-24T00:47:03.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.931+0000 7ff1ede81640 1 Processor -- start 2024-03-24T00:47:03.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.931+0000 7ff1ede81640 1 -- start start 2024-03-24T00:47:03.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.932+0000 7ff1ede81640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1e80b96d0 0x7ff1e815ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:03.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.932+0000 7ff1ede81640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1e815dfe0 0x7ff1e8155220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:03.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.932+0000 7ff1ede81640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff1e8161bb0 0x7ff1e8151e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:03.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.932+0000 7ff1ede81640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e80d1120 con 0x7ff1e815dfe0 2024-03-24T00:47:03.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.932+0000 7ff1ede81640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e815e3c0 con 0x7ff1e80b96d0 2024-03-24T00:47:03.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.932+0000 7ff1ede81640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e80d12a0 con 0x7ff1e8161bb0 2024-03-24T00:47:03.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.932+0000 7ff1e6ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1e815dfe0 0x7ff1e8155220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:03.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.932+0000 7ff1e6ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1e815dfe0 0x7ff1e8155220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39252/0 (socket says 172.21.15.67:39252) 2024-03-24T00:47:03.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.932+0000 7ff1e6ffd640 1 -- 172.21.15.67:0/1677818419 learned_addr learned my addr 172.21.15.67:0/1677818419 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:03.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.932+0000 7ff1e7fff640 1 --2- 172.21.15.67:0/1677818419 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff1e8161bb0 0x7ff1e8151e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:03.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.932+0000 7ff1e77fe640 1 --2- 172.21.15.67:0/1677818419 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1e80b96d0 0x7ff1e815ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:03.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.932+0000 7ff1e6ffd640 1 -- 172.21.15.67:0/1677818419 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff1e8161bb0 msgr2=0x7ff1e8151e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:03.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.932+0000 7ff1e6ffd640 1 --2- 172.21.15.67:0/1677818419 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff1e8161bb0 0x7ff1e8151e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:03.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.932+0000 7ff1e6ffd640 1 -- 172.21.15.67:0/1677818419 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1e80b96d0 msgr2=0x7ff1e815ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:03.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.932+0000 7ff1e6ffd640 1 --2- 172.21.15.67:0/1677818419 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1e80b96d0 0x7ff1e815ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:03.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.932+0000 7ff1e6ffd640 1 -- 172.21.15.67:0/1677818419 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1e80cd2e0 con 0x7ff1e815dfe0 2024-03-24T00:47:03.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.932+0000 7ff1e7fff640 1 --2- 172.21.15.67:0/1677818419 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff1e8161bb0 0x7ff1e8151e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:47:03.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.932+0000 7ff1e6ffd640 1 --2- 172.21.15.67:0/1677818419 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1e815dfe0 0x7ff1e8155220 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7ff1d800d7d0 tx=0x7ff1d800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:03.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.932+0000 7ff1dffff640 1 -- 172.21.15.67:0/1677818419 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d8004030 con 0x7ff1e815dfe0 2024-03-24T00:47:03.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.932+0000 7ff1dffff640 1 -- 172.21.15.67:0/1677818419 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1d80041d0 con 0x7ff1e815dfe0 2024-03-24T00:47:03.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.932+0000 7ff1ede81640 1 -- 172.21.15.67:0/1677818419 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1e8152d80 con 0x7ff1e815dfe0 2024-03-24T00:47:03.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.932+0000 7ff1dffff640 1 -- 172.21.15.67:0/1677818419 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d80096a0 con 0x7ff1e815dfe0 2024-03-24T00:47:03.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.933+0000 7ff1ede81640 1 -- 172.21.15.67:0/1677818419 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1e8155760 con 0x7ff1e815dfe0 2024-03-24T00:47:03.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.934+0000 7ff1dffff640 1 -- 172.21.15.67:0/1677818419 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff1d8009840 con 0x7ff1e815dfe0 2024-03-24T00:47:03.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.934+0000 7ff1ede81640 1 -- 172.21.15.67:0/1677818419 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1e806b180 con 0x7ff1e815dfe0 2024-03-24T00:47:03.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.934+0000 7ff1dffff640 1 --2- 172.21.15.67:0/1677818419 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff1c806bbd0 0x7ff1c806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:03.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.934+0000 7ff1dffff640 1 -- 172.21.15.67:0/1677818419 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6389+0+0 (secure 0 0 0) 0x7ff1d808afb0 con 0x7ff1e815dfe0 2024-03-24T00:47:03.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.935+0000 7ff1e77fe640 1 --2- 172.21.15.67:0/1677818419 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff1c806bbd0 0x7ff1c806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:03.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.935+0000 7ff1e77fe640 1 --2- 172.21.15.67:0/1677818419 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff1c806bbd0 0x7ff1c806e090 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7ff1d0036000 tx=0x7ff1d003b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:03.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:03.939+0000 7ff1dffff640 1 -- 172.21.15.67:0/1677818419 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff1d8059880 con 0x7ff1e815dfe0 2024-03-24T00:47:04.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.179+0000 7ff1ede81640 1 -- 172.21.15.67:0/1677818419 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff1e8152b70 con 0x7ff1e815dfe0 2024-03-24T00:47:04.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.180+0000 7ff1dffff640 1 -- 172.21.15.67:0/1677818419 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4980 (secure 0 0 0) 0x7ff1d8059180 con 0x7ff1e815dfe0 2024-03-24T00:47:04.181 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:47:04.181 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":66,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6359,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2664460392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2664460392},{"type":"v1","addr":"172.21.15.67:6835","nonce":2664460392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6368,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/3323873506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3323873506},{"type":"v1","addr":"172.21.15.72:6837","nonce":3323873506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":7150,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/2744016217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2744016217},{"type":"v1","addr":"172.21.15.72:6836","nonce":2744016217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:44:54.749656+0000","modified":"2024-03-24T00:46:24.852760+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8163},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8163":{"gid":8163,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6833/3311719072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3311719072},{"type":"v1","addr":"172.21.15.67:6833","nonce":3311719072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8163,"qdb_cluster":[8163]},"id":6}]} 2024-03-24T00:47:04.181 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 66 2024-03-24T00:47:04.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.181+0000 7ff1ede81640 1 -- 172.21.15.67:0/1677818419 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff1c806bbd0 msgr2=0x7ff1c806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:04.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.181+0000 7ff1ede81640 1 --2- 172.21.15.67:0/1677818419 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff1c806bbd0 0x7ff1c806e090 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7ff1d0036000 tx=0x7ff1d003b040 comp rx=0 tx=0).stop 2024-03-24T00:47:04.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.181+0000 7ff1ede81640 1 -- 172.21.15.67:0/1677818419 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1e815dfe0 msgr2=0x7ff1e8155220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:04.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.181+0000 7ff1ede81640 1 --2- 172.21.15.67:0/1677818419 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1e815dfe0 0x7ff1e8155220 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7ff1d800d7d0 tx=0x7ff1d800dc90 comp rx=0 tx=0).stop 2024-03-24T00:47:04.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.181+0000 7ff1ede81640 1 -- 172.21.15.67:0/1677818419 shutdown_connections 2024-03-24T00:47:04.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.181+0000 7ff1ede81640 1 --2- 172.21.15.67:0/1677818419 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff1c806bbd0 0x7ff1c806e090 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:04.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.181+0000 7ff1ede81640 1 --2- 172.21.15.67:0/1677818419 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff1e8161bb0 0x7ff1e8151e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:04.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.181+0000 7ff1ede81640 1 --2- 172.21.15.67:0/1677818419 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1e815dfe0 0x7ff1e8155220 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:04.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.181+0000 7ff1ede81640 1 --2- 172.21.15.67:0/1677818419 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1e80b96d0 0x7ff1e815ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:04.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.181+0000 7ff1ede81640 1 -- 172.21.15.67:0/1677818419 >> 172.21.15.67:0/1677818419 conn(0x7ff1e806c890 msgr2=0x7ff1e80b3730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:04.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.182+0000 7ff1ede81640 1 -- 172.21.15.67:0/1677818419 shutdown_connections 2024-03-24T00:47:04.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.182+0000 7ff1ede81640 1 -- 172.21.15.67:0/1677818419 wait complete. 2024-03-24T00:47:04.193 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:47:04.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.294+0000 7fb482590640 1 Processor -- start 2024-03-24T00:47:04.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.294+0000 7fb482590640 1 -- start start 2024-03-24T00:47:04.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.295+0000 7fb482590640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb47c15bf70 0x7fb47c0cc690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:04.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.295+0000 7fb482590640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb47c0ce930 0x7fb47c0b1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:04.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.295+0000 7fb482590640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb47c0ccfe0 con 0x7fb47c0cfc70 2024-03-24T00:47:04.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.295+0000 7fb482590640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb47c0cd2c0 con 0x7fb47c0ce930 2024-03-24T00:47:04.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.295+0000 7fb482590640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c0cd5a0 con 0x7fb47c15bf70 2024-03-24T00:47:04.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.295+0000 7fb47b7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb47c0cfc70 0x7fb47c0ce220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46798/0 (socket says 172.21.15.67:46798) 2024-03-24T00:47:04.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.295+0000 7fb47b7fe640 1 -- 172.21.15.67:0/3314852713 learned_addr learned my addr 172.21.15.67:0/3314852713 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:47:04.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.295+0000 7fb47bfff640 1 --2- 172.21.15.67:0/3314852713 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb47c15bf70 0x7fb47c0cc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:04.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.295+0000 7fb480b2d640 1 --2- 172.21.15.67:0/3314852713 >> v2:172.21.15.72:3300/0 conn(0x7fb47c0ce930 0x7fb47c0b1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:04.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.295+0000 7fb47affd640 1 -- 172.21.15.67:0/3314852713 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2023634802 0 0) 0x7fb47c0ccfe0 con 0x7fb47c0cfc70 2024-03-24T00:47:04.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.295+0000 7fb47affd640 1 -- 172.21.15.67:0/3314852713 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb464003520 con 0x7fb47c0cfc70 2024-03-24T00:47:04.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.295+0000 7fb47bfff640 1 -- 172.21.15.67:0/3314852713 >> v2:172.21.15.72:3300/0 conn(0x7fb47c0ce930 msgr2=0x7fb47c0b1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:04.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.295+0000 7fb47bfff640 1 --2- 172.21.15.67:0/3314852713 >> v2:172.21.15.72:3300/0 conn(0x7fb47c0ce930 0x7fb47c0b1cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:04.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.295+0000 7fb47affd640 1 -- 172.21.15.67:0/3314852713 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2056390211 0 0) 0x7fb464003520 con 0x7fb47c0cfc70 2024-03-24T00:47:04.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.295+0000 7fb47bfff640 1 -- 172.21.15.67:0/3314852713 >> v1:172.21.15.67:6789/0 conn(0x7fb47c0cfc70 legacy=0x7fb47c0ce220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:04.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.295+0000 7fb47bfff640 1 -- 172.21.15.67:0/3314852713 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb47c0cd880 con 0x7fb47c15bf70 2024-03-24T00:47:04.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.295+0000 7fb480b2d640 1 --2- 172.21.15.67:0/3314852713 >> v2:172.21.15.72:3300/0 conn(0x7fb47c0ce930 0x7fb47c0b1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:47:04.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.296+0000 7fb47bfff640 1 --2- 172.21.15.67:0/3314852713 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb47c15bf70 0x7fb47c0cc690 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb46c0097a0 tx=0x7fb46c02b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9f9a1a3bfe3157c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:04.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.296+0000 7fb47affd640 1 -- 172.21.15.67:0/3314852713 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46c037070 con 0x7fb47c15bf70 2024-03-24T00:47:04.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.296+0000 7fb47affd640 1 -- 172.21.15.67:0/3314852713 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb46c030510 con 0x7fb47c15bf70 2024-03-24T00:47:04.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.296+0000 7fb47affd640 1 -- 172.21.15.67:0/3314852713 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb470001670 con 0x7fb47c15bf70 2024-03-24T00:47:04.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.297+0000 7fb482590640 1 -- 172.21.15.67:0/3314852713 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb47c15bf70 msgr2=0x7fb47c0cc690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:04.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.297+0000 7fb482590640 1 --2- 172.21.15.67:0/3314852713 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb47c15bf70 0x7fb47c0cc690 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb46c0097a0 tx=0x7fb46c02b790 comp rx=0 tx=0).stop 2024-03-24T00:47:04.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.297+0000 7fb482590640 1 -- 172.21.15.67:0/3314852713 shutdown_connections 2024-03-24T00:47:04.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.297+0000 7fb482590640 1 --2- 172.21.15.67:0/3314852713 >> v2:172.21.15.72:3300/0 conn(0x7fb47c0ce930 0x7fb47c0b1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:04.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.297+0000 7fb482590640 1 --2- 172.21.15.67:0/3314852713 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb47c15bf70 0x7fb47c0cc690 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:04.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.297+0000 7fb482590640 1 -- 172.21.15.67:0/3314852713 >> 172.21.15.67:0/3314852713 conn(0x7fb47c0d1bf0 msgr2=0x7fb47c0d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:04.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.297+0000 7fb482590640 1 -- 172.21.15.67:0/3314852713 shutdown_connections 2024-03-24T00:47:04.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.297+0000 7fb482590640 1 -- 172.21.15.67:0/3314852713 wait complete. 2024-03-24T00:47:04.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.298+0000 7fb482590640 1 Processor -- start 2024-03-24T00:47:04.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.298+0000 7fb482590640 1 -- start start 2024-03-24T00:47:04.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.299+0000 7fb482590640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb47c0ce930 0x7fb47c150200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:04.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.299+0000 7fb482590640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb47c0cfc70 0x7fb47c15d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:04.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.299+0000 7fb482590640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb47c15bf70 0x7fb47c0b69b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:04.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.299+0000 7fb482590640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c0ccd50 con 0x7fb47c15bf70 2024-03-24T00:47:04.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.299+0000 7fb482590640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c0ccbd0 con 0x7fb47c0cfc70 2024-03-24T00:47:04.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.299+0000 7fb482590640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c0b2210 con 0x7fb47c0ce930 2024-03-24T00:47:04.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.299+0000 7fb480b2d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb47c15bf70 0x7fb47c0b69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:04.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.299+0000 7fb480b2d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb47c15bf70 0x7fb47c0b69b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39268/0 (socket says 172.21.15.67:39268) 2024-03-24T00:47:04.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.299+0000 7fb480b2d640 1 -- 172.21.15.67:0/2556799265 learned_addr learned my addr 172.21.15.67:0/2556799265 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:04.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.299+0000 7fb47b7fe640 1 --2- 172.21.15.67:0/2556799265 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb47c0cfc70 0x7fb47c15d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:04.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.299+0000 7fb47bfff640 1 --2- 172.21.15.67:0/2556799265 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb47c0ce930 0x7fb47c150200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:04.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.299+0000 7fb480b2d640 1 -- 172.21.15.67:0/2556799265 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb47c0ce930 msgr2=0x7fb47c150200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:04.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.299+0000 7fb480b2d640 1 --2- 172.21.15.67:0/2556799265 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb47c0ce930 0x7fb47c150200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:04.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.299+0000 7fb480b2d640 1 -- 172.21.15.67:0/2556799265 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb47c0cfc70 msgr2=0x7fb47c15d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:04.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.299+0000 7fb480b2d640 1 --2- 172.21.15.67:0/2556799265 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb47c0cfc70 0x7fb47c15d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:04.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.299+0000 7fb480b2d640 1 -- 172.21.15.67:0/2556799265 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb47c15e220 con 0x7fb47c15bf70 2024-03-24T00:47:04.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.299+0000 7fb47bfff640 1 --2- 172.21.15.67:0/2556799265 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb47c0ce930 0x7fb47c150200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:47:04.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.299+0000 7fb47b7fe640 1 --2- 172.21.15.67:0/2556799265 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb47c0cfc70 0x7fb47c15d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:47:04.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.300+0000 7fb480b2d640 1 --2- 172.21.15.67:0/2556799265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb47c15bf70 0x7fb47c0b69b0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7fb474009eb0 tx=0x7fb47400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:04.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.300+0000 7fb478ff9640 1 -- 172.21.15.67:0/2556799265 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb474018020 con 0x7fb47c15bf70 2024-03-24T00:47:04.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.300+0000 7fb478ff9640 1 -- 172.21.15.67:0/2556799265 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb474014480 con 0x7fb47c15bf70 2024-03-24T00:47:04.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.300+0000 7fb482590640 1 -- 172.21.15.67:0/2556799265 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb47c15e500 con 0x7fb47c15bf70 2024-03-24T00:47:04.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.300+0000 7fb478ff9640 1 -- 172.21.15.67:0/2556799265 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb474014850 con 0x7fb47c15bf70 2024-03-24T00:47:04.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.300+0000 7fb482590640 1 -- 172.21.15.67:0/2556799265 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb47c15eac0 con 0x7fb47c15bf70 2024-03-24T00:47:04.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.301+0000 7fb482590640 1 -- 172.21.15.67:0/2556799265 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb47c04f400 con 0x7fb47c15bf70 2024-03-24T00:47:04.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.301+0000 7fb478ff9640 1 -- 172.21.15.67:0/2556799265 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb474005d10 con 0x7fb47c15bf70 2024-03-24T00:47:04.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.302+0000 7fb478ff9640 1 --2- 172.21.15.67:0/2556799265 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb46806bbd0 0x7fb46806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:04.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.302+0000 7fb478ff9640 1 -- 172.21.15.67:0/2556799265 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6389+0+0 (secure 0 0 0) 0x7fb47408bb50 con 0x7fb47c15bf70 2024-03-24T00:47:04.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.302+0000 7fb47bfff640 1 --2- 172.21.15.67:0/2556799265 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb46806bbd0 0x7fb46806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:04.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.302+0000 7fb47bfff640 1 --2- 172.21.15.67:0/2556799265 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb46806bbd0 0x7fb46806e090 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fb46c034000 tx=0x7fb46c02f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:04.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.306+0000 7fb478ff9640 1 -- 172.21.15.67:0/2556799265 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb474022030 con 0x7fb47c15bf70 2024-03-24T00:47:04.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.532+0000 7fb482590640 1 -- 172.21.15.67:0/2556799265 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb47c0b2e80 con 0x7fb47c15bf70 2024-03-24T00:47:04.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.534+0000 7fb478ff9640 1 -- 172.21.15.67:0/2556799265 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+17278 (secure 0 0 0) 0x7fb474010040 con 0x7fb47c15bf70 2024-03-24T00:47:04.535 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:47:04.535 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":93,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:46:20.877901+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:44:53.025965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-03-24T00:44:53.712957+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.32","mappings":[{"from":1,"to":7}]},{"pgid":"14.36","mappings":[{"from":1,"to":5}]},{"pgid":"14.38","mappings":[{"from":1,"to":2}]},{"pgid":"14.3a","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6833/1336395002":"2024-03-25T00:46:20.877877+0000","172.21.15.67:6832/1336395002":"2024-03-25T00:46:20.877877+0000","172.21.15.67:6833/3021622863":"2024-03-25T00:45:09.102827+0000","172.21.15.67:6832/3021622863":"2024-03-25T00:45:09.102827+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:47:04.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.536+0000 7fb482590640 1 -- 172.21.15.67:0/2556799265 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb46806bbd0 msgr2=0x7fb46806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:04.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.536+0000 7fb482590640 1 --2- 172.21.15.67:0/2556799265 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb46806bbd0 0x7fb46806e090 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fb46c034000 tx=0x7fb46c02f000 comp rx=0 tx=0).stop 2024-03-24T00:47:04.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.536+0000 7fb482590640 1 -- 172.21.15.67:0/2556799265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb47c15bf70 msgr2=0x7fb47c0b69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:04.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.536+0000 7fb482590640 1 --2- 172.21.15.67:0/2556799265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb47c15bf70 0x7fb47c0b69b0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7fb474009eb0 tx=0x7fb47400c490 comp rx=0 tx=0).stop 2024-03-24T00:47:04.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.536+0000 7fb482590640 1 -- 172.21.15.67:0/2556799265 shutdown_connections 2024-03-24T00:47:04.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.536+0000 7fb482590640 1 --2- 172.21.15.67:0/2556799265 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb46806bbd0 0x7fb46806e090 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:04.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.536+0000 7fb482590640 1 --2- 172.21.15.67:0/2556799265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb47c15bf70 0x7fb47c0b69b0 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:04.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.536+0000 7fb482590640 1 --2- 172.21.15.67:0/2556799265 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb47c0cfc70 0x7fb47c15d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:04.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.536+0000 7fb482590640 1 --2- 172.21.15.67:0/2556799265 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb47c0ce930 0x7fb47c150200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:04.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.536+0000 7fb482590640 1 -- 172.21.15.67:0/2556799265 >> 172.21.15.67:0/2556799265 conn(0x7fb47c0d1bf0 msgr2=0x7fb47c070780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:04.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.536+0000 7fb482590640 1 -- 172.21.15.67:0/2556799265 shutdown_connections 2024-03-24T00:47:04.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.536+0000 7fb482590640 1 -- 172.21.15.67:0/2556799265 wait complete. 2024-03-24T00:47:04.549 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-24T00:47:04.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.661+0000 7f14cc3e8640 1 Processor -- start 2024-03-24T00:47:04.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.661+0000 7f14cc3e8640 1 -- start start 2024-03-24T00:47:04.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.663+0000 7f14cc3e8640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f14c415e0a0 0x7f14c40cd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:04.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.663+0000 7f14cc3e8640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f14c40cdc30 0x7f14c406b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:04.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.663+0000 7f14cc3e8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f14c406b580 0x7f14c40d0a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:04.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.663+0000 7f14cc3e8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14c40afd80 con 0x7f14c406b580 2024-03-24T00:47:04.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.663+0000 7f14cc3e8640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14c40cb0f0 con 0x7f14c40cdc30 2024-03-24T00:47:04.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.663+0000 7f14cc3e8640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14c40d0080 con 0x7f14c415e0a0 2024-03-24T00:47:04.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.663+0000 7f14ca985640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f14c406b580 0x7f14c40d0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:04.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.663+0000 7f14ca985640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f14c406b580 0x7f14c40d0a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39276/0 (socket says 172.21.15.67:39276) 2024-03-24T00:47:04.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.663+0000 7f14ca985640 1 -- 172.21.15.67:0/4154764431 learned_addr learned my addr 172.21.15.67:0/4154764431 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:04.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.663+0000 7f14ca184640 1 --2- 172.21.15.67:0/4154764431 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f14c415e0a0 0x7f14c40cd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:04.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.663+0000 7f14c9983640 1 --2- 172.21.15.67:0/4154764431 >> v2:172.21.15.72:3300/0 conn(0x7f14c40cdc30 0x7f14c406b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:04.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.663+0000 7f14ca985640 1 -- 172.21.15.67:0/4154764431 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f14c415e0a0 msgr2=0x7f14c40cd6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:04.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.663+0000 7f14ca985640 1 --2- 172.21.15.67:0/4154764431 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f14c415e0a0 0x7f14c40cd6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:04.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.663+0000 7f14ca985640 1 -- 172.21.15.67:0/4154764431 >> v2:172.21.15.72:3300/0 conn(0x7f14c40cdc30 msgr2=0x7f14c406b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:04.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.663+0000 7f14ca985640 1 --2- 172.21.15.67:0/4154764431 >> v2:172.21.15.72:3300/0 conn(0x7f14c40cdc30 0x7f14c406b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:04.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.663+0000 7f14ca985640 1 -- 172.21.15.67:0/4154764431 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14c40d0360 con 0x7f14c406b580 2024-03-24T00:47:04.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.664+0000 7f14ca985640 1 --2- 172.21.15.67:0/4154764431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14c406b580 0x7f14c40d0a30 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f14c40d1b30 tx=0x7f14c002bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=321fdab34a1fc91a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:04.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.664+0000 7f14c9182640 1 -- 172.21.15.67:0/4154764431 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14c002c950 con 0x7f14c406b580 2024-03-24T00:47:04.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.664+0000 7f14c9182640 1 -- 172.21.15.67:0/4154764431 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14c002caf0 con 0x7f14c406b580 2024-03-24T00:47:04.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.664+0000 7f14c9182640 1 -- 172.21.15.67:0/4154764431 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14c00343f0 con 0x7f14c406b580 2024-03-24T00:47:04.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.665+0000 7f14cc3e8640 1 -- 172.21.15.67:0/4154764431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14c406b580 msgr2=0x7f14c40d0a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:04.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.665+0000 7f14cc3e8640 1 --2- 172.21.15.67:0/4154764431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14c406b580 0x7f14c40d0a30 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f14c40d1b30 tx=0x7f14c002bdb0 comp rx=0 tx=0).stop 2024-03-24T00:47:04.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.665+0000 7f14cc3e8640 1 -- 172.21.15.67:0/4154764431 shutdown_connections 2024-03-24T00:47:04.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.665+0000 7f14cc3e8640 1 --2- 172.21.15.67:0/4154764431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14c406b580 0x7f14c40d0a30 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:04.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.665+0000 7f14cc3e8640 1 --2- 172.21.15.67:0/4154764431 >> v2:172.21.15.72:3300/0 conn(0x7f14c40cdc30 0x7f14c406b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:04.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.665+0000 7f14cc3e8640 1 --2- 172.21.15.67:0/4154764431 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f14c415e0a0 0x7f14c40cd6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:04.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.665+0000 7f14cc3e8640 1 -- 172.21.15.67:0/4154764431 >> 172.21.15.67:0/4154764431 conn(0x7f14c4099a30 msgr2=0x7f14c4099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:04.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.665+0000 7f14cc3e8640 1 -- 172.21.15.67:0/4154764431 shutdown_connections 2024-03-24T00:47:04.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.665+0000 7f14cc3e8640 1 -- 172.21.15.67:0/4154764431 wait complete. 2024-03-24T00:47:04.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.666+0000 7f14cc3e8640 1 Processor -- start 2024-03-24T00:47:04.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.666+0000 7f14cc3e8640 1 -- start start 2024-03-24T00:47:04.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.667+0000 7f14cc3e8640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f14c406b580 0x7f14c4155060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:04.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.667+0000 7f14cc3e8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14c40cdc30 0x7f14c40c6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:04.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.667+0000 7f14cc3e8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14c415e0a0 0x7f14c40c6a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:04.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.667+0000 7f14cc3e8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c40ce4d0 con 0x7f14c40cdc30 2024-03-24T00:47:04.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.667+0000 7f14cc3e8640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c40ce350 con 0x7f14c415e0a0 2024-03-24T00:47:04.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.667+0000 7f14cc3e8640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14c40cfca0 con 0x7f14c406b580 2024-03-24T00:47:04.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.667+0000 7f14c9983640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14c40cdc30 0x7f14c40c6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:04.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.667+0000 7f14c9983640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14c40cdc30 0x7f14c40c6510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39280/0 (socket says 172.21.15.67:39280) 2024-03-24T00:47:04.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.667+0000 7f14c9983640 1 -- 172.21.15.67:0/1589299530 learned_addr learned my addr 172.21.15.67:0/1589299530 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:04.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.667+0000 7f14ca184640 1 --2- 172.21.15.67:0/1589299530 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f14c406b580 0x7f14c4155060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:04.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.667+0000 7f14ca985640 1 --2- 172.21.15.67:0/1589299530 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14c415e0a0 0x7f14c40c6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:04.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.667+0000 7f14c9983640 1 -- 172.21.15.67:0/1589299530 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f14c406b580 msgr2=0x7f14c4155060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:04.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.667+0000 7f14c9983640 1 --2- 172.21.15.67:0/1589299530 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f14c406b580 0x7f14c4155060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:04.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.667+0000 7f14c9983640 1 -- 172.21.15.67:0/1589299530 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14c415e0a0 msgr2=0x7f14c40c6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:04.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.667+0000 7f14c9983640 1 --2- 172.21.15.67:0/1589299530 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14c415e0a0 0x7f14c40c6a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:04.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.667+0000 7f14c9983640 1 -- 172.21.15.67:0/1589299530 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14c40c7060 con 0x7f14c40cdc30 2024-03-24T00:47:04.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.667+0000 7f14ca184640 1 --2- 172.21.15.67:0/1589299530 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f14c406b580 0x7f14c4155060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:47:04.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.667+0000 7f14c9983640 1 --2- 172.21.15.67:0/1589299530 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14c40cdc30 0x7f14c40c6510 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f14b400e840 tx=0x7f14b400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:04.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.667+0000 7f14baffd640 1 -- 172.21.15.67:0/1589299530 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b4017070 con 0x7f14c40cdc30 2024-03-24T00:47:04.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.667+0000 7f14baffd640 1 -- 172.21.15.67:0/1589299530 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14b400c940 con 0x7f14c40cdc30 2024-03-24T00:47:04.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.667+0000 7f14cc3e8640 1 -- 172.21.15.67:0/1589299530 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14c40c7340 con 0x7f14c40cdc30 2024-03-24T00:47:04.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.668+0000 7f14baffd640 1 -- 172.21.15.67:0/1589299530 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b400cd10 con 0x7f14c40cdc30 2024-03-24T00:47:04.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.668+0000 7f14cc3e8640 1 -- 172.21.15.67:0/1589299530 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14c415f750 con 0x7f14c40cdc30 2024-03-24T00:47:04.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.669+0000 7f14cc3e8640 1 -- 172.21.15.67:0/1589299530 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14c404f400 con 0x7f14c40cdc30 2024-03-24T00:47:04.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.670+0000 7f14baffd640 1 -- 172.21.15.67:0/1589299530 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f14b401a070 con 0x7f14c40cdc30 2024-03-24T00:47:04.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.670+0000 7f14baffd640 1 --2- 172.21.15.67:0/1589299530 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f14a806bc00 0x7f14a806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:04.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.670+0000 7f14baffd640 1 -- 172.21.15.67:0/1589299530 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f14b408af50 con 0x7f14c40cdc30 2024-03-24T00:47:04.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.670+0000 7f14ca184640 1 --2- 172.21.15.67:0/1589299530 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f14a806bc00 0x7f14a806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:04.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.671+0000 7f14ca184640 1 --2- 172.21.15.67:0/1589299530 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f14a806bc00 0x7f14a806e0c0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f14bc009b10 tx=0x7f14bc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:04.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:04.675+0000 7f14baffd640 1 -- 172.21.15.67:0/1589299530 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f14b4059790 con 0x7f14c40cdc30 2024-03-24T00:47:05.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.070+0000 7f14cc3e8640 1 -- 172.21.15.67:0/1589299530 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f14c40cd260 con 0x7f14c40cdc30 2024-03-24T00:47:05.072 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-24T00:47:05.071+0000 7fbfcfa11640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T00:47:05.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.072+0000 7f14baffd640 1 -- 172.21.15.67:0/1589299530 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v67) v1 ==== 161+0+0 (secure 0 0 0) 0x7f14b4059090 con 0x7f14c40cdc30 2024-03-24T00:47:05.073 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-24T00:47:05.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.076+0000 7f14cc3e8640 1 -- 172.21.15.67:0/1589299530 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f14a806bc00 msgr2=0x7f14a806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:05.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.076+0000 7f14cc3e8640 1 --2- 172.21.15.67:0/1589299530 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f14a806bc00 0x7f14a806e0c0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f14bc009b10 tx=0x7f14bc008040 comp rx=0 tx=0).stop 2024-03-24T00:47:05.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.076+0000 7f14cc3e8640 1 -- 172.21.15.67:0/1589299530 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14c40cdc30 msgr2=0x7f14c40c6510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:05.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.076+0000 7f14cc3e8640 1 --2- 172.21.15.67:0/1589299530 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14c40cdc30 0x7f14c40c6510 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f14b400e840 tx=0x7f14b400ed00 comp rx=0 tx=0).stop 2024-03-24T00:47:05.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.076+0000 7f14cc3e8640 1 -- 172.21.15.67:0/1589299530 shutdown_connections 2024-03-24T00:47:05.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.076+0000 7f14cc3e8640 1 --2- 172.21.15.67:0/1589299530 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f14a806bc00 0x7f14a806e0c0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:05.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.076+0000 7f14cc3e8640 1 --2- 172.21.15.67:0/1589299530 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14c415e0a0 0x7f14c40c6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:05.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.076+0000 7f14cc3e8640 1 --2- 172.21.15.67:0/1589299530 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14c40cdc30 0x7f14c40c6510 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:05.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.076+0000 7f14cc3e8640 1 --2- 172.21.15.67:0/1589299530 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f14c406b580 0x7f14c4155060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:05.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.076+0000 7f14cc3e8640 1 -- 172.21.15.67:0/1589299530 >> 172.21.15.67:0/1589299530 conn(0x7f14c4099a30 msgr2=0x7f14c40b2f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:05.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.076+0000 7f14cc3e8640 1 -- 172.21.15.67:0/1589299530 shutdown_connections 2024-03-24T00:47:05.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.076+0000 7f14cc3e8640 1 -- 172.21.15.67:0/1589299530 wait complete. 2024-03-24T00:47:05.080 INFO:tasks.ceph.mds.c.smithi067.stderr: -2328> 2024-03-24T00:46:26.851+0000 7f97dcee7640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T00:47:05.080 INFO:tasks.ceph.mds.c.smithi067.stderr: -1952> 2024-03-24T00:46:27.623+0000 7f97d8edf640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [...2,head] /dir/file auth v1 ap=1 DIRTYPARENT s=0 nl=2 n() (iversion lock) | dirtyparent=1 authpin=1 scrubqueue=0 0x55e4b9e62100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(14)0x10000000001:[<0x10000000000/file v8>,<0x1/dir v15>]//[]","memoryvalue":"(14)0x10000000001:[<0x10000000000/file v1>,<0x1/dir v22>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-24T00:47:05.088 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-24T00:47:05.105 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-03-24T00:47:05.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.192+0000 7fe8aa9cd640 1 Processor -- start 2024-03-24T00:47:05.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.192+0000 7fe8aa9cd640 1 -- start start 2024-03-24T00:47:05.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.193+0000 7fe8aa9cd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe8a4099d00 0x7fe8a409a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:05.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.193+0000 7fe8aa9cd640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe8a40b49b0 0x7fe8a40d0e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:05.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.193+0000 7fe8aa9cd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe8a40d1360 0x7fe8a415d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:05.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.193+0000 7fe8aa9cd640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8a40cc1b0 con 0x7fe8a40d1360 2024-03-24T00:47:05.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.193+0000 7fe8aa9cd640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8a409a620 con 0x7fe8a4099d00 2024-03-24T00:47:05.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.193+0000 7fe8aa9cd640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a40cc400 con 0x7fe8a40b49b0 2024-03-24T00:47:05.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.193+0000 7fe8a8f6a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe8a40d1360 0x7fe8a415d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:05.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.193+0000 7fe8a8f6a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe8a40d1360 0x7fe8a415d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39316/0 (socket says 172.21.15.67:39316) 2024-03-24T00:47:05.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.193+0000 7fe8a8f6a640 1 -- 172.21.15.67:0/3602821831 learned_addr learned my addr 172.21.15.67:0/3602821831 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:05.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.194+0000 7fe8a37fe640 1 --2- 172.21.15.67:0/3602821831 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe8a40b49b0 0x7fe8a40d0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:05.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.194+0000 7fe8a3fff640 1 --2- 172.21.15.67:0/3602821831 >> v2:172.21.15.72:3300/0 conn(0x7fe8a4099d00 0x7fe8a409a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:05.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.194+0000 7fe8a8f6a640 1 -- 172.21.15.67:0/3602821831 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe8a40b49b0 msgr2=0x7fe8a40d0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:05.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.194+0000 7fe8a8f6a640 1 --2- 172.21.15.67:0/3602821831 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe8a40b49b0 0x7fe8a40d0e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:05.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.194+0000 7fe8a8f6a640 1 -- 172.21.15.67:0/3602821831 >> v2:172.21.15.72:3300/0 conn(0x7fe8a4099d00 msgr2=0x7fe8a409a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:05.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.194+0000 7fe8a8f6a640 1 --2- 172.21.15.67:0/3602821831 >> v2:172.21.15.72:3300/0 conn(0x7fe8a4099d00 0x7fe8a409a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:05.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.194+0000 7fe8a8f6a640 1 -- 172.21.15.67:0/3602821831 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8a40cc6e0 con 0x7fe8a40d1360 2024-03-24T00:47:05.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.194+0000 7fe8a37fe640 1 --2- 172.21.15.67:0/3602821831 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe8a40b49b0 0x7fe8a40d0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:47:05.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.194+0000 7fe8a3fff640 1 --2- 172.21.15.67:0/3602821831 >> v2:172.21.15.72:3300/0 conn(0x7fe8a4099d00 0x7fe8a409a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:47:05.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.194+0000 7fe8a8f6a640 1 --2- 172.21.15.67:0/3602821831 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8a40d1360 0x7fe8a415d7d0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7fe8a408b260 tx=0x7fe89c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a34866aa3f795fd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:05.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.195+0000 7fe8a2ffd640 1 -- 172.21.15.67:0/3602821831 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe89c02ca70 con 0x7fe8a40d1360 2024-03-24T00:47:05.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.195+0000 7fe8a2ffd640 1 -- 172.21.15.67:0/3602821831 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe89c02cc10 con 0x7fe8a40d1360 2024-03-24T00:47:05.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.195+0000 7fe8a2ffd640 1 -- 172.21.15.67:0/3602821831 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe89c0343f0 con 0x7fe8a40d1360 2024-03-24T00:47:05.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.195+0000 7fe8aa9cd640 1 -- 172.21.15.67:0/3602821831 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8a40d1360 msgr2=0x7fe8a415d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:05.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.195+0000 7fe8aa9cd640 1 --2- 172.21.15.67:0/3602821831 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8a40d1360 0x7fe8a415d7d0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7fe8a408b260 tx=0x7fe89c02bed0 comp rx=0 tx=0).stop 2024-03-24T00:47:05.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.195+0000 7fe8aa9cd640 1 -- 172.21.15.67:0/3602821831 shutdown_connections 2024-03-24T00:47:05.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.195+0000 7fe8aa9cd640 1 --2- 172.21.15.67:0/3602821831 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8a40d1360 0x7fe8a415d7d0 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:05.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.195+0000 7fe8aa9cd640 1 --2- 172.21.15.67:0/3602821831 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe8a40b49b0 0x7fe8a40d0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:05.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.195+0000 7fe8aa9cd640 1 --2- 172.21.15.67:0/3602821831 >> v2:172.21.15.72:3300/0 conn(0x7fe8a4099d00 0x7fe8a409a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:05.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.195+0000 7fe8aa9cd640 1 -- 172.21.15.67:0/3602821831 >> 172.21.15.67:0/3602821831 conn(0x7fe8a40b3650 msgr2=0x7fe8a40b3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:05.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.195+0000 7fe8aa9cd640 1 -- 172.21.15.67:0/3602821831 shutdown_connections 2024-03-24T00:47:05.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.195+0000 7fe8aa9cd640 1 -- 172.21.15.67:0/3602821831 wait complete. 2024-03-24T00:47:05.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.196+0000 7fe8aa9cd640 1 Processor -- start 2024-03-24T00:47:05.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.197+0000 7fe8aa9cd640 1 -- start start 2024-03-24T00:47:05.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.197+0000 7fe8aa9cd640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe8a4099d00 0x7fe8a41f4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:05.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.197+0000 7fe8aa9cd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8a40b49b0 0x7fe8a41571e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:05.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.197+0000 7fe8aa9cd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8a40d1360 0x7fe8a4159df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:05.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.197+0000 7fe8aa9cd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a40d01c0 con 0x7fe8a40d1360 2024-03-24T00:47:05.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.197+0000 7fe8aa9cd640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a40ccca0 con 0x7fe8a40b49b0 2024-03-24T00:47:05.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.197+0000 7fe8aa9cd640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a40b32a0 con 0x7fe8a4099d00 2024-03-24T00:47:05.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.197+0000 7fe8a8f6a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8a40d1360 0x7fe8a4159df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:05.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.197+0000 7fe8a8f6a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8a40d1360 0x7fe8a4159df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39324/0 (socket says 172.21.15.67:39324) 2024-03-24T00:47:05.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.197+0000 7fe8a8f6a640 1 -- 172.21.15.67:0/2961835172 learned_addr learned my addr 172.21.15.67:0/2961835172 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:05.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.198+0000 7fe8a3fff640 1 --2- 172.21.15.67:0/2961835172 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe8a4099d00 0x7fe8a41f4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:05.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.198+0000 7fe8a37fe640 1 --2- 172.21.15.67:0/2961835172 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8a40b49b0 0x7fe8a41571e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:05.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.198+0000 7fe8a8f6a640 1 -- 172.21.15.67:0/2961835172 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe8a4099d00 msgr2=0x7fe8a41f4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:05.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.198+0000 7fe8a8f6a640 1 --2- 172.21.15.67:0/2961835172 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe8a4099d00 0x7fe8a41f4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:05.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.198+0000 7fe8a8f6a640 1 -- 172.21.15.67:0/2961835172 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8a40b49b0 msgr2=0x7fe8a41571e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:05.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.198+0000 7fe8a8f6a640 1 --2- 172.21.15.67:0/2961835172 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8a40b49b0 0x7fe8a41571e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:05.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.198+0000 7fe8a8f6a640 1 -- 172.21.15.67:0/2961835172 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8a4157720 con 0x7fe8a40d1360 2024-03-24T00:47:05.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.198+0000 7fe8a8f6a640 1 --2- 172.21.15.67:0/2961835172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8a40d1360 0x7fe8a4159df0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7fe89c009f40 tx=0x7fe89c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:05.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.198+0000 7fe8a0ff9640 1 -- 172.21.15.67:0/2961835172 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe89c043020 con 0x7fe8a40d1360 2024-03-24T00:47:05.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.198+0000 7fe8a0ff9640 1 -- 172.21.15.67:0/2961835172 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe89c03e450 con 0x7fe8a40d1360 2024-03-24T00:47:05.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.198+0000 7fe8aa9cd640 1 -- 172.21.15.67:0/2961835172 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8a4158410 con 0x7fe8a40d1360 2024-03-24T00:47:05.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.198+0000 7fe8a0ff9640 1 -- 172.21.15.67:0/2961835172 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe89c03e820 con 0x7fe8a40d1360 2024-03-24T00:47:05.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.198+0000 7fe8aa9cd640 1 -- 172.21.15.67:0/2961835172 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8a41589d0 con 0x7fe8a40d1360 2024-03-24T00:47:05.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.199+0000 7fe8aa9cd640 1 -- 172.21.15.67:0/2961835172 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8a404f400 con 0x7fe8a40d1360 2024-03-24T00:47:05.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.199+0000 7fe8a0ff9640 1 -- 172.21.15.67:0/2961835172 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe89c046020 con 0x7fe8a40d1360 2024-03-24T00:47:05.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.200+0000 7fe8a0ff9640 1 --2- 172.21.15.67:0/2961835172 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe89006bbd0 0x7fe89006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:05.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.200+0000 7fe8a0ff9640 1 -- 172.21.15.67:0/2961835172 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6475+0+0 (secure 0 0 0) 0x7fe89c0ad5d0 con 0x7fe8a40d1360 2024-03-24T00:47:05.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.200+0000 7fe8a3fff640 1 --2- 172.21.15.67:0/2961835172 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe89006bbd0 0x7fe89006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:05.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.201+0000 7fe8a3fff640 1 --2- 172.21.15.67:0/2961835172 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe89006bbd0 0x7fe89006e090 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fe8a406ceb0 tx=0x7fe894008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:05.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.204+0000 7fe8a0ff9640 1 -- 172.21.15.67:0/2961835172 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe89c04d030 con 0x7fe8a40d1360 2024-03-24T00:47:05.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:05.431+0000 7fe8aa9cd640 1 -- 172.21.15.67:0/2961835172 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fe8a40cbe90 con 0x7fe8a40d1360 2024-03-24T00:47:06.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.081+0000 7fe8a0ff9640 1 -- 172.21.15.67:0/2961835172 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v68) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe89c07b770 con 0x7fe8a40d1360 2024-03-24T00:47:06.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.084+0000 7fe8aa9cd640 1 -- 172.21.15.67:0/2961835172 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe89006bbd0 msgr2=0x7fe89006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:06.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.084+0000 7fe8aa9cd640 1 --2- 172.21.15.67:0/2961835172 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe89006bbd0 0x7fe89006e090 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fe8a406ceb0 tx=0x7fe894008040 comp rx=0 tx=0).stop 2024-03-24T00:47:06.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.084+0000 7fe8aa9cd640 1 -- 172.21.15.67:0/2961835172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8a40d1360 msgr2=0x7fe8a4159df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:06.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.084+0000 7fe8aa9cd640 1 --2- 172.21.15.67:0/2961835172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8a40d1360 0x7fe8a4159df0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7fe89c009f40 tx=0x7fe89c004a40 comp rx=0 tx=0).stop 2024-03-24T00:47:06.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.084+0000 7fe8aa9cd640 1 -- 172.21.15.67:0/2961835172 shutdown_connections 2024-03-24T00:47:06.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.084+0000 7fe8aa9cd640 1 --2- 172.21.15.67:0/2961835172 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe89006bbd0 0x7fe89006e090 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:06.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.084+0000 7fe8aa9cd640 1 --2- 172.21.15.67:0/2961835172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8a40d1360 0x7fe8a4159df0 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:06.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.084+0000 7fe8aa9cd640 1 --2- 172.21.15.67:0/2961835172 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8a40b49b0 0x7fe8a41571e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:06.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.084+0000 7fe8aa9cd640 1 --2- 172.21.15.67:0/2961835172 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe8a4099d00 0x7fe8a41f4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:06.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.084+0000 7fe8aa9cd640 1 -- 172.21.15.67:0/2961835172 >> 172.21.15.67:0/2961835172 conn(0x7fe8a40b3650 msgr2=0x7fe8a40ce3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:06.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.084+0000 7fe8aa9cd640 1 -- 172.21.15.67:0/2961835172 shutdown_connections 2024-03-24T00:47:06.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.085+0000 7fe8aa9cd640 1 -- 172.21.15.67:0/2961835172 wait complete. 2024-03-24T00:47:06.099 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-24T00:47:06.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.253+0000 7f2ac34bf640 1 Processor -- start 2024-03-24T00:47:06.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.253+0000 7f2ac34bf640 1 -- start start 2024-03-24T00:47:06.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.253+0000 7f2ac34bf640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2abc15bc40 0x7f2abc09ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:06.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.254+0000 7f2ac34bf640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2abc0cd130 0x7f2abc0cd590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:06.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.254+0000 7f2ac34bf640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2abc0d0130 0x7f2abc0d05c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:06.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.254+0000 7f2ac34bf640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2abc0d0b00 con 0x7f2abc15bc40 2024-03-24T00:47:06.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.254+0000 7f2ac34bf640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2abc099d70 con 0x7f2abc0cd130 2024-03-24T00:47:06.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.254+0000 7f2ac34bf640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2abc09a050 con 0x7f2abc0d0130 2024-03-24T00:47:06.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.254+0000 7f2ac125b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2abc15bc40 0x7f2abc09ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:06.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.254+0000 7f2ac125b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2abc15bc40 0x7f2abc09ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39334/0 (socket says 172.21.15.67:39334) 2024-03-24T00:47:06.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.254+0000 7f2ac125b640 1 -- 172.21.15.67:0/1168966560 learned_addr learned my addr 172.21.15.67:0/1168966560 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:06.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.254+0000 7f2ac0a5a640 1 --2- 172.21.15.67:0/1168966560 >> v2:172.21.15.72:3300/0 conn(0x7f2abc0cd130 0x7f2abc0cd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:06.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.254+0000 7f2ac1a5c640 1 --2- 172.21.15.67:0/1168966560 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2abc0d0130 0x7f2abc0d05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:06.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.254+0000 7f2ac125b640 1 -- 172.21.15.67:0/1168966560 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2abc0d0130 msgr2=0x7f2abc0d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:06.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.254+0000 7f2ac125b640 1 --2- 172.21.15.67:0/1168966560 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2abc0d0130 0x7f2abc0d05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:06.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.254+0000 7f2ac125b640 1 -- 172.21.15.67:0/1168966560 >> v2:172.21.15.72:3300/0 conn(0x7f2abc0cd130 msgr2=0x7f2abc0cd590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:06.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.254+0000 7f2ac125b640 1 --2- 172.21.15.67:0/1168966560 >> v2:172.21.15.72:3300/0 conn(0x7f2abc0cd130 0x7f2abc0cd590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:06.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.254+0000 7f2ac125b640 1 -- 172.21.15.67:0/1168966560 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2abc09a330 con 0x7f2abc15bc40 2024-03-24T00:47:06.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.255+0000 7f2ac125b640 1 --2- 172.21.15.67:0/1168966560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2abc15bc40 0x7f2abc09ae10 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f2abc073680 tx=0x7f2ab802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83d6bb8cfe0db089 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:06.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.255+0000 7f2ab3fff640 1 -- 172.21.15.67:0/1168966560 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab8035020 con 0x7f2abc15bc40 2024-03-24T00:47:06.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.255+0000 7f2ab3fff640 1 -- 172.21.15.67:0/1168966560 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ab8031870 con 0x7f2abc15bc40 2024-03-24T00:47:06.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.255+0000 7f2ab3fff640 1 -- 172.21.15.67:0/1168966560 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab8031c40 con 0x7f2abc15bc40 2024-03-24T00:47:06.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.256+0000 7f2ac34bf640 1 -- 172.21.15.67:0/1168966560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2abc15bc40 msgr2=0x7f2abc09ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:06.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.256+0000 7f2ac34bf640 1 --2- 172.21.15.67:0/1168966560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2abc15bc40 0x7f2abc09ae10 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f2abc073680 tx=0x7f2ab802b910 comp rx=0 tx=0).stop 2024-03-24T00:47:06.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.256+0000 7f2ac34bf640 1 -- 172.21.15.67:0/1168966560 shutdown_connections 2024-03-24T00:47:06.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.256+0000 7f2ac34bf640 1 --2- 172.21.15.67:0/1168966560 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2abc0d0130 0x7f2abc0d05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:06.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.256+0000 7f2ac34bf640 1 --2- 172.21.15.67:0/1168966560 >> v2:172.21.15.72:3300/0 conn(0x7f2abc0cd130 0x7f2abc0cd590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:06.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.256+0000 7f2ac34bf640 1 --2- 172.21.15.67:0/1168966560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2abc15bc40 0x7f2abc09ae10 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:06.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.256+0000 7f2ac34bf640 1 -- 172.21.15.67:0/1168966560 >> 172.21.15.67:0/1168966560 conn(0x7f2abc0b3640 msgr2=0x7f2abc0b3a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:06.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.256+0000 7f2ac34bf640 1 -- 172.21.15.67:0/1168966560 shutdown_connections 2024-03-24T00:47:06.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.256+0000 7f2ac34bf640 1 -- 172.21.15.67:0/1168966560 wait complete. 2024-03-24T00:47:06.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.257+0000 7f2ac34bf640 1 Processor -- start 2024-03-24T00:47:06.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.257+0000 7f2ac34bf640 1 -- start start 2024-03-24T00:47:06.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.258+0000 7f2ac34bf640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2abc0cd130 0x7f2abc0c9c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:06.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.258+0000 7f2ac34bf640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2abc0d0130 0x7f2abc14fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:06.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.258+0000 7f2ac34bf640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2abc15bc40 0x7f2abc1e5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:06.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.258+0000 7f2ac34bf640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2abc0cf950 con 0x7f2abc0cd130 2024-03-24T00:47:06.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.258+0000 7f2ac34bf640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2abc09a8f0 con 0x7f2abc0d0130 2024-03-24T00:47:06.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.258+0000 7f2ac34bf640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2abc15c020 con 0x7f2abc15bc40 2024-03-24T00:47:06.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.258+0000 7f2ac125b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2abc0cd130 0x7f2abc0c9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:06.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.258+0000 7f2ac125b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2abc0cd130 0x7f2abc0c9c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39344/0 (socket says 172.21.15.67:39344) 2024-03-24T00:47:06.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.258+0000 7f2ac125b640 1 -- 172.21.15.67:0/1391497293 learned_addr learned my addr 172.21.15.67:0/1391497293 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:06.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.258+0000 7f2ac0a5a640 1 --2- 172.21.15.67:0/1391497293 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2abc0d0130 0x7f2abc14fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:06.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.258+0000 7f2ac125b640 1 -- 172.21.15.67:0/1391497293 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2abc15bc40 msgr2=0x7f2abc1e5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:06.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.258+0000 7f2ac125b640 1 --2- 172.21.15.67:0/1391497293 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2abc15bc40 0x7f2abc1e5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:06.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.258+0000 7f2ac125b640 1 -- 172.21.15.67:0/1391497293 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2abc0d0130 msgr2=0x7f2abc14fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:06.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.258+0000 7f2ac125b640 1 --2- 172.21.15.67:0/1391497293 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2abc0d0130 0x7f2abc14fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:06.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.258+0000 7f2ac125b640 1 -- 172.21.15.67:0/1391497293 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2abc0ca1b0 con 0x7f2abc0cd130 2024-03-24T00:47:06.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.258+0000 7f2ac0a5a640 1 --2- 172.21.15.67:0/1391497293 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2abc0d0130 0x7f2abc14fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:47:06.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.258+0000 7f2ac125b640 1 --2- 172.21.15.67:0/1391497293 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2abc0cd130 0x7f2abc0c9c70 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f2ab802b950 tx=0x7f2ab80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:06.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.258+0000 7f2ab1ffb640 1 -- 172.21.15.67:0/1391497293 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab8040070 con 0x7f2abc0cd130 2024-03-24T00:47:06.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.258+0000 7f2ab1ffb640 1 -- 172.21.15.67:0/1391497293 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ab8030d00 con 0x7f2abc0cd130 2024-03-24T00:47:06.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.258+0000 7f2ac34bf640 1 -- 172.21.15.67:0/1391497293 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2abc0ca450 con 0x7f2abc0cd130 2024-03-24T00:47:06.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.259+0000 7f2ab1ffb640 1 -- 172.21.15.67:0/1391497293 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab8035040 con 0x7f2abc0cd130 2024-03-24T00:47:06.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.259+0000 7f2ac34bf640 1 -- 172.21.15.67:0/1391497293 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2abc0cacf0 con 0x7f2abc0cd130 2024-03-24T00:47:06.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.260+0000 7f2ac34bf640 1 -- 172.21.15.67:0/1391497293 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2abc04f400 con 0x7f2abc0cd130 2024-03-24T00:47:06.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.260+0000 7f2ab1ffb640 1 -- 172.21.15.67:0/1391497293 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2ab804b060 con 0x7f2abc0cd130 2024-03-24T00:47:06.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.260+0000 7f2ab1ffb640 1 --2- 172.21.15.67:0/1391497293 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2aa006bbd0 0x7f2aa006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:06.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.261+0000 7f2ab1ffb640 1 -- 172.21.15.67:0/1391497293 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6475+0+0 (secure 0 0 0) 0x7f2ab80acac0 con 0x7f2abc0cd130 2024-03-24T00:47:06.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.261+0000 7f2ac0a5a640 1 --2- 172.21.15.67:0/1391497293 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2aa006bbd0 0x7f2aa006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:06.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.261+0000 7f2ac0a5a640 1 --2- 172.21.15.67:0/1391497293 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2aa006bbd0 0x7f2aa006e090 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f2abc0cbd00 tx=0x7f2aac008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:06.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.265+0000 7f2ab1ffb640 1 -- 172.21.15.67:0/1391497293 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2ab807b2b0 con 0x7f2abc0cd130 2024-03-24T00:47:06.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:06.447+0000 7f2ac34bf640 1 -- 172.21.15.67:0/1391497293 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2abc0b4550 con 0x7f2abc0cd130 2024-03-24T00:47:07.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.085+0000 7f2ab1ffb640 1 -- 172.21.15.67:0/1391497293 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v95) v1 ==== 181+0+0 (secure 0 0 0) 0x7f2ab807abb0 con 0x7f2abc0cd130 2024-03-24T00:47:07.085 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' removed 2024-03-24T00:47:07.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.087+0000 7f2ac34bf640 1 -- 172.21.15.67:0/1391497293 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2aa006bbd0 msgr2=0x7f2aa006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:07.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.087+0000 7f2ac34bf640 1 --2- 172.21.15.67:0/1391497293 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2aa006bbd0 0x7f2aa006e090 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f2abc0cbd00 tx=0x7f2aac008040 comp rx=0 tx=0).stop 2024-03-24T00:47:07.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.087+0000 7f2ac34bf640 1 -- 172.21.15.67:0/1391497293 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2abc0cd130 msgr2=0x7f2abc0c9c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:07.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.087+0000 7f2ac34bf640 1 --2- 172.21.15.67:0/1391497293 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2abc0cd130 0x7f2abc0c9c70 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f2ab802b950 tx=0x7f2ab80046c0 comp rx=0 tx=0).stop 2024-03-24T00:47:07.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.087+0000 7f2ac34bf640 1 -- 172.21.15.67:0/1391497293 shutdown_connections 2024-03-24T00:47:07.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.087+0000 7f2ac34bf640 1 --2- 172.21.15.67:0/1391497293 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2aa006bbd0 0x7f2aa006e090 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:07.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.087+0000 7f2ac34bf640 1 --2- 172.21.15.67:0/1391497293 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2abc15bc40 0x7f2abc1e5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:07.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.087+0000 7f2ac34bf640 1 --2- 172.21.15.67:0/1391497293 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2abc0d0130 0x7f2abc14fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:07.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.087+0000 7f2ac34bf640 1 --2- 172.21.15.67:0/1391497293 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2abc0cd130 0x7f2abc0c9c70 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:07.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.087+0000 7f2ac34bf640 1 -- 172.21.15.67:0/1391497293 >> 172.21.15.67:0/1391497293 conn(0x7f2abc0b3640 msgr2=0x7f2abc0b6ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:07.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.087+0000 7f2ac34bf640 1 -- 172.21.15.67:0/1391497293 shutdown_connections 2024-03-24T00:47:07.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.087+0000 7f2ac34bf640 1 -- 172.21.15.67:0/1391497293 wait complete. 2024-03-24T00:47:07.101 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-24T00:47:07.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.210+0000 7fc01abab640 1 Processor -- start 2024-03-24T00:47:07.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.211+0000 7fc01abab640 1 -- start start 2024-03-24T00:47:07.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.211+0000 7fc01abab640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc01406cb60 0x7fc01409a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:07.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.211+0000 7fc01abab640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc01406cf40 0x7fc0140cfe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:07.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.211+0000 7fc01abab640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0140d06d0 con 0x7fc0140b2800 2024-03-24T00:47:07.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.211+0000 7fc01abab640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0140d09b0 con 0x7fc01406cf40 2024-03-24T00:47:07.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.211+0000 7fc01abab640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0140d0c90 con 0x7fc01406cb60 2024-03-24T00:47:07.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.211+0000 7fc013fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc01406cf40 0x7fc0140cfe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:07.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.211+0000 7fc019148640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc0140b2800 0x7fc0140b2c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46806/0 (socket says 172.21.15.67:46806) 2024-03-24T00:47:07.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.211+0000 7fc013fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc01406cf40 0x7fc0140cfe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39350/0 (socket says 172.21.15.67:39350) 2024-03-24T00:47:07.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.211+0000 7fc019148640 1 -- 172.21.15.67:0/2851541334 learned_addr learned my addr 172.21.15.67:0/2851541334 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:47:07.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.212+0000 7fc018947640 1 --2- 172.21.15.67:0/2851541334 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc01406cb60 0x7fc01409a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:07.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.212+0000 7fc013fff640 1 -- 172.21.15.67:0/2851541334 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc01406cb60 msgr2=0x7fc01409a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:07.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.212+0000 7fc013fff640 1 --2- 172.21.15.67:0/2851541334 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc01406cb60 0x7fc01409a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:07.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.212+0000 7fc013fff640 1 -- 172.21.15.67:0/2851541334 >> v1:172.21.15.67:6789/0 conn(0x7fc0140b2800 legacy=0x7fc0140b2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:07.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.212+0000 7fc013fff640 1 -- 172.21.15.67:0/2851541334 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0140d0f70 con 0x7fc01406cf40 2024-03-24T00:47:07.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.212+0000 7fc013fff640 1 --2- 172.21.15.67:0/2851541334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc01406cf40 0x7fc0140cfe10 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7fc0000097a0 tx=0x7fc00002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a17187d943580284 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:07.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.213+0000 7fc0137fe640 1 -- 172.21.15.67:0/2851541334 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0000315a0 con 0x7fc01406cf40 2024-03-24T00:47:07.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.213+0000 7fc0137fe640 1 -- 172.21.15.67:0/2851541334 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc000037070 con 0x7fc01406cf40 2024-03-24T00:47:07.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.213+0000 7fc0137fe640 1 -- 172.21.15.67:0/2851541334 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc000031960 con 0x7fc01406cf40 2024-03-24T00:47:07.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.213+0000 7fc01abab640 1 -- 172.21.15.67:0/2851541334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc01406cf40 msgr2=0x7fc0140cfe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:07.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.213+0000 7fc01abab640 1 --2- 172.21.15.67:0/2851541334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc01406cf40 0x7fc0140cfe10 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7fc0000097a0 tx=0x7fc00002b840 comp rx=0 tx=0).stop 2024-03-24T00:47:07.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.213+0000 7fc01abab640 1 -- 172.21.15.67:0/2851541334 shutdown_connections 2024-03-24T00:47:07.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.213+0000 7fc01abab640 1 --2- 172.21.15.67:0/2851541334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc01406cf40 0x7fc0140cfe10 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:07.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.213+0000 7fc01abab640 1 --2- 172.21.15.67:0/2851541334 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc01406cb60 0x7fc01409a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:07.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.213+0000 7fc01abab640 1 -- 172.21.15.67:0/2851541334 >> 172.21.15.67:0/2851541334 conn(0x7fc0140b36c0 msgr2=0x7fc0140b3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:07.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.213+0000 7fc01abab640 1 -- 172.21.15.67:0/2851541334 shutdown_connections 2024-03-24T00:47:07.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.213+0000 7fc01abab640 1 -- 172.21.15.67:0/2851541334 wait complete. 2024-03-24T00:47:07.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.214+0000 7fc01abab640 1 Processor -- start 2024-03-24T00:47:07.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.214+0000 7fc01abab640 1 -- start start 2024-03-24T00:47:07.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.215+0000 7fc01abab640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc01406cb60 0x7fc0140b1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:07.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.215+0000 7fc01abab640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc01406cf40 0x7fc01415b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:07.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.215+0000 7fc01abab640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc0140b2800 0x7fc014160680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:07.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.215+0000 7fc01abab640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0140cd230 con 0x7fc01406cb60 2024-03-24T00:47:07.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.215+0000 7fc01abab640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0140d1770 con 0x7fc0140b2800 2024-03-24T00:47:07.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.215+0000 7fc01abab640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc01406d3c0 con 0x7fc01406cf40 2024-03-24T00:47:07.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.215+0000 7fc018947640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc01406cb60 0x7fc0140b1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:07.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.215+0000 7fc018947640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc01406cb60 0x7fc0140b1fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39352/0 (socket says 172.21.15.67:39352) 2024-03-24T00:47:07.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.215+0000 7fc018947640 1 -- 172.21.15.67:0/349414683 learned_addr learned my addr 172.21.15.67:0/349414683 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:07.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.215+0000 7fc013fff640 1 --2- 172.21.15.67:0/349414683 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc01406cf40 0x7fc01415b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:07.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.215+0000 7fc019148640 1 --2- 172.21.15.67:0/349414683 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc0140b2800 0x7fc014160680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:07.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.216+0000 7fc018947640 1 -- 172.21.15.67:0/349414683 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc01406cf40 msgr2=0x7fc01415b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:07.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.216+0000 7fc018947640 1 --2- 172.21.15.67:0/349414683 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc01406cf40 0x7fc01415b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:07.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.216+0000 7fc018947640 1 -- 172.21.15.67:0/349414683 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc0140b2800 msgr2=0x7fc014160680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:07.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.216+0000 7fc018947640 1 --2- 172.21.15.67:0/349414683 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc0140b2800 0x7fc014160680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:07.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.216+0000 7fc018947640 1 -- 172.21.15.67:0/349414683 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0140b46e0 con 0x7fc01406cb60 2024-03-24T00:47:07.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.216+0000 7fc019148640 1 --2- 172.21.15.67:0/349414683 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc0140b2800 0x7fc014160680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:47:07.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.216+0000 7fc018947640 1 --2- 172.21.15.67:0/349414683 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc01406cb60 0x7fc0140b1fb0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7fc00800d5f0 tx=0x7fc00800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:07.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.216+0000 7fc0117fa640 1 -- 172.21.15.67:0/349414683 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0080125f0 con 0x7fc01406cb60 2024-03-24T00:47:07.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.216+0000 7fc0117fa640 1 -- 172.21.15.67:0/349414683 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc008019070 con 0x7fc01406cb60 2024-03-24T00:47:07.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.216+0000 7fc01abab640 1 -- 172.21.15.67:0/349414683 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0140b49c0 con 0x7fc01406cb60 2024-03-24T00:47:07.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.216+0000 7fc0117fa640 1 -- 172.21.15.67:0/349414683 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0080129d0 con 0x7fc01406cb60 2024-03-24T00:47:07.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.216+0000 7fc01abab640 1 -- 172.21.15.67:0/349414683 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0140b4f80 con 0x7fc01406cb60 2024-03-24T00:47:07.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.217+0000 7fc01abab640 1 -- 172.21.15.67:0/349414683 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc01404f400 con 0x7fc01406cb60 2024-03-24T00:47:07.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.217+0000 7fc0117fa640 1 -- 172.21.15.67:0/349414683 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc008012b70 con 0x7fc01406cb60 2024-03-24T00:47:07.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.218+0000 7fc0117fa640 1 --2- 172.21.15.67:0/349414683 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbff806bbd0 0x7fbff806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:07.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.218+0000 7fc0117fa640 1 -- 172.21.15.67:0/349414683 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6037+0+0 (secure 0 0 0) 0x7fc00808af30 con 0x7fc01406cb60 2024-03-24T00:47:07.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.218+0000 7fc013fff640 1 --2- 172.21.15.67:0/349414683 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbff806bbd0 0x7fbff806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:07.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.218+0000 7fc013fff640 1 --2- 172.21.15.67:0/349414683 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbff806bbd0 0x7fbff806e090 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fc000036000 tx=0x7fc00003d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:07.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.222+0000 7fc0117fa640 1 -- 172.21.15.67:0/349414683 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc008059960 con 0x7fc01406cb60 2024-03-24T00:47:07.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:07.404+0000 7fc01abab640 1 -- 172.21.15.67:0/349414683 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc0140cd830 con 0x7fc01406cb60 2024-03-24T00:47:08.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.091+0000 7fc0117fa640 1 -- 172.21.15.67:0/349414683 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v96) v1 ==== 169+0+0 (secure 0 0 0) 0x7fc008059260 con 0x7fc01406cb60 2024-03-24T00:47:08.092 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' removed 2024-03-24T00:47:08.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.093+0000 7fc01abab640 1 -- 172.21.15.67:0/349414683 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbff806bbd0 msgr2=0x7fbff806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:08.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.094+0000 7fc01abab640 1 --2- 172.21.15.67:0/349414683 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbff806bbd0 0x7fbff806e090 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fc000036000 tx=0x7fc00003d040 comp rx=0 tx=0).stop 2024-03-24T00:47:08.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.094+0000 7fc01abab640 1 -- 172.21.15.67:0/349414683 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc01406cb60 msgr2=0x7fc0140b1fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:08.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.094+0000 7fc01abab640 1 --2- 172.21.15.67:0/349414683 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc01406cb60 0x7fc0140b1fb0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7fc00800d5f0 tx=0x7fc00800dab0 comp rx=0 tx=0).stop 2024-03-24T00:47:08.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.094+0000 7fc01abab640 1 -- 172.21.15.67:0/349414683 shutdown_connections 2024-03-24T00:47:08.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.094+0000 7fc01abab640 1 --2- 172.21.15.67:0/349414683 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbff806bbd0 0x7fbff806e090 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:08.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.094+0000 7fc01abab640 1 --2- 172.21.15.67:0/349414683 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc0140b2800 0x7fc014160680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:08.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.094+0000 7fc01abab640 1 --2- 172.21.15.67:0/349414683 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc01406cf40 0x7fc01415b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:08.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.094+0000 7fc01abab640 1 --2- 172.21.15.67:0/349414683 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc01406cb60 0x7fc0140b1fb0 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:08.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.094+0000 7fc01abab640 1 -- 172.21.15.67:0/349414683 >> 172.21.15.67:0/349414683 conn(0x7fc0140b36c0 msgr2=0x7fc014099d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:08.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.094+0000 7fc01abab640 1 -- 172.21.15.67:0/349414683 shutdown_connections 2024-03-24T00:47:08.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.094+0000 7fc01abab640 1 -- 172.21.15.67:0/349414683 wait complete. 2024-03-24T00:47:08.110 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-24T00:47:08.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.215+0000 7fde0495b640 1 Processor -- start 2024-03-24T00:47:08.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.215+0000 7fde0495b640 1 -- start start 2024-03-24T00:47:08.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.215+0000 7fde0495b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fde000d0d50 0x7fde000ce610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:08.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.216+0000 7fde0495b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fde000d22b0 0x7fde000d2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:08.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.216+0000 7fde0495b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde000cf570 con 0x7fde0015e130 2024-03-24T00:47:08.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.216+0000 7fde0495b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde000ceb50 con 0x7fde000d22b0 2024-03-24T00:47:08.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.216+0000 7fde0495b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde000cecf0 con 0x7fde000d0d50 2024-03-24T00:47:08.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.216+0000 7fddfdd9b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fde000d0d50 0x7fde000ce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:08.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.216+0000 7fddfed9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fde000d22b0 0x7fde000d2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:08.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.216+0000 7fddfdd9b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fde000d0d50 0x7fde000ce610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.67:60188/0 (socket says 172.21.15.67:60188) 2024-03-24T00:47:08.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.216+0000 7fddfed9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fde000d22b0 0x7fde000d2760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:37738/0 (socket says 172.21.15.67:37738) 2024-03-24T00:47:08.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.216+0000 7fddfdd9b640 1 -- 172.21.15.67:0/1430311663 learned_addr learned my addr 172.21.15.67:0/1430311663 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:08.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.216+0000 7fddfe59c640 1 --1- 172.21.15.67:0/1430311663 >> v1:172.21.15.72:6789/0 conn(0x7fde0015e130 0x7fde000d0640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:58056/0 (socket says 172.21.15.67:58056) 2024-03-24T00:47:08.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.216+0000 7fddfed9d640 1 -- 172.21.15.67:0/1430311663 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fde000d0d50 msgr2=0x7fde000ce610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:08.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.216+0000 7fddfed9d640 1 --2- 172.21.15.67:0/1430311663 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fde000d0d50 0x7fde000ce610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:08.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.216+0000 7fddfed9d640 1 -- 172.21.15.67:0/1430311663 >> v1:172.21.15.72:6789/0 conn(0x7fde0015e130 legacy=0x7fde000d0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:08.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.216+0000 7fddfed9d640 1 -- 172.21.15.67:0/1430311663 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde00162a60 con 0x7fde000d22b0 2024-03-24T00:47:08.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.216+0000 7fddfd59a640 1 -- 172.21.15.67:0/1430311663 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2446702715 0 0) 0x7fde000cf570 con 0x7fde0015e130 2024-03-24T00:47:08.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.216+0000 7fddfdd9b640 1 --2- 172.21.15.67:0/1430311663 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fde000d0d50 0x7fde000ce610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:47:08.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.217+0000 7fddfed9d640 1 --2- 172.21.15.67:0/1430311663 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde000d22b0 0x7fde000d2760 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fddf8009d90 tx=0x7fddf802bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8da3580f8783b46 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:08.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.217+0000 7fddfd59a640 1 -- 172.21.15.67:0/1430311663 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf8039070 con 0x7fde000d22b0 2024-03-24T00:47:08.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.217+0000 7fddfd59a640 1 -- 172.21.15.67:0/1430311663 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddf802cc50 con 0x7fde000d22b0 2024-03-24T00:47:08.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.217+0000 7fddfd59a640 1 -- 172.21.15.67:0/1430311663 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf8034440 con 0x7fde000d22b0 2024-03-24T00:47:08.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.218+0000 7fde0495b640 1 -- 172.21.15.67:0/1430311663 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde000d22b0 msgr2=0x7fde000d2760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:08.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.218+0000 7fde0495b640 1 --2- 172.21.15.67:0/1430311663 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde000d22b0 0x7fde000d2760 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fddf8009d90 tx=0x7fddf802bea0 comp rx=0 tx=0).stop 2024-03-24T00:47:08.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.218+0000 7fde0495b640 1 -- 172.21.15.67:0/1430311663 shutdown_connections 2024-03-24T00:47:08.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.218+0000 7fde0495b640 1 --2- 172.21.15.67:0/1430311663 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde000d22b0 0x7fde000d2760 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:08.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.218+0000 7fde0495b640 1 --2- 172.21.15.67:0/1430311663 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fde000d0d50 0x7fde000ce610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:08.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.218+0000 7fde0495b640 1 -- 172.21.15.67:0/1430311663 >> 172.21.15.67:0/1430311663 conn(0x7fde0006c8a0 msgr2=0x7fde0006ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:08.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.218+0000 7fde0495b640 1 -- 172.21.15.67:0/1430311663 shutdown_connections 2024-03-24T00:47:08.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.218+0000 7fde0495b640 1 -- 172.21.15.67:0/1430311663 wait complete. 2024-03-24T00:47:08.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.219+0000 7fde0495b640 1 Processor -- start 2024-03-24T00:47:08.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.219+0000 7fde0495b640 1 -- start start 2024-03-24T00:47:08.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.220+0000 7fde0495b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde000d0d50 0x7fde001f6a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:08.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.220+0000 7fde0495b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fde000d22b0 0x7fde001f3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:08.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.220+0000 7fde0495b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde0015e130 0x7fde0015f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:08.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.220+0000 7fde0495b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde0015e510 con 0x7fde000d0d50 2024-03-24T00:47:08.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.220+0000 7fde0495b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde000cf290 con 0x7fde0015e130 2024-03-24T00:47:08.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.220+0000 7fde0495b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde000b2e10 con 0x7fde000d22b0 2024-03-24T00:47:08.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.220+0000 7fddfe59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde000d0d50 0x7fde001f6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:08.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.220+0000 7fddfe59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde000d0d50 0x7fde001f6a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39360/0 (socket says 172.21.15.67:39360) 2024-03-24T00:47:08.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.220+0000 7fddfe59c640 1 -- 172.21.15.67:0/868389912 learned_addr learned my addr 172.21.15.67:0/868389912 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:08.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.220+0000 7fddfdd9b640 1 --2- 172.21.15.67:0/868389912 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fde000d22b0 0x7fde001f3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:08.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.220+0000 7fddfed9d640 1 --2- 172.21.15.67:0/868389912 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde0015e130 0x7fde0015f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:08.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.221+0000 7fddfe59c640 1 -- 172.21.15.67:0/868389912 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fde000d22b0 msgr2=0x7fde001f3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:08.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.221+0000 7fddfe59c640 1 --2- 172.21.15.67:0/868389912 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fde000d22b0 0x7fde001f3c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:08.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.221+0000 7fddfe59c640 1 -- 172.21.15.67:0/868389912 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde0015e130 msgr2=0x7fde0015f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:08.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.221+0000 7fddfe59c640 1 --2- 172.21.15.67:0/868389912 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde0015e130 0x7fde0015f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:08.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.221+0000 7fddfe59c640 1 -- 172.21.15.67:0/868389912 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde000b3890 con 0x7fde000d0d50 2024-03-24T00:47:08.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.221+0000 7fddfed9d640 1 --2- 172.21.15.67:0/868389912 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde0015e130 0x7fde0015f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:47:08.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.221+0000 7fddfe59c640 1 --2- 172.21.15.67:0/868389912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde000d0d50 0x7fde001f6a70 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7fddf000c430 tx=0x7fddf000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:08.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.221+0000 7fddeeffd640 1 -- 172.21.15.67:0/868389912 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf00074e0 con 0x7fde000d0d50 2024-03-24T00:47:08.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.221+0000 7fddeeffd640 1 -- 172.21.15.67:0/868389912 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddf00047e0 con 0x7fde000d0d50 2024-03-24T00:47:08.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.221+0000 7fddeeffd640 1 -- 172.21.15.67:0/868389912 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf0004bd0 con 0x7fde000d0d50 2024-03-24T00:47:08.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.221+0000 7fde0495b640 1 -- 172.21.15.67:0/868389912 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde000b3b70 con 0x7fde000d0d50 2024-03-24T00:47:08.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.221+0000 7fde0495b640 1 -- 172.21.15.67:0/868389912 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde001f6fb0 con 0x7fde000d0d50 2024-03-24T00:47:08.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.222+0000 7fde0495b640 1 -- 172.21.15.67:0/868389912 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde0004f400 con 0x7fde000d0d50 2024-03-24T00:47:08.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.223+0000 7fddeeffd640 1 -- 172.21.15.67:0/868389912 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fddf0011070 con 0x7fde000d0d50 2024-03-24T00:47:08.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.223+0000 7fddeeffd640 1 --2- 172.21.15.67:0/868389912 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdde006bbd0 0x7fdde006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:08.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.223+0000 7fddeeffd640 1 -- 172.21.15.67:0/868389912 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5507+0+0 (secure 0 0 0) 0x7fddf008a660 con 0x7fde000d0d50 2024-03-24T00:47:08.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.223+0000 7fddfdd9b640 1 --2- 172.21.15.67:0/868389912 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdde006bbd0 0x7fdde006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:08.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.225+0000 7fddfdd9b640 1 --2- 172.21.15.67:0/868389912 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdde006bbd0 0x7fdde006e090 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fde000b2800 tx=0x7fddf4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:08.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.230+0000 7fddeeffd640 1 -- 172.21.15.67:0/868389912 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fddf000e280 con 0x7fde000d0d50 2024-03-24T00:47:08.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.413+0000 7fde0495b640 1 -- 172.21.15.67:0/868389912 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fde0015d7c0 con 0x7fde000d0d50 2024-03-24T00:47:08.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.416+0000 7fddeeffd640 1 -- 172.21.15.67:0/868389912 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v19) v1 ==== 104+0+0 (secure 0 0 0) 0x7fddf0059220 con 0x7fde000d0d50 2024-03-24T00:47:08.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.418+0000 7fde0495b640 1 -- 172.21.15.67:0/868389912 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdde006bbd0 msgr2=0x7fdde006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:08.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.418+0000 7fde0495b640 1 --2- 172.21.15.67:0/868389912 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdde006bbd0 0x7fdde006e090 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fde000b2800 tx=0x7fddf4008040 comp rx=0 tx=0).stop 2024-03-24T00:47:08.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.418+0000 7fde0495b640 1 -- 172.21.15.67:0/868389912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde000d0d50 msgr2=0x7fde001f6a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:08.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.418+0000 7fde0495b640 1 --2- 172.21.15.67:0/868389912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde000d0d50 0x7fde001f6a70 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7fddf000c430 tx=0x7fddf000c8f0 comp rx=0 tx=0).stop 2024-03-24T00:47:08.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.418+0000 7fde0495b640 1 -- 172.21.15.67:0/868389912 shutdown_connections 2024-03-24T00:47:08.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.418+0000 7fde0495b640 1 --2- 172.21.15.67:0/868389912 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdde006bbd0 0x7fdde006e090 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:08.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.418+0000 7fde0495b640 1 --2- 172.21.15.67:0/868389912 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde0015e130 0x7fde0015f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:08.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.418+0000 7fde0495b640 1 --2- 172.21.15.67:0/868389912 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fde000d22b0 0x7fde001f3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:08.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.418+0000 7fde0495b640 1 --2- 172.21.15.67:0/868389912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde000d0d50 0x7fde001f6a70 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:08.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.418+0000 7fde0495b640 1 -- 172.21.15.67:0/868389912 >> 172.21.15.67:0/868389912 conn(0x7fde0006c8a0 msgr2=0x7fde000c98c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:08.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.418+0000 7fde0495b640 1 -- 172.21.15.67:0/868389912 shutdown_connections 2024-03-24T00:47:08.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.418+0000 7fde0495b640 1 -- 172.21.15.67:0/868389912 wait complete. 2024-03-24T00:47:08.430 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dirfrag_repair' 2024-03-24T00:47:08.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.530+0000 7f7be3cff640 1 Processor -- start 2024-03-24T00:47:08.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.530+0000 7f7be3cff640 1 -- start start 2024-03-24T00:47:08.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.531+0000 7f7be3cff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7bdc0d1550 0x7f7bdc06cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:08.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.531+0000 7f7be3cff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7bdc09a480 0x7f7bdc0b2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:08.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.531+0000 7f7be3cff640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bdc0d1930 con 0x7f7bdc0b2780 2024-03-24T00:47:08.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.531+0000 7f7be3cff640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bdc06d420 con 0x7f7bdc0d1550 2024-03-24T00:47:08.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.531+0000 7f7be3cff640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bdc06ca50 con 0x7f7bdc09a480 2024-03-24T00:47:08.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.531+0000 7f7be1a9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7bdc0d1550 0x7f7bdc06cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:08.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.531+0000 7f7be1a9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7bdc0d1550 0x7f7bdc06cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39364/0 (socket says 172.21.15.67:39364) 2024-03-24T00:47:08.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.531+0000 7f7be129a640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7bdc0b2780 0x7f7bdc099d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46810/0 (socket says 172.21.15.67:46810) 2024-03-24T00:47:08.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.531+0000 7f7be1a9b640 1 -- 172.21.15.67:0/2230861799 learned_addr learned my addr 172.21.15.67:0/2230861799 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:08.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.531+0000 7f7be229c640 1 --2- 172.21.15.67:0/2230861799 >> v2:172.21.15.72:3300/0 conn(0x7f7bdc09a480 0x7f7bdc0b2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:08.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.531+0000 7f7be1a9b640 1 -- 172.21.15.67:0/2230861799 >> v2:172.21.15.72:3300/0 conn(0x7f7bdc09a480 msgr2=0x7f7bdc0b2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:08.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.531+0000 7f7be1a9b640 1 --2- 172.21.15.67:0/2230861799 >> v2:172.21.15.72:3300/0 conn(0x7f7bdc09a480 0x7f7bdc0b2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:08.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.531+0000 7f7be1a9b640 1 -- 172.21.15.67:0/2230861799 >> v1:172.21.15.67:6789/0 conn(0x7f7bdc0b2780 legacy=0x7f7bdc099d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:08.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.531+0000 7f7be0a99640 1 -- 172.21.15.67:0/2230861799 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1525189367 0 0) 0x7f7bdc0d1930 con 0x7f7bdc0b2780 2024-03-24T00:47:08.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.531+0000 7f7be1a9b640 1 -- 172.21.15.67:0/2230861799 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bdc0b1f60 con 0x7f7bdc0d1550 2024-03-24T00:47:08.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.531+0000 7f7be229c640 1 --2- 172.21.15.67:0/2230861799 >> v2:172.21.15.72:3300/0 conn(0x7f7bdc09a480 0x7f7bdc0b2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:47:08.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.532+0000 7f7be1a9b640 1 --2- 172.21.15.67:0/2230861799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bdc0d1550 0x7f7bdc06cee0 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc009e50 tx=0x7f7bcc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d814773feef3a6a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:08.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.532+0000 7f7be0a99640 1 -- 172.21.15.67:0/2230861799 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bcc02ca80 con 0x7f7bdc0d1550 2024-03-24T00:47:08.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.532+0000 7f7be0a99640 1 -- 172.21.15.67:0/2230861799 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bcc02cc20 con 0x7f7bdc0d1550 2024-03-24T00:47:08.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.532+0000 7f7be0a99640 1 -- 172.21.15.67:0/2230861799 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bcc0343f0 con 0x7f7bdc0d1550 2024-03-24T00:47:08.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.533+0000 7f7be3cff640 1 -- 172.21.15.67:0/2230861799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bdc0d1550 msgr2=0x7f7bdc06cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:08.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.533+0000 7f7be3cff640 1 --2- 172.21.15.67:0/2230861799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bdc0d1550 0x7f7bdc06cee0 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc009e50 tx=0x7f7bcc02bee0 comp rx=0 tx=0).stop 2024-03-24T00:47:08.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.533+0000 7f7be3cff640 1 -- 172.21.15.67:0/2230861799 shutdown_connections 2024-03-24T00:47:08.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.533+0000 7f7be3cff640 1 --2- 172.21.15.67:0/2230861799 >> v2:172.21.15.72:3300/0 conn(0x7f7bdc09a480 0x7f7bdc0b2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:08.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.533+0000 7f7be3cff640 1 --2- 172.21.15.67:0/2230861799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bdc0d1550 0x7f7bdc06cee0 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:08.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.533+0000 7f7be3cff640 1 -- 172.21.15.67:0/2230861799 >> 172.21.15.67:0/2230861799 conn(0x7f7bdc0b36e0 msgr2=0x7f7bdc0b3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:08.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.533+0000 7f7be3cff640 1 -- 172.21.15.67:0/2230861799 shutdown_connections 2024-03-24T00:47:08.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.533+0000 7f7be3cff640 1 -- 172.21.15.67:0/2230861799 wait complete. 2024-03-24T00:47:08.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.534+0000 7f7be3cff640 1 Processor -- start 2024-03-24T00:47:08.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.534+0000 7f7be3cff640 1 -- start start 2024-03-24T00:47:08.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.535+0000 7f7be3cff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7bdc09a480 0x7f7bdc1601d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:08.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.535+0000 7f7be3cff640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7bdc0b2780 0x7f7bdc160710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:08.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.535+0000 7f7be3cff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bdc0d1550 0x7f7bdc1f7e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:08.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.535+0000 7f7be3cff640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bdc06c400 con 0x7f7bdc0d1550 2024-03-24T00:47:08.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.535+0000 7f7be3cff640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bdc0b2520 con 0x7f7bdc09a480 2024-03-24T00:47:08.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.535+0000 7f7be3cff640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bdc06c6e0 con 0x7f7bdc0b2780 2024-03-24T00:47:08.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.535+0000 7f7be229c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bdc0d1550 0x7f7bdc1f7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:08.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.535+0000 7f7be229c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bdc0d1550 0x7f7bdc1f7e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39374/0 (socket says 172.21.15.67:39374) 2024-03-24T00:47:08.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.535+0000 7f7be229c640 1 -- 172.21.15.67:0/2398200644 learned_addr learned my addr 172.21.15.67:0/2398200644 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:08.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.535+0000 7f7be1a9b640 1 --2- 172.21.15.67:0/2398200644 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7bdc09a480 0x7f7bdc1601d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:08.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.535+0000 7f7be129a640 1 --2- 172.21.15.67:0/2398200644 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7bdc0b2780 0x7f7bdc160710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:08.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.535+0000 7f7be229c640 1 -- 172.21.15.67:0/2398200644 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7bdc0b2780 msgr2=0x7f7bdc160710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:08.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.535+0000 7f7be229c640 1 --2- 172.21.15.67:0/2398200644 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7bdc0b2780 0x7f7bdc160710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:08.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.535+0000 7f7be229c640 1 -- 172.21.15.67:0/2398200644 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7bdc09a480 msgr2=0x7f7bdc1601d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:08.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.535+0000 7f7be229c640 1 --2- 172.21.15.67:0/2398200644 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7bdc09a480 0x7f7bdc1601d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:08.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.535+0000 7f7be229c640 1 -- 172.21.15.67:0/2398200644 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bdc1f5670 con 0x7f7bdc0d1550 2024-03-24T00:47:08.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.535+0000 7f7be1a9b640 1 --2- 172.21.15.67:0/2398200644 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7bdc09a480 0x7f7bdc1601d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:47:08.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.535+0000 7f7be229c640 1 --2- 172.21.15.67:0/2398200644 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bdc0d1550 0x7f7bdc1f7e20 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f7bd400c5f0 tx=0x7f7bd400cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:08.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.536+0000 7f7bda7fc640 1 -- 172.21.15.67:0/2398200644 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd4015070 con 0x7f7bdc0d1550 2024-03-24T00:47:08.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.536+0000 7f7bda7fc640 1 -- 172.21.15.67:0/2398200644 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bd4011820 con 0x7f7bdc0d1550 2024-03-24T00:47:08.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.536+0000 7f7bda7fc640 1 -- 172.21.15.67:0/2398200644 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd4011c10 con 0x7f7bdc0d1550 2024-03-24T00:47:08.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.536+0000 7f7be3cff640 1 -- 172.21.15.67:0/2398200644 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bdc1f8ba0 con 0x7f7bdc0d1550 2024-03-24T00:47:08.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.536+0000 7f7be3cff640 1 -- 172.21.15.67:0/2398200644 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bdc1f4b90 con 0x7f7bdc0d1550 2024-03-24T00:47:08.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.537+0000 7f7be3cff640 1 -- 172.21.15.67:0/2398200644 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bdc04f400 con 0x7f7bdc0d1550 2024-03-24T00:47:08.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.537+0000 7f7bda7fc640 1 -- 172.21.15.67:0/2398200644 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7bd4018070 con 0x7f7bdc0d1550 2024-03-24T00:47:08.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.537+0000 7f7bda7fc640 1 --2- 172.21.15.67:0/2398200644 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7bc806bbd0 0x7f7bc806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:08.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.538+0000 7f7bda7fc640 1 -- 172.21.15.67:0/2398200644 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f7bd408a8b0 con 0x7f7bdc0d1550 2024-03-24T00:47:08.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.538+0000 7f7be1a9b640 1 --2- 172.21.15.67:0/2398200644 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7bc806bbd0 0x7f7bc806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:08.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.538+0000 7f7be1a9b640 1 --2- 172.21.15.67:0/2398200644 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7bc806bbd0 0x7f7bc806e090 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc035000 tx=0x7f7bcc030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:08.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.542+0000 7f7bda7fc640 1 -- 172.21.15.67:0/2398200644 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7bd4059470 con 0x7f7bdc0d1550 2024-03-24T00:47:08.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:08.799+0000 7f7be3cff640 1 -- 172.21.15.67:0/2398200644 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dirfrag_repair"]} v 0) v1 -- 0x7f7bdc0b7a70 con 0x7f7bdc0d1550 2024-03-24T00:47:09.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.092+0000 7f7bda7fc640 1 -- 172.21.15.67:0/2398200644 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dirfrag_repair"]}]=0 v391) v1 ==== 167+0+0 (secure 0 0 0) 0x7f7bd4058d70 con 0x7f7bdc0d1550 2024-03-24T00:47:09.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.095+0000 7f7be3cff640 1 -- 172.21.15.67:0/2398200644 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7bc806bbd0 msgr2=0x7f7bc806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:09.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.095+0000 7f7be3cff640 1 --2- 172.21.15.67:0/2398200644 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7bc806bbd0 0x7f7bc806e090 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc035000 tx=0x7f7bcc030000 comp rx=0 tx=0).stop 2024-03-24T00:47:09.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.095+0000 7f7be3cff640 1 -- 172.21.15.67:0/2398200644 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bdc0d1550 msgr2=0x7f7bdc1f7e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:09.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.095+0000 7f7be3cff640 1 --2- 172.21.15.67:0/2398200644 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bdc0d1550 0x7f7bdc1f7e20 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f7bd400c5f0 tx=0x7f7bd400cab0 comp rx=0 tx=0).stop 2024-03-24T00:47:09.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.095+0000 7f7be3cff640 1 -- 172.21.15.67:0/2398200644 shutdown_connections 2024-03-24T00:47:09.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.095+0000 7f7be3cff640 1 --2- 172.21.15.67:0/2398200644 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7bc806bbd0 0x7f7bc806e090 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:09.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.095+0000 7f7be3cff640 1 --2- 172.21.15.67:0/2398200644 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bdc0d1550 0x7f7bdc1f7e20 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:09.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.095+0000 7f7be3cff640 1 --2- 172.21.15.67:0/2398200644 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7bdc0b2780 0x7f7bdc160710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:09.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.095+0000 7f7be3cff640 1 --2- 172.21.15.67:0/2398200644 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7bdc09a480 0x7f7bdc1601d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:09.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.095+0000 7f7be3cff640 1 -- 172.21.15.67:0/2398200644 >> 172.21.15.67:0/2398200644 conn(0x7f7bdc0b36e0 msgr2=0x7f7bdc0b3ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:09.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.095+0000 7f7be3cff640 1 -- 172.21.15.67:0/2398200644 shutdown_connections 2024-03-24T00:47:09.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.095+0000 7f7be3cff640 1 -- 172.21.15.67:0/2398200644 wait complete. 2024-03-24T00:47:09.107 INFO:tasks.cephfs_test_runner:Test that the damage health status is cleared ... ok 2024-03-24T00:47:09.107 INFO:tasks.cephfs_test_runner:Starting test: test_inotable_sync (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2024-03-24T00:47:09.107 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:47:09.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.209+0000 7f028bb1b640 1 Processor -- start 2024-03-24T00:47:09.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.209+0000 7f028bb1b640 1 -- start start 2024-03-24T00:47:09.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.209+0000 7f028bb1b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f028415df30 0x7f0284162610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:09.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.209+0000 7f028bb1b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f028415e310 0x7f02840d0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:09.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.209+0000 7f028bb1b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02840ce920 con 0x7f02840d0b80 2024-03-24T00:47:09.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.209+0000 7f028bb1b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0284162b50 con 0x7f028415df30 2024-03-24T00:47:09.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.209+0000 7f028bb1b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02840d0fd0 con 0x7f028415e310 2024-03-24T00:47:09.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.210+0000 7f02898b7640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f028415df30 0x7f0284162610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:09.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.210+0000 7f02890b6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f028415e310 0x7f02840d0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:09.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.210+0000 7f02898b7640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f028415df30 0x7f0284162610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:37772/0 (socket says 172.21.15.67:37772) 2024-03-24T00:47:09.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.210+0000 7f02898b7640 1 -- 172.21.15.67:0/2408448665 learned_addr learned my addr 172.21.15.67:0/2408448665 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:09.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.210+0000 7f028a0b8640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f02840d0b80 0x7f02840b57a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:58066/0 (socket says 172.21.15.67:58066) 2024-03-24T00:47:09.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.210+0000 7f02898b7640 1 -- 172.21.15.67:0/2408448665 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f028415e310 msgr2=0x7f02840d0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:09.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.210+0000 7f02898b7640 1 --2- 172.21.15.67:0/2408448665 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f028415e310 0x7f02840d0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:09.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.210+0000 7f02898b7640 1 -- 172.21.15.67:0/2408448665 >> v1:172.21.15.72:6789/0 conn(0x7f02840d0b80 legacy=0x7f02840b57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:09.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.210+0000 7f02898b7640 1 -- 172.21.15.67:0/2408448665 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02840cef20 con 0x7f028415df30 2024-03-24T00:47:09.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.211+0000 7f02898b7640 1 --2- 172.21.15.67:0/2408448665 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f028415df30 0x7f0284162610 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f027c0098a0 tx=0x7f027c02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16f1d15c600abd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:09.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.211+0000 7f02888b5640 1 -- 172.21.15.67:0/2408448665 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f027c035020 con 0x7f028415df30 2024-03-24T00:47:09.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.211+0000 7f02888b5640 1 -- 172.21.15.67:0/2408448665 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f027c031640 con 0x7f028415df30 2024-03-24T00:47:09.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.211+0000 7f02888b5640 1 -- 172.21.15.67:0/2408448665 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f027c02d040 con 0x7f028415df30 2024-03-24T00:47:09.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.212+0000 7f028bb1b640 1 -- 172.21.15.67:0/2408448665 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f028415df30 msgr2=0x7f0284162610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:09.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.212+0000 7f028bb1b640 1 --2- 172.21.15.67:0/2408448665 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f028415df30 0x7f0284162610 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f027c0098a0 tx=0x7f027c02b8f0 comp rx=0 tx=0).stop 2024-03-24T00:47:09.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.212+0000 7f028bb1b640 1 -- 172.21.15.67:0/2408448665 shutdown_connections 2024-03-24T00:47:09.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.212+0000 7f028bb1b640 1 --2- 172.21.15.67:0/2408448665 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f028415e310 0x7f02840d0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:09.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.212+0000 7f028bb1b640 1 --2- 172.21.15.67:0/2408448665 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f028415df30 0x7f0284162610 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:09.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.212+0000 7f028bb1b640 1 -- 172.21.15.67:0/2408448665 >> 172.21.15.67:0/2408448665 conn(0x7f028406c880 msgr2=0x7f028406cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:09.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.212+0000 7f028bb1b640 1 -- 172.21.15.67:0/2408448665 shutdown_connections 2024-03-24T00:47:09.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.212+0000 7f028bb1b640 1 -- 172.21.15.67:0/2408448665 wait complete. 2024-03-24T00:47:09.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.213+0000 7f028bb1b640 1 Processor -- start 2024-03-24T00:47:09.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.213+0000 7f028bb1b640 1 -- start start 2024-03-24T00:47:09.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.214+0000 7f028bb1b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02840d0b80 0x7f02841f31c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:09.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.214+0000 7f028bb1b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f028415df30 0x7f02840c7850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:09.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.214+0000 7f028bb1b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02840c7d90 0x7f02840b5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:09.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.214+0000 7f028bb1b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02840b32f0 con 0x7f02840d0b80 2024-03-24T00:47:09.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.214+0000 7f028bb1b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02840cf4e0 con 0x7f028415df30 2024-03-24T00:47:09.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.214+0000 7f028bb1b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02840ce580 con 0x7f02840c7d90 2024-03-24T00:47:09.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.214+0000 7f02898b7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02840d0b80 0x7f02841f31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:09.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.214+0000 7f02898b7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02840d0b80 0x7f02841f31c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39384/0 (socket says 172.21.15.67:39384) 2024-03-24T00:47:09.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.214+0000 7f02898b7640 1 -- 172.21.15.67:0/1995115687 learned_addr learned my addr 172.21.15.67:0/1995115687 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:09.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.214+0000 7f028a0b8640 1 --2- 172.21.15.67:0/1995115687 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02840c7d90 0x7f02840b5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:09.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.214+0000 7f02890b6640 1 --2- 172.21.15.67:0/1995115687 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f028415df30 0x7f02840c7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:09.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.214+0000 7f02898b7640 1 -- 172.21.15.67:0/1995115687 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02840c7d90 msgr2=0x7f02840b5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:09.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.214+0000 7f02898b7640 1 --2- 172.21.15.67:0/1995115687 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02840c7d90 0x7f02840b5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:09.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.214+0000 7f02898b7640 1 -- 172.21.15.67:0/1995115687 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f028415df30 msgr2=0x7f02840c7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:09.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.214+0000 7f02898b7640 1 --2- 172.21.15.67:0/1995115687 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f028415df30 0x7f02840c7850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:09.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.214+0000 7f02898b7640 1 -- 172.21.15.67:0/1995115687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02840cbf70 con 0x7f02840d0b80 2024-03-24T00:47:09.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.214+0000 7f02890b6640 1 --2- 172.21.15.67:0/1995115687 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f028415df30 0x7f02840c7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:47:09.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.214+0000 7f02898b7640 1 --2- 172.21.15.67:0/1995115687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02840d0b80 0x7f02841f31c0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f027c02b930 tx=0x7f027c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:09.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.214+0000 7f02827fc640 1 -- 172.21.15.67:0/1995115687 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f027c040070 con 0x7f02840d0b80 2024-03-24T00:47:09.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.215+0000 7f028bb1b640 1 -- 172.21.15.67:0/1995115687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02840cc250 con 0x7f02840d0b80 2024-03-24T00:47:09.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.215+0000 7f02827fc640 1 -- 172.21.15.67:0/1995115687 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f027c0399f0 con 0x7f02840d0b80 2024-03-24T00:47:09.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.215+0000 7f02827fc640 1 -- 172.21.15.67:0/1995115687 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f027c035040 con 0x7f02840d0b80 2024-03-24T00:47:09.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.215+0000 7f028bb1b640 1 -- 172.21.15.67:0/1995115687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02840cc810 con 0x7f02840d0b80 2024-03-24T00:47:09.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.215+0000 7f028bb1b640 1 -- 172.21.15.67:0/1995115687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f028404f400 con 0x7f02840d0b80 2024-03-24T00:47:09.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.216+0000 7f02827fc640 1 -- 172.21.15.67:0/1995115687 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f027c04b060 con 0x7f02840d0b80 2024-03-24T00:47:09.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.216+0000 7f02827fc640 1 --2- 172.21.15.67:0/1995115687 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f027006bbd0 0x7f027006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:09.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.216+0000 7f02827fc640 1 -- 172.21.15.67:0/1995115687 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f027c0ac1b0 con 0x7f02840d0b80 2024-03-24T00:47:09.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.216+0000 7f02890b6640 1 --2- 172.21.15.67:0/1995115687 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f027006bbd0 0x7f027006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:09.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.217+0000 7f02890b6640 1 --2- 172.21.15.67:0/1995115687 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f027006bbd0 0x7f027006e090 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f02840b4f60 tx=0x7f0278008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:09.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.221+0000 7f02827fc640 1 -- 172.21.15.67:0/1995115687 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f027c07ad70 con 0x7f02840d0b80 2024-03-24T00:47:09.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.402+0000 7f028bb1b640 1 -- 172.21.15.67:0/1995115687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f02840d1a30 con 0x7f02840d0b80 2024-03-24T00:47:09.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.403+0000 7f02827fc640 1 -- 172.21.15.67:0/1995115687 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) v1 ==== 74+0+13494 (secure 0 0 0) 0x7f027c07a670 con 0x7f02840d0b80 2024-03-24T00:47:09.404 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:47:09.404 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":96,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:47:08.086167+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6832/3311719072":"2024-03-25T00:47:05.072561+0000","172.21.15.67:6833/1336395002":"2024-03-25T00:46:20.877877+0000","172.21.15.67:6832/1336395002":"2024-03-25T00:46:20.877877+0000","172.21.15.67:6833/3311719072":"2024-03-25T00:47:05.072561+0000","172.21.15.67:6833/3021622863":"2024-03-25T00:45:09.102827+0000","172.21.15.67:6832/3021622863":"2024-03-25T00:45:09.102827+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:47:09.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.405+0000 7f028bb1b640 1 -- 172.21.15.67:0/1995115687 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f027006bbd0 msgr2=0x7f027006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:09.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.405+0000 7f028bb1b640 1 --2- 172.21.15.67:0/1995115687 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f027006bbd0 0x7f027006e090 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f02840b4f60 tx=0x7f0278008040 comp rx=0 tx=0).stop 2024-03-24T00:47:09.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.405+0000 7f028bb1b640 1 -- 172.21.15.67:0/1995115687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02840d0b80 msgr2=0x7f02841f31c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:09.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.405+0000 7f028bb1b640 1 --2- 172.21.15.67:0/1995115687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02840d0b80 0x7f02841f31c0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f027c02b930 tx=0x7f027c0046c0 comp rx=0 tx=0).stop 2024-03-24T00:47:09.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.405+0000 7f028bb1b640 1 -- 172.21.15.67:0/1995115687 shutdown_connections 2024-03-24T00:47:09.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.405+0000 7f028bb1b640 1 --2- 172.21.15.67:0/1995115687 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f027006bbd0 0x7f027006e090 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:09.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.405+0000 7f028bb1b640 1 --2- 172.21.15.67:0/1995115687 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02840c7d90 0x7f02840b5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:09.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.405+0000 7f028bb1b640 1 --2- 172.21.15.67:0/1995115687 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f028415df30 0x7f02840c7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:09.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.405+0000 7f028bb1b640 1 --2- 172.21.15.67:0/1995115687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02840d0b80 0x7f02841f31c0 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:09.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.405+0000 7f028bb1b640 1 -- 172.21.15.67:0/1995115687 >> 172.21.15.67:0/1995115687 conn(0x7f028406c880 msgr2=0x7f028406cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:09.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.405+0000 7f028bb1b640 1 -- 172.21.15.67:0/1995115687 shutdown_connections 2024-03-24T00:47:09.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.405+0000 7f028bb1b640 1 -- 172.21.15.67:0/1995115687 wait complete. 2024-03-24T00:47:09.417 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:38:39.364950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:47:09.417 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:47:09.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.520+0000 7f030684b640 1 Processor -- start 2024-03-24T00:47:09.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.520+0000 7f030684b640 1 -- start start 2024-03-24T00:47:09.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.521+0000 7f030684b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f03000b9690 0x7f03000ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:09.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.521+0000 7f030684b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0300162290 con 0x7f0300161370 2024-03-24T00:47:09.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.521+0000 7f030684b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0300162570 con 0x7f03000cef30 2024-03-24T00:47:09.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.521+0000 7f030684b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03000b6e10 con 0x7f03000b9690 2024-03-24T00:47:09.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.521+0000 7f02fffff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0300161370 0x7f0300161ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46812/0 (socket says 172.21.15.67:46812) 2024-03-24T00:47:09.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.521+0000 7f02fffff640 1 -- 172.21.15.67:0/211288747 learned_addr learned my addr 172.21.15.67:0/211288747 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:47:09.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.522+0000 7f02feffd640 1 -- 172.21.15.67:0/211288747 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2125218608 0 0) 0x7f0300162290 con 0x7f0300161370 2024-03-24T00:47:09.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.522+0000 7f02ff7fe640 1 --2- 172.21.15.67:0/211288747 >> v2:172.21.15.72:3300/0 conn(0x7f03000b9690 0x7f03000ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:09.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.522+0000 7f02feffd640 1 -- 172.21.15.67:0/211288747 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02e8003520 con 0x7f0300161370 2024-03-24T00:47:09.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.522+0000 7f02feffd640 1 -- 172.21.15.67:0/211288747 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1880349454 0 0) 0x7f02e8003520 con 0x7f0300161370 2024-03-24T00:47:09.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.522+0000 7f02feffd640 1 -- 172.21.15.67:0/211288747 >> v2:172.21.15.72:3300/0 conn(0x7f03000b9690 msgr2=0x7f03000ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:09.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.522+0000 7f02feffd640 1 --2- 172.21.15.67:0/211288747 >> v2:172.21.15.72:3300/0 conn(0x7f03000b9690 0x7f03000ce9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:09.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.522+0000 7f02feffd640 1 -- 172.21.15.67:0/211288747 >> v1:172.21.15.72:6789/0 conn(0x7f03000cef30 legacy=0x7f03000d2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:09.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.522+0000 7f02feffd640 1 -- 172.21.15.67:0/211288747 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0300162850 con 0x7f0300161370 2024-03-24T00:47:09.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.522+0000 7f02feffd640 1 -- 172.21.15.67:0/211288747 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f02f0001670 con 0x7f0300161370 2024-03-24T00:47:09.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.522+0000 7f02feffd640 1 -- 172.21.15.67:0/211288747 >> v1:172.21.15.67:6789/0 conn(0x7f0300161370 legacy=0x7f0300161ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:09.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.522+0000 7f02feffd640 1 --2- 172.21.15.67:0/211288747 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02e80039b0 0x7f02e8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:09.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.522+0000 7f02feffd640 1 --2- 172.21.15.67:0/211288747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02e8004800 0x7f02e8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:09.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.522+0000 7f02feffd640 1 --2- 172.21.15.67:0/211288747 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02e80051c0 0x7f02e800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:09.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.522+0000 7f02feffd640 1 -- 172.21.15.67:0/211288747 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02e800c040 con 0x7f02e8004800 2024-03-24T00:47:09.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.522+0000 7f02feffd640 1 -- 172.21.15.67:0/211288747 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02e800c1f0 con 0x7f02e80051c0 2024-03-24T00:47:09.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.522+0000 7f02feffd640 1 -- 172.21.15.67:0/211288747 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02e800c3f0 con 0x7f02e80039b0 2024-03-24T00:47:09.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.522+0000 7f02ff7fe640 1 --2- 172.21.15.67:0/211288747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02e8004800 0x7f02e8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:09.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.522+0000 7f0304de8640 1 --2- 172.21.15.67:0/211288747 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02e80051c0 0x7f02e800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:09.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.522+0000 7f02fffff640 1 --2- 172.21.15.67:0/211288747 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02e80039b0 0x7f02e8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:09.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.522+0000 7f02ff7fe640 1 -- 172.21.15.67:0/211288747 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02e80039b0 msgr2=0x7f02e8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:09.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.522+0000 7f02ff7fe640 1 --2- 172.21.15.67:0/211288747 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02e80039b0 0x7f02e8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:09.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.522+0000 7f02ff7fe640 1 -- 172.21.15.67:0/211288747 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02e80051c0 msgr2=0x7f02e800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:09.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.522+0000 7f02ff7fe640 1 --2- 172.21.15.67:0/211288747 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02e80051c0 0x7f02e800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:09.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.522+0000 7f0304de8640 1 -- 172.21.15.67:0/211288747 reap_dead start 2024-03-24T00:47:09.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.522+0000 7f02ff7fe640 1 -- 172.21.15.67:0/211288747 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e8003520 con 0x7f02e8004800 2024-03-24T00:47:09.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.523+0000 7f02ff7fe640 1 --2- 172.21.15.67:0/211288747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02e8004800 0x7f02e8004c80 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f02f4009ac0 tx=0x7f02e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8595d6964a4b6df server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:09.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.523+0000 7f02feffd640 1 -- 172.21.15.67:0/211288747 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f4039070 con 0x7f02e8004800 2024-03-24T00:47:09.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.523+0000 7f02feffd640 1 -- 172.21.15.67:0/211288747 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02f402c6b0 con 0x7f02e8004800 2024-03-24T00:47:09.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.523+0000 7f02feffd640 1 -- 172.21.15.67:0/211288747 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f4009690 con 0x7f02e8004800 2024-03-24T00:47:09.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.524+0000 7f030684b640 1 -- 172.21.15.67:0/211288747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02e8004800 msgr2=0x7f02e8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:09.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.524+0000 7f030684b640 1 --2- 172.21.15.67:0/211288747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02e8004800 0x7f02e8004c80 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f02f4009ac0 tx=0x7f02e8005830 comp rx=0 tx=0).stop 2024-03-24T00:47:09.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.524+0000 7f030684b640 1 -- 172.21.15.67:0/211288747 shutdown_connections 2024-03-24T00:47:09.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.524+0000 7f030684b640 1 --2- 172.21.15.67:0/211288747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02e8004800 0x7f02e8004c80 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:09.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.524+0000 7f030684b640 1 -- 172.21.15.67:0/211288747 >> 172.21.15.67:0/211288747 conn(0x7f030006c880 msgr2=0x7f030006cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:09.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.524+0000 7f030684b640 1 -- 172.21.15.67:0/211288747 shutdown_connections 2024-03-24T00:47:09.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.524+0000 7f030684b640 1 -- 172.21.15.67:0/211288747 wait complete. 2024-03-24T00:47:09.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.525+0000 7f030684b640 1 Processor -- start 2024-03-24T00:47:09.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.525+0000 7f030684b640 1 -- start start 2024-03-24T00:47:09.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.526+0000 7f030684b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02e8004800 0x7f0300151d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:09.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.526+0000 7f030684b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03000b60d0 0x7f030015f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:09.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.526+0000 7f030684b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03000cd560 0x7f030015f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:09.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.526+0000 7f030684b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03000d2190 con 0x7f03000b60d0 2024-03-24T00:47:09.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.526+0000 7f030684b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03000cf380 con 0x7f03000cd560 2024-03-24T00:47:09.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.526+0000 7f030684b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03000ce9f0 con 0x7f02e8004800 2024-03-24T00:47:09.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.526+0000 7f02ff7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03000b60d0 0x7f030015f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:09.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.526+0000 7f02ff7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03000b60d0 0x7f030015f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39416/0 (socket says 172.21.15.67:39416) 2024-03-24T00:47:09.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.526+0000 7f02ff7fe640 1 -- 172.21.15.67:0/4145181759 learned_addr learned my addr 172.21.15.67:0/4145181759 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:09.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.526+0000 7f02ff7fe640 1 -- 172.21.15.67:0/4145181759 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02e8004800 msgr2=0x7f0300151d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:09.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.526+0000 7f02ff7fe640 1 --2- 172.21.15.67:0/4145181759 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02e8004800 0x7f0300151d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:09.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.526+0000 7f02ff7fe640 1 -- 172.21.15.67:0/4145181759 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03000cd560 msgr2=0x7f030015f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:09.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.526+0000 7f02ff7fe640 1 --2- 172.21.15.67:0/4145181759 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03000cd560 0x7f030015f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:09.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.526+0000 7f02ff7fe640 1 -- 172.21.15.67:0/4145181759 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f030015f580 con 0x7f03000b60d0 2024-03-24T00:47:09.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.526+0000 7f02ff7fe640 1 --2- 172.21.15.67:0/4145181759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03000b60d0 0x7f030015f040 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f02f4009ac0 tx=0x7f02f4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:09.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.526+0000 7f02fcff9640 1 -- 172.21.15.67:0/4145181759 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f4039050 con 0x7f03000b60d0 2024-03-24T00:47:09.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.527+0000 7f02fcff9640 1 -- 172.21.15.67:0/4145181759 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02f4045070 con 0x7f03000b60d0 2024-03-24T00:47:09.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.527+0000 7f030684b640 1 -- 172.21.15.67:0/4145181759 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03000b65b0 con 0x7f03000b60d0 2024-03-24T00:47:09.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.527+0000 7f02fcff9640 1 -- 172.21.15.67:0/4145181759 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f40044e0 con 0x7f03000b60d0 2024-03-24T00:47:09.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.527+0000 7f030684b640 1 -- 172.21.15.67:0/4145181759 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f030015e840 con 0x7f03000b60d0 2024-03-24T00:47:09.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.528+0000 7f030684b640 1 -- 172.21.15.67:0/4145181759 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f030004f400 con 0x7f03000b60d0 2024-03-24T00:47:09.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.528+0000 7f02fcff9640 1 -- 172.21.15.67:0/4145181759 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f02f4003850 con 0x7f03000b60d0 2024-03-24T00:47:09.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.528+0000 7f02fcff9640 1 --2- 172.21.15.67:0/4145181759 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f02ec06bc00 0x7f02ec06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:09.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.528+0000 7f02fcff9640 1 -- 172.21.15.67:0/4145181759 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f02f40ac540 con 0x7f03000b60d0 2024-03-24T00:47:09.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.529+0000 7f02fffff640 1 --2- 172.21.15.67:0/4145181759 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f02ec06bc00 0x7f02ec06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:09.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.529+0000 7f02fffff640 1 --2- 172.21.15.67:0/4145181759 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f02ec06bc00 0x7f02ec06e0c0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f02f0001200 tx=0x7f02f0001110 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:09.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.533+0000 7f02fcff9640 1 -- 172.21.15.67:0/4145181759 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f02f404e020 con 0x7f03000b60d0 2024-03-24T00:47:09.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.717+0000 7f030684b640 1 -- 172.21.15.67:0/4145181759 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f03000d1aa0 con 0x7f03000b60d0 2024-03-24T00:47:09.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.718+0000 7f02fcff9640 1 -- 172.21.15.67:0/4145181759 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v96) v1 ==== 93+0+10 (secure 0 0 0) 0x7f02f407b210 con 0x7f03000b60d0 2024-03-24T00:47:09.719 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-24T00:47:09.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.720+0000 7f030684b640 1 -- 172.21.15.67:0/4145181759 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f02ec06bc00 msgr2=0x7f02ec06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:09.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.720+0000 7f030684b640 1 --2- 172.21.15.67:0/4145181759 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f02ec06bc00 0x7f02ec06e0c0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f02f0001200 tx=0x7f02f0001110 comp rx=0 tx=0).stop 2024-03-24T00:47:09.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.720+0000 7f030684b640 1 -- 172.21.15.67:0/4145181759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03000b60d0 msgr2=0x7f030015f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:09.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.720+0000 7f030684b640 1 --2- 172.21.15.67:0/4145181759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03000b60d0 0x7f030015f040 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f02f4009ac0 tx=0x7f02f4004730 comp rx=0 tx=0).stop 2024-03-24T00:47:09.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.720+0000 7f030684b640 1 -- 172.21.15.67:0/4145181759 shutdown_connections 2024-03-24T00:47:09.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.720+0000 7f030684b640 1 --2- 172.21.15.67:0/4145181759 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f02ec06bc00 0x7f02ec06e0c0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:09.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.720+0000 7f030684b640 1 --2- 172.21.15.67:0/4145181759 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03000cd560 0x7f030015f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:09.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.720+0000 7f030684b640 1 --2- 172.21.15.67:0/4145181759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03000b60d0 0x7f030015f040 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:09.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.720+0000 7f030684b640 1 --2- 172.21.15.67:0/4145181759 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02e8004800 0x7f0300151d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:09.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.720+0000 7f030684b640 1 -- 172.21.15.67:0/4145181759 >> 172.21.15.67:0/4145181759 conn(0x7f030006c880 msgr2=0x7f030006cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:09.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.720+0000 7f030684b640 1 -- 172.21.15.67:0/4145181759 shutdown_connections 2024-03-24T00:47:09.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.720+0000 7f030684b640 1 -- 172.21.15.67:0/4145181759 wait complete. 2024-03-24T00:47:09.732 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_inotable_sync' 2024-03-24T00:47:09.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.833+0000 7f7172cdc640 1 Processor -- start 2024-03-24T00:47:09.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.833+0000 7f7172cdc640 1 -- start start 2024-03-24T00:47:09.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.833+0000 7f7172cdc640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f716c15dec0 0x7f716c0ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:09.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.833+0000 7f7172cdc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f716c15e2a0 0x7f716c162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:09.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.833+0000 7f7172cdc640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f716c0cf640 con 0x7f716c0d07e0 2024-03-24T00:47:09.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.833+0000 7f7172cdc640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f716c0cee30 con 0x7f716c15e2a0 2024-03-24T00:47:09.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.833+0000 7f7172cdc640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f716c0d2120 con 0x7f716c15dec0 2024-03-24T00:47:09.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.834+0000 7f7171279640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f716c0d07e0 0x7f716c0b5850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46822/0 (socket says 172.21.15.67:46822) 2024-03-24T00:47:09.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.834+0000 7f7171279640 1 -- 172.21.15.67:0/251182847 learned_addr learned my addr 172.21.15.67:0/251182847 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:47:09.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.834+0000 7f7170a78640 1 --2- 172.21.15.67:0/251182847 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f716c15dec0 0x7f716c0ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:09.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.834+0000 7f716bfff640 1 --2- 172.21.15.67:0/251182847 >> v2:172.21.15.72:3300/0 conn(0x7f716c15e2a0 0x7f716c162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:09.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.834+0000 7f716b7fe640 1 -- 172.21.15.67:0/251182847 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3663845132 0 0) 0x7f7154001110 con 0x7f716c0d07e0 2024-03-24T00:47:09.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.834+0000 7f716b7fe640 1 -- 172.21.15.67:0/251182847 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7158003520 con 0x7f716c0d07e0 2024-03-24T00:47:09.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.834+0000 7f716bfff640 1 -- 172.21.15.67:0/251182847 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f716c15dec0 msgr2=0x7f716c0ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:09.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.834+0000 7f716bfff640 1 --2- 172.21.15.67:0/251182847 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f716c15dec0 0x7f716c0ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:09.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.834+0000 7f716bfff640 1 -- 172.21.15.67:0/251182847 >> v1:172.21.15.67:6789/0 conn(0x7f716c0d07e0 legacy=0x7f716c0b5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:09.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.834+0000 7f716b7fe640 1 -- 172.21.15.67:0/251182847 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4269998112 0 0) 0x7f7158003520 con 0x7f716c0d07e0 2024-03-24T00:47:09.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.834+0000 7f716bfff640 1 -- 172.21.15.67:0/251182847 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f716c0cf080 con 0x7f716c15e2a0 2024-03-24T00:47:09.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.835+0000 7f716bfff640 1 --2- 172.21.15.67:0/251182847 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f716c15e2a0 0x7f716c162630 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f716c0cefb0 tx=0x7f715c02b6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca869e80ee093485 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:09.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.835+0000 7f716b7fe640 1 -- 172.21.15.67:0/251182847 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f715c037070 con 0x7f716c15e2a0 2024-03-24T00:47:09.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.835+0000 7f716b7fe640 1 -- 172.21.15.67:0/251182847 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f715c030510 con 0x7f716c15e2a0 2024-03-24T00:47:09.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.835+0000 7f716b7fe640 1 -- 172.21.15.67:0/251182847 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7154001840 con 0x7f716c15e2a0 2024-03-24T00:47:09.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.836+0000 7f7172cdc640 1 -- 172.21.15.67:0/251182847 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f716c15e2a0 msgr2=0x7f716c162630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:09.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.836+0000 7f7172cdc640 1 --2- 172.21.15.67:0/251182847 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f716c15e2a0 0x7f716c162630 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f716c0cefb0 tx=0x7f715c02b6f0 comp rx=0 tx=0).stop 2024-03-24T00:47:09.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.836+0000 7f7172cdc640 1 -- 172.21.15.67:0/251182847 shutdown_connections 2024-03-24T00:47:09.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.836+0000 7f7172cdc640 1 --2- 172.21.15.67:0/251182847 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f716c15e2a0 0x7f716c162630 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:09.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.836+0000 7f7172cdc640 1 --2- 172.21.15.67:0/251182847 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f716c15dec0 0x7f716c0ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:09.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.836+0000 7f7172cdc640 1 -- 172.21.15.67:0/251182847 >> 172.21.15.67:0/251182847 conn(0x7f716c06c810 msgr2=0x7f716c06cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:09.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.836+0000 7f7172cdc640 1 -- 172.21.15.67:0/251182847 shutdown_connections 2024-03-24T00:47:09.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.836+0000 7f7172cdc640 1 -- 172.21.15.67:0/251182847 wait complete. 2024-03-24T00:47:09.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.837+0000 7f7172cdc640 1 Processor -- start 2024-03-24T00:47:09.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.837+0000 7f7172cdc640 1 -- start start 2024-03-24T00:47:09.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.838+0000 7f7172cdc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f716c0d07e0 0x7f716c156d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:09.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.838+0000 7f7172cdc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f716c15dec0 0x7f716c0b5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:09.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.838+0000 7f7172cdc640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f716c15e2a0 0x7f716c1f7490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:09.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.838+0000 7f7172cdc640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f716c0d3740 con 0x7f716c0d07e0 2024-03-24T00:47:09.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.838+0000 7f7172cdc640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f716c0d35c0 con 0x7f716c15dec0 2024-03-24T00:47:09.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.838+0000 7f7172cdc640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f716c0d38c0 con 0x7f716c15e2a0 2024-03-24T00:47:09.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.838+0000 7f7170a78640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f716c0d07e0 0x7f716c156d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:09.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.838+0000 7f7170a78640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f716c0d07e0 0x7f716c156d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39420/0 (socket says 172.21.15.67:39420) 2024-03-24T00:47:09.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.838+0000 7f7170a78640 1 -- 172.21.15.67:0/1081620531 learned_addr learned my addr 172.21.15.67:0/1081620531 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:09.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.838+0000 7f716bfff640 1 --2- 172.21.15.67:0/1081620531 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f716c15dec0 0x7f716c0b5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:09.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.838+0000 7f7171279640 1 --2- 172.21.15.67:0/1081620531 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f716c15e2a0 0x7f716c1f7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:09.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.838+0000 7f7170a78640 1 -- 172.21.15.67:0/1081620531 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f716c15e2a0 msgr2=0x7f716c1f7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:09.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.838+0000 7f7170a78640 1 --2- 172.21.15.67:0/1081620531 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f716c15e2a0 0x7f716c1f7490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:09.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.838+0000 7f7170a78640 1 -- 172.21.15.67:0/1081620531 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f716c15dec0 msgr2=0x7f716c0b5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:09.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.838+0000 7f7170a78640 1 --2- 172.21.15.67:0/1081620531 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f716c15dec0 0x7f716c0b5bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:09.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.838+0000 7f7170a78640 1 -- 172.21.15.67:0/1081620531 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f716c1572b0 con 0x7f716c0d07e0 2024-03-24T00:47:09.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.838+0000 7f716bfff640 1 --2- 172.21.15.67:0/1081620531 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f716c15dec0 0x7f716c0b5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:47:09.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.838+0000 7f7170a78640 1 --2- 172.21.15.67:0/1081620531 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f716c0d07e0 0x7f716c156d70 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f716400efc0 tx=0x7f716400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:09.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.839+0000 7f71697fa640 1 -- 172.21.15.67:0/1081620531 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7164018020 con 0x7f716c0d07e0 2024-03-24T00:47:09.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.839+0000 7f71697fa640 1 -- 172.21.15.67:0/1081620531 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7164014480 con 0x7f716c0d07e0 2024-03-24T00:47:09.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.839+0000 7f7172cdc640 1 -- 172.21.15.67:0/1081620531 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f716c157560 con 0x7f716c0d07e0 2024-03-24T00:47:09.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.839+0000 7f71697fa640 1 -- 172.21.15.67:0/1081620531 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7164014850 con 0x7f716c0d07e0 2024-03-24T00:47:09.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.839+0000 7f7172cdc640 1 -- 172.21.15.67:0/1081620531 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f716c157e00 con 0x7f716c0d07e0 2024-03-24T00:47:09.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.840+0000 7f7172cdc640 1 -- 172.21.15.67:0/1081620531 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f716c04f400 con 0x7f716c0d07e0 2024-03-24T00:47:09.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.840+0000 7f71697fa640 1 -- 172.21.15.67:0/1081620531 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7164005d10 con 0x7f716c0d07e0 2024-03-24T00:47:09.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.840+0000 7f71697fa640 1 --2- 172.21.15.67:0/1081620531 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f714c06bbd0 0x7f714c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:09.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.840+0000 7f71697fa640 1 -- 172.21.15.67:0/1081620531 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f716408b740 con 0x7f716c0d07e0 2024-03-24T00:47:09.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.841+0000 7f716bfff640 1 --2- 172.21.15.67:0/1081620531 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f714c06bbd0 0x7f714c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:09.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.841+0000 7f716bfff640 1 --2- 172.21.15.67:0/1081620531 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f714c06bbd0 0x7f714c06e090 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f715c034000 tx=0x7f715c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:09.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:09.845+0000 7f71697fa640 1 -- 172.21.15.67:0/1081620531 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7164022030 con 0x7f716c0d07e0 2024-03-24T00:47:10.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:10.097+0000 7f7172cdc640 1 -- 172.21.15.67:0/1081620531 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_inotable_sync"]} v 0) v1 -- 0x7f716c161430 con 0x7f716c0d07e0 2024-03-24T00:47:11.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.099+0000 7f71697fa640 1 -- 172.21.15.67:0/1081620531 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_inotable_sync"]}]=0 v393) v1 ==== 149+0+0 (secure 0 0 0) 0x7f7164010040 con 0x7f716c0d07e0 2024-03-24T00:47:11.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.101+0000 7f7172cdc640 1 -- 172.21.15.67:0/1081620531 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f714c06bbd0 msgr2=0x7f714c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:11.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.101+0000 7f7172cdc640 1 --2- 172.21.15.67:0/1081620531 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f714c06bbd0 0x7f714c06e090 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f715c034000 tx=0x7f715c008030 comp rx=0 tx=0).stop 2024-03-24T00:47:11.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.102+0000 7f7172cdc640 1 -- 172.21.15.67:0/1081620531 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f716c0d07e0 msgr2=0x7f716c156d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:11.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.102+0000 7f7172cdc640 1 --2- 172.21.15.67:0/1081620531 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f716c0d07e0 0x7f716c156d70 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f716400efc0 tx=0x7f716400c490 comp rx=0 tx=0).stop 2024-03-24T00:47:11.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.102+0000 7f7172cdc640 1 -- 172.21.15.67:0/1081620531 shutdown_connections 2024-03-24T00:47:11.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.102+0000 7f7172cdc640 1 --2- 172.21.15.67:0/1081620531 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f714c06bbd0 0x7f714c06e090 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:11.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.102+0000 7f7172cdc640 1 --2- 172.21.15.67:0/1081620531 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f716c15e2a0 0x7f716c1f7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:11.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.102+0000 7f7172cdc640 1 --2- 172.21.15.67:0/1081620531 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f716c15dec0 0x7f716c0b5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:11.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.102+0000 7f7172cdc640 1 --2- 172.21.15.67:0/1081620531 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f716c0d07e0 0x7f716c156d70 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:11.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.102+0000 7f7172cdc640 1 -- 172.21.15.67:0/1081620531 >> 172.21.15.67:0/1081620531 conn(0x7f716c06c810 msgr2=0x7f716c0c9ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:11.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.102+0000 7f7172cdc640 1 -- 172.21.15.67:0/1081620531 shutdown_connections 2024-03-24T00:47:11.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.102+0000 7f7172cdc640 1 -- 172.21.15.67:0/1081620531 wait complete. 2024-03-24T00:47:11.116 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-24T00:47:11.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.226+0000 7f4311626640 1 Processor -- start 2024-03-24T00:47:11.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.226+0000 7f4311626640 1 -- start start 2024-03-24T00:47:11.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.226+0000 7f4311626640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f430c145540 0x7f430c160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:11.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.226+0000 7f4311626640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f430c06dcf0 con 0x7f430c071af0 2024-03-24T00:47:11.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.226+0000 7f4311626640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f430c06dfd0 con 0x7f430c15c2a0 2024-03-24T00:47:11.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.226+0000 7f4311626640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f430c06e620 con 0x7f430c145540 2024-03-24T00:47:11.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.227+0000 7f430b7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f430c071af0 0x7f430c071ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46828/0 (socket says 172.21.15.67:46828) 2024-03-24T00:47:11.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.227+0000 7f430b7fe640 1 -- 172.21.15.67:0/580973636 learned_addr learned my addr 172.21.15.67:0/580973636 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:47:11.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.227+0000 7f430affd640 1 --2- 172.21.15.67:0/580973636 >> v2:172.21.15.72:3300/0 conn(0x7f430c145540 0x7f430c160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:11.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.227+0000 7f4309ffb640 1 -- 172.21.15.67:0/580973636 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2275872427 0 0) 0x7f430c06dcf0 con 0x7f430c071af0 2024-03-24T00:47:11.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.227+0000 7f4309ffb640 1 -- 172.21.15.67:0/580973636 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42f4003520 con 0x7f430c071af0 2024-03-24T00:47:11.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.227+0000 7f4309ffb640 1 -- 172.21.15.67:0/580973636 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 149322102 0 0) 0x7f42f4003520 con 0x7f430c071af0 2024-03-24T00:47:11.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.227+0000 7f4309ffb640 1 -- 172.21.15.67:0/580973636 >> v2:172.21.15.72:3300/0 conn(0x7f430c145540 msgr2=0x7f430c160420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:11.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.227+0000 7f4309ffb640 1 --2- 172.21.15.67:0/580973636 >> v2:172.21.15.72:3300/0 conn(0x7f430c145540 0x7f430c160420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:11.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.227+0000 7f4309ffb640 1 -- 172.21.15.67:0/580973636 >> v1:172.21.15.72:6789/0 conn(0x7f430c15c2a0 legacy=0x7f430c0cef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:11.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.227+0000 7f4309ffb640 1 -- 172.21.15.67:0/580973636 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f430c06e2b0 con 0x7f430c071af0 2024-03-24T00:47:11.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.227+0000 7f4309ffb640 1 -- 172.21.15.67:0/580973636 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f43040019f0 con 0x7f430c071af0 2024-03-24T00:47:11.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.227+0000 7f4309ffb640 1 -- 172.21.15.67:0/580973636 >> v1:172.21.15.67:6789/0 conn(0x7f430c071af0 legacy=0x7f430c071ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:11.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.227+0000 7f4309ffb640 1 --2- 172.21.15.67:0/580973636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42f40039b0 0x7f42f4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:11.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.227+0000 7f4309ffb640 1 --2- 172.21.15.67:0/580973636 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f42f40043b0 0x7f42f4004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:11.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.227+0000 7f4309ffb640 1 --2- 172.21.15.67:0/580973636 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f42f4005080 0x7f42f400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:11.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.227+0000 7f4309ffb640 1 -- 172.21.15.67:0/580973636 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f430c06e620 con 0x7f42f40039b0 2024-03-24T00:47:11.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.227+0000 7f4309ffb640 1 -- 172.21.15.67:0/580973636 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f400bfe0 con 0x7f42f40043b0 2024-03-24T00:47:11.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.227+0000 7f4309ffb640 1 -- 172.21.15.67:0/580973636 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42f400c1e0 con 0x7f42f4005080 2024-03-24T00:47:11.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.227+0000 7f430affd640 1 --2- 172.21.15.67:0/580973636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42f40039b0 0x7f42f4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:11.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.228+0000 7f430b7fe640 1 --2- 172.21.15.67:0/580973636 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f42f4005080 0x7f42f400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:11.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.228+0000 7f430a7fc640 1 --2- 172.21.15.67:0/580973636 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f42f40043b0 0x7f42f4004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:11.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.228+0000 7f430affd640 1 -- 172.21.15.67:0/580973636 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f42f4005080 msgr2=0x7f42f400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:11.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.228+0000 7f430affd640 1 --2- 172.21.15.67:0/580973636 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f42f4005080 0x7f42f400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:11.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.228+0000 7f430affd640 1 -- 172.21.15.67:0/580973636 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f42f40043b0 msgr2=0x7f42f4004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:11.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.228+0000 7f430affd640 1 --2- 172.21.15.67:0/580973636 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f42f40043b0 0x7f42f4004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:11.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.228+0000 7f430affd640 1 -- 172.21.15.67:0/580973636 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42f4003520 con 0x7f42f40039b0 2024-03-24T00:47:11.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.228+0000 7f430b7fe640 1 --2- 172.21.15.67:0/580973636 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f42f4005080 0x7f42f400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:47:11.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.228+0000 7f430b7fe640 1 -- 172.21.15.67:0/580973636 reap_dead start 2024-03-24T00:47:11.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.228+0000 7f430affd640 1 --2- 172.21.15.67:0/580973636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42f40039b0 0x7f42f4003d90 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f42fc009650 tx=0x7f42f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4046c321ca4fd4b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:11.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.229+0000 7f4309ffb640 1 -- 172.21.15.67:0/580973636 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42fc03d070 con 0x7f42f40039b0 2024-03-24T00:47:11.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.229+0000 7f4309ffb640 1 -- 172.21.15.67:0/580973636 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42fc030ac0 con 0x7f42f40039b0 2024-03-24T00:47:11.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.229+0000 7f4309ffb640 1 -- 172.21.15.67:0/580973636 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42fc009690 con 0x7f42f40039b0 2024-03-24T00:47:11.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.229+0000 7f4311626640 1 -- 172.21.15.67:0/580973636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42f40039b0 msgr2=0x7f42f4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:11.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.229+0000 7f4311626640 1 --2- 172.21.15.67:0/580973636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42f40039b0 0x7f42f4003d90 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f42fc009650 tx=0x7f42f4005830 comp rx=0 tx=0).stop 2024-03-24T00:47:11.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.229+0000 7f4311626640 1 -- 172.21.15.67:0/580973636 shutdown_connections 2024-03-24T00:47:11.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.229+0000 7f4311626640 1 --2- 172.21.15.67:0/580973636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42f40039b0 0x7f42f4003d90 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:11.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.229+0000 7f4311626640 1 -- 172.21.15.67:0/580973636 >> 172.21.15.67:0/580973636 conn(0x7f430c0cf9e0 msgr2=0x7f430c0cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:11.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.229+0000 7f4311626640 1 -- 172.21.15.67:0/580973636 shutdown_connections 2024-03-24T00:47:11.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.229+0000 7f4311626640 1 -- 172.21.15.67:0/580973636 wait complete. 2024-03-24T00:47:11.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.230+0000 7f4311626640 1 Processor -- start 2024-03-24T00:47:11.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.231+0000 7f4311626640 1 -- start start 2024-03-24T00:47:11.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.231+0000 7f4311626640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42f40039b0 0x7f430c15b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:11.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.231+0000 7f4311626640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f430c15d0c0 0x7f430c159c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:11.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.231+0000 7f4311626640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f430c15a170 0x7f430c135a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:11.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.231+0000 7f4311626640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f430c1605a0 con 0x7f42f40039b0 2024-03-24T00:47:11.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.231+0000 7f4311626640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f430c160420 con 0x7f430c15d0c0 2024-03-24T00:47:11.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.231+0000 7f4311626640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f430c160720 con 0x7f430c15a170 2024-03-24T00:47:11.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.231+0000 7f430affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42f40039b0 0x7f430c15b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:11.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.231+0000 7f430affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42f40039b0 0x7f430c15b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39434/0 (socket says 172.21.15.67:39434) 2024-03-24T00:47:11.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.231+0000 7f430affd640 1 -- 172.21.15.67:0/4043643043 learned_addr learned my addr 172.21.15.67:0/4043643043 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:11.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.232+0000 7f430a7fc640 1 --2- 172.21.15.67:0/4043643043 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f430c15d0c0 0x7f430c159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:11.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.232+0000 7f430affd640 1 -- 172.21.15.67:0/4043643043 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f430c15a170 msgr2=0x7f430c135a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:11.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.232+0000 7f430b7fe640 1 --2- 172.21.15.67:0/4043643043 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f430c15a170 0x7f430c135a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:11.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.232+0000 7f430affd640 1 --2- 172.21.15.67:0/4043643043 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f430c15a170 0x7f430c135a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:11.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.232+0000 7f430affd640 1 -- 172.21.15.67:0/4043643043 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f430c15d0c0 msgr2=0x7f430c159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:11.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.232+0000 7f430affd640 1 --2- 172.21.15.67:0/4043643043 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f430c15d0c0 0x7f430c159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:11.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.232+0000 7f430affd640 1 -- 172.21.15.67:0/4043643043 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f430c1349e0 con 0x7f42f40039b0 2024-03-24T00:47:11.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.232+0000 7f430a7fc640 1 --2- 172.21.15.67:0/4043643043 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f430c15d0c0 0x7f430c159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:47:11.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.232+0000 7f430affd640 1 --2- 172.21.15.67:0/4043643043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42f40039b0 0x7f430c15b900 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f42fc03b040 tx=0x7f42fc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:11.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.232+0000 7f42f37fe640 1 -- 172.21.15.67:0/4043643043 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42fc03d050 con 0x7f42f40039b0 2024-03-24T00:47:11.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.232+0000 7f42f37fe640 1 -- 172.21.15.67:0/4043643043 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42fc049070 con 0x7f42f40039b0 2024-03-24T00:47:11.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.232+0000 7f4311626640 1 -- 172.21.15.67:0/4043643043 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f430c134cc0 con 0x7f42f40039b0 2024-03-24T00:47:11.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.232+0000 7f42f37fe640 1 -- 172.21.15.67:0/4043643043 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42fc003c40 con 0x7f42f40039b0 2024-03-24T00:47:11.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.232+0000 7f4311626640 1 -- 172.21.15.67:0/4043643043 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f430c135280 con 0x7f42f40039b0 2024-03-24T00:47:11.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.233+0000 7f4311626640 1 -- 172.21.15.67:0/4043643043 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f430c04f400 con 0x7f42f40039b0 2024-03-24T00:47:11.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.234+0000 7f42f37fe640 1 -- 172.21.15.67:0/4043643043 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f42fc044450 con 0x7f42f40039b0 2024-03-24T00:47:11.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.234+0000 7f42f37fe640 1 --2- 172.21.15.67:0/4043643043 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f42f806bbd0 0x7f42f806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:11.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.234+0000 7f42f37fe640 1 -- 172.21.15.67:0/4043643043 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f42fc0b0850 con 0x7f42f40039b0 2024-03-24T00:47:11.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.234+0000 7f430a7fc640 1 --2- 172.21.15.67:0/4043643043 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f42f806bbd0 0x7f42f806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:11.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.235+0000 7f430a7fc640 1 --2- 172.21.15.67:0/4043643043 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f42f806bbd0 0x7f42f806e090 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f430c140bc0 tx=0x7f4300007b70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:11.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.238+0000 7f42f37fe640 1 -- 172.21.15.67:0/4043643043 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f42fc052020 con 0x7f42f40039b0 2024-03-24T00:47:11.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.422+0000 7f4311626640 1 -- 172.21.15.67:0/4043643043 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f430c0d0740 con 0x7f42f40039b0 2024-03-24T00:47:11.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.425+0000 7f42f37fe640 1 -- 172.21.15.67:0/4043643043 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v20)=0 v20) v1 ==== 122+0+0 (secure 0 0 0) 0x7f42fc07f490 con 0x7f42f40039b0 2024-03-24T00:47:11.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.428+0000 7f4311626640 1 -- 172.21.15.67:0/4043643043 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f42f806bbd0 msgr2=0x7f42f806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:11.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.428+0000 7f4311626640 1 --2- 172.21.15.67:0/4043643043 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f42f806bbd0 0x7f42f806e090 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f430c140bc0 tx=0x7f4300007b70 comp rx=0 tx=0).stop 2024-03-24T00:47:11.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.428+0000 7f4311626640 1 -- 172.21.15.67:0/4043643043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42f40039b0 msgr2=0x7f430c15b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:11.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.428+0000 7f4311626640 1 --2- 172.21.15.67:0/4043643043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42f40039b0 0x7f430c15b900 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f42fc03b040 tx=0x7f42fc004730 comp rx=0 tx=0).stop 2024-03-24T00:47:11.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.428+0000 7f4311626640 1 -- 172.21.15.67:0/4043643043 shutdown_connections 2024-03-24T00:47:11.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.428+0000 7f4311626640 1 --2- 172.21.15.67:0/4043643043 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f42f806bbd0 0x7f42f806e090 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:11.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.428+0000 7f4311626640 1 --2- 172.21.15.67:0/4043643043 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f430c15a170 0x7f430c135a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:11.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.428+0000 7f4311626640 1 --2- 172.21.15.67:0/4043643043 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f430c15d0c0 0x7f430c159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:11.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.428+0000 7f4311626640 1 --2- 172.21.15.67:0/4043643043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42f40039b0 0x7f430c15b900 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:11.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.428+0000 7f4311626640 1 -- 172.21.15.67:0/4043643043 >> 172.21.15.67:0/4043643043 conn(0x7f430c0cf9e0 msgr2=0x7f430c071ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:11.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.428+0000 7f4311626640 1 -- 172.21.15.67:0/4043643043 shutdown_connections 2024-03-24T00:47:11.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.428+0000 7f4311626640 1 -- 172.21.15.67:0/4043643043 wait complete. 2024-03-24T00:47:11.439 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-24T00:47:11.440 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:47:11.479 DEBUG:teuthology.orchestra.run.smithi072:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:47:11.527 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-24T00:47:11.527 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:47:11.527 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:47:11.542 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056375,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15771 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T00:47:11.542 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:47:11.543 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:47:11.560 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17590 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248808k,nr_inodes=812202,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:47:11.560 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:47:11.560 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:47:11.596 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056375,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15771 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T00:47:11.597 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:47:11.597 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:47:11.618 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17590 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248808k,nr_inodes=812202,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:47:11.618 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-24T00:47:11.618 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:47:11.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.737+0000 7fed7e1b9640 1 Processor -- start 2024-03-24T00:47:11.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.737+0000 7fed7e1b9640 1 -- start start 2024-03-24T00:47:11.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.738+0000 7fed7e1b9640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fed780ccce0 0x7fed780cd0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:11.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.738+0000 7fed7e1b9640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed781606d0 con 0x7fed780cef40 2024-03-24T00:47:11.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.738+0000 7fed7e1b9640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed780cd600 con 0x7fed7806ecf0 2024-03-24T00:47:11.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.738+0000 7fed7e1b9640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed78072890 con 0x7fed780ccce0 2024-03-24T00:47:11.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.738+0000 7fed76ffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fed780cef40 0x7fed7815f8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46838/0 (socket says 172.21.15.67:46838) 2024-03-24T00:47:11.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.738+0000 7fed76ffd640 1 -- 172.21.15.67:0/807738430 learned_addr learned my addr 172.21.15.67:0/807738430 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:47:11.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.738+0000 7fed777fe640 1 --2- 172.21.15.67:0/807738430 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fed780ccce0 0x7fed780cd0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:11.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.738+0000 7fed767fc640 1 -- 172.21.15.67:0/807738430 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 463950078 0 0) 0x7fed781606d0 con 0x7fed780cef40 2024-03-24T00:47:11.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.738+0000 7fed767fc640 1 -- 172.21.15.67:0/807738430 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed60003520 con 0x7fed780cef40 2024-03-24T00:47:11.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.738+0000 7fed767fc640 1 -- 172.21.15.67:0/807738430 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1915698934 0 0) 0x7fed780cd600 con 0x7fed7806ecf0 2024-03-24T00:47:11.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.738+0000 7fed767fc640 1 -- 172.21.15.67:0/807738430 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed781606d0 con 0x7fed7806ecf0 2024-03-24T00:47:11.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.738+0000 7fed767fc640 1 -- 172.21.15.67:0/807738430 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1133012363 0 0) 0x7fed60003520 con 0x7fed780cef40 2024-03-24T00:47:11.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.738+0000 7fed777fe640 1 -- 172.21.15.67:0/807738430 >> v1:172.21.15.72:6789/0 conn(0x7fed7806ecf0 legacy=0x7fed7815ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:11.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.738+0000 7fed777fe640 1 -- 172.21.15.67:0/807738430 >> v1:172.21.15.67:6789/0 conn(0x7fed780cef40 legacy=0x7fed7815f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:11.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.738+0000 7fed777fe640 1 -- 172.21.15.67:0/807738430 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed7815c580 con 0x7fed780ccce0 2024-03-24T00:47:11.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.739+0000 7fed777fe640 1 --2- 172.21.15.67:0/807738430 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fed780ccce0 0x7fed780cd0c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fed680097d0 tx=0x7fed6802b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6fa67446f39fac4b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:11.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.739+0000 7fed767fc640 1 -- 172.21.15.67:0/807738430 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed68037070 con 0x7fed780ccce0 2024-03-24T00:47:11.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.739+0000 7fed767fc640 1 -- 172.21.15.67:0/807738430 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed68030510 con 0x7fed780ccce0 2024-03-24T00:47:11.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.739+0000 7fed767fc640 1 -- 172.21.15.67:0/807738430 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6c001670 con 0x7fed780ccce0 2024-03-24T00:47:11.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.740+0000 7fed7e1b9640 1 -- 172.21.15.67:0/807738430 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fed780ccce0 msgr2=0x7fed780cd0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:11.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.740+0000 7fed7e1b9640 1 --2- 172.21.15.67:0/807738430 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fed780ccce0 0x7fed780cd0c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fed680097d0 tx=0x7fed6802b800 comp rx=0 tx=0).stop 2024-03-24T00:47:11.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.740+0000 7fed7e1b9640 1 -- 172.21.15.67:0/807738430 shutdown_connections 2024-03-24T00:47:11.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.740+0000 7fed7e1b9640 1 --2- 172.21.15.67:0/807738430 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fed780ccce0 0x7fed780cd0c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:11.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.740+0000 7fed7e1b9640 1 -- 172.21.15.67:0/807738430 >> 172.21.15.67:0/807738430 conn(0x7fed780cf9e0 msgr2=0x7fed780cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:11.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.740+0000 7fed7e1b9640 1 -- 172.21.15.67:0/807738430 shutdown_connections 2024-03-24T00:47:11.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.740+0000 7fed7e1b9640 1 -- 172.21.15.67:0/807738430 wait complete. 2024-03-24T00:47:11.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.741+0000 7fed7e1b9640 1 Processor -- start 2024-03-24T00:47:11.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.741+0000 7fed7e1b9640 1 -- start start 2024-03-24T00:47:11.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.742+0000 7fed7e1b9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fed7806ecf0 0x7fed781f5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:11.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.742+0000 7fed7e1b9640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fed780ccce0 0x7fed7815ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:11.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.742+0000 7fed7e1b9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed780cef40 0x7fed780ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:11.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.742+0000 7fed7e1b9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed7815ce20 con 0x7fed780cef40 2024-03-24T00:47:11.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.742+0000 7fed7e1b9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed7815cb40 con 0x7fed7806ecf0 2024-03-24T00:47:11.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.742+0000 7fed7e1b9640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed7815c290 con 0x7fed780ccce0 2024-03-24T00:47:11.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.742+0000 7fed77fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed780cef40 0x7fed780ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:11.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.742+0000 7fed77fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed780cef40 0x7fed780ae8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39446/0 (socket says 172.21.15.67:39446) 2024-03-24T00:47:11.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.742+0000 7fed77fff640 1 -- 172.21.15.67:0/248739521 learned_addr learned my addr 172.21.15.67:0/248739521 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:11.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.742+0000 7fed777fe640 1 --2- 172.21.15.67:0/248739521 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fed7806ecf0 0x7fed781f5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:11.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.742+0000 7fed76ffd640 1 --2- 172.21.15.67:0/248739521 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fed780ccce0 0x7fed7815ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:11.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.742+0000 7fed77fff640 1 -- 172.21.15.67:0/248739521 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fed780ccce0 msgr2=0x7fed7815ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:11.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.742+0000 7fed77fff640 1 --2- 172.21.15.67:0/248739521 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fed780ccce0 0x7fed7815ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:11.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.742+0000 7fed77fff640 1 -- 172.21.15.67:0/248739521 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fed7806ecf0 msgr2=0x7fed781f5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:11.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.742+0000 7fed77fff640 1 --2- 172.21.15.67:0/248739521 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fed7806ecf0 0x7fed781f5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:11.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.742+0000 7fed77fff640 1 -- 172.21.15.67:0/248739521 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed781f64e0 con 0x7fed780cef40 2024-03-24T00:47:11.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.742+0000 7fed777fe640 1 --2- 172.21.15.67:0/248739521 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fed7806ecf0 0x7fed781f5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:47:11.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.742+0000 7fed76ffd640 1 --2- 172.21.15.67:0/248739521 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fed780ccce0 0x7fed7815ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:47:11.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.743+0000 7fed77fff640 1 --2- 172.21.15.67:0/248739521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed780cef40 0x7fed780ae8e0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7fed7000c7a0 tx=0x7fed7000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:11.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.743+0000 7fed5ffff640 1 -- 172.21.15.67:0/248739521 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed70003b70 con 0x7fed780cef40 2024-03-24T00:47:11.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.743+0000 7fed5ffff640 1 -- 172.21.15.67:0/248739521 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed70003d10 con 0x7fed780cef40 2024-03-24T00:47:11.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.743+0000 7fed5ffff640 1 -- 172.21.15.67:0/248739521 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7000aa90 con 0x7fed780cef40 2024-03-24T00:47:11.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.743+0000 7fed7e1b9640 1 -- 172.21.15.67:0/248739521 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed7815b070 con 0x7fed780cef40 2024-03-24T00:47:11.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.743+0000 7fed7e1b9640 1 -- 172.21.15.67:0/248739521 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed781e5c20 con 0x7fed780cef40 2024-03-24T00:47:11.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.744+0000 7fed7e1b9640 1 -- 172.21.15.67:0/248739521 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed781604f0 con 0x7fed780cef40 2024-03-24T00:47:11.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.744+0000 7fed5ffff640 1 -- 172.21.15.67:0/248739521 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fed7000ac30 con 0x7fed780cef40 2024-03-24T00:47:11.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.745+0000 7fed5ffff640 1 --2- 172.21.15.67:0/248739521 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fed6406bbd0 0x7fed6406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:11.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.745+0000 7fed5ffff640 1 -- 172.21.15.67:0/248739521 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5507+0+0 (secure 0 0 0) 0x7fed7008a940 con 0x7fed780cef40 2024-03-24T00:47:11.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.745+0000 7fed777fe640 1 --2- 172.21.15.67:0/248739521 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fed6406bbd0 0x7fed6406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:11.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.745+0000 7fed777fe640 1 --2- 172.21.15.67:0/248739521 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fed6406bbd0 0x7fed6406e090 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fed68034000 tx=0x7fed68008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:11.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.750+0000 7fed5ffff640 1 -- 172.21.15.67:0/248739521 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fed781604f0 con 0x7fed780cef40 2024-03-24T00:47:11.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.968+0000 7fed7e1b9640 1 -- 172.21.15.67:0/248739521 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed7804f400 con 0x7fed780cef40 2024-03-24T00:47:11.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.968+0000 7fed5ffff640 1 -- 172.21.15.67:0/248739521 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+3696 (secure 0 0 0) 0x7fed781f64e0 con 0x7fed780cef40 2024-03-24T00:47:11.969 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:47:11.969 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":68,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6359,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2664460392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2664460392},{"type":"v1","addr":"172.21.15.67:6835","nonce":2664460392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6368,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/3323873506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3323873506},{"type":"v1","addr":"172.21.15.72:6837","nonce":3323873506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":7150,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/2744016217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2744016217},{"type":"v1","addr":"172.21.15.72:6836","nonce":2744016217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":8472,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/820406445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":820406445},{"type":"v1","addr":"172.21.15.67:6833","nonce":820406445}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68}],"filesystems":[]} 2024-03-24T00:47:11.969 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 68 2024-03-24T00:47:11.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.970+0000 7fed7e1b9640 1 -- 172.21.15.67:0/248739521 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fed6406bbd0 msgr2=0x7fed6406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:11.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.970+0000 7fed7e1b9640 1 --2- 172.21.15.67:0/248739521 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fed6406bbd0 0x7fed6406e090 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fed68034000 tx=0x7fed68008030 comp rx=0 tx=0).stop 2024-03-24T00:47:11.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.970+0000 7fed7e1b9640 1 -- 172.21.15.67:0/248739521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed780cef40 msgr2=0x7fed780ae8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:11.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.970+0000 7fed7e1b9640 1 --2- 172.21.15.67:0/248739521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed780cef40 0x7fed780ae8e0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7fed7000c7a0 tx=0x7fed7000cc60 comp rx=0 tx=0).stop 2024-03-24T00:47:11.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.970+0000 7fed7e1b9640 1 -- 172.21.15.67:0/248739521 shutdown_connections 2024-03-24T00:47:11.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.970+0000 7fed7e1b9640 1 --2- 172.21.15.67:0/248739521 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fed6406bbd0 0x7fed6406e090 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:11.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.970+0000 7fed7e1b9640 1 --2- 172.21.15.67:0/248739521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed780cef40 0x7fed780ae8e0 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:11.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.970+0000 7fed7e1b9640 1 --2- 172.21.15.67:0/248739521 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fed780ccce0 0x7fed7815ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:11.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.970+0000 7fed7e1b9640 1 --2- 172.21.15.67:0/248739521 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fed7806ecf0 0x7fed781f5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:11.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.970+0000 7fed7e1b9640 1 -- 172.21.15.67:0/248739521 >> 172.21.15.67:0/248739521 conn(0x7fed780cf9e0 msgr2=0x7fed780b0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:11.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.970+0000 7fed7e1b9640 1 -- 172.21.15.67:0/248739521 shutdown_connections 2024-03-24T00:47:11.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:11.970+0000 7fed7e1b9640 1 -- 172.21.15.67:0/248739521 wait complete. 2024-03-24T00:47:11.982 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-24T00:47:11.982 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-24T00:47:11.982 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-24T00:47:11.982 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:47:11.983 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-24T00:47:11.983 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-24T00:47:11.983 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-24T00:47:11.983 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:47:11.983 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-24T00:47:11.983 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-24T00:47:11.983 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-24T00:47:11.983 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:47:11.983 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-24T00:47:11.984 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-24T00:47:11.984 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-24T00:47:11.984 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:47:11.991 INFO:tasks.ceph.mds.a:Stopped 2024-03-24T00:47:11.992 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-24T00:47:11.992 INFO:tasks.ceph.mds.c:Stopped 2024-03-24T00:47:11.993 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-24T00:47:11.995 INFO:tasks.ceph.mds.b:Stopped 2024-03-24T00:47:11.995 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-24T00:47:11.996 INFO:tasks.ceph.mds.d:Stopped 2024-03-24T00:47:11.996 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-24T00:47:12.034 INFO:tasks.ceph.mds.a:Started 2024-03-24T00:47:12.034 DEBUG:teuthology.parallel:result is None 2024-03-24T00:47:12.034 INFO:tasks.ceph.mds.c:Started 2024-03-24T00:47:12.035 DEBUG:teuthology.parallel:result is None 2024-03-24T00:47:12.037 INFO:tasks.ceph.mds.b:Started 2024-03-24T00:47:12.038 DEBUG:teuthology.parallel:result is None 2024-03-24T00:47:12.038 INFO:tasks.ceph.mds.d:Started 2024-03-24T00:47:12.038 DEBUG:teuthology.parallel:result is None 2024-03-24T00:47:12.039 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-24T00:47:12.110 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-03-24T00:47:12.111 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-03-24T00:47:12.121 INFO:tasks.ceph.mds.d.smithi072.stdout:starting mds.d at 2024-03-24T00:47:12.121 INFO:tasks.ceph.mds.b.smithi072.stdout:starting mds.b at 2024-03-24T00:47:12.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.146+0000 7f19fcbd4640 1 Processor -- start 2024-03-24T00:47:12.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.146+0000 7f19fcbd4640 1 -- start start 2024-03-24T00:47:12.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.146+0000 7f19fcbd4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f19f815e0f0 0x7f19f80b95b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:12.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.146+0000 7f19fcbd4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f19f80b9af0 0x7f19f80d32d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:12.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.147+0000 7f19fcbd4640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19f80d2230 con 0x7f19f80d3810 2024-03-24T00:47:12.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.147+0000 7f19f659c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f19f815e0f0 0x7f19f80b95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:12.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.147+0000 7f19f6d9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f19f80d3810 0x7f19f80b6dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46864/0 (socket says 172.21.15.67:46864) 2024-03-24T00:47:12.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.147+0000 7f19f6d9d640 1 -- 172.21.15.67:0/4176125093 learned_addr learned my addr 172.21.15.67:0/4176125093 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:47:12.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.147+0000 7f19fcbd4640 1 -- 172.21.15.67:0/4176125093 --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19f80d2510 con 0x7f19f815e0f0 2024-03-24T00:47:12.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.147+0000 7f19fcbd4640 1 -- 172.21.15.67:0/4176125093 --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19f80d27f0 con 0x7f19f80b9af0 2024-03-24T00:47:12.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.147+0000 7f19f5d9b640 1 --2- 172.21.15.67:0/4176125093 >> v2:172.21.15.72:3300/0 conn(0x7f19f80b9af0 0x7f19f80d32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:12.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.147+0000 7f19f659c640 1 -- 172.21.15.67:0/4176125093 >> v2:172.21.15.72:3300/0 conn(0x7f19f80b9af0 msgr2=0x7f19f80d32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:12.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.147+0000 7f19f659c640 1 --2- 172.21.15.67:0/4176125093 >> v2:172.21.15.72:3300/0 conn(0x7f19f80b9af0 0x7f19f80d32d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:12.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.147+0000 7f19f659c640 1 -- 172.21.15.67:0/4176125093 >> v1:172.21.15.67:6789/0 conn(0x7f19f80d3810 legacy=0x7f19f80b6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:12.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.147+0000 7f19f659c640 1 -- 172.21.15.67:0/4176125093 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f80d2ad0 con 0x7f19f815e0f0 2024-03-24T00:47:12.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.147+0000 7f19f559a640 1 -- 172.21.15.67:0/4176125093 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 984473734 0 0) 0x7f19f80d2230 con 0x7f19f80d3810 2024-03-24T00:47:12.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.148+0000 7f19f659c640 1 --2- 172.21.15.67:0/4176125093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19f815e0f0 0x7f19f80b95b0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f19f80cacc0 tx=0x7f19e802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99a92305f7d29ae5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:12.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.148+0000 7f19f559a640 1 -- 172.21.15.67:0/4176125093 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e8035020 con 0x7f19f815e0f0 2024-03-24T00:47:12.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.148+0000 7f19f559a640 1 -- 172.21.15.67:0/4176125093 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19e8031830 con 0x7f19f815e0f0 2024-03-24T00:47:12.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.148+0000 7f19f559a640 1 -- 172.21.15.67:0/4176125093 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e8031c00 con 0x7f19f815e0f0 2024-03-24T00:47:12.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.148+0000 7f19fcbd4640 1 -- 172.21.15.67:0/4176125093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19f815e0f0 msgr2=0x7f19f80b95b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:12.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.148+0000 7f19fcbd4640 1 --2- 172.21.15.67:0/4176125093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19f815e0f0 0x7f19f80b95b0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f19f80cacc0 tx=0x7f19e802b8d0 comp rx=0 tx=0).stop 2024-03-24T00:47:12.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.148+0000 7f19fcbd4640 1 -- 172.21.15.67:0/4176125093 shutdown_connections 2024-03-24T00:47:12.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.148+0000 7f19fcbd4640 1 --2- 172.21.15.67:0/4176125093 >> v2:172.21.15.72:3300/0 conn(0x7f19f80b9af0 0x7f19f80d32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:12.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.148+0000 7f19fcbd4640 1 --2- 172.21.15.67:0/4176125093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19f815e0f0 0x7f19f80b95b0 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:12.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.148+0000 7f19fcbd4640 1 -- 172.21.15.67:0/4176125093 >> 172.21.15.67:0/4176125093 conn(0x7f19f8072b60 msgr2=0x7f19f8072f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:12.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.148+0000 7f19fcbd4640 1 -- 172.21.15.67:0/4176125093 shutdown_connections 2024-03-24T00:47:12.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.149+0000 7f19fcbd4640 1 -- 172.21.15.67:0/4176125093 wait complete. 2024-03-24T00:47:12.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.150+0000 7f19fcbd4640 1 Processor -- start 2024-03-24T00:47:12.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.150+0000 7f19fcbd4640 1 -- start start 2024-03-24T00:47:12.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.150+0000 7f19fcbd4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19f80b9af0 0x7f19f80cc640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:12.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.150+0000 7f19fcbd4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f19f80d3810 0x7f19f80ccb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:12.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.150+0000 7f19fcbd4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19f815e0f0 0x7f19f81f3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:12.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.150+0000 7f19fcbd4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f80b3350 con 0x7f19f80b9af0 2024-03-24T00:47:12.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.150+0000 7f19fcbd4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f80b31d0 con 0x7f19f815e0f0 2024-03-24T00:47:12.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.150+0000 7f19fcbd4640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19f80b74d0 con 0x7f19f80d3810 2024-03-24T00:47:12.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.151+0000 7f19f659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19f80b9af0 0x7f19f80cc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:12.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.151+0000 7f19f659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19f80b9af0 0x7f19f80cc640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39470/0 (socket says 172.21.15.67:39470) 2024-03-24T00:47:12.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.151+0000 7f19f659c640 1 -- 172.21.15.67:0/2863332184 learned_addr learned my addr 172.21.15.67:0/2863332184 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:12.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.151+0000 7f19f6d9d640 1 --2- 172.21.15.67:0/2863332184 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19f815e0f0 0x7f19f81f3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:12.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.151+0000 7f19f659c640 1 -- 172.21.15.67:0/2863332184 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f19f80d3810 msgr2=0x7f19f80ccb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:12.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.151+0000 7f19f659c640 1 --2- 172.21.15.67:0/2863332184 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f19f80d3810 0x7f19f80ccb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:12.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.151+0000 7f19f659c640 1 -- 172.21.15.67:0/2863332184 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19f815e0f0 msgr2=0x7f19f81f3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:12.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.151+0000 7f19f659c640 1 --2- 172.21.15.67:0/2863332184 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19f815e0f0 0x7f19f81f3680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:12.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.151+0000 7f19f659c640 1 -- 172.21.15.67:0/2863332184 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f80cd270 con 0x7f19f80b9af0 2024-03-24T00:47:12.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.151+0000 7f19f659c640 1 --2- 172.21.15.67:0/2863332184 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19f80b9af0 0x7f19f80cc640 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f19e8009890 tx=0x7f19e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:12.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.151+0000 7f19eeffd640 1 -- 172.21.15.67:0/2863332184 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e8040070 con 0x7f19f80b9af0 2024-03-24T00:47:12.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.151+0000 7f19eeffd640 1 -- 172.21.15.67:0/2863332184 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19e803a8f0 con 0x7f19f80b9af0 2024-03-24T00:47:12.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.151+0000 7f19eeffd640 1 -- 172.21.15.67:0/2863332184 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e8035040 con 0x7f19f80b9af0 2024-03-24T00:47:12.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.151+0000 7f19fcbd4640 1 -- 172.21.15.67:0/2863332184 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19f80cd550 con 0x7f19f80b9af0 2024-03-24T00:47:12.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.151+0000 7f19fcbd4640 1 -- 172.21.15.67:0/2863332184 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19f8160f30 con 0x7f19f80b9af0 2024-03-24T00:47:12.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.152+0000 7f19fcbd4640 1 -- 172.21.15.67:0/2863332184 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19f804f400 con 0x7f19f80b9af0 2024-03-24T00:47:12.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.153+0000 7f19eeffd640 1 -- 172.21.15.67:0/2863332184 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f19e8038020 con 0x7f19f80b9af0 2024-03-24T00:47:12.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.153+0000 7f19eeffd640 1 --2- 172.21.15.67:0/2863332184 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f19d806bbd0 0x7f19d806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:12.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.153+0000 7f19eeffd640 1 -- 172.21.15.67:0/2863332184 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f19e80ac2f0 con 0x7f19f80b9af0 2024-03-24T00:47:12.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.153+0000 7f19f5d9b640 1 --2- 172.21.15.67:0/2863332184 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f19d806bbd0 0x7f19d806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:12.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.154+0000 7f19f5d9b640 1 --2- 172.21.15.67:0/2863332184 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f19d806bbd0 0x7f19d806e090 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f19e0006e20 tx=0x7f19e0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:12.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.157+0000 7f19eeffd640 1 -- 172.21.15.67:0/2863332184 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f19e807aea0 con 0x7f19f80b9af0 2024-03-24T00:47:12.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:12.352+0000 7f19fcbd4640 1 -- 172.21.15.67:0/2863332184 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f19f815e4d0 con 0x7f19f80b9af0 2024-03-24T00:47:13.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.448+0000 7f19eeffd640 1 -- 172.21.15.67:0/2863332184 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v101) v1 ==== 97+0+0 (secure 0 0 0) 0x7f19e807a7a0 con 0x7f19f80b9af0 2024-03-24T00:47:13.448 INFO:teuthology.orchestra.run.smithi067.stderr: removed all blocklist entries 2024-03-24T00:47:13.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.451+0000 7f19fcbd4640 1 -- 172.21.15.67:0/2863332184 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f19d806bbd0 msgr2=0x7f19d806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:13.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.451+0000 7f19fcbd4640 1 --2- 172.21.15.67:0/2863332184 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f19d806bbd0 0x7f19d806e090 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f19e0006e20 tx=0x7f19e0008040 comp rx=0 tx=0).stop 2024-03-24T00:47:13.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.451+0000 7f19fcbd4640 1 -- 172.21.15.67:0/2863332184 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19f80b9af0 msgr2=0x7f19f80cc640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:13.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.451+0000 7f19fcbd4640 1 --2- 172.21.15.67:0/2863332184 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19f80b9af0 0x7f19f80cc640 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f19e8009890 tx=0x7f19e80046c0 comp rx=0 tx=0).stop 2024-03-24T00:47:13.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.451+0000 7f19fcbd4640 1 -- 172.21.15.67:0/2863332184 shutdown_connections 2024-03-24T00:47:13.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.451+0000 7f19fcbd4640 1 --2- 172.21.15.67:0/2863332184 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f19d806bbd0 0x7f19d806e090 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:13.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.451+0000 7f19fcbd4640 1 --2- 172.21.15.67:0/2863332184 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19f815e0f0 0x7f19f81f3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:13.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.451+0000 7f19fcbd4640 1 --2- 172.21.15.67:0/2863332184 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f19f80d3810 0x7f19f80ccb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:13.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.451+0000 7f19fcbd4640 1 --2- 172.21.15.67:0/2863332184 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19f80b9af0 0x7f19f80cc640 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:13.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.451+0000 7f19fcbd4640 1 -- 172.21.15.67:0/2863332184 >> 172.21.15.67:0/2863332184 conn(0x7f19f8072b60 msgr2=0x7f19f80d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:13.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.451+0000 7f19fcbd4640 1 -- 172.21.15.67:0/2863332184 shutdown_connections 2024-03-24T00:47:13.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.451+0000 7f19fcbd4640 1 -- 172.21.15.67:0/2863332184 wait complete. 2024-03-24T00:47:13.465 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-24T00:47:13.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.573+0000 7fb4deac6640 1 Processor -- start 2024-03-24T00:47:13.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.573+0000 7fb4deac6640 1 -- start start 2024-03-24T00:47:13.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.574+0000 7fb4deac6640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb4d809a410 0x7fb4d80cfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:13.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.574+0000 7fb4deac6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb4d80b43a0 0x7fb4d80cc880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:13.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.574+0000 7fb4deac6640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4d80d0fc0 con 0x7fb4d815e1b0 2024-03-24T00:47:13.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.574+0000 7fb4deac6640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4d815e590 con 0x7fb4d809a410 2024-03-24T00:47:13.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.574+0000 7fb4deac6640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d80d1820 con 0x7fb4d80b43a0 2024-03-24T00:47:13.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.574+0000 7fb4d7fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb4d809a410 0x7fb4d80cfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:13.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.574+0000 7fb4d7fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb4d809a410 0x7fb4d80cfe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47066/0 (socket says 172.21.15.67:47066) 2024-03-24T00:47:13.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.574+0000 7fb4d7fff640 1 -- 172.21.15.67:0/3898904457 learned_addr learned my addr 172.21.15.67:0/3898904457 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:13.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.574+0000 7fb4dd063640 1 --2- 172.21.15.67:0/3898904457 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb4d80b43a0 0x7fb4d80cc880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:13.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.574+0000 7fb4d7fff640 1 -- 172.21.15.67:0/3898904457 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb4d80b43a0 msgr2=0x7fb4d80cc880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:13.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.574+0000 7fb4d7fff640 1 --2- 172.21.15.67:0/3898904457 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb4d80b43a0 0x7fb4d80cc880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:13.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.574+0000 7fb4d7fff640 1 -- 172.21.15.67:0/3898904457 >> v1:172.21.15.72:6789/0 conn(0x7fb4d815e1b0 legacy=0x7fb4d8099d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:13.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.574+0000 7fb4d7fff640 1 -- 172.21.15.67:0/3898904457 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4d80d1260 con 0x7fb4d809a410 2024-03-24T00:47:13.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.575+0000 7fb4d7fff640 1 --2- 172.21.15.67:0/3898904457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4d809a410 0x7fb4d80cfe20 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7fb4c8009e50 tx=0x7fb4c802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e23c44843c8997f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:13.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.575+0000 7fb4d77fe640 1 -- 172.21.15.67:0/3898904457 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c802c9e0 con 0x7fb4d809a410 2024-03-24T00:47:13.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.575+0000 7fb4d77fe640 1 -- 172.21.15.67:0/3898904457 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4c802cb80 con 0x7fb4d809a410 2024-03-24T00:47:13.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.575+0000 7fb4d77fe640 1 -- 172.21.15.67:0/3898904457 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c80343f0 con 0x7fb4d809a410 2024-03-24T00:47:13.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.576+0000 7fb4deac6640 1 -- 172.21.15.67:0/3898904457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4d809a410 msgr2=0x7fb4d80cfe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:13.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.576+0000 7fb4deac6640 1 --2- 172.21.15.67:0/3898904457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4d809a410 0x7fb4d80cfe20 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7fb4c8009e50 tx=0x7fb4c802be40 comp rx=0 tx=0).stop 2024-03-24T00:47:13.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.576+0000 7fb4deac6640 1 -- 172.21.15.67:0/3898904457 shutdown_connections 2024-03-24T00:47:13.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.576+0000 7fb4deac6640 1 --2- 172.21.15.67:0/3898904457 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb4d80b43a0 0x7fb4d80cc880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:13.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.576+0000 7fb4deac6640 1 --2- 172.21.15.67:0/3898904457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4d809a410 0x7fb4d80cfe20 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:13.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.576+0000 7fb4deac6640 1 -- 172.21.15.67:0/3898904457 >> 172.21.15.67:0/3898904457 conn(0x7fb4d80b36d0 msgr2=0x7fb4d80b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:13.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.576+0000 7fb4deac6640 1 -- 172.21.15.67:0/3898904457 shutdown_connections 2024-03-24T00:47:13.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.576+0000 7fb4deac6640 1 -- 172.21.15.67:0/3898904457 wait complete. 2024-03-24T00:47:13.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.577+0000 7fb4deac6640 1 Processor -- start 2024-03-24T00:47:13.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.577+0000 7fb4deac6640 1 -- start start 2024-03-24T00:47:13.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.578+0000 7fb4deac6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4d809a410 0x7fb4d8158730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:13.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.578+0000 7fb4deac6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb4d80b43a0 0x7fb4d80ca9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:13.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.578+0000 7fb4deac6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb4d815e1b0 0x7fb4d81e49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:13.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.578+0000 7fb4deac6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d809a9a0 con 0x7fb4d809a410 2024-03-24T00:47:13.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.578+0000 7fb4deac6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d80cc360 con 0x7fb4d815e1b0 2024-03-24T00:47:13.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.578+0000 7fb4deac6640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d80ccdc0 con 0x7fb4d80b43a0 2024-03-24T00:47:13.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.578+0000 7fb4dc862640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4d809a410 0x7fb4d8158730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:13.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.578+0000 7fb4dc862640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4d809a410 0x7fb4d8158730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47072/0 (socket says 172.21.15.67:47072) 2024-03-24T00:47:13.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.578+0000 7fb4dc862640 1 -- 172.21.15.67:0/1914657797 learned_addr learned my addr 172.21.15.67:0/1914657797 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:13.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.578+0000 7fb4dc862640 1 -- 172.21.15.67:0/1914657797 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb4d80b43a0 msgr2=0x7fb4d80ca9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:13.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.578+0000 7fb4dc862640 1 --2- 172.21.15.67:0/1914657797 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb4d80b43a0 0x7fb4d80ca9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:13.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.578+0000 7fb4dc862640 1 -- 172.21.15.67:0/1914657797 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb4d815e1b0 msgr2=0x7fb4d81e49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:13.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.578+0000 7fb4dc862640 1 --2- 172.21.15.67:0/1914657797 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb4d815e1b0 0x7fb4d81e49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:13.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.578+0000 7fb4dc862640 1 -- 172.21.15.67:0/1914657797 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4d8158c70 con 0x7fb4d809a410 2024-03-24T00:47:13.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.579+0000 7fb4dc862640 1 --2- 172.21.15.67:0/1914657797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4d809a410 0x7fb4d8158730 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7fb4d000c340 tx=0x7fb4d000c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:13.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.579+0000 7fb4d57fa640 1 -- 172.21.15.67:0/1914657797 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d0017070 con 0x7fb4d809a410 2024-03-24T00:47:13.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.579+0000 7fb4d57fa640 1 -- 172.21.15.67:0/1914657797 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4d00052d0 con 0x7fb4d809a410 2024-03-24T00:47:13.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.579+0000 7fb4d57fa640 1 -- 172.21.15.67:0/1914657797 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d0003850 con 0x7fb4d809a410 2024-03-24T00:47:13.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.579+0000 7fb4deac6640 1 -- 172.21.15.67:0/1914657797 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4d80b4a20 con 0x7fb4d809a410 2024-03-24T00:47:13.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.579+0000 7fb4deac6640 1 -- 172.21.15.67:0/1914657797 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4d80b4fe0 con 0x7fb4d809a410 2024-03-24T00:47:13.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.580+0000 7fb4deac6640 1 -- 172.21.15.67:0/1914657797 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4d804f400 con 0x7fb4d809a410 2024-03-24T00:47:13.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.580+0000 7fb4d57fa640 1 -- 172.21.15.67:0/1914657797 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb4d001a070 con 0x7fb4d809a410 2024-03-24T00:47:13.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.580+0000 7fb4d57fa640 1 --2- 172.21.15.67:0/1914657797 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb4b806bbd0 0x7fb4b806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:13.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.581+0000 7fb4d57fa640 1 -- 172.21.15.67:0/1914657797 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fb4d0089bd0 con 0x7fb4d809a410 2024-03-24T00:47:13.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.581+0000 7fb4d7fff640 1 --2- 172.21.15.67:0/1914657797 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb4b806bbd0 0x7fb4b806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:13.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.581+0000 7fb4d7fff640 1 --2- 172.21.15.67:0/1914657797 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb4b806bbd0 0x7fb4b806e090 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fb4c8035000 tx=0x7fb4c802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:13.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.585+0000 7fb4d57fa640 1 -- 172.21.15.67:0/1914657797 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb4d0058880 con 0x7fb4d809a410 2024-03-24T00:47:13.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.843+0000 7fb4deac6640 1 -- 172.21.15.67:0/1914657797 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fb4d809a080 con 0x7fb4d809a410 2024-03-24T00:47:13.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.844+0000 7fb4d57fa640 1 -- 172.21.15.67:0/1914657797 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v18) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fb4d0058180 con 0x7fb4d809a410 2024-03-24T00:47:13.844 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:47:13.844 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:47:13.844 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_dump": [ 2024-03-24T00:47:13.844 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:47:13.844 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.a", 2024-03-24T00:47:13.844 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9l4UENDhAASKUDtqcbSQ7wADj7uo4eyQ==", 2024-03-24T00:47:13.844 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:47:13.844 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:47:13.844 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-03-24T00:47:13.844 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-03-24T00:47:13.844 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:47:13.844 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:47:13.845 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:47:13.845 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:47:13.845 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.b", 2024-03-24T00:47:13.845 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9luhJWGhAAsS6PFHV2XdF9htjIjzdrdA==", 2024-03-24T00:47:13.845 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:47:13.845 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:47:13.845 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-03-24T00:47:13.845 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-03-24T00:47:13.845 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:47:13.845 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:47:13.845 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:47:13.845 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:47:13.845 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.c", 2024-03-24T00:47:13.845 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9l8mRjFRAAQ73yexNddpcyp8ayTcRKCg==", 2024-03-24T00:47:13.846 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:47:13.846 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:47:13.846 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-03-24T00:47:13.846 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-03-24T00:47:13.846 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:47:13.846 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:47:13.846 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:47:13.846 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:47:13.846 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.d", 2024-03-24T00:47:13.846 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9lcJyvIRAA2tpDb6qxn1MtljjEno4qPA==", 2024-03-24T00:47:13.846 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:47:13.846 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:47:13.846 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-03-24T00:47:13.846 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-03-24T00:47:13.847 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:47:13.847 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:47:13.847 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:47:13.847 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:47:13.847 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.0", 2024-03-24T00:47:13.847 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDgdf9lV/NUBRAAyJNajFAnNABJErnBlB8TEA==", 2024-03-24T00:47:13.847 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:47:13.847 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:47:13.847 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:47:13.847 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:47:13.847 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:47:13.847 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:47:13.847 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:47:13.847 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.1", 2024-03-24T00:47:13.847 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDjdf9l0J6lAxAA9RInt/X22jqzthE0vOiiFg==", 2024-03-24T00:47:13.847 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:47:13.848 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:47:13.848 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:47:13.848 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:47:13.848 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:47:13.848 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:47:13.848 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:47:13.848 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.2", 2024-03-24T00:47:13.848 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDmdf9lSmrEARAAwmtSQRNePsn4WIZODnjhfQ==", 2024-03-24T00:47:13.848 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:47:13.848 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:47:13.848 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:47:13.848 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:47:13.848 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:47:13.848 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:47:13.849 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:47:13.849 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.3", 2024-03-24T00:47:13.849 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDpdf9lm59bABAAgJ5AV774lw2BgkSCgaweyA==", 2024-03-24T00:47:13.849 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:47:13.849 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:47:13.849 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:47:13.849 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:47:13.849 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:47:13.849 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:47:13.849 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:47:13.849 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.4", 2024-03-24T00:47:13.849 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDvdf9lCo+pCRAANHkL0pPxxok6IsBCLAFCbg==", 2024-03-24T00:47:13.849 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:47:13.849 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:47:13.849 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:47:13.850 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:47:13.850 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:47:13.850 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:47:13.850 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:47:13.850 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.5", 2024-03-24T00:47:13.850 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDydf9lyGEdCBAAcuszPsthGlLvVryE7+D9iQ==", 2024-03-24T00:47:13.850 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:47:13.850 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:47:13.850 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:47:13.850 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:47:13.850 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:47:13.850 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:47:13.850 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:47:13.850 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.6", 2024-03-24T00:47:13.851 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQD1df9ljuZqBhAACt2OPNkjKOD3rK7g2mVBBA==", 2024-03-24T00:47:13.851 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:47:13.851 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:47:13.851 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:47:13.851 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:47:13.851 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:47:13.851 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:47:13.851 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:47:13.851 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.7", 2024-03-24T00:47:13.851 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQD4df9lYKBfBRAAy/cWFF6pvGDjXZHAuBYw7g==", 2024-03-24T00:47:13.851 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:47:13.851 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:47:13.851 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:47:13.851 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:47:13.851 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:47:13.851 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:47:13.852 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:47:13.852 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.0", 2024-03-24T00:47:13.852 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9lY4nvJRAA6HWOZDWA+SSvJVmyjG+51g==", 2024-03-24T00:47:13.852 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:47:13.852 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:47:13.852 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow r", 2024-03-24T00:47:13.852 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-24T00:47:13.852 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:47:13.852 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:47:13.852 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:47:13.852 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.1", 2024-03-24T00:47:13.852 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9lonsXLxAAaltVMW3n0qHX8u9X8EkSmg==", 2024-03-24T00:47:13.852 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:47:13.852 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:47:13.852 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-03-24T00:47:13.853 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-03-24T00:47:13.853 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-03-24T00:47:13.853 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:47:13.853 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:47:13.853 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:47:13.853 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.2", 2024-03-24T00:47:13.853 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9lLEm2KxAAkT65mhLrTirlDnOUyvBY7g==", 2024-03-24T00:47:13.853 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:47:13.853 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:47:13.853 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-03-24T00:47:13.853 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-03-24T00:47:13.853 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-03-24T00:47:13.853 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:47:13.853 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:47:13.853 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:47:13.854 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.3", 2024-03-24T00:47:13.854 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9l8njZNBAAaAktZpX2SPcJR9ymPm9ebw==", 2024-03-24T00:47:13.854 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:47:13.854 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:47:13.854 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-03-24T00:47:13.854 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-03-24T00:47:13.854 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-03-24T00:47:13.854 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:47:13.854 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:47:13.854 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:47:13.854 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.admin", 2024-03-24T00:47:13.854 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDbdf9lDTIwLRAAMajqhM/KfUNF/SjVv9OJDw==", 2024-03-24T00:47:13.855 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:47:13.855 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-03-24T00:47:13.855 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-03-24T00:47:13.855 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-03-24T00:47:13.855 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:47:13.855 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:47:13.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:47:13.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:47:13.855 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.x", 2024-03-24T00:47:13.855 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9lGwKvChAAxl+fRnjbAQ7dYJ727QFUgw==", 2024-03-24T00:47:13.855 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:47:13.855 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-03-24T00:47:13.855 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-03-24T00:47:13.855 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:47:13.855 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:47:13.856 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:47:13.856 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:47:13.856 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.y", 2024-03-24T00:47:13.856 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9lSvufBhAAoQ9L1GjLkstjrAw2lqlryA==", 2024-03-24T00:47:13.856 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:47:13.856 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-03-24T00:47:13.856 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-03-24T00:47:13.856 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:47:13.856 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:47:13.856 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:47:13.856 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-24T00:47:13.856 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:47:13.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.846+0000 7fb4deac6640 1 -- 172.21.15.67:0/1914657797 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb4b806bbd0 msgr2=0x7fb4b806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:13.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.846+0000 7fb4deac6640 1 --2- 172.21.15.67:0/1914657797 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb4b806bbd0 0x7fb4b806e090 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fb4c8035000 tx=0x7fb4c802e000 comp rx=0 tx=0).stop 2024-03-24T00:47:13.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.846+0000 7fb4deac6640 1 -- 172.21.15.67:0/1914657797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4d809a410 msgr2=0x7fb4d8158730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:13.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.846+0000 7fb4deac6640 1 --2- 172.21.15.67:0/1914657797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4d809a410 0x7fb4d8158730 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7fb4d000c340 tx=0x7fb4d000c800 comp rx=0 tx=0).stop 2024-03-24T00:47:13.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.846+0000 7fb4deac6640 1 -- 172.21.15.67:0/1914657797 shutdown_connections 2024-03-24T00:47:13.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.846+0000 7fb4deac6640 1 --2- 172.21.15.67:0/1914657797 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb4b806bbd0 0x7fb4b806e090 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:13.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.846+0000 7fb4deac6640 1 --2- 172.21.15.67:0/1914657797 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb4d815e1b0 0x7fb4d81e49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:13.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.846+0000 7fb4deac6640 1 --2- 172.21.15.67:0/1914657797 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb4d80b43a0 0x7fb4d80ca9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:13.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.846+0000 7fb4deac6640 1 --2- 172.21.15.67:0/1914657797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4d809a410 0x7fb4d8158730 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:13.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.846+0000 7fb4deac6640 1 -- 172.21.15.67:0/1914657797 >> 172.21.15.67:0/1914657797 conn(0x7fb4d80b36d0 msgr2=0x7fb4d806fe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:13.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.846+0000 7fb4deac6640 1 -- 172.21.15.67:0/1914657797 shutdown_connections 2024-03-24T00:47:13.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.846+0000 7fb4deac6640 1 -- 172.21.15.67:0/1914657797 wait complete. 2024-03-24T00:47:13.859 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:47:13.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.961+0000 7f96a1f97640 1 Processor -- start 2024-03-24T00:47:13.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.961+0000 7f96a1f97640 1 -- start start 2024-03-24T00:47:13.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.962+0000 7f96a1f97640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f969c0cd350 0x7f969c06b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:13.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.962+0000 7f96a1f97640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f969c0af2e0 0x7f969c0b58f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:13.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.962+0000 7f96a1f97640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f969c0b2ca0 con 0x7f969c15be40 2024-03-24T00:47:13.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.962+0000 7f96a1f97640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f969c0afb80 con 0x7f969c0cd350 2024-03-24T00:47:13.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.962+0000 7f96a1f97640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f969c15c220 con 0x7f969c0af2e0 2024-03-24T00:47:13.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.962+0000 7f969bfff640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f969c0af2e0 0x7f969c0b58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:13.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.962+0000 7f969affd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f969c0cd350 0x7f969c06b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:13.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.962+0000 7f969bfff640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f969c0af2e0 0x7f969c0b58f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.67:38452/0 (socket says 172.21.15.67:38452) 2024-03-24T00:47:13.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.962+0000 7f969bfff640 1 -- 172.21.15.67:0/575155220 learned_addr learned my addr 172.21.15.67:0/575155220 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:13.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.962+0000 7f969bfff640 1 -- 172.21.15.67:0/575155220 >> v2:172.21.15.72:3300/0 conn(0x7f969c0cd350 msgr2=0x7f969c06b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:13.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.962+0000 7f969bfff640 1 --2- 172.21.15.67:0/575155220 >> v2:172.21.15.72:3300/0 conn(0x7f969c0cd350 0x7f969c06b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:13.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.962+0000 7f969bfff640 1 -- 172.21.15.67:0/575155220 >> v1:172.21.15.72:6789/0 conn(0x7f969c15be40 legacy=0x7f969c0ccc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:13.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.962+0000 7f969bfff640 1 -- 172.21.15.67:0/575155220 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f969c0b26e0 con 0x7f969c0af2e0 2024-03-24T00:47:13.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.963+0000 7f969bfff640 1 --2- 172.21.15.67:0/575155220 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f969c0af2e0 0x7f969c0b58f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9694009e50 tx=0x7f969402bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88efeded5c1d6583 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:13.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.963+0000 7f969a7fc640 1 -- 172.21.15.67:0/575155220 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f969402ca70 con 0x7f969c0af2e0 2024-03-24T00:47:13.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.963+0000 7f969a7fc640 1 -- 172.21.15.67:0/575155220 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f969402cc10 con 0x7f969c0af2e0 2024-03-24T00:47:13.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.963+0000 7f969a7fc640 1 -- 172.21.15.67:0/575155220 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96940343f0 con 0x7f969c0af2e0 2024-03-24T00:47:13.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.964+0000 7f96a1f97640 1 -- 172.21.15.67:0/575155220 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f969c0af2e0 msgr2=0x7f969c0b58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:13.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.964+0000 7f96a1f97640 1 --2- 172.21.15.67:0/575155220 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f969c0af2e0 0x7f969c0b58f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9694009e50 tx=0x7f969402bed0 comp rx=0 tx=0).stop 2024-03-24T00:47:13.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.964+0000 7f96a1f97640 1 -- 172.21.15.67:0/575155220 shutdown_connections 2024-03-24T00:47:13.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.964+0000 7f96a1f97640 1 --2- 172.21.15.67:0/575155220 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f969c0af2e0 0x7f969c0b58f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:13.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.964+0000 7f96a1f97640 1 --2- 172.21.15.67:0/575155220 >> v2:172.21.15.72:3300/0 conn(0x7f969c0cd350 0x7f969c06b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:13.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.964+0000 7f96a1f97640 1 -- 172.21.15.67:0/575155220 >> 172.21.15.67:0/575155220 conn(0x7f969c0b3c40 msgr2=0x7f969c0b1dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:13.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.964+0000 7f96a1f97640 1 -- 172.21.15.67:0/575155220 shutdown_connections 2024-03-24T00:47:13.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.964+0000 7f96a1f97640 1 -- 172.21.15.67:0/575155220 wait complete. 2024-03-24T00:47:13.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.965+0000 7f96a1f97640 1 Processor -- start 2024-03-24T00:47:13.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.965+0000 7f96a1f97640 1 -- start start 2024-03-24T00:47:13.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.966+0000 7f96a1f97640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f969c0af2e0 0x7f969c0c4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:13.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.966+0000 7f96a1f97640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f969c0cd350 0x7f969c0c6b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:13.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.966+0000 7f96a1f97640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f969c15be40 0x7f969c156d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:13.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.966+0000 7f96a1f97640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f969c0cf400 con 0x7f969c15be40 2024-03-24T00:47:13.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.966+0000 7f96a1f97640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f969c0ceee0 con 0x7f969c0cd350 2024-03-24T00:47:13.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.966+0000 7f96a1f97640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f969c0cf580 con 0x7f969c0af2e0 2024-03-24T00:47:13.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.966+0000 7f969bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f969c15be40 0x7f969c156d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:13.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.966+0000 7f969bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f969c15be40 0x7f969c156d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47076/0 (socket says 172.21.15.67:47076) 2024-03-24T00:47:13.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.966+0000 7f969bfff640 1 -- 172.21.15.67:0/3827135427 learned_addr learned my addr 172.21.15.67:0/3827135427 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:13.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.966+0000 7f969affd640 1 --2- 172.21.15.67:0/3827135427 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f969c0cd350 0x7f969c0c6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:13.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.966+0000 7f969b7fe640 1 --2- 172.21.15.67:0/3827135427 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f969c0af2e0 0x7f969c0c4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:13.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.966+0000 7f969bfff640 1 -- 172.21.15.67:0/3827135427 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f969c0af2e0 msgr2=0x7f969c0c4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:13.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.966+0000 7f969bfff640 1 --2- 172.21.15.67:0/3827135427 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f969c0af2e0 0x7f969c0c4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:13.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.966+0000 7f969bfff640 1 -- 172.21.15.67:0/3827135427 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f969c0cd350 msgr2=0x7f969c0c6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:13.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.967+0000 7f969bfff640 1 --2- 172.21.15.67:0/3827135427 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f969c0cd350 0x7f969c0c6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:13.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.967+0000 7f969bfff640 1 -- 172.21.15.67:0/3827135427 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f969c0c70c0 con 0x7f969c15be40 2024-03-24T00:47:13.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.967+0000 7f969bfff640 1 --2- 172.21.15.67:0/3827135427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f969c15be40 0x7f969c156d80 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f9694036040 tx=0x7f9694004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:13.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.967+0000 7f9693fff640 1 -- 172.21.15.67:0/3827135427 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9694043020 con 0x7f969c15be40 2024-03-24T00:47:13.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.967+0000 7f9693fff640 1 -- 172.21.15.67:0/3827135427 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f969403e450 con 0x7f969c15be40 2024-03-24T00:47:13.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.967+0000 7f96a1f97640 1 -- 172.21.15.67:0/3827135427 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f969c0c5150 con 0x7f969c15be40 2024-03-24T00:47:13.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.967+0000 7f9693fff640 1 -- 172.21.15.67:0/3827135427 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9694009510 con 0x7f969c15be40 2024-03-24T00:47:13.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.967+0000 7f96a1f97640 1 -- 172.21.15.67:0/3827135427 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f969c0c5710 con 0x7f969c15be40 2024-03-24T00:47:13.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.968+0000 7f96a1f97640 1 -- 172.21.15.67:0/3827135427 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f969c04f400 con 0x7f969c15be40 2024-03-24T00:47:13.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.968+0000 7f9693fff640 1 -- 172.21.15.67:0/3827135427 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9694046020 con 0x7f969c15be40 2024-03-24T00:47:13.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.969+0000 7f9693fff640 1 --2- 172.21.15.67:0/3827135427 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f967c06bc00 0x7f967c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:13.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.969+0000 7f9693fff640 1 -- 172.21.15.67:0/3827135427 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f96940ace40 con 0x7f969c15be40 2024-03-24T00:47:13.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.969+0000 7f969b7fe640 1 --2- 172.21.15.67:0/3827135427 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f967c06bc00 0x7f967c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:13.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.970+0000 7f969b7fe640 1 --2- 172.21.15.67:0/3827135427 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f967c06bc00 0x7f967c06e0c0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f968c0016e0 tx=0x7f968c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:13.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:13.973+0000 7f9693fff640 1 -- 172.21.15.67:0/3827135427 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f969404d030 con 0x7f969c15be40 2024-03-24T00:47:14.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.157+0000 7f96a1f97640 1 -- 172.21.15.67:0/3827135427 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f969c0aeb30 con 0x7f969c15be40 2024-03-24T00:47:14.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.158+0000 7f9693fff640 1 -- 172.21.15.67:0/3827135427 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) v1 ==== 74+0+13106 (secure 0 0 0) 0x7f969407b530 con 0x7f969c15be40 2024-03-24T00:47:14.157 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:47:14.158 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":101,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:47:13.447099+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:47:14.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.159+0000 7f96a1f97640 1 -- 172.21.15.67:0/3827135427 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f967c06bc00 msgr2=0x7f967c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:14.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.159+0000 7f96a1f97640 1 --2- 172.21.15.67:0/3827135427 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f967c06bc00 0x7f967c06e0c0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f968c0016e0 tx=0x7f968c009040 comp rx=0 tx=0).stop 2024-03-24T00:47:14.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.159+0000 7f96a1f97640 1 -- 172.21.15.67:0/3827135427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f969c15be40 msgr2=0x7f969c156d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:14.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.159+0000 7f96a1f97640 1 --2- 172.21.15.67:0/3827135427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f969c15be40 0x7f969c156d80 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f9694036040 tx=0x7f9694004a40 comp rx=0 tx=0).stop 2024-03-24T00:47:14.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.159+0000 7f96a1f97640 1 -- 172.21.15.67:0/3827135427 shutdown_connections 2024-03-24T00:47:14.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.159+0000 7f96a1f97640 1 --2- 172.21.15.67:0/3827135427 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f967c06bc00 0x7f967c06e0c0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:14.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.159+0000 7f96a1f97640 1 --2- 172.21.15.67:0/3827135427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f969c15be40 0x7f969c156d80 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:14.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.159+0000 7f96a1f97640 1 --2- 172.21.15.67:0/3827135427 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f969c0cd350 0x7f969c0c6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:14.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.159+0000 7f96a1f97640 1 --2- 172.21.15.67:0/3827135427 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f969c0af2e0 0x7f969c0c4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:14.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.160+0000 7f96a1f97640 1 -- 172.21.15.67:0/3827135427 >> 172.21.15.67:0/3827135427 conn(0x7f969c0b3c40 msgr2=0x7f969c157650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:14.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.160+0000 7f96a1f97640 1 -- 172.21.15.67:0/3827135427 shutdown_connections 2024-03-24T00:47:14.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.160+0000 7f96a1f97640 1 -- 172.21.15.67:0/3827135427 wait complete. 2024-03-24T00:47:14.170 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:38:39.364950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:47:14.171 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:47:14.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.275+0000 7f41fe3f3640 1 Processor -- start 2024-03-24T00:47:14.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.275+0000 7f41fe3f3640 1 -- start start 2024-03-24T00:47:14.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.276+0000 7f41fe3f3640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f41f815bc30 0x7f41f80b1a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:14.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.276+0000 7f41fe3f3640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41f80b3ae0 con 0x7f41f80b34a0 2024-03-24T00:47:14.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.276+0000 7f41fe3f3640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41f80b3dc0 con 0x7f41f815c010 2024-03-24T00:47:14.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.276+0000 7f41fe3f3640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41f80cca60 con 0x7f41f815bc30 2024-03-24T00:47:14.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.276+0000 7f41f7fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f41f815bc30 0x7f41f80b1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:14.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.276+0000 7f41fc990640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f41f80b34a0 0x7f41f80ccef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48220/0 (socket says 172.21.15.67:48220) 2024-03-24T00:47:14.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.276+0000 7f41fc990640 1 -- 172.21.15.67:0/2470280472 learned_addr learned my addr 172.21.15.67:0/2470280472 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:47:14.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.277+0000 7f41f7fff640 1 -- 172.21.15.67:0/2470280472 >> v1:172.21.15.72:6789/0 conn(0x7f41f815c010 legacy=0x7f41f80b2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:14.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.277+0000 7f41f7fff640 1 -- 172.21.15.67:0/2470280472 >> v1:172.21.15.67:6789/0 conn(0x7f41f80b34a0 legacy=0x7f41f80ccef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:14.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.277+0000 7f41f6ffd640 1 -- 172.21.15.67:0/2470280472 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 556057035 0 0) 0x7f41f80b3ae0 con 0x7f41f80b34a0 2024-03-24T00:47:14.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.277+0000 7f41f7fff640 1 -- 172.21.15.67:0/2470280472 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41f80b40a0 con 0x7f41f815bc30 2024-03-24T00:47:14.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.277+0000 7f41f7fff640 1 --2- 172.21.15.67:0/2470280472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41f815bc30 0x7f41f80b1a00 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f41ec0097a0 tx=0x7f41ec02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e3bd6a45c23a04f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:14.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.278+0000 7f41f6ffd640 1 -- 172.21.15.67:0/2470280472 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41ec035020 con 0x7f41f815bc30 2024-03-24T00:47:14.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.278+0000 7f41f6ffd640 1 -- 172.21.15.67:0/2470280472 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41ec031830 con 0x7f41f815bc30 2024-03-24T00:47:14.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.278+0000 7f41f6ffd640 1 -- 172.21.15.67:0/2470280472 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41ec031c00 con 0x7f41f815bc30 2024-03-24T00:47:14.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.278+0000 7f41fe3f3640 1 -- 172.21.15.67:0/2470280472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41f815bc30 msgr2=0x7f41f80b1a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:14.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.278+0000 7f41fe3f3640 1 --2- 172.21.15.67:0/2470280472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41f815bc30 0x7f41f80b1a00 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f41ec0097a0 tx=0x7f41ec02b8d0 comp rx=0 tx=0).stop 2024-03-24T00:47:14.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.278+0000 7f41fe3f3640 1 -- 172.21.15.67:0/2470280472 shutdown_connections 2024-03-24T00:47:14.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.278+0000 7f41fe3f3640 1 --2- 172.21.15.67:0/2470280472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41f815bc30 0x7f41f80b1a00 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:14.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.278+0000 7f41fe3f3640 1 -- 172.21.15.67:0/2470280472 >> 172.21.15.67:0/2470280472 conn(0x7f41f808b180 msgr2=0x7f41f808b590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:14.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.278+0000 7f41fe3f3640 1 -- 172.21.15.67:0/2470280472 shutdown_connections 2024-03-24T00:47:14.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.278+0000 7f41fe3f3640 1 -- 172.21.15.67:0/2470280472 wait complete. 2024-03-24T00:47:14.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.279+0000 7f41fe3f3640 1 Processor -- start 2024-03-24T00:47:14.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.280+0000 7f41fe3f3640 1 -- start start 2024-03-24T00:47:14.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.280+0000 7f41fe3f3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f41f80b34a0 0x7f41f80c55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:14.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.280+0000 7f41fe3f3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41f815bc30 0x7f41f81500e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:14.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.280+0000 7f41fe3f3640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f41f815c010 0x7f41f81579c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:14.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.280+0000 7f41fe3f3640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f80b20c0 con 0x7f41f815bc30 2024-03-24T00:47:14.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.280+0000 7f41fe3f3640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f80b1f40 con 0x7f41f80b34a0 2024-03-24T00:47:14.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.280+0000 7f41fe3f3640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41f80b2240 con 0x7f41f815c010 2024-03-24T00:47:14.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.280+0000 7f41f77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41f815bc30 0x7f41f81500e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:14.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.280+0000 7f41f77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41f815bc30 0x7f41f81500e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47088/0 (socket says 172.21.15.67:47088) 2024-03-24T00:47:14.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.280+0000 7f41f77fe640 1 -- 172.21.15.67:0/3951863471 learned_addr learned my addr 172.21.15.67:0/3951863471 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:14.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.281+0000 7f41fc990640 1 --2- 172.21.15.67:0/3951863471 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f41f815c010 0x7f41f81579c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:14.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.281+0000 7f41f77fe640 1 -- 172.21.15.67:0/3951863471 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f41f815c010 msgr2=0x7f41f81579c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:14.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.281+0000 7f41f7fff640 1 --2- 172.21.15.67:0/3951863471 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f41f80b34a0 0x7f41f80c55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:14.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.281+0000 7f41f77fe640 1 --2- 172.21.15.67:0/3951863471 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f41f815c010 0x7f41f81579c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:14.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.281+0000 7f41f77fe640 1 -- 172.21.15.67:0/3951863471 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f41f80b34a0 msgr2=0x7f41f80c55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:14.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.281+0000 7f41f77fe640 1 --2- 172.21.15.67:0/3951863471 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f41f80b34a0 0x7f41f80c55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:14.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.281+0000 7f41f77fe640 1 -- 172.21.15.67:0/3951863471 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41f80c5b00 con 0x7f41f815bc30 2024-03-24T00:47:14.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.281+0000 7f41fc990640 1 --2- 172.21.15.67:0/3951863471 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f41f815c010 0x7f41f81579c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:47:14.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.281+0000 7f41f7fff640 1 --2- 172.21.15.67:0/3951863471 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f41f80b34a0 0x7f41f80c55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:47:14.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.281+0000 7f41f77fe640 1 --2- 172.21.15.67:0/3951863471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41f815bc30 0x7f41f81500e0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f41e800b3e0 tx=0x7f41e800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:14.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.281+0000 7f41f4ff9640 1 -- 172.21.15.67:0/3951863471 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e8015020 con 0x7f41f815bc30 2024-03-24T00:47:14.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.281+0000 7f41f4ff9640 1 -- 172.21.15.67:0/3951863471 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41e8011850 con 0x7f41f815bc30 2024-03-24T00:47:14.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.281+0000 7f41fe3f3640 1 -- 172.21.15.67:0/3951863471 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41f80b5ef0 con 0x7f41f815bc30 2024-03-24T00:47:14.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.281+0000 7f41f4ff9640 1 -- 172.21.15.67:0/3951863471 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e8011c40 con 0x7f41f815bc30 2024-03-24T00:47:14.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.281+0000 7f41fe3f3640 1 -- 172.21.15.67:0/3951863471 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41f80b64b0 con 0x7f41f815bc30 2024-03-24T00:47:14.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.282+0000 7f41fe3f3640 1 -- 172.21.15.67:0/3951863471 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41f804f400 con 0x7f41f815bc30 2024-03-24T00:47:14.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.282+0000 7f41f4ff9640 1 -- 172.21.15.67:0/3951863471 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f41e8018020 con 0x7f41f815bc30 2024-03-24T00:47:14.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.283+0000 7f41f4ff9640 1 --2- 172.21.15.67:0/3951863471 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f41e406bbd0 0x7f41e406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:14.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.283+0000 7f41f4ff9640 1 -- 172.21.15.67:0/3951863471 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f41e808a7e0 con 0x7f41f815bc30 2024-03-24T00:47:14.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.283+0000 7f41f7fff640 1 --2- 172.21.15.67:0/3951863471 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f41e406bbd0 0x7f41e406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:14.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.284+0000 7f41f7fff640 1 --2- 172.21.15.67:0/3951863471 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f41e406bbd0 0x7f41e406e090 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f41ec036000 tx=0x7f41ec008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:14.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.287+0000 7f41f4ff9640 1 -- 172.21.15.67:0/3951863471 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f41e801f020 con 0x7f41f815bc30 2024-03-24T00:47:14.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.469+0000 7f41fe3f3640 1 -- 172.21.15.67:0/3951863471 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f41f80b17e0 con 0x7f41f815bc30 2024-03-24T00:47:14.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.469+0000 7f41f4ff9640 1 -- 172.21.15.67:0/3951863471 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v101) v1 ==== 93+0+10 (secure 0 0 0) 0x7f41e80172c0 con 0x7f41f815bc30 2024-03-24T00:47:14.469 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-24T00:47:14.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.471+0000 7f41fe3f3640 1 -- 172.21.15.67:0/3951863471 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f41e406bbd0 msgr2=0x7f41e406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:14.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.471+0000 7f41fe3f3640 1 --2- 172.21.15.67:0/3951863471 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f41e406bbd0 0x7f41e406e090 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f41ec036000 tx=0x7f41ec008030 comp rx=0 tx=0).stop 2024-03-24T00:47:14.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.471+0000 7f41fe3f3640 1 -- 172.21.15.67:0/3951863471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41f815bc30 msgr2=0x7f41f81500e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:14.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.471+0000 7f41fe3f3640 1 --2- 172.21.15.67:0/3951863471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41f815bc30 0x7f41f81500e0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f41e800b3e0 tx=0x7f41e800b8a0 comp rx=0 tx=0).stop 2024-03-24T00:47:14.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.471+0000 7f41fe3f3640 1 -- 172.21.15.67:0/3951863471 shutdown_connections 2024-03-24T00:47:14.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.471+0000 7f41fe3f3640 1 --2- 172.21.15.67:0/3951863471 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f41e406bbd0 0x7f41e406e090 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:14.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.471+0000 7f41fe3f3640 1 --2- 172.21.15.67:0/3951863471 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f41f815c010 0x7f41f81579c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:14.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.471+0000 7f41fe3f3640 1 --2- 172.21.15.67:0/3951863471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41f815bc30 0x7f41f81500e0 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:14.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.471+0000 7f41fe3f3640 1 --2- 172.21.15.67:0/3951863471 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f41f80b34a0 0x7f41f80c55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:14.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.471+0000 7f41fe3f3640 1 -- 172.21.15.67:0/3951863471 >> 172.21.15.67:0/3951863471 conn(0x7f41f808b180 msgr2=0x7f41f80ce920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:14.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.471+0000 7f41fe3f3640 1 -- 172.21.15.67:0/3951863471 shutdown_connections 2024-03-24T00:47:14.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.471+0000 7f41fe3f3640 1 -- 172.21.15.67:0/3951863471 wait complete. 2024-03-24T00:47:14.482 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-24T00:47:14.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.583+0000 7f2bf2085640 1 Processor -- start 2024-03-24T00:47:14.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.583+0000 7f2bf2085640 1 -- start start 2024-03-24T00:47:14.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.584+0000 7f2bf2085640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2bec070670 0x7f2bec0b5c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:14.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.584+0000 7f2bf2085640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bec15ece0 con 0x7f2bec15bb20 2024-03-24T00:47:14.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.584+0000 7f2bf2085640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bec15efc0 con 0x7f2bec0b2f30 2024-03-24T00:47:14.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.584+0000 7f2bf2085640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bec0ae630 con 0x7f2bec070670 2024-03-24T00:47:14.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.584+0000 7f2beb7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2bec15bb20 0x7f2bec15e440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48232/0 (socket says 172.21.15.67:48232) 2024-03-24T00:47:14.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.584+0000 7f2beb7fe640 1 -- 172.21.15.67:0/1852086429 learned_addr learned my addr 172.21.15.67:0/1852086429 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:47:14.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.584+0000 7f2bebfff640 1 --2- 172.21.15.67:0/1852086429 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2bec070670 0x7f2bec0b5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:14.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.584+0000 7f2bea7fc640 1 -- 172.21.15.67:0/1852086429 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3033877264 0 0) 0x7f2bec15ece0 con 0x7f2bec15bb20 2024-03-24T00:47:14.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.584+0000 7f2bea7fc640 1 -- 172.21.15.67:0/1852086429 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bd4003520 con 0x7f2bec15bb20 2024-03-24T00:47:14.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.585+0000 7f2bea7fc640 1 -- 172.21.15.67:0/1852086429 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3947058858 0 0) 0x7f2bd4003520 con 0x7f2bec15bb20 2024-03-24T00:47:14.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.585+0000 7f2bebfff640 1 -- 172.21.15.67:0/1852086429 >> v1:172.21.15.72:6789/0 conn(0x7f2bec0b2f30 legacy=0x7f2bec06ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:14.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.585+0000 7f2bebfff640 1 -- 172.21.15.67:0/1852086429 >> v1:172.21.15.67:6789/0 conn(0x7f2bec15bb20 legacy=0x7f2bec15e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:14.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.585+0000 7f2bebfff640 1 -- 172.21.15.67:0/1852086429 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bec15f2a0 con 0x7f2bec070670 2024-03-24T00:47:14.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.585+0000 7f2bebfff640 1 --2- 172.21.15.67:0/1852086429 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2bec070670 0x7f2bec0b5c80 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2be4009dc0 tx=0x7f2be402bc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=167b807b3272aeea server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:14.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.586+0000 7f2bea7fc640 1 -- 172.21.15.67:0/1852086429 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be4039070 con 0x7f2bec070670 2024-03-24T00:47:14.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.586+0000 7f2bea7fc640 1 -- 172.21.15.67:0/1852086429 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2be402c770 con 0x7f2bec070670 2024-03-24T00:47:14.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.586+0000 7f2bea7fc640 1 -- 172.21.15.67:0/1852086429 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bdc001670 con 0x7f2bec070670 2024-03-24T00:47:14.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.586+0000 7f2bf2085640 1 -- 172.21.15.67:0/1852086429 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2bec070670 msgr2=0x7f2bec0b5c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:14.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.586+0000 7f2bf2085640 1 --2- 172.21.15.67:0/1852086429 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2bec070670 0x7f2bec0b5c80 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2be4009dc0 tx=0x7f2be402bc40 comp rx=0 tx=0).stop 2024-03-24T00:47:14.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.586+0000 7f2bf2085640 1 -- 172.21.15.67:0/1852086429 shutdown_connections 2024-03-24T00:47:14.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.586+0000 7f2bf2085640 1 --2- 172.21.15.67:0/1852086429 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2bec070670 0x7f2bec0b5c80 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:14.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.586+0000 7f2bf2085640 1 -- 172.21.15.67:0/1852086429 >> 172.21.15.67:0/1852086429 conn(0x7f2bec071ef0 msgr2=0x7f2bec072300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:14.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.586+0000 7f2bf2085640 1 -- 172.21.15.67:0/1852086429 shutdown_connections 2024-03-24T00:47:14.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.586+0000 7f2bf2085640 1 -- 172.21.15.67:0/1852086429 wait complete. 2024-03-24T00:47:14.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.587+0000 7f2bf2085640 1 Processor -- start 2024-03-24T00:47:14.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.588+0000 7f2bf2085640 1 -- start start 2024-03-24T00:47:14.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.588+0000 7f2bf2085640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2bec070670 0x7f2bec15bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:14.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.588+0000 7f2bf2085640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2bec0b2f30 0x7f2bec1589a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:14.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.588+0000 7f2bf2085640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2bec15bb20 0x7f2bec15a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:14.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.588+0000 7f2bf2085640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bec168f20 con 0x7f2bec0b2f30 2024-03-24T00:47:14.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.588+0000 7f2bf2085640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bec168da0 con 0x7f2bec15bb20 2024-03-24T00:47:14.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.588+0000 7f2bf2085640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bec1690a0 con 0x7f2bec070670 2024-03-24T00:47:14.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.588+0000 7f2beaffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2bec0b2f30 0x7f2bec1589a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:14.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.588+0000 7f2beaffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2bec0b2f30 0x7f2bec1589a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47100/0 (socket says 172.21.15.67:47100) 2024-03-24T00:47:14.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.588+0000 7f2beaffd640 1 -- 172.21.15.67:0/872700618 learned_addr learned my addr 172.21.15.67:0/872700618 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:14.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.588+0000 7f2beb7fe640 1 --2- 172.21.15.67:0/872700618 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2bec070670 0x7f2bec15bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:14.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.588+0000 7f2bebfff640 1 --2- 172.21.15.67:0/872700618 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2bec15bb20 0x7f2bec15a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:14.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.589+0000 7f2beaffd640 1 -- 172.21.15.67:0/872700618 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2bec070670 msgr2=0x7f2bec15bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:14.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.589+0000 7f2beaffd640 1 --2- 172.21.15.67:0/872700618 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2bec070670 0x7f2bec15bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:14.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.589+0000 7f2beaffd640 1 -- 172.21.15.67:0/872700618 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2bec15bb20 msgr2=0x7f2bec15a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:14.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.589+0000 7f2beaffd640 1 --2- 172.21.15.67:0/872700618 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2bec15bb20 0x7f2bec15a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:14.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.589+0000 7f2beaffd640 1 -- 172.21.15.67:0/872700618 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bec159ef0 con 0x7f2bec0b2f30 2024-03-24T00:47:14.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.589+0000 7f2beaffd640 1 --2- 172.21.15.67:0/872700618 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2bec0b2f30 0x7f2bec1589a0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f2be000c780 tx=0x7f2be000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:14.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.589+0000 7f2bd3fff640 1 -- 172.21.15.67:0/872700618 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be0017070 con 0x7f2bec0b2f30 2024-03-24T00:47:14.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.589+0000 7f2bd3fff640 1 -- 172.21.15.67:0/872700618 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2be0005270 con 0x7f2bec0b2f30 2024-03-24T00:47:14.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.589+0000 7f2bf2085640 1 -- 172.21.15.67:0/872700618 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bec15a1d0 con 0x7f2bec0b2f30 2024-03-24T00:47:14.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.589+0000 7f2bd3fff640 1 -- 172.21.15.67:0/872700618 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be000a840 con 0x7f2bec0b2f30 2024-03-24T00:47:14.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.589+0000 7f2bf2085640 1 -- 172.21.15.67:0/872700618 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bec15c440 con 0x7f2bec0b2f30 2024-03-24T00:47:14.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.590+0000 7f2bf2085640 1 -- 172.21.15.67:0/872700618 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bec04f400 con 0x7f2bec0b2f30 2024-03-24T00:47:14.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.590+0000 7f2bd3fff640 1 -- 172.21.15.67:0/872700618 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2be001a070 con 0x7f2bec0b2f30 2024-03-24T00:47:14.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.591+0000 7f2bd3fff640 1 --2- 172.21.15.67:0/872700618 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2bd806bbd0 0x7f2bd806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:14.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.591+0000 7f2bd3fff640 1 -- 172.21.15.67:0/872700618 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f2be0089cb0 con 0x7f2bec0b2f30 2024-03-24T00:47:14.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.591+0000 7f2beb7fe640 1 --2- 172.21.15.67:0/872700618 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2bd806bbd0 0x7f2bd806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:14.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.592+0000 7f2beb7fe640 1 --2- 172.21.15.67:0/872700618 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2bd806bbd0 0x7f2bd806e090 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc001eb0 tx=0x7f2bdc001d80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:14.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.595+0000 7f2bd3fff640 1 -- 172.21.15.67:0/872700618 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2be00583a0 con 0x7f2bec0b2f30 2024-03-24T00:47:14.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.779+0000 7f2bf2085640 1 -- 172.21.15.67:0/872700618 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f2bec06fd40 con 0x7f2bec0b2f30 2024-03-24T00:47:14.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.779+0000 7f2bd3fff640 1 -- 172.21.15.67:0/872700618 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v101) v1 ==== 84+0+65 (secure 0 0 0) 0x7f2be005c440 con 0x7f2bec0b2f30 2024-03-24T00:47:14.779 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:47:14.779 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-03-24T00:47:14.779 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:47:14.779 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2024-03-24T00:47:14.780 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2024-03-24T00:47:14.780 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:47:14.780 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-03-24T00:47:14.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.781+0000 7f2bf2085640 1 -- 172.21.15.67:0/872700618 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2bd806bbd0 msgr2=0x7f2bd806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:14.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.781+0000 7f2bf2085640 1 --2- 172.21.15.67:0/872700618 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2bd806bbd0 0x7f2bd806e090 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc001eb0 tx=0x7f2bdc001d80 comp rx=0 tx=0).stop 2024-03-24T00:47:14.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.781+0000 7f2bf2085640 1 -- 172.21.15.67:0/872700618 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2bec0b2f30 msgr2=0x7f2bec1589a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:14.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.781+0000 7f2bf2085640 1 --2- 172.21.15.67:0/872700618 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2bec0b2f30 0x7f2bec1589a0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f2be000c780 tx=0x7f2be000cc40 comp rx=0 tx=0).stop 2024-03-24T00:47:14.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.781+0000 7f2bf2085640 1 -- 172.21.15.67:0/872700618 shutdown_connections 2024-03-24T00:47:14.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.781+0000 7f2bf2085640 1 --2- 172.21.15.67:0/872700618 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2bd806bbd0 0x7f2bd806e090 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:14.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.781+0000 7f2bf2085640 1 --2- 172.21.15.67:0/872700618 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2bec15bb20 0x7f2bec15a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:14.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.781+0000 7f2bf2085640 1 --2- 172.21.15.67:0/872700618 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2bec0b2f30 0x7f2bec1589a0 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:14.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.781+0000 7f2bf2085640 1 --2- 172.21.15.67:0/872700618 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2bec070670 0x7f2bec15bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:14.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.781+0000 7f2bf2085640 1 -- 172.21.15.67:0/872700618 >> 172.21.15.67:0/872700618 conn(0x7f2bec071ef0 msgr2=0x7f2bec0b5210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:14.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.781+0000 7f2bf2085640 1 -- 172.21.15.67:0/872700618 shutdown_connections 2024-03-24T00:47:14.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.781+0000 7f2bf2085640 1 -- 172.21.15.67:0/872700618 wait complete. 2024-03-24T00:47:14.792 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-24T00:47:14.792 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-24T00:47:14.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.897+0000 7fec304dc640 1 Processor -- start 2024-03-24T00:47:14.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.897+0000 7fec304dc640 1 -- start start 2024-03-24T00:47:14.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.898+0000 7fec304dc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fec2815e1b0 0x7fec280b6230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:14.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.898+0000 7fec304dc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fec280d0e60 0x7fec280d1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:14.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.898+0000 7fec304dc640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec28161d60 con 0x7fec280b2d80 2024-03-24T00:47:14.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.898+0000 7fec304dc640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec28162970 con 0x7fec2815e1b0 2024-03-24T00:47:14.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.898+0000 7fec304dc640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec28162040 con 0x7fec280d0e60 2024-03-24T00:47:14.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.898+0000 7fec2e278640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fec2815e1b0 0x7fec280b6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:14.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.898+0000 7fec2da77640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fec280b2d80 0x7fec280d0750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48246/0 (socket says 172.21.15.67:48246) 2024-03-24T00:47:14.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.898+0000 7fec2da77640 1 -- 172.21.15.67:0/2441839663 learned_addr learned my addr 172.21.15.67:0/2441839663 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:47:14.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.898+0000 7fec2e278640 1 -- 172.21.15.67:0/2441839663 >> v2:172.21.15.72:3300/0 conn(0x7fec280d0e60 msgr2=0x7fec280d1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:14.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.898+0000 7fec2e278640 1 --2- 172.21.15.67:0/2441839663 >> v2:172.21.15.72:3300/0 conn(0x7fec280d0e60 0x7fec280d1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:14.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.898+0000 7fec2e278640 1 -- 172.21.15.67:0/2441839663 >> v1:172.21.15.67:6789/0 conn(0x7fec280b2d80 legacy=0x7fec280d0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:14.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.898+0000 7fec2e278640 1 -- 172.21.15.67:0/2441839663 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec28162320 con 0x7fec2815e1b0 2024-03-24T00:47:14.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.898+0000 7fec2d276640 1 -- 172.21.15.67:0/2441839663 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 43810451 0 0) 0x7fec28161d60 con 0x7fec280b2d80 2024-03-24T00:47:14.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.899+0000 7fec2e278640 1 --2- 172.21.15.67:0/2441839663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec2815e1b0 0x7fec280b6230 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7fec28162e00 tx=0x7fec2002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26fd4c92e367c523 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:14.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.899+0000 7fec2d276640 1 -- 172.21.15.67:0/2441839663 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec200317c0 con 0x7fec2815e1b0 2024-03-24T00:47:14.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.899+0000 7fec2d276640 1 -- 172.21.15.67:0/2441839663 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec20037070 con 0x7fec2815e1b0 2024-03-24T00:47:14.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.899+0000 7fec2d276640 1 -- 172.21.15.67:0/2441839663 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec20031b80 con 0x7fec2815e1b0 2024-03-24T00:47:14.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.900+0000 7fec304dc640 1 -- 172.21.15.67:0/2441839663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec2815e1b0 msgr2=0x7fec280b6230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:14.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.900+0000 7fec304dc640 1 --2- 172.21.15.67:0/2441839663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec2815e1b0 0x7fec280b6230 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7fec28162e00 tx=0x7fec2002b860 comp rx=0 tx=0).stop 2024-03-24T00:47:14.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.900+0000 7fec304dc640 1 -- 172.21.15.67:0/2441839663 shutdown_connections 2024-03-24T00:47:14.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.900+0000 7fec304dc640 1 --2- 172.21.15.67:0/2441839663 >> v2:172.21.15.72:3300/0 conn(0x7fec280d0e60 0x7fec280d1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:14.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.900+0000 7fec304dc640 1 --2- 172.21.15.67:0/2441839663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec2815e1b0 0x7fec280b6230 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:14.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.900+0000 7fec304dc640 1 -- 172.21.15.67:0/2441839663 >> 172.21.15.67:0/2441839663 conn(0x7fec2806c8f0 msgr2=0x7fec2806ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:14.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.900+0000 7fec304dc640 1 -- 172.21.15.67:0/2441839663 shutdown_connections 2024-03-24T00:47:14.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.900+0000 7fec304dc640 1 -- 172.21.15.67:0/2441839663 wait complete. 2024-03-24T00:47:14.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.901+0000 7fec304dc640 1 Processor -- start 2024-03-24T00:47:14.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.901+0000 7fec304dc640 1 -- start start 2024-03-24T00:47:14.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.902+0000 7fec304dc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec280b2d80 0x7fec28152290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:14.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.902+0000 7fec304dc640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fec280d0e60 0x7fec2815ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:14.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.902+0000 7fec304dc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec2815e1b0 0x7fec281544c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:14.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.902+0000 7fec304dc640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec2815e590 con 0x7fec2815e1b0 2024-03-24T00:47:14.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.902+0000 7fec304dc640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec280b3570 con 0x7fec280b2d80 2024-03-24T00:47:14.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.902+0000 7fec304dc640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec280d2510 con 0x7fec280d0e60 2024-03-24T00:47:14.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.902+0000 7fec2ea79640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec2815e1b0 0x7fec281544c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:14.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.902+0000 7fec2ea79640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec2815e1b0 0x7fec281544c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47106/0 (socket says 172.21.15.67:47106) 2024-03-24T00:47:14.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.902+0000 7fec2ea79640 1 -- 172.21.15.67:0/714465622 learned_addr learned my addr 172.21.15.67:0/714465622 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:14.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.902+0000 7fec2da77640 1 --2- 172.21.15.67:0/714465622 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fec280d0e60 0x7fec2815ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:14.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.902+0000 7fec2e278640 1 --2- 172.21.15.67:0/714465622 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec280b2d80 0x7fec28152290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:14.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.902+0000 7fec2ea79640 1 -- 172.21.15.67:0/714465622 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fec280d0e60 msgr2=0x7fec2815ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:14.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.902+0000 7fec2ea79640 1 --2- 172.21.15.67:0/714465622 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fec280d0e60 0x7fec2815ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:14.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.902+0000 7fec2ea79640 1 -- 172.21.15.67:0/714465622 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec280b2d80 msgr2=0x7fec28152290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:14.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.902+0000 7fec2ea79640 1 --2- 172.21.15.67:0/714465622 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec280b2d80 0x7fec28152290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:14.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.902+0000 7fec2ea79640 1 -- 172.21.15.67:0/714465622 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec280cbed0 con 0x7fec2815e1b0 2024-03-24T00:47:14.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.902+0000 7fec2e278640 1 --2- 172.21.15.67:0/714465622 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec280b2d80 0x7fec28152290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:47:14.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.902+0000 7fec2ea79640 1 --2- 172.21.15.67:0/714465622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec2815e1b0 0x7fec281544c0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7fec2400b9e0 tx=0x7fec2400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:14.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.903+0000 7fec1effd640 1 -- 172.21.15.67:0/714465622 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec24019070 con 0x7fec2815e1b0 2024-03-24T00:47:14.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.903+0000 7fec1effd640 1 -- 172.21.15.67:0/714465622 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec2400cb20 con 0x7fec2815e1b0 2024-03-24T00:47:14.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.903+0000 7fec304dc640 1 -- 172.21.15.67:0/714465622 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec280cc1b0 con 0x7fec2815e1b0 2024-03-24T00:47:14.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.903+0000 7fec1effd640 1 -- 172.21.15.67:0/714465622 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec24014440 con 0x7fec2815e1b0 2024-03-24T00:47:14.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.903+0000 7fec304dc640 1 -- 172.21.15.67:0/714465622 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec280cc770 con 0x7fec2815e1b0 2024-03-24T00:47:14.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.904+0000 7fec304dc640 1 -- 172.21.15.67:0/714465622 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec2804f400 con 0x7fec2815e1b0 2024-03-24T00:47:14.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.904+0000 7fec1effd640 1 -- 172.21.15.67:0/714465622 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fec2401c070 con 0x7fec2815e1b0 2024-03-24T00:47:14.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.905+0000 7fec1effd640 1 --2- 172.21.15.67:0/714465622 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fec0c06bc00 0x7fec0c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:14.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.905+0000 7fec1effd640 1 -- 172.21.15.67:0/714465622 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fec2408a490 con 0x7fec2815e1b0 2024-03-24T00:47:14.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.905+0000 7fec2e278640 1 --2- 172.21.15.67:0/714465622 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fec0c06bc00 0x7fec0c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:14.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.905+0000 7fec2e278640 1 --2- 172.21.15.67:0/714465622 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fec0c06bc00 0x7fec0c06e0c0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fec20036000 tx=0x7fec2003d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:14.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:14.909+0000 7fec1effd640 1 -- 172.21.15.67:0/714465622 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fec24058af0 con 0x7fec2815e1b0 2024-03-24T00:47:15.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.092+0000 7fec304dc640 1 -- 172.21.15.67:0/714465622 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fec280d1b80 con 0x7fec2815e1b0 2024-03-24T00:47:15.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.119+0000 7fec1effd640 1 -- 172.21.15.67:0/714465622 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v102) v1 ==== 138+0+0 (secure 0 0 0) 0x7fec2405cb90 con 0x7fec2815e1b0 2024-03-24T00:47:15.119 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' created 2024-03-24T00:47:15.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.121+0000 7fec304dc640 1 -- 172.21.15.67:0/714465622 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fec0c06bc00 msgr2=0x7fec0c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:15.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.121+0000 7fec304dc640 1 --2- 172.21.15.67:0/714465622 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fec0c06bc00 0x7fec0c06e0c0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fec20036000 tx=0x7fec2003d040 comp rx=0 tx=0).stop 2024-03-24T00:47:15.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.121+0000 7fec304dc640 1 -- 172.21.15.67:0/714465622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec2815e1b0 msgr2=0x7fec281544c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:15.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.121+0000 7fec304dc640 1 --2- 172.21.15.67:0/714465622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec2815e1b0 0x7fec281544c0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7fec2400b9e0 tx=0x7fec2400bea0 comp rx=0 tx=0).stop 2024-03-24T00:47:15.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.121+0000 7fec304dc640 1 -- 172.21.15.67:0/714465622 shutdown_connections 2024-03-24T00:47:15.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.121+0000 7fec304dc640 1 --2- 172.21.15.67:0/714465622 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fec0c06bc00 0x7fec0c06e0c0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:15.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.121+0000 7fec304dc640 1 --2- 172.21.15.67:0/714465622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec2815e1b0 0x7fec281544c0 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:15.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.121+0000 7fec304dc640 1 --2- 172.21.15.67:0/714465622 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fec280d0e60 0x7fec2815ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:15.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.121+0000 7fec304dc640 1 --2- 172.21.15.67:0/714465622 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec280b2d80 0x7fec28152290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:15.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.121+0000 7fec304dc640 1 -- 172.21.15.67:0/714465622 >> 172.21.15.67:0/714465622 conn(0x7fec2806c8f0 msgr2=0x7fec2815d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:15.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.121+0000 7fec304dc640 1 -- 172.21.15.67:0/714465622 shutdown_connections 2024-03-24T00:47:15.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.121+0000 7fec304dc640 1 -- 172.21.15.67:0/714465622 wait complete. 2024-03-24T00:47:15.135 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-24T00:47:15.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.243+0000 7f541ce9a640 1 Processor -- start 2024-03-24T00:47:15.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.244+0000 7f541ce9a640 1 -- start start 2024-03-24T00:47:15.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.244+0000 7f541ce9a640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f541815e170 0x7f54180d21c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:15.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.244+0000 7f541ce9a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f54180cf2d0 0x7f54180b6f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:15.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.244+0000 7f541ce9a640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54180d2af0 con 0x7f54180b9b50 2024-03-24T00:47:15.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.244+0000 7f541ce9a640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54180d2dd0 con 0x7f54180cf2d0 2024-03-24T00:47:15.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.244+0000 7f541ce9a640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54180d30b0 con 0x7f541815e170 2024-03-24T00:47:15.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.244+0000 7f5416d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f54180cf2d0 0x7f54180b6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:15.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.244+0000 7f540ffff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f54180b9b50 0x7f54180cebc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48248/0 (socket says 172.21.15.67:48248) 2024-03-24T00:47:15.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.244+0000 7f5416d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f54180cf2d0 0x7f54180b6f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47120/0 (socket says 172.21.15.67:47120) 2024-03-24T00:47:15.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.244+0000 7f540ffff640 1 -- 172.21.15.67:0/1247939036 learned_addr learned my addr 172.21.15.67:0/1247939036 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:47:15.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.245+0000 7f541659c640 1 --2- 172.21.15.67:0/1247939036 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f541815e170 0x7f54180d21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:15.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.245+0000 7f5416d9d640 1 -- 172.21.15.67:0/1247939036 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f541815e170 msgr2=0x7f54180d21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:15.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.245+0000 7f5416d9d640 1 --2- 172.21.15.67:0/1247939036 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f541815e170 0x7f54180d21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:15.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.245+0000 7f5416d9d640 1 -- 172.21.15.67:0/1247939036 >> v1:172.21.15.67:6789/0 conn(0x7f54180b9b50 legacy=0x7f54180cebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:15.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.245+0000 7f5416d9d640 1 -- 172.21.15.67:0/1247939036 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54180d3390 con 0x7f54180cf2d0 2024-03-24T00:47:15.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.245+0000 7f5415d9b640 1 -- 172.21.15.67:0/1247939036 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1708030885 0 0) 0x7f54180d2af0 con 0x7f54180b9b50 2024-03-24T00:47:15.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.246+0000 7f5416d9d640 1 --2- 172.21.15.67:0/1247939036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54180cf2d0 0x7f54180b6f20 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f5410009d90 tx=0x7f541002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ec727df2efb0175 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:15.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.246+0000 7f5415d9b640 1 -- 172.21.15.67:0/1247939036 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5410039070 con 0x7f54180cf2d0 2024-03-24T00:47:15.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.246+0000 7f5415d9b640 1 -- 172.21.15.67:0/1247939036 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f541002cc50 con 0x7f54180cf2d0 2024-03-24T00:47:15.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.246+0000 7f5415d9b640 1 -- 172.21.15.67:0/1247939036 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5410034440 con 0x7f54180cf2d0 2024-03-24T00:47:15.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.246+0000 7f541ce9a640 1 -- 172.21.15.67:0/1247939036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54180cf2d0 msgr2=0x7f54180b6f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:15.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.246+0000 7f541ce9a640 1 --2- 172.21.15.67:0/1247939036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54180cf2d0 0x7f54180b6f20 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f5410009d90 tx=0x7f541002bea0 comp rx=0 tx=0).stop 2024-03-24T00:47:15.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.246+0000 7f541ce9a640 1 -- 172.21.15.67:0/1247939036 shutdown_connections 2024-03-24T00:47:15.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.246+0000 7f541ce9a640 1 --2- 172.21.15.67:0/1247939036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54180cf2d0 0x7f54180b6f20 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:15.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.246+0000 7f541ce9a640 1 --2- 172.21.15.67:0/1247939036 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f541815e170 0x7f54180d21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:15.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.246+0000 7f541ce9a640 1 -- 172.21.15.67:0/1247939036 >> 172.21.15.67:0/1247939036 conn(0x7f541806c810 msgr2=0x7f541806cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:15.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.246+0000 7f541ce9a640 1 -- 172.21.15.67:0/1247939036 shutdown_connections 2024-03-24T00:47:15.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.246+0000 7f541ce9a640 1 -- 172.21.15.67:0/1247939036 wait complete. 2024-03-24T00:47:15.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.248+0000 7f541ce9a640 1 Processor -- start 2024-03-24T00:47:15.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.248+0000 7f541ce9a640 1 -- start start 2024-03-24T00:47:15.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.248+0000 7f541ce9a640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f54180b9b50 0x7f54180c72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:15.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.248+0000 7f541ce9a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54180cf2d0 0x7f54181e7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:15.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.248+0000 7f541ce9a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f541815e170 0x7f541814a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:15.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.248+0000 7f541ce9a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54180d2880 con 0x7f54180cf2d0 2024-03-24T00:47:15.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.248+0000 7f541ce9a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54180d2700 con 0x7f541815e170 2024-03-24T00:47:15.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.248+0000 7f541ce9a640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54180b7460 con 0x7f54180b9b50 2024-03-24T00:47:15.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.248+0000 7f540ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54180cf2d0 0x7f54181e7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:15.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.248+0000 7f540ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54180cf2d0 0x7f54181e7470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47136/0 (socket says 172.21.15.67:47136) 2024-03-24T00:47:15.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.248+0000 7f540ffff640 1 -- 172.21.15.67:0/3604301067 learned_addr learned my addr 172.21.15.67:0/3604301067 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:15.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.248+0000 7f5416d9d640 1 --2- 172.21.15.67:0/3604301067 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f541815e170 0x7f541814a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:15.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.248+0000 7f541659c640 1 --2- 172.21.15.67:0/3604301067 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f54180b9b50 0x7f54180c72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:15.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.249+0000 7f540ffff640 1 -- 172.21.15.67:0/3604301067 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f54180b9b50 msgr2=0x7f54180c72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:15.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.249+0000 7f540ffff640 1 --2- 172.21.15.67:0/3604301067 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f54180b9b50 0x7f54180c72b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:15.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.249+0000 7f540ffff640 1 -- 172.21.15.67:0/3604301067 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f541815e170 msgr2=0x7f541814a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:15.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.249+0000 7f540ffff640 1 --2- 172.21.15.67:0/3604301067 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f541815e170 0x7f541814a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:15.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.249+0000 7f540ffff640 1 -- 172.21.15.67:0/3604301067 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5418149140 con 0x7f54180cf2d0 2024-03-24T00:47:15.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.249+0000 7f540ffff640 1 --2- 172.21.15.67:0/3604301067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54180cf2d0 0x7f54181e7470 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f540000c4d0 tx=0x7f540000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:15.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.249+0000 7f540effd640 1 -- 172.21.15.67:0/3604301067 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5400018070 con 0x7f54180cf2d0 2024-03-24T00:47:15.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.249+0000 7f540effd640 1 -- 172.21.15.67:0/3604301067 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5400007500 con 0x7f54180cf2d0 2024-03-24T00:47:15.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.249+0000 7f540effd640 1 -- 172.21.15.67:0/3604301067 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5400004a20 con 0x7f54180cf2d0 2024-03-24T00:47:15.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.249+0000 7f541ce9a640 1 -- 172.21.15.67:0/3604301067 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5418149420 con 0x7f54180cf2d0 2024-03-24T00:47:15.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.249+0000 7f541ce9a640 1 -- 172.21.15.67:0/3604301067 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54181499e0 con 0x7f54180cf2d0 2024-03-24T00:47:15.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.250+0000 7f541ce9a640 1 -- 172.21.15.67:0/3604301067 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f541804f400 con 0x7f54180cf2d0 2024-03-24T00:47:15.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.251+0000 7f540effd640 1 -- 172.21.15.67:0/3604301067 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f540001b070 con 0x7f54180cf2d0 2024-03-24T00:47:15.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.251+0000 7f540effd640 1 --2- 172.21.15.67:0/3604301067 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f53f806bbd0 0x7f53f806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:15.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.251+0000 7f540effd640 1 -- 172.21.15.67:0/3604301067 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f5400089c40 con 0x7f54180cf2d0 2024-03-24T00:47:15.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.251+0000 7f541659c640 1 --2- 172.21.15.67:0/3604301067 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f53f806bbd0 0x7f53f806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:15.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.252+0000 7f541659c640 1 --2- 172.21.15.67:0/3604301067 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f53f806bbd0 0x7f53f806e090 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f54080079a0 tx=0x7f5408008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:15.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.256+0000 7f540effd640 1 -- 172.21.15.67:0/3604301067 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f54000581c0 con 0x7f54180cf2d0 2024-03-24T00:47:15.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:15.436+0000 7f541ce9a640 1 -- 172.21.15.67:0/3604301067 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f54180b63d0 con 0x7f54180cf2d0 2024-03-24T00:47:16.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.125+0000 7f540effd640 1 -- 172.21.15.67:0/3604301067 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v103) v1 ==== 170+0+0 (secure 0 0 0) 0x7f540005c260 con 0x7f54180cf2d0 2024-03-24T00:47:16.125 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' created 2024-03-24T00:47:16.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.127+0000 7f541ce9a640 1 -- 172.21.15.67:0/3604301067 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f53f806bbd0 msgr2=0x7f53f806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:16.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.127+0000 7f541ce9a640 1 --2- 172.21.15.67:0/3604301067 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f53f806bbd0 0x7f53f806e090 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f54080079a0 tx=0x7f5408008040 comp rx=0 tx=0).stop 2024-03-24T00:47:16.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.127+0000 7f541ce9a640 1 -- 172.21.15.67:0/3604301067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54180cf2d0 msgr2=0x7f54181e7470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:16.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.127+0000 7f541ce9a640 1 --2- 172.21.15.67:0/3604301067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54180cf2d0 0x7f54181e7470 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f540000c4d0 tx=0x7f540000c990 comp rx=0 tx=0).stop 2024-03-24T00:47:16.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.128+0000 7f541ce9a640 1 -- 172.21.15.67:0/3604301067 shutdown_connections 2024-03-24T00:47:16.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.128+0000 7f541ce9a640 1 --2- 172.21.15.67:0/3604301067 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f53f806bbd0 0x7f53f806e090 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:16.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.128+0000 7f541ce9a640 1 --2- 172.21.15.67:0/3604301067 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f541815e170 0x7f541814a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:16.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.128+0000 7f541ce9a640 1 --2- 172.21.15.67:0/3604301067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54180cf2d0 0x7f54181e7470 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:16.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.128+0000 7f541ce9a640 1 --2- 172.21.15.67:0/3604301067 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f54180b9b50 0x7f54180c72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:16.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.128+0000 7f541ce9a640 1 -- 172.21.15.67:0/3604301067 >> 172.21.15.67:0/3604301067 conn(0x7f541806c810 msgr2=0x7f54180b5820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:16.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.128+0000 7f541ce9a640 1 -- 172.21.15.67:0/3604301067 shutdown_connections 2024-03-24T00:47:16.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.128+0000 7f541ce9a640 1 -- 172.21.15.67:0/3604301067 wait complete. 2024-03-24T00:47:16.148 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-24T00:47:16.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.257+0000 7f490b313640 1 Processor -- start 2024-03-24T00:47:16.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.257+0000 7f490b313640 1 -- start start 2024-03-24T00:47:16.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.258+0000 7f490b313640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f490406b750 0x7f490415e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:16.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.258+0000 7f490b313640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49040d1630 con 0x7f49040af4e0 2024-03-24T00:47:16.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.258+0000 7f490b313640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49040ad9f0 con 0x7f490415de90 2024-03-24T00:47:16.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.258+0000 7f490b313640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49040d1910 con 0x7f490406b750 2024-03-24T00:47:16.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.258+0000 7f49088ae640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f49040af4e0 0x7f49040cfd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48252/0 (socket says 172.21.15.67:48252) 2024-03-24T00:47:16.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.258+0000 7f49088ae640 1 -- 172.21.15.67:0/2148647810 learned_addr learned my addr 172.21.15.67:0/2148647810 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:47:16.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.258+0000 7f49098b0640 1 --2- 172.21.15.67:0/2148647810 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f490406b750 0x7f490415e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:16.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.259+0000 7f4903fff640 1 -- 172.21.15.67:0/2148647810 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1566651217 0 0) 0x7f49040d1630 con 0x7f49040af4e0 2024-03-24T00:47:16.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.259+0000 7f4903fff640 1 -- 172.21.15.67:0/2148647810 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48ec003520 con 0x7f49040af4e0 2024-03-24T00:47:16.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.259+0000 7f4903fff640 1 -- 172.21.15.67:0/2148647810 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3215065858 0 0) 0x7f49040ad9f0 con 0x7f490415de90 2024-03-24T00:47:16.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.259+0000 7f4903fff640 1 -- 172.21.15.67:0/2148647810 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49040d1630 con 0x7f490415de90 2024-03-24T00:47:16.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.259+0000 7f4903fff640 1 -- 172.21.15.67:0/2148647810 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3283501402 0 0) 0x7f48ec003520 con 0x7f49040af4e0 2024-03-24T00:47:16.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.259+0000 7f49098b0640 1 -- 172.21.15.67:0/2148647810 >> v1:172.21.15.72:6789/0 conn(0x7f490415de90 legacy=0x7f490406b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:16.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.259+0000 7f49098b0640 1 -- 172.21.15.67:0/2148647810 >> v1:172.21.15.67:6789/0 conn(0x7f49040af4e0 legacy=0x7f49040cfd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:16.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.259+0000 7f49098b0640 1 -- 172.21.15.67:0/2148647810 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49040adcd0 con 0x7f490406b750 2024-03-24T00:47:16.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.260+0000 7f49098b0640 1 --2- 172.21.15.67:0/2148647810 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f490406b750 0x7f490415e270 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f48f4009dc0 tx=0x7f48f402bd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7aae3fb520fa110d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:16.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.260+0000 7f4903fff640 1 -- 172.21.15.67:0/2148647810 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f4039070 con 0x7f490406b750 2024-03-24T00:47:16.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.260+0000 7f4903fff640 1 -- 172.21.15.67:0/2148647810 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48f402c880 con 0x7f490406b750 2024-03-24T00:47:16.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.260+0000 7f4903fff640 1 -- 172.21.15.67:0/2148647810 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f8001670 con 0x7f490406b750 2024-03-24T00:47:16.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.260+0000 7f490b313640 1 -- 172.21.15.67:0/2148647810 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f490406b750 msgr2=0x7f490415e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:16.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.260+0000 7f490b313640 1 --2- 172.21.15.67:0/2148647810 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f490406b750 0x7f490415e270 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f48f4009dc0 tx=0x7f48f402bd50 comp rx=0 tx=0).stop 2024-03-24T00:47:16.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.260+0000 7f490b313640 1 -- 172.21.15.67:0/2148647810 shutdown_connections 2024-03-24T00:47:16.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.260+0000 7f490b313640 1 --2- 172.21.15.67:0/2148647810 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f490406b750 0x7f490415e270 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:16.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.260+0000 7f490b313640 1 -- 172.21.15.67:0/2148647810 >> 172.21.15.67:0/2148647810 conn(0x7f4904099a30 msgr2=0x7f4904099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:16.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.260+0000 7f490b313640 1 -- 172.21.15.67:0/2148647810 shutdown_connections 2024-03-24T00:47:16.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.260+0000 7f490b313640 1 -- 172.21.15.67:0/2148647810 wait complete. 2024-03-24T00:47:16.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.261+0000 7f490b313640 1 Processor -- start 2024-03-24T00:47:16.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.262+0000 7f490b313640 1 -- start start 2024-03-24T00:47:16.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.262+0000 7f490b313640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f490406b750 0x7f4904151d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:16.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.262+0000 7f490b313640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49040af4e0 0x7f49041592d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:16.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.262+0000 7f490b313640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f490415de90 0x7f49040c67e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:16.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.262+0000 7f490b313640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49040cd460 con 0x7f49040af4e0 2024-03-24T00:47:16.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.262+0000 7f490b313640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49040cd2e0 con 0x7f490415de90 2024-03-24T00:47:16.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.262+0000 7f490b313640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49040cd5e0 con 0x7f490406b750 2024-03-24T00:47:16.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.262+0000 7f49088ae640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49040af4e0 0x7f49041592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:16.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.262+0000 7f49088ae640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49040af4e0 0x7f49041592d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47138/0 (socket says 172.21.15.67:47138) 2024-03-24T00:47:16.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.262+0000 7f49088ae640 1 -- 172.21.15.67:0/3203802022 learned_addr learned my addr 172.21.15.67:0/3203802022 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:16.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.262+0000 7f49090af640 1 --2- 172.21.15.67:0/3203802022 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f490406b750 0x7f4904151d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:16.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.262+0000 7f49098b0640 1 --2- 172.21.15.67:0/3203802022 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f490415de90 0x7f49040c67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:16.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.263+0000 7f49088ae640 1 -- 172.21.15.67:0/3203802022 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f490406b750 msgr2=0x7f4904151d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:16.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.263+0000 7f49088ae640 1 --2- 172.21.15.67:0/3203802022 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f490406b750 0x7f4904151d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:16.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.263+0000 7f49088ae640 1 -- 172.21.15.67:0/3203802022 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f490415de90 msgr2=0x7f49040c67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:16.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.263+0000 7f49088ae640 1 --2- 172.21.15.67:0/3203802022 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f490415de90 0x7f49040c67e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:16.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.263+0000 7f49088ae640 1 -- 172.21.15.67:0/3203802022 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f490414fbe0 con 0x7f49040af4e0 2024-03-24T00:47:16.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.263+0000 7f49088ae640 1 --2- 172.21.15.67:0/3203802022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49040af4e0 0x7f49041592d0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f48f800c5d0 tx=0x7f48f800ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:16.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.263+0000 7f4901ffb640 1 -- 172.21.15.67:0/3203802022 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f8017070 con 0x7f49040af4e0 2024-03-24T00:47:16.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.263+0000 7f4901ffb640 1 -- 172.21.15.67:0/3203802022 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48f8005ce0 con 0x7f49040af4e0 2024-03-24T00:47:16.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.263+0000 7f490b313640 1 -- 172.21.15.67:0/3203802022 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4904158a40 con 0x7f49040af4e0 2024-03-24T00:47:16.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.263+0000 7f4901ffb640 1 -- 172.21.15.67:0/3203802022 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f80072b0 con 0x7f49040af4e0 2024-03-24T00:47:16.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.263+0000 7f490b313640 1 -- 172.21.15.67:0/3203802022 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4904159000 con 0x7f49040af4e0 2024-03-24T00:47:16.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.264+0000 7f490b313640 1 -- 172.21.15.67:0/3203802022 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f490404f400 con 0x7f49040af4e0 2024-03-24T00:47:16.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.265+0000 7f4901ffb640 1 -- 172.21.15.67:0/3203802022 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f48f801a070 con 0x7f49040af4e0 2024-03-24T00:47:16.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.265+0000 7f4901ffb640 1 --2- 172.21.15.67:0/3203802022 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f48f006bbd0 0x7f48f006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:16.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.265+0000 7f4901ffb640 1 -- 172.21.15.67:0/3203802022 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f48f808a0c0 con 0x7f49040af4e0 2024-03-24T00:47:16.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.265+0000 7f49090af640 1 --2- 172.21.15.67:0/3203802022 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f48f006bbd0 0x7f48f006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:16.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.266+0000 7f49090af640 1 --2- 172.21.15.67:0/3203802022 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f48f006bbd0 0x7f48f006e090 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f48fc0011e0 tx=0x7f48fc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:16.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.269+0000 7f4901ffb640 1 -- 172.21.15.67:0/3203802022 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f48f8058410 con 0x7f49040af4e0 2024-03-24T00:47:16.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:16.491+0000 7f490b313640 1 -- 172.21.15.67:0/3203802022 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f49040cead0 con 0x7f49040af4e0 2024-03-24T00:47:16.949 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:47:16.952 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:47:17.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.128+0000 7f4901ffb640 1 -- 172.21.15.67:0/3203802022 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 15 and data pool 16 v70) v1 ==== 174+0+0 (secure 0 0 0) 0x7f48f805c4b0 con 0x7f49040af4e0 2024-03-24T00:47:17.128 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 15 and data pool 16 2024-03-24T00:47:17.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.133+0000 7f490b313640 1 -- 172.21.15.67:0/3203802022 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f48f006bbd0 msgr2=0x7f48f006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:17.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.133+0000 7f490b313640 1 --2- 172.21.15.67:0/3203802022 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f48f006bbd0 0x7f48f006e090 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f48fc0011e0 tx=0x7f48fc009040 comp rx=0 tx=0).stop 2024-03-24T00:47:17.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.133+0000 7f490b313640 1 -- 172.21.15.67:0/3203802022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49040af4e0 msgr2=0x7f49041592d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:17.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.133+0000 7f490b313640 1 --2- 172.21.15.67:0/3203802022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49040af4e0 0x7f49041592d0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f48f800c5d0 tx=0x7f48f800ca90 comp rx=0 tx=0).stop 2024-03-24T00:47:17.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.134+0000 7f490b313640 1 -- 172.21.15.67:0/3203802022 shutdown_connections 2024-03-24T00:47:17.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.134+0000 7f490b313640 1 --2- 172.21.15.67:0/3203802022 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f48f006bbd0 0x7f48f006e090 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:17.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.134+0000 7f490b313640 1 --2- 172.21.15.67:0/3203802022 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f490415de90 0x7f49040c67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:17.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.134+0000 7f490b313640 1 --2- 172.21.15.67:0/3203802022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49040af4e0 0x7f49041592d0 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:17.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.134+0000 7f490b313640 1 --2- 172.21.15.67:0/3203802022 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f490406b750 0x7f4904151d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:17.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.134+0000 7f490b313640 1 -- 172.21.15.67:0/3203802022 >> 172.21.15.67:0/3203802022 conn(0x7f4904099a30 msgr2=0x7f49040d0580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:17.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.141+0000 7f490b313640 1 -- 172.21.15.67:0/3203802022 shutdown_connections 2024-03-24T00:47:17.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.141+0000 7f490b313640 1 -- 172.21.15.67:0/3203802022 wait complete. 2024-03-24T00:47:17.170 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:47:17.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.275+0000 7f5cbcd10640 1 Processor -- start 2024-03-24T00:47:17.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.275+0000 7f5cbcd10640 1 -- start start 2024-03-24T00:47:17.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.276+0000 7f5cbcd10640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5cb815e2a0 0x7f5cb80ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:17.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.276+0000 7f5cbcd10640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5cb80d0ef0 0x7f5cb8162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:17.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.276+0000 7f5cbcd10640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cb80cf640 con 0x7f5cb815dec0 2024-03-24T00:47:17.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.276+0000 7f5cbcd10640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cb80cee30 con 0x7f5cb815e2a0 2024-03-24T00:47:17.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.276+0000 7f5cbcd10640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb80d2120 con 0x7f5cb80d0ef0 2024-03-24T00:47:17.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.276+0000 7f5cb6d9d640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5cb80d0ef0 0x7f5cb8162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:17.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.276+0000 7f5cb5d9b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5cb815e2a0 0x7f5cb80ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:17.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.276+0000 7f5cb6d9d640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5cb80d0ef0 0x7f5cb8162630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.67:38534/0 (socket says 172.21.15.67:38534) 2024-03-24T00:47:17.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.277+0000 7f5cb6d9d640 1 -- 172.21.15.67:0/2293948333 learned_addr learned my addr 172.21.15.67:0/2293948333 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:17.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.277+0000 7f5cb559a640 1 -- 172.21.15.67:0/2293948333 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2758935179 0 0) 0x7f5ca8001490 con 0x7f5cb815dec0 2024-03-24T00:47:17.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.277+0000 7f5cb559a640 1 -- 172.21.15.67:0/2293948333 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ca4003520 con 0x7f5cb815dec0 2024-03-24T00:47:17.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.277+0000 7f5cb6d9d640 1 -- 172.21.15.67:0/2293948333 >> v2:172.21.15.72:3300/0 conn(0x7f5cb815e2a0 msgr2=0x7f5cb80ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:17.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.277+0000 7f5cb6d9d640 1 --2- 172.21.15.67:0/2293948333 >> v2:172.21.15.72:3300/0 conn(0x7f5cb815e2a0 0x7f5cb80ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:17.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.277+0000 7f5cb6d9d640 1 -- 172.21.15.67:0/2293948333 >> v1:172.21.15.72:6789/0 conn(0x7f5cb815dec0 legacy=0x7f5cb80d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:17.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.277+0000 7f5cb6d9d640 1 -- 172.21.15.67:0/2293948333 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cb80cf080 con 0x7f5cb80d0ef0 2024-03-24T00:47:17.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.277+0000 7f5cb5d9b640 1 --2- 172.21.15.67:0/2293948333 >> v2:172.21.15.72:3300/0 conn(0x7f5cb815e2a0 0x7f5cb80ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:47:17.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.278+0000 7f5cb6d9d640 1 --2- 172.21.15.67:0/2293948333 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5cb80d0ef0 0x7f5cb8162630 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5cb0009870 tx=0x7f5cb002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75a213e9710bfee7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:17.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.278+0000 7f5cb559a640 1 -- 172.21.15.67:0/2293948333 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb0035020 con 0x7f5cb80d0ef0 2024-03-24T00:47:17.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.278+0000 7f5cb559a640 1 -- 172.21.15.67:0/2293948333 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cb0031640 con 0x7f5cb80d0ef0 2024-03-24T00:47:17.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.278+0000 7f5cb559a640 1 -- 172.21.15.67:0/2293948333 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb002d040 con 0x7f5cb80d0ef0 2024-03-24T00:47:17.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.279+0000 7f5cbcd10640 1 -- 172.21.15.67:0/2293948333 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5cb80d0ef0 msgr2=0x7f5cb8162630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:17.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.279+0000 7f5cbcd10640 1 --2- 172.21.15.67:0/2293948333 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5cb80d0ef0 0x7f5cb8162630 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5cb0009870 tx=0x7f5cb002b8f0 comp rx=0 tx=0).stop 2024-03-24T00:47:17.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.279+0000 7f5cbcd10640 1 -- 172.21.15.67:0/2293948333 shutdown_connections 2024-03-24T00:47:17.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.279+0000 7f5cbcd10640 1 --2- 172.21.15.67:0/2293948333 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5cb80d0ef0 0x7f5cb8162630 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:17.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.279+0000 7f5cbcd10640 1 --2- 172.21.15.67:0/2293948333 >> v2:172.21.15.72:3300/0 conn(0x7f5cb815e2a0 0x7f5cb80ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:17.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.279+0000 7f5cbcd10640 1 -- 172.21.15.67:0/2293948333 >> 172.21.15.67:0/2293948333 conn(0x7f5cb806c810 msgr2=0x7f5cb806cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:17.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.279+0000 7f5cbcd10640 1 -- 172.21.15.67:0/2293948333 shutdown_connections 2024-03-24T00:47:17.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.279+0000 7f5cbcd10640 1 -- 172.21.15.67:0/2293948333 wait complete. 2024-03-24T00:47:17.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.280+0000 7f5cbcd10640 1 Processor -- start 2024-03-24T00:47:17.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.280+0000 7f5cbcd10640 1 -- start start 2024-03-24T00:47:17.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.281+0000 7f5cbcd10640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5cb80d0ef0 0x7f5cb81f7320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:17.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.281+0000 7f5cbcd10640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cb815dec0 0x7f5cb80b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:17.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.281+0000 7f5cbcd10640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5cb815e2a0 0x7f5cb80d2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:17.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.281+0000 7f5cbcd10640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb80d2730 con 0x7f5cb815dec0 2024-03-24T00:47:17.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.281+0000 7f5cbcd10640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb80d25b0 con 0x7f5cb80d0ef0 2024-03-24T00:47:17.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.281+0000 7f5cbcd10640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb80d28b0 con 0x7f5cb815e2a0 2024-03-24T00:47:17.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.281+0000 7f5cb5d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cb815dec0 0x7f5cb80b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:17.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.281+0000 7f5cb5d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cb815dec0 0x7f5cb80b65f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47146/0 (socket says 172.21.15.67:47146) 2024-03-24T00:47:17.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.281+0000 7f5cb5d9b640 1 -- 172.21.15.67:0/4173775844 learned_addr learned my addr 172.21.15.67:0/4173775844 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:17.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.281+0000 7f5cb659c640 1 --2- 172.21.15.67:0/4173775844 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5cb80d0ef0 0x7f5cb81f7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:17.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.281+0000 7f5cb6d9d640 1 --2- 172.21.15.67:0/4173775844 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5cb815e2a0 0x7f5cb80d2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:17.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.281+0000 7f5cb5d9b640 1 -- 172.21.15.67:0/4173775844 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5cb815e2a0 msgr2=0x7f5cb80d2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:17.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.281+0000 7f5cb5d9b640 1 --2- 172.21.15.67:0/4173775844 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5cb815e2a0 0x7f5cb80d2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:17.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.281+0000 7f5cb5d9b640 1 -- 172.21.15.67:0/4173775844 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5cb80d0ef0 msgr2=0x7f5cb81f7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:17.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.281+0000 7f5cb5d9b640 1 --2- 172.21.15.67:0/4173775844 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5cb80d0ef0 0x7f5cb81f7320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:17.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.281+0000 7f5cb5d9b640 1 -- 172.21.15.67:0/4173775844 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cb81f3e40 con 0x7f5cb815dec0 2024-03-24T00:47:17.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.282+0000 7f5cb5d9b640 1 --2- 172.21.15.67:0/4173775844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cb815dec0 0x7f5cb80b65f0 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f5ca000e900 tx=0x7f5ca000edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:17.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.282+0000 7f5caeffd640 1 -- 172.21.15.67:0/4173775844 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca0017070 con 0x7f5cb815dec0 2024-03-24T00:47:17.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.282+0000 7f5caeffd640 1 -- 172.21.15.67:0/4173775844 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ca000ca00 con 0x7f5cb815dec0 2024-03-24T00:47:17.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.282+0000 7f5caeffd640 1 -- 172.21.15.67:0/4173775844 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca000cdf0 con 0x7f5cb815dec0 2024-03-24T00:47:17.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.282+0000 7f5cbcd10640 1 -- 172.21.15.67:0/4173775844 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cb81f7900 con 0x7f5cb815dec0 2024-03-24T00:47:17.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.282+0000 7f5cbcd10640 1 -- 172.21.15.67:0/4173775844 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cb81f7ec0 con 0x7f5cb815dec0 2024-03-24T00:47:17.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.283+0000 7f5cbcd10640 1 -- 172.21.15.67:0/4173775844 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cb804f400 con 0x7f5cb815dec0 2024-03-24T00:47:17.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.283+0000 7f5caeffd640 1 -- 172.21.15.67:0/4173775844 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5ca001a070 con 0x7f5cb815dec0 2024-03-24T00:47:17.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.284+0000 7f5caeffd640 1 --2- 172.21.15.67:0/4173775844 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5c9806bbd0 0x7f5c9806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:17.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.284+0000 7f5caeffd640 1 -- 172.21.15.67:0/4173775844 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5ca008a710 con 0x7f5cb815dec0 2024-03-24T00:47:17.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.284+0000 7f5cb659c640 1 --2- 172.21.15.67:0/4173775844 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5c9806bbd0 0x7f5c9806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:17.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.284+0000 7f5cb659c640 1 --2- 172.21.15.67:0/4173775844 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5c9806bbd0 0x7f5c9806e090 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f5cb80b4960 tx=0x7f5ca8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:17.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.288+0000 7f5caeffd640 1 -- 172.21.15.67:0/4173775844 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5ca00589a0 con 0x7f5cb815dec0 2024-03-24T00:47:17.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.470+0000 7f5cbcd10640 1 -- 172.21.15.67:0/4173775844 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5cb8160170 con 0x7f5cb815dec0 2024-03-24T00:47:17.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.471+0000 7f5caeffd640 1 -- 172.21.15.67:0/4173775844 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) v1 ==== 74+0+16837 (secure 0 0 0) 0x7f5ca005ca40 con 0x7f5cb815dec0 2024-03-24T00:47:17.471 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:47:17.471 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":104,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:47:16.492174+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:47:15.111478+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-03-24T00:47:15.455832+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:47:17.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.473+0000 7f5cbcd10640 1 -- 172.21.15.67:0/4173775844 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5c9806bbd0 msgr2=0x7f5c9806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:17.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.473+0000 7f5cbcd10640 1 --2- 172.21.15.67:0/4173775844 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5c9806bbd0 0x7f5c9806e090 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f5cb80b4960 tx=0x7f5ca8009040 comp rx=0 tx=0).stop 2024-03-24T00:47:17.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.473+0000 7f5cbcd10640 1 -- 172.21.15.67:0/4173775844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cb815dec0 msgr2=0x7f5cb80b65f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:17.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.473+0000 7f5cbcd10640 1 --2- 172.21.15.67:0/4173775844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cb815dec0 0x7f5cb80b65f0 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f5ca000e900 tx=0x7f5ca000edc0 comp rx=0 tx=0).stop 2024-03-24T00:47:17.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.473+0000 7f5cbcd10640 1 -- 172.21.15.67:0/4173775844 shutdown_connections 2024-03-24T00:47:17.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.473+0000 7f5cbcd10640 1 --2- 172.21.15.67:0/4173775844 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5c9806bbd0 0x7f5c9806e090 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:17.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.473+0000 7f5cbcd10640 1 --2- 172.21.15.67:0/4173775844 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5cb815e2a0 0x7f5cb80d2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:17.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.473+0000 7f5cbcd10640 1 --2- 172.21.15.67:0/4173775844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cb815dec0 0x7f5cb80b65f0 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:17.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.473+0000 7f5cbcd10640 1 --2- 172.21.15.67:0/4173775844 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5cb80d0ef0 0x7f5cb81f7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:17.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.473+0000 7f5cbcd10640 1 -- 172.21.15.67:0/4173775844 >> 172.21.15.67:0/4173775844 conn(0x7f5cb806c810 msgr2=0x7f5cb815d300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:17.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.473+0000 7f5cbcd10640 1 -- 172.21.15.67:0/4173775844 shutdown_connections 2024-03-24T00:47:17.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.473+0000 7f5cbcd10640 1 -- 172.21.15.67:0/4173775844 wait complete. 2024-03-24T00:47:17.484 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:47:17.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.586+0000 7fbeff0dc640 1 Processor -- start 2024-03-24T00:47:17.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.586+0000 7fbeff0dc640 1 -- start start 2024-03-24T00:47:17.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.586+0000 7fbeff0dc640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbef815c140 0x7fbef80d01c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:17.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.587+0000 7fbeff0dc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbef809a410 0x7fbef8165a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:17.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.587+0000 7fbeff0dc640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbef80cdaa0 con 0x7fbef815bd60 2024-03-24T00:47:17.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.587+0000 7fbeff0dc640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbef80d08c0 con 0x7fbef809a410 2024-03-24T00:47:17.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.587+0000 7fbeff0dc640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbef80c58d0 con 0x7fbef815c140 2024-03-24T00:47:17.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.587+0000 7fbefd679640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbef809a410 0x7fbef8165a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:17.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.587+0000 7fbefce78640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbef815bd60 0x7fbef8099d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48260/0 (socket says 172.21.15.67:48260) 2024-03-24T00:47:17.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.587+0000 7fbefce78640 1 -- 172.21.15.67:0/306005377 learned_addr learned my addr 172.21.15.67:0/306005377 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:47:17.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.587+0000 7fbef7fff640 1 --2- 172.21.15.67:0/306005377 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbef815c140 0x7fbef80d01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:17.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.587+0000 7fbefd679640 1 -- 172.21.15.67:0/306005377 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbef815c140 msgr2=0x7fbef80d01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:17.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.587+0000 7fbefd679640 1 --2- 172.21.15.67:0/306005377 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbef815c140 0x7fbef80d01c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:17.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.587+0000 7fbefd679640 1 -- 172.21.15.67:0/306005377 >> v1:172.21.15.67:6789/0 conn(0x7fbef815bd60 legacy=0x7fbef8099d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:17.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.587+0000 7fbefd679640 1 -- 172.21.15.67:0/306005377 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef80c5bb0 con 0x7fbef809a410 2024-03-24T00:47:17.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.587+0000 7fbef7fff640 1 --2- 172.21.15.67:0/306005377 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbef815c140 0x7fbef80d01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:47:17.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.588+0000 7fbefd679640 1 --2- 172.21.15.67:0/306005377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbef809a410 0x7fbef8165a90 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7fbef0009d90 tx=0x7fbef002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9a32c8b88fa049d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:17.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.588+0000 7fbef77fe640 1 -- 172.21.15.67:0/306005377 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef0039070 con 0x7fbef809a410 2024-03-24T00:47:17.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.588+0000 7fbef77fe640 1 -- 172.21.15.67:0/306005377 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbef002cc50 con 0x7fbef809a410 2024-03-24T00:47:17.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.588+0000 7fbef77fe640 1 -- 172.21.15.67:0/306005377 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef0034440 con 0x7fbef809a410 2024-03-24T00:47:17.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.588+0000 7fbeff0dc640 1 -- 172.21.15.67:0/306005377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbef809a410 msgr2=0x7fbef8165a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:17.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.588+0000 7fbeff0dc640 1 --2- 172.21.15.67:0/306005377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbef809a410 0x7fbef8165a90 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7fbef0009d90 tx=0x7fbef002bea0 comp rx=0 tx=0).stop 2024-03-24T00:47:17.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.588+0000 7fbeff0dc640 1 -- 172.21.15.67:0/306005377 shutdown_connections 2024-03-24T00:47:17.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.588+0000 7fbeff0dc640 1 --2- 172.21.15.67:0/306005377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbef809a410 0x7fbef8165a90 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:17.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.588+0000 7fbeff0dc640 1 --2- 172.21.15.67:0/306005377 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbef815c140 0x7fbef80d01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:17.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.589+0000 7fbeff0dc640 1 -- 172.21.15.67:0/306005377 >> 172.21.15.67:0/306005377 conn(0x7fbef80b36d0 msgr2=0x7fbef80b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:17.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.589+0000 7fbeff0dc640 1 -- 172.21.15.67:0/306005377 shutdown_connections 2024-03-24T00:47:17.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.589+0000 7fbeff0dc640 1 -- 172.21.15.67:0/306005377 wait complete. 2024-03-24T00:47:17.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.590+0000 7fbeff0dc640 1 Processor -- start 2024-03-24T00:47:17.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.590+0000 7fbeff0dc640 1 -- start start 2024-03-24T00:47:17.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.591+0000 7fbeff0dc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbef809a410 0x7fbef81e4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:17.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.591+0000 7fbeff0dc640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbef815bd60 0x7fbef81e4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:17.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.591+0000 7fbeff0dc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbef815c140 0x7fbef80b6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:17.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.591+0000 7fbeff0dc640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef80c8540 con 0x7fbef809a410 2024-03-24T00:47:17.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.591+0000 7fbeff0dc640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef80c83c0 con 0x7fbef815c140 2024-03-24T00:47:17.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.591+0000 7fbeff0dc640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbef809ad40 con 0x7fbef815bd60 2024-03-24T00:47:17.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.591+0000 7fbefce78640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbef809a410 0x7fbef81e4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:17.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.591+0000 7fbefce78640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbef809a410 0x7fbef81e4eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47160/0 (socket says 172.21.15.67:47160) 2024-03-24T00:47:17.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.591+0000 7fbefce78640 1 -- 172.21.15.67:0/3658176129 learned_addr learned my addr 172.21.15.67:0/3658176129 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:17.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.591+0000 7fbef7fff640 1 --2- 172.21.15.67:0/3658176129 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbef815bd60 0x7fbef81e4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:17.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.591+0000 7fbefd679640 1 --2- 172.21.15.67:0/3658176129 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbef815c140 0x7fbef80b6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:17.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.591+0000 7fbefce78640 1 -- 172.21.15.67:0/3658176129 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbef815bd60 msgr2=0x7fbef81e4670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:17.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.591+0000 7fbefce78640 1 --2- 172.21.15.67:0/3658176129 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbef815bd60 0x7fbef81e4670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:17.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.591+0000 7fbefce78640 1 -- 172.21.15.67:0/3658176129 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbef815c140 msgr2=0x7fbef80b6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:17.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.591+0000 7fbefce78640 1 --2- 172.21.15.67:0/3658176129 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbef815c140 0x7fbef80b6060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:17.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.591+0000 7fbefce78640 1 -- 172.21.15.67:0/3658176129 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef815a1e0 con 0x7fbef809a410 2024-03-24T00:47:17.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.591+0000 7fbefd679640 1 --2- 172.21.15.67:0/3658176129 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbef815c140 0x7fbef80b6060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:47:17.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.591+0000 7fbefce78640 1 --2- 172.21.15.67:0/3658176129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbef809a410 0x7fbef81e4eb0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7fbeec00c430 tx=0x7fbeec00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:17.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.591+0000 7fbef57fa640 1 -- 172.21.15.67:0/3658176129 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeec017070 con 0x7fbef809a410 2024-03-24T00:47:17.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.591+0000 7fbef57fa640 1 -- 172.21.15.67:0/3658176129 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeec005ce0 con 0x7fbef809a410 2024-03-24T00:47:17.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.591+0000 7fbef57fa640 1 -- 172.21.15.67:0/3658176129 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeec0072f0 con 0x7fbef809a410 2024-03-24T00:47:17.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.591+0000 7fbeff0dc640 1 -- 172.21.15.67:0/3658176129 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbef81e5570 con 0x7fbef809a410 2024-03-24T00:47:17.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.592+0000 7fbeff0dc640 1 -- 172.21.15.67:0/3658176129 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbef815a780 con 0x7fbef809a410 2024-03-24T00:47:17.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.593+0000 7fbeff0dc640 1 -- 172.21.15.67:0/3658176129 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbef80c8c10 con 0x7fbef809a410 2024-03-24T00:47:17.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.593+0000 7fbef57fa640 1 -- 172.21.15.67:0/3658176129 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fbeec01a070 con 0x7fbef809a410 2024-03-24T00:47:17.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.593+0000 7fbef57fa640 1 --2- 172.21.15.67:0/3658176129 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbedc06bbd0 0x7fbedc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:17.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.593+0000 7fbef57fa640 1 -- 172.21.15.67:0/3658176129 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fbeec089ff0 con 0x7fbef809a410 2024-03-24T00:47:17.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.593+0000 7fbef7fff640 1 --2- 172.21.15.67:0/3658176129 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbedc06bbd0 0x7fbedc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:17.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.594+0000 7fbef7fff640 1 --2- 172.21.15.67:0/3658176129 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbedc06bbd0 0x7fbedc06e090 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fbef81e68b0 tx=0x7fbee4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:17.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.598+0000 7fbef57fa640 1 -- 172.21.15.67:0/3658176129 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fbeec058270 con 0x7fbef809a410 2024-03-24T00:47:17.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.781+0000 7fbeff0dc640 1 -- 172.21.15.67:0/3658176129 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbef804f400 con 0x7fbef809a410 2024-03-24T00:47:17.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.782+0000 7fbef57fa640 1 -- 172.21.15.67:0/3658176129 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) v1 ==== 74+0+16837 (secure 0 0 0) 0x7fbeec05c310 con 0x7fbef809a410 2024-03-24T00:47:17.782 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:47:17.782 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":104,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:47:16.492174+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:47:15.111478+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-03-24T00:47:15.455832+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:47:17.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.784+0000 7fbeff0dc640 1 -- 172.21.15.67:0/3658176129 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbedc06bbd0 msgr2=0x7fbedc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:17.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.784+0000 7fbeff0dc640 1 --2- 172.21.15.67:0/3658176129 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbedc06bbd0 0x7fbedc06e090 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fbef81e68b0 tx=0x7fbee4008040 comp rx=0 tx=0).stop 2024-03-24T00:47:17.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.784+0000 7fbeff0dc640 1 -- 172.21.15.67:0/3658176129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbef809a410 msgr2=0x7fbef81e4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:17.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.784+0000 7fbeff0dc640 1 --2- 172.21.15.67:0/3658176129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbef809a410 0x7fbef81e4eb0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7fbeec00c430 tx=0x7fbeec00c8f0 comp rx=0 tx=0).stop 2024-03-24T00:47:17.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.784+0000 7fbeff0dc640 1 -- 172.21.15.67:0/3658176129 shutdown_connections 2024-03-24T00:47:17.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.784+0000 7fbeff0dc640 1 --2- 172.21.15.67:0/3658176129 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbedc06bbd0 0x7fbedc06e090 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:17.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.784+0000 7fbeff0dc640 1 --2- 172.21.15.67:0/3658176129 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbef815c140 0x7fbef80b6060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:17.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.784+0000 7fbeff0dc640 1 --2- 172.21.15.67:0/3658176129 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbef815bd60 0x7fbef81e4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:17.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.784+0000 7fbeff0dc640 1 --2- 172.21.15.67:0/3658176129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbef809a410 0x7fbef81e4eb0 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:17.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.784+0000 7fbeff0dc640 1 -- 172.21.15.67:0/3658176129 >> 172.21.15.67:0/3658176129 conn(0x7fbef80b36d0 msgr2=0x7fbef8099d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:17.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.784+0000 7fbeff0dc640 1 -- 172.21.15.67:0/3658176129 shutdown_connections 2024-03-24T00:47:17.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.784+0000 7fbeff0dc640 1 -- 172.21.15.67:0/3658176129 wait complete. 2024-03-24T00:47:17.795 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-24T00:47:17.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.899+0000 7f7f0c90c640 1 Processor -- start 2024-03-24T00:47:17.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.899+0000 7f7f0c90c640 1 -- start start 2024-03-24T00:47:17.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.900+0000 7f7f0c90c640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7f0806eb30 0x7f7f080ad140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:17.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.900+0000 7f7f0c90c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7f080b51d0 0x7f7f080ad680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:17.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.900+0000 7f7f0c90c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f080b6150 con 0x7f7f0815de30 2024-03-24T00:47:17.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.900+0000 7f7f0c90c640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f0806c380 con 0x7f7f080b51d0 2024-03-24T00:47:17.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.900+0000 7f7f0c90c640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0806ba50 con 0x7f7f0806eb30 2024-03-24T00:47:17.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.900+0000 7f7f06d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7f080b51d0 0x7f7f080ad680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:17.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.900+0000 7f7f06d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7f080b51d0 0x7f7f080ad680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47164/0 (socket says 172.21.15.67:47164) 2024-03-24T00:47:17.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.900+0000 7f7f06d9d640 1 -- 172.21.15.67:0/3884441013 learned_addr learned my addr 172.21.15.67:0/3884441013 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:17.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.900+0000 7f7f05d9b640 1 --2- 172.21.15.67:0/3884441013 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7f0806eb30 0x7f7f080ad140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:17.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.900+0000 7f7f06d9d640 1 -- 172.21.15.67:0/3884441013 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7f0806eb30 msgr2=0x7f7f080ad140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:17.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.901+0000 7f7f06d9d640 1 --2- 172.21.15.67:0/3884441013 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7f0806eb30 0x7f7f080ad140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:17.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.901+0000 7f7f06d9d640 1 -- 172.21.15.67:0/3884441013 >> v1:172.21.15.72:6789/0 conn(0x7f7f0815de30 legacy=0x7f7f0806e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:17.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.901+0000 7f7f06d9d640 1 -- 172.21.15.67:0/3884441013 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f0806bd30 con 0x7f7f080b51d0 2024-03-24T00:47:17.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.901+0000 7f7f06d9d640 1 --2- 172.21.15.67:0/3884441013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f080b51d0 0x7f7f080ad680 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8009e50 tx=0x7f7ef802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16c3fa5f5dc5971c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:17.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.901+0000 7f7f0559a640 1 -- 172.21.15.67:0/3884441013 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef802cb00 con 0x7f7f080b51d0 2024-03-24T00:47:17.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.901+0000 7f7f0559a640 1 -- 172.21.15.67:0/3884441013 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ef802cca0 con 0x7f7f080b51d0 2024-03-24T00:47:17.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.901+0000 7f7f0559a640 1 -- 172.21.15.67:0/3884441013 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef8034610 con 0x7f7f080b51d0 2024-03-24T00:47:17.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.902+0000 7f7f0c90c640 1 -- 172.21.15.67:0/3884441013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f080b51d0 msgr2=0x7f7f080ad680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:17.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.902+0000 7f7f0c90c640 1 --2- 172.21.15.67:0/3884441013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f080b51d0 0x7f7f080ad680 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8009e50 tx=0x7f7ef802bf60 comp rx=0 tx=0).stop 2024-03-24T00:47:17.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.902+0000 7f7f0c90c640 1 -- 172.21.15.67:0/3884441013 shutdown_connections 2024-03-24T00:47:17.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.902+0000 7f7f0c90c640 1 --2- 172.21.15.67:0/3884441013 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7f0806eb30 0x7f7f080ad140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:17.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.902+0000 7f7f0c90c640 1 --2- 172.21.15.67:0/3884441013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f080b51d0 0x7f7f080ad680 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:17.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.902+0000 7f7f0c90c640 1 -- 172.21.15.67:0/3884441013 >> 172.21.15.67:0/3884441013 conn(0x7f7f08099a30 msgr2=0x7f7f080b5680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:17.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.902+0000 7f7f0c90c640 1 -- 172.21.15.67:0/3884441013 shutdown_connections 2024-03-24T00:47:17.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.902+0000 7f7f0c90c640 1 -- 172.21.15.67:0/3884441013 wait complete. 2024-03-24T00:47:17.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.903+0000 7f7f0c90c640 1 Processor -- start 2024-03-24T00:47:17.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.903+0000 7f7f0c90c640 1 -- start start 2024-03-24T00:47:17.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.904+0000 7f7f0c90c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7f0806eb30 0x7f7f081f4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:17.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.904+0000 7f7f0c90c640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7f080b51d0 0x7f7f08154ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:17.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.904+0000 7f7f0c90c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f0815de30 0x7f7f081f7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:17.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.904+0000 7f7f0c90c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0815e210 con 0x7f7f0815de30 2024-03-24T00:47:17.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.904+0000 7f7f0c90c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0806c810 con 0x7f7f0806eb30 2024-03-24T00:47:17.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.904+0000 7f7f0c90c640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f081f3fd0 con 0x7f7f080b51d0 2024-03-24T00:47:17.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.904+0000 7f7f06d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f0815de30 0x7f7f081f7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:17.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.904+0000 7f7f06d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f0815de30 0x7f7f081f7040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47174/0 (socket says 172.21.15.67:47174) 2024-03-24T00:47:17.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.904+0000 7f7f06d9d640 1 -- 172.21.15.67:0/3954347231 learned_addr learned my addr 172.21.15.67:0/3954347231 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:17.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.904+0000 7f7f05d9b640 1 --2- 172.21.15.67:0/3954347231 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7f080b51d0 0x7f7f08154ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:17.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.904+0000 7f7f0659c640 1 --2- 172.21.15.67:0/3954347231 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7f0806eb30 0x7f7f081f4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:17.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.905+0000 7f7f06d9d640 1 -- 172.21.15.67:0/3954347231 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7f080b51d0 msgr2=0x7f7f08154ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:17.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.905+0000 7f7f06d9d640 1 --2- 172.21.15.67:0/3954347231 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7f080b51d0 0x7f7f08154ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:17.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.905+0000 7f7f06d9d640 1 -- 172.21.15.67:0/3954347231 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7f0806eb30 msgr2=0x7f7f081f4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:17.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.905+0000 7f7f06d9d640 1 --2- 172.21.15.67:0/3954347231 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7f0806eb30 0x7f7f081f4460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:17.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.905+0000 7f7f06d9d640 1 -- 172.21.15.67:0/3954347231 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f081f49a0 con 0x7f7f0815de30 2024-03-24T00:47:17.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.905+0000 7f7f06d9d640 1 --2- 172.21.15.67:0/3954347231 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f0815de30 0x7f7f081f7040 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8009e50 tx=0x7f7ef8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:17.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.905+0000 7f7efeffd640 1 -- 172.21.15.67:0/3954347231 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef8043020 con 0x7f7f0815de30 2024-03-24T00:47:17.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.905+0000 7f7efeffd640 1 -- 172.21.15.67:0/3954347231 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ef802ce50 con 0x7f7f0815de30 2024-03-24T00:47:17.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.905+0000 7f7f0c90c640 1 -- 172.21.15.67:0/3954347231 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f081f79b0 con 0x7f7f0815de30 2024-03-24T00:47:17.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.905+0000 7f7efeffd640 1 -- 172.21.15.67:0/3954347231 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef8009510 con 0x7f7f0815de30 2024-03-24T00:47:17.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.905+0000 7f7f0c90c640 1 -- 172.21.15.67:0/3954347231 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f081f3c60 con 0x7f7f0815de30 2024-03-24T00:47:17.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.906+0000 7f7f0c90c640 1 -- 172.21.15.67:0/3954347231 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f081f7ed0 con 0x7f7f0815de30 2024-03-24T00:47:17.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.907+0000 7f7efeffd640 1 -- 172.21.15.67:0/3954347231 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7ef8046020 con 0x7f7f0815de30 2024-03-24T00:47:17.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.907+0000 7f7efeffd640 1 --2- 172.21.15.67:0/3954347231 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7ee806bbd0 0x7f7ee806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:17.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.907+0000 7f7efeffd640 1 -- 172.21.15.67:0/3954347231 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7ef80ad140 con 0x7f7f0815de30 2024-03-24T00:47:17.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.907+0000 7f7f0659c640 1 --2- 172.21.15.67:0/3954347231 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7ee806bbd0 0x7f7ee806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:17.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.908+0000 7f7f0659c640 1 --2- 172.21.15.67:0/3954347231 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7ee806bbd0 0x7f7ee806e090 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f7f08160b90 tx=0x7f7f00009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:17.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:17.911+0000 7f7efeffd640 1 -- 172.21.15.67:0/3954347231 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7f081f7ed0 con 0x7f7f0815de30 2024-03-24T00:47:18.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.133+0000 7f7f0c90c640 1 -- 172.21.15.67:0/3954347231 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f7f081f3a50 con 0x7f7f0815de30 2024-03-24T00:47:18.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.147+0000 7f7efeffd640 1 -- 172.21.15.67:0/3954347231 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v71) v1 ==== 118+0+0 (secure 0 0 0) 0x7f7ef804d030 con 0x7f7f0815de30 2024-03-24T00:47:18.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.149+0000 7f7f0c90c640 1 -- 172.21.15.67:0/3954347231 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7ee806bbd0 msgr2=0x7f7ee806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:18.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.149+0000 7f7f0c90c640 1 --2- 172.21.15.67:0/3954347231 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7ee806bbd0 0x7f7ee806e090 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f7f08160b90 tx=0x7f7f00009040 comp rx=0 tx=0).stop 2024-03-24T00:47:18.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.149+0000 7f7f0c90c640 1 -- 172.21.15.67:0/3954347231 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f0815de30 msgr2=0x7f7f081f7040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:18.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.149+0000 7f7f0c90c640 1 --2- 172.21.15.67:0/3954347231 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f0815de30 0x7f7f081f7040 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8009e50 tx=0x7f7ef8004a40 comp rx=0 tx=0).stop 2024-03-24T00:47:18.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.149+0000 7f7f0c90c640 1 -- 172.21.15.67:0/3954347231 shutdown_connections 2024-03-24T00:47:18.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.149+0000 7f7f0c90c640 1 --2- 172.21.15.67:0/3954347231 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7ee806bbd0 0x7f7ee806e090 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:18.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.149+0000 7f7f0c90c640 1 --2- 172.21.15.67:0/3954347231 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f0815de30 0x7f7f081f7040 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:18.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.149+0000 7f7f0c90c640 1 --2- 172.21.15.67:0/3954347231 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7f080b51d0 0x7f7f08154ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:18.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.149+0000 7f7f0c90c640 1 --2- 172.21.15.67:0/3954347231 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7f0806eb30 0x7f7f081f4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:18.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.149+0000 7f7f0c90c640 1 -- 172.21.15.67:0/3954347231 >> 172.21.15.67:0/3954347231 conn(0x7f7f08099a30 msgr2=0x7f7f0815ee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:18.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.149+0000 7f7f0c90c640 1 -- 172.21.15.67:0/3954347231 shutdown_connections 2024-03-24T00:47:18.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.149+0000 7f7f0c90c640 1 -- 172.21.15.67:0/3954347231 wait complete. 2024-03-24T00:47:18.159 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-24T00:47:18.160 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-24T00:47:18.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.264+0000 7f23f5ed1640 1 Processor -- start 2024-03-24T00:47:18.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.264+0000 7f23f5ed1640 1 -- start start 2024-03-24T00:47:18.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.265+0000 7f23f5ed1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f23f015df00 0x7f23f015ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:18.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.265+0000 7f23f5ed1640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f23f015e580 0x7f23f00ce3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:18.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.265+0000 7f23f5ed1640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23f015e2e0 con 0x7f23f00b2920 2024-03-24T00:47:18.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.265+0000 7f23f5ed1640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23f00b1880 con 0x7f23f015e580 2024-03-24T00:47:18.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.265+0000 7f23f5ed1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23f00b1b60 con 0x7f23f015df00 2024-03-24T00:47:18.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.265+0000 7f23effff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f23f00b2920 0x7f23f006c360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48276/0 (socket says 172.21.15.67:48276) 2024-03-24T00:47:18.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.265+0000 7f23effff640 1 -- 172.21.15.67:0/882251811 learned_addr learned my addr 172.21.15.67:0/882251811 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:47:18.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.266+0000 7f23ef7fe640 1 --2- 172.21.15.67:0/882251811 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f23f015df00 0x7f23f015ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:18.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.266+0000 7f23eeffd640 1 --2- 172.21.15.67:0/882251811 >> v2:172.21.15.72:3300/0 conn(0x7f23f015e580 0x7f23f00ce3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:18.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.266+0000 7f23ee7fc640 1 -- 172.21.15.67:0/882251811 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4030178681 0 0) 0x7f23f015e2e0 con 0x7f23f00b2920 2024-03-24T00:47:18.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.266+0000 7f23ee7fc640 1 -- 172.21.15.67:0/882251811 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23d8003520 con 0x7f23f00b2920 2024-03-24T00:47:18.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.266+0000 7f23ee7fc640 1 -- 172.21.15.67:0/882251811 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4014488739 0 0) 0x7f23d8003520 con 0x7f23f00b2920 2024-03-24T00:47:18.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.266+0000 7f23ee7fc640 1 -- 172.21.15.67:0/882251811 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f23f015df00 msgr2=0x7f23f015ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:18.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.266+0000 7f23ee7fc640 1 --2- 172.21.15.67:0/882251811 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f23f015df00 0x7f23f015ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:18.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.266+0000 7f23ee7fc640 1 -- 172.21.15.67:0/882251811 >> v2:172.21.15.72:3300/0 conn(0x7f23f015e580 msgr2=0x7f23f00ce3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:18.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.266+0000 7f23ee7fc640 1 --2- 172.21.15.67:0/882251811 >> v2:172.21.15.72:3300/0 conn(0x7f23f015e580 0x7f23f00ce3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:18.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.266+0000 7f23ee7fc640 1 -- 172.21.15.67:0/882251811 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23f00b1e40 con 0x7f23f00b2920 2024-03-24T00:47:18.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.266+0000 7f23ee7fc640 1 -- 172.21.15.67:0/882251811 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f23e80019f0 con 0x7f23f00b2920 2024-03-24T00:47:18.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.266+0000 7f23ee7fc640 1 -- 172.21.15.67:0/882251811 >> v1:172.21.15.67:6789/0 conn(0x7f23f00b2920 legacy=0x7f23f006c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:18.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.266+0000 7f23ee7fc640 1 --2- 172.21.15.67:0/882251811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23d80039b0 0x7f23d8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:18.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.266+0000 7f23ee7fc640 1 --2- 172.21.15.67:0/882251811 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f23f015df00 0x7f23d8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:18.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.266+0000 7f23ee7fc640 1 --2- 172.21.15.67:0/882251811 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23d8004d40 0x7f23d80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:18.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.266+0000 7f23ee7fc640 1 -- 172.21.15.67:0/882251811 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f00b1880 con 0x7f23d80039b0 2024-03-24T00:47:18.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.266+0000 7f23ee7fc640 1 -- 172.21.15.67:0/882251811 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f00b1b60 con 0x7f23d8004d40 2024-03-24T00:47:18.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.266+0000 7f23ee7fc640 1 -- 172.21.15.67:0/882251811 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23d800ba40 con 0x7f23f015df00 2024-03-24T00:47:18.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.266+0000 7f23ef7fe640 1 --2- 172.21.15.67:0/882251811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23d80039b0 0x7f23d8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:18.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.266+0000 7f23effff640 1 --2- 172.21.15.67:0/882251811 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23d8004d40 0x7f23d80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:18.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.266+0000 7f23eeffd640 1 --2- 172.21.15.67:0/882251811 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f23f015df00 0x7f23d8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:18.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.266+0000 7f23ef7fe640 1 -- 172.21.15.67:0/882251811 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f23f015df00 msgr2=0x7f23d8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:18.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.266+0000 7f23ef7fe640 1 --2- 172.21.15.67:0/882251811 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f23f015df00 0x7f23d8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:18.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.266+0000 7f23ef7fe640 1 -- 172.21.15.67:0/882251811 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23d8004d40 msgr2=0x7f23d80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:18.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.266+0000 7f23ef7fe640 1 --2- 172.21.15.67:0/882251811 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23d8004d40 0x7f23d80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:18.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.266+0000 7f23ef7fe640 1 -- 172.21.15.67:0/882251811 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23d8003520 con 0x7f23d80039b0 2024-03-24T00:47:18.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.268+0000 7f23ef7fe640 1 --2- 172.21.15.67:0/882251811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23d80039b0 0x7f23d8003dd0 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f23e0009ac0 tx=0x7f23d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63461f16942f2e14 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:18.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.268+0000 7f23ee7fc640 1 -- 172.21.15.67:0/882251811 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e0039070 con 0x7f23d80039b0 2024-03-24T00:47:18.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.268+0000 7f23ee7fc640 1 -- 172.21.15.67:0/882251811 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23e002c6b0 con 0x7f23d80039b0 2024-03-24T00:47:18.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.268+0000 7f23ee7fc640 1 -- 172.21.15.67:0/882251811 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e0009690 con 0x7f23d80039b0 2024-03-24T00:47:18.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.268+0000 7f23f5ed1640 1 -- 172.21.15.67:0/882251811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23d80039b0 msgr2=0x7f23d8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:18.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.268+0000 7f23f5ed1640 1 --2- 172.21.15.67:0/882251811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23d80039b0 0x7f23d8003dd0 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f23e0009ac0 tx=0x7f23d8005830 comp rx=0 tx=0).stop 2024-03-24T00:47:18.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.268+0000 7f23effff640 1 -- 172.21.15.67:0/882251811 reap_dead start 2024-03-24T00:47:18.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.268+0000 7f23f5ed1640 1 -- 172.21.15.67:0/882251811 shutdown_connections 2024-03-24T00:47:18.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.268+0000 7f23f5ed1640 1 -- 172.21.15.67:0/882251811 >> 172.21.15.67:0/882251811 conn(0x7f23f008b180 msgr2=0x7f23f008b590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:18.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.268+0000 7f23f5ed1640 1 -- 172.21.15.67:0/882251811 shutdown_connections 2024-03-24T00:47:18.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.268+0000 7f23f5ed1640 1 -- 172.21.15.67:0/882251811 wait complete. 2024-03-24T00:47:18.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.270+0000 7f23f5ed1640 1 Processor -- start 2024-03-24T00:47:18.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.270+0000 7f23f5ed1640 1 -- start start 2024-03-24T00:47:18.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.272+0000 7f23f5ed1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23d80039b0 0x7f23f014a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:18.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.272+0000 7f23f5ed1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f23f0160760 0x7f23f009c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:18.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.272+0000 7f23f5ed1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23f009ced0 0x7f23f00ae690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:18.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.272+0000 7f23f5ed1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f00ce900 con 0x7f23d80039b0 2024-03-24T00:47:18.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.272+0000 7f23f5ed1640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f00b2400 con 0x7f23f009ced0 2024-03-24T00:47:18.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.272+0000 7f23f5ed1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23f00cea80 con 0x7f23f0160760 2024-03-24T00:47:18.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.272+0000 7f23ef7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23d80039b0 0x7f23f014a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:18.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.272+0000 7f23ef7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23d80039b0 0x7f23f014a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47190/0 (socket says 172.21.15.67:47190) 2024-03-24T00:47:18.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.272+0000 7f23ef7fe640 1 -- 172.21.15.67:0/803533905 learned_addr learned my addr 172.21.15.67:0/803533905 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:18.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.272+0000 7f23effff640 1 --2- 172.21.15.67:0/803533905 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23f009ced0 0x7f23f00ae690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:18.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.272+0000 7f23ef7fe640 1 -- 172.21.15.67:0/803533905 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f23f0160760 msgr2=0x7f23f009c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:18.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.272+0000 7f23eeffd640 1 --2- 172.21.15.67:0/803533905 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f23f0160760 0x7f23f009c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:18.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.272+0000 7f23ef7fe640 1 --2- 172.21.15.67:0/803533905 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f23f0160760 0x7f23f009c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:18.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.272+0000 7f23ef7fe640 1 -- 172.21.15.67:0/803533905 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23f009ced0 msgr2=0x7f23f00ae690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:18.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.272+0000 7f23ef7fe640 1 --2- 172.21.15.67:0/803533905 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23f009ced0 0x7f23f00ae690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:18.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.272+0000 7f23ef7fe640 1 -- 172.21.15.67:0/803533905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23f014ac80 con 0x7f23d80039b0 2024-03-24T00:47:18.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.272+0000 7f23effff640 1 --2- 172.21.15.67:0/803533905 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23f009ced0 0x7f23f00ae690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:47:18.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.272+0000 7f23ef7fe640 1 --2- 172.21.15.67:0/803533905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23d80039b0 0x7f23f014a520 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f23e0000c00 tx=0x7f23e0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:18.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.272+0000 7f23d7fff640 1 -- 172.21.15.67:0/803533905 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e0042020 con 0x7f23d80039b0 2024-03-24T00:47:18.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.272+0000 7f23d7fff640 1 -- 172.21.15.67:0/803533905 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23e003a040 con 0x7f23d80039b0 2024-03-24T00:47:18.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.272+0000 7f23f5ed1640 1 -- 172.21.15.67:0/803533905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23f014af60 con 0x7f23d80039b0 2024-03-24T00:47:18.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.272+0000 7f23d7fff640 1 -- 172.21.15.67:0/803533905 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e0039070 con 0x7f23d80039b0 2024-03-24T00:47:18.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.272+0000 7f23f5ed1640 1 -- 172.21.15.67:0/803533905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23f014b520 con 0x7f23d80039b0 2024-03-24T00:47:18.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.274+0000 7f23f5ed1640 1 -- 172.21.15.67:0/803533905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23f004f400 con 0x7f23d80039b0 2024-03-24T00:47:18.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.274+0000 7f23d7fff640 1 -- 172.21.15.67:0/803533905 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f23e0045020 con 0x7f23d80039b0 2024-03-24T00:47:18.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.274+0000 7f23d7fff640 1 --2- 172.21.15.67:0/803533905 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f23dc06bbd0 0x7f23dc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:18.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.274+0000 7f23d7fff640 1 -- 172.21.15.67:0/803533905 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f23e00ac120 con 0x7f23d80039b0 2024-03-24T00:47:18.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.275+0000 7f23eeffd640 1 --2- 172.21.15.67:0/803533905 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f23dc06bbd0 0x7f23dc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:18.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.275+0000 7f23eeffd640 1 --2- 172.21.15.67:0/803533905 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f23dc06bbd0 0x7f23dc06e090 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f23f006cb10 tx=0x7f23e4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:18.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.283+0000 7f23d7fff640 1 -- 172.21.15.67:0/803533905 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f23e004c070 con 0x7f23d80039b0 2024-03-24T00:47:18.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:18.506+0000 7f23f5ed1640 1 -- 172.21.15.67:0/803533905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f23f00b7ac0 con 0x7f23d80039b0 2024-03-24T00:47:19.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.152+0000 7f23d7fff640 1 -- 172.21.15.67:0/803533905 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v72) v1 ==== 122+0+0 (secure 0 0 0) 0x7f23e007aa80 con 0x7f23d80039b0 2024-03-24T00:47:19.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.154+0000 7f23f5ed1640 1 -- 172.21.15.67:0/803533905 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f23dc06bbd0 msgr2=0x7f23dc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:19.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.154+0000 7f23f5ed1640 1 --2- 172.21.15.67:0/803533905 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f23dc06bbd0 0x7f23dc06e090 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f23f006cb10 tx=0x7f23e4008040 comp rx=0 tx=0).stop 2024-03-24T00:47:19.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.154+0000 7f23f5ed1640 1 -- 172.21.15.67:0/803533905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23d80039b0 msgr2=0x7f23f014a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:19.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.154+0000 7f23f5ed1640 1 --2- 172.21.15.67:0/803533905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23d80039b0 0x7f23f014a520 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f23e0000c00 tx=0x7f23e0004730 comp rx=0 tx=0).stop 2024-03-24T00:47:19.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.154+0000 7f23f5ed1640 1 -- 172.21.15.67:0/803533905 shutdown_connections 2024-03-24T00:47:19.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.154+0000 7f23f5ed1640 1 --2- 172.21.15.67:0/803533905 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f23dc06bbd0 0x7f23dc06e090 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:19.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.154+0000 7f23f5ed1640 1 --2- 172.21.15.67:0/803533905 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23f009ced0 0x7f23f00ae690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:19.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.154+0000 7f23f5ed1640 1 --2- 172.21.15.67:0/803533905 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f23f0160760 0x7f23f009c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:19.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.154+0000 7f23f5ed1640 1 --2- 172.21.15.67:0/803533905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23d80039b0 0x7f23f014a520 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:19.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.154+0000 7f23f5ed1640 1 -- 172.21.15.67:0/803533905 >> 172.21.15.67:0/803533905 conn(0x7f23f008b180 msgr2=0x7f23f015c450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:19.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.154+0000 7f23f5ed1640 1 -- 172.21.15.67:0/803533905 shutdown_connections 2024-03-24T00:47:19.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.154+0000 7f23f5ed1640 1 -- 172.21.15.67:0/803533905 wait complete. 2024-03-24T00:47:19.167 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-24T00:47:19.167 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:47:19.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.268+0000 7f7b6944b640 1 Processor -- start 2024-03-24T00:47:19.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.269+0000 7f7b6944b640 1 -- start start 2024-03-24T00:47:19.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.269+0000 7f7b6944b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7b6415e1b0 0x7f7b640cfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:19.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.269+0000 7f7b6944b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7b640b43a0 0x7f7b64099d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:19.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.270+0000 7f7b6944b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7b6409a240 0x7f7b6406c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:19.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.270+0000 7f7b6944b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b6415e590 con 0x7f7b6415e1b0 2024-03-24T00:47:19.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.270+0000 7f7b6944b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b6409a690 con 0x7f7b640b43a0 2024-03-24T00:47:19.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.270+0000 7f7b6944b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b640d1820 con 0x7f7b6409a240 2024-03-24T00:47:19.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.270+0000 7f7b62ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7b6415e1b0 0x7f7b640cfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:19.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.270+0000 7f7b62ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7b6415e1b0 0x7f7b640cfe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47192/0 (socket says 172.21.15.67:47192) 2024-03-24T00:47:19.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.270+0000 7f7b62ffd640 1 -- 172.21.15.67:0/2517568190 learned_addr learned my addr 172.21.15.67:0/2517568190 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:19.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.270+0000 7f7b627fc640 1 --2- 172.21.15.67:0/2517568190 >> v2:172.21.15.72:3300/0 conn(0x7f7b640b43a0 0x7f7b64099d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:19.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.270+0000 7f7b637fe640 1 --2- 172.21.15.67:0/2517568190 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7b6409a240 0x7f7b6406c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:19.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.270+0000 7f7b62ffd640 1 -- 172.21.15.67:0/2517568190 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7b6409a240 msgr2=0x7f7b6406c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:19.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.270+0000 7f7b62ffd640 1 --2- 172.21.15.67:0/2517568190 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7b6409a240 0x7f7b6406c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:19.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.270+0000 7f7b62ffd640 1 -- 172.21.15.67:0/2517568190 >> v2:172.21.15.72:3300/0 conn(0x7f7b640b43a0 msgr2=0x7f7b64099d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:19.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.270+0000 7f7b62ffd640 1 --2- 172.21.15.67:0/2517568190 >> v2:172.21.15.72:3300/0 conn(0x7f7b640b43a0 0x7f7b64099d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:19.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.270+0000 7f7b62ffd640 1 -- 172.21.15.67:0/2517568190 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b640d0fc0 con 0x7f7b6415e1b0 2024-03-24T00:47:19.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.271+0000 7f7b62ffd640 1 --2- 172.21.15.67:0/2517568190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b6415e1b0 0x7f7b640cfe20 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f7b640c8c30 tx=0x7f7b5402b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38032b094ff10a00 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:19.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.271+0000 7f7b61ffb640 1 -- 172.21.15.67:0/2517568190 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b54038070 con 0x7f7b6415e1b0 2024-03-24T00:47:19.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.271+0000 7f7b61ffb640 1 -- 172.21.15.67:0/2517568190 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b54031740 con 0x7f7b6415e1b0 2024-03-24T00:47:19.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.271+0000 7f7b61ffb640 1 -- 172.21.15.67:0/2517568190 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b5402f050 con 0x7f7b6415e1b0 2024-03-24T00:47:19.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.272+0000 7f7b6944b640 1 -- 172.21.15.67:0/2517568190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b6415e1b0 msgr2=0x7f7b640cfe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:19.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.272+0000 7f7b6944b640 1 --2- 172.21.15.67:0/2517568190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b6415e1b0 0x7f7b640cfe20 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f7b640c8c30 tx=0x7f7b5402b7e0 comp rx=0 tx=0).stop 2024-03-24T00:47:19.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.272+0000 7f7b6944b640 1 -- 172.21.15.67:0/2517568190 shutdown_connections 2024-03-24T00:47:19.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.272+0000 7f7b6944b640 1 --2- 172.21.15.67:0/2517568190 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7b6409a240 0x7f7b6406c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:19.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.272+0000 7f7b6944b640 1 --2- 172.21.15.67:0/2517568190 >> v2:172.21.15.72:3300/0 conn(0x7f7b640b43a0 0x7f7b64099d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:19.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.272+0000 7f7b6944b640 1 --2- 172.21.15.67:0/2517568190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b6415e1b0 0x7f7b640cfe20 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:19.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.272+0000 7f7b6944b640 1 -- 172.21.15.67:0/2517568190 >> 172.21.15.67:0/2517568190 conn(0x7f7b640b36d0 msgr2=0x7f7b640b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:19.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.272+0000 7f7b6944b640 1 -- 172.21.15.67:0/2517568190 shutdown_connections 2024-03-24T00:47:19.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.272+0000 7f7b6944b640 1 -- 172.21.15.67:0/2517568190 wait complete. 2024-03-24T00:47:19.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.274+0000 7f7b6944b640 1 Processor -- start 2024-03-24T00:47:19.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.274+0000 7f7b6944b640 1 -- start start 2024-03-24T00:47:19.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.275+0000 7f7b6944b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b6409a240 0x7f7b640c9d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:19.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.275+0000 7f7b6944b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7b640b43a0 0x7f7b6414ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:19.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.275+0000 7f7b6944b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b6415e1b0 0x7f7b640c5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:19.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.275+0000 7f7b6944b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b640cc360 con 0x7f7b6409a240 2024-03-24T00:47:19.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.275+0000 7f7b6944b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b640d1540 con 0x7f7b6415e1b0 2024-03-24T00:47:19.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.275+0000 7f7b6944b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6409a9a0 con 0x7f7b640b43a0 2024-03-24T00:47:19.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.275+0000 7f7b62ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b6409a240 0x7f7b640c9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:19.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.275+0000 7f7b62ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b6409a240 0x7f7b640c9d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47194/0 (socket says 172.21.15.67:47194) 2024-03-24T00:47:19.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.275+0000 7f7b62ffd640 1 -- 172.21.15.67:0/2332383423 learned_addr learned my addr 172.21.15.67:0/2332383423 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:19.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.275+0000 7f7b637fe640 1 --2- 172.21.15.67:0/2332383423 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b6415e1b0 0x7f7b640c5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:19.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.275+0000 7f7b627fc640 1 --2- 172.21.15.67:0/2332383423 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7b640b43a0 0x7f7b6414ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:19.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.275+0000 7f7b62ffd640 1 -- 172.21.15.67:0/2332383423 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7b640b43a0 msgr2=0x7f7b6414ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:19.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.275+0000 7f7b62ffd640 1 --2- 172.21.15.67:0/2332383423 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7b640b43a0 0x7f7b6414ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:19.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.275+0000 7f7b62ffd640 1 -- 172.21.15.67:0/2332383423 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b6415e1b0 msgr2=0x7f7b640c5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:19.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.275+0000 7f7b62ffd640 1 --2- 172.21.15.67:0/2332383423 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b6415e1b0 0x7f7b640c5710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:19.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.275+0000 7f7b62ffd640 1 -- 172.21.15.67:0/2332383423 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b640ca240 con 0x7f7b6409a240 2024-03-24T00:47:19.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.275+0000 7f7b627fc640 1 --2- 172.21.15.67:0/2332383423 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7b640b43a0 0x7f7b6414ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:47:19.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.275+0000 7f7b637fe640 1 --2- 172.21.15.67:0/2332383423 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b6415e1b0 0x7f7b640c5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:47:19.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.276+0000 7f7b62ffd640 1 --2- 172.21.15.67:0/2332383423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b6409a240 0x7f7b640c9d00 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f7b54000c00 tx=0x7f7b540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:19.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.276+0000 7f7b5b7fe640 1 -- 172.21.15.67:0/2332383423 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b54042070 con 0x7f7b6409a240 2024-03-24T00:47:19.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.276+0000 7f7b5b7fe640 1 -- 172.21.15.67:0/2332383423 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b5403d870 con 0x7f7b6409a240 2024-03-24T00:47:19.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.276+0000 7f7b6944b640 1 -- 172.21.15.67:0/2332383423 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b640ca4e0 con 0x7f7b6409a240 2024-03-24T00:47:19.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.276+0000 7f7b5b7fe640 1 -- 172.21.15.67:0/2332383423 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b54038040 con 0x7f7b6409a240 2024-03-24T00:47:19.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.276+0000 7f7b6944b640 1 -- 172.21.15.67:0/2332383423 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b640caaa0 con 0x7f7b6409a240 2024-03-24T00:47:19.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.277+0000 7f7b6944b640 1 -- 172.21.15.67:0/2332383423 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b6404f400 con 0x7f7b6409a240 2024-03-24T00:47:19.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.277+0000 7f7b5b7fe640 1 -- 172.21.15.67:0/2332383423 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7b54045070 con 0x7f7b6409a240 2024-03-24T00:47:19.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.277+0000 7f7b5b7fe640 1 --2- 172.21.15.67:0/2332383423 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7b4406bbd0 0x7f7b4406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:19.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.278+0000 7f7b5b7fe640 1 -- 172.21.15.67:0/2332383423 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7b540ac650 con 0x7f7b6409a240 2024-03-24T00:47:19.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.278+0000 7f7b627fc640 1 --2- 172.21.15.67:0/2332383423 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7b4406bbd0 0x7f7b4406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:19.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.278+0000 7f7b627fc640 1 --2- 172.21.15.67:0/2332383423 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7b4406bbd0 0x7f7b4406e090 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f7b6415a5c0 tx=0x7f7b4c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:19.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.286+0000 7f7b5b7fe640 1 -- 172.21.15.67:0/2332383423 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7b5404c030 con 0x7f7b6409a240 2024-03-24T00:47:19.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.640+0000 7f7b6944b640 1 -- 172.21.15.67:0/2332383423 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b640cd980 con 0x7f7b6409a240 2024-03-24T00:47:19.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.641+0000 7f7b5b7fe640 1 -- 172.21.15.67:0/2332383423 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+4976 (secure 0 0 0) 0x7f7b54045e40 con 0x7f7b6409a240 2024-03-24T00:47:19.641 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:47:19.641 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":72,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6902,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3672294289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3672294289},{"type":"v1","addr":"172.21.15.67:6835","nonce":3672294289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":6908,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/1286271878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":1286271878},{"type":"v1","addr":"172.21.15.72:6836","nonce":1286271878}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":6911,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/140622129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":140622129},{"type":"v1","addr":"172.21.15.72:6837","nonce":140622129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:47:16.492485+0000","modified":"2024-03-24T00:47:18.507677+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8541},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8541":{"gid":8541,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6833/1826974843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1826974843},{"type":"v1","addr":"172.21.15.67:6833","nonce":1826974843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8541,"qdb_cluster":[8541]},"id":7}]} 2024-03-24T00:47:19.641 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 72 2024-03-24T00:47:19.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.643+0000 7f7b6944b640 1 -- 172.21.15.67:0/2332383423 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7b4406bbd0 msgr2=0x7f7b4406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:19.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.643+0000 7f7b6944b640 1 --2- 172.21.15.67:0/2332383423 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7b4406bbd0 0x7f7b4406e090 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f7b6415a5c0 tx=0x7f7b4c008040 comp rx=0 tx=0).stop 2024-03-24T00:47:19.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.643+0000 7f7b6944b640 1 -- 172.21.15.67:0/2332383423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b6409a240 msgr2=0x7f7b640c9d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:19.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.643+0000 7f7b6944b640 1 --2- 172.21.15.67:0/2332383423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b6409a240 0x7f7b640c9d00 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f7b54000c00 tx=0x7f7b540046c0 comp rx=0 tx=0).stop 2024-03-24T00:47:19.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.643+0000 7f7b6944b640 1 -- 172.21.15.67:0/2332383423 shutdown_connections 2024-03-24T00:47:19.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.643+0000 7f7b6944b640 1 --2- 172.21.15.67:0/2332383423 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7b4406bbd0 0x7f7b4406e090 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:19.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.643+0000 7f7b6944b640 1 --2- 172.21.15.67:0/2332383423 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b6415e1b0 0x7f7b640c5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:19.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.643+0000 7f7b6944b640 1 --2- 172.21.15.67:0/2332383423 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7b640b43a0 0x7f7b6414ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:19.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.643+0000 7f7b6944b640 1 --2- 172.21.15.67:0/2332383423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b6409a240 0x7f7b640c9d00 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:19.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.643+0000 7f7b6944b640 1 -- 172.21.15.67:0/2332383423 >> 172.21.15.67:0/2332383423 conn(0x7f7b640b36d0 msgr2=0x7f7b64070c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:19.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.643+0000 7f7b6944b640 1 -- 172.21.15.67:0/2332383423 shutdown_connections 2024-03-24T00:47:19.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.643+0000 7f7b6944b640 1 -- 172.21.15.67:0/2332383423 wait complete. 2024-03-24T00:47:19.654 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:47:19.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.755+0000 7f5c3cfdf640 1 Processor -- start 2024-03-24T00:47:19.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.755+0000 7f5c3cfdf640 1 -- start start 2024-03-24T00:47:19.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.757+0000 7f5c3cfdf640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5c3815dec0 0x7f5c380ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:19.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.757+0000 7f5c3cfdf640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5c3815e2a0 0x7f5c38162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:19.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.757+0000 7f5c3cfdf640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c380cf640 con 0x7f5c380d07e0 2024-03-24T00:47:19.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.757+0000 7f5c3cfdf640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c380cee30 con 0x7f5c3815dec0 2024-03-24T00:47:19.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.757+0000 7f5c3cfdf640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c380d2120 con 0x7f5c3815e2a0 2024-03-24T00:47:19.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.757+0000 7f5c3659c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5c3815dec0 0x7f5c380ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:19.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.757+0000 7f5c3659c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5c3815dec0 0x7f5c380ce630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47198/0 (socket says 172.21.15.67:47198) 2024-03-24T00:47:19.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.757+0000 7f5c3659c640 1 -- 172.21.15.67:0/955858748 learned_addr learned my addr 172.21.15.67:0/955858748 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:19.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.757+0000 7f5c35d9b640 1 --2- 172.21.15.67:0/955858748 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5c3815e2a0 0x7f5c38162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:19.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.757+0000 7f5c3659c640 1 -- 172.21.15.67:0/955858748 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5c3815e2a0 msgr2=0x7f5c38162630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:19.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.757+0000 7f5c3659c640 1 --2- 172.21.15.67:0/955858748 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5c3815e2a0 0x7f5c38162630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:19.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.757+0000 7f5c3659c640 1 -- 172.21.15.67:0/955858748 >> v1:172.21.15.72:6789/0 conn(0x7f5c380d07e0 legacy=0x7f5c380b5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:19.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.757+0000 7f5c3659c640 1 -- 172.21.15.67:0/955858748 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c380cf080 con 0x7f5c3815dec0 2024-03-24T00:47:19.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.758+0000 7f5c3659c640 1 --2- 172.21.15.67:0/955858748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c3815dec0 0x7f5c380ce630 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f5c380cefb0 tx=0x7f5c3002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba23fedab9d4886e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:19.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.758+0000 7f5c3559a640 1 -- 172.21.15.67:0/955858748 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c30035020 con 0x7f5c3815dec0 2024-03-24T00:47:19.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.758+0000 7f5c3559a640 1 -- 172.21.15.67:0/955858748 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c30031640 con 0x7f5c3815dec0 2024-03-24T00:47:19.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.758+0000 7f5c3559a640 1 -- 172.21.15.67:0/955858748 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3002d040 con 0x7f5c3815dec0 2024-03-24T00:47:19.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.759+0000 7f5c3cfdf640 1 -- 172.21.15.67:0/955858748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c3815dec0 msgr2=0x7f5c380ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:19.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.759+0000 7f5c3cfdf640 1 --2- 172.21.15.67:0/955858748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c3815dec0 0x7f5c380ce630 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f5c380cefb0 tx=0x7f5c3002b8f0 comp rx=0 tx=0).stop 2024-03-24T00:47:19.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.759+0000 7f5c3cfdf640 1 -- 172.21.15.67:0/955858748 shutdown_connections 2024-03-24T00:47:19.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.759+0000 7f5c3cfdf640 1 --2- 172.21.15.67:0/955858748 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5c3815e2a0 0x7f5c38162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:19.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.759+0000 7f5c3cfdf640 1 --2- 172.21.15.67:0/955858748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c3815dec0 0x7f5c380ce630 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:19.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.759+0000 7f5c3cfdf640 1 -- 172.21.15.67:0/955858748 >> 172.21.15.67:0/955858748 conn(0x7f5c3806c810 msgr2=0x7f5c3806cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:19.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.759+0000 7f5c3cfdf640 1 -- 172.21.15.67:0/955858748 shutdown_connections 2024-03-24T00:47:19.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.759+0000 7f5c3cfdf640 1 -- 172.21.15.67:0/955858748 wait complete. 2024-03-24T00:47:19.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.760+0000 7f5c3cfdf640 1 Processor -- start 2024-03-24T00:47:19.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.760+0000 7f5c3cfdf640 1 -- start start 2024-03-24T00:47:19.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.761+0000 7f5c3cfdf640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c380d07e0 0x7f5c380cc990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:19.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.761+0000 7f5c3cfdf640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c3815dec0 0x7f5c38155b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:19.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.761+0000 7f5c3cfdf640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5c3815e2a0 0x7f5c380caea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:19.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.761+0000 7f5c3cfdf640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c380d3740 con 0x7f5c3815dec0 2024-03-24T00:47:19.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.761+0000 7f5c3cfdf640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c380d35c0 con 0x7f5c380d07e0 2024-03-24T00:47:19.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.761+0000 7f5c3cfdf640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c380d38c0 con 0x7f5c3815e2a0 2024-03-24T00:47:19.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.761+0000 7f5c35d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c3815dec0 0x7f5c38155b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:19.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.761+0000 7f5c35d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c3815dec0 0x7f5c38155b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47210/0 (socket says 172.21.15.67:47210) 2024-03-24T00:47:19.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.761+0000 7f5c35d9b640 1 -- 172.21.15.67:0/2137707444 learned_addr learned my addr 172.21.15.67:0/2137707444 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:19.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.761+0000 7f5c3659c640 1 --2- 172.21.15.67:0/2137707444 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c380d07e0 0x7f5c380cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:19.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.761+0000 7f5c36d9d640 1 --2- 172.21.15.67:0/2137707444 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5c3815e2a0 0x7f5c380caea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:19.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.762+0000 7f5c35d9b640 1 -- 172.21.15.67:0/2137707444 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5c3815e2a0 msgr2=0x7f5c380caea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:19.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.762+0000 7f5c35d9b640 1 --2- 172.21.15.67:0/2137707444 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5c3815e2a0 0x7f5c380caea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:19.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.762+0000 7f5c35d9b640 1 -- 172.21.15.67:0/2137707444 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c380d07e0 msgr2=0x7f5c380cc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:19.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.762+0000 7f5c35d9b640 1 --2- 172.21.15.67:0/2137707444 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c380d07e0 0x7f5c380cc990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:19.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.762+0000 7f5c35d9b640 1 -- 172.21.15.67:0/2137707444 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c381560c0 con 0x7f5c3815dec0 2024-03-24T00:47:19.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.762+0000 7f5c36d9d640 1 --2- 172.21.15.67:0/2137707444 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5c3815e2a0 0x7f5c380caea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:47:19.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.762+0000 7f5c3659c640 1 --2- 172.21.15.67:0/2137707444 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c380d07e0 0x7f5c380cc990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:47:19.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.762+0000 7f5c35d9b640 1 --2- 172.21.15.67:0/2137707444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c3815dec0 0x7f5c38155b80 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f5c2c00d5c0 tx=0x7f5c2c00da80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:19.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.762+0000 7f5c26ffd640 1 -- 172.21.15.67:0/2137707444 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c2c0125f0 con 0x7f5c3815dec0 2024-03-24T00:47:19.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.762+0000 7f5c26ffd640 1 -- 172.21.15.67:0/2137707444 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c2c019070 con 0x7f5c3815dec0 2024-03-24T00:47:19.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.762+0000 7f5c3cfdf640 1 -- 172.21.15.67:0/2137707444 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c380b6620 con 0x7f5c3815dec0 2024-03-24T00:47:19.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.762+0000 7f5c26ffd640 1 -- 172.21.15.67:0/2137707444 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c2c0129d0 con 0x7f5c3815dec0 2024-03-24T00:47:19.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.762+0000 7f5c3cfdf640 1 -- 172.21.15.67:0/2137707444 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c380b6be0 con 0x7f5c3815dec0 2024-03-24T00:47:19.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.763+0000 7f5c3cfdf640 1 -- 172.21.15.67:0/2137707444 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c3804f400 con 0x7f5c3815dec0 2024-03-24T00:47:19.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.763+0000 7f5c26ffd640 1 -- 172.21.15.67:0/2137707444 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5c2c012b70 con 0x7f5c3815dec0 2024-03-24T00:47:19.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.764+0000 7f5c26ffd640 1 --2- 172.21.15.67:0/2137707444 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5c1806bbd0 0x7f5c1806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:19.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.764+0000 7f5c26ffd640 1 -- 172.21.15.67:0/2137707444 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5c2c08afd0 con 0x7f5c3815dec0 2024-03-24T00:47:19.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.764+0000 7f5c3659c640 1 --2- 172.21.15.67:0/2137707444 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5c1806bbd0 0x7f5c1806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:19.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.765+0000 7f5c3659c640 1 --2- 172.21.15.67:0/2137707444 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5c1806bbd0 0x7f5c1806e090 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f5c30036000 tx=0x7f5c30008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:19.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.769+0000 7f5c26ffd640 1 -- 172.21.15.67:0/2137707444 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5c2c059930 con 0x7f5c3815dec0 2024-03-24T00:47:19.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.955+0000 7f5c3cfdf640 1 -- 172.21.15.67:0/2137707444 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5c380d1aa0 con 0x7f5c3815dec0 2024-03-24T00:47:19.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.956+0000 7f5c26ffd640 1 -- 172.21.15.67:0/2137707444 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v105) v1 ==== 74+0+16824 (secure 0 0 0) 0x7f5c2c059230 con 0x7f5c3815dec0 2024-03-24T00:47:19.956 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:47:19.956 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":105,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:47:18.128465+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:47:15.111478+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-03-24T00:47:15.455832+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:47:19.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.958+0000 7f5c3cfdf640 1 -- 172.21.15.67:0/2137707444 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5c1806bbd0 msgr2=0x7f5c1806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:19.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.958+0000 7f5c3cfdf640 1 --2- 172.21.15.67:0/2137707444 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5c1806bbd0 0x7f5c1806e090 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f5c30036000 tx=0x7f5c30008030 comp rx=0 tx=0).stop 2024-03-24T00:47:19.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.958+0000 7f5c3cfdf640 1 -- 172.21.15.67:0/2137707444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c3815dec0 msgr2=0x7f5c38155b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:19.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.958+0000 7f5c3cfdf640 1 --2- 172.21.15.67:0/2137707444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c3815dec0 0x7f5c38155b80 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f5c2c00d5c0 tx=0x7f5c2c00da80 comp rx=0 tx=0).stop 2024-03-24T00:47:19.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.958+0000 7f5c3cfdf640 1 -- 172.21.15.67:0/2137707444 shutdown_connections 2024-03-24T00:47:19.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.958+0000 7f5c3cfdf640 1 --2- 172.21.15.67:0/2137707444 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5c1806bbd0 0x7f5c1806e090 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:19.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.958+0000 7f5c3cfdf640 1 --2- 172.21.15.67:0/2137707444 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5c3815e2a0 0x7f5c380caea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:19.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.958+0000 7f5c3cfdf640 1 --2- 172.21.15.67:0/2137707444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c3815dec0 0x7f5c38155b80 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:19.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.958+0000 7f5c3cfdf640 1 --2- 172.21.15.67:0/2137707444 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c380d07e0 0x7f5c380cc990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:19.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.958+0000 7f5c3cfdf640 1 -- 172.21.15.67:0/2137707444 >> 172.21.15.67:0/2137707444 conn(0x7f5c3806c810 msgr2=0x7f5c3815c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:19.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.958+0000 7f5c3cfdf640 1 -- 172.21.15.67:0/2137707444 shutdown_connections 2024-03-24T00:47:19.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:19.958+0000 7f5c3cfdf640 1 -- 172.21.15.67:0/2137707444 wait complete. 2024-03-24T00:47:19.969 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-24T00:47:19.969 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:47:20.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.070+0000 7f84126ff640 1 Processor -- start 2024-03-24T00:47:20.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.070+0000 7f84126ff640 1 -- start start 2024-03-24T00:47:20.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.071+0000 7f84126ff640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f840c15dec0 0x7f840c0ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:20.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.071+0000 7f84126ff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f840c0d0ef0 0x7f840c162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:20.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.071+0000 7f84126ff640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f840c0cf640 con 0x7f840c15e2a0 2024-03-24T00:47:20.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.071+0000 7f84126ff640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f840c0cee30 con 0x7f840c0d0ef0 2024-03-24T00:47:20.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.071+0000 7f84126ff640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f840c0d2120 con 0x7f840c15dec0 2024-03-24T00:47:20.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.071+0000 7f8410c9c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f840c0d0ef0 0x7f840c162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:20.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.071+0000 7f8410c9c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f840c0d0ef0 0x7f840c162630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47220/0 (socket says 172.21.15.67:47220) 2024-03-24T00:47:20.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.071+0000 7f8410c9c640 1 -- 172.21.15.67:0/131445270 learned_addr learned my addr 172.21.15.67:0/131445270 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:20.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.072+0000 7f840bfff640 1 --2- 172.21.15.67:0/131445270 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f840c15dec0 0x7f840c0ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:20.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.072+0000 7f8410c9c640 1 -- 172.21.15.67:0/131445270 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f840c15dec0 msgr2=0x7f840c0ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:20.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.072+0000 7f8410c9c640 1 --2- 172.21.15.67:0/131445270 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f840c15dec0 0x7f840c0ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:20.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.072+0000 7f8410c9c640 1 -- 172.21.15.67:0/131445270 >> v1:172.21.15.72:6789/0 conn(0x7f840c15e2a0 legacy=0x7f840c0d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:20.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.072+0000 7f8410c9c640 1 -- 172.21.15.67:0/131445270 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f840c0cf080 con 0x7f840c0d0ef0 2024-03-24T00:47:20.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.072+0000 7f840bfff640 1 --2- 172.21.15.67:0/131445270 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f840c15dec0 0x7f840c0ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:47:20.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.072+0000 7f8410c9c640 1 --2- 172.21.15.67:0/131445270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f840c0d0ef0 0x7f840c162630 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f8404009e50 tx=0x7f840402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5406dc49ccb6e82e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:20.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.073+0000 7f840affd640 1 -- 172.21.15.67:0/131445270 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f840402c9e0 con 0x7f840c0d0ef0 2024-03-24T00:47:20.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.073+0000 7f840affd640 1 -- 172.21.15.67:0/131445270 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f840402cb80 con 0x7f840c0d0ef0 2024-03-24T00:47:20.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.073+0000 7f840affd640 1 -- 172.21.15.67:0/131445270 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84040343f0 con 0x7f840c0d0ef0 2024-03-24T00:47:20.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.073+0000 7f84126ff640 1 -- 172.21.15.67:0/131445270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f840c0d0ef0 msgr2=0x7f840c162630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:20.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.073+0000 7f84126ff640 1 --2- 172.21.15.67:0/131445270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f840c0d0ef0 0x7f840c162630 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f8404009e50 tx=0x7f840402be40 comp rx=0 tx=0).stop 2024-03-24T00:47:20.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.073+0000 7f84126ff640 1 -- 172.21.15.67:0/131445270 shutdown_connections 2024-03-24T00:47:20.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.073+0000 7f84126ff640 1 --2- 172.21.15.67:0/131445270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f840c0d0ef0 0x7f840c162630 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:20.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.073+0000 7f84126ff640 1 --2- 172.21.15.67:0/131445270 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f840c15dec0 0x7f840c0ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:20.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.073+0000 7f84126ff640 1 -- 172.21.15.67:0/131445270 >> 172.21.15.67:0/131445270 conn(0x7f840c06c810 msgr2=0x7f840c06cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:20.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.073+0000 7f84126ff640 1 -- 172.21.15.67:0/131445270 shutdown_connections 2024-03-24T00:47:20.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.073+0000 7f84126ff640 1 -- 172.21.15.67:0/131445270 wait complete. 2024-03-24T00:47:20.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.075+0000 7f84126ff640 1 Processor -- start 2024-03-24T00:47:20.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.075+0000 7f84126ff640 1 -- start start 2024-03-24T00:47:20.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.076+0000 7f84126ff640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f840c0d0ef0 0x7f840c0b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:20.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.076+0000 7f84126ff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f840c15dec0 0x7f840c0cd7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:20.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.076+0000 7f84126ff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f840c15e2a0 0x7f840c0b5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:20.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.076+0000 7f84126ff640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f840c0d2730 con 0x7f840c15dec0 2024-03-24T00:47:20.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.076+0000 7f84126ff640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f840c0d25b0 con 0x7f840c15e2a0 2024-03-24T00:47:20.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.076+0000 7f84126ff640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f840c0d28b0 con 0x7f840c0d0ef0 2024-03-24T00:47:20.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.076+0000 7f840b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f840c15dec0 0x7f840c0cd7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:20.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.077+0000 7f840b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f840c15dec0 0x7f840c0cd7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47228/0 (socket says 172.21.15.67:47228) 2024-03-24T00:47:20.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.077+0000 7f840b7fe640 1 -- 172.21.15.67:0/2219016152 learned_addr learned my addr 172.21.15.67:0/2219016152 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:20.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.077+0000 7f8410c9c640 1 --2- 172.21.15.67:0/2219016152 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f840c15e2a0 0x7f840c0b5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:20.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.077+0000 7f840bfff640 1 --2- 172.21.15.67:0/2219016152 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f840c0d0ef0 0x7f840c0b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:20.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.077+0000 7f840b7fe640 1 -- 172.21.15.67:0/2219016152 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f840c0d0ef0 msgr2=0x7f840c0b4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:20.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.077+0000 7f840b7fe640 1 --2- 172.21.15.67:0/2219016152 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f840c0d0ef0 0x7f840c0b4ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:20.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.077+0000 7f840b7fe640 1 -- 172.21.15.67:0/2219016152 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f840c15e2a0 msgr2=0x7f840c0b5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:20.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.077+0000 7f840b7fe640 1 --2- 172.21.15.67:0/2219016152 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f840c15e2a0 0x7f840c0b5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:20.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.077+0000 7f840b7fe640 1 -- 172.21.15.67:0/2219016152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f840c159020 con 0x7f840c15dec0 2024-03-24T00:47:20.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.077+0000 7f840b7fe640 1 --2- 172.21.15.67:0/2219016152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f840c15dec0 0x7f840c0cd7f0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f840000b3c0 tx=0x7f840000b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:20.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.077+0000 7f8408ff9640 1 -- 172.21.15.67:0/2219016152 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8400011830 con 0x7f840c15dec0 2024-03-24T00:47:20.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.077+0000 7f8408ff9640 1 -- 172.21.15.67:0/2219016152 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8400016070 con 0x7f840c15dec0 2024-03-24T00:47:20.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.077+0000 7f8408ff9640 1 -- 172.21.15.67:0/2219016152 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8400011c50 con 0x7f840c15dec0 2024-03-24T00:47:20.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.077+0000 7f84126ff640 1 -- 172.21.15.67:0/2219016152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f840c159300 con 0x7f840c15dec0 2024-03-24T00:47:20.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.077+0000 7f84126ff640 1 -- 172.21.15.67:0/2219016152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f840c1581c0 con 0x7f840c15dec0 2024-03-24T00:47:20.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.078+0000 7f84126ff640 1 -- 172.21.15.67:0/2219016152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f840c157d10 con 0x7f840c15dec0 2024-03-24T00:47:20.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.079+0000 7f8408ff9640 1 -- 172.21.15.67:0/2219016152 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f840001b460 con 0x7f840c15dec0 2024-03-24T00:47:20.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.079+0000 7f8408ff9640 1 --2- 172.21.15.67:0/2219016152 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f83f806bbd0 0x7f83f806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:20.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.079+0000 7f8408ff9640 1 -- 172.21.15.67:0/2219016152 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f840008a6c0 con 0x7f840c15dec0 2024-03-24T00:47:20.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.079+0000 7f840bfff640 1 --2- 172.21.15.67:0/2219016152 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f83f806bbd0 0x7f83f806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:20.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.080+0000 7f840bfff640 1 --2- 172.21.15.67:0/2219016152 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f83f806bbd0 0x7f83f806e090 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f83fc00a990 tx=0x7f83fc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:20.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.084+0000 7f8408ff9640 1 -- 172.21.15.67:0/2219016152 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8400020020 con 0x7f840c15dec0 2024-03-24T00:47:20.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.279+0000 7f84126ff640 1 -- 172.21.15.67:0/2219016152 --> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f840c158dd0 con 0x7f83f806bbd0 2024-03-24T00:47:20.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.286+0000 7f8408ff9640 1 -- 172.21.15.67:0/2219016152 <== mgr.4105 v2:172.21.15.72:6832/2768292274 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252485 (secure 0 0 0) 0x7f840c158dd0 con 0x7f83f806bbd0 2024-03-24T00:47:20.288 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:47:20.289 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-24T00:47:20.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.288+0000 7f84126ff640 1 -- 172.21.15.67:0/2219016152 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f83f806bbd0 msgr2=0x7f83f806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:20.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.288+0000 7f84126ff640 1 --2- 172.21.15.67:0/2219016152 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f83f806bbd0 0x7f83f806e090 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f83fc00a990 tx=0x7f83fc008040 comp rx=0 tx=0).stop 2024-03-24T00:47:20.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.288+0000 7f84126ff640 1 -- 172.21.15.67:0/2219016152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f840c15dec0 msgr2=0x7f840c0cd7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:20.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.288+0000 7f84126ff640 1 --2- 172.21.15.67:0/2219016152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f840c15dec0 0x7f840c0cd7f0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f840000b3c0 tx=0x7f840000b880 comp rx=0 tx=0).stop 2024-03-24T00:47:20.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.288+0000 7f84126ff640 1 -- 172.21.15.67:0/2219016152 shutdown_connections 2024-03-24T00:47:20.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.288+0000 7f84126ff640 1 --2- 172.21.15.67:0/2219016152 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f83f806bbd0 0x7f83f806e090 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:20.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.288+0000 7f84126ff640 1 --2- 172.21.15.67:0/2219016152 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f840c15e2a0 0x7f840c0b5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:20.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.289+0000 7f84126ff640 1 --2- 172.21.15.67:0/2219016152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f840c15dec0 0x7f840c0cd7f0 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:20.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.289+0000 7f84126ff640 1 --2- 172.21.15.67:0/2219016152 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f840c0d0ef0 0x7f840c0b4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:20.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.289+0000 7f84126ff640 1 -- 172.21.15.67:0/2219016152 >> 172.21.15.67:0/2219016152 conn(0x7f840c06c810 msgr2=0x7f840c160c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:20.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.289+0000 7f84126ff640 1 -- 172.21.15.67:0/2219016152 shutdown_connections 2024-03-24T00:47:20.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.289+0000 7f84126ff640 1 -- 172.21.15.67:0/2219016152 wait complete. 2024-03-24T00:47:20.299 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":358,"stamp":"2024-03-24T00:47:19.443999+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":156,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":260844,"kb_used_data":11376,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754713876,"statfs":{"total":773094113280,"available":772827009024,"internally_reserved":0,"allocated":11649024,"data_stored":6023068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.323344"},"pg_stats":[{"pgid":"16.3f","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.131414+0000","last_change":"2024-03-24T00:47:17.147223+0000","last_active":"2024-03-24T00:47:18.131414+0000","last_peered":"2024-03-24T00:47:18.131414+0000","last_clean":"2024-03-24T00:47:18.131414+0000","last_became_active":"2024-03-24T00:47:17.146523+0000","last_became_peered":"2024-03-24T00:47:17.146523+0000","last_unstale":"2024-03-24T00:47:18.131414+0000","last_undegraded":"2024-03-24T00:47:18.131414+0000","last_fullsized":"2024-03-24T00:47:18.131414+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:05:38.608602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3e","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477708+0000","last_change":"2024-03-24T00:47:17.151017+0000","last_active":"2024-03-24T00:47:18.477708+0000","last_peered":"2024-03-24T00:47:18.477708+0000","last_clean":"2024-03-24T00:47:18.477708+0000","last_became_active":"2024-03-24T00:47:17.150743+0000","last_became_peered":"2024-03-24T00:47:17.150743+0000","last_unstale":"2024-03-24T00:47:18.477708+0000","last_undegraded":"2024-03-24T00:47:18.477708+0000","last_fullsized":"2024-03-24T00:47:18.477708+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:46:04.860603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.3d","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.131683+0000","last_change":"2024-03-24T00:47:17.140449+0000","last_active":"2024-03-24T00:47:18.131683+0000","last_peered":"2024-03-24T00:47:18.131683+0000","last_clean":"2024-03-24T00:47:18.131683+0000","last_became_active":"2024-03-24T00:47:17.140123+0000","last_became_peered":"2024-03-24T00:47:17.140123+0000","last_unstale":"2024-03-24T00:47:18.131683+0000","last_undegraded":"2024-03-24T00:47:18.131683+0000","last_fullsized":"2024-03-24T00:47:18.131683+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:24:51.948592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3c","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.478311+0000","last_change":"2024-03-24T00:47:17.155744+0000","last_active":"2024-03-24T00:47:18.478311+0000","last_peered":"2024-03-24T00:47:18.478311+0000","last_clean":"2024-03-24T00:47:18.478311+0000","last_became_active":"2024-03-24T00:47:17.155459+0000","last_became_peered":"2024-03-24T00:47:17.155459+0000","last_unstale":"2024-03-24T00:47:18.478311+0000","last_undegraded":"2024-03-24T00:47:18.478311+0000","last_fullsized":"2024-03-24T00:47:18.478311+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:45:21.442043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.3b","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477298+0000","last_change":"2024-03-24T00:47:17.151494+0000","last_active":"2024-03-24T00:47:18.477298+0000","last_peered":"2024-03-24T00:47:18.477298+0000","last_clean":"2024-03-24T00:47:18.477298+0000","last_became_active":"2024-03-24T00:47:17.150735+0000","last_became_peered":"2024-03-24T00:47:17.150735+0000","last_unstale":"2024-03-24T00:47:18.477298+0000","last_undegraded":"2024-03-24T00:47:18.477298+0000","last_fullsized":"2024-03-24T00:47:18.477298+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:15:29.178086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.3a","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477547+0000","last_change":"2024-03-24T00:47:17.161009+0000","last_active":"2024-03-24T00:47:18.477547+0000","last_peered":"2024-03-24T00:47:18.477547+0000","last_clean":"2024-03-24T00:47:18.477547+0000","last_became_active":"2024-03-24T00:47:17.160357+0000","last_became_peered":"2024-03-24T00:47:17.160357+0000","last_unstale":"2024-03-24T00:47:18.477547+0000","last_undegraded":"2024-03-24T00:47:18.477547+0000","last_fullsized":"2024-03-24T00:47:18.477547+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:13:47.628605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.39","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477198+0000","last_change":"2024-03-24T00:47:17.160775+0000","last_active":"2024-03-24T00:47:18.477198+0000","last_peered":"2024-03-24T00:47:18.477198+0000","last_clean":"2024-03-24T00:47:18.477198+0000","last_became_active":"2024-03-24T00:47:17.160297+0000","last_became_peered":"2024-03-24T00:47:17.160297+0000","last_unstale":"2024-03-24T00:47:18.477198+0000","last_undegraded":"2024-03-24T00:47:18.477198+0000","last_fullsized":"2024-03-24T00:47:18.477198+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:12:49.377452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.38","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.476798+0000","last_change":"2024-03-24T00:47:17.152081+0000","last_active":"2024-03-24T00:47:18.476798+0000","last_peered":"2024-03-24T00:47:18.476798+0000","last_clean":"2024-03-24T00:47:18.476798+0000","last_became_active":"2024-03-24T00:47:17.151796+0000","last_became_peered":"2024-03-24T00:47:17.151796+0000","last_unstale":"2024-03-24T00:47:18.476798+0000","last_undegraded":"2024-03-24T00:47:18.476798+0000","last_fullsized":"2024-03-24T00:47:18.476798+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:14:27.957978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.37","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477577+0000","last_change":"2024-03-24T00:47:17.157494+0000","last_active":"2024-03-24T00:47:18.477577+0000","last_peered":"2024-03-24T00:47:18.477577+0000","last_clean":"2024-03-24T00:47:18.477577+0000","last_became_active":"2024-03-24T00:47:17.157217+0000","last_became_peered":"2024-03-24T00:47:17.157217+0000","last_unstale":"2024-03-24T00:47:18.477577+0000","last_undegraded":"2024-03-24T00:47:18.477577+0000","last_fullsized":"2024-03-24T00:47:18.477577+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:00:00.156306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.36","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.478216+0000","last_change":"2024-03-24T00:47:17.149194+0000","last_active":"2024-03-24T00:47:18.478216+0000","last_peered":"2024-03-24T00:47:18.478216+0000","last_clean":"2024-03-24T00:47:18.478216+0000","last_became_active":"2024-03-24T00:47:17.148885+0000","last_became_peered":"2024-03-24T00:47:17.148885+0000","last_unstale":"2024-03-24T00:47:18.478216+0000","last_undegraded":"2024-03-24T00:47:18.478216+0000","last_fullsized":"2024-03-24T00:47:18.478216+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:37:11.302019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.35","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.135187+0000","last_change":"2024-03-24T00:47:17.153113+0000","last_active":"2024-03-24T00:47:18.135187+0000","last_peered":"2024-03-24T00:47:18.135187+0000","last_clean":"2024-03-24T00:47:18.135187+0000","last_became_active":"2024-03-24T00:47:17.152718+0000","last_became_peered":"2024-03-24T00:47:17.152718+0000","last_unstale":"2024-03-24T00:47:18.135187+0000","last_undegraded":"2024-03-24T00:47:18.135187+0000","last_fullsized":"2024-03-24T00:47:18.135187+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:27:13.774555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.34","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.134714+0000","last_change":"2024-03-24T00:47:17.158198+0000","last_active":"2024-03-24T00:47:18.134714+0000","last_peered":"2024-03-24T00:47:18.134714+0000","last_clean":"2024-03-24T00:47:18.134714+0000","last_became_active":"2024-03-24T00:47:17.156616+0000","last_became_peered":"2024-03-24T00:47:17.156616+0000","last_unstale":"2024-03-24T00:47:18.134714+0000","last_undegraded":"2024-03-24T00:47:18.134714+0000","last_fullsized":"2024-03-24T00:47:18.134714+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:37:46.124771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.33","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477357+0000","last_change":"2024-03-24T00:47:17.147949+0000","last_active":"2024-03-24T00:47:18.477357+0000","last_peered":"2024-03-24T00:47:18.477357+0000","last_clean":"2024-03-24T00:47:18.477357+0000","last_became_active":"2024-03-24T00:47:17.147656+0000","last_became_peered":"2024-03-24T00:47:17.147656+0000","last_unstale":"2024-03-24T00:47:18.477357+0000","last_undegraded":"2024-03-24T00:47:18.477357+0000","last_fullsized":"2024-03-24T00:47:18.477357+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:52:34.023840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.32","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477547+0000","last_change":"2024-03-24T00:47:17.147768+0000","last_active":"2024-03-24T00:47:18.477547+0000","last_peered":"2024-03-24T00:47:18.477547+0000","last_clean":"2024-03-24T00:47:18.477547+0000","last_became_active":"2024-03-24T00:47:17.147516+0000","last_became_peered":"2024-03-24T00:47:17.147516+0000","last_unstale":"2024-03-24T00:47:18.477547+0000","last_undegraded":"2024-03-24T00:47:18.477547+0000","last_fullsized":"2024-03-24T00:47:18.477547+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:44:57.547374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.31","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477390+0000","last_change":"2024-03-24T00:47:17.154501+0000","last_active":"2024-03-24T00:47:18.477390+0000","last_peered":"2024-03-24T00:47:18.477390+0000","last_clean":"2024-03-24T00:47:18.477390+0000","last_became_active":"2024-03-24T00:47:17.154232+0000","last_became_peered":"2024-03-24T00:47:17.154232+0000","last_unstale":"2024-03-24T00:47:18.477390+0000","last_undegraded":"2024-03-24T00:47:18.477390+0000","last_fullsized":"2024-03-24T00:47:18.477390+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:27:13.774555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.30","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477498+0000","last_change":"2024-03-24T00:47:17.149209+0000","last_active":"2024-03-24T00:47:18.477498+0000","last_peered":"2024-03-24T00:47:18.477498+0000","last_clean":"2024-03-24T00:47:18.477498+0000","last_became_active":"2024-03-24T00:47:17.148887+0000","last_became_peered":"2024-03-24T00:47:17.148887+0000","last_unstale":"2024-03-24T00:47:18.477498+0000","last_undegraded":"2024-03-24T00:47:18.477498+0000","last_fullsized":"2024-03-24T00:47:18.477498+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:08:14.585720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.2f","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.131555+0000","last_change":"2024-03-24T00:47:17.140341+0000","last_active":"2024-03-24T00:47:18.131555+0000","last_peered":"2024-03-24T00:47:18.131555+0000","last_clean":"2024-03-24T00:47:18.131555+0000","last_became_active":"2024-03-24T00:47:17.140058+0000","last_became_peered":"2024-03-24T00:47:17.140058+0000","last_unstale":"2024-03-24T00:47:18.131555+0000","last_undegraded":"2024-03-24T00:47:18.131555+0000","last_fullsized":"2024-03-24T00:47:18.131555+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:35:51.675202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2e","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477876+0000","last_change":"2024-03-24T00:47:17.150236+0000","last_active":"2024-03-24T00:47:18.477876+0000","last_peered":"2024-03-24T00:47:18.477876+0000","last_clean":"2024-03-24T00:47:18.477876+0000","last_became_active":"2024-03-24T00:47:17.149800+0000","last_became_peered":"2024-03-24T00:47:17.149800+0000","last_unstale":"2024-03-24T00:47:18.477876+0000","last_undegraded":"2024-03-24T00:47:18.477876+0000","last_fullsized":"2024-03-24T00:47:18.477876+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:37:46.124771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.2d","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.131740+0000","last_change":"2024-03-24T00:47:17.155358+0000","last_active":"2024-03-24T00:47:18.131740+0000","last_peered":"2024-03-24T00:47:18.131740+0000","last_clean":"2024-03-24T00:47:18.131740+0000","last_became_active":"2024-03-24T00:47:17.154156+0000","last_became_peered":"2024-03-24T00:47:17.154156+0000","last_unstale":"2024-03-24T00:47:18.131740+0000","last_undegraded":"2024-03-24T00:47:18.131740+0000","last_fullsized":"2024-03-24T00:47:18.131740+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:08:48.681541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2c","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.134922+0000","last_change":"2024-03-24T00:47:17.152539+0000","last_active":"2024-03-24T00:47:18.134922+0000","last_peered":"2024-03-24T00:47:18.134922+0000","last_clean":"2024-03-24T00:47:18.134922+0000","last_became_active":"2024-03-24T00:47:17.152273+0000","last_became_peered":"2024-03-24T00:47:17.152273+0000","last_unstale":"2024-03-24T00:47:18.134922+0000","last_undegraded":"2024-03-24T00:47:18.134922+0000","last_fullsized":"2024-03-24T00:47:18.134922+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:50:00.059710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2b","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477513+0000","last_change":"2024-03-24T00:47:17.148950+0000","last_active":"2024-03-24T00:47:18.477513+0000","last_peered":"2024-03-24T00:47:18.477513+0000","last_clean":"2024-03-24T00:47:18.477513+0000","last_became_active":"2024-03-24T00:47:17.148661+0000","last_became_peered":"2024-03-24T00:47:17.148661+0000","last_unstale":"2024-03-24T00:47:18.477513+0000","last_undegraded":"2024-03-24T00:47:18.477513+0000","last_fullsized":"2024-03-24T00:47:18.477513+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:41:54.376692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2a","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477113+0000","last_change":"2024-03-24T00:47:17.152120+0000","last_active":"2024-03-24T00:47:18.477113+0000","last_peered":"2024-03-24T00:47:18.477113+0000","last_clean":"2024-03-24T00:47:18.477113+0000","last_became_active":"2024-03-24T00:47:17.151841+0000","last_became_peered":"2024-03-24T00:47:17.151841+0000","last_unstale":"2024-03-24T00:47:18.477113+0000","last_undegraded":"2024-03-24T00:47:18.477113+0000","last_fullsized":"2024-03-24T00:47:18.477113+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:13:45.186309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.29","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.131457+0000","last_change":"2024-03-24T00:47:17.145741+0000","last_active":"2024-03-24T00:47:18.131457+0000","last_peered":"2024-03-24T00:47:18.131457+0000","last_clean":"2024-03-24T00:47:18.131457+0000","last_became_active":"2024-03-24T00:47:17.145434+0000","last_became_peered":"2024-03-24T00:47:17.145434+0000","last_unstale":"2024-03-24T00:47:18.131457+0000","last_undegraded":"2024-03-24T00:47:18.131457+0000","last_fullsized":"2024-03-24T00:47:18.131457+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:30:56.194752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.28","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477468+0000","last_change":"2024-03-24T00:47:17.149650+0000","last_active":"2024-03-24T00:47:18.477468+0000","last_peered":"2024-03-24T00:47:18.477468+0000","last_clean":"2024-03-24T00:47:18.477468+0000","last_became_active":"2024-03-24T00:47:17.149343+0000","last_became_peered":"2024-03-24T00:47:17.149343+0000","last_unstale":"2024-03-24T00:47:18.477468+0000","last_undegraded":"2024-03-24T00:47:18.477468+0000","last_fullsized":"2024-03-24T00:47:18.477468+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:09:34.392689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.27","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.134893+0000","last_change":"2024-03-24T00:47:17.158496+0000","last_active":"2024-03-24T00:47:18.134893+0000","last_peered":"2024-03-24T00:47:18.134893+0000","last_clean":"2024-03-24T00:47:18.134893+0000","last_became_active":"2024-03-24T00:47:17.156810+0000","last_became_peered":"2024-03-24T00:47:17.156810+0000","last_unstale":"2024-03-24T00:47:18.134893+0000","last_undegraded":"2024-03-24T00:47:18.134893+0000","last_fullsized":"2024-03-24T00:47:18.134893+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:40:03.538155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.26","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.135225+0000","last_change":"2024-03-24T00:47:17.149097+0000","last_active":"2024-03-24T00:47:18.135225+0000","last_peered":"2024-03-24T00:47:18.135225+0000","last_clean":"2024-03-24T00:47:18.135225+0000","last_became_active":"2024-03-24T00:47:17.148813+0000","last_became_peered":"2024-03-24T00:47:17.148813+0000","last_unstale":"2024-03-24T00:47:18.135225+0000","last_undegraded":"2024-03-24T00:47:18.135225+0000","last_fullsized":"2024-03-24T00:47:18.135225+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:27:55.664267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.25","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.135397+0000","last_change":"2024-03-24T00:47:17.145605+0000","last_active":"2024-03-24T00:47:18.135397+0000","last_peered":"2024-03-24T00:47:18.135397+0000","last_clean":"2024-03-24T00:47:18.135397+0000","last_became_active":"2024-03-24T00:47:17.144961+0000","last_became_peered":"2024-03-24T00:47:17.144961+0000","last_unstale":"2024-03-24T00:47:18.135397+0000","last_undegraded":"2024-03-24T00:47:18.135397+0000","last_fullsized":"2024-03-24T00:47:18.135397+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:08:14.585720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.24","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477677+0000","last_change":"2024-03-24T00:47:17.153966+0000","last_active":"2024-03-24T00:47:18.477677+0000","last_peered":"2024-03-24T00:47:18.477677+0000","last_clean":"2024-03-24T00:47:18.477677+0000","last_became_active":"2024-03-24T00:47:17.153683+0000","last_became_peered":"2024-03-24T00:47:17.153683+0000","last_unstale":"2024-03-24T00:47:18.477677+0000","last_undegraded":"2024-03-24T00:47:18.477677+0000","last_fullsized":"2024-03-24T00:47:18.477677+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:53:00.441548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.23","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477660+0000","last_change":"2024-03-24T00:47:17.150198+0000","last_active":"2024-03-24T00:47:18.477660+0000","last_peered":"2024-03-24T00:47:18.477660+0000","last_clean":"2024-03-24T00:47:18.477660+0000","last_became_active":"2024-03-24T00:47:17.149903+0000","last_became_peered":"2024-03-24T00:47:17.149903+0000","last_unstale":"2024-03-24T00:47:18.477660+0000","last_undegraded":"2024-03-24T00:47:18.477660+0000","last_fullsized":"2024-03-24T00:47:18.477660+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:35:33.604622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.22","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.134432+0000","last_change":"2024-03-24T00:47:17.158820+0000","last_active":"2024-03-24T00:47:18.134432+0000","last_peered":"2024-03-24T00:47:18.134432+0000","last_clean":"2024-03-24T00:47:18.134432+0000","last_became_active":"2024-03-24T00:47:17.157921+0000","last_became_peered":"2024-03-24T00:47:17.157921+0000","last_unstale":"2024-03-24T00:47:18.134432+0000","last_undegraded":"2024-03-24T00:47:18.134432+0000","last_fullsized":"2024-03-24T00:47:18.134432+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:43:46.684428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.21","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477443+0000","last_change":"2024-03-24T00:47:17.159992+0000","last_active":"2024-03-24T00:47:18.477443+0000","last_peered":"2024-03-24T00:47:18.477443+0000","last_clean":"2024-03-24T00:47:18.477443+0000","last_became_active":"2024-03-24T00:47:17.159695+0000","last_became_peered":"2024-03-24T00:47:17.159695+0000","last_unstale":"2024-03-24T00:47:18.477443+0000","last_undegraded":"2024-03-24T00:47:18.477443+0000","last_fullsized":"2024-03-24T00:47:18.477443+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:49:59.144522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.20","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.476949+0000","last_change":"2024-03-24T00:47:17.154783+0000","last_active":"2024-03-24T00:47:18.476949+0000","last_peered":"2024-03-24T00:47:18.476949+0000","last_clean":"2024-03-24T00:47:18.476949+0000","last_became_active":"2024-03-24T00:47:17.154490+0000","last_became_peered":"2024-03-24T00:47:17.154490+0000","last_unstale":"2024-03-24T00:47:18.476949+0000","last_undegraded":"2024-03-24T00:47:18.476949+0000","last_fullsized":"2024-03-24T00:47:18.476949+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:46:36.771424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.131360+0000","last_change":"2024-03-24T00:47:17.155763+0000","last_active":"2024-03-24T00:47:18.131360+0000","last_peered":"2024-03-24T00:47:18.131360+0000","last_clean":"2024-03-24T00:47:18.131360+0000","last_became_active":"2024-03-24T00:47:17.155486+0000","last_became_peered":"2024-03-24T00:47:17.155486+0000","last_unstale":"2024-03-24T00:47:18.131360+0000","last_undegraded":"2024-03-24T00:47:18.131360+0000","last_fullsized":"2024-03-24T00:47:18.131360+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:19:44.866507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"104'4","reported_seq":18,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477978+0000","last_change":"2024-03-24T00:47:16.149652+0000","last_active":"2024-03-24T00:47:18.477978+0000","last_peered":"2024-03-24T00:47:18.477978+0000","last_clean":"2024-03-24T00:47:18.477978+0000","last_became_active":"2024-03-24T00:47:16.149370+0000","last_became_peered":"2024-03-24T00:47:16.149370+0000","last_unstale":"2024-03-24T00:47:18.477978+0000","last_undegraded":"2024-03-24T00:47:18.477978+0000","last_fullsized":"2024-03-24T00:47:18.477978+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:01:39.510298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.476960+0000","last_change":"2024-03-24T00:47:17.159058+0000","last_active":"2024-03-24T00:47:18.476960+0000","last_peered":"2024-03-24T00:47:18.476960+0000","last_clean":"2024-03-24T00:47:18.476960+0000","last_became_active":"2024-03-24T00:47:17.158779+0000","last_became_peered":"2024-03-24T00:47:17.158779+0000","last_unstale":"2024-03-24T00:47:18.476960+0000","last_undegraded":"2024-03-24T00:47:18.476960+0000","last_fullsized":"2024-03-24T00:47:18.476960+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:23:31.948889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1e","version":"104'1","reported_seq":15,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.478445+0000","last_change":"2024-03-24T00:47:16.145718+0000","last_active":"2024-03-24T00:47:18.478445+0000","last_peered":"2024-03-24T00:47:18.478445+0000","last_clean":"2024-03-24T00:47:18.478445+0000","last_became_active":"2024-03-24T00:47:16.145418+0000","last_became_peered":"2024-03-24T00:47:16.145418+0000","last_unstale":"2024-03-24T00:47:18.478445+0000","last_undegraded":"2024-03-24T00:47:18.478445+0000","last_fullsized":"2024-03-24T00:47:18.478445+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:32:00.796419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.134431+0000","last_change":"2024-03-24T00:47:17.159926+0000","last_active":"2024-03-24T00:47:18.134431+0000","last_peered":"2024-03-24T00:47:18.134431+0000","last_clean":"2024-03-24T00:47:18.134431+0000","last_became_active":"2024-03-24T00:47:17.159640+0000","last_became_peered":"2024-03-24T00:47:17.159640+0000","last_unstale":"2024-03-24T00:47:18.134431+0000","last_undegraded":"2024-03-24T00:47:18.134431+0000","last_fullsized":"2024-03-24T00:47:18.134431+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:46:04.860603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"104'2","reported_seq":16,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.135136+0000","last_change":"2024-03-24T00:47:16.146156+0000","last_active":"2024-03-24T00:47:18.135136+0000","last_peered":"2024-03-24T00:47:18.135136+0000","last_clean":"2024-03-24T00:47:18.135136+0000","last_became_active":"2024-03-24T00:47:16.145793+0000","last_became_peered":"2024-03-24T00:47:16.145793+0000","last_unstale":"2024-03-24T00:47:18.135136+0000","last_undegraded":"2024-03-24T00:47:18.135136+0000","last_fullsized":"2024-03-24T00:47:18.135136+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:35:32.597867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.134743+0000","last_change":"2024-03-24T00:47:17.153905+0000","last_active":"2024-03-24T00:47:18.134743+0000","last_peered":"2024-03-24T00:47:18.134743+0000","last_clean":"2024-03-24T00:47:18.134743+0000","last_became_active":"2024-03-24T00:47:17.153630+0000","last_became_peered":"2024-03-24T00:47:17.153630+0000","last_unstale":"2024-03-24T00:47:18.134743+0000","last_undegraded":"2024-03-24T00:47:18.134743+0000","last_fullsized":"2024-03-24T00:47:18.134743+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:13:58.356201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1c","version":"104'2","reported_seq":16,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477718+0000","last_change":"2024-03-24T00:47:16.150021+0000","last_active":"2024-03-24T00:47:18.477718+0000","last_peered":"2024-03-24T00:47:18.477718+0000","last_clean":"2024-03-24T00:47:18.477718+0000","last_became_active":"2024-03-24T00:47:16.149706+0000","last_became_peered":"2024-03-24T00:47:16.149706+0000","last_unstale":"2024-03-24T00:47:18.477718+0000","last_undegraded":"2024-03-24T00:47:18.477718+0000","last_fullsized":"2024-03-24T00:47:18.477718+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:37:45.118015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.478317+0000","last_change":"2024-03-24T00:47:17.150669+0000","last_active":"2024-03-24T00:47:18.478317+0000","last_peered":"2024-03-24T00:47:18.478317+0000","last_clean":"2024-03-24T00:47:18.478317+0000","last_became_active":"2024-03-24T00:47:17.150354+0000","last_became_peered":"2024-03-24T00:47:17.150354+0000","last_unstale":"2024-03-24T00:47:18.478317+0000","last_undegraded":"2024-03-24T00:47:18.478317+0000","last_fullsized":"2024-03-24T00:47:18.478317+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:20:13.126290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"104'1","reported_seq":15,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.478046+0000","last_change":"2024-03-24T00:47:16.149722+0000","last_active":"2024-03-24T00:47:18.478046+0000","last_peered":"2024-03-24T00:47:18.478046+0000","last_clean":"2024-03-24T00:47:18.478046+0000","last_became_active":"2024-03-24T00:47:16.148764+0000","last_became_peered":"2024-03-24T00:47:16.148764+0000","last_unstale":"2024-03-24T00:47:18.478046+0000","last_undegraded":"2024-03-24T00:47:18.478046+0000","last_fullsized":"2024-03-24T00:47:18.478046+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:37:10.295263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.131823+0000","last_change":"2024-03-24T00:47:17.154855+0000","last_active":"2024-03-24T00:47:18.131823+0000","last_peered":"2024-03-24T00:47:18.131823+0000","last_clean":"2024-03-24T00:47:18.131823+0000","last_became_active":"2024-03-24T00:47:17.153756+0000","last_became_peered":"2024-03-24T00:47:17.153756+0000","last_unstale":"2024-03-24T00:47:18.131823+0000","last_undegraded":"2024-03-24T00:47:18.131823+0000","last_fullsized":"2024-03-24T00:47:18.131823+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:56:42.633611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"104'1","reported_seq":15,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.134103+0000","last_change":"2024-03-24T00:47:16.147625+0000","last_active":"2024-03-24T00:47:18.134103+0000","last_peered":"2024-03-24T00:47:18.134103+0000","last_clean":"2024-03-24T00:47:18.134103+0000","last_became_active":"2024-03-24T00:47:16.147334+0000","last_became_peered":"2024-03-24T00:47:16.147334+0000","last_unstale":"2024-03-24T00:47:18.134103+0000","last_undegraded":"2024-03-24T00:47:18.134103+0000","last_fullsized":"2024-03-24T00:47:18.134103+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:13:44.179554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477954+0000","last_change":"2024-03-24T00:47:17.156056+0000","last_active":"2024-03-24T00:47:18.477954+0000","last_peered":"2024-03-24T00:47:18.477954+0000","last_clean":"2024-03-24T00:47:18.477954+0000","last_became_active":"2024-03-24T00:47:17.155771+0000","last_became_peered":"2024-03-24T00:47:17.155771+0000","last_unstale":"2024-03-24T00:47:18.477954+0000","last_undegraded":"2024-03-24T00:47:18.477954+0000","last_fullsized":"2024-03-24T00:47:18.477954+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:40:03.538155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":14,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.134512+0000","last_change":"2024-03-24T00:47:16.151295+0000","last_active":"2024-03-24T00:47:18.134512+0000","last_peered":"2024-03-24T00:47:18.134512+0000","last_clean":"2024-03-24T00:47:18.134512+0000","last_became_active":"2024-03-24T00:47:16.150902+0000","last_became_peered":"2024-03-24T00:47:16.150902+0000","last_unstale":"2024-03-24T00:47:18.134512+0000","last_undegraded":"2024-03-24T00:47:18.134512+0000","last_fullsized":"2024-03-24T00:47:18.134512+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:43:04.319575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477877+0000","last_change":"2024-03-24T00:47:17.143976+0000","last_active":"2024-03-24T00:47:18.477877+0000","last_peered":"2024-03-24T00:47:18.477877+0000","last_clean":"2024-03-24T00:47:18.477877+0000","last_became_active":"2024-03-24T00:47:17.143638+0000","last_became_peered":"2024-03-24T00:47:17.143638+0000","last_unstale":"2024-03-24T00:47:18.477877+0000","last_undegraded":"2024-03-24T00:47:18.477877+0000","last_fullsized":"2024-03-24T00:47:18.477877+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:41:11.873811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":14,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.478060+0000","last_change":"2024-03-24T00:47:16.150403+0000","last_active":"2024-03-24T00:47:18.478060+0000","last_peered":"2024-03-24T00:47:18.478060+0000","last_clean":"2024-03-24T00:47:18.478060+0000","last_became_active":"2024-03-24T00:47:16.150114+0000","last_became_peered":"2024-03-24T00:47:16.150114+0000","last_unstale":"2024-03-24T00:47:18.478060+0000","last_undegraded":"2024-03-24T00:47:18.478060+0000","last_fullsized":"2024-03-24T00:47:18.478060+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:05:30.554415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.131225+0000","last_change":"2024-03-24T00:47:17.156494+0000","last_active":"2024-03-24T00:47:18.131225+0000","last_peered":"2024-03-24T00:47:18.131225+0000","last_clean":"2024-03-24T00:47:18.131225+0000","last_became_active":"2024-03-24T00:47:17.156197+0000","last_became_peered":"2024-03-24T00:47:17.156197+0000","last_unstale":"2024-03-24T00:47:18.131225+0000","last_undegraded":"2024-03-24T00:47:18.131225+0000","last_fullsized":"2024-03-24T00:47:18.131225+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:31:56.106499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":14,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.135011+0000","last_change":"2024-03-24T00:47:16.147099+0000","last_active":"2024-03-24T00:47:18.135011+0000","last_peered":"2024-03-24T00:47:18.135011+0000","last_clean":"2024-03-24T00:47:18.135011+0000","last_became_active":"2024-03-24T00:47:16.145707+0000","last_became_peered":"2024-03-24T00:47:16.145707+0000","last_unstale":"2024-03-24T00:47:18.135011+0000","last_undegraded":"2024-03-24T00:47:18.135011+0000","last_fullsized":"2024-03-24T00:47:18.135011+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:20:09.491226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477066+0000","last_change":"2024-03-24T00:47:17.149301+0000","last_active":"2024-03-24T00:47:18.477066+0000","last_peered":"2024-03-24T00:47:18.477066+0000","last_clean":"2024-03-24T00:47:18.477066+0000","last_became_active":"2024-03-24T00:47:17.148972+0000","last_became_peered":"2024-03-24T00:47:17.148972+0000","last_unstale":"2024-03-24T00:47:18.477066+0000","last_undegraded":"2024-03-24T00:47:18.477066+0000","last_fullsized":"2024-03-24T00:47:18.477066+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:06:17.216361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":14,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.131300+0000","last_change":"2024-03-24T00:47:16.146221+0000","last_active":"2024-03-24T00:47:18.131300+0000","last_peered":"2024-03-24T00:47:18.131300+0000","last_clean":"2024-03-24T00:47:18.131300+0000","last_became_active":"2024-03-24T00:47:16.145948+0000","last_became_peered":"2024-03-24T00:47:16.145948+0000","last_unstale":"2024-03-24T00:47:18.131300+0000","last_undegraded":"2024-03-24T00:47:18.131300+0000","last_fullsized":"2024-03-24T00:47:18.131300+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:44:20.360231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477207+0000","last_change":"2024-03-24T00:47:17.154820+0000","last_active":"2024-03-24T00:47:18.477207+0000","last_peered":"2024-03-24T00:47:18.477207+0000","last_clean":"2024-03-24T00:47:18.477207+0000","last_became_active":"2024-03-24T00:47:17.154524+0000","last_became_peered":"2024-03-24T00:47:17.154524+0000","last_unstale":"2024-03-24T00:47:18.477207+0000","last_undegraded":"2024-03-24T00:47:18.477207+0000","last_fullsized":"2024-03-24T00:47:18.477207+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:20:10.497981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.15","version":"104'3","reported_seq":17,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477497+0000","last_change":"2024-03-24T00:47:16.146600+0000","last_active":"2024-03-24T00:47:18.477497+0000","last_peered":"2024-03-24T00:47:18.477497+0000","last_clean":"2024-03-24T00:47:18.477497+0000","last_became_active":"2024-03-24T00:47:16.146279+0000","last_became_peered":"2024-03-24T00:47:16.146279+0000","last_unstale":"2024-03-24T00:47:18.477497+0000","last_undegraded":"2024-03-24T00:47:18.477497+0000","last_fullsized":"2024-03-24T00:47:18.477497+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:48:22.549360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477592+0000","last_change":"2024-03-24T00:47:17.151264+0000","last_active":"2024-03-24T00:47:18.477592+0000","last_peered":"2024-03-24T00:47:18.477592+0000","last_clean":"2024-03-24T00:47:18.477592+0000","last_became_active":"2024-03-24T00:47:17.150449+0000","last_became_peered":"2024-03-24T00:47:17.150449+0000","last_unstale":"2024-03-24T00:47:18.477592+0000","last_undegraded":"2024-03-24T00:47:18.477592+0000","last_fullsized":"2024-03-24T00:47:18.477592+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:48:23.556116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.14","version":"104'6","reported_seq":20,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.478075+0000","last_change":"2024-03-24T00:47:16.152236+0000","last_active":"2024-03-24T00:47:18.478075+0000","last_peered":"2024-03-24T00:47:18.478075+0000","last_clean":"2024-03-24T00:47:18.478075+0000","last_became_active":"2024-03-24T00:47:16.152019+0000","last_became_peered":"2024-03-24T00:47:16.152019+0000","last_unstale":"2024-03-24T00:47:18.478075+0000","last_undegraded":"2024-03-24T00:47:18.478075+0000","last_fullsized":"2024-03-24T00:47:18.478075+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:52:59.339383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.135375+0000","last_change":"2024-03-24T00:47:17.136903+0000","last_active":"2024-03-24T00:47:18.135375+0000","last_peered":"2024-03-24T00:47:18.135375+0000","last_clean":"2024-03-24T00:47:18.135375+0000","last_became_active":"2024-03-24T00:47:17.136568+0000","last_became_peered":"2024-03-24T00:47:17.136568+0000","last_unstale":"2024-03-24T00:47:18.135375+0000","last_undegraded":"2024-03-24T00:47:18.135375+0000","last_fullsized":"2024-03-24T00:47:18.135375+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:44:57.547374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.13","version":"104'2","reported_seq":16,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477771+0000","last_change":"2024-03-24T00:47:16.149612+0000","last_active":"2024-03-24T00:47:18.477771+0000","last_peered":"2024-03-24T00:47:18.477771+0000","last_clean":"2024-03-24T00:47:18.477771+0000","last_became_active":"2024-03-24T00:47:16.149303+0000","last_became_peered":"2024-03-24T00:47:16.149303+0000","last_unstale":"2024-03-24T00:47:18.477771+0000","last_undegraded":"2024-03-24T00:47:18.477771+0000","last_fullsized":"2024-03-24T00:47:18.477771+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:49:59.052954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477777+0000","last_change":"2024-03-24T00:47:17.156700+0000","last_active":"2024-03-24T00:47:18.477777+0000","last_peered":"2024-03-24T00:47:18.477777+0000","last_clean":"2024-03-24T00:47:18.477777+0000","last_became_active":"2024-03-24T00:47:17.156438+0000","last_became_peered":"2024-03-24T00:47:17.156438+0000","last_unstale":"2024-03-24T00:47:18.477777+0000","last_undegraded":"2024-03-24T00:47:18.477777+0000","last_fullsized":"2024-03-24T00:47:18.477777+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:43:46.684428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":14,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477138+0000","last_change":"2024-03-24T00:47:16.150026+0000","last_active":"2024-03-24T00:47:18.477138+0000","last_peered":"2024-03-24T00:47:18.477138+0000","last_clean":"2024-03-24T00:47:18.477138+0000","last_became_active":"2024-03-24T00:47:16.149739+0000","last_became_peered":"2024-03-24T00:47:16.149739+0000","last_unstale":"2024-03-24T00:47:18.477138+0000","last_undegraded":"2024-03-24T00:47:18.477138+0000","last_fullsized":"2024-03-24T00:47:18.477138+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:36:40.034011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.478408+0000","last_change":"2024-03-24T00:47:17.154037+0000","last_active":"2024-03-24T00:47:18.478408+0000","last_peered":"2024-03-24T00:47:18.478408+0000","last_clean":"2024-03-24T00:47:18.478408+0000","last_became_active":"2024-03-24T00:47:17.153759+0000","last_became_peered":"2024-03-24T00:47:17.153759+0000","last_unstale":"2024-03-24T00:47:18.478408+0000","last_undegraded":"2024-03-24T00:47:18.478408+0000","last_fullsized":"2024-03-24T00:47:18.478408+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:24:33.435385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":14,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.476839+0000","last_change":"2024-03-24T00:47:16.150357+0000","last_active":"2024-03-24T00:47:18.476839+0000","last_peered":"2024-03-24T00:47:18.476839+0000","last_clean":"2024-03-24T00:47:18.476839+0000","last_became_active":"2024-03-24T00:47:16.150061+0000","last_became_peered":"2024-03-24T00:47:16.150061+0000","last_unstale":"2024-03-24T00:47:18.476839+0000","last_undegraded":"2024-03-24T00:47:18.476839+0000","last_fullsized":"2024-03-24T00:47:18.476839+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:48:13.364368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.131709+0000","last_change":"2024-03-24T00:47:17.139195+0000","last_active":"2024-03-24T00:47:18.131709+0000","last_peered":"2024-03-24T00:47:18.131709+0000","last_clean":"2024-03-24T00:47:18.131709+0000","last_became_active":"2024-03-24T00:47:17.138927+0000","last_became_peered":"2024-03-24T00:47:17.138927+0000","last_unstale":"2024-03-24T00:47:18.131709+0000","last_undegraded":"2024-03-24T00:47:18.131709+0000","last_fullsized":"2024-03-24T00:47:18.131709+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:23:08.092507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.10","version":"104'8","reported_seq":22,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.134434+0000","last_change":"2024-03-24T00:47:16.146521+0000","last_active":"2024-03-24T00:47:18.134434+0000","last_peered":"2024-03-24T00:47:18.134434+0000","last_clean":"2024-03-24T00:47:18.134434+0000","last_became_active":"2024-03-24T00:47:16.145984+0000","last_became_peered":"2024-03-24T00:47:16.145984+0000","last_unstale":"2024-03-24T00:47:18.134434+0000","last_undegraded":"2024-03-24T00:47:18.134434+0000","last_fullsized":"2024-03-24T00:47:18.134434+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:34:57.983478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477087+0000","last_change":"2024-03-24T00:47:17.147660+0000","last_active":"2024-03-24T00:47:18.477087+0000","last_peered":"2024-03-24T00:47:18.477087+0000","last_clean":"2024-03-24T00:47:18.477087+0000","last_became_active":"2024-03-24T00:47:17.147391+0000","last_became_peered":"2024-03-24T00:47:17.147391+0000","last_unstale":"2024-03-24T00:47:18.477087+0000","last_undegraded":"2024-03-24T00:47:18.477087+0000","last_fullsized":"2024-03-24T00:47:18.477087+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:43:05.326331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.f","version":"104'1","reported_seq":15,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477693+0000","last_change":"2024-03-24T00:47:16.149894+0000","last_active":"2024-03-24T00:47:18.477693+0000","last_peered":"2024-03-24T00:47:18.477693+0000","last_clean":"2024-03-24T00:47:18.477693+0000","last_became_active":"2024-03-24T00:47:16.149616+0000","last_became_peered":"2024-03-24T00:47:16.149616+0000","last_unstale":"2024-03-24T00:47:18.477693+0000","last_undegraded":"2024-03-24T00:47:18.477693+0000","last_fullsized":"2024-03-24T00:47:18.477693+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:06:24.193452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477503+0000","last_change":"2024-03-24T00:47:17.158745+0000","last_active":"2024-03-24T00:47:18.477503+0000","last_peered":"2024-03-24T00:47:18.477503+0000","last_clean":"2024-03-24T00:47:18.477503+0000","last_became_active":"2024-03-24T00:47:17.158474+0000","last_became_peered":"2024-03-24T00:47:17.158474+0000","last_unstale":"2024-03-24T00:47:18.477503+0000","last_undegraded":"2024-03-24T00:47:18.477503+0000","last_fullsized":"2024-03-24T00:47:18.477503+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:18:15.311145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"104'1","reported_seq":15,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477886+0000","last_change":"2024-03-24T00:47:16.148297+0000","last_active":"2024-03-24T00:47:18.477886+0000","last_peered":"2024-03-24T00:47:18.477886+0000","last_clean":"2024-03-24T00:47:18.477886+0000","last_became_active":"2024-03-24T00:47:16.148023+0000","last_became_peered":"2024-03-24T00:47:16.148023+0000","last_unstale":"2024-03-24T00:47:18.477886+0000","last_undegraded":"2024-03-24T00:47:18.477886+0000","last_fullsized":"2024-03-24T00:47:18.477886+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:57:16.256195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477449+0000","last_change":"2024-03-24T00:47:17.151006+0000","last_active":"2024-03-24T00:47:18.477449+0000","last_peered":"2024-03-24T00:47:18.477449+0000","last_clean":"2024-03-24T00:47:18.477449+0000","last_became_active":"2024-03-24T00:47:17.150156+0000","last_became_peered":"2024-03-24T00:47:17.150156+0000","last_unstale":"2024-03-24T00:47:18.477449+0000","last_undegraded":"2024-03-24T00:47:18.477449+0000","last_fullsized":"2024-03-24T00:47:18.477449+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:42:33.045278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":14,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.134877+0000","last_change":"2024-03-24T00:47:16.149966+0000","last_active":"2024-03-24T00:47:18.134877+0000","last_peered":"2024-03-24T00:47:18.134877+0000","last_clean":"2024-03-24T00:47:18.134877+0000","last_became_active":"2024-03-24T00:47:16.149684+0000","last_became_peered":"2024-03-24T00:47:16.149684+0000","last_unstale":"2024-03-24T00:47:18.134877+0000","last_undegraded":"2024-03-24T00:47:18.134877+0000","last_fullsized":"2024-03-24T00:47:18.134877+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:06:16.209605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.131474+0000","last_change":"2024-03-24T00:47:17.146131+0000","last_active":"2024-03-24T00:47:18.131474+0000","last_peered":"2024-03-24T00:47:18.131474+0000","last_clean":"2024-03-24T00:47:18.131474+0000","last_became_active":"2024-03-24T00:47:17.145869+0000","last_became_peered":"2024-03-24T00:47:17.145869+0000","last_unstale":"2024-03-24T00:47:18.131474+0000","last_undegraded":"2024-03-24T00:47:18.131474+0000","last_fullsized":"2024-03-24T00:47:18.131474+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:25:02.770395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"104'2","reported_seq":16,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477781+0000","last_change":"2024-03-24T00:47:16.138026+0000","last_active":"2024-03-24T00:47:18.477781+0000","last_peered":"2024-03-24T00:47:18.477781+0000","last_clean":"2024-03-24T00:47:18.477781+0000","last_became_active":"2024-03-24T00:47:16.137765+0000","last_became_peered":"2024-03-24T00:47:16.137765+0000","last_unstale":"2024-03-24T00:47:18.477781+0000","last_undegraded":"2024-03-24T00:47:18.477781+0000","last_fullsized":"2024-03-24T00:47:18.477781+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:31:01.641338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.131770+0000","last_change":"2024-03-24T00:47:17.155612+0000","last_active":"2024-03-24T00:47:18.131770+0000","last_peered":"2024-03-24T00:47:18.131770+0000","last_clean":"2024-03-24T00:47:18.131770+0000","last_became_active":"2024-03-24T00:47:17.154532+0000","last_became_peered":"2024-03-24T00:47:17.154532+0000","last_unstale":"2024-03-24T00:47:18.131770+0000","last_undegraded":"2024-03-24T00:47:18.131770+0000","last_fullsized":"2024-03-24T00:47:18.131770+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:59:27.246748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":14,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477062+0000","last_change":"2024-03-24T00:47:16.148191+0000","last_active":"2024-03-24T00:47:18.477062+0000","last_peered":"2024-03-24T00:47:18.477062+0000","last_clean":"2024-03-24T00:47:18.477062+0000","last_became_active":"2024-03-24T00:47:16.147927+0000","last_became_peered":"2024-03-24T00:47:16.147927+0000","last_unstale":"2024-03-24T00:47:18.477062+0000","last_undegraded":"2024-03-24T00:47:18.477062+0000","last_fullsized":"2024-03-24T00:47:18.477062+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:57:16.124980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477513+0000","last_change":"2024-03-24T00:47:17.152591+0000","last_active":"2024-03-24T00:47:18.477513+0000","last_peered":"2024-03-24T00:47:18.477513+0000","last_clean":"2024-03-24T00:47:18.477513+0000","last_became_active":"2024-03-24T00:47:17.152329+0000","last_became_peered":"2024-03-24T00:47:17.152329+0000","last_unstale":"2024-03-24T00:47:18.477513+0000","last_undegraded":"2024-03-24T00:47:18.477513+0000","last_fullsized":"2024-03-24T00:47:18.477513+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:17:29.186551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":14,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477313+0000","last_change":"2024-03-24T00:47:16.151542+0000","last_active":"2024-03-24T00:47:18.477313+0000","last_peered":"2024-03-24T00:47:18.477313+0000","last_clean":"2024-03-24T00:47:18.477313+0000","last_became_active":"2024-03-24T00:47:16.151219+0000","last_became_peered":"2024-03-24T00:47:16.151219+0000","last_unstale":"2024-03-24T00:47:18.477313+0000","last_undegraded":"2024-03-24T00:47:18.477313+0000","last_fullsized":"2024-03-24T00:47:18.477313+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:21:29.220609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477769+0000","last_change":"2024-03-24T00:47:17.159413+0000","last_active":"2024-03-24T00:47:18.477769+0000","last_peered":"2024-03-24T00:47:18.477769+0000","last_clean":"2024-03-24T00:47:18.477769+0000","last_became_active":"2024-03-24T00:47:17.158993+0000","last_became_peered":"2024-03-24T00:47:17.158993+0000","last_unstale":"2024-03-24T00:47:18.477769+0000","last_undegraded":"2024-03-24T00:47:18.477769+0000","last_fullsized":"2024-03-24T00:47:18.477769+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:11:45.335007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":14,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477181+0000","last_change":"2024-03-24T00:47:16.144375+0000","last_active":"2024-03-24T00:47:18.477181+0000","last_peered":"2024-03-24T00:47:18.477181+0000","last_clean":"2024-03-24T00:47:18.477181+0000","last_became_active":"2024-03-24T00:47:16.144044+0000","last_became_peered":"2024-03-24T00:47:16.144044+0000","last_unstale":"2024-03-24T00:47:18.477181+0000","last_undegraded":"2024-03-24T00:47:18.477181+0000","last_fullsized":"2024-03-24T00:47:18.477181+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:41:43.839546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.131235+0000","last_change":"2024-03-24T00:47:17.145454+0000","last_active":"2024-03-24T00:47:18.131235+0000","last_peered":"2024-03-24T00:47:18.131235+0000","last_clean":"2024-03-24T00:47:18.131235+0000","last_became_active":"2024-03-24T00:47:17.145141+0000","last_became_peered":"2024-03-24T00:47:17.145141+0000","last_unstale":"2024-03-24T00:47:18.131235+0000","last_undegraded":"2024-03-24T00:47:18.131235+0000","last_fullsized":"2024-03-24T00:47:18.131235+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:27:30.445712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"104'2","reported_seq":16,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477996+0000","last_change":"2024-03-24T00:47:16.144152+0000","last_active":"2024-03-24T00:47:18.477996+0000","last_peered":"2024-03-24T00:47:18.477996+0000","last_clean":"2024-03-24T00:47:18.477996+0000","last_became_active":"2024-03-24T00:47:16.143883+0000","last_became_peered":"2024-03-24T00:47:16.143883+0000","last_unstale":"2024-03-24T00:47:18.477996+0000","last_undegraded":"2024-03-24T00:47:18.477996+0000","last_fullsized":"2024-03-24T00:47:18.477996+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:27:54.657512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.134386+0000","last_change":"2024-03-24T00:47:17.147483+0000","last_active":"2024-03-24T00:47:18.134386+0000","last_peered":"2024-03-24T00:47:18.134386+0000","last_clean":"2024-03-24T00:47:18.134386+0000","last_became_active":"2024-03-24T00:47:17.146934+0000","last_became_peered":"2024-03-24T00:47:17.146934+0000","last_unstale":"2024-03-24T00:47:18.134386+0000","last_undegraded":"2024-03-24T00:47:18.134386+0000","last_fullsized":"2024-03-24T00:47:18.134386+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:53:00.441548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.6","version":"105'1","reported_seq":15,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:19.135413+0000","last_change":"2024-03-24T00:47:16.135888+0000","last_active":"2024-03-24T00:47:19.135413+0000","last_peered":"2024-03-24T00:47:19.135413+0000","last_clean":"2024-03-24T00:47:19.135413+0000","last_became_active":"2024-03-24T00:47:16.135601+0000","last_became_peered":"2024-03-24T00:47:16.135601+0000","last_unstale":"2024-03-24T00:47:19.135413+0000","last_undegraded":"2024-03-24T00:47:19.135413+0000","last_fullsized":"2024-03-24T00:47:19.135413+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:33:40.537415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477045+0000","last_change":"2024-03-24T00:47:17.151857+0000","last_active":"2024-03-24T00:47:18.477045+0000","last_peered":"2024-03-24T00:47:18.477045+0000","last_clean":"2024-03-24T00:47:18.477045+0000","last_became_active":"2024-03-24T00:47:17.151589+0000","last_became_peered":"2024-03-24T00:47:17.151589+0000","last_unstale":"2024-03-24T00:47:18.477045+0000","last_undegraded":"2024-03-24T00:47:18.477045+0000","last_fullsized":"2024-03-24T00:47:18.477045+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:50:41.868971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":14,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477819+0000","last_change":"2024-03-24T00:47:16.148013+0000","last_active":"2024-03-24T00:47:18.477819+0000","last_peered":"2024-03-24T00:47:18.477819+0000","last_clean":"2024-03-24T00:47:18.477819+0000","last_became_active":"2024-03-24T00:47:16.147658+0000","last_became_peered":"2024-03-24T00:47:16.147658+0000","last_unstale":"2024-03-24T00:47:18.477819+0000","last_undegraded":"2024-03-24T00:47:18.477819+0000","last_fullsized":"2024-03-24T00:47:18.477819+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:13:47.270490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.134734+0000","last_change":"2024-03-24T00:47:17.152727+0000","last_active":"2024-03-24T00:47:18.134734+0000","last_peered":"2024-03-24T00:47:18.134734+0000","last_clean":"2024-03-24T00:47:18.134734+0000","last_became_active":"2024-03-24T00:47:17.152454+0000","last_became_peered":"2024-03-24T00:47:17.152454+0000","last_unstale":"2024-03-24T00:47:18.134734+0000","last_undegraded":"2024-03-24T00:47:18.134734+0000","last_fullsized":"2024-03-24T00:47:18.134734+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:05:31.561170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":14,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477806+0000","last_change":"2024-03-24T00:47:16.149651+0000","last_active":"2024-03-24T00:47:18.477806+0000","last_peered":"2024-03-24T00:47:18.477806+0000","last_clean":"2024-03-24T00:47:18.477806+0000","last_became_active":"2024-03-24T00:47:16.149285+0000","last_became_peered":"2024-03-24T00:47:16.149285+0000","last_unstale":"2024-03-24T00:47:18.477806+0000","last_undegraded":"2024-03-24T00:47:18.477806+0000","last_fullsized":"2024-03-24T00:47:18.477806+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:40:02.531399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477638+0000","last_change":"2024-03-24T00:47:17.152781+0000","last_active":"2024-03-24T00:47:18.477638+0000","last_peered":"2024-03-24T00:47:18.477638+0000","last_clean":"2024-03-24T00:47:18.477638+0000","last_became_active":"2024-03-24T00:47:17.152426+0000","last_became_peered":"2024-03-24T00:47:17.152426+0000","last_unstale":"2024-03-24T00:47:18.477638+0000","last_undegraded":"2024-03-24T00:47:18.477638+0000","last_fullsized":"2024-03-24T00:47:18.477638+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:34:58.990234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":14,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.134661+0000","last_change":"2024-03-24T00:47:16.147713+0000","last_active":"2024-03-24T00:47:18.134661+0000","last_peered":"2024-03-24T00:47:18.134661+0000","last_clean":"2024-03-24T00:47:18.134661+0000","last_became_active":"2024-03-24T00:47:16.147366+0000","last_became_peered":"2024-03-24T00:47:16.147366+0000","last_unstale":"2024-03-24T00:47:18.134661+0000","last_undegraded":"2024-03-24T00:47:18.134661+0000","last_fullsized":"2024-03-24T00:47:18.134661+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:41:53.369936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477924+0000","last_change":"2024-03-24T00:47:17.153365+0000","last_active":"2024-03-24T00:47:18.477924+0000","last_peered":"2024-03-24T00:47:18.477924+0000","last_clean":"2024-03-24T00:47:18.477924+0000","last_became_active":"2024-03-24T00:47:17.153073+0000","last_became_peered":"2024-03-24T00:47:17.153073+0000","last_unstale":"2024-03-24T00:47:18.477924+0000","last_undegraded":"2024-03-24T00:47:18.477924+0000","last_fullsized":"2024-03-24T00:47:18.477924+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:55:30.418171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":14,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477300+0000","last_change":"2024-03-24T00:47:16.148454+0000","last_active":"2024-03-24T00:47:18.477300+0000","last_peered":"2024-03-24T00:47:18.477300+0000","last_clean":"2024-03-24T00:47:18.477300+0000","last_became_active":"2024-03-24T00:47:16.147257+0000","last_became_peered":"2024-03-24T00:47:16.147257+0000","last_unstale":"2024-03-24T00:47:18.477300+0000","last_undegraded":"2024-03-24T00:47:18.477300+0000","last_fullsized":"2024-03-24T00:47:18.477300+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:22:00.243619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477762+0000","last_change":"2024-03-24T00:47:17.153777+0000","last_active":"2024-03-24T00:47:18.477762+0000","last_peered":"2024-03-24T00:47:18.477762+0000","last_clean":"2024-03-24T00:47:18.477762+0000","last_became_active":"2024-03-24T00:47:17.153452+0000","last_became_peered":"2024-03-24T00:47:17.153452+0000","last_unstale":"2024-03-24T00:47:18.477762+0000","last_undegraded":"2024-03-24T00:47:18.477762+0000","last_fullsized":"2024-03-24T00:47:18.477762+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:53:45.283188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":14,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477539+0000","last_change":"2024-03-24T00:47:16.148022+0000","last_active":"2024-03-24T00:47:18.477539+0000","last_peered":"2024-03-24T00:47:18.477539+0000","last_clean":"2024-03-24T00:47:18.477539+0000","last_became_active":"2024-03-24T00:47:16.147216+0000","last_became_peered":"2024-03-24T00:47:16.147216+0000","last_unstale":"2024-03-24T00:47:18.477539+0000","last_undegraded":"2024-03-24T00:47:18.477539+0000","last_fullsized":"2024-03-24T00:47:18.477539+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:36:10.435766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.135579+0000","last_change":"2024-03-24T00:47:17.143293+0000","last_active":"2024-03-24T00:47:18.135579+0000","last_peered":"2024-03-24T00:47:18.135579+0000","last_clean":"2024-03-24T00:47:18.135579+0000","last_became_active":"2024-03-24T00:47:17.142992+0000","last_became_peered":"2024-03-24T00:47:17.142992+0000","last_unstale":"2024-03-24T00:47:18.135579+0000","last_undegraded":"2024-03-24T00:47:18.135579+0000","last_fullsized":"2024-03-24T00:47:18.135579+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:53:45.283188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":14,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477271+0000","last_change":"2024-03-24T00:47:16.150192+0000","last_active":"2024-03-24T00:47:18.477271+0000","last_peered":"2024-03-24T00:47:18.477271+0000","last_clean":"2024-03-24T00:47:18.477271+0000","last_became_active":"2024-03-24T00:47:16.148847+0000","last_became_peered":"2024-03-24T00:47:16.148847+0000","last_unstale":"2024-03-24T00:47:18.477271+0000","last_undegraded":"2024-03-24T00:47:18.477271+0000","last_fullsized":"2024-03-24T00:47:18.477271+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:40:56.988956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.134209+0000","last_change":"2024-03-24T00:47:17.145149+0000","last_active":"2024-03-24T00:47:18.134209+0000","last_peered":"2024-03-24T00:47:18.134209+0000","last_clean":"2024-03-24T00:47:18.134209+0000","last_became_active":"2024-03-24T00:47:17.144806+0000","last_became_peered":"2024-03-24T00:47:17.144806+0000","last_unstale":"2024-03-24T00:47:18.134209+0000","last_undegraded":"2024-03-24T00:47:18.134209+0000","last_fullsized":"2024-03-24T00:47:18.134209+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:01:40.517054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.e","version":"104'2","reported_seq":16,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.478466+0000","last_change":"2024-03-24T00:47:16.147574+0000","last_active":"2024-03-24T00:47:18.478466+0000","last_peered":"2024-03-24T00:47:18.478466+0000","last_clean":"2024-03-24T00:47:18.478466+0000","last_became_active":"2024-03-24T00:47:16.147236+0000","last_became_peered":"2024-03-24T00:47:16.147236+0000","last_unstale":"2024-03-24T00:47:18.478466+0000","last_undegraded":"2024-03-24T00:47:18.478466+0000","last_fullsized":"2024-03-24T00:47:18.478466+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:19:32.345194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":329,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477566+0000","last_change":"2024-03-24T00:38:40.953871+0000","last_active":"2024-03-24T00:47:18.477566+0000","last_peered":"2024-03-24T00:47:18.477566+0000","last_clean":"2024-03-24T00:47:18.477566+0000","last_became_active":"2024-03-24T00:38:40.952926+0000","last_became_peered":"2024-03-24T00:38:40.952926+0000","last_unstale":"2024-03-24T00:47:18.477566+0000","last_undegraded":"2024-03-24T00:47:18.477566+0000","last_fullsized":"2024-03-24T00:47:18.477566+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_clean_scrub_stamp":"2024-03-24T00:38:39.722194+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:05.397257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":16,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574957,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33044,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338796,"statfs":{"total":96636764160,"available":96602927104,"internally_reserved":0,"allocated":1904640,"data_stored":1196890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574956,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32476,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339364,"statfs":{"total":96636764160,"available":96603508736,"internally_reserved":0,"allocated":1323008,"data_stored":608578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574956,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32456,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339384,"statfs":{"total":96636764160,"available":96603529216,"internally_reserved":0,"allocated":1302528,"data_stored":606362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574955,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32464,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339376,"statfs":{"total":96636764160,"available":96603521024,"internally_reserved":0,"allocated":1310720,"data_stored":606454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574955,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32444,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339396,"statfs":{"total":96636764160,"available":96603541504,"internally_reserved":0,"allocated":1294336,"data_stored":600252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574956,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32460,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339380,"statfs":{"total":96636764160,"available":96603525120,"internally_reserved":0,"allocated":1306624,"data_stored":606396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574956,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32448,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339392,"statfs":{"total":96636764160,"available":96603537408,"internally_reserved":0,"allocated":1294336,"data_stored":599704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574955,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33052,"kb_used_data":1868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338788,"statfs":{"total":96636764160,"available":96602918912,"internally_reserved":0,"allocated":1912832,"data_stored":1198432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:47:20.304 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:47:20.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.451+0000 7fd806353640 1 Processor -- start 2024-03-24T00:47:20.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.451+0000 7fd806353640 1 -- start start 2024-03-24T00:47:20.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.453+0000 7fd806353640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd8000d0ef0 0x7fd8000ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:20.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.453+0000 7fd806353640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8000cf640 con 0x7fd80015dec0 2024-03-24T00:47:20.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.453+0000 7fd806353640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8000cf080 con 0x7fd80015e2a0 2024-03-24T00:47:20.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.453+0000 7fd806353640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8000cee30 con 0x7fd8000d0ef0 2024-03-24T00:47:20.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.453+0000 7fd8048f0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd8000d0ef0 0x7fd8000ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:20.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.453+0000 7fd7fffff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd80015dec0 0x7fd8000d07e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48286/0 (socket says 172.21.15.67:48286) 2024-03-24T00:47:20.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.453+0000 7fd7fffff640 1 -- 172.21.15.67:0/2582547769 learned_addr learned my addr 172.21.15.67:0/2582547769 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:47:20.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.453+0000 7fd8048f0640 1 -- 172.21.15.67:0/2582547769 >> v1:172.21.15.72:6789/0 conn(0x7fd80015e2a0 legacy=0x7fd8000d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:20.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.453+0000 7fd8048f0640 1 -- 172.21.15.67:0/2582547769 >> v1:172.21.15.67:6789/0 conn(0x7fd80015dec0 legacy=0x7fd8000d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:20.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.453+0000 7fd7feffd640 1 -- 172.21.15.67:0/2582547769 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4068458100 0 0) 0x7fd7f0001110 con 0x7fd80015dec0 2024-03-24T00:47:20.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.453+0000 7fd8048f0640 1 -- 172.21.15.67:0/2582547769 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8000cf360 con 0x7fd8000d0ef0 2024-03-24T00:47:20.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.454+0000 7fd8048f0640 1 --2- 172.21.15.67:0/2582547769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8000d0ef0 0x7fd8000ce630 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7fd8000cefb0 tx=0x7fd7f802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f916ee213c7fcbd3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:20.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.454+0000 7fd7feffd640 1 -- 172.21.15.67:0/2582547769 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7f8039070 con 0x7fd8000d0ef0 2024-03-24T00:47:20.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.454+0000 7fd7feffd640 1 -- 172.21.15.67:0/2582547769 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7f802cc50 con 0x7fd8000d0ef0 2024-03-24T00:47:20.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.454+0000 7fd7feffd640 1 -- 172.21.15.67:0/2582547769 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7f8034440 con 0x7fd8000d0ef0 2024-03-24T00:47:20.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.455+0000 7fd806353640 1 -- 172.21.15.67:0/2582547769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8000d0ef0 msgr2=0x7fd8000ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:20.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.455+0000 7fd806353640 1 --2- 172.21.15.67:0/2582547769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8000d0ef0 0x7fd8000ce630 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7fd8000cefb0 tx=0x7fd7f802bea0 comp rx=0 tx=0).stop 2024-03-24T00:47:20.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.455+0000 7fd806353640 1 -- 172.21.15.67:0/2582547769 shutdown_connections 2024-03-24T00:47:20.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.455+0000 7fd806353640 1 --2- 172.21.15.67:0/2582547769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8000d0ef0 0x7fd8000ce630 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:20.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.455+0000 7fd806353640 1 -- 172.21.15.67:0/2582547769 >> 172.21.15.67:0/2582547769 conn(0x7fd80006c810 msgr2=0x7fd80006cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:20.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.455+0000 7fd806353640 1 -- 172.21.15.67:0/2582547769 shutdown_connections 2024-03-24T00:47:20.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.455+0000 7fd806353640 1 -- 172.21.15.67:0/2582547769 wait complete. 2024-03-24T00:47:20.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.456+0000 7fd806353640 1 Processor -- start 2024-03-24T00:47:20.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.457+0000 7fd806353640 1 -- start start 2024-03-24T00:47:20.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.458+0000 7fd806353640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8000d0ef0 0x7fd800159460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:20.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.458+0000 7fd806353640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd80015dec0 0x7fd8000b5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:20.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.458+0000 7fd806353640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd80015e2a0 0x7fd8000b5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:20.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.458+0000 7fd806353640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd800162a40 con 0x7fd8000d0ef0 2024-03-24T00:47:20.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.458+0000 7fd806353640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8001628c0 con 0x7fd80015dec0 2024-03-24T00:47:20.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.458+0000 7fd806353640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8000d2120 con 0x7fd80015e2a0 2024-03-24T00:47:20.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.458+0000 7fd7fffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8000d0ef0 0x7fd800159460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:20.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.458+0000 7fd7fffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8000d0ef0 0x7fd800159460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47252/0 (socket says 172.21.15.67:47252) 2024-03-24T00:47:20.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.458+0000 7fd7fffff640 1 -- 172.21.15.67:0/1648517326 learned_addr learned my addr 172.21.15.67:0/1648517326 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:20.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.458+0000 7fd8048f0640 1 --2- 172.21.15.67:0/1648517326 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd80015e2a0 0x7fd8000b5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:20.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.458+0000 7fd7ff7fe640 1 --2- 172.21.15.67:0/1648517326 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd80015dec0 0x7fd8000b5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:20.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.458+0000 7fd7fffff640 1 -- 172.21.15.67:0/1648517326 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd80015e2a0 msgr2=0x7fd8000b5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:20.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.458+0000 7fd7fffff640 1 --2- 172.21.15.67:0/1648517326 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd80015e2a0 0x7fd8000b5ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:20.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.458+0000 7fd7fffff640 1 -- 172.21.15.67:0/1648517326 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd80015dec0 msgr2=0x7fd8000b5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:20.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.458+0000 7fd7fffff640 1 --2- 172.21.15.67:0/1648517326 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd80015dec0 0x7fd8000b5990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:20.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.458+0000 7fd7fffff640 1 -- 172.21.15.67:0/1648517326 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8001e92a0 con 0x7fd8000d0ef0 2024-03-24T00:47:20.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.458+0000 7fd7ff7fe640 1 --2- 172.21.15.67:0/1648517326 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd80015dec0 0x7fd8000b5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:47:20.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.459+0000 7fd7fffff640 1 --2- 172.21.15.67:0/1648517326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8000d0ef0 0x7fd800159460 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7fd7f000c780 tx=0x7fd7f000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:20.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.459+0000 7fd7fcff9640 1 -- 172.21.15.67:0/1648517326 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7f0018070 con 0x7fd8000d0ef0 2024-03-24T00:47:20.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.459+0000 7fd7fcff9640 1 -- 172.21.15.67:0/1648517326 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7f00072d0 con 0x7fd8000d0ef0 2024-03-24T00:47:20.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.459+0000 7fd806353640 1 -- 172.21.15.67:0/1648517326 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8001e8fc0 con 0x7fd8000d0ef0 2024-03-24T00:47:20.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.459+0000 7fd7fcff9640 1 -- 172.21.15.67:0/1648517326 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7f00047e0 con 0x7fd8000d0ef0 2024-03-24T00:47:20.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.459+0000 7fd806353640 1 -- 172.21.15.67:0/1648517326 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8000cc3f0 con 0x7fd8000d0ef0 2024-03-24T00:47:20.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.460+0000 7fd806353640 1 -- 172.21.15.67:0/1648517326 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd80004f400 con 0x7fd8000d0ef0 2024-03-24T00:47:20.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.460+0000 7fd7fcff9640 1 -- 172.21.15.67:0/1648517326 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd7f001b070 con 0x7fd8000d0ef0 2024-03-24T00:47:20.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.460+0000 7fd7fcff9640 1 --2- 172.21.15.67:0/1648517326 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd7ec06bbd0 0x7fd7ec06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:20.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.461+0000 7fd7fcff9640 1 -- 172.21.15.67:0/1648517326 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd7f0089e20 con 0x7fd8000d0ef0 2024-03-24T00:47:20.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.461+0000 7fd7ff7fe640 1 --2- 172.21.15.67:0/1648517326 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd7ec06bbd0 0x7fd7ec06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:20.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.461+0000 7fd7ff7fe640 1 --2- 172.21.15.67:0/1648517326 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd7ec06bbd0 0x7fd7ec06e090 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fd8001599a0 tx=0x7fd7f4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:20.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.468+0000 7fd7fcff9640 1 -- 172.21.15.67:0/1648517326 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd8000cf640 con 0x7fd8000d0ef0 2024-03-24T00:47:20.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.645+0000 7fd806353640 1 -- 172.21.15.67:0/1648517326 --> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd8000d1aa0 con 0x7fd7ec06bbd0 2024-03-24T00:47:20.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.652+0000 7fd7fcff9640 1 -- 172.21.15.67:0/1648517326 <== mgr.4105 v2:172.21.15.72:6832/2768292274 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252485 (secure 0 0 0) 0x7fd8000d1aa0 con 0x7fd7ec06bbd0 2024-03-24T00:47:20.654 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:47:20.655 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-24T00:47:20.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.655+0000 7fd806353640 1 -- 172.21.15.67:0/1648517326 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd7ec06bbd0 msgr2=0x7fd7ec06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:20.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.655+0000 7fd806353640 1 --2- 172.21.15.67:0/1648517326 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd7ec06bbd0 0x7fd7ec06e090 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fd8001599a0 tx=0x7fd7f4009040 comp rx=0 tx=0).stop 2024-03-24T00:47:20.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.655+0000 7fd806353640 1 -- 172.21.15.67:0/1648517326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8000d0ef0 msgr2=0x7fd800159460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:20.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.655+0000 7fd806353640 1 --2- 172.21.15.67:0/1648517326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8000d0ef0 0x7fd800159460 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7fd7f000c780 tx=0x7fd7f000cc40 comp rx=0 tx=0).stop 2024-03-24T00:47:20.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.655+0000 7fd806353640 1 -- 172.21.15.67:0/1648517326 shutdown_connections 2024-03-24T00:47:20.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.655+0000 7fd806353640 1 --2- 172.21.15.67:0/1648517326 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd7ec06bbd0 0x7fd7ec06e090 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:20.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.655+0000 7fd806353640 1 --2- 172.21.15.67:0/1648517326 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd80015e2a0 0x7fd8000b5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:20.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.655+0000 7fd806353640 1 --2- 172.21.15.67:0/1648517326 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd80015dec0 0x7fd8000b5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:20.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.655+0000 7fd806353640 1 --2- 172.21.15.67:0/1648517326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8000d0ef0 0x7fd800159460 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:20.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.655+0000 7fd806353640 1 -- 172.21.15.67:0/1648517326 >> 172.21.15.67:0/1648517326 conn(0x7fd80006c810 msgr2=0x7fd8000d3170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:20.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.655+0000 7fd806353640 1 -- 172.21.15.67:0/1648517326 shutdown_connections 2024-03-24T00:47:20.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.655+0000 7fd806353640 1 -- 172.21.15.67:0/1648517326 wait complete. 2024-03-24T00:47:20.665 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":358,"stamp":"2024-03-24T00:47:19.443999+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":156,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":260844,"kb_used_data":11376,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754713876,"statfs":{"total":773094113280,"available":772827009024,"internally_reserved":0,"allocated":11649024,"data_stored":6023068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.323344"},"pg_stats":[{"pgid":"16.3f","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.131414+0000","last_change":"2024-03-24T00:47:17.147223+0000","last_active":"2024-03-24T00:47:18.131414+0000","last_peered":"2024-03-24T00:47:18.131414+0000","last_clean":"2024-03-24T00:47:18.131414+0000","last_became_active":"2024-03-24T00:47:17.146523+0000","last_became_peered":"2024-03-24T00:47:17.146523+0000","last_unstale":"2024-03-24T00:47:18.131414+0000","last_undegraded":"2024-03-24T00:47:18.131414+0000","last_fullsized":"2024-03-24T00:47:18.131414+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:05:38.608602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3e","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477708+0000","last_change":"2024-03-24T00:47:17.151017+0000","last_active":"2024-03-24T00:47:18.477708+0000","last_peered":"2024-03-24T00:47:18.477708+0000","last_clean":"2024-03-24T00:47:18.477708+0000","last_became_active":"2024-03-24T00:47:17.150743+0000","last_became_peered":"2024-03-24T00:47:17.150743+0000","last_unstale":"2024-03-24T00:47:18.477708+0000","last_undegraded":"2024-03-24T00:47:18.477708+0000","last_fullsized":"2024-03-24T00:47:18.477708+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:46:04.860603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.3d","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.131683+0000","last_change":"2024-03-24T00:47:17.140449+0000","last_active":"2024-03-24T00:47:18.131683+0000","last_peered":"2024-03-24T00:47:18.131683+0000","last_clean":"2024-03-24T00:47:18.131683+0000","last_became_active":"2024-03-24T00:47:17.140123+0000","last_became_peered":"2024-03-24T00:47:17.140123+0000","last_unstale":"2024-03-24T00:47:18.131683+0000","last_undegraded":"2024-03-24T00:47:18.131683+0000","last_fullsized":"2024-03-24T00:47:18.131683+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:24:51.948592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3c","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.478311+0000","last_change":"2024-03-24T00:47:17.155744+0000","last_active":"2024-03-24T00:47:18.478311+0000","last_peered":"2024-03-24T00:47:18.478311+0000","last_clean":"2024-03-24T00:47:18.478311+0000","last_became_active":"2024-03-24T00:47:17.155459+0000","last_became_peered":"2024-03-24T00:47:17.155459+0000","last_unstale":"2024-03-24T00:47:18.478311+0000","last_undegraded":"2024-03-24T00:47:18.478311+0000","last_fullsized":"2024-03-24T00:47:18.478311+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:45:21.442043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.3b","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477298+0000","last_change":"2024-03-24T00:47:17.151494+0000","last_active":"2024-03-24T00:47:18.477298+0000","last_peered":"2024-03-24T00:47:18.477298+0000","last_clean":"2024-03-24T00:47:18.477298+0000","last_became_active":"2024-03-24T00:47:17.150735+0000","last_became_peered":"2024-03-24T00:47:17.150735+0000","last_unstale":"2024-03-24T00:47:18.477298+0000","last_undegraded":"2024-03-24T00:47:18.477298+0000","last_fullsized":"2024-03-24T00:47:18.477298+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:15:29.178086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.3a","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477547+0000","last_change":"2024-03-24T00:47:17.161009+0000","last_active":"2024-03-24T00:47:18.477547+0000","last_peered":"2024-03-24T00:47:18.477547+0000","last_clean":"2024-03-24T00:47:18.477547+0000","last_became_active":"2024-03-24T00:47:17.160357+0000","last_became_peered":"2024-03-24T00:47:17.160357+0000","last_unstale":"2024-03-24T00:47:18.477547+0000","last_undegraded":"2024-03-24T00:47:18.477547+0000","last_fullsized":"2024-03-24T00:47:18.477547+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:13:47.628605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.39","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477198+0000","last_change":"2024-03-24T00:47:17.160775+0000","last_active":"2024-03-24T00:47:18.477198+0000","last_peered":"2024-03-24T00:47:18.477198+0000","last_clean":"2024-03-24T00:47:18.477198+0000","last_became_active":"2024-03-24T00:47:17.160297+0000","last_became_peered":"2024-03-24T00:47:17.160297+0000","last_unstale":"2024-03-24T00:47:18.477198+0000","last_undegraded":"2024-03-24T00:47:18.477198+0000","last_fullsized":"2024-03-24T00:47:18.477198+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:12:49.377452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.38","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.476798+0000","last_change":"2024-03-24T00:47:17.152081+0000","last_active":"2024-03-24T00:47:18.476798+0000","last_peered":"2024-03-24T00:47:18.476798+0000","last_clean":"2024-03-24T00:47:18.476798+0000","last_became_active":"2024-03-24T00:47:17.151796+0000","last_became_peered":"2024-03-24T00:47:17.151796+0000","last_unstale":"2024-03-24T00:47:18.476798+0000","last_undegraded":"2024-03-24T00:47:18.476798+0000","last_fullsized":"2024-03-24T00:47:18.476798+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:14:27.957978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.37","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477577+0000","last_change":"2024-03-24T00:47:17.157494+0000","last_active":"2024-03-24T00:47:18.477577+0000","last_peered":"2024-03-24T00:47:18.477577+0000","last_clean":"2024-03-24T00:47:18.477577+0000","last_became_active":"2024-03-24T00:47:17.157217+0000","last_became_peered":"2024-03-24T00:47:17.157217+0000","last_unstale":"2024-03-24T00:47:18.477577+0000","last_undegraded":"2024-03-24T00:47:18.477577+0000","last_fullsized":"2024-03-24T00:47:18.477577+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:00:00.156306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.36","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.478216+0000","last_change":"2024-03-24T00:47:17.149194+0000","last_active":"2024-03-24T00:47:18.478216+0000","last_peered":"2024-03-24T00:47:18.478216+0000","last_clean":"2024-03-24T00:47:18.478216+0000","last_became_active":"2024-03-24T00:47:17.148885+0000","last_became_peered":"2024-03-24T00:47:17.148885+0000","last_unstale":"2024-03-24T00:47:18.478216+0000","last_undegraded":"2024-03-24T00:47:18.478216+0000","last_fullsized":"2024-03-24T00:47:18.478216+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:37:11.302019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.35","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.135187+0000","last_change":"2024-03-24T00:47:17.153113+0000","last_active":"2024-03-24T00:47:18.135187+0000","last_peered":"2024-03-24T00:47:18.135187+0000","last_clean":"2024-03-24T00:47:18.135187+0000","last_became_active":"2024-03-24T00:47:17.152718+0000","last_became_peered":"2024-03-24T00:47:17.152718+0000","last_unstale":"2024-03-24T00:47:18.135187+0000","last_undegraded":"2024-03-24T00:47:18.135187+0000","last_fullsized":"2024-03-24T00:47:18.135187+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:27:13.774555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.34","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.134714+0000","last_change":"2024-03-24T00:47:17.158198+0000","last_active":"2024-03-24T00:47:18.134714+0000","last_peered":"2024-03-24T00:47:18.134714+0000","last_clean":"2024-03-24T00:47:18.134714+0000","last_became_active":"2024-03-24T00:47:17.156616+0000","last_became_peered":"2024-03-24T00:47:17.156616+0000","last_unstale":"2024-03-24T00:47:18.134714+0000","last_undegraded":"2024-03-24T00:47:18.134714+0000","last_fullsized":"2024-03-24T00:47:18.134714+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:37:46.124771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.33","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477357+0000","last_change":"2024-03-24T00:47:17.147949+0000","last_active":"2024-03-24T00:47:18.477357+0000","last_peered":"2024-03-24T00:47:18.477357+0000","last_clean":"2024-03-24T00:47:18.477357+0000","last_became_active":"2024-03-24T00:47:17.147656+0000","last_became_peered":"2024-03-24T00:47:17.147656+0000","last_unstale":"2024-03-24T00:47:18.477357+0000","last_undegraded":"2024-03-24T00:47:18.477357+0000","last_fullsized":"2024-03-24T00:47:18.477357+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:52:34.023840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.32","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477547+0000","last_change":"2024-03-24T00:47:17.147768+0000","last_active":"2024-03-24T00:47:18.477547+0000","last_peered":"2024-03-24T00:47:18.477547+0000","last_clean":"2024-03-24T00:47:18.477547+0000","last_became_active":"2024-03-24T00:47:17.147516+0000","last_became_peered":"2024-03-24T00:47:17.147516+0000","last_unstale":"2024-03-24T00:47:18.477547+0000","last_undegraded":"2024-03-24T00:47:18.477547+0000","last_fullsized":"2024-03-24T00:47:18.477547+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:44:57.547374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.31","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477390+0000","last_change":"2024-03-24T00:47:17.154501+0000","last_active":"2024-03-24T00:47:18.477390+0000","last_peered":"2024-03-24T00:47:18.477390+0000","last_clean":"2024-03-24T00:47:18.477390+0000","last_became_active":"2024-03-24T00:47:17.154232+0000","last_became_peered":"2024-03-24T00:47:17.154232+0000","last_unstale":"2024-03-24T00:47:18.477390+0000","last_undegraded":"2024-03-24T00:47:18.477390+0000","last_fullsized":"2024-03-24T00:47:18.477390+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:27:13.774555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.30","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477498+0000","last_change":"2024-03-24T00:47:17.149209+0000","last_active":"2024-03-24T00:47:18.477498+0000","last_peered":"2024-03-24T00:47:18.477498+0000","last_clean":"2024-03-24T00:47:18.477498+0000","last_became_active":"2024-03-24T00:47:17.148887+0000","last_became_peered":"2024-03-24T00:47:17.148887+0000","last_unstale":"2024-03-24T00:47:18.477498+0000","last_undegraded":"2024-03-24T00:47:18.477498+0000","last_fullsized":"2024-03-24T00:47:18.477498+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:08:14.585720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.2f","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.131555+0000","last_change":"2024-03-24T00:47:17.140341+0000","last_active":"2024-03-24T00:47:18.131555+0000","last_peered":"2024-03-24T00:47:18.131555+0000","last_clean":"2024-03-24T00:47:18.131555+0000","last_became_active":"2024-03-24T00:47:17.140058+0000","last_became_peered":"2024-03-24T00:47:17.140058+0000","last_unstale":"2024-03-24T00:47:18.131555+0000","last_undegraded":"2024-03-24T00:47:18.131555+0000","last_fullsized":"2024-03-24T00:47:18.131555+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:35:51.675202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2e","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477876+0000","last_change":"2024-03-24T00:47:17.150236+0000","last_active":"2024-03-24T00:47:18.477876+0000","last_peered":"2024-03-24T00:47:18.477876+0000","last_clean":"2024-03-24T00:47:18.477876+0000","last_became_active":"2024-03-24T00:47:17.149800+0000","last_became_peered":"2024-03-24T00:47:17.149800+0000","last_unstale":"2024-03-24T00:47:18.477876+0000","last_undegraded":"2024-03-24T00:47:18.477876+0000","last_fullsized":"2024-03-24T00:47:18.477876+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:37:46.124771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.2d","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.131740+0000","last_change":"2024-03-24T00:47:17.155358+0000","last_active":"2024-03-24T00:47:18.131740+0000","last_peered":"2024-03-24T00:47:18.131740+0000","last_clean":"2024-03-24T00:47:18.131740+0000","last_became_active":"2024-03-24T00:47:17.154156+0000","last_became_peered":"2024-03-24T00:47:17.154156+0000","last_unstale":"2024-03-24T00:47:18.131740+0000","last_undegraded":"2024-03-24T00:47:18.131740+0000","last_fullsized":"2024-03-24T00:47:18.131740+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:08:48.681541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2c","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.134922+0000","last_change":"2024-03-24T00:47:17.152539+0000","last_active":"2024-03-24T00:47:18.134922+0000","last_peered":"2024-03-24T00:47:18.134922+0000","last_clean":"2024-03-24T00:47:18.134922+0000","last_became_active":"2024-03-24T00:47:17.152273+0000","last_became_peered":"2024-03-24T00:47:17.152273+0000","last_unstale":"2024-03-24T00:47:18.134922+0000","last_undegraded":"2024-03-24T00:47:18.134922+0000","last_fullsized":"2024-03-24T00:47:18.134922+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:50:00.059710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2b","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477513+0000","last_change":"2024-03-24T00:47:17.148950+0000","last_active":"2024-03-24T00:47:18.477513+0000","last_peered":"2024-03-24T00:47:18.477513+0000","last_clean":"2024-03-24T00:47:18.477513+0000","last_became_active":"2024-03-24T00:47:17.148661+0000","last_became_peered":"2024-03-24T00:47:17.148661+0000","last_unstale":"2024-03-24T00:47:18.477513+0000","last_undegraded":"2024-03-24T00:47:18.477513+0000","last_fullsized":"2024-03-24T00:47:18.477513+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:41:54.376692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2a","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477113+0000","last_change":"2024-03-24T00:47:17.152120+0000","last_active":"2024-03-24T00:47:18.477113+0000","last_peered":"2024-03-24T00:47:18.477113+0000","last_clean":"2024-03-24T00:47:18.477113+0000","last_became_active":"2024-03-24T00:47:17.151841+0000","last_became_peered":"2024-03-24T00:47:17.151841+0000","last_unstale":"2024-03-24T00:47:18.477113+0000","last_undegraded":"2024-03-24T00:47:18.477113+0000","last_fullsized":"2024-03-24T00:47:18.477113+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:13:45.186309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.29","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.131457+0000","last_change":"2024-03-24T00:47:17.145741+0000","last_active":"2024-03-24T00:47:18.131457+0000","last_peered":"2024-03-24T00:47:18.131457+0000","last_clean":"2024-03-24T00:47:18.131457+0000","last_became_active":"2024-03-24T00:47:17.145434+0000","last_became_peered":"2024-03-24T00:47:17.145434+0000","last_unstale":"2024-03-24T00:47:18.131457+0000","last_undegraded":"2024-03-24T00:47:18.131457+0000","last_fullsized":"2024-03-24T00:47:18.131457+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:30:56.194752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.28","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477468+0000","last_change":"2024-03-24T00:47:17.149650+0000","last_active":"2024-03-24T00:47:18.477468+0000","last_peered":"2024-03-24T00:47:18.477468+0000","last_clean":"2024-03-24T00:47:18.477468+0000","last_became_active":"2024-03-24T00:47:17.149343+0000","last_became_peered":"2024-03-24T00:47:17.149343+0000","last_unstale":"2024-03-24T00:47:18.477468+0000","last_undegraded":"2024-03-24T00:47:18.477468+0000","last_fullsized":"2024-03-24T00:47:18.477468+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:09:34.392689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.27","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.134893+0000","last_change":"2024-03-24T00:47:17.158496+0000","last_active":"2024-03-24T00:47:18.134893+0000","last_peered":"2024-03-24T00:47:18.134893+0000","last_clean":"2024-03-24T00:47:18.134893+0000","last_became_active":"2024-03-24T00:47:17.156810+0000","last_became_peered":"2024-03-24T00:47:17.156810+0000","last_unstale":"2024-03-24T00:47:18.134893+0000","last_undegraded":"2024-03-24T00:47:18.134893+0000","last_fullsized":"2024-03-24T00:47:18.134893+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:40:03.538155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.26","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.135225+0000","last_change":"2024-03-24T00:47:17.149097+0000","last_active":"2024-03-24T00:47:18.135225+0000","last_peered":"2024-03-24T00:47:18.135225+0000","last_clean":"2024-03-24T00:47:18.135225+0000","last_became_active":"2024-03-24T00:47:17.148813+0000","last_became_peered":"2024-03-24T00:47:17.148813+0000","last_unstale":"2024-03-24T00:47:18.135225+0000","last_undegraded":"2024-03-24T00:47:18.135225+0000","last_fullsized":"2024-03-24T00:47:18.135225+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:27:55.664267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.25","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.135397+0000","last_change":"2024-03-24T00:47:17.145605+0000","last_active":"2024-03-24T00:47:18.135397+0000","last_peered":"2024-03-24T00:47:18.135397+0000","last_clean":"2024-03-24T00:47:18.135397+0000","last_became_active":"2024-03-24T00:47:17.144961+0000","last_became_peered":"2024-03-24T00:47:17.144961+0000","last_unstale":"2024-03-24T00:47:18.135397+0000","last_undegraded":"2024-03-24T00:47:18.135397+0000","last_fullsized":"2024-03-24T00:47:18.135397+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:08:14.585720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.24","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477677+0000","last_change":"2024-03-24T00:47:17.153966+0000","last_active":"2024-03-24T00:47:18.477677+0000","last_peered":"2024-03-24T00:47:18.477677+0000","last_clean":"2024-03-24T00:47:18.477677+0000","last_became_active":"2024-03-24T00:47:17.153683+0000","last_became_peered":"2024-03-24T00:47:17.153683+0000","last_unstale":"2024-03-24T00:47:18.477677+0000","last_undegraded":"2024-03-24T00:47:18.477677+0000","last_fullsized":"2024-03-24T00:47:18.477677+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:53:00.441548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.23","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477660+0000","last_change":"2024-03-24T00:47:17.150198+0000","last_active":"2024-03-24T00:47:18.477660+0000","last_peered":"2024-03-24T00:47:18.477660+0000","last_clean":"2024-03-24T00:47:18.477660+0000","last_became_active":"2024-03-24T00:47:17.149903+0000","last_became_peered":"2024-03-24T00:47:17.149903+0000","last_unstale":"2024-03-24T00:47:18.477660+0000","last_undegraded":"2024-03-24T00:47:18.477660+0000","last_fullsized":"2024-03-24T00:47:18.477660+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:35:33.604622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.22","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.134432+0000","last_change":"2024-03-24T00:47:17.158820+0000","last_active":"2024-03-24T00:47:18.134432+0000","last_peered":"2024-03-24T00:47:18.134432+0000","last_clean":"2024-03-24T00:47:18.134432+0000","last_became_active":"2024-03-24T00:47:17.157921+0000","last_became_peered":"2024-03-24T00:47:17.157921+0000","last_unstale":"2024-03-24T00:47:18.134432+0000","last_undegraded":"2024-03-24T00:47:18.134432+0000","last_fullsized":"2024-03-24T00:47:18.134432+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:43:46.684428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.21","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477443+0000","last_change":"2024-03-24T00:47:17.159992+0000","last_active":"2024-03-24T00:47:18.477443+0000","last_peered":"2024-03-24T00:47:18.477443+0000","last_clean":"2024-03-24T00:47:18.477443+0000","last_became_active":"2024-03-24T00:47:17.159695+0000","last_became_peered":"2024-03-24T00:47:17.159695+0000","last_unstale":"2024-03-24T00:47:18.477443+0000","last_undegraded":"2024-03-24T00:47:18.477443+0000","last_fullsized":"2024-03-24T00:47:18.477443+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:49:59.144522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.20","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.476949+0000","last_change":"2024-03-24T00:47:17.154783+0000","last_active":"2024-03-24T00:47:18.476949+0000","last_peered":"2024-03-24T00:47:18.476949+0000","last_clean":"2024-03-24T00:47:18.476949+0000","last_became_active":"2024-03-24T00:47:17.154490+0000","last_became_peered":"2024-03-24T00:47:17.154490+0000","last_unstale":"2024-03-24T00:47:18.476949+0000","last_undegraded":"2024-03-24T00:47:18.476949+0000","last_fullsized":"2024-03-24T00:47:18.476949+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:46:36.771424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.131360+0000","last_change":"2024-03-24T00:47:17.155763+0000","last_active":"2024-03-24T00:47:18.131360+0000","last_peered":"2024-03-24T00:47:18.131360+0000","last_clean":"2024-03-24T00:47:18.131360+0000","last_became_active":"2024-03-24T00:47:17.155486+0000","last_became_peered":"2024-03-24T00:47:17.155486+0000","last_unstale":"2024-03-24T00:47:18.131360+0000","last_undegraded":"2024-03-24T00:47:18.131360+0000","last_fullsized":"2024-03-24T00:47:18.131360+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:19:44.866507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"104'4","reported_seq":18,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477978+0000","last_change":"2024-03-24T00:47:16.149652+0000","last_active":"2024-03-24T00:47:18.477978+0000","last_peered":"2024-03-24T00:47:18.477978+0000","last_clean":"2024-03-24T00:47:18.477978+0000","last_became_active":"2024-03-24T00:47:16.149370+0000","last_became_peered":"2024-03-24T00:47:16.149370+0000","last_unstale":"2024-03-24T00:47:18.477978+0000","last_undegraded":"2024-03-24T00:47:18.477978+0000","last_fullsized":"2024-03-24T00:47:18.477978+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:01:39.510298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.476960+0000","last_change":"2024-03-24T00:47:17.159058+0000","last_active":"2024-03-24T00:47:18.476960+0000","last_peered":"2024-03-24T00:47:18.476960+0000","last_clean":"2024-03-24T00:47:18.476960+0000","last_became_active":"2024-03-24T00:47:17.158779+0000","last_became_peered":"2024-03-24T00:47:17.158779+0000","last_unstale":"2024-03-24T00:47:18.476960+0000","last_undegraded":"2024-03-24T00:47:18.476960+0000","last_fullsized":"2024-03-24T00:47:18.476960+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:23:31.948889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1e","version":"104'1","reported_seq":15,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.478445+0000","last_change":"2024-03-24T00:47:16.145718+0000","last_active":"2024-03-24T00:47:18.478445+0000","last_peered":"2024-03-24T00:47:18.478445+0000","last_clean":"2024-03-24T00:47:18.478445+0000","last_became_active":"2024-03-24T00:47:16.145418+0000","last_became_peered":"2024-03-24T00:47:16.145418+0000","last_unstale":"2024-03-24T00:47:18.478445+0000","last_undegraded":"2024-03-24T00:47:18.478445+0000","last_fullsized":"2024-03-24T00:47:18.478445+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:32:00.796419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.134431+0000","last_change":"2024-03-24T00:47:17.159926+0000","last_active":"2024-03-24T00:47:18.134431+0000","last_peered":"2024-03-24T00:47:18.134431+0000","last_clean":"2024-03-24T00:47:18.134431+0000","last_became_active":"2024-03-24T00:47:17.159640+0000","last_became_peered":"2024-03-24T00:47:17.159640+0000","last_unstale":"2024-03-24T00:47:18.134431+0000","last_undegraded":"2024-03-24T00:47:18.134431+0000","last_fullsized":"2024-03-24T00:47:18.134431+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:46:04.860603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"104'2","reported_seq":16,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.135136+0000","last_change":"2024-03-24T00:47:16.146156+0000","last_active":"2024-03-24T00:47:18.135136+0000","last_peered":"2024-03-24T00:47:18.135136+0000","last_clean":"2024-03-24T00:47:18.135136+0000","last_became_active":"2024-03-24T00:47:16.145793+0000","last_became_peered":"2024-03-24T00:47:16.145793+0000","last_unstale":"2024-03-24T00:47:18.135136+0000","last_undegraded":"2024-03-24T00:47:18.135136+0000","last_fullsized":"2024-03-24T00:47:18.135136+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:35:32.597867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.134743+0000","last_change":"2024-03-24T00:47:17.153905+0000","last_active":"2024-03-24T00:47:18.134743+0000","last_peered":"2024-03-24T00:47:18.134743+0000","last_clean":"2024-03-24T00:47:18.134743+0000","last_became_active":"2024-03-24T00:47:17.153630+0000","last_became_peered":"2024-03-24T00:47:17.153630+0000","last_unstale":"2024-03-24T00:47:18.134743+0000","last_undegraded":"2024-03-24T00:47:18.134743+0000","last_fullsized":"2024-03-24T00:47:18.134743+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:13:58.356201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1c","version":"104'2","reported_seq":16,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477718+0000","last_change":"2024-03-24T00:47:16.150021+0000","last_active":"2024-03-24T00:47:18.477718+0000","last_peered":"2024-03-24T00:47:18.477718+0000","last_clean":"2024-03-24T00:47:18.477718+0000","last_became_active":"2024-03-24T00:47:16.149706+0000","last_became_peered":"2024-03-24T00:47:16.149706+0000","last_unstale":"2024-03-24T00:47:18.477718+0000","last_undegraded":"2024-03-24T00:47:18.477718+0000","last_fullsized":"2024-03-24T00:47:18.477718+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:37:45.118015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.478317+0000","last_change":"2024-03-24T00:47:17.150669+0000","last_active":"2024-03-24T00:47:18.478317+0000","last_peered":"2024-03-24T00:47:18.478317+0000","last_clean":"2024-03-24T00:47:18.478317+0000","last_became_active":"2024-03-24T00:47:17.150354+0000","last_became_peered":"2024-03-24T00:47:17.150354+0000","last_unstale":"2024-03-24T00:47:18.478317+0000","last_undegraded":"2024-03-24T00:47:18.478317+0000","last_fullsized":"2024-03-24T00:47:18.478317+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:20:13.126290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"104'1","reported_seq":15,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.478046+0000","last_change":"2024-03-24T00:47:16.149722+0000","last_active":"2024-03-24T00:47:18.478046+0000","last_peered":"2024-03-24T00:47:18.478046+0000","last_clean":"2024-03-24T00:47:18.478046+0000","last_became_active":"2024-03-24T00:47:16.148764+0000","last_became_peered":"2024-03-24T00:47:16.148764+0000","last_unstale":"2024-03-24T00:47:18.478046+0000","last_undegraded":"2024-03-24T00:47:18.478046+0000","last_fullsized":"2024-03-24T00:47:18.478046+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:37:10.295263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.131823+0000","last_change":"2024-03-24T00:47:17.154855+0000","last_active":"2024-03-24T00:47:18.131823+0000","last_peered":"2024-03-24T00:47:18.131823+0000","last_clean":"2024-03-24T00:47:18.131823+0000","last_became_active":"2024-03-24T00:47:17.153756+0000","last_became_peered":"2024-03-24T00:47:17.153756+0000","last_unstale":"2024-03-24T00:47:18.131823+0000","last_undegraded":"2024-03-24T00:47:18.131823+0000","last_fullsized":"2024-03-24T00:47:18.131823+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:56:42.633611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"104'1","reported_seq":15,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.134103+0000","last_change":"2024-03-24T00:47:16.147625+0000","last_active":"2024-03-24T00:47:18.134103+0000","last_peered":"2024-03-24T00:47:18.134103+0000","last_clean":"2024-03-24T00:47:18.134103+0000","last_became_active":"2024-03-24T00:47:16.147334+0000","last_became_peered":"2024-03-24T00:47:16.147334+0000","last_unstale":"2024-03-24T00:47:18.134103+0000","last_undegraded":"2024-03-24T00:47:18.134103+0000","last_fullsized":"2024-03-24T00:47:18.134103+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:13:44.179554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477954+0000","last_change":"2024-03-24T00:47:17.156056+0000","last_active":"2024-03-24T00:47:18.477954+0000","last_peered":"2024-03-24T00:47:18.477954+0000","last_clean":"2024-03-24T00:47:18.477954+0000","last_became_active":"2024-03-24T00:47:17.155771+0000","last_became_peered":"2024-03-24T00:47:17.155771+0000","last_unstale":"2024-03-24T00:47:18.477954+0000","last_undegraded":"2024-03-24T00:47:18.477954+0000","last_fullsized":"2024-03-24T00:47:18.477954+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:40:03.538155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":14,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.134512+0000","last_change":"2024-03-24T00:47:16.151295+0000","last_active":"2024-03-24T00:47:18.134512+0000","last_peered":"2024-03-24T00:47:18.134512+0000","last_clean":"2024-03-24T00:47:18.134512+0000","last_became_active":"2024-03-24T00:47:16.150902+0000","last_became_peered":"2024-03-24T00:47:16.150902+0000","last_unstale":"2024-03-24T00:47:18.134512+0000","last_undegraded":"2024-03-24T00:47:18.134512+0000","last_fullsized":"2024-03-24T00:47:18.134512+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:43:04.319575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477877+0000","last_change":"2024-03-24T00:47:17.143976+0000","last_active":"2024-03-24T00:47:18.477877+0000","last_peered":"2024-03-24T00:47:18.477877+0000","last_clean":"2024-03-24T00:47:18.477877+0000","last_became_active":"2024-03-24T00:47:17.143638+0000","last_became_peered":"2024-03-24T00:47:17.143638+0000","last_unstale":"2024-03-24T00:47:18.477877+0000","last_undegraded":"2024-03-24T00:47:18.477877+0000","last_fullsized":"2024-03-24T00:47:18.477877+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:41:11.873811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":14,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.478060+0000","last_change":"2024-03-24T00:47:16.150403+0000","last_active":"2024-03-24T00:47:18.478060+0000","last_peered":"2024-03-24T00:47:18.478060+0000","last_clean":"2024-03-24T00:47:18.478060+0000","last_became_active":"2024-03-24T00:47:16.150114+0000","last_became_peered":"2024-03-24T00:47:16.150114+0000","last_unstale":"2024-03-24T00:47:18.478060+0000","last_undegraded":"2024-03-24T00:47:18.478060+0000","last_fullsized":"2024-03-24T00:47:18.478060+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:05:30.554415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.131225+0000","last_change":"2024-03-24T00:47:17.156494+0000","last_active":"2024-03-24T00:47:18.131225+0000","last_peered":"2024-03-24T00:47:18.131225+0000","last_clean":"2024-03-24T00:47:18.131225+0000","last_became_active":"2024-03-24T00:47:17.156197+0000","last_became_peered":"2024-03-24T00:47:17.156197+0000","last_unstale":"2024-03-24T00:47:18.131225+0000","last_undegraded":"2024-03-24T00:47:18.131225+0000","last_fullsized":"2024-03-24T00:47:18.131225+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:31:56.106499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":14,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.135011+0000","last_change":"2024-03-24T00:47:16.147099+0000","last_active":"2024-03-24T00:47:18.135011+0000","last_peered":"2024-03-24T00:47:18.135011+0000","last_clean":"2024-03-24T00:47:18.135011+0000","last_became_active":"2024-03-24T00:47:16.145707+0000","last_became_peered":"2024-03-24T00:47:16.145707+0000","last_unstale":"2024-03-24T00:47:18.135011+0000","last_undegraded":"2024-03-24T00:47:18.135011+0000","last_fullsized":"2024-03-24T00:47:18.135011+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:20:09.491226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477066+0000","last_change":"2024-03-24T00:47:17.149301+0000","last_active":"2024-03-24T00:47:18.477066+0000","last_peered":"2024-03-24T00:47:18.477066+0000","last_clean":"2024-03-24T00:47:18.477066+0000","last_became_active":"2024-03-24T00:47:17.148972+0000","last_became_peered":"2024-03-24T00:47:17.148972+0000","last_unstale":"2024-03-24T00:47:18.477066+0000","last_undegraded":"2024-03-24T00:47:18.477066+0000","last_fullsized":"2024-03-24T00:47:18.477066+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:06:17.216361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":14,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.131300+0000","last_change":"2024-03-24T00:47:16.146221+0000","last_active":"2024-03-24T00:47:18.131300+0000","last_peered":"2024-03-24T00:47:18.131300+0000","last_clean":"2024-03-24T00:47:18.131300+0000","last_became_active":"2024-03-24T00:47:16.145948+0000","last_became_peered":"2024-03-24T00:47:16.145948+0000","last_unstale":"2024-03-24T00:47:18.131300+0000","last_undegraded":"2024-03-24T00:47:18.131300+0000","last_fullsized":"2024-03-24T00:47:18.131300+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:44:20.360231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477207+0000","last_change":"2024-03-24T00:47:17.154820+0000","last_active":"2024-03-24T00:47:18.477207+0000","last_peered":"2024-03-24T00:47:18.477207+0000","last_clean":"2024-03-24T00:47:18.477207+0000","last_became_active":"2024-03-24T00:47:17.154524+0000","last_became_peered":"2024-03-24T00:47:17.154524+0000","last_unstale":"2024-03-24T00:47:18.477207+0000","last_undegraded":"2024-03-24T00:47:18.477207+0000","last_fullsized":"2024-03-24T00:47:18.477207+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:20:10.497981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.15","version":"104'3","reported_seq":17,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477497+0000","last_change":"2024-03-24T00:47:16.146600+0000","last_active":"2024-03-24T00:47:18.477497+0000","last_peered":"2024-03-24T00:47:18.477497+0000","last_clean":"2024-03-24T00:47:18.477497+0000","last_became_active":"2024-03-24T00:47:16.146279+0000","last_became_peered":"2024-03-24T00:47:16.146279+0000","last_unstale":"2024-03-24T00:47:18.477497+0000","last_undegraded":"2024-03-24T00:47:18.477497+0000","last_fullsized":"2024-03-24T00:47:18.477497+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:48:22.549360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477592+0000","last_change":"2024-03-24T00:47:17.151264+0000","last_active":"2024-03-24T00:47:18.477592+0000","last_peered":"2024-03-24T00:47:18.477592+0000","last_clean":"2024-03-24T00:47:18.477592+0000","last_became_active":"2024-03-24T00:47:17.150449+0000","last_became_peered":"2024-03-24T00:47:17.150449+0000","last_unstale":"2024-03-24T00:47:18.477592+0000","last_undegraded":"2024-03-24T00:47:18.477592+0000","last_fullsized":"2024-03-24T00:47:18.477592+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:48:23.556116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.14","version":"104'6","reported_seq":20,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.478075+0000","last_change":"2024-03-24T00:47:16.152236+0000","last_active":"2024-03-24T00:47:18.478075+0000","last_peered":"2024-03-24T00:47:18.478075+0000","last_clean":"2024-03-24T00:47:18.478075+0000","last_became_active":"2024-03-24T00:47:16.152019+0000","last_became_peered":"2024-03-24T00:47:16.152019+0000","last_unstale":"2024-03-24T00:47:18.478075+0000","last_undegraded":"2024-03-24T00:47:18.478075+0000","last_fullsized":"2024-03-24T00:47:18.478075+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:52:59.339383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.135375+0000","last_change":"2024-03-24T00:47:17.136903+0000","last_active":"2024-03-24T00:47:18.135375+0000","last_peered":"2024-03-24T00:47:18.135375+0000","last_clean":"2024-03-24T00:47:18.135375+0000","last_became_active":"2024-03-24T00:47:17.136568+0000","last_became_peered":"2024-03-24T00:47:17.136568+0000","last_unstale":"2024-03-24T00:47:18.135375+0000","last_undegraded":"2024-03-24T00:47:18.135375+0000","last_fullsized":"2024-03-24T00:47:18.135375+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:44:57.547374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.13","version":"104'2","reported_seq":16,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477771+0000","last_change":"2024-03-24T00:47:16.149612+0000","last_active":"2024-03-24T00:47:18.477771+0000","last_peered":"2024-03-24T00:47:18.477771+0000","last_clean":"2024-03-24T00:47:18.477771+0000","last_became_active":"2024-03-24T00:47:16.149303+0000","last_became_peered":"2024-03-24T00:47:16.149303+0000","last_unstale":"2024-03-24T00:47:18.477771+0000","last_undegraded":"2024-03-24T00:47:18.477771+0000","last_fullsized":"2024-03-24T00:47:18.477771+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:49:59.052954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477777+0000","last_change":"2024-03-24T00:47:17.156700+0000","last_active":"2024-03-24T00:47:18.477777+0000","last_peered":"2024-03-24T00:47:18.477777+0000","last_clean":"2024-03-24T00:47:18.477777+0000","last_became_active":"2024-03-24T00:47:17.156438+0000","last_became_peered":"2024-03-24T00:47:17.156438+0000","last_unstale":"2024-03-24T00:47:18.477777+0000","last_undegraded":"2024-03-24T00:47:18.477777+0000","last_fullsized":"2024-03-24T00:47:18.477777+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:43:46.684428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":14,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477138+0000","last_change":"2024-03-24T00:47:16.150026+0000","last_active":"2024-03-24T00:47:18.477138+0000","last_peered":"2024-03-24T00:47:18.477138+0000","last_clean":"2024-03-24T00:47:18.477138+0000","last_became_active":"2024-03-24T00:47:16.149739+0000","last_became_peered":"2024-03-24T00:47:16.149739+0000","last_unstale":"2024-03-24T00:47:18.477138+0000","last_undegraded":"2024-03-24T00:47:18.477138+0000","last_fullsized":"2024-03-24T00:47:18.477138+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:36:40.034011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.478408+0000","last_change":"2024-03-24T00:47:17.154037+0000","last_active":"2024-03-24T00:47:18.478408+0000","last_peered":"2024-03-24T00:47:18.478408+0000","last_clean":"2024-03-24T00:47:18.478408+0000","last_became_active":"2024-03-24T00:47:17.153759+0000","last_became_peered":"2024-03-24T00:47:17.153759+0000","last_unstale":"2024-03-24T00:47:18.478408+0000","last_undegraded":"2024-03-24T00:47:18.478408+0000","last_fullsized":"2024-03-24T00:47:18.478408+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:24:33.435385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":14,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.476839+0000","last_change":"2024-03-24T00:47:16.150357+0000","last_active":"2024-03-24T00:47:18.476839+0000","last_peered":"2024-03-24T00:47:18.476839+0000","last_clean":"2024-03-24T00:47:18.476839+0000","last_became_active":"2024-03-24T00:47:16.150061+0000","last_became_peered":"2024-03-24T00:47:16.150061+0000","last_unstale":"2024-03-24T00:47:18.476839+0000","last_undegraded":"2024-03-24T00:47:18.476839+0000","last_fullsized":"2024-03-24T00:47:18.476839+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:48:13.364368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.131709+0000","last_change":"2024-03-24T00:47:17.139195+0000","last_active":"2024-03-24T00:47:18.131709+0000","last_peered":"2024-03-24T00:47:18.131709+0000","last_clean":"2024-03-24T00:47:18.131709+0000","last_became_active":"2024-03-24T00:47:17.138927+0000","last_became_peered":"2024-03-24T00:47:17.138927+0000","last_unstale":"2024-03-24T00:47:18.131709+0000","last_undegraded":"2024-03-24T00:47:18.131709+0000","last_fullsized":"2024-03-24T00:47:18.131709+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:23:08.092507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.10","version":"104'8","reported_seq":22,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.134434+0000","last_change":"2024-03-24T00:47:16.146521+0000","last_active":"2024-03-24T00:47:18.134434+0000","last_peered":"2024-03-24T00:47:18.134434+0000","last_clean":"2024-03-24T00:47:18.134434+0000","last_became_active":"2024-03-24T00:47:16.145984+0000","last_became_peered":"2024-03-24T00:47:16.145984+0000","last_unstale":"2024-03-24T00:47:18.134434+0000","last_undegraded":"2024-03-24T00:47:18.134434+0000","last_fullsized":"2024-03-24T00:47:18.134434+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:34:57.983478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477087+0000","last_change":"2024-03-24T00:47:17.147660+0000","last_active":"2024-03-24T00:47:18.477087+0000","last_peered":"2024-03-24T00:47:18.477087+0000","last_clean":"2024-03-24T00:47:18.477087+0000","last_became_active":"2024-03-24T00:47:17.147391+0000","last_became_peered":"2024-03-24T00:47:17.147391+0000","last_unstale":"2024-03-24T00:47:18.477087+0000","last_undegraded":"2024-03-24T00:47:18.477087+0000","last_fullsized":"2024-03-24T00:47:18.477087+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:43:05.326331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.f","version":"104'1","reported_seq":15,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477693+0000","last_change":"2024-03-24T00:47:16.149894+0000","last_active":"2024-03-24T00:47:18.477693+0000","last_peered":"2024-03-24T00:47:18.477693+0000","last_clean":"2024-03-24T00:47:18.477693+0000","last_became_active":"2024-03-24T00:47:16.149616+0000","last_became_peered":"2024-03-24T00:47:16.149616+0000","last_unstale":"2024-03-24T00:47:18.477693+0000","last_undegraded":"2024-03-24T00:47:18.477693+0000","last_fullsized":"2024-03-24T00:47:18.477693+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:06:24.193452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477503+0000","last_change":"2024-03-24T00:47:17.158745+0000","last_active":"2024-03-24T00:47:18.477503+0000","last_peered":"2024-03-24T00:47:18.477503+0000","last_clean":"2024-03-24T00:47:18.477503+0000","last_became_active":"2024-03-24T00:47:17.158474+0000","last_became_peered":"2024-03-24T00:47:17.158474+0000","last_unstale":"2024-03-24T00:47:18.477503+0000","last_undegraded":"2024-03-24T00:47:18.477503+0000","last_fullsized":"2024-03-24T00:47:18.477503+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:18:15.311145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"104'1","reported_seq":15,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477886+0000","last_change":"2024-03-24T00:47:16.148297+0000","last_active":"2024-03-24T00:47:18.477886+0000","last_peered":"2024-03-24T00:47:18.477886+0000","last_clean":"2024-03-24T00:47:18.477886+0000","last_became_active":"2024-03-24T00:47:16.148023+0000","last_became_peered":"2024-03-24T00:47:16.148023+0000","last_unstale":"2024-03-24T00:47:18.477886+0000","last_undegraded":"2024-03-24T00:47:18.477886+0000","last_fullsized":"2024-03-24T00:47:18.477886+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:57:16.256195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477449+0000","last_change":"2024-03-24T00:47:17.151006+0000","last_active":"2024-03-24T00:47:18.477449+0000","last_peered":"2024-03-24T00:47:18.477449+0000","last_clean":"2024-03-24T00:47:18.477449+0000","last_became_active":"2024-03-24T00:47:17.150156+0000","last_became_peered":"2024-03-24T00:47:17.150156+0000","last_unstale":"2024-03-24T00:47:18.477449+0000","last_undegraded":"2024-03-24T00:47:18.477449+0000","last_fullsized":"2024-03-24T00:47:18.477449+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:42:33.045278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":14,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.134877+0000","last_change":"2024-03-24T00:47:16.149966+0000","last_active":"2024-03-24T00:47:18.134877+0000","last_peered":"2024-03-24T00:47:18.134877+0000","last_clean":"2024-03-24T00:47:18.134877+0000","last_became_active":"2024-03-24T00:47:16.149684+0000","last_became_peered":"2024-03-24T00:47:16.149684+0000","last_unstale":"2024-03-24T00:47:18.134877+0000","last_undegraded":"2024-03-24T00:47:18.134877+0000","last_fullsized":"2024-03-24T00:47:18.134877+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:06:16.209605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.131474+0000","last_change":"2024-03-24T00:47:17.146131+0000","last_active":"2024-03-24T00:47:18.131474+0000","last_peered":"2024-03-24T00:47:18.131474+0000","last_clean":"2024-03-24T00:47:18.131474+0000","last_became_active":"2024-03-24T00:47:17.145869+0000","last_became_peered":"2024-03-24T00:47:17.145869+0000","last_unstale":"2024-03-24T00:47:18.131474+0000","last_undegraded":"2024-03-24T00:47:18.131474+0000","last_fullsized":"2024-03-24T00:47:18.131474+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:25:02.770395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"104'2","reported_seq":16,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477781+0000","last_change":"2024-03-24T00:47:16.138026+0000","last_active":"2024-03-24T00:47:18.477781+0000","last_peered":"2024-03-24T00:47:18.477781+0000","last_clean":"2024-03-24T00:47:18.477781+0000","last_became_active":"2024-03-24T00:47:16.137765+0000","last_became_peered":"2024-03-24T00:47:16.137765+0000","last_unstale":"2024-03-24T00:47:18.477781+0000","last_undegraded":"2024-03-24T00:47:18.477781+0000","last_fullsized":"2024-03-24T00:47:18.477781+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:31:01.641338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.131770+0000","last_change":"2024-03-24T00:47:17.155612+0000","last_active":"2024-03-24T00:47:18.131770+0000","last_peered":"2024-03-24T00:47:18.131770+0000","last_clean":"2024-03-24T00:47:18.131770+0000","last_became_active":"2024-03-24T00:47:17.154532+0000","last_became_peered":"2024-03-24T00:47:17.154532+0000","last_unstale":"2024-03-24T00:47:18.131770+0000","last_undegraded":"2024-03-24T00:47:18.131770+0000","last_fullsized":"2024-03-24T00:47:18.131770+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:59:27.246748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":14,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477062+0000","last_change":"2024-03-24T00:47:16.148191+0000","last_active":"2024-03-24T00:47:18.477062+0000","last_peered":"2024-03-24T00:47:18.477062+0000","last_clean":"2024-03-24T00:47:18.477062+0000","last_became_active":"2024-03-24T00:47:16.147927+0000","last_became_peered":"2024-03-24T00:47:16.147927+0000","last_unstale":"2024-03-24T00:47:18.477062+0000","last_undegraded":"2024-03-24T00:47:18.477062+0000","last_fullsized":"2024-03-24T00:47:18.477062+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:57:16.124980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477513+0000","last_change":"2024-03-24T00:47:17.152591+0000","last_active":"2024-03-24T00:47:18.477513+0000","last_peered":"2024-03-24T00:47:18.477513+0000","last_clean":"2024-03-24T00:47:18.477513+0000","last_became_active":"2024-03-24T00:47:17.152329+0000","last_became_peered":"2024-03-24T00:47:17.152329+0000","last_unstale":"2024-03-24T00:47:18.477513+0000","last_undegraded":"2024-03-24T00:47:18.477513+0000","last_fullsized":"2024-03-24T00:47:18.477513+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:17:29.186551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":14,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477313+0000","last_change":"2024-03-24T00:47:16.151542+0000","last_active":"2024-03-24T00:47:18.477313+0000","last_peered":"2024-03-24T00:47:18.477313+0000","last_clean":"2024-03-24T00:47:18.477313+0000","last_became_active":"2024-03-24T00:47:16.151219+0000","last_became_peered":"2024-03-24T00:47:16.151219+0000","last_unstale":"2024-03-24T00:47:18.477313+0000","last_undegraded":"2024-03-24T00:47:18.477313+0000","last_fullsized":"2024-03-24T00:47:18.477313+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:21:29.220609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477769+0000","last_change":"2024-03-24T00:47:17.159413+0000","last_active":"2024-03-24T00:47:18.477769+0000","last_peered":"2024-03-24T00:47:18.477769+0000","last_clean":"2024-03-24T00:47:18.477769+0000","last_became_active":"2024-03-24T00:47:17.158993+0000","last_became_peered":"2024-03-24T00:47:17.158993+0000","last_unstale":"2024-03-24T00:47:18.477769+0000","last_undegraded":"2024-03-24T00:47:18.477769+0000","last_fullsized":"2024-03-24T00:47:18.477769+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:11:45.335007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":14,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477181+0000","last_change":"2024-03-24T00:47:16.144375+0000","last_active":"2024-03-24T00:47:18.477181+0000","last_peered":"2024-03-24T00:47:18.477181+0000","last_clean":"2024-03-24T00:47:18.477181+0000","last_became_active":"2024-03-24T00:47:16.144044+0000","last_became_peered":"2024-03-24T00:47:16.144044+0000","last_unstale":"2024-03-24T00:47:18.477181+0000","last_undegraded":"2024-03-24T00:47:18.477181+0000","last_fullsized":"2024-03-24T00:47:18.477181+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:41:43.839546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.131235+0000","last_change":"2024-03-24T00:47:17.145454+0000","last_active":"2024-03-24T00:47:18.131235+0000","last_peered":"2024-03-24T00:47:18.131235+0000","last_clean":"2024-03-24T00:47:18.131235+0000","last_became_active":"2024-03-24T00:47:17.145141+0000","last_became_peered":"2024-03-24T00:47:17.145141+0000","last_unstale":"2024-03-24T00:47:18.131235+0000","last_undegraded":"2024-03-24T00:47:18.131235+0000","last_fullsized":"2024-03-24T00:47:18.131235+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:27:30.445712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"104'2","reported_seq":16,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477996+0000","last_change":"2024-03-24T00:47:16.144152+0000","last_active":"2024-03-24T00:47:18.477996+0000","last_peered":"2024-03-24T00:47:18.477996+0000","last_clean":"2024-03-24T00:47:18.477996+0000","last_became_active":"2024-03-24T00:47:16.143883+0000","last_became_peered":"2024-03-24T00:47:16.143883+0000","last_unstale":"2024-03-24T00:47:18.477996+0000","last_undegraded":"2024-03-24T00:47:18.477996+0000","last_fullsized":"2024-03-24T00:47:18.477996+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:27:54.657512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.134386+0000","last_change":"2024-03-24T00:47:17.147483+0000","last_active":"2024-03-24T00:47:18.134386+0000","last_peered":"2024-03-24T00:47:18.134386+0000","last_clean":"2024-03-24T00:47:18.134386+0000","last_became_active":"2024-03-24T00:47:17.146934+0000","last_became_peered":"2024-03-24T00:47:17.146934+0000","last_unstale":"2024-03-24T00:47:18.134386+0000","last_undegraded":"2024-03-24T00:47:18.134386+0000","last_fullsized":"2024-03-24T00:47:18.134386+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:53:00.441548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.6","version":"105'1","reported_seq":15,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:19.135413+0000","last_change":"2024-03-24T00:47:16.135888+0000","last_active":"2024-03-24T00:47:19.135413+0000","last_peered":"2024-03-24T00:47:19.135413+0000","last_clean":"2024-03-24T00:47:19.135413+0000","last_became_active":"2024-03-24T00:47:16.135601+0000","last_became_peered":"2024-03-24T00:47:16.135601+0000","last_unstale":"2024-03-24T00:47:19.135413+0000","last_undegraded":"2024-03-24T00:47:19.135413+0000","last_fullsized":"2024-03-24T00:47:19.135413+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:33:40.537415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477045+0000","last_change":"2024-03-24T00:47:17.151857+0000","last_active":"2024-03-24T00:47:18.477045+0000","last_peered":"2024-03-24T00:47:18.477045+0000","last_clean":"2024-03-24T00:47:18.477045+0000","last_became_active":"2024-03-24T00:47:17.151589+0000","last_became_peered":"2024-03-24T00:47:17.151589+0000","last_unstale":"2024-03-24T00:47:18.477045+0000","last_undegraded":"2024-03-24T00:47:18.477045+0000","last_fullsized":"2024-03-24T00:47:18.477045+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:50:41.868971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":14,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477819+0000","last_change":"2024-03-24T00:47:16.148013+0000","last_active":"2024-03-24T00:47:18.477819+0000","last_peered":"2024-03-24T00:47:18.477819+0000","last_clean":"2024-03-24T00:47:18.477819+0000","last_became_active":"2024-03-24T00:47:16.147658+0000","last_became_peered":"2024-03-24T00:47:16.147658+0000","last_unstale":"2024-03-24T00:47:18.477819+0000","last_undegraded":"2024-03-24T00:47:18.477819+0000","last_fullsized":"2024-03-24T00:47:18.477819+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:13:47.270490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.134734+0000","last_change":"2024-03-24T00:47:17.152727+0000","last_active":"2024-03-24T00:47:18.134734+0000","last_peered":"2024-03-24T00:47:18.134734+0000","last_clean":"2024-03-24T00:47:18.134734+0000","last_became_active":"2024-03-24T00:47:17.152454+0000","last_became_peered":"2024-03-24T00:47:17.152454+0000","last_unstale":"2024-03-24T00:47:18.134734+0000","last_undegraded":"2024-03-24T00:47:18.134734+0000","last_fullsized":"2024-03-24T00:47:18.134734+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:05:31.561170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":14,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477806+0000","last_change":"2024-03-24T00:47:16.149651+0000","last_active":"2024-03-24T00:47:18.477806+0000","last_peered":"2024-03-24T00:47:18.477806+0000","last_clean":"2024-03-24T00:47:18.477806+0000","last_became_active":"2024-03-24T00:47:16.149285+0000","last_became_peered":"2024-03-24T00:47:16.149285+0000","last_unstale":"2024-03-24T00:47:18.477806+0000","last_undegraded":"2024-03-24T00:47:18.477806+0000","last_fullsized":"2024-03-24T00:47:18.477806+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:40:02.531399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477638+0000","last_change":"2024-03-24T00:47:17.152781+0000","last_active":"2024-03-24T00:47:18.477638+0000","last_peered":"2024-03-24T00:47:18.477638+0000","last_clean":"2024-03-24T00:47:18.477638+0000","last_became_active":"2024-03-24T00:47:17.152426+0000","last_became_peered":"2024-03-24T00:47:17.152426+0000","last_unstale":"2024-03-24T00:47:18.477638+0000","last_undegraded":"2024-03-24T00:47:18.477638+0000","last_fullsized":"2024-03-24T00:47:18.477638+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:34:58.990234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":14,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.134661+0000","last_change":"2024-03-24T00:47:16.147713+0000","last_active":"2024-03-24T00:47:18.134661+0000","last_peered":"2024-03-24T00:47:18.134661+0000","last_clean":"2024-03-24T00:47:18.134661+0000","last_became_active":"2024-03-24T00:47:16.147366+0000","last_became_peered":"2024-03-24T00:47:16.147366+0000","last_unstale":"2024-03-24T00:47:18.134661+0000","last_undegraded":"2024-03-24T00:47:18.134661+0000","last_fullsized":"2024-03-24T00:47:18.134661+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:41:53.369936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477924+0000","last_change":"2024-03-24T00:47:17.153365+0000","last_active":"2024-03-24T00:47:18.477924+0000","last_peered":"2024-03-24T00:47:18.477924+0000","last_clean":"2024-03-24T00:47:18.477924+0000","last_became_active":"2024-03-24T00:47:17.153073+0000","last_became_peered":"2024-03-24T00:47:17.153073+0000","last_unstale":"2024-03-24T00:47:18.477924+0000","last_undegraded":"2024-03-24T00:47:18.477924+0000","last_fullsized":"2024-03-24T00:47:18.477924+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:55:30.418171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":14,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477300+0000","last_change":"2024-03-24T00:47:16.148454+0000","last_active":"2024-03-24T00:47:18.477300+0000","last_peered":"2024-03-24T00:47:18.477300+0000","last_clean":"2024-03-24T00:47:18.477300+0000","last_became_active":"2024-03-24T00:47:16.147257+0000","last_became_peered":"2024-03-24T00:47:16.147257+0000","last_unstale":"2024-03-24T00:47:18.477300+0000","last_undegraded":"2024-03-24T00:47:18.477300+0000","last_fullsized":"2024-03-24T00:47:18.477300+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:22:00.243619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477762+0000","last_change":"2024-03-24T00:47:17.153777+0000","last_active":"2024-03-24T00:47:18.477762+0000","last_peered":"2024-03-24T00:47:18.477762+0000","last_clean":"2024-03-24T00:47:18.477762+0000","last_became_active":"2024-03-24T00:47:17.153452+0000","last_became_peered":"2024-03-24T00:47:17.153452+0000","last_unstale":"2024-03-24T00:47:18.477762+0000","last_undegraded":"2024-03-24T00:47:18.477762+0000","last_fullsized":"2024-03-24T00:47:18.477762+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:53:45.283188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":14,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477539+0000","last_change":"2024-03-24T00:47:16.148022+0000","last_active":"2024-03-24T00:47:18.477539+0000","last_peered":"2024-03-24T00:47:18.477539+0000","last_clean":"2024-03-24T00:47:18.477539+0000","last_became_active":"2024-03-24T00:47:16.147216+0000","last_became_peered":"2024-03-24T00:47:16.147216+0000","last_unstale":"2024-03-24T00:47:18.477539+0000","last_undegraded":"2024-03-24T00:47:18.477539+0000","last_fullsized":"2024-03-24T00:47:18.477539+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:36:10.435766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.135579+0000","last_change":"2024-03-24T00:47:17.143293+0000","last_active":"2024-03-24T00:47:18.135579+0000","last_peered":"2024-03-24T00:47:18.135579+0000","last_clean":"2024-03-24T00:47:18.135579+0000","last_became_active":"2024-03-24T00:47:17.142992+0000","last_became_peered":"2024-03-24T00:47:17.142992+0000","last_unstale":"2024-03-24T00:47:18.135579+0000","last_undegraded":"2024-03-24T00:47:18.135579+0000","last_fullsized":"2024-03-24T00:47:18.135579+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:53:45.283188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":14,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477271+0000","last_change":"2024-03-24T00:47:16.150192+0000","last_active":"2024-03-24T00:47:18.477271+0000","last_peered":"2024-03-24T00:47:18.477271+0000","last_clean":"2024-03-24T00:47:18.477271+0000","last_became_active":"2024-03-24T00:47:16.148847+0000","last_became_peered":"2024-03-24T00:47:16.148847+0000","last_unstale":"2024-03-24T00:47:18.477271+0000","last_undegraded":"2024-03-24T00:47:18.477271+0000","last_fullsized":"2024-03-24T00:47:18.477271+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:40:56.988956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.134209+0000","last_change":"2024-03-24T00:47:17.145149+0000","last_active":"2024-03-24T00:47:18.134209+0000","last_peered":"2024-03-24T00:47:18.134209+0000","last_clean":"2024-03-24T00:47:18.134209+0000","last_became_active":"2024-03-24T00:47:17.144806+0000","last_became_peered":"2024-03-24T00:47:17.144806+0000","last_unstale":"2024-03-24T00:47:18.134209+0000","last_undegraded":"2024-03-24T00:47:18.134209+0000","last_fullsized":"2024-03-24T00:47:18.134209+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:16.119951+0000","last_clean_scrub_stamp":"2024-03-24T00:47:16.119951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:01:40.517054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.e","version":"104'2","reported_seq":16,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.478466+0000","last_change":"2024-03-24T00:47:16.147574+0000","last_active":"2024-03-24T00:47:18.478466+0000","last_peered":"2024-03-24T00:47:18.478466+0000","last_clean":"2024-03-24T00:47:18.478466+0000","last_became_active":"2024-03-24T00:47:16.147236+0000","last_became_peered":"2024-03-24T00:47:16.147236+0000","last_unstale":"2024-03-24T00:47:18.478466+0000","last_undegraded":"2024-03-24T00:47:18.478466+0000","last_fullsized":"2024-03-24T00:47:18.478466+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:47:15.113196+0000","last_clean_scrub_stamp":"2024-03-24T00:47:15.113196+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:19:32.345194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":329,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:47:18.477566+0000","last_change":"2024-03-24T00:38:40.953871+0000","last_active":"2024-03-24T00:47:18.477566+0000","last_peered":"2024-03-24T00:47:18.477566+0000","last_clean":"2024-03-24T00:47:18.477566+0000","last_became_active":"2024-03-24T00:38:40.952926+0000","last_became_peered":"2024-03-24T00:38:40.952926+0000","last_unstale":"2024-03-24T00:47:18.477566+0000","last_undegraded":"2024-03-24T00:47:18.477566+0000","last_fullsized":"2024-03-24T00:47:18.477566+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_clean_scrub_stamp":"2024-03-24T00:38:39.722194+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:05.397257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":16,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574957,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33044,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338796,"statfs":{"total":96636764160,"available":96602927104,"internally_reserved":0,"allocated":1904640,"data_stored":1196890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574956,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32476,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339364,"statfs":{"total":96636764160,"available":96603508736,"internally_reserved":0,"allocated":1323008,"data_stored":608578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574956,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32456,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339384,"statfs":{"total":96636764160,"available":96603529216,"internally_reserved":0,"allocated":1302528,"data_stored":606362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574955,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32464,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339376,"statfs":{"total":96636764160,"available":96603521024,"internally_reserved":0,"allocated":1310720,"data_stored":606454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574955,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32444,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339396,"statfs":{"total":96636764160,"available":96603541504,"internally_reserved":0,"allocated":1294336,"data_stored":600252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574956,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32460,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339380,"statfs":{"total":96636764160,"available":96603525120,"internally_reserved":0,"allocated":1306624,"data_stored":606396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574956,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32448,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339392,"statfs":{"total":96636764160,"available":96603537408,"internally_reserved":0,"allocated":1294336,"data_stored":599704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574955,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33052,"kb_used_data":1868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338788,"statfs":{"total":96636764160,"available":96602918912,"internally_reserved":0,"allocated":1912832,"data_stored":1198432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:47:20.669 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-24T00:47:20.669 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-24T00:47:20.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.830+0000 7f992daac640 1 Processor -- start 2024-03-24T00:47:20.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.830+0000 7f992daac640 1 -- start start 2024-03-24T00:47:20.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.831+0000 7f992daac640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f99280b1b50 0x7f99280b1f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:20.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.831+0000 7f992daac640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f992806b040 0x7f992806b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:20.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.831+0000 7f992daac640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f992815e4c0 con 0x7f992815e0e0 2024-03-24T00:47:20.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.831+0000 7f992daac640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99280b24d0 con 0x7f99280b1b50 2024-03-24T00:47:20.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.831+0000 7f992daac640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99280b0c80 con 0x7f992806b040 2024-03-24T00:47:20.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.831+0000 7f9926ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f99280b1b50 0x7f99280b1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:20.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.831+0000 7f99277fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f992815e0e0 0x7f99280b1110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48302/0 (socket says 172.21.15.67:48302) 2024-03-24T00:47:20.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.831+0000 7f99277fe640 1 -- 172.21.15.67:0/2855016916 learned_addr learned my addr 172.21.15.67:0/2855016916 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:47:20.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.832+0000 7f9927fff640 1 --2- 172.21.15.67:0/2855016916 >> v2:172.21.15.72:3300/0 conn(0x7f992806b040 0x7f992806b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:20.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.832+0000 7f9926ffd640 1 -- 172.21.15.67:0/2855016916 >> v2:172.21.15.72:3300/0 conn(0x7f992806b040 msgr2=0x7f992806b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:20.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.832+0000 7f9926ffd640 1 --2- 172.21.15.67:0/2855016916 >> v2:172.21.15.72:3300/0 conn(0x7f992806b040 0x7f992806b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:20.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.832+0000 7f9926ffd640 1 -- 172.21.15.67:0/2855016916 >> v1:172.21.15.67:6789/0 conn(0x7f992815e0e0 legacy=0x7f99280b1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:20.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.832+0000 7f99267fc640 1 -- 172.21.15.67:0/2855016916 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 784654885 0 0) 0x7f992815e4c0 con 0x7f992815e0e0 2024-03-24T00:47:20.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.832+0000 7f9926ffd640 1 -- 172.21.15.67:0/2855016916 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99280b1820 con 0x7f99280b1b50 2024-03-24T00:47:20.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.832+0000 7f9927fff640 1 --2- 172.21.15.67:0/2855016916 >> v2:172.21.15.72:3300/0 conn(0x7f992806b040 0x7f992806b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:47:20.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.833+0000 7f9926ffd640 1 --2- 172.21.15.67:0/2855016916 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99280b1b50 0x7f99280b1f90 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f9928071380 tx=0x7f991802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ae16a773ea45aae server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:20.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.833+0000 7f99267fc640 1 -- 172.21.15.67:0/2855016916 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9918034070 con 0x7f99280b1b50 2024-03-24T00:47:20.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.833+0000 7f99267fc640 1 -- 172.21.15.67:0/2855016916 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9918030610 con 0x7f99280b1b50 2024-03-24T00:47:20.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.833+0000 7f99267fc640 1 -- 172.21.15.67:0/2855016916 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99180309e0 con 0x7f99280b1b50 2024-03-24T00:47:20.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.833+0000 7f992daac640 1 -- 172.21.15.67:0/2855016916 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99280b1b50 msgr2=0x7f99280b1f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:20.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.833+0000 7f992daac640 1 --2- 172.21.15.67:0/2855016916 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99280b1b50 0x7f99280b1f90 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f9928071380 tx=0x7f991802b960 comp rx=0 tx=0).stop 2024-03-24T00:47:20.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.833+0000 7f992daac640 1 -- 172.21.15.67:0/2855016916 shutdown_connections 2024-03-24T00:47:20.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.833+0000 7f992daac640 1 --2- 172.21.15.67:0/2855016916 >> v2:172.21.15.72:3300/0 conn(0x7f992806b040 0x7f992806b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:20.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.833+0000 7f992daac640 1 --2- 172.21.15.67:0/2855016916 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99280b1b50 0x7f99280b1f90 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:20.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.833+0000 7f992daac640 1 -- 172.21.15.67:0/2855016916 >> 172.21.15.67:0/2855016916 conn(0x7f9928099a30 msgr2=0x7f9928099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:20.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.833+0000 7f992daac640 1 -- 172.21.15.67:0/2855016916 shutdown_connections 2024-03-24T00:47:20.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.833+0000 7f992daac640 1 -- 172.21.15.67:0/2855016916 wait complete. 2024-03-24T00:47:20.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.835+0000 7f992daac640 1 Processor -- start 2024-03-24T00:47:20.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.835+0000 7f992daac640 1 -- start start 2024-03-24T00:47:20.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.836+0000 7f992daac640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f992806b040 0x7f99281523a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:20.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.836+0000 7f992daac640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f99280b1b50 0x7f99280cc180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:20.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.836+0000 7f992daac640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f992815e0e0 0x7f99280cc6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:20.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.836+0000 7f992daac640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99280b0630 con 0x7f992806b040 2024-03-24T00:47:20.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.836+0000 7f992daac640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99280b0350 con 0x7f992815e0e0 2024-03-24T00:47:20.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.836+0000 7f992daac640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99280b0910 con 0x7f99280b1b50 2024-03-24T00:47:20.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.837+0000 7f99277fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f992806b040 0x7f99281523a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:20.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.837+0000 7f99277fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f992806b040 0x7f99281523a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47264/0 (socket says 172.21.15.67:47264) 2024-03-24T00:47:20.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.837+0000 7f99277fe640 1 -- 172.21.15.67:0/3429026654 learned_addr learned my addr 172.21.15.67:0/3429026654 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:20.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.837+0000 7f9927fff640 1 --2- 172.21.15.67:0/3429026654 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f992815e0e0 0x7f99280cc6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:20.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.837+0000 7f9926ffd640 1 --2- 172.21.15.67:0/3429026654 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f99280b1b50 0x7f99280cc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:20.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.837+0000 7f99277fe640 1 -- 172.21.15.67:0/3429026654 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f99280b1b50 msgr2=0x7f99280cc180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:20.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.837+0000 7f99277fe640 1 --2- 172.21.15.67:0/3429026654 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f99280b1b50 0x7f99280cc180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:20.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.837+0000 7f99277fe640 1 -- 172.21.15.67:0/3429026654 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f992815e0e0 msgr2=0x7f99280cc6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:20.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.837+0000 7f99277fe640 1 --2- 172.21.15.67:0/3429026654 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f992815e0e0 0x7f99280cc6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:20.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.837+0000 7f99277fe640 1 -- 172.21.15.67:0/3429026654 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99281404e0 con 0x7f992806b040 2024-03-24T00:47:20.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.837+0000 7f99277fe640 1 --2- 172.21.15.67:0/3429026654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f992806b040 0x7f99281523a0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f991000c460 tx=0x7f991000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:20.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.837+0000 7f991ffff640 1 -- 172.21.15.67:0/3429026654 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9910018070 con 0x7f992806b040 2024-03-24T00:47:20.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.837+0000 7f991ffff640 1 -- 172.21.15.67:0/3429026654 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9910007500 con 0x7f992806b040 2024-03-24T00:47:20.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.837+0000 7f991ffff640 1 -- 172.21.15.67:0/3429026654 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9910004a20 con 0x7f992806b040 2024-03-24T00:47:20.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.837+0000 7f992daac640 1 -- 172.21.15.67:0/3429026654 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99281407c0 con 0x7f992806b040 2024-03-24T00:47:20.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.837+0000 7f992daac640 1 -- 172.21.15.67:0/3429026654 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9928140d80 con 0x7f992806b040 2024-03-24T00:47:20.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.838+0000 7f992daac640 1 -- 172.21.15.67:0/3429026654 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f992804f400 con 0x7f992806b040 2024-03-24T00:47:20.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.839+0000 7f991ffff640 1 -- 172.21.15.67:0/3429026654 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9910004bc0 con 0x7f992806b040 2024-03-24T00:47:20.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.839+0000 7f991ffff640 1 --2- 172.21.15.67:0/3429026654 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f990806bbd0 0x7f990806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:20.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.839+0000 7f991ffff640 1 -- 172.21.15.67:0/3429026654 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9910089ec0 con 0x7f992806b040 2024-03-24T00:47:20.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.839+0000 7f9926ffd640 1 --2- 172.21.15.67:0/3429026654 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f990806bbd0 0x7f990806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:20.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.840+0000 7f9926ffd640 1 --2- 172.21.15.67:0/3429026654 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f990806bbd0 0x7f990806e090 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f9918036000 tx=0x7f991803b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:20.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:20.848+0000 7f991ffff640 1 -- 172.21.15.67:0/3429026654 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f99100581e0 con 0x7f992806b040 2024-03-24T00:47:21.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.100+0000 7f992daac640 1 -- 172.21.15.67:0/3429026654 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f992815bb70 con 0x7f992806b040 2024-03-24T00:47:21.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.103+0000 7f991ffff640 1 -- 172.21.15.67:0/3429026654 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v19) v1 ==== 190+0+147 (secure 0 0 0) 0x7f991005c280 con 0x7f992806b040 2024-03-24T00:47:21.103 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-03-24T00:47:21.103 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQDcdf9lY4nvJRAA6HWOZDWA+SSvJVmyjG+51g== 2024-03-24T00:47:21.103 INFO:teuthology.orchestra.run.smithi067.stdout: caps mds = "allow" 2024-03-24T00:47:21.103 INFO:teuthology.orchestra.run.smithi067.stdout: caps mon = "allow r" 2024-03-24T00:47:21.103 INFO:teuthology.orchestra.run.smithi067.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-24T00:47:21.103 INFO:teuthology.orchestra.run.smithi067.stderr:updated caps for client.0 2024-03-24T00:47:21.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.105+0000 7f992daac640 1 -- 172.21.15.67:0/3429026654 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f990806bbd0 msgr2=0x7f990806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:21.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.105+0000 7f992daac640 1 --2- 172.21.15.67:0/3429026654 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f990806bbd0 0x7f990806e090 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f9918036000 tx=0x7f991803b040 comp rx=0 tx=0).stop 2024-03-24T00:47:21.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.105+0000 7f992daac640 1 -- 172.21.15.67:0/3429026654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f992806b040 msgr2=0x7f99281523a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:21.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.105+0000 7f992daac640 1 --2- 172.21.15.67:0/3429026654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f992806b040 0x7f99281523a0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f991000c460 tx=0x7f991000c920 comp rx=0 tx=0).stop 2024-03-24T00:47:21.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.105+0000 7f992daac640 1 -- 172.21.15.67:0/3429026654 shutdown_connections 2024-03-24T00:47:21.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.105+0000 7f992daac640 1 --2- 172.21.15.67:0/3429026654 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f990806bbd0 0x7f990806e090 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:21.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.105+0000 7f992daac640 1 --2- 172.21.15.67:0/3429026654 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f992815e0e0 0x7f99280cc6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:21.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.105+0000 7f992daac640 1 --2- 172.21.15.67:0/3429026654 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f99280b1b50 0x7f99280cc180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:21.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.105+0000 7f992daac640 1 --2- 172.21.15.67:0/3429026654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f992806b040 0x7f99281523a0 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:21.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.105+0000 7f992daac640 1 -- 172.21.15.67:0/3429026654 >> 172.21.15.67:0/3429026654 conn(0x7f9928099a30 msgr2=0x7f99280cfcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:21.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.106+0000 7f992daac640 1 -- 172.21.15.67:0/3429026654 shutdown_connections 2024-03-24T00:47:21.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.106+0000 7f992daac640 1 -- 172.21.15.67:0/3429026654 wait complete. 2024-03-24T00:47:21.116 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:47:21.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.218+0000 7f282fd29640 1 Processor -- start 2024-03-24T00:47:21.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.218+0000 7f282fd29640 1 -- start start 2024-03-24T00:47:21.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.219+0000 7f282fd29640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f282815e2a0 0x7f28280ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:21.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.219+0000 7f282fd29640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28280cf640 con 0x7f282815dec0 2024-03-24T00:47:21.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.219+0000 7f282fd29640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28280cf080 con 0x7f28280d0ef0 2024-03-24T00:47:21.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.219+0000 7f282fd29640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28280cee30 con 0x7f282815e2a0 2024-03-24T00:47:21.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.219+0000 7f282d2c4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f282815e2a0 0x7f28280ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:21.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.219+0000 7f282dac5640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f282815dec0 0x7f28280d07e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48308/0 (socket says 172.21.15.67:48308) 2024-03-24T00:47:21.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.219+0000 7f282dac5640 1 -- 172.21.15.67:0/887065308 learned_addr learned my addr 172.21.15.67:0/887065308 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:47:21.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.220+0000 7f282d2c4640 1 -- 172.21.15.67:0/887065308 >> v1:172.21.15.72:6789/0 conn(0x7f28280d0ef0 legacy=0x7f28280d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:21.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.220+0000 7f282d2c4640 1 -- 172.21.15.67:0/887065308 >> v1:172.21.15.67:6789/0 conn(0x7f282815dec0 legacy=0x7f28280d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:21.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.220+0000 7f282d2c4640 1 -- 172.21.15.67:0/887065308 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28280cf360 con 0x7f282815e2a0 2024-03-24T00:47:21.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.220+0000 7f282cac3640 1 -- 172.21.15.67:0/887065308 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 99898196 0 0) 0x7f2824001110 con 0x7f282815dec0 2024-03-24T00:47:21.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.220+0000 7f282d2c4640 1 --2- 172.21.15.67:0/887065308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f282815e2a0 0x7f28280ce630 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f28280cefb0 tx=0x7f281802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da1812cb5c970186 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:21.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.221+0000 7f282cac3640 1 -- 172.21.15.67:0/887065308 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28180315a0 con 0x7f282815e2a0 2024-03-24T00:47:21.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.221+0000 7f282cac3640 1 -- 172.21.15.67:0/887065308 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2818037070 con 0x7f282815e2a0 2024-03-24T00:47:21.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.221+0000 7f282cac3640 1 -- 172.21.15.67:0/887065308 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2818031960 con 0x7f282815e2a0 2024-03-24T00:47:21.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.221+0000 7f282fd29640 1 -- 172.21.15.67:0/887065308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f282815e2a0 msgr2=0x7f28280ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:21.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.221+0000 7f282fd29640 1 --2- 172.21.15.67:0/887065308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f282815e2a0 0x7f28280ce630 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f28280cefb0 tx=0x7f281802b840 comp rx=0 tx=0).stop 2024-03-24T00:47:21.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.221+0000 7f282fd29640 1 -- 172.21.15.67:0/887065308 shutdown_connections 2024-03-24T00:47:21.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.221+0000 7f282fd29640 1 --2- 172.21.15.67:0/887065308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f282815e2a0 0x7f28280ce630 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:21.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.221+0000 7f282fd29640 1 -- 172.21.15.67:0/887065308 >> 172.21.15.67:0/887065308 conn(0x7f282806c810 msgr2=0x7f282806cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:21.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.221+0000 7f282fd29640 1 -- 172.21.15.67:0/887065308 shutdown_connections 2024-03-24T00:47:21.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.221+0000 7f282fd29640 1 -- 172.21.15.67:0/887065308 wait complete. 2024-03-24T00:47:21.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.223+0000 7f282fd29640 1 Processor -- start 2024-03-24T00:47:21.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.223+0000 7f282fd29640 1 -- start start 2024-03-24T00:47:21.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.224+0000 7f282fd29640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28280d0ef0 0x7f28280b6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:21.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.224+0000 7f282fd29640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f282815dec0 0x7f28280b3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:21.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.224+0000 7f282fd29640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f282815e2a0 0x7f2828159330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:21.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.224+0000 7f282fd29640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2828162a40 con 0x7f28280d0ef0 2024-03-24T00:47:21.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.224+0000 7f282fd29640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28281628c0 con 0x7f282815e2a0 2024-03-24T00:47:21.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.224+0000 7f282fd29640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28280d2120 con 0x7f282815dec0 2024-03-24T00:47:21.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.224+0000 7f282dac5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28280d0ef0 0x7f28280b6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:21.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.224+0000 7f282dac5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28280d0ef0 0x7f28280b6360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47288/0 (socket says 172.21.15.67:47288) 2024-03-24T00:47:21.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.224+0000 7f282dac5640 1 -- 172.21.15.67:0/1128065760 learned_addr learned my addr 172.21.15.67:0/1128065760 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:21.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.224+0000 7f282d2c4640 1 --2- 172.21.15.67:0/1128065760 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f282815dec0 0x7f28280b3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:21.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.224+0000 7f282dac5640 1 -- 172.21.15.67:0/1128065760 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f282815dec0 msgr2=0x7f28280b3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:21.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.224+0000 7f282dac5640 1 --2- 172.21.15.67:0/1128065760 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f282815dec0 0x7f28280b3810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:21.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.224+0000 7f282e2c6640 1 --2- 172.21.15.67:0/1128065760 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f282815e2a0 0x7f2828159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:21.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.224+0000 7f282dac5640 1 -- 172.21.15.67:0/1128065760 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f282815e2a0 msgr2=0x7f2828159330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:21.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.224+0000 7f282dac5640 1 --2- 172.21.15.67:0/1128065760 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f282815e2a0 0x7f2828159330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:21.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.224+0000 7f282dac5640 1 -- 172.21.15.67:0/1128065760 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28280cd3b0 con 0x7f28280d0ef0 2024-03-24T00:47:21.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.224+0000 7f282dac5640 1 --2- 172.21.15.67:0/1128065760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28280d0ef0 0x7f28280b6360 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f282400c7b0 tx=0x7f282400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:21.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.224+0000 7f281e7fc640 1 -- 172.21.15.67:0/1128065760 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2824018070 con 0x7f28280d0ef0 2024-03-24T00:47:21.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.224+0000 7f281e7fc640 1 -- 172.21.15.67:0/1128065760 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28240072d0 con 0x7f28280d0ef0 2024-03-24T00:47:21.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.224+0000 7f282fd29640 1 -- 172.21.15.67:0/1128065760 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28281e7b70 con 0x7f28280d0ef0 2024-03-24T00:47:21.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.225+0000 7f281e7fc640 1 -- 172.21.15.67:0/1128065760 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28240047e0 con 0x7f28280d0ef0 2024-03-24T00:47:21.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.225+0000 7f282fd29640 1 -- 172.21.15.67:0/1128065760 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28281e8130 con 0x7f28280d0ef0 2024-03-24T00:47:21.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.225+0000 7f282fd29640 1 -- 172.21.15.67:0/1128065760 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28281e7960 con 0x7f28280d0ef0 2024-03-24T00:47:21.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.226+0000 7f281e7fc640 1 -- 172.21.15.67:0/1128065760 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f282401b070 con 0x7f28280d0ef0 2024-03-24T00:47:21.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.226+0000 7f281e7fc640 1 --2- 172.21.15.67:0/1128065760 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f280c06bbd0 0x7f280c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:21.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.226+0000 7f281e7fc640 1 -- 172.21.15.67:0/1128065760 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2824089f40 con 0x7f28280d0ef0 2024-03-24T00:47:21.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.226+0000 7f282d2c4640 1 --2- 172.21.15.67:0/1128065760 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f280c06bbd0 0x7f280c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:21.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.227+0000 7f282d2c4640 1 --2- 172.21.15.67:0/1128065760 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f280c06bbd0 0x7f280c06e090 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f2818036000 tx=0x7f2818008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:21.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.231+0000 7f281e7fc640 1 -- 172.21.15.67:0/1128065760 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f28280cf640 con 0x7f28280d0ef0 2024-03-24T00:47:21.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.451+0000 7f282fd29640 1 -- 172.21.15.67:0/1128065760 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f282804f400 con 0x7f28280d0ef0 2024-03-24T00:47:21.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.452+0000 7f281e7fc640 1 -- 172.21.15.67:0/1128065760 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+4976 (secure 0 0 0) 0x7f28240588a0 con 0x7f28280d0ef0 2024-03-24T00:47:21.452 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:47:21.452 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":72,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6902,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3672294289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3672294289},{"type":"v1","addr":"172.21.15.67:6835","nonce":3672294289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":6908,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/1286271878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":1286271878},{"type":"v1","addr":"172.21.15.72:6836","nonce":1286271878}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":6911,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/140622129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":140622129},{"type":"v1","addr":"172.21.15.72:6837","nonce":140622129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:47:16.492485+0000","modified":"2024-03-24T00:47:18.507677+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8541},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8541":{"gid":8541,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6833/1826974843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1826974843},{"type":"v1","addr":"172.21.15.67:6833","nonce":1826974843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8541,"qdb_cluster":[8541]},"id":7}]} 2024-03-24T00:47:21.452 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 72 2024-03-24T00:47:21.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.454+0000 7f282fd29640 1 -- 172.21.15.67:0/1128065760 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f280c06bbd0 msgr2=0x7f280c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:21.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.454+0000 7f282fd29640 1 --2- 172.21.15.67:0/1128065760 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f280c06bbd0 0x7f280c06e090 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f2818036000 tx=0x7f2818008030 comp rx=0 tx=0).stop 2024-03-24T00:47:21.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.454+0000 7f282fd29640 1 -- 172.21.15.67:0/1128065760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28280d0ef0 msgr2=0x7f28280b6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:21.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.454+0000 7f282fd29640 1 --2- 172.21.15.67:0/1128065760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28280d0ef0 0x7f28280b6360 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f282400c7b0 tx=0x7f282400cc70 comp rx=0 tx=0).stop 2024-03-24T00:47:21.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.454+0000 7f282fd29640 1 -- 172.21.15.67:0/1128065760 shutdown_connections 2024-03-24T00:47:21.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.454+0000 7f282fd29640 1 --2- 172.21.15.67:0/1128065760 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f280c06bbd0 0x7f280c06e090 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:21.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.454+0000 7f282fd29640 1 --2- 172.21.15.67:0/1128065760 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f282815e2a0 0x7f2828159330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:21.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.454+0000 7f282fd29640 1 --2- 172.21.15.67:0/1128065760 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f282815dec0 0x7f28280b3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:21.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.454+0000 7f282fd29640 1 --2- 172.21.15.67:0/1128065760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28280d0ef0 0x7f28280b6360 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:21.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.454+0000 7f282fd29640 1 -- 172.21.15.67:0/1128065760 >> 172.21.15.67:0/1128065760 conn(0x7f282806c810 msgr2=0x7f282806cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:21.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.454+0000 7f282fd29640 1 -- 172.21.15.67:0/1128065760 shutdown_connections 2024-03-24T00:47:21.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.454+0000 7f282fd29640 1 -- 172.21.15.67:0/1128065760 wait complete. 2024-03-24T00:47:21.868 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 72, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:47:16.492485+0000', 'modified': '2024-03-24T00:47:18.507677+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8541}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8541': {'gid': 8541, 'name': 'c', 'rank': 0, 'incarnation': 70, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.67:6833/1826974843', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6832', 'nonce': 1826974843}, {'type': 'v1', 'addr': '172.21.15.67:6833', 'nonce': 1826974843}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8541, 'qdb_cluster': [8541]} 2024-03-24T00:47:21.868 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-24T00:47:21.868 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-24T00:47:21.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.972+0000 7fcb8fdf4640 1 Processor -- start 2024-03-24T00:47:21.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.972+0000 7fcb8fdf4640 1 -- start start 2024-03-24T00:47:21.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.974+0000 7fcb8fdf4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcb8815e0f0 0x7fcb880b95b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:21.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.974+0000 7fcb8fdf4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fcb880b9af0 0x7fcb880d32d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:21.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.974+0000 7fcb8fdf4640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb880d2230 con 0x7fcb880d3810 2024-03-24T00:47:21.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.974+0000 7fcb8fdf4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb880d2510 con 0x7fcb880b9af0 2024-03-24T00:47:21.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.974+0000 7fcb8fdf4640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb880d27f0 con 0x7fcb8815e0f0 2024-03-24T00:47:21.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.974+0000 7fcb8e391640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fcb880d3810 0x7fcb880b6dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48312/0 (socket says 172.21.15.67:48312) 2024-03-24T00:47:21.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.974+0000 7fcb8e391640 1 -- 172.21.15.67:0/3041122951 learned_addr learned my addr 172.21.15.67:0/3041122951 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:47:21.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.974+0000 7fcb8db90640 1 --2- 172.21.15.67:0/3041122951 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcb8815e0f0 0x7fcb880b95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:21.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.974+0000 7fcb8d38f640 1 --2- 172.21.15.67:0/3041122951 >> v2:172.21.15.72:3300/0 conn(0x7fcb880b9af0 0x7fcb880d32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:21.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.974+0000 7fcb8cb8e640 1 -- 172.21.15.67:0/3041122951 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1053885811 0 0) 0x7fcb880d2230 con 0x7fcb880d3810 2024-03-24T00:47:21.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.974+0000 7fcb8cb8e640 1 -- 172.21.15.67:0/3041122951 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb74003520 con 0x7fcb880d3810 2024-03-24T00:47:21.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.974+0000 7fcb8cb8e640 1 -- 172.21.15.67:0/3041122951 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 705599333 0 0) 0x7fcb74003520 con 0x7fcb880d3810 2024-03-24T00:47:21.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.974+0000 7fcb8cb8e640 1 -- 172.21.15.67:0/3041122951 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcb8815e0f0 msgr2=0x7fcb880b95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:21.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.974+0000 7fcb8cb8e640 1 --2- 172.21.15.67:0/3041122951 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcb8815e0f0 0x7fcb880b95b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:21.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.974+0000 7fcb8cb8e640 1 -- 172.21.15.67:0/3041122951 >> v2:172.21.15.72:3300/0 conn(0x7fcb880b9af0 msgr2=0x7fcb880d32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:21.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.974+0000 7fcb8cb8e640 1 --2- 172.21.15.67:0/3041122951 >> v2:172.21.15.72:3300/0 conn(0x7fcb880b9af0 0x7fcb880d32d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:21.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.974+0000 7fcb8cb8e640 1 -- 172.21.15.67:0/3041122951 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb880d2ad0 con 0x7fcb880d3810 2024-03-24T00:47:21.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.974+0000 7fcb8cb8e640 1 -- 172.21.15.67:0/3041122951 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7fcb84001670 con 0x7fcb880d3810 2024-03-24T00:47:21.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.974+0000 7fcb8cb8e640 1 -- 172.21.15.67:0/3041122951 >> v1:172.21.15.67:6789/0 conn(0x7fcb880d3810 legacy=0x7fcb880b6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:21.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.975+0000 7fcb8cb8e640 1 --2- 172.21.15.67:0/3041122951 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcb8815e0f0 0x7fcb740039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:21.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.975+0000 7fcb8cb8e640 1 --2- 172.21.15.67:0/3041122951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb74003f30 0x7fcb74004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:21.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.975+0000 7fcb8cb8e640 1 --2- 172.21.15.67:0/3041122951 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb74004d40 0x7fcb740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:21.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.975+0000 7fcb8cb8e640 1 -- 172.21.15.67:0/3041122951 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb880d2510 con 0x7fcb74003f30 2024-03-24T00:47:21.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.975+0000 7fcb8cb8e640 1 -- 172.21.15.67:0/3041122951 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb880d27f0 con 0x7fcb74004d40 2024-03-24T00:47:21.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.975+0000 7fcb8cb8e640 1 -- 172.21.15.67:0/3041122951 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7400b9c0 con 0x7fcb8815e0f0 2024-03-24T00:47:21.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.975+0000 7fcb8d38f640 1 --2- 172.21.15.67:0/3041122951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb74003f30 0x7fcb74004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:21.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.975+0000 7fcb8db90640 1 --2- 172.21.15.67:0/3041122951 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcb8815e0f0 0x7fcb740039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:21.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.975+0000 7fcb8e391640 1 --2- 172.21.15.67:0/3041122951 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb74004d40 0x7fcb740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:21.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.975+0000 7fcb8d38f640 1 -- 172.21.15.67:0/3041122951 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcb8815e0f0 msgr2=0x7fcb740039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:21.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.975+0000 7fcb8d38f640 1 --2- 172.21.15.67:0/3041122951 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcb8815e0f0 0x7fcb740039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:21.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.975+0000 7fcb8d38f640 1 -- 172.21.15.67:0/3041122951 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb74004d40 msgr2=0x7fcb740051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:21.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.975+0000 7fcb8d38f640 1 --2- 172.21.15.67:0/3041122951 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb74004d40 0x7fcb740051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:21.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.975+0000 7fcb8d38f640 1 -- 172.21.15.67:0/3041122951 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb74003520 con 0x7fcb74003f30 2024-03-24T00:47:21.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.976+0000 7fcb8d38f640 1 --2- 172.21.15.67:0/3041122951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb74003f30 0x7fcb74004800 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7fcb78009ab0 tx=0x7fcb74005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36cae0942791a3ee server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:21.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.976+0000 7fcb8cb8e640 1 -- 172.21.15.67:0/3041122951 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb78009680 con 0x7fcb74003f30 2024-03-24T00:47:21.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.976+0000 7fcb8cb8e640 1 -- 172.21.15.67:0/3041122951 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb78005990 con 0x7fcb74003f30 2024-03-24T00:47:21.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.976+0000 7fcb8cb8e640 1 -- 172.21.15.67:0/3041122951 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb78005d80 con 0x7fcb74003f30 2024-03-24T00:47:21.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.976+0000 7fcb8fdf4640 1 -- 172.21.15.67:0/3041122951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb74003f30 msgr2=0x7fcb74004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:21.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.976+0000 7fcb8fdf4640 1 --2- 172.21.15.67:0/3041122951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb74003f30 0x7fcb74004800 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7fcb78009ab0 tx=0x7fcb74005830 comp rx=0 tx=0).stop 2024-03-24T00:47:21.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.976+0000 7fcb8e391640 1 -- 172.21.15.67:0/3041122951 reap_dead start 2024-03-24T00:47:21.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.976+0000 7fcb8fdf4640 1 -- 172.21.15.67:0/3041122951 shutdown_connections 2024-03-24T00:47:21.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.976+0000 7fcb8fdf4640 1 -- 172.21.15.67:0/3041122951 >> 172.21.15.67:0/3041122951 conn(0x7fcb88072b60 msgr2=0x7fcb88072f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:21.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.977+0000 7fcb8fdf4640 1 -- 172.21.15.67:0/3041122951 shutdown_connections 2024-03-24T00:47:21.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.977+0000 7fcb8fdf4640 1 -- 172.21.15.67:0/3041122951 wait complete. 2024-03-24T00:47:21.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.978+0000 7fcb8fdf4640 1 Processor -- start 2024-03-24T00:47:21.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.978+0000 7fcb8fdf4640 1 -- start start 2024-03-24T00:47:21.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.979+0000 7fcb8fdf4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb74003f30 0x7fcb880c9fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:21.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.979+0000 7fcb8fdf4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcb881f6f90 0x7fcb881f73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:21.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.979+0000 7fcb8fdf4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb88155bc0 0x7fcb88156040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:21.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.979+0000 7fcb8fdf4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb880b3350 con 0x7fcb88155bc0 2024-03-24T00:47:21.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.979+0000 7fcb8fdf4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb880b31d0 con 0x7fcb74003f30 2024-03-24T00:47:21.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.979+0000 7fcb8fdf4640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb880b74d0 con 0x7fcb881f6f90 2024-03-24T00:47:21.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.979+0000 7fcb8e391640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb88155bc0 0x7fcb88156040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:21.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.979+0000 7fcb8e391640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb88155bc0 0x7fcb88156040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47306/0 (socket says 172.21.15.67:47306) 2024-03-24T00:47:21.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.979+0000 7fcb8e391640 1 -- 172.21.15.67:0/1484841732 learned_addr learned my addr 172.21.15.67:0/1484841732 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:21.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.980+0000 7fcb8db90640 1 --2- 172.21.15.67:0/1484841732 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb74003f30 0x7fcb880c9fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:21.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.980+0000 7fcb8d38f640 1 --2- 172.21.15.67:0/1484841732 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcb881f6f90 0x7fcb881f73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:21.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.980+0000 7fcb8e391640 1 -- 172.21.15.67:0/1484841732 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcb881f6f90 msgr2=0x7fcb881f73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:21.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.980+0000 7fcb8e391640 1 --2- 172.21.15.67:0/1484841732 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcb881f6f90 0x7fcb881f73f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:21.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.980+0000 7fcb8e391640 1 -- 172.21.15.67:0/1484841732 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb74003f30 msgr2=0x7fcb880c9fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:21.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.980+0000 7fcb8e391640 1 --2- 172.21.15.67:0/1484841732 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb74003f30 0x7fcb880c9fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:21.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.980+0000 7fcb8e391640 1 -- 172.21.15.67:0/1484841732 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb880ca510 con 0x7fcb88155bc0 2024-03-24T00:47:21.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.980+0000 7fcb8d38f640 1 --2- 172.21.15.67:0/1484841732 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcb881f6f90 0x7fcb881f73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:47:21.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.980+0000 7fcb8e391640 1 --2- 172.21.15.67:0/1484841732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb88155bc0 0x7fcb88156040 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7fcb880d2970 tx=0x7fcb84007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:21.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.980+0000 7fcb7e7fc640 1 -- 172.21.15.67:0/1484841732 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb8401a070 con 0x7fcb88155bc0 2024-03-24T00:47:21.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.980+0000 7fcb7e7fc640 1 -- 172.21.15.67:0/1484841732 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb8400f680 con 0x7fcb88155bc0 2024-03-24T00:47:21.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.980+0000 7fcb7e7fc640 1 -- 172.21.15.67:0/1484841732 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb8400fa50 con 0x7fcb88155bc0 2024-03-24T00:47:21.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.980+0000 7fcb8fdf4640 1 -- 172.21.15.67:0/1484841732 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb88156930 con 0x7fcb88155bc0 2024-03-24T00:47:21.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.980+0000 7fcb8fdf4640 1 -- 172.21.15.67:0/1484841732 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb880b4650 con 0x7fcb88155bc0 2024-03-24T00:47:21.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.982+0000 7fcb7e7fc640 1 -- 172.21.15.67:0/1484841732 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fcb8401d070 con 0x7fcb88155bc0 2024-03-24T00:47:21.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.983+0000 7fcb7e7fc640 1 --2- 172.21.15.67:0/1484841732 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcb6c06bbd0 0x7fcb6c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:21.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.983+0000 7fcb7e7fc640 1 -- 172.21.15.67:0/1484841732 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fcb8408a260 con 0x7fcb88155bc0 2024-03-24T00:47:21.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.983+0000 7fcb8db90640 1 --2- 172.21.15.67:0/1484841732 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcb6c06bbd0 0x7fcb6c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:21.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.983+0000 7fcb929e5740 1 Processor -- start 2024-03-24T00:47:21.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.983+0000 7fcb929e5740 1 -- start start 2024-03-24T00:47:21.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.984+0000 7fcb8db90640 1 --2- 172.21.15.67:0/1484841732 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcb6c06bbd0 0x7fcb6c06e090 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fcb80014430 tx=0x7fcb8000b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:21.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.984+0000 7fcb929e5740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56554c8adc40 0x56554c8d4770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:21.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.984+0000 7fcb929e5740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56554c8d4cb0 0x56554c8acc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:21.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.984+0000 7fcb929e5740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56554c8ae020 0x56554c8ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:21.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.984+0000 7fcb929e5740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x56554c811c20 con 0x56554c8d4cb0 2024-03-24T00:47:21.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.984+0000 7fcb929e5740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x56554c7baa40 con 0x56554c8ae020 2024-03-24T00:47:21.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.984+0000 7fcb929e5740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x56554c80dcc0 con 0x56554c8adc40 2024-03-24T00:47:21.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.984+0000 7fcb8d38f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56554c8d4cb0 0x56554c8acc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:21.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.984+0000 7fcb8d38f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56554c8d4cb0 0x56554c8acc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47308/0 (socket says 172.21.15.67:47308) 2024-03-24T00:47:21.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.984+0000 7fcb8d38f640 1 -- 172.21.15.67:0/3795779897 learned_addr learned my addr 172.21.15.67:0/3795779897 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:21.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.984+0000 7fcb8db90640 1 --2- 172.21.15.67:0/3795779897 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56554c8adc40 0x56554c8d4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:21.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.984+0000 7fcb8e391640 1 --2- 172.21.15.67:0/3795779897 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56554c8ae020 0x56554c8ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:21.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.984+0000 7fcb8d38f640 1 -- 172.21.15.67:0/3795779897 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56554c8adc40 msgr2=0x56554c8d4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:21.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.984+0000 7fcb8d38f640 1 --2- 172.21.15.67:0/3795779897 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56554c8adc40 0x56554c8d4770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:21.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.984+0000 7fcb8d38f640 1 -- 172.21.15.67:0/3795779897 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56554c8ae020 msgr2=0x56554c8ad1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:21.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.984+0000 7fcb8d38f640 1 --2- 172.21.15.67:0/3795779897 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56554c8ae020 0x56554c8ad1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:21.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.984+0000 7fcb8d38f640 1 -- 172.21.15.67:0/3795779897 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56554c7dcf90 con 0x56554c8d4cb0 2024-03-24T00:47:21.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.985+0000 7fcb8d38f640 1 --2- 172.21.15.67:0/3795779897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56554c8d4cb0 0x56554c8acc80 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fcb78000c00 tx=0x7fcb7802e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb3f140ee7834d30 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:21.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.985+0000 7fcb6b7fe640 1 -- 172.21.15.67:0/3795779897 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb78044070 con 0x56554c8d4cb0 2024-03-24T00:47:21.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.985+0000 7fcb6b7fe640 1 -- 172.21.15.67:0/3795779897 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb78005990 con 0x56554c8d4cb0 2024-03-24T00:47:21.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.985+0000 7fcb6b7fe640 1 -- 172.21.15.67:0/3795779897 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb78005d40 con 0x56554c8d4cb0 2024-03-24T00:47:21.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.985+0000 7fcb929e5740 1 -- 172.21.15.67:0/3795779897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56554c8d4cb0 msgr2=0x56554c8acc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:21.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.985+0000 7fcb929e5740 1 --2- 172.21.15.67:0/3795779897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56554c8d4cb0 0x56554c8acc80 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fcb78000c00 tx=0x7fcb7802e000 comp rx=0 tx=0).stop 2024-03-24T00:47:21.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.985+0000 7fcb929e5740 1 -- 172.21.15.67:0/3795779897 shutdown_connections 2024-03-24T00:47:21.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.985+0000 7fcb929e5740 1 --2- 172.21.15.67:0/3795779897 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56554c8ae020 0x56554c8ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:21.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.985+0000 7fcb929e5740 1 --2- 172.21.15.67:0/3795779897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56554c8d4cb0 0x56554c8acc80 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:21.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.985+0000 7fcb929e5740 1 --2- 172.21.15.67:0/3795779897 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56554c8adc40 0x56554c8d4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:21.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.985+0000 7fcb929e5740 1 -- 172.21.15.67:0/3795779897 >> 172.21.15.67:0/3795779897 conn(0x56554c874a60 msgr2=0x56554c8f2960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:21.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.985+0000 7fcb929e5740 1 -- 172.21.15.67:0/3795779897 shutdown_connections 2024-03-24T00:47:21.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.985+0000 7fcb929e5740 1 -- 172.21.15.67:0/3795779897 wait complete. 2024-03-24T00:47:21.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.986+0000 7fcb929e5740 1 -- start start 2024-03-24T00:47:21.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.986+0000 7fcb929e5740 1 Processor -- start 2024-03-24T00:47:21.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.987+0000 7fcb929e5740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56554c8adc40 0x56554c8f59c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:21.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.987+0000 7fcb929e5740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56554c8ae020 0x56554c8f5f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:21.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.987+0000 7fcb929e5740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56554c8cb990 0x56554c8cdf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:21.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.987+0000 7fcb929e5740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x56554c8d42e0 con 0x56554c8adc40 2024-03-24T00:47:21.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.987+0000 7fcb929e5740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x56554c7cde90 con 0x56554c8cb990 2024-03-24T00:47:21.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.987+0000 7fcb929e5740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x56554c8cd060 con 0x56554c8ae020 2024-03-24T00:47:21.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.987+0000 7fcb929e5740 10 client.0 _ll_register_callbacks cb 0x56554c525ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:47:21.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.987+0000 7fcb8db90640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56554c8adc40 0x56554c8f59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:21.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.987+0000 7fcb8db90640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56554c8adc40 0x56554c8f59c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47324/0 (socket says 172.21.15.67:47324) 2024-03-24T00:47:21.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.987+0000 7fcb8db90640 1 -- 172.21.15.67:0/1546263835 learned_addr learned my addr 172.21.15.67:0/1546263835 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:21.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.987+0000 7fcb8d38f640 1 --2- 172.21.15.67:0/1546263835 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56554c8ae020 0x56554c8f5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:21.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.987+0000 7fcb8e391640 1 --2- 172.21.15.67:0/1546263835 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56554c8cb990 0x56554c8cdf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:21.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.987+0000 7fcb8db90640 1 -- 172.21.15.67:0/1546263835 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56554c8ae020 msgr2=0x56554c8f5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:21.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.987+0000 7fcb8db90640 1 --2- 172.21.15.67:0/1546263835 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56554c8ae020 0x56554c8f5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:21.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.987+0000 7fcb8db90640 1 -- 172.21.15.67:0/1546263835 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56554c8cb990 msgr2=0x56554c8cdf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:21.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.987+0000 7fcb8db90640 1 --2- 172.21.15.67:0/1546263835 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56554c8cb990 0x56554c8cdf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:21.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.987+0000 7fcb8db90640 1 -- 172.21.15.67:0/1546263835 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56554c7da9d0 con 0x56554c8adc40 2024-03-24T00:47:21.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.987+0000 7fcb8e391640 1 --2- 172.21.15.67:0/1546263835 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56554c8cb990 0x56554c8cdf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:47:21.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.988+0000 7fcb8db90640 1 --2- 172.21.15.67:0/1546263835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56554c8adc40 0x56554c8f59c0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7fcb80016f60 tx=0x7fcb80004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b042b4571596ff4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:21.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.988+0000 7fcb68ff9640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:47:21.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.988+0000 7fcb68ff9640 1 -- 172.21.15.67:0/1546263835 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb80003b90 con 0x56554c8adc40 2024-03-24T00:47:21.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.988+0000 7fcb68ff9640 1 -- 172.21.15.67:0/1546263835 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fcb80003d30 con 0x56554c8adc40 2024-03-24T00:47:21.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.988+0000 7fcb929e5740 1 -- 172.21.15.67:0/1546263835 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcb78005990 con 0x56554c8adc40 2024-03-24T00:47:21.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.988+0000 7fcb68ff9640 1 -- 172.21.15.67:0/1546263835 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcb80034050 con 0x56554c8adc40 2024-03-24T00:47:21.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.988+0000 7fcb929e5740 10 client.8694 fetch_fsmap learned FSMap version 72 2024-03-24T00:47:21.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.988+0000 7fcb929e5740 1 -- 172.21.15.67:0/1546263835 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=72,monmap=2+}) v3 -- 0x56554c5262d0 con 0x56554c8adc40 2024-03-24T00:47:21.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.988+0000 7fcb68ff9640 1 -- 172.21.15.67:0/1546263835 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb78005990 con 0x56554c8adc40 2024-03-24T00:47:21.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.988+0000 7fcb68ff9640 1 -- 172.21.15.67:0/1546263835 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 72) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fcb80004020 con 0x56554c8adc40 2024-03-24T00:47:21.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.988+0000 7fcb929e5740 10 client.8694 fetch_fsmap finished waiting for FSMap version 72 2024-03-24T00:47:21.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.988+0000 7fcb929e5740 10 client.8694 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T00:47:21.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.988+0000 7fcb929e5740 20 client.8694 populate_metadata read hostname 'smithi067' 2024-03-24T00:47:21.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.988+0000 7fcb929e5740 1 --2- 172.21.15.67:0/1546263835 >> [v2:172.21.15.67:6832/1826974843,v1:172.21.15.67:6833/1826974843] conn(0x56554c8d8cf0 0x56554c8d9110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:21.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.988+0000 7fcb929e5740 4 client.8694 mds_command: new command op to 8541 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:47:21.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.988+0000 7fcb929e5740 1 -- 172.21.15.67:0/1546263835 --> [v2:172.21.15.67:6832/1826974843,v1:172.21.15.67:6833/1826974843] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56554c81f160 con 0x56554c8d8cf0 2024-03-24T00:47:21.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.988+0000 7fcb8d38f640 1 --2- 172.21.15.67:0/1546263835 >> [v2:172.21.15.67:6832/1826974843,v1:172.21.15.67:6833/1826974843] conn(0x56554c8d8cf0 0x56554c8d9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:21.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.989+0000 7fcb68ff9640 10 client.8694 ms_handle_connect on v2:172.21.15.67:6832/1826974843 2024-03-24T00:47:21.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.989+0000 7fcb8d38f640 1 --2- 172.21.15.67:0/1546263835 >> [v2:172.21.15.67:6832/1826974843,v1:172.21.15.67:6833/1826974843] conn(0x56554c8d8cf0 0x56554c8d9110 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=843118c8d5e21698 server_cookie=4e540cf026ebe451 in_seq=0 out_seq=0 2024-03-24T00:47:21.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.989+0000 7fcb68ff9640 1 -- 172.21.15.67:0/1546263835 <== mds.0 v2:172.21.15.67:6832/1826974843 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56554c81f160 con 0x56554c8d8cf0 2024-03-24T00:47:21.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.989+0000 7fcb68ff9640 10 client.8694 handle_command_reply: tid=0 2024-03-24T00:47:21.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.989+0000 7fcb929e5740 1 client.8694 shutdown 2024-03-24T00:47:21.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.990+0000 7fcb929e5740 1 -- 172.21.15.67:0/1546263835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56554c8adc40 msgr2=0x56554c8f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:21.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.990+0000 7fcb929e5740 1 --2- 172.21.15.67:0/1546263835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56554c8adc40 0x56554c8f59c0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7fcb80016f60 tx=0x7fcb80004ab0 comp rx=0 tx=0).stop 2024-03-24T00:47:21.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.990+0000 7fcb929e5740 1 -- 172.21.15.67:0/1546263835 shutdown_connections 2024-03-24T00:47:21.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.990+0000 7fcb929e5740 1 --2- 172.21.15.67:0/1546263835 >> [v2:172.21.15.67:6832/1826974843,v1:172.21.15.67:6833/1826974843] conn(0x56554c8d8cf0 0x56554c8d9110 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:21.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.990+0000 7fcb929e5740 1 --2- 172.21.15.67:0/1546263835 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56554c8cb990 0x56554c8cdf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:21.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.990+0000 7fcb929e5740 1 --2- 172.21.15.67:0/1546263835 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56554c8ae020 0x56554c8f5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:21.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.990+0000 7fcb68ff9640 0 client.8694 ms_handle_reset on v2:172.21.15.67:6832/1826974843 2024-03-24T00:47:21.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.990+0000 7fcb929e5740 1 --2- 172.21.15.67:0/1546263835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56554c8adc40 0x56554c8f59c0 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:21.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.990+0000 7fcb929e5740 1 -- 172.21.15.67:0/1546263835 >> 172.21.15.67:0/1546263835 conn(0x56554c874a60 msgr2=0x56554c8f3290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:21.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.990+0000 7fcb929e5740 1 -- 172.21.15.67:0/1546263835 shutdown_connections 2024-03-24T00:47:21.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.990+0000 7fcb929e5740 1 -- 172.21.15.67:0/1546263835 wait complete. 2024-03-24T00:47:21.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:21.990+0000 7fcb929e5740 20 client.8694 trim_cache size 0 max 16384 2024-03-24T00:47:22.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.023+0000 7fcb929e5740 1 Processor -- start 2024-03-24T00:47:22.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.023+0000 7fcb929e5740 1 -- start start 2024-03-24T00:47:22.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.024+0000 7fcb929e5740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56554c8adc40 0x56554c8a6a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:22.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.024+0000 7fcb929e5740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56554c8ae020 0x56554c901d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:22.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.024+0000 7fcb929e5740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56554c8cb990 0x56554c95cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:22.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.024+0000 7fcb929e5740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb88200020 con 0x56554c8ae020 2024-03-24T00:47:22.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.024+0000 7fcb929e5740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb881ff420 con 0x56554c8cb990 2024-03-24T00:47:22.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.024+0000 7fcb929e5740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb881ffa20 con 0x56554c8adc40 2024-03-24T00:47:22.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.024+0000 7fcb8d38f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56554c8ae020 0x56554c901d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:22.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.024+0000 7fcb8d38f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56554c8ae020 0x56554c901d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47340/0 (socket says 172.21.15.67:47340) 2024-03-24T00:47:22.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.024+0000 7fcb8d38f640 1 -- 172.21.15.67:0/3099366079 learned_addr learned my addr 172.21.15.67:0/3099366079 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:22.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.024+0000 7fcb8db90640 1 --2- 172.21.15.67:0/3099366079 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56554c8adc40 0x56554c8a6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:22.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.024+0000 7fcb8d38f640 1 -- 172.21.15.67:0/3099366079 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56554c8adc40 msgr2=0x56554c8a6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:22.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.024+0000 7fcb8d38f640 1 --2- 172.21.15.67:0/3099366079 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56554c8adc40 0x56554c8a6a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:22.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.024+0000 7fcb8d38f640 1 -- 172.21.15.67:0/3099366079 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56554c8cb990 msgr2=0x56554c95cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:22.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.024+0000 7fcb8e391640 1 --2- 172.21.15.67:0/3099366079 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56554c8cb990 0x56554c95cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:22.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.024+0000 7fcb8d38f640 1 --2- 172.21.15.67:0/3099366079 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56554c8cb990 0x56554c95cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:22.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.024+0000 7fcb8d38f640 1 -- 172.21.15.67:0/3099366079 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56554c5f06b0 con 0x56554c8ae020 2024-03-24T00:47:22.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.024+0000 7fcb8d38f640 1 --2- 172.21.15.67:0/3099366079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56554c8ae020 0x56554c901d50 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7fcb78009bd0 tx=0x7fcb78008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d732e90541f89952 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:22.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.025+0000 7fcb6affd640 1 -- 172.21.15.67:0/3099366079 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb78044040 con 0x56554c8ae020 2024-03-24T00:47:22.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.025+0000 7fcb6affd640 1 -- 172.21.15.67:0/3099366079 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb78056280 con 0x56554c8ae020 2024-03-24T00:47:22.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.025+0000 7fcb6affd640 1 -- 172.21.15.67:0/3099366079 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb7802cab0 con 0x56554c8ae020 2024-03-24T00:47:22.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.025+0000 7fcb929e5740 1 -- 172.21.15.67:0/3099366079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56554c8ae020 msgr2=0x56554c901d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:22.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.025+0000 7fcb929e5740 1 --2- 172.21.15.67:0/3099366079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56554c8ae020 0x56554c901d50 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7fcb78009bd0 tx=0x7fcb78008030 comp rx=0 tx=0).stop 2024-03-24T00:47:22.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.025+0000 7fcb929e5740 1 -- 172.21.15.67:0/3099366079 shutdown_connections 2024-03-24T00:47:22.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.025+0000 7fcb929e5740 1 --2- 172.21.15.67:0/3099366079 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56554c8cb990 0x56554c95cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:22.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.025+0000 7fcb929e5740 1 --2- 172.21.15.67:0/3099366079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56554c8ae020 0x56554c901d50 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:22.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.025+0000 7fcb929e5740 1 --2- 172.21.15.67:0/3099366079 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56554c8adc40 0x56554c8a6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:22.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.025+0000 7fcb929e5740 1 -- 172.21.15.67:0/3099366079 >> 172.21.15.67:0/3099366079 conn(0x56554c874a60 msgr2=0x56554c8a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:22.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.025+0000 7fcb929e5740 1 -- 172.21.15.67:0/3099366079 shutdown_connections 2024-03-24T00:47:22.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.025+0000 7fcb929e5740 1 -- 172.21.15.67:0/3099366079 wait complete. 2024-03-24T00:47:22.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.026+0000 7fcb929e5740 1 -- start start 2024-03-24T00:47:22.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.026+0000 7fcb929e5740 1 Processor -- start 2024-03-24T00:47:22.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.027+0000 7fcb929e5740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56554c8adc40 0x56554c8f20b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:22.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.027+0000 7fcb929e5740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56554c8ae020 0x56554c946df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:22.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.027+0000 7fcb929e5740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56554c8cb990 0x56554c904e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:22.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.027+0000 7fcb929e5740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x56554c8cc290 con 0x56554c8cb990 2024-03-24T00:47:22.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.027+0000 7fcb929e5740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x56554c8da750 con 0x56554c8ae020 2024-03-24T00:47:22.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.027+0000 7fcb929e5740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x56554c8da320 con 0x56554c8adc40 2024-03-24T00:47:22.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.027+0000 7fcb929e5740 10 client.0 _ll_register_callbacks cb 0x7fcb80009d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:47:22.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.027+0000 7fcb8e391640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56554c8cb990 0x56554c904e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:22.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.027+0000 7fcb8e391640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56554c8cb990 0x56554c904e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47356/0 (socket says 172.21.15.67:47356) 2024-03-24T00:47:22.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.027+0000 7fcb8e391640 1 -- 172.21.15.67:0/2554474480 learned_addr learned my addr 172.21.15.67:0/2554474480 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:22.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.027+0000 7fcb8db90640 1 --2- 172.21.15.67:0/2554474480 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56554c8adc40 0x56554c8f20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:22.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.027+0000 7fcb8d38f640 1 --2- 172.21.15.67:0/2554474480 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56554c8ae020 0x56554c946df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:22.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.027+0000 7fcb8e391640 1 -- 172.21.15.67:0/2554474480 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56554c8adc40 msgr2=0x56554c8f20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:22.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.027+0000 7fcb8e391640 1 --2- 172.21.15.67:0/2554474480 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56554c8adc40 0x56554c8f20b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:22.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.027+0000 7fcb8e391640 1 -- 172.21.15.67:0/2554474480 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56554c8ae020 msgr2=0x56554c946df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:22.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.027+0000 7fcb8e391640 1 --2- 172.21.15.67:0/2554474480 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56554c8ae020 0x56554c946df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:22.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.027+0000 7fcb8e391640 1 -- 172.21.15.67:0/2554474480 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56554c8f59a0 con 0x56554c8cb990 2024-03-24T00:47:22.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.027+0000 7fcb8d38f640 1 --2- 172.21.15.67:0/2554474480 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56554c8ae020 0x56554c946df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:47:22.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.027+0000 7fcb8e391640 1 --2- 172.21.15.67:0/2554474480 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56554c8cb990 0x56554c904e10 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7fcb84054a10 tx=0x7fcb84058600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f56a7e9d6f96ffe6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:22.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.027+0000 7fcb68ff9640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:47:22.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.027+0000 7fcb68ff9640 1 -- 172.21.15.67:0/2554474480 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb840703d0 con 0x56554c8cb990 2024-03-24T00:47:22.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.027+0000 7fcb68ff9640 1 -- 172.21.15.67:0/2554474480 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fcb8400fbf0 con 0x56554c8cb990 2024-03-24T00:47:22.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.027+0000 7fcb929e5740 1 -- 172.21.15.67:0/2554474480 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcb78056280 con 0x56554c8cb990 2024-03-24T00:47:22.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.028+0000 7fcb68ff9640 1 -- 172.21.15.67:0/2554474480 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcb8400fdf0 con 0x56554c8cb990 2024-03-24T00:47:22.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.028+0000 7fcb929e5740 10 client.8700 fetch_fsmap learned FSMap version 72 2024-03-24T00:47:22.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.028+0000 7fcb929e5740 1 -- 172.21.15.67:0/2554474480 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=72,monmap=2+}) v3 -- 0x56554c947330 con 0x56554c8cb990 2024-03-24T00:47:22.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.028+0000 7fcb68ff9640 1 -- 172.21.15.67:0/2554474480 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb78056280 con 0x56554c8cb990 2024-03-24T00:47:22.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.028+0000 7fcb68ff9640 1 -- 172.21.15.67:0/2554474480 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 72) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fcb840880d0 con 0x56554c8cb990 2024-03-24T00:47:22.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.028+0000 7fcb929e5740 10 client.8700 fetch_fsmap finished waiting for FSMap version 72 2024-03-24T00:47:22.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.028+0000 7fcb929e5740 10 client.8700 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T00:47:22.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.028+0000 7fcb929e5740 20 client.8700 populate_metadata read hostname 'smithi067' 2024-03-24T00:47:22.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.028+0000 7fcb929e5740 1 --2- 172.21.15.67:0/2554474480 >> [v2:172.21.15.67:6832/1826974843,v1:172.21.15.67:6833/1826974843] conn(0x56554c8d7c20 0x56554c8e82f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:22.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.028+0000 7fcb929e5740 4 client.8700 mds_command: new command op to 8541 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:47:22.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.028+0000 7fcb929e5740 1 -- 172.21.15.67:0/2554474480 --> [v2:172.21.15.67:6832/1826974843,v1:172.21.15.67:6833/1826974843] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56554c8194b0 con 0x56554c8d7c20 2024-03-24T00:47:22.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.028+0000 7fcb8db90640 1 --2- 172.21.15.67:0/2554474480 >> [v2:172.21.15.67:6832/1826974843,v1:172.21.15.67:6833/1826974843] conn(0x56554c8d7c20 0x56554c8e82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:22.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.028+0000 7fcb68ff9640 10 client.8700 ms_handle_connect on v2:172.21.15.67:6832/1826974843 2024-03-24T00:47:22.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.028+0000 7fcb8db90640 1 --2- 172.21.15.67:0/2554474480 >> [v2:172.21.15.67:6832/1826974843,v1:172.21.15.67:6833/1826974843] conn(0x56554c8d7c20 0x56554c8e82f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f70f9b3466f28a3c server_cookie=1a1f00c76a5cac9d in_seq=0 out_seq=0 2024-03-24T00:47:22.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.029+0000 7fcb68ff9640 1 -- 172.21.15.67:0/2554474480 <== mds.0 v2:172.21.15.67:6832/1826974843 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x56554c8194b0 con 0x56554c8d7c20 2024-03-24T00:47:22.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.029+0000 7fcb68ff9640 10 client.8700 handle_command_reply: tid=0 2024-03-24T00:47:22.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.029+0000 7fcb929e5740 1 client.8700 shutdown 2024-03-24T00:47:22.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.029+0000 7fcb929e5740 1 -- 172.21.15.67:0/2554474480 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56554c8cb990 msgr2=0x56554c904e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:22.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.029+0000 7fcb929e5740 1 --2- 172.21.15.67:0/2554474480 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56554c8cb990 0x56554c904e10 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7fcb84054a10 tx=0x7fcb84058600 comp rx=0 tx=0).stop 2024-03-24T00:47:22.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.029+0000 7fcb929e5740 1 -- 172.21.15.67:0/2554474480 shutdown_connections 2024-03-24T00:47:22.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.029+0000 7fcb929e5740 1 --2- 172.21.15.67:0/2554474480 >> [v2:172.21.15.67:6832/1826974843,v1:172.21.15.67:6833/1826974843] conn(0x56554c8d7c20 0x56554c8e82f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:22.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.029+0000 7fcb929e5740 1 --2- 172.21.15.67:0/2554474480 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56554c8cb990 0x56554c904e10 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:22.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.029+0000 7fcb929e5740 1 --2- 172.21.15.67:0/2554474480 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56554c8ae020 0x56554c946df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:22.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.029+0000 7fcb68ff9640 0 client.8700 ms_handle_reset on v2:172.21.15.67:6832/1826974843 2024-03-24T00:47:22.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.029+0000 7fcb929e5740 1 --2- 172.21.15.67:0/2554474480 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56554c8adc40 0x56554c8f20b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:22.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.029+0000 7fcb929e5740 1 -- 172.21.15.67:0/2554474480 >> 172.21.15.67:0/2554474480 conn(0x56554c874a60 msgr2=0x56554c8a6a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:22.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.029+0000 7fcb929e5740 1 -- 172.21.15.67:0/2554474480 shutdown_connections 2024-03-24T00:47:22.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.029+0000 7fcb929e5740 1 -- 172.21.15.67:0/2554474480 wait complete. 2024-03-24T00:47:22.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.029+0000 7fcb929e5740 20 client.8700 trim_cache size 0 max 16384 2024-03-24T00:47:22.035 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:47:22.035 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "de1d0178-808e-49ca-a986-e297e523b976", 2024-03-24T00:47:22.035 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-03-24T00:47:22.035 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 8541, 2024-03-24T00:47:22.036 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-03-24T00:47:22.036 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-03-24T00:47:22.036 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-03-24T00:47:22.036 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 4.9002239080000001, 2024-03-24T00:47:22.036 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 72, 2024-03-24T00:47:22.036 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 105, 2024-03-24T00:47:22.036 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 105, 2024-03-24T00:47:22.036 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 9.9165267499999992 2024-03-24T00:47:22.036 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:47:22.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.030+0000 7fcb8fdf4640 1 -- 172.21.15.67:0/1484841732 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcb6c06bbd0 msgr2=0x7fcb6c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:22.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.030+0000 7fcb8fdf4640 1 --2- 172.21.15.67:0/1484841732 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcb6c06bbd0 0x7fcb6c06e090 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fcb80014430 tx=0x7fcb8000b040 comp rx=0 tx=0).stop 2024-03-24T00:47:22.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.030+0000 7fcb8fdf4640 1 -- 172.21.15.67:0/1484841732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb88155bc0 msgr2=0x7fcb88156040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:22.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.030+0000 7fcb8fdf4640 1 --2- 172.21.15.67:0/1484841732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb88155bc0 0x7fcb88156040 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7fcb880d2970 tx=0x7fcb84007430 comp rx=0 tx=0).stop 2024-03-24T00:47:22.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.030+0000 7fcb8fdf4640 1 -- 172.21.15.67:0/1484841732 shutdown_connections 2024-03-24T00:47:22.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.030+0000 7fcb8fdf4640 1 --2- 172.21.15.67:0/1484841732 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcb6c06bbd0 0x7fcb6c06e090 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:22.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.030+0000 7fcb8fdf4640 1 --2- 172.21.15.67:0/1484841732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb88155bc0 0x7fcb88156040 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:22.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.030+0000 7fcb8fdf4640 1 --2- 172.21.15.67:0/1484841732 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcb881f6f90 0x7fcb881f73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:22.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.030+0000 7fcb8fdf4640 1 --2- 172.21.15.67:0/1484841732 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb74003f30 0x7fcb880c9fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:22.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.030+0000 7fcb8fdf4640 1 -- 172.21.15.67:0/1484841732 >> 172.21.15.67:0/1484841732 conn(0x7fcb88072b60 msgr2=0x7fcb88072f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:22.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.030+0000 7fcb8fdf4640 1 -- 172.21.15.67:0/1484841732 shutdown_connections 2024-03-24T00:47:22.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.030+0000 7fcb8fdf4640 1 -- 172.21.15.67:0/1484841732 wait complete. 2024-03-24T00:47:22.040 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T00:47:22.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.142+0000 7fb7c63a9640 1 Processor -- start 2024-03-24T00:47:22.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.142+0000 7fb7c63a9640 1 -- start start 2024-03-24T00:47:22.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.143+0000 7fb7c63a9640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb7c00ce930 0x7fb7c0166b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:22.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.143+0000 7fb7c63a9640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7c00ccfe0 con 0x7fb7c015bf70 2024-03-24T00:47:22.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.143+0000 7fb7c63a9640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7c00cd2c0 con 0x7fb7c00cfc70 2024-03-24T00:47:22.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.143+0000 7fb7c63a9640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c00ccda0 con 0x7fb7c00ce930 2024-03-24T00:47:22.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.143+0000 7fb7bffff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb7c015bf70 0x7fb7c00cc690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48324/0 (socket says 172.21.15.67:48324) 2024-03-24T00:47:22.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.143+0000 7fb7bffff640 1 -- 172.21.15.67:0/3158542854 learned_addr learned my addr 172.21.15.67:0/3158542854 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:47:22.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.143+0000 7fb7c4946640 1 --2- 172.21.15.67:0/3158542854 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb7c00ce930 0x7fb7c0166b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:22.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.143+0000 7fb7beffd640 1 -- 172.21.15.67:0/3158542854 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1537183527 0 0) 0x7fb7c00ccfe0 con 0x7fb7c015bf70 2024-03-24T00:47:22.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.143+0000 7fb7beffd640 1 -- 172.21.15.67:0/3158542854 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7a8003520 con 0x7fb7c015bf70 2024-03-24T00:47:22.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.143+0000 7fb7beffd640 1 -- 172.21.15.67:0/3158542854 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 350644209 0 0) 0x7fb7c00cd2c0 con 0x7fb7c00cfc70 2024-03-24T00:47:22.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.143+0000 7fb7beffd640 1 -- 172.21.15.67:0/3158542854 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7c00ccfe0 con 0x7fb7c00cfc70 2024-03-24T00:47:22.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.144+0000 7fb7beffd640 1 -- 172.21.15.67:0/3158542854 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3382126387 0 0) 0x7fb7a8003520 con 0x7fb7c015bf70 2024-03-24T00:47:22.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.144+0000 7fb7c4946640 1 -- 172.21.15.67:0/3158542854 >> v1:172.21.15.72:6789/0 conn(0x7fb7c00cfc70 legacy=0x7fb7c00ce220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:22.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.144+0000 7fb7c4946640 1 -- 172.21.15.67:0/3158542854 >> v1:172.21.15.67:6789/0 conn(0x7fb7c015bf70 legacy=0x7fb7c00cc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:22.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.144+0000 7fb7c4946640 1 -- 172.21.15.67:0/3158542854 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c00cd5a0 con 0x7fb7c00ce930 2024-03-24T00:47:22.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.145+0000 7fb7c4946640 1 --2- 172.21.15.67:0/3158542854 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb7c00ce930 0x7fb7c0166b10 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb7b8009dc0 tx=0x7fb7b802bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75f435618fea13eb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:22.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.145+0000 7fb7beffd640 1 -- 172.21.15.67:0/3158542854 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b8039070 con 0x7fb7c00ce930 2024-03-24T00:47:22.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.145+0000 7fb7beffd640 1 -- 172.21.15.67:0/3158542854 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7b802c910 con 0x7fb7c00ce930 2024-03-24T00:47:22.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.145+0000 7fb7beffd640 1 -- 172.21.15.67:0/3158542854 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b4001670 con 0x7fb7c00ce930 2024-03-24T00:47:22.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.145+0000 7fb7c63a9640 1 -- 172.21.15.67:0/3158542854 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb7c00ce930 msgr2=0x7fb7c0166b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:22.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.145+0000 7fb7c63a9640 1 --2- 172.21.15.67:0/3158542854 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb7c00ce930 0x7fb7c0166b10 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb7b8009dc0 tx=0x7fb7b802bde0 comp rx=0 tx=0).stop 2024-03-24T00:47:22.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.145+0000 7fb7c63a9640 1 -- 172.21.15.67:0/3158542854 shutdown_connections 2024-03-24T00:47:22.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.145+0000 7fb7c63a9640 1 --2- 172.21.15.67:0/3158542854 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb7c00ce930 0x7fb7c0166b10 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:22.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.145+0000 7fb7c63a9640 1 -- 172.21.15.67:0/3158542854 >> 172.21.15.67:0/3158542854 conn(0x7fb7c00d1bf0 msgr2=0x7fb7c00d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:22.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.146+0000 7fb7c63a9640 1 -- 172.21.15.67:0/3158542854 shutdown_connections 2024-03-24T00:47:22.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.146+0000 7fb7c63a9640 1 -- 172.21.15.67:0/3158542854 wait complete. 2024-03-24T00:47:22.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.147+0000 7fb7c63a9640 1 Processor -- start 2024-03-24T00:47:22.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.147+0000 7fb7c63a9640 1 -- start start 2024-03-24T00:47:22.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.148+0000 7fb7c63a9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7c00ce930 0x7fb7c0151010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:22.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.148+0000 7fb7c63a9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7c00cfc70 0x7fb7c015d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:22.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.148+0000 7fb7c63a9640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb7c015bf70 0x7fb7c015dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:22.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.148+0000 7fb7c63a9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c00b2ce0 con 0x7fb7c00cfc70 2024-03-24T00:47:22.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.148+0000 7fb7c63a9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c015c350 con 0x7fb7c00ce930 2024-03-24T00:47:22.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.148+0000 7fb7c63a9640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c00b2e60 con 0x7fb7c015bf70 2024-03-24T00:47:22.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.148+0000 7fb7bf7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7c00cfc70 0x7fb7c015d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:22.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.148+0000 7fb7bf7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7c00cfc70 0x7fb7c015d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47364/0 (socket says 172.21.15.67:47364) 2024-03-24T00:47:22.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.148+0000 7fb7bf7fe640 1 -- 172.21.15.67:0/1185048590 learned_addr learned my addr 172.21.15.67:0/1185048590 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:22.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.148+0000 7fb7c4946640 1 --2- 172.21.15.67:0/1185048590 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb7c015bf70 0x7fb7c015dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:22.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.148+0000 7fb7bffff640 1 --2- 172.21.15.67:0/1185048590 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7c00ce930 0x7fb7c0151010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:22.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.148+0000 7fb7bf7fe640 1 -- 172.21.15.67:0/1185048590 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb7c015bf70 msgr2=0x7fb7c015dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:22.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.148+0000 7fb7bf7fe640 1 --2- 172.21.15.67:0/1185048590 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb7c015bf70 0x7fb7c015dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:22.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.148+0000 7fb7bf7fe640 1 -- 172.21.15.67:0/1185048590 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7c00ce930 msgr2=0x7fb7c0151010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:22.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.148+0000 7fb7bf7fe640 1 --2- 172.21.15.67:0/1185048590 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7c00ce930 0x7fb7c0151010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:22.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.148+0000 7fb7bf7fe640 1 -- 172.21.15.67:0/1185048590 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c0153f10 con 0x7fb7c00cfc70 2024-03-24T00:47:22.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.148+0000 7fb7bffff640 1 --2- 172.21.15.67:0/1185048590 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7c00ce930 0x7fb7c0151010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:47:22.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.148+0000 7fb7bf7fe640 1 --2- 172.21.15.67:0/1185048590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7c00cfc70 0x7fb7c015d7b0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7fb7b000c400 tx=0x7fb7b000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:22.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.148+0000 7fb7bcff9640 1 -- 172.21.15.67:0/1185048590 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b0017070 con 0x7fb7c00cfc70 2024-03-24T00:47:22.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.148+0000 7fb7bcff9640 1 -- 172.21.15.67:0/1185048590 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7b00052d0 con 0x7fb7c00cfc70 2024-03-24T00:47:22.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.148+0000 7fb7c63a9640 1 -- 172.21.15.67:0/1185048590 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7c01541f0 con 0x7fb7c00cfc70 2024-03-24T00:47:22.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.148+0000 7fb7bcff9640 1 -- 172.21.15.67:0/1185048590 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b0003810 con 0x7fb7c00cfc70 2024-03-24T00:47:22.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.148+0000 7fb7c63a9640 1 -- 172.21.15.67:0/1185048590 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7c015eac0 con 0x7fb7c00cfc70 2024-03-24T00:47:22.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.149+0000 7fb7c63a9640 1 -- 172.21.15.67:0/1185048590 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7c004f400 con 0x7fb7c00cfc70 2024-03-24T00:47:22.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.150+0000 7fb7bcff9640 1 -- 172.21.15.67:0/1185048590 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb7b001a070 con 0x7fb7c00cfc70 2024-03-24T00:47:22.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.150+0000 7fb7bcff9640 1 --2- 172.21.15.67:0/1185048590 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb7ac06bbd0 0x7fb7ac06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:22.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.150+0000 7fb7bcff9640 1 -- 172.21.15.67:0/1185048590 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb7b0089fa0 con 0x7fb7c00cfc70 2024-03-24T00:47:22.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.151+0000 7fb7bffff640 1 --2- 172.21.15.67:0/1185048590 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb7ac06bbd0 0x7fb7ac06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:22.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.151+0000 7fb7bffff640 1 --2- 172.21.15.67:0/1185048590 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb7ac06bbd0 0x7fb7ac06e090 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fb7b4001380 tx=0x7fb7b4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:22.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.155+0000 7fb7bcff9640 1 -- 172.21.15.67:0/1185048590 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb7b0058990 con 0x7fb7c00cfc70 2024-03-24T00:47:22.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.378+0000 7fb7c63a9640 1 -- 172.21.15.67:0/1185048590 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fb7c008bc80 con 0x7fb7c00cfc70 2024-03-24T00:47:22.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.378+0000 7fb7bcff9640 1 -- 172.21.15.67:0/1185048590 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v72) v1 ==== 53+0+73 (secure 0 0 0) 0x7fb7b0058290 con 0x7fb7c00cfc70 2024-03-24T00:47:22.378 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-24T00:47:22.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.380+0000 7fb7c63a9640 1 -- 172.21.15.67:0/1185048590 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb7ac06bbd0 msgr2=0x7fb7ac06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:22.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.380+0000 7fb7c63a9640 1 --2- 172.21.15.67:0/1185048590 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb7ac06bbd0 0x7fb7ac06e090 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fb7b4001380 tx=0x7fb7b4009040 comp rx=0 tx=0).stop 2024-03-24T00:47:22.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.380+0000 7fb7c63a9640 1 -- 172.21.15.67:0/1185048590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7c00cfc70 msgr2=0x7fb7c015d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:22.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.380+0000 7fb7c63a9640 1 --2- 172.21.15.67:0/1185048590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7c00cfc70 0x7fb7c015d7b0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7fb7b000c400 tx=0x7fb7b000c8c0 comp rx=0 tx=0).stop 2024-03-24T00:47:22.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.380+0000 7fb7c63a9640 1 -- 172.21.15.67:0/1185048590 shutdown_connections 2024-03-24T00:47:22.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.380+0000 7fb7c63a9640 1 --2- 172.21.15.67:0/1185048590 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb7ac06bbd0 0x7fb7ac06e090 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:22.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.380+0000 7fb7c63a9640 1 --2- 172.21.15.67:0/1185048590 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb7c015bf70 0x7fb7c015dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:22.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.380+0000 7fb7c63a9640 1 --2- 172.21.15.67:0/1185048590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7c00cfc70 0x7fb7c015d7b0 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:22.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.380+0000 7fb7c63a9640 1 --2- 172.21.15.67:0/1185048590 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7c00ce930 0x7fb7c0151010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:22.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.380+0000 7fb7c63a9640 1 -- 172.21.15.67:0/1185048590 >> 172.21.15.67:0/1185048590 conn(0x7fb7c00d1bf0 msgr2=0x7fb7c015b140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:22.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.381+0000 7fb7c63a9640 1 -- 172.21.15.67:0/1185048590 shutdown_connections 2024-03-24T00:47:22.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:22.381+0000 7fb7c63a9640 1 -- 172.21.15.67:0/1185048590 wait complete. 2024-03-24T00:47:22.391 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-24T00:47:22.391 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T00:47:22.391 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-03-24T00:47:22.391 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-24T00:47:22.391 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T00:47:22.391 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-24T00:47:22.391 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-24T00:47:22.391 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T00:47:22.391 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:47:22.392 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:47:22.392 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-03-24T00:47:22.407 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T00:47:22.407 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T00:47:22.407 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T00:47:22.407 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:47:22.407 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-03-24T00:47:22.407 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:47:22.407 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:47:22.407 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-03-24T00:47:22.407 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-03-24T00:47:22.407 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:47:22.407 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-03-24T00:47:22.408 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-03-24T00:47:22.408 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:47:22.408 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-03-24T00:47:22.408 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T00:47:22.408 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-03-24T00:47:22.408 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-24T00:47:22.408 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 42071sec preferred_lft 42071sec 2024-03-24T00:47:22.408 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-03-24T00:47:22.408 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:47:22.408 INFO:teuthology.orchestra.run.smithi067.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T00:47:22.408 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 96:cf:26:c8:bf:62 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:47:22.408 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-24T00:47:22.408 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:47:22.408 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::7c6a:5fff:feaa:d41/64 scope link 2024-03-24T00:47:22.408 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:47:22.409 INFO:teuthology.orchestra.run.smithi067.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T00:47:22.409 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ea:22:4b:d5:48:3d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:47:22.409 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::e822:4bff:fed5:483d/64 scope link 2024-03-24T00:47:22.409 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:47:22.409 INFO:teuthology.orchestra.run.smithi067.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T00:47:22.409 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 96:cf:26:c8:bf:62 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:47:22.409 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::94cf:26ff:fec8:bf62/64 scope link 2024-03-24T00:47:22.409 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:47:22.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:47:22.409 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T00:47:22.474 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-03-24T00:47:22.476 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:47:22.476 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-03-24T00:47:22.531 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-03-24T00:47:22.532 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T00:47:22.532 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-24T00:47:22.532 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-24T00:47:22.532 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T00:47:22.532 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:47:22.532 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-03-24T00:47:22.532 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-03-24T00:47:22.532 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-24T00:47:22.532 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-24T00:47:22.532 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-24T00:47:22.532 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-03-24T00:47:22.627 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:47:22.627 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-03-24T00:47:22.641 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-24T00:47:22.642 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-24T00:47:22.642 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:47:22.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:47:22.642 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-03-24T00:47:22.708 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:47:22.708 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-24T00:47:22.762 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:47:22.762 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:47:22.762 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:47:22.816 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-24T00:47:22.816 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:47:22.817 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:47:22.900 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:47:22.900 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T00:47:22.900 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-03-24T00:47:22.908 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:47:22.908 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T00:47:22.908 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-03-24T00:47:22.908 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.72:3300,172.21.15.72:3301" 2024-03-24T00:47:22.908 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-24T00:47:22.913 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:47:22.914 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T00:47:22.914 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-03-24T00:47:22.914 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.72:3300,172.21.15.72:3301" 2024-03-24T00:47:22.914 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-24T00:47:22.914 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:6789,172.21.15.72:6789,172.21.15.72:6790" 2024-03-24T00:47:22.914 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.67:6789,172.21.15.72:6789,172.21.15.72:6790:/ 2024-03-24T00:47:22.914 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=de1d0178-808e-49ca-a986-e297e523b976" will pass to kernel 2024-03-24T00:47:22.914 INFO:teuthology.orchestra.run.smithi067.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-24T00:47:22.914 INFO:teuthology.orchestra.run.smithi067.stdout: You just mounted a file system that supports labels which does not 2024-03-24T00:47:22.915 INFO:teuthology.orchestra.run.smithi067.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-24T00:47:22.915 INFO:teuthology.orchestra.run.smithi067.stdout: applications will generate AVC messages and not be allowed access to 2024-03-24T00:47:22.915 INFO:teuthology.orchestra.run.smithi067.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-24T00:47:22.916 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-24T00:47:22.916 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:47:22.916 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:47:22.942 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:47:22.943 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T00:47:23.002 INFO:teuthology.orchestra.run.smithi067.stdout:client7873 2024-03-24T00:47:23.002 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:47:23.002 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:47:23.058 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:47:23.058 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:47:23.110 INFO:tasks.ceph.mds.c.smithi067.stderr:2024-03-24T00:47:23.111+0000 7f41f367b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T00:47:23.114 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:47:23.114 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=file1_sixmegs bs=1M conv=fdatasync count=6 seek=0) 2024-03-24T00:47:23.215 INFO:teuthology.orchestra.run.smithi067.stderr:6+0 records in 2024-03-24T00:47:23.215 INFO:teuthology.orchestra.run.smithi067.stderr:6+0 records out 2024-03-24T00:47:23.215 INFO:teuthology.orchestra.run.smithi067.stderr:6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.042111 s, 149 MB/s 2024-03-24T00:47:23.216 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:47:23.216 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:47:23.231 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:47:23.231 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:47:23.286 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:47:23.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.426+0000 7f7d075b3640 1 Processor -- start 2024-03-24T00:47:23.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.427+0000 7f7d075b3640 1 -- start start 2024-03-24T00:47:23.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.427+0000 7f7d075b3640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7d0015e2a0 0x7f7d000ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:23.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.427+0000 7f7d075b3640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7d000d0ef0 0x7f7d00162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:23.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.427+0000 7f7d075b3640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d000cf640 con 0x7f7d0015dec0 2024-03-24T00:47:23.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.427+0000 7f7d075b3640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d000cee30 con 0x7f7d000d0ef0 2024-03-24T00:47:23.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.427+0000 7f7d075b3640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d000d2120 con 0x7f7d0015e2a0 2024-03-24T00:47:23.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.428+0000 7f7d05b50640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7d000d0ef0 0x7f7d00162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:23.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.428+0000 7f7d04b4e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7d0015e2a0 0x7f7d000ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:23.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.428+0000 7f7d0534f640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f7d0015dec0 0x7f7d000d07e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:38024/0 (socket says 172.21.15.67:38024) 2024-03-24T00:47:23.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.428+0000 7f7d0534f640 1 -- 172.21.15.67:0/61716592 learned_addr learned my addr 172.21.15.67:0/61716592 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:47:23.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.428+0000 7f7cf7fff640 1 -- 172.21.15.67:0/61716592 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1022424488 0 0) 0x7f7cfc001110 con 0x7f7d0015dec0 2024-03-24T00:47:23.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.428+0000 7f7cf7fff640 1 -- 172.21.15.67:0/61716592 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cec003520 con 0x7f7d0015dec0 2024-03-24T00:47:23.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.428+0000 7f7d05b50640 1 -- 172.21.15.67:0/61716592 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7d0015e2a0 msgr2=0x7f7d000ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:23.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.428+0000 7f7d05b50640 1 --2- 172.21.15.67:0/61716592 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7d0015e2a0 0x7f7d000ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:23.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.428+0000 7f7d05b50640 1 -- 172.21.15.67:0/61716592 >> v1:172.21.15.72:6789/0 conn(0x7f7d0015dec0 legacy=0x7f7d000d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:23.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.428+0000 7f7d05b50640 1 -- 172.21.15.67:0/61716592 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d000cf080 con 0x7f7d000d0ef0 2024-03-24T00:47:23.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.428+0000 7f7d04b4e640 1 --2- 172.21.15.67:0/61716592 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7d0015e2a0 0x7f7d000ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:47:23.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.429+0000 7f7d05b50640 1 --2- 172.21.15.67:0/61716592 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d000d0ef0 0x7f7d00162630 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f7cf8009d90 tx=0x7f7cf802be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d469f60b1f8c2a8d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:23.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.429+0000 7f7cf7fff640 1 -- 172.21.15.67:0/61716592 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf8039070 con 0x7f7d000d0ef0 2024-03-24T00:47:23.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.429+0000 7f7cf7fff640 1 -- 172.21.15.67:0/61716592 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cf802cbc0 con 0x7f7d000d0ef0 2024-03-24T00:47:23.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.429+0000 7f7cf7fff640 1 -- 172.21.15.67:0/61716592 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf8034440 con 0x7f7d000d0ef0 2024-03-24T00:47:23.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.430+0000 7f7d075b3640 1 -- 172.21.15.67:0/61716592 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d000d0ef0 msgr2=0x7f7d00162630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:23.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.430+0000 7f7d075b3640 1 --2- 172.21.15.67:0/61716592 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d000d0ef0 0x7f7d00162630 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f7cf8009d90 tx=0x7f7cf802be10 comp rx=0 tx=0).stop 2024-03-24T00:47:23.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.430+0000 7f7d075b3640 1 -- 172.21.15.67:0/61716592 shutdown_connections 2024-03-24T00:47:23.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.430+0000 7f7d075b3640 1 --2- 172.21.15.67:0/61716592 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d000d0ef0 0x7f7d00162630 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:23.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.430+0000 7f7d075b3640 1 --2- 172.21.15.67:0/61716592 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7d0015e2a0 0x7f7d000ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:23.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.430+0000 7f7d075b3640 1 -- 172.21.15.67:0/61716592 >> 172.21.15.67:0/61716592 conn(0x7f7d0006c810 msgr2=0x7f7d0006cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:23.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.430+0000 7f7d075b3640 1 -- 172.21.15.67:0/61716592 shutdown_connections 2024-03-24T00:47:23.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.430+0000 7f7d075b3640 1 -- 172.21.15.67:0/61716592 wait complete. 2024-03-24T00:47:23.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.431+0000 7f7d075b3640 1 Processor -- start 2024-03-24T00:47:23.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.431+0000 7f7d075b3640 1 -- start start 2024-03-24T00:47:23.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.432+0000 7f7d075b3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d000d0ef0 0x7f7d0015ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:23.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.432+0000 7f7d075b3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d0015dec0 0x7f7d000b42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:23.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.432+0000 7f7d075b3640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7d0015e2a0 0x7f7d000b47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:23.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.432+0000 7f7d075b3640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d000d2730 con 0x7f7d000d0ef0 2024-03-24T00:47:23.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.432+0000 7f7d075b3640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d000d25b0 con 0x7f7d0015dec0 2024-03-24T00:47:23.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.432+0000 7f7d075b3640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d000d28b0 con 0x7f7d0015e2a0 2024-03-24T00:47:23.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.432+0000 7f7d0534f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d000d0ef0 0x7f7d0015ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:23.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.432+0000 7f7d0534f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d000d0ef0 0x7f7d0015ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47376/0 (socket says 172.21.15.67:47376) 2024-03-24T00:47:23.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.432+0000 7f7d0534f640 1 -- 172.21.15.67:0/1199364896 learned_addr learned my addr 172.21.15.67:0/1199364896 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:23.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.432+0000 7f7d04b4e640 1 --2- 172.21.15.67:0/1199364896 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d0015dec0 0x7f7d000b42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:23.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.432+0000 7f7d05b50640 1 --2- 172.21.15.67:0/1199364896 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7d0015e2a0 0x7f7d000b47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:23.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.432+0000 7f7d0534f640 1 -- 172.21.15.67:0/1199364896 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7d0015e2a0 msgr2=0x7f7d000b47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:23.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.432+0000 7f7d0534f640 1 --2- 172.21.15.67:0/1199364896 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7d0015e2a0 0x7f7d000b47f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:23.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.432+0000 7f7d0534f640 1 -- 172.21.15.67:0/1199364896 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d0015dec0 msgr2=0x7f7d000b42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:23.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.432+0000 7f7d0534f640 1 --2- 172.21.15.67:0/1199364896 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d0015dec0 0x7f7d000b42b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:23.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.432+0000 7f7d0534f640 1 -- 172.21.15.67:0/1199364896 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d000b5260 con 0x7f7d000d0ef0 2024-03-24T00:47:23.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.432+0000 7f7d04b4e640 1 --2- 172.21.15.67:0/1199364896 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d0015dec0 0x7f7d000b42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:47:23.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.432+0000 7f7d05b50640 1 --2- 172.21.15.67:0/1199364896 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7d0015e2a0 0x7f7d000b47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:47:23.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.432+0000 7f7d0534f640 1 --2- 172.21.15.67:0/1199364896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d000d0ef0 0x7f7d0015ae00 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc00c730 tx=0x7f7cfc00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:23.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.433+0000 7f7cf5ffb640 1 -- 172.21.15.67:0/1199364896 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cfc017070 con 0x7f7d000d0ef0 2024-03-24T00:47:23.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.433+0000 7f7cf5ffb640 1 -- 172.21.15.67:0/1199364896 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cfc005ad0 con 0x7f7d000d0ef0 2024-03-24T00:47:23.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.433+0000 7f7d075b3640 1 -- 172.21.15.67:0/1199364896 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d000b5500 con 0x7f7d000d0ef0 2024-03-24T00:47:23.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.433+0000 7f7cf5ffb640 1 -- 172.21.15.67:0/1199364896 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cfc007e20 con 0x7f7d000d0ef0 2024-03-24T00:47:23.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.433+0000 7f7d075b3640 1 -- 172.21.15.67:0/1199364896 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d000b5ac0 con 0x7f7d000d0ef0 2024-03-24T00:47:23.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.434+0000 7f7d075b3640 1 -- 172.21.15.67:0/1199364896 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d0004f400 con 0x7f7d000d0ef0 2024-03-24T00:47:23.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.435+0000 7f7cf5ffb640 1 -- 172.21.15.67:0/1199364896 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7cfc01a070 con 0x7f7d000d0ef0 2024-03-24T00:47:23.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.435+0000 7f7cf5ffb640 1 --2- 172.21.15.67:0/1199364896 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7ce406bbd0 0x7f7ce406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:23.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.435+0000 7f7cf5ffb640 1 -- 172.21.15.67:0/1199364896 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7cfc08a030 con 0x7f7d000d0ef0 2024-03-24T00:47:23.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.435+0000 7f7d04b4e640 1 --2- 172.21.15.67:0/1199364896 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7ce406bbd0 0x7f7ce406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:23.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.436+0000 7f7d04b4e640 1 --2- 172.21.15.67:0/1199364896 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7ce406bbd0 0x7f7ce406e090 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f7d0015d720 tx=0x7f7cf0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:23.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.440+0000 7f7cf5ffb640 1 -- 172.21.15.67:0/1199364896 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7d000cf640 con 0x7f7d000d0ef0 2024-03-24T00:47:23.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.712+0000 7f7d075b3640 1 -- 172.21.15.67:0/1199364896 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d000d1aa0 con 0x7f7d000d0ef0 2024-03-24T00:47:23.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.713+0000 7f7cf5ffb640 1 -- 172.21.15.67:0/1199364896 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v105) v1 ==== 74+0+16824 (secure 0 0 0) 0x7f7cfc058990 con 0x7f7d000d0ef0 2024-03-24T00:47:23.713 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:47:23.714 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":105,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:47:18.128465+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:47:15.111478+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-03-24T00:47:15.455832+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:47:23.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.716+0000 7f7d075b3640 1 -- 172.21.15.67:0/1199364896 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7ce406bbd0 msgr2=0x7f7ce406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:23.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.716+0000 7f7d075b3640 1 --2- 172.21.15.67:0/1199364896 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7ce406bbd0 0x7f7ce406e090 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f7d0015d720 tx=0x7f7cf0008040 comp rx=0 tx=0).stop 2024-03-24T00:47:23.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.716+0000 7f7d075b3640 1 -- 172.21.15.67:0/1199364896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d000d0ef0 msgr2=0x7f7d0015ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:23.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.716+0000 7f7d075b3640 1 --2- 172.21.15.67:0/1199364896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d000d0ef0 0x7f7d0015ae00 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc00c730 tx=0x7f7cfc00cbf0 comp rx=0 tx=0).stop 2024-03-24T00:47:23.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.716+0000 7f7d075b3640 1 -- 172.21.15.67:0/1199364896 shutdown_connections 2024-03-24T00:47:23.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.716+0000 7f7d075b3640 1 --2- 172.21.15.67:0/1199364896 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7ce406bbd0 0x7f7ce406e090 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:23.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.716+0000 7f7d075b3640 1 --2- 172.21.15.67:0/1199364896 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7d0015e2a0 0x7f7d000b47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:23.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.716+0000 7f7d075b3640 1 --2- 172.21.15.67:0/1199364896 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d0015dec0 0x7f7d000b42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:23.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.716+0000 7f7d075b3640 1 --2- 172.21.15.67:0/1199364896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d000d0ef0 0x7f7d0015ae00 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:23.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.716+0000 7f7d075b3640 1 -- 172.21.15.67:0/1199364896 >> 172.21.15.67:0/1199364896 conn(0x7f7d0006c810 msgr2=0x7f7d0015f990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:23.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.716+0000 7f7d075b3640 1 -- 172.21.15.67:0/1199364896 shutdown_connections 2024-03-24T00:47:23.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.716+0000 7f7d075b3640 1 -- 172.21.15.67:0/1199364896 wait complete. 2024-03-24T00:47:23.727 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:38:39.364950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:47:15.111478+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '104', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:47:15.455832+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '105', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:47:23.727 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:47:23.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.836+0000 7fd4c9beb640 1 Processor -- start 2024-03-24T00:47:23.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.836+0000 7fd4c9beb640 1 -- start start 2024-03-24T00:47:23.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.837+0000 7fd4c9beb640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd4c40cd360 0x7fd4c4099a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:23.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.837+0000 7fd4c9beb640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4c40af730 con 0x7fd4c4160430 2024-03-24T00:47:23.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.837+0000 7fd4c9beb640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4c40b2880 con 0x7fd4c415bee0 2024-03-24T00:47:23.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.837+0000 7fd4c9beb640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4c40cf180 con 0x7fd4c40cd360 2024-03-24T00:47:23.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.837+0000 7fd4c3fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd4c40cd360 0x7fd4c4099a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:23.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.837+0000 7fd4baffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd4c4160430 0x7fd4c40ccc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45274/0 (socket says 172.21.15.67:45274) 2024-03-24T00:47:23.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.837+0000 7fd4baffd640 1 -- 172.21.15.67:0/3017208936 learned_addr learned my addr 172.21.15.67:0/3017208936 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:47:23.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.837+0000 7fd4c3fff640 1 -- 172.21.15.67:0/3017208936 >> v1:172.21.15.72:6789/0 conn(0x7fd4c415bee0 legacy=0x7fd4c406b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:23.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.837+0000 7fd4c3fff640 1 -- 172.21.15.67:0/3017208936 >> v1:172.21.15.67:6789/0 conn(0x7fd4c4160430 legacy=0x7fd4c40ccc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:23.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.837+0000 7fd4c3fff640 1 -- 172.21.15.67:0/3017208936 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4c40b2b60 con 0x7fd4c40cd360 2024-03-24T00:47:23.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.837+0000 7fd4c2ffd640 1 -- 172.21.15.67:0/3017208936 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3033151023 0 0) 0x7fd4c40af730 con 0x7fd4c4160430 2024-03-24T00:47:23.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.838+0000 7fd4c3fff640 1 --2- 172.21.15.67:0/3017208936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4c40cd360 0x7fd4c4099a30 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc009d90 tx=0x7fd4bc02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1235f44fdf9de375 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:23.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.838+0000 7fd4c2ffd640 1 -- 172.21.15.67:0/3017208936 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4bc039070 con 0x7fd4c40cd360 2024-03-24T00:47:23.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.838+0000 7fd4c2ffd640 1 -- 172.21.15.67:0/3017208936 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4bc02cc60 con 0x7fd4c40cd360 2024-03-24T00:47:23.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.838+0000 7fd4c2ffd640 1 -- 172.21.15.67:0/3017208936 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4bc034660 con 0x7fd4c40cd360 2024-03-24T00:47:23.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.839+0000 7fd4c9beb640 1 -- 172.21.15.67:0/3017208936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4c40cd360 msgr2=0x7fd4c4099a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:23.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.839+0000 7fd4c9beb640 1 --2- 172.21.15.67:0/3017208936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4c40cd360 0x7fd4c4099a30 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc009d90 tx=0x7fd4bc02beb0 comp rx=0 tx=0).stop 2024-03-24T00:47:23.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.839+0000 7fd4c9beb640 1 -- 172.21.15.67:0/3017208936 shutdown_connections 2024-03-24T00:47:23.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.839+0000 7fd4c9beb640 1 --2- 172.21.15.67:0/3017208936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4c40cd360 0x7fd4c4099a30 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:23.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.839+0000 7fd4c9beb640 1 -- 172.21.15.67:0/3017208936 >> 172.21.15.67:0/3017208936 conn(0x7fd4c40b3c40 msgr2=0x7fd4c40b1dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:23.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.839+0000 7fd4c9beb640 1 -- 172.21.15.67:0/3017208936 shutdown_connections 2024-03-24T00:47:23.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.839+0000 7fd4c9beb640 1 -- 172.21.15.67:0/3017208936 wait complete. 2024-03-24T00:47:23.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.840+0000 7fd4c9beb640 1 Processor -- start 2024-03-24T00:47:23.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.840+0000 7fd4c9beb640 1 -- start start 2024-03-24T00:47:23.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.841+0000 7fd4c9beb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4c40cd360 0x7fd4c40c7930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:23.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.841+0000 7fd4c9beb640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4c415bee0 0x7fd4c41e5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:23.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.841+0000 7fd4c9beb640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd4c4160430 0x7fd4c40b0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:23.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.841+0000 7fd4c9beb640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c40cf610 con 0x7fd4c415bee0 2024-03-24T00:47:23.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.841+0000 7fd4c9beb640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c40ceda0 con 0x7fd4c40cd360 2024-03-24T00:47:23.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.841+0000 7fd4c9beb640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c40cf790 con 0x7fd4c4160430 2024-03-24T00:47:23.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.841+0000 7fd4baffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4c415bee0 0x7fd4c41e5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:23.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.841+0000 7fd4baffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4c415bee0 0x7fd4c41e5920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60266/0 (socket says 172.21.15.67:60266) 2024-03-24T00:47:23.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.841+0000 7fd4baffd640 1 -- 172.21.15.67:0/1152172066 learned_addr learned my addr 172.21.15.67:0/1152172066 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:23.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.841+0000 7fd4c37fe640 1 --2- 172.21.15.67:0/1152172066 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4c40cd360 0x7fd4c40c7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:23.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.841+0000 7fd4c3fff640 1 --2- 172.21.15.67:0/1152172066 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd4c4160430 0x7fd4c40b0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:23.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.841+0000 7fd4baffd640 1 -- 172.21.15.67:0/1152172066 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd4c4160430 msgr2=0x7fd4c40b0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:23.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.841+0000 7fd4baffd640 1 --2- 172.21.15.67:0/1152172066 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd4c4160430 0x7fd4c40b0e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:23.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.841+0000 7fd4baffd640 1 -- 172.21.15.67:0/1152172066 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4c40cd360 msgr2=0x7fd4c40c7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:23.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.841+0000 7fd4baffd640 1 --2- 172.21.15.67:0/1152172066 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4c40cd360 0x7fd4c40c7930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:23.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.841+0000 7fd4baffd640 1 -- 172.21.15.67:0/1152172066 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4c40afd60 con 0x7fd4c415bee0 2024-03-24T00:47:23.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.842+0000 7fd4baffd640 1 --2- 172.21.15.67:0/1152172066 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4c415bee0 0x7fd4c41e5920 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac00c410 tx=0x7fd4ac00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:23.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.842+0000 7fd4c0ff9640 1 -- 172.21.15.67:0/1152172066 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4ac0074e0 con 0x7fd4c415bee0 2024-03-24T00:47:23.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.842+0000 7fd4c0ff9640 1 -- 172.21.15.67:0/1152172066 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4ac0047e0 con 0x7fd4c415bee0 2024-03-24T00:47:23.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.842+0000 7fd4c9beb640 1 -- 172.21.15.67:0/1152172066 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4c40b0040 con 0x7fd4c415bee0 2024-03-24T00:47:23.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.842+0000 7fd4c0ff9640 1 -- 172.21.15.67:0/1152172066 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4ac004bd0 con 0x7fd4c415bee0 2024-03-24T00:47:23.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.842+0000 7fd4c9beb640 1 -- 172.21.15.67:0/1152172066 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4c40b0600 con 0x7fd4c415bee0 2024-03-24T00:47:23.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.843+0000 7fd4c9beb640 1 -- 172.21.15.67:0/1152172066 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4c404f400 con 0x7fd4c415bee0 2024-03-24T00:47:23.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.843+0000 7fd4c0ff9640 1 -- 172.21.15.67:0/1152172066 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd4ac011070 con 0x7fd4c415bee0 2024-03-24T00:47:23.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.844+0000 7fd4c0ff9640 1 --2- 172.21.15.67:0/1152172066 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd4a406bbd0 0x7fd4a406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:23.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.844+0000 7fd4c0ff9640 1 -- 172.21.15.67:0/1152172066 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd4ac08a850 con 0x7fd4c415bee0 2024-03-24T00:47:23.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.844+0000 7fd4c37fe640 1 --2- 172.21.15.67:0/1152172066 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd4a406bbd0 0x7fd4a406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:23.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.844+0000 7fd4c37fe640 1 --2- 172.21.15.67:0/1152172066 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd4a406bbd0 0x7fd4a406e090 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fd4b4004ea0 tx=0x7fd4b4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:23.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:23.848+0000 7fd4c0ff9640 1 -- 172.21.15.67:0/1152172066 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd4ac059240 con 0x7fd4c415bee0 2024-03-24T00:47:24.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.029+0000 7fd4c9beb640 1 -- 172.21.15.67:0/1152172066 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd4c40cda40 con 0x7fd4c415bee0 2024-03-24T00:47:24.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.029+0000 7fd4c0ff9640 1 -- 172.21.15.67:0/1152172066 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v105) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd4ac058b40 con 0x7fd4c415bee0 2024-03-24T00:47:24.029 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-24T00:47:24.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.031+0000 7fd4c9beb640 1 -- 172.21.15.67:0/1152172066 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd4a406bbd0 msgr2=0x7fd4a406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:24.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.031+0000 7fd4c9beb640 1 --2- 172.21.15.67:0/1152172066 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd4a406bbd0 0x7fd4a406e090 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fd4b4004ea0 tx=0x7fd4b4009040 comp rx=0 tx=0).stop 2024-03-24T00:47:24.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.031+0000 7fd4c9beb640 1 -- 172.21.15.67:0/1152172066 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4c415bee0 msgr2=0x7fd4c41e5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:24.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.031+0000 7fd4c9beb640 1 --2- 172.21.15.67:0/1152172066 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4c415bee0 0x7fd4c41e5920 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac00c410 tx=0x7fd4ac00c8d0 comp rx=0 tx=0).stop 2024-03-24T00:47:24.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.031+0000 7fd4c9beb640 1 -- 172.21.15.67:0/1152172066 shutdown_connections 2024-03-24T00:47:24.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.031+0000 7fd4c9beb640 1 --2- 172.21.15.67:0/1152172066 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd4a406bbd0 0x7fd4a406e090 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:24.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.031+0000 7fd4c9beb640 1 --2- 172.21.15.67:0/1152172066 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd4c4160430 0x7fd4c40b0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:24.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.031+0000 7fd4c9beb640 1 --2- 172.21.15.67:0/1152172066 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4c415bee0 0x7fd4c41e5920 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:24.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.031+0000 7fd4c9beb640 1 --2- 172.21.15.67:0/1152172066 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4c40cd360 0x7fd4c40c7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:24.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.031+0000 7fd4c9beb640 1 -- 172.21.15.67:0/1152172066 >> 172.21.15.67:0/1152172066 conn(0x7fd4c40b3c40 msgr2=0x7fd4c40cb890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:24.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.031+0000 7fd4c9beb640 1 -- 172.21.15.67:0/1152172066 shutdown_connections 2024-03-24T00:47:24.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.031+0000 7fd4c9beb640 1 -- 172.21.15.67:0/1152172066 wait complete. 2024-03-24T00:47:24.041 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:47:24.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.148+0000 7f1b0de36640 1 Processor -- start 2024-03-24T00:47:24.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.148+0000 7f1b0de36640 1 -- start start 2024-03-24T00:47:24.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.149+0000 7f1b0de36640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1b080d3ae0 0x7f1b080b5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:24.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.149+0000 7f1b0de36640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b080d2250 con 0x7f1b0815dea0 2024-03-24T00:47:24.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.149+0000 7f1b0de36640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b080d2530 con 0x7f1b0815e280 2024-03-24T00:47:24.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.149+0000 7f1b0de36640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b080d2810 con 0x7f1b080d3ae0 2024-03-24T00:47:24.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.149+0000 7f1b077fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1b0815dea0 0x7f1b080d33d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45280/0 (socket says 172.21.15.67:45280) 2024-03-24T00:47:24.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.149+0000 7f1b077fe640 1 -- 172.21.15.67:0/1491941433 learned_addr learned my addr 172.21.15.67:0/1491941433 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:47:24.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.149+0000 7f1b06ffd640 1 --2- 172.21.15.67:0/1491941433 >> v2:172.21.15.67:3300/0 conn(0x7f1b080d3ae0 0x7f1b080b5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:24.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.149+0000 7f1b06ffd640 1 -- 172.21.15.67:0/1491941433 >> v1:172.21.15.72:6789/0 conn(0x7f1b0815e280 legacy=0x7f1b080b70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:24.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.149+0000 7f1b06ffd640 1 -- 172.21.15.67:0/1491941433 >> v1:172.21.15.67:6789/0 conn(0x7f1b0815dea0 legacy=0x7f1b080d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:24.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.149+0000 7f1b067fc640 1 -- 172.21.15.67:0/1491941433 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3798358872 0 0) 0x7f1b080d2530 con 0x7f1b0815e280 2024-03-24T00:47:24.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.149+0000 7f1b06ffd640 1 -- 172.21.15.67:0/1491941433 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b080d2af0 con 0x7f1b080d3ae0 2024-03-24T00:47:24.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.150+0000 7f1b06ffd640 1 --2- 172.21.15.67:0/1491941433 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b080d3ae0 0x7f1b080b5a60 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f1afc0097d0 tx=0x7f1afc02b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d4365a4d280268c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:24.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.150+0000 7f1b067fc640 1 -- 172.21.15.67:0/1491941433 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1afc037070 con 0x7f1b080d3ae0 2024-03-24T00:47:24.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.150+0000 7f1b067fc640 1 -- 172.21.15.67:0/1491941433 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1afc030710 con 0x7f1b080d3ae0 2024-03-24T00:47:24.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.150+0000 7f1b067fc640 1 -- 172.21.15.67:0/1491941433 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1afc030ae0 con 0x7f1b080d3ae0 2024-03-24T00:47:24.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.151+0000 7f1b0de36640 1 -- 172.21.15.67:0/1491941433 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b080d3ae0 msgr2=0x7f1b080b5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:24.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.151+0000 7f1b0de36640 1 --2- 172.21.15.67:0/1491941433 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b080d3ae0 0x7f1b080b5a60 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f1afc0097d0 tx=0x7f1afc02b710 comp rx=0 tx=0).stop 2024-03-24T00:47:24.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.151+0000 7f1b0de36640 1 -- 172.21.15.67:0/1491941433 shutdown_connections 2024-03-24T00:47:24.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.151+0000 7f1b0de36640 1 --2- 172.21.15.67:0/1491941433 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b080d3ae0 0x7f1b080b5a60 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:24.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.151+0000 7f1b0de36640 1 -- 172.21.15.67:0/1491941433 >> 172.21.15.67:0/1491941433 conn(0x7f1b08072da0 msgr2=0x7f1b080d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:24.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.151+0000 7f1b0de36640 1 -- 172.21.15.67:0/1491941433 shutdown_connections 2024-03-24T00:47:24.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.151+0000 7f1b0de36640 1 -- 172.21.15.67:0/1491941433 wait complete. 2024-03-24T00:47:24.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.153+0000 7f1b0de36640 1 Processor -- start 2024-03-24T00:47:24.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.153+0000 7f1b0de36640 1 -- start start 2024-03-24T00:47:24.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.154+0000 7f1b0de36640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b080d3ae0 0x7f1b080c9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:24.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.154+0000 7f1b0de36640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b0815dea0 0x7f1b0815a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:24.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.154+0000 7f1b0de36640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1b0815e280 0x7f1b0815ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:24.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.154+0000 7f1b0de36640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b080b6370 con 0x7f1b0815dea0 2024-03-24T00:47:24.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.154+0000 7f1b0de36640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b080b61f0 con 0x7f1b080d3ae0 2024-03-24T00:47:24.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.154+0000 7f1b0de36640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b080b7800 con 0x7f1b0815e280 2024-03-24T00:47:24.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.154+0000 7f1b06ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b0815dea0 0x7f1b0815a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:24.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.154+0000 7f1b06ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b0815dea0 0x7f1b0815a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60284/0 (socket says 172.21.15.67:60284) 2024-03-24T00:47:24.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.154+0000 7f1b06ffd640 1 -- 172.21.15.67:0/3435136472 learned_addr learned my addr 172.21.15.67:0/3435136472 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:24.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.155+0000 7f1b07fff640 1 --2- 172.21.15.67:0/3435136472 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1b0815e280 0x7f1b0815ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:24.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.155+0000 7f1b077fe640 1 --2- 172.21.15.67:0/3435136472 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b080d3ae0 0x7f1b080c9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:24.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.155+0000 7f1b06ffd640 1 -- 172.21.15.67:0/3435136472 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1b0815e280 msgr2=0x7f1b0815ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:24.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.155+0000 7f1b06ffd640 1 --2- 172.21.15.67:0/3435136472 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1b0815e280 0x7f1b0815ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:24.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.155+0000 7f1b06ffd640 1 -- 172.21.15.67:0/3435136472 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b080d3ae0 msgr2=0x7f1b080c9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:24.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.155+0000 7f1b06ffd640 1 --2- 172.21.15.67:0/3435136472 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b080d3ae0 0x7f1b080c9ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:24.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.155+0000 7f1b06ffd640 1 -- 172.21.15.67:0/3435136472 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b081f4b80 con 0x7f1b0815dea0 2024-03-24T00:47:24.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.155+0000 7f1b077fe640 1 --2- 172.21.15.67:0/3435136472 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b080d3ae0 0x7f1b080c9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:47:24.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.155+0000 7f1b07fff640 1 --2- 172.21.15.67:0/3435136472 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1b0815e280 0x7f1b0815ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:47:24.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.155+0000 7f1b06ffd640 1 --2- 172.21.15.67:0/3435136472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b0815dea0 0x7f1b0815a820 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f1afc0097d0 tx=0x7f1afc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:24.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.155+0000 7f1aeffff640 1 -- 172.21.15.67:0/3435136472 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1afc042070 con 0x7f1b0815dea0 2024-03-24T00:47:24.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.155+0000 7f1aeffff640 1 -- 172.21.15.67:0/3435136472 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1afc032a40 con 0x7f1b0815dea0 2024-03-24T00:47:24.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.155+0000 7f1b0de36640 1 -- 172.21.15.67:0/3435136472 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b081f3dc0 con 0x7f1b0815dea0 2024-03-24T00:47:24.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.155+0000 7f1aeffff640 1 -- 172.21.15.67:0/3435136472 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1afc037040 con 0x7f1b0815dea0 2024-03-24T00:47:24.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.155+0000 7f1b0de36640 1 -- 172.21.15.67:0/3435136472 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b081f4380 con 0x7f1b0815dea0 2024-03-24T00:47:24.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.156+0000 7f1b0de36640 1 -- 172.21.15.67:0/3435136472 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b0804f400 con 0x7f1b0815dea0 2024-03-24T00:47:24.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.156+0000 7f1aeffff640 1 -- 172.21.15.67:0/3435136472 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1afc045070 con 0x7f1b0815dea0 2024-03-24T00:47:24.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.157+0000 7f1aeffff640 1 --2- 172.21.15.67:0/3435136472 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1af406bbd0 0x7f1af406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:24.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.157+0000 7f1aeffff640 1 -- 172.21.15.67:0/3435136472 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1afc0aca20 con 0x7f1b0815dea0 2024-03-24T00:47:24.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.157+0000 7f1b077fe640 1 --2- 172.21.15.67:0/3435136472 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1af406bbd0 0x7f1af406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:24.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.158+0000 7f1b077fe640 1 --2- 172.21.15.67:0/3435136472 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1af406bbd0 0x7f1af406e090 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f1af8000f60 tx=0x7f1af8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:24.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.161+0000 7f1aeffff640 1 -- 172.21.15.67:0/3435136472 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1afc07b530 con 0x7f1b0815dea0 2024-03-24T00:47:24.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.342+0000 7f1b0de36640 1 -- 172.21.15.67:0/3435136472 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1b08161630 con 0x7f1b0815dea0 2024-03-24T00:47:24.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.342+0000 7f1aeffff640 1 -- 172.21.15.67:0/3435136472 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v105) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1afc07ae30 con 0x7f1b0815dea0 2024-03-24T00:47:24.342 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-24T00:47:24.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.344+0000 7f1b0de36640 1 -- 172.21.15.67:0/3435136472 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1af406bbd0 msgr2=0x7f1af406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:24.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.344+0000 7f1b0de36640 1 --2- 172.21.15.67:0/3435136472 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1af406bbd0 0x7f1af406e090 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f1af8000f60 tx=0x7f1af8009040 comp rx=0 tx=0).stop 2024-03-24T00:47:24.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.344+0000 7f1b0de36640 1 -- 172.21.15.67:0/3435136472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b0815dea0 msgr2=0x7f1b0815a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:24.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.344+0000 7f1b0de36640 1 --2- 172.21.15.67:0/3435136472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b0815dea0 0x7f1b0815a820 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f1afc0097d0 tx=0x7f1afc0046c0 comp rx=0 tx=0).stop 2024-03-24T00:47:24.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.344+0000 7f1b0de36640 1 -- 172.21.15.67:0/3435136472 shutdown_connections 2024-03-24T00:47:24.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.344+0000 7f1b0de36640 1 --2- 172.21.15.67:0/3435136472 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1af406bbd0 0x7f1af406e090 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:24.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.344+0000 7f1b0de36640 1 --2- 172.21.15.67:0/3435136472 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1b0815e280 0x7f1b0815ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:24.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.344+0000 7f1b0de36640 1 --2- 172.21.15.67:0/3435136472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b0815dea0 0x7f1b0815a820 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:24.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.344+0000 7f1b0de36640 1 --2- 172.21.15.67:0/3435136472 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b080d3ae0 0x7f1b080c9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:24.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.344+0000 7f1b0de36640 1 -- 172.21.15.67:0/3435136472 >> 172.21.15.67:0/3435136472 conn(0x7f1b08072da0 msgr2=0x7f1b080d09b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:24.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.344+0000 7f1b0de36640 1 -- 172.21.15.67:0/3435136472 shutdown_connections 2024-03-24T00:47:24.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.344+0000 7f1b0de36640 1 -- 172.21.15.67:0/3435136472 wait complete. 2024-03-24T00:47:24.355 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:47:24.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.459+0000 7fb88febf640 1 Processor -- start 2024-03-24T00:47:24.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.459+0000 7fb88febf640 1 -- start start 2024-03-24T00:47:24.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.460+0000 7fb88febf640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb88815dea0 0x7fb8880d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:24.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.460+0000 7fb88febf640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb88815e280 0x7fb8880d3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:24.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.460+0000 7fb88febf640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8880d2250 con 0x7fb8880b5a60 2024-03-24T00:47:24.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.460+0000 7fb88febf640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8880d2530 con 0x7fb88815e280 2024-03-24T00:47:24.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.460+0000 7fb88febf640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8880d2810 con 0x7fb88815dea0 2024-03-24T00:47:24.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.460+0000 7fb88e45c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb8880b5a60 0x7fb88806c640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45282/0 (socket says 172.21.15.67:45282) 2024-03-24T00:47:24.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.460+0000 7fb88e45c640 1 -- 172.21.15.67:0/4088508141 learned_addr learned my addr 172.21.15.67:0/4088508141 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:47:24.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.460+0000 7fb88d45a640 1 --2- 172.21.15.67:0/4088508141 >> v2:172.21.15.72:3300/0 conn(0x7fb88815e280 0x7fb8880d3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:24.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.460+0000 7fb88dc5b640 1 --2- 172.21.15.67:0/4088508141 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb88815dea0 0x7fb8880d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:24.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.460+0000 7fb88cc59640 1 -- 172.21.15.67:0/4088508141 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1198766613 0 0) 0x7fb8880d2250 con 0x7fb8880b5a60 2024-03-24T00:47:24.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.460+0000 7fb88cc59640 1 -- 172.21.15.67:0/4088508141 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb874003520 con 0x7fb8880b5a60 2024-03-24T00:47:24.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.461+0000 7fb88cc59640 1 -- 172.21.15.67:0/4088508141 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 825498806 0 0) 0x7fb874003520 con 0x7fb8880b5a60 2024-03-24T00:47:24.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.461+0000 7fb88cc59640 1 -- 172.21.15.67:0/4088508141 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb88815dea0 msgr2=0x7fb8880d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:24.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.461+0000 7fb88cc59640 1 --2- 172.21.15.67:0/4088508141 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb88815dea0 0x7fb8880d33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:24.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.461+0000 7fb88cc59640 1 -- 172.21.15.67:0/4088508141 >> v2:172.21.15.72:3300/0 conn(0x7fb88815e280 msgr2=0x7fb8880d3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:24.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.461+0000 7fb88cc59640 1 --2- 172.21.15.67:0/4088508141 >> v2:172.21.15.72:3300/0 conn(0x7fb88815e280 0x7fb8880d3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:24.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.461+0000 7fb88cc59640 1 -- 172.21.15.67:0/4088508141 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8880d2af0 con 0x7fb8880b5a60 2024-03-24T00:47:24.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.461+0000 7fb88cc59640 1 -- 172.21.15.67:0/4088508141 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7fb8800019f0 con 0x7fb8880b5a60 2024-03-24T00:47:24.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.461+0000 7fb88d45a640 1 --2- 172.21.15.67:0/4088508141 >> v2:172.21.15.72:3300/0 conn(0x7fb88815e280 0x7fb8880d3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:47:24.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.461+0000 7fb88cc59640 1 -- 172.21.15.67:0/4088508141 >> v1:172.21.15.67:6789/0 conn(0x7fb8880b5a60 legacy=0x7fb88806c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:24.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.461+0000 7fb88cc59640 1 --2- 172.21.15.67:0/4088508141 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb88815dea0 0x7fb8740039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:24.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.461+0000 7fb88cc59640 1 --2- 172.21.15.67:0/4088508141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb874003f30 0x7fb874004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:24.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.461+0000 7fb88cc59640 1 --2- 172.21.15.67:0/4088508141 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb874004d40 0x7fb8740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:24.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.461+0000 7fb88cc59640 1 -- 172.21.15.67:0/4088508141 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8880d2530 con 0x7fb874003f30 2024-03-24T00:47:24.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.461+0000 7fb88cc59640 1 -- 172.21.15.67:0/4088508141 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8880d2810 con 0x7fb874004d40 2024-03-24T00:47:24.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.461+0000 7fb88cc59640 1 -- 172.21.15.67:0/4088508141 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb87400ba10 con 0x7fb88815dea0 2024-03-24T00:47:24.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.461+0000 7fb88dc5b640 1 --2- 172.21.15.67:0/4088508141 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb88815dea0 0x7fb8740039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:24.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.461+0000 7fb88d45a640 1 --2- 172.21.15.67:0/4088508141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb874003f30 0x7fb874004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:24.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.461+0000 7fb88e45c640 1 --2- 172.21.15.67:0/4088508141 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb874004d40 0x7fb8740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:24.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.461+0000 7fb88d45a640 1 -- 172.21.15.67:0/4088508141 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb88815dea0 msgr2=0x7fb8740039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:24.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.461+0000 7fb88d45a640 1 --2- 172.21.15.67:0/4088508141 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb88815dea0 0x7fb8740039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:24.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.461+0000 7fb88d45a640 1 -- 172.21.15.67:0/4088508141 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb874004d40 msgr2=0x7fb8740051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:24.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.461+0000 7fb88d45a640 1 --2- 172.21.15.67:0/4088508141 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb874004d40 0x7fb8740051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:24.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.461+0000 7fb88d45a640 1 -- 172.21.15.67:0/4088508141 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb874003520 con 0x7fb874003f30 2024-03-24T00:47:24.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.462+0000 7fb88d45a640 1 --2- 172.21.15.67:0/4088508141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb874003f30 0x7fb874004800 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7fb878009cc0 tx=0x7fb874005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6726137f4cfd51c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:24.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.462+0000 7fb88cc59640 1 -- 172.21.15.67:0/4088508141 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb878039070 con 0x7fb874003f30 2024-03-24T00:47:24.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.462+0000 7fb88cc59640 1 -- 172.21.15.67:0/4088508141 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb87802c960 con 0x7fb874003f30 2024-03-24T00:47:24.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.462+0000 7fb88cc59640 1 -- 172.21.15.67:0/4088508141 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb878009860 con 0x7fb874003f30 2024-03-24T00:47:24.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.463+0000 7fb88febf640 1 -- 172.21.15.67:0/4088508141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb874003f30 msgr2=0x7fb874004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:24.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.463+0000 7fb88febf640 1 --2- 172.21.15.67:0/4088508141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb874003f30 0x7fb874004800 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7fb878009cc0 tx=0x7fb874005830 comp rx=0 tx=0).stop 2024-03-24T00:47:24.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.463+0000 7fb88e45c640 1 -- 172.21.15.67:0/4088508141 reap_dead start 2024-03-24T00:47:24.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.463+0000 7fb88febf640 1 -- 172.21.15.67:0/4088508141 shutdown_connections 2024-03-24T00:47:24.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.463+0000 7fb88febf640 1 -- 172.21.15.67:0/4088508141 >> 172.21.15.67:0/4088508141 conn(0x7fb888072da0 msgr2=0x7fb8880d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:24.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.463+0000 7fb88febf640 1 -- 172.21.15.67:0/4088508141 shutdown_connections 2024-03-24T00:47:24.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.463+0000 7fb88febf640 1 -- 172.21.15.67:0/4088508141 wait complete. 2024-03-24T00:47:24.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.465+0000 7fb88febf640 1 Processor -- start 2024-03-24T00:47:24.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.465+0000 7fb88febf640 1 -- start start 2024-03-24T00:47:24.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.466+0000 7fb88febf640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb874003f30 0x7fb88815ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:24.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.466+0000 7fb88febf640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb888154f90 0x7fb888160d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:24.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.466+0000 7fb88febf640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb8880b39e0 0x7fb8880b3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:24.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.466+0000 7fb88febf640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8880d33d0 con 0x7fb874003f30 2024-03-24T00:47:24.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.466+0000 7fb88febf640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8880b8310 con 0x7fb8880b39e0 2024-03-24T00:47:24.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.466+0000 7fb88febf640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8880d3550 con 0x7fb888154f90 2024-03-24T00:47:24.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.466+0000 7fb88dc5b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb874003f30 0x7fb88815ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:24.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.466+0000 7fb88dc5b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb874003f30 0x7fb88815ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60290/0 (socket says 172.21.15.67:60290) 2024-03-24T00:47:24.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.466+0000 7fb88dc5b640 1 -- 172.21.15.67:0/1810847348 learned_addr learned my addr 172.21.15.67:0/1810847348 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:24.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.466+0000 7fb88e45c640 1 --2- 172.21.15.67:0/1810847348 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb8880b39e0 0x7fb8880b3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:24.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.466+0000 7fb88d45a640 1 --2- 172.21.15.67:0/1810847348 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb888154f90 0x7fb888160d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:24.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.466+0000 7fb88dc5b640 1 -- 172.21.15.67:0/1810847348 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb888154f90 msgr2=0x7fb888160d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:24.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.466+0000 7fb88dc5b640 1 --2- 172.21.15.67:0/1810847348 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb888154f90 0x7fb888160d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:24.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.466+0000 7fb88dc5b640 1 -- 172.21.15.67:0/1810847348 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb8880b39e0 msgr2=0x7fb8880b3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:24.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.466+0000 7fb88dc5b640 1 --2- 172.21.15.67:0/1810847348 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb8880b39e0 0x7fb8880b3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:24.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.466+0000 7fb88dc5b640 1 -- 172.21.15.67:0/1810847348 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb888159ad0 con 0x7fb874003f30 2024-03-24T00:47:24.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.466+0000 7fb88e45c640 1 --2- 172.21.15.67:0/1810847348 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb8880b39e0 0x7fb8880b3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:47:24.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.466+0000 7fb88dc5b640 1 --2- 172.21.15.67:0/1810847348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb874003f30 0x7fb88815ab70 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7fb8840119a0 tx=0x7fb884011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:24.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.466+0000 7fb87e7fc640 1 -- 172.21.15.67:0/1810847348 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb88400faf0 con 0x7fb874003f30 2024-03-24T00:47:24.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.467+0000 7fb87e7fc640 1 -- 172.21.15.67:0/1810847348 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb88400fc90 con 0x7fb874003f30 2024-03-24T00:47:24.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.467+0000 7fb88febf640 1 -- 172.21.15.67:0/1810847348 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb888159db0 con 0x7fb874003f30 2024-03-24T00:47:24.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.467+0000 7fb87e7fc640 1 -- 172.21.15.67:0/1810847348 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb884014440 con 0x7fb874003f30 2024-03-24T00:47:24.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.467+0000 7fb88febf640 1 -- 172.21.15.67:0/1810847348 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb88815a370 con 0x7fb874003f30 2024-03-24T00:47:24.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.467+0000 7fb88febf640 1 -- 172.21.15.67:0/1810847348 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb88804f400 con 0x7fb874003f30 2024-03-24T00:47:24.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.468+0000 7fb87e7fc640 1 -- 172.21.15.67:0/1810847348 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb884012040 con 0x7fb874003f30 2024-03-24T00:47:24.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.468+0000 7fb87e7fc640 1 --2- 172.21.15.67:0/1810847348 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb86c06bbd0 0x7fb86c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:24.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.469+0000 7fb87e7fc640 1 -- 172.21.15.67:0/1810847348 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb88408b220 con 0x7fb874003f30 2024-03-24T00:47:24.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.469+0000 7fb88d45a640 1 --2- 172.21.15.67:0/1810847348 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb86c06bbd0 0x7fb86c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:24.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.469+0000 7fb88d45a640 1 --2- 172.21.15.67:0/1810847348 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb86c06bbd0 0x7fb86c06e090 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fb878036000 tx=0x7fb878030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:24.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.473+0000 7fb87e7fc640 1 -- 172.21.15.67:0/1810847348 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb884019030 con 0x7fb874003f30 2024-03-24T00:47:24.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.662+0000 7fb88febf640 1 -- 172.21.15.67:0/1810847348 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb8880ca9e0 con 0x7fb874003f30 2024-03-24T00:47:24.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.663+0000 7fb87e7fc640 1 -- 172.21.15.67:0/1810847348 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v105) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb884059530 con 0x7fb874003f30 2024-03-24T00:47:24.663 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-03-24T00:47:24.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.665+0000 7fb88febf640 1 -- 172.21.15.67:0/1810847348 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb86c06bbd0 msgr2=0x7fb86c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:24.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.665+0000 7fb88febf640 1 --2- 172.21.15.67:0/1810847348 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb86c06bbd0 0x7fb86c06e090 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fb878036000 tx=0x7fb878030000 comp rx=0 tx=0).stop 2024-03-24T00:47:24.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.665+0000 7fb88febf640 1 -- 172.21.15.67:0/1810847348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb874003f30 msgr2=0x7fb88815ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:24.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.665+0000 7fb88febf640 1 --2- 172.21.15.67:0/1810847348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb874003f30 0x7fb88815ab70 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7fb8840119a0 tx=0x7fb884011e60 comp rx=0 tx=0).stop 2024-03-24T00:47:24.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.665+0000 7fb88febf640 1 -- 172.21.15.67:0/1810847348 shutdown_connections 2024-03-24T00:47:24.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.665+0000 7fb88febf640 1 --2- 172.21.15.67:0/1810847348 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb86c06bbd0 0x7fb86c06e090 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:24.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.665+0000 7fb88febf640 1 --2- 172.21.15.67:0/1810847348 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb8880b39e0 0x7fb8880b3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:24.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.665+0000 7fb88febf640 1 --2- 172.21.15.67:0/1810847348 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb888154f90 0x7fb888160d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:24.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.665+0000 7fb88febf640 1 --2- 172.21.15.67:0/1810847348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb874003f30 0x7fb88815ab70 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:24.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.665+0000 7fb88febf640 1 -- 172.21.15.67:0/1810847348 >> 172.21.15.67:0/1810847348 conn(0x7fb888072da0 msgr2=0x7fb8880b52c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:24.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.665+0000 7fb88febf640 1 -- 172.21.15.67:0/1810847348 shutdown_connections 2024-03-24T00:47:24.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.665+0000 7fb88febf640 1 -- 172.21.15.67:0/1810847348 wait complete. 2024-03-24T00:47:24.675 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T00:47:24.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.778+0000 7f659f47e640 1 Processor -- start 2024-03-24T00:47:24.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.778+0000 7f659f47e640 1 -- start start 2024-03-24T00:47:24.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.779+0000 7f659f47e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f659806de60 0x7f659809ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:24.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.779+0000 7f659f47e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f65980b1eb0 0x7f6598099d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:24.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.779+0000 7f659f47e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65980b29b0 con 0x7f659809a240 2024-03-24T00:47:24.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.779+0000 7f659f47e640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65980cc9e0 con 0x7f659806de60 2024-03-24T00:47:24.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.779+0000 7f659f47e640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65980ccf80 con 0x7f65980b1eb0 2024-03-24T00:47:24.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.779+0000 7f659d21a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f659806de60 0x7f659809ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:24.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.779+0000 7f659da1b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f659809a240 0x7f65980cff00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45290/0 (socket says 172.21.15.67:45290) 2024-03-24T00:47:24.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.779+0000 7f659da1b640 1 -- 172.21.15.67:0/3601054602 learned_addr learned my addr 172.21.15.67:0/3601054602 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:47:24.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.779+0000 7f659ca19640 1 --2- 172.21.15.67:0/3601054602 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f65980b1eb0 0x7f6598099d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:24.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.779+0000 7f658ffff640 1 -- 172.21.15.67:0/3601054602 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 820071705 0 0) 0x7f65980b29b0 con 0x7f659809a240 2024-03-24T00:47:24.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.779+0000 7f658ffff640 1 -- 172.21.15.67:0/3601054602 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6584003520 con 0x7f659809a240 2024-03-24T00:47:24.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.779+0000 7f659d21a640 1 -- 172.21.15.67:0/3601054602 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f65980b1eb0 msgr2=0x7f6598099d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:24.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.779+0000 7f659d21a640 1 --2- 172.21.15.67:0/3601054602 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f65980b1eb0 0x7f6598099d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:24.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.779+0000 7f659d21a640 1 -- 172.21.15.67:0/3601054602 >> v1:172.21.15.67:6789/0 conn(0x7f659809a240 legacy=0x7f65980cff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:24.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.779+0000 7f659d21a640 1 -- 172.21.15.67:0/3601054602 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65980b23f0 con 0x7f659806de60 2024-03-24T00:47:24.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.780+0000 7f659d21a640 1 --2- 172.21.15.67:0/3601054602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f659806de60 0x7f659809ae80 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f65940097a0 tx=0x7f659402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bdbcef8c8750645 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:24.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.780+0000 7f658ffff640 1 -- 172.21.15.67:0/3601054602 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6594035020 con 0x7f659806de60 2024-03-24T00:47:24.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.780+0000 7f658ffff640 1 -- 172.21.15.67:0/3601054602 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6594031830 con 0x7f659806de60 2024-03-24T00:47:24.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.780+0000 7f658ffff640 1 -- 172.21.15.67:0/3601054602 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6594031c00 con 0x7f659806de60 2024-03-24T00:47:24.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.781+0000 7f659f47e640 1 -- 172.21.15.67:0/3601054602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f659806de60 msgr2=0x7f659809ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:24.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.781+0000 7f659f47e640 1 --2- 172.21.15.67:0/3601054602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f659806de60 0x7f659809ae80 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f65940097a0 tx=0x7f659402b8d0 comp rx=0 tx=0).stop 2024-03-24T00:47:24.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.781+0000 7f659f47e640 1 -- 172.21.15.67:0/3601054602 shutdown_connections 2024-03-24T00:47:24.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.781+0000 7f659f47e640 1 --2- 172.21.15.67:0/3601054602 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f65980b1eb0 0x7f6598099d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:24.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.781+0000 7f659f47e640 1 --2- 172.21.15.67:0/3601054602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f659806de60 0x7f659809ae80 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:24.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.781+0000 7f659f47e640 1 -- 172.21.15.67:0/3601054602 >> 172.21.15.67:0/3601054602 conn(0x7f65980b3650 msgr2=0x7f65980b3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:24.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.781+0000 7f659f47e640 1 -- 172.21.15.67:0/3601054602 shutdown_connections 2024-03-24T00:47:24.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.781+0000 7f659f47e640 1 -- 172.21.15.67:0/3601054602 wait complete. 2024-03-24T00:47:24.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.782+0000 7f659f47e640 1 Processor -- start 2024-03-24T00:47:24.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.782+0000 7f659f47e640 1 -- start start 2024-03-24T00:47:24.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.783+0000 7f659f47e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f659806de60 0x7f6598150b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:24.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.783+0000 7f659f47e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f659809a240 0x7f6598070f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:24.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.783+0000 7f659f47e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f65980b1eb0 0x7f6598071440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:24.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.783+0000 7f659f47e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f659809b3c0 con 0x7f659809a240 2024-03-24T00:47:24.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.783+0000 7f659f47e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65980ccc10 con 0x7f65980b1eb0 2024-03-24T00:47:24.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.783+0000 7f659f47e640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f659806c150 con 0x7f659806de60 2024-03-24T00:47:24.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.783+0000 7f659ca19640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f659809a240 0x7f6598070f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:24.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.783+0000 7f659ca19640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f659809a240 0x7f6598070f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60310/0 (socket says 172.21.15.67:60310) 2024-03-24T00:47:24.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.783+0000 7f659ca19640 1 -- 172.21.15.67:0/3691209318 learned_addr learned my addr 172.21.15.67:0/3691209318 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:24.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.783+0000 7f659da1b640 1 --2- 172.21.15.67:0/3691209318 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f65980b1eb0 0x7f6598071440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:24.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.783+0000 7f659d21a640 1 --2- 172.21.15.67:0/3691209318 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f659806de60 0x7f6598150b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:24.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.783+0000 7f659ca19640 1 -- 172.21.15.67:0/3691209318 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f659806de60 msgr2=0x7f6598150b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:24.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.783+0000 7f659ca19640 1 --2- 172.21.15.67:0/3691209318 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f659806de60 0x7f6598150b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:24.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.783+0000 7f659ca19640 1 -- 172.21.15.67:0/3691209318 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f65980b1eb0 msgr2=0x7f6598071440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:24.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.783+0000 7f659ca19640 1 --2- 172.21.15.67:0/3691209318 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f65980b1eb0 0x7f6598071440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:24.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.783+0000 7f659ca19640 1 -- 172.21.15.67:0/3691209318 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6598151560 con 0x7f659809a240 2024-03-24T00:47:24.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.783+0000 7f659da1b640 1 --2- 172.21.15.67:0/3691209318 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f65980b1eb0 0x7f6598071440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:47:24.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.783+0000 7f659ca19640 1 --2- 172.21.15.67:0/3691209318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f659809a240 0x7f6598070f00 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7f658800e870 tx=0x7f658800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:24.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.783+0000 7f658dffb640 1 -- 172.21.15.67:0/3691209318 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6588017070 con 0x7f659809a240 2024-03-24T00:47:24.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.783+0000 7f658dffb640 1 -- 172.21.15.67:0/3691209318 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f658800c940 con 0x7f659809a240 2024-03-24T00:47:24.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.783+0000 7f658dffb640 1 -- 172.21.15.67:0/3691209318 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f658800cd30 con 0x7f659809a240 2024-03-24T00:47:24.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.783+0000 7f659f47e640 1 -- 172.21.15.67:0/3691209318 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6598151840 con 0x7f659809a240 2024-03-24T00:47:24.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.784+0000 7f659f47e640 1 -- 172.21.15.67:0/3691209318 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6598151e00 con 0x7f659809a240 2024-03-24T00:47:24.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.785+0000 7f659f47e640 1 -- 172.21.15.67:0/3691209318 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f659804f400 con 0x7f659809a240 2024-03-24T00:47:24.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.786+0000 7f658dffb640 1 -- 172.21.15.67:0/3691209318 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f658801a070 con 0x7f659809a240 2024-03-24T00:47:24.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.786+0000 7f658dffb640 1 --2- 172.21.15.67:0/3691209318 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f657c06bbd0 0x7f657c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:24.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.786+0000 7f658dffb640 1 -- 172.21.15.67:0/3691209318 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f658808aea0 con 0x7f659809a240 2024-03-24T00:47:24.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.786+0000 7f659d21a640 1 --2- 172.21.15.67:0/3691209318 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f657c06bbd0 0x7f657c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:24.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.787+0000 7f659d21a640 1 --2- 172.21.15.67:0/3691209318 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f657c06bbd0 0x7f657c06e090 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f6594036000 tx=0x7f6594008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:24.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.791+0000 7f658dffb640 1 -- 172.21.15.67:0/3691209318 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6588059800 con 0x7f659809a240 2024-03-24T00:47:24.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.985+0000 7f659f47e640 1 -- 172.21.15.67:0/3691209318 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f65980b3db0 con 0x7f659809a240 2024-03-24T00:47:24.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.985+0000 7f658dffb640 1 -- 172.21.15.67:0/3691209318 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v105) v1 ==== 80+0+0 (secure 0 0 0) 0x7f6588059100 con 0x7f659809a240 2024-03-24T00:47:24.989 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-03-24T00:47:24.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.987+0000 7f659f47e640 1 -- 172.21.15.67:0/3691209318 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f657c06bbd0 msgr2=0x7f657c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:24.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.987+0000 7f659f47e640 1 --2- 172.21.15.67:0/3691209318 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f657c06bbd0 0x7f657c06e090 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f6594036000 tx=0x7f6594008030 comp rx=0 tx=0).stop 2024-03-24T00:47:24.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.987+0000 7f659f47e640 1 -- 172.21.15.67:0/3691209318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f659809a240 msgr2=0x7f6598070f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:24.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.987+0000 7f659f47e640 1 --2- 172.21.15.67:0/3691209318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f659809a240 0x7f6598070f00 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7f658800e870 tx=0x7f658800ed30 comp rx=0 tx=0).stop 2024-03-24T00:47:24.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.987+0000 7f659f47e640 1 -- 172.21.15.67:0/3691209318 shutdown_connections 2024-03-24T00:47:24.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.987+0000 7f659f47e640 1 --2- 172.21.15.67:0/3691209318 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f657c06bbd0 0x7f657c06e090 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:24.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.987+0000 7f659f47e640 1 --2- 172.21.15.67:0/3691209318 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f65980b1eb0 0x7f6598071440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:24.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.987+0000 7f659f47e640 1 --2- 172.21.15.67:0/3691209318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f659809a240 0x7f6598070f00 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:24.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.987+0000 7f659f47e640 1 --2- 172.21.15.67:0/3691209318 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f659806de60 0x7f6598150b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:24.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.987+0000 7f659f47e640 1 -- 172.21.15.67:0/3691209318 >> 172.21.15.67:0/3691209318 conn(0x7f65980b3650 msgr2=0x7f65980b0760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:24.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.987+0000 7f659f47e640 1 -- 172.21.15.67:0/3691209318 shutdown_connections 2024-03-24T00:47:24.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:24.987+0000 7f659f47e640 1 -- 172.21.15.67:0/3691209318 wait complete. 2024-03-24T00:47:24.998 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-24T00:47:24.998 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:47:24.998 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-24T00:47:25.030 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-03-24T00:47:25.030 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:47:25.030 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-24T00:47:25.085 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:47:25.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.261+0000 7fdfa6ccc640 1 Processor -- start 2024-03-24T00:47:25.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.261+0000 7fdfa6ccc640 1 -- start start 2024-03-24T00:47:25.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.263+0000 7fdfa6ccc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdfa00cc360 0x7fdfa0066e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:25.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.263+0000 7fdfa6ccc640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdfa006ba50 0x7fdfa00ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:25.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.263+0000 7fdfa6ccc640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfa00b1c40 con 0x7fdfa00cd240 2024-03-24T00:47:25.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.263+0000 7fdfa6ccc640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfa00cd6b0 con 0x7fdfa00cc360 2024-03-24T00:47:25.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.263+0000 7fdfa6ccc640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa00caad0 con 0x7fdfa006ba50 2024-03-24T00:47:25.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.263+0000 7fdfa4a68640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdfa00cc360 0x7fdfa0066e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:25.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.263+0000 7fdfa4a68640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdfa00cc360 0x7fdfa0066e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60318/0 (socket says 172.21.15.67:60318) 2024-03-24T00:47:25.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.263+0000 7fdfa4a68640 1 -- 172.21.15.67:0/4020389761 learned_addr learned my addr 172.21.15.67:0/4020389761 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:25.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.263+0000 7fdf9ffff640 1 --2- 172.21.15.67:0/4020389761 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdfa006ba50 0x7fdfa00ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:25.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.263+0000 7fdfa4a68640 1 -- 172.21.15.67:0/4020389761 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdfa006ba50 msgr2=0x7fdfa00ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:25.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.263+0000 7fdfa4a68640 1 --2- 172.21.15.67:0/4020389761 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdfa006ba50 0x7fdfa00ccd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:25.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.263+0000 7fdfa4a68640 1 -- 172.21.15.67:0/4020389761 >> v1:172.21.15.72:6789/0 conn(0x7fdfa00cd240 legacy=0x7fdfa00d0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:25.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.263+0000 7fdfa4a68640 1 -- 172.21.15.67:0/4020389761 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa006c210 con 0x7fdfa00cc360 2024-03-24T00:47:25.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.264+0000 7fdfa4a68640 1 --2- 172.21.15.67:0/4020389761 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfa00cc360 0x7fdfa0066e40 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7fdf90009870 tx=0x7fdf9002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae22176fc7dbd62f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:25.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.264+0000 7fdf9f7fe640 1 -- 172.21.15.67:0/4020389761 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf90034070 con 0x7fdfa00cc360 2024-03-24T00:47:25.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.264+0000 7fdf9f7fe640 1 -- 172.21.15.67:0/4020389761 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf90030840 con 0x7fdfa00cc360 2024-03-24T00:47:25.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.264+0000 7fdf9f7fe640 1 -- 172.21.15.67:0/4020389761 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf90030c10 con 0x7fdfa00cc360 2024-03-24T00:47:25.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.264+0000 7fdfa6ccc640 1 -- 172.21.15.67:0/4020389761 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfa00cc360 msgr2=0x7fdfa0066e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:25.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.264+0000 7fdfa6ccc640 1 --2- 172.21.15.67:0/4020389761 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfa00cc360 0x7fdfa0066e40 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7fdf90009870 tx=0x7fdf9002b980 comp rx=0 tx=0).stop 2024-03-24T00:47:25.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.265+0000 7fdfa6ccc640 1 -- 172.21.15.67:0/4020389761 shutdown_connections 2024-03-24T00:47:25.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.265+0000 7fdfa6ccc640 1 --2- 172.21.15.67:0/4020389761 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdfa006ba50 0x7fdfa00ccd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:25.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.265+0000 7fdfa6ccc640 1 --2- 172.21.15.67:0/4020389761 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfa00cc360 0x7fdfa0066e40 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:25.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.265+0000 7fdfa6ccc640 1 -- 172.21.15.67:0/4020389761 >> 172.21.15.67:0/4020389761 conn(0x7fdfa00b5440 msgr2=0x7fdfa00b33d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:25.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.265+0000 7fdfa6ccc640 1 -- 172.21.15.67:0/4020389761 shutdown_connections 2024-03-24T00:47:25.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.265+0000 7fdfa6ccc640 1 -- 172.21.15.67:0/4020389761 wait complete. 2024-03-24T00:47:25.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.266+0000 7fdfa6ccc640 1 Processor -- start 2024-03-24T00:47:25.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.266+0000 7fdfa6ccc640 1 -- start start 2024-03-24T00:47:25.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.267+0000 7fdfa6ccc640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdfa006ba50 0x7fdfa014f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:25.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.267+0000 7fdfa6ccc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdfa00cc360 0x7fdfa015d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:25.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.267+0000 7fdfa6ccc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfa00cd240 0x7fdfa015d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:25.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.267+0000 7fdfa6ccc640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa00b3c80 con 0x7fdfa00cd240 2024-03-24T00:47:25.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.267+0000 7fdfa6ccc640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa006bf30 con 0x7fdfa00cc360 2024-03-24T00:47:25.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.267+0000 7fdfa6ccc640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa00b3f60 con 0x7fdfa006ba50 2024-03-24T00:47:25.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.267+0000 7fdfa5269640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfa00cd240 0x7fdfa015d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:25.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.267+0000 7fdfa5269640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfa00cd240 0x7fdfa015d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60332/0 (socket says 172.21.15.67:60332) 2024-03-24T00:47:25.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.267+0000 7fdfa5269640 1 -- 172.21.15.67:0/3624085556 learned_addr learned my addr 172.21.15.67:0/3624085556 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:25.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.267+0000 7fdfa4a68640 1 --2- 172.21.15.67:0/3624085556 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdfa006ba50 0x7fdfa014f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:25.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.267+0000 7fdf9ffff640 1 --2- 172.21.15.67:0/3624085556 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdfa00cc360 0x7fdfa015d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:25.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.267+0000 7fdfa5269640 1 -- 172.21.15.67:0/3624085556 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdfa006ba50 msgr2=0x7fdfa014f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:25.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.267+0000 7fdfa5269640 1 --2- 172.21.15.67:0/3624085556 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdfa006ba50 0x7fdfa014f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:25.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.267+0000 7fdfa5269640 1 -- 172.21.15.67:0/3624085556 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdfa00cc360 msgr2=0x7fdfa015d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:25.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.267+0000 7fdfa5269640 1 --2- 172.21.15.67:0/3624085556 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdfa00cc360 0x7fdfa015d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:25.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.267+0000 7fdfa5269640 1 -- 172.21.15.67:0/3624085556 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa00c7550 con 0x7fdfa00cd240 2024-03-24T00:47:25.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.267+0000 7fdfa5269640 1 --2- 172.21.15.67:0/3624085556 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfa00cd240 0x7fdfa015d680 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7fdf9800c7a0 tx=0x7fdf9800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:25.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.267+0000 7fdf9d7fa640 1 -- 172.21.15.67:0/3624085556 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf98003b70 con 0x7fdfa00cd240 2024-03-24T00:47:25.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.267+0000 7fdf9d7fa640 1 -- 172.21.15.67:0/3624085556 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf98003d30 con 0x7fdfa00cd240 2024-03-24T00:47:25.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.267+0000 7fdfa6ccc640 1 -- 172.21.15.67:0/3624085556 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfa00c9710 con 0x7fdfa00cd240 2024-03-24T00:47:25.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.268+0000 7fdf9d7fa640 1 -- 172.21.15.67:0/3624085556 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf9800aa90 con 0x7fdfa00cd240 2024-03-24T00:47:25.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.268+0000 7fdfa6ccc640 1 -- 172.21.15.67:0/3624085556 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfa00c7af0 con 0x7fdfa00cd240 2024-03-24T00:47:25.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.269+0000 7fdfa6ccc640 1 -- 172.21.15.67:0/3624085556 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfa004f400 con 0x7fdfa00cd240 2024-03-24T00:47:25.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.269+0000 7fdf9d7fa640 1 -- 172.21.15.67:0/3624085556 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fdf9800ac30 con 0x7fdfa00cd240 2024-03-24T00:47:25.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.269+0000 7fdf9d7fa640 1 --2- 172.21.15.67:0/3624085556 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdf8c06bbd0 0x7fdf8c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:25.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.269+0000 7fdf9d7fa640 1 -- 172.21.15.67:0/3624085556 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fdf9808ab40 con 0x7fdfa00cd240 2024-03-24T00:47:25.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.270+0000 7fdfa4a68640 1 --2- 172.21.15.67:0/3624085556 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdf8c06bbd0 0x7fdf8c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:25.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.270+0000 7fdfa4a68640 1 --2- 172.21.15.67:0/3624085556 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdf8c06bbd0 0x7fdf8c06e090 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fdf90036000 tx=0x7fdf9003b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:25.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.274+0000 7fdf9d7fa640 1 -- 172.21.15.67:0/3624085556 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fdf980594a0 con 0x7fdfa00cd240 2024-03-24T00:47:25.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.496+0000 7fdfa6ccc640 1 -- 172.21.15.67:0/3624085556 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdfa0072b30 con 0x7fdfa00cd240 2024-03-24T00:47:25.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.496+0000 7fdf9d7fa640 1 -- 172.21.15.67:0/3624085556 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+4976 (secure 0 0 0) 0x7fdf98058da0 con 0x7fdfa00cd240 2024-03-24T00:47:25.496 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:47:25.497 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":72,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6902,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3672294289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3672294289},{"type":"v1","addr":"172.21.15.67:6835","nonce":3672294289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":6908,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/1286271878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":1286271878},{"type":"v1","addr":"172.21.15.72:6836","nonce":1286271878}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":6911,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/140622129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":140622129},{"type":"v1","addr":"172.21.15.72:6837","nonce":140622129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:47:16.492485+0000","modified":"2024-03-24T00:47:18.507677+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8541},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8541":{"gid":8541,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6833/1826974843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1826974843},{"type":"v1","addr":"172.21.15.67:6833","nonce":1826974843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8541,"qdb_cluster":[8541]},"id":7}]} 2024-03-24T00:47:25.497 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 72 2024-03-24T00:47:25.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.499+0000 7fdfa6ccc640 1 -- 172.21.15.67:0/3624085556 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdf8c06bbd0 msgr2=0x7fdf8c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:25.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.499+0000 7fdfa6ccc640 1 --2- 172.21.15.67:0/3624085556 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdf8c06bbd0 0x7fdf8c06e090 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fdf90036000 tx=0x7fdf9003b040 comp rx=0 tx=0).stop 2024-03-24T00:47:25.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.499+0000 7fdfa6ccc640 1 -- 172.21.15.67:0/3624085556 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfa00cd240 msgr2=0x7fdfa015d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:25.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.499+0000 7fdfa6ccc640 1 --2- 172.21.15.67:0/3624085556 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfa00cd240 0x7fdfa015d680 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7fdf9800c7a0 tx=0x7fdf9800cc60 comp rx=0 tx=0).stop 2024-03-24T00:47:25.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.499+0000 7fdfa6ccc640 1 -- 172.21.15.67:0/3624085556 shutdown_connections 2024-03-24T00:47:25.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.499+0000 7fdfa6ccc640 1 --2- 172.21.15.67:0/3624085556 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdf8c06bbd0 0x7fdf8c06e090 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:25.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.499+0000 7fdfa6ccc640 1 --2- 172.21.15.67:0/3624085556 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfa00cd240 0x7fdfa015d680 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:25.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.499+0000 7fdfa6ccc640 1 --2- 172.21.15.67:0/3624085556 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdfa00cc360 0x7fdfa015d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:25.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.499+0000 7fdfa6ccc640 1 --2- 172.21.15.67:0/3624085556 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdfa006ba50 0x7fdfa014f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:25.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.499+0000 7fdfa6ccc640 1 -- 172.21.15.67:0/3624085556 >> 172.21.15.67:0/3624085556 conn(0x7fdfa00b5440 msgr2=0x7fdfa00af250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:25.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.499+0000 7fdfa6ccc640 1 -- 172.21.15.67:0/3624085556 shutdown_connections 2024-03-24T00:47:25.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.499+0000 7fdfa6ccc640 1 -- 172.21.15.67:0/3624085556 wait complete. 2024-03-24T00:47:25.510 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-24T00:47:25.632 INFO:teuthology.orchestra.run.smithi067.stdout:{"message":"","return_code":0} 2024-03-24T00:47:25.632 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-24T00:47:25.632 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:47:25.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.733+0000 7fded3f05640 1 Processor -- start 2024-03-24T00:47:25.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.733+0000 7fded3f05640 1 -- start start 2024-03-24T00:47:25.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.734+0000 7fded3f05640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdecc0cfc70 0x7fdecc0ce220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:25.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.734+0000 7fded3f05640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdecc0ce760 0x7fdecc0b1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:25.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.734+0000 7fded3f05640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdecc0ccfe0 con 0x7fdecc15bf70 2024-03-24T00:47:25.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.734+0000 7fded3f05640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdecc0ccda0 con 0x7fdecc0cfc70 2024-03-24T00:47:25.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.734+0000 7fded3f05640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdecc0cd2c0 con 0x7fdecc0ce760 2024-03-24T00:47:25.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.734+0000 7fded14a0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdecc0cfc70 0x7fdecc0ce220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:25.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.734+0000 7fded14a0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdecc0cfc70 0x7fdecc0ce220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60338/0 (socket says 172.21.15.67:60338) 2024-03-24T00:47:25.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.734+0000 7fded1ca1640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fdecc15bf70 0x7fdecc0cc690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45296/0 (socket says 172.21.15.67:45296) 2024-03-24T00:47:25.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.734+0000 7fded14a0640 1 -- 172.21.15.67:0/17691439 learned_addr learned my addr 172.21.15.67:0/17691439 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:25.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.735+0000 7fded24a2640 1 --2- 172.21.15.67:0/17691439 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdecc0ce760 0x7fdecc0b1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:25.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.735+0000 7fded14a0640 1 -- 172.21.15.67:0/17691439 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdecc0ce760 msgr2=0x7fdecc0b1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:25.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.735+0000 7fded14a0640 1 --2- 172.21.15.67:0/17691439 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdecc0ce760 0x7fdecc0b1cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:25.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.735+0000 7fded14a0640 1 -- 172.21.15.67:0/17691439 >> v1:172.21.15.67:6789/0 conn(0x7fdecc15bf70 legacy=0x7fdecc0cc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:25.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.735+0000 7fded14a0640 1 -- 172.21.15.67:0/17691439 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdecc0cd5a0 con 0x7fdecc0cfc70 2024-03-24T00:47:25.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.736+0000 7fded14a0640 1 --2- 172.21.15.67:0/17691439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdecc0cfc70 0x7fdecc0ce220 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7fdebc009870 tx=0x7fdebc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eca660e28bafadda server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:25.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.736+0000 7fded0c9f640 1 -- 172.21.15.67:0/17691439 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdebc034070 con 0x7fdecc0cfc70 2024-03-24T00:47:25.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.736+0000 7fded0c9f640 1 -- 172.21.15.67:0/17691439 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdebc030840 con 0x7fdecc0cfc70 2024-03-24T00:47:25.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.736+0000 7fded0c9f640 1 -- 172.21.15.67:0/17691439 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdebc030c10 con 0x7fdecc0cfc70 2024-03-24T00:47:25.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.736+0000 7fded3f05640 1 -- 172.21.15.67:0/17691439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdecc0cfc70 msgr2=0x7fdecc0ce220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:25.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.736+0000 7fded3f05640 1 --2- 172.21.15.67:0/17691439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdecc0cfc70 0x7fdecc0ce220 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7fdebc009870 tx=0x7fdebc02b980 comp rx=0 tx=0).stop 2024-03-24T00:47:25.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.736+0000 7fded3f05640 1 -- 172.21.15.67:0/17691439 shutdown_connections 2024-03-24T00:47:25.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.736+0000 7fded3f05640 1 --2- 172.21.15.67:0/17691439 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdecc0ce760 0x7fdecc0b1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:25.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.736+0000 7fded3f05640 1 --2- 172.21.15.67:0/17691439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdecc0cfc70 0x7fdecc0ce220 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:25.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.736+0000 7fded3f05640 1 -- 172.21.15.67:0/17691439 >> 172.21.15.67:0/17691439 conn(0x7fdecc0d1bf0 msgr2=0x7fdecc0d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:25.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.736+0000 7fded3f05640 1 -- 172.21.15.67:0/17691439 shutdown_connections 2024-03-24T00:47:25.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.736+0000 7fded3f05640 1 -- 172.21.15.67:0/17691439 wait complete. 2024-03-24T00:47:25.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.737+0000 7fded3f05640 1 Processor -- start 2024-03-24T00:47:25.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.738+0000 7fded3f05640 1 -- start start 2024-03-24T00:47:25.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.738+0000 7fded3f05640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdecc0ce760 0x7fdecc0cbea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:25.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.738+0000 7fded3f05640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdecc0cfc70 0x7fdecc0b47e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:25.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.738+0000 7fded3f05640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdecc15bf70 0x7fdecc0af0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:25.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.738+0000 7fded3f05640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdecc0b2210 con 0x7fdecc0cfc70 2024-03-24T00:47:25.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.738+0000 7fded3f05640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdecc15c350 con 0x7fdecc0ce760 2024-03-24T00:47:25.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.738+0000 7fded3f05640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdecc0b2390 con 0x7fdecc15bf70 2024-03-24T00:47:25.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.738+0000 7fded14a0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdecc0cfc70 0x7fdecc0b47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:25.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.738+0000 7fded14a0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdecc0cfc70 0x7fdecc0b47e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60342/0 (socket says 172.21.15.67:60342) 2024-03-24T00:47:25.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.738+0000 7fded14a0640 1 -- 172.21.15.67:0/4052761786 learned_addr learned my addr 172.21.15.67:0/4052761786 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:25.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.739+0000 7fded24a2640 1 --2- 172.21.15.67:0/4052761786 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdecc15bf70 0x7fdecc0af0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:25.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.739+0000 7fded1ca1640 1 --2- 172.21.15.67:0/4052761786 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdecc0ce760 0x7fdecc0cbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:25.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.739+0000 7fded14a0640 1 -- 172.21.15.67:0/4052761786 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdecc15bf70 msgr2=0x7fdecc0af0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:25.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.739+0000 7fded14a0640 1 --2- 172.21.15.67:0/4052761786 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdecc15bf70 0x7fdecc0af0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:25.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.739+0000 7fded14a0640 1 -- 172.21.15.67:0/4052761786 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdecc0ce760 msgr2=0x7fdecc0cbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:25.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.739+0000 7fded14a0640 1 --2- 172.21.15.67:0/4052761786 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdecc0ce760 0x7fdecc0cbea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:25.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.739+0000 7fded14a0640 1 -- 172.21.15.67:0/4052761786 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdecc0b4f50 con 0x7fdecc0cfc70 2024-03-24T00:47:25.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.739+0000 7fded24a2640 1 --2- 172.21.15.67:0/4052761786 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdecc15bf70 0x7fdecc0af0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:47:25.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.739+0000 7fded1ca1640 1 --2- 172.21.15.67:0/4052761786 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdecc0ce760 0x7fdecc0cbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:47:25.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.739+0000 7fded14a0640 1 --2- 172.21.15.67:0/4052761786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdecc0cfc70 0x7fdecc0b47e0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7fdebc02b9c0 tx=0x7fdebc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:25.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.739+0000 7fdec27fc640 1 -- 172.21.15.67:0/4052761786 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdebc048070 con 0x7fdecc0cfc70 2024-03-24T00:47:25.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.739+0000 7fdec27fc640 1 -- 172.21.15.67:0/4052761786 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdebc030640 con 0x7fdecc0cfc70 2024-03-24T00:47:25.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.739+0000 7fded3f05640 1 -- 172.21.15.67:0/4052761786 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdecc0b5230 con 0x7fdecc0cfc70 2024-03-24T00:47:25.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.739+0000 7fdec27fc640 1 -- 172.21.15.67:0/4052761786 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdebc034040 con 0x7fdecc0cfc70 2024-03-24T00:47:25.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.739+0000 7fded3f05640 1 -- 172.21.15.67:0/4052761786 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdecc0b57f0 con 0x7fdecc0cfc70 2024-03-24T00:47:25.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.740+0000 7fded3f05640 1 -- 172.21.15.67:0/4052761786 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdecc04f400 con 0x7fdecc0cfc70 2024-03-24T00:47:25.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.741+0000 7fdec27fc640 1 -- 172.21.15.67:0/4052761786 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fdebc04b070 con 0x7fdecc0cfc70 2024-03-24T00:47:25.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.741+0000 7fdec27fc640 1 --2- 172.21.15.67:0/4052761786 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdeb006bbd0 0x7fdeb006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:25.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.741+0000 7fdec27fc640 1 -- 172.21.15.67:0/4052761786 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fdebc0acb50 con 0x7fdecc0cfc70 2024-03-24T00:47:25.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.742+0000 7fded1ca1640 1 --2- 172.21.15.67:0/4052761786 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdeb006bbd0 0x7fdeb006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:25.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.742+0000 7fded1ca1640 1 --2- 172.21.15.67:0/4052761786 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdeb006bbd0 0x7fdeb006e090 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fdec8000f60 tx=0x7fdec8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:25.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:25.746+0000 7fdec27fc640 1 -- 172.21.15.67:0/4052761786 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fdebc07b5d0 con 0x7fdecc0cfc70 2024-03-24T00:47:26.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.058+0000 7fded3f05640 1 -- 172.21.15.67:0/4052761786 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdecc0c8670 con 0x7fdecc0cfc70 2024-03-24T00:47:26.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.058+0000 7fdec27fc640 1 -- 172.21.15.67:0/4052761786 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+4976 (secure 0 0 0) 0x7fdebc030810 con 0x7fdecc0cfc70 2024-03-24T00:47:26.058 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:47:26.058 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":72,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6902,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3672294289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3672294289},{"type":"v1","addr":"172.21.15.67:6835","nonce":3672294289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":6908,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/1286271878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":1286271878},{"type":"v1","addr":"172.21.15.72:6836","nonce":1286271878}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":6911,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/140622129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":140622129},{"type":"v1","addr":"172.21.15.72:6837","nonce":140622129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:47:16.492485+0000","modified":"2024-03-24T00:47:18.507677+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8541},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8541":{"gid":8541,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6833/1826974843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1826974843},{"type":"v1","addr":"172.21.15.67:6833","nonce":1826974843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8541,"qdb_cluster":[8541]},"id":7}]} 2024-03-24T00:47:26.059 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 72 2024-03-24T00:47:26.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.060+0000 7fded3f05640 1 -- 172.21.15.67:0/4052761786 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdeb006bbd0 msgr2=0x7fdeb006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:26.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.060+0000 7fded3f05640 1 --2- 172.21.15.67:0/4052761786 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdeb006bbd0 0x7fdeb006e090 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fdec8000f60 tx=0x7fdec8009040 comp rx=0 tx=0).stop 2024-03-24T00:47:26.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.060+0000 7fded3f05640 1 -- 172.21.15.67:0/4052761786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdecc0cfc70 msgr2=0x7fdecc0b47e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:26.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.060+0000 7fded3f05640 1 --2- 172.21.15.67:0/4052761786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdecc0cfc70 0x7fdecc0b47e0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7fdebc02b9c0 tx=0x7fdebc004730 comp rx=0 tx=0).stop 2024-03-24T00:47:26.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.060+0000 7fded3f05640 1 -- 172.21.15.67:0/4052761786 shutdown_connections 2024-03-24T00:47:26.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.060+0000 7fded3f05640 1 --2- 172.21.15.67:0/4052761786 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdeb006bbd0 0x7fdeb006e090 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:26.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.060+0000 7fded3f05640 1 --2- 172.21.15.67:0/4052761786 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdecc15bf70 0x7fdecc0af0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:26.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.060+0000 7fded3f05640 1 --2- 172.21.15.67:0/4052761786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdecc0cfc70 0x7fdecc0b47e0 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:26.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.060+0000 7fded3f05640 1 --2- 172.21.15.67:0/4052761786 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdecc0ce760 0x7fdecc0cbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:26.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.060+0000 7fded3f05640 1 -- 172.21.15.67:0/4052761786 >> 172.21.15.67:0/4052761786 conn(0x7fdecc0d1bf0 msgr2=0x7fdecc0b0810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:26.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.060+0000 7fded3f05640 1 -- 172.21.15.67:0/4052761786 shutdown_connections 2024-03-24T00:47:26.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.060+0000 7fded3f05640 1 -- 172.21.15.67:0/4052761786 wait complete. 2024-03-24T00:47:26.071 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata get mds0_inotable - 2024-03-24T00:47:26.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.113+0000 7faa9e235f40 1 Processor -- start 2024-03-24T00:47:26.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.113+0000 7faa9e235f40 1 -- start start 2024-03-24T00:47:26.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.114+0000 7faa9e235f40 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558e79e32210 0x558e79ec1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:26.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.114+0000 7faa9e235f40 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558e79e325f0 0x558e79ebfcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:26.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.114+0000 7faa9e235f40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558e79eb88a0 0x558e79ec0210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:26.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.114+0000 7faa9e235f40 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e79ebf050 con 0x558e79e325f0 2024-03-24T00:47:26.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.114+0000 7faa9e235f40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e79d900d0 con 0x558e79eb88a0 2024-03-24T00:47:26.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.114+0000 7faa9e235f40 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x558e79ebf1d0 con 0x558e79e32210 2024-03-24T00:47:26.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.114+0000 7faa93fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558e79e325f0 0x558e79ebfcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:26.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.114+0000 7faa93fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558e79e325f0 0x558e79ebfcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60368/0 (socket says 172.21.15.67:60368) 2024-03-24T00:47:26.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.114+0000 7faa93fff640 1 -- 172.21.15.67:0/1215670776 learned_addr learned my addr 172.21.15.67:0/1215670776 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:26.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.114+0000 7faa93fff640 1 -- 172.21.15.67:0/1215670776 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558e79e32210 msgr2=0x558e79ec1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:26.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.114+0000 7faa93fff640 1 --2- 172.21.15.67:0/1215670776 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558e79e32210 0x558e79ec1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:26.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.114+0000 7faa93fff640 1 -- 172.21.15.67:0/1215670776 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558e79eb88a0 msgr2=0x558e79ec0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:26.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.114+0000 7faa93fff640 1 --2- 172.21.15.67:0/1215670776 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558e79eb88a0 0x558e79ec0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:26.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.114+0000 7faa93fff640 1 -- 172.21.15.67:0/1215670776 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558e79ec0750 con 0x558e79e325f0 2024-03-24T00:47:26.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.114+0000 7faa93fff640 1 --2- 172.21.15.67:0/1215670776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558e79e325f0 0x558e79ebfcd0 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7faa8400a440 tx=0x7faa8404c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6c355f8a978c22c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:26.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.114+0000 7faa9a4db640 1 -- 172.21.15.67:0/1215670776 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa84057070 con 0x558e79e325f0 2024-03-24T00:47:26.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.114+0000 7faa9a4db640 1 -- 172.21.15.67:0/1215670776 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa84061070 con 0x558e79e325f0 2024-03-24T00:47:26.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.114+0000 7faa9a4db640 1 -- 172.21.15.67:0/1215670776 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa84052900 con 0x558e79e325f0 2024-03-24T00:47:26.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.115+0000 7faa9e235f40 1 -- 172.21.15.67:0/1215670776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558e79e325f0 msgr2=0x558e79ebfcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:26.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.115+0000 7faa9e235f40 1 --2- 172.21.15.67:0/1215670776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558e79e325f0 0x558e79ebfcd0 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7faa8400a440 tx=0x7faa8404c000 comp rx=0 tx=0).stop 2024-03-24T00:47:26.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.115+0000 7faa9e235f40 1 -- 172.21.15.67:0/1215670776 shutdown_connections 2024-03-24T00:47:26.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.115+0000 7faa9e235f40 1 --2- 172.21.15.67:0/1215670776 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558e79eb88a0 0x558e79ec0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:26.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.115+0000 7faa9e235f40 1 --2- 172.21.15.67:0/1215670776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558e79e325f0 0x558e79ebfcd0 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:26.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.115+0000 7faa9e235f40 1 --2- 172.21.15.67:0/1215670776 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558e79e32210 0x558e79ec1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:26.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.115+0000 7faa9e235f40 1 -- 172.21.15.67:0/1215670776 >> 172.21.15.67:0/1215670776 conn(0x558e79e15320 msgr2=0x558e79e9e2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:26.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.115+0000 7faa9e235f40 1 -- 172.21.15.67:0/1215670776 shutdown_connections 2024-03-24T00:47:26.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.115+0000 7faa9e235f40 1 -- 172.21.15.67:0/1215670776 wait complete. 2024-03-24T00:47:26.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.115+0000 7faa9e235f40 1 Processor -- start 2024-03-24T00:47:26.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.115+0000 7faa9e235f40 1 -- start start 2024-03-24T00:47:26.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.116+0000 7faa9e235f40 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558e79e32210 0x558e79eb7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:26.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.116+0000 7faa9e235f40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558e79e325f0 0x558e79e13a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:26.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.116+0000 7faa9e235f40 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558e79eb88a0 0x558e79e13f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:26.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.116+0000 7faa9e235f40 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e79de81b0 con 0x558e79eb88a0 2024-03-24T00:47:26.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.116+0000 7faa9e235f40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e79ea2f90 con 0x558e79e325f0 2024-03-24T00:47:26.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.116+0000 7faa9e235f40 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x558e79de8790 con 0x558e79e32210 2024-03-24T00:47:26.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.116+0000 7faa9bcde640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558e79eb88a0 0x558e79e13f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:26.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.116+0000 7faa9bcde640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558e79eb88a0 0x558e79e13f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60382/0 (socket says 172.21.15.67:60382) 2024-03-24T00:47:26.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.116+0000 7faa9bcde640 1 -- 172.21.15.67:0/3493871595 learned_addr learned my addr 172.21.15.67:0/3493871595 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:26.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.116+0000 7faa9b4dd640 1 --2- 172.21.15.67:0/3493871595 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558e79e32210 0x558e79eb7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:26.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.116+0000 7faa93fff640 1 --2- 172.21.15.67:0/3493871595 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558e79e325f0 0x558e79e13a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:26.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.117+0000 7faa9bcde640 1 -- 172.21.15.67:0/3493871595 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558e79e32210 msgr2=0x558e79eb7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:26.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.117+0000 7faa9bcde640 1 --2- 172.21.15.67:0/3493871595 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558e79e32210 0x558e79eb7ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:26.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.117+0000 7faa9bcde640 1 -- 172.21.15.67:0/3493871595 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558e79e325f0 msgr2=0x558e79e13a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:26.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.117+0000 7faa9bcde640 1 --2- 172.21.15.67:0/3493871595 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558e79e325f0 0x558e79e13a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:26.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.117+0000 7faa9bcde640 1 -- 172.21.15.67:0/3493871595 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558e79e17b80 con 0x558e79eb88a0 2024-03-24T00:47:26.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.117+0000 7faa9bcde640 1 --2- 172.21.15.67:0/3493871595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558e79eb88a0 0x558e79e13f40 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7faa94005f80 tx=0x7faa9400a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:26.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.117+0000 7faa994d9640 1 -- 172.21.15.67:0/3493871595 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa94013430 con 0x558e79eb88a0 2024-03-24T00:47:26.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.117+0000 7faa994d9640 1 -- 172.21.15.67:0/3493871595 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa94018070 con 0x558e79eb88a0 2024-03-24T00:47:26.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.117+0000 7faa994d9640 1 -- 172.21.15.67:0/3493871595 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa94013830 con 0x558e79eb88a0 2024-03-24T00:47:26.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.117+0000 7faa9e235f40 1 -- 172.21.15.67:0/3493871595 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x558e79eb7ff0 con 0x558e79eb88a0 2024-03-24T00:47:26.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.117+0000 7faa9e235f40 1 -- 172.21.15.67:0/3493871595 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558e79e17df0 con 0x558e79eb88a0 2024-03-24T00:47:26.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.118+0000 7faa994d9640 1 -- 172.21.15.67:0/3493871595 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7faa940139d0 con 0x558e79eb88a0 2024-03-24T00:47:26.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.119+0000 7faa994d9640 1 --2- 172.21.15.67:0/3493871595 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7faa7c06a5b0 0x7faa7c06ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:26.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.119+0000 7faa994d9640 1 -- 172.21.15.67:0/3493871595 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6101+0+0 (secure 0 0 0) 0x7faa94089840 con 0x558e79eb88a0 2024-03-24T00:47:26.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.119+0000 7faa9b4dd640 1 --2- 172.21.15.67:0/3493871595 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7faa7c06a5b0 0x7faa7c06ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:26.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.119+0000 7faa9e235f40 1 --2- 172.21.15.67:0/3493871595 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x558e79ea0c60 0x558e79f5cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:26.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.119+0000 7faa9e235f40 1 -- 172.21.15.67:0/3493871595 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:1 15.13 15:c91d4a1d:::mds0_inotable:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e105) v9 -- 0x558e79e988e0 con 0x558e79ea0c60 2024-03-24T00:47:26.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.119+0000 7faa93fff640 1 --2- 172.21.15.67:0/3493871595 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x558e79ea0c60 0x558e79f5cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:26.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.119+0000 7faa9b4dd640 1 --2- 172.21.15.67:0/3493871595 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7faa7c06a5b0 0x7faa7c06ca70 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7faa8c000f60 tx=0x7faa8c0016f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:26.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.119+0000 7faa93fff640 1 --2- 172.21.15.67:0/3493871595 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x558e79ea0c60 0x558e79f5cf90 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:26.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.120+0000 7faa93fff640 1 -- 172.21.15.67:0/3493871595 <== osd.1 v2:172.21.15.67:6800/4062510404 1 ==== osd_op_reply(1 mds0_inotable [read 0~34 out=34b] v0'0 uv3 ondisk = 0) v8 ==== 157+0+34 (crc 0 0 0) 0x7faa8405a070 con 0x558e79ea0c60 2024-03-24T00:47:26.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.120+0000 7faa9e235f40 1 -- 172.21.15.67:0/3493871595 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x558e79ea0c60 msgr2=0x558e79f5cf90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:26.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.120+0000 7faa9e235f40 1 --2- 172.21.15.67:0/3493871595 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x558e79ea0c60 0x558e79f5cf90 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:26.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.120+0000 7faa9e235f40 1 -- 172.21.15.67:0/3493871595 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7faa7c06a5b0 msgr2=0x7faa7c06ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:26.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.120+0000 7faa9e235f40 1 --2- 172.21.15.67:0/3493871595 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7faa7c06a5b0 0x7faa7c06ca70 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7faa8c000f60 tx=0x7faa8c0016f0 comp rx=0 tx=0).stop 2024-03-24T00:47:26.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.120+0000 7faa9e235f40 1 -- 172.21.15.67:0/3493871595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558e79eb88a0 msgr2=0x558e79e13f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:26.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.120+0000 7faa9e235f40 1 --2- 172.21.15.67:0/3493871595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558e79eb88a0 0x558e79e13f40 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7faa94005f80 tx=0x7faa9400a570 comp rx=0 tx=0).stop 2024-03-24T00:47:26.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.120+0000 7faa9bcde640 1 -- 172.21.15.67:0/3493871595 reap_dead start 2024-03-24T00:47:26.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.120+0000 7faa9e235f40 1 -- 172.21.15.67:0/3493871595 shutdown_connections 2024-03-24T00:47:26.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.120+0000 7faa9e235f40 1 -- 172.21.15.67:0/3493871595 >> 172.21.15.67:0/3493871595 conn(0x558e79e15320 msgr2=0x558e79e9fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:26.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.120+0000 7faa9e235f40 1 -- 172.21.15.67:0/3493871595 shutdown_connections 2024-03-24T00:47:26.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.120+0000 7faa9e235f40 1 -- 172.21.15.67:0/3493871595 wait complete. 2024-03-24T00:47:26.125 INFO:teuthology.orchestra.run.smithi067.stdout:����� 2024-03-24T00:47:26.125 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T00:47:26.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.266+0000 7f0e10280640 1 Processor -- start 2024-03-24T00:47:26.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.266+0000 7f0e10280640 1 -- start start 2024-03-24T00:47:26.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.267+0000 7f0e10280640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f0e0815bf70 0x7f0e080cc690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:26.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.267+0000 7f0e10280640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0e080ce930 0x7f0e080b1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:26.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.267+0000 7f0e10280640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e080ccfe0 con 0x7f0e080cfc70 2024-03-24T00:47:26.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.267+0000 7f0e10280640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e080cd2c0 con 0x7f0e080ce930 2024-03-24T00:47:26.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.267+0000 7f0e10280640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e080cd5a0 con 0x7f0e0815bf70 2024-03-24T00:47:26.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.267+0000 7f0e0e81d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0e080ce930 0x7f0e080b1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:26.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.267+0000 7f0e0d81b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0e080cfc70 0x7f0e080ce220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45304/0 (socket says 172.21.15.67:45304) 2024-03-24T00:47:26.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.267+0000 7f0e0d81b640 1 -- 172.21.15.67:0/1928807860 learned_addr learned my addr 172.21.15.67:0/1928807860 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:47:26.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.268+0000 7f0e0e01c640 1 --2- 172.21.15.67:0/1928807860 >> v2:172.21.15.72:3300/0 conn(0x7f0e0815bf70 0x7f0e080cc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:26.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.268+0000 7f0e0e81d640 1 -- 172.21.15.67:0/1928807860 >> v2:172.21.15.72:3300/0 conn(0x7f0e0815bf70 msgr2=0x7f0e080cc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:26.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.268+0000 7f0e0e81d640 1 --2- 172.21.15.67:0/1928807860 >> v2:172.21.15.72:3300/0 conn(0x7f0e0815bf70 0x7f0e080cc690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:26.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.268+0000 7f0e0e81d640 1 -- 172.21.15.67:0/1928807860 >> v1:172.21.15.67:6789/0 conn(0x7f0e080cfc70 legacy=0x7f0e080ce220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:26.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.268+0000 7f0e0e81d640 1 -- 172.21.15.67:0/1928807860 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e080cd880 con 0x7f0e080ce930 2024-03-24T00:47:26.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.269+0000 7f0e0e81d640 1 --2- 172.21.15.67:0/1928807860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e080ce930 0x7f0e080b1cd0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f0e0808c2a0 tx=0x7f0e0002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4b2a917082e5bc7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:26.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.269+0000 7f0e0d01a640 1 -- 172.21.15.67:0/1928807860 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e00039070 con 0x7f0e080ce930 2024-03-24T00:47:26.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.269+0000 7f0e0d01a640 1 -- 172.21.15.67:0/1928807860 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e0002cc80 con 0x7f0e080ce930 2024-03-24T00:47:26.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.269+0000 7f0e0d01a640 1 -- 172.21.15.67:0/1928807860 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e00034660 con 0x7f0e080ce930 2024-03-24T00:47:26.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.269+0000 7f0e10280640 1 -- 172.21.15.67:0/1928807860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e080ce930 msgr2=0x7f0e080b1cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:26.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.269+0000 7f0e10280640 1 --2- 172.21.15.67:0/1928807860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e080ce930 0x7f0e080b1cd0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f0e0808c2a0 tx=0x7f0e0002bed0 comp rx=0 tx=0).stop 2024-03-24T00:47:26.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.269+0000 7f0e10280640 1 -- 172.21.15.67:0/1928807860 shutdown_connections 2024-03-24T00:47:26.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.269+0000 7f0e10280640 1 --2- 172.21.15.67:0/1928807860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e080ce930 0x7f0e080b1cd0 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:26.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.269+0000 7f0e10280640 1 --2- 172.21.15.67:0/1928807860 >> v2:172.21.15.72:3300/0 conn(0x7f0e0815bf70 0x7f0e080cc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:26.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.269+0000 7f0e10280640 1 -- 172.21.15.67:0/1928807860 >> 172.21.15.67:0/1928807860 conn(0x7f0e080d1bf0 msgr2=0x7f0e080d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:26.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.269+0000 7f0e10280640 1 -- 172.21.15.67:0/1928807860 shutdown_connections 2024-03-24T00:47:26.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.270+0000 7f0e10280640 1 -- 172.21.15.67:0/1928807860 wait complete. 2024-03-24T00:47:26.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.271+0000 7f0e10280640 1 Processor -- start 2024-03-24T00:47:26.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.271+0000 7f0e10280640 1 -- start start 2024-03-24T00:47:26.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.272+0000 7f0e10280640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e080ce930 0x7f0e08158260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:26.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.272+0000 7f0e10280640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0e080cfc70 0x7f0e080b47e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:26.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.272+0000 7f0e10280640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0e0815bf70 0x7f0e080cacd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:26.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.272+0000 7f0e10280640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e080ccd50 con 0x7f0e080ce930 2024-03-24T00:47:26.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.272+0000 7f0e10280640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e080ccbd0 con 0x7f0e0815bf70 2024-03-24T00:47:26.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.272+0000 7f0e10280640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e080b2210 con 0x7f0e080cfc70 2024-03-24T00:47:26.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.273+0000 7f0e0e01c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e080ce930 0x7f0e08158260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:26.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.273+0000 7f0e0e01c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e080ce930 0x7f0e08158260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60398/0 (socket says 172.21.15.67:60398) 2024-03-24T00:47:26.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.273+0000 7f0e0e01c640 1 -- 172.21.15.67:0/4190941733 learned_addr learned my addr 172.21.15.67:0/4190941733 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:26.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.273+0000 7f0e0e81d640 1 --2- 172.21.15.67:0/4190941733 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0e0815bf70 0x7f0e080cacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:26.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.273+0000 7f0e0d81b640 1 --2- 172.21.15.67:0/4190941733 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0e080cfc70 0x7f0e080b47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:26.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.273+0000 7f0e0e01c640 1 -- 172.21.15.67:0/4190941733 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0e080cfc70 msgr2=0x7f0e080b47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:26.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.273+0000 7f0e0e01c640 1 --2- 172.21.15.67:0/4190941733 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0e080cfc70 0x7f0e080b47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:26.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.273+0000 7f0e0e01c640 1 -- 172.21.15.67:0/4190941733 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0e0815bf70 msgr2=0x7f0e080cacd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:26.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.273+0000 7f0e0e01c640 1 --2- 172.21.15.67:0/4190941733 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0e0815bf70 0x7f0e080cacd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:26.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.273+0000 7f0e0e01c640 1 -- 172.21.15.67:0/4190941733 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e080b51c0 con 0x7f0e080ce930 2024-03-24T00:47:26.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.273+0000 7f0e0d81b640 1 --2- 172.21.15.67:0/4190941733 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0e080cfc70 0x7f0e080b47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:47:26.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.273+0000 7f0e0e01c640 1 --2- 172.21.15.67:0/4190941733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e080ce930 0x7f0e08158260 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f0e0400d5f0 tx=0x7f0e0400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:26.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.273+0000 7f0dfeffd640 1 -- 172.21.15.67:0/4190941733 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e040125f0 con 0x7f0e080ce930 2024-03-24T00:47:26.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.273+0000 7f0dfeffd640 1 -- 172.21.15.67:0/4190941733 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e04019070 con 0x7f0e080ce930 2024-03-24T00:47:26.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.273+0000 7f0e10280640 1 -- 172.21.15.67:0/4190941733 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e080b54a0 con 0x7f0e080ce930 2024-03-24T00:47:26.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.273+0000 7f0dfeffd640 1 -- 172.21.15.67:0/4190941733 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e040129d0 con 0x7f0e080ce930 2024-03-24T00:47:26.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.273+0000 7f0e10280640 1 -- 172.21.15.67:0/4190941733 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e080b5a60 con 0x7f0e080ce930 2024-03-24T00:47:26.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.274+0000 7f0e10280640 1 -- 172.21.15.67:0/4190941733 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e080b6450 con 0x7f0e080ce930 2024-03-24T00:47:26.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.275+0000 7f0dfeffd640 1 -- 172.21.15.67:0/4190941733 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0e04012b70 con 0x7f0e080ce930 2024-03-24T00:47:26.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.275+0000 7f0dfeffd640 1 --2- 172.21.15.67:0/4190941733 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0dec06bbd0 0x7f0dec06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:26.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.275+0000 7f0dfeffd640 1 -- 172.21.15.67:0/4190941733 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0e0408afe0 con 0x7f0e080ce930 2024-03-24T00:47:26.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.276+0000 7f0e0d81b640 1 --2- 172.21.15.67:0/4190941733 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0dec06bbd0 0x7f0dec06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:26.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.276+0000 7f0e0d81b640 1 --2- 172.21.15.67:0/4190941733 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0dec06bbd0 0x7f0dec06e090 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f0e080cb930 tx=0x7f0df8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:26.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.284+0000 7f0dfeffd640 1 -- 172.21.15.67:0/4190941733 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0e04059940 con 0x7f0e080ce930 2024-03-24T00:47:26.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.503+0000 7f0e10280640 1 -- 172.21.15.67:0/4190941733 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f0e0804f400 con 0x7f0e080ce930 2024-03-24T00:47:26.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.503+0000 7f0dfeffd640 1 -- 172.21.15.67:0/4190941733 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v72) v1 ==== 53+0+73 (secure 0 0 0) 0x7f0e04059240 con 0x7f0e080ce930 2024-03-24T00:47:26.504 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-24T00:47:26.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.506+0000 7f0e10280640 1 -- 172.21.15.67:0/4190941733 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0dec06bbd0 msgr2=0x7f0dec06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:26.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.506+0000 7f0e10280640 1 --2- 172.21.15.67:0/4190941733 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0dec06bbd0 0x7f0dec06e090 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f0e080cb930 tx=0x7f0df8009040 comp rx=0 tx=0).stop 2024-03-24T00:47:26.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.506+0000 7f0e10280640 1 -- 172.21.15.67:0/4190941733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e080ce930 msgr2=0x7f0e08158260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:26.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.506+0000 7f0e10280640 1 --2- 172.21.15.67:0/4190941733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e080ce930 0x7f0e08158260 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f0e0400d5f0 tx=0x7f0e0400dab0 comp rx=0 tx=0).stop 2024-03-24T00:47:26.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.506+0000 7f0e10280640 1 -- 172.21.15.67:0/4190941733 shutdown_connections 2024-03-24T00:47:26.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.506+0000 7f0e10280640 1 --2- 172.21.15.67:0/4190941733 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0dec06bbd0 0x7f0dec06e090 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:26.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.506+0000 7f0e10280640 1 --2- 172.21.15.67:0/4190941733 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0e0815bf70 0x7f0e080cacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:26.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.506+0000 7f0e10280640 1 --2- 172.21.15.67:0/4190941733 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0e080cfc70 0x7f0e080b47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:26.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.506+0000 7f0e10280640 1 --2- 172.21.15.67:0/4190941733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e080ce930 0x7f0e08158260 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:26.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.506+0000 7f0e10280640 1 -- 172.21.15.67:0/4190941733 >> 172.21.15.67:0/4190941733 conn(0x7f0e080d1bf0 msgr2=0x7f0e080cde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:26.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.506+0000 7f0e10280640 1 -- 172.21.15.67:0/4190941733 shutdown_connections 2024-03-24T00:47:26.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:26.506+0000 7f0e10280640 1 -- 172.21.15.67:0/4190941733 wait complete. 2024-03-24T00:47:26.519 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-24T00:47:26.519 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T00:47:26.519 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-03-24T00:47:26.520 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-24T00:47:26.520 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T00:47:26.520 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-24T00:47:26.520 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-24T00:47:26.520 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T00:47:26.520 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:47:26.520 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:47:26.520 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-03-24T00:47:26.535 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T00:47:26.535 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T00:47:26.535 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T00:47:26.535 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:47:26.535 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-03-24T00:47:26.536 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:47:26.536 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:47:26.536 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-03-24T00:47:26.536 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-03-24T00:47:26.536 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:47:26.536 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-03-24T00:47:26.536 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-03-24T00:47:26.536 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:47:26.536 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-03-24T00:47:26.536 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T00:47:26.536 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-03-24T00:47:26.536 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-24T00:47:26.536 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 42067sec preferred_lft 42067sec 2024-03-24T00:47:26.536 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-03-24T00:47:26.537 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:47:26.537 INFO:teuthology.orchestra.run.smithi067.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T00:47:26.537 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 96:cf:26:c8:bf:62 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:47:26.537 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-24T00:47:26.537 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:47:26.537 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::7c6a:5fff:feaa:d41/64 scope link 2024-03-24T00:47:26.537 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:47:26.537 INFO:teuthology.orchestra.run.smithi067.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T00:47:26.537 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ea:22:4b:d5:48:3d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:47:26.537 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::e822:4bff:fed5:483d/64 scope link 2024-03-24T00:47:26.537 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:47:26.537 INFO:teuthology.orchestra.run.smithi067.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T00:47:26.537 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 96:cf:26:c8:bf:62 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:47:26.537 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::94cf:26ff:fec8:bf62/64 scope link 2024-03-24T00:47:26.538 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:47:26.538 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:47:26.538 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T00:47:26.600 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-03-24T00:47:26.602 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:47:26.602 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-03-24T00:47:26.658 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-03-24T00:47:26.658 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T00:47:26.658 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-24T00:47:26.658 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-24T00:47:26.658 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T00:47:26.658 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:47:26.658 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-03-24T00:47:26.658 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-03-24T00:47:26.659 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-24T00:47:26.659 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-24T00:47:26.659 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-24T00:47:26.659 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-03-24T00:47:26.753 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:47:26.753 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-03-24T00:47:26.768 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-24T00:47:26.768 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-24T00:47:26.768 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:47:26.768 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:47:26.769 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-03-24T00:47:26.835 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:47:26.835 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-24T00:47:26.890 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:47:26.890 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:47:26.890 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:47:26.944 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-24T00:47:26.945 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:47:26.945 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:47:27.030 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:47:27.030 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T00:47:27.030 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-03-24T00:47:27.036 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:47:27.036 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T00:47:27.036 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-03-24T00:47:27.036 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.72:3300,172.21.15.72:3301" 2024-03-24T00:47:27.037 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-24T00:47:27.044 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:47:27.044 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T00:47:27.044 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-03-24T00:47:27.044 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.72:3300,172.21.15.72:3301" 2024-03-24T00:47:27.045 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-24T00:47:27.045 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:6789,172.21.15.72:6789,172.21.15.72:6790" 2024-03-24T00:47:27.045 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.67:6789,172.21.15.72:6789,172.21.15.72:6790:/ 2024-03-24T00:47:27.045 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=de1d0178-808e-49ca-a986-e297e523b976" will pass to kernel 2024-03-24T00:47:27.045 INFO:teuthology.orchestra.run.smithi067.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-24T00:47:27.045 INFO:teuthology.orchestra.run.smithi067.stdout: You just mounted a file system that supports labels which does not 2024-03-24T00:47:27.045 INFO:teuthology.orchestra.run.smithi067.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-24T00:47:27.045 INFO:teuthology.orchestra.run.smithi067.stdout: applications will generate AVC messages and not be allowed access to 2024-03-24T00:47:27.046 INFO:teuthology.orchestra.run.smithi067.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-24T00:47:27.047 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-24T00:47:27.047 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:47:27.047 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:47:27.113 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:47:27.113 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T00:47:27.169 INFO:teuthology.orchestra.run.smithi067.stdout:client8781 2024-03-24T00:47:27.169 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:47:27.169 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:47:27.224 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:47:27.224 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:47:27.279 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:47:27.279 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=file2_sixmegs bs=1M conv=fdatasync count=6 seek=0) 2024-03-24T00:47:27.369 INFO:teuthology.orchestra.run.smithi067.stderr:6+0 records in 2024-03-24T00:47:27.369 INFO:teuthology.orchestra.run.smithi067.stderr:6+0 records out 2024-03-24T00:47:27.369 INFO:teuthology.orchestra.run.smithi067.stderr:6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.0334519 s, 188 MB/s 2024-03-24T00:47:27.369 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:47:27.369 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:47:27.384 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:47:27.384 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=file3_sixmegs bs=1M conv=fdatasync count=6 seek=0) 2024-03-24T00:47:27.471 INFO:teuthology.orchestra.run.smithi067.stderr:6+0 records in 2024-03-24T00:47:27.472 INFO:teuthology.orchestra.run.smithi067.stderr:6+0 records out 2024-03-24T00:47:27.472 INFO:teuthology.orchestra.run.smithi067.stderr:6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.0316415 s, 199 MB/s 2024-03-24T00:47:27.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:47:27.472 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec find ./) 2024-03-24T00:47:27.487 INFO:teuthology.orchestra.run.smithi067.stdout:./ 2024-03-24T00:47:27.487 INFO:teuthology.orchestra.run.smithi067.stdout:./file2_sixmegs 2024-03-24T00:47:27.488 INFO:teuthology.orchestra.run.smithi067.stdout:./file3_sixmegs 2024-03-24T00:47:27.488 INFO:teuthology.orchestra.run.smithi067.stdout:./file1_sixmegs 2024-03-24T00:47:27.488 DEBUG:teuthology.orchestra.run.smithi067:> stdin-killer -- python3 -c ' 2024-03-24T00:47:27.488 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-03-24T00:47:27.488 DEBUG:teuthology.orchestra.run.smithi067:> import stat 2024-03-24T00:47:27.488 DEBUG:teuthology.orchestra.run.smithi067:> 2024-03-24T00:47:27.488 DEBUG:teuthology.orchestra.run.smithi067:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./").st_ino) 2024-03-24T00:47:27.488 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-03-24T00:47:27.592 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-03-24T00:47:27.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:47:27.601 DEBUG:teuthology.orchestra.run.smithi067:> stdin-killer -- python3 -c ' 2024-03-24T00:47:27.601 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-03-24T00:47:27.601 DEBUG:teuthology.orchestra.run.smithi067:> import stat 2024-03-24T00:47:27.601 DEBUG:teuthology.orchestra.run.smithi067:> 2024-03-24T00:47:27.601 DEBUG:teuthology.orchestra.run.smithi067:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./file2_sixmegs").st_ino) 2024-03-24T00:47:27.601 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-03-24T00:47:27.706 INFO:teuthology.orchestra.run.smithi067.stdout:1099511627777 2024-03-24T00:47:27.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:47:27.715 DEBUG:teuthology.orchestra.run.smithi067:> stdin-killer -- python3 -c ' 2024-03-24T00:47:27.715 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-03-24T00:47:27.715 DEBUG:teuthology.orchestra.run.smithi067:> import stat 2024-03-24T00:47:27.715 DEBUG:teuthology.orchestra.run.smithi067:> 2024-03-24T00:47:27.715 DEBUG:teuthology.orchestra.run.smithi067:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./file3_sixmegs").st_ino) 2024-03-24T00:47:27.715 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-03-24T00:47:27.820 INFO:teuthology.orchestra.run.smithi067.stdout:1099511627778 2024-03-24T00:47:27.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:47:27.829 DEBUG:teuthology.orchestra.run.smithi067:> stdin-killer -- python3 -c ' 2024-03-24T00:47:27.829 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-03-24T00:47:27.829 DEBUG:teuthology.orchestra.run.smithi067:> import stat 2024-03-24T00:47:27.829 DEBUG:teuthology.orchestra.run.smithi067:> 2024-03-24T00:47:27.829 DEBUG:teuthology.orchestra.run.smithi067:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./file1_sixmegs").st_ino) 2024-03-24T00:47:27.829 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-03-24T00:47:27.934 INFO:teuthology.orchestra.run.smithi067.stdout:1099511627776 2024-03-24T00:47:27.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:47:27.943 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:47:27.943 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:47:27.999 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:47:27.999 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:47:28.053 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:47:28.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.195+0000 7f48964c0640 1 Processor -- start 2024-03-24T00:47:28.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.195+0000 7f48964c0640 1 -- start start 2024-03-24T00:47:28.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.196+0000 7f48964c0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f489015e2a0 0x7f48900ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:28.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.196+0000 7f48964c0640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f48900d0ef0 0x7f4890162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:28.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.196+0000 7f48964c0640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48900cf640 con 0x7f489015dec0 2024-03-24T00:47:28.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.196+0000 7f48964c0640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48900cee30 con 0x7f489015e2a0 2024-03-24T00:47:28.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.196+0000 7f48964c0640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48900d2120 con 0x7f48900d0ef0 2024-03-24T00:47:28.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.196+0000 7f488ffff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f489015dec0 0x7f48900d07e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45316/0 (socket says 172.21.15.67:45316) 2024-03-24T00:47:28.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.196+0000 7f488ffff640 1 -- 172.21.15.67:0/194785145 learned_addr learned my addr 172.21.15.67:0/194785145 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:47:28.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.196+0000 7f488f7fe640 1 --2- 172.21.15.67:0/194785145 >> v2:172.21.15.67:3300/0 conn(0x7f489015e2a0 0x7f48900ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:28.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.196+0000 7f4894a5d640 1 --2- 172.21.15.67:0/194785145 >> v2:172.21.15.72:3300/0 conn(0x7f48900d0ef0 0x7f4890162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:28.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.196+0000 7f488f7fe640 1 -- 172.21.15.67:0/194785145 >> v2:172.21.15.72:3300/0 conn(0x7f48900d0ef0 msgr2=0x7f4890162630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:28.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.196+0000 7f488f7fe640 1 --2- 172.21.15.67:0/194785145 >> v2:172.21.15.72:3300/0 conn(0x7f48900d0ef0 0x7f4890162630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:28.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.196+0000 7f488f7fe640 1 -- 172.21.15.67:0/194785145 >> v1:172.21.15.67:6789/0 conn(0x7f489015dec0 legacy=0x7f48900d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:28.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.196+0000 7f488f7fe640 1 -- 172.21.15.67:0/194785145 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48900cf080 con 0x7f489015e2a0 2024-03-24T00:47:28.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.196+0000 7f488effd640 1 -- 172.21.15.67:0/194785145 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1146001474 0 0) 0x7f4884001110 con 0x7f489015dec0 2024-03-24T00:47:28.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.197+0000 7f488f7fe640 1 --2- 172.21.15.67:0/194785145 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f489015e2a0 0x7f48900ce630 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f48900cae20 tx=0x7f488002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df038d54485c8d2c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:28.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.197+0000 7f488effd640 1 -- 172.21.15.67:0/194785145 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48800315a0 con 0x7f489015e2a0 2024-03-24T00:47:28.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.197+0000 7f488effd640 1 -- 172.21.15.67:0/194785145 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4880037070 con 0x7f489015e2a0 2024-03-24T00:47:28.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.197+0000 7f488effd640 1 -- 172.21.15.67:0/194785145 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4880031960 con 0x7f489015e2a0 2024-03-24T00:47:28.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.198+0000 7f48964c0640 1 -- 172.21.15.67:0/194785145 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f489015e2a0 msgr2=0x7f48900ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:28.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.198+0000 7f48964c0640 1 --2- 172.21.15.67:0/194785145 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f489015e2a0 0x7f48900ce630 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f48900cae20 tx=0x7f488002b840 comp rx=0 tx=0).stop 2024-03-24T00:47:28.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.198+0000 7f48964c0640 1 -- 172.21.15.67:0/194785145 shutdown_connections 2024-03-24T00:47:28.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.198+0000 7f48964c0640 1 --2- 172.21.15.67:0/194785145 >> v2:172.21.15.72:3300/0 conn(0x7f48900d0ef0 0x7f4890162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:28.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.198+0000 7f48964c0640 1 --2- 172.21.15.67:0/194785145 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f489015e2a0 0x7f48900ce630 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:28.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.198+0000 7f48964c0640 1 -- 172.21.15.67:0/194785145 >> 172.21.15.67:0/194785145 conn(0x7f489006c810 msgr2=0x7f489006cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:28.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.198+0000 7f48964c0640 1 -- 172.21.15.67:0/194785145 shutdown_connections 2024-03-24T00:47:28.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.198+0000 7f48964c0640 1 -- 172.21.15.67:0/194785145 wait complete. 2024-03-24T00:47:28.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.199+0000 7f48964c0640 1 Processor -- start 2024-03-24T00:47:28.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.200+0000 7f48964c0640 1 -- start start 2024-03-24T00:47:28.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.201+0000 7f48964c0640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f48900d0ef0 0x7f48901f6d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:28.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.201+0000 7f48964c0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f489015dec0 0x7f48900ad0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:28.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.201+0000 7f48964c0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f489015e2a0 0x7f48900b77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:28.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.201+0000 7f48964c0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48900d2730 con 0x7f489015e2a0 2024-03-24T00:47:28.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.201+0000 7f48964c0640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48900d25b0 con 0x7f489015dec0 2024-03-24T00:47:28.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.201+0000 7f48964c0640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48900d28b0 con 0x7f48900d0ef0 2024-03-24T00:47:28.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.201+0000 7f4894a5d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f489015e2a0 0x7f48900b77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:28.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.201+0000 7f4894a5d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f489015e2a0 0x7f48900b77c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60406/0 (socket says 172.21.15.67:60406) 2024-03-24T00:47:28.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.201+0000 7f4894a5d640 1 -- 172.21.15.67:0/3726572668 learned_addr learned my addr 172.21.15.67:0/3726572668 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:28.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.201+0000 7f488ffff640 1 --2- 172.21.15.67:0/3726572668 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f48900d0ef0 0x7f48901f6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:28.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.201+0000 7f488f7fe640 1 --2- 172.21.15.67:0/3726572668 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f489015dec0 0x7f48900ad0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:28.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.201+0000 7f4894a5d640 1 -- 172.21.15.67:0/3726572668 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f48900d0ef0 msgr2=0x7f48901f6d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:28.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.201+0000 7f4894a5d640 1 --2- 172.21.15.67:0/3726572668 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f48900d0ef0 0x7f48901f6d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:28.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.201+0000 7f4894a5d640 1 -- 172.21.15.67:0/3726572668 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f489015dec0 msgr2=0x7f48900ad0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:28.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.201+0000 7f4894a5d640 1 --2- 172.21.15.67:0/3726572668 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f489015dec0 0x7f48900ad0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:28.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.201+0000 7f4894a5d640 1 -- 172.21.15.67:0/3726572668 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48900abb80 con 0x7f489015e2a0 2024-03-24T00:47:28.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.201+0000 7f488f7fe640 1 --2- 172.21.15.67:0/3726572668 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f489015dec0 0x7f48900ad0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:47:28.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.201+0000 7f4894a5d640 1 --2- 172.21.15.67:0/3726572668 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f489015e2a0 0x7f48900b77c0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f4888007e40 tx=0x7f488800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:28.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.202+0000 7f488cff9640 1 -- 172.21.15.67:0/3726572668 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4888014620 con 0x7f489015e2a0 2024-03-24T00:47:28.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.202+0000 7f488cff9640 1 -- 172.21.15.67:0/3726572668 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4888019070 con 0x7f489015e2a0 2024-03-24T00:47:28.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.202+0000 7f48964c0640 1 -- 172.21.15.67:0/3726572668 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48900aaae0 con 0x7f489015e2a0 2024-03-24T00:47:28.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.202+0000 7f488cff9640 1 -- 172.21.15.67:0/3726572668 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4888014a20 con 0x7f489015e2a0 2024-03-24T00:47:28.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.202+0000 7f48964c0640 1 -- 172.21.15.67:0/3726572668 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48900ab0a0 con 0x7f489015e2a0 2024-03-24T00:47:28.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.203+0000 7f48964c0640 1 -- 172.21.15.67:0/3726572668 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f489004f400 con 0x7f489015e2a0 2024-03-24T00:47:28.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.203+0000 7f488cff9640 1 -- 172.21.15.67:0/3726572668 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4888014bc0 con 0x7f489015e2a0 2024-03-24T00:47:28.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.203+0000 7f488cff9640 1 --2- 172.21.15.67:0/3726572668 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f487c06bbd0 0x7f487c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:28.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.204+0000 7f488cff9640 1 -- 172.21.15.67:0/3726572668 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f488808b540 con 0x7f489015e2a0 2024-03-24T00:47:28.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.204+0000 7f488ffff640 1 --2- 172.21.15.67:0/3726572668 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f487c06bbd0 0x7f487c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:28.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.204+0000 7f488ffff640 1 --2- 172.21.15.67:0/3726572668 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f487c06bbd0 0x7f487c06e090 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f4884000f60 tx=0x7f4884007970 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:28.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.212+0000 7f488cff9640 1 -- 172.21.15.67:0/3726572668 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4888059ea0 con 0x7f489015e2a0 2024-03-24T00:47:28.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.394+0000 7f48964c0640 1 -- 172.21.15.67:0/3726572668 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f48900ce6f0 con 0x7f489015e2a0 2024-03-24T00:47:28.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.396+0000 7f488cff9640 1 -- 172.21.15.67:0/3726572668 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v105) v1 ==== 74+0+16824 (secure 0 0 0) 0x7f48880597a0 con 0x7f489015e2a0 2024-03-24T00:47:28.397 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:47:28.397 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":105,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:47:18.128465+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:47:15.111478+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-03-24T00:47:15.455832+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:47:28.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.398+0000 7f48964c0640 1 -- 172.21.15.67:0/3726572668 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f487c06bbd0 msgr2=0x7f487c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:28.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.398+0000 7f48964c0640 1 --2- 172.21.15.67:0/3726572668 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f487c06bbd0 0x7f487c06e090 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f4884000f60 tx=0x7f4884007970 comp rx=0 tx=0).stop 2024-03-24T00:47:28.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.398+0000 7f48964c0640 1 -- 172.21.15.67:0/3726572668 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f489015e2a0 msgr2=0x7f48900b77c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:28.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.398+0000 7f48964c0640 1 --2- 172.21.15.67:0/3726572668 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f489015e2a0 0x7f48900b77c0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f4888007e40 tx=0x7f488800e4b0 comp rx=0 tx=0).stop 2024-03-24T00:47:28.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.398+0000 7f48964c0640 1 -- 172.21.15.67:0/3726572668 shutdown_connections 2024-03-24T00:47:28.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.398+0000 7f48964c0640 1 --2- 172.21.15.67:0/3726572668 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f487c06bbd0 0x7f487c06e090 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:28.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.398+0000 7f48964c0640 1 --2- 172.21.15.67:0/3726572668 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f489015e2a0 0x7f48900b77c0 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:28.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.398+0000 7f48964c0640 1 --2- 172.21.15.67:0/3726572668 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f489015dec0 0x7f48900ad0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:28.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.398+0000 7f48964c0640 1 --2- 172.21.15.67:0/3726572668 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f48900d0ef0 0x7f48901f6d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:28.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.398+0000 7f48964c0640 1 -- 172.21.15.67:0/3726572668 >> 172.21.15.67:0/3726572668 conn(0x7f489006c810 msgr2=0x7f489015ecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:28.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.398+0000 7f48964c0640 1 -- 172.21.15.67:0/3726572668 shutdown_connections 2024-03-24T00:47:28.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.398+0000 7f48964c0640 1 -- 172.21.15.67:0/3726572668 wait complete. 2024-03-24T00:47:28.409 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:38:39.364950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:47:15.111478+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '104', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:47:15.455832+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '105', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:47:28.410 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:47:28.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.520+0000 7f46d6a73640 1 Processor -- start 2024-03-24T00:47:28.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.520+0000 7f46d6a73640 1 -- start start 2024-03-24T00:47:28.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.521+0000 7f46d6a73640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f46d015dea0 0x7f46d00d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:28.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.521+0000 7f46d6a73640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46d00d2250 con 0x7f46d00b6170 2024-03-24T00:47:28.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.521+0000 7f46d6a73640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46d00d2530 con 0x7f46d015e280 2024-03-24T00:47:28.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.521+0000 7f46d6a73640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46d00d2810 con 0x7f46d015dea0 2024-03-24T00:47:28.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.521+0000 7f46d480f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f46d015dea0 0x7f46d00d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:28.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.521+0000 7f46d5010640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f46d00b6170 0x7f46d00b70f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45324/0 (socket says 172.21.15.67:45324) 2024-03-24T00:47:28.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.521+0000 7f46d5010640 1 -- 172.21.15.67:0/3287801986 learned_addr learned my addr 172.21.15.67:0/3287801986 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:47:28.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.521+0000 7f46d480f640 1 -- 172.21.15.67:0/3287801986 >> v1:172.21.15.72:6789/0 conn(0x7f46d015e280 legacy=0x7f46d00b5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:28.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.521+0000 7f46d480f640 1 -- 172.21.15.67:0/3287801986 >> v1:172.21.15.67:6789/0 conn(0x7f46d00b6170 legacy=0x7f46d00b70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:28.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.521+0000 7f46cf7fe640 1 -- 172.21.15.67:0/3287801986 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2218680831 0 0) 0x7f46d00d2250 con 0x7f46d00b6170 2024-03-24T00:47:28.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.521+0000 7f46d480f640 1 -- 172.21.15.67:0/3287801986 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d00d2af0 con 0x7f46d015dea0 2024-03-24T00:47:28.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.522+0000 7f46d480f640 1 --2- 172.21.15.67:0/3287801986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46d015dea0 0x7f46d00d33d0 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f46c40097d0 tx=0x7f46c402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19b740d6b5271f2e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:28.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.522+0000 7f46cf7fe640 1 -- 172.21.15.67:0/3287801986 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c4035020 con 0x7f46d015dea0 2024-03-24T00:47:28.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.522+0000 7f46cf7fe640 1 -- 172.21.15.67:0/3287801986 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46c4031830 con 0x7f46d015dea0 2024-03-24T00:47:28.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.522+0000 7f46cf7fe640 1 -- 172.21.15.67:0/3287801986 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c4031c00 con 0x7f46d015dea0 2024-03-24T00:47:28.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.522+0000 7f46d6a73640 1 -- 172.21.15.67:0/3287801986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46d015dea0 msgr2=0x7f46d00d33d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:28.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.522+0000 7f46d6a73640 1 --2- 172.21.15.67:0/3287801986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46d015dea0 0x7f46d00d33d0 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f46c40097d0 tx=0x7f46c402b8d0 comp rx=0 tx=0).stop 2024-03-24T00:47:28.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.522+0000 7f46d6a73640 1 -- 172.21.15.67:0/3287801986 shutdown_connections 2024-03-24T00:47:28.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.522+0000 7f46d6a73640 1 --2- 172.21.15.67:0/3287801986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46d015dea0 0x7f46d00d33d0 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:28.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.522+0000 7f46d6a73640 1 -- 172.21.15.67:0/3287801986 >> 172.21.15.67:0/3287801986 conn(0x7f46d0072da0 msgr2=0x7f46d00d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:28.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.523+0000 7f46d6a73640 1 -- 172.21.15.67:0/3287801986 shutdown_connections 2024-03-24T00:47:28.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.523+0000 7f46d6a73640 1 -- 172.21.15.67:0/3287801986 wait complete. 2024-03-24T00:47:28.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.524+0000 7f46d6a73640 1 Processor -- start 2024-03-24T00:47:28.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.524+0000 7f46d6a73640 1 -- start start 2024-03-24T00:47:28.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.524+0000 7f46d6a73640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46d00b6170 0x7f46d00b2270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:28.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.524+0000 7f46d6a73640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f46d015dea0 0x7f46d00ad9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:28.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.524+0000 7f46d6a73640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f46d015e280 0x7f46d00adf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:28.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.524+0000 7f46d6a73640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d00d3ca0 con 0x7f46d00b6170 2024-03-24T00:47:28.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.524+0000 7f46d6a73640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d00d3b20 con 0x7f46d015dea0 2024-03-24T00:47:28.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.524+0000 7f46d6a73640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46d00b7800 con 0x7f46d015e280 2024-03-24T00:47:28.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.525+0000 7f46d480f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46d00b6170 0x7f46d00b2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:28.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.525+0000 7f46d480f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46d00b6170 0x7f46d00b2270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60426/0 (socket says 172.21.15.67:60426) 2024-03-24T00:47:28.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.525+0000 7f46d480f640 1 -- 172.21.15.67:0/1933073612 learned_addr learned my addr 172.21.15.67:0/1933073612 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:28.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.525+0000 7f46cffff640 1 --2- 172.21.15.67:0/1933073612 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f46d015dea0 0x7f46d00ad9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:28.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.525+0000 7f46d5010640 1 --2- 172.21.15.67:0/1933073612 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f46d015e280 0x7f46d00adf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:28.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.525+0000 7f46d480f640 1 -- 172.21.15.67:0/1933073612 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f46d015e280 msgr2=0x7f46d00adf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:28.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.525+0000 7f46d480f640 1 --2- 172.21.15.67:0/1933073612 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f46d015e280 0x7f46d00adf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:28.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.525+0000 7f46d480f640 1 -- 172.21.15.67:0/1933073612 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f46d015dea0 msgr2=0x7f46d00ad9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:28.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.525+0000 7f46d480f640 1 --2- 172.21.15.67:0/1933073612 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f46d015dea0 0x7f46d00ad9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:28.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.525+0000 7f46d480f640 1 -- 172.21.15.67:0/1933073612 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d00b27b0 con 0x7f46d00b6170 2024-03-24T00:47:28.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.525+0000 7f46cffff640 1 --2- 172.21.15.67:0/1933073612 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f46d015dea0 0x7f46d00ad9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:47:28.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.525+0000 7f46d480f640 1 --2- 172.21.15.67:0/1933073612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46d00b6170 0x7f46d00b2270 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f46c40098f0 tx=0x7f46c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:28.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.525+0000 7f46cd7fa640 1 -- 172.21.15.67:0/1933073612 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c4041070 con 0x7f46d00b6170 2024-03-24T00:47:28.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.525+0000 7f46cd7fa640 1 -- 172.21.15.67:0/1933073612 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46c403aa30 con 0x7f46d00b6170 2024-03-24T00:47:28.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.525+0000 7f46d6a73640 1 -- 172.21.15.67:0/1933073612 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46d00cdb50 con 0x7f46d00b6170 2024-03-24T00:47:28.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.525+0000 7f46cd7fa640 1 -- 172.21.15.67:0/1933073612 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c4035040 con 0x7f46d00b6170 2024-03-24T00:47:28.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.525+0000 7f46d6a73640 1 -- 172.21.15.67:0/1933073612 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46d00cd870 con 0x7f46d00b6170 2024-03-24T00:47:28.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.527+0000 7f46d6a73640 1 -- 172.21.15.67:0/1933073612 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46d004f400 con 0x7f46d00b6170 2024-03-24T00:47:28.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.527+0000 7f46cd7fa640 1 -- 172.21.15.67:0/1933073612 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f46c404c060 con 0x7f46d00b6170 2024-03-24T00:47:28.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.527+0000 7f46cd7fa640 1 --2- 172.21.15.67:0/1933073612 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f46b406bbd0 0x7f46b406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:28.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.527+0000 7f46cd7fa640 1 -- 172.21.15.67:0/1933073612 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f46c40ac610 con 0x7f46d00b6170 2024-03-24T00:47:28.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.527+0000 7f46cffff640 1 --2- 172.21.15.67:0/1933073612 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f46b406bbd0 0x7f46b406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:28.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.528+0000 7f46cffff640 1 --2- 172.21.15.67:0/1933073612 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f46b406bbd0 0x7f46b406e090 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f46bc007fd0 tx=0x7f46bc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:28.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.532+0000 7f46cd7fa640 1 -- 172.21.15.67:0/1933073612 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f46c407b000 con 0x7f46d00b6170 2024-03-24T00:47:28.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.712+0000 7f46d6a73640 1 -- 172.21.15.67:0/1933073612 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f46d006b990 con 0x7f46d00b6170 2024-03-24T00:47:28.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.712+0000 7f46cd7fa640 1 -- 172.21.15.67:0/1933073612 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v105) v1 ==== 93+0+10 (secure 0 0 0) 0x7f46c407a900 con 0x7f46d00b6170 2024-03-24T00:47:28.713 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-24T00:47:28.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.715+0000 7f46d6a73640 1 -- 172.21.15.67:0/1933073612 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f46b406bbd0 msgr2=0x7f46b406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:28.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.715+0000 7f46d6a73640 1 --2- 172.21.15.67:0/1933073612 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f46b406bbd0 0x7f46b406e090 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f46bc007fd0 tx=0x7f46bc009040 comp rx=0 tx=0).stop 2024-03-24T00:47:28.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.715+0000 7f46d6a73640 1 -- 172.21.15.67:0/1933073612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46d00b6170 msgr2=0x7f46d00b2270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:28.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.715+0000 7f46d6a73640 1 --2- 172.21.15.67:0/1933073612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46d00b6170 0x7f46d00b2270 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f46c40098f0 tx=0x7f46c40046c0 comp rx=0 tx=0).stop 2024-03-24T00:47:28.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.715+0000 7f46d6a73640 1 -- 172.21.15.67:0/1933073612 shutdown_connections 2024-03-24T00:47:28.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.715+0000 7f46d6a73640 1 --2- 172.21.15.67:0/1933073612 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f46b406bbd0 0x7f46b406e090 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:28.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.715+0000 7f46d6a73640 1 --2- 172.21.15.67:0/1933073612 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f46d015e280 0x7f46d00adf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:28.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.715+0000 7f46d6a73640 1 --2- 172.21.15.67:0/1933073612 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f46d015dea0 0x7f46d00ad9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:28.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.715+0000 7f46d6a73640 1 --2- 172.21.15.67:0/1933073612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46d00b6170 0x7f46d00b2270 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:28.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.715+0000 7f46d6a73640 1 -- 172.21.15.67:0/1933073612 >> 172.21.15.67:0/1933073612 conn(0x7f46d0072da0 msgr2=0x7f46d00d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:28.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.715+0000 7f46d6a73640 1 -- 172.21.15.67:0/1933073612 shutdown_connections 2024-03-24T00:47:28.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.715+0000 7f46d6a73640 1 -- 172.21.15.67:0/1933073612 wait complete. 2024-03-24T00:47:28.728 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:47:28.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.832+0000 7fbf66c63640 1 Processor -- start 2024-03-24T00:47:28.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.832+0000 7fbf66c63640 1 -- start start 2024-03-24T00:47:28.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.833+0000 7fbf66c63640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbf6006b750 0x7fbf6015e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:28.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.833+0000 7fbf66c63640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf600d1630 con 0x7fbf6015de90 2024-03-24T00:47:28.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.833+0000 7fbf66c63640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf600ad9f0 con 0x7fbf600af4e0 2024-03-24T00:47:28.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.833+0000 7fbf66c63640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf600d1910 con 0x7fbf6006b750 2024-03-24T00:47:28.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.833+0000 7fbf649ff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbf6015de90 0x7fbf6006b040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45336/0 (socket says 172.21.15.67:45336) 2024-03-24T00:47:28.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.833+0000 7fbf649ff640 1 -- 172.21.15.67:0/1610207460 learned_addr learned my addr 172.21.15.67:0/1610207460 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:47:28.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.833+0000 7fbf65200640 1 --2- 172.21.15.67:0/1610207460 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbf6006b750 0x7fbf6015e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:28.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.833+0000 7fbf5f7fe640 1 -- 172.21.15.67:0/1610207460 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2248947669 0 0) 0x7fbf600d1630 con 0x7fbf6015de90 2024-03-24T00:47:28.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.833+0000 7fbf5f7fe640 1 -- 172.21.15.67:0/1610207460 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf48003520 con 0x7fbf6015de90 2024-03-24T00:47:28.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.833+0000 7fbf5f7fe640 1 -- 172.21.15.67:0/1610207460 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3218678631 0 0) 0x7fbf600ad9f0 con 0x7fbf600af4e0 2024-03-24T00:47:28.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.833+0000 7fbf5f7fe640 1 -- 172.21.15.67:0/1610207460 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf600d1630 con 0x7fbf600af4e0 2024-03-24T00:47:28.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.833+0000 7fbf5f7fe640 1 -- 172.21.15.67:0/1610207460 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 24374541 0 0) 0x7fbf48003520 con 0x7fbf6015de90 2024-03-24T00:47:28.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.833+0000 7fbf65200640 1 -- 172.21.15.67:0/1610207460 >> v1:172.21.15.72:6789/0 conn(0x7fbf600af4e0 legacy=0x7fbf600cfd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:28.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.834+0000 7fbf65200640 1 -- 172.21.15.67:0/1610207460 >> v1:172.21.15.67:6789/0 conn(0x7fbf6015de90 legacy=0x7fbf6006b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:28.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.834+0000 7fbf65200640 1 -- 172.21.15.67:0/1610207460 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf600adcd0 con 0x7fbf6006b750 2024-03-24T00:47:28.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.834+0000 7fbf65200640 1 --2- 172.21.15.67:0/1610207460 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbf6006b750 0x7fbf6015e270 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fbf50009dc0 tx=0x7fbf5002bd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fafc91b1cb94070a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:28.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.834+0000 7fbf5f7fe640 1 -- 172.21.15.67:0/1610207460 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf50039070 con 0x7fbf6006b750 2024-03-24T00:47:28.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.835+0000 7fbf5f7fe640 1 -- 172.21.15.67:0/1610207460 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf5002c880 con 0x7fbf6006b750 2024-03-24T00:47:28.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.835+0000 7fbf5f7fe640 1 -- 172.21.15.67:0/1610207460 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf58001670 con 0x7fbf6006b750 2024-03-24T00:47:28.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.835+0000 7fbf66c63640 1 -- 172.21.15.67:0/1610207460 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbf6006b750 msgr2=0x7fbf6015e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:28.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.835+0000 7fbf66c63640 1 --2- 172.21.15.67:0/1610207460 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbf6006b750 0x7fbf6015e270 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fbf50009dc0 tx=0x7fbf5002bd50 comp rx=0 tx=0).stop 2024-03-24T00:47:28.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.835+0000 7fbf66c63640 1 -- 172.21.15.67:0/1610207460 shutdown_connections 2024-03-24T00:47:28.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.835+0000 7fbf66c63640 1 --2- 172.21.15.67:0/1610207460 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbf6006b750 0x7fbf6015e270 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:28.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.835+0000 7fbf66c63640 1 -- 172.21.15.67:0/1610207460 >> 172.21.15.67:0/1610207460 conn(0x7fbf60099a30 msgr2=0x7fbf60099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:28.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.835+0000 7fbf66c63640 1 -- 172.21.15.67:0/1610207460 shutdown_connections 2024-03-24T00:47:28.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.835+0000 7fbf66c63640 1 -- 172.21.15.67:0/1610207460 wait complete. 2024-03-24T00:47:28.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.836+0000 7fbf66c63640 1 Processor -- start 2024-03-24T00:47:28.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.836+0000 7fbf66c63640 1 -- start start 2024-03-24T00:47:28.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.837+0000 7fbf66c63640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbf6006b750 0x7fbf60150000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:28.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.837+0000 7fbf66c63640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf600af4e0 0x7fbf600c4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:28.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.837+0000 7fbf66c63640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf6015de90 0x7fbf6006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:28.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.837+0000 7fbf66c63640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf600cd460 con 0x7fbf6015de90 2024-03-24T00:47:28.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.837+0000 7fbf66c63640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf600cd2e0 con 0x7fbf600af4e0 2024-03-24T00:47:28.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.837+0000 7fbf66c63640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf600cd5e0 con 0x7fbf6006b750 2024-03-24T00:47:28.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.837+0000 7fbf65200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf6015de90 0x7fbf6006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:28.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.837+0000 7fbf65200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf6015de90 0x7fbf6006ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60442/0 (socket says 172.21.15.67:60442) 2024-03-24T00:47:28.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.837+0000 7fbf65200640 1 -- 172.21.15.67:0/2893369391 learned_addr learned my addr 172.21.15.67:0/2893369391 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:28.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.838+0000 7fbf5ffff640 1 --2- 172.21.15.67:0/2893369391 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf600af4e0 0x7fbf600c4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:28.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.838+0000 7fbf649ff640 1 --2- 172.21.15.67:0/2893369391 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbf6006b750 0x7fbf60150000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:28.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.838+0000 7fbf65200640 1 -- 172.21.15.67:0/2893369391 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbf6006b750 msgr2=0x7fbf60150000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:28.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.838+0000 7fbf65200640 1 --2- 172.21.15.67:0/2893369391 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbf6006b750 0x7fbf60150000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:28.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.838+0000 7fbf65200640 1 -- 172.21.15.67:0/2893369391 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf600af4e0 msgr2=0x7fbf600c4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:28.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.838+0000 7fbf65200640 1 --2- 172.21.15.67:0/2893369391 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf600af4e0 0x7fbf600c4610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:28.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.838+0000 7fbf65200640 1 -- 172.21.15.67:0/2893369391 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf60150540 con 0x7fbf6015de90 2024-03-24T00:47:28.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.838+0000 7fbf5ffff640 1 --2- 172.21.15.67:0/2893369391 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf600af4e0 0x7fbf600c4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:47:28.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.838+0000 7fbf65200640 1 --2- 172.21.15.67:0/2893369391 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf6015de90 0x7fbf6006ec70 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7fbf50009ee0 tx=0x7fbf50004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:28.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.838+0000 7fbf5d7fa640 1 -- 172.21.15.67:0/2893369391 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf50039050 con 0x7fbf6015de90 2024-03-24T00:47:28.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.838+0000 7fbf5d7fa640 1 -- 172.21.15.67:0/2893369391 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf50045070 con 0x7fbf6015de90 2024-03-24T00:47:28.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.838+0000 7fbf66c63640 1 -- 172.21.15.67:0/2893369391 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf600c5040 con 0x7fbf6015de90 2024-03-24T00:47:28.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.838+0000 7fbf5d7fa640 1 -- 172.21.15.67:0/2893369391 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf50009450 con 0x7fbf6015de90 2024-03-24T00:47:28.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.838+0000 7fbf66c63640 1 -- 172.21.15.67:0/2893369391 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf600b3960 con 0x7fbf6015de90 2024-03-24T00:47:28.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.839+0000 7fbf66c63640 1 -- 172.21.15.67:0/2893369391 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf600c79a0 con 0x7fbf6015de90 2024-03-24T00:47:28.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.840+0000 7fbf5d7fa640 1 -- 172.21.15.67:0/2893369391 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fbf50004690 con 0x7fbf6015de90 2024-03-24T00:47:28.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.840+0000 7fbf5d7fa640 1 --2- 172.21.15.67:0/2893369391 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbf4c06bbd0 0x7fbf4c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:28.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.840+0000 7fbf5d7fa640 1 -- 172.21.15.67:0/2893369391 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fbf500ad460 con 0x7fbf6015de90 2024-03-24T00:47:28.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.840+0000 7fbf649ff640 1 --2- 172.21.15.67:0/2893369391 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbf4c06bbd0 0x7fbf4c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:28.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.841+0000 7fbf649ff640 1 --2- 172.21.15.67:0/2893369391 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbf4c06bbd0 0x7fbf4c06e090 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fbf58000ff0 tx=0x7fbf58009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:28.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:28.845+0000 7fbf5d7fa640 1 -- 172.21.15.67:0/2893369391 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fbf60150540 con 0x7fbf6015de90 2024-03-24T00:47:29.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.023+0000 7fbf66c63640 1 -- 172.21.15.67:0/2893369391 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbf6004f400 con 0x7fbf6015de90 2024-03-24T00:47:29.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.024+0000 7fbf5d7fa640 1 -- 172.21.15.67:0/2893369391 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v105) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbf5007be50 con 0x7fbf6015de90 2024-03-24T00:47:29.025 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-24T00:47:29.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.026+0000 7fbf66c63640 1 -- 172.21.15.67:0/2893369391 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbf4c06bbd0 msgr2=0x7fbf4c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:29.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.026+0000 7fbf66c63640 1 --2- 172.21.15.67:0/2893369391 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbf4c06bbd0 0x7fbf4c06e090 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fbf58000ff0 tx=0x7fbf58009040 comp rx=0 tx=0).stop 2024-03-24T00:47:29.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.026+0000 7fbf66c63640 1 -- 172.21.15.67:0/2893369391 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf6015de90 msgr2=0x7fbf6006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:29.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.026+0000 7fbf66c63640 1 --2- 172.21.15.67:0/2893369391 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf6015de90 0x7fbf6006ec70 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7fbf50009ee0 tx=0x7fbf50004a40 comp rx=0 tx=0).stop 2024-03-24T00:47:29.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.026+0000 7fbf66c63640 1 -- 172.21.15.67:0/2893369391 shutdown_connections 2024-03-24T00:47:29.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.026+0000 7fbf66c63640 1 --2- 172.21.15.67:0/2893369391 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbf4c06bbd0 0x7fbf4c06e090 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:29.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.026+0000 7fbf66c63640 1 --2- 172.21.15.67:0/2893369391 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf6015de90 0x7fbf6006ec70 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:29.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.026+0000 7fbf66c63640 1 --2- 172.21.15.67:0/2893369391 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf600af4e0 0x7fbf600c4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:29.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.026+0000 7fbf66c63640 1 --2- 172.21.15.67:0/2893369391 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbf6006b750 0x7fbf60150000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:29.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.026+0000 7fbf66c63640 1 -- 172.21.15.67:0/2893369391 >> 172.21.15.67:0/2893369391 conn(0x7fbf60099a30 msgr2=0x7fbf6006ff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:29.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.026+0000 7fbf66c63640 1 -- 172.21.15.67:0/2893369391 shutdown_connections 2024-03-24T00:47:29.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.026+0000 7fbf66c63640 1 -- 172.21.15.67:0/2893369391 wait complete. 2024-03-24T00:47:29.038 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:47:29.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.141+0000 7fca79d1d640 1 Processor -- start 2024-03-24T00:47:29.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.141+0000 7fca79d1d640 1 -- start start 2024-03-24T00:47:29.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.142+0000 7fca79d1d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fca7415dea0 0x7fca740d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:29.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.142+0000 7fca79d1d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fca740b6170 0x7fca740d3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:29.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.142+0000 7fca79d1d640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca740d2250 con 0x7fca7415e280 2024-03-24T00:47:29.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.142+0000 7fca79d1d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca740d2530 con 0x7fca740b6170 2024-03-24T00:47:29.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.142+0000 7fca79d1d640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca740d2810 con 0x7fca7415dea0 2024-03-24T00:47:29.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.142+0000 7fca73fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fca740b6170 0x7fca740d3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:29.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.142+0000 7fca73fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fca740b6170 0x7fca740d3910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60456/0 (socket says 172.21.15.67:60456) 2024-03-24T00:47:29.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.142+0000 7fca73fff640 1 -- 172.21.15.67:0/709679503 learned_addr learned my addr 172.21.15.67:0/709679503 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:29.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.142+0000 7fca737fe640 1 --2- 172.21.15.67:0/709679503 >> v2:172.21.15.72:3300/0 conn(0x7fca7415dea0 0x7fca740d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:29.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.142+0000 7fca73fff640 1 -- 172.21.15.67:0/709679503 >> v2:172.21.15.72:3300/0 conn(0x7fca7415dea0 msgr2=0x7fca740d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:29.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.142+0000 7fca73fff640 1 --2- 172.21.15.67:0/709679503 >> v2:172.21.15.72:3300/0 conn(0x7fca7415dea0 0x7fca740d33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:29.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.142+0000 7fca73fff640 1 -- 172.21.15.67:0/709679503 >> v1:172.21.15.72:6789/0 conn(0x7fca7415e280 legacy=0x7fca740b5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:29.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.142+0000 7fca73fff640 1 -- 172.21.15.67:0/709679503 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca740d2af0 con 0x7fca740b6170 2024-03-24T00:47:29.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.143+0000 7fca73fff640 1 --2- 172.21.15.67:0/709679503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca740b6170 0x7fca740d3910 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7fca6c009e50 tx=0x7fca6c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe693401e009e5ad server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:29.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.143+0000 7fca727fc640 1 -- 172.21.15.67:0/709679503 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca6c02ca80 con 0x7fca740b6170 2024-03-24T00:47:29.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.143+0000 7fca727fc640 1 -- 172.21.15.67:0/709679503 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca6c02cc20 con 0x7fca740b6170 2024-03-24T00:47:29.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.143+0000 7fca727fc640 1 -- 172.21.15.67:0/709679503 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca6c0343f0 con 0x7fca740b6170 2024-03-24T00:47:29.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.144+0000 7fca79d1d640 1 -- 172.21.15.67:0/709679503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca740b6170 msgr2=0x7fca740d3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:29.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.144+0000 7fca79d1d640 1 --2- 172.21.15.67:0/709679503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca740b6170 0x7fca740d3910 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7fca6c009e50 tx=0x7fca6c02bee0 comp rx=0 tx=0).stop 2024-03-24T00:47:29.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.144+0000 7fca79d1d640 1 -- 172.21.15.67:0/709679503 shutdown_connections 2024-03-24T00:47:29.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.144+0000 7fca79d1d640 1 --2- 172.21.15.67:0/709679503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca740b6170 0x7fca740d3910 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:29.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.144+0000 7fca79d1d640 1 --2- 172.21.15.67:0/709679503 >> v2:172.21.15.72:3300/0 conn(0x7fca7415dea0 0x7fca740d33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:29.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.144+0000 7fca79d1d640 1 -- 172.21.15.67:0/709679503 >> 172.21.15.67:0/709679503 conn(0x7fca74072da0 msgr2=0x7fca740d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:29.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.144+0000 7fca79d1d640 1 -- 172.21.15.67:0/709679503 shutdown_connections 2024-03-24T00:47:29.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.144+0000 7fca79d1d640 1 -- 172.21.15.67:0/709679503 wait complete. 2024-03-24T00:47:29.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.146+0000 7fca79d1d640 1 Processor -- start 2024-03-24T00:47:29.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.146+0000 7fca79d1d640 1 -- start start 2024-03-24T00:47:29.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.147+0000 7fca79d1d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca740b6170 0x7fca7414f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:29.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.147+0000 7fca79d1d640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fca7415dea0 0x7fca740cb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:29.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.147+0000 7fca79d1d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca7415e280 0x7fca741e8640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:29.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.147+0000 7fca79d1d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca740b7480 con 0x7fca7415e280 2024-03-24T00:47:29.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.147+0000 7fca79d1d640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca740b7300 con 0x7fca740b6170 2024-03-24T00:47:29.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.147+0000 7fca79d1d640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca740b7600 con 0x7fca7415dea0 2024-03-24T00:47:29.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.147+0000 7fca73fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca7415e280 0x7fca741e8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:29.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.148+0000 7fca73fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca7415e280 0x7fca741e8640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60466/0 (socket says 172.21.15.67:60466) 2024-03-24T00:47:29.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.148+0000 7fca73fff640 1 -- 172.21.15.67:0/420513346 learned_addr learned my addr 172.21.15.67:0/420513346 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:29.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.148+0000 7fca72ffd640 1 --2- 172.21.15.67:0/420513346 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fca7415dea0 0x7fca740cb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:29.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.148+0000 7fca737fe640 1 --2- 172.21.15.67:0/420513346 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca740b6170 0x7fca7414f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:29.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.148+0000 7fca73fff640 1 -- 172.21.15.67:0/420513346 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fca7415dea0 msgr2=0x7fca740cb0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:29.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.148+0000 7fca73fff640 1 --2- 172.21.15.67:0/420513346 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fca7415dea0 0x7fca740cb0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:29.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.148+0000 7fca73fff640 1 -- 172.21.15.67:0/420513346 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca740b6170 msgr2=0x7fca7414f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:29.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.148+0000 7fca73fff640 1 --2- 172.21.15.67:0/420513346 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca740b6170 0x7fca7414f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:29.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.148+0000 7fca73fff640 1 -- 172.21.15.67:0/420513346 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca740cba20 con 0x7fca7415e280 2024-03-24T00:47:29.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.148+0000 7fca73fff640 1 --2- 172.21.15.67:0/420513346 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca7415e280 0x7fca741e8640 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7fca6c009e50 tx=0x7fca6c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:29.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.148+0000 7fca5bfff640 1 -- 172.21.15.67:0/420513346 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca6c043020 con 0x7fca7415e280 2024-03-24T00:47:29.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.148+0000 7fca5bfff640 1 -- 172.21.15.67:0/420513346 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca6c03e450 con 0x7fca7415e280 2024-03-24T00:47:29.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.148+0000 7fca79d1d640 1 -- 172.21.15.67:0/420513346 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca740cbd00 con 0x7fca7415e280 2024-03-24T00:47:29.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.148+0000 7fca5bfff640 1 -- 172.21.15.67:0/420513346 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca6c009510 con 0x7fca7415e280 2024-03-24T00:47:29.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.148+0000 7fca79d1d640 1 -- 172.21.15.67:0/420513346 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca740cc2c0 con 0x7fca7415e280 2024-03-24T00:47:29.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.149+0000 7fca79d1d640 1 -- 172.21.15.67:0/420513346 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca7404f400 con 0x7fca7415e280 2024-03-24T00:47:29.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.150+0000 7fca5bfff640 1 -- 172.21.15.67:0/420513346 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fca6c046020 con 0x7fca7415e280 2024-03-24T00:47:29.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.150+0000 7fca5bfff640 1 --2- 172.21.15.67:0/420513346 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fca6006bbd0 0x7fca6006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:29.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.150+0000 7fca5bfff640 1 -- 172.21.15.67:0/420513346 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fca6c0ad170 con 0x7fca7415e280 2024-03-24T00:47:29.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.150+0000 7fca737fe640 1 --2- 172.21.15.67:0/420513346 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fca6006bbd0 0x7fca6006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:29.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.151+0000 7fca737fe640 1 --2- 172.21.15.67:0/420513346 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fca6006bbd0 0x7fca6006e090 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fca740ccd60 tx=0x7fca64009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:29.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.155+0000 7fca5bfff640 1 -- 172.21.15.67:0/420513346 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fca6c04d030 con 0x7fca7415e280 2024-03-24T00:47:29.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.337+0000 7fca79d1d640 1 -- 172.21.15.67:0/420513346 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fca7414f8e0 con 0x7fca7415e280 2024-03-24T00:47:29.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.338+0000 7fca5bfff640 1 -- 172.21.15.67:0/420513346 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v105) v1 ==== 100+0+11 (secure 0 0 0) 0x7fca6c07b480 con 0x7fca7415e280 2024-03-24T00:47:29.339 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-03-24T00:47:29.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.340+0000 7fca79d1d640 1 -- 172.21.15.67:0/420513346 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fca6006bbd0 msgr2=0x7fca6006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:29.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.340+0000 7fca79d1d640 1 --2- 172.21.15.67:0/420513346 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fca6006bbd0 0x7fca6006e090 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fca740ccd60 tx=0x7fca64009190 comp rx=0 tx=0).stop 2024-03-24T00:47:29.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.340+0000 7fca79d1d640 1 -- 172.21.15.67:0/420513346 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca7415e280 msgr2=0x7fca741e8640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:29.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.340+0000 7fca79d1d640 1 --2- 172.21.15.67:0/420513346 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca7415e280 0x7fca741e8640 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7fca6c009e50 tx=0x7fca6c004a40 comp rx=0 tx=0).stop 2024-03-24T00:47:29.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.340+0000 7fca79d1d640 1 -- 172.21.15.67:0/420513346 shutdown_connections 2024-03-24T00:47:29.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.340+0000 7fca79d1d640 1 --2- 172.21.15.67:0/420513346 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fca6006bbd0 0x7fca6006e090 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:29.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.340+0000 7fca79d1d640 1 --2- 172.21.15.67:0/420513346 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca7415e280 0x7fca741e8640 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:29.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.340+0000 7fca79d1d640 1 --2- 172.21.15.67:0/420513346 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fca7415dea0 0x7fca740cb0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:29.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.340+0000 7fca79d1d640 1 --2- 172.21.15.67:0/420513346 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca740b6170 0x7fca7414f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:29.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.340+0000 7fca79d1d640 1 -- 172.21.15.67:0/420513346 >> 172.21.15.67:0/420513346 conn(0x7fca74072da0 msgr2=0x7fca740b8d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:29.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.340+0000 7fca79d1d640 1 -- 172.21.15.67:0/420513346 shutdown_connections 2024-03-24T00:47:29.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.340+0000 7fca79d1d640 1 -- 172.21.15.67:0/420513346 wait complete. 2024-03-24T00:47:29.352 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T00:47:29.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.454+0000 7f1fa873b640 1 Processor -- start 2024-03-24T00:47:29.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.454+0000 7f1fa873b640 1 -- start start 2024-03-24T00:47:29.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.454+0000 7f1fa873b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1fa00b1eb0 0x7f1fa0099d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:29.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.454+0000 7f1fa873b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1fa009a240 0x7f1fa00cd410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:29.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.454+0000 7f1fa873b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fa00b29b0 con 0x7f1fa006de60 2024-03-24T00:47:29.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.455+0000 7f1fa873b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fa00cc9e0 con 0x7f1fa00b1eb0 2024-03-24T00:47:29.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.455+0000 7f1fa873b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa00ccf80 con 0x7f1fa009a240 2024-03-24T00:47:29.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.455+0000 7f1fa6cd8640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1fa009a240 0x7f1fa00cd410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:29.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.455+0000 7f1fa5cd6640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1fa00b1eb0 0x7f1fa0099d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:29.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.455+0000 7f1fa5cd6640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1fa00b1eb0 0x7f1fa0099d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:53184/0 (socket says 172.21.15.67:53184) 2024-03-24T00:47:29.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.455+0000 7f1fa6cd8640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1fa009a240 0x7f1fa00cd410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.67:46444/0 (socket says 172.21.15.67:46444) 2024-03-24T00:47:29.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.455+0000 7f1fa5cd6640 1 -- 172.21.15.67:0/1675311306 learned_addr learned my addr 172.21.15.67:0/1675311306 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:29.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.455+0000 7f1fa6cd8640 1 -- 172.21.15.67:0/1675311306 >> v2:172.21.15.72:3300/0 conn(0x7f1fa00b1eb0 msgr2=0x7f1fa0099d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:29.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.455+0000 7f1fa6cd8640 1 --2- 172.21.15.67:0/1675311306 >> v2:172.21.15.72:3300/0 conn(0x7f1fa00b1eb0 0x7f1fa0099d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:29.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.455+0000 7f1fa6cd8640 1 -- 172.21.15.67:0/1675311306 >> v1:172.21.15.72:6789/0 conn(0x7f1fa006de60 legacy=0x7f1fa009ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:29.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.455+0000 7f1fa54d5640 1 -- 172.21.15.67:0/1675311306 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 700122923 0 0) 0x7f1fa00b29b0 con 0x7f1fa006de60 2024-03-24T00:47:29.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.455+0000 7f1fa6cd8640 1 -- 172.21.15.67:0/1675311306 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fa00b23f0 con 0x7f1fa009a240 2024-03-24T00:47:29.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.455+0000 7f1fa5cd6640 1 --2- 172.21.15.67:0/1675311306 >> v2:172.21.15.72:3300/0 conn(0x7f1fa00b1eb0 0x7f1fa0099d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:47:29.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.456+0000 7f1fa6cd8640 1 --2- 172.21.15.67:0/1675311306 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1fa009a240 0x7f1fa00cd410 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1f90009d90 tx=0x7f1f9002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92cce6c9495634df server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:29.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.456+0000 7f1fa54d5640 1 -- 172.21.15.67:0/1675311306 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f90039070 con 0x7f1fa009a240 2024-03-24T00:47:29.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.456+0000 7f1fa54d5640 1 -- 172.21.15.67:0/1675311306 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f9002cc50 con 0x7f1fa009a240 2024-03-24T00:47:29.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.456+0000 7f1fa54d5640 1 -- 172.21.15.67:0/1675311306 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f90034440 con 0x7f1fa009a240 2024-03-24T00:47:29.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.457+0000 7f1fa873b640 1 -- 172.21.15.67:0/1675311306 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1fa009a240 msgr2=0x7f1fa00cd410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:29.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.457+0000 7f1fa873b640 1 --2- 172.21.15.67:0/1675311306 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1fa009a240 0x7f1fa00cd410 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1f90009d90 tx=0x7f1f9002bea0 comp rx=0 tx=0).stop 2024-03-24T00:47:29.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.457+0000 7f1fa873b640 1 -- 172.21.15.67:0/1675311306 shutdown_connections 2024-03-24T00:47:29.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.457+0000 7f1fa873b640 1 --2- 172.21.15.67:0/1675311306 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1fa009a240 0x7f1fa00cd410 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:29.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.457+0000 7f1fa873b640 1 --2- 172.21.15.67:0/1675311306 >> v2:172.21.15.72:3300/0 conn(0x7f1fa00b1eb0 0x7f1fa0099d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:29.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.457+0000 7f1fa873b640 1 -- 172.21.15.67:0/1675311306 >> 172.21.15.67:0/1675311306 conn(0x7f1fa00b3650 msgr2=0x7f1fa00b3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:29.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.457+0000 7f1fa873b640 1 -- 172.21.15.67:0/1675311306 shutdown_connections 2024-03-24T00:47:29.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.457+0000 7f1fa873b640 1 -- 172.21.15.67:0/1675311306 wait complete. 2024-03-24T00:47:29.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.459+0000 7f1fa873b640 1 Processor -- start 2024-03-24T00:47:29.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.459+0000 7f1fa873b640 1 -- start start 2024-03-24T00:47:29.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.460+0000 7f1fa873b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1fa006de60 0x7f1fa0154d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:29.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.460+0000 7f1fa873b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fa009a240 0x7f1fa00c7340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:29.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.460+0000 7f1fa873b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1fa00b1eb0 0x7f1fa00cfbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:29.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.460+0000 7f1fa873b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa006c150 con 0x7f1fa009a240 2024-03-24T00:47:29.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.460+0000 7f1fa873b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa00ccc10 con 0x7f1fa00b1eb0 2024-03-24T00:47:29.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.460+0000 7f1fa873b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa006c2d0 con 0x7f1fa006de60 2024-03-24T00:47:29.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.460+0000 7f1fa5cd6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fa009a240 0x7f1fa00c7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:29.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.460+0000 7f1fa5cd6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fa009a240 0x7f1fa00c7340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60478/0 (socket says 172.21.15.67:60478) 2024-03-24T00:47:29.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.460+0000 7f1fa5cd6640 1 -- 172.21.15.67:0/4037904782 learned_addr learned my addr 172.21.15.67:0/4037904782 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:29.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.460+0000 7f1fa64d7640 1 --2- 172.21.15.67:0/4037904782 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1fa006de60 0x7f1fa0154d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:29.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.460+0000 7f1fa6cd8640 1 --2- 172.21.15.67:0/4037904782 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1fa00b1eb0 0x7f1fa00cfbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:29.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.460+0000 7f1fa5cd6640 1 -- 172.21.15.67:0/4037904782 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1fa006de60 msgr2=0x7f1fa0154d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:29.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.460+0000 7f1fa5cd6640 1 --2- 172.21.15.67:0/4037904782 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1fa006de60 0x7f1fa0154d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:29.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.460+0000 7f1fa5cd6640 1 -- 172.21.15.67:0/4037904782 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1fa00b1eb0 msgr2=0x7f1fa00cfbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:29.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.460+0000 7f1fa5cd6640 1 --2- 172.21.15.67:0/4037904782 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1fa00b1eb0 0x7f1fa00cfbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:29.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.460+0000 7f1fa5cd6640 1 -- 172.21.15.67:0/4037904782 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fa01552b0 con 0x7f1fa009a240 2024-03-24T00:47:29.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.460+0000 7f1fa6cd8640 1 --2- 172.21.15.67:0/4037904782 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1fa00b1eb0 0x7f1fa00cfbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:47:29.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.461+0000 7f1fa5cd6640 1 --2- 172.21.15.67:0/4037904782 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fa009a240 0x7f1fa00c7340 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f1f9400e9e0 tx=0x7f1f9400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:29.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.461+0000 7f1f9effd640 1 -- 172.21.15.67:0/4037904782 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f94017070 con 0x7f1fa009a240 2024-03-24T00:47:29.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.461+0000 7f1f9effd640 1 -- 172.21.15.67:0/4037904782 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f9400c9d0 con 0x7f1fa009a240 2024-03-24T00:47:29.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.461+0000 7f1fa873b640 1 -- 172.21.15.67:0/4037904782 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fa00cac90 con 0x7f1fa009a240 2024-03-24T00:47:29.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.461+0000 7f1f9effd640 1 -- 172.21.15.67:0/4037904782 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f9400cd80 con 0x7f1fa009a240 2024-03-24T00:47:29.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.461+0000 7f1fa873b640 1 -- 172.21.15.67:0/4037904782 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fa00cb250 con 0x7f1fa009a240 2024-03-24T00:47:29.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.462+0000 7f1fa873b640 1 -- 172.21.15.67:0/4037904782 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fa004f400 con 0x7f1fa009a240 2024-03-24T00:47:29.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.462+0000 7f1f9effd640 1 -- 172.21.15.67:0/4037904782 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1f9401a070 con 0x7f1fa009a240 2024-03-24T00:47:29.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.463+0000 7f1f9effd640 1 --2- 172.21.15.67:0/4037904782 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1f8c06bbd0 0x7f1f8c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:29.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.463+0000 7f1f9effd640 1 -- 172.21.15.67:0/4037904782 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1f9408aad0 con 0x7f1fa009a240 2024-03-24T00:47:29.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.463+0000 7f1fa64d7640 1 --2- 172.21.15.67:0/4037904782 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1f8c06bbd0 0x7f1f8c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:29.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.463+0000 7f1fa64d7640 1 --2- 172.21.15.67:0/4037904782 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1f8c06bbd0 0x7f1f8c06e090 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f1f98007fd0 tx=0x7f1f98009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:29.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.467+0000 7f1f9effd640 1 -- 172.21.15.67:0/4037904782 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1f94059430 con 0x7f1fa009a240 2024-03-24T00:47:29.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.661+0000 7f1fa873b640 1 -- 172.21.15.67:0/4037904782 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f1fa009a8c0 con 0x7f1fa009a240 2024-03-24T00:47:29.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.661+0000 7f1f9effd640 1 -- 172.21.15.67:0/4037904782 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v105) v1 ==== 80+0+0 (secure 0 0 0) 0x7f1f94058d30 con 0x7f1fa009a240 2024-03-24T00:47:29.662 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-03-24T00:47:29.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.663+0000 7f1fa873b640 1 -- 172.21.15.67:0/4037904782 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1f8c06bbd0 msgr2=0x7f1f8c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:29.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.663+0000 7f1fa873b640 1 --2- 172.21.15.67:0/4037904782 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1f8c06bbd0 0x7f1f8c06e090 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f1f98007fd0 tx=0x7f1f98009040 comp rx=0 tx=0).stop 2024-03-24T00:47:29.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.663+0000 7f1fa873b640 1 -- 172.21.15.67:0/4037904782 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fa009a240 msgr2=0x7f1fa00c7340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:29.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.663+0000 7f1fa873b640 1 --2- 172.21.15.67:0/4037904782 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fa009a240 0x7f1fa00c7340 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f1f9400e9e0 tx=0x7f1f9400eea0 comp rx=0 tx=0).stop 2024-03-24T00:47:29.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.663+0000 7f1fa873b640 1 -- 172.21.15.67:0/4037904782 shutdown_connections 2024-03-24T00:47:29.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.663+0000 7f1fa873b640 1 --2- 172.21.15.67:0/4037904782 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1f8c06bbd0 0x7f1f8c06e090 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:29.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.663+0000 7f1fa873b640 1 --2- 172.21.15.67:0/4037904782 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1fa00b1eb0 0x7f1fa00cfbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:29.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.663+0000 7f1fa873b640 1 --2- 172.21.15.67:0/4037904782 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fa009a240 0x7f1fa00c7340 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:29.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.663+0000 7f1fa873b640 1 --2- 172.21.15.67:0/4037904782 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1fa006de60 0x7f1fa0154d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:29.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.663+0000 7f1fa873b640 1 -- 172.21.15.67:0/4037904782 >> 172.21.15.67:0/4037904782 conn(0x7f1fa00b3650 msgr2=0x7f1fa00b0760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:29.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.663+0000 7f1fa873b640 1 -- 172.21.15.67:0/4037904782 shutdown_connections 2024-03-24T00:47:29.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.663+0000 7f1fa873b640 1 -- 172.21.15.67:0/4037904782 wait complete. 2024-03-24T00:47:29.675 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-24T00:47:29.675 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:47:29.675 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-24T00:47:29.729 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-03-24T00:47:29.729 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:47:29.729 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-24T00:47:29.754 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:47:29.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.896+0000 7fce339d4640 1 Processor -- start 2024-03-24T00:47:29.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.896+0000 7fce339d4640 1 -- start start 2024-03-24T00:47:29.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.897+0000 7fce339d4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fce2c15dec0 0x7fce2c0ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:29.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.897+0000 7fce339d4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fce2c0d0ef0 0x7fce2c162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:29.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.897+0000 7fce339d4640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce2c0cf640 con 0x7fce2c15e2a0 2024-03-24T00:47:29.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.897+0000 7fce339d4640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce2c0cee30 con 0x7fce2c15dec0 2024-03-24T00:47:29.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.897+0000 7fce339d4640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce2c0d2120 con 0x7fce2c0d0ef0 2024-03-24T00:47:29.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.897+0000 7fce31770640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fce2c15dec0 0x7fce2c0ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:29.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.897+0000 7fce31770640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fce2c15dec0 0x7fce2c0ce630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60486/0 (socket says 172.21.15.67:60486) 2024-03-24T00:47:29.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.897+0000 7fce31770640 1 -- 172.21.15.67:0/1074014891 learned_addr learned my addr 172.21.15.67:0/1074014891 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:29.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.897+0000 7fce31f71640 1 --2- 172.21.15.67:0/1074014891 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fce2c0d0ef0 0x7fce2c162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:29.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.897+0000 7fce31770640 1 -- 172.21.15.67:0/1074014891 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fce2c0d0ef0 msgr2=0x7fce2c162630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:29.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.897+0000 7fce31770640 1 --2- 172.21.15.67:0/1074014891 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fce2c0d0ef0 0x7fce2c162630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:29.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.897+0000 7fce31770640 1 -- 172.21.15.67:0/1074014891 >> v1:172.21.15.72:6789/0 conn(0x7fce2c15e2a0 legacy=0x7fce2c0d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:29.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.897+0000 7fce31770640 1 -- 172.21.15.67:0/1074014891 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce2c0cf080 con 0x7fce2c15dec0 2024-03-24T00:47:29.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.898+0000 7fce31770640 1 --2- 172.21.15.67:0/1074014891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce2c15dec0 0x7fce2c0ce630 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7fce28009870 tx=0x7fce2802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d4206c21e8d9684 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:29.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.898+0000 7fce23fff640 1 -- 172.21.15.67:0/1074014891 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce280317c0 con 0x7fce2c15dec0 2024-03-24T00:47:29.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.898+0000 7fce23fff640 1 -- 172.21.15.67:0/1074014891 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce28037070 con 0x7fce2c15dec0 2024-03-24T00:47:29.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.898+0000 7fce23fff640 1 -- 172.21.15.67:0/1074014891 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce28031b80 con 0x7fce2c15dec0 2024-03-24T00:47:29.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.898+0000 7fce339d4640 1 -- 172.21.15.67:0/1074014891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce2c15dec0 msgr2=0x7fce2c0ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:29.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.898+0000 7fce339d4640 1 --2- 172.21.15.67:0/1074014891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce2c15dec0 0x7fce2c0ce630 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7fce28009870 tx=0x7fce2802b860 comp rx=0 tx=0).stop 2024-03-24T00:47:29.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.898+0000 7fce339d4640 1 -- 172.21.15.67:0/1074014891 shutdown_connections 2024-03-24T00:47:29.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.898+0000 7fce339d4640 1 --2- 172.21.15.67:0/1074014891 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fce2c0d0ef0 0x7fce2c162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:29.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.898+0000 7fce339d4640 1 --2- 172.21.15.67:0/1074014891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce2c15dec0 0x7fce2c0ce630 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:29.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.899+0000 7fce339d4640 1 -- 172.21.15.67:0/1074014891 >> 172.21.15.67:0/1074014891 conn(0x7fce2c06c810 msgr2=0x7fce2c06cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:29.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.899+0000 7fce339d4640 1 -- 172.21.15.67:0/1074014891 shutdown_connections 2024-03-24T00:47:29.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.899+0000 7fce339d4640 1 -- 172.21.15.67:0/1074014891 wait complete. 2024-03-24T00:47:29.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.900+0000 7fce339d4640 1 Processor -- start 2024-03-24T00:47:29.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.900+0000 7fce339d4640 1 -- start start 2024-03-24T00:47:29.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.900+0000 7fce339d4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fce2c0d0ef0 0x7fce2c15f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:29.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.900+0000 7fce339d4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fce2c15dec0 0x7fce2c0ab2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:29.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.900+0000 7fce339d4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce2c15e2a0 0x7fce2c0ab820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:29.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.900+0000 7fce339d4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce2c0d2730 con 0x7fce2c15e2a0 2024-03-24T00:47:29.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.900+0000 7fce339d4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce2c0d25b0 con 0x7fce2c15dec0 2024-03-24T00:47:29.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.900+0000 7fce339d4640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce2c0d28b0 con 0x7fce2c0d0ef0 2024-03-24T00:47:29.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.901+0000 7fce31f71640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce2c15e2a0 0x7fce2c0ab820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:29.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.901+0000 7fce31f71640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce2c15e2a0 0x7fce2c0ab820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60500/0 (socket says 172.21.15.67:60500) 2024-03-24T00:47:29.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.901+0000 7fce31f71640 1 -- 172.21.15.67:0/954899970 learned_addr learned my addr 172.21.15.67:0/954899970 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:29.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.901+0000 7fce30f6f640 1 --2- 172.21.15.67:0/954899970 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fce2c15dec0 0x7fce2c0ab2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:29.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.901+0000 7fce31770640 1 --2- 172.21.15.67:0/954899970 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fce2c0d0ef0 0x7fce2c15f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:29.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.901+0000 7fce31f71640 1 -- 172.21.15.67:0/954899970 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fce2c0d0ef0 msgr2=0x7fce2c15f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:29.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.901+0000 7fce31f71640 1 --2- 172.21.15.67:0/954899970 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fce2c0d0ef0 0x7fce2c15f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:29.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.901+0000 7fce31f71640 1 -- 172.21.15.67:0/954899970 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fce2c15dec0 msgr2=0x7fce2c0ab2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:29.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.901+0000 7fce31f71640 1 --2- 172.21.15.67:0/954899970 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fce2c15dec0 0x7fce2c0ab2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:29.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.901+0000 7fce31f71640 1 -- 172.21.15.67:0/954899970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce2c0abd90 con 0x7fce2c15e2a0 2024-03-24T00:47:29.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.901+0000 7fce30f6f640 1 --2- 172.21.15.67:0/954899970 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fce2c15dec0 0x7fce2c0ab2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:47:29.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.901+0000 7fce31f71640 1 --2- 172.21.15.67:0/954899970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce2c15e2a0 0x7fce2c0ab820 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7fce2400cd40 tx=0x7fce2400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:29.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.901+0000 7fce21ffb640 1 -- 172.21.15.67:0/954899970 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce24003b90 con 0x7fce2c15e2a0 2024-03-24T00:47:29.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.901+0000 7fce21ffb640 1 -- 172.21.15.67:0/954899970 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce24003d30 con 0x7fce2c15e2a0 2024-03-24T00:47:29.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.901+0000 7fce339d4640 1 -- 172.21.15.67:0/954899970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce2c0cb160 con 0x7fce2c15e2a0 2024-03-24T00:47:29.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.901+0000 7fce21ffb640 1 -- 172.21.15.67:0/954899970 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce24004240 con 0x7fce2c15e2a0 2024-03-24T00:47:29.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.901+0000 7fce339d4640 1 -- 172.21.15.67:0/954899970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce2c0cb720 con 0x7fce2c15e2a0 2024-03-24T00:47:29.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.903+0000 7fce339d4640 1 -- 172.21.15.67:0/954899970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce2c04f400 con 0x7fce2c15e2a0 2024-03-24T00:47:29.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.903+0000 7fce21ffb640 1 -- 172.21.15.67:0/954899970 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fce240043e0 con 0x7fce2c15e2a0 2024-03-24T00:47:29.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.903+0000 7fce21ffb640 1 --2- 172.21.15.67:0/954899970 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fce1006bbd0 0x7fce1006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:29.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.903+0000 7fce21ffb640 1 -- 172.21.15.67:0/954899970 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fce2408b5d0 con 0x7fce2c15e2a0 2024-03-24T00:47:29.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.903+0000 7fce31770640 1 --2- 172.21.15.67:0/954899970 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fce1006bbd0 0x7fce1006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:29.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.904+0000 7fce31770640 1 --2- 172.21.15.67:0/954899970 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fce1006bbd0 0x7fce1006e090 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fce28036000 tx=0x7fce28008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:29.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:29.907+0000 7fce21ffb640 1 -- 172.21.15.67:0/954899970 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fce2c0abd90 con 0x7fce2c15e2a0 2024-03-24T00:47:30.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.127+0000 7fce339d4640 1 -- 172.21.15.67:0/954899970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce2c0caea0 con 0x7fce2c15e2a0 2024-03-24T00:47:30.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.128+0000 7fce21ffb640 1 -- 172.21.15.67:0/954899970 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+4976 (secure 0 0 0) 0x7fce24059fc0 con 0x7fce2c15e2a0 2024-03-24T00:47:30.129 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:47:30.129 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":72,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6902,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3672294289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3672294289},{"type":"v1","addr":"172.21.15.67:6835","nonce":3672294289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":6908,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/1286271878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":1286271878},{"type":"v1","addr":"172.21.15.72:6836","nonce":1286271878}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":6911,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/140622129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":140622129},{"type":"v1","addr":"172.21.15.72:6837","nonce":140622129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:47:16.492485+0000","modified":"2024-03-24T00:47:18.507677+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8541},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8541":{"gid":8541,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6833/1826974843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1826974843},{"type":"v1","addr":"172.21.15.67:6833","nonce":1826974843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8541,"qdb_cluster":[8541]},"id":7}]} 2024-03-24T00:47:30.129 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 72 2024-03-24T00:47:30.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.130+0000 7fce339d4640 1 -- 172.21.15.67:0/954899970 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fce1006bbd0 msgr2=0x7fce1006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:30.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.130+0000 7fce339d4640 1 --2- 172.21.15.67:0/954899970 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fce1006bbd0 0x7fce1006e090 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fce28036000 tx=0x7fce28008030 comp rx=0 tx=0).stop 2024-03-24T00:47:30.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.130+0000 7fce339d4640 1 -- 172.21.15.67:0/954899970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce2c15e2a0 msgr2=0x7fce2c0ab820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:30.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.130+0000 7fce339d4640 1 --2- 172.21.15.67:0/954899970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce2c15e2a0 0x7fce2c0ab820 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7fce2400cd40 tx=0x7fce2400a660 comp rx=0 tx=0).stop 2024-03-24T00:47:30.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.130+0000 7fce339d4640 1 -- 172.21.15.67:0/954899970 shutdown_connections 2024-03-24T00:47:30.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.130+0000 7fce339d4640 1 --2- 172.21.15.67:0/954899970 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fce1006bbd0 0x7fce1006e090 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.130+0000 7fce339d4640 1 --2- 172.21.15.67:0/954899970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce2c15e2a0 0x7fce2c0ab820 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.130+0000 7fce339d4640 1 --2- 172.21.15.67:0/954899970 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fce2c15dec0 0x7fce2c0ab2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.130+0000 7fce339d4640 1 --2- 172.21.15.67:0/954899970 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fce2c0d0ef0 0x7fce2c15f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.130+0000 7fce339d4640 1 -- 172.21.15.67:0/954899970 >> 172.21.15.67:0/954899970 conn(0x7fce2c06c810 msgr2=0x7fce2c15bf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:30.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.130+0000 7fce339d4640 1 -- 172.21.15.67:0/954899970 shutdown_connections 2024-03-24T00:47:30.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.130+0000 7fce339d4640 1 -- 172.21.15.67:0/954899970 wait complete. 2024-03-24T00:47:30.142 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-24T00:47:30.273 INFO:teuthology.orchestra.run.smithi067.stdout:{"message":"","return_code":0} 2024-03-24T00:47:30.274 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-24T00:47:30.274 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:47:30.274 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:47:30.288 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056375,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15771 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T00:47:30.289 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-03-24T00:47:30.289 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:47:30.289 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-24T00:47:30.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:47:30.364 INFO:teuthology.orchestra.run.smithi067.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-24T00:47:30.364 DEBUG:teuthology.orchestra.run.smithi067:> sudo daemon-helper kill ceph --cluster ceph -w 2024-03-24T00:47:30.406 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:47:30.406 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.7:0 scrub start / repair,recursive 2024-03-24T00:47:30.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.511+0000 7fdc0513b640 1 Processor -- start 2024-03-24T00:47:30.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.511+0000 7fdc0513b640 1 -- start start 2024-03-24T00:47:30.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.512+0000 7fdc0513b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fdc0015dfc0 0x7fdc001624c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:30.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.512+0000 7fdc0513b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdc000d0d50 0x7fdc000d2b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:30.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.512+0000 7fdc0513b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc00162a00 con 0x7fdc000ced60 2024-03-24T00:47:30.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.512+0000 7fdc0513b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc0015e3a0 con 0x7fdc000d0d50 2024-03-24T00:47:30.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.512+0000 7fdc0513b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc000cf4d0 con 0x7fdc0015dfc0 2024-03-24T00:47:30.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.513+0000 7fdbff59e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdc000d0d50 0x7fdc000d2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:30.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.513+0000 7fdbff59e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdc000d0d50 0x7fdc000d2b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60504/0 (socket says 172.21.15.67:60504) 2024-03-24T00:47:30.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.513+0000 7fdbff59e640 1 -- 172.21.15.67:0/1644305838 learned_addr learned my addr 172.21.15.67:0/1644305838 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:30.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.513+0000 7fdbfed9d640 1 --2- 172.21.15.67:0/1644305838 >> v2:172.21.15.72:3300/0 conn(0x7fdc0015dfc0 0x7fdc001624c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:30.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.513+0000 7fdbff59e640 1 -- 172.21.15.67:0/1644305838 >> v2:172.21.15.72:3300/0 conn(0x7fdc0015dfc0 msgr2=0x7fdc001624c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:30.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.513+0000 7fdbff59e640 1 --2- 172.21.15.67:0/1644305838 >> v2:172.21.15.72:3300/0 conn(0x7fdc0015dfc0 0x7fdc001624c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.513+0000 7fdbff59e640 1 -- 172.21.15.67:0/1644305838 >> v1:172.21.15.72:6789/0 conn(0x7fdc000ced60 legacy=0x7fdc000d0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:30.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.513+0000 7fdbff59e640 1 -- 172.21.15.67:0/1644305838 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc000cf1f0 con 0x7fdc000d0d50 2024-03-24T00:47:30.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.513+0000 7fdbfed9d640 1 --2- 172.21.15.67:0/1644305838 >> v2:172.21.15.72:3300/0 conn(0x7fdc0015dfc0 0x7fdc001624c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:47:30.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.514+0000 7fdbff59e640 1 --2- 172.21.15.67:0/1644305838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc000d0d50 0x7fdc000d2b00 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7fdc0006b450 tx=0x7fdbf802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f054085278906800 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:30.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.514+0000 7fdbfe59c640 1 -- 172.21.15.67:0/1644305838 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf802ca80 con 0x7fdc000d0d50 2024-03-24T00:47:30.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.514+0000 7fdbfe59c640 1 -- 172.21.15.67:0/1644305838 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbf802cc20 con 0x7fdc000d0d50 2024-03-24T00:47:30.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.514+0000 7fdbfe59c640 1 -- 172.21.15.67:0/1644305838 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf80343f0 con 0x7fdc000d0d50 2024-03-24T00:47:30.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.514+0000 7fdc0513b640 1 -- 172.21.15.67:0/1644305838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc000d0d50 msgr2=0x7fdc000d2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:30.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.514+0000 7fdc0513b640 1 --2- 172.21.15.67:0/1644305838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc000d0d50 0x7fdc000d2b00 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7fdc0006b450 tx=0x7fdbf802bee0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.515+0000 7fdc0513b640 1 -- 172.21.15.67:0/1644305838 shutdown_connections 2024-03-24T00:47:30.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.515+0000 7fdc0513b640 1 --2- 172.21.15.67:0/1644305838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc000d0d50 0x7fdc000d2b00 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.515+0000 7fdc0513b640 1 --2- 172.21.15.67:0/1644305838 >> v2:172.21.15.72:3300/0 conn(0x7fdc0015dfc0 0x7fdc001624c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.515+0000 7fdc0513b640 1 -- 172.21.15.67:0/1644305838 >> 172.21.15.67:0/1644305838 conn(0x7fdc0006c880 msgr2=0x7fdc0006cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:30.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.515+0000 7fdc0513b640 1 -- 172.21.15.67:0/1644305838 shutdown_connections 2024-03-24T00:47:30.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.515+0000 7fdc0513b640 1 -- 172.21.15.67:0/1644305838 wait complete. 2024-03-24T00:47:30.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.516+0000 7fdc0513b640 1 Processor -- start 2024-03-24T00:47:30.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.516+0000 7fdc0513b640 1 -- start start 2024-03-24T00:47:30.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.517+0000 7fdc0513b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdc000d0d50 0x7fdc000b4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:30.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.517+0000 7fdc0513b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc0015dfc0 0x7fdc0015fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:30.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.517+0000 7fdc0513b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc000b6440 0x7fdc00160270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:30.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.517+0000 7fdc0513b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc000d25e0 con 0x7fdc000b6440 2024-03-24T00:47:30.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.517+0000 7fdc0513b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc000d2300 con 0x7fdc0015dfc0 2024-03-24T00:47:30.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.517+0000 7fdc0513b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc000b6ea0 con 0x7fdc000d0d50 2024-03-24T00:47:30.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.517+0000 7fdbff59e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc000b6440 0x7fdc00160270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:30.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.517+0000 7fdbff59e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc000b6440 0x7fdc00160270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60518/0 (socket says 172.21.15.67:60518) 2024-03-24T00:47:30.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.517+0000 7fdbff59e640 1 -- 172.21.15.67:0/1580313941 learned_addr learned my addr 172.21.15.67:0/1580313941 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:30.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.517+0000 7fdbfed9d640 1 --2- 172.21.15.67:0/1580313941 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdc000d0d50 0x7fdc000b4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:30.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.517+0000 7fdbf7fff640 1 --2- 172.21.15.67:0/1580313941 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc0015dfc0 0x7fdc0015fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:30.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.517+0000 7fdbff59e640 1 -- 172.21.15.67:0/1580313941 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdc000d0d50 msgr2=0x7fdc000b4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:30.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.517+0000 7fdbff59e640 1 --2- 172.21.15.67:0/1580313941 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdc000d0d50 0x7fdc000b4cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.517+0000 7fdbff59e640 1 -- 172.21.15.67:0/1580313941 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc0015dfc0 msgr2=0x7fdc0015fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:30.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.517+0000 7fdbff59e640 1 --2- 172.21.15.67:0/1580313941 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc0015dfc0 0x7fdc0015fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.517+0000 7fdbff59e640 1 -- 172.21.15.67:0/1580313941 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc000b53a0 con 0x7fdc000b6440 2024-03-24T00:47:30.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.517+0000 7fdbfed9d640 1 --2- 172.21.15.67:0/1580313941 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdc000d0d50 0x7fdc000b4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:47:30.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.517+0000 7fdbf7fff640 1 --2- 172.21.15.67:0/1580313941 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc0015dfc0 0x7fdc0015fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:47:30.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.517+0000 7fdbff59e640 1 --2- 172.21.15.67:0/1580313941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc000b6440 0x7fdc00160270 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8009f40 tx=0x7fdbf8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:30.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.517+0000 7fdbf77fe640 1 -- 172.21.15.67:0/1580313941 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf8043020 con 0x7fdc000b6440 2024-03-24T00:47:30.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.518+0000 7fdbf77fe640 1 -- 172.21.15.67:0/1580313941 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbf803e450 con 0x7fdc000b6440 2024-03-24T00:47:30.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.518+0000 7fdc0513b640 1 -- 172.21.15.67:0/1580313941 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc000b5680 con 0x7fdc000b6440 2024-03-24T00:47:30.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.518+0000 7fdbf77fe640 1 -- 172.21.15.67:0/1580313941 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf803e860 con 0x7fdc000b6440 2024-03-24T00:47:30.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.518+0000 7fdc0513b640 1 -- 172.21.15.67:0/1580313941 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc000b5c40 con 0x7fdc000b6440 2024-03-24T00:47:30.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.519+0000 7fdbf77fe640 1 -- 172.21.15.67:0/1580313941 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fdbf8046020 con 0x7fdc000b6440 2024-03-24T00:47:30.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.519+0000 7fdbf77fe640 1 --2- 172.21.15.67:0/1580313941 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdbe006bbd0 0x7fdbe006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:30.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.520+0000 7fdbf77fe640 1 -- 172.21.15.67:0/1580313941 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fdbf80ad610 con 0x7fdc000b6440 2024-03-24T00:47:30.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.520+0000 7fdbfed9d640 1 --2- 172.21.15.67:0/1580313941 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdbe006bbd0 0x7fdbe006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:30.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.520+0000 7fdbfed9d640 1 --2- 172.21.15.67:0/1580313941 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdbe006bbd0 0x7fdbe006e090 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fdbf000a990 tx=0x7fdbf0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:30.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.521+0000 7fdc07e08740 1 Processor -- start 2024-03-24T00:47:30.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.522+0000 7fdc07e08740 1 -- start start 2024-03-24T00:47:30.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.523+0000 7fdc07e08740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e346be0e60 0x55e346c07990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:30.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.523+0000 7fdc07e08740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e346c07ed0 0x55e346bdfea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:30.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.523+0000 7fdc07e08740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e346be1240 0x55e346be03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:30.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.523+0000 7fdc07e08740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e346b1fc90 con 0x55e346c07ed0 2024-03-24T00:47:30.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.523+0000 7fdc07e08740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e346b45070 con 0x55e346be1240 2024-03-24T00:47:30.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.523+0000 7fdc07e08740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e346aedc80 con 0x55e346be0e60 2024-03-24T00:47:30.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.523+0000 7fdbf7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e346c07ed0 0x55e346bdfea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:30.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.523+0000 7fdbf7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e346c07ed0 0x55e346bdfea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60526/0 (socket says 172.21.15.67:60526) 2024-03-24T00:47:30.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.523+0000 7fdbf7fff640 1 -- 172.21.15.67:0/3170126442 learned_addr learned my addr 172.21.15.67:0/3170126442 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:30.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.523+0000 7fdbff59e640 1 --2- 172.21.15.67:0/3170126442 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e346be1240 0x55e346be03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:30.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.523+0000 7fdbfed9d640 1 --2- 172.21.15.67:0/3170126442 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e346be0e60 0x55e346c07990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:30.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.523+0000 7fdbf7fff640 1 -- 172.21.15.67:0/3170126442 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e346be0e60 msgr2=0x55e346c07990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:30.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.523+0000 7fdbf7fff640 1 --2- 172.21.15.67:0/3170126442 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e346be0e60 0x55e346c07990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.523+0000 7fdbf7fff640 1 -- 172.21.15.67:0/3170126442 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e346be1240 msgr2=0x55e346be03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:30.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.523+0000 7fdbf7fff640 1 --2- 172.21.15.67:0/3170126442 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e346be1240 0x55e346be03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.523+0000 7fdbf7fff640 1 -- 172.21.15.67:0/3170126442 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e346ae5d10 con 0x55e346c07ed0 2024-03-24T00:47:30.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.523+0000 7fdbfed9d640 1 --2- 172.21.15.67:0/3170126442 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e346be0e60 0x55e346c07990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:47:30.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.524+0000 7fdbf7fff640 1 --2- 172.21.15.67:0/3170126442 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e346c07ed0 0x55e346bdfea0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7fdbe8003d60 tx=0x7fdbe8004220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81a8d2cb5ead2038 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:30.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.524+0000 7fdbf4ff9640 1 -- 172.21.15.67:0/3170126442 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbe8018020 con 0x55e346c07ed0 2024-03-24T00:47:30.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.524+0000 7fdbf4ff9640 1 -- 172.21.15.67:0/3170126442 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbe8005030 con 0x55e346c07ed0 2024-03-24T00:47:30.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.524+0000 7fdbf4ff9640 1 -- 172.21.15.67:0/3170126442 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbe8005400 con 0x55e346c07ed0 2024-03-24T00:47:30.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.525+0000 7fdc07e08740 1 -- 172.21.15.67:0/3170126442 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e346c07ed0 msgr2=0x55e346bdfea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:30.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.525+0000 7fdc07e08740 1 --2- 172.21.15.67:0/3170126442 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e346c07ed0 0x55e346bdfea0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7fdbe8003d60 tx=0x7fdbe8004220 comp rx=0 tx=0).stop 2024-03-24T00:47:30.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.525+0000 7fdc07e08740 1 -- 172.21.15.67:0/3170126442 shutdown_connections 2024-03-24T00:47:30.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.525+0000 7fdc07e08740 1 --2- 172.21.15.67:0/3170126442 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e346be1240 0x55e346be03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.525+0000 7fdc07e08740 1 --2- 172.21.15.67:0/3170126442 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e346c07ed0 0x55e346bdfea0 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.525+0000 7fdc07e08740 1 --2- 172.21.15.67:0/3170126442 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e346be0e60 0x55e346c07990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.525+0000 7fdc07e08740 1 -- 172.21.15.67:0/3170126442 >> 172.21.15.67:0/3170126442 conn(0x55e346ba81a0 msgr2=0x55e346c25b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:30.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.525+0000 7fdc07e08740 1 -- 172.21.15.67:0/3170126442 shutdown_connections 2024-03-24T00:47:30.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.525+0000 7fdc07e08740 1 -- 172.21.15.67:0/3170126442 wait complete. 2024-03-24T00:47:30.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.525+0000 7fdc07e08740 1 -- start start 2024-03-24T00:47:30.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.526+0000 7fdc07e08740 1 Processor -- start 2024-03-24T00:47:30.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.526+0000 7fdc07e08740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e346be0e60 0x55e346c0e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:30.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.526+0000 7fdc07e08740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e346be1240 0x55e346c38e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:30.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.526+0000 7fdc07e08740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e346c04c10 0x55e346c05090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:30.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.526+0000 7fdc07e08740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e346b010d0 con 0x55e346be0e60 2024-03-24T00:47:30.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.526+0000 7fdc07e08740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e346b41110 con 0x55e346c04c10 2024-03-24T00:47:30.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.526+0000 7fdc07e08740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e346c07500 con 0x55e346be1240 2024-03-24T00:47:30.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.526+0000 7fdc07e08740 10 client.0 _ll_register_callbacks cb 0x55e346b3ce60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:47:30.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.526+0000 7fdbfed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e346be0e60 0x55e346c0e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:30.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.527+0000 7fdbfed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e346be0e60 0x55e346c0e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60528/0 (socket says 172.21.15.67:60528) 2024-03-24T00:47:30.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.527+0000 7fdbfed9d640 1 -- 172.21.15.67:0/1157795297 learned_addr learned my addr 172.21.15.67:0/1157795297 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:30.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.527+0000 7fdbf7fff640 1 --2- 172.21.15.67:0/1157795297 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e346be1240 0x55e346c38e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:30.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.527+0000 7fdbff59e640 1 --2- 172.21.15.67:0/1157795297 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e346c04c10 0x55e346c05090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:30.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.527+0000 7fdbfed9d640 1 -- 172.21.15.67:0/1157795297 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e346be1240 msgr2=0x55e346c38e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:30.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.527+0000 7fdbfed9d640 1 --2- 172.21.15.67:0/1157795297 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e346be1240 0x55e346c38e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.527+0000 7fdbfed9d640 1 -- 172.21.15.67:0/1157795297 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e346c04c10 msgr2=0x55e346c05090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:30.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.527+0000 7fdbfed9d640 1 --2- 172.21.15.67:0/1157795297 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e346c04c10 0x55e346c05090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.527+0000 7fdbfed9d640 1 -- 172.21.15.67:0/1157795297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e3469217e0 con 0x55e346be0e60 2024-03-24T00:47:30.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.527+0000 7fdbfed9d640 1 --2- 172.21.15.67:0/1157795297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e346be0e60 0x55e346c0e3f0 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7fdbf000a380 tx=0x7fdbf0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ae42571651d3aba server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:30.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.527+0000 7fdbddffb640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:47:30.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.527+0000 7fdbddffb640 1 -- 172.21.15.67:0/1157795297 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf00158e0 con 0x55e346be0e60 2024-03-24T00:47:30.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.527+0000 7fdbddffb640 1 -- 172.21.15.67:0/1157795297 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fdbf0015a80 con 0x55e346be0e60 2024-03-24T00:47:30.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.527+0000 7fdc07e08740 1 -- 172.21.15.67:0/1157795297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdbe8005030 con 0x55e346be0e60 2024-03-24T00:47:30.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.527+0000 7fdbddffb640 1 -- 172.21.15.67:0/1157795297 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdbf0015c80 con 0x55e346be0e60 2024-03-24T00:47:30.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.527+0000 7fdc07e08740 10 client.8832 fetch_fsmap learned FSMap version 72 2024-03-24T00:47:30.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.527+0000 7fdc07e08740 1 -- 172.21.15.67:0/1157795297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=72,monmap=2+}) v3 -- 0x55e346981f70 con 0x55e346be0e60 2024-03-24T00:47:30.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.528+0000 7fdbddffb640 1 -- 172.21.15.67:0/1157795297 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbe8005030 con 0x55e346be0e60 2024-03-24T00:47:30.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.528+0000 7fdbddffb640 1 -- 172.21.15.67:0/1157795297 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 72) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fdbf0005990 con 0x55e346be0e60 2024-03-24T00:47:30.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.528+0000 7fdc07e08740 10 client.8832 fetch_fsmap finished waiting for FSMap version 72 2024-03-24T00:47:30.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.528+0000 7fdc07e08740 10 client.8832 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.c 2024-03-24T00:47:30.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.528+0000 7fdc07e08740 20 client.8832 populate_metadata read hostname 'smithi067' 2024-03-24T00:47:30.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.528+0000 7fdc07e08740 1 --2- 172.21.15.67:0/1157795297 >> [v2:172.21.15.67:6832/1826974843,v1:172.21.15.67:6833/1826974843] conn(0x55e346c00750 0x55e346c00b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:30.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.528+0000 7fdc07e08740 4 client.8832 mds_command: new command op to 8541 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:47:30.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.528+0000 7fdc07e08740 1 -- 172.21.15.67:0/1157795297 --> [v2:172.21.15.67:6832/1826974843,v1:172.21.15.67:6833/1826974843] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e346b525b0 con 0x55e346c00750 2024-03-24T00:47:30.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.528+0000 7fdbf7fff640 1 --2- 172.21.15.67:0/1157795297 >> [v2:172.21.15.67:6832/1826974843,v1:172.21.15.67:6833/1826974843] conn(0x55e346c00750 0x55e346c00b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:30.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.528+0000 7fdbddffb640 10 client.8832 ms_handle_connect on v2:172.21.15.67:6832/1826974843 2024-03-24T00:47:30.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.528+0000 7fdbf7fff640 1 --2- 172.21.15.67:0/1157795297 >> [v2:172.21.15.67:6832/1826974843,v1:172.21.15.67:6833/1826974843] conn(0x55e346c00750 0x55e346c00b80 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ea85b2e03534665 server_cookie=1b82c5b65ef9e820 in_seq=0 out_seq=0 2024-03-24T00:47:30.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.529+0000 7fdbddffb640 1 -- 172.21.15.67:0/1157795297 <== mds.0 v2:172.21.15.67:6832/1826974843 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e346b525b0 con 0x55e346c00750 2024-03-24T00:47:30.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.529+0000 7fdbddffb640 10 client.8832 handle_command_reply: tid=0 2024-03-24T00:47:30.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.529+0000 7fdc07e08740 1 client.8832 shutdown 2024-03-24T00:47:30.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.529+0000 7fdc07e08740 1 -- 172.21.15.67:0/1157795297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e346be0e60 msgr2=0x55e346c0e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:30.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.529+0000 7fdc07e08740 1 --2- 172.21.15.67:0/1157795297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e346be0e60 0x55e346c0e3f0 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7fdbf000a380 tx=0x7fdbf0004730 comp rx=0 tx=0).stop 2024-03-24T00:47:30.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.529+0000 7fdc07e08740 1 -- 172.21.15.67:0/1157795297 shutdown_connections 2024-03-24T00:47:30.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.529+0000 7fdc07e08740 1 --2- 172.21.15.67:0/1157795297 >> [v2:172.21.15.67:6832/1826974843,v1:172.21.15.67:6833/1826974843] conn(0x55e346c00750 0x55e346c00b80 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.529+0000 7fdc07e08740 1 --2- 172.21.15.67:0/1157795297 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e346c04c10 0x55e346c05090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.529+0000 7fdc07e08740 1 --2- 172.21.15.67:0/1157795297 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e346be1240 0x55e346c38e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.529+0000 7fdc07e08740 1 --2- 172.21.15.67:0/1157795297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e346be0e60 0x55e346c0e3f0 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.529+0000 7fdbddffb640 0 client.8832 ms_handle_reset on v2:172.21.15.67:6832/1826974843 2024-03-24T00:47:30.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.529+0000 7fdc07e08740 1 -- 172.21.15.67:0/1157795297 >> 172.21.15.67:0/1157795297 conn(0x55e346ba81a0 msgr2=0x55e346c069d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:30.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.529+0000 7fdc07e08740 1 -- 172.21.15.67:0/1157795297 shutdown_connections 2024-03-24T00:47:30.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.529+0000 7fdc07e08740 1 -- 172.21.15.67:0/1157795297 wait complete. 2024-03-24T00:47:30.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.529+0000 7fdc07e08740 20 client.8832 trim_cache size 0 max 16384 2024-03-24T00:47:30.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.534+0000 7f75f56ff640 1 Processor -- start 2024-03-24T00:47:30.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.534+0000 7f75f56ff640 1 -- start start 2024-03-24T00:47:30.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.535+0000 7f75f56ff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f75f015bb70 0x7f75f01629a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:30.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.535+0000 7f75f56ff640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75f0161f00 con 0x7f75f015bf50 2024-03-24T00:47:30.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.535+0000 7f75f56ff640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75f00cf540 con 0x7f75f00d0f80 2024-03-24T00:47:30.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.535+0000 7f75f56ff640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75f0162510 con 0x7f75f015bb70 2024-03-24T00:47:30.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.536+0000 7f75eeffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f75f015bb70 0x7f75f01629a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:30.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.536+0000 7f75eeffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f75f015bb70 0x7f75f01629a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60530/0 (socket says 172.21.15.67:60530) 2024-03-24T00:47:30.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.536+0000 7f75ee7fc640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f75f015bf50 0x7f75f00d0870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45346/0 (socket says 172.21.15.67:45346) 2024-03-24T00:47:30.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.536+0000 7f75eeffd640 1 -- 172.21.15.67:0/2979559129 learned_addr learned my addr 172.21.15.67:0/2979559129 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:30.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.536+0000 7f75eeffd640 1 -- 172.21.15.67:0/2979559129 >> v1:172.21.15.72:6789/0 conn(0x7f75f00d0f80 legacy=0x7f75f00ce8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:30.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.536+0000 7f75eeffd640 1 -- 172.21.15.67:0/2979559129 >> v1:172.21.15.67:6789/0 conn(0x7f75f015bf50 legacy=0x7f75f00d0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:30.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.536+0000 7f75eeffd640 1 -- 172.21.15.67:0/2979559129 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f01621a0 con 0x7f75f015bb70 2024-03-24T00:47:30.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.537+0000 7f75eeffd640 1 --2- 172.21.15.67:0/2979559129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75f015bb70 0x7f75f01629a0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f75e0009870 tx=0x7f75e002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40c42f2d1cdaf30a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:30.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.537+0000 7f75edffb640 1 -- 172.21.15.67:0/2979559129 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e0035020 con 0x7f75f015bb70 2024-03-24T00:47:30.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.537+0000 7f75edffb640 1 -- 172.21.15.67:0/2979559129 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75e0031870 con 0x7f75f015bb70 2024-03-24T00:47:30.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.537+0000 7f75edffb640 1 -- 172.21.15.67:0/2979559129 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e0031c40 con 0x7f75f015bb70 2024-03-24T00:47:30.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.538+0000 7f75f56ff640 1 -- 172.21.15.67:0/2979559129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75f015bb70 msgr2=0x7f75f01629a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:30.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.538+0000 7f75f56ff640 1 --2- 172.21.15.67:0/2979559129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75f015bb70 0x7f75f01629a0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f75e0009870 tx=0x7f75e002b910 comp rx=0 tx=0).stop 2024-03-24T00:47:30.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.538+0000 7f75f56ff640 1 -- 172.21.15.67:0/2979559129 shutdown_connections 2024-03-24T00:47:30.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.538+0000 7f75f56ff640 1 --2- 172.21.15.67:0/2979559129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75f015bb70 0x7f75f01629a0 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.538+0000 7f75f56ff640 1 -- 172.21.15.67:0/2979559129 >> 172.21.15.67:0/2979559129 conn(0x7f75f006c8a0 msgr2=0x7f75f006ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:30.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.538+0000 7f75f56ff640 1 -- 172.21.15.67:0/2979559129 shutdown_connections 2024-03-24T00:47:30.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.538+0000 7f75f56ff640 1 -- 172.21.15.67:0/2979559129 wait complete. 2024-03-24T00:47:30.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.539+0000 7f75f56ff640 1 Processor -- start 2024-03-24T00:47:30.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.539+0000 7f75f56ff640 1 -- start start 2024-03-24T00:47:30.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.540+0000 7f75f56ff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f75f00d0f80 0x7f75f00b4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:30.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.540+0000 7f75f56ff640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f75f015bb70 0x7f75f0158d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:30.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.540+0000 7f75f56ff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75f015bf50 0x7f75f00ccdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:30.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.540+0000 7f75f56ff640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f00cefd0 con 0x7f75f015bf50 2024-03-24T00:47:30.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.540+0000 7f75f56ff640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f00cf820 con 0x7f75f00d0f80 2024-03-24T00:47:30.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.540+0000 7f75f56ff640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75f00cf150 con 0x7f75f015bb70 2024-03-24T00:47:30.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.540+0000 7f75ef7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75f015bf50 0x7f75f00ccdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:30.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.540+0000 7f75ef7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75f015bf50 0x7f75f00ccdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60544/0 (socket says 172.21.15.67:60544) 2024-03-24T00:47:30.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.540+0000 7f75ef7fe640 1 -- 172.21.15.67:0/1288347833 learned_addr learned my addr 172.21.15.67:0/1288347833 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:30.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.540+0000 7f75eeffd640 1 --2- 172.21.15.67:0/1288347833 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f75f00d0f80 0x7f75f00b4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:30.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.540+0000 7f75ee7fc640 1 --2- 172.21.15.67:0/1288347833 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f75f015bb70 0x7f75f0158d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:30.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.540+0000 7f75ef7fe640 1 -- 172.21.15.67:0/1288347833 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f75f015bb70 msgr2=0x7f75f0158d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:30.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.540+0000 7f75ef7fe640 1 --2- 172.21.15.67:0/1288347833 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f75f015bb70 0x7f75f0158d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.540+0000 7f75ef7fe640 1 -- 172.21.15.67:0/1288347833 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f75f00d0f80 msgr2=0x7f75f00b4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:30.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.540+0000 7f75ef7fe640 1 --2- 172.21.15.67:0/1288347833 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f75f00d0f80 0x7f75f00b4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.540+0000 7f75ef7fe640 1 -- 172.21.15.67:0/1288347833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f00b4ac0 con 0x7f75f015bf50 2024-03-24T00:47:30.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.540+0000 7f75eeffd640 1 --2- 172.21.15.67:0/1288347833 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f75f00d0f80 0x7f75f00b4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:47:30.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.540+0000 7f75ef7fe640 1 --2- 172.21.15.67:0/1288347833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75f015bf50 0x7f75f00ccdf0 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f75e800c700 tx=0x7f75e800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:30.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.540+0000 7f75d77fe640 1 -- 172.21.15.67:0/1288347833 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e8016020 con 0x7f75f015bf50 2024-03-24T00:47:30.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.540+0000 7f75d77fe640 1 -- 172.21.15.67:0/1288347833 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75e800aa50 con 0x7f75f015bf50 2024-03-24T00:47:30.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.540+0000 7f75f56ff640 1 -- 172.21.15.67:0/1288347833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75f00b4da0 con 0x7f75f015bf50 2024-03-24T00:47:30.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.540+0000 7f75d77fe640 1 -- 172.21.15.67:0/1288347833 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e800ce10 con 0x7f75f015bf50 2024-03-24T00:47:30.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.541+0000 7f75f56ff640 1 -- 172.21.15.67:0/1288347833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75f00b5360 con 0x7f75f015bf50 2024-03-24T00:47:30.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.541+0000 7f75f56ff640 1 -- 172.21.15.67:0/1288347833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f75f004f400 con 0x7f75f015bf50 2024-03-24T00:47:30.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.542+0000 7f75d77fe640 1 -- 172.21.15.67:0/1288347833 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f75e8019020 con 0x7f75f015bf50 2024-03-24T00:47:30.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.542+0000 7f75d77fe640 1 --2- 172.21.15.67:0/1288347833 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f75dc06bbd0 0x7f75dc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:30.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.542+0000 7f75d77fe640 1 -- 172.21.15.67:0/1288347833 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f75e808b550 con 0x7f75f015bf50 2024-03-24T00:47:30.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.542+0000 7f75eeffd640 1 --2- 172.21.15.67:0/1288347833 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f75dc06bbd0 0x7f75dc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:30.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.542+0000 7f75d77fe640 1 -- 172.21.15.67:0/1288347833 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+511 (secure 0 0 0) 0x7f75e8020020 con 0x7f75f015bf50 2024-03-24T00:47:30.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.543+0000 7f75f56ff640 1 -- 172.21.15.67:0/1288347833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({log-info=0+}) v3 -- 0x7f75f00ce800 con 0x7f75f015bf50 2024-03-24T00:47:30.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.543+0000 7f75eeffd640 1 --2- 172.21.15.67:0/1288347833 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f75dc06bbd0 0x7f75dc06e090 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f75e0036000 tx=0x7f75e003c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:30.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.552+0000 7fdc07e08740 1 Processor -- start 2024-03-24T00:47:30.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.552+0000 7fdc07e08740 1 -- start start 2024-03-24T00:47:30.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.553+0000 7fdc07e08740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e346be0e60 0x55e346c223a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:30.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.553+0000 7fdc07e08740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e346be1240 0x55e346c0ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:30.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.553+0000 7fdc07e08740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e346c04c10 0x55e346c3ff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:30.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.553+0000 7fdc07e08740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc00200a30 con 0x55e346c04c10 2024-03-24T00:47:30.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.553+0000 7fdc07e08740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc001ffc90 con 0x55e346be0e60 2024-03-24T00:47:30.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.553+0000 7fdc07e08740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc00156110 con 0x55e346be1240 2024-03-24T00:47:30.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.553+0000 7fdbff59e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e346c04c10 0x55e346c3ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:30.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.553+0000 7fdbff59e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e346c04c10 0x55e346c3ff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60552/0 (socket says 172.21.15.67:60552) 2024-03-24T00:47:30.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.553+0000 7fdbff59e640 1 -- 172.21.15.67:0/3586241581 learned_addr learned my addr 172.21.15.67:0/3586241581 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:30.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.553+0000 7fdbff59e640 1 -- 172.21.15.67:0/3586241581 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e346be1240 msgr2=0x55e346c0ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:30.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.553+0000 7fdbf7fff640 1 --2- 172.21.15.67:0/3586241581 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e346be1240 0x55e346c0ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:30.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.553+0000 7fdbfed9d640 1 --2- 172.21.15.67:0/3586241581 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e346be0e60 0x55e346c223a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:30.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.553+0000 7fdbff59e640 1 --2- 172.21.15.67:0/3586241581 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e346be1240 0x55e346c0ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.553+0000 7fdbff59e640 1 -- 172.21.15.67:0/3586241581 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e346be0e60 msgr2=0x55e346c223a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:30.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.553+0000 7fdbff59e640 1 --2- 172.21.15.67:0/3586241581 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e346be0e60 0x55e346c223a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.553+0000 7fdbff59e640 1 -- 172.21.15.67:0/3586241581 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e346819170 con 0x55e346c04c10 2024-03-24T00:47:30.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.554+0000 7fdbff59e640 1 --2- 172.21.15.67:0/3586241581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e346c04c10 0x55e346c3ff10 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7fdbf80ab9e0 tx=0x7fdbf8009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=354ffc33d72e3d04 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:30.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.554+0000 7fdbdffff640 1 -- 172.21.15.67:0/3586241581 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf807fa50 con 0x55e346c04c10 2024-03-24T00:47:30.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.554+0000 7fdbdffff640 1 -- 172.21.15.67:0/3586241581 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbf8084010 con 0x55e346c04c10 2024-03-24T00:47:30.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.554+0000 7fdbdffff640 1 -- 172.21.15.67:0/3586241581 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf8077f70 con 0x55e346c04c10 2024-03-24T00:47:30.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.554+0000 7fdc07e08740 1 -- 172.21.15.67:0/3586241581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e346c04c10 msgr2=0x55e346c3ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:30.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.554+0000 7fdc07e08740 1 --2- 172.21.15.67:0/3586241581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e346c04c10 0x55e346c3ff10 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7fdbf80ab9e0 tx=0x7fdbf8009040 comp rx=0 tx=0).stop 2024-03-24T00:47:30.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.554+0000 7fdc07e08740 1 -- 172.21.15.67:0/3586241581 shutdown_connections 2024-03-24T00:47:30.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.554+0000 7fdc07e08740 1 --2- 172.21.15.67:0/3586241581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e346c04c10 0x55e346c3ff10 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.554+0000 7fdc07e08740 1 --2- 172.21.15.67:0/3586241581 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e346be1240 0x55e346c0ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.554+0000 7fdc07e08740 1 --2- 172.21.15.67:0/3586241581 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e346be0e60 0x55e346c223a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.554+0000 7fdc07e08740 1 -- 172.21.15.67:0/3586241581 >> 172.21.15.67:0/3586241581 conn(0x55e346ba81a0 msgr2=0x55e346bff300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:30.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.554+0000 7fdc07e08740 1 -- 172.21.15.67:0/3586241581 shutdown_connections 2024-03-24T00:47:30.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.554+0000 7fdc07e08740 1 -- 172.21.15.67:0/3586241581 wait complete. 2024-03-24T00:47:30.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.555+0000 7fdc07e08740 1 -- start start 2024-03-24T00:47:30.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.555+0000 7fdc07e08740 1 Processor -- start 2024-03-24T00:47:30.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.556+0000 7fdc07e08740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e346be0e60 0x55e346c1b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:30.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.556+0000 7fdc07e08740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e346be1240 0x55e346c1ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:30.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.556+0000 7fdc07e08740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e346c04c10 0x55e346c98790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:30.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.556+0000 7fdc07e08740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc000b42c0 con 0x55e346c04c10 2024-03-24T00:47:30.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.556+0000 7fdc07e08740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e346b12280 con 0x55e346be0e60 2024-03-24T00:47:30.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.556+0000 7fdc07e08740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e346c051c0 con 0x55e346be1240 2024-03-24T00:47:30.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.556+0000 7fdc07e08740 10 client.0 _ll_register_callbacks cb 0x7fdbf000a660 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:47:30.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.556+0000 7fdbfed9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e346be0e60 0x55e346c1b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:30.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.556+0000 7fdbff59e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e346c04c10 0x55e346c98790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:30.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.556+0000 7fdbf7fff640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e346be1240 0x55e346c1ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:30.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.556+0000 7fdbfed9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e346be0e60 0x55e346c1b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:53262/0 (socket says 172.21.15.67:53262) 2024-03-24T00:47:30.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.557+0000 7fdbff59e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e346c04c10 0x55e346c98790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60554/0 (socket says 172.21.15.67:60554) 2024-03-24T00:47:30.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.557+0000 7fdbfed9d640 1 -- 172.21.15.67:0/2415720935 learned_addr learned my addr 172.21.15.67:0/2415720935 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:30.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.557+0000 7fdbfed9d640 1 -- 172.21.15.67:0/2415720935 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e346be1240 msgr2=0x55e346c1ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:30.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.557+0000 7fdbfed9d640 1 --2- 172.21.15.67:0/2415720935 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e346be1240 0x55e346c1ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.557+0000 7fdbfed9d640 1 -- 172.21.15.67:0/2415720935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e346c04c10 msgr2=0x55e346c98790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:30.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.557+0000 7fdbfed9d640 1 --2- 172.21.15.67:0/2415720935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e346c04c10 0x55e346c98790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.557+0000 7fdbfed9d640 1 -- 172.21.15.67:0/2415720935 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e346c21cc0 con 0x55e346be0e60 2024-03-24T00:47:30.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.557+0000 7fdbff59e640 1 --2- 172.21.15.67:0/2415720935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e346c04c10 0x55e346c98790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:47:30.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.557+0000 7fdbf7fff640 1 --2- 172.21.15.67:0/2415720935 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e346be1240 0x55e346c1ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:47:30.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.557+0000 7fdbfed9d640 1 --2- 172.21.15.67:0/2415720935 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e346be0e60 0x55e346c1b510 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fdbf0014fd0 tx=0x7fdbf0014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d3c881bed2818ec server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:30.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.557+0000 7fdbddffb640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-24T00:47:30.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.557+0000 7fdbddffb640 1 -- 172.21.15.67:0/2415720935 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf00062f0 con 0x55e346be0e60 2024-03-24T00:47:30.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.557+0000 7fdc07e08740 1 -- 172.21.15.67:0/2415720935 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdbf8084010 con 0x55e346be0e60 2024-03-24T00:47:30.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.557+0000 7fdbddffb640 1 -- 172.21.15.67:0/2415720935 <== mon.1 v2:172.21.15.72:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fdbf000f630 con 0x55e346be0e60 2024-03-24T00:47:30.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.558+0000 7fdbddffb640 1 -- 172.21.15.67:0/2415720935 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdbf0015e20 con 0x55e346be0e60 2024-03-24T00:47:30.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.558+0000 7fdc07e08740 10 client.7954 fetch_fsmap learned FSMap version 72 2024-03-24T00:47:30.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.558+0000 7fdc07e08740 1 -- 172.21.15.67:0/2415720935 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=72,monmap=2+}) v3 -- 0x55e346c07360 con 0x55e346be0e60 2024-03-24T00:47:30.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.558+0000 7fdbddffb640 1 -- 172.21.15.67:0/2415720935 <== mon.1 v2:172.21.15.72:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbf8084010 con 0x55e346be0e60 2024-03-24T00:47:30.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.558+0000 7fdbddffb640 1 -- 172.21.15.67:0/2415720935 <== mon.1 v2:172.21.15.72:3300/0 5 ==== fsmap(e 72) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fdbf0044080 con 0x55e346be0e60 2024-03-24T00:47:30.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.558+0000 7fdc07e08740 10 client.7954 fetch_fsmap finished waiting for FSMap version 72 2024-03-24T00:47:30.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.558+0000 7fdc07e08740 10 client.7954 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.c 2024-03-24T00:47:30.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.558+0000 7fdc07e08740 20 client.7954 populate_metadata read hostname 'smithi067' 2024-03-24T00:47:30.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.558+0000 7fdc07e08740 1 --2- 172.21.15.67:0/2415720935 >> [v2:172.21.15.67:6832/1826974843,v1:172.21.15.67:6833/1826974843] conn(0x55e346c92900 0x55e346c98cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:30.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.558+0000 7fdc07e08740 4 client.7954 mds_command: new command op to 8541 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive"]}] 2024-03-24T00:47:30.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.558+0000 7fdbf7fff640 1 --2- 172.21.15.67:0/2415720935 >> [v2:172.21.15.67:6832/1826974843,v1:172.21.15.67:6833/1826974843] conn(0x55e346c92900 0x55e346c98cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:30.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.558+0000 7fdc07e08740 1 -- 172.21.15.67:0/2415720935 --> [v2:172.21.15.67:6832/1826974843,v1:172.21.15.67:6833/1826974843] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive"]}) v1 -- 0x55e346b4c900 con 0x55e346c92900 2024-03-24T00:47:30.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.558+0000 7fdbf7fff640 1 --2- 172.21.15.67:0/2415720935 >> [v2:172.21.15.67:6832/1826974843,v1:172.21.15.67:6833/1826974843] conn(0x55e346c92900 0x55e346c98cd0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10fa4d907e0bb056 server_cookie=394b1a83fe47f439 in_seq=0 out_seq=0 2024-03-24T00:47:30.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.558+0000 7fdbddffb640 10 client.7954 ms_handle_connect on v2:172.21.15.67:6832/1826974843 2024-03-24T00:47:30.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.559+0000 7fdbddffb640 1 -- 172.21.15.67:0/2415720935 <== mds.0 v2:172.21.15.67:6832/1826974843 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55e346b4c900 con 0x55e346c92900 2024-03-24T00:47:30.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.559+0000 7fdbddffb640 10 client.7954 handle_command_reply: tid=0 2024-03-24T00:47:30.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.559+0000 7fdc07e08740 1 client.7954 shutdown 2024-03-24T00:47:30.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.559+0000 7fdc07e08740 1 -- 172.21.15.67:0/2415720935 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e346be0e60 msgr2=0x55e346c1b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:30.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.559+0000 7fdc07e08740 1 --2- 172.21.15.67:0/2415720935 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e346be0e60 0x55e346c1b510 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fdbf0014fd0 tx=0x7fdbf0014000 comp rx=0 tx=0).stop 2024-03-24T00:47:30.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.559+0000 7fdc07e08740 1 -- 172.21.15.67:0/2415720935 shutdown_connections 2024-03-24T00:47:30.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.559+0000 7fdc07e08740 1 --2- 172.21.15.67:0/2415720935 >> [v2:172.21.15.67:6832/1826974843,v1:172.21.15.67:6833/1826974843] conn(0x55e346c92900 0x55e346c98cd0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.559+0000 7fdc07e08740 1 --2- 172.21.15.67:0/2415720935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e346c04c10 0x55e346c98790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.559+0000 7fdc07e08740 1 --2- 172.21.15.67:0/2415720935 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e346be1240 0x55e346c1ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.559+0000 7fdbddffb640 0 client.7954 ms_handle_reset on v2:172.21.15.67:6832/1826974843 2024-03-24T00:47:30.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.559+0000 7fdc07e08740 1 --2- 172.21.15.67:0/2415720935 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e346be0e60 0x55e346c1b510 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.559+0000 7fdc07e08740 1 -- 172.21.15.67:0/2415720935 >> 172.21.15.67:0/2415720935 conn(0x55e346ba81a0 msgr2=0x55e346c35040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:30.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.560+0000 7fdc07e08740 1 -- 172.21.15.67:0/2415720935 shutdown_connections 2024-03-24T00:47:30.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.560+0000 7fdc07e08740 1 -- 172.21.15.67:0/2415720935 wait complete. 2024-03-24T00:47:30.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.560+0000 7fdc07e08740 20 client.7954 trim_cache size 0 max 16384 2024-03-24T00:47:30.566 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:47:30.566 INFO:teuthology.orchestra.run.smithi067.stdout: "return_code": 0, 2024-03-24T00:47:30.566 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_tag": "c2714944-9dfb-4bbd-b289-824099840a5e", 2024-03-24T00:47:30.566 INFO:teuthology.orchestra.run.smithi067.stdout: "mode": "asynchronous" 2024-03-24T00:47:30.566 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:47:30.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.560+0000 7fdc0513b640 1 -- 172.21.15.67:0/1580313941 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdbe006bbd0 msgr2=0x7fdbe006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:30.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.560+0000 7fdc0513b640 1 --2- 172.21.15.67:0/1580313941 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdbe006bbd0 0x7fdbe006e090 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fdbf000a990 tx=0x7fdbf0008040 comp rx=0 tx=0).stop 2024-03-24T00:47:30.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.560+0000 7fdc0513b640 1 -- 172.21.15.67:0/1580313941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc000b6440 msgr2=0x7fdc00160270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:30.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.560+0000 7fdc0513b640 1 --2- 172.21.15.67:0/1580313941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc000b6440 0x7fdc00160270 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8009f40 tx=0x7fdbf8004a40 comp rx=0 tx=0).stop 2024-03-24T00:47:30.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.560+0000 7fdc0513b640 1 -- 172.21.15.67:0/1580313941 shutdown_connections 2024-03-24T00:47:30.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.560+0000 7fdc0513b640 1 --2- 172.21.15.67:0/1580313941 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdbe006bbd0 0x7fdbe006e090 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.560+0000 7fdc0513b640 1 --2- 172.21.15.67:0/1580313941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc000b6440 0x7fdc00160270 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.560+0000 7fdc0513b640 1 --2- 172.21.15.67:0/1580313941 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc0015dfc0 0x7fdc0015fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.560+0000 7fdc0513b640 1 --2- 172.21.15.67:0/1580313941 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdc000d0d50 0x7fdc000b4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.560+0000 7fdc0513b640 1 -- 172.21.15.67:0/1580313941 >> 172.21.15.67:0/1580313941 conn(0x7fdc0006c880 msgr2=0x7fdc000b8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:30.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.561+0000 7fdc0513b640 1 -- 172.21.15.67:0/1580313941 shutdown_connections 2024-03-24T00:47:30.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.561+0000 7fdc0513b640 1 -- 172.21.15.67:0/1580313941 wait complete. 2024-03-24T00:47:30.576 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:47:30.576 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.7:0 scrub status 2024-03-24T00:47:30.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.678+0000 7fd927bc6640 1 Processor -- start 2024-03-24T00:47:30.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.678+0000 7fd927bc6640 1 -- start start 2024-03-24T00:47:30.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.679+0000 7fd927bc6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd9200b2110 0x7fd9200b2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:30.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.679+0000 7fd927bc6640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd92015c010 0x7fd9200b32d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:30.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.679+0000 7fd927bc6640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9200b3810 con 0x7fd92015bc30 2024-03-24T00:47:30.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.679+0000 7fd927bc6640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9200cca60 con 0x7fd92015c010 2024-03-24T00:47:30.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.679+0000 7fd927bc6640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9200b3ae0 con 0x7fd9200b2110 2024-03-24T00:47:30.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.679+0000 7fd926163640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd92015c010 0x7fd9200b32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:30.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.679+0000 7fd925161640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd9200b2110 0x7fd9200b2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:30.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.679+0000 7fd926163640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd92015c010 0x7fd9200b32d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:53272/0 (socket says 172.21.15.67:53272) 2024-03-24T00:47:30.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.679+0000 7fd925161640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd9200b2110 0x7fd9200b2d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.67:46538/0 (socket says 172.21.15.67:46538) 2024-03-24T00:47:30.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.679+0000 7fd926163640 1 -- 172.21.15.67:0/3876747132 learned_addr learned my addr 172.21.15.67:0/3876747132 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:30.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.679+0000 7fd925962640 1 --1- 172.21.15.67:0/3876747132 >> v1:172.21.15.72:6789/0 conn(0x7fd92015bc30 0x7fd9200b1a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:47072/0 (socket says 172.21.15.67:47072) 2024-03-24T00:47:30.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.680+0000 7fd926163640 1 -- 172.21.15.67:0/3876747132 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd9200b2110 msgr2=0x7fd9200b2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:30.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.680+0000 7fd926163640 1 --2- 172.21.15.67:0/3876747132 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd9200b2110 0x7fd9200b2d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.680+0000 7fd926163640 1 -- 172.21.15.67:0/3876747132 >> v1:172.21.15.72:6789/0 conn(0x7fd92015bc30 legacy=0x7fd9200b1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:30.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.680+0000 7fd926163640 1 -- 172.21.15.67:0/3876747132 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9200b3dc0 con 0x7fd92015c010 2024-03-24T00:47:30.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.680+0000 7fd924960640 1 -- 172.21.15.67:0/3876747132 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3362822664 0 0) 0x7fd9200b3810 con 0x7fd92015bc30 2024-03-24T00:47:30.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.680+0000 7fd925161640 1 --2- 172.21.15.67:0/3876747132 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd9200b2110 0x7fd9200b2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:47:30.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.680+0000 7fd926163640 1 --2- 172.21.15.67:0/3876747132 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd92015c010 0x7fd9200b32d0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fd918009e50 tx=0x7fd91802bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5bffbe25f63bcdf3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:30.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.680+0000 7fd924960640 1 -- 172.21.15.67:0/3876747132 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd91802cb00 con 0x7fd92015c010 2024-03-24T00:47:30.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.681+0000 7fd924960640 1 -- 172.21.15.67:0/3876747132 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd91802cca0 con 0x7fd92015c010 2024-03-24T00:47:30.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.681+0000 7fd924960640 1 -- 172.21.15.67:0/3876747132 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd918034610 con 0x7fd92015c010 2024-03-24T00:47:30.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.681+0000 7fd927bc6640 1 -- 172.21.15.67:0/3876747132 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd92015c010 msgr2=0x7fd9200b32d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:30.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.681+0000 7fd927bc6640 1 --2- 172.21.15.67:0/3876747132 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd92015c010 0x7fd9200b32d0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fd918009e50 tx=0x7fd91802bf60 comp rx=0 tx=0).stop 2024-03-24T00:47:30.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.681+0000 7fd927bc6640 1 -- 172.21.15.67:0/3876747132 shutdown_connections 2024-03-24T00:47:30.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.681+0000 7fd927bc6640 1 --2- 172.21.15.67:0/3876747132 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd92015c010 0x7fd9200b32d0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.681+0000 7fd927bc6640 1 --2- 172.21.15.67:0/3876747132 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd9200b2110 0x7fd9200b2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.681+0000 7fd927bc6640 1 -- 172.21.15.67:0/3876747132 >> 172.21.15.67:0/3876747132 conn(0x7fd92008b180 msgr2=0x7fd92008b590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:30.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.681+0000 7fd927bc6640 1 -- 172.21.15.67:0/3876747132 shutdown_connections 2024-03-24T00:47:30.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.681+0000 7fd927bc6640 1 -- 172.21.15.67:0/3876747132 wait complete. 2024-03-24T00:47:30.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.682+0000 7fd927bc6640 1 Processor -- start 2024-03-24T00:47:30.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.682+0000 7fd927bc6640 1 -- start start 2024-03-24T00:47:30.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.683+0000 7fd927bc6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd9200b2110 0x7fd9200b0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:30.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.683+0000 7fd927bc6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd92015bc30 0x7fd9200b10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:30.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.683+0000 7fd927bc6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd92015c010 0x7fd9200c6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:30.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.683+0000 7fd927bc6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9200ccef0 con 0x7fd92015c010 2024-03-24T00:47:30.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.683+0000 7fd927bc6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9200b4380 con 0x7fd92015bc30 2024-03-24T00:47:30.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.683+0000 7fd927bc6640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9200cd070 con 0x7fd9200b2110 2024-03-24T00:47:30.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.683+0000 7fd926163640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd92015c010 0x7fd9200c6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:30.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.683+0000 7fd926163640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd92015c010 0x7fd9200c6ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60566/0 (socket says 172.21.15.67:60566) 2024-03-24T00:47:30.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.683+0000 7fd926163640 1 -- 172.21.15.67:0/3867082294 learned_addr learned my addr 172.21.15.67:0/3867082294 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:30.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.683+0000 7fd925962640 1 --2- 172.21.15.67:0/3867082294 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd9200b2110 0x7fd9200b0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:30.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.683+0000 7fd925161640 1 --2- 172.21.15.67:0/3867082294 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd92015bc30 0x7fd9200b10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:30.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.684+0000 7fd926163640 1 -- 172.21.15.67:0/3867082294 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd9200b2110 msgr2=0x7fd9200b0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:30.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.684+0000 7fd926163640 1 --2- 172.21.15.67:0/3867082294 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd9200b2110 0x7fd9200b0b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.684+0000 7fd926163640 1 -- 172.21.15.67:0/3867082294 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd92015bc30 msgr2=0x7fd9200b10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:30.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.684+0000 7fd926163640 1 --2- 172.21.15.67:0/3867082294 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd92015bc30 0x7fd9200b10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.684+0000 7fd926163640 1 -- 172.21.15.67:0/3867082294 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9200c7ef0 con 0x7fd92015c010 2024-03-24T00:47:30.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.684+0000 7fd926163640 1 --2- 172.21.15.67:0/3867082294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd92015c010 0x7fd9200c6ee0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7fd918036040 tx=0x7fd918004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:30.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.684+0000 7fd9167fc640 1 -- 172.21.15.67:0/3867082294 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd918043030 con 0x7fd92015c010 2024-03-24T00:47:30.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.684+0000 7fd9167fc640 1 -- 172.21.15.67:0/3867082294 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd91802ce50 con 0x7fd92015c010 2024-03-24T00:47:30.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.684+0000 7fd927bc6640 1 -- 172.21.15.67:0/3867082294 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9200c81d0 con 0x7fd92015c010 2024-03-24T00:47:30.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.684+0000 7fd9167fc640 1 -- 172.21.15.67:0/3867082294 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd918009510 con 0x7fd92015c010 2024-03-24T00:47:30.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.684+0000 7fd927bc6640 1 -- 172.21.15.67:0/3867082294 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9200c84b0 con 0x7fd92015c010 2024-03-24T00:47:30.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.686+0000 7fd9167fc640 1 -- 172.21.15.67:0/3867082294 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd918046070 con 0x7fd92015c010 2024-03-24T00:47:30.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.686+0000 7fd9167fc640 1 --2- 172.21.15.67:0/3867082294 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd90406bbd0 0x7fd90406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:30.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.686+0000 7fd9167fc640 1 -- 172.21.15.67:0/3867082294 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd9180accc0 con 0x7fd92015c010 2024-03-24T00:47:30.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.686+0000 7fd925962640 1 --2- 172.21.15.67:0/3867082294 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd90406bbd0 0x7fd90406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:30.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.687+0000 7fd925962640 1 --2- 172.21.15.67:0/3867082294 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd90406bbd0 0x7fd90406e090 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fd91c000f60 tx=0x7fd91c007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:30.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.687+0000 7fd92a7b1740 1 Processor -- start 2024-03-24T00:47:30.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.687+0000 7fd92a7b1740 1 -- start start 2024-03-24T00:47:30.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.688+0000 7fd92a7b1740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x564cc87858d0 0x564cc8783590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:30.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.688+0000 7fd92a7b1740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564cc8785cb0 0x564cc873cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:30.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.688+0000 7fd92a7b1740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc8765210 0x564cc873d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:30.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.688+0000 7fd92a7b1740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x564cc86a23d0 con 0x564cc8765210 2024-03-24T00:47:30.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.688+0000 7fd92a7b1740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x564cc864aad0 con 0x564cc8785cb0 2024-03-24T00:47:30.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.688+0000 7fd92a7b1740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x564cc869e470 con 0x564cc87858d0 2024-03-24T00:47:30.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.688+0000 7fd926163640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc8765210 0x564cc873d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:30.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.688+0000 7fd926163640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc8765210 0x564cc873d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60578/0 (socket says 172.21.15.67:60578) 2024-03-24T00:47:30.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.688+0000 7fd926163640 1 -- 172.21.15.67:0/3633417150 learned_addr learned my addr 172.21.15.67:0/3633417150 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:30.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.688+0000 7fd925962640 1 --2- 172.21.15.67:0/3633417150 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x564cc87858d0 0x564cc8783590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:30.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.688+0000 7fd925161640 1 --2- 172.21.15.67:0/3633417150 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564cc8785cb0 0x564cc873cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:30.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.688+0000 7fd926163640 1 -- 172.21.15.67:0/3633417150 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x564cc87858d0 msgr2=0x564cc8783590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:30.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.688+0000 7fd926163640 1 --2- 172.21.15.67:0/3633417150 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x564cc87858d0 0x564cc8783590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.688+0000 7fd926163640 1 -- 172.21.15.67:0/3633417150 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564cc8785cb0 msgr2=0x564cc873cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:30.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.688+0000 7fd926163640 1 --2- 172.21.15.67:0/3633417150 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564cc8785cb0 0x564cc873cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.688+0000 7fd926163640 1 -- 172.21.15.67:0/3633417150 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564cc84807d0 con 0x564cc8765210 2024-03-24T00:47:30.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.688+0000 7fd926163640 1 --2- 172.21.15.67:0/3633417150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc8765210 0x564cc873d440 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7fd9180aafd0 tx=0x7fd91807ab50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db78cc264166283e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:30.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.688+0000 7fd9037fe640 1 -- 172.21.15.67:0/3633417150 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9180835b0 con 0x564cc8765210 2024-03-24T00:47:30.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.688+0000 7fd9037fe640 1 -- 172.21.15.67:0/3633417150 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd918078ff0 con 0x564cc8765210 2024-03-24T00:47:30.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.688+0000 7fd9037fe640 1 -- 172.21.15.67:0/3633417150 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9180775c0 con 0x564cc8765210 2024-03-24T00:47:30.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.689+0000 7fd92a7b1740 1 -- 172.21.15.67:0/3633417150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc8765210 msgr2=0x564cc873d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:30.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.689+0000 7fd92a7b1740 1 --2- 172.21.15.67:0/3633417150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc8765210 0x564cc873d440 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7fd9180aafd0 tx=0x7fd91807ab50 comp rx=0 tx=0).stop 2024-03-24T00:47:30.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.689+0000 7fd92a7b1740 1 -- 172.21.15.67:0/3633417150 shutdown_connections 2024-03-24T00:47:30.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.689+0000 7fd92a7b1740 1 --2- 172.21.15.67:0/3633417150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc8765210 0x564cc873d440 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.689+0000 7fd92a7b1740 1 --2- 172.21.15.67:0/3633417150 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564cc8785cb0 0x564cc873cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.689+0000 7fd92a7b1740 1 --2- 172.21.15.67:0/3633417150 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x564cc87858d0 0x564cc8783590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.689+0000 7fd92a7b1740 1 -- 172.21.15.67:0/3633417150 >> 172.21.15.67:0/3633417150 conn(0x564cc873a060 msgr2=0x564cc8782be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:30.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.689+0000 7fd92a7b1740 1 -- 172.21.15.67:0/3633417150 shutdown_connections 2024-03-24T00:47:30.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.689+0000 7fd92a7b1740 1 -- 172.21.15.67:0/3633417150 wait complete. 2024-03-24T00:47:30.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.690+0000 7fd92a7b1740 1 -- start start 2024-03-24T00:47:30.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.690+0000 7fd92a7b1740 1 Processor -- start 2024-03-24T00:47:30.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.691+0000 7fd92a7b1740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564cc87858d0 0x564cc875e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:30.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.691+0000 7fd92a7b1740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc8785cb0 0x564cc879a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:30.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.691+0000 7fd92a7b1740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x564cc87821f0 0x564cc879a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:30.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.691+0000 7fd92a7b1740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x564cc864aad0 con 0x564cc8785cb0 2024-03-24T00:47:30.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.691+0000 7fd92a7b1740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x564cc865e060 con 0x564cc87858d0 2024-03-24T00:47:30.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.691+0000 7fd92a7b1740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x564cc86a23d0 con 0x564cc87821f0 2024-03-24T00:47:30.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.691+0000 7fd92a7b1740 10 client.0 _ll_register_callbacks cb 0x564cc86406a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:47:30.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.691+0000 7fd925161640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc8785cb0 0x564cc879a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:30.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.691+0000 7fd925161640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc8785cb0 0x564cc879a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60586/0 (socket says 172.21.15.67:60586) 2024-03-24T00:47:30.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.691+0000 7fd925161640 1 -- 172.21.15.67:0/2587769804 learned_addr learned my addr 172.21.15.67:0/2587769804 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:30.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.691+0000 7fd925962640 1 --2- 172.21.15.67:0/2587769804 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564cc87858d0 0x564cc875e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:30.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.691+0000 7fd926163640 1 --2- 172.21.15.67:0/2587769804 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x564cc87821f0 0x564cc879a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:30.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.691+0000 7fd925161640 1 -- 172.21.15.67:0/2587769804 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x564cc87821f0 msgr2=0x564cc879a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:30.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.691+0000 7fd925161640 1 --2- 172.21.15.67:0/2587769804 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x564cc87821f0 0x564cc879a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.691+0000 7fd925161640 1 -- 172.21.15.67:0/2587769804 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564cc87858d0 msgr2=0x564cc875e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:30.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.691+0000 7fd925161640 1 --2- 172.21.15.67:0/2587769804 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564cc87858d0 0x564cc875e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.691+0000 7fd925161640 1 -- 172.21.15.67:0/2587769804 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564cc83b63f0 con 0x564cc8785cb0 2024-03-24T00:47:30.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.691+0000 7fd925161640 1 --2- 172.21.15.67:0/2587769804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc8785cb0 0x564cc879a390 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7fd9100147a0 tx=0x7fd910014c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e22afdbfe04a11fb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:30.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.691+0000 7fd900ff9640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:47:30.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.691+0000 7fd900ff9640 1 -- 172.21.15.67:0/2587769804 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd910022060 con 0x564cc8785cb0 2024-03-24T00:47:30.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.691+0000 7fd900ff9640 1 -- 172.21.15.67:0/2587769804 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd910012ac0 con 0x564cc8785cb0 2024-03-24T00:47:30.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.691+0000 7fd92a7b1740 1 -- 172.21.15.67:0/2587769804 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd918078ff0 con 0x564cc8785cb0 2024-03-24T00:47:30.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.692+0000 7fd900ff9640 1 -- 172.21.15.67:0/2587769804 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd910028050 con 0x564cc8785cb0 2024-03-24T00:47:30.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.692+0000 7fd92a7b1740 10 client.8856 fetch_fsmap learned FSMap version 72 2024-03-24T00:47:30.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.692+0000 7fd92a7b1740 1 -- 172.21.15.67:0/2587769804 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=72,monmap=2+}) v3 -- 0x564cc83760f0 con 0x564cc8785cb0 2024-03-24T00:47:30.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.692+0000 7fd900ff9640 1 -- 172.21.15.67:0/2587769804 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd918078ff0 con 0x564cc8785cb0 2024-03-24T00:47:30.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.692+0000 7fd900ff9640 1 -- 172.21.15.67:0/2587769804 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 72) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fd91002b020 con 0x564cc8785cb0 2024-03-24T00:47:30.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.692+0000 7fd92a7b1740 10 client.8856 fetch_fsmap finished waiting for FSMap version 72 2024-03-24T00:47:30.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.692+0000 7fd92a7b1740 10 client.8856 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.c 2024-03-24T00:47:30.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.692+0000 7fd92a7b1740 20 client.8856 populate_metadata read hostname 'smithi067' 2024-03-24T00:47:30.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.692+0000 7fd92a7b1740 1 --2- 172.21.15.67:0/2587769804 >> [v2:172.21.15.67:6832/1826974843,v1:172.21.15.67:6833/1826974843] conn(0x564cc8763a00 0x564cc8763de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:30.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.692+0000 7fd92a7b1740 4 client.8856 mds_command: new command op to 8541 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:47:30.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.692+0000 7fd92a7b1740 1 -- 172.21.15.67:0/2587769804 --> [v2:172.21.15.67:6832/1826974843,v1:172.21.15.67:6833/1826974843] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564cc86af910 con 0x564cc8763a00 2024-03-24T00:47:30.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.692+0000 7fd925962640 1 --2- 172.21.15.67:0/2587769804 >> [v2:172.21.15.67:6832/1826974843,v1:172.21.15.67:6833/1826974843] conn(0x564cc8763a00 0x564cc8763de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:30.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.692+0000 7fd900ff9640 10 client.8856 ms_handle_connect on v2:172.21.15.67:6832/1826974843 2024-03-24T00:47:30.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.692+0000 7fd925962640 1 --2- 172.21.15.67:0/2587769804 >> [v2:172.21.15.67:6832/1826974843,v1:172.21.15.67:6833/1826974843] conn(0x564cc8763a00 0x564cc8763de0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71b8d6d0755c86aa server_cookie=ce13b17e25882c4b in_seq=0 out_seq=0 2024-03-24T00:47:30.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.693+0000 7fd900ff9640 1 -- 172.21.15.67:0/2587769804 <== mds.0 v2:172.21.15.67:6832/1826974843 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564cc86af910 con 0x564cc8763a00 2024-03-24T00:47:30.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.693+0000 7fd900ff9640 10 client.8856 handle_command_reply: tid=0 2024-03-24T00:47:30.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.693+0000 7fd92a7b1740 1 client.8856 shutdown 2024-03-24T00:47:30.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.693+0000 7fd92a7b1740 1 -- 172.21.15.67:0/2587769804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc8785cb0 msgr2=0x564cc879a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:30.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.693+0000 7fd92a7b1740 1 --2- 172.21.15.67:0/2587769804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc8785cb0 0x564cc879a390 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7fd9100147a0 tx=0x7fd910014c60 comp rx=0 tx=0).stop 2024-03-24T00:47:30.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.693+0000 7fd92a7b1740 1 -- 172.21.15.67:0/2587769804 shutdown_connections 2024-03-24T00:47:30.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.693+0000 7fd92a7b1740 1 --2- 172.21.15.67:0/2587769804 >> [v2:172.21.15.67:6832/1826974843,v1:172.21.15.67:6833/1826974843] conn(0x564cc8763a00 0x564cc8763de0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.693+0000 7fd92a7b1740 1 --2- 172.21.15.67:0/2587769804 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x564cc87821f0 0x564cc879a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.693+0000 7fd92a7b1740 1 --2- 172.21.15.67:0/2587769804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc8785cb0 0x564cc879a390 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.693+0000 7fd900ff9640 0 client.8856 ms_handle_reset on v2:172.21.15.67:6832/1826974843 2024-03-24T00:47:30.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.693+0000 7fd92a7b1740 1 --2- 172.21.15.67:0/2587769804 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564cc87858d0 0x564cc875e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.693+0000 7fd92a7b1740 1 -- 172.21.15.67:0/2587769804 >> 172.21.15.67:0/2587769804 conn(0x564cc8765210 msgr2=0x564cc8784a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:30.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.693+0000 7fd92a7b1740 1 -- 172.21.15.67:0/2587769804 shutdown_connections 2024-03-24T00:47:30.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.693+0000 7fd92a7b1740 1 -- 172.21.15.67:0/2587769804 wait complete. 2024-03-24T00:47:30.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.693+0000 7fd92a7b1740 20 client.8856 trim_cache size 0 max 16384 2024-03-24T00:47:30.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.709+0000 7fd92a7b1740 1 Processor -- start 2024-03-24T00:47:30.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.709+0000 7fd92a7b1740 1 -- start start 2024-03-24T00:47:30.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.710+0000 7fd92a7b1740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x564cc87821f0 0x564cc873ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:30.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.710+0000 7fd92a7b1740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc87858d0 0x564cc8784fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:30.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.710+0000 7fd92a7b1740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564cc8785cb0 0x564cc879a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:30.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.710+0000 7fd92a7b1740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd92006e170 con 0x564cc87858d0 2024-03-24T00:47:30.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.710+0000 7fd92a7b1740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd92006d360 con 0x564cc8785cb0 2024-03-24T00:47:30.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.710+0000 7fd92a7b1740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9200af860 con 0x564cc87821f0 2024-03-24T00:47:30.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.710+0000 7fd925161640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc87858d0 0x564cc8784fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:30.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.710+0000 7fd925161640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc87858d0 0x564cc8784fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60592/0 (socket says 172.21.15.67:60592) 2024-03-24T00:47:30.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.710+0000 7fd925161640 1 -- 172.21.15.67:0/539409286 learned_addr learned my addr 172.21.15.67:0/539409286 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:30.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.710+0000 7fd925962640 1 --2- 172.21.15.67:0/539409286 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x564cc87821f0 0x564cc873ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:30.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.710+0000 7fd926163640 1 --2- 172.21.15.67:0/539409286 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564cc8785cb0 0x564cc879a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:30.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.710+0000 7fd925161640 1 -- 172.21.15.67:0/539409286 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x564cc87821f0 msgr2=0x564cc873ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:30.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.710+0000 7fd925161640 1 --2- 172.21.15.67:0/539409286 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x564cc87821f0 0x564cc873ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.710+0000 7fd925161640 1 -- 172.21.15.67:0/539409286 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564cc8785cb0 msgr2=0x564cc879a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:30.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.710+0000 7fd925161640 1 --2- 172.21.15.67:0/539409286 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564cc8785cb0 0x564cc879a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.710+0000 7fd925161640 1 -- 172.21.15.67:0/539409286 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564cc878f060 con 0x564cc87858d0 2024-03-24T00:47:30.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.711+0000 7fd926163640 1 --2- 172.21.15.67:0/539409286 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564cc8785cb0 0x564cc879a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:47:30.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.711+0000 7fd925161640 1 --2- 172.21.15.67:0/539409286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc87858d0 0x564cc8784fa0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7fd910009f60 tx=0x7fd910017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32564b883599243e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:30.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.711+0000 7fd902ffd640 1 -- 172.21.15.67:0/539409286 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd910012700 con 0x564cc87858d0 2024-03-24T00:47:30.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.711+0000 7fd902ffd640 1 -- 172.21.15.67:0/539409286 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd910012d10 con 0x564cc87858d0 2024-03-24T00:47:30.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.711+0000 7fd902ffd640 1 -- 172.21.15.67:0/539409286 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd910005bb0 con 0x564cc87858d0 2024-03-24T00:47:30.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.711+0000 7fd92a7b1740 1 -- 172.21.15.67:0/539409286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc87858d0 msgr2=0x564cc8784fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:30.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.711+0000 7fd92a7b1740 1 --2- 172.21.15.67:0/539409286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc87858d0 0x564cc8784fa0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7fd910009f60 tx=0x7fd910017000 comp rx=0 tx=0).stop 2024-03-24T00:47:30.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.711+0000 7fd92a7b1740 1 -- 172.21.15.67:0/539409286 shutdown_connections 2024-03-24T00:47:30.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.711+0000 7fd92a7b1740 1 --2- 172.21.15.67:0/539409286 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564cc8785cb0 0x564cc879a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.711+0000 7fd92a7b1740 1 --2- 172.21.15.67:0/539409286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc87858d0 0x564cc8784fa0 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.711+0000 7fd92a7b1740 1 --2- 172.21.15.67:0/539409286 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x564cc87821f0 0x564cc873ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.711+0000 7fd92a7b1740 1 -- 172.21.15.67:0/539409286 >> 172.21.15.67:0/539409286 conn(0x564cc8765210 msgr2=0x564cc87656f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:30.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.711+0000 7fd92a7b1740 1 -- 172.21.15.67:0/539409286 shutdown_connections 2024-03-24T00:47:30.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.711+0000 7fd92a7b1740 1 -- 172.21.15.67:0/539409286 wait complete. 2024-03-24T00:47:30.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.712+0000 7fd92a7b1740 1 -- start start 2024-03-24T00:47:30.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.712+0000 7fd92a7b1740 1 Processor -- start 2024-03-24T00:47:30.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.713+0000 7fd92a7b1740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564cc87821f0 0x564cc875d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:30.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.713+0000 7fd92a7b1740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x564cc87858d0 0x564cc87809a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:30.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.713+0000 7fd92a7b1740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc8785cb0 0x564cc8780ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:30.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.713+0000 7fd92a7b1740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x564cc867da20 con 0x564cc8785cb0 2024-03-24T00:47:30.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.713+0000 7fd92a7b1740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9200afe60 con 0x564cc87821f0 2024-03-24T00:47:30.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.713+0000 7fd92a7b1740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x564cc873d1c0 con 0x564cc87858d0 2024-03-24T00:47:30.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.713+0000 7fd92a7b1740 10 client.0 _ll_register_callbacks cb 0x7fd9100097c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:47:30.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.713+0000 7fd926163640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc8785cb0 0x564cc8780ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:30.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.713+0000 7fd926163640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc8785cb0 0x564cc8780ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60598/0 (socket says 172.21.15.67:60598) 2024-03-24T00:47:30.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.713+0000 7fd926163640 1 -- 172.21.15.67:0/67354295 learned_addr learned my addr 172.21.15.67:0/67354295 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:47:30.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.713+0000 7fd925962640 1 --2- 172.21.15.67:0/67354295 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564cc87821f0 0x564cc875d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:30.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.713+0000 7fd925161640 1 --2- 172.21.15.67:0/67354295 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x564cc87858d0 0x564cc87809a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:30.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.713+0000 7fd926163640 1 -- 172.21.15.67:0/67354295 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x564cc87858d0 msgr2=0x564cc87809a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:30.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.713+0000 7fd926163640 1 --2- 172.21.15.67:0/67354295 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x564cc87858d0 0x564cc87809a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.713+0000 7fd926163640 1 -- 172.21.15.67:0/67354295 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564cc87821f0 msgr2=0x564cc875d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:47:30.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.713+0000 7fd926163640 1 --2- 172.21.15.67:0/67354295 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564cc87821f0 0x564cc875d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.713+0000 7fd926163640 1 -- 172.21.15.67:0/67354295 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564cc87f6560 con 0x564cc8785cb0 2024-03-24T00:47:30.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.713+0000 7fd925161640 1 --2- 172.21.15.67:0/67354295 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x564cc87858d0 0x564cc87809a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:47:30.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.714+0000 7fd926163640 1 --2- 172.21.15.67:0/67354295 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc8785cb0 0x564cc8780ee0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7fd9180ab0b0 tx=0x7fd918081040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7cf039030ad1511 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:47:30.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.714+0000 7fd900ff9640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:47:30.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.714+0000 7fd900ff9640 1 -- 172.21.15.67:0/67354295 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9180835b0 con 0x564cc8785cb0 2024-03-24T00:47:30.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.714+0000 7fd900ff9640 1 -- 172.21.15.67:0/67354295 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd918079250 con 0x564cc8785cb0 2024-03-24T00:47:30.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.714+0000 7fd92a7b1740 1 -- 172.21.15.67:0/67354295 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd910012d10 con 0x564cc8785cb0 2024-03-24T00:47:30.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.714+0000 7fd900ff9640 1 -- 172.21.15.67:0/67354295 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd9180775c0 con 0x564cc8785cb0 2024-03-24T00:47:30.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.714+0000 7fd92a7b1740 10 client.8862 fetch_fsmap learned FSMap version 72 2024-03-24T00:47:30.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.714+0000 7fd92a7b1740 1 -- 172.21.15.67:0/67354295 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=72,monmap=2+}) v3 -- 0x564cc878ecf0 con 0x564cc8785cb0 2024-03-24T00:47:30.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.714+0000 7fd900ff9640 1 -- 172.21.15.67:0/67354295 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd910012d10 con 0x564cc8785cb0 2024-03-24T00:47:30.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.714+0000 7fd900ff9640 1 -- 172.21.15.67:0/67354295 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 72) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fd918092070 con 0x564cc8785cb0 2024-03-24T00:47:30.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.714+0000 7fd92a7b1740 10 client.8862 fetch_fsmap finished waiting for FSMap version 72 2024-03-24T00:47:30.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.714+0000 7fd92a7b1740 10 client.8862 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.c 2024-03-24T00:47:30.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.714+0000 7fd92a7b1740 20 client.8862 populate_metadata read hostname 'smithi067' 2024-03-24T00:47:30.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.714+0000 7fd92a7b1740 1 --2- 172.21.15.67:0/67354295 >> [v2:172.21.15.67:6832/1826974843,v1:172.21.15.67:6833/1826974843] conn(0x564cc87f5760 0x564cc8767f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:30.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.714+0000 7fd92a7b1740 4 client.8862 mds_command: new command op to 8541 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:47:30.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.714+0000 7fd92a7b1740 1 -- 172.21.15.67:0/67354295 --> [v2:172.21.15.67:6832/1826974843,v1:172.21.15.67:6833/1826974843] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x564cc86a9c60 con 0x564cc87f5760 2024-03-24T00:47:30.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.714+0000 7fd925962640 1 --2- 172.21.15.67:0/67354295 >> [v2:172.21.15.67:6832/1826974843,v1:172.21.15.67:6833/1826974843] conn(0x564cc87f5760 0x564cc8767f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:47:30.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.715+0000 7fd925962640 1 --2- 172.21.15.67:0/67354295 >> [v2:172.21.15.67:6832/1826974843,v1:172.21.15.67:6833/1826974843] conn(0x564cc87f5760 0x564cc8767f60 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc5a356e3b6e1de6 server_cookie=8a50f7996a1d784b in_seq=0 out_seq=0 2024-03-24T00:47:30.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.715+0000 7fd900ff9640 10 client.8862 ms_handle_connect on v2:172.21.15.67:6832/1826974843 2024-03-24T00:47:30.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.715+0000 7fd900ff9640 1 -- 172.21.15.67:0/67354295 <== mds.0 v2:172.21.15.67:6832/1826974843 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+260 (crc 0 0 0) 0x564cc86a9c60 con 0x564cc87f5760 2024-03-24T00:47:30.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.715+0000 7fd900ff9640 10 client.8862 handle_command_reply: tid=0 2024-03-24T00:47:30.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.715+0000 7fd92a7b1740 1 client.8862 shutdown 2024-03-24T00:47:30.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.715+0000 7fd92a7b1740 1 -- 172.21.15.67:0/67354295 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc8785cb0 msgr2=0x564cc8780ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:30.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.715+0000 7fd92a7b1740 1 --2- 172.21.15.67:0/67354295 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc8785cb0 0x564cc8780ee0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7fd9180ab0b0 tx=0x7fd918081040 comp rx=0 tx=0).stop 2024-03-24T00:47:30.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.715+0000 7fd92a7b1740 1 -- 172.21.15.67:0/67354295 shutdown_connections 2024-03-24T00:47:30.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.715+0000 7fd92a7b1740 1 --2- 172.21.15.67:0/67354295 >> [v2:172.21.15.67:6832/1826974843,v1:172.21.15.67:6833/1826974843] conn(0x564cc87f5760 0x564cc8767f60 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.715+0000 7fd92a7b1740 1 --2- 172.21.15.67:0/67354295 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc8785cb0 0x564cc8780ee0 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.715+0000 7fd92a7b1740 1 --2- 172.21.15.67:0/67354295 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x564cc87858d0 0x564cc87809a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.715+0000 7fd900ff9640 0 client.8862 ms_handle_reset on v2:172.21.15.67:6832/1826974843 2024-03-24T00:47:30.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.715+0000 7fd92a7b1740 1 --2- 172.21.15.67:0/67354295 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564cc87821f0 0x564cc875d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.715+0000 7fd92a7b1740 1 -- 172.21.15.67:0/67354295 >> 172.21.15.67:0/67354295 conn(0x564cc8765210 msgr2=0x564cc8761f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:30.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.715+0000 7fd92a7b1740 1 -- 172.21.15.67:0/67354295 shutdown_connections 2024-03-24T00:47:30.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.715+0000 7fd92a7b1740 1 -- 172.21.15.67:0/67354295 wait complete. 2024-03-24T00:47:30.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.715+0000 7fd92a7b1740 20 client.8862 trim_cache size 0 max 16384 2024-03-24T00:47:30.722 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:47:30.722 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T00:47:30.722 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": { 2024-03-24T00:47:30.722 INFO:teuthology.orchestra.run.smithi067.stdout: "c2714944-9dfb-4bbd-b289-824099840a5e": { 2024-03-24T00:47:30.722 INFO:teuthology.orchestra.run.smithi067.stdout: "path": "/", 2024-03-24T00:47:30.722 INFO:teuthology.orchestra.run.smithi067.stdout: "tag": "c2714944-9dfb-4bbd-b289-824099840a5e", 2024-03-24T00:47:30.722 INFO:teuthology.orchestra.run.smithi067.stdout: "options": "recursive,repair" 2024-03-24T00:47:30.722 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:47:30.722 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:47:30.722 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:47:30.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.716+0000 7fd927bc6640 1 -- 172.21.15.67:0/3867082294 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd90406bbd0 msgr2=0x7fd90406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:30.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.716+0000 7fd927bc6640 1 --2- 172.21.15.67:0/3867082294 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd90406bbd0 0x7fd90406e090 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fd91c000f60 tx=0x7fd91c007d50 comp rx=0 tx=0).stop 2024-03-24T00:47:30.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.716+0000 7fd927bc6640 1 -- 172.21.15.67:0/3867082294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd92015c010 msgr2=0x7fd9200c6ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:47:30.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.716+0000 7fd927bc6640 1 --2- 172.21.15.67:0/3867082294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd92015c010 0x7fd9200c6ee0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7fd918036040 tx=0x7fd918004a40 comp rx=0 tx=0).stop 2024-03-24T00:47:30.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.716+0000 7fd927bc6640 1 -- 172.21.15.67:0/3867082294 shutdown_connections 2024-03-24T00:47:30.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.716+0000 7fd927bc6640 1 --2- 172.21.15.67:0/3867082294 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd90406bbd0 0x7fd90406e090 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.716+0000 7fd927bc6640 1 --2- 172.21.15.67:0/3867082294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd92015c010 0x7fd9200c6ee0 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.716+0000 7fd927bc6640 1 --2- 172.21.15.67:0/3867082294 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd92015bc30 0x7fd9200b10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.716+0000 7fd927bc6640 1 --2- 172.21.15.67:0/3867082294 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd9200b2110 0x7fd9200b0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:30.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.716+0000 7fd927bc6640 1 -- 172.21.15.67:0/3867082294 >> 172.21.15.67:0/3867082294 conn(0x7fd92008b180 msgr2=0x7fd920156de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:47:30.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.716+0000 7fd927bc6640 1 -- 172.21.15.67:0/3867082294 shutdown_connections 2024-03-24T00:47:30.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:30.716+0000 7fd927bc6640 1 -- 172.21.15.67:0/3867082294 wait complete. 2024-03-24T00:47:30.727 INFO:tasks.cephfs.filesystem:scrub status for tag:c2714944-9dfb-4bbd-b289-824099840a5e - {'path': '/', 'tag': 'c2714944-9dfb-4bbd-b289-824099840a5e', 'options': 'recursive,repair'} 2024-03-24T00:47:31.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:31.190+0000 7f75d77fe640 1 -- 172.21.15.67:0/1288347833 <== mon.0 v2:172.21.15.67:3300/0 7 ==== log(4 entries from seq 1 at 2024-03-24T00:47:30.560418+0000) v1 ==== 695+0+0 (secure 0 0 0) 0x7f75e8089a00 con 0x7f75f015bf50 2024-03-24T00:47:31.191 INFO:teuthology.orchestra.run.smithi067.stdout: cluster: 2024-03-24T00:47:31.191 INFO:teuthology.orchestra.run.smithi067.stdout: id: de1d0178-808e-49ca-a986-e297e523b976 2024-03-24T00:47:31.191 INFO:teuthology.orchestra.run.smithi067.stdout: health: HEALTH_OK 2024-03-24T00:47:31.191 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:47:31.191 INFO:teuthology.orchestra.run.smithi067.stdout: services: 2024-03-24T00:47:31.192 INFO:teuthology.orchestra.run.smithi067.stdout: mon: 3 daemons, quorum a,b,c (age 9m) 2024-03-24T00:47:31.192 INFO:teuthology.orchestra.run.smithi067.stdout: mgr: x(active, since 8m), standbys: y 2024-03-24T00:47:31.192 INFO:teuthology.orchestra.run.smithi067.stdout: mds: 1/1 daemons up, 3 standby 2024-03-24T00:47:31.192 INFO:teuthology.orchestra.run.smithi067.stdout: osd: 8 osds: 8 up (since 8m), 8 in (since 9m) 2024-03-24T00:47:31.192 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:47:31.192 INFO:teuthology.orchestra.run.smithi067.stdout: data: 2024-03-24T00:47:31.192 INFO:teuthology.orchestra.run.smithi067.stdout: volumes: 1/1 healthy 2024-03-24T00:47:31.192 INFO:teuthology.orchestra.run.smithi067.stdout: pools: 3 pools, 97 pgs 2024-03-24T00:47:31.192 INFO:teuthology.orchestra.run.smithi067.stdout: objects: 30 objects, 19 MiB 2024-03-24T00:47:31.192 INFO:teuthology.orchestra.run.smithi067.stdout: usage: 291 MiB used, 720 GiB / 720 GiB avail 2024-03-24T00:47:31.192 INFO:teuthology.orchestra.run.smithi067.stdout: pgs: 97 active+clean 2024-03-24T00:47:31.192 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:47:31.192 INFO:teuthology.orchestra.run.smithi067.stdout: io: 2024-03-24T00:47:31.192 INFO:teuthology.orchestra.run.smithi067.stdout: client: 90 B/s rd, 1.6 MiB/s wr, 0 op/s rd, 7 op/s wr 2024-03-24T00:47:31.193 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:47:31.193 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:47:31.193 INFO:teuthology.orchestra.run.smithi067.stdout:2024-03-24T00:47:30.560418+0000 mds.c [INF] scrub queued for path: / 2024-03-24T00:47:31.193 INFO:teuthology.orchestra.run.smithi067.stdout:2024-03-24T00:47:30.560423+0000 mds.c [INF] scrub summary: idle+waiting paths [/] 2024-03-24T00:47:31.193 INFO:teuthology.orchestra.run.smithi067.stdout:2024-03-24T00:47:30.560427+0000 mds.c [INF] scrub summary: active paths [/] 2024-03-24T00:47:31.193 INFO:teuthology.orchestra.run.smithi067.stdout:2024-03-24T00:47:30.561514+0000 mds.c [INF] scrub summary: idle+waiting paths [/] 2024-03-24T00:47:33.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:33.196+0000 7f75d77fe640 1 -- 172.21.15.67:0/1288347833 <== mon.0 v2:172.21.15.67:3300/0 8 ==== log(1 entries from seq 5 at 2024-03-24T00:47:32.115208+0000) v1 ==== 189+0+0 (secure 0 0 0) 0x7f75e8057850 con 0x7f75f015bf50 2024-03-24T00:47:33.197 INFO:teuthology.orchestra.run.smithi067.stdout:2024-03-24T00:47:32.115208+0000 mds.c [INF] scrub summary: idle 2024-03-24T00:47:34.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:34.197+0000 7f75d77fe640 1 -- 172.21.15.67:0/1288347833 <== mon.0 v2:172.21.15.67:3300/0 9 ==== log(2 entries from seq 161 at 2024-03-24T00:47:33.510661+0000) v1 ==== 581+0+0 (secure 0 0 0) 0x7f75e8057ab0 con 0x7f75f015bf50 2024-03-24T00:47:35.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:47:35.206+0000 7f75d77fe640 1 -- 172.21.15.67:0/1288347833 <== mon.0 v2:172.21.15.67:3300/0 10 ==== log(1 entries from seq 1036 at 2024-03-24T00:47:34.201767+0000) v1 ==== 299+0+0 (secure 0 0 0) 0x7f75e8059a10 con 0x7f75f015bf50 2024-03-24T00:47:47.015 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:47:47.018 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:48:00.728 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:48:00.729 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.7:0 scrub status 2024-03-24T00:48:00.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.835+0000 7f676bc85640 1 Processor -- start 2024-03-24T00:48:00.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.835+0000 7f676bc85640 1 -- start start 2024-03-24T00:48:00.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.836+0000 7f676bc85640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f676415dea0 0x7f67640d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:00.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.836+0000 7f676bc85640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67640d2250 con 0x7f676415e280 2024-03-24T00:48:00.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.836+0000 7f676bc85640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67640d2530 con 0x7f67640b6170 2024-03-24T00:48:00.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.836+0000 7f676bc85640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67640d2810 con 0x7f676415dea0 2024-03-24T00:48:00.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.836+0000 7f6769220640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f676415e280 0x7f67640b5a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:51390/0 (socket says 172.21.15.67:51390) 2024-03-24T00:48:00.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.836+0000 7f6769220640 1 -- 172.21.15.67:0/2241687710 learned_addr learned my addr 172.21.15.67:0/2241687710 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:48:00.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.836+0000 7f6769a21640 1 --2- 172.21.15.67:0/2241687710 >> v2:172.21.15.72:3300/0 conn(0x7f676415dea0 0x7f67640d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:00.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.836+0000 7f6768a1f640 1 -- 172.21.15.67:0/2241687710 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2411981744 0 0) 0x7f67640d2250 con 0x7f676415e280 2024-03-24T00:48:00.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.836+0000 7f6768a1f640 1 -- 172.21.15.67:0/2241687710 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6750003520 con 0x7f676415e280 2024-03-24T00:48:00.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.837+0000 7f6768a1f640 1 -- 172.21.15.67:0/2241687710 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2798758117 0 0) 0x7f6750003520 con 0x7f676415e280 2024-03-24T00:48:00.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.837+0000 7f6768a1f640 1 -- 172.21.15.67:0/2241687710 >> v2:172.21.15.72:3300/0 conn(0x7f676415dea0 msgr2=0x7f67640d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:00.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.837+0000 7f6768a1f640 1 --2- 172.21.15.67:0/2241687710 >> v2:172.21.15.72:3300/0 conn(0x7f676415dea0 0x7f67640d33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:00.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.837+0000 7f6768a1f640 1 -- 172.21.15.67:0/2241687710 >> v1:172.21.15.72:6789/0 conn(0x7f67640b6170 legacy=0x7f67640b70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:00.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.837+0000 7f6768a1f640 1 -- 172.21.15.67:0/2241687710 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67640d2af0 con 0x7f676415e280 2024-03-24T00:48:00.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.837+0000 7f6768a1f640 1 -- 172.21.15.67:0/2241687710 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f6754001670 con 0x7f676415e280 2024-03-24T00:48:00.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.837+0000 7f6768a1f640 1 -- 172.21.15.67:0/2241687710 >> v1:172.21.15.67:6789/0 conn(0x7f676415e280 legacy=0x7f67640b5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:00.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.837+0000 7f6768a1f640 1 --2- 172.21.15.67:0/2241687710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67500039b0 0x7f6750003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:00.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.837+0000 7f6768a1f640 1 --2- 172.21.15.67:0/2241687710 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6750004800 0x7f6750004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:00.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.837+0000 7f6768a1f640 1 --2- 172.21.15.67:0/2241687710 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f67500051c0 0x7f675000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:00.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.837+0000 7f6768a1f640 1 -- 172.21.15.67:0/2241687710 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67640d2810 con 0x7f67500039b0 2024-03-24T00:48:00.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.837+0000 7f6768a1f640 1 -- 172.21.15.67:0/2241687710 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f675000c160 con 0x7f6750004800 2024-03-24T00:48:00.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.837+0000 7f6768a1f640 1 -- 172.21.15.67:0/2241687710 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f675000c360 con 0x7f67500051c0 2024-03-24T00:48:00.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.837+0000 7f6769a21640 1 --2- 172.21.15.67:0/2241687710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67500039b0 0x7f6750003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:00.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.837+0000 7f6769220640 1 --2- 172.21.15.67:0/2241687710 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6750004800 0x7f6750004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:00.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.837+0000 7f676a222640 1 --2- 172.21.15.67:0/2241687710 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f67500051c0 0x7f675000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:00.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.837+0000 7f6769a21640 1 -- 172.21.15.67:0/2241687710 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f67500051c0 msgr2=0x7f675000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:00.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.837+0000 7f6769a21640 1 --2- 172.21.15.67:0/2241687710 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f67500051c0 0x7f675000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:00.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.837+0000 7f6769a21640 1 -- 172.21.15.67:0/2241687710 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6750004800 msgr2=0x7f6750004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:00.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.837+0000 7f6769a21640 1 --2- 172.21.15.67:0/2241687710 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6750004800 0x7f6750004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:00.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.837+0000 7f6769a21640 1 -- 172.21.15.67:0/2241687710 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6750003520 con 0x7f67500039b0 2024-03-24T00:48:00.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.837+0000 7f676a222640 1 -- 172.21.15.67:0/2241687710 reap_dead start 2024-03-24T00:48:00.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.838+0000 7f6769a21640 1 --2- 172.21.15.67:0/2241687710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67500039b0 0x7f6750003dd0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f6760009ac0 tx=0x7f6750005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80d715dece5f7d84 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:00.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.838+0000 7f6768a1f640 1 -- 172.21.15.67:0/2241687710 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6760039070 con 0x7f67500039b0 2024-03-24T00:48:00.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.838+0000 7f6768a1f640 1 -- 172.21.15.67:0/2241687710 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f676002c6b0 con 0x7f67500039b0 2024-03-24T00:48:00.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.838+0000 7f6768a1f640 1 -- 172.21.15.67:0/2241687710 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6760009690 con 0x7f67500039b0 2024-03-24T00:48:00.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.838+0000 7f676bc85640 1 -- 172.21.15.67:0/2241687710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67500039b0 msgr2=0x7f6750003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:00.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.838+0000 7f676bc85640 1 --2- 172.21.15.67:0/2241687710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67500039b0 0x7f6750003dd0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f6760009ac0 tx=0x7f6750005830 comp rx=0 tx=0).stop 2024-03-24T00:48:00.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.839+0000 7f676bc85640 1 -- 172.21.15.67:0/2241687710 shutdown_connections 2024-03-24T00:48:00.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.839+0000 7f676bc85640 1 --2- 172.21.15.67:0/2241687710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67500039b0 0x7f6750003dd0 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:00.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.839+0000 7f676bc85640 1 -- 172.21.15.67:0/2241687710 >> 172.21.15.67:0/2241687710 conn(0x7f6764072da0 msgr2=0x7f67640d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:00.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.839+0000 7f676bc85640 1 -- 172.21.15.67:0/2241687710 shutdown_connections 2024-03-24T00:48:00.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.839+0000 7f676bc85640 1 -- 172.21.15.67:0/2241687710 wait complete. 2024-03-24T00:48:00.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.840+0000 7f676bc85640 1 Processor -- start 2024-03-24T00:48:00.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.840+0000 7f676bc85640 1 -- start start 2024-03-24T00:48:00.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.840+0000 7f676bc85640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f67500039b0 0x7f6764156c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:00.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.840+0000 7f676bc85640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f67640cbbe0 0x7f67640cd050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:00.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.840+0000 7f676bc85640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67640cd590 0x7f676415f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:00.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.840+0000 7f676bc85640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67640b5df0 con 0x7f67640cd590 2024-03-24T00:48:00.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.840+0000 7f676bc85640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67640b5c70 con 0x7f67500039b0 2024-03-24T00:48:00.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.840+0000 7f676bc85640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67640b5f70 con 0x7f67640cbbe0 2024-03-24T00:48:00.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.841+0000 7f676a222640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67640cd590 0x7f676415f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:00.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.841+0000 7f676a222640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67640cd590 0x7f676415f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36050/0 (socket says 172.21.15.67:36050) 2024-03-24T00:48:00.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.841+0000 7f676a222640 1 -- 172.21.15.67:0/136446100 learned_addr learned my addr 172.21.15.67:0/136446100 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:00.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.841+0000 7f6769220640 1 --2- 172.21.15.67:0/136446100 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f67640cbbe0 0x7f67640cd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:00.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.841+0000 7f6769a21640 1 --2- 172.21.15.67:0/136446100 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f67500039b0 0x7f6764156c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:00.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.841+0000 7f676a222640 1 -- 172.21.15.67:0/136446100 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f67640cbbe0 msgr2=0x7f67640cd050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:00.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.841+0000 7f676a222640 1 --2- 172.21.15.67:0/136446100 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f67640cbbe0 0x7f67640cd050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:00.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.841+0000 7f676a222640 1 -- 172.21.15.67:0/136446100 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f67500039b0 msgr2=0x7f6764156c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:00.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.841+0000 7f676a222640 1 --2- 172.21.15.67:0/136446100 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f67500039b0 0x7f6764156c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:00.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.841+0000 7f676a222640 1 -- 172.21.15.67:0/136446100 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f676415e8c0 con 0x7f67640cd590 2024-03-24T00:48:00.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.841+0000 7f676a222640 1 --2- 172.21.15.67:0/136446100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67640cd590 0x7f676415f360 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f676406da80 tx=0x7f675c00fbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:00.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.841+0000 7f675a7fc640 1 -- 172.21.15.67:0/136446100 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f675c003970 con 0x7f67640cd590 2024-03-24T00:48:00.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.841+0000 7f675a7fc640 1 -- 172.21.15.67:0/136446100 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f675c018070 con 0x7f67640cd590 2024-03-24T00:48:00.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.841+0000 7f676bc85640 1 -- 172.21.15.67:0/136446100 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67640cc3a0 con 0x7f67640cd590 2024-03-24T00:48:00.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.841+0000 7f675a7fc640 1 -- 172.21.15.67:0/136446100 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f675c003d70 con 0x7f67640cd590 2024-03-24T00:48:00.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.841+0000 7f676bc85640 1 -- 172.21.15.67:0/136446100 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67640cc060 con 0x7f67640cd590 2024-03-24T00:48:00.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.843+0000 7f675a7fc640 1 -- 172.21.15.67:0/136446100 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f675c0050d0 con 0x7f67640cd590 2024-03-24T00:48:00.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.843+0000 7f675a7fc640 1 --2- 172.21.15.67:0/136446100 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f674806bbd0 0x7f674806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:00.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.843+0000 7f675a7fc640 1 -- 172.21.15.67:0/136446100 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6130+0+0 (secure 0 0 0) 0x7f675c08a840 con 0x7f67640cd590 2024-03-24T00:48:00.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.844+0000 7f6769a21640 1 --2- 172.21.15.67:0/136446100 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f674806bbd0 0x7f674806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:00.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.844+0000 7f6769a21640 1 --2- 172.21.15.67:0/136446100 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f674806bbd0 0x7f674806e090 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f6760036000 tx=0x7f6760030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:00.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.844+0000 7f676e87c740 1 Processor -- start 2024-03-24T00:48:00.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.844+0000 7f676e87c740 1 -- start start 2024-03-24T00:48:00.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.845+0000 7f676e87c740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603e6bc7e50 0x5603e6b9adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:00.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.845+0000 7f676e87c740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5603e6bc8230 0x5603e6bcabe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:00.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.845+0000 7f676e87c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603e6b9b330 0x5603e6bcb120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:00.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.845+0000 7f676e87c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603e6b053d0 con 0x5603e6b9b330 2024-03-24T00:48:00.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.845+0000 7f676e87c740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603e6aadad0 con 0x5603e6bc7e50 2024-03-24T00:48:00.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.845+0000 7f676e87c740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5603e6b01470 con 0x5603e6bc8230 2024-03-24T00:48:00.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.845+0000 7f676a222640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603e6b9b330 0x5603e6bcb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:00.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.845+0000 7f676a222640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603e6b9b330 0x5603e6bcb120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36056/0 (socket says 172.21.15.67:36056) 2024-03-24T00:48:00.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.845+0000 7f676a222640 1 -- 172.21.15.67:0/3055867961 learned_addr learned my addr 172.21.15.67:0/3055867961 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:00.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.845+0000 7f6769a21640 1 --2- 172.21.15.67:0/3055867961 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603e6bc7e50 0x5603e6b9adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:00.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.845+0000 7f6769220640 1 --2- 172.21.15.67:0/3055867961 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5603e6bc8230 0x5603e6bcabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:00.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.845+0000 7f676a222640 1 -- 172.21.15.67:0/3055867961 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5603e6bc8230 msgr2=0x5603e6bcabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:00.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.845+0000 7f676a222640 1 --2- 172.21.15.67:0/3055867961 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5603e6bc8230 0x5603e6bcabe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:00.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.845+0000 7f676a222640 1 -- 172.21.15.67:0/3055867961 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603e6bc7e50 msgr2=0x5603e6b9adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:00.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.845+0000 7f676a222640 1 --2- 172.21.15.67:0/3055867961 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603e6bc7e50 0x5603e6b9adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:00.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.845+0000 7f676a222640 1 -- 172.21.15.67:0/3055867961 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5603e68e37d0 con 0x5603e6b9b330 2024-03-24T00:48:00.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.846+0000 7f676a222640 1 --2- 172.21.15.67:0/3055867961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603e6b9b330 0x5603e6bcb120 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f675c088bf0 tx=0x7f675c059210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8e8fc9a8c9fd534 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:00.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.846+0000 7f67477fe640 1 -- 172.21.15.67:0/3055867961 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f675c06ce60 con 0x5603e6b9b330 2024-03-24T00:48:00.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.846+0000 7f67477fe640 1 -- 172.21.15.67:0/3055867961 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f675c05cc60 con 0x5603e6b9b330 2024-03-24T00:48:00.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.846+0000 7f67477fe640 1 -- 172.21.15.67:0/3055867961 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f675c061220 con 0x5603e6b9b330 2024-03-24T00:48:00.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.846+0000 7f676e87c740 1 -- 172.21.15.67:0/3055867961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603e6b9b330 msgr2=0x5603e6bcb120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:00.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.846+0000 7f676e87c740 1 --2- 172.21.15.67:0/3055867961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603e6b9b330 0x5603e6bcb120 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f675c088bf0 tx=0x7f675c059210 comp rx=0 tx=0).stop 2024-03-24T00:48:00.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.846+0000 7f676e87c740 1 -- 172.21.15.67:0/3055867961 shutdown_connections 2024-03-24T00:48:00.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.846+0000 7f676e87c740 1 --2- 172.21.15.67:0/3055867961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603e6b9b330 0x5603e6bcb120 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:00.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.846+0000 7f676e87c740 1 --2- 172.21.15.67:0/3055867961 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5603e6bc8230 0x5603e6bcabe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:00.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.846+0000 7f676e87c740 1 --2- 172.21.15.67:0/3055867961 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603e6bc7e50 0x5603e6b9adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:00.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.846+0000 7f676e87c740 1 -- 172.21.15.67:0/3055867961 >> 172.21.15.67:0/3055867961 conn(0x5603e6b9d060 msgr2=0x5603e6be5be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:00.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.846+0000 7f676e87c740 1 -- 172.21.15.67:0/3055867961 shutdown_connections 2024-03-24T00:48:00.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.847+0000 7f676e87c740 1 -- 172.21.15.67:0/3055867961 wait complete. 2024-03-24T00:48:00.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.847+0000 7f676e87c740 1 -- start start 2024-03-24T00:48:00.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.847+0000 7f676e87c740 1 Processor -- start 2024-03-24T00:48:00.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.848+0000 7f676e87c740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5603e6bc7e50 0x5603e6bf7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:00.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.848+0000 7f676e87c740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603e6bc8230 0x5603e6be5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:00.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.848+0000 7f676e87c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603e6bc1530 0x5603e6bca680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:00.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.848+0000 7f676e87c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603e6b9f700 con 0x5603e6bc1530 2024-03-24T00:48:00.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.848+0000 7f676e87c740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603e6ac1060 con 0x5603e6bc8230 2024-03-24T00:48:00.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.848+0000 7f676e87c740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5603e6ba1f40 con 0x5603e6bc7e50 2024-03-24T00:48:00.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.848+0000 7f676e87c740 10 client.0 _ll_register_callbacks cb 0x5603e6aa36a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:48:00.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.848+0000 7f676a222640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603e6bc1530 0x5603e6bca680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:00.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.848+0000 7f6769a21640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5603e6bc7e50 0x5603e6bf7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:00.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.848+0000 7f676a222640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603e6bc1530 0x5603e6bca680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36068/0 (socket says 172.21.15.67:36068) 2024-03-24T00:48:00.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.848+0000 7f676a222640 1 -- 172.21.15.67:0/1601321850 learned_addr learned my addr 172.21.15.67:0/1601321850 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:00.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.848+0000 7f6769220640 1 --2- 172.21.15.67:0/1601321850 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603e6bc8230 0x5603e6be5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:00.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.848+0000 7f676a222640 1 -- 172.21.15.67:0/1601321850 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5603e6bc7e50 msgr2=0x5603e6bf7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:00.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.848+0000 7f676a222640 1 --2- 172.21.15.67:0/1601321850 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5603e6bc7e50 0x5603e6bf7430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:00.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.848+0000 7f676a222640 1 -- 172.21.15.67:0/1601321850 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603e6bc8230 msgr2=0x5603e6be5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:00.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.848+0000 7f676a222640 1 --2- 172.21.15.67:0/1601321850 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603e6bc8230 0x5603e6be5ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:00.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.848+0000 7f676a222640 1 -- 172.21.15.67:0/1601321850 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5603e68193f0 con 0x5603e6bc1530 2024-03-24T00:48:00.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.849+0000 7f676a222640 1 --2- 172.21.15.67:0/1601321850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603e6bc1530 0x5603e6bca680 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f675c0593c0 tx=0x7f675c0011e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbe50d408601aaa9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:00.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.849+0000 7f6744ff9640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:48:00.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.849+0000 7f6744ff9640 1 -- 172.21.15.67:0/1601321850 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f675c075060 con 0x5603e6bc1530 2024-03-24T00:48:00.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.849+0000 7f6744ff9640 1 -- 172.21.15.67:0/1601321850 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(107..107 src has 1..107) v4 ==== 6130+0+0 (secure 0 0 0) 0x7f675c05dd80 con 0x5603e6bc1530 2024-03-24T00:48:00.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.849+0000 7f676e87c740 1 -- 172.21.15.67:0/1601321850 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f675c05cc60 con 0x5603e6bc1530 2024-03-24T00:48:00.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.849+0000 7f6744ff9640 1 -- 172.21.15.67:0/1601321850 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7f675c06ce60 con 0x5603e6bc1530 2024-03-24T00:48:00.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.849+0000 7f676e87c740 10 client.8874 fetch_fsmap learned FSMap version 72 2024-03-24T00:48:00.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.849+0000 7f676e87c740 1 -- 172.21.15.67:0/1601321850 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=72,monmap=2+}) v3 -- 0x5603e67d90f0 con 0x5603e6bc1530 2024-03-24T00:48:00.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.849+0000 7f6744ff9640 1 -- 172.21.15.67:0/1601321850 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f675c05cc60 con 0x5603e6bc1530 2024-03-24T00:48:00.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.849+0000 7f6744ff9640 1 -- 172.21.15.67:0/1601321850 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 72) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f675c057f60 con 0x5603e6bc1530 2024-03-24T00:48:00.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.849+0000 7f676e87c740 10 client.8874 fetch_fsmap finished waiting for FSMap version 72 2024-03-24T00:48:00.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.849+0000 7f676e87c740 10 client.8874 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.c 2024-03-24T00:48:00.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.849+0000 7f676e87c740 20 client.8874 populate_metadata read hostname 'smithi067' 2024-03-24T00:48:00.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.849+0000 7f676e87c740 1 --2- 172.21.15.67:0/1601321850 >> [v2:172.21.15.67:6832/1826974843,v1:172.21.15.67:6833/1826974843] conn(0x5603e6bc3350 0x5603e6bc3760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:00.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.849+0000 7f676e87c740 4 client.8874 mds_command: new command op to 8541 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:48:00.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.849+0000 7f676e87c740 1 -- 172.21.15.67:0/1601321850 --> [v2:172.21.15.67:6832/1826974843,v1:172.21.15.67:6833/1826974843] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5603e6b12910 con 0x5603e6bc3350 2024-03-24T00:48:00.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.849+0000 7f6769a21640 1 --2- 172.21.15.67:0/1601321850 >> [v2:172.21.15.67:6832/1826974843,v1:172.21.15.67:6833/1826974843] conn(0x5603e6bc3350 0x5603e6bc3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:00.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.850+0000 7f6744ff9640 10 client.8874 ms_handle_connect on v2:172.21.15.67:6832/1826974843 2024-03-24T00:48:00.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.850+0000 7f6769a21640 1 --2- 172.21.15.67:0/1601321850 >> [v2:172.21.15.67:6832/1826974843,v1:172.21.15.67:6833/1826974843] conn(0x5603e6bc3350 0x5603e6bc3760 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=565f300bad1c6d5 server_cookie=eebd944d85b44ded in_seq=0 out_seq=0 2024-03-24T00:48:00.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.850+0000 7f6744ff9640 1 -- 172.21.15.67:0/1601321850 <== mds.0 v2:172.21.15.67:6832/1826974843 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5603e6b12910 con 0x5603e6bc3350 2024-03-24T00:48:00.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.850+0000 7f6744ff9640 10 client.8874 handle_command_reply: tid=0 2024-03-24T00:48:00.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.850+0000 7f676e87c740 1 client.8874 shutdown 2024-03-24T00:48:00.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.850+0000 7f676e87c740 1 -- 172.21.15.67:0/1601321850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603e6bc1530 msgr2=0x5603e6bca680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:00.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.850+0000 7f676e87c740 1 --2- 172.21.15.67:0/1601321850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603e6bc1530 0x5603e6bca680 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f675c0593c0 tx=0x7f675c0011e0 comp rx=0 tx=0).stop 2024-03-24T00:48:00.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.851+0000 7f676e87c740 1 -- 172.21.15.67:0/1601321850 shutdown_connections 2024-03-24T00:48:00.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.851+0000 7f676e87c740 1 --2- 172.21.15.67:0/1601321850 >> [v2:172.21.15.67:6832/1826974843,v1:172.21.15.67:6833/1826974843] conn(0x5603e6bc3350 0x5603e6bc3760 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:00.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.851+0000 7f676e87c740 1 --2- 172.21.15.67:0/1601321850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603e6bc1530 0x5603e6bca680 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:00.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.851+0000 7f676e87c740 1 --2- 172.21.15.67:0/1601321850 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603e6bc8230 0x5603e6be5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:00.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.851+0000 7f676e87c740 1 --2- 172.21.15.67:0/1601321850 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5603e6bc7e50 0x5603e6bf7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:00.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.851+0000 7f676e87c740 1 -- 172.21.15.67:0/1601321850 >> 172.21.15.67:0/1601321850 conn(0x5603e6b9b330 msgr2=0x5603e6b9edd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:00.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.851+0000 7f6744ff9640 0 client.8874 ms_handle_reset on v2:172.21.15.67:6832/1826974843 2024-03-24T00:48:00.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.851+0000 7f676e87c740 1 -- 172.21.15.67:0/1601321850 shutdown_connections 2024-03-24T00:48:00.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.851+0000 7f676e87c740 1 -- 172.21.15.67:0/1601321850 wait complete. 2024-03-24T00:48:00.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.851+0000 7f676e87c740 20 client.8874 trim_cache size 0 max 16384 2024-03-24T00:48:00.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.867+0000 7f676e87c740 1 Processor -- start 2024-03-24T00:48:00.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.867+0000 7f676e87c740 1 -- start start 2024-03-24T00:48:00.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.867+0000 7f676e87c740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603e6bc1530 0x5603e6bf7410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:00.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.867+0000 7f676e87c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603e6bc7e50 0x5603e6bffde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:00.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.867+0000 7f676e87c740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5603e6bc8230 0x5603e6bc0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:00.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.867+0000 7f676e87c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6764200ff0 con 0x5603e6bc7e50 2024-03-24T00:48:00.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.867+0000 7f676e87c740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6764200190 con 0x5603e6bc1530 2024-03-24T00:48:00.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.867+0000 7f676e87c740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67641ff590 con 0x5603e6bc8230 2024-03-24T00:48:00.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.868+0000 7f6769220640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603e6bc7e50 0x5603e6bffde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:00.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.868+0000 7f6769220640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603e6bc7e50 0x5603e6bffde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36070/0 (socket says 172.21.15.67:36070) 2024-03-24T00:48:00.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.868+0000 7f6769220640 1 -- 172.21.15.67:0/1779667283 learned_addr learned my addr 172.21.15.67:0/1779667283 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:00.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.868+0000 7f6769220640 1 -- 172.21.15.67:0/1779667283 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5603e6bc8230 msgr2=0x5603e6bc0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:00.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.868+0000 7f6769220640 1 --2- 172.21.15.67:0/1779667283 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5603e6bc8230 0x5603e6bc0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:00.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.868+0000 7f6769220640 1 -- 172.21.15.67:0/1779667283 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603e6bc1530 msgr2=0x5603e6bf7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:00.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.868+0000 7f6769220640 1 --2- 172.21.15.67:0/1779667283 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603e6bc1530 0x5603e6bf7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:00.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.868+0000 7f6769220640 1 -- 172.21.15.67:0/1779667283 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5603e6bcb6f0 con 0x5603e6bc7e50 2024-03-24T00:48:00.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.868+0000 7f6769220640 1 --2- 172.21.15.67:0/1779667283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603e6bc7e50 0x5603e6bffde0 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f6754007f60 tx=0x7f67540078f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=698a59454dadd172 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:00.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.868+0000 7f6746ffd640 1 -- 172.21.15.67:0/1779667283 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6754007730 con 0x5603e6bc7e50 2024-03-24T00:48:00.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.868+0000 7f6746ffd640 1 -- 172.21.15.67:0/1779667283 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6754016420 con 0x5603e6bc7e50 2024-03-24T00:48:00.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.868+0000 7f6746ffd640 1 -- 172.21.15.67:0/1779667283 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67540167d0 con 0x5603e6bc7e50 2024-03-24T00:48:00.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.869+0000 7f676e87c740 1 -- 172.21.15.67:0/1779667283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603e6bc7e50 msgr2=0x5603e6bffde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:00.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.869+0000 7f676e87c740 1 --2- 172.21.15.67:0/1779667283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603e6bc7e50 0x5603e6bffde0 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f6754007f60 tx=0x7f67540078f0 comp rx=0 tx=0).stop 2024-03-24T00:48:00.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.869+0000 7f676e87c740 1 -- 172.21.15.67:0/1779667283 shutdown_connections 2024-03-24T00:48:00.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.869+0000 7f676e87c740 1 --2- 172.21.15.67:0/1779667283 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5603e6bc8230 0x5603e6bc0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:00.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.869+0000 7f676e87c740 1 --2- 172.21.15.67:0/1779667283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603e6bc7e50 0x5603e6bffde0 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:00.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.869+0000 7f676e87c740 1 --2- 172.21.15.67:0/1779667283 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603e6bc1530 0x5603e6bf7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:00.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.869+0000 7f676e87c740 1 -- 172.21.15.67:0/1779667283 >> 172.21.15.67:0/1779667283 conn(0x5603e6b9b330 msgr2=0x5603e6be5cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:00.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.869+0000 7f676e87c740 1 -- 172.21.15.67:0/1779667283 shutdown_connections 2024-03-24T00:48:00.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.869+0000 7f676e87c740 1 -- 172.21.15.67:0/1779667283 wait complete. 2024-03-24T00:48:00.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.869+0000 7f676e87c740 1 -- start start 2024-03-24T00:48:00.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.869+0000 7f676e87c740 1 Processor -- start 2024-03-24T00:48:00.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.870+0000 7f676e87c740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603e6bc1530 0x5603e6c45a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:00.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.870+0000 7f676e87c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603e6bc7e50 0x5603e6c48da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:00.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.870+0000 7f676e87c740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5603e6bc8230 0x5603e6c492e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:00.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.870+0000 7f676e87c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603e6b9c960 con 0x5603e6bc7e50 2024-03-24T00:48:00.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.870+0000 7f676e87c740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67641ffb90 con 0x5603e6bc1530 2024-03-24T00:48:00.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.870+0000 7f676e87c740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5603e6bcb310 con 0x5603e6bc8230 2024-03-24T00:48:00.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.870+0000 7f676e87c740 10 client.0 _ll_register_callbacks cb 0x7f675c06f070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:48:00.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.870+0000 7f6769220640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603e6bc7e50 0x5603e6c48da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:00.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.870+0000 7f6769220640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603e6bc7e50 0x5603e6c48da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36080/0 (socket says 172.21.15.67:36080) 2024-03-24T00:48:00.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.870+0000 7f6769220640 1 -- 172.21.15.67:0/3889515253 learned_addr learned my addr 172.21.15.67:0/3889515253 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:00.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.870+0000 7f676a222640 1 --2- 172.21.15.67:0/3889515253 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5603e6bc8230 0x5603e6c492e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:00.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.871+0000 7f6769a21640 1 --2- 172.21.15.67:0/3889515253 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603e6bc1530 0x5603e6c45a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:00.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.871+0000 7f6769220640 1 -- 172.21.15.67:0/3889515253 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5603e6bc8230 msgr2=0x5603e6c492e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:00.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.871+0000 7f6769220640 1 --2- 172.21.15.67:0/3889515253 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5603e6bc8230 0x5603e6c492e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:00.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.871+0000 7f6769220640 1 -- 172.21.15.67:0/3889515253 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603e6bc1530 msgr2=0x5603e6c45a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:00.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.871+0000 7f6769220640 1 --2- 172.21.15.67:0/3889515253 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603e6bc1530 0x5603e6c45a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:00.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.871+0000 7f6769220640 1 -- 172.21.15.67:0/3889515253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5603e6c48a60 con 0x5603e6bc7e50 2024-03-24T00:48:00.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.871+0000 7f676a222640 1 --2- 172.21.15.67:0/3889515253 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5603e6bc8230 0x5603e6c492e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:48:00.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.871+0000 7f6769220640 1 --2- 172.21.15.67:0/3889515253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603e6bc7e50 0x5603e6c48da0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f6764050d00 tx=0x7f67640d2700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e58d4461a4e854 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:00.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.871+0000 7f6744ff9640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:48:00.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.871+0000 7f6744ff9640 1 -- 172.21.15.67:0/3889515253 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6754016860 con 0x5603e6bc7e50 2024-03-24T00:48:00.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.871+0000 7f6744ff9640 1 -- 172.21.15.67:0/3889515253 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(107..107 src has 1..107) v4 ==== 6130+0+0 (secure 0 0 0) 0x7f675401c050 con 0x5603e6bc7e50 2024-03-24T00:48:00.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.871+0000 7f676e87c740 1 -- 172.21.15.67:0/3889515253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6754016420 con 0x5603e6bc7e50 2024-03-24T00:48:00.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.871+0000 7f6744ff9640 1 -- 172.21.15.67:0/3889515253 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6754033050 con 0x5603e6bc7e50 2024-03-24T00:48:00.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.871+0000 7f676e87c740 10 client.8880 fetch_fsmap learned FSMap version 72 2024-03-24T00:48:00.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.871+0000 7f676e87c740 1 -- 172.21.15.67:0/3889515253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=72,monmap=2+}) v3 -- 0x5603e6bc86d0 con 0x5603e6bc7e50 2024-03-24T00:48:00.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.871+0000 7f6744ff9640 1 -- 172.21.15.67:0/3889515253 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6754016420 con 0x5603e6bc7e50 2024-03-24T00:48:00.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.871+0000 7f6744ff9640 1 -- 172.21.15.67:0/3889515253 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 72) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f6754036020 con 0x5603e6bc7e50 2024-03-24T00:48:00.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.871+0000 7f676e87c740 10 client.8880 fetch_fsmap finished waiting for FSMap version 72 2024-03-24T00:48:00.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.871+0000 7f676e87c740 10 client.8880 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.c 2024-03-24T00:48:00.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.871+0000 7f676e87c740 20 client.8880 populate_metadata read hostname 'smithi067' 2024-03-24T00:48:00.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.871+0000 7f676e87c740 1 --2- 172.21.15.67:0/3889515253 >> [v2:172.21.15.67:6832/1826974843,v1:172.21.15.67:6833/1826974843] conn(0x5603e6c46050 0x5603e6c4b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:00.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.871+0000 7f676e87c740 4 client.8880 mds_command: new command op to 8541 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:48:00.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.871+0000 7f676e87c740 1 -- 172.21.15.67:0/3889515253 --> [v2:172.21.15.67:6832/1826974843,v1:172.21.15.67:6833/1826974843] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5603e6b0cc60 con 0x5603e6c46050 2024-03-24T00:48:00.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.872+0000 7f6769a21640 1 --2- 172.21.15.67:0/3889515253 >> [v2:172.21.15.67:6832/1826974843,v1:172.21.15.67:6833/1826974843] conn(0x5603e6c46050 0x5603e6c4b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:00.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.872+0000 7f6744ff9640 10 client.8880 ms_handle_connect on v2:172.21.15.67:6832/1826974843 2024-03-24T00:48:00.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.872+0000 7f6769a21640 1 --2- 172.21.15.67:0/3889515253 >> [v2:172.21.15.67:6832/1826974843,v1:172.21.15.67:6833/1826974843] conn(0x5603e6c46050 0x5603e6c4b0e0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac7add203aae095c server_cookie=1a14430768289312 in_seq=0 out_seq=0 2024-03-24T00:48:00.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.872+0000 7f6744ff9640 1 -- 172.21.15.67:0/3889515253 <== mds.0 v2:172.21.15.67:6832/1826974843 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5603e6b0cc60 con 0x5603e6c46050 2024-03-24T00:48:00.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.872+0000 7f6744ff9640 10 client.8880 handle_command_reply: tid=0 2024-03-24T00:48:00.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.872+0000 7f676e87c740 1 client.8880 shutdown 2024-03-24T00:48:00.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.872+0000 7f676e87c740 1 -- 172.21.15.67:0/3889515253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603e6bc7e50 msgr2=0x5603e6c48da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:00.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.872+0000 7f676e87c740 1 --2- 172.21.15.67:0/3889515253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603e6bc7e50 0x5603e6c48da0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f6764050d00 tx=0x7f67640d2700 comp rx=0 tx=0).stop 2024-03-24T00:48:00.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.872+0000 7f676e87c740 1 -- 172.21.15.67:0/3889515253 shutdown_connections 2024-03-24T00:48:00.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.872+0000 7f676e87c740 1 --2- 172.21.15.67:0/3889515253 >> [v2:172.21.15.67:6832/1826974843,v1:172.21.15.67:6833/1826974843] conn(0x5603e6c46050 0x5603e6c4b0e0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:00.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.872+0000 7f676e87c740 1 --2- 172.21.15.67:0/3889515253 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5603e6bc8230 0x5603e6c492e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:00.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.872+0000 7f676e87c740 1 --2- 172.21.15.67:0/3889515253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603e6bc7e50 0x5603e6c48da0 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:00.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.872+0000 7f6744ff9640 0 client.8880 ms_handle_reset on v2:172.21.15.67:6832/1826974843 2024-03-24T00:48:00.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.872+0000 7f676e87c740 1 --2- 172.21.15.67:0/3889515253 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603e6bc1530 0x5603e6c45a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:00.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.872+0000 7f676e87c740 1 -- 172.21.15.67:0/3889515253 >> 172.21.15.67:0/3889515253 conn(0x5603e6b9b330 msgr2=0x5603e6bc0b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:00.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.872+0000 7f676e87c740 1 -- 172.21.15.67:0/3889515253 shutdown_connections 2024-03-24T00:48:00.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.872+0000 7f676e87c740 1 -- 172.21.15.67:0/3889515253 wait complete. 2024-03-24T00:48:00.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.872+0000 7f676e87c740 20 client.8880 trim_cache size 0 max 16384 2024-03-24T00:48:00.878 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:48:00.878 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "no active scrubs running", 2024-03-24T00:48:00.878 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": {} 2024-03-24T00:48:00.878 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:48:00.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.873+0000 7f676bc85640 1 -- 172.21.15.67:0/136446100 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f674806bbd0 msgr2=0x7f674806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:00.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.873+0000 7f676bc85640 1 --2- 172.21.15.67:0/136446100 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f674806bbd0 0x7f674806e090 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f6760036000 tx=0x7f6760030000 comp rx=0 tx=0).stop 2024-03-24T00:48:00.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.873+0000 7f676bc85640 1 -- 172.21.15.67:0/136446100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67640cd590 msgr2=0x7f676415f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:00.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.873+0000 7f676bc85640 1 --2- 172.21.15.67:0/136446100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67640cd590 0x7f676415f360 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f676406da80 tx=0x7f675c00fbb0 comp rx=0 tx=0).stop 2024-03-24T00:48:00.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.873+0000 7f676bc85640 1 -- 172.21.15.67:0/136446100 shutdown_connections 2024-03-24T00:48:00.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.873+0000 7f676bc85640 1 --2- 172.21.15.67:0/136446100 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f674806bbd0 0x7f674806e090 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:00.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.873+0000 7f676bc85640 1 --2- 172.21.15.67:0/136446100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67640cd590 0x7f676415f360 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:00.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.873+0000 7f676bc85640 1 --2- 172.21.15.67:0/136446100 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f67640cbbe0 0x7f67640cd050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:00.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.873+0000 7f676bc85640 1 --2- 172.21.15.67:0/136446100 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f67500039b0 0x7f6764156c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:00.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.873+0000 7f676bc85640 1 -- 172.21.15.67:0/136446100 >> 172.21.15.67:0/136446100 conn(0x7f6764072da0 msgr2=0x7f67640b39e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:00.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.873+0000 7f676bc85640 1 -- 172.21.15.67:0/136446100 shutdown_connections 2024-03-24T00:48:00.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:00.873+0000 7f676bc85640 1 -- 172.21.15.67:0/136446100 wait complete. 2024-03-24T00:48:00.883 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T00:48:00.984 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-24T00:48:01.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.087+0000 7f6b6e8a1640 1 Processor -- start 2024-03-24T00:48:01.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.087+0000 7f6b6e8a1640 1 -- start start 2024-03-24T00:48:01.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.089+0000 7f6b6e8a1640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6b6815e0f0 0x7f6b680b95b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:01.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.089+0000 7f6b6e8a1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6b680b6dc0 0x7f6b680b7210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:01.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.089+0000 7f6b6e8a1640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b680d2230 con 0x7f6b680b9af0 2024-03-24T00:48:01.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.089+0000 7f6b6e8a1640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b680b7750 con 0x7f6b6815e0f0 2024-03-24T00:48:01.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.089+0000 7f6b6e8a1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b680d2510 con 0x7f6b680b6dc0 2024-03-24T00:48:01.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.089+0000 7f6b67fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6b6815e0f0 0x7f6b680b95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:01.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.089+0000 7f6b67fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6b6815e0f0 0x7f6b680b95b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36090/0 (socket says 172.21.15.67:36090) 2024-03-24T00:48:01.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.089+0000 7f6b67fff640 1 -- 172.21.15.67:0/3410196611 learned_addr learned my addr 172.21.15.67:0/3410196611 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:01.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.089+0000 7f6b6ce3e640 1 --2- 172.21.15.67:0/3410196611 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6b680b6dc0 0x7f6b680b7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:01.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.089+0000 7f6b67fff640 1 -- 172.21.15.67:0/3410196611 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6b680b6dc0 msgr2=0x7f6b680b7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:01.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.089+0000 7f6b67fff640 1 --2- 172.21.15.67:0/3410196611 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6b680b6dc0 0x7f6b680b7210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.089+0000 7f6b67fff640 1 -- 172.21.15.67:0/3410196611 >> v1:172.21.15.72:6789/0 conn(0x7f6b680b9af0 legacy=0x7f6b680d32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:01.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.089+0000 7f6b67fff640 1 -- 172.21.15.67:0/3410196611 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b680d27f0 con 0x7f6b6815e0f0 2024-03-24T00:48:01.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.090+0000 7f6b67fff640 1 --2- 172.21.15.67:0/3410196611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b6815e0f0 0x7f6b680b95b0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f6b58009870 tx=0x7f6b5802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95e6a0a358e07f11 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:01.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.090+0000 7f6b66ffd640 1 -- 172.21.15.67:0/3410196611 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b58034070 con 0x7f6b6815e0f0 2024-03-24T00:48:01.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.090+0000 7f6b66ffd640 1 -- 172.21.15.67:0/3410196611 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b58030840 con 0x7f6b6815e0f0 2024-03-24T00:48:01.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.090+0000 7f6b66ffd640 1 -- 172.21.15.67:0/3410196611 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b58030c10 con 0x7f6b6815e0f0 2024-03-24T00:48:01.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.091+0000 7f6b6e8a1640 1 -- 172.21.15.67:0/3410196611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b6815e0f0 msgr2=0x7f6b680b95b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:01.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.091+0000 7f6b6e8a1640 1 --2- 172.21.15.67:0/3410196611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b6815e0f0 0x7f6b680b95b0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f6b58009870 tx=0x7f6b5802b980 comp rx=0 tx=0).stop 2024-03-24T00:48:01.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.091+0000 7f6b6e8a1640 1 -- 172.21.15.67:0/3410196611 shutdown_connections 2024-03-24T00:48:01.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.091+0000 7f6b6e8a1640 1 --2- 172.21.15.67:0/3410196611 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6b680b6dc0 0x7f6b680b7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.091+0000 7f6b6e8a1640 1 --2- 172.21.15.67:0/3410196611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b6815e0f0 0x7f6b680b95b0 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.091+0000 7f6b6e8a1640 1 -- 172.21.15.67:0/3410196611 >> 172.21.15.67:0/3410196611 conn(0x7f6b68072b60 msgr2=0x7f6b68072f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:01.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.091+0000 7f6b6e8a1640 1 -- 172.21.15.67:0/3410196611 shutdown_connections 2024-03-24T00:48:01.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.091+0000 7f6b6e8a1640 1 -- 172.21.15.67:0/3410196611 wait complete. 2024-03-24T00:48:01.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.092+0000 7f6b6e8a1640 1 Processor -- start 2024-03-24T00:48:01.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.092+0000 7f6b6e8a1640 1 -- start start 2024-03-24T00:48:01.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.093+0000 7f6b6e8a1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b680b6dc0 0x7f6b681f3430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:01.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.093+0000 7f6b6e8a1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b680b9af0 0x7f6b680ccc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:01.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.093+0000 7f6b6e8a1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6b6815e0f0 0x7f6b680cd180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:01.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.093+0000 7f6b6e8a1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b680b31d0 con 0x7f6b680b9af0 2024-03-24T00:48:01.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.093+0000 7f6b6e8a1640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b680d2db0 con 0x7f6b680b6dc0 2024-03-24T00:48:01.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.093+0000 7f6b6e8a1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b680b3350 con 0x7f6b6815e0f0 2024-03-24T00:48:01.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.093+0000 7f6b677fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b680b9af0 0x7f6b680ccc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:01.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.093+0000 7f6b677fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b680b9af0 0x7f6b680ccc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36104/0 (socket says 172.21.15.67:36104) 2024-03-24T00:48:01.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.093+0000 7f6b677fe640 1 -- 172.21.15.67:0/4157102128 learned_addr learned my addr 172.21.15.67:0/4157102128 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:01.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.093+0000 7f6b6ce3e640 1 --2- 172.21.15.67:0/4157102128 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6b6815e0f0 0x7f6b680cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:01.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.093+0000 7f6b67fff640 1 --2- 172.21.15.67:0/4157102128 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b680b6dc0 0x7f6b681f3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:01.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.093+0000 7f6b677fe640 1 -- 172.21.15.67:0/4157102128 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6b6815e0f0 msgr2=0x7f6b680cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:01.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.094+0000 7f6b677fe640 1 --2- 172.21.15.67:0/4157102128 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6b6815e0f0 0x7f6b680cd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.094+0000 7f6b677fe640 1 -- 172.21.15.67:0/4157102128 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b680b6dc0 msgr2=0x7f6b681f3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:01.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.094+0000 7f6b677fe640 1 --2- 172.21.15.67:0/4157102128 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b680b6dc0 0x7f6b681f3430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.094+0000 7f6b677fe640 1 -- 172.21.15.67:0/4157102128 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b6814bee0 con 0x7f6b680b9af0 2024-03-24T00:48:01.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.094+0000 7f6b677fe640 1 --2- 172.21.15.67:0/4157102128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b680b9af0 0x7f6b680ccc40 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c00c440 tx=0x7f6b5c001ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:01.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.094+0000 7f6b64ff9640 1 -- 172.21.15.67:0/4157102128 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b5c017070 con 0x7f6b680b9af0 2024-03-24T00:48:01.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.094+0000 7f6b64ff9640 1 -- 172.21.15.67:0/4157102128 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b5c0052f0 con 0x7f6b680b9af0 2024-03-24T00:48:01.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.094+0000 7f6b6e8a1640 1 -- 172.21.15.67:0/4157102128 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b68159930 con 0x7f6b680b9af0 2024-03-24T00:48:01.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.094+0000 7f6b64ff9640 1 -- 172.21.15.67:0/4157102128 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b5c003660 con 0x7f6b680b9af0 2024-03-24T00:48:01.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.094+0000 7f6b6e8a1640 1 -- 172.21.15.67:0/4157102128 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b68159ef0 con 0x7f6b680b9af0 2024-03-24T00:48:01.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.095+0000 7f6b6e8a1640 1 -- 172.21.15.67:0/4157102128 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b680b6a80 con 0x7f6b680b9af0 2024-03-24T00:48:01.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.095+0000 7f6b64ff9640 1 -- 172.21.15.67:0/4157102128 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6b5c01a070 con 0x7f6b680b9af0 2024-03-24T00:48:01.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.096+0000 7f6b64ff9640 1 --2- 172.21.15.67:0/4157102128 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6b5406bbd0 0x7f6b5406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:01.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.096+0000 7f6b64ff9640 1 -- 172.21.15.67:0/4157102128 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6130+0+0 (secure 0 0 0) 0x7f6b5c089fa0 con 0x7f6b680b9af0 2024-03-24T00:48:01.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.096+0000 7f6b67fff640 1 --2- 172.21.15.67:0/4157102128 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6b5406bbd0 0x7f6b5406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:01.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.097+0000 7f6b67fff640 1 --2- 172.21.15.67:0/4157102128 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6b5406bbd0 0x7f6b5406e090 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f6b58036000 tx=0x7f6b5803b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:01.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.101+0000 7f6b64ff9640 1 -- 172.21.15.67:0/4157102128 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6b680b6a80 con 0x7f6b680b9af0 2024-03-24T00:48:01.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.321+0000 7f6b6e8a1640 1 -- 172.21.15.67:0/4157102128 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f6b680ca620 con 0x7f6b680b9af0 2024-03-24T00:48:01.322 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-24T00:48:01.322+0000 7fbfcfa11640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T00:48:01.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.324+0000 7f6b64ff9640 1 -- 172.21.15.67:0/4157102128 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v73) v1 ==== 161+0+0 (secure 0 0 0) 0x7f6b5c058290 con 0x7f6b680b9af0 2024-03-24T00:48:01.323 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-24T00:48:01.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.327+0000 7f6b6e8a1640 1 -- 172.21.15.67:0/4157102128 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6b5406bbd0 msgr2=0x7f6b5406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:01.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.327+0000 7f6b6e8a1640 1 --2- 172.21.15.67:0/4157102128 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6b5406bbd0 0x7f6b5406e090 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f6b58036000 tx=0x7f6b5803b040 comp rx=0 tx=0).stop 2024-03-24T00:48:01.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.327+0000 7f6b6e8a1640 1 -- 172.21.15.67:0/4157102128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b680b9af0 msgr2=0x7f6b680ccc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:01.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.327+0000 7f6b6e8a1640 1 --2- 172.21.15.67:0/4157102128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b680b9af0 0x7f6b680ccc40 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c00c440 tx=0x7f6b5c001ec0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.327+0000 7f6b6e8a1640 1 -- 172.21.15.67:0/4157102128 shutdown_connections 2024-03-24T00:48:01.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.327+0000 7f6b6e8a1640 1 --2- 172.21.15.67:0/4157102128 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6b5406bbd0 0x7f6b5406e090 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.327+0000 7f6b6e8a1640 1 --2- 172.21.15.67:0/4157102128 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6b6815e0f0 0x7f6b680cd180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.327+0000 7f6b6e8a1640 1 --2- 172.21.15.67:0/4157102128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b680b9af0 0x7f6b680ccc40 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.327+0000 7f6b6e8a1640 1 --2- 172.21.15.67:0/4157102128 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b680b6dc0 0x7f6b681f3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.327+0000 7f6b6e8a1640 1 -- 172.21.15.67:0/4157102128 >> 172.21.15.67:0/4157102128 conn(0x7f6b68072b60 msgr2=0x7f6b680b4b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:01.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.327+0000 7f6b6e8a1640 1 -- 172.21.15.67:0/4157102128 shutdown_connections 2024-03-24T00:48:01.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.327+0000 7f6b6e8a1640 1 -- 172.21.15.67:0/4157102128 wait complete. 2024-03-24T00:48:01.332 INFO:tasks.ceph.mds.c.smithi067.stderr: -5411> 2024-03-24T00:47:23.111+0000 7f41f367b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T00:48:01.341 DEBUG:teuthology.orchestra.run.smithi067:> cephfs-journal-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 --rank cephfs:0 event splice --inode=1099511627777 summary 2024-03-24T00:48:01.361 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-03-24T00:48:01.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.382+0000 7f571f20f280 1 -- start start 2024-03-24T00:48:01.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.382+0000 7f571f20f280 1 Processor -- start 2024-03-24T00:48:01.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.383+0000 7f571f20f280 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563e7af44c00 0x563e7afec580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:01.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.383+0000 7f571f20f280 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563e7af45800 0x563e7afecb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:01.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.383+0000 7f571f20f280 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e7af45c00 0x563e7afed080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:01.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.383+0000 7f571f20f280 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e7bd18600 con 0x563e7af45c00 2024-03-24T00:48:01.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.383+0000 7f571f20f280 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e7bd18480 con 0x563e7af45800 2024-03-24T00:48:01.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.383+0000 7f571f20f280 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x563e7bd18300 con 0x563e7af44c00 2024-03-24T00:48:01.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.383+0000 7f571d609640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e7af45c00 0x563e7afed080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:01.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.383+0000 7f571d609640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e7af45c00 0x563e7afed080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36128/0 (socket says 172.21.15.67:36128) 2024-03-24T00:48:01.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.383+0000 7f571d609640 1 -- 172.21.15.67:0/723879601 learned_addr learned my addr 172.21.15.67:0/723879601 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:01.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.383+0000 7f571ce08640 1 --2- 172.21.15.67:0/723879601 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563e7af44c00 0x563e7afec580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:01.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.383+0000 7f571d609640 1 -- 172.21.15.67:0/723879601 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563e7af44c00 msgr2=0x563e7afec580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:01.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.383+0000 7f571d609640 1 --2- 172.21.15.67:0/723879601 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563e7af44c00 0x563e7afec580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.383+0000 7f571c607640 1 --2- 172.21.15.67:0/723879601 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563e7af45800 0x563e7afecb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:01.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.383+0000 7f571d609640 1 -- 172.21.15.67:0/723879601 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563e7af45800 msgr2=0x563e7afecb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:01.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.383+0000 7f571d609640 1 --2- 172.21.15.67:0/723879601 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563e7af45800 0x563e7afecb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.383+0000 7f571d609640 1 -- 172.21.15.67:0/723879601 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563e7af7a780 con 0x563e7af45c00 2024-03-24T00:48:01.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.384+0000 7f571d609640 1 --2- 172.21.15.67:0/723879601 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e7af45c00 0x563e7afed080 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x563e7bd5f3e0 tx=0x563e7bcdf500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b7556ac5b47adf5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:01.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.384+0000 7f571a603640 1 -- 172.21.15.67:0/723879601 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x563e7af6a680 con 0x563e7af45c00 2024-03-24T00:48:01.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.384+0000 7f571a603640 1 -- 172.21.15.67:0/723879601 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x563e7af70a80 con 0x563e7af45c00 2024-03-24T00:48:01.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.384+0000 7f571a603640 1 -- 172.21.15.67:0/723879601 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x563e7af6ab60 con 0x563e7af45c00 2024-03-24T00:48:01.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.384+0000 7f571f20f280 1 -- 172.21.15.67:0/723879601 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563e7af7a3c0 con 0x563e7af45c00 2024-03-24T00:48:01.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.384+0000 7f571f20f280 1 -- 172.21.15.67:0/723879601 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x563e7af7b680 con 0x563e7af45c00 2024-03-24T00:48:01.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.384+0000 7f571f20f280 4 waiting for MDS map... 2024-03-24T00:48:01.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.384+0000 7f571a603640 1 -- 172.21.15.67:0/723879601 <== mon.0 v2:172.21.15.67:3300/0 4 ==== osd_map(108..108 src has 1..108) v4 ==== 6216+0+0 (secure 0 0 0) 0x563e7be08600 con 0x563e7af45c00 2024-03-24T00:48:01.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.384+0000 7f571a603640 1 -- 172.21.15.67:0/723879601 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 73) v1 ==== 2735+0+0 (secure 0 0 0) 0x563e7af9f180 con 0x563e7af45c00 2024-03-24T00:48:01.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.384+0000 7f571f20f280 4 Got MDS map 73 2024-03-24T00:48:01.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.384+0000 7f571f20f280 10 main: JournalTool::main 2024-03-24T00:48:01.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.384+0000 7f571f20f280 4 main: JournalTool: connecting to RADOS... 2024-03-24T00:48:01.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.385+0000 7f571f20f280 1 Processor -- start 2024-03-24T00:48:01.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.385+0000 7f571f20f280 1 -- start start 2024-03-24T00:48:01.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.385+0000 7f571f20f280 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563e7be0b400 0x563e7afed600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:01.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.385+0000 7f571f20f280 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563e7be0b800 0x563e7afedb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:01.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.385+0000 7f571f20f280 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e7be0bc00 0x563e7afee100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:01.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.385+0000 7f571f20f280 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e7bd18180 con 0x563e7be0bc00 2024-03-24T00:48:01.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.385+0000 7f571f20f280 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e7afa6480 con 0x563e7be0b800 2024-03-24T00:48:01.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.385+0000 7f571f20f280 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x563e7afa6900 con 0x563e7be0b400 2024-03-24T00:48:01.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.385+0000 7f571c607640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e7be0bc00 0x563e7afee100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:01.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.385+0000 7f571c607640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e7be0bc00 0x563e7afee100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36144/0 (socket says 172.21.15.67:36144) 2024-03-24T00:48:01.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.385+0000 7f571c607640 1 -- 172.21.15.67:0/1386182699 learned_addr learned my addr 172.21.15.67:0/1386182699 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:01.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.385+0000 7f571c607640 1 --2- 172.21.15.67:0/1386182699 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563e7be0b400 0x563e7afed600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:01.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.385+0000 7f571ce08640 1 --2- 172.21.15.67:0/1386182699 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563e7be0b800 0x563e7afedb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:01.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.386+0000 7f571c607640 1 -- 172.21.15.67:0/1386182699 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563e7be0b400 msgr2=0x563e7afed600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:01.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.386+0000 7f571c607640 1 --2- 172.21.15.67:0/1386182699 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563e7be0b400 0x563e7afed600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.386+0000 7f571c607640 1 -- 172.21.15.67:0/1386182699 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563e7be0b800 msgr2=0x563e7afedb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:01.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.386+0000 7f571c607640 1 --2- 172.21.15.67:0/1386182699 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563e7be0b800 0x563e7afedb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.386+0000 7f571c607640 1 -- 172.21.15.67:0/1386182699 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563e7af7b4a0 con 0x563e7be0bc00 2024-03-24T00:48:01.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.386+0000 7f571c607640 1 --2- 172.21.15.67:0/1386182699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e7be0bc00 0x563e7afee100 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x563e7be24690 tx=0x563e7be27980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3b379fea9523ac4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:01.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.386+0000 7f57185ff640 1 -- 172.21.15.67:0/1386182699 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x563e7af6a680 con 0x563e7be0bc00 2024-03-24T00:48:01.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.386+0000 7f57185ff640 1 -- 172.21.15.67:0/1386182699 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x563e7af70e00 con 0x563e7be0bc00 2024-03-24T00:48:01.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.386+0000 7f57185ff640 1 -- 172.21.15.67:0/1386182699 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x563e7af6ab60 con 0x563e7be0bc00 2024-03-24T00:48:01.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.386+0000 7f571f20f280 1 -- 172.21.15.67:0/1386182699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e7be0bc00 msgr2=0x563e7afee100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:01.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.386+0000 7f571f20f280 1 --2- 172.21.15.67:0/1386182699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e7be0bc00 0x563e7afee100 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x563e7be24690 tx=0x563e7be27980 comp rx=0 tx=0).stop 2024-03-24T00:48:01.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.386+0000 7f571f20f280 1 -- 172.21.15.67:0/1386182699 shutdown_connections 2024-03-24T00:48:01.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.386+0000 7f571f20f280 1 --2- 172.21.15.67:0/1386182699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e7be0bc00 0x563e7afee100 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.386+0000 7f571f20f280 1 --2- 172.21.15.67:0/1386182699 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563e7be0b800 0x563e7afedb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.386+0000 7f571f20f280 1 --2- 172.21.15.67:0/1386182699 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563e7be0b400 0x563e7afed600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.386+0000 7f571f20f280 1 -- 172.21.15.67:0/1386182699 >> 172.21.15.67:0/1386182699 conn(0x563e7be0b000 msgr2=0x563e7bcc6800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:01.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.386+0000 7f571f20f280 1 -- 172.21.15.67:0/1386182699 shutdown_connections 2024-03-24T00:48:01.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.386+0000 7f571f20f280 1 -- 172.21.15.67:0/1386182699 wait complete. 2024-03-24T00:48:01.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.387+0000 7f571f20f280 1 Processor -- start 2024-03-24T00:48:01.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.387+0000 7f571f20f280 1 -- start start 2024-03-24T00:48:01.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.387+0000 7f571f20f280 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563e7be0b400 0x563e7afef180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:01.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.387+0000 7f571f20f280 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563e7be0b800 0x563e7afeec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:01.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.387+0000 7f571f20f280 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e7be0bc00 0x563e7afee680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:01.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.387+0000 7f571f20f280 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e7bd19380 con 0x563e7be0bc00 2024-03-24T00:48:01.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.387+0000 7f571f20f280 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e7bd18180 con 0x563e7be0b800 2024-03-24T00:48:01.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.387+0000 7f571f20f280 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x563e7bd19500 con 0x563e7be0b400 2024-03-24T00:48:01.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.388+0000 7f571c607640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e7be0bc00 0x563e7afee680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:01.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.388+0000 7f571c607640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e7be0bc00 0x563e7afee680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36158/0 (socket says 172.21.15.67:36158) 2024-03-24T00:48:01.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.388+0000 7f571c607640 1 -- 172.21.15.67:0/736635304 learned_addr learned my addr 172.21.15.67:0/736635304 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:01.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.388+0000 7f571ce08640 1 --2- 172.21.15.67:0/736635304 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563e7be0b800 0x563e7afeec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:01.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.388+0000 7f571c607640 1 --2- 172.21.15.67:0/736635304 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563e7be0b400 0x563e7afef180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:01.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.388+0000 7f571c607640 1 -- 172.21.15.67:0/736635304 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563e7be0b400 msgr2=0x563e7afef180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:01.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.388+0000 7f571c607640 1 --2- 172.21.15.67:0/736635304 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563e7be0b400 0x563e7afef180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.388+0000 7f571c607640 1 -- 172.21.15.67:0/736635304 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563e7be0b800 msgr2=0x563e7afeec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:01.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.388+0000 7f571c607640 1 --2- 172.21.15.67:0/736635304 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563e7be0b800 0x563e7afeec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.388+0000 7f571c607640 1 -- 172.21.15.67:0/736635304 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563e7af7a780 con 0x563e7be0bc00 2024-03-24T00:48:01.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.388+0000 7f571c607640 1 --2- 172.21.15.67:0/736635304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e7be0bc00 0x563e7afee680 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x563e7be25350 tx=0x563e7be278c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:01.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.388+0000 7f5716dfc640 1 -- 172.21.15.67:0/736635304 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x563e7af6a4e0 con 0x563e7be0bc00 2024-03-24T00:48:01.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.388+0000 7f5716dfc640 1 -- 172.21.15.67:0/736635304 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x563e7af70c40 con 0x563e7be0bc00 2024-03-24T00:48:01.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.388+0000 7f571f20f280 1 -- 172.21.15.67:0/736635304 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x563e7af7a3c0 con 0x563e7be0bc00 2024-03-24T00:48:01.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.388+0000 7f5716dfc640 1 -- 172.21.15.67:0/736635304 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x563e7af6a680 con 0x563e7be0bc00 2024-03-24T00:48:01.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.388+0000 7f571f20f280 1 -- 172.21.15.67:0/736635304 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563e7af7b680 con 0x563e7be0bc00 2024-03-24T00:48:01.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.388+0000 7f571f20f280 4 main: JournalTool: resolving pool 15 2024-03-24T00:48:01.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.389+0000 7f5716dfc640 1 -- 172.21.15.67:0/736635304 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x563e7be92c00 con 0x563e7be0bc00 2024-03-24T00:48:01.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.390+0000 7f5716dfc640 1 --2- 172.21.15.67:0/736635304 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x563e7be8ac00 0x563e7afef700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:01.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.390+0000 7f5716dfc640 1 -- 172.21.15.67:0/736635304 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6216+0+0 (secure 0 0 0) 0x563e7be8ca00 con 0x563e7be0bc00 2024-03-24T00:48:01.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.390+0000 7f571f20f280 4 main: JournalTool: creating IoCtx.. 2024-03-24T00:48:01.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.390+0000 7f571f20f280 4 main: Executing for rank 0 2024-03-24T00:48:01.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.390+0000 7f571f20f280 4 Filtering by inode '1099511627777' 2024-03-24T00:48:01.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.390+0000 7f571f20f280 1 --2- 172.21.15.67:0/736635304 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x563e7be8b000 0x563e7afee100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:01.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.390+0000 7f571f20f280 1 -- 172.21.15.67:0/736635304 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:1 15.f 15:f1f69726:::400.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108) v9 -- 0x563e7be8b400 con 0x563e7be8b000 2024-03-24T00:48:01.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.390+0000 7f571ce08640 1 --2- 172.21.15.67:0/736635304 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x563e7be8ac00 0x563e7afef700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:01.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.390+0000 7f571c607640 1 --2- 172.21.15.67:0/736635304 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x563e7be8b000 0x563e7afee100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:01.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.390+0000 7f571ce08640 1 --2- 172.21.15.67:0/736635304 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x563e7be8ac00 0x563e7afef700 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x563e7afe7e30 tx=0x563e7bcdf2c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:01.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.390+0000 7f571c607640 1 --2- 172.21.15.67:0/736635304 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x563e7be8b000 0x563e7afee100 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:01.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.393+0000 7f571c607640 1 -- 172.21.15.67:0/736635304 <== osd.6 v2:172.21.15.72:6816/1725206562 1 ==== osd_op_reply(1 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+22 (crc 0 0 0) 0x563e7b03a000 con 0x563e7be8b000 2024-03-24T00:48:01.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.393+0000 7f571f20f280 4 Pointer 400.00000000 is readable 2024-03-24T00:48:01.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.393+0000 7f571f20f280 4 JournalScanner::scan: reading header object '200.00000000' 2024-03-24T00:48:01.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.393+0000 7f571f20f280 1 -- 172.21.15.67:0/736635304 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:2 15.14 15:292cf221:::200.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108) v9 -- 0x563e7be8b800 con 0x563e7be8b000 2024-03-24T00:48:01.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.394+0000 7f571c607640 1 -- 172.21.15.67:0/736635304 <== osd.6 v2:172.21.15.72:6816/1725206562 2 ==== osd_op_reply(2 200.00000000 [read 0~90 out=90b] v0'0 uv31 ondisk = 0) v8 ==== 156+0+90 (crc 0 0 0) 0x563e7b03a000 con 0x563e7be8b000 2024-03-24T00:48:01.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.394+0000 7f571f20f280 10 Header 0x400000 0x407d21 0x40809d 2024-03-24T00:48:01.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.394+0000 7f571f20f280 10 Starting journal scan from offset 0x407d21 2024-03-24T00:48:01.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.394+0000 7f571f20f280 1 -- 172.21.15.67:0/736635304 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:3 15.14 15:2e2fa760:::200.00000001:head [read 32033~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108) v9 -- 0x563e7be8bc00 con 0x563e7be8b000 2024-03-24T00:48:01.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.394+0000 7f571c607640 1 -- 172.21.15.67:0/736635304 <== osd.6 v2:172.21.15.72:6816/1725206562 3 ==== osd_op_reply(3 200.00000001 [read 32033~892 out=892b] v0'0 uv28 ondisk = 0) v8 ==== 156+0+892 (crc 0 0 0) 0x563e7b03a000 con 0x563e7be8b000 2024-03-24T00:48:01.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.394+0000 7f571f20f280 4 Read 0x37c bytes from 200.00000001 gap=0 2024-03-24T00:48:01.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.394+0000 7f571f20f280 10 Parsing data, 0x37c bytes available 2024-03-24T00:48:01.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.394+0000 7f571f20f280 10 Attempting decode at 0x407d21 2024-03-24T00:48:01.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.394+0000 7f571f20f280 10 Consumed 0x37c bytes 2024-03-24T00:48:01.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.394+0000 7f571f20f280 10 Valid entry at 0x407d21 2024-03-24T00:48:01.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.394+0000 7f571f20f280 1 --2- 172.21.15.67:0/736635304 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x563e7b2c8000 0x563e7afedb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:01.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.394+0000 7f571f20f280 1 -- 172.21.15.67:0/736635304 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:4 15.1b 15:ddb4e4d7:::200.00000002:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108) v9 -- 0x563e7b2c8400 con 0x563e7b2c8000 2024-03-24T00:48:01.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.395+0000 7f571d609640 1 --2- 172.21.15.67:0/736635304 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x563e7b2c8000 0x563e7afedb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:01.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.395+0000 7f571d609640 1 --2- 172.21.15.67:0/736635304 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x563e7b2c8000 0x563e7afedb80 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:01.400 INFO:teuthology.orchestra.run.smithi067.stdout:Events by type: 2024-03-24T00:48:01.400 INFO:teuthology.orchestra.run.smithi067.stdout:Errors: 0 2024-03-24T00:48:01.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.397+0000 7f571d609640 1 -- 172.21.15.67:0/736635304 <== osd.7 v2:172.21.15.72:6824/2893038746 1 ==== osd_op_reply(4 200.00000002 [read 0~2147483647] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x563e7b03a500 con 0x563e7b2c8000 2024-03-24T00:48:01.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.397+0000 7f571f20f280 4 Reached end of journal objects 2024-03-24T00:48:01.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.397+0000 7f571f20f280 4 Scanned objects, 0 missing, 1 valid 2024-03-24T00:48:01.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.397+0000 7f571f20f280 4 Events scanned, 0 gaps 2024-03-24T00:48:01.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.397+0000 7f571f20f280 4 Found 1 valid events 2024-03-24T00:48:01.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.397+0000 7f571f20f280 4 Selected 0 events events for processing 2024-03-24T00:48:01.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.397+0000 7f571f20f280 4 0 events 2024-03-24T00:48:01.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.397+0000 7f571f20f280 1 -- 172.21.15.67:0/736635304 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x563e7be8b000 msgr2=0x563e7afee100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:01.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.397+0000 7f571f20f280 1 --2- 172.21.15.67:0/736635304 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x563e7be8b000 0x563e7afee100 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.397+0000 7f571f20f280 1 -- 172.21.15.67:0/736635304 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x563e7b2c8000 msgr2=0x563e7afedb80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:01.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.397+0000 7f571f20f280 1 --2- 172.21.15.67:0/736635304 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x563e7b2c8000 0x563e7afedb80 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.397+0000 7f571f20f280 1 -- 172.21.15.67:0/736635304 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x563e7be8ac00 msgr2=0x563e7afef700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:01.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.397+0000 7f571f20f280 1 --2- 172.21.15.67:0/736635304 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x563e7be8ac00 0x563e7afef700 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x563e7afe7e30 tx=0x563e7bcdf2c0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.397+0000 7f571f20f280 1 -- 172.21.15.67:0/736635304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e7be0bc00 msgr2=0x563e7afee680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:01.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.397+0000 7f571f20f280 1 --2- 172.21.15.67:0/736635304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e7be0bc00 0x563e7afee680 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x563e7be25350 tx=0x563e7be278c0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.397+0000 7f571ce08640 1 -- 172.21.15.67:0/736635304 reap_dead start 2024-03-24T00:48:01.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.397+0000 7f571ce08640 1 -- 172.21.15.67:0/736635304 reap_dead start 2024-03-24T00:48:01.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.397+0000 7f571f20f280 1 -- 172.21.15.67:0/736635304 shutdown_connections 2024-03-24T00:48:01.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.397+0000 7f571f20f280 1 -- 172.21.15.67:0/736635304 >> 172.21.15.67:0/736635304 conn(0x563e7be0b000 msgr2=0x563e7bcc6800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:01.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.397+0000 7f571f20f280 1 -- 172.21.15.67:0/736635304 shutdown_connections 2024-03-24T00:48:01.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.397+0000 7f571f20f280 1 -- 172.21.15.67:0/736635304 wait complete. 2024-03-24T00:48:01.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.397+0000 7f571f20f280 1 -- 172.21.15.67:0/723879601 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e7af45c00 msgr2=0x563e7afed080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:01.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.397+0000 7f571f20f280 1 --2- 172.21.15.67:0/723879601 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e7af45c00 0x563e7afed080 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x563e7bd5f3e0 tx=0x563e7bcdf500 comp rx=0 tx=0).stop 2024-03-24T00:48:01.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.397+0000 7f571f20f280 1 -- 172.21.15.67:0/723879601 shutdown_connections 2024-03-24T00:48:01.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.397+0000 7f571f20f280 1 --2- 172.21.15.67:0/723879601 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e7af45c00 0x563e7afed080 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.397+0000 7f571f20f280 1 --2- 172.21.15.67:0/723879601 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563e7af45800 0x563e7afecb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.397+0000 7f571f20f280 1 --2- 172.21.15.67:0/723879601 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563e7af44c00 0x563e7afec580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.398+0000 7f571f20f280 1 -- 172.21.15.67:0/723879601 >> 172.21.15.67:0/723879601 conn(0x563e7af45000 msgr2=0x563e7bcc7000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:01.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.398+0000 7f571f20f280 1 -- 172.21.15.67:0/723879601 shutdown_connections 2024-03-24T00:48:01.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.398+0000 7f571f20f280 1 -- 172.21.15.67:0/723879601 wait complete. 2024-03-24T00:48:01.404 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-journal-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', '--rank', 'cephfs:0', 'event', 'splice', '--inode=1099511627777', 'summary'] in time 0:00:00.063518, result: Events by type: Errors: 0 2024-03-24T00:48:01.404 DEBUG:teuthology.orchestra.run.smithi067:> cephfs-journal-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 --rank cephfs:0 event splice --inode=1099511627778 summary 2024-03-24T00:48:01.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.483+0000 7f9327039280 1 -- start start 2024-03-24T00:48:01.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.483+0000 7f9327039280 1 Processor -- start 2024-03-24T00:48:01.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.484+0000 7f9327039280 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b58d518c00 0x55b58d5c0580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:01.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.484+0000 7f9327039280 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b58d519800 0x55b58d5c0b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:01.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.484+0000 7f9327039280 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b58d519c00 0x55b58d5c1080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:01.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.484+0000 7f9327039280 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b58e2f8180 con 0x55b58d519800 2024-03-24T00:48:01.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.484+0000 7f9327039280 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b58e2f8300 con 0x55b58d518c00 2024-03-24T00:48:01.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.484+0000 7f9327039280 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b58e2f8480 con 0x55b58d519c00 2024-03-24T00:48:01.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.484+0000 7f9324739640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b58d519800 0x55b58d5c0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:01.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.484+0000 7f9324739640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b58d519800 0x55b58d5c0b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36180/0 (socket says 172.21.15.67:36180) 2024-03-24T00:48:01.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.484+0000 7f9324739640 1 -- 172.21.15.67:0/3151851989 learned_addr learned my addr 172.21.15.67:0/3151851989 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:01.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.484+0000 7f932573b640 1 --2- 172.21.15.67:0/3151851989 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b58d519c00 0x55b58d5c1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:01.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.484+0000 7f9324f3a640 1 --2- 172.21.15.67:0/3151851989 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b58d518c00 0x55b58d5c0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:01.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.484+0000 7f9324739640 1 -- 172.21.15.67:0/3151851989 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b58d519c00 msgr2=0x55b58d5c1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:01.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.484+0000 7f9324739640 1 --2- 172.21.15.67:0/3151851989 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b58d519c00 0x55b58d5c1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.484+0000 7f9324739640 1 -- 172.21.15.67:0/3151851989 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b58d518c00 msgr2=0x55b58d5c0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:01.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.484+0000 7f9324739640 1 --2- 172.21.15.67:0/3151851989 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b58d518c00 0x55b58d5c0580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.484+0000 7f9324739640 1 -- 172.21.15.67:0/3151851989 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b58d54e780 con 0x55b58d519800 2024-03-24T00:48:01.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.485+0000 7f9324739640 1 --2- 172.21.15.67:0/3151851989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b58d519800 0x55b58d5c0b00 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x55b58e339440 tx=0x55b58e2b22a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14fe71f8c5dfb42a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:01.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.485+0000 7f9322735640 1 -- 172.21.15.67:0/3151851989 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55b58d53e680 con 0x55b58d519800 2024-03-24T00:48:01.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.485+0000 7f9322735640 1 -- 172.21.15.67:0/3151851989 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b58d544700 con 0x55b58d519800 2024-03-24T00:48:01.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.485+0000 7f9322735640 1 -- 172.21.15.67:0/3151851989 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55b58d53eb60 con 0x55b58d519800 2024-03-24T00:48:01.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.485+0000 7f9327039280 1 -- 172.21.15.67:0/3151851989 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b58d54f4a0 con 0x55b58d519800 2024-03-24T00:48:01.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.485+0000 7f9327039280 1 -- 172.21.15.67:0/3151851989 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x55b58d54e000 con 0x55b58d519800 2024-03-24T00:48:01.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.485+0000 7f9327039280 4 waiting for MDS map... 2024-03-24T00:48:01.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.485+0000 7f9322735640 1 -- 172.21.15.67:0/3151851989 <== mon.0 v2:172.21.15.67:3300/0 4 ==== osd_map(108..108 src has 1..108) v4 ==== 6216+0+0 (secure 0 0 0) 0x55b58e3faa00 con 0x55b58d519800 2024-03-24T00:48:01.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.485+0000 7f9322735640 1 -- 172.21.15.67:0/3151851989 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 73) v1 ==== 2735+0+0 (secure 0 0 0) 0x55b58d573180 con 0x55b58d519800 2024-03-24T00:48:01.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.485+0000 7f9327039280 4 Got MDS map 73 2024-03-24T00:48:01.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.485+0000 7f9327039280 10 main: JournalTool::main 2024-03-24T00:48:01.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.485+0000 7f9327039280 4 main: JournalTool: connecting to RADOS... 2024-03-24T00:48:01.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.486+0000 7f9327039280 1 Processor -- start 2024-03-24T00:48:01.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.486+0000 7f9327039280 1 -- start start 2024-03-24T00:48:01.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.486+0000 7f9327039280 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b58e3e3400 0x55b58d5c1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:01.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.486+0000 7f9327039280 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b58e3e3800 0x55b58d5c1b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:01.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.486+0000 7f9327039280 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b58e3e3c00 0x55b58d5c2100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:01.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.486+0000 7f9327039280 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b58e2f8600 con 0x55b58e3e3800 2024-03-24T00:48:01.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.486+0000 7f9327039280 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b58e2f8780 con 0x55b58e3e3c00 2024-03-24T00:48:01.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.486+0000 7f9327039280 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b58e2f8900 con 0x55b58e3e3400 2024-03-24T00:48:01.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.486+0000 7f9324f3a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b58e3e3800 0x55b58d5c1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:01.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.486+0000 7f9324f3a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b58e3e3800 0x55b58d5c1b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36182/0 (socket says 172.21.15.67:36182) 2024-03-24T00:48:01.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.486+0000 7f9324f3a640 1 -- 172.21.15.67:0/2990554287 learned_addr learned my addr 172.21.15.67:0/2990554287 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:01.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.486+0000 7f932573b640 1 --2- 172.21.15.67:0/2990554287 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b58e3e3400 0x55b58d5c1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:01.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.486+0000 7f9324739640 1 --2- 172.21.15.67:0/2990554287 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b58e3e3c00 0x55b58d5c2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:01.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.487+0000 7f9324f3a640 1 -- 172.21.15.67:0/2990554287 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b58e3e3400 msgr2=0x55b58d5c1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:01.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.487+0000 7f9324f3a640 1 --2- 172.21.15.67:0/2990554287 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b58e3e3400 0x55b58d5c1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.487+0000 7f9324f3a640 1 -- 172.21.15.67:0/2990554287 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b58e3e3c00 msgr2=0x55b58d5c2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:01.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.487+0000 7f9324f3a640 1 --2- 172.21.15.67:0/2990554287 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b58e3e3c00 0x55b58d5c2100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.487+0000 7f9324f3a640 1 -- 172.21.15.67:0/2990554287 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b58d54e3c0 con 0x55b58e3e3800 2024-03-24T00:48:01.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.487+0000 7f9324f3a640 1 --2- 172.21.15.67:0/2990554287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b58e3e3800 0x55b58d5c1b80 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x55b58e338a80 tx=0x55b58e2b0840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50edcdfd95279d63 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:01.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.487+0000 7f9320731640 1 -- 172.21.15.67:0/2990554287 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55b58d53e9c0 con 0x55b58e3e3800 2024-03-24T00:48:01.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.487+0000 7f9320731640 1 -- 172.21.15.67:0/2990554287 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b58d544fc0 con 0x55b58e3e3800 2024-03-24T00:48:01.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.487+0000 7f9320731640 1 -- 172.21.15.67:0/2990554287 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55b58d53e680 con 0x55b58e3e3800 2024-03-24T00:48:01.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.487+0000 7f9327039280 1 -- 172.21.15.67:0/2990554287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b58e3e3800 msgr2=0x55b58d5c1b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:01.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.487+0000 7f9327039280 1 --2- 172.21.15.67:0/2990554287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b58e3e3800 0x55b58d5c1b80 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x55b58e338a80 tx=0x55b58e2b0840 comp rx=0 tx=0).stop 2024-03-24T00:48:01.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.487+0000 7f9327039280 1 -- 172.21.15.67:0/2990554287 shutdown_connections 2024-03-24T00:48:01.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.487+0000 7f9327039280 1 --2- 172.21.15.67:0/2990554287 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b58e3e3c00 0x55b58d5c2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.487+0000 7f9327039280 1 --2- 172.21.15.67:0/2990554287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b58e3e3800 0x55b58d5c1b80 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.487+0000 7f9327039280 1 --2- 172.21.15.67:0/2990554287 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b58e3e3400 0x55b58d5c1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.487+0000 7f9327039280 1 -- 172.21.15.67:0/2990554287 >> 172.21.15.67:0/2990554287 conn(0x55b58e3e3000 msgr2=0x55b58d609000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:01.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.487+0000 7f9327039280 1 -- 172.21.15.67:0/2990554287 shutdown_connections 2024-03-24T00:48:01.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.487+0000 7f9327039280 1 -- 172.21.15.67:0/2990554287 wait complete. 2024-03-24T00:48:01.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.488+0000 7f9327039280 1 Processor -- start 2024-03-24T00:48:01.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.488+0000 7f9327039280 1 -- start start 2024-03-24T00:48:01.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.488+0000 7f9327039280 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b58e3e3400 0x55b58d5c3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:01.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.488+0000 7f9327039280 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b58e3e3800 0x55b58d5c2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:01.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.488+0000 7f9327039280 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b58e3e3c00 0x55b58d5c2680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:01.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.488+0000 7f9327039280 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b58e2f9980 con 0x55b58e3e3400 2024-03-24T00:48:01.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.488+0000 7f9327039280 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b58e2f9800 con 0x55b58e3e3c00 2024-03-24T00:48:01.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.488+0000 7f9327039280 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b58e2f9680 con 0x55b58e3e3800 2024-03-24T00:48:01.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.489+0000 7f932573b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b58e3e3400 0x55b58d5c3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:01.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.489+0000 7f932573b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b58e3e3400 0x55b58d5c3180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36188/0 (socket says 172.21.15.67:36188) 2024-03-24T00:48:01.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.489+0000 7f932573b640 1 -- 172.21.15.67:0/2751217784 learned_addr learned my addr 172.21.15.67:0/2751217784 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:01.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.489+0000 7f9324f3a640 1 --2- 172.21.15.67:0/2751217784 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b58e3e3800 0x55b58d5c2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:01.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.489+0000 7f9324739640 1 --2- 172.21.15.67:0/2751217784 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b58e3e3c00 0x55b58d5c2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:01.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.489+0000 7f932573b640 1 -- 172.21.15.67:0/2751217784 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b58e3e3800 msgr2=0x55b58d5c2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:01.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.489+0000 7f932573b640 1 --2- 172.21.15.67:0/2751217784 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b58e3e3800 0x55b58d5c2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.489+0000 7f932573b640 1 -- 172.21.15.67:0/2751217784 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b58e3e3c00 msgr2=0x55b58d5c2680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:01.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.489+0000 7f932573b640 1 --2- 172.21.15.67:0/2751217784 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b58e3e3c00 0x55b58d5c2680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.489+0000 7f932573b640 1 -- 172.21.15.67:0/2751217784 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b58d54e780 con 0x55b58e3e3400 2024-03-24T00:48:01.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.489+0000 7f9324739640 1 --2- 172.21.15.67:0/2751217784 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b58e3e3c00 0x55b58d5c2680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:48:01.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.489+0000 7f932573b640 1 --2- 172.21.15.67:0/2751217784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b58e3e3400 0x55b58d5c3180 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x55b58e3f9680 tx=0x55b58e405800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:01.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.489+0000 7f931ef2e640 1 -- 172.21.15.67:0/2751217784 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55b58d53e9c0 con 0x55b58e3e3400 2024-03-24T00:48:01.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.489+0000 7f931ef2e640 1 -- 172.21.15.67:0/2751217784 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b58d544e00 con 0x55b58e3e3400 2024-03-24T00:48:01.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.489+0000 7f9327039280 1 -- 172.21.15.67:0/2751217784 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55b58d54f4a0 con 0x55b58e3e3400 2024-03-24T00:48:01.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.489+0000 7f931ef2e640 1 -- 172.21.15.67:0/2751217784 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55b58d53e680 con 0x55b58e3e3400 2024-03-24T00:48:01.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.489+0000 7f9327039280 1 -- 172.21.15.67:0/2751217784 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b58d54e960 con 0x55b58e3e3400 2024-03-24T00:48:01.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.489+0000 7f9327039280 4 main: JournalTool: resolving pool 15 2024-03-24T00:48:01.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.490+0000 7f931ef2e640 1 -- 172.21.15.67:0/2751217784 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x55b58e3e6c00 con 0x55b58e3e3400 2024-03-24T00:48:01.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.491+0000 7f931ef2e640 1 --2- 172.21.15.67:0/2751217784 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x55b58e460c00 0x55b58d5c3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:01.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.491+0000 7f931ef2e640 1 -- 172.21.15.67:0/2751217784 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6216+0+0 (secure 0 0 0) 0x55b58e462a00 con 0x55b58e3e3400 2024-03-24T00:48:01.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.491+0000 7f9327039280 4 main: JournalTool: creating IoCtx.. 2024-03-24T00:48:01.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.491+0000 7f9327039280 4 main: Executing for rank 0 2024-03-24T00:48:01.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.491+0000 7f9327039280 4 Filtering by inode '1099511627778' 2024-03-24T00:48:01.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.491+0000 7f9327039280 1 --2- 172.21.15.67:0/2751217784 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x55b58e461000 0x55b58d5c2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:01.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.491+0000 7f9327039280 1 -- 172.21.15.67:0/2751217784 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:1 15.f 15:f1f69726:::400.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108) v9 -- 0x55b58e461400 con 0x55b58e461000 2024-03-24T00:48:01.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.491+0000 7f9324f3a640 1 --2- 172.21.15.67:0/2751217784 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x55b58e460c00 0x55b58d5c3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:01.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.491+0000 7f9324739640 1 --2- 172.21.15.67:0/2751217784 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x55b58e461000 0x55b58d5c2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:01.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.491+0000 7f9324739640 1 --2- 172.21.15.67:0/2751217784 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x55b58e461000 0x55b58d5c2100 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:01.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.491+0000 7f9324f3a640 1 --2- 172.21.15.67:0/2751217784 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x55b58e460c00 0x55b58d5c3700 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x55b58e338a80 tx=0x55b58e2b2240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:01.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.492+0000 7f9324739640 1 -- 172.21.15.67:0/2751217784 <== osd.6 v2:172.21.15.72:6816/1725206562 1 ==== osd_op_reply(1 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+22 (crc 0 0 0) 0x55b58d60e000 con 0x55b58e461000 2024-03-24T00:48:01.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.492+0000 7f9327039280 4 Pointer 400.00000000 is readable 2024-03-24T00:48:01.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.492+0000 7f9327039280 4 JournalScanner::scan: reading header object '200.00000000' 2024-03-24T00:48:01.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.492+0000 7f9327039280 1 -- 172.21.15.67:0/2751217784 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:2 15.14 15:292cf221:::200.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108) v9 -- 0x55b58e461800 con 0x55b58e461000 2024-03-24T00:48:01.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.492+0000 7f9324739640 1 -- 172.21.15.67:0/2751217784 <== osd.6 v2:172.21.15.72:6816/1725206562 2 ==== osd_op_reply(2 200.00000000 [read 0~90 out=90b] v0'0 uv31 ondisk = 0) v8 ==== 156+0+90 (crc 0 0 0) 0x55b58d60e000 con 0x55b58e461000 2024-03-24T00:48:01.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.492+0000 7f9327039280 10 Header 0x400000 0x407d21 0x40809d 2024-03-24T00:48:01.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.492+0000 7f9327039280 10 Starting journal scan from offset 0x407d21 2024-03-24T00:48:01.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.492+0000 7f9327039280 1 -- 172.21.15.67:0/2751217784 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:3 15.14 15:2e2fa760:::200.00000001:head [read 32033~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108) v9 -- 0x55b58e461c00 con 0x55b58e461000 2024-03-24T00:48:01.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.492+0000 7f9324739640 1 -- 172.21.15.67:0/2751217784 <== osd.6 v2:172.21.15.72:6816/1725206562 3 ==== osd_op_reply(3 200.00000001 [read 32033~892 out=892b] v0'0 uv28 ondisk = 0) v8 ==== 156+0+892 (crc 0 0 0) 0x55b58d60e000 con 0x55b58e461000 2024-03-24T00:48:01.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.492+0000 7f9327039280 4 Read 0x37c bytes from 200.00000001 gap=0 2024-03-24T00:48:01.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.492+0000 7f9327039280 10 Parsing data, 0x37c bytes available 2024-03-24T00:48:01.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.492+0000 7f9327039280 10 Attempting decode at 0x407d21 2024-03-24T00:48:01.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.492+0000 7f9327039280 10 Consumed 0x37c bytes 2024-03-24T00:48:01.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.492+0000 7f9327039280 10 Valid entry at 0x407d21 2024-03-24T00:48:01.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.492+0000 7f9327039280 1 --2- 172.21.15.67:0/2751217784 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x55b58d892000 0x55b58d5c1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:01.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.492+0000 7f9327039280 1 -- 172.21.15.67:0/2751217784 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:4 15.1b 15:ddb4e4d7:::200.00000002:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108) v9 -- 0x55b58d892400 con 0x55b58d892000 2024-03-24T00:48:01.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.493+0000 7f932573b640 1 --2- 172.21.15.67:0/2751217784 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x55b58d892000 0x55b58d5c1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:01.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.493+0000 7f932573b640 1 --2- 172.21.15.67:0/2751217784 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x55b58d892000 0x55b58d5c1b80 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:01.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.493+0000 7f932573b640 1 -- 172.21.15.67:0/2751217784 <== osd.7 v2:172.21.15.72:6824/2893038746 1 ==== osd_op_reply(4 200.00000002 [read 0~2147483647] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x55b58d60e500 con 0x55b58d892000 2024-03-24T00:48:01.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.493+0000 7f9327039280 4 Reached end of journal objects 2024-03-24T00:48:01.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.493+0000 7f9327039280 4 Scanned objects, 0 missing, 1 valid 2024-03-24T00:48:01.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.493+0000 7f9327039280 4 Events scanned, 0 gaps 2024-03-24T00:48:01.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.493+0000 7f9327039280 4 Found 1 valid events 2024-03-24T00:48:01.500 INFO:teuthology.orchestra.run.smithi067.stdout:Events by type: 2024-03-24T00:48:01.500 INFO:teuthology.orchestra.run.smithi067.stdout:Errors: 0 2024-03-24T00:48:01.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.493+0000 7f9327039280 4 Selected 0 events events for processing 2024-03-24T00:48:01.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.493+0000 7f9327039280 4 0 events 2024-03-24T00:48:01.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.493+0000 7f9327039280 1 -- 172.21.15.67:0/2751217784 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x55b58e461000 msgr2=0x55b58d5c2100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:01.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.493+0000 7f9327039280 1 --2- 172.21.15.67:0/2751217784 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x55b58e461000 0x55b58d5c2100 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.493+0000 7f9327039280 1 -- 172.21.15.67:0/2751217784 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x55b58d892000 msgr2=0x55b58d5c1b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:01.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.493+0000 7f9327039280 1 --2- 172.21.15.67:0/2751217784 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x55b58d892000 0x55b58d5c1b80 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.493+0000 7f9327039280 1 -- 172.21.15.67:0/2751217784 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x55b58e460c00 msgr2=0x55b58d5c3700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:01.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.493+0000 7f9327039280 1 --2- 172.21.15.67:0/2751217784 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x55b58e460c00 0x55b58d5c3700 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x55b58e338a80 tx=0x55b58e2b2240 comp rx=0 tx=0).stop 2024-03-24T00:48:01.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.493+0000 7f9327039280 1 -- 172.21.15.67:0/2751217784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b58e3e3400 msgr2=0x55b58d5c3180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:01.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.493+0000 7f9327039280 1 --2- 172.21.15.67:0/2751217784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b58e3e3400 0x55b58d5c3180 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x55b58e3f9680 tx=0x55b58e405800 comp rx=0 tx=0).stop 2024-03-24T00:48:01.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.493+0000 7f9324f3a640 1 -- 172.21.15.67:0/2751217784 reap_dead start 2024-03-24T00:48:01.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.493+0000 7f9324f3a640 1 -- 172.21.15.67:0/2751217784 reap_dead start 2024-03-24T00:48:01.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.493+0000 7f9327039280 1 -- 172.21.15.67:0/2751217784 shutdown_connections 2024-03-24T00:48:01.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.493+0000 7f9327039280 1 -- 172.21.15.67:0/2751217784 >> 172.21.15.67:0/2751217784 conn(0x55b58e3e3000 msgr2=0x55b58d609000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:01.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.493+0000 7f9327039280 1 -- 172.21.15.67:0/2751217784 shutdown_connections 2024-03-24T00:48:01.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.493+0000 7f9327039280 1 -- 172.21.15.67:0/2751217784 wait complete. 2024-03-24T00:48:01.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.494+0000 7f9327039280 1 -- 172.21.15.67:0/3151851989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b58d519800 msgr2=0x55b58d5c0b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:01.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.494+0000 7f9327039280 1 --2- 172.21.15.67:0/3151851989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b58d519800 0x55b58d5c0b00 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x55b58e339440 tx=0x55b58e2b22a0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.494+0000 7f9327039280 1 -- 172.21.15.67:0/3151851989 shutdown_connections 2024-03-24T00:48:01.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.494+0000 7f9327039280 1 --2- 172.21.15.67:0/3151851989 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55b58d519c00 0x55b58d5c1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.494+0000 7f9327039280 1 --2- 172.21.15.67:0/3151851989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b58d519800 0x55b58d5c0b00 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.494+0000 7f9327039280 1 --2- 172.21.15.67:0/3151851989 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b58d518c00 0x55b58d5c0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.494+0000 7f9327039280 1 -- 172.21.15.67:0/3151851989 >> 172.21.15.67:0/3151851989 conn(0x55b58d519000 msgr2=0x55b58d608800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:01.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.494+0000 7f9327039280 1 -- 172.21.15.67:0/3151851989 shutdown_connections 2024-03-24T00:48:01.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.494+0000 7f9327039280 1 -- 172.21.15.67:0/3151851989 wait complete. 2024-03-24T00:48:01.504 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-journal-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', '--rank', 'cephfs:0', 'event', 'splice', '--inode=1099511627778', 'summary'] in time 0:00:00.099806, result: Events by type: Errors: 0 2024-03-24T00:48:01.504 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata put mds0_inotable - 2024-03-24T00:48:01.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.584+0000 7f8f45f7df40 1 Processor -- start 2024-03-24T00:48:01.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.584+0000 7f8f45f7df40 1 -- start start 2024-03-24T00:48:01.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.585+0000 7f8f45f7df40 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55a2e49061c0 0x55a2e4919e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:01.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.585+0000 7f8f45f7df40 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a2e491ed90 0x55a2e491a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:01.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.585+0000 7f8f45f7df40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a2e49ab150 0x55a2e4a43d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:01.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.585+0000 7f8f45f7df40 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2e49ab780 con 0x55a2e491ed90 2024-03-24T00:48:01.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.585+0000 7f8f45f7df40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2e487d0d0 con 0x55a2e49ab150 2024-03-24T00:48:01.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.585+0000 7f8f45f7df40 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a2e491f510 con 0x55a2e49061c0 2024-03-24T00:48:01.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.585+0000 7f8f4327a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a2e491ed90 0x55a2e491a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:01.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.585+0000 7f8f4327a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a2e491ed90 0x55a2e491a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36216/0 (socket says 172.21.15.67:36216) 2024-03-24T00:48:01.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.585+0000 7f8f4327a640 1 -- 172.21.15.67:0/1093548917 learned_addr learned my addr 172.21.15.67:0/1093548917 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:01.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.585+0000 7f8f4427c640 1 --2- 172.21.15.67:0/1093548917 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a2e49ab150 0x55a2e4a43d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:01.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.585+0000 7f8f43a7b640 1 --2- 172.21.15.67:0/1093548917 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55a2e49061c0 0x55a2e4919e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:01.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.585+0000 7f8f4327a640 1 -- 172.21.15.67:0/1093548917 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55a2e49061c0 msgr2=0x55a2e4919e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:01.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.585+0000 7f8f4327a640 1 --2- 172.21.15.67:0/1093548917 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55a2e49061c0 0x55a2e4919e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.585+0000 7f8f4327a640 1 -- 172.21.15.67:0/1093548917 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a2e49ab150 msgr2=0x55a2e4a43d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:01.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.585+0000 7f8f4327a640 1 --2- 172.21.15.67:0/1093548917 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a2e49ab150 0x55a2e4a43d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.585+0000 7f8f4327a640 1 -- 172.21.15.67:0/1093548917 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a2e49aa6d0 con 0x55a2e491ed90 2024-03-24T00:48:01.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.585+0000 7f8f4427c640 1 --2- 172.21.15.67:0/1093548917 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a2e49ab150 0x55a2e4a43d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:48:01.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.585+0000 7f8f4327a640 1 --2- 172.21.15.67:0/1093548917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a2e491ed90 0x55a2e491a3b0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c002e10 tx=0x7f8f2c00b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34a2a58cf1bbbb0c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:01.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.585+0000 7f8f42278640 1 -- 172.21.15.67:0/1093548917 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f2c013020 con 0x55a2e491ed90 2024-03-24T00:48:01.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.585+0000 7f8f42278640 1 -- 172.21.15.67:0/1093548917 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f2c00f400 con 0x55a2e491ed90 2024-03-24T00:48:01.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.585+0000 7f8f42278640 1 -- 172.21.15.67:0/1093548917 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f2c00f7f0 con 0x55a2e491ed90 2024-03-24T00:48:01.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.586+0000 7f8f45f7df40 1 -- 172.21.15.67:0/1093548917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a2e491ed90 msgr2=0x55a2e491a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:01.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.586+0000 7f8f45f7df40 1 --2- 172.21.15.67:0/1093548917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a2e491ed90 0x55a2e491a3b0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c002e10 tx=0x7f8f2c00b5b0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.586+0000 7f8f45f7df40 1 -- 172.21.15.67:0/1093548917 shutdown_connections 2024-03-24T00:48:01.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.586+0000 7f8f45f7df40 1 --2- 172.21.15.67:0/1093548917 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a2e49ab150 0x55a2e4a43d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.586+0000 7f8f45f7df40 1 --2- 172.21.15.67:0/1093548917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a2e491ed90 0x55a2e491a3b0 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.586+0000 7f8f45f7df40 1 --2- 172.21.15.67:0/1093548917 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55a2e49061c0 0x55a2e4919e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.586+0000 7f8f45f7df40 1 -- 172.21.15.67:0/1093548917 >> 172.21.15.67:0/1093548917 conn(0x55a2e48d4e70 msgr2=0x55a2e48d9320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:01.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.586+0000 7f8f45f7df40 1 -- 172.21.15.67:0/1093548917 shutdown_connections 2024-03-24T00:48:01.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.586+0000 7f8f45f7df40 1 -- 172.21.15.67:0/1093548917 wait complete. 2024-03-24T00:48:01.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.587+0000 7f8f45f7df40 1 Processor -- start 2024-03-24T00:48:01.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.587+0000 7f8f45f7df40 1 -- start start 2024-03-24T00:48:01.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.587+0000 7f8f45f7df40 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55a2e49061c0 0x55a2e4988030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:01.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.587+0000 7f8f45f7df40 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a2e491ed90 0x55a2e4919080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:01.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.587+0000 7f8f45f7df40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a2e49ab150 0x55a2e498ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:01.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.587+0000 7f8f45f7df40 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2e49aa3f0 con 0x55a2e491ed90 2024-03-24T00:48:01.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.587+0000 7f8f45f7df40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2e498ab90 con 0x55a2e49ab150 2024-03-24T00:48:01.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.587+0000 7f8f45f7df40 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a2e491ae70 con 0x55a2e49061c0 2024-03-24T00:48:01.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.587+0000 7f8f4327a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a2e491ed90 0x55a2e4919080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:01.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.587+0000 7f8f4327a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a2e491ed90 0x55a2e4919080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36226/0 (socket says 172.21.15.67:36226) 2024-03-24T00:48:01.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.587+0000 7f8f4327a640 1 -- 172.21.15.67:0/2694241593 learned_addr learned my addr 172.21.15.67:0/2694241593 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:01.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.587+0000 7f8f43a7b640 1 --2- 172.21.15.67:0/2694241593 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55a2e49061c0 0x55a2e4988030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:01.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.587+0000 7f8f4427c640 1 --2- 172.21.15.67:0/2694241593 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a2e49ab150 0x55a2e498ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:01.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.588+0000 7f8f4327a640 1 -- 172.21.15.67:0/2694241593 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55a2e49061c0 msgr2=0x55a2e4988030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:01.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.588+0000 7f8f4327a640 1 --2- 172.21.15.67:0/2694241593 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55a2e49061c0 0x55a2e4988030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.588+0000 7f8f4327a640 1 -- 172.21.15.67:0/2694241593 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a2e49ab150 msgr2=0x55a2e498ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:01.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.588+0000 7f8f4327a640 1 --2- 172.21.15.67:0/2694241593 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a2e49ab150 0x55a2e498ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.588+0000 7f8f4327a640 1 -- 172.21.15.67:0/2694241593 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a2e49a4650 con 0x55a2e491ed90 2024-03-24T00:48:01.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.588+0000 7f8f4327a640 1 --2- 172.21.15.67:0/2694241593 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a2e491ed90 0x55a2e4919080 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c00b5f0 tx=0x7f8f2c001fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:01.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.588+0000 7f8f41276640 1 -- 172.21.15.67:0/2694241593 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f2c027070 con 0x55a2e491ed90 2024-03-24T00:48:01.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.588+0000 7f8f41276640 1 -- 172.21.15.67:0/2694241593 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f2c00fe20 con 0x55a2e491ed90 2024-03-24T00:48:01.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.588+0000 7f8f41276640 1 -- 172.21.15.67:0/2694241593 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f2c018890 con 0x55a2e491ed90 2024-03-24T00:48:01.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.588+0000 7f8f45f7df40 1 -- 172.21.15.67:0/2694241593 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55a2e49a4930 con 0x55a2e491ed90 2024-03-24T00:48:01.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.588+0000 7f8f45f7df40 1 -- 172.21.15.67:0/2694241593 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a2e49a4ef0 con 0x55a2e491ed90 2024-03-24T00:48:01.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.589+0000 7f8f41276640 1 -- 172.21.15.67:0/2694241593 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8f2c02a070 con 0x55a2e491ed90 2024-03-24T00:48:01.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.590+0000 7f8f41276640 1 --2- 172.21.15.67:0/2694241593 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8f2406a5b0 0x7f8f2406ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:01.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.590+0000 7f8f41276640 1 -- 172.21.15.67:0/2694241593 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f8f2c08a850 con 0x55a2e491ed90 2024-03-24T00:48:01.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.590+0000 7f8f43a7b640 1 --2- 172.21.15.67:0/2694241593 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8f2406a5b0 0x7f8f2406ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:01.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.590+0000 7f8f45f7df40 1 --2- 172.21.15.67:0/2694241593 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x55a2e499ab90 0x55a2e49a75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:01.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.590+0000 7f8f45f7df40 1 -- 172.21.15.67:0/2694241593 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:1 15.13 15:c91d4a1d:::mds0_inotable:head [writefull 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) v9 -- 0x55a2e49a7b00 con 0x55a2e499ab90 2024-03-24T00:48:01.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.590+0000 7f8f4427c640 1 --2- 172.21.15.67:0/2694241593 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x55a2e499ab90 0x55a2e49a75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:01.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.590+0000 7f8f4427c640 1 --2- 172.21.15.67:0/2694241593 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x55a2e499ab90 0x55a2e49a75c0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:01.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.590+0000 7f8f43a7b640 1 --2- 172.21.15.67:0/2694241593 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8f2406a5b0 0x7f8f2406ca70 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x55a2e4920460 tx=0x7f8f3c0059e0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:01.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.593+0000 7f8f4427c640 1 -- 172.21.15.67:0/2694241593 <== osd.1 v2:172.21.15.67:6800/4062510404 1 ==== osd_op_reply(1 mds0_inotable [writefull 0~34] v108'5 uv5 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x7f8f3405a040 con 0x55a2e499ab90 2024-03-24T00:48:01.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.593+0000 7f8f45f7df40 1 -- 172.21.15.67:0/2694241593 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x55a2e499ab90 msgr2=0x55a2e49a75c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:01.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.593+0000 7f8f45f7df40 1 --2- 172.21.15.67:0/2694241593 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x55a2e499ab90 0x55a2e49a75c0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.593+0000 7f8f45f7df40 1 -- 172.21.15.67:0/2694241593 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8f2406a5b0 msgr2=0x7f8f2406ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:01.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.593+0000 7f8f45f7df40 1 --2- 172.21.15.67:0/2694241593 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8f2406a5b0 0x7f8f2406ca70 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x55a2e4920460 tx=0x7f8f3c0059e0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.593+0000 7f8f45f7df40 1 -- 172.21.15.67:0/2694241593 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a2e491ed90 msgr2=0x55a2e4919080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:01.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.593+0000 7f8f45f7df40 1 --2- 172.21.15.67:0/2694241593 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a2e491ed90 0x55a2e4919080 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c00b5f0 tx=0x7f8f2c001fe0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.593+0000 7f8f4427c640 1 -- 172.21.15.67:0/2694241593 reap_dead start 2024-03-24T00:48:01.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.593+0000 7f8f45f7df40 1 -- 172.21.15.67:0/2694241593 shutdown_connections 2024-03-24T00:48:01.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.593+0000 7f8f45f7df40 1 -- 172.21.15.67:0/2694241593 >> 172.21.15.67:0/2694241593 conn(0x55a2e48d4e70 msgr2=0x55a2e4919c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:01.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.593+0000 7f8f45f7df40 1 -- 172.21.15.67:0/2694241593 shutdown_connections 2024-03-24T00:48:01.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.594+0000 7f8f45f7df40 1 -- 172.21.15.67:0/2694241593 wait complete. 2024-03-24T00:48:01.596 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-03-24T00:48:01.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.742+0000 7f87cb259640 1 Processor -- start 2024-03-24T00:48:01.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.742+0000 7f87cb259640 1 -- start start 2024-03-24T00:48:01.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.742+0000 7f87cb259640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f87c415de50 0x7f87c40b96d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:01.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.742+0000 7f87cb259640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f87c415e230 0x7f87c40cea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:01.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.742+0000 7f87cb259640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87c40d23d0 con 0x7f87c40d37b0 2024-03-24T00:48:01.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.742+0000 7f87cb259640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87c40b9c10 con 0x7f87c415de50 2024-03-24T00:48:01.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.742+0000 7f87cb259640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87c40d2180 con 0x7f87c415e230 2024-03-24T00:48:01.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.742+0000 7f87c8ff5640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f87c415de50 0x7f87c40b96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:01.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.743+0000 7f87c8ff5640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f87c415de50 0x7f87c40b96d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36234/0 (socket says 172.21.15.67:36234) 2024-03-24T00:48:01.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.743+0000 7f87c8ff5640 1 -- 172.21.15.67:0/2879869869 learned_addr learned my addr 172.21.15.67:0/2879869869 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:01.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.743+0000 7f87c3fff640 1 --2- 172.21.15.67:0/2879869869 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f87c415e230 0x7f87c40cea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:01.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.743+0000 7f87c8ff5640 1 -- 172.21.15.67:0/2879869869 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f87c415e230 msgr2=0x7f87c40cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:01.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.743+0000 7f87c8ff5640 1 --2- 172.21.15.67:0/2879869869 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f87c415e230 0x7f87c40cea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.743+0000 7f87c8ff5640 1 -- 172.21.15.67:0/2879869869 >> v1:172.21.15.72:6789/0 conn(0x7f87c40d37b0 legacy=0x7f87c40b57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:01.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.743+0000 7f87c8ff5640 1 -- 172.21.15.67:0/2879869869 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87c40d26b0 con 0x7f87c415de50 2024-03-24T00:48:01.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.743+0000 7f87c8ff5640 1 --2- 172.21.15.67:0/2879869869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87c415de50 0x7f87c40b96d0 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f87b8009870 tx=0x7f87b802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=472eab6d7537e5c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:01.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.744+0000 7f87c37fe640 1 -- 172.21.15.67:0/2879869869 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b8034070 con 0x7f87c415de50 2024-03-24T00:48:01.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.744+0000 7f87c37fe640 1 -- 172.21.15.67:0/2879869869 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87b8030840 con 0x7f87c415de50 2024-03-24T00:48:01.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.744+0000 7f87c37fe640 1 -- 172.21.15.67:0/2879869869 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b8030c10 con 0x7f87c415de50 2024-03-24T00:48:01.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.744+0000 7f87cb259640 1 -- 172.21.15.67:0/2879869869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87c415de50 msgr2=0x7f87c40b96d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:01.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.744+0000 7f87cb259640 1 --2- 172.21.15.67:0/2879869869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87c415de50 0x7f87c40b96d0 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f87b8009870 tx=0x7f87b802b980 comp rx=0 tx=0).stop 2024-03-24T00:48:01.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.744+0000 7f87cb259640 1 -- 172.21.15.67:0/2879869869 shutdown_connections 2024-03-24T00:48:01.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.744+0000 7f87cb259640 1 --2- 172.21.15.67:0/2879869869 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f87c415e230 0x7f87c40cea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.744+0000 7f87cb259640 1 --2- 172.21.15.67:0/2879869869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87c415de50 0x7f87c40b96d0 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.744+0000 7f87cb259640 1 -- 172.21.15.67:0/2879869869 >> 172.21.15.67:0/2879869869 conn(0x7f87c4072b60 msgr2=0x7f87c4072f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:01.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.744+0000 7f87cb259640 1 -- 172.21.15.67:0/2879869869 shutdown_connections 2024-03-24T00:48:01.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.744+0000 7f87cb259640 1 -- 172.21.15.67:0/2879869869 wait complete. 2024-03-24T00:48:01.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.745+0000 7f87cb259640 1 Processor -- start 2024-03-24T00:48:01.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.746+0000 7f87cb259640 1 -- start start 2024-03-24T00:48:01.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.746+0000 7f87cb259640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f87c40d37b0 0x7f87c40cc880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:01.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.746+0000 7f87cb259640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87c415de50 0x7f87c415f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:01.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.746+0000 7f87cb259640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87c415e230 0x7f87c41f2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:01.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.746+0000 7f87cb259640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c40d3190 con 0x7f87c415e230 2024-03-24T00:48:01.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.746+0000 7f87cb259640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c40d2c70 con 0x7f87c415de50 2024-03-24T00:48:01.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.746+0000 7f87cb259640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87c40d3310 con 0x7f87c40d37b0 2024-03-24T00:48:01.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.747+0000 7f87c97f6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87c415e230 0x7f87c41f2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:01.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.747+0000 7f87c97f6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87c415e230 0x7f87c41f2ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36242/0 (socket says 172.21.15.67:36242) 2024-03-24T00:48:01.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.747+0000 7f87c97f6640 1 -- 172.21.15.67:0/3628087538 learned_addr learned my addr 172.21.15.67:0/3628087538 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:01.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.747+0000 7f87c3fff640 1 --2- 172.21.15.67:0/3628087538 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87c415de50 0x7f87c415f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:01.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.747+0000 7f87c8ff5640 1 --2- 172.21.15.67:0/3628087538 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f87c40d37b0 0x7f87c40cc880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:01.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.747+0000 7f87c97f6640 1 -- 172.21.15.67:0/3628087538 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f87c40d37b0 msgr2=0x7f87c40cc880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:01.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.747+0000 7f87c97f6640 1 --2- 172.21.15.67:0/3628087538 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f87c40d37b0 0x7f87c40cc880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.747+0000 7f87c97f6640 1 -- 172.21.15.67:0/3628087538 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87c415de50 msgr2=0x7f87c415f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:01.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.747+0000 7f87c97f6640 1 --2- 172.21.15.67:0/3628087538 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87c415de50 0x7f87c415f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.747+0000 7f87c97f6640 1 -- 172.21.15.67:0/3628087538 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87c40cd950 con 0x7f87c415e230 2024-03-24T00:48:01.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.747+0000 7f87c97f6640 1 --2- 172.21.15.67:0/3628087538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87c415e230 0x7f87c41f2ed0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f87bc00c780 tx=0x7f87bc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:01.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.747+0000 7f87c17fa640 1 -- 172.21.15.67:0/3628087538 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87bc017070 con 0x7f87c415e230 2024-03-24T00:48:01.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.747+0000 7f87c17fa640 1 -- 172.21.15.67:0/3628087538 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87bc003da0 con 0x7f87c415e230 2024-03-24T00:48:01.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.747+0000 7f87cb259640 1 -- 172.21.15.67:0/3628087538 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87c40cd0b0 con 0x7f87c415e230 2024-03-24T00:48:01.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.747+0000 7f87c17fa640 1 -- 172.21.15.67:0/3628087538 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87bc0052b0 con 0x7f87c415e230 2024-03-24T00:48:01.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.747+0000 7f87cb259640 1 -- 172.21.15.67:0/3628087538 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87c40cd670 con 0x7f87c415e230 2024-03-24T00:48:01.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.748+0000 7f87cb259640 1 -- 172.21.15.67:0/3628087538 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87c40b3900 con 0x7f87c415e230 2024-03-24T00:48:01.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.749+0000 7f87c17fa640 1 -- 172.21.15.67:0/3628087538 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f87bc01a070 con 0x7f87c415e230 2024-03-24T00:48:01.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.749+0000 7f87c17fa640 1 --2- 172.21.15.67:0/3628087538 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f87a806bbd0 0x7f87a806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:01.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.749+0000 7f87c17fa640 1 -- 172.21.15.67:0/3628087538 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f87bc08a270 con 0x7f87c415e230 2024-03-24T00:48:01.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.749+0000 7f87c8ff5640 1 --2- 172.21.15.67:0/3628087538 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f87a806bbd0 0x7f87a806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:01.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.750+0000 7f87c8ff5640 1 --2- 172.21.15.67:0/3628087538 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f87a806bbd0 0x7f87a806e090 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f87b8036000 tx=0x7f87b803b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:01.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.753+0000 7f87c17fa640 1 -- 172.21.15.67:0/3628087538 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f87bc058bf0 con 0x7f87c415e230 2024-03-24T00:48:01.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:01.978+0000 7f87cb259640 1 -- 172.21.15.67:0/3628087538 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) v1 -- 0x7f87c404f400 con 0x7f87c415e230 2024-03-24T00:48:02.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.329+0000 7f87c17fa640 1 -- 172.21.15.67:0/3628087538 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v74) v1 ==== 162+0+0 (secure 0 0 0) 0x7f87bc0584f0 con 0x7f87c415e230 2024-03-24T00:48:02.329 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked joinable; MDS may join as newly active. 2024-03-24T00:48:02.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.331+0000 7f87cb259640 1 -- 172.21.15.67:0/3628087538 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f87a806bbd0 msgr2=0x7f87a806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:02.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.331+0000 7f87cb259640 1 --2- 172.21.15.67:0/3628087538 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f87a806bbd0 0x7f87a806e090 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f87b8036000 tx=0x7f87b803b040 comp rx=0 tx=0).stop 2024-03-24T00:48:02.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.331+0000 7f87cb259640 1 -- 172.21.15.67:0/3628087538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87c415e230 msgr2=0x7f87c41f2ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:02.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.331+0000 7f87cb259640 1 --2- 172.21.15.67:0/3628087538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87c415e230 0x7f87c41f2ed0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f87bc00c780 tx=0x7f87bc00cc40 comp rx=0 tx=0).stop 2024-03-24T00:48:02.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.332+0000 7f87cb259640 1 -- 172.21.15.67:0/3628087538 shutdown_connections 2024-03-24T00:48:02.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.332+0000 7f87cb259640 1 --2- 172.21.15.67:0/3628087538 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f87a806bbd0 0x7f87a806e090 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:02.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.332+0000 7f87cb259640 1 --2- 172.21.15.67:0/3628087538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87c415e230 0x7f87c41f2ed0 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:02.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.332+0000 7f87cb259640 1 --2- 172.21.15.67:0/3628087538 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87c415de50 0x7f87c415f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:02.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.332+0000 7f87cb259640 1 --2- 172.21.15.67:0/3628087538 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f87c40d37b0 0x7f87c40cc880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:02.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.332+0000 7f87cb259640 1 -- 172.21.15.67:0/3628087538 >> 172.21.15.67:0/3628087538 conn(0x7f87c4072b60 msgr2=0x7f87c40b6de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:02.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.332+0000 7f87cb259640 1 -- 172.21.15.67:0/3628087538 shutdown_connections 2024-03-24T00:48:02.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.332+0000 7f87cb259640 1 -- 172.21.15.67:0/3628087538 wait complete. 2024-03-24T00:48:02.345 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:48:02.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.447+0000 7f43931b9640 1 Processor -- start 2024-03-24T00:48:02.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.447+0000 7f43931b9640 1 -- start start 2024-03-24T00:48:02.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.447+0000 7f43931b9640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f438c072a70 0x7f438c165090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:02.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.448+0000 7f43931b9640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f438c0b4da0 con 0x7f438c15b430 2024-03-24T00:48:02.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.448+0000 7f43931b9640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f438c06db30 con 0x7f438c0b49c0 2024-03-24T00:48:02.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.448+0000 7f43931b9640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f438c06de10 con 0x7f438c072a70 2024-03-24T00:48:02.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.448+0000 7f438bfff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f438c15b430 0x7f438c072360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:51420/0 (socket says 172.21.15.67:51420) 2024-03-24T00:48:02.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.448+0000 7f438bfff640 1 -- 172.21.15.67:0/864199434 learned_addr learned my addr 172.21.15.67:0/864199434 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:48:02.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.448+0000 7f4391756640 1 --2- 172.21.15.67:0/864199434 >> v2:172.21.15.72:3300/0 conn(0x7f438c072a70 0x7f438c165090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:02.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.448+0000 7f438b7fe640 1 -- 172.21.15.67:0/864199434 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3199522202 0 0) 0x7f438c0b4da0 con 0x7f438c15b430 2024-03-24T00:48:02.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.448+0000 7f438b7fe640 1 -- 172.21.15.67:0/864199434 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f437c004630 con 0x7f438c15b430 2024-03-24T00:48:02.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.448+0000 7f438b7fe640 1 -- 172.21.15.67:0/864199434 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1505620659 0 0) 0x7f437c004630 con 0x7f438c15b430 2024-03-24T00:48:02.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.448+0000 7f438b7fe640 1 -- 172.21.15.67:0/864199434 >> v2:172.21.15.72:3300/0 conn(0x7f438c072a70 msgr2=0x7f438c165090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:02.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.448+0000 7f438b7fe640 1 --2- 172.21.15.67:0/864199434 >> v2:172.21.15.72:3300/0 conn(0x7f438c072a70 0x7f438c165090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:02.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.448+0000 7f438b7fe640 1 -- 172.21.15.67:0/864199434 >> v1:172.21.15.72:6789/0 conn(0x7f438c0b49c0 legacy=0x7f438c06f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:02.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.448+0000 7f438b7fe640 1 -- 172.21.15.67:0/864199434 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f438c06e0f0 con 0x7f438c15b430 2024-03-24T00:48:02.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.448+0000 7f438b7fe640 1 -- 172.21.15.67:0/864199434 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f4378001670 con 0x7f438c15b430 2024-03-24T00:48:02.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.448+0000 7f438b7fe640 1 -- 172.21.15.67:0/864199434 >> v1:172.21.15.67:6789/0 conn(0x7f438c15b430 legacy=0x7f438c072360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:02.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.448+0000 7f438b7fe640 1 --2- 172.21.15.67:0/864199434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f437c004ac0 0x7f437c004ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:02.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.448+0000 7f438b7fe640 1 --2- 172.21.15.67:0/864199434 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f437c005910 0x7f437c005d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:02.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.448+0000 7f438b7fe640 1 --2- 172.21.15.67:0/864199434 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f437c0062d0 0x7f437c00cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:02.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.448+0000 7f438b7fe640 1 -- 172.21.15.67:0/864199434 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f438c06de10 con 0x7f437c004ac0 2024-03-24T00:48:02.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.448+0000 7f438b7fe640 1 -- 172.21.15.67:0/864199434 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f437c00d270 con 0x7f437c0062d0 2024-03-24T00:48:02.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.448+0000 7f438b7fe640 1 -- 172.21.15.67:0/864199434 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f437c00d470 con 0x7f437c005910 2024-03-24T00:48:02.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.448+0000 7f4390f55640 1 --2- 172.21.15.67:0/864199434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f437c004ac0 0x7f437c004ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:02.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.449+0000 7f438bfff640 1 --2- 172.21.15.67:0/864199434 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f437c005910 0x7f437c005d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:02.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.449+0000 7f4391756640 1 --2- 172.21.15.67:0/864199434 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f437c0062d0 0x7f437c00cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:02.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.449+0000 7f4390f55640 1 -- 172.21.15.67:0/864199434 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f437c005910 msgr2=0x7f437c005d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:02.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.449+0000 7f4390f55640 1 --2- 172.21.15.67:0/864199434 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f437c005910 0x7f437c005d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:02.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.449+0000 7f4390f55640 1 -- 172.21.15.67:0/864199434 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f437c0062d0 msgr2=0x7f437c00cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:02.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.449+0000 7f4390f55640 1 --2- 172.21.15.67:0/864199434 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f437c0062d0 0x7f437c00cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:02.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.449+0000 7f4390f55640 1 -- 172.21.15.67:0/864199434 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f437c004630 con 0x7f437c004ac0 2024-03-24T00:48:02.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.449+0000 7f4391756640 1 -- 172.21.15.67:0/864199434 reap_dead start 2024-03-24T00:48:02.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.449+0000 7f4390f55640 1 --2- 172.21.15.67:0/864199434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f437c004ac0 0x7f437c004ee0 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f4380001750 tx=0x7f437c006940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c17880c596bda088 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:02.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.450+0000 7f438b7fe640 1 -- 172.21.15.67:0/864199434 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4380001370 con 0x7f437c004ac0 2024-03-24T00:48:02.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.450+0000 7f438b7fe640 1 -- 172.21.15.67:0/864199434 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43800052d0 con 0x7f437c004ac0 2024-03-24T00:48:02.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.450+0000 7f438b7fe640 1 -- 172.21.15.67:0/864199434 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4380003830 con 0x7f437c004ac0 2024-03-24T00:48:02.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.450+0000 7f43931b9640 1 -- 172.21.15.67:0/864199434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f437c004ac0 msgr2=0x7f437c004ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:02.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.450+0000 7f43931b9640 1 --2- 172.21.15.67:0/864199434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f437c004ac0 0x7f437c004ee0 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f4380001750 tx=0x7f437c006940 comp rx=0 tx=0).stop 2024-03-24T00:48:02.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.450+0000 7f43931b9640 1 -- 172.21.15.67:0/864199434 shutdown_connections 2024-03-24T00:48:02.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.450+0000 7f43931b9640 1 --2- 172.21.15.67:0/864199434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f437c004ac0 0x7f437c004ee0 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:02.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.450+0000 7f43931b9640 1 -- 172.21.15.67:0/864199434 >> 172.21.15.67:0/864199434 conn(0x7f438c0cf9e0 msgr2=0x7f438c0cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:02.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.450+0000 7f43931b9640 1 -- 172.21.15.67:0/864199434 shutdown_connections 2024-03-24T00:48:02.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.450+0000 7f43931b9640 1 -- 172.21.15.67:0/864199434 wait complete. 2024-03-24T00:48:02.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.451+0000 7f43931b9640 1 Processor -- start 2024-03-24T00:48:02.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.451+0000 7f43931b9640 1 -- start start 2024-03-24T00:48:02.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.452+0000 7f43931b9640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f437c004ac0 0x7f438c145280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:02.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.452+0000 7f43931b9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f438c148430 0x7f438c15caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:02.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.452+0000 7f43931b9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f438c15d030 0x7f438c1f2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:02.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.452+0000 7f43931b9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f438c0b3d30 con 0x7f438c15d030 2024-03-24T00:48:02.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.452+0000 7f43931b9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f438c06e6b0 con 0x7f438c148430 2024-03-24T00:48:02.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.452+0000 7f43931b9640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f438c0b3eb0 con 0x7f437c004ac0 2024-03-24T00:48:02.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.452+0000 7f4391756640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f438c15d030 0x7f438c1f2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:02.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.452+0000 7f4391756640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f438c15d030 0x7f438c1f2490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36262/0 (socket says 172.21.15.67:36262) 2024-03-24T00:48:02.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.452+0000 7f4391756640 1 -- 172.21.15.67:0/932556641 learned_addr learned my addr 172.21.15.67:0/932556641 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:02.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.452+0000 7f4390f55640 1 --2- 172.21.15.67:0/932556641 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f437c004ac0 0x7f438c145280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:02.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.452+0000 7f438bfff640 1 --2- 172.21.15.67:0/932556641 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f438c148430 0x7f438c15caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:02.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.453+0000 7f4391756640 1 -- 172.21.15.67:0/932556641 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f437c004ac0 msgr2=0x7f438c145280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:02.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.453+0000 7f4391756640 1 --2- 172.21.15.67:0/932556641 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f437c004ac0 0x7f438c145280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:02.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.453+0000 7f4391756640 1 -- 172.21.15.67:0/932556641 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f438c148430 msgr2=0x7f438c15caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:02.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.453+0000 7f4391756640 1 --2- 172.21.15.67:0/932556641 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f438c148430 0x7f438c15caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:02.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.453+0000 7f4391756640 1 -- 172.21.15.67:0/932556641 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f438c0ae430 con 0x7f438c15d030 2024-03-24T00:48:02.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.453+0000 7f4391756640 1 --2- 172.21.15.67:0/932556641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f438c15d030 0x7f438c1f2490 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f438c0cd030 tx=0x7f438400f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:02.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.453+0000 7f43897fa640 1 -- 172.21.15.67:0/932556641 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f438400fd80 con 0x7f438c15d030 2024-03-24T00:48:02.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.453+0000 7f43897fa640 1 -- 172.21.15.67:0/932556641 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f438401d070 con 0x7f438c15d030 2024-03-24T00:48:02.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.453+0000 7f43931b9640 1 -- 172.21.15.67:0/932556641 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f438c0ae710 con 0x7f438c15d030 2024-03-24T00:48:02.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.453+0000 7f43897fa640 1 -- 172.21.15.67:0/932556641 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4384015670 con 0x7f438c15d030 2024-03-24T00:48:02.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.453+0000 7f43931b9640 1 -- 172.21.15.67:0/932556641 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f438c0aecd0 con 0x7f438c15d030 2024-03-24T00:48:02.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.454+0000 7f43931b9640 1 -- 172.21.15.67:0/932556641 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f438c04f400 con 0x7f438c15d030 2024-03-24T00:48:02.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.455+0000 7f43897fa640 1 -- 172.21.15.67:0/932556641 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4384015810 con 0x7f438c15d030 2024-03-24T00:48:02.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.455+0000 7f43897fa640 1 --2- 172.21.15.67:0/932556641 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f437006bbd0 0x7f437006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:02.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.455+0000 7f43897fa640 1 -- 172.21.15.67:0/932556641 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f438408b570 con 0x7f438c15d030 2024-03-24T00:48:02.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.455+0000 7f4390f55640 1 --2- 172.21.15.67:0/932556641 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f437006bbd0 0x7f437006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:02.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.456+0000 7f4390f55640 1 --2- 172.21.15.67:0/932556641 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f437006bbd0 0x7f437006e090 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f4380033000 tx=0x7f4380009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:02.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.460+0000 7f43897fa640 1 -- 172.21.15.67:0/932556641 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4384027020 con 0x7f438c15d030 2024-03-24T00:48:02.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.682+0000 7f43931b9640 1 -- 172.21.15.67:0/932556641 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f438c15c5d0 con 0x7f438c15d030 2024-03-24T00:48:02.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.683+0000 7f43897fa640 1 -- 172.21.15.67:0/932556641 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+4971 (secure 0 0 0) 0x7f4384010040 con 0x7f438c15d030 2024-03-24T00:48:02.683 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:48:02.683 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":75,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6902,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3672294289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3672294289},{"type":"v1","addr":"172.21.15.67:6835","nonce":3672294289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":6908,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/1286271878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":1286271878},{"type":"v1","addr":"172.21.15.72:6836","nonce":1286271878}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":6911,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/140622129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":140622129},{"type":"v1","addr":"172.21.15.72:6837","nonce":140622129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:47:16.492485+0000","modified":"2024-03-24T00:48:02.331579+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":108,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8895},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8895":{"gid":8895,"name":"c","rank":0,"incarnation":75,"state":"up:replay","state_seq":1,"addr":"172.21.15.67:6833/3197261495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3197261495},{"type":"v1","addr":"172.21.15.67:6833","nonce":3197261495}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-03-24T00:48:02.683 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 75 2024-03-24T00:48:02.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.684+0000 7f43931b9640 1 -- 172.21.15.67:0/932556641 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f437006bbd0 msgr2=0x7f437006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:02.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.684+0000 7f43931b9640 1 --2- 172.21.15.67:0/932556641 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f437006bbd0 0x7f437006e090 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f4380033000 tx=0x7f4380009030 comp rx=0 tx=0).stop 2024-03-24T00:48:02.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.684+0000 7f43931b9640 1 -- 172.21.15.67:0/932556641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f438c15d030 msgr2=0x7f438c1f2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:02.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.684+0000 7f43931b9640 1 --2- 172.21.15.67:0/932556641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f438c15d030 0x7f438c1f2490 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f438c0cd030 tx=0x7f438400f580 comp rx=0 tx=0).stop 2024-03-24T00:48:02.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.685+0000 7f43931b9640 1 -- 172.21.15.67:0/932556641 shutdown_connections 2024-03-24T00:48:02.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.685+0000 7f43931b9640 1 --2- 172.21.15.67:0/932556641 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f437006bbd0 0x7f437006e090 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:02.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.685+0000 7f43931b9640 1 --2- 172.21.15.67:0/932556641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f438c15d030 0x7f438c1f2490 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:02.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.685+0000 7f43931b9640 1 --2- 172.21.15.67:0/932556641 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f438c148430 0x7f438c15caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:02.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.685+0000 7f43931b9640 1 --2- 172.21.15.67:0/932556641 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f437c004ac0 0x7f438c145280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:02.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.685+0000 7f43931b9640 1 -- 172.21.15.67:0/932556641 >> 172.21.15.67:0/932556641 conn(0x7f438c0cf9e0 msgr2=0x7f438c154550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:02.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.685+0000 7f43931b9640 1 -- 172.21.15.67:0/932556641 shutdown_connections 2024-03-24T00:48:02.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:02.685+0000 7f43931b9640 1 -- 172.21.15.67:0/932556641 wait complete. 2024-03-24T00:48:03.097 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 75, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:47:16.492485+0000', 'modified': '2024-03-24T00:48:02.331579+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 108, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8895}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8895': {'gid': 8895, 'name': 'c', 'rank': 0, 'incarnation': 75, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.67:6833/3197261495', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6832', 'nonce': 3197261495}, {'type': 'v1', 'addr': '172.21.15.67:6833', 'nonce': 3197261495}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-24T00:48:03.097 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8895:up:replay 2024-03-24T00:48:04.099 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:48:04.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.213+0000 7f4c9f5f3640 1 Processor -- start 2024-03-24T00:48:04.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.214+0000 7f4c9f5f3640 1 -- start start 2024-03-24T00:48:04.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.214+0000 7f4c9f5f3640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4c9815e2a0 0x7f4c980ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:04.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.214+0000 7f4c9f5f3640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4c980d0ef0 0x7f4c98162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:04.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.214+0000 7f4c9f5f3640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c980cf640 con 0x7f4c9815dec0 2024-03-24T00:48:04.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.214+0000 7f4c9f5f3640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c980cee30 con 0x7f4c9815e2a0 2024-03-24T00:48:04.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.214+0000 7f4c9f5f3640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c980d2120 con 0x7f4c980d0ef0 2024-03-24T00:48:04.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.214+0000 7f4c9cb8e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4c9815e2a0 0x7f4c980ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:04.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.214+0000 7f4c9cb8e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4c9815e2a0 0x7f4c980ce630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41016/0 (socket says 172.21.15.67:41016) 2024-03-24T00:48:04.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.214+0000 7f4c9cb8e640 1 -- 172.21.15.67:0/1241142771 learned_addr learned my addr 172.21.15.67:0/1241142771 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:04.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.215+0000 7f4c9db90640 1 --2- 172.21.15.67:0/1241142771 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4c980d0ef0 0x7f4c98162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:04.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.215+0000 7f4c9cb8e640 1 -- 172.21.15.67:0/1241142771 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4c980d0ef0 msgr2=0x7f4c98162630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:04.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.215+0000 7f4c9cb8e640 1 --2- 172.21.15.67:0/1241142771 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4c980d0ef0 0x7f4c98162630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:04.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.215+0000 7f4c9cb8e640 1 -- 172.21.15.67:0/1241142771 >> v1:172.21.15.72:6789/0 conn(0x7f4c9815dec0 legacy=0x7f4c980d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:04.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.215+0000 7f4c9cb8e640 1 -- 172.21.15.67:0/1241142771 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c980cf080 con 0x7f4c9815e2a0 2024-03-24T00:48:04.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.216+0000 7f4c9cb8e640 1 --2- 172.21.15.67:0/1241142771 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c9815e2a0 0x7f4c980ce630 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f4c88009870 tx=0x7f4c8802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1774e23f97c17b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:04.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.216+0000 7f4c8ffff640 1 -- 172.21.15.67:0/1241142771 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c88035020 con 0x7f4c9815e2a0 2024-03-24T00:48:04.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.216+0000 7f4c8ffff640 1 -- 172.21.15.67:0/1241142771 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c88031640 con 0x7f4c9815e2a0 2024-03-24T00:48:04.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.216+0000 7f4c8ffff640 1 -- 172.21.15.67:0/1241142771 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c8802d040 con 0x7f4c9815e2a0 2024-03-24T00:48:04.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.216+0000 7f4c9f5f3640 1 -- 172.21.15.67:0/1241142771 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c9815e2a0 msgr2=0x7f4c980ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:04.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.216+0000 7f4c9f5f3640 1 --2- 172.21.15.67:0/1241142771 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c9815e2a0 0x7f4c980ce630 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f4c88009870 tx=0x7f4c8802b8f0 comp rx=0 tx=0).stop 2024-03-24T00:48:04.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.216+0000 7f4c9f5f3640 1 -- 172.21.15.67:0/1241142771 shutdown_connections 2024-03-24T00:48:04.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.216+0000 7f4c9f5f3640 1 --2- 172.21.15.67:0/1241142771 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4c980d0ef0 0x7f4c98162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:04.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.216+0000 7f4c9f5f3640 1 --2- 172.21.15.67:0/1241142771 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c9815e2a0 0x7f4c980ce630 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:04.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.216+0000 7f4c9f5f3640 1 -- 172.21.15.67:0/1241142771 >> 172.21.15.67:0/1241142771 conn(0x7f4c9806c810 msgr2=0x7f4c9806cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:04.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.216+0000 7f4c9f5f3640 1 -- 172.21.15.67:0/1241142771 shutdown_connections 2024-03-24T00:48:04.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.216+0000 7f4c9f5f3640 1 -- 172.21.15.67:0/1241142771 wait complete. 2024-03-24T00:48:04.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.218+0000 7f4c9f5f3640 1 Processor -- start 2024-03-24T00:48:04.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.218+0000 7f4c9f5f3640 1 -- start start 2024-03-24T00:48:04.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.218+0000 7f4c9f5f3640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4c980d0ef0 0x7f4c9815b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:04.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.218+0000 7f4c9f5f3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c9815dec0 0x7f4c980caea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:04.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.218+0000 7f4c9f5f3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c9815e2a0 0x7f4c9815fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:04.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.218+0000 7f4c9f5f3640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c980d2730 con 0x7f4c9815e2a0 2024-03-24T00:48:04.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.218+0000 7f4c9f5f3640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c980d25b0 con 0x7f4c9815dec0 2024-03-24T00:48:04.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.218+0000 7f4c9f5f3640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c980d28b0 con 0x7f4c980d0ef0 2024-03-24T00:48:04.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.218+0000 7f4c9db90640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c9815e2a0 0x7f4c9815fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:04.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.218+0000 7f4c9db90640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c9815e2a0 0x7f4c9815fed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41024/0 (socket says 172.21.15.67:41024) 2024-03-24T00:48:04.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.218+0000 7f4c9db90640 1 -- 172.21.15.67:0/1274455811 learned_addr learned my addr 172.21.15.67:0/1274455811 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:04.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.218+0000 7f4c9d38f640 1 --2- 172.21.15.67:0/1274455811 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4c980d0ef0 0x7f4c9815b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:04.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.219+0000 7f4c9cb8e640 1 --2- 172.21.15.67:0/1274455811 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c9815dec0 0x7f4c980caea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:04.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.219+0000 7f4c9db90640 1 -- 172.21.15.67:0/1274455811 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4c980d0ef0 msgr2=0x7f4c9815b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:04.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.219+0000 7f4c9db90640 1 --2- 172.21.15.67:0/1274455811 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4c980d0ef0 0x7f4c9815b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:04.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.219+0000 7f4c9db90640 1 -- 172.21.15.67:0/1274455811 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c9815dec0 msgr2=0x7f4c980caea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:04.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.219+0000 7f4c9db90640 1 --2- 172.21.15.67:0/1274455811 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c9815dec0 0x7f4c980caea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:04.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.219+0000 7f4c9db90640 1 -- 172.21.15.67:0/1274455811 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c980cb6d0 con 0x7f4c9815e2a0 2024-03-24T00:48:04.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.219+0000 7f4c9db90640 1 --2- 172.21.15.67:0/1274455811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c9815e2a0 0x7f4c9815fed0 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f4c90007e60 tx=0x7f4c9000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:04.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.219+0000 7f4c8dffb640 1 -- 172.21.15.67:0/1274455811 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c90014620 con 0x7f4c9815e2a0 2024-03-24T00:48:04.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.219+0000 7f4c8dffb640 1 -- 172.21.15.67:0/1274455811 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c90019070 con 0x7f4c9815e2a0 2024-03-24T00:48:04.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.219+0000 7f4c9f5f3640 1 -- 172.21.15.67:0/1274455811 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c980cb9b0 con 0x7f4c9815e2a0 2024-03-24T00:48:04.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.219+0000 7f4c8dffb640 1 -- 172.21.15.67:0/1274455811 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c90014a00 con 0x7f4c9815e2a0 2024-03-24T00:48:04.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.219+0000 7f4c9f5f3640 1 -- 172.21.15.67:0/1274455811 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c980cc4b0 con 0x7f4c9815e2a0 2024-03-24T00:48:04.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.220+0000 7f4c9f5f3640 1 -- 172.21.15.67:0/1274455811 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c9804f400 con 0x7f4c9815e2a0 2024-03-24T00:48:04.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.221+0000 7f4c8dffb640 1 -- 172.21.15.67:0/1274455811 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4c90014ba0 con 0x7f4c9815e2a0 2024-03-24T00:48:04.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.221+0000 7f4c8dffb640 1 --2- 172.21.15.67:0/1274455811 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4c7c06bbd0 0x7f4c7c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:04.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.222+0000 7f4c8dffb640 1 -- 172.21.15.67:0/1274455811 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f4c9008b370 con 0x7f4c9815e2a0 2024-03-24T00:48:04.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.222+0000 7f4c9d38f640 1 --2- 172.21.15.67:0/1274455811 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4c7c06bbd0 0x7f4c7c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:04.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.222+0000 7f4c9d38f640 1 --2- 172.21.15.67:0/1274455811 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4c7c06bbd0 0x7f4c7c06e090 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f4c94001780 tx=0x7f4c94009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:04.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.226+0000 7f4c8dffb640 1 -- 172.21.15.67:0/1274455811 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4c900596a0 con 0x7f4c9815e2a0 2024-03-24T00:48:04.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.447+0000 7f4c9f5f3640 1 -- 172.21.15.67:0/1274455811 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4c980d1aa0 con 0x7f4c9815e2a0 2024-03-24T00:48:04.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.448+0000 7f4c8dffb640 1 -- 172.21.15.67:0/1274455811 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+4971 (secure 0 0 0) 0x7f4c9005d740 con 0x7f4c9815e2a0 2024-03-24T00:48:04.448 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:48:04.448 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":77,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6902,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3672294289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3672294289},{"type":"v1","addr":"172.21.15.67:6835","nonce":3672294289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":6908,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/1286271878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":1286271878},{"type":"v1","addr":"172.21.15.72:6836","nonce":1286271878}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":6911,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/140622129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":140622129},{"type":"v1","addr":"172.21.15.72:6837","nonce":140622129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:47:16.492485+0000","modified":"2024-03-24T00:48:03.337083+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":108,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8895},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8895":{"gid":8895,"name":"c","rank":0,"incarnation":75,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.67:6833/3197261495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3197261495},{"type":"v1","addr":"172.21.15.67:6833","nonce":3197261495}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-03-24T00:48:04.448 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 77 2024-03-24T00:48:04.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.450+0000 7f4c9f5f3640 1 -- 172.21.15.67:0/1274455811 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4c7c06bbd0 msgr2=0x7f4c7c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:04.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.450+0000 7f4c9f5f3640 1 --2- 172.21.15.67:0/1274455811 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4c7c06bbd0 0x7f4c7c06e090 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f4c94001780 tx=0x7f4c94009040 comp rx=0 tx=0).stop 2024-03-24T00:48:04.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.450+0000 7f4c9f5f3640 1 -- 172.21.15.67:0/1274455811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c9815e2a0 msgr2=0x7f4c9815fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:04.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.450+0000 7f4c9f5f3640 1 --2- 172.21.15.67:0/1274455811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c9815e2a0 0x7f4c9815fed0 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f4c90007e60 tx=0x7f4c9000e4b0 comp rx=0 tx=0).stop 2024-03-24T00:48:04.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.450+0000 7f4c9f5f3640 1 -- 172.21.15.67:0/1274455811 shutdown_connections 2024-03-24T00:48:04.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.450+0000 7f4c9f5f3640 1 --2- 172.21.15.67:0/1274455811 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4c7c06bbd0 0x7f4c7c06e090 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:04.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.450+0000 7f4c9f5f3640 1 --2- 172.21.15.67:0/1274455811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c9815e2a0 0x7f4c9815fed0 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:04.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.450+0000 7f4c9f5f3640 1 --2- 172.21.15.67:0/1274455811 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c9815dec0 0x7f4c980caea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:04.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.450+0000 7f4c9f5f3640 1 --2- 172.21.15.67:0/1274455811 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4c980d0ef0 0x7f4c9815b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:04.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.450+0000 7f4c9f5f3640 1 -- 172.21.15.67:0/1274455811 >> 172.21.15.67:0/1274455811 conn(0x7f4c9806c810 msgr2=0x7f4c9806cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:04.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.450+0000 7f4c9f5f3640 1 -- 172.21.15.67:0/1274455811 shutdown_connections 2024-03-24T00:48:04.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:04.450+0000 7f4c9f5f3640 1 -- 172.21.15.67:0/1274455811 wait complete. 2024-03-24T00:48:04.863 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:47:16.492485+0000', 'modified': '2024-03-24T00:48:03.337083+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 108, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8895}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8895': {'gid': 8895, 'name': 'c', 'rank': 0, 'incarnation': 75, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.67:6833/3197261495', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6832', 'nonce': 3197261495}, {'type': 'v1', 'addr': '172.21.15.67:6833', 'nonce': 3197261495}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-24T00:48:04.863 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8895:up:rejoin 2024-03-24T00:48:05.864 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:48:05.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.966+0000 7eff607e6640 1 Processor -- start 2024-03-24T00:48:05.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.966+0000 7eff607e6640 1 -- start start 2024-03-24T00:48:05.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.967+0000 7eff607e6640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7eff580b18f0 0x7eff580b1d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:05.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.967+0000 7eff607e6640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7eff580d0440 0x7eff580d08d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:05.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.967+0000 7eff607e6640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff580b2590 con 0x7eff5806ca40 2024-03-24T00:48:05.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.967+0000 7eff607e6640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff580b2870 con 0x7eff580d0440 2024-03-24T00:48:05.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.967+0000 7eff607e6640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff580b2b50 con 0x7eff580b18f0 2024-03-24T00:48:05.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.967+0000 7eff5ed83640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7eff580d0440 0x7eff580d08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:05.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.967+0000 7eff5e582640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7eff5806ca40 0x7eff5806ce20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48844/0 (socket says 172.21.15.67:48844) 2024-03-24T00:48:05.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.967+0000 7eff5e582640 1 -- 172.21.15.67:0/2381291507 learned_addr learned my addr 172.21.15.67:0/2381291507 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:48:05.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.968+0000 7eff5ed83640 1 -- 172.21.15.67:0/2381291507 >> v2:172.21.15.72:3300/0 conn(0x7eff580b18f0 msgr2=0x7eff580b1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:05.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.968+0000 7eff5dd81640 1 --2- 172.21.15.67:0/2381291507 >> v2:172.21.15.72:3300/0 conn(0x7eff580b18f0 0x7eff580b1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:05.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.968+0000 7eff5ed83640 1 --2- 172.21.15.67:0/2381291507 >> v2:172.21.15.72:3300/0 conn(0x7eff580b18f0 0x7eff580b1d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:05.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.968+0000 7eff5ed83640 1 -- 172.21.15.67:0/2381291507 >> v1:172.21.15.67:6789/0 conn(0x7eff5806ca40 legacy=0x7eff5806ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:05.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.968+0000 7eff5d580640 1 -- 172.21.15.67:0/2381291507 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2744062696 0 0) 0x7eff580b2590 con 0x7eff5806ca40 2024-03-24T00:48:05.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.968+0000 7eff5ed83640 1 -- 172.21.15.67:0/2381291507 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff580b2e30 con 0x7eff580d0440 2024-03-24T00:48:05.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.968+0000 7eff5dd81640 1 --2- 172.21.15.67:0/2381291507 >> v2:172.21.15.72:3300/0 conn(0x7eff580b18f0 0x7eff580b1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:48:05.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.968+0000 7eff5ed83640 1 --2- 172.21.15.67:0/2381291507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff580d0440 0x7eff580d08d0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7eff50009d90 tx=0x7eff5002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e096d5be1afdbb6e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:05.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.969+0000 7eff5d580640 1 -- 172.21.15.67:0/2381291507 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff50039070 con 0x7eff580d0440 2024-03-24T00:48:05.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.969+0000 7eff5d580640 1 -- 172.21.15.67:0/2381291507 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff5002cc60 con 0x7eff580d0440 2024-03-24T00:48:05.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.969+0000 7eff5d580640 1 -- 172.21.15.67:0/2381291507 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff50034660 con 0x7eff580d0440 2024-03-24T00:48:05.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.969+0000 7eff607e6640 1 -- 172.21.15.67:0/2381291507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff580d0440 msgr2=0x7eff580d08d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:05.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.969+0000 7eff607e6640 1 --2- 172.21.15.67:0/2381291507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff580d0440 0x7eff580d08d0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7eff50009d90 tx=0x7eff5002beb0 comp rx=0 tx=0).stop 2024-03-24T00:48:05.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.969+0000 7eff607e6640 1 -- 172.21.15.67:0/2381291507 shutdown_connections 2024-03-24T00:48:05.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.969+0000 7eff607e6640 1 --2- 172.21.15.67:0/2381291507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff580d0440 0x7eff580d08d0 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:05.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.969+0000 7eff607e6640 1 --2- 172.21.15.67:0/2381291507 >> v2:172.21.15.72:3300/0 conn(0x7eff580b18f0 0x7eff580b1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:05.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.969+0000 7eff607e6640 1 -- 172.21.15.67:0/2381291507 >> 172.21.15.67:0/2381291507 conn(0x7eff580d1bf0 msgr2=0x7eff580d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:05.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.969+0000 7eff607e6640 1 -- 172.21.15.67:0/2381291507 shutdown_connections 2024-03-24T00:48:05.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.969+0000 7eff607e6640 1 -- 172.21.15.67:0/2381291507 wait complete. 2024-03-24T00:48:05.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.970+0000 7eff607e6640 1 Processor -- start 2024-03-24T00:48:05.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.970+0000 7eff607e6640 1 -- start start 2024-03-24T00:48:05.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.971+0000 7eff607e6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7eff5806ca40 0x7eff580b70b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:05.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.971+0000 7eff607e6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff580b18f0 0x7eff580b01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:05.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.971+0000 7eff607e6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff580d0440 0x7eff581f3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:05.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.971+0000 7eff607e6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff5815e370 con 0x7eff580d0440 2024-03-24T00:48:05.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.971+0000 7eff607e6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff580b2270 con 0x7eff580b18f0 2024-03-24T00:48:05.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.971+0000 7eff607e6640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff5815e4f0 con 0x7eff5806ca40 2024-03-24T00:48:05.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.971+0000 7eff5ed83640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff580d0440 0x7eff581f3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:05.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.971+0000 7eff5ed83640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff580d0440 0x7eff581f3a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41032/0 (socket says 172.21.15.67:41032) 2024-03-24T00:48:05.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.971+0000 7eff5ed83640 1 -- 172.21.15.67:0/721470883 learned_addr learned my addr 172.21.15.67:0/721470883 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:05.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.971+0000 7eff5dd81640 1 --2- 172.21.15.67:0/721470883 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff580b18f0 0x7eff580b01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:05.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.972+0000 7eff5e582640 1 --2- 172.21.15.67:0/721470883 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7eff5806ca40 0x7eff580b70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:05.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.972+0000 7eff5ed83640 1 -- 172.21.15.67:0/721470883 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7eff5806ca40 msgr2=0x7eff580b70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:05.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.972+0000 7eff5ed83640 1 --2- 172.21.15.67:0/721470883 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7eff5806ca40 0x7eff580b70b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:05.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.972+0000 7eff5ed83640 1 -- 172.21.15.67:0/721470883 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff580b18f0 msgr2=0x7eff580b01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:05.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.972+0000 7eff5ed83640 1 --2- 172.21.15.67:0/721470883 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff580b18f0 0x7eff580b01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:05.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.972+0000 7eff5ed83640 1 -- 172.21.15.67:0/721470883 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff580b75f0 con 0x7eff580d0440 2024-03-24T00:48:05.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.972+0000 7eff5ed83640 1 --2- 172.21.15.67:0/721470883 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff580d0440 0x7eff581f3a20 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7eff50009d90 tx=0x7eff50004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:05.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.972+0000 7eff4effd640 1 -- 172.21.15.67:0/721470883 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff50039050 con 0x7eff580d0440 2024-03-24T00:48:05.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.972+0000 7eff4effd640 1 -- 172.21.15.67:0/721470883 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff50045070 con 0x7eff580d0440 2024-03-24T00:48:05.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.972+0000 7eff4effd640 1 -- 172.21.15.67:0/721470883 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff5003e830 con 0x7eff580d0440 2024-03-24T00:48:05.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.972+0000 7eff607e6640 1 -- 172.21.15.67:0/721470883 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff581f4430 con 0x7eff580d0440 2024-03-24T00:48:05.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.972+0000 7eff607e6640 1 -- 172.21.15.67:0/721470883 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff580af9a0 con 0x7eff580d0440 2024-03-24T00:48:05.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.973+0000 7eff607e6640 1 -- 172.21.15.67:0/721470883 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff5804f400 con 0x7eff580d0440 2024-03-24T00:48:05.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.974+0000 7eff4effd640 1 -- 172.21.15.67:0/721470883 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7eff5003e9d0 con 0x7eff580d0440 2024-03-24T00:48:05.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.974+0000 7eff4effd640 1 --2- 172.21.15.67:0/721470883 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7eff3c06bc00 0x7eff3c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:05.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.974+0000 7eff4effd640 1 -- 172.21.15.67:0/721470883 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6216+0+0 (secure 0 0 0) 0x7eff500ada50 con 0x7eff580d0440 2024-03-24T00:48:05.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.974+0000 7eff5e582640 1 --2- 172.21.15.67:0/721470883 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7eff3c06bc00 0x7eff3c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:05.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.975+0000 7eff5e582640 1 --2- 172.21.15.67:0/721470883 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7eff3c06bc00 0x7eff3c06e0c0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7eff54000f60 tx=0x7eff54007c00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:05.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:05.979+0000 7eff4effd640 1 -- 172.21.15.67:0/721470883 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7eff50032e20 con 0x7eff580d0440 2024-03-24T00:48:06.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.198+0000 7eff607e6640 1 -- 172.21.15.67:0/721470883 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff580b30d0 con 0x7eff580d0440 2024-03-24T00:48:06.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.199+0000 7eff4effd640 1 -- 172.21.15.67:0/721470883 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+4978 (secure 0 0 0) 0x7eff5007c460 con 0x7eff580d0440 2024-03-24T00:48:06.199 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:48:06.199 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":78,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6902,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3672294289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3672294289},{"type":"v1","addr":"172.21.15.67:6835","nonce":3672294289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":6908,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/1286271878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":1286271878},{"type":"v1","addr":"172.21.15.72:6836","nonce":1286271878}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":6911,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/140622129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":140622129},{"type":"v1","addr":"172.21.15.72:6837","nonce":140622129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:47:16.492485+0000","modified":"2024-03-24T00:48:05.339444+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":108,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8895},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8895":{"gid":8895,"name":"c","rank":0,"incarnation":75,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6833/3197261495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3197261495},{"type":"v1","addr":"172.21.15.67:6833","nonce":3197261495}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8895,"qdb_cluster":[8895]},"id":7}]} 2024-03-24T00:48:06.199 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 78 2024-03-24T00:48:06.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.200+0000 7eff607e6640 1 -- 172.21.15.67:0/721470883 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7eff3c06bc00 msgr2=0x7eff3c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:06.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.200+0000 7eff607e6640 1 --2- 172.21.15.67:0/721470883 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7eff3c06bc00 0x7eff3c06e0c0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7eff54000f60 tx=0x7eff54007c00 comp rx=0 tx=0).stop 2024-03-24T00:48:06.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.200+0000 7eff607e6640 1 -- 172.21.15.67:0/721470883 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff580d0440 msgr2=0x7eff581f3a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:06.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.200+0000 7eff607e6640 1 --2- 172.21.15.67:0/721470883 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff580d0440 0x7eff581f3a20 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7eff50009d90 tx=0x7eff50004a40 comp rx=0 tx=0).stop 2024-03-24T00:48:06.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.200+0000 7eff607e6640 1 -- 172.21.15.67:0/721470883 shutdown_connections 2024-03-24T00:48:06.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.200+0000 7eff607e6640 1 --2- 172.21.15.67:0/721470883 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7eff3c06bc00 0x7eff3c06e0c0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.200+0000 7eff607e6640 1 --2- 172.21.15.67:0/721470883 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff580d0440 0x7eff581f3a20 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.200+0000 7eff607e6640 1 --2- 172.21.15.67:0/721470883 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff580b18f0 0x7eff580b01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.200+0000 7eff607e6640 1 --2- 172.21.15.67:0/721470883 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7eff5806ca40 0x7eff580b70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.201+0000 7eff607e6640 1 -- 172.21.15.67:0/721470883 >> 172.21.15.67:0/721470883 conn(0x7eff580d1bf0 msgr2=0x7eff5815c170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:06.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.201+0000 7eff607e6640 1 -- 172.21.15.67:0/721470883 shutdown_connections 2024-03-24T00:48:06.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.201+0000 7eff607e6640 1 -- 172.21.15.67:0/721470883 wait complete. 2024-03-24T00:48:06.613 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 78, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:47:16.492485+0000', 'modified': '2024-03-24T00:48:05.339444+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 108, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8895}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8895': {'gid': 8895, 'name': 'c', 'rank': 0, 'incarnation': 75, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.67:6833/3197261495', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6832', 'nonce': 3197261495}, {'type': 'v1', 'addr': '172.21.15.67:6833', 'nonce': 3197261495}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8895, 'qdb_cluster': [8895]} 2024-03-24T00:48:06.614 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-24T00:48:06.614 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-24T00:48:06.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.718+0000 7f5c33661640 1 Processor -- start 2024-03-24T00:48:06.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.718+0000 7f5c33661640 1 -- start start 2024-03-24T00:48:06.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.719+0000 7f5c33661640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5c2c06b040 0x7f5c2c06b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:06.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.719+0000 7f5c33661640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c2c0cc9a0 con 0x7f5c2c0cc5c0 2024-03-24T00:48:06.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.719+0000 7f5c33661640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c2c0b1f90 con 0x7f5c2c0cfe70 2024-03-24T00:48:06.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.719+0000 7f5c33661640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c2c0b0e60 con 0x7f5c2c06b040 2024-03-24T00:48:06.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.719+0000 7f5c313fd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5c2c0cc5c0 0x7f5c2c0ccde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48848/0 (socket says 172.21.15.67:48848) 2024-03-24T00:48:06.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.719+0000 7f5c30bfc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5c2c06b040 0x7f5c2c06b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:06.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.719+0000 7f5c313fd640 1 -- 172.21.15.67:0/398420398 learned_addr learned my addr 172.21.15.67:0/398420398 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:48:06.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.719+0000 7f5c23fff640 1 -- 172.21.15.67:0/398420398 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1924331795 0 0) 0x7f5c2c0cc9a0 con 0x7f5c2c0cc5c0 2024-03-24T00:48:06.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.719+0000 7f5c23fff640 1 -- 172.21.15.67:0/398420398 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c18003520 con 0x7f5c2c0cc5c0 2024-03-24T00:48:06.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.720+0000 7f5c30bfc640 1 -- 172.21.15.67:0/398420398 >> v1:172.21.15.72:6789/0 conn(0x7f5c2c0cfe70 legacy=0x7f5c2c0d0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:06.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.720+0000 7f5c30bfc640 1 -- 172.21.15.67:0/398420398 >> v1:172.21.15.67:6789/0 conn(0x7f5c2c0cc5c0 legacy=0x7f5c2c0ccde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:06.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.720+0000 7f5c30bfc640 1 -- 172.21.15.67:0/398420398 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c2c0b1140 con 0x7f5c2c06b040 2024-03-24T00:48:06.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.720+0000 7f5c30bfc640 1 --2- 172.21.15.67:0/398420398 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c2c06b040 0x7f5c2c06b4d0 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c0097a0 tx=0x7f5c1c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3561c9c5f8855f4a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:06.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.720+0000 7f5c23fff640 1 -- 172.21.15.67:0/398420398 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c1c034070 con 0x7f5c2c06b040 2024-03-24T00:48:06.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.721+0000 7f5c23fff640 1 -- 172.21.15.67:0/398420398 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c1c030610 con 0x7f5c2c06b040 2024-03-24T00:48:06.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.721+0000 7f5c23fff640 1 -- 172.21.15.67:0/398420398 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c1c0309e0 con 0x7f5c2c06b040 2024-03-24T00:48:06.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.721+0000 7f5c33661640 1 -- 172.21.15.67:0/398420398 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c2c06b040 msgr2=0x7f5c2c06b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:06.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.721+0000 7f5c33661640 1 --2- 172.21.15.67:0/398420398 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c2c06b040 0x7f5c2c06b4d0 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c0097a0 tx=0x7f5c1c02b960 comp rx=0 tx=0).stop 2024-03-24T00:48:06.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.721+0000 7f5c33661640 1 -- 172.21.15.67:0/398420398 shutdown_connections 2024-03-24T00:48:06.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.721+0000 7f5c33661640 1 --2- 172.21.15.67:0/398420398 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c2c06b040 0x7f5c2c06b4d0 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.721+0000 7f5c33661640 1 -- 172.21.15.67:0/398420398 >> 172.21.15.67:0/398420398 conn(0x7f5c2c099a30 msgr2=0x7f5c2c099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:06.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.721+0000 7f5c33661640 1 -- 172.21.15.67:0/398420398 shutdown_connections 2024-03-24T00:48:06.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.721+0000 7f5c33661640 1 -- 172.21.15.67:0/398420398 wait complete. 2024-03-24T00:48:06.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.722+0000 7f5c33661640 1 Processor -- start 2024-03-24T00:48:06.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.722+0000 7f5c33661640 1 -- start start 2024-03-24T00:48:06.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.723+0000 7f5c33661640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c2c06b040 0x7f5c2c150070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:06.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.723+0000 7f5c33661640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5c2c0cc5c0 0x7f5c2c0c9050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:06.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.723+0000 7f5c33661640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c2c0cfe70 0x7f5c2c09ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:06.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.723+0000 7f5c33661640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c2c0b2270 con 0x7f5c2c0cfe70 2024-03-24T00:48:06.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.723+0000 7f5c33661640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c2c0ce970 con 0x7f5c2c06b040 2024-03-24T00:48:06.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.723+0000 7f5c33661640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c2c0b1bb0 con 0x7f5c2c0cc5c0 2024-03-24T00:48:06.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.723+0000 7f5c31bfe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c2c0cfe70 0x7f5c2c09ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:06.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.723+0000 7f5c31bfe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c2c0cfe70 0x7f5c2c09ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41044/0 (socket says 172.21.15.67:41044) 2024-03-24T00:48:06.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.723+0000 7f5c31bfe640 1 -- 172.21.15.67:0/1436426476 learned_addr learned my addr 172.21.15.67:0/1436426476 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:06.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.723+0000 7f5c313fd640 1 --2- 172.21.15.67:0/1436426476 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c2c06b040 0x7f5c2c150070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:06.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.723+0000 7f5c30bfc640 1 --2- 172.21.15.67:0/1436426476 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5c2c0cc5c0 0x7f5c2c0c9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:06.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.723+0000 7f5c31bfe640 1 -- 172.21.15.67:0/1436426476 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5c2c0cc5c0 msgr2=0x7f5c2c0c9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:06.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.723+0000 7f5c31bfe640 1 --2- 172.21.15.67:0/1436426476 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5c2c0cc5c0 0x7f5c2c0c9050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.723+0000 7f5c31bfe640 1 -- 172.21.15.67:0/1436426476 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c2c06b040 msgr2=0x7f5c2c150070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:06.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.723+0000 7f5c31bfe640 1 --2- 172.21.15.67:0/1436426476 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c2c06b040 0x7f5c2c150070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.723+0000 7f5c31bfe640 1 -- 172.21.15.67:0/1436426476 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c2c0c9590 con 0x7f5c2c0cfe70 2024-03-24T00:48:06.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.724+0000 7f5c31bfe640 1 --2- 172.21.15.67:0/1436426476 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c2c0cfe70 0x7f5c2c09ca70 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f5c2400c6b0 tx=0x7f5c2400cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:06.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.724+0000 7f5c21ffb640 1 -- 172.21.15.67:0/1436426476 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c24003b70 con 0x7f5c2c0cfe70 2024-03-24T00:48:06.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.724+0000 7f5c21ffb640 1 -- 172.21.15.67:0/1436426476 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c24003d30 con 0x7f5c2c0cfe70 2024-03-24T00:48:06.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.724+0000 7f5c33661640 1 -- 172.21.15.67:0/1436426476 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c2c0c7fb0 con 0x7f5c2c0cfe70 2024-03-24T00:48:06.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.724+0000 7f5c21ffb640 1 -- 172.21.15.67:0/1436426476 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c24005310 con 0x7f5c2c0cfe70 2024-03-24T00:48:06.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.724+0000 7f5c33661640 1 -- 172.21.15.67:0/1436426476 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c2c0c8570 con 0x7f5c2c0cfe70 2024-03-24T00:48:06.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.726+0000 7f5c21ffb640 1 -- 172.21.15.67:0/1436426476 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5c240054b0 con 0x7f5c2c0cfe70 2024-03-24T00:48:06.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.726+0000 7f5c21ffb640 1 --2- 172.21.15.67:0/1436426476 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5c1006bbd0 0x7f5c1006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:06.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.726+0000 7f5c21ffb640 1 -- 172.21.15.67:0/1436426476 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f5c2408ab70 con 0x7f5c2c0cfe70 2024-03-24T00:48:06.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.726+0000 7f5c313fd640 1 --2- 172.21.15.67:0/1436426476 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5c1006bbd0 0x7f5c1006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:06.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.727+0000 7f5c313fd640 1 --2- 172.21.15.67:0/1436426476 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5c1006bbd0 0x7f5c1006e090 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f5c280011e0 tx=0x7f5c28009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:06.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.727+0000 7f5c36252740 1 Processor -- start 2024-03-24T00:48:06.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.727+0000 7f5c36252740 1 -- start start 2024-03-24T00:48:06.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.728+0000 7f5c36252740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55dc58d7dc40 0x55dc58da4770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:06.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.728+0000 7f5c36252740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc58da4cb0 0x55dc58d7cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:06.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.728+0000 7f5c36252740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dc58d7e020 0x55dc58d7d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:06.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.728+0000 7f5c36252740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc58ce1c20 con 0x55dc58da4cb0 2024-03-24T00:48:06.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.728+0000 7f5c36252740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc58c8aa40 con 0x55dc58d7e020 2024-03-24T00:48:06.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.728+0000 7f5c36252740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55dc58cddcc0 con 0x55dc58d7dc40 2024-03-24T00:48:06.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.728+0000 7f5c30bfc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc58da4cb0 0x55dc58d7cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:06.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.728+0000 7f5c30bfc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc58da4cb0 0x55dc58d7cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41046/0 (socket says 172.21.15.67:41046) 2024-03-24T00:48:06.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.728+0000 7f5c30bfc640 1 -- 172.21.15.67:0/3880882225 learned_addr learned my addr 172.21.15.67:0/3880882225 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:06.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.728+0000 7f5c313fd640 1 --2- 172.21.15.67:0/3880882225 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55dc58d7dc40 0x55dc58da4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:06.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.728+0000 7f5c31bfe640 1 --2- 172.21.15.67:0/3880882225 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dc58d7e020 0x55dc58d7d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:06.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.728+0000 7f5c30bfc640 1 -- 172.21.15.67:0/3880882225 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55dc58d7dc40 msgr2=0x55dc58da4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:06.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.728+0000 7f5c30bfc640 1 --2- 172.21.15.67:0/3880882225 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55dc58d7dc40 0x55dc58da4770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.728+0000 7f5c30bfc640 1 -- 172.21.15.67:0/3880882225 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dc58d7e020 msgr2=0x55dc58d7d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:06.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.728+0000 7f5c30bfc640 1 --2- 172.21.15.67:0/3880882225 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dc58d7e020 0x55dc58d7d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.728+0000 7f5c30bfc640 1 -- 172.21.15.67:0/3880882225 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dc58cacf90 con 0x55dc58da4cb0 2024-03-24T00:48:06.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.728+0000 7f5c31bfe640 1 --2- 172.21.15.67:0/3880882225 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dc58d7e020 0x55dc58d7d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:48:06.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.729+0000 7f5c30bfc640 1 --2- 172.21.15.67:0/3880882225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc58da4cb0 0x55dc58d7cc80 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c0098c0 tx=0x7f5c1c008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8493596ab95babd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:06.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.729+0000 7f5c16ffd640 1 -- 172.21.15.67:0/3880882225 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c1c034040 con 0x55dc58da4cb0 2024-03-24T00:48:06.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.729+0000 7f5c16ffd640 1 -- 172.21.15.67:0/3880882225 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c1c039690 con 0x55dc58da4cb0 2024-03-24T00:48:06.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.729+0000 7f5c16ffd640 1 -- 172.21.15.67:0/3880882225 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c1c039a40 con 0x55dc58da4cb0 2024-03-24T00:48:06.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.729+0000 7f5c36252740 1 -- 172.21.15.67:0/3880882225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc58da4cb0 msgr2=0x55dc58d7cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:06.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.729+0000 7f5c36252740 1 --2- 172.21.15.67:0/3880882225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc58da4cb0 0x55dc58d7cc80 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c0098c0 tx=0x7f5c1c008030 comp rx=0 tx=0).stop 2024-03-24T00:48:06.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.729+0000 7f5c36252740 1 -- 172.21.15.67:0/3880882225 shutdown_connections 2024-03-24T00:48:06.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.729+0000 7f5c36252740 1 --2- 172.21.15.67:0/3880882225 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dc58d7e020 0x55dc58d7d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.729+0000 7f5c36252740 1 --2- 172.21.15.67:0/3880882225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc58da4cb0 0x55dc58d7cc80 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.729+0000 7f5c36252740 1 --2- 172.21.15.67:0/3880882225 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55dc58d7dc40 0x55dc58da4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.729+0000 7f5c36252740 1 -- 172.21.15.67:0/3880882225 >> 172.21.15.67:0/3880882225 conn(0x55dc58d44a60 msgr2=0x55dc58dc2960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:06.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.729+0000 7f5c36252740 1 -- 172.21.15.67:0/3880882225 shutdown_connections 2024-03-24T00:48:06.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.729+0000 7f5c36252740 1 -- 172.21.15.67:0/3880882225 wait complete. 2024-03-24T00:48:06.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.730+0000 7f5c36252740 1 -- start start 2024-03-24T00:48:06.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.730+0000 7f5c36252740 1 Processor -- start 2024-03-24T00:48:06.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.731+0000 7f5c36252740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc58d7dc40 0x55dc58dab930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:06.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.731+0000 7f5c36252740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55dc58d7e020 0x55dc58d9bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:06.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.731+0000 7f5c36252740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dc58d9d210 0x55dc58d9d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:06.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.731+0000 7f5c36252740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc58da42e0 con 0x55dc58d7dc40 2024-03-24T00:48:06.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.731+0000 7f5c36252740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc58c9de90 con 0x55dc58d9d210 2024-03-24T00:48:06.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.731+0000 7f5c36252740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55dc58d9d060 con 0x55dc58d7e020 2024-03-24T00:48:06.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.731+0000 7f5c36252740 10 client.0 _ll_register_callbacks cb 0x55dc589f5ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:48:06.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.731+0000 7f5c313fd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc58d7dc40 0x55dc58dab930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:06.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.731+0000 7f5c313fd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc58d7dc40 0x55dc58dab930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41056/0 (socket says 172.21.15.67:41056) 2024-03-24T00:48:06.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.731+0000 7f5c313fd640 1 -- 172.21.15.67:0/1477517295 learned_addr learned my addr 172.21.15.67:0/1477517295 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:06.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.731+0000 7f5c30bfc640 1 --2- 172.21.15.67:0/1477517295 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55dc58d7e020 0x55dc58d9bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:06.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.731+0000 7f5c31bfe640 1 --2- 172.21.15.67:0/1477517295 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dc58d9d210 0x55dc58d9d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:06.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.731+0000 7f5c313fd640 1 -- 172.21.15.67:0/1477517295 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55dc58d7e020 msgr2=0x55dc58d9bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:06.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.731+0000 7f5c313fd640 1 --2- 172.21.15.67:0/1477517295 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55dc58d7e020 0x55dc58d9bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.731+0000 7f5c313fd640 1 -- 172.21.15.67:0/1477517295 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dc58d9d210 msgr2=0x55dc58d9d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:06.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.731+0000 7f5c313fd640 1 --2- 172.21.15.67:0/1477517295 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dc58d9d210 0x55dc58d9d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.731+0000 7f5c313fd640 1 -- 172.21.15.67:0/1477517295 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dc58caa9d0 con 0x55dc58d7dc40 2024-03-24T00:48:06.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.732+0000 7f5c313fd640 1 --2- 172.21.15.67:0/1477517295 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc58d7dc40 0x55dc58dab930 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f5c28005f80 tx=0x7f5c28001150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b96678370725b043 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:06.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.732+0000 7f5c0bfff640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:48:06.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.732+0000 7f5c0bfff640 1 -- 172.21.15.67:0/1477517295 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c28027060 con 0x55dc58d7dc40 2024-03-24T00:48:06.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.732+0000 7f5c0bfff640 1 -- 172.21.15.67:0/1477517295 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f5c28010040 con 0x55dc58d7dc40 2024-03-24T00:48:06.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.732+0000 7f5c36252740 1 -- 172.21.15.67:0/1477517295 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5c1c039690 con 0x55dc58d7dc40 2024-03-24T00:48:06.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.732+0000 7f5c0bfff640 1 -- 172.21.15.67:0/1477517295 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5c28013420 con 0x55dc58d7dc40 2024-03-24T00:48:06.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.732+0000 7f5c36252740 10 client.8973 fetch_fsmap learned FSMap version 78 2024-03-24T00:48:06.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.732+0000 7f5c36252740 1 -- 172.21.15.67:0/1477517295 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=2+}) v3 -- 0x55dc589f62d0 con 0x55dc58d7dc40 2024-03-24T00:48:06.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.732+0000 7f5c0bfff640 1 -- 172.21.15.67:0/1477517295 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c1c039690 con 0x55dc58d7dc40 2024-03-24T00:48:06.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.732+0000 7f5c0bfff640 1 -- 172.21.15.67:0/1477517295 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 78) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f5c280135c0 con 0x55dc58d7dc40 2024-03-24T00:48:06.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.732+0000 7f5c36252740 10 client.8973 fetch_fsmap finished waiting for FSMap version 78 2024-03-24T00:48:06.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.732+0000 7f5c36252740 10 client.8973 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T00:48:06.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.732+0000 7f5c36252740 20 client.8973 populate_metadata read hostname 'smithi067' 2024-03-24T00:48:06.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.732+0000 7f5c36252740 1 --2- 172.21.15.67:0/1477517295 >> [v2:172.21.15.67:6832/3197261495,v1:172.21.15.67:6833/3197261495] conn(0x55dc58dd5af0 0x55dc58dd5f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:06.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.732+0000 7f5c36252740 4 client.8973 mds_command: new command op to 8895 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:48:06.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.732+0000 7f5c36252740 1 -- 172.21.15.67:0/1477517295 --> [v2:172.21.15.67:6832/3197261495,v1:172.21.15.67:6833/3197261495] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dc58cef160 con 0x55dc58dd5af0 2024-03-24T00:48:06.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.732+0000 7f5c30bfc640 1 --2- 172.21.15.67:0/1477517295 >> [v2:172.21.15.67:6832/3197261495,v1:172.21.15.67:6833/3197261495] conn(0x55dc58dd5af0 0x55dc58dd5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:06.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.733+0000 7f5c0bfff640 10 client.8973 ms_handle_connect on v2:172.21.15.67:6832/3197261495 2024-03-24T00:48:06.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.733+0000 7f5c30bfc640 1 --2- 172.21.15.67:0/1477517295 >> [v2:172.21.15.67:6832/3197261495,v1:172.21.15.67:6833/3197261495] conn(0x55dc58dd5af0 0x55dc58dd5f10 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae0df02371ec963 server_cookie=68ed1293250a4b5d in_seq=0 out_seq=0 2024-03-24T00:48:06.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.733+0000 7f5c0bfff640 1 -- 172.21.15.67:0/1477517295 <== mds.0 v2:172.21.15.67:6832/3197261495 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55dc58cef160 con 0x55dc58dd5af0 2024-03-24T00:48:06.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.733+0000 7f5c0bfff640 10 client.8973 handle_command_reply: tid=0 2024-03-24T00:48:06.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.733+0000 7f5c36252740 1 client.8973 shutdown 2024-03-24T00:48:06.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.733+0000 7f5c36252740 1 -- 172.21.15.67:0/1477517295 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc58d7dc40 msgr2=0x55dc58dab930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:06.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.733+0000 7f5c36252740 1 --2- 172.21.15.67:0/1477517295 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc58d7dc40 0x55dc58dab930 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f5c28005f80 tx=0x7f5c28001150 comp rx=0 tx=0).stop 2024-03-24T00:48:06.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.733+0000 7f5c36252740 1 -- 172.21.15.67:0/1477517295 shutdown_connections 2024-03-24T00:48:06.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.733+0000 7f5c36252740 1 --2- 172.21.15.67:0/1477517295 >> [v2:172.21.15.67:6832/3197261495,v1:172.21.15.67:6833/3197261495] conn(0x55dc58dd5af0 0x55dc58dd5f10 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.733+0000 7f5c36252740 1 --2- 172.21.15.67:0/1477517295 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dc58d9d210 0x55dc58d9d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.733+0000 7f5c0bfff640 0 client.8973 ms_handle_reset on v2:172.21.15.67:6832/3197261495 2024-03-24T00:48:06.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.733+0000 7f5c36252740 1 --2- 172.21.15.67:0/1477517295 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55dc58d7e020 0x55dc58d9bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.733+0000 7f5c36252740 1 --2- 172.21.15.67:0/1477517295 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc58d7dc40 0x55dc58dab930 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.733+0000 7f5c36252740 1 -- 172.21.15.67:0/1477517295 >> 172.21.15.67:0/1477517295 conn(0x55dc58d44a60 msgr2=0x55dc58da5940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:06.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.734+0000 7f5c36252740 1 -- 172.21.15.67:0/1477517295 shutdown_connections 2024-03-24T00:48:06.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.734+0000 7f5c36252740 1 -- 172.21.15.67:0/1477517295 wait complete. 2024-03-24T00:48:06.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.734+0000 7f5c36252740 20 client.8973 trim_cache size 0 max 16384 2024-03-24T00:48:06.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.750+0000 7f5c36252740 1 Processor -- start 2024-03-24T00:48:06.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.750+0000 7f5c36252740 1 -- start start 2024-03-24T00:48:06.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.751+0000 7f5c36252740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55dc58d7dc40 0x55dc58d78a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:06.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.751+0000 7f5c36252740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dc58d7e020 0x55dc58dd4b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:06.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.752+0000 7f5c36252740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc58d9d210 0x55dc58dd1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:06.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.752+0000 7f5c36252740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c2c1ffc10 con 0x55dc58d9d210 2024-03-24T00:48:06.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.752+0000 7f5c36252740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c2c09c150 con 0x55dc58d7e020 2024-03-24T00:48:06.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.752+0000 7f5c36252740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c2c1ff8b0 con 0x55dc58d7dc40 2024-03-24T00:48:06.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.752+0000 7f5c31bfe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc58d9d210 0x55dc58dd1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:06.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.752+0000 7f5c31bfe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc58d9d210 0x55dc58dd1290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41068/0 (socket says 172.21.15.67:41068) 2024-03-24T00:48:06.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.752+0000 7f5c31bfe640 1 -- 172.21.15.67:0/2011679056 learned_addr learned my addr 172.21.15.67:0/2011679056 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:06.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.752+0000 7f5c313fd640 1 --2- 172.21.15.67:0/2011679056 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55dc58d7dc40 0x55dc58d78a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:06.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.752+0000 7f5c30bfc640 1 --2- 172.21.15.67:0/2011679056 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dc58d7e020 0x55dc58dd4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:06.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.752+0000 7f5c31bfe640 1 -- 172.21.15.67:0/2011679056 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55dc58d7dc40 msgr2=0x55dc58d78a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:06.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.752+0000 7f5c31bfe640 1 --2- 172.21.15.67:0/2011679056 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55dc58d7dc40 0x55dc58d78a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.752+0000 7f5c31bfe640 1 -- 172.21.15.67:0/2011679056 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dc58d7e020 msgr2=0x55dc58dd4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:06.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.752+0000 7f5c31bfe640 1 --2- 172.21.15.67:0/2011679056 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dc58d7e020 0x55dc58dd4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.752+0000 7f5c31bfe640 1 -- 172.21.15.67:0/2011679056 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dc58ac06b0 con 0x55dc58d9d210 2024-03-24T00:48:06.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.752+0000 7f5c30bfc640 1 --2- 172.21.15.67:0/2011679056 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dc58d7e020 0x55dc58dd4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:48:06.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.752+0000 7f5c31bfe640 1 --2- 172.21.15.67:0/2011679056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc58d9d210 0x55dc58dd1290 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f5c2400cfd0 tx=0x7f5c2401d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22a03d7d01c963f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:06.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.752+0000 7f5c167fc640 1 -- 172.21.15.67:0/2011679056 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c24061470 con 0x55dc58d9d210 2024-03-24T00:48:06.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.752+0000 7f5c167fc640 1 -- 172.21.15.67:0/2011679056 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c24056d70 con 0x55dc58d9d210 2024-03-24T00:48:06.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.752+0000 7f5c167fc640 1 -- 172.21.15.67:0/2011679056 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c24055340 con 0x55dc58d9d210 2024-03-24T00:48:06.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.753+0000 7f5c36252740 1 -- 172.21.15.67:0/2011679056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc58d9d210 msgr2=0x55dc58dd1290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:06.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.753+0000 7f5c36252740 1 --2- 172.21.15.67:0/2011679056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc58d9d210 0x55dc58dd1290 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f5c2400cfd0 tx=0x7f5c2401d040 comp rx=0 tx=0).stop 2024-03-24T00:48:06.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.753+0000 7f5c36252740 1 -- 172.21.15.67:0/2011679056 shutdown_connections 2024-03-24T00:48:06.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.753+0000 7f5c36252740 1 --2- 172.21.15.67:0/2011679056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc58d9d210 0x55dc58dd1290 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.753+0000 7f5c36252740 1 --2- 172.21.15.67:0/2011679056 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dc58d7e020 0x55dc58dd4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.753+0000 7f5c36252740 1 --2- 172.21.15.67:0/2011679056 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55dc58d7dc40 0x55dc58d78a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.753+0000 7f5c36252740 1 -- 172.21.15.67:0/2011679056 >> 172.21.15.67:0/2011679056 conn(0x55dc58d44a60 msgr2=0x55dc58d9b9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:06.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.753+0000 7f5c36252740 1 -- 172.21.15.67:0/2011679056 shutdown_connections 2024-03-24T00:48:06.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.753+0000 7f5c36252740 1 -- 172.21.15.67:0/2011679056 wait complete. 2024-03-24T00:48:06.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.753+0000 7f5c36252740 1 -- start start 2024-03-24T00:48:06.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.753+0000 7f5c36252740 1 Processor -- start 2024-03-24T00:48:06.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.754+0000 7f5c36252740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55dc58d7dc40 0x55dc58e23b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:06.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.754+0000 7f5c36252740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dc58d7e020 0x55dc58d9dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:06.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.754+0000 7f5c36252740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc58d9d210 0x55dc58db82f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:06.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.754+0000 7f5c36252740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc58d9fb00 con 0x55dc58d9d210 2024-03-24T00:48:06.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.754+0000 7f5c36252740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc58da9190 con 0x55dc58d7e020 2024-03-24T00:48:06.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.754+0000 7f5c36252740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55dc58da1140 con 0x55dc58d7dc40 2024-03-24T00:48:06.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.754+0000 7f5c36252740 10 client.0 _ll_register_callbacks cb 0x7f5c28001dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:48:06.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.754+0000 7f5c31bfe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc58d9d210 0x55dc58db82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:06.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.754+0000 7f5c31bfe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc58d9d210 0x55dc58db82f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41082/0 (socket says 172.21.15.67:41082) 2024-03-24T00:48:06.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.755+0000 7f5c31bfe640 1 -- 172.21.15.67:0/61068080 learned_addr learned my addr 172.21.15.67:0/61068080 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:06.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.755+0000 7f5c313fd640 1 --2- 172.21.15.67:0/61068080 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55dc58d7dc40 0x55dc58e23b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:06.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.755+0000 7f5c30bfc640 1 --2- 172.21.15.67:0/61068080 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dc58d7e020 0x55dc58d9dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:06.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.755+0000 7f5c31bfe640 1 -- 172.21.15.67:0/61068080 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55dc58d7dc40 msgr2=0x55dc58e23b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:06.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.755+0000 7f5c31bfe640 1 --2- 172.21.15.67:0/61068080 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55dc58d7dc40 0x55dc58e23b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.755+0000 7f5c31bfe640 1 -- 172.21.15.67:0/61068080 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dc58d7e020 msgr2=0x55dc58d9dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:06.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.755+0000 7f5c31bfe640 1 --2- 172.21.15.67:0/61068080 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dc58d7e020 0x55dc58d9dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.755+0000 7f5c31bfe640 1 -- 172.21.15.67:0/61068080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dc58da34f0 con 0x55dc58d9d210 2024-03-24T00:48:06.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.755+0000 7f5c31bfe640 1 --2- 172.21.15.67:0/61068080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc58d9d210 0x55dc58db82f0 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f5c2401a000 tx=0x7f5c24088d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2163e8efc7499e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:06.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.755+0000 7f5c0bfff640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:48:06.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.755+0000 7f5c0bfff640 1 -- 172.21.15.67:0/61068080 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c24061470 con 0x55dc58d9d210 2024-03-24T00:48:06.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.755+0000 7f5c0bfff640 1 -- 172.21.15.67:0/61068080 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f5c24055340 con 0x55dc58d9d210 2024-03-24T00:48:06.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.755+0000 7f5c36252740 1 -- 172.21.15.67:0/61068080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5c24056d70 con 0x55dc58d9d210 2024-03-24T00:48:06.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.755+0000 7f5c0bfff640 1 -- 172.21.15.67:0/61068080 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5c24082050 con 0x55dc58d9d210 2024-03-24T00:48:06.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.755+0000 7f5c36252740 10 client.8979 fetch_fsmap learned FSMap version 78 2024-03-24T00:48:06.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.755+0000 7f5c36252740 1 -- 172.21.15.67:0/61068080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=2+}) v3 -- 0x55dc58e26ea0 con 0x55dc58d9d210 2024-03-24T00:48:06.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.755+0000 7f5c0bfff640 1 -- 172.21.15.67:0/61068080 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c24056d70 con 0x55dc58d9d210 2024-03-24T00:48:06.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.756+0000 7f5c0bfff640 1 -- 172.21.15.67:0/61068080 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 78) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f5c24058120 con 0x55dc58d9d210 2024-03-24T00:48:06.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.756+0000 7f5c36252740 10 client.8979 fetch_fsmap finished waiting for FSMap version 78 2024-03-24T00:48:06.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.756+0000 7f5c36252740 10 client.8979 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T00:48:06.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.756+0000 7f5c36252740 20 client.8979 populate_metadata read hostname 'smithi067' 2024-03-24T00:48:06.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.756+0000 7f5c36252740 1 --2- 172.21.15.67:0/61068080 >> [v2:172.21.15.67:6832/3197261495,v1:172.21.15.67:6833/3197261495] conn(0x55dc58e27db0 0x55dc58e28190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:06.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.756+0000 7f5c36252740 4 client.8979 mds_command: new command op to 8895 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:48:06.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.756+0000 7f5c36252740 1 -- 172.21.15.67:0/61068080 --> [v2:172.21.15.67:6832/3197261495,v1:172.21.15.67:6833/3197261495] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55dc58da0cb0 con 0x55dc58e27db0 2024-03-24T00:48:06.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.756+0000 7f5c313fd640 1 --2- 172.21.15.67:0/61068080 >> [v2:172.21.15.67:6832/3197261495,v1:172.21.15.67:6833/3197261495] conn(0x55dc58e27db0 0x55dc58e28190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:06.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.756+0000 7f5c0bfff640 10 client.8979 ms_handle_connect on v2:172.21.15.67:6832/3197261495 2024-03-24T00:48:06.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.756+0000 7f5c313fd640 1 --2- 172.21.15.67:0/61068080 >> [v2:172.21.15.67:6832/3197261495,v1:172.21.15.67:6833/3197261495] conn(0x55dc58e27db0 0x55dc58e28190 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a88f61d6188ebf5 server_cookie=e198114406ce8332 in_seq=0 out_seq=0 2024-03-24T00:48:06.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.756+0000 7f5c0bfff640 1 -- 172.21.15.67:0/61068080 <== mds.0 v2:172.21.15.67:6832/3197261495 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x55dc58da0cb0 con 0x55dc58e27db0 2024-03-24T00:48:06.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.756+0000 7f5c0bfff640 10 client.8979 handle_command_reply: tid=0 2024-03-24T00:48:06.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.756+0000 7f5c36252740 1 client.8979 shutdown 2024-03-24T00:48:06.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.756+0000 7f5c36252740 1 -- 172.21.15.67:0/61068080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc58d9d210 msgr2=0x55dc58db82f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:06.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.756+0000 7f5c36252740 1 --2- 172.21.15.67:0/61068080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc58d9d210 0x55dc58db82f0 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f5c2401a000 tx=0x7f5c24088d70 comp rx=0 tx=0).stop 2024-03-24T00:48:06.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.757+0000 7f5c36252740 1 -- 172.21.15.67:0/61068080 shutdown_connections 2024-03-24T00:48:06.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.757+0000 7f5c36252740 1 --2- 172.21.15.67:0/61068080 >> [v2:172.21.15.67:6832/3197261495,v1:172.21.15.67:6833/3197261495] conn(0x55dc58e27db0 0x55dc58e28190 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.757+0000 7f5c36252740 1 --2- 172.21.15.67:0/61068080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc58d9d210 0x55dc58db82f0 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.757+0000 7f5c36252740 1 --2- 172.21.15.67:0/61068080 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dc58d7e020 0x55dc58d9dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.757+0000 7f5c36252740 1 --2- 172.21.15.67:0/61068080 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55dc58d7dc40 0x55dc58e23b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.757+0000 7f5c0bfff640 0 client.8979 ms_handle_reset on v2:172.21.15.67:6832/3197261495 2024-03-24T00:48:06.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.757+0000 7f5c36252740 1 -- 172.21.15.67:0/61068080 >> 172.21.15.67:0/61068080 conn(0x55dc58d44a60 msgr2=0x55dc58dc6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:06.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.757+0000 7f5c36252740 1 -- 172.21.15.67:0/61068080 shutdown_connections 2024-03-24T00:48:06.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.757+0000 7f5c36252740 1 -- 172.21.15.67:0/61068080 wait complete. 2024-03-24T00:48:06.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.757+0000 7f5c36252740 20 client.8979 trim_cache size 0 max 16384 2024-03-24T00:48:06.764 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:48:06.764 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "de1d0178-808e-49ca-a986-e297e523b976", 2024-03-24T00:48:06.764 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-03-24T00:48:06.764 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 8895, 2024-03-24T00:48:06.764 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-03-24T00:48:06.764 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-03-24T00:48:06.764 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-03-24T00:48:06.764 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 4.4212533379999996, 2024-03-24T00:48:06.765 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 78, 2024-03-24T00:48:06.765 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 108, 2024-03-24T00:48:06.765 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 108, 2024-03-24T00:48:06.765 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 5.3937527870000004 2024-03-24T00:48:06.765 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:48:06.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.757+0000 7f5c33661640 1 -- 172.21.15.67:0/1436426476 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5c1006bbd0 msgr2=0x7f5c1006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:06.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.757+0000 7f5c33661640 1 --2- 172.21.15.67:0/1436426476 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5c1006bbd0 0x7f5c1006e090 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f5c280011e0 tx=0x7f5c28009040 comp rx=0 tx=0).stop 2024-03-24T00:48:06.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.757+0000 7f5c33661640 1 -- 172.21.15.67:0/1436426476 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c2c0cfe70 msgr2=0x7f5c2c09ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:06.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.757+0000 7f5c33661640 1 --2- 172.21.15.67:0/1436426476 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c2c0cfe70 0x7f5c2c09ca70 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f5c2400c6b0 tx=0x7f5c2400cb70 comp rx=0 tx=0).stop 2024-03-24T00:48:06.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.757+0000 7f5c33661640 1 -- 172.21.15.67:0/1436426476 shutdown_connections 2024-03-24T00:48:06.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.757+0000 7f5c33661640 1 --2- 172.21.15.67:0/1436426476 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5c1006bbd0 0x7f5c1006e090 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.757+0000 7f5c33661640 1 --2- 172.21.15.67:0/1436426476 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c2c0cfe70 0x7f5c2c09ca70 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.757+0000 7f5c33661640 1 --2- 172.21.15.67:0/1436426476 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5c2c0cc5c0 0x7f5c2c0c9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.757+0000 7f5c33661640 1 --2- 172.21.15.67:0/1436426476 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c2c06b040 0x7f5c2c150070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.757+0000 7f5c33661640 1 -- 172.21.15.67:0/1436426476 >> 172.21.15.67:0/1436426476 conn(0x7f5c2c099a30 msgr2=0x7f5c2c0ccb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:06.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.757+0000 7f5c33661640 1 -- 172.21.15.67:0/1436426476 shutdown_connections 2024-03-24T00:48:06.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.758+0000 7f5c33661640 1 -- 172.21.15.67:0/1436426476 wait complete. 2024-03-24T00:48:06.768 DEBUG:teuthology.orchestra.run.smithi067:> sudo daemon-helper kill ceph --cluster ceph -w 2024-03-24T00:48:06.770 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:48:06.770 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.7:0 scrub start / repair,recursive 2024-03-24T00:48:06.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.882+0000 7f82f4ce8640 1 Processor -- start 2024-03-24T00:48:06.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.882+0000 7f82f4ce8640 1 -- start start 2024-03-24T00:48:06.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.883+0000 7f82f4ce8640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f82f015e240 0x7f82f00b2d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:06.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.883+0000 7f82f4ce8640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f82f00d0a20 0x7f82f0099d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:06.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.883+0000 7f82f4ce8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82f00d1100 con 0x7f82f009a240 2024-03-24T00:48:06.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.883+0000 7f82f4ce8640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82f00b4650 con 0x7f82f015e240 2024-03-24T00:48:06.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.883+0000 7f82f4ce8640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82f00cc360 con 0x7f82f00d0a20 2024-03-24T00:48:06.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.883+0000 7f82eed9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f82f009a240 0x7f82f006c3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48854/0 (socket says 172.21.15.67:48854) 2024-03-24T00:48:06.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.883+0000 7f82eed9d640 1 -- 172.21.15.67:0/774207469 learned_addr learned my addr 172.21.15.67:0/774207469 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:48:06.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.883+0000 7f82edd9b640 1 --2- 172.21.15.67:0/774207469 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f82f00d0a20 0x7f82f0099d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:06.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.883+0000 7f82ee59c640 1 --2- 172.21.15.67:0/774207469 >> v2:172.21.15.72:3300/0 conn(0x7f82f015e240 0x7f82f00b2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:06.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.883+0000 7f82ed59a640 1 -- 172.21.15.67:0/774207469 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 583287095 0 0) 0x7f82f00d1100 con 0x7f82f009a240 2024-03-24T00:48:06.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.883+0000 7f82ed59a640 1 -- 172.21.15.67:0/774207469 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82d8003520 con 0x7f82f009a240 2024-03-24T00:48:06.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.883+0000 7f82edd9b640 1 -- 172.21.15.67:0/774207469 >> v2:172.21.15.72:3300/0 conn(0x7f82f015e240 msgr2=0x7f82f00b2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:06.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.883+0000 7f82edd9b640 1 --2- 172.21.15.67:0/774207469 >> v2:172.21.15.72:3300/0 conn(0x7f82f015e240 0x7f82f00b2d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.883+0000 7f82edd9b640 1 -- 172.21.15.67:0/774207469 >> v1:172.21.15.67:6789/0 conn(0x7f82f009a240 legacy=0x7f82f006c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:06.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.883+0000 7f82edd9b640 1 -- 172.21.15.67:0/774207469 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f00d13e0 con 0x7f82f00d0a20 2024-03-24T00:48:06.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.884+0000 7f82edd9b640 1 --2- 172.21.15.67:0/774207469 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f82f00d0a20 0x7f82f0099d00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f82e40097a0 tx=0x7f82e402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3c8e4eec53b3fb1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:06.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.884+0000 7f82ed59a640 1 -- 172.21.15.67:0/774207469 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e4035020 con 0x7f82f00d0a20 2024-03-24T00:48:06.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.884+0000 7f82ed59a640 1 -- 172.21.15.67:0/774207469 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82e4031830 con 0x7f82f00d0a20 2024-03-24T00:48:06.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.884+0000 7f82ed59a640 1 -- 172.21.15.67:0/774207469 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e4031be0 con 0x7f82f00d0a20 2024-03-24T00:48:06.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.885+0000 7f82f4ce8640 1 -- 172.21.15.67:0/774207469 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f82f00d0a20 msgr2=0x7f82f0099d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:06.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.885+0000 7f82f4ce8640 1 --2- 172.21.15.67:0/774207469 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f82f00d0a20 0x7f82f0099d00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f82e40097a0 tx=0x7f82e402b8d0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.885+0000 7f82f4ce8640 1 -- 172.21.15.67:0/774207469 shutdown_connections 2024-03-24T00:48:06.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.885+0000 7f82f4ce8640 1 --2- 172.21.15.67:0/774207469 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f82f00d0a20 0x7f82f0099d00 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.885+0000 7f82f4ce8640 1 --2- 172.21.15.67:0/774207469 >> v2:172.21.15.72:3300/0 conn(0x7f82f015e240 0x7f82f00b2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.885+0000 7f82f4ce8640 1 -- 172.21.15.67:0/774207469 >> 172.21.15.67:0/774207469 conn(0x7f82f00b36d0 msgr2=0x7f82f00b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:06.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.885+0000 7f82f4ce8640 1 -- 172.21.15.67:0/774207469 shutdown_connections 2024-03-24T00:48:06.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.885+0000 7f82f4ce8640 1 -- 172.21.15.67:0/774207469 wait complete. 2024-03-24T00:48:06.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.886+0000 7f82f4ce8640 1 Processor -- start 2024-03-24T00:48:06.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.886+0000 7f82f4ce8640 1 -- start start 2024-03-24T00:48:06.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.887+0000 7f82f4ce8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f82f009a240 0x7f82f00b4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:06.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.887+0000 7f82f4ce8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82f00d0a20 0x7f82f01513f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:06.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.887+0000 7f82f4ce8640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f82f015e240 0x7f82f00cae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:06.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.887+0000 7f82f4ce8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f00d19a0 con 0x7f82f00d0a20 2024-03-24T00:48:06.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.887+0000 7f82f4ce8640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f00d0ea0 con 0x7f82f009a240 2024-03-24T00:48:06.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.887+0000 7f82f4ce8640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82f015e620 con 0x7f82f015e240 2024-03-24T00:48:06.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.887+0000 7f82edd9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82f00d0a20 0x7f82f01513f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:06.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.887+0000 7f82edd9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82f00d0a20 0x7f82f01513f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41090/0 (socket says 172.21.15.67:41090) 2024-03-24T00:48:06.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.887+0000 7f82edd9b640 1 -- 172.21.15.67:0/652708072 learned_addr learned my addr 172.21.15.67:0/652708072 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:06.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.887+0000 7f82eed9d640 1 --2- 172.21.15.67:0/652708072 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f82f015e240 0x7f82f00cae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:06.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.887+0000 7f82ee59c640 1 --2- 172.21.15.67:0/652708072 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f82f009a240 0x7f82f00b4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:06.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.888+0000 7f82edd9b640 1 -- 172.21.15.67:0/652708072 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f82f015e240 msgr2=0x7f82f00cae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:06.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.888+0000 7f82edd9b640 1 --2- 172.21.15.67:0/652708072 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f82f015e240 0x7f82f00cae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.888+0000 7f82edd9b640 1 -- 172.21.15.67:0/652708072 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f82f009a240 msgr2=0x7f82f00b4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:06.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.888+0000 7f82edd9b640 1 --2- 172.21.15.67:0/652708072 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f82f009a240 0x7f82f00b4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.888+0000 7f82edd9b640 1 -- 172.21.15.67:0/652708072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f00c9da0 con 0x7f82f00d0a20 2024-03-24T00:48:06.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.888+0000 7f82edd9b640 1 --2- 172.21.15.67:0/652708072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82f00d0a20 0x7f82f01513f0 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f82e4000c00 tx=0x7f82e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:06.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.888+0000 7f82deffd640 1 -- 172.21.15.67:0/652708072 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e4048070 con 0x7f82f00d0a20 2024-03-24T00:48:06.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.888+0000 7f82deffd640 1 -- 172.21.15.67:0/652708072 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82e403a6f0 con 0x7f82f00d0a20 2024-03-24T00:48:06.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.888+0000 7f82deffd640 1 -- 172.21.15.67:0/652708072 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e403aae0 con 0x7f82f00d0a20 2024-03-24T00:48:06.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.888+0000 7f82f4ce8640 1 -- 172.21.15.67:0/652708072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82f00ca080 con 0x7f82f00d0a20 2024-03-24T00:48:06.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.888+0000 7f82f4ce8640 1 -- 172.21.15.67:0/652708072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82f00ca640 con 0x7f82f00d0a20 2024-03-24T00:48:06.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.890+0000 7f82deffd640 1 -- 172.21.15.67:0/652708072 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f82e404b070 con 0x7f82f00d0a20 2024-03-24T00:48:06.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.890+0000 7f82deffd640 1 --2- 172.21.15.67:0/652708072 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f82d006bbd0 0x7f82d006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:06.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.890+0000 7f82deffd640 1 -- 172.21.15.67:0/652708072 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f82e40ac2e0 con 0x7f82f00d0a20 2024-03-24T00:48:06.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.890+0000 7f82ee59c640 1 --2- 172.21.15.67:0/652708072 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f82d006bbd0 0x7f82d006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:06.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.891+0000 7f82ee59c640 1 --2- 172.21.15.67:0/652708072 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f82d006bbd0 0x7f82d006e090 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f82e0009ad0 tx=0x7f82e0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:06.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.891+0000 7f82f79ad740 1 Processor -- start 2024-03-24T00:48:06.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.891+0000 7f82f79ad740 1 -- start start 2024-03-24T00:48:06.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.892+0000 7f82f79ad740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558e218b9e60 0x558e218e0990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:06.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.892+0000 7f82f79ad740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558e218e0ed0 0x558e218b8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:06.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.892+0000 7f82f79ad740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558e218ba240 0x558e218b93e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:06.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.892+0000 7f82f79ad740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e217f8c90 con 0x558e218ba240 2024-03-24T00:48:06.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.892+0000 7f82f79ad740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e2181e070 con 0x558e218b9e60 2024-03-24T00:48:06.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.892+0000 7f82f79ad740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x558e217c6c80 con 0x558e218e0ed0 2024-03-24T00:48:06.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.892+0000 7f82eed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558e218ba240 0x558e218b93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:06.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.892+0000 7f82eed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558e218ba240 0x558e218b93e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41104/0 (socket says 172.21.15.67:41104) 2024-03-24T00:48:06.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.892+0000 7f82eed9d640 1 -- 172.21.15.67:0/2210819440 learned_addr learned my addr 172.21.15.67:0/2210819440 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:06.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.892+0000 7f82edd9b640 1 --2- 172.21.15.67:0/2210819440 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558e218e0ed0 0x558e218b8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:06.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.892+0000 7f82ee59c640 1 --2- 172.21.15.67:0/2210819440 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558e218b9e60 0x558e218e0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:06.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.892+0000 7f82eed9d640 1 -- 172.21.15.67:0/2210819440 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558e218e0ed0 msgr2=0x558e218b8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:06.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.892+0000 7f82eed9d640 1 --2- 172.21.15.67:0/2210819440 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558e218e0ed0 0x558e218b8ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.892+0000 7f82eed9d640 1 -- 172.21.15.67:0/2210819440 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558e218b9e60 msgr2=0x558e218e0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:06.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.893+0000 7f82eed9d640 1 --2- 172.21.15.67:0/2210819440 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558e218b9e60 0x558e218e0990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.893+0000 7f82eed9d640 1 -- 172.21.15.67:0/2210819440 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558e217bed10 con 0x558e218ba240 2024-03-24T00:48:06.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.893+0000 7f82edd9b640 1 --2- 172.21.15.67:0/2210819440 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558e218e0ed0 0x558e218b8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:48:06.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.893+0000 7f82eed9d640 1 --2- 172.21.15.67:0/2210819440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558e218ba240 0x558e218b93e0 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f82e8001fa0 tx=0x7f82e800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c815ba7ef4bd574 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:06.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.893+0000 7f82cffff640 1 -- 172.21.15.67:0/2210819440 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e8019070 con 0x558e218ba240 2024-03-24T00:48:06.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.893+0000 7f82cffff640 1 -- 172.21.15.67:0/2210819440 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82e800edf0 con 0x558e218ba240 2024-03-24T00:48:06.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.893+0000 7f82cffff640 1 -- 172.21.15.67:0/2210819440 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e8014680 con 0x558e218ba240 2024-03-24T00:48:06.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.894+0000 7f82f79ad740 1 -- 172.21.15.67:0/2210819440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558e218ba240 msgr2=0x558e218b93e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:06.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.894+0000 7f82f79ad740 1 --2- 172.21.15.67:0/2210819440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558e218ba240 0x558e218b93e0 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f82e8001fa0 tx=0x7f82e800e530 comp rx=0 tx=0).stop 2024-03-24T00:48:06.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.894+0000 7f82f79ad740 1 -- 172.21.15.67:0/2210819440 shutdown_connections 2024-03-24T00:48:06.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.894+0000 7f82f79ad740 1 --2- 172.21.15.67:0/2210819440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558e218ba240 0x558e218b93e0 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.894+0000 7f82f79ad740 1 --2- 172.21.15.67:0/2210819440 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558e218e0ed0 0x558e218b8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.894+0000 7f82f79ad740 1 --2- 172.21.15.67:0/2210819440 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558e218b9e60 0x558e218e0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.894+0000 7f82f79ad740 1 -- 172.21.15.67:0/2210819440 >> 172.21.15.67:0/2210819440 conn(0x558e218811a0 msgr2=0x558e218feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:06.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.894+0000 7f82f79ad740 1 -- 172.21.15.67:0/2210819440 shutdown_connections 2024-03-24T00:48:06.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.894+0000 7f82f79ad740 1 -- 172.21.15.67:0/2210819440 wait complete. 2024-03-24T00:48:06.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.895+0000 7f82f79ad740 1 -- start start 2024-03-24T00:48:06.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.895+0000 7f82f79ad740 1 Processor -- start 2024-03-24T00:48:06.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.896+0000 7f82f79ad740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558e218b9e60 0x558e218d9c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:06.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.896+0000 7f82f79ad740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558e218ba240 0x558e218de620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:06.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.896+0000 7f82f79ad740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558e218fbe10 0x558e218deb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:06.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.896+0000 7f82f79ad740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e217da0d0 con 0x558e218ba240 2024-03-24T00:48:06.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.896+0000 7f82f79ad740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e2181a110 con 0x558e218fbe10 2024-03-24T00:48:06.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.896+0000 7f82f79ad740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x558e218e0500 con 0x558e218b9e60 2024-03-24T00:48:06.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.896+0000 7f82f79ad740 10 client.0 _ll_register_callbacks cb 0x558e21815e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:48:06.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.896+0000 7f82edd9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558e218ba240 0x558e218de620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:06.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.896+0000 7f82edd9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558e218ba240 0x558e218de620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41110/0 (socket says 172.21.15.67:41110) 2024-03-24T00:48:06.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.896+0000 7f82edd9b640 1 -- 172.21.15.67:0/157735253 learned_addr learned my addr 172.21.15.67:0/157735253 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:06.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.896+0000 7f82ee59c640 1 --2- 172.21.15.67:0/157735253 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558e218b9e60 0x558e218d9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:06.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.896+0000 7f82eed9d640 1 --2- 172.21.15.67:0/157735253 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558e218fbe10 0x558e218deb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:06.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.897+0000 7f82edd9b640 1 -- 172.21.15.67:0/157735253 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558e218b9e60 msgr2=0x558e218d9c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:06.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.897+0000 7f82edd9b640 1 --2- 172.21.15.67:0/157735253 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558e218b9e60 0x558e218d9c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.897+0000 7f82edd9b640 1 -- 172.21.15.67:0/157735253 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558e218fbe10 msgr2=0x558e218deb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:06.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.897+0000 7f82edd9b640 1 --2- 172.21.15.67:0/157735253 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558e218fbe10 0x558e218deb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.897+0000 7f82edd9b640 1 -- 172.21.15.67:0/157735253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558e215fa7e0 con 0x558e218ba240 2024-03-24T00:48:06.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.897+0000 7f82edd9b640 1 --2- 172.21.15.67:0/157735253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558e218ba240 0x558e218de620 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f82e4006f80 tx=0x7f82e407ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bf12d023dc1119a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:06.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.897+0000 7f82cd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:48:06.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.897+0000 7f82cd7fa640 1 -- 172.21.15.67:0/157735253 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e4078860 con 0x558e218ba240 2024-03-24T00:48:06.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.897+0000 7f82cd7fa640 1 -- 172.21.15.67:0/157735253 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f82e4076bd0 con 0x558e218ba240 2024-03-24T00:48:06.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.897+0000 7f82f79ad740 1 -- 172.21.15.67:0/157735253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f82e800edf0 con 0x558e218ba240 2024-03-24T00:48:06.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.897+0000 7f82cd7fa640 1 -- 172.21.15.67:0/157735253 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) v2 ==== 24+0+0 (secure 0 0 0) 0x7f82e403ac80 con 0x558e218ba240 2024-03-24T00:48:06.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.897+0000 7f82f79ad740 10 client.8991 fetch_fsmap learned FSMap version 78 2024-03-24T00:48:06.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.897+0000 7f82f79ad740 1 -- 172.21.15.67:0/157735253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=2+}) v3 -- 0x558e2165af70 con 0x558e218ba240 2024-03-24T00:48:06.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.898+0000 7f82cd7fa640 1 -- 172.21.15.67:0/157735253 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82e800edf0 con 0x558e218ba240 2024-03-24T00:48:06.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.898+0000 7f82cd7fa640 1 -- 172.21.15.67:0/157735253 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 78) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f82e40aa170 con 0x558e218ba240 2024-03-24T00:48:06.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.898+0000 7f82f79ad740 10 client.8991 fetch_fsmap finished waiting for FSMap version 78 2024-03-24T00:48:06.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.898+0000 7f82f79ad740 10 client.8991 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.c 2024-03-24T00:48:06.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.898+0000 7f82f79ad740 20 client.8991 populate_metadata read hostname 'smithi067' 2024-03-24T00:48:06.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.898+0000 7f82f79ad740 1 --2- 172.21.15.67:0/157735253 >> [v2:172.21.15.67:6832/3197261495,v1:172.21.15.67:6833/3197261495] conn(0x558e218dcbf0 0x558e218dd030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:06.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.898+0000 7f82f79ad740 4 client.8991 mds_command: new command op to 8895 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:48:06.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.898+0000 7f82f79ad740 1 -- 172.21.15.67:0/157735253 --> [v2:172.21.15.67:6832/3197261495,v1:172.21.15.67:6833/3197261495] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558e2182b5b0 con 0x558e218dcbf0 2024-03-24T00:48:06.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.899+0000 7f82ee59c640 1 --2- 172.21.15.67:0/157735253 >> [v2:172.21.15.67:6832/3197261495,v1:172.21.15.67:6833/3197261495] conn(0x558e218dcbf0 0x558e218dd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:06.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.899+0000 7f82cd7fa640 10 client.8991 ms_handle_connect on v2:172.21.15.67:6832/3197261495 2024-03-24T00:48:06.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.899+0000 7f82ee59c640 1 --2- 172.21.15.67:0/157735253 >> [v2:172.21.15.67:6832/3197261495,v1:172.21.15.67:6833/3197261495] conn(0x558e218dcbf0 0x558e218dd030 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57980fe4a21827a5 server_cookie=d0d63a9f7deff74f in_seq=0 out_seq=0 2024-03-24T00:48:06.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.900+0000 7f82cd7fa640 1 -- 172.21.15.67:0/157735253 <== mds.0 v2:172.21.15.67:6832/3197261495 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x558e2182b5b0 con 0x558e218dcbf0 2024-03-24T00:48:06.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.900+0000 7f82cd7fa640 10 client.8991 handle_command_reply: tid=0 2024-03-24T00:48:06.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.900+0000 7f82f79ad740 1 client.8991 shutdown 2024-03-24T00:48:06.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.900+0000 7f82f79ad740 1 -- 172.21.15.67:0/157735253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558e218ba240 msgr2=0x558e218de620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:06.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.900+0000 7f82f79ad740 1 --2- 172.21.15.67:0/157735253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558e218ba240 0x558e218de620 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f82e4006f80 tx=0x7f82e407ada0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.900+0000 7f82f79ad740 1 -- 172.21.15.67:0/157735253 shutdown_connections 2024-03-24T00:48:06.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.900+0000 7f82f79ad740 1 --2- 172.21.15.67:0/157735253 >> [v2:172.21.15.67:6832/3197261495,v1:172.21.15.67:6833/3197261495] conn(0x558e218dcbf0 0x558e218dd030 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.900+0000 7f82f79ad740 1 --2- 172.21.15.67:0/157735253 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558e218fbe10 0x558e218deb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.900+0000 7f82cd7fa640 0 client.8991 ms_handle_reset on v2:172.21.15.67:6832/3197261495 2024-03-24T00:48:06.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.900+0000 7f82f79ad740 1 --2- 172.21.15.67:0/157735253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558e218ba240 0x558e218de620 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.900+0000 7f82f79ad740 1 --2- 172.21.15.67:0/157735253 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558e218b9e60 0x558e218d9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.900+0000 7f82f79ad740 1 -- 172.21.15.67:0/157735253 >> 172.21.15.67:0/157735253 conn(0x558e218811a0 msgr2=0x558e218dfad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:06.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.900+0000 7f82f79ad740 1 -- 172.21.15.67:0/157735253 shutdown_connections 2024-03-24T00:48:06.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.900+0000 7f82f79ad740 1 -- 172.21.15.67:0/157735253 wait complete. 2024-03-24T00:48:06.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.900+0000 7f82f79ad740 20 client.8991 trim_cache size 0 max 16384 2024-03-24T00:48:06.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.903+0000 7ff5d81d4640 1 Processor -- start 2024-03-24T00:48:06.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.903+0000 7ff5d81d4640 1 -- start start 2024-03-24T00:48:06.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.905+0000 7ff5d81d4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff5d00cd3f0 0x7ff5d00ccda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:06.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.905+0000 7ff5d81d4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff5d00b1fe0 0x7ff5d00cb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:06.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.905+0000 7ff5d81d4640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5d015e580 con 0x7ff5d015e1a0 2024-03-24T00:48:06.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.905+0000 7ff5d81d4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5d00cd880 con 0x7ff5d00b1fe0 2024-03-24T00:48:06.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.905+0000 7ff5d81d4640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d00cdb60 con 0x7ff5d00cd3f0 2024-03-24T00:48:06.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.905+0000 7ff5d5f70640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff5d015e1a0 0x7ff5d00b18d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48862/0 (socket says 172.21.15.67:48862) 2024-03-24T00:48:06.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.905+0000 7ff5d5f70640 1 -- 172.21.15.67:0/4146985248 learned_addr learned my addr 172.21.15.67:0/4146985248 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:48:06.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.905+0000 7ff5d576f640 1 --2- 172.21.15.67:0/4146985248 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff5d00cd3f0 0x7ff5d00ccda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:06.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.905+0000 7ff5d6771640 1 --2- 172.21.15.67:0/4146985248 >> v2:172.21.15.72:3300/0 conn(0x7ff5d00b1fe0 0x7ff5d00cb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:06.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.905+0000 7ff5d4f6e640 1 -- 172.21.15.67:0/4146985248 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 689747346 0 0) 0x7ff5d015e580 con 0x7ff5d015e1a0 2024-03-24T00:48:06.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.905+0000 7ff5d4f6e640 1 -- 172.21.15.67:0/4146985248 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5bc003520 con 0x7ff5d015e1a0 2024-03-24T00:48:06.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.905+0000 7ff5d576f640 1 -- 172.21.15.67:0/4146985248 >> v2:172.21.15.72:3300/0 conn(0x7ff5d00b1fe0 msgr2=0x7ff5d00cb4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:06.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.905+0000 7ff5d576f640 1 --2- 172.21.15.67:0/4146985248 >> v2:172.21.15.72:3300/0 conn(0x7ff5d00b1fe0 0x7ff5d00cb4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.905+0000 7ff5d576f640 1 -- 172.21.15.67:0/4146985248 >> v1:172.21.15.67:6789/0 conn(0x7ff5d015e1a0 legacy=0x7ff5d00b18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:06.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.905+0000 7ff5d4f6e640 1 -- 172.21.15.67:0/4146985248 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3546815409 0 0) 0x7ff5bc003520 con 0x7ff5d015e1a0 2024-03-24T00:48:06.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.905+0000 7ff5d576f640 1 -- 172.21.15.67:0/4146985248 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5d00cde40 con 0x7ff5d00cd3f0 2024-03-24T00:48:06.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.906+0000 7ff5d576f640 1 --2- 172.21.15.67:0/4146985248 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff5d00cd3f0 0x7ff5d00ccda0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff5c00097a0 tx=0x7ff5c002b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4af3662337a0c298 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:06.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.906+0000 7ff5d4f6e640 1 -- 172.21.15.67:0/4146985248 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5c0037070 con 0x7ff5d00cd3f0 2024-03-24T00:48:06.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.906+0000 7ff5d4f6e640 1 -- 172.21.15.67:0/4146985248 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5c0030510 con 0x7ff5d00cd3f0 2024-03-24T00:48:06.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.906+0000 7ff5d4f6e640 1 -- 172.21.15.67:0/4146985248 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5cc001670 con 0x7ff5d00cd3f0 2024-03-24T00:48:06.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.907+0000 7ff5d81d4640 1 -- 172.21.15.67:0/4146985248 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff5d00cd3f0 msgr2=0x7ff5d00ccda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:06.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.907+0000 7ff5d81d4640 1 --2- 172.21.15.67:0/4146985248 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff5d00cd3f0 0x7ff5d00ccda0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff5c00097a0 tx=0x7ff5c002b790 comp rx=0 tx=0).stop 2024-03-24T00:48:06.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.907+0000 7ff5d81d4640 1 -- 172.21.15.67:0/4146985248 shutdown_connections 2024-03-24T00:48:06.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.907+0000 7ff5d81d4640 1 --2- 172.21.15.67:0/4146985248 >> v2:172.21.15.72:3300/0 conn(0x7ff5d00b1fe0 0x7ff5d00cb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.907+0000 7ff5d81d4640 1 --2- 172.21.15.67:0/4146985248 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff5d00cd3f0 0x7ff5d00ccda0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.907+0000 7ff5d81d4640 1 -- 172.21.15.67:0/4146985248 >> 172.21.15.67:0/4146985248 conn(0x7ff5d00d0aa0 msgr2=0x7ff5d00d0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:06.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.907+0000 7ff5d81d4640 1 -- 172.21.15.67:0/4146985248 shutdown_connections 2024-03-24T00:48:06.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.907+0000 7ff5d81d4640 1 -- 172.21.15.67:0/4146985248 wait complete. 2024-03-24T00:48:06.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.909+0000 7ff5d81d4640 1 Processor -- start 2024-03-24T00:48:06.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.909+0000 7ff5d81d4640 1 -- start start 2024-03-24T00:48:06.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.910+0000 7ff5d81d4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff5d00b1fe0 0x7ff5d015ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:06.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.910+0000 7ff5d81d4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5d015e1a0 0x7ff5d0160e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:06.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.910+0000 7ff5d81d4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff5d014a940 0x7ff5d014adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:06.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.910+0000 7ff5d81d4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d00ce8a0 con 0x7ff5d015e1a0 2024-03-24T00:48:06.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.910+0000 7ff5d81d4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d00ce120 con 0x7ff5d00b1fe0 2024-03-24T00:48:06.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.910+0000 7ff5d81d4640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d00cea20 con 0x7ff5d014a940 2024-03-24T00:48:06.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.910+0000 7ff5d576f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5d015e1a0 0x7ff5d0160e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:06.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.910+0000 7ff5d576f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5d015e1a0 0x7ff5d0160e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41122/0 (socket says 172.21.15.67:41122) 2024-03-24T00:48:06.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.910+0000 7ff5d576f640 1 -- 172.21.15.67:0/1575450538 learned_addr learned my addr 172.21.15.67:0/1575450538 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:06.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.911+0000 7ff5d6771640 1 --2- 172.21.15.67:0/1575450538 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff5d014a940 0x7ff5d014adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:06.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.911+0000 7ff5d5f70640 1 --2- 172.21.15.67:0/1575450538 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff5d00b1fe0 0x7ff5d015ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:06.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.911+0000 7ff5d576f640 1 -- 172.21.15.67:0/1575450538 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff5d014a940 msgr2=0x7ff5d014adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:06.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.911+0000 7ff5d576f640 1 --2- 172.21.15.67:0/1575450538 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff5d014a940 0x7ff5d014adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.911+0000 7ff5d576f640 1 -- 172.21.15.67:0/1575450538 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff5d00b1fe0 msgr2=0x7ff5d015ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:06.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.911+0000 7ff5d576f640 1 --2- 172.21.15.67:0/1575450538 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff5d00b1fe0 0x7ff5d015ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.911+0000 7ff5d576f640 1 -- 172.21.15.67:0/1575450538 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5d00c9c10 con 0x7ff5d015e1a0 2024-03-24T00:48:06.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.911+0000 7ff5d576f640 1 --2- 172.21.15.67:0/1575450538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5d015e1a0 0x7ff5d0160e60 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7ff5c00097a0 tx=0x7ff5c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:06.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.911+0000 7ff5c67fc640 1 -- 172.21.15.67:0/1575450538 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5c0042070 con 0x7ff5d015e1a0 2024-03-24T00:48:06.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.911+0000 7ff5c67fc640 1 -- 172.21.15.67:0/1575450538 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5c003e870 con 0x7ff5d015e1a0 2024-03-24T00:48:06.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.911+0000 7ff5d81d4640 1 -- 172.21.15.67:0/1575450538 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5d00c9e90 con 0x7ff5d015e1a0 2024-03-24T00:48:06.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.911+0000 7ff5c67fc640 1 -- 172.21.15.67:0/1575450538 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5c0037040 con 0x7ff5d015e1a0 2024-03-24T00:48:06.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.911+0000 7ff5d81d4640 1 -- 172.21.15.67:0/1575450538 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5d00ca450 con 0x7ff5d015e1a0 2024-03-24T00:48:06.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.912+0000 7ff5d81d4640 1 -- 172.21.15.67:0/1575450538 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7ff5d004f400 con 0x7ff5d015e1a0 2024-03-24T00:48:06.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.913+0000 7ff5c67fc640 1 -- 172.21.15.67:0/1575450538 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff5c0045070 con 0x7ff5d015e1a0 2024-03-24T00:48:06.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.913+0000 7ff5c67fc640 1 --2- 172.21.15.67:0/1575450538 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff5b406bbd0 0x7ff5b406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:06.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.913+0000 7ff5c67fc640 1 -- 172.21.15.67:0/1575450538 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6216+0+0 (secure 0 0 0) 0x7ff5c00acd10 con 0x7ff5d015e1a0 2024-03-24T00:48:06.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.913+0000 7ff5c67fc640 1 -- 172.21.15.67:0/1575450538 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+512 (secure 0 0 0) 0x7ff5d00c9c10 con 0x7ff5d015e1a0 2024-03-24T00:48:06.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.913+0000 7ff5d5f70640 1 --2- 172.21.15.67:0/1575450538 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff5b406bbd0 0x7ff5b406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:06.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.914+0000 7ff5b3fff640 1 -- 172.21.15.67:0/1575450538 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({log-info=0+}) v3 -- 0x7ff5d015c1e0 con 0x7ff5d015e1a0 2024-03-24T00:48:06.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.914+0000 7ff5d5f70640 1 --2- 172.21.15.67:0/1575450538 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff5b406bbd0 0x7ff5b406e090 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc001320 tx=0x7ff5cc0012b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:06.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.918+0000 7f82f79ad740 1 Processor -- start 2024-03-24T00:48:06.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.918+0000 7f82f79ad740 1 -- start start 2024-03-24T00:48:06.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.919+0000 7f82f79ad740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558e218b9e60 0x558e218b2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:06.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.919+0000 7f82f79ad740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558e218ba240 0x558e218d7a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:06.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.919+0000 7f82f79ad740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558e218fbe10 0x558e218dcfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:06.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.919+0000 7f82f79ad740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f0200910 con 0x558e218fbe10 2024-03-24T00:48:06.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.919+0000 7f82f79ad740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f014e300 con 0x558e218ba240 2024-03-24T00:48:06.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.919+0000 7f82f79ad740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82f01f7c60 con 0x558e218b9e60 2024-03-24T00:48:06.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.919+0000 7f82eed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558e218fbe10 0x558e218dcfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:06.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.919+0000 7f82eed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558e218fbe10 0x558e218dcfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41126/0 (socket says 172.21.15.67:41126) 2024-03-24T00:48:06.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.919+0000 7f82eed9d640 1 -- 172.21.15.67:0/233059662 learned_addr learned my addr 172.21.15.67:0/233059662 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:06.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.919+0000 7f82ee59c640 1 --2- 172.21.15.67:0/233059662 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558e218b9e60 0x558e218b2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:06.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.919+0000 7f82edd9b640 1 --2- 172.21.15.67:0/233059662 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558e218ba240 0x558e218d7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:06.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.919+0000 7f82eed9d640 1 -- 172.21.15.67:0/233059662 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558e218b9e60 msgr2=0x558e218b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:06.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.919+0000 7f82eed9d640 1 --2- 172.21.15.67:0/233059662 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558e218b9e60 0x558e218b2ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.919+0000 7f82eed9d640 1 -- 172.21.15.67:0/233059662 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558e218ba240 msgr2=0x558e218d7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:06.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.919+0000 7f82eed9d640 1 --2- 172.21.15.67:0/233059662 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558e218ba240 0x558e218d7a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.919+0000 7f82eed9d640 1 -- 172.21.15.67:0/233059662 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558e214f2170 con 0x558e218fbe10 2024-03-24T00:48:06.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.919+0000 7f82ee59c640 1 --2- 172.21.15.67:0/233059662 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558e218b9e60 0x558e218b2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:48:06.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.919+0000 7f82edd9b640 1 --2- 172.21.15.67:0/233059662 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558e218ba240 0x558e218d7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:48:06.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.919+0000 7f82eed9d640 1 --2- 172.21.15.67:0/233059662 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558e218fbe10 0x558e218dcfd0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f82e8013000 tx=0x7f82e8007000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24fd7e4868e6020d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:06.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.919+0000 7f82cf7fe640 1 -- 172.21.15.67:0/233059662 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e8019040 con 0x558e218fbe10 2024-03-24T00:48:06.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.919+0000 7f82cf7fe640 1 -- 172.21.15.67:0/233059662 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82e8024070 con 0x558e218fbe10 2024-03-24T00:48:06.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.919+0000 7f82cf7fe640 1 -- 172.21.15.67:0/233059662 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e80146e0 con 0x558e218fbe10 2024-03-24T00:48:06.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.920+0000 7f82f79ad740 1 -- 172.21.15.67:0/233059662 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558e218fbe10 msgr2=0x558e218dcfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:06.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.920+0000 7f82f79ad740 1 --2- 172.21.15.67:0/233059662 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558e218fbe10 0x558e218dcfd0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f82e8013000 tx=0x7f82e8007000 comp rx=0 tx=0).stop 2024-03-24T00:48:06.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.920+0000 7f82f79ad740 1 -- 172.21.15.67:0/233059662 shutdown_connections 2024-03-24T00:48:06.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.920+0000 7f82f79ad740 1 --2- 172.21.15.67:0/233059662 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558e218fbe10 0x558e218dcfd0 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.920+0000 7f82f79ad740 1 --2- 172.21.15.67:0/233059662 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558e218ba240 0x558e218d7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.920+0000 7f82f79ad740 1 --2- 172.21.15.67:0/233059662 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558e218b9e60 0x558e218b2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.920+0000 7f82f79ad740 1 -- 172.21.15.67:0/233059662 >> 172.21.15.67:0/233059662 conn(0x558e218811a0 msgr2=0x558e218db070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:06.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.920+0000 7f82f79ad740 1 -- 172.21.15.67:0/233059662 shutdown_connections 2024-03-24T00:48:06.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.920+0000 7f82f79ad740 1 -- 172.21.15.67:0/233059662 wait complete. 2024-03-24T00:48:06.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.921+0000 7f82f79ad740 1 -- start start 2024-03-24T00:48:06.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.921+0000 7f82f79ad740 1 Processor -- start 2024-03-24T00:48:06.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.922+0000 7f82f79ad740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558e218b9e60 0x558e21921df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:06.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.922+0000 7f82f79ad740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558e218ba240 0x558e219112d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:06.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.922+0000 7f82f79ad740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558e218fbe10 0x558e218e6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:06.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.922+0000 7f82f79ad740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e218deeb0 con 0x558e218fbe10 2024-03-24T00:48:06.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.922+0000 7f82f79ad740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f014dd00 con 0x558e218ba240 2024-03-24T00:48:06.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.922+0000 7f82f79ad740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x558e218fbc20 con 0x558e218b9e60 2024-03-24T00:48:06.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.922+0000 7f82f79ad740 10 client.0 _ll_register_callbacks cb 0x7f82e407a0a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:48:06.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.922+0000 7f82eed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558e218fbe10 0x558e218e6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:06.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.922+0000 7f82eed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558e218fbe10 0x558e218e6d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41136/0 (socket says 172.21.15.67:41136) 2024-03-24T00:48:06.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.922+0000 7f82eed9d640 1 -- 172.21.15.67:0/3140788495 learned_addr learned my addr 172.21.15.67:0/3140788495 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:06.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.922+0000 7f82ee59c640 1 --2- 172.21.15.67:0/3140788495 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558e218b9e60 0x558e21921df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:06.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.922+0000 7f82edd9b640 1 --2- 172.21.15.67:0/3140788495 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558e218ba240 0x558e219112d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:06.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.922+0000 7f82eed9d640 1 -- 172.21.15.67:0/3140788495 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558e218b9e60 msgr2=0x558e21921df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:06.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.922+0000 7f82eed9d640 1 --2- 172.21.15.67:0/3140788495 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558e218b9e60 0x558e21921df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.922+0000 7f82eed9d640 1 -- 172.21.15.67:0/3140788495 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558e218ba240 msgr2=0x558e219112d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:06.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.922+0000 7f82eed9d640 1 --2- 172.21.15.67:0/3140788495 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558e218ba240 0x558e219112d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.922+0000 7f82eed9d640 1 -- 172.21.15.67:0/3140788495 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558e218f45b0 con 0x558e218fbe10 2024-03-24T00:48:06.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.922+0000 7f82edd9b640 1 --2- 172.21.15.67:0/3140788495 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558e218ba240 0x558e219112d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:48:06.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.922+0000 7f82eed9d640 1 --2- 172.21.15.67:0/3140788495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558e218fbe10 0x558e218e6d60 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f82e8010010 tx=0x7f82e8001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd6441bd8ca20a62 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:06.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.922+0000 7f82cd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:48:06.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.923+0000 7f82cd7fa640 1 -- 172.21.15.67:0/3140788495 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e802e020 con 0x558e218fbe10 2024-03-24T00:48:06.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.923+0000 7f82cd7fa640 1 -- 172.21.15.67:0/3140788495 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f82e8014480 con 0x558e218fbe10 2024-03-24T00:48:06.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.923+0000 7f82f79ad740 1 -- 172.21.15.67:0/3140788495 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f82e8024070 con 0x558e218fbe10 2024-03-24T00:48:06.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.923+0000 7f82cd7fa640 1 -- 172.21.15.67:0/3140788495 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) v2 ==== 24+0+0 (secure 0 0 0) 0x7f82e8019040 con 0x558e218fbe10 2024-03-24T00:48:06.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.923+0000 7f82f79ad740 10 client.9003 fetch_fsmap learned FSMap version 78 2024-03-24T00:48:06.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.923+0000 7f82f79ad740 1 -- 172.21.15.67:0/3140788495 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=2+}) v3 -- 0x558e21921220 con 0x558e218fbe10 2024-03-24T00:48:06.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.923+0000 7f82cd7fa640 1 -- 172.21.15.67:0/3140788495 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82e8024070 con 0x558e218fbe10 2024-03-24T00:48:06.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.923+0000 7f82cd7fa640 1 -- 172.21.15.67:0/3140788495 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 78) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f82e8004b60 con 0x558e218fbe10 2024-03-24T00:48:06.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.923+0000 7f82f79ad740 10 client.9003 fetch_fsmap finished waiting for FSMap version 78 2024-03-24T00:48:06.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.923+0000 7f82f79ad740 10 client.9003 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.c 2024-03-24T00:48:06.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.923+0000 7f82f79ad740 20 client.9003 populate_metadata read hostname 'smithi067' 2024-03-24T00:48:06.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.923+0000 7f82f79ad740 1 --2- 172.21.15.67:0/3140788495 >> [v2:172.21.15.67:6832/3197261495,v1:172.21.15.67:6833/3197261495] conn(0x558e21963dc0 0x558e218e72a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:06.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.923+0000 7f82f79ad740 4 client.9003 mds_command: new command op to 8895 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive"]}] 2024-03-24T00:48:06.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.923+0000 7f82f79ad740 1 -- 172.21.15.67:0/3140788495 --> [v2:172.21.15.67:6832/3197261495,v1:172.21.15.67:6833/3197261495] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive"]}) v1 -- 0x558e218dab70 con 0x558e21963dc0 2024-03-24T00:48:06.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.923+0000 7f82ee59c640 1 --2- 172.21.15.67:0/3140788495 >> [v2:172.21.15.67:6832/3197261495,v1:172.21.15.67:6833/3197261495] conn(0x558e21963dc0 0x558e218e72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:06.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.924+0000 7f82ee59c640 1 --2- 172.21.15.67:0/3140788495 >> [v2:172.21.15.67:6832/3197261495,v1:172.21.15.67:6833/3197261495] conn(0x558e21963dc0 0x558e218e72a0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22ef16ca71c69b85 server_cookie=2e6213b5987045d7 in_seq=0 out_seq=0 2024-03-24T00:48:06.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.924+0000 7f82cd7fa640 10 client.9003 ms_handle_connect on v2:172.21.15.67:6832/3197261495 2024-03-24T00:48:06.929 INFO:tasks.ceph.mds.c.smithi067.stderr:2024-03-24T00:48:06.924+0000 7efc3e158640 -1 log_channel(cluster) log [ERR] : scrub: inode wrongly marked free: 0x10000000001 2024-03-24T00:48:06.929 INFO:tasks.ceph.mds.c.smithi067.stderr:2024-03-24T00:48:06.924+0000 7efc3e158640 -1 log_channel(cluster) log [ERR] : inode table repaired for inode: 0x10000000001 2024-03-24T00:48:06.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.924+0000 7f82cd7fa640 1 -- 172.21.15.67:0/3140788495 <== mds.0 v2:172.21.15.67:6832/3197261495 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x558e218dab70 con 0x558e21963dc0 2024-03-24T00:48:06.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.924+0000 7f82cd7fa640 10 client.9003 handle_command_reply: tid=0 2024-03-24T00:48:06.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.924+0000 7f82f79ad740 1 client.9003 shutdown 2024-03-24T00:48:06.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.924+0000 7f82f79ad740 1 -- 172.21.15.67:0/3140788495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558e218fbe10 msgr2=0x558e218e6d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:06.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.924+0000 7f82f79ad740 1 --2- 172.21.15.67:0/3140788495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558e218fbe10 0x558e218e6d60 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f82e8010010 tx=0x7f82e8001250 comp rx=0 tx=0).stop 2024-03-24T00:48:06.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.925+0000 7f82f79ad740 1 -- 172.21.15.67:0/3140788495 shutdown_connections 2024-03-24T00:48:06.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.925+0000 7f82f79ad740 1 --2- 172.21.15.67:0/3140788495 >> [v2:172.21.15.67:6832/3197261495,v1:172.21.15.67:6833/3197261495] conn(0x558e21963dc0 0x558e218e72a0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.925+0000 7f82cd7fa640 0 client.9003 ms_handle_reset on v2:172.21.15.67:6832/3197261495 2024-03-24T00:48:06.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.925+0000 7f82f79ad740 1 --2- 172.21.15.67:0/3140788495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558e218fbe10 0x558e218e6d60 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.925+0000 7f82f79ad740 1 --2- 172.21.15.67:0/3140788495 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558e218ba240 0x558e219112d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.925+0000 7f82f79ad740 1 --2- 172.21.15.67:0/3140788495 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558e218b9e60 0x558e21921df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.925+0000 7f82f79ad740 1 -- 172.21.15.67:0/3140788495 >> 172.21.15.67:0/3140788495 conn(0x558e218811a0 msgr2=0x558e218b2ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:06.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.925+0000 7f82f79ad740 1 -- 172.21.15.67:0/3140788495 shutdown_connections 2024-03-24T00:48:06.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.925+0000 7f82f79ad740 1 -- 172.21.15.67:0/3140788495 wait complete. 2024-03-24T00:48:06.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.925+0000 7f82f79ad740 20 client.9003 trim_cache size 0 max 16384 2024-03-24T00:48:06.931 INFO:tasks.ceph.mds.c.smithi067.stderr:2024-03-24T00:48:06.925+0000 7efc3e158640 -1 log_channel(cluster) log [ERR] : scrub: inode wrongly marked free: 0x10000000002 2024-03-24T00:48:06.931 INFO:tasks.ceph.mds.c.smithi067.stderr:2024-03-24T00:48:06.925+0000 7efc3e158640 -1 log_channel(cluster) log [ERR] : inode table repaired for inode: 0x10000000002 2024-03-24T00:48:06.931 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:48:06.932 INFO:teuthology.orchestra.run.smithi067.stdout: "return_code": 0, 2024-03-24T00:48:06.932 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_tag": "77723f88-cdd1-4cac-8bac-4b8ebc9d1c43", 2024-03-24T00:48:06.932 INFO:teuthology.orchestra.run.smithi067.stdout: "mode": "asynchronous" 2024-03-24T00:48:06.932 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:48:06.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.926+0000 7f82f4ce8640 1 -- 172.21.15.67:0/652708072 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f82d006bbd0 msgr2=0x7f82d006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:06.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.926+0000 7f82f4ce8640 1 --2- 172.21.15.67:0/652708072 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f82d006bbd0 0x7f82d006e090 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f82e0009ad0 tx=0x7f82e0008040 comp rx=0 tx=0).stop 2024-03-24T00:48:06.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.926+0000 7f82f4ce8640 1 -- 172.21.15.67:0/652708072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82f00d0a20 msgr2=0x7f82f01513f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:06.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.926+0000 7f82f4ce8640 1 --2- 172.21.15.67:0/652708072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82f00d0a20 0x7f82f01513f0 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f82e4000c00 tx=0x7f82e40046c0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.926+0000 7f82f4ce8640 1 -- 172.21.15.67:0/652708072 shutdown_connections 2024-03-24T00:48:06.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.926+0000 7f82f4ce8640 1 --2- 172.21.15.67:0/652708072 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f82d006bbd0 0x7f82d006e090 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.926+0000 7f82f4ce8640 1 --2- 172.21.15.67:0/652708072 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f82f015e240 0x7f82f00cae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.926+0000 7f82f4ce8640 1 --2- 172.21.15.67:0/652708072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82f00d0a20 0x7f82f01513f0 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.926+0000 7f82f4ce8640 1 --2- 172.21.15.67:0/652708072 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f82f009a240 0x7f82f00b4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:06.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.926+0000 7f82f4ce8640 1 -- 172.21.15.67:0/652708072 >> 172.21.15.67:0/652708072 conn(0x7f82f00b36d0 msgr2=0x7f82f0071160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:06.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.926+0000 7f82f4ce8640 1 -- 172.21.15.67:0/652708072 shutdown_connections 2024-03-24T00:48:06.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:06.926+0000 7f82f4ce8640 1 -- 172.21.15.67:0/652708072 wait complete. 2024-03-24T00:48:06.937 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:48:06.937 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.7:0 scrub status 2024-03-24T00:48:07.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.040+0000 7f9c4159f640 1 Processor -- start 2024-03-24T00:48:07.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.040+0000 7f9c4159f640 1 -- start start 2024-03-24T00:48:07.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.041+0000 7f9c4159f640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f9c3c15aca0 0x7f9c3c0ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:07.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.041+0000 7f9c4159f640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9c3c0af890 0x7f9c3c0d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:07.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.041+0000 7f9c4159f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c3c15ea10 con 0x7f9c3c0d0df0 2024-03-24T00:48:07.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.041+0000 7f9c4159f640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c3c06ee70 con 0x7f9c3c15aca0 2024-03-24T00:48:07.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.041+0000 7f9c4159f640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3c15ecf0 con 0x7f9c3c0af890 2024-03-24T00:48:07.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.042+0000 7f9c40d9e640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9c3c0d0df0 0x7f9c3c0cd2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48876/0 (socket says 172.21.15.67:48876) 2024-03-24T00:48:07.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.042+0000 7f9c40d9e640 1 -- 172.21.15.67:0/2757041619 learned_addr learned my addr 172.21.15.67:0/2757041619 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:48:07.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.042+0000 7f9c3b7fe640 1 --2- 172.21.15.67:0/2757041619 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9c3c0af890 0x7f9c3c0d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:07.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.042+0000 7f9c3bfff640 1 --2- 172.21.15.67:0/2757041619 >> v2:172.21.15.72:3300/0 conn(0x7f9c3c15aca0 0x7f9c3c0ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:07.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.042+0000 7f9c3affd640 1 -- 172.21.15.67:0/2757041619 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2408569915 0 0) 0x7f9c3c15ea10 con 0x7f9c3c0d0df0 2024-03-24T00:48:07.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.042+0000 7f9c3affd640 1 -- 172.21.15.67:0/2757041619 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c24003520 con 0x7f9c3c0d0df0 2024-03-24T00:48:07.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.042+0000 7f9c3affd640 1 -- 172.21.15.67:0/2757041619 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 902941675 0 0) 0x7f9c24003520 con 0x7f9c3c0d0df0 2024-03-24T00:48:07.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.042+0000 7f9c3affd640 1 -- 172.21.15.67:0/2757041619 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9c3c0af890 msgr2=0x7f9c3c0d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:07.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.042+0000 7f9c3affd640 1 --2- 172.21.15.67:0/2757041619 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9c3c0af890 0x7f9c3c0d08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:07.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.042+0000 7f9c3affd640 1 -- 172.21.15.67:0/2757041619 >> v2:172.21.15.72:3300/0 conn(0x7f9c3c15aca0 msgr2=0x7f9c3c0ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:07.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.042+0000 7f9c3affd640 1 --2- 172.21.15.67:0/2757041619 >> v2:172.21.15.72:3300/0 conn(0x7f9c3c15aca0 0x7f9c3c0ce4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:07.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.042+0000 7f9c3affd640 1 -- 172.21.15.67:0/2757041619 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c3c15efd0 con 0x7f9c3c0d0df0 2024-03-24T00:48:07.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.042+0000 7f9c3affd640 1 -- 172.21.15.67:0/2757041619 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f9c340019f0 con 0x7f9c3c0d0df0 2024-03-24T00:48:07.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.042+0000 7f9c3affd640 1 -- 172.21.15.67:0/2757041619 >> v1:172.21.15.67:6789/0 conn(0x7f9c3c0d0df0 legacy=0x7f9c3c0cd2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:07.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.042+0000 7f9c3affd640 1 --2- 172.21.15.67:0/2757041619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c240039b0 0x7f9c24003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:07.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.042+0000 7f9c3affd640 1 --2- 172.21.15.67:0/2757041619 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9c24004800 0x7f9c24004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:07.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.042+0000 7f9c3affd640 1 --2- 172.21.15.67:0/2757041619 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9c3c0af890 0x7f9c240051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:07.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.042+0000 7f9c3affd640 1 -- 172.21.15.67:0/2757041619 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3c15ecf0 con 0x7f9c240039b0 2024-03-24T00:48:07.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.042+0000 7f9c3affd640 1 -- 172.21.15.67:0/2757041619 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2400b9c0 con 0x7f9c24004800 2024-03-24T00:48:07.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.042+0000 7f9c3affd640 1 -- 172.21.15.67:0/2757041619 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2400bb40 con 0x7f9c3c0af890 2024-03-24T00:48:07.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.042+0000 7f9c3bfff640 1 --2- 172.21.15.67:0/2757041619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c240039b0 0x7f9c24003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:07.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.042+0000 7f9c3b7fe640 1 --2- 172.21.15.67:0/2757041619 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9c24004800 0x7f9c24004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:07.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.042+0000 7f9c40d9e640 1 --2- 172.21.15.67:0/2757041619 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9c3c0af890 0x7f9c240051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:07.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.043+0000 7f9c3bfff640 1 -- 172.21.15.67:0/2757041619 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9c3c0af890 msgr2=0x7f9c240051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:07.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.043+0000 7f9c3bfff640 1 --2- 172.21.15.67:0/2757041619 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9c3c0af890 0x7f9c240051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:07.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.043+0000 7f9c3bfff640 1 -- 172.21.15.67:0/2757041619 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9c24004800 msgr2=0x7f9c24004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:07.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.043+0000 7f9c3bfff640 1 --2- 172.21.15.67:0/2757041619 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9c24004800 0x7f9c24004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:07.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.043+0000 7f9c3bfff640 1 -- 172.21.15.67:0/2757041619 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c24003520 con 0x7f9c240039b0 2024-03-24T00:48:07.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.043+0000 7f9c40d9e640 1 --2- 172.21.15.67:0/2757041619 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9c3c0af890 0x7f9c240051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:48:07.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.043+0000 7f9c3bfff640 1 --2- 172.21.15.67:0/2757041619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c240039b0 0x7f9c24003dd0 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c009af0 tx=0x7f9c24005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=412afd3ed6057902 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:07.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.043+0000 7f9c3affd640 1 -- 172.21.15.67:0/2757041619 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c2c039070 con 0x7f9c240039b0 2024-03-24T00:48:07.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.044+0000 7f9c3affd640 1 -- 172.21.15.67:0/2757041619 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c2c02c710 con 0x7f9c240039b0 2024-03-24T00:48:07.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.044+0000 7f9c3affd640 1 -- 172.21.15.67:0/2757041619 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c2c009690 con 0x7f9c240039b0 2024-03-24T00:48:07.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.044+0000 7f9c4159f640 1 -- 172.21.15.67:0/2757041619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c240039b0 msgr2=0x7f9c24003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:07.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.044+0000 7f9c4159f640 1 --2- 172.21.15.67:0/2757041619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c240039b0 0x7f9c24003dd0 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c009af0 tx=0x7f9c24005830 comp rx=0 tx=0).stop 2024-03-24T00:48:07.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.044+0000 7f9c40d9e640 1 -- 172.21.15.67:0/2757041619 reap_dead start 2024-03-24T00:48:07.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.044+0000 7f9c4159f640 1 -- 172.21.15.67:0/2757041619 shutdown_connections 2024-03-24T00:48:07.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.044+0000 7f9c4159f640 1 -- 172.21.15.67:0/2757041619 >> 172.21.15.67:0/2757041619 conn(0x7f9c3c099a30 msgr2=0x7f9c3c099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:07.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.044+0000 7f9c4159f640 1 -- 172.21.15.67:0/2757041619 shutdown_connections 2024-03-24T00:48:07.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.044+0000 7f9c4159f640 1 -- 172.21.15.67:0/2757041619 wait complete. 2024-03-24T00:48:07.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.045+0000 7f9c4159f640 1 Processor -- start 2024-03-24T00:48:07.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.045+0000 7f9c4159f640 1 -- start start 2024-03-24T00:48:07.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.046+0000 7f9c4159f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c240039b0 0x7f9c3c0ca160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:07.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.046+0000 7f9c4159f640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9c3c0b2630 0x7f9c3c14fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:07.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.046+0000 7f9c4159f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9c3c0b2ab0 0x7f9c3c14ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:07.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.046+0000 7f9c4159f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3c15b080 con 0x7f9c240039b0 2024-03-24T00:48:07.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.046+0000 7f9c4159f640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3c15f590 con 0x7f9c3c0b2ab0 2024-03-24T00:48:07.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.046+0000 7f9c4159f640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3c0cc650 con 0x7f9c3c0b2630 2024-03-24T00:48:07.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.046+0000 7f9c3bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c240039b0 0x7f9c3c0ca160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:07.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.046+0000 7f9c3bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c240039b0 0x7f9c3c0ca160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41160/0 (socket says 172.21.15.67:41160) 2024-03-24T00:48:07.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.046+0000 7f9c3bfff640 1 -- 172.21.15.67:0/1160443881 learned_addr learned my addr 172.21.15.67:0/1160443881 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:07.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.046+0000 7f9c3b7fe640 1 --2- 172.21.15.67:0/1160443881 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9c3c0b2630 0x7f9c3c14fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:07.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.046+0000 7f9c40d9e640 1 --2- 172.21.15.67:0/1160443881 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9c3c0b2ab0 0x7f9c3c14ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:07.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.047+0000 7f9c3bfff640 1 -- 172.21.15.67:0/1160443881 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9c3c0b2630 msgr2=0x7f9c3c14fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:07.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.047+0000 7f9c3bfff640 1 --2- 172.21.15.67:0/1160443881 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9c3c0b2630 0x7f9c3c14fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:07.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.047+0000 7f9c3bfff640 1 -- 172.21.15.67:0/1160443881 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9c3c0b2ab0 msgr2=0x7f9c3c14ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:07.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.047+0000 7f9c3bfff640 1 --2- 172.21.15.67:0/1160443881 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9c3c0b2ab0 0x7f9c3c14ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:07.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.047+0000 7f9c3bfff640 1 -- 172.21.15.67:0/1160443881 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c3c0b3520 con 0x7f9c240039b0 2024-03-24T00:48:07.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.047+0000 7f9c40d9e640 1 --2- 172.21.15.67:0/1160443881 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9c3c0b2ab0 0x7f9c3c14ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:48:07.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.047+0000 7f9c3b7fe640 1 --2- 172.21.15.67:0/1160443881 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9c3c0b2630 0x7f9c3c14fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:48:07.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.047+0000 7f9c3bfff640 1 --2- 172.21.15.67:0/1160443881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c240039b0 0x7f9c3c0ca160 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c037040 tx=0x7f9c2c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:07.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.047+0000 7f9c38ff9640 1 -- 172.21.15.67:0/1160443881 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c2c039050 con 0x7f9c240039b0 2024-03-24T00:48:07.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.047+0000 7f9c38ff9640 1 -- 172.21.15.67:0/1160443881 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c2c045070 con 0x7f9c240039b0 2024-03-24T00:48:07.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.047+0000 7f9c4159f640 1 -- 172.21.15.67:0/1160443881 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c3c0c66a0 con 0x7f9c240039b0 2024-03-24T00:48:07.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.047+0000 7f9c38ff9640 1 -- 172.21.15.67:0/1160443881 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c2c003a50 con 0x7f9c240039b0 2024-03-24T00:48:07.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.047+0000 7f9c4159f640 1 -- 172.21.15.67:0/1160443881 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c3c0c6c60 con 0x7f9c240039b0 2024-03-24T00:48:07.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.049+0000 7f9c38ff9640 1 -- 172.21.15.67:0/1160443881 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9c2c003bf0 con 0x7f9c240039b0 2024-03-24T00:48:07.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.049+0000 7f9c38ff9640 1 --2- 172.21.15.67:0/1160443881 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9c2806bbd0 0x7f9c2806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:07.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.049+0000 7f9c38ff9640 1 -- 172.21.15.67:0/1160443881 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f9c2c0ac700 con 0x7f9c240039b0 2024-03-24T00:48:07.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.049+0000 7f9c3b7fe640 1 --2- 172.21.15.67:0/1160443881 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9c2806bbd0 0x7f9c2806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:07.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.050+0000 7f9c453e0740 1 Processor -- start 2024-03-24T00:48:07.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.050+0000 7f9c453e0740 1 -- start start 2024-03-24T00:48:07.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.050+0000 7f9c3b7fe640 1 --2- 172.21.15.67:0/1160443881 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9c2806bbd0 0x7f9c2806e090 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c0cd840 tx=0x7f9c30008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:07.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.050+0000 7f9c453e0740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ed5c4ad030 0x55ed5c47ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:07.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.050+0000 7f9c453e0740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed5c4ad410 0x55ed5c4afdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:07.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.051+0000 7f9c453e0740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ed5c480510 0x55ed5c4b0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:07.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.051+0000 7f9c453e0740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed5c3ea3d0 con 0x55ed5c4ad410 2024-03-24T00:48:07.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.051+0000 7f9c453e0740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed5c392ad0 con 0x55ed5c4ad030 2024-03-24T00:48:07.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.051+0000 7f9c453e0740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ed5c3e6470 con 0x55ed5c480510 2024-03-24T00:48:07.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.051+0000 7f9c3b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed5c4ad410 0x55ed5c4afdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:07.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.051+0000 7f9c3b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed5c4ad410 0x55ed5c4afdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41176/0 (socket says 172.21.15.67:41176) 2024-03-24T00:48:07.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.051+0000 7f9c3b7fe640 1 -- 172.21.15.67:0/2759674476 learned_addr learned my addr 172.21.15.67:0/2759674476 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:07.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.051+0000 7f9c3bfff640 1 --2- 172.21.15.67:0/2759674476 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ed5c4ad030 0x55ed5c47ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:07.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.051+0000 7f9c3b7fe640 1 -- 172.21.15.67:0/2759674476 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ed5c480510 msgr2=0x55ed5c4b0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:07.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.051+0000 7f9c40d9e640 1 --2- 172.21.15.67:0/2759674476 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ed5c480510 0x55ed5c4b0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:07.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.051+0000 7f9c3b7fe640 1 --2- 172.21.15.67:0/2759674476 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ed5c480510 0x55ed5c4b0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:07.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.051+0000 7f9c3b7fe640 1 -- 172.21.15.67:0/2759674476 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ed5c4ad030 msgr2=0x55ed5c47ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:07.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.051+0000 7f9c3b7fe640 1 --2- 172.21.15.67:0/2759674476 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ed5c4ad030 0x55ed5c47ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:07.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.051+0000 7f9c3b7fe640 1 -- 172.21.15.67:0/2759674476 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ed5c1c87d0 con 0x55ed5c4ad410 2024-03-24T00:48:07.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.051+0000 7f9c3bfff640 1 --2- 172.21.15.67:0/2759674476 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ed5c4ad030 0x55ed5c47ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:48:07.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.051+0000 7f9c3b7fe640 1 --2- 172.21.15.67:0/2759674476 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed5c4ad410 0x55ed5c4afdc0 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f9c30013e40 tx=0x7f9c30016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=761c9b8bc7df5bde server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:07.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.051+0000 7f9c21ffb640 1 -- 172.21.15.67:0/2759674476 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c30013e70 con 0x55ed5c4ad410 2024-03-24T00:48:07.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.051+0000 7f9c21ffb640 1 -- 172.21.15.67:0/2759674476 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c30028070 con 0x55ed5c4ad410 2024-03-24T00:48:07.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.051+0000 7f9c21ffb640 1 -- 172.21.15.67:0/2759674476 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c30023660 con 0x55ed5c4ad410 2024-03-24T00:48:07.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.052+0000 7f9c453e0740 1 -- 172.21.15.67:0/2759674476 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed5c4ad410 msgr2=0x55ed5c4afdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:07.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.052+0000 7f9c453e0740 1 --2- 172.21.15.67:0/2759674476 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed5c4ad410 0x55ed5c4afdc0 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f9c30013e40 tx=0x7f9c30016000 comp rx=0 tx=0).stop 2024-03-24T00:48:07.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.052+0000 7f9c453e0740 1 -- 172.21.15.67:0/2759674476 shutdown_connections 2024-03-24T00:48:07.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.052+0000 7f9c453e0740 1 --2- 172.21.15.67:0/2759674476 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ed5c480510 0x55ed5c4b0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:07.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.052+0000 7f9c453e0740 1 --2- 172.21.15.67:0/2759674476 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed5c4ad410 0x55ed5c4afdc0 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:07.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.052+0000 7f9c453e0740 1 --2- 172.21.15.67:0/2759674476 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ed5c4ad030 0x55ed5c47ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:07.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.052+0000 7f9c453e0740 1 -- 172.21.15.67:0/2759674476 >> 172.21.15.67:0/2759674476 conn(0x55ed5c482240 msgr2=0x55ed5c4cadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:07.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.052+0000 7f9c453e0740 1 -- 172.21.15.67:0/2759674476 shutdown_connections 2024-03-24T00:48:07.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.052+0000 7f9c453e0740 1 -- 172.21.15.67:0/2759674476 wait complete. 2024-03-24T00:48:07.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.052+0000 7f9c453e0740 1 -- start start 2024-03-24T00:48:07.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.053+0000 7f9c453e0740 1 Processor -- start 2024-03-24T00:48:07.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.053+0000 7f9c453e0740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ed5c4ad030 0x55ed5c4b5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:07.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.053+0000 7f9c453e0740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ed5c4ad410 0x55ed5c4ac6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:07.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.053+0000 7f9c453e0740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed5c4ca380 0x55ed5c4ca800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:07.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.053+0000 7f9c453e0740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed5c4848e0 con 0x55ed5c4ca380 2024-03-24T00:48:07.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.053+0000 7f9c453e0740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed5c3a6060 con 0x55ed5c4ad410 2024-03-24T00:48:07.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.053+0000 7f9c453e0740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ed5c487120 con 0x55ed5c4ad030 2024-03-24T00:48:07.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.054+0000 7f9c453e0740 10 client.0 _ll_register_callbacks cb 0x55ed5c3886a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:48:07.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.054+0000 7f9c40d9e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed5c4ca380 0x55ed5c4ca800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:07.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.054+0000 7f9c40d9e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed5c4ca380 0x55ed5c4ca800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41178/0 (socket says 172.21.15.67:41178) 2024-03-24T00:48:07.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.054+0000 7f9c40d9e640 1 -- 172.21.15.67:0/172182764 learned_addr learned my addr 172.21.15.67:0/172182764 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:07.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.054+0000 7f9c3bfff640 1 --2- 172.21.15.67:0/172182764 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ed5c4ad030 0x55ed5c4b5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:07.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.054+0000 7f9c3b7fe640 1 --2- 172.21.15.67:0/172182764 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ed5c4ad410 0x55ed5c4ac6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:07.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.054+0000 7f9c40d9e640 1 -- 172.21.15.67:0/172182764 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ed5c4ad030 msgr2=0x55ed5c4b5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:07.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.054+0000 7f9c40d9e640 1 --2- 172.21.15.67:0/172182764 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ed5c4ad030 0x55ed5c4b5cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:07.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.054+0000 7f9c40d9e640 1 -- 172.21.15.67:0/172182764 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ed5c4ad410 msgr2=0x55ed5c4ac6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:07.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.054+0000 7f9c40d9e640 1 --2- 172.21.15.67:0/172182764 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ed5c4ad410 0x55ed5c4ac6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:07.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.054+0000 7f9c40d9e640 1 -- 172.21.15.67:0/172182764 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ed5c0fe3f0 con 0x55ed5c4ca380 2024-03-24T00:48:07.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.054+0000 7f9c40d9e640 1 --2- 172.21.15.67:0/172182764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed5c4ca380 0x55ed5c4ca800 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c072960 tx=0x7f9c34007a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a75a051770a33825 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:07.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.054+0000 7f9c1affd640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:48:07.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.054+0000 7f9c1affd640 1 -- 172.21.15.67:0/172182764 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c3400d030 con 0x55ed5c4ca380 2024-03-24T00:48:07.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.054+0000 7f9c1affd640 1 -- 172.21.15.67:0/172182764 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f9c34004d60 con 0x55ed5c4ca380 2024-03-24T00:48:07.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.054+0000 7f9c453e0740 1 -- 172.21.15.67:0/172182764 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9c30028070 con 0x55ed5c4ca380 2024-03-24T00:48:07.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.054+0000 7f9c1affd640 1 -- 172.21.15.67:0/172182764 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9c34010040 con 0x55ed5c4ca380 2024-03-24T00:48:07.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.054+0000 7f9c453e0740 10 client.9015 fetch_fsmap learned FSMap version 78 2024-03-24T00:48:07.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.054+0000 7f9c453e0740 1 -- 172.21.15.67:0/172182764 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=2+}) v3 -- 0x55ed5c0be0f0 con 0x55ed5c4ca380 2024-03-24T00:48:07.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.055+0000 7f9c1affd640 1 -- 172.21.15.67:0/172182764 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c30028070 con 0x55ed5c4ca380 2024-03-24T00:48:07.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.055+0000 7f9c1affd640 1 -- 172.21.15.67:0/172182764 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 78) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f9c3401d050 con 0x55ed5c4ca380 2024-03-24T00:48:07.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.055+0000 7f9c453e0740 10 client.9015 fetch_fsmap finished waiting for FSMap version 78 2024-03-24T00:48:07.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.055+0000 7f9c453e0740 10 client.9015 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.c 2024-03-24T00:48:07.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.055+0000 7f9c453e0740 20 client.9015 populate_metadata read hostname 'smithi067' 2024-03-24T00:48:07.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.055+0000 7f9c453e0740 1 --2- 172.21.15.67:0/172182764 >> [v2:172.21.15.67:6832/3197261495,v1:172.21.15.67:6833/3197261495] conn(0x55ed5c4dbcf0 0x55ed5c4b0f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:07.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.055+0000 7f9c453e0740 4 client.9015 mds_command: new command op to 8895 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:48:07.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.055+0000 7f9c453e0740 1 -- 172.21.15.67:0/172182764 --> [v2:172.21.15.67:6832/3197261495,v1:172.21.15.67:6833/3197261495] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ed5c3f7910 con 0x55ed5c4dbcf0 2024-03-24T00:48:07.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.055+0000 7f9c3bfff640 1 --2- 172.21.15.67:0/172182764 >> [v2:172.21.15.67:6832/3197261495,v1:172.21.15.67:6833/3197261495] conn(0x55ed5c4dbcf0 0x55ed5c4b0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:07.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.055+0000 7f9c3bfff640 1 --2- 172.21.15.67:0/172182764 >> [v2:172.21.15.67:6832/3197261495,v1:172.21.15.67:6833/3197261495] conn(0x55ed5c4dbcf0 0x55ed5c4b0f60 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5106ac2e560aeaa7 server_cookie=20e2f1552318e69a in_seq=0 out_seq=0 2024-03-24T00:48:07.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.055+0000 7f9c1affd640 10 client.9015 ms_handle_connect on v2:172.21.15.67:6832/3197261495 2024-03-24T00:48:07.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.056+0000 7f9c1affd640 1 -- 172.21.15.67:0/172182764 <== mds.0 v2:172.21.15.67:6832/3197261495 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ed5c3f7910 con 0x55ed5c4dbcf0 2024-03-24T00:48:07.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.056+0000 7f9c1affd640 10 client.9015 handle_command_reply: tid=0 2024-03-24T00:48:07.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.056+0000 7f9c453e0740 1 client.9015 shutdown 2024-03-24T00:48:07.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.056+0000 7f9c453e0740 1 -- 172.21.15.67:0/172182764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed5c4ca380 msgr2=0x55ed5c4ca800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:07.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.056+0000 7f9c453e0740 1 --2- 172.21.15.67:0/172182764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed5c4ca380 0x55ed5c4ca800 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c072960 tx=0x7f9c34007a70 comp rx=0 tx=0).stop 2024-03-24T00:48:07.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.056+0000 7f9c453e0740 1 -- 172.21.15.67:0/172182764 shutdown_connections 2024-03-24T00:48:07.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.056+0000 7f9c453e0740 1 --2- 172.21.15.67:0/172182764 >> [v2:172.21.15.67:6832/3197261495,v1:172.21.15.67:6833/3197261495] conn(0x55ed5c4dbcf0 0x55ed5c4b0f60 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:07.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.056+0000 7f9c453e0740 1 --2- 172.21.15.67:0/172182764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed5c4ca380 0x55ed5c4ca800 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:07.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.056+0000 7f9c453e0740 1 --2- 172.21.15.67:0/172182764 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ed5c4ad410 0x55ed5c4ac6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:07.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.056+0000 7f9c1affd640 0 client.9015 ms_handle_reset on v2:172.21.15.67:6832/3197261495 2024-03-24T00:48:07.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.056+0000 7f9c453e0740 1 --2- 172.21.15.67:0/172182764 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ed5c4ad030 0x55ed5c4b5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:07.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.056+0000 7f9c453e0740 1 -- 172.21.15.67:0/172182764 >> 172.21.15.67:0/172182764 conn(0x55ed5c480510 msgr2=0x55ed5c484050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:07.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.056+0000 7f9c453e0740 1 -- 172.21.15.67:0/172182764 shutdown_connections 2024-03-24T00:48:07.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.056+0000 7f9c453e0740 1 -- 172.21.15.67:0/172182764 wait complete. 2024-03-24T00:48:07.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.056+0000 7f9c453e0740 20 client.9015 trim_cache size 0 max 16384 2024-03-24T00:48:07.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.072+0000 7f9c453e0740 1 Processor -- start 2024-03-24T00:48:07.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.072+0000 7f9c453e0740 1 -- start start 2024-03-24T00:48:07.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.073+0000 7f9c453e0740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed5c4ad030 0x55ed5c4b2680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:07.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.073+0000 7f9c453e0740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ed5c4ad410 0x55ed5c4cdb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:07.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.073+0000 7f9c453e0740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ed5c4ca380 0x55ed5c4ce070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:07.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.073+0000 7f9c453e0740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3c200ea0 con 0x55ed5c4ad030 2024-03-24T00:48:07.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.073+0000 7f9c453e0740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3c200020 con 0x55ed5c4ad410 2024-03-24T00:48:07.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.073+0000 7f9c453e0740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3c1ff4c0 con 0x55ed5c4ca380 2024-03-24T00:48:07.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.073+0000 7f9c3bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed5c4ad030 0x55ed5c4b2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:07.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.073+0000 7f9c3bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed5c4ad030 0x55ed5c4b2680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41192/0 (socket says 172.21.15.67:41192) 2024-03-24T00:48:07.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.073+0000 7f9c3bfff640 1 -- 172.21.15.67:0/767598119 learned_addr learned my addr 172.21.15.67:0/767598119 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:07.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.073+0000 7f9c40d9e640 1 --2- 172.21.15.67:0/767598119 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ed5c4ca380 0x55ed5c4ce070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:07.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.073+0000 7f9c3b7fe640 1 --2- 172.21.15.67:0/767598119 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ed5c4ad410 0x55ed5c4cdb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:07.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.073+0000 7f9c3bfff640 1 -- 172.21.15.67:0/767598119 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ed5c4ca380 msgr2=0x55ed5c4ce070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:07.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.073+0000 7f9c3bfff640 1 --2- 172.21.15.67:0/767598119 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ed5c4ca380 0x55ed5c4ce070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:07.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.073+0000 7f9c3bfff640 1 -- 172.21.15.67:0/767598119 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ed5c4ad410 msgr2=0x55ed5c4cdb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:07.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.073+0000 7f9c3bfff640 1 --2- 172.21.15.67:0/767598119 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ed5c4ad410 0x55ed5c4cdb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:07.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.073+0000 7f9c3bfff640 1 -- 172.21.15.67:0/767598119 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ed5c4b6420 con 0x55ed5c4ad030 2024-03-24T00:48:07.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.073+0000 7f9c3bfff640 1 --2- 172.21.15.67:0/767598119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed5c4ad030 0x55ed5c4b2680 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c07a880 tx=0x7f9c2c0aa900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4b89f5596ea8bf6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:07.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.073+0000 7f9c217fa640 1 -- 172.21.15.67:0/767598119 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c2c07a8f0 con 0x55ed5c4ad030 2024-03-24T00:48:07.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.073+0000 7f9c217fa640 1 -- 172.21.15.67:0/767598119 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c2c0aae50 con 0x55ed5c4ad030 2024-03-24T00:48:07.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.073+0000 7f9c217fa640 1 -- 172.21.15.67:0/767598119 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c2c07ea40 con 0x55ed5c4ad030 2024-03-24T00:48:07.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.074+0000 7f9c453e0740 1 -- 172.21.15.67:0/767598119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed5c4ad030 msgr2=0x55ed5c4b2680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:07.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.074+0000 7f9c453e0740 1 --2- 172.21.15.67:0/767598119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed5c4ad030 0x55ed5c4b2680 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c07a880 tx=0x7f9c2c0aa900 comp rx=0 tx=0).stop 2024-03-24T00:48:07.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.074+0000 7f9c453e0740 1 -- 172.21.15.67:0/767598119 shutdown_connections 2024-03-24T00:48:07.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.074+0000 7f9c453e0740 1 --2- 172.21.15.67:0/767598119 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ed5c4ca380 0x55ed5c4ce070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:07.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.074+0000 7f9c453e0740 1 --2- 172.21.15.67:0/767598119 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ed5c4ad410 0x55ed5c4cdb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:07.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.074+0000 7f9c453e0740 1 --2- 172.21.15.67:0/767598119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed5c4ad030 0x55ed5c4b2680 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:07.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.074+0000 7f9c453e0740 1 -- 172.21.15.67:0/767598119 >> 172.21.15.67:0/767598119 conn(0x55ed5c480510 msgr2=0x55ed5c47ef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:07.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.074+0000 7f9c453e0740 1 -- 172.21.15.67:0/767598119 shutdown_connections 2024-03-24T00:48:07.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.074+0000 7f9c453e0740 1 -- 172.21.15.67:0/767598119 wait complete. 2024-03-24T00:48:07.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.074+0000 7f9c453e0740 1 -- start start 2024-03-24T00:48:07.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.075+0000 7f9c453e0740 1 Processor -- start 2024-03-24T00:48:07.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.075+0000 7f9c453e0740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ed5c4ad030 0x55ed5c4b2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:07.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.075+0000 7f9c453e0740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ed5c4ad410 0x55ed5c53dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:07.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.075+0000 7f9c453e0740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed5c4ca380 0x55ed5c4cdf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:07.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.075+0000 7f9c453e0740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3c1ffa20 con 0x55ed5c4ca380 2024-03-24T00:48:07.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.075+0000 7f9c453e0740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed5c3f7910 con 0x55ed5c4ad030 2024-03-24T00:48:07.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.075+0000 7f9c453e0740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ed5c4eeb90 con 0x55ed5c4ad410 2024-03-24T00:48:07.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.076+0000 7f9c453e0740 10 client.0 _ll_register_callbacks cb 0x7f9c34001490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:48:07.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.076+0000 7f9c40d9e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed5c4ca380 0x55ed5c4cdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:07.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.076+0000 7f9c40d9e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed5c4ca380 0x55ed5c4cdf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41208/0 (socket says 172.21.15.67:41208) 2024-03-24T00:48:07.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.076+0000 7f9c40d9e640 1 -- 172.21.15.67:0/1003397207 learned_addr learned my addr 172.21.15.67:0/1003397207 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:07.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.076+0000 7f9c3b7fe640 1 --2- 172.21.15.67:0/1003397207 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ed5c4ad410 0x55ed5c53dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:07.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.076+0000 7f9c3bfff640 1 --2- 172.21.15.67:0/1003397207 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ed5c4ad030 0x55ed5c4b2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:07.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.076+0000 7f9c40d9e640 1 -- 172.21.15.67:0/1003397207 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ed5c4ad410 msgr2=0x55ed5c53dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:07.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.076+0000 7f9c40d9e640 1 --2- 172.21.15.67:0/1003397207 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ed5c4ad410 0x55ed5c53dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:07.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.076+0000 7f9c40d9e640 1 -- 172.21.15.67:0/1003397207 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ed5c4ad030 msgr2=0x55ed5c4b2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:07.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.076+0000 7f9c40d9e640 1 --2- 172.21.15.67:0/1003397207 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ed5c4ad030 0x55ed5c4b2a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:07.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.076+0000 7f9c40d9e640 1 -- 172.21.15.67:0/1003397207 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ed5c4ce470 con 0x55ed5c4ca380 2024-03-24T00:48:07.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.076+0000 7f9c40d9e640 1 --2- 172.21.15.67:0/1003397207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed5c4ca380 0x55ed5c4cdf30 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f9c34000f80 tx=0x7f9c34001360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5b4784e5fbec05c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:07.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.076+0000 7f9c1affd640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:48:07.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.076+0000 7f9c1affd640 1 -- 172.21.15.67:0/1003397207 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c3400d040 con 0x55ed5c4ca380 2024-03-24T00:48:07.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.076+0000 7f9c1affd640 1 -- 172.21.15.67:0/1003397207 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f9c3400fc70 con 0x55ed5c4ca380 2024-03-24T00:48:07.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.076+0000 7f9c453e0740 1 -- 172.21.15.67:0/1003397207 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9c2c0aae50 con 0x55ed5c4ca380 2024-03-24T00:48:07.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.076+0000 7f9c1affd640 1 -- 172.21.15.67:0/1003397207 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9c3402d050 con 0x55ed5c4ca380 2024-03-24T00:48:07.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.076+0000 7f9c453e0740 10 client.9021 fetch_fsmap learned FSMap version 78 2024-03-24T00:48:07.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.076+0000 7f9c453e0740 1 -- 172.21.15.67:0/1003397207 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=2+}) v3 -- 0x55ed5c4e52a0 con 0x55ed5c4ca380 2024-03-24T00:48:07.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.077+0000 7f9c1affd640 1 -- 172.21.15.67:0/1003397207 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c2c0aae50 con 0x55ed5c4ca380 2024-03-24T00:48:07.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.077+0000 7f9c1affd640 1 -- 172.21.15.67:0/1003397207 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 78) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f9c34030020 con 0x55ed5c4ca380 2024-03-24T00:48:07.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.077+0000 7f9c453e0740 10 client.9021 fetch_fsmap finished waiting for FSMap version 78 2024-03-24T00:48:07.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.077+0000 7f9c453e0740 10 client.9021 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.c 2024-03-24T00:48:07.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.077+0000 7f9c453e0740 20 client.9021 populate_metadata read hostname 'smithi067' 2024-03-24T00:48:07.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.077+0000 7f9c453e0740 1 --2- 172.21.15.67:0/1003397207 >> [v2:172.21.15.67:6832/3197261495,v1:172.21.15.67:6833/3197261495] conn(0x55ed5c4edfe0 0x55ed5c539260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:07.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.077+0000 7f9c453e0740 4 client.9021 mds_command: new command op to 8895 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:48:07.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.077+0000 7f9c453e0740 1 -- 172.21.15.67:0/1003397207 --> [v2:172.21.15.67:6832/3197261495,v1:172.21.15.67:6833/3197261495] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55ed5c4b7260 con 0x55ed5c4edfe0 2024-03-24T00:48:07.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.077+0000 7f9c3bfff640 1 --2- 172.21.15.67:0/1003397207 >> [v2:172.21.15.67:6832/3197261495,v1:172.21.15.67:6833/3197261495] conn(0x55ed5c4edfe0 0x55ed5c539260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:07.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.077+0000 7f9c1affd640 10 client.9021 ms_handle_connect on v2:172.21.15.67:6832/3197261495 2024-03-24T00:48:07.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.077+0000 7f9c3bfff640 1 --2- 172.21.15.67:0/1003397207 >> [v2:172.21.15.67:6832/3197261495,v1:172.21.15.67:6833/3197261495] conn(0x55ed5c4edfe0 0x55ed5c539260 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a269f8b32ce8b3e server_cookie=37d7bfc30b7ef3f2 in_seq=0 out_seq=0 2024-03-24T00:48:07.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.077+0000 7f9c1affd640 1 -- 172.21.15.67:0/1003397207 <== mds.0 v2:172.21.15.67:6832/3197261495 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+260 (crc 0 0 0) 0x55ed5c4b7260 con 0x55ed5c4edfe0 2024-03-24T00:48:07.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.077+0000 7f9c1affd640 10 client.9021 handle_command_reply: tid=0 2024-03-24T00:48:07.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.077+0000 7f9c453e0740 1 client.9021 shutdown 2024-03-24T00:48:07.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.078+0000 7f9c453e0740 1 -- 172.21.15.67:0/1003397207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed5c4ca380 msgr2=0x55ed5c4cdf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:07.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.078+0000 7f9c453e0740 1 --2- 172.21.15.67:0/1003397207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed5c4ca380 0x55ed5c4cdf30 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f9c34000f80 tx=0x7f9c34001360 comp rx=0 tx=0).stop 2024-03-24T00:48:07.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.078+0000 7f9c453e0740 1 -- 172.21.15.67:0/1003397207 shutdown_connections 2024-03-24T00:48:07.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.078+0000 7f9c453e0740 1 --2- 172.21.15.67:0/1003397207 >> [v2:172.21.15.67:6832/3197261495,v1:172.21.15.67:6833/3197261495] conn(0x55ed5c4edfe0 0x55ed5c539260 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:07.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.078+0000 7f9c453e0740 1 --2- 172.21.15.67:0/1003397207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed5c4ca380 0x55ed5c4cdf30 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:07.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.078+0000 7f9c453e0740 1 --2- 172.21.15.67:0/1003397207 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55ed5c4ad410 0x55ed5c53dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:07.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.078+0000 7f9c453e0740 1 --2- 172.21.15.67:0/1003397207 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ed5c4ad030 0x55ed5c4b2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:07.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.078+0000 7f9c1affd640 0 client.9021 ms_handle_reset on v2:172.21.15.67:6832/3197261495 2024-03-24T00:48:07.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.078+0000 7f9c453e0740 1 -- 172.21.15.67:0/1003397207 >> 172.21.15.67:0/1003397207 conn(0x55ed5c480510 msgr2=0x55ed5c4c7770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:07.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.078+0000 7f9c453e0740 1 -- 172.21.15.67:0/1003397207 shutdown_connections 2024-03-24T00:48:07.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.078+0000 7f9c453e0740 1 -- 172.21.15.67:0/1003397207 wait complete. 2024-03-24T00:48:07.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.078+0000 7f9c453e0740 20 client.9021 trim_cache size 0 max 16384 2024-03-24T00:48:07.083 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:48:07.083 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T00:48:07.083 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": { 2024-03-24T00:48:07.083 INFO:teuthology.orchestra.run.smithi067.stdout: "77723f88-cdd1-4cac-8bac-4b8ebc9d1c43": { 2024-03-24T00:48:07.083 INFO:teuthology.orchestra.run.smithi067.stdout: "path": "/", 2024-03-24T00:48:07.083 INFO:teuthology.orchestra.run.smithi067.stdout: "tag": "77723f88-cdd1-4cac-8bac-4b8ebc9d1c43", 2024-03-24T00:48:07.083 INFO:teuthology.orchestra.run.smithi067.stdout: "options": "recursive,repair" 2024-03-24T00:48:07.083 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:48:07.084 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:48:07.084 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:48:07.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.078+0000 7f9c4159f640 1 -- 172.21.15.67:0/1160443881 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9c2806bbd0 msgr2=0x7f9c2806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:07.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.078+0000 7f9c4159f640 1 --2- 172.21.15.67:0/1160443881 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9c2806bbd0 0x7f9c2806e090 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c0cd840 tx=0x7f9c30008040 comp rx=0 tx=0).stop 2024-03-24T00:48:07.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.078+0000 7f9c4159f640 1 -- 172.21.15.67:0/1160443881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c240039b0 msgr2=0x7f9c3c0ca160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:07.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.078+0000 7f9c4159f640 1 --2- 172.21.15.67:0/1160443881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c240039b0 0x7f9c3c0ca160 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c037040 tx=0x7f9c2c004730 comp rx=0 tx=0).stop 2024-03-24T00:48:07.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.078+0000 7f9c4159f640 1 -- 172.21.15.67:0/1160443881 shutdown_connections 2024-03-24T00:48:07.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.078+0000 7f9c4159f640 1 --2- 172.21.15.67:0/1160443881 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9c2806bbd0 0x7f9c2806e090 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:07.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.078+0000 7f9c4159f640 1 --2- 172.21.15.67:0/1160443881 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9c3c0b2ab0 0x7f9c3c14ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:07.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.078+0000 7f9c4159f640 1 --2- 172.21.15.67:0/1160443881 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9c3c0b2630 0x7f9c3c14fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:07.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.078+0000 7f9c4159f640 1 --2- 172.21.15.67:0/1160443881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c240039b0 0x7f9c3c0ca160 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:07.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.078+0000 7f9c4159f640 1 -- 172.21.15.67:0/1160443881 >> 172.21.15.67:0/1160443881 conn(0x7f9c3c099a30 msgr2=0x7f9c3c06eed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:07.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.079+0000 7f9c4159f640 1 -- 172.21.15.67:0/1160443881 shutdown_connections 2024-03-24T00:48:07.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.079+0000 7f9c4159f640 1 -- 172.21.15.67:0/1160443881 wait complete. 2024-03-24T00:48:07.089 INFO:tasks.cephfs.filesystem:scrub status for tag:77723f88-cdd1-4cac-8bac-4b8ebc9d1c43 - {'path': '/', 'tag': '77723f88-cdd1-4cac-8bac-4b8ebc9d1c43', 'options': 'recursive,repair'} 2024-03-24T00:48:07.336 INFO:tasks.ceph.mds.c.smithi067.stderr:2024-03-24T00:48:07.336+0000 7efc42160640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T00:48:07.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:07.545+0000 7ff5c67fc640 1 -- 172.21.15.67:0/1575450538 <== mon.0 v2:172.21.15.67:3300/0 7 ==== log(8 entries from seq 1 at 2024-03-24T00:48:06.924617+0000) v1 ==== 1407+0+0 (secure 0 0 0) 0x7ff5c00ab150 con 0x7ff5d015e1a0 2024-03-24T00:48:07.545 INFO:teuthology.orchestra.run.smithi067.stdout: cluster: 2024-03-24T00:48:07.545 INFO:teuthology.orchestra.run.smithi067.stdout: id: de1d0178-808e-49ca-a986-e297e523b976 2024-03-24T00:48:07.545 INFO:teuthology.orchestra.run.smithi067.stdout: health: HEALTH_OK 2024-03-24T00:48:07.546 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:48:07.546 INFO:teuthology.orchestra.run.smithi067.stdout: services: 2024-03-24T00:48:07.546 INFO:teuthology.orchestra.run.smithi067.stdout: mon: 3 daemons, quorum a,b,c (age 9m) 2024-03-24T00:48:07.546 INFO:teuthology.orchestra.run.smithi067.stdout: mgr: x(active, since 9m), standbys: y 2024-03-24T00:48:07.546 INFO:teuthology.orchestra.run.smithi067.stdout: mds: 1/1 daemons up, 3 standby 2024-03-24T00:48:07.546 INFO:teuthology.orchestra.run.smithi067.stdout: osd: 8 osds: 8 up (since 9m), 8 in (since 9m) 2024-03-24T00:48:07.546 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:48:07.546 INFO:teuthology.orchestra.run.smithi067.stdout: data: 2024-03-24T00:48:07.546 INFO:teuthology.orchestra.run.smithi067.stdout: volumes: 1/1 healthy 2024-03-24T00:48:07.546 INFO:teuthology.orchestra.run.smithi067.stdout: pools: 3 pools, 97 pgs 2024-03-24T00:48:07.546 INFO:teuthology.orchestra.run.smithi067.stdout: objects: 30 objects, 19 MiB 2024-03-24T00:48:07.546 INFO:teuthology.orchestra.run.smithi067.stdout: usage: 291 MiB used, 720 GiB / 720 GiB avail 2024-03-24T00:48:07.546 INFO:teuthology.orchestra.run.smithi067.stdout: pgs: 97 active+clean 2024-03-24T00:48:07.547 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:48:07.547 INFO:teuthology.orchestra.run.smithi067.stdout: io: 2024-03-24T00:48:07.547 INFO:teuthology.orchestra.run.smithi067.stdout: client: 1.8 KiB/s rd, 102 B/s wr, 1 op/s rd, 0 op/s wr 2024-03-24T00:48:07.547 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:48:07.547 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:48:07.547 INFO:teuthology.orchestra.run.smithi067.stdout:2024-03-24T00:48:06.924617+0000 mds.c [INF] scrub queued for path: / 2024-03-24T00:48:07.547 INFO:teuthology.orchestra.run.smithi067.stdout:2024-03-24T00:48:06.924622+0000 mds.c [INF] scrub summary: idle+waiting paths [/] 2024-03-24T00:48:07.547 INFO:teuthology.orchestra.run.smithi067.stdout:2024-03-24T00:48:06.924629+0000 mds.c [INF] scrub summary: active paths [/] 2024-03-24T00:48:07.547 INFO:teuthology.orchestra.run.smithi067.stdout:2024-03-24T00:48:06.925171+0000 mds.c [ERR] scrub: inode wrongly marked free: 0x10000000001 2024-03-24T00:48:07.547 INFO:teuthology.orchestra.run.smithi067.stdout:2024-03-24T00:48:06.925179+0000 mds.c [ERR] inode table repaired for inode: 0x10000000001 2024-03-24T00:48:07.547 INFO:teuthology.orchestra.run.smithi067.stdout:2024-03-24T00:48:06.925975+0000 mds.c [ERR] scrub: inode wrongly marked free: 0x10000000002 2024-03-24T00:48:07.547 INFO:teuthology.orchestra.run.smithi067.stdout:2024-03-24T00:48:06.925979+0000 mds.c [ERR] inode table repaired for inode: 0x10000000002 2024-03-24T00:48:07.547 INFO:teuthology.orchestra.run.smithi067.stdout:2024-03-24T00:48:06.926330+0000 mds.c [INF] scrub summary: idle+waiting paths [/] 2024-03-24T00:48:12.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:12.549+0000 7ff5c67fc640 1 -- 172.21.15.67:0/1575450538 <== mon.0 v2:172.21.15.67:3300/0 8 ==== log(1 entries from seq 9 at 2024-03-24T00:48:11.365148+0000) v1 ==== 189+0+0 (secure 0 0 0) 0x7ff5c0079050 con 0x7ff5d015e1a0 2024-03-24T00:48:12.549 INFO:teuthology.orchestra.run.smithi067.stdout:2024-03-24T00:48:11.365148+0000 mds.c [INF] scrub summary: idle 2024-03-24T00:48:17.073 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:48:17.076 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:48:37.090 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:48:37.090 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.7:0 scrub status 2024-03-24T00:48:37.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.196+0000 7f3d0407e640 1 Processor -- start 2024-03-24T00:48:37.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.196+0000 7f3d0407e640 1 -- start start 2024-03-24T00:48:37.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.196+0000 7f3d0407e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3cfc15dea0 0x7f3cfc0d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:37.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.197+0000 7f3d0407e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3cfc15e280 0x7f3cfc0d3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:37.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.197+0000 7f3d0407e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cfc0d2250 con 0x7f3cfc0b5a60 2024-03-24T00:48:37.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.197+0000 7f3d0407e640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cfc0d2530 con 0x7f3cfc15e280 2024-03-24T00:48:37.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.197+0000 7f3d0407e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cfc0d2810 con 0x7f3cfc15dea0 2024-03-24T00:48:37.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.197+0000 7f3d01619640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3cfc15e280 0x7f3cfc0d3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:37.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.197+0000 7f3d01619640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3cfc15e280 0x7f3cfc0d3910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36024/0 (socket says 172.21.15.67:36024) 2024-03-24T00:48:37.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.197+0000 7f3d01619640 1 -- 172.21.15.67:0/900812753 learned_addr learned my addr 172.21.15.67:0/900812753 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:37.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.197+0000 7f3d01e1a640 1 --2- 172.21.15.67:0/900812753 >> v2:172.21.15.72:3300/0 conn(0x7f3cfc15dea0 0x7f3cfc0d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:37.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.197+0000 7f3d01619640 1 -- 172.21.15.67:0/900812753 >> v2:172.21.15.72:3300/0 conn(0x7f3cfc15dea0 msgr2=0x7f3cfc0d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:37.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.197+0000 7f3d01619640 1 --2- 172.21.15.67:0/900812753 >> v2:172.21.15.72:3300/0 conn(0x7f3cfc15dea0 0x7f3cfc0d33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.197+0000 7f3d01619640 1 -- 172.21.15.67:0/900812753 >> v1:172.21.15.72:6789/0 conn(0x7f3cfc0b5a60 legacy=0x7f3cfc06c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:37.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.197+0000 7f3d01619640 1 -- 172.21.15.67:0/900812753 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cfc0d2af0 con 0x7f3cfc15e280 2024-03-24T00:48:37.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.198+0000 7f3d01619640 1 --2- 172.21.15.67:0/900812753 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cfc15e280 0x7f3cfc0d3910 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc0d4c30 tx=0x7f3cec02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=651dc70b7779227 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:37.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.198+0000 7f3d00e18640 1 -- 172.21.15.67:0/900812753 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cec035020 con 0x7f3cfc15e280 2024-03-24T00:48:37.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.198+0000 7f3d00e18640 1 -- 172.21.15.67:0/900812753 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cec031870 con 0x7f3cfc15e280 2024-03-24T00:48:37.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.198+0000 7f3d00e18640 1 -- 172.21.15.67:0/900812753 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cec031c40 con 0x7f3cfc15e280 2024-03-24T00:48:37.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.199+0000 7f3d0407e640 1 -- 172.21.15.67:0/900812753 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cfc15e280 msgr2=0x7f3cfc0d3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:37.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.199+0000 7f3d0407e640 1 --2- 172.21.15.67:0/900812753 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cfc15e280 0x7f3cfc0d3910 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc0d4c30 tx=0x7f3cec02b910 comp rx=0 tx=0).stop 2024-03-24T00:48:37.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.199+0000 7f3d0407e640 1 -- 172.21.15.67:0/900812753 shutdown_connections 2024-03-24T00:48:37.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.199+0000 7f3d0407e640 1 --2- 172.21.15.67:0/900812753 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cfc15e280 0x7f3cfc0d3910 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.199+0000 7f3d0407e640 1 --2- 172.21.15.67:0/900812753 >> v2:172.21.15.72:3300/0 conn(0x7f3cfc15dea0 0x7f3cfc0d33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.199+0000 7f3d0407e640 1 -- 172.21.15.67:0/900812753 >> 172.21.15.67:0/900812753 conn(0x7f3cfc072da0 msgr2=0x7f3cfc0d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:37.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.199+0000 7f3d0407e640 1 -- 172.21.15.67:0/900812753 shutdown_connections 2024-03-24T00:48:37.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.199+0000 7f3d0407e640 1 -- 172.21.15.67:0/900812753 wait complete. 2024-03-24T00:48:37.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.200+0000 7f3d0407e640 1 Processor -- start 2024-03-24T00:48:37.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.200+0000 7f3d0407e640 1 -- start start 2024-03-24T00:48:37.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.201+0000 7f3d0407e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cfc0b5a60 0x7f3cfc153340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:37.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.201+0000 7f3d0407e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3cfc15dea0 0x7f3cfc1610c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:37.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.201+0000 7f3d0407e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3cfc15e280 0x7f3cfc0b4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:37.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.201+0000 7f3d0407e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc0b5eb0 con 0x7f3cfc0b5a60 2024-03-24T00:48:37.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.201+0000 7f3d0407e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc0b8310 con 0x7f3cfc15dea0 2024-03-24T00:48:37.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.201+0000 7f3d0407e640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc0b6030 con 0x7f3cfc15e280 2024-03-24T00:48:37.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.201+0000 7f3d01e1a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cfc0b5a60 0x7f3cfc153340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:37.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.201+0000 7f3d01e1a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cfc0b5a60 0x7f3cfc153340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36026/0 (socket says 172.21.15.67:36026) 2024-03-24T00:48:37.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.201+0000 7f3d01e1a640 1 -- 172.21.15.67:0/1433471073 learned_addr learned my addr 172.21.15.67:0/1433471073 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:37.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.201+0000 7f3d01619640 1 --2- 172.21.15.67:0/1433471073 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3cfc15dea0 0x7f3cfc1610c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:37.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.201+0000 7f3d0261b640 1 --2- 172.21.15.67:0/1433471073 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3cfc15e280 0x7f3cfc0b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:37.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.201+0000 7f3d01e1a640 1 -- 172.21.15.67:0/1433471073 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3cfc15e280 msgr2=0x7f3cfc0b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:37.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.201+0000 7f3d01e1a640 1 --2- 172.21.15.67:0/1433471073 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3cfc15e280 0x7f3cfc0b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.201+0000 7f3d01e1a640 1 -- 172.21.15.67:0/1433471073 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3cfc15dea0 msgr2=0x7f3cfc1610c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:37.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.201+0000 7f3d01e1a640 1 --2- 172.21.15.67:0/1433471073 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3cfc15dea0 0x7f3cfc1610c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.201+0000 7f3d01e1a640 1 -- 172.21.15.67:0/1433471073 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cfc15f8c0 con 0x7f3cfc0b5a60 2024-03-24T00:48:37.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.201+0000 7f3d0261b640 1 --2- 172.21.15.67:0/1433471073 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3cfc15e280 0x7f3cfc0b4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:48:37.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.202+0000 7f3d01e1a640 1 --2- 172.21.15.67:0/1433471073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cfc0b5a60 0x7f3cfc153340 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f3cf800e9b0 tx=0x7f3cf800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:37.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.202+0000 7f3cf27fc640 1 -- 172.21.15.67:0/1433471073 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf8017070 con 0x7f3cfc0b5a60 2024-03-24T00:48:37.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.202+0000 7f3cf27fc640 1 -- 172.21.15.67:0/1433471073 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cf800c9a0 con 0x7f3cfc0b5a60 2024-03-24T00:48:37.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.202+0000 7f3d0407e640 1 -- 172.21.15.67:0/1433471073 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cfc153880 con 0x7f3cfc0b5a60 2024-03-24T00:48:37.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.202+0000 7f3cf27fc640 1 -- 172.21.15.67:0/1433471073 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf800cd90 con 0x7f3cfc0b5a60 2024-03-24T00:48:37.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.202+0000 7f3d0407e640 1 -- 172.21.15.67:0/1433471073 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cfc15fe40 con 0x7f3cfc0b5a60 2024-03-24T00:48:37.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.204+0000 7f3cf27fc640 1 -- 172.21.15.67:0/1433471073 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3cf801a070 con 0x7f3cfc0b5a60 2024-03-24T00:48:37.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.204+0000 7f3cf27fc640 1 --2- 172.21.15.67:0/1433471073 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3ce006bbd0 0x7f3ce006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:37.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.204+0000 7f3cf27fc640 1 -- 172.21.15.67:0/1433471073 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f3cf808ac50 con 0x7f3cfc0b5a60 2024-03-24T00:48:37.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.205+0000 7f3d06c62740 1 Processor -- start 2024-03-24T00:48:37.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.205+0000 7f3d01619640 1 --2- 172.21.15.67:0/1433471073 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3ce006bbd0 0x7f3ce006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:37.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.205+0000 7f3d06c62740 1 -- start start 2024-03-24T00:48:37.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.205+0000 7f3d01619640 1 --2- 172.21.15.67:0/1433471073 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3ce006bbd0 0x7f3ce006e090 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f3cec036000 tx=0x7f3cec03c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:37.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.205+0000 7f3d06c62740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563b47d16e50 0x563b47ce9df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:37.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.205+0000 7f3d06c62740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563b47d17230 0x563b47d19be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:37.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.205+0000 7f3d06c62740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563b47cea330 0x563b47d1a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:37.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.205+0000 7f3d06c62740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b47c543d0 con 0x563b47cea330 2024-03-24T00:48:37.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.205+0000 7f3d06c62740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b47bfcad0 con 0x563b47d17230 2024-03-24T00:48:37.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.205+0000 7f3d06c62740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x563b47c50470 con 0x563b47d16e50 2024-03-24T00:48:37.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.205+0000 7f3d0261b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563b47cea330 0x563b47d1a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:37.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.205+0000 7f3d0261b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563b47cea330 0x563b47d1a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36042/0 (socket says 172.21.15.67:36042) 2024-03-24T00:48:37.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.205+0000 7f3d0261b640 1 -- 172.21.15.67:0/3506063561 learned_addr learned my addr 172.21.15.67:0/3506063561 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:37.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.206+0000 7f3d01e1a640 1 --2- 172.21.15.67:0/3506063561 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563b47d16e50 0x563b47ce9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:37.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.206+0000 7f3d01619640 1 --2- 172.21.15.67:0/3506063561 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563b47d17230 0x563b47d19be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:37.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.206+0000 7f3d0261b640 1 -- 172.21.15.67:0/3506063561 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563b47d16e50 msgr2=0x563b47ce9df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:37.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.206+0000 7f3d0261b640 1 --2- 172.21.15.67:0/3506063561 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563b47d16e50 0x563b47ce9df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.206+0000 7f3d0261b640 1 -- 172.21.15.67:0/3506063561 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563b47d17230 msgr2=0x563b47d19be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:37.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.206+0000 7f3d0261b640 1 --2- 172.21.15.67:0/3506063561 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563b47d17230 0x563b47d19be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.206+0000 7f3d0261b640 1 -- 172.21.15.67:0/3506063561 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563b47a327d0 con 0x563b47cea330 2024-03-24T00:48:37.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.206+0000 7f3d0261b640 1 --2- 172.21.15.67:0/3506063561 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563b47cea330 0x563b47d1a120 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f3cf4001d30 tx=0x7f3cf40021f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9504661a5387778f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:37.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.206+0000 7f3cdf7fe640 1 -- 172.21.15.67:0/3506063561 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf4019070 con 0x563b47cea330 2024-03-24T00:48:37.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.206+0000 7f3cdf7fe640 1 -- 172.21.15.67:0/3506063561 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cf4003b80 con 0x563b47cea330 2024-03-24T00:48:37.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.206+0000 7f3cdf7fe640 1 -- 172.21.15.67:0/3506063561 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf400e860 con 0x563b47cea330 2024-03-24T00:48:37.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.207+0000 7f3d06c62740 1 -- 172.21.15.67:0/3506063561 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563b47cea330 msgr2=0x563b47d1a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:37.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.207+0000 7f3d06c62740 1 --2- 172.21.15.67:0/3506063561 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563b47cea330 0x563b47d1a120 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f3cf4001d30 tx=0x7f3cf40021f0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.207+0000 7f3d06c62740 1 -- 172.21.15.67:0/3506063561 shutdown_connections 2024-03-24T00:48:37.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.207+0000 7f3d06c62740 1 --2- 172.21.15.67:0/3506063561 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563b47cea330 0x563b47d1a120 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.207+0000 7f3d06c62740 1 --2- 172.21.15.67:0/3506063561 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563b47d17230 0x563b47d19be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.207+0000 7f3d06c62740 1 --2- 172.21.15.67:0/3506063561 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563b47d16e50 0x563b47ce9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.207+0000 7f3d06c62740 1 -- 172.21.15.67:0/3506063561 >> 172.21.15.67:0/3506063561 conn(0x563b47cec060 msgr2=0x563b47d34be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:37.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.207+0000 7f3d06c62740 1 -- 172.21.15.67:0/3506063561 shutdown_connections 2024-03-24T00:48:37.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.207+0000 7f3d06c62740 1 -- 172.21.15.67:0/3506063561 wait complete. 2024-03-24T00:48:37.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.207+0000 7f3d06c62740 1 -- start start 2024-03-24T00:48:37.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.207+0000 7f3d06c62740 1 Processor -- start 2024-03-24T00:48:37.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.208+0000 7f3d06c62740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563b47d16e50 0x563b47d209d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:37.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.208+0000 7f3d06c62740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563b47d17230 0x563b47d09c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:37.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.208+0000 7f3d06c62740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563b47d1d900 0x563b47d0a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:37.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.208+0000 7f3d06c62740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b47cee700 con 0x563b47d16e50 2024-03-24T00:48:37.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.208+0000 7f3d06c62740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b47c10060 con 0x563b47d1d900 2024-03-24T00:48:37.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.208+0000 7f3d06c62740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x563b47cf0f40 con 0x563b47d17230 2024-03-24T00:48:37.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.208+0000 7f3d06c62740 10 client.0 _ll_register_callbacks cb 0x563b47bf26a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:48:37.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.208+0000 7f3d01e1a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563b47d16e50 0x563b47d209d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:37.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.208+0000 7f3d01e1a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563b47d16e50 0x563b47d209d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36044/0 (socket says 172.21.15.67:36044) 2024-03-24T00:48:37.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.208+0000 7f3d01e1a640 1 -- 172.21.15.67:0/2436234070 learned_addr learned my addr 172.21.15.67:0/2436234070 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:37.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.209+0000 7f3d0261b640 1 --2- 172.21.15.67:0/2436234070 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563b47d1d900 0x563b47d0a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:37.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.209+0000 7f3d01619640 1 --2- 172.21.15.67:0/2436234070 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563b47d17230 0x563b47d09c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:37.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.209+0000 7f3d01e1a640 1 -- 172.21.15.67:0/2436234070 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563b47d17230 msgr2=0x563b47d09c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:37.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.209+0000 7f3d01e1a640 1 --2- 172.21.15.67:0/2436234070 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563b47d17230 0x563b47d09c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.209+0000 7f3d01e1a640 1 -- 172.21.15.67:0/2436234070 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563b47d1d900 msgr2=0x563b47d0a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:37.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.209+0000 7f3d01e1a640 1 --2- 172.21.15.67:0/2436234070 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563b47d1d900 0x563b47d0a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.209+0000 7f3d01e1a640 1 -- 172.21.15.67:0/2436234070 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563b479683f0 con 0x563b47d16e50 2024-03-24T00:48:37.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.209+0000 7f3d01e1a640 1 --2- 172.21.15.67:0/2436234070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563b47d16e50 0x563b47d209d0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f3cf8058c00 tx=0x7f3cf805d060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5788172ff135a442 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:37.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.209+0000 7f3cdcff9640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:48:37.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.209+0000 7f3cdcff9640 1 -- 172.21.15.67:0/2436234070 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf8075060 con 0x563b47d16e50 2024-03-24T00:48:37.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.209+0000 7f3cdcff9640 1 -- 172.21.15.67:0/2436234070 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f3cf80612d0 con 0x563b47d16e50 2024-03-24T00:48:37.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.209+0000 7f3d06c62740 1 -- 172.21.15.67:0/2436234070 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3cf4003b80 con 0x563b47d16e50 2024-03-24T00:48:37.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.209+0000 7f3cdcff9640 1 -- 172.21.15.67:0/2436234070 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3cf8021040 con 0x563b47d16e50 2024-03-24T00:48:37.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.209+0000 7f3d06c62740 10 client.9033 fetch_fsmap learned FSMap version 78 2024-03-24T00:48:37.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.209+0000 7f3d06c62740 1 -- 172.21.15.67:0/2436234070 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=2+}) v3 -- 0x563b479280f0 con 0x563b47d16e50 2024-03-24T00:48:37.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.209+0000 7f3cdcff9640 1 -- 172.21.15.67:0/2436234070 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cf4003b80 con 0x563b47d16e50 2024-03-24T00:48:37.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.210+0000 7f3cdcff9640 1 -- 172.21.15.67:0/2436234070 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 78) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f3cf8024060 con 0x563b47d16e50 2024-03-24T00:48:37.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.210+0000 7f3d06c62740 10 client.9033 fetch_fsmap finished waiting for FSMap version 78 2024-03-24T00:48:37.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.210+0000 7f3d06c62740 10 client.9033 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.c 2024-03-24T00:48:37.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.210+0000 7f3d06c62740 20 client.9033 populate_metadata read hostname 'smithi067' 2024-03-24T00:48:37.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.210+0000 7f3d06c62740 1 --2- 172.21.15.67:0/2436234070 >> [v2:172.21.15.67:6832/3197261495,v1:172.21.15.67:6833/3197261495] conn(0x563b47d487d0 0x563b47d48be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:37.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.210+0000 7f3d06c62740 4 client.9033 mds_command: new command op to 8895 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:48:37.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.210+0000 7f3d06c62740 1 -- 172.21.15.67:0/2436234070 --> [v2:172.21.15.67:6832/3197261495,v1:172.21.15.67:6833/3197261495] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563b47c61910 con 0x563b47d487d0 2024-03-24T00:48:37.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.210+0000 7f3d01619640 1 --2- 172.21.15.67:0/2436234070 >> [v2:172.21.15.67:6832/3197261495,v1:172.21.15.67:6833/3197261495] conn(0x563b47d487d0 0x563b47d48be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:37.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.210+0000 7f3cdcff9640 10 client.9033 ms_handle_connect on v2:172.21.15.67:6832/3197261495 2024-03-24T00:48:37.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.210+0000 7f3d01619640 1 --2- 172.21.15.67:0/2436234070 >> [v2:172.21.15.67:6832/3197261495,v1:172.21.15.67:6833/3197261495] conn(0x563b47d487d0 0x563b47d48be0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4dd056fe1afafae server_cookie=a072ed3930a0f93 in_seq=0 out_seq=0 2024-03-24T00:48:37.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.211+0000 7f3cdcff9640 1 -- 172.21.15.67:0/2436234070 <== mds.0 v2:172.21.15.67:6832/3197261495 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563b47c61910 con 0x563b47d487d0 2024-03-24T00:48:37.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.211+0000 7f3cdcff9640 10 client.9033 handle_command_reply: tid=0 2024-03-24T00:48:37.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.211+0000 7f3d06c62740 1 client.9033 shutdown 2024-03-24T00:48:37.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.211+0000 7f3d06c62740 1 -- 172.21.15.67:0/2436234070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563b47d16e50 msgr2=0x563b47d209d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:37.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.211+0000 7f3d06c62740 1 --2- 172.21.15.67:0/2436234070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563b47d16e50 0x563b47d209d0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f3cf8058c00 tx=0x7f3cf805d060 comp rx=0 tx=0).stop 2024-03-24T00:48:37.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.211+0000 7f3d06c62740 1 -- 172.21.15.67:0/2436234070 shutdown_connections 2024-03-24T00:48:37.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.211+0000 7f3d06c62740 1 --2- 172.21.15.67:0/2436234070 >> [v2:172.21.15.67:6832/3197261495,v1:172.21.15.67:6833/3197261495] conn(0x563b47d487d0 0x563b47d48be0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.211+0000 7f3d06c62740 1 --2- 172.21.15.67:0/2436234070 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563b47d1d900 0x563b47d0a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.211+0000 7f3d06c62740 1 --2- 172.21.15.67:0/2436234070 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563b47d17230 0x563b47d09c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.211+0000 7f3cdcff9640 0 client.9033 ms_handle_reset on v2:172.21.15.67:6832/3197261495 2024-03-24T00:48:37.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.211+0000 7f3d06c62740 1 --2- 172.21.15.67:0/2436234070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563b47d16e50 0x563b47d209d0 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.211+0000 7f3d06c62740 1 -- 172.21.15.67:0/2436234070 >> 172.21.15.67:0/2436234070 conn(0x563b47cea330 msgr2=0x563b47cedd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:37.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.211+0000 7f3d06c62740 1 -- 172.21.15.67:0/2436234070 shutdown_connections 2024-03-24T00:48:37.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.211+0000 7f3d06c62740 1 -- 172.21.15.67:0/2436234070 wait complete. 2024-03-24T00:48:37.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.211+0000 7f3d06c62740 20 client.9033 trim_cache size 0 max 16384 2024-03-24T00:48:37.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.228+0000 7f3d06c62740 1 Processor -- start 2024-03-24T00:48:37.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.228+0000 7f3d06c62740 1 -- start start 2024-03-24T00:48:37.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.228+0000 7f3d06c62740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563b47d16e50 0x563b47d37db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:37.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.228+0000 7f3d06c62740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563b47d17230 0x563b47cedb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:37.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.228+0000 7f3d06c62740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563b47d1d900 0x563b47ce8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:37.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.228+0000 7f3d06c62740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc200900 con 0x563b47d17230 2024-03-24T00:48:37.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.228+0000 7f3d06c62740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc1ffab0 con 0x563b47d1d900 2024-03-24T00:48:37.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.228+0000 7f3d06c62740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc0abce0 con 0x563b47d16e50 2024-03-24T00:48:37.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.228+0000 7f3d01619640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563b47d17230 0x563b47cedb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:37.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.228+0000 7f3d01619640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563b47d17230 0x563b47cedb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36046/0 (socket says 172.21.15.67:36046) 2024-03-24T00:48:37.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.228+0000 7f3d01619640 1 -- 172.21.15.67:0/2347647575 learned_addr learned my addr 172.21.15.67:0/2347647575 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:37.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.229+0000 7f3d0261b640 1 --2- 172.21.15.67:0/2347647575 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563b47d1d900 0x563b47ce8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:37.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.229+0000 7f3d01e1a640 1 --2- 172.21.15.67:0/2347647575 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563b47d16e50 0x563b47d37db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:37.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.229+0000 7f3d01619640 1 -- 172.21.15.67:0/2347647575 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563b47d16e50 msgr2=0x563b47d37db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:37.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.229+0000 7f3d01619640 1 --2- 172.21.15.67:0/2347647575 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563b47d16e50 0x563b47d37db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.229+0000 7f3d01619640 1 -- 172.21.15.67:0/2347647575 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563b47d1d900 msgr2=0x563b47ce8ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:37.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.229+0000 7f3d01619640 1 --2- 172.21.15.67:0/2347647575 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563b47d1d900 0x563b47ce8ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.229+0000 7f3d01619640 1 -- 172.21.15.67:0/2347647575 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563b47d200e0 con 0x563b47d17230 2024-03-24T00:48:37.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.229+0000 7f3d01619640 1 --2- 172.21.15.67:0/2347647575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563b47d17230 0x563b47cedb40 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f3cec0090b0 tx=0x7f3cec030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3b0c8dbe7e5691a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:37.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.229+0000 7f3cdeffd640 1 -- 172.21.15.67:0/2347647575 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cec035040 con 0x563b47d17230 2024-03-24T00:48:37.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.229+0000 7f3cdeffd640 1 -- 172.21.15.67:0/2347647575 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cec03a9b0 con 0x563b47d17230 2024-03-24T00:48:37.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.229+0000 7f3cdeffd640 1 -- 172.21.15.67:0/2347647575 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cec053310 con 0x563b47d17230 2024-03-24T00:48:37.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.229+0000 7f3d06c62740 1 -- 172.21.15.67:0/2347647575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563b47d17230 msgr2=0x563b47cedb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:37.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.229+0000 7f3d06c62740 1 --2- 172.21.15.67:0/2347647575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563b47d17230 0x563b47cedb40 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f3cec0090b0 tx=0x7f3cec030000 comp rx=0 tx=0).stop 2024-03-24T00:48:37.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.230+0000 7f3d06c62740 1 -- 172.21.15.67:0/2347647575 shutdown_connections 2024-03-24T00:48:37.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.230+0000 7f3d06c62740 1 --2- 172.21.15.67:0/2347647575 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563b47d1d900 0x563b47ce8ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.230+0000 7f3d06c62740 1 --2- 172.21.15.67:0/2347647575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563b47d17230 0x563b47cedb40 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.230+0000 7f3d06c62740 1 --2- 172.21.15.67:0/2347647575 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563b47d16e50 0x563b47d37db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.230+0000 7f3d06c62740 1 -- 172.21.15.67:0/2347647575 >> 172.21.15.67:0/2347647575 conn(0x563b47cea330 msgr2=0x563b47d1e510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:37.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.230+0000 7f3d06c62740 1 -- 172.21.15.67:0/2347647575 shutdown_connections 2024-03-24T00:48:37.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.230+0000 7f3d06c62740 1 -- 172.21.15.67:0/2347647575 wait complete. 2024-03-24T00:48:37.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.230+0000 7f3d06c62740 1 -- start start 2024-03-24T00:48:37.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.230+0000 7f3d06c62740 1 Processor -- start 2024-03-24T00:48:37.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.231+0000 7f3d06c62740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563b47d16e50 0x563b47da3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:37.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.231+0000 7f3d06c62740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563b47d17230 0x563b47d47ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:37.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.231+0000 7f3d06c62740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563b47d1d900 0x563b47d37fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:37.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.231+0000 7f3d06c62740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b47d14e10 con 0x563b47d16e50 2024-03-24T00:48:37.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.231+0000 7f3d06c62740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc1ff4b0 con 0x563b47d1d900 2024-03-24T00:48:37.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.231+0000 7f3d06c62740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x563b47c59650 con 0x563b47d17230 2024-03-24T00:48:37.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.232+0000 7f3d06c62740 10 client.0 _ll_register_callbacks cb 0x7f3cf8058a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:48:37.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.232+0000 7f3d01e1a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563b47d16e50 0x563b47da3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:37.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.232+0000 7f3d01e1a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563b47d16e50 0x563b47da3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36060/0 (socket says 172.21.15.67:36060) 2024-03-24T00:48:37.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.232+0000 7f3d01e1a640 1 -- 172.21.15.67:0/565691993 learned_addr learned my addr 172.21.15.67:0/565691993 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:37.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.232+0000 7f3d01619640 1 --2- 172.21.15.67:0/565691993 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563b47d17230 0x563b47d47ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:37.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.232+0000 7f3d0261b640 1 --2- 172.21.15.67:0/565691993 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563b47d1d900 0x563b47d37fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:37.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.232+0000 7f3d01e1a640 1 -- 172.21.15.67:0/565691993 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563b47d17230 msgr2=0x563b47d47ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:37.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.232+0000 7f3d01e1a640 1 --2- 172.21.15.67:0/565691993 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563b47d17230 0x563b47d47ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.232+0000 7f3d01e1a640 1 -- 172.21.15.67:0/565691993 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563b47d1d900 msgr2=0x563b47d37fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:37.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.232+0000 7f3d01e1a640 1 --2- 172.21.15.67:0/565691993 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563b47d1d900 0x563b47d37fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.232+0000 7f3d01e1a640 1 -- 172.21.15.67:0/565691993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563b47cf0440 con 0x563b47d16e50 2024-03-24T00:48:37.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.232+0000 7f3d01619640 1 --2- 172.21.15.67:0/565691993 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563b47d17230 0x563b47d47ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:48:37.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.232+0000 7f3d01e1a640 1 --2- 172.21.15.67:0/565691993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563b47d16e50 0x563b47da3e20 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f3cf8089060 tx=0x7f3cf8088ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c69281c05ee5813 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:37.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.232+0000 7f3cdcff9640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:48:37.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.232+0000 7f3cdcff9640 1 -- 172.21.15.67:0/565691993 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf80551a0 con 0x563b47d16e50 2024-03-24T00:48:37.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.232+0000 7f3cdcff9640 1 -- 172.21.15.67:0/565691993 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f3cf807ecc0 con 0x563b47d16e50 2024-03-24T00:48:37.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.232+0000 7f3d06c62740 1 -- 172.21.15.67:0/565691993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3cec03a9b0 con 0x563b47d16e50 2024-03-24T00:48:37.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.232+0000 7f3cdcff9640 1 -- 172.21.15.67:0/565691993 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3cf8058350 con 0x563b47d16e50 2024-03-24T00:48:37.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.232+0000 7f3d06c62740 10 client.9039 fetch_fsmap learned FSMap version 78 2024-03-24T00:48:37.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.232+0000 7f3d06c62740 1 -- 172.21.15.67:0/565691993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=2+}) v3 -- 0x563b47d16850 con 0x563b47d16e50 2024-03-24T00:48:37.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.233+0000 7f3cdcff9640 1 -- 172.21.15.67:0/565691993 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cec03a9b0 con 0x563b47d16e50 2024-03-24T00:48:37.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.233+0000 7f3cdcff9640 1 -- 172.21.15.67:0/565691993 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 78) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f3cf8054c50 con 0x563b47d16e50 2024-03-24T00:48:37.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.233+0000 7f3d06c62740 10 client.9039 fetch_fsmap finished waiting for FSMap version 78 2024-03-24T00:48:37.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.233+0000 7f3d06c62740 10 client.9039 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.c 2024-03-24T00:48:37.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.233+0000 7f3d06c62740 20 client.9039 populate_metadata read hostname 'smithi067' 2024-03-24T00:48:37.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.233+0000 7f3d06c62740 1 --2- 172.21.15.67:0/565691993 >> [v2:172.21.15.67:6832/3197261495,v1:172.21.15.67:6833/3197261495] conn(0x563b47d38510 0x563b47da44e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:37.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.233+0000 7f3d06c62740 4 client.9039 mds_command: new command op to 8895 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:48:37.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.233+0000 7f3d06c62740 1 -- 172.21.15.67:0/565691993 --> [v2:172.21.15.67:6832/3197261495,v1:172.21.15.67:6833/3197261495] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x563b47c5bc60 con 0x563b47d38510 2024-03-24T00:48:37.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.233+0000 7f3d01619640 1 --2- 172.21.15.67:0/565691993 >> [v2:172.21.15.67:6832/3197261495,v1:172.21.15.67:6833/3197261495] conn(0x563b47d38510 0x563b47da44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:37.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.233+0000 7f3cdcff9640 10 client.9039 ms_handle_connect on v2:172.21.15.67:6832/3197261495 2024-03-24T00:48:37.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.233+0000 7f3d01619640 1 --2- 172.21.15.67:0/565691993 >> [v2:172.21.15.67:6832/3197261495,v1:172.21.15.67:6833/3197261495] conn(0x563b47d38510 0x563b47da44e0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1f9d7bc4ba095bd server_cookie=860b09a4e5224c16 in_seq=0 out_seq=0 2024-03-24T00:48:37.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.233+0000 7f3cdcff9640 1 -- 172.21.15.67:0/565691993 <== mds.0 v2:172.21.15.67:6832/3197261495 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x563b47c5bc60 con 0x563b47d38510 2024-03-24T00:48:37.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.233+0000 7f3cdcff9640 10 client.9039 handle_command_reply: tid=0 2024-03-24T00:48:37.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.233+0000 7f3d06c62740 1 client.9039 shutdown 2024-03-24T00:48:37.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.233+0000 7f3d06c62740 1 -- 172.21.15.67:0/565691993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563b47d16e50 msgr2=0x563b47da3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:37.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.233+0000 7f3d06c62740 1 --2- 172.21.15.67:0/565691993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563b47d16e50 0x563b47da3e20 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f3cf8089060 tx=0x7f3cf8088ee0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.233+0000 7f3d06c62740 1 -- 172.21.15.67:0/565691993 shutdown_connections 2024-03-24T00:48:37.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.233+0000 7f3d06c62740 1 --2- 172.21.15.67:0/565691993 >> [v2:172.21.15.67:6832/3197261495,v1:172.21.15.67:6833/3197261495] conn(0x563b47d38510 0x563b47da44e0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.233+0000 7f3d06c62740 1 --2- 172.21.15.67:0/565691993 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563b47d1d900 0x563b47d37fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.233+0000 7f3d06c62740 1 --2- 172.21.15.67:0/565691993 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563b47d17230 0x563b47d47ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.233+0000 7f3cdcff9640 0 client.9039 ms_handle_reset on v2:172.21.15.67:6832/3197261495 2024-03-24T00:48:37.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.233+0000 7f3d06c62740 1 --2- 172.21.15.67:0/565691993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563b47d16e50 0x563b47da3e20 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.233+0000 7f3d06c62740 1 -- 172.21.15.67:0/565691993 >> 172.21.15.67:0/565691993 conn(0x563b47cea330 msgr2=0x563b47ce5390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:37.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.234+0000 7f3d06c62740 1 -- 172.21.15.67:0/565691993 shutdown_connections 2024-03-24T00:48:37.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.234+0000 7f3d06c62740 1 -- 172.21.15.67:0/565691993 wait complete. 2024-03-24T00:48:37.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.234+0000 7f3d06c62740 20 client.9039 trim_cache size 0 max 16384 2024-03-24T00:48:37.242 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:48:37.242 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "no active scrubs running", 2024-03-24T00:48:37.243 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": {} 2024-03-24T00:48:37.243 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:48:37.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.234+0000 7f3d0407e640 1 -- 172.21.15.67:0/1433471073 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3ce006bbd0 msgr2=0x7f3ce006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:37.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.234+0000 7f3d0407e640 1 --2- 172.21.15.67:0/1433471073 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3ce006bbd0 0x7f3ce006e090 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f3cec036000 tx=0x7f3cec03c040 comp rx=0 tx=0).stop 2024-03-24T00:48:37.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.234+0000 7f3d0407e640 1 -- 172.21.15.67:0/1433471073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cfc0b5a60 msgr2=0x7f3cfc153340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:37.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.234+0000 7f3d0407e640 1 --2- 172.21.15.67:0/1433471073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cfc0b5a60 0x7f3cfc153340 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f3cf800e9b0 tx=0x7f3cf800ee70 comp rx=0 tx=0).stop 2024-03-24T00:48:37.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.234+0000 7f3d0407e640 1 -- 172.21.15.67:0/1433471073 shutdown_connections 2024-03-24T00:48:37.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.234+0000 7f3d0407e640 1 --2- 172.21.15.67:0/1433471073 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3ce006bbd0 0x7f3ce006e090 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.234+0000 7f3d0407e640 1 --2- 172.21.15.67:0/1433471073 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3cfc15e280 0x7f3cfc0b4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.234+0000 7f3d0407e640 1 --2- 172.21.15.67:0/1433471073 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3cfc15dea0 0x7f3cfc1610c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.234+0000 7f3d0407e640 1 --2- 172.21.15.67:0/1433471073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cfc0b5a60 0x7f3cfc153340 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.234+0000 7f3d0407e640 1 -- 172.21.15.67:0/1433471073 >> 172.21.15.67:0/1433471073 conn(0x7f3cfc072da0 msgr2=0x7f3cfc0b7120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:37.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.234+0000 7f3d0407e640 1 -- 172.21.15.67:0/1433471073 shutdown_connections 2024-03-24T00:48:37.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.234+0000 7f3d0407e640 1 -- 172.21.15.67:0/1433471073 wait complete. 2024-03-24T00:48:37.246 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T00:48:37.347 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-24T00:48:37.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.450+0000 7fe7b240b640 1 Processor -- start 2024-03-24T00:48:37.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.450+0000 7fe7b240b640 1 -- start start 2024-03-24T00:48:37.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.451+0000 7fe7b240b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe7ac0d32d0 0x7fe7ac0d3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:37.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.451+0000 7fe7b240b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe7ac0b6dc0 0x7fe7ac0b7250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:37.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.451+0000 7fe7b240b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7ac0b9cc0 con 0x7fe7ac15e0f0 2024-03-24T00:48:37.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.451+0000 7fe7b240b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7ac0b7790 con 0x7fe7ac0b6dc0 2024-03-24T00:48:37.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.451+0000 7fe7b240b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7ac0d2230 con 0x7fe7ac0d32d0 2024-03-24T00:48:37.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.451+0000 7fe7abfff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe7ac15e0f0 0x7fe7ac0b95b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45576/0 (socket says 172.21.15.67:45576) 2024-03-24T00:48:37.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.451+0000 7fe7b09a8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe7ac0b6dc0 0x7fe7ac0b7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:37.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.451+0000 7fe7abfff640 1 -- 172.21.15.67:0/4169117020 learned_addr learned my addr 172.21.15.67:0/4169117020 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:48:37.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.451+0000 7fe7ab7fe640 1 --2- 172.21.15.67:0/4169117020 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe7ac0d32d0 0x7fe7ac0d3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:37.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.451+0000 7fe7aaffd640 1 -- 172.21.15.67:0/4169117020 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 637885532 0 0) 0x7fe7ac0b9cc0 con 0x7fe7ac15e0f0 2024-03-24T00:48:37.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.451+0000 7fe7aaffd640 1 -- 172.21.15.67:0/4169117020 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe794003520 con 0x7fe7ac15e0f0 2024-03-24T00:48:37.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.451+0000 7fe7b09a8640 1 -- 172.21.15.67:0/4169117020 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe7ac0d32d0 msgr2=0x7fe7ac0d3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:37.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.451+0000 7fe7b09a8640 1 --2- 172.21.15.67:0/4169117020 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe7ac0d32d0 0x7fe7ac0d3710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.451+0000 7fe7b09a8640 1 -- 172.21.15.67:0/4169117020 >> v1:172.21.15.67:6789/0 conn(0x7fe7ac15e0f0 legacy=0x7fe7ac0b95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:37.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.451+0000 7fe7b09a8640 1 -- 172.21.15.67:0/4169117020 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7ac0d2510 con 0x7fe7ac0b6dc0 2024-03-24T00:48:37.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.452+0000 7fe7b09a8640 1 --2- 172.21.15.67:0/4169117020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7ac0b6dc0 0x7fe7ac0b7250 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4009d90 tx=0x7fe7a402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a97e5e4828c8d3dc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:37.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.452+0000 7fe7aaffd640 1 -- 172.21.15.67:0/4169117020 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a4039070 con 0x7fe7ac0b6dc0 2024-03-24T00:48:37.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.452+0000 7fe7aaffd640 1 -- 172.21.15.67:0/4169117020 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7a402cc50 con 0x7fe7ac0b6dc0 2024-03-24T00:48:37.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.452+0000 7fe7aaffd640 1 -- 172.21.15.67:0/4169117020 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a4034440 con 0x7fe7ac0b6dc0 2024-03-24T00:48:37.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.453+0000 7fe7b240b640 1 -- 172.21.15.67:0/4169117020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7ac0b6dc0 msgr2=0x7fe7ac0b7250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:37.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.453+0000 7fe7b240b640 1 --2- 172.21.15.67:0/4169117020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7ac0b6dc0 0x7fe7ac0b7250 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4009d90 tx=0x7fe7a402bea0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.453+0000 7fe7b240b640 1 -- 172.21.15.67:0/4169117020 shutdown_connections 2024-03-24T00:48:37.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.453+0000 7fe7b240b640 1 --2- 172.21.15.67:0/4169117020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7ac0b6dc0 0x7fe7ac0b7250 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.453+0000 7fe7b240b640 1 --2- 172.21.15.67:0/4169117020 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe7ac0d32d0 0x7fe7ac0d3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.453+0000 7fe7b240b640 1 -- 172.21.15.67:0/4169117020 >> 172.21.15.67:0/4169117020 conn(0x7fe7ac072b60 msgr2=0x7fe7ac072f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:37.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.453+0000 7fe7b240b640 1 -- 172.21.15.67:0/4169117020 shutdown_connections 2024-03-24T00:48:37.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.453+0000 7fe7b240b640 1 -- 172.21.15.67:0/4169117020 wait complete. 2024-03-24T00:48:37.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.454+0000 7fe7b240b640 1 Processor -- start 2024-03-24T00:48:37.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.454+0000 7fe7b240b640 1 -- start start 2024-03-24T00:48:37.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.455+0000 7fe7b240b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7ac0b6dc0 0x7fe7ac15b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:37.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.455+0000 7fe7b240b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe7ac0d32d0 0x7fe7ac0cd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:37.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.455+0000 7fe7b240b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe7ac15e0f0 0x7fe7ac0cd810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:37.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.455+0000 7fe7b240b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7ac0d2db0 con 0x7fe7ac0b6dc0 2024-03-24T00:48:37.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.455+0000 7fe7b240b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7ac0d2ad0 con 0x7fe7ac15e0f0 2024-03-24T00:48:37.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.455+0000 7fe7b240b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7ac0b31d0 con 0x7fe7ac0d32d0 2024-03-24T00:48:37.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.455+0000 7fe7abfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7ac0b6dc0 0x7fe7ac15b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:37.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.455+0000 7fe7abfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7ac0b6dc0 0x7fe7ac15b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36078/0 (socket says 172.21.15.67:36078) 2024-03-24T00:48:37.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.455+0000 7fe7abfff640 1 -- 172.21.15.67:0/4078245014 learned_addr learned my addr 172.21.15.67:0/4078245014 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:37.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.455+0000 7fe7ab7fe640 1 --2- 172.21.15.67:0/4078245014 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe7ac0d32d0 0x7fe7ac0cd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:37.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.455+0000 7fe7b09a8640 1 --2- 172.21.15.67:0/4078245014 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe7ac15e0f0 0x7fe7ac0cd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:37.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.455+0000 7fe7abfff640 1 -- 172.21.15.67:0/4078245014 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe7ac0d32d0 msgr2=0x7fe7ac0cd2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:37.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.455+0000 7fe7abfff640 1 --2- 172.21.15.67:0/4078245014 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe7ac0d32d0 0x7fe7ac0cd2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.455+0000 7fe7abfff640 1 -- 172.21.15.67:0/4078245014 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe7ac15e0f0 msgr2=0x7fe7ac0cd810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:37.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.455+0000 7fe7abfff640 1 --2- 172.21.15.67:0/4078245014 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe7ac15e0f0 0x7fe7ac0cd810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.456+0000 7fe7abfff640 1 -- 172.21.15.67:0/4078245014 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7ac0b5cb0 con 0x7fe7ac0b6dc0 2024-03-24T00:48:37.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.456+0000 7fe7ab7fe640 1 --2- 172.21.15.67:0/4078245014 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe7ac0d32d0 0x7fe7ac0cd2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:48:37.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.456+0000 7fe7abfff640 1 --2- 172.21.15.67:0/4078245014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7ac0b6dc0 0x7fe7ac15b1d0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7fe79c00c430 tx=0x7fe79c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:37.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.456+0000 7fe7a8ff9640 1 -- 172.21.15.67:0/4078245014 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe79c017070 con 0x7fe7ac0b6dc0 2024-03-24T00:48:37.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.456+0000 7fe7a8ff9640 1 -- 172.21.15.67:0/4078245014 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe79c005ce0 con 0x7fe7ac0b6dc0 2024-03-24T00:48:37.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.456+0000 7fe7b240b640 1 -- 172.21.15.67:0/4078245014 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7ac0b3630 con 0x7fe7ac0b6dc0 2024-03-24T00:48:37.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.456+0000 7fe7a8ff9640 1 -- 172.21.15.67:0/4078245014 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe79c0072f0 con 0x7fe7ac0b6dc0 2024-03-24T00:48:37.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.456+0000 7fe7b240b640 1 -- 172.21.15.67:0/4078245014 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7ac0b3bf0 con 0x7fe7ac0b6dc0 2024-03-24T00:48:37.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.457+0000 7fe7b240b640 1 -- 172.21.15.67:0/4078245014 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7ac04f400 con 0x7fe7ac0b6dc0 2024-03-24T00:48:37.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.457+0000 7fe7a8ff9640 1 -- 172.21.15.67:0/4078245014 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe79c01a070 con 0x7fe7ac0b6dc0 2024-03-24T00:48:37.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.458+0000 7fe7a8ff9640 1 --2- 172.21.15.67:0/4078245014 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe79806bbd0 0x7fe79806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:37.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.458+0000 7fe7a8ff9640 1 -- 172.21.15.67:0/4078245014 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6216+0+0 (secure 0 0 0) 0x7fe79c08a130 con 0x7fe7ac0b6dc0 2024-03-24T00:48:37.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.458+0000 7fe7ab7fe640 1 --2- 172.21.15.67:0/4078245014 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe79806bbd0 0x7fe79806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:37.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.459+0000 7fe7ab7fe640 1 --2- 172.21.15.67:0/4078245014 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe79806bbd0 0x7fe79806e090 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac0b4d80 tx=0x7fe7a0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:37.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.462+0000 7fe7a8ff9640 1 -- 172.21.15.67:0/4078245014 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe7ac0b5cb0 con 0x7fe7ac0b6dc0 2024-03-24T00:48:37.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.686+0000 7fe7b240b640 1 -- 172.21.15.67:0/4078245014 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fe7ac0c6fc0 con 0x7fe7ac0b6dc0 2024-03-24T00:48:37.687 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-24T00:48:37.686+0000 7fbfcfa11640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T00:48:37.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.688+0000 7fe7a8ff9640 1 -- 172.21.15.67:0/4078245014 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v79) v1 ==== 161+0+0 (secure 0 0 0) 0x7fe79c058a20 con 0x7fe7ac0b6dc0 2024-03-24T00:48:37.689 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-24T00:48:37.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.690+0000 7fe7b240b640 1 -- 172.21.15.67:0/4078245014 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe79806bbd0 msgr2=0x7fe79806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:37.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.690+0000 7fe7b240b640 1 --2- 172.21.15.67:0/4078245014 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe79806bbd0 0x7fe79806e090 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac0b4d80 tx=0x7fe7a0008040 comp rx=0 tx=0).stop 2024-03-24T00:48:37.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.690+0000 7fe7b240b640 1 -- 172.21.15.67:0/4078245014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7ac0b6dc0 msgr2=0x7fe7ac15b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:37.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.690+0000 7fe7b240b640 1 --2- 172.21.15.67:0/4078245014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7ac0b6dc0 0x7fe7ac15b1d0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7fe79c00c430 tx=0x7fe79c00c8f0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.690+0000 7fe7b240b640 1 -- 172.21.15.67:0/4078245014 shutdown_connections 2024-03-24T00:48:37.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.690+0000 7fe7b240b640 1 --2- 172.21.15.67:0/4078245014 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe79806bbd0 0x7fe79806e090 secure :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac0b4d80 tx=0x7fe7a0008040 comp rx=0 tx=0).stop 2024-03-24T00:48:37.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.690+0000 7fe7b240b640 1 --2- 172.21.15.67:0/4078245014 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe7ac15e0f0 0x7fe7ac0cd810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.690+0000 7fe7b240b640 1 --2- 172.21.15.67:0/4078245014 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe7ac0d32d0 0x7fe7ac0cd2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.690+0000 7fe7b240b640 1 --2- 172.21.15.67:0/4078245014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7ac0b6dc0 0x7fe7ac15b1d0 secure :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7fe79c00c430 tx=0x7fe79c00c8f0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.690+0000 7fe7b240b640 1 -- 172.21.15.67:0/4078245014 >> 172.21.15.67:0/4078245014 conn(0x7fe7ac072b60 msgr2=0x7fe7ac0c88d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:37.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.691+0000 7fe7b240b640 1 -- 172.21.15.67:0/4078245014 shutdown_connections 2024-03-24T00:48:37.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.692+0000 7fe7b240b640 1 -- 172.21.15.67:0/4078245014 wait complete. 2024-03-24T00:48:37.693 INFO:tasks.ceph.mds.c.smithi067.stderr: -1149> 2024-03-24T00:48:06.924+0000 7efc3e158640 -1 log_channel(cluster) log [ERR] : scrub: inode wrongly marked free: 0x10000000001 2024-03-24T00:48:37.694 INFO:tasks.ceph.mds.c.smithi067.stderr: -1146> 2024-03-24T00:48:06.924+0000 7efc3e158640 -1 log_channel(cluster) log [ERR] : inode table repaired for inode: 0x10000000001 2024-03-24T00:48:37.694 INFO:tasks.ceph.mds.c.smithi067.stderr: -1118> 2024-03-24T00:48:06.925+0000 7efc3e158640 -1 log_channel(cluster) log [ERR] : scrub: inode wrongly marked free: 0x10000000002 2024-03-24T00:48:37.694 INFO:tasks.ceph.mds.c.smithi067.stderr: -1115> 2024-03-24T00:48:06.925+0000 7efc3e158640 -1 log_channel(cluster) log [ERR] : inode table repaired for inode: 0x10000000002 2024-03-24T00:48:37.694 INFO:tasks.ceph.mds.c.smithi067.stderr: -995> 2024-03-24T00:48:07.336+0000 7efc42160640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T00:48:37.704 DEBUG:teuthology.orchestra.run.smithi067:> cephfs-table-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 0 show inode 2024-03-24T00:48:37.718 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-03-24T00:48:37.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.741+0000 7f589d80e280 1 -- start start 2024-03-24T00:48:37.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.741+0000 7f589d80e280 1 Processor -- start 2024-03-24T00:48:37.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.742+0000 7f589d80e280 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558477970c00 0x558477a19b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:37.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.742+0000 7f589d80e280 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558477971800 0x558477a19600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:37.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.742+0000 7f589d80e280 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558477971c00 0x558477a19080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:37.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.742+0000 7f589d80e280 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x558478746180 con 0x558477971800 2024-03-24T00:48:37.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.742+0000 7f589d80e280 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584779d2900 con 0x558477971c00 2024-03-24T00:48:37.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.742+0000 7f589d80e280 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5584779d2480 con 0x558477970c00 2024-03-24T00:48:37.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.742+0000 7f589ab72640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558477971800 0x558477a19600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:37.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.742+0000 7f589ab72640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558477971800 0x558477a19600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36112/0 (socket says 172.21.15.67:36112) 2024-03-24T00:48:37.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.742+0000 7f589ab72640 1 -- 172.21.15.67:0/541294989 learned_addr learned my addr 172.21.15.67:0/541294989 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:37.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.742+0000 7f589ab72640 1 -- 172.21.15.67:0/541294989 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558477970c00 msgr2=0x558477a19b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:37.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.742+0000 7f589b373640 1 --2- 172.21.15.67:0/541294989 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558477970c00 0x558477a19b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:37.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.742+0000 7f589bb74640 1 --2- 172.21.15.67:0/541294989 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558477971c00 0x558477a19080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:37.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.742+0000 7f589ab72640 1 --2- 172.21.15.67:0/541294989 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558477970c00 0x558477a19b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.742+0000 7f589ab72640 1 -- 172.21.15.67:0/541294989 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558477971c00 msgr2=0x558477a19080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:37.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.742+0000 7f589ab72640 1 --2- 172.21.15.67:0/541294989 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558477971c00 0x558477a19080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.742+0000 7f589ab72640 1 -- 172.21.15.67:0/541294989 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5584779a6780 con 0x558477971800 2024-03-24T00:48:37.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.742+0000 7f589b373640 1 --2- 172.21.15.67:0/541294989 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558477970c00 0x558477a19b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:48:37.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.742+0000 7f589bb74640 1 --2- 172.21.15.67:0/541294989 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558477971c00 0x558477a19080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:48:37.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.742+0000 7f589ab72640 1 --2- 172.21.15.67:0/541294989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558477971800 0x558477a19600 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x5584787870e0 tx=0x558478708a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=997adf853de85611 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:37.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.742+0000 7f5898b6e640 1 -- 172.21.15.67:0/541294989 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5584779969c0 con 0x558477971800 2024-03-24T00:48:37.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.742+0000 7f5898b6e640 1 -- 172.21.15.67:0/541294989 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55847799c700 con 0x558477971800 2024-03-24T00:48:37.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.742+0000 7f589d80e280 1 -- 172.21.15.67:0/541294989 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5584779a74a0 con 0x558477971800 2024-03-24T00:48:37.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.742+0000 7f589d80e280 1 -- 172.21.15.67:0/541294989 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x5584779a6000 con 0x558477971800 2024-03-24T00:48:37.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.742+0000 7f589d80e280 4 waiting for MDS map... 2024-03-24T00:48:37.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.742+0000 7f5898b6e640 1 -- 172.21.15.67:0/541294989 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x558477996680 con 0x558477971800 2024-03-24T00:48:37.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.742+0000 7f5898b6e640 1 -- 172.21.15.67:0/541294989 <== mon.0 v2:172.21.15.67:3300/0 4 ==== osd_map(109..109 src has 1..109) v4 ==== 6302+0+0 (secure 0 0 0) 0x558478838200 con 0x558477971800 2024-03-24T00:48:37.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.743+0000 7f5898b6e640 1 -- 172.21.15.67:0/541294989 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 79) v1 ==== 2735+0+0 (secure 0 0 0) 0x5584779cb180 con 0x558477971800 2024-03-24T00:48:37.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.743+0000 7f589d80e280 4 Got MDS map 79 2024-03-24T00:48:37.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.743+0000 7f589d80e280 10 main: main 2024-03-24T00:48:37.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.743+0000 7f589d80e280 4 main: connecting to RADOS... 2024-03-24T00:48:37.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.743+0000 7f589d80e280 1 Processor -- start 2024-03-24T00:48:37.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.743+0000 7f589d80e280 1 -- start start 2024-03-24T00:48:37.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.744+0000 7f589d80e280 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558478894000 0x558477a18b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:37.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.744+0000 7f589d80e280 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558478894400 0x558477a18580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:37.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.744+0000 7f589d80e280 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558478894800 0x558477a1a100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:37.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.744+0000 7f589d80e280 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584779d2180 con 0x558478894400 2024-03-24T00:48:37.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.744+0000 7f589d80e280 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x558478746d80 con 0x558478894000 2024-03-24T00:48:37.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.744+0000 7f589d80e280 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x558478746f00 con 0x558478894800 2024-03-24T00:48:37.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.744+0000 7f589b373640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558478894400 0x558477a18580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:37.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.744+0000 7f589b373640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558478894400 0x558477a18580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36124/0 (socket says 172.21.15.67:36124) 2024-03-24T00:48:37.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.744+0000 7f589b373640 1 -- 172.21.15.67:0/2789765951 learned_addr learned my addr 172.21.15.67:0/2789765951 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:37.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.744+0000 7f589ab72640 1 --2- 172.21.15.67:0/2789765951 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558478894800 0x558477a1a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:37.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.744+0000 7f589bb74640 1 --2- 172.21.15.67:0/2789765951 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558478894000 0x558477a18b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:37.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.744+0000 7f589b373640 1 -- 172.21.15.67:0/2789765951 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558478894800 msgr2=0x558477a1a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:37.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.744+0000 7f589b373640 1 --2- 172.21.15.67:0/2789765951 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558478894800 0x558477a1a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.744+0000 7f589b373640 1 -- 172.21.15.67:0/2789765951 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558478894000 msgr2=0x558477a18b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:37.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.744+0000 7f589b373640 1 --2- 172.21.15.67:0/2789765951 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558478894000 0x558477a18b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.744+0000 7f589b373640 1 -- 172.21.15.67:0/2789765951 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5584779a63c0 con 0x558478894400 2024-03-24T00:48:37.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.744+0000 7f589b373640 1 --2- 172.21.15.67:0/2789765951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558478894400 0x558477a18580 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x5584788421e0 tx=0x5584788458c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b42b25ff28ed3eb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:37.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.744+0000 7f5896b6a640 1 -- 172.21.15.67:0/2789765951 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5584779969c0 con 0x558478894400 2024-03-24T00:48:37.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.744+0000 7f5896b6a640 1 -- 172.21.15.67:0/2789765951 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55847799ce00 con 0x558478894400 2024-03-24T00:48:37.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.744+0000 7f5896b6a640 1 -- 172.21.15.67:0/2789765951 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x558477996680 con 0x558478894400 2024-03-24T00:48:37.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.745+0000 7f589d80e280 1 -- 172.21.15.67:0/2789765951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558478894400 msgr2=0x558477a18580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:37.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.745+0000 7f589d80e280 1 --2- 172.21.15.67:0/2789765951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558478894400 0x558477a18580 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x5584788421e0 tx=0x5584788458c0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.745+0000 7f589d80e280 1 -- 172.21.15.67:0/2789765951 shutdown_connections 2024-03-24T00:48:37.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.745+0000 7f589d80e280 1 --2- 172.21.15.67:0/2789765951 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558478894800 0x558477a1a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.745+0000 7f589d80e280 1 --2- 172.21.15.67:0/2789765951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558478894400 0x558477a18580 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.745+0000 7f589d80e280 1 --2- 172.21.15.67:0/2789765951 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558478894000 0x558477a18b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.745+0000 7f589d80e280 1 -- 172.21.15.67:0/2789765951 >> 172.21.15.67:0/2789765951 conn(0x55847874dc00 msgr2=0x558477a61000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:37.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.745+0000 7f589d80e280 1 -- 172.21.15.67:0/2789765951 shutdown_connections 2024-03-24T00:48:37.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.745+0000 7f589d80e280 1 -- 172.21.15.67:0/2789765951 wait complete. 2024-03-24T00:48:37.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.745+0000 7f589d80e280 1 Processor -- start 2024-03-24T00:48:37.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.745+0000 7f589d80e280 1 -- start start 2024-03-24T00:48:37.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.746+0000 7f589d80e280 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558478894000 0x558477a18b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:37.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.746+0000 7f589d80e280 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558478894400 0x558477a18580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:37.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.746+0000 7f589d80e280 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558478894800 0x558477a1a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:37.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.746+0000 7f589d80e280 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x558478747200 con 0x558478894400 2024-03-24T00:48:37.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.746+0000 7f589d80e280 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x558478747080 con 0x558478894800 2024-03-24T00:48:37.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.746+0000 7f589d80e280 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x558478747380 con 0x558478894000 2024-03-24T00:48:37.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.746+0000 7f589b373640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558478894400 0x558477a18580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:37.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.746+0000 7f589b373640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558478894400 0x558477a18580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36138/0 (socket says 172.21.15.67:36138) 2024-03-24T00:48:37.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.746+0000 7f589b373640 1 -- 172.21.15.67:0/1499018874 learned_addr learned my addr 172.21.15.67:0/1499018874 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:37.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.746+0000 7f589ab72640 1 --2- 172.21.15.67:0/1499018874 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558478894800 0x558477a1a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:37.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.746+0000 7f589bb74640 1 --2- 172.21.15.67:0/1499018874 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558478894000 0x558477a18b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:37.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.746+0000 7f589b373640 1 -- 172.21.15.67:0/1499018874 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558478894000 msgr2=0x558477a18b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:37.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.746+0000 7f589b373640 1 --2- 172.21.15.67:0/1499018874 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558478894000 0x558477a18b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.746+0000 7f589b373640 1 -- 172.21.15.67:0/1499018874 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558478894800 msgr2=0x558477a1a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:37.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.746+0000 7f589b373640 1 --2- 172.21.15.67:0/1499018874 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558478894800 0x558477a1a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.746+0000 7f589b373640 1 -- 172.21.15.67:0/1499018874 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5584779a6960 con 0x558478894400 2024-03-24T00:48:37.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.746+0000 7f589bb74640 1 --2- 172.21.15.67:0/1499018874 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558478894000 0x558477a18b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:48:37.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.746+0000 7f589b373640 1 --2- 172.21.15.67:0/1499018874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558478894400 0x558477a18580 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x558478843320 tx=0x558478708900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:37.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.747+0000 7f5895367640 1 -- 172.21.15.67:0/1499018874 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5584779964e0 con 0x558478894400 2024-03-24T00:48:37.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.747+0000 7f5895367640 1 -- 172.21.15.67:0/1499018874 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55847799ca80 con 0x558478894400 2024-03-24T00:48:37.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.747+0000 7f589d80e280 1 -- 172.21.15.67:0/1499018874 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5584779a7680 con 0x558478894400 2024-03-24T00:48:37.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.747+0000 7f5895367640 1 -- 172.21.15.67:0/1499018874 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5584779969c0 con 0x558478894400 2024-03-24T00:48:37.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.747+0000 7f589d80e280 1 -- 172.21.15.67:0/1499018874 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5584779a7860 con 0x558478894400 2024-03-24T00:48:37.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.747+0000 7f589d80e280 4 main: resolving pool 15 2024-03-24T00:48:37.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.748+0000 7f5895367640 1 -- 172.21.15.67:0/1499018874 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x5584788a0c00 con 0x558478894400 2024-03-24T00:48:37.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.748+0000 7f5895367640 1 --2- 172.21.15.67:0/1499018874 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x558477cd8000 0x558477cda000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:37.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.748+0000 7f5895367640 1 -- 172.21.15.67:0/1499018874 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6302+0+0 (secure 0 0 0) 0x55847889b400 con 0x558478894400 2024-03-24T00:48:37.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.748+0000 7f589d80e280 4 main: creating IoCtx.. 2024-03-24T00:48:37.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.748+0000 7f589d80e280 1 --2- 172.21.15.67:0/1499018874 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x558478894c00 0x558477a1a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:37.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.748+0000 7f589d80e280 1 -- 172.21.15.67:0/1499018874 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:1 15.13 15:c91d4a1d:::mds0_inotable:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e109) v9 -- 0x558478895000 con 0x558478894c00 2024-03-24T00:48:37.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.748+0000 7f589bb74640 1 --2- 172.21.15.67:0/1499018874 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x558477cd8000 0x558477cda000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:37.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.748+0000 7f589ab72640 1 --2- 172.21.15.67:0/1499018874 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x558478894c00 0x558477a1a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:37.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.749+0000 7f589ab72640 1 --2- 172.21.15.67:0/1499018874 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x558478894c00 0x558477a1a680 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:37.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.749+0000 7f589bb74640 1 --2- 172.21.15.67:0/1499018874 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x558477cd8000 0x558477cda000 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x5584786f86f0 tx=0x558477c6ade0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:37.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.751+0000 7f589ab72640 1 -- 172.21.15.67:0/1499018874 <== osd.1 v2:172.21.15.67:6800/4062510404 1 ==== osd_op_reply(1 mds0_inotable [read 0~34 out=34b] v0'0 uv7 ondisk = 0) v8 ==== 157+0+34 (crc 0 0 0) 0x558477a66000 con 0x558478894c00 2024-03-24T00:48:37.757 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:48:37.757 INFO:teuthology.orchestra.run.smithi067.stdout: "0": { 2024-03-24T00:48:37.757 INFO:teuthology.orchestra.run.smithi067.stdout: "data": { 2024-03-24T00:48:37.757 INFO:teuthology.orchestra.run.smithi067.stdout: "version": 5, 2024-03-24T00:48:37.757 INFO:teuthology.orchestra.run.smithi067.stdout: "inotable": { 2024-03-24T00:48:37.757 INFO:teuthology.orchestra.run.smithi067.stdout: "projected_free": [ 2024-03-24T00:48:37.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:48:37.758 INFO:teuthology.orchestra.run.smithi067.stdout: "start": 1099511627779, 2024-03-24T00:48:37.758 INFO:teuthology.orchestra.run.smithi067.stdout: "len": 1099511627773 2024-03-24T00:48:37.758 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:48:37.758 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-24T00:48:37.758 INFO:teuthology.orchestra.run.smithi067.stdout: "free": [ 2024-03-24T00:48:37.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:48:37.758 INFO:teuthology.orchestra.run.smithi067.stdout: "start": 1099511627779, 2024-03-24T00:48:37.758 INFO:teuthology.orchestra.run.smithi067.stdout: "len": 1099511627773 2024-03-24T00:48:37.758 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:48:37.758 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-24T00:48:37.758 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:48:37.758 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:48:37.758 INFO:teuthology.orchestra.run.smithi067.stdout: "result": 0 2024-03-24T00:48:37.759 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:48:37.759 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:48:37.759 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:48:37.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.751+0000 7f589d80e280 1 -- 172.21.15.67:0/1499018874 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x558478894c00 msgr2=0x558477a1a680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:37.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.751+0000 7f589d80e280 1 --2- 172.21.15.67:0/1499018874 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x558478894c00 0x558477a1a680 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.751+0000 7f589d80e280 1 -- 172.21.15.67:0/1499018874 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x558477cd8000 msgr2=0x558477cda000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:37.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.751+0000 7f589d80e280 1 --2- 172.21.15.67:0/1499018874 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x558477cd8000 0x558477cda000 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x5584786f86f0 tx=0x558477c6ade0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.751+0000 7f589d80e280 1 -- 172.21.15.67:0/1499018874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558478894400 msgr2=0x558477a18580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:37.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.751+0000 7f589d80e280 1 --2- 172.21.15.67:0/1499018874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558478894400 0x558477a18580 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x558478843320 tx=0x558478708900 comp rx=0 tx=0).stop 2024-03-24T00:48:37.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.751+0000 7f589b373640 1 -- 172.21.15.67:0/1499018874 reap_dead start 2024-03-24T00:48:37.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.751+0000 7f589d80e280 1 -- 172.21.15.67:0/1499018874 shutdown_connections 2024-03-24T00:48:37.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.751+0000 7f589d80e280 1 -- 172.21.15.67:0/1499018874 >> 172.21.15.67:0/1499018874 conn(0x55847874dc00 msgr2=0x558477a61000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:37.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.751+0000 7f589d80e280 1 -- 172.21.15.67:0/1499018874 shutdown_connections 2024-03-24T00:48:37.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.751+0000 7f589d80e280 1 -- 172.21.15.67:0/1499018874 wait complete. 2024-03-24T00:48:37.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.751+0000 7f589d80e280 1 -- 172.21.15.67:0/541294989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558477971800 msgr2=0x558477a19600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:37.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.751+0000 7f589d80e280 1 --2- 172.21.15.67:0/541294989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558477971800 0x558477a19600 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x5584787870e0 tx=0x558478708a80 comp rx=0 tx=0).stop 2024-03-24T00:48:37.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.751+0000 7f589d80e280 1 -- 172.21.15.67:0/541294989 shutdown_connections 2024-03-24T00:48:37.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.751+0000 7f589d80e280 1 --2- 172.21.15.67:0/541294989 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558477971c00 0x558477a19080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.751+0000 7f589d80e280 1 --2- 172.21.15.67:0/541294989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558477971800 0x558477a19600 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.751+0000 7f589d80e280 1 --2- 172.21.15.67:0/541294989 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558477970c00 0x558477a19b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.751+0000 7f589d80e280 1 -- 172.21.15.67:0/541294989 >> 172.21.15.67:0/541294989 conn(0x558477971000 msgr2=0x558477a60800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:37.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.752+0000 7f589d80e280 1 -- 172.21.15.67:0/541294989 shutdown_connections 2024-03-24T00:48:37.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:37.752+0000 7f589d80e280 1 -- 172.21.15.67:0/541294989 wait complete. 2024-03-24T00:48:37.762 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-table-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', '0', 'show', 'inode'] in time 0:00:00.058562, result: { "0": { "data": { "version": 5, "inotable": { "projected_free": [ { "start": 1099511627779, "len": 1099511627773 } ], "free": [ { "start": 1099511627779, "len": 1099511627773 } ] } }, "result": 0 } } 2024-03-24T00:48:37.763 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-24T00:48:37.763 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:48:37.844 DEBUG:teuthology.orchestra.run.smithi072:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:48:37.894 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-24T00:48:37.897 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:48:37.897 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:48:37.911 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056375,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15771 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T00:48:37.912 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:48:37.912 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:48:37.929 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17590 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248808k,nr_inodes=812202,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:48:37.929 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:48:37.929 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:48:37.966 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056375,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15771 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T00:48:37.966 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:48:37.966 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:48:37.987 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17590 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248808k,nr_inodes=812202,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:48:37.987 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:48:38.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.106+0000 7f9d56212640 1 Processor -- start 2024-03-24T00:48:38.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.106+0000 7f9d56212640 1 -- start start 2024-03-24T00:48:38.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.107+0000 7f9d56212640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9d500d0ef0 0x7f9d500ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:38.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.107+0000 7f9d56212640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d500cf640 con 0x7f9d5015e2a0 2024-03-24T00:48:38.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.107+0000 7f9d56212640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d500cf080 con 0x7f9d5015dec0 2024-03-24T00:48:38.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.107+0000 7f9d56212640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d500cee30 con 0x7f9d500d0ef0 2024-03-24T00:48:38.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.107+0000 7f9d4ffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9d500d0ef0 0x7f9d500ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:38.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.107+0000 7f9d4effd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9d5015e2a0 0x7f9d500d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45584/0 (socket says 172.21.15.67:45584) 2024-03-24T00:48:38.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.107+0000 7f9d4effd640 1 -- 172.21.15.67:0/81741411 learned_addr learned my addr 172.21.15.67:0/81741411 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:48:38.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.108+0000 7f9d4ffff640 1 -- 172.21.15.67:0/81741411 >> v1:172.21.15.72:6789/0 conn(0x7f9d5015dec0 legacy=0x7f9d500d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:38.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.108+0000 7f9d4ffff640 1 -- 172.21.15.67:0/81741411 >> v1:172.21.15.67:6789/0 conn(0x7f9d5015e2a0 legacy=0x7f9d500d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:38.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.108+0000 7f9d4e7fc640 1 -- 172.21.15.67:0/81741411 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2963053335 0 0) 0x7f9d44001110 con 0x7f9d5015e2a0 2024-03-24T00:48:38.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.108+0000 7f9d4ffff640 1 -- 172.21.15.67:0/81741411 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d500cf360 con 0x7f9d500d0ef0 2024-03-24T00:48:38.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.108+0000 7f9d4ffff640 1 --2- 172.21.15.67:0/81741411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d500d0ef0 0x7f9d500ce630 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f9d500cefb0 tx=0x7f9d4802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=702d7cdfb40e326a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:38.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.108+0000 7f9d4e7fc640 1 -- 172.21.15.67:0/81741411 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d48035020 con 0x7f9d500d0ef0 2024-03-24T00:48:38.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.108+0000 7f9d4e7fc640 1 -- 172.21.15.67:0/81741411 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d48031830 con 0x7f9d500d0ef0 2024-03-24T00:48:38.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.108+0000 7f9d4e7fc640 1 -- 172.21.15.67:0/81741411 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d48031c00 con 0x7f9d500d0ef0 2024-03-24T00:48:38.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.109+0000 7f9d56212640 1 -- 172.21.15.67:0/81741411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d500d0ef0 msgr2=0x7f9d500ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:38.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.109+0000 7f9d56212640 1 --2- 172.21.15.67:0/81741411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d500d0ef0 0x7f9d500ce630 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f9d500cefb0 tx=0x7f9d4802b8d0 comp rx=0 tx=0).stop 2024-03-24T00:48:38.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.109+0000 7f9d56212640 1 -- 172.21.15.67:0/81741411 shutdown_connections 2024-03-24T00:48:38.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.109+0000 7f9d56212640 1 --2- 172.21.15.67:0/81741411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d500d0ef0 0x7f9d500ce630 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:38.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.109+0000 7f9d56212640 1 -- 172.21.15.67:0/81741411 >> 172.21.15.67:0/81741411 conn(0x7f9d5006c810 msgr2=0x7f9d5006cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:38.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.109+0000 7f9d56212640 1 -- 172.21.15.67:0/81741411 shutdown_connections 2024-03-24T00:48:38.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.109+0000 7f9d56212640 1 -- 172.21.15.67:0/81741411 wait complete. 2024-03-24T00:48:38.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.110+0000 7f9d56212640 1 Processor -- start 2024-03-24T00:48:38.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.110+0000 7f9d56212640 1 -- start start 2024-03-24T00:48:38.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.111+0000 7f9d56212640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9d500d0ef0 0x7f9d500b4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:38.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.111+0000 7f9d56212640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d5015dec0 0x7f9d500b3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:38.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.111+0000 7f9d56212640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d5015e2a0 0x7f9d500cbf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:38.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.111+0000 7f9d56212640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d50162a40 con 0x7f9d5015e2a0 2024-03-24T00:48:38.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.111+0000 7f9d56212640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d501628c0 con 0x7f9d5015dec0 2024-03-24T00:48:38.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.111+0000 7f9d56212640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d500d2120 con 0x7f9d500d0ef0 2024-03-24T00:48:38.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.111+0000 7f9d4ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d5015e2a0 0x7f9d500cbf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:38.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.111+0000 7f9d4ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d5015e2a0 0x7f9d500cbf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36150/0 (socket says 172.21.15.67:36150) 2024-03-24T00:48:38.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.111+0000 7f9d4ffff640 1 -- 172.21.15.67:0/1324690593 learned_addr learned my addr 172.21.15.67:0/1324690593 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:38.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.111+0000 7f9d4effd640 1 --2- 172.21.15.67:0/1324690593 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d5015dec0 0x7f9d500b3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:38.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.111+0000 7f9d4f7fe640 1 --2- 172.21.15.67:0/1324690593 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9d500d0ef0 0x7f9d500b4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:38.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.111+0000 7f9d4ffff640 1 -- 172.21.15.67:0/1324690593 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9d500d0ef0 msgr2=0x7f9d500b4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:38.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.111+0000 7f9d4ffff640 1 --2- 172.21.15.67:0/1324690593 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9d500d0ef0 0x7f9d500b4f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:38.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.111+0000 7f9d4ffff640 1 -- 172.21.15.67:0/1324690593 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d5015dec0 msgr2=0x7f9d500b3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:38.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.111+0000 7f9d4ffff640 1 --2- 172.21.15.67:0/1324690593 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d5015dec0 0x7f9d500b3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:38.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.111+0000 7f9d4ffff640 1 -- 172.21.15.67:0/1324690593 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d500b3ea0 con 0x7f9d5015e2a0 2024-03-24T00:48:38.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.111+0000 7f9d4f7fe640 1 --2- 172.21.15.67:0/1324690593 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9d500d0ef0 0x7f9d500b4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:48:38.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.111+0000 7f9d4effd640 1 --2- 172.21.15.67:0/1324690593 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d5015dec0 0x7f9d500b3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:48:38.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.112+0000 7f9d4ffff640 1 --2- 172.21.15.67:0/1324690593 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d5015e2a0 0x7f9d500cbf60 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f9d48009890 tx=0x7f9d480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:38.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.112+0000 7f9d37fff640 1 -- 172.21.15.67:0/1324690593 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d48035050 con 0x7f9d5015e2a0 2024-03-24T00:48:38.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.112+0000 7f9d37fff640 1 -- 172.21.15.67:0/1324690593 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d48004270 con 0x7f9d5015e2a0 2024-03-24T00:48:38.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.112+0000 7f9d56212640 1 -- 172.21.15.67:0/1324690593 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d500b4180 con 0x7f9d5015e2a0 2024-03-24T00:48:38.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.112+0000 7f9d37fff640 1 -- 172.21.15.67:0/1324690593 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d4803b880 con 0x7f9d5015e2a0 2024-03-24T00:48:38.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.112+0000 7f9d56212640 1 -- 172.21.15.67:0/1324690593 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d500b4740 con 0x7f9d5015e2a0 2024-03-24T00:48:38.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.113+0000 7f9d56212640 1 -- 172.21.15.67:0/1324690593 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d5004f400 con 0x7f9d5015e2a0 2024-03-24T00:48:38.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.113+0000 7f9d37fff640 1 -- 172.21.15.67:0/1324690593 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9d48038020 con 0x7f9d5015e2a0 2024-03-24T00:48:38.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.114+0000 7f9d37fff640 1 --2- 172.21.15.67:0/1324690593 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9d3c06bbd0 0x7f9d3c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:38.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.114+0000 7f9d37fff640 1 -- 172.21.15.67:0/1324690593 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6302+0+0 (secure 0 0 0) 0x7f9d480acaf0 con 0x7f9d5015e2a0 2024-03-24T00:48:38.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.114+0000 7f9d4f7fe640 1 --2- 172.21.15.67:0/1324690593 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9d3c06bbd0 0x7f9d3c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:38.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.114+0000 7f9d4f7fe640 1 --2- 172.21.15.67:0/1324690593 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9d3c06bbd0 0x7f9d3c06e090 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f9d400012e0 tx=0x7f9d40009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:38.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.118+0000 7f9d37fff640 1 -- 172.21.15.67:0/1324690593 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9d48040070 con 0x7f9d5015e2a0 2024-03-24T00:48:38.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.338+0000 7f9d56212640 1 -- 172.21.15.67:0/1324690593 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d500d1aa0 con 0x7f9d5015e2a0 2024-03-24T00:48:38.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.339+0000 7f9d37fff640 1 -- 172.21.15.67:0/1324690593 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+4172 (secure 0 0 0) 0x7f9d4807b420 con 0x7f9d5015e2a0 2024-03-24T00:48:38.340 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:48:38.340 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":79,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6902,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3672294289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3672294289},{"type":"v1","addr":"172.21.15.67:6835","nonce":3672294289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":6908,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/1286271878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":1286271878},{"type":"v1","addr":"172.21.15.72:6836","nonce":1286271878}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":6911,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/140622129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":140622129},{"type":"v1","addr":"172.21.15.72:6837","nonce":140622129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":79,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:47:16.492485+0000","modified":"2024-03-24T00:48:37.687883+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":109,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-03-24T00:48:38.340 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 79 2024-03-24T00:48:38.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.340+0000 7f9d56212640 1 -- 172.21.15.67:0/1324690593 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9d3c06bbd0 msgr2=0x7f9d3c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:38.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.341+0000 7f9d56212640 1 --2- 172.21.15.67:0/1324690593 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9d3c06bbd0 0x7f9d3c06e090 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f9d400012e0 tx=0x7f9d40009040 comp rx=0 tx=0).stop 2024-03-24T00:48:38.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.341+0000 7f9d56212640 1 -- 172.21.15.67:0/1324690593 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d5015e2a0 msgr2=0x7f9d500cbf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:38.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.341+0000 7f9d56212640 1 --2- 172.21.15.67:0/1324690593 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d5015e2a0 0x7f9d500cbf60 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f9d48009890 tx=0x7f9d480046c0 comp rx=0 tx=0).stop 2024-03-24T00:48:38.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.341+0000 7f9d56212640 1 -- 172.21.15.67:0/1324690593 shutdown_connections 2024-03-24T00:48:38.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.341+0000 7f9d56212640 1 --2- 172.21.15.67:0/1324690593 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9d3c06bbd0 0x7f9d3c06e090 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:38.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.341+0000 7f9d56212640 1 --2- 172.21.15.67:0/1324690593 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d5015e2a0 0x7f9d500cbf60 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:38.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.341+0000 7f9d56212640 1 --2- 172.21.15.67:0/1324690593 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d5015dec0 0x7f9d500b3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:38.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.341+0000 7f9d56212640 1 --2- 172.21.15.67:0/1324690593 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9d500d0ef0 0x7f9d500b4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:38.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.341+0000 7f9d56212640 1 -- 172.21.15.67:0/1324690593 >> 172.21.15.67:0/1324690593 conn(0x7f9d5006c810 msgr2=0x7f9d5015e6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:38.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.341+0000 7f9d56212640 1 -- 172.21.15.67:0/1324690593 shutdown_connections 2024-03-24T00:48:38.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.341+0000 7f9d56212640 1 -- 172.21.15.67:0/1324690593 wait complete. 2024-03-24T00:48:38.352 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:48:38.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.453+0000 7f6f1751b640 1 Processor -- start 2024-03-24T00:48:38.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.453+0000 7f6f1751b640 1 -- start start 2024-03-24T00:48:38.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.454+0000 7f6f1751b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6f1015dfb0 0x7f6f100cfe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:38.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.454+0000 7f6f1751b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6f1015e390 0x7f6f1006b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:38.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.454+0000 7f6f1751b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f100d07d0 con 0x7f6f100cded0 2024-03-24T00:48:38.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.454+0000 7f6f1751b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f100d0ab0 con 0x7f6f1015e390 2024-03-24T00:48:38.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.454+0000 7f6f1751b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f100d0d90 con 0x7f6f1015dfb0 2024-03-24T00:48:38.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.454+0000 7f6f14ab6640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6f1015e390 0x7f6f1006b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:38.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.454+0000 7f6f14ab6640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6f1015e390 0x7f6f1006b040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36154/0 (socket says 172.21.15.67:36154) 2024-03-24T00:48:38.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.454+0000 7f6f14ab6640 1 -- 172.21.15.67:0/2619917430 learned_addr learned my addr 172.21.15.67:0/2619917430 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:38.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.454+0000 7f6f152b7640 1 --2- 172.21.15.67:0/2619917430 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6f1015dfb0 0x7f6f100cfe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:38.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.454+0000 7f6f14ab6640 1 -- 172.21.15.67:0/2619917430 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6f1015dfb0 msgr2=0x7f6f100cfe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:38.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.454+0000 7f6f14ab6640 1 --2- 172.21.15.67:0/2619917430 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6f1015dfb0 0x7f6f100cfe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:38.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.454+0000 7f6f14ab6640 1 -- 172.21.15.67:0/2619917430 >> v1:172.21.15.72:6789/0 conn(0x7f6f100cded0 legacy=0x7f6f10161d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:38.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.454+0000 7f6f14ab6640 1 -- 172.21.15.67:0/2619917430 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f100d1070 con 0x7f6f1015e390 2024-03-24T00:48:38.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.454+0000 7f6f152b7640 1 --2- 172.21.15.67:0/2619917430 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6f1015dfb0 0x7f6f100cfe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:48:38.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.455+0000 7f6f14ab6640 1 --2- 172.21.15.67:0/2619917430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f1015e390 0x7f6f1006b040 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f6f00009870 tx=0x7f6f0002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e484aaa7f1b1091a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:38.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.455+0000 7f6f07fff640 1 -- 172.21.15.67:0/2619917430 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f00034070 con 0x7f6f1015e390 2024-03-24T00:48:38.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.455+0000 7f6f07fff640 1 -- 172.21.15.67:0/2619917430 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f00030840 con 0x7f6f1015e390 2024-03-24T00:48:38.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.455+0000 7f6f07fff640 1 -- 172.21.15.67:0/2619917430 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f00030c10 con 0x7f6f1015e390 2024-03-24T00:48:38.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.456+0000 7f6f1751b640 1 -- 172.21.15.67:0/2619917430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f1015e390 msgr2=0x7f6f1006b040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:38.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.456+0000 7f6f1751b640 1 --2- 172.21.15.67:0/2619917430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f1015e390 0x7f6f1006b040 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f6f00009870 tx=0x7f6f0002b980 comp rx=0 tx=0).stop 2024-03-24T00:48:38.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.456+0000 7f6f1751b640 1 -- 172.21.15.67:0/2619917430 shutdown_connections 2024-03-24T00:48:38.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.456+0000 7f6f1751b640 1 --2- 172.21.15.67:0/2619917430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f1015e390 0x7f6f1006b040 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:38.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.456+0000 7f6f1751b640 1 --2- 172.21.15.67:0/2619917430 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6f1015dfb0 0x7f6f100cfe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:38.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.456+0000 7f6f1751b640 1 -- 172.21.15.67:0/2619917430 >> 172.21.15.67:0/2619917430 conn(0x7f6f10099a30 msgr2=0x7f6f10099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:38.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.456+0000 7f6f1751b640 1 -- 172.21.15.67:0/2619917430 shutdown_connections 2024-03-24T00:48:38.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.456+0000 7f6f1751b640 1 -- 172.21.15.67:0/2619917430 wait complete. 2024-03-24T00:48:38.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.457+0000 7f6f1751b640 1 Processor -- start 2024-03-24T00:48:38.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.457+0000 7f6f1751b640 1 -- start start 2024-03-24T00:48:38.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.458+0000 7f6f1751b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f100cded0 0x7f6f10151070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:38.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.458+0000 7f6f1751b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6f1015dfb0 0x7f6f100ca450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:38.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.458+0000 7f6f1751b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f1015e390 0x7f6f100c6830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:38.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.458+0000 7f6f1751b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f100d1870 con 0x7f6f100cded0 2024-03-24T00:48:38.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.458+0000 7f6f1751b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f100cc130 con 0x7f6f1015e390 2024-03-24T00:48:38.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.458+0000 7f6f1751b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f100d03c0 con 0x7f6f1015dfb0 2024-03-24T00:48:38.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.458+0000 7f6f152b7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f100cded0 0x7f6f10151070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:38.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.458+0000 7f6f152b7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f100cded0 0x7f6f10151070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36160/0 (socket says 172.21.15.67:36160) 2024-03-24T00:48:38.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.458+0000 7f6f152b7640 1 -- 172.21.15.67:0/3189987278 learned_addr learned my addr 172.21.15.67:0/3189987278 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:38.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.458+0000 7f6f14ab6640 1 --2- 172.21.15.67:0/3189987278 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6f1015dfb0 0x7f6f100ca450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:38.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.458+0000 7f6f152b7640 1 -- 172.21.15.67:0/3189987278 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6f1015dfb0 msgr2=0x7f6f100ca450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:38.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.458+0000 7f6f152b7640 1 --2- 172.21.15.67:0/3189987278 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6f1015dfb0 0x7f6f100ca450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:38.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.458+0000 7f6f152b7640 1 -- 172.21.15.67:0/3189987278 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f1015e390 msgr2=0x7f6f100c6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:38.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.458+0000 7f6f15ab8640 1 --2- 172.21.15.67:0/3189987278 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f1015e390 0x7f6f100c6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:38.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.458+0000 7f6f152b7640 1 --2- 172.21.15.67:0/3189987278 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f1015e390 0x7f6f100c6830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:38.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.458+0000 7f6f152b7640 1 -- 172.21.15.67:0/3189987278 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f100ca990 con 0x7f6f100cded0 2024-03-24T00:48:38.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.458+0000 7f6f14ab6640 1 --2- 172.21.15.67:0/3189987278 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6f1015dfb0 0x7f6f100ca450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:48:38.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.458+0000 7f6f15ab8640 1 --2- 172.21.15.67:0/3189987278 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f1015e390 0x7f6f100c6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:48:38.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.458+0000 7f6f152b7640 1 --2- 172.21.15.67:0/3189987278 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f100cded0 0x7f6f10151070 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c00e7e0 tx=0x7f6f0c00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:38.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.458+0000 7f6f05ffb640 1 -- 172.21.15.67:0/3189987278 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f0c00f040 con 0x7f6f100cded0 2024-03-24T00:48:38.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.458+0000 7f6f05ffb640 1 -- 172.21.15.67:0/3189987278 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f0c004030 con 0x7f6f100cded0 2024-03-24T00:48:38.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.458+0000 7f6f05ffb640 1 -- 172.21.15.67:0/3189987278 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f0c004400 con 0x7f6f100cded0 2024-03-24T00:48:38.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.458+0000 7f6f1751b640 1 -- 172.21.15.67:0/3189987278 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f100c5790 con 0x7f6f100cded0 2024-03-24T00:48:38.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.459+0000 7f6f1751b640 1 -- 172.21.15.67:0/3189987278 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f100c5d50 con 0x7f6f100cded0 2024-03-24T00:48:38.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.460+0000 7f6f1751b640 1 -- 172.21.15.67:0/3189987278 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f1004f400 con 0x7f6f100cded0 2024-03-24T00:48:38.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.460+0000 7f6f05ffb640 1 -- 172.21.15.67:0/3189987278 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6f0c00c6d0 con 0x7f6f100cded0 2024-03-24T00:48:38.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.460+0000 7f6f05ffb640 1 --2- 172.21.15.67:0/3189987278 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6ef406bbd0 0x7f6ef406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:38.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.460+0000 7f6f05ffb640 1 -- 172.21.15.67:0/3189987278 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6302+0+0 (secure 0 0 0) 0x7f6f0c08aa60 con 0x7f6f100cded0 2024-03-24T00:48:38.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.461+0000 7f6f14ab6640 1 --2- 172.21.15.67:0/3189987278 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6ef406bbd0 0x7f6ef406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:38.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.461+0000 7f6f14ab6640 1 --2- 172.21.15.67:0/3189987278 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6ef406bbd0 0x7f6ef406e090 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f6f00036000 tx=0x7f6f0003b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:38.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.465+0000 7f6f05ffb640 1 -- 172.21.15.67:0/3189987278 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6f0c017020 con 0x7f6f100cded0 2024-03-24T00:48:38.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.648+0000 7f6f1751b640 1 -- 172.21.15.67:0/3189987278 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6f100cae00 con 0x7f6f100cded0 2024-03-24T00:48:38.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.649+0000 7f6f05ffb640 1 -- 172.21.15.67:0/3189987278 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) v1 ==== 74+0+17133 (secure 0 0 0) 0x7f6f0c059300 con 0x7f6f100cded0 2024-03-24T00:48:38.650 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:48:38.651 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":109,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:48:37.687686+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:47:15.111478+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-03-24T00:47:15.455832+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"16.1b","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6833/3197261495":"2024-03-25T00:48:37.687657+0000","172.21.15.67:6832/3197261495":"2024-03-25T00:48:37.687657+0000","172.21.15.67:6833/1826974843":"2024-03-25T00:48:01.322612+0000","172.21.15.67:6832/1826974843":"2024-03-25T00:48:01.322612+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:48:38.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.651+0000 7f6f1751b640 1 -- 172.21.15.67:0/3189987278 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6ef406bbd0 msgr2=0x7f6ef406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:38.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.651+0000 7f6f1751b640 1 --2- 172.21.15.67:0/3189987278 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6ef406bbd0 0x7f6ef406e090 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f6f00036000 tx=0x7f6f0003b040 comp rx=0 tx=0).stop 2024-03-24T00:48:38.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.651+0000 7f6f1751b640 1 -- 172.21.15.67:0/3189987278 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f100cded0 msgr2=0x7f6f10151070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:38.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.651+0000 7f6f1751b640 1 --2- 172.21.15.67:0/3189987278 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f100cded0 0x7f6f10151070 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c00e7e0 tx=0x7f6f0c00eca0 comp rx=0 tx=0).stop 2024-03-24T00:48:38.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.651+0000 7f6f1751b640 1 -- 172.21.15.67:0/3189987278 shutdown_connections 2024-03-24T00:48:38.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.651+0000 7f6f1751b640 1 --2- 172.21.15.67:0/3189987278 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6ef406bbd0 0x7f6ef406e090 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:38.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.651+0000 7f6f1751b640 1 --2- 172.21.15.67:0/3189987278 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f1015e390 0x7f6f100c6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:38.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.651+0000 7f6f1751b640 1 --2- 172.21.15.67:0/3189987278 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6f1015dfb0 0x7f6f100ca450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:38.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.651+0000 7f6f1751b640 1 --2- 172.21.15.67:0/3189987278 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f100cded0 0x7f6f10151070 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:38.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.651+0000 7f6f1751b640 1 -- 172.21.15.67:0/3189987278 >> 172.21.15.67:0/3189987278 conn(0x7f6f10099a30 msgr2=0x7f6f1006b040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:38.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.651+0000 7f6f1751b640 1 -- 172.21.15.67:0/3189987278 shutdown_connections 2024-03-24T00:48:38.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.651+0000 7f6f1751b640 1 -- 172.21.15.67:0/3189987278 wait complete. 2024-03-24T00:48:38.663 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:38:39.364950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:47:15.111478+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '104', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:47:15.455832+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '105', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:48:38.663 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:48:38.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.768+0000 7fa357e4b640 1 Processor -- start 2024-03-24T00:48:38.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.768+0000 7fa357e4b640 1 -- start start 2024-03-24T00:48:38.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.769+0000 7fa357e4b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa3500b49b0 0x7fa3500d0e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:38.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.769+0000 7fa357e4b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa3500d1360 0x7fa35015d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:38.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.769+0000 7fa357e4b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3500cc400 con 0x7fa350099d00 2024-03-24T00:48:38.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.769+0000 7fa357e4b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3500cc1b0 con 0x7fa3500d1360 2024-03-24T00:48:38.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.769+0000 7fa357e4b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3500cc6e0 con 0x7fa3500b49b0 2024-03-24T00:48:38.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.769+0000 7fa3563e8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa3500d1360 0x7fa35015d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:38.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.769+0000 7fa3563e8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa3500d1360 0x7fa35015d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36166/0 (socket says 172.21.15.67:36166) 2024-03-24T00:48:38.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.769+0000 7fa3563e8640 1 -- 172.21.15.67:0/261198620 learned_addr learned my addr 172.21.15.67:0/261198620 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:38.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.769+0000 7fa3553e6640 1 --2- 172.21.15.67:0/261198620 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa3500b49b0 0x7fa3500d0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:38.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.769+0000 7fa3563e8640 1 -- 172.21.15.67:0/261198620 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa3500b49b0 msgr2=0x7fa3500d0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:38.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.769+0000 7fa3563e8640 1 --2- 172.21.15.67:0/261198620 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa3500b49b0 0x7fa3500d0e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:38.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.769+0000 7fa3563e8640 1 -- 172.21.15.67:0/261198620 >> v1:172.21.15.72:6789/0 conn(0x7fa350099d00 legacy=0x7fa35009a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:38.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.769+0000 7fa3563e8640 1 -- 172.21.15.67:0/261198620 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3500cc9c0 con 0x7fa3500d1360 2024-03-24T00:48:38.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.769+0000 7fa3553e6640 1 --2- 172.21.15.67:0/261198620 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa3500b49b0 0x7fa3500d0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:48:38.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.770+0000 7fa3563e8640 1 --2- 172.21.15.67:0/261198620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3500d1360 0x7fa35015d7d0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7fa348009e50 tx=0x7fa34802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ff4fdc7dab07f4f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:38.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.770+0000 7fa354be5640 1 -- 172.21.15.67:0/261198620 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa34802cb00 con 0x7fa3500d1360 2024-03-24T00:48:38.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.770+0000 7fa354be5640 1 -- 172.21.15.67:0/261198620 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa34802cca0 con 0x7fa3500d1360 2024-03-24T00:48:38.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.770+0000 7fa354be5640 1 -- 172.21.15.67:0/261198620 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa348034610 con 0x7fa3500d1360 2024-03-24T00:48:38.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.771+0000 7fa357e4b640 1 -- 172.21.15.67:0/261198620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3500d1360 msgr2=0x7fa35015d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:38.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.771+0000 7fa357e4b640 1 --2- 172.21.15.67:0/261198620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3500d1360 0x7fa35015d7d0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7fa348009e50 tx=0x7fa34802bf60 comp rx=0 tx=0).stop 2024-03-24T00:48:38.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.771+0000 7fa357e4b640 1 -- 172.21.15.67:0/261198620 shutdown_connections 2024-03-24T00:48:38.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.771+0000 7fa357e4b640 1 --2- 172.21.15.67:0/261198620 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa3500b49b0 0x7fa3500d0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:38.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.771+0000 7fa357e4b640 1 --2- 172.21.15.67:0/261198620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3500d1360 0x7fa35015d7d0 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:38.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.771+0000 7fa357e4b640 1 -- 172.21.15.67:0/261198620 >> 172.21.15.67:0/261198620 conn(0x7fa3500b3650 msgr2=0x7fa3500b3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:38.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.771+0000 7fa357e4b640 1 -- 172.21.15.67:0/261198620 shutdown_connections 2024-03-24T00:48:38.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.771+0000 7fa357e4b640 1 -- 172.21.15.67:0/261198620 wait complete. 2024-03-24T00:48:38.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.772+0000 7fa357e4b640 1 Processor -- start 2024-03-24T00:48:38.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.772+0000 7fa357e4b640 1 -- start start 2024-03-24T00:48:38.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.772+0000 7fa357e4b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa350099d00 0x7fa35009d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:38.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.773+0000 7fa357e4b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa3500b49b0 0x7fa350160350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:38.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.773+0000 7fa357e4b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3500d1360 0x7fa35009b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:38.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.773+0000 7fa357e4b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3500b32a0 con 0x7fa350099d00 2024-03-24T00:48:38.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.773+0000 7fa357e4b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3500d01c0 con 0x7fa3500d1360 2024-03-24T00:48:38.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.773+0000 7fa357e4b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3500b3420 con 0x7fa3500b49b0 2024-03-24T00:48:38.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.773+0000 7fa355be7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa350099d00 0x7fa35009d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:38.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.773+0000 7fa355be7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa350099d00 0x7fa35009d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36168/0 (socket says 172.21.15.67:36168) 2024-03-24T00:48:38.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.773+0000 7fa355be7640 1 -- 172.21.15.67:0/1334768798 learned_addr learned my addr 172.21.15.67:0/1334768798 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:38.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.773+0000 7fa3553e6640 1 --2- 172.21.15.67:0/1334768798 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa3500b49b0 0x7fa350160350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:38.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.773+0000 7fa3563e8640 1 --2- 172.21.15.67:0/1334768798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3500d1360 0x7fa35009b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:38.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.773+0000 7fa355be7640 1 -- 172.21.15.67:0/1334768798 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa3500b49b0 msgr2=0x7fa350160350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:38.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.773+0000 7fa355be7640 1 --2- 172.21.15.67:0/1334768798 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa3500b49b0 0x7fa350160350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:38.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.773+0000 7fa355be7640 1 -- 172.21.15.67:0/1334768798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3500d1360 msgr2=0x7fa35009b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:38.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.773+0000 7fa355be7640 1 --2- 172.21.15.67:0/1334768798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3500d1360 0x7fa35009b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:38.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.773+0000 7fa355be7640 1 -- 172.21.15.67:0/1334768798 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa350160890 con 0x7fa350099d00 2024-03-24T00:48:38.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.773+0000 7fa355be7640 1 --2- 172.21.15.67:0/1334768798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa350099d00 0x7fa35009d2c0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7fa34c00c3f0 tx=0x7fa34c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:38.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.773+0000 7fa3467fc640 1 -- 172.21.15.67:0/1334768798 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa34c0052d0 con 0x7fa350099d00 2024-03-24T00:48:38.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.773+0000 7fa3467fc640 1 -- 172.21.15.67:0/1334768798 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa34c005490 con 0x7fa350099d00 2024-03-24T00:48:38.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.773+0000 7fa3467fc640 1 -- 172.21.15.67:0/1334768798 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa34c007260 con 0x7fa350099d00 2024-03-24T00:48:38.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.773+0000 7fa357e4b640 1 -- 172.21.15.67:0/1334768798 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa35009c220 con 0x7fa350099d00 2024-03-24T00:48:38.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.773+0000 7fa357e4b640 1 -- 172.21.15.67:0/1334768798 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa35009c7e0 con 0x7fa350099d00 2024-03-24T00:48:38.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.774+0000 7fa357e4b640 1 -- 172.21.15.67:0/1334768798 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa35004f400 con 0x7fa350099d00 2024-03-24T00:48:38.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.775+0000 7fa3467fc640 1 -- 172.21.15.67:0/1334768798 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa34c007400 con 0x7fa350099d00 2024-03-24T00:48:38.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.775+0000 7fa3467fc640 1 --2- 172.21.15.67:0/1334768798 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa33406bbd0 0x7fa33406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:38.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.775+0000 7fa3467fc640 1 -- 172.21.15.67:0/1334768798 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6302+0+0 (secure 0 0 0) 0x7fa34c08a060 con 0x7fa350099d00 2024-03-24T00:48:38.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.775+0000 7fa3553e6640 1 --2- 172.21.15.67:0/1334768798 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa33406bbd0 0x7fa33406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:38.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.776+0000 7fa3553e6640 1 --2- 172.21.15.67:0/1334768798 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa33406bbd0 0x7fa33406e090 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fa3500c6820 tx=0x7fa340008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:38.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.779+0000 7fa3467fc640 1 -- 172.21.15.67:0/1334768798 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa34c058340 con 0x7fa350099d00 2024-03-24T00:48:38.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.960+0000 7fa357e4b640 1 -- 172.21.15.67:0/1334768798 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa350070090 con 0x7fa350099d00 2024-03-24T00:48:38.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.960+0000 7fa3467fc640 1 -- 172.21.15.67:0/1334768798 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v109) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa34c05c3e0 con 0x7fa350099d00 2024-03-24T00:48:38.961 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-24T00:48:38.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.962+0000 7fa357e4b640 1 -- 172.21.15.67:0/1334768798 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa33406bbd0 msgr2=0x7fa33406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:38.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.962+0000 7fa357e4b640 1 --2- 172.21.15.67:0/1334768798 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa33406bbd0 0x7fa33406e090 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fa3500c6820 tx=0x7fa340008040 comp rx=0 tx=0).stop 2024-03-24T00:48:38.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.962+0000 7fa357e4b640 1 -- 172.21.15.67:0/1334768798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa350099d00 msgr2=0x7fa35009d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:38.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.962+0000 7fa357e4b640 1 --2- 172.21.15.67:0/1334768798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa350099d00 0x7fa35009d2c0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7fa34c00c3f0 tx=0x7fa34c00c8b0 comp rx=0 tx=0).stop 2024-03-24T00:48:38.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.962+0000 7fa357e4b640 1 -- 172.21.15.67:0/1334768798 shutdown_connections 2024-03-24T00:48:38.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.962+0000 7fa357e4b640 1 --2- 172.21.15.67:0/1334768798 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa33406bbd0 0x7fa33406e090 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:38.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.962+0000 7fa357e4b640 1 --2- 172.21.15.67:0/1334768798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3500d1360 0x7fa35009b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:38.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.962+0000 7fa357e4b640 1 --2- 172.21.15.67:0/1334768798 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa3500b49b0 0x7fa350160350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:38.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.962+0000 7fa357e4b640 1 --2- 172.21.15.67:0/1334768798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa350099d00 0x7fa35009d2c0 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:38.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.962+0000 7fa357e4b640 1 -- 172.21.15.67:0/1334768798 >> 172.21.15.67:0/1334768798 conn(0x7fa3500b3650 msgr2=0x7fa3500b4e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:38.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.962+0000 7fa357e4b640 1 -- 172.21.15.67:0/1334768798 shutdown_connections 2024-03-24T00:48:38.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:38.962+0000 7fa357e4b640 1 -- 172.21.15.67:0/1334768798 wait complete. 2024-03-24T00:48:38.974 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:48:39.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.077+0000 7fc76e3f4640 1 Processor -- start 2024-03-24T00:48:39.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.077+0000 7fc76e3f4640 1 -- start start 2024-03-24T00:48:39.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.078+0000 7fc76e3f4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc76815de90 0x7fc76815e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:39.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.078+0000 7fc76e3f4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc7680af4e0 0x7fc7680cfd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:39.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.078+0000 7fc76e3f4640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7680d1630 con 0x7fc7680cd2e0 2024-03-24T00:48:39.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.078+0000 7fc76e3f4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7680d1910 con 0x7fc76815de90 2024-03-24T00:48:39.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.078+0000 7fc76e3f4640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7680ad9f0 con 0x7fc7680af4e0 2024-03-24T00:48:39.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.078+0000 7fc7677fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc7680cd2e0 0x7fc76806b040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45600/0 (socket says 172.21.15.67:45600) 2024-03-24T00:48:39.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.078+0000 7fc7677fe640 1 -- 172.21.15.67:0/3385606219 learned_addr learned my addr 172.21.15.67:0/3385606219 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:48:39.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.078+0000 7fc76c991640 1 --2- 172.21.15.67:0/3385606219 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc7680af4e0 0x7fc7680cfd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:39.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.078+0000 7fc767fff640 1 --2- 172.21.15.67:0/3385606219 >> v2:172.21.15.72:3300/0 conn(0x7fc76815de90 0x7fc76815e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:39.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.078+0000 7fc766ffd640 1 -- 172.21.15.67:0/3385606219 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1707922810 0 0) 0x7fc7680d1630 con 0x7fc7680cd2e0 2024-03-24T00:48:39.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.078+0000 7fc766ffd640 1 -- 172.21.15.67:0/3385606219 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc750003520 con 0x7fc7680cd2e0 2024-03-24T00:48:39.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.078+0000 7fc766ffd640 1 -- 172.21.15.67:0/3385606219 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 210157733 0 0) 0x7fc750003520 con 0x7fc7680cd2e0 2024-03-24T00:48:39.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.078+0000 7fc766ffd640 1 -- 172.21.15.67:0/3385606219 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc7680af4e0 msgr2=0x7fc7680cfd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:39.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.078+0000 7fc766ffd640 1 --2- 172.21.15.67:0/3385606219 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc7680af4e0 0x7fc7680cfd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:39.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.078+0000 7fc766ffd640 1 -- 172.21.15.67:0/3385606219 >> v2:172.21.15.72:3300/0 conn(0x7fc76815de90 msgr2=0x7fc76815e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:39.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.078+0000 7fc766ffd640 1 --2- 172.21.15.67:0/3385606219 >> v2:172.21.15.72:3300/0 conn(0x7fc76815de90 0x7fc76815e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:39.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.078+0000 7fc766ffd640 1 -- 172.21.15.67:0/3385606219 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7680adcd0 con 0x7fc7680cd2e0 2024-03-24T00:48:39.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.078+0000 7fc766ffd640 1 -- 172.21.15.67:0/3385606219 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7fc758001670 con 0x7fc7680cd2e0 2024-03-24T00:48:39.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.078+0000 7fc766ffd640 1 -- 172.21.15.67:0/3385606219 >> v1:172.21.15.67:6789/0 conn(0x7fc7680cd2e0 legacy=0x7fc76806b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:39.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.078+0000 7fc766ffd640 1 --2- 172.21.15.67:0/3385606219 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7500039b0 0x7fc750003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:39.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.078+0000 7fc766ffd640 1 --2- 172.21.15.67:0/3385606219 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc750004800 0x7fc750004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:39.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.078+0000 7fc766ffd640 1 --2- 172.21.15.67:0/3385606219 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc7500051c0 0x7fc75000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:39.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.078+0000 7fc766ffd640 1 -- 172.21.15.67:0/3385606219 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7680d1910 con 0x7fc7500039b0 2024-03-24T00:48:39.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.078+0000 7fc766ffd640 1 -- 172.21.15.67:0/3385606219 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7680ad9f0 con 0x7fc750004800 2024-03-24T00:48:39.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.078+0000 7fc766ffd640 1 -- 172.21.15.67:0/3385606219 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc75000bf30 con 0x7fc7500051c0 2024-03-24T00:48:39.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.078+0000 7fc767fff640 1 --2- 172.21.15.67:0/3385606219 >> v2:172.21.15.72:3300/0 conn(0x7fc76815de90 0x7fc76815e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:48:39.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.078+0000 7fc76c991640 1 --2- 172.21.15.67:0/3385606219 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc7680af4e0 0x7fc7680cfd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:48:39.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.079+0000 7fc767fff640 1 --2- 172.21.15.67:0/3385606219 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7500039b0 0x7fc750003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:39.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.079+0000 7fc7677fe640 1 --2- 172.21.15.67:0/3385606219 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc750004800 0x7fc750004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:39.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.079+0000 7fc76c991640 1 --2- 172.21.15.67:0/3385606219 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc7500051c0 0x7fc75000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:39.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.079+0000 7fc767fff640 1 -- 172.21.15.67:0/3385606219 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc7500051c0 msgr2=0x7fc75000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:39.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.079+0000 7fc767fff640 1 --2- 172.21.15.67:0/3385606219 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc7500051c0 0x7fc75000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:39.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.079+0000 7fc767fff640 1 -- 172.21.15.67:0/3385606219 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc750004800 msgr2=0x7fc750004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:39.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.079+0000 7fc767fff640 1 --2- 172.21.15.67:0/3385606219 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc750004800 0x7fc750004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:39.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.079+0000 7fc767fff640 1 -- 172.21.15.67:0/3385606219 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc750003520 con 0x7fc7500039b0 2024-03-24T00:48:39.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.079+0000 7fc7677fe640 1 --2- 172.21.15.67:0/3385606219 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc750004800 0x7fc750004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:48:39.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.079+0000 7fc76c991640 1 --2- 172.21.15.67:0/3385606219 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc7500051c0 0x7fc75000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:48:39.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.080+0000 7fc767fff640 1 --2- 172.21.15.67:0/3385606219 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7500039b0 0x7fc750003dd0 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7fc75c009c70 tx=0x7fc750005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5b8d997fecb6bb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:39.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.080+0000 7fc766ffd640 1 -- 172.21.15.67:0/3385606219 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc75c009840 con 0x7fc7500039b0 2024-03-24T00:48:39.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.080+0000 7fc766ffd640 1 -- 172.21.15.67:0/3385606219 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc75c004020 con 0x7fc7500039b0 2024-03-24T00:48:39.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.080+0000 7fc766ffd640 1 -- 172.21.15.67:0/3385606219 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc75c004410 con 0x7fc7500039b0 2024-03-24T00:48:39.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.080+0000 7fc76e3f4640 1 -- 172.21.15.67:0/3385606219 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7500039b0 msgr2=0x7fc750003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:39.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.080+0000 7fc76e3f4640 1 --2- 172.21.15.67:0/3385606219 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7500039b0 0x7fc750003dd0 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7fc75c009c70 tx=0x7fc750005830 comp rx=0 tx=0).stop 2024-03-24T00:48:39.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.080+0000 7fc76c991640 1 -- 172.21.15.67:0/3385606219 reap_dead start 2024-03-24T00:48:39.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.080+0000 7fc76e3f4640 1 -- 172.21.15.67:0/3385606219 shutdown_connections 2024-03-24T00:48:39.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.080+0000 7fc76e3f4640 1 -- 172.21.15.67:0/3385606219 >> 172.21.15.67:0/3385606219 conn(0x7fc768099a30 msgr2=0x7fc768099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:39.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.080+0000 7fc76e3f4640 1 -- 172.21.15.67:0/3385606219 shutdown_connections 2024-03-24T00:48:39.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.080+0000 7fc76e3f4640 1 -- 172.21.15.67:0/3385606219 wait complete. 2024-03-24T00:48:39.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.081+0000 7fc76e3f4640 1 Processor -- start 2024-03-24T00:48:39.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.082+0000 7fc76e3f4640 1 -- start start 2024-03-24T00:48:39.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.082+0000 7fc76e3f4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7500039b0 0x7fc768154020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:39.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.082+0000 7fc76e3f4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc7680b09c0 0x7fc7680b4150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:39.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.082+0000 7fc76e3f4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc7680b4690 0x7fc7680b5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:39.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.082+0000 7fc76e3f4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc76806b8d0 con 0x7fc7500039b0 2024-03-24T00:48:39.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.082+0000 7fc76e3f4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc76806b750 con 0x7fc7680b09c0 2024-03-24T00:48:39.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.082+0000 7fc76e3f4640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7680cfd30 con 0x7fc7680b4690 2024-03-24T00:48:39.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.082+0000 7fc767fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7500039b0 0x7fc768154020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:39.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.082+0000 7fc767fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7500039b0 0x7fc768154020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36178/0 (socket says 172.21.15.67:36178) 2024-03-24T00:48:39.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.082+0000 7fc767fff640 1 -- 172.21.15.67:0/3578109900 learned_addr learned my addr 172.21.15.67:0/3578109900 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:39.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.083+0000 7fc76c991640 1 --2- 172.21.15.67:0/3578109900 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc7680b4690 0x7fc7680b5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:39.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.083+0000 7fc7677fe640 1 --2- 172.21.15.67:0/3578109900 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc7680b09c0 0x7fc7680b4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:39.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.083+0000 7fc767fff640 1 -- 172.21.15.67:0/3578109900 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc7680b4690 msgr2=0x7fc7680b5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:39.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.083+0000 7fc767fff640 1 --2- 172.21.15.67:0/3578109900 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc7680b4690 0x7fc7680b5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:39.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.083+0000 7fc767fff640 1 -- 172.21.15.67:0/3578109900 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc7680b09c0 msgr2=0x7fc7680b4150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:39.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.083+0000 7fc767fff640 1 --2- 172.21.15.67:0/3578109900 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc7680b09c0 0x7fc7680b4150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:39.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.083+0000 7fc767fff640 1 -- 172.21.15.67:0/3578109900 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc768154560 con 0x7fc7500039b0 2024-03-24T00:48:39.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.083+0000 7fc76c991640 1 --2- 172.21.15.67:0/3578109900 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc7680b4690 0x7fc7680b5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:48:39.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.083+0000 7fc7677fe640 1 --2- 172.21.15.67:0/3578109900 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc7680b09c0 0x7fc7680b4150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:48:39.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.083+0000 7fc767fff640 1 --2- 172.21.15.67:0/3578109900 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7500039b0 0x7fc768154020 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7fc75c02c560 tx=0x7fc75c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:39.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.083+0000 7fc764ff9640 1 -- 172.21.15.67:0/3578109900 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc75c043020 con 0x7fc7500039b0 2024-03-24T00:48:39.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.083+0000 7fc764ff9640 1 -- 172.21.15.67:0/3578109900 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc75c02c950 con 0x7fc7500039b0 2024-03-24T00:48:39.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.083+0000 7fc76e3f4640 1 -- 172.21.15.67:0/3578109900 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7680b4e10 con 0x7fc7500039b0 2024-03-24T00:48:39.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.083+0000 7fc764ff9640 1 -- 172.21.15.67:0/3578109900 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc75c006e10 con 0x7fc7500039b0 2024-03-24T00:48:39.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.083+0000 7fc76e3f4640 1 -- 172.21.15.67:0/3578109900 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc76815e310 con 0x7fc7500039b0 2024-03-24T00:48:39.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.084+0000 7fc76e3f4640 1 -- 172.21.15.67:0/3578109900 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc76804f400 con 0x7fc7500039b0 2024-03-24T00:48:39.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.084+0000 7fc764ff9640 1 -- 172.21.15.67:0/3578109900 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc75c046020 con 0x7fc7500039b0 2024-03-24T00:48:39.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.085+0000 7fc764ff9640 1 --2- 172.21.15.67:0/3578109900 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc75406bbd0 0x7fc75406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:39.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.085+0000 7fc764ff9640 1 -- 172.21.15.67:0/3578109900 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6302+0+0 (secure 0 0 0) 0x7fc75c0acdc0 con 0x7fc7500039b0 2024-03-24T00:48:39.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.085+0000 7fc7677fe640 1 --2- 172.21.15.67:0/3578109900 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc75406bbd0 0x7fc75406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:39.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.086+0000 7fc7677fe640 1 --2- 172.21.15.67:0/3578109900 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc75406bbd0 0x7fc75406e090 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fc758000ff0 tx=0x7fc758009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:39.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.089+0000 7fc764ff9640 1 -- 172.21.15.67:0/3578109900 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc75c04d030 con 0x7fc7500039b0 2024-03-24T00:48:39.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.273+0000 7fc76e3f4640 1 -- 172.21.15.67:0/3578109900 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc7680cead0 con 0x7fc7500039b0 2024-03-24T00:48:39.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.273+0000 7fc764ff9640 1 -- 172.21.15.67:0/3578109900 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v109) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc75c02cd00 con 0x7fc7500039b0 2024-03-24T00:48:39.274 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-24T00:48:39.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.275+0000 7fc76e3f4640 1 -- 172.21.15.67:0/3578109900 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc75406bbd0 msgr2=0x7fc75406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:39.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.275+0000 7fc76e3f4640 1 --2- 172.21.15.67:0/3578109900 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc75406bbd0 0x7fc75406e090 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fc758000ff0 tx=0x7fc758009040 comp rx=0 tx=0).stop 2024-03-24T00:48:39.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.275+0000 7fc76e3f4640 1 -- 172.21.15.67:0/3578109900 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7500039b0 msgr2=0x7fc768154020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:39.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.275+0000 7fc76e3f4640 1 --2- 172.21.15.67:0/3578109900 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7500039b0 0x7fc768154020 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7fc75c02c560 tx=0x7fc75c004730 comp rx=0 tx=0).stop 2024-03-24T00:48:39.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.275+0000 7fc76e3f4640 1 -- 172.21.15.67:0/3578109900 shutdown_connections 2024-03-24T00:48:39.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.275+0000 7fc76e3f4640 1 --2- 172.21.15.67:0/3578109900 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc75406bbd0 0x7fc75406e090 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:39.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.275+0000 7fc76e3f4640 1 --2- 172.21.15.67:0/3578109900 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc7680b4690 0x7fc7680b5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:39.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.275+0000 7fc76e3f4640 1 --2- 172.21.15.67:0/3578109900 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc7680b09c0 0x7fc7680b4150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:39.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.275+0000 7fc76e3f4640 1 --2- 172.21.15.67:0/3578109900 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7500039b0 0x7fc768154020 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:39.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.275+0000 7fc76e3f4640 1 -- 172.21.15.67:0/3578109900 >> 172.21.15.67:0/3578109900 conn(0x7fc768099a30 msgr2=0x7fc7680d0130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:39.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.275+0000 7fc76e3f4640 1 -- 172.21.15.67:0/3578109900 shutdown_connections 2024-03-24T00:48:39.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.275+0000 7fc76e3f4640 1 -- 172.21.15.67:0/3578109900 wait complete. 2024-03-24T00:48:39.287 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:48:39.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.390+0000 7fa2ec33b640 1 Processor -- start 2024-03-24T00:48:39.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.390+0000 7fa2ec33b640 1 -- start start 2024-03-24T00:48:39.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.391+0000 7fa2ec33b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa2e415bc30 0x7fa2e40b1a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:39.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.391+0000 7fa2ec33b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa2e415c010 0x7fa2e40b1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:39.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.391+0000 7fa2ec33b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2e40b3ae0 con 0x7fa2e40b2d90 2024-03-24T00:48:39.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.391+0000 7fa2ec33b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2e40cca60 con 0x7fa2e415c010 2024-03-24T00:48:39.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.391+0000 7fa2ec33b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e40b3dc0 con 0x7fa2e415bc30 2024-03-24T00:48:39.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.391+0000 7fa2ea8d8640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa2e40b2d90 0x7fa2e40b3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45614/0 (socket says 172.21.15.67:45614) 2024-03-24T00:48:39.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.391+0000 7fa2ea8d8640 1 -- 172.21.15.67:0/3284338801 learned_addr learned my addr 172.21.15.67:0/3284338801 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:48:39.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.392+0000 7fa2ea0d7640 1 --2- 172.21.15.67:0/3284338801 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa2e415bc30 0x7fa2e40b1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:39.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.392+0000 7fa2e90d5640 1 -- 172.21.15.67:0/3284338801 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1642139343 0 0) 0x7fa2e40b3ae0 con 0x7fa2e40b2d90 2024-03-24T00:48:39.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.392+0000 7fa2e98d6640 1 --2- 172.21.15.67:0/3284338801 >> v2:172.21.15.72:3300/0 conn(0x7fa2e415c010 0x7fa2e40b1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:39.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.392+0000 7fa2e90d5640 1 -- 172.21.15.67:0/3284338801 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2cc003520 con 0x7fa2e40b2d90 2024-03-24T00:48:39.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.392+0000 7fa2e90d5640 1 -- 172.21.15.67:0/3284338801 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3161546650 0 0) 0x7fa2cc003520 con 0x7fa2e40b2d90 2024-03-24T00:48:39.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.392+0000 7fa2e90d5640 1 -- 172.21.15.67:0/3284338801 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa2e415bc30 msgr2=0x7fa2e40b1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:39.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.392+0000 7fa2e90d5640 1 --2- 172.21.15.67:0/3284338801 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa2e415bc30 0x7fa2e40b1a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:39.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.392+0000 7fa2e90d5640 1 -- 172.21.15.67:0/3284338801 >> v2:172.21.15.72:3300/0 conn(0x7fa2e415c010 msgr2=0x7fa2e40b1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:39.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.392+0000 7fa2e90d5640 1 --2- 172.21.15.67:0/3284338801 >> v2:172.21.15.72:3300/0 conn(0x7fa2e415c010 0x7fa2e40b1f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:39.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.392+0000 7fa2e90d5640 1 -- 172.21.15.67:0/3284338801 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e40b40a0 con 0x7fa2e40b2d90 2024-03-24T00:48:39.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.392+0000 7fa2e90d5640 1 -- 172.21.15.67:0/3284338801 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7fa2dc001670 con 0x7fa2e40b2d90 2024-03-24T00:48:39.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.392+0000 7fa2e90d5640 1 -- 172.21.15.67:0/3284338801 >> v1:172.21.15.67:6789/0 conn(0x7fa2e40b2d90 legacy=0x7fa2e40b3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:39.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.392+0000 7fa2e90d5640 1 --2- 172.21.15.67:0/3284338801 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2cc0039b0 0x7fa2cc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:39.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.392+0000 7fa2e90d5640 1 --2- 172.21.15.67:0/3284338801 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa2e415bc30 0x7fa2cc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:39.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.392+0000 7fa2e90d5640 1 --2- 172.21.15.67:0/3284338801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2cc004d40 0x7fa2cc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:39.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.392+0000 7fa2e90d5640 1 -- 172.21.15.67:0/3284338801 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e40cca60 con 0x7fa2cc004d40 2024-03-24T00:48:39.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.392+0000 7fa2e90d5640 1 -- 172.21.15.67:0/3284338801 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e40b3dc0 con 0x7fa2cc0039b0 2024-03-24T00:48:39.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.392+0000 7fa2e90d5640 1 -- 172.21.15.67:0/3284338801 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2cc00fdd0 con 0x7fa2e415bc30 2024-03-24T00:48:39.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.392+0000 7fa2ea8d8640 1 --2- 172.21.15.67:0/3284338801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2cc004d40 0x7fa2cc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:39.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.392+0000 7fa2e98d6640 1 --2- 172.21.15.67:0/3284338801 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa2e415bc30 0x7fa2cc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:39.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.392+0000 7fa2ea0d7640 1 --2- 172.21.15.67:0/3284338801 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2cc0039b0 0x7fa2cc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:39.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.392+0000 7fa2ea8d8640 1 -- 172.21.15.67:0/3284338801 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa2e415bc30 msgr2=0x7fa2cc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:39.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.392+0000 7fa2ea8d8640 1 --2- 172.21.15.67:0/3284338801 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa2e415bc30 0x7fa2cc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:39.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.392+0000 7fa2ea8d8640 1 -- 172.21.15.67:0/3284338801 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2cc0039b0 msgr2=0x7fa2cc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:39.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.392+0000 7fa2ea8d8640 1 --2- 172.21.15.67:0/3284338801 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2cc0039b0 0x7fa2cc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:39.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.392+0000 7fa2ea8d8640 1 -- 172.21.15.67:0/3284338801 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2cc003520 con 0x7fa2cc004d40 2024-03-24T00:48:39.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.393+0000 7fa2ea8d8640 1 --2- 172.21.15.67:0/3284338801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2cc004d40 0x7fa2cc0051c0 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc007fd0 tx=0x7fa2cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58b0d80da1aeff9f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:39.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.393+0000 7fa2e90d5640 1 -- 172.21.15.67:0/3284338801 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2dc007d50 con 0x7fa2cc004d40 2024-03-24T00:48:39.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.393+0000 7fa2e90d5640 1 -- 172.21.15.67:0/3284338801 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2dc030070 con 0x7fa2cc004d40 2024-03-24T00:48:39.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.393+0000 7fa2e90d5640 1 -- 172.21.15.67:0/3284338801 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2dc02cd40 con 0x7fa2cc004d40 2024-03-24T00:48:39.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.394+0000 7fa2ec33b640 1 -- 172.21.15.67:0/3284338801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2cc004d40 msgr2=0x7fa2cc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:39.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.394+0000 7fa2ec33b640 1 --2- 172.21.15.67:0/3284338801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2cc004d40 0x7fa2cc0051c0 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc007fd0 tx=0x7fa2cc005830 comp rx=0 tx=0).stop 2024-03-24T00:48:39.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.394+0000 7fa2ea8d8640 1 -- 172.21.15.67:0/3284338801 reap_dead start 2024-03-24T00:48:39.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.394+0000 7fa2ec33b640 1 -- 172.21.15.67:0/3284338801 shutdown_connections 2024-03-24T00:48:39.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.394+0000 7fa2ec33b640 1 -- 172.21.15.67:0/3284338801 >> 172.21.15.67:0/3284338801 conn(0x7fa2e408b180 msgr2=0x7fa2e408b590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:39.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.394+0000 7fa2ec33b640 1 -- 172.21.15.67:0/3284338801 shutdown_connections 2024-03-24T00:48:39.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.394+0000 7fa2ec33b640 1 -- 172.21.15.67:0/3284338801 wait complete. 2024-03-24T00:48:39.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.395+0000 7fa2ec33b640 1 Processor -- start 2024-03-24T00:48:39.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.395+0000 7fa2ec33b640 1 -- start start 2024-03-24T00:48:39.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.396+0000 7fa2ec33b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2cc004d40 0x7fa2e40af080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:39.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.396+0000 7fa2ec33b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa2e414dd90 0x7fa2e415afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:39.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.396+0000 7fa2ec33b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2e414d210 0x7fa2e40b4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:39.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.396+0000 7fa2ec33b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e40b1b80 con 0x7fa2e414d210 2024-03-24T00:48:39.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.396+0000 7fa2ec33b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e40b1a00 con 0x7fa2cc004d40 2024-03-24T00:48:39.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.396+0000 7fa2ec33b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e40b1d00 con 0x7fa2e414dd90 2024-03-24T00:48:39.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.396+0000 7fa2ea8d8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2e414d210 0x7fa2e40b4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:39.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.396+0000 7fa2ea8d8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2e414d210 0x7fa2e40b4640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36206/0 (socket says 172.21.15.67:36206) 2024-03-24T00:48:39.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.396+0000 7fa2ea8d8640 1 -- 172.21.15.67:0/1894964362 learned_addr learned my addr 172.21.15.67:0/1894964362 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:39.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.396+0000 7fa2e98d6640 1 --2- 172.21.15.67:0/1894964362 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa2e414dd90 0x7fa2e415afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:39.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.396+0000 7fa2ea0d7640 1 --2- 172.21.15.67:0/1894964362 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2cc004d40 0x7fa2e40af080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:39.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.396+0000 7fa2ea8d8640 1 -- 172.21.15.67:0/1894964362 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa2e414dd90 msgr2=0x7fa2e415afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:39.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.396+0000 7fa2ea8d8640 1 --2- 172.21.15.67:0/1894964362 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa2e414dd90 0x7fa2e415afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:39.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.396+0000 7fa2ea8d8640 1 -- 172.21.15.67:0/1894964362 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2cc004d40 msgr2=0x7fa2e40af080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:39.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.396+0000 7fa2ea8d8640 1 --2- 172.21.15.67:0/1894964362 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2cc004d40 0x7fa2e40af080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:39.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.396+0000 7fa2ea8d8640 1 -- 172.21.15.67:0/1894964362 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e414d990 con 0x7fa2e414d210 2024-03-24T00:48:39.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.396+0000 7fa2ea8d8640 1 --2- 172.21.15.67:0/1894964362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2e414d210 0x7fa2e40b4640 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc000c00 tx=0x7fa2e40cc9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:39.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.396+0000 7fa2e2ffd640 1 -- 172.21.15.67:0/1894964362 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2dc001cb0 con 0x7fa2e414d210 2024-03-24T00:48:39.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.397+0000 7fa2e2ffd640 1 -- 172.21.15.67:0/1894964362 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2dc004bf0 con 0x7fa2e414d210 2024-03-24T00:48:39.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.397+0000 7fa2e2ffd640 1 -- 172.21.15.67:0/1894964362 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2dc007070 con 0x7fa2e414d210 2024-03-24T00:48:39.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.397+0000 7fa2ec33b640 1 -- 172.21.15.67:0/1894964362 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2e415d6a0 con 0x7fa2e414d210 2024-03-24T00:48:39.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.397+0000 7fa2ec33b640 1 -- 172.21.15.67:0/1894964362 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2e414d6a0 con 0x7fa2e414d210 2024-03-24T00:48:39.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.398+0000 7fa2ec33b640 1 -- 172.21.15.67:0/1894964362 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2e404f400 con 0x7fa2e414d210 2024-03-24T00:48:39.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.398+0000 7fa2e2ffd640 1 -- 172.21.15.67:0/1894964362 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa2dc007210 con 0x7fa2e414d210 2024-03-24T00:48:39.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.398+0000 7fa2e2ffd640 1 --2- 172.21.15.67:0/1894964362 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa2d006bc00 0x7fa2d006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:39.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.398+0000 7fa2e2ffd640 1 -- 172.21.15.67:0/1894964362 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6302+0+0 (secure 0 0 0) 0x7fa2dc0ac270 con 0x7fa2e414d210 2024-03-24T00:48:39.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.399+0000 7fa2ea0d7640 1 --2- 172.21.15.67:0/1894964362 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa2d006bc00 0x7fa2d006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:39.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.399+0000 7fa2ea0d7640 1 --2- 172.21.15.67:0/1894964362 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa2d006bc00 0x7fa2d006e0c0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fa2d4003b80 tx=0x7fa2d400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:39.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.403+0000 7fa2e2ffd640 1 -- 172.21.15.67:0/1894964362 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa2dc07a4c0 con 0x7fa2e414d210 2024-03-24T00:48:39.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.587+0000 7fa2ec33b640 1 -- 172.21.15.67:0/1894964362 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa2e40b17e0 con 0x7fa2e414d210 2024-03-24T00:48:39.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.587+0000 7fa2e2ffd640 1 -- 172.21.15.67:0/1894964362 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v109) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa2dc07e560 con 0x7fa2e414d210 2024-03-24T00:48:39.588 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-03-24T00:48:39.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.589+0000 7fa2ec33b640 1 -- 172.21.15.67:0/1894964362 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa2d006bc00 msgr2=0x7fa2d006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:39.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.589+0000 7fa2ec33b640 1 --2- 172.21.15.67:0/1894964362 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa2d006bc00 0x7fa2d006e0c0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fa2d4003b80 tx=0x7fa2d400b040 comp rx=0 tx=0).stop 2024-03-24T00:48:39.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.589+0000 7fa2ec33b640 1 -- 172.21.15.67:0/1894964362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2e414d210 msgr2=0x7fa2e40b4640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:39.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.589+0000 7fa2ec33b640 1 --2- 172.21.15.67:0/1894964362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2e414d210 0x7fa2e40b4640 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc000c00 tx=0x7fa2e40cc9d0 comp rx=0 tx=0).stop 2024-03-24T00:48:39.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.589+0000 7fa2ec33b640 1 -- 172.21.15.67:0/1894964362 shutdown_connections 2024-03-24T00:48:39.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.589+0000 7fa2ec33b640 1 --2- 172.21.15.67:0/1894964362 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa2d006bc00 0x7fa2d006e0c0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:39.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.589+0000 7fa2ec33b640 1 --2- 172.21.15.67:0/1894964362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2e414d210 0x7fa2e40b4640 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:39.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.589+0000 7fa2ec33b640 1 --2- 172.21.15.67:0/1894964362 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa2e414dd90 0x7fa2e415afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:39.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.589+0000 7fa2ec33b640 1 --2- 172.21.15.67:0/1894964362 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2cc004d40 0x7fa2e40af080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:39.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.589+0000 7fa2ec33b640 1 -- 172.21.15.67:0/1894964362 >> 172.21.15.67:0/1894964362 conn(0x7fa2e408b180 msgr2=0x7fa2e408b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:39.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.589+0000 7fa2ec33b640 1 -- 172.21.15.67:0/1894964362 shutdown_connections 2024-03-24T00:48:39.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.589+0000 7fa2ec33b640 1 -- 172.21.15.67:0/1894964362 wait complete. 2024-03-24T00:48:39.600 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:48:39.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.701+0000 7faf90edc640 1 Processor -- start 2024-03-24T00:48:39.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.701+0000 7faf90edc640 1 -- start start 2024-03-24T00:48:39.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.702+0000 7faf90edc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7faf8c15dfb0 0x7faf8c0cfe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:39.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.702+0000 7faf90edc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faf8c06b750 0x7faf8c0cb120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:39.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.702+0000 7faf90edc640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf8c0d07d0 con 0x7faf8c15e390 2024-03-24T00:48:39.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.702+0000 7faf90edc640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf8c0d0ab0 con 0x7faf8c06b750 2024-03-24T00:48:39.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.702+0000 7faf90edc640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf8c0d0d90 con 0x7faf8c15dfb0 2024-03-24T00:48:39.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.702+0000 7faf8ad9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faf8c06b750 0x7faf8c0cb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:39.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.702+0000 7faf83fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7faf8c15e390 0x7faf8c06b040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45622/0 (socket says 172.21.15.67:45622) 2024-03-24T00:48:39.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.702+0000 7faf83fff640 1 -- 172.21.15.67:0/1178564738 learned_addr learned my addr 172.21.15.67:0/1178564738 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:48:39.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.702+0000 7faf8ad9d640 1 -- 172.21.15.67:0/1178564738 >> v2:172.21.15.72:3300/0 conn(0x7faf8c15dfb0 msgr2=0x7faf8c0cfe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:39.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.702+0000 7faf8a59c640 1 --2- 172.21.15.67:0/1178564738 >> v2:172.21.15.72:3300/0 conn(0x7faf8c15dfb0 0x7faf8c0cfe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:39.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.702+0000 7faf89d9b640 1 -- 172.21.15.67:0/1178564738 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1883313320 0 0) 0x7faf8c0d07d0 con 0x7faf8c15e390 2024-03-24T00:48:39.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.702+0000 7faf8ad9d640 1 --2- 172.21.15.67:0/1178564738 >> v2:172.21.15.72:3300/0 conn(0x7faf8c15dfb0 0x7faf8c0cfe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:39.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.702+0000 7faf8ad9d640 1 -- 172.21.15.67:0/1178564738 >> v1:172.21.15.67:6789/0 conn(0x7faf8c15e390 legacy=0x7faf8c06b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:39.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.702+0000 7faf8ad9d640 1 -- 172.21.15.67:0/1178564738 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf8c0d1070 con 0x7faf8c06b750 2024-03-24T00:48:39.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.702+0000 7faf8a59c640 1 --2- 172.21.15.67:0/1178564738 >> v2:172.21.15.72:3300/0 conn(0x7faf8c15dfb0 0x7faf8c0cfe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:48:39.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.703+0000 7faf8ad9d640 1 --2- 172.21.15.67:0/1178564738 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf8c06b750 0x7faf8c0cb120 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7faf8c0d1b70 tx=0x7faf8402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c24eec810dd9b817 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:39.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.703+0000 7faf89d9b640 1 -- 172.21.15.67:0/1178564738 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf84039070 con 0x7faf8c06b750 2024-03-24T00:48:39.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.703+0000 7faf89d9b640 1 -- 172.21.15.67:0/1178564738 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf8402cc80 con 0x7faf8c06b750 2024-03-24T00:48:39.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.703+0000 7faf89d9b640 1 -- 172.21.15.67:0/1178564738 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf84034660 con 0x7faf8c06b750 2024-03-24T00:48:39.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.703+0000 7faf90edc640 1 -- 172.21.15.67:0/1178564738 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf8c06b750 msgr2=0x7faf8c0cb120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:39.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.703+0000 7faf90edc640 1 --2- 172.21.15.67:0/1178564738 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf8c06b750 0x7faf8c0cb120 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7faf8c0d1b70 tx=0x7faf8402bed0 comp rx=0 tx=0).stop 2024-03-24T00:48:39.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.704+0000 7faf90edc640 1 -- 172.21.15.67:0/1178564738 shutdown_connections 2024-03-24T00:48:39.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.704+0000 7faf90edc640 1 --2- 172.21.15.67:0/1178564738 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf8c06b750 0x7faf8c0cb120 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:39.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.704+0000 7faf90edc640 1 --2- 172.21.15.67:0/1178564738 >> v2:172.21.15.72:3300/0 conn(0x7faf8c15dfb0 0x7faf8c0cfe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:39.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.704+0000 7faf90edc640 1 -- 172.21.15.67:0/1178564738 >> 172.21.15.67:0/1178564738 conn(0x7faf8c099a30 msgr2=0x7faf8c099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:39.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.704+0000 7faf90edc640 1 -- 172.21.15.67:0/1178564738 shutdown_connections 2024-03-24T00:48:39.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.704+0000 7faf90edc640 1 -- 172.21.15.67:0/1178564738 wait complete. 2024-03-24T00:48:39.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.705+0000 7faf90edc640 1 Processor -- start 2024-03-24T00:48:39.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.705+0000 7faf90edc640 1 -- start start 2024-03-24T00:48:39.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.706+0000 7faf90edc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faf8c06b750 0x7faf8c09a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:39.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.706+0000 7faf90edc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf8c15dfb0 0x7faf8c0705a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:39.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.706+0000 7faf90edc640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faf8c15e390 0x7faf8c0c9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:39.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.706+0000 7faf90edc640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf8c0d03c0 con 0x7faf8c15dfb0 2024-03-24T00:48:39.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.706+0000 7faf90edc640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf8c0d1870 con 0x7faf8c06b750 2024-03-24T00:48:39.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.706+0000 7faf90edc640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf8c0d0540 con 0x7faf8c15e390 2024-03-24T00:48:39.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.706+0000 7faf83fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf8c15dfb0 0x7faf8c0705a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:39.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.706+0000 7faf83fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf8c15dfb0 0x7faf8c0705a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36222/0 (socket says 172.21.15.67:36222) 2024-03-24T00:48:39.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.706+0000 7faf83fff640 1 -- 172.21.15.67:0/537118070 learned_addr learned my addr 172.21.15.67:0/537118070 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:39.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.706+0000 7faf8ad9d640 1 --2- 172.21.15.67:0/537118070 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faf8c15e390 0x7faf8c0c9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:39.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.706+0000 7faf8a59c640 1 --2- 172.21.15.67:0/537118070 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faf8c06b750 0x7faf8c09a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:39.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.706+0000 7faf83fff640 1 -- 172.21.15.67:0/537118070 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faf8c15e390 msgr2=0x7faf8c0c9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:39.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.706+0000 7faf83fff640 1 --2- 172.21.15.67:0/537118070 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faf8c15e390 0x7faf8c0c9fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:39.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.706+0000 7faf83fff640 1 -- 172.21.15.67:0/537118070 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faf8c06b750 msgr2=0x7faf8c09a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:39.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.706+0000 7faf83fff640 1 --2- 172.21.15.67:0/537118070 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faf8c06b750 0x7faf8c09a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:39.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.706+0000 7faf83fff640 1 -- 172.21.15.67:0/537118070 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf8c15a900 con 0x7faf8c15dfb0 2024-03-24T00:48:39.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.706+0000 7faf83fff640 1 --2- 172.21.15.67:0/537118070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf8c15dfb0 0x7faf8c0705a0 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7faf7400c4d0 tx=0x7faf7400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:39.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.706+0000 7faf82ffd640 1 -- 172.21.15.67:0/537118070 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf74018070 con 0x7faf8c15dfb0 2024-03-24T00:48:39.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.706+0000 7faf82ffd640 1 -- 172.21.15.67:0/537118070 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf74007500 con 0x7faf8c15dfb0 2024-03-24T00:48:39.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.706+0000 7faf82ffd640 1 -- 172.21.15.67:0/537118070 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf74004a20 con 0x7faf8c15dfb0 2024-03-24T00:48:39.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.706+0000 7faf90edc640 1 -- 172.21.15.67:0/537118070 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf8c09e3b0 con 0x7faf8c15dfb0 2024-03-24T00:48:39.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.706+0000 7faf90edc640 1 -- 172.21.15.67:0/537118070 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf8c09afb0 con 0x7faf8c15dfb0 2024-03-24T00:48:39.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.707+0000 7faf90edc640 1 -- 172.21.15.67:0/537118070 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf8c04f400 con 0x7faf8c15dfb0 2024-03-24T00:48:39.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.708+0000 7faf82ffd640 1 -- 172.21.15.67:0/537118070 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7faf7401b070 con 0x7faf8c15dfb0 2024-03-24T00:48:39.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.708+0000 7faf82ffd640 1 --2- 172.21.15.67:0/537118070 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7faf6c06bbd0 0x7faf6c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:39.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.708+0000 7faf82ffd640 1 -- 172.21.15.67:0/537118070 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6302+0+0 (secure 0 0 0) 0x7faf74089fc0 con 0x7faf8c15dfb0 2024-03-24T00:48:39.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.709+0000 7faf8a59c640 1 --2- 172.21.15.67:0/537118070 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7faf6c06bbd0 0x7faf6c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:39.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.709+0000 7faf8a59c640 1 --2- 172.21.15.67:0/537118070 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7faf6c06bbd0 0x7faf6c06e090 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7faf7c00a990 tx=0x7faf7c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:39.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.713+0000 7faf82ffd640 1 -- 172.21.15.67:0/537118070 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7faf74058860 con 0x7faf8c15dfb0 2024-03-24T00:48:39.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.934+0000 7faf90edc640 1 -- 172.21.15.67:0/537118070 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf8c0b6a50 con 0x7faf8c15dfb0 2024-03-24T00:48:39.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.935+0000 7faf82ffd640 1 -- 172.21.15.67:0/537118070 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+4963 (secure 0 0 0) 0x7faf74058160 con 0x7faf8c15dfb0 2024-03-24T00:48:39.935 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:48:39.936 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":80,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6902,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3672294289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3672294289},{"type":"v1","addr":"172.21.15.67:6835","nonce":3672294289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":6908,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/1286271878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":1286271878},{"type":"v1","addr":"172.21.15.72:6836","nonce":1286271878}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":6911,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/140622129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":140622129},{"type":"v1","addr":"172.21.15.72:6837","nonce":140622129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":9054,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/3799780628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3799780628},{"type":"v1","addr":"172.21.15.67:6833","nonce":3799780628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":79,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:47:16.492485+0000","modified":"2024-03-24T00:48:37.687883+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":109,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-03-24T00:48:39.936 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 80 2024-03-24T00:48:39.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.937+0000 7faf90edc640 1 -- 172.21.15.67:0/537118070 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7faf6c06bbd0 msgr2=0x7faf6c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:39.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.937+0000 7faf90edc640 1 --2- 172.21.15.67:0/537118070 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7faf6c06bbd0 0x7faf6c06e090 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7faf7c00a990 tx=0x7faf7c008040 comp rx=0 tx=0).stop 2024-03-24T00:48:39.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.937+0000 7faf90edc640 1 -- 172.21.15.67:0/537118070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf8c15dfb0 msgr2=0x7faf8c0705a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:39.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.937+0000 7faf90edc640 1 --2- 172.21.15.67:0/537118070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf8c15dfb0 0x7faf8c0705a0 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7faf7400c4d0 tx=0x7faf7400c990 comp rx=0 tx=0).stop 2024-03-24T00:48:39.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.937+0000 7faf90edc640 1 -- 172.21.15.67:0/537118070 shutdown_connections 2024-03-24T00:48:39.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.937+0000 7faf90edc640 1 --2- 172.21.15.67:0/537118070 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7faf6c06bbd0 0x7faf6c06e090 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:39.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.937+0000 7faf90edc640 1 --2- 172.21.15.67:0/537118070 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faf8c15e390 0x7faf8c0c9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:39.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.937+0000 7faf90edc640 1 --2- 172.21.15.67:0/537118070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf8c15dfb0 0x7faf8c0705a0 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:39.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.937+0000 7faf90edc640 1 --2- 172.21.15.67:0/537118070 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faf8c06b750 0x7faf8c09a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:39.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.937+0000 7faf90edc640 1 -- 172.21.15.67:0/537118070 >> 172.21.15.67:0/537118070 conn(0x7faf8c099a30 msgr2=0x7faf8c06b040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:39.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.937+0000 7faf90edc640 1 -- 172.21.15.67:0/537118070 shutdown_connections 2024-03-24T00:48:39.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:39.937+0000 7faf90edc640 1 -- 172.21.15.67:0/537118070 wait complete. 2024-03-24T00:48:39.950 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:48:40.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.050+0000 7f96d9959640 1 Processor -- start 2024-03-24T00:48:40.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.050+0000 7f96d9959640 1 -- start start 2024-03-24T00:48:40.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.051+0000 7f96d9959640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f96d40d1550 0x7f96d406cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:40.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.051+0000 7f96d9959640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f96d409a480 0x7f96d40b2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:40.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.051+0000 7f96d9959640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96d40d1930 con 0x7f96d40b2780 2024-03-24T00:48:40.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.051+0000 7f96d9959640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96d406d420 con 0x7f96d409a480 2024-03-24T00:48:40.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.051+0000 7f96d9959640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96d406ca50 con 0x7f96d40d1550 2024-03-24T00:48:40.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.051+0000 7f96d37fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f96d409a480 0x7f96d40b2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:40.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.051+0000 7f96cbfff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f96d40b2780 0x7f96d4099d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45638/0 (socket says 172.21.15.67:45638) 2024-03-24T00:48:40.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.051+0000 7f96cbfff640 1 -- 172.21.15.67:0/813314361 learned_addr learned my addr 172.21.15.67:0/813314361 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:48:40.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.051+0000 7f96d2ffd640 1 --2- 172.21.15.67:0/813314361 >> v2:172.21.15.72:3300/0 conn(0x7f96d40d1550 0x7f96d406cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:40.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.051+0000 7f96d37fe640 1 -- 172.21.15.67:0/813314361 >> v2:172.21.15.72:3300/0 conn(0x7f96d40d1550 msgr2=0x7f96d406cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:40.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.051+0000 7f96d37fe640 1 --2- 172.21.15.67:0/813314361 >> v2:172.21.15.72:3300/0 conn(0x7f96d40d1550 0x7f96d406cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:40.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.051+0000 7f96d37fe640 1 -- 172.21.15.67:0/813314361 >> v1:172.21.15.67:6789/0 conn(0x7f96d40b2780 legacy=0x7f96d4099d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:40.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.051+0000 7f96d27fc640 1 -- 172.21.15.67:0/813314361 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1204369441 0 0) 0x7f96d40d1930 con 0x7f96d40b2780 2024-03-24T00:48:40.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.051+0000 7f96d37fe640 1 -- 172.21.15.67:0/813314361 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96d40b1f60 con 0x7f96d409a480 2024-03-24T00:48:40.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.052+0000 7f96d37fe640 1 --2- 172.21.15.67:0/813314361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96d409a480 0x7f96d40b2c30 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f96cc009d90 tx=0x7f96cc02be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9b486102ecc5003 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:40.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.052+0000 7f96d27fc640 1 -- 172.21.15.67:0/813314361 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96cc039070 con 0x7f96d409a480 2024-03-24T00:48:40.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.052+0000 7f96d27fc640 1 -- 172.21.15.67:0/813314361 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96cc02cbd0 con 0x7f96d409a480 2024-03-24T00:48:40.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.052+0000 7f96d27fc640 1 -- 172.21.15.67:0/813314361 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96cc034440 con 0x7f96d409a480 2024-03-24T00:48:40.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.053+0000 7f96d9959640 1 -- 172.21.15.67:0/813314361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96d409a480 msgr2=0x7f96d40b2c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:40.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.053+0000 7f96d9959640 1 --2- 172.21.15.67:0/813314361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96d409a480 0x7f96d40b2c30 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f96cc009d90 tx=0x7f96cc02be20 comp rx=0 tx=0).stop 2024-03-24T00:48:40.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.053+0000 7f96d9959640 1 -- 172.21.15.67:0/813314361 shutdown_connections 2024-03-24T00:48:40.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.053+0000 7f96d9959640 1 --2- 172.21.15.67:0/813314361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96d409a480 0x7f96d40b2c30 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:40.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.053+0000 7f96d9959640 1 --2- 172.21.15.67:0/813314361 >> v2:172.21.15.72:3300/0 conn(0x7f96d40d1550 0x7f96d406cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:40.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.053+0000 7f96d9959640 1 -- 172.21.15.67:0/813314361 >> 172.21.15.67:0/813314361 conn(0x7f96d40b36e0 msgr2=0x7f96d40b3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:40.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.053+0000 7f96d9959640 1 -- 172.21.15.67:0/813314361 shutdown_connections 2024-03-24T00:48:40.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.053+0000 7f96d9959640 1 -- 172.21.15.67:0/813314361 wait complete. 2024-03-24T00:48:40.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.054+0000 7f96d9959640 1 Processor -- start 2024-03-24T00:48:40.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.054+0000 7f96d9959640 1 -- start start 2024-03-24T00:48:40.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.055+0000 7f96d9959640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96d409a480 0x7f96d41e8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:40.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.055+0000 7f96d9959640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f96d40b2780 0x7f96d414fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:40.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.055+0000 7f96d9959640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f96d40d1550 0x7f96d414cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:40.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.055+0000 7f96d9959640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96d406c400 con 0x7f96d409a480 2024-03-24T00:48:40.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.055+0000 7f96d9959640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96d40b2520 con 0x7f96d40b2780 2024-03-24T00:48:40.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.055+0000 7f96d9959640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96d406c6e0 con 0x7f96d40d1550 2024-03-24T00:48:40.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.055+0000 7f96d2ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96d409a480 0x7f96d41e8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:40.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.055+0000 7f96d2ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96d409a480 0x7f96d41e8bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36238/0 (socket says 172.21.15.67:36238) 2024-03-24T00:48:40.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.055+0000 7f96d2ffd640 1 -- 172.21.15.67:0/3124959657 learned_addr learned my addr 172.21.15.67:0/3124959657 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:40.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.055+0000 7f96d37fe640 1 --2- 172.21.15.67:0/3124959657 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f96d40d1550 0x7f96d414cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:40.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.055+0000 7f96d2ffd640 1 -- 172.21.15.67:0/3124959657 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f96d40d1550 msgr2=0x7f96d414cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:40.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.055+0000 7f96cbfff640 1 --2- 172.21.15.67:0/3124959657 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f96d40b2780 0x7f96d414fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:40.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.055+0000 7f96d2ffd640 1 --2- 172.21.15.67:0/3124959657 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f96d40d1550 0x7f96d414cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:40.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.055+0000 7f96d2ffd640 1 -- 172.21.15.67:0/3124959657 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f96d40b2780 msgr2=0x7f96d414fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:40.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.055+0000 7f96d2ffd640 1 --2- 172.21.15.67:0/3124959657 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f96d40b2780 0x7f96d414fec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:40.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.055+0000 7f96d2ffd640 1 -- 172.21.15.67:0/3124959657 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96d41f7750 con 0x7f96d409a480 2024-03-24T00:48:40.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.055+0000 7f96d2ffd640 1 --2- 172.21.15.67:0/3124959657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96d409a480 0x7f96d41e8bc0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f96c400c7f0 tx=0x7f96c400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:40.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.055+0000 7f96d0ff9640 1 -- 172.21.15.67:0/3124959657 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c4015070 con 0x7f96d409a480 2024-03-24T00:48:40.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.055+0000 7f96d0ff9640 1 -- 172.21.15.67:0/3124959657 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96c4004240 con 0x7f96d409a480 2024-03-24T00:48:40.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.055+0000 7f96d9959640 1 -- 172.21.15.67:0/3124959657 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96d41f7a30 con 0x7f96d409a480 2024-03-24T00:48:40.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.055+0000 7f96d0ff9640 1 -- 172.21.15.67:0/3124959657 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c4007620 con 0x7f96d409a480 2024-03-24T00:48:40.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.055+0000 7f96d9959640 1 -- 172.21.15.67:0/3124959657 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96d41f7ff0 con 0x7f96d409a480 2024-03-24T00:48:40.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.056+0000 7f96d9959640 1 -- 172.21.15.67:0/3124959657 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96d404f400 con 0x7f96d409a480 2024-03-24T00:48:40.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.057+0000 7f96d0ff9640 1 -- 172.21.15.67:0/3124959657 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f96c40079d0 con 0x7f96d409a480 2024-03-24T00:48:40.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.057+0000 7f96d0ff9640 1 --2- 172.21.15.67:0/3124959657 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f96b406bbd0 0x7f96b406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:40.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.057+0000 7f96d0ff9640 1 -- 172.21.15.67:0/3124959657 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6302+0+0 (secure 0 0 0) 0x7f96c408b110 con 0x7f96d409a480 2024-03-24T00:48:40.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.057+0000 7f96cbfff640 1 --2- 172.21.15.67:0/3124959657 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f96b406bbd0 0x7f96b406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:40.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.058+0000 7f96cbfff640 1 --2- 172.21.15.67:0/3124959657 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f96b406bbd0 0x7f96b406e090 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f96bc007fd0 tx=0x7f96bc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:40.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.061+0000 7f96d0ff9640 1 -- 172.21.15.67:0/3124959657 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f96c40599b0 con 0x7f96d409a480 2024-03-24T00:48:40.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.247+0000 7f96d9959640 1 -- 172.21.15.67:0/3124959657 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f96d40b1310 con 0x7f96d409a480 2024-03-24T00:48:40.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.248+0000 7f96d0ff9640 1 -- 172.21.15.67:0/3124959657 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) v1 ==== 74+0+17133 (secure 0 0 0) 0x7f96c40592b0 con 0x7f96d409a480 2024-03-24T00:48:40.249 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:48:40.249 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":109,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:48:37.687686+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:47:15.111478+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-03-24T00:47:15.455832+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"16.1b","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6833/3197261495":"2024-03-25T00:48:37.687657+0000","172.21.15.67:6832/3197261495":"2024-03-25T00:48:37.687657+0000","172.21.15.67:6833/1826974843":"2024-03-25T00:48:01.322612+0000","172.21.15.67:6832/1826974843":"2024-03-25T00:48:01.322612+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:48:40.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.250+0000 7f96d9959640 1 -- 172.21.15.67:0/3124959657 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f96b406bbd0 msgr2=0x7f96b406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:40.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.250+0000 7f96d9959640 1 --2- 172.21.15.67:0/3124959657 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f96b406bbd0 0x7f96b406e090 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f96bc007fd0 tx=0x7f96bc009040 comp rx=0 tx=0).stop 2024-03-24T00:48:40.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.250+0000 7f96d9959640 1 -- 172.21.15.67:0/3124959657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96d409a480 msgr2=0x7f96d41e8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:40.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.250+0000 7f96d9959640 1 --2- 172.21.15.67:0/3124959657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96d409a480 0x7f96d41e8bc0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f96c400c7f0 tx=0x7f96c400ccb0 comp rx=0 tx=0).stop 2024-03-24T00:48:40.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.250+0000 7f96d9959640 1 -- 172.21.15.67:0/3124959657 shutdown_connections 2024-03-24T00:48:40.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.250+0000 7f96d9959640 1 --2- 172.21.15.67:0/3124959657 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f96b406bbd0 0x7f96b406e090 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:40.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.250+0000 7f96d9959640 1 --2- 172.21.15.67:0/3124959657 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f96d40d1550 0x7f96d414cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:40.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.250+0000 7f96d9959640 1 --2- 172.21.15.67:0/3124959657 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f96d40b2780 0x7f96d414fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:40.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.250+0000 7f96d9959640 1 --2- 172.21.15.67:0/3124959657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96d409a480 0x7f96d41e8bc0 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:40.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.250+0000 7f96d9959640 1 -- 172.21.15.67:0/3124959657 >> 172.21.15.67:0/3124959657 conn(0x7f96d40b36e0 msgr2=0x7f96d40b1530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:40.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.250+0000 7f96d9959640 1 -- 172.21.15.67:0/3124959657 shutdown_connections 2024-03-24T00:48:40.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.250+0000 7f96d9959640 1 -- 172.21.15.67:0/3124959657 wait complete. 2024-03-24T00:48:40.262 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-24T00:48:40.262 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:48:40.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.364+0000 7fa055a27640 1 Processor -- start 2024-03-24T00:48:40.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.364+0000 7fa055a27640 1 -- start start 2024-03-24T00:48:40.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.364+0000 7fa055a27640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa05006cee0 0x7fa050099d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:40.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.364+0000 7fa055a27640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa05009a2b0 0x7fa0500cfd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:40.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.364+0000 7fa055a27640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0500d1930 con 0x7fa0500d1550 2024-03-24T00:48:40.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.364+0000 7fa055a27640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa05006ca50 con 0x7fa05009a2b0 2024-03-24T00:48:40.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.364+0000 7fa055a27640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0500b1f60 con 0x7fa05006cee0 2024-03-24T00:48:40.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.365+0000 7fa04effd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa0500d1550 0x7fa0500b2780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45648/0 (socket says 172.21.15.67:45648) 2024-03-24T00:48:40.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.365+0000 7fa04effd640 1 -- 172.21.15.67:0/97779880 learned_addr learned my addr 172.21.15.67:0/97779880 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:48:40.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.365+0000 7fa04e7fc640 1 --2- 172.21.15.67:0/97779880 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa05006cee0 0x7fa050099d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:40.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.365+0000 7fa04dffb640 1 -- 172.21.15.67:0/97779880 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2318175847 0 0) 0x7fa0500d1930 con 0x7fa0500d1550 2024-03-24T00:48:40.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.365+0000 7fa04dffb640 1 -- 172.21.15.67:0/97779880 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa038003520 con 0x7fa0500d1550 2024-03-24T00:48:40.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.365+0000 7fa04f7fe640 1 --2- 172.21.15.67:0/97779880 >> v2:172.21.15.72:3300/0 conn(0x7fa05009a2b0 0x7fa0500cfd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:40.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.365+0000 7fa04dffb640 1 -- 172.21.15.67:0/97779880 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3197239 0 0) 0x7fa038003520 con 0x7fa0500d1550 2024-03-24T00:48:40.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.365+0000 7fa04dffb640 1 -- 172.21.15.67:0/97779880 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa05006cee0 msgr2=0x7fa050099d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:40.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.365+0000 7fa04dffb640 1 --2- 172.21.15.67:0/97779880 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa05006cee0 0x7fa050099d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:40.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.365+0000 7fa04dffb640 1 -- 172.21.15.67:0/97779880 >> v2:172.21.15.72:3300/0 conn(0x7fa05009a2b0 msgr2=0x7fa0500cfd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:40.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.365+0000 7fa04dffb640 1 --2- 172.21.15.67:0/97779880 >> v2:172.21.15.72:3300/0 conn(0x7fa05009a2b0 0x7fa0500cfd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:40.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.365+0000 7fa04dffb640 1 -- 172.21.15.67:0/97779880 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0500b2240 con 0x7fa0500d1550 2024-03-24T00:48:40.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.365+0000 7fa04dffb640 1 -- 172.21.15.67:0/97779880 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7fa040001670 con 0x7fa0500d1550 2024-03-24T00:48:40.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.365+0000 7fa04dffb640 1 -- 172.21.15.67:0/97779880 >> v1:172.21.15.67:6789/0 conn(0x7fa0500d1550 legacy=0x7fa0500b2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:40.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.365+0000 7fa04dffb640 1 --2- 172.21.15.67:0/97779880 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa05006cee0 0x7fa0380039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:40.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.365+0000 7fa04dffb640 1 --2- 172.21.15.67:0/97779880 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa038003f30 0x7fa038004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:40.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.365+0000 7fa04dffb640 1 --2- 172.21.15.67:0/97779880 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa038004d40 0x7fa0380051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:40.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.365+0000 7fa04dffb640 1 -- 172.21.15.67:0/97779880 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa05006ca50 con 0x7fa038004d40 2024-03-24T00:48:40.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.365+0000 7fa04dffb640 1 -- 172.21.15.67:0/97779880 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0500b1f60 con 0x7fa038003f30 2024-03-24T00:48:40.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.365+0000 7fa04dffb640 1 -- 172.21.15.67:0/97779880 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa03800b9c0 con 0x7fa05006cee0 2024-03-24T00:48:40.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.365+0000 7fa04f7fe640 1 --2- 172.21.15.67:0/97779880 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa038004d40 0x7fa0380051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:40.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.365+0000 7fa04effd640 1 --2- 172.21.15.67:0/97779880 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa05006cee0 0x7fa0380039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:40.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.365+0000 7fa04e7fc640 1 --2- 172.21.15.67:0/97779880 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa038003f30 0x7fa038004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:40.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.366+0000 7fa04f7fe640 1 -- 172.21.15.67:0/97779880 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa05006cee0 msgr2=0x7fa0380039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:40.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.366+0000 7fa04f7fe640 1 --2- 172.21.15.67:0/97779880 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa05006cee0 0x7fa0380039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:40.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.366+0000 7fa04f7fe640 1 -- 172.21.15.67:0/97779880 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa038003f30 msgr2=0x7fa038004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:40.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.366+0000 7fa04f7fe640 1 --2- 172.21.15.67:0/97779880 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa038003f30 0x7fa038004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:40.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.366+0000 7fa04f7fe640 1 -- 172.21.15.67:0/97779880 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa038003520 con 0x7fa038004d40 2024-03-24T00:48:40.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.366+0000 7fa04f7fe640 1 --2- 172.21.15.67:0/97779880 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa038004d40 0x7fa0380051c0 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7fa048009fc0 tx=0x7fa038005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a54726a32f594ad7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:40.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.367+0000 7fa04dffb640 1 -- 172.21.15.67:0/97779880 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa048039070 con 0x7fa038004d40 2024-03-24T00:48:40.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.367+0000 7fa04dffb640 1 -- 172.21.15.67:0/97779880 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa04802cc40 con 0x7fa038004d40 2024-03-24T00:48:40.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.367+0000 7fa04dffb640 1 -- 172.21.15.67:0/97779880 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa048009c10 con 0x7fa038004d40 2024-03-24T00:48:40.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.367+0000 7fa055a27640 1 -- 172.21.15.67:0/97779880 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa038004d40 msgr2=0x7fa0380051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:40.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.367+0000 7fa055a27640 1 --2- 172.21.15.67:0/97779880 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa038004d40 0x7fa0380051c0 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7fa048009fc0 tx=0x7fa038005830 comp rx=0 tx=0).stop 2024-03-24T00:48:40.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.367+0000 7fa04f7fe640 1 -- 172.21.15.67:0/97779880 reap_dead start 2024-03-24T00:48:40.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.367+0000 7fa055a27640 1 -- 172.21.15.67:0/97779880 shutdown_connections 2024-03-24T00:48:40.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.367+0000 7fa055a27640 1 -- 172.21.15.67:0/97779880 >> 172.21.15.67:0/97779880 conn(0x7fa0500b36e0 msgr2=0x7fa0500b3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:40.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.367+0000 7fa055a27640 1 -- 172.21.15.67:0/97779880 shutdown_connections 2024-03-24T00:48:40.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.367+0000 7fa055a27640 1 -- 172.21.15.67:0/97779880 wait complete. 2024-03-24T00:48:40.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.368+0000 7fa055a27640 1 Processor -- start 2024-03-24T00:48:40.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.368+0000 7fa055a27640 1 -- start start 2024-03-24T00:48:40.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.369+0000 7fa055a27640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0500b6550 0x7fa0500b4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:40.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.369+0000 7fa055a27640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa0500b6930 0x7fa0500b4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:40.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.369+0000 7fa055a27640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa050154ec0 0x7fa050155340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:40.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.369+0000 7fa055a27640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa05006c6e0 con 0x7fa0500b6550 2024-03-24T00:48:40.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.369+0000 7fa055a27640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa05006c400 con 0x7fa050154ec0 2024-03-24T00:48:40.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.369+0000 7fa055a27640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa05006d360 con 0x7fa0500b6930 2024-03-24T00:48:40.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.369+0000 7fa04effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0500b6550 0x7fa0500b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:40.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.369+0000 7fa04effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0500b6550 0x7fa0500b4730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36246/0 (socket says 172.21.15.67:36246) 2024-03-24T00:48:40.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.369+0000 7fa04effd640 1 -- 172.21.15.67:0/4098950353 learned_addr learned my addr 172.21.15.67:0/4098950353 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:40.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.369+0000 7fa04f7fe640 1 --2- 172.21.15.67:0/4098950353 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa050154ec0 0x7fa050155340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:40.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.369+0000 7fa04e7fc640 1 --2- 172.21.15.67:0/4098950353 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa0500b6930 0x7fa0500b4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:40.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.370+0000 7fa04effd640 1 -- 172.21.15.67:0/4098950353 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa0500b6930 msgr2=0x7fa0500b4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:40.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.370+0000 7fa04effd640 1 --2- 172.21.15.67:0/4098950353 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa0500b6930 0x7fa0500b4c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:40.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.370+0000 7fa04effd640 1 -- 172.21.15.67:0/4098950353 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa050154ec0 msgr2=0x7fa050155340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:40.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.370+0000 7fa04effd640 1 --2- 172.21.15.67:0/4098950353 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa050154ec0 0x7fa050155340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:40.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.370+0000 7fa04effd640 1 -- 172.21.15.67:0/4098950353 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0500b5310 con 0x7fa0500b6550 2024-03-24T00:48:40.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.370+0000 7fa04effd640 1 --2- 172.21.15.67:0/4098950353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0500b6550 0x7fa0500b4730 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7fa0400017e0 tx=0x7fa040007660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:40.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.370+0000 7fa037fff640 1 -- 172.21.15.67:0/4098950353 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa04000f8e0 con 0x7fa0500b6550 2024-03-24T00:48:40.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.370+0000 7fa037fff640 1 -- 172.21.15.67:0/4098950353 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa040018070 con 0x7fa0500b6550 2024-03-24T00:48:40.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.370+0000 7fa055a27640 1 -- 172.21.15.67:0/4098950353 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0500b55f0 con 0x7fa0500b6550 2024-03-24T00:48:40.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.370+0000 7fa037fff640 1 -- 172.21.15.67:0/4098950353 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa04000fd00 con 0x7fa0500b6550 2024-03-24T00:48:40.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.370+0000 7fa055a27640 1 -- 172.21.15.67:0/4098950353 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0500c8e60 con 0x7fa0500b6550 2024-03-24T00:48:40.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.371+0000 7fa055a27640 1 -- 172.21.15.67:0/4098950353 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0500b5b10 con 0x7fa0500b6550 2024-03-24T00:48:40.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.371+0000 7fa037fff640 1 -- 172.21.15.67:0/4098950353 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa04001d460 con 0x7fa0500b6550 2024-03-24T00:48:40.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.372+0000 7fa037fff640 1 --2- 172.21.15.67:0/4098950353 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa03c06bbd0 0x7fa03c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:40.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.372+0000 7fa037fff640 1 -- 172.21.15.67:0/4098950353 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6302+0+0 (secure 0 0 0) 0x7fa04008ae00 con 0x7fa0500b6550 2024-03-24T00:48:40.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.372+0000 7fa04e7fc640 1 --2- 172.21.15.67:0/4098950353 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa03c06bbd0 0x7fa03c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:40.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.372+0000 7fa04e7fc640 1 --2- 172.21.15.67:0/4098950353 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa03c06bbd0 0x7fa03c06e090 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fa0500c8c00 tx=0x7fa044009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:40.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.376+0000 7fa037fff640 1 -- 172.21.15.67:0/4098950353 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa0500b5b10 con 0x7fa0500b6550 2024-03-24T00:48:40.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.596+0000 7fa055a27640 1 -- 172.21.15.67:0/4098950353 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0500c9060 con 0x7fa0500b6550 2024-03-24T00:48:40.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.596+0000 7fa037fff640 1 -- 172.21.15.67:0/4098950353 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+4963 (secure 0 0 0) 0x7fa0500c9060 con 0x7fa0500b6550 2024-03-24T00:48:40.597 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:48:40.597 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":80,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6902,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3672294289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3672294289},{"type":"v1","addr":"172.21.15.67:6835","nonce":3672294289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":6908,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/1286271878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":1286271878},{"type":"v1","addr":"172.21.15.72:6836","nonce":1286271878}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":6911,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/140622129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":140622129},{"type":"v1","addr":"172.21.15.72:6837","nonce":140622129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":9054,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/3799780628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3799780628},{"type":"v1","addr":"172.21.15.67:6833","nonce":3799780628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":79,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:47:16.492485+0000","modified":"2024-03-24T00:48:37.687883+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":109,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-03-24T00:48:40.597 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 80 2024-03-24T00:48:40.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.598+0000 7fa055a27640 1 -- 172.21.15.67:0/4098950353 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa03c06bbd0 msgr2=0x7fa03c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:40.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.598+0000 7fa055a27640 1 --2- 172.21.15.67:0/4098950353 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa03c06bbd0 0x7fa03c06e090 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fa0500c8c00 tx=0x7fa044009190 comp rx=0 tx=0).stop 2024-03-24T00:48:40.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.598+0000 7fa055a27640 1 -- 172.21.15.67:0/4098950353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0500b6550 msgr2=0x7fa0500b4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:40.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.598+0000 7fa055a27640 1 --2- 172.21.15.67:0/4098950353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0500b6550 0x7fa0500b4730 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7fa0400017e0 tx=0x7fa040007660 comp rx=0 tx=0).stop 2024-03-24T00:48:40.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.598+0000 7fa055a27640 1 -- 172.21.15.67:0/4098950353 shutdown_connections 2024-03-24T00:48:40.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.598+0000 7fa055a27640 1 --2- 172.21.15.67:0/4098950353 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa03c06bbd0 0x7fa03c06e090 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:40.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.598+0000 7fa055a27640 1 --2- 172.21.15.67:0/4098950353 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa050154ec0 0x7fa050155340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:40.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.598+0000 7fa055a27640 1 --2- 172.21.15.67:0/4098950353 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa0500b6930 0x7fa0500b4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:40.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.598+0000 7fa055a27640 1 --2- 172.21.15.67:0/4098950353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0500b6550 0x7fa0500b4730 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:40.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.598+0000 7fa055a27640 1 -- 172.21.15.67:0/4098950353 >> 172.21.15.67:0/4098950353 conn(0x7fa0500b36e0 msgr2=0x7fa0500703a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:40.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.598+0000 7fa055a27640 1 -- 172.21.15.67:0/4098950353 shutdown_connections 2024-03-24T00:48:40.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.598+0000 7fa055a27640 1 -- 172.21.15.67:0/4098950353 wait complete. 2024-03-24T00:48:40.610 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:48:40.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.710+0000 7f760b6dc640 1 Processor -- start 2024-03-24T00:48:40.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.710+0000 7f760b6dc640 1 -- start start 2024-03-24T00:48:40.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.711+0000 7f760b6dc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f760415e310 0x7f7604162610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:40.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.711+0000 7f760b6dc640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76040ce920 con 0x7f76040d0d50 2024-03-24T00:48:40.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.711+0000 7f760b6dc640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76040cef20 con 0x7f760415df30 2024-03-24T00:48:40.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.711+0000 7f760b6dc640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7604162b50 con 0x7f760415e310 2024-03-24T00:48:40.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.711+0000 7f7609c79640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f76040d0d50 0x7f76040d2490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45650/0 (socket says 172.21.15.67:45650) 2024-03-24T00:48:40.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.711+0000 7f7609c79640 1 -- 172.21.15.67:0/863018751 learned_addr learned my addr 172.21.15.67:0/863018751 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:48:40.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.711+0000 7f7608c77640 1 --2- 172.21.15.67:0/863018751 >> v2:172.21.15.72:3300/0 conn(0x7f760415e310 0x7f7604162610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:40.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.712+0000 7f7603fff640 1 -- 172.21.15.67:0/863018751 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2880791066 0 0) 0x7f76040ce920 con 0x7f76040d0d50 2024-03-24T00:48:40.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.712+0000 7f7603fff640 1 -- 172.21.15.67:0/863018751 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75ec003520 con 0x7f76040d0d50 2024-03-24T00:48:40.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.712+0000 7f7608c77640 1 -- 172.21.15.67:0/863018751 >> v1:172.21.15.72:6789/0 conn(0x7f760415df30 legacy=0x7f76040d0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:40.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.712+0000 7f7608c77640 1 -- 172.21.15.67:0/863018751 >> v1:172.21.15.67:6789/0 conn(0x7f76040d0d50 legacy=0x7f76040d2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:40.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.712+0000 7f7603fff640 1 -- 172.21.15.67:0/863018751 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 132551274 0 0) 0x7f76040cef20 con 0x7f760415df30 2024-03-24T00:48:40.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.712+0000 7f7608c77640 1 -- 172.21.15.67:0/863018751 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76040cf200 con 0x7f760415e310 2024-03-24T00:48:40.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.712+0000 7f7608c77640 1 --2- 172.21.15.67:0/863018751 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f760415e310 0x7f7604162610 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f75f80097a0 tx=0x7f75f802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca7183969b296d26 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:40.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.713+0000 7f7603fff640 1 -- 172.21.15.67:0/863018751 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f8035020 con 0x7f760415e310 2024-03-24T00:48:40.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.713+0000 7f7603fff640 1 -- 172.21.15.67:0/863018751 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75f8031830 con 0x7f760415e310 2024-03-24T00:48:40.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.713+0000 7f7603fff640 1 -- 172.21.15.67:0/863018751 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f8031be0 con 0x7f760415e310 2024-03-24T00:48:40.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.713+0000 7f760b6dc640 1 -- 172.21.15.67:0/863018751 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f760415e310 msgr2=0x7f7604162610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:40.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.713+0000 7f760b6dc640 1 --2- 172.21.15.67:0/863018751 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f760415e310 0x7f7604162610 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f75f80097a0 tx=0x7f75f802b8d0 comp rx=0 tx=0).stop 2024-03-24T00:48:40.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.713+0000 7f760b6dc640 1 -- 172.21.15.67:0/863018751 shutdown_connections 2024-03-24T00:48:40.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.713+0000 7f760b6dc640 1 --2- 172.21.15.67:0/863018751 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f760415e310 0x7f7604162610 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:40.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.713+0000 7f760b6dc640 1 -- 172.21.15.67:0/863018751 >> 172.21.15.67:0/863018751 conn(0x7f760406c880 msgr2=0x7f760406cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:40.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.713+0000 7f760b6dc640 1 -- 172.21.15.67:0/863018751 shutdown_connections 2024-03-24T00:48:40.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.713+0000 7f760b6dc640 1 -- 172.21.15.67:0/863018751 wait complete. 2024-03-24T00:48:40.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.714+0000 7f760b6dc640 1 Processor -- start 2024-03-24T00:48:40.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.714+0000 7f760b6dc640 1 -- start start 2024-03-24T00:48:40.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.715+0000 7f760b6dc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76040d0d50 0x7f76040cbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:40.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.715+0000 7f760b6dc640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f760415df30 0x7f760415c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:40.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.715+0000 7f760b6dc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f760415cb50 0x7f76040aa720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:40.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.715+0000 7f760b6dc640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76040ce580 con 0x7f76040d0d50 2024-03-24T00:48:40.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.715+0000 7f760b6dc640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76040b32f0 con 0x7f760415cb50 2024-03-24T00:48:40.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.715+0000 7f760b6dc640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76040ce700 con 0x7f760415df30 2024-03-24T00:48:40.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.715+0000 7f7609478640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76040d0d50 0x7f76040cbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:40.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.715+0000 7f7609478640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76040d0d50 0x7f76040cbaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36258/0 (socket says 172.21.15.67:36258) 2024-03-24T00:48:40.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.715+0000 7f7609478640 1 -- 172.21.15.67:0/1929513006 learned_addr learned my addr 172.21.15.67:0/1929513006 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:40.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.715+0000 7f7608c77640 1 --2- 172.21.15.67:0/1929513006 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f760415df30 0x7f760415c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:40.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.715+0000 7f7609478640 1 -- 172.21.15.67:0/1929513006 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f760415df30 msgr2=0x7f760415c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:40.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.715+0000 7f7609c79640 1 --2- 172.21.15.67:0/1929513006 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f760415cb50 0x7f76040aa720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:40.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.715+0000 7f7609478640 1 --2- 172.21.15.67:0/1929513006 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f760415df30 0x7f760415c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:40.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.715+0000 7f7609478640 1 -- 172.21.15.67:0/1929513006 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f760415cb50 msgr2=0x7f76040aa720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:40.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.715+0000 7f7609478640 1 --2- 172.21.15.67:0/1929513006 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f760415cb50 0x7f76040aa720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:40.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.716+0000 7f7609478640 1 -- 172.21.15.67:0/1929513006 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76041f4000 con 0x7f76040d0d50 2024-03-24T00:48:40.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.716+0000 7f7608c77640 1 --2- 172.21.15.67:0/1929513006 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f760415df30 0x7f760415c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:48:40.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.716+0000 7f7609c79640 1 --2- 172.21.15.67:0/1929513006 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f760415cb50 0x7f76040aa720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:48:40.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.716+0000 7f7609478640 1 --2- 172.21.15.67:0/1929513006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76040d0d50 0x7f76040cbaa0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f75f400c890 tx=0x7f75f400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:40.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.716+0000 7f7601ffb640 1 -- 172.21.15.67:0/1929513006 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f4017070 con 0x7f76040d0d50 2024-03-24T00:48:40.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.716+0000 7f7601ffb640 1 -- 172.21.15.67:0/1929513006 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75f4003dc0 con 0x7f76040d0d50 2024-03-24T00:48:40.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.716+0000 7f760b6dc640 1 -- 172.21.15.67:0/1929513006 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f760415d3b0 con 0x7f76040d0d50 2024-03-24T00:48:40.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.716+0000 7f7601ffb640 1 -- 172.21.15.67:0/1929513006 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f40052b0 con 0x7f76040d0d50 2024-03-24T00:48:40.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.716+0000 7f760b6dc640 1 -- 172.21.15.67:0/1929513006 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76040cc480 con 0x7f76040d0d50 2024-03-24T00:48:40.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.717+0000 7f760b6dc640 1 -- 172.21.15.67:0/1929513006 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f760404f400 con 0x7f76040d0d50 2024-03-24T00:48:40.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.717+0000 7f7601ffb640 1 -- 172.21.15.67:0/1929513006 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f75f401a070 con 0x7f76040d0d50 2024-03-24T00:48:40.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.718+0000 7f7601ffb640 1 --2- 172.21.15.67:0/1929513006 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f75f006bbd0 0x7f75f006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:40.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.718+0000 7f7601ffb640 1 -- 172.21.15.67:0/1929513006 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6302+0+0 (secure 0 0 0) 0x7f75f408a430 con 0x7f76040d0d50 2024-03-24T00:48:40.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.718+0000 7f7608c77640 1 --2- 172.21.15.67:0/1929513006 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f75f006bbd0 0x7f75f006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:40.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.719+0000 7f7608c77640 1 --2- 172.21.15.67:0/1929513006 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f75f006bbd0 0x7f75f006e090 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f75f8036000 tx=0x7f75f803c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:40.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.722+0000 7f7601ffb640 1 -- 172.21.15.67:0/1929513006 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f75f4058cd0 con 0x7f76040d0d50 2024-03-24T00:48:40.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.942+0000 7f760b6dc640 1 -- 172.21.15.67:0/1929513006 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7604161e30 con 0x7f76040d0d50 2024-03-24T00:48:40.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.943+0000 7f7601ffb640 1 -- 172.21.15.67:0/1929513006 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+4963 (secure 0 0 0) 0x7f75f40585d0 con 0x7f76040d0d50 2024-03-24T00:48:40.944 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:48:40.944 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":80,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6902,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3672294289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3672294289},{"type":"v1","addr":"172.21.15.67:6835","nonce":3672294289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":6908,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/1286271878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":1286271878},{"type":"v1","addr":"172.21.15.72:6836","nonce":1286271878}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":6911,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/140622129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":140622129},{"type":"v1","addr":"172.21.15.72:6837","nonce":140622129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":9054,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/3799780628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3799780628},{"type":"v1","addr":"172.21.15.67:6833","nonce":3799780628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":79,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:47:16.492485+0000","modified":"2024-03-24T00:48:37.687883+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":109,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-03-24T00:48:40.944 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 80 2024-03-24T00:48:40.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.945+0000 7f760b6dc640 1 -- 172.21.15.67:0/1929513006 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f75f006bbd0 msgr2=0x7f75f006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:40.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.945+0000 7f760b6dc640 1 --2- 172.21.15.67:0/1929513006 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f75f006bbd0 0x7f75f006e090 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f75f8036000 tx=0x7f75f803c040 comp rx=0 tx=0).stop 2024-03-24T00:48:40.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.945+0000 7f760b6dc640 1 -- 172.21.15.67:0/1929513006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76040d0d50 msgr2=0x7f76040cbaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:40.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.945+0000 7f760b6dc640 1 --2- 172.21.15.67:0/1929513006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76040d0d50 0x7f76040cbaa0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f75f400c890 tx=0x7f75f400cd50 comp rx=0 tx=0).stop 2024-03-24T00:48:40.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.945+0000 7f760b6dc640 1 -- 172.21.15.67:0/1929513006 shutdown_connections 2024-03-24T00:48:40.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.945+0000 7f760b6dc640 1 --2- 172.21.15.67:0/1929513006 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f75f006bbd0 0x7f75f006e090 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:40.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.945+0000 7f760b6dc640 1 --2- 172.21.15.67:0/1929513006 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f760415cb50 0x7f76040aa720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:40.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.945+0000 7f760b6dc640 1 --2- 172.21.15.67:0/1929513006 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f760415df30 0x7f760415c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:40.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.945+0000 7f760b6dc640 1 --2- 172.21.15.67:0/1929513006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76040d0d50 0x7f76040cbaa0 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:40.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.945+0000 7f760b6dc640 1 -- 172.21.15.67:0/1929513006 >> 172.21.15.67:0/1929513006 conn(0x7f760406c880 msgr2=0x7f760406cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:40.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.945+0000 7f760b6dc640 1 -- 172.21.15.67:0/1929513006 shutdown_connections 2024-03-24T00:48:40.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:40.945+0000 7f760b6dc640 1 -- 172.21.15.67:0/1929513006 wait complete. 2024-03-24T00:48:40.956 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:48:41.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.056+0000 7f8ddd57e640 1 Processor -- start 2024-03-24T00:48:41.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.057+0000 7f8ddd57e640 1 -- start start 2024-03-24T00:48:41.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.057+0000 7f8ddd57e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8dd815e1b0 0x7f8dd80cfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:41.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.057+0000 7f8ddd57e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8dd80b43a0 0x7f8dd8099d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:41.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.057+0000 7f8ddd57e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dd80d0fc0 con 0x7f8dd809a240 2024-03-24T00:48:41.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.057+0000 7f8ddd57e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dd815e590 con 0x7f8dd815e1b0 2024-03-24T00:48:41.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.057+0000 7f8ddd57e640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd809a690 con 0x7f8dd80b43a0 2024-03-24T00:48:41.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.058+0000 7f8dd67fc640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8dd80b43a0 0x7f8dd8099d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:41.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.058+0000 7f8dd6ffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8dd815e1b0 0x7f8dd80cfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:41.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.058+0000 7f8dd67fc640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8dd80b43a0 0x7f8dd8099d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.67:53440/0 (socket says 172.21.15.67:53440) 2024-03-24T00:48:41.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.058+0000 7f8dd67fc640 1 -- 172.21.15.67:0/1374346047 learned_addr learned my addr 172.21.15.67:0/1374346047 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:41.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.058+0000 7f8dd67fc640 1 -- 172.21.15.67:0/1374346047 >> v2:172.21.15.72:3300/0 conn(0x7f8dd815e1b0 msgr2=0x7f8dd80cfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:41.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.058+0000 7f8dd67fc640 1 --2- 172.21.15.67:0/1374346047 >> v2:172.21.15.72:3300/0 conn(0x7f8dd815e1b0 0x7f8dd80cfe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:41.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.058+0000 7f8dd67fc640 1 -- 172.21.15.67:0/1374346047 >> v1:172.21.15.72:6789/0 conn(0x7f8dd809a240 legacy=0x7f8dd806c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:41.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.058+0000 7f8dd67fc640 1 -- 172.21.15.67:0/1374346047 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd80d1260 con 0x7f8dd80b43a0 2024-03-24T00:48:41.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.058+0000 7f8dd6ffd640 1 --2- 172.21.15.67:0/1374346047 >> v2:172.21.15.72:3300/0 conn(0x7f8dd815e1b0 0x7f8dd80cfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:48:41.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.059+0000 7f8dd67fc640 1 --2- 172.21.15.67:0/1374346047 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8dd80b43a0 0x7f8dd8099d00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc009870 tx=0x7f8dcc02b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a74518cab5b168c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:41.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.059+0000 7f8dd5ffb640 1 -- 172.21.15.67:0/1374346047 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dcc0317c0 con 0x7f8dd80b43a0 2024-03-24T00:48:41.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.059+0000 7f8dd5ffb640 1 -- 172.21.15.67:0/1374346047 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dcc037070 con 0x7f8dd80b43a0 2024-03-24T00:48:41.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.059+0000 7f8dd5ffb640 1 -- 172.21.15.67:0/1374346047 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dcc031b80 con 0x7f8dd80b43a0 2024-03-24T00:48:41.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.060+0000 7f8ddd57e640 1 -- 172.21.15.67:0/1374346047 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8dd80b43a0 msgr2=0x7f8dd8099d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:41.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.060+0000 7f8ddd57e640 1 --2- 172.21.15.67:0/1374346047 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8dd80b43a0 0x7f8dd8099d00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc009870 tx=0x7f8dcc02b860 comp rx=0 tx=0).stop 2024-03-24T00:48:41.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.060+0000 7f8ddd57e640 1 -- 172.21.15.67:0/1374346047 shutdown_connections 2024-03-24T00:48:41.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.060+0000 7f8ddd57e640 1 --2- 172.21.15.67:0/1374346047 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8dd80b43a0 0x7f8dd8099d00 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:41.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.060+0000 7f8ddd57e640 1 --2- 172.21.15.67:0/1374346047 >> v2:172.21.15.72:3300/0 conn(0x7f8dd815e1b0 0x7f8dd80cfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:41.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.060+0000 7f8ddd57e640 1 -- 172.21.15.67:0/1374346047 >> 172.21.15.67:0/1374346047 conn(0x7f8dd80b36d0 msgr2=0x7f8dd80b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:41.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.060+0000 7f8ddd57e640 1 -- 172.21.15.67:0/1374346047 shutdown_connections 2024-03-24T00:48:41.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.060+0000 7f8ddd57e640 1 -- 172.21.15.67:0/1374346047 wait complete. 2024-03-24T00:48:41.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.061+0000 7f8ddd57e640 1 Processor -- start 2024-03-24T00:48:41.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.061+0000 7f8ddd57e640 1 -- start start 2024-03-24T00:48:41.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.062+0000 7f8ddd57e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8dd809a240 0x7f8dd8147eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:41.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.062+0000 7f8ddd57e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8dd80b43a0 0x7f8dd81e7b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:41.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.062+0000 7f8ddd57e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8dd815e1b0 0x7f8dd81e8050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:41.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.062+0000 7f8ddd57e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd80cc360 con 0x7f8dd815e1b0 2024-03-24T00:48:41.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.062+0000 7f8ddd57e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd80d1820 con 0x7f8dd809a240 2024-03-24T00:48:41.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.062+0000 7f8ddd57e640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd809a9a0 con 0x7f8dd80b43a0 2024-03-24T00:48:41.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.062+0000 7f8dd77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8dd815e1b0 0x7f8dd81e8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:41.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.062+0000 7f8dd77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8dd815e1b0 0x7f8dd81e8050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36274/0 (socket says 172.21.15.67:36274) 2024-03-24T00:48:41.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.062+0000 7f8dd77fe640 1 -- 172.21.15.67:0/155543892 learned_addr learned my addr 172.21.15.67:0/155543892 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:41.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.062+0000 7f8dd67fc640 1 --2- 172.21.15.67:0/155543892 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8dd80b43a0 0x7f8dd81e7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:41.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.062+0000 7f8dd6ffd640 1 --2- 172.21.15.67:0/155543892 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8dd809a240 0x7f8dd8147eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:41.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.062+0000 7f8dd77fe640 1 -- 172.21.15.67:0/155543892 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8dd80b43a0 msgr2=0x7f8dd81e7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:41.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.062+0000 7f8dd77fe640 1 --2- 172.21.15.67:0/155543892 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8dd80b43a0 0x7f8dd81e7b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:41.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.062+0000 7f8dd77fe640 1 -- 172.21.15.67:0/155543892 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8dd809a240 msgr2=0x7f8dd8147eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:41.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.062+0000 7f8dd77fe640 1 --2- 172.21.15.67:0/155543892 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8dd809a240 0x7f8dd8147eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:41.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.062+0000 7f8dd77fe640 1 -- 172.21.15.67:0/155543892 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd815c650 con 0x7f8dd815e1b0 2024-03-24T00:48:41.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.063+0000 7f8dd77fe640 1 --2- 172.21.15.67:0/155543892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8dd815e1b0 0x7f8dd81e8050 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f8dd000c730 tx=0x7f8dd000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:41.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.063+0000 7f8dbf7fe640 1 -- 172.21.15.67:0/155543892 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd0003b90 con 0x7f8dd815e1b0 2024-03-24T00:48:41.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.063+0000 7f8dbf7fe640 1 -- 172.21.15.67:0/155543892 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dd0003d30 con 0x7f8dd815e1b0 2024-03-24T00:48:41.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.063+0000 7f8ddd57e640 1 -- 172.21.15.67:0/155543892 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dd815c930 con 0x7f8dd815e1b0 2024-03-24T00:48:41.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.063+0000 7f8dbf7fe640 1 -- 172.21.15.67:0/155543892 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd0005310 con 0x7f8dd815e1b0 2024-03-24T00:48:41.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.063+0000 7f8ddd57e640 1 -- 172.21.15.67:0/155543892 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dd815cef0 con 0x7f8dd815e1b0 2024-03-24T00:48:41.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.064+0000 7f8ddd57e640 1 -- 172.21.15.67:0/155543892 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dd804f400 con 0x7f8dd815e1b0 2024-03-24T00:48:41.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.064+0000 7f8dbf7fe640 1 -- 172.21.15.67:0/155543892 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8dd00056c0 con 0x7f8dd815e1b0 2024-03-24T00:48:41.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.064+0000 7f8dbf7fe640 1 --2- 172.21.15.67:0/155543892 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8dc406bc00 0x7f8dc406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:41.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.065+0000 7f8dbf7fe640 1 -- 172.21.15.67:0/155543892 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6302+0+0 (secure 0 0 0) 0x7f8dd008ae20 con 0x7f8dd815e1b0 2024-03-24T00:48:41.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.065+0000 7f8dd6ffd640 1 --2- 172.21.15.67:0/155543892 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8dc406bc00 0x7f8dc406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:41.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.065+0000 7f8dd6ffd640 1 --2- 172.21.15.67:0/155543892 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8dc406bc00 0x7f8dc406e0c0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f8dc8009a10 tx=0x7f8dc8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:41.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.069+0000 7f8dbf7fe640 1 -- 172.21.15.67:0/155543892 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8dd0059630 con 0x7f8dd815e1b0 2024-03-24T00:48:41.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.254+0000 7f8ddd57e640 1 -- 172.21.15.67:0/155543892 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8dd80b7a40 con 0x7f8dd815e1b0 2024-03-24T00:48:41.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.255+0000 7f8dbf7fe640 1 -- 172.21.15.67:0/155543892 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) v1 ==== 74+0+17133 (secure 0 0 0) 0x7f8dd0058f30 con 0x7f8dd815e1b0 2024-03-24T00:48:41.256 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:48:41.256 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":109,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:48:37.687686+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:47:15.111478+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-03-24T00:47:15.455832+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"16.1b","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6833/3197261495":"2024-03-25T00:48:37.687657+0000","172.21.15.67:6832/3197261495":"2024-03-25T00:48:37.687657+0000","172.21.15.67:6833/1826974843":"2024-03-25T00:48:01.322612+0000","172.21.15.67:6832/1826974843":"2024-03-25T00:48:01.322612+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:48:41.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.257+0000 7f8ddd57e640 1 -- 172.21.15.67:0/155543892 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8dc406bc00 msgr2=0x7f8dc406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:41.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.257+0000 7f8ddd57e640 1 --2- 172.21.15.67:0/155543892 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8dc406bc00 0x7f8dc406e0c0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f8dc8009a10 tx=0x7f8dc8008040 comp rx=0 tx=0).stop 2024-03-24T00:48:41.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.257+0000 7f8ddd57e640 1 -- 172.21.15.67:0/155543892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8dd815e1b0 msgr2=0x7f8dd81e8050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:41.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.257+0000 7f8ddd57e640 1 --2- 172.21.15.67:0/155543892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8dd815e1b0 0x7f8dd81e8050 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f8dd000c730 tx=0x7f8dd000cbf0 comp rx=0 tx=0).stop 2024-03-24T00:48:41.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.257+0000 7f8ddd57e640 1 -- 172.21.15.67:0/155543892 shutdown_connections 2024-03-24T00:48:41.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.257+0000 7f8ddd57e640 1 --2- 172.21.15.67:0/155543892 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8dc406bc00 0x7f8dc406e0c0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:41.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.257+0000 7f8ddd57e640 1 --2- 172.21.15.67:0/155543892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8dd815e1b0 0x7f8dd81e8050 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:41.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.257+0000 7f8ddd57e640 1 --2- 172.21.15.67:0/155543892 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8dd80b43a0 0x7f8dd81e7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:41.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.257+0000 7f8ddd57e640 1 --2- 172.21.15.67:0/155543892 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8dd809a240 0x7f8dd8147eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:41.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.257+0000 7f8ddd57e640 1 -- 172.21.15.67:0/155543892 >> 172.21.15.67:0/155543892 conn(0x7f8dd80b36d0 msgr2=0x7f8dd80cda10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:41.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.257+0000 7f8ddd57e640 1 -- 172.21.15.67:0/155543892 shutdown_connections 2024-03-24T00:48:41.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.257+0000 7f8ddd57e640 1 -- 172.21.15.67:0/155543892 wait complete. 2024-03-24T00:48:41.269 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-24T00:48:41.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.371+0000 7f0421bf4640 1 Processor -- start 2024-03-24T00:48:41.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.371+0000 7f0421bf4640 1 -- start start 2024-03-24T00:48:41.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.372+0000 7f0421bf4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f041c099d00 0x7f041c0cc7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:41.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.372+0000 7f0421bf4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f041c0cf8e0 0x7f041c0cfd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:41.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.372+0000 7f0421bf4640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f041c0cce30 con 0x7f041c15ce90 2024-03-24T00:48:41.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.372+0000 7f0421bf4640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f041c0cda40 con 0x7f041c0cf8e0 2024-03-24T00:48:41.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.372+0000 7f0421bf4640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f041c0cd110 con 0x7f041c099d00 2024-03-24T00:48:41.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.372+0000 7f041bfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f041c0cf8e0 0x7f041c0cfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:41.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.372+0000 7f0413fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f041c15ce90 0x7f041c09a0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45658/0 (socket says 172.21.15.67:45658) 2024-03-24T00:48:41.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.372+0000 7f041bfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f041c0cf8e0 0x7f041c0cfd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36290/0 (socket says 172.21.15.67:36290) 2024-03-24T00:48:41.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.372+0000 7f0413fff640 1 -- 172.21.15.67:0/3532499540 learned_addr learned my addr 172.21.15.67:0/3532499540 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:48:41.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.372+0000 7f041b7fe640 1 --2- 172.21.15.67:0/3532499540 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f041c099d00 0x7f041c0cc7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:41.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.372+0000 7f041bfff640 1 -- 172.21.15.67:0/3532499540 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f041c099d00 msgr2=0x7f041c0cc7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:41.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.372+0000 7f041bfff640 1 --2- 172.21.15.67:0/3532499540 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f041c099d00 0x7f041c0cc7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:41.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.372+0000 7f041bfff640 1 -- 172.21.15.67:0/3532499540 >> v1:172.21.15.67:6789/0 conn(0x7f041c15ce90 legacy=0x7f041c09a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:41.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.372+0000 7f041bfff640 1 -- 172.21.15.67:0/3532499540 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f041c0cd3f0 con 0x7f041c0cf8e0 2024-03-24T00:48:41.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.372+0000 7f041affd640 1 -- 172.21.15.67:0/3532499540 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3061947547 0 0) 0x7f041c0cce30 con 0x7f041c15ce90 2024-03-24T00:48:41.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.373+0000 7f041bfff640 1 --2- 172.21.15.67:0/3532499540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f041c0cf8e0 0x7f041c0cfd70 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f0414009d90 tx=0x7f041402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f15affcf5538ea30 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:41.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.373+0000 7f041affd640 1 -- 172.21.15.67:0/3532499540 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0414039070 con 0x7f041c0cf8e0 2024-03-24T00:48:41.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.373+0000 7f041affd640 1 -- 172.21.15.67:0/3532499540 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f041402cc50 con 0x7f041c0cf8e0 2024-03-24T00:48:41.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.373+0000 7f041affd640 1 -- 172.21.15.67:0/3532499540 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0414034440 con 0x7f041c0cf8e0 2024-03-24T00:48:41.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.374+0000 7f0421bf4640 1 -- 172.21.15.67:0/3532499540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f041c0cf8e0 msgr2=0x7f041c0cfd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:41.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.374+0000 7f0421bf4640 1 --2- 172.21.15.67:0/3532499540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f041c0cf8e0 0x7f041c0cfd70 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f0414009d90 tx=0x7f041402bea0 comp rx=0 tx=0).stop 2024-03-24T00:48:41.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.374+0000 7f0421bf4640 1 -- 172.21.15.67:0/3532499540 shutdown_connections 2024-03-24T00:48:41.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.374+0000 7f0421bf4640 1 --2- 172.21.15.67:0/3532499540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f041c0cf8e0 0x7f041c0cfd70 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:41.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.374+0000 7f0421bf4640 1 --2- 172.21.15.67:0/3532499540 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f041c099d00 0x7f041c0cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:41.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.374+0000 7f0421bf4640 1 -- 172.21.15.67:0/3532499540 >> 172.21.15.67:0/3532499540 conn(0x7f041c0b3650 msgr2=0x7f041c0b3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:41.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.374+0000 7f0421bf4640 1 -- 172.21.15.67:0/3532499540 shutdown_connections 2024-03-24T00:48:41.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.374+0000 7f0421bf4640 1 -- 172.21.15.67:0/3532499540 wait complete. 2024-03-24T00:48:41.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.375+0000 7f0421bf4640 1 Processor -- start 2024-03-24T00:48:41.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.375+0000 7f0421bf4640 1 -- start start 2024-03-24T00:48:41.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.376+0000 7f0421bf4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f041c099d00 0x7f041c1562f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:41.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.376+0000 7f0421bf4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f041c0cf8e0 0x7f041c0b5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:41.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.376+0000 7f0421bf4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f041c15ce90 0x7f041c157a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:41.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.376+0000 7f0421bf4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f041c159cc0 con 0x7f041c099d00 2024-03-24T00:48:41.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.376+0000 7f0421bf4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f041c15d310 con 0x7f041c0cf8e0 2024-03-24T00:48:41.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.376+0000 7f0421bf4640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f041c159e40 con 0x7f041c15ce90 2024-03-24T00:48:41.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.376+0000 7f041b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f041c099d00 0x7f041c1562f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:41.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.376+0000 7f041b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f041c099d00 0x7f041c1562f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36294/0 (socket says 172.21.15.67:36294) 2024-03-24T00:48:41.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.376+0000 7f041b7fe640 1 -- 172.21.15.67:0/1115517786 learned_addr learned my addr 172.21.15.67:0/1115517786 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:41.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.376+0000 7f041bfff640 1 --2- 172.21.15.67:0/1115517786 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f041c15ce90 0x7f041c157a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:41.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.376+0000 7f0413fff640 1 --2- 172.21.15.67:0/1115517786 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f041c0cf8e0 0x7f041c0b5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:41.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.376+0000 7f041b7fe640 1 -- 172.21.15.67:0/1115517786 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f041c15ce90 msgr2=0x7f041c157a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:41.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.376+0000 7f041b7fe640 1 --2- 172.21.15.67:0/1115517786 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f041c15ce90 0x7f041c157a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:41.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.376+0000 7f041b7fe640 1 -- 172.21.15.67:0/1115517786 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f041c0cf8e0 msgr2=0x7f041c0b5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:41.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.376+0000 7f041b7fe640 1 --2- 172.21.15.67:0/1115517786 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f041c0cf8e0 0x7f041c0b5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:41.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.376+0000 7f041b7fe640 1 -- 172.21.15.67:0/1115517786 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f041c0c7200 con 0x7f041c099d00 2024-03-24T00:48:41.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.376+0000 7f041bfff640 1 --2- 172.21.15.67:0/1115517786 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f041c15ce90 0x7f041c157a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:48:41.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.376+0000 7f041b7fe640 1 --2- 172.21.15.67:0/1115517786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f041c099d00 0x7f041c1562f0 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f040c00d5f0 tx=0x7f040c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:41.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.376+0000 7f0418ff9640 1 -- 172.21.15.67:0/1115517786 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f040c0125f0 con 0x7f041c099d00 2024-03-24T00:48:41.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.376+0000 7f0418ff9640 1 -- 172.21.15.67:0/1115517786 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f040c019070 con 0x7f041c099d00 2024-03-24T00:48:41.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.376+0000 7f0421bf4640 1 -- 172.21.15.67:0/1115517786 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f041c0c74e0 con 0x7f041c099d00 2024-03-24T00:48:41.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.376+0000 7f0418ff9640 1 -- 172.21.15.67:0/1115517786 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f040c0129d0 con 0x7f041c099d00 2024-03-24T00:48:41.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.376+0000 7f0421bf4640 1 -- 172.21.15.67:0/1115517786 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f041c0c7aa0 con 0x7f041c099d00 2024-03-24T00:48:41.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.377+0000 7f0421bf4640 1 -- 172.21.15.67:0/1115517786 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f041c04f400 con 0x7f041c099d00 2024-03-24T00:48:41.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.378+0000 7f0418ff9640 1 -- 172.21.15.67:0/1115517786 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f040c012b70 con 0x7f041c099d00 2024-03-24T00:48:41.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.378+0000 7f0418ff9640 1 --2- 172.21.15.67:0/1115517786 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f03fc06bbd0 0x7f03fc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:41.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.378+0000 7f0418ff9640 1 -- 172.21.15.67:0/1115517786 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6302+0+0 (secure 0 0 0) 0x7f040c08b010 con 0x7f041c099d00 2024-03-24T00:48:41.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.378+0000 7f0413fff640 1 --2- 172.21.15.67:0/1115517786 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f03fc06bbd0 0x7f03fc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:41.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.379+0000 7f0413fff640 1 --2- 172.21.15.67:0/1115517786 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f03fc06bbd0 0x7f03fc06e090 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f041c15b030 tx=0x7f0404007b50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:41.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.383+0000 7f0418ff9640 1 -- 172.21.15.67:0/1115517786 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f040c059940 con 0x7f041c099d00 2024-03-24T00:48:41.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:41.604+0000 7f0421bf4640 1 -- 172.21.15.67:0/1115517786 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f041c0d1170 con 0x7f041c099d00 2024-03-24T00:48:42.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.594+0000 7f0418ff9640 1 -- 172.21.15.67:0/1115517786 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v81) v1 ==== 161+0+0 (secure 0 0 0) 0x7f040c059240 con 0x7f041c099d00 2024-03-24T00:48:42.595 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-24T00:48:42.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.597+0000 7f0421bf4640 1 -- 172.21.15.67:0/1115517786 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f03fc06bbd0 msgr2=0x7f03fc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:42.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.597+0000 7f0421bf4640 1 --2- 172.21.15.67:0/1115517786 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f03fc06bbd0 0x7f03fc06e090 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f041c15b030 tx=0x7f0404007b50 comp rx=0 tx=0).stop 2024-03-24T00:48:42.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.597+0000 7f0421bf4640 1 -- 172.21.15.67:0/1115517786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f041c099d00 msgr2=0x7f041c1562f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:42.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.597+0000 7f0421bf4640 1 --2- 172.21.15.67:0/1115517786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f041c099d00 0x7f041c1562f0 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f040c00d5f0 tx=0x7f040c00dab0 comp rx=0 tx=0).stop 2024-03-24T00:48:42.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.597+0000 7f0421bf4640 1 -- 172.21.15.67:0/1115517786 shutdown_connections 2024-03-24T00:48:42.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.597+0000 7f0421bf4640 1 --2- 172.21.15.67:0/1115517786 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f03fc06bbd0 0x7f03fc06e090 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:42.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.597+0000 7f0421bf4640 1 --2- 172.21.15.67:0/1115517786 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f041c15ce90 0x7f041c157a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:42.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.597+0000 7f0421bf4640 1 --2- 172.21.15.67:0/1115517786 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f041c0cf8e0 0x7f041c0b5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:42.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.597+0000 7f0421bf4640 1 --2- 172.21.15.67:0/1115517786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f041c099d00 0x7f041c1562f0 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:42.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.597+0000 7f0421bf4640 1 -- 172.21.15.67:0/1115517786 >> 172.21.15.67:0/1115517786 conn(0x7f041c0b3650 msgr2=0x7f041c0cfcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:42.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.597+0000 7f0421bf4640 1 -- 172.21.15.67:0/1115517786 shutdown_connections 2024-03-24T00:48:42.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.597+0000 7f0421bf4640 1 -- 172.21.15.67:0/1115517786 wait complete. 2024-03-24T00:48:42.611 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-24T00:48:42.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.721+0000 7f74b8f52640 1 Processor -- start 2024-03-24T00:48:42.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.721+0000 7f74b8f52640 1 -- start start 2024-03-24T00:48:42.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.722+0000 7f74b8f52640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f74b40d0d50 0x7f74b40ce610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:42.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.722+0000 7f74b8f52640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b40cf570 con 0x7f74b415e130 2024-03-24T00:48:42.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.722+0000 7f74b8f52640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b40cecf0 con 0x7f74b40d22b0 2024-03-24T00:48:42.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.722+0000 7f74b8f52640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74b40ceb50 con 0x7f74b40d0d50 2024-03-24T00:48:42.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.722+0000 7f74abfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f74b40d0d50 0x7f74b40ce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:42.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.722+0000 7f74b259c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f74b415e130 0x7f74b40d0640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45662/0 (socket says 172.21.15.67:45662) 2024-03-24T00:48:42.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.722+0000 7f74b259c640 1 -- 172.21.15.67:0/2015224154 learned_addr learned my addr 172.21.15.67:0/2015224154 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:48:42.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.722+0000 7f74abfff640 1 -- 172.21.15.67:0/2015224154 >> v1:172.21.15.72:6789/0 conn(0x7f74b40d22b0 legacy=0x7f74b40d2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:42.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.722+0000 7f74abfff640 1 -- 172.21.15.67:0/2015224154 >> v1:172.21.15.67:6789/0 conn(0x7f74b415e130 legacy=0x7f74b40d0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:42.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.722+0000 7f74b1d9b640 1 -- 172.21.15.67:0/2015224154 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2668720130 0 0) 0x7f74b40cf570 con 0x7f74b415e130 2024-03-24T00:48:42.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.722+0000 7f74abfff640 1 -- 172.21.15.67:0/2015224154 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b4162a60 con 0x7f74b40d0d50 2024-03-24T00:48:42.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.723+0000 7f74abfff640 1 --2- 172.21.15.67:0/2015224154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74b40d0d50 0x7f74b40ce610 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f749c0097a0 tx=0x7f749c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a18c6f170f73948c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:42.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.723+0000 7f74b1d9b640 1 -- 172.21.15.67:0/2015224154 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f749c035020 con 0x7f74b40d0d50 2024-03-24T00:48:42.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.723+0000 7f74b1d9b640 1 -- 172.21.15.67:0/2015224154 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f749c031830 con 0x7f74b40d0d50 2024-03-24T00:48:42.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.723+0000 7f74b1d9b640 1 -- 172.21.15.67:0/2015224154 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f749c031c00 con 0x7f74b40d0d50 2024-03-24T00:48:42.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.724+0000 7f74b8f52640 1 -- 172.21.15.67:0/2015224154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74b40d0d50 msgr2=0x7f74b40ce610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:42.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.724+0000 7f74b8f52640 1 --2- 172.21.15.67:0/2015224154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74b40d0d50 0x7f74b40ce610 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f749c0097a0 tx=0x7f749c02b8d0 comp rx=0 tx=0).stop 2024-03-24T00:48:42.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.724+0000 7f74b8f52640 1 -- 172.21.15.67:0/2015224154 shutdown_connections 2024-03-24T00:48:42.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.724+0000 7f74b8f52640 1 --2- 172.21.15.67:0/2015224154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74b40d0d50 0x7f74b40ce610 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:42.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.724+0000 7f74b8f52640 1 -- 172.21.15.67:0/2015224154 >> 172.21.15.67:0/2015224154 conn(0x7f74b406c8a0 msgr2=0x7f74b406ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:42.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.724+0000 7f74b8f52640 1 -- 172.21.15.67:0/2015224154 shutdown_connections 2024-03-24T00:48:42.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.724+0000 7f74b8f52640 1 -- 172.21.15.67:0/2015224154 wait complete. 2024-03-24T00:48:42.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.725+0000 7f74b8f52640 1 Processor -- start 2024-03-24T00:48:42.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.725+0000 7f74b8f52640 1 -- start start 2024-03-24T00:48:42.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.726+0000 7f74b8f52640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74b40d0d50 0x7f74b414d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:42.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.726+0000 7f74b8f52640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f74b40d22b0 0x7f74b40cc950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:42.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.726+0000 7f74b8f52640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f74b415e130 0x7f74b40cadf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:42.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.726+0000 7f74b8f52640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b415e510 con 0x7f74b40d0d50 2024-03-24T00:48:42.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.726+0000 7f74b8f52640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b40cf290 con 0x7f74b415e130 2024-03-24T00:48:42.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.726+0000 7f74b8f52640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74b40b2e10 con 0x7f74b40d22b0 2024-03-24T00:48:42.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.726+0000 7f74b259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74b40d0d50 0x7f74b414d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:42.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.726+0000 7f74b259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74b40d0d50 0x7f74b414d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36312/0 (socket says 172.21.15.67:36312) 2024-03-24T00:48:42.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.726+0000 7f74b259c640 1 -- 172.21.15.67:0/1336443578 learned_addr learned my addr 172.21.15.67:0/1336443578 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:42.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.726+0000 7f74b259c640 1 -- 172.21.15.67:0/1336443578 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f74b40d22b0 msgr2=0x7f74b40cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:42.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.726+0000 7f74b259c640 1 --2- 172.21.15.67:0/1336443578 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f74b40d22b0 0x7f74b40cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:42.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.726+0000 7f74b259c640 1 -- 172.21.15.67:0/1336443578 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f74b415e130 msgr2=0x7f74b40cadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:42.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.726+0000 7f74b259c640 1 --2- 172.21.15.67:0/1336443578 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f74b415e130 0x7f74b40cadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:42.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.726+0000 7f74b259c640 1 -- 172.21.15.67:0/1336443578 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b40cb8b0 con 0x7f74b40d0d50 2024-03-24T00:48:42.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.726+0000 7f74b259c640 1 --2- 172.21.15.67:0/1336443578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74b40d0d50 0x7f74b414d2c0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f74a400c430 tx=0x7f74a400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:42.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.726+0000 7f74aaffd640 1 -- 172.21.15.67:0/1336443578 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a4017070 con 0x7f74b40d0d50 2024-03-24T00:48:42.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.726+0000 7f74aaffd640 1 -- 172.21.15.67:0/1336443578 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74a4005ce0 con 0x7f74b40d0d50 2024-03-24T00:48:42.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.727+0000 7f74b8f52640 1 -- 172.21.15.67:0/1336443578 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74b40cbb90 con 0x7f74b40d0d50 2024-03-24T00:48:42.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.727+0000 7f74aaffd640 1 -- 172.21.15.67:0/1336443578 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a4007310 con 0x7f74b40d0d50 2024-03-24T00:48:42.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.727+0000 7f74b8f52640 1 -- 172.21.15.67:0/1336443578 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74b40cc150 con 0x7f74b40d0d50 2024-03-24T00:48:42.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.728+0000 7f74b8f52640 1 -- 172.21.15.67:0/1336443578 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74b404f400 con 0x7f74b40d0d50 2024-03-24T00:48:42.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.728+0000 7f74aaffd640 1 -- 172.21.15.67:0/1336443578 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f74a401a070 con 0x7f74b40d0d50 2024-03-24T00:48:42.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.728+0000 7f74aaffd640 1 --2- 172.21.15.67:0/1336443578 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f749406bbd0 0x7f749406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:42.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.729+0000 7f74aaffd640 1 -- 172.21.15.67:0/1336443578 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6302+0+0 (secure 0 0 0) 0x7f74a408a0f0 con 0x7f74b40d0d50 2024-03-24T00:48:42.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.729+0000 7f74abfff640 1 --2- 172.21.15.67:0/1336443578 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f749406bbd0 0x7f749406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:42.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.729+0000 7f74abfff640 1 --2- 172.21.15.67:0/1336443578 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f749406bbd0 0x7f749406e090 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f749c036000 tx=0x7f749c030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:42.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.733+0000 7f74aaffd640 1 -- 172.21.15.67:0/1336443578 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f74a4058990 con 0x7f74b40d0d50 2024-03-24T00:48:42.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:42.954+0000 7f74b8f52640 1 -- 172.21.15.67:0/1336443578 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f74b40d19f0 con 0x7f74b40d0d50 2024-03-24T00:48:43.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.600+0000 7f74aaffd640 1 -- 172.21.15.67:0/1336443578 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v82) v1 ==== 104+0+0 (secure 0 0 0) 0x7f74a4058290 con 0x7f74b40d0d50 2024-03-24T00:48:43.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.603+0000 7f74b8f52640 1 -- 172.21.15.67:0/1336443578 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f749406bbd0 msgr2=0x7f749406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:43.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.603+0000 7f74b8f52640 1 --2- 172.21.15.67:0/1336443578 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f749406bbd0 0x7f749406e090 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f749c036000 tx=0x7f749c030000 comp rx=0 tx=0).stop 2024-03-24T00:48:43.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.603+0000 7f74b8f52640 1 -- 172.21.15.67:0/1336443578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74b40d0d50 msgr2=0x7f74b414d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:43.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.603+0000 7f74b8f52640 1 --2- 172.21.15.67:0/1336443578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74b40d0d50 0x7f74b414d2c0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f74a400c430 tx=0x7f74a400c8f0 comp rx=0 tx=0).stop 2024-03-24T00:48:43.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.604+0000 7f74b8f52640 1 -- 172.21.15.67:0/1336443578 shutdown_connections 2024-03-24T00:48:43.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.604+0000 7f74b8f52640 1 --2- 172.21.15.67:0/1336443578 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f749406bbd0 0x7f749406e090 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:43.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.604+0000 7f74b8f52640 1 --2- 172.21.15.67:0/1336443578 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f74b415e130 0x7f74b40cadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:43.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.604+0000 7f74b8f52640 1 --2- 172.21.15.67:0/1336443578 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f74b40d22b0 0x7f74b40cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:43.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.604+0000 7f74b8f52640 1 --2- 172.21.15.67:0/1336443578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74b40d0d50 0x7f74b414d2c0 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:43.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.604+0000 7f74b8f52640 1 -- 172.21.15.67:0/1336443578 >> 172.21.15.67:0/1336443578 conn(0x7f74b406c8a0 msgr2=0x7f74b40b3760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:43.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.604+0000 7f74b8f52640 1 -- 172.21.15.67:0/1336443578 shutdown_connections 2024-03-24T00:48:43.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.604+0000 7f74b8f52640 1 -- 172.21.15.67:0/1336443578 wait complete. 2024-03-24T00:48:43.618 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-24T00:48:43.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.729+0000 7fdc45fcd640 1 Processor -- start 2024-03-24T00:48:43.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.729+0000 7fdc45fcd640 1 -- start start 2024-03-24T00:48:43.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.730+0000 7fdc45fcd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdc40099d70 0x7fdc4009adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:43.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.730+0000 7fdc45fcd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fdc4015cd40 0x7fdc400cf7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:43.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.730+0000 7fdc45fcd640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc4015d1b0 con 0x7fdc400cd6d0 2024-03-24T00:48:43.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.730+0000 7fdc45fcd640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc400cc6e0 con 0x7fdc40099d70 2024-03-24T00:48:43.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.730+0000 7fdc45fcd640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc4015d470 con 0x7fdc4015cd40 2024-03-24T00:48:43.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.730+0000 7fdc3f7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdc40099d70 0x7fdc4009adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:43.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.730+0000 7fdc3effd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fdc400cd6d0 0x7fdc4009a150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48560/0 (socket says 172.21.15.67:48560) 2024-03-24T00:48:43.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.730+0000 7fdc3f7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdc40099d70 0x7fdc4009adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34670/0 (socket says 172.21.15.67:34670) 2024-03-24T00:48:43.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.730+0000 7fdc3effd640 1 -- 172.21.15.67:0/690799233 learned_addr learned my addr 172.21.15.67:0/690799233 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:48:43.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.730+0000 7fdc3ffff640 1 --2- 172.21.15.67:0/690799233 >> v2:172.21.15.72:3300/0 conn(0x7fdc4015cd40 0x7fdc400cf7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:43.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.730+0000 7fdc3f7fe640 1 -- 172.21.15.67:0/690799233 >> v2:172.21.15.72:3300/0 conn(0x7fdc4015cd40 msgr2=0x7fdc400cf7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:43.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.730+0000 7fdc3f7fe640 1 --2- 172.21.15.67:0/690799233 >> v2:172.21.15.72:3300/0 conn(0x7fdc4015cd40 0x7fdc400cf7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:43.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.730+0000 7fdc3f7fe640 1 -- 172.21.15.67:0/690799233 >> v1:172.21.15.67:6789/0 conn(0x7fdc400cd6d0 legacy=0x7fdc4009a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:43.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.730+0000 7fdc3f7fe640 1 -- 172.21.15.67:0/690799233 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc400cc910 con 0x7fdc40099d70 2024-03-24T00:48:43.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.730+0000 7fdc3e7fc640 1 -- 172.21.15.67:0/690799233 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4080248964 0 0) 0x7fdc4015d1b0 con 0x7fdc400cd6d0 2024-03-24T00:48:43.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.731+0000 7fdc3f7fe640 1 --2- 172.21.15.67:0/690799233 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc40099d70 0x7fdc4009adb0 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7fdc40073300 tx=0x7fdc3002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c29688854777617 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:43.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.731+0000 7fdc3e7fc640 1 -- 172.21.15.67:0/690799233 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc30035020 con 0x7fdc40099d70 2024-03-24T00:48:43.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.731+0000 7fdc3e7fc640 1 -- 172.21.15.67:0/690799233 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc30031830 con 0x7fdc40099d70 2024-03-24T00:48:43.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.731+0000 7fdc3e7fc640 1 -- 172.21.15.67:0/690799233 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc30031c00 con 0x7fdc40099d70 2024-03-24T00:48:43.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.731+0000 7fdc45fcd640 1 -- 172.21.15.67:0/690799233 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc40099d70 msgr2=0x7fdc4009adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:43.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.731+0000 7fdc45fcd640 1 --2- 172.21.15.67:0/690799233 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc40099d70 0x7fdc4009adb0 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7fdc40073300 tx=0x7fdc3002b8d0 comp rx=0 tx=0).stop 2024-03-24T00:48:43.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.732+0000 7fdc45fcd640 1 -- 172.21.15.67:0/690799233 shutdown_connections 2024-03-24T00:48:43.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.732+0000 7fdc45fcd640 1 --2- 172.21.15.67:0/690799233 >> v2:172.21.15.72:3300/0 conn(0x7fdc4015cd40 0x7fdc400cf7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:43.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.732+0000 7fdc45fcd640 1 --2- 172.21.15.67:0/690799233 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc40099d70 0x7fdc4009adb0 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:43.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.732+0000 7fdc45fcd640 1 -- 172.21.15.67:0/690799233 >> 172.21.15.67:0/690799233 conn(0x7fdc400b36c0 msgr2=0x7fdc400b3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:43.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.732+0000 7fdc45fcd640 1 -- 172.21.15.67:0/690799233 shutdown_connections 2024-03-24T00:48:43.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.732+0000 7fdc45fcd640 1 -- 172.21.15.67:0/690799233 wait complete. 2024-03-24T00:48:43.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.733+0000 7fdc45fcd640 1 Processor -- start 2024-03-24T00:48:43.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.733+0000 7fdc45fcd640 1 -- start start 2024-03-24T00:48:43.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.734+0000 7fdc45fcd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc40099d70 0x7fdc401570a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:43.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.734+0000 7fdc45fcd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc400cd6d0 0x7fdc400b4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:43.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.734+0000 7fdc45fcd640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdc4015cd40 0x7fdc400ca170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:43.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.734+0000 7fdc45fcd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc400cd1b0 con 0x7fdc40099d70 2024-03-24T00:48:43.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.734+0000 7fdc45fcd640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc400cced0 con 0x7fdc400cd6d0 2024-03-24T00:48:43.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.734+0000 7fdc45fcd640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc400cfce0 con 0x7fdc4015cd40 2024-03-24T00:48:43.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.734+0000 7fdc3f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc40099d70 0x7fdc401570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:43.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.734+0000 7fdc3f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc40099d70 0x7fdc401570a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34684/0 (socket says 172.21.15.67:34684) 2024-03-24T00:48:43.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.734+0000 7fdc3f7fe640 1 -- 172.21.15.67:0/526176918 learned_addr learned my addr 172.21.15.67:0/526176918 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:43.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.734+0000 7fdc3f7fe640 1 -- 172.21.15.67:0/526176918 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdc4015cd40 msgr2=0x7fdc400ca170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:43.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.734+0000 7fdc3f7fe640 1 --2- 172.21.15.67:0/526176918 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdc4015cd40 0x7fdc400ca170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:43.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.734+0000 7fdc3f7fe640 1 -- 172.21.15.67:0/526176918 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc400cd6d0 msgr2=0x7fdc400b4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:43.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.734+0000 7fdc3f7fe640 1 --2- 172.21.15.67:0/526176918 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc400cd6d0 0x7fdc400b4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:43.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.734+0000 7fdc3f7fe640 1 -- 172.21.15.67:0/526176918 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc40158c30 con 0x7fdc40099d70 2024-03-24T00:48:43.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.734+0000 7fdc3f7fe640 1 --2- 172.21.15.67:0/526176918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc40099d70 0x7fdc401570a0 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7fdc30034040 tx=0x7fdc300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:43.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.734+0000 7fdc27fff640 1 -- 172.21.15.67:0/526176918 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc30040070 con 0x7fdc40099d70 2024-03-24T00:48:43.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.734+0000 7fdc27fff640 1 -- 172.21.15.67:0/526176918 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc300399a0 con 0x7fdc40099d70 2024-03-24T00:48:43.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.734+0000 7fdc45fcd640 1 -- 172.21.15.67:0/526176918 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc400b5770 con 0x7fdc40099d70 2024-03-24T00:48:43.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.734+0000 7fdc27fff640 1 -- 172.21.15.67:0/526176918 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc30035040 con 0x7fdc40099d70 2024-03-24T00:48:43.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.734+0000 7fdc45fcd640 1 -- 172.21.15.67:0/526176918 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc401591b0 con 0x7fdc40099d70 2024-03-24T00:48:43.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.735+0000 7fdc45fcd640 1 -- 172.21.15.67:0/526176918 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc4004f400 con 0x7fdc40099d70 2024-03-24T00:48:43.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.736+0000 7fdc27fff640 1 -- 172.21.15.67:0/526176918 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fdc30038020 con 0x7fdc40099d70 2024-03-24T00:48:43.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.736+0000 7fdc27fff640 1 --2- 172.21.15.67:0/526176918 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdc2c06bbd0 0x7fdc2c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:43.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.736+0000 7fdc27fff640 1 -- 172.21.15.67:0/526176918 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6302+0+0 (secure 0 0 0) 0x7fdc300ac6c0 con 0x7fdc40099d70 2024-03-24T00:48:43.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.737+0000 7fdc3effd640 1 --2- 172.21.15.67:0/526176918 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdc2c06bbd0 0x7fdc2c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:43.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.737+0000 7fdc3effd640 1 --2- 172.21.15.67:0/526176918 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdc2c06bbd0 0x7fdc2c06e090 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fdc34007fd0 tx=0x7fdc34007920 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:43.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.741+0000 7fdc27fff640 1 -- 172.21.15.67:0/526176918 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fdc3007af70 con 0x7fdc40099d70 2024-03-24T00:48:43.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:43.922+0000 7fdc45fcd640 1 -- 172.21.15.67:0/526176918 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fdc400cdd60 con 0x7fdc40099d70 2024-03-24T00:48:44.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.608+0000 7fdc27fff640 1 -- 172.21.15.67:0/526176918 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v110) v1 ==== 181+0+0 (secure 0 0 0) 0x7fdc3007a870 con 0x7fdc40099d70 2024-03-24T00:48:44.609 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' removed 2024-03-24T00:48:44.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.612+0000 7fdc45fcd640 1 -- 172.21.15.67:0/526176918 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdc2c06bbd0 msgr2=0x7fdc2c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:44.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.612+0000 7fdc45fcd640 1 --2- 172.21.15.67:0/526176918 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdc2c06bbd0 0x7fdc2c06e090 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fdc34007fd0 tx=0x7fdc34007920 comp rx=0 tx=0).stop 2024-03-24T00:48:44.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.612+0000 7fdc45fcd640 1 -- 172.21.15.67:0/526176918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc40099d70 msgr2=0x7fdc401570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:44.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.612+0000 7fdc45fcd640 1 --2- 172.21.15.67:0/526176918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc40099d70 0x7fdc401570a0 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7fdc30034040 tx=0x7fdc300046c0 comp rx=0 tx=0).stop 2024-03-24T00:48:44.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.612+0000 7fdc45fcd640 1 -- 172.21.15.67:0/526176918 shutdown_connections 2024-03-24T00:48:44.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.612+0000 7fdc45fcd640 1 --2- 172.21.15.67:0/526176918 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdc2c06bbd0 0x7fdc2c06e090 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:44.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.612+0000 7fdc45fcd640 1 --2- 172.21.15.67:0/526176918 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdc4015cd40 0x7fdc400ca170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:44.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.612+0000 7fdc45fcd640 1 --2- 172.21.15.67:0/526176918 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc400cd6d0 0x7fdc400b4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:44.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.612+0000 7fdc45fcd640 1 --2- 172.21.15.67:0/526176918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc40099d70 0x7fdc401570a0 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:44.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.612+0000 7fdc45fcd640 1 -- 172.21.15.67:0/526176918 >> 172.21.15.67:0/526176918 conn(0x7fdc400b36c0 msgr2=0x7fdc400b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:44.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.612+0000 7fdc45fcd640 1 -- 172.21.15.67:0/526176918 shutdown_connections 2024-03-24T00:48:44.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.612+0000 7fdc45fcd640 1 -- 172.21.15.67:0/526176918 wait complete. 2024-03-24T00:48:44.627 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-24T00:48:44.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.737+0000 7fdf58bf3640 1 Processor -- start 2024-03-24T00:48:44.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.737+0000 7fdf58bf3640 1 -- start start 2024-03-24T00:48:44.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.738+0000 7fdf58bf3640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdf540cdc10 0x7fdf540cfd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:44.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.738+0000 7fdf58bf3640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdf5406d270 0x7fdf540cc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:44.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.738+0000 7fdf58bf3640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf540cdff0 con 0x7fdf54161ac0 2024-03-24T00:48:44.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.738+0000 7fdf58bf3640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf540d06a0 con 0x7fdf5406d270 2024-03-24T00:48:44.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.738+0000 7fdf58bf3640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf540d0980 con 0x7fdf540cdc10 2024-03-24T00:48:44.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.738+0000 7fdf52d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdf5406d270 0x7fdf540cc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:44.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.738+0000 7fdf4bfff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fdf54161ac0 0x7fdf5406cb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48570/0 (socket says 172.21.15.67:48570) 2024-03-24T00:48:44.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.738+0000 7fdf4bfff640 1 -- 172.21.15.67:0/1398084029 learned_addr learned my addr 172.21.15.67:0/1398084029 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:48:44.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.738+0000 7fdf52d9d640 1 --2- 172.21.15.67:0/1398084029 >> v2:172.21.15.67:3300/0 conn(0x7fdf5406d270 0x7fdf540cc1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34698/0 (socket says 172.21.15.67:34698) 2024-03-24T00:48:44.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.738+0000 7fdf5259c640 1 --2- 172.21.15.67:0/1398084029 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdf540cdc10 0x7fdf540cfd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:44.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.738+0000 7fdf52d9d640 1 -- 172.21.15.67:0/1398084029 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdf540cdc10 msgr2=0x7fdf540cfd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:44.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.738+0000 7fdf52d9d640 1 --2- 172.21.15.67:0/1398084029 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdf540cdc10 0x7fdf540cfd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:44.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.738+0000 7fdf52d9d640 1 -- 172.21.15.67:0/1398084029 >> v1:172.21.15.67:6789/0 conn(0x7fdf54161ac0 legacy=0x7fdf5406cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:44.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.738+0000 7fdf52d9d640 1 -- 172.21.15.67:0/1398084029 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf540d0c60 con 0x7fdf5406d270 2024-03-24T00:48:44.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.738+0000 7fdf51d9b640 1 -- 172.21.15.67:0/1398084029 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1929028500 0 0) 0x7fdf540cdff0 con 0x7fdf54161ac0 2024-03-24T00:48:44.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.739+0000 7fdf52d9d640 1 --2- 172.21.15.67:0/1398084029 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf5406d270 0x7fdf540cc1a0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c009d90 tx=0x7fdf4c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d48691cccb8576bf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:44.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.739+0000 7fdf51d9b640 1 -- 172.21.15.67:0/1398084029 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf4c039070 con 0x7fdf5406d270 2024-03-24T00:48:44.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.739+0000 7fdf51d9b640 1 -- 172.21.15.67:0/1398084029 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf4c02cc50 con 0x7fdf5406d270 2024-03-24T00:48:44.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.739+0000 7fdf51d9b640 1 -- 172.21.15.67:0/1398084029 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf4c034440 con 0x7fdf5406d270 2024-03-24T00:48:44.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.740+0000 7fdf58bf3640 1 -- 172.21.15.67:0/1398084029 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf5406d270 msgr2=0x7fdf540cc1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:44.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.740+0000 7fdf58bf3640 1 --2- 172.21.15.67:0/1398084029 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf5406d270 0x7fdf540cc1a0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c009d90 tx=0x7fdf4c02bea0 comp rx=0 tx=0).stop 2024-03-24T00:48:44.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.740+0000 7fdf58bf3640 1 -- 172.21.15.67:0/1398084029 shutdown_connections 2024-03-24T00:48:44.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.740+0000 7fdf58bf3640 1 --2- 172.21.15.67:0/1398084029 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf5406d270 0x7fdf540cc1a0 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:44.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.740+0000 7fdf58bf3640 1 --2- 172.21.15.67:0/1398084029 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdf540cdc10 0x7fdf540cfd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:44.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.740+0000 7fdf58bf3640 1 -- 172.21.15.67:0/1398084029 >> 172.21.15.67:0/1398084029 conn(0x7fdf540b3640 msgr2=0x7fdf540b3a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:44.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.740+0000 7fdf58bf3640 1 -- 172.21.15.67:0/1398084029 shutdown_connections 2024-03-24T00:48:44.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.740+0000 7fdf58bf3640 1 -- 172.21.15.67:0/1398084029 wait complete. 2024-03-24T00:48:44.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.741+0000 7fdf58bf3640 1 Processor -- start 2024-03-24T00:48:44.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.741+0000 7fdf58bf3640 1 -- start start 2024-03-24T00:48:44.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.742+0000 7fdf58bf3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf5406d270 0x7fdf54148ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:44.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.742+0000 7fdf58bf3640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdf540cdc10 0x7fdf540b2900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:44.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.742+0000 7fdf58bf3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf54161ac0 0x7fdf540b0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:44.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.742+0000 7fdf58bf3640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf540d1740 con 0x7fdf54161ac0 2024-03-24T00:48:44.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.742+0000 7fdf58bf3640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf540d1220 con 0x7fdf5406d270 2024-03-24T00:48:44.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.742+0000 7fdf58bf3640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf540d0290 con 0x7fdf540cdc10 2024-03-24T00:48:44.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.742+0000 7fdf52d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf54161ac0 0x7fdf540b0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:44.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.742+0000 7fdf52d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf54161ac0 0x7fdf540b0f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34704/0 (socket says 172.21.15.67:34704) 2024-03-24T00:48:44.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.742+0000 7fdf52d9d640 1 -- 172.21.15.67:0/2581497323 learned_addr learned my addr 172.21.15.67:0/2581497323 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:44.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.742+0000 7fdf52d9d640 1 -- 172.21.15.67:0/2581497323 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdf540cdc10 msgr2=0x7fdf540b2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:44.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.742+0000 7fdf52d9d640 1 --2- 172.21.15.67:0/2581497323 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdf540cdc10 0x7fdf540b2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:44.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.742+0000 7fdf52d9d640 1 -- 172.21.15.67:0/2581497323 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf5406d270 msgr2=0x7fdf54148ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:44.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.742+0000 7fdf52d9d640 1 --2- 172.21.15.67:0/2581497323 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf5406d270 0x7fdf54148ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:44.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.742+0000 7fdf52d9d640 1 -- 172.21.15.67:0/2581497323 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf540b1860 con 0x7fdf54161ac0 2024-03-24T00:48:44.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.742+0000 7fdf52d9d640 1 --2- 172.21.15.67:0/2581497323 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf54161ac0 0x7fdf540b0f80 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c037040 tx=0x7fdf4c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:44.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.742+0000 7fdf4affd640 1 -- 172.21.15.67:0/2581497323 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf4c044020 con 0x7fdf54161ac0 2024-03-24T00:48:44.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.743+0000 7fdf4affd640 1 -- 172.21.15.67:0/2581497323 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf4c03a040 con 0x7fdf54161ac0 2024-03-24T00:48:44.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.743+0000 7fdf58bf3640 1 -- 172.21.15.67:0/2581497323 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf540b1b40 con 0x7fdf54161ac0 2024-03-24T00:48:44.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.743+0000 7fdf4affd640 1 -- 172.21.15.67:0/2581497323 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf4c039050 con 0x7fdf54161ac0 2024-03-24T00:48:44.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.743+0000 7fdf58bf3640 1 -- 172.21.15.67:0/2581497323 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf540b2100 con 0x7fdf54161ac0 2024-03-24T00:48:44.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.744+0000 7fdf58bf3640 1 -- 172.21.15.67:0/2581497323 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf5404f400 con 0x7fdf54161ac0 2024-03-24T00:48:44.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.744+0000 7fdf4affd640 1 -- 172.21.15.67:0/2581497323 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fdf4c003990 con 0x7fdf54161ac0 2024-03-24T00:48:44.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.744+0000 7fdf4affd640 1 --2- 172.21.15.67:0/2581497323 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdf3406bc00 0x7fdf3406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:44.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.744+0000 7fdf4affd640 1 -- 172.21.15.67:0/2581497323 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5864+0+0 (secure 0 0 0) 0x7fdf4c0ace90 con 0x7fdf54161ac0 2024-03-24T00:48:44.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.745+0000 7fdf5259c640 1 --2- 172.21.15.67:0/2581497323 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdf3406bc00 0x7fdf3406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:44.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.745+0000 7fdf5259c640 1 --2- 172.21.15.67:0/2581497323 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdf3406bc00 0x7fdf3406e0c0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fdf5415b570 tx=0x7fdf44008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:44.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.749+0000 7fdf4affd640 1 -- 172.21.15.67:0/2581497323 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fdf4c04e060 con 0x7fdf54161ac0 2024-03-24T00:48:44.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:44.929+0000 7fdf58bf3640 1 -- 172.21.15.67:0/2581497323 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fdf54070aa0 con 0x7fdf54161ac0 2024-03-24T00:48:45.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.613+0000 7fdf4affd640 1 -- 172.21.15.67:0/2581497323 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v111) v1 ==== 169+0+0 (secure 0 0 0) 0x7fdf4c07b970 con 0x7fdf54161ac0 2024-03-24T00:48:45.614 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' removed 2024-03-24T00:48:45.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.615+0000 7fdf58bf3640 1 -- 172.21.15.67:0/2581497323 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdf3406bc00 msgr2=0x7fdf3406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:45.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.615+0000 7fdf58bf3640 1 --2- 172.21.15.67:0/2581497323 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdf3406bc00 0x7fdf3406e0c0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fdf5415b570 tx=0x7fdf44008040 comp rx=0 tx=0).stop 2024-03-24T00:48:45.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.615+0000 7fdf58bf3640 1 -- 172.21.15.67:0/2581497323 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf54161ac0 msgr2=0x7fdf540b0f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:45.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.615+0000 7fdf58bf3640 1 --2- 172.21.15.67:0/2581497323 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf54161ac0 0x7fdf540b0f80 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c037040 tx=0x7fdf4c004a40 comp rx=0 tx=0).stop 2024-03-24T00:48:45.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.616+0000 7fdf58bf3640 1 -- 172.21.15.67:0/2581497323 shutdown_connections 2024-03-24T00:48:45.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.616+0000 7fdf58bf3640 1 --2- 172.21.15.67:0/2581497323 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fdf3406bc00 0x7fdf3406e0c0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:45.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.616+0000 7fdf58bf3640 1 --2- 172.21.15.67:0/2581497323 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf54161ac0 0x7fdf540b0f80 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:45.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.616+0000 7fdf58bf3640 1 --2- 172.21.15.67:0/2581497323 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdf540cdc10 0x7fdf540b2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:45.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.616+0000 7fdf58bf3640 1 --2- 172.21.15.67:0/2581497323 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf5406d270 0x7fdf54148ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:45.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.616+0000 7fdf58bf3640 1 -- 172.21.15.67:0/2581497323 >> 172.21.15.67:0/2581497323 conn(0x7fdf540b3640 msgr2=0x7fdf540b3a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:45.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.617+0000 7fdf58bf3640 1 -- 172.21.15.67:0/2581497323 shutdown_connections 2024-03-24T00:48:45.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.617+0000 7fdf58bf3640 1 -- 172.21.15.67:0/2581497323 wait complete. 2024-03-24T00:48:45.631 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-24T00:48:45.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.736+0000 7f7dc8b90640 1 Processor -- start 2024-03-24T00:48:45.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.736+0000 7f7dc8b90640 1 -- start start 2024-03-24T00:48:45.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.737+0000 7f7dc8b90640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7dc4145540 0x7f7dc4160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:45.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.737+0000 7f7dc8b90640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7dc4071af0 0x7f7dc4071ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:45.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.737+0000 7f7dc8b90640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dc4072410 con 0x7f7dc415c2a0 2024-03-24T00:48:45.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.737+0000 7f7dc8b90640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dc406e620 con 0x7f7dc4071af0 2024-03-24T00:48:45.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.737+0000 7f7dc8b90640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dc406dcf0 con 0x7f7dc4145540 2024-03-24T00:48:45.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.737+0000 7f7dc2d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7dc4071af0 0x7f7dc4071ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:45.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.737+0000 7f7dc2d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7dc4071af0 0x7f7dc4071ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34716/0 (socket says 172.21.15.67:34716) 2024-03-24T00:48:45.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.737+0000 7f7dc2d9d640 1 -- 172.21.15.67:0/4266173138 learned_addr learned my addr 172.21.15.67:0/4266173138 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:45.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.737+0000 7f7dc2d9d640 1 -- 172.21.15.67:0/4266173138 >> v2:172.21.15.72:3300/0 conn(0x7f7dc4145540 msgr2=0x7f7dc4160420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:45.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.737+0000 7f7dc2d9d640 1 --2- 172.21.15.67:0/4266173138 >> v2:172.21.15.72:3300/0 conn(0x7f7dc4145540 0x7f7dc4160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:45.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.737+0000 7f7dc2d9d640 1 -- 172.21.15.67:0/4266173138 >> v1:172.21.15.72:6789/0 conn(0x7f7dc415c2a0 legacy=0x7f7dc40cef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:45.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.737+0000 7f7dc2d9d640 1 -- 172.21.15.67:0/4266173138 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dc406dfd0 con 0x7f7dc4071af0 2024-03-24T00:48:45.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.738+0000 7f7dc2d9d640 1 --2- 172.21.15.67:0/4266173138 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dc4071af0 0x7f7dc4071ed0 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f7dc406c0b0 tx=0x7f7dbc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cf920cd5bcf1076 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:45.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.738+0000 7f7dc159a640 1 -- 172.21.15.67:0/4266173138 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dbc035020 con 0x7f7dc4071af0 2024-03-24T00:48:45.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.738+0000 7f7dc159a640 1 -- 172.21.15.67:0/4266173138 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dbc031870 con 0x7f7dc4071af0 2024-03-24T00:48:45.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.738+0000 7f7dc159a640 1 -- 172.21.15.67:0/4266173138 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dbc031c40 con 0x7f7dc4071af0 2024-03-24T00:48:45.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.739+0000 7f7dc8b90640 1 -- 172.21.15.67:0/4266173138 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dc4071af0 msgr2=0x7f7dc4071ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:45.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.739+0000 7f7dc8b90640 1 --2- 172.21.15.67:0/4266173138 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dc4071af0 0x7f7dc4071ed0 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f7dc406c0b0 tx=0x7f7dbc02b910 comp rx=0 tx=0).stop 2024-03-24T00:48:45.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.739+0000 7f7dc8b90640 1 -- 172.21.15.67:0/4266173138 shutdown_connections 2024-03-24T00:48:45.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.739+0000 7f7dc8b90640 1 --2- 172.21.15.67:0/4266173138 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dc4071af0 0x7f7dc4071ed0 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:45.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.739+0000 7f7dc8b90640 1 --2- 172.21.15.67:0/4266173138 >> v2:172.21.15.72:3300/0 conn(0x7f7dc4145540 0x7f7dc4160420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:45.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.739+0000 7f7dc8b90640 1 -- 172.21.15.67:0/4266173138 >> 172.21.15.67:0/4266173138 conn(0x7f7dc40cf9e0 msgr2=0x7f7dc40cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:45.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.739+0000 7f7dc8b90640 1 -- 172.21.15.67:0/4266173138 shutdown_connections 2024-03-24T00:48:45.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.739+0000 7f7dc8b90640 1 -- 172.21.15.67:0/4266173138 wait complete. 2024-03-24T00:48:45.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.740+0000 7f7dc8b90640 1 Processor -- start 2024-03-24T00:48:45.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.740+0000 7f7dc8b90640 1 -- start start 2024-03-24T00:48:45.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.741+0000 7f7dc8b90640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7dc4071af0 0x7f7dc415b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:45.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.741+0000 7f7dc8b90640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7dc4145540 0x7f7dc413d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:45.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.741+0000 7f7dc8b90640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dc415c2a0 0x7f7dc41e6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:45.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.741+0000 7f7dc8b90640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc406eab0 con 0x7f7dc415c2a0 2024-03-24T00:48:45.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.741+0000 7f7dc8b90640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc4145920 con 0x7f7dc4145540 2024-03-24T00:48:45.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.741+0000 7f7dc8b90640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc406ec30 con 0x7f7dc4071af0 2024-03-24T00:48:45.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.741+0000 7f7dc2d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dc415c2a0 0x7f7dc41e6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:45.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.741+0000 7f7dc2d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dc415c2a0 0x7f7dc41e6d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34720/0 (socket says 172.21.15.67:34720) 2024-03-24T00:48:45.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.741+0000 7f7dc2d9d640 1 -- 172.21.15.67:0/2601611987 learned_addr learned my addr 172.21.15.67:0/2601611987 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:45.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.741+0000 7f7dc1d9b640 1 --2- 172.21.15.67:0/2601611987 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7dc4145540 0x7f7dc413d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:45.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.741+0000 7f7dc259c640 1 --2- 172.21.15.67:0/2601611987 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7dc4071af0 0x7f7dc415b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:45.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.741+0000 7f7dc2d9d640 1 -- 172.21.15.67:0/2601611987 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7dc4071af0 msgr2=0x7f7dc415b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:45.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.741+0000 7f7dc2d9d640 1 --2- 172.21.15.67:0/2601611987 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7dc4071af0 0x7f7dc415b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:45.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.741+0000 7f7dc2d9d640 1 -- 172.21.15.67:0/2601611987 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7dc4145540 msgr2=0x7f7dc413d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:45.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.741+0000 7f7dc2d9d640 1 --2- 172.21.15.67:0/2601611987 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7dc4145540 0x7f7dc413d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:45.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.741+0000 7f7dc2d9d640 1 -- 172.21.15.67:0/2601611987 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dc413da00 con 0x7f7dc415c2a0 2024-03-24T00:48:45.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.741+0000 7f7dc1d9b640 1 --2- 172.21.15.67:0/2601611987 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7dc4145540 0x7f7dc413d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:48:45.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.741+0000 7f7dc259c640 1 --2- 172.21.15.67:0/2601611987 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7dc4071af0 0x7f7dc415b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:48:45.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.741+0000 7f7dc2d9d640 1 --2- 172.21.15.67:0/2601611987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dc415c2a0 0x7f7dc41e6d80 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc02b950 tx=0x7f7dbc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:45.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.742+0000 7f7dbaffd640 1 -- 172.21.15.67:0/2601611987 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dbc0443f0 con 0x7f7dc415c2a0 2024-03-24T00:48:45.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.742+0000 7f7dbaffd640 1 -- 172.21.15.67:0/2601611987 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dbc0445b0 con 0x7f7dc415c2a0 2024-03-24T00:48:45.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.742+0000 7f7dc8b90640 1 -- 172.21.15.67:0/2601611987 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dc41374c0 con 0x7f7dc415c2a0 2024-03-24T00:48:45.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.742+0000 7f7dbaffd640 1 -- 172.21.15.67:0/2601611987 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dbc0449a0 con 0x7f7dc415c2a0 2024-03-24T00:48:45.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.742+0000 7f7dc8b90640 1 -- 172.21.15.67:0/2601611987 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dc4137a80 con 0x7f7dc415c2a0 2024-03-24T00:48:45.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.743+0000 7f7dc8b90640 1 -- 172.21.15.67:0/2601611987 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dc4137c80 con 0x7f7dc415c2a0 2024-03-24T00:48:45.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.743+0000 7f7dbaffd640 1 -- 172.21.15.67:0/2601611987 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7dbc038020 con 0x7f7dc415c2a0 2024-03-24T00:48:45.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.743+0000 7f7dbaffd640 1 --2- 172.21.15.67:0/2601611987 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7da406bbd0 0x7f7da406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:45.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.744+0000 7f7dbaffd640 1 -- 172.21.15.67:0/2601611987 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f7dbc0acb80 con 0x7f7dc415c2a0 2024-03-24T00:48:45.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.744+0000 7f7dc259c640 1 --2- 172.21.15.67:0/2601611987 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7da406bbd0 0x7f7da406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:45.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.744+0000 7f7dc259c640 1 --2- 172.21.15.67:0/2601611987 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7da406bbd0 0x7f7da406e090 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f7db4007eb0 tx=0x7f7db40073c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:45.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.748+0000 7f7dbaffd640 1 -- 172.21.15.67:0/2601611987 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7dc4137c80 con 0x7f7dc415c2a0 2024-03-24T00:48:45.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.929+0000 7f7dc8b90640 1 -- 172.21.15.67:0/2601611987 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f7dc404f400 con 0x7f7dc415c2a0 2024-03-24T00:48:45.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.932+0000 7f7dbaffd640 1 -- 172.21.15.67:0/2601611987 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v21) v1 ==== 104+0+0 (secure 0 0 0) 0x7f7dbc03f070 con 0x7f7dc415c2a0 2024-03-24T00:48:45.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.934+0000 7f7dc8b90640 1 -- 172.21.15.67:0/2601611987 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7da406bbd0 msgr2=0x7f7da406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:45.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.934+0000 7f7dc8b90640 1 --2- 172.21.15.67:0/2601611987 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7da406bbd0 0x7f7da406e090 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f7db4007eb0 tx=0x7f7db40073c0 comp rx=0 tx=0).stop 2024-03-24T00:48:45.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.934+0000 7f7dc8b90640 1 -- 172.21.15.67:0/2601611987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dc415c2a0 msgr2=0x7f7dc41e6d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:45.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.934+0000 7f7dc8b90640 1 --2- 172.21.15.67:0/2601611987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dc415c2a0 0x7f7dc41e6d80 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc02b950 tx=0x7f7dbc0046c0 comp rx=0 tx=0).stop 2024-03-24T00:48:45.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.934+0000 7f7dc8b90640 1 -- 172.21.15.67:0/2601611987 shutdown_connections 2024-03-24T00:48:45.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.934+0000 7f7dc8b90640 1 --2- 172.21.15.67:0/2601611987 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7da406bbd0 0x7f7da406e090 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:45.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.934+0000 7f7dc8b90640 1 --2- 172.21.15.67:0/2601611987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dc415c2a0 0x7f7dc41e6d80 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:45.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.934+0000 7f7dc8b90640 1 --2- 172.21.15.67:0/2601611987 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7dc4145540 0x7f7dc413d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:45.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.934+0000 7f7dc8b90640 1 --2- 172.21.15.67:0/2601611987 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7dc4071af0 0x7f7dc415b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:45.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.934+0000 7f7dc8b90640 1 -- 172.21.15.67:0/2601611987 >> 172.21.15.67:0/2601611987 conn(0x7f7dc40cf9e0 msgr2=0x7f7dc415e620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:45.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.934+0000 7f7dc8b90640 1 -- 172.21.15.67:0/2601611987 shutdown_connections 2024-03-24T00:48:45.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:45.934+0000 7f7dc8b90640 1 -- 172.21.15.67:0/2601611987 wait complete. 2024-03-24T00:48:45.946 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_inotable_sync' 2024-03-24T00:48:46.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.049+0000 7f99089c6640 1 Processor -- start 2024-03-24T00:48:46.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.049+0000 7f99089c6640 1 -- start start 2024-03-24T00:48:46.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.050+0000 7f99089c6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f990415df30 0x7f9904162610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:46.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.050+0000 7f99089c6640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f990415e310 0x7f99040d0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:46.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.050+0000 7f99089c6640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99040ce920 con 0x7f99040d0b80 2024-03-24T00:48:46.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.050+0000 7f99089c6640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9904162b50 con 0x7f990415e310 2024-03-24T00:48:46.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.050+0000 7f99089c6640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99040d0fd0 con 0x7f990415df30 2024-03-24T00:48:46.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.050+0000 7f9902d9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f99040d0b80 0x7f99040b57a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48582/0 (socket says 172.21.15.67:48582) 2024-03-24T00:48:46.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.050+0000 7f9902d9d640 1 -- 172.21.15.67:0/3943305399 learned_addr learned my addr 172.21.15.67:0/3943305399 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:48:46.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.050+0000 7f990259c640 1 --2- 172.21.15.67:0/3943305399 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f990415df30 0x7f9904162610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:46.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.050+0000 7f9901d9b640 1 --2- 172.21.15.67:0/3943305399 >> v2:172.21.15.72:3300/0 conn(0x7f990415e310 0x7f99040d0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:46.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.051+0000 7f990159a640 1 -- 172.21.15.67:0/3943305399 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 831379329 0 0) 0x7f99040ce920 con 0x7f99040d0b80 2024-03-24T00:48:46.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.051+0000 7f990159a640 1 -- 172.21.15.67:0/3943305399 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98f0003520 con 0x7f99040d0b80 2024-03-24T00:48:46.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.051+0000 7f990259c640 1 -- 172.21.15.67:0/3943305399 >> v2:172.21.15.72:3300/0 conn(0x7f990415e310 msgr2=0x7f99040d0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:46.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.051+0000 7f990259c640 1 --2- 172.21.15.67:0/3943305399 >> v2:172.21.15.72:3300/0 conn(0x7f990415e310 0x7f99040d0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:46.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.051+0000 7f990259c640 1 -- 172.21.15.67:0/3943305399 >> v1:172.21.15.67:6789/0 conn(0x7f99040d0b80 legacy=0x7f99040b57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:46.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.051+0000 7f990259c640 1 -- 172.21.15.67:0/3943305399 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99040cef20 con 0x7f990415df30 2024-03-24T00:48:46.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.051+0000 7f9901d9b640 1 --2- 172.21.15.67:0/3943305399 >> v2:172.21.15.72:3300/0 conn(0x7f990415e310 0x7f99040d0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:48:46.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.052+0000 7f990259c640 1 --2- 172.21.15.67:0/3943305399 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f990415df30 0x7f9904162610 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f98fc0097a0 tx=0x7f98fc02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30b00edf2a811eb3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:46.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.052+0000 7f990159a640 1 -- 172.21.15.67:0/3943305399 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98fc0315a0 con 0x7f990415df30 2024-03-24T00:48:46.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.052+0000 7f990159a640 1 -- 172.21.15.67:0/3943305399 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98fc037070 con 0x7f990415df30 2024-03-24T00:48:46.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.052+0000 7f990159a640 1 -- 172.21.15.67:0/3943305399 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98fc031960 con 0x7f990415df30 2024-03-24T00:48:46.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.052+0000 7f99089c6640 1 -- 172.21.15.67:0/3943305399 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f990415df30 msgr2=0x7f9904162610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:46.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.052+0000 7f99089c6640 1 --2- 172.21.15.67:0/3943305399 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f990415df30 0x7f9904162610 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f98fc0097a0 tx=0x7f98fc02b840 comp rx=0 tx=0).stop 2024-03-24T00:48:46.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.052+0000 7f99089c6640 1 -- 172.21.15.67:0/3943305399 shutdown_connections 2024-03-24T00:48:46.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.052+0000 7f99089c6640 1 --2- 172.21.15.67:0/3943305399 >> v2:172.21.15.72:3300/0 conn(0x7f990415e310 0x7f99040d0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:46.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.052+0000 7f99089c6640 1 --2- 172.21.15.67:0/3943305399 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f990415df30 0x7f9904162610 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:46.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.052+0000 7f99089c6640 1 -- 172.21.15.67:0/3943305399 >> 172.21.15.67:0/3943305399 conn(0x7f990406c880 msgr2=0x7f990406cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:46.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.052+0000 7f99089c6640 1 -- 172.21.15.67:0/3943305399 shutdown_connections 2024-03-24T00:48:46.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.052+0000 7f99089c6640 1 -- 172.21.15.67:0/3943305399 wait complete. 2024-03-24T00:48:46.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.054+0000 7f99089c6640 1 Processor -- start 2024-03-24T00:48:46.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.054+0000 7f99089c6640 1 -- start start 2024-03-24T00:48:46.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.054+0000 7f99089c6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99040d0b80 0x7f99041f7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:46.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.054+0000 7f99089c6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f990415df30 0x7f9904160e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:46.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.054+0000 7f99089c6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9904161370 0x7f99040b5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:46.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.054+0000 7f99089c6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99040b32f0 con 0x7f99040d0b80 2024-03-24T00:48:46.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.054+0000 7f99089c6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99040cf4e0 con 0x7f9904161370 2024-03-24T00:48:46.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.054+0000 7f99089c6640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99040ce580 con 0x7f990415df30 2024-03-24T00:48:46.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.054+0000 7f990259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99040d0b80 0x7f99041f7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:46.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.054+0000 7f990259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99040d0b80 0x7f99041f7a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34736/0 (socket says 172.21.15.67:34736) 2024-03-24T00:48:46.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.055+0000 7f990259c640 1 -- 172.21.15.67:0/2176686023 learned_addr learned my addr 172.21.15.67:0/2176686023 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:46.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.055+0000 7f9901d9b640 1 --2- 172.21.15.67:0/2176686023 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f990415df30 0x7f9904160e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:46.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.055+0000 7f9902d9d640 1 --2- 172.21.15.67:0/2176686023 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9904161370 0x7f99040b5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:46.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.055+0000 7f990259c640 1 -- 172.21.15.67:0/2176686023 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f990415df30 msgr2=0x7f9904160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:46.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.055+0000 7f990259c640 1 --2- 172.21.15.67:0/2176686023 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f990415df30 0x7f9904160e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:46.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.055+0000 7f990259c640 1 -- 172.21.15.67:0/2176686023 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9904161370 msgr2=0x7f99040b5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:46.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.055+0000 7f990259c640 1 --2- 172.21.15.67:0/2176686023 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9904161370 0x7f99040b5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:46.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.055+0000 7f990259c640 1 -- 172.21.15.67:0/2176686023 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f990414ec50 con 0x7f99040d0b80 2024-03-24T00:48:46.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.055+0000 7f990259c640 1 --2- 172.21.15.67:0/2176686023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99040d0b80 0x7f99041f7a80 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f98fc000c00 tx=0x7f98fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:46.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.055+0000 7f98faffd640 1 -- 172.21.15.67:0/2176686023 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98fc032630 con 0x7f99040d0b80 2024-03-24T00:48:46.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.055+0000 7f98faffd640 1 -- 172.21.15.67:0/2176686023 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98fc0327f0 con 0x7f99040d0b80 2024-03-24T00:48:46.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.055+0000 7f99089c6640 1 -- 172.21.15.67:0/2176686023 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99041f7fc0 con 0x7f99040d0b80 2024-03-24T00:48:46.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.055+0000 7f98faffd640 1 -- 172.21.15.67:0/2176686023 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98fc032be0 con 0x7f99040d0b80 2024-03-24T00:48:46.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.055+0000 7f99089c6640 1 -- 172.21.15.67:0/2176686023 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f990414f1a0 con 0x7f99040d0b80 2024-03-24T00:48:46.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.056+0000 7f99089c6640 1 -- 172.21.15.67:0/2176686023 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f990404f400 con 0x7f99040d0b80 2024-03-24T00:48:46.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.057+0000 7f98faffd640 1 -- 172.21.15.67:0/2176686023 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f98fc04d3f0 con 0x7f99040d0b80 2024-03-24T00:48:46.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.057+0000 7f98faffd640 1 --2- 172.21.15.67:0/2176686023 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f98e406bbd0 0x7f98e406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:46.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.057+0000 7f98faffd640 1 -- 172.21.15.67:0/2176686023 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f98fc02f030 con 0x7f99040d0b80 2024-03-24T00:48:46.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.058+0000 7f9901d9b640 1 --2- 172.21.15.67:0/2176686023 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f98e406bbd0 0x7f98e406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:46.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.058+0000 7f9901d9b640 1 --2- 172.21.15.67:0/2176686023 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f98e406bbd0 0x7f98e406e090 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f98ec009560 tx=0x7f98ec009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:46.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.062+0000 7f98faffd640 1 -- 172.21.15.67:0/2176686023 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f98fc040070 con 0x7f99040d0b80 2024-03-24T00:48:46.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.319+0000 7f99089c6640 1 -- 172.21.15.67:0/2176686023 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_inotable_sync"]} v 0) v1 -- 0x7f99040d1a30 con 0x7f99040d0b80 2024-03-24T00:48:46.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.614+0000 7f98faffd640 1 -- 172.21.15.67:0/2176686023 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_inotable_sync"]}]=0 v461) v1 ==== 146+0+0 (secure 0 0 0) 0x7f98fc07a740 con 0x7f99040d0b80 2024-03-24T00:48:46.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.617+0000 7f99089c6640 1 -- 172.21.15.67:0/2176686023 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f98e406bbd0 msgr2=0x7f98e406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:46.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.617+0000 7f99089c6640 1 --2- 172.21.15.67:0/2176686023 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f98e406bbd0 0x7f98e406e090 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f98ec009560 tx=0x7f98ec009190 comp rx=0 tx=0).stop 2024-03-24T00:48:46.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.617+0000 7f99089c6640 1 -- 172.21.15.67:0/2176686023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99040d0b80 msgr2=0x7f99041f7a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:46.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.617+0000 7f99089c6640 1 --2- 172.21.15.67:0/2176686023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99040d0b80 0x7f99041f7a80 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f98fc000c00 tx=0x7f98fc0046c0 comp rx=0 tx=0).stop 2024-03-24T00:48:46.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.617+0000 7f99089c6640 1 -- 172.21.15.67:0/2176686023 shutdown_connections 2024-03-24T00:48:46.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.617+0000 7f99089c6640 1 --2- 172.21.15.67:0/2176686023 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f98e406bbd0 0x7f98e406e090 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:46.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.617+0000 7f99089c6640 1 --2- 172.21.15.67:0/2176686023 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9904161370 0x7f99040b5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:46.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.617+0000 7f99089c6640 1 --2- 172.21.15.67:0/2176686023 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f990415df30 0x7f9904160e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:46.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.617+0000 7f99089c6640 1 --2- 172.21.15.67:0/2176686023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99040d0b80 0x7f99041f7a80 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:46.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.617+0000 7f99089c6640 1 -- 172.21.15.67:0/2176686023 >> 172.21.15.67:0/2176686023 conn(0x7f990406c880 msgr2=0x7f99040c7e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:46.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.617+0000 7f99089c6640 1 -- 172.21.15.67:0/2176686023 shutdown_connections 2024-03-24T00:48:46.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.617+0000 7f99089c6640 1 -- 172.21.15.67:0/2176686023 wait complete. 2024-03-24T00:48:46.629 INFO:tasks.cephfs_test_runner:test_inotable_sync (tasks.cephfs.test_forward_scrub.TestForwardScrub) ... ok 2024-03-24T00:48:46.629 INFO:tasks.cephfs_test_runner:Starting test: test_orphan_scan (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2024-03-24T00:48:46.630 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:48:46.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.732+0000 7f5e5cfe6640 1 Processor -- start 2024-03-24T00:48:46.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.732+0000 7f5e5cfe6640 1 -- start start 2024-03-24T00:48:46.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.733+0000 7f5e5cfe6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5e5806ec40 0x7f5e580cdd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:46.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.733+0000 7f5e5cfe6640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5e58071ad0 0x7f5e5815bd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:46.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.733+0000 7f5e5cfe6640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e58071f50 con 0x7f5e5806f020 2024-03-24T00:48:46.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.733+0000 7f5e5cfe6640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e58072a70 con 0x7f5e58071ad0 2024-03-24T00:48:46.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.733+0000 7f5e5cfe6640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e580721d0 con 0x7f5e5806ec40 2024-03-24T00:48:46.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.733+0000 7f5e56d9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5e5806f020 0x7f5e5815e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48586/0 (socket says 172.21.15.67:48586) 2024-03-24T00:48:46.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.733+0000 7f5e56d9d640 1 -- 172.21.15.67:0/3084696874 learned_addr learned my addr 172.21.15.67:0/3084696874 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:48:46.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.733+0000 7f5e5659c640 1 --2- 172.21.15.67:0/3084696874 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5e5806ec40 0x7f5e580cdd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:46.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.733+0000 7f5e4ffff640 1 --2- 172.21.15.67:0/3084696874 >> v2:172.21.15.72:3300/0 conn(0x7f5e58071ad0 0x7f5e5815bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:46.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.733+0000 7f5e55d9b640 1 -- 172.21.15.67:0/3084696874 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3618911154 0 0) 0x7f5e50001490 con 0x7f5e5806f020 2024-03-24T00:48:46.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.733+0000 7f5e55d9b640 1 -- 172.21.15.67:0/3084696874 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e44003520 con 0x7f5e5806f020 2024-03-24T00:48:46.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.733+0000 7f5e55d9b640 1 -- 172.21.15.67:0/3084696874 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2330393234 0 0) 0x7f5e44003520 con 0x7f5e5806f020 2024-03-24T00:48:46.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.733+0000 7f5e5659c640 1 -- 172.21.15.67:0/3084696874 >> v2:172.21.15.72:3300/0 conn(0x7f5e58071ad0 msgr2=0x7f5e5815bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:46.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.733+0000 7f5e5659c640 1 --2- 172.21.15.67:0/3084696874 >> v2:172.21.15.72:3300/0 conn(0x7f5e58071ad0 0x7f5e5815bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:46.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.733+0000 7f5e5659c640 1 -- 172.21.15.67:0/3084696874 >> v1:172.21.15.67:6789/0 conn(0x7f5e5806f020 legacy=0x7f5e5815e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:46.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.733+0000 7f5e5659c640 1 -- 172.21.15.67:0/3084696874 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e580724b0 con 0x7f5e5806ec40 2024-03-24T00:48:46.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.734+0000 7f5e5659c640 1 --2- 172.21.15.67:0/3084696874 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5e5806ec40 0x7f5e580cdd50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5e480097a0 tx=0x7f5e4802b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=307e5d65ae52a4e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:46.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.734+0000 7f5e55d9b640 1 -- 172.21.15.67:0/3084696874 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e48037070 con 0x7f5e5806ec40 2024-03-24T00:48:46.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.734+0000 7f5e55d9b640 1 -- 172.21.15.67:0/3084696874 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e48030510 con 0x7f5e5806ec40 2024-03-24T00:48:46.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.734+0000 7f5e55d9b640 1 -- 172.21.15.67:0/3084696874 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e50001bc0 con 0x7f5e5806ec40 2024-03-24T00:48:46.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.735+0000 7f5e5cfe6640 1 -- 172.21.15.67:0/3084696874 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5e5806ec40 msgr2=0x7f5e580cdd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:46.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.735+0000 7f5e5cfe6640 1 --2- 172.21.15.67:0/3084696874 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5e5806ec40 0x7f5e580cdd50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5e480097a0 tx=0x7f5e4802b710 comp rx=0 tx=0).stop 2024-03-24T00:48:46.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.735+0000 7f5e5cfe6640 1 -- 172.21.15.67:0/3084696874 shutdown_connections 2024-03-24T00:48:46.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.735+0000 7f5e5cfe6640 1 --2- 172.21.15.67:0/3084696874 >> v2:172.21.15.72:3300/0 conn(0x7f5e58071ad0 0x7f5e5815bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:46.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.735+0000 7f5e5cfe6640 1 --2- 172.21.15.67:0/3084696874 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5e5806ec40 0x7f5e580cdd50 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:46.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.735+0000 7f5e5cfe6640 1 -- 172.21.15.67:0/3084696874 >> 172.21.15.67:0/3084696874 conn(0x7f5e580cf9e0 msgr2=0x7f5e580cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:46.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.735+0000 7f5e5cfe6640 1 -- 172.21.15.67:0/3084696874 shutdown_connections 2024-03-24T00:48:46.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.735+0000 7f5e5cfe6640 1 -- 172.21.15.67:0/3084696874 wait complete. 2024-03-24T00:48:46.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.736+0000 7f5e5cfe6640 1 Processor -- start 2024-03-24T00:48:46.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.736+0000 7f5e5cfe6640 1 -- start start 2024-03-24T00:48:46.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.737+0000 7f5e5cfe6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5e5806ec40 0x7f5e58154c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:46.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.737+0000 7f5e5cfe6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e5806f020 0x7f5e58155190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:46.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.737+0000 7f5e5cfe6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e58071ad0 0x7f5e581f13d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:46.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.737+0000 7f5e5cfe6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5806f4b0 con 0x7f5e58071ad0 2024-03-24T00:48:46.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.737+0000 7f5e5cfe6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e580b2b40 con 0x7f5e5806f020 2024-03-24T00:48:46.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.737+0000 7f5e5cfe6640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e58144850 con 0x7f5e5806ec40 2024-03-24T00:48:46.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.737+0000 7f5e56d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e58071ad0 0x7f5e581f13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:46.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.737+0000 7f5e56d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e58071ad0 0x7f5e581f13d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34750/0 (socket says 172.21.15.67:34750) 2024-03-24T00:48:46.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.737+0000 7f5e56d9d640 1 -- 172.21.15.67:0/1759110988 learned_addr learned my addr 172.21.15.67:0/1759110988 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:46.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.737+0000 7f5e5659c640 1 --2- 172.21.15.67:0/1759110988 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5e5806ec40 0x7f5e58154c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:46.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.737+0000 7f5e4ffff640 1 --2- 172.21.15.67:0/1759110988 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e5806f020 0x7f5e58155190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:46.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.737+0000 7f5e56d9d640 1 -- 172.21.15.67:0/1759110988 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5e5806ec40 msgr2=0x7f5e58154c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:46.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.737+0000 7f5e56d9d640 1 --2- 172.21.15.67:0/1759110988 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5e5806ec40 0x7f5e58154c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:46.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.737+0000 7f5e56d9d640 1 -- 172.21.15.67:0/1759110988 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e5806f020 msgr2=0x7f5e58155190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:46.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.737+0000 7f5e56d9d640 1 --2- 172.21.15.67:0/1759110988 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e5806f020 0x7f5e58155190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:46.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.737+0000 7f5e56d9d640 1 -- 172.21.15.67:0/1759110988 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e58129680 con 0x7f5e58071ad0 2024-03-24T00:48:46.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.737+0000 7f5e4ffff640 1 --2- 172.21.15.67:0/1759110988 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e5806f020 0x7f5e58155190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:48:46.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.738+0000 7f5e56d9d640 1 --2- 172.21.15.67:0/1759110988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e58071ad0 0x7f5e581f13d0 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f5e50001a90 tx=0x7f5e5000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:46.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.738+0000 7f5e4effd640 1 -- 172.21.15.67:0/1759110988 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e50012400 con 0x7f5e58071ad0 2024-03-24T00:48:46.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.738+0000 7f5e4effd640 1 -- 172.21.15.67:0/1759110988 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e500125c0 con 0x7f5e58071ad0 2024-03-24T00:48:46.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.738+0000 7f5e5cfe6640 1 -- 172.21.15.67:0/1759110988 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e58129960 con 0x7f5e58071ad0 2024-03-24T00:48:46.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.738+0000 7f5e4effd640 1 -- 172.21.15.67:0/1759110988 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e500129b0 con 0x7f5e58071ad0 2024-03-24T00:48:46.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.738+0000 7f5e5cfe6640 1 -- 172.21.15.67:0/1759110988 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e58129f20 con 0x7f5e58071ad0 2024-03-24T00:48:46.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.739+0000 7f5e5cfe6640 1 -- 172.21.15.67:0/1759110988 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e580b27e0 con 0x7f5e58071ad0 2024-03-24T00:48:46.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.739+0000 7f5e4effd640 1 -- 172.21.15.67:0/1759110988 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5e50012b50 con 0x7f5e58071ad0 2024-03-24T00:48:46.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.739+0000 7f5e4effd640 1 --2- 172.21.15.67:0/1759110988 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5e3806bbd0 0x7f5e3806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:46.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.740+0000 7f5e4effd640 1 -- 172.21.15.67:0/1759110988 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f5e5008b230 con 0x7f5e58071ad0 2024-03-24T00:48:46.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.740+0000 7f5e5659c640 1 --2- 172.21.15.67:0/1759110988 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5e3806bbd0 0x7f5e3806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:46.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.740+0000 7f5e5659c640 1 --2- 172.21.15.67:0/1759110988 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5e3806bbd0 0x7f5e3806e090 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f5e48034000 tx=0x7f5e48008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:46.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.744+0000 7f5e4effd640 1 -- 172.21.15.67:0/1759110988 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5e580b27e0 con 0x7f5e58071ad0 2024-03-24T00:48:46.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.927+0000 7f5e5cfe6640 1 -- 172.21.15.67:0/1759110988 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e5804f400 con 0x7f5e58071ad0 2024-03-24T00:48:46.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.928+0000 7f5e4effd640 1 -- 172.21.15.67:0/1759110988 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v111) v1 ==== 74+0+13373 (secure 0 0 0) 0x7f5e58071f50 con 0x7f5e58071ad0 2024-03-24T00:48:46.929 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:48:46.929 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":111,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:48:45.610169+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6833/3197261495":"2024-03-25T00:48:37.687657+0000","172.21.15.67:6832/3197261495":"2024-03-25T00:48:37.687657+0000","172.21.15.67:6833/1826974843":"2024-03-25T00:48:01.322612+0000","172.21.15.67:6832/1826974843":"2024-03-25T00:48:01.322612+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:48:46.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.930+0000 7f5e5cfe6640 1 -- 172.21.15.67:0/1759110988 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5e3806bbd0 msgr2=0x7f5e3806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:46.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.930+0000 7f5e5cfe6640 1 --2- 172.21.15.67:0/1759110988 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5e3806bbd0 0x7f5e3806e090 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f5e48034000 tx=0x7f5e48008030 comp rx=0 tx=0).stop 2024-03-24T00:48:46.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.930+0000 7f5e5cfe6640 1 -- 172.21.15.67:0/1759110988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e58071ad0 msgr2=0x7f5e581f13d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:46.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.930+0000 7f5e5cfe6640 1 --2- 172.21.15.67:0/1759110988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e58071ad0 0x7f5e581f13d0 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f5e50001a90 tx=0x7f5e5000a430 comp rx=0 tx=0).stop 2024-03-24T00:48:46.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.930+0000 7f5e5cfe6640 1 -- 172.21.15.67:0/1759110988 shutdown_connections 2024-03-24T00:48:46.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.930+0000 7f5e5cfe6640 1 --2- 172.21.15.67:0/1759110988 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5e3806bbd0 0x7f5e3806e090 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:46.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.930+0000 7f5e5cfe6640 1 --2- 172.21.15.67:0/1759110988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e58071ad0 0x7f5e581f13d0 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:46.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.930+0000 7f5e5cfe6640 1 --2- 172.21.15.67:0/1759110988 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e5806f020 0x7f5e58155190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:46.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.930+0000 7f5e5cfe6640 1 --2- 172.21.15.67:0/1759110988 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5e5806ec40 0x7f5e58154c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:46.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.930+0000 7f5e5cfe6640 1 -- 172.21.15.67:0/1759110988 >> 172.21.15.67:0/1759110988 conn(0x7f5e580cf9e0 msgr2=0x7f5e581405a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:46.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.930+0000 7f5e5cfe6640 1 -- 172.21.15.67:0/1759110988 shutdown_connections 2024-03-24T00:48:46.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:46.930+0000 7f5e5cfe6640 1 -- 172.21.15.67:0/1759110988 wait complete. 2024-03-24T00:48:46.942 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:38:39.364950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:48:46.942 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:48:47.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.046+0000 7f0224af8640 1 Processor -- start 2024-03-24T00:48:47.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.046+0000 7f0224af8640 1 -- start start 2024-03-24T00:48:47.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.047+0000 7f0224af8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f022015dea0 0x7f02200d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:47.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.047+0000 7f0224af8640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f022015e280 0x7f02200d3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:47.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.047+0000 7f0224af8640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02200b5a60 0x7f02200b5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:47.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.047+0000 7f0224af8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02200b63f0 con 0x7f022015dea0 2024-03-24T00:48:47.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.047+0000 7f0224af8640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02200d2250 con 0x7f022015e280 2024-03-24T00:48:47.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.047+0000 7f0224af8640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02200d2530 con 0x7f02200b5a60 2024-03-24T00:48:47.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.048+0000 7f021e59c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f022015dea0 0x7f02200d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:47.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.048+0000 7f021e59c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f022015dea0 0x7f02200d33d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34764/0 (socket says 172.21.15.67:34764) 2024-03-24T00:48:47.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.048+0000 7f021e59c640 1 -- 172.21.15.67:0/25605404 learned_addr learned my addr 172.21.15.67:0/25605404 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:47.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.048+0000 7f021ed9d640 1 --2- 172.21.15.67:0/25605404 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02200b5a60 0x7f02200b5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:47.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.048+0000 7f021dd9b640 1 --2- 172.21.15.67:0/25605404 >> v2:172.21.15.72:3300/0 conn(0x7f022015e280 0x7f02200d3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:47.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.048+0000 7f021e59c640 1 -- 172.21.15.67:0/25605404 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02200b5a60 msgr2=0x7f02200b5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:47.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.048+0000 7f021e59c640 1 --2- 172.21.15.67:0/25605404 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02200b5a60 0x7f02200b5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:47.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.048+0000 7f021e59c640 1 -- 172.21.15.67:0/25605404 >> v2:172.21.15.72:3300/0 conn(0x7f022015e280 msgr2=0x7f02200d3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:47.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.048+0000 7f021e59c640 1 --2- 172.21.15.67:0/25605404 >> v2:172.21.15.72:3300/0 conn(0x7f022015e280 0x7f02200d3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:47.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.048+0000 7f021e59c640 1 -- 172.21.15.67:0/25605404 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02200d2810 con 0x7f022015dea0 2024-03-24T00:48:47.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.048+0000 7f021ed9d640 1 --2- 172.21.15.67:0/25605404 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02200b5a60 0x7f02200b5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:48:47.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.049+0000 7f021e59c640 1 --2- 172.21.15.67:0/25605404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f022015dea0 0x7f02200d33d0 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f02200d04d0 tx=0x7f021802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=358b405b3c30272c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:47.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.049+0000 7f021d59a640 1 -- 172.21.15.67:0/25605404 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f021802c9e0 con 0x7f022015dea0 2024-03-24T00:48:47.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.049+0000 7f021d59a640 1 -- 172.21.15.67:0/25605404 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f021802cb80 con 0x7f022015dea0 2024-03-24T00:48:47.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.049+0000 7f021d59a640 1 -- 172.21.15.67:0/25605404 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02180343f0 con 0x7f022015dea0 2024-03-24T00:48:47.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.049+0000 7f0224af8640 1 -- 172.21.15.67:0/25605404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f022015dea0 msgr2=0x7f02200d33d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:47.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.049+0000 7f0224af8640 1 --2- 172.21.15.67:0/25605404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f022015dea0 0x7f02200d33d0 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f02200d04d0 tx=0x7f021802be40 comp rx=0 tx=0).stop 2024-03-24T00:48:47.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.049+0000 7f0224af8640 1 -- 172.21.15.67:0/25605404 shutdown_connections 2024-03-24T00:48:47.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.049+0000 7f0224af8640 1 --2- 172.21.15.67:0/25605404 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02200b5a60 0x7f02200b5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:47.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.049+0000 7f0224af8640 1 --2- 172.21.15.67:0/25605404 >> v2:172.21.15.72:3300/0 conn(0x7f022015e280 0x7f02200d3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:47.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.049+0000 7f0224af8640 1 --2- 172.21.15.67:0/25605404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f022015dea0 0x7f02200d33d0 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:47.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.049+0000 7f0224af8640 1 -- 172.21.15.67:0/25605404 >> 172.21.15.67:0/25605404 conn(0x7f0220072da0 msgr2=0x7f02200d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:47.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.049+0000 7f0224af8640 1 -- 172.21.15.67:0/25605404 shutdown_connections 2024-03-24T00:48:47.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.049+0000 7f0224af8640 1 -- 172.21.15.67:0/25605404 wait complete. 2024-03-24T00:48:47.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.051+0000 7f0224af8640 1 Processor -- start 2024-03-24T00:48:47.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.051+0000 7f0224af8640 1 -- start start 2024-03-24T00:48:47.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.051+0000 7f0224af8640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02200b5a60 0x7f02200ad2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:47.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.051+0000 7f0224af8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f022015dea0 0x7f0220160410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:47.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.051+0000 7f0224af8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f022015e280 0x7f0220160950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:47.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.051+0000 7f0224af8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f022006c7c0 con 0x7f022015e280 2024-03-24T00:48:47.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.051+0000 7f0224af8640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f022006c640 con 0x7f022015dea0 2024-03-24T00:48:47.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.051+0000 7f0224af8640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f022006c940 con 0x7f02200b5a60 2024-03-24T00:48:47.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.052+0000 7f021ed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f022015e280 0x7f0220160950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:47.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.052+0000 7f021ed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f022015e280 0x7f0220160950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34778/0 (socket says 172.21.15.67:34778) 2024-03-24T00:48:47.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.052+0000 7f021ed9d640 1 -- 172.21.15.67:0/3782421760 learned_addr learned my addr 172.21.15.67:0/3782421760 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:47.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.052+0000 7f021e59c640 1 --2- 172.21.15.67:0/3782421760 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02200b5a60 0x7f02200ad2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:47.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.052+0000 7f021dd9b640 1 --2- 172.21.15.67:0/3782421760 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f022015dea0 0x7f0220160410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:47.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.052+0000 7f021ed9d640 1 -- 172.21.15.67:0/3782421760 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02200b5a60 msgr2=0x7f02200ad2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:47.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.052+0000 7f021ed9d640 1 --2- 172.21.15.67:0/3782421760 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02200b5a60 0x7f02200ad2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:47.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.052+0000 7f021ed9d640 1 -- 172.21.15.67:0/3782421760 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f022015dea0 msgr2=0x7f0220160410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:47.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.052+0000 7f021ed9d640 1 --2- 172.21.15.67:0/3782421760 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f022015dea0 0x7f0220160410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:47.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.052+0000 7f021ed9d640 1 -- 172.21.15.67:0/3782421760 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02200ab930 con 0x7f022015e280 2024-03-24T00:48:47.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.052+0000 7f021ed9d640 1 --2- 172.21.15.67:0/3782421760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f022015e280 0x7f0220160950 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f021000e4f0 tx=0x7f021000e9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:47.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.052+0000 7f0216ffd640 1 -- 172.21.15.67:0/3782421760 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f021000f040 con 0x7f022015e280 2024-03-24T00:48:47.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.052+0000 7f0216ffd640 1 -- 172.21.15.67:0/3782421760 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0210004230 con 0x7f022015e280 2024-03-24T00:48:47.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.052+0000 7f0216ffd640 1 -- 172.21.15.67:0/3782421760 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0210003610 con 0x7f022015e280 2024-03-24T00:48:47.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.052+0000 7f0224af8640 1 -- 172.21.15.67:0/3782421760 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02200abc10 con 0x7f022015e280 2024-03-24T00:48:47.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.052+0000 7f0224af8640 1 -- 172.21.15.67:0/3782421760 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02200ac1d0 con 0x7f022015e280 2024-03-24T00:48:47.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.053+0000 7f0224af8640 1 -- 172.21.15.67:0/3782421760 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02200cf300 con 0x7f022015e280 2024-03-24T00:48:47.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.054+0000 7f0216ffd640 1 -- 172.21.15.67:0/3782421760 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f02100037b0 con 0x7f022015e280 2024-03-24T00:48:47.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.054+0000 7f0216ffd640 1 --2- 172.21.15.67:0/3782421760 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f020006bbd0 0x7f020006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:47.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.054+0000 7f0216ffd640 1 -- 172.21.15.67:0/3782421760 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f021008a400 con 0x7f022015e280 2024-03-24T00:48:47.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.054+0000 7f021e59c640 1 --2- 172.21.15.67:0/3782421760 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f020006bbd0 0x7f020006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:47.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.055+0000 7f021e59c640 1 --2- 172.21.15.67:0/3782421760 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f020006bbd0 0x7f020006e090 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f0218035000 tx=0x7f0218030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:47.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.059+0000 7f0216ffd640 1 -- 172.21.15.67:0/3782421760 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f02200cf300 con 0x7f022015e280 2024-03-24T00:48:47.154 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:48:47.157 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:48:47.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.244+0000 7f0224af8640 1 -- 172.21.15.67:0/3782421760 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f02200b96e0 con 0x7f022015e280 2024-03-24T00:48:47.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.244+0000 7f0216ffd640 1 -- 172.21.15.67:0/3782421760 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v111) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0210017020 con 0x7f022015e280 2024-03-24T00:48:47.244 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-24T00:48:47.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.246+0000 7f0224af8640 1 -- 172.21.15.67:0/3782421760 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f020006bbd0 msgr2=0x7f020006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:47.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.246+0000 7f0224af8640 1 --2- 172.21.15.67:0/3782421760 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f020006bbd0 0x7f020006e090 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f0218035000 tx=0x7f0218030000 comp rx=0 tx=0).stop 2024-03-24T00:48:47.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.246+0000 7f0224af8640 1 -- 172.21.15.67:0/3782421760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f022015e280 msgr2=0x7f0220160950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:47.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.246+0000 7f0224af8640 1 --2- 172.21.15.67:0/3782421760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f022015e280 0x7f0220160950 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f021000e4f0 tx=0x7f021000e9b0 comp rx=0 tx=0).stop 2024-03-24T00:48:47.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.246+0000 7f0224af8640 1 -- 172.21.15.67:0/3782421760 shutdown_connections 2024-03-24T00:48:47.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.246+0000 7f0224af8640 1 --2- 172.21.15.67:0/3782421760 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f020006bbd0 0x7f020006e090 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:47.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.246+0000 7f0224af8640 1 --2- 172.21.15.67:0/3782421760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f022015e280 0x7f0220160950 unknown :-1 s=CLOSED pgs=1664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:47.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.246+0000 7f0224af8640 1 --2- 172.21.15.67:0/3782421760 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f022015dea0 0x7f0220160410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:47.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.246+0000 7f0224af8640 1 --2- 172.21.15.67:0/3782421760 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02200b5a60 0x7f02200ad2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:47.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.246+0000 7f0224af8640 1 -- 172.21.15.67:0/3782421760 >> 172.21.15.67:0/3782421760 conn(0x7f0220072da0 msgr2=0x7f02200b8cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:47.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.246+0000 7f0224af8640 1 -- 172.21.15.67:0/3782421760 shutdown_connections 2024-03-24T00:48:47.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.246+0000 7f0224af8640 1 -- 172.21.15.67:0/3782421760 wait complete. 2024-03-24T00:48:47.257 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_orphan_scan' 2024-03-24T00:48:47.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.399+0000 7fb2a2e85640 1 Processor -- start 2024-03-24T00:48:47.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.399+0000 7fb2a2e85640 1 -- start start 2024-03-24T00:48:47.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.400+0000 7fb2a2e85640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb29c1445a0 0x7fb29c0cee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:47.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.400+0000 7fb2a2e85640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb29c0b48d0 0x7fb29c072550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:47.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.400+0000 7fb2a2e85640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb29c0b4d30 con 0x7fb29c06ec40 2024-03-24T00:48:47.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.400+0000 7fb2a2e85640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb29c0ce990 con 0x7fb29c1445a0 2024-03-24T00:48:47.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.400+0000 7fb2a2e85640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb29c0cddc0 con 0x7fb29c0b48d0 2024-03-24T00:48:47.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.400+0000 7fb29bfff640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb29c0b48d0 0x7fb29c072550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:47.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.400+0000 7fb2a0c21640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb29c1445a0 0x7fb29c0cee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:47.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.400+0000 7fb2a1422640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fb29c06ec40 0x7fb29c0b2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:41554/0 (socket says 172.21.15.67:41554) 2024-03-24T00:48:47.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.400+0000 7fb2a1422640 1 -- 172.21.15.67:0/2857658386 learned_addr learned my addr 172.21.15.67:0/2857658386 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:48:47.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.401+0000 7fb29b7fe640 1 -- 172.21.15.67:0/2857658386 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3319705633 0 0) 0x7fb294001490 con 0x7fb29c06ec40 2024-03-24T00:48:47.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.401+0000 7fb29b7fe640 1 -- 172.21.15.67:0/2857658386 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb28c003520 con 0x7fb29c06ec40 2024-03-24T00:48:47.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.401+0000 7fb2a0c21640 1 -- 172.21.15.67:0/2857658386 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb29c0b48d0 msgr2=0x7fb29c072550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:47.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.401+0000 7fb2a0c21640 1 --2- 172.21.15.67:0/2857658386 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb29c0b48d0 0x7fb29c072550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:47.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.401+0000 7fb2a0c21640 1 -- 172.21.15.67:0/2857658386 >> v1:172.21.15.72:6789/0 conn(0x7fb29c06ec40 legacy=0x7fb29c0b2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:47.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.401+0000 7fb2a0c21640 1 -- 172.21.15.67:0/2857658386 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb29c0ce060 con 0x7fb29c1445a0 2024-03-24T00:48:47.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.401+0000 7fb29bfff640 1 --2- 172.21.15.67:0/2857658386 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb29c0b48d0 0x7fb29c072550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:48:47.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.402+0000 7fb2a0c21640 1 --2- 172.21.15.67:0/2857658386 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb29c1445a0 0x7fb29c0cee20 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fb2900097a0 tx=0x7fb29002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee3e9260630177e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:47.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.402+0000 7fb29b7fe640 1 -- 172.21.15.67:0/2857658386 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb290035020 con 0x7fb29c1445a0 2024-03-24T00:48:47.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.402+0000 7fb29b7fe640 1 -- 172.21.15.67:0/2857658386 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb290031830 con 0x7fb29c1445a0 2024-03-24T00:48:47.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.402+0000 7fb29b7fe640 1 -- 172.21.15.67:0/2857658386 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb290031be0 con 0x7fb29c1445a0 2024-03-24T00:48:47.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.402+0000 7fb2a2e85640 1 -- 172.21.15.67:0/2857658386 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb29c1445a0 msgr2=0x7fb29c0cee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:47.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.402+0000 7fb2a2e85640 1 --2- 172.21.15.67:0/2857658386 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb29c1445a0 0x7fb29c0cee20 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fb2900097a0 tx=0x7fb29002b8d0 comp rx=0 tx=0).stop 2024-03-24T00:48:47.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.402+0000 7fb2a2e85640 1 -- 172.21.15.67:0/2857658386 shutdown_connections 2024-03-24T00:48:47.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.402+0000 7fb2a2e85640 1 --2- 172.21.15.67:0/2857658386 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb29c0b48d0 0x7fb29c072550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:47.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.402+0000 7fb2a2e85640 1 --2- 172.21.15.67:0/2857658386 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb29c1445a0 0x7fb29c0cee20 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:47.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.402+0000 7fb2a2e85640 1 -- 172.21.15.67:0/2857658386 >> 172.21.15.67:0/2857658386 conn(0x7fb29c0cf9e0 msgr2=0x7fb29c0cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:47.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.402+0000 7fb2a2e85640 1 -- 172.21.15.67:0/2857658386 shutdown_connections 2024-03-24T00:48:47.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.402+0000 7fb2a2e85640 1 -- 172.21.15.67:0/2857658386 wait complete. 2024-03-24T00:48:47.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.403+0000 7fb2a2e85640 1 Processor -- start 2024-03-24T00:48:47.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.404+0000 7fb2a2e85640 1 -- start start 2024-03-24T00:48:47.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.404+0000 7fb2a2e85640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb29c06ec40 0x7fb29c1687c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:47.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.404+0000 7fb2a2e85640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb29c0b48d0 0x7fb29c15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:47.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.404+0000 7fb2a2e85640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb29c1445a0 0x7fb29c071ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:47.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.404+0000 7fb2a2e85640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb29c06f0d0 con 0x7fb29c1445a0 2024-03-24T00:48:47.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.404+0000 7fb2a2e85640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb29c0ce620 con 0x7fb29c0b48d0 2024-03-24T00:48:47.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.404+0000 7fb2a2e85640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb29c06f250 con 0x7fb29c06ec40 2024-03-24T00:48:47.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.404+0000 7fb2a1422640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb29c1445a0 0x7fb29c071ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:47.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.405+0000 7fb2a1422640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb29c1445a0 0x7fb29c071ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34782/0 (socket says 172.21.15.67:34782) 2024-03-24T00:48:47.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.405+0000 7fb2a1422640 1 -- 172.21.15.67:0/4221218381 learned_addr learned my addr 172.21.15.67:0/4221218381 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:47.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.405+0000 7fb2a0c21640 1 --2- 172.21.15.67:0/4221218381 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb29c06ec40 0x7fb29c1687c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:47.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.405+0000 7fb29bfff640 1 --2- 172.21.15.67:0/4221218381 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb29c0b48d0 0x7fb29c15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:47.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.405+0000 7fb2a1422640 1 -- 172.21.15.67:0/4221218381 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb29c06ec40 msgr2=0x7fb29c1687c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:47.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.405+0000 7fb2a1422640 1 --2- 172.21.15.67:0/4221218381 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb29c06ec40 0x7fb29c1687c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:47.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.405+0000 7fb2a1422640 1 -- 172.21.15.67:0/4221218381 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb29c0b48d0 msgr2=0x7fb29c15cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:47.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.405+0000 7fb2a1422640 1 --2- 172.21.15.67:0/4221218381 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb29c0b48d0 0x7fb29c15cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:47.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.405+0000 7fb2a1422640 1 -- 172.21.15.67:0/4221218381 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb29c15d7c0 con 0x7fb29c1445a0 2024-03-24T00:48:47.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.405+0000 7fb2a1422640 1 --2- 172.21.15.67:0/4221218381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb29c1445a0 0x7fb29c071ad0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7fb29400cb80 tx=0x7fb29400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:47.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.405+0000 7fb2997fa640 1 -- 172.21.15.67:0/4221218381 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb294017070 con 0x7fb29c1445a0 2024-03-24T00:48:47.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.405+0000 7fb2997fa640 1 -- 172.21.15.67:0/4221218381 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb29400ad40 con 0x7fb29c1445a0 2024-03-24T00:48:47.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.405+0000 7fb2997fa640 1 -- 172.21.15.67:0/4221218381 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb294012620 con 0x7fb29c1445a0 2024-03-24T00:48:47.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.405+0000 7fb2a2e85640 1 -- 172.21.15.67:0/4221218381 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb29c0ac940 con 0x7fb29c1445a0 2024-03-24T00:48:47.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.405+0000 7fb2a2e85640 1 -- 172.21.15.67:0/4221218381 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb29c0acf00 con 0x7fb29c1445a0 2024-03-24T00:48:47.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.406+0000 7fb2a2e85640 1 -- 172.21.15.67:0/4221218381 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb29c04f400 con 0x7fb29c1445a0 2024-03-24T00:48:47.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.407+0000 7fb2997fa640 1 -- 172.21.15.67:0/4221218381 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb29401a070 con 0x7fb29c1445a0 2024-03-24T00:48:47.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.407+0000 7fb2997fa640 1 --2- 172.21.15.67:0/4221218381 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb28006bc00 0x7fb28006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:47.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.407+0000 7fb2997fa640 1 -- 172.21.15.67:0/4221218381 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fb29408a860 con 0x7fb29c1445a0 2024-03-24T00:48:47.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.407+0000 7fb2a0c21640 1 --2- 172.21.15.67:0/4221218381 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb28006bc00 0x7fb28006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:47.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.408+0000 7fb2a0c21640 1 --2- 172.21.15.67:0/4221218381 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb28006bc00 0x7fb28006e0c0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fb290036000 tx=0x7fb29003c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:47.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.411+0000 7fb2997fa640 1 -- 172.21.15.67:0/4221218381 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb29c0b4d30 con 0x7fb29c1445a0 2024-03-24T00:48:47.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:47.665+0000 7fb2a2e85640 1 -- 172.21.15.67:0/4221218381 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_orphan_scan"]} v 0) v1 -- 0x7fb29c155cd0 con 0x7fb29c1445a0 2024-03-24T00:48:48.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.624+0000 7fb2997fa640 1 -- 172.21.15.67:0/4221218381 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_orphan_scan"]}]=0 v463) v1 ==== 147+0+0 (secure 0 0 0) 0x7fb2940593e0 con 0x7fb29c1445a0 2024-03-24T00:48:48.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.626+0000 7fb2a2e85640 1 -- 172.21.15.67:0/4221218381 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb28006bc00 msgr2=0x7fb28006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:48.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.626+0000 7fb2a2e85640 1 --2- 172.21.15.67:0/4221218381 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb28006bc00 0x7fb28006e0c0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fb290036000 tx=0x7fb29003c040 comp rx=0 tx=0).stop 2024-03-24T00:48:48.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.627+0000 7fb2a2e85640 1 -- 172.21.15.67:0/4221218381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb29c1445a0 msgr2=0x7fb29c071ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:48.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.627+0000 7fb2a2e85640 1 --2- 172.21.15.67:0/4221218381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb29c1445a0 0x7fb29c071ad0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7fb29400cb80 tx=0x7fb29400a430 comp rx=0 tx=0).stop 2024-03-24T00:48:48.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.627+0000 7fb2a2e85640 1 -- 172.21.15.67:0/4221218381 shutdown_connections 2024-03-24T00:48:48.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.627+0000 7fb2a2e85640 1 --2- 172.21.15.67:0/4221218381 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb28006bc00 0x7fb28006e0c0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:48.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.627+0000 7fb2a2e85640 1 --2- 172.21.15.67:0/4221218381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb29c1445a0 0x7fb29c071ad0 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:48.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.627+0000 7fb2a2e85640 1 --2- 172.21.15.67:0/4221218381 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb29c0b48d0 0x7fb29c15cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:48.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.627+0000 7fb2a2e85640 1 --2- 172.21.15.67:0/4221218381 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb29c06ec40 0x7fb29c1687c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:48.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.627+0000 7fb2a2e85640 1 -- 172.21.15.67:0/4221218381 >> 172.21.15.67:0/4221218381 conn(0x7fb29c0cf9e0 msgr2=0x7fb29c0cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:48.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.627+0000 7fb2a2e85640 1 -- 172.21.15.67:0/4221218381 shutdown_connections 2024-03-24T00:48:48.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.627+0000 7fb2a2e85640 1 -- 172.21.15.67:0/4221218381 wait complete. 2024-03-24T00:48:48.640 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-24T00:48:48.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.751+0000 7fa0c6468640 1 Processor -- start 2024-03-24T00:48:48.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.751+0000 7fa0c6468640 1 -- start start 2024-03-24T00:48:48.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.752+0000 7fa0c6468640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa0c0144670 0x7fa0c00b3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:48.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.752+0000 7fa0c6468640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa0c006ec60 0x7fa0c006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:48.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.752+0000 7fa0c6468640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0c00b2ab0 con 0x7fa0c00b4a00 2024-03-24T00:48:48.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.752+0000 7fa0c6468640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0c006f610 con 0x7fa0c0144670 2024-03-24T00:48:48.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.752+0000 7fa0c6468640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c00b2d90 con 0x7fa0c006ec60 2024-03-24T00:48:48.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.752+0000 7fa0bffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa0c0144670 0x7fa0c00b3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:48.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.752+0000 7fa0bf7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa0c00b4a00 0x7fa0c0072500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48592/0 (socket says 172.21.15.67:48592) 2024-03-24T00:48:48.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.752+0000 7fa0bf7fe640 1 -- 172.21.15.67:0/3130536329 learned_addr learned my addr 172.21.15.67:0/3130536329 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:48:48.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.752+0000 7fa0bffff640 1 -- 172.21.15.67:0/3130536329 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa0c006ec60 msgr2=0x7fa0c006f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:48.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.752+0000 7fa0c4a05640 1 --2- 172.21.15.67:0/3130536329 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa0c006ec60 0x7fa0c006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:48.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.752+0000 7fa0beffd640 1 -- 172.21.15.67:0/3130536329 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2582739078 0 0) 0x7fa0c00b2ab0 con 0x7fa0c00b4a00 2024-03-24T00:48:48.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.752+0000 7fa0bffff640 1 --2- 172.21.15.67:0/3130536329 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa0c006ec60 0x7fa0c006f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:48.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.752+0000 7fa0bffff640 1 -- 172.21.15.67:0/3130536329 >> v1:172.21.15.67:6789/0 conn(0x7fa0c00b4a00 legacy=0x7fa0c0072500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:48.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.752+0000 7fa0bffff640 1 -- 172.21.15.67:0/3130536329 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c00b3070 con 0x7fa0c0144670 2024-03-24T00:48:48.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.753+0000 7fa0bffff640 1 --2- 172.21.15.67:0/3130536329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0c0144670 0x7fa0c00b3870 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7fa0b40097a0 tx=0x7fa0b402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d8bcb28135ed32b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:48.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.753+0000 7fa0beffd640 1 -- 172.21.15.67:0/3130536329 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b4035020 con 0x7fa0c0144670 2024-03-24T00:48:48.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.753+0000 7fa0beffd640 1 -- 172.21.15.67:0/3130536329 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0b4031830 con 0x7fa0c0144670 2024-03-24T00:48:48.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.753+0000 7fa0beffd640 1 -- 172.21.15.67:0/3130536329 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b4031c00 con 0x7fa0c0144670 2024-03-24T00:48:48.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.754+0000 7fa0c6468640 1 -- 172.21.15.67:0/3130536329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0c0144670 msgr2=0x7fa0c00b3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:48.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.754+0000 7fa0c6468640 1 --2- 172.21.15.67:0/3130536329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0c0144670 0x7fa0c00b3870 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7fa0b40097a0 tx=0x7fa0b402b8d0 comp rx=0 tx=0).stop 2024-03-24T00:48:48.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.754+0000 7fa0c6468640 1 -- 172.21.15.67:0/3130536329 shutdown_connections 2024-03-24T00:48:48.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.754+0000 7fa0c6468640 1 --2- 172.21.15.67:0/3130536329 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa0c006ec60 0x7fa0c006f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:48.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.754+0000 7fa0c6468640 1 --2- 172.21.15.67:0/3130536329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0c0144670 0x7fa0c00b3870 unknown :-1 s=CLOSED pgs=1667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:48.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.754+0000 7fa0c6468640 1 -- 172.21.15.67:0/3130536329 >> 172.21.15.67:0/3130536329 conn(0x7fa0c00cf9e0 msgr2=0x7fa0c00cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:48.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.754+0000 7fa0c6468640 1 -- 172.21.15.67:0/3130536329 shutdown_connections 2024-03-24T00:48:48.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.754+0000 7fa0c6468640 1 -- 172.21.15.67:0/3130536329 wait complete. 2024-03-24T00:48:48.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.755+0000 7fa0c6468640 1 Processor -- start 2024-03-24T00:48:48.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.755+0000 7fa0c6468640 1 -- start start 2024-03-24T00:48:48.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.756+0000 7fa0c6468640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0c006ec60 0x7fa0c0142690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:48.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.756+0000 7fa0c6468640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa0c00b4a00 0x7fa0c01f06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:48.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.756+0000 7fa0c6468640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa0c0144670 0x7fa0c013d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:48.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.756+0000 7fa0c6468640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c00cde30 con 0x7fa0c006ec60 2024-03-24T00:48:48.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.756+0000 7fa0c6468640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c0072d30 con 0x7fa0c0144670 2024-03-24T00:48:48.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.756+0000 7fa0c6468640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c00cdfb0 con 0x7fa0c00b4a00 2024-03-24T00:48:48.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.756+0000 7fa0bffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0c006ec60 0x7fa0c0142690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:48.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.756+0000 7fa0bffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0c006ec60 0x7fa0c0142690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34790/0 (socket says 172.21.15.67:34790) 2024-03-24T00:48:48.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.756+0000 7fa0bffff640 1 -- 172.21.15.67:0/441614974 learned_addr learned my addr 172.21.15.67:0/441614974 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:48.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.756+0000 7fa0bffff640 1 -- 172.21.15.67:0/441614974 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa0c00b4a00 msgr2=0x7fa0c01f06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:48.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.756+0000 7fa0bffff640 1 --2- 172.21.15.67:0/441614974 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa0c00b4a00 0x7fa0c01f06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:48.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.756+0000 7fa0bffff640 1 -- 172.21.15.67:0/441614974 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa0c0144670 msgr2=0x7fa0c013d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:48.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.756+0000 7fa0bffff640 1 --2- 172.21.15.67:0/441614974 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa0c0144670 0x7fa0c013d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:48.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.756+0000 7fa0bffff640 1 -- 172.21.15.67:0/441614974 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c013d920 con 0x7fa0c006ec60 2024-03-24T00:48:48.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.756+0000 7fa0bffff640 1 --2- 172.21.15.67:0/441614974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0c006ec60 0x7fa0c0142690 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7fa0b40097a0 tx=0x7fa0b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:48.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.756+0000 7fa0bcff9640 1 -- 172.21.15.67:0/441614974 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b4041070 con 0x7fa0c006ec60 2024-03-24T00:48:48.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.756+0000 7fa0bcff9640 1 -- 172.21.15.67:0/441614974 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0b4004230 con 0x7fa0c006ec60 2024-03-24T00:48:48.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.757+0000 7fa0c6468640 1 -- 172.21.15.67:0/441614974 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0c013a540 con 0x7fa0c006ec60 2024-03-24T00:48:48.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.757+0000 7fa0bcff9640 1 -- 172.21.15.67:0/441614974 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b4035040 con 0x7fa0c006ec60 2024-03-24T00:48:48.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.757+0000 7fa0c6468640 1 -- 172.21.15.67:0/441614974 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0c01f3540 con 0x7fa0c006ec60 2024-03-24T00:48:48.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.757+0000 7fa0c6468640 1 -- 172.21.15.67:0/441614974 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0c004f400 con 0x7fa0c006ec60 2024-03-24T00:48:48.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.758+0000 7fa0bcff9640 1 -- 172.21.15.67:0/441614974 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa0b4038020 con 0x7fa0c006ec60 2024-03-24T00:48:48.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.758+0000 7fa0bcff9640 1 --2- 172.21.15.67:0/441614974 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa0ac06bc00 0x7fa0ac06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:48.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.759+0000 7fa0bcff9640 1 -- 172.21.15.67:0/441614974 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fa0b40ac3d0 con 0x7fa0c006ec60 2024-03-24T00:48:48.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.759+0000 7fa0bf7fe640 1 --2- 172.21.15.67:0/441614974 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa0ac06bc00 0x7fa0ac06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:48.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.759+0000 7fa0bf7fe640 1 --2- 172.21.15.67:0/441614974 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa0ac06bc00 0x7fa0ac06e0c0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fa0b00011e0 tx=0x7fa0b0007950 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:48.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.763+0000 7fa0bcff9640 1 -- 172.21.15.67:0/441614974 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa0b407afe0 con 0x7fa0c006ec60 2024-03-24T00:48:48.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.946+0000 7fa0c6468640 1 -- 172.21.15.67:0/441614974 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fa0c015e1c0 con 0x7fa0c006ec60 2024-03-24T00:48:48.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.950+0000 7fa0bcff9640 1 -- 172.21.15.67:0/441614974 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa0b407a8e0 con 0x7fa0c006ec60 2024-03-24T00:48:48.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.952+0000 7fa0c6468640 1 -- 172.21.15.67:0/441614974 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa0ac06bc00 msgr2=0x7fa0ac06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:48.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.952+0000 7fa0c6468640 1 --2- 172.21.15.67:0/441614974 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa0ac06bc00 0x7fa0ac06e0c0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fa0b00011e0 tx=0x7fa0b0007950 comp rx=0 tx=0).stop 2024-03-24T00:48:48.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.952+0000 7fa0c6468640 1 -- 172.21.15.67:0/441614974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0c006ec60 msgr2=0x7fa0c0142690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:48.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.952+0000 7fa0c6468640 1 --2- 172.21.15.67:0/441614974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0c006ec60 0x7fa0c0142690 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7fa0b40097a0 tx=0x7fa0b40046c0 comp rx=0 tx=0).stop 2024-03-24T00:48:48.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.952+0000 7fa0c6468640 1 -- 172.21.15.67:0/441614974 shutdown_connections 2024-03-24T00:48:48.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.952+0000 7fa0c6468640 1 --2- 172.21.15.67:0/441614974 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa0ac06bc00 0x7fa0ac06e0c0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:48.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.952+0000 7fa0c6468640 1 --2- 172.21.15.67:0/441614974 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa0c0144670 0x7fa0c013d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:48.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.952+0000 7fa0c6468640 1 --2- 172.21.15.67:0/441614974 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa0c00b4a00 0x7fa0c01f06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:48.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.952+0000 7fa0c6468640 1 --2- 172.21.15.67:0/441614974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0c006ec60 0x7fa0c0142690 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:48.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.952+0000 7fa0c6468640 1 -- 172.21.15.67:0/441614974 >> 172.21.15.67:0/441614974 conn(0x7fa0c00cf9e0 msgr2=0x7fa0c00ce3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:48.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.953+0000 7fa0c6468640 1 -- 172.21.15.67:0/441614974 shutdown_connections 2024-03-24T00:48:48.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:48.953+0000 7fa0c6468640 1 -- 172.21.15.67:0/441614974 wait complete. 2024-03-24T00:48:48.963 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-24T00:48:48.963 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:48:49.003 DEBUG:teuthology.orchestra.run.smithi072:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:48:49.055 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-24T00:48:49.055 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:48:49.055 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:48:49.070 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056375,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15771 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T00:48:49.070 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:48:49.070 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:48:49.087 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17590 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248808k,nr_inodes=812202,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:48:49.087 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:48:49.087 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:48:49.125 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056375,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15771 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T00:48:49.125 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:48:49.125 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:48:49.144 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17590 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248808k,nr_inodes=812202,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:48:49.144 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-24T00:48:49.144 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:48:49.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.265+0000 7f1b9c1e6640 1 Processor -- start 2024-03-24T00:48:49.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.265+0000 7f1b9c1e6640 1 -- start start 2024-03-24T00:48:49.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.266+0000 7f1b9c1e6640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1b9406ca40 0x7f1b9406ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:49.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.266+0000 7f1b9c1e6640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b9406d360 con 0x7f1b940b18f0 2024-03-24T00:48:49.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.266+0000 7f1b9c1e6640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b940b2590 con 0x7f1b940d0440 2024-03-24T00:48:49.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.266+0000 7f1b9c1e6640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b940b2870 con 0x7f1b9406ca40 2024-03-24T00:48:49.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.266+0000 7f1b99781640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1b940b18f0 0x7f1b940b1d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48596/0 (socket says 172.21.15.67:48596) 2024-03-24T00:48:49.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.266+0000 7f1b99781640 1 -- 172.21.15.67:0/3413609529 learned_addr learned my addr 172.21.15.67:0/3413609529 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:48:49.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.266+0000 7f1b99f82640 1 --2- 172.21.15.67:0/3413609529 >> v2:172.21.15.72:3300/0 conn(0x7f1b9406ca40 0x7f1b9406ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:49.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.267+0000 7f1b98f80640 1 -- 172.21.15.67:0/3413609529 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2508376686 0 0) 0x7f1b9406d360 con 0x7f1b940b18f0 2024-03-24T00:48:49.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.267+0000 7f1b98f80640 1 -- 172.21.15.67:0/3413609529 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b7c003520 con 0x7f1b940b18f0 2024-03-24T00:48:49.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.267+0000 7f1b98f80640 1 -- 172.21.15.67:0/3413609529 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2915029331 0 0) 0x7f1b940b2590 con 0x7f1b940d0440 2024-03-24T00:48:49.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.267+0000 7f1b98f80640 1 -- 172.21.15.67:0/3413609529 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b9406d360 con 0x7f1b940d0440 2024-03-24T00:48:49.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.267+0000 7f1b98f80640 1 -- 172.21.15.67:0/3413609529 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4105098603 0 0) 0x7f1b7c003520 con 0x7f1b940b18f0 2024-03-24T00:48:49.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.267+0000 7f1b98f80640 1 -- 172.21.15.67:0/3413609529 >> v2:172.21.15.72:3300/0 conn(0x7f1b9406ca40 msgr2=0x7f1b9406ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:49.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.267+0000 7f1b98f80640 1 --2- 172.21.15.67:0/3413609529 >> v2:172.21.15.72:3300/0 conn(0x7f1b9406ca40 0x7f1b9406ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:49.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.267+0000 7f1b98f80640 1 -- 172.21.15.67:0/3413609529 >> v1:172.21.15.72:6789/0 conn(0x7f1b940d0440 legacy=0x7f1b940d08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:49.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.267+0000 7f1b98f80640 1 -- 172.21.15.67:0/3413609529 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b940b2b50 con 0x7f1b940b18f0 2024-03-24T00:48:49.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.267+0000 7f1b98f80640 1 -- 172.21.15.67:0/3413609529 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f1b88001670 con 0x7f1b940b18f0 2024-03-24T00:48:49.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.267+0000 7f1b98f80640 1 -- 172.21.15.67:0/3413609529 >> v1:172.21.15.67:6789/0 conn(0x7f1b940b18f0 legacy=0x7f1b940b1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:49.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.267+0000 7f1b98f80640 1 --2- 172.21.15.67:0/3413609529 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b7c003b40 0x7f1b7c003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:49.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.267+0000 7f1b98f80640 1 --2- 172.21.15.67:0/3413609529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b7c004800 0x7f1b7c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:49.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.267+0000 7f1b98f80640 1 --2- 172.21.15.67:0/3413609529 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1b7c0051c0 0x7f1b7c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:49.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.267+0000 7f1b98f80640 1 -- 172.21.15.67:0/3413609529 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b940b2870 con 0x7f1b7c004800 2024-03-24T00:48:49.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.267+0000 7f1b98f80640 1 -- 172.21.15.67:0/3413609529 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7c00c040 con 0x7f1b7c003b40 2024-03-24T00:48:49.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.267+0000 7f1b98f80640 1 -- 172.21.15.67:0/3413609529 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7c00c240 con 0x7f1b7c0051c0 2024-03-24T00:48:49.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.267+0000 7f1b99781640 1 --2- 172.21.15.67:0/3413609529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b7c004800 0x7f1b7c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:49.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.267+0000 7f1b99f82640 1 --2- 172.21.15.67:0/3413609529 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b7c003b40 0x7f1b7c003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:49.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.267+0000 7f1b9a783640 1 --2- 172.21.15.67:0/3413609529 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1b7c0051c0 0x7f1b7c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:49.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.267+0000 7f1b99781640 1 -- 172.21.15.67:0/3413609529 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1b7c0051c0 msgr2=0x7f1b7c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:49.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.267+0000 7f1b99781640 1 --2- 172.21.15.67:0/3413609529 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1b7c0051c0 0x7f1b7c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:49.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.267+0000 7f1b99781640 1 -- 172.21.15.67:0/3413609529 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b7c003b40 msgr2=0x7f1b7c003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:49.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.267+0000 7f1b99781640 1 --2- 172.21.15.67:0/3413609529 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b7c003b40 0x7f1b7c003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:49.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.267+0000 7f1b99781640 1 -- 172.21.15.67:0/3413609529 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b7c003520 con 0x7f1b7c004800 2024-03-24T00:48:49.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.267+0000 7f1b9a783640 1 -- 172.21.15.67:0/3413609529 reap_dead start 2024-03-24T00:48:49.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.268+0000 7f1b99781640 1 --2- 172.21.15.67:0/3413609529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b7c004800 0x7f1b7c004c80 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f1b8800c1c0 tx=0x7f1b7c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac8cf94504511b21 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:49.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.268+0000 7f1b98f80640 1 -- 172.21.15.67:0/3413609529 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b88007cc0 con 0x7f1b7c004800 2024-03-24T00:48:49.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.268+0000 7f1b98f80640 1 -- 172.21.15.67:0/3413609529 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b88030070 con 0x7f1b7c004800 2024-03-24T00:48:49.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.268+0000 7f1b98f80640 1 -- 172.21.15.67:0/3413609529 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b88005d70 con 0x7f1b7c004800 2024-03-24T00:48:49.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.269+0000 7f1b9c1e6640 1 -- 172.21.15.67:0/3413609529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b7c004800 msgr2=0x7f1b7c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:49.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.269+0000 7f1b9c1e6640 1 --2- 172.21.15.67:0/3413609529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b7c004800 0x7f1b7c004c80 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f1b8800c1c0 tx=0x7f1b7c005830 comp rx=0 tx=0).stop 2024-03-24T00:48:49.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.269+0000 7f1b9c1e6640 1 -- 172.21.15.67:0/3413609529 shutdown_connections 2024-03-24T00:48:49.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.269+0000 7f1b9c1e6640 1 --2- 172.21.15.67:0/3413609529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b7c004800 0x7f1b7c004c80 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:49.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.269+0000 7f1b9c1e6640 1 -- 172.21.15.67:0/3413609529 >> 172.21.15.67:0/3413609529 conn(0x7f1b940d1bf0 msgr2=0x7f1b940d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:49.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.269+0000 7f1b9c1e6640 1 -- 172.21.15.67:0/3413609529 shutdown_connections 2024-03-24T00:48:49.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.269+0000 7f1b9c1e6640 1 -- 172.21.15.67:0/3413609529 wait complete. 2024-03-24T00:48:49.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.270+0000 7f1b9c1e6640 1 Processor -- start 2024-03-24T00:48:49.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.270+0000 7f1b9c1e6640 1 -- start start 2024-03-24T00:48:49.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.271+0000 7f1b9c1e6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b7c004800 0x7f1b940c9200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:49.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.271+0000 7f1b9c1e6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1b941f34b0 0x7f1b940c81b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:49.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.271+0000 7f1b9c1e6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b940ce590 0x7f1b940cea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:49.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.271+0000 7f1b9c1e6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b9415e370 con 0x7f1b940ce590 2024-03-24T00:48:49.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.271+0000 7f1b9c1e6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b940cff20 con 0x7f1b7c004800 2024-03-24T00:48:49.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.271+0000 7f1b9c1e6640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b9415e4f0 con 0x7f1b941f34b0 2024-03-24T00:48:49.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.271+0000 7f1b9a783640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b940ce590 0x7f1b940cea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:49.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.271+0000 7f1b9a783640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b940ce590 0x7f1b940cea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34806/0 (socket says 172.21.15.67:34806) 2024-03-24T00:48:49.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.271+0000 7f1b9a783640 1 -- 172.21.15.67:0/1495555034 learned_addr learned my addr 172.21.15.67:0/1495555034 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:49.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.271+0000 7f1b99f82640 1 --2- 172.21.15.67:0/1495555034 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b7c004800 0x7f1b940c9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:49.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.271+0000 7f1b99781640 1 --2- 172.21.15.67:0/1495555034 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1b941f34b0 0x7f1b940c81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:49.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.271+0000 7f1b9a783640 1 -- 172.21.15.67:0/1495555034 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1b941f34b0 msgr2=0x7f1b940c81b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:49.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.271+0000 7f1b9a783640 1 --2- 172.21.15.67:0/1495555034 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1b941f34b0 0x7f1b940c81b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:49.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.271+0000 7f1b9a783640 1 -- 172.21.15.67:0/1495555034 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b7c004800 msgr2=0x7f1b940c9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:49.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.271+0000 7f1b9a783640 1 --2- 172.21.15.67:0/1495555034 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b7c004800 0x7f1b940c9200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:49.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.271+0000 7f1b9a783640 1 -- 172.21.15.67:0/1495555034 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b940c9870 con 0x7f1b940ce590 2024-03-24T00:48:49.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.271+0000 7f1b9a783640 1 --2- 172.21.15.67:0/1495555034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b940ce590 0x7f1b940cea10 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f1b8400fae0 tx=0x7f1b8400fee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:49.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.272+0000 7f1b927fc640 1 -- 172.21.15.67:0/1495555034 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b8401a070 con 0x7f1b940ce590 2024-03-24T00:48:49.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.272+0000 7f1b927fc640 1 -- 172.21.15.67:0/1495555034 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b84006d60 con 0x7f1b940ce590 2024-03-24T00:48:49.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.272+0000 7f1b9c1e6640 1 -- 172.21.15.67:0/1495555034 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b940c9b50 con 0x7f1b940ce590 2024-03-24T00:48:49.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.272+0000 7f1b927fc640 1 -- 172.21.15.67:0/1495555034 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b84003bd0 con 0x7f1b940ce590 2024-03-24T00:48:49.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.272+0000 7f1b9c1e6640 1 -- 172.21.15.67:0/1495555034 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b940ca110 con 0x7f1b940ce590 2024-03-24T00:48:49.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.273+0000 7f1b9c1e6640 1 -- 172.21.15.67:0/1495555034 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b9404f400 con 0x7f1b940ce590 2024-03-24T00:48:49.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.273+0000 7f1b927fc640 1 -- 172.21.15.67:0/1495555034 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1b8401d070 con 0x7f1b940ce590 2024-03-24T00:48:49.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.273+0000 7f1b927fc640 1 --2- 172.21.15.67:0/1495555034 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1b8006bbd0 0x7f1b8006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:49.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.274+0000 7f1b927fc640 1 -- 172.21.15.67:0/1495555034 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f1b84089e40 con 0x7f1b940ce590 2024-03-24T00:48:49.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.274+0000 7f1b99f82640 1 --2- 172.21.15.67:0/1495555034 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1b8006bbd0 0x7f1b8006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:49.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.274+0000 7f1b99f82640 1 --2- 172.21.15.67:0/1495555034 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1b8006bbd0 0x7f1b8006e090 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f1b94050be0 tx=0x7f1b8c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:49.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.278+0000 7f1b927fc640 1 -- 172.21.15.67:0/1495555034 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1b84058490 con 0x7f1b940ce590 2024-03-24T00:48:49.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.498+0000 7f1b9c1e6640 1 -- 172.21.15.67:0/1495555034 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b940cef50 con 0x7f1b940ce590 2024-03-24T00:48:49.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.499+0000 7f1b927fc640 1 -- 172.21.15.67:0/1495555034 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3696 (secure 0 0 0) 0x7f1b8405c530 con 0x7f1b940ce590 2024-03-24T00:48:49.499 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:48:49.499 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":82,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6902,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3672294289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3672294289},{"type":"v1","addr":"172.21.15.67:6835","nonce":3672294289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":6908,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/1286271878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":1286271878},{"type":"v1","addr":"172.21.15.72:6836","nonce":1286271878}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":6911,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/140622129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":140622129},{"type":"v1","addr":"172.21.15.72:6837","nonce":140622129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":9054,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/3799780628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3799780628},{"type":"v1","addr":"172.21.15.67:6833","nonce":3799780628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[]} 2024-03-24T00:48:49.500 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 82 2024-03-24T00:48:49.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.501+0000 7f1b9c1e6640 1 -- 172.21.15.67:0/1495555034 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1b8006bbd0 msgr2=0x7f1b8006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:49.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.501+0000 7f1b9c1e6640 1 --2- 172.21.15.67:0/1495555034 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1b8006bbd0 0x7f1b8006e090 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f1b94050be0 tx=0x7f1b8c009190 comp rx=0 tx=0).stop 2024-03-24T00:48:49.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.501+0000 7f1b9c1e6640 1 -- 172.21.15.67:0/1495555034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b940ce590 msgr2=0x7f1b940cea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:49.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.501+0000 7f1b9c1e6640 1 --2- 172.21.15.67:0/1495555034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b940ce590 0x7f1b940cea10 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f1b8400fae0 tx=0x7f1b8400fee0 comp rx=0 tx=0).stop 2024-03-24T00:48:49.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.501+0000 7f1b9c1e6640 1 -- 172.21.15.67:0/1495555034 shutdown_connections 2024-03-24T00:48:49.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.501+0000 7f1b9c1e6640 1 --2- 172.21.15.67:0/1495555034 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1b8006bbd0 0x7f1b8006e090 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:49.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.501+0000 7f1b9c1e6640 1 --2- 172.21.15.67:0/1495555034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b940ce590 0x7f1b940cea10 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:49.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.501+0000 7f1b9c1e6640 1 --2- 172.21.15.67:0/1495555034 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1b941f34b0 0x7f1b940c81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:49.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.501+0000 7f1b9c1e6640 1 --2- 172.21.15.67:0/1495555034 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b7c004800 0x7f1b940c9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:49.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.501+0000 7f1b9c1e6640 1 -- 172.21.15.67:0/1495555034 >> 172.21.15.67:0/1495555034 conn(0x7f1b940d1bf0 msgr2=0x7f1b94070d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:49.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.501+0000 7f1b9c1e6640 1 -- 172.21.15.67:0/1495555034 shutdown_connections 2024-03-24T00:48:49.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.501+0000 7f1b9c1e6640 1 -- 172.21.15.67:0/1495555034 wait complete. 2024-03-24T00:48:49.512 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-24T00:48:49.512 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-24T00:48:49.512 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-24T00:48:49.513 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:48:49.513 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-24T00:48:49.513 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-24T00:48:49.513 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-24T00:48:49.513 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:48:49.513 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-24T00:48:49.514 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-24T00:48:49.514 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-24T00:48:49.514 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:48:49.514 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-24T00:48:49.514 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-24T00:48:49.514 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-24T00:48:49.514 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:48:49.521 INFO:tasks.ceph.mds.a:Stopped 2024-03-24T00:48:49.522 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-24T00:48:49.523 INFO:tasks.ceph.mds.c:Stopped 2024-03-24T00:48:49.523 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-24T00:48:49.526 INFO:tasks.ceph.mds.b:Stopped 2024-03-24T00:48:49.526 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-24T00:48:49.527 INFO:tasks.ceph.mds.d:Stopped 2024-03-24T00:48:49.527 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-24T00:48:49.564 INFO:tasks.ceph.mds.a:Started 2024-03-24T00:48:49.564 DEBUG:teuthology.parallel:result is None 2024-03-24T00:48:49.565 INFO:tasks.ceph.mds.c:Started 2024-03-24T00:48:49.565 DEBUG:teuthology.parallel:result is None 2024-03-24T00:48:49.568 INFO:tasks.ceph.mds.b:Started 2024-03-24T00:48:49.569 DEBUG:teuthology.parallel:result is None 2024-03-24T00:48:49.569 INFO:tasks.ceph.mds.d:Started 2024-03-24T00:48:49.569 DEBUG:teuthology.parallel:result is None 2024-03-24T00:48:49.570 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-24T00:48:49.641 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-03-24T00:48:49.641 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-03-24T00:48:49.651 INFO:tasks.ceph.mds.b.smithi072.stdout:starting mds.b at 2024-03-24T00:48:49.652 INFO:tasks.ceph.mds.d.smithi072.stdout:starting mds.d at 2024-03-24T00:48:49.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.677+0000 7f80214f8640 1 Processor -- start 2024-03-24T00:48:49.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.677+0000 7f80214f8640 1 -- start start 2024-03-24T00:48:49.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.677+0000 7f80214f8640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f801c162010 0x7f801c0b6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:49.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.677+0000 7f80214f8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f801c0ce310 0x7f801c0ce760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:49.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.677+0000 7f80214f8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f801c0d4470 con 0x7f801c1623f0 2024-03-24T00:48:49.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.677+0000 7f80214f8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f801c0ceca0 con 0x7f801c0ce310 2024-03-24T00:48:49.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.677+0000 7f80214f8640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f801c0d4220 con 0x7f801c162010 2024-03-24T00:48:49.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.678+0000 7f801b7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f801c0ce310 0x7f801c0ce760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:49.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.678+0000 7f801a7fc640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f801c1623f0 0x7f801c0d0d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48606/0 (socket says 172.21.15.67:48606) 2024-03-24T00:48:49.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.678+0000 7f801a7fc640 1 -- 172.21.15.67:0/1927413249 learned_addr learned my addr 172.21.15.67:0/1927413249 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:48:49.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.678+0000 7f801affd640 1 --2- 172.21.15.67:0/1927413249 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f801c162010 0x7f801c0b6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:49.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.678+0000 7f801b7fe640 1 -- 172.21.15.67:0/1927413249 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f801c162010 msgr2=0x7f801c0b6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:49.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.678+0000 7f801b7fe640 1 --2- 172.21.15.67:0/1927413249 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f801c162010 0x7f801c0b6ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:49.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.678+0000 7f801b7fe640 1 -- 172.21.15.67:0/1927413249 >> v1:172.21.15.67:6789/0 conn(0x7f801c1623f0 legacy=0x7f801c0d0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:49.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.678+0000 7f801b7fe640 1 -- 172.21.15.67:0/1927413249 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f801c0d4750 con 0x7f801c0ce310 2024-03-24T00:48:49.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.678+0000 7f801affd640 1 --2- 172.21.15.67:0/1927413249 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f801c162010 0x7f801c0b6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:48:49.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.678+0000 7f8019ffb640 1 -- 172.21.15.67:0/1927413249 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1064555648 0 0) 0x7f801c0d4470 con 0x7f801c1623f0 2024-03-24T00:48:49.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.679+0000 7f801b7fe640 1 --2- 172.21.15.67:0/1927413249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f801c0ce310 0x7f801c0ce760 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f8014009d90 tx=0x7f801402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3ae551f45f6d12b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:49.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.679+0000 7f8019ffb640 1 -- 172.21.15.67:0/1927413249 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8014039070 con 0x7f801c0ce310 2024-03-24T00:48:49.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.679+0000 7f8019ffb640 1 -- 172.21.15.67:0/1927413249 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f801402cbd0 con 0x7f801c0ce310 2024-03-24T00:48:49.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.679+0000 7f8019ffb640 1 -- 172.21.15.67:0/1927413249 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8014034440 con 0x7f801c0ce310 2024-03-24T00:48:49.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.679+0000 7f80214f8640 1 -- 172.21.15.67:0/1927413249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f801c0ce310 msgr2=0x7f801c0ce760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:49.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.679+0000 7f80214f8640 1 --2- 172.21.15.67:0/1927413249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f801c0ce310 0x7f801c0ce760 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f8014009d90 tx=0x7f801402be20 comp rx=0 tx=0).stop 2024-03-24T00:48:49.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.679+0000 7f80214f8640 1 -- 172.21.15.67:0/1927413249 shutdown_connections 2024-03-24T00:48:49.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.679+0000 7f80214f8640 1 --2- 172.21.15.67:0/1927413249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f801c0ce310 0x7f801c0ce760 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:49.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.679+0000 7f80214f8640 1 --2- 172.21.15.67:0/1927413249 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f801c162010 0x7f801c0b6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:49.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.679+0000 7f80214f8640 1 -- 172.21.15.67:0/1927413249 >> 172.21.15.67:0/1927413249 conn(0x7f801c072b60 msgr2=0x7f801c072f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:49.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.679+0000 7f80214f8640 1 -- 172.21.15.67:0/1927413249 shutdown_connections 2024-03-24T00:48:49.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.679+0000 7f80214f8640 1 -- 172.21.15.67:0/1927413249 wait complete. 2024-03-24T00:48:49.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.681+0000 7f80214f8640 1 Processor -- start 2024-03-24T00:48:49.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.681+0000 7f80214f8640 1 -- start start 2024-03-24T00:48:49.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.681+0000 7f80214f8640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f801c0ce310 0x7f801c14d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:49.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.681+0000 7f80214f8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f801c162010 0x7f801c14e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:49.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.681+0000 7f80214f8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f801c1623f0 0x7f801c165fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:49.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.681+0000 7f80214f8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f801c0d1ae0 con 0x7f801c1623f0 2024-03-24T00:48:49.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.681+0000 7f80214f8640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f801c0d1890 con 0x7f801c162010 2024-03-24T00:48:49.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.681+0000 7f80214f8640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f801c0d1410 con 0x7f801c0ce310 2024-03-24T00:48:49.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.682+0000 7f801b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f801c1623f0 0x7f801c165fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:49.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.682+0000 7f801b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f801c1623f0 0x7f801c165fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34834/0 (socket says 172.21.15.67:34834) 2024-03-24T00:48:49.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.682+0000 7f801b7fe640 1 -- 172.21.15.67:0/3751234839 learned_addr learned my addr 172.21.15.67:0/3751234839 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:49.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.682+0000 7f801a7fc640 1 --2- 172.21.15.67:0/3751234839 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f801c162010 0x7f801c14e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:49.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.682+0000 7f801affd640 1 --2- 172.21.15.67:0/3751234839 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f801c0ce310 0x7f801c14d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:49.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.682+0000 7f801b7fe640 1 -- 172.21.15.67:0/3751234839 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f801c0ce310 msgr2=0x7f801c14d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:49.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.682+0000 7f801b7fe640 1 --2- 172.21.15.67:0/3751234839 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f801c0ce310 0x7f801c14d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:49.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.682+0000 7f801b7fe640 1 -- 172.21.15.67:0/3751234839 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f801c162010 msgr2=0x7f801c14e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:49.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.682+0000 7f801b7fe640 1 --2- 172.21.15.67:0/3751234839 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f801c162010 0x7f801c14e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:49.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.682+0000 7f801b7fe640 1 -- 172.21.15.67:0/3751234839 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f801c14d7a0 con 0x7f801c1623f0 2024-03-24T00:48:49.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.682+0000 7f801b7fe640 1 --2- 172.21.15.67:0/3751234839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f801c1623f0 0x7f801c165fa0 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f8014009d90 tx=0x7f8014004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:49.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.682+0000 7f80037fe640 1 -- 172.21.15.67:0/3751234839 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8014044020 con 0x7f801c1623f0 2024-03-24T00:48:49.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.682+0000 7f80037fe640 1 -- 172.21.15.67:0/3751234839 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f801403a040 con 0x7f801c1623f0 2024-03-24T00:48:49.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.682+0000 7f80037fe640 1 -- 172.21.15.67:0/3751234839 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8014039050 con 0x7f801c1623f0 2024-03-24T00:48:49.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.682+0000 7f80214f8640 1 -- 172.21.15.67:0/3751234839 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f801c14da80 con 0x7f801c1623f0 2024-03-24T00:48:49.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.682+0000 7f80214f8640 1 -- 172.21.15.67:0/3751234839 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f801c14e040 con 0x7f801c1623f0 2024-03-24T00:48:49.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.683+0000 7f80214f8640 1 -- 172.21.15.67:0/3751234839 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f801c04f400 con 0x7f801c1623f0 2024-03-24T00:48:49.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.684+0000 7f80037fe640 1 -- 172.21.15.67:0/3751234839 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8014003bb0 con 0x7f801c1623f0 2024-03-24T00:48:49.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.684+0000 7f80037fe640 1 --2- 172.21.15.67:0/3751234839 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f800806bbd0 0x7f800806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:49.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.684+0000 7f80037fe640 1 -- 172.21.15.67:0/3751234839 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f80140acf70 con 0x7f801c1623f0 2024-03-24T00:48:49.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.685+0000 7f801affd640 1 --2- 172.21.15.67:0/3751234839 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f800806bbd0 0x7f800806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:49.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.685+0000 7f801affd640 1 --2- 172.21.15.67:0/3751234839 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f800806bbd0 0x7f800806e090 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f801c0cda40 tx=0x7f800c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:49.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.689+0000 7f80037fe640 1 -- 172.21.15.67:0/3751234839 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f801404e060 con 0x7f801c1623f0 2024-03-24T00:48:49.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:49.883+0000 7f80214f8640 1 -- 172.21.15.67:0/3751234839 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f801c0d23f0 con 0x7f801c1623f0 2024-03-24T00:48:50.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:50.966+0000 7f80037fe640 1 -- 172.21.15.67:0/3751234839 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v116) v1 ==== 97+0+0 (secure 0 0 0) 0x7f801407bc10 con 0x7f801c1623f0 2024-03-24T00:48:50.966 INFO:teuthology.orchestra.run.smithi067.stderr: removed all blocklist entries 2024-03-24T00:48:50.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:50.969+0000 7f80214f8640 1 -- 172.21.15.67:0/3751234839 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f800806bbd0 msgr2=0x7f800806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:50.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:50.969+0000 7f80214f8640 1 --2- 172.21.15.67:0/3751234839 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f800806bbd0 0x7f800806e090 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f801c0cda40 tx=0x7f800c008040 comp rx=0 tx=0).stop 2024-03-24T00:48:50.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:50.969+0000 7f80214f8640 1 -- 172.21.15.67:0/3751234839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f801c1623f0 msgr2=0x7f801c165fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:50.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:50.969+0000 7f80214f8640 1 --2- 172.21.15.67:0/3751234839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f801c1623f0 0x7f801c165fa0 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f8014009d90 tx=0x7f8014004a40 comp rx=0 tx=0).stop 2024-03-24T00:48:50.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:50.969+0000 7f80214f8640 1 -- 172.21.15.67:0/3751234839 shutdown_connections 2024-03-24T00:48:50.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:50.970+0000 7f80214f8640 1 --2- 172.21.15.67:0/3751234839 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f800806bbd0 0x7f800806e090 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:50.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:50.970+0000 7f80214f8640 1 --2- 172.21.15.67:0/3751234839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f801c1623f0 0x7f801c165fa0 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:50.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:50.970+0000 7f80214f8640 1 --2- 172.21.15.67:0/3751234839 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f801c162010 0x7f801c14e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:50.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:50.970+0000 7f80214f8640 1 --2- 172.21.15.67:0/3751234839 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f801c0ce310 0x7f801c14d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:50.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:50.970+0000 7f80214f8640 1 -- 172.21.15.67:0/3751234839 >> 172.21.15.67:0/3751234839 conn(0x7f801c072b60 msgr2=0x7f801c0b5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:50.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:50.970+0000 7f80214f8640 1 -- 172.21.15.67:0/3751234839 shutdown_connections 2024-03-24T00:48:50.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:50.970+0000 7f80214f8640 1 -- 172.21.15.67:0/3751234839 wait complete. 2024-03-24T00:48:50.983 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-24T00:48:51.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.091+0000 7ff1a5d87640 1 Processor -- start 2024-03-24T00:48:51.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.091+0000 7ff1a5d87640 1 -- start start 2024-03-24T00:48:51.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.092+0000 7ff1a5d87640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff1a0145640 0x7ff1a00ccc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:51.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.092+0000 7ff1a5d87640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1a015fb10 con 0x7ff1a00cf570 2024-03-24T00:48:51.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.092+0000 7ff1a5d87640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1a015fdf0 con 0x7ff1a00cd1b0 2024-03-24T00:48:51.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.092+0000 7ff1a5d87640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1a00cd610 con 0x7ff1a0145640 2024-03-24T00:48:51.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.092+0000 7ff19ffff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff1a00cf570 0x7ff1a006dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48614/0 (socket says 172.21.15.67:48614) 2024-03-24T00:48:51.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.092+0000 7ff19ffff640 1 -- 172.21.15.67:0/1507047421 learned_addr learned my addr 172.21.15.67:0/1507047421 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:48:51.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.092+0000 7ff19f7fe640 1 --2- 172.21.15.67:0/1507047421 >> v2:172.21.15.72:3300/0 conn(0x7ff1a0145640 0x7ff1a00ccc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:51.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.092+0000 7ff19e7fc640 1 -- 172.21.15.67:0/1507047421 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1058726102 0 0) 0x7ff1a015fb10 con 0x7ff1a00cf570 2024-03-24T00:48:51.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.092+0000 7ff19e7fc640 1 -- 172.21.15.67:0/1507047421 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff188003520 con 0x7ff1a00cf570 2024-03-24T00:48:51.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.092+0000 7ff19e7fc640 1 -- 172.21.15.67:0/1507047421 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1355598053 0 0) 0x7ff188003520 con 0x7ff1a00cf570 2024-03-24T00:48:51.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.092+0000 7ff19e7fc640 1 -- 172.21.15.67:0/1507047421 >> v2:172.21.15.72:3300/0 conn(0x7ff1a0145640 msgr2=0x7ff1a00ccc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:51.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.092+0000 7ff19e7fc640 1 --2- 172.21.15.67:0/1507047421 >> v2:172.21.15.72:3300/0 conn(0x7ff1a0145640 0x7ff1a00ccc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:51.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.092+0000 7ff19e7fc640 1 -- 172.21.15.67:0/1507047421 >> v1:172.21.15.72:6789/0 conn(0x7ff1a00cd1b0 legacy=0x7ff1a00cee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:51.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.092+0000 7ff19e7fc640 1 -- 172.21.15.67:0/1507047421 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1a01600d0 con 0x7ff1a00cf570 2024-03-24T00:48:51.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.092+0000 7ff19e7fc640 1 -- 172.21.15.67:0/1507047421 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7ff1980019f0 con 0x7ff1a00cf570 2024-03-24T00:48:51.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.092+0000 7ff19e7fc640 1 -- 172.21.15.67:0/1507047421 >> v1:172.21.15.67:6789/0 conn(0x7ff1a00cf570 legacy=0x7ff1a006dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:51.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.092+0000 7ff19e7fc640 1 --2- 172.21.15.67:0/1507047421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1880039b0 0x7ff188003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:51.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.092+0000 7ff19e7fc640 1 --2- 172.21.15.67:0/1507047421 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff188004800 0x7ff188004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:51.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.093+0000 7ff19e7fc640 1 --2- 172.21.15.67:0/1507047421 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1880051c0 0x7ff18800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:51.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.093+0000 7ff19e7fc640 1 -- 172.21.15.67:0/1507047421 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18800c030 con 0x7ff1880039b0 2024-03-24T00:48:51.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.093+0000 7ff19e7fc640 1 -- 172.21.15.67:0/1507047421 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18800c1e0 con 0x7ff1880051c0 2024-03-24T00:48:51.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.093+0000 7ff19e7fc640 1 -- 172.21.15.67:0/1507047421 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff18800c3e0 con 0x7ff188004800 2024-03-24T00:48:51.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.093+0000 7ff19f7fe640 1 --2- 172.21.15.67:0/1507047421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1880039b0 0x7ff188003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:51.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.093+0000 7ff19ffff640 1 --2- 172.21.15.67:0/1507047421 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1880051c0 0x7ff18800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:51.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.093+0000 7ff19effd640 1 --2- 172.21.15.67:0/1507047421 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff188004800 0x7ff188004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:51.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.093+0000 7ff19f7fe640 1 -- 172.21.15.67:0/1507047421 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff188004800 msgr2=0x7ff188004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:51.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.093+0000 7ff19f7fe640 1 --2- 172.21.15.67:0/1507047421 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff188004800 0x7ff188004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:51.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.093+0000 7ff19f7fe640 1 -- 172.21.15.67:0/1507047421 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1880051c0 msgr2=0x7ff18800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:51.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.093+0000 7ff19f7fe640 1 --2- 172.21.15.67:0/1507047421 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1880051c0 0x7ff18800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:51.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.093+0000 7ff19ffff640 1 -- 172.21.15.67:0/1507047421 reap_dead start 2024-03-24T00:48:51.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.093+0000 7ff19f7fe640 1 -- 172.21.15.67:0/1507047421 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff188003520 con 0x7ff1880039b0 2024-03-24T00:48:51.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.093+0000 7ff19effd640 1 --2- 172.21.15.67:0/1507047421 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff188004800 0x7ff188004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:48:51.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.094+0000 7ff19f7fe640 1 --2- 172.21.15.67:0/1507047421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1880039b0 0x7ff188003dd0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7ff190009ac0 tx=0x7ff188005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29c846ff9039f283 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:51.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.094+0000 7ff19e7fc640 1 -- 172.21.15.67:0/1507047421 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff190039070 con 0x7ff1880039b0 2024-03-24T00:48:51.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.094+0000 7ff19e7fc640 1 -- 172.21.15.67:0/1507047421 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff19002c6b0 con 0x7ff1880039b0 2024-03-24T00:48:51.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.094+0000 7ff19e7fc640 1 -- 172.21.15.67:0/1507047421 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff190009690 con 0x7ff1880039b0 2024-03-24T00:48:51.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.094+0000 7ff1a5d87640 1 -- 172.21.15.67:0/1507047421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1880039b0 msgr2=0x7ff188003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:51.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.094+0000 7ff1a5d87640 1 --2- 172.21.15.67:0/1507047421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1880039b0 0x7ff188003dd0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7ff190009ac0 tx=0x7ff188005830 comp rx=0 tx=0).stop 2024-03-24T00:48:51.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.094+0000 7ff1a5d87640 1 -- 172.21.15.67:0/1507047421 shutdown_connections 2024-03-24T00:48:51.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.094+0000 7ff1a5d87640 1 --2- 172.21.15.67:0/1507047421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1880039b0 0x7ff188003dd0 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:51.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.094+0000 7ff1a5d87640 1 -- 172.21.15.67:0/1507047421 >> 172.21.15.67:0/1507047421 conn(0x7ff1a00cf9e0 msgr2=0x7ff1a00cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:51.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.094+0000 7ff1a5d87640 1 -- 172.21.15.67:0/1507047421 shutdown_connections 2024-03-24T00:48:51.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.094+0000 7ff1a5d87640 1 -- 172.21.15.67:0/1507047421 wait complete. 2024-03-24T00:48:51.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.096+0000 7ff1a5d87640 1 Processor -- start 2024-03-24T00:48:51.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.096+0000 7ff1a5d87640 1 -- start start 2024-03-24T00:48:51.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.096+0000 7ff1a5d87640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff1880039b0 0x7ff1a01f2650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:51.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.097+0000 7ff1a5d87640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1a0158ff0 0x7ff1a013f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:51.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.097+0000 7ff1a5d87640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1a013f590 0x7ff1a013b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:51.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.097+0000 7ff1a5d87640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a0160720 con 0x7ff1a0158ff0 2024-03-24T00:48:51.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.097+0000 7ff1a5d87640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a015f8c0 con 0x7ff1a013f590 2024-03-24T00:48:51.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.097+0000 7ff1a5d87640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a00ccc70 con 0x7ff1880039b0 2024-03-24T00:48:51.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.097+0000 7ff19effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1a0158ff0 0x7ff1a013f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:51.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.097+0000 7ff19effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1a0158ff0 0x7ff1a013f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34840/0 (socket says 172.21.15.67:34840) 2024-03-24T00:48:51.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.097+0000 7ff19effd640 1 -- 172.21.15.67:0/3380712540 learned_addr learned my addr 172.21.15.67:0/3380712540 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:51.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.097+0000 7ff19f7fe640 1 --2- 172.21.15.67:0/3380712540 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff1880039b0 0x7ff1a01f2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:51.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.097+0000 7ff19effd640 1 -- 172.21.15.67:0/3380712540 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff1880039b0 msgr2=0x7ff1a01f2650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:51.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.097+0000 7ff19effd640 1 --2- 172.21.15.67:0/3380712540 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff1880039b0 0x7ff1a01f2650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:51.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.097+0000 7ff19effd640 1 -- 172.21.15.67:0/3380712540 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1a013f590 msgr2=0x7ff1a013b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:51.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.097+0000 7ff19ffff640 1 --2- 172.21.15.67:0/3380712540 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1a013f590 0x7ff1a013b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:51.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.097+0000 7ff19effd640 1 --2- 172.21.15.67:0/3380712540 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1a013f590 0x7ff1a013b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:51.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.097+0000 7ff19effd640 1 -- 172.21.15.67:0/3380712540 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1a013c0e0 con 0x7ff1a0158ff0 2024-03-24T00:48:51.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.097+0000 7ff19f7fe640 1 --2- 172.21.15.67:0/3380712540 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff1880039b0 0x7ff1a01f2650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:48:51.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.097+0000 7ff19ffff640 1 --2- 172.21.15.67:0/3380712540 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1a013f590 0x7ff1a013b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:48:51.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.097+0000 7ff19effd640 1 --2- 172.21.15.67:0/3380712540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1a0158ff0 0x7ff1a013f050 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7ff19400e4b0 tx=0x7ff19400e970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:51.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.097+0000 7ff187fff640 1 -- 172.21.15.67:0/3380712540 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1940052e0 con 0x7ff1a0158ff0 2024-03-24T00:48:51.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.097+0000 7ff187fff640 1 -- 172.21.15.67:0/3380712540 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff194007020 con 0x7ff1a0158ff0 2024-03-24T00:48:51.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.097+0000 7ff187fff640 1 -- 172.21.15.67:0/3380712540 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff194007210 con 0x7ff1a0158ff0 2024-03-24T00:48:51.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.097+0000 7ff1a5d87640 1 -- 172.21.15.67:0/3380712540 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1a01f1b70 con 0x7ff1a0158ff0 2024-03-24T00:48:51.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.097+0000 7ff1a5d87640 1 -- 172.21.15.67:0/3380712540 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1a0159470 con 0x7ff1a0158ff0 2024-03-24T00:48:51.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.098+0000 7ff1a5d87640 1 -- 172.21.15.67:0/3380712540 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1a01f1980 con 0x7ff1a0158ff0 2024-03-24T00:48:51.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.099+0000 7ff187fff640 1 -- 172.21.15.67:0/3380712540 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff19401a020 con 0x7ff1a0158ff0 2024-03-24T00:48:51.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.099+0000 7ff187fff640 1 --2- 172.21.15.67:0/3380712540 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff18c06bbd0 0x7ff18c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:51.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.099+0000 7ff187fff640 1 -- 172.21.15.67:0/3380712540 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5249+0+0 (secure 0 0 0) 0x7ff194089ed0 con 0x7ff1a0158ff0 2024-03-24T00:48:51.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.099+0000 7ff19f7fe640 1 --2- 172.21.15.67:0/3380712540 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff18c06bbd0 0x7ff18c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:51.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.100+0000 7ff19f7fe640 1 --2- 172.21.15.67:0/3380712540 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff18c06bbd0 0x7ff18c06e090 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7ff190036000 tx=0x7ff190030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:51.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.104+0000 7ff187fff640 1 -- 172.21.15.67:0/3380712540 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff19401a310 con 0x7ff1a0158ff0 2024-03-24T00:48:51.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.354+0000 7ff1a5d87640 1 -- 172.21.15.67:0/3380712540 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7ff1a004f400 con 0x7ff1a0158ff0 2024-03-24T00:48:51.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.355+0000 7ff187fff640 1 -- 172.21.15.67:0/3380712540 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v19) v1 ==== 80+0+5304 (secure 0 0 0) 0x7ff194058af0 con 0x7ff1a0158ff0 2024-03-24T00:48:51.355 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:48:51.355 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:48:51.355 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_dump": [ 2024-03-24T00:48:51.355 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:48:51.355 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.a", 2024-03-24T00:48:51.355 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9l4UENDhAASKUDtqcbSQ7wADj7uo4eyQ==", 2024-03-24T00:48:51.355 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:48:51.355 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:48:51.355 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-03-24T00:48:51.355 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-03-24T00:48:51.355 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:48:51.355 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:48:51.356 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:48:51.356 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:48:51.356 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.b", 2024-03-24T00:48:51.356 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9luhJWGhAAsS6PFHV2XdF9htjIjzdrdA==", 2024-03-24T00:48:51.356 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:48:51.356 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:48:51.356 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-03-24T00:48:51.356 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-03-24T00:48:51.356 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:48:51.356 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:48:51.356 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:48:51.356 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:48:51.356 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.c", 2024-03-24T00:48:51.356 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9l8mRjFRAAQ73yexNddpcyp8ayTcRKCg==", 2024-03-24T00:48:51.356 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:48:51.357 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:48:51.357 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-03-24T00:48:51.357 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-03-24T00:48:51.357 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:48:51.357 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:48:51.357 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:48:51.357 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:48:51.357 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.d", 2024-03-24T00:48:51.357 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9lcJyvIRAA2tpDb6qxn1MtljjEno4qPA==", 2024-03-24T00:48:51.357 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:48:51.357 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:48:51.357 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-03-24T00:48:51.357 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-03-24T00:48:51.357 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:48:51.357 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:48:51.358 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:48:51.358 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:48:51.358 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.0", 2024-03-24T00:48:51.358 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDgdf9lV/NUBRAAyJNajFAnNABJErnBlB8TEA==", 2024-03-24T00:48:51.358 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:48:51.358 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:48:51.358 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:48:51.358 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:48:51.358 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:48:51.358 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:48:51.358 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:48:51.358 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.1", 2024-03-24T00:48:51.358 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDjdf9l0J6lAxAA9RInt/X22jqzthE0vOiiFg==", 2024-03-24T00:48:51.358 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:48:51.359 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:48:51.359 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:48:51.359 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:48:51.359 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:48:51.359 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:48:51.359 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:48:51.359 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.2", 2024-03-24T00:48:51.359 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDmdf9lSmrEARAAwmtSQRNePsn4WIZODnjhfQ==", 2024-03-24T00:48:51.359 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:48:51.359 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:48:51.359 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:48:51.359 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:48:51.359 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:48:51.359 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:48:51.359 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:48:51.360 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.3", 2024-03-24T00:48:51.360 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDpdf9lm59bABAAgJ5AV774lw2BgkSCgaweyA==", 2024-03-24T00:48:51.360 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:48:51.360 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:48:51.360 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:48:51.360 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:48:51.360 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:48:51.360 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:48:51.360 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:48:51.360 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.4", 2024-03-24T00:48:51.360 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDvdf9lCo+pCRAANHkL0pPxxok6IsBCLAFCbg==", 2024-03-24T00:48:51.360 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:48:51.360 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:48:51.361 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:48:51.361 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:48:51.361 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:48:51.361 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:48:51.361 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:48:51.361 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.5", 2024-03-24T00:48:51.361 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDydf9lyGEdCBAAcuszPsthGlLvVryE7+D9iQ==", 2024-03-24T00:48:51.361 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:48:51.361 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:48:51.361 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:48:51.361 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:48:51.361 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:48:51.361 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:48:51.361 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:48:51.362 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.6", 2024-03-24T00:48:51.362 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQD1df9ljuZqBhAACt2OPNkjKOD3rK7g2mVBBA==", 2024-03-24T00:48:51.362 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:48:51.362 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:48:51.362 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:48:51.362 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:48:51.362 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:48:51.362 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:48:51.362 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:48:51.362 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.7", 2024-03-24T00:48:51.362 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQD4df9lYKBfBRAAy/cWFF6pvGDjXZHAuBYw7g==", 2024-03-24T00:48:51.362 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:48:51.362 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-03-24T00:48:51.362 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-03-24T00:48:51.363 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:48:51.363 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:48:51.363 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:48:51.363 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:48:51.363 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.0", 2024-03-24T00:48:51.363 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9lY4nvJRAA6HWOZDWA+SSvJVmyjG+51g==", 2024-03-24T00:48:51.363 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:48:51.363 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:48:51.363 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow r", 2024-03-24T00:48:51.363 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-24T00:48:51.363 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:48:51.363 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:48:51.363 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:48:51.363 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.1", 2024-03-24T00:48:51.364 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9lonsXLxAAaltVMW3n0qHX8u9X8EkSmg==", 2024-03-24T00:48:51.364 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:48:51.364 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:48:51.364 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-03-24T00:48:51.364 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-03-24T00:48:51.364 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-03-24T00:48:51.364 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:48:51.364 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:48:51.364 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:48:51.364 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.2", 2024-03-24T00:48:51.364 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9lLEm2KxAAkT65mhLrTirlDnOUyvBY7g==", 2024-03-24T00:48:51.364 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:48:51.364 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:48:51.365 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-03-24T00:48:51.365 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-03-24T00:48:51.365 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-03-24T00:48:51.365 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:48:51.365 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:48:51.365 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:48:51.365 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.3", 2024-03-24T00:48:51.365 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9l8njZNBAAaAktZpX2SPcJR9ymPm9ebw==", 2024-03-24T00:48:51.365 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:48:51.365 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-03-24T00:48:51.365 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-03-24T00:48:51.365 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-03-24T00:48:51.365 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-03-24T00:48:51.365 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:48:51.366 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:48:51.366 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:48:51.366 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.admin", 2024-03-24T00:48:51.366 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDbdf9lDTIwLRAAMajqhM/KfUNF/SjVv9OJDw==", 2024-03-24T00:48:51.366 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:48:51.366 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-03-24T00:48:51.366 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-03-24T00:48:51.366 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-03-24T00:48:51.366 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:48:51.366 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:48:51.366 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:48:51.366 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:48:51.366 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.x", 2024-03-24T00:48:51.366 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9lGwKvChAAxl+fRnjbAQ7dYJ727QFUgw==", 2024-03-24T00:48:51.367 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:48:51.367 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-03-24T00:48:51.367 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-03-24T00:48:51.367 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:48:51.367 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:48:51.367 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-24T00:48:51.367 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:48:51.367 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.y", 2024-03-24T00:48:51.367 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDcdf9lSvufBhAAoQ9L1GjLkstjrAw2lqlryA==", 2024-03-24T00:48:51.367 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-03-24T00:48:51.367 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-03-24T00:48:51.367 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-03-24T00:48:51.367 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-03-24T00:48:51.367 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:48:51.368 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:48:51.368 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-24T00:48:51.368 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:48:51.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.356+0000 7ff1a5d87640 1 -- 172.21.15.67:0/3380712540 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff18c06bbd0 msgr2=0x7ff18c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:51.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.356+0000 7ff1a5d87640 1 --2- 172.21.15.67:0/3380712540 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff18c06bbd0 0x7ff18c06e090 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7ff190036000 tx=0x7ff190030000 comp rx=0 tx=0).stop 2024-03-24T00:48:51.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.356+0000 7ff1a5d87640 1 -- 172.21.15.67:0/3380712540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1a0158ff0 msgr2=0x7ff1a013f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:51.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.356+0000 7ff1a5d87640 1 --2- 172.21.15.67:0/3380712540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1a0158ff0 0x7ff1a013f050 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7ff19400e4b0 tx=0x7ff19400e970 comp rx=0 tx=0).stop 2024-03-24T00:48:51.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.356+0000 7ff1a5d87640 1 -- 172.21.15.67:0/3380712540 shutdown_connections 2024-03-24T00:48:51.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.356+0000 7ff1a5d87640 1 --2- 172.21.15.67:0/3380712540 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff18c06bbd0 0x7ff18c06e090 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:51.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.356+0000 7ff1a5d87640 1 --2- 172.21.15.67:0/3380712540 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1a013f590 0x7ff1a013b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:51.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.356+0000 7ff1a5d87640 1 --2- 172.21.15.67:0/3380712540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1a0158ff0 0x7ff1a013f050 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:51.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.356+0000 7ff1a5d87640 1 --2- 172.21.15.67:0/3380712540 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff1880039b0 0x7ff1a01f2650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:51.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.356+0000 7ff1a5d87640 1 -- 172.21.15.67:0/3380712540 >> 172.21.15.67:0/3380712540 conn(0x7ff1a00cf9e0 msgr2=0x7ff1a00cee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:51.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.357+0000 7ff1a5d87640 1 -- 172.21.15.67:0/3380712540 shutdown_connections 2024-03-24T00:48:51.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.357+0000 7ff1a5d87640 1 -- 172.21.15.67:0/3380712540 wait complete. 2024-03-24T00:48:51.371 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:48:51.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.472+0000 7f84523b3640 1 Processor -- start 2024-03-24T00:48:51.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.472+0000 7f84523b3640 1 -- start start 2024-03-24T00:48:51.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.473+0000 7f84523b3640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f844c15dfb0 0x7f844c0cfe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:51.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.473+0000 7f84523b3640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f844c06b750 0x7f844c0cb120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:51.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.473+0000 7f84523b3640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f844c0d07d0 con 0x7f844c15e390 2024-03-24T00:48:51.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.473+0000 7f84523b3640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f844c0d0ab0 con 0x7f844c06b750 2024-03-24T00:48:51.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.473+0000 7f84523b3640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f844c0d0d90 con 0x7f844c15dfb0 2024-03-24T00:48:51.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.473+0000 7f8450950640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f844c06b750 0x7f844c0cb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:51.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.473+0000 7f844b7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f844c15e390 0x7f844c06b040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48622/0 (socket says 172.21.15.67:48622) 2024-03-24T00:48:51.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.473+0000 7f844b7fe640 1 -- 172.21.15.67:0/567294438 learned_addr learned my addr 172.21.15.67:0/567294438 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:48:51.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.473+0000 7f844bfff640 1 --2- 172.21.15.67:0/567294438 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f844c15dfb0 0x7f844c0cfe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:51.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.473+0000 7f8450950640 1 -- 172.21.15.67:0/567294438 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f844c15dfb0 msgr2=0x7f844c0cfe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:51.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.473+0000 7f8450950640 1 --2- 172.21.15.67:0/567294438 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f844c15dfb0 0x7f844c0cfe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:51.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.473+0000 7f8450950640 1 -- 172.21.15.67:0/567294438 >> v1:172.21.15.67:6789/0 conn(0x7f844c15e390 legacy=0x7f844c06b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:51.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.474+0000 7f8450950640 1 -- 172.21.15.67:0/567294438 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f844c0d1070 con 0x7f844c06b750 2024-03-24T00:48:51.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.474+0000 7f844bfff640 1 --2- 172.21.15.67:0/567294438 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f844c15dfb0 0x7f844c0cfe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:48:51.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.474+0000 7f8450950640 1 --2- 172.21.15.67:0/567294438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f844c06b750 0x7f844c0cb120 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f8444009d90 tx=0x7f844402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f07f2646955cf9f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:51.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.474+0000 7f844affd640 1 -- 172.21.15.67:0/567294438 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8444039070 con 0x7f844c06b750 2024-03-24T00:48:51.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.474+0000 7f844affd640 1 -- 172.21.15.67:0/567294438 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f844402cc50 con 0x7f844c06b750 2024-03-24T00:48:51.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.474+0000 7f844affd640 1 -- 172.21.15.67:0/567294438 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8444034440 con 0x7f844c06b750 2024-03-24T00:48:51.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.475+0000 7f84523b3640 1 -- 172.21.15.67:0/567294438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f844c06b750 msgr2=0x7f844c0cb120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:51.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.475+0000 7f84523b3640 1 --2- 172.21.15.67:0/567294438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f844c06b750 0x7f844c0cb120 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f8444009d90 tx=0x7f844402bea0 comp rx=0 tx=0).stop 2024-03-24T00:48:51.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.475+0000 7f84523b3640 1 -- 172.21.15.67:0/567294438 shutdown_connections 2024-03-24T00:48:51.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.475+0000 7f84523b3640 1 --2- 172.21.15.67:0/567294438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f844c06b750 0x7f844c0cb120 unknown :-1 s=CLOSED pgs=1685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:51.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.475+0000 7f84523b3640 1 --2- 172.21.15.67:0/567294438 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f844c15dfb0 0x7f844c0cfe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:51.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.475+0000 7f84523b3640 1 -- 172.21.15.67:0/567294438 >> 172.21.15.67:0/567294438 conn(0x7f844c099a30 msgr2=0x7f844c099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:51.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.475+0000 7f84523b3640 1 -- 172.21.15.67:0/567294438 shutdown_connections 2024-03-24T00:48:51.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.475+0000 7f84523b3640 1 -- 172.21.15.67:0/567294438 wait complete. 2024-03-24T00:48:51.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.476+0000 7f84523b3640 1 Processor -- start 2024-03-24T00:48:51.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.476+0000 7f84523b3640 1 -- start start 2024-03-24T00:48:51.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.477+0000 7f84523b3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f844c06b750 0x7f844c09a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:51.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.477+0000 7f84523b3640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f844c15dfb0 0x7f844c15c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:51.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.477+0000 7f84523b3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f844c15e390 0x7f844c0c71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:51.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.477+0000 7f84523b3640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f844c0d03c0 con 0x7f844c06b750 2024-03-24T00:48:51.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.477+0000 7f84523b3640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f844c0d1870 con 0x7f844c15e390 2024-03-24T00:48:51.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.477+0000 7f84523b3640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f844c0d0540 con 0x7f844c15dfb0 2024-03-24T00:48:51.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.477+0000 7f844bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f844c06b750 0x7f844c09a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:51.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.477+0000 7f844bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f844c06b750 0x7f844c09a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34848/0 (socket says 172.21.15.67:34848) 2024-03-24T00:48:51.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.477+0000 7f844bfff640 1 -- 172.21.15.67:0/2251394123 learned_addr learned my addr 172.21.15.67:0/2251394123 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:51.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.477+0000 7f844b7fe640 1 --2- 172.21.15.67:0/2251394123 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f844c15dfb0 0x7f844c15c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:51.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.477+0000 7f8450950640 1 --2- 172.21.15.67:0/2251394123 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f844c15e390 0x7f844c0c71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:51.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.477+0000 7f844bfff640 1 -- 172.21.15.67:0/2251394123 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f844c15dfb0 msgr2=0x7f844c15c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:51.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.477+0000 7f844bfff640 1 --2- 172.21.15.67:0/2251394123 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f844c15dfb0 0x7f844c15c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:51.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.477+0000 7f844bfff640 1 -- 172.21.15.67:0/2251394123 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f844c15e390 msgr2=0x7f844c0c71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:51.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.477+0000 7f844bfff640 1 --2- 172.21.15.67:0/2251394123 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f844c15e390 0x7f844c0c71d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:51.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.477+0000 7f844bfff640 1 -- 172.21.15.67:0/2251394123 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f844c1f51f0 con 0x7f844c06b750 2024-03-24T00:48:51.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.477+0000 7f8450950640 1 --2- 172.21.15.67:0/2251394123 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f844c15e390 0x7f844c0c71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:48:51.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.478+0000 7f844bfff640 1 --2- 172.21.15.67:0/2251394123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f844c06b750 0x7f844c09a9d0 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f844000c7f0 tx=0x7f844000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:51.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.478+0000 7f8448ff9640 1 -- 172.21.15.67:0/2251394123 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8440015070 con 0x7f844c06b750 2024-03-24T00:48:51.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.478+0000 7f8448ff9640 1 -- 172.21.15.67:0/2251394123 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8440007a30 con 0x7f844c06b750 2024-03-24T00:48:51.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.478+0000 7f84523b3640 1 -- 172.21.15.67:0/2251394123 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f844c1f4670 con 0x7f844c06b750 2024-03-24T00:48:51.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.478+0000 7f8448ff9640 1 -- 172.21.15.67:0/2251394123 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8440007e00 con 0x7f844c06b750 2024-03-24T00:48:51.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.478+0000 7f84523b3640 1 -- 172.21.15.67:0/2251394123 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f844c1f4c30 con 0x7f844c06b750 2024-03-24T00:48:51.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.479+0000 7f84523b3640 1 -- 172.21.15.67:0/2251394123 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f844c04f400 con 0x7f844c06b750 2024-03-24T00:48:51.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.479+0000 7f8448ff9640 1 -- 172.21.15.67:0/2251394123 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8440018070 con 0x7f844c06b750 2024-03-24T00:48:51.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.480+0000 7f8448ff9640 1 --2- 172.21.15.67:0/2251394123 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f843806bbd0 0x7f843806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:51.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.480+0000 7f8448ff9640 1 -- 172.21.15.67:0/2251394123 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f844008a720 con 0x7f844c06b750 2024-03-24T00:48:51.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.480+0000 7f844b7fe640 1 --2- 172.21.15.67:0/2251394123 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f843806bbd0 0x7f843806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:51.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.480+0000 7f844b7fe640 1 --2- 172.21.15.67:0/2251394123 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f843806bbd0 0x7f843806e090 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f844c067430 tx=0x7f843c007e80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:51.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.484+0000 7f8448ff9640 1 -- 172.21.15.67:0/2251394123 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8440058e10 con 0x7f844c06b750 2024-03-24T00:48:51.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.667+0000 7f84523b3640 1 -- 172.21.15.67:0/2251394123 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f844c0b6a50 con 0x7f844c06b750 2024-03-24T00:48:51.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.668+0000 7f8448ff9640 1 -- 172.21.15.67:0/2251394123 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) v1 ==== 74+0+13114 (secure 0 0 0) 0x7f844005ceb0 con 0x7f844c06b750 2024-03-24T00:48:51.668 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:48:51.668 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":116,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:48:50.965081+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:48:51.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.670+0000 7f84523b3640 1 -- 172.21.15.67:0/2251394123 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f843806bbd0 msgr2=0x7f843806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:51.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.670+0000 7f84523b3640 1 --2- 172.21.15.67:0/2251394123 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f843806bbd0 0x7f843806e090 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f844c067430 tx=0x7f843c007e80 comp rx=0 tx=0).stop 2024-03-24T00:48:51.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.670+0000 7f84523b3640 1 -- 172.21.15.67:0/2251394123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f844c06b750 msgr2=0x7f844c09a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:51.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.670+0000 7f84523b3640 1 --2- 172.21.15.67:0/2251394123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f844c06b750 0x7f844c09a9d0 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f844000c7f0 tx=0x7f844000cbf0 comp rx=0 tx=0).stop 2024-03-24T00:48:51.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.670+0000 7f84523b3640 1 -- 172.21.15.67:0/2251394123 shutdown_connections 2024-03-24T00:48:51.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.670+0000 7f84523b3640 1 --2- 172.21.15.67:0/2251394123 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f843806bbd0 0x7f843806e090 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:51.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.670+0000 7f84523b3640 1 --2- 172.21.15.67:0/2251394123 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f844c15e390 0x7f844c0c71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:51.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.670+0000 7f84523b3640 1 --2- 172.21.15.67:0/2251394123 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f844c15dfb0 0x7f844c15c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:51.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.670+0000 7f84523b3640 1 --2- 172.21.15.67:0/2251394123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f844c06b750 0x7f844c09a9d0 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:51.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.670+0000 7f84523b3640 1 -- 172.21.15.67:0/2251394123 >> 172.21.15.67:0/2251394123 conn(0x7f844c099a30 msgr2=0x7f844c0cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:51.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.670+0000 7f84523b3640 1 -- 172.21.15.67:0/2251394123 shutdown_connections 2024-03-24T00:48:51.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.670+0000 7f84523b3640 1 -- 172.21.15.67:0/2251394123 wait complete. 2024-03-24T00:48:51.681 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:38:39.364950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:48:51.681 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:48:51.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.786+0000 7f6e6803f640 1 Processor -- start 2024-03-24T00:48:51.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.787+0000 7f6e6803f640 1 -- start start 2024-03-24T00:48:51.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.787+0000 7f6e6803f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6e600d3ae0 0x7f6e600b5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:51.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.787+0000 7f6e6803f640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6e6015e280 0x7f6e600b5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:51.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.787+0000 7f6e6803f640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e600d2250 con 0x7f6e6015dea0 2024-03-24T00:48:51.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.787+0000 7f6e6803f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e600d2530 con 0x7f6e600d3ae0 2024-03-24T00:48:51.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.787+0000 7f6e6803f640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e600d2810 con 0x7f6e6015e280 2024-03-24T00:48:51.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.787+0000 7f6e655da640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6e600d3ae0 0x7f6e600b5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:51.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.787+0000 7f6e655da640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6e600d3ae0 0x7f6e600b5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34860/0 (socket says 172.21.15.67:34860) 2024-03-24T00:48:51.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.788+0000 7f6e655da640 1 -- 172.21.15.67:0/1306257093 learned_addr learned my addr 172.21.15.67:0/1306257093 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:51.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.788+0000 7f6e665dc640 1 --2- 172.21.15.67:0/1306257093 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6e6015e280 0x7f6e600b5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:51.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.788+0000 7f6e655da640 1 -- 172.21.15.67:0/1306257093 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6e6015e280 msgr2=0x7f6e600b5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:51.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.788+0000 7f6e655da640 1 --2- 172.21.15.67:0/1306257093 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6e6015e280 0x7f6e600b5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:51.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.788+0000 7f6e655da640 1 -- 172.21.15.67:0/1306257093 >> v1:172.21.15.72:6789/0 conn(0x7f6e6015dea0 legacy=0x7f6e600d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:51.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.788+0000 7f6e655da640 1 -- 172.21.15.67:0/1306257093 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e600d2af0 con 0x7f6e600d3ae0 2024-03-24T00:48:51.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.789+0000 7f6e655da640 1 --2- 172.21.15.67:0/1306257093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e600d3ae0 0x7f6e600b5a60 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c0098a0 tx=0x7f6e5c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=943900188388f42a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:51.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.789+0000 7f6e64dd9640 1 -- 172.21.15.67:0/1306257093 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e5c035020 con 0x7f6e600d3ae0 2024-03-24T00:48:51.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.789+0000 7f6e64dd9640 1 -- 172.21.15.67:0/1306257093 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e5c031640 con 0x7f6e600d3ae0 2024-03-24T00:48:51.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.789+0000 7f6e64dd9640 1 -- 172.21.15.67:0/1306257093 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e5c02d040 con 0x7f6e600d3ae0 2024-03-24T00:48:51.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.789+0000 7f6e6803f640 1 -- 172.21.15.67:0/1306257093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e600d3ae0 msgr2=0x7f6e600b5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:51.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.789+0000 7f6e6803f640 1 --2- 172.21.15.67:0/1306257093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e600d3ae0 0x7f6e600b5a60 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c0098a0 tx=0x7f6e5c02b8f0 comp rx=0 tx=0).stop 2024-03-24T00:48:51.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.789+0000 7f6e6803f640 1 -- 172.21.15.67:0/1306257093 shutdown_connections 2024-03-24T00:48:51.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.789+0000 7f6e6803f640 1 --2- 172.21.15.67:0/1306257093 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6e6015e280 0x7f6e600b5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:51.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.789+0000 7f6e6803f640 1 --2- 172.21.15.67:0/1306257093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e600d3ae0 0x7f6e600b5a60 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:51.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.789+0000 7f6e6803f640 1 -- 172.21.15.67:0/1306257093 >> 172.21.15.67:0/1306257093 conn(0x7f6e60072da0 msgr2=0x7f6e600d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:51.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.789+0000 7f6e6803f640 1 -- 172.21.15.67:0/1306257093 shutdown_connections 2024-03-24T00:48:51.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.789+0000 7f6e6803f640 1 -- 172.21.15.67:0/1306257093 wait complete. 2024-03-24T00:48:51.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.791+0000 7f6e6803f640 1 Processor -- start 2024-03-24T00:48:51.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.791+0000 7f6e6803f640 1 -- start start 2024-03-24T00:48:51.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.791+0000 7f6e6803f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6e600d3ae0 0x7f6e600c9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:51.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.791+0000 7f6e6803f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e6015dea0 0x7f6e60159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:51.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.791+0000 7f6e6803f640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6e6015e280 0x7f6e6014ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:51.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.791+0000 7f6e6803f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e600b7480 con 0x7f6e6015dea0 2024-03-24T00:48:51.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.791+0000 7f6e6803f640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e600b7300 con 0x7f6e600d3ae0 2024-03-24T00:48:51.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.791+0000 7f6e6803f640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e600b7600 con 0x7f6e6015e280 2024-03-24T00:48:51.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.791+0000 7f6e655da640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e6015dea0 0x7f6e60159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:51.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.791+0000 7f6e655da640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e6015dea0 0x7f6e60159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34868/0 (socket says 172.21.15.67:34868) 2024-03-24T00:48:51.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.791+0000 7f6e655da640 1 -- 172.21.15.67:0/101657530 learned_addr learned my addr 172.21.15.67:0/101657530 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:51.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.792+0000 7f6e665dc640 1 --2- 172.21.15.67:0/101657530 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6e6015e280 0x7f6e6014ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:51.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.792+0000 7f6e65ddb640 1 --2- 172.21.15.67:0/101657530 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6e600d3ae0 0x7f6e600c9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:51.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.792+0000 7f6e655da640 1 -- 172.21.15.67:0/101657530 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6e6015e280 msgr2=0x7f6e6014ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:51.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.792+0000 7f6e655da640 1 --2- 172.21.15.67:0/101657530 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6e6015e280 0x7f6e6014ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:51.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.792+0000 7f6e655da640 1 -- 172.21.15.67:0/101657530 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6e600d3ae0 msgr2=0x7f6e600c9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:51.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.792+0000 7f6e655da640 1 --2- 172.21.15.67:0/101657530 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6e600d3ae0 0x7f6e600c9cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:51.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.792+0000 7f6e655da640 1 -- 172.21.15.67:0/101657530 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e60158f10 con 0x7f6e6015dea0 2024-03-24T00:48:51.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.792+0000 7f6e65ddb640 1 --2- 172.21.15.67:0/101657530 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6e600d3ae0 0x7f6e600c9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:48:51.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.792+0000 7f6e655da640 1 --2- 172.21.15.67:0/101657530 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e6015dea0 0x7f6e60159fb0 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c0099c0 tx=0x7f6e5c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:51.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.792+0000 7f6e527fc640 1 -- 172.21.15.67:0/101657530 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e5c040070 con 0x7f6e6015dea0 2024-03-24T00:48:51.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.792+0000 7f6e527fc640 1 -- 172.21.15.67:0/101657530 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e5c03e070 con 0x7f6e6015dea0 2024-03-24T00:48:51.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.792+0000 7f6e527fc640 1 -- 172.21.15.67:0/101657530 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e5c035040 con 0x7f6e6015dea0 2024-03-24T00:48:51.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.792+0000 7f6e6803f640 1 -- 172.21.15.67:0/101657530 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e601591f0 con 0x7f6e6015dea0 2024-03-24T00:48:51.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.792+0000 7f6e6803f640 1 -- 172.21.15.67:0/101657530 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e601597b0 con 0x7f6e6015dea0 2024-03-24T00:48:51.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.793+0000 7f6e6803f640 1 -- 172.21.15.67:0/101657530 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e6004f400 con 0x7f6e6015dea0 2024-03-24T00:48:51.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.794+0000 7f6e527fc640 1 -- 172.21.15.67:0/101657530 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6e5c038020 con 0x7f6e6015dea0 2024-03-24T00:48:51.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.794+0000 7f6e527fc640 1 --2- 172.21.15.67:0/101657530 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6e4406bbd0 0x7f6e4406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:51.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.794+0000 7f6e527fc640 1 -- 172.21.15.67:0/101657530 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f6e5c0ac200 con 0x7f6e6015dea0 2024-03-24T00:48:51.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.794+0000 7f6e65ddb640 1 --2- 172.21.15.67:0/101657530 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6e4406bbd0 0x7f6e4406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:51.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.795+0000 7f6e65ddb640 1 --2- 172.21.15.67:0/101657530 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6e4406bbd0 0x7f6e4406e090 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f6e54001680 tx=0x7f6e54009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:51.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.799+0000 7f6e527fc640 1 -- 172.21.15.67:0/101657530 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6e5c07aeb0 con 0x7f6e6015dea0 2024-03-24T00:48:51.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.981+0000 7f6e6803f640 1 -- 172.21.15.67:0/101657530 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6e6006b990 con 0x7f6e6015dea0 2024-03-24T00:48:51.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.981+0000 7f6e527fc640 1 -- 172.21.15.67:0/101657530 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v116) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6e5c07a7b0 con 0x7f6e6015dea0 2024-03-24T00:48:51.981 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-24T00:48:51.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.983+0000 7f6e6803f640 1 -- 172.21.15.67:0/101657530 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6e4406bbd0 msgr2=0x7f6e4406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:51.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.983+0000 7f6e6803f640 1 --2- 172.21.15.67:0/101657530 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6e4406bbd0 0x7f6e4406e090 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f6e54001680 tx=0x7f6e54009040 comp rx=0 tx=0).stop 2024-03-24T00:48:51.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.983+0000 7f6e6803f640 1 -- 172.21.15.67:0/101657530 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e6015dea0 msgr2=0x7f6e60159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:51.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.983+0000 7f6e6803f640 1 --2- 172.21.15.67:0/101657530 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e6015dea0 0x7f6e60159fb0 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c0099c0 tx=0x7f6e5c0046c0 comp rx=0 tx=0).stop 2024-03-24T00:48:51.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.983+0000 7f6e6803f640 1 -- 172.21.15.67:0/101657530 shutdown_connections 2024-03-24T00:48:51.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.983+0000 7f6e6803f640 1 --2- 172.21.15.67:0/101657530 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f6e4406bbd0 0x7f6e4406e090 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:51.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.983+0000 7f6e6803f640 1 --2- 172.21.15.67:0/101657530 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f6e6015e280 0x7f6e6014ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:51.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.983+0000 7f6e6803f640 1 --2- 172.21.15.67:0/101657530 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e6015dea0 0x7f6e60159fb0 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:51.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.983+0000 7f6e6803f640 1 --2- 172.21.15.67:0/101657530 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6e600d3ae0 0x7f6e600c9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:51.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.983+0000 7f6e6803f640 1 -- 172.21.15.67:0/101657530 >> 172.21.15.67:0/101657530 conn(0x7f6e60072da0 msgr2=0x7f6e600b42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:51.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.984+0000 7f6e6803f640 1 -- 172.21.15.67:0/101657530 shutdown_connections 2024-03-24T00:48:51.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:51.984+0000 7f6e6803f640 1 -- 172.21.15.67:0/101657530 wait complete. 2024-03-24T00:48:51.994 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-24T00:48:52.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.096+0000 7fe546280640 1 Processor -- start 2024-03-24T00:48:52.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.096+0000 7fe546280640 1 -- start start 2024-03-24T00:48:52.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.097+0000 7fe546280640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe54015f410 0x7fe5400cf7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:52.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.097+0000 7fe546280640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe5400d0940 0x7fe5400cfd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:52.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.097+0000 7fe546280640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe540099d70 0x7fe5400d0240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:52.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.097+0000 7fe546280640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5400cdb50 con 0x7fe540099d70 2024-03-24T00:48:52.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.097+0000 7fe546280640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5400cd2b0 con 0x7fe54015f410 2024-03-24T00:48:52.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.097+0000 7fe546280640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5400cd590 con 0x7fe5400d0940 2024-03-24T00:48:52.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.097+0000 7fe54481d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe540099d70 0x7fe5400d0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:52.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.097+0000 7fe54481d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe540099d70 0x7fe5400d0240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34874/0 (socket says 172.21.15.67:34874) 2024-03-24T00:48:52.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.097+0000 7fe54481d640 1 -- 172.21.15.67:0/932155927 learned_addr learned my addr 172.21.15.67:0/932155927 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:52.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.097+0000 7fe53ffff640 1 --2- 172.21.15.67:0/932155927 >> v2:172.21.15.72:3300/0 conn(0x7fe54015f410 0x7fe5400cf7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:52.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.097+0000 7fe53f7fe640 1 --2- 172.21.15.67:0/932155927 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe5400d0940 0x7fe5400cfd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:52.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.097+0000 7fe54481d640 1 -- 172.21.15.67:0/932155927 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe5400d0940 msgr2=0x7fe5400cfd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:52.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.097+0000 7fe54481d640 1 --2- 172.21.15.67:0/932155927 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe5400d0940 0x7fe5400cfd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:52.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.097+0000 7fe54481d640 1 -- 172.21.15.67:0/932155927 >> v2:172.21.15.72:3300/0 conn(0x7fe54015f410 msgr2=0x7fe5400cf7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:52.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.097+0000 7fe54481d640 1 --2- 172.21.15.67:0/932155927 >> v2:172.21.15.72:3300/0 conn(0x7fe54015f410 0x7fe5400cf7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:52.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.097+0000 7fe54481d640 1 -- 172.21.15.67:0/932155927 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5400cd870 con 0x7fe540099d70 2024-03-24T00:48:52.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.097+0000 7fe53ffff640 1 --2- 172.21.15.67:0/932155927 >> v2:172.21.15.72:3300/0 conn(0x7fe54015f410 0x7fe5400cf7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:48:52.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.097+0000 7fe53f7fe640 1 --2- 172.21.15.67:0/932155927 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe5400d0940 0x7fe5400cfd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:48:52.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.098+0000 7fe54481d640 1 --2- 172.21.15.67:0/932155927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe540099d70 0x7fe5400d0240 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7fe5400733e0 tx=0x7fe53802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bb6ebcf1d8dc3c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:52.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.098+0000 7fe53effd640 1 -- 172.21.15.67:0/932155927 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53802ca70 con 0x7fe540099d70 2024-03-24T00:48:52.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.098+0000 7fe53effd640 1 -- 172.21.15.67:0/932155927 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe53802cc10 con 0x7fe540099d70 2024-03-24T00:48:52.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.098+0000 7fe53effd640 1 -- 172.21.15.67:0/932155927 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5380343f0 con 0x7fe540099d70 2024-03-24T00:48:52.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.099+0000 7fe546280640 1 -- 172.21.15.67:0/932155927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe540099d70 msgr2=0x7fe5400d0240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:52.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.099+0000 7fe546280640 1 --2- 172.21.15.67:0/932155927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe540099d70 0x7fe5400d0240 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7fe5400733e0 tx=0x7fe53802bed0 comp rx=0 tx=0).stop 2024-03-24T00:48:52.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.099+0000 7fe546280640 1 -- 172.21.15.67:0/932155927 shutdown_connections 2024-03-24T00:48:52.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.099+0000 7fe546280640 1 --2- 172.21.15.67:0/932155927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe540099d70 0x7fe5400d0240 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:52.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.099+0000 7fe546280640 1 --2- 172.21.15.67:0/932155927 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe5400d0940 0x7fe5400cfd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:52.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.099+0000 7fe546280640 1 --2- 172.21.15.67:0/932155927 >> v2:172.21.15.72:3300/0 conn(0x7fe54015f410 0x7fe5400cf7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:52.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.099+0000 7fe546280640 1 -- 172.21.15.67:0/932155927 >> 172.21.15.67:0/932155927 conn(0x7fe5400b36e0 msgr2=0x7fe5400b3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:52.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.099+0000 7fe546280640 1 -- 172.21.15.67:0/932155927 shutdown_connections 2024-03-24T00:48:52.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.099+0000 7fe546280640 1 -- 172.21.15.67:0/932155927 wait complete. 2024-03-24T00:48:52.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.100+0000 7fe546280640 1 Processor -- start 2024-03-24T00:48:52.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.100+0000 7fe546280640 1 -- start start 2024-03-24T00:48:52.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.101+0000 7fe546280640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe540099d70 0x7fe54014feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:52.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.101+0000 7fe546280640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe54015f410 0x7fe54015b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:52.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.101+0000 7fe546280640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe54015b680 0x7fe5400af9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:52.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.101+0000 7fe546280640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54015f7f0 con 0x7fe54015b680 2024-03-24T00:48:52.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.101+0000 7fe546280640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54015fb70 con 0x7fe540099d70 2024-03-24T00:48:52.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.101+0000 7fe546280640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5400c7740 con 0x7fe54015f410 2024-03-24T00:48:52.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.101+0000 7fe54481d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe54015b680 0x7fe5400af9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:52.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.101+0000 7fe54481d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe54015b680 0x7fe5400af9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34882/0 (socket says 172.21.15.67:34882) 2024-03-24T00:48:52.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.101+0000 7fe54481d640 1 -- 172.21.15.67:0/1576200786 learned_addr learned my addr 172.21.15.67:0/1576200786 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:52.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.101+0000 7fe53f7fe640 1 --2- 172.21.15.67:0/1576200786 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe54015f410 0x7fe54015b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:52.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.101+0000 7fe53ffff640 1 --2- 172.21.15.67:0/1576200786 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe540099d70 0x7fe54014feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:52.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.101+0000 7fe54481d640 1 -- 172.21.15.67:0/1576200786 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe54015f410 msgr2=0x7fe54015b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:52.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.101+0000 7fe54481d640 1 --2- 172.21.15.67:0/1576200786 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe54015f410 0x7fe54015b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:52.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.101+0000 7fe54481d640 1 -- 172.21.15.67:0/1576200786 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe540099d70 msgr2=0x7fe54014feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:52.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.101+0000 7fe54481d640 1 --2- 172.21.15.67:0/1576200786 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe540099d70 0x7fe54014feb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:52.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.101+0000 7fe54481d640 1 -- 172.21.15.67:0/1576200786 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5400c8ff0 con 0x7fe54015b680 2024-03-24T00:48:52.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.102+0000 7fe54481d640 1 --2- 172.21.15.67:0/1576200786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe54015b680 0x7fe5400af9d0 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7fe538009f40 tx=0x7fe538004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:52.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.102+0000 7fe53cff9640 1 -- 172.21.15.67:0/1576200786 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe538043020 con 0x7fe54015b680 2024-03-24T00:48:52.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.102+0000 7fe53cff9640 1 -- 172.21.15.67:0/1576200786 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe53803e450 con 0x7fe54015b680 2024-03-24T00:48:52.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.102+0000 7fe546280640 1 -- 172.21.15.67:0/1576200786 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5400c6b30 con 0x7fe54015b680 2024-03-24T00:48:52.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.102+0000 7fe53cff9640 1 -- 172.21.15.67:0/1576200786 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe538009510 con 0x7fe54015b680 2024-03-24T00:48:52.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.102+0000 7fe546280640 1 -- 172.21.15.67:0/1576200786 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5400c70f0 con 0x7fe54015b680 2024-03-24T00:48:52.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.103+0000 7fe546280640 1 -- 172.21.15.67:0/1576200786 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe54004f400 con 0x7fe54015b680 2024-03-24T00:48:52.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.103+0000 7fe53cff9640 1 -- 172.21.15.67:0/1576200786 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe538046020 con 0x7fe54015b680 2024-03-24T00:48:52.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.103+0000 7fe53cff9640 1 --2- 172.21.15.67:0/1576200786 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe52006bc00 0x7fe52006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:52.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.104+0000 7fe53cff9640 1 -- 172.21.15.67:0/1576200786 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fe5380ad410 con 0x7fe54015b680 2024-03-24T00:48:52.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.104+0000 7fe53ffff640 1 --2- 172.21.15.67:0/1576200786 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe52006bc00 0x7fe52006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:52.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.104+0000 7fe53ffff640 1 --2- 172.21.15.67:0/1576200786 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe52006bc00 0x7fe52006e0c0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fe5280096d0 tx=0x7fe528008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:52.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.108+0000 7fe53cff9640 1 -- 172.21.15.67:0/1576200786 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe53804d020 con 0x7fe54015b680 2024-03-24T00:48:52.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.292+0000 7fe546280640 1 -- 172.21.15.67:0/1576200786 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fe5400d19d0 con 0x7fe54015b680 2024-03-24T00:48:52.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.292+0000 7fe53cff9640 1 -- 172.21.15.67:0/1576200786 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v116) v1 ==== 84+0+65 (secure 0 0 0) 0x7fe5400c8ff0 con 0x7fe54015b680 2024-03-24T00:48:52.292 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:48:52.292 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-03-24T00:48:52.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-24T00:48:52.293 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2024-03-24T00:48:52.293 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2024-03-24T00:48:52.293 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-24T00:48:52.293 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-03-24T00:48:52.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.294+0000 7fe546280640 1 -- 172.21.15.67:0/1576200786 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe52006bc00 msgr2=0x7fe52006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:52.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.294+0000 7fe546280640 1 --2- 172.21.15.67:0/1576200786 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe52006bc00 0x7fe52006e0c0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fe5280096d0 tx=0x7fe528008040 comp rx=0 tx=0).stop 2024-03-24T00:48:52.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.294+0000 7fe546280640 1 -- 172.21.15.67:0/1576200786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe54015b680 msgr2=0x7fe5400af9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:52.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.294+0000 7fe546280640 1 --2- 172.21.15.67:0/1576200786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe54015b680 0x7fe5400af9d0 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7fe538009f40 tx=0x7fe538004a40 comp rx=0 tx=0).stop 2024-03-24T00:48:52.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.294+0000 7fe546280640 1 -- 172.21.15.67:0/1576200786 shutdown_connections 2024-03-24T00:48:52.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.294+0000 7fe546280640 1 --2- 172.21.15.67:0/1576200786 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe52006bc00 0x7fe52006e0c0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:52.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.294+0000 7fe546280640 1 --2- 172.21.15.67:0/1576200786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe54015b680 0x7fe5400af9d0 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:52.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.294+0000 7fe546280640 1 --2- 172.21.15.67:0/1576200786 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe54015f410 0x7fe54015b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:52.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.294+0000 7fe546280640 1 --2- 172.21.15.67:0/1576200786 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe540099d70 0x7fe54014feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:52.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.294+0000 7fe546280640 1 -- 172.21.15.67:0/1576200786 >> 172.21.15.67:0/1576200786 conn(0x7fe5400b36e0 msgr2=0x7fe540159fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:52.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.294+0000 7fe546280640 1 -- 172.21.15.67:0/1576200786 shutdown_connections 2024-03-24T00:48:52.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.294+0000 7fe546280640 1 -- 172.21.15.67:0/1576200786 wait complete. 2024-03-24T00:48:52.305 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-24T00:48:52.305 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-24T00:48:52.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.410+0000 7fbc1247e640 1 Processor -- start 2024-03-24T00:48:52.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.410+0000 7fbc1247e640 1 -- start start 2024-03-24T00:48:52.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.411+0000 7fbc1247e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbc0c0aeee0 0x7fbc0c0cef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:52.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.411+0000 7fbc1247e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc0c072300 con 0x7fbc0c0cd340 2024-03-24T00:48:52.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.411+0000 7fbc1247e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc0c0725e0 con 0x7fbc0c071ad0 2024-03-24T00:48:52.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.411+0000 7fbc1247e640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc0c06db30 con 0x7fbc0c0aeee0 2024-03-24T00:48:52.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.411+0000 7fbc0bfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbc0c0aeee0 0x7fbc0c0cef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:52.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.411+0000 7fbc10a1b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbc0c0cd340 0x7fbc0c06dfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48630/0 (socket says 172.21.15.67:48630) 2024-03-24T00:48:52.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.411+0000 7fbc10a1b640 1 -- 172.21.15.67:0/1385226838 learned_addr learned my addr 172.21.15.67:0/1385226838 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:48:52.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.411+0000 7fbc0affd640 1 -- 172.21.15.67:0/1385226838 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3782122822 0 0) 0x7fbc0c072300 con 0x7fbc0c0cd340 2024-03-24T00:48:52.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.411+0000 7fbc0affd640 1 -- 172.21.15.67:0/1385226838 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbf4003520 con 0x7fbc0c0cd340 2024-03-24T00:48:52.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.411+0000 7fbc0bfff640 1 -- 172.21.15.67:0/1385226838 >> v1:172.21.15.72:6789/0 conn(0x7fbc0c071ad0 legacy=0x7fbc0c0ccc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:52.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.411+0000 7fbc0bfff640 1 -- 172.21.15.67:0/1385226838 >> v1:172.21.15.67:6789/0 conn(0x7fbc0c0cd340 legacy=0x7fbc0c06dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:52.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.411+0000 7fbc0bfff640 1 -- 172.21.15.67:0/1385226838 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc0c0728c0 con 0x7fbc0c0aeee0 2024-03-24T00:48:52.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.412+0000 7fbc0bfff640 1 --2- 172.21.15.67:0/1385226838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc0c0aeee0 0x7fbc0c0cef40 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7fbc000097a0 tx=0x7fbc0002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf12f9f49167c312 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:52.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.412+0000 7fbc0affd640 1 -- 172.21.15.67:0/1385226838 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc00035020 con 0x7fbc0c0aeee0 2024-03-24T00:48:52.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.412+0000 7fbc0affd640 1 -- 172.21.15.67:0/1385226838 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc00031830 con 0x7fbc0c0aeee0 2024-03-24T00:48:52.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.412+0000 7fbc0affd640 1 -- 172.21.15.67:0/1385226838 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc00031c00 con 0x7fbc0c0aeee0 2024-03-24T00:48:52.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.413+0000 7fbc1247e640 1 -- 172.21.15.67:0/1385226838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc0c0aeee0 msgr2=0x7fbc0c0cef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:52.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.413+0000 7fbc1247e640 1 --2- 172.21.15.67:0/1385226838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc0c0aeee0 0x7fbc0c0cef40 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7fbc000097a0 tx=0x7fbc0002b8d0 comp rx=0 tx=0).stop 2024-03-24T00:48:52.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.413+0000 7fbc1247e640 1 -- 172.21.15.67:0/1385226838 shutdown_connections 2024-03-24T00:48:52.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.413+0000 7fbc1247e640 1 --2- 172.21.15.67:0/1385226838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc0c0aeee0 0x7fbc0c0cef40 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:52.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.413+0000 7fbc1247e640 1 -- 172.21.15.67:0/1385226838 >> 172.21.15.67:0/1385226838 conn(0x7fbc0c0cf9e0 msgr2=0x7fbc0c0cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:52.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.413+0000 7fbc1247e640 1 -- 172.21.15.67:0/1385226838 shutdown_connections 2024-03-24T00:48:52.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.413+0000 7fbc1247e640 1 -- 172.21.15.67:0/1385226838 wait complete. 2024-03-24T00:48:52.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.414+0000 7fbc1247e640 1 Processor -- start 2024-03-24T00:48:52.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.414+0000 7fbc1247e640 1 -- start start 2024-03-24T00:48:52.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.415+0000 7fbc1247e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc0c071ad0 0x7fbc0c1f5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:52.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.415+0000 7fbc1247e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc0c0aeee0 0x7fbc0c1f0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:52.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.415+0000 7fbc1247e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbc0c0cd340 0x7fbc0c143190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:52.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.415+0000 7fbc1247e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0c0720c0 con 0x7fbc0c0aeee0 2024-03-24T00:48:52.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.415+0000 7fbc1247e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0c071f40 con 0x7fbc0c071ad0 2024-03-24T00:48:52.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.415+0000 7fbc1247e640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0c06e6d0 con 0x7fbc0c0cd340 2024-03-24T00:48:52.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.415+0000 7fbc0b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc0c0aeee0 0x7fbc0c1f0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:52.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.415+0000 7fbc0b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc0c0aeee0 0x7fbc0c1f0e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34900/0 (socket says 172.21.15.67:34900) 2024-03-24T00:48:52.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.415+0000 7fbc0b7fe640 1 -- 172.21.15.67:0/3568522086 learned_addr learned my addr 172.21.15.67:0/3568522086 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:52.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.415+0000 7fbc0bfff640 1 --2- 172.21.15.67:0/3568522086 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc0c071ad0 0x7fbc0c1f5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:52.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.415+0000 7fbc10a1b640 1 --2- 172.21.15.67:0/3568522086 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbc0c0cd340 0x7fbc0c143190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:52.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.415+0000 7fbc0b7fe640 1 -- 172.21.15.67:0/3568522086 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbc0c0cd340 msgr2=0x7fbc0c143190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:52.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.415+0000 7fbc0b7fe640 1 --2- 172.21.15.67:0/3568522086 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbc0c0cd340 0x7fbc0c143190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:52.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.415+0000 7fbc0b7fe640 1 -- 172.21.15.67:0/3568522086 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc0c071ad0 msgr2=0x7fbc0c1f5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:52.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.415+0000 7fbc0b7fe640 1 --2- 172.21.15.67:0/3568522086 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc0c071ad0 0x7fbc0c1f5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:52.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.415+0000 7fbc0b7fe640 1 -- 172.21.15.67:0/3568522086 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc0c1f59b0 con 0x7fbc0c0aeee0 2024-03-24T00:48:52.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.415+0000 7fbc0b7fe640 1 --2- 172.21.15.67:0/3568522086 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc0c0aeee0 0x7fbc0c1f0e60 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc00c460 tx=0x7fbbfc00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:52.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.416+0000 7fbc08ff9640 1 -- 172.21.15.67:0/3568522086 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc016020 con 0x7fbc0c0aeee0 2024-03-24T00:48:52.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.416+0000 7fbc08ff9640 1 -- 172.21.15.67:0/3568522086 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbfc012870 con 0x7fbc0c0aeee0 2024-03-24T00:48:52.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.416+0000 7fbc1247e640 1 -- 172.21.15.67:0/3568522086 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc0c1f13a0 con 0x7fbc0c0aeee0 2024-03-24T00:48:52.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.416+0000 7fbc08ff9640 1 -- 172.21.15.67:0/3568522086 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc012c60 con 0x7fbc0c0aeee0 2024-03-24T00:48:52.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.416+0000 7fbc1247e640 1 -- 172.21.15.67:0/3568522086 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc0c14d010 con 0x7fbc0c0aeee0 2024-03-24T00:48:52.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.417+0000 7fbc1247e640 1 -- 172.21.15.67:0/3568522086 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc0c04f400 con 0x7fbc0c0aeee0 2024-03-24T00:48:52.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.417+0000 7fbc08ff9640 1 -- 172.21.15.67:0/3568522086 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fbbfc019020 con 0x7fbc0c0aeee0 2024-03-24T00:48:52.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.418+0000 7fbc08ff9640 1 --2- 172.21.15.67:0/3568522086 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbbf806bbd0 0x7fbbf806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:52.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.418+0000 7fbc08ff9640 1 -- 172.21.15.67:0/3568522086 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fbbfc08a320 con 0x7fbc0c0aeee0 2024-03-24T00:48:52.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.418+0000 7fbc0bfff640 1 --2- 172.21.15.67:0/3568522086 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbbf806bbd0 0x7fbbf806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:52.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.418+0000 7fbc0bfff640 1 --2- 172.21.15.67:0/3568522086 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbbf806bbd0 0x7fbbf806e090 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fbc00036000 tx=0x7fbc0003c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:52.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.422+0000 7fbc08ff9640 1 -- 172.21.15.67:0/3568522086 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fbbfc020070 con 0x7fbc0c0aeee0 2024-03-24T00:48:52.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.603+0000 7fbc1247e640 1 -- 172.21.15.67:0/3568522086 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fbc0c0d0680 con 0x7fbc0c0aeee0 2024-03-24T00:48:52.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.969+0000 7fbc08ff9640 1 -- 172.21.15.67:0/3568522086 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v117) v1 ==== 138+0+0 (secure 0 0 0) 0x7fbbfc058fd0 con 0x7fbc0c0aeee0 2024-03-24T00:48:52.969 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' created 2024-03-24T00:48:52.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.971+0000 7fbc1247e640 1 -- 172.21.15.67:0/3568522086 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbbf806bbd0 msgr2=0x7fbbf806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:52.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.971+0000 7fbc1247e640 1 --2- 172.21.15.67:0/3568522086 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbbf806bbd0 0x7fbbf806e090 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fbc00036000 tx=0x7fbc0003c040 comp rx=0 tx=0).stop 2024-03-24T00:48:52.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.971+0000 7fbc1247e640 1 -- 172.21.15.67:0/3568522086 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc0c0aeee0 msgr2=0x7fbc0c1f0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:52.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.971+0000 7fbc1247e640 1 --2- 172.21.15.67:0/3568522086 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc0c0aeee0 0x7fbc0c1f0e60 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc00c460 tx=0x7fbbfc00c920 comp rx=0 tx=0).stop 2024-03-24T00:48:52.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.972+0000 7fbc1247e640 1 -- 172.21.15.67:0/3568522086 shutdown_connections 2024-03-24T00:48:52.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.972+0000 7fbc1247e640 1 --2- 172.21.15.67:0/3568522086 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fbbf806bbd0 0x7fbbf806e090 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:52.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.972+0000 7fbc1247e640 1 --2- 172.21.15.67:0/3568522086 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbc0c0cd340 0x7fbc0c143190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:52.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.972+0000 7fbc1247e640 1 --2- 172.21.15.67:0/3568522086 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc0c0aeee0 0x7fbc0c1f0e60 unknown :-1 s=CLOSED pgs=1693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:52.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.972+0000 7fbc1247e640 1 --2- 172.21.15.67:0/3568522086 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc0c071ad0 0x7fbc0c1f5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:52.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.972+0000 7fbc1247e640 1 -- 172.21.15.67:0/3568522086 >> 172.21.15.67:0/3568522086 conn(0x7fbc0c0cf9e0 msgr2=0x7fbc0c144ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:52.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.972+0000 7fbc1247e640 1 -- 172.21.15.67:0/3568522086 shutdown_connections 2024-03-24T00:48:52.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:52.972+0000 7fbc1247e640 1 -- 172.21.15.67:0/3568522086 wait complete. 2024-03-24T00:48:52.987 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-24T00:48:53.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.097+0000 7f3be6bf3640 1 Processor -- start 2024-03-24T00:48:53.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.097+0000 7f3be6bf3640 1 -- start start 2024-03-24T00:48:53.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.098+0000 7f3be6bf3640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3be006b040 0x7f3be015ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:53.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.098+0000 7f3be6bf3640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3be015c1e0 0x7f3be006b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:53.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.098+0000 7f3be6bf3640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3be00ce060 0x7f3be00cc620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:53.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.098+0000 7f3be6bf3640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3be00b24b0 con 0x7f3be015c1e0 2024-03-24T00:48:53.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.098+0000 7f3be6bf3640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3be015c9d0 con 0x7f3be006b040 2024-03-24T00:48:53.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.098+0000 7f3be6bf3640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3be006b960 con 0x7f3be00ce060 2024-03-24T00:48:53.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.098+0000 7f3bdffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3be015c1e0 0x7f3be006b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:53.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.098+0000 7f3bdffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3be015c1e0 0x7f3be006b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34904/0 (socket says 172.21.15.67:34904) 2024-03-24T00:48:53.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.098+0000 7f3bdffff640 1 -- 172.21.15.67:0/2412935983 learned_addr learned my addr 172.21.15.67:0/2412935983 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:53.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.098+0000 7f3be5190640 1 --2- 172.21.15.67:0/2412935983 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3be00ce060 0x7f3be00cc620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:53.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.098+0000 7f3be498f640 1 --2- 172.21.15.67:0/2412935983 >> v2:172.21.15.72:3300/0 conn(0x7f3be006b040 0x7f3be015ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:53.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.098+0000 7f3bdffff640 1 -- 172.21.15.67:0/2412935983 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3be00ce060 msgr2=0x7f3be00cc620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:53.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.098+0000 7f3bdffff640 1 --2- 172.21.15.67:0/2412935983 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3be00ce060 0x7f3be00cc620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:53.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.099+0000 7f3bdffff640 1 -- 172.21.15.67:0/2412935983 >> v2:172.21.15.72:3300/0 conn(0x7f3be006b040 msgr2=0x7f3be015ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:53.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.099+0000 7f3bdffff640 1 --2- 172.21.15.67:0/2412935983 >> v2:172.21.15.72:3300/0 conn(0x7f3be006b040 0x7f3be015ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:53.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.099+0000 7f3bdffff640 1 -- 172.21.15.67:0/2412935983 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3be00ccee0 con 0x7f3be015c1e0 2024-03-24T00:48:53.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.099+0000 7f3bdffff640 1 --2- 172.21.15.67:0/2412935983 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3be015c1e0 0x7f3be006b420 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f3be0071210 tx=0x7f3bd402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73d20ab37632b16f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:53.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.099+0000 7f3bdf7fe640 1 -- 172.21.15.67:0/2412935983 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd4035020 con 0x7f3be015c1e0 2024-03-24T00:48:53.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.099+0000 7f3bdf7fe640 1 -- 172.21.15.67:0/2412935983 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bd4031870 con 0x7f3be015c1e0 2024-03-24T00:48:53.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.099+0000 7f3bdf7fe640 1 -- 172.21.15.67:0/2412935983 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd4031c40 con 0x7f3be015c1e0 2024-03-24T00:48:53.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.100+0000 7f3be6bf3640 1 -- 172.21.15.67:0/2412935983 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3be015c1e0 msgr2=0x7f3be006b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:53.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.100+0000 7f3be6bf3640 1 --2- 172.21.15.67:0/2412935983 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3be015c1e0 0x7f3be006b420 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f3be0071210 tx=0x7f3bd402b910 comp rx=0 tx=0).stop 2024-03-24T00:48:53.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.100+0000 7f3be6bf3640 1 -- 172.21.15.67:0/2412935983 shutdown_connections 2024-03-24T00:48:53.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.100+0000 7f3be6bf3640 1 --2- 172.21.15.67:0/2412935983 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3be00ce060 0x7f3be00cc620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:53.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.100+0000 7f3be6bf3640 1 --2- 172.21.15.67:0/2412935983 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3be015c1e0 0x7f3be006b420 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:53.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.100+0000 7f3be6bf3640 1 --2- 172.21.15.67:0/2412935983 >> v2:172.21.15.72:3300/0 conn(0x7f3be006b040 0x7f3be015ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:53.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.100+0000 7f3be6bf3640 1 -- 172.21.15.67:0/2412935983 >> 172.21.15.67:0/2412935983 conn(0x7f3be0099a30 msgr2=0x7f3be0099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:53.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.100+0000 7f3be6bf3640 1 -- 172.21.15.67:0/2412935983 shutdown_connections 2024-03-24T00:48:53.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.100+0000 7f3be6bf3640 1 -- 172.21.15.67:0/2412935983 wait complete. 2024-03-24T00:48:53.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.101+0000 7f3be6bf3640 1 Processor -- start 2024-03-24T00:48:53.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.101+0000 7f3be6bf3640 1 -- start start 2024-03-24T00:48:53.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.102+0000 7f3be6bf3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3be006b040 0x7f3be00c45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:53.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.102+0000 7f3be6bf3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3be00ce060 0x7f3be014a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:53.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.102+0000 7f3be6bf3640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3be015c1e0 0x7f3be01f7d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:53.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.102+0000 7f3be6bf3640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3be00ccb60 con 0x7f3be006b040 2024-03-24T00:48:53.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.102+0000 7f3be6bf3640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3be015c660 con 0x7f3be00ce060 2024-03-24T00:48:53.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.102+0000 7f3be6bf3640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3be015f920 con 0x7f3be015c1e0 2024-03-24T00:48:53.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.102+0000 7f3be498f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3be006b040 0x7f3be00c45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:53.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.102+0000 7f3be498f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3be006b040 0x7f3be00c45d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34906/0 (socket says 172.21.15.67:34906) 2024-03-24T00:48:53.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.102+0000 7f3be498f640 1 -- 172.21.15.67:0/3299875753 learned_addr learned my addr 172.21.15.67:0/3299875753 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:53.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.102+0000 7f3be5190640 1 --2- 172.21.15.67:0/3299875753 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3be015c1e0 0x7f3be01f7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:53.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.102+0000 7f3bdffff640 1 --2- 172.21.15.67:0/3299875753 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3be00ce060 0x7f3be014a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:53.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.102+0000 7f3be498f640 1 -- 172.21.15.67:0/3299875753 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3be015c1e0 msgr2=0x7f3be01f7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:53.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.102+0000 7f3be498f640 1 --2- 172.21.15.67:0/3299875753 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3be015c1e0 0x7f3be01f7d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:53.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.102+0000 7f3be498f640 1 -- 172.21.15.67:0/3299875753 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3be00ce060 msgr2=0x7f3be014a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:53.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.102+0000 7f3be498f640 1 --2- 172.21.15.67:0/3299875753 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3be00ce060 0x7f3be014a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:53.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.102+0000 7f3be498f640 1 -- 172.21.15.67:0/3299875753 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3be014a6f0 con 0x7f3be006b040 2024-03-24T00:48:53.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.102+0000 7f3be5190640 1 --2- 172.21.15.67:0/3299875753 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3be015c1e0 0x7f3be01f7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:48:53.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.103+0000 7f3be498f640 1 --2- 172.21.15.67:0/3299875753 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3be006b040 0x7f3be00c45d0 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f3bd000c7f0 tx=0x7f3bd000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:53.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.103+0000 7f3bdd7fa640 1 -- 172.21.15.67:0/3299875753 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd0015070 con 0x7f3be006b040 2024-03-24T00:48:53.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.103+0000 7f3bdd7fa640 1 -- 172.21.15.67:0/3299875753 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bd0004220 con 0x7f3be006b040 2024-03-24T00:48:53.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.103+0000 7f3be6bf3640 1 -- 172.21.15.67:0/3299875753 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3be00c63e0 con 0x7f3be006b040 2024-03-24T00:48:53.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.103+0000 7f3bdd7fa640 1 -- 172.21.15.67:0/3299875753 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd0007620 con 0x7f3be006b040 2024-03-24T00:48:53.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.103+0000 7f3be6bf3640 1 -- 172.21.15.67:0/3299875753 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3be00c69a0 con 0x7f3be006b040 2024-03-24T00:48:53.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.104+0000 7f3be6bf3640 1 -- 172.21.15.67:0/3299875753 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3be004f400 con 0x7f3be006b040 2024-03-24T00:48:53.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.104+0000 7f3bdd7fa640 1 -- 172.21.15.67:0/3299875753 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3bd00079d0 con 0x7f3be006b040 2024-03-24T00:48:53.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.105+0000 7f3bdd7fa640 1 --2- 172.21.15.67:0/3299875753 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3bcc06bbd0 0x7f3bcc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:53.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.105+0000 7f3bdd7fa640 1 -- 172.21.15.67:0/3299875753 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f3bd008aae0 con 0x7f3be006b040 2024-03-24T00:48:53.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.105+0000 7f3bdffff640 1 --2- 172.21.15.67:0/3299875753 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3bcc06bbd0 0x7f3bcc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:53.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.105+0000 7f3bdffff640 1 --2- 172.21.15.67:0/3299875753 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3bcc06bbd0 0x7f3bcc06e090 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f3bd4036000 tx=0x7f3bd403c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:53.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.109+0000 7f3bdd7fa640 1 -- 172.21.15.67:0/3299875753 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3bd0059590 con 0x7f3be006b040 2024-03-24T00:48:53.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.292+0000 7f3be6bf3640 1 -- 172.21.15.67:0/3299875753 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f3be00cd830 con 0x7f3be006b040 2024-03-24T00:48:53.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.974+0000 7f3bdd7fa640 1 -- 172.21.15.67:0/3299875753 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v118) v1 ==== 170+0+0 (secure 0 0 0) 0x7f3bd0058e90 con 0x7f3be006b040 2024-03-24T00:48:53.975 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' created 2024-03-24T00:48:53.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.979+0000 7f3be6bf3640 1 -- 172.21.15.67:0/3299875753 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3bcc06bbd0 msgr2=0x7f3bcc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:53.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.979+0000 7f3be6bf3640 1 --2- 172.21.15.67:0/3299875753 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3bcc06bbd0 0x7f3bcc06e090 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f3bd4036000 tx=0x7f3bd403c040 comp rx=0 tx=0).stop 2024-03-24T00:48:53.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.979+0000 7f3be6bf3640 1 -- 172.21.15.67:0/3299875753 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3be006b040 msgr2=0x7f3be00c45d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:53.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.979+0000 7f3be6bf3640 1 --2- 172.21.15.67:0/3299875753 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3be006b040 0x7f3be00c45d0 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f3bd000c7f0 tx=0x7f3bd000ccb0 comp rx=0 tx=0).stop 2024-03-24T00:48:53.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.979+0000 7f3be6bf3640 1 -- 172.21.15.67:0/3299875753 shutdown_connections 2024-03-24T00:48:53.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.979+0000 7f3be6bf3640 1 --2- 172.21.15.67:0/3299875753 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3bcc06bbd0 0x7f3bcc06e090 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:53.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.979+0000 7f3be6bf3640 1 --2- 172.21.15.67:0/3299875753 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3be015c1e0 0x7f3be01f7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:53.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.979+0000 7f3be6bf3640 1 --2- 172.21.15.67:0/3299875753 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3be00ce060 0x7f3be014a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:53.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.979+0000 7f3be6bf3640 1 --2- 172.21.15.67:0/3299875753 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3be006b040 0x7f3be00c45d0 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:53.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.979+0000 7f3be6bf3640 1 -- 172.21.15.67:0/3299875753 >> 172.21.15.67:0/3299875753 conn(0x7f3be0099a30 msgr2=0x7f3be015e070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:53.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.981+0000 7f3be6bf3640 1 -- 172.21.15.67:0/3299875753 shutdown_connections 2024-03-24T00:48:53.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:53.984+0000 7f3be6bf3640 1 -- 172.21.15.67:0/3299875753 wait complete. 2024-03-24T00:48:54.007 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-24T00:48:54.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.117+0000 7f11fdb87640 1 Processor -- start 2024-03-24T00:48:54.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.117+0000 7f11fdb87640 1 -- start start 2024-03-24T00:48:54.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.118+0000 7f11fdb87640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f11f80b4a00 0x7f11f80b3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:54.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.118+0000 7f11fdb87640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11f80b2ab0 con 0x7f11f806ec60 2024-03-24T00:48:54.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.118+0000 7f11fdb87640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11f80b2d90 con 0x7f11f8144670 2024-03-24T00:48:54.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.118+0000 7f11fdb87640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11f80b3070 con 0x7f11f80b4a00 2024-03-24T00:48:54.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.118+0000 7f11f7fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f11f806ec60 0x7f11f806f080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35888/0 (socket says 172.21.15.67:35888) 2024-03-24T00:48:54.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.118+0000 7f11f7fff640 1 -- 172.21.15.67:0/271936315 learned_addr learned my addr 172.21.15.67:0/271936315 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:48:54.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.118+0000 7f11f6ffd640 1 --2- 172.21.15.67:0/271936315 >> v2:172.21.15.72:3300/0 conn(0x7f11f80b4a00 0x7f11f80b3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:54.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.118+0000 7f11f67fc640 1 -- 172.21.15.67:0/271936315 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3922473926 0 0) 0x7f11f80b2ab0 con 0x7f11f806ec60 2024-03-24T00:48:54.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.118+0000 7f11f67fc640 1 -- 172.21.15.67:0/271936315 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11e0003520 con 0x7f11f806ec60 2024-03-24T00:48:54.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.118+0000 7f11f67fc640 1 -- 172.21.15.67:0/271936315 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 226133894 0 0) 0x7f11e0003520 con 0x7f11f806ec60 2024-03-24T00:48:54.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.118+0000 7f11f67fc640 1 -- 172.21.15.67:0/271936315 >> v2:172.21.15.72:3300/0 conn(0x7f11f80b4a00 msgr2=0x7f11f80b3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:54.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.118+0000 7f11f67fc640 1 --2- 172.21.15.67:0/271936315 >> v2:172.21.15.72:3300/0 conn(0x7f11f80b4a00 0x7f11f80b3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.118+0000 7f11f67fc640 1 -- 172.21.15.67:0/271936315 >> v1:172.21.15.72:6789/0 conn(0x7f11f8144670 legacy=0x7f11f8072500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:54.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.118+0000 7f11f67fc640 1 -- 172.21.15.67:0/271936315 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11f80b3350 con 0x7f11f806ec60 2024-03-24T00:48:54.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.118+0000 7f11f67fc640 1 -- 172.21.15.67:0/271936315 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f11f00019f0 con 0x7f11f806ec60 2024-03-24T00:48:54.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.118+0000 7f11f67fc640 1 -- 172.21.15.67:0/271936315 >> v1:172.21.15.67:6789/0 conn(0x7f11f806ec60 legacy=0x7f11f806f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:54.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.118+0000 7f11f67fc640 1 --2- 172.21.15.67:0/271936315 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f11e00039b0 0x7f11e0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:54.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.118+0000 7f11f67fc640 1 --2- 172.21.15.67:0/271936315 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11e0004800 0x7f11e0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:54.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.118+0000 7f11f67fc640 1 --2- 172.21.15.67:0/271936315 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f11e00051c0 0x7f11e000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:54.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.118+0000 7f11f67fc640 1 -- 172.21.15.67:0/271936315 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11f80b3070 con 0x7f11e0004800 2024-03-24T00:48:54.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.118+0000 7f11f67fc640 1 -- 172.21.15.67:0/271936315 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e000c060 con 0x7f11e00039b0 2024-03-24T00:48:54.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.118+0000 7f11f67fc640 1 -- 172.21.15.67:0/271936315 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11e000c260 con 0x7f11e00051c0 2024-03-24T00:48:54.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.118+0000 7f11f6ffd640 1 --2- 172.21.15.67:0/271936315 >> v2:172.21.15.72:3300/0 conn(0x7f11f80b4a00 0x7f11f80b3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:48:54.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.119+0000 7f11f77fe640 1 --2- 172.21.15.67:0/271936315 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f11e00039b0 0x7f11e0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:54.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.119+0000 7f11f6ffd640 1 --2- 172.21.15.67:0/271936315 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11e0004800 0x7f11e0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:54.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.119+0000 7f11f7fff640 1 --2- 172.21.15.67:0/271936315 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f11e00051c0 0x7f11e000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:54.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.119+0000 7f11f77fe640 1 -- 172.21.15.67:0/271936315 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f11e00051c0 msgr2=0x7f11e000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:54.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.119+0000 7f11f77fe640 1 --2- 172.21.15.67:0/271936315 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f11e00051c0 0x7f11e000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.119+0000 7f11f77fe640 1 -- 172.21.15.67:0/271936315 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11e0004800 msgr2=0x7f11e0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:54.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.119+0000 7f11f77fe640 1 --2- 172.21.15.67:0/271936315 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11e0004800 0x7f11e0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.119+0000 7f11f77fe640 1 -- 172.21.15.67:0/271936315 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11e0003520 con 0x7f11e00039b0 2024-03-24T00:48:54.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.119+0000 7f11f7fff640 1 -- 172.21.15.67:0/271936315 reap_dead start 2024-03-24T00:48:54.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.119+0000 7f11f6ffd640 1 --2- 172.21.15.67:0/271936315 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11e0004800 0x7f11e0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:48:54.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.120+0000 7f11f77fe640 1 --2- 172.21.15.67:0/271936315 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f11e00039b0 0x7f11e0003dd0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f11ec001750 tx=0x7f11e0005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1b65929ee59abbe server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:54.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.120+0000 7f11f67fc640 1 -- 172.21.15.67:0/271936315 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ec001370 con 0x7f11e00039b0 2024-03-24T00:48:54.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.120+0000 7f11f67fc640 1 -- 172.21.15.67:0/271936315 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11ec0052d0 con 0x7f11e00039b0 2024-03-24T00:48:54.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.120+0000 7f11f67fc640 1 -- 172.21.15.67:0/271936315 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ec003830 con 0x7f11e00039b0 2024-03-24T00:48:54.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.120+0000 7f11fdb87640 1 -- 172.21.15.67:0/271936315 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f11e00039b0 msgr2=0x7f11e0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:54.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.120+0000 7f11fdb87640 1 --2- 172.21.15.67:0/271936315 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f11e00039b0 0x7f11e0003dd0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f11ec001750 tx=0x7f11e0005830 comp rx=0 tx=0).stop 2024-03-24T00:48:54.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.120+0000 7f11fdb87640 1 -- 172.21.15.67:0/271936315 shutdown_connections 2024-03-24T00:48:54.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.120+0000 7f11fdb87640 1 --2- 172.21.15.67:0/271936315 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f11e00039b0 0x7f11e0003dd0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.120+0000 7f11fdb87640 1 -- 172.21.15.67:0/271936315 >> 172.21.15.67:0/271936315 conn(0x7f11f80cf9e0 msgr2=0x7f11f80cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:54.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.120+0000 7f11fdb87640 1 -- 172.21.15.67:0/271936315 shutdown_connections 2024-03-24T00:48:54.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.120+0000 7f11fdb87640 1 -- 172.21.15.67:0/271936315 wait complete. 2024-03-24T00:48:54.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.122+0000 7f11fdb87640 1 Processor -- start 2024-03-24T00:48:54.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.122+0000 7f11fdb87640 1 -- start start 2024-03-24T00:48:54.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.122+0000 7f11fdb87640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f11e00039b0 0x7f11f80c4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:54.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.122+0000 7f11fdb87640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f11f81f15f0 0x7f11f815d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:54.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.122+0000 7f11fdb87640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11f80ad320 0x7f11f806db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:54.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.122+0000 7f11fdb87640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11f80b39f0 con 0x7f11f80ad320 2024-03-24T00:48:54.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.122+0000 7f11fdb87640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11f80b3870 con 0x7f11e00039b0 2024-03-24T00:48:54.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.122+0000 7f11fdb87640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11f80b3b70 con 0x7f11f81f15f0 2024-03-24T00:48:54.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.123+0000 7f11f7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11f80ad320 0x7f11f806db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:54.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.123+0000 7f11f7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11f80ad320 0x7f11f806db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57346/0 (socket says 172.21.15.67:57346) 2024-03-24T00:48:54.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.123+0000 7f11f7fff640 1 -- 172.21.15.67:0/4049731384 learned_addr learned my addr 172.21.15.67:0/4049731384 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:54.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.123+0000 7f11f77fe640 1 --2- 172.21.15.67:0/4049731384 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f11e00039b0 0x7f11f80c4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:54.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.123+0000 7f11f6ffd640 1 --2- 172.21.15.67:0/4049731384 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f11f81f15f0 0x7f11f815d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:54.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.123+0000 7f11f7fff640 1 -- 172.21.15.67:0/4049731384 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f11f81f15f0 msgr2=0x7f11f815d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:54.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.123+0000 7f11f7fff640 1 --2- 172.21.15.67:0/4049731384 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f11f81f15f0 0x7f11f815d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.123+0000 7f11f7fff640 1 -- 172.21.15.67:0/4049731384 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f11e00039b0 msgr2=0x7f11f80c4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:54.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.123+0000 7f11f7fff640 1 --2- 172.21.15.67:0/4049731384 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f11e00039b0 0x7f11f80c4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.123+0000 7f11f7fff640 1 -- 172.21.15.67:0/4049731384 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11f81f1d60 con 0x7f11f80ad320 2024-03-24T00:48:54.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.123+0000 7f11f6ffd640 1 --2- 172.21.15.67:0/4049731384 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f11f81f15f0 0x7f11f815d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:48:54.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.123+0000 7f11f77fe640 1 --2- 172.21.15.67:0/4049731384 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f11e00039b0 0x7f11f80c4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:48:54.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.123+0000 7f11f7fff640 1 --2- 172.21.15.67:0/4049731384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11f80ad320 0x7f11f806db50 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f11f0001f00 tx=0x7f11f0007b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:54.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.123+0000 7f11dffff640 1 -- 172.21.15.67:0/4049731384 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f0017020 con 0x7f11f80ad320 2024-03-24T00:48:54.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.123+0000 7f11dffff640 1 -- 172.21.15.67:0/4049731384 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11f000ab60 con 0x7f11f80ad320 2024-03-24T00:48:54.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.123+0000 7f11fdb87640 1 -- 172.21.15.67:0/4049731384 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11f81f2040 con 0x7f11f80ad320 2024-03-24T00:48:54.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.123+0000 7f11dffff640 1 -- 172.21.15.67:0/4049731384 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f00133f0 con 0x7f11f80ad320 2024-03-24T00:48:54.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.123+0000 7f11fdb87640 1 -- 172.21.15.67:0/4049731384 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11f80ab890 con 0x7f11f80ad320 2024-03-24T00:48:54.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.124+0000 7f11fdb87640 1 -- 172.21.15.67:0/4049731384 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11f804f400 con 0x7f11f80ad320 2024-03-24T00:48:54.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.125+0000 7f11dffff640 1 -- 172.21.15.67:0/4049731384 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f11f001a020 con 0x7f11f80ad320 2024-03-24T00:48:54.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.125+0000 7f11dffff640 1 --2- 172.21.15.67:0/4049731384 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f11e406bbd0 0x7f11e406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:54.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.125+0000 7f11dffff640 1 -- 172.21.15.67:0/4049731384 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f11f008b3a0 con 0x7f11f80ad320 2024-03-24T00:48:54.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.125+0000 7f11f77fe640 1 --2- 172.21.15.67:0/4049731384 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f11e406bbd0 0x7f11e406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:54.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.126+0000 7f11f77fe640 1 --2- 172.21.15.67:0/4049731384 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f11e406bbd0 0x7f11e406e090 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f11ec033000 tx=0x7f11ec009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:54.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.130+0000 7f11dffff640 1 -- 172.21.15.67:0/4049731384 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f11f0021020 con 0x7f11f80ad320 2024-03-24T00:48:54.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.350+0000 7f11fdb87640 1 -- 172.21.15.67:0/4049731384 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f11f81f2560 con 0x7f11f80ad320 2024-03-24T00:48:54.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.976+0000 7f11dffff640 1 -- 172.21.15.67:0/4049731384 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 17 and data pool 18 v84) v1 ==== 174+0+0 (secure 0 0 0) 0x7f11f0059710 con 0x7f11f80ad320 2024-03-24T00:48:54.976 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 17 and data pool 18 2024-03-24T00:48:54.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.982+0000 7f11fdb87640 1 -- 172.21.15.67:0/4049731384 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f11e406bbd0 msgr2=0x7f11e406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:54.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.982+0000 7f11fdb87640 1 --2- 172.21.15.67:0/4049731384 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f11e406bbd0 0x7f11e406e090 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f11ec033000 tx=0x7f11ec009030 comp rx=0 tx=0).stop 2024-03-24T00:48:54.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.983+0000 7f11fdb87640 1 -- 172.21.15.67:0/4049731384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11f80ad320 msgr2=0x7f11f806db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:54.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.983+0000 7f11fdb87640 1 --2- 172.21.15.67:0/4049731384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11f80ad320 0x7f11f806db50 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f11f0001f00 tx=0x7f11f0007b20 comp rx=0 tx=0).stop 2024-03-24T00:48:54.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.986+0000 7f11fdb87640 1 -- 172.21.15.67:0/4049731384 shutdown_connections 2024-03-24T00:48:54.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.986+0000 7f11fdb87640 1 --2- 172.21.15.67:0/4049731384 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f11e406bbd0 0x7f11e406e090 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.986+0000 7f11fdb87640 1 --2- 172.21.15.67:0/4049731384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11f80ad320 0x7f11f806db50 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.986+0000 7f11fdb87640 1 --2- 172.21.15.67:0/4049731384 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f11f81f15f0 0x7f11f815d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.986+0000 7f11fdb87640 1 --2- 172.21.15.67:0/4049731384 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f11e00039b0 0x7f11f80c4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:54.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.986+0000 7f11fdb87640 1 -- 172.21.15.67:0/4049731384 >> 172.21.15.67:0/4049731384 conn(0x7f11f80cf9e0 msgr2=0x7f11f80721f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:54.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.989+0000 7f11fdb87640 1 -- 172.21.15.67:0/4049731384 shutdown_connections 2024-03-24T00:48:54.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:54.992+0000 7f11fdb87640 1 -- 172.21.15.67:0/4049731384 wait complete. 2024-03-24T00:48:55.017 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:48:55.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.122+0000 7f4dbdf7e640 1 Processor -- start 2024-03-24T00:48:55.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.122+0000 7f4dbdf7e640 1 -- start start 2024-03-24T00:48:55.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.123+0000 7f4dbdf7e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4db815def0 0x7f4db806c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:55.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.123+0000 7f4dbdf7e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4db80ce460 0x7f4db806cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:55.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.123+0000 7f4dbdf7e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4db80cecb0 con 0x7f4db815e2d0 2024-03-24T00:48:55.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.123+0000 7f4dbdf7e640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4db80cef90 con 0x7f4db80ce460 2024-03-24T00:48:55.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.123+0000 7f4dbdf7e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4db80cf270 con 0x7f4db815def0 2024-03-24T00:48:55.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.123+0000 7f4db7fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4db80ce460 0x7f4db806cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:55.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.123+0000 7f4db6ffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4db815e2d0 0x7f4db80cff50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35890/0 (socket says 172.21.15.67:35890) 2024-03-24T00:48:55.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.123+0000 7f4db6ffd640 1 -- 172.21.15.67:0/2337832989 learned_addr learned my addr 172.21.15.67:0/2337832989 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:48:55.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.124+0000 7f4db77fe640 1 --2- 172.21.15.67:0/2337832989 >> v2:172.21.15.72:3300/0 conn(0x7f4db815def0 0x7f4db806c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:55.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.124+0000 7f4db7fff640 1 -- 172.21.15.67:0/2337832989 >> v2:172.21.15.72:3300/0 conn(0x7f4db815def0 msgr2=0x7f4db806c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:55.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.124+0000 7f4db7fff640 1 --2- 172.21.15.67:0/2337832989 >> v2:172.21.15.72:3300/0 conn(0x7f4db815def0 0x7f4db806c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:55.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.124+0000 7f4db7fff640 1 -- 172.21.15.67:0/2337832989 >> v1:172.21.15.67:6789/0 conn(0x7f4db815e2d0 legacy=0x7f4db80cff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:55.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.124+0000 7f4db7fff640 1 -- 172.21.15.67:0/2337832989 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db80b1c10 con 0x7f4db80ce460 2024-03-24T00:48:55.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.124+0000 7f4db67fc640 1 -- 172.21.15.67:0/2337832989 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2209255166 0 0) 0x7f4db80cecb0 con 0x7f4db815e2d0 2024-03-24T00:48:55.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.124+0000 7f4db77fe640 1 --2- 172.21.15.67:0/2337832989 >> v2:172.21.15.72:3300/0 conn(0x7f4db815def0 0x7f4db806c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:48:55.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.124+0000 7f4db7fff640 1 --2- 172.21.15.67:0/2337832989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4db80ce460 0x7f4db806cb60 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f4db80678b0 tx=0x7f4db002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ea4199cf48c781e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:55.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.125+0000 7f4db67fc640 1 -- 172.21.15.67:0/2337832989 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db0039070 con 0x7f4db80ce460 2024-03-24T00:48:55.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.125+0000 7f4db67fc640 1 -- 172.21.15.67:0/2337832989 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4db002cbf0 con 0x7f4db80ce460 2024-03-24T00:48:55.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.125+0000 7f4db67fc640 1 -- 172.21.15.67:0/2337832989 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db0034440 con 0x7f4db80ce460 2024-03-24T00:48:55.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.125+0000 7f4dbdf7e640 1 -- 172.21.15.67:0/2337832989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4db80ce460 msgr2=0x7f4db806cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:55.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.125+0000 7f4dbdf7e640 1 --2- 172.21.15.67:0/2337832989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4db80ce460 0x7f4db806cb60 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f4db80678b0 tx=0x7f4db002be40 comp rx=0 tx=0).stop 2024-03-24T00:48:55.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.125+0000 7f4dbdf7e640 1 -- 172.21.15.67:0/2337832989 shutdown_connections 2024-03-24T00:48:55.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.125+0000 7f4dbdf7e640 1 --2- 172.21.15.67:0/2337832989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4db80ce460 0x7f4db806cb60 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:55.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.125+0000 7f4dbdf7e640 1 --2- 172.21.15.67:0/2337832989 >> v2:172.21.15.72:3300/0 conn(0x7f4db815def0 0x7f4db806c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:55.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.125+0000 7f4dbdf7e640 1 -- 172.21.15.67:0/2337832989 >> 172.21.15.67:0/2337832989 conn(0x7f4db808b1b0 msgr2=0x7f4db808b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:55.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.125+0000 7f4dbdf7e640 1 -- 172.21.15.67:0/2337832989 shutdown_connections 2024-03-24T00:48:55.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.125+0000 7f4dbdf7e640 1 -- 172.21.15.67:0/2337832989 wait complete. 2024-03-24T00:48:55.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.126+0000 7f4dbdf7e640 1 Processor -- start 2024-03-24T00:48:55.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.127+0000 7f4dbdf7e640 1 -- start start 2024-03-24T00:48:55.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.127+0000 7f4dbdf7e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4db80ce460 0x7f4db81541a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:55.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.127+0000 7f4dbdf7e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4db815def0 0x7f4db81607a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:55.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.127+0000 7f4dbdf7e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4db8160ce0 0x7f4db815c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:55.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.127+0000 7f4dbdf7e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db806d220 con 0x7f4db8160ce0 2024-03-24T00:48:55.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.127+0000 7f4dbdf7e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db806d0a0 con 0x7f4db80ce460 2024-03-24T00:48:55.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.127+0000 7f4dbdf7e640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4db806d3a0 con 0x7f4db815def0 2024-03-24T00:48:55.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.127+0000 7f4db7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4db8160ce0 0x7f4db815c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:55.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.127+0000 7f4db7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4db8160ce0 0x7f4db815c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57368/0 (socket says 172.21.15.67:57368) 2024-03-24T00:48:55.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.127+0000 7f4db7fff640 1 -- 172.21.15.67:0/3344582821 learned_addr learned my addr 172.21.15.67:0/3344582821 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:55.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.128+0000 7f4db77fe640 1 --2- 172.21.15.67:0/3344582821 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4db80ce460 0x7f4db81541a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:55.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.128+0000 7f4db6ffd640 1 --2- 172.21.15.67:0/3344582821 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4db815def0 0x7f4db81607a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:55.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.128+0000 7f4db7fff640 1 -- 172.21.15.67:0/3344582821 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4db815def0 msgr2=0x7f4db81607a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:55.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.128+0000 7f4db7fff640 1 --2- 172.21.15.67:0/3344582821 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4db815def0 0x7f4db81607a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:55.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.128+0000 7f4db7fff640 1 -- 172.21.15.67:0/3344582821 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4db80ce460 msgr2=0x7f4db81541a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:55.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.128+0000 7f4db7fff640 1 --2- 172.21.15.67:0/3344582821 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4db80ce460 0x7f4db81541a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:55.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.128+0000 7f4db7fff640 1 -- 172.21.15.67:0/3344582821 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db8161240 con 0x7f4db8160ce0 2024-03-24T00:48:55.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.128+0000 7f4db6ffd640 1 --2- 172.21.15.67:0/3344582821 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4db815def0 0x7f4db81607a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:48:55.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.128+0000 7f4db7fff640 1 --2- 172.21.15.67:0/3344582821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4db8160ce0 0x7f4db815c530 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f4db0009e80 tx=0x7f4db0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:55.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.128+0000 7f4d9ffff640 1 -- 172.21.15.67:0/3344582821 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db0039050 con 0x7f4db8160ce0 2024-03-24T00:48:55.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.128+0000 7f4d9ffff640 1 -- 172.21.15.67:0/3344582821 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4db003a040 con 0x7f4db8160ce0 2024-03-24T00:48:55.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.128+0000 7f4dbdf7e640 1 -- 172.21.15.67:0/3344582821 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4db80c8e30 con 0x7f4db8160ce0 2024-03-24T00:48:55.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.128+0000 7f4d9ffff640 1 -- 172.21.15.67:0/3344582821 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db003e610 con 0x7f4db8160ce0 2024-03-24T00:48:55.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.128+0000 7f4dbdf7e640 1 -- 172.21.15.67:0/3344582821 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4db8160040 con 0x7f4db8160ce0 2024-03-24T00:48:55.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.129+0000 7f4dbdf7e640 1 -- 172.21.15.67:0/3344582821 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4db804f400 con 0x7f4db8160ce0 2024-03-24T00:48:55.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.130+0000 7f4d9ffff640 1 -- 172.21.15.67:0/3344582821 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4db0047020 con 0x7f4db8160ce0 2024-03-24T00:48:55.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.130+0000 7f4d9ffff640 1 --2- 172.21.15.67:0/3344582821 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4da406bbd0 0x7f4da406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:55.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.130+0000 7f4d9ffff640 1 -- 172.21.15.67:0/3344582821 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4db00acec0 con 0x7f4db8160ce0 2024-03-24T00:48:55.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.130+0000 7f4db77fe640 1 --2- 172.21.15.67:0/3344582821 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4da406bbd0 0x7f4da406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:55.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.131+0000 7f4db77fe640 1 --2- 172.21.15.67:0/3344582821 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4da406bbd0 0x7f4da406e090 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f4da8006e80 tx=0x7f4da8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:55.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.134+0000 7f4d9ffff640 1 -- 172.21.15.67:0/3344582821 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4db004e020 con 0x7f4db8160ce0 2024-03-24T00:48:55.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.318+0000 7f4dbdf7e640 1 -- 172.21.15.67:0/3344582821 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4db80b1640 con 0x7f4db8160ce0 2024-03-24T00:48:55.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.319+0000 7f4d9ffff640 1 -- 172.21.15.67:0/3344582821 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v119) v1 ==== 74+0+16905 (secure 0 0 0) 0x7f4db007b820 con 0x7f4db8160ce0 2024-03-24T00:48:55.319 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:48:55.319 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":119,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:48:54.351267+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:48:52.622356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-03-24T00:48:53.311452+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:48:55.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.321+0000 7f4dbdf7e640 1 -- 172.21.15.67:0/3344582821 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4da406bbd0 msgr2=0x7f4da406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:55.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.321+0000 7f4dbdf7e640 1 --2- 172.21.15.67:0/3344582821 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4da406bbd0 0x7f4da406e090 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f4da8006e80 tx=0x7f4da8008040 comp rx=0 tx=0).stop 2024-03-24T00:48:55.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.321+0000 7f4dbdf7e640 1 -- 172.21.15.67:0/3344582821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4db8160ce0 msgr2=0x7f4db815c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:55.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.321+0000 7f4dbdf7e640 1 --2- 172.21.15.67:0/3344582821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4db8160ce0 0x7f4db815c530 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f4db0009e80 tx=0x7f4db0004a40 comp rx=0 tx=0).stop 2024-03-24T00:48:55.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.321+0000 7f4dbdf7e640 1 -- 172.21.15.67:0/3344582821 shutdown_connections 2024-03-24T00:48:55.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.321+0000 7f4dbdf7e640 1 --2- 172.21.15.67:0/3344582821 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4da406bbd0 0x7f4da406e090 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:55.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.321+0000 7f4dbdf7e640 1 --2- 172.21.15.67:0/3344582821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4db8160ce0 0x7f4db815c530 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:55.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.321+0000 7f4dbdf7e640 1 --2- 172.21.15.67:0/3344582821 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4db815def0 0x7f4db81607a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:55.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.321+0000 7f4dbdf7e640 1 --2- 172.21.15.67:0/3344582821 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4db80ce460 0x7f4db81541a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:55.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.321+0000 7f4dbdf7e640 1 -- 172.21.15.67:0/3344582821 >> 172.21.15.67:0/3344582821 conn(0x7f4db808b1b0 msgr2=0x7f4db8070480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:55.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.321+0000 7f4dbdf7e640 1 -- 172.21.15.67:0/3344582821 shutdown_connections 2024-03-24T00:48:55.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.321+0000 7f4dbdf7e640 1 -- 172.21.15.67:0/3344582821 wait complete. 2024-03-24T00:48:55.332 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:48:55.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.433+0000 7ff1c0e73640 1 Processor -- start 2024-03-24T00:48:55.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.433+0000 7ff1c0e73640 1 -- start start 2024-03-24T00:48:55.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.434+0000 7ff1c0e73640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff1bc15dec0 0x7ff1bc0ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:55.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.434+0000 7ff1c0e73640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1bc0cf640 con 0x7ff1bc0d0ef0 2024-03-24T00:48:55.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.434+0000 7ff1c0e73640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1bc0cf080 con 0x7ff1bc15e2a0 2024-03-24T00:48:55.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.434+0000 7ff1c0e73640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc0cee30 con 0x7ff1bc15dec0 2024-03-24T00:48:55.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.434+0000 7ff1bad9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff1bc0d0ef0 0x7ff1bc0d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35896/0 (socket says 172.21.15.67:35896) 2024-03-24T00:48:55.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.434+0000 7ff1bad9d640 1 -- 172.21.15.67:0/1496935454 learned_addr learned my addr 172.21.15.67:0/1496935454 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:48:55.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.435+0000 7ff1b959a640 1 -- 172.21.15.67:0/1496935454 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1470270552 0 0) 0x7ff1ac001490 con 0x7ff1bc0d0ef0 2024-03-24T00:48:55.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.435+0000 7ff1ba59c640 1 --2- 172.21.15.67:0/1496935454 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff1bc15dec0 0x7ff1bc0ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:55.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.435+0000 7ff1b959a640 1 -- 172.21.15.67:0/1496935454 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1a8003520 con 0x7ff1bc0d0ef0 2024-03-24T00:48:55.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.435+0000 7ff1b959a640 1 -- 172.21.15.67:0/1496935454 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 530359893 0 0) 0x7ff1a8003520 con 0x7ff1bc0d0ef0 2024-03-24T00:48:55.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.435+0000 7ff1b959a640 1 -- 172.21.15.67:0/1496935454 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff1bc15dec0 msgr2=0x7ff1bc0ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:55.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.435+0000 7ff1b959a640 1 --2- 172.21.15.67:0/1496935454 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff1bc15dec0 0x7ff1bc0ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:55.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.435+0000 7ff1b959a640 1 -- 172.21.15.67:0/1496935454 >> v1:172.21.15.72:6789/0 conn(0x7ff1bc15e2a0 legacy=0x7ff1bc0d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:55.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.435+0000 7ff1b959a640 1 -- 172.21.15.67:0/1496935454 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1bc0cf360 con 0x7ff1bc0d0ef0 2024-03-24T00:48:55.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.435+0000 7ff1b959a640 1 -- 172.21.15.67:0/1496935454 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7ff1ac001bc0 con 0x7ff1bc0d0ef0 2024-03-24T00:48:55.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.435+0000 7ff1b959a640 1 -- 172.21.15.67:0/1496935454 >> v1:172.21.15.67:6789/0 conn(0x7ff1bc0d0ef0 legacy=0x7ff1bc0d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:55.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.435+0000 7ff1b959a640 1 --2- 172.21.15.67:0/1496935454 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1a80039e0 0x7ff1a8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:55.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.435+0000 7ff1b959a640 1 --2- 172.21.15.67:0/1496935454 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1a8004800 0x7ff1a8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:55.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.435+0000 7ff1b959a640 1 --2- 172.21.15.67:0/1496935454 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff1bc15dec0 0x7ff1a8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:55.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.435+0000 7ff1b959a640 1 -- 172.21.15.67:0/1496935454 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc0cee30 con 0x7ff1a80039e0 2024-03-24T00:48:55.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.435+0000 7ff1b959a640 1 -- 172.21.15.67:0/1496935454 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a800baf0 con 0x7ff1a8004800 2024-03-24T00:48:55.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.435+0000 7ff1b959a640 1 -- 172.21.15.67:0/1496935454 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a800bcf0 con 0x7ff1bc15dec0 2024-03-24T00:48:55.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.435+0000 7ff1ba59c640 1 --2- 172.21.15.67:0/1496935454 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1a80039e0 0x7ff1a8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:55.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.435+0000 7ff1bad9d640 1 --2- 172.21.15.67:0/1496935454 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff1bc15dec0 0x7ff1a8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:55.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.435+0000 7ff1b9d9b640 1 --2- 172.21.15.67:0/1496935454 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1a8004800 0x7ff1a8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:55.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.435+0000 7ff1ba59c640 1 -- 172.21.15.67:0/1496935454 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff1bc15dec0 msgr2=0x7ff1a8005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:55.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.435+0000 7ff1ba59c640 1 --2- 172.21.15.67:0/1496935454 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff1bc15dec0 0x7ff1a8005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:55.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.435+0000 7ff1ba59c640 1 -- 172.21.15.67:0/1496935454 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1a8004800 msgr2=0x7ff1a8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:55.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.435+0000 7ff1ba59c640 1 --2- 172.21.15.67:0/1496935454 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1a8004800 0x7ff1a8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:55.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.435+0000 7ff1ba59c640 1 -- 172.21.15.67:0/1496935454 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1a8003520 con 0x7ff1a80039e0 2024-03-24T00:48:55.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.435+0000 7ff1b9d9b640 1 --2- 172.21.15.67:0/1496935454 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1a8004800 0x7ff1a8004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:48:55.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.435+0000 7ff1bad9d640 1 --2- 172.21.15.67:0/1496935454 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff1bc15dec0 0x7ff1a8005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:48:55.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.436+0000 7ff1ba59c640 1 --2- 172.21.15.67:0/1496935454 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1a80039e0 0x7ff1a8003e00 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4009ac0 tx=0x7ff1a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76e80d631def49d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:55.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.436+0000 7ff1b959a640 1 -- 172.21.15.67:0/1496935454 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1b4039070 con 0x7ff1a80039e0 2024-03-24T00:48:55.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.436+0000 7ff1b959a640 1 -- 172.21.15.67:0/1496935454 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1b402c6b0 con 0x7ff1a80039e0 2024-03-24T00:48:55.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.436+0000 7ff1b959a640 1 -- 172.21.15.67:0/1496935454 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1b4009690 con 0x7ff1a80039e0 2024-03-24T00:48:55.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.437+0000 7ff1c0e73640 1 -- 172.21.15.67:0/1496935454 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1a80039e0 msgr2=0x7ff1a8003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:55.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.437+0000 7ff1c0e73640 1 --2- 172.21.15.67:0/1496935454 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1a80039e0 0x7ff1a8003e00 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4009ac0 tx=0x7ff1a8005830 comp rx=0 tx=0).stop 2024-03-24T00:48:55.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.437+0000 7ff1bad9d640 1 -- 172.21.15.67:0/1496935454 reap_dead start 2024-03-24T00:48:55.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.437+0000 7ff1c0e73640 1 -- 172.21.15.67:0/1496935454 shutdown_connections 2024-03-24T00:48:55.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.437+0000 7ff1c0e73640 1 -- 172.21.15.67:0/1496935454 >> 172.21.15.67:0/1496935454 conn(0x7ff1bc06c810 msgr2=0x7ff1bc06cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:55.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.437+0000 7ff1c0e73640 1 -- 172.21.15.67:0/1496935454 shutdown_connections 2024-03-24T00:48:55.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.437+0000 7ff1c0e73640 1 -- 172.21.15.67:0/1496935454 wait complete. 2024-03-24T00:48:55.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.438+0000 7ff1c0e73640 1 Processor -- start 2024-03-24T00:48:55.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.438+0000 7ff1c0e73640 1 -- start start 2024-03-24T00:48:55.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.439+0000 7ff1c0e73640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff1a80039e0 0x7ff1bc152330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:55.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.439+0000 7ff1c0e73640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1bc0b6550 0x7ff1bc1f4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:55.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.439+0000 7ff1c0e73640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1bc1588c0 0x7ff1bc0b70e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:55.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.439+0000 7ff1c0e73640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc0cea40 con 0x7ff1bc0b6550 2024-03-24T00:48:55.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.439+0000 7ff1c0e73640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc0ce8c0 con 0x7ff1bc1588c0 2024-03-24T00:48:55.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.439+0000 7ff1c0e73640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc0d2120 con 0x7ff1a80039e0 2024-03-24T00:48:55.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.439+0000 7ff1b9d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1bc0b6550 0x7ff1bc1f4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:55.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.439+0000 7ff1b9d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1bc0b6550 0x7ff1bc1f4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57388/0 (socket says 172.21.15.67:57388) 2024-03-24T00:48:55.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.439+0000 7ff1b9d9b640 1 -- 172.21.15.67:0/3106042683 learned_addr learned my addr 172.21.15.67:0/3106042683 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:55.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.439+0000 7ff1ba59c640 1 --2- 172.21.15.67:0/3106042683 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff1a80039e0 0x7ff1bc152330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:55.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.439+0000 7ff1bad9d640 1 --2- 172.21.15.67:0/3106042683 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1bc1588c0 0x7ff1bc0b70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:55.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.439+0000 7ff1b9d9b640 1 -- 172.21.15.67:0/3106042683 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff1a80039e0 msgr2=0x7ff1bc152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:55.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.439+0000 7ff1b9d9b640 1 --2- 172.21.15.67:0/3106042683 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff1a80039e0 0x7ff1bc152330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:55.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.439+0000 7ff1b9d9b640 1 -- 172.21.15.67:0/3106042683 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1bc1588c0 msgr2=0x7ff1bc0b70e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:55.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.439+0000 7ff1b9d9b640 1 --2- 172.21.15.67:0/3106042683 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1bc1588c0 0x7ff1bc0b70e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:55.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.439+0000 7ff1b9d9b640 1 -- 172.21.15.67:0/3106042683 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1bc1f50c0 con 0x7ff1bc0b6550 2024-03-24T00:48:55.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.440+0000 7ff1b9d9b640 1 --2- 172.21.15.67:0/3106042683 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1bc0b6550 0x7ff1bc1f4b80 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7ff1a400e570 tx=0x7ff1a400ea30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:55.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.440+0000 7ff1b2ffd640 1 -- 172.21.15.67:0/3106042683 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a40052e0 con 0x7ff1bc0b6550 2024-03-24T00:48:55.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.440+0000 7ff1b2ffd640 1 -- 172.21.15.67:0/3106042683 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1a4018070 con 0x7ff1bc0b6550 2024-03-24T00:48:55.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.440+0000 7ff1c0e73640 1 -- 172.21.15.67:0/3106042683 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1bc0b6af0 con 0x7ff1bc0b6550 2024-03-24T00:48:55.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.440+0000 7ff1b2ffd640 1 -- 172.21.15.67:0/3106042683 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a4003810 con 0x7ff1bc0b6550 2024-03-24T00:48:55.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.440+0000 7ff1c0e73640 1 -- 172.21.15.67:0/3106042683 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1bc157de0 con 0x7ff1bc0b6550 2024-03-24T00:48:55.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.441+0000 7ff1c0e73640 1 -- 172.21.15.67:0/3106042683 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1bc04f400 con 0x7ff1bc0b6550 2024-03-24T00:48:55.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.441+0000 7ff1b2ffd640 1 -- 172.21.15.67:0/3106042683 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff1a40039b0 con 0x7ff1bc0b6550 2024-03-24T00:48:55.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.441+0000 7ff1b2ffd640 1 --2- 172.21.15.67:0/3106042683 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff19c06bbd0 0x7ff19c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:55.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.442+0000 7ff1b2ffd640 1 -- 172.21.15.67:0/3106042683 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff1a408a7b0 con 0x7ff1bc0b6550 2024-03-24T00:48:55.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.442+0000 7ff1ba59c640 1 --2- 172.21.15.67:0/3106042683 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff19c06bbd0 0x7ff19c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:55.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.442+0000 7ff1ba59c640 1 --2- 172.21.15.67:0/3106042683 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff19c06bbd0 0x7ff19c06e090 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4036000 tx=0x7ff1b4030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:55.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.446+0000 7ff1b2ffd640 1 -- 172.21.15.67:0/3106042683 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff1a4058b50 con 0x7ff1bc0b6550 2024-03-24T00:48:55.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.629+0000 7ff1c0e73640 1 -- 172.21.15.67:0/3106042683 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff1bc158dc0 con 0x7ff1bc0b6550 2024-03-24T00:48:55.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.630+0000 7ff1b2ffd640 1 -- 172.21.15.67:0/3106042683 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v119) v1 ==== 74+0+16905 (secure 0 0 0) 0x7ff1a405cbf0 con 0x7ff1bc0b6550 2024-03-24T00:48:55.630 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:48:55.630 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":119,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:48:54.351267+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:48:52.622356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-03-24T00:48:53.311452+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:48:55.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.632+0000 7ff1c0e73640 1 -- 172.21.15.67:0/3106042683 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff19c06bbd0 msgr2=0x7ff19c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:55.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.632+0000 7ff1c0e73640 1 --2- 172.21.15.67:0/3106042683 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff19c06bbd0 0x7ff19c06e090 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4036000 tx=0x7ff1b4030000 comp rx=0 tx=0).stop 2024-03-24T00:48:55.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.632+0000 7ff1c0e73640 1 -- 172.21.15.67:0/3106042683 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1bc0b6550 msgr2=0x7ff1bc1f4b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:55.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.632+0000 7ff1c0e73640 1 --2- 172.21.15.67:0/3106042683 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1bc0b6550 0x7ff1bc1f4b80 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7ff1a400e570 tx=0x7ff1a400ea30 comp rx=0 tx=0).stop 2024-03-24T00:48:55.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.632+0000 7ff1c0e73640 1 -- 172.21.15.67:0/3106042683 shutdown_connections 2024-03-24T00:48:55.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.632+0000 7ff1c0e73640 1 --2- 172.21.15.67:0/3106042683 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff19c06bbd0 0x7ff19c06e090 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:55.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.632+0000 7ff1c0e73640 1 --2- 172.21.15.67:0/3106042683 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1bc1588c0 0x7ff1bc0b70e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:55.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.632+0000 7ff1c0e73640 1 --2- 172.21.15.67:0/3106042683 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1bc0b6550 0x7ff1bc1f4b80 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:55.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.632+0000 7ff1c0e73640 1 --2- 172.21.15.67:0/3106042683 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff1a80039e0 0x7ff1bc152330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:55.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.632+0000 7ff1c0e73640 1 -- 172.21.15.67:0/3106042683 >> 172.21.15.67:0/3106042683 conn(0x7ff1bc06c810 msgr2=0x7ff1bc152a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:55.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.632+0000 7ff1c0e73640 1 -- 172.21.15.67:0/3106042683 shutdown_connections 2024-03-24T00:48:55.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.632+0000 7ff1c0e73640 1 -- 172.21.15.67:0/3106042683 wait complete. 2024-03-24T00:48:55.643 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-24T00:48:55.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.747+0000 7fe71d885640 1 Processor -- start 2024-03-24T00:48:55.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.747+0000 7fe71d885640 1 -- start start 2024-03-24T00:48:55.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.748+0000 7fe71d885640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe7180d3ae0 0x7fe7180b5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:55.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.748+0000 7fe71d885640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7180d2250 con 0x7fe71815e280 2024-03-24T00:48:55.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.748+0000 7fe71d885640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7180d2530 con 0x7fe71815dea0 2024-03-24T00:48:55.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.748+0000 7fe71d885640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7180d2810 con 0x7fe7180d3ae0 2024-03-24T00:48:55.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.748+0000 7fe7177fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe71815e280 0x7fe7180b70f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35898/0 (socket says 172.21.15.67:35898) 2024-03-24T00:48:55.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.748+0000 7fe7177fe640 1 -- 172.21.15.67:0/1387553451 learned_addr learned my addr 172.21.15.67:0/1387553451 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:48:55.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.748+0000 7fe7167fc640 1 --2- 172.21.15.67:0/1387553451 >> v2:172.21.15.72:3300/0 conn(0x7fe7180d3ae0 0x7fe7180b5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:55.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.748+0000 7fe715ffb640 1 -- 172.21.15.67:0/1387553451 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 768251684 0 0) 0x7fe7180d2250 con 0x7fe71815e280 2024-03-24T00:48:55.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.748+0000 7fe715ffb640 1 -- 172.21.15.67:0/1387553451 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe704003520 con 0x7fe71815e280 2024-03-24T00:48:55.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.749+0000 7fe7167fc640 1 -- 172.21.15.67:0/1387553451 >> v1:172.21.15.72:6789/0 conn(0x7fe71815dea0 legacy=0x7fe7180d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:55.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.749+0000 7fe7167fc640 1 -- 172.21.15.67:0/1387553451 >> v1:172.21.15.67:6789/0 conn(0x7fe71815e280 legacy=0x7fe7180b70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:55.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.749+0000 7fe7167fc640 1 -- 172.21.15.67:0/1387553451 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7180d2af0 con 0x7fe7180d3ae0 2024-03-24T00:48:55.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.749+0000 7fe7167fc640 1 --2- 172.21.15.67:0/1387553451 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe7180d3ae0 0x7fe7180b5a60 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fe7000097d0 tx=0x7fe70002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1f5309dc8c35958 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:55.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.750+0000 7fe715ffb640 1 -- 172.21.15.67:0/1387553451 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe700035020 con 0x7fe7180d3ae0 2024-03-24T00:48:55.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.750+0000 7fe715ffb640 1 -- 172.21.15.67:0/1387553451 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe700031830 con 0x7fe7180d3ae0 2024-03-24T00:48:55.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.750+0000 7fe715ffb640 1 -- 172.21.15.67:0/1387553451 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe700031be0 con 0x7fe7180d3ae0 2024-03-24T00:48:55.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.750+0000 7fe71d885640 1 -- 172.21.15.67:0/1387553451 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe7180d3ae0 msgr2=0x7fe7180b5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:55.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.750+0000 7fe71d885640 1 --2- 172.21.15.67:0/1387553451 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe7180d3ae0 0x7fe7180b5a60 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fe7000097d0 tx=0x7fe70002b8d0 comp rx=0 tx=0).stop 2024-03-24T00:48:55.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.750+0000 7fe71d885640 1 -- 172.21.15.67:0/1387553451 shutdown_connections 2024-03-24T00:48:55.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.750+0000 7fe71d885640 1 --2- 172.21.15.67:0/1387553451 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe7180d3ae0 0x7fe7180b5a60 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:55.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.750+0000 7fe71d885640 1 -- 172.21.15.67:0/1387553451 >> 172.21.15.67:0/1387553451 conn(0x7fe718072da0 msgr2=0x7fe7180d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:55.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.750+0000 7fe71d885640 1 -- 172.21.15.67:0/1387553451 shutdown_connections 2024-03-24T00:48:55.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.750+0000 7fe71d885640 1 -- 172.21.15.67:0/1387553451 wait complete. 2024-03-24T00:48:55.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.751+0000 7fe71d885640 1 Processor -- start 2024-03-24T00:48:55.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.751+0000 7fe71d885640 1 -- start start 2024-03-24T00:48:55.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.752+0000 7fe71d885640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe7180d3ae0 0x7fe718160e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:55.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.752+0000 7fe71d885640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe71815dea0 0x7fe718158120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:55.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.752+0000 7fe71d885640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe71815e280 0x7fe718158660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:55.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.752+0000 7fe71d885640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7180b6370 con 0x7fe71815dea0 2024-03-24T00:48:55.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.752+0000 7fe71d885640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7180b61f0 con 0x7fe71815e280 2024-03-24T00:48:55.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.752+0000 7fe71d885640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7180b7800 con 0x7fe7180d3ae0 2024-03-24T00:48:55.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.752+0000 7fe7167fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe71815dea0 0x7fe718158120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:55.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.752+0000 7fe7167fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe71815dea0 0x7fe718158120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57394/0 (socket says 172.21.15.67:57394) 2024-03-24T00:48:55.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.752+0000 7fe7167fc640 1 -- 172.21.15.67:0/3923905184 learned_addr learned my addr 172.21.15.67:0/3923905184 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:55.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.752+0000 7fe716ffd640 1 --2- 172.21.15.67:0/3923905184 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe7180d3ae0 0x7fe718160e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:55.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.752+0000 7fe7177fe640 1 --2- 172.21.15.67:0/3923905184 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe71815e280 0x7fe718158660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:55.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.753+0000 7fe7167fc640 1 -- 172.21.15.67:0/3923905184 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe7180d3ae0 msgr2=0x7fe718160e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:55.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.753+0000 7fe7167fc640 1 --2- 172.21.15.67:0/3923905184 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe7180d3ae0 0x7fe718160e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:55.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.753+0000 7fe7167fc640 1 -- 172.21.15.67:0/3923905184 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe71815e280 msgr2=0x7fe718158660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:55.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.753+0000 7fe7167fc640 1 --2- 172.21.15.67:0/3923905184 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe71815e280 0x7fe718158660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:55.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.753+0000 7fe7167fc640 1 -- 172.21.15.67:0/3923905184 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe718158d00 con 0x7fe71815dea0 2024-03-24T00:48:55.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.753+0000 7fe7167fc640 1 --2- 172.21.15.67:0/3923905184 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe71815dea0 0x7fe718158120 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7fe7000097d0 tx=0x7fe7000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:55.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.753+0000 7fe70ffff640 1 -- 172.21.15.67:0/3923905184 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe700035070 con 0x7fe71815dea0 2024-03-24T00:48:55.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.753+0000 7fe70ffff640 1 -- 172.21.15.67:0/3923905184 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe700004230 con 0x7fe71815dea0 2024-03-24T00:48:55.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.753+0000 7fe71d885640 1 -- 172.21.15.67:0/3923905184 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe718158fe0 con 0x7fe71815dea0 2024-03-24T00:48:55.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.753+0000 7fe70ffff640 1 -- 172.21.15.67:0/3923905184 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe70003a630 con 0x7fe71815dea0 2024-03-24T00:48:55.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.753+0000 7fe71d885640 1 -- 172.21.15.67:0/3923905184 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7181595a0 con 0x7fe71815dea0 2024-03-24T00:48:55.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.754+0000 7fe71d885640 1 -- 172.21.15.67:0/3923905184 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe71804f400 con 0x7fe71815dea0 2024-03-24T00:48:55.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.754+0000 7fe70ffff640 1 -- 172.21.15.67:0/3923905184 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe700038020 con 0x7fe71815dea0 2024-03-24T00:48:55.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.755+0000 7fe70ffff640 1 --2- 172.21.15.67:0/3923905184 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe6f806bbd0 0x7fe6f806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:55.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.755+0000 7fe70ffff640 1 -- 172.21.15.67:0/3923905184 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe7000ac740 con 0x7fe71815dea0 2024-03-24T00:48:55.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.755+0000 7fe716ffd640 1 --2- 172.21.15.67:0/3923905184 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe6f806bbd0 0x7fe6f806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:55.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.755+0000 7fe716ffd640 1 --2- 172.21.15.67:0/3923905184 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe6f806bbd0 0x7fe6f806e090 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fe710001780 tx=0x7fe710009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:55.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.759+0000 7fe70ffff640 1 -- 172.21.15.67:0/3923905184 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe700040070 con 0x7fe71815dea0 2024-03-24T00:48:55.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.979+0000 7fe71d885640 1 -- 172.21.15.67:0/3923905184 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fe7180d1450 con 0x7fe71815dea0 2024-03-24T00:48:55.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.993+0000 7fe70ffff640 1 -- 172.21.15.67:0/3923905184 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v85) v1 ==== 118+0+0 (secure 0 0 0) 0x7fe70007aa50 con 0x7fe71815dea0 2024-03-24T00:48:55.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.996+0000 7fe71d885640 1 -- 172.21.15.67:0/3923905184 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe6f806bbd0 msgr2=0x7fe6f806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:55.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.996+0000 7fe71d885640 1 --2- 172.21.15.67:0/3923905184 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe6f806bbd0 0x7fe6f806e090 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fe710001780 tx=0x7fe710009040 comp rx=0 tx=0).stop 2024-03-24T00:48:55.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.996+0000 7fe71d885640 1 -- 172.21.15.67:0/3923905184 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe71815dea0 msgr2=0x7fe718158120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:55.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.996+0000 7fe71d885640 1 --2- 172.21.15.67:0/3923905184 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe71815dea0 0x7fe718158120 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7fe7000097d0 tx=0x7fe7000046c0 comp rx=0 tx=0).stop 2024-03-24T00:48:55.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.996+0000 7fe71d885640 1 -- 172.21.15.67:0/3923905184 shutdown_connections 2024-03-24T00:48:55.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.996+0000 7fe71d885640 1 --2- 172.21.15.67:0/3923905184 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe6f806bbd0 0x7fe6f806e090 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:55.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.996+0000 7fe71d885640 1 --2- 172.21.15.67:0/3923905184 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe71815e280 0x7fe718158660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:55.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.996+0000 7fe71d885640 1 --2- 172.21.15.67:0/3923905184 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe71815dea0 0x7fe718158120 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:55.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.996+0000 7fe71d885640 1 --2- 172.21.15.67:0/3923905184 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe7180d3ae0 0x7fe718160e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:55.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.996+0000 7fe71d885640 1 -- 172.21.15.67:0/3923905184 >> 172.21.15.67:0/3923905184 conn(0x7fe718072da0 msgr2=0x7fe7180cd0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:55.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.996+0000 7fe71d885640 1 -- 172.21.15.67:0/3923905184 shutdown_connections 2024-03-24T00:48:55.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:55.996+0000 7fe71d885640 1 -- 172.21.15.67:0/3923905184 wait complete. 2024-03-24T00:48:56.008 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-24T00:48:56.008 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-24T00:48:56.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.113+0000 7fad9a3b3640 1 Processor -- start 2024-03-24T00:48:56.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.113+0000 7fad9a3b3640 1 -- start start 2024-03-24T00:48:56.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.114+0000 7fad9a3b3640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fad9406eb30 0x7fad940ad140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:56.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.114+0000 7fad9a3b3640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad940b6150 con 0x7fad9415de30 2024-03-24T00:48:56.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.114+0000 7fad9a3b3640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad9406ba50 con 0x7fad940b51d0 2024-03-24T00:48:56.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.114+0000 7fad9a3b3640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad9406c380 con 0x7fad9406eb30 2024-03-24T00:48:56.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.114+0000 7fad93fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fad9415de30 0x7fad9406e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35908/0 (socket says 172.21.15.67:35908) 2024-03-24T00:48:56.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.114+0000 7fad93fff640 1 -- 172.21.15.67:0/540737001 learned_addr learned my addr 172.21.15.67:0/540737001 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:48:56.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.114+0000 7fad92ffd640 1 -- 172.21.15.67:0/540737001 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3865084903 0 0) 0x7fad940b6150 con 0x7fad9415de30 2024-03-24T00:48:56.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.114+0000 7fad937fe640 1 --2- 172.21.15.67:0/540737001 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fad9406eb30 0x7fad940ad140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:56.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.114+0000 7fad92ffd640 1 -- 172.21.15.67:0/540737001 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad80003520 con 0x7fad9415de30 2024-03-24T00:48:56.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.114+0000 7fad92ffd640 1 -- 172.21.15.67:0/540737001 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 859890133 0 0) 0x7fad80003520 con 0x7fad9415de30 2024-03-24T00:48:56.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.114+0000 7fad92ffd640 1 -- 172.21.15.67:0/540737001 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fad9406eb30 msgr2=0x7fad940ad140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:56.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.114+0000 7fad92ffd640 1 --2- 172.21.15.67:0/540737001 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fad9406eb30 0x7fad940ad140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.114+0000 7fad92ffd640 1 -- 172.21.15.67:0/540737001 >> v1:172.21.15.72:6789/0 conn(0x7fad940b51d0 legacy=0x7fad940ad680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:56.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.115+0000 7fad92ffd640 1 -- 172.21.15.67:0/540737001 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad9406bd30 con 0x7fad9415de30 2024-03-24T00:48:56.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.115+0000 7fad92ffd640 1 -- 172.21.15.67:0/540737001 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7fad7c001670 con 0x7fad9415de30 2024-03-24T00:48:56.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.115+0000 7fad92ffd640 1 -- 172.21.15.67:0/540737001 >> v1:172.21.15.67:6789/0 conn(0x7fad9415de30 legacy=0x7fad9406e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:56.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.115+0000 7fad92ffd640 1 --2- 172.21.15.67:0/540737001 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad800039e0 0x7fad80003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:56.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.115+0000 7fad92ffd640 1 --2- 172.21.15.67:0/540737001 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fad9406eb30 0x7fad80004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:56.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.115+0000 7fad92ffd640 1 --2- 172.21.15.67:0/540737001 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad80004d40 0x7fad800051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:56.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.115+0000 7fad92ffd640 1 -- 172.21.15.67:0/540737001 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad9406c380 con 0x7fad800039e0 2024-03-24T00:48:56.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.115+0000 7fad92ffd640 1 -- 172.21.15.67:0/540737001 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad8000ba70 con 0x7fad80004d40 2024-03-24T00:48:56.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.115+0000 7fad92ffd640 1 -- 172.21.15.67:0/540737001 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad8000bc70 con 0x7fad9406eb30 2024-03-24T00:48:56.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.115+0000 7fad93fff640 1 --2- 172.21.15.67:0/540737001 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad800039e0 0x7fad80003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:56.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.115+0000 7fad98950640 1 --2- 172.21.15.67:0/540737001 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad80004d40 0x7fad800051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:56.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.115+0000 7fad937fe640 1 --2- 172.21.15.67:0/540737001 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fad9406eb30 0x7fad80004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:56.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.115+0000 7fad93fff640 1 -- 172.21.15.67:0/540737001 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fad9406eb30 msgr2=0x7fad80004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:56.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.115+0000 7fad93fff640 1 --2- 172.21.15.67:0/540737001 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fad9406eb30 0x7fad80004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.115+0000 7fad93fff640 1 -- 172.21.15.67:0/540737001 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad80004d40 msgr2=0x7fad800051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:56.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.115+0000 7fad93fff640 1 --2- 172.21.15.67:0/540737001 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad80004d40 0x7fad800051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.115+0000 7fad93fff640 1 -- 172.21.15.67:0/540737001 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad80003520 con 0x7fad800039e0 2024-03-24T00:48:56.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.116+0000 7fad93fff640 1 --2- 172.21.15.67:0/540737001 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad800039e0 0x7fad80003e00 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7fad7c00c210 tx=0x7fad80005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f91c5cf805c99959 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:56.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.116+0000 7fad92ffd640 1 -- 172.21.15.67:0/540737001 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad7c007cf0 con 0x7fad800039e0 2024-03-24T00:48:56.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.116+0000 7fad92ffd640 1 -- 172.21.15.67:0/540737001 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad7c030070 con 0x7fad800039e0 2024-03-24T00:48:56.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.116+0000 7fad92ffd640 1 -- 172.21.15.67:0/540737001 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad7c005d30 con 0x7fad800039e0 2024-03-24T00:48:56.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.117+0000 7fad9a3b3640 1 -- 172.21.15.67:0/540737001 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad800039e0 msgr2=0x7fad80003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:56.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.117+0000 7fad9a3b3640 1 --2- 172.21.15.67:0/540737001 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad800039e0 0x7fad80003e00 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7fad7c00c210 tx=0x7fad80005830 comp rx=0 tx=0).stop 2024-03-24T00:48:56.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.117+0000 7fad98950640 1 -- 172.21.15.67:0/540737001 reap_dead start 2024-03-24T00:48:56.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.117+0000 7fad9a3b3640 1 -- 172.21.15.67:0/540737001 shutdown_connections 2024-03-24T00:48:56.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.117+0000 7fad9a3b3640 1 -- 172.21.15.67:0/540737001 >> 172.21.15.67:0/540737001 conn(0x7fad94099a30 msgr2=0x7fad940b5680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:56.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.117+0000 7fad9a3b3640 1 -- 172.21.15.67:0/540737001 shutdown_connections 2024-03-24T00:48:56.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.117+0000 7fad9a3b3640 1 -- 172.21.15.67:0/540737001 wait complete. 2024-03-24T00:48:56.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.118+0000 7fad9a3b3640 1 Processor -- start 2024-03-24T00:48:56.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.118+0000 7fad9a3b3640 1 -- start start 2024-03-24T00:48:56.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.119+0000 7fad9a3b3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad800039e0 0x7fad94149830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:56.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.119+0000 7fad9a3b3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad9414cfc0 0x7fad9414a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:56.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.119+0000 7fad9a3b3640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fad94146360 0x7fad940b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:56.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.119+0000 7fad9a3b3640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad9415e210 con 0x7fad9414cfc0 2024-03-24T00:48:56.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.119+0000 7fad9a3b3640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad9406c810 con 0x7fad800039e0 2024-03-24T00:48:56.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.119+0000 7fad9a3b3640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad94147b40 con 0x7fad94146360 2024-03-24T00:48:56.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.119+0000 7fad937fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad9414cfc0 0x7fad9414a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:56.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.119+0000 7fad937fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad9414cfc0 0x7fad9414a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57410/0 (socket says 172.21.15.67:57410) 2024-03-24T00:48:56.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.119+0000 7fad937fe640 1 -- 172.21.15.67:0/2759315698 learned_addr learned my addr 172.21.15.67:0/2759315698 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:56.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.119+0000 7fad98950640 1 --2- 172.21.15.67:0/2759315698 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fad94146360 0x7fad940b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:56.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.119+0000 7fad93fff640 1 --2- 172.21.15.67:0/2759315698 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad800039e0 0x7fad94149830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:56.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.119+0000 7fad937fe640 1 -- 172.21.15.67:0/2759315698 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fad94146360 msgr2=0x7fad940b1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:56.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.119+0000 7fad937fe640 1 --2- 172.21.15.67:0/2759315698 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fad94146360 0x7fad940b1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.119+0000 7fad937fe640 1 -- 172.21.15.67:0/2759315698 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad800039e0 msgr2=0x7fad94149830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:56.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.119+0000 7fad937fe640 1 --2- 172.21.15.67:0/2759315698 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad800039e0 0x7fad94149830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:56.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.119+0000 7fad937fe640 1 -- 172.21.15.67:0/2759315698 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad94146f30 con 0x7fad9414cfc0 2024-03-24T00:48:56.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.119+0000 7fad93fff640 1 --2- 172.21.15.67:0/2759315698 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad800039e0 0x7fad94149830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:48:56.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.119+0000 7fad937fe640 1 --2- 172.21.15.67:0/2759315698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad9414cfc0 0x7fad9414a280 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7fad840119a0 tx=0x7fad84011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:56.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.120+0000 7fad90ff9640 1 -- 172.21.15.67:0/2759315698 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad8400faf0 con 0x7fad9414cfc0 2024-03-24T00:48:56.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.120+0000 7fad90ff9640 1 -- 172.21.15.67:0/2759315698 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad8400fc90 con 0x7fad9414cfc0 2024-03-24T00:48:56.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.120+0000 7fad9a3b3640 1 -- 172.21.15.67:0/2759315698 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad94147210 con 0x7fad9414cfc0 2024-03-24T00:48:56.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.120+0000 7fad90ff9640 1 -- 172.21.15.67:0/2759315698 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad84014440 con 0x7fad9414cfc0 2024-03-24T00:48:56.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.120+0000 7fad9a3b3640 1 -- 172.21.15.67:0/2759315698 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad941477d0 con 0x7fad9414cfc0 2024-03-24T00:48:56.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.121+0000 7fad9a3b3640 1 -- 172.21.15.67:0/2759315698 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad9404f400 con 0x7fad9414cfc0 2024-03-24T00:48:56.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.121+0000 7fad90ff9640 1 -- 172.21.15.67:0/2759315698 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fad84012040 con 0x7fad9414cfc0 2024-03-24T00:48:56.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.122+0000 7fad90ff9640 1 --2- 172.21.15.67:0/2759315698 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fad7406bc00 0x7fad7406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:56.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.122+0000 7fad90ff9640 1 -- 172.21.15.67:0/2759315698 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fad8408b120 con 0x7fad9414cfc0 2024-03-24T00:48:56.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.122+0000 7fad93fff640 1 --2- 172.21.15.67:0/2759315698 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fad7406bc00 0x7fad7406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:56.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.122+0000 7fad93fff640 1 --2- 172.21.15.67:0/2759315698 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fad7406bc00 0x7fad7406e0c0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fad7c034000 tx=0x7fad7c007000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:56.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.130+0000 7fad90ff9640 1 -- 172.21.15.67:0/2759315698 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fad84019030 con 0x7fad9414cfc0 2024-03-24T00:48:56.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:56.350+0000 7fad9a3b3640 1 -- 172.21.15.67:0/2759315698 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fad94159ef0 con 0x7fad9414cfc0 2024-03-24T00:48:57.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.003+0000 7fad90ff9640 1 -- 172.21.15.67:0/2759315698 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v86) v1 ==== 122+0+0 (secure 0 0 0) 0x7fad840594c0 con 0x7fad9414cfc0 2024-03-24T00:48:57.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.005+0000 7fad9a3b3640 1 -- 172.21.15.67:0/2759315698 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fad7406bc00 msgr2=0x7fad7406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:57.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.005+0000 7fad9a3b3640 1 --2- 172.21.15.67:0/2759315698 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fad7406bc00 0x7fad7406e0c0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fad7c034000 tx=0x7fad7c007000 comp rx=0 tx=0).stop 2024-03-24T00:48:57.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.005+0000 7fad9a3b3640 1 -- 172.21.15.67:0/2759315698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad9414cfc0 msgr2=0x7fad9414a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:57.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.005+0000 7fad9a3b3640 1 --2- 172.21.15.67:0/2759315698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad9414cfc0 0x7fad9414a280 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7fad840119a0 tx=0x7fad84011e60 comp rx=0 tx=0).stop 2024-03-24T00:48:57.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.005+0000 7fad9a3b3640 1 -- 172.21.15.67:0/2759315698 shutdown_connections 2024-03-24T00:48:57.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.005+0000 7fad9a3b3640 1 --2- 172.21.15.67:0/2759315698 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fad7406bc00 0x7fad7406e0c0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.005+0000 7fad9a3b3640 1 --2- 172.21.15.67:0/2759315698 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fad94146360 0x7fad940b1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.005+0000 7fad9a3b3640 1 --2- 172.21.15.67:0/2759315698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad9414cfc0 0x7fad9414a280 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.005+0000 7fad9a3b3640 1 --2- 172.21.15.67:0/2759315698 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad800039e0 0x7fad94149830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.005+0000 7fad9a3b3640 1 -- 172.21.15.67:0/2759315698 >> 172.21.15.67:0/2759315698 conn(0x7fad94099a30 msgr2=0x7fad940b1710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:57.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.005+0000 7fad9a3b3640 1 -- 172.21.15.67:0/2759315698 shutdown_connections 2024-03-24T00:48:57.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.005+0000 7fad9a3b3640 1 -- 172.21.15.67:0/2759315698 wait complete. 2024-03-24T00:48:57.016 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-24T00:48:57.016 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:48:57.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.118+0000 7fb68b8ff640 1 Processor -- start 2024-03-24T00:48:57.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.118+0000 7fb68b8ff640 1 -- start start 2024-03-24T00:48:57.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.119+0000 7fb68b8ff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb68415bf70 0x7fb6840cc690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:57.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.119+0000 7fb68b8ff640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb6840cfc70 0x7fb6840ce220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:57.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.119+0000 7fb68b8ff640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6840ccfe0 con 0x7fb6840ce760 2024-03-24T00:48:57.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.119+0000 7fb68b8ff640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6840cd2c0 con 0x7fb68415bf70 2024-03-24T00:48:57.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.119+0000 7fb68b8ff640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6840cd5a0 con 0x7fb6840cfc70 2024-03-24T00:48:57.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.119+0000 7fb68969b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb68415bf70 0x7fb6840cc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:57.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.119+0000 7fb68969b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb68415bf70 0x7fb6840cc690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57418/0 (socket says 172.21.15.67:57418) 2024-03-24T00:48:57.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.119+0000 7fb68969b640 1 -- 172.21.15.67:0/577460440 learned_addr learned my addr 172.21.15.67:0/577460440 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:57.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.119+0000 7fb68969b640 1 -- 172.21.15.67:0/577460440 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb6840cfc70 msgr2=0x7fb6840ce220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:57.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.119+0000 7fb688e9a640 1 --2- 172.21.15.67:0/577460440 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb6840cfc70 0x7fb6840ce220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:57.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.119+0000 7fb68969b640 1 --2- 172.21.15.67:0/577460440 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb6840cfc70 0x7fb6840ce220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.119+0000 7fb68969b640 1 -- 172.21.15.67:0/577460440 >> v1:172.21.15.72:6789/0 conn(0x7fb6840ce760 legacy=0x7fb68408b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:57.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.119+0000 7fb68969b640 1 -- 172.21.15.67:0/577460440 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6840cd880 con 0x7fb68415bf70 2024-03-24T00:48:57.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.119+0000 7fb688e9a640 1 --2- 172.21.15.67:0/577460440 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb6840cfc70 0x7fb6840ce220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:48:57.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.120+0000 7fb68969b640 1 --2- 172.21.15.67:0/577460440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb68415bf70 0x7fb6840cc690 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7fb680009870 tx=0x7fb68002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9d36fd5e95ca841 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:57.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.120+0000 7fb67bfff640 1 -- 172.21.15.67:0/577460440 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb680035020 con 0x7fb68415bf70 2024-03-24T00:48:57.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.120+0000 7fb67bfff640 1 -- 172.21.15.67:0/577460440 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb680031640 con 0x7fb68415bf70 2024-03-24T00:48:57.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.120+0000 7fb67bfff640 1 -- 172.21.15.67:0/577460440 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb68002d040 con 0x7fb68415bf70 2024-03-24T00:48:57.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.120+0000 7fb68b8ff640 1 -- 172.21.15.67:0/577460440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb68415bf70 msgr2=0x7fb6840cc690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:57.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.120+0000 7fb68b8ff640 1 --2- 172.21.15.67:0/577460440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb68415bf70 0x7fb6840cc690 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7fb680009870 tx=0x7fb68002b8f0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.120+0000 7fb68b8ff640 1 -- 172.21.15.67:0/577460440 shutdown_connections 2024-03-24T00:48:57.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.120+0000 7fb68b8ff640 1 --2- 172.21.15.67:0/577460440 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb6840cfc70 0x7fb6840ce220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.120+0000 7fb68b8ff640 1 --2- 172.21.15.67:0/577460440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb68415bf70 0x7fb6840cc690 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.120+0000 7fb68b8ff640 1 -- 172.21.15.67:0/577460440 >> 172.21.15.67:0/577460440 conn(0x7fb6840d1bf0 msgr2=0x7fb6840d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:57.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.121+0000 7fb68b8ff640 1 -- 172.21.15.67:0/577460440 shutdown_connections 2024-03-24T00:48:57.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.121+0000 7fb68b8ff640 1 -- 172.21.15.67:0/577460440 wait complete. 2024-03-24T00:48:57.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.122+0000 7fb68b8ff640 1 Processor -- start 2024-03-24T00:48:57.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.122+0000 7fb68b8ff640 1 -- start start 2024-03-24T00:48:57.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.122+0000 7fb68b8ff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb6840ce760 0x7fb68415c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:57.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.122+0000 7fb68b8ff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6840cfc70 0x7fb68415cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:57.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.122+0000 7fb68b8ff640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb68415bf70 0x7fb68415d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:57.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.122+0000 7fb68b8ff640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6840ccd50 con 0x7fb6840cfc70 2024-03-24T00:48:57.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.122+0000 7fb68b8ff640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6840ccbd0 con 0x7fb6840ce760 2024-03-24T00:48:57.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.122+0000 7fb68b8ff640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6840b2ce0 con 0x7fb68415bf70 2024-03-24T00:48:57.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.123+0000 7fb688e9a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6840cfc70 0x7fb68415cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:57.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.123+0000 7fb688e9a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6840cfc70 0x7fb68415cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57420/0 (socket says 172.21.15.67:57420) 2024-03-24T00:48:57.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.123+0000 7fb688e9a640 1 -- 172.21.15.67:0/2288101590 learned_addr learned my addr 172.21.15.67:0/2288101590 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:57.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.123+0000 7fb689e9c640 1 --2- 172.21.15.67:0/2288101590 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb68415bf70 0x7fb68415d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:57.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.123+0000 7fb68969b640 1 --2- 172.21.15.67:0/2288101590 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb6840ce760 0x7fb68415c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:57.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.123+0000 7fb688e9a640 1 -- 172.21.15.67:0/2288101590 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb68415bf70 msgr2=0x7fb68415d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:57.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.123+0000 7fb688e9a640 1 --2- 172.21.15.67:0/2288101590 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb68415bf70 0x7fb68415d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.123+0000 7fb688e9a640 1 -- 172.21.15.67:0/2288101590 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb6840ce760 msgr2=0x7fb68415c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:57.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.123+0000 7fb688e9a640 1 --2- 172.21.15.67:0/2288101590 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb6840ce760 0x7fb68415c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.123+0000 7fb688e9a640 1 -- 172.21.15.67:0/2288101590 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6840c6670 con 0x7fb6840cfc70 2024-03-24T00:48:57.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.123+0000 7fb689e9c640 1 --2- 172.21.15.67:0/2288101590 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb68415bf70 0x7fb68415d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:48:57.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.123+0000 7fb68969b640 1 --2- 172.21.15.67:0/2288101590 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb6840ce760 0x7fb68415c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:48:57.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.123+0000 7fb688e9a640 1 --2- 172.21.15.67:0/2288101590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6840cfc70 0x7fb68415cd40 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7fb67400c7f0 tx=0x7fb67400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:57.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.123+0000 7fb679ffb640 1 -- 172.21.15.67:0/2288101590 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb674015070 con 0x7fb6840cfc70 2024-03-24T00:48:57.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.123+0000 7fb679ffb640 1 -- 172.21.15.67:0/2288101590 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb674007820 con 0x7fb6840cfc70 2024-03-24T00:48:57.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.123+0000 7fb68b8ff640 1 -- 172.21.15.67:0/2288101590 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6840c6930 con 0x7fb6840cfc70 2024-03-24T00:48:57.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.123+0000 7fb679ffb640 1 -- 172.21.15.67:0/2288101590 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb674007bd0 con 0x7fb6840cfc70 2024-03-24T00:48:57.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.124+0000 7fb68b8ff640 1 -- 172.21.15.67:0/2288101590 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6840c6ef0 con 0x7fb6840cfc70 2024-03-24T00:48:57.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.124+0000 7fb68b8ff640 1 -- 172.21.15.67:0/2288101590 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb68404f400 con 0x7fb6840cfc70 2024-03-24T00:48:57.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.125+0000 7fb679ffb640 1 -- 172.21.15.67:0/2288101590 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb674018070 con 0x7fb6840cfc70 2024-03-24T00:48:57.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.125+0000 7fb679ffb640 1 --2- 172.21.15.67:0/2288101590 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb66806bbd0 0x7fb66806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:57.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.125+0000 7fb679ffb640 1 -- 172.21.15.67:0/2288101590 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb67408ad30 con 0x7fb6840cfc70 2024-03-24T00:48:57.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.125+0000 7fb68969b640 1 --2- 172.21.15.67:0/2288101590 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb66806bbd0 0x7fb66806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:57.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.126+0000 7fb68969b640 1 --2- 172.21.15.67:0/2288101590 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb66806bbd0 0x7fb66806e090 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fb680036000 tx=0x7fb680008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:57.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.130+0000 7fb679ffb640 1 -- 172.21.15.67:0/2288101590 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb674059690 con 0x7fb6840cfc70 2024-03-24T00:48:57.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.354+0000 7fb68b8ff640 1 -- 172.21.15.67:0/2288101590 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb6840cbcf0 con 0x7fb6840cfc70 2024-03-24T00:48:57.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.354+0000 7fb679ffb640 1 -- 172.21.15.67:0/2288101590 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+4973 (secure 0 0 0) 0x7fb674058f90 con 0x7fb6840cfc70 2024-03-24T00:48:57.354 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:48:57.355 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":86,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7298,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/839277571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":839277571},{"type":"v1","addr":"172.21.15.72:6836","nonce":839277571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":8224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2052380377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2052380377},{"type":"v1","addr":"172.21.15.67:6834","nonce":2052380377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":8239,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2196618204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2196618204},{"type":"v1","addr":"172.21.15.72:6837","nonce":2196618204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:48:54.351575+0000","modified":"2024-03-24T00:48:56.351565+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9219},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9219":{"gid":9219,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/312063805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":312063805},{"type":"v1","addr":"172.21.15.67:6835","nonce":312063805}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9219,"qdb_cluster":[9219]},"id":8}]} 2024-03-24T00:48:57.355 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 86 2024-03-24T00:48:57.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.356+0000 7fb68b8ff640 1 -- 172.21.15.67:0/2288101590 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb66806bbd0 msgr2=0x7fb66806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:57.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.356+0000 7fb68b8ff640 1 --2- 172.21.15.67:0/2288101590 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb66806bbd0 0x7fb66806e090 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fb680036000 tx=0x7fb680008030 comp rx=0 tx=0).stop 2024-03-24T00:48:57.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.356+0000 7fb68b8ff640 1 -- 172.21.15.67:0/2288101590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6840cfc70 msgr2=0x7fb68415cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:57.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.356+0000 7fb68b8ff640 1 --2- 172.21.15.67:0/2288101590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6840cfc70 0x7fb68415cd40 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7fb67400c7f0 tx=0x7fb67400ccb0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.356+0000 7fb68b8ff640 1 -- 172.21.15.67:0/2288101590 shutdown_connections 2024-03-24T00:48:57.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.356+0000 7fb68b8ff640 1 --2- 172.21.15.67:0/2288101590 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb66806bbd0 0x7fb66806e090 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.356+0000 7fb68b8ff640 1 --2- 172.21.15.67:0/2288101590 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb68415bf70 0x7fb68415d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.356+0000 7fb68b8ff640 1 --2- 172.21.15.67:0/2288101590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6840cfc70 0x7fb68415cd40 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.356+0000 7fb68b8ff640 1 --2- 172.21.15.67:0/2288101590 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb6840ce760 0x7fb68415c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.356+0000 7fb68b8ff640 1 -- 172.21.15.67:0/2288101590 >> 172.21.15.67:0/2288101590 conn(0x7fb6840d1bf0 msgr2=0x7fb68406fce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:57.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.357+0000 7fb68b8ff640 1 -- 172.21.15.67:0/2288101590 shutdown_connections 2024-03-24T00:48:57.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.357+0000 7fb68b8ff640 1 -- 172.21.15.67:0/2288101590 wait complete. 2024-03-24T00:48:57.367 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:48:57.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.468+0000 7fca70aea640 1 Processor -- start 2024-03-24T00:48:57.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.468+0000 7fca70aea640 1 -- start start 2024-03-24T00:48:57.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.469+0000 7fca70aea640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fca6c15e1b0 0x7fca6c0cfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:57.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.469+0000 7fca70aea640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fca6c09a410 0x7fca6c0cc880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:57.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.469+0000 7fca70aea640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca6c0d0fc0 con 0x7fca6c0b43a0 2024-03-24T00:48:57.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.469+0000 7fca70aea640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca6c15e590 con 0x7fca6c09a410 2024-03-24T00:48:57.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.469+0000 7fca70aea640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca6c0d1820 con 0x7fca6c15e1b0 2024-03-24T00:48:57.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.469+0000 7fca6ad9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fca6c09a410 0x7fca6c0cc880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:57.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.469+0000 7fca6ad9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fca6c09a410 0x7fca6c0cc880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57426/0 (socket says 172.21.15.67:57426) 2024-03-24T00:48:57.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.469+0000 7fca6ad9d640 1 -- 172.21.15.67:0/3487583284 learned_addr learned my addr 172.21.15.67:0/3487583284 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:57.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.469+0000 7fca6a59c640 1 --2- 172.21.15.67:0/3487583284 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fca6c15e1b0 0x7fca6c0cfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:57.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.470+0000 7fca6ad9d640 1 -- 172.21.15.67:0/3487583284 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fca6c15e1b0 msgr2=0x7fca6c0cfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:57.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.470+0000 7fca6ad9d640 1 --2- 172.21.15.67:0/3487583284 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fca6c15e1b0 0x7fca6c0cfe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.470+0000 7fca6ad9d640 1 -- 172.21.15.67:0/3487583284 >> v1:172.21.15.72:6789/0 conn(0x7fca6c0b43a0 legacy=0x7fca6c099d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:57.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.470+0000 7fca6ad9d640 1 -- 172.21.15.67:0/3487583284 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca6c0d1260 con 0x7fca6c09a410 2024-03-24T00:48:57.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.470+0000 7fca6ad9d640 1 --2- 172.21.15.67:0/3487583284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca6c09a410 0x7fca6c0cc880 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7fca64009e50 tx=0x7fca6402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbc72b48f2bc5007 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:57.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.471+0000 7fca6959a640 1 -- 172.21.15.67:0/3487583284 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca6402c9e0 con 0x7fca6c09a410 2024-03-24T00:48:57.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.471+0000 7fca6959a640 1 -- 172.21.15.67:0/3487583284 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca6402cb80 con 0x7fca6c09a410 2024-03-24T00:48:57.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.471+0000 7fca6959a640 1 -- 172.21.15.67:0/3487583284 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca640343f0 con 0x7fca6c09a410 2024-03-24T00:48:57.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.471+0000 7fca70aea640 1 -- 172.21.15.67:0/3487583284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca6c09a410 msgr2=0x7fca6c0cc880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:57.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.471+0000 7fca70aea640 1 --2- 172.21.15.67:0/3487583284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca6c09a410 0x7fca6c0cc880 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7fca64009e50 tx=0x7fca6402be40 comp rx=0 tx=0).stop 2024-03-24T00:48:57.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.471+0000 7fca70aea640 1 -- 172.21.15.67:0/3487583284 shutdown_connections 2024-03-24T00:48:57.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.471+0000 7fca70aea640 1 --2- 172.21.15.67:0/3487583284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca6c09a410 0x7fca6c0cc880 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.471+0000 7fca70aea640 1 --2- 172.21.15.67:0/3487583284 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fca6c15e1b0 0x7fca6c0cfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.471+0000 7fca70aea640 1 -- 172.21.15.67:0/3487583284 >> 172.21.15.67:0/3487583284 conn(0x7fca6c0b36d0 msgr2=0x7fca6c0b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:57.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.471+0000 7fca70aea640 1 -- 172.21.15.67:0/3487583284 shutdown_connections 2024-03-24T00:48:57.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.471+0000 7fca70aea640 1 -- 172.21.15.67:0/3487583284 wait complete. 2024-03-24T00:48:57.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.473+0000 7fca70aea640 1 Processor -- start 2024-03-24T00:48:57.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.473+0000 7fca70aea640 1 -- start start 2024-03-24T00:48:57.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.473+0000 7fca70aea640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca6c09a410 0x7fca6c150250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:57.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.473+0000 7fca70aea640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fca6c0b43a0 0x7fca6c14f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:57.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.473+0000 7fca70aea640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca6c15e1b0 0x7fca6c0aeb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:57.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.473+0000 7fca70aea640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca6c09a9a0 con 0x7fca6c09a410 2024-03-24T00:48:57.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.473+0000 7fca70aea640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca6c0cc360 con 0x7fca6c15e1b0 2024-03-24T00:48:57.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.473+0000 7fca70aea640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca6c0ccdc0 con 0x7fca6c0b43a0 2024-03-24T00:48:57.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.473+0000 7fca6a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca6c09a410 0x7fca6c150250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:57.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.473+0000 7fca6a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca6c09a410 0x7fca6c150250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57428/0 (socket says 172.21.15.67:57428) 2024-03-24T00:48:57.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.473+0000 7fca6a59c640 1 -- 172.21.15.67:0/3173167860 learned_addr learned my addr 172.21.15.67:0/3173167860 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:57.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.474+0000 7fca69d9b640 1 --2- 172.21.15.67:0/3173167860 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fca6c0b43a0 0x7fca6c14f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:57.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.474+0000 7fca6ad9d640 1 --2- 172.21.15.67:0/3173167860 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca6c15e1b0 0x7fca6c0aeb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:57.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.474+0000 7fca6a59c640 1 -- 172.21.15.67:0/3173167860 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fca6c0b43a0 msgr2=0x7fca6c14f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:57.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.474+0000 7fca6a59c640 1 --2- 172.21.15.67:0/3173167860 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fca6c0b43a0 0x7fca6c14f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.474+0000 7fca6a59c640 1 -- 172.21.15.67:0/3173167860 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca6c15e1b0 msgr2=0x7fca6c0aeb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:57.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.474+0000 7fca6a59c640 1 --2- 172.21.15.67:0/3173167860 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca6c15e1b0 0x7fca6c0aeb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.474+0000 7fca6a59c640 1 -- 172.21.15.67:0/3173167860 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca6c14ff20 con 0x7fca6c09a410 2024-03-24T00:48:57.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.474+0000 7fca6a59c640 1 --2- 172.21.15.67:0/3173167860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca6c09a410 0x7fca6c150250 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fca5c00e870 tx=0x7fca5c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:57.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.474+0000 7fca62ffd640 1 -- 172.21.15.67:0/3173167860 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca5c017070 con 0x7fca6c09a410 2024-03-24T00:48:57.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.474+0000 7fca62ffd640 1 -- 172.21.15.67:0/3173167860 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca5c00c940 con 0x7fca6c09a410 2024-03-24T00:48:57.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.474+0000 7fca70aea640 1 -- 172.21.15.67:0/3173167860 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca6c1525c0 con 0x7fca6c09a410 2024-03-24T00:48:57.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.474+0000 7fca62ffd640 1 -- 172.21.15.67:0/3173167860 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca5c00cd30 con 0x7fca6c09a410 2024-03-24T00:48:57.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.474+0000 7fca70aea640 1 -- 172.21.15.67:0/3173167860 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca6c152b80 con 0x7fca6c09a410 2024-03-24T00:48:57.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.475+0000 7fca70aea640 1 -- 172.21.15.67:0/3173167860 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca6c04f400 con 0x7fca6c09a410 2024-03-24T00:48:57.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.476+0000 7fca62ffd640 1 -- 172.21.15.67:0/3173167860 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fca5c01a070 con 0x7fca6c09a410 2024-03-24T00:48:57.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.476+0000 7fca62ffd640 1 --2- 172.21.15.67:0/3173167860 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fca4c06bbd0 0x7fca4c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:57.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.476+0000 7fca62ffd640 1 -- 172.21.15.67:0/3173167860 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fca5c08aa20 con 0x7fca6c09a410 2024-03-24T00:48:57.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.477+0000 7fca69d9b640 1 --2- 172.21.15.67:0/3173167860 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fca4c06bbd0 0x7fca4c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:57.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.477+0000 7fca69d9b640 1 --2- 172.21.15.67:0/3173167860 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fca4c06bbd0 0x7fca4c06e090 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fca6c0cc9f0 tx=0x7fca54009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:57.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.485+0000 7fca62ffd640 1 -- 172.21.15.67:0/3173167860 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fca5c058d30 con 0x7fca6c09a410 2024-03-24T00:48:57.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.670+0000 7fca70aea640 1 -- 172.21.15.67:0/3173167860 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fca6c1531d0 con 0x7fca6c09a410 2024-03-24T00:48:57.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.671+0000 7fca62ffd640 1 -- 172.21.15.67:0/3173167860 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v120) v1 ==== 74+0+16892 (secure 0 0 0) 0x7fca5c05cdd0 con 0x7fca6c09a410 2024-03-24T00:48:57.671 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:48:57.671 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":120,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:48:55.987829+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:48:52.622356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-03-24T00:48:53.311452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:48:57.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.673+0000 7fca70aea640 1 -- 172.21.15.67:0/3173167860 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fca4c06bbd0 msgr2=0x7fca4c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:57.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.673+0000 7fca70aea640 1 --2- 172.21.15.67:0/3173167860 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fca4c06bbd0 0x7fca4c06e090 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fca6c0cc9f0 tx=0x7fca54009040 comp rx=0 tx=0).stop 2024-03-24T00:48:57.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.673+0000 7fca70aea640 1 -- 172.21.15.67:0/3173167860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca6c09a410 msgr2=0x7fca6c150250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:57.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.673+0000 7fca70aea640 1 --2- 172.21.15.67:0/3173167860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca6c09a410 0x7fca6c150250 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fca5c00e870 tx=0x7fca5c00ed30 comp rx=0 tx=0).stop 2024-03-24T00:48:57.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.673+0000 7fca70aea640 1 -- 172.21.15.67:0/3173167860 shutdown_connections 2024-03-24T00:48:57.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.673+0000 7fca70aea640 1 --2- 172.21.15.67:0/3173167860 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fca4c06bbd0 0x7fca4c06e090 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.673+0000 7fca70aea640 1 --2- 172.21.15.67:0/3173167860 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca6c15e1b0 0x7fca6c0aeb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.673+0000 7fca70aea640 1 --2- 172.21.15.67:0/3173167860 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fca6c0b43a0 0x7fca6c14f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.673+0000 7fca70aea640 1 --2- 172.21.15.67:0/3173167860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca6c09a410 0x7fca6c150250 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.673+0000 7fca70aea640 1 -- 172.21.15.67:0/3173167860 >> 172.21.15.67:0/3173167860 conn(0x7fca6c0b36d0 msgr2=0x7fca6c070760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:57.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.673+0000 7fca70aea640 1 -- 172.21.15.67:0/3173167860 shutdown_connections 2024-03-24T00:48:57.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.673+0000 7fca70aea640 1 -- 172.21.15.67:0/3173167860 wait complete. 2024-03-24T00:48:57.684 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-24T00:48:57.684 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:48:57.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.786+0000 7fd8f5481640 1 Processor -- start 2024-03-24T00:48:57.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.786+0000 7fd8f5481640 1 -- start start 2024-03-24T00:48:57.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.787+0000 7fd8f5481640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd8f015dec0 0x7fd8f00ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:57.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.787+0000 7fd8f5481640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8f00cf640 con 0x7fd8f015e2a0 2024-03-24T00:48:57.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.787+0000 7fd8f5481640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8f00cf080 con 0x7fd8f00d0ef0 2024-03-24T00:48:57.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.787+0000 7fd8f5481640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8f00cee30 con 0x7fd8f015dec0 2024-03-24T00:48:57.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.787+0000 7fd8ee7fc640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd8f015e2a0 0x7fd8f00d07e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35916/0 (socket says 172.21.15.67:35916) 2024-03-24T00:48:57.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.787+0000 7fd8ee7fc640 1 -- 172.21.15.67:0/555607356 learned_addr learned my addr 172.21.15.67:0/555607356 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:48:57.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.787+0000 7fd8eeffd640 1 --2- 172.21.15.67:0/555607356 >> v2:172.21.15.72:3300/0 conn(0x7fd8f015dec0 0x7fd8f00ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:57.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.787+0000 7fd8edffb640 1 -- 172.21.15.67:0/555607356 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 10623353 0 0) 0x7fd8e4001110 con 0x7fd8f015e2a0 2024-03-24T00:48:57.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.787+0000 7fd8edffb640 1 -- 172.21.15.67:0/555607356 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8d8003520 con 0x7fd8f015e2a0 2024-03-24T00:48:57.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.788+0000 7fd8eeffd640 1 -- 172.21.15.67:0/555607356 >> v1:172.21.15.72:6789/0 conn(0x7fd8f00d0ef0 legacy=0x7fd8f00d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:57.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.788+0000 7fd8edffb640 1 -- 172.21.15.67:0/555607356 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1252939708 0 0) 0x7fd8f00cf080 con 0x7fd8f00d0ef0 2024-03-24T00:48:57.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.788+0000 7fd8eeffd640 1 -- 172.21.15.67:0/555607356 >> v1:172.21.15.67:6789/0 conn(0x7fd8f015e2a0 legacy=0x7fd8f00d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:57.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.788+0000 7fd8eeffd640 1 -- 172.21.15.67:0/555607356 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8f00cf360 con 0x7fd8f015dec0 2024-03-24T00:48:57.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.788+0000 7fd8eeffd640 1 --2- 172.21.15.67:0/555607356 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd8f015dec0 0x7fd8f00ce630 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fd8f00cefb0 tx=0x7fd8e002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=748e80519cdc8bdc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:57.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.789+0000 7fd8edffb640 1 -- 172.21.15.67:0/555607356 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e0035020 con 0x7fd8f015dec0 2024-03-24T00:48:57.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.789+0000 7fd8edffb640 1 -- 172.21.15.67:0/555607356 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8e0031830 con 0x7fd8f015dec0 2024-03-24T00:48:57.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.789+0000 7fd8edffb640 1 -- 172.21.15.67:0/555607356 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e0031c00 con 0x7fd8f015dec0 2024-03-24T00:48:57.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.789+0000 7fd8f5481640 1 -- 172.21.15.67:0/555607356 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd8f015dec0 msgr2=0x7fd8f00ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:57.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.789+0000 7fd8f5481640 1 --2- 172.21.15.67:0/555607356 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd8f015dec0 0x7fd8f00ce630 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fd8f00cefb0 tx=0x7fd8e002b8d0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.789+0000 7fd8f5481640 1 -- 172.21.15.67:0/555607356 shutdown_connections 2024-03-24T00:48:57.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.789+0000 7fd8f5481640 1 --2- 172.21.15.67:0/555607356 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd8f015dec0 0x7fd8f00ce630 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.789+0000 7fd8f5481640 1 -- 172.21.15.67:0/555607356 >> 172.21.15.67:0/555607356 conn(0x7fd8f006c810 msgr2=0x7fd8f006cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:57.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.789+0000 7fd8f5481640 1 -- 172.21.15.67:0/555607356 shutdown_connections 2024-03-24T00:48:57.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.789+0000 7fd8f5481640 1 -- 172.21.15.67:0/555607356 wait complete. 2024-03-24T00:48:57.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.790+0000 7fd8f5481640 1 Processor -- start 2024-03-24T00:48:57.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.790+0000 7fd8f5481640 1 -- start start 2024-03-24T00:48:57.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.791+0000 7fd8f5481640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd8f00d0ef0 0x7fd8f015ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:57.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.791+0000 7fd8f5481640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd8f015dec0 0x7fd8f00aa870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:57.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.791+0000 7fd8f5481640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8f015e2a0 0x7fd8f00aadb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:57.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.791+0000 7fd8f5481640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f0162a40 con 0x7fd8f015e2a0 2024-03-24T00:48:57.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.791+0000 7fd8f5481640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f01628c0 con 0x7fd8f00d0ef0 2024-03-24T00:48:57.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.791+0000 7fd8f5481640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f00d2120 con 0x7fd8f015dec0 2024-03-24T00:48:57.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.791+0000 7fd8ef7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8f015e2a0 0x7fd8f00aadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:57.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.791+0000 7fd8ef7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8f015e2a0 0x7fd8f00aadb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57442/0 (socket says 172.21.15.67:57442) 2024-03-24T00:48:57.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.791+0000 7fd8ef7fe640 1 -- 172.21.15.67:0/1526170666 learned_addr learned my addr 172.21.15.67:0/1526170666 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:57.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.791+0000 7fd8eeffd640 1 --2- 172.21.15.67:0/1526170666 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd8f00d0ef0 0x7fd8f015ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:57.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.791+0000 7fd8ee7fc640 1 --2- 172.21.15.67:0/1526170666 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd8f015dec0 0x7fd8f00aa870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:57.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.792+0000 7fd8ef7fe640 1 -- 172.21.15.67:0/1526170666 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd8f015dec0 msgr2=0x7fd8f00aa870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:57.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.792+0000 7fd8ef7fe640 1 --2- 172.21.15.67:0/1526170666 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd8f015dec0 0x7fd8f00aa870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.792+0000 7fd8ef7fe640 1 -- 172.21.15.67:0/1526170666 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd8f00d0ef0 msgr2=0x7fd8f015ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:57.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.792+0000 7fd8ef7fe640 1 --2- 172.21.15.67:0/1526170666 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd8f00d0ef0 0x7fd8f015ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.792+0000 7fd8ef7fe640 1 -- 172.21.15.67:0/1526170666 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8f01f4f30 con 0x7fd8f015e2a0 2024-03-24T00:48:57.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.792+0000 7fd8ef7fe640 1 --2- 172.21.15.67:0/1526170666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8f015e2a0 0x7fd8f00aadb0 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7fd8e800c760 tx=0x7fd8e800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:57.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.792+0000 7fd8d77fe640 1 -- 172.21.15.67:0/1526170666 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e8003da0 con 0x7fd8f015e2a0 2024-03-24T00:48:57.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.792+0000 7fd8d77fe640 1 -- 172.21.15.67:0/1526170666 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8e8005070 con 0x7fd8f015e2a0 2024-03-24T00:48:57.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.792+0000 7fd8f5481640 1 -- 172.21.15.67:0/1526170666 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8f01f51d0 con 0x7fd8f015e2a0 2024-03-24T00:48:57.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.792+0000 7fd8d77fe640 1 -- 172.21.15.67:0/1526170666 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e8005460 con 0x7fd8f015e2a0 2024-03-24T00:48:57.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.792+0000 7fd8f5481640 1 -- 172.21.15.67:0/1526170666 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8f01f5790 con 0x7fd8f015e2a0 2024-03-24T00:48:57.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.793+0000 7fd8f5481640 1 -- 172.21.15.67:0/1526170666 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8f004f400 con 0x7fd8f015e2a0 2024-03-24T00:48:57.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.794+0000 7fd8d77fe640 1 -- 172.21.15.67:0/1526170666 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd8e8005600 con 0x7fd8f015e2a0 2024-03-24T00:48:57.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.794+0000 7fd8d77fe640 1 --2- 172.21.15.67:0/1526170666 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd8dc06bbd0 0x7fd8dc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:57.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.794+0000 7fd8d77fe640 1 -- 172.21.15.67:0/1526170666 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd8e808a8f0 con 0x7fd8f015e2a0 2024-03-24T00:48:57.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.794+0000 7fd8eeffd640 1 --2- 172.21.15.67:0/1526170666 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd8dc06bbd0 0x7fd8dc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:57.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.795+0000 7fd8eeffd640 1 --2- 172.21.15.67:0/1526170666 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd8dc06bbd0 0x7fd8dc06e090 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fd8e0036000 tx=0x7fd8e0008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:57.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.798+0000 7fd8d77fe640 1 -- 172.21.15.67:0/1526170666 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd8e8058c90 con 0x7fd8f015e2a0 2024-03-24T00:48:57.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.975+0000 7fd8f5481640 1 -- 172.21.15.67:0/1526170666 --> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd8f00d1aa0 con 0x7fd8dc06bbd0 2024-03-24T00:48:57.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.981+0000 7fd8d77fe640 1 -- 172.21.15.67:0/1526170666 <== mgr.4105 v2:172.21.15.72:6832/2768292274 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+247087 (secure 0 0 0) 0x7fd8e0086540 con 0x7fd8dc06bbd0 2024-03-24T00:48:57.984 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:48:57.985 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-24T00:48:57.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.985+0000 7fd8f5481640 1 -- 172.21.15.67:0/1526170666 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd8dc06bbd0 msgr2=0x7fd8dc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:57.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.985+0000 7fd8f5481640 1 --2- 172.21.15.67:0/1526170666 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd8dc06bbd0 0x7fd8dc06e090 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fd8e0036000 tx=0x7fd8e0008030 comp rx=0 tx=0).stop 2024-03-24T00:48:57.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.985+0000 7fd8f5481640 1 -- 172.21.15.67:0/1526170666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8f015e2a0 msgr2=0x7fd8f00aadb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:57.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.985+0000 7fd8f5481640 1 --2- 172.21.15.67:0/1526170666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8f015e2a0 0x7fd8f00aadb0 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7fd8e800c760 tx=0x7fd8e800cc20 comp rx=0 tx=0).stop 2024-03-24T00:48:57.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.985+0000 7fd8f5481640 1 -- 172.21.15.67:0/1526170666 shutdown_connections 2024-03-24T00:48:57.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.985+0000 7fd8f5481640 1 --2- 172.21.15.67:0/1526170666 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd8dc06bbd0 0x7fd8dc06e090 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.985+0000 7fd8f5481640 1 --2- 172.21.15.67:0/1526170666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8f015e2a0 0x7fd8f00aadb0 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.985+0000 7fd8f5481640 1 --2- 172.21.15.67:0/1526170666 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd8f015dec0 0x7fd8f00aa870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.985+0000 7fd8f5481640 1 --2- 172.21.15.67:0/1526170666 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd8f00d0ef0 0x7fd8f015ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:57.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.985+0000 7fd8f5481640 1 -- 172.21.15.67:0/1526170666 >> 172.21.15.67:0/1526170666 conn(0x7fd8f006c810 msgr2=0x7fd8f00d25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:57.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.985+0000 7fd8f5481640 1 -- 172.21.15.67:0/1526170666 shutdown_connections 2024-03-24T00:48:57.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:57.985+0000 7fd8f5481640 1 -- 172.21.15.67:0/1526170666 wait complete. 2024-03-24T00:48:57.996 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":422,"stamp":"2024-03-24T00:48:57.468891+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":8,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":262036,"kb_used_data":12588,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754712684,"statfs":{"total":773094113280,"available":772825788416,"internally_reserved":0,"allocated":12890112,"data_stored":6632745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.496750"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.981887+0000","last_change":"2024-03-24T00:48:53.981887+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.981887+0000","last_undegraded":"2024-03-24T00:48:53.981887+0000","last_fullsized":"2024-03-24T00:48:53.981887+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.987478+0000","last_change":"2024-03-24T00:48:53.987478+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.987478+0000","last_undegraded":"2024-03-24T00:48:53.987478+0000","last_fullsized":"2024-03-24T00:48:53.987478+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.986072+0000","last_change":"2024-03-24T00:48:53.986072+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.986072+0000","last_undegraded":"2024-03-24T00:48:53.986072+0000","last_fullsized":"2024-03-24T00:48:53.986072+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.981339+0000","last_change":"2024-03-24T00:48:53.981339+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.981339+0000","last_undegraded":"2024-03-24T00:48:53.981339+0000","last_fullsized":"2024-03-24T00:48:53.981339+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.986981+0000","last_change":"2024-03-24T00:48:53.986981+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.986981+0000","last_undegraded":"2024-03-24T00:48:53.986981+0000","last_fullsized":"2024-03-24T00:48:53.986981+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.988748+0000","last_change":"2024-03-24T00:48:53.988748+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.988748+0000","last_undegraded":"2024-03-24T00:48:53.988748+0000","last_fullsized":"2024-03-24T00:48:53.988748+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.987478+0000","last_change":"2024-03-24T00:48:53.987478+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.987478+0000","last_undegraded":"2024-03-24T00:48:53.987478+0000","last_fullsized":"2024-03-24T00:48:53.987478+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.986010+0000","last_change":"2024-03-24T00:48:53.986010+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.986010+0000","last_undegraded":"2024-03-24T00:48:53.986010+0000","last_fullsized":"2024-03-24T00:48:53.986010+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.982242+0000","last_change":"2024-03-24T00:48:53.982242+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.982242+0000","last_undegraded":"2024-03-24T00:48:53.982242+0000","last_fullsized":"2024-03-24T00:48:53.982242+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.990365+0000","last_change":"2024-03-24T00:48:53.990365+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.990365+0000","last_undegraded":"2024-03-24T00:48:53.990365+0000","last_fullsized":"2024-03-24T00:48:53.990365+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.981850+0000","last_change":"2024-03-24T00:48:53.981850+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.981850+0000","last_undegraded":"2024-03-24T00:48:53.981850+0000","last_fullsized":"2024-03-24T00:48:53.981850+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.987024+0000","last_change":"2024-03-24T00:48:53.987024+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.987024+0000","last_undegraded":"2024-03-24T00:48:53.987024+0000","last_fullsized":"2024-03-24T00:48:53.987024+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.989677+0000","last_change":"2024-03-24T00:48:53.989677+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.989677+0000","last_undegraded":"2024-03-24T00:48:53.989677+0000","last_fullsized":"2024-03-24T00:48:53.989677+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.985341+0000","last_change":"2024-03-24T00:48:53.985341+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.985341+0000","last_undegraded":"2024-03-24T00:48:53.985341+0000","last_fullsized":"2024-03-24T00:48:53.985341+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.981660+0000","last_change":"2024-03-24T00:48:53.981660+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.981660+0000","last_undegraded":"2024-03-24T00:48:53.981660+0000","last_fullsized":"2024-03-24T00:48:53.981660+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.986575+0000","last_change":"2024-03-24T00:48:53.986575+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.986575+0000","last_undegraded":"2024-03-24T00:48:53.986575+0000","last_fullsized":"2024-03-24T00:48:53.986575+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.989175+0000","last_change":"2024-03-24T00:48:53.989175+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.989175+0000","last_undegraded":"2024-03-24T00:48:53.989175+0000","last_fullsized":"2024-03-24T00:48:53.989175+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.984128+0000","last_change":"2024-03-24T00:48:53.984128+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.984128+0000","last_undegraded":"2024-03-24T00:48:53.984128+0000","last_fullsized":"2024-03-24T00:48:53.984128+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.983103+0000","last_change":"2024-03-24T00:48:53.983103+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.983103+0000","last_undegraded":"2024-03-24T00:48:53.983103+0000","last_fullsized":"2024-03-24T00:48:53.983103+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.977625+0000","last_change":"2024-03-24T00:48:53.977625+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.977625+0000","last_undegraded":"2024-03-24T00:48:53.977625+0000","last_fullsized":"2024-03-24T00:48:53.977625+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.982297+0000","last_change":"2024-03-24T00:48:53.982297+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.982297+0000","last_undegraded":"2024-03-24T00:48:53.982297+0000","last_fullsized":"2024-03-24T00:48:53.982297+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.982445+0000","last_change":"2024-03-24T00:48:53.982445+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.982445+0000","last_undegraded":"2024-03-24T00:48:53.982445+0000","last_fullsized":"2024-03-24T00:48:53.982445+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.982875+0000","last_change":"2024-03-24T00:48:53.982875+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.982875+0000","last_undegraded":"2024-03-24T00:48:53.982875+0000","last_fullsized":"2024-03-24T00:48:53.982875+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.979109+0000","last_change":"2024-03-24T00:48:53.979109+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.979109+0000","last_undegraded":"2024-03-24T00:48:53.979109+0000","last_fullsized":"2024-03-24T00:48:53.979109+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.979993+0000","last_change":"2024-03-24T00:48:53.979993+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.979993+0000","last_undegraded":"2024-03-24T00:48:53.979993+0000","last_fullsized":"2024-03-24T00:48:53.979993+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.978637+0000","last_change":"2024-03-24T00:48:53.978637+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.978637+0000","last_undegraded":"2024-03-24T00:48:53.978637+0000","last_fullsized":"2024-03-24T00:48:53.978637+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.982655+0000","last_change":"2024-03-24T00:48:53.982655+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.982655+0000","last_undegraded":"2024-03-24T00:48:53.982655+0000","last_fullsized":"2024-03-24T00:48:53.982655+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.986026+0000","last_change":"2024-03-24T00:48:53.986026+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.986026+0000","last_undegraded":"2024-03-24T00:48:53.986026+0000","last_fullsized":"2024-03-24T00:48:53.986026+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.977622+0000","last_change":"2024-03-24T00:48:53.977622+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.977622+0000","last_undegraded":"2024-03-24T00:48:53.977622+0000","last_fullsized":"2024-03-24T00:48:53.977622+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.984581+0000","last_change":"2024-03-24T00:48:53.984581+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.984581+0000","last_undegraded":"2024-03-24T00:48:53.984581+0000","last_fullsized":"2024-03-24T00:48:53.984581+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.979193+0000","last_change":"2024-03-24T00:48:53.979193+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.979193+0000","last_undegraded":"2024-03-24T00:48:53.979193+0000","last_fullsized":"2024-03-24T00:48:53.979193+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.978546+0000","last_change":"2024-03-24T00:48:53.978546+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.978546+0000","last_undegraded":"2024-03-24T00:48:53.978546+0000","last_fullsized":"2024-03-24T00:48:53.978546+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.990974+0000","last_change":"2024-03-24T00:48:53.990974+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.990974+0000","last_undegraded":"2024-03-24T00:48:53.990974+0000","last_fullsized":"2024-03-24T00:48:53.990974+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.997599+0000","last_change":"2024-03-24T00:48:53.992874+0000","last_active":"2024-03-24T00:48:53.997599+0000","last_peered":"2024-03-24T00:48:53.997599+0000","last_clean":"2024-03-24T00:48:53.997599+0000","last_became_active":"2024-03-24T00:48:53.992507+0000","last_became_peered":"2024-03-24T00:48:53.992507+0000","last_unstale":"2024-03-24T00:48:53.997599+0000","last_undegraded":"2024-03-24T00:48:53.997599+0000","last_fullsized":"2024-03-24T00:48:53.997599+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:17:05.808342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.985412+0000","last_change":"2024-03-24T00:48:53.985412+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.985412+0000","last_undegraded":"2024-03-24T00:48:53.985412+0000","last_fullsized":"2024-03-24T00:48:53.985412+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.984444+0000","last_change":"2024-03-24T00:48:53.982887+0000","last_active":"2024-03-24T00:48:53.984444+0000","last_peered":"2024-03-24T00:48:53.984444+0000","last_clean":"2024-03-24T00:48:53.984444+0000","last_became_active":"2024-03-24T00:48:53.982515+0000","last_became_peered":"2024-03-24T00:48:53.982515+0000","last_unstale":"2024-03-24T00:48:53.984444+0000","last_undegraded":"2024-03-24T00:48:53.984444+0000","last_fullsized":"2024-03-24T00:48:53.984444+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:31:25.866307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.978888+0000","last_change":"2024-03-24T00:48:53.978888+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.978888+0000","last_undegraded":"2024-03-24T00:48:53.978888+0000","last_fullsized":"2024-03-24T00:48:53.978888+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.997498+0000","last_change":"2024-03-24T00:48:53.993536+0000","last_active":"2024-03-24T00:48:53.997498+0000","last_peered":"2024-03-24T00:48:53.997498+0000","last_clean":"2024-03-24T00:48:53.997498+0000","last_became_active":"2024-03-24T00:48:53.993136+0000","last_became_peered":"2024-03-24T00:48:53.993136+0000","last_unstale":"2024-03-24T00:48:53.997498+0000","last_undegraded":"2024-03-24T00:48:53.997498+0000","last_fullsized":"2024-03-24T00:48:53.997498+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:26:39.618413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.981809+0000","last_change":"2024-03-24T00:48:53.981809+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.981809+0000","last_undegraded":"2024-03-24T00:48:53.981809+0000","last_fullsized":"2024-03-24T00:48:53.981809+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.996455+0000","last_change":"2024-03-24T00:48:53.993401+0000","last_active":"2024-03-24T00:48:53.996455+0000","last_peered":"2024-03-24T00:48:53.996455+0000","last_clean":"2024-03-24T00:48:53.996455+0000","last_became_active":"2024-03-24T00:48:53.991401+0000","last_became_peered":"2024-03-24T00:48:53.991401+0000","last_unstale":"2024-03-24T00:48:53.996455+0000","last_undegraded":"2024-03-24T00:48:53.996455+0000","last_fullsized":"2024-03-24T00:48:53.996455+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:32.954517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.986877+0000","last_change":"2024-03-24T00:48:53.986877+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.986877+0000","last_undegraded":"2024-03-24T00:48:53.986877+0000","last_fullsized":"2024-03-24T00:48:53.986877+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.997557+0000","last_change":"2024-03-24T00:48:53.994950+0000","last_active":"2024-03-24T00:48:53.997557+0000","last_peered":"2024-03-24T00:48:53.997557+0000","last_clean":"2024-03-24T00:48:53.997557+0000","last_became_active":"2024-03-24T00:48:53.994709+0000","last_became_peered":"2024-03-24T00:48:53.994709+0000","last_unstale":"2024-03-24T00:48:53.997557+0000","last_undegraded":"2024-03-24T00:48:53.997557+0000","last_fullsized":"2024-03-24T00:48:53.997557+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:45:58.215004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.982871+0000","last_change":"2024-03-24T00:48:53.982871+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.982871+0000","last_undegraded":"2024-03-24T00:48:53.982871+0000","last_fullsized":"2024-03-24T00:48:53.982871+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:54.000260+0000","last_change":"2024-03-24T00:48:53.997359+0000","last_active":"2024-03-24T00:48:54.000260+0000","last_peered":"2024-03-24T00:48:54.000260+0000","last_clean":"2024-03-24T00:48:54.000260+0000","last_became_active":"2024-03-24T00:48:53.997039+0000","last_became_peered":"2024-03-24T00:48:53.997039+0000","last_unstale":"2024-03-24T00:48:54.000260+0000","last_undegraded":"2024-03-24T00:48:54.000260+0000","last_fullsized":"2024-03-24T00:48:54.000260+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:03:39.280823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.984118+0000","last_change":"2024-03-24T00:48:53.984118+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.984118+0000","last_undegraded":"2024-03-24T00:48:53.984118+0000","last_fullsized":"2024-03-24T00:48:53.984118+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.992054+0000","last_change":"2024-03-24T00:48:53.988155+0000","last_active":"2024-03-24T00:48:53.992054+0000","last_peered":"2024-03-24T00:48:53.992054+0000","last_clean":"2024-03-24T00:48:53.992054+0000","last_became_active":"2024-03-24T00:48:53.987659+0000","last_became_peered":"2024-03-24T00:48:53.987659+0000","last_unstale":"2024-03-24T00:48:53.992054+0000","last_undegraded":"2024-03-24T00:48:53.992054+0000","last_fullsized":"2024-03-24T00:48:53.992054+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:32:33.042770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.987465+0000","last_change":"2024-03-24T00:48:53.987465+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.987465+0000","last_undegraded":"2024-03-24T00:48:53.987465+0000","last_fullsized":"2024-03-24T00:48:53.987465+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.998570+0000","last_change":"2024-03-24T00:48:53.993610+0000","last_active":"2024-03-24T00:48:53.998570+0000","last_peered":"2024-03-24T00:48:53.998570+0000","last_clean":"2024-03-24T00:48:53.998570+0000","last_became_active":"2024-03-24T00:48:53.993326+0000","last_became_peered":"2024-03-24T00:48:53.993326+0000","last_unstale":"2024-03-24T00:48:53.998570+0000","last_undegraded":"2024-03-24T00:48:53.998570+0000","last_fullsized":"2024-03-24T00:48:53.998570+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:20:08.465917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.983527+0000","last_change":"2024-03-24T00:48:53.983527+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.983527+0000","last_undegraded":"2024-03-24T00:48:53.983527+0000","last_fullsized":"2024-03-24T00:48:53.983527+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:54.003266+0000","last_change":"2024-03-24T00:48:53.999330+0000","last_active":"2024-03-24T00:48:54.003266+0000","last_peered":"2024-03-24T00:48:54.003266+0000","last_clean":"2024-03-24T00:48:54.003266+0000","last_became_active":"2024-03-24T00:48:53.999065+0000","last_became_peered":"2024-03-24T00:48:53.999065+0000","last_unstale":"2024-03-24T00:48:54.003266+0000","last_undegraded":"2024-03-24T00:48:54.003266+0000","last_fullsized":"2024-03-24T00:48:54.003266+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:45:58.215004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.978769+0000","last_change":"2024-03-24T00:48:53.978769+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.978769+0000","last_undegraded":"2024-03-24T00:48:53.978769+0000","last_fullsized":"2024-03-24T00:48:53.978769+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.991791+0000","last_change":"2024-03-24T00:48:53.985480+0000","last_active":"2024-03-24T00:48:53.991791+0000","last_peered":"2024-03-24T00:48:53.991791+0000","last_clean":"2024-03-24T00:48:53.991791+0000","last_became_active":"2024-03-24T00:48:53.985203+0000","last_became_peered":"2024-03-24T00:48:53.985203+0000","last_unstale":"2024-03-24T00:48:53.991791+0000","last_undegraded":"2024-03-24T00:48:53.991791+0000","last_fullsized":"2024-03-24T00:48:53.991791+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:21:21.714524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.983568+0000","last_change":"2024-03-24T00:48:53.983568+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.983568+0000","last_undegraded":"2024-03-24T00:48:53.983568+0000","last_fullsized":"2024-03-24T00:48:53.983568+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.992006+0000","last_change":"2024-03-24T00:48:53.987256+0000","last_active":"2024-03-24T00:48:53.992006+0000","last_peered":"2024-03-24T00:48:53.992006+0000","last_clean":"2024-03-24T00:48:53.992006+0000","last_became_active":"2024-03-24T00:48:53.986435+0000","last_became_peered":"2024-03-24T00:48:53.986435+0000","last_unstale":"2024-03-24T00:48:53.992006+0000","last_undegraded":"2024-03-24T00:48:53.992006+0000","last_fullsized":"2024-03-24T00:48:53.992006+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:07:15.456620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.979598+0000","last_change":"2024-03-24T00:48:53.979598+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.979598+0000","last_undegraded":"2024-03-24T00:48:53.979598+0000","last_fullsized":"2024-03-24T00:48:53.979598+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.997830+0000","last_change":"2024-03-24T00:48:53.992589+0000","last_active":"2024-03-24T00:48:53.997830+0000","last_peered":"2024-03-24T00:48:53.997830+0000","last_clean":"2024-03-24T00:48:53.997830+0000","last_became_active":"2024-03-24T00:48:53.992273+0000","last_became_peered":"2024-03-24T00:48:53.992273+0000","last_unstale":"2024-03-24T00:48:53.997830+0000","last_undegraded":"2024-03-24T00:48:53.997830+0000","last_fullsized":"2024-03-24T00:48:53.997830+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:01:04.094765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.986841+0000","last_change":"2024-03-24T00:48:53.986841+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.986841+0000","last_undegraded":"2024-03-24T00:48:53.986841+0000","last_fullsized":"2024-03-24T00:48:53.986841+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.992329+0000","last_change":"2024-03-24T00:48:53.988410+0000","last_active":"2024-03-24T00:48:53.992329+0000","last_peered":"2024-03-24T00:48:53.992329+0000","last_clean":"2024-03-24T00:48:53.992329+0000","last_became_active":"2024-03-24T00:48:53.987901+0000","last_became_peered":"2024-03-24T00:48:53.987901+0000","last_unstale":"2024-03-24T00:48:53.992329+0000","last_undegraded":"2024-03-24T00:48:53.992329+0000","last_fullsized":"2024-03-24T00:48:53.992329+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:10:25.529558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.986869+0000","last_change":"2024-03-24T00:48:53.986869+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.986869+0000","last_undegraded":"2024-03-24T00:48:53.986869+0000","last_fullsized":"2024-03-24T00:48:53.986869+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.992260+0000","last_change":"2024-03-24T00:48:53.988076+0000","last_active":"2024-03-24T00:48:53.992260+0000","last_peered":"2024-03-24T00:48:53.992260+0000","last_clean":"2024-03-24T00:48:53.992260+0000","last_became_active":"2024-03-24T00:48:53.987752+0000","last_became_peered":"2024-03-24T00:48:53.987752+0000","last_unstale":"2024-03-24T00:48:53.992260+0000","last_undegraded":"2024-03-24T00:48:53.992260+0000","last_fullsized":"2024-03-24T00:48:53.992260+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:24:44.940524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.977688+0000","last_change":"2024-03-24T00:48:53.977688+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.977688+0000","last_undegraded":"2024-03-24T00:48:53.977688+0000","last_fullsized":"2024-03-24T00:48:53.977688+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.997839+0000","last_change":"2024-03-24T00:48:53.992865+0000","last_active":"2024-03-24T00:48:53.997839+0000","last_peered":"2024-03-24T00:48:53.997839+0000","last_clean":"2024-03-24T00:48:53.997839+0000","last_became_active":"2024-03-24T00:48:53.992565+0000","last_became_peered":"2024-03-24T00:48:53.992565+0000","last_unstale":"2024-03-24T00:48:53.997839+0000","last_undegraded":"2024-03-24T00:48:53.997839+0000","last_fullsized":"2024-03-24T00:48:53.997839+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:58:19.481628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.978648+0000","last_change":"2024-03-24T00:48:53.978648+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.978648+0000","last_undegraded":"2024-03-24T00:48:53.978648+0000","last_fullsized":"2024-03-24T00:48:53.978648+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.994162+0000","last_change":"2024-03-24T00:48:53.988733+0000","last_active":"2024-03-24T00:48:53.994162+0000","last_peered":"2024-03-24T00:48:53.994162+0000","last_clean":"2024-03-24T00:48:53.994162+0000","last_became_active":"2024-03-24T00:48:53.988394+0000","last_became_peered":"2024-03-24T00:48:53.988394+0000","last_unstale":"2024-03-24T00:48:53.994162+0000","last_undegraded":"2024-03-24T00:48:53.994162+0000","last_fullsized":"2024-03-24T00:48:53.994162+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:14:52.551288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.981376+0000","last_change":"2024-03-24T00:48:53.981376+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.981376+0000","last_undegraded":"2024-03-24T00:48:53.981376+0000","last_fullsized":"2024-03-24T00:48:53.981376+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.997444+0000","last_change":"2024-03-24T00:48:53.989354+0000","last_active":"2024-03-24T00:48:53.997444+0000","last_peered":"2024-03-24T00:48:53.997444+0000","last_clean":"2024-03-24T00:48:53.997444+0000","last_became_active":"2024-03-24T00:48:53.989074+0000","last_became_peered":"2024-03-24T00:48:53.989074+0000","last_unstale":"2024-03-24T00:48:53.997444+0000","last_undegraded":"2024-03-24T00:48:53.997444+0000","last_fullsized":"2024-03-24T00:48:53.997444+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:15:35.204218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.984924+0000","last_change":"2024-03-24T00:48:53.984924+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.984924+0000","last_undegraded":"2024-03-24T00:48:53.984924+0000","last_fullsized":"2024-03-24T00:48:53.984924+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.997478+0000","last_change":"2024-03-24T00:48:53.992745+0000","last_active":"2024-03-24T00:48:53.997478+0000","last_peered":"2024-03-24T00:48:53.997478+0000","last_clean":"2024-03-24T00:48:53.997478+0000","last_became_active":"2024-03-24T00:48:53.992433+0000","last_became_peered":"2024-03-24T00:48:53.992433+0000","last_unstale":"2024-03-24T00:48:53.997478+0000","last_undegraded":"2024-03-24T00:48:53.997478+0000","last_fullsized":"2024-03-24T00:48:53.997478+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:07:15.456620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.979161+0000","last_change":"2024-03-24T00:48:53.979161+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.979161+0000","last_undegraded":"2024-03-24T00:48:53.979161+0000","last_fullsized":"2024-03-24T00:48:53.979161+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:54.000511+0000","last_change":"2024-03-24T00:48:53.997898+0000","last_active":"2024-03-24T00:48:54.000511+0000","last_peered":"2024-03-24T00:48:54.000511+0000","last_clean":"2024-03-24T00:48:54.000511+0000","last_became_active":"2024-03-24T00:48:53.997647+0000","last_became_peered":"2024-03-24T00:48:53.997647+0000","last_unstale":"2024-03-24T00:48:54.000511+0000","last_undegraded":"2024-03-24T00:48:54.000511+0000","last_fullsized":"2024-03-24T00:48:54.000511+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:26:28.796609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.984378+0000","last_change":"2024-03-24T00:48:53.984378+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.984378+0000","last_undegraded":"2024-03-24T00:48:53.984378+0000","last_fullsized":"2024-03-24T00:48:53.984378+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.996693+0000","last_change":"2024-03-24T00:48:53.989088+0000","last_active":"2024-03-24T00:48:53.996693+0000","last_peered":"2024-03-24T00:48:53.996693+0000","last_clean":"2024-03-24T00:48:53.996693+0000","last_became_active":"2024-03-24T00:48:53.988812+0000","last_became_peered":"2024-03-24T00:48:53.988812+0000","last_unstale":"2024-03-24T00:48:53.996693+0000","last_undegraded":"2024-03-24T00:48:53.996693+0000","last_fullsized":"2024-03-24T00:48:53.996693+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:46:58.027816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.983531+0000","last_change":"2024-03-24T00:48:53.983531+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.983531+0000","last_undegraded":"2024-03-24T00:48:53.983531+0000","last_fullsized":"2024-03-24T00:48:53.983531+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.997431+0000","last_change":"2024-03-24T00:48:53.992661+0000","last_active":"2024-03-24T00:48:53.997431+0000","last_peered":"2024-03-24T00:48:53.997431+0000","last_clean":"2024-03-24T00:48:53.997431+0000","last_became_active":"2024-03-24T00:48:53.992347+0000","last_became_peered":"2024-03-24T00:48:53.992347+0000","last_unstale":"2024-03-24T00:48:53.997431+0000","last_undegraded":"2024-03-24T00:48:53.997431+0000","last_fullsized":"2024-03-24T00:48:53.997431+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:21:21.714524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.982683+0000","last_change":"2024-03-24T00:48:53.982683+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.982683+0000","last_undegraded":"2024-03-24T00:48:53.982683+0000","last_fullsized":"2024-03-24T00:48:53.982683+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.997862+0000","last_change":"2024-03-24T00:48:53.995870+0000","last_active":"2024-03-24T00:48:53.997862+0000","last_peered":"2024-03-24T00:48:53.997862+0000","last_clean":"2024-03-24T00:48:53.997862+0000","last_became_active":"2024-03-24T00:48:53.995507+0000","last_became_peered":"2024-03-24T00:48:53.995507+0000","last_unstale":"2024-03-24T00:48:53.997862+0000","last_undegraded":"2024-03-24T00:48:53.997862+0000","last_fullsized":"2024-03-24T00:48:53.997862+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:07:44.083528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.978432+0000","last_change":"2024-03-24T00:48:53.978432+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.978432+0000","last_undegraded":"2024-03-24T00:48:53.978432+0000","last_fullsized":"2024-03-24T00:48:53.978432+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.997550+0000","last_change":"2024-03-24T00:48:53.993911+0000","last_active":"2024-03-24T00:48:53.997550+0000","last_peered":"2024-03-24T00:48:53.997550+0000","last_clean":"2024-03-24T00:48:53.997550+0000","last_became_active":"2024-03-24T00:48:53.993618+0000","last_became_peered":"2024-03-24T00:48:53.993618+0000","last_unstale":"2024-03-24T00:48:53.997550+0000","last_undegraded":"2024-03-24T00:48:53.997550+0000","last_fullsized":"2024-03-24T00:48:53.997550+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:37:28.523220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.983938+0000","last_change":"2024-03-24T00:48:53.983938+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.983938+0000","last_undegraded":"2024-03-24T00:48:53.983938+0000","last_fullsized":"2024-03-24T00:48:53.983938+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.998467+0000","last_change":"2024-03-24T00:48:53.994470+0000","last_active":"2024-03-24T00:48:53.998467+0000","last_peered":"2024-03-24T00:48:53.998467+0000","last_clean":"2024-03-24T00:48:53.998467+0000","last_became_active":"2024-03-24T00:48:53.994193+0000","last_became_peered":"2024-03-24T00:48:53.994193+0000","last_unstale":"2024-03-24T00:48:53.998467+0000","last_undegraded":"2024-03-24T00:48:53.998467+0000","last_fullsized":"2024-03-24T00:48:53.998467+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:30:58.413234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.986488+0000","last_change":"2024-03-24T00:48:53.986488+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.986488+0000","last_undegraded":"2024-03-24T00:48:53.986488+0000","last_fullsized":"2024-03-24T00:48:53.986488+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.997675+0000","last_change":"2024-03-24T00:48:53.995779+0000","last_active":"2024-03-24T00:48:53.997675+0000","last_peered":"2024-03-24T00:48:53.997675+0000","last_clean":"2024-03-24T00:48:53.997675+0000","last_became_active":"2024-03-24T00:48:53.995516+0000","last_became_peered":"2024-03-24T00:48:53.995516+0000","last_unstale":"2024-03-24T00:48:53.997675+0000","last_undegraded":"2024-03-24T00:48:53.997675+0000","last_fullsized":"2024-03-24T00:48:53.997675+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:55:30.565494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.982630+0000","last_change":"2024-03-24T00:48:53.982630+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.982630+0000","last_undegraded":"2024-03-24T00:48:53.982630+0000","last_fullsized":"2024-03-24T00:48:53.982630+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.997656+0000","last_change":"2024-03-24T00:48:53.995776+0000","last_active":"2024-03-24T00:48:53.997656+0000","last_peered":"2024-03-24T00:48:53.997656+0000","last_clean":"2024-03-24T00:48:53.997656+0000","last_became_active":"2024-03-24T00:48:53.995494+0000","last_became_peered":"2024-03-24T00:48:53.995494+0000","last_unstale":"2024-03-24T00:48:53.997656+0000","last_undegraded":"2024-03-24T00:48:53.997656+0000","last_fullsized":"2024-03-24T00:48:53.997656+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:16:17.295799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.977688+0000","last_change":"2024-03-24T00:48:53.977688+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.977688+0000","last_undegraded":"2024-03-24T00:48:53.977688+0000","last_fullsized":"2024-03-24T00:48:53.977688+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:54.000204+0000","last_change":"2024-03-24T00:48:53.996637+0000","last_active":"2024-03-24T00:48:54.000204+0000","last_peered":"2024-03-24T00:48:54.000204+0000","last_clean":"2024-03-24T00:48:54.000204+0000","last_became_active":"2024-03-24T00:48:53.996356+0000","last_became_peered":"2024-03-24T00:48:53.996356+0000","last_unstale":"2024-03-24T00:48:54.000204+0000","last_undegraded":"2024-03-24T00:48:54.000204+0000","last_fullsized":"2024-03-24T00:48:54.000204+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:13:07.121290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.983425+0000","last_change":"2024-03-24T00:48:53.983425+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.983425+0000","last_undegraded":"2024-03-24T00:48:53.983425+0000","last_fullsized":"2024-03-24T00:48:53.983425+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:54.003146+0000","last_change":"2024-03-24T00:48:53.998500+0000","last_active":"2024-03-24T00:48:54.003146+0000","last_peered":"2024-03-24T00:48:54.003146+0000","last_clean":"2024-03-24T00:48:54.003146+0000","last_became_active":"2024-03-24T00:48:53.998219+0000","last_became_peered":"2024-03-24T00:48:53.998219+0000","last_unstale":"2024-03-24T00:48:54.003146+0000","last_undegraded":"2024-03-24T00:48:54.003146+0000","last_fullsized":"2024-03-24T00:48:54.003146+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:35:18.392188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.983109+0000","last_change":"2024-03-24T00:48:53.983109+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.983109+0000","last_undegraded":"2024-03-24T00:48:53.983109+0000","last_fullsized":"2024-03-24T00:48:53.983109+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.997008+0000","last_change":"2024-03-24T00:48:53.992750+0000","last_active":"2024-03-24T00:48:53.997008+0000","last_peered":"2024-03-24T00:48:53.997008+0000","last_clean":"2024-03-24T00:48:53.997008+0000","last_became_active":"2024-03-24T00:48:53.992433+0000","last_became_peered":"2024-03-24T00:48:53.992433+0000","last_unstale":"2024-03-24T00:48:53.997008+0000","last_undegraded":"2024-03-24T00:48:53.997008+0000","last_fullsized":"2024-03-24T00:48:53.997008+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:55:30.565494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.981416+0000","last_change":"2024-03-24T00:48:53.981416+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.981416+0000","last_undegraded":"2024-03-24T00:48:53.981416+0000","last_fullsized":"2024-03-24T00:48:53.981416+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:54.000238+0000","last_change":"2024-03-24T00:48:53.997216+0000","last_active":"2024-03-24T00:48:54.000238+0000","last_peered":"2024-03-24T00:48:54.000238+0000","last_clean":"2024-03-24T00:48:54.000238+0000","last_became_active":"2024-03-24T00:48:53.996916+0000","last_became_peered":"2024-03-24T00:48:53.996916+0000","last_unstale":"2024-03-24T00:48:54.000238+0000","last_undegraded":"2024-03-24T00:48:54.000238+0000","last_fullsized":"2024-03-24T00:48:54.000238+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:54:53.066944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.985442+0000","last_change":"2024-03-24T00:48:53.985442+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.985442+0000","last_undegraded":"2024-03-24T00:48:53.985442+0000","last_fullsized":"2024-03-24T00:48:53.985442+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.997530+0000","last_change":"2024-03-24T00:48:53.993376+0000","last_active":"2024-03-24T00:48:53.997530+0000","last_peered":"2024-03-24T00:48:53.997530+0000","last_clean":"2024-03-24T00:48:53.997530+0000","last_became_active":"2024-03-24T00:48:53.993104+0000","last_became_peered":"2024-03-24T00:48:53.993104+0000","last_unstale":"2024-03-24T00:48:53.997530+0000","last_undegraded":"2024-03-24T00:48:53.997530+0000","last_fullsized":"2024-03-24T00:48:53.997530+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:32:33.042770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.981895+0000","last_change":"2024-03-24T00:48:53.981895+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.981895+0000","last_undegraded":"2024-03-24T00:48:53.981895+0000","last_fullsized":"2024-03-24T00:48:53.981895+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.993067+0000","last_change":"2024-03-24T00:48:53.988008+0000","last_active":"2024-03-24T00:48:53.993067+0000","last_peered":"2024-03-24T00:48:53.993067+0000","last_clean":"2024-03-24T00:48:53.993067+0000","last_became_active":"2024-03-24T00:48:53.987085+0000","last_became_peered":"2024-03-24T00:48:53.987085+0000","last_unstale":"2024-03-24T00:48:53.993067+0000","last_undegraded":"2024-03-24T00:48:53.993067+0000","last_fullsized":"2024-03-24T00:48:53.993067+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:26:28.796609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":354,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.985066+0000","last_change":"2024-03-24T00:38:40.953871+0000","last_active":"2024-03-24T00:48:53.985066+0000","last_peered":"2024-03-24T00:48:53.985066+0000","last_clean":"2024-03-24T00:48:53.985066+0000","last_became_active":"2024-03-24T00:38:40.952926+0000","last_became_peered":"2024-03-24T00:38:40.952926+0000","last_unstale":"2024-03-24T00:48:53.985066+0000","last_undegraded":"2024-03-24T00:48:53.985066+0000","last_fullsized":"2024-03-24T00:48:53.985066+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_clean_scrub_stamp":"2024-03-24T00:38:39.722194+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:05.397257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574976,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33184,"kb_used_data":2004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338656,"statfs":{"total":96636764160,"available":96602783744,"internally_reserved":0,"allocated":2052096,"data_stored":1269369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574975,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32604,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339236,"statfs":{"total":96636764160,"available":96603377664,"internally_reserved":0,"allocated":1458176,"data_stored":679001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574975,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32604,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339236,"statfs":{"total":96636764160,"available":96603377664,"internally_reserved":0,"allocated":1458176,"data_stored":679001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574974,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32620,"kb_used_data":1436,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339220,"statfs":{"total":96636764160,"available":96603361280,"internally_reserved":0,"allocated":1470464,"data_stored":685668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574974,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32604,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339236,"statfs":{"total":96636764160,"available":96603377664,"internally_reserved":0,"allocated":1458176,"data_stored":679001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574975,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32604,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339236,"statfs":{"total":96636764160,"available":96603377664,"internally_reserved":0,"allocated":1458176,"data_stored":679001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574975,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32620,"kb_used_data":1436,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339220,"statfs":{"total":96636764160,"available":96603361280,"internally_reserved":0,"allocated":1470464,"data_stored":685668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574974,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33196,"kb_used_data":2016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338644,"statfs":{"total":96636764160,"available":96602771456,"internally_reserved":0,"allocated":2064384,"data_stored":1276036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:48:58.000 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:48:58.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.140+0000 7fb161990640 1 Processor -- start 2024-03-24T00:48:58.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.140+0000 7fb161990640 1 -- start start 2024-03-24T00:48:58.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.140+0000 7fb161990640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb15c15dfb0 0x7fb15c0cfe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:58.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.140+0000 7fb161990640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb15c15e390 0x7fb15c06b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:58.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.140+0000 7fb161990640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb15c0d07d0 con 0x7fb15c0cded0 2024-03-24T00:48:58.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.140+0000 7fb161990640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb15c0d0ab0 con 0x7fb15c15dfb0 2024-03-24T00:48:58.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.140+0000 7fb161990640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb15c0d0d90 con 0x7fb15c15e390 2024-03-24T00:48:58.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.141+0000 7fb15affd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb15c15dfb0 0x7fb15c0cfe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:58.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.141+0000 7fb15b7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb15c0cded0 0x7fb15c161d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35926/0 (socket says 172.21.15.67:35926) 2024-03-24T00:48:58.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.141+0000 7fb15b7fe640 1 -- 172.21.15.67:0/3333024514 learned_addr learned my addr 172.21.15.67:0/3333024514 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:48:58.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.141+0000 7fb15a7fc640 1 --2- 172.21.15.67:0/3333024514 >> v2:172.21.15.72:3300/0 conn(0x7fb15c15e390 0x7fb15c06b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:58.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.141+0000 7fb15affd640 1 -- 172.21.15.67:0/3333024514 >> v2:172.21.15.72:3300/0 conn(0x7fb15c15e390 msgr2=0x7fb15c06b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:58.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.141+0000 7fb15affd640 1 --2- 172.21.15.67:0/3333024514 >> v2:172.21.15.72:3300/0 conn(0x7fb15c15e390 0x7fb15c06b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:58.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.141+0000 7fb15affd640 1 -- 172.21.15.67:0/3333024514 >> v1:172.21.15.67:6789/0 conn(0x7fb15c0cded0 legacy=0x7fb15c161d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:58.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.141+0000 7fb15affd640 1 -- 172.21.15.67:0/3333024514 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb15c0d1070 con 0x7fb15c15dfb0 2024-03-24T00:48:58.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.141+0000 7fb159ffb640 1 -- 172.21.15.67:0/3333024514 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2683723877 0 0) 0x7fb15c0d07d0 con 0x7fb15c0cded0 2024-03-24T00:48:58.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.142+0000 7fb15affd640 1 --2- 172.21.15.67:0/3333024514 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb15c15dfb0 0x7fb15c0cfe80 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7fb15c0b35f0 tx=0x7fb14c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e515518120e24f68 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:58.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.142+0000 7fb159ffb640 1 -- 172.21.15.67:0/3333024514 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb14c035020 con 0x7fb15c15dfb0 2024-03-24T00:48:58.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.142+0000 7fb159ffb640 1 -- 172.21.15.67:0/3333024514 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb14c031830 con 0x7fb15c15dfb0 2024-03-24T00:48:58.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.142+0000 7fb159ffb640 1 -- 172.21.15.67:0/3333024514 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb14c031c00 con 0x7fb15c15dfb0 2024-03-24T00:48:58.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.143+0000 7fb161990640 1 -- 172.21.15.67:0/3333024514 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb15c15dfb0 msgr2=0x7fb15c0cfe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:58.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.143+0000 7fb161990640 1 --2- 172.21.15.67:0/3333024514 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb15c15dfb0 0x7fb15c0cfe80 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7fb15c0b35f0 tx=0x7fb14c02b8d0 comp rx=0 tx=0).stop 2024-03-24T00:48:58.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.143+0000 7fb161990640 1 -- 172.21.15.67:0/3333024514 shutdown_connections 2024-03-24T00:48:58.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.143+0000 7fb161990640 1 --2- 172.21.15.67:0/3333024514 >> v2:172.21.15.72:3300/0 conn(0x7fb15c15e390 0x7fb15c06b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:58.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.143+0000 7fb161990640 1 --2- 172.21.15.67:0/3333024514 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb15c15dfb0 0x7fb15c0cfe80 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:58.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.143+0000 7fb161990640 1 -- 172.21.15.67:0/3333024514 >> 172.21.15.67:0/3333024514 conn(0x7fb15c099a30 msgr2=0x7fb15c099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:58.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.143+0000 7fb161990640 1 -- 172.21.15.67:0/3333024514 shutdown_connections 2024-03-24T00:48:58.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.143+0000 7fb161990640 1 -- 172.21.15.67:0/3333024514 wait complete. 2024-03-24T00:48:58.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.144+0000 7fb161990640 1 Processor -- start 2024-03-24T00:48:58.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.144+0000 7fb161990640 1 -- start start 2024-03-24T00:48:58.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.145+0000 7fb161990640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb15c0cded0 0x7fb15c156570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:58.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.145+0000 7fb161990640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb15c15dfb0 0x7fb15c0c67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:58.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.145+0000 7fb161990640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb15c15e390 0x7fb15c15b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:58.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.145+0000 7fb161990640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb15c0d1870 con 0x7fb15c15e390 2024-03-24T00:48:58.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.145+0000 7fb161990640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb15c0cc130 con 0x7fb15c15dfb0 2024-03-24T00:48:58.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.145+0000 7fb161990640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb15c0d03c0 con 0x7fb15c0cded0 2024-03-24T00:48:58.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.145+0000 7fb15b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb15c15e390 0x7fb15c15b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:58.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.145+0000 7fb15b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb15c15e390 0x7fb15c15b5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57456/0 (socket says 172.21.15.67:57456) 2024-03-24T00:48:58.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.145+0000 7fb15b7fe640 1 -- 172.21.15.67:0/539624208 learned_addr learned my addr 172.21.15.67:0/539624208 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:58.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.145+0000 7fb15a7fc640 1 --2- 172.21.15.67:0/539624208 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb15c15dfb0 0x7fb15c0c67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:58.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.145+0000 7fb15b7fe640 1 -- 172.21.15.67:0/539624208 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb15c0cded0 msgr2=0x7fb15c156570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:58.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.145+0000 7fb15affd640 1 --2- 172.21.15.67:0/539624208 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb15c0cded0 0x7fb15c156570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:58.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.145+0000 7fb15b7fe640 1 --2- 172.21.15.67:0/539624208 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb15c0cded0 0x7fb15c156570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:58.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.145+0000 7fb15b7fe640 1 -- 172.21.15.67:0/539624208 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb15c15dfb0 msgr2=0x7fb15c0c67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:58.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.145+0000 7fb15b7fe640 1 --2- 172.21.15.67:0/539624208 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb15c15dfb0 0x7fb15c0c67b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:58.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.145+0000 7fb15b7fe640 1 -- 172.21.15.67:0/539624208 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb15c0c7270 con 0x7fb15c15e390 2024-03-24T00:48:58.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.145+0000 7fb15b7fe640 1 --2- 172.21.15.67:0/539624208 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb15c15e390 0x7fb15c15b5e0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7fb15400c7a0 tx=0x7fb15400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:58.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.146+0000 7fb143fff640 1 -- 172.21.15.67:0/539624208 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb154018070 con 0x7fb15c15e390 2024-03-24T00:48:58.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.146+0000 7fb143fff640 1 -- 172.21.15.67:0/539624208 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb15400a640 con 0x7fb15c15e390 2024-03-24T00:48:58.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.146+0000 7fb161990640 1 -- 172.21.15.67:0/539624208 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb15c0c7550 con 0x7fb15c15e390 2024-03-24T00:48:58.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.146+0000 7fb143fff640 1 -- 172.21.15.67:0/539624208 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15400aab0 con 0x7fb15c15e390 2024-03-24T00:48:58.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.146+0000 7fb161990640 1 -- 172.21.15.67:0/539624208 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb15c15ee80 con 0x7fb15c15e390 2024-03-24T00:48:58.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.147+0000 7fb161990640 1 -- 172.21.15.67:0/539624208 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb15c15e990 con 0x7fb15c15e390 2024-03-24T00:48:58.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.147+0000 7fb143fff640 1 -- 172.21.15.67:0/539624208 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb15401b070 con 0x7fb15c15e390 2024-03-24T00:48:58.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.148+0000 7fb143fff640 1 --2- 172.21.15.67:0/539624208 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb14806bc00 0x7fb14806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:58.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.148+0000 7fb143fff640 1 -- 172.21.15.67:0/539624208 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb15408a220 con 0x7fb15c15e390 2024-03-24T00:48:58.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.148+0000 7fb15affd640 1 --2- 172.21.15.67:0/539624208 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb14806bc00 0x7fb14806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:58.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.148+0000 7fb15affd640 1 --2- 172.21.15.67:0/539624208 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb14806bc00 0x7fb14806e0c0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fb14c036000 tx=0x7fb14c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:58.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.152+0000 7fb143fff640 1 -- 172.21.15.67:0/539624208 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb154058b80 con 0x7fb15c15e390 2024-03-24T00:48:58.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.330+0000 7fb161990640 1 -- 172.21.15.67:0/539624208 --> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb15c04f400 con 0x7fb14806bc00 2024-03-24T00:48:58.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.337+0000 7fb143fff640 1 -- 172.21.15.67:0/539624208 <== mgr.4105 v2:172.21.15.72:6832/2768292274 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+247087 (secure 0 0 0) 0x7fb15c04f400 con 0x7fb14806bc00 2024-03-24T00:48:58.339 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:48:58.340 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-24T00:48:58.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.340+0000 7fb161990640 1 -- 172.21.15.67:0/539624208 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb14806bc00 msgr2=0x7fb14806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:58.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.340+0000 7fb161990640 1 --2- 172.21.15.67:0/539624208 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb14806bc00 0x7fb14806e0c0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fb14c036000 tx=0x7fb14c008030 comp rx=0 tx=0).stop 2024-03-24T00:48:58.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.340+0000 7fb161990640 1 -- 172.21.15.67:0/539624208 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb15c15e390 msgr2=0x7fb15c15b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:58.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.340+0000 7fb161990640 1 --2- 172.21.15.67:0/539624208 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb15c15e390 0x7fb15c15b5e0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7fb15400c7a0 tx=0x7fb15400cc60 comp rx=0 tx=0).stop 2024-03-24T00:48:58.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.340+0000 7fb161990640 1 -- 172.21.15.67:0/539624208 shutdown_connections 2024-03-24T00:48:58.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.340+0000 7fb161990640 1 --2- 172.21.15.67:0/539624208 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb14806bc00 0x7fb14806e0c0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:58.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.340+0000 7fb161990640 1 --2- 172.21.15.67:0/539624208 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb15c15e390 0x7fb15c15b5e0 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:58.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.340+0000 7fb161990640 1 --2- 172.21.15.67:0/539624208 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb15c15dfb0 0x7fb15c0c67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:58.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.340+0000 7fb161990640 1 --2- 172.21.15.67:0/539624208 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb15c0cded0 0x7fb15c156570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:58.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.340+0000 7fb161990640 1 -- 172.21.15.67:0/539624208 >> 172.21.15.67:0/539624208 conn(0x7fb15c099a30 msgr2=0x7fb15c06f560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:58.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.340+0000 7fb161990640 1 -- 172.21.15.67:0/539624208 shutdown_connections 2024-03-24T00:48:58.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.340+0000 7fb161990640 1 -- 172.21.15.67:0/539624208 wait complete. 2024-03-24T00:48:58.350 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":422,"stamp":"2024-03-24T00:48:57.468891+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":8,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":262036,"kb_used_data":12588,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754712684,"statfs":{"total":773094113280,"available":772825788416,"internally_reserved":0,"allocated":12890112,"data_stored":6632745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.496750"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.981887+0000","last_change":"2024-03-24T00:48:53.981887+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.981887+0000","last_undegraded":"2024-03-24T00:48:53.981887+0000","last_fullsized":"2024-03-24T00:48:53.981887+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.987478+0000","last_change":"2024-03-24T00:48:53.987478+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.987478+0000","last_undegraded":"2024-03-24T00:48:53.987478+0000","last_fullsized":"2024-03-24T00:48:53.987478+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.986072+0000","last_change":"2024-03-24T00:48:53.986072+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.986072+0000","last_undegraded":"2024-03-24T00:48:53.986072+0000","last_fullsized":"2024-03-24T00:48:53.986072+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.981339+0000","last_change":"2024-03-24T00:48:53.981339+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.981339+0000","last_undegraded":"2024-03-24T00:48:53.981339+0000","last_fullsized":"2024-03-24T00:48:53.981339+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.986981+0000","last_change":"2024-03-24T00:48:53.986981+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.986981+0000","last_undegraded":"2024-03-24T00:48:53.986981+0000","last_fullsized":"2024-03-24T00:48:53.986981+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.988748+0000","last_change":"2024-03-24T00:48:53.988748+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.988748+0000","last_undegraded":"2024-03-24T00:48:53.988748+0000","last_fullsized":"2024-03-24T00:48:53.988748+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.987478+0000","last_change":"2024-03-24T00:48:53.987478+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.987478+0000","last_undegraded":"2024-03-24T00:48:53.987478+0000","last_fullsized":"2024-03-24T00:48:53.987478+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.986010+0000","last_change":"2024-03-24T00:48:53.986010+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.986010+0000","last_undegraded":"2024-03-24T00:48:53.986010+0000","last_fullsized":"2024-03-24T00:48:53.986010+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.982242+0000","last_change":"2024-03-24T00:48:53.982242+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.982242+0000","last_undegraded":"2024-03-24T00:48:53.982242+0000","last_fullsized":"2024-03-24T00:48:53.982242+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.990365+0000","last_change":"2024-03-24T00:48:53.990365+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.990365+0000","last_undegraded":"2024-03-24T00:48:53.990365+0000","last_fullsized":"2024-03-24T00:48:53.990365+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.981850+0000","last_change":"2024-03-24T00:48:53.981850+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.981850+0000","last_undegraded":"2024-03-24T00:48:53.981850+0000","last_fullsized":"2024-03-24T00:48:53.981850+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.987024+0000","last_change":"2024-03-24T00:48:53.987024+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.987024+0000","last_undegraded":"2024-03-24T00:48:53.987024+0000","last_fullsized":"2024-03-24T00:48:53.987024+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.989677+0000","last_change":"2024-03-24T00:48:53.989677+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.989677+0000","last_undegraded":"2024-03-24T00:48:53.989677+0000","last_fullsized":"2024-03-24T00:48:53.989677+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.985341+0000","last_change":"2024-03-24T00:48:53.985341+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.985341+0000","last_undegraded":"2024-03-24T00:48:53.985341+0000","last_fullsized":"2024-03-24T00:48:53.985341+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.981660+0000","last_change":"2024-03-24T00:48:53.981660+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.981660+0000","last_undegraded":"2024-03-24T00:48:53.981660+0000","last_fullsized":"2024-03-24T00:48:53.981660+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.986575+0000","last_change":"2024-03-24T00:48:53.986575+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.986575+0000","last_undegraded":"2024-03-24T00:48:53.986575+0000","last_fullsized":"2024-03-24T00:48:53.986575+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.989175+0000","last_change":"2024-03-24T00:48:53.989175+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.989175+0000","last_undegraded":"2024-03-24T00:48:53.989175+0000","last_fullsized":"2024-03-24T00:48:53.989175+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.984128+0000","last_change":"2024-03-24T00:48:53.984128+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.984128+0000","last_undegraded":"2024-03-24T00:48:53.984128+0000","last_fullsized":"2024-03-24T00:48:53.984128+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.983103+0000","last_change":"2024-03-24T00:48:53.983103+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.983103+0000","last_undegraded":"2024-03-24T00:48:53.983103+0000","last_fullsized":"2024-03-24T00:48:53.983103+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.977625+0000","last_change":"2024-03-24T00:48:53.977625+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.977625+0000","last_undegraded":"2024-03-24T00:48:53.977625+0000","last_fullsized":"2024-03-24T00:48:53.977625+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.982297+0000","last_change":"2024-03-24T00:48:53.982297+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.982297+0000","last_undegraded":"2024-03-24T00:48:53.982297+0000","last_fullsized":"2024-03-24T00:48:53.982297+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.982445+0000","last_change":"2024-03-24T00:48:53.982445+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.982445+0000","last_undegraded":"2024-03-24T00:48:53.982445+0000","last_fullsized":"2024-03-24T00:48:53.982445+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.982875+0000","last_change":"2024-03-24T00:48:53.982875+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.982875+0000","last_undegraded":"2024-03-24T00:48:53.982875+0000","last_fullsized":"2024-03-24T00:48:53.982875+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.979109+0000","last_change":"2024-03-24T00:48:53.979109+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.979109+0000","last_undegraded":"2024-03-24T00:48:53.979109+0000","last_fullsized":"2024-03-24T00:48:53.979109+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.979993+0000","last_change":"2024-03-24T00:48:53.979993+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.979993+0000","last_undegraded":"2024-03-24T00:48:53.979993+0000","last_fullsized":"2024-03-24T00:48:53.979993+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.978637+0000","last_change":"2024-03-24T00:48:53.978637+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.978637+0000","last_undegraded":"2024-03-24T00:48:53.978637+0000","last_fullsized":"2024-03-24T00:48:53.978637+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.982655+0000","last_change":"2024-03-24T00:48:53.982655+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.982655+0000","last_undegraded":"2024-03-24T00:48:53.982655+0000","last_fullsized":"2024-03-24T00:48:53.982655+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.986026+0000","last_change":"2024-03-24T00:48:53.986026+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.986026+0000","last_undegraded":"2024-03-24T00:48:53.986026+0000","last_fullsized":"2024-03-24T00:48:53.986026+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.977622+0000","last_change":"2024-03-24T00:48:53.977622+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.977622+0000","last_undegraded":"2024-03-24T00:48:53.977622+0000","last_fullsized":"2024-03-24T00:48:53.977622+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.984581+0000","last_change":"2024-03-24T00:48:53.984581+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.984581+0000","last_undegraded":"2024-03-24T00:48:53.984581+0000","last_fullsized":"2024-03-24T00:48:53.984581+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.979193+0000","last_change":"2024-03-24T00:48:53.979193+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.979193+0000","last_undegraded":"2024-03-24T00:48:53.979193+0000","last_fullsized":"2024-03-24T00:48:53.979193+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.978546+0000","last_change":"2024-03-24T00:48:53.978546+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.978546+0000","last_undegraded":"2024-03-24T00:48:53.978546+0000","last_fullsized":"2024-03-24T00:48:53.978546+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.990974+0000","last_change":"2024-03-24T00:48:53.990974+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.990974+0000","last_undegraded":"2024-03-24T00:48:53.990974+0000","last_fullsized":"2024-03-24T00:48:53.990974+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.997599+0000","last_change":"2024-03-24T00:48:53.992874+0000","last_active":"2024-03-24T00:48:53.997599+0000","last_peered":"2024-03-24T00:48:53.997599+0000","last_clean":"2024-03-24T00:48:53.997599+0000","last_became_active":"2024-03-24T00:48:53.992507+0000","last_became_peered":"2024-03-24T00:48:53.992507+0000","last_unstale":"2024-03-24T00:48:53.997599+0000","last_undegraded":"2024-03-24T00:48:53.997599+0000","last_fullsized":"2024-03-24T00:48:53.997599+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:17:05.808342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.985412+0000","last_change":"2024-03-24T00:48:53.985412+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.985412+0000","last_undegraded":"2024-03-24T00:48:53.985412+0000","last_fullsized":"2024-03-24T00:48:53.985412+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.984444+0000","last_change":"2024-03-24T00:48:53.982887+0000","last_active":"2024-03-24T00:48:53.984444+0000","last_peered":"2024-03-24T00:48:53.984444+0000","last_clean":"2024-03-24T00:48:53.984444+0000","last_became_active":"2024-03-24T00:48:53.982515+0000","last_became_peered":"2024-03-24T00:48:53.982515+0000","last_unstale":"2024-03-24T00:48:53.984444+0000","last_undegraded":"2024-03-24T00:48:53.984444+0000","last_fullsized":"2024-03-24T00:48:53.984444+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:31:25.866307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.978888+0000","last_change":"2024-03-24T00:48:53.978888+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.978888+0000","last_undegraded":"2024-03-24T00:48:53.978888+0000","last_fullsized":"2024-03-24T00:48:53.978888+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.997498+0000","last_change":"2024-03-24T00:48:53.993536+0000","last_active":"2024-03-24T00:48:53.997498+0000","last_peered":"2024-03-24T00:48:53.997498+0000","last_clean":"2024-03-24T00:48:53.997498+0000","last_became_active":"2024-03-24T00:48:53.993136+0000","last_became_peered":"2024-03-24T00:48:53.993136+0000","last_unstale":"2024-03-24T00:48:53.997498+0000","last_undegraded":"2024-03-24T00:48:53.997498+0000","last_fullsized":"2024-03-24T00:48:53.997498+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:26:39.618413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.981809+0000","last_change":"2024-03-24T00:48:53.981809+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.981809+0000","last_undegraded":"2024-03-24T00:48:53.981809+0000","last_fullsized":"2024-03-24T00:48:53.981809+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.996455+0000","last_change":"2024-03-24T00:48:53.993401+0000","last_active":"2024-03-24T00:48:53.996455+0000","last_peered":"2024-03-24T00:48:53.996455+0000","last_clean":"2024-03-24T00:48:53.996455+0000","last_became_active":"2024-03-24T00:48:53.991401+0000","last_became_peered":"2024-03-24T00:48:53.991401+0000","last_unstale":"2024-03-24T00:48:53.996455+0000","last_undegraded":"2024-03-24T00:48:53.996455+0000","last_fullsized":"2024-03-24T00:48:53.996455+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:32.954517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.986877+0000","last_change":"2024-03-24T00:48:53.986877+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.986877+0000","last_undegraded":"2024-03-24T00:48:53.986877+0000","last_fullsized":"2024-03-24T00:48:53.986877+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.997557+0000","last_change":"2024-03-24T00:48:53.994950+0000","last_active":"2024-03-24T00:48:53.997557+0000","last_peered":"2024-03-24T00:48:53.997557+0000","last_clean":"2024-03-24T00:48:53.997557+0000","last_became_active":"2024-03-24T00:48:53.994709+0000","last_became_peered":"2024-03-24T00:48:53.994709+0000","last_unstale":"2024-03-24T00:48:53.997557+0000","last_undegraded":"2024-03-24T00:48:53.997557+0000","last_fullsized":"2024-03-24T00:48:53.997557+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:45:58.215004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.982871+0000","last_change":"2024-03-24T00:48:53.982871+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.982871+0000","last_undegraded":"2024-03-24T00:48:53.982871+0000","last_fullsized":"2024-03-24T00:48:53.982871+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:54.000260+0000","last_change":"2024-03-24T00:48:53.997359+0000","last_active":"2024-03-24T00:48:54.000260+0000","last_peered":"2024-03-24T00:48:54.000260+0000","last_clean":"2024-03-24T00:48:54.000260+0000","last_became_active":"2024-03-24T00:48:53.997039+0000","last_became_peered":"2024-03-24T00:48:53.997039+0000","last_unstale":"2024-03-24T00:48:54.000260+0000","last_undegraded":"2024-03-24T00:48:54.000260+0000","last_fullsized":"2024-03-24T00:48:54.000260+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:03:39.280823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.984118+0000","last_change":"2024-03-24T00:48:53.984118+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.984118+0000","last_undegraded":"2024-03-24T00:48:53.984118+0000","last_fullsized":"2024-03-24T00:48:53.984118+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.992054+0000","last_change":"2024-03-24T00:48:53.988155+0000","last_active":"2024-03-24T00:48:53.992054+0000","last_peered":"2024-03-24T00:48:53.992054+0000","last_clean":"2024-03-24T00:48:53.992054+0000","last_became_active":"2024-03-24T00:48:53.987659+0000","last_became_peered":"2024-03-24T00:48:53.987659+0000","last_unstale":"2024-03-24T00:48:53.992054+0000","last_undegraded":"2024-03-24T00:48:53.992054+0000","last_fullsized":"2024-03-24T00:48:53.992054+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:32:33.042770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.987465+0000","last_change":"2024-03-24T00:48:53.987465+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.987465+0000","last_undegraded":"2024-03-24T00:48:53.987465+0000","last_fullsized":"2024-03-24T00:48:53.987465+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.998570+0000","last_change":"2024-03-24T00:48:53.993610+0000","last_active":"2024-03-24T00:48:53.998570+0000","last_peered":"2024-03-24T00:48:53.998570+0000","last_clean":"2024-03-24T00:48:53.998570+0000","last_became_active":"2024-03-24T00:48:53.993326+0000","last_became_peered":"2024-03-24T00:48:53.993326+0000","last_unstale":"2024-03-24T00:48:53.998570+0000","last_undegraded":"2024-03-24T00:48:53.998570+0000","last_fullsized":"2024-03-24T00:48:53.998570+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:20:08.465917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.983527+0000","last_change":"2024-03-24T00:48:53.983527+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.983527+0000","last_undegraded":"2024-03-24T00:48:53.983527+0000","last_fullsized":"2024-03-24T00:48:53.983527+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:54.003266+0000","last_change":"2024-03-24T00:48:53.999330+0000","last_active":"2024-03-24T00:48:54.003266+0000","last_peered":"2024-03-24T00:48:54.003266+0000","last_clean":"2024-03-24T00:48:54.003266+0000","last_became_active":"2024-03-24T00:48:53.999065+0000","last_became_peered":"2024-03-24T00:48:53.999065+0000","last_unstale":"2024-03-24T00:48:54.003266+0000","last_undegraded":"2024-03-24T00:48:54.003266+0000","last_fullsized":"2024-03-24T00:48:54.003266+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:45:58.215004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.978769+0000","last_change":"2024-03-24T00:48:53.978769+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.978769+0000","last_undegraded":"2024-03-24T00:48:53.978769+0000","last_fullsized":"2024-03-24T00:48:53.978769+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.991791+0000","last_change":"2024-03-24T00:48:53.985480+0000","last_active":"2024-03-24T00:48:53.991791+0000","last_peered":"2024-03-24T00:48:53.991791+0000","last_clean":"2024-03-24T00:48:53.991791+0000","last_became_active":"2024-03-24T00:48:53.985203+0000","last_became_peered":"2024-03-24T00:48:53.985203+0000","last_unstale":"2024-03-24T00:48:53.991791+0000","last_undegraded":"2024-03-24T00:48:53.991791+0000","last_fullsized":"2024-03-24T00:48:53.991791+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:21:21.714524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.983568+0000","last_change":"2024-03-24T00:48:53.983568+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.983568+0000","last_undegraded":"2024-03-24T00:48:53.983568+0000","last_fullsized":"2024-03-24T00:48:53.983568+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.992006+0000","last_change":"2024-03-24T00:48:53.987256+0000","last_active":"2024-03-24T00:48:53.992006+0000","last_peered":"2024-03-24T00:48:53.992006+0000","last_clean":"2024-03-24T00:48:53.992006+0000","last_became_active":"2024-03-24T00:48:53.986435+0000","last_became_peered":"2024-03-24T00:48:53.986435+0000","last_unstale":"2024-03-24T00:48:53.992006+0000","last_undegraded":"2024-03-24T00:48:53.992006+0000","last_fullsized":"2024-03-24T00:48:53.992006+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:07:15.456620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.979598+0000","last_change":"2024-03-24T00:48:53.979598+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.979598+0000","last_undegraded":"2024-03-24T00:48:53.979598+0000","last_fullsized":"2024-03-24T00:48:53.979598+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.997830+0000","last_change":"2024-03-24T00:48:53.992589+0000","last_active":"2024-03-24T00:48:53.997830+0000","last_peered":"2024-03-24T00:48:53.997830+0000","last_clean":"2024-03-24T00:48:53.997830+0000","last_became_active":"2024-03-24T00:48:53.992273+0000","last_became_peered":"2024-03-24T00:48:53.992273+0000","last_unstale":"2024-03-24T00:48:53.997830+0000","last_undegraded":"2024-03-24T00:48:53.997830+0000","last_fullsized":"2024-03-24T00:48:53.997830+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:01:04.094765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.986841+0000","last_change":"2024-03-24T00:48:53.986841+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.986841+0000","last_undegraded":"2024-03-24T00:48:53.986841+0000","last_fullsized":"2024-03-24T00:48:53.986841+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.992329+0000","last_change":"2024-03-24T00:48:53.988410+0000","last_active":"2024-03-24T00:48:53.992329+0000","last_peered":"2024-03-24T00:48:53.992329+0000","last_clean":"2024-03-24T00:48:53.992329+0000","last_became_active":"2024-03-24T00:48:53.987901+0000","last_became_peered":"2024-03-24T00:48:53.987901+0000","last_unstale":"2024-03-24T00:48:53.992329+0000","last_undegraded":"2024-03-24T00:48:53.992329+0000","last_fullsized":"2024-03-24T00:48:53.992329+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:10:25.529558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.986869+0000","last_change":"2024-03-24T00:48:53.986869+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.986869+0000","last_undegraded":"2024-03-24T00:48:53.986869+0000","last_fullsized":"2024-03-24T00:48:53.986869+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.992260+0000","last_change":"2024-03-24T00:48:53.988076+0000","last_active":"2024-03-24T00:48:53.992260+0000","last_peered":"2024-03-24T00:48:53.992260+0000","last_clean":"2024-03-24T00:48:53.992260+0000","last_became_active":"2024-03-24T00:48:53.987752+0000","last_became_peered":"2024-03-24T00:48:53.987752+0000","last_unstale":"2024-03-24T00:48:53.992260+0000","last_undegraded":"2024-03-24T00:48:53.992260+0000","last_fullsized":"2024-03-24T00:48:53.992260+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:24:44.940524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.977688+0000","last_change":"2024-03-24T00:48:53.977688+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.977688+0000","last_undegraded":"2024-03-24T00:48:53.977688+0000","last_fullsized":"2024-03-24T00:48:53.977688+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.997839+0000","last_change":"2024-03-24T00:48:53.992865+0000","last_active":"2024-03-24T00:48:53.997839+0000","last_peered":"2024-03-24T00:48:53.997839+0000","last_clean":"2024-03-24T00:48:53.997839+0000","last_became_active":"2024-03-24T00:48:53.992565+0000","last_became_peered":"2024-03-24T00:48:53.992565+0000","last_unstale":"2024-03-24T00:48:53.997839+0000","last_undegraded":"2024-03-24T00:48:53.997839+0000","last_fullsized":"2024-03-24T00:48:53.997839+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:58:19.481628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.978648+0000","last_change":"2024-03-24T00:48:53.978648+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.978648+0000","last_undegraded":"2024-03-24T00:48:53.978648+0000","last_fullsized":"2024-03-24T00:48:53.978648+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.994162+0000","last_change":"2024-03-24T00:48:53.988733+0000","last_active":"2024-03-24T00:48:53.994162+0000","last_peered":"2024-03-24T00:48:53.994162+0000","last_clean":"2024-03-24T00:48:53.994162+0000","last_became_active":"2024-03-24T00:48:53.988394+0000","last_became_peered":"2024-03-24T00:48:53.988394+0000","last_unstale":"2024-03-24T00:48:53.994162+0000","last_undegraded":"2024-03-24T00:48:53.994162+0000","last_fullsized":"2024-03-24T00:48:53.994162+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:14:52.551288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.981376+0000","last_change":"2024-03-24T00:48:53.981376+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.981376+0000","last_undegraded":"2024-03-24T00:48:53.981376+0000","last_fullsized":"2024-03-24T00:48:53.981376+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.997444+0000","last_change":"2024-03-24T00:48:53.989354+0000","last_active":"2024-03-24T00:48:53.997444+0000","last_peered":"2024-03-24T00:48:53.997444+0000","last_clean":"2024-03-24T00:48:53.997444+0000","last_became_active":"2024-03-24T00:48:53.989074+0000","last_became_peered":"2024-03-24T00:48:53.989074+0000","last_unstale":"2024-03-24T00:48:53.997444+0000","last_undegraded":"2024-03-24T00:48:53.997444+0000","last_fullsized":"2024-03-24T00:48:53.997444+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:15:35.204218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.984924+0000","last_change":"2024-03-24T00:48:53.984924+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.984924+0000","last_undegraded":"2024-03-24T00:48:53.984924+0000","last_fullsized":"2024-03-24T00:48:53.984924+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.997478+0000","last_change":"2024-03-24T00:48:53.992745+0000","last_active":"2024-03-24T00:48:53.997478+0000","last_peered":"2024-03-24T00:48:53.997478+0000","last_clean":"2024-03-24T00:48:53.997478+0000","last_became_active":"2024-03-24T00:48:53.992433+0000","last_became_peered":"2024-03-24T00:48:53.992433+0000","last_unstale":"2024-03-24T00:48:53.997478+0000","last_undegraded":"2024-03-24T00:48:53.997478+0000","last_fullsized":"2024-03-24T00:48:53.997478+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:07:15.456620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.979161+0000","last_change":"2024-03-24T00:48:53.979161+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.979161+0000","last_undegraded":"2024-03-24T00:48:53.979161+0000","last_fullsized":"2024-03-24T00:48:53.979161+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:54.000511+0000","last_change":"2024-03-24T00:48:53.997898+0000","last_active":"2024-03-24T00:48:54.000511+0000","last_peered":"2024-03-24T00:48:54.000511+0000","last_clean":"2024-03-24T00:48:54.000511+0000","last_became_active":"2024-03-24T00:48:53.997647+0000","last_became_peered":"2024-03-24T00:48:53.997647+0000","last_unstale":"2024-03-24T00:48:54.000511+0000","last_undegraded":"2024-03-24T00:48:54.000511+0000","last_fullsized":"2024-03-24T00:48:54.000511+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:26:28.796609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.984378+0000","last_change":"2024-03-24T00:48:53.984378+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.984378+0000","last_undegraded":"2024-03-24T00:48:53.984378+0000","last_fullsized":"2024-03-24T00:48:53.984378+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.996693+0000","last_change":"2024-03-24T00:48:53.989088+0000","last_active":"2024-03-24T00:48:53.996693+0000","last_peered":"2024-03-24T00:48:53.996693+0000","last_clean":"2024-03-24T00:48:53.996693+0000","last_became_active":"2024-03-24T00:48:53.988812+0000","last_became_peered":"2024-03-24T00:48:53.988812+0000","last_unstale":"2024-03-24T00:48:53.996693+0000","last_undegraded":"2024-03-24T00:48:53.996693+0000","last_fullsized":"2024-03-24T00:48:53.996693+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:46:58.027816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.983531+0000","last_change":"2024-03-24T00:48:53.983531+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.983531+0000","last_undegraded":"2024-03-24T00:48:53.983531+0000","last_fullsized":"2024-03-24T00:48:53.983531+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.997431+0000","last_change":"2024-03-24T00:48:53.992661+0000","last_active":"2024-03-24T00:48:53.997431+0000","last_peered":"2024-03-24T00:48:53.997431+0000","last_clean":"2024-03-24T00:48:53.997431+0000","last_became_active":"2024-03-24T00:48:53.992347+0000","last_became_peered":"2024-03-24T00:48:53.992347+0000","last_unstale":"2024-03-24T00:48:53.997431+0000","last_undegraded":"2024-03-24T00:48:53.997431+0000","last_fullsized":"2024-03-24T00:48:53.997431+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:21:21.714524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.982683+0000","last_change":"2024-03-24T00:48:53.982683+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.982683+0000","last_undegraded":"2024-03-24T00:48:53.982683+0000","last_fullsized":"2024-03-24T00:48:53.982683+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.997862+0000","last_change":"2024-03-24T00:48:53.995870+0000","last_active":"2024-03-24T00:48:53.997862+0000","last_peered":"2024-03-24T00:48:53.997862+0000","last_clean":"2024-03-24T00:48:53.997862+0000","last_became_active":"2024-03-24T00:48:53.995507+0000","last_became_peered":"2024-03-24T00:48:53.995507+0000","last_unstale":"2024-03-24T00:48:53.997862+0000","last_undegraded":"2024-03-24T00:48:53.997862+0000","last_fullsized":"2024-03-24T00:48:53.997862+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:07:44.083528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.978432+0000","last_change":"2024-03-24T00:48:53.978432+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.978432+0000","last_undegraded":"2024-03-24T00:48:53.978432+0000","last_fullsized":"2024-03-24T00:48:53.978432+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.997550+0000","last_change":"2024-03-24T00:48:53.993911+0000","last_active":"2024-03-24T00:48:53.997550+0000","last_peered":"2024-03-24T00:48:53.997550+0000","last_clean":"2024-03-24T00:48:53.997550+0000","last_became_active":"2024-03-24T00:48:53.993618+0000","last_became_peered":"2024-03-24T00:48:53.993618+0000","last_unstale":"2024-03-24T00:48:53.997550+0000","last_undegraded":"2024-03-24T00:48:53.997550+0000","last_fullsized":"2024-03-24T00:48:53.997550+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:37:28.523220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.983938+0000","last_change":"2024-03-24T00:48:53.983938+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.983938+0000","last_undegraded":"2024-03-24T00:48:53.983938+0000","last_fullsized":"2024-03-24T00:48:53.983938+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.998467+0000","last_change":"2024-03-24T00:48:53.994470+0000","last_active":"2024-03-24T00:48:53.998467+0000","last_peered":"2024-03-24T00:48:53.998467+0000","last_clean":"2024-03-24T00:48:53.998467+0000","last_became_active":"2024-03-24T00:48:53.994193+0000","last_became_peered":"2024-03-24T00:48:53.994193+0000","last_unstale":"2024-03-24T00:48:53.998467+0000","last_undegraded":"2024-03-24T00:48:53.998467+0000","last_fullsized":"2024-03-24T00:48:53.998467+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:30:58.413234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.986488+0000","last_change":"2024-03-24T00:48:53.986488+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.986488+0000","last_undegraded":"2024-03-24T00:48:53.986488+0000","last_fullsized":"2024-03-24T00:48:53.986488+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.997675+0000","last_change":"2024-03-24T00:48:53.995779+0000","last_active":"2024-03-24T00:48:53.997675+0000","last_peered":"2024-03-24T00:48:53.997675+0000","last_clean":"2024-03-24T00:48:53.997675+0000","last_became_active":"2024-03-24T00:48:53.995516+0000","last_became_peered":"2024-03-24T00:48:53.995516+0000","last_unstale":"2024-03-24T00:48:53.997675+0000","last_undegraded":"2024-03-24T00:48:53.997675+0000","last_fullsized":"2024-03-24T00:48:53.997675+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:55:30.565494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.982630+0000","last_change":"2024-03-24T00:48:53.982630+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.982630+0000","last_undegraded":"2024-03-24T00:48:53.982630+0000","last_fullsized":"2024-03-24T00:48:53.982630+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.997656+0000","last_change":"2024-03-24T00:48:53.995776+0000","last_active":"2024-03-24T00:48:53.997656+0000","last_peered":"2024-03-24T00:48:53.997656+0000","last_clean":"2024-03-24T00:48:53.997656+0000","last_became_active":"2024-03-24T00:48:53.995494+0000","last_became_peered":"2024-03-24T00:48:53.995494+0000","last_unstale":"2024-03-24T00:48:53.997656+0000","last_undegraded":"2024-03-24T00:48:53.997656+0000","last_fullsized":"2024-03-24T00:48:53.997656+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:16:17.295799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.977688+0000","last_change":"2024-03-24T00:48:53.977688+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.977688+0000","last_undegraded":"2024-03-24T00:48:53.977688+0000","last_fullsized":"2024-03-24T00:48:53.977688+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:54.000204+0000","last_change":"2024-03-24T00:48:53.996637+0000","last_active":"2024-03-24T00:48:54.000204+0000","last_peered":"2024-03-24T00:48:54.000204+0000","last_clean":"2024-03-24T00:48:54.000204+0000","last_became_active":"2024-03-24T00:48:53.996356+0000","last_became_peered":"2024-03-24T00:48:53.996356+0000","last_unstale":"2024-03-24T00:48:54.000204+0000","last_undegraded":"2024-03-24T00:48:54.000204+0000","last_fullsized":"2024-03-24T00:48:54.000204+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:13:07.121290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.983425+0000","last_change":"2024-03-24T00:48:53.983425+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.983425+0000","last_undegraded":"2024-03-24T00:48:53.983425+0000","last_fullsized":"2024-03-24T00:48:53.983425+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:54.003146+0000","last_change":"2024-03-24T00:48:53.998500+0000","last_active":"2024-03-24T00:48:54.003146+0000","last_peered":"2024-03-24T00:48:54.003146+0000","last_clean":"2024-03-24T00:48:54.003146+0000","last_became_active":"2024-03-24T00:48:53.998219+0000","last_became_peered":"2024-03-24T00:48:53.998219+0000","last_unstale":"2024-03-24T00:48:54.003146+0000","last_undegraded":"2024-03-24T00:48:54.003146+0000","last_fullsized":"2024-03-24T00:48:54.003146+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:35:18.392188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.983109+0000","last_change":"2024-03-24T00:48:53.983109+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.983109+0000","last_undegraded":"2024-03-24T00:48:53.983109+0000","last_fullsized":"2024-03-24T00:48:53.983109+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.997008+0000","last_change":"2024-03-24T00:48:53.992750+0000","last_active":"2024-03-24T00:48:53.997008+0000","last_peered":"2024-03-24T00:48:53.997008+0000","last_clean":"2024-03-24T00:48:53.997008+0000","last_became_active":"2024-03-24T00:48:53.992433+0000","last_became_peered":"2024-03-24T00:48:53.992433+0000","last_unstale":"2024-03-24T00:48:53.997008+0000","last_undegraded":"2024-03-24T00:48:53.997008+0000","last_fullsized":"2024-03-24T00:48:53.997008+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:55:30.565494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.981416+0000","last_change":"2024-03-24T00:48:53.981416+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.981416+0000","last_undegraded":"2024-03-24T00:48:53.981416+0000","last_fullsized":"2024-03-24T00:48:53.981416+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:54.000238+0000","last_change":"2024-03-24T00:48:53.997216+0000","last_active":"2024-03-24T00:48:54.000238+0000","last_peered":"2024-03-24T00:48:54.000238+0000","last_clean":"2024-03-24T00:48:54.000238+0000","last_became_active":"2024-03-24T00:48:53.996916+0000","last_became_peered":"2024-03-24T00:48:53.996916+0000","last_unstale":"2024-03-24T00:48:54.000238+0000","last_undegraded":"2024-03-24T00:48:54.000238+0000","last_fullsized":"2024-03-24T00:48:54.000238+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:54:53.066944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.985442+0000","last_change":"2024-03-24T00:48:53.985442+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.985442+0000","last_undegraded":"2024-03-24T00:48:53.985442+0000","last_fullsized":"2024-03-24T00:48:53.985442+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.997530+0000","last_change":"2024-03-24T00:48:53.993376+0000","last_active":"2024-03-24T00:48:53.997530+0000","last_peered":"2024-03-24T00:48:53.997530+0000","last_clean":"2024-03-24T00:48:53.997530+0000","last_became_active":"2024-03-24T00:48:53.993104+0000","last_became_peered":"2024-03-24T00:48:53.993104+0000","last_unstale":"2024-03-24T00:48:53.997530+0000","last_undegraded":"2024-03-24T00:48:53.997530+0000","last_fullsized":"2024-03-24T00:48:53.997530+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:32:33.042770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.981895+0000","last_change":"2024-03-24T00:48:53.981895+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.981895+0000","last_undegraded":"2024-03-24T00:48:53.981895+0000","last_fullsized":"2024-03-24T00:48:53.981895+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.993067+0000","last_change":"2024-03-24T00:48:53.988008+0000","last_active":"2024-03-24T00:48:53.993067+0000","last_peered":"2024-03-24T00:48:53.993067+0000","last_clean":"2024-03-24T00:48:53.993067+0000","last_became_active":"2024-03-24T00:48:53.987085+0000","last_became_peered":"2024-03-24T00:48:53.987085+0000","last_unstale":"2024-03-24T00:48:53.993067+0000","last_undegraded":"2024-03-24T00:48:53.993067+0000","last_fullsized":"2024-03-24T00:48:53.993067+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:26:28.796609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":354,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.985066+0000","last_change":"2024-03-24T00:38:40.953871+0000","last_active":"2024-03-24T00:48:53.985066+0000","last_peered":"2024-03-24T00:48:53.985066+0000","last_clean":"2024-03-24T00:48:53.985066+0000","last_became_active":"2024-03-24T00:38:40.952926+0000","last_became_peered":"2024-03-24T00:38:40.952926+0000","last_unstale":"2024-03-24T00:48:53.985066+0000","last_undegraded":"2024-03-24T00:48:53.985066+0000","last_fullsized":"2024-03-24T00:48:53.985066+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_clean_scrub_stamp":"2024-03-24T00:38:39.722194+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:05.397257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574976,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33184,"kb_used_data":2004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338656,"statfs":{"total":96636764160,"available":96602783744,"internally_reserved":0,"allocated":2052096,"data_stored":1269369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574975,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32604,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339236,"statfs":{"total":96636764160,"available":96603377664,"internally_reserved":0,"allocated":1458176,"data_stored":679001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574975,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32604,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339236,"statfs":{"total":96636764160,"available":96603377664,"internally_reserved":0,"allocated":1458176,"data_stored":679001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574974,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32620,"kb_used_data":1436,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339220,"statfs":{"total":96636764160,"available":96603361280,"internally_reserved":0,"allocated":1470464,"data_stored":685668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574974,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32604,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339236,"statfs":{"total":96636764160,"available":96603377664,"internally_reserved":0,"allocated":1458176,"data_stored":679001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574975,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32604,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339236,"statfs":{"total":96636764160,"available":96603377664,"internally_reserved":0,"allocated":1458176,"data_stored":679001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574975,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32620,"kb_used_data":1436,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339220,"statfs":{"total":96636764160,"available":96603361280,"internally_reserved":0,"allocated":1470464,"data_stored":685668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574974,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33196,"kb_used_data":2016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338644,"statfs":{"total":96636764160,"available":96602771456,"internally_reserved":0,"allocated":2064384,"data_stored":1276036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:48:58.354 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:48:58.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.492+0000 7f4cd31f3640 1 Processor -- start 2024-03-24T00:48:58.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.492+0000 7f4cd31f3640 1 -- start start 2024-03-24T00:48:58.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.493+0000 7f4cd31f3640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4ccc15bf70 0x7f4ccc0cc690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:58.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.493+0000 7f4cd31f3640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4ccc0cfc70 0x7f4ccc0ce220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:58.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.493+0000 7f4cd31f3640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ccc0ccfe0 con 0x7f4ccc0ce760 2024-03-24T00:48:58.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.493+0000 7f4cd31f3640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ccc0cd2c0 con 0x7f4ccc0cfc70 2024-03-24T00:48:58.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.493+0000 7f4cd31f3640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ccc0cd5a0 con 0x7f4ccc15bf70 2024-03-24T00:48:58.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.493+0000 7f4ccbfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4ccc0cfc70 0x7f4ccc0ce220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:58.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.493+0000 7f4ccbfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4ccc0cfc70 0x7f4ccc0ce220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57470/0 (socket says 172.21.15.67:57470) 2024-03-24T00:48:58.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.493+0000 7f4ccbfff640 1 -- 172.21.15.67:0/2155150513 learned_addr learned my addr 172.21.15.67:0/2155150513 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:58.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.494+0000 7f4cd0f8f640 1 --2- 172.21.15.67:0/2155150513 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4ccc15bf70 0x7f4ccc0cc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:58.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.494+0000 7f4ccbfff640 1 -- 172.21.15.67:0/2155150513 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4ccc15bf70 msgr2=0x7f4ccc0cc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:58.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.494+0000 7f4ccbfff640 1 --2- 172.21.15.67:0/2155150513 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4ccc15bf70 0x7f4ccc0cc690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:58.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.494+0000 7f4ccbfff640 1 -- 172.21.15.67:0/2155150513 >> v1:172.21.15.72:6789/0 conn(0x7f4ccc0ce760 legacy=0x7f4ccc08b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:58.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.494+0000 7f4ccbfff640 1 -- 172.21.15.67:0/2155150513 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ccc0cd880 con 0x7f4ccc0cfc70 2024-03-24T00:48:58.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.494+0000 7f4ccbfff640 1 --2- 172.21.15.67:0/2155150513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ccc0cfc70 0x7f4ccc0ce220 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f4cb8009870 tx=0x7f4cb802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95e5924cce3c168f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:58.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.495+0000 7f4ccb7fe640 1 -- 172.21.15.67:0/2155150513 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb8034070 con 0x7f4ccc0cfc70 2024-03-24T00:48:58.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.495+0000 7f4ccb7fe640 1 -- 172.21.15.67:0/2155150513 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cb8030840 con 0x7f4ccc0cfc70 2024-03-24T00:48:58.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.495+0000 7f4ccb7fe640 1 -- 172.21.15.67:0/2155150513 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb8030c10 con 0x7f4ccc0cfc70 2024-03-24T00:48:58.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.495+0000 7f4cd31f3640 1 -- 172.21.15.67:0/2155150513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ccc0cfc70 msgr2=0x7f4ccc0ce220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:58.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.495+0000 7f4cd31f3640 1 --2- 172.21.15.67:0/2155150513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ccc0cfc70 0x7f4ccc0ce220 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f4cb8009870 tx=0x7f4cb802b980 comp rx=0 tx=0).stop 2024-03-24T00:48:58.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.495+0000 7f4cd31f3640 1 -- 172.21.15.67:0/2155150513 shutdown_connections 2024-03-24T00:48:58.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.495+0000 7f4cd31f3640 1 --2- 172.21.15.67:0/2155150513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ccc0cfc70 0x7f4ccc0ce220 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:58.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.495+0000 7f4cd31f3640 1 --2- 172.21.15.67:0/2155150513 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4ccc15bf70 0x7f4ccc0cc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:58.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.495+0000 7f4cd31f3640 1 -- 172.21.15.67:0/2155150513 >> 172.21.15.67:0/2155150513 conn(0x7f4ccc0d1bf0 msgr2=0x7f4ccc0d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:58.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.495+0000 7f4cd31f3640 1 -- 172.21.15.67:0/2155150513 shutdown_connections 2024-03-24T00:48:58.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.495+0000 7f4cd31f3640 1 -- 172.21.15.67:0/2155150513 wait complete. 2024-03-24T00:48:58.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.497+0000 7f4cd31f3640 1 Processor -- start 2024-03-24T00:48:58.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.497+0000 7f4cd31f3640 1 -- start start 2024-03-24T00:48:58.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.498+0000 7f4cd31f3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ccc0ce760 0x7f4ccc1561f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:58.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.498+0000 7f4cd31f3640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4ccc0cfc70 0x7f4ccc1f4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:58.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.498+0000 7f4cd31f3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ccc15bf70 0x7f4ccc070860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:58.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.498+0000 7f4cd31f3640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ccc0ccd50 con 0x7f4ccc15bf70 2024-03-24T00:48:58.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.498+0000 7f4cd31f3640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ccc0ccbd0 con 0x7f4ccc0ce760 2024-03-24T00:48:58.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.498+0000 7f4cd31f3640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ccc0b2ce0 con 0x7f4ccc0cfc70 2024-03-24T00:48:58.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.498+0000 7f4cd1790640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ccc15bf70 0x7f4ccc070860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:58.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.499+0000 7f4cd1790640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ccc15bf70 0x7f4ccc070860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57478/0 (socket says 172.21.15.67:57478) 2024-03-24T00:48:58.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.499+0000 7f4cd1790640 1 -- 172.21.15.67:0/1483389179 learned_addr learned my addr 172.21.15.67:0/1483389179 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:58.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.499+0000 7f4ccbfff640 1 --2- 172.21.15.67:0/1483389179 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4ccc0cfc70 0x7f4ccc1f4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:58.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.499+0000 7f4cd0f8f640 1 --2- 172.21.15.67:0/1483389179 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ccc0ce760 0x7f4ccc1561f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:58.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.499+0000 7f4cd1790640 1 -- 172.21.15.67:0/1483389179 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4ccc0cfc70 msgr2=0x7f4ccc1f4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:58.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.499+0000 7f4cd1790640 1 --2- 172.21.15.67:0/1483389179 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4ccc0cfc70 0x7f4ccc1f4ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:58.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.499+0000 7f4cd1790640 1 -- 172.21.15.67:0/1483389179 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ccc0ce760 msgr2=0x7f4ccc1561f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:58.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.499+0000 7f4cd1790640 1 --2- 172.21.15.67:0/1483389179 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ccc0ce760 0x7f4ccc1561f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:58.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.499+0000 7f4cd1790640 1 -- 172.21.15.67:0/1483389179 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ccc1f5e70 con 0x7f4ccc15bf70 2024-03-24T00:48:58.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.499+0000 7f4cd1790640 1 --2- 172.21.15.67:0/1483389179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ccc15bf70 0x7f4ccc070860 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f4cc400c6b0 tx=0x7f4cc400cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:58.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.499+0000 7f4cc97fa640 1 -- 172.21.15.67:0/1483389179 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc4003b70 con 0x7f4ccc15bf70 2024-03-24T00:48:58.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.499+0000 7f4cc97fa640 1 -- 172.21.15.67:0/1483389179 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cc4003d30 con 0x7f4ccc15bf70 2024-03-24T00:48:58.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.499+0000 7f4cd31f3640 1 -- 172.21.15.67:0/1483389179 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ccc156730 con 0x7f4ccc15bf70 2024-03-24T00:48:58.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.499+0000 7f4cc97fa640 1 -- 172.21.15.67:0/1483389179 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc4005310 con 0x7f4ccc15bf70 2024-03-24T00:48:58.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.499+0000 7f4cd31f3640 1 -- 172.21.15.67:0/1483389179 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ccc1f5b90 con 0x7f4ccc15bf70 2024-03-24T00:48:58.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.501+0000 7f4cd31f3640 1 -- 172.21.15.67:0/1483389179 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ccc04f400 con 0x7f4ccc15bf70 2024-03-24T00:48:58.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.501+0000 7f4cc97fa640 1 -- 172.21.15.67:0/1483389179 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4cc40054b0 con 0x7f4ccc15bf70 2024-03-24T00:48:58.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.501+0000 7f4cc97fa640 1 --2- 172.21.15.67:0/1483389179 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4cb006bc00 0x7f4cb006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:58.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.501+0000 7f4cc97fa640 1 -- 172.21.15.67:0/1483389179 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4cc408a890 con 0x7f4ccc15bf70 2024-03-24T00:48:58.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.501+0000 7f4cd0f8f640 1 --2- 172.21.15.67:0/1483389179 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4cb006bc00 0x7f4cb006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:58.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.502+0000 7f4cd0f8f640 1 --2- 172.21.15.67:0/1483389179 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4cb006bc00 0x7f4cb006e0c0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f4cc0006fd0 tx=0x7f4cc0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:58.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.505+0000 7f4cc97fa640 1 -- 172.21.15.67:0/1483389179 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4cc4058c30 con 0x7f4ccc15bf70 2024-03-24T00:48:58.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.684+0000 7f4cd31f3640 1 -- 172.21.15.67:0/1483389179 --> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4ccc0cc690 con 0x7f4cb006bc00 2024-03-24T00:48:58.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.690+0000 7f4cc97fa640 1 -- 172.21.15.67:0/1483389179 <== mgr.4105 v2:172.21.15.72:6832/2768292274 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+247087 (secure 0 0 0) 0x7f4ccc0cc690 con 0x7f4cb006bc00 2024-03-24T00:48:58.693 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:48:58.694 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-24T00:48:58.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.693+0000 7f4cd31f3640 1 -- 172.21.15.67:0/1483389179 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4cb006bc00 msgr2=0x7f4cb006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:58.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.693+0000 7f4cd31f3640 1 --2- 172.21.15.67:0/1483389179 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4cb006bc00 0x7f4cb006e0c0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f4cc0006fd0 tx=0x7f4cc0008040 comp rx=0 tx=0).stop 2024-03-24T00:48:58.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.693+0000 7f4cd31f3640 1 -- 172.21.15.67:0/1483389179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ccc15bf70 msgr2=0x7f4ccc070860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:58.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.693+0000 7f4cd31f3640 1 --2- 172.21.15.67:0/1483389179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ccc15bf70 0x7f4ccc070860 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f4cc400c6b0 tx=0x7f4cc400cb70 comp rx=0 tx=0).stop 2024-03-24T00:48:58.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.693+0000 7f4cd31f3640 1 -- 172.21.15.67:0/1483389179 shutdown_connections 2024-03-24T00:48:58.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.693+0000 7f4cd31f3640 1 --2- 172.21.15.67:0/1483389179 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4cb006bc00 0x7f4cb006e0c0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:58.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.693+0000 7f4cd31f3640 1 --2- 172.21.15.67:0/1483389179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ccc15bf70 0x7f4ccc070860 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:58.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.693+0000 7f4cd31f3640 1 --2- 172.21.15.67:0/1483389179 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4ccc0cfc70 0x7f4ccc1f4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:58.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.693+0000 7f4cd31f3640 1 --2- 172.21.15.67:0/1483389179 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ccc0ce760 0x7f4ccc1561f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:58.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.693+0000 7f4cd31f3640 1 -- 172.21.15.67:0/1483389179 >> 172.21.15.67:0/1483389179 conn(0x7f4ccc0d1bf0 msgr2=0x7f4ccc0b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:58.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.693+0000 7f4cd31f3640 1 -- 172.21.15.67:0/1483389179 shutdown_connections 2024-03-24T00:48:58.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.693+0000 7f4cd31f3640 1 -- 172.21.15.67:0/1483389179 wait complete. 2024-03-24T00:48:58.704 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":422,"stamp":"2024-03-24T00:48:57.468891+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":8,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":262036,"kb_used_data":12588,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754712684,"statfs":{"total":773094113280,"available":772825788416,"internally_reserved":0,"allocated":12890112,"data_stored":6632745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.496750"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.981887+0000","last_change":"2024-03-24T00:48:53.981887+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.981887+0000","last_undegraded":"2024-03-24T00:48:53.981887+0000","last_fullsized":"2024-03-24T00:48:53.981887+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.987478+0000","last_change":"2024-03-24T00:48:53.987478+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.987478+0000","last_undegraded":"2024-03-24T00:48:53.987478+0000","last_fullsized":"2024-03-24T00:48:53.987478+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.986072+0000","last_change":"2024-03-24T00:48:53.986072+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.986072+0000","last_undegraded":"2024-03-24T00:48:53.986072+0000","last_fullsized":"2024-03-24T00:48:53.986072+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.981339+0000","last_change":"2024-03-24T00:48:53.981339+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.981339+0000","last_undegraded":"2024-03-24T00:48:53.981339+0000","last_fullsized":"2024-03-24T00:48:53.981339+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.986981+0000","last_change":"2024-03-24T00:48:53.986981+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.986981+0000","last_undegraded":"2024-03-24T00:48:53.986981+0000","last_fullsized":"2024-03-24T00:48:53.986981+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.988748+0000","last_change":"2024-03-24T00:48:53.988748+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.988748+0000","last_undegraded":"2024-03-24T00:48:53.988748+0000","last_fullsized":"2024-03-24T00:48:53.988748+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.987478+0000","last_change":"2024-03-24T00:48:53.987478+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.987478+0000","last_undegraded":"2024-03-24T00:48:53.987478+0000","last_fullsized":"2024-03-24T00:48:53.987478+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.986010+0000","last_change":"2024-03-24T00:48:53.986010+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.986010+0000","last_undegraded":"2024-03-24T00:48:53.986010+0000","last_fullsized":"2024-03-24T00:48:53.986010+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.982242+0000","last_change":"2024-03-24T00:48:53.982242+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.982242+0000","last_undegraded":"2024-03-24T00:48:53.982242+0000","last_fullsized":"2024-03-24T00:48:53.982242+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.990365+0000","last_change":"2024-03-24T00:48:53.990365+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.990365+0000","last_undegraded":"2024-03-24T00:48:53.990365+0000","last_fullsized":"2024-03-24T00:48:53.990365+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.981850+0000","last_change":"2024-03-24T00:48:53.981850+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.981850+0000","last_undegraded":"2024-03-24T00:48:53.981850+0000","last_fullsized":"2024-03-24T00:48:53.981850+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.987024+0000","last_change":"2024-03-24T00:48:53.987024+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.987024+0000","last_undegraded":"2024-03-24T00:48:53.987024+0000","last_fullsized":"2024-03-24T00:48:53.987024+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.989677+0000","last_change":"2024-03-24T00:48:53.989677+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.989677+0000","last_undegraded":"2024-03-24T00:48:53.989677+0000","last_fullsized":"2024-03-24T00:48:53.989677+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.985341+0000","last_change":"2024-03-24T00:48:53.985341+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.985341+0000","last_undegraded":"2024-03-24T00:48:53.985341+0000","last_fullsized":"2024-03-24T00:48:53.985341+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.981660+0000","last_change":"2024-03-24T00:48:53.981660+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.981660+0000","last_undegraded":"2024-03-24T00:48:53.981660+0000","last_fullsized":"2024-03-24T00:48:53.981660+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.986575+0000","last_change":"2024-03-24T00:48:53.986575+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.986575+0000","last_undegraded":"2024-03-24T00:48:53.986575+0000","last_fullsized":"2024-03-24T00:48:53.986575+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.989175+0000","last_change":"2024-03-24T00:48:53.989175+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.989175+0000","last_undegraded":"2024-03-24T00:48:53.989175+0000","last_fullsized":"2024-03-24T00:48:53.989175+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.984128+0000","last_change":"2024-03-24T00:48:53.984128+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.984128+0000","last_undegraded":"2024-03-24T00:48:53.984128+0000","last_fullsized":"2024-03-24T00:48:53.984128+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.983103+0000","last_change":"2024-03-24T00:48:53.983103+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.983103+0000","last_undegraded":"2024-03-24T00:48:53.983103+0000","last_fullsized":"2024-03-24T00:48:53.983103+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.977625+0000","last_change":"2024-03-24T00:48:53.977625+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.977625+0000","last_undegraded":"2024-03-24T00:48:53.977625+0000","last_fullsized":"2024-03-24T00:48:53.977625+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.982297+0000","last_change":"2024-03-24T00:48:53.982297+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.982297+0000","last_undegraded":"2024-03-24T00:48:53.982297+0000","last_fullsized":"2024-03-24T00:48:53.982297+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.982445+0000","last_change":"2024-03-24T00:48:53.982445+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.982445+0000","last_undegraded":"2024-03-24T00:48:53.982445+0000","last_fullsized":"2024-03-24T00:48:53.982445+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.982875+0000","last_change":"2024-03-24T00:48:53.982875+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.982875+0000","last_undegraded":"2024-03-24T00:48:53.982875+0000","last_fullsized":"2024-03-24T00:48:53.982875+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.979109+0000","last_change":"2024-03-24T00:48:53.979109+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.979109+0000","last_undegraded":"2024-03-24T00:48:53.979109+0000","last_fullsized":"2024-03-24T00:48:53.979109+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.979993+0000","last_change":"2024-03-24T00:48:53.979993+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.979993+0000","last_undegraded":"2024-03-24T00:48:53.979993+0000","last_fullsized":"2024-03-24T00:48:53.979993+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.978637+0000","last_change":"2024-03-24T00:48:53.978637+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.978637+0000","last_undegraded":"2024-03-24T00:48:53.978637+0000","last_fullsized":"2024-03-24T00:48:53.978637+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.982655+0000","last_change":"2024-03-24T00:48:53.982655+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.982655+0000","last_undegraded":"2024-03-24T00:48:53.982655+0000","last_fullsized":"2024-03-24T00:48:53.982655+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.986026+0000","last_change":"2024-03-24T00:48:53.986026+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.986026+0000","last_undegraded":"2024-03-24T00:48:53.986026+0000","last_fullsized":"2024-03-24T00:48:53.986026+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.977622+0000","last_change":"2024-03-24T00:48:53.977622+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.977622+0000","last_undegraded":"2024-03-24T00:48:53.977622+0000","last_fullsized":"2024-03-24T00:48:53.977622+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.984581+0000","last_change":"2024-03-24T00:48:53.984581+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.984581+0000","last_undegraded":"2024-03-24T00:48:53.984581+0000","last_fullsized":"2024-03-24T00:48:53.984581+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.979193+0000","last_change":"2024-03-24T00:48:53.979193+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.979193+0000","last_undegraded":"2024-03-24T00:48:53.979193+0000","last_fullsized":"2024-03-24T00:48:53.979193+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.978546+0000","last_change":"2024-03-24T00:48:53.978546+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.978546+0000","last_undegraded":"2024-03-24T00:48:53.978546+0000","last_fullsized":"2024-03-24T00:48:53.978546+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.990974+0000","last_change":"2024-03-24T00:48:53.990974+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.990974+0000","last_undegraded":"2024-03-24T00:48:53.990974+0000","last_fullsized":"2024-03-24T00:48:53.990974+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.997599+0000","last_change":"2024-03-24T00:48:53.992874+0000","last_active":"2024-03-24T00:48:53.997599+0000","last_peered":"2024-03-24T00:48:53.997599+0000","last_clean":"2024-03-24T00:48:53.997599+0000","last_became_active":"2024-03-24T00:48:53.992507+0000","last_became_peered":"2024-03-24T00:48:53.992507+0000","last_unstale":"2024-03-24T00:48:53.997599+0000","last_undegraded":"2024-03-24T00:48:53.997599+0000","last_fullsized":"2024-03-24T00:48:53.997599+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:17:05.808342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.985412+0000","last_change":"2024-03-24T00:48:53.985412+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.985412+0000","last_undegraded":"2024-03-24T00:48:53.985412+0000","last_fullsized":"2024-03-24T00:48:53.985412+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.984444+0000","last_change":"2024-03-24T00:48:53.982887+0000","last_active":"2024-03-24T00:48:53.984444+0000","last_peered":"2024-03-24T00:48:53.984444+0000","last_clean":"2024-03-24T00:48:53.984444+0000","last_became_active":"2024-03-24T00:48:53.982515+0000","last_became_peered":"2024-03-24T00:48:53.982515+0000","last_unstale":"2024-03-24T00:48:53.984444+0000","last_undegraded":"2024-03-24T00:48:53.984444+0000","last_fullsized":"2024-03-24T00:48:53.984444+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:31:25.866307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.978888+0000","last_change":"2024-03-24T00:48:53.978888+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.978888+0000","last_undegraded":"2024-03-24T00:48:53.978888+0000","last_fullsized":"2024-03-24T00:48:53.978888+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.997498+0000","last_change":"2024-03-24T00:48:53.993536+0000","last_active":"2024-03-24T00:48:53.997498+0000","last_peered":"2024-03-24T00:48:53.997498+0000","last_clean":"2024-03-24T00:48:53.997498+0000","last_became_active":"2024-03-24T00:48:53.993136+0000","last_became_peered":"2024-03-24T00:48:53.993136+0000","last_unstale":"2024-03-24T00:48:53.997498+0000","last_undegraded":"2024-03-24T00:48:53.997498+0000","last_fullsized":"2024-03-24T00:48:53.997498+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:26:39.618413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.981809+0000","last_change":"2024-03-24T00:48:53.981809+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.981809+0000","last_undegraded":"2024-03-24T00:48:53.981809+0000","last_fullsized":"2024-03-24T00:48:53.981809+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.996455+0000","last_change":"2024-03-24T00:48:53.993401+0000","last_active":"2024-03-24T00:48:53.996455+0000","last_peered":"2024-03-24T00:48:53.996455+0000","last_clean":"2024-03-24T00:48:53.996455+0000","last_became_active":"2024-03-24T00:48:53.991401+0000","last_became_peered":"2024-03-24T00:48:53.991401+0000","last_unstale":"2024-03-24T00:48:53.996455+0000","last_undegraded":"2024-03-24T00:48:53.996455+0000","last_fullsized":"2024-03-24T00:48:53.996455+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:32.954517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.986877+0000","last_change":"2024-03-24T00:48:53.986877+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.986877+0000","last_undegraded":"2024-03-24T00:48:53.986877+0000","last_fullsized":"2024-03-24T00:48:53.986877+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.997557+0000","last_change":"2024-03-24T00:48:53.994950+0000","last_active":"2024-03-24T00:48:53.997557+0000","last_peered":"2024-03-24T00:48:53.997557+0000","last_clean":"2024-03-24T00:48:53.997557+0000","last_became_active":"2024-03-24T00:48:53.994709+0000","last_became_peered":"2024-03-24T00:48:53.994709+0000","last_unstale":"2024-03-24T00:48:53.997557+0000","last_undegraded":"2024-03-24T00:48:53.997557+0000","last_fullsized":"2024-03-24T00:48:53.997557+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:45:58.215004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.982871+0000","last_change":"2024-03-24T00:48:53.982871+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.982871+0000","last_undegraded":"2024-03-24T00:48:53.982871+0000","last_fullsized":"2024-03-24T00:48:53.982871+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:54.000260+0000","last_change":"2024-03-24T00:48:53.997359+0000","last_active":"2024-03-24T00:48:54.000260+0000","last_peered":"2024-03-24T00:48:54.000260+0000","last_clean":"2024-03-24T00:48:54.000260+0000","last_became_active":"2024-03-24T00:48:53.997039+0000","last_became_peered":"2024-03-24T00:48:53.997039+0000","last_unstale":"2024-03-24T00:48:54.000260+0000","last_undegraded":"2024-03-24T00:48:54.000260+0000","last_fullsized":"2024-03-24T00:48:54.000260+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:03:39.280823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.984118+0000","last_change":"2024-03-24T00:48:53.984118+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.984118+0000","last_undegraded":"2024-03-24T00:48:53.984118+0000","last_fullsized":"2024-03-24T00:48:53.984118+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.992054+0000","last_change":"2024-03-24T00:48:53.988155+0000","last_active":"2024-03-24T00:48:53.992054+0000","last_peered":"2024-03-24T00:48:53.992054+0000","last_clean":"2024-03-24T00:48:53.992054+0000","last_became_active":"2024-03-24T00:48:53.987659+0000","last_became_peered":"2024-03-24T00:48:53.987659+0000","last_unstale":"2024-03-24T00:48:53.992054+0000","last_undegraded":"2024-03-24T00:48:53.992054+0000","last_fullsized":"2024-03-24T00:48:53.992054+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:32:33.042770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.987465+0000","last_change":"2024-03-24T00:48:53.987465+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.987465+0000","last_undegraded":"2024-03-24T00:48:53.987465+0000","last_fullsized":"2024-03-24T00:48:53.987465+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.998570+0000","last_change":"2024-03-24T00:48:53.993610+0000","last_active":"2024-03-24T00:48:53.998570+0000","last_peered":"2024-03-24T00:48:53.998570+0000","last_clean":"2024-03-24T00:48:53.998570+0000","last_became_active":"2024-03-24T00:48:53.993326+0000","last_became_peered":"2024-03-24T00:48:53.993326+0000","last_unstale":"2024-03-24T00:48:53.998570+0000","last_undegraded":"2024-03-24T00:48:53.998570+0000","last_fullsized":"2024-03-24T00:48:53.998570+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:20:08.465917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.983527+0000","last_change":"2024-03-24T00:48:53.983527+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.983527+0000","last_undegraded":"2024-03-24T00:48:53.983527+0000","last_fullsized":"2024-03-24T00:48:53.983527+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:54.003266+0000","last_change":"2024-03-24T00:48:53.999330+0000","last_active":"2024-03-24T00:48:54.003266+0000","last_peered":"2024-03-24T00:48:54.003266+0000","last_clean":"2024-03-24T00:48:54.003266+0000","last_became_active":"2024-03-24T00:48:53.999065+0000","last_became_peered":"2024-03-24T00:48:53.999065+0000","last_unstale":"2024-03-24T00:48:54.003266+0000","last_undegraded":"2024-03-24T00:48:54.003266+0000","last_fullsized":"2024-03-24T00:48:54.003266+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:45:58.215004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.978769+0000","last_change":"2024-03-24T00:48:53.978769+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.978769+0000","last_undegraded":"2024-03-24T00:48:53.978769+0000","last_fullsized":"2024-03-24T00:48:53.978769+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.991791+0000","last_change":"2024-03-24T00:48:53.985480+0000","last_active":"2024-03-24T00:48:53.991791+0000","last_peered":"2024-03-24T00:48:53.991791+0000","last_clean":"2024-03-24T00:48:53.991791+0000","last_became_active":"2024-03-24T00:48:53.985203+0000","last_became_peered":"2024-03-24T00:48:53.985203+0000","last_unstale":"2024-03-24T00:48:53.991791+0000","last_undegraded":"2024-03-24T00:48:53.991791+0000","last_fullsized":"2024-03-24T00:48:53.991791+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:21:21.714524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.983568+0000","last_change":"2024-03-24T00:48:53.983568+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.983568+0000","last_undegraded":"2024-03-24T00:48:53.983568+0000","last_fullsized":"2024-03-24T00:48:53.983568+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.992006+0000","last_change":"2024-03-24T00:48:53.987256+0000","last_active":"2024-03-24T00:48:53.992006+0000","last_peered":"2024-03-24T00:48:53.992006+0000","last_clean":"2024-03-24T00:48:53.992006+0000","last_became_active":"2024-03-24T00:48:53.986435+0000","last_became_peered":"2024-03-24T00:48:53.986435+0000","last_unstale":"2024-03-24T00:48:53.992006+0000","last_undegraded":"2024-03-24T00:48:53.992006+0000","last_fullsized":"2024-03-24T00:48:53.992006+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:07:15.456620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.979598+0000","last_change":"2024-03-24T00:48:53.979598+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.979598+0000","last_undegraded":"2024-03-24T00:48:53.979598+0000","last_fullsized":"2024-03-24T00:48:53.979598+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.997830+0000","last_change":"2024-03-24T00:48:53.992589+0000","last_active":"2024-03-24T00:48:53.997830+0000","last_peered":"2024-03-24T00:48:53.997830+0000","last_clean":"2024-03-24T00:48:53.997830+0000","last_became_active":"2024-03-24T00:48:53.992273+0000","last_became_peered":"2024-03-24T00:48:53.992273+0000","last_unstale":"2024-03-24T00:48:53.997830+0000","last_undegraded":"2024-03-24T00:48:53.997830+0000","last_fullsized":"2024-03-24T00:48:53.997830+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:01:04.094765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.986841+0000","last_change":"2024-03-24T00:48:53.986841+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.986841+0000","last_undegraded":"2024-03-24T00:48:53.986841+0000","last_fullsized":"2024-03-24T00:48:53.986841+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.992329+0000","last_change":"2024-03-24T00:48:53.988410+0000","last_active":"2024-03-24T00:48:53.992329+0000","last_peered":"2024-03-24T00:48:53.992329+0000","last_clean":"2024-03-24T00:48:53.992329+0000","last_became_active":"2024-03-24T00:48:53.987901+0000","last_became_peered":"2024-03-24T00:48:53.987901+0000","last_unstale":"2024-03-24T00:48:53.992329+0000","last_undegraded":"2024-03-24T00:48:53.992329+0000","last_fullsized":"2024-03-24T00:48:53.992329+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:10:25.529558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.986869+0000","last_change":"2024-03-24T00:48:53.986869+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.986869+0000","last_undegraded":"2024-03-24T00:48:53.986869+0000","last_fullsized":"2024-03-24T00:48:53.986869+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.992260+0000","last_change":"2024-03-24T00:48:53.988076+0000","last_active":"2024-03-24T00:48:53.992260+0000","last_peered":"2024-03-24T00:48:53.992260+0000","last_clean":"2024-03-24T00:48:53.992260+0000","last_became_active":"2024-03-24T00:48:53.987752+0000","last_became_peered":"2024-03-24T00:48:53.987752+0000","last_unstale":"2024-03-24T00:48:53.992260+0000","last_undegraded":"2024-03-24T00:48:53.992260+0000","last_fullsized":"2024-03-24T00:48:53.992260+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:24:44.940524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.977688+0000","last_change":"2024-03-24T00:48:53.977688+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.977688+0000","last_undegraded":"2024-03-24T00:48:53.977688+0000","last_fullsized":"2024-03-24T00:48:53.977688+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.997839+0000","last_change":"2024-03-24T00:48:53.992865+0000","last_active":"2024-03-24T00:48:53.997839+0000","last_peered":"2024-03-24T00:48:53.997839+0000","last_clean":"2024-03-24T00:48:53.997839+0000","last_became_active":"2024-03-24T00:48:53.992565+0000","last_became_peered":"2024-03-24T00:48:53.992565+0000","last_unstale":"2024-03-24T00:48:53.997839+0000","last_undegraded":"2024-03-24T00:48:53.997839+0000","last_fullsized":"2024-03-24T00:48:53.997839+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:58:19.481628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.978648+0000","last_change":"2024-03-24T00:48:53.978648+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.978648+0000","last_undegraded":"2024-03-24T00:48:53.978648+0000","last_fullsized":"2024-03-24T00:48:53.978648+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.994162+0000","last_change":"2024-03-24T00:48:53.988733+0000","last_active":"2024-03-24T00:48:53.994162+0000","last_peered":"2024-03-24T00:48:53.994162+0000","last_clean":"2024-03-24T00:48:53.994162+0000","last_became_active":"2024-03-24T00:48:53.988394+0000","last_became_peered":"2024-03-24T00:48:53.988394+0000","last_unstale":"2024-03-24T00:48:53.994162+0000","last_undegraded":"2024-03-24T00:48:53.994162+0000","last_fullsized":"2024-03-24T00:48:53.994162+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:14:52.551288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.981376+0000","last_change":"2024-03-24T00:48:53.981376+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.981376+0000","last_undegraded":"2024-03-24T00:48:53.981376+0000","last_fullsized":"2024-03-24T00:48:53.981376+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.997444+0000","last_change":"2024-03-24T00:48:53.989354+0000","last_active":"2024-03-24T00:48:53.997444+0000","last_peered":"2024-03-24T00:48:53.997444+0000","last_clean":"2024-03-24T00:48:53.997444+0000","last_became_active":"2024-03-24T00:48:53.989074+0000","last_became_peered":"2024-03-24T00:48:53.989074+0000","last_unstale":"2024-03-24T00:48:53.997444+0000","last_undegraded":"2024-03-24T00:48:53.997444+0000","last_fullsized":"2024-03-24T00:48:53.997444+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:15:35.204218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.984924+0000","last_change":"2024-03-24T00:48:53.984924+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.984924+0000","last_undegraded":"2024-03-24T00:48:53.984924+0000","last_fullsized":"2024-03-24T00:48:53.984924+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.997478+0000","last_change":"2024-03-24T00:48:53.992745+0000","last_active":"2024-03-24T00:48:53.997478+0000","last_peered":"2024-03-24T00:48:53.997478+0000","last_clean":"2024-03-24T00:48:53.997478+0000","last_became_active":"2024-03-24T00:48:53.992433+0000","last_became_peered":"2024-03-24T00:48:53.992433+0000","last_unstale":"2024-03-24T00:48:53.997478+0000","last_undegraded":"2024-03-24T00:48:53.997478+0000","last_fullsized":"2024-03-24T00:48:53.997478+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:07:15.456620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.979161+0000","last_change":"2024-03-24T00:48:53.979161+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.979161+0000","last_undegraded":"2024-03-24T00:48:53.979161+0000","last_fullsized":"2024-03-24T00:48:53.979161+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:54.000511+0000","last_change":"2024-03-24T00:48:53.997898+0000","last_active":"2024-03-24T00:48:54.000511+0000","last_peered":"2024-03-24T00:48:54.000511+0000","last_clean":"2024-03-24T00:48:54.000511+0000","last_became_active":"2024-03-24T00:48:53.997647+0000","last_became_peered":"2024-03-24T00:48:53.997647+0000","last_unstale":"2024-03-24T00:48:54.000511+0000","last_undegraded":"2024-03-24T00:48:54.000511+0000","last_fullsized":"2024-03-24T00:48:54.000511+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:26:28.796609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.984378+0000","last_change":"2024-03-24T00:48:53.984378+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.984378+0000","last_undegraded":"2024-03-24T00:48:53.984378+0000","last_fullsized":"2024-03-24T00:48:53.984378+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.996693+0000","last_change":"2024-03-24T00:48:53.989088+0000","last_active":"2024-03-24T00:48:53.996693+0000","last_peered":"2024-03-24T00:48:53.996693+0000","last_clean":"2024-03-24T00:48:53.996693+0000","last_became_active":"2024-03-24T00:48:53.988812+0000","last_became_peered":"2024-03-24T00:48:53.988812+0000","last_unstale":"2024-03-24T00:48:53.996693+0000","last_undegraded":"2024-03-24T00:48:53.996693+0000","last_fullsized":"2024-03-24T00:48:53.996693+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:46:58.027816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.983531+0000","last_change":"2024-03-24T00:48:53.983531+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.983531+0000","last_undegraded":"2024-03-24T00:48:53.983531+0000","last_fullsized":"2024-03-24T00:48:53.983531+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.997431+0000","last_change":"2024-03-24T00:48:53.992661+0000","last_active":"2024-03-24T00:48:53.997431+0000","last_peered":"2024-03-24T00:48:53.997431+0000","last_clean":"2024-03-24T00:48:53.997431+0000","last_became_active":"2024-03-24T00:48:53.992347+0000","last_became_peered":"2024-03-24T00:48:53.992347+0000","last_unstale":"2024-03-24T00:48:53.997431+0000","last_undegraded":"2024-03-24T00:48:53.997431+0000","last_fullsized":"2024-03-24T00:48:53.997431+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:21:21.714524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.982683+0000","last_change":"2024-03-24T00:48:53.982683+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.982683+0000","last_undegraded":"2024-03-24T00:48:53.982683+0000","last_fullsized":"2024-03-24T00:48:53.982683+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.997862+0000","last_change":"2024-03-24T00:48:53.995870+0000","last_active":"2024-03-24T00:48:53.997862+0000","last_peered":"2024-03-24T00:48:53.997862+0000","last_clean":"2024-03-24T00:48:53.997862+0000","last_became_active":"2024-03-24T00:48:53.995507+0000","last_became_peered":"2024-03-24T00:48:53.995507+0000","last_unstale":"2024-03-24T00:48:53.997862+0000","last_undegraded":"2024-03-24T00:48:53.997862+0000","last_fullsized":"2024-03-24T00:48:53.997862+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:07:44.083528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.978432+0000","last_change":"2024-03-24T00:48:53.978432+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.978432+0000","last_undegraded":"2024-03-24T00:48:53.978432+0000","last_fullsized":"2024-03-24T00:48:53.978432+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.997550+0000","last_change":"2024-03-24T00:48:53.993911+0000","last_active":"2024-03-24T00:48:53.997550+0000","last_peered":"2024-03-24T00:48:53.997550+0000","last_clean":"2024-03-24T00:48:53.997550+0000","last_became_active":"2024-03-24T00:48:53.993618+0000","last_became_peered":"2024-03-24T00:48:53.993618+0000","last_unstale":"2024-03-24T00:48:53.997550+0000","last_undegraded":"2024-03-24T00:48:53.997550+0000","last_fullsized":"2024-03-24T00:48:53.997550+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:37:28.523220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.983938+0000","last_change":"2024-03-24T00:48:53.983938+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.983938+0000","last_undegraded":"2024-03-24T00:48:53.983938+0000","last_fullsized":"2024-03-24T00:48:53.983938+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.998467+0000","last_change":"2024-03-24T00:48:53.994470+0000","last_active":"2024-03-24T00:48:53.998467+0000","last_peered":"2024-03-24T00:48:53.998467+0000","last_clean":"2024-03-24T00:48:53.998467+0000","last_became_active":"2024-03-24T00:48:53.994193+0000","last_became_peered":"2024-03-24T00:48:53.994193+0000","last_unstale":"2024-03-24T00:48:53.998467+0000","last_undegraded":"2024-03-24T00:48:53.998467+0000","last_fullsized":"2024-03-24T00:48:53.998467+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:30:58.413234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.986488+0000","last_change":"2024-03-24T00:48:53.986488+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.986488+0000","last_undegraded":"2024-03-24T00:48:53.986488+0000","last_fullsized":"2024-03-24T00:48:53.986488+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.997675+0000","last_change":"2024-03-24T00:48:53.995779+0000","last_active":"2024-03-24T00:48:53.997675+0000","last_peered":"2024-03-24T00:48:53.997675+0000","last_clean":"2024-03-24T00:48:53.997675+0000","last_became_active":"2024-03-24T00:48:53.995516+0000","last_became_peered":"2024-03-24T00:48:53.995516+0000","last_unstale":"2024-03-24T00:48:53.997675+0000","last_undegraded":"2024-03-24T00:48:53.997675+0000","last_fullsized":"2024-03-24T00:48:53.997675+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:55:30.565494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.982630+0000","last_change":"2024-03-24T00:48:53.982630+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.982630+0000","last_undegraded":"2024-03-24T00:48:53.982630+0000","last_fullsized":"2024-03-24T00:48:53.982630+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.997656+0000","last_change":"2024-03-24T00:48:53.995776+0000","last_active":"2024-03-24T00:48:53.997656+0000","last_peered":"2024-03-24T00:48:53.997656+0000","last_clean":"2024-03-24T00:48:53.997656+0000","last_became_active":"2024-03-24T00:48:53.995494+0000","last_became_peered":"2024-03-24T00:48:53.995494+0000","last_unstale":"2024-03-24T00:48:53.997656+0000","last_undegraded":"2024-03-24T00:48:53.997656+0000","last_fullsized":"2024-03-24T00:48:53.997656+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:16:17.295799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.977688+0000","last_change":"2024-03-24T00:48:53.977688+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.977688+0000","last_undegraded":"2024-03-24T00:48:53.977688+0000","last_fullsized":"2024-03-24T00:48:53.977688+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:54.000204+0000","last_change":"2024-03-24T00:48:53.996637+0000","last_active":"2024-03-24T00:48:54.000204+0000","last_peered":"2024-03-24T00:48:54.000204+0000","last_clean":"2024-03-24T00:48:54.000204+0000","last_became_active":"2024-03-24T00:48:53.996356+0000","last_became_peered":"2024-03-24T00:48:53.996356+0000","last_unstale":"2024-03-24T00:48:54.000204+0000","last_undegraded":"2024-03-24T00:48:54.000204+0000","last_fullsized":"2024-03-24T00:48:54.000204+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:13:07.121290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.983425+0000","last_change":"2024-03-24T00:48:53.983425+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.983425+0000","last_undegraded":"2024-03-24T00:48:53.983425+0000","last_fullsized":"2024-03-24T00:48:53.983425+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:54.003146+0000","last_change":"2024-03-24T00:48:53.998500+0000","last_active":"2024-03-24T00:48:54.003146+0000","last_peered":"2024-03-24T00:48:54.003146+0000","last_clean":"2024-03-24T00:48:54.003146+0000","last_became_active":"2024-03-24T00:48:53.998219+0000","last_became_peered":"2024-03-24T00:48:53.998219+0000","last_unstale":"2024-03-24T00:48:54.003146+0000","last_undegraded":"2024-03-24T00:48:54.003146+0000","last_fullsized":"2024-03-24T00:48:54.003146+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:35:18.392188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.983109+0000","last_change":"2024-03-24T00:48:53.983109+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.983109+0000","last_undegraded":"2024-03-24T00:48:53.983109+0000","last_fullsized":"2024-03-24T00:48:53.983109+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.997008+0000","last_change":"2024-03-24T00:48:53.992750+0000","last_active":"2024-03-24T00:48:53.997008+0000","last_peered":"2024-03-24T00:48:53.997008+0000","last_clean":"2024-03-24T00:48:53.997008+0000","last_became_active":"2024-03-24T00:48:53.992433+0000","last_became_peered":"2024-03-24T00:48:53.992433+0000","last_unstale":"2024-03-24T00:48:53.997008+0000","last_undegraded":"2024-03-24T00:48:53.997008+0000","last_fullsized":"2024-03-24T00:48:53.997008+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:55:30.565494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.981416+0000","last_change":"2024-03-24T00:48:53.981416+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.981416+0000","last_undegraded":"2024-03-24T00:48:53.981416+0000","last_fullsized":"2024-03-24T00:48:53.981416+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:54.000238+0000","last_change":"2024-03-24T00:48:53.997216+0000","last_active":"2024-03-24T00:48:54.000238+0000","last_peered":"2024-03-24T00:48:54.000238+0000","last_clean":"2024-03-24T00:48:54.000238+0000","last_became_active":"2024-03-24T00:48:53.996916+0000","last_became_peered":"2024-03-24T00:48:53.996916+0000","last_unstale":"2024-03-24T00:48:54.000238+0000","last_undegraded":"2024-03-24T00:48:54.000238+0000","last_fullsized":"2024-03-24T00:48:54.000238+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:54:53.066944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.985442+0000","last_change":"2024-03-24T00:48:53.985442+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.985442+0000","last_undegraded":"2024-03-24T00:48:53.985442+0000","last_fullsized":"2024-03-24T00:48:53.985442+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.997530+0000","last_change":"2024-03-24T00:48:53.993376+0000","last_active":"2024-03-24T00:48:53.997530+0000","last_peered":"2024-03-24T00:48:53.997530+0000","last_clean":"2024-03-24T00:48:53.997530+0000","last_became_active":"2024-03-24T00:48:53.993104+0000","last_became_peered":"2024-03-24T00:48:53.993104+0000","last_unstale":"2024-03-24T00:48:53.997530+0000","last_undegraded":"2024-03-24T00:48:53.997530+0000","last_fullsized":"2024-03-24T00:48:53.997530+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:32:33.042770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":1,"reported_epoch":118,"state":"creating+peering","last_fresh":"2024-03-24T00:48:53.981895+0000","last_change":"2024-03-24T00:48:53.981895+0000","last_active":"2024-03-24T00:48:53.970087+0000","last_peered":"2024-03-24T00:48:53.970087+0000","last_clean":"2024-03-24T00:48:53.970087+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:48:53.981895+0000","last_undegraded":"2024-03-24T00:48:53.981895+0000","last_fullsized":"2024-03-24T00:48:53.981895+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.993067+0000","last_change":"2024-03-24T00:48:53.988008+0000","last_active":"2024-03-24T00:48:53.993067+0000","last_peered":"2024-03-24T00:48:53.993067+0000","last_clean":"2024-03-24T00:48:53.993067+0000","last_became_active":"2024-03-24T00:48:53.987085+0000","last_became_peered":"2024-03-24T00:48:53.987085+0000","last_unstale":"2024-03-24T00:48:53.993067+0000","last_undegraded":"2024-03-24T00:48:53.993067+0000","last_fullsized":"2024-03-24T00:48:53.993067+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:26:28.796609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":354,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:48:53.985066+0000","last_change":"2024-03-24T00:38:40.953871+0000","last_active":"2024-03-24T00:48:53.985066+0000","last_peered":"2024-03-24T00:48:53.985066+0000","last_clean":"2024-03-24T00:48:53.985066+0000","last_became_active":"2024-03-24T00:38:40.952926+0000","last_became_peered":"2024-03-24T00:38:40.952926+0000","last_unstale":"2024-03-24T00:48:53.985066+0000","last_undegraded":"2024-03-24T00:48:53.985066+0000","last_fullsized":"2024-03-24T00:48:53.985066+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_clean_scrub_stamp":"2024-03-24T00:38:39.722194+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:05.397257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574976,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33184,"kb_used_data":2004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338656,"statfs":{"total":96636764160,"available":96602783744,"internally_reserved":0,"allocated":2052096,"data_stored":1269369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574975,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32604,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339236,"statfs":{"total":96636764160,"available":96603377664,"internally_reserved":0,"allocated":1458176,"data_stored":679001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574975,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32604,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339236,"statfs":{"total":96636764160,"available":96603377664,"internally_reserved":0,"allocated":1458176,"data_stored":679001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574974,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32620,"kb_used_data":1436,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339220,"statfs":{"total":96636764160,"available":96603361280,"internally_reserved":0,"allocated":1470464,"data_stored":685668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574974,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32604,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339236,"statfs":{"total":96636764160,"available":96603377664,"internally_reserved":0,"allocated":1458176,"data_stored":679001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574975,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32604,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339236,"statfs":{"total":96636764160,"available":96603377664,"internally_reserved":0,"allocated":1458176,"data_stored":679001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574975,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32620,"kb_used_data":1436,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339220,"statfs":{"total":96636764160,"available":96603361280,"internally_reserved":0,"allocated":1470464,"data_stored":685668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574974,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33196,"kb_used_data":2016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338644,"statfs":{"total":96636764160,"available":96602771456,"internally_reserved":0,"allocated":2064384,"data_stored":1276036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:48:58.707 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.3f is not active+clean 2024-03-24T00:48:58.708 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.981887+0000', 'last_change': '2024-03-24T00:48:53.981887+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.981887+0000', 'last_undegraded': '2024-03-24T00:48:53.981887+0000', 'last_fullsized': '2024-03-24T00:48:53.981887+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-24T00:48:58.708 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.3e is not active+clean 2024-03-24T00:48:58.708 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.987478+0000', 'last_change': '2024-03-24T00:48:53.987478+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.987478+0000', 'last_undegraded': '2024-03-24T00:48:53.987478+0000', 'last_fullsized': '2024-03-24T00:48:53.987478+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T00:48:58.708 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.3d is not active+clean 2024-03-24T00:48:58.708 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.986072+0000', 'last_change': '2024-03-24T00:48:53.986072+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.986072+0000', 'last_undegraded': '2024-03-24T00:48:53.986072+0000', 'last_fullsized': '2024-03-24T00:48:53.986072+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T00:48:58.708 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.3c is not active+clean 2024-03-24T00:48:58.708 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.981339+0000', 'last_change': '2024-03-24T00:48:53.981339+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.981339+0000', 'last_undegraded': '2024-03-24T00:48:53.981339+0000', 'last_fullsized': '2024-03-24T00:48:53.981339+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T00:48:58.708 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.3b is not active+clean 2024-03-24T00:48:58.708 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.986981+0000', 'last_change': '2024-03-24T00:48:53.986981+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.986981+0000', 'last_undegraded': '2024-03-24T00:48:53.986981+0000', 'last_fullsized': '2024-03-24T00:48:53.986981+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-24T00:48:58.708 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.3a is not active+clean 2024-03-24T00:48:58.708 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.988748+0000', 'last_change': '2024-03-24T00:48:53.988748+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.988748+0000', 'last_undegraded': '2024-03-24T00:48:53.988748+0000', 'last_fullsized': '2024-03-24T00:48:53.988748+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T00:48:58.709 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.39 is not active+clean 2024-03-24T00:48:58.709 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.987478+0000', 'last_change': '2024-03-24T00:48:53.987478+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.987478+0000', 'last_undegraded': '2024-03-24T00:48:53.987478+0000', 'last_fullsized': '2024-03-24T00:48:53.987478+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T00:48:58.709 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.38 is not active+clean 2024-03-24T00:48:58.709 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.986010+0000', 'last_change': '2024-03-24T00:48:53.986010+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.986010+0000', 'last_undegraded': '2024-03-24T00:48:53.986010+0000', 'last_fullsized': '2024-03-24T00:48:53.986010+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T00:48:58.709 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.37 is not active+clean 2024-03-24T00:48:58.709 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.982242+0000', 'last_change': '2024-03-24T00:48:53.982242+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.982242+0000', 'last_undegraded': '2024-03-24T00:48:53.982242+0000', 'last_fullsized': '2024-03-24T00:48:53.982242+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T00:48:58.709 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.36 is not active+clean 2024-03-24T00:48:58.709 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.990365+0000', 'last_change': '2024-03-24T00:48:53.990365+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.990365+0000', 'last_undegraded': '2024-03-24T00:48:53.990365+0000', 'last_fullsized': '2024-03-24T00:48:53.990365+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-24T00:48:58.709 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.35 is not active+clean 2024-03-24T00:48:58.709 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.981850+0000', 'last_change': '2024-03-24T00:48:53.981850+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.981850+0000', 'last_undegraded': '2024-03-24T00:48:53.981850+0000', 'last_fullsized': '2024-03-24T00:48:53.981850+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-24T00:48:58.709 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.34 is not active+clean 2024-03-24T00:48:58.709 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.987024+0000', 'last_change': '2024-03-24T00:48:53.987024+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.987024+0000', 'last_undegraded': '2024-03-24T00:48:53.987024+0000', 'last_fullsized': '2024-03-24T00:48:53.987024+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T00:48:58.710 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.33 is not active+clean 2024-03-24T00:48:58.710 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.989677+0000', 'last_change': '2024-03-24T00:48:53.989677+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.989677+0000', 'last_undegraded': '2024-03-24T00:48:53.989677+0000', 'last_fullsized': '2024-03-24T00:48:53.989677+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-24T00:48:58.710 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.32 is not active+clean 2024-03-24T00:48:58.710 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.985341+0000', 'last_change': '2024-03-24T00:48:53.985341+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.985341+0000', 'last_undegraded': '2024-03-24T00:48:53.985341+0000', 'last_fullsized': '2024-03-24T00:48:53.985341+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T00:48:58.710 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.31 is not active+clean 2024-03-24T00:48:58.710 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.981660+0000', 'last_change': '2024-03-24T00:48:53.981660+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.981660+0000', 'last_undegraded': '2024-03-24T00:48:53.981660+0000', 'last_fullsized': '2024-03-24T00:48:53.981660+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-24T00:48:58.710 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.30 is not active+clean 2024-03-24T00:48:58.710 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.986575+0000', 'last_change': '2024-03-24T00:48:53.986575+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.986575+0000', 'last_undegraded': '2024-03-24T00:48:53.986575+0000', 'last_fullsized': '2024-03-24T00:48:53.986575+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T00:48:58.710 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.2f is not active+clean 2024-03-24T00:48:58.710 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.989175+0000', 'last_change': '2024-03-24T00:48:53.989175+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.989175+0000', 'last_undegraded': '2024-03-24T00:48:53.989175+0000', 'last_fullsized': '2024-03-24T00:48:53.989175+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-24T00:48:58.711 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.2e is not active+clean 2024-03-24T00:48:58.711 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.984128+0000', 'last_change': '2024-03-24T00:48:53.984128+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.984128+0000', 'last_undegraded': '2024-03-24T00:48:53.984128+0000', 'last_fullsized': '2024-03-24T00:48:53.984128+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T00:48:58.711 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.2d is not active+clean 2024-03-24T00:48:58.711 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.983103+0000', 'last_change': '2024-03-24T00:48:53.983103+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.983103+0000', 'last_undegraded': '2024-03-24T00:48:53.983103+0000', 'last_fullsized': '2024-03-24T00:48:53.983103+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-24T00:48:58.711 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.2c is not active+clean 2024-03-24T00:48:58.711 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.977625+0000', 'last_change': '2024-03-24T00:48:53.977625+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.977625+0000', 'last_undegraded': '2024-03-24T00:48:53.977625+0000', 'last_fullsized': '2024-03-24T00:48:53.977625+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-24T00:48:58.711 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.2b is not active+clean 2024-03-24T00:48:58.711 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.982297+0000', 'last_change': '2024-03-24T00:48:53.982297+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.982297+0000', 'last_undegraded': '2024-03-24T00:48:53.982297+0000', 'last_fullsized': '2024-03-24T00:48:53.982297+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T00:48:58.711 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.2a is not active+clean 2024-03-24T00:48:58.711 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.982445+0000', 'last_change': '2024-03-24T00:48:53.982445+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.982445+0000', 'last_undegraded': '2024-03-24T00:48:53.982445+0000', 'last_fullsized': '2024-03-24T00:48:53.982445+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-24T00:48:58.711 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.29 is not active+clean 2024-03-24T00:48:58.711 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.982875+0000', 'last_change': '2024-03-24T00:48:53.982875+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.982875+0000', 'last_undegraded': '2024-03-24T00:48:53.982875+0000', 'last_fullsized': '2024-03-24T00:48:53.982875+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T00:48:58.712 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.28 is not active+clean 2024-03-24T00:48:58.712 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.979109+0000', 'last_change': '2024-03-24T00:48:53.979109+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.979109+0000', 'last_undegraded': '2024-03-24T00:48:53.979109+0000', 'last_fullsized': '2024-03-24T00:48:53.979109+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-24T00:48:58.712 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.27 is not active+clean 2024-03-24T00:48:58.712 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.979993+0000', 'last_change': '2024-03-24T00:48:53.979993+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.979993+0000', 'last_undegraded': '2024-03-24T00:48:53.979993+0000', 'last_fullsized': '2024-03-24T00:48:53.979993+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-24T00:48:58.712 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.26 is not active+clean 2024-03-24T00:48:58.712 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.978637+0000', 'last_change': '2024-03-24T00:48:53.978637+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.978637+0000', 'last_undegraded': '2024-03-24T00:48:53.978637+0000', 'last_fullsized': '2024-03-24T00:48:53.978637+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-24T00:48:58.712 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.25 is not active+clean 2024-03-24T00:48:58.712 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.982655+0000', 'last_change': '2024-03-24T00:48:53.982655+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.982655+0000', 'last_undegraded': '2024-03-24T00:48:53.982655+0000', 'last_fullsized': '2024-03-24T00:48:53.982655+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-24T00:48:58.712 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.24 is not active+clean 2024-03-24T00:48:58.712 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.986026+0000', 'last_change': '2024-03-24T00:48:53.986026+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.986026+0000', 'last_undegraded': '2024-03-24T00:48:53.986026+0000', 'last_fullsized': '2024-03-24T00:48:53.986026+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T00:48:58.712 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.23 is not active+clean 2024-03-24T00:48:58.713 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.977622+0000', 'last_change': '2024-03-24T00:48:53.977622+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.977622+0000', 'last_undegraded': '2024-03-24T00:48:53.977622+0000', 'last_fullsized': '2024-03-24T00:48:53.977622+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-24T00:48:58.713 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.22 is not active+clean 2024-03-24T00:48:58.713 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.984581+0000', 'last_change': '2024-03-24T00:48:53.984581+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.984581+0000', 'last_undegraded': '2024-03-24T00:48:53.984581+0000', 'last_fullsized': '2024-03-24T00:48:53.984581+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T00:48:58.713 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.21 is not active+clean 2024-03-24T00:48:58.713 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.979193+0000', 'last_change': '2024-03-24T00:48:53.979193+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.979193+0000', 'last_undegraded': '2024-03-24T00:48:53.979193+0000', 'last_fullsized': '2024-03-24T00:48:53.979193+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-24T00:48:58.713 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.20 is not active+clean 2024-03-24T00:48:58.713 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.978546+0000', 'last_change': '2024-03-24T00:48:53.978546+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.978546+0000', 'last_undegraded': '2024-03-24T00:48:53.978546+0000', 'last_fullsized': '2024-03-24T00:48:53.978546+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-24T00:48:58.713 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.1c is not active+clean 2024-03-24T00:48:58.713 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.990974+0000', 'last_change': '2024-03-24T00:48:53.990974+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.990974+0000', 'last_undegraded': '2024-03-24T00:48:53.990974+0000', 'last_fullsized': '2024-03-24T00:48:53.990974+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-24T00:48:58.713 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.1d is not active+clean 2024-03-24T00:48:58.713 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.985412+0000', 'last_change': '2024-03-24T00:48:53.985412+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.985412+0000', 'last_undegraded': '2024-03-24T00:48:53.985412+0000', 'last_fullsized': '2024-03-24T00:48:53.985412+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T00:48:58.714 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.1e is not active+clean 2024-03-24T00:48:58.714 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.978888+0000', 'last_change': '2024-03-24T00:48:53.978888+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.978888+0000', 'last_undegraded': '2024-03-24T00:48:53.978888+0000', 'last_fullsized': '2024-03-24T00:48:53.978888+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-24T00:48:58.714 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.1f is not active+clean 2024-03-24T00:48:58.714 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.981809+0000', 'last_change': '2024-03-24T00:48:53.981809+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.981809+0000', 'last_undegraded': '2024-03-24T00:48:53.981809+0000', 'last_fullsized': '2024-03-24T00:48:53.981809+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T00:48:58.714 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.18 is not active+clean 2024-03-24T00:48:58.714 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.986877+0000', 'last_change': '2024-03-24T00:48:53.986877+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.986877+0000', 'last_undegraded': '2024-03-24T00:48:53.986877+0000', 'last_fullsized': '2024-03-24T00:48:53.986877+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-24T00:48:58.714 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.19 is not active+clean 2024-03-24T00:48:58.714 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.982871+0000', 'last_change': '2024-03-24T00:48:53.982871+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.982871+0000', 'last_undegraded': '2024-03-24T00:48:53.982871+0000', 'last_fullsized': '2024-03-24T00:48:53.982871+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-24T00:48:58.714 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.1a is not active+clean 2024-03-24T00:48:58.714 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.984118+0000', 'last_change': '2024-03-24T00:48:53.984118+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.984118+0000', 'last_undegraded': '2024-03-24T00:48:53.984118+0000', 'last_fullsized': '2024-03-24T00:48:53.984118+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T00:48:58.714 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.1b is not active+clean 2024-03-24T00:48:58.714 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.987465+0000', 'last_change': '2024-03-24T00:48:53.987465+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.987465+0000', 'last_undegraded': '2024-03-24T00:48:53.987465+0000', 'last_fullsized': '2024-03-24T00:48:53.987465+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-24T00:48:58.715 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.14 is not active+clean 2024-03-24T00:48:58.715 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.983527+0000', 'last_change': '2024-03-24T00:48:53.983527+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.983527+0000', 'last_undegraded': '2024-03-24T00:48:53.983527+0000', 'last_fullsized': '2024-03-24T00:48:53.983527+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-24T00:48:58.715 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.15 is not active+clean 2024-03-24T00:48:58.715 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.978769+0000', 'last_change': '2024-03-24T00:48:53.978769+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.978769+0000', 'last_undegraded': '2024-03-24T00:48:53.978769+0000', 'last_fullsized': '2024-03-24T00:48:53.978769+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-24T00:48:58.715 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.16 is not active+clean 2024-03-24T00:48:58.715 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.983568+0000', 'last_change': '2024-03-24T00:48:53.983568+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.983568+0000', 'last_undegraded': '2024-03-24T00:48:53.983568+0000', 'last_fullsized': '2024-03-24T00:48:53.983568+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T00:48:58.715 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.17 is not active+clean 2024-03-24T00:48:58.715 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.979598+0000', 'last_change': '2024-03-24T00:48:53.979598+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.979598+0000', 'last_undegraded': '2024-03-24T00:48:53.979598+0000', 'last_fullsized': '2024-03-24T00:48:53.979598+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-24T00:48:58.715 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.10 is not active+clean 2024-03-24T00:48:58.715 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.986841+0000', 'last_change': '2024-03-24T00:48:53.986841+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.986841+0000', 'last_undegraded': '2024-03-24T00:48:53.986841+0000', 'last_fullsized': '2024-03-24T00:48:53.986841+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-24T00:48:58.716 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.11 is not active+clean 2024-03-24T00:48:58.716 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.986869+0000', 'last_change': '2024-03-24T00:48:53.986869+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.986869+0000', 'last_undegraded': '2024-03-24T00:48:53.986869+0000', 'last_fullsized': '2024-03-24T00:48:53.986869+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-24T00:48:58.716 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.12 is not active+clean 2024-03-24T00:48:58.716 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.977688+0000', 'last_change': '2024-03-24T00:48:53.977688+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.977688+0000', 'last_undegraded': '2024-03-24T00:48:53.977688+0000', 'last_fullsized': '2024-03-24T00:48:53.977688+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-24T00:48:58.716 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.c is not active+clean 2024-03-24T00:48:58.716 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.978648+0000', 'last_change': '2024-03-24T00:48:53.978648+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.978648+0000', 'last_undegraded': '2024-03-24T00:48:53.978648+0000', 'last_fullsized': '2024-03-24T00:48:53.978648+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-24T00:48:58.716 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.d is not active+clean 2024-03-24T00:48:58.716 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.981376+0000', 'last_change': '2024-03-24T00:48:53.981376+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.981376+0000', 'last_undegraded': '2024-03-24T00:48:53.981376+0000', 'last_fullsized': '2024-03-24T00:48:53.981376+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-24T00:48:58.716 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.e is not active+clean 2024-03-24T00:48:58.716 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.984924+0000', 'last_change': '2024-03-24T00:48:53.984924+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.984924+0000', 'last_undegraded': '2024-03-24T00:48:53.984924+0000', 'last_fullsized': '2024-03-24T00:48:53.984924+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T00:48:58.716 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.f is not active+clean 2024-03-24T00:48:58.716 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.979161+0000', 'last_change': '2024-03-24T00:48:53.979161+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.979161+0000', 'last_undegraded': '2024-03-24T00:48:53.979161+0000', 'last_fullsized': '2024-03-24T00:48:53.979161+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-24T00:48:58.717 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.8 is not active+clean 2024-03-24T00:48:58.717 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.984378+0000', 'last_change': '2024-03-24T00:48:53.984378+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.984378+0000', 'last_undegraded': '2024-03-24T00:48:53.984378+0000', 'last_fullsized': '2024-03-24T00:48:53.984378+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T00:48:58.717 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.9 is not active+clean 2024-03-24T00:48:58.717 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.983531+0000', 'last_change': '2024-03-24T00:48:53.983531+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.983531+0000', 'last_undegraded': '2024-03-24T00:48:53.983531+0000', 'last_fullsized': '2024-03-24T00:48:53.983531+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T00:48:58.717 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.a is not active+clean 2024-03-24T00:48:58.717 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.982683+0000', 'last_change': '2024-03-24T00:48:53.982683+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.982683+0000', 'last_undegraded': '2024-03-24T00:48:53.982683+0000', 'last_fullsized': '2024-03-24T00:48:53.982683+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T00:48:58.717 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.b is not active+clean 2024-03-24T00:48:58.717 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.978432+0000', 'last_change': '2024-03-24T00:48:53.978432+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.978432+0000', 'last_undegraded': '2024-03-24T00:48:53.978432+0000', 'last_fullsized': '2024-03-24T00:48:53.978432+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-24T00:48:58.717 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.4 is not active+clean 2024-03-24T00:48:58.717 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.983938+0000', 'last_change': '2024-03-24T00:48:53.983938+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.983938+0000', 'last_undegraded': '2024-03-24T00:48:53.983938+0000', 'last_fullsized': '2024-03-24T00:48:53.983938+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T00:48:58.717 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.5 is not active+clean 2024-03-24T00:48:58.717 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.986488+0000', 'last_change': '2024-03-24T00:48:53.986488+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.986488+0000', 'last_undegraded': '2024-03-24T00:48:53.986488+0000', 'last_fullsized': '2024-03-24T00:48:53.986488+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-24T00:48:58.718 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.6 is not active+clean 2024-03-24T00:48:58.718 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.982630+0000', 'last_change': '2024-03-24T00:48:53.982630+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.982630+0000', 'last_undegraded': '2024-03-24T00:48:53.982630+0000', 'last_fullsized': '2024-03-24T00:48:53.982630+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T00:48:58.718 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.7 is not active+clean 2024-03-24T00:48:58.718 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.977688+0000', 'last_change': '2024-03-24T00:48:53.977688+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.977688+0000', 'last_undegraded': '2024-03-24T00:48:53.977688+0000', 'last_fullsized': '2024-03-24T00:48:53.977688+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-24T00:48:58.718 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.0 is not active+clean 2024-03-24T00:48:58.718 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.983425+0000', 'last_change': '2024-03-24T00:48:53.983425+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.983425+0000', 'last_undegraded': '2024-03-24T00:48:53.983425+0000', 'last_fullsized': '2024-03-24T00:48:53.983425+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T00:48:58.718 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.1 is not active+clean 2024-03-24T00:48:58.718 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.983109+0000', 'last_change': '2024-03-24T00:48:53.983109+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.983109+0000', 'last_undegraded': '2024-03-24T00:48:53.983109+0000', 'last_fullsized': '2024-03-24T00:48:53.983109+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T00:48:58.718 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.2 is not active+clean 2024-03-24T00:48:58.718 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.981416+0000', 'last_change': '2024-03-24T00:48:53.981416+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.981416+0000', 'last_undegraded': '2024-03-24T00:48:53.981416+0000', 'last_fullsized': '2024-03-24T00:48:53.981416+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T00:48:58.719 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.3 is not active+clean 2024-03-24T00:48:58.719 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.985442+0000', 'last_change': '2024-03-24T00:48:53.985442+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.985442+0000', 'last_undegraded': '2024-03-24T00:48:53.985442+0000', 'last_fullsized': '2024-03-24T00:48:53.985442+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-24T00:48:58.719 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.13 is not active+clean 2024-03-24T00:48:58.719 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 118, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:48:53.981895+0000', 'last_change': '2024-03-24T00:48:53.981895+0000', 'last_active': '2024-03-24T00:48:53.970087+0000', 'last_peered': '2024-03-24T00:48:53.970087+0000', 'last_clean': '2024-03-24T00:48:53.970087+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:48:53.981895+0000', 'last_undegraded': '2024-03-24T00:48:53.981895+0000', 'last_fullsized': '2024-03-24T00:48:53.981895+0000', 'mapping_epoch': 118, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 118, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'last_clean_scrub_stamp': '2024-03-24T00:48:53.970087+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T00:48:58.719 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-24T00:48:58.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.846+0000 7fadfda97640 1 Processor -- start 2024-03-24T00:48:58.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.846+0000 7fadfda97640 1 -- start start 2024-03-24T00:48:58.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.847+0000 7fadfda97640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fadf80d0ef0 0x7fadf80ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:58.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.847+0000 7fadfda97640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadf80cf640 con 0x7fadf815dec0 2024-03-24T00:48:58.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.847+0000 7fadfda97640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadf80cf080 con 0x7fadf815e2a0 2024-03-24T00:48:58.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.847+0000 7fadfda97640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadf80cee30 con 0x7fadf80d0ef0 2024-03-24T00:48:58.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.847+0000 7fadf77fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fadf815dec0 0x7fadf80d07e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35938/0 (socket says 172.21.15.67:35938) 2024-03-24T00:48:58.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.847+0000 7fadf77fe640 1 -- 172.21.15.67:0/1224581816 learned_addr learned my addr 172.21.15.67:0/1224581816 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:48:58.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.847+0000 7fadf7fff640 1 --2- 172.21.15.67:0/1224581816 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fadf80d0ef0 0x7fadf80ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:58.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.847+0000 7fadf67fc640 1 -- 172.21.15.67:0/1224581816 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3148156096 0 0) 0x7fadec001110 con 0x7fadf815dec0 2024-03-24T00:48:58.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.847+0000 7fadf67fc640 1 -- 172.21.15.67:0/1224581816 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fade0003520 con 0x7fadf815dec0 2024-03-24T00:48:58.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.847+0000 7fadf7fff640 1 -- 172.21.15.67:0/1224581816 >> v1:172.21.15.72:6789/0 conn(0x7fadf815e2a0 legacy=0x7fadf80d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:58.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.847+0000 7fadf67fc640 1 -- 172.21.15.67:0/1224581816 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2521937462 0 0) 0x7fadf80cf080 con 0x7fadf815e2a0 2024-03-24T00:48:58.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.847+0000 7fadf7fff640 1 -- 172.21.15.67:0/1224581816 >> v1:172.21.15.67:6789/0 conn(0x7fadf815dec0 legacy=0x7fadf80d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:58.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.847+0000 7fadf7fff640 1 -- 172.21.15.67:0/1224581816 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf80cf360 con 0x7fadf80d0ef0 2024-03-24T00:48:58.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.848+0000 7fadf7fff640 1 --2- 172.21.15.67:0/1224581816 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fadf80d0ef0 0x7fadf80ce630 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fadf0009dc0 tx=0x7fadf002be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef7da67a868696b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:58.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.848+0000 7fadf67fc640 1 -- 172.21.15.67:0/1224581816 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf0039070 con 0x7fadf80d0ef0 2024-03-24T00:48:58.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.848+0000 7fadf67fc640 1 -- 172.21.15.67:0/1224581816 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadf002cc40 con 0x7fadf80d0ef0 2024-03-24T00:48:58.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.848+0000 7fadf67fc640 1 -- 172.21.15.67:0/1224581816 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf0034440 con 0x7fadf80d0ef0 2024-03-24T00:48:58.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.849+0000 7fadfda97640 1 -- 172.21.15.67:0/1224581816 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fadf80d0ef0 msgr2=0x7fadf80ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:58.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.849+0000 7fadfda97640 1 --2- 172.21.15.67:0/1224581816 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fadf80d0ef0 0x7fadf80ce630 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fadf0009dc0 tx=0x7fadf002be90 comp rx=0 tx=0).stop 2024-03-24T00:48:58.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.849+0000 7fadfda97640 1 -- 172.21.15.67:0/1224581816 shutdown_connections 2024-03-24T00:48:58.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.849+0000 7fadfda97640 1 --2- 172.21.15.67:0/1224581816 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fadf80d0ef0 0x7fadf80ce630 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:58.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.849+0000 7fadfda97640 1 -- 172.21.15.67:0/1224581816 >> 172.21.15.67:0/1224581816 conn(0x7fadf806c810 msgr2=0x7fadf806cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:58.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.849+0000 7fadfda97640 1 -- 172.21.15.67:0/1224581816 shutdown_connections 2024-03-24T00:48:58.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.849+0000 7fadfda97640 1 -- 172.21.15.67:0/1224581816 wait complete. 2024-03-24T00:48:58.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.850+0000 7fadfda97640 1 Processor -- start 2024-03-24T00:48:58.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.850+0000 7fadfda97640 1 -- start start 2024-03-24T00:48:58.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.851+0000 7fadfda97640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fadf80d0ef0 0x7fadf80cc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:58.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.851+0000 7fadfda97640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadf815dec0 0x7fadf80c9990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:58.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.851+0000 7fadfda97640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fadf815e2a0 0x7fadf815be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:58.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.851+0000 7fadfda97640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf8162a40 con 0x7fadf815dec0 2024-03-24T00:48:58.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.851+0000 7fadfda97640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf81628c0 con 0x7fadf815e2a0 2024-03-24T00:48:58.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.851+0000 7fadfda97640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadf80d2120 con 0x7fadf80d0ef0 2024-03-24T00:48:58.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.851+0000 7fadf6ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadf815dec0 0x7fadf80c9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:58.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.851+0000 7fadf6ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadf815dec0 0x7fadf80c9990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57486/0 (socket says 172.21.15.67:57486) 2024-03-24T00:48:58.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.851+0000 7fadf6ffd640 1 -- 172.21.15.67:0/641623153 learned_addr learned my addr 172.21.15.67:0/641623153 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:58.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.851+0000 7fadf77fe640 1 --2- 172.21.15.67:0/641623153 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fadf80d0ef0 0x7fadf80cc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:58.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.851+0000 7fadf7fff640 1 --2- 172.21.15.67:0/641623153 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fadf815e2a0 0x7fadf815be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:58.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.851+0000 7fadf6ffd640 1 -- 172.21.15.67:0/641623153 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fadf80d0ef0 msgr2=0x7fadf80cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:58.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.851+0000 7fadf6ffd640 1 --2- 172.21.15.67:0/641623153 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fadf80d0ef0 0x7fadf80cc2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:58.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.851+0000 7fadf6ffd640 1 -- 172.21.15.67:0/641623153 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fadf815e2a0 msgr2=0x7fadf815be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:58.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.851+0000 7fadf6ffd640 1 --2- 172.21.15.67:0/641623153 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fadf815e2a0 0x7fadf815be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:58.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.851+0000 7fadf6ffd640 1 -- 172.21.15.67:0/641623153 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf80cc9a0 con 0x7fadf815dec0 2024-03-24T00:48:58.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.852+0000 7fadf6ffd640 1 --2- 172.21.15.67:0/641623153 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadf815dec0 0x7fadf80c9990 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7fade800c550 tx=0x7fade800ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:58.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.852+0000 7faddffff640 1 -- 172.21.15.67:0/641623153 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade8017070 con 0x7fadf815dec0 2024-03-24T00:48:58.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.852+0000 7faddffff640 1 -- 172.21.15.67:0/641623153 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fade80035f0 con 0x7fadf815dec0 2024-03-24T00:48:58.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.852+0000 7fadfda97640 1 -- 172.21.15.67:0/641623153 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadf80ccc80 con 0x7fadf815dec0 2024-03-24T00:48:58.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.852+0000 7faddffff640 1 -- 172.21.15.67:0/641623153 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade80039e0 con 0x7fadf815dec0 2024-03-24T00:48:58.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.852+0000 7fadfda97640 1 -- 172.21.15.67:0/641623153 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadf80cd240 con 0x7fadf815dec0 2024-03-24T00:48:58.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.853+0000 7fadfda97640 1 -- 172.21.15.67:0/641623153 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadf804f400 con 0x7fadf815dec0 2024-03-24T00:48:58.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.853+0000 7faddffff640 1 -- 172.21.15.67:0/641623153 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fade801a070 con 0x7fadf815dec0 2024-03-24T00:48:58.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.854+0000 7faddffff640 1 --2- 172.21.15.67:0/641623153 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fade406bbd0 0x7fade406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:58.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.854+0000 7faddffff640 1 -- 172.21.15.67:0/641623153 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fade8089fa0 con 0x7fadf815dec0 2024-03-24T00:48:58.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.854+0000 7fadf77fe640 1 --2- 172.21.15.67:0/641623153 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fade406bbd0 0x7fade406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:58.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.855+0000 7fadf77fe640 1 --2- 172.21.15.67:0/641623153 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fade406bbd0 0x7fade406e090 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fadec005880 tx=0x7fadec009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:58.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:58.858+0000 7faddffff640 1 -- 172.21.15.67:0/641623153 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fade8058990 con 0x7fadf815dec0 2024-03-24T00:48:59.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.288+0000 7fadfda97640 1 -- 172.21.15.67:0/641623153 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fadf80d1aa0 con 0x7fadf815dec0 2024-03-24T00:48:59.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.288+0000 7faddffff640 1 -- 172.21.15.67:0/641623153 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1197 (secure 0 0 0) 0x7fade8058290 con 0x7fadf815dec0 2024-03-24T00:48:59.288 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:48:59.288 INFO:teuthology.orchestra.run.smithi067.stdout:{"fsid":"de1d0178-808e-49ca-a986-e297e523b976","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":632,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":120,"num_osds":8,"num_up_osds":8,"osd_up_since":1711240718,"num_in_osds":8,"osd_in_since":1711240709,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":33}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":268324864,"bytes_avail":772825788416,"bytes_total":773094113280,"inactive_pgs_ratio":0.65979379415512085},"fsmap":{"epoch":86,"id":8,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":8,"rank":0,"name":"a","status":"up:active","gid":9219}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":23,"modified":"2024-03-24T00:48:13.458286+0000","services":{"mds":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-24T00:48:59.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.290+0000 7fadfda97640 1 -- 172.21.15.67:0/641623153 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fade406bbd0 msgr2=0x7fade406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:59.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.290+0000 7fadfda97640 1 --2- 172.21.15.67:0/641623153 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fade406bbd0 0x7fade406e090 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fadec005880 tx=0x7fadec009040 comp rx=0 tx=0).stop 2024-03-24T00:48:59.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.290+0000 7fadfda97640 1 -- 172.21.15.67:0/641623153 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadf815dec0 msgr2=0x7fadf80c9990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:59.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.290+0000 7fadfda97640 1 --2- 172.21.15.67:0/641623153 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadf815dec0 0x7fadf80c9990 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7fade800c550 tx=0x7fade800ca10 comp rx=0 tx=0).stop 2024-03-24T00:48:59.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.290+0000 7fadfda97640 1 -- 172.21.15.67:0/641623153 shutdown_connections 2024-03-24T00:48:59.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.290+0000 7fadfda97640 1 --2- 172.21.15.67:0/641623153 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fade406bbd0 0x7fade406e090 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:59.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.290+0000 7fadfda97640 1 --2- 172.21.15.67:0/641623153 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fadf815e2a0 0x7fadf815be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:59.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.290+0000 7fadfda97640 1 --2- 172.21.15.67:0/641623153 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadf815dec0 0x7fadf80c9990 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:59.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.290+0000 7fadfda97640 1 --2- 172.21.15.67:0/641623153 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fadf80d0ef0 0x7fadf80cc2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:59.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.290+0000 7fadfda97640 1 -- 172.21.15.67:0/641623153 >> 172.21.15.67:0/641623153 conn(0x7fadf806c810 msgr2=0x7fadf815d490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:59.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.290+0000 7fadfda97640 1 -- 172.21.15.67:0/641623153 shutdown_connections 2024-03-24T00:48:59.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.290+0000 7fadfda97640 1 -- 172.21.15.67:0/641623153 wait complete. 2024-03-24T00:48:59.301 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-24T00:48:59.301 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:48:59.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.403+0000 7fc1cc20b640 1 Processor -- start 2024-03-24T00:48:59.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.403+0000 7fc1cc20b640 1 -- start start 2024-03-24T00:48:59.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.404+0000 7fc1cc20b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc1c415bf70 0x7fc1c40cc690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:59.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.404+0000 7fc1cc20b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc1c40ce930 0x7fc1c40b1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:59.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.404+0000 7fc1cc20b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1c40ccfe0 con 0x7fc1c40cfc70 2024-03-24T00:48:59.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.404+0000 7fc1cc20b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1c40cd2c0 con 0x7fc1c415bf70 2024-03-24T00:48:59.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.404+0000 7fc1cc20b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c40cd5a0 con 0x7fc1c40ce930 2024-03-24T00:48:59.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.404+0000 7fc1c97a6640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc1c40cfc70 0x7fc1c40ce220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35946/0 (socket says 172.21.15.67:35946) 2024-03-24T00:48:59.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.404+0000 7fc1c97a6640 1 -- 172.21.15.67:0/3526868440 learned_addr learned my addr 172.21.15.67:0/3526868440 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:48:59.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.404+0000 7fc1ca7a8640 1 --2- 172.21.15.67:0/3526868440 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc1c40ce930 0x7fc1c40b1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:59.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.404+0000 7fc1c9fa7640 1 --2- 172.21.15.67:0/3526868440 >> v2:172.21.15.72:3300/0 conn(0x7fc1c415bf70 0x7fc1c40cc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:59.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.404+0000 7fc1c8fa5640 1 -- 172.21.15.67:0/3526868440 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1180685638 0 0) 0x7fc1c40ccfe0 con 0x7fc1c40cfc70 2024-03-24T00:48:59.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.404+0000 7fc1c8fa5640 1 -- 172.21.15.67:0/3526868440 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1b0003520 con 0x7fc1c40cfc70 2024-03-24T00:48:59.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.405+0000 7fc1c9fa7640 1 -- 172.21.15.67:0/3526868440 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc1c40ce930 msgr2=0x7fc1c40b1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:48:59.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.405+0000 7fc1c9fa7640 1 --2- 172.21.15.67:0/3526868440 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc1c40ce930 0x7fc1c40b1cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:59.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.405+0000 7fc1c9fa7640 1 -- 172.21.15.67:0/3526868440 >> v1:172.21.15.67:6789/0 conn(0x7fc1c40cfc70 legacy=0x7fc1c40ce220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:59.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.405+0000 7fc1c9fa7640 1 -- 172.21.15.67:0/3526868440 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c40cd880 con 0x7fc1c415bf70 2024-03-24T00:48:59.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.405+0000 7fc1c9fa7640 1 --2- 172.21.15.67:0/3526868440 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc1c415bf70 0x7fc1c40cc690 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fc1c00097a0 tx=0x7fc1c002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=128fbdd4ebb13343 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:59.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.406+0000 7fc1c8fa5640 1 -- 172.21.15.67:0/3526868440 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c0035020 con 0x7fc1c415bf70 2024-03-24T00:48:59.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.406+0000 7fc1c8fa5640 1 -- 172.21.15.67:0/3526868440 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1c0031830 con 0x7fc1c415bf70 2024-03-24T00:48:59.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.406+0000 7fc1c8fa5640 1 -- 172.21.15.67:0/3526868440 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c0031be0 con 0x7fc1c415bf70 2024-03-24T00:48:59.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.406+0000 7fc1cc20b640 1 -- 172.21.15.67:0/3526868440 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc1c415bf70 msgr2=0x7fc1c40cc690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:59.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.406+0000 7fc1cc20b640 1 --2- 172.21.15.67:0/3526868440 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc1c415bf70 0x7fc1c40cc690 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fc1c00097a0 tx=0x7fc1c002b8d0 comp rx=0 tx=0).stop 2024-03-24T00:48:59.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.406+0000 7fc1cc20b640 1 -- 172.21.15.67:0/3526868440 shutdown_connections 2024-03-24T00:48:59.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.406+0000 7fc1cc20b640 1 --2- 172.21.15.67:0/3526868440 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc1c40ce930 0x7fc1c40b1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:59.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.406+0000 7fc1cc20b640 1 --2- 172.21.15.67:0/3526868440 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc1c415bf70 0x7fc1c40cc690 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:59.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.406+0000 7fc1cc20b640 1 -- 172.21.15.67:0/3526868440 >> 172.21.15.67:0/3526868440 conn(0x7fc1c40d1bf0 msgr2=0x7fc1c40d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:59.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.406+0000 7fc1cc20b640 1 -- 172.21.15.67:0/3526868440 shutdown_connections 2024-03-24T00:48:59.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.406+0000 7fc1cc20b640 1 -- 172.21.15.67:0/3526868440 wait complete. 2024-03-24T00:48:59.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.407+0000 7fc1cc20b640 1 Processor -- start 2024-03-24T00:48:59.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.407+0000 7fc1cc20b640 1 -- start start 2024-03-24T00:48:59.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.408+0000 7fc1cc20b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc1c40ce930 0x7fc1c41f4a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:59.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.408+0000 7fc1cc20b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1c40cfc70 0x7fc1c41f1b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:59.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.408+0000 7fc1cc20b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc1c415bf70 0x7fc1c40c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:59.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.408+0000 7fc1cc20b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c40ccd50 con 0x7fc1c40cfc70 2024-03-24T00:48:59.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.408+0000 7fc1cc20b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c40ccbd0 con 0x7fc1c40ce930 2024-03-24T00:48:59.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.408+0000 7fc1cc20b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c40b2210 con 0x7fc1c415bf70 2024-03-24T00:48:59.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.408+0000 7fc1c97a6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1c40cfc70 0x7fc1c41f1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:59.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.408+0000 7fc1c97a6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1c40cfc70 0x7fc1c41f1b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57500/0 (socket says 172.21.15.67:57500) 2024-03-24T00:48:59.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.408+0000 7fc1c97a6640 1 -- 172.21.15.67:0/4082047815 learned_addr learned my addr 172.21.15.67:0/4082047815 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:48:59.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.408+0000 7fc1c9fa7640 1 --2- 172.21.15.67:0/4082047815 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc1c40ce930 0x7fc1c41f4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:59.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.408+0000 7fc1ca7a8640 1 --2- 172.21.15.67:0/4082047815 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc1c415bf70 0x7fc1c40c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:59.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.409+0000 7fc1c97a6640 1 -- 172.21.15.67:0/4082047815 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc1c415bf70 msgr2=0x7fc1c40c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:59.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.409+0000 7fc1c97a6640 1 --2- 172.21.15.67:0/4082047815 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc1c415bf70 0x7fc1c40c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:59.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.409+0000 7fc1c97a6640 1 -- 172.21.15.67:0/4082047815 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc1c40ce930 msgr2=0x7fc1c41f4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:59.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.409+0000 7fc1c97a6640 1 --2- 172.21.15.67:0/4082047815 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc1c40ce930 0x7fc1c41f4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:59.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.409+0000 7fc1c97a6640 1 -- 172.21.15.67:0/4082047815 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c40c9d70 con 0x7fc1c40cfc70 2024-03-24T00:48:59.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.409+0000 7fc1c97a6640 1 --2- 172.21.15.67:0/4082047815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1c40cfc70 0x7fc1c41f1b20 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7fc1b400c400 tx=0x7fc1b400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:59.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.409+0000 7fc1ba7fc640 1 -- 172.21.15.67:0/4082047815 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b4017070 con 0x7fc1c40cfc70 2024-03-24T00:48:59.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.409+0000 7fc1ba7fc640 1 -- 172.21.15.67:0/4082047815 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1b4005d00 con 0x7fc1c40cfc70 2024-03-24T00:48:59.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.409+0000 7fc1cc20b640 1 -- 172.21.15.67:0/4082047815 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1c40c9200 con 0x7fc1c40cfc70 2024-03-24T00:48:59.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.409+0000 7fc1ba7fc640 1 -- 172.21.15.67:0/4082047815 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b4007310 con 0x7fc1c40cfc70 2024-03-24T00:48:59.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.409+0000 7fc1cc20b640 1 -- 172.21.15.67:0/4082047815 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1c40c97b0 con 0x7fc1c40cfc70 2024-03-24T00:48:59.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.410+0000 7fc1cc20b640 1 -- 172.21.15.67:0/4082047815 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1c404f400 con 0x7fc1c40cfc70 2024-03-24T00:48:59.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.410+0000 7fc1ba7fc640 1 -- 172.21.15.67:0/4082047815 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc1b401a070 con 0x7fc1c40cfc70 2024-03-24T00:48:59.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.411+0000 7fc1ba7fc640 1 --2- 172.21.15.67:0/4082047815 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc1a806bbd0 0x7fc1a806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:59.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.411+0000 7fc1ba7fc640 1 -- 172.21.15.67:0/4082047815 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc1b4089f40 con 0x7fc1c40cfc70 2024-03-24T00:48:59.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.411+0000 7fc1c9fa7640 1 --2- 172.21.15.67:0/4082047815 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc1a806bbd0 0x7fc1a806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:48:59.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.412+0000 7fc1c9fa7640 1 --2- 172.21.15.67:0/4082047815 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc1a806bbd0 0x7fc1a806e090 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fc1c0036000 tx=0x7fc1c0008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:48:59.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.415+0000 7fc1ba7fc640 1 -- 172.21.15.67:0/4082047815 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc1b40582e0 con 0x7fc1c40cfc70 2024-03-24T00:48:59.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.593+0000 7fc1cc20b640 1 -- 172.21.15.67:0/4082047815 --> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc1c40b0a40 con 0x7fc1a806bbd0 2024-03-24T00:48:59.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.600+0000 7fc1ba7fc640 1 -- 172.21.15.67:0/4082047815 <== mgr.4105 v2:172.21.15.72:6832/2768292274 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252488 (secure 0 0 0) 0x7fc1c0086a60 con 0x7fc1a806bbd0 2024-03-24T00:48:59.602 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:48:59.604 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-24T00:48:59.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.603+0000 7fc1cc20b640 1 -- 172.21.15.67:0/4082047815 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc1a806bbd0 msgr2=0x7fc1a806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:59.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.603+0000 7fc1cc20b640 1 --2- 172.21.15.67:0/4082047815 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc1a806bbd0 0x7fc1a806e090 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fc1c0036000 tx=0x7fc1c0008030 comp rx=0 tx=0).stop 2024-03-24T00:48:59.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.603+0000 7fc1cc20b640 1 -- 172.21.15.67:0/4082047815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1c40cfc70 msgr2=0x7fc1c41f1b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:48:59.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.603+0000 7fc1cc20b640 1 --2- 172.21.15.67:0/4082047815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1c40cfc70 0x7fc1c41f1b20 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7fc1b400c400 tx=0x7fc1b400c8c0 comp rx=0 tx=0).stop 2024-03-24T00:48:59.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.603+0000 7fc1cc20b640 1 -- 172.21.15.67:0/4082047815 shutdown_connections 2024-03-24T00:48:59.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.603+0000 7fc1cc20b640 1 --2- 172.21.15.67:0/4082047815 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc1a806bbd0 0x7fc1a806e090 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:59.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.603+0000 7fc1cc20b640 1 --2- 172.21.15.67:0/4082047815 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc1c415bf70 0x7fc1c40c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:59.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.603+0000 7fc1cc20b640 1 --2- 172.21.15.67:0/4082047815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1c40cfc70 0x7fc1c41f1b20 unknown :-1 s=CLOSED pgs=1723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:59.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.603+0000 7fc1cc20b640 1 --2- 172.21.15.67:0/4082047815 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc1c40ce930 0x7fc1c41f4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:59.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.603+0000 7fc1cc20b640 1 -- 172.21.15.67:0/4082047815 >> 172.21.15.67:0/4082047815 conn(0x7fc1c40d1bf0 msgr2=0x7fc1c4070580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:48:59.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.603+0000 7fc1cc20b640 1 -- 172.21.15.67:0/4082047815 shutdown_connections 2024-03-24T00:48:59.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:48:59.603+0000 7fc1cc20b640 1 -- 172.21.15.67:0/4082047815 wait complete. 2024-03-24T00:48:59.613 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":423,"stamp":"2024-03-24T00:48:59.469284+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":262344,"kb_used_data":12904,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754712376,"statfs":{"total":773094113280,"available":772825473024,"internally_reserved":0,"allocated":13213696,"data_stored":6781772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001159"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.421413+0000","last_change":"2024-03-24T00:48:55.004268+0000","last_active":"2024-03-24T00:48:56.421413+0000","last_peered":"2024-03-24T00:48:56.421413+0000","last_clean":"2024-03-24T00:48:56.421413+0000","last_became_active":"2024-03-24T00:48:55.003913+0000","last_became_peered":"2024-03-24T00:48:55.003913+0000","last_unstale":"2024-03-24T00:48:56.421413+0000","last_undegraded":"2024-03-24T00:48:56.421413+0000","last_fullsized":"2024-03-24T00:48:56.421413+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:26:40.620531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420338+0000","last_change":"2024-03-24T00:48:54.999852+0000","last_active":"2024-03-24T00:48:56.420338+0000","last_peered":"2024-03-24T00:48:56.420338+0000","last_clean":"2024-03-24T00:48:56.420338+0000","last_became_active":"2024-03-24T00:48:54.999575+0000","last_became_peered":"2024-03-24T00:48:54.999575+0000","last_unstale":"2024-03-24T00:48:56.420338+0000","last_undegraded":"2024-03-24T00:48:56.420338+0000","last_fullsized":"2024-03-24T00:48:56.420338+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:57:11.627849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420528+0000","last_change":"2024-03-24T00:48:54.995387+0000","last_active":"2024-03-24T00:48:56.420528+0000","last_peered":"2024-03-24T00:48:56.420528+0000","last_clean":"2024-03-24T00:48:56.420528+0000","last_became_active":"2024-03-24T00:48:54.994895+0000","last_became_peered":"2024-03-24T00:48:54.994895+0000","last_unstale":"2024-03-24T00:48:56.420528+0000","last_undegraded":"2024-03-24T00:48:56.420528+0000","last_fullsized":"2024-03-24T00:48:56.420528+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:33:03.600384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.364218+0000","last_change":"2024-03-24T00:48:55.006393+0000","last_active":"2024-03-24T00:48:56.364218+0000","last_peered":"2024-03-24T00:48:56.364218+0000","last_clean":"2024-03-24T00:48:56.364218+0000","last_became_active":"2024-03-24T00:48:55.005929+0000","last_became_peered":"2024-03-24T00:48:55.005929+0000","last_unstale":"2024-03-24T00:48:56.364218+0000","last_undegraded":"2024-03-24T00:48:56.364218+0000","last_fullsized":"2024-03-24T00:48:56.364218+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:49:00.743606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420894+0000","last_change":"2024-03-24T00:48:55.007877+0000","last_active":"2024-03-24T00:48:56.420894+0000","last_peered":"2024-03-24T00:48:56.420894+0000","last_clean":"2024-03-24T00:48:56.420894+0000","last_became_active":"2024-03-24T00:48:55.007552+0000","last_became_peered":"2024-03-24T00:48:55.007552+0000","last_unstale":"2024-03-24T00:48:56.420894+0000","last_undegraded":"2024-03-24T00:48:56.420894+0000","last_fullsized":"2024-03-24T00:48:56.420894+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:21:22.716643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.419988+0000","last_change":"2024-03-24T00:48:54.992733+0000","last_active":"2024-03-24T00:48:56.419988+0000","last_peered":"2024-03-24T00:48:56.419988+0000","last_clean":"2024-03-24T00:48:56.419988+0000","last_became_active":"2024-03-24T00:48:54.992479+0000","last_became_peered":"2024-03-24T00:48:54.992479+0000","last_unstale":"2024-03-24T00:48:56.419988+0000","last_undegraded":"2024-03-24T00:48:56.419988+0000","last_fullsized":"2024-03-24T00:48:56.419988+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:57:07.482366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.419674+0000","last_change":"2024-03-24T00:48:54.994860+0000","last_active":"2024-03-24T00:48:56.419674+0000","last_peered":"2024-03-24T00:48:56.419674+0000","last_clean":"2024-03-24T00:48:56.419674+0000","last_became_active":"2024-03-24T00:48:54.994481+0000","last_became_peered":"2024-03-24T00:48:54.994481+0000","last_unstale":"2024-03-24T00:48:56.419674+0000","last_undegraded":"2024-03-24T00:48:56.419674+0000","last_fullsized":"2024-03-24T00:48:56.419674+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:21:45.017588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.419778+0000","last_change":"2024-03-24T00:48:54.988130+0000","last_active":"2024-03-24T00:48:56.419778+0000","last_peered":"2024-03-24T00:48:56.419778+0000","last_clean":"2024-03-24T00:48:56.419778+0000","last_became_active":"2024-03-24T00:48:54.987865+0000","last_became_peered":"2024-03-24T00:48:54.987865+0000","last_unstale":"2024-03-24T00:48:56.419778+0000","last_undegraded":"2024-03-24T00:48:56.419778+0000","last_fullsized":"2024-03-24T00:48:56.419778+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:57:22.250932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.364452+0000","last_change":"2024-03-24T00:48:55.012922+0000","last_active":"2024-03-24T00:48:56.364452+0000","last_peered":"2024-03-24T00:48:56.364452+0000","last_clean":"2024-03-24T00:48:56.364452+0000","last_became_active":"2024-03-24T00:48:55.012673+0000","last_became_peered":"2024-03-24T00:48:55.012673+0000","last_unstale":"2024-03-24T00:48:56.364452+0000","last_undegraded":"2024-03-24T00:48:56.364452+0000","last_fullsized":"2024-03-24T00:48:56.364452+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:31:13.279647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:55.998290+0000","last_change":"2024-03-24T00:48:54.992179+0000","last_active":"2024-03-24T00:48:55.998290+0000","last_peered":"2024-03-24T00:48:55.998290+0000","last_clean":"2024-03-24T00:48:55.998290+0000","last_became_active":"2024-03-24T00:48:54.991916+0000","last_became_peered":"2024-03-24T00:48:54.991916+0000","last_unstale":"2024-03-24T00:48:55.998290+0000","last_undegraded":"2024-03-24T00:48:55.998290+0000","last_fullsized":"2024-03-24T00:48:55.998290+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:48:09.727445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420848+0000","last_change":"2024-03-24T00:48:54.998698+0000","last_active":"2024-03-24T00:48:56.420848+0000","last_peered":"2024-03-24T00:48:56.420848+0000","last_clean":"2024-03-24T00:48:56.420848+0000","last_became_active":"2024-03-24T00:48:54.998394+0000","last_became_peered":"2024-03-24T00:48:54.998394+0000","last_unstale":"2024-03-24T00:48:56.420848+0000","last_undegraded":"2024-03-24T00:48:56.420848+0000","last_fullsized":"2024-03-24T00:48:56.420848+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:29:08.295849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420272+0000","last_change":"2024-03-24T00:48:55.004680+0000","last_active":"2024-03-24T00:48:56.420272+0000","last_peered":"2024-03-24T00:48:56.420272+0000","last_clean":"2024-03-24T00:48:56.420272+0000","last_became_active":"2024-03-24T00:48:55.004371+0000","last_became_peered":"2024-03-24T00:48:55.004371+0000","last_unstale":"2024-03-24T00:48:56.420272+0000","last_undegraded":"2024-03-24T00:48:56.420272+0000","last_fullsized":"2024-03-24T00:48:56.420272+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:27:26.283205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:55.997967+0000","last_change":"2024-03-24T00:48:54.997783+0000","last_active":"2024-03-24T00:48:55.997967+0000","last_peered":"2024-03-24T00:48:55.997967+0000","last_clean":"2024-03-24T00:48:55.997967+0000","last_became_active":"2024-03-24T00:48:54.997235+0000","last_became_peered":"2024-03-24T00:48:54.997235+0000","last_unstale":"2024-03-24T00:48:55.997967+0000","last_undegraded":"2024-03-24T00:48:55.997967+0000","last_fullsized":"2024-03-24T00:48:55.997967+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:26:33.324100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420123+0000","last_change":"2024-03-24T00:48:54.990561+0000","last_active":"2024-03-24T00:48:56.420123+0000","last_peered":"2024-03-24T00:48:56.420123+0000","last_clean":"2024-03-24T00:48:56.420123+0000","last_became_active":"2024-03-24T00:48:54.990287+0000","last_became_peered":"2024-03-24T00:48:54.990287+0000","last_unstale":"2024-03-24T00:48:56.420123+0000","last_undegraded":"2024-03-24T00:48:56.420123+0000","last_fullsized":"2024-03-24T00:48:56.420123+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:15:44.739583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420799+0000","last_change":"2024-03-24T00:48:55.000100+0000","last_active":"2024-03-24T00:48:56.420799+0000","last_peered":"2024-03-24T00:48:56.420799+0000","last_clean":"2024-03-24T00:48:56.420799+0000","last_became_active":"2024-03-24T00:48:54.999462+0000","last_became_peered":"2024-03-24T00:48:54.999462+0000","last_unstale":"2024-03-24T00:48:56.420799+0000","last_undegraded":"2024-03-24T00:48:56.420799+0000","last_fullsized":"2024-03-24T00:48:56.420799+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:33.956635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.419638+0000","last_change":"2024-03-24T00:48:54.995544+0000","last_active":"2024-03-24T00:48:56.419638+0000","last_peered":"2024-03-24T00:48:56.419638+0000","last_clean":"2024-03-24T00:48:56.419638+0000","last_became_active":"2024-03-24T00:48:54.995254+0000","last_became_peered":"2024-03-24T00:48:54.995254+0000","last_unstale":"2024-03-24T00:48:56.419638+0000","last_undegraded":"2024-03-24T00:48:56.419638+0000","last_fullsized":"2024-03-24T00:48:56.419638+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:12:57.852089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:55.998271+0000","last_change":"2024-03-24T00:48:54.999212+0000","last_active":"2024-03-24T00:48:55.998271+0000","last_peered":"2024-03-24T00:48:55.998271+0000","last_clean":"2024-03-24T00:48:55.998271+0000","last_became_active":"2024-03-24T00:48:54.998297+0000","last_became_peered":"2024-03-24T00:48:54.998297+0000","last_unstale":"2024-03-24T00:48:55.998271+0000","last_undegraded":"2024-03-24T00:48:55.998271+0000","last_fullsized":"2024-03-24T00:48:55.998271+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:21:34.837676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420484+0000","last_change":"2024-03-24T00:48:54.999152+0000","last_active":"2024-03-24T00:48:56.420484+0000","last_peered":"2024-03-24T00:48:56.420484+0000","last_clean":"2024-03-24T00:48:56.420484+0000","last_became_active":"2024-03-24T00:48:54.998859+0000","last_became_peered":"2024-03-24T00:48:54.998859+0000","last_unstale":"2024-03-24T00:48:56.420484+0000","last_undegraded":"2024-03-24T00:48:56.420484+0000","last_fullsized":"2024-03-24T00:48:56.420484+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:15:16.407168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.421223+0000","last_change":"2024-03-24T00:48:55.001267+0000","last_active":"2024-03-24T00:48:56.421223+0000","last_peered":"2024-03-24T00:48:56.421223+0000","last_clean":"2024-03-24T00:48:56.421223+0000","last_became_active":"2024-03-24T00:48:55.000987+0000","last_became_peered":"2024-03-24T00:48:55.000987+0000","last_unstale":"2024-03-24T00:48:56.421223+0000","last_undegraded":"2024-03-24T00:48:56.421223+0000","last_fullsized":"2024-03-24T00:48:56.421223+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:12:35.490886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:55.999080+0000","last_change":"2024-03-24T00:48:54.995352+0000","last_active":"2024-03-24T00:48:55.999080+0000","last_peered":"2024-03-24T00:48:55.999080+0000","last_clean":"2024-03-24T00:48:55.999080+0000","last_became_active":"2024-03-24T00:48:54.995048+0000","last_became_peered":"2024-03-24T00:48:54.995048+0000","last_unstale":"2024-03-24T00:48:55.999080+0000","last_undegraded":"2024-03-24T00:48:55.999080+0000","last_fullsized":"2024-03-24T00:48:55.999080+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:19:07.036687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.364446+0000","last_change":"2024-03-24T00:48:54.999294+0000","last_active":"2024-03-24T00:48:56.364446+0000","last_peered":"2024-03-24T00:48:56.364446+0000","last_clean":"2024-03-24T00:48:56.364446+0000","last_became_active":"2024-03-24T00:48:54.998655+0000","last_became_peered":"2024-03-24T00:48:54.998655+0000","last_unstale":"2024-03-24T00:48:56.364446+0000","last_undegraded":"2024-03-24T00:48:56.364446+0000","last_fullsized":"2024-03-24T00:48:56.364446+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:20:56.647531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.421231+0000","last_change":"2024-03-24T00:48:55.012977+0000","last_active":"2024-03-24T00:48:56.421231+0000","last_peered":"2024-03-24T00:48:56.421231+0000","last_clean":"2024-03-24T00:48:56.421231+0000","last_became_active":"2024-03-24T00:48:55.012648+0000","last_became_peered":"2024-03-24T00:48:55.012648+0000","last_unstale":"2024-03-24T00:48:56.421231+0000","last_undegraded":"2024-03-24T00:48:56.421231+0000","last_fullsized":"2024-03-24T00:48:56.421231+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:32:34.044889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.364223+0000","last_change":"2024-03-24T00:48:55.010392+0000","last_active":"2024-03-24T00:48:56.364223+0000","last_peered":"2024-03-24T00:48:56.364223+0000","last_clean":"2024-03-24T00:48:56.364223+0000","last_became_active":"2024-03-24T00:48:55.010108+0000","last_became_peered":"2024-03-24T00:48:55.010108+0000","last_unstale":"2024-03-24T00:48:56.364223+0000","last_undegraded":"2024-03-24T00:48:56.364223+0000","last_fullsized":"2024-03-24T00:48:56.364223+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:34:54.718836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.001994+0000","last_change":"2024-03-24T00:48:55.002573+0000","last_active":"2024-03-24T00:48:56.001994+0000","last_peered":"2024-03-24T00:48:56.001994+0000","last_clean":"2024-03-24T00:48:56.001994+0000","last_became_active":"2024-03-24T00:48:55.002046+0000","last_became_peered":"2024-03-24T00:48:55.002046+0000","last_unstale":"2024-03-24T00:48:56.001994+0000","last_undegraded":"2024-03-24T00:48:56.001994+0000","last_fullsized":"2024-03-24T00:48:56.001994+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:21:34.837676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.002783+0000","last_change":"2024-03-24T00:48:55.005021+0000","last_active":"2024-03-24T00:48:56.002783+0000","last_peered":"2024-03-24T00:48:56.002783+0000","last_clean":"2024-03-24T00:48:56.002783+0000","last_became_active":"2024-03-24T00:48:55.004687+0000","last_became_peered":"2024-03-24T00:48:55.004687+0000","last_unstale":"2024-03-24T00:48:56.002783+0000","last_undegraded":"2024-03-24T00:48:56.002783+0000","last_fullsized":"2024-03-24T00:48:56.002783+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:49:00.743606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:55.999401+0000","last_change":"2024-03-24T00:48:55.007440+0000","last_active":"2024-03-24T00:48:55.999401+0000","last_peered":"2024-03-24T00:48:55.999401+0000","last_clean":"2024-03-24T00:48:55.999401+0000","last_became_active":"2024-03-24T00:48:55.007147+0000","last_became_peered":"2024-03-24T00:48:55.007147+0000","last_unstale":"2024-03-24T00:48:55.999401+0000","last_undegraded":"2024-03-24T00:48:55.999401+0000","last_fullsized":"2024-03-24T00:48:55.999401+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:23:29.268243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.421388+0000","last_change":"2024-03-24T00:48:55.000536+0000","last_active":"2024-03-24T00:48:56.421388+0000","last_peered":"2024-03-24T00:48:56.421388+0000","last_clean":"2024-03-24T00:48:56.421388+0000","last_became_active":"2024-03-24T00:48:55.000274+0000","last_became_peered":"2024-03-24T00:48:55.000274+0000","last_unstale":"2024-03-24T00:48:56.421388+0000","last_undegraded":"2024-03-24T00:48:56.421388+0000","last_fullsized":"2024-03-24T00:48:56.421388+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:14:53.553406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420398+0000","last_change":"2024-03-24T00:48:54.996234+0000","last_active":"2024-03-24T00:48:56.420398+0000","last_peered":"2024-03-24T00:48:56.420398+0000","last_clean":"2024-03-24T00:48:56.420398+0000","last_became_active":"2024-03-24T00:48:54.995959+0000","last_became_peered":"2024-03-24T00:48:54.995959+0000","last_unstale":"2024-03-24T00:48:56.420398+0000","last_undegraded":"2024-03-24T00:48:56.420398+0000","last_fullsized":"2024-03-24T00:48:56.420398+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:50:43.626322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:55.999391+0000","last_change":"2024-03-24T00:48:54.995982+0000","last_active":"2024-03-24T00:48:55.999391+0000","last_peered":"2024-03-24T00:48:55.999391+0000","last_clean":"2024-03-24T00:48:55.999391+0000","last_became_active":"2024-03-24T00:48:54.995699+0000","last_became_peered":"2024-03-24T00:48:54.995699+0000","last_unstale":"2024-03-24T00:48:55.999391+0000","last_undegraded":"2024-03-24T00:48:55.999391+0000","last_fullsized":"2024-03-24T00:48:55.999391+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:21:50.976426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420264+0000","last_change":"2024-03-24T00:48:55.006635+0000","last_active":"2024-03-24T00:48:56.420264+0000","last_peered":"2024-03-24T00:48:56.420264+0000","last_clean":"2024-03-24T00:48:56.420264+0000","last_became_active":"2024-03-24T00:48:55.006319+0000","last_became_peered":"2024-03-24T00:48:55.006319+0000","last_unstale":"2024-03-24T00:48:56.420264+0000","last_undegraded":"2024-03-24T00:48:56.420264+0000","last_fullsized":"2024-03-24T00:48:56.420264+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:37:40.578548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.002141+0000","last_change":"2024-03-24T00:48:54.996797+0000","last_active":"2024-03-24T00:48:56.002141+0000","last_peered":"2024-03-24T00:48:56.002141+0000","last_clean":"2024-03-24T00:48:56.002141+0000","last_became_active":"2024-03-24T00:48:54.995962+0000","last_became_peered":"2024-03-24T00:48:54.995962+0000","last_unstale":"2024-03-24T00:48:56.002141+0000","last_undegraded":"2024-03-24T00:48:56.002141+0000","last_fullsized":"2024-03-24T00:48:56.002141+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:48:56.876244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.002105+0000","last_change":"2024-03-24T00:48:55.002328+0000","last_active":"2024-03-24T00:48:56.002105+0000","last_peered":"2024-03-24T00:48:56.002105+0000","last_clean":"2024-03-24T00:48:56.002105+0000","last_became_active":"2024-03-24T00:48:55.001687+0000","last_became_peered":"2024-03-24T00:48:55.001687+0000","last_unstale":"2024-03-24T00:48:56.002105+0000","last_undegraded":"2024-03-24T00:48:56.002105+0000","last_fullsized":"2024-03-24T00:48:56.002105+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:48:09.727445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:55.998379+0000","last_change":"2024-03-24T00:48:54.995323+0000","last_active":"2024-03-24T00:48:55.998379+0000","last_peered":"2024-03-24T00:48:55.998379+0000","last_clean":"2024-03-24T00:48:55.998379+0000","last_became_active":"2024-03-24T00:48:54.995035+0000","last_became_peered":"2024-03-24T00:48:54.995035+0000","last_unstale":"2024-03-24T00:48:55.998379+0000","last_undegraded":"2024-03-24T00:48:55.998379+0000","last_fullsized":"2024-03-24T00:48:55.998379+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:48:56.876244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1f","version":"119'4","reported_seq":18,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.421388+0000","last_change":"2024-03-24T00:48:53.992874+0000","last_active":"2024-03-24T00:48:56.421388+0000","last_peered":"2024-03-24T00:48:56.421388+0000","last_clean":"2024-03-24T00:48:56.421388+0000","last_became_active":"2024-03-24T00:48:53.992507+0000","last_became_peered":"2024-03-24T00:48:53.992507+0000","last_unstale":"2024-03-24T00:48:56.421388+0000","last_undegraded":"2024-03-24T00:48:56.421388+0000","last_fullsized":"2024-03-24T00:48:56.421388+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:21:59.631832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420115+0000","last_change":"2024-03-24T00:48:54.991001+0000","last_active":"2024-03-24T00:48:56.420115+0000","last_peered":"2024-03-24T00:48:56.420115+0000","last_clean":"2024-03-24T00:48:56.420115+0000","last_became_active":"2024-03-24T00:48:54.990700+0000","last_became_peered":"2024-03-24T00:48:54.990700+0000","last_unstale":"2024-03-24T00:48:56.420115+0000","last_undegraded":"2024-03-24T00:48:56.420115+0000","last_fullsized":"2024-03-24T00:48:56.420115+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:58:43.903904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1e","version":"119'1","reported_seq":15,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:55.998384+0000","last_change":"2024-03-24T00:48:53.982887+0000","last_active":"2024-03-24T00:48:55.998384+0000","last_peered":"2024-03-24T00:48:55.998384+0000","last_clean":"2024-03-24T00:48:55.998384+0000","last_became_active":"2024-03-24T00:48:53.982515+0000","last_became_peered":"2024-03-24T00:48:53.982515+0000","last_unstale":"2024-03-24T00:48:55.998384+0000","last_undegraded":"2024-03-24T00:48:55.998384+0000","last_fullsized":"2024-03-24T00:48:55.998384+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:29:07.293730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.002283+0000","last_change":"2024-03-24T00:48:55.005866+0000","last_active":"2024-03-24T00:48:56.002283+0000","last_peered":"2024-03-24T00:48:56.002283+0000","last_clean":"2024-03-24T00:48:56.002283+0000","last_became_active":"2024-03-24T00:48:55.004659+0000","last_became_peered":"2024-03-24T00:48:55.004659+0000","last_unstale":"2024-03-24T00:48:56.002283+0000","last_undegraded":"2024-03-24T00:48:56.002283+0000","last_fullsized":"2024-03-24T00:48:56.002283+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:43:34.664046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"119'2","reported_seq":16,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.364266+0000","last_change":"2024-03-24T00:48:53.993536+0000","last_active":"2024-03-24T00:48:56.364266+0000","last_peered":"2024-03-24T00:48:56.364266+0000","last_clean":"2024-03-24T00:48:56.364266+0000","last_became_active":"2024-03-24T00:48:53.993136+0000","last_became_peered":"2024-03-24T00:48:53.993136+0000","last_unstale":"2024-03-24T00:48:56.364266+0000","last_undegraded":"2024-03-24T00:48:56.364266+0000","last_fullsized":"2024-03-24T00:48:56.364266+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:07:44.083528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.364481+0000","last_change":"2024-03-24T00:48:55.005809+0000","last_active":"2024-03-24T00:48:56.364481+0000","last_peered":"2024-03-24T00:48:56.364481+0000","last_clean":"2024-03-24T00:48:56.364481+0000","last_became_active":"2024-03-24T00:48:55.005520+0000","last_became_peered":"2024-03-24T00:48:55.005520+0000","last_unstale":"2024-03-24T00:48:56.364481+0000","last_undegraded":"2024-03-24T00:48:56.364481+0000","last_fullsized":"2024-03-24T00:48:56.364481+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:12:42.264405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"119'2","reported_seq":16,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:55.998505+0000","last_change":"2024-03-24T00:48:53.993401+0000","last_active":"2024-03-24T00:48:55.998505+0000","last_peered":"2024-03-24T00:48:55.998505+0000","last_clean":"2024-03-24T00:48:55.998505+0000","last_became_active":"2024-03-24T00:48:53.991401+0000","last_became_peered":"2024-03-24T00:48:53.991401+0000","last_unstale":"2024-03-24T00:48:55.998505+0000","last_undegraded":"2024-03-24T00:48:55.998505+0000","last_fullsized":"2024-03-24T00:48:55.998505+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:21:21.714524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:55.997959+0000","last_change":"2024-03-24T00:48:55.001388+0000","last_active":"2024-03-24T00:48:55.997959+0000","last_peered":"2024-03-24T00:48:55.997959+0000","last_clean":"2024-03-24T00:48:55.997959+0000","last_became_active":"2024-03-24T00:48:55.000323+0000","last_became_peered":"2024-03-24T00:48:55.000323+0000","last_unstale":"2024-03-24T00:48:55.997959+0000","last_undegraded":"2024-03-24T00:48:55.997959+0000","last_fullsized":"2024-03-24T00:48:55.997959+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:08:25.466057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"119'1","reported_seq":15,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.421024+0000","last_change":"2024-03-24T00:48:53.994950+0000","last_active":"2024-03-24T00:48:56.421024+0000","last_peered":"2024-03-24T00:48:56.421024+0000","last_clean":"2024-03-24T00:48:56.421024+0000","last_became_active":"2024-03-24T00:48:53.994709+0000","last_became_peered":"2024-03-24T00:48:53.994709+0000","last_unstale":"2024-03-24T00:48:56.421024+0000","last_undegraded":"2024-03-24T00:48:56.421024+0000","last_fullsized":"2024-03-24T00:48:56.421024+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:14:11.273118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420634+0000","last_change":"2024-03-24T00:48:55.000402+0000","last_active":"2024-03-24T00:48:56.420634+0000","last_peered":"2024-03-24T00:48:56.420634+0000","last_clean":"2024-03-24T00:48:56.420634+0000","last_became_active":"2024-03-24T00:48:55.000100+0000","last_became_peered":"2024-03-24T00:48:55.000100+0000","last_unstale":"2024-03-24T00:48:56.420634+0000","last_undegraded":"2024-03-24T00:48:56.420634+0000","last_fullsized":"2024-03-24T00:48:56.420634+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:14:53.553406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"119'1","reported_seq":15,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420089+0000","last_change":"2024-03-24T00:48:53.997359+0000","last_active":"2024-03-24T00:48:56.420089+0000","last_peered":"2024-03-24T00:48:56.420089+0000","last_clean":"2024-03-24T00:48:56.420089+0000","last_became_active":"2024-03-24T00:48:53.997039+0000","last_became_peered":"2024-03-24T00:48:53.997039+0000","last_unstale":"2024-03-24T00:48:56.420089+0000","last_undegraded":"2024-03-24T00:48:56.420089+0000","last_fullsized":"2024-03-24T00:48:56.420089+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:12:22.878628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420408+0000","last_change":"2024-03-24T00:48:55.001833+0000","last_active":"2024-03-24T00:48:56.420408+0000","last_peered":"2024-03-24T00:48:56.420408+0000","last_clean":"2024-03-24T00:48:56.420408+0000","last_became_active":"2024-03-24T00:48:55.000964+0000","last_became_peered":"2024-03-24T00:48:55.000964+0000","last_unstale":"2024-03-24T00:48:56.420408+0000","last_undegraded":"2024-03-24T00:48:56.420408+0000","last_fullsized":"2024-03-24T00:48:56.420408+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:04:20.294834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":14,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.002235+0000","last_change":"2024-03-24T00:48:53.988155+0000","last_active":"2024-03-24T00:48:56.002235+0000","last_peered":"2024-03-24T00:48:56.002235+0000","last_clean":"2024-03-24T00:48:56.002235+0000","last_became_active":"2024-03-24T00:48:53.987659+0000","last_became_peered":"2024-03-24T00:48:53.987659+0000","last_unstale":"2024-03-24T00:48:56.002235+0000","last_undegraded":"2024-03-24T00:48:56.002235+0000","last_fullsized":"2024-03-24T00:48:56.002235+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:26:28.796609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:55.998147+0000","last_change":"2024-03-24T00:48:55.010365+0000","last_active":"2024-03-24T00:48:55.998147+0000","last_peered":"2024-03-24T00:48:55.998147+0000","last_clean":"2024-03-24T00:48:55.998147+0000","last_became_active":"2024-03-24T00:48:55.010131+0000","last_became_peered":"2024-03-24T00:48:55.010131+0000","last_unstale":"2024-03-24T00:48:55.998147+0000","last_undegraded":"2024-03-24T00:48:55.998147+0000","last_fullsized":"2024-03-24T00:48:55.998147+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:15:03.063582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":14,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.419829+0000","last_change":"2024-03-24T00:48:53.993610+0000","last_active":"2024-03-24T00:48:56.419829+0000","last_peered":"2024-03-24T00:48:56.419829+0000","last_clean":"2024-03-24T00:48:56.419829+0000","last_became_active":"2024-03-24T00:48:53.993326+0000","last_became_peered":"2024-03-24T00:48:53.993326+0000","last_unstale":"2024-03-24T00:48:56.419829+0000","last_undegraded":"2024-03-24T00:48:56.419829+0000","last_fullsized":"2024-03-24T00:48:56.419829+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:07:03.572587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420867+0000","last_change":"2024-03-24T00:48:54.999636+0000","last_active":"2024-03-24T00:48:56.420867+0000","last_peered":"2024-03-24T00:48:56.420867+0000","last_clean":"2024-03-24T00:48:56.420867+0000","last_became_active":"2024-03-24T00:48:54.999197+0000","last_became_peered":"2024-03-24T00:48:54.999197+0000","last_unstale":"2024-03-24T00:48:56.420867+0000","last_undegraded":"2024-03-24T00:48:56.420867+0000","last_fullsized":"2024-03-24T00:48:56.420867+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:34:55.898803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":14,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.421402+0000","last_change":"2024-03-24T00:48:53.999330+0000","last_active":"2024-03-24T00:48:56.421402+0000","last_peered":"2024-03-24T00:48:56.421402+0000","last_clean":"2024-03-24T00:48:56.421402+0000","last_became_active":"2024-03-24T00:48:53.999065+0000","last_became_peered":"2024-03-24T00:48:53.999065+0000","last_unstale":"2024-03-24T00:48:56.421402+0000","last_undegraded":"2024-03-24T00:48:56.421402+0000","last_fullsized":"2024-03-24T00:48:56.421402+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:14:11.273118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.002545+0000","last_change":"2024-03-24T00:48:54.994492+0000","last_active":"2024-03-24T00:48:56.002545+0000","last_peered":"2024-03-24T00:48:56.002545+0000","last_clean":"2024-03-24T00:48:56.002545+0000","last_became_active":"2024-03-24T00:48:54.994195+0000","last_became_peered":"2024-03-24T00:48:54.994195+0000","last_unstale":"2024-03-24T00:48:56.002545+0000","last_undegraded":"2024-03-24T00:48:56.002545+0000","last_fullsized":"2024-03-24T00:48:56.002545+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:34:54.718836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":14,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.002174+0000","last_change":"2024-03-24T00:48:53.985480+0000","last_active":"2024-03-24T00:48:56.002174+0000","last_peered":"2024-03-24T00:48:56.002174+0000","last_clean":"2024-03-24T00:48:56.002174+0000","last_became_active":"2024-03-24T00:48:53.985203+0000","last_became_peered":"2024-03-24T00:48:53.985203+0000","last_unstale":"2024-03-24T00:48:56.002174+0000","last_undegraded":"2024-03-24T00:48:56.002174+0000","last_fullsized":"2024-03-24T00:48:56.002174+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:37:28.523220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420636+0000","last_change":"2024-03-24T00:48:54.993477+0000","last_active":"2024-03-24T00:48:56.420636+0000","last_peered":"2024-03-24T00:48:56.420636+0000","last_clean":"2024-03-24T00:48:56.420636+0000","last_became_active":"2024-03-24T00:48:54.993207+0000","last_became_peered":"2024-03-24T00:48:54.993207+0000","last_unstale":"2024-03-24T00:48:56.420636+0000","last_undegraded":"2024-03-24T00:48:56.420636+0000","last_fullsized":"2024-03-24T00:48:56.420636+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:27:27.533944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.15","version":"119'3","reported_seq":17,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.002520+0000","last_change":"2024-03-24T00:48:53.987256+0000","last_active":"2024-03-24T00:48:56.002520+0000","last_peered":"2024-03-24T00:48:56.002520+0000","last_clean":"2024-03-24T00:48:56.002520+0000","last_became_active":"2024-03-24T00:48:53.986435+0000","last_became_peered":"2024-03-24T00:48:53.986435+0000","last_unstale":"2024-03-24T00:48:56.002520+0000","last_undegraded":"2024-03-24T00:48:56.002520+0000","last_fullsized":"2024-03-24T00:48:56.002520+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:26:39.618413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.002842+0000","last_change":"2024-03-24T00:48:55.001414+0000","last_active":"2024-03-24T00:48:56.002842+0000","last_peered":"2024-03-24T00:48:56.002842+0000","last_clean":"2024-03-24T00:48:56.002842+0000","last_became_active":"2024-03-24T00:48:55.001126+0000","last_became_peered":"2024-03-24T00:48:55.001126+0000","last_unstale":"2024-03-24T00:48:56.002842+0000","last_undegraded":"2024-03-24T00:48:56.002842+0000","last_fullsized":"2024-03-24T00:48:56.002842+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:31:13.279647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.14","version":"119'6","reported_seq":20,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420741+0000","last_change":"2024-03-24T00:48:53.992589+0000","last_active":"2024-03-24T00:48:56.420741+0000","last_peered":"2024-03-24T00:48:56.420741+0000","last_clean":"2024-03-24T00:48:56.420741+0000","last_became_active":"2024-03-24T00:48:53.992273+0000","last_became_peered":"2024-03-24T00:48:53.992273+0000","last_unstale":"2024-03-24T00:48:56.420741+0000","last_undegraded":"2024-03-24T00:48:56.420741+0000","last_fullsized":"2024-03-24T00:48:56.420741+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:21:33.835558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.421262+0000","last_change":"2024-03-24T00:48:54.987532+0000","last_active":"2024-03-24T00:48:56.421262+0000","last_peered":"2024-03-24T00:48:56.421262+0000","last_clean":"2024-03-24T00:48:56.421262+0000","last_became_active":"2024-03-24T00:48:54.987241+0000","last_became_peered":"2024-03-24T00:48:54.987241+0000","last_unstale":"2024-03-24T00:48:56.421262+0000","last_undegraded":"2024-03-24T00:48:56.421262+0000","last_fullsized":"2024-03-24T00:48:56.421262+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:31:26.868426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.13","version":"119'2","reported_seq":16,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.419984+0000","last_change":"2024-03-24T00:48:53.988410+0000","last_active":"2024-03-24T00:48:56.419984+0000","last_peered":"2024-03-24T00:48:56.419984+0000","last_clean":"2024-03-24T00:48:56.419984+0000","last_became_active":"2024-03-24T00:48:53.987901+0000","last_became_peered":"2024-03-24T00:48:53.987901+0000","last_unstale":"2024-03-24T00:48:56.419984+0000","last_undegraded":"2024-03-24T00:48:56.419984+0000","last_fullsized":"2024-03-24T00:48:56.419984+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:23:37.568996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:55.998042+0000","last_change":"2024-03-24T00:48:54.996570+0000","last_active":"2024-03-24T00:48:55.998042+0000","last_peered":"2024-03-24T00:48:55.998042+0000","last_clean":"2024-03-24T00:48:55.998042+0000","last_became_active":"2024-03-24T00:48:54.996219+0000","last_became_peered":"2024-03-24T00:48:54.996219+0000","last_unstale":"2024-03-24T00:48:55.998042+0000","last_undegraded":"2024-03-24T00:48:55.998042+0000","last_fullsized":"2024-03-24T00:48:55.998042+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:35:09.413477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":14,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420599+0000","last_change":"2024-03-24T00:48:53.988076+0000","last_active":"2024-03-24T00:48:56.420599+0000","last_peered":"2024-03-24T00:48:56.420599+0000","last_clean":"2024-03-24T00:48:56.420599+0000","last_became_active":"2024-03-24T00:48:53.987752+0000","last_became_peered":"2024-03-24T00:48:53.987752+0000","last_unstale":"2024-03-24T00:48:56.420599+0000","last_undegraded":"2024-03-24T00:48:56.420599+0000","last_fullsized":"2024-03-24T00:48:56.420599+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:48:08.725327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:55.999118+0000","last_change":"2024-03-24T00:48:55.005093+0000","last_active":"2024-03-24T00:48:55.999118+0000","last_peered":"2024-03-24T00:48:55.999118+0000","last_clean":"2024-03-24T00:48:55.999118+0000","last_became_active":"2024-03-24T00:48:55.004817+0000","last_became_peered":"2024-03-24T00:48:55.004817+0000","last_unstale":"2024-03-24T00:48:55.999118+0000","last_undegraded":"2024-03-24T00:48:55.999118+0000","last_fullsized":"2024-03-24T00:48:55.999118+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:42:49.723947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":14,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420113+0000","last_change":"2024-03-24T00:48:53.992865+0000","last_active":"2024-03-24T00:48:56.420113+0000","last_peered":"2024-03-24T00:48:56.420113+0000","last_clean":"2024-03-24T00:48:56.420113+0000","last_became_active":"2024-03-24T00:48:53.992565+0000","last_became_peered":"2024-03-24T00:48:53.992565+0000","last_unstale":"2024-03-24T00:48:56.420113+0000","last_undegraded":"2024-03-24T00:48:56.420113+0000","last_fullsized":"2024-03-24T00:48:56.420113+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:35:08.411358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.002343+0000","last_change":"2024-03-24T00:48:55.002859+0000","last_active":"2024-03-24T00:48:56.002343+0000","last_peered":"2024-03-24T00:48:56.002343+0000","last_clean":"2024-03-24T00:48:56.002343+0000","last_became_active":"2024-03-24T00:48:55.002568+0000","last_became_peered":"2024-03-24T00:48:55.002568+0000","last_unstale":"2024-03-24T00:48:56.002343+0000","last_undegraded":"2024-03-24T00:48:56.002343+0000","last_fullsized":"2024-03-24T00:48:56.002343+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:15:22.617558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"119'1","reported_seq":15,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:55.998366+0000","last_change":"2024-03-24T00:48:53.988733+0000","last_active":"2024-03-24T00:48:55.998366+0000","last_peered":"2024-03-24T00:48:55.998366+0000","last_clean":"2024-03-24T00:48:55.998366+0000","last_became_active":"2024-03-24T00:48:53.988394+0000","last_became_peered":"2024-03-24T00:48:53.988394+0000","last_unstale":"2024-03-24T00:48:55.998366+0000","last_undegraded":"2024-03-24T00:48:55.998366+0000","last_fullsized":"2024-03-24T00:48:55.998366+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:07:15.456620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.421054+0000","last_change":"2024-03-24T00:48:55.013412+0000","last_active":"2024-03-24T00:48:56.421054+0000","last_peered":"2024-03-24T00:48:56.421054+0000","last_clean":"2024-03-24T00:48:56.421054+0000","last_became_active":"2024-03-24T00:48:55.013075+0000","last_became_peered":"2024-03-24T00:48:55.013075+0000","last_unstale":"2024-03-24T00:48:56.421054+0000","last_undegraded":"2024-03-24T00:48:56.421054+0000","last_fullsized":"2024-03-24T00:48:56.421054+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:07:16.458739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"119'2","reported_seq":16,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:55.999500+0000","last_change":"2024-03-24T00:48:53.989354+0000","last_active":"2024-03-24T00:48:55.999500+0000","last_peered":"2024-03-24T00:48:55.999500+0000","last_clean":"2024-03-24T00:48:55.999500+0000","last_became_active":"2024-03-24T00:48:53.989074+0000","last_became_peered":"2024-03-24T00:48:53.989074+0000","last_unstale":"2024-03-24T00:48:55.999500+0000","last_undegraded":"2024-03-24T00:48:55.999500+0000","last_fullsized":"2024-03-24T00:48:55.999500+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:09:51.433738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420447+0000","last_change":"2024-03-24T00:48:55.002046+0000","last_active":"2024-03-24T00:48:56.420447+0000","last_peered":"2024-03-24T00:48:56.420447+0000","last_clean":"2024-03-24T00:48:56.420447+0000","last_became_active":"2024-03-24T00:48:55.001773+0000","last_became_peered":"2024-03-24T00:48:55.001773+0000","last_unstale":"2024-03-24T00:48:56.420447+0000","last_undegraded":"2024-03-24T00:48:56.420447+0000","last_fullsized":"2024-03-24T00:48:56.420447+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:25:31.343032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"119'1","reported_seq":15,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.364313+0000","last_change":"2024-03-24T00:48:53.992745+0000","last_active":"2024-03-24T00:48:56.364313+0000","last_peered":"2024-03-24T00:48:56.364313+0000","last_clean":"2024-03-24T00:48:56.364313+0000","last_became_active":"2024-03-24T00:48:53.992433+0000","last_became_peered":"2024-03-24T00:48:53.992433+0000","last_unstale":"2024-03-24T00:48:56.364313+0000","last_undegraded":"2024-03-24T00:48:56.364313+0000","last_fullsized":"2024-03-24T00:48:56.364313+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:08:24.463939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.002907+0000","last_change":"2024-03-24T00:48:55.007080+0000","last_active":"2024-03-24T00:48:56.002907+0000","last_peered":"2024-03-24T00:48:56.002907+0000","last_clean":"2024-03-24T00:48:56.002907+0000","last_became_active":"2024-03-24T00:48:55.006437+0000","last_became_peered":"2024-03-24T00:48:55.006437+0000","last_unstale":"2024-03-24T00:48:56.002907+0000","last_undegraded":"2024-03-24T00:48:56.002907+0000","last_fullsized":"2024-03-24T00:48:56.002907+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:20:56.647531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":14,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.364137+0000","last_change":"2024-03-24T00:48:53.997898+0000","last_active":"2024-03-24T00:48:56.364137+0000","last_peered":"2024-03-24T00:48:56.364137+0000","last_clean":"2024-03-24T00:48:56.364137+0000","last_became_active":"2024-03-24T00:48:53.997647+0000","last_became_peered":"2024-03-24T00:48:53.997647+0000","last_unstale":"2024-03-24T00:48:56.364137+0000","last_undegraded":"2024-03-24T00:48:56.364137+0000","last_fullsized":"2024-03-24T00:48:56.364137+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:58:19.481628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.419638+0000","last_change":"2024-03-24T00:48:54.999193+0000","last_active":"2024-03-24T00:48:56.419638+0000","last_peered":"2024-03-24T00:48:56.419638+0000","last_clean":"2024-03-24T00:48:56.419638+0000","last_became_active":"2024-03-24T00:48:54.998895+0000","last_became_peered":"2024-03-24T00:48:54.998895+0000","last_unstale":"2024-03-24T00:48:56.419638+0000","last_undegraded":"2024-03-24T00:48:56.419638+0000","last_fullsized":"2024-03-24T00:48:56.419638+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:22:30.282268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"119'2","reported_seq":16,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420210+0000","last_change":"2024-03-24T00:48:53.989088+0000","last_active":"2024-03-24T00:48:56.420210+0000","last_peered":"2024-03-24T00:48:56.420210+0000","last_clean":"2024-03-24T00:48:56.420210+0000","last_became_active":"2024-03-24T00:48:53.988812+0000","last_became_peered":"2024-03-24T00:48:53.988812+0000","last_unstale":"2024-03-24T00:48:56.420210+0000","last_undegraded":"2024-03-24T00:48:56.420210+0000","last_fullsized":"2024-03-24T00:48:56.420210+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:54:55.973101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.419960+0000","last_change":"2024-03-24T00:48:54.996761+0000","last_active":"2024-03-24T00:48:56.419960+0000","last_peered":"2024-03-24T00:48:56.419960+0000","last_clean":"2024-03-24T00:48:56.419960+0000","last_became_active":"2024-03-24T00:48:54.996464+0000","last_became_peered":"2024-03-24T00:48:54.996464+0000","last_unstale":"2024-03-24T00:48:56.419960+0000","last_undegraded":"2024-03-24T00:48:56.419960+0000","last_fullsized":"2024-03-24T00:48:56.419960+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:54:00.503746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":14,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.363977+0000","last_change":"2024-03-24T00:48:53.992661+0000","last_active":"2024-03-24T00:48:56.363977+0000","last_peered":"2024-03-24T00:48:56.363977+0000","last_clean":"2024-03-24T00:48:56.363977+0000","last_became_active":"2024-03-24T00:48:53.992347+0000","last_became_peered":"2024-03-24T00:48:53.992347+0000","last_unstale":"2024-03-24T00:48:56.363977+0000","last_undegraded":"2024-03-24T00:48:56.363977+0000","last_fullsized":"2024-03-24T00:48:56.363977+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:01:04.094765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420604+0000","last_change":"2024-03-24T00:48:54.989841+0000","last_active":"2024-03-24T00:48:56.420604+0000","last_peered":"2024-03-24T00:48:56.420604+0000","last_clean":"2024-03-24T00:48:56.420604+0000","last_became_active":"2024-03-24T00:48:54.989577+0000","last_became_peered":"2024-03-24T00:48:54.989577+0000","last_unstale":"2024-03-24T00:48:56.420604+0000","last_undegraded":"2024-03-24T00:48:56.420604+0000","last_fullsized":"2024-03-24T00:48:56.420604+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:25:52.856404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":14,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420197+0000","last_change":"2024-03-24T00:48:53.995870+0000","last_active":"2024-03-24T00:48:56.420197+0000","last_peered":"2024-03-24T00:48:56.420197+0000","last_clean":"2024-03-24T00:48:56.420197+0000","last_became_active":"2024-03-24T00:48:53.995507+0000","last_became_peered":"2024-03-24T00:48:53.995507+0000","last_unstale":"2024-03-24T00:48:56.420197+0000","last_undegraded":"2024-03-24T00:48:56.420197+0000","last_fullsized":"2024-03-24T00:48:56.420197+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:15:02.061464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.002232+0000","last_change":"2024-03-24T00:48:54.996407+0000","last_active":"2024-03-24T00:48:56.002232+0000","last_peered":"2024-03-24T00:48:56.002232+0000","last_clean":"2024-03-24T00:48:56.002232+0000","last_became_active":"2024-03-24T00:48:54.996118+0000","last_became_peered":"2024-03-24T00:48:54.996118+0000","last_unstale":"2024-03-24T00:48:56.002232+0000","last_undegraded":"2024-03-24T00:48:56.002232+0000","last_fullsized":"2024-03-24T00:48:56.002232+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:03:17.187222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":14,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.363817+0000","last_change":"2024-03-24T00:48:53.993911+0000","last_active":"2024-03-24T00:48:56.363817+0000","last_peered":"2024-03-24T00:48:56.363817+0000","last_clean":"2024-03-24T00:48:56.363817+0000","last_became_active":"2024-03-24T00:48:53.993618+0000","last_became_peered":"2024-03-24T00:48:53.993618+0000","last_unstale":"2024-03-24T00:48:56.363817+0000","last_undegraded":"2024-03-24T00:48:56.363817+0000","last_fullsized":"2024-03-24T00:48:56.363817+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:24:44.940524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420427+0000","last_change":"2024-03-24T00:48:54.994862+0000","last_active":"2024-03-24T00:48:56.420427+0000","last_peered":"2024-03-24T00:48:56.420427+0000","last_clean":"2024-03-24T00:48:56.420427+0000","last_became_active":"2024-03-24T00:48:54.994511+0000","last_became_peered":"2024-03-24T00:48:54.994511+0000","last_unstale":"2024-03-24T00:48:56.420427+0000","last_undegraded":"2024-03-24T00:48:56.420427+0000","last_fullsized":"2024-03-24T00:48:56.420427+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:54:33.905780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.7","version":"119'2","reported_seq":16,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420300+0000","last_change":"2024-03-24T00:48:53.994470+0000","last_active":"2024-03-24T00:48:56.420300+0000","last_peered":"2024-03-24T00:48:56.420300+0000","last_clean":"2024-03-24T00:48:56.420300+0000","last_became_active":"2024-03-24T00:48:53.994193+0000","last_became_peered":"2024-03-24T00:48:53.994193+0000","last_unstale":"2024-03-24T00:48:56.420300+0000","last_undegraded":"2024-03-24T00:48:56.420300+0000","last_fullsized":"2024-03-24T00:48:56.420300+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:21:24.620572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:55.998229+0000","last_change":"2024-03-24T00:48:55.004460+0000","last_active":"2024-03-24T00:48:55.998229+0000","last_peered":"2024-03-24T00:48:55.998229+0000","last_clean":"2024-03-24T00:48:55.998229+0000","last_became_active":"2024-03-24T00:48:55.003966+0000","last_became_peered":"2024-03-24T00:48:55.003966+0000","last_unstale":"2024-03-24T00:48:55.998229+0000","last_undegraded":"2024-03-24T00:48:55.998229+0000","last_fullsized":"2024-03-24T00:48:55.998229+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:03:17.187222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.6","version":"120'1","reported_seq":15,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.421551+0000","last_change":"2024-03-24T00:48:53.995779+0000","last_active":"2024-03-24T00:48:56.421551+0000","last_peered":"2024-03-24T00:48:56.421551+0000","last_clean":"2024-03-24T00:48:56.421551+0000","last_became_active":"2024-03-24T00:48:53.995516+0000","last_became_peered":"2024-03-24T00:48:53.995516+0000","last_unstale":"2024-03-24T00:48:56.421551+0000","last_undegraded":"2024-03-24T00:48:56.421551+0000","last_fullsized":"2024-03-24T00:48:56.421551+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:30:35.317515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.364522+0000","last_change":"2024-03-24T00:48:55.000436+0000","last_active":"2024-03-24T00:48:56.364522+0000","last_peered":"2024-03-24T00:48:56.364522+0000","last_clean":"2024-03-24T00:48:56.364522+0000","last_became_active":"2024-03-24T00:48:54.999859+0000","last_became_peered":"2024-03-24T00:48:54.999859+0000","last_unstale":"2024-03-24T00:48:56.364522+0000","last_undegraded":"2024-03-24T00:48:56.364522+0000","last_fullsized":"2024-03-24T00:48:56.364522+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:46:56.230850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":14,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.421402+0000","last_change":"2024-03-24T00:48:53.995776+0000","last_active":"2024-03-24T00:48:56.421402+0000","last_peered":"2024-03-24T00:48:56.421402+0000","last_clean":"2024-03-24T00:48:56.421402+0000","last_became_active":"2024-03-24T00:48:53.995494+0000","last_became_peered":"2024-03-24T00:48:53.995494+0000","last_unstale":"2024-03-24T00:48:56.421402+0000","last_undegraded":"2024-03-24T00:48:56.421402+0000","last_fullsized":"2024-03-24T00:48:56.421402+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:35:18.392188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:55.999180+0000","last_change":"2024-03-24T00:48:54.998094+0000","last_active":"2024-03-24T00:48:55.999180+0000","last_peered":"2024-03-24T00:48:55.999180+0000","last_clean":"2024-03-24T00:48:55.999180+0000","last_became_active":"2024-03-24T00:48:54.997823+0000","last_became_peered":"2024-03-24T00:48:54.997823+0000","last_unstale":"2024-03-24T00:48:55.999180+0000","last_undegraded":"2024-03-24T00:48:55.999180+0000","last_fullsized":"2024-03-24T00:48:55.999180+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:46:59.292179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":14,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420491+0000","last_change":"2024-03-24T00:48:53.996637+0000","last_active":"2024-03-24T00:48:56.420491+0000","last_peered":"2024-03-24T00:48:56.420491+0000","last_clean":"2024-03-24T00:48:56.420491+0000","last_became_active":"2024-03-24T00:48:53.996356+0000","last_became_peered":"2024-03-24T00:48:53.996356+0000","last_unstale":"2024-03-24T00:48:56.420491+0000","last_undegraded":"2024-03-24T00:48:56.420491+0000","last_fullsized":"2024-03-24T00:48:56.420491+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:04:03.246665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.419792+0000","last_change":"2024-03-24T00:48:55.001161+0000","last_active":"2024-03-24T00:48:56.419792+0000","last_peered":"2024-03-24T00:48:56.419792+0000","last_clean":"2024-03-24T00:48:56.419792+0000","last_became_active":"2024-03-24T00:48:55.000901+0000","last_became_peered":"2024-03-24T00:48:55.000901+0000","last_unstale":"2024-03-24T00:48:56.419792+0000","last_undegraded":"2024-03-24T00:48:56.419792+0000","last_fullsized":"2024-03-24T00:48:56.419792+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:48:52.719348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":14,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.421208+0000","last_change":"2024-03-24T00:48:53.998500+0000","last_active":"2024-03-24T00:48:56.421208+0000","last_peered":"2024-03-24T00:48:56.421208+0000","last_clean":"2024-03-24T00:48:56.421208+0000","last_became_active":"2024-03-24T00:48:53.998219+0000","last_became_peered":"2024-03-24T00:48:53.998219+0000","last_unstale":"2024-03-24T00:48:56.421208+0000","last_undegraded":"2024-03-24T00:48:56.421208+0000","last_fullsized":"2024-03-24T00:48:56.421208+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:30:35.317515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420109+0000","last_change":"2024-03-24T00:48:54.989178+0000","last_active":"2024-03-24T00:48:56.420109+0000","last_peered":"2024-03-24T00:48:56.420109+0000","last_clean":"2024-03-24T00:48:56.420109+0000","last_became_active":"2024-03-24T00:48:54.988884+0000","last_became_peered":"2024-03-24T00:48:54.988884+0000","last_unstale":"2024-03-24T00:48:56.420109+0000","last_undegraded":"2024-03-24T00:48:56.420109+0000","last_fullsized":"2024-03-24T00:48:56.420109+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:19:24.977405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":14,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.421336+0000","last_change":"2024-03-24T00:48:53.992750+0000","last_active":"2024-03-24T00:48:56.421336+0000","last_peered":"2024-03-24T00:48:56.421336+0000","last_clean":"2024-03-24T00:48:56.421336+0000","last_became_active":"2024-03-24T00:48:53.992433+0000","last_became_peered":"2024-03-24T00:48:53.992433+0000","last_unstale":"2024-03-24T00:48:56.421336+0000","last_undegraded":"2024-03-24T00:48:56.421336+0000","last_fullsized":"2024-03-24T00:48:56.421336+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:21:59.631832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.364100+0000","last_change":"2024-03-24T00:48:55.001650+0000","last_active":"2024-03-24T00:48:56.364100+0000","last_peered":"2024-03-24T00:48:56.364100+0000","last_clean":"2024-03-24T00:48:56.364100+0000","last_became_active":"2024-03-24T00:48:55.001336+0000","last_became_peered":"2024-03-24T00:48:55.001336+0000","last_unstale":"2024-03-24T00:48:56.364100+0000","last_undegraded":"2024-03-24T00:48:56.364100+0000","last_fullsized":"2024-03-24T00:48:56.364100+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:15:22.617558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":14,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.419686+0000","last_change":"2024-03-24T00:48:53.997216+0000","last_active":"2024-03-24T00:48:56.419686+0000","last_peered":"2024-03-24T00:48:56.419686+0000","last_clean":"2024-03-24T00:48:56.419686+0000","last_became_active":"2024-03-24T00:48:53.996916+0000","last_became_peered":"2024-03-24T00:48:53.996916+0000","last_unstale":"2024-03-24T00:48:56.419686+0000","last_undegraded":"2024-03-24T00:48:56.419686+0000","last_fullsized":"2024-03-24T00:48:56.419686+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:18:02.497937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:55.998181+0000","last_change":"2024-03-24T00:48:55.001355+0000","last_active":"2024-03-24T00:48:55.998181+0000","last_peered":"2024-03-24T00:48:55.998181+0000","last_clean":"2024-03-24T00:48:55.998181+0000","last_became_active":"2024-03-24T00:48:54.996881+0000","last_became_peered":"2024-03-24T00:48:54.996881+0000","last_unstale":"2024-03-24T00:48:55.998181+0000","last_undegraded":"2024-03-24T00:48:55.998181+0000","last_fullsized":"2024-03-24T00:48:55.998181+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:23:38.571115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":14,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.363827+0000","last_change":"2024-03-24T00:48:53.993376+0000","last_active":"2024-03-24T00:48:56.363827+0000","last_peered":"2024-03-24T00:48:56.363827+0000","last_clean":"2024-03-24T00:48:56.363827+0000","last_became_active":"2024-03-24T00:48:53.993104+0000","last_became_peered":"2024-03-24T00:48:53.993104+0000","last_unstale":"2024-03-24T00:48:56.363827+0000","last_undegraded":"2024-03-24T00:48:56.363827+0000","last_fullsized":"2024-03-24T00:48:56.363827+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:10:25.529558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.364486+0000","last_change":"2024-03-24T00:48:54.999535+0000","last_active":"2024-03-24T00:48:56.364486+0000","last_peered":"2024-03-24T00:48:56.364486+0000","last_clean":"2024-03-24T00:48:56.364486+0000","last_became_active":"2024-03-24T00:48:54.999019+0000","last_became_peered":"2024-03-24T00:48:54.999019+0000","last_unstale":"2024-03-24T00:48:56.364486+0000","last_undegraded":"2024-03-24T00:48:56.364486+0000","last_fullsized":"2024-03-24T00:48:56.364486+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:13:46.178006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.10","version":"119'8","reported_seq":22,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.419795+0000","last_change":"2024-03-24T00:48:53.988008+0000","last_active":"2024-03-24T00:48:56.419795+0000","last_peered":"2024-03-24T00:48:56.419795+0000","last_clean":"2024-03-24T00:48:56.419795+0000","last_became_active":"2024-03-24T00:48:53.987085+0000","last_became_peered":"2024-03-24T00:48:53.987085+0000","last_unstale":"2024-03-24T00:48:56.419795+0000","last_undegraded":"2024-03-24T00:48:56.419795+0000","last_fullsized":"2024-03-24T00:48:56.419795+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:26:32.321981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":358,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.363981+0000","last_change":"2024-03-24T00:38:40.953871+0000","last_active":"2024-03-24T00:48:56.363981+0000","last_peered":"2024-03-24T00:48:56.363981+0000","last_clean":"2024-03-24T00:48:56.363981+0000","last_became_active":"2024-03-24T00:38:40.952926+0000","last_became_peered":"2024-03-24T00:38:40.952926+0000","last_unstale":"2024-03-24T00:48:56.363981+0000","last_undegraded":"2024-03-24T00:48:56.363981+0000","last_fullsized":"2024-03-24T00:48:56.363981+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_clean_scrub_stamp":"2024-03-24T00:38:39.722194+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:05.397257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574977,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33236,"kb_used_data":2056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338604,"statfs":{"total":96636764160,"available":96602730496,"internally_reserved":0,"allocated":2105344,"data_stored":1290671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574976,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32644,"kb_used_data":1464,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339196,"statfs":{"total":96636764160,"available":96603336704,"internally_reserved":0,"allocated":1499136,"data_stored":700109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574976,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32648,"kb_used_data":1468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339192,"statfs":{"total":96636764160,"available":96603332608,"internally_reserved":0,"allocated":1503232,"data_stored":699687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574975,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32652,"kb_used_data":1472,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339188,"statfs":{"total":96636764160,"available":96603328512,"internally_reserved":0,"allocated":1507328,"data_stored":701139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574975,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32656,"kb_used_data":1476,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339184,"statfs":{"total":96636764160,"available":96603324416,"internally_reserved":0,"allocated":1511424,"data_stored":701173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574976,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32640,"kb_used_data":1460,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339200,"statfs":{"total":96636764160,"available":96603340800,"internally_reserved":0,"allocated":1495040,"data_stored":699527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574976,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32644,"kb_used_data":1464,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339196,"statfs":{"total":96636764160,"available":96603336704,"internally_reserved":0,"allocated":1499136,"data_stored":699549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574975,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33224,"kb_used_data":2044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338616,"statfs":{"total":96636764160,"available":96602742784,"internally_reserved":0,"allocated":2093056,"data_stored":1289917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:49:00.635 INFO:tasks.ceph.mds.a.smithi067.stderr:2024-03-24T00:49:00.635+0000 7f4e8788a640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T00:49:02.617 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:49:02.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.719+0000 7ffaaf036640 1 Processor -- start 2024-03-24T00:49:02.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.719+0000 7ffaaf036640 1 -- start start 2024-03-24T00:49:02.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.720+0000 7ffaaf036640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ffaa815e2a0 0x7ffaa80ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:02.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.720+0000 7ffaaf036640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaa80cf640 con 0x7ffaa815dec0 2024-03-24T00:49:02.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.720+0000 7ffaaf036640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaa80cf080 con 0x7ffaa80d0ef0 2024-03-24T00:49:02.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.720+0000 7ffaaf036640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaa80cee30 con 0x7ffaa815e2a0 2024-03-24T00:49:02.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.720+0000 7ffaacdd2640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ffaa815dec0 0x7ffaa80d07e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35952/0 (socket says 172.21.15.67:35952) 2024-03-24T00:49:02.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.720+0000 7ffaacdd2640 1 -- 172.21.15.67:0/1579283504 learned_addr learned my addr 172.21.15.67:0/1579283504 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:49:02.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.720+0000 7ffaa7fff640 1 --2- 172.21.15.67:0/1579283504 >> v2:172.21.15.72:3300/0 conn(0x7ffaa815e2a0 0x7ffaa80ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:02.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.720+0000 7ffaa77fe640 1 -- 172.21.15.67:0/1579283504 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3442067570 0 0) 0x7ffa9c001110 con 0x7ffaa815dec0 2024-03-24T00:49:02.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.720+0000 7ffaa77fe640 1 -- 172.21.15.67:0/1579283504 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa98003520 con 0x7ffaa815dec0 2024-03-24T00:49:02.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.720+0000 7ffaa77fe640 1 -- 172.21.15.67:0/1579283504 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2471073060 0 0) 0x7ffa98003520 con 0x7ffaa815dec0 2024-03-24T00:49:02.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.720+0000 7ffaa77fe640 1 -- 172.21.15.67:0/1579283504 >> v2:172.21.15.72:3300/0 conn(0x7ffaa815e2a0 msgr2=0x7ffaa80ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:02.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.720+0000 7ffaa77fe640 1 --2- 172.21.15.67:0/1579283504 >> v2:172.21.15.72:3300/0 conn(0x7ffaa815e2a0 0x7ffaa80ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:02.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.720+0000 7ffaa77fe640 1 -- 172.21.15.67:0/1579283504 >> v1:172.21.15.72:6789/0 conn(0x7ffaa80d0ef0 legacy=0x7ffaa80d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:02.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.720+0000 7ffaa77fe640 1 -- 172.21.15.67:0/1579283504 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaa80cf360 con 0x7ffaa815dec0 2024-03-24T00:49:02.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.720+0000 7ffaa77fe640 1 -- 172.21.15.67:0/1579283504 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7ffa9c001840 con 0x7ffaa815dec0 2024-03-24T00:49:02.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.721+0000 7ffaa77fe640 1 -- 172.21.15.67:0/1579283504 >> v1:172.21.15.67:6789/0 conn(0x7ffaa815dec0 legacy=0x7ffaa80d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:02.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.721+0000 7ffaa77fe640 1 --2- 172.21.15.67:0/1579283504 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ffa980039b0 0x7ffa98003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:02.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.721+0000 7ffaa77fe640 1 --2- 172.21.15.67:0/1579283504 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffa980043d0 0x7ffa980048a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:02.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.721+0000 7ffaa77fe640 1 --2- 172.21.15.67:0/1579283504 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa98005140 0x7ffa9800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:02.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.721+0000 7ffaa77fe640 1 -- 172.21.15.67:0/1579283504 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa80cee30 con 0x7ffa98005140 2024-03-24T00:49:02.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.721+0000 7ffaa77fe640 1 -- 172.21.15.67:0/1579283504 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa9800c090 con 0x7ffa980043d0 2024-03-24T00:49:02.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.721+0000 7ffaa77fe640 1 -- 172.21.15.67:0/1579283504 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa9800c290 con 0x7ffa980039b0 2024-03-24T00:49:02.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.721+0000 7ffaad5d3640 1 --2- 172.21.15.67:0/1579283504 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa98005140 0x7ffa9800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:02.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.721+0000 7ffaa7fff640 1 --2- 172.21.15.67:0/1579283504 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffa980043d0 0x7ffa980048a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:02.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.721+0000 7ffaacdd2640 1 --2- 172.21.15.67:0/1579283504 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ffa980039b0 0x7ffa98003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:02.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.721+0000 7ffaad5d3640 1 -- 172.21.15.67:0/1579283504 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ffa980039b0 msgr2=0x7ffa98003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:02.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.721+0000 7ffaad5d3640 1 --2- 172.21.15.67:0/1579283504 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ffa980039b0 0x7ffa98003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:02.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.721+0000 7ffaad5d3640 1 -- 172.21.15.67:0/1579283504 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffa980043d0 msgr2=0x7ffa980048a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:02.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.721+0000 7ffaad5d3640 1 --2- 172.21.15.67:0/1579283504 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffa980043d0 0x7ffa980048a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:02.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.721+0000 7ffaad5d3640 1 -- 172.21.15.67:0/1579283504 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa98003520 con 0x7ffa98005140 2024-03-24T00:49:02.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.722+0000 7ffaad5d3640 1 -- 172.21.15.67:0/1579283504 reap_dead start 2024-03-24T00:49:02.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.722+0000 7ffaad5d3640 1 --2- 172.21.15.67:0/1579283504 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa98005140 0x7ffa9800b9c0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7ffaa0001f60 tx=0x7ffa98005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe360a2322d2efa8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:02.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.722+0000 7ffaa77fe640 1 -- 172.21.15.67:0/1579283504 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa0001b30 con 0x7ffa98005140 2024-03-24T00:49:02.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.722+0000 7ffaa77fe640 1 -- 172.21.15.67:0/1579283504 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaa0003b90 con 0x7ffa98005140 2024-03-24T00:49:02.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.722+0000 7ffaa77fe640 1 -- 172.21.15.67:0/1579283504 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa0005070 con 0x7ffa98005140 2024-03-24T00:49:02.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.722+0000 7ffaaf036640 1 -- 172.21.15.67:0/1579283504 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa98005140 msgr2=0x7ffa9800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:02.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.722+0000 7ffaaf036640 1 --2- 172.21.15.67:0/1579283504 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa98005140 0x7ffa9800b9c0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7ffaa0001f60 tx=0x7ffa98005830 comp rx=0 tx=0).stop 2024-03-24T00:49:02.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.722+0000 7ffaaf036640 1 -- 172.21.15.67:0/1579283504 shutdown_connections 2024-03-24T00:49:02.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.722+0000 7ffaaf036640 1 --2- 172.21.15.67:0/1579283504 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa98005140 0x7ffa9800b9c0 unknown :-1 s=CLOSED pgs=1725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:02.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.722+0000 7ffaaf036640 1 -- 172.21.15.67:0/1579283504 >> 172.21.15.67:0/1579283504 conn(0x7ffaa806c810 msgr2=0x7ffaa806cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:02.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.723+0000 7ffaaf036640 1 -- 172.21.15.67:0/1579283504 shutdown_connections 2024-03-24T00:49:02.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.723+0000 7ffaaf036640 1 -- 172.21.15.67:0/1579283504 wait complete. 2024-03-24T00:49:02.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.724+0000 7ffaaf036640 1 Processor -- start 2024-03-24T00:49:02.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.724+0000 7ffaaf036640 1 -- start start 2024-03-24T00:49:02.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.725+0000 7ffaaf036640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa98005140 0x7ffaa81e9370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:02.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.725+0000 7ffaaf036640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffaa80cb850 0x7ffaa815f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:02.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.725+0000 7ffaaf036640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ffaa80cbcd0 0x7ffaa815f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:02.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.725+0000 7ffaaf036640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa80ce9c0 con 0x7ffa98005140 2024-03-24T00:49:02.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.725+0000 7ffaaf036640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa80ce840 con 0x7ffaa80cb850 2024-03-24T00:49:02.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.725+0000 7ffaaf036640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa80d2120 con 0x7ffaa80cbcd0 2024-03-24T00:49:02.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.725+0000 7ffaacdd2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa98005140 0x7ffaa81e9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:02.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.725+0000 7ffaacdd2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa98005140 0x7ffaa81e9370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57512/0 (socket says 172.21.15.67:57512) 2024-03-24T00:49:02.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.725+0000 7ffaacdd2640 1 -- 172.21.15.67:0/2933485014 learned_addr learned my addr 172.21.15.67:0/2933485014 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:02.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.725+0000 7ffaad5d3640 1 --2- 172.21.15.67:0/2933485014 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ffaa80cbcd0 0x7ffaa815f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:02.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.725+0000 7ffaa7fff640 1 --2- 172.21.15.67:0/2933485014 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffaa80cb850 0x7ffaa815f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:02.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.725+0000 7ffaacdd2640 1 -- 172.21.15.67:0/2933485014 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ffaa80cbcd0 msgr2=0x7ffaa815f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:02.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.725+0000 7ffaacdd2640 1 --2- 172.21.15.67:0/2933485014 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ffaa80cbcd0 0x7ffaa815f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:02.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.725+0000 7ffaacdd2640 1 -- 172.21.15.67:0/2933485014 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffaa80cb850 msgr2=0x7ffaa815f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:02.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.725+0000 7ffaacdd2640 1 --2- 172.21.15.67:0/2933485014 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffaa80cb850 0x7ffaa815f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:02.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.725+0000 7ffaacdd2640 1 -- 172.21.15.67:0/2933485014 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaa80cc290 con 0x7ffa98005140 2024-03-24T00:49:02.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.725+0000 7ffaa7fff640 1 --2- 172.21.15.67:0/2933485014 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffaa80cb850 0x7ffaa815f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:02.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.725+0000 7ffaacdd2640 1 --2- 172.21.15.67:0/2933485014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa98005140 0x7ffaa81e9370 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c0072f0 tx=0x7ffa9c0077b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:02.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.725+0000 7ffaa57fa640 1 -- 172.21.15.67:0/2933485014 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa9c01a070 con 0x7ffa98005140 2024-03-24T00:49:02.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.725+0000 7ffaa57fa640 1 -- 172.21.15.67:0/2933485014 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa9c001810 con 0x7ffa98005140 2024-03-24T00:49:02.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.725+0000 7ffaaf036640 1 -- 172.21.15.67:0/2933485014 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaa80b45c0 con 0x7ffa98005140 2024-03-24T00:49:02.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.725+0000 7ffaa57fa640 1 -- 172.21.15.67:0/2933485014 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa9c00fb60 con 0x7ffa98005140 2024-03-24T00:49:02.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.725+0000 7ffaaf036640 1 -- 172.21.15.67:0/2933485014 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaa80b4b80 con 0x7ffa98005140 2024-03-24T00:49:02.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.727+0000 7ffaaf036640 1 -- 172.21.15.67:0/2933485014 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaa804f400 con 0x7ffa98005140 2024-03-24T00:49:02.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.727+0000 7ffaa57fa640 1 -- 172.21.15.67:0/2933485014 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ffa9c01d070 con 0x7ffa98005140 2024-03-24T00:49:02.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.727+0000 7ffaa57fa640 1 --2- 172.21.15.67:0/2933485014 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ffa8c06bc00 0x7ffa8c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:02.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.727+0000 7ffaa57fa640 1 -- 172.21.15.67:0/2933485014 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ffa9c08a5a0 con 0x7ffa98005140 2024-03-24T00:49:02.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.728+0000 7ffaa7fff640 1 --2- 172.21.15.67:0/2933485014 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ffa8c06bc00 0x7ffa8c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:02.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.728+0000 7ffaa7fff640 1 --2- 172.21.15.67:0/2933485014 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ffa8c06bc00 0x7ffa8c06e0c0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7ffa940093d0 tx=0x7ffa94008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:02.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.732+0000 7ffaa57fa640 1 -- 172.21.15.67:0/2933485014 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ffaa80cf640 con 0x7ffa98005140 2024-03-24T00:49:02.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.910+0000 7ffaaf036640 1 -- 172.21.15.67:0/2933485014 --> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ffaa80d36e0 con 0x7ffa8c06bc00 2024-03-24T00:49:02.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.916+0000 7ffaa57fa640 1 -- 172.21.15.67:0/2933485014 <== mgr.4105 v2:172.21.15.72:6832/2768292274 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252488 (secure 0 0 0) 0x7ffaa80d36e0 con 0x7ffa8c06bc00 2024-03-24T00:49:02.920 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:02.921 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-24T00:49:02.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.919+0000 7ffaaf036640 1 -- 172.21.15.67:0/2933485014 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ffa8c06bc00 msgr2=0x7ffa8c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:02.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.919+0000 7ffaaf036640 1 --2- 172.21.15.67:0/2933485014 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ffa8c06bc00 0x7ffa8c06e0c0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7ffa940093d0 tx=0x7ffa94008040 comp rx=0 tx=0).stop 2024-03-24T00:49:02.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.919+0000 7ffaaf036640 1 -- 172.21.15.67:0/2933485014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa98005140 msgr2=0x7ffaa81e9370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:02.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.919+0000 7ffaaf036640 1 --2- 172.21.15.67:0/2933485014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa98005140 0x7ffaa81e9370 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c0072f0 tx=0x7ffa9c0077b0 comp rx=0 tx=0).stop 2024-03-24T00:49:02.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.919+0000 7ffaaf036640 1 -- 172.21.15.67:0/2933485014 shutdown_connections 2024-03-24T00:49:02.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.919+0000 7ffaaf036640 1 --2- 172.21.15.67:0/2933485014 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ffa8c06bc00 0x7ffa8c06e0c0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:02.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.919+0000 7ffaaf036640 1 --2- 172.21.15.67:0/2933485014 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ffaa80cbcd0 0x7ffaa815f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:02.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.919+0000 7ffaaf036640 1 --2- 172.21.15.67:0/2933485014 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffaa80cb850 0x7ffaa815f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:02.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.919+0000 7ffaaf036640 1 --2- 172.21.15.67:0/2933485014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa98005140 0x7ffaa81e9370 unknown :-1 s=CLOSED pgs=1726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:02.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.919+0000 7ffaaf036640 1 -- 172.21.15.67:0/2933485014 >> 172.21.15.67:0/2933485014 conn(0x7ffaa806c810 msgr2=0x7ffaa8160190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:02.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.919+0000 7ffaaf036640 1 -- 172.21.15.67:0/2933485014 shutdown_connections 2024-03-24T00:49:02.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:02.919+0000 7ffaaf036640 1 -- 172.21.15.67:0/2933485014 wait complete. 2024-03-24T00:49:02.931 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":424,"stamp":"2024-03-24T00:49:01.469863+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":262344,"kb_used_data":12904,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754712376,"statfs":{"total":773094113280,"available":772825473024,"internally_reserved":0,"allocated":13213696,"data_stored":6781772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.493683"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.421413+0000","last_change":"2024-03-24T00:48:55.004268+0000","last_active":"2024-03-24T00:48:56.421413+0000","last_peered":"2024-03-24T00:48:56.421413+0000","last_clean":"2024-03-24T00:48:56.421413+0000","last_became_active":"2024-03-24T00:48:55.003913+0000","last_became_peered":"2024-03-24T00:48:55.003913+0000","last_unstale":"2024-03-24T00:48:56.421413+0000","last_undegraded":"2024-03-24T00:48:56.421413+0000","last_fullsized":"2024-03-24T00:48:56.421413+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:26:40.620531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420338+0000","last_change":"2024-03-24T00:48:54.999852+0000","last_active":"2024-03-24T00:48:56.420338+0000","last_peered":"2024-03-24T00:48:56.420338+0000","last_clean":"2024-03-24T00:48:56.420338+0000","last_became_active":"2024-03-24T00:48:54.999575+0000","last_became_peered":"2024-03-24T00:48:54.999575+0000","last_unstale":"2024-03-24T00:48:56.420338+0000","last_undegraded":"2024-03-24T00:48:56.420338+0000","last_fullsized":"2024-03-24T00:48:56.420338+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:57:11.627849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420528+0000","last_change":"2024-03-24T00:48:54.995387+0000","last_active":"2024-03-24T00:48:56.420528+0000","last_peered":"2024-03-24T00:48:56.420528+0000","last_clean":"2024-03-24T00:48:56.420528+0000","last_became_active":"2024-03-24T00:48:54.994895+0000","last_became_peered":"2024-03-24T00:48:54.994895+0000","last_unstale":"2024-03-24T00:48:56.420528+0000","last_undegraded":"2024-03-24T00:48:56.420528+0000","last_fullsized":"2024-03-24T00:48:56.420528+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:33:03.600384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.364218+0000","last_change":"2024-03-24T00:48:55.006393+0000","last_active":"2024-03-24T00:48:56.364218+0000","last_peered":"2024-03-24T00:48:56.364218+0000","last_clean":"2024-03-24T00:48:56.364218+0000","last_became_active":"2024-03-24T00:48:55.005929+0000","last_became_peered":"2024-03-24T00:48:55.005929+0000","last_unstale":"2024-03-24T00:48:56.364218+0000","last_undegraded":"2024-03-24T00:48:56.364218+0000","last_fullsized":"2024-03-24T00:48:56.364218+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:49:00.743606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420894+0000","last_change":"2024-03-24T00:48:55.007877+0000","last_active":"2024-03-24T00:48:56.420894+0000","last_peered":"2024-03-24T00:48:56.420894+0000","last_clean":"2024-03-24T00:48:56.420894+0000","last_became_active":"2024-03-24T00:48:55.007552+0000","last_became_peered":"2024-03-24T00:48:55.007552+0000","last_unstale":"2024-03-24T00:48:56.420894+0000","last_undegraded":"2024-03-24T00:48:56.420894+0000","last_fullsized":"2024-03-24T00:48:56.420894+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:21:22.716643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.419988+0000","last_change":"2024-03-24T00:48:54.992733+0000","last_active":"2024-03-24T00:48:56.419988+0000","last_peered":"2024-03-24T00:48:56.419988+0000","last_clean":"2024-03-24T00:48:56.419988+0000","last_became_active":"2024-03-24T00:48:54.992479+0000","last_became_peered":"2024-03-24T00:48:54.992479+0000","last_unstale":"2024-03-24T00:48:56.419988+0000","last_undegraded":"2024-03-24T00:48:56.419988+0000","last_fullsized":"2024-03-24T00:48:56.419988+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:57:07.482366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.419674+0000","last_change":"2024-03-24T00:48:54.994860+0000","last_active":"2024-03-24T00:48:56.419674+0000","last_peered":"2024-03-24T00:48:56.419674+0000","last_clean":"2024-03-24T00:48:56.419674+0000","last_became_active":"2024-03-24T00:48:54.994481+0000","last_became_peered":"2024-03-24T00:48:54.994481+0000","last_unstale":"2024-03-24T00:48:56.419674+0000","last_undegraded":"2024-03-24T00:48:56.419674+0000","last_fullsized":"2024-03-24T00:48:56.419674+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:21:45.017588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.419778+0000","last_change":"2024-03-24T00:48:54.988130+0000","last_active":"2024-03-24T00:48:56.419778+0000","last_peered":"2024-03-24T00:48:56.419778+0000","last_clean":"2024-03-24T00:48:56.419778+0000","last_became_active":"2024-03-24T00:48:54.987865+0000","last_became_peered":"2024-03-24T00:48:54.987865+0000","last_unstale":"2024-03-24T00:48:56.419778+0000","last_undegraded":"2024-03-24T00:48:56.419778+0000","last_fullsized":"2024-03-24T00:48:56.419778+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:57:22.250932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.364452+0000","last_change":"2024-03-24T00:48:55.012922+0000","last_active":"2024-03-24T00:48:56.364452+0000","last_peered":"2024-03-24T00:48:56.364452+0000","last_clean":"2024-03-24T00:48:56.364452+0000","last_became_active":"2024-03-24T00:48:55.012673+0000","last_became_peered":"2024-03-24T00:48:55.012673+0000","last_unstale":"2024-03-24T00:48:56.364452+0000","last_undegraded":"2024-03-24T00:48:56.364452+0000","last_fullsized":"2024-03-24T00:48:56.364452+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:31:13.279647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:55.998290+0000","last_change":"2024-03-24T00:48:54.992179+0000","last_active":"2024-03-24T00:48:55.998290+0000","last_peered":"2024-03-24T00:48:55.998290+0000","last_clean":"2024-03-24T00:48:55.998290+0000","last_became_active":"2024-03-24T00:48:54.991916+0000","last_became_peered":"2024-03-24T00:48:54.991916+0000","last_unstale":"2024-03-24T00:48:55.998290+0000","last_undegraded":"2024-03-24T00:48:55.998290+0000","last_fullsized":"2024-03-24T00:48:55.998290+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:48:09.727445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420848+0000","last_change":"2024-03-24T00:48:54.998698+0000","last_active":"2024-03-24T00:48:56.420848+0000","last_peered":"2024-03-24T00:48:56.420848+0000","last_clean":"2024-03-24T00:48:56.420848+0000","last_became_active":"2024-03-24T00:48:54.998394+0000","last_became_peered":"2024-03-24T00:48:54.998394+0000","last_unstale":"2024-03-24T00:48:56.420848+0000","last_undegraded":"2024-03-24T00:48:56.420848+0000","last_fullsized":"2024-03-24T00:48:56.420848+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:29:08.295849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420272+0000","last_change":"2024-03-24T00:48:55.004680+0000","last_active":"2024-03-24T00:48:56.420272+0000","last_peered":"2024-03-24T00:48:56.420272+0000","last_clean":"2024-03-24T00:48:56.420272+0000","last_became_active":"2024-03-24T00:48:55.004371+0000","last_became_peered":"2024-03-24T00:48:55.004371+0000","last_unstale":"2024-03-24T00:48:56.420272+0000","last_undegraded":"2024-03-24T00:48:56.420272+0000","last_fullsized":"2024-03-24T00:48:56.420272+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:27:26.283205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:55.997967+0000","last_change":"2024-03-24T00:48:54.997783+0000","last_active":"2024-03-24T00:48:55.997967+0000","last_peered":"2024-03-24T00:48:55.997967+0000","last_clean":"2024-03-24T00:48:55.997967+0000","last_became_active":"2024-03-24T00:48:54.997235+0000","last_became_peered":"2024-03-24T00:48:54.997235+0000","last_unstale":"2024-03-24T00:48:55.997967+0000","last_undegraded":"2024-03-24T00:48:55.997967+0000","last_fullsized":"2024-03-24T00:48:55.997967+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:26:33.324100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420123+0000","last_change":"2024-03-24T00:48:54.990561+0000","last_active":"2024-03-24T00:48:56.420123+0000","last_peered":"2024-03-24T00:48:56.420123+0000","last_clean":"2024-03-24T00:48:56.420123+0000","last_became_active":"2024-03-24T00:48:54.990287+0000","last_became_peered":"2024-03-24T00:48:54.990287+0000","last_unstale":"2024-03-24T00:48:56.420123+0000","last_undegraded":"2024-03-24T00:48:56.420123+0000","last_fullsized":"2024-03-24T00:48:56.420123+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:15:44.739583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420799+0000","last_change":"2024-03-24T00:48:55.000100+0000","last_active":"2024-03-24T00:48:56.420799+0000","last_peered":"2024-03-24T00:48:56.420799+0000","last_clean":"2024-03-24T00:48:56.420799+0000","last_became_active":"2024-03-24T00:48:54.999462+0000","last_became_peered":"2024-03-24T00:48:54.999462+0000","last_unstale":"2024-03-24T00:48:56.420799+0000","last_undegraded":"2024-03-24T00:48:56.420799+0000","last_fullsized":"2024-03-24T00:48:56.420799+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:33.956635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.419638+0000","last_change":"2024-03-24T00:48:54.995544+0000","last_active":"2024-03-24T00:48:56.419638+0000","last_peered":"2024-03-24T00:48:56.419638+0000","last_clean":"2024-03-24T00:48:56.419638+0000","last_became_active":"2024-03-24T00:48:54.995254+0000","last_became_peered":"2024-03-24T00:48:54.995254+0000","last_unstale":"2024-03-24T00:48:56.419638+0000","last_undegraded":"2024-03-24T00:48:56.419638+0000","last_fullsized":"2024-03-24T00:48:56.419638+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:12:57.852089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:55.998271+0000","last_change":"2024-03-24T00:48:54.999212+0000","last_active":"2024-03-24T00:48:55.998271+0000","last_peered":"2024-03-24T00:48:55.998271+0000","last_clean":"2024-03-24T00:48:55.998271+0000","last_became_active":"2024-03-24T00:48:54.998297+0000","last_became_peered":"2024-03-24T00:48:54.998297+0000","last_unstale":"2024-03-24T00:48:55.998271+0000","last_undegraded":"2024-03-24T00:48:55.998271+0000","last_fullsized":"2024-03-24T00:48:55.998271+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:21:34.837676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420484+0000","last_change":"2024-03-24T00:48:54.999152+0000","last_active":"2024-03-24T00:48:56.420484+0000","last_peered":"2024-03-24T00:48:56.420484+0000","last_clean":"2024-03-24T00:48:56.420484+0000","last_became_active":"2024-03-24T00:48:54.998859+0000","last_became_peered":"2024-03-24T00:48:54.998859+0000","last_unstale":"2024-03-24T00:48:56.420484+0000","last_undegraded":"2024-03-24T00:48:56.420484+0000","last_fullsized":"2024-03-24T00:48:56.420484+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:15:16.407168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.421223+0000","last_change":"2024-03-24T00:48:55.001267+0000","last_active":"2024-03-24T00:48:56.421223+0000","last_peered":"2024-03-24T00:48:56.421223+0000","last_clean":"2024-03-24T00:48:56.421223+0000","last_became_active":"2024-03-24T00:48:55.000987+0000","last_became_peered":"2024-03-24T00:48:55.000987+0000","last_unstale":"2024-03-24T00:48:56.421223+0000","last_undegraded":"2024-03-24T00:48:56.421223+0000","last_fullsized":"2024-03-24T00:48:56.421223+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:12:35.490886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:55.999080+0000","last_change":"2024-03-24T00:48:54.995352+0000","last_active":"2024-03-24T00:48:55.999080+0000","last_peered":"2024-03-24T00:48:55.999080+0000","last_clean":"2024-03-24T00:48:55.999080+0000","last_became_active":"2024-03-24T00:48:54.995048+0000","last_became_peered":"2024-03-24T00:48:54.995048+0000","last_unstale":"2024-03-24T00:48:55.999080+0000","last_undegraded":"2024-03-24T00:48:55.999080+0000","last_fullsized":"2024-03-24T00:48:55.999080+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:19:07.036687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.364446+0000","last_change":"2024-03-24T00:48:54.999294+0000","last_active":"2024-03-24T00:48:56.364446+0000","last_peered":"2024-03-24T00:48:56.364446+0000","last_clean":"2024-03-24T00:48:56.364446+0000","last_became_active":"2024-03-24T00:48:54.998655+0000","last_became_peered":"2024-03-24T00:48:54.998655+0000","last_unstale":"2024-03-24T00:48:56.364446+0000","last_undegraded":"2024-03-24T00:48:56.364446+0000","last_fullsized":"2024-03-24T00:48:56.364446+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:20:56.647531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.421231+0000","last_change":"2024-03-24T00:48:55.012977+0000","last_active":"2024-03-24T00:48:56.421231+0000","last_peered":"2024-03-24T00:48:56.421231+0000","last_clean":"2024-03-24T00:48:56.421231+0000","last_became_active":"2024-03-24T00:48:55.012648+0000","last_became_peered":"2024-03-24T00:48:55.012648+0000","last_unstale":"2024-03-24T00:48:56.421231+0000","last_undegraded":"2024-03-24T00:48:56.421231+0000","last_fullsized":"2024-03-24T00:48:56.421231+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:32:34.044889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.364223+0000","last_change":"2024-03-24T00:48:55.010392+0000","last_active":"2024-03-24T00:48:56.364223+0000","last_peered":"2024-03-24T00:48:56.364223+0000","last_clean":"2024-03-24T00:48:56.364223+0000","last_became_active":"2024-03-24T00:48:55.010108+0000","last_became_peered":"2024-03-24T00:48:55.010108+0000","last_unstale":"2024-03-24T00:48:56.364223+0000","last_undegraded":"2024-03-24T00:48:56.364223+0000","last_fullsized":"2024-03-24T00:48:56.364223+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:34:54.718836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.001994+0000","last_change":"2024-03-24T00:48:55.002573+0000","last_active":"2024-03-24T00:48:56.001994+0000","last_peered":"2024-03-24T00:48:56.001994+0000","last_clean":"2024-03-24T00:48:56.001994+0000","last_became_active":"2024-03-24T00:48:55.002046+0000","last_became_peered":"2024-03-24T00:48:55.002046+0000","last_unstale":"2024-03-24T00:48:56.001994+0000","last_undegraded":"2024-03-24T00:48:56.001994+0000","last_fullsized":"2024-03-24T00:48:56.001994+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:21:34.837676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.002783+0000","last_change":"2024-03-24T00:48:55.005021+0000","last_active":"2024-03-24T00:48:56.002783+0000","last_peered":"2024-03-24T00:48:56.002783+0000","last_clean":"2024-03-24T00:48:56.002783+0000","last_became_active":"2024-03-24T00:48:55.004687+0000","last_became_peered":"2024-03-24T00:48:55.004687+0000","last_unstale":"2024-03-24T00:48:56.002783+0000","last_undegraded":"2024-03-24T00:48:56.002783+0000","last_fullsized":"2024-03-24T00:48:56.002783+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:49:00.743606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:55.999401+0000","last_change":"2024-03-24T00:48:55.007440+0000","last_active":"2024-03-24T00:48:55.999401+0000","last_peered":"2024-03-24T00:48:55.999401+0000","last_clean":"2024-03-24T00:48:55.999401+0000","last_became_active":"2024-03-24T00:48:55.007147+0000","last_became_peered":"2024-03-24T00:48:55.007147+0000","last_unstale":"2024-03-24T00:48:55.999401+0000","last_undegraded":"2024-03-24T00:48:55.999401+0000","last_fullsized":"2024-03-24T00:48:55.999401+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:23:29.268243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.421388+0000","last_change":"2024-03-24T00:48:55.000536+0000","last_active":"2024-03-24T00:48:56.421388+0000","last_peered":"2024-03-24T00:48:56.421388+0000","last_clean":"2024-03-24T00:48:56.421388+0000","last_became_active":"2024-03-24T00:48:55.000274+0000","last_became_peered":"2024-03-24T00:48:55.000274+0000","last_unstale":"2024-03-24T00:48:56.421388+0000","last_undegraded":"2024-03-24T00:48:56.421388+0000","last_fullsized":"2024-03-24T00:48:56.421388+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:14:53.553406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420398+0000","last_change":"2024-03-24T00:48:54.996234+0000","last_active":"2024-03-24T00:48:56.420398+0000","last_peered":"2024-03-24T00:48:56.420398+0000","last_clean":"2024-03-24T00:48:56.420398+0000","last_became_active":"2024-03-24T00:48:54.995959+0000","last_became_peered":"2024-03-24T00:48:54.995959+0000","last_unstale":"2024-03-24T00:48:56.420398+0000","last_undegraded":"2024-03-24T00:48:56.420398+0000","last_fullsized":"2024-03-24T00:48:56.420398+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:50:43.626322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:55.999391+0000","last_change":"2024-03-24T00:48:54.995982+0000","last_active":"2024-03-24T00:48:55.999391+0000","last_peered":"2024-03-24T00:48:55.999391+0000","last_clean":"2024-03-24T00:48:55.999391+0000","last_became_active":"2024-03-24T00:48:54.995699+0000","last_became_peered":"2024-03-24T00:48:54.995699+0000","last_unstale":"2024-03-24T00:48:55.999391+0000","last_undegraded":"2024-03-24T00:48:55.999391+0000","last_fullsized":"2024-03-24T00:48:55.999391+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:21:50.976426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420264+0000","last_change":"2024-03-24T00:48:55.006635+0000","last_active":"2024-03-24T00:48:56.420264+0000","last_peered":"2024-03-24T00:48:56.420264+0000","last_clean":"2024-03-24T00:48:56.420264+0000","last_became_active":"2024-03-24T00:48:55.006319+0000","last_became_peered":"2024-03-24T00:48:55.006319+0000","last_unstale":"2024-03-24T00:48:56.420264+0000","last_undegraded":"2024-03-24T00:48:56.420264+0000","last_fullsized":"2024-03-24T00:48:56.420264+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:37:40.578548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.002141+0000","last_change":"2024-03-24T00:48:54.996797+0000","last_active":"2024-03-24T00:48:56.002141+0000","last_peered":"2024-03-24T00:48:56.002141+0000","last_clean":"2024-03-24T00:48:56.002141+0000","last_became_active":"2024-03-24T00:48:54.995962+0000","last_became_peered":"2024-03-24T00:48:54.995962+0000","last_unstale":"2024-03-24T00:48:56.002141+0000","last_undegraded":"2024-03-24T00:48:56.002141+0000","last_fullsized":"2024-03-24T00:48:56.002141+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:48:56.876244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.002105+0000","last_change":"2024-03-24T00:48:55.002328+0000","last_active":"2024-03-24T00:48:56.002105+0000","last_peered":"2024-03-24T00:48:56.002105+0000","last_clean":"2024-03-24T00:48:56.002105+0000","last_became_active":"2024-03-24T00:48:55.001687+0000","last_became_peered":"2024-03-24T00:48:55.001687+0000","last_unstale":"2024-03-24T00:48:56.002105+0000","last_undegraded":"2024-03-24T00:48:56.002105+0000","last_fullsized":"2024-03-24T00:48:56.002105+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:48:09.727445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:55.998379+0000","last_change":"2024-03-24T00:48:54.995323+0000","last_active":"2024-03-24T00:48:55.998379+0000","last_peered":"2024-03-24T00:48:55.998379+0000","last_clean":"2024-03-24T00:48:55.998379+0000","last_became_active":"2024-03-24T00:48:54.995035+0000","last_became_peered":"2024-03-24T00:48:54.995035+0000","last_unstale":"2024-03-24T00:48:55.998379+0000","last_undegraded":"2024-03-24T00:48:55.998379+0000","last_fullsized":"2024-03-24T00:48:55.998379+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:48:56.876244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1f","version":"119'4","reported_seq":18,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.421388+0000","last_change":"2024-03-24T00:48:53.992874+0000","last_active":"2024-03-24T00:48:56.421388+0000","last_peered":"2024-03-24T00:48:56.421388+0000","last_clean":"2024-03-24T00:48:56.421388+0000","last_became_active":"2024-03-24T00:48:53.992507+0000","last_became_peered":"2024-03-24T00:48:53.992507+0000","last_unstale":"2024-03-24T00:48:56.421388+0000","last_undegraded":"2024-03-24T00:48:56.421388+0000","last_fullsized":"2024-03-24T00:48:56.421388+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:21:59.631832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420115+0000","last_change":"2024-03-24T00:48:54.991001+0000","last_active":"2024-03-24T00:48:56.420115+0000","last_peered":"2024-03-24T00:48:56.420115+0000","last_clean":"2024-03-24T00:48:56.420115+0000","last_became_active":"2024-03-24T00:48:54.990700+0000","last_became_peered":"2024-03-24T00:48:54.990700+0000","last_unstale":"2024-03-24T00:48:56.420115+0000","last_undegraded":"2024-03-24T00:48:56.420115+0000","last_fullsized":"2024-03-24T00:48:56.420115+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:58:43.903904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1e","version":"119'1","reported_seq":15,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:55.998384+0000","last_change":"2024-03-24T00:48:53.982887+0000","last_active":"2024-03-24T00:48:55.998384+0000","last_peered":"2024-03-24T00:48:55.998384+0000","last_clean":"2024-03-24T00:48:55.998384+0000","last_became_active":"2024-03-24T00:48:53.982515+0000","last_became_peered":"2024-03-24T00:48:53.982515+0000","last_unstale":"2024-03-24T00:48:55.998384+0000","last_undegraded":"2024-03-24T00:48:55.998384+0000","last_fullsized":"2024-03-24T00:48:55.998384+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:29:07.293730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.002283+0000","last_change":"2024-03-24T00:48:55.005866+0000","last_active":"2024-03-24T00:48:56.002283+0000","last_peered":"2024-03-24T00:48:56.002283+0000","last_clean":"2024-03-24T00:48:56.002283+0000","last_became_active":"2024-03-24T00:48:55.004659+0000","last_became_peered":"2024-03-24T00:48:55.004659+0000","last_unstale":"2024-03-24T00:48:56.002283+0000","last_undegraded":"2024-03-24T00:48:56.002283+0000","last_fullsized":"2024-03-24T00:48:56.002283+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:43:34.664046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"119'2","reported_seq":16,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.364266+0000","last_change":"2024-03-24T00:48:53.993536+0000","last_active":"2024-03-24T00:48:56.364266+0000","last_peered":"2024-03-24T00:48:56.364266+0000","last_clean":"2024-03-24T00:48:56.364266+0000","last_became_active":"2024-03-24T00:48:53.993136+0000","last_became_peered":"2024-03-24T00:48:53.993136+0000","last_unstale":"2024-03-24T00:48:56.364266+0000","last_undegraded":"2024-03-24T00:48:56.364266+0000","last_fullsized":"2024-03-24T00:48:56.364266+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:07:44.083528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.364481+0000","last_change":"2024-03-24T00:48:55.005809+0000","last_active":"2024-03-24T00:48:56.364481+0000","last_peered":"2024-03-24T00:48:56.364481+0000","last_clean":"2024-03-24T00:48:56.364481+0000","last_became_active":"2024-03-24T00:48:55.005520+0000","last_became_peered":"2024-03-24T00:48:55.005520+0000","last_unstale":"2024-03-24T00:48:56.364481+0000","last_undegraded":"2024-03-24T00:48:56.364481+0000","last_fullsized":"2024-03-24T00:48:56.364481+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:12:42.264405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"119'2","reported_seq":16,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:55.998505+0000","last_change":"2024-03-24T00:48:53.993401+0000","last_active":"2024-03-24T00:48:55.998505+0000","last_peered":"2024-03-24T00:48:55.998505+0000","last_clean":"2024-03-24T00:48:55.998505+0000","last_became_active":"2024-03-24T00:48:53.991401+0000","last_became_peered":"2024-03-24T00:48:53.991401+0000","last_unstale":"2024-03-24T00:48:55.998505+0000","last_undegraded":"2024-03-24T00:48:55.998505+0000","last_fullsized":"2024-03-24T00:48:55.998505+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:21:21.714524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:55.997959+0000","last_change":"2024-03-24T00:48:55.001388+0000","last_active":"2024-03-24T00:48:55.997959+0000","last_peered":"2024-03-24T00:48:55.997959+0000","last_clean":"2024-03-24T00:48:55.997959+0000","last_became_active":"2024-03-24T00:48:55.000323+0000","last_became_peered":"2024-03-24T00:48:55.000323+0000","last_unstale":"2024-03-24T00:48:55.997959+0000","last_undegraded":"2024-03-24T00:48:55.997959+0000","last_fullsized":"2024-03-24T00:48:55.997959+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:08:25.466057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"119'1","reported_seq":15,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.421024+0000","last_change":"2024-03-24T00:48:53.994950+0000","last_active":"2024-03-24T00:48:56.421024+0000","last_peered":"2024-03-24T00:48:56.421024+0000","last_clean":"2024-03-24T00:48:56.421024+0000","last_became_active":"2024-03-24T00:48:53.994709+0000","last_became_peered":"2024-03-24T00:48:53.994709+0000","last_unstale":"2024-03-24T00:48:56.421024+0000","last_undegraded":"2024-03-24T00:48:56.421024+0000","last_fullsized":"2024-03-24T00:48:56.421024+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:14:11.273118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420634+0000","last_change":"2024-03-24T00:48:55.000402+0000","last_active":"2024-03-24T00:48:56.420634+0000","last_peered":"2024-03-24T00:48:56.420634+0000","last_clean":"2024-03-24T00:48:56.420634+0000","last_became_active":"2024-03-24T00:48:55.000100+0000","last_became_peered":"2024-03-24T00:48:55.000100+0000","last_unstale":"2024-03-24T00:48:56.420634+0000","last_undegraded":"2024-03-24T00:48:56.420634+0000","last_fullsized":"2024-03-24T00:48:56.420634+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:14:53.553406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"119'1","reported_seq":15,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420089+0000","last_change":"2024-03-24T00:48:53.997359+0000","last_active":"2024-03-24T00:48:56.420089+0000","last_peered":"2024-03-24T00:48:56.420089+0000","last_clean":"2024-03-24T00:48:56.420089+0000","last_became_active":"2024-03-24T00:48:53.997039+0000","last_became_peered":"2024-03-24T00:48:53.997039+0000","last_unstale":"2024-03-24T00:48:56.420089+0000","last_undegraded":"2024-03-24T00:48:56.420089+0000","last_fullsized":"2024-03-24T00:48:56.420089+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:12:22.878628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420408+0000","last_change":"2024-03-24T00:48:55.001833+0000","last_active":"2024-03-24T00:48:56.420408+0000","last_peered":"2024-03-24T00:48:56.420408+0000","last_clean":"2024-03-24T00:48:56.420408+0000","last_became_active":"2024-03-24T00:48:55.000964+0000","last_became_peered":"2024-03-24T00:48:55.000964+0000","last_unstale":"2024-03-24T00:48:56.420408+0000","last_undegraded":"2024-03-24T00:48:56.420408+0000","last_fullsized":"2024-03-24T00:48:56.420408+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:04:20.294834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":14,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.002235+0000","last_change":"2024-03-24T00:48:53.988155+0000","last_active":"2024-03-24T00:48:56.002235+0000","last_peered":"2024-03-24T00:48:56.002235+0000","last_clean":"2024-03-24T00:48:56.002235+0000","last_became_active":"2024-03-24T00:48:53.987659+0000","last_became_peered":"2024-03-24T00:48:53.987659+0000","last_unstale":"2024-03-24T00:48:56.002235+0000","last_undegraded":"2024-03-24T00:48:56.002235+0000","last_fullsized":"2024-03-24T00:48:56.002235+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:26:28.796609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:55.998147+0000","last_change":"2024-03-24T00:48:55.010365+0000","last_active":"2024-03-24T00:48:55.998147+0000","last_peered":"2024-03-24T00:48:55.998147+0000","last_clean":"2024-03-24T00:48:55.998147+0000","last_became_active":"2024-03-24T00:48:55.010131+0000","last_became_peered":"2024-03-24T00:48:55.010131+0000","last_unstale":"2024-03-24T00:48:55.998147+0000","last_undegraded":"2024-03-24T00:48:55.998147+0000","last_fullsized":"2024-03-24T00:48:55.998147+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:15:03.063582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":14,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.419829+0000","last_change":"2024-03-24T00:48:53.993610+0000","last_active":"2024-03-24T00:48:56.419829+0000","last_peered":"2024-03-24T00:48:56.419829+0000","last_clean":"2024-03-24T00:48:56.419829+0000","last_became_active":"2024-03-24T00:48:53.993326+0000","last_became_peered":"2024-03-24T00:48:53.993326+0000","last_unstale":"2024-03-24T00:48:56.419829+0000","last_undegraded":"2024-03-24T00:48:56.419829+0000","last_fullsized":"2024-03-24T00:48:56.419829+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:07:03.572587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420867+0000","last_change":"2024-03-24T00:48:54.999636+0000","last_active":"2024-03-24T00:48:56.420867+0000","last_peered":"2024-03-24T00:48:56.420867+0000","last_clean":"2024-03-24T00:48:56.420867+0000","last_became_active":"2024-03-24T00:48:54.999197+0000","last_became_peered":"2024-03-24T00:48:54.999197+0000","last_unstale":"2024-03-24T00:48:56.420867+0000","last_undegraded":"2024-03-24T00:48:56.420867+0000","last_fullsized":"2024-03-24T00:48:56.420867+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:34:55.898803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":14,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.421402+0000","last_change":"2024-03-24T00:48:53.999330+0000","last_active":"2024-03-24T00:48:56.421402+0000","last_peered":"2024-03-24T00:48:56.421402+0000","last_clean":"2024-03-24T00:48:56.421402+0000","last_became_active":"2024-03-24T00:48:53.999065+0000","last_became_peered":"2024-03-24T00:48:53.999065+0000","last_unstale":"2024-03-24T00:48:56.421402+0000","last_undegraded":"2024-03-24T00:48:56.421402+0000","last_fullsized":"2024-03-24T00:48:56.421402+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:14:11.273118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.002545+0000","last_change":"2024-03-24T00:48:54.994492+0000","last_active":"2024-03-24T00:48:56.002545+0000","last_peered":"2024-03-24T00:48:56.002545+0000","last_clean":"2024-03-24T00:48:56.002545+0000","last_became_active":"2024-03-24T00:48:54.994195+0000","last_became_peered":"2024-03-24T00:48:54.994195+0000","last_unstale":"2024-03-24T00:48:56.002545+0000","last_undegraded":"2024-03-24T00:48:56.002545+0000","last_fullsized":"2024-03-24T00:48:56.002545+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:34:54.718836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":14,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.002174+0000","last_change":"2024-03-24T00:48:53.985480+0000","last_active":"2024-03-24T00:48:56.002174+0000","last_peered":"2024-03-24T00:48:56.002174+0000","last_clean":"2024-03-24T00:48:56.002174+0000","last_became_active":"2024-03-24T00:48:53.985203+0000","last_became_peered":"2024-03-24T00:48:53.985203+0000","last_unstale":"2024-03-24T00:48:56.002174+0000","last_undegraded":"2024-03-24T00:48:56.002174+0000","last_fullsized":"2024-03-24T00:48:56.002174+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:37:28.523220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420636+0000","last_change":"2024-03-24T00:48:54.993477+0000","last_active":"2024-03-24T00:48:56.420636+0000","last_peered":"2024-03-24T00:48:56.420636+0000","last_clean":"2024-03-24T00:48:56.420636+0000","last_became_active":"2024-03-24T00:48:54.993207+0000","last_became_peered":"2024-03-24T00:48:54.993207+0000","last_unstale":"2024-03-24T00:48:56.420636+0000","last_undegraded":"2024-03-24T00:48:56.420636+0000","last_fullsized":"2024-03-24T00:48:56.420636+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:27:27.533944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.15","version":"119'3","reported_seq":17,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.002520+0000","last_change":"2024-03-24T00:48:53.987256+0000","last_active":"2024-03-24T00:48:56.002520+0000","last_peered":"2024-03-24T00:48:56.002520+0000","last_clean":"2024-03-24T00:48:56.002520+0000","last_became_active":"2024-03-24T00:48:53.986435+0000","last_became_peered":"2024-03-24T00:48:53.986435+0000","last_unstale":"2024-03-24T00:48:56.002520+0000","last_undegraded":"2024-03-24T00:48:56.002520+0000","last_fullsized":"2024-03-24T00:48:56.002520+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:26:39.618413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.002842+0000","last_change":"2024-03-24T00:48:55.001414+0000","last_active":"2024-03-24T00:48:56.002842+0000","last_peered":"2024-03-24T00:48:56.002842+0000","last_clean":"2024-03-24T00:48:56.002842+0000","last_became_active":"2024-03-24T00:48:55.001126+0000","last_became_peered":"2024-03-24T00:48:55.001126+0000","last_unstale":"2024-03-24T00:48:56.002842+0000","last_undegraded":"2024-03-24T00:48:56.002842+0000","last_fullsized":"2024-03-24T00:48:56.002842+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:31:13.279647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.14","version":"119'6","reported_seq":20,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420741+0000","last_change":"2024-03-24T00:48:53.992589+0000","last_active":"2024-03-24T00:48:56.420741+0000","last_peered":"2024-03-24T00:48:56.420741+0000","last_clean":"2024-03-24T00:48:56.420741+0000","last_became_active":"2024-03-24T00:48:53.992273+0000","last_became_peered":"2024-03-24T00:48:53.992273+0000","last_unstale":"2024-03-24T00:48:56.420741+0000","last_undegraded":"2024-03-24T00:48:56.420741+0000","last_fullsized":"2024-03-24T00:48:56.420741+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:21:33.835558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.421262+0000","last_change":"2024-03-24T00:48:54.987532+0000","last_active":"2024-03-24T00:48:56.421262+0000","last_peered":"2024-03-24T00:48:56.421262+0000","last_clean":"2024-03-24T00:48:56.421262+0000","last_became_active":"2024-03-24T00:48:54.987241+0000","last_became_peered":"2024-03-24T00:48:54.987241+0000","last_unstale":"2024-03-24T00:48:56.421262+0000","last_undegraded":"2024-03-24T00:48:56.421262+0000","last_fullsized":"2024-03-24T00:48:56.421262+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:31:26.868426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.13","version":"119'2","reported_seq":16,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.419984+0000","last_change":"2024-03-24T00:48:53.988410+0000","last_active":"2024-03-24T00:48:56.419984+0000","last_peered":"2024-03-24T00:48:56.419984+0000","last_clean":"2024-03-24T00:48:56.419984+0000","last_became_active":"2024-03-24T00:48:53.987901+0000","last_became_peered":"2024-03-24T00:48:53.987901+0000","last_unstale":"2024-03-24T00:48:56.419984+0000","last_undegraded":"2024-03-24T00:48:56.419984+0000","last_fullsized":"2024-03-24T00:48:56.419984+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:23:37.568996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:55.998042+0000","last_change":"2024-03-24T00:48:54.996570+0000","last_active":"2024-03-24T00:48:55.998042+0000","last_peered":"2024-03-24T00:48:55.998042+0000","last_clean":"2024-03-24T00:48:55.998042+0000","last_became_active":"2024-03-24T00:48:54.996219+0000","last_became_peered":"2024-03-24T00:48:54.996219+0000","last_unstale":"2024-03-24T00:48:55.998042+0000","last_undegraded":"2024-03-24T00:48:55.998042+0000","last_fullsized":"2024-03-24T00:48:55.998042+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:35:09.413477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":14,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420599+0000","last_change":"2024-03-24T00:48:53.988076+0000","last_active":"2024-03-24T00:48:56.420599+0000","last_peered":"2024-03-24T00:48:56.420599+0000","last_clean":"2024-03-24T00:48:56.420599+0000","last_became_active":"2024-03-24T00:48:53.987752+0000","last_became_peered":"2024-03-24T00:48:53.987752+0000","last_unstale":"2024-03-24T00:48:56.420599+0000","last_undegraded":"2024-03-24T00:48:56.420599+0000","last_fullsized":"2024-03-24T00:48:56.420599+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:48:08.725327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:55.999118+0000","last_change":"2024-03-24T00:48:55.005093+0000","last_active":"2024-03-24T00:48:55.999118+0000","last_peered":"2024-03-24T00:48:55.999118+0000","last_clean":"2024-03-24T00:48:55.999118+0000","last_became_active":"2024-03-24T00:48:55.004817+0000","last_became_peered":"2024-03-24T00:48:55.004817+0000","last_unstale":"2024-03-24T00:48:55.999118+0000","last_undegraded":"2024-03-24T00:48:55.999118+0000","last_fullsized":"2024-03-24T00:48:55.999118+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:42:49.723947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":14,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420113+0000","last_change":"2024-03-24T00:48:53.992865+0000","last_active":"2024-03-24T00:48:56.420113+0000","last_peered":"2024-03-24T00:48:56.420113+0000","last_clean":"2024-03-24T00:48:56.420113+0000","last_became_active":"2024-03-24T00:48:53.992565+0000","last_became_peered":"2024-03-24T00:48:53.992565+0000","last_unstale":"2024-03-24T00:48:56.420113+0000","last_undegraded":"2024-03-24T00:48:56.420113+0000","last_fullsized":"2024-03-24T00:48:56.420113+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:35:08.411358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.002343+0000","last_change":"2024-03-24T00:48:55.002859+0000","last_active":"2024-03-24T00:48:56.002343+0000","last_peered":"2024-03-24T00:48:56.002343+0000","last_clean":"2024-03-24T00:48:56.002343+0000","last_became_active":"2024-03-24T00:48:55.002568+0000","last_became_peered":"2024-03-24T00:48:55.002568+0000","last_unstale":"2024-03-24T00:48:56.002343+0000","last_undegraded":"2024-03-24T00:48:56.002343+0000","last_fullsized":"2024-03-24T00:48:56.002343+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:15:22.617558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"119'1","reported_seq":15,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:55.998366+0000","last_change":"2024-03-24T00:48:53.988733+0000","last_active":"2024-03-24T00:48:55.998366+0000","last_peered":"2024-03-24T00:48:55.998366+0000","last_clean":"2024-03-24T00:48:55.998366+0000","last_became_active":"2024-03-24T00:48:53.988394+0000","last_became_peered":"2024-03-24T00:48:53.988394+0000","last_unstale":"2024-03-24T00:48:55.998366+0000","last_undegraded":"2024-03-24T00:48:55.998366+0000","last_fullsized":"2024-03-24T00:48:55.998366+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:07:15.456620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.421054+0000","last_change":"2024-03-24T00:48:55.013412+0000","last_active":"2024-03-24T00:48:56.421054+0000","last_peered":"2024-03-24T00:48:56.421054+0000","last_clean":"2024-03-24T00:48:56.421054+0000","last_became_active":"2024-03-24T00:48:55.013075+0000","last_became_peered":"2024-03-24T00:48:55.013075+0000","last_unstale":"2024-03-24T00:48:56.421054+0000","last_undegraded":"2024-03-24T00:48:56.421054+0000","last_fullsized":"2024-03-24T00:48:56.421054+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:07:16.458739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"119'2","reported_seq":16,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:55.999500+0000","last_change":"2024-03-24T00:48:53.989354+0000","last_active":"2024-03-24T00:48:55.999500+0000","last_peered":"2024-03-24T00:48:55.999500+0000","last_clean":"2024-03-24T00:48:55.999500+0000","last_became_active":"2024-03-24T00:48:53.989074+0000","last_became_peered":"2024-03-24T00:48:53.989074+0000","last_unstale":"2024-03-24T00:48:55.999500+0000","last_undegraded":"2024-03-24T00:48:55.999500+0000","last_fullsized":"2024-03-24T00:48:55.999500+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:09:51.433738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420447+0000","last_change":"2024-03-24T00:48:55.002046+0000","last_active":"2024-03-24T00:48:56.420447+0000","last_peered":"2024-03-24T00:48:56.420447+0000","last_clean":"2024-03-24T00:48:56.420447+0000","last_became_active":"2024-03-24T00:48:55.001773+0000","last_became_peered":"2024-03-24T00:48:55.001773+0000","last_unstale":"2024-03-24T00:48:56.420447+0000","last_undegraded":"2024-03-24T00:48:56.420447+0000","last_fullsized":"2024-03-24T00:48:56.420447+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:25:31.343032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"119'1","reported_seq":15,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.364313+0000","last_change":"2024-03-24T00:48:53.992745+0000","last_active":"2024-03-24T00:48:56.364313+0000","last_peered":"2024-03-24T00:48:56.364313+0000","last_clean":"2024-03-24T00:48:56.364313+0000","last_became_active":"2024-03-24T00:48:53.992433+0000","last_became_peered":"2024-03-24T00:48:53.992433+0000","last_unstale":"2024-03-24T00:48:56.364313+0000","last_undegraded":"2024-03-24T00:48:56.364313+0000","last_fullsized":"2024-03-24T00:48:56.364313+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:08:24.463939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.002907+0000","last_change":"2024-03-24T00:48:55.007080+0000","last_active":"2024-03-24T00:48:56.002907+0000","last_peered":"2024-03-24T00:48:56.002907+0000","last_clean":"2024-03-24T00:48:56.002907+0000","last_became_active":"2024-03-24T00:48:55.006437+0000","last_became_peered":"2024-03-24T00:48:55.006437+0000","last_unstale":"2024-03-24T00:48:56.002907+0000","last_undegraded":"2024-03-24T00:48:56.002907+0000","last_fullsized":"2024-03-24T00:48:56.002907+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:20:56.647531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":14,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.364137+0000","last_change":"2024-03-24T00:48:53.997898+0000","last_active":"2024-03-24T00:48:56.364137+0000","last_peered":"2024-03-24T00:48:56.364137+0000","last_clean":"2024-03-24T00:48:56.364137+0000","last_became_active":"2024-03-24T00:48:53.997647+0000","last_became_peered":"2024-03-24T00:48:53.997647+0000","last_unstale":"2024-03-24T00:48:56.364137+0000","last_undegraded":"2024-03-24T00:48:56.364137+0000","last_fullsized":"2024-03-24T00:48:56.364137+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:58:19.481628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.419638+0000","last_change":"2024-03-24T00:48:54.999193+0000","last_active":"2024-03-24T00:48:56.419638+0000","last_peered":"2024-03-24T00:48:56.419638+0000","last_clean":"2024-03-24T00:48:56.419638+0000","last_became_active":"2024-03-24T00:48:54.998895+0000","last_became_peered":"2024-03-24T00:48:54.998895+0000","last_unstale":"2024-03-24T00:48:56.419638+0000","last_undegraded":"2024-03-24T00:48:56.419638+0000","last_fullsized":"2024-03-24T00:48:56.419638+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:22:30.282268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"119'2","reported_seq":16,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420210+0000","last_change":"2024-03-24T00:48:53.989088+0000","last_active":"2024-03-24T00:48:56.420210+0000","last_peered":"2024-03-24T00:48:56.420210+0000","last_clean":"2024-03-24T00:48:56.420210+0000","last_became_active":"2024-03-24T00:48:53.988812+0000","last_became_peered":"2024-03-24T00:48:53.988812+0000","last_unstale":"2024-03-24T00:48:56.420210+0000","last_undegraded":"2024-03-24T00:48:56.420210+0000","last_fullsized":"2024-03-24T00:48:56.420210+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:54:55.973101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.419960+0000","last_change":"2024-03-24T00:48:54.996761+0000","last_active":"2024-03-24T00:48:56.419960+0000","last_peered":"2024-03-24T00:48:56.419960+0000","last_clean":"2024-03-24T00:48:56.419960+0000","last_became_active":"2024-03-24T00:48:54.996464+0000","last_became_peered":"2024-03-24T00:48:54.996464+0000","last_unstale":"2024-03-24T00:48:56.419960+0000","last_undegraded":"2024-03-24T00:48:56.419960+0000","last_fullsized":"2024-03-24T00:48:56.419960+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:54:00.503746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":14,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.363977+0000","last_change":"2024-03-24T00:48:53.992661+0000","last_active":"2024-03-24T00:48:56.363977+0000","last_peered":"2024-03-24T00:48:56.363977+0000","last_clean":"2024-03-24T00:48:56.363977+0000","last_became_active":"2024-03-24T00:48:53.992347+0000","last_became_peered":"2024-03-24T00:48:53.992347+0000","last_unstale":"2024-03-24T00:48:56.363977+0000","last_undegraded":"2024-03-24T00:48:56.363977+0000","last_fullsized":"2024-03-24T00:48:56.363977+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:01:04.094765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420604+0000","last_change":"2024-03-24T00:48:54.989841+0000","last_active":"2024-03-24T00:48:56.420604+0000","last_peered":"2024-03-24T00:48:56.420604+0000","last_clean":"2024-03-24T00:48:56.420604+0000","last_became_active":"2024-03-24T00:48:54.989577+0000","last_became_peered":"2024-03-24T00:48:54.989577+0000","last_unstale":"2024-03-24T00:48:56.420604+0000","last_undegraded":"2024-03-24T00:48:56.420604+0000","last_fullsized":"2024-03-24T00:48:56.420604+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:25:52.856404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":14,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420197+0000","last_change":"2024-03-24T00:48:53.995870+0000","last_active":"2024-03-24T00:48:56.420197+0000","last_peered":"2024-03-24T00:48:56.420197+0000","last_clean":"2024-03-24T00:48:56.420197+0000","last_became_active":"2024-03-24T00:48:53.995507+0000","last_became_peered":"2024-03-24T00:48:53.995507+0000","last_unstale":"2024-03-24T00:48:56.420197+0000","last_undegraded":"2024-03-24T00:48:56.420197+0000","last_fullsized":"2024-03-24T00:48:56.420197+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:15:02.061464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.002232+0000","last_change":"2024-03-24T00:48:54.996407+0000","last_active":"2024-03-24T00:48:56.002232+0000","last_peered":"2024-03-24T00:48:56.002232+0000","last_clean":"2024-03-24T00:48:56.002232+0000","last_became_active":"2024-03-24T00:48:54.996118+0000","last_became_peered":"2024-03-24T00:48:54.996118+0000","last_unstale":"2024-03-24T00:48:56.002232+0000","last_undegraded":"2024-03-24T00:48:56.002232+0000","last_fullsized":"2024-03-24T00:48:56.002232+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:03:17.187222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":14,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.363817+0000","last_change":"2024-03-24T00:48:53.993911+0000","last_active":"2024-03-24T00:48:56.363817+0000","last_peered":"2024-03-24T00:48:56.363817+0000","last_clean":"2024-03-24T00:48:56.363817+0000","last_became_active":"2024-03-24T00:48:53.993618+0000","last_became_peered":"2024-03-24T00:48:53.993618+0000","last_unstale":"2024-03-24T00:48:56.363817+0000","last_undegraded":"2024-03-24T00:48:56.363817+0000","last_fullsized":"2024-03-24T00:48:56.363817+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:24:44.940524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420427+0000","last_change":"2024-03-24T00:48:54.994862+0000","last_active":"2024-03-24T00:48:56.420427+0000","last_peered":"2024-03-24T00:48:56.420427+0000","last_clean":"2024-03-24T00:48:56.420427+0000","last_became_active":"2024-03-24T00:48:54.994511+0000","last_became_peered":"2024-03-24T00:48:54.994511+0000","last_unstale":"2024-03-24T00:48:56.420427+0000","last_undegraded":"2024-03-24T00:48:56.420427+0000","last_fullsized":"2024-03-24T00:48:56.420427+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:54:33.905780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.7","version":"119'2","reported_seq":16,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420300+0000","last_change":"2024-03-24T00:48:53.994470+0000","last_active":"2024-03-24T00:48:56.420300+0000","last_peered":"2024-03-24T00:48:56.420300+0000","last_clean":"2024-03-24T00:48:56.420300+0000","last_became_active":"2024-03-24T00:48:53.994193+0000","last_became_peered":"2024-03-24T00:48:53.994193+0000","last_unstale":"2024-03-24T00:48:56.420300+0000","last_undegraded":"2024-03-24T00:48:56.420300+0000","last_fullsized":"2024-03-24T00:48:56.420300+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:21:24.620572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:55.998229+0000","last_change":"2024-03-24T00:48:55.004460+0000","last_active":"2024-03-24T00:48:55.998229+0000","last_peered":"2024-03-24T00:48:55.998229+0000","last_clean":"2024-03-24T00:48:55.998229+0000","last_became_active":"2024-03-24T00:48:55.003966+0000","last_became_peered":"2024-03-24T00:48:55.003966+0000","last_unstale":"2024-03-24T00:48:55.998229+0000","last_undegraded":"2024-03-24T00:48:55.998229+0000","last_fullsized":"2024-03-24T00:48:55.998229+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:03:17.187222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.6","version":"120'1","reported_seq":15,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.421551+0000","last_change":"2024-03-24T00:48:53.995779+0000","last_active":"2024-03-24T00:48:56.421551+0000","last_peered":"2024-03-24T00:48:56.421551+0000","last_clean":"2024-03-24T00:48:56.421551+0000","last_became_active":"2024-03-24T00:48:53.995516+0000","last_became_peered":"2024-03-24T00:48:53.995516+0000","last_unstale":"2024-03-24T00:48:56.421551+0000","last_undegraded":"2024-03-24T00:48:56.421551+0000","last_fullsized":"2024-03-24T00:48:56.421551+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:30:35.317515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.364522+0000","last_change":"2024-03-24T00:48:55.000436+0000","last_active":"2024-03-24T00:48:56.364522+0000","last_peered":"2024-03-24T00:48:56.364522+0000","last_clean":"2024-03-24T00:48:56.364522+0000","last_became_active":"2024-03-24T00:48:54.999859+0000","last_became_peered":"2024-03-24T00:48:54.999859+0000","last_unstale":"2024-03-24T00:48:56.364522+0000","last_undegraded":"2024-03-24T00:48:56.364522+0000","last_fullsized":"2024-03-24T00:48:56.364522+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:46:56.230850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":14,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.421402+0000","last_change":"2024-03-24T00:48:53.995776+0000","last_active":"2024-03-24T00:48:56.421402+0000","last_peered":"2024-03-24T00:48:56.421402+0000","last_clean":"2024-03-24T00:48:56.421402+0000","last_became_active":"2024-03-24T00:48:53.995494+0000","last_became_peered":"2024-03-24T00:48:53.995494+0000","last_unstale":"2024-03-24T00:48:56.421402+0000","last_undegraded":"2024-03-24T00:48:56.421402+0000","last_fullsized":"2024-03-24T00:48:56.421402+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:35:18.392188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:55.999180+0000","last_change":"2024-03-24T00:48:54.998094+0000","last_active":"2024-03-24T00:48:55.999180+0000","last_peered":"2024-03-24T00:48:55.999180+0000","last_clean":"2024-03-24T00:48:55.999180+0000","last_became_active":"2024-03-24T00:48:54.997823+0000","last_became_peered":"2024-03-24T00:48:54.997823+0000","last_unstale":"2024-03-24T00:48:55.999180+0000","last_undegraded":"2024-03-24T00:48:55.999180+0000","last_fullsized":"2024-03-24T00:48:55.999180+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:46:59.292179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":14,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420491+0000","last_change":"2024-03-24T00:48:53.996637+0000","last_active":"2024-03-24T00:48:56.420491+0000","last_peered":"2024-03-24T00:48:56.420491+0000","last_clean":"2024-03-24T00:48:56.420491+0000","last_became_active":"2024-03-24T00:48:53.996356+0000","last_became_peered":"2024-03-24T00:48:53.996356+0000","last_unstale":"2024-03-24T00:48:56.420491+0000","last_undegraded":"2024-03-24T00:48:56.420491+0000","last_fullsized":"2024-03-24T00:48:56.420491+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:04:03.246665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.419792+0000","last_change":"2024-03-24T00:48:55.001161+0000","last_active":"2024-03-24T00:48:56.419792+0000","last_peered":"2024-03-24T00:48:56.419792+0000","last_clean":"2024-03-24T00:48:56.419792+0000","last_became_active":"2024-03-24T00:48:55.000901+0000","last_became_peered":"2024-03-24T00:48:55.000901+0000","last_unstale":"2024-03-24T00:48:56.419792+0000","last_undegraded":"2024-03-24T00:48:56.419792+0000","last_fullsized":"2024-03-24T00:48:56.419792+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:48:52.719348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":14,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.421208+0000","last_change":"2024-03-24T00:48:53.998500+0000","last_active":"2024-03-24T00:48:56.421208+0000","last_peered":"2024-03-24T00:48:56.421208+0000","last_clean":"2024-03-24T00:48:56.421208+0000","last_became_active":"2024-03-24T00:48:53.998219+0000","last_became_peered":"2024-03-24T00:48:53.998219+0000","last_unstale":"2024-03-24T00:48:56.421208+0000","last_undegraded":"2024-03-24T00:48:56.421208+0000","last_fullsized":"2024-03-24T00:48:56.421208+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:30:35.317515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.420109+0000","last_change":"2024-03-24T00:48:54.989178+0000","last_active":"2024-03-24T00:48:56.420109+0000","last_peered":"2024-03-24T00:48:56.420109+0000","last_clean":"2024-03-24T00:48:56.420109+0000","last_became_active":"2024-03-24T00:48:54.988884+0000","last_became_peered":"2024-03-24T00:48:54.988884+0000","last_unstale":"2024-03-24T00:48:56.420109+0000","last_undegraded":"2024-03-24T00:48:56.420109+0000","last_fullsized":"2024-03-24T00:48:56.420109+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:19:24.977405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":14,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.421336+0000","last_change":"2024-03-24T00:48:53.992750+0000","last_active":"2024-03-24T00:48:56.421336+0000","last_peered":"2024-03-24T00:48:56.421336+0000","last_clean":"2024-03-24T00:48:56.421336+0000","last_became_active":"2024-03-24T00:48:53.992433+0000","last_became_peered":"2024-03-24T00:48:53.992433+0000","last_unstale":"2024-03-24T00:48:56.421336+0000","last_undegraded":"2024-03-24T00:48:56.421336+0000","last_fullsized":"2024-03-24T00:48:56.421336+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:21:59.631832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.364100+0000","last_change":"2024-03-24T00:48:55.001650+0000","last_active":"2024-03-24T00:48:56.364100+0000","last_peered":"2024-03-24T00:48:56.364100+0000","last_clean":"2024-03-24T00:48:56.364100+0000","last_became_active":"2024-03-24T00:48:55.001336+0000","last_became_peered":"2024-03-24T00:48:55.001336+0000","last_unstale":"2024-03-24T00:48:56.364100+0000","last_undegraded":"2024-03-24T00:48:56.364100+0000","last_fullsized":"2024-03-24T00:48:56.364100+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:15:22.617558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":14,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.419686+0000","last_change":"2024-03-24T00:48:53.997216+0000","last_active":"2024-03-24T00:48:56.419686+0000","last_peered":"2024-03-24T00:48:56.419686+0000","last_clean":"2024-03-24T00:48:56.419686+0000","last_became_active":"2024-03-24T00:48:53.996916+0000","last_became_peered":"2024-03-24T00:48:53.996916+0000","last_unstale":"2024-03-24T00:48:56.419686+0000","last_undegraded":"2024-03-24T00:48:56.419686+0000","last_fullsized":"2024-03-24T00:48:56.419686+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:18:02.497937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:55.998181+0000","last_change":"2024-03-24T00:48:55.001355+0000","last_active":"2024-03-24T00:48:55.998181+0000","last_peered":"2024-03-24T00:48:55.998181+0000","last_clean":"2024-03-24T00:48:55.998181+0000","last_became_active":"2024-03-24T00:48:54.996881+0000","last_became_peered":"2024-03-24T00:48:54.996881+0000","last_unstale":"2024-03-24T00:48:55.998181+0000","last_undegraded":"2024-03-24T00:48:55.998181+0000","last_fullsized":"2024-03-24T00:48:55.998181+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:23:38.571115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":14,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.363827+0000","last_change":"2024-03-24T00:48:53.993376+0000","last_active":"2024-03-24T00:48:56.363827+0000","last_peered":"2024-03-24T00:48:56.363827+0000","last_clean":"2024-03-24T00:48:56.363827+0000","last_became_active":"2024-03-24T00:48:53.993104+0000","last_became_peered":"2024-03-24T00:48:53.993104+0000","last_unstale":"2024-03-24T00:48:56.363827+0000","last_undegraded":"2024-03-24T00:48:56.363827+0000","last_fullsized":"2024-03-24T00:48:56.363827+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:10:25.529558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":12,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.364486+0000","last_change":"2024-03-24T00:48:54.999535+0000","last_active":"2024-03-24T00:48:56.364486+0000","last_peered":"2024-03-24T00:48:56.364486+0000","last_clean":"2024-03-24T00:48:56.364486+0000","last_became_active":"2024-03-24T00:48:54.999019+0000","last_became_peered":"2024-03-24T00:48:54.999019+0000","last_unstale":"2024-03-24T00:48:56.364486+0000","last_undegraded":"2024-03-24T00:48:56.364486+0000","last_fullsized":"2024-03-24T00:48:56.364486+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:53.970087+0000","last_clean_scrub_stamp":"2024-03-24T00:48:53.970087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:13:46.178006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.10","version":"119'8","reported_seq":22,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.419795+0000","last_change":"2024-03-24T00:48:53.988008+0000","last_active":"2024-03-24T00:48:56.419795+0000","last_peered":"2024-03-24T00:48:56.419795+0000","last_clean":"2024-03-24T00:48:56.419795+0000","last_became_active":"2024-03-24T00:48:53.987085+0000","last_became_peered":"2024-03-24T00:48:53.987085+0000","last_unstale":"2024-03-24T00:48:56.419795+0000","last_undegraded":"2024-03-24T00:48:56.419795+0000","last_fullsized":"2024-03-24T00:48:56.419795+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:48:52.967969+0000","last_clean_scrub_stamp":"2024-03-24T00:48:52.967969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:26:32.321981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":358,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-24T00:48:56.363981+0000","last_change":"2024-03-24T00:38:40.953871+0000","last_active":"2024-03-24T00:48:56.363981+0000","last_peered":"2024-03-24T00:48:56.363981+0000","last_clean":"2024-03-24T00:48:56.363981+0000","last_became_active":"2024-03-24T00:38:40.952926+0000","last_became_peered":"2024-03-24T00:38:40.952926+0000","last_unstale":"2024-03-24T00:48:56.363981+0000","last_undegraded":"2024-03-24T00:48:56.363981+0000","last_fullsized":"2024-03-24T00:48:56.363981+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_clean_scrub_stamp":"2024-03-24T00:38:39.722194+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:05.397257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574977,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33236,"kb_used_data":2056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338604,"statfs":{"total":96636764160,"available":96602730496,"internally_reserved":0,"allocated":2105344,"data_stored":1290671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574976,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32644,"kb_used_data":1464,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339196,"statfs":{"total":96636764160,"available":96603336704,"internally_reserved":0,"allocated":1499136,"data_stored":700109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574976,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32648,"kb_used_data":1468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339192,"statfs":{"total":96636764160,"available":96603332608,"internally_reserved":0,"allocated":1503232,"data_stored":699687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574975,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32652,"kb_used_data":1472,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339188,"statfs":{"total":96636764160,"available":96603328512,"internally_reserved":0,"allocated":1507328,"data_stored":701139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574975,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32656,"kb_used_data":1476,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339184,"statfs":{"total":96636764160,"available":96603324416,"internally_reserved":0,"allocated":1511424,"data_stored":701173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574976,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32640,"kb_used_data":1460,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339200,"statfs":{"total":96636764160,"available":96603340800,"internally_reserved":0,"allocated":1495040,"data_stored":699527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574976,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32644,"kb_used_data":1464,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339196,"statfs":{"total":96636764160,"available":96603336704,"internally_reserved":0,"allocated":1499136,"data_stored":699549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574975,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33224,"kb_used_data":2044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338616,"statfs":{"total":96636764160,"available":96602742784,"internally_reserved":0,"allocated":2093056,"data_stored":1289917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:49:02.935 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-24T00:49:02.935 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-24T00:49:03.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.084+0000 7f7517b77640 1 Processor -- start 2024-03-24T00:49:03.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.084+0000 7f7517b77640 1 -- start start 2024-03-24T00:49:03.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.085+0000 7f7517b77640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f751015e380 0x7f75100d11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:03.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.085+0000 7f7517b77640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7510099d70 0x7f751009a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:03.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.085+0000 7f7517b77640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f751006cc00 con 0x7f751006c440 2024-03-24T00:49:03.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.085+0000 7f7517b77640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75100b33d0 con 0x7f751015e380 2024-03-24T00:49:03.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.085+0000 7f7517b77640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f751009a700 con 0x7f7510099d70 2024-03-24T00:49:03.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.085+0000 7f7515913640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f751015e380 0x7f75100d11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:03.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.085+0000 7f7515913640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f751015e380 0x7f75100d11f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57522/0 (socket says 172.21.15.67:57522) 2024-03-24T00:49:03.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.085+0000 7f7515913640 1 -- 172.21.15.67:0/1185045948 learned_addr learned my addr 172.21.15.67:0/1185045948 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:03.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.085+0000 7f7516114640 1 --2- 172.21.15.67:0/1185045948 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7510099d70 0x7f751009a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:03.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.085+0000 7f7515913640 1 -- 172.21.15.67:0/1185045948 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7510099d70 msgr2=0x7f751009a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:03.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.085+0000 7f7515913640 1 --2- 172.21.15.67:0/1185045948 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7510099d70 0x7f751009a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:03.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.085+0000 7f7515913640 1 -- 172.21.15.67:0/1185045948 >> v1:172.21.15.67:6789/0 conn(0x7f751006c440 legacy=0x7f75100b2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:03.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.085+0000 7f7515913640 1 -- 172.21.15.67:0/1185045948 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f751006cee0 con 0x7f751015e380 2024-03-24T00:49:03.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.085+0000 7f7514911640 1 -- 172.21.15.67:0/1185045948 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2345683253 0 0) 0x7f751006cc00 con 0x7f751006c440 2024-03-24T00:49:03.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.086+0000 7f7515913640 1 --2- 172.21.15.67:0/1185045948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f751015e380 0x7f75100d11f0 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f750c009870 tx=0x7f750c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79c96865c4a624a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:03.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.086+0000 7f7514911640 1 -- 172.21.15.67:0/1185045948 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f750c034070 con 0x7f751015e380 2024-03-24T00:49:03.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.086+0000 7f7514911640 1 -- 172.21.15.67:0/1185045948 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f750c030840 con 0x7f751015e380 2024-03-24T00:49:03.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.086+0000 7f7514911640 1 -- 172.21.15.67:0/1185045948 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f750c030c10 con 0x7f751015e380 2024-03-24T00:49:03.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.087+0000 7f7517b77640 1 -- 172.21.15.67:0/1185045948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f751015e380 msgr2=0x7f75100d11f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:03.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.087+0000 7f7517b77640 1 --2- 172.21.15.67:0/1185045948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f751015e380 0x7f75100d11f0 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f750c009870 tx=0x7f750c02b980 comp rx=0 tx=0).stop 2024-03-24T00:49:03.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.087+0000 7f7517b77640 1 -- 172.21.15.67:0/1185045948 shutdown_connections 2024-03-24T00:49:03.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.087+0000 7f7517b77640 1 --2- 172.21.15.67:0/1185045948 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7510099d70 0x7f751009a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:03.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.087+0000 7f7517b77640 1 --2- 172.21.15.67:0/1185045948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f751015e380 0x7f75100d11f0 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:03.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.087+0000 7f7517b77640 1 -- 172.21.15.67:0/1185045948 >> 172.21.15.67:0/1185045948 conn(0x7f75100b3660 msgr2=0x7f75100b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:03.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.087+0000 7f7517b77640 1 -- 172.21.15.67:0/1185045948 shutdown_connections 2024-03-24T00:49:03.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.087+0000 7f7517b77640 1 -- 172.21.15.67:0/1185045948 wait complete. 2024-03-24T00:49:03.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.088+0000 7f7517b77640 1 Processor -- start 2024-03-24T00:49:03.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.088+0000 7f7517b77640 1 -- start start 2024-03-24T00:49:03.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.088+0000 7f7517b77640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f751006c440 0x7f75100c6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:03.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.088+0000 7f7517b77640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7510099d70 0x7f75100c9c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:03.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.088+0000 7f7517b77640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f751015e380 0x7f75101f3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:03.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.089+0000 7f7517b77640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75100d1730 con 0x7f751015e380 2024-03-24T00:49:03.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.089+0000 7f7517b77640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75100d0cd0 con 0x7f751006c440 2024-03-24T00:49:03.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.089+0000 7f7517b77640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f751006d1c0 con 0x7f7510099d70 2024-03-24T00:49:03.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.089+0000 7f7516114640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f751015e380 0x7f75101f3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:03.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.089+0000 7f7516114640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f751015e380 0x7f75101f3d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57524/0 (socket says 172.21.15.67:57524) 2024-03-24T00:49:03.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.089+0000 7f7516114640 1 -- 172.21.15.67:0/3910195481 learned_addr learned my addr 172.21.15.67:0/3910195481 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:03.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.089+0000 7f7515913640 1 --2- 172.21.15.67:0/3910195481 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f751006c440 0x7f75100c6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:03.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.089+0000 7f7515112640 1 --2- 172.21.15.67:0/3910195481 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7510099d70 0x7f75100c9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:03.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.089+0000 7f7516114640 1 -- 172.21.15.67:0/3910195481 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7510099d70 msgr2=0x7f75100c9c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:03.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.089+0000 7f7516114640 1 --2- 172.21.15.67:0/3910195481 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7510099d70 0x7f75100c9c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:03.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.089+0000 7f7516114640 1 -- 172.21.15.67:0/3910195481 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f751006c440 msgr2=0x7f75100c6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:03.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.089+0000 7f7516114640 1 --2- 172.21.15.67:0/3910195481 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f751006c440 0x7f75100c6590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:03.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.089+0000 7f7516114640 1 -- 172.21.15.67:0/3910195481 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75100c5940 con 0x7f751015e380 2024-03-24T00:49:03.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.089+0000 7f7515913640 1 --2- 172.21.15.67:0/3910195481 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f751006c440 0x7f75100c6590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:03.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.089+0000 7f7515112640 1 --2- 172.21.15.67:0/3910195481 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7510099d70 0x7f75100c9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:03.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.089+0000 7f7516114640 1 --2- 172.21.15.67:0/3910195481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f751015e380 0x7f75101f3d00 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f7508007e90 tx=0x7f750800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:03.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.089+0000 7f75067fc640 1 -- 172.21.15.67:0/3910195481 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7508014420 con 0x7f751015e380 2024-03-24T00:49:03.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.089+0000 7f75067fc640 1 -- 172.21.15.67:0/3910195481 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7508019070 con 0x7f751015e380 2024-03-24T00:49:03.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.089+0000 7f7517b77640 1 -- 172.21.15.67:0/3910195481 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75100ca300 con 0x7f751015e380 2024-03-24T00:49:03.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.089+0000 7f75067fc640 1 -- 172.21.15.67:0/3910195481 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7508014800 con 0x7f751015e380 2024-03-24T00:49:03.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.089+0000 7f7517b77640 1 -- 172.21.15.67:0/3910195481 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75100ca8c0 con 0x7f751015e380 2024-03-24T00:49:03.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.090+0000 7f7517b77640 1 -- 172.21.15.67:0/3910195481 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f751004f400 con 0x7f751015e380 2024-03-24T00:49:03.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.091+0000 7f75067fc640 1 -- 172.21.15.67:0/3910195481 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f75080149a0 con 0x7f751015e380 2024-03-24T00:49:03.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.091+0000 7f75067fc640 1 --2- 172.21.15.67:0/3910195481 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f74f406bc00 0x7f74f406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:03.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.091+0000 7f75067fc640 1 -- 172.21.15.67:0/3910195481 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f750808b3a0 con 0x7f751015e380 2024-03-24T00:49:03.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.091+0000 7f7515913640 1 --2- 172.21.15.67:0/3910195481 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f74f406bc00 0x7f74f406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:03.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.092+0000 7f7515913640 1 --2- 172.21.15.67:0/3910195481 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f74f406bc00 0x7f74f406e0c0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f750c036000 tx=0x7f750c03b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:03.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.096+0000 7f75067fc640 1 -- 172.21.15.67:0/3910195481 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7508059d90 con 0x7f751015e380 2024-03-24T00:49:03.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.364+0000 7f7517b77640 1 -- 172.21.15.67:0/3910195481 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f75100b2a30 con 0x7f751015e380 2024-03-24T00:49:03.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.366+0000 7f75067fc640 1 -- 172.21.15.67:0/3910195481 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v20) v1 ==== 190+0+147 (secure 0 0 0) 0x7f750808b1c0 con 0x7f751015e380 2024-03-24T00:49:03.367 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-03-24T00:49:03.367 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQDcdf9lY4nvJRAA6HWOZDWA+SSvJVmyjG+51g== 2024-03-24T00:49:03.367 INFO:teuthology.orchestra.run.smithi067.stdout: caps mds = "allow" 2024-03-24T00:49:03.367 INFO:teuthology.orchestra.run.smithi067.stdout: caps mon = "allow r" 2024-03-24T00:49:03.367 INFO:teuthology.orchestra.run.smithi067.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-24T00:49:03.367 INFO:teuthology.orchestra.run.smithi067.stderr:updated caps for client.0 2024-03-24T00:49:03.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.368+0000 7f7517b77640 1 -- 172.21.15.67:0/3910195481 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f74f406bc00 msgr2=0x7f74f406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:03.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.368+0000 7f7517b77640 1 --2- 172.21.15.67:0/3910195481 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f74f406bc00 0x7f74f406e0c0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f750c036000 tx=0x7f750c03b040 comp rx=0 tx=0).stop 2024-03-24T00:49:03.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.368+0000 7f7517b77640 1 -- 172.21.15.67:0/3910195481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f751015e380 msgr2=0x7f75101f3d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:03.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.368+0000 7f7517b77640 1 --2- 172.21.15.67:0/3910195481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f751015e380 0x7f75101f3d00 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f7508007e90 tx=0x7f750800e4b0 comp rx=0 tx=0).stop 2024-03-24T00:49:03.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.368+0000 7f7517b77640 1 -- 172.21.15.67:0/3910195481 shutdown_connections 2024-03-24T00:49:03.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.368+0000 7f7517b77640 1 --2- 172.21.15.67:0/3910195481 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f74f406bc00 0x7f74f406e0c0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:03.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.368+0000 7f7517b77640 1 --2- 172.21.15.67:0/3910195481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f751015e380 0x7f75101f3d00 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:03.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.368+0000 7f7517b77640 1 --2- 172.21.15.67:0/3910195481 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7510099d70 0x7f75100c9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:03.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.368+0000 7f7517b77640 1 --2- 172.21.15.67:0/3910195481 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f751006c440 0x7f75100c6590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:03.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.368+0000 7f7517b77640 1 -- 172.21.15.67:0/3910195481 >> 172.21.15.67:0/3910195481 conn(0x7f75100b3660 msgr2=0x7f75100ccff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:03.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.368+0000 7f7517b77640 1 -- 172.21.15.67:0/3910195481 shutdown_connections 2024-03-24T00:49:03.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.368+0000 7f7517b77640 1 -- 172.21.15.67:0/3910195481 wait complete. 2024-03-24T00:49:03.380 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:49:03.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.480+0000 7fa38f0f1640 1 Processor -- start 2024-03-24T00:49:03.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.480+0000 7fa38f0f1640 1 -- start start 2024-03-24T00:49:03.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.481+0000 7fa38f0f1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa3880b2e80 0x7fa3880b32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:03.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.481+0000 7fa38f0f1640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa3880cf4a0 0x7fa3880d0d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:03.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.481+0000 7fa38f0f1640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa388162b40 con 0x7fa3880d0990 2024-03-24T00:49:03.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.481+0000 7fa38f0f1640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3880ce770 con 0x7fa3880cf4a0 2024-03-24T00:49:03.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.481+0000 7fa38f0f1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3880ce9c0 con 0x7fa3880b2e80 2024-03-24T00:49:03.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.481+0000 7fa38d68e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa3880cf4a0 0x7fa3880d0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:03.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.481+0000 7fa38d68e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa3880cf4a0 0x7fa3880d0d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57536/0 (socket says 172.21.15.67:57536) 2024-03-24T00:49:03.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.481+0000 7fa38d68e640 1 -- 172.21.15.67:0/2963693309 learned_addr learned my addr 172.21.15.67:0/2963693309 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:03.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.482+0000 7fa37ffff640 1 --2- 172.21.15.67:0/2963693309 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa3880b2e80 0x7fa3880b32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:03.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.482+0000 7fa38d68e640 1 -- 172.21.15.67:0/2963693309 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa3880b2e80 msgr2=0x7fa3880b32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:03.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.482+0000 7fa38d68e640 1 --2- 172.21.15.67:0/2963693309 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa3880b2e80 0x7fa3880b32f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:03.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.482+0000 7fa38d68e640 1 -- 172.21.15.67:0/2963693309 >> v1:172.21.15.72:6789/0 conn(0x7fa3880d0990 legacy=0x7fa38815dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:03.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.482+0000 7fa38d68e640 1 -- 172.21.15.67:0/2963693309 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3880ceca0 con 0x7fa3880cf4a0 2024-03-24T00:49:03.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.482+0000 7fa38d68e640 1 --2- 172.21.15.67:0/2963693309 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3880cf4a0 0x7fa3880d0d70 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7fa380009e50 tx=0x7fa38002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69e960aa89eee5ab server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:03.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.483+0000 7fa37f7fe640 1 -- 172.21.15.67:0/2963693309 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa38002ca70 con 0x7fa3880cf4a0 2024-03-24T00:49:03.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.483+0000 7fa37f7fe640 1 -- 172.21.15.67:0/2963693309 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa38002cc10 con 0x7fa3880cf4a0 2024-03-24T00:49:03.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.483+0000 7fa37f7fe640 1 -- 172.21.15.67:0/2963693309 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3800343f0 con 0x7fa3880cf4a0 2024-03-24T00:49:03.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.483+0000 7fa38f0f1640 1 -- 172.21.15.67:0/2963693309 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3880cf4a0 msgr2=0x7fa3880d0d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:03.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.483+0000 7fa38f0f1640 1 --2- 172.21.15.67:0/2963693309 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3880cf4a0 0x7fa3880d0d70 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7fa380009e50 tx=0x7fa38002bed0 comp rx=0 tx=0).stop 2024-03-24T00:49:03.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.483+0000 7fa38f0f1640 1 -- 172.21.15.67:0/2963693309 shutdown_connections 2024-03-24T00:49:03.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.483+0000 7fa38f0f1640 1 --2- 172.21.15.67:0/2963693309 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa3880b2e80 0x7fa3880b32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:03.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.483+0000 7fa38f0f1640 1 --2- 172.21.15.67:0/2963693309 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3880cf4a0 0x7fa3880d0d70 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:03.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.483+0000 7fa38f0f1640 1 -- 172.21.15.67:0/2963693309 >> 172.21.15.67:0/2963693309 conn(0x7fa38806c830 msgr2=0x7fa38806cc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:03.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.483+0000 7fa38f0f1640 1 -- 172.21.15.67:0/2963693309 shutdown_connections 2024-03-24T00:49:03.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.483+0000 7fa38f0f1640 1 -- 172.21.15.67:0/2963693309 wait complete. 2024-03-24T00:49:03.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.484+0000 7fa38f0f1640 1 Processor -- start 2024-03-24T00:49:03.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.484+0000 7fa38f0f1640 1 -- start start 2024-03-24T00:49:03.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.485+0000 7fa38f0f1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa3880b2e80 0x7fa38814d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:03.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.485+0000 7fa38f0f1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3880cf4a0 0x7fa3880c80e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:03.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.485+0000 7fa38f0f1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3880d0990 0x7fa388157f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:03.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.485+0000 7fa38f0f1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa38815e380 con 0x7fa3880d0990 2024-03-24T00:49:03.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.485+0000 7fa38f0f1640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3881627c0 con 0x7fa3880cf4a0 2024-03-24T00:49:03.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.485+0000 7fa38f0f1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa38815e500 con 0x7fa3880b2e80 2024-03-24T00:49:03.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.485+0000 7fa38d68e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3880d0990 0x7fa388157f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:03.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.485+0000 7fa38d68e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3880d0990 0x7fa388157f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42290/0 (socket says 172.21.15.67:42290) 2024-03-24T00:49:03.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.485+0000 7fa38d68e640 1 -- 172.21.15.67:0/2840155848 learned_addr learned my addr 172.21.15.67:0/2840155848 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:03.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.485+0000 7fa37ffff640 1 --2- 172.21.15.67:0/2840155848 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3880cf4a0 0x7fa3880c80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:03.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.485+0000 7fa38ce8d640 1 --2- 172.21.15.67:0/2840155848 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa3880b2e80 0x7fa38814d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:03.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.486+0000 7fa38d68e640 1 -- 172.21.15.67:0/2840155848 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa3880b2e80 msgr2=0x7fa38814d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:03.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.486+0000 7fa38d68e640 1 --2- 172.21.15.67:0/2840155848 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa3880b2e80 0x7fa38814d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:03.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.486+0000 7fa38d68e640 1 -- 172.21.15.67:0/2840155848 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3880cf4a0 msgr2=0x7fa3880c80e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:03.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.486+0000 7fa38d68e640 1 --2- 172.21.15.67:0/2840155848 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3880cf4a0 0x7fa3880c80e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:03.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.486+0000 7fa38d68e640 1 -- 172.21.15.67:0/2840155848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3880c9980 con 0x7fa3880d0990 2024-03-24T00:49:03.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.486+0000 7fa38d68e640 1 --2- 172.21.15.67:0/2840155848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3880d0990 0x7fa388157f10 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7fa380009e50 tx=0x7fa380004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:03.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.486+0000 7fa37d7fa640 1 -- 172.21.15.67:0/2840155848 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa380043020 con 0x7fa3880d0990 2024-03-24T00:49:03.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.486+0000 7fa37d7fa640 1 -- 172.21.15.67:0/2840155848 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa38003e450 con 0x7fa3880d0990 2024-03-24T00:49:03.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.486+0000 7fa38f0f1640 1 -- 172.21.15.67:0/2840155848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3880c9c60 con 0x7fa3880d0990 2024-03-24T00:49:03.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.486+0000 7fa37d7fa640 1 -- 172.21.15.67:0/2840155848 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa380009510 con 0x7fa3880d0990 2024-03-24T00:49:03.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.486+0000 7fa38f0f1640 1 -- 172.21.15.67:0/2840155848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa38815fbb0 con 0x7fa3880d0990 2024-03-24T00:49:03.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.487+0000 7fa38f0f1640 1 -- 172.21.15.67:0/2840155848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa38804f400 con 0x7fa3880d0990 2024-03-24T00:49:03.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.488+0000 7fa37d7fa640 1 -- 172.21.15.67:0/2840155848 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa380046020 con 0x7fa3880d0990 2024-03-24T00:49:03.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.488+0000 7fa37d7fa640 1 --2- 172.21.15.67:0/2840155848 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa37406bbd0 0x7fa37406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:03.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.488+0000 7fa37d7fa640 1 -- 172.21.15.67:0/2840155848 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa3800ad310 con 0x7fa3880d0990 2024-03-24T00:49:03.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.488+0000 7fa38ce8d640 1 --2- 172.21.15.67:0/2840155848 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa37406bbd0 0x7fa37406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:03.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.489+0000 7fa38ce8d640 1 --2- 172.21.15.67:0/2840155848 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa37406bbd0 0x7fa37406e090 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fa384006fd0 tx=0x7fa384009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:03.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.493+0000 7fa37d7fa640 1 -- 172.21.15.67:0/2840155848 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa38004d030 con 0x7fa3880d0990 2024-03-24T00:49:03.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.713+0000 7fa38f0f1640 1 -- 172.21.15.67:0/2840155848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3880d1be0 con 0x7fa3880d0990 2024-03-24T00:49:03.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.713+0000 7fa37d7fa640 1 -- 172.21.15.67:0/2840155848 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+4973 (secure 0 0 0) 0x7fa38007b620 con 0x7fa3880d0990 2024-03-24T00:49:03.714 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:03.714 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":86,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7298,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/839277571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":839277571},{"type":"v1","addr":"172.21.15.72:6836","nonce":839277571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":8224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2052380377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2052380377},{"type":"v1","addr":"172.21.15.67:6834","nonce":2052380377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":8239,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2196618204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2196618204},{"type":"v1","addr":"172.21.15.72:6837","nonce":2196618204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:48:54.351575+0000","modified":"2024-03-24T00:48:56.351565+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9219},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9219":{"gid":9219,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/312063805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":312063805},{"type":"v1","addr":"172.21.15.67:6835","nonce":312063805}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9219,"qdb_cluster":[9219]},"id":8}]} 2024-03-24T00:49:03.715 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 86 2024-03-24T00:49:03.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.715+0000 7fa38f0f1640 1 -- 172.21.15.67:0/2840155848 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa37406bbd0 msgr2=0x7fa37406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:03.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.715+0000 7fa38f0f1640 1 --2- 172.21.15.67:0/2840155848 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa37406bbd0 0x7fa37406e090 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fa384006fd0 tx=0x7fa384009060 comp rx=0 tx=0).stop 2024-03-24T00:49:03.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.715+0000 7fa38f0f1640 1 -- 172.21.15.67:0/2840155848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3880d0990 msgr2=0x7fa388157f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:03.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.715+0000 7fa38f0f1640 1 --2- 172.21.15.67:0/2840155848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3880d0990 0x7fa388157f10 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7fa380009e50 tx=0x7fa380004a40 comp rx=0 tx=0).stop 2024-03-24T00:49:03.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.716+0000 7fa38f0f1640 1 -- 172.21.15.67:0/2840155848 shutdown_connections 2024-03-24T00:49:03.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.716+0000 7fa38f0f1640 1 --2- 172.21.15.67:0/2840155848 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa37406bbd0 0x7fa37406e090 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:03.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.716+0000 7fa38f0f1640 1 --2- 172.21.15.67:0/2840155848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3880d0990 0x7fa388157f10 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:03.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.716+0000 7fa38f0f1640 1 --2- 172.21.15.67:0/2840155848 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3880cf4a0 0x7fa3880c80e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:03.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.716+0000 7fa38f0f1640 1 --2- 172.21.15.67:0/2840155848 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa3880b2e80 0x7fa38814d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:03.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.716+0000 7fa38f0f1640 1 -- 172.21.15.67:0/2840155848 >> 172.21.15.67:0/2840155848 conn(0x7fa38806c830 msgr2=0x7fa3880b44d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:03.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.716+0000 7fa38f0f1640 1 -- 172.21.15.67:0/2840155848 shutdown_connections 2024-03-24T00:49:03.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:03.716+0000 7fa38f0f1640 1 -- 172.21.15.67:0/2840155848 wait complete. 2024-03-24T00:49:04.130 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 86, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:48:54.351575+0000', 'modified': '2024-03-24T00:48:56.351565+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9219}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9219': {'gid': 9219, 'name': 'a', 'rank': 0, 'incarnation': 84, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.67:6835/312063805', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6833', 'nonce': 312063805}, {'type': 'v1', 'addr': '172.21.15.67:6835', 'nonce': 312063805}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9219, 'qdb_cluster': [9219]} 2024-03-24T00:49:04.130 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-24T00:49:04.130 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-24T00:49:04.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.232+0000 7f50d10f8640 1 Processor -- start 2024-03-24T00:49:04.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.232+0000 7f50d10f8640 1 -- start start 2024-03-24T00:49:04.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.233+0000 7f50d10f8640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f50cc15e0f0 0x7f50cc0b95b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:04.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.233+0000 7f50d10f8640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f50cc0b6dc0 0x7f50cc0b7230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:04.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.233+0000 7f50d10f8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50cc0d2230 con 0x7f50cc0b9af0 2024-03-24T00:49:04.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.233+0000 7f50d10f8640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50cc0b7770 con 0x7f50cc0b6dc0 2024-03-24T00:49:04.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.233+0000 7f50d10f8640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50cc0d2510 con 0x7f50cc15e0f0 2024-03-24T00:49:04.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.233+0000 7f50ca59c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f50cc0b9af0 0x7f50cc0d32d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48676/0 (socket says 172.21.15.67:48676) 2024-03-24T00:49:04.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.234+0000 7f50ca59c640 1 -- 172.21.15.67:0/1176320109 learned_addr learned my addr 172.21.15.67:0/1176320109 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:49:04.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.234+0000 7f50cb59e640 1 --2- 172.21.15.67:0/1176320109 >> v2:172.21.15.72:3300/0 conn(0x7f50cc0b6dc0 0x7f50cc0b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:04.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.234+0000 7f50cad9d640 1 --2- 172.21.15.67:0/1176320109 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f50cc15e0f0 0x7f50cc0b95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:04.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.234+0000 7f50c9d9b640 1 -- 172.21.15.67:0/1176320109 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1524471055 0 0) 0x7f50cc0d2230 con 0x7f50cc0b9af0 2024-03-24T00:49:04.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.234+0000 7f50c9d9b640 1 -- 172.21.15.67:0/1176320109 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50b4003520 con 0x7f50cc0b9af0 2024-03-24T00:49:04.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.234+0000 7f50cb59e640 1 -- 172.21.15.67:0/1176320109 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f50cc15e0f0 msgr2=0x7f50cc0b95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:04.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.234+0000 7f50cb59e640 1 --2- 172.21.15.67:0/1176320109 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f50cc15e0f0 0x7f50cc0b95b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:04.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.234+0000 7f50cb59e640 1 -- 172.21.15.67:0/1176320109 >> v1:172.21.15.67:6789/0 conn(0x7f50cc0b9af0 legacy=0x7f50cc0d32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:04.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.234+0000 7f50cb59e640 1 -- 172.21.15.67:0/1176320109 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50cc0d27f0 con 0x7f50cc0b6dc0 2024-03-24T00:49:04.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.235+0000 7f50cb59e640 1 --2- 172.21.15.67:0/1176320109 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50cc0b6dc0 0x7f50cc0b7230 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f50c4009d90 tx=0x7f50c402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0994c30de4f8971 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:04.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.235+0000 7f50c9d9b640 1 -- 172.21.15.67:0/1176320109 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c4039070 con 0x7f50cc0b6dc0 2024-03-24T00:49:04.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.235+0000 7f50c9d9b640 1 -- 172.21.15.67:0/1176320109 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50c402cc50 con 0x7f50cc0b6dc0 2024-03-24T00:49:04.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.235+0000 7f50c9d9b640 1 -- 172.21.15.67:0/1176320109 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c4034440 con 0x7f50cc0b6dc0 2024-03-24T00:49:04.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.235+0000 7f50d10f8640 1 -- 172.21.15.67:0/1176320109 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50cc0b6dc0 msgr2=0x7f50cc0b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:04.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.235+0000 7f50d10f8640 1 --2- 172.21.15.67:0/1176320109 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50cc0b6dc0 0x7f50cc0b7230 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f50c4009d90 tx=0x7f50c402bea0 comp rx=0 tx=0).stop 2024-03-24T00:49:04.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.236+0000 7f50d10f8640 1 -- 172.21.15.67:0/1176320109 shutdown_connections 2024-03-24T00:49:04.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.236+0000 7f50d10f8640 1 --2- 172.21.15.67:0/1176320109 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50cc0b6dc0 0x7f50cc0b7230 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:04.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.236+0000 7f50d10f8640 1 --2- 172.21.15.67:0/1176320109 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f50cc15e0f0 0x7f50cc0b95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:04.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.236+0000 7f50d10f8640 1 -- 172.21.15.67:0/1176320109 >> 172.21.15.67:0/1176320109 conn(0x7f50cc072b60 msgr2=0x7f50cc072f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:04.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.236+0000 7f50d10f8640 1 -- 172.21.15.67:0/1176320109 shutdown_connections 2024-03-24T00:49:04.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.236+0000 7f50d10f8640 1 -- 172.21.15.67:0/1176320109 wait complete. 2024-03-24T00:49:04.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.237+0000 7f50d10f8640 1 Processor -- start 2024-03-24T00:49:04.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.237+0000 7f50d10f8640 1 -- start start 2024-03-24T00:49:04.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.238+0000 7f50d10f8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50cc0b6dc0 0x7f50cc14e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:04.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.238+0000 7f50d10f8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50cc0b9af0 0x7f50cc15e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:04.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.238+0000 7f50d10f8640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f50cc15e0f0 0x7f50cc0c9290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:04.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.238+0000 7f50d10f8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50cc0b31d0 con 0x7f50cc0b9af0 2024-03-24T00:49:04.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.238+0000 7f50d10f8640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50cc0d2db0 con 0x7f50cc0b6dc0 2024-03-24T00:49:04.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.238+0000 7f50d10f8640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50cc0b3350 con 0x7f50cc15e0f0 2024-03-24T00:49:04.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.238+0000 7f50ca59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50cc0b9af0 0x7f50cc15e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:04.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.238+0000 7f50ca59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50cc0b9af0 0x7f50cc15e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42306/0 (socket says 172.21.15.67:42306) 2024-03-24T00:49:04.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.238+0000 7f50ca59c640 1 -- 172.21.15.67:0/3037965365 learned_addr learned my addr 172.21.15.67:0/3037965365 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:04.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.238+0000 7f50cad9d640 1 --2- 172.21.15.67:0/3037965365 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50cc0b6dc0 0x7f50cc14e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:04.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.238+0000 7f50cb59e640 1 --2- 172.21.15.67:0/3037965365 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f50cc15e0f0 0x7f50cc0c9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:04.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.238+0000 7f50ca59c640 1 -- 172.21.15.67:0/3037965365 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f50cc15e0f0 msgr2=0x7f50cc0c9290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:04.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.238+0000 7f50ca59c640 1 --2- 172.21.15.67:0/3037965365 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f50cc15e0f0 0x7f50cc0c9290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:04.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.238+0000 7f50ca59c640 1 -- 172.21.15.67:0/3037965365 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50cc0b6dc0 msgr2=0x7f50cc14e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:04.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.238+0000 7f50ca59c640 1 --2- 172.21.15.67:0/3037965365 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50cc0b6dc0 0x7f50cc14e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:04.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.238+0000 7f50ca59c640 1 -- 172.21.15.67:0/3037965365 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50cc0caf30 con 0x7f50cc0b9af0 2024-03-24T00:49:04.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.238+0000 7f50ca59c640 1 --2- 172.21.15.67:0/3037965365 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50cc0b9af0 0x7f50cc15e4d0 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f50bc00c730 tx=0x7f50bc00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:04.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.239+0000 7f50b37fe640 1 -- 172.21.15.67:0/3037965365 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50bc017070 con 0x7f50cc0b9af0 2024-03-24T00:49:04.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.239+0000 7f50b37fe640 1 -- 172.21.15.67:0/3037965365 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50bc007290 con 0x7f50cc0b9af0 2024-03-24T00:49:04.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.239+0000 7f50b37fe640 1 -- 172.21.15.67:0/3037965365 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50bc001d40 con 0x7f50cc0b9af0 2024-03-24T00:49:04.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.239+0000 7f50d10f8640 1 -- 172.21.15.67:0/3037965365 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50cc0cb210 con 0x7f50cc0b9af0 2024-03-24T00:49:04.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.239+0000 7f50d10f8640 1 -- 172.21.15.67:0/3037965365 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50cc0cb7d0 con 0x7f50cc0b9af0 2024-03-24T00:49:04.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.240+0000 7f50b37fe640 1 -- 172.21.15.67:0/3037965365 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f50bc01a070 con 0x7f50cc0b9af0 2024-03-24T00:49:04.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.241+0000 7f50b37fe640 1 --2- 172.21.15.67:0/3037965365 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f50b806bbd0 0x7f50b806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:04.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.241+0000 7f50b37fe640 1 -- 172.21.15.67:0/3037965365 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f50bc089f80 con 0x7f50cc0b9af0 2024-03-24T00:49:04.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.241+0000 7f50cad9d640 1 --2- 172.21.15.67:0/3037965365 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f50b806bbd0 0x7f50b806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:04.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.241+0000 7f50cad9d640 1 --2- 172.21.15.67:0/3037965365 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f50b806bbd0 0x7f50b806e090 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f50c0009aa0 tx=0x7f50c0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:04.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.242+0000 7f50d3dc4740 1 Processor -- start 2024-03-24T00:49:04.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.242+0000 7f50d3dc4740 1 -- start start 2024-03-24T00:49:04.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.242+0000 7f50d3dc4740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55980384ac40 0x559803871770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:04.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.242+0000 7f50d3dc4740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559803871cb0 0x559803849c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:04.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.242+0000 7f50d3dc4740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55980384b020 0x55980384a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:04.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.242+0000 7f50d3dc4740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598037aec20 con 0x559803871cb0 2024-03-24T00:49:04.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.243+0000 7f50d3dc4740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x559803757a40 con 0x55980384b020 2024-03-24T00:49:04.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.243+0000 7f50d3dc4740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5598037aacc0 con 0x55980384ac40 2024-03-24T00:49:04.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.243+0000 7f50ca59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559803871cb0 0x559803849c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:04.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.243+0000 7f50ca59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559803871cb0 0x559803849c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42314/0 (socket says 172.21.15.67:42314) 2024-03-24T00:49:04.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.243+0000 7f50ca59c640 1 -- 172.21.15.67:0/1329866849 learned_addr learned my addr 172.21.15.67:0/1329866849 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:04.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.243+0000 7f50cb59e640 1 --2- 172.21.15.67:0/1329866849 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55980384b020 0x55980384a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:04.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.243+0000 7f50cad9d640 1 --2- 172.21.15.67:0/1329866849 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55980384ac40 0x559803871770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:04.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.243+0000 7f50ca59c640 1 -- 172.21.15.67:0/1329866849 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55980384ac40 msgr2=0x559803871770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:04.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.243+0000 7f50ca59c640 1 --2- 172.21.15.67:0/1329866849 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55980384ac40 0x559803871770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:04.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.243+0000 7f50ca59c640 1 -- 172.21.15.67:0/1329866849 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55980384b020 msgr2=0x55980384a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:04.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.243+0000 7f50ca59c640 1 --2- 172.21.15.67:0/1329866849 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55980384b020 0x55980384a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:04.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.243+0000 7f50ca59c640 1 -- 172.21.15.67:0/1329866849 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559803779f90 con 0x559803871cb0 2024-03-24T00:49:04.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.243+0000 7f50cb59e640 1 --2- 172.21.15.67:0/1329866849 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55980384b020 0x55980384a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:04.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.243+0000 7f50ca59c640 1 --2- 172.21.15.67:0/1329866849 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559803871cb0 0x559803849c80 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f50bc088320 tx=0x7f50bc058350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d81a313ddec86414 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:04.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.243+0000 7f50b0ff9640 1 -- 172.21.15.67:0/1329866849 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50bc0562b0 con 0x559803871cb0 2024-03-24T00:49:04.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.243+0000 7f50b0ff9640 1 -- 172.21.15.67:0/1329866849 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50bc0770c0 con 0x559803871cb0 2024-03-24T00:49:04.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.243+0000 7f50b0ff9640 1 -- 172.21.15.67:0/1329866849 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50bc054880 con 0x559803871cb0 2024-03-24T00:49:04.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.244+0000 7f50d3dc4740 1 -- 172.21.15.67:0/1329866849 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559803871cb0 msgr2=0x559803849c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:04.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.244+0000 7f50d3dc4740 1 --2- 172.21.15.67:0/1329866849 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559803871cb0 0x559803849c80 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f50bc088320 tx=0x7f50bc058350 comp rx=0 tx=0).stop 2024-03-24T00:49:04.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.244+0000 7f50d3dc4740 1 -- 172.21.15.67:0/1329866849 shutdown_connections 2024-03-24T00:49:04.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.244+0000 7f50d3dc4740 1 --2- 172.21.15.67:0/1329866849 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55980384b020 0x55980384a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:04.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.244+0000 7f50d3dc4740 1 --2- 172.21.15.67:0/1329866849 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559803871cb0 0x559803849c80 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:04.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.244+0000 7f50d3dc4740 1 --2- 172.21.15.67:0/1329866849 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55980384ac40 0x559803871770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:04.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.244+0000 7f50d3dc4740 1 -- 172.21.15.67:0/1329866849 >> 172.21.15.67:0/1329866849 conn(0x559803811a60 msgr2=0x55980388f960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:04.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.244+0000 7f50d3dc4740 1 -- 172.21.15.67:0/1329866849 shutdown_connections 2024-03-24T00:49:04.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.244+0000 7f50d3dc4740 1 -- 172.21.15.67:0/1329866849 wait complete. 2024-03-24T00:49:04.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.245+0000 7f50d3dc4740 1 -- start start 2024-03-24T00:49:04.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.245+0000 7f50d3dc4740 1 Processor -- start 2024-03-24T00:49:04.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.245+0000 7f50d3dc4740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55980384ac40 0x55980388d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:04.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.245+0000 7f50d3dc4740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55980384b020 0x559803878260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:04.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.245+0000 7f50d3dc4740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55980389fa80 0x559803868dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:04.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.245+0000 7f50d3dc4740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598038712e0 con 0x55980384ac40 2024-03-24T00:49:04.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.245+0000 7f50d3dc4740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55980376ae90 con 0x55980389fa80 2024-03-24T00:49:04.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.245+0000 7f50d3dc4740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55980386a060 con 0x55980384b020 2024-03-24T00:49:04.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.245+0000 7f50d3dc4740 10 client.0 _ll_register_callbacks cb 0x5598034c2ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:49:04.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.245+0000 7f50cad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55980384ac40 0x55980388d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:04.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.245+0000 7f50cad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55980384ac40 0x55980388d270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42324/0 (socket says 172.21.15.67:42324) 2024-03-24T00:49:04.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.245+0000 7f50cad9d640 1 -- 172.21.15.67:0/3014912343 learned_addr learned my addr 172.21.15.67:0/3014912343 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:04.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.246+0000 7f50cb59e640 1 --2- 172.21.15.67:0/3014912343 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55980389fa80 0x559803868dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:04.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.246+0000 7f50ca59c640 1 --2- 172.21.15.67:0/3014912343 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55980384b020 0x559803878260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:04.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.246+0000 7f50cad9d640 1 -- 172.21.15.67:0/3014912343 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55980384b020 msgr2=0x559803878260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:04.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.246+0000 7f50cad9d640 1 --2- 172.21.15.67:0/3014912343 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55980384b020 0x559803878260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:04.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.246+0000 7f50cad9d640 1 -- 172.21.15.67:0/3014912343 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55980389fa80 msgr2=0x559803868dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:04.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.246+0000 7f50cad9d640 1 --2- 172.21.15.67:0/3014912343 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55980389fa80 0x559803868dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:04.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.246+0000 7f50cad9d640 1 -- 172.21.15.67:0/3014912343 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5598037779d0 con 0x55980384ac40 2024-03-24T00:49:04.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.246+0000 7f50ca59c640 1 --2- 172.21.15.67:0/3014912343 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55980384b020 0x559803878260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:04.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.246+0000 7f50cb59e640 1 --2- 172.21.15.67:0/3014912343 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55980389fa80 0x559803868dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:04.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.246+0000 7f50cad9d640 1 --2- 172.21.15.67:0/3014912343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55980384ac40 0x55980388d270 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f50c0012010 tx=0x7f50c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=810567a10f99a76d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:04.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.246+0000 7f50a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:49:04.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.246+0000 7f50a9ffb640 1 -- 172.21.15.67:0/3014912343 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c001e050 con 0x55980384ac40 2024-03-24T00:49:04.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.246+0000 7f50a9ffb640 1 -- 172.21.15.67:0/3014912343 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(120..120 src has 1..120) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f50c0003da0 con 0x55980384ac40 2024-03-24T00:49:04.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.246+0000 7f50d3dc4740 1 -- 172.21.15.67:0/3014912343 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f50bc0770c0 con 0x55980384ac40 2024-03-24T00:49:04.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.246+0000 7f50a9ffb640 1 -- 172.21.15.67:0/3014912343 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f50c0030050 con 0x55980384ac40 2024-03-24T00:49:04.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.246+0000 7f50d3dc4740 10 client.9399 fetch_fsmap learned FSMap version 86 2024-03-24T00:49:04.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.246+0000 7f50d3dc4740 1 -- 172.21.15.67:0/3014912343 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=86,monmap=2+}) v3 -- 0x5598034c32d0 con 0x55980384ac40 2024-03-24T00:49:04.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.247+0000 7f50a9ffb640 1 -- 172.21.15.67:0/3014912343 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50bc0770c0 con 0x55980384ac40 2024-03-24T00:49:04.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.247+0000 7f50a9ffb640 1 -- 172.21.15.67:0/3014912343 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 86) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f50c0033020 con 0x55980384ac40 2024-03-24T00:49:04.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.247+0000 7f50d3dc4740 10 client.9399 fetch_fsmap finished waiting for FSMap version 86 2024-03-24T00:49:04.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.247+0000 7f50d3dc4740 10 client.9399 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-24T00:49:04.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.247+0000 7f50d3dc4740 20 client.9399 populate_metadata read hostname 'smithi067' 2024-03-24T00:49:04.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.247+0000 7f50d3dc4740 1 --2- 172.21.15.67:0/3014912343 >> [v2:172.21.15.67:6833/312063805,v1:172.21.15.67:6835/312063805] conn(0x5598038ab4d0 0x5598038ad8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:04.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.247+0000 7f50d3dc4740 4 client.9399 mds_command: new command op to 9219 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:49:04.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.247+0000 7f50d3dc4740 1 -- 172.21.15.67:0/3014912343 --> [v2:172.21.15.67:6833/312063805,v1:172.21.15.67:6835/312063805] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5598037bc160 con 0x5598038ab4d0 2024-03-24T00:49:04.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.247+0000 7f50ca59c640 1 --2- 172.21.15.67:0/3014912343 >> [v2:172.21.15.67:6833/312063805,v1:172.21.15.67:6835/312063805] conn(0x5598038ab4d0 0x5598038ad8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:04.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.247+0000 7f50a9ffb640 10 client.9399 ms_handle_connect on v2:172.21.15.67:6833/312063805 2024-03-24T00:49:04.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.247+0000 7f50ca59c640 1 --2- 172.21.15.67:0/3014912343 >> [v2:172.21.15.67:6833/312063805,v1:172.21.15.67:6835/312063805] conn(0x5598038ab4d0 0x5598038ad8c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=817b83fe224171c8 server_cookie=5d069d6d4679f02c in_seq=0 out_seq=0 2024-03-24T00:49:04.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.248+0000 7f50a9ffb640 1 -- 172.21.15.67:0/3014912343 <== mds.0 v2:172.21.15.67:6833/312063805 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5598037bc160 con 0x5598038ab4d0 2024-03-24T00:49:04.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.248+0000 7f50a9ffb640 10 client.9399 handle_command_reply: tid=0 2024-03-24T00:49:04.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.248+0000 7f50d3dc4740 1 client.9399 shutdown 2024-03-24T00:49:04.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.248+0000 7f50d3dc4740 1 -- 172.21.15.67:0/3014912343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55980384ac40 msgr2=0x55980388d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:04.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.248+0000 7f50d3dc4740 1 --2- 172.21.15.67:0/3014912343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55980384ac40 0x55980388d270 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f50c0012010 tx=0x7f50c00046c0 comp rx=0 tx=0).stop 2024-03-24T00:49:04.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.248+0000 7f50d3dc4740 1 -- 172.21.15.67:0/3014912343 shutdown_connections 2024-03-24T00:49:04.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.248+0000 7f50d3dc4740 1 --2- 172.21.15.67:0/3014912343 >> [v2:172.21.15.67:6833/312063805,v1:172.21.15.67:6835/312063805] conn(0x5598038ab4d0 0x5598038ad8c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:04.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.248+0000 7f50d3dc4740 1 --2- 172.21.15.67:0/3014912343 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55980389fa80 0x559803868dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:04.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.248+0000 7f50d3dc4740 1 --2- 172.21.15.67:0/3014912343 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55980384b020 0x559803878260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:04.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.248+0000 7f50a9ffb640 0 client.9399 ms_handle_reset on v2:172.21.15.67:6833/312063805 2024-03-24T00:49:04.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.248+0000 7f50d3dc4740 1 --2- 172.21.15.67:0/3014912343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55980384ac40 0x55980388d270 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:04.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.248+0000 7f50d3dc4740 1 -- 172.21.15.67:0/3014912343 >> 172.21.15.67:0/3014912343 conn(0x559803811a60 msgr2=0x5598038486d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:04.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.248+0000 7f50d3dc4740 1 -- 172.21.15.67:0/3014912343 shutdown_connections 2024-03-24T00:49:04.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.248+0000 7f50d3dc4740 1 -- 172.21.15.67:0/3014912343 wait complete. 2024-03-24T00:49:04.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.248+0000 7f50d3dc4740 20 client.9399 trim_cache size 0 max 16384 2024-03-24T00:49:04.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.265+0000 7f50d3dc4740 1 Processor -- start 2024-03-24T00:49:04.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.265+0000 7f50d3dc4740 1 -- start start 2024-03-24T00:49:04.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.266+0000 7f50d3dc4740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55980384ac40 0x55980388cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:04.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.266+0000 7f50d3dc4740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55980384b020 0x5598038b18e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:04.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.266+0000 7f50d3dc4740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55980389fa80 0x5598038b1e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:04.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.266+0000 7f50d3dc4740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50cc1ff8a0 con 0x55980389fa80 2024-03-24T00:49:04.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.266+0000 7f50d3dc4740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50cc0cd000 con 0x55980384ac40 2024-03-24T00:49:04.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.266+0000 7f50d3dc4740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50cc0cd600 con 0x55980384b020 2024-03-24T00:49:04.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.266+0000 7f50cb59e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55980389fa80 0x5598038b1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:04.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.266+0000 7f50cb59e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55980389fa80 0x5598038b1e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42330/0 (socket says 172.21.15.67:42330) 2024-03-24T00:49:04.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.266+0000 7f50cb59e640 1 -- 172.21.15.67:0/1597347772 learned_addr learned my addr 172.21.15.67:0/1597347772 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:04.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.266+0000 7f50cad9d640 1 --2- 172.21.15.67:0/1597347772 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55980384ac40 0x55980388cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:04.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.266+0000 7f50ca59c640 1 --2- 172.21.15.67:0/1597347772 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55980384b020 0x5598038b18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:04.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.266+0000 7f50cb59e640 1 -- 172.21.15.67:0/1597347772 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55980384b020 msgr2=0x5598038b18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:04.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.266+0000 7f50cb59e640 1 --2- 172.21.15.67:0/1597347772 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55980384b020 0x5598038b18e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:04.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.266+0000 7f50cb59e640 1 -- 172.21.15.67:0/1597347772 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55980384ac40 msgr2=0x55980388cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:04.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.266+0000 7f50cb59e640 1 --2- 172.21.15.67:0/1597347772 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55980384ac40 0x55980388cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:04.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.266+0000 7f50cb59e640 1 -- 172.21.15.67:0/1597347772 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559803871ae0 con 0x55980389fa80 2024-03-24T00:49:04.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.266+0000 7f50cad9d640 1 --2- 172.21.15.67:0/1597347772 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55980384ac40 0x55980388cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:49:04.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.267+0000 7f50cb59e640 1 --2- 172.21.15.67:0/1597347772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55980389fa80 0x5598038b1e20 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f50c402bee0 tx=0x7f50c402e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36de7424b1bd3d47 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:04.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.267+0000 7f50abfff640 1 -- 172.21.15.67:0/1597347772 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c4045070 con 0x55980389fa80 2024-03-24T00:49:04.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.267+0000 7f50abfff640 1 -- 172.21.15.67:0/1597347772 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50c403a040 con 0x55980389fa80 2024-03-24T00:49:04.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.267+0000 7f50abfff640 1 -- 172.21.15.67:0/1597347772 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c4039040 con 0x55980389fa80 2024-03-24T00:49:04.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.267+0000 7f50d3dc4740 1 -- 172.21.15.67:0/1597347772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55980389fa80 msgr2=0x5598038b1e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:04.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.267+0000 7f50d3dc4740 1 --2- 172.21.15.67:0/1597347772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55980389fa80 0x5598038b1e20 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f50c402bee0 tx=0x7f50c402e000 comp rx=0 tx=0).stop 2024-03-24T00:49:04.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.267+0000 7f50d3dc4740 1 -- 172.21.15.67:0/1597347772 shutdown_connections 2024-03-24T00:49:04.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.267+0000 7f50d3dc4740 1 --2- 172.21.15.67:0/1597347772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55980389fa80 0x5598038b1e20 unknown :-1 s=CLOSED pgs=1736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:04.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.267+0000 7f50d3dc4740 1 --2- 172.21.15.67:0/1597347772 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55980384b020 0x5598038b18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:04.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.267+0000 7f50d3dc4740 1 --2- 172.21.15.67:0/1597347772 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55980384ac40 0x55980388cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:04.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.267+0000 7f50d3dc4740 1 -- 172.21.15.67:0/1597347772 >> 172.21.15.67:0/1597347772 conn(0x559803811a60 msgr2=0x559803870280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:04.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.267+0000 7f50d3dc4740 1 -- 172.21.15.67:0/1597347772 shutdown_connections 2024-03-24T00:49:04.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.267+0000 7f50d3dc4740 1 -- 172.21.15.67:0/1597347772 wait complete. 2024-03-24T00:49:04.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.268+0000 7f50d3dc4740 1 -- start start 2024-03-24T00:49:04.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.268+0000 7f50d3dc4740 1 Processor -- start 2024-03-24T00:49:04.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.269+0000 7f50d3dc4740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55980384ac40 0x5598038fe990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:04.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.269+0000 7f50d3dc4740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55980384b020 0x559803844300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:04.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.269+0000 7f50d3dc4740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55980389fa80 0x55980386d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:04.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.269+0000 7f50d3dc4740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55980388f2b0 con 0x55980389fa80 2024-03-24T00:49:04.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.269+0000 7f50d3dc4740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55980389ff90 con 0x55980384b020 2024-03-24T00:49:04.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.269+0000 7f50d3dc4740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x559803876d50 con 0x55980384ac40 2024-03-24T00:49:04.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.270+0000 7f50d3dc4740 10 client.0 _ll_register_callbacks cb 0x7f50c0011c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:49:04.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.270+0000 7f50cb59e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55980389fa80 0x55980386d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:04.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.270+0000 7f50cb59e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55980389fa80 0x55980386d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42336/0 (socket says 172.21.15.67:42336) 2024-03-24T00:49:04.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.270+0000 7f50cb59e640 1 -- 172.21.15.67:0/3024446889 learned_addr learned my addr 172.21.15.67:0/3024446889 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:04.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.270+0000 7f50ca59c640 1 --2- 172.21.15.67:0/3024446889 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55980384b020 0x559803844300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:04.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.270+0000 7f50cad9d640 1 --2- 172.21.15.67:0/3024446889 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55980384ac40 0x5598038fe990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:04.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.270+0000 7f50cb59e640 1 -- 172.21.15.67:0/3024446889 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55980384ac40 msgr2=0x5598038fe990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:04.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.270+0000 7f50cb59e640 1 --2- 172.21.15.67:0/3024446889 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55980384ac40 0x5598038fe990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:04.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.270+0000 7f50cb59e640 1 -- 172.21.15.67:0/3024446889 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55980384b020 msgr2=0x559803844300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:04.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.270+0000 7f50cb59e640 1 --2- 172.21.15.67:0/3024446889 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55980384b020 0x559803844300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:04.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.270+0000 7f50cb59e640 1 -- 172.21.15.67:0/3024446889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55980358d6b0 con 0x55980389fa80 2024-03-24T00:49:04.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.270+0000 7f50ca59c640 1 --2- 172.21.15.67:0/3024446889 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55980384b020 0x559803844300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:04.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.270+0000 7f50cb59e640 1 --2- 172.21.15.67:0/3024446889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55980389fa80 0x55980386d750 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f50c402bee0 tx=0x7f50c4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=799ab34ad078f9a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:04.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.270+0000 7f50a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:49:04.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.270+0000 7f50a9ffb640 1 -- 172.21.15.67:0/3024446889 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c4051070 con 0x55980389fa80 2024-03-24T00:49:04.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.270+0000 7f50a9ffb640 1 -- 172.21.15.67:0/3024446889 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(120..120 src has 1..120) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f50c4041880 con 0x55980389fa80 2024-03-24T00:49:04.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.270+0000 7f50d3dc4740 1 -- 172.21.15.67:0/3024446889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f50c403a040 con 0x55980389fa80 2024-03-24T00:49:04.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.271+0000 7f50a9ffb640 1 -- 172.21.15.67:0/3024446889 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f50c4039040 con 0x55980389fa80 2024-03-24T00:49:04.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.271+0000 7f50d3dc4740 10 client.9405 fetch_fsmap learned FSMap version 86 2024-03-24T00:49:04.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.271+0000 7f50d3dc4740 1 -- 172.21.15.67:0/3024446889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=86,monmap=2+}) v3 -- 0x559803878db0 con 0x55980389fa80 2024-03-24T00:49:04.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.271+0000 7f50a9ffb640 1 -- 172.21.15.67:0/3024446889 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50c403a040 con 0x55980389fa80 2024-03-24T00:49:04.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.271+0000 7f50a9ffb640 1 -- 172.21.15.67:0/3024446889 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 86) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f50c405f080 con 0x55980389fa80 2024-03-24T00:49:04.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.271+0000 7f50d3dc4740 10 client.9405 fetch_fsmap finished waiting for FSMap version 86 2024-03-24T00:49:04.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.271+0000 7f50d3dc4740 10 client.9405 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-24T00:49:04.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.271+0000 7f50d3dc4740 20 client.9405 populate_metadata read hostname 'smithi067' 2024-03-24T00:49:04.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.271+0000 7f50d3dc4740 1 --2- 172.21.15.67:0/3024446889 >> [v2:172.21.15.67:6833/312063805,v1:172.21.15.67:6835/312063805] conn(0x5598038708f0 0x5598038852f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:04.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.271+0000 7f50d3dc4740 4 client.9405 mds_command: new command op to 9219 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:49:04.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.271+0000 7f50d3dc4740 1 -- 172.21.15.67:0/3024446889 --> [v2:172.21.15.67:6833/312063805,v1:172.21.15.67:6835/312063805] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5598037b64b0 con 0x5598038708f0 2024-03-24T00:49:04.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.271+0000 7f50cad9d640 1 --2- 172.21.15.67:0/3024446889 >> [v2:172.21.15.67:6833/312063805,v1:172.21.15.67:6835/312063805] conn(0x5598038708f0 0x5598038852f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:04.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.271+0000 7f50a9ffb640 10 client.9405 ms_handle_connect on v2:172.21.15.67:6833/312063805 2024-03-24T00:49:04.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.271+0000 7f50cad9d640 1 --2- 172.21.15.67:0/3024446889 >> [v2:172.21.15.67:6833/312063805,v1:172.21.15.67:6835/312063805] conn(0x5598038708f0 0x5598038852f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e74c1219c9440742 server_cookie=ed6150a3ea690109 in_seq=0 out_seq=0 2024-03-24T00:49:04.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.271+0000 7f50a9ffb640 1 -- 172.21.15.67:0/3024446889 <== mds.0 v2:172.21.15.67:6833/312063805 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+320 (crc 0 0 0) 0x5598037b64b0 con 0x5598038708f0 2024-03-24T00:49:04.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.271+0000 7f50a9ffb640 10 client.9405 handle_command_reply: tid=0 2024-03-24T00:49:04.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.271+0000 7f50d3dc4740 1 client.9405 shutdown 2024-03-24T00:49:04.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.272+0000 7f50d3dc4740 1 -- 172.21.15.67:0/3024446889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55980389fa80 msgr2=0x55980386d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:04.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.272+0000 7f50d3dc4740 1 --2- 172.21.15.67:0/3024446889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55980389fa80 0x55980386d750 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f50c402bee0 tx=0x7f50c4004ab0 comp rx=0 tx=0).stop 2024-03-24T00:49:04.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.272+0000 7f50d3dc4740 1 -- 172.21.15.67:0/3024446889 shutdown_connections 2024-03-24T00:49:04.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.272+0000 7f50d3dc4740 1 --2- 172.21.15.67:0/3024446889 >> [v2:172.21.15.67:6833/312063805,v1:172.21.15.67:6835/312063805] conn(0x5598038708f0 0x5598038852f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:04.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.272+0000 7f50d3dc4740 1 --2- 172.21.15.67:0/3024446889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55980389fa80 0x55980386d750 unknown :-1 s=CLOSED pgs=1737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:04.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.272+0000 7f50d3dc4740 1 --2- 172.21.15.67:0/3024446889 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55980384b020 0x559803844300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:04.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.272+0000 7f50d3dc4740 1 --2- 172.21.15.67:0/3024446889 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55980384ac40 0x5598038fe990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:04.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.272+0000 7f50a9ffb640 0 client.9405 ms_handle_reset on v2:172.21.15.67:6833/312063805 2024-03-24T00:49:04.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.272+0000 7f50d3dc4740 1 -- 172.21.15.67:0/3024446889 >> 172.21.15.67:0/3024446889 conn(0x559803811a60 msgr2=0x55980384bad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:04.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.272+0000 7f50d3dc4740 1 -- 172.21.15.67:0/3024446889 shutdown_connections 2024-03-24T00:49:04.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.272+0000 7f50d3dc4740 1 -- 172.21.15.67:0/3024446889 wait complete. 2024-03-24T00:49:04.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.272+0000 7f50d3dc4740 20 client.9405 trim_cache size 0 max 16384 2024-03-24T00:49:04.280 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:49:04.280 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "de1d0178-808e-49ca-a986-e297e523b976", 2024-03-24T00:49:04.280 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-03-24T00:49:04.280 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 9219, 2024-03-24T00:49:04.280 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-03-24T00:49:04.280 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-03-24T00:49:04.281 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-03-24T00:49:04.281 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 9.295545122, 2024-03-24T00:49:04.281 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 86, 2024-03-24T00:49:04.281 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 120, 2024-03-24T00:49:04.281 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 120, 2024-03-24T00:49:04.281 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 14.629804474 2024-03-24T00:49:04.281 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:49:04.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.272+0000 7f50d10f8640 1 -- 172.21.15.67:0/3037965365 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f50b806bbd0 msgr2=0x7f50b806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:04.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.272+0000 7f50d10f8640 1 --2- 172.21.15.67:0/3037965365 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f50b806bbd0 0x7f50b806e090 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f50c0009aa0 tx=0x7f50c0009190 comp rx=0 tx=0).stop 2024-03-24T00:49:04.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.272+0000 7f50d10f8640 1 -- 172.21.15.67:0/3037965365 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50cc0b9af0 msgr2=0x7f50cc15e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:04.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.272+0000 7f50d10f8640 1 --2- 172.21.15.67:0/3037965365 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50cc0b9af0 0x7f50cc15e4d0 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f50bc00c730 tx=0x7f50bc00cbf0 comp rx=0 tx=0).stop 2024-03-24T00:49:04.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.272+0000 7f50d10f8640 1 -- 172.21.15.67:0/3037965365 shutdown_connections 2024-03-24T00:49:04.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.272+0000 7f50d10f8640 1 --2- 172.21.15.67:0/3037965365 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f50b806bbd0 0x7f50b806e090 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:04.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.272+0000 7f50d10f8640 1 --2- 172.21.15.67:0/3037965365 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f50cc15e0f0 0x7f50cc0c9290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:04.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.272+0000 7f50d10f8640 1 --2- 172.21.15.67:0/3037965365 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50cc0b9af0 0x7f50cc15e4d0 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:04.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.272+0000 7f50d10f8640 1 --2- 172.21.15.67:0/3037965365 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50cc0b6dc0 0x7f50cc14e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:04.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.272+0000 7f50d10f8640 1 -- 172.21.15.67:0/3037965365 >> 172.21.15.67:0/3037965365 conn(0x7f50cc072b60 msgr2=0x7f50cc0b9370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:04.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.272+0000 7f50d10f8640 1 -- 172.21.15.67:0/3037965365 shutdown_connections 2024-03-24T00:49:04.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.272+0000 7f50d10f8640 1 -- 172.21.15.67:0/3037965365 wait complete. 2024-03-24T00:49:04.284 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T00:49:04.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.386+0000 7fa2b1a01640 1 Processor -- start 2024-03-24T00:49:04.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.386+0000 7fa2b1a01640 1 -- start start 2024-03-24T00:49:04.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.387+0000 7fa2b1a01640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa2ac0d0700 0x7fa2ac0b1360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:04.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.387+0000 7fa2b1a01640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2ac15c1f0 con 0x7fa2ac0cf1f0 2024-03-24T00:49:04.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.387+0000 7fa2b1a01640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2ac0b1a20 con 0x7fa2ac15bc20 2024-03-24T00:49:04.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.387+0000 7fa2b1a01640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2ac0cf670 con 0x7fa2ac0d0700 2024-03-24T00:49:04.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.387+0000 7fa2aaffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa2ac0d0700 0x7fa2ac0b1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:04.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.387+0000 7fa2aa7fc640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa2ac0cf1f0 0x7fa2ac0d0ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48690/0 (socket says 172.21.15.67:48690) 2024-03-24T00:49:04.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.387+0000 7fa2aa7fc640 1 -- 172.21.15.67:0/747679711 learned_addr learned my addr 172.21.15.67:0/747679711 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:49:04.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.387+0000 7fa2aaffd640 1 -- 172.21.15.67:0/747679711 >> v1:172.21.15.72:6789/0 conn(0x7fa2ac15bc20 legacy=0x7fa2ac15cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:04.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.387+0000 7fa2aaffd640 1 -- 172.21.15.67:0/747679711 >> v1:172.21.15.67:6789/0 conn(0x7fa2ac0cf1f0 legacy=0x7fa2ac0d0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:04.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.387+0000 7fa2aaffd640 1 -- 172.21.15.67:0/747679711 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2ac15c460 con 0x7fa2ac0d0700 2024-03-24T00:49:04.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.388+0000 7fa2aaffd640 1 --2- 172.21.15.67:0/747679711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2ac0d0700 0x7fa2ac0b1360 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7fa2a40097a0 tx=0x7fa2a402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5928ecb136f4a1dd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:04.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.388+0000 7fa2a9ffb640 1 -- 172.21.15.67:0/747679711 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a4035020 con 0x7fa2ac0d0700 2024-03-24T00:49:04.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.388+0000 7fa2a9ffb640 1 -- 172.21.15.67:0/747679711 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2a4031830 con 0x7fa2ac0d0700 2024-03-24T00:49:04.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.388+0000 7fa2a9ffb640 1 -- 172.21.15.67:0/747679711 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a4031c00 con 0x7fa2ac0d0700 2024-03-24T00:49:04.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.389+0000 7fa2b1a01640 1 -- 172.21.15.67:0/747679711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2ac0d0700 msgr2=0x7fa2ac0b1360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:04.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.389+0000 7fa2b1a01640 1 --2- 172.21.15.67:0/747679711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2ac0d0700 0x7fa2ac0b1360 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7fa2a40097a0 tx=0x7fa2a402b8d0 comp rx=0 tx=0).stop 2024-03-24T00:49:04.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.389+0000 7fa2b1a01640 1 -- 172.21.15.67:0/747679711 shutdown_connections 2024-03-24T00:49:04.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.389+0000 7fa2b1a01640 1 --2- 172.21.15.67:0/747679711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2ac0d0700 0x7fa2ac0b1360 unknown :-1 s=CLOSED pgs=1739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:04.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.389+0000 7fa2b1a01640 1 -- 172.21.15.67:0/747679711 >> 172.21.15.67:0/747679711 conn(0x7fa2ac099a30 msgr2=0x7fa2ac099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:04.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.389+0000 7fa2b1a01640 1 -- 172.21.15.67:0/747679711 shutdown_connections 2024-03-24T00:49:04.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.389+0000 7fa2b1a01640 1 -- 172.21.15.67:0/747679711 wait complete. 2024-03-24T00:49:04.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.390+0000 7fa2b1a01640 1 Processor -- start 2024-03-24T00:49:04.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.390+0000 7fa2b1a01640 1 -- start start 2024-03-24T00:49:04.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.391+0000 7fa2b1a01640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa2ac0cf1f0 0x7fa2ac155070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:04.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.391+0000 7fa2b1a01640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2ac0d0700 0x7fa2ac0c8e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:04.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.391+0000 7fa2b1a01640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2ac15bc20 0x7fa2ac0c6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:04.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.391+0000 7fa2b1a01640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2ac15d370 con 0x7fa2ac15bc20 2024-03-24T00:49:04.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.391+0000 7fa2b1a01640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2ac0b1d00 con 0x7fa2ac0d0700 2024-03-24T00:49:04.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.391+0000 7fa2b1a01640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2ac15d4f0 con 0x7fa2ac0cf1f0 2024-03-24T00:49:04.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.391+0000 7fa2ab7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2ac15bc20 0x7fa2ac0c6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:04.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.391+0000 7fa2ab7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2ac15bc20 0x7fa2ac0c6f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42354/0 (socket says 172.21.15.67:42354) 2024-03-24T00:49:04.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.391+0000 7fa2ab7fe640 1 -- 172.21.15.67:0/2375063530 learned_addr learned my addr 172.21.15.67:0/2375063530 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:04.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.391+0000 7fa2aa7fc640 1 --2- 172.21.15.67:0/2375063530 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2ac0d0700 0x7fa2ac0c8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:04.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.391+0000 7fa2aaffd640 1 --2- 172.21.15.67:0/2375063530 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa2ac0cf1f0 0x7fa2ac155070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:04.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.392+0000 7fa2ab7fe640 1 -- 172.21.15.67:0/2375063530 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa2ac0cf1f0 msgr2=0x7fa2ac155070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:04.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.392+0000 7fa2ab7fe640 1 --2- 172.21.15.67:0/2375063530 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa2ac0cf1f0 0x7fa2ac155070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:04.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.392+0000 7fa2ab7fe640 1 -- 172.21.15.67:0/2375063530 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2ac0d0700 msgr2=0x7fa2ac0c8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:04.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.392+0000 7fa2ab7fe640 1 --2- 172.21.15.67:0/2375063530 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2ac0d0700 0x7fa2ac0c8e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:04.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.392+0000 7fa2ab7fe640 1 -- 172.21.15.67:0/2375063530 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2ac1555b0 con 0x7fa2ac15bc20 2024-03-24T00:49:04.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.392+0000 7fa2ab7fe640 1 --2- 172.21.15.67:0/2375063530 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2ac15bc20 0x7fa2ac0c6f40 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7fa29c009f30 tx=0x7fa29c009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:04.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.392+0000 7fa293fff640 1 -- 172.21.15.67:0/2375063530 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa29c013710 con 0x7fa2ac15bc20 2024-03-24T00:49:04.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.392+0000 7fa293fff640 1 -- 172.21.15.67:0/2375063530 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa29c0138f0 con 0x7fa2ac15bc20 2024-03-24T00:49:04.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.392+0000 7fa2b1a01640 1 -- 172.21.15.67:0/2375063530 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2ac0c7d70 con 0x7fa2ac15bc20 2024-03-24T00:49:04.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.392+0000 7fa293fff640 1 -- 172.21.15.67:0/2375063530 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa29c013ce0 con 0x7fa2ac15bc20 2024-03-24T00:49:04.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.392+0000 7fa2b1a01640 1 -- 172.21.15.67:0/2375063530 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2ac0c8330 con 0x7fa2ac15bc20 2024-03-24T00:49:04.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.393+0000 7fa2b1a01640 1 -- 172.21.15.67:0/2375063530 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2ac04f400 con 0x7fa2ac15bc20 2024-03-24T00:49:04.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.394+0000 7fa293fff640 1 -- 172.21.15.67:0/2375063530 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa29c01e400 con 0x7fa2ac15bc20 2024-03-24T00:49:04.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.394+0000 7fa293fff640 1 --2- 172.21.15.67:0/2375063530 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa29806bc00 0x7fa29806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:04.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.394+0000 7fa293fff640 1 -- 172.21.15.67:0/2375063530 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa29c08d820 con 0x7fa2ac15bc20 2024-03-24T00:49:04.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.394+0000 7fa2aaffd640 1 --2- 172.21.15.67:0/2375063530 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa29806bc00 0x7fa29806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:04.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.395+0000 7fa2aaffd640 1 --2- 172.21.15.67:0/2375063530 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa29806bc00 0x7fa29806e0c0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fa2a4036000 tx=0x7fa2a4008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:04.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.399+0000 7fa293fff640 1 -- 172.21.15.67:0/2375063530 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa29c05bbc0 con 0x7fa2ac15bc20 2024-03-24T00:49:04.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.622+0000 7fa2b1a01640 1 -- 172.21.15.67:0/2375063530 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fa2ac0b0080 con 0x7fa2ac15bc20 2024-03-24T00:49:04.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.622+0000 7fa293fff640 1 -- 172.21.15.67:0/2375063530 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v86) v1 ==== 53+0+73 (secure 0 0 0) 0x7fa29c05fc60 con 0x7fa2ac15bc20 2024-03-24T00:49:04.623 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-24T00:49:04.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.624+0000 7fa2b1a01640 1 -- 172.21.15.67:0/2375063530 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa29806bc00 msgr2=0x7fa29806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:04.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.624+0000 7fa2b1a01640 1 --2- 172.21.15.67:0/2375063530 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa29806bc00 0x7fa29806e0c0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fa2a4036000 tx=0x7fa2a4008030 comp rx=0 tx=0).stop 2024-03-24T00:49:04.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.624+0000 7fa2b1a01640 1 -- 172.21.15.67:0/2375063530 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2ac15bc20 msgr2=0x7fa2ac0c6f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:04.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.624+0000 7fa2b1a01640 1 --2- 172.21.15.67:0/2375063530 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2ac15bc20 0x7fa2ac0c6f40 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7fa29c009f30 tx=0x7fa29c009f60 comp rx=0 tx=0).stop 2024-03-24T00:49:04.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.624+0000 7fa2b1a01640 1 -- 172.21.15.67:0/2375063530 shutdown_connections 2024-03-24T00:49:04.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.624+0000 7fa2b1a01640 1 --2- 172.21.15.67:0/2375063530 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa29806bc00 0x7fa29806e0c0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:04.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.624+0000 7fa2b1a01640 1 --2- 172.21.15.67:0/2375063530 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2ac15bc20 0x7fa2ac0c6f40 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:04.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.624+0000 7fa2b1a01640 1 --2- 172.21.15.67:0/2375063530 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2ac0d0700 0x7fa2ac0c8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:04.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.624+0000 7fa2b1a01640 1 --2- 172.21.15.67:0/2375063530 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa2ac0cf1f0 0x7fa2ac155070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:04.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.624+0000 7fa2b1a01640 1 -- 172.21.15.67:0/2375063530 >> 172.21.15.67:0/2375063530 conn(0x7fa2ac099a30 msgr2=0x7fa2ac15dac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:04.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.624+0000 7fa2b1a01640 1 -- 172.21.15.67:0/2375063530 shutdown_connections 2024-03-24T00:49:04.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:04.624+0000 7fa2b1a01640 1 -- 172.21.15.67:0/2375063530 wait complete. 2024-03-24T00:49:04.636 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-24T00:49:04.636 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T00:49:04.636 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-03-24T00:49:04.636 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-24T00:49:04.636 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T00:49:04.636 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-24T00:49:04.636 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-24T00:49:04.637 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T00:49:04.637 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:49:04.637 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:04.637 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-03-24T00:49:04.652 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T00:49:04.652 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T00:49:04.652 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T00:49:04.653 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:49:04.653 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-03-24T00:49:04.653 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:49:04.653 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:49:04.653 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-03-24T00:49:04.653 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-03-24T00:49:04.653 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:49:04.653 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-03-24T00:49:04.653 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-03-24T00:49:04.653 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:49:04.653 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-03-24T00:49:04.653 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T00:49:04.654 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-03-24T00:49:04.654 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-24T00:49:04.654 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 41968sec preferred_lft 41968sec 2024-03-24T00:49:04.654 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-03-24T00:49:04.654 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:49:04.654 INFO:teuthology.orchestra.run.smithi067.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T00:49:04.654 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 96:cf:26:c8:bf:62 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:49:04.654 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-24T00:49:04.654 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:49:04.654 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::7c6a:5fff:feaa:d41/64 scope link 2024-03-24T00:49:04.654 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:49:04.655 INFO:teuthology.orchestra.run.smithi067.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T00:49:04.655 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ea:22:4b:d5:48:3d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:49:04.655 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::e822:4bff:fed5:483d/64 scope link 2024-03-24T00:49:04.655 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:49:04.655 INFO:teuthology.orchestra.run.smithi067.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T00:49:04.655 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 96:cf:26:c8:bf:62 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:49:04.655 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::94cf:26ff:fec8:bf62/64 scope link 2024-03-24T00:49:04.655 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:49:04.655 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:04.655 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T00:49:04.718 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-03-24T00:49:04.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:04.720 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-03-24T00:49:04.776 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-03-24T00:49:04.776 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T00:49:04.777 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-24T00:49:04.777 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-24T00:49:04.777 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T00:49:04.777 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:04.778 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-03-24T00:49:04.778 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-03-24T00:49:04.778 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-24T00:49:04.778 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-24T00:49:04.779 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-24T00:49:04.779 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-03-24T00:49:04.872 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:04.872 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-03-24T00:49:04.887 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-24T00:49:04.887 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-24T00:49:04.888 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:49:04.888 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:04.888 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-03-24T00:49:04.955 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:49:04.955 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-24T00:49:05.010 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:49:05.010 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:49:05.010 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:49:05.064 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-24T00:49:05.065 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:05.065 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:49:05.152 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:49:05.152 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T00:49:05.153 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-03-24T00:49:05.160 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:49:05.160 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T00:49:05.160 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-03-24T00:49:05.161 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.72:3300,172.21.15.72:3301" 2024-03-24T00:49:05.161 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-24T00:49:05.176 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:49:05.176 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T00:49:05.176 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-03-24T00:49:05.176 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.72:3300,172.21.15.72:3301" 2024-03-24T00:49:05.176 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-24T00:49:05.176 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:6789,172.21.15.72:6789,172.21.15.72:6790" 2024-03-24T00:49:05.176 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.67:6789,172.21.15.72:6789,172.21.15.72:6790:/ 2024-03-24T00:49:05.177 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=de1d0178-808e-49ca-a986-e297e523b976" will pass to kernel 2024-03-24T00:49:05.177 INFO:teuthology.orchestra.run.smithi067.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-24T00:49:05.177 INFO:teuthology.orchestra.run.smithi067.stdout: You just mounted a file system that supports labels which does not 2024-03-24T00:49:05.177 INFO:teuthology.orchestra.run.smithi067.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-24T00:49:05.177 INFO:teuthology.orchestra.run.smithi067.stdout: applications will generate AVC messages and not be allowed access to 2024-03-24T00:49:05.177 INFO:teuthology.orchestra.run.smithi067.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-24T00:49:05.179 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-24T00:49:05.179 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:05.179 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:49:05.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:05.206 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T00:49:05.265 INFO:teuthology.orchestra.run.smithi067.stdout:client7391 2024-03-24T00:49:05.265 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:49:05.265 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:49:05.322 DEBUG:teuthology.orchestra.run.smithi067:> stdin-killer -- python3 -c ' 2024-03-24T00:49:05.322 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-03-24T00:49:05.322 DEBUG:teuthology.orchestra.run.smithi067:> mount_point = "/home/ubuntu/cephtest/mnt.0" 2024-03-24T00:49:05.322 DEBUG:teuthology.orchestra.run.smithi067:> parent = os.path.join(mount_point, "parent") 2024-03-24T00:49:05.322 DEBUG:teuthology.orchestra.run.smithi067:> os.mkdir(parent) 2024-03-24T00:49:05.322 DEBUG:teuthology.orchestra.run.smithi067:> flushed = os.path.join(parent, "flushed") 2024-03-24T00:49:05.322 DEBUG:teuthology.orchestra.run.smithi067:> os.mkdir(flushed) 2024-03-24T00:49:05.322 DEBUG:teuthology.orchestra.run.smithi067:> for f in ["alpha", "bravo", "charlie"]: 2024-03-24T00:49:05.322 DEBUG:teuthology.orchestra.run.smithi067:> open(os.path.join(flushed, f), '"'"'w'"'"').write(f) 2024-03-24T00:49:05.323 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-03-24T00:49:05.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:49:05.449 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:05.449 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec find ./) 2024-03-24T00:49:05.506 INFO:teuthology.orchestra.run.smithi067.stdout:./ 2024-03-24T00:49:05.506 INFO:teuthology.orchestra.run.smithi067.stdout:./parent 2024-03-24T00:49:05.507 INFO:teuthology.orchestra.run.smithi067.stdout:./parent/flushed 2024-03-24T00:49:05.507 INFO:teuthology.orchestra.run.smithi067.stdout:./parent/flushed/charlie 2024-03-24T00:49:05.507 INFO:teuthology.orchestra.run.smithi067.stdout:./parent/flushed/alpha 2024-03-24T00:49:05.507 INFO:teuthology.orchestra.run.smithi067.stdout:./parent/flushed/bravo 2024-03-24T00:49:05.507 DEBUG:teuthology.orchestra.run.smithi067:> stdin-killer -- python3 -c ' 2024-03-24T00:49:05.507 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-03-24T00:49:05.507 DEBUG:teuthology.orchestra.run.smithi067:> import stat 2024-03-24T00:49:05.507 DEBUG:teuthology.orchestra.run.smithi067:> 2024-03-24T00:49:05.507 DEBUG:teuthology.orchestra.run.smithi067:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./").st_ino) 2024-03-24T00:49:05.507 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-03-24T00:49:05.613 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-03-24T00:49:05.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:49:05.621 DEBUG:teuthology.orchestra.run.smithi067:> stdin-killer -- python3 -c ' 2024-03-24T00:49:05.622 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-03-24T00:49:05.622 DEBUG:teuthology.orchestra.run.smithi067:> import stat 2024-03-24T00:49:05.622 DEBUG:teuthology.orchestra.run.smithi067:> 2024-03-24T00:49:05.622 DEBUG:teuthology.orchestra.run.smithi067:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent").st_ino) 2024-03-24T00:49:05.622 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-03-24T00:49:05.728 INFO:teuthology.orchestra.run.smithi067.stdout:1099511627776 2024-03-24T00:49:05.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:49:05.738 DEBUG:teuthology.orchestra.run.smithi067:> stdin-killer -- python3 -c ' 2024-03-24T00:49:05.738 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-03-24T00:49:05.738 DEBUG:teuthology.orchestra.run.smithi067:> import stat 2024-03-24T00:49:05.739 DEBUG:teuthology.orchestra.run.smithi067:> 2024-03-24T00:49:05.739 DEBUG:teuthology.orchestra.run.smithi067:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed").st_ino) 2024-03-24T00:49:05.739 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-03-24T00:49:05.843 INFO:teuthology.orchestra.run.smithi067.stdout:1099511627777 2024-03-24T00:49:05.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:49:05.851 DEBUG:teuthology.orchestra.run.smithi067:> stdin-killer -- python3 -c ' 2024-03-24T00:49:05.852 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-03-24T00:49:05.852 DEBUG:teuthology.orchestra.run.smithi067:> import stat 2024-03-24T00:49:05.852 DEBUG:teuthology.orchestra.run.smithi067:> 2024-03-24T00:49:05.852 DEBUG:teuthology.orchestra.run.smithi067:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed/charlie").st_ino) 2024-03-24T00:49:05.852 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-03-24T00:49:05.957 INFO:teuthology.orchestra.run.smithi067.stdout:1099511628280 2024-03-24T00:49:05.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:49:05.965 DEBUG:teuthology.orchestra.run.smithi067:> stdin-killer -- python3 -c ' 2024-03-24T00:49:05.965 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-03-24T00:49:05.966 DEBUG:teuthology.orchestra.run.smithi067:> import stat 2024-03-24T00:49:05.966 DEBUG:teuthology.orchestra.run.smithi067:> 2024-03-24T00:49:05.966 DEBUG:teuthology.orchestra.run.smithi067:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed/alpha").st_ino) 2024-03-24T00:49:05.966 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-03-24T00:49:06.071 INFO:teuthology.orchestra.run.smithi067.stdout:1099511627778 2024-03-24T00:49:06.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:49:06.080 DEBUG:teuthology.orchestra.run.smithi067:> stdin-killer -- python3 -c ' 2024-03-24T00:49:06.080 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-03-24T00:49:06.080 DEBUG:teuthology.orchestra.run.smithi067:> import stat 2024-03-24T00:49:06.081 DEBUG:teuthology.orchestra.run.smithi067:> 2024-03-24T00:49:06.081 DEBUG:teuthology.orchestra.run.smithi067:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed/bravo").st_ino) 2024-03-24T00:49:06.081 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-03-24T00:49:06.186 INFO:teuthology.orchestra.run.smithi067.stdout:1099511628279 2024-03-24T00:49:06.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:49:06.194 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:49:06.194 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:49:06.249 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:49:06.250 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:49:06.304 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:49:06.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.473+0000 7fdefcc75640 1 Processor -- start 2024-03-24T00:49:06.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.473+0000 7fdefcc75640 1 -- start start 2024-03-24T00:49:06.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.474+0000 7fdefcc75640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdef815dec0 0x7fdef80ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:06.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.474+0000 7fdefcc75640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdef80d0ef0 0x7fdef8162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:06.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.474+0000 7fdefcc75640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdef80cf640 con 0x7fdef815e2a0 2024-03-24T00:49:06.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.474+0000 7fdefcc75640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdef80cee30 con 0x7fdef815dec0 2024-03-24T00:49:06.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.474+0000 7fdefcc75640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdef80d2120 con 0x7fdef80d0ef0 2024-03-24T00:49:06.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.474+0000 7fdef659c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdef815dec0 0x7fdef80ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:06.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.474+0000 7fdef659c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdef815dec0 0x7fdef80ce630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42368/0 (socket says 172.21.15.67:42368) 2024-03-24T00:49:06.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.474+0000 7fdef659c640 1 -- 172.21.15.67:0/2161084758 learned_addr learned my addr 172.21.15.67:0/2161084758 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:06.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.474+0000 7fdef6d9d640 1 --2- 172.21.15.67:0/2161084758 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdef80d0ef0 0x7fdef8162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:06.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.474+0000 7fdef659c640 1 -- 172.21.15.67:0/2161084758 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdef80d0ef0 msgr2=0x7fdef8162630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:06.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.474+0000 7fdef659c640 1 --2- 172.21.15.67:0/2161084758 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdef80d0ef0 0x7fdef8162630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:06.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.474+0000 7fdef659c640 1 -- 172.21.15.67:0/2161084758 >> v1:172.21.15.72:6789/0 conn(0x7fdef815e2a0 legacy=0x7fdef80d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:06.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.474+0000 7fdef659c640 1 -- 172.21.15.67:0/2161084758 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdef80cf080 con 0x7fdef815dec0 2024-03-24T00:49:06.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.475+0000 7fdef659c640 1 --2- 172.21.15.67:0/2161084758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdef815dec0 0x7fdef80ce630 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7fdee8009870 tx=0x7fdee802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adbe8f1abc84af server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:06.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.475+0000 7fdef5d9b640 1 -- 172.21.15.67:0/2161084758 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee8035020 con 0x7fdef815dec0 2024-03-24T00:49:06.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.475+0000 7fdef5d9b640 1 -- 172.21.15.67:0/2161084758 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdee8031640 con 0x7fdef815dec0 2024-03-24T00:49:06.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.475+0000 7fdef5d9b640 1 -- 172.21.15.67:0/2161084758 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee802d040 con 0x7fdef815dec0 2024-03-24T00:49:06.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.476+0000 7fdefcc75640 1 -- 172.21.15.67:0/2161084758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdef815dec0 msgr2=0x7fdef80ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:06.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.476+0000 7fdefcc75640 1 --2- 172.21.15.67:0/2161084758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdef815dec0 0x7fdef80ce630 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7fdee8009870 tx=0x7fdee802b8f0 comp rx=0 tx=0).stop 2024-03-24T00:49:06.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.476+0000 7fdefcc75640 1 -- 172.21.15.67:0/2161084758 shutdown_connections 2024-03-24T00:49:06.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.476+0000 7fdefcc75640 1 --2- 172.21.15.67:0/2161084758 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdef80d0ef0 0x7fdef8162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:06.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.476+0000 7fdefcc75640 1 --2- 172.21.15.67:0/2161084758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdef815dec0 0x7fdef80ce630 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:06.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.476+0000 7fdefcc75640 1 -- 172.21.15.67:0/2161084758 >> 172.21.15.67:0/2161084758 conn(0x7fdef806c810 msgr2=0x7fdef806cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:06.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.476+0000 7fdefcc75640 1 -- 172.21.15.67:0/2161084758 shutdown_connections 2024-03-24T00:49:06.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.476+0000 7fdefcc75640 1 -- 172.21.15.67:0/2161084758 wait complete. 2024-03-24T00:49:06.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.478+0000 7fdefcc75640 1 Processor -- start 2024-03-24T00:49:06.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.478+0000 7fdefcc75640 1 -- start start 2024-03-24T00:49:06.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.479+0000 7fdefcc75640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdef80d0ef0 0x7fdef80b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:06.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.479+0000 7fdefcc75640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdef815dec0 0x7fdef815aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:06.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.479+0000 7fdefcc75640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdef815e2a0 0x7fdef80b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:06.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.479+0000 7fdefcc75640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef80d2730 con 0x7fdef815e2a0 2024-03-24T00:49:06.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.479+0000 7fdefcc75640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef80d25b0 con 0x7fdef80d0ef0 2024-03-24T00:49:06.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.479+0000 7fdefcc75640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdef80d28b0 con 0x7fdef815dec0 2024-03-24T00:49:06.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.479+0000 7fdef6d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdef815e2a0 0x7fdef80b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:06.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.479+0000 7fdef6d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdef815e2a0 0x7fdef80b41c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42370/0 (socket says 172.21.15.67:42370) 2024-03-24T00:49:06.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.479+0000 7fdef6d9d640 1 -- 172.21.15.67:0/2895140325 learned_addr learned my addr 172.21.15.67:0/2895140325 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:06.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.479+0000 7fdef659c640 1 --2- 172.21.15.67:0/2895140325 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdef80d0ef0 0x7fdef80b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:06.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.479+0000 7fdeeffff640 1 --2- 172.21.15.67:0/2895140325 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdef815dec0 0x7fdef815aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:06.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.479+0000 7fdef6d9d640 1 -- 172.21.15.67:0/2895140325 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdef815dec0 msgr2=0x7fdef815aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:06.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.479+0000 7fdef6d9d640 1 --2- 172.21.15.67:0/2895140325 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdef815dec0 0x7fdef815aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:06.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.479+0000 7fdef6d9d640 1 -- 172.21.15.67:0/2895140325 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdef80d0ef0 msgr2=0x7fdef80b6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:06.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.479+0000 7fdef6d9d640 1 --2- 172.21.15.67:0/2895140325 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdef80d0ef0 0x7fdef80b6160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:06.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.479+0000 7fdef6d9d640 1 -- 172.21.15.67:0/2895140325 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdef815b1f0 con 0x7fdef815e2a0 2024-03-24T00:49:06.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.479+0000 7fdef659c640 1 --2- 172.21.15.67:0/2895140325 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdef80d0ef0 0x7fdef80b6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:06.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.479+0000 7fdeeffff640 1 --2- 172.21.15.67:0/2895140325 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdef815dec0 0x7fdef815aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:06.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.479+0000 7fdef6d9d640 1 --2- 172.21.15.67:0/2895140325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdef815e2a0 0x7fdef80b41c0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7fdef0007e30 tx=0x7fdef000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:06.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.479+0000 7fdeeeffd640 1 -- 172.21.15.67:0/2895140325 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef0014620 con 0x7fdef815e2a0 2024-03-24T00:49:06.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.480+0000 7fdeeeffd640 1 -- 172.21.15.67:0/2895140325 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdef0019070 con 0x7fdef815e2a0 2024-03-24T00:49:06.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.480+0000 7fdeeeffd640 1 -- 172.21.15.67:0/2895140325 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef0014a00 con 0x7fdef815e2a0 2024-03-24T00:49:06.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.480+0000 7fdefcc75640 1 -- 172.21.15.67:0/2895140325 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdef815b4d0 con 0x7fdef815e2a0 2024-03-24T00:49:06.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.480+0000 7fdefcc75640 1 -- 172.21.15.67:0/2895140325 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdef815ba90 con 0x7fdef815e2a0 2024-03-24T00:49:06.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.480+0000 7fdefcc75640 1 -- 172.21.15.67:0/2895140325 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdef804f400 con 0x7fdef815e2a0 2024-03-24T00:49:06.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.481+0000 7fdeeeffd640 1 -- 172.21.15.67:0/2895140325 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fdef0014ba0 con 0x7fdef815e2a0 2024-03-24T00:49:06.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.481+0000 7fdeeeffd640 1 --2- 172.21.15.67:0/2895140325 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fded806bbd0 0x7fded806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:06.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.482+0000 7fdeeeffd640 1 -- 172.21.15.67:0/2895140325 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fdef008b510 con 0x7fdef815e2a0 2024-03-24T00:49:06.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.482+0000 7fdef659c640 1 --2- 172.21.15.67:0/2895140325 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fded806bbd0 0x7fded806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:06.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.482+0000 7fdef659c640 1 --2- 172.21.15.67:0/2895140325 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fded806bbd0 0x7fded806e090 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fdee8036000 tx=0x7fdee8008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:06.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.486+0000 7fdeeeffd640 1 -- 172.21.15.67:0/2895140325 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fdef0059e70 con 0x7fdef815e2a0 2024-03-24T00:49:06.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.669+0000 7fdefcc75640 1 -- 172.21.15.67:0/2895140325 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdef80d1aa0 con 0x7fdef815e2a0 2024-03-24T00:49:06.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.670+0000 7fdeeeffd640 1 -- 172.21.15.67:0/2895140325 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v120) v1 ==== 74+0+16892 (secure 0 0 0) 0x7fdef0059770 con 0x7fdef815e2a0 2024-03-24T00:49:06.671 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:06.671 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":120,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:48:55.987829+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:48:52.622356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-03-24T00:48:53.311452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:49:06.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.672+0000 7fdefcc75640 1 -- 172.21.15.67:0/2895140325 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fded806bbd0 msgr2=0x7fded806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:06.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.672+0000 7fdefcc75640 1 --2- 172.21.15.67:0/2895140325 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fded806bbd0 0x7fded806e090 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fdee8036000 tx=0x7fdee8008030 comp rx=0 tx=0).stop 2024-03-24T00:49:06.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.672+0000 7fdefcc75640 1 -- 172.21.15.67:0/2895140325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdef815e2a0 msgr2=0x7fdef80b41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:06.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.672+0000 7fdefcc75640 1 --2- 172.21.15.67:0/2895140325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdef815e2a0 0x7fdef80b41c0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7fdef0007e30 tx=0x7fdef000e4b0 comp rx=0 tx=0).stop 2024-03-24T00:49:06.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.672+0000 7fdefcc75640 1 -- 172.21.15.67:0/2895140325 shutdown_connections 2024-03-24T00:49:06.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.672+0000 7fdefcc75640 1 --2- 172.21.15.67:0/2895140325 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fded806bbd0 0x7fded806e090 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:06.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.672+0000 7fdefcc75640 1 --2- 172.21.15.67:0/2895140325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdef815e2a0 0x7fdef80b41c0 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:06.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.672+0000 7fdefcc75640 1 --2- 172.21.15.67:0/2895140325 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdef815dec0 0x7fdef815aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:06.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.672+0000 7fdefcc75640 1 --2- 172.21.15.67:0/2895140325 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdef80d0ef0 0x7fdef80b6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:06.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.672+0000 7fdefcc75640 1 -- 172.21.15.67:0/2895140325 >> 172.21.15.67:0/2895140325 conn(0x7fdef806c810 msgr2=0x7fdef80b2ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:06.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.672+0000 7fdefcc75640 1 -- 172.21.15.67:0/2895140325 shutdown_connections 2024-03-24T00:49:06.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.672+0000 7fdefcc75640 1 -- 172.21.15.67:0/2895140325 wait complete. 2024-03-24T00:49:06.684 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:38:39.364950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:48:52.622356+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '119', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:48:53.311452+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '120', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:49:06.684 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:49:06.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.788+0000 7f2c35b6f640 1 Processor -- start 2024-03-24T00:49:06.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.788+0000 7f2c35b6f640 1 -- start start 2024-03-24T00:49:06.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.789+0000 7f2c35b6f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2c3015dea0 0x7f2c300d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:06.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.789+0000 7f2c35b6f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c300d2250 con 0x7f2c3015e280 2024-03-24T00:49:06.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.789+0000 7f2c35b6f640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c300d2530 con 0x7f2c300b6170 2024-03-24T00:49:06.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.789+0000 7f2c35b6f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c300d2810 con 0x7f2c3015dea0 2024-03-24T00:49:06.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.789+0000 7f2c2f7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2c3015dea0 0x7f2c300d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:06.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.789+0000 7f2c2effd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2c3015e280 0x7f2c300b5a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48698/0 (socket says 172.21.15.67:48698) 2024-03-24T00:49:06.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.789+0000 7f2c2effd640 1 -- 172.21.15.67:0/437314823 learned_addr learned my addr 172.21.15.67:0/437314823 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:49:06.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.789+0000 7f2c2f7fe640 1 -- 172.21.15.67:0/437314823 >> v1:172.21.15.72:6789/0 conn(0x7f2c300b6170 legacy=0x7f2c300b70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:06.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.789+0000 7f2c2f7fe640 1 -- 172.21.15.67:0/437314823 >> v1:172.21.15.67:6789/0 conn(0x7f2c3015e280 legacy=0x7f2c300b5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:06.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.789+0000 7f2c2f7fe640 1 -- 172.21.15.67:0/437314823 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c300d2af0 con 0x7f2c3015dea0 2024-03-24T00:49:06.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.789+0000 7f2c2e7fc640 1 -- 172.21.15.67:0/437314823 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3142787201 0 0) 0x7f2c300d2250 con 0x7f2c3015e280 2024-03-24T00:49:06.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.791+0000 7f2c2f7fe640 1 --2- 172.21.15.67:0/437314823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c3015dea0 0x7f2c300d33d0 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f2c200097d0 tx=0x7f2c2002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9803b7a6580fb11a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:06.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.791+0000 7f2c2e7fc640 1 -- 172.21.15.67:0/437314823 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c200315a0 con 0x7f2c3015dea0 2024-03-24T00:49:06.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.791+0000 7f2c2e7fc640 1 -- 172.21.15.67:0/437314823 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c20037070 con 0x7f2c3015dea0 2024-03-24T00:49:06.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.791+0000 7f2c2e7fc640 1 -- 172.21.15.67:0/437314823 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c20031960 con 0x7f2c3015dea0 2024-03-24T00:49:06.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.791+0000 7f2c35b6f640 1 -- 172.21.15.67:0/437314823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c3015dea0 msgr2=0x7f2c300d33d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:06.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.791+0000 7f2c35b6f640 1 --2- 172.21.15.67:0/437314823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c3015dea0 0x7f2c300d33d0 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f2c200097d0 tx=0x7f2c2002b840 comp rx=0 tx=0).stop 2024-03-24T00:49:06.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.791+0000 7f2c35b6f640 1 -- 172.21.15.67:0/437314823 shutdown_connections 2024-03-24T00:49:06.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.791+0000 7f2c35b6f640 1 --2- 172.21.15.67:0/437314823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c3015dea0 0x7f2c300d33d0 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:06.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.791+0000 7f2c35b6f640 1 -- 172.21.15.67:0/437314823 >> 172.21.15.67:0/437314823 conn(0x7f2c30072da0 msgr2=0x7f2c300d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:06.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.791+0000 7f2c35b6f640 1 -- 172.21.15.67:0/437314823 shutdown_connections 2024-03-24T00:49:06.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.791+0000 7f2c35b6f640 1 -- 172.21.15.67:0/437314823 wait complete. 2024-03-24T00:49:06.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.793+0000 7f2c35b6f640 1 Processor -- start 2024-03-24T00:49:06.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.793+0000 7f2c35b6f640 1 -- start start 2024-03-24T00:49:06.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.794+0000 7f2c35b6f640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2c300b6170 0x7f2c300cd650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:06.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.794+0000 7f2c35b6f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c3015dea0 0x7f2c3014dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:06.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.794+0000 7f2c35b6f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2c3015e280 0x7f2c3014d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:06.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.794+0000 7f2c35b6f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c300d3ca0 con 0x7f2c3015dea0 2024-03-24T00:49:06.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.794+0000 7f2c35b6f640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c300d3b20 con 0x7f2c3015e280 2024-03-24T00:49:06.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.794+0000 7f2c35b6f640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c300b7800 con 0x7f2c300b6170 2024-03-24T00:49:06.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.794+0000 7f2c2effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c3015dea0 0x7f2c3014dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:06.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.794+0000 7f2c2effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c3015dea0 0x7f2c3014dc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42396/0 (socket says 172.21.15.67:42396) 2024-03-24T00:49:06.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.794+0000 7f2c2effd640 1 -- 172.21.15.67:0/136509712 learned_addr learned my addr 172.21.15.67:0/136509712 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:06.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.795+0000 7f2c2f7fe640 1 --2- 172.21.15.67:0/136509712 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2c300b6170 0x7f2c300cd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:06.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.795+0000 7f2c2ffff640 1 --2- 172.21.15.67:0/136509712 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2c3015e280 0x7f2c3014d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:06.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.795+0000 7f2c2effd640 1 -- 172.21.15.67:0/136509712 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2c300b6170 msgr2=0x7f2c300cd650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:06.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.795+0000 7f2c2effd640 1 --2- 172.21.15.67:0/136509712 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2c300b6170 0x7f2c300cd650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:06.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.795+0000 7f2c2effd640 1 -- 172.21.15.67:0/136509712 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2c3015e280 msgr2=0x7f2c3014d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:06.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.795+0000 7f2c2effd640 1 --2- 172.21.15.67:0/136509712 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2c3015e280 0x7f2c3014d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:06.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.795+0000 7f2c2effd640 1 -- 172.21.15.67:0/136509712 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c3014d930 con 0x7f2c3015dea0 2024-03-24T00:49:06.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.795+0000 7f2c2effd640 1 --2- 172.21.15.67:0/136509712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c3015dea0 0x7f2c3014dc50 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f2c1800c430 tx=0x7f2c1800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:06.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.795+0000 7f2c27fff640 1 -- 172.21.15.67:0/136509712 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c180074e0 con 0x7f2c3015dea0 2024-03-24T00:49:06.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.795+0000 7f2c27fff640 1 -- 172.21.15.67:0/136509712 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c180047e0 con 0x7f2c3015dea0 2024-03-24T00:49:06.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.795+0000 7f2c27fff640 1 -- 172.21.15.67:0/136509712 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c18004bd0 con 0x7f2c3015dea0 2024-03-24T00:49:06.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.795+0000 7f2c35b6f640 1 -- 172.21.15.67:0/136509712 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c3014e350 con 0x7f2c3015dea0 2024-03-24T00:49:06.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.795+0000 7f2c35b6f640 1 -- 172.21.15.67:0/136509712 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c3014e910 con 0x7f2c3015dea0 2024-03-24T00:49:06.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.796+0000 7f2c35b6f640 1 -- 172.21.15.67:0/136509712 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c3004f400 con 0x7f2c3015dea0 2024-03-24T00:49:06.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.796+0000 7f2c27fff640 1 -- 172.21.15.67:0/136509712 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2c18011070 con 0x7f2c3015dea0 2024-03-24T00:49:06.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.797+0000 7f2c27fff640 1 --2- 172.21.15.67:0/136509712 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2c1006bc00 0x7f2c1006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:06.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.797+0000 7f2c27fff640 1 -- 172.21.15.67:0/136509712 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2c1808a880 con 0x7f2c3015dea0 2024-03-24T00:49:06.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.797+0000 7f2c2f7fe640 1 --2- 172.21.15.67:0/136509712 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2c1006bc00 0x7f2c1006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:06.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.798+0000 7f2c2f7fe640 1 --2- 172.21.15.67:0/136509712 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2c1006bc00 0x7f2c1006e0c0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f2c20036000 tx=0x7f2c20008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:06.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:06.801+0000 7f2c27fff640 1 -- 172.21.15.67:0/136509712 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2c1800e280 con 0x7f2c3015dea0 2024-03-24T00:49:07.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.136+0000 7f2c35b6f640 1 -- 172.21.15.67:0/136509712 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2c300cedc0 con 0x7f2c3015dea0 2024-03-24T00:49:07.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.136+0000 7f2c27fff640 1 -- 172.21.15.67:0/136509712 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v120) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2c18059150 con 0x7f2c3015dea0 2024-03-24T00:49:07.137 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-24T00:49:07.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.139+0000 7f2c35b6f640 1 -- 172.21.15.67:0/136509712 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2c1006bc00 msgr2=0x7f2c1006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:07.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.139+0000 7f2c35b6f640 1 --2- 172.21.15.67:0/136509712 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2c1006bc00 0x7f2c1006e0c0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f2c20036000 tx=0x7f2c20008030 comp rx=0 tx=0).stop 2024-03-24T00:49:07.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.139+0000 7f2c35b6f640 1 -- 172.21.15.67:0/136509712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c3015dea0 msgr2=0x7f2c3014dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:07.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.139+0000 7f2c35b6f640 1 --2- 172.21.15.67:0/136509712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c3015dea0 0x7f2c3014dc50 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f2c1800c430 tx=0x7f2c1800c8f0 comp rx=0 tx=0).stop 2024-03-24T00:49:07.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.139+0000 7f2c35b6f640 1 -- 172.21.15.67:0/136509712 shutdown_connections 2024-03-24T00:49:07.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.139+0000 7f2c35b6f640 1 --2- 172.21.15.67:0/136509712 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2c1006bc00 0x7f2c1006e0c0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:07.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.139+0000 7f2c35b6f640 1 --2- 172.21.15.67:0/136509712 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2c3015e280 0x7f2c3014d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:07.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.139+0000 7f2c35b6f640 1 --2- 172.21.15.67:0/136509712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c3015dea0 0x7f2c3014dc50 unknown :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:07.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.139+0000 7f2c35b6f640 1 --2- 172.21.15.67:0/136509712 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2c300b6170 0x7f2c300cd650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:07.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.139+0000 7f2c35b6f640 1 -- 172.21.15.67:0/136509712 >> 172.21.15.67:0/136509712 conn(0x7f2c30072da0 msgr2=0x7f2c300b5380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:07.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.139+0000 7f2c35b6f640 1 -- 172.21.15.67:0/136509712 shutdown_connections 2024-03-24T00:49:07.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.139+0000 7f2c35b6f640 1 -- 172.21.15.67:0/136509712 wait complete. 2024-03-24T00:49:07.151 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:49:07.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.254+0000 7f668b012640 1 Processor -- start 2024-03-24T00:49:07.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.254+0000 7f668b012640 1 -- start start 2024-03-24T00:49:07.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.255+0000 7f668b012640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f668415dea0 0x7f66840d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:07.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.255+0000 7f668b012640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f668415e280 0x7f66840d3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:07.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.255+0000 7f668b012640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66840d2250 con 0x7f66840b5a60 2024-03-24T00:49:07.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.255+0000 7f668b012640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66840d2530 con 0x7f668415e280 2024-03-24T00:49:07.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.255+0000 7f668b012640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66840d2810 con 0x7f668415dea0 2024-03-24T00:49:07.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.255+0000 7f6688dae640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f668415dea0 0x7f66840d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:07.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.255+0000 7f6688dae640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f668415dea0 0x7f66840d33d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.67:46594/0 (socket says 172.21.15.67:46594) 2024-03-24T00:49:07.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.255+0000 7f6688dae640 1 -- 172.21.15.67:0/939936066 learned_addr learned my addr 172.21.15.67:0/939936066 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:07.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.255+0000 7f667bfff640 1 --2- 172.21.15.67:0/939936066 >> v2:172.21.15.72:3300/0 conn(0x7f668415e280 0x7f66840d3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:07.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.255+0000 7f6688dae640 1 -- 172.21.15.67:0/939936066 >> v2:172.21.15.72:3300/0 conn(0x7f668415e280 msgr2=0x7f66840d3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:07.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.255+0000 7f6688dae640 1 --2- 172.21.15.67:0/939936066 >> v2:172.21.15.72:3300/0 conn(0x7f668415e280 0x7f66840d3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:07.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.255+0000 7f6688dae640 1 -- 172.21.15.67:0/939936066 >> v1:172.21.15.72:6789/0 conn(0x7f66840b5a60 legacy=0x7f668406c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:07.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.256+0000 7f6688dae640 1 -- 172.21.15.67:0/939936066 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66840d2af0 con 0x7f668415dea0 2024-03-24T00:49:07.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.256+0000 7f667bfff640 1 --2- 172.21.15.67:0/939936066 >> v2:172.21.15.72:3300/0 conn(0x7f668415e280 0x7f66840d3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:49:07.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.256+0000 7f6688dae640 1 --2- 172.21.15.67:0/939936066 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f668415dea0 0x7f66840d33d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6680009870 tx=0x7f668002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=716b9a84592e3463 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:07.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.256+0000 7f667b7fe640 1 -- 172.21.15.67:0/939936066 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6680035020 con 0x7f668415dea0 2024-03-24T00:49:07.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.256+0000 7f667b7fe640 1 -- 172.21.15.67:0/939936066 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6680031640 con 0x7f668415dea0 2024-03-24T00:49:07.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.257+0000 7f667b7fe640 1 -- 172.21.15.67:0/939936066 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f668002d040 con 0x7f668415dea0 2024-03-24T00:49:07.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.257+0000 7f668b012640 1 -- 172.21.15.67:0/939936066 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f668415dea0 msgr2=0x7f66840d33d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:07.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.257+0000 7f668b012640 1 --2- 172.21.15.67:0/939936066 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f668415dea0 0x7f66840d33d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6680009870 tx=0x7f668002b8f0 comp rx=0 tx=0).stop 2024-03-24T00:49:07.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.257+0000 7f668b012640 1 -- 172.21.15.67:0/939936066 shutdown_connections 2024-03-24T00:49:07.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.257+0000 7f668b012640 1 --2- 172.21.15.67:0/939936066 >> v2:172.21.15.72:3300/0 conn(0x7f668415e280 0x7f66840d3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:07.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.257+0000 7f668b012640 1 --2- 172.21.15.67:0/939936066 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f668415dea0 0x7f66840d33d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:07.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.257+0000 7f668b012640 1 -- 172.21.15.67:0/939936066 >> 172.21.15.67:0/939936066 conn(0x7f6684072da0 msgr2=0x7f66840d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:07.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.257+0000 7f668b012640 1 -- 172.21.15.67:0/939936066 shutdown_connections 2024-03-24T00:49:07.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.257+0000 7f668b012640 1 -- 172.21.15.67:0/939936066 wait complete. 2024-03-24T00:49:07.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.259+0000 7f668b012640 1 Processor -- start 2024-03-24T00:49:07.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.259+0000 7f668b012640 1 -- start start 2024-03-24T00:49:07.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.260+0000 7f668b012640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f66840b5a60 0x7f668414cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:07.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.260+0000 7f668b012640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f668415dea0 0x7f66840ad6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:07.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.260+0000 7f668b012640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f668415e280 0x7f66840adc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:07.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.260+0000 7f668b012640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66840b5ed0 con 0x7f668415dea0 2024-03-24T00:49:07.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.260+0000 7f668b012640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66840b8350 con 0x7f668415e280 2024-03-24T00:49:07.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.260+0000 7f668b012640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66840b6050 con 0x7f66840b5a60 2024-03-24T00:49:07.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.260+0000 7f667bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f668415dea0 0x7f66840ad6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:07.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.260+0000 7f667bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f668415dea0 0x7f66840ad6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42404/0 (socket says 172.21.15.67:42404) 2024-03-24T00:49:07.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.260+0000 7f667bfff640 1 -- 172.21.15.67:0/3561778124 learned_addr learned my addr 172.21.15.67:0/3561778124 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:07.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.260+0000 7f6688dae640 1 --2- 172.21.15.67:0/3561778124 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f66840b5a60 0x7f668414cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:07.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.260+0000 7f66895af640 1 --2- 172.21.15.67:0/3561778124 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f668415e280 0x7f66840adc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:07.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.260+0000 7f667bfff640 1 -- 172.21.15.67:0/3561778124 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f66840b5a60 msgr2=0x7f668414cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:07.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.260+0000 7f667bfff640 1 --2- 172.21.15.67:0/3561778124 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f66840b5a60 0x7f668414cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:07.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.260+0000 7f667bfff640 1 -- 172.21.15.67:0/3561778124 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f668415e280 msgr2=0x7f66840adc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:07.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.260+0000 7f667bfff640 1 --2- 172.21.15.67:0/3561778124 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f668415e280 0x7f66840adc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:07.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.260+0000 7f667bfff640 1 -- 172.21.15.67:0/3561778124 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66840cd840 con 0x7f668415dea0 2024-03-24T00:49:07.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.260+0000 7f667bfff640 1 --2- 172.21.15.67:0/3561778124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f668415dea0 0x7f66840ad6c0 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f667400e960 tx=0x7f667400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:07.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.261+0000 7f66797fa640 1 -- 172.21.15.67:0/3561778124 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6674017070 con 0x7f668415dea0 2024-03-24T00:49:07.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.261+0000 7f66797fa640 1 -- 172.21.15.67:0/3561778124 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f667400ca00 con 0x7f668415dea0 2024-03-24T00:49:07.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.261+0000 7f668b012640 1 -- 172.21.15.67:0/3561778124 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f668414da90 con 0x7f668415dea0 2024-03-24T00:49:07.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.261+0000 7f66797fa640 1 -- 172.21.15.67:0/3561778124 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f667400cdd0 con 0x7f668415dea0 2024-03-24T00:49:07.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.261+0000 7f668b012640 1 -- 172.21.15.67:0/3561778124 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f668414d500 con 0x7f668415dea0 2024-03-24T00:49:07.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.262+0000 7f668b012640 1 -- 172.21.15.67:0/3561778124 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66840ab520 con 0x7f668415dea0 2024-03-24T00:49:07.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.262+0000 7f66797fa640 1 -- 172.21.15.67:0/3561778124 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f667401a070 con 0x7f668415dea0 2024-03-24T00:49:07.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.262+0000 7f66797fa640 1 --2- 172.21.15.67:0/3561778124 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f667006bbd0 0x7f667006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:07.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.262+0000 7f66797fa640 1 -- 172.21.15.67:0/3561778124 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f667408adb0 con 0x7f668415dea0 2024-03-24T00:49:07.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.263+0000 7f6688dae640 1 --2- 172.21.15.67:0/3561778124 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f667006bbd0 0x7f667006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:07.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.263+0000 7f6688dae640 1 --2- 172.21.15.67:0/3561778124 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f667006bbd0 0x7f667006e090 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f6680036000 tx=0x7f668003c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:07.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.267+0000 7f66797fa640 1 -- 172.21.15.67:0/3561778124 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f66740590c0 con 0x7f668415dea0 2024-03-24T00:49:07.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.448+0000 7f668b012640 1 -- 172.21.15.67:0/3561778124 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f668404f400 con 0x7f668415dea0 2024-03-24T00:49:07.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.449+0000 7f66797fa640 1 -- 172.21.15.67:0/3561778124 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v120) v1 ==== 104+0+11 (secure 0 0 0) 0x7f667405d160 con 0x7f668415dea0 2024-03-24T00:49:07.450 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-24T00:49:07.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.451+0000 7f668b012640 1 -- 172.21.15.67:0/3561778124 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f667006bbd0 msgr2=0x7f667006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:07.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.451+0000 7f668b012640 1 --2- 172.21.15.67:0/3561778124 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f667006bbd0 0x7f667006e090 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f6680036000 tx=0x7f668003c040 comp rx=0 tx=0).stop 2024-03-24T00:49:07.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.451+0000 7f668b012640 1 -- 172.21.15.67:0/3561778124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f668415dea0 msgr2=0x7f66840ad6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:07.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.451+0000 7f668b012640 1 --2- 172.21.15.67:0/3561778124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f668415dea0 0x7f66840ad6c0 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f667400e960 tx=0x7f667400ee20 comp rx=0 tx=0).stop 2024-03-24T00:49:07.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.451+0000 7f668b012640 1 -- 172.21.15.67:0/3561778124 shutdown_connections 2024-03-24T00:49:07.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.451+0000 7f668b012640 1 --2- 172.21.15.67:0/3561778124 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f667006bbd0 0x7f667006e090 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:07.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.451+0000 7f668b012640 1 --2- 172.21.15.67:0/3561778124 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f668415e280 0x7f66840adc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:07.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.451+0000 7f668b012640 1 --2- 172.21.15.67:0/3561778124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f668415dea0 0x7f66840ad6c0 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:07.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.451+0000 7f668b012640 1 --2- 172.21.15.67:0/3561778124 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f66840b5a60 0x7f668414cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:07.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.451+0000 7f668b012640 1 -- 172.21.15.67:0/3561778124 >> 172.21.15.67:0/3561778124 conn(0x7f6684072da0 msgr2=0x7f66840b39e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:07.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.451+0000 7f668b012640 1 -- 172.21.15.67:0/3561778124 shutdown_connections 2024-03-24T00:49:07.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.451+0000 7f668b012640 1 -- 172.21.15.67:0/3561778124 wait complete. 2024-03-24T00:49:07.462 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:49:07.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.566+0000 7ff5c59e6640 1 Processor -- start 2024-03-24T00:49:07.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.566+0000 7ff5c59e6640 1 -- start start 2024-03-24T00:49:07.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.567+0000 7ff5c59e6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff5c00d18b0 0x7ff5c015c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:07.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.567+0000 7ff5c59e6640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff5c0160430 0x7ff5c00cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:07.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.567+0000 7ff5c59e6640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5c015c7c0 con 0x7ff5c00d1c90 2024-03-24T00:49:07.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.567+0000 7ff5c59e6640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5c015caa0 con 0x7ff5c0160430 2024-03-24T00:49:07.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.567+0000 7ff5c59e6640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c015cd80 con 0x7ff5c00d18b0 2024-03-24T00:49:07.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.567+0000 7ff5be7fc640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff5c00d1c90 0x7ff5c015fd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48710/0 (socket says 172.21.15.67:48710) 2024-03-24T00:49:07.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.567+0000 7ff5be7fc640 1 -- 172.21.15.67:0/3647214291 learned_addr learned my addr 172.21.15.67:0/3647214291 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:49:07.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.568+0000 7ff5bdffb640 1 -- 172.21.15.67:0/3647214291 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3817046553 0 0) 0x7ff5c015c7c0 con 0x7ff5c00d1c90 2024-03-24T00:49:07.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.568+0000 7ff5beffd640 1 --2- 172.21.15.67:0/3647214291 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff5c00d18b0 0x7ff5c015c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:07.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.568+0000 7ff5bdffb640 1 -- 172.21.15.67:0/3647214291 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5ac003520 con 0x7ff5c00d1c90 2024-03-24T00:49:07.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.568+0000 7ff5bf7fe640 1 --2- 172.21.15.67:0/3647214291 >> v2:172.21.15.72:3300/0 conn(0x7ff5c0160430 0x7ff5c00cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:07.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.568+0000 7ff5bdffb640 1 -- 172.21.15.67:0/3647214291 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1465918116 0 0) 0x7ff5ac003520 con 0x7ff5c00d1c90 2024-03-24T00:49:07.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.568+0000 7ff5bdffb640 1 -- 172.21.15.67:0/3647214291 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff5c00d18b0 msgr2=0x7ff5c015c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:07.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.568+0000 7ff5bdffb640 1 --2- 172.21.15.67:0/3647214291 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff5c00d18b0 0x7ff5c015c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:07.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.568+0000 7ff5bdffb640 1 -- 172.21.15.67:0/3647214291 >> v2:172.21.15.72:3300/0 conn(0x7ff5c0160430 msgr2=0x7ff5c00cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:07.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.568+0000 7ff5bdffb640 1 --2- 172.21.15.67:0/3647214291 >> v2:172.21.15.72:3300/0 conn(0x7ff5c0160430 0x7ff5c00cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:07.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.568+0000 7ff5bdffb640 1 -- 172.21.15.67:0/3647214291 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5c015d060 con 0x7ff5c00d1c90 2024-03-24T00:49:07.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.568+0000 7ff5bdffb640 1 -- 172.21.15.67:0/3647214291 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7ff5b0001670 con 0x7ff5c00d1c90 2024-03-24T00:49:07.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.568+0000 7ff5bdffb640 1 -- 172.21.15.67:0/3647214291 >> v1:172.21.15.67:6789/0 conn(0x7ff5c00d1c90 legacy=0x7ff5c015fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:07.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.568+0000 7ff5bdffb640 1 --2- 172.21.15.67:0/3647214291 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff5c00d18b0 0x7ff5ac0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:07.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.568+0000 7ff5bdffb640 1 --2- 172.21.15.67:0/3647214291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5ac003f30 0x7ff5ac004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:07.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.568+0000 7ff5bdffb640 1 --2- 172.21.15.67:0/3647214291 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff5ac004d40 0x7ff5ac0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:07.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.568+0000 7ff5bdffb640 1 -- 172.21.15.67:0/3647214291 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c015caa0 con 0x7ff5ac003f30 2024-03-24T00:49:07.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.568+0000 7ff5bdffb640 1 -- 172.21.15.67:0/3647214291 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c015cd80 con 0x7ff5ac004d40 2024-03-24T00:49:07.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.568+0000 7ff5bdffb640 1 -- 172.21.15.67:0/3647214291 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5ac00ba10 con 0x7ff5c00d18b0 2024-03-24T00:49:07.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.568+0000 7ff5be7fc640 1 --2- 172.21.15.67:0/3647214291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5ac003f30 0x7ff5ac004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:07.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.568+0000 7ff5bf7fe640 1 --2- 172.21.15.67:0/3647214291 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff5ac004d40 0x7ff5ac0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:07.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.568+0000 7ff5beffd640 1 --2- 172.21.15.67:0/3647214291 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff5c00d18b0 0x7ff5ac0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:07.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.568+0000 7ff5be7fc640 1 -- 172.21.15.67:0/3647214291 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff5c00d18b0 msgr2=0x7ff5ac0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:07.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.568+0000 7ff5be7fc640 1 --2- 172.21.15.67:0/3647214291 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff5c00d18b0 0x7ff5ac0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:07.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.568+0000 7ff5be7fc640 1 -- 172.21.15.67:0/3647214291 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff5ac004d40 msgr2=0x7ff5ac0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:07.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.568+0000 7ff5be7fc640 1 --2- 172.21.15.67:0/3647214291 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff5ac004d40 0x7ff5ac0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:07.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.568+0000 7ff5be7fc640 1 -- 172.21.15.67:0/3647214291 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5ac003520 con 0x7ff5ac003f30 2024-03-24T00:49:07.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.569+0000 7ff5be7fc640 1 --2- 172.21.15.67:0/3647214291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5ac003f30 0x7ff5ac004800 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7ff5b000c240 tx=0x7ff5ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d6f3a946d235975 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:07.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.569+0000 7ff5bdffb640 1 -- 172.21.15.67:0/3647214291 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5b0007d50 con 0x7ff5ac003f30 2024-03-24T00:49:07.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.569+0000 7ff5bdffb640 1 -- 172.21.15.67:0/3647214291 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5b0030070 con 0x7ff5ac003f30 2024-03-24T00:49:07.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.569+0000 7ff5bdffb640 1 -- 172.21.15.67:0/3647214291 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5b002cdc0 con 0x7ff5ac003f30 2024-03-24T00:49:07.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.570+0000 7ff5c59e6640 1 -- 172.21.15.67:0/3647214291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5ac003f30 msgr2=0x7ff5ac004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:07.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.570+0000 7ff5c59e6640 1 --2- 172.21.15.67:0/3647214291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5ac003f30 0x7ff5ac004800 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7ff5b000c240 tx=0x7ff5ac005830 comp rx=0 tx=0).stop 2024-03-24T00:49:07.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.570+0000 7ff5bf7fe640 1 -- 172.21.15.67:0/3647214291 reap_dead start 2024-03-24T00:49:07.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.570+0000 7ff5c59e6640 1 -- 172.21.15.67:0/3647214291 shutdown_connections 2024-03-24T00:49:07.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.570+0000 7ff5c59e6640 1 -- 172.21.15.67:0/3647214291 >> 172.21.15.67:0/3647214291 conn(0x7ff5c008b1b0 msgr2=0x7ff5c008b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:07.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.570+0000 7ff5c59e6640 1 -- 172.21.15.67:0/3647214291 shutdown_connections 2024-03-24T00:49:07.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.570+0000 7ff5c59e6640 1 -- 172.21.15.67:0/3647214291 wait complete. 2024-03-24T00:49:07.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.571+0000 7ff5c59e6640 1 Processor -- start 2024-03-24T00:49:07.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.571+0000 7ff5c59e6640 1 -- start start 2024-03-24T00:49:07.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.572+0000 7ff5c59e6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff5ac003f30 0x7ff5c00b0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:07.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.572+0000 7ff5c59e6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff5c00c7da0 0x7ff5c00b0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:07.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.572+0000 7ff5c59e6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5c00c8220 0x7ff5c00b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:07.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.572+0000 7ff5c59e6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c015c1b0 con 0x7ff5c00c8220 2024-03-24T00:49:07.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.572+0000 7ff5c59e6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c00d2300 con 0x7ff5c00c7da0 2024-03-24T00:49:07.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.572+0000 7ff5c59e6640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c015c330 con 0x7ff5ac003f30 2024-03-24T00:49:07.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.572+0000 7ff5bf7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5c00c8220 0x7ff5c00b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:07.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.572+0000 7ff5bf7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5c00c8220 0x7ff5c00b12a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42420/0 (socket says 172.21.15.67:42420) 2024-03-24T00:49:07.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.572+0000 7ff5bf7fe640 1 -- 172.21.15.67:0/3050590939 learned_addr learned my addr 172.21.15.67:0/3050590939 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:07.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.572+0000 7ff5be7fc640 1 --2- 172.21.15.67:0/3050590939 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff5c00c7da0 0x7ff5c00b0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:07.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.572+0000 7ff5beffd640 1 --2- 172.21.15.67:0/3050590939 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff5ac003f30 0x7ff5c00b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:07.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.572+0000 7ff5bf7fe640 1 -- 172.21.15.67:0/3050590939 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff5ac003f30 msgr2=0x7ff5c00b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:07.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.572+0000 7ff5bf7fe640 1 --2- 172.21.15.67:0/3050590939 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff5ac003f30 0x7ff5c00b0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:07.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.572+0000 7ff5bf7fe640 1 -- 172.21.15.67:0/3050590939 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff5c00c7da0 msgr2=0x7ff5c00b0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:07.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.572+0000 7ff5bf7fe640 1 --2- 172.21.15.67:0/3050590939 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff5c00c7da0 0x7ff5c00b0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:07.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.572+0000 7ff5bf7fe640 1 -- 172.21.15.67:0/3050590939 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5c00af100 con 0x7ff5c00c8220 2024-03-24T00:49:07.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.573+0000 7ff5bf7fe640 1 --2- 172.21.15.67:0/3050590939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5c00c8220 0x7ff5c00b12a0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7ff5c00b4790 tx=0x7ff5b800f1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:07.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.573+0000 7ff5b7fff640 1 -- 172.21.15.67:0/3050590939 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5b800fce0 con 0x7ff5c00c8220 2024-03-24T00:49:07.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.573+0000 7ff5b7fff640 1 -- 172.21.15.67:0/3050590939 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5b800fe80 con 0x7ff5c00c8220 2024-03-24T00:49:07.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.573+0000 7ff5c59e6640 1 -- 172.21.15.67:0/3050590939 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5c00af3c0 con 0x7ff5c00c8220 2024-03-24T00:49:07.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.573+0000 7ff5b7fff640 1 -- 172.21.15.67:0/3050590939 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5b8015670 con 0x7ff5c00c8220 2024-03-24T00:49:07.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.573+0000 7ff5c59e6640 1 -- 172.21.15.67:0/3050590939 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5c00afc60 con 0x7ff5c00c8220 2024-03-24T00:49:07.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.574+0000 7ff5c59e6640 1 -- 172.21.15.67:0/3050590939 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5c004f400 con 0x7ff5c00c8220 2024-03-24T00:49:07.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.574+0000 7ff5b7fff640 1 -- 172.21.15.67:0/3050590939 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff5b8015810 con 0x7ff5c00c8220 2024-03-24T00:49:07.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.575+0000 7ff5b7fff640 1 --2- 172.21.15.67:0/3050590939 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff5a006bc00 0x7ff5a006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:07.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.575+0000 7ff5b7fff640 1 -- 172.21.15.67:0/3050590939 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff5b808b920 con 0x7ff5c00c8220 2024-03-24T00:49:07.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.575+0000 7ff5beffd640 1 --2- 172.21.15.67:0/3050590939 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff5a006bc00 0x7ff5a006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:07.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.575+0000 7ff5beffd640 1 --2- 172.21.15.67:0/3050590939 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff5a006bc00 0x7ff5a006e0c0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7ff5a8009a60 tx=0x7ff5a8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:07.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.583+0000 7ff5b7fff640 1 -- 172.21.15.67:0/3050590939 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff5b8019030 con 0x7ff5c00c8220 2024-03-24T00:49:07.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.765+0000 7ff5c59e6640 1 -- 172.21.15.67:0/3050590939 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff5c00b1840 con 0x7ff5c00c8220 2024-03-24T00:49:07.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.765+0000 7ff5b7fff640 1 -- 172.21.15.67:0/3050590939 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v120) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff5b8010040 con 0x7ff5c00c8220 2024-03-24T00:49:07.766 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-03-24T00:49:07.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.767+0000 7ff5c59e6640 1 -- 172.21.15.67:0/3050590939 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff5a006bc00 msgr2=0x7ff5a006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:07.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.767+0000 7ff5c59e6640 1 --2- 172.21.15.67:0/3050590939 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff5a006bc00 0x7ff5a006e0c0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7ff5a8009a60 tx=0x7ff5a8009190 comp rx=0 tx=0).stop 2024-03-24T00:49:07.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.767+0000 7ff5c59e6640 1 -- 172.21.15.67:0/3050590939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5c00c8220 msgr2=0x7ff5c00b12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:07.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.767+0000 7ff5c59e6640 1 --2- 172.21.15.67:0/3050590939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5c00c8220 0x7ff5c00b12a0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7ff5c00b4790 tx=0x7ff5b800f1b0 comp rx=0 tx=0).stop 2024-03-24T00:49:07.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.767+0000 7ff5c59e6640 1 -- 172.21.15.67:0/3050590939 shutdown_connections 2024-03-24T00:49:07.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.767+0000 7ff5c59e6640 1 --2- 172.21.15.67:0/3050590939 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff5a006bc00 0x7ff5a006e0c0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:07.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.767+0000 7ff5c59e6640 1 --2- 172.21.15.67:0/3050590939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5c00c8220 0x7ff5c00b12a0 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:07.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.767+0000 7ff5c59e6640 1 --2- 172.21.15.67:0/3050590939 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff5c00c7da0 0x7ff5c00b0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:07.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.767+0000 7ff5c59e6640 1 --2- 172.21.15.67:0/3050590939 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff5ac003f30 0x7ff5c00b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:07.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.767+0000 7ff5c59e6640 1 -- 172.21.15.67:0/3050590939 >> 172.21.15.67:0/3050590939 conn(0x7ff5c008b1b0 msgr2=0x7ff5c00ce6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:07.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.767+0000 7ff5c59e6640 1 -- 172.21.15.67:0/3050590939 shutdown_connections 2024-03-24T00:49:07.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.767+0000 7ff5c59e6640 1 -- 172.21.15.67:0/3050590939 wait complete. 2024-03-24T00:49:07.780 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T00:49:07.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.885+0000 7f7965d1b640 1 Processor -- start 2024-03-24T00:49:07.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.885+0000 7f7965d1b640 1 -- start start 2024-03-24T00:49:07.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.885+0000 7f7965d1b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f796015e0f0 0x7f79600b95b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:07.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.885+0000 7f7965d1b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f79600b6dc0 0x7f79600b7210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:07.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.885+0000 7f7965d1b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79600d2230 con 0x7f79600b9af0 2024-03-24T00:49:07.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.885+0000 7f7965d1b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79600b7750 con 0x7f796015e0f0 2024-03-24T00:49:07.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.885+0000 7f7965d1b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79600d2510 con 0x7f79600b6dc0 2024-03-24T00:49:07.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.886+0000 7f795ffff640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f79600b6dc0 0x7f79600b7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:07.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.886+0000 7f795f7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f796015e0f0 0x7f79600b95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:07.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.886+0000 7f795f7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f796015e0f0 0x7f79600b95b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:59460/0 (socket says 172.21.15.67:59460) 2024-03-24T00:49:07.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.886+0000 7f795ffff640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f79600b6dc0 0x7f79600b7210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.67:46636/0 (socket says 172.21.15.67:46636) 2024-03-24T00:49:07.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.886+0000 7f795f7fe640 1 -- 172.21.15.67:0/420467165 learned_addr learned my addr 172.21.15.67:0/420467165 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:07.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.886+0000 7f795effd640 1 --1- 172.21.15.67:0/420467165 >> v1:172.21.15.72:6789/0 conn(0x7f79600b9af0 0x7f79600d32d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:40432/0 (socket says 172.21.15.67:40432) 2024-03-24T00:49:07.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.886+0000 7f795f7fe640 1 -- 172.21.15.67:0/420467165 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f79600b6dc0 msgr2=0x7f79600b7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:07.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.886+0000 7f795f7fe640 1 --2- 172.21.15.67:0/420467165 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f79600b6dc0 0x7f79600b7210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:07.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.886+0000 7f795f7fe640 1 -- 172.21.15.67:0/420467165 >> v1:172.21.15.72:6789/0 conn(0x7f79600b9af0 legacy=0x7f79600d32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:07.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.886+0000 7f795e7fc640 1 -- 172.21.15.67:0/420467165 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2184977362 0 0) 0x7f79600d2230 con 0x7f79600b9af0 2024-03-24T00:49:07.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.886+0000 7f795f7fe640 1 -- 172.21.15.67:0/420467165 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79600d27f0 con 0x7f796015e0f0 2024-03-24T00:49:07.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.887+0000 7f795f7fe640 1 --2- 172.21.15.67:0/420467165 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f796015e0f0 0x7f79600b95b0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f7950009870 tx=0x7f795002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dac75931c047cf5b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:07.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.888+0000 7f795e7fc640 1 -- 172.21.15.67:0/420467165 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7950034070 con 0x7f796015e0f0 2024-03-24T00:49:07.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.888+0000 7f795e7fc640 1 -- 172.21.15.67:0/420467165 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7950030840 con 0x7f796015e0f0 2024-03-24T00:49:07.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.888+0000 7f795e7fc640 1 -- 172.21.15.67:0/420467165 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7950030c10 con 0x7f796015e0f0 2024-03-24T00:49:07.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.888+0000 7f7965d1b640 1 -- 172.21.15.67:0/420467165 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f796015e0f0 msgr2=0x7f79600b95b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:07.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.888+0000 7f7965d1b640 1 --2- 172.21.15.67:0/420467165 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f796015e0f0 0x7f79600b95b0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f7950009870 tx=0x7f795002b980 comp rx=0 tx=0).stop 2024-03-24T00:49:07.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.888+0000 7f7965d1b640 1 -- 172.21.15.67:0/420467165 shutdown_connections 2024-03-24T00:49:07.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.888+0000 7f7965d1b640 1 --2- 172.21.15.67:0/420467165 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f79600b6dc0 0x7f79600b7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:07.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.888+0000 7f7965d1b640 1 --2- 172.21.15.67:0/420467165 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f796015e0f0 0x7f79600b95b0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:07.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.888+0000 7f7965d1b640 1 -- 172.21.15.67:0/420467165 >> 172.21.15.67:0/420467165 conn(0x7f7960072b60 msgr2=0x7f7960072f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:07.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.888+0000 7f7965d1b640 1 -- 172.21.15.67:0/420467165 shutdown_connections 2024-03-24T00:49:07.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.888+0000 7f7965d1b640 1 -- 172.21.15.67:0/420467165 wait complete. 2024-03-24T00:49:07.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.889+0000 7f7965d1b640 1 Processor -- start 2024-03-24T00:49:07.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.889+0000 7f7965d1b640 1 -- start start 2024-03-24T00:49:07.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.890+0000 7f7965d1b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79600b6dc0 0x7f7960160c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:07.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.890+0000 7f7965d1b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f79600b9af0 0x7f79600ab130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:07.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.890+0000 7f7965d1b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f796015e0f0 0x7f79600ac660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:07.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.890+0000 7f7965d1b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79600b31d0 con 0x7f796015e0f0 2024-03-24T00:49:07.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.890+0000 7f7965d1b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79600d2db0 con 0x7f79600b6dc0 2024-03-24T00:49:07.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.890+0000 7f7965d1b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79600b3350 con 0x7f79600b9af0 2024-03-24T00:49:07.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.890+0000 7f795ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f796015e0f0 0x7f79600ac660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:07.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.890+0000 7f795ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f796015e0f0 0x7f79600ac660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42422/0 (socket says 172.21.15.67:42422) 2024-03-24T00:49:07.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.890+0000 7f795ffff640 1 -- 172.21.15.67:0/2466653987 learned_addr learned my addr 172.21.15.67:0/2466653987 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:07.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.890+0000 7f795f7fe640 1 --2- 172.21.15.67:0/2466653987 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79600b6dc0 0x7f7960160c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:07.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.890+0000 7f795effd640 1 --2- 172.21.15.67:0/2466653987 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f79600b9af0 0x7f79600ab130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:07.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.890+0000 7f795ffff640 1 -- 172.21.15.67:0/2466653987 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f79600b9af0 msgr2=0x7f79600ab130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:07.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.890+0000 7f795ffff640 1 --2- 172.21.15.67:0/2466653987 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f79600b9af0 0x7f79600ab130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:07.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.890+0000 7f795ffff640 1 -- 172.21.15.67:0/2466653987 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79600b6dc0 msgr2=0x7f7960160c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:07.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.891+0000 7f795ffff640 1 --2- 172.21.15.67:0/2466653987 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79600b6dc0 0x7f7960160c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:07.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.891+0000 7f795ffff640 1 -- 172.21.15.67:0/2466653987 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79600ab670 con 0x7f796015e0f0 2024-03-24T00:49:07.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.891+0000 7f795ffff640 1 --2- 172.21.15.67:0/2466653987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f796015e0f0 0x7f79600ac660 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7f795800efc0 tx=0x7f795800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:07.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.891+0000 7f7947fff640 1 -- 172.21.15.67:0/2466653987 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7958018020 con 0x7f796015e0f0 2024-03-24T00:49:07.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.891+0000 7f7947fff640 1 -- 172.21.15.67:0/2466653987 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7958014480 con 0x7f796015e0f0 2024-03-24T00:49:07.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.891+0000 7f7965d1b640 1 -- 172.21.15.67:0/2466653987 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79600ab910 con 0x7f796015e0f0 2024-03-24T00:49:07.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.891+0000 7f7947fff640 1 -- 172.21.15.67:0/2466653987 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7958014870 con 0x7f796015e0f0 2024-03-24T00:49:07.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.891+0000 7f7965d1b640 1 -- 172.21.15.67:0/2466653987 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79600abed0 con 0x7f796015e0f0 2024-03-24T00:49:07.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.892+0000 7f7965d1b640 1 -- 172.21.15.67:0/2466653987 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f796015e560 con 0x7f796015e0f0 2024-03-24T00:49:07.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.892+0000 7f7947fff640 1 -- 172.21.15.67:0/2466653987 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7958005d10 con 0x7f796015e0f0 2024-03-24T00:49:07.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.893+0000 7f7947fff640 1 --2- 172.21.15.67:0/2466653987 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f794c06bbd0 0x7f794c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:07.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.893+0000 7f7947fff640 1 -- 172.21.15.67:0/2466653987 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f795808b4f0 con 0x7f796015e0f0 2024-03-24T00:49:07.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.893+0000 7f795f7fe640 1 --2- 172.21.15.67:0/2466653987 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f794c06bbd0 0x7f794c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:07.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.893+0000 7f795f7fe640 1 --2- 172.21.15.67:0/2466653987 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f794c06bbd0 0x7f794c06e090 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f7950036000 tx=0x7f795003b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:07.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:07.897+0000 7f7947fff640 1 -- 172.21.15.67:0/2466653987 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7958022070 con 0x7f796015e0f0 2024-03-24T00:49:08.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.178+0000 7f7965d1b640 1 -- 172.21.15.67:0/2466653987 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f79600b4300 con 0x7f796015e0f0 2024-03-24T00:49:08.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.178+0000 7f7947fff640 1 -- 172.21.15.67:0/2466653987 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v120) v1 ==== 80+0+0 (secure 0 0 0) 0x7f7958010040 con 0x7f796015e0f0 2024-03-24T00:49:08.179 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-03-24T00:49:08.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.180+0000 7f7965d1b640 1 -- 172.21.15.67:0/2466653987 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f794c06bbd0 msgr2=0x7f794c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:08.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.180+0000 7f7965d1b640 1 --2- 172.21.15.67:0/2466653987 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f794c06bbd0 0x7f794c06e090 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f7950036000 tx=0x7f795003b040 comp rx=0 tx=0).stop 2024-03-24T00:49:08.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.180+0000 7f7965d1b640 1 -- 172.21.15.67:0/2466653987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f796015e0f0 msgr2=0x7f79600ac660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:08.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.180+0000 7f7965d1b640 1 --2- 172.21.15.67:0/2466653987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f796015e0f0 0x7f79600ac660 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7f795800efc0 tx=0x7f795800c490 comp rx=0 tx=0).stop 2024-03-24T00:49:08.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.180+0000 7f7965d1b640 1 -- 172.21.15.67:0/2466653987 shutdown_connections 2024-03-24T00:49:08.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.180+0000 7f7965d1b640 1 --2- 172.21.15.67:0/2466653987 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f794c06bbd0 0x7f794c06e090 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:08.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.180+0000 7f7965d1b640 1 --2- 172.21.15.67:0/2466653987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f796015e0f0 0x7f79600ac660 unknown :-1 s=CLOSED pgs=1750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:08.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.180+0000 7f7965d1b640 1 --2- 172.21.15.67:0/2466653987 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f79600b9af0 0x7f79600ab130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:08.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.180+0000 7f7965d1b640 1 --2- 172.21.15.67:0/2466653987 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79600b6dc0 0x7f7960160c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:08.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.181+0000 7f7965d1b640 1 -- 172.21.15.67:0/2466653987 >> 172.21.15.67:0/2466653987 conn(0x7f7960072b60 msgr2=0x7f79600d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:08.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.181+0000 7f7965d1b640 1 -- 172.21.15.67:0/2466653987 shutdown_connections 2024-03-24T00:49:08.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.181+0000 7f7965d1b640 1 -- 172.21.15.67:0/2466653987 wait complete. 2024-03-24T00:49:08.192 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-24T00:49:08.192 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:08.192 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-24T00:49:08.259 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-03-24T00:49:08.259 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:08.259 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-24T00:49:08.278 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:49:08.279 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.8:0 flush journal 2024-03-24T00:49:08.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.422+0000 7f0bcaa63640 1 Processor -- start 2024-03-24T00:49:08.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.422+0000 7f0bcaa63640 1 -- start start 2024-03-24T00:49:08.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.423+0000 7f0bcaa63640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f0bc415dea0 0x7f0bc40d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:08.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.423+0000 7f0bcaa63640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0bc415e280 0x7f0bc40d3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:08.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.423+0000 7f0bcaa63640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bc40d2250 con 0x7f0bc40b5a60 2024-03-24T00:49:08.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.423+0000 7f0bcaa63640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bc40d2530 con 0x7f0bc415dea0 2024-03-24T00:49:08.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.423+0000 7f0bcaa63640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc40d2810 con 0x7f0bc415e280 2024-03-24T00:49:08.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.423+0000 7f0bc9000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0bc40b5a60 0x7f0bc406c640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48714/0 (socket says 172.21.15.67:48714) 2024-03-24T00:49:08.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.423+0000 7f0bc9000640 1 -- 172.21.15.67:0/3716733185 learned_addr learned my addr 172.21.15.67:0/3716733185 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:49:08.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.423+0000 7f0bc3fff640 1 --2- 172.21.15.67:0/3716733185 >> v2:172.21.15.72:3300/0 conn(0x7f0bc415dea0 0x7f0bc40d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:08.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.423+0000 7f0bbbfff640 1 --2- 172.21.15.67:0/3716733185 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0bc415e280 0x7f0bc40d3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:08.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.423+0000 7f0bc37fe640 1 -- 172.21.15.67:0/3716733185 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1262952100 0 0) 0x7f0bc40d2250 con 0x7f0bc40b5a60 2024-03-24T00:49:08.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.423+0000 7f0bc37fe640 1 -- 172.21.15.67:0/3716733185 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bb0003520 con 0x7f0bc40b5a60 2024-03-24T00:49:08.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.423+0000 7f0bc37fe640 1 -- 172.21.15.67:0/3716733185 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3976692321 0 0) 0x7f0bb0003520 con 0x7f0bc40b5a60 2024-03-24T00:49:08.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.423+0000 7f0bc37fe640 1 -- 172.21.15.67:0/3716733185 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0bc415e280 msgr2=0x7f0bc40d3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:08.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.423+0000 7f0bc37fe640 1 --2- 172.21.15.67:0/3716733185 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0bc415e280 0x7f0bc40d3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:08.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.423+0000 7f0bc37fe640 1 -- 172.21.15.67:0/3716733185 >> v2:172.21.15.72:3300/0 conn(0x7f0bc415dea0 msgr2=0x7f0bc40d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:08.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.423+0000 7f0bc37fe640 1 --2- 172.21.15.67:0/3716733185 >> v2:172.21.15.72:3300/0 conn(0x7f0bc415dea0 0x7f0bc40d33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:08.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.423+0000 7f0bc37fe640 1 -- 172.21.15.67:0/3716733185 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc40d2af0 con 0x7f0bc40b5a60 2024-03-24T00:49:08.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.423+0000 7f0bc37fe640 1 -- 172.21.15.67:0/3716733185 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f0bbc0019f0 con 0x7f0bc40b5a60 2024-03-24T00:49:08.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.423+0000 7f0bc37fe640 1 -- 172.21.15.67:0/3716733185 >> v1:172.21.15.67:6789/0 conn(0x7f0bc40b5a60 legacy=0x7f0bc406c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:08.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.423+0000 7f0bc37fe640 1 --2- 172.21.15.67:0/3716733185 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0bc415e280 0x7f0bb00039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:08.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.423+0000 7f0bc37fe640 1 --2- 172.21.15.67:0/3716733185 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0bb0003f30 0x7f0bb0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:08.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.424+0000 7f0bc37fe640 1 --2- 172.21.15.67:0/3716733185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bb0004d40 0x7f0bb00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:08.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.424+0000 7f0bc37fe640 1 -- 172.21.15.67:0/3716733185 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc40d2530 con 0x7f0bb0004d40 2024-03-24T00:49:08.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.424+0000 7f0bc37fe640 1 -- 172.21.15.67:0/3716733185 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc40d2810 con 0x7f0bb0003f30 2024-03-24T00:49:08.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.424+0000 7f0bc37fe640 1 -- 172.21.15.67:0/3716733185 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb000ba10 con 0x7f0bc415e280 2024-03-24T00:49:08.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.424+0000 7f0bc9000640 1 --2- 172.21.15.67:0/3716733185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bb0004d40 0x7f0bb00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:08.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.424+0000 7f0bc3fff640 1 --2- 172.21.15.67:0/3716733185 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0bc415e280 0x7f0bb00039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:08.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.424+0000 7f0bbbfff640 1 --2- 172.21.15.67:0/3716733185 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0bb0003f30 0x7f0bb0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:08.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.424+0000 7f0bc9000640 1 -- 172.21.15.67:0/3716733185 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0bc415e280 msgr2=0x7f0bb00039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:08.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.424+0000 7f0bc9000640 1 --2- 172.21.15.67:0/3716733185 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0bc415e280 0x7f0bb00039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:08.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.424+0000 7f0bc9000640 1 -- 172.21.15.67:0/3716733185 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0bb0003f30 msgr2=0x7f0bb0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:08.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.424+0000 7f0bc9000640 1 --2- 172.21.15.67:0/3716733185 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0bb0003f30 0x7f0bb0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:08.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.424+0000 7f0bc9000640 1 -- 172.21.15.67:0/3716733185 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb0003520 con 0x7f0bb0004d40 2024-03-24T00:49:08.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.425+0000 7f0bc9000640 1 --2- 172.21.15.67:0/3716733185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bb0004d40 0x7f0bb00051c0 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc00a8b0 tx=0x7f0bb0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7177a2c084df49c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:08.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.425+0000 7f0bc37fe640 1 -- 172.21.15.67:0/3716733185 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bbc00a4a0 con 0x7f0bb0004d40 2024-03-24T00:49:08.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.425+0000 7f0bc37fe640 1 -- 172.21.15.67:0/3716733185 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bbc034650 con 0x7f0bb0004d40 2024-03-24T00:49:08.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.425+0000 7f0bc37fe640 1 -- 172.21.15.67:0/3716733185 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bbc034a40 con 0x7f0bb0004d40 2024-03-24T00:49:08.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.425+0000 7f0bcaa63640 1 -- 172.21.15.67:0/3716733185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bb0004d40 msgr2=0x7f0bb00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:08.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.425+0000 7f0bcaa63640 1 --2- 172.21.15.67:0/3716733185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bb0004d40 0x7f0bb00051c0 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc00a8b0 tx=0x7f0bb0005830 comp rx=0 tx=0).stop 2024-03-24T00:49:08.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.425+0000 7f0bc9000640 1 -- 172.21.15.67:0/3716733185 reap_dead start 2024-03-24T00:49:08.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.426+0000 7f0bcaa63640 1 -- 172.21.15.67:0/3716733185 shutdown_connections 2024-03-24T00:49:08.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.426+0000 7f0bcaa63640 1 -- 172.21.15.67:0/3716733185 >> 172.21.15.67:0/3716733185 conn(0x7f0bc4072da0 msgr2=0x7f0bc40d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:08.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.426+0000 7f0bcaa63640 1 -- 172.21.15.67:0/3716733185 shutdown_connections 2024-03-24T00:49:08.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.426+0000 7f0bcaa63640 1 -- 172.21.15.67:0/3716733185 wait complete. 2024-03-24T00:49:08.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.427+0000 7f0bcaa63640 1 Processor -- start 2024-03-24T00:49:08.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.428+0000 7f0bcaa63640 1 -- start start 2024-03-24T00:49:08.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.429+0000 7f0bcaa63640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bc41e9ba0 0x7f0bc414cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:08.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.429+0000 7f0bcaa63640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0bc414a0f0 0x7f0bc41e7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:08.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.429+0000 7f0bcaa63640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0bc41e7e50 0x7f0bc414b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:08.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.429+0000 7f0bcaa63640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc40d3910 con 0x7f0bc41e9ba0 2024-03-24T00:49:08.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.429+0000 7f0bcaa63640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc40b8310 con 0x7f0bc41e7e50 2024-03-24T00:49:08.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.429+0000 7f0bcaa63640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc40d3a90 con 0x7f0bc414a0f0 2024-03-24T00:49:08.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.429+0000 7f0bc3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bc41e9ba0 0x7f0bc414cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:08.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.429+0000 7f0bc3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bc41e9ba0 0x7f0bc414cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42438/0 (socket says 172.21.15.67:42438) 2024-03-24T00:49:08.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.429+0000 7f0bc3fff640 1 -- 172.21.15.67:0/925172904 learned_addr learned my addr 172.21.15.67:0/925172904 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:08.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.429+0000 7f0bbbfff640 1 --2- 172.21.15.67:0/925172904 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0bc414a0f0 0x7f0bc41e7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:08.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.429+0000 7f0bc9000640 1 --2- 172.21.15.67:0/925172904 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0bc41e7e50 0x7f0bc414b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:08.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.429+0000 7f0bc3fff640 1 -- 172.21.15.67:0/925172904 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0bc414a0f0 msgr2=0x7f0bc41e7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:08.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.429+0000 7f0bc3fff640 1 --2- 172.21.15.67:0/925172904 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0bc414a0f0 0x7f0bc41e7910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:08.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.429+0000 7f0bc3fff640 1 -- 172.21.15.67:0/925172904 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0bc41e7e50 msgr2=0x7f0bc414b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:08.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.429+0000 7f0bc3fff640 1 --2- 172.21.15.67:0/925172904 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0bc41e7e50 0x7f0bc414b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:08.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.429+0000 7f0bc3fff640 1 -- 172.21.15.67:0/925172904 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc415d520 con 0x7f0bc41e9ba0 2024-03-24T00:49:08.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.429+0000 7f0bc9000640 1 --2- 172.21.15.67:0/925172904 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0bc41e7e50 0x7f0bc414b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:08.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.429+0000 7f0bc3fff640 1 --2- 172.21.15.67:0/925172904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bc41e9ba0 0x7f0bc414cb90 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f0bb4011a70 tx=0x7f0bb4011f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:08.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.429+0000 7f0bc17fa640 1 -- 172.21.15.67:0/925172904 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb400fb50 con 0x7f0bc41e9ba0 2024-03-24T00:49:08.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.429+0000 7f0bc17fa640 1 -- 172.21.15.67:0/925172904 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bb400fcf0 con 0x7f0bc41e9ba0 2024-03-24T00:49:08.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.429+0000 7f0bcaa63640 1 -- 172.21.15.67:0/925172904 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bc414d0d0 con 0x7f0bc41e9ba0 2024-03-24T00:49:08.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.429+0000 7f0bc17fa640 1 -- 172.21.15.67:0/925172904 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb400fea0 con 0x7f0bc41e9ba0 2024-03-24T00:49:08.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.430+0000 7f0bcaa63640 1 -- 172.21.15.67:0/925172904 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bc414a890 con 0x7f0bc41e9ba0 2024-03-24T00:49:08.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.431+0000 7f0bc17fa640 1 -- 172.21.15.67:0/925172904 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0bb4012040 con 0x7f0bc41e9ba0 2024-03-24T00:49:08.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.431+0000 7f0bc17fa640 1 --2- 172.21.15.67:0/925172904 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0ba406bbd0 0x7f0ba406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:08.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.432+0000 7f0bc17fa640 1 -- 172.21.15.67:0/925172904 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0bb408b330 con 0x7f0bc41e9ba0 2024-03-24T00:49:08.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.432+0000 7f0bbbfff640 1 --2- 172.21.15.67:0/925172904 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0ba406bbd0 0x7f0ba406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:08.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.432+0000 7f0bbbfff640 1 --2- 172.21.15.67:0/925172904 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0ba406bbd0 0x7f0ba406e090 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f0bc41f73e0 tx=0x7f0bac008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:08.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.433+0000 7f0bcd649740 1 Processor -- start 2024-03-24T00:49:08.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.434+0000 7f0bcd649740 1 -- start start 2024-03-24T00:49:08.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.434+0000 7f0bcd649740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560b25ef6e50 0x560b25ec9df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:08.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.434+0000 7f0bcd649740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x560b25ef7230 0x560b25ef9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:08.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.434+0000 7f0bcd649740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560b25eca330 0x560b25efa120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:08.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.434+0000 7f0bcd649740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b25e343d0 con 0x560b25eca330 2024-03-24T00:49:08.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.434+0000 7f0bcd649740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b25ddcad0 con 0x560b25ef6e50 2024-03-24T00:49:08.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.434+0000 7f0bcd649740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x560b25e30470 con 0x560b25ef7230 2024-03-24T00:49:08.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.434+0000 7f0bc9000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560b25eca330 0x560b25efa120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:08.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.434+0000 7f0bc9000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560b25eca330 0x560b25efa120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42444/0 (socket says 172.21.15.67:42444) 2024-03-24T00:49:08.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.434+0000 7f0bc9000640 1 -- 172.21.15.67:0/3679748179 learned_addr learned my addr 172.21.15.67:0/3679748179 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:08.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.435+0000 7f0bc3fff640 1 --2- 172.21.15.67:0/3679748179 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560b25ef6e50 0x560b25ec9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:08.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.435+0000 7f0bbbfff640 1 --2- 172.21.15.67:0/3679748179 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x560b25ef7230 0x560b25ef9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:08.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.435+0000 7f0bc9000640 1 -- 172.21.15.67:0/3679748179 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x560b25ef7230 msgr2=0x560b25ef9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:08.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.435+0000 7f0bc9000640 1 --2- 172.21.15.67:0/3679748179 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x560b25ef7230 0x560b25ef9be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:08.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.435+0000 7f0bc9000640 1 -- 172.21.15.67:0/3679748179 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560b25ef6e50 msgr2=0x560b25ec9df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:08.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.435+0000 7f0bc9000640 1 --2- 172.21.15.67:0/3679748179 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560b25ef6e50 0x560b25ec9df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:08.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.435+0000 7f0bc9000640 1 -- 172.21.15.67:0/3679748179 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560b25c127d0 con 0x560b25eca330 2024-03-24T00:49:08.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.435+0000 7f0bc9000640 1 --2- 172.21.15.67:0/3679748179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560b25eca330 0x560b25efa120 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc00a9a0 tx=0x7f0bbc007000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0e2c2b52b31a9f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:08.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.435+0000 7f0bb9ffb640 1 -- 172.21.15.67:0/3679748179 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bbc00a520 con 0x560b25eca330 2024-03-24T00:49:08.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.435+0000 7f0bb9ffb640 1 -- 172.21.15.67:0/3679748179 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bbc045070 con 0x560b25eca330 2024-03-24T00:49:08.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.435+0000 7f0bb9ffb640 1 -- 172.21.15.67:0/3679748179 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bbc0019b0 con 0x560b25eca330 2024-03-24T00:49:08.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.436+0000 7f0bcd649740 1 -- 172.21.15.67:0/3679748179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560b25eca330 msgr2=0x560b25efa120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:08.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.436+0000 7f0bcd649740 1 --2- 172.21.15.67:0/3679748179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560b25eca330 0x560b25efa120 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc00a9a0 tx=0x7f0bbc007000 comp rx=0 tx=0).stop 2024-03-24T00:49:08.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.436+0000 7f0bcd649740 1 -- 172.21.15.67:0/3679748179 shutdown_connections 2024-03-24T00:49:08.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.436+0000 7f0bcd649740 1 --2- 172.21.15.67:0/3679748179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560b25eca330 0x560b25efa120 unknown :-1 s=CLOSED pgs=1754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:08.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.436+0000 7f0bcd649740 1 --2- 172.21.15.67:0/3679748179 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x560b25ef7230 0x560b25ef9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:08.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.436+0000 7f0bcd649740 1 --2- 172.21.15.67:0/3679748179 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560b25ef6e50 0x560b25ec9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:08.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.436+0000 7f0bcd649740 1 -- 172.21.15.67:0/3679748179 >> 172.21.15.67:0/3679748179 conn(0x560b25ecc060 msgr2=0x560b25f14be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:08.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.436+0000 7f0bcd649740 1 -- 172.21.15.67:0/3679748179 shutdown_connections 2024-03-24T00:49:08.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.436+0000 7f0bcd649740 1 -- 172.21.15.67:0/3679748179 wait complete. 2024-03-24T00:49:08.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.436+0000 7f0bcd649740 1 -- start start 2024-03-24T00:49:08.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.436+0000 7f0bcd649740 1 Processor -- start 2024-03-24T00:49:08.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.437+0000 7f0bcd649740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560b25ef6e50 0x560b25f12110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:08.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.437+0000 7f0bcd649740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560b25ef7230 0x560b25efcf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:08.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.437+0000 7f0bcd649740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x560b25efdbc0 0x560b25efd4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:08.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.437+0000 7f0bcd649740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b25ece700 con 0x560b25ef7230 2024-03-24T00:49:08.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.437+0000 7f0bcd649740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b25df0060 con 0x560b25ef6e50 2024-03-24T00:49:08.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.437+0000 7f0bcd649740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x560b25ed0f40 con 0x560b25efdbc0 2024-03-24T00:49:08.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.437+0000 7f0bcd649740 10 client.0 _ll_register_callbacks cb 0x560b25dd26a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:49:08.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.437+0000 7f0bbbfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560b25ef7230 0x560b25efcf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:08.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.437+0000 7f0bbbfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560b25ef7230 0x560b25efcf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42456/0 (socket says 172.21.15.67:42456) 2024-03-24T00:49:08.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.437+0000 7f0bbbfff640 1 -- 172.21.15.67:0/2411765515 learned_addr learned my addr 172.21.15.67:0/2411765515 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:08.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.437+0000 7f0bc3fff640 1 --2- 172.21.15.67:0/2411765515 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560b25ef6e50 0x560b25f12110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:08.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.437+0000 7f0bc9000640 1 --2- 172.21.15.67:0/2411765515 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x560b25efdbc0 0x560b25efd4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:08.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.438+0000 7f0bbbfff640 1 -- 172.21.15.67:0/2411765515 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x560b25efdbc0 msgr2=0x560b25efd4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:08.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.438+0000 7f0bbbfff640 1 --2- 172.21.15.67:0/2411765515 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x560b25efdbc0 0x560b25efd4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:08.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.438+0000 7f0bbbfff640 1 -- 172.21.15.67:0/2411765515 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560b25ef6e50 msgr2=0x560b25f12110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:08.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.438+0000 7f0bbbfff640 1 --2- 172.21.15.67:0/2411765515 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560b25ef6e50 0x560b25f12110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:08.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.438+0000 7f0bbbfff640 1 -- 172.21.15.67:0/2411765515 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560b25b483f0 con 0x560b25ef7230 2024-03-24T00:49:08.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.438+0000 7f0bc3fff640 1 --2- 172.21.15.67:0/2411765515 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560b25ef6e50 0x560b25f12110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:49:08.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.438+0000 7f0bbbfff640 1 --2- 172.21.15.67:0/2411765515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560b25ef7230 0x560b25efcf90 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f0bac016a10 tx=0x7f0bac004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b46c63419e2864f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:08.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.438+0000 7f0ba2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:49:08.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.438+0000 7f0ba2ffd640 1 -- 172.21.15.67:0/2411765515 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bac015b50 con 0x560b25ef7230 2024-03-24T00:49:08.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.438+0000 7f0bcd649740 1 -- 172.21.15.67:0/2411765515 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0bbc045070 con 0x560b25ef7230 2024-03-24T00:49:08.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.438+0000 7f0ba2ffd640 1 -- 172.21.15.67:0/2411765515 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(120..120 src has 1..120) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0bac015cf0 con 0x560b25ef7230 2024-03-24T00:49:08.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.438+0000 7f0ba2ffd640 1 -- 172.21.15.67:0/2411765515 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0bac024430 con 0x560b25ef7230 2024-03-24T00:49:08.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.438+0000 7f0bcd649740 10 client.9453 fetch_fsmap learned FSMap version 86 2024-03-24T00:49:08.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.438+0000 7f0bcd649740 1 -- 172.21.15.67:0/2411765515 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=86,monmap=2+}) v3 -- 0x560b25b080f0 con 0x560b25ef7230 2024-03-24T00:49:08.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.438+0000 7f0ba2ffd640 1 -- 172.21.15.67:0/2411765515 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bbc045070 con 0x560b25ef7230 2024-03-24T00:49:08.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.438+0000 7f0ba2ffd640 1 -- 172.21.15.67:0/2411765515 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 86) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f0bac0245d0 con 0x560b25ef7230 2024-03-24T00:49:08.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.438+0000 7f0bcd649740 10 client.9453 fetch_fsmap finished waiting for FSMap version 86 2024-03-24T00:49:08.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.438+0000 7f0bcd649740 10 client.9453 resolve_mds: resolved 8:0 to role '8:0' aka daemon mds.a 2024-03-24T00:49:08.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.438+0000 7f0bcd649740 20 client.9453 populate_metadata read hostname 'smithi067' 2024-03-24T00:49:08.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.439+0000 7f0bcd649740 1 --2- 172.21.15.67:0/2411765515 >> [v2:172.21.15.67:6833/312063805,v1:172.21.15.67:6835/312063805] conn(0x560b25f28350 0x560b25f28790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:08.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.439+0000 7f0bcd649740 4 client.9453 mds_command: new command op to 9219 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:49:08.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.439+0000 7f0bcd649740 1 -- 172.21.15.67:0/2411765515 --> [v2:172.21.15.67:6833/312063805,v1:172.21.15.67:6835/312063805] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560b25e41910 con 0x560b25f28350 2024-03-24T00:49:08.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.439+0000 7f0bc3fff640 1 --2- 172.21.15.67:0/2411765515 >> [v2:172.21.15.67:6833/312063805,v1:172.21.15.67:6835/312063805] conn(0x560b25f28350 0x560b25f28790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:08.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.439+0000 7f0ba2ffd640 10 client.9453 ms_handle_connect on v2:172.21.15.67:6833/312063805 2024-03-24T00:49:08.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.439+0000 7f0bc3fff640 1 --2- 172.21.15.67:0/2411765515 >> [v2:172.21.15.67:6833/312063805,v1:172.21.15.67:6835/312063805] conn(0x560b25f28350 0x560b25f28790 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=746d3a6d0c7a6a2d server_cookie=c6375fd6220f91ab in_seq=0 out_seq=0 2024-03-24T00:49:08.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.440+0000 7f0ba2ffd640 1 -- 172.21.15.67:0/2411765515 <== mds.0 v2:172.21.15.67:6833/312063805 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560b25e41910 con 0x560b25f28350 2024-03-24T00:49:08.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.440+0000 7f0ba2ffd640 10 client.9453 handle_command_reply: tid=0 2024-03-24T00:49:08.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.440+0000 7f0bcd649740 1 client.9453 shutdown 2024-03-24T00:49:08.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.440+0000 7f0bcd649740 1 -- 172.21.15.67:0/2411765515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560b25ef7230 msgr2=0x560b25efcf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:08.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.440+0000 7f0bcd649740 1 --2- 172.21.15.67:0/2411765515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560b25ef7230 0x560b25efcf90 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f0bac016a10 tx=0x7f0bac004730 comp rx=0 tx=0).stop 2024-03-24T00:49:08.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.440+0000 7f0bcd649740 1 -- 172.21.15.67:0/2411765515 shutdown_connections 2024-03-24T00:49:08.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.440+0000 7f0bcd649740 1 --2- 172.21.15.67:0/2411765515 >> [v2:172.21.15.67:6833/312063805,v1:172.21.15.67:6835/312063805] conn(0x560b25f28350 0x560b25f28790 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:08.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.440+0000 7f0bcd649740 1 --2- 172.21.15.67:0/2411765515 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x560b25efdbc0 0x560b25efd4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:08.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.440+0000 7f0bcd649740 1 --2- 172.21.15.67:0/2411765515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560b25ef7230 0x560b25efcf90 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:08.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.440+0000 7f0bcd649740 1 --2- 172.21.15.67:0/2411765515 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560b25ef6e50 0x560b25f12110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:08.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.440+0000 7f0bcd649740 1 -- 172.21.15.67:0/2411765515 >> 172.21.15.67:0/2411765515 conn(0x560b25eca330 msgr2=0x560b25ecde00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:08.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.440+0000 7f0ba2ffd640 0 client.9453 ms_handle_reset on v2:172.21.15.67:6833/312063805 2024-03-24T00:49:08.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.440+0000 7f0bcd649740 1 -- 172.21.15.67:0/2411765515 shutdown_connections 2024-03-24T00:49:08.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.440+0000 7f0bcd649740 1 -- 172.21.15.67:0/2411765515 wait complete. 2024-03-24T00:49:08.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.440+0000 7f0bcd649740 20 client.9453 trim_cache size 0 max 16384 2024-03-24T00:49:08.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.454+0000 7f0bcd649740 1 Processor -- start 2024-03-24T00:49:08.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.454+0000 7f0bcd649740 1 -- start start 2024-03-24T00:49:08.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.455+0000 7f0bcd649740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560b25ef6e50 0x560b25f14740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:08.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.455+0000 7f0bcd649740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x560b25ef7230 0x560b25f18700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:08.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.455+0000 7f0bcd649740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560b25ecdaf0 0x560b25f27280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:08.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.455+0000 7f0bcd649740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc42014f0 con 0x560b25ecdaf0 2024-03-24T00:49:08.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.455+0000 7f0bcd649740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc4200630 con 0x560b25ef6e50 2024-03-24T00:49:08.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.455+0000 7f0bcd649740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc41e6fe0 con 0x560b25ef7230 2024-03-24T00:49:08.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.455+0000 7f0bc9000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560b25ecdaf0 0x560b25f27280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:08.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.455+0000 7f0bc9000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560b25ecdaf0 0x560b25f27280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42460/0 (socket says 172.21.15.67:42460) 2024-03-24T00:49:08.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.455+0000 7f0bc9000640 1 -- 172.21.15.67:0/3241056619 learned_addr learned my addr 172.21.15.67:0/3241056619 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:08.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.455+0000 7f0bbbfff640 1 --2- 172.21.15.67:0/3241056619 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x560b25ef7230 0x560b25f18700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:08.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.455+0000 7f0bc3fff640 1 --2- 172.21.15.67:0/3241056619 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560b25ef6e50 0x560b25f14740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:08.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.455+0000 7f0bc9000640 1 -- 172.21.15.67:0/3241056619 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x560b25ef7230 msgr2=0x560b25f18700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:08.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.455+0000 7f0bc9000640 1 --2- 172.21.15.67:0/3241056619 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x560b25ef7230 0x560b25f18700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:08.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.455+0000 7f0bc9000640 1 -- 172.21.15.67:0/3241056619 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560b25ef6e50 msgr2=0x560b25f14740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:08.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.455+0000 7f0bc9000640 1 --2- 172.21.15.67:0/3241056619 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560b25ef6e50 0x560b25f14740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:08.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.455+0000 7f0bc9000640 1 -- 172.21.15.67:0/3241056619 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560b25ef51d0 con 0x560b25ecdaf0 2024-03-24T00:49:08.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.455+0000 7f0bc3fff640 1 --2- 172.21.15.67:0/3241056619 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560b25ef6e50 0x560b25f14740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:08.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.455+0000 7f0bc9000640 1 --2- 172.21.15.67:0/3241056619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560b25ecdaf0 0x560b25f27280 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc00a9d0 tx=0x7f0bbc02f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17e4e6d08a47c3ee server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:08.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.455+0000 7f0bb97fa640 1 -- 172.21.15.67:0/3241056619 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bbc00a520 con 0x560b25ecdaf0 2024-03-24T00:49:08.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.455+0000 7f0bb97fa640 1 -- 172.21.15.67:0/3241056619 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bbc045040 con 0x560b25ecdaf0 2024-03-24T00:49:08.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.455+0000 7f0bb97fa640 1 -- 172.21.15.67:0/3241056619 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bbc0019f0 con 0x560b25ecdaf0 2024-03-24T00:49:08.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.456+0000 7f0bcd649740 1 -- 172.21.15.67:0/3241056619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560b25ecdaf0 msgr2=0x560b25f27280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:08.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.456+0000 7f0bcd649740 1 --2- 172.21.15.67:0/3241056619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560b25ecdaf0 0x560b25f27280 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc00a9d0 tx=0x7f0bbc02f000 comp rx=0 tx=0).stop 2024-03-24T00:49:08.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.456+0000 7f0bcd649740 1 -- 172.21.15.67:0/3241056619 shutdown_connections 2024-03-24T00:49:08.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.456+0000 7f0bcd649740 1 --2- 172.21.15.67:0/3241056619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560b25ecdaf0 0x560b25f27280 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:08.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.456+0000 7f0bcd649740 1 --2- 172.21.15.67:0/3241056619 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x560b25ef7230 0x560b25f18700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:08.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.456+0000 7f0bcd649740 1 --2- 172.21.15.67:0/3241056619 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560b25ef6e50 0x560b25f14740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:08.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.456+0000 7f0bcd649740 1 -- 172.21.15.67:0/3241056619 >> 172.21.15.67:0/3241056619 conn(0x560b25eca330 msgr2=0x560b25f11f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:08.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.456+0000 7f0bcd649740 1 -- 172.21.15.67:0/3241056619 shutdown_connections 2024-03-24T00:49:08.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.456+0000 7f0bcd649740 1 -- 172.21.15.67:0/3241056619 wait complete. 2024-03-24T00:49:08.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.456+0000 7f0bcd649740 1 -- start start 2024-03-24T00:49:08.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.457+0000 7f0bcd649740 1 Processor -- start 2024-03-24T00:49:08.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.457+0000 7f0bcd649740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560b25ef6e50 0x560b25f830c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:08.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.457+0000 7f0bcd649740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560b25ef7230 0x560b25eeffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:08.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.457+0000 7f0bcd649740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x560b25ef9df0 0x560b25ef3370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:08.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.457+0000 7f0bcd649740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b25ef1090 con 0x560b25ef6e50 2024-03-24T00:49:08.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.457+0000 7f0bcd649740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc4200030 con 0x560b25ef7230 2024-03-24T00:49:08.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.457+0000 7f0bcd649740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x560b25ed0b70 con 0x560b25ef9df0 2024-03-24T00:49:08.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.457+0000 7f0bcd649740 10 client.0 _ll_register_callbacks cb 0x7f0bac01a070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:49:08.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.457+0000 7f0bc3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560b25ef6e50 0x560b25f830c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:08.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.457+0000 7f0bc3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560b25ef6e50 0x560b25f830c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42466/0 (socket says 172.21.15.67:42466) 2024-03-24T00:49:08.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.458+0000 7f0bc3fff640 1 -- 172.21.15.67:0/731496091 learned_addr learned my addr 172.21.15.67:0/731496091 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:08.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.458+0000 7f0bc9000640 1 --2- 172.21.15.67:0/731496091 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x560b25ef9df0 0x560b25ef3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:08.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.458+0000 7f0bbbfff640 1 --2- 172.21.15.67:0/731496091 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560b25ef7230 0x560b25eeffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:08.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.458+0000 7f0bc3fff640 1 -- 172.21.15.67:0/731496091 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x560b25ef9df0 msgr2=0x560b25ef3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:08.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.458+0000 7f0bc3fff640 1 --2- 172.21.15.67:0/731496091 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x560b25ef9df0 0x560b25ef3370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:08.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.458+0000 7f0bc3fff640 1 -- 172.21.15.67:0/731496091 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560b25ef7230 msgr2=0x560b25eeffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:08.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.458+0000 7f0bc3fff640 1 --2- 172.21.15.67:0/731496091 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560b25ef7230 0x560b25eeffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:08.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.458+0000 7f0bc3fff640 1 -- 172.21.15.67:0/731496091 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560b25f379e0 con 0x560b25ef6e50 2024-03-24T00:49:08.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.458+0000 7f0bc3fff640 1 --2- 172.21.15.67:0/731496091 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560b25ef6e50 0x560b25f830c0 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7f0bb4055aa0 tx=0x7f0bb405d7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65351de26fa73b08 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:08.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.458+0000 7f0ba2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:49:08.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.458+0000 7f0ba2ffd640 1 -- 172.21.15.67:0/731496091 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb4057940 con 0x560b25ef6e50 2024-03-24T00:49:08.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.458+0000 7f0ba2ffd640 1 -- 172.21.15.67:0/731496091 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(120..120 src has 1..120) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0bb408b040 con 0x560b25ef6e50 2024-03-24T00:49:08.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.458+0000 7f0bcd649740 1 -- 172.21.15.67:0/731496091 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0bbc045040 con 0x560b25ef6e50 2024-03-24T00:49:08.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.458+0000 7f0ba2ffd640 1 -- 172.21.15.67:0/731496091 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0bb4089320 con 0x560b25ef6e50 2024-03-24T00:49:08.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.458+0000 7f0bcd649740 10 client.9459 fetch_fsmap learned FSMap version 86 2024-03-24T00:49:08.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.458+0000 7f0bcd649740 1 -- 172.21.15.67:0/731496091 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=86,monmap=2+}) v3 -- 0x560b25f2f240 con 0x560b25ef6e50 2024-03-24T00:49:08.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.458+0000 7f0ba2ffd640 1 -- 172.21.15.67:0/731496091 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bbc045040 con 0x560b25ef6e50 2024-03-24T00:49:08.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.459+0000 7f0ba2ffd640 1 -- 172.21.15.67:0/731496091 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 86) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f0bb4025030 con 0x560b25ef6e50 2024-03-24T00:49:08.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.459+0000 7f0bcd649740 10 client.9459 fetch_fsmap finished waiting for FSMap version 86 2024-03-24T00:49:08.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.459+0000 7f0bcd649740 10 client.9459 resolve_mds: resolved 8:0 to role '8:0' aka daemon mds.a 2024-03-24T00:49:08.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.459+0000 7f0bcd649740 20 client.9459 populate_metadata read hostname 'smithi067' 2024-03-24T00:49:08.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.459+0000 7f0bcd649740 1 --2- 172.21.15.67:0/731496091 >> [v2:172.21.15.67:6833/312063805,v1:172.21.15.67:6835/312063805] conn(0x560b25ecab40 0x560b25f27e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:08.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.459+0000 7f0bcd649740 4 client.9459 mds_command: new command op to 9219 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:49:08.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.459+0000 7f0bcd649740 1 -- 172.21.15.67:0/731496091 --> [v2:172.21.15.67:6833/312063805,v1:172.21.15.67:6835/312063805] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x560b25e3bc60 con 0x560b25ecab40 2024-03-24T00:49:08.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.459+0000 7f0bbbfff640 1 --2- 172.21.15.67:0/731496091 >> [v2:172.21.15.67:6833/312063805,v1:172.21.15.67:6835/312063805] conn(0x560b25ecab40 0x560b25f27e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:08.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.459+0000 7f0ba2ffd640 10 client.9459 ms_handle_connect on v2:172.21.15.67:6833/312063805 2024-03-24T00:49:08.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.459+0000 7f0bbbfff640 1 --2- 172.21.15.67:0/731496091 >> [v2:172.21.15.67:6833/312063805,v1:172.21.15.67:6835/312063805] conn(0x560b25ecab40 0x560b25f27e70 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5d531c8c81d58df server_cookie=74aa5b5a65ff7793 in_seq=0 out_seq=0 2024-03-24T00:49:08.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.472+0000 7f0ba2ffd640 1 -- 172.21.15.67:0/731496091 <== mds.0 v2:172.21.15.67:6833/312063805 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x560b25e3bc60 con 0x560b25ecab40 2024-03-24T00:49:08.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.472+0000 7f0ba2ffd640 10 client.9459 handle_command_reply: tid=0 2024-03-24T00:49:08.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.472+0000 7f0bcd649740 1 client.9459 shutdown 2024-03-24T00:49:08.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.472+0000 7f0bcd649740 1 -- 172.21.15.67:0/731496091 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560b25ef6e50 msgr2=0x560b25f830c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:08.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.472+0000 7f0bcd649740 1 --2- 172.21.15.67:0/731496091 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560b25ef6e50 0x560b25f830c0 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7f0bb4055aa0 tx=0x7f0bb405d7a0 comp rx=0 tx=0).stop 2024-03-24T00:49:08.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.472+0000 7f0bcd649740 1 -- 172.21.15.67:0/731496091 shutdown_connections 2024-03-24T00:49:08.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.472+0000 7f0bcd649740 1 --2- 172.21.15.67:0/731496091 >> [v2:172.21.15.67:6833/312063805,v1:172.21.15.67:6835/312063805] conn(0x560b25ecab40 0x560b25f27e70 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:08.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.472+0000 7f0bcd649740 1 --2- 172.21.15.67:0/731496091 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x560b25ef9df0 0x560b25ef3370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:08.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.472+0000 7f0bcd649740 1 --2- 172.21.15.67:0/731496091 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560b25ef7230 0x560b25eeffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:08.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.472+0000 7f0ba2ffd640 0 client.9459 ms_handle_reset on v2:172.21.15.67:6833/312063805 2024-03-24T00:49:08.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.472+0000 7f0bcd649740 1 --2- 172.21.15.67:0/731496091 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560b25ef6e50 0x560b25f830c0 unknown :-1 s=CLOSED pgs=1757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:08.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.472+0000 7f0bcd649740 1 -- 172.21.15.67:0/731496091 >> 172.21.15.67:0/731496091 conn(0x560b25eca330 msgr2=0x560b25ef96e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:08.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.472+0000 7f0bcd649740 1 -- 172.21.15.67:0/731496091 shutdown_connections 2024-03-24T00:49:08.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.472+0000 7f0bcd649740 1 -- 172.21.15.67:0/731496091 wait complete. 2024-03-24T00:49:08.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.472+0000 7f0bcd649740 20 client.9459 trim_cache size 0 max 16384 2024-03-24T00:49:08.474 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:49:08.475 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "", 2024-03-24T00:49:08.475 INFO:teuthology.orchestra.run.smithi067.stdout: "return_code": 0 2024-03-24T00:49:08.475 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:49:08.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.473+0000 7f0bcaa63640 1 -- 172.21.15.67:0/925172904 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0ba406bbd0 msgr2=0x7f0ba406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:08.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.473+0000 7f0bcaa63640 1 --2- 172.21.15.67:0/925172904 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0ba406bbd0 0x7f0ba406e090 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f0bc41f73e0 tx=0x7f0bac008040 comp rx=0 tx=0).stop 2024-03-24T00:49:08.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.473+0000 7f0bcaa63640 1 -- 172.21.15.67:0/925172904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bc41e9ba0 msgr2=0x7f0bc414cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:08.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.473+0000 7f0bcaa63640 1 --2- 172.21.15.67:0/925172904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bc41e9ba0 0x7f0bc414cb90 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f0bb4011a70 tx=0x7f0bb4011f30 comp rx=0 tx=0).stop 2024-03-24T00:49:08.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.473+0000 7f0bcaa63640 1 -- 172.21.15.67:0/925172904 shutdown_connections 2024-03-24T00:49:08.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.473+0000 7f0bcaa63640 1 --2- 172.21.15.67:0/925172904 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f0ba406bbd0 0x7f0ba406e090 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:08.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.473+0000 7f0bcaa63640 1 --2- 172.21.15.67:0/925172904 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0bc41e7e50 0x7f0bc414b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:08.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.473+0000 7f0bcaa63640 1 --2- 172.21.15.67:0/925172904 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f0bc414a0f0 0x7f0bc41e7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:08.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.473+0000 7f0bcaa63640 1 --2- 172.21.15.67:0/925172904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bc41e9ba0 0x7f0bc414cb90 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:08.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.473+0000 7f0bcaa63640 1 -- 172.21.15.67:0/925172904 >> 172.21.15.67:0/925172904 conn(0x7f0bc4072da0 msgr2=0x7f0bc40b8ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:08.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.473+0000 7f0bcaa63640 1 -- 172.21.15.67:0/925172904 shutdown_connections 2024-03-24T00:49:08.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.473+0000 7f0bcaa63640 1 -- 172.21.15.67:0/925172904 wait complete. 2024-03-24T00:49:08.485 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T00:49:08.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.585+0000 7faa1bb3b640 1 Processor -- start 2024-03-24T00:49:08.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.585+0000 7faa1bb3b640 1 -- start start 2024-03-24T00:49:08.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.586+0000 7faa1bb3b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7faa1415dec0 0x7faa140ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:08.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.586+0000 7faa1bb3b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faa1415e2a0 0x7faa14162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:08.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.586+0000 7faa1bb3b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faa140d07e0 0x7faa140d0c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:08.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.586+0000 7faa1bb3b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa140d1190 con 0x7faa140d07e0 2024-03-24T00:49:08.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.586+0000 7faa1bb3b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa140cee30 con 0x7faa1415dec0 2024-03-24T00:49:08.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.586+0000 7faa1bb3b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa140d2120 con 0x7faa1415e2a0 2024-03-24T00:49:08.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.586+0000 7faa1a0d8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faa140d07e0 0x7faa140d0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:08.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.586+0000 7faa1a0d8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faa140d07e0 0x7faa140d0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42470/0 (socket says 172.21.15.67:42470) 2024-03-24T00:49:08.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.586+0000 7faa1a0d8640 1 -- 172.21.15.67:0/3940036002 learned_addr learned my addr 172.21.15.67:0/3940036002 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:08.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.586+0000 7faa190d6640 1 --2- 172.21.15.67:0/3940036002 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faa1415e2a0 0x7faa14162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:08.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.586+0000 7faa198d7640 1 --2- 172.21.15.67:0/3940036002 >> v2:172.21.15.72:3300/0 conn(0x7faa1415dec0 0x7faa140ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:08.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.586+0000 7faa1a0d8640 1 -- 172.21.15.67:0/3940036002 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faa1415e2a0 msgr2=0x7faa14162630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:08.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.586+0000 7faa1a0d8640 1 --2- 172.21.15.67:0/3940036002 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faa1415e2a0 0x7faa14162630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:08.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.586+0000 7faa1a0d8640 1 -- 172.21.15.67:0/3940036002 >> v2:172.21.15.72:3300/0 conn(0x7faa1415dec0 msgr2=0x7faa140ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:08.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.586+0000 7faa1a0d8640 1 --2- 172.21.15.67:0/3940036002 >> v2:172.21.15.72:3300/0 conn(0x7faa1415dec0 0x7faa140ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:08.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.586+0000 7faa1a0d8640 1 -- 172.21.15.67:0/3940036002 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa140cf640 con 0x7faa140d07e0 2024-03-24T00:49:08.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.586+0000 7faa198d7640 1 --2- 172.21.15.67:0/3940036002 >> v2:172.21.15.72:3300/0 conn(0x7faa1415dec0 0x7faa140ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:08.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.587+0000 7faa1a0d8640 1 --2- 172.21.15.67:0/3940036002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa140d07e0 0x7faa140d0c50 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7faa140d14a0 tx=0x7faa0c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a152ca13ff69c20 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:08.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.587+0000 7faa188d5640 1 -- 172.21.15.67:0/3940036002 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa0c02c9e0 con 0x7faa140d07e0 2024-03-24T00:49:08.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.587+0000 7faa188d5640 1 -- 172.21.15.67:0/3940036002 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa0c02cb80 con 0x7faa140d07e0 2024-03-24T00:49:08.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.587+0000 7faa188d5640 1 -- 172.21.15.67:0/3940036002 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa0c0343f0 con 0x7faa140d07e0 2024-03-24T00:49:08.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.588+0000 7faa1bb3b640 1 -- 172.21.15.67:0/3940036002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa140d07e0 msgr2=0x7faa140d0c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:08.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.588+0000 7faa1bb3b640 1 --2- 172.21.15.67:0/3940036002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa140d07e0 0x7faa140d0c50 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7faa140d14a0 tx=0x7faa0c02be40 comp rx=0 tx=0).stop 2024-03-24T00:49:08.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.588+0000 7faa1bb3b640 1 -- 172.21.15.67:0/3940036002 shutdown_connections 2024-03-24T00:49:08.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.588+0000 7faa1bb3b640 1 --2- 172.21.15.67:0/3940036002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa140d07e0 0x7faa140d0c50 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:08.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.588+0000 7faa1bb3b640 1 --2- 172.21.15.67:0/3940036002 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faa1415e2a0 0x7faa14162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:08.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.588+0000 7faa1bb3b640 1 --2- 172.21.15.67:0/3940036002 >> v2:172.21.15.72:3300/0 conn(0x7faa1415dec0 0x7faa140ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:08.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.588+0000 7faa1bb3b640 1 -- 172.21.15.67:0/3940036002 >> 172.21.15.67:0/3940036002 conn(0x7faa1406c810 msgr2=0x7faa1406cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:08.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.588+0000 7faa1bb3b640 1 -- 172.21.15.67:0/3940036002 shutdown_connections 2024-03-24T00:49:08.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.588+0000 7faa1bb3b640 1 -- 172.21.15.67:0/3940036002 wait complete. 2024-03-24T00:49:08.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.589+0000 7faa1bb3b640 1 Processor -- start 2024-03-24T00:49:08.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.589+0000 7faa1bb3b640 1 -- start start 2024-03-24T00:49:08.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.590+0000 7faa1bb3b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa140d07e0 0x7faa141511b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:08.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.590+0000 7faa1bb3b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faa1415dec0 0x7faa140caea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:08.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.590+0000 7faa1bb3b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa1415e2a0 0x7faa140d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:08.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.590+0000 7faa1bb3b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa140b5850 con 0x7faa140d07e0 2024-03-24T00:49:08.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.590+0000 7faa1bb3b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa140cf360 con 0x7faa1415e2a0 2024-03-24T00:49:08.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.590+0000 7faa1bb3b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa140b59d0 con 0x7faa1415dec0 2024-03-24T00:49:08.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.590+0000 7faa198d7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa140d07e0 0x7faa141511b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:08.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.590+0000 7faa198d7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa140d07e0 0x7faa141511b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42478/0 (socket says 172.21.15.67:42478) 2024-03-24T00:49:08.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.590+0000 7faa198d7640 1 -- 172.21.15.67:0/3712611169 learned_addr learned my addr 172.21.15.67:0/3712611169 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:08.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.590+0000 7faa198d7640 1 -- 172.21.15.67:0/3712611169 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faa1415dec0 msgr2=0x7faa140caea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:08.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.590+0000 7faa198d7640 1 --2- 172.21.15.67:0/3712611169 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faa1415dec0 0x7faa140caea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:08.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.590+0000 7faa198d7640 1 -- 172.21.15.67:0/3712611169 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa1415e2a0 msgr2=0x7faa140d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:08.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.590+0000 7faa1a0d8640 1 --2- 172.21.15.67:0/3712611169 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa1415e2a0 0x7faa140d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:08.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.590+0000 7faa198d7640 1 --2- 172.21.15.67:0/3712611169 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa1415e2a0 0x7faa140d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:08.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.590+0000 7faa198d7640 1 -- 172.21.15.67:0/3712611169 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa140cb8f0 con 0x7faa140d07e0 2024-03-24T00:49:08.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.590+0000 7faa1a0d8640 1 --2- 172.21.15.67:0/3712611169 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa1415e2a0 0x7faa140d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:08.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.590+0000 7faa198d7640 1 --2- 172.21.15.67:0/3712611169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa140d07e0 0x7faa141511b0 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7faa1000c820 tx=0x7faa1000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:08.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.590+0000 7faa0a7fc640 1 -- 172.21.15.67:0/3712611169 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa10015070 con 0x7faa140d07e0 2024-03-24T00:49:08.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.591+0000 7faa0a7fc640 1 -- 172.21.15.67:0/3712611169 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa10007a30 con 0x7faa140d07e0 2024-03-24T00:49:08.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.591+0000 7faa1bb3b640 1 -- 172.21.15.67:0/3712611169 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa140cbbd0 con 0x7faa140d07e0 2024-03-24T00:49:08.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.591+0000 7faa0a7fc640 1 -- 172.21.15.67:0/3712611169 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa10007e00 con 0x7faa140d07e0 2024-03-24T00:49:08.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.591+0000 7faa1bb3b640 1 -- 172.21.15.67:0/3712611169 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa1415b510 con 0x7faa140d07e0 2024-03-24T00:49:08.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.592+0000 7faa1bb3b640 1 -- 172.21.15.67:0/3712611169 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa140cc300 con 0x7faa140d07e0 2024-03-24T00:49:08.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.592+0000 7faa0a7fc640 1 -- 172.21.15.67:0/3712611169 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7faa10018070 con 0x7faa140d07e0 2024-03-24T00:49:08.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.593+0000 7faa0a7fc640 1 --2- 172.21.15.67:0/3712611169 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa9f806bbd0 0x7fa9f806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:08.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.593+0000 7faa0a7fc640 1 -- 172.21.15.67:0/3712611169 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6101+0+0 (secure 0 0 0) 0x7faa1008abf0 con 0x7faa140d07e0 2024-03-24T00:49:08.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.593+0000 7faa190d6640 1 --2- 172.21.15.67:0/3712611169 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa9f806bbd0 0x7fa9f806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:08.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.593+0000 7faa190d6640 1 --2- 172.21.15.67:0/3712611169 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa9f806bbd0 0x7fa9f806e090 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7faa140ce760 tx=0x7faa04008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:08.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.597+0000 7faa0a7fc640 1 -- 172.21.15.67:0/3712611169 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7faa10059550 con 0x7faa140d07e0 2024-03-24T00:49:08.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.820+0000 7faa1bb3b640 1 -- 172.21.15.67:0/3712611169 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7faa140cc0f0 con 0x7faa140d07e0 2024-03-24T00:49:08.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.821+0000 7faa0a7fc640 1 -- 172.21.15.67:0/3712611169 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v86) v1 ==== 53+0+73 (secure 0 0 0) 0x7faa140cc0f0 con 0x7faa140d07e0 2024-03-24T00:49:08.822 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-24T00:49:08.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.822+0000 7faa1bb3b640 1 -- 172.21.15.67:0/3712611169 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa9f806bbd0 msgr2=0x7fa9f806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:08.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.822+0000 7faa1bb3b640 1 --2- 172.21.15.67:0/3712611169 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa9f806bbd0 0x7fa9f806e090 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7faa140ce760 tx=0x7faa04008040 comp rx=0 tx=0).stop 2024-03-24T00:49:08.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.822+0000 7faa1bb3b640 1 -- 172.21.15.67:0/3712611169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa140d07e0 msgr2=0x7faa141511b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:08.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.822+0000 7faa1bb3b640 1 --2- 172.21.15.67:0/3712611169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa140d07e0 0x7faa141511b0 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7faa1000c820 tx=0x7faa1000cc20 comp rx=0 tx=0).stop 2024-03-24T00:49:08.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.822+0000 7faa1bb3b640 1 -- 172.21.15.67:0/3712611169 shutdown_connections 2024-03-24T00:49:08.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.822+0000 7faa1bb3b640 1 --2- 172.21.15.67:0/3712611169 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa9f806bbd0 0x7fa9f806e090 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:08.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.822+0000 7faa1bb3b640 1 --2- 172.21.15.67:0/3712611169 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa1415e2a0 0x7faa140d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:08.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.822+0000 7faa1bb3b640 1 --2- 172.21.15.67:0/3712611169 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faa1415dec0 0x7faa140caea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:08.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.822+0000 7faa1bb3b640 1 --2- 172.21.15.67:0/3712611169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa140d07e0 0x7faa141511b0 unknown :-1 s=CLOSED pgs=1759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:08.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.822+0000 7faa1bb3b640 1 -- 172.21.15.67:0/3712611169 >> 172.21.15.67:0/3712611169 conn(0x7faa1406c810 msgr2=0x7faa140b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:08.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.822+0000 7faa1bb3b640 1 -- 172.21.15.67:0/3712611169 shutdown_connections 2024-03-24T00:49:08.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:08.822+0000 7faa1bb3b640 1 -- 172.21.15.67:0/3712611169 wait complete. 2024-03-24T00:49:08.836 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-24T00:49:08.836 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T00:49:08.836 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-03-24T00:49:08.836 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-24T00:49:08.836 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T00:49:08.836 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-24T00:49:08.837 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-24T00:49:08.837 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T00:49:08.837 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:49:08.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:08.837 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-03-24T00:49:08.853 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T00:49:08.853 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T00:49:08.853 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T00:49:08.853 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:49:08.853 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-03-24T00:49:08.854 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:49:08.854 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:49:08.854 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-03-24T00:49:08.854 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-03-24T00:49:08.854 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:49:08.854 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-03-24T00:49:08.854 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-03-24T00:49:08.854 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:49:08.854 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-03-24T00:49:08.854 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T00:49:08.854 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-03-24T00:49:08.854 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-24T00:49:08.854 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 41964sec preferred_lft 41964sec 2024-03-24T00:49:08.854 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-03-24T00:49:08.855 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:49:08.855 INFO:teuthology.orchestra.run.smithi067.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T00:49:08.855 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 96:cf:26:c8:bf:62 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:49:08.855 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-24T00:49:08.855 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:49:08.855 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::7c6a:5fff:feaa:d41/64 scope link 2024-03-24T00:49:08.855 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:49:08.855 INFO:teuthology.orchestra.run.smithi067.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T00:49:08.855 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ea:22:4b:d5:48:3d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:49:08.855 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::e822:4bff:fed5:483d/64 scope link 2024-03-24T00:49:08.855 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:49:08.855 INFO:teuthology.orchestra.run.smithi067.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T00:49:08.855 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 96:cf:26:c8:bf:62 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:49:08.855 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::94cf:26ff:fec8:bf62/64 scope link 2024-03-24T00:49:08.856 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:49:08.856 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:08.856 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T00:49:08.919 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-03-24T00:49:08.920 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:08.920 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-03-24T00:49:08.976 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-03-24T00:49:08.976 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T00:49:08.976 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-24T00:49:08.976 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-24T00:49:08.976 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T00:49:08.977 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:08.977 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-03-24T00:49:08.977 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-03-24T00:49:08.977 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-24T00:49:08.977 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-24T00:49:08.977 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-24T00:49:08.977 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-03-24T00:49:09.072 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:09.072 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-03-24T00:49:09.086 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-24T00:49:09.086 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-24T00:49:09.086 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:49:09.086 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:09.086 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-03-24T00:49:09.153 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:49:09.153 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-24T00:49:09.208 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:49:09.208 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:49:09.208 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:49:09.263 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-24T00:49:09.263 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:09.263 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:49:09.346 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:49:09.346 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T00:49:09.346 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-03-24T00:49:09.354 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:49:09.354 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T00:49:09.354 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-03-24T00:49:09.354 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.72:3300,172.21.15.72:3301" 2024-03-24T00:49:09.355 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-24T00:49:09.359 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:49:09.359 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T00:49:09.359 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-03-24T00:49:09.359 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.72:3300,172.21.15.72:3301" 2024-03-24T00:49:09.360 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-24T00:49:09.360 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:6789,172.21.15.72:6789,172.21.15.72:6790" 2024-03-24T00:49:09.360 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.67:6789,172.21.15.72:6789,172.21.15.72:6790:/ 2024-03-24T00:49:09.360 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=de1d0178-808e-49ca-a986-e297e523b976" will pass to kernel 2024-03-24T00:49:09.361 INFO:teuthology.orchestra.run.smithi067.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-24T00:49:09.361 INFO:teuthology.orchestra.run.smithi067.stdout: You just mounted a file system that supports labels which does not 2024-03-24T00:49:09.361 INFO:teuthology.orchestra.run.smithi067.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-24T00:49:09.361 INFO:teuthology.orchestra.run.smithi067.stdout: applications will generate AVC messages and not be allowed access to 2024-03-24T00:49:09.361 INFO:teuthology.orchestra.run.smithi067.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-24T00:49:09.362 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-24T00:49:09.362 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:09.362 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:49:09.433 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:09.434 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T00:49:09.490 INFO:teuthology.orchestra.run.smithi067.stdout:client8392 2024-03-24T00:49:09.490 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:49:09.490 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:49:09.545 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:09.545 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir parent/unflushed) 2024-03-24T00:49:09.603 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:09.603 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=./parent/unflushed/jfile bs=1M count=8) 2024-03-24T00:49:09.681 INFO:teuthology.orchestra.run.smithi067.stderr:8+0 records in 2024-03-24T00:49:09.681 INFO:teuthology.orchestra.run.smithi067.stderr:8+0 records out 2024-03-24T00:49:09.681 INFO:teuthology.orchestra.run.smithi067.stderr:8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.0233881 s, 359 MB/s 2024-03-24T00:49:09.682 DEBUG:teuthology.orchestra.run.smithi067:> stdin-killer -- python3 -c ' 2024-03-24T00:49:09.682 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-03-24T00:49:09.682 DEBUG:teuthology.orchestra.run.smithi067:> import stat 2024-03-24T00:49:09.682 DEBUG:teuthology.orchestra.run.smithi067:> 2024-03-24T00:49:09.682 DEBUG:teuthology.orchestra.run.smithi067:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/unflushed").st_ino) 2024-03-24T00:49:09.683 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-03-24T00:49:09.787 INFO:teuthology.orchestra.run.smithi067.stdout:1099511627779 2024-03-24T00:49:09.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:49:09.796 DEBUG:teuthology.orchestra.run.smithi067:> stdin-killer -- python3 -c ' 2024-03-24T00:49:09.796 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-03-24T00:49:09.796 DEBUG:teuthology.orchestra.run.smithi067:> import stat 2024-03-24T00:49:09.796 DEBUG:teuthology.orchestra.run.smithi067:> 2024-03-24T00:49:09.796 DEBUG:teuthology.orchestra.run.smithi067:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/unflushed/jfile").st_ino) 2024-03-24T00:49:09.796 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-03-24T00:49:09.901 INFO:teuthology.orchestra.run.smithi067.stdout:1099511627780 2024-03-24T00:49:09.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:49:09.909 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:49:09.909 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:49:09.965 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:49:09.966 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:49:10.022 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:49:10.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.162+0000 7ff74ba30640 1 Processor -- start 2024-03-24T00:49:10.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.162+0000 7ff74ba30640 1 -- start start 2024-03-24T00:49:10.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.163+0000 7ff74ba30640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff74415dec0 0x7ff7440ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:10.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.163+0000 7ff74ba30640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff74415e2a0 0x7ff744162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:10.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.163+0000 7ff74ba30640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7440cf640 con 0x7ff7440d07e0 2024-03-24T00:49:10.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.163+0000 7ff74ba30640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7440cee30 con 0x7ff74415dec0 2024-03-24T00:49:10.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.163+0000 7ff74ba30640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7440d2120 con 0x7ff74415e2a0 2024-03-24T00:49:10.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.163+0000 7ff749fcd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff7440d07e0 0x7ff7440b5850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48720/0 (socket says 172.21.15.67:48720) 2024-03-24T00:49:10.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.163+0000 7ff749fcd640 1 -- 172.21.15.67:0/2262083857 learned_addr learned my addr 172.21.15.67:0/2262083857 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:49:10.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.164+0000 7ff737fff640 1 -- 172.21.15.67:0/2262083857 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1068975538 0 0) 0x7ff73c001490 con 0x7ff7440d07e0 2024-03-24T00:49:10.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.164+0000 7ff737fff640 1 -- 172.21.15.67:0/2262083857 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff730003520 con 0x7ff7440d07e0 2024-03-24T00:49:10.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.164+0000 7ff748fcb640 1 --2- 172.21.15.67:0/2262083857 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff74415e2a0 0x7ff744162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:10.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.164+0000 7ff7497cc640 1 --2- 172.21.15.67:0/2262083857 >> v2:172.21.15.72:3300/0 conn(0x7ff74415dec0 0x7ff7440ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:10.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.164+0000 7ff737fff640 1 -- 172.21.15.67:0/2262083857 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4046449509 0 0) 0x7ff730003520 con 0x7ff7440d07e0 2024-03-24T00:49:10.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.164+0000 7ff737fff640 1 -- 172.21.15.67:0/2262083857 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff74415e2a0 msgr2=0x7ff744162630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:10.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.164+0000 7ff737fff640 1 --2- 172.21.15.67:0/2262083857 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff74415e2a0 0x7ff744162630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:10.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.164+0000 7ff737fff640 1 -- 172.21.15.67:0/2262083857 >> v2:172.21.15.72:3300/0 conn(0x7ff74415dec0 msgr2=0x7ff7440ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:10.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.164+0000 7ff737fff640 1 --2- 172.21.15.67:0/2262083857 >> v2:172.21.15.72:3300/0 conn(0x7ff74415dec0 0x7ff7440ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:10.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.164+0000 7ff737fff640 1 -- 172.21.15.67:0/2262083857 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7440cf080 con 0x7ff7440d07e0 2024-03-24T00:49:10.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.164+0000 7ff737fff640 1 -- 172.21.15.67:0/2262083857 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7ff73c001bc0 con 0x7ff7440d07e0 2024-03-24T00:49:10.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.164+0000 7ff7497cc640 1 --2- 172.21.15.67:0/2262083857 >> v2:172.21.15.72:3300/0 conn(0x7ff74415dec0 0x7ff7440ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:49:10.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.164+0000 7ff737fff640 1 -- 172.21.15.67:0/2262083857 >> v1:172.21.15.67:6789/0 conn(0x7ff7440d07e0 legacy=0x7ff7440b5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:10.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.164+0000 7ff737fff640 1 --2- 172.21.15.67:0/2262083857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7300039b0 0x7ff730003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:10.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.164+0000 7ff737fff640 1 --2- 172.21.15.67:0/2262083857 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff730004800 0x7ff730004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:10.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.164+0000 7ff737fff640 1 --2- 172.21.15.67:0/2262083857 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff74415e2a0 0x7ff7300051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:10.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.164+0000 7ff737fff640 1 -- 172.21.15.67:0/2262083857 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7440cee30 con 0x7ff7300039b0 2024-03-24T00:49:10.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.164+0000 7ff737fff640 1 -- 172.21.15.67:0/2262083857 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7440d2120 con 0x7ff730004800 2024-03-24T00:49:10.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.164+0000 7ff737fff640 1 -- 172.21.15.67:0/2262083857 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff73000bb40 con 0x7ff74415e2a0 2024-03-24T00:49:10.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.164+0000 7ff7497cc640 1 --2- 172.21.15.67:0/2262083857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7300039b0 0x7ff730003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:10.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.164+0000 7ff748fcb640 1 --2- 172.21.15.67:0/2262083857 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff730004800 0x7ff730004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:10.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.164+0000 7ff749fcd640 1 --2- 172.21.15.67:0/2262083857 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff74415e2a0 0x7ff7300051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:10.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.164+0000 7ff7497cc640 1 -- 172.21.15.67:0/2262083857 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff74415e2a0 msgr2=0x7ff7300051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:10.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.164+0000 7ff7497cc640 1 --2- 172.21.15.67:0/2262083857 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff74415e2a0 0x7ff7300051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:10.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.164+0000 7ff7497cc640 1 -- 172.21.15.67:0/2262083857 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff730004800 msgr2=0x7ff730004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:10.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.164+0000 7ff7497cc640 1 --2- 172.21.15.67:0/2262083857 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff730004800 0x7ff730004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:10.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.164+0000 7ff7497cc640 1 -- 172.21.15.67:0/2262083857 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff730003520 con 0x7ff7300039b0 2024-03-24T00:49:10.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.166+0000 7ff7497cc640 1 --2- 172.21.15.67:0/2262083857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7300039b0 0x7ff730003df0 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7ff7380098a0 tx=0x7ff730005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=292568728943761a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:10.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.166+0000 7ff737fff640 1 -- 172.21.15.67:0/2262083857 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff73800f040 con 0x7ff7300039b0 2024-03-24T00:49:10.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.166+0000 7ff737fff640 1 -- 172.21.15.67:0/2262083857 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff738004230 con 0x7ff7300039b0 2024-03-24T00:49:10.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.166+0000 7ff737fff640 1 -- 172.21.15.67:0/2262083857 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff738003610 con 0x7ff7300039b0 2024-03-24T00:49:10.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.166+0000 7ff74ba30640 1 -- 172.21.15.67:0/2262083857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7300039b0 msgr2=0x7ff730003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:10.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.166+0000 7ff74ba30640 1 --2- 172.21.15.67:0/2262083857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7300039b0 0x7ff730003df0 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7ff7380098a0 tx=0x7ff730005830 comp rx=0 tx=0).stop 2024-03-24T00:49:10.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.166+0000 7ff749fcd640 1 -- 172.21.15.67:0/2262083857 reap_dead start 2024-03-24T00:49:10.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.167+0000 7ff74ba30640 1 -- 172.21.15.67:0/2262083857 shutdown_connections 2024-03-24T00:49:10.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.167+0000 7ff74ba30640 1 -- 172.21.15.67:0/2262083857 >> 172.21.15.67:0/2262083857 conn(0x7ff74406c810 msgr2=0x7ff74406cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:10.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.167+0000 7ff74ba30640 1 -- 172.21.15.67:0/2262083857 shutdown_connections 2024-03-24T00:49:10.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.167+0000 7ff74ba30640 1 -- 172.21.15.67:0/2262083857 wait complete. 2024-03-24T00:49:10.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.168+0000 7ff74ba30640 1 Processor -- start 2024-03-24T00:49:10.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.168+0000 7ff74ba30640 1 -- start start 2024-03-24T00:49:10.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.168+0000 7ff74ba30640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff7300039b0 0x7ff7440b9030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:10.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.168+0000 7ff74ba30640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff744152250 0x7ff744158690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:10.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.168+0000 7ff74ba30640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff744158bd0 0x7ff7440b70e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:10.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.168+0000 7ff74ba30640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7440d3740 con 0x7ff744158bd0 2024-03-24T00:49:10.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.168+0000 7ff74ba30640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7440d35c0 con 0x7ff7300039b0 2024-03-24T00:49:10.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.168+0000 7ff74ba30640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7440d38c0 con 0x7ff744152250 2024-03-24T00:49:10.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.169+0000 7ff749fcd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff744158bd0 0x7ff7440b70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:10.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.169+0000 7ff749fcd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff744158bd0 0x7ff7440b70e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42508/0 (socket says 172.21.15.67:42508) 2024-03-24T00:49:10.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.169+0000 7ff749fcd640 1 -- 172.21.15.67:0/572494750 learned_addr learned my addr 172.21.15.67:0/572494750 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:10.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.169+0000 7ff7497cc640 1 --2- 172.21.15.67:0/572494750 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff7300039b0 0x7ff7440b9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:10.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.169+0000 7ff748fcb640 1 --2- 172.21.15.67:0/572494750 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff744152250 0x7ff744158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:10.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.169+0000 7ff749fcd640 1 -- 172.21.15.67:0/572494750 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff744152250 msgr2=0x7ff744158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:10.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.169+0000 7ff749fcd640 1 --2- 172.21.15.67:0/572494750 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff744152250 0x7ff744158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:10.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.169+0000 7ff749fcd640 1 -- 172.21.15.67:0/572494750 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff7300039b0 msgr2=0x7ff7440b9030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:10.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.169+0000 7ff749fcd640 1 --2- 172.21.15.67:0/572494750 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff7300039b0 0x7ff7440b9030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:10.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.169+0000 7ff749fcd640 1 -- 172.21.15.67:0/572494750 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7440b8a60 con 0x7ff744158bd0 2024-03-24T00:49:10.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.169+0000 7ff7497cc640 1 --2- 172.21.15.67:0/572494750 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff7300039b0 0x7ff7440b9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:10.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.169+0000 7ff748fcb640 1 --2- 172.21.15.67:0/572494750 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff744152250 0x7ff744158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:10.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.169+0000 7ff749fcd640 1 --2- 172.21.15.67:0/572494750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff744158bd0 0x7ff7440b70e0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7ff73c001330 tx=0x7ff73c00aec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:10.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.169+0000 7ff735ffb640 1 -- 172.21.15.67:0/572494750 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff73c017020 con 0x7ff744158bd0 2024-03-24T00:49:10.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.169+0000 7ff735ffb640 1 -- 172.21.15.67:0/572494750 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff73c007980 con 0x7ff744158bd0 2024-03-24T00:49:10.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.169+0000 7ff735ffb640 1 -- 172.21.15.67:0/572494750 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff73c013630 con 0x7ff744158bd0 2024-03-24T00:49:10.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.169+0000 7ff74ba30640 1 -- 172.21.15.67:0/572494750 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7441578b0 con 0x7ff744158bd0 2024-03-24T00:49:10.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.170+0000 7ff74ba30640 1 -- 172.21.15.67:0/572494750 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff744157e70 con 0x7ff744158bd0 2024-03-24T00:49:10.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.171+0000 7ff74ba30640 1 -- 172.21.15.67:0/572494750 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff74404f400 con 0x7ff744158bd0 2024-03-24T00:49:10.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.171+0000 7ff735ffb640 1 -- 172.21.15.67:0/572494750 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff73c01a020 con 0x7ff744158bd0 2024-03-24T00:49:10.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.171+0000 7ff735ffb640 1 --2- 172.21.15.67:0/572494750 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff72806bbd0 0x7ff72806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:10.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.171+0000 7ff735ffb640 1 -- 172.21.15.67:0/572494750 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff73c08b490 con 0x7ff744158bd0 2024-03-24T00:49:10.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.171+0000 7ff7497cc640 1 --2- 172.21.15.67:0/572494750 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff72806bbd0 0x7ff72806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:10.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.172+0000 7ff7497cc640 1 --2- 172.21.15.67:0/572494750 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff72806bbd0 0x7ff72806e090 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7ff738035000 tx=0x7ff738039040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:10.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.176+0000 7ff735ffb640 1 -- 172.21.15.67:0/572494750 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff73c021020 con 0x7ff744158bd0 2024-03-24T00:49:10.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.456+0000 7ff74ba30640 1 -- 172.21.15.67:0/572494750 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff744158040 con 0x7ff744158bd0 2024-03-24T00:49:10.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.457+0000 7ff735ffb640 1 -- 172.21.15.67:0/572494750 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v120) v1 ==== 74+0+16892 (secure 0 0 0) 0x7ff7440cf640 con 0x7ff744158bd0 2024-03-24T00:49:10.458 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:10.458 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":120,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:48:55.987829+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:48:52.622356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-03-24T00:48:53.311452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:49:10.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.459+0000 7ff74ba30640 1 -- 172.21.15.67:0/572494750 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff72806bbd0 msgr2=0x7ff72806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:10.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.459+0000 7ff74ba30640 1 --2- 172.21.15.67:0/572494750 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff72806bbd0 0x7ff72806e090 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7ff738035000 tx=0x7ff738039040 comp rx=0 tx=0).stop 2024-03-24T00:49:10.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.459+0000 7ff74ba30640 1 -- 172.21.15.67:0/572494750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff744158bd0 msgr2=0x7ff7440b70e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:10.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.459+0000 7ff74ba30640 1 --2- 172.21.15.67:0/572494750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff744158bd0 0x7ff7440b70e0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7ff73c001330 tx=0x7ff73c00aec0 comp rx=0 tx=0).stop 2024-03-24T00:49:10.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.459+0000 7ff74ba30640 1 -- 172.21.15.67:0/572494750 shutdown_connections 2024-03-24T00:49:10.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.459+0000 7ff74ba30640 1 --2- 172.21.15.67:0/572494750 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff72806bbd0 0x7ff72806e090 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:10.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.459+0000 7ff74ba30640 1 --2- 172.21.15.67:0/572494750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff744158bd0 0x7ff7440b70e0 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:10.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.459+0000 7ff74ba30640 1 --2- 172.21.15.67:0/572494750 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff744152250 0x7ff744158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:10.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.459+0000 7ff74ba30640 1 --2- 172.21.15.67:0/572494750 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff7300039b0 0x7ff7440b9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:10.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.459+0000 7ff74ba30640 1 -- 172.21.15.67:0/572494750 >> 172.21.15.67:0/572494750 conn(0x7ff74406c810 msgr2=0x7ff74415f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:10.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.459+0000 7ff74ba30640 1 -- 172.21.15.67:0/572494750 shutdown_connections 2024-03-24T00:49:10.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.459+0000 7ff74ba30640 1 -- 172.21.15.67:0/572494750 wait complete. 2024-03-24T00:49:10.471 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:38:39.364950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:48:52.622356+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '119', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:48:53.311452+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '120', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:49:10.471 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:49:10.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.576+0000 7f3bfa35b640 1 Processor -- start 2024-03-24T00:49:10.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.576+0000 7f3bfa35b640 1 -- start start 2024-03-24T00:49:10.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.577+0000 7f3bfa35b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bf40d3ae0 0x7f3bf40b5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:10.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.577+0000 7f3bfa35b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3bf415e280 0x7f3bf40b5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:10.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.577+0000 7f3bfa35b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bf40d2250 con 0x7f3bf415dea0 2024-03-24T00:49:10.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.577+0000 7f3bfa35b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bf40d2530 con 0x7f3bf415e280 2024-03-24T00:49:10.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.577+0000 7f3bfa35b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf40d2810 con 0x7f3bf40d3ae0 2024-03-24T00:49:10.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.577+0000 7f3bf3fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3bf415dea0 0x7f3bf40d33d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48732/0 (socket says 172.21.15.67:48732) 2024-03-24T00:49:10.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.577+0000 7f3bf3fff640 1 -- 172.21.15.67:0/890372107 learned_addr learned my addr 172.21.15.67:0/890372107 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:49:10.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.577+0000 7f3bf37fe640 1 --2- 172.21.15.67:0/890372107 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bf40d3ae0 0x7f3bf40b5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:10.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.577+0000 7f3bf88f8640 1 --2- 172.21.15.67:0/890372107 >> v2:172.21.15.72:3300/0 conn(0x7f3bf415e280 0x7f3bf40b5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:10.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.577+0000 7f3bf2ffd640 1 -- 172.21.15.67:0/890372107 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1637434385 0 0) 0x7f3bf40d2250 con 0x7f3bf415dea0 2024-03-24T00:49:10.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.577+0000 7f3bf2ffd640 1 -- 172.21.15.67:0/890372107 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bdc003520 con 0x7f3bf415dea0 2024-03-24T00:49:10.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.577+0000 7f3bf2ffd640 1 -- 172.21.15.67:0/890372107 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3068425068 0 0) 0x7f3bdc003520 con 0x7f3bf415dea0 2024-03-24T00:49:10.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.577+0000 7f3bf2ffd640 1 -- 172.21.15.67:0/890372107 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bf40d3ae0 msgr2=0x7f3bf40b5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:10.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.577+0000 7f3bf2ffd640 1 --2- 172.21.15.67:0/890372107 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bf40d3ae0 0x7f3bf40b5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:10.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.577+0000 7f3bf2ffd640 1 -- 172.21.15.67:0/890372107 >> v2:172.21.15.72:3300/0 conn(0x7f3bf415e280 msgr2=0x7f3bf40b5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:10.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.578+0000 7f3bf2ffd640 1 --2- 172.21.15.67:0/890372107 >> v2:172.21.15.72:3300/0 conn(0x7f3bf415e280 0x7f3bf40b5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:10.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.578+0000 7f3bf2ffd640 1 -- 172.21.15.67:0/890372107 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf40d2af0 con 0x7f3bf415dea0 2024-03-24T00:49:10.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.578+0000 7f3bf2ffd640 1 -- 172.21.15.67:0/890372107 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f3be4001670 con 0x7f3bf415dea0 2024-03-24T00:49:10.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.578+0000 7f3bf2ffd640 1 -- 172.21.15.67:0/890372107 >> v1:172.21.15.67:6789/0 conn(0x7f3bf415dea0 legacy=0x7f3bf40d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:10.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.578+0000 7f3bf2ffd640 1 --2- 172.21.15.67:0/890372107 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bdc0039b0 0x7f3bdc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:10.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.578+0000 7f3bf2ffd640 1 --2- 172.21.15.67:0/890372107 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bdc004800 0x7f3bdc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:10.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.578+0000 7f3bf2ffd640 1 --2- 172.21.15.67:0/890372107 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bf40d3ae0 0x7f3bdc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:10.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.578+0000 7f3bf2ffd640 1 -- 172.21.15.67:0/890372107 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf40d2530 con 0x7f3bdc004800 2024-03-24T00:49:10.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.578+0000 7f3bf2ffd640 1 -- 172.21.15.67:0/890372107 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf40d2810 con 0x7f3bdc0039b0 2024-03-24T00:49:10.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.578+0000 7f3bf2ffd640 1 -- 172.21.15.67:0/890372107 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bdc00b9c0 con 0x7f3bf40d3ae0 2024-03-24T00:49:10.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.578+0000 7f3bf37fe640 1 --2- 172.21.15.67:0/890372107 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bdc004800 0x7f3bdc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:10.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.578+0000 7f3bf3fff640 1 --2- 172.21.15.67:0/890372107 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bdc0039b0 0x7f3bdc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:10.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.578+0000 7f3bf88f8640 1 --2- 172.21.15.67:0/890372107 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bf40d3ae0 0x7f3bdc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:10.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.578+0000 7f3bf37fe640 1 -- 172.21.15.67:0/890372107 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bf40d3ae0 msgr2=0x7f3bdc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:10.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.578+0000 7f3bf37fe640 1 --2- 172.21.15.67:0/890372107 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bf40d3ae0 0x7f3bdc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:10.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.578+0000 7f3bf37fe640 1 -- 172.21.15.67:0/890372107 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bdc0039b0 msgr2=0x7f3bdc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:10.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.578+0000 7f3bf37fe640 1 --2- 172.21.15.67:0/890372107 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bdc0039b0 0x7f3bdc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:10.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.578+0000 7f3bf37fe640 1 -- 172.21.15.67:0/890372107 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bdc003520 con 0x7f3bdc004800 2024-03-24T00:49:10.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.579+0000 7f3bf37fe640 1 --2- 172.21.15.67:0/890372107 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bdc004800 0x7f3bdc004c80 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f3be8009ac0 tx=0x7f3bdc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=751a4ea44fdaa96a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:10.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.579+0000 7f3bf2ffd640 1 -- 172.21.15.67:0/890372107 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be8039070 con 0x7f3bdc004800 2024-03-24T00:49:10.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.579+0000 7f3bf2ffd640 1 -- 172.21.15.67:0/890372107 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3be802c680 con 0x7f3bdc004800 2024-03-24T00:49:10.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.579+0000 7f3bf2ffd640 1 -- 172.21.15.67:0/890372107 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be8009690 con 0x7f3bdc004800 2024-03-24T00:49:10.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.579+0000 7f3bfa35b640 1 -- 172.21.15.67:0/890372107 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bdc004800 msgr2=0x7f3bdc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:10.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.579+0000 7f3bfa35b640 1 --2- 172.21.15.67:0/890372107 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bdc004800 0x7f3bdc004c80 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f3be8009ac0 tx=0x7f3bdc005830 comp rx=0 tx=0).stop 2024-03-24T00:49:10.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.579+0000 7f3bf88f8640 1 -- 172.21.15.67:0/890372107 reap_dead start 2024-03-24T00:49:10.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.579+0000 7f3bfa35b640 1 -- 172.21.15.67:0/890372107 shutdown_connections 2024-03-24T00:49:10.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.579+0000 7f3bfa35b640 1 -- 172.21.15.67:0/890372107 >> 172.21.15.67:0/890372107 conn(0x7f3bf4072da0 msgr2=0x7f3bf40d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:10.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.579+0000 7f3bfa35b640 1 -- 172.21.15.67:0/890372107 shutdown_connections 2024-03-24T00:49:10.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.580+0000 7f3bfa35b640 1 -- 172.21.15.67:0/890372107 wait complete. 2024-03-24T00:49:10.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.581+0000 7f3bfa35b640 1 Processor -- start 2024-03-24T00:49:10.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.582+0000 7f3bfa35b640 1 -- start start 2024-03-24T00:49:10.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.583+0000 7f3bfa35b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bdc004800 0x7f3bf4153340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:10.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.583+0000 7f3bfa35b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bf40cd280 0x7f3bf40ac360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:10.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.583+0000 7f3bfa35b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bf40d3fc0 0x7f3bf40d4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:10.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.583+0000 7f3bfa35b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf40b70f0 con 0x7f3bf40cd280 2024-03-24T00:49:10.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.583+0000 7f3bfa35b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf40b5cb0 con 0x7f3bf40d3fc0 2024-03-24T00:49:10.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.583+0000 7f3bfa35b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf40b7270 con 0x7f3bdc004800 2024-03-24T00:49:10.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.583+0000 7f3bf37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bf40cd280 0x7f3bf40ac360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:10.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.583+0000 7f3bf37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bf40cd280 0x7f3bf40ac360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42530/0 (socket says 172.21.15.67:42530) 2024-03-24T00:49:10.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.583+0000 7f3bf37fe640 1 -- 172.21.15.67:0/2206754798 learned_addr learned my addr 172.21.15.67:0/2206754798 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:10.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.583+0000 7f3bf88f8640 1 --2- 172.21.15.67:0/2206754798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bf40d3fc0 0x7f3bf40d4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:10.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.583+0000 7f3bf3fff640 1 --2- 172.21.15.67:0/2206754798 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bdc004800 0x7f3bf4153340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:10.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.583+0000 7f3bf37fe640 1 -- 172.21.15.67:0/2206754798 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bdc004800 msgr2=0x7f3bf4153340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:10.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.583+0000 7f3bf37fe640 1 --2- 172.21.15.67:0/2206754798 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bdc004800 0x7f3bf4153340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:10.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.583+0000 7f3bf37fe640 1 -- 172.21.15.67:0/2206754798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bf40d3fc0 msgr2=0x7f3bf40d4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:10.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.583+0000 7f3bf37fe640 1 --2- 172.21.15.67:0/2206754798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bf40d3fc0 0x7f3bf40d4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:10.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.583+0000 7f3bf37fe640 1 -- 172.21.15.67:0/2206754798 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf4153880 con 0x7f3bf40cd280 2024-03-24T00:49:10.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.583+0000 7f3bf88f8640 1 --2- 172.21.15.67:0/2206754798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bf40d3fc0 0x7f3bf40d4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:10.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.583+0000 7f3bf3fff640 1 --2- 172.21.15.67:0/2206754798 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bdc004800 0x7f3bf4153340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:10.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.583+0000 7f3bf37fe640 1 --2- 172.21.15.67:0/2206754798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bf40cd280 0x7f3bf40ac360 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f3be8009be0 tx=0x7f3be8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:10.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.583+0000 7f3bf0ff9640 1 -- 172.21.15.67:0/2206754798 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be8042020 con 0x7f3bf40cd280 2024-03-24T00:49:10.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.583+0000 7f3bf0ff9640 1 -- 172.21.15.67:0/2206754798 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3be803a040 con 0x7f3bf40cd280 2024-03-24T00:49:10.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.584+0000 7f3bfa35b640 1 -- 172.21.15.67:0/2206754798 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bf40acc00 con 0x7f3bf40cd280 2024-03-24T00:49:10.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.584+0000 7f3bf0ff9640 1 -- 172.21.15.67:0/2206754798 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be8039070 con 0x7f3bf40cd280 2024-03-24T00:49:10.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.584+0000 7f3bfa35b640 1 -- 172.21.15.67:0/2206754798 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bf40ad1c0 con 0x7f3bf40cd280 2024-03-24T00:49:10.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.585+0000 7f3bfa35b640 1 -- 172.21.15.67:0/2206754798 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bf404f400 con 0x7f3bf40cd280 2024-03-24T00:49:10.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.586+0000 7f3bf0ff9640 1 -- 172.21.15.67:0/2206754798 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3be8045020 con 0x7f3bf40cd280 2024-03-24T00:49:10.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.586+0000 7f3bf0ff9640 1 --2- 172.21.15.67:0/2206754798 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3be006bbd0 0x7f3be006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:10.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.586+0000 7f3bf0ff9640 1 -- 172.21.15.67:0/2206754798 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3be80ac6e0 con 0x7f3bf40cd280 2024-03-24T00:49:10.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.586+0000 7f3bf3fff640 1 --2- 172.21.15.67:0/2206754798 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3be006bbd0 0x7f3be006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:10.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.587+0000 7f3bf3fff640 1 --2- 172.21.15.67:0/2206754798 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3be006bbd0 0x7f3be006e090 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f3be40014f0 tx=0x7f3be4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:10.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.591+0000 7f3bf0ff9640 1 -- 172.21.15.67:0/2206754798 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3be804c020 con 0x7f3bf40cd280 2024-03-24T00:49:10.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.773+0000 7f3bfa35b640 1 -- 172.21.15.67:0/2206754798 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3bf40cee20 con 0x7f3bf40cd280 2024-03-24T00:49:10.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.774+0000 7f3bf0ff9640 1 -- 172.21.15.67:0/2206754798 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v120) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3be807b160 con 0x7f3bf40cd280 2024-03-24T00:49:10.775 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-24T00:49:10.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.776+0000 7f3bfa35b640 1 -- 172.21.15.67:0/2206754798 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3be006bbd0 msgr2=0x7f3be006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:10.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.776+0000 7f3bfa35b640 1 --2- 172.21.15.67:0/2206754798 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3be006bbd0 0x7f3be006e090 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f3be40014f0 tx=0x7f3be4009040 comp rx=0 tx=0).stop 2024-03-24T00:49:10.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.776+0000 7f3bfa35b640 1 -- 172.21.15.67:0/2206754798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bf40cd280 msgr2=0x7f3bf40ac360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:10.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.776+0000 7f3bfa35b640 1 --2- 172.21.15.67:0/2206754798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bf40cd280 0x7f3bf40ac360 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f3be8009be0 tx=0x7f3be8004730 comp rx=0 tx=0).stop 2024-03-24T00:49:10.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.776+0000 7f3bfa35b640 1 -- 172.21.15.67:0/2206754798 shutdown_connections 2024-03-24T00:49:10.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.776+0000 7f3bfa35b640 1 --2- 172.21.15.67:0/2206754798 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3be006bbd0 0x7f3be006e090 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:10.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.776+0000 7f3bfa35b640 1 --2- 172.21.15.67:0/2206754798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bf40d3fc0 0x7f3bf40d4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:10.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.776+0000 7f3bfa35b640 1 --2- 172.21.15.67:0/2206754798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bf40cd280 0x7f3bf40ac360 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:10.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.776+0000 7f3bfa35b640 1 --2- 172.21.15.67:0/2206754798 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3bdc004800 0x7f3bf4153340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:10.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.776+0000 7f3bfa35b640 1 -- 172.21.15.67:0/2206754798 >> 172.21.15.67:0/2206754798 conn(0x7f3bf4072da0 msgr2=0x7f3bf40b8090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:10.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.776+0000 7f3bfa35b640 1 -- 172.21.15.67:0/2206754798 shutdown_connections 2024-03-24T00:49:10.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.776+0000 7f3bfa35b640 1 -- 172.21.15.67:0/2206754798 wait complete. 2024-03-24T00:49:10.788 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:49:10.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.891+0000 7f08888c0640 1 Processor -- start 2024-03-24T00:49:10.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.891+0000 7f08888c0640 1 -- start start 2024-03-24T00:49:10.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.893+0000 7f08888c0640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f08840d3ae0 0x7f08840b5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:10.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.893+0000 7f08888c0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f088415e280 0x7f08840b5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:10.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.893+0000 7f08888c0640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08840d2250 con 0x7f088415dea0 2024-03-24T00:49:10.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.893+0000 7f08888c0640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08840d2530 con 0x7f088415e280 2024-03-24T00:49:10.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.893+0000 7f08888c0640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08840d2810 con 0x7f08840d3ae0 2024-03-24T00:49:10.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.893+0000 7f0882d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f088415e280 0x7f08840b5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:10.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.893+0000 7f088259c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f088415dea0 0x7f08840d33d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48744/0 (socket says 172.21.15.67:48744) 2024-03-24T00:49:10.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.893+0000 7f088259c640 1 -- 172.21.15.67:0/2992074673 learned_addr learned my addr 172.21.15.67:0/2992074673 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:49:10.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.893+0000 7f0881d9b640 1 --2- 172.21.15.67:0/2992074673 >> v2:172.21.15.72:3300/0 conn(0x7f08840d3ae0 0x7f08840b5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:10.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.893+0000 7f0882d9d640 1 -- 172.21.15.67:0/2992074673 >> v2:172.21.15.72:3300/0 conn(0x7f08840d3ae0 msgr2=0x7f08840b5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:10.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.893+0000 7f0882d9d640 1 --2- 172.21.15.67:0/2992074673 >> v2:172.21.15.72:3300/0 conn(0x7f08840d3ae0 0x7f08840b5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:10.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.893+0000 7f0882d9d640 1 -- 172.21.15.67:0/2992074673 >> v1:172.21.15.67:6789/0 conn(0x7f088415dea0 legacy=0x7f08840d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:10.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.893+0000 7f088159a640 1 -- 172.21.15.67:0/2992074673 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2967242962 0 0) 0x7f08840d2250 con 0x7f088415dea0 2024-03-24T00:49:10.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.893+0000 7f0882d9d640 1 -- 172.21.15.67:0/2992074673 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08840d2af0 con 0x7f088415e280 2024-03-24T00:49:10.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.893+0000 7f0881d9b640 1 --2- 172.21.15.67:0/2992074673 >> v2:172.21.15.72:3300/0 conn(0x7f08840d3ae0 0x7f08840b5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:10.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.894+0000 7f0882d9d640 1 --2- 172.21.15.67:0/2992074673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f088415e280 0x7f08840b5fa0 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f0874009d90 tx=0x7f087402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2152b971f6cf54c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:10.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.894+0000 7f088159a640 1 -- 172.21.15.67:0/2992074673 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0874039070 con 0x7f088415e280 2024-03-24T00:49:10.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.894+0000 7f088159a640 1 -- 172.21.15.67:0/2992074673 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f087402cbd0 con 0x7f088415e280 2024-03-24T00:49:10.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.894+0000 7f088159a640 1 -- 172.21.15.67:0/2992074673 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0874034440 con 0x7f088415e280 2024-03-24T00:49:10.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.895+0000 7f08888c0640 1 -- 172.21.15.67:0/2992074673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f088415e280 msgr2=0x7f08840b5fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:10.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.895+0000 7f08888c0640 1 --2- 172.21.15.67:0/2992074673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f088415e280 0x7f08840b5fa0 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f0874009d90 tx=0x7f087402be20 comp rx=0 tx=0).stop 2024-03-24T00:49:10.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.895+0000 7f08888c0640 1 -- 172.21.15.67:0/2992074673 shutdown_connections 2024-03-24T00:49:10.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.895+0000 7f08888c0640 1 --2- 172.21.15.67:0/2992074673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f088415e280 0x7f08840b5fa0 unknown :-1 s=CLOSED pgs=1767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:10.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.895+0000 7f08888c0640 1 --2- 172.21.15.67:0/2992074673 >> v2:172.21.15.72:3300/0 conn(0x7f08840d3ae0 0x7f08840b5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:10.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.895+0000 7f08888c0640 1 -- 172.21.15.67:0/2992074673 >> 172.21.15.67:0/2992074673 conn(0x7f0884072da0 msgr2=0x7f08840d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:10.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.895+0000 7f08888c0640 1 -- 172.21.15.67:0/2992074673 shutdown_connections 2024-03-24T00:49:10.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.895+0000 7f08888c0640 1 -- 172.21.15.67:0/2992074673 wait complete. 2024-03-24T00:49:10.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.896+0000 7f08888c0640 1 Processor -- start 2024-03-24T00:49:10.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.896+0000 7f08888c0640 1 -- start start 2024-03-24T00:49:10.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.897+0000 7f08888c0640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f08840d3ae0 0x7f088415f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:10.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.897+0000 7f08888c0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f088415dea0 0x7f088414ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:10.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.897+0000 7f08888c0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f088415e280 0x7f08841f2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:10.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.897+0000 7f08888c0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08840b7480 con 0x7f088415dea0 2024-03-24T00:49:10.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.897+0000 7f08888c0640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08840b7300 con 0x7f088415e280 2024-03-24T00:49:10.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.897+0000 7f08888c0640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08840b7600 con 0x7f08840d3ae0 2024-03-24T00:49:10.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.897+0000 7f0881d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f088415dea0 0x7f088414ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:10.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.897+0000 7f0881d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f088415dea0 0x7f088414ae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42560/0 (socket says 172.21.15.67:42560) 2024-03-24T00:49:10.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.897+0000 7f0881d9b640 1 -- 172.21.15.67:0/2221628597 learned_addr learned my addr 172.21.15.67:0/2221628597 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:10.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.897+0000 7f0882d9d640 1 --2- 172.21.15.67:0/2221628597 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f088415e280 0x7f08841f2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:10.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.897+0000 7f088259c640 1 --2- 172.21.15.67:0/2221628597 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f08840d3ae0 0x7f088415f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:10.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.897+0000 7f0881d9b640 1 -- 172.21.15.67:0/2221628597 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f08840d3ae0 msgr2=0x7f088415f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:10.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.897+0000 7f0881d9b640 1 --2- 172.21.15.67:0/2221628597 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f08840d3ae0 0x7f088415f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:10.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.897+0000 7f0881d9b640 1 -- 172.21.15.67:0/2221628597 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f088415e280 msgr2=0x7f08841f2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:10.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.897+0000 7f0881d9b640 1 --2- 172.21.15.67:0/2221628597 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f088415e280 0x7f08841f2ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:10.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.897+0000 7f0881d9b640 1 -- 172.21.15.67:0/2221628597 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0884158890 con 0x7f088415dea0 2024-03-24T00:49:10.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.898+0000 7f0881d9b640 1 --2- 172.21.15.67:0/2221628597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f088415dea0 0x7f088414ae90 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f086c00c7f0 tx=0x7f086c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:10.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.898+0000 7f087affd640 1 -- 172.21.15.67:0/2221628597 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086c015070 con 0x7f088415dea0 2024-03-24T00:49:10.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.898+0000 7f087affd640 1 -- 172.21.15.67:0/2221628597 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f086c007a30 con 0x7f088415dea0 2024-03-24T00:49:10.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.898+0000 7f08888c0640 1 -- 172.21.15.67:0/2221628597 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0884158b70 con 0x7f088415dea0 2024-03-24T00:49:10.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.898+0000 7f087affd640 1 -- 172.21.15.67:0/2221628597 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086c007e00 con 0x7f088415dea0 2024-03-24T00:49:10.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.898+0000 7f08888c0640 1 -- 172.21.15.67:0/2221628597 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f088415e730 con 0x7f088415dea0 2024-03-24T00:49:10.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.899+0000 7f08888c0640 1 -- 172.21.15.67:0/2221628597 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f088404f400 con 0x7f088415dea0 2024-03-24T00:49:10.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.899+0000 7f087affd640 1 -- 172.21.15.67:0/2221628597 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f086c018070 con 0x7f088415dea0 2024-03-24T00:49:10.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.900+0000 7f087affd640 1 --2- 172.21.15.67:0/2221628597 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f086406bbd0 0x7f086406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:10.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.900+0000 7f087affd640 1 -- 172.21.15.67:0/2221628597 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f086c08ab40 con 0x7f088415dea0 2024-03-24T00:49:10.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.900+0000 7f088259c640 1 --2- 172.21.15.67:0/2221628597 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f086406bbd0 0x7f086406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:10.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.901+0000 7f088259c640 1 --2- 172.21.15.67:0/2221628597 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f086406bbd0 0x7f086406e090 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f087c000f60 tx=0x7f087c007b50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:10.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:10.904+0000 7f087affd640 1 -- 172.21.15.67:0/2221628597 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f086c059530 con 0x7f088415dea0 2024-03-24T00:49:11.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.089+0000 7f08888c0640 1 -- 172.21.15.67:0/2221628597 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f08840d34e0 con 0x7f088415dea0 2024-03-24T00:49:11.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.089+0000 7f087affd640 1 -- 172.21.15.67:0/2221628597 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v120) v1 ==== 104+0+11 (secure 0 0 0) 0x7f086c058e30 con 0x7f088415dea0 2024-03-24T00:49:11.090 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-24T00:49:11.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.091+0000 7f08888c0640 1 -- 172.21.15.67:0/2221628597 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f086406bbd0 msgr2=0x7f086406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:11.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.091+0000 7f08888c0640 1 --2- 172.21.15.67:0/2221628597 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f086406bbd0 0x7f086406e090 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f087c000f60 tx=0x7f087c007b50 comp rx=0 tx=0).stop 2024-03-24T00:49:11.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.091+0000 7f08888c0640 1 -- 172.21.15.67:0/2221628597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f088415dea0 msgr2=0x7f088414ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:11.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.091+0000 7f08888c0640 1 --2- 172.21.15.67:0/2221628597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f088415dea0 0x7f088414ae90 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f086c00c7f0 tx=0x7f086c00cbf0 comp rx=0 tx=0).stop 2024-03-24T00:49:11.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.091+0000 7f08888c0640 1 -- 172.21.15.67:0/2221628597 shutdown_connections 2024-03-24T00:49:11.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.091+0000 7f08888c0640 1 --2- 172.21.15.67:0/2221628597 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f086406bbd0 0x7f086406e090 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:11.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.091+0000 7f08888c0640 1 --2- 172.21.15.67:0/2221628597 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f088415e280 0x7f08841f2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:11.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.091+0000 7f08888c0640 1 --2- 172.21.15.67:0/2221628597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f088415dea0 0x7f088414ae90 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:11.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.091+0000 7f08888c0640 1 --2- 172.21.15.67:0/2221628597 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f08840d3ae0 0x7f088415f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:11.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.091+0000 7f08888c0640 1 -- 172.21.15.67:0/2221628597 >> 172.21.15.67:0/2221628597 conn(0x7f0884072da0 msgr2=0x7f08840b79f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:11.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.092+0000 7f08888c0640 1 -- 172.21.15.67:0/2221628597 shutdown_connections 2024-03-24T00:49:11.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.092+0000 7f08888c0640 1 -- 172.21.15.67:0/2221628597 wait complete. 2024-03-24T00:49:11.103 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:49:11.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.206+0000 7f84af3da640 1 Processor -- start 2024-03-24T00:49:11.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.206+0000 7f84af3da640 1 -- start start 2024-03-24T00:49:11.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.207+0000 7f84af3da640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f84a8145540 0x7f84a8160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:11.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.207+0000 7f84af3da640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f84a815c2a0 0x7f84a80cef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:11.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.207+0000 7f84af3da640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f84a8071af0 0x7f84a8071f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:11.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.207+0000 7f84af3da640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84a80724a0 con 0x7f84a8071af0 2024-03-24T00:49:11.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.207+0000 7f84af3da640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84a806e620 con 0x7f84a815c2a0 2024-03-24T00:49:11.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.207+0000 7f84af3da640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84a806dcf0 con 0x7f84a8145540 2024-03-24T00:49:11.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.208+0000 7f84ad977640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f84a8071af0 0x7f84a8071f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:11.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.208+0000 7f84ad977640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f84a8071af0 0x7f84a8071f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42576/0 (socket says 172.21.15.67:42576) 2024-03-24T00:49:11.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.208+0000 7f84ad977640 1 -- 172.21.15.67:0/3332387935 learned_addr learned my addr 172.21.15.67:0/3332387935 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:11.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.208+0000 7f84ac975640 1 --2- 172.21.15.67:0/3332387935 >> v2:172.21.15.72:3300/0 conn(0x7f84a815c2a0 0x7f84a80cef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:11.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.208+0000 7f84ad176640 1 --2- 172.21.15.67:0/3332387935 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f84a8145540 0x7f84a8160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:11.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.208+0000 7f84ad977640 1 -- 172.21.15.67:0/3332387935 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f84a8145540 msgr2=0x7f84a8160420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:11.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.208+0000 7f84ad977640 1 --2- 172.21.15.67:0/3332387935 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f84a8145540 0x7f84a8160420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:11.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.208+0000 7f84ad977640 1 -- 172.21.15.67:0/3332387935 >> v2:172.21.15.72:3300/0 conn(0x7f84a815c2a0 msgr2=0x7f84a80cef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:11.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.208+0000 7f84ad977640 1 --2- 172.21.15.67:0/3332387935 >> v2:172.21.15.72:3300/0 conn(0x7f84a815c2a0 0x7f84a80cef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:11.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.208+0000 7f84ad977640 1 -- 172.21.15.67:0/3332387935 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a806dfd0 con 0x7f84a8071af0 2024-03-24T00:49:11.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.208+0000 7f84ac975640 1 --2- 172.21.15.67:0/3332387935 >> v2:172.21.15.72:3300/0 conn(0x7f84a815c2a0 0x7f84a80cef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:11.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.209+0000 7f84ad977640 1 --2- 172.21.15.67:0/3332387935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84a8071af0 0x7f84a8071f60 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f84a80b4ee0 tx=0x7f84a002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c8a55dc69ef02f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:11.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.209+0000 7f849bfff640 1 -- 172.21.15.67:0/3332387935 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a002ca70 con 0x7f84a8071af0 2024-03-24T00:49:11.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.209+0000 7f849bfff640 1 -- 172.21.15.67:0/3332387935 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84a002cc10 con 0x7f84a8071af0 2024-03-24T00:49:11.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.209+0000 7f849bfff640 1 -- 172.21.15.67:0/3332387935 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a00343f0 con 0x7f84a8071af0 2024-03-24T00:49:11.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.210+0000 7f84af3da640 1 -- 172.21.15.67:0/3332387935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84a8071af0 msgr2=0x7f84a8071f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:11.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.210+0000 7f84af3da640 1 --2- 172.21.15.67:0/3332387935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84a8071af0 0x7f84a8071f60 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f84a80b4ee0 tx=0x7f84a002bed0 comp rx=0 tx=0).stop 2024-03-24T00:49:11.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.210+0000 7f84af3da640 1 -- 172.21.15.67:0/3332387935 shutdown_connections 2024-03-24T00:49:11.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.210+0000 7f84af3da640 1 --2- 172.21.15.67:0/3332387935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84a8071af0 0x7f84a8071f60 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:11.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.210+0000 7f84af3da640 1 --2- 172.21.15.67:0/3332387935 >> v2:172.21.15.72:3300/0 conn(0x7f84a815c2a0 0x7f84a80cef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:11.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.210+0000 7f84af3da640 1 --2- 172.21.15.67:0/3332387935 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f84a8145540 0x7f84a8160420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:11.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.210+0000 7f84af3da640 1 -- 172.21.15.67:0/3332387935 >> 172.21.15.67:0/3332387935 conn(0x7f84a80cf9e0 msgr2=0x7f84a80cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:11.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.210+0000 7f84af3da640 1 -- 172.21.15.67:0/3332387935 shutdown_connections 2024-03-24T00:49:11.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.210+0000 7f84af3da640 1 -- 172.21.15.67:0/3332387935 wait complete. 2024-03-24T00:49:11.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.212+0000 7f84af3da640 1 Processor -- start 2024-03-24T00:49:11.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.212+0000 7f84af3da640 1 -- start start 2024-03-24T00:49:11.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.213+0000 7f84af3da640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f84a8071af0 0x7f84a80c7e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:11.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.213+0000 7f84af3da640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f84a8145540 0x7f84a815e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:11.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.213+0000 7f84af3da640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84a815c2a0 0x7f84a815ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:11.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.213+0000 7f84af3da640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a80cf480 con 0x7f84a815c2a0 2024-03-24T00:49:11.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.213+0000 7f84af3da640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a8145920 con 0x7f84a8071af0 2024-03-24T00:49:11.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.213+0000 7f84af3da640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84a80cf600 con 0x7f84a8145540 2024-03-24T00:49:11.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.213+0000 7f84ad977640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84a815c2a0 0x7f84a815ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:11.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.213+0000 7f84ad977640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84a815c2a0 0x7f84a815ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42578/0 (socket says 172.21.15.67:42578) 2024-03-24T00:49:11.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.213+0000 7f84ad977640 1 -- 172.21.15.67:0/4279629756 learned_addr learned my addr 172.21.15.67:0/4279629756 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:11.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.213+0000 7f84ac975640 1 --2- 172.21.15.67:0/4279629756 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f84a8145540 0x7f84a815e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:11.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.213+0000 7f84ad176640 1 --2- 172.21.15.67:0/4279629756 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f84a8071af0 0x7f84a80c7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:11.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.213+0000 7f84ad977640 1 -- 172.21.15.67:0/4279629756 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f84a8145540 msgr2=0x7f84a815e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:11.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.213+0000 7f84ad977640 1 --2- 172.21.15.67:0/4279629756 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f84a8145540 0x7f84a815e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:11.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.213+0000 7f84ad977640 1 -- 172.21.15.67:0/4279629756 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f84a8071af0 msgr2=0x7f84a80c7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:11.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.213+0000 7f84ad977640 1 --2- 172.21.15.67:0/4279629756 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f84a8071af0 0x7f84a80c7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:11.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.213+0000 7f84ad977640 1 -- 172.21.15.67:0/4279629756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a80c5ba0 con 0x7f84a815c2a0 2024-03-24T00:49:11.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.213+0000 7f84ac975640 1 --2- 172.21.15.67:0/4279629756 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f84a8145540 0x7f84a815e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:11.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.213+0000 7f84ad176640 1 --2- 172.21.15.67:0/4279629756 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f84a8071af0 0x7f84a80c7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:11.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.213+0000 7f84ad977640 1 --2- 172.21.15.67:0/4279629756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84a815c2a0 0x7f84a815ec00 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f84a0036040 tx=0x7f84a0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:11.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.214+0000 7f8499ffb640 1 -- 172.21.15.67:0/4279629756 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a0043020 con 0x7f84a815c2a0 2024-03-24T00:49:11.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.214+0000 7f8499ffb640 1 -- 172.21.15.67:0/4279629756 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84a003e450 con 0x7f84a815c2a0 2024-03-24T00:49:11.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.214+0000 7f84af3da640 1 -- 172.21.15.67:0/4279629756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84a80c5e80 con 0x7f84a815c2a0 2024-03-24T00:49:11.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.214+0000 7f8499ffb640 1 -- 172.21.15.67:0/4279629756 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a0009510 con 0x7f84a815c2a0 2024-03-24T00:49:11.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.214+0000 7f84af3da640 1 -- 172.21.15.67:0/4279629756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84a80c6440 con 0x7f84a815c2a0 2024-03-24T00:49:11.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.215+0000 7f84af3da640 1 -- 172.21.15.67:0/4279629756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84a804f400 con 0x7f84a815c2a0 2024-03-24T00:49:11.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.215+0000 7f8499ffb640 1 -- 172.21.15.67:0/4279629756 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f84a0046020 con 0x7f84a815c2a0 2024-03-24T00:49:11.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.216+0000 7f8499ffb640 1 --2- 172.21.15.67:0/4279629756 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f848c06bc00 0x7f848c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:11.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.216+0000 7f8499ffb640 1 -- 172.21.15.67:0/4279629756 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f84a00ad490 con 0x7f84a815c2a0 2024-03-24T00:49:11.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.216+0000 7f84ad176640 1 --2- 172.21.15.67:0/4279629756 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f848c06bc00 0x7f848c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:11.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.216+0000 7f84ad176640 1 --2- 172.21.15.67:0/4279629756 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f848c06bc00 0x7f848c06e0c0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f849c00b820 tx=0x7f849c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:11.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.220+0000 7f8499ffb640 1 -- 172.21.15.67:0/4279629756 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f84a004d030 con 0x7f84a815c2a0 2024-03-24T00:49:11.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.428+0000 7f84af3da640 1 -- 172.21.15.67:0/4279629756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f84a80d0740 con 0x7f84a815c2a0 2024-03-24T00:49:11.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.429+0000 7f8499ffb640 1 -- 172.21.15.67:0/4279629756 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v120) v1 ==== 100+0+11 (secure 0 0 0) 0x7f84a003c350 con 0x7f84a815c2a0 2024-03-24T00:49:11.430 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-03-24T00:49:11.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.431+0000 7f84af3da640 1 -- 172.21.15.67:0/4279629756 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f848c06bc00 msgr2=0x7f848c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:11.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.431+0000 7f84af3da640 1 --2- 172.21.15.67:0/4279629756 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f848c06bc00 0x7f848c06e0c0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f849c00b820 tx=0x7f849c008040 comp rx=0 tx=0).stop 2024-03-24T00:49:11.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.431+0000 7f84af3da640 1 -- 172.21.15.67:0/4279629756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84a815c2a0 msgr2=0x7f84a815ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:11.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.431+0000 7f84af3da640 1 --2- 172.21.15.67:0/4279629756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84a815c2a0 0x7f84a815ec00 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f84a0036040 tx=0x7f84a0004a40 comp rx=0 tx=0).stop 2024-03-24T00:49:11.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.431+0000 7f84af3da640 1 -- 172.21.15.67:0/4279629756 shutdown_connections 2024-03-24T00:49:11.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.431+0000 7f84af3da640 1 --2- 172.21.15.67:0/4279629756 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f848c06bc00 0x7f848c06e0c0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:11.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.431+0000 7f84af3da640 1 --2- 172.21.15.67:0/4279629756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84a815c2a0 0x7f84a815ec00 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:11.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.431+0000 7f84af3da640 1 --2- 172.21.15.67:0/4279629756 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f84a8145540 0x7f84a815e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:11.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.431+0000 7f84af3da640 1 --2- 172.21.15.67:0/4279629756 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f84a8071af0 0x7f84a80c7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:11.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.431+0000 7f84af3da640 1 -- 172.21.15.67:0/4279629756 >> 172.21.15.67:0/4279629756 conn(0x7f84a80cf9e0 msgr2=0x7f84a8140d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:11.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.431+0000 7f84af3da640 1 -- 172.21.15.67:0/4279629756 shutdown_connections 2024-03-24T00:49:11.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.431+0000 7f84af3da640 1 -- 172.21.15.67:0/4279629756 wait complete. 2024-03-24T00:49:11.443 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T00:49:11.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.544+0000 7f957215b640 1 Processor -- start 2024-03-24T00:49:11.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.544+0000 7f957215b640 1 -- start start 2024-03-24T00:49:11.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.545+0000 7f957215b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f956c06b750 0x7f956c0d01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:11.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.545+0000 7f957215b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f956c0ce970 con 0x7f956c15e200 2024-03-24T00:49:11.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.545+0000 7f957215b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f956c162120 con 0x7f956c0aeba0 2024-03-24T00:49:11.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.545+0000 7f957215b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f956c0ad700 con 0x7f956c06b750 2024-03-24T00:49:11.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.545+0000 7f956b7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f956c15e200 0x7f956c06b040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48752/0 (socket says 172.21.15.67:48752) 2024-03-24T00:49:11.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.545+0000 7f956b7fe640 1 -- 172.21.15.67:0/451451684 learned_addr learned my addr 172.21.15.67:0/451451684 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:49:11.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.545+0000 7f956affd640 1 --2- 172.21.15.67:0/451451684 >> v2:172.21.15.72:3300/0 conn(0x7f956c06b750 0x7f956c0d01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:11.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.546+0000 7f956a7fc640 1 -- 172.21.15.67:0/451451684 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2823271534 0 0) 0x7f956c0ce970 con 0x7f956c15e200 2024-03-24T00:49:11.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.546+0000 7f956a7fc640 1 -- 172.21.15.67:0/451451684 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9554003520 con 0x7f956c15e200 2024-03-24T00:49:11.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.546+0000 7f956affd640 1 -- 172.21.15.67:0/451451684 >> v1:172.21.15.72:6789/0 conn(0x7f956c0aeba0 legacy=0x7f956c0d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:11.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.546+0000 7f956a7fc640 1 -- 172.21.15.67:0/451451684 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2076520836 0 0) 0x7f956c162120 con 0x7f956c0aeba0 2024-03-24T00:49:11.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.546+0000 7f956affd640 1 -- 172.21.15.67:0/451451684 >> v1:172.21.15.67:6789/0 conn(0x7f956c15e200 legacy=0x7f956c06b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:11.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.546+0000 7f956affd640 1 -- 172.21.15.67:0/451451684 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f956c162400 con 0x7f956c06b750 2024-03-24T00:49:11.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.547+0000 7f956affd640 1 --2- 172.21.15.67:0/451451684 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f956c06b750 0x7f956c0d01a0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f95600097a0 tx=0x7f956002b6b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c7dd53e807ff4a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:11.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.547+0000 7f956a7fc640 1 -- 172.21.15.67:0/451451684 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9560037070 con 0x7f956c06b750 2024-03-24T00:49:11.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.547+0000 7f956a7fc640 1 -- 172.21.15.67:0/451451684 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9560030510 con 0x7f956c06b750 2024-03-24T00:49:11.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.547+0000 7f956a7fc640 1 -- 172.21.15.67:0/451451684 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f955c001670 con 0x7f956c06b750 2024-03-24T00:49:11.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.547+0000 7f957215b640 1 -- 172.21.15.67:0/451451684 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f956c06b750 msgr2=0x7f956c0d01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:11.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.547+0000 7f957215b640 1 --2- 172.21.15.67:0/451451684 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f956c06b750 0x7f956c0d01a0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f95600097a0 tx=0x7f956002b6b0 comp rx=0 tx=0).stop 2024-03-24T00:49:11.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.547+0000 7f957215b640 1 -- 172.21.15.67:0/451451684 shutdown_connections 2024-03-24T00:49:11.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.547+0000 7f957215b640 1 --2- 172.21.15.67:0/451451684 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f956c06b750 0x7f956c0d01a0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:11.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.547+0000 7f957215b640 1 -- 172.21.15.67:0/451451684 >> 172.21.15.67:0/451451684 conn(0x7f956c099a30 msgr2=0x7f956c099e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:11.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.547+0000 7f957215b640 1 -- 172.21.15.67:0/451451684 shutdown_connections 2024-03-24T00:49:11.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.547+0000 7f957215b640 1 -- 172.21.15.67:0/451451684 wait complete. 2024-03-24T00:49:11.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.549+0000 7f957215b640 1 Processor -- start 2024-03-24T00:49:11.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.549+0000 7f957215b640 1 -- start start 2024-03-24T00:49:11.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.549+0000 7f957215b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f956c06b750 0x7f956c0c5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:11.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.549+0000 7f957215b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f956c0aeba0 0x7f956c157db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:11.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.549+0000 7f957215b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f956c15e200 0x7f956c0b5890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:11.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.549+0000 7f957215b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f956c161bb0 con 0x7f956c0aeba0 2024-03-24T00:49:11.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.549+0000 7f957215b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f956c1629c0 con 0x7f956c15e200 2024-03-24T00:49:11.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.549+0000 7f957215b640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f956c161d30 con 0x7f956c06b750 2024-03-24T00:49:11.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.550+0000 7f956affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f956c0aeba0 0x7f956c157db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:11.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.550+0000 7f956affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f956c0aeba0 0x7f956c157db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42582/0 (socket says 172.21.15.67:42582) 2024-03-24T00:49:11.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.550+0000 7f956affd640 1 -- 172.21.15.67:0/834737642 learned_addr learned my addr 172.21.15.67:0/834737642 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:11.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.550+0000 7f956bfff640 1 --2- 172.21.15.67:0/834737642 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f956c15e200 0x7f956c0b5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:11.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.550+0000 7f956b7fe640 1 --2- 172.21.15.67:0/834737642 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f956c06b750 0x7f956c0c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:11.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.550+0000 7f956affd640 1 -- 172.21.15.67:0/834737642 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f956c06b750 msgr2=0x7f956c0c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:11.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.550+0000 7f956affd640 1 --2- 172.21.15.67:0/834737642 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f956c06b750 0x7f956c0c5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:11.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.550+0000 7f956affd640 1 -- 172.21.15.67:0/834737642 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f956c15e200 msgr2=0x7f956c0b5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:11.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.550+0000 7f956affd640 1 --2- 172.21.15.67:0/834737642 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f956c15e200 0x7f956c0b5890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:11.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.550+0000 7f956affd640 1 -- 172.21.15.67:0/834737642 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f956c0c45b0 con 0x7f956c0aeba0 2024-03-24T00:49:11.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.550+0000 7f956b7fe640 1 --2- 172.21.15.67:0/834737642 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f956c06b750 0x7f956c0c5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:11.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.550+0000 7f956affd640 1 --2- 172.21.15.67:0/834737642 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f956c0aeba0 0x7f956c157db0 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f95600097a0 tx=0x7f95600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:11.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.550+0000 7f9553fff640 1 -- 172.21.15.67:0/834737642 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9560042070 con 0x7f956c0aeba0 2024-03-24T00:49:11.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.550+0000 7f9553fff640 1 -- 172.21.15.67:0/834737642 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f956003e930 con 0x7f956c0aeba0 2024-03-24T00:49:11.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.550+0000 7f957215b640 1 -- 172.21.15.67:0/834737642 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f956c0c4850 con 0x7f956c0aeba0 2024-03-24T00:49:11.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.550+0000 7f9553fff640 1 -- 172.21.15.67:0/834737642 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9560037040 con 0x7f956c0aeba0 2024-03-24T00:49:11.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.550+0000 7f957215b640 1 -- 172.21.15.67:0/834737642 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f956c0c4e10 con 0x7f956c0aeba0 2024-03-24T00:49:11.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.552+0000 7f957215b640 1 -- 172.21.15.67:0/834737642 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f956c04f400 con 0x7f956c0aeba0 2024-03-24T00:49:11.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.552+0000 7f9553fff640 1 -- 172.21.15.67:0/834737642 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9560045070 con 0x7f956c0aeba0 2024-03-24T00:49:11.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.552+0000 7f9553fff640 1 --2- 172.21.15.67:0/834737642 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f955806bbd0 0x7f955806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:11.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.552+0000 7f9553fff640 1 -- 172.21.15.67:0/834737642 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f95600ac8d0 con 0x7f956c0aeba0 2024-03-24T00:49:11.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.553+0000 7f956b7fe640 1 --2- 172.21.15.67:0/834737642 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f955806bbd0 0x7f955806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:11.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.553+0000 7f956b7fe640 1 --2- 172.21.15.67:0/834737642 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f955806bbd0 0x7f955806e090 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f955c000c00 tx=0x7f955c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:11.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.557+0000 7f9553fff640 1 -- 172.21.15.67:0/834737642 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f956007b2c0 con 0x7f956c0aeba0 2024-03-24T00:49:11.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.850+0000 7f957215b640 1 -- 172.21.15.67:0/834737642 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f956c0ce020 con 0x7f956c0aeba0 2024-03-24T00:49:11.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.850+0000 7f9553fff640 1 -- 172.21.15.67:0/834737642 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v120) v1 ==== 80+0+0 (secure 0 0 0) 0x7f956007abc0 con 0x7f956c0aeba0 2024-03-24T00:49:11.851 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-03-24T00:49:11.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.852+0000 7f957215b640 1 -- 172.21.15.67:0/834737642 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f955806bbd0 msgr2=0x7f955806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:11.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.852+0000 7f957215b640 1 --2- 172.21.15.67:0/834737642 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f955806bbd0 0x7f955806e090 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f955c000c00 tx=0x7f955c009040 comp rx=0 tx=0).stop 2024-03-24T00:49:11.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.852+0000 7f957215b640 1 -- 172.21.15.67:0/834737642 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f956c0aeba0 msgr2=0x7f956c157db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:11.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.852+0000 7f957215b640 1 --2- 172.21.15.67:0/834737642 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f956c0aeba0 0x7f956c157db0 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f95600097a0 tx=0x7f95600046c0 comp rx=0 tx=0).stop 2024-03-24T00:49:11.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.852+0000 7f957215b640 1 -- 172.21.15.67:0/834737642 shutdown_connections 2024-03-24T00:49:11.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.852+0000 7f957215b640 1 --2- 172.21.15.67:0/834737642 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f955806bbd0 0x7f955806e090 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:11.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.852+0000 7f957215b640 1 --2- 172.21.15.67:0/834737642 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f956c15e200 0x7f956c0b5890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:11.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.852+0000 7f957215b640 1 --2- 172.21.15.67:0/834737642 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f956c0aeba0 0x7f956c157db0 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:11.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.852+0000 7f957215b640 1 --2- 172.21.15.67:0/834737642 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f956c06b750 0x7f956c0c5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:11.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.852+0000 7f957215b640 1 -- 172.21.15.67:0/834737642 >> 172.21.15.67:0/834737642 conn(0x7f956c099a30 msgr2=0x7f956c0cc810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:11.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.852+0000 7f957215b640 1 -- 172.21.15.67:0/834737642 shutdown_connections 2024-03-24T00:49:11.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:11.852+0000 7f957215b640 1 -- 172.21.15.67:0/834737642 wait complete. 2024-03-24T00:49:11.864 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-24T00:49:11.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:11.864 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-24T00:49:11.936 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-03-24T00:49:11.936 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:11.936 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-24T00:49:11.968 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:49:11.969 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:49:12.024 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056375,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15771 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T00:49:12.024 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-03-24T00:49:12.024 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:12.024 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-24T00:49:12.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:49:12.079 INFO:teuthology.orchestra.run.smithi067.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-24T00:49:12.079 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-24T00:49:12.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.221+0000 7f5abee44640 1 Processor -- start 2024-03-24T00:49:12.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.221+0000 7f5abee44640 1 -- start start 2024-03-24T00:49:12.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.222+0000 7f5abee44640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5ab815e0f0 0x7f5ab80b95b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:12.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.222+0000 7f5abee44640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ab80d2230 con 0x7f5ab80b6dc0 2024-03-24T00:49:12.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.222+0000 7f5abee44640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ab80d2510 con 0x7f5ab80b9af0 2024-03-24T00:49:12.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.222+0000 7f5abee44640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ab80d27f0 con 0x7f5ab815e0f0 2024-03-24T00:49:12.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.222+0000 7f5abd3e1640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5ab80b6dc0 0x7f5ab80b71c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48760/0 (socket says 172.21.15.67:48760) 2024-03-24T00:49:12.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.222+0000 7f5abd3e1640 1 -- 172.21.15.67:0/2383296484 learned_addr learned my addr 172.21.15.67:0/2383296484 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:49:12.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.222+0000 7f5abcbe0640 1 --2- 172.21.15.67:0/2383296484 >> v2:172.21.15.72:3300/0 conn(0x7f5ab815e0f0 0x7f5ab80b95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:12.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.222+0000 7f5ab77fe640 1 -- 172.21.15.67:0/2383296484 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3153323181 0 0) 0x7f5ab80d2230 con 0x7f5ab80b6dc0 2024-03-24T00:49:12.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.222+0000 7f5ab77fe640 1 -- 172.21.15.67:0/2383296484 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5aa8003520 con 0x7f5ab80b6dc0 2024-03-24T00:49:12.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.222+0000 7f5ab77fe640 1 -- 172.21.15.67:0/2383296484 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1308528138 0 0) 0x7f5ab80d2510 con 0x7f5ab80b9af0 2024-03-24T00:49:12.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.222+0000 7f5ab77fe640 1 -- 172.21.15.67:0/2383296484 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ab80d2230 con 0x7f5ab80b9af0 2024-03-24T00:49:12.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.222+0000 7f5abcbe0640 1 -- 172.21.15.67:0/2383296484 >> v1:172.21.15.72:6789/0 conn(0x7f5ab80b9af0 legacy=0x7f5ab80d32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:12.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.222+0000 7f5ab77fe640 1 -- 172.21.15.67:0/2383296484 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2607151107 0 0) 0x7f5aa8003520 con 0x7f5ab80b6dc0 2024-03-24T00:49:12.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.222+0000 7f5abcbe0640 1 -- 172.21.15.67:0/2383296484 >> v1:172.21.15.67:6789/0 conn(0x7f5ab80b6dc0 legacy=0x7f5ab80b71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:12.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.222+0000 7f5abcbe0640 1 -- 172.21.15.67:0/2383296484 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab80d2ad0 con 0x7f5ab815e0f0 2024-03-24T00:49:12.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.223+0000 7f5abcbe0640 1 --2- 172.21.15.67:0/2383296484 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ab815e0f0 0x7f5ab80b95b0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f5aac0097a0 tx=0x7f5aac02b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d888c524946d53c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:12.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.223+0000 7f5ab77fe640 1 -- 172.21.15.67:0/2383296484 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aac038070 con 0x7f5ab815e0f0 2024-03-24T00:49:12.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.223+0000 7f5ab77fe640 1 -- 172.21.15.67:0/2383296484 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5aac031500 con 0x7f5ab815e0f0 2024-03-24T00:49:12.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.223+0000 7f5ab77fe640 1 -- 172.21.15.67:0/2383296484 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab00019f0 con 0x7f5ab815e0f0 2024-03-24T00:49:12.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.224+0000 7f5abee44640 1 -- 172.21.15.67:0/2383296484 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ab815e0f0 msgr2=0x7f5ab80b95b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:12.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.224+0000 7f5abee44640 1 --2- 172.21.15.67:0/2383296484 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ab815e0f0 0x7f5ab80b95b0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f5aac0097a0 tx=0x7f5aac02b820 comp rx=0 tx=0).stop 2024-03-24T00:49:12.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.224+0000 7f5abee44640 1 -- 172.21.15.67:0/2383296484 shutdown_connections 2024-03-24T00:49:12.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.224+0000 7f5abee44640 1 --2- 172.21.15.67:0/2383296484 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ab815e0f0 0x7f5ab80b95b0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:12.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.224+0000 7f5abee44640 1 -- 172.21.15.67:0/2383296484 >> 172.21.15.67:0/2383296484 conn(0x7f5ab8072b60 msgr2=0x7f5ab8072f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:12.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.224+0000 7f5abee44640 1 -- 172.21.15.67:0/2383296484 shutdown_connections 2024-03-24T00:49:12.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.224+0000 7f5abee44640 1 -- 172.21.15.67:0/2383296484 wait complete. 2024-03-24T00:49:12.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.225+0000 7f5abee44640 1 Processor -- start 2024-03-24T00:49:12.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.225+0000 7f5abee44640 1 -- start start 2024-03-24T00:49:12.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.226+0000 7f5abee44640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ab80b6dc0 0x7f5ab80aba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:12.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.226+0000 7f5abee44640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ab80b9af0 0x7f5ab80b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:12.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.226+0000 7f5abee44640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5ab815e0f0 0x7f5ab80afa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:12.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.226+0000 7f5abee44640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab80b3350 con 0x7f5ab80b9af0 2024-03-24T00:49:12.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.226+0000 7f5abee44640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab80b31d0 con 0x7f5ab80b6dc0 2024-03-24T00:49:12.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.226+0000 7f5abee44640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab80d39e0 con 0x7f5ab815e0f0 2024-03-24T00:49:12.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.226+0000 7f5ab7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ab80b9af0 0x7f5ab80b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:12.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.226+0000 7f5ab7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ab80b9af0 0x7f5ab80b2070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42592/0 (socket says 172.21.15.67:42592) 2024-03-24T00:49:12.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.226+0000 7f5ab7fff640 1 -- 172.21.15.67:0/789095679 learned_addr learned my addr 172.21.15.67:0/789095679 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:12.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.226+0000 7f5abcbe0640 1 --2- 172.21.15.67:0/789095679 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ab80b6dc0 0x7f5ab80aba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:12.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.226+0000 7f5abd3e1640 1 --2- 172.21.15.67:0/789095679 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5ab815e0f0 0x7f5ab80afa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:12.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.226+0000 7f5ab7fff640 1 -- 172.21.15.67:0/789095679 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5ab815e0f0 msgr2=0x7f5ab80afa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:12.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.226+0000 7f5ab7fff640 1 --2- 172.21.15.67:0/789095679 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5ab815e0f0 0x7f5ab80afa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:12.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.226+0000 7f5ab7fff640 1 -- 172.21.15.67:0/789095679 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ab80b6dc0 msgr2=0x7f5ab80aba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:12.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.226+0000 7f5ab7fff640 1 --2- 172.21.15.67:0/789095679 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ab80b6dc0 0x7f5ab80aba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:12.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.226+0000 7f5ab7fff640 1 -- 172.21.15.67:0/789095679 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab80ac300 con 0x7f5ab80b9af0 2024-03-24T00:49:12.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.226+0000 7f5ab7fff640 1 --2- 172.21.15.67:0/789095679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ab80b9af0 0x7f5ab80b2070 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f5aa400c500 tx=0x7f5aa400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:12.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.227+0000 7f5ab57fa640 1 -- 172.21.15.67:0/789095679 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa40052d0 con 0x7f5ab80b9af0 2024-03-24T00:49:12.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.227+0000 7f5ab57fa640 1 -- 172.21.15.67:0/789095679 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5aa4005470 con 0x7f5ab80b9af0 2024-03-24T00:49:12.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.227+0000 7f5ab57fa640 1 -- 172.21.15.67:0/789095679 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa4003880 con 0x7f5ab80b9af0 2024-03-24T00:49:12.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.227+0000 7f5abee44640 1 -- 172.21.15.67:0/789095679 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ab80ac5e0 con 0x7f5ab80b9af0 2024-03-24T00:49:12.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.227+0000 7f5abee44640 1 -- 172.21.15.67:0/789095679 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ab80acba0 con 0x7f5ab80b9af0 2024-03-24T00:49:12.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.228+0000 7f5abee44640 1 -- 172.21.15.67:0/789095679 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ab80b62f0 con 0x7f5ab80b9af0 2024-03-24T00:49:12.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.228+0000 7f5ab57fa640 1 -- 172.21.15.67:0/789095679 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5aa4003a20 con 0x7f5ab80b9af0 2024-03-24T00:49:12.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.229+0000 7f5ab57fa640 1 --2- 172.21.15.67:0/789095679 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5a9c06bbd0 0x7f5a9c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:12.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.229+0000 7f5ab57fa640 1 -- 172.21.15.67:0/789095679 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5aa408a790 con 0x7f5ab80b9af0 2024-03-24T00:49:12.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.229+0000 7f5abcbe0640 1 --2- 172.21.15.67:0/789095679 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5a9c06bbd0 0x7f5a9c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:12.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.229+0000 7f5abcbe0640 1 --2- 172.21.15.67:0/789095679 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5a9c06bbd0 0x7f5a9c06e090 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f5aac035000 tx=0x7f5aac008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:12.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.233+0000 7f5ab57fa640 1 -- 172.21.15.67:0/789095679 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5ab80b62f0 con 0x7f5ab80b9af0 2024-03-24T00:49:12.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.454+0000 7f5abee44640 1 -- 172.21.15.67:0/789095679 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f5ab804f400 con 0x7f5ab80b9af0 2024-03-24T00:49:12.456 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-24T00:49:12.455+0000 7fbfcfa11640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T00:49:12.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.465+0000 7f5ab57fa640 1 -- 172.21.15.67:0/789095679 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v87) v1 ==== 161+0+0 (secure 0 0 0) 0x7f5aa4058b70 con 0x7f5ab80b9af0 2024-03-24T00:49:12.466 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-24T00:49:12.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.467+0000 7f5abee44640 1 -- 172.21.15.67:0/789095679 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5a9c06bbd0 msgr2=0x7f5a9c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:12.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.467+0000 7f5abee44640 1 --2- 172.21.15.67:0/789095679 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5a9c06bbd0 0x7f5a9c06e090 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f5aac035000 tx=0x7f5aac008030 comp rx=0 tx=0).stop 2024-03-24T00:49:12.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.467+0000 7f5abee44640 1 -- 172.21.15.67:0/789095679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ab80b9af0 msgr2=0x7f5ab80b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:12.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.467+0000 7f5abee44640 1 --2- 172.21.15.67:0/789095679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ab80b9af0 0x7f5ab80b2070 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f5aa400c500 tx=0x7f5aa400c9c0 comp rx=0 tx=0).stop 2024-03-24T00:49:12.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.467+0000 7f5abee44640 1 -- 172.21.15.67:0/789095679 shutdown_connections 2024-03-24T00:49:12.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.467+0000 7f5abee44640 1 --2- 172.21.15.67:0/789095679 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5a9c06bbd0 0x7f5a9c06e090 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:12.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.467+0000 7f5abee44640 1 --2- 172.21.15.67:0/789095679 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5ab815e0f0 0x7f5ab80afa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:12.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.467+0000 7f5abee44640 1 --2- 172.21.15.67:0/789095679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ab80b9af0 0x7f5ab80b2070 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:12.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.467+0000 7f5abee44640 1 --2- 172.21.15.67:0/789095679 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ab80b6dc0 0x7f5ab80aba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:12.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.467+0000 7f5abee44640 1 -- 172.21.15.67:0/789095679 >> 172.21.15.67:0/789095679 conn(0x7f5ab8072b60 msgr2=0x7f5ab80b3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:12.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.467+0000 7f5abee44640 1 -- 172.21.15.67:0/789095679 shutdown_connections 2024-03-24T00:49:12.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.468+0000 7f5abee44640 1 -- 172.21.15.67:0/789095679 wait complete. 2024-03-24T00:49:12.473 INFO:tasks.ceph.mds.a.smithi067.stderr: -8432> 2024-03-24T00:49:00.635+0000 7f4e8788a640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T00:49:12.481 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-24T00:49:12.484 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-24T00:49:12.505 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-03-24T00:49:12.533 INFO:teuthology.orchestra.run.smithi067.stdout:[global] 2024-03-24T00:49:12.533 INFO:teuthology.orchestra.run.smithi067.stdout: chdir = "" 2024-03-24T00:49:12.533 INFO:teuthology.orchestra.run.smithi067.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-24T00:49:12.533 INFO:teuthology.orchestra.run.smithi067.stdout: auth supported = cephx 2024-03-24T00:49:12.533 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.533 INFO:teuthology.orchestra.run.smithi067.stdout: filestore xattr use omap = true 2024-03-24T00:49:12.533 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.533 INFO:teuthology.orchestra.run.smithi067.stdout: mon clock drift allowed = 1.000 2024-03-24T00:49:12.533 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.534 INFO:teuthology.orchestra.run.smithi067.stdout: osd crush chooseleaf type = 0 2024-03-24T00:49:12.534 INFO:teuthology.orchestra.run.smithi067.stdout: auth debug = true 2024-03-24T00:49:12.534 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.534 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on old message = true 2024-03-24T00:49:12.534 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on bug = true 2024-03-24T00:49:12.534 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.534 INFO:teuthology.orchestra.run.smithi067.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-24T00:49:12.534 INFO:teuthology.orchestra.run.smithi067.stdout: mon pg warn max object skew = 0 2024-03-24T00:49:12.534 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.534 INFO:teuthology.orchestra.run.smithi067.stdout: # disable pg_autoscaler by default for new pools 2024-03-24T00:49:12.534 INFO:teuthology.orchestra.run.smithi067.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-24T00:49:12.534 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.534 INFO:teuthology.orchestra.run.smithi067.stdout: osd pool default size = 2 2024-03-24T00:49:12.534 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.534 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd allow primary affinity = true 2024-03-24T00:49:12.535 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd allow pg remap = true 2024-03-24T00:49:12.535 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on legacy crush tunables = false 2024-03-24T00:49:12.535 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on crush straw calc version zero = false 2024-03-24T00:49:12.535 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on no sortbitwise = false 2024-03-24T00:49:12.535 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on osd down out interval zero = false 2024-03-24T00:49:12.535 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on too few osds = false 2024-03-24T00:49:12.535 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-24T00:49:12.535 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-24T00:49:12.535 INFO:teuthology.orchestra.run.smithi067.stdout: mon_allow_pool_size_one = true 2024-03-24T00:49:12.535 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.535 INFO:teuthology.orchestra.run.smithi067.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-24T00:49:12.535 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.535 INFO:teuthology.orchestra.run.smithi067.stdout: osd default data pool replay window = 5 2024-03-24T00:49:12.535 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.535 INFO:teuthology.orchestra.run.smithi067.stdout: mon allow pool delete = true 2024-03-24T00:49:12.535 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.535 INFO:teuthology.orchestra.run.smithi067.stdout: mon cluster log file level = debug 2024-03-24T00:49:12.536 INFO:teuthology.orchestra.run.smithi067.stdout: debug asserts on shutdown = true 2024-03-24T00:49:12.536 INFO:teuthology.orchestra.run.smithi067.stdout: mon health detail to clog = false 2024-03-24T00:49:12.536 INFO:teuthology.orchestra.run.smithi067.stdout: mon host = "172.21.15.67,172.21.15.72,[v2:172.21.15.72:3301,v1:172.21.15.72:6790]" 2024-03-24T00:49:12.536 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on skipped message = False 2024-03-24T00:49:12.536 INFO:teuthology.orchestra.run.smithi067.stdout: fsid = de1d0178-808e-49ca-a986-e297e523b976 2024-03-24T00:49:12.536 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.536 INFO:teuthology.orchestra.run.smithi067.stdout:[osd] 2024-03-24T00:49:12.536 INFO:teuthology.orchestra.run.smithi067.stdout: osd journal size = 100 2024-03-24T00:49:12.536 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.536 INFO:teuthology.orchestra.run.smithi067.stdout: osd scrub load threshold = 5.0 2024-03-24T00:49:12.536 INFO:teuthology.orchestra.run.smithi067.stdout: osd scrub max interval = 600 2024-03-24T00:49:12.536 INFO:teuthology.orchestra.run.smithi067.stdout: osd mclock profile = high_recovery_ops 2024-03-24T00:49:12.536 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.536 INFO:teuthology.orchestra.run.smithi067.stdout: osd recover clone overlap = true 2024-03-24T00:49:12.536 INFO:teuthology.orchestra.run.smithi067.stdout: osd recovery max chunk = 1048576 2024-03-24T00:49:12.537 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.537 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug shutdown = true 2024-03-24T00:49:12.537 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug op order = true 2024-03-24T00:49:12.537 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug verify stray on activate = true 2024-03-24T00:49:12.537 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug trim objects = true 2024-03-24T00:49:12.537 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.537 INFO:teuthology.orchestra.run.smithi067.stdout: osd open classes on start = true 2024-03-24T00:49:12.537 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug pg log writeout = true 2024-03-24T00:49:12.537 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.537 INFO:teuthology.orchestra.run.smithi067.stdout: osd deep scrub update digest min age = 30 2024-03-24T00:49:12.538 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.538 INFO:teuthology.orchestra.run.smithi067.stdout: osd map max advance = 10 2024-03-24T00:49:12.538 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.538 INFO:teuthology.orchestra.run.smithi067.stdout: journal zero on create = true 2024-03-24T00:49:12.538 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.538 INFO:teuthology.orchestra.run.smithi067.stdout: filestore ondisk finisher threads = 3 2024-03-24T00:49:12.538 INFO:teuthology.orchestra.run.smithi067.stdout: filestore apply finisher threads = 3 2024-03-24T00:49:12.538 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.538 INFO:teuthology.orchestra.run.smithi067.stdout: bdev debug aio = true 2024-03-24T00:49:12.538 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug misdirected ops = true 2024-03-24T00:49:12.538 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore block size = 96636764160 2024-03-24T00:49:12.538 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore fsck on mount = True 2024-03-24T00:49:12.538 INFO:teuthology.orchestra.run.smithi067.stdout: debug bluefs = 20 2024-03-24T00:49:12.538 INFO:teuthology.orchestra.run.smithi067.stdout: debug bluestore = 20 2024-03-24T00:49:12.539 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-03-24T00:49:12.539 INFO:teuthology.orchestra.run.smithi067.stdout: debug osd = 20 2024-03-24T00:49:12.539 INFO:teuthology.orchestra.run.smithi067.stdout: debug rocksdb = 10 2024-03-24T00:49:12.539 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd backfillfull_ratio = 0.85 2024-03-24T00:49:12.539 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd full ratio = 0.9 2024-03-24T00:49:12.539 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd nearfull ratio = 0.8 2024-03-24T00:49:12.539 INFO:teuthology.orchestra.run.smithi067.stdout: osd failsafe full ratio = 0.95 2024-03-24T00:49:12.539 INFO:teuthology.orchestra.run.smithi067.stdout: osd objectstore = bluestore 2024-03-24T00:49:12.539 INFO:teuthology.orchestra.run.smithi067.stdout: osd op complaint time = 180 2024-03-24T00:49:12.539 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.539 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr] 2024-03-24T00:49:12.539 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-03-24T00:49:12.539 INFO:teuthology.orchestra.run.smithi067.stdout: debug mgr = 20 2024-03-24T00:49:12.539 INFO:teuthology.orchestra.run.smithi067.stdout: debug mon = 20 2024-03-24T00:49:12.539 INFO:teuthology.orchestra.run.smithi067.stdout: debug auth = 20 2024-03-24T00:49:12.539 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min pgs per osd = 4 2024-03-24T00:49:12.539 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min bytes per osd = 10 2024-03-24T00:49:12.540 INFO:teuthology.orchestra.run.smithi067.stdout: mgr/telemetry/nag = false 2024-03-24T00:49:12.540 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.540 INFO:teuthology.orchestra.run.smithi067.stdout:[mon] 2024-03-24T00:49:12.540 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-03-24T00:49:12.540 INFO:teuthology.orchestra.run.smithi067.stdout: debug mon = 20 2024-03-24T00:49:12.540 INFO:teuthology.orchestra.run.smithi067.stdout: debug paxos = 20 2024-03-24T00:49:12.540 INFO:teuthology.orchestra.run.smithi067.stdout: debug auth = 20 2024-03-24T00:49:12.540 INFO:teuthology.orchestra.run.smithi067.stdout: mon data avail warn = 5 2024-03-24T00:49:12.540 INFO:teuthology.orchestra.run.smithi067.stdout: mon mgr mkfs grace = 240 2024-03-24T00:49:12.540 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min pgs per osd = 4 2024-03-24T00:49:12.540 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd reporter subtree level = osd 2024-03-24T00:49:12.540 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd prime pg temp = true 2024-03-24T00:49:12.541 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min bytes per osd = 10 2024-03-24T00:49:12.541 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.541 INFO:teuthology.orchestra.run.smithi067.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-24T00:49:12.541 INFO:teuthology.orchestra.run.smithi067.stdout: auth mon ticket ttl = 660 # 11m 2024-03-24T00:49:12.541 INFO:teuthology.orchestra.run.smithi067.stdout: auth service ticket ttl = 240 # 4m 2024-03-24T00:49:12.541 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.541 INFO:teuthology.orchestra.run.smithi067.stdout: # don't complain about insecure global_id in the test suite 2024-03-24T00:49:12.541 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-24T00:49:12.541 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-24T00:49:12.542 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.542 INFO:teuthology.orchestra.run.smithi067.stdout: # 1m isn't quite enough 2024-03-24T00:49:12.542 INFO:teuthology.orchestra.run.smithi067.stdout: mon_down_mkfs_grace = 2m 2024-03-24T00:49:12.542 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.542 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_filestore_osds = false 2024-03-24T00:49:12.542 INFO:teuthology.orchestra.run.smithi067.stdout: mon op complaint time = 120 2024-03-24T00:49:12.542 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.542 INFO:teuthology.orchestra.run.smithi067.stdout:[client] 2024-03-24T00:49:12.542 INFO:teuthology.orchestra.run.smithi067.stdout: rgw cache enabled = true 2024-03-24T00:49:12.542 INFO:teuthology.orchestra.run.smithi067.stdout: rgw enable ops log = true 2024-03-24T00:49:12.542 INFO:teuthology.orchestra.run.smithi067.stdout: rgw enable usage log = true 2024-03-24T00:49:12.542 INFO:teuthology.orchestra.run.smithi067.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-24T00:49:12.542 INFO:teuthology.orchestra.run.smithi067.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-24T00:49:12.542 INFO:teuthology.orchestra.run.smithi067.stdout: client mount timeout = 600 2024-03-24T00:49:12.543 INFO:teuthology.orchestra.run.smithi067.stdout: debug client = 20 2024-03-24T00:49:12.543 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-03-24T00:49:12.543 INFO:teuthology.orchestra.run.smithi067.stdout: pid file = "" 2024-03-24T00:49:12.543 INFO:teuthology.orchestra.run.smithi067.stdout: rados mon op timeout = 900 2024-03-24T00:49:12.543 INFO:teuthology.orchestra.run.smithi067.stdout: rados osd op timeout = 900 2024-03-24T00:49:12.543 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.a] 2024-03-24T00:49:12.543 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.b] 2024-03-24T00:49:12.543 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.c] 2024-03-24T00:49:12.543 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.a] 2024-03-24T00:49:12.543 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.c] 2024-03-24T00:49:12.543 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.b] 2024-03-24T00:49:12.543 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.d] 2024-03-24T00:49:12.543 INFO:teuthology.orchestra.run.smithi067.stdout:[mds] 2024-03-24T00:49:12.543 INFO:teuthology.orchestra.run.smithi067.stdout: debug mds = 20 2024-03-24T00:49:12.544 INFO:teuthology.orchestra.run.smithi067.stdout: debug mds balancer = 20 2024-03-24T00:49:12.544 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-03-24T00:49:12.544 INFO:teuthology.orchestra.run.smithi067.stdout: mds debug frag = True 2024-03-24T00:49:12.544 INFO:teuthology.orchestra.run.smithi067.stdout: mds debug scatterstat = True 2024-03-24T00:49:12.544 INFO:teuthology.orchestra.run.smithi067.stdout: mds op complaint time = 180 2024-03-24T00:49:12.544 INFO:teuthology.orchestra.run.smithi067.stdout: mds verify scatter = False 2024-03-24T00:49:12.544 INFO:teuthology.orchestra.run.smithi067.stdout: osd op complaint time = 180 2024-03-24T00:49:12.544 INFO:teuthology.orchestra.run.smithi067.stdout: rados mon op timeout = 900 2024-03-24T00:49:12.544 INFO:teuthology.orchestra.run.smithi067.stdout: rados osd op timeout = 900 2024-03-24T00:49:12.544 INFO:teuthology.orchestra.run.smithi072.stdout:[global] 2024-03-24T00:49:12.544 INFO:teuthology.orchestra.run.smithi072.stdout: chdir = "" 2024-03-24T00:49:12.545 INFO:teuthology.orchestra.run.smithi072.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-24T00:49:12.545 INFO:teuthology.orchestra.run.smithi072.stdout: auth supported = cephx 2024-03-24T00:49:12.545 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.545 INFO:teuthology.orchestra.run.smithi072.stdout: filestore xattr use omap = true 2024-03-24T00:49:12.545 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.545 INFO:teuthology.orchestra.run.smithi072.stdout: mon clock drift allowed = 1.000 2024-03-24T00:49:12.545 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.545 INFO:teuthology.orchestra.run.smithi072.stdout: osd crush chooseleaf type = 0 2024-03-24T00:49:12.545 INFO:teuthology.orchestra.run.smithi072.stdout: auth debug = true 2024-03-24T00:49:12.545 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.545 INFO:teuthology.orchestra.run.smithi072.stdout: ms die on old message = true 2024-03-24T00:49:12.545 INFO:teuthology.orchestra.run.smithi072.stdout: ms die on bug = true 2024-03-24T00:49:12.545 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.545 INFO:teuthology.orchestra.run.smithi072.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-24T00:49:12.545 INFO:teuthology.orchestra.run.smithi072.stdout: mon pg warn max object skew = 0 2024-03-24T00:49:12.545 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.545 INFO:teuthology.orchestra.run.smithi072.stdout: # disable pg_autoscaler by default for new pools 2024-03-24T00:49:12.546 INFO:teuthology.orchestra.run.smithi072.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-24T00:49:12.546 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.546 INFO:teuthology.orchestra.run.smithi072.stdout: osd pool default size = 2 2024-03-24T00:49:12.546 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.546 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd allow primary affinity = true 2024-03-24T00:49:12.546 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd allow pg remap = true 2024-03-24T00:49:12.546 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on legacy crush tunables = false 2024-03-24T00:49:12.546 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on crush straw calc version zero = false 2024-03-24T00:49:12.546 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on no sortbitwise = false 2024-03-24T00:49:12.546 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on osd down out interval zero = false 2024-03-24T00:49:12.546 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on too few osds = false 2024-03-24T00:49:12.546 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-24T00:49:12.546 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-24T00:49:12.546 INFO:teuthology.orchestra.run.smithi072.stdout: mon_allow_pool_size_one = true 2024-03-24T00:49:12.546 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.547 INFO:teuthology.orchestra.run.smithi072.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-24T00:49:12.547 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.547 INFO:teuthology.orchestra.run.smithi072.stdout: osd default data pool replay window = 5 2024-03-24T00:49:12.547 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.547 INFO:teuthology.orchestra.run.smithi072.stdout: mon allow pool delete = true 2024-03-24T00:49:12.547 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.547 INFO:teuthology.orchestra.run.smithi072.stdout: mon cluster log file level = debug 2024-03-24T00:49:12.547 INFO:teuthology.orchestra.run.smithi072.stdout: debug asserts on shutdown = true 2024-03-24T00:49:12.547 INFO:teuthology.orchestra.run.smithi072.stdout: mon health detail to clog = false 2024-03-24T00:49:12.547 INFO:teuthology.orchestra.run.smithi072.stdout: mon host = "172.21.15.67,172.21.15.72,[v2:172.21.15.72:3301,v1:172.21.15.72:6790]" 2024-03-24T00:49:12.547 INFO:teuthology.orchestra.run.smithi072.stdout: ms die on skipped message = False 2024-03-24T00:49:12.547 INFO:teuthology.orchestra.run.smithi072.stdout: fsid = de1d0178-808e-49ca-a986-e297e523b976 2024-03-24T00:49:12.547 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.547 INFO:teuthology.orchestra.run.smithi072.stdout:[osd] 2024-03-24T00:49:12.547 INFO:teuthology.orchestra.run.smithi072.stdout: osd journal size = 100 2024-03-24T00:49:12.547 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.548 INFO:teuthology.orchestra.run.smithi072.stdout: osd scrub load threshold = 5.0 2024-03-24T00:49:12.548 INFO:teuthology.orchestra.run.smithi072.stdout: osd scrub max interval = 600 2024-03-24T00:49:12.548 INFO:teuthology.orchestra.run.smithi072.stdout: osd mclock profile = high_recovery_ops 2024-03-24T00:49:12.548 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.548 INFO:teuthology.orchestra.run.smithi072.stdout: osd recover clone overlap = true 2024-03-24T00:49:12.548 INFO:teuthology.orchestra.run.smithi072.stdout: osd recovery max chunk = 1048576 2024-03-24T00:49:12.548 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.548 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug shutdown = true 2024-03-24T00:49:12.548 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug op order = true 2024-03-24T00:49:12.548 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug verify stray on activate = true 2024-03-24T00:49:12.548 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug trim objects = true 2024-03-24T00:49:12.548 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.548 INFO:teuthology.orchestra.run.smithi072.stdout: osd open classes on start = true 2024-03-24T00:49:12.548 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug pg log writeout = true 2024-03-24T00:49:12.548 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.548 INFO:teuthology.orchestra.run.smithi072.stdout: osd deep scrub update digest min age = 30 2024-03-24T00:49:12.549 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.549 INFO:teuthology.orchestra.run.smithi072.stdout: osd map max advance = 10 2024-03-24T00:49:12.549 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.549 INFO:teuthology.orchestra.run.smithi072.stdout: journal zero on create = true 2024-03-24T00:49:12.549 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.549 INFO:teuthology.orchestra.run.smithi072.stdout: filestore ondisk finisher threads = 3 2024-03-24T00:49:12.549 INFO:teuthology.orchestra.run.smithi072.stdout: filestore apply finisher threads = 3 2024-03-24T00:49:12.549 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.549 INFO:teuthology.orchestra.run.smithi072.stdout: bdev debug aio = true 2024-03-24T00:49:12.549 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug misdirected ops = true 2024-03-24T00:49:12.549 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore block size = 96636764160 2024-03-24T00:49:12.549 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore fsck on mount = True 2024-03-24T00:49:12.549 INFO:teuthology.orchestra.run.smithi072.stdout: debug bluefs = 20 2024-03-24T00:49:12.549 INFO:teuthology.orchestra.run.smithi072.stdout: debug bluestore = 20 2024-03-24T00:49:12.549 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-24T00:49:12.549 INFO:teuthology.orchestra.run.smithi072.stdout: debug osd = 20 2024-03-24T00:49:12.549 INFO:teuthology.orchestra.run.smithi072.stdout: debug rocksdb = 10 2024-03-24T00:49:12.550 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd backfillfull_ratio = 0.85 2024-03-24T00:49:12.550 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd full ratio = 0.9 2024-03-24T00:49:12.550 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd nearfull ratio = 0.8 2024-03-24T00:49:12.550 INFO:teuthology.orchestra.run.smithi072.stdout: osd failsafe full ratio = 0.95 2024-03-24T00:49:12.550 INFO:teuthology.orchestra.run.smithi072.stdout: osd objectstore = bluestore 2024-03-24T00:49:12.550 INFO:teuthology.orchestra.run.smithi072.stdout: osd op complaint time = 180 2024-03-24T00:49:12.550 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.550 INFO:teuthology.orchestra.run.smithi072.stdout:[mgr] 2024-03-24T00:49:12.550 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-24T00:49:12.550 INFO:teuthology.orchestra.run.smithi072.stdout: debug mgr = 20 2024-03-24T00:49:12.550 INFO:teuthology.orchestra.run.smithi072.stdout: debug mon = 20 2024-03-24T00:49:12.550 INFO:teuthology.orchestra.run.smithi072.stdout: debug auth = 20 2024-03-24T00:49:12.550 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min pgs per osd = 4 2024-03-24T00:49:12.551 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min bytes per osd = 10 2024-03-24T00:49:12.551 INFO:teuthology.orchestra.run.smithi072.stdout: mgr/telemetry/nag = false 2024-03-24T00:49:12.551 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.551 INFO:teuthology.orchestra.run.smithi072.stdout:[mon] 2024-03-24T00:49:12.551 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-24T00:49:12.551 INFO:teuthology.orchestra.run.smithi072.stdout: debug mon = 20 2024-03-24T00:49:12.551 INFO:teuthology.orchestra.run.smithi072.stdout: debug paxos = 20 2024-03-24T00:49:12.551 INFO:teuthology.orchestra.run.smithi072.stdout: debug auth = 20 2024-03-24T00:49:12.552 INFO:teuthology.orchestra.run.smithi072.stdout: mon data avail warn = 5 2024-03-24T00:49:12.552 INFO:teuthology.orchestra.run.smithi072.stdout: mon mgr mkfs grace = 240 2024-03-24T00:49:12.552 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min pgs per osd = 4 2024-03-24T00:49:12.552 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd reporter subtree level = osd 2024-03-24T00:49:12.552 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd prime pg temp = true 2024-03-24T00:49:12.552 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min bytes per osd = 10 2024-03-24T00:49:12.552 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.552 INFO:teuthology.orchestra.run.smithi072.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-24T00:49:12.552 INFO:teuthology.orchestra.run.smithi072.stdout: auth mon ticket ttl = 660 # 11m 2024-03-24T00:49:12.552 INFO:teuthology.orchestra.run.smithi072.stdout: auth service ticket ttl = 240 # 4m 2024-03-24T00:49:12.552 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.552 INFO:teuthology.orchestra.run.smithi072.stdout: # don't complain about insecure global_id in the test suite 2024-03-24T00:49:12.552 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-24T00:49:12.552 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-24T00:49:12.552 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.553 INFO:teuthology.orchestra.run.smithi072.stdout: # 1m isn't quite enough 2024-03-24T00:49:12.553 INFO:teuthology.orchestra.run.smithi072.stdout: mon_down_mkfs_grace = 2m 2024-03-24T00:49:12.553 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.553 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_filestore_osds = false 2024-03-24T00:49:12.553 INFO:teuthology.orchestra.run.smithi072.stdout: mon op complaint time = 120 2024-03-24T00:49:12.553 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.553 INFO:teuthology.orchestra.run.smithi072.stdout:[client] 2024-03-24T00:49:12.553 INFO:teuthology.orchestra.run.smithi072.stdout: rgw cache enabled = true 2024-03-24T00:49:12.553 INFO:teuthology.orchestra.run.smithi072.stdout: rgw enable ops log = true 2024-03-24T00:49:12.553 INFO:teuthology.orchestra.run.smithi072.stdout: rgw enable usage log = true 2024-03-24T00:49:12.553 INFO:teuthology.orchestra.run.smithi072.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-24T00:49:12.553 INFO:teuthology.orchestra.run.smithi072.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-24T00:49:12.553 INFO:teuthology.orchestra.run.smithi072.stdout: client mount timeout = 600 2024-03-24T00:49:12.553 INFO:teuthology.orchestra.run.smithi072.stdout: debug client = 20 2024-03-24T00:49:12.554 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-24T00:49:12.554 INFO:teuthology.orchestra.run.smithi072.stdout: pid file = "" 2024-03-24T00:49:12.554 INFO:teuthology.orchestra.run.smithi072.stdout: rados mon op timeout = 900 2024-03-24T00:49:12.554 INFO:teuthology.orchestra.run.smithi072.stdout: rados osd op timeout = 900 2024-03-24T00:49:12.554 INFO:teuthology.orchestra.run.smithi072.stdout:[mon.a] 2024-03-24T00:49:12.554 INFO:teuthology.orchestra.run.smithi072.stdout:[mon.b] 2024-03-24T00:49:12.554 INFO:teuthology.orchestra.run.smithi072.stdout:[mon.c] 2024-03-24T00:49:12.554 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.a] 2024-03-24T00:49:12.554 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.c] 2024-03-24T00:49:12.554 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.b] 2024-03-24T00:49:12.554 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.d] 2024-03-24T00:49:12.554 INFO:teuthology.orchestra.run.smithi072.stdout:[mds] 2024-03-24T00:49:12.554 INFO:teuthology.orchestra.run.smithi072.stdout: debug mds = 20 2024-03-24T00:49:12.554 INFO:teuthology.orchestra.run.smithi072.stdout: debug mds balancer = 20 2024-03-24T00:49:12.554 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-24T00:49:12.555 INFO:teuthology.orchestra.run.smithi072.stdout: mds debug frag = True 2024-03-24T00:49:12.555 INFO:teuthology.orchestra.run.smithi072.stdout: mds debug scatterstat = True 2024-03-24T00:49:12.555 INFO:teuthology.orchestra.run.smithi072.stdout: mds op complaint time = 180 2024-03-24T00:49:12.555 INFO:teuthology.orchestra.run.smithi072.stdout: mds verify scatter = False 2024-03-24T00:49:12.555 INFO:teuthology.orchestra.run.smithi072.stdout: osd op complaint time = 180 2024-03-24T00:49:12.555 INFO:teuthology.orchestra.run.smithi072.stdout: rados mon op timeout = 900 2024-03-24T00:49:12.555 INFO:teuthology.orchestra.run.smithi072.stdout: rados osd op timeout = 900 2024-03-24T00:49:12.557 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-24T00:49:12.599 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-24T00:49:12.647 INFO:teuthology.orchestra.run.smithi067.stdout:[global] 2024-03-24T00:49:12.647 INFO:teuthology.orchestra.run.smithi067.stdout: chdir = "" 2024-03-24T00:49:12.647 INFO:teuthology.orchestra.run.smithi067.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-24T00:49:12.647 INFO:teuthology.orchestra.run.smithi067.stdout: auth supported = cephx 2024-03-24T00:49:12.647 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.647 INFO:teuthology.orchestra.run.smithi067.stdout: filestore xattr use omap = true 2024-03-24T00:49:12.647 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.647 INFO:teuthology.orchestra.run.smithi067.stdout: mon clock drift allowed = 1.000 2024-03-24T00:49:12.647 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.648 INFO:teuthology.orchestra.run.smithi067.stdout: osd crush chooseleaf type = 0 2024-03-24T00:49:12.648 INFO:teuthology.orchestra.run.smithi067.stdout: auth debug = true 2024-03-24T00:49:12.648 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.648 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on old message = true 2024-03-24T00:49:12.648 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on bug = true 2024-03-24T00:49:12.648 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.648 INFO:teuthology.orchestra.run.smithi067.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-24T00:49:12.648 INFO:teuthology.orchestra.run.smithi067.stdout: mon pg warn max object skew = 0 2024-03-24T00:49:12.648 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.648 INFO:teuthology.orchestra.run.smithi067.stdout: # disable pg_autoscaler by default for new pools 2024-03-24T00:49:12.648 INFO:teuthology.orchestra.run.smithi067.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-24T00:49:12.648 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.648 INFO:teuthology.orchestra.run.smithi067.stdout: osd pool default size = 2 2024-03-24T00:49:12.648 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.649 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd allow primary affinity = true 2024-03-24T00:49:12.649 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd allow pg remap = true 2024-03-24T00:49:12.649 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on legacy crush tunables = false 2024-03-24T00:49:12.649 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on crush straw calc version zero = false 2024-03-24T00:49:12.649 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on no sortbitwise = false 2024-03-24T00:49:12.649 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on osd down out interval zero = false 2024-03-24T00:49:12.649 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on too few osds = false 2024-03-24T00:49:12.649 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-24T00:49:12.649 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-24T00:49:12.649 INFO:teuthology.orchestra.run.smithi067.stdout: mon_allow_pool_size_one = true 2024-03-24T00:49:12.649 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.649 INFO:teuthology.orchestra.run.smithi067.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-24T00:49:12.649 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.649 INFO:teuthology.orchestra.run.smithi067.stdout: osd default data pool replay window = 5 2024-03-24T00:49:12.649 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.650 INFO:teuthology.orchestra.run.smithi067.stdout: mon allow pool delete = true 2024-03-24T00:49:12.650 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.650 INFO:teuthology.orchestra.run.smithi067.stdout: mon cluster log file level = debug 2024-03-24T00:49:12.650 INFO:teuthology.orchestra.run.smithi067.stdout: debug asserts on shutdown = true 2024-03-24T00:49:12.650 INFO:teuthology.orchestra.run.smithi067.stdout: mon health detail to clog = false 2024-03-24T00:49:12.650 INFO:teuthology.orchestra.run.smithi067.stdout: mon host = "172.21.15.67,172.21.15.72,[v2:172.21.15.72:3301,v1:172.21.15.72:6790]" 2024-03-24T00:49:12.650 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on skipped message = False 2024-03-24T00:49:12.650 INFO:teuthology.orchestra.run.smithi067.stdout: fsid = de1d0178-808e-49ca-a986-e297e523b976 2024-03-24T00:49:12.650 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.650 INFO:teuthology.orchestra.run.smithi067.stdout:[osd] 2024-03-24T00:49:12.650 INFO:teuthology.orchestra.run.smithi067.stdout: osd journal size = 100 2024-03-24T00:49:12.650 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.651 INFO:teuthology.orchestra.run.smithi067.stdout: osd scrub load threshold = 5.0 2024-03-24T00:49:12.651 INFO:teuthology.orchestra.run.smithi067.stdout: osd scrub max interval = 600 2024-03-24T00:49:12.651 INFO:teuthology.orchestra.run.smithi067.stdout: osd mclock profile = high_recovery_ops 2024-03-24T00:49:12.651 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.651 INFO:teuthology.orchestra.run.smithi067.stdout: osd recover clone overlap = true 2024-03-24T00:49:12.651 INFO:teuthology.orchestra.run.smithi067.stdout: osd recovery max chunk = 1048576 2024-03-24T00:49:12.651 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.651 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug shutdown = true 2024-03-24T00:49:12.651 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug op order = true 2024-03-24T00:49:12.651 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug verify stray on activate = true 2024-03-24T00:49:12.651 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug trim objects = true 2024-03-24T00:49:12.651 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.651 INFO:teuthology.orchestra.run.smithi067.stdout: osd open classes on start = true 2024-03-24T00:49:12.652 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug pg log writeout = true 2024-03-24T00:49:12.652 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.652 INFO:teuthology.orchestra.run.smithi067.stdout: osd deep scrub update digest min age = 30 2024-03-24T00:49:12.652 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.652 INFO:teuthology.orchestra.run.smithi067.stdout: osd map max advance = 10 2024-03-24T00:49:12.652 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.652 INFO:teuthology.orchestra.run.smithi067.stdout: journal zero on create = true 2024-03-24T00:49:12.652 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.652 INFO:teuthology.orchestra.run.smithi067.stdout: filestore ondisk finisher threads = 3 2024-03-24T00:49:12.652 INFO:teuthology.orchestra.run.smithi067.stdout: filestore apply finisher threads = 3 2024-03-24T00:49:12.652 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.652 INFO:teuthology.orchestra.run.smithi067.stdout: bdev debug aio = true 2024-03-24T00:49:12.652 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug misdirected ops = true 2024-03-24T00:49:12.653 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore block size = 96636764160 2024-03-24T00:49:12.653 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore fsck on mount = True 2024-03-24T00:49:12.653 INFO:teuthology.orchestra.run.smithi067.stdout: debug bluefs = 20 2024-03-24T00:49:12.653 INFO:teuthology.orchestra.run.smithi067.stdout: debug bluestore = 20 2024-03-24T00:49:12.653 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-03-24T00:49:12.653 INFO:teuthology.orchestra.run.smithi067.stdout: debug osd = 20 2024-03-24T00:49:12.653 INFO:teuthology.orchestra.run.smithi067.stdout: debug rocksdb = 10 2024-03-24T00:49:12.653 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd backfillfull_ratio = 0.85 2024-03-24T00:49:12.653 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd full ratio = 0.9 2024-03-24T00:49:12.653 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd nearfull ratio = 0.8 2024-03-24T00:49:12.653 INFO:teuthology.orchestra.run.smithi067.stdout: osd failsafe full ratio = 0.95 2024-03-24T00:49:12.653 INFO:teuthology.orchestra.run.smithi067.stdout: osd objectstore = bluestore 2024-03-24T00:49:12.653 INFO:teuthology.orchestra.run.smithi067.stdout: osd op complaint time = 180 2024-03-24T00:49:12.653 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.654 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr] 2024-03-24T00:49:12.654 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-03-24T00:49:12.654 INFO:teuthology.orchestra.run.smithi067.stdout: debug mgr = 20 2024-03-24T00:49:12.654 INFO:teuthology.orchestra.run.smithi067.stdout: debug mon = 20 2024-03-24T00:49:12.654 INFO:teuthology.orchestra.run.smithi067.stdout: debug auth = 20 2024-03-24T00:49:12.654 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min pgs per osd = 4 2024-03-24T00:49:12.654 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min bytes per osd = 10 2024-03-24T00:49:12.654 INFO:teuthology.orchestra.run.smithi067.stdout: mgr/telemetry/nag = false 2024-03-24T00:49:12.655 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.655 INFO:teuthology.orchestra.run.smithi067.stdout:[mon] 2024-03-24T00:49:12.655 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-03-24T00:49:12.655 INFO:teuthology.orchestra.run.smithi067.stdout: debug mon = 20 2024-03-24T00:49:12.655 INFO:teuthology.orchestra.run.smithi067.stdout: debug paxos = 20 2024-03-24T00:49:12.655 INFO:teuthology.orchestra.run.smithi067.stdout: debug auth = 20 2024-03-24T00:49:12.655 INFO:teuthology.orchestra.run.smithi067.stdout: mon data avail warn = 5 2024-03-24T00:49:12.655 INFO:teuthology.orchestra.run.smithi067.stdout: mon mgr mkfs grace = 240 2024-03-24T00:49:12.655 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min pgs per osd = 4 2024-03-24T00:49:12.655 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd reporter subtree level = osd 2024-03-24T00:49:12.655 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd prime pg temp = true 2024-03-24T00:49:12.655 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min bytes per osd = 10 2024-03-24T00:49:12.655 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.655 INFO:teuthology.orchestra.run.smithi067.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-24T00:49:12.655 INFO:teuthology.orchestra.run.smithi067.stdout: auth mon ticket ttl = 660 # 11m 2024-03-24T00:49:12.656 INFO:teuthology.orchestra.run.smithi067.stdout: auth service ticket ttl = 240 # 4m 2024-03-24T00:49:12.656 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.656 INFO:teuthology.orchestra.run.smithi067.stdout: # don't complain about insecure global_id in the test suite 2024-03-24T00:49:12.656 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-24T00:49:12.656 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-24T00:49:12.656 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.656 INFO:teuthology.orchestra.run.smithi067.stdout: # 1m isn't quite enough 2024-03-24T00:49:12.656 INFO:teuthology.orchestra.run.smithi067.stdout: mon_down_mkfs_grace = 2m 2024-03-24T00:49:12.656 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.656 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_filestore_osds = false 2024-03-24T00:49:12.656 INFO:teuthology.orchestra.run.smithi067.stdout: mon op complaint time = 120 2024-03-24T00:49:12.656 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:12.657 INFO:teuthology.orchestra.run.smithi067.stdout:[client] 2024-03-24T00:49:12.657 INFO:teuthology.orchestra.run.smithi067.stdout: rgw cache enabled = true 2024-03-24T00:49:12.657 INFO:teuthology.orchestra.run.smithi067.stdout: rgw enable ops log = true 2024-03-24T00:49:12.657 INFO:teuthology.orchestra.run.smithi067.stdout: rgw enable usage log = true 2024-03-24T00:49:12.657 INFO:teuthology.orchestra.run.smithi067.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-24T00:49:12.657 INFO:teuthology.orchestra.run.smithi067.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-24T00:49:12.657 INFO:teuthology.orchestra.run.smithi067.stdout: client mount timeout = 600 2024-03-24T00:49:12.657 INFO:teuthology.orchestra.run.smithi067.stdout: debug client = 20 2024-03-24T00:49:12.657 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-03-24T00:49:12.657 INFO:teuthology.orchestra.run.smithi067.stdout: pid file = "" 2024-03-24T00:49:12.657 INFO:teuthology.orchestra.run.smithi067.stdout: rados mon op timeout = 900 2024-03-24T00:49:12.658 INFO:teuthology.orchestra.run.smithi067.stdout: rados osd op timeout = 900 2024-03-24T00:49:12.658 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.a] 2024-03-24T00:49:12.658 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.b] 2024-03-24T00:49:12.658 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.c] 2024-03-24T00:49:12.658 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.a] 2024-03-24T00:49:12.658 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.c] 2024-03-24T00:49:12.658 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.b] 2024-03-24T00:49:12.658 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.d] 2024-03-24T00:49:12.658 INFO:teuthology.orchestra.run.smithi067.stdout:[mds] 2024-03-24T00:49:12.658 INFO:teuthology.orchestra.run.smithi067.stdout: debug mds = 20 2024-03-24T00:49:12.658 INFO:teuthology.orchestra.run.smithi067.stdout: debug mds balancer = 20 2024-03-24T00:49:12.658 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-03-24T00:49:12.658 INFO:teuthology.orchestra.run.smithi067.stdout: mds debug frag = True 2024-03-24T00:49:12.658 INFO:teuthology.orchestra.run.smithi067.stdout: mds debug scatterstat = False 2024-03-24T00:49:12.659 INFO:teuthology.orchestra.run.smithi067.stdout: mds op complaint time = 180 2024-03-24T00:49:12.659 INFO:teuthology.orchestra.run.smithi067.stdout: mds verify scatter = False 2024-03-24T00:49:12.659 INFO:teuthology.orchestra.run.smithi067.stdout: osd op complaint time = 180 2024-03-24T00:49:12.659 INFO:teuthology.orchestra.run.smithi067.stdout: rados mon op timeout = 900 2024-03-24T00:49:12.659 INFO:teuthology.orchestra.run.smithi067.stdout: rados osd op timeout = 900 2024-03-24T00:49:12.659 INFO:teuthology.orchestra.run.smithi072.stdout:[global] 2024-03-24T00:49:12.659 INFO:teuthology.orchestra.run.smithi072.stdout: chdir = "" 2024-03-24T00:49:12.659 INFO:teuthology.orchestra.run.smithi072.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-24T00:49:12.660 INFO:teuthology.orchestra.run.smithi072.stdout: auth supported = cephx 2024-03-24T00:49:12.660 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.660 INFO:teuthology.orchestra.run.smithi072.stdout: filestore xattr use omap = true 2024-03-24T00:49:12.660 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.660 INFO:teuthology.orchestra.run.smithi072.stdout: mon clock drift allowed = 1.000 2024-03-24T00:49:12.660 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.660 INFO:teuthology.orchestra.run.smithi072.stdout: osd crush chooseleaf type = 0 2024-03-24T00:49:12.660 INFO:teuthology.orchestra.run.smithi072.stdout: auth debug = true 2024-03-24T00:49:12.660 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.660 INFO:teuthology.orchestra.run.smithi072.stdout: ms die on old message = true 2024-03-24T00:49:12.660 INFO:teuthology.orchestra.run.smithi072.stdout: ms die on bug = true 2024-03-24T00:49:12.660 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.660 INFO:teuthology.orchestra.run.smithi072.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-24T00:49:12.660 INFO:teuthology.orchestra.run.smithi072.stdout: mon pg warn max object skew = 0 2024-03-24T00:49:12.661 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.661 INFO:teuthology.orchestra.run.smithi072.stdout: # disable pg_autoscaler by default for new pools 2024-03-24T00:49:12.661 INFO:teuthology.orchestra.run.smithi072.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-24T00:49:12.661 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.661 INFO:teuthology.orchestra.run.smithi072.stdout: osd pool default size = 2 2024-03-24T00:49:12.661 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.661 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd allow primary affinity = true 2024-03-24T00:49:12.661 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd allow pg remap = true 2024-03-24T00:49:12.661 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on legacy crush tunables = false 2024-03-24T00:49:12.661 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on crush straw calc version zero = false 2024-03-24T00:49:12.661 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on no sortbitwise = false 2024-03-24T00:49:12.661 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on osd down out interval zero = false 2024-03-24T00:49:12.661 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on too few osds = false 2024-03-24T00:49:12.661 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-24T00:49:12.661 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-24T00:49:12.662 INFO:teuthology.orchestra.run.smithi072.stdout: mon_allow_pool_size_one = true 2024-03-24T00:49:12.662 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.662 INFO:teuthology.orchestra.run.smithi072.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-24T00:49:12.662 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.662 INFO:teuthology.orchestra.run.smithi072.stdout: osd default data pool replay window = 5 2024-03-24T00:49:12.662 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.662 INFO:teuthology.orchestra.run.smithi072.stdout: mon allow pool delete = true 2024-03-24T00:49:12.662 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.662 INFO:teuthology.orchestra.run.smithi072.stdout: mon cluster log file level = debug 2024-03-24T00:49:12.662 INFO:teuthology.orchestra.run.smithi072.stdout: debug asserts on shutdown = true 2024-03-24T00:49:12.662 INFO:teuthology.orchestra.run.smithi072.stdout: mon health detail to clog = false 2024-03-24T00:49:12.662 INFO:teuthology.orchestra.run.smithi072.stdout: mon host = "172.21.15.67,172.21.15.72,[v2:172.21.15.72:3301,v1:172.21.15.72:6790]" 2024-03-24T00:49:12.662 INFO:teuthology.orchestra.run.smithi072.stdout: ms die on skipped message = False 2024-03-24T00:49:12.662 INFO:teuthology.orchestra.run.smithi072.stdout: fsid = de1d0178-808e-49ca-a986-e297e523b976 2024-03-24T00:49:12.663 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.663 INFO:teuthology.orchestra.run.smithi072.stdout:[osd] 2024-03-24T00:49:12.663 INFO:teuthology.orchestra.run.smithi072.stdout: osd journal size = 100 2024-03-24T00:49:12.663 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.663 INFO:teuthology.orchestra.run.smithi072.stdout: osd scrub load threshold = 5.0 2024-03-24T00:49:12.663 INFO:teuthology.orchestra.run.smithi072.stdout: osd scrub max interval = 600 2024-03-24T00:49:12.663 INFO:teuthology.orchestra.run.smithi072.stdout: osd mclock profile = high_recovery_ops 2024-03-24T00:49:12.663 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.663 INFO:teuthology.orchestra.run.smithi072.stdout: osd recover clone overlap = true 2024-03-24T00:49:12.663 INFO:teuthology.orchestra.run.smithi072.stdout: osd recovery max chunk = 1048576 2024-03-24T00:49:12.663 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.663 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug shutdown = true 2024-03-24T00:49:12.663 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug op order = true 2024-03-24T00:49:12.663 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug verify stray on activate = true 2024-03-24T00:49:12.664 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug trim objects = true 2024-03-24T00:49:12.664 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.664 INFO:teuthology.orchestra.run.smithi072.stdout: osd open classes on start = true 2024-03-24T00:49:12.664 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug pg log writeout = true 2024-03-24T00:49:12.664 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.664 INFO:teuthology.orchestra.run.smithi072.stdout: osd deep scrub update digest min age = 30 2024-03-24T00:49:12.664 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.664 INFO:teuthology.orchestra.run.smithi072.stdout: osd map max advance = 10 2024-03-24T00:49:12.664 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.664 INFO:teuthology.orchestra.run.smithi072.stdout: journal zero on create = true 2024-03-24T00:49:12.664 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.664 INFO:teuthology.orchestra.run.smithi072.stdout: filestore ondisk finisher threads = 3 2024-03-24T00:49:12.664 INFO:teuthology.orchestra.run.smithi072.stdout: filestore apply finisher threads = 3 2024-03-24T00:49:12.665 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.665 INFO:teuthology.orchestra.run.smithi072.stdout: bdev debug aio = true 2024-03-24T00:49:12.665 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug misdirected ops = true 2024-03-24T00:49:12.665 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore block size = 96636764160 2024-03-24T00:49:12.665 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore fsck on mount = True 2024-03-24T00:49:12.665 INFO:teuthology.orchestra.run.smithi072.stdout: debug bluefs = 20 2024-03-24T00:49:12.665 INFO:teuthology.orchestra.run.smithi072.stdout: debug bluestore = 20 2024-03-24T00:49:12.665 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-24T00:49:12.665 INFO:teuthology.orchestra.run.smithi072.stdout: debug osd = 20 2024-03-24T00:49:12.665 INFO:teuthology.orchestra.run.smithi072.stdout: debug rocksdb = 10 2024-03-24T00:49:12.665 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd backfillfull_ratio = 0.85 2024-03-24T00:49:12.665 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd full ratio = 0.9 2024-03-24T00:49:12.666 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd nearfull ratio = 0.8 2024-03-24T00:49:12.666 INFO:teuthology.orchestra.run.smithi072.stdout: osd failsafe full ratio = 0.95 2024-03-24T00:49:12.666 INFO:teuthology.orchestra.run.smithi072.stdout: osd objectstore = bluestore 2024-03-24T00:49:12.666 INFO:teuthology.orchestra.run.smithi072.stdout: osd op complaint time = 180 2024-03-24T00:49:12.666 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.666 INFO:teuthology.orchestra.run.smithi072.stdout:[mgr] 2024-03-24T00:49:12.666 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-24T00:49:12.666 INFO:teuthology.orchestra.run.smithi072.stdout: debug mgr = 20 2024-03-24T00:49:12.666 INFO:teuthology.orchestra.run.smithi072.stdout: debug mon = 20 2024-03-24T00:49:12.666 INFO:teuthology.orchestra.run.smithi072.stdout: debug auth = 20 2024-03-24T00:49:12.666 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min pgs per osd = 4 2024-03-24T00:49:12.666 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min bytes per osd = 10 2024-03-24T00:49:12.667 INFO:teuthology.orchestra.run.smithi072.stdout: mgr/telemetry/nag = false 2024-03-24T00:49:12.667 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.667 INFO:teuthology.orchestra.run.smithi072.stdout:[mon] 2024-03-24T00:49:12.667 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-24T00:49:12.667 INFO:teuthology.orchestra.run.smithi072.stdout: debug mon = 20 2024-03-24T00:49:12.667 INFO:teuthology.orchestra.run.smithi072.stdout: debug paxos = 20 2024-03-24T00:49:12.667 INFO:teuthology.orchestra.run.smithi072.stdout: debug auth = 20 2024-03-24T00:49:12.667 INFO:teuthology.orchestra.run.smithi072.stdout: mon data avail warn = 5 2024-03-24T00:49:12.667 INFO:teuthology.orchestra.run.smithi072.stdout: mon mgr mkfs grace = 240 2024-03-24T00:49:12.667 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min pgs per osd = 4 2024-03-24T00:49:12.667 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd reporter subtree level = osd 2024-03-24T00:49:12.667 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd prime pg temp = true 2024-03-24T00:49:12.667 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min bytes per osd = 10 2024-03-24T00:49:12.667 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.668 INFO:teuthology.orchestra.run.smithi072.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-24T00:49:12.668 INFO:teuthology.orchestra.run.smithi072.stdout: auth mon ticket ttl = 660 # 11m 2024-03-24T00:49:12.668 INFO:teuthology.orchestra.run.smithi072.stdout: auth service ticket ttl = 240 # 4m 2024-03-24T00:49:12.668 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.668 INFO:teuthology.orchestra.run.smithi072.stdout: # don't complain about insecure global_id in the test suite 2024-03-24T00:49:12.668 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-24T00:49:12.668 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-24T00:49:12.668 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.668 INFO:teuthology.orchestra.run.smithi072.stdout: # 1m isn't quite enough 2024-03-24T00:49:12.668 INFO:teuthology.orchestra.run.smithi072.stdout: mon_down_mkfs_grace = 2m 2024-03-24T00:49:12.668 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.668 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_filestore_osds = false 2024-03-24T00:49:12.668 INFO:teuthology.orchestra.run.smithi072.stdout: mon op complaint time = 120 2024-03-24T00:49:12.668 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:12.669 INFO:teuthology.orchestra.run.smithi072.stdout:[client] 2024-03-24T00:49:12.669 INFO:teuthology.orchestra.run.smithi072.stdout: rgw cache enabled = true 2024-03-24T00:49:12.669 INFO:teuthology.orchestra.run.smithi072.stdout: rgw enable ops log = true 2024-03-24T00:49:12.669 INFO:teuthology.orchestra.run.smithi072.stdout: rgw enable usage log = true 2024-03-24T00:49:12.669 INFO:teuthology.orchestra.run.smithi072.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-24T00:49:12.669 INFO:teuthology.orchestra.run.smithi072.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-24T00:49:12.669 INFO:teuthology.orchestra.run.smithi072.stdout: client mount timeout = 600 2024-03-24T00:49:12.669 INFO:teuthology.orchestra.run.smithi072.stdout: debug client = 20 2024-03-24T00:49:12.669 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-24T00:49:12.669 INFO:teuthology.orchestra.run.smithi072.stdout: pid file = "" 2024-03-24T00:49:12.669 INFO:teuthology.orchestra.run.smithi072.stdout: rados mon op timeout = 900 2024-03-24T00:49:12.669 INFO:teuthology.orchestra.run.smithi072.stdout: rados osd op timeout = 900 2024-03-24T00:49:12.669 INFO:teuthology.orchestra.run.smithi072.stdout:[mon.a] 2024-03-24T00:49:12.669 INFO:teuthology.orchestra.run.smithi072.stdout:[mon.b] 2024-03-24T00:49:12.669 INFO:teuthology.orchestra.run.smithi072.stdout:[mon.c] 2024-03-24T00:49:12.669 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.a] 2024-03-24T00:49:12.670 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.c] 2024-03-24T00:49:12.670 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.b] 2024-03-24T00:49:12.670 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.d] 2024-03-24T00:49:12.670 INFO:teuthology.orchestra.run.smithi072.stdout:[mds] 2024-03-24T00:49:12.670 INFO:teuthology.orchestra.run.smithi072.stdout: debug mds = 20 2024-03-24T00:49:12.670 INFO:teuthology.orchestra.run.smithi072.stdout: debug mds balancer = 20 2024-03-24T00:49:12.670 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-24T00:49:12.670 INFO:teuthology.orchestra.run.smithi072.stdout: mds debug frag = True 2024-03-24T00:49:12.670 INFO:teuthology.orchestra.run.smithi072.stdout: mds debug scatterstat = False 2024-03-24T00:49:12.670 INFO:teuthology.orchestra.run.smithi072.stdout: mds op complaint time = 180 2024-03-24T00:49:12.670 INFO:teuthology.orchestra.run.smithi072.stdout: mds verify scatter = False 2024-03-24T00:49:12.670 INFO:teuthology.orchestra.run.smithi072.stdout: osd op complaint time = 180 2024-03-24T00:49:12.670 INFO:teuthology.orchestra.run.smithi072.stdout: rados mon op timeout = 900 2024-03-24T00:49:12.671 INFO:teuthology.orchestra.run.smithi072.stdout: rados osd op timeout = 900 2024-03-24T00:49:12.672 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rmomapkey 10000000001.00000000 bravo_head 2024-03-24T00:49:12.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.753+0000 7fcce0e41f40 1 Processor -- start 2024-03-24T00:49:12.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.753+0000 7fcce0e41f40 1 -- start start 2024-03-24T00:49:12.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.754+0000 7fcce0e41f40 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56546d0f20c0 0x56546d1969f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:12.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.754+0000 7fcce0e41f40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56546d10cf80 0x56546d195a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:12.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.754+0000 7fcce0e41f40 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56546d10d360 0x56546d195fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:12.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.754+0000 7fcce0e41f40 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x56546d18e690 con 0x56546d10d360 2024-03-24T00:49:12.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.754+0000 7fcce0e41f40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x56546d06a0d0 con 0x56546d10cf80 2024-03-24T00:49:12.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.754+0000 7fcce0e41f40 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x56546d0ecff0 con 0x56546d0f20c0 2024-03-24T00:49:12.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.754+0000 7fccde94e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56546d10d360 0x56546d195fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:12.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.754+0000 7fccde94e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56546d10d360 0x56546d195fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42606/0 (socket says 172.21.15.67:42606) 2024-03-24T00:49:12.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.754+0000 7fccde94e640 1 -- 172.21.15.67:0/2470946692 learned_addr learned my addr 172.21.15.67:0/2470946692 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:12.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.754+0000 7fccdd94c640 1 --2- 172.21.15.67:0/2470946692 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56546d10cf80 0x56546d195a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:12.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.754+0000 7fccde14d640 1 --2- 172.21.15.67:0/2470946692 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56546d0f20c0 0x56546d1969f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:12.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.754+0000 7fccde94e640 1 -- 172.21.15.67:0/2470946692 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56546d0f20c0 msgr2=0x56546d1969f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:12.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.754+0000 7fccde94e640 1 --2- 172.21.15.67:0/2470946692 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56546d0f20c0 0x56546d1969f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:12.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.754+0000 7fccde94e640 1 -- 172.21.15.67:0/2470946692 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56546d10cf80 msgr2=0x56546d195a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:12.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.754+0000 7fccde94e640 1 --2- 172.21.15.67:0/2470946692 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56546d10cf80 0x56546d195a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:12.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.754+0000 7fccde94e640 1 -- 172.21.15.67:0/2470946692 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56546d194bf0 con 0x56546d10d360 2024-03-24T00:49:12.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.754+0000 7fccdd94c640 1 --2- 172.21.15.67:0/2470946692 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56546d10cf80 0x56546d195a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:49:12.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.754+0000 7fccde94e640 1 --2- 172.21.15.67:0/2470946692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56546d10d360 0x56546d195fd0 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7fccd801bdb0 tx=0x7fccd804c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f092264153d65c56 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:12.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.755+0000 7fccdc94a640 1 -- 172.21.15.67:0/2470946692 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd8061070 con 0x56546d10d360 2024-03-24T00:49:12.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.755+0000 7fccdc94a640 1 -- 172.21.15.67:0/2470946692 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccd8052930 con 0x56546d10d360 2024-03-24T00:49:12.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.755+0000 7fccdc94a640 1 -- 172.21.15.67:0/2470946692 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd8057040 con 0x56546d10d360 2024-03-24T00:49:12.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.755+0000 7fcce0e41f40 1 -- 172.21.15.67:0/2470946692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56546d10d360 msgr2=0x56546d195fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:12.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.755+0000 7fcce0e41f40 1 --2- 172.21.15.67:0/2470946692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56546d10d360 0x56546d195fd0 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7fccd801bdb0 tx=0x7fccd804c000 comp rx=0 tx=0).stop 2024-03-24T00:49:12.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.755+0000 7fcce0e41f40 1 -- 172.21.15.67:0/2470946692 shutdown_connections 2024-03-24T00:49:12.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.755+0000 7fcce0e41f40 1 --2- 172.21.15.67:0/2470946692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56546d10d360 0x56546d195fd0 unknown :-1 s=CLOSED pgs=1778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:12.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.755+0000 7fcce0e41f40 1 --2- 172.21.15.67:0/2470946692 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56546d10cf80 0x56546d195a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:12.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.755+0000 7fcce0e41f40 1 --2- 172.21.15.67:0/2470946692 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56546d0f20c0 0x56546d1969f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:12.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.755+0000 7fcce0e41f40 1 -- 172.21.15.67:0/2470946692 >> 172.21.15.67:0/2470946692 conn(0x56546d0c3650 msgr2=0x56546d195320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:12.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.755+0000 7fcce0e41f40 1 -- 172.21.15.67:0/2470946692 shutdown_connections 2024-03-24T00:49:12.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.755+0000 7fcce0e41f40 1 -- 172.21.15.67:0/2470946692 wait complete. 2024-03-24T00:49:12.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.756+0000 7fcce0e41f40 1 Processor -- start 2024-03-24T00:49:12.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.756+0000 7fcce0e41f40 1 -- start start 2024-03-24T00:49:12.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.756+0000 7fcce0e41f40 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56546d0f20c0 0x56546d1864e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:12.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.756+0000 7fcce0e41f40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56546d10cf80 0x56546d10aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:12.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.756+0000 7fcce0e41f40 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56546d10d360 0x56546d10afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:12.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.756+0000 7fcce0e41f40 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x56546d194eb0 con 0x56546d10d360 2024-03-24T00:49:12.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.756+0000 7fcce0e41f40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x56546d198880 con 0x56546d10cf80 2024-03-24T00:49:12.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.756+0000 7fcce0e41f40 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x56546d109d70 con 0x56546d0f20c0 2024-03-24T00:49:12.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.756+0000 7fccde94e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56546d10d360 0x56546d10afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:12.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.756+0000 7fccde94e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56546d10d360 0x56546d10afa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42614/0 (socket says 172.21.15.67:42614) 2024-03-24T00:49:12.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.756+0000 7fccde94e640 1 -- 172.21.15.67:0/672141441 learned_addr learned my addr 172.21.15.67:0/672141441 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:12.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.756+0000 7fccdd94c640 1 --2- 172.21.15.67:0/672141441 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56546d10cf80 0x56546d10aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:12.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.757+0000 7fccde14d640 1 --2- 172.21.15.67:0/672141441 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56546d0f20c0 0x56546d1864e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:12.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.757+0000 7fccde94e640 1 -- 172.21.15.67:0/672141441 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56546d0f20c0 msgr2=0x56546d1864e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:12.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.757+0000 7fccde94e640 1 --2- 172.21.15.67:0/672141441 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56546d0f20c0 0x56546d1864e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:12.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.757+0000 7fccde94e640 1 -- 172.21.15.67:0/672141441 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56546d10cf80 msgr2=0x56546d10aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:12.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.757+0000 7fccde94e640 1 --2- 172.21.15.67:0/672141441 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56546d10cf80 0x56546d10aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:12.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.757+0000 7fccde94e640 1 -- 172.21.15.67:0/672141441 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56546d194690 con 0x56546d10d360 2024-03-24T00:49:12.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.757+0000 7fccde94e640 1 --2- 172.21.15.67:0/672141441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56546d10d360 0x56546d10afa0 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7fccd801cce0 tx=0x7fccd8003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:12.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.757+0000 7fcccf7fe640 1 -- 172.21.15.67:0/672141441 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd8061070 con 0x56546d10d360 2024-03-24T00:49:12.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.757+0000 7fcccf7fe640 1 -- 172.21.15.67:0/672141441 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccd8062040 con 0x56546d10d360 2024-03-24T00:49:12.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.757+0000 7fcce0e41f40 1 -- 172.21.15.67:0/672141441 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x56546d1849f0 con 0x56546d10d360 2024-03-24T00:49:12.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.757+0000 7fcccf7fe640 1 -- 172.21.15.67:0/672141441 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd80688c0 con 0x56546d10d360 2024-03-24T00:49:12.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.757+0000 7fcce0e41f40 1 -- 172.21.15.67:0/672141441 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56546d184710 con 0x56546d10d360 2024-03-24T00:49:12.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.758+0000 7fcccf7fe640 1 -- 172.21.15.67:0/672141441 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fccd8050070 con 0x56546d10d360 2024-03-24T00:49:12.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.759+0000 7fcccf7fe640 1 --2- 172.21.15.67:0/672141441 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fccc006a5b0 0x7fccc006ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:12.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.759+0000 7fcccf7fe640 1 -- 172.21.15.67:0/672141441 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fccd80cc970 con 0x56546d10d360 2024-03-24T00:49:12.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.759+0000 7fccde14d640 1 --2- 172.21.15.67:0/672141441 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fccc006a5b0 0x7fccc006ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:12.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.759+0000 7fcce0e41f40 1 --2- 172.21.15.67:0/672141441 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x56546d182130 0x56546d183560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:12.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.759+0000 7fcce0e41f40 1 -- 172.21.15.67:0/672141441 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [omap-rm-keys in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) v9 -- 0x56546d183aa0 con 0x56546d182130 2024-03-24T00:49:12.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.759+0000 7fccdd94c640 1 --2- 172.21.15.67:0/672141441 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x56546d182130 0x56546d183560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:12.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.760+0000 7fccde14d640 1 --2- 172.21.15.67:0/672141441 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fccc006a5b0 0x7fccc006ca70 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x56546d193d70 tx=0x7fccd0003870 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:12.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.760+0000 7fccdd94c640 1 --2- 172.21.15.67:0/672141441 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x56546d182130 0x56546d183560 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:12.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.761+0000 7fccdd94c640 1 -- 172.21.15.67:0/672141441 <== osd.0 v2:172.21.15.67:6808/2628130519 1 ==== osd_op_reply(1 10000000001.00000000 [omap-rm-keys] v121'5 uv5 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7fccd40099c0 con 0x56546d182130 2024-03-24T00:49:12.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.761+0000 7fcce0e41f40 1 -- 172.21.15.67:0/672141441 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x56546d182130 msgr2=0x56546d183560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:12.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.761+0000 7fcce0e41f40 1 --2- 172.21.15.67:0/672141441 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x56546d182130 0x56546d183560 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:12.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.761+0000 7fcce0e41f40 1 -- 172.21.15.67:0/672141441 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fccc006a5b0 msgr2=0x7fccc006ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:12.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.761+0000 7fcce0e41f40 1 --2- 172.21.15.67:0/672141441 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fccc006a5b0 0x7fccc006ca70 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x56546d193d70 tx=0x7fccd0003870 comp rx=0 tx=0).stop 2024-03-24T00:49:12.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.761+0000 7fcce0e41f40 1 -- 172.21.15.67:0/672141441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56546d10d360 msgr2=0x56546d10afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:12.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.761+0000 7fcce0e41f40 1 --2- 172.21.15.67:0/672141441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56546d10d360 0x56546d10afa0 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7fccd801cce0 tx=0x7fccd8003330 comp rx=0 tx=0).stop 2024-03-24T00:49:12.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.761+0000 7fccde94e640 1 -- 172.21.15.67:0/672141441 reap_dead start 2024-03-24T00:49:12.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.761+0000 7fcce0e41f40 1 -- 172.21.15.67:0/672141441 shutdown_connections 2024-03-24T00:49:12.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.761+0000 7fcce0e41f40 1 -- 172.21.15.67:0/672141441 >> 172.21.15.67:0/672141441 conn(0x56546d0c3650 msgr2=0x56546d195320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:12.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.761+0000 7fcce0e41f40 1 -- 172.21.15.67:0/672141441 shutdown_connections 2024-03-24T00:49:12.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.761+0000 7fcce0e41f40 1 -- 172.21.15.67:0/672141441 wait complete. 2024-03-24T00:49:12.767 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-03-24T00:49:12.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.910+0000 7f66537b2640 1 Processor -- start 2024-03-24T00:49:12.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.910+0000 7f66537b2640 1 -- start start 2024-03-24T00:49:12.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.911+0000 7f66537b2640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f664c0cd730 0x7f664c0b30e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:12.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.911+0000 7f66537b2640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f664c0b1d00 0x7f664c0cdb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:12.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.911+0000 7f66537b2640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f664c0cc170 con 0x7f664c15abc0 2024-03-24T00:49:12.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.911+0000 7f66537b2640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f664c0cca10 con 0x7f664c0cd730 2024-03-24T00:49:12.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.911+0000 7f66537b2640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f664c0cc450 con 0x7f664c0b1d00 2024-03-24T00:49:12.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.911+0000 7f665154e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f664c0cd730 0x7f664c0b30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:12.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.911+0000 7f665154e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f664c0cd730 0x7f664c0b30e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42628/0 (socket says 172.21.15.67:42628) 2024-03-24T00:49:12.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.911+0000 7f665154e640 1 -- 172.21.15.67:0/2039313661 learned_addr learned my addr 172.21.15.67:0/2039313661 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:12.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.911+0000 7f6650d4d640 1 --2- 172.21.15.67:0/2039313661 >> v2:172.21.15.72:3300/0 conn(0x7f664c0b1d00 0x7f664c0cdb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:12.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.911+0000 7f665154e640 1 -- 172.21.15.67:0/2039313661 >> v2:172.21.15.72:3300/0 conn(0x7f664c0b1d00 msgr2=0x7f664c0cdb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:12.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.911+0000 7f665154e640 1 --2- 172.21.15.67:0/2039313661 >> v2:172.21.15.72:3300/0 conn(0x7f664c0b1d00 0x7f664c0cdb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:12.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.911+0000 7f665154e640 1 -- 172.21.15.67:0/2039313661 >> v1:172.21.15.72:6789/0 conn(0x7f664c15abc0 legacy=0x7f664c15b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:12.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.911+0000 7f665154e640 1 -- 172.21.15.67:0/2039313661 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f664c0cc730 con 0x7f664c0cd730 2024-03-24T00:49:12.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.911+0000 7f6650d4d640 1 --2- 172.21.15.67:0/2039313661 >> v2:172.21.15.72:3300/0 conn(0x7f664c0b1d00 0x7f664c0cdb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:49:12.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.912+0000 7f665154e640 1 --2- 172.21.15.67:0/2039313661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f664c0cd730 0x7f664c0b30e0 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f664c0b1ba0 tx=0x7f664402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f1812fbceaedf94 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:12.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.912+0000 7f664bfff640 1 -- 172.21.15.67:0/2039313661 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f664402ca80 con 0x7f664c0cd730 2024-03-24T00:49:12.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.912+0000 7f664bfff640 1 -- 172.21.15.67:0/2039313661 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f664402cc20 con 0x7f664c0cd730 2024-03-24T00:49:12.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.912+0000 7f664bfff640 1 -- 172.21.15.67:0/2039313661 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66440343f0 con 0x7f664c0cd730 2024-03-24T00:49:12.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.913+0000 7f66537b2640 1 -- 172.21.15.67:0/2039313661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f664c0cd730 msgr2=0x7f664c0b30e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:12.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.913+0000 7f66537b2640 1 --2- 172.21.15.67:0/2039313661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f664c0cd730 0x7f664c0b30e0 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f664c0b1ba0 tx=0x7f664402bee0 comp rx=0 tx=0).stop 2024-03-24T00:49:12.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.913+0000 7f66537b2640 1 -- 172.21.15.67:0/2039313661 shutdown_connections 2024-03-24T00:49:12.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.913+0000 7f66537b2640 1 --2- 172.21.15.67:0/2039313661 >> v2:172.21.15.72:3300/0 conn(0x7f664c0b1d00 0x7f664c0cdb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:12.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.913+0000 7f66537b2640 1 --2- 172.21.15.67:0/2039313661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f664c0cd730 0x7f664c0b30e0 unknown :-1 s=CLOSED pgs=1780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:12.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.913+0000 7f66537b2640 1 -- 172.21.15.67:0/2039313661 >> 172.21.15.67:0/2039313661 conn(0x7f664c0b25c0 msgr2=0x7f664c0b29d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:12.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.913+0000 7f66537b2640 1 -- 172.21.15.67:0/2039313661 shutdown_connections 2024-03-24T00:49:12.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.913+0000 7f66537b2640 1 -- 172.21.15.67:0/2039313661 wait complete. 2024-03-24T00:49:12.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.914+0000 7f66537b2640 1 Processor -- start 2024-03-24T00:49:12.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.914+0000 7f66537b2640 1 -- start start 2024-03-24T00:49:12.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.915+0000 7f66537b2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f664c0b1d00 0x7f664c155160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:12.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.915+0000 7f66537b2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f664c0cd730 0x7f664c0c6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:12.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.915+0000 7f66537b2640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f664c15abc0 0x7f664c1f1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:12.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.915+0000 7f66537b2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f664c06b0c0 con 0x7f664c0cd730 2024-03-24T00:49:12.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.915+0000 7f66537b2640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f664c0ce390 con 0x7f664c0b1d00 2024-03-24T00:49:12.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.915+0000 7f66537b2640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f664c06b240 con 0x7f664c15abc0 2024-03-24T00:49:12.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.915+0000 7f6650d4d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f664c0cd730 0x7f664c0c6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:12.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.915+0000 7f6650d4d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f664c0cd730 0x7f664c0c6730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42642/0 (socket says 172.21.15.67:42642) 2024-03-24T00:49:12.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.915+0000 7f6650d4d640 1 -- 172.21.15.67:0/1165134292 learned_addr learned my addr 172.21.15.67:0/1165134292 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:12.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.915+0000 7f665154e640 1 --2- 172.21.15.67:0/1165134292 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f664c0b1d00 0x7f664c155160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:12.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.915+0000 7f6651d4f640 1 --2- 172.21.15.67:0/1165134292 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f664c15abc0 0x7f664c1f1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:12.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.915+0000 7f6650d4d640 1 -- 172.21.15.67:0/1165134292 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f664c15abc0 msgr2=0x7f664c1f1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:12.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.915+0000 7f6650d4d640 1 --2- 172.21.15.67:0/1165134292 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f664c15abc0 0x7f664c1f1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:12.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.915+0000 7f6650d4d640 1 -- 172.21.15.67:0/1165134292 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f664c0b1d00 msgr2=0x7f664c155160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:12.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.915+0000 7f6650d4d640 1 --2- 172.21.15.67:0/1165134292 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f664c0b1d00 0x7f664c155160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:12.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.915+0000 7f6650d4d640 1 -- 172.21.15.67:0/1165134292 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f664c1556a0 con 0x7f664c0cd730 2024-03-24T00:49:12.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.915+0000 7f665154e640 1 --2- 172.21.15.67:0/1165134292 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f664c0b1d00 0x7f664c155160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:49:12.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.915+0000 7f6650d4d640 1 --2- 172.21.15.67:0/1165134292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f664c0cd730 0x7f664c0c6730 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f664000e730 tx=0x7f664000ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:12.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.915+0000 7f6649ffb640 1 -- 172.21.15.67:0/1165134292 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f664000f040 con 0x7f664c0cd730 2024-03-24T00:49:12.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.916+0000 7f6649ffb640 1 -- 172.21.15.67:0/1165134292 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6640004230 con 0x7f664c0cd730 2024-03-24T00:49:12.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.916+0000 7f66537b2640 1 -- 172.21.15.67:0/1165134292 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f664c0c9290 con 0x7f664c0cd730 2024-03-24T00:49:12.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.916+0000 7f6649ffb640 1 -- 172.21.15.67:0/1165134292 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f664000ee40 con 0x7f664c0cd730 2024-03-24T00:49:12.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.916+0000 7f66537b2640 1 -- 172.21.15.67:0/1165134292 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f664c0c9850 con 0x7f664c0cd730 2024-03-24T00:49:12.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.917+0000 7f66537b2640 1 -- 172.21.15.67:0/1165134292 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f664c04f400 con 0x7f664c0cd730 2024-03-24T00:49:12.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.917+0000 7f6649ffb640 1 -- 172.21.15.67:0/1165134292 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f664000c6d0 con 0x7f664c0cd730 2024-03-24T00:49:12.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.917+0000 7f6649ffb640 1 --2- 172.21.15.67:0/1165134292 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f663806bbd0 0x7f663806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:12.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.917+0000 7f6649ffb640 1 -- 172.21.15.67:0/1165134292 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f664008a950 con 0x7f664c0cd730 2024-03-24T00:49:12.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.918+0000 7f665154e640 1 --2- 172.21.15.67:0/1165134292 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f663806bbd0 0x7f663806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:12.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.918+0000 7f665154e640 1 --2- 172.21.15.67:0/1165134292 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f663806bbd0 0x7f663806e090 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f6644035000 tx=0x7f6644030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:12.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:12.922+0000 7f6649ffb640 1 -- 172.21.15.67:0/1165134292 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6640017020 con 0x7f664c0cd730 2024-03-24T00:49:13.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.148+0000 7f66537b2640 1 -- 172.21.15.67:0/1165134292 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) v1 -- 0x7f664c0ceee0 con 0x7f664c0cd730 2024-03-24T00:49:13.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.467+0000 7f6649ffb640 1 -- 172.21.15.67:0/1165134292 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v88) v1 ==== 162+0+0 (secure 0 0 0) 0x7f6640059260 con 0x7f664c0cd730 2024-03-24T00:49:13.469 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked joinable; MDS may join as newly active. 2024-03-24T00:49:13.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.470+0000 7f66537b2640 1 -- 172.21.15.67:0/1165134292 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f663806bbd0 msgr2=0x7f663806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:13.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.470+0000 7f66537b2640 1 --2- 172.21.15.67:0/1165134292 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f663806bbd0 0x7f663806e090 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f6644035000 tx=0x7f6644030000 comp rx=0 tx=0).stop 2024-03-24T00:49:13.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.470+0000 7f66537b2640 1 -- 172.21.15.67:0/1165134292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f664c0cd730 msgr2=0x7f664c0c6730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:13.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.470+0000 7f66537b2640 1 --2- 172.21.15.67:0/1165134292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f664c0cd730 0x7f664c0c6730 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f664000e730 tx=0x7f664000ebf0 comp rx=0 tx=0).stop 2024-03-24T00:49:13.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.470+0000 7f66537b2640 1 -- 172.21.15.67:0/1165134292 shutdown_connections 2024-03-24T00:49:13.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.470+0000 7f66537b2640 1 --2- 172.21.15.67:0/1165134292 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f663806bbd0 0x7f663806e090 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:13.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.470+0000 7f66537b2640 1 --2- 172.21.15.67:0/1165134292 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f664c15abc0 0x7f664c1f1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:13.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.470+0000 7f66537b2640 1 --2- 172.21.15.67:0/1165134292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f664c0cd730 0x7f664c0c6730 unknown :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:13.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.470+0000 7f66537b2640 1 --2- 172.21.15.67:0/1165134292 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f664c0b1d00 0x7f664c155160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:13.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.470+0000 7f66537b2640 1 -- 172.21.15.67:0/1165134292 >> 172.21.15.67:0/1165134292 conn(0x7f664c0b25c0 msgr2=0x7f664c0ccc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:13.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.470+0000 7f66537b2640 1 -- 172.21.15.67:0/1165134292 shutdown_connections 2024-03-24T00:49:13.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.470+0000 7f66537b2640 1 -- 172.21.15.67:0/1165134292 wait complete. 2024-03-24T00:49:13.485 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:49:13.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.587+0000 7fd34ac85640 1 Processor -- start 2024-03-24T00:49:13.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.587+0000 7fd34ac85640 1 -- start start 2024-03-24T00:49:13.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.588+0000 7fd34ac85640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd3440cd4b0 0x7fd3440b2b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:13.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.588+0000 7fd34ac85640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd3440b14c0 0x7fd3440b1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:13.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.588+0000 7fd34ac85640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3440cdd50 con 0x7fd344099d00 2024-03-24T00:49:13.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.588+0000 7fd34ac85640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3440d0030 con 0x7fd3440b14c0 2024-03-24T00:49:13.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.588+0000 7fd34ac85640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3440cda70 con 0x7fd3440cd4b0 2024-03-24T00:49:13.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.588+0000 7fd349222640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd3440b14c0 0x7fd3440b1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:13.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.588+0000 7fd348a21640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd344099d00 0x7fd34409a0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43684/0 (socket says 172.21.15.67:43684) 2024-03-24T00:49:13.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.588+0000 7fd348a21640 1 -- 172.21.15.67:0/1771655767 learned_addr learned my addr 172.21.15.67:0/1771655767 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:49:13.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.588+0000 7fd343fff640 1 --2- 172.21.15.67:0/1771655767 >> v2:172.21.15.72:3300/0 conn(0x7fd3440cd4b0 0x7fd3440b2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:13.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.588+0000 7fd349222640 1 -- 172.21.15.67:0/1771655767 >> v2:172.21.15.72:3300/0 conn(0x7fd3440cd4b0 msgr2=0x7fd3440b2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:13.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.588+0000 7fd349222640 1 --2- 172.21.15.67:0/1771655767 >> v2:172.21.15.72:3300/0 conn(0x7fd3440cd4b0 0x7fd3440b2b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:13.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.588+0000 7fd349222640 1 -- 172.21.15.67:0/1771655767 >> v1:172.21.15.67:6789/0 conn(0x7fd344099d00 legacy=0x7fd34409a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:13.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.588+0000 7fd3437fe640 1 -- 172.21.15.67:0/1771655767 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2853605118 0 0) 0x7fd3440cdd50 con 0x7fd344099d00 2024-03-24T00:49:13.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.588+0000 7fd349222640 1 -- 172.21.15.67:0/1771655767 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3440cf9e0 con 0x7fd3440b14c0 2024-03-24T00:49:13.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.589+0000 7fd349222640 1 --2- 172.21.15.67:0/1771655767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3440b14c0 0x7fd3440b1950 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7fd34406f5a0 tx=0x7fd33c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bc5000995f7e180 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:13.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.589+0000 7fd3437fe640 1 -- 172.21.15.67:0/1771655767 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33c039070 con 0x7fd3440b14c0 2024-03-24T00:49:13.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.589+0000 7fd3437fe640 1 -- 172.21.15.67:0/1771655767 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd33c02cc80 con 0x7fd3440b14c0 2024-03-24T00:49:13.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.589+0000 7fd3437fe640 1 -- 172.21.15.67:0/1771655767 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33c034660 con 0x7fd3440b14c0 2024-03-24T00:49:13.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.590+0000 7fd34ac85640 1 -- 172.21.15.67:0/1771655767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3440b14c0 msgr2=0x7fd3440b1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:13.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.590+0000 7fd34ac85640 1 --2- 172.21.15.67:0/1771655767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3440b14c0 0x7fd3440b1950 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7fd34406f5a0 tx=0x7fd33c02bed0 comp rx=0 tx=0).stop 2024-03-24T00:49:13.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.590+0000 7fd34ac85640 1 -- 172.21.15.67:0/1771655767 shutdown_connections 2024-03-24T00:49:13.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.590+0000 7fd34ac85640 1 --2- 172.21.15.67:0/1771655767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3440b14c0 0x7fd3440b1950 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:13.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.590+0000 7fd34ac85640 1 --2- 172.21.15.67:0/1771655767 >> v2:172.21.15.72:3300/0 conn(0x7fd3440cd4b0 0x7fd3440b2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:13.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.590+0000 7fd34ac85640 1 -- 172.21.15.67:0/1771655767 >> 172.21.15.67:0/1771655767 conn(0x7fd3440b3650 msgr2=0x7fd3440b3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:13.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.590+0000 7fd34ac85640 1 -- 172.21.15.67:0/1771655767 shutdown_connections 2024-03-24T00:49:13.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.590+0000 7fd34ac85640 1 -- 172.21.15.67:0/1771655767 wait complete. 2024-03-24T00:49:13.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.591+0000 7fd34ac85640 1 Processor -- start 2024-03-24T00:49:13.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.591+0000 7fd34ac85640 1 -- start start 2024-03-24T00:49:13.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.592+0000 7fd34ac85640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd344099d00 0x7fd3440b4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:13.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.592+0000 7fd34ac85640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd3440b14c0 0x7fd3440b7300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:13.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.592+0000 7fd34ac85640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3440cd4b0 0x7fd3441534d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:13.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.592+0000 7fd34ac85640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3440b1160 con 0x7fd3440cd4b0 2024-03-24T00:49:13.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.592+0000 7fd34ac85640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd34409ace0 con 0x7fd344099d00 2024-03-24T00:49:13.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.592+0000 7fd34ac85640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3440d04c0 con 0x7fd3440b14c0 2024-03-24T00:49:13.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.592+0000 7fd349222640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3440cd4b0 0x7fd3441534d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:13.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.592+0000 7fd349222640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3440cd4b0 0x7fd3441534d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46358/0 (socket says 172.21.15.67:46358) 2024-03-24T00:49:13.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.592+0000 7fd349222640 1 -- 172.21.15.67:0/234272820 learned_addr learned my addr 172.21.15.67:0/234272820 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:13.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.592+0000 7fd348a21640 1 --2- 172.21.15.67:0/234272820 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd344099d00 0x7fd3440b4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:13.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.592+0000 7fd343fff640 1 --2- 172.21.15.67:0/234272820 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd3440b14c0 0x7fd3440b7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:13.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.592+0000 7fd349222640 1 -- 172.21.15.67:0/234272820 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd3440b14c0 msgr2=0x7fd3440b7300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:13.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.592+0000 7fd349222640 1 --2- 172.21.15.67:0/234272820 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd3440b14c0 0x7fd3440b7300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:13.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.592+0000 7fd349222640 1 -- 172.21.15.67:0/234272820 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd344099d00 msgr2=0x7fd3440b4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:13.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.592+0000 7fd349222640 1 --2- 172.21.15.67:0/234272820 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd344099d00 0x7fd3440b4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:13.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.592+0000 7fd349222640 1 -- 172.21.15.67:0/234272820 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd344153a10 con 0x7fd3440cd4b0 2024-03-24T00:49:13.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.592+0000 7fd343fff640 1 --2- 172.21.15.67:0/234272820 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd3440b14c0 0x7fd3440b7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:49:13.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.593+0000 7fd349222640 1 --2- 172.21.15.67:0/234272820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3440cd4b0 0x7fd3441534d0 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7fd33c037040 tx=0x7fd33c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:13.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.593+0000 7fd3417fa640 1 -- 172.21.15.67:0/234272820 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33c044020 con 0x7fd3440cd4b0 2024-03-24T00:49:13.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.593+0000 7fd3417fa640 1 -- 172.21.15.67:0/234272820 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd33c03a040 con 0x7fd3440cd4b0 2024-03-24T00:49:13.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.593+0000 7fd34ac85640 1 -- 172.21.15.67:0/234272820 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3440b4df0 con 0x7fd3440cd4b0 2024-03-24T00:49:13.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.593+0000 7fd3417fa640 1 -- 172.21.15.67:0/234272820 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33c039050 con 0x7fd3440cd4b0 2024-03-24T00:49:13.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.593+0000 7fd34ac85640 1 -- 172.21.15.67:0/234272820 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3440b6b00 con 0x7fd3440cd4b0 2024-03-24T00:49:13.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.594+0000 7fd34ac85640 1 -- 172.21.15.67:0/234272820 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd34404f400 con 0x7fd3440cd4b0 2024-03-24T00:49:13.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.594+0000 7fd3417fa640 1 -- 172.21.15.67:0/234272820 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd33c003b90 con 0x7fd3440cd4b0 2024-03-24T00:49:13.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.595+0000 7fd3417fa640 1 --2- 172.21.15.67:0/234272820 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd32806bbd0 0x7fd32806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:13.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.595+0000 7fd3417fa640 1 -- 172.21.15.67:0/234272820 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fd33c0ad3c0 con 0x7fd3440cd4b0 2024-03-24T00:49:13.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.595+0000 7fd348a21640 1 --2- 172.21.15.67:0/234272820 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd32806bbd0 0x7fd32806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:13.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.595+0000 7fd348a21640 1 --2- 172.21.15.67:0/234272820 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd32806bbd0 0x7fd32806e090 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fd338000f60 tx=0x7fd338007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:13.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.599+0000 7fd3417fa640 1 -- 172.21.15.67:0/234272820 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd33c04e020 con 0x7fd3440cd4b0 2024-03-24T00:49:13.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.823+0000 7fd34ac85640 1 -- 172.21.15.67:0/234272820 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd344070a10 con 0x7fd3440cd4b0 2024-03-24T00:49:13.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.824+0000 7fd3417fa640 1 -- 172.21.15.67:0/234272820 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+4971 (secure 0 0 0) 0x7fd33c07bd60 con 0x7fd3440cd4b0 2024-03-24T00:49:13.825 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:13.825 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":89,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7298,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/839277571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":839277571},{"type":"v1","addr":"172.21.15.72:6836","nonce":839277571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":8224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2052380377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2052380377},{"type":"v1","addr":"172.21.15.67:6834","nonce":2052380377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":8239,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2196618204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2196618204},{"type":"v1","addr":"172.21.15.72:6837","nonce":2196618204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:48:54.351575+0000","modified":"2024-03-24T00:49:13.471352+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":121,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9510},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9510":{"gid":9510,"name":"a","rank":0,"incarnation":89,"state":"up:replay","state_seq":1,"addr":"172.21.15.67:6835/3833808901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":3833808901},{"type":"v1","addr":"172.21.15.67:6835","nonce":3833808901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":8}]} 2024-03-24T00:49:13.825 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 89 2024-03-24T00:49:13.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.826+0000 7fd34ac85640 1 -- 172.21.15.67:0/234272820 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd32806bbd0 msgr2=0x7fd32806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:13.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.826+0000 7fd34ac85640 1 --2- 172.21.15.67:0/234272820 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd32806bbd0 0x7fd32806e090 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fd338000f60 tx=0x7fd338007d50 comp rx=0 tx=0).stop 2024-03-24T00:49:13.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.826+0000 7fd34ac85640 1 -- 172.21.15.67:0/234272820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3440cd4b0 msgr2=0x7fd3441534d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:13.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.826+0000 7fd34ac85640 1 --2- 172.21.15.67:0/234272820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3440cd4b0 0x7fd3441534d0 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7fd33c037040 tx=0x7fd33c004a40 comp rx=0 tx=0).stop 2024-03-24T00:49:13.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.826+0000 7fd34ac85640 1 -- 172.21.15.67:0/234272820 shutdown_connections 2024-03-24T00:49:13.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.826+0000 7fd34ac85640 1 --2- 172.21.15.67:0/234272820 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd32806bbd0 0x7fd32806e090 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:13.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.826+0000 7fd34ac85640 1 --2- 172.21.15.67:0/234272820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3440cd4b0 0x7fd3441534d0 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:13.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.826+0000 7fd34ac85640 1 --2- 172.21.15.67:0/234272820 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd3440b14c0 0x7fd3440b7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:13.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.826+0000 7fd34ac85640 1 --2- 172.21.15.67:0/234272820 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd344099d00 0x7fd3440b4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:13.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.826+0000 7fd34ac85640 1 -- 172.21.15.67:0/234272820 >> 172.21.15.67:0/234272820 conn(0x7fd3440b3650 msgr2=0x7fd34409a0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:13.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.826+0000 7fd34ac85640 1 -- 172.21.15.67:0/234272820 shutdown_connections 2024-03-24T00:49:13.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:13.826+0000 7fd34ac85640 1 -- 172.21.15.67:0/234272820 wait complete. 2024-03-24T00:49:14.241 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:48:54.351575+0000', 'modified': '2024-03-24T00:49:13.471352+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 121, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9510}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9510': {'gid': 9510, 'name': 'a', 'rank': 0, 'incarnation': 89, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.67:6835/3833808901', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6833', 'nonce': 3833808901}, {'type': 'v1', 'addr': '172.21.15.67:6835', 'nonce': 3833808901}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-24T00:49:14.241 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9510:up:replay 2024-03-24T00:49:15.242 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:49:15.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.356+0000 7f884bfc4640 1 Processor -- start 2024-03-24T00:49:15.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.356+0000 7f884bfc4640 1 -- start start 2024-03-24T00:49:15.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.357+0000 7f884bfc4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f884415bd60 0x7f88440cc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:15.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.357+0000 7f884bfc4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f88440cdda0 0x7f88440d0600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:15.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.357+0000 7f884bfc4640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88440ccd00 con 0x7f8844099d70 2024-03-24T00:49:15.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.357+0000 7f884bfc4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88440cfa50 con 0x7f88440cdda0 2024-03-24T00:49:15.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.357+0000 7f884bfc4640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88440ccfe0 con 0x7f884415bd60 2024-03-24T00:49:15.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.357+0000 7f884a561640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8844099d70 0x7f884415f8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43698/0 (socket says 172.21.15.67:43698) 2024-03-24T00:49:15.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.357+0000 7f884a561640 1 -- 172.21.15.67:0/1929767257 learned_addr learned my addr 172.21.15.67:0/1929767257 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:49:15.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.357+0000 7f8849d60640 1 --2- 172.21.15.67:0/1929767257 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f884415bd60 0x7f88440cc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:15.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.358+0000 7f884955f640 1 --2- 172.21.15.67:0/1929767257 >> v2:172.21.15.72:3300/0 conn(0x7f88440cdda0 0x7f88440d0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:15.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.358+0000 7f8848d5e640 1 -- 172.21.15.67:0/1929767257 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3612726162 0 0) 0x7f88440ccd00 con 0x7f8844099d70 2024-03-24T00:49:15.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.358+0000 7f8848d5e640 1 -- 172.21.15.67:0/1929767257 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8830003520 con 0x7f8844099d70 2024-03-24T00:49:15.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.358+0000 7f8849d60640 1 -- 172.21.15.67:0/1929767257 >> v2:172.21.15.72:3300/0 conn(0x7f88440cdda0 msgr2=0x7f88440d0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:15.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.358+0000 7f8849d60640 1 --2- 172.21.15.67:0/1929767257 >> v2:172.21.15.72:3300/0 conn(0x7f88440cdda0 0x7f88440d0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:15.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.358+0000 7f8849d60640 1 -- 172.21.15.67:0/1929767257 >> v1:172.21.15.67:6789/0 conn(0x7f8844099d70 legacy=0x7f884415f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:15.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.358+0000 7f8849d60640 1 -- 172.21.15.67:0/1929767257 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88440cd2c0 con 0x7f884415bd60 2024-03-24T00:49:15.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.358+0000 7f884955f640 1 --2- 172.21.15.67:0/1929767257 >> v2:172.21.15.72:3300/0 conn(0x7f88440cdda0 0x7f88440d0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:49:15.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.359+0000 7f8849d60640 1 --2- 172.21.15.67:0/1929767257 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f884415bd60 0x7f88440cc6d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f883c0097a0 tx=0x7f883c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30557a75365ff9b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:15.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.359+0000 7f8848d5e640 1 -- 172.21.15.67:0/1929767257 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f883c035020 con 0x7f884415bd60 2024-03-24T00:49:15.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.359+0000 7f8848d5e640 1 -- 172.21.15.67:0/1929767257 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f883c031830 con 0x7f884415bd60 2024-03-24T00:49:15.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.359+0000 7f8848d5e640 1 -- 172.21.15.67:0/1929767257 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f883c031c00 con 0x7f884415bd60 2024-03-24T00:49:15.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.359+0000 7f884bfc4640 1 -- 172.21.15.67:0/1929767257 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f884415bd60 msgr2=0x7f88440cc6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:15.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.359+0000 7f884bfc4640 1 --2- 172.21.15.67:0/1929767257 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f884415bd60 0x7f88440cc6d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f883c0097a0 tx=0x7f883c02b8d0 comp rx=0 tx=0).stop 2024-03-24T00:49:15.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.359+0000 7f884bfc4640 1 -- 172.21.15.67:0/1929767257 shutdown_connections 2024-03-24T00:49:15.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.359+0000 7f884bfc4640 1 --2- 172.21.15.67:0/1929767257 >> v2:172.21.15.72:3300/0 conn(0x7f88440cdda0 0x7f88440d0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:15.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.360+0000 7f884bfc4640 1 --2- 172.21.15.67:0/1929767257 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f884415bd60 0x7f88440cc6d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:15.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.360+0000 7f884bfc4640 1 -- 172.21.15.67:0/1929767257 >> 172.21.15.67:0/1929767257 conn(0x7f88440b3640 msgr2=0x7f88440b3a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:15.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.360+0000 7f884bfc4640 1 -- 172.21.15.67:0/1929767257 shutdown_connections 2024-03-24T00:49:15.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.360+0000 7f884bfc4640 1 -- 172.21.15.67:0/1929767257 wait complete. 2024-03-24T00:49:15.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.361+0000 7f884bfc4640 1 Processor -- start 2024-03-24T00:49:15.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.361+0000 7f884bfc4640 1 -- start start 2024-03-24T00:49:15.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.362+0000 7f884bfc4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8844099d70 0x7f88440c7240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:15.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.362+0000 7f884bfc4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88440cdda0 0x7f88440b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:15.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.362+0000 7f884bfc4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f884415bd60 0x7f884415a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:15.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.362+0000 7f884bfc4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f884415c140 con 0x7f88440cdda0 2024-03-24T00:49:15.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.362+0000 7f884bfc4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88440cd880 con 0x7f8844099d70 2024-03-24T00:49:15.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.362+0000 7f884bfc4640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88440d0b40 con 0x7f884415bd60 2024-03-24T00:49:15.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.362+0000 7f884955f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88440cdda0 0x7f88440b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:15.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.362+0000 7f884955f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88440cdda0 0x7f88440b49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46366/0 (socket says 172.21.15.67:46366) 2024-03-24T00:49:15.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.362+0000 7f884955f640 1 -- 172.21.15.67:0/1546391215 learned_addr learned my addr 172.21.15.67:0/1546391215 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:15.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.362+0000 7f8849d60640 1 --2- 172.21.15.67:0/1546391215 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8844099d70 0x7f88440c7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:15.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.362+0000 7f884a561640 1 --2- 172.21.15.67:0/1546391215 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f884415bd60 0x7f884415a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:15.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.362+0000 7f884955f640 1 -- 172.21.15.67:0/1546391215 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f884415bd60 msgr2=0x7f884415a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:15.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.362+0000 7f884955f640 1 --2- 172.21.15.67:0/1546391215 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f884415bd60 0x7f884415a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:15.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.362+0000 7f884955f640 1 -- 172.21.15.67:0/1546391215 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8844099d70 msgr2=0x7f88440c7240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:15.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.362+0000 7f884955f640 1 --2- 172.21.15.67:0/1546391215 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8844099d70 0x7f88440c7240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:15.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.362+0000 7f884955f640 1 -- 172.21.15.67:0/1546391215 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88440b50d0 con 0x7f88440cdda0 2024-03-24T00:49:15.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.362+0000 7f8849d60640 1 --2- 172.21.15.67:0/1546391215 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8844099d70 0x7f88440c7240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:49:15.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.362+0000 7f884955f640 1 --2- 172.21.15.67:0/1546391215 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88440cdda0 0x7f88440b49a0 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f883400ebe0 tx=0x7f883400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:15.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.362+0000 7f883a7fc640 1 -- 172.21.15.67:0/1546391215 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8834017070 con 0x7f88440cdda0 2024-03-24T00:49:15.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.362+0000 7f883a7fc640 1 -- 172.21.15.67:0/1546391215 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f883400cc30 con 0x7f88440cdda0 2024-03-24T00:49:15.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.362+0000 7f884bfc4640 1 -- 172.21.15.67:0/1546391215 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88440b53b0 con 0x7f88440cdda0 2024-03-24T00:49:15.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.362+0000 7f883a7fc640 1 -- 172.21.15.67:0/1546391215 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8834013470 con 0x7f88440cdda0 2024-03-24T00:49:15.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.362+0000 7f884bfc4640 1 -- 172.21.15.67:0/1546391215 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88440b5970 con 0x7f88440cdda0 2024-03-24T00:49:15.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.363+0000 7f884bfc4640 1 -- 172.21.15.67:0/1546391215 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f884404f400 con 0x7f88440cdda0 2024-03-24T00:49:15.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.364+0000 7f883a7fc640 1 -- 172.21.15.67:0/1546391215 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f883401a070 con 0x7f88440cdda0 2024-03-24T00:49:15.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.364+0000 7f883a7fc640 1 --2- 172.21.15.67:0/1546391215 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f882806bbd0 0x7f882806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:15.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.364+0000 7f883a7fc640 1 -- 172.21.15.67:0/1546391215 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f883408aff0 con 0x7f88440cdda0 2024-03-24T00:49:15.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.364+0000 7f8849d60640 1 --2- 172.21.15.67:0/1546391215 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f882806bbd0 0x7f882806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:15.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.365+0000 7f8849d60640 1 --2- 172.21.15.67:0/1546391215 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f882806bbd0 0x7f882806e090 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f883c036000 tx=0x7f883c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:15.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.369+0000 7f883a7fc640 1 -- 172.21.15.67:0/1546391215 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8834059870 con 0x7f88440cdda0 2024-03-24T00:49:15.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.590+0000 7f884bfc4640 1 -- 172.21.15.67:0/1546391215 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f884409a8a0 con 0x7f88440cdda0 2024-03-24T00:49:15.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.591+0000 7f883a7fc640 1 -- 172.21.15.67:0/1546391215 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+4971 (secure 0 0 0) 0x7f8834059170 con 0x7f88440cdda0 2024-03-24T00:49:15.592 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:15.592 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":91,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7298,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/839277571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":839277571},{"type":"v1","addr":"172.21.15.72:6836","nonce":839277571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":8224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2052380377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2052380377},{"type":"v1","addr":"172.21.15.67:6834","nonce":2052380377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":8239,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2196618204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2196618204},{"type":"v1","addr":"172.21.15.72:6837","nonce":2196618204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:48:54.351575+0000","modified":"2024-03-24T00:49:14.484223+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":121,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9510},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9510":{"gid":9510,"name":"a","rank":0,"incarnation":89,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.67:6835/3833808901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":3833808901},{"type":"v1","addr":"172.21.15.67:6835","nonce":3833808901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":8}]} 2024-03-24T00:49:15.592 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 91 2024-03-24T00:49:15.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.592+0000 7f884bfc4640 1 -- 172.21.15.67:0/1546391215 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f882806bbd0 msgr2=0x7f882806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:15.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.592+0000 7f884bfc4640 1 --2- 172.21.15.67:0/1546391215 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f882806bbd0 0x7f882806e090 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f883c036000 tx=0x7f883c008030 comp rx=0 tx=0).stop 2024-03-24T00:49:15.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.592+0000 7f884bfc4640 1 -- 172.21.15.67:0/1546391215 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88440cdda0 msgr2=0x7f88440b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:15.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.592+0000 7f884bfc4640 1 --2- 172.21.15.67:0/1546391215 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88440cdda0 0x7f88440b49a0 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f883400ebe0 tx=0x7f883400c520 comp rx=0 tx=0).stop 2024-03-24T00:49:15.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.592+0000 7f884bfc4640 1 -- 172.21.15.67:0/1546391215 shutdown_connections 2024-03-24T00:49:15.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.592+0000 7f884bfc4640 1 --2- 172.21.15.67:0/1546391215 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f882806bbd0 0x7f882806e090 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:15.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.592+0000 7f884bfc4640 1 --2- 172.21.15.67:0/1546391215 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f884415bd60 0x7f884415a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:15.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.592+0000 7f884bfc4640 1 --2- 172.21.15.67:0/1546391215 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88440cdda0 0x7f88440b49a0 unknown :-1 s=CLOSED pgs=1786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:15.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.592+0000 7f884bfc4640 1 --2- 172.21.15.67:0/1546391215 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8844099d70 0x7f88440c7240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:15.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.592+0000 7f884bfc4640 1 -- 172.21.15.67:0/1546391215 >> 172.21.15.67:0/1546391215 conn(0x7f88440b3640 msgr2=0x7f8844070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:15.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.592+0000 7f884bfc4640 1 -- 172.21.15.67:0/1546391215 shutdown_connections 2024-03-24T00:49:15.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:15.593+0000 7f884bfc4640 1 -- 172.21.15.67:0/1546391215 wait complete. 2024-03-24T00:49:16.007 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 91, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:48:54.351575+0000', 'modified': '2024-03-24T00:49:14.484223+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 121, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9510}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9510': {'gid': 9510, 'name': 'a', 'rank': 0, 'incarnation': 89, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.67:6835/3833808901', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6833', 'nonce': 3833808901}, {'type': 'v1', 'addr': '172.21.15.67:6835', 'nonce': 3833808901}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-24T00:49:16.007 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9510:up:rejoin 2024-03-24T00:49:17.008 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:49:17.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.109+0000 7f1dcbab2640 1 Processor -- start 2024-03-24T00:49:17.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.109+0000 7f1dcbab2640 1 -- start start 2024-03-24T00:49:17.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.110+0000 7f1dcbab2640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1dc4099d00 0x7f1dc40b2b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:17.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.110+0000 7f1dcbab2640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1dc40cd4b0 0x7f1dc409a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:17.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.110+0000 7f1dcbab2640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1dc40b14c0 0x7f1dc40b1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:17.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.110+0000 7f1dcbab2640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dc40d0030 con 0x7f1dc40cd4b0 2024-03-24T00:49:17.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.110+0000 7f1dcbab2640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dc409a620 con 0x7f1dc4099d00 2024-03-24T00:49:17.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.110+0000 7f1dcbab2640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc40cdd50 con 0x7f1dc40b14c0 2024-03-24T00:49:17.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.110+0000 7f1dc904d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1dc40cd4b0 0x7f1dc409a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:17.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.110+0000 7f1dc904d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1dc40cd4b0 0x7f1dc409a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46370/0 (socket says 172.21.15.67:46370) 2024-03-24T00:49:17.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.110+0000 7f1dc904d640 1 -- 172.21.15.67:0/3126908763 learned_addr learned my addr 172.21.15.67:0/3126908763 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:17.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.110+0000 7f1dca04f640 1 --2- 172.21.15.67:0/3126908763 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1dc40b14c0 0x7f1dc40b1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:17.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.110+0000 7f1dc984e640 1 --2- 172.21.15.67:0/3126908763 >> v2:172.21.15.72:3300/0 conn(0x7f1dc4099d00 0x7f1dc40b2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:17.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.110+0000 7f1dc904d640 1 -- 172.21.15.67:0/3126908763 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1dc40b14c0 msgr2=0x7f1dc40b1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:17.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.110+0000 7f1dc904d640 1 --2- 172.21.15.67:0/3126908763 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1dc40b14c0 0x7f1dc40b1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:17.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.110+0000 7f1dc904d640 1 -- 172.21.15.67:0/3126908763 >> v2:172.21.15.72:3300/0 conn(0x7f1dc4099d00 msgr2=0x7f1dc40b2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:17.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.110+0000 7f1dc904d640 1 --2- 172.21.15.67:0/3126908763 >> v2:172.21.15.72:3300/0 conn(0x7f1dc4099d00 0x7f1dc40b2b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:17.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.110+0000 7f1dc904d640 1 -- 172.21.15.67:0/3126908763 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dc40cda70 con 0x7f1dc40cd4b0 2024-03-24T00:49:17.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.110+0000 7f1dc984e640 1 --2- 172.21.15.67:0/3126908763 >> v2:172.21.15.72:3300/0 conn(0x7f1dc4099d00 0x7f1dc40b2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:49:17.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.111+0000 7f1dc904d640 1 --2- 172.21.15.67:0/3126908763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1dc40cd4b0 0x7f1dc409a0e0 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f1dc40cf660 tx=0x7f1db402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78cff99fd8dccc6b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:17.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.111+0000 7f1dc884c640 1 -- 172.21.15.67:0/3126908763 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db4035020 con 0x7f1dc40cd4b0 2024-03-24T00:49:17.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.111+0000 7f1dc884c640 1 -- 172.21.15.67:0/3126908763 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1db4031870 con 0x7f1dc40cd4b0 2024-03-24T00:49:17.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.111+0000 7f1dc884c640 1 -- 172.21.15.67:0/3126908763 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db4031c40 con 0x7f1dc40cd4b0 2024-03-24T00:49:17.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.112+0000 7f1dcbab2640 1 -- 172.21.15.67:0/3126908763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1dc40cd4b0 msgr2=0x7f1dc409a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:17.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.112+0000 7f1dcbab2640 1 --2- 172.21.15.67:0/3126908763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1dc40cd4b0 0x7f1dc409a0e0 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f1dc40cf660 tx=0x7f1db402b910 comp rx=0 tx=0).stop 2024-03-24T00:49:17.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.112+0000 7f1dcbab2640 1 -- 172.21.15.67:0/3126908763 shutdown_connections 2024-03-24T00:49:17.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.112+0000 7f1dcbab2640 1 --2- 172.21.15.67:0/3126908763 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1dc40b14c0 0x7f1dc40b1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:17.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.112+0000 7f1dcbab2640 1 --2- 172.21.15.67:0/3126908763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1dc40cd4b0 0x7f1dc409a0e0 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:17.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.112+0000 7f1dcbab2640 1 --2- 172.21.15.67:0/3126908763 >> v2:172.21.15.72:3300/0 conn(0x7f1dc4099d00 0x7f1dc40b2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:17.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.112+0000 7f1dcbab2640 1 -- 172.21.15.67:0/3126908763 >> 172.21.15.67:0/3126908763 conn(0x7f1dc40b3650 msgr2=0x7f1dc40b3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:17.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.112+0000 7f1dcbab2640 1 -- 172.21.15.67:0/3126908763 shutdown_connections 2024-03-24T00:49:17.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.112+0000 7f1dcbab2640 1 -- 172.21.15.67:0/3126908763 wait complete. 2024-03-24T00:49:17.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.113+0000 7f1dcbab2640 1 Processor -- start 2024-03-24T00:49:17.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.113+0000 7f1dcbab2640 1 -- start start 2024-03-24T00:49:17.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.114+0000 7f1dcbab2640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1dc4099d00 0x7f1dc4155130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:17.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.114+0000 7f1dcbab2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1dc40b14c0 0x7f1dc40c8c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:17.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.114+0000 7f1dcbab2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1dc40cd4b0 0x7f1dc406d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:17.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.114+0000 7f1dcbab2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc409ace0 con 0x7f1dc40cd4b0 2024-03-24T00:49:17.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.114+0000 7f1dcbab2640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc40cfcc0 con 0x7f1dc40b14c0 2024-03-24T00:49:17.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.114+0000 7f1dcbab2640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc40b1160 con 0x7f1dc4099d00 2024-03-24T00:49:17.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.114+0000 7f1dca04f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1dc40cd4b0 0x7f1dc406d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:17.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.114+0000 7f1dca04f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1dc40cd4b0 0x7f1dc406d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46376/0 (socket says 172.21.15.67:46376) 2024-03-24T00:49:17.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.114+0000 7f1dca04f640 1 -- 172.21.15.67:0/588032418 learned_addr learned my addr 172.21.15.67:0/588032418 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:17.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.114+0000 7f1dc904d640 1 --2- 172.21.15.67:0/588032418 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1dc40b14c0 0x7f1dc40c8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:17.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.114+0000 7f1dc984e640 1 --2- 172.21.15.67:0/588032418 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1dc4099d00 0x7f1dc4155130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:17.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.114+0000 7f1dca04f640 1 -- 172.21.15.67:0/588032418 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1dc4099d00 msgr2=0x7f1dc4155130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:17.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.114+0000 7f1dca04f640 1 --2- 172.21.15.67:0/588032418 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1dc4099d00 0x7f1dc4155130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:17.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.114+0000 7f1dca04f640 1 -- 172.21.15.67:0/588032418 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1dc40b14c0 msgr2=0x7f1dc40c8c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:17.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.114+0000 7f1dca04f640 1 --2- 172.21.15.67:0/588032418 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1dc40b14c0 0x7f1dc40c8c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:17.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.114+0000 7f1dca04f640 1 -- 172.21.15.67:0/588032418 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dc4155670 con 0x7f1dc40cd4b0 2024-03-24T00:49:17.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.114+0000 7f1dc904d640 1 --2- 172.21.15.67:0/588032418 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1dc40b14c0 0x7f1dc40c8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:49:17.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.115+0000 7f1dca04f640 1 --2- 172.21.15.67:0/588032418 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1dc40cd4b0 0x7f1dc406d2e0 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f1dc0009d30 tx=0x7f1dc000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:17.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.115+0000 7f1dba7fc640 1 -- 172.21.15.67:0/588032418 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dc0018020 con 0x7f1dc40cd4b0 2024-03-24T00:49:17.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.115+0000 7f1dba7fc640 1 -- 172.21.15.67:0/588032418 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dc0014480 con 0x7f1dc40cd4b0 2024-03-24T00:49:17.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.115+0000 7f1dcbab2640 1 -- 172.21.15.67:0/588032418 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dc40ca200 con 0x7f1dc40cd4b0 2024-03-24T00:49:17.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.115+0000 7f1dba7fc640 1 -- 172.21.15.67:0/588032418 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dc0014870 con 0x7f1dc40cd4b0 2024-03-24T00:49:17.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.115+0000 7f1dcbab2640 1 -- 172.21.15.67:0/588032418 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dc40c9960 con 0x7f1dc40cd4b0 2024-03-24T00:49:17.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.116+0000 7f1dcbab2640 1 -- 172.21.15.67:0/588032418 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dc404f400 con 0x7f1dc40cd4b0 2024-03-24T00:49:17.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.116+0000 7f1dba7fc640 1 -- 172.21.15.67:0/588032418 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1dc0005d10 con 0x7f1dc40cd4b0 2024-03-24T00:49:17.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.116+0000 7f1dba7fc640 1 --2- 172.21.15.67:0/588032418 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1da806bbd0 0x7f1da806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:17.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.117+0000 7f1dba7fc640 1 -- 172.21.15.67:0/588032418 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f1dc008b6b0 con 0x7f1dc40cd4b0 2024-03-24T00:49:17.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.117+0000 7f1dc984e640 1 --2- 172.21.15.67:0/588032418 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1da806bbd0 0x7f1da806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:17.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.117+0000 7f1dc984e640 1 --2- 172.21.15.67:0/588032418 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1da806bbd0 0x7f1da806e090 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc006fd0 tx=0x7f1dbc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:17.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.125+0000 7f1dba7fc640 1 -- 172.21.15.67:0/588032418 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1dc0022070 con 0x7f1dc40cd4b0 2024-03-24T00:49:17.212 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:49:17.214 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:49:17.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.358+0000 7f1dcbab2640 1 -- 172.21.15.67:0/588032418 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1dc406c650 con 0x7f1dc40cd4b0 2024-03-24T00:49:17.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.358+0000 7f1dba7fc640 1 -- 172.21.15.67:0/588032418 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+4978 (secure 0 0 0) 0x7f1dc0089e20 con 0x7f1dc40cd4b0 2024-03-24T00:49:17.360 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:17.360 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":92,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7298,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/839277571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":839277571},{"type":"v1","addr":"172.21.15.72:6836","nonce":839277571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":8224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2052380377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2052380377},{"type":"v1","addr":"172.21.15.67:6834","nonce":2052380377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":8239,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2196618204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2196618204},{"type":"v1","addr":"172.21.15.72:6837","nonce":2196618204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:48:54.351575+0000","modified":"2024-03-24T00:49:16.485067+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":121,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9510},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9510":{"gid":9510,"name":"a","rank":0,"incarnation":89,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6835/3833808901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":3833808901},{"type":"v1","addr":"172.21.15.67:6835","nonce":3833808901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9510,"qdb_cluster":[9510]},"id":8}]} 2024-03-24T00:49:17.360 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 92 2024-03-24T00:49:17.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.361+0000 7f1dcbab2640 1 -- 172.21.15.67:0/588032418 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1da806bbd0 msgr2=0x7f1da806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:17.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.361+0000 7f1dcbab2640 1 --2- 172.21.15.67:0/588032418 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1da806bbd0 0x7f1da806e090 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc006fd0 tx=0x7f1dbc008040 comp rx=0 tx=0).stop 2024-03-24T00:49:17.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.361+0000 7f1dcbab2640 1 -- 172.21.15.67:0/588032418 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1dc40cd4b0 msgr2=0x7f1dc406d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:17.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.361+0000 7f1dcbab2640 1 --2- 172.21.15.67:0/588032418 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1dc40cd4b0 0x7f1dc406d2e0 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f1dc0009d30 tx=0x7f1dc000c490 comp rx=0 tx=0).stop 2024-03-24T00:49:17.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.361+0000 7f1dcbab2640 1 -- 172.21.15.67:0/588032418 shutdown_connections 2024-03-24T00:49:17.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.361+0000 7f1dcbab2640 1 --2- 172.21.15.67:0/588032418 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1da806bbd0 0x7f1da806e090 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:17.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.361+0000 7f1dcbab2640 1 --2- 172.21.15.67:0/588032418 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1dc40cd4b0 0x7f1dc406d2e0 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:17.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.361+0000 7f1dcbab2640 1 --2- 172.21.15.67:0/588032418 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1dc40b14c0 0x7f1dc40c8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:17.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.361+0000 7f1dcbab2640 1 --2- 172.21.15.67:0/588032418 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1dc4099d00 0x7f1dc4155130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:17.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.361+0000 7f1dcbab2640 1 -- 172.21.15.67:0/588032418 >> 172.21.15.67:0/588032418 conn(0x7f1dc40b3650 msgr2=0x7f1dc406cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:17.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.361+0000 7f1dcbab2640 1 -- 172.21.15.67:0/588032418 shutdown_connections 2024-03-24T00:49:17.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.361+0000 7f1dcbab2640 1 -- 172.21.15.67:0/588032418 wait complete. 2024-03-24T00:49:17.778 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 92, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:48:54.351575+0000', 'modified': '2024-03-24T00:49:16.485067+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 121, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9510}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9510': {'gid': 9510, 'name': 'a', 'rank': 0, 'incarnation': 89, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.67:6835/3833808901', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6833', 'nonce': 3833808901}, {'type': 'v1', 'addr': '172.21.15.67:6835', 'nonce': 3833808901}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9510, 'qdb_cluster': [9510]} 2024-03-24T00:49:17.779 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-24T00:49:17.779 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-24T00:49:17.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.880+0000 7f6011eb8640 1 Processor -- start 2024-03-24T00:49:17.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.880+0000 7f6011eb8640 1 -- start start 2024-03-24T00:49:17.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.881+0000 7f6011eb8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f600c15dfc0 0x7f600c161bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:17.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.881+0000 7f6011eb8640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f600c0d10b0 0x7f600c0ceb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:17.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.881+0000 7f6011eb8640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f600c162a50 con 0x7f600c0cf050 2024-03-24T00:49:17.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.881+0000 7f6011eb8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f600c162490 con 0x7f600c15dfc0 2024-03-24T00:49:17.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.881+0000 7f6011eb8640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f600c162770 con 0x7f600c0d10b0 2024-03-24T00:49:17.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.881+0000 7f600b7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f600c15dfc0 0x7f600c161bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:17.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.881+0000 7f600b7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f600c15dfc0 0x7f600c161bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46382/0 (socket says 172.21.15.67:46382) 2024-03-24T00:49:17.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.881+0000 7f600b7fe640 1 -- 172.21.15.67:0/368379912 learned_addr learned my addr 172.21.15.67:0/368379912 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:17.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.882+0000 7f600affd640 1 --2- 172.21.15.67:0/368379912 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f600c0d10b0 0x7f600c0ceb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:17.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.882+0000 7f600b7fe640 1 -- 172.21.15.67:0/368379912 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f600c0d10b0 msgr2=0x7f600c0ceb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:17.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.882+0000 7f600b7fe640 1 --2- 172.21.15.67:0/368379912 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f600c0d10b0 0x7f600c0ceb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:17.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.882+0000 7f600b7fe640 1 -- 172.21.15.67:0/368379912 >> v1:172.21.15.72:6789/0 conn(0x7f600c0cf050 legacy=0x7f600c0b57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:17.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.882+0000 7f600b7fe640 1 -- 172.21.15.67:0/368379912 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f600c0d1f40 con 0x7f600c15dfc0 2024-03-24T00:49:17.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.882+0000 7f600b7fe640 1 --2- 172.21.15.67:0/368379912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f600c15dfc0 0x7f600c161bb0 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc009870 tx=0x7f5ffc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9aeb17d71554205 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:17.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.883+0000 7f600a7fc640 1 -- 172.21.15.67:0/368379912 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc034070 con 0x7f600c15dfc0 2024-03-24T00:49:17.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.883+0000 7f600a7fc640 1 -- 172.21.15.67:0/368379912 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ffc030840 con 0x7f600c15dfc0 2024-03-24T00:49:17.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.883+0000 7f600a7fc640 1 -- 172.21.15.67:0/368379912 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc030c10 con 0x7f600c15dfc0 2024-03-24T00:49:17.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.883+0000 7f6011eb8640 1 -- 172.21.15.67:0/368379912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f600c15dfc0 msgr2=0x7f600c161bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:17.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.883+0000 7f6011eb8640 1 --2- 172.21.15.67:0/368379912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f600c15dfc0 0x7f600c161bb0 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc009870 tx=0x7f5ffc02b980 comp rx=0 tx=0).stop 2024-03-24T00:49:17.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.883+0000 7f6011eb8640 1 -- 172.21.15.67:0/368379912 shutdown_connections 2024-03-24T00:49:17.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.883+0000 7f6011eb8640 1 --2- 172.21.15.67:0/368379912 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f600c0d10b0 0x7f600c0ceb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:17.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.883+0000 7f6011eb8640 1 --2- 172.21.15.67:0/368379912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f600c15dfc0 0x7f600c161bb0 unknown :-1 s=CLOSED pgs=1789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:17.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.883+0000 7f6011eb8640 1 -- 172.21.15.67:0/368379912 >> 172.21.15.67:0/368379912 conn(0x7f600c06c930 msgr2=0x7f600c06cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:17.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.883+0000 7f6011eb8640 1 -- 172.21.15.67:0/368379912 shutdown_connections 2024-03-24T00:49:17.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.883+0000 7f6011eb8640 1 -- 172.21.15.67:0/368379912 wait complete. 2024-03-24T00:49:17.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.884+0000 7f6011eb8640 1 Processor -- start 2024-03-24T00:49:17.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.885+0000 7f6011eb8640 1 -- start start 2024-03-24T00:49:17.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.885+0000 7f6011eb8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f600c0cf050 0x7f600c154c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:17.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.885+0000 7f6011eb8640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f600c0d10b0 0x7f600c0ccbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:17.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.885+0000 7f6011eb8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f600c15dfc0 0x7f600c0cd100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:17.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.885+0000 7f6011eb8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f600c162270 con 0x7f600c0cf050 2024-03-24T00:49:17.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.885+0000 7f6011eb8640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f600c1620f0 con 0x7f600c15dfc0 2024-03-24T00:49:17.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.885+0000 7f6011eb8640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f600c15e3a0 con 0x7f600c0d10b0 2024-03-24T00:49:17.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.885+0000 7f600b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f600c0cf050 0x7f600c154c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:17.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.885+0000 7f600b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f600c0cf050 0x7f600c154c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46398/0 (socket says 172.21.15.67:46398) 2024-03-24T00:49:17.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.885+0000 7f600b7fe640 1 -- 172.21.15.67:0/3309024838 learned_addr learned my addr 172.21.15.67:0/3309024838 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:17.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.885+0000 7f600affd640 1 --2- 172.21.15.67:0/3309024838 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f600c0d10b0 0x7f600c0ccbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:17.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.885+0000 7f600bfff640 1 --2- 172.21.15.67:0/3309024838 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f600c15dfc0 0x7f600c0cd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:17.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.886+0000 7f600b7fe640 1 -- 172.21.15.67:0/3309024838 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f600c0d10b0 msgr2=0x7f600c0ccbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:17.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.886+0000 7f600b7fe640 1 --2- 172.21.15.67:0/3309024838 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f600c0d10b0 0x7f600c0ccbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:17.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.886+0000 7f600b7fe640 1 -- 172.21.15.67:0/3309024838 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f600c15dfc0 msgr2=0x7f600c0cd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:17.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.886+0000 7f600b7fe640 1 --2- 172.21.15.67:0/3309024838 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f600c15dfc0 0x7f600c0cd100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:17.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.886+0000 7f600b7fe640 1 -- 172.21.15.67:0/3309024838 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f600c155140 con 0x7f600c0cf050 2024-03-24T00:49:17.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.886+0000 7f600b7fe640 1 --2- 172.21.15.67:0/3309024838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f600c0cf050 0x7f600c154c00 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc000c00 tx=0x7f5ffc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:17.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.886+0000 7f5ff3fff640 1 -- 172.21.15.67:0/3309024838 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc034040 con 0x7f600c0cf050 2024-03-24T00:49:17.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.886+0000 7f5ff3fff640 1 -- 172.21.15.67:0/3309024838 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ffc039d90 con 0x7f600c0cf050 2024-03-24T00:49:17.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.886+0000 7f5ff3fff640 1 -- 172.21.15.67:0/3309024838 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc043610 con 0x7f600c0cf050 2024-03-24T00:49:17.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.886+0000 7f6011eb8640 1 -- 172.21.15.67:0/3309024838 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f600c1f4f40 con 0x7f600c0cf050 2024-03-24T00:49:17.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.886+0000 7f6011eb8640 1 -- 172.21.15.67:0/3309024838 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f600c1f4920 con 0x7f600c0cf050 2024-03-24T00:49:17.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.887+0000 7f5ff3fff640 1 -- 172.21.15.67:0/3309024838 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5ffc04a060 con 0x7f600c0cf050 2024-03-24T00:49:17.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.888+0000 7f5ff3fff640 1 --2- 172.21.15.67:0/3309024838 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5ff806bbd0 0x7f5ff806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:17.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.888+0000 7f5ff3fff640 1 -- 172.21.15.67:0/3309024838 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f5ffc0acbd0 con 0x7f600c0cf050 2024-03-24T00:49:17.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.888+0000 7f600affd640 1 --2- 172.21.15.67:0/3309024838 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5ff806bbd0 0x7f5ff806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:17.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.889+0000 7f600affd640 1 --2- 172.21.15.67:0/3309024838 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5ff806bbd0 0x7f5ff806e090 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f600c161950 tx=0x7f6000008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:17.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.889+0000 7f6014b78740 1 Processor -- start 2024-03-24T00:49:17.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.889+0000 7f6014b78740 1 -- start start 2024-03-24T00:49:17.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.890+0000 7f6014b78740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55828577cc40 0x5582857a3770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:17.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.890+0000 7f6014b78740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5582857a3cb0 0x55828577bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:17.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.890+0000 7f6014b78740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55828577d020 0x55828577c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:17.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.890+0000 7f6014b78740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582856e0c20 con 0x55828577cc40 2024-03-24T00:49:17.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.890+0000 7f6014b78740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x558285689a40 con 0x55828577d020 2024-03-24T00:49:17.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.890+0000 7f6014b78740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5582856dccc0 con 0x5582857a3cb0 2024-03-24T00:49:17.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.890+0000 7f600b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55828577cc40 0x5582857a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:17.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.890+0000 7f600b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55828577cc40 0x5582857a3770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46410/0 (socket says 172.21.15.67:46410) 2024-03-24T00:49:17.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.890+0000 7f600b7fe640 1 -- 172.21.15.67:0/1234627736 learned_addr learned my addr 172.21.15.67:0/1234627736 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:17.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.890+0000 7f600affd640 1 --2- 172.21.15.67:0/1234627736 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5582857a3cb0 0x55828577bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:17.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.890+0000 7f600bfff640 1 --2- 172.21.15.67:0/1234627736 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55828577d020 0x55828577c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:17.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.891+0000 7f600b7fe640 1 -- 172.21.15.67:0/1234627736 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5582857a3cb0 msgr2=0x55828577bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:17.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.891+0000 7f600b7fe640 1 --2- 172.21.15.67:0/1234627736 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5582857a3cb0 0x55828577bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:17.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.891+0000 7f600b7fe640 1 -- 172.21.15.67:0/1234627736 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55828577d020 msgr2=0x55828577c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:17.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.891+0000 7f600b7fe640 1 --2- 172.21.15.67:0/1234627736 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55828577d020 0x55828577c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:17.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.891+0000 7f600b7fe640 1 -- 172.21.15.67:0/1234627736 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5582856abf90 con 0x55828577cc40 2024-03-24T00:49:17.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.891+0000 7f600bfff640 1 --2- 172.21.15.67:0/1234627736 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55828577d020 0x55828577c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:49:17.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.891+0000 7f600b7fe640 1 --2- 172.21.15.67:0/1234627736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55828577cc40 0x5582857a3770 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc006c80 tx=0x7f5ffc07aec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d21d2769a7a8de server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:17.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.891+0000 7f5ff17fa640 1 -- 172.21.15.67:0/1234627736 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc0834f0 con 0x55828577cc40 2024-03-24T00:49:17.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.891+0000 7f5ff17fa640 1 -- 172.21.15.67:0/1234627736 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ffc078ea0 con 0x55828577cc40 2024-03-24T00:49:17.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.891+0000 7f5ff17fa640 1 -- 172.21.15.67:0/1234627736 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc077470 con 0x55828577cc40 2024-03-24T00:49:17.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.892+0000 7f6014b78740 1 -- 172.21.15.67:0/1234627736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55828577cc40 msgr2=0x5582857a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:17.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.892+0000 7f6014b78740 1 --2- 172.21.15.67:0/1234627736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55828577cc40 0x5582857a3770 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc006c80 tx=0x7f5ffc07aec0 comp rx=0 tx=0).stop 2024-03-24T00:49:17.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.892+0000 7f6014b78740 1 -- 172.21.15.67:0/1234627736 shutdown_connections 2024-03-24T00:49:17.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.892+0000 7f6014b78740 1 --2- 172.21.15.67:0/1234627736 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55828577d020 0x55828577c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:17.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.892+0000 7f6014b78740 1 --2- 172.21.15.67:0/1234627736 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5582857a3cb0 0x55828577bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:17.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.892+0000 7f6014b78740 1 --2- 172.21.15.67:0/1234627736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55828577cc40 0x5582857a3770 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:17.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.892+0000 7f6014b78740 1 -- 172.21.15.67:0/1234627736 >> 172.21.15.67:0/1234627736 conn(0x558285743a60 msgr2=0x5582857c1960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:17.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.892+0000 7f6014b78740 1 -- 172.21.15.67:0/1234627736 shutdown_connections 2024-03-24T00:49:17.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.892+0000 7f6014b78740 1 -- 172.21.15.67:0/1234627736 wait complete. 2024-03-24T00:49:17.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.892+0000 7f6014b78740 1 -- start start 2024-03-24T00:49:17.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.892+0000 7f6014b78740 1 Processor -- start 2024-03-24T00:49:17.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.893+0000 7f6014b78740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55828577cc40 0x5582857aa140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:17.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.893+0000 7f6014b78740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55828577d020 0x5582857bfba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:17.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.893+0000 7f6014b78740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558285790140 0x5582857c4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:17.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.893+0000 7f6014b78740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x558285689a40 con 0x558285790140 2024-03-24T00:49:17.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.893+0000 7f6014b78740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55828569ce90 con 0x55828577cc40 2024-03-24T00:49:17.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.893+0000 7f6014b78740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5582856e0c20 con 0x55828577d020 2024-03-24T00:49:17.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.893+0000 7f6014b78740 10 client.0 _ll_register_callbacks cb 0x5582853f4ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:49:17.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.893+0000 7f600bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558285790140 0x5582857c4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:17.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.893+0000 7f600bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558285790140 0x5582857c4a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46426/0 (socket says 172.21.15.67:46426) 2024-03-24T00:49:17.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.893+0000 7f600bfff640 1 -- 172.21.15.67:0/1589124554 learned_addr learned my addr 172.21.15.67:0/1589124554 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:17.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.893+0000 7f600b7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55828577cc40 0x5582857aa140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:17.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.893+0000 7f600affd640 1 --2- 172.21.15.67:0/1589124554 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55828577d020 0x5582857bfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:17.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.894+0000 7f600bfff640 1 -- 172.21.15.67:0/1589124554 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55828577d020 msgr2=0x5582857bfba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:17.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.894+0000 7f600bfff640 1 --2- 172.21.15.67:0/1589124554 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55828577d020 0x5582857bfba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:17.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.894+0000 7f600bfff640 1 -- 172.21.15.67:0/1589124554 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55828577cc40 msgr2=0x5582857aa140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:17.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.894+0000 7f600bfff640 1 --2- 172.21.15.67:0/1589124554 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55828577cc40 0x5582857aa140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:17.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.894+0000 7f600bfff640 1 -- 172.21.15.67:0/1589124554 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5582856a99d0 con 0x558285790140 2024-03-24T00:49:17.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.894+0000 7f600bfff640 1 --2- 172.21.15.67:0/1589124554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558285790140 0x5582857c4a00 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f6004002080 tx=0x7f6004005420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55e9b906ef3f6fbf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:17.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.894+0000 7f5fea7fc640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:49:17.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.894+0000 7f5fea7fc640 1 -- 172.21.15.67:0/1589124554 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6004013600 con 0x558285790140 2024-03-24T00:49:17.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.894+0000 7f5fea7fc640 1 -- 172.21.15.67:0/1589124554 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(121..121 src has 1..121) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f60040137a0 con 0x558285790140 2024-03-24T00:49:17.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.894+0000 7f6014b78740 1 -- 172.21.15.67:0/1589124554 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5ffc078ea0 con 0x558285790140 2024-03-24T00:49:17.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.894+0000 7f5fea7fc640 1 -- 172.21.15.67:0/1589124554 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6004022050 con 0x558285790140 2024-03-24T00:49:17.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.894+0000 7f6014b78740 10 client.9558 fetch_fsmap learned FSMap version 92 2024-03-24T00:49:17.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.894+0000 7f6014b78740 1 -- 172.21.15.67:0/1589124554 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=92,monmap=2+}) v3 -- 0x5582853f52d0 con 0x558285790140 2024-03-24T00:49:17.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.894+0000 7f5fea7fc640 1 -- 172.21.15.67:0/1589124554 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ffc078ea0 con 0x558285790140 2024-03-24T00:49:17.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.894+0000 7f5fea7fc640 1 -- 172.21.15.67:0/1589124554 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 92) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f6004025020 con 0x558285790140 2024-03-24T00:49:17.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.894+0000 7f6014b78740 10 client.9558 fetch_fsmap finished waiting for FSMap version 92 2024-03-24T00:49:17.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.894+0000 7f6014b78740 10 client.9558 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-24T00:49:17.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.894+0000 7f6014b78740 20 client.9558 populate_metadata read hostname 'smithi067' 2024-03-24T00:49:17.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.894+0000 7f6014b78740 1 --2- 172.21.15.67:0/1589124554 >> [v2:172.21.15.67:6833/3833808901,v1:172.21.15.67:6835/3833808901] conn(0x5582857a7d70 0x5582857a8150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:17.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.894+0000 7f6014b78740 4 client.9558 mds_command: new command op to 9510 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:49:17.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.894+0000 7f6014b78740 1 -- 172.21.15.67:0/1589124554 --> [v2:172.21.15.67:6833/3833808901,v1:172.21.15.67:6835/3833808901] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5582856ee160 con 0x5582857a7d70 2024-03-24T00:49:17.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.895+0000 7f600b7fe640 1 --2- 172.21.15.67:0/1589124554 >> [v2:172.21.15.67:6833/3833808901,v1:172.21.15.67:6835/3833808901] conn(0x5582857a7d70 0x5582857a8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:17.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.895+0000 7f600b7fe640 1 --2- 172.21.15.67:0/1589124554 >> [v2:172.21.15.67:6833/3833808901,v1:172.21.15.67:6835/3833808901] conn(0x5582857a7d70 0x5582857a8150 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f6e75ab9c27b88 server_cookie=f81f3c6a34f5c6ef in_seq=0 out_seq=0 2024-03-24T00:49:17.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.895+0000 7f5fea7fc640 10 client.9558 ms_handle_connect on v2:172.21.15.67:6833/3833808901 2024-03-24T00:49:17.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.895+0000 7f5fea7fc640 1 -- 172.21.15.67:0/1589124554 <== mds.0 v2:172.21.15.67:6833/3833808901 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5582856ee160 con 0x5582857a7d70 2024-03-24T00:49:17.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.895+0000 7f5fea7fc640 10 client.9558 handle_command_reply: tid=0 2024-03-24T00:49:17.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.895+0000 7f6014b78740 1 client.9558 shutdown 2024-03-24T00:49:17.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.896+0000 7f6014b78740 1 -- 172.21.15.67:0/1589124554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558285790140 msgr2=0x5582857c4a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:17.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.896+0000 7f6014b78740 1 --2- 172.21.15.67:0/1589124554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558285790140 0x5582857c4a00 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f6004002080 tx=0x7f6004005420 comp rx=0 tx=0).stop 2024-03-24T00:49:17.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.896+0000 7f6014b78740 1 -- 172.21.15.67:0/1589124554 shutdown_connections 2024-03-24T00:49:17.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.896+0000 7f6014b78740 1 --2- 172.21.15.67:0/1589124554 >> [v2:172.21.15.67:6833/3833808901,v1:172.21.15.67:6835/3833808901] conn(0x5582857a7d70 0x5582857a8150 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:17.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.896+0000 7f6014b78740 1 --2- 172.21.15.67:0/1589124554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558285790140 0x5582857c4a00 unknown :-1 s=CLOSED pgs=1792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:17.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.896+0000 7f6014b78740 1 --2- 172.21.15.67:0/1589124554 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55828577d020 0x5582857bfba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:17.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.896+0000 7f5fea7fc640 0 client.9558 ms_handle_reset on v2:172.21.15.67:6833/3833808901 2024-03-24T00:49:17.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.896+0000 7f6014b78740 1 --2- 172.21.15.67:0/1589124554 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55828577cc40 0x5582857aa140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:17.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.896+0000 7f6014b78740 1 -- 172.21.15.67:0/1589124554 >> 172.21.15.67:0/1589124554 conn(0x558285743a60 msgr2=0x5582857a24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:17.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.896+0000 7f6014b78740 1 -- 172.21.15.67:0/1589124554 shutdown_connections 2024-03-24T00:49:17.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.896+0000 7f6014b78740 1 -- 172.21.15.67:0/1589124554 wait complete. 2024-03-24T00:49:17.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.896+0000 7f6014b78740 20 client.9558 trim_cache size 0 max 16384 2024-03-24T00:49:17.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.912+0000 7f6014b78740 1 Processor -- start 2024-03-24T00:49:17.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.913+0000 7f6014b78740 1 -- start start 2024-03-24T00:49:17.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.913+0000 7f6014b78740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55828577cc40 0x5582857a8b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:17.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.913+0000 7f6014b78740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55828577d020 0x5582857a9090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:17.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.913+0000 7f6014b78740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558285790140 0x5582857d0080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:17.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.913+0000 7f6014b78740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f600c1ffc50 con 0x55828577d020 2024-03-24T00:49:17.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.913+0000 7f6014b78740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f600c0c9ec0 con 0x558285790140 2024-03-24T00:49:17.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.913+0000 7f6014b78740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f600c1ff650 con 0x55828577cc40 2024-03-24T00:49:17.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.913+0000 7f600affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55828577d020 0x5582857a9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:17.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.913+0000 7f600affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55828577d020 0x5582857a9090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46434/0 (socket says 172.21.15.67:46434) 2024-03-24T00:49:17.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.913+0000 7f600affd640 1 -- 172.21.15.67:0/171120533 learned_addr learned my addr 172.21.15.67:0/171120533 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:17.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.914+0000 7f600bfff640 1 --2- 172.21.15.67:0/171120533 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558285790140 0x5582857d0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:17.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.914+0000 7f600b7fe640 1 --2- 172.21.15.67:0/171120533 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55828577cc40 0x5582857a8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:17.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.914+0000 7f600affd640 1 -- 172.21.15.67:0/171120533 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55828577cc40 msgr2=0x5582857a8b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:17.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.914+0000 7f600affd640 1 --2- 172.21.15.67:0/171120533 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55828577cc40 0x5582857a8b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:17.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.914+0000 7f600affd640 1 -- 172.21.15.67:0/171120533 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558285790140 msgr2=0x5582857d0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:17.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.914+0000 7f600affd640 1 --2- 172.21.15.67:0/171120533 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558285790140 0x5582857d0080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:17.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.914+0000 7f600affd640 1 -- 172.21.15.67:0/171120533 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5582854bf6b0 con 0x55828577d020 2024-03-24T00:49:17.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.914+0000 7f600bfff640 1 --2- 172.21.15.67:0/171120533 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558285790140 0x5582857d0080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:17.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.914+0000 7f600b7fe640 1 --2- 172.21.15.67:0/171120533 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55828577cc40 0x5582857a8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:17.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.914+0000 7f600affd640 1 --2- 172.21.15.67:0/171120533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55828577d020 0x5582857a9090 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f600c15cac0 tx=0x7f6000013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eae2df82fdf867f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:17.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.914+0000 7f5ff0ff9640 1 -- 172.21.15.67:0/171120533 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6000028070 con 0x55828577d020 2024-03-24T00:49:17.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.914+0000 7f5ff0ff9640 1 -- 172.21.15.67:0/171120533 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f600001d040 con 0x55828577d020 2024-03-24T00:49:17.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.914+0000 7f5ff0ff9640 1 -- 172.21.15.67:0/171120533 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f600001c040 con 0x55828577d020 2024-03-24T00:49:17.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.915+0000 7f6014b78740 1 -- 172.21.15.67:0/171120533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55828577d020 msgr2=0x5582857a9090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:17.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.915+0000 7f6014b78740 1 --2- 172.21.15.67:0/171120533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55828577d020 0x5582857a9090 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f600c15cac0 tx=0x7f6000013000 comp rx=0 tx=0).stop 2024-03-24T00:49:17.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.915+0000 7f6014b78740 1 -- 172.21.15.67:0/171120533 shutdown_connections 2024-03-24T00:49:17.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.915+0000 7f6014b78740 1 --2- 172.21.15.67:0/171120533 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558285790140 0x5582857d0080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:17.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.915+0000 7f6014b78740 1 --2- 172.21.15.67:0/171120533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55828577d020 0x5582857a9090 unknown :-1 s=CLOSED pgs=1793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:17.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.915+0000 7f6014b78740 1 --2- 172.21.15.67:0/171120533 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55828577cc40 0x5582857a8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:17.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.915+0000 7f6014b78740 1 -- 172.21.15.67:0/171120533 >> 172.21.15.67:0/171120533 conn(0x558285743a60 msgr2=0x5582857c03c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:17.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.915+0000 7f6014b78740 1 -- 172.21.15.67:0/171120533 shutdown_connections 2024-03-24T00:49:17.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.915+0000 7f6014b78740 1 -- 172.21.15.67:0/171120533 wait complete. 2024-03-24T00:49:17.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.915+0000 7f6014b78740 1 -- start start 2024-03-24T00:49:17.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.915+0000 7f6014b78740 1 Processor -- start 2024-03-24T00:49:17.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.916+0000 7f6014b78740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55828577cc40 0x5582857b7500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:17.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.916+0000 7f6014b78740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55828577d020 0x5582857b7a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:17.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.916+0000 7f6014b78740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558285790140 0x558285775a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:17.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.916+0000 7f6014b78740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582857c5170 con 0x55828577d020 2024-03-24T00:49:17.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.916+0000 7f6014b78740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582857c0140 con 0x55828577cc40 2024-03-24T00:49:17.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.916+0000 7f6014b78740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5582857d1900 con 0x558285790140 2024-03-24T00:49:17.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.916+0000 7f6014b78740 10 client.0 _ll_register_callbacks cb 0x7f6004003a50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:49:17.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.916+0000 7f600affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55828577d020 0x5582857b7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:17.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.916+0000 7f600affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55828577d020 0x5582857b7a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46448/0 (socket says 172.21.15.67:46448) 2024-03-24T00:49:17.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.916+0000 7f600affd640 1 -- 172.21.15.67:0/2798677072 learned_addr learned my addr 172.21.15.67:0/2798677072 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:17.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.916+0000 7f600bfff640 1 --2- 172.21.15.67:0/2798677072 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558285790140 0x558285775a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:17.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.916+0000 7f600b7fe640 1 --2- 172.21.15.67:0/2798677072 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55828577cc40 0x5582857b7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:17.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.916+0000 7f600affd640 1 -- 172.21.15.67:0/2798677072 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558285790140 msgr2=0x558285775a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:17.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.916+0000 7f600affd640 1 --2- 172.21.15.67:0/2798677072 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558285790140 0x558285775a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:17.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.916+0000 7f600affd640 1 -- 172.21.15.67:0/2798677072 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55828577cc40 msgr2=0x5582857b7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:17.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.917+0000 7f600affd640 1 --2- 172.21.15.67:0/2798677072 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55828577cc40 0x5582857b7500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:17.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.917+0000 7f600affd640 1 -- 172.21.15.67:0/2798677072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5582857a6ec0 con 0x55828577d020 2024-03-24T00:49:17.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.917+0000 7f600b7fe640 1 --2- 172.21.15.67:0/2798677072 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55828577cc40 0x5582857b7500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:49:17.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.917+0000 7f600affd640 1 --2- 172.21.15.67:0/2798677072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55828577d020 0x5582857b7a40 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f6000006fd0 tx=0x7f6000004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b684d05439023e56 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:17.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.917+0000 7f5fea7fc640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:49:17.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.917+0000 7f5fea7fc640 1 -- 172.21.15.67:0/2798677072 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f600001c070 con 0x55828577d020 2024-03-24T00:49:17.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.917+0000 7f5fea7fc640 1 -- 172.21.15.67:0/2798677072 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(121..121 src has 1..121) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f6000024ab0 con 0x55828577d020 2024-03-24T00:49:17.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.917+0000 7f6014b78740 1 -- 172.21.15.67:0/2798677072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f600001d040 con 0x55828577d020 2024-03-24T00:49:17.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.917+0000 7f5fea7fc640 1 -- 172.21.15.67:0/2798677072 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7f600003e050 con 0x55828577d020 2024-03-24T00:49:17.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.917+0000 7f6014b78740 10 client.9564 fetch_fsmap learned FSMap version 92 2024-03-24T00:49:17.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.917+0000 7f6014b78740 1 -- 172.21.15.67:0/2798677072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=92,monmap=2+}) v3 -- 0x5582858312f0 con 0x55828577d020 2024-03-24T00:49:17.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.917+0000 7f5fea7fc640 1 -- 172.21.15.67:0/2798677072 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f600001d040 con 0x55828577d020 2024-03-24T00:49:17.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.917+0000 7f5fea7fc640 1 -- 172.21.15.67:0/2798677072 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 92) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f6000024cb0 con 0x55828577d020 2024-03-24T00:49:17.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.917+0000 7f6014b78740 10 client.9564 fetch_fsmap finished waiting for FSMap version 92 2024-03-24T00:49:17.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.917+0000 7f6014b78740 10 client.9564 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-24T00:49:17.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.917+0000 7f6014b78740 20 client.9564 populate_metadata read hostname 'smithi067' 2024-03-24T00:49:17.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.917+0000 7f6014b78740 1 --2- 172.21.15.67:0/2798677072 >> [v2:172.21.15.67:6833/3833808901,v1:172.21.15.67:6835/3833808901] conn(0x5582857765a0 0x5582858301f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:17.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.917+0000 7f6014b78740 4 client.9564 mds_command: new command op to 9510 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:49:17.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.917+0000 7f6014b78740 1 -- 172.21.15.67:0/2798677072 --> [v2:172.21.15.67:6833/3833808901,v1:172.21.15.67:6835/3833808901] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5582856e84b0 con 0x5582857765a0 2024-03-24T00:49:17.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.917+0000 7f600b7fe640 1 --2- 172.21.15.67:0/2798677072 >> [v2:172.21.15.67:6833/3833808901,v1:172.21.15.67:6835/3833808901] conn(0x5582857765a0 0x5582858301f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:17.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.918+0000 7f5fea7fc640 10 client.9564 ms_handle_connect on v2:172.21.15.67:6833/3833808901 2024-03-24T00:49:17.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.918+0000 7f600b7fe640 1 --2- 172.21.15.67:0/2798677072 >> [v2:172.21.15.67:6833/3833808901,v1:172.21.15.67:6835/3833808901] conn(0x5582857765a0 0x5582858301f0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df4fba13d9477b55 server_cookie=7ea0f3f8f760bb1c in_seq=0 out_seq=0 2024-03-24T00:49:17.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.918+0000 7f5fea7fc640 1 -- 172.21.15.67:0/2798677072 <== mds.0 v2:172.21.15.67:6833/3833808901 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x5582856e84b0 con 0x5582857765a0 2024-03-24T00:49:17.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.918+0000 7f5fea7fc640 10 client.9564 handle_command_reply: tid=0 2024-03-24T00:49:17.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.918+0000 7f6014b78740 1 client.9564 shutdown 2024-03-24T00:49:17.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.918+0000 7f6014b78740 1 -- 172.21.15.67:0/2798677072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55828577d020 msgr2=0x5582857b7a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:17.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.918+0000 7f6014b78740 1 --2- 172.21.15.67:0/2798677072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55828577d020 0x5582857b7a40 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f6000006fd0 tx=0x7f6000004730 comp rx=0 tx=0).stop 2024-03-24T00:49:17.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.918+0000 7f6014b78740 1 -- 172.21.15.67:0/2798677072 shutdown_connections 2024-03-24T00:49:17.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.918+0000 7f6014b78740 1 --2- 172.21.15.67:0/2798677072 >> [v2:172.21.15.67:6833/3833808901,v1:172.21.15.67:6835/3833808901] conn(0x5582857765a0 0x5582858301f0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:17.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.918+0000 7f6014b78740 1 --2- 172.21.15.67:0/2798677072 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x558285790140 0x558285775a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:17.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.918+0000 7f6014b78740 1 --2- 172.21.15.67:0/2798677072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55828577d020 0x5582857b7a40 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:17.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.918+0000 7f5fea7fc640 0 client.9564 ms_handle_reset on v2:172.21.15.67:6833/3833808901 2024-03-24T00:49:17.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.918+0000 7f6014b78740 1 --2- 172.21.15.67:0/2798677072 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55828577cc40 0x5582857b7500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:17.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.918+0000 7f6014b78740 1 -- 172.21.15.67:0/2798677072 >> 172.21.15.67:0/2798677072 conn(0x558285743a60 msgr2=0x5582857a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:17.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.918+0000 7f6014b78740 1 -- 172.21.15.67:0/2798677072 shutdown_connections 2024-03-24T00:49:17.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.918+0000 7f6014b78740 1 -- 172.21.15.67:0/2798677072 wait complete. 2024-03-24T00:49:17.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.918+0000 7f6014b78740 20 client.9564 trim_cache size 0 max 16384 2024-03-24T00:49:17.926 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:49:17.926 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "de1d0178-808e-49ca-a986-e297e523b976", 2024-03-24T00:49:17.926 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-03-24T00:49:17.926 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 9510, 2024-03-24T00:49:17.926 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-03-24T00:49:17.926 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-03-24T00:49:17.926 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-03-24T00:49:17.926 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 4.4433723079999998, 2024-03-24T00:49:17.926 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 92, 2024-03-24T00:49:17.926 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 121, 2024-03-24T00:49:17.926 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 121, 2024-03-24T00:49:17.926 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 5.4117037010000004 2024-03-24T00:49:17.926 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:49:17.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.919+0000 7f6011eb8640 1 -- 172.21.15.67:0/3309024838 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5ff806bbd0 msgr2=0x7f5ff806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:17.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.919+0000 7f6011eb8640 1 --2- 172.21.15.67:0/3309024838 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5ff806bbd0 0x7f5ff806e090 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f600c161950 tx=0x7f6000008040 comp rx=0 tx=0).stop 2024-03-24T00:49:17.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.919+0000 7f6011eb8640 1 -- 172.21.15.67:0/3309024838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f600c0cf050 msgr2=0x7f600c154c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:17.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.919+0000 7f6011eb8640 1 --2- 172.21.15.67:0/3309024838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f600c0cf050 0x7f600c154c00 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc000c00 tx=0x7f5ffc0046c0 comp rx=0 tx=0).stop 2024-03-24T00:49:17.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.919+0000 7f6011eb8640 1 -- 172.21.15.67:0/3309024838 shutdown_connections 2024-03-24T00:49:17.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.919+0000 7f6011eb8640 1 --2- 172.21.15.67:0/3309024838 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5ff806bbd0 0x7f5ff806e090 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:17.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.919+0000 7f6011eb8640 1 --2- 172.21.15.67:0/3309024838 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f600c15dfc0 0x7f600c0cd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:17.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.919+0000 7f6011eb8640 1 --2- 172.21.15.67:0/3309024838 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f600c0d10b0 0x7f600c0ccbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:17.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.919+0000 7f6011eb8640 1 --2- 172.21.15.67:0/3309024838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f600c0cf050 0x7f600c154c00 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:17.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.919+0000 7f6011eb8640 1 -- 172.21.15.67:0/3309024838 >> 172.21.15.67:0/3309024838 conn(0x7f600c06c930 msgr2=0x7f600c0b85d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:17.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.919+0000 7f6011eb8640 1 -- 172.21.15.67:0/3309024838 shutdown_connections 2024-03-24T00:49:17.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:17.919+0000 7f6011eb8640 1 -- 172.21.15.67:0/3309024838 wait complete. 2024-03-24T00:49:17.930 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T00:49:18.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.034+0000 7fb15359e640 1 Processor -- start 2024-03-24T00:49:18.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.034+0000 7fb15359e640 1 -- start start 2024-03-24T00:49:18.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.036+0000 7fb15359e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb1540cf590 0x7fb1540cd8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:18.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.036+0000 7fb15359e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb1540b1f20 0x7fb1540d13c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:18.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.036+0000 7fb15359e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1540b3790 con 0x7fb1540b1b40 2024-03-24T00:49:18.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.036+0000 7fb15359e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1540ce9d0 con 0x7fb1540b1f20 2024-03-24T00:49:18.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.036+0000 7fb15359e640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1540b4360 con 0x7fb1540cf590 2024-03-24T00:49:18.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.036+0000 7fb152d9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb1540b1f20 0x7fb1540d13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:18.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.036+0000 7fb14bfff640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb1540cf590 0x7fb1540cd8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:18.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.036+0000 7fb152d9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb1540b1f20 0x7fb1540d13c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:33588/0 (socket says 172.21.15.67:33588) 2024-03-24T00:49:18.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.036+0000 7fb15259c640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fb1540b1b40 0x7fb1540cee80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:50522/0 (socket says 172.21.15.67:50522) 2024-03-24T00:49:18.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.036+0000 7fb152d9d640 1 -- 172.21.15.67:0/350437566 learned_addr learned my addr 172.21.15.67:0/350437566 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:18.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.036+0000 7fb14bfff640 1 -- 172.21.15.67:0/350437566 >> v2:172.21.15.72:3300/0 conn(0x7fb1540b1f20 msgr2=0x7fb1540d13c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:18.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.036+0000 7fb14bfff640 1 --2- 172.21.15.67:0/350437566 >> v2:172.21.15.72:3300/0 conn(0x7fb1540b1f20 0x7fb1540d13c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:18.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.036+0000 7fb14bfff640 1 -- 172.21.15.67:0/350437566 >> v1:172.21.15.72:6789/0 conn(0x7fb1540b1b40 legacy=0x7fb1540cee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:18.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.036+0000 7fb14bfff640 1 -- 172.21.15.67:0/350437566 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1540b3a30 con 0x7fb1540cf590 2024-03-24T00:49:18.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.036+0000 7fb151d9b640 1 -- 172.21.15.67:0/350437566 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3637115632 0 0) 0x7fb1540b3790 con 0x7fb1540b1b40 2024-03-24T00:49:18.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.037+0000 7fb14bfff640 1 --2- 172.21.15.67:0/350437566 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb1540cf590 0x7fb1540cd8e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb13c0097a0 tx=0x7fb13c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee11fc00052be6a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:18.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.037+0000 7fb151d9b640 1 -- 172.21.15.67:0/350437566 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13c035020 con 0x7fb1540cf590 2024-03-24T00:49:18.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.037+0000 7fb151d9b640 1 -- 172.21.15.67:0/350437566 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb13c031830 con 0x7fb1540cf590 2024-03-24T00:49:18.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.037+0000 7fb151d9b640 1 -- 172.21.15.67:0/350437566 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13c031c00 con 0x7fb1540cf590 2024-03-24T00:49:18.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.038+0000 7fb15359e640 1 -- 172.21.15.67:0/350437566 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb1540cf590 msgr2=0x7fb1540cd8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:18.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.038+0000 7fb15359e640 1 --2- 172.21.15.67:0/350437566 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb1540cf590 0x7fb1540cd8e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb13c0097a0 tx=0x7fb13c02b8d0 comp rx=0 tx=0).stop 2024-03-24T00:49:18.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.038+0000 7fb15359e640 1 -- 172.21.15.67:0/350437566 shutdown_connections 2024-03-24T00:49:18.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.038+0000 7fb15359e640 1 --2- 172.21.15.67:0/350437566 >> v2:172.21.15.72:3300/0 conn(0x7fb1540b1f20 0x7fb1540d13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:18.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.038+0000 7fb15359e640 1 --2- 172.21.15.67:0/350437566 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb1540cf590 0x7fb1540cd8e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:18.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.038+0000 7fb15359e640 1 -- 172.21.15.67:0/350437566 >> 172.21.15.67:0/350437566 conn(0x7fb1540d08a0 msgr2=0x7fb1540d0cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:18.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.038+0000 7fb15359e640 1 -- 172.21.15.67:0/350437566 shutdown_connections 2024-03-24T00:49:18.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.038+0000 7fb15359e640 1 -- 172.21.15.67:0/350437566 wait complete. 2024-03-24T00:49:18.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.039+0000 7fb15359e640 1 Processor -- start 2024-03-24T00:49:18.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.039+0000 7fb15359e640 1 -- start start 2024-03-24T00:49:18.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.040+0000 7fb15359e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb1540b1b40 0x7fb1541f1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:18.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.040+0000 7fb15359e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1540b1f20 0x7fb15406bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:18.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.040+0000 7fb15359e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1540cf590 0x7fb154150130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:18.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.040+0000 7fb15359e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb15415a4f0 con 0x7fb1540cf590 2024-03-24T00:49:18.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.040+0000 7fb15359e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1540b3ff0 con 0x7fb1540b1f20 2024-03-24T00:49:18.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.040+0000 7fb15359e640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb15415a670 con 0x7fb1540b1b40 2024-03-24T00:49:18.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.040+0000 7fb152d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1540cf590 0x7fb154150130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:18.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.040+0000 7fb152d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1540cf590 0x7fb154150130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46462/0 (socket says 172.21.15.67:46462) 2024-03-24T00:49:18.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.040+0000 7fb152d9d640 1 -- 172.21.15.67:0/356966166 learned_addr learned my addr 172.21.15.67:0/356966166 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:18.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.040+0000 7fb15259c640 1 --2- 172.21.15.67:0/356966166 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb1540b1b40 0x7fb1541f1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:18.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.040+0000 7fb14bfff640 1 --2- 172.21.15.67:0/356966166 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1540b1f20 0x7fb15406bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:18.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.040+0000 7fb152d9d640 1 -- 172.21.15.67:0/356966166 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb1540b1b40 msgr2=0x7fb1541f1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:18.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.040+0000 7fb152d9d640 1 --2- 172.21.15.67:0/356966166 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb1540b1b40 0x7fb1541f1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:18.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.040+0000 7fb152d9d640 1 -- 172.21.15.67:0/356966166 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1540b1f20 msgr2=0x7fb15406bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:18.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.040+0000 7fb152d9d640 1 --2- 172.21.15.67:0/356966166 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1540b1f20 0x7fb15406bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:18.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.040+0000 7fb152d9d640 1 -- 172.21.15.67:0/356966166 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb15406e9d0 con 0x7fb1540cf590 2024-03-24T00:49:18.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.040+0000 7fb152d9d640 1 --2- 172.21.15.67:0/356966166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1540cf590 0x7fb154150130 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7fb14c009d90 tx=0x7fb14c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:18.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.041+0000 7fb14affd640 1 -- 172.21.15.67:0/356966166 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb14c00cd20 con 0x7fb1540cf590 2024-03-24T00:49:18.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.041+0000 7fb14affd640 1 -- 172.21.15.67:0/356966166 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb14c003990 con 0x7fb1540cf590 2024-03-24T00:49:18.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.041+0000 7fb15359e640 1 -- 172.21.15.67:0/356966166 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1541f1960 con 0x7fb1540cf590 2024-03-24T00:49:18.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.041+0000 7fb14affd640 1 -- 172.21.15.67:0/356966166 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb14c003d60 con 0x7fb1540cf590 2024-03-24T00:49:18.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.041+0000 7fb15359e640 1 -- 172.21.15.67:0/356966166 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1541f1f20 con 0x7fb1540cf590 2024-03-24T00:49:18.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.042+0000 7fb15359e640 1 -- 172.21.15.67:0/356966166 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1540751c0 con 0x7fb1540cf590 2024-03-24T00:49:18.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.042+0000 7fb14affd640 1 -- 172.21.15.67:0/356966166 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb14c005d10 con 0x7fb1540cf590 2024-03-24T00:49:18.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.043+0000 7fb14affd640 1 --2- 172.21.15.67:0/356966166 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb13406bbd0 0x7fb13406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:18.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.043+0000 7fb14affd640 1 -- 172.21.15.67:0/356966166 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fb14c08b220 con 0x7fb1540cf590 2024-03-24T00:49:18.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.043+0000 7fb15259c640 1 --2- 172.21.15.67:0/356966166 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb13406bbd0 0x7fb13406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:18.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.043+0000 7fb15259c640 1 --2- 172.21.15.67:0/356966166 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb13406bbd0 0x7fb13406e090 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fb144000f60 tx=0x7fb144007b70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:18.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.047+0000 7fb14affd640 1 -- 172.21.15.67:0/356966166 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb14c022020 con 0x7fb1540cf590 2024-03-24T00:49:18.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.268+0000 7fb15359e640 1 -- 172.21.15.67:0/356966166 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fb1540c3ea0 con 0x7fb1540cf590 2024-03-24T00:49:18.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.268+0000 7fb14affd640 1 -- 172.21.15.67:0/356966166 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v92) v1 ==== 53+0+73 (secure 0 0 0) 0x7fb14c010040 con 0x7fb1540cf590 2024-03-24T00:49:18.269 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-24T00:49:18.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.270+0000 7fb15359e640 1 -- 172.21.15.67:0/356966166 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb13406bbd0 msgr2=0x7fb13406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:18.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.270+0000 7fb15359e640 1 --2- 172.21.15.67:0/356966166 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb13406bbd0 0x7fb13406e090 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fb144000f60 tx=0x7fb144007b70 comp rx=0 tx=0).stop 2024-03-24T00:49:18.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.270+0000 7fb15359e640 1 -- 172.21.15.67:0/356966166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1540cf590 msgr2=0x7fb154150130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:18.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.270+0000 7fb15359e640 1 --2- 172.21.15.67:0/356966166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1540cf590 0x7fb154150130 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7fb14c009d90 tx=0x7fb14c00c490 comp rx=0 tx=0).stop 2024-03-24T00:49:18.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.270+0000 7fb15359e640 1 -- 172.21.15.67:0/356966166 shutdown_connections 2024-03-24T00:49:18.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.270+0000 7fb15359e640 1 --2- 172.21.15.67:0/356966166 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb13406bbd0 0x7fb13406e090 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:18.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.270+0000 7fb15359e640 1 --2- 172.21.15.67:0/356966166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1540cf590 0x7fb154150130 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:18.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.270+0000 7fb15359e640 1 --2- 172.21.15.67:0/356966166 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1540b1f20 0x7fb15406bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:18.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.270+0000 7fb15359e640 1 --2- 172.21.15.67:0/356966166 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb1540b1b40 0x7fb1541f1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:18.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.270+0000 7fb15359e640 1 -- 172.21.15.67:0/356966166 >> 172.21.15.67:0/356966166 conn(0x7fb1540d08a0 msgr2=0x7fb154070040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:18.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.270+0000 7fb15359e640 1 -- 172.21.15.67:0/356966166 shutdown_connections 2024-03-24T00:49:18.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:18.270+0000 7fb15359e640 1 -- 172.21.15.67:0/356966166 wait complete. 2024-03-24T00:49:18.282 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-24T00:49:18.282 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T00:49:18.282 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-03-24T00:49:18.282 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-24T00:49:18.282 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T00:49:18.282 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-24T00:49:18.282 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-24T00:49:18.282 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T00:49:18.282 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:49:18.283 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:18.283 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-03-24T00:49:18.297 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T00:49:18.298 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T00:49:18.298 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T00:49:18.298 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:49:18.298 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-03-24T00:49:18.298 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:49:18.298 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:49:18.298 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-03-24T00:49:18.298 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-03-24T00:49:18.298 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:49:18.298 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-03-24T00:49:18.298 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-03-24T00:49:18.298 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:49:18.299 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-03-24T00:49:18.299 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T00:49:18.299 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-03-24T00:49:18.299 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-24T00:49:18.299 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 41955sec preferred_lft 41955sec 2024-03-24T00:49:18.299 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-03-24T00:49:18.299 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:49:18.299 INFO:teuthology.orchestra.run.smithi067.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T00:49:18.299 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 96:cf:26:c8:bf:62 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:49:18.299 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-24T00:49:18.299 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:49:18.299 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::7c6a:5fff:feaa:d41/64 scope link 2024-03-24T00:49:18.300 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:49:18.300 INFO:teuthology.orchestra.run.smithi067.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T00:49:18.300 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ea:22:4b:d5:48:3d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:49:18.300 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::e822:4bff:fed5:483d/64 scope link 2024-03-24T00:49:18.300 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:49:18.300 INFO:teuthology.orchestra.run.smithi067.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T00:49:18.300 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 96:cf:26:c8:bf:62 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:49:18.300 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::94cf:26ff:fec8:bf62/64 scope link 2024-03-24T00:49:18.300 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:49:18.300 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:18.300 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T00:49:18.363 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-03-24T00:49:18.364 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:18.364 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-03-24T00:49:18.420 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-03-24T00:49:18.420 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T00:49:18.420 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-24T00:49:18.420 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-24T00:49:18.420 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T00:49:18.420 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:18.421 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-03-24T00:49:18.421 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-03-24T00:49:18.421 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-24T00:49:18.421 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-24T00:49:18.421 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-24T00:49:18.421 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-03-24T00:49:18.517 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:18.518 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-03-24T00:49:18.531 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-24T00:49:18.531 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-24T00:49:18.532 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:49:18.532 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:18.532 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-03-24T00:49:18.598 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:49:18.598 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-24T00:49:18.652 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:49:18.652 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:49:18.652 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:49:18.706 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-24T00:49:18.706 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:18.706 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:49:18.789 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:49:18.789 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T00:49:18.789 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-03-24T00:49:18.797 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:49:18.797 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T00:49:18.797 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-03-24T00:49:18.797 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.72:3300,172.21.15.72:3301" 2024-03-24T00:49:18.797 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-24T00:49:18.806 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:49:18.806 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T00:49:18.807 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-03-24T00:49:18.807 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.72:3300,172.21.15.72:3301" 2024-03-24T00:49:18.807 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-24T00:49:18.807 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:6789,172.21.15.72:6789,172.21.15.72:6790" 2024-03-24T00:49:18.807 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.67:6789,172.21.15.72:6789,172.21.15.72:6790:/ 2024-03-24T00:49:18.807 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=de1d0178-808e-49ca-a986-e297e523b976" will pass to kernel 2024-03-24T00:49:18.813 INFO:teuthology.orchestra.run.smithi067.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-24T00:49:18.813 INFO:teuthology.orchestra.run.smithi067.stdout: You just mounted a file system that supports labels which does not 2024-03-24T00:49:18.813 INFO:teuthology.orchestra.run.smithi067.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-24T00:49:18.813 INFO:teuthology.orchestra.run.smithi067.stdout: applications will generate AVC messages and not be allowed access to 2024-03-24T00:49:18.813 INFO:teuthology.orchestra.run.smithi067.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-24T00:49:18.813 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-24T00:49:18.813 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:18.813 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:49:18.884 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:18.884 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T00:49:18.939 INFO:teuthology.orchestra.run.smithi067.stdout:client9570 2024-03-24T00:49:18.940 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:49:18.940 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:49:18.995 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:18.995 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec find ./) 2024-03-24T00:49:19.052 INFO:teuthology.orchestra.run.smithi067.stdout:./ 2024-03-24T00:49:19.052 INFO:teuthology.orchestra.run.smithi067.stdout:./parent 2024-03-24T00:49:19.052 INFO:teuthology.orchestra.run.smithi067.stdout:./parent/flushed 2024-03-24T00:49:19.052 INFO:teuthology.orchestra.run.smithi067.stdout:./parent/flushed/charlie 2024-03-24T00:49:19.052 INFO:teuthology.orchestra.run.smithi067.stdout:./parent/flushed/alpha 2024-03-24T00:49:19.052 INFO:teuthology.orchestra.run.smithi067.stdout:./parent/unflushed 2024-03-24T00:49:19.052 INFO:teuthology.orchestra.run.smithi067.stdout:./parent/unflushed/jfile 2024-03-24T00:49:19.052 DEBUG:teuthology.orchestra.run.smithi067:> stdin-killer -- python3 -c ' 2024-03-24T00:49:19.052 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-03-24T00:49:19.053 DEBUG:teuthology.orchestra.run.smithi067:> import stat 2024-03-24T00:49:19.053 DEBUG:teuthology.orchestra.run.smithi067:> 2024-03-24T00:49:19.053 DEBUG:teuthology.orchestra.run.smithi067:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./").st_ino) 2024-03-24T00:49:19.053 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-03-24T00:49:19.157 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-03-24T00:49:19.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:49:19.166 DEBUG:teuthology.orchestra.run.smithi067:> stdin-killer -- python3 -c ' 2024-03-24T00:49:19.166 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-03-24T00:49:19.166 DEBUG:teuthology.orchestra.run.smithi067:> import stat 2024-03-24T00:49:19.166 DEBUG:teuthology.orchestra.run.smithi067:> 2024-03-24T00:49:19.166 DEBUG:teuthology.orchestra.run.smithi067:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent").st_ino) 2024-03-24T00:49:19.166 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-03-24T00:49:19.270 INFO:teuthology.orchestra.run.smithi067.stdout:1099511627776 2024-03-24T00:49:19.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:49:19.278 DEBUG:teuthology.orchestra.run.smithi067:> stdin-killer -- python3 -c ' 2024-03-24T00:49:19.278 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-03-24T00:49:19.278 DEBUG:teuthology.orchestra.run.smithi067:> import stat 2024-03-24T00:49:19.279 DEBUG:teuthology.orchestra.run.smithi067:> 2024-03-24T00:49:19.279 DEBUG:teuthology.orchestra.run.smithi067:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed").st_ino) 2024-03-24T00:49:19.279 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-03-24T00:49:19.383 INFO:teuthology.orchestra.run.smithi067.stdout:1099511627777 2024-03-24T00:49:19.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:49:19.392 DEBUG:teuthology.orchestra.run.smithi067:> stdin-killer -- python3 -c ' 2024-03-24T00:49:19.392 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-03-24T00:49:19.392 DEBUG:teuthology.orchestra.run.smithi067:> import stat 2024-03-24T00:49:19.392 DEBUG:teuthology.orchestra.run.smithi067:> 2024-03-24T00:49:19.392 DEBUG:teuthology.orchestra.run.smithi067:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed/charlie").st_ino) 2024-03-24T00:49:19.392 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-03-24T00:49:19.474 INFO:tasks.ceph.mds.a.smithi067.stderr:2024-03-24T00:49:19.474+0000 7fa4888e4640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T00:49:19.514 INFO:teuthology.orchestra.run.smithi067.stdout:1099511628280 2024-03-24T00:49:19.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:49:19.525 DEBUG:teuthology.orchestra.run.smithi067:> stdin-killer -- python3 -c ' 2024-03-24T00:49:19.525 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-03-24T00:49:19.525 DEBUG:teuthology.orchestra.run.smithi067:> import stat 2024-03-24T00:49:19.525 DEBUG:teuthology.orchestra.run.smithi067:> 2024-03-24T00:49:19.525 DEBUG:teuthology.orchestra.run.smithi067:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed/alpha").st_ino) 2024-03-24T00:49:19.525 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-03-24T00:49:19.649 INFO:teuthology.orchestra.run.smithi067.stdout:1099511627778 2024-03-24T00:49:19.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:49:19.658 DEBUG:teuthology.orchestra.run.smithi067:> stdin-killer -- python3 -c ' 2024-03-24T00:49:19.658 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-03-24T00:49:19.658 DEBUG:teuthology.orchestra.run.smithi067:> import stat 2024-03-24T00:49:19.659 DEBUG:teuthology.orchestra.run.smithi067:> 2024-03-24T00:49:19.659 DEBUG:teuthology.orchestra.run.smithi067:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/unflushed").st_ino) 2024-03-24T00:49:19.659 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-03-24T00:49:19.763 INFO:teuthology.orchestra.run.smithi067.stdout:1099511627779 2024-03-24T00:49:19.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:49:19.772 DEBUG:teuthology.orchestra.run.smithi067:> stdin-killer -- python3 -c ' 2024-03-24T00:49:19.772 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-03-24T00:49:19.772 DEBUG:teuthology.orchestra.run.smithi067:> import stat 2024-03-24T00:49:19.772 DEBUG:teuthology.orchestra.run.smithi067:> 2024-03-24T00:49:19.772 DEBUG:teuthology.orchestra.run.smithi067:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/unflushed/jfile").st_ino) 2024-03-24T00:49:19.772 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-03-24T00:49:19.877 INFO:teuthology.orchestra.run.smithi067.stdout:1099511627780 2024-03-24T00:49:19.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:49:19.885 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:49:19.886 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:49:19.941 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:49:19.941 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:49:19.997 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:49:20.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.137+0000 7f92da5b9640 1 Processor -- start 2024-03-24T00:49:20.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.137+0000 7f92da5b9640 1 -- start start 2024-03-24T00:49:20.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.137+0000 7f92da5b9640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f92d415dfc0 0x7f92d4161bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:20.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.138+0000 7f92da5b9640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f92d40cf220 0x7f92d40d2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:20.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.138+0000 7f92da5b9640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92d4162a50 con 0x7f92d40d10b0 2024-03-24T00:49:20.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.138+0000 7f92da5b9640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92d4162490 con 0x7f92d415dfc0 2024-03-24T00:49:20.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.138+0000 7f92da5b9640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92d4162770 con 0x7f92d40cf220 2024-03-24T00:49:20.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.138+0000 7f92d3fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f92d415dfc0 0x7f92d4161bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:20.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.138+0000 7f92d8b56640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f92d40cf220 0x7f92d40d2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:20.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.138+0000 7f92d3fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f92d415dfc0 0x7f92d4161bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:33604/0 (socket says 172.21.15.67:33604) 2024-03-24T00:49:20.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.138+0000 7f92d3fff640 1 -- 172.21.15.67:0/770028521 learned_addr learned my addr 172.21.15.67:0/770028521 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:20.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.138+0000 7f92d37fe640 1 --1- 172.21.15.67:0/770028521 >> v1:172.21.15.72:6789/0 conn(0x7f92d40d10b0 0x7f92d40ceb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:50532/0 (socket says 172.21.15.67:50532) 2024-03-24T00:49:20.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.138+0000 7f92d3fff640 1 -- 172.21.15.67:0/770028521 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f92d40cf220 msgr2=0x7f92d40d2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:20.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.138+0000 7f92d3fff640 1 --2- 172.21.15.67:0/770028521 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f92d40cf220 0x7f92d40d2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.138+0000 7f92d3fff640 1 -- 172.21.15.67:0/770028521 >> v1:172.21.15.72:6789/0 conn(0x7f92d40d10b0 legacy=0x7f92d40ceb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:20.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.138+0000 7f92d3fff640 1 -- 172.21.15.67:0/770028521 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92d40d1f40 con 0x7f92d415dfc0 2024-03-24T00:49:20.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.139+0000 7f92d3fff640 1 --2- 172.21.15.67:0/770028521 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f92d415dfc0 0x7f92d4161bb0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f92c4009870 tx=0x7f92c402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e433ec66890bdc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:20.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.139+0000 7f92d2ffd640 1 -- 172.21.15.67:0/770028521 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c4034070 con 0x7f92d415dfc0 2024-03-24T00:49:20.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.139+0000 7f92d2ffd640 1 -- 172.21.15.67:0/770028521 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92c4030840 con 0x7f92d415dfc0 2024-03-24T00:49:20.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.139+0000 7f92d2ffd640 1 -- 172.21.15.67:0/770028521 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c4030c10 con 0x7f92d415dfc0 2024-03-24T00:49:20.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.140+0000 7f92da5b9640 1 -- 172.21.15.67:0/770028521 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f92d415dfc0 msgr2=0x7f92d4161bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:20.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.140+0000 7f92da5b9640 1 --2- 172.21.15.67:0/770028521 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f92d415dfc0 0x7f92d4161bb0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f92c4009870 tx=0x7f92c402b980 comp rx=0 tx=0).stop 2024-03-24T00:49:20.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.140+0000 7f92da5b9640 1 -- 172.21.15.67:0/770028521 shutdown_connections 2024-03-24T00:49:20.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.140+0000 7f92da5b9640 1 --2- 172.21.15.67:0/770028521 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f92d40cf220 0x7f92d40d2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.140+0000 7f92da5b9640 1 --2- 172.21.15.67:0/770028521 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f92d415dfc0 0x7f92d4161bb0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.140+0000 7f92da5b9640 1 -- 172.21.15.67:0/770028521 >> 172.21.15.67:0/770028521 conn(0x7f92d406c930 msgr2=0x7f92d406cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:20.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.140+0000 7f92da5b9640 1 -- 172.21.15.67:0/770028521 shutdown_connections 2024-03-24T00:49:20.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.140+0000 7f92da5b9640 1 -- 172.21.15.67:0/770028521 wait complete. 2024-03-24T00:49:20.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.141+0000 7f92da5b9640 1 Processor -- start 2024-03-24T00:49:20.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.141+0000 7f92da5b9640 1 -- start start 2024-03-24T00:49:20.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.142+0000 7f92da5b9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92d40cf220 0x7f92d414ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:20.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.142+0000 7f92da5b9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f92d40d10b0 0x7f92d40c8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:20.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.142+0000 7f92da5b9640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f92d415dfc0 0x7f92d40c87c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:20.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.142+0000 7f92da5b9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92d4162270 con 0x7f92d40cf220 2024-03-24T00:49:20.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.142+0000 7f92da5b9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92d41620f0 con 0x7f92d40d10b0 2024-03-24T00:49:20.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.142+0000 7f92da5b9640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92d415e3a0 con 0x7f92d415dfc0 2024-03-24T00:49:20.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.142+0000 7f92d3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92d40cf220 0x7f92d414ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:20.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.142+0000 7f92d3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92d40cf220 0x7f92d414ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46474/0 (socket says 172.21.15.67:46474) 2024-03-24T00:49:20.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.142+0000 7f92d3fff640 1 -- 172.21.15.67:0/281525228 learned_addr learned my addr 172.21.15.67:0/281525228 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:20.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.142+0000 7f92d8b56640 1 --2- 172.21.15.67:0/281525228 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f92d415dfc0 0x7f92d40c87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:20.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.142+0000 7f92d37fe640 1 --2- 172.21.15.67:0/281525228 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f92d40d10b0 0x7f92d40c8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:20.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.142+0000 7f92d3fff640 1 -- 172.21.15.67:0/281525228 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f92d415dfc0 msgr2=0x7f92d40c87c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:20.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.142+0000 7f92d3fff640 1 --2- 172.21.15.67:0/281525228 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f92d415dfc0 0x7f92d40c87c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.142+0000 7f92d3fff640 1 -- 172.21.15.67:0/281525228 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f92d40d10b0 msgr2=0x7f92d40c8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:20.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.142+0000 7f92d3fff640 1 --2- 172.21.15.67:0/281525228 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f92d40d10b0 0x7f92d40c8280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.142+0000 7f92d3fff640 1 -- 172.21.15.67:0/281525228 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92d40c8d00 con 0x7f92d40cf220 2024-03-24T00:49:20.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.142+0000 7f92d3fff640 1 --2- 172.21.15.67:0/281525228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92d40cf220 0x7f92d414ee40 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f92c4000c00 tx=0x7f92c4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:20.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.143+0000 7f92d0ff9640 1 -- 172.21.15.67:0/281525228 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c4048070 con 0x7f92d40cf220 2024-03-24T00:49:20.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.143+0000 7f92d0ff9640 1 -- 172.21.15.67:0/281525228 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92c4030640 con 0x7f92d40cf220 2024-03-24T00:49:20.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.143+0000 7f92da5b9640 1 -- 172.21.15.67:0/281525228 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92d40c8fb0 con 0x7f92d40cf220 2024-03-24T00:49:20.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.143+0000 7f92d0ff9640 1 -- 172.21.15.67:0/281525228 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c4034040 con 0x7f92d40cf220 2024-03-24T00:49:20.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.143+0000 7f92da5b9640 1 -- 172.21.15.67:0/281525228 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92d40c9570 con 0x7f92d40cf220 2024-03-24T00:49:20.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.144+0000 7f92da5b9640 1 -- 172.21.15.67:0/281525228 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92d40b4080 con 0x7f92d40cf220 2024-03-24T00:49:20.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.145+0000 7f92d0ff9640 1 -- 172.21.15.67:0/281525228 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f92c4037070 con 0x7f92d40cf220 2024-03-24T00:49:20.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.145+0000 7f92d0ff9640 1 --2- 172.21.15.67:0/281525228 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f92c006bc00 0x7f92c006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:20.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.145+0000 7f92d0ff9640 1 -- 172.21.15.67:0/281525228 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f92c40ac770 con 0x7f92d40cf220 2024-03-24T00:49:20.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.145+0000 7f92d37fe640 1 --2- 172.21.15.67:0/281525228 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f92c006bc00 0x7f92c006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:20.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.146+0000 7f92d37fe640 1 --2- 172.21.15.67:0/281525228 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f92c006bc00 0x7f92c006e0c0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f92d40cd1e0 tx=0x7f92c8007e20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:20.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.150+0000 7f92d0ff9640 1 -- 172.21.15.67:0/281525228 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f92c407aa40 con 0x7f92d40cf220 2024-03-24T00:49:20.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.331+0000 7f92da5b9640 1 -- 172.21.15.67:0/281525228 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f92d40d1a50 con 0x7f92d40cf220 2024-03-24T00:49:20.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.332+0000 7f92d0ff9640 1 -- 172.21.15.67:0/281525228 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v121) v1 ==== 74+0+17019 (secure 0 0 0) 0x7f92c407eae0 con 0x7f92d40cf220 2024-03-24T00:49:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:20.334 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":121,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:49:12.456396+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:48:52.622356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-03-24T00:48:53.311452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6835/312063805":"2024-03-25T00:49:12.456364+0000","172.21.15.67:6833/312063805":"2024-03-25T00:49:12.456364+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:49:20.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.334+0000 7f92da5b9640 1 -- 172.21.15.67:0/281525228 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f92c006bc00 msgr2=0x7f92c006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:20.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.334+0000 7f92da5b9640 1 --2- 172.21.15.67:0/281525228 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f92c006bc00 0x7f92c006e0c0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f92d40cd1e0 tx=0x7f92c8007e20 comp rx=0 tx=0).stop 2024-03-24T00:49:20.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.334+0000 7f92da5b9640 1 -- 172.21.15.67:0/281525228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92d40cf220 msgr2=0x7f92d414ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:20.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.334+0000 7f92da5b9640 1 --2- 172.21.15.67:0/281525228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92d40cf220 0x7f92d414ee40 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f92c4000c00 tx=0x7f92c4004730 comp rx=0 tx=0).stop 2024-03-24T00:49:20.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.334+0000 7f92da5b9640 1 -- 172.21.15.67:0/281525228 shutdown_connections 2024-03-24T00:49:20.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.334+0000 7f92da5b9640 1 --2- 172.21.15.67:0/281525228 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f92c006bc00 0x7f92c006e0c0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.334+0000 7f92da5b9640 1 --2- 172.21.15.67:0/281525228 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f92d415dfc0 0x7f92d40c87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.334+0000 7f92da5b9640 1 --2- 172.21.15.67:0/281525228 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f92d40d10b0 0x7f92d40c8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.334+0000 7f92da5b9640 1 --2- 172.21.15.67:0/281525228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92d40cf220 0x7f92d414ee40 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.334+0000 7f92da5b9640 1 -- 172.21.15.67:0/281525228 >> 172.21.15.67:0/281525228 conn(0x7f92d406c930 msgr2=0x7f92d40b8880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:20.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.335+0000 7f92da5b9640 1 -- 172.21.15.67:0/281525228 shutdown_connections 2024-03-24T00:49:20.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.335+0000 7f92da5b9640 1 -- 172.21.15.67:0/281525228 wait complete. 2024-03-24T00:49:20.346 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:38:39.364950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:48:52.622356+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '119', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:48:53.311452+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '120', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:49:20.347 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:49:20.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.453+0000 7f2f75036640 1 Processor -- start 2024-03-24T00:49:20.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.454+0000 7f2f75036640 1 -- start start 2024-03-24T00:49:20.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.454+0000 7f2f75036640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2f7009a190 0x7f2f700cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:20.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.454+0000 7f2f75036640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f7009ade0 con 0x7f2f70099d00 2024-03-24T00:49:20.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.454+0000 7f2f75036640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f7009a600 con 0x7f2f7015f7b0 2024-03-24T00:49:20.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.454+0000 7f2f75036640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f700cf9e0 con 0x7f2f7009a190 2024-03-24T00:49:20.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.454+0000 7f2f6659c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2f70099d00 0x7f2f700ccd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43708/0 (socket says 172.21.15.67:43708) 2024-03-24T00:49:20.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.454+0000 7f2f6659c640 1 -- 172.21.15.67:0/2535693194 learned_addr learned my addr 172.21.15.67:0/2535693194 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:49:20.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.455+0000 7f2f6f59e640 1 --2- 172.21.15.67:0/2535693194 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2f7009a190 0x7f2f700cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:20.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.455+0000 7f2f6e59c640 1 -- 172.21.15.67:0/2535693194 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3192488398 0 0) 0x7f2f7009ade0 con 0x7f2f70099d00 2024-03-24T00:49:20.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.455+0000 7f2f6e59c640 1 -- 172.21.15.67:0/2535693194 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f5c003520 con 0x7f2f70099d00 2024-03-24T00:49:20.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.455+0000 7f2f6e59c640 1 -- 172.21.15.67:0/2535693194 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2922352890 0 0) 0x7f2f5c003520 con 0x7f2f70099d00 2024-03-24T00:49:20.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.455+0000 7f2f6e59c640 1 -- 172.21.15.67:0/2535693194 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2f7009a190 msgr2=0x7f2f700cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:20.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.455+0000 7f2f6e59c640 1 --2- 172.21.15.67:0/2535693194 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2f7009a190 0x7f2f700cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.455+0000 7f2f6e59c640 1 -- 172.21.15.67:0/2535693194 >> v1:172.21.15.72:6789/0 conn(0x7f2f7015f7b0 legacy=0x7f2f700b2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:20.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.455+0000 7f2f6e59c640 1 -- 172.21.15.67:0/2535693194 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f7009b080 con 0x7f2f70099d00 2024-03-24T00:49:20.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.455+0000 7f2f6e59c640 1 -- 172.21.15.67:0/2535693194 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f2f58001670 con 0x7f2f70099d00 2024-03-24T00:49:20.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.455+0000 7f2f6e59c640 1 -- 172.21.15.67:0/2535693194 >> v1:172.21.15.67:6789/0 conn(0x7f2f70099d00 legacy=0x7f2f700ccd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:20.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.455+0000 7f2f6e59c640 1 --2- 172.21.15.67:0/2535693194 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f5c0039e0 0x7f2f5c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:20.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.455+0000 7f2f6e59c640 1 --2- 172.21.15.67:0/2535693194 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2f7009a190 0x7f2f5c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:20.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.455+0000 7f2f6e59c640 1 --2- 172.21.15.67:0/2535693194 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f5c004d40 0x7f2f5c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:20.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.455+0000 7f2f6e59c640 1 -- 172.21.15.67:0/2535693194 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f700cf9e0 con 0x7f2f5c004d40 2024-03-24T00:49:20.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.455+0000 7f2f6e59c640 1 -- 172.21.15.67:0/2535693194 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f5c00b9c0 con 0x7f2f5c0039e0 2024-03-24T00:49:20.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.455+0000 7f2f6e59c640 1 -- 172.21.15.67:0/2535693194 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f5c00bbc0 con 0x7f2f7009a190 2024-03-24T00:49:20.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.455+0000 7f2f6f59e640 1 --2- 172.21.15.67:0/2535693194 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f5c004d40 0x7f2f5c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:20.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.455+0000 7f2f6ed9d640 1 --2- 172.21.15.67:0/2535693194 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f5c0039e0 0x7f2f5c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:20.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.455+0000 7f2f6659c640 1 --2- 172.21.15.67:0/2535693194 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2f7009a190 0x7f2f5c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:20.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.456+0000 7f2f6f59e640 1 -- 172.21.15.67:0/2535693194 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2f7009a190 msgr2=0x7f2f5c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:20.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.456+0000 7f2f6f59e640 1 --2- 172.21.15.67:0/2535693194 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2f7009a190 0x7f2f5c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.456+0000 7f2f6f59e640 1 -- 172.21.15.67:0/2535693194 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f5c0039e0 msgr2=0x7f2f5c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:20.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.456+0000 7f2f6f59e640 1 --2- 172.21.15.67:0/2535693194 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f5c0039e0 0x7f2f5c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.456+0000 7f2f6f59e640 1 -- 172.21.15.67:0/2535693194 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f5c003520 con 0x7f2f5c004d40 2024-03-24T00:49:20.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.456+0000 7f2f6f59e640 1 --2- 172.21.15.67:0/2535693194 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f5c004d40 0x7f2f5c0051c0 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f2f68009fc0 tx=0x7f2f5c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d79a1b74ae9c3c77 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:20.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.456+0000 7f2f6e59c640 1 -- 172.21.15.67:0/2535693194 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f68039070 con 0x7f2f5c004d40 2024-03-24T00:49:20.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.456+0000 7f2f6e59c640 1 -- 172.21.15.67:0/2535693194 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f6802cc20 con 0x7f2f5c004d40 2024-03-24T00:49:20.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.456+0000 7f2f6e59c640 1 -- 172.21.15.67:0/2535693194 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f68009c10 con 0x7f2f5c004d40 2024-03-24T00:49:20.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.457+0000 7f2f75036640 1 -- 172.21.15.67:0/2535693194 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f5c004d40 msgr2=0x7f2f5c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:20.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.457+0000 7f2f75036640 1 --2- 172.21.15.67:0/2535693194 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f5c004d40 0x7f2f5c0051c0 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f2f68009fc0 tx=0x7f2f5c005830 comp rx=0 tx=0).stop 2024-03-24T00:49:20.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.457+0000 7f2f6f59e640 1 -- 172.21.15.67:0/2535693194 reap_dead start 2024-03-24T00:49:20.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.457+0000 7f2f75036640 1 -- 172.21.15.67:0/2535693194 shutdown_connections 2024-03-24T00:49:20.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.457+0000 7f2f75036640 1 -- 172.21.15.67:0/2535693194 >> 172.21.15.67:0/2535693194 conn(0x7f2f700b3650 msgr2=0x7f2f700b3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:20.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.457+0000 7f2f75036640 1 -- 172.21.15.67:0/2535693194 shutdown_connections 2024-03-24T00:49:20.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.457+0000 7f2f75036640 1 -- 172.21.15.67:0/2535693194 wait complete. 2024-03-24T00:49:20.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.458+0000 7f2f75036640 1 Processor -- start 2024-03-24T00:49:20.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.459+0000 7f2f75036640 1 -- start start 2024-03-24T00:49:20.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.459+0000 7f2f75036640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2f700c7f70 0x7f2f700b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:20.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.459+0000 7f2f75036640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f701f7930 0x7f2f701f7db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:20.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.459+0000 7f2f75036640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f7015cd80 0x7f2f7015d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:20.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.459+0000 7f2f75036640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f700ae750 con 0x7f2f7015cd80 2024-03-24T00:49:20.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.459+0000 7f2f75036640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f700cfcc0 con 0x7f2f701f7930 2024-03-24T00:49:20.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.459+0000 7f2f75036640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f701f83b0 con 0x7f2f700c7f70 2024-03-24T00:49:20.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.459+0000 7f2f6f59e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f7015cd80 0x7f2f7015d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:20.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.459+0000 7f2f6f59e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f7015cd80 0x7f2f7015d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46494/0 (socket says 172.21.15.67:46494) 2024-03-24T00:49:20.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.459+0000 7f2f6f59e640 1 -- 172.21.15.67:0/2709369288 learned_addr learned my addr 172.21.15.67:0/2709369288 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:20.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.460+0000 7f2f6659c640 1 --2- 172.21.15.67:0/2709369288 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f701f7930 0x7f2f701f7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:20.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.460+0000 7f2f6ed9d640 1 --2- 172.21.15.67:0/2709369288 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2f700c7f70 0x7f2f700b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:20.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.460+0000 7f2f6f59e640 1 -- 172.21.15.67:0/2709369288 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2f700c7f70 msgr2=0x7f2f700b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:20.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.460+0000 7f2f6f59e640 1 --2- 172.21.15.67:0/2709369288 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2f700c7f70 0x7f2f700b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.460+0000 7f2f6f59e640 1 -- 172.21.15.67:0/2709369288 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f701f7930 msgr2=0x7f2f701f7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:20.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.460+0000 7f2f6f59e640 1 --2- 172.21.15.67:0/2709369288 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f701f7930 0x7f2f701f7db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.460+0000 7f2f6f59e640 1 -- 172.21.15.67:0/2709369288 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f701f8690 con 0x7f2f7015cd80 2024-03-24T00:49:20.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.460+0000 7f2f6f59e640 1 --2- 172.21.15.67:0/2709369288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f7015cd80 0x7f2f7015d200 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f2f68002230 tx=0x7f2f68009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:20.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.460+0000 7f2f67fff640 1 -- 172.21.15.67:0/2709369288 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f68039030 con 0x7f2f7015cd80 2024-03-24T00:49:20.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.460+0000 7f2f67fff640 1 -- 172.21.15.67:0/2709369288 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f68043070 con 0x7f2f7015cd80 2024-03-24T00:49:20.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.460+0000 7f2f67fff640 1 -- 172.21.15.67:0/2709369288 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f68009c10 con 0x7f2f7015cd80 2024-03-24T00:49:20.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.460+0000 7f2f75036640 1 -- 172.21.15.67:0/2709369288 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f700afcc0 con 0x7f2f7015cd80 2024-03-24T00:49:20.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.460+0000 7f2f75036640 1 -- 172.21.15.67:0/2709369288 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f700b0a00 con 0x7f2f7015cd80 2024-03-24T00:49:20.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.462+0000 7f2f75036640 1 -- 172.21.15.67:0/2709369288 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f700afab0 con 0x7f2f7015cd80 2024-03-24T00:49:20.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.462+0000 7f2f67fff640 1 -- 172.21.15.67:0/2709369288 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2f68005d10 con 0x7f2f7015cd80 2024-03-24T00:49:20.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.462+0000 7f2f67fff640 1 --2- 172.21.15.67:0/2709369288 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2f5006bbd0 0x7f2f5006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:20.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.462+0000 7f2f67fff640 1 -- 172.21.15.67:0/2709369288 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f2f680accf0 con 0x7f2f7015cd80 2024-03-24T00:49:20.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.462+0000 7f2f6ed9d640 1 --2- 172.21.15.67:0/2709369288 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2f5006bbd0 0x7f2f5006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:20.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.463+0000 7f2f6ed9d640 1 --2- 172.21.15.67:0/2709369288 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2f5006bbd0 0x7f2f5006e090 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f2f60000c00 tx=0x7f2f60009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:20.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.467+0000 7f2f67fff640 1 -- 172.21.15.67:0/2709369288 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2f680b1070 con 0x7f2f7015cd80 2024-03-24T00:49:20.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.649+0000 7f2f75036640 1 -- 172.21.15.67:0/2709369288 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2f7004f400 con 0x7f2f7015cd80 2024-03-24T00:49:20.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.649+0000 7f2f67fff640 1 -- 172.21.15.67:0/2709369288 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v121) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2f6807b070 con 0x7f2f7015cd80 2024-03-24T00:49:20.650 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-24T00:49:20.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.651+0000 7f2f75036640 1 -- 172.21.15.67:0/2709369288 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2f5006bbd0 msgr2=0x7f2f5006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:20.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.651+0000 7f2f75036640 1 --2- 172.21.15.67:0/2709369288 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2f5006bbd0 0x7f2f5006e090 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f2f60000c00 tx=0x7f2f60009040 comp rx=0 tx=0).stop 2024-03-24T00:49:20.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.651+0000 7f2f75036640 1 -- 172.21.15.67:0/2709369288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f7015cd80 msgr2=0x7f2f7015d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:20.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.651+0000 7f2f75036640 1 --2- 172.21.15.67:0/2709369288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f7015cd80 0x7f2f7015d200 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f2f68002230 tx=0x7f2f68009750 comp rx=0 tx=0).stop 2024-03-24T00:49:20.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.651+0000 7f2f75036640 1 -- 172.21.15.67:0/2709369288 shutdown_connections 2024-03-24T00:49:20.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.651+0000 7f2f75036640 1 --2- 172.21.15.67:0/2709369288 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2f5006bbd0 0x7f2f5006e090 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.651+0000 7f2f75036640 1 --2- 172.21.15.67:0/2709369288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f7015cd80 0x7f2f7015d200 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.651+0000 7f2f75036640 1 --2- 172.21.15.67:0/2709369288 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f701f7930 0x7f2f701f7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.651+0000 7f2f75036640 1 --2- 172.21.15.67:0/2709369288 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2f700c7f70 0x7f2f700b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.651+0000 7f2f75036640 1 -- 172.21.15.67:0/2709369288 >> 172.21.15.67:0/2709369288 conn(0x7f2f700b3650 msgr2=0x7f2f70073b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:20.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.651+0000 7f2f75036640 1 -- 172.21.15.67:0/2709369288 shutdown_connections 2024-03-24T00:49:20.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.651+0000 7f2f75036640 1 -- 172.21.15.67:0/2709369288 wait complete. 2024-03-24T00:49:20.663 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:49:20.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.771+0000 7fba2afd3640 1 Processor -- start 2024-03-24T00:49:20.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.771+0000 7fba2afd3640 1 -- start start 2024-03-24T00:49:20.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.771+0000 7fba2afd3640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fba2415dfc0 0x7fba24161bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:20.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.771+0000 7fba2afd3640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fba240cf220 0x7fba240d2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:20.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.771+0000 7fba2afd3640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba24162a50 con 0x7fba240d10b0 2024-03-24T00:49:20.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.771+0000 7fba2afd3640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba24162490 con 0x7fba240cf220 2024-03-24T00:49:20.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.771+0000 7fba2afd3640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba24162770 con 0x7fba2415dfc0 2024-03-24T00:49:20.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.772+0000 7fba28d6f640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fba2415dfc0 0x7fba24161bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:20.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.772+0000 7fba29570640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fba240cf220 0x7fba240d2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:20.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.772+0000 7fba23fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fba240d10b0 0x7fba240ceb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:50558/0 (socket says 172.21.15.67:50558) 2024-03-24T00:49:20.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.772+0000 7fba23fff640 1 -- 172.21.15.67:0/2103672230 learned_addr learned my addr 172.21.15.67:0/2103672230 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:49:20.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.772+0000 7fba29570640 1 -- 172.21.15.67:0/2103672230 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fba2415dfc0 msgr2=0x7fba24161bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:20.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.772+0000 7fba29570640 1 --2- 172.21.15.67:0/2103672230 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fba2415dfc0 0x7fba24161bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.772+0000 7fba29570640 1 -- 172.21.15.67:0/2103672230 >> v1:172.21.15.72:6789/0 conn(0x7fba240d10b0 legacy=0x7fba240ceb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:20.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.772+0000 7fba29570640 1 -- 172.21.15.67:0/2103672230 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba240d1f40 con 0x7fba240cf220 2024-03-24T00:49:20.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.773+0000 7fba29570640 1 --2- 172.21.15.67:0/2103672230 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba240cf220 0x7fba240d2740 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fba1c009d90 tx=0x7fba1c02bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d079c67a93d8207 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:20.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.773+0000 7fba237fe640 1 -- 172.21.15.67:0/2103672230 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba1c039070 con 0x7fba240cf220 2024-03-24T00:49:20.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.773+0000 7fba237fe640 1 -- 172.21.15.67:0/2103672230 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba1c02cc50 con 0x7fba240cf220 2024-03-24T00:49:20.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.773+0000 7fba237fe640 1 -- 172.21.15.67:0/2103672230 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba1c034440 con 0x7fba240cf220 2024-03-24T00:49:20.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.774+0000 7fba2afd3640 1 -- 172.21.15.67:0/2103672230 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba240cf220 msgr2=0x7fba240d2740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:20.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.774+0000 7fba2afd3640 1 --2- 172.21.15.67:0/2103672230 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba240cf220 0x7fba240d2740 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fba1c009d90 tx=0x7fba1c02bea0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.774+0000 7fba2afd3640 1 -- 172.21.15.67:0/2103672230 shutdown_connections 2024-03-24T00:49:20.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.774+0000 7fba2afd3640 1 --2- 172.21.15.67:0/2103672230 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba240cf220 0x7fba240d2740 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.774+0000 7fba2afd3640 1 --2- 172.21.15.67:0/2103672230 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fba2415dfc0 0x7fba24161bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.774+0000 7fba2afd3640 1 -- 172.21.15.67:0/2103672230 >> 172.21.15.67:0/2103672230 conn(0x7fba2406c930 msgr2=0x7fba2406cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:20.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.774+0000 7fba2afd3640 1 -- 172.21.15.67:0/2103672230 shutdown_connections 2024-03-24T00:49:20.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.774+0000 7fba2afd3640 1 -- 172.21.15.67:0/2103672230 wait complete. 2024-03-24T00:49:20.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.775+0000 7fba2afd3640 1 Processor -- start 2024-03-24T00:49:20.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.775+0000 7fba2afd3640 1 -- start start 2024-03-24T00:49:20.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.776+0000 7fba2afd3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba240cf220 0x7fba241f7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:20.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.776+0000 7fba2afd3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba240d10b0 0x7fba240d3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:20.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.776+0000 7fba2afd3640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fba2415dfc0 0x7fba241f3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:20.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.776+0000 7fba2afd3640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba24162270 con 0x7fba240d10b0 2024-03-24T00:49:20.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.776+0000 7fba2afd3640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba241620f0 con 0x7fba240cf220 2024-03-24T00:49:20.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.776+0000 7fba2afd3640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba2415e3a0 con 0x7fba2415dfc0 2024-03-24T00:49:20.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.776+0000 7fba23fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba240d10b0 0x7fba240d3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:20.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.776+0000 7fba23fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba240d10b0 0x7fba240d3f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46496/0 (socket says 172.21.15.67:46496) 2024-03-24T00:49:20.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.776+0000 7fba23fff640 1 -- 172.21.15.67:0/1734767822 learned_addr learned my addr 172.21.15.67:0/1734767822 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:20.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.776+0000 7fba28d6f640 1 --2- 172.21.15.67:0/1734767822 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba240cf220 0x7fba241f7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:20.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.776+0000 7fba29570640 1 --2- 172.21.15.67:0/1734767822 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fba2415dfc0 0x7fba241f3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:20.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.776+0000 7fba23fff640 1 -- 172.21.15.67:0/1734767822 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fba2415dfc0 msgr2=0x7fba241f3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:20.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.776+0000 7fba23fff640 1 --2- 172.21.15.67:0/1734767822 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fba2415dfc0 0x7fba241f3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.776+0000 7fba23fff640 1 -- 172.21.15.67:0/1734767822 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba240cf220 msgr2=0x7fba241f7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:20.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.776+0000 7fba23fff640 1 --2- 172.21.15.67:0/1734767822 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba240cf220 0x7fba241f7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.776+0000 7fba23fff640 1 -- 172.21.15.67:0/1734767822 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba240c9f80 con 0x7fba240d10b0 2024-03-24T00:49:20.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.776+0000 7fba23fff640 1 --2- 172.21.15.67:0/1734767822 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba240d10b0 0x7fba240d3f90 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7fba1000c400 tx=0x7fba1000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:20.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.777+0000 7fba217fa640 1 -- 172.21.15.67:0/1734767822 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba10017070 con 0x7fba240d10b0 2024-03-24T00:49:20.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.777+0000 7fba217fa640 1 -- 172.21.15.67:0/1734767822 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba10005ce0 con 0x7fba240d10b0 2024-03-24T00:49:20.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.777+0000 7fba2afd3640 1 -- 172.21.15.67:0/1734767822 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba241f4690 con 0x7fba240d10b0 2024-03-24T00:49:20.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.777+0000 7fba217fa640 1 -- 172.21.15.67:0/1734767822 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba100072f0 con 0x7fba240d10b0 2024-03-24T00:49:20.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.777+0000 7fba2afd3640 1 -- 172.21.15.67:0/1734767822 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba241f73e0 con 0x7fba240d10b0 2024-03-24T00:49:20.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.778+0000 7fba2afd3640 1 -- 172.21.15.67:0/1734767822 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba2404f400 con 0x7fba240d10b0 2024-03-24T00:49:20.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.778+0000 7fba217fa640 1 -- 172.21.15.67:0/1734767822 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fba1001a070 con 0x7fba240d10b0 2024-03-24T00:49:20.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.778+0000 7fba217fa640 1 --2- 172.21.15.67:0/1734767822 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fba0806bbd0 0x7fba0806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:20.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.779+0000 7fba217fa640 1 -- 172.21.15.67:0/1734767822 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fba1008a070 con 0x7fba240d10b0 2024-03-24T00:49:20.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.779+0000 7fba28d6f640 1 --2- 172.21.15.67:0/1734767822 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fba0806bbd0 0x7fba0806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:20.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.779+0000 7fba28d6f640 1 --2- 172.21.15.67:0/1734767822 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fba0806bbd0 0x7fba0806e090 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fba240d4d20 tx=0x7fba18008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:20.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.787+0000 7fba217fa640 1 -- 172.21.15.67:0/1734767822 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fba10058960 con 0x7fba240d10b0 2024-03-24T00:49:20.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.967+0000 7fba2afd3640 1 -- 172.21.15.67:0/1734767822 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fba2415c9a0 con 0x7fba240d10b0 2024-03-24T00:49:20.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.968+0000 7fba217fa640 1 -- 172.21.15.67:0/1734767822 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v121) v1 ==== 104+0+11 (secure 0 0 0) 0x7fba10058260 con 0x7fba240d10b0 2024-03-24T00:49:20.969 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-24T00:49:20.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.969+0000 7fba2afd3640 1 -- 172.21.15.67:0/1734767822 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fba0806bbd0 msgr2=0x7fba0806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:20.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.969+0000 7fba2afd3640 1 --2- 172.21.15.67:0/1734767822 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fba0806bbd0 0x7fba0806e090 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fba240d4d20 tx=0x7fba18008040 comp rx=0 tx=0).stop 2024-03-24T00:49:20.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.969+0000 7fba2afd3640 1 -- 172.21.15.67:0/1734767822 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba240d10b0 msgr2=0x7fba240d3f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:20.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.969+0000 7fba2afd3640 1 --2- 172.21.15.67:0/1734767822 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba240d10b0 0x7fba240d3f90 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7fba1000c400 tx=0x7fba1000c8c0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.970+0000 7fba2afd3640 1 -- 172.21.15.67:0/1734767822 shutdown_connections 2024-03-24T00:49:20.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.970+0000 7fba2afd3640 1 --2- 172.21.15.67:0/1734767822 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fba0806bbd0 0x7fba0806e090 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.970+0000 7fba2afd3640 1 --2- 172.21.15.67:0/1734767822 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fba2415dfc0 0x7fba241f3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.970+0000 7fba2afd3640 1 --2- 172.21.15.67:0/1734767822 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba240d10b0 0x7fba240d3f90 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.970+0000 7fba2afd3640 1 --2- 172.21.15.67:0/1734767822 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba240cf220 0x7fba241f7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:20.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.970+0000 7fba2afd3640 1 -- 172.21.15.67:0/1734767822 >> 172.21.15.67:0/1734767822 conn(0x7fba2406c930 msgr2=0x7fba240b8850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:20.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.970+0000 7fba2afd3640 1 -- 172.21.15.67:0/1734767822 shutdown_connections 2024-03-24T00:49:20.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:20.970+0000 7fba2afd3640 1 -- 172.21.15.67:0/1734767822 wait complete. 2024-03-24T00:49:20.981 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:49:21.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.084+0000 7f9aa3b09640 1 Processor -- start 2024-03-24T00:49:21.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.085+0000 7f9aa3b09640 1 -- start start 2024-03-24T00:49:21.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.085+0000 7f9aa3b09640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9a9c15dfc0 0x7f9a9c161bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:21.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.085+0000 7f9aa3b09640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9a9c0d10b0 0x7f9a9c0ceb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:21.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.085+0000 7f9aa3b09640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a9c162a50 con 0x7f9a9c0cf050 2024-03-24T00:49:21.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.085+0000 7f9aa3b09640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a9c162490 con 0x7f9a9c15dfc0 2024-03-24T00:49:21.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.085+0000 7f9aa3b09640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9c162770 con 0x7f9a9c0d10b0 2024-03-24T00:49:21.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.086+0000 7f9aa18a5640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9a9c15dfc0 0x7f9a9c161bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:21.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.086+0000 7f9aa18a5640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9a9c15dfc0 0x7f9a9c161bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46512/0 (socket says 172.21.15.67:46512) 2024-03-24T00:49:21.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.086+0000 7f9aa18a5640 1 -- 172.21.15.67:0/3495511615 learned_addr learned my addr 172.21.15.67:0/3495511615 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:21.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.086+0000 7f9aa10a4640 1 --2- 172.21.15.67:0/3495511615 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9a9c0d10b0 0x7f9a9c0ceb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:21.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.086+0000 7f9aa18a5640 1 -- 172.21.15.67:0/3495511615 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9a9c0d10b0 msgr2=0x7f9a9c0ceb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:21.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.086+0000 7f9aa18a5640 1 --2- 172.21.15.67:0/3495511615 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9a9c0d10b0 0x7f9a9c0ceb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:21.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.086+0000 7f9aa18a5640 1 -- 172.21.15.67:0/3495511615 >> v1:172.21.15.72:6789/0 conn(0x7f9a9c0cf050 legacy=0x7f9a9c0b57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:21.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.086+0000 7f9aa18a5640 1 -- 172.21.15.67:0/3495511615 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a9c0d1f40 con 0x7f9a9c15dfc0 2024-03-24T00:49:21.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.087+0000 7f9aa18a5640 1 --2- 172.21.15.67:0/3495511615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a9c15dfc0 0x7f9a9c161bb0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f9a94009e50 tx=0x7f9a9402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b459317cc61e0b51 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:21.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.087+0000 7f9aa08a3640 1 -- 172.21.15.67:0/3495511615 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a9402cb00 con 0x7f9a9c15dfc0 2024-03-24T00:49:21.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.087+0000 7f9aa08a3640 1 -- 172.21.15.67:0/3495511615 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a9402cca0 con 0x7f9a9c15dfc0 2024-03-24T00:49:21.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.087+0000 7f9aa08a3640 1 -- 172.21.15.67:0/3495511615 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a94034610 con 0x7f9a9c15dfc0 2024-03-24T00:49:21.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.087+0000 7f9aa3b09640 1 -- 172.21.15.67:0/3495511615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a9c15dfc0 msgr2=0x7f9a9c161bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:21.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.087+0000 7f9aa3b09640 1 --2- 172.21.15.67:0/3495511615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a9c15dfc0 0x7f9a9c161bb0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f9a94009e50 tx=0x7f9a9402bf60 comp rx=0 tx=0).stop 2024-03-24T00:49:21.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.087+0000 7f9aa3b09640 1 -- 172.21.15.67:0/3495511615 shutdown_connections 2024-03-24T00:49:21.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.087+0000 7f9aa3b09640 1 --2- 172.21.15.67:0/3495511615 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9a9c0d10b0 0x7f9a9c0ceb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:21.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.087+0000 7f9aa3b09640 1 --2- 172.21.15.67:0/3495511615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a9c15dfc0 0x7f9a9c161bb0 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:21.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.087+0000 7f9aa3b09640 1 -- 172.21.15.67:0/3495511615 >> 172.21.15.67:0/3495511615 conn(0x7f9a9c06c930 msgr2=0x7f9a9c06cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:21.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.088+0000 7f9aa3b09640 1 -- 172.21.15.67:0/3495511615 shutdown_connections 2024-03-24T00:49:21.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.088+0000 7f9aa3b09640 1 -- 172.21.15.67:0/3495511615 wait complete. 2024-03-24T00:49:21.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.089+0000 7f9aa3b09640 1 Processor -- start 2024-03-24T00:49:21.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.089+0000 7f9aa3b09640 1 -- start start 2024-03-24T00:49:21.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.089+0000 7f9aa3b09640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a9c0cf050 0x7f9a9c0b36a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:21.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.089+0000 7f9aa3b09640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9a9c0d10b0 0x7f9a9c0c9560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:21.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.089+0000 7f9aa3b09640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a9c15dfc0 0x7f9a9c15fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:21.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.089+0000 7f9aa3b09640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9c162270 con 0x7f9a9c0cf050 2024-03-24T00:49:21.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.089+0000 7f9aa3b09640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9c1620f0 con 0x7f9a9c15dfc0 2024-03-24T00:49:21.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.089+0000 7f9aa3b09640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9c15e3a0 con 0x7f9a9c0d10b0 2024-03-24T00:49:21.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.089+0000 7f9aa18a5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a9c0cf050 0x7f9a9c0b36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:21.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.090+0000 7f9aa18a5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a9c0cf050 0x7f9a9c0b36a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46524/0 (socket says 172.21.15.67:46524) 2024-03-24T00:49:21.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.090+0000 7f9aa18a5640 1 -- 172.21.15.67:0/1716523816 learned_addr learned my addr 172.21.15.67:0/1716523816 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:21.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.090+0000 7f9aa10a4640 1 --2- 172.21.15.67:0/1716523816 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9a9c0d10b0 0x7f9a9c0c9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:21.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.090+0000 7f9aa20a6640 1 --2- 172.21.15.67:0/1716523816 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a9c15dfc0 0x7f9a9c15fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:21.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.090+0000 7f9aa18a5640 1 -- 172.21.15.67:0/1716523816 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9a9c0d10b0 msgr2=0x7f9a9c0c9560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:21.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.090+0000 7f9aa18a5640 1 --2- 172.21.15.67:0/1716523816 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9a9c0d10b0 0x7f9a9c0c9560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:21.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.090+0000 7f9aa18a5640 1 -- 172.21.15.67:0/1716523816 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a9c15dfc0 msgr2=0x7f9a9c15fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:21.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.090+0000 7f9aa18a5640 1 --2- 172.21.15.67:0/1716523816 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a9c15dfc0 0x7f9a9c15fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:21.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.090+0000 7f9aa18a5640 1 -- 172.21.15.67:0/1716523816 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a9c1f7590 con 0x7f9a9c0cf050 2024-03-24T00:49:21.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.090+0000 7f9aa18a5640 1 --2- 172.21.15.67:0/1716523816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a9c0cf050 0x7f9a9c0b36a0 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f9a94000f80 tx=0x7f9a94004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:21.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.090+0000 7f9a9a7fc640 1 -- 172.21.15.67:0/1716523816 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a94043020 con 0x7f9a9c0cf050 2024-03-24T00:49:21.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.090+0000 7f9a9a7fc640 1 -- 172.21.15.67:0/1716523816 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a9402ce50 con 0x7f9a9c0cf050 2024-03-24T00:49:21.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.090+0000 7f9aa3b09640 1 -- 172.21.15.67:0/1716523816 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a9c1f7870 con 0x7f9a9c0cf050 2024-03-24T00:49:21.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.090+0000 7f9a9a7fc640 1 -- 172.21.15.67:0/1716523816 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a94009510 con 0x7f9a9c0cf050 2024-03-24T00:49:21.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.090+0000 7f9aa3b09640 1 -- 172.21.15.67:0/1716523816 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a9c1f7e30 con 0x7f9a9c0cf050 2024-03-24T00:49:21.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.091+0000 7f9aa3b09640 1 -- 172.21.15.67:0/1716523816 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a9c0d0330 con 0x7f9a9c0cf050 2024-03-24T00:49:21.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.092+0000 7f9a9a7fc640 1 -- 172.21.15.67:0/1716523816 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9a94046020 con 0x7f9a9c0cf050 2024-03-24T00:49:21.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.092+0000 7f9a9a7fc640 1 --2- 172.21.15.67:0/1716523816 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9a8806bbd0 0x7f9a8806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:21.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.092+0000 7f9a9a7fc640 1 -- 172.21.15.67:0/1716523816 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f9a940ad300 con 0x7f9a9c0cf050 2024-03-24T00:49:21.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.092+0000 7f9aa10a4640 1 --2- 172.21.15.67:0/1716523816 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9a8806bbd0 0x7f9a8806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:21.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.093+0000 7f9aa10a4640 1 --2- 172.21.15.67:0/1716523816 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9a8806bbd0 0x7f9a8806e090 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c0ca890 tx=0x7f9a90008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:21.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.101+0000 7f9a9a7fc640 1 -- 172.21.15.67:0/1716523816 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9a9c0d0330 con 0x7f9a9c0cf050 2024-03-24T00:49:21.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.283+0000 7f9aa3b09640 1 -- 172.21.15.67:0/1716523816 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9a9c04f400 con 0x7f9a9c0cf050 2024-03-24T00:49:21.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.284+0000 7f9a9a7fc640 1 -- 172.21.15.67:0/1716523816 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v121) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9a9404d030 con 0x7f9a9c0cf050 2024-03-24T00:49:21.285 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-03-24T00:49:21.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.286+0000 7f9aa3b09640 1 -- 172.21.15.67:0/1716523816 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9a8806bbd0 msgr2=0x7f9a8806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:21.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.286+0000 7f9aa3b09640 1 --2- 172.21.15.67:0/1716523816 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9a8806bbd0 0x7f9a8806e090 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c0ca890 tx=0x7f9a90008040 comp rx=0 tx=0).stop 2024-03-24T00:49:21.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.286+0000 7f9aa3b09640 1 -- 172.21.15.67:0/1716523816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a9c0cf050 msgr2=0x7f9a9c0b36a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:21.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.286+0000 7f9aa3b09640 1 --2- 172.21.15.67:0/1716523816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a9c0cf050 0x7f9a9c0b36a0 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f9a94000f80 tx=0x7f9a94004a40 comp rx=0 tx=0).stop 2024-03-24T00:49:21.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.286+0000 7f9aa3b09640 1 -- 172.21.15.67:0/1716523816 shutdown_connections 2024-03-24T00:49:21.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.286+0000 7f9aa3b09640 1 --2- 172.21.15.67:0/1716523816 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9a8806bbd0 0x7f9a8806e090 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:21.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.286+0000 7f9aa3b09640 1 --2- 172.21.15.67:0/1716523816 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a9c15dfc0 0x7f9a9c15fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:21.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.286+0000 7f9aa3b09640 1 --2- 172.21.15.67:0/1716523816 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9a9c0d10b0 0x7f9a9c0c9560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:21.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.286+0000 7f9aa3b09640 1 --2- 172.21.15.67:0/1716523816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a9c0cf050 0x7f9a9c0b36a0 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:21.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.286+0000 7f9aa3b09640 1 -- 172.21.15.67:0/1716523816 >> 172.21.15.67:0/1716523816 conn(0x7f9a9c06c930 msgr2=0x7f9a9c0b88b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:21.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.286+0000 7f9aa3b09640 1 -- 172.21.15.67:0/1716523816 shutdown_connections 2024-03-24T00:49:21.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.286+0000 7f9aa3b09640 1 -- 172.21.15.67:0/1716523816 wait complete. 2024-03-24T00:49:21.297 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T00:49:21.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.402+0000 7f1f9d020640 1 Processor -- start 2024-03-24T00:49:21.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.402+0000 7f1f9d020640 1 -- start start 2024-03-24T00:49:21.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.403+0000 7f1f9d020640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1f9815dfc0 0x7f1f98161bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:21.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.403+0000 7f1f9d020640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1f980d10b0 0x7f1f980ceb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:21.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.403+0000 7f1f9d020640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1f980cf050 0x7f1f980b57c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:21.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.403+0000 7f1f9d020640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f98162a50 con 0x7f1f980d10b0 2024-03-24T00:49:21.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.403+0000 7f1f9d020640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f98162490 con 0x7f1f9815dfc0 2024-03-24T00:49:21.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.403+0000 7f1f9d020640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f98162770 con 0x7f1f980cf050 2024-03-24T00:49:21.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.403+0000 7f1f9659c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1f980d10b0 0x7f1f980ceb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:21.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.403+0000 7f1f9659c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1f980d10b0 0x7f1f980ceb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46528/0 (socket says 172.21.15.67:46528) 2024-03-24T00:49:21.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.403+0000 7f1f9659c640 1 -- 172.21.15.67:0/1466668662 learned_addr learned my addr 172.21.15.67:0/1466668662 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:21.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.403+0000 7f1f96d9d640 1 --2- 172.21.15.67:0/1466668662 >> v2:172.21.15.72:3300/0 conn(0x7f1f9815dfc0 0x7f1f98161bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:21.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.403+0000 7f1f9759e640 1 --2- 172.21.15.67:0/1466668662 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1f980cf050 0x7f1f980b57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:21.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.403+0000 7f1f9659c640 1 -- 172.21.15.67:0/1466668662 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1f980cf050 msgr2=0x7f1f980b57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:21.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.403+0000 7f1f9659c640 1 --2- 172.21.15.67:0/1466668662 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1f980cf050 0x7f1f980b57c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:21.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.403+0000 7f1f9659c640 1 -- 172.21.15.67:0/1466668662 >> v2:172.21.15.72:3300/0 conn(0x7f1f9815dfc0 msgr2=0x7f1f98161bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:21.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.403+0000 7f1f9659c640 1 --2- 172.21.15.67:0/1466668662 >> v2:172.21.15.72:3300/0 conn(0x7f1f9815dfc0 0x7f1f98161bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:21.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.403+0000 7f1f9659c640 1 -- 172.21.15.67:0/1466668662 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f980d1f40 con 0x7f1f980d10b0 2024-03-24T00:49:21.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.403+0000 7f1f96d9d640 1 --2- 172.21.15.67:0/1466668662 >> v2:172.21.15.72:3300/0 conn(0x7f1f9815dfc0 0x7f1f98161bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:49:21.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.404+0000 7f1f9659c640 1 --2- 172.21.15.67:0/1466668662 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f980d10b0 0x7f1f980ceb10 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f1f9806b250 tx=0x7f1f8c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1a331547b6dd70a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:21.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.404+0000 7f1f95d9b640 1 -- 172.21.15.67:0/1466668662 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8c035020 con 0x7f1f980d10b0 2024-03-24T00:49:21.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.404+0000 7f1f95d9b640 1 -- 172.21.15.67:0/1466668662 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f8c031870 con 0x7f1f980d10b0 2024-03-24T00:49:21.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.404+0000 7f1f95d9b640 1 -- 172.21.15.67:0/1466668662 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8c031c40 con 0x7f1f980d10b0 2024-03-24T00:49:21.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.405+0000 7f1f9d020640 1 -- 172.21.15.67:0/1466668662 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f980d10b0 msgr2=0x7f1f980ceb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:21.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.405+0000 7f1f9d020640 1 --2- 172.21.15.67:0/1466668662 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f980d10b0 0x7f1f980ceb10 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f1f9806b250 tx=0x7f1f8c02b910 comp rx=0 tx=0).stop 2024-03-24T00:49:21.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.405+0000 7f1f9d020640 1 -- 172.21.15.67:0/1466668662 shutdown_connections 2024-03-24T00:49:21.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.405+0000 7f1f9d020640 1 --2- 172.21.15.67:0/1466668662 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1f980cf050 0x7f1f980b57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:21.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.405+0000 7f1f9d020640 1 --2- 172.21.15.67:0/1466668662 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f980d10b0 0x7f1f980ceb10 unknown :-1 s=CLOSED pgs=1804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:21.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.405+0000 7f1f9d020640 1 --2- 172.21.15.67:0/1466668662 >> v2:172.21.15.72:3300/0 conn(0x7f1f9815dfc0 0x7f1f98161bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:21.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.405+0000 7f1f9d020640 1 -- 172.21.15.67:0/1466668662 >> 172.21.15.67:0/1466668662 conn(0x7f1f9806c930 msgr2=0x7f1f9806cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:21.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.405+0000 7f1f9d020640 1 -- 172.21.15.67:0/1466668662 shutdown_connections 2024-03-24T00:49:21.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.405+0000 7f1f9d020640 1 -- 172.21.15.67:0/1466668662 wait complete. 2024-03-24T00:49:21.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.406+0000 7f1f9d020640 1 Processor -- start 2024-03-24T00:49:21.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.406+0000 7f1f9d020640 1 -- start start 2024-03-24T00:49:21.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.407+0000 7f1f9d020640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f980cf050 0x7f1f98151a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:21.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.407+0000 7f1f9d020640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f980d10b0 0x7f1f980ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:21.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.407+0000 7f1f9d020640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1f9815dfc0 0x7f1f980cd470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:21.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.407+0000 7f1f9d020640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f98162270 con 0x7f1f980d10b0 2024-03-24T00:49:21.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.407+0000 7f1f9d020640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f981620f0 con 0x7f1f980cf050 2024-03-24T00:49:21.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.407+0000 7f1f9d020640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9815e3a0 con 0x7f1f9815dfc0 2024-03-24T00:49:21.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.407+0000 7f1f9659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f980d10b0 0x7f1f980ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:21.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.408+0000 7f1f9659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f980d10b0 0x7f1f980ccf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46538/0 (socket says 172.21.15.67:46538) 2024-03-24T00:49:21.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.408+0000 7f1f9659c640 1 -- 172.21.15.67:0/2268041566 learned_addr learned my addr 172.21.15.67:0/2268041566 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:21.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.408+0000 7f1f96d9d640 1 --2- 172.21.15.67:0/2268041566 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f980cf050 0x7f1f98151a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:21.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.408+0000 7f1f9759e640 1 --2- 172.21.15.67:0/2268041566 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1f9815dfc0 0x7f1f980cd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:21.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.408+0000 7f1f9659c640 1 -- 172.21.15.67:0/2268041566 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1f9815dfc0 msgr2=0x7f1f980cd470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:21.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.408+0000 7f1f9659c640 1 --2- 172.21.15.67:0/2268041566 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1f9815dfc0 0x7f1f980cd470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:21.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.408+0000 7f1f9659c640 1 -- 172.21.15.67:0/2268041566 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f980cf050 msgr2=0x7f1f98151a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:21.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.408+0000 7f1f9659c640 1 --2- 172.21.15.67:0/2268041566 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f980cf050 0x7f1f98151a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:21.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.408+0000 7f1f9659c640 1 -- 172.21.15.67:0/2268041566 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f980c7a80 con 0x7f1f980d10b0 2024-03-24T00:49:21.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.408+0000 7f1f96d9d640 1 --2- 172.21.15.67:0/2268041566 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f980cf050 0x7f1f98151a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:21.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.408+0000 7f1f9659c640 1 --2- 172.21.15.67:0/2268041566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f980d10b0 0x7f1f980ccf30 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c000c00 tx=0x7f1f8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:21.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.408+0000 7f1f7f7fe640 1 -- 172.21.15.67:0/2268041566 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8c040070 con 0x7f1f980d10b0 2024-03-24T00:49:21.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.408+0000 7f1f7f7fe640 1 -- 172.21.15.67:0/2268041566 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f8c03e070 con 0x7f1f980d10b0 2024-03-24T00:49:21.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.408+0000 7f1f7f7fe640 1 -- 172.21.15.67:0/2268041566 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8c035040 con 0x7f1f980d10b0 2024-03-24T00:49:21.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.408+0000 7f1f9d020640 1 -- 172.21.15.67:0/2268041566 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f980c7d60 con 0x7f1f980d10b0 2024-03-24T00:49:21.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.408+0000 7f1f9d020640 1 -- 172.21.15.67:0/2268041566 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f98151fd0 con 0x7f1f980d10b0 2024-03-24T00:49:21.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.410+0000 7f1f9d020640 1 -- 172.21.15.67:0/2268041566 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f9804f400 con 0x7f1f980d10b0 2024-03-24T00:49:21.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.410+0000 7f1f7f7fe640 1 -- 172.21.15.67:0/2268041566 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1f8c038020 con 0x7f1f980d10b0 2024-03-24T00:49:21.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.410+0000 7f1f7f7fe640 1 --2- 172.21.15.67:0/2268041566 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1f8406bc00 0x7f1f8406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:21.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.410+0000 7f1f7f7fe640 1 -- 172.21.15.67:0/2268041566 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f1f8c0aca10 con 0x7f1f980d10b0 2024-03-24T00:49:21.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.410+0000 7f1f96d9d640 1 --2- 172.21.15.67:0/2268041566 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1f8406bc00 0x7f1f8406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:21.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.411+0000 7f1f96d9d640 1 --2- 172.21.15.67:0/2268041566 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1f8406bc00 0x7f1f8406e0c0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f1f980ca5a0 tx=0x7f1f88008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:21.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.414+0000 7f1f7f7fe640 1 -- 172.21.15.67:0/2268041566 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1f8c02f320 con 0x7f1f980d10b0 2024-03-24T00:49:21.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.778+0000 7f1f9d020640 1 -- 172.21.15.67:0/2268041566 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f1f980d1a50 con 0x7f1f980d10b0 2024-03-24T00:49:21.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.778+0000 7f1f7f7fe640 1 -- 172.21.15.67:0/2268041566 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v121) v1 ==== 80+0+120 (secure 0 0 0) 0x7f1f8c07b3b0 con 0x7f1f980d10b0 2024-03-24T00:49:21.780 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6835/312063805 2024-03-25T00:49:12.456364+0000 2024-03-24T00:49:21.780 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6833/312063805 2024-03-25T00:49:12.456364+0000 2024-03-24T00:49:21.780 INFO:teuthology.orchestra.run.smithi067.stderr:listed 2 entries 2024-03-24T00:49:21.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.781+0000 7f1f9d020640 1 -- 172.21.15.67:0/2268041566 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1f8406bc00 msgr2=0x7f1f8406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:21.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.781+0000 7f1f9d020640 1 --2- 172.21.15.67:0/2268041566 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1f8406bc00 0x7f1f8406e0c0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f1f980ca5a0 tx=0x7f1f88008040 comp rx=0 tx=0).stop 2024-03-24T00:49:21.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.781+0000 7f1f9d020640 1 -- 172.21.15.67:0/2268041566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f980d10b0 msgr2=0x7f1f980ccf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:21.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.781+0000 7f1f9d020640 1 --2- 172.21.15.67:0/2268041566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f980d10b0 0x7f1f980ccf30 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c000c00 tx=0x7f1f8c0046c0 comp rx=0 tx=0).stop 2024-03-24T00:49:21.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.781+0000 7f1f9d020640 1 -- 172.21.15.67:0/2268041566 shutdown_connections 2024-03-24T00:49:21.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.781+0000 7f1f9d020640 1 --2- 172.21.15.67:0/2268041566 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1f8406bc00 0x7f1f8406e0c0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:21.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.781+0000 7f1f9d020640 1 --2- 172.21.15.67:0/2268041566 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1f9815dfc0 0x7f1f980cd470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:21.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.781+0000 7f1f9d020640 1 --2- 172.21.15.67:0/2268041566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f980d10b0 0x7f1f980ccf30 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:21.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.781+0000 7f1f9d020640 1 --2- 172.21.15.67:0/2268041566 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f980cf050 0x7f1f98151a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:21.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.781+0000 7f1f9d020640 1 -- 172.21.15.67:0/2268041566 >> 172.21.15.67:0/2268041566 conn(0x7f1f9806c930 msgr2=0x7f1f980b86f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:21.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.781+0000 7f1f9d020640 1 -- 172.21.15.67:0/2268041566 shutdown_connections 2024-03-24T00:49:21.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:21.781+0000 7f1f9d020640 1 -- 172.21.15.67:0/2268041566 wait complete. 2024-03-24T00:49:21.793 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-24T00:49:21.793 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:21.793 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-24T00:49:21.823 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-03-24T00:49:21.823 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:21.823 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-24T00:49:21.879 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:49:22.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.018+0000 7f0217bf4640 1 Processor -- start 2024-03-24T00:49:22.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.019+0000 7f0217bf4640 1 -- start start 2024-03-24T00:49:22.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.019+0000 7f0217bf4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f021015dfc0 0x7f0210161bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.019+0000 7f0217bf4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f02100d10b0 0x7f02100ceb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.019+0000 7f0217bf4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02100cf050 0x7f02100b57c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.019+0000 7f0217bf4640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0210162a50 con 0x7f02100d10b0 2024-03-24T00:49:22.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.019+0000 7f0217bf4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0210162490 con 0x7f021015dfc0 2024-03-24T00:49:22.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.019+0000 7f0217bf4640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0210162770 con 0x7f02100cf050 2024-03-24T00:49:22.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.020+0000 7f021518f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f02100d10b0 0x7f02100ceb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.020+0000 7f021518f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f02100d10b0 0x7f02100ceb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46554/0 (socket says 172.21.15.67:46554) 2024-03-24T00:49:22.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.020+0000 7f021518f640 1 -- 172.21.15.67:0/2336404422 learned_addr learned my addr 172.21.15.67:0/2336404422 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:22.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.020+0000 7f0216191640 1 --2- 172.21.15.67:0/2336404422 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02100cf050 0x7f02100b57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.020+0000 7f0215990640 1 --2- 172.21.15.67:0/2336404422 >> v2:172.21.15.72:3300/0 conn(0x7f021015dfc0 0x7f0210161bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.020+0000 7f021518f640 1 -- 172.21.15.67:0/2336404422 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02100cf050 msgr2=0x7f02100b57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:22.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.020+0000 7f021518f640 1 --2- 172.21.15.67:0/2336404422 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02100cf050 0x7f02100b57c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.020+0000 7f021518f640 1 -- 172.21.15.67:0/2336404422 >> v2:172.21.15.72:3300/0 conn(0x7f021015dfc0 msgr2=0x7f0210161bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:22.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.020+0000 7f021518f640 1 --2- 172.21.15.67:0/2336404422 >> v2:172.21.15.72:3300/0 conn(0x7f021015dfc0 0x7f0210161bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.020+0000 7f021518f640 1 -- 172.21.15.67:0/2336404422 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02100d1f40 con 0x7f02100d10b0 2024-03-24T00:49:22.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.021+0000 7f021518f640 1 --2- 172.21.15.67:0/2336404422 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02100d10b0 0x7f02100ceb10 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f021006b250 tx=0x7f020002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=295834771658bd1b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:22.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.021+0000 7f021498e640 1 -- 172.21.15.67:0/2336404422 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0200035020 con 0x7f02100d10b0 2024-03-24T00:49:22.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.021+0000 7f021498e640 1 -- 172.21.15.67:0/2336404422 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0200031870 con 0x7f02100d10b0 2024-03-24T00:49:22.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.021+0000 7f021498e640 1 -- 172.21.15.67:0/2336404422 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0200031c40 con 0x7f02100d10b0 2024-03-24T00:49:22.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.021+0000 7f0217bf4640 1 -- 172.21.15.67:0/2336404422 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02100d10b0 msgr2=0x7f02100ceb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:22.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.021+0000 7f0217bf4640 1 --2- 172.21.15.67:0/2336404422 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02100d10b0 0x7f02100ceb10 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f021006b250 tx=0x7f020002b910 comp rx=0 tx=0).stop 2024-03-24T00:49:22.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.021+0000 7f0217bf4640 1 -- 172.21.15.67:0/2336404422 shutdown_connections 2024-03-24T00:49:22.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.021+0000 7f0217bf4640 1 --2- 172.21.15.67:0/2336404422 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02100cf050 0x7f02100b57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.021+0000 7f0217bf4640 1 --2- 172.21.15.67:0/2336404422 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02100d10b0 0x7f02100ceb10 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.021+0000 7f0217bf4640 1 --2- 172.21.15.67:0/2336404422 >> v2:172.21.15.72:3300/0 conn(0x7f021015dfc0 0x7f0210161bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.021+0000 7f0217bf4640 1 -- 172.21.15.67:0/2336404422 >> 172.21.15.67:0/2336404422 conn(0x7f021006c930 msgr2=0x7f021006cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:22.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.021+0000 7f0217bf4640 1 -- 172.21.15.67:0/2336404422 shutdown_connections 2024-03-24T00:49:22.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.021+0000 7f0217bf4640 1 -- 172.21.15.67:0/2336404422 wait complete. 2024-03-24T00:49:22.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.023+0000 7f0217bf4640 1 Processor -- start 2024-03-24T00:49:22.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.023+0000 7f0217bf4640 1 -- start start 2024-03-24T00:49:22.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.023+0000 7f0217bf4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02100cf050 0x7f021015b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.023+0000 7f0217bf4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02100d10b0 0x7f02100b4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.023+0000 7f0217bf4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f021015dfc0 0x7f02100b5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.023+0000 7f0217bf4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0210162270 con 0x7f021015dfc0 2024-03-24T00:49:22.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.023+0000 7f0217bf4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02101620f0 con 0x7f02100d10b0 2024-03-24T00:49:22.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.023+0000 7f0217bf4640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f021015e3a0 con 0x7f02100cf050 2024-03-24T00:49:22.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.023+0000 7f0216191640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f021015dfc0 0x7f02100b5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.023+0000 7f0216191640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f021015dfc0 0x7f02100b5320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46566/0 (socket says 172.21.15.67:46566) 2024-03-24T00:49:22.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.024+0000 7f0216191640 1 -- 172.21.15.67:0/689302591 learned_addr learned my addr 172.21.15.67:0/689302591 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:22.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.024+0000 7f0215990640 1 --2- 172.21.15.67:0/689302591 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02100cf050 0x7f021015b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.024+0000 7f021518f640 1 --2- 172.21.15.67:0/689302591 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02100d10b0 0x7f02100b4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.024+0000 7f0216191640 1 -- 172.21.15.67:0/689302591 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02100cf050 msgr2=0x7f021015b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:22.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.024+0000 7f0216191640 1 --2- 172.21.15.67:0/689302591 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02100cf050 0x7f021015b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.024+0000 7f0216191640 1 -- 172.21.15.67:0/689302591 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02100d10b0 msgr2=0x7f02100b4de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:22.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.024+0000 7f0216191640 1 --2- 172.21.15.67:0/689302591 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02100d10b0 0x7f02100b4de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.024+0000 7f0216191640 1 -- 172.21.15.67:0/689302591 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02100b4a70 con 0x7f021015dfc0 2024-03-24T00:49:22.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.024+0000 7f021518f640 1 --2- 172.21.15.67:0/689302591 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02100d10b0 0x7f02100b4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:49:22.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.024+0000 7f0216191640 1 --2- 172.21.15.67:0/689302591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f021015dfc0 0x7f02100b5320 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f020800c4c0 tx=0x7f020800c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:22.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.024+0000 7f02067fc640 1 -- 172.21.15.67:0/689302591 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0208015070 con 0x7f021015dfc0 2024-03-24T00:49:22.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.024+0000 7f02067fc640 1 -- 172.21.15.67:0/689302591 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0208004220 con 0x7f021015dfc0 2024-03-24T00:49:22.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.024+0000 7f0217bf4640 1 -- 172.21.15.67:0/689302591 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02100b5950 con 0x7f021015dfc0 2024-03-24T00:49:22.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.024+0000 7f02067fc640 1 -- 172.21.15.67:0/689302591 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0208003610 con 0x7f021015dfc0 2024-03-24T00:49:22.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.024+0000 7f0217bf4640 1 -- 172.21.15.67:0/689302591 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02100b5f10 con 0x7f021015dfc0 2024-03-24T00:49:22.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.025+0000 7f0217bf4640 1 -- 172.21.15.67:0/689302591 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f021004f400 con 0x7f021015dfc0 2024-03-24T00:49:22.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.026+0000 7f02067fc640 1 -- 172.21.15.67:0/689302591 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f02080037b0 con 0x7f021015dfc0 2024-03-24T00:49:22.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.026+0000 7f02067fc640 1 --2- 172.21.15.67:0/689302591 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f01f406bbd0 0x7f01f406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.026+0000 7f02067fc640 1 -- 172.21.15.67:0/689302591 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f020808a8a0 con 0x7f021015dfc0 2024-03-24T00:49:22.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.026+0000 7f0215990640 1 --2- 172.21.15.67:0/689302591 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f01f406bbd0 0x7f01f406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.027+0000 7f0215990640 1 --2- 172.21.15.67:0/689302591 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f01f406bbd0 0x7f01f406e090 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f020c00f360 tx=0x7f020c007710 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:22.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.031+0000 7f02067fc640 1 -- 172.21.15.67:0/689302591 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0208058bf0 con 0x7f021015dfc0 2024-03-24T00:49:22.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.328+0000 7f0217bf4640 1 -- 172.21.15.67:0/689302591 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02100d1a50 con 0x7f021015dfc0 2024-03-24T00:49:22.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.328+0000 7f02067fc640 1 -- 172.21.15.67:0/689302591 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+4978 (secure 0 0 0) 0x7f020805cc90 con 0x7f021015dfc0 2024-03-24T00:49:22.329 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:22.330 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":92,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7298,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/839277571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":839277571},{"type":"v1","addr":"172.21.15.72:6836","nonce":839277571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":8224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2052380377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2052380377},{"type":"v1","addr":"172.21.15.67:6834","nonce":2052380377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":8239,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2196618204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2196618204},{"type":"v1","addr":"172.21.15.72:6837","nonce":2196618204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:48:54.351575+0000","modified":"2024-03-24T00:49:16.485067+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":121,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9510},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9510":{"gid":9510,"name":"a","rank":0,"incarnation":89,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6835/3833808901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":3833808901},{"type":"v1","addr":"172.21.15.67:6835","nonce":3833808901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9510,"qdb_cluster":[9510]},"id":8}]} 2024-03-24T00:49:22.330 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 92 2024-03-24T00:49:22.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.331+0000 7f0217bf4640 1 -- 172.21.15.67:0/689302591 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f01f406bbd0 msgr2=0x7f01f406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:22.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.331+0000 7f0217bf4640 1 --2- 172.21.15.67:0/689302591 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f01f406bbd0 0x7f01f406e090 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f020c00f360 tx=0x7f020c007710 comp rx=0 tx=0).stop 2024-03-24T00:49:22.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.331+0000 7f0217bf4640 1 -- 172.21.15.67:0/689302591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f021015dfc0 msgr2=0x7f02100b5320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:22.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.331+0000 7f0217bf4640 1 --2- 172.21.15.67:0/689302591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f021015dfc0 0x7f02100b5320 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f020800c4c0 tx=0x7f020800c980 comp rx=0 tx=0).stop 2024-03-24T00:49:22.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.331+0000 7f0217bf4640 1 -- 172.21.15.67:0/689302591 shutdown_connections 2024-03-24T00:49:22.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.331+0000 7f0217bf4640 1 --2- 172.21.15.67:0/689302591 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f01f406bbd0 0x7f01f406e090 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.331+0000 7f0217bf4640 1 --2- 172.21.15.67:0/689302591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f021015dfc0 0x7f02100b5320 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.331+0000 7f0217bf4640 1 --2- 172.21.15.67:0/689302591 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02100d10b0 0x7f02100b4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.331+0000 7f0217bf4640 1 --2- 172.21.15.67:0/689302591 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f02100cf050 0x7f021015b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.331+0000 7f0217bf4640 1 -- 172.21.15.67:0/689302591 >> 172.21.15.67:0/689302591 conn(0x7f021006c930 msgr2=0x7f02100ce7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:22.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.332+0000 7f0217bf4640 1 -- 172.21.15.67:0/689302591 shutdown_connections 2024-03-24T00:49:22.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.332+0000 7f0217bf4640 1 -- 172.21.15.67:0/689302591 wait complete. 2024-03-24T00:49:22.344 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json tag path /parent mytag123 2024-03-24T00:49:22.466 INFO:teuthology.orchestra.run.smithi067.stdout:{"return_code":0,"scrub_tag":"mytag123","mode":"asynchronous"} 2024-03-24T00:49:22.467 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mode": "asynchronous", "return_code": 0, "scrub_tag": "mytag123" } 2024-03-24T00:49:22.467 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 100000001f7.00000000 scrub_tag 2024-03-24T00:49:22.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.511+0000 7f52aa641f40 1 Processor -- start 2024-03-24T00:49:22.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.511+0000 7f52aa641f40 1 -- start start 2024-03-24T00:49:22.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.512+0000 7f52aa641f40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d25b1592b0 0x55d25b13e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.512+0000 7f52aa641f40 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d25b1e5cf0 0x55d25b1dd860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.512+0000 7f52aa641f40 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d25b1e69c0 0x55d25b1ddda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.512+0000 7f52aa641f40 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d25b1e6460 con 0x55d25b1e5cf0 2024-03-24T00:49:22.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.512+0000 7f52aa641f40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d25b0b70d0 con 0x55d25b1592b0 2024-03-24T00:49:22.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.512+0000 7f52aa641f40 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d25b1e6700 con 0x55d25b1e69c0 2024-03-24T00:49:22.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.512+0000 7f52a70c2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d25b1e5cf0 0x55d25b1dd860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.512+0000 7f52a70c2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d25b1e5cf0 0x55d25b1dd860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46580/0 (socket says 172.21.15.67:46580) 2024-03-24T00:49:22.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.512+0000 7f52a70c2640 1 -- 172.21.15.67:0/4272212253 learned_addr learned my addr 172.21.15.67:0/4272212253 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:22.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.512+0000 7f52a78c3640 1 --2- 172.21.15.67:0/4272212253 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d25b1592b0 0x55d25b13e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.512+0000 7f52a80c4640 1 --2- 172.21.15.67:0/4272212253 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d25b1e69c0 0x55d25b1ddda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.512+0000 7f52a70c2640 1 -- 172.21.15.67:0/4272212253 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d25b1e69c0 msgr2=0x55d25b1ddda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:22.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.512+0000 7f52a70c2640 1 --2- 172.21.15.67:0/4272212253 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d25b1e69c0 0x55d25b1ddda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.512+0000 7f52a70c2640 1 -- 172.21.15.67:0/4272212253 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d25b1592b0 msgr2=0x55d25b13e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:22.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.512+0000 7f52a70c2640 1 --2- 172.21.15.67:0/4272212253 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d25b1592b0 0x55d25b13e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.512+0000 7f52a70c2640 1 -- 172.21.15.67:0/4272212253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d25b0fa9f0 con 0x55d25b1e5cf0 2024-03-24T00:49:22.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.513+0000 7f52a70c2640 1 --2- 172.21.15.67:0/4272212253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d25b1e5cf0 0x55d25b1dd860 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f529c00a440 tx=0x7f529c04c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcbca049949008c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:22.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.513+0000 7f52a60c0640 1 -- 172.21.15.67:0/4272212253 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f529c057030 con 0x55d25b1e5cf0 2024-03-24T00:49:22.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.513+0000 7f52a60c0640 1 -- 172.21.15.67:0/4272212253 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f529c061070 con 0x55d25b1e5cf0 2024-03-24T00:49:22.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.513+0000 7f52a60c0640 1 -- 172.21.15.67:0/4272212253 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f529c052920 con 0x55d25b1e5cf0 2024-03-24T00:49:22.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.513+0000 7f52aa641f40 1 -- 172.21.15.67:0/4272212253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d25b1e5cf0 msgr2=0x55d25b1dd860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:22.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.513+0000 7f52aa641f40 1 --2- 172.21.15.67:0/4272212253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d25b1e5cf0 0x55d25b1dd860 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f529c00a440 tx=0x7f529c04c000 comp rx=0 tx=0).stop 2024-03-24T00:49:22.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.513+0000 7f52aa641f40 1 -- 172.21.15.67:0/4272212253 shutdown_connections 2024-03-24T00:49:22.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.513+0000 7f52aa641f40 1 --2- 172.21.15.67:0/4272212253 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d25b1e69c0 0x55d25b1ddda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.513+0000 7f52aa641f40 1 --2- 172.21.15.67:0/4272212253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d25b1e5cf0 0x55d25b1dd860 unknown :-1 s=CLOSED pgs=1809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.513+0000 7f52aa641f40 1 --2- 172.21.15.67:0/4272212253 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d25b1592b0 0x55d25b13e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.513+0000 7f52aa641f40 1 -- 172.21.15.67:0/4272212253 >> 172.21.15.67:0/4272212253 conn(0x55d25b110650 msgr2=0x55d25b1e82f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:22.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.513+0000 7f52aa641f40 1 -- 172.21.15.67:0/4272212253 shutdown_connections 2024-03-24T00:49:22.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.513+0000 7f52aa641f40 1 -- 172.21.15.67:0/4272212253 wait complete. 2024-03-24T00:49:22.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.514+0000 7f52aa641f40 1 Processor -- start 2024-03-24T00:49:22.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.514+0000 7f52aa641f40 1 -- start start 2024-03-24T00:49:22.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.515+0000 7f52aa641f40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d25b1592b0 0x55d25b1d3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.515+0000 7f52aa641f40 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d25b1e5cf0 0x55d25b1e7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.515+0000 7f52aa641f40 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d25b1e69c0 0x55d25b1c0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.515+0000 7f52aa641f40 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d25b0facb0 con 0x55d25b1e69c0 2024-03-24T00:49:22.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.515+0000 7f52aa641f40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d25b1deff0 con 0x55d25b1592b0 2024-03-24T00:49:22.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.515+0000 7f52aa641f40 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d25b1df760 con 0x55d25b1e5cf0 2024-03-24T00:49:22.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.515+0000 7f52a80c4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d25b1e69c0 0x55d25b1c0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.515+0000 7f52a80c4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d25b1e69c0 0x55d25b1c0c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46588/0 (socket says 172.21.15.67:46588) 2024-03-24T00:49:22.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.515+0000 7f52a80c4640 1 -- 172.21.15.67:0/3440215688 learned_addr learned my addr 172.21.15.67:0/3440215688 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:22.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.515+0000 7f52a78c3640 1 --2- 172.21.15.67:0/3440215688 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d25b1592b0 0x55d25b1d3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.515+0000 7f52a70c2640 1 --2- 172.21.15.67:0/3440215688 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d25b1e5cf0 0x55d25b1e7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.515+0000 7f52a80c4640 1 -- 172.21.15.67:0/3440215688 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d25b1e5cf0 msgr2=0x55d25b1e7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:22.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.515+0000 7f52a80c4640 1 --2- 172.21.15.67:0/3440215688 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55d25b1e5cf0 0x55d25b1e7230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.515+0000 7f52a80c4640 1 -- 172.21.15.67:0/3440215688 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d25b1592b0 msgr2=0x55d25b1d3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:22.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.515+0000 7f52a80c4640 1 --2- 172.21.15.67:0/3440215688 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d25b1592b0 0x55d25b1d3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.515+0000 7f52a80c4640 1 -- 172.21.15.67:0/3440215688 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d25b1dd8b0 con 0x55d25b1e69c0 2024-03-24T00:49:22.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.515+0000 7f52a80c4640 1 --2- 172.21.15.67:0/3440215688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d25b1e69c0 0x55d25b1c0c30 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f52a00028d0 tx=0x7f52a0002d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:22.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.515+0000 7f52a50be640 1 -- 172.21.15.67:0/3440215688 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a0017070 con 0x55d25b1e69c0 2024-03-24T00:49:22.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.515+0000 7f52a50be640 1 -- 172.21.15.67:0/3440215688 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52a000f9c0 con 0x55d25b1e69c0 2024-03-24T00:49:22.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.515+0000 7f52a50be640 1 -- 172.21.15.67:0/3440215688 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a000fdd0 con 0x55d25b1e69c0 2024-03-24T00:49:22.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.515+0000 7f52aa641f40 1 -- 172.21.15.67:0/3440215688 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55d25b1ddb90 con 0x55d25b1e69c0 2024-03-24T00:49:22.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.516+0000 7f52aa641f40 1 -- 172.21.15.67:0/3440215688 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d25b1de150 con 0x55d25b1e69c0 2024-03-24T00:49:22.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:49:22.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.517+0000 7f52a50be640 1 -- 172.21.15.67:0/3440215688 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f52a001a070 con 0x55d25b1e69c0 2024-03-24T00:49:22.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.517+0000 7f52a50be640 1 --2- 172.21.15.67:0/3440215688 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f529406a5e0 0x7f529406caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.517+0000 7f52a50be640 1 -- 172.21.15.67:0/3440215688 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f52a008a0c0 con 0x55d25b1e69c0 2024-03-24T00:49:22.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.517+0000 7f52a78c3640 1 --2- 172.21.15.67:0/3440215688 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f529406a5e0 0x7f529406caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.517+0000 7f52aa641f40 1 --2- 172.21.15.67:0/3440215688 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x55d25b1e7770 0x55d25b1e2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.517+0000 7f52aa641f40 1 -- 172.21.15.67:0/3440215688 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:1 18.36 18:6fe38dc2:::100000001f7.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e121) v9 -- 0x55d25b1dd3b0 con 0x55d25b1e7770 2024-03-24T00:49:22.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.517+0000 7f52a70c2640 1 --2- 172.21.15.67:0/3440215688 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x55d25b1e7770 0x55d25b1e2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.518+0000 7f52a78c3640 1 --2- 172.21.15.67:0/3440215688 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f529406a5e0 0x7f529406caa0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f5298001ed0 tx=0x7f52980019f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:22.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.518+0000 7f52a70c2640 1 --2- 172.21.15.67:0/3440215688 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x55d25b1e7770 0x55d25b1e2e00 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:22.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.518+0000 7f52a70c2640 1 -- 172.21.15.67:0/3440215688 <== osd.1 v2:172.21.15.67:6800/4062510404 1 ==== osd_op_reply(1 100000001f7.00000000 [getxattr] v0'0 uv0 ondisk = -61 ((61) No data available)) v8 ==== 164+0+0 (crc 0 0 0) 0x7f529c05a070 con 0x55d25b1e7770 2024-03-24T00:49:22.524 INFO:teuthology.orchestra.run.smithi067.stderr:error getting xattr cephfs_data/100000001f7.00000000/scrub_tag: (61) No data available 2024-03-24T00:49:22.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.518+0000 7f52aa641f40 1 -- 172.21.15.67:0/3440215688 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x55d25b1e7770 msgr2=0x55d25b1e2e00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:22.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.518+0000 7f52aa641f40 1 --2- 172.21.15.67:0/3440215688 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x55d25b1e7770 0x55d25b1e2e00 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.518+0000 7f52aa641f40 1 -- 172.21.15.67:0/3440215688 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f529406a5e0 msgr2=0x7f529406caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:22.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.518+0000 7f52aa641f40 1 --2- 172.21.15.67:0/3440215688 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f529406a5e0 0x7f529406caa0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f5298001ed0 tx=0x7f52980019f0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.518+0000 7f52aa641f40 1 -- 172.21.15.67:0/3440215688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d25b1e69c0 msgr2=0x55d25b1c0c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:22.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.518+0000 7f52aa641f40 1 --2- 172.21.15.67:0/3440215688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d25b1e69c0 0x55d25b1c0c30 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f52a00028d0 tx=0x7f52a0002d90 comp rx=0 tx=0).stop 2024-03-24T00:49:22.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.518+0000 7f52a80c4640 1 -- 172.21.15.67:0/3440215688 reap_dead start 2024-03-24T00:49:22.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.518+0000 7f52aa641f40 1 -- 172.21.15.67:0/3440215688 shutdown_connections 2024-03-24T00:49:22.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.518+0000 7f52aa641f40 1 -- 172.21.15.67:0/3440215688 >> 172.21.15.67:0/3440215688 conn(0x55d25b110650 msgr2=0x55d25b1e80e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:22.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.518+0000 7f52aa641f40 1 -- 172.21.15.67:0/3440215688 shutdown_connections 2024-03-24T00:49:22.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.518+0000 7f52aa641f40 1 -- 172.21.15.67:0/3440215688 wait complete. 2024-03-24T00:49:22.525 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000002.00000000 scrub_tag 2024-03-24T00:49:22.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.567+0000 7fc34300bf40 1 Processor -- start 2024-03-24T00:49:22.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.567+0000 7fc34300bf40 1 -- start start 2024-03-24T00:49:22.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.567+0000 7fc34300bf40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c0d6062e20 0x55c0d608f940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.567+0000 7fc34300bf40 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c0d608b1c0 0x55c0d6060950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.567+0000 7fc34300bf40 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55c0d6090290 0x55c0d6118410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.567+0000 7fc34300bf40 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0d611bdc0 con 0x55c0d608b1c0 2024-03-24T00:49:22.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.567+0000 7fc34300bf40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0d60070d0 con 0x55c0d6062e20 2024-03-24T00:49:22.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.567+0000 7fc34300bf40 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c0d611bf40 con 0x55c0d6090290 2024-03-24T00:49:22.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.568+0000 7fc33f64b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c0d608b1c0 0x55c0d6060950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.568+0000 7fc33f64b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c0d608b1c0 0x55c0d6060950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46600/0 (socket says 172.21.15.67:46600) 2024-03-24T00:49:22.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.568+0000 7fc33f64b640 1 -- 172.21.15.67:0/2453750064 learned_addr learned my addr 172.21.15.67:0/2453750064 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:22.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.568+0000 7fc33f64b640 1 -- 172.21.15.67:0/2453750064 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55c0d6090290 msgr2=0x55c0d6118410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:22.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.568+0000 7fc33f64b640 1 --2- 172.21.15.67:0/2453750064 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55c0d6090290 0x55c0d6118410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.568+0000 7fc33f64b640 1 -- 172.21.15.67:0/2453750064 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c0d6062e20 msgr2=0x55c0d608f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:22.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.568+0000 7fc33fe4c640 1 --2- 172.21.15.67:0/2453750064 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c0d6062e20 0x55c0d608f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.568+0000 7fc33f64b640 1 --2- 172.21.15.67:0/2453750064 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c0d6062e20 0x55c0d608f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.568+0000 7fc33f64b640 1 -- 172.21.15.67:0/2453750064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c0d60a4f20 con 0x55c0d608b1c0 2024-03-24T00:49:22.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.568+0000 7fc33fe4c640 1 --2- 172.21.15.67:0/2453750064 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c0d6062e20 0x55c0d608f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:22.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.568+0000 7fc33f64b640 1 --2- 172.21.15.67:0/2453750064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c0d608b1c0 0x55c0d6060950 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7fc32800a4f0 tx=0x7fc32804c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ba68f12d95dcb29 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:22.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.568+0000 7fc33e649640 1 -- 172.21.15.67:0/2453750064 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc328057040 con 0x55c0d608b1c0 2024-03-24T00:49:22.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.568+0000 7fc33e649640 1 -- 172.21.15.67:0/2453750064 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3280527f0 con 0x55c0d608b1c0 2024-03-24T00:49:22.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.568+0000 7fc33e649640 1 -- 172.21.15.67:0/2453750064 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc328052be0 con 0x55c0d608b1c0 2024-03-24T00:49:22.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.569+0000 7fc34300bf40 1 -- 172.21.15.67:0/2453750064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c0d608b1c0 msgr2=0x55c0d6060950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:22.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.569+0000 7fc34300bf40 1 --2- 172.21.15.67:0/2453750064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c0d608b1c0 0x55c0d6060950 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7fc32800a4f0 tx=0x7fc32804c000 comp rx=0 tx=0).stop 2024-03-24T00:49:22.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.569+0000 7fc34300bf40 1 -- 172.21.15.67:0/2453750064 shutdown_connections 2024-03-24T00:49:22.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.569+0000 7fc34300bf40 1 --2- 172.21.15.67:0/2453750064 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55c0d6090290 0x55c0d6118410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.569+0000 7fc34300bf40 1 --2- 172.21.15.67:0/2453750064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c0d608b1c0 0x55c0d6060950 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.569+0000 7fc34300bf40 1 --2- 172.21.15.67:0/2453750064 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c0d6062e20 0x55c0d608f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.569+0000 7fc34300bf40 1 -- 172.21.15.67:0/2453750064 >> 172.21.15.67:0/2453750064 conn(0x55c0d608d770 msgr2=0x55c0d60a8f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:22.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.569+0000 7fc34300bf40 1 -- 172.21.15.67:0/2453750064 shutdown_connections 2024-03-24T00:49:22.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.569+0000 7fc34300bf40 1 -- 172.21.15.67:0/2453750064 wait complete. 2024-03-24T00:49:22.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.569+0000 7fc34300bf40 1 Processor -- start 2024-03-24T00:49:22.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.569+0000 7fc34300bf40 1 -- start start 2024-03-24T00:49:22.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.570+0000 7fc34300bf40 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c0d6062e20 0x55c0d6060910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.570+0000 7fc34300bf40 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55c0d608b1c0 0x55c0d605de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.570+0000 7fc34300bf40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c0d6090290 0x55c0d6111e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.570+0000 7fc34300bf40 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0d60a9cc0 con 0x55c0d6062e20 2024-03-24T00:49:22.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.570+0000 7fc34300bf40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0d60a5200 con 0x55c0d6090290 2024-03-24T00:49:22.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.570+0000 7fc34300bf40 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c0d60a9fa0 con 0x55c0d608b1c0 2024-03-24T00:49:22.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.570+0000 7fc33fe4c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c0d6062e20 0x55c0d6060910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.570+0000 7fc33fe4c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c0d6062e20 0x55c0d6060910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46614/0 (socket says 172.21.15.67:46614) 2024-03-24T00:49:22.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.570+0000 7fc33fe4c640 1 -- 172.21.15.67:0/822797326 learned_addr learned my addr 172.21.15.67:0/822797326 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:22.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.570+0000 7fc33f64b640 1 --2- 172.21.15.67:0/822797326 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55c0d608b1c0 0x55c0d605de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.570+0000 7fc34064d640 1 --2- 172.21.15.67:0/822797326 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c0d6090290 0x55c0d6111e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.570+0000 7fc33fe4c640 1 -- 172.21.15.67:0/822797326 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55c0d608b1c0 msgr2=0x55c0d605de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:22.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.570+0000 7fc33fe4c640 1 --2- 172.21.15.67:0/822797326 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55c0d608b1c0 0x55c0d605de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.570+0000 7fc33fe4c640 1 -- 172.21.15.67:0/822797326 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c0d6090290 msgr2=0x55c0d6111e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:22.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.571+0000 7fc33fe4c640 1 --2- 172.21.15.67:0/822797326 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c0d6090290 0x55c0d6111e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.571+0000 7fc33fe4c640 1 -- 172.21.15.67:0/822797326 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c0d6061080 con 0x55c0d6062e20 2024-03-24T00:49:22.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.571+0000 7fc33f64b640 1 --2- 172.21.15.67:0/822797326 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55c0d608b1c0 0x55c0d605de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:22.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.571+0000 7fc34064d640 1 --2- 172.21.15.67:0/822797326 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c0d6090290 0x55c0d6111e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:22.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.571+0000 7fc33fe4c640 1 --2- 172.21.15.67:0/822797326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c0d6062e20 0x55c0d6060910 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7fc33000cec0 tx=0x7fc33000a5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:22.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.571+0000 7fc33d647640 1 -- 172.21.15.67:0/822797326 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc330010470 con 0x55c0d6062e20 2024-03-24T00:49:22.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.571+0000 7fc33d647640 1 -- 172.21.15.67:0/822797326 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc330010610 con 0x55c0d6062e20 2024-03-24T00:49:22.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.571+0000 7fc34300bf40 1 -- 172.21.15.67:0/822797326 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55c0d6061360 con 0x55c0d6062e20 2024-03-24T00:49:22.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.571+0000 7fc33d647640 1 -- 172.21.15.67:0/822797326 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3300109e0 con 0x55c0d6062e20 2024-03-24T00:49:22.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.571+0000 7fc34300bf40 1 -- 172.21.15.67:0/822797326 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c0d611d860 con 0x55c0d6062e20 2024-03-24T00:49:22.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.572+0000 7fc33d647640 1 -- 172.21.15.67:0/822797326 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc33000e040 con 0x55c0d6062e20 2024-03-24T00:49:22.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.573+0000 7fc33d647640 1 --2- 172.21.15.67:0/822797326 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc32006a5e0 0x7fc32006caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.573+0000 7fc33d647640 1 -- 172.21.15.67:0/822797326 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fc3300897c0 con 0x55c0d6062e20 2024-03-24T00:49:22.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.573+0000 7fc33f64b640 1 --2- 172.21.15.67:0/822797326 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc32006a5e0 0x7fc32006caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.573+0000 7fc34300bf40 1 --2- 172.21.15.67:0/822797326 >> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] conn(0x55c0d605e3c0 0x55c0d611e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.573+0000 7fc34300bf40 1 -- 172.21.15.67:0/822797326 --> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] -- osd_op(unknown.0.0:1 18.12 18:4bec3886:::10000000002.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e121) v9 -- 0x55c0d611e5a0 con 0x55c0d605e3c0 2024-03-24T00:49:22.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.573+0000 7fc34064d640 1 --2- 172.21.15.67:0/822797326 >> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] conn(0x55c0d605e3c0 0x55c0d611e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.573+0000 7fc33f64b640 1 --2- 172.21.15.67:0/822797326 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc32006a5e0 0x7fc32006caa0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x55c0d60a8a10 tx=0x7fc32804f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:22.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.574+0000 7fc34064d640 1 --2- 172.21.15.67:0/822797326 >> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] conn(0x55c0d605e3c0 0x55c0d611e060 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:22.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.574+0000 7fc34064d640 1 -- 172.21.15.67:0/822797326 <== osd.2 v2:172.21.15.67:6817/1268432561 1 ==== osd_op_reply(1 10000000002.00000000 [getxattr (12) out=12b] v0'0 uv22 ondisk = 0) v8 ==== 164+0+12 (crc 0 0 0) 0x7fc3380027e0 con 0x55c0d605e3c0 2024-03-24T00:49:22.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.574+0000 7fc34300bf40 1 -- 172.21.15.67:0/822797326 >> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] conn(0x55c0d605e3c0 msgr2=0x55c0d611e060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:22.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.574+0000 7fc34300bf40 1 --2- 172.21.15.67:0/822797326 >> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] conn(0x55c0d605e3c0 0x55c0d611e060 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.574+0000 7fc34300bf40 1 -- 172.21.15.67:0/822797326 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc32006a5e0 msgr2=0x7fc32006caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:22.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.574+0000 7fc34300bf40 1 --2- 172.21.15.67:0/822797326 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc32006a5e0 0x7fc32006caa0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x55c0d60a8a10 tx=0x7fc32804f000 comp rx=0 tx=0).stop 2024-03-24T00:49:22.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.574+0000 7fc34300bf40 1 -- 172.21.15.67:0/822797326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c0d6062e20 msgr2=0x55c0d6060910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:22.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.574+0000 7fc34300bf40 1 --2- 172.21.15.67:0/822797326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c0d6062e20 0x55c0d6060910 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7fc33000cec0 tx=0x7fc33000a5a0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.574+0000 7fc34064d640 1 -- 172.21.15.67:0/822797326 reap_dead start 2024-03-24T00:49:22.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.574+0000 7fc34300bf40 1 -- 172.21.15.67:0/822797326 shutdown_connections 2024-03-24T00:49:22.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.574+0000 7fc34300bf40 1 -- 172.21.15.67:0/822797326 >> 172.21.15.67:0/822797326 conn(0x55c0d608d770 msgr2=0x55c0d60a8f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:22.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.574+0000 7fc34300bf40 1 -- 172.21.15.67:0/822797326 shutdown_connections 2024-03-24T00:49:22.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.574+0000 7fc34300bf40 1 -- 172.21.15.67:0/822797326 wait complete. 2024-03-24T00:49:22.582 INFO:teuthology.orchestra.run.smithi067.stdout:mytag123 2024-03-24T00:49:22.582 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 100000001f8.00000000 scrub_tag 2024-03-24T00:49:22.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.664+0000 7f659e035f40 1 Processor -- start 2024-03-24T00:49:22.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.664+0000 7f659e035f40 1 -- start start 2024-03-24T00:49:22.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.665+0000 7f659e035f40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559b7751b190 0x559b77563b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.665+0000 7f659e035f40 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b775eb8e0 0x559b775ef6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.665+0000 7f659e035f40 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559b775ebcc0 0x559b775efc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.665+0000 7f659e035f40 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b774c10d0 con 0x559b775eb8e0 2024-03-24T00:49:22.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.665+0000 7f659e035f40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b7751b5a0 con 0x559b7751b190 2024-03-24T00:49:22.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.665+0000 7f659e035f40 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x559b77544c50 con 0x559b775ebcc0 2024-03-24T00:49:22.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.665+0000 7f659ab74640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b775eb8e0 0x559b775ef6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.665+0000 7f659ab74640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b775eb8e0 0x559b775ef6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46630/0 (socket says 172.21.15.67:46630) 2024-03-24T00:49:22.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.665+0000 7f659ab74640 1 -- 172.21.15.67:0/688546350 learned_addr learned my addr 172.21.15.67:0/688546350 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:22.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.665+0000 7f659b375640 1 --2- 172.21.15.67:0/688546350 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559b7751b190 0x559b77563b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.665+0000 7f659bb76640 1 --2- 172.21.15.67:0/688546350 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559b775ebcc0 0x559b775efc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.665+0000 7f659ab74640 1 -- 172.21.15.67:0/688546350 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559b775ebcc0 msgr2=0x559b775efc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:22.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.665+0000 7f659ab74640 1 --2- 172.21.15.67:0/688546350 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559b775ebcc0 0x559b775efc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.665+0000 7f659ab74640 1 -- 172.21.15.67:0/688546350 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559b7751b190 msgr2=0x559b77563b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:22.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.665+0000 7f659ab74640 1 --2- 172.21.15.67:0/688546350 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559b7751b190 0x559b77563b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.665+0000 7f659ab74640 1 -- 172.21.15.67:0/688546350 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559b77546570 con 0x559b775eb8e0 2024-03-24T00:49:22.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.665+0000 7f659ab74640 1 --2- 172.21.15.67:0/688546350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b775eb8e0 0x559b775ef6d0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f6590009790 tx=0x7f6590005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ca1f82208cd7ba0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:22.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.666+0000 7f6599b72640 1 -- 172.21.15.67:0/688546350 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6590008870 con 0x559b775eb8e0 2024-03-24T00:49:22.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.666+0000 7f6599b72640 1 -- 172.21.15.67:0/688546350 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6590018070 con 0x559b775eb8e0 2024-03-24T00:49:22.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.666+0000 7f6599b72640 1 -- 172.21.15.67:0/688546350 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6590008c50 con 0x559b775eb8e0 2024-03-24T00:49:22.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.666+0000 7f659e035f40 1 -- 172.21.15.67:0/688546350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b775eb8e0 msgr2=0x559b775ef6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:22.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.666+0000 7f659e035f40 1 --2- 172.21.15.67:0/688546350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b775eb8e0 0x559b775ef6d0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f6590009790 tx=0x7f6590005a00 comp rx=0 tx=0).stop 2024-03-24T00:49:22.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.666+0000 7f659e035f40 1 -- 172.21.15.67:0/688546350 shutdown_connections 2024-03-24T00:49:22.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.666+0000 7f659e035f40 1 --2- 172.21.15.67:0/688546350 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559b775ebcc0 0x559b775efc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.666+0000 7f659e035f40 1 --2- 172.21.15.67:0/688546350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b775eb8e0 0x559b775ef6d0 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.666+0000 7f659e035f40 1 --2- 172.21.15.67:0/688546350 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559b7751b190 0x559b77563b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.666+0000 7f659e035f40 1 -- 172.21.15.67:0/688546350 >> 172.21.15.67:0/688546350 conn(0x559b775460f0 msgr2=0x559b77543c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:22.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.666+0000 7f659e035f40 1 -- 172.21.15.67:0/688546350 shutdown_connections 2024-03-24T00:49:22.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.666+0000 7f659e035f40 1 -- 172.21.15.67:0/688546350 wait complete. 2024-03-24T00:49:22.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.667+0000 7f659e035f40 1 Processor -- start 2024-03-24T00:49:22.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.667+0000 7f659e035f40 1 -- start start 2024-03-24T00:49:22.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.668+0000 7f659e035f40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559b7751b190 0x559b7768e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.668+0000 7f659e035f40 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559b775eb8e0 0x559b775e75e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.668+0000 7f659e035f40 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b775ebcc0 0x559b775db300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.668+0000 7f659e035f40 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b775448e0 con 0x559b775ebcc0 2024-03-24T00:49:22.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.668+0000 7f659e035f40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b77546850 con 0x559b7751b190 2024-03-24T00:49:22.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.668+0000 7f659e035f40 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x559b7751aed0 con 0x559b775eb8e0 2024-03-24T00:49:22.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.668+0000 7f659bb76640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b775ebcc0 0x559b775db300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.668+0000 7f659bb76640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b775ebcc0 0x559b775db300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46638/0 (socket says 172.21.15.67:46638) 2024-03-24T00:49:22.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.668+0000 7f659bb76640 1 -- 172.21.15.67:0/1846701746 learned_addr learned my addr 172.21.15.67:0/1846701746 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:22.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.668+0000 7f659b375640 1 --2- 172.21.15.67:0/1846701746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559b7751b190 0x559b7768e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.668+0000 7f659ab74640 1 --2- 172.21.15.67:0/1846701746 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559b775eb8e0 0x559b775e75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.668+0000 7f659bb76640 1 -- 172.21.15.67:0/1846701746 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559b775eb8e0 msgr2=0x559b775e75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:22.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.668+0000 7f659bb76640 1 --2- 172.21.15.67:0/1846701746 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x559b775eb8e0 0x559b775e75e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.668+0000 7f659bb76640 1 -- 172.21.15.67:0/1846701746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559b7751b190 msgr2=0x559b7768e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:22.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.668+0000 7f659bb76640 1 --2- 172.21.15.67:0/1846701746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559b7751b190 0x559b7768e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.668+0000 7f659bb76640 1 -- 172.21.15.67:0/1846701746 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559b775e5c20 con 0x559b775ebcc0 2024-03-24T00:49:22.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.668+0000 7f659bb76640 1 --2- 172.21.15.67:0/1846701746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b775ebcc0 0x559b775db300 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f659400d410 tx=0x7f659400d8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:22.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.668+0000 7f6598b70640 1 -- 172.21.15.67:0/1846701746 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6594018070 con 0x559b775ebcc0 2024-03-24T00:49:22.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.668+0000 7f6598b70640 1 -- 172.21.15.67:0/1846701746 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f659400f510 con 0x559b775ebcc0 2024-03-24T00:49:22.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.668+0000 7f6598b70640 1 -- 172.21.15.67:0/1846701746 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f659400f8e0 con 0x559b775ebcc0 2024-03-24T00:49:22.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.668+0000 7f659e035f40 1 -- 172.21.15.67:0/1846701746 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x559b775e5f00 con 0x559b775ebcc0 2024-03-24T00:49:22.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.668+0000 7f659e035f40 1 -- 172.21.15.67:0/1846701746 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559b775dbce0 con 0x559b775ebcc0 2024-03-24T00:49:22.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.670+0000 7f6598b70640 1 -- 172.21.15.67:0/1846701746 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f659401b070 con 0x559b775ebcc0 2024-03-24T00:49:22.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.671+0000 7f6598b70640 1 --2- 172.21.15.67:0/1846701746 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f658806a5b0 0x7f658806ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.671+0000 7f6598b70640 1 -- 172.21.15.67:0/1846701746 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f65940890b0 con 0x559b775ebcc0 2024-03-24T00:49:22.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.671+0000 7f659e035f40 1 --2- 172.21.15.67:0/1846701746 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x559b775db840 0x559b775dc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.671+0000 7f659e035f40 1 -- 172.21.15.67:0/1846701746 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:1 18.19 18:997b7129:::100000001f8.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e121) v9 -- 0x559b775e40d0 con 0x559b775db840 2024-03-24T00:49:22.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.671+0000 7f659b375640 1 --2- 172.21.15.67:0/1846701746 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f658806a5b0 0x7f658806ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.671+0000 7f659ab74640 1 --2- 172.21.15.67:0/1846701746 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x559b775db840 0x559b775dc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.671+0000 7f659b375640 1 --2- 172.21.15.67:0/1846701746 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f658806a5b0 0x7f658806ca70 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f658c054000 tx=0x7f658c04e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:22.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.672+0000 7f659ab74640 1 --2- 172.21.15.67:0/1846701746 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x559b775db840 0x559b775dc4e0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:22.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.672+0000 7f659ab74640 1 -- 172.21.15.67:0/1846701746 <== osd.4 v2:172.21.15.72:6808/477220793 1 ==== osd_op_reply(1 100000001f8.00000000 [getxattr (12) out=12b] v0'0 uv23 ondisk = 0) v8 ==== 164+0+12 (crc 0 0 0) 0x7f65900086f0 con 0x559b775db840 2024-03-24T00:49:22.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.672+0000 7f659e035f40 1 -- 172.21.15.67:0/1846701746 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x559b775db840 msgr2=0x559b775dc4e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:22.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.672+0000 7f659e035f40 1 --2- 172.21.15.67:0/1846701746 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x559b775db840 0x559b775dc4e0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.672+0000 7f659e035f40 1 -- 172.21.15.67:0/1846701746 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f658806a5b0 msgr2=0x7f658806ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:22.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.672+0000 7f659e035f40 1 --2- 172.21.15.67:0/1846701746 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f658806a5b0 0x7f658806ca70 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f658c054000 tx=0x7f658c04e000 comp rx=0 tx=0).stop 2024-03-24T00:49:22.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.672+0000 7f659e035f40 1 -- 172.21.15.67:0/1846701746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b775ebcc0 msgr2=0x559b775db300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:22.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.672+0000 7f659e035f40 1 --2- 172.21.15.67:0/1846701746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b775ebcc0 0x559b775db300 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f659400d410 tx=0x7f659400d8d0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.672+0000 7f659bb76640 1 -- 172.21.15.67:0/1846701746 reap_dead start 2024-03-24T00:49:22.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.672+0000 7f659e035f40 1 -- 172.21.15.67:0/1846701746 shutdown_connections 2024-03-24T00:49:22.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.672+0000 7f659e035f40 1 -- 172.21.15.67:0/1846701746 >> 172.21.15.67:0/1846701746 conn(0x559b775460f0 msgr2=0x559b77517c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:22.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.672+0000 7f659e035f40 1 -- 172.21.15.67:0/1846701746 shutdown_connections 2024-03-24T00:49:22.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.672+0000 7f659e035f40 1 -- 172.21.15.67:0/1846701746 wait complete. 2024-03-24T00:49:22.678 INFO:teuthology.orchestra.run.smithi067.stdout:mytag123 2024-03-24T00:49:22.678 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000004.00000000 scrub_tag 2024-03-24T00:49:22.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.760+0000 7fe6dfed2f40 1 Processor -- start 2024-03-24T00:49:22.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.760+0000 7fe6dfed2f40 1 -- start start 2024-03-24T00:49:22.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.761+0000 7fe6dfed2f40 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563f1529d250 0x563f15329b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.761+0000 7fe6dfed2f40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563f1530d8c0 0x563f1530c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.761+0000 7fe6dfed2f40 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563f15324480 0x563f1530cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.761+0000 7fe6dfed2f40 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f152557f0 con 0x563f1529d250 2024-03-24T00:49:22.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.761+0000 7fe6dfed2f40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f151fd0d0 con 0x563f1530d8c0 2024-03-24T00:49:22.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.761+0000 7fe6dfed2f40 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x563f15255970 con 0x563f15324480 2024-03-24T00:49:22.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.761+0000 7fe6dd9d0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563f1529d250 0x563f15329b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.761+0000 7fe6dd9d0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563f1529d250 0x563f15329b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46654/0 (socket says 172.21.15.67:46654) 2024-03-24T00:49:22.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.761+0000 7fe6dd9d0640 1 -- 172.21.15.67:0/1357264530 learned_addr learned my addr 172.21.15.67:0/1357264530 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:22.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.761+0000 7fe6de1d1640 1 --2- 172.21.15.67:0/1357264530 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563f15324480 0x563f1530cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.761+0000 7fe6dd1cf640 1 --2- 172.21.15.67:0/1357264530 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563f1530d8c0 0x563f1530c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.761+0000 7fe6dd9d0640 1 -- 172.21.15.67:0/1357264530 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563f15324480 msgr2=0x563f1530cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:22.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.761+0000 7fe6dd9d0640 1 --2- 172.21.15.67:0/1357264530 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563f15324480 0x563f1530cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.761+0000 7fe6dd9d0640 1 -- 172.21.15.67:0/1357264530 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563f1530d8c0 msgr2=0x563f1530c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:22.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.761+0000 7fe6dd9d0640 1 --2- 172.21.15.67:0/1357264530 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563f1530d8c0 0x563f1530c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.761+0000 7fe6dd9d0640 1 -- 172.21.15.67:0/1357264530 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563f1530d250 con 0x563f1529d250 2024-03-24T00:49:22.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.761+0000 7fe6de1d1640 1 --2- 172.21.15.67:0/1357264530 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563f15324480 0x563f1530cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:22.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.762+0000 7fe6dd9d0640 1 --2- 172.21.15.67:0/1357264530 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563f1529d250 0x563f15329b70 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7fe6d000a520 tx=0x7fe6d004c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bb8c4ea902d495f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:22.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.762+0000 7fe6cffff640 1 -- 172.21.15.67:0/1357264530 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d0061070 con 0x563f1529d250 2024-03-24T00:49:22.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.762+0000 7fe6cffff640 1 -- 172.21.15.67:0/1357264530 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6d0058040 con 0x563f1529d250 2024-03-24T00:49:22.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.762+0000 7fe6cffff640 1 -- 172.21.15.67:0/1357264530 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d0057070 con 0x563f1529d250 2024-03-24T00:49:22.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.762+0000 7fe6dfed2f40 1 -- 172.21.15.67:0/1357264530 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563f1529d250 msgr2=0x563f15329b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:22.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.762+0000 7fe6dfed2f40 1 --2- 172.21.15.67:0/1357264530 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563f1529d250 0x563f15329b70 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7fe6d000a520 tx=0x7fe6d004c000 comp rx=0 tx=0).stop 2024-03-24T00:49:22.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.762+0000 7fe6dfed2f40 1 -- 172.21.15.67:0/1357264530 shutdown_connections 2024-03-24T00:49:22.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.762+0000 7fe6dfed2f40 1 --2- 172.21.15.67:0/1357264530 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563f15324480 0x563f1530cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.762+0000 7fe6dfed2f40 1 --2- 172.21.15.67:0/1357264530 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563f1530d8c0 0x563f1530c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.762+0000 7fe6dfed2f40 1 --2- 172.21.15.67:0/1357264530 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563f1529d250 0x563f15329b70 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.762+0000 7fe6dfed2f40 1 -- 172.21.15.67:0/1357264530 >> 172.21.15.67:0/1357264530 conn(0x563f152820f0 msgr2=0x563f1532c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:22.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.762+0000 7fe6dfed2f40 1 -- 172.21.15.67:0/1357264530 shutdown_connections 2024-03-24T00:49:22.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.762+0000 7fe6dfed2f40 1 -- 172.21.15.67:0/1357264530 wait complete. 2024-03-24T00:49:22.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.763+0000 7fe6dfed2f40 1 Processor -- start 2024-03-24T00:49:22.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.763+0000 7fe6dfed2f40 1 -- start start 2024-03-24T00:49:22.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.763+0000 7fe6dfed2f40 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563f1529d250 0x563f1532bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.763+0000 7fe6dfed2f40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563f1530d8c0 0x563f1529b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.763+0000 7fe6dfed2f40 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563f15324480 0x563f1530c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.764+0000 7fe6dfed2f40 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f15282da0 con 0x563f1529d250 2024-03-24T00:49:22.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.764+0000 7fe6dfed2f40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f15280060 con 0x563f1530d8c0 2024-03-24T00:49:22.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.764+0000 7fe6dfed2f40 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x563f15283230 con 0x563f15324480 2024-03-24T00:49:22.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.764+0000 7fe6dd9d0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563f1529d250 0x563f1532bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.764+0000 7fe6dd9d0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563f1529d250 0x563f1532bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46668/0 (socket says 172.21.15.67:46668) 2024-03-24T00:49:22.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.764+0000 7fe6dd9d0640 1 -- 172.21.15.67:0/4162869393 learned_addr learned my addr 172.21.15.67:0/4162869393 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:22.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.764+0000 7fe6dd1cf640 1 --2- 172.21.15.67:0/4162869393 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563f1530d8c0 0x563f1529b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.764+0000 7fe6de1d1640 1 --2- 172.21.15.67:0/4162869393 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563f15324480 0x563f1530c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.764+0000 7fe6dd9d0640 1 -- 172.21.15.67:0/4162869393 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563f15324480 msgr2=0x563f1530c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:22.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.764+0000 7fe6dd9d0640 1 --2- 172.21.15.67:0/4162869393 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x563f15324480 0x563f1530c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.764+0000 7fe6dd9d0640 1 -- 172.21.15.67:0/4162869393 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563f1530d8c0 msgr2=0x563f1529b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:22.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.764+0000 7fe6dd9d0640 1 --2- 172.21.15.67:0/4162869393 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563f1530d8c0 0x563f1529b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.764+0000 7fe6dd9d0640 1 -- 172.21.15.67:0/4162869393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563f1530b610 con 0x563f1529d250 2024-03-24T00:49:22.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.764+0000 7fe6dd1cf640 1 --2- 172.21.15.67:0/4162869393 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563f1530d8c0 0x563f1529b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:49:22.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.764+0000 7fe6dd9d0640 1 --2- 172.21.15.67:0/4162869393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563f1529d250 0x563f1532bd40 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7fe6d0000c00 tx=0x7fe6d0003020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:22.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.764+0000 7fe6ceffd640 1 -- 172.21.15.67:0/4162869393 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d006a070 con 0x563f1529d250 2024-03-24T00:49:22.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.764+0000 7fe6ceffd640 1 -- 172.21.15.67:0/4162869393 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6d0052640 con 0x563f1529d250 2024-03-24T00:49:22.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.764+0000 7fe6dfed2f40 1 -- 172.21.15.67:0/4162869393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x563f1530b8f0 con 0x563f1529d250 2024-03-24T00:49:22.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.764+0000 7fe6ceffd640 1 -- 172.21.15.67:0/4162869393 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d0057070 con 0x563f1529d250 2024-03-24T00:49:22.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.764+0000 7fe6dfed2f40 1 -- 172.21.15.67:0/4162869393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563f1530beb0 con 0x563f1529d250 2024-03-24T00:49:22.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.766+0000 7fe6ceffd640 1 -- 172.21.15.67:0/4162869393 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe6d0052a70 con 0x563f1529d250 2024-03-24T00:49:22.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.766+0000 7fe6ceffd640 1 --2- 172.21.15.67:0/4162869393 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe6c006a5b0 0x7fe6c006ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.766+0000 7fe6ceffd640 1 -- 172.21.15.67:0/4162869393 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fe6d00cb080 con 0x563f1529d250 2024-03-24T00:49:22.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.766+0000 7fe6dfed2f40 1 --2- 172.21.15.67:0/4162869393 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x563f1530b170 0x563f153c6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.766+0000 7fe6dd1cf640 1 --2- 172.21.15.67:0/4162869393 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe6c006a5b0 0x7fe6c006ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.766+0000 7fe6dfed2f40 1 -- 172.21.15.67:0/4162869393 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:1 18.5 18:a367a3e8:::10000000004.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e121) v9 -- 0x563f1530cbf0 con 0x563f1530b170 2024-03-24T00:49:22.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.766+0000 7fe6de1d1640 1 --2- 172.21.15.67:0/4162869393 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x563f1530b170 0x563f153c6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.767+0000 7fe6de1d1640 1 --2- 172.21.15.67:0/4162869393 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x563f1530b170 0x563f153c6360 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:22.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.767+0000 7fe6dd1cf640 1 --2- 172.21.15.67:0/4162869393 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe6c006a5b0 0x7fe6c006ca70 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x563f1530a760 tx=0x7fe6d400c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:22.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.767+0000 7fe6de1d1640 1 -- 172.21.15.67:0/4162869393 <== osd.1 v2:172.21.15.67:6800/4062510404 1 ==== osd_op_reply(1 10000000004.00000000 [getxattr (12) out=12b] v0'0 uv12 ondisk = 0) v8 ==== 164+0+12 (crc 0 0 0) 0x7fe6d8001800 con 0x563f1530b170 2024-03-24T00:49:22.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.767+0000 7fe6dfed2f40 1 -- 172.21.15.67:0/4162869393 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x563f1530b170 msgr2=0x563f153c6360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:22.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.767+0000 7fe6dfed2f40 1 --2- 172.21.15.67:0/4162869393 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x563f1530b170 0x563f153c6360 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.767+0000 7fe6dfed2f40 1 -- 172.21.15.67:0/4162869393 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe6c006a5b0 msgr2=0x7fe6c006ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:22.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.767+0000 7fe6dfed2f40 1 --2- 172.21.15.67:0/4162869393 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe6c006a5b0 0x7fe6c006ca70 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x563f1530a760 tx=0x7fe6d400c040 comp rx=0 tx=0).stop 2024-03-24T00:49:22.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.767+0000 7fe6dfed2f40 1 -- 172.21.15.67:0/4162869393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563f1529d250 msgr2=0x563f1532bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:22.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.767+0000 7fe6dfed2f40 1 --2- 172.21.15.67:0/4162869393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563f1529d250 0x563f1532bd40 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7fe6d0000c00 tx=0x7fe6d0003020 comp rx=0 tx=0).stop 2024-03-24T00:49:22.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.767+0000 7fe6de1d1640 1 -- 172.21.15.67:0/4162869393 reap_dead start 2024-03-24T00:49:22.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.767+0000 7fe6dfed2f40 1 -- 172.21.15.67:0/4162869393 shutdown_connections 2024-03-24T00:49:22.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.767+0000 7fe6dfed2f40 1 -- 172.21.15.67:0/4162869393 >> 172.21.15.67:0/4162869393 conn(0x563f152820f0 msgr2=0x563f1532ca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:22.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.767+0000 7fe6dfed2f40 1 -- 172.21.15.67:0/4162869393 shutdown_connections 2024-03-24T00:49:22.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.767+0000 7fe6dfed2f40 1 -- 172.21.15.67:0/4162869393 wait complete. 2024-03-24T00:49:22.777 INFO:teuthology.orchestra.run.smithi067.stdout:mytag123 2024-03-24T00:49:22.777 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:49:22.777 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.8:0 flush journal 2024-03-24T00:49:22.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.920+0000 7fa485b59640 1 Processor -- start 2024-03-24T00:49:22.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.920+0000 7fa485b59640 1 -- start start 2024-03-24T00:49:22.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.921+0000 7fa485b59640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa48015f7b0 0x7fa4800b2660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.921+0000 7fa485b59640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa4800b2ba0 0x7fa480099d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.921+0000 7fa485b59640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa48009ade0 con 0x7fa48009a240 2024-03-24T00:49:22.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.921+0000 7fa485b59640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa48009a690 con 0x7fa4800b2ba0 2024-03-24T00:49:22.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.921+0000 7fa485b59640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4800cdd60 con 0x7fa48015f7b0 2024-03-24T00:49:22.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.921+0000 7fa47effd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa4800b2ba0 0x7fa480099d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.921+0000 7fa47effd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa4800b2ba0 0x7fa480099d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46684/0 (socket says 172.21.15.67:46684) 2024-03-24T00:49:22.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.921+0000 7fa47effd640 1 -- 172.21.15.67:0/4143539308 learned_addr learned my addr 172.21.15.67:0/4143539308 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:22.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.921+0000 7fa47f7fe640 1 --2- 172.21.15.67:0/4143539308 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa48015f7b0 0x7fa4800b2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.921+0000 7fa47effd640 1 -- 172.21.15.67:0/4143539308 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa48015f7b0 msgr2=0x7fa4800b2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:22.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.921+0000 7fa47effd640 1 --2- 172.21.15.67:0/4143539308 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa48015f7b0 0x7fa4800b2660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.921+0000 7fa47effd640 1 -- 172.21.15.67:0/4143539308 >> v1:172.21.15.72:6789/0 conn(0x7fa48009a240 legacy=0x7fa4800b1490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:22.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.921+0000 7fa47effd640 1 -- 172.21.15.67:0/4143539308 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4800cf9e0 con 0x7fa4800b2ba0 2024-03-24T00:49:22.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.922+0000 7fa47effd640 1 --2- 172.21.15.67:0/4143539308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4800b2ba0 0x7fa480099d00 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7fa470009870 tx=0x7fa47002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25dbcd666ba57d43 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:22.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.922+0000 7fa47e7fc640 1 -- 172.21.15.67:0/4143539308 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa470034070 con 0x7fa4800b2ba0 2024-03-24T00:49:22.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.922+0000 7fa47e7fc640 1 -- 172.21.15.67:0/4143539308 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa470030840 con 0x7fa4800b2ba0 2024-03-24T00:49:22.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.922+0000 7fa47e7fc640 1 -- 172.21.15.67:0/4143539308 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa470030c10 con 0x7fa4800b2ba0 2024-03-24T00:49:22.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.922+0000 7fa485b59640 1 -- 172.21.15.67:0/4143539308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4800b2ba0 msgr2=0x7fa480099d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:22.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.922+0000 7fa485b59640 1 --2- 172.21.15.67:0/4143539308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4800b2ba0 0x7fa480099d00 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7fa470009870 tx=0x7fa47002b980 comp rx=0 tx=0).stop 2024-03-24T00:49:22.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.923+0000 7fa485b59640 1 -- 172.21.15.67:0/4143539308 shutdown_connections 2024-03-24T00:49:22.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.923+0000 7fa485b59640 1 --2- 172.21.15.67:0/4143539308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4800b2ba0 0x7fa480099d00 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.923+0000 7fa485b59640 1 --2- 172.21.15.67:0/4143539308 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa48015f7b0 0x7fa4800b2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.923+0000 7fa485b59640 1 -- 172.21.15.67:0/4143539308 >> 172.21.15.67:0/4143539308 conn(0x7fa4800b3650 msgr2=0x7fa4800b3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:22.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.923+0000 7fa485b59640 1 -- 172.21.15.67:0/4143539308 shutdown_connections 2024-03-24T00:49:22.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.923+0000 7fa485b59640 1 -- 172.21.15.67:0/4143539308 wait complete. 2024-03-24T00:49:22.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.924+0000 7fa485b59640 1 Processor -- start 2024-03-24T00:49:22.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.924+0000 7fa485b59640 1 -- start start 2024-03-24T00:49:22.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.925+0000 7fa485b59640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa48009a240 0x7fa480159950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.925+0000 7fa485b59640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa4800b2ba0 0x7fa480070fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.925+0000 7fa485b59640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa48015f7b0 0x7fa480071500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.925+0000 7fa485b59640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4800cfcc0 con 0x7fa48009a240 2024-03-24T00:49:22.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.925+0000 7fa485b59640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4800cc820 con 0x7fa48015f7b0 2024-03-24T00:49:22.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.925+0000 7fa485b59640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4800ae750 con 0x7fa4800b2ba0 2024-03-24T00:49:22.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.925+0000 7fa47f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa48009a240 0x7fa480159950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.925+0000 7fa47f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa48009a240 0x7fa480159950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46698/0 (socket says 172.21.15.67:46698) 2024-03-24T00:49:22.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.925+0000 7fa47f7fe640 1 -- 172.21.15.67:0/1727183402 learned_addr learned my addr 172.21.15.67:0/1727183402 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:22.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.925+0000 7fa47ffff640 1 --2- 172.21.15.67:0/1727183402 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa48015f7b0 0x7fa480071500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.925+0000 7fa47effd640 1 --2- 172.21.15.67:0/1727183402 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa4800b2ba0 0x7fa480070fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.925+0000 7fa47f7fe640 1 -- 172.21.15.67:0/1727183402 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa4800b2ba0 msgr2=0x7fa480070fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:22.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.925+0000 7fa47f7fe640 1 --2- 172.21.15.67:0/1727183402 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa4800b2ba0 0x7fa480070fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.925+0000 7fa47f7fe640 1 -- 172.21.15.67:0/1727183402 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa48015f7b0 msgr2=0x7fa480071500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:22.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.925+0000 7fa47f7fe640 1 --2- 172.21.15.67:0/1727183402 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa48015f7b0 0x7fa480071500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.925+0000 7fa47f7fe640 1 -- 172.21.15.67:0/1727183402 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4800cace0 con 0x7fa48009a240 2024-03-24T00:49:22.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.925+0000 7fa47ffff640 1 --2- 172.21.15.67:0/1727183402 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa48015f7b0 0x7fa480071500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:22.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.925+0000 7fa47f7fe640 1 --2- 172.21.15.67:0/1727183402 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa48009a240 0x7fa480159950 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7fa47400d5f0 tx=0x7fa47400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:22.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.925+0000 7fa467fff640 1 -- 172.21.15.67:0/1727183402 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4740125f0 con 0x7fa48009a240 2024-03-24T00:49:22.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.925+0000 7fa467fff640 1 -- 172.21.15.67:0/1727183402 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa474019070 con 0x7fa48009a240 2024-03-24T00:49:22.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.925+0000 7fa467fff640 1 -- 172.21.15.67:0/1727183402 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4740129d0 con 0x7fa48009a240 2024-03-24T00:49:22.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.925+0000 7fa485b59640 1 -- 172.21.15.67:0/1727183402 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4800caf80 con 0x7fa48009a240 2024-03-24T00:49:22.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.925+0000 7fa485b59640 1 -- 172.21.15.67:0/1727183402 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4800cb540 con 0x7fa48009a240 2024-03-24T00:49:22.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.927+0000 7fa467fff640 1 -- 172.21.15.67:0/1727183402 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa474012b70 con 0x7fa48009a240 2024-03-24T00:49:22.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.928+0000 7fa467fff640 1 --2- 172.21.15.67:0/1727183402 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa46c06bbd0 0x7fa46c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.928+0000 7fa467fff640 1 -- 172.21.15.67:0/1727183402 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fa47408b080 con 0x7fa48009a240 2024-03-24T00:49:22.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.928+0000 7fa47effd640 1 --2- 172.21.15.67:0/1727183402 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa46c06bbd0 0x7fa46c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.928+0000 7fa488813740 1 Processor -- start 2024-03-24T00:49:22.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.928+0000 7fa488813740 1 -- start start 2024-03-24T00:49:22.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.928+0000 7fa47effd640 1 --2- 172.21.15.67:0/1727183402 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa46c06bbd0 0x7fa46c06e090 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fa470036000 tx=0x7fa47003b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:22.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.929+0000 7fa488813740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5627896098d0 0x562789607590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.929+0000 7fa488813740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x562789609cb0 0x5627895c0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.929+0000 7fa488813740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5627895e9210 0x5627895c1440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.929+0000 7fa488813740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627895263d0 con 0x5627895e9210 2024-03-24T00:49:22.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.929+0000 7fa488813740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627894cead0 con 0x562789609cb0 2024-03-24T00:49:22.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.929+0000 7fa488813740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x562789522470 con 0x5627896098d0 2024-03-24T00:49:22.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.929+0000 7fa47ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5627895e9210 0x5627895c1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.929+0000 7fa47ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5627895e9210 0x5627895c1440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46714/0 (socket says 172.21.15.67:46714) 2024-03-24T00:49:22.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.929+0000 7fa47ffff640 1 -- 172.21.15.67:0/3416956167 learned_addr learned my addr 172.21.15.67:0/3416956167 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:22.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.929+0000 7fa47effd640 1 --2- 172.21.15.67:0/3416956167 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x562789609cb0 0x5627895c0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.929+0000 7fa47f7fe640 1 --2- 172.21.15.67:0/3416956167 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5627896098d0 0x562789607590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.929+0000 7fa47ffff640 1 -- 172.21.15.67:0/3416956167 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5627896098d0 msgr2=0x562789607590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:22.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.929+0000 7fa47ffff640 1 --2- 172.21.15.67:0/3416956167 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5627896098d0 0x562789607590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.929+0000 7fa47ffff640 1 -- 172.21.15.67:0/3416956167 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x562789609cb0 msgr2=0x5627895c0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:22.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.929+0000 7fa47ffff640 1 --2- 172.21.15.67:0/3416956167 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x562789609cb0 0x5627895c0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.929+0000 7fa47ffff640 1 -- 172.21.15.67:0/3416956167 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5627893047d0 con 0x5627895e9210 2024-03-24T00:49:22.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.930+0000 7fa47ffff640 1 --2- 172.21.15.67:0/3416956167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5627895e9210 0x5627895c1440 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7fa478001e70 tx=0x7fa47800d520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c091518686bcba65 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:22.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.930+0000 7fa4657fa640 1 -- 172.21.15.67:0/3416956167 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa478018070 con 0x5627895e9210 2024-03-24T00:49:22.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.930+0000 7fa4657fa640 1 -- 172.21.15.67:0/3416956167 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa478002080 con 0x5627895e9210 2024-03-24T00:49:22.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.930+0000 7fa4657fa640 1 -- 172.21.15.67:0/3416956167 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa478003b90 con 0x5627895e9210 2024-03-24T00:49:22.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.930+0000 7fa488813740 1 -- 172.21.15.67:0/3416956167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5627895e9210 msgr2=0x5627895c1440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:22.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.930+0000 7fa488813740 1 --2- 172.21.15.67:0/3416956167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5627895e9210 0x5627895c1440 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7fa478001e70 tx=0x7fa47800d520 comp rx=0 tx=0).stop 2024-03-24T00:49:22.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.930+0000 7fa488813740 1 -- 172.21.15.67:0/3416956167 shutdown_connections 2024-03-24T00:49:22.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.930+0000 7fa488813740 1 --2- 172.21.15.67:0/3416956167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5627895e9210 0x5627895c1440 unknown :-1 s=CLOSED pgs=1822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.930+0000 7fa488813740 1 --2- 172.21.15.67:0/3416956167 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x562789609cb0 0x5627895c0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.930+0000 7fa488813740 1 --2- 172.21.15.67:0/3416956167 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5627896098d0 0x562789607590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.930+0000 7fa488813740 1 -- 172.21.15.67:0/3416956167 >> 172.21.15.67:0/3416956167 conn(0x5627895be060 msgr2=0x562789606be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:22.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.931+0000 7fa488813740 1 -- 172.21.15.67:0/3416956167 shutdown_connections 2024-03-24T00:49:22.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.931+0000 7fa488813740 1 -- 172.21.15.67:0/3416956167 wait complete. 2024-03-24T00:49:22.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.931+0000 7fa488813740 1 -- start start 2024-03-24T00:49:22.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.931+0000 7fa488813740 1 Processor -- start 2024-03-24T00:49:22.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.932+0000 7fa488813740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5627896098d0 0x5627895eee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.932+0000 7fa488813740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562789609cb0 0x5627895e3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.932+0000 7fa488813740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56278961ee40 0x5627895e4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.932+0000 7fa488813740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627895bc690 con 0x562789609cb0 2024-03-24T00:49:22.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.932+0000 7fa488813740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627894e2060 con 0x56278961ee40 2024-03-24T00:49:22.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.932+0000 7fa488813740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5627895c0490 con 0x5627896098d0 2024-03-24T00:49:22.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.932+0000 7fa488813740 10 client.0 _ll_register_callbacks cb 0x5627894c46a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:49:22.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.932+0000 7fa47effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562789609cb0 0x5627895e3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.932+0000 7fa47effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562789609cb0 0x5627895e3a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46728/0 (socket says 172.21.15.67:46728) 2024-03-24T00:49:22.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.932+0000 7fa47effd640 1 -- 172.21.15.67:0/960995039 learned_addr learned my addr 172.21.15.67:0/960995039 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:22.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.932+0000 7fa47f7fe640 1 --2- 172.21.15.67:0/960995039 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5627896098d0 0x5627895eee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.932+0000 7fa47ffff640 1 --2- 172.21.15.67:0/960995039 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56278961ee40 0x5627895e4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.932+0000 7fa47effd640 1 -- 172.21.15.67:0/960995039 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5627896098d0 msgr2=0x5627895eee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:22.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.932+0000 7fa47effd640 1 --2- 172.21.15.67:0/960995039 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5627896098d0 0x5627895eee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.932+0000 7fa47effd640 1 -- 172.21.15.67:0/960995039 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56278961ee40 msgr2=0x5627895e4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:22.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.932+0000 7fa47effd640 1 --2- 172.21.15.67:0/960995039 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56278961ee40 0x5627895e4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.932+0000 7fa47effd640 1 -- 172.21.15.67:0/960995039 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56278923a3f0 con 0x562789609cb0 2024-03-24T00:49:22.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.932+0000 7fa47effd640 1 --2- 172.21.15.67:0/960995039 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562789609cb0 0x5627895e3a20 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7fa47002bf90 tx=0x7fa470004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66975c619c084a29 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:22.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.933+0000 7fa45e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:49:22.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.933+0000 7fa45e7fc640 1 -- 172.21.15.67:0/960995039 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa470053060 con 0x562789609cb0 2024-03-24T00:49:22.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.933+0000 7fa45e7fc640 1 -- 172.21.15.67:0/960995039 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(121..121 src has 1..121) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fa470043810 con 0x562789609cb0 2024-03-24T00:49:22.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.933+0000 7fa488813740 1 -- 172.21.15.67:0/960995039 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa478002080 con 0x562789609cb0 2024-03-24T00:49:22.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.933+0000 7fa45e7fc640 1 -- 172.21.15.67:0/960995039 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa470059050 con 0x562789609cb0 2024-03-24T00:49:22.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.933+0000 7fa488813740 10 client.9648 fetch_fsmap learned FSMap version 92 2024-03-24T00:49:22.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.933+0000 7fa488813740 1 -- 172.21.15.67:0/960995039 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=92,monmap=2+}) v3 -- 0x5627891fa0f0 con 0x562789609cb0 2024-03-24T00:49:22.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.933+0000 7fa45e7fc640 1 -- 172.21.15.67:0/960995039 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa478002080 con 0x562789609cb0 2024-03-24T00:49:22.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.933+0000 7fa45e7fc640 1 -- 172.21.15.67:0/960995039 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 92) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fa47005c020 con 0x562789609cb0 2024-03-24T00:49:22.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.933+0000 7fa488813740 10 client.9648 fetch_fsmap finished waiting for FSMap version 92 2024-03-24T00:49:22.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.933+0000 7fa488813740 10 client.9648 resolve_mds: resolved 8:0 to role '8:0' aka daemon mds.a 2024-03-24T00:49:22.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.933+0000 7fa488813740 20 client.9648 populate_metadata read hostname 'smithi067' 2024-03-24T00:49:22.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.933+0000 7fa488813740 1 --2- 172.21.15.67:0/960995039 >> [v2:172.21.15.67:6833/3833808901,v1:172.21.15.67:6835/3833808901] conn(0x5627895ec9c0 0x5627895ee040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.933+0000 7fa488813740 4 client.9648 mds_command: new command op to 9510 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:49:22.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.933+0000 7fa488813740 1 -- 172.21.15.67:0/960995039 --> [v2:172.21.15.67:6833/3833808901,v1:172.21.15.67:6835/3833808901] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562789533910 con 0x5627895ec9c0 2024-03-24T00:49:22.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.933+0000 7fa47f7fe640 1 --2- 172.21.15.67:0/960995039 >> [v2:172.21.15.67:6833/3833808901,v1:172.21.15.67:6835/3833808901] conn(0x5627895ec9c0 0x5627895ee040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.934+0000 7fa47f7fe640 1 --2- 172.21.15.67:0/960995039 >> [v2:172.21.15.67:6833/3833808901,v1:172.21.15.67:6835/3833808901] conn(0x5627895ec9c0 0x5627895ee040 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6bfcd9979a9c384 server_cookie=d2d2b641c42fddd2 in_seq=0 out_seq=0 2024-03-24T00:49:22.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.934+0000 7fa45e7fc640 10 client.9648 ms_handle_connect on v2:172.21.15.67:6833/3833808901 2024-03-24T00:49:22.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.934+0000 7fa45e7fc640 1 -- 172.21.15.67:0/960995039 <== mds.0 v2:172.21.15.67:6833/3833808901 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x562789533910 con 0x5627895ec9c0 2024-03-24T00:49:22.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.934+0000 7fa45e7fc640 10 client.9648 handle_command_reply: tid=0 2024-03-24T00:49:22.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.934+0000 7fa488813740 1 client.9648 shutdown 2024-03-24T00:49:22.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.934+0000 7fa488813740 1 -- 172.21.15.67:0/960995039 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562789609cb0 msgr2=0x5627895e3a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:22.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.934+0000 7fa488813740 1 --2- 172.21.15.67:0/960995039 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562789609cb0 0x5627895e3a20 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7fa47002bf90 tx=0x7fa470004730 comp rx=0 tx=0).stop 2024-03-24T00:49:22.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.935+0000 7fa488813740 1 -- 172.21.15.67:0/960995039 shutdown_connections 2024-03-24T00:49:22.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.935+0000 7fa488813740 1 --2- 172.21.15.67:0/960995039 >> [v2:172.21.15.67:6833/3833808901,v1:172.21.15.67:6835/3833808901] conn(0x5627895ec9c0 0x5627895ee040 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.935+0000 7fa488813740 1 --2- 172.21.15.67:0/960995039 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56278961ee40 0x5627895e4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.935+0000 7fa45e7fc640 0 client.9648 ms_handle_reset on v2:172.21.15.67:6833/3833808901 2024-03-24T00:49:22.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.935+0000 7fa488813740 1 --2- 172.21.15.67:0/960995039 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562789609cb0 0x5627895e3a20 unknown :-1 s=CLOSED pgs=1823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.935+0000 7fa488813740 1 --2- 172.21.15.67:0/960995039 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5627896098d0 0x5627895eee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.935+0000 7fa488813740 1 -- 172.21.15.67:0/960995039 >> 172.21.15.67:0/960995039 conn(0x5627895e9210 msgr2=0x562789606a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:22.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.935+0000 7fa488813740 1 -- 172.21.15.67:0/960995039 shutdown_connections 2024-03-24T00:49:22.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.935+0000 7fa488813740 1 -- 172.21.15.67:0/960995039 wait complete. 2024-03-24T00:49:22.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.935+0000 7fa488813740 20 client.9648 trim_cache size 0 max 16384 2024-03-24T00:49:22.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.948+0000 7fa488813740 1 Processor -- start 2024-03-24T00:49:22.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.948+0000 7fa488813740 1 -- start start 2024-03-24T00:49:22.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.949+0000 7fa488813740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5627896098d0 0x5627895c1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.949+0000 7fa488813740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562789609cb0 0x5627895c1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.949+0000 7fa488813740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56278961ee40 0x5627895c05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.949+0000 7fa488813740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa48006e090 con 0x562789609cb0 2024-03-24T00:49:22.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.949+0000 7fa488813740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4800b0590 con 0x5627896098d0 2024-03-24T00:49:22.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.949+0000 7fa488813740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4800b5110 con 0x56278961ee40 2024-03-24T00:49:22.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.949+0000 7fa47effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562789609cb0 0x5627895c1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.949+0000 7fa47effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562789609cb0 0x5627895c1ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46740/0 (socket says 172.21.15.67:46740) 2024-03-24T00:49:22.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.949+0000 7fa47effd640 1 -- 172.21.15.67:0/2777548725 learned_addr learned my addr 172.21.15.67:0/2777548725 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:22.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.950+0000 7fa47effd640 1 -- 172.21.15.67:0/2777548725 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56278961ee40 msgr2=0x5627895c05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:22.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.950+0000 7fa47f7fe640 1 --2- 172.21.15.67:0/2777548725 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5627896098d0 0x5627895c1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.950+0000 7fa47ffff640 1 --2- 172.21.15.67:0/2777548725 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56278961ee40 0x5627895c05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.950+0000 7fa47effd640 1 --2- 172.21.15.67:0/2777548725 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56278961ee40 0x5627895c05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.950+0000 7fa47effd640 1 -- 172.21.15.67:0/2777548725 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5627896098d0 msgr2=0x5627895c1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:22.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.950+0000 7fa47effd640 1 --2- 172.21.15.67:0/2777548725 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5627896098d0 0x5627895c1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.950+0000 7fa47effd640 1 -- 172.21.15.67:0/2777548725 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5627895ef720 con 0x562789609cb0 2024-03-24T00:49:22.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.950+0000 7fa47f7fe640 1 --2- 172.21.15.67:0/2777548725 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5627896098d0 0x5627895c1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:22.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.950+0000 7fa47effd640 1 --2- 172.21.15.67:0/2777548725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562789609cb0 0x5627895c1ec0 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7fa470000c00 tx=0x7fa47004b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bbbf0c6f08a2f18 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:22.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.950+0000 7fa464ff9640 1 -- 172.21.15.67:0/2777548725 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa470034040 con 0x562789609cb0 2024-03-24T00:49:22.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.950+0000 7fa464ff9640 1 -- 172.21.15.67:0/2777548725 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa470047030 con 0x562789609cb0 2024-03-24T00:49:22.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.950+0000 7fa464ff9640 1 -- 172.21.15.67:0/2777548725 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa470043890 con 0x562789609cb0 2024-03-24T00:49:22.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.951+0000 7fa488813740 1 -- 172.21.15.67:0/2777548725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562789609cb0 msgr2=0x5627895c1ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:22.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.951+0000 7fa488813740 1 --2- 172.21.15.67:0/2777548725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562789609cb0 0x5627895c1ec0 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7fa470000c00 tx=0x7fa47004b000 comp rx=0 tx=0).stop 2024-03-24T00:49:22.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.951+0000 7fa488813740 1 -- 172.21.15.67:0/2777548725 shutdown_connections 2024-03-24T00:49:22.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.951+0000 7fa488813740 1 --2- 172.21.15.67:0/2777548725 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56278961ee40 0x5627895c05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.951+0000 7fa488813740 1 --2- 172.21.15.67:0/2777548725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562789609cb0 0x5627895c1ec0 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.951+0000 7fa488813740 1 --2- 172.21.15.67:0/2777548725 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5627896098d0 0x5627895c1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.951+0000 7fa488813740 1 -- 172.21.15.67:0/2777548725 >> 172.21.15.67:0/2777548725 conn(0x5627895e9210 msgr2=0x562789612c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:22.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.951+0000 7fa488813740 1 -- 172.21.15.67:0/2777548725 shutdown_connections 2024-03-24T00:49:22.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.951+0000 7fa488813740 1 -- 172.21.15.67:0/2777548725 wait complete. 2024-03-24T00:49:22.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.952+0000 7fa488813740 1 -- start start 2024-03-24T00:49:22.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.952+0000 7fa488813740 1 Processor -- start 2024-03-24T00:49:22.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.953+0000 7fa488813740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5627896098d0 0x56278965fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.953+0000 7fa488813740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562789609cb0 0x5627895ece10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.953+0000 7fa488813740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56278961ee40 0x5627895c05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.953+0000 7fa488813740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627895c2fe0 con 0x562789609cb0 2024-03-24T00:49:22.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.953+0000 7fa488813740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4800aff90 con 0x56278961ee40 2024-03-24T00:49:22.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.953+0000 7fa488813740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5627895ee5d0 con 0x5627896098d0 2024-03-24T00:49:22.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.953+0000 7fa488813740 10 client.0 _ll_register_callbacks cb 0x7fa4700096f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:49:22.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.953+0000 7fa47effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562789609cb0 0x5627895ece10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.953+0000 7fa47effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562789609cb0 0x5627895ece10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46746/0 (socket says 172.21.15.67:46746) 2024-03-24T00:49:22.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.953+0000 7fa47effd640 1 -- 172.21.15.67:0/1435100070 learned_addr learned my addr 172.21.15.67:0/1435100070 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:22.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.953+0000 7fa47ffff640 1 --2- 172.21.15.67:0/1435100070 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56278961ee40 0x5627895c05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.953+0000 7fa47f7fe640 1 --2- 172.21.15.67:0/1435100070 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5627896098d0 0x56278965fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.953+0000 7fa47effd640 1 -- 172.21.15.67:0/1435100070 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5627896098d0 msgr2=0x56278965fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:22.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.953+0000 7fa47effd640 1 --2- 172.21.15.67:0/1435100070 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5627896098d0 0x56278965fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.953+0000 7fa47effd640 1 -- 172.21.15.67:0/1435100070 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56278961ee40 msgr2=0x5627895c05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:22.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.953+0000 7fa47effd640 1 --2- 172.21.15.67:0/1435100070 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56278961ee40 0x5627895c05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.953+0000 7fa47effd640 1 -- 172.21.15.67:0/1435100070 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5627895ea1a0 con 0x562789609cb0 2024-03-24T00:49:22.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.954+0000 7fa47effd640 1 --2- 172.21.15.67:0/1435100070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562789609cb0 0x5627895ece10 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7fa47002b9c0 tx=0x7fa470031000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a2bbd3fb2810c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:22.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.954+0000 7fa45e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:49:22.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.954+0000 7fa45e7fc640 1 -- 172.21.15.67:0/1435100070 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa470069060 con 0x562789609cb0 2024-03-24T00:49:22.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.954+0000 7fa45e7fc640 1 -- 172.21.15.67:0/1435100070 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(121..121 src has 1..121) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fa470003810 con 0x562789609cb0 2024-03-24T00:49:22.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.954+0000 7fa488813740 1 -- 172.21.15.67:0/1435100070 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa470047030 con 0x562789609cb0 2024-03-24T00:49:22.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.954+0000 7fa45e7fc640 1 -- 172.21.15.67:0/1435100070 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa47006f050 con 0x562789609cb0 2024-03-24T00:49:22.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.954+0000 7fa488813740 10 client.9654 fetch_fsmap learned FSMap version 92 2024-03-24T00:49:22.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.954+0000 7fa488813740 1 -- 172.21.15.67:0/1435100070 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=92,monmap=2+}) v3 -- 0x5627895e8a80 con 0x562789609cb0 2024-03-24T00:49:22.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.954+0000 7fa45e7fc640 1 -- 172.21.15.67:0/1435100070 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa470047030 con 0x562789609cb0 2024-03-24T00:49:22.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.954+0000 7fa45e7fc640 1 -- 172.21.15.67:0/1435100070 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 92) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fa470072020 con 0x562789609cb0 2024-03-24T00:49:22.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.954+0000 7fa488813740 10 client.9654 fetch_fsmap finished waiting for FSMap version 92 2024-03-24T00:49:22.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.954+0000 7fa488813740 10 client.9654 resolve_mds: resolved 8:0 to role '8:0' aka daemon mds.a 2024-03-24T00:49:22.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.954+0000 7fa488813740 20 client.9654 populate_metadata read hostname 'smithi067' 2024-03-24T00:49:22.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.954+0000 7fa488813740 1 --2- 172.21.15.67:0/1435100070 >> [v2:172.21.15.67:6833/3833808901,v1:172.21.15.67:6835/3833808901] conn(0x5627895ed350 0x5627895c0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:22.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.954+0000 7fa488813740 4 client.9654 mds_command: new command op to 9510 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:49:22.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.954+0000 7fa488813740 1 -- 172.21.15.67:0/1435100070 --> [v2:172.21.15.67:6833/3833808901,v1:172.21.15.67:6835/3833808901] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56278952dc60 con 0x5627895ed350 2024-03-24T00:49:22.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.954+0000 7fa47f7fe640 1 --2- 172.21.15.67:0/1435100070 >> [v2:172.21.15.67:6833/3833808901,v1:172.21.15.67:6835/3833808901] conn(0x5627895ed350 0x5627895c0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:22.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.955+0000 7fa45e7fc640 10 client.9654 ms_handle_connect on v2:172.21.15.67:6833/3833808901 2024-03-24T00:49:22.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.955+0000 7fa47f7fe640 1 --2- 172.21.15.67:0/1435100070 >> [v2:172.21.15.67:6833/3833808901,v1:172.21.15.67:6835/3833808901] conn(0x5627895ed350 0x5627895c0b20 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=637beb52ab3eeba3 server_cookie=9321c4d1cf7cbd2b in_seq=0 out_seq=0 2024-03-24T00:49:22.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.968+0000 7fa45e7fc640 1 -- 172.21.15.67:0/1435100070 <== mds.0 v2:172.21.15.67:6833/3833808901 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56278952dc60 con 0x5627895ed350 2024-03-24T00:49:22.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.968+0000 7fa45e7fc640 10 client.9654 handle_command_reply: tid=0 2024-03-24T00:49:22.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.968+0000 7fa488813740 1 client.9654 shutdown 2024-03-24T00:49:22.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.968+0000 7fa488813740 1 -- 172.21.15.67:0/1435100070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562789609cb0 msgr2=0x5627895ece10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:22.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.968+0000 7fa488813740 1 --2- 172.21.15.67:0/1435100070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562789609cb0 0x5627895ece10 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7fa47002b9c0 tx=0x7fa470031000 comp rx=0 tx=0).stop 2024-03-24T00:49:22.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.968+0000 7fa488813740 1 -- 172.21.15.67:0/1435100070 shutdown_connections 2024-03-24T00:49:22.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.968+0000 7fa488813740 1 --2- 172.21.15.67:0/1435100070 >> [v2:172.21.15.67:6833/3833808901,v1:172.21.15.67:6835/3833808901] conn(0x5627895ed350 0x5627895c0b20 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.968+0000 7fa488813740 1 --2- 172.21.15.67:0/1435100070 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56278961ee40 0x5627895c05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.968+0000 7fa488813740 1 --2- 172.21.15.67:0/1435100070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562789609cb0 0x5627895ece10 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.968+0000 7fa488813740 1 --2- 172.21.15.67:0/1435100070 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5627896098d0 0x56278965fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.968+0000 7fa45e7fc640 0 client.9654 ms_handle_reset on v2:172.21.15.67:6833/3833808901 2024-03-24T00:49:22.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.968+0000 7fa488813740 1 -- 172.21.15.67:0/1435100070 >> 172.21.15.67:0/1435100070 conn(0x5627895e9210 msgr2=0x56278961e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:22.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.969+0000 7fa488813740 1 -- 172.21.15.67:0/1435100070 shutdown_connections 2024-03-24T00:49:22.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.969+0000 7fa488813740 1 -- 172.21.15.67:0/1435100070 wait complete. 2024-03-24T00:49:22.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.969+0000 7fa488813740 20 client.9654 trim_cache size 0 max 16384 2024-03-24T00:49:22.971 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:49:22.971 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "", 2024-03-24T00:49:22.972 INFO:teuthology.orchestra.run.smithi067.stdout: "return_code": 0 2024-03-24T00:49:22.972 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:49:22.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.969+0000 7fa485b59640 1 -- 172.21.15.67:0/1727183402 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa46c06bbd0 msgr2=0x7fa46c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:22.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.969+0000 7fa485b59640 1 --2- 172.21.15.67:0/1727183402 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa46c06bbd0 0x7fa46c06e090 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fa470036000 tx=0x7fa47003b040 comp rx=0 tx=0).stop 2024-03-24T00:49:22.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.969+0000 7fa485b59640 1 -- 172.21.15.67:0/1727183402 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa48009a240 msgr2=0x7fa480159950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:22.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.969+0000 7fa485b59640 1 --2- 172.21.15.67:0/1727183402 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa48009a240 0x7fa480159950 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7fa47400d5f0 tx=0x7fa47400dab0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.969+0000 7fa485b59640 1 -- 172.21.15.67:0/1727183402 shutdown_connections 2024-03-24T00:49:22.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.969+0000 7fa485b59640 1 --2- 172.21.15.67:0/1727183402 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa46c06bbd0 0x7fa46c06e090 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.969+0000 7fa485b59640 1 --2- 172.21.15.67:0/1727183402 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa48015f7b0 0x7fa480071500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.969+0000 7fa485b59640 1 --2- 172.21.15.67:0/1727183402 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa4800b2ba0 0x7fa480070fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.969+0000 7fa485b59640 1 --2- 172.21.15.67:0/1727183402 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa48009a240 0x7fa480159950 unknown :-1 s=CLOSED pgs=1821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:22.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.969+0000 7fa485b59640 1 -- 172.21.15.67:0/1727183402 >> 172.21.15.67:0/1727183402 conn(0x7fa4800b3650 msgr2=0x7fa48015ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:22.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.969+0000 7fa485b59640 1 -- 172.21.15.67:0/1727183402 shutdown_connections 2024-03-24T00:49:22.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:22.969+0000 7fa485b59640 1 -- 172.21.15.67:0/1727183402 wait complete. 2024-03-24T00:49:22.981 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-24T00:49:23.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.082+0000 7f5925a27640 1 Processor -- start 2024-03-24T00:49:23.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.082+0000 7f5925a27640 1 -- start start 2024-03-24T00:49:23.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.083+0000 7f5925a27640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f592006d400 0x7f59200b6d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.083+0000 7f5925a27640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f59200d3420 0x7f59200b72c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.083+0000 7f5925a27640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f592015e3e0 con 0x7f592015e000 2024-03-24T00:49:23.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.083+0000 7f5925a27640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59200b8950 con 0x7f592006d400 2024-03-24T00:49:23.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.083+0000 7f5925a27640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59200b8c30 con 0x7f59200d3420 2024-03-24T00:49:23.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.083+0000 7f59167fc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f592006d400 0x7f59200b6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.083+0000 7f59167fc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f592006d400 0x7f59200b6d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46754/0 (socket says 172.21.15.67:46754) 2024-03-24T00:49:23.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.083+0000 7f59167fc640 1 -- 172.21.15.67:0/2320110831 learned_addr learned my addr 172.21.15.67:0/2320110831 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:23.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.083+0000 7f59167fc640 1 -- 172.21.15.67:0/2320110831 >> v2:172.21.15.72:3300/0 conn(0x7f59200d3420 msgr2=0x7f59200b72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:23.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.083+0000 7f59167fc640 1 --2- 172.21.15.67:0/2320110831 >> v2:172.21.15.72:3300/0 conn(0x7f59200d3420 0x7f59200b72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.083+0000 7f59167fc640 1 -- 172.21.15.67:0/2320110831 >> v1:172.21.15.72:6789/0 conn(0x7f592015e000 legacy=0x7f592006ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:23.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.083+0000 7f59167fc640 1 -- 172.21.15.67:0/2320110831 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59200b8f10 con 0x7f592006d400 2024-03-24T00:49:23.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.084+0000 7f59167fc640 1 --2- 172.21.15.67:0/2320110831 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f592006d400 0x7f59200b6d80 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f59200ce2c0 tx=0x7f590802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c41730268c74dec server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:23.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.084+0000 7f591e7fc640 1 -- 172.21.15.67:0/2320110831 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5908037070 con 0x7f592006d400 2024-03-24T00:49:23.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.084+0000 7f591e7fc640 1 -- 172.21.15.67:0/2320110831 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5908004230 con 0x7f592006d400 2024-03-24T00:49:23.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.084+0000 7f591e7fc640 1 -- 172.21.15.67:0/2320110831 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5908003610 con 0x7f592006d400 2024-03-24T00:49:23.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.085+0000 7f5925a27640 1 -- 172.21.15.67:0/2320110831 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f592006d400 msgr2=0x7f59200b6d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.085+0000 7f5925a27640 1 --2- 172.21.15.67:0/2320110831 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f592006d400 0x7f59200b6d80 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f59200ce2c0 tx=0x7f590802b860 comp rx=0 tx=0).stop 2024-03-24T00:49:23.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.085+0000 7f5925a27640 1 -- 172.21.15.67:0/2320110831 shutdown_connections 2024-03-24T00:49:23.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.085+0000 7f5925a27640 1 --2- 172.21.15.67:0/2320110831 >> v2:172.21.15.72:3300/0 conn(0x7f59200d3420 0x7f59200b72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.085+0000 7f5925a27640 1 --2- 172.21.15.67:0/2320110831 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f592006d400 0x7f59200b6d80 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.085+0000 7f5925a27640 1 -- 172.21.15.67:0/2320110831 >> 172.21.15.67:0/2320110831 conn(0x7f5920067220 msgr2=0x7f5920067630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:23.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.085+0000 7f5925a27640 1 -- 172.21.15.67:0/2320110831 shutdown_connections 2024-03-24T00:49:23.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.085+0000 7f5925a27640 1 -- 172.21.15.67:0/2320110831 wait complete. 2024-03-24T00:49:23.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.086+0000 7f5925a27640 1 Processor -- start 2024-03-24T00:49:23.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.086+0000 7f5925a27640 1 -- start start 2024-03-24T00:49:23.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.087+0000 7f5925a27640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f592006d400 0x7f592014cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.087+0000 7f5925a27640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f59200d3420 0x7f59200c9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.087+0000 7f5925a27640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f592015e000 0x7f59200b3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.087+0000 7f5925a27640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59200b9b30 con 0x7f592015e000 2024-03-24T00:49:23.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.087+0000 7f5925a27640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59200b7800 con 0x7f592006d400 2024-03-24T00:49:23.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.087+0000 7f5925a27640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59201614d0 con 0x7f59200d3420 2024-03-24T00:49:23.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.087+0000 7f591f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f592015e000 0x7f59200b3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.087+0000 7f591f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f592015e000 0x7f59200b3770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46770/0 (socket says 172.21.15.67:46770) 2024-03-24T00:49:23.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.087+0000 7f591f7fe640 1 -- 172.21.15.67:0/1019476297 learned_addr learned my addr 172.21.15.67:0/1019476297 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:23.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.087+0000 7f591effd640 1 --2- 172.21.15.67:0/1019476297 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f592006d400 0x7f592014cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.087+0000 7f59167fc640 1 --2- 172.21.15.67:0/1019476297 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f59200d3420 0x7f59200c9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.087+0000 7f591f7fe640 1 -- 172.21.15.67:0/1019476297 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f59200d3420 msgr2=0x7f59200c9fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.087+0000 7f591f7fe640 1 --2- 172.21.15.67:0/1019476297 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f59200d3420 0x7f59200c9fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.087+0000 7f591f7fe640 1 -- 172.21.15.67:0/1019476297 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f592006d400 msgr2=0x7f592014cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.087+0000 7f591f7fe640 1 --2- 172.21.15.67:0/1019476297 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f592006d400 0x7f592014cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.087+0000 7f591f7fe640 1 -- 172.21.15.67:0/1019476297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5920160c30 con 0x7f592015e000 2024-03-24T00:49:23.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.087+0000 7f591f7fe640 1 --2- 172.21.15.67:0/1019476297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f592015e000 0x7f59200b3770 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f591800b970 tx=0x7f591800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:23.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.088+0000 7f591cff9640 1 -- 172.21.15.67:0/1019476297 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5918019070 con 0x7f592015e000 2024-03-24T00:49:23.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.088+0000 7f591cff9640 1 -- 172.21.15.67:0/1019476297 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f591800cb60 con 0x7f592015e000 2024-03-24T00:49:23.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.088+0000 7f5925a27640 1 -- 172.21.15.67:0/1019476297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5920160f10 con 0x7f592015e000 2024-03-24T00:49:23.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.088+0000 7f591cff9640 1 -- 172.21.15.67:0/1019476297 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5918014440 con 0x7f592015e000 2024-03-24T00:49:23.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.088+0000 7f5925a27640 1 -- 172.21.15.67:0/1019476297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59200c7930 con 0x7f592015e000 2024-03-24T00:49:23.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.089+0000 7f5925a27640 1 -- 172.21.15.67:0/1019476297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f592004f400 con 0x7f592015e000 2024-03-24T00:49:23.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.089+0000 7f591cff9640 1 -- 172.21.15.67:0/1019476297 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f591801c070 con 0x7f592015e000 2024-03-24T00:49:23.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.089+0000 7f591cff9640 1 --2- 172.21.15.67:0/1019476297 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f590006bbd0 0x7f590006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.090+0000 7f591cff9640 1 -- 172.21.15.67:0/1019476297 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f591808a910 con 0x7f592015e000 2024-03-24T00:49:23.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.090+0000 7f591effd640 1 --2- 172.21.15.67:0/1019476297 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f590006bbd0 0x7f590006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.090+0000 7f591effd640 1 --2- 172.21.15.67:0/1019476297 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f590006bbd0 0x7f590006e090 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f591000a730 tx=0x7f5910008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:23.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.094+0000 7f591cff9640 1 -- 172.21.15.67:0/1019476297 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f59180592b0 con 0x7f592015e000 2024-03-24T00:49:23.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.314+0000 7f5925a27640 1 -- 172.21.15.67:0/1019476297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f59200d0fa0 con 0x7f592015e000 2024-03-24T00:49:23.315 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-24T00:49:23.314+0000 7fbfcfa11640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T00:49:23.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.316+0000 7f591cff9640 1 -- 172.21.15.67:0/1019476297 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v93) v1 ==== 161+0+0 (secure 0 0 0) 0x7f5918058bb0 con 0x7f592015e000 2024-03-24T00:49:23.316 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-24T00:49:23.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.319+0000 7f5925a27640 1 -- 172.21.15.67:0/1019476297 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f590006bbd0 msgr2=0x7f590006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.319+0000 7f5925a27640 1 --2- 172.21.15.67:0/1019476297 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f590006bbd0 0x7f590006e090 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f591000a730 tx=0x7f5910008040 comp rx=0 tx=0).stop 2024-03-24T00:49:23.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.319+0000 7f5925a27640 1 -- 172.21.15.67:0/1019476297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f592015e000 msgr2=0x7f59200b3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.319+0000 7f5925a27640 1 --2- 172.21.15.67:0/1019476297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f592015e000 0x7f59200b3770 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f591800b970 tx=0x7f591800be30 comp rx=0 tx=0).stop 2024-03-24T00:49:23.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.319+0000 7f5925a27640 1 -- 172.21.15.67:0/1019476297 shutdown_connections 2024-03-24T00:49:23.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.319+0000 7f5925a27640 1 --2- 172.21.15.67:0/1019476297 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f590006bbd0 0x7f590006e090 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.319+0000 7f5925a27640 1 --2- 172.21.15.67:0/1019476297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f592015e000 0x7f59200b3770 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.319+0000 7f5925a27640 1 --2- 172.21.15.67:0/1019476297 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f59200d3420 0x7f59200c9fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.319+0000 7f5925a27640 1 --2- 172.21.15.67:0/1019476297 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f592006d400 0x7f592014cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.319+0000 7f5925a27640 1 -- 172.21.15.67:0/1019476297 >> 172.21.15.67:0/1019476297 conn(0x7f5920067220 msgr2=0x7f59200d17a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:23.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.319+0000 7f5925a27640 1 -- 172.21.15.67:0/1019476297 shutdown_connections 2024-03-24T00:49:23.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.319+0000 7f5925a27640 1 -- 172.21.15.67:0/1019476297 wait complete. 2024-03-24T00:49:23.322 INFO:tasks.ceph.mds.a.smithi067.stderr: -1113> 2024-03-24T00:49:19.474+0000 7fa4888e4640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T00:49:23.332 DEBUG:teuthology.orchestra.run.smithi067:> cephfs-journal-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 --rank cephfs:0 journal reset --force 2024-03-24T00:49:23.347 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-03-24T00:49:23.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.372+0000 7f49dbcda280 1 -- start start 2024-03-24T00:49:23.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.372+0000 7f49dbcda280 1 Processor -- start 2024-03-24T00:49:23.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.373+0000 7f49dbcda280 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556cf8aaac00 0x556cf8b52580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.373+0000 7f49dbcda280 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556cf8aab800 0x556cf8b52b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.373+0000 7f49dbcda280 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556cf8aabc00 0x556cf8b53080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.373+0000 7f49dbcda280 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cf98da180 con 0x556cf8aaac00 2024-03-24T00:49:23.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.373+0000 7f49dbcda280 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cf98da300 con 0x556cf8aab800 2024-03-24T00:49:23.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.373+0000 7f49dbcda280 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x556cf98da480 con 0x556cf8aabc00 2024-03-24T00:49:23.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.373+0000 7f49da488640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556cf8aaac00 0x556cf8b52580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.373+0000 7f49da488640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556cf8aaac00 0x556cf8b52580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46810/0 (socket says 172.21.15.67:46810) 2024-03-24T00:49:23.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.373+0000 7f49da488640 1 -- 172.21.15.67:0/224569962 learned_addr learned my addr 172.21.15.67:0/224569962 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:23.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.373+0000 7f49da488640 1 -- 172.21.15.67:0/224569962 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556cf8aabc00 msgr2=0x556cf8b53080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:23.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.373+0000 7f49da488640 1 --2- 172.21.15.67:0/224569962 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556cf8aabc00 0x556cf8b53080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.373+0000 7f49da488640 1 -- 172.21.15.67:0/224569962 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556cf8aab800 msgr2=0x556cf8b52b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:23.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.373+0000 7f49d9c87640 1 --2- 172.21.15.67:0/224569962 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556cf8aab800 0x556cf8b52b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.373+0000 7f49da488640 1 --2- 172.21.15.67:0/224569962 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556cf8aab800 0x556cf8b52b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.373+0000 7f49da488640 1 -- 172.21.15.67:0/224569962 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556cf8ae0780 con 0x556cf8aaac00 2024-03-24T00:49:23.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.373+0000 7f49da488640 1 --2- 172.21.15.67:0/224569962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556cf8aaac00 0x556cf8b52580 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x556cf98c1440 tx=0x556cf9842b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e2c0be642d4d1eb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:23.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.373+0000 7f49d7c83640 1 -- 172.21.15.67:0/224569962 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x556cf8ad0680 con 0x556cf8aaac00 2024-03-24T00:49:23.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.373+0000 7f49d7c83640 1 -- 172.21.15.67:0/224569962 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x556cf8ad6700 con 0x556cf8aaac00 2024-03-24T00:49:23.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.373+0000 7f49d7c83640 1 -- 172.21.15.67:0/224569962 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x556cf8ad0b60 con 0x556cf8aaac00 2024-03-24T00:49:23.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.373+0000 7f49dbcda280 1 -- 172.21.15.67:0/224569962 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556cf8ae14a0 con 0x556cf8aaac00 2024-03-24T00:49:23.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.373+0000 7f49dbcda280 1 -- 172.21.15.67:0/224569962 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x556cf8ae0000 con 0x556cf8aaac00 2024-03-24T00:49:23.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.373+0000 7f49dbcda280 4 waiting for MDS map... 2024-03-24T00:49:23.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.374+0000 7f49d7c83640 1 -- 172.21.15.67:0/224569962 <== mon.0 v2:172.21.15.67:3300/0 4 ==== osd_map(122..122 src has 1..122) v4 ==== 6273+0+0 (secure 0 0 0) 0x556cf9976600 con 0x556cf8aaac00 2024-03-24T00:49:23.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.374+0000 7f49d7c83640 1 -- 172.21.15.67:0/224569962 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 93) v1 ==== 2735+0+0 (secure 0 0 0) 0x556cf8b05180 con 0x556cf8aaac00 2024-03-24T00:49:23.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.374+0000 7f49dbcda280 4 Got MDS map 93 2024-03-24T00:49:23.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.374+0000 7f49dbcda280 10 main: JournalTool::main 2024-03-24T00:49:23.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.374+0000 7f49dbcda280 4 main: JournalTool: connecting to RADOS... 2024-03-24T00:49:23.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.374+0000 7f49dbcda280 1 Processor -- start 2024-03-24T00:49:23.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.374+0000 7f49dbcda280 1 -- start start 2024-03-24T00:49:23.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.375+0000 7f49dbcda280 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556cf997d400 0x556cf8b53600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.375+0000 7f49dbcda280 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556cf997d800 0x556cf8b53b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.375+0000 7f49dbcda280 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556cf997dc00 0x556cf8b54100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.375+0000 7f49dbcda280 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cf98da600 con 0x556cf997dc00 2024-03-24T00:49:23.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.375+0000 7f49dbcda280 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cf98da780 con 0x556cf997d800 2024-03-24T00:49:23.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.375+0000 7f49dbcda280 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x556cf98da900 con 0x556cf997d400 2024-03-24T00:49:23.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.375+0000 7f49d9c87640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556cf997dc00 0x556cf8b54100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.375+0000 7f49d9c87640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556cf997dc00 0x556cf8b54100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46824/0 (socket says 172.21.15.67:46824) 2024-03-24T00:49:23.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.375+0000 7f49d9c87640 1 -- 172.21.15.67:0/2566159950 learned_addr learned my addr 172.21.15.67:0/2566159950 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:23.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.375+0000 7f49dac89640 1 --2- 172.21.15.67:0/2566159950 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556cf997d400 0x556cf8b53600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.375+0000 7f49da488640 1 --2- 172.21.15.67:0/2566159950 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556cf997d800 0x556cf8b53b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.375+0000 7f49d9c87640 1 -- 172.21.15.67:0/2566159950 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556cf997d400 msgr2=0x556cf8b53600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:23.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.375+0000 7f49d9c87640 1 --2- 172.21.15.67:0/2566159950 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556cf997d400 0x556cf8b53600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.375+0000 7f49d9c87640 1 -- 172.21.15.67:0/2566159950 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556cf997d800 msgr2=0x556cf8b53b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:23.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.375+0000 7f49d9c87640 1 --2- 172.21.15.67:0/2566159950 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556cf997d800 0x556cf8b53b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.375+0000 7f49d9c87640 1 -- 172.21.15.67:0/2566159950 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556cf8ae03c0 con 0x556cf997dc00 2024-03-24T00:49:23.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.375+0000 7f49dac89640 1 --2- 172.21.15.67:0/2566159950 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556cf997d400 0x556cf8b53600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:49:23.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.375+0000 7f49da488640 1 --2- 172.21.15.67:0/2566159950 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556cf997d800 0x556cf8b53b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:49:23.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.375+0000 7f49d9c87640 1 --2- 172.21.15.67:0/2566159950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556cf997dc00 0x556cf8b54100 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x556cf996e8d0 tx=0x556cf9992fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39e4bad347b8fa52 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:23.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.375+0000 7f49d5c7f640 1 -- 172.21.15.67:0/2566159950 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x556cf8ad0680 con 0x556cf997dc00 2024-03-24T00:49:23.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.376+0000 7f49d5c7f640 1 -- 172.21.15.67:0/2566159950 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x556cf8ad6fc0 con 0x556cf997dc00 2024-03-24T00:49:23.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.376+0000 7f49d5c7f640 1 -- 172.21.15.67:0/2566159950 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x556cf8ad0b60 con 0x556cf997dc00 2024-03-24T00:49:23.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.376+0000 7f49dbcda280 1 -- 172.21.15.67:0/2566159950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556cf997dc00 msgr2=0x556cf8b54100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.376+0000 7f49dbcda280 1 --2- 172.21.15.67:0/2566159950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556cf997dc00 0x556cf8b54100 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x556cf996e8d0 tx=0x556cf9992fc0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.376+0000 7f49dbcda280 1 -- 172.21.15.67:0/2566159950 shutdown_connections 2024-03-24T00:49:23.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.376+0000 7f49dbcda280 1 --2- 172.21.15.67:0/2566159950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556cf997dc00 0x556cf8b54100 unknown :-1 s=CLOSED pgs=1832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.376+0000 7f49dbcda280 1 --2- 172.21.15.67:0/2566159950 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556cf997d800 0x556cf8b53b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.376+0000 7f49dbcda280 1 --2- 172.21.15.67:0/2566159950 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556cf997d400 0x556cf8b53600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.376+0000 7f49dbcda280 1 -- 172.21.15.67:0/2566159950 >> 172.21.15.67:0/2566159950 conn(0x556cf997d000 msgr2=0x556cf8b9b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:23.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.376+0000 7f49dbcda280 1 -- 172.21.15.67:0/2566159950 shutdown_connections 2024-03-24T00:49:23.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.376+0000 7f49dbcda280 1 -- 172.21.15.67:0/2566159950 wait complete. 2024-03-24T00:49:23.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.376+0000 7f49dbcda280 1 Processor -- start 2024-03-24T00:49:23.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.376+0000 7f49dbcda280 1 -- start start 2024-03-24T00:49:23.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.377+0000 7f49dbcda280 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556cf997d400 0x556cf8b55180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.377+0000 7f49dbcda280 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556cf997d800 0x556cf8b54c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.377+0000 7f49dbcda280 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556cf997dc00 0x556cf8b54680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.377+0000 7f49dbcda280 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cf98dbc80 con 0x556cf997d800 2024-03-24T00:49:23.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.377+0000 7f49dbcda280 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cf98dbb00 con 0x556cf997d400 2024-03-24T00:49:23.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.377+0000 7f49dbcda280 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x556cf98db980 con 0x556cf997dc00 2024-03-24T00:49:23.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.377+0000 7f49da488640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556cf997d800 0x556cf8b54c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.377+0000 7f49da488640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556cf997d800 0x556cf8b54c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46834/0 (socket says 172.21.15.67:46834) 2024-03-24T00:49:23.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.377+0000 7f49da488640 1 -- 172.21.15.67:0/671152449 learned_addr learned my addr 172.21.15.67:0/671152449 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:23.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.377+0000 7f49d9c87640 1 --2- 172.21.15.67:0/671152449 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556cf997dc00 0x556cf8b54680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.377+0000 7f49dac89640 1 --2- 172.21.15.67:0/671152449 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556cf997d400 0x556cf8b55180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.377+0000 7f49da488640 1 -- 172.21.15.67:0/671152449 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556cf997dc00 msgr2=0x556cf8b54680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.377+0000 7f49da488640 1 --2- 172.21.15.67:0/671152449 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556cf997dc00 0x556cf8b54680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.377+0000 7f49da488640 1 -- 172.21.15.67:0/671152449 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556cf997d400 msgr2=0x556cf8b55180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.377+0000 7f49da488640 1 --2- 172.21.15.67:0/671152449 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556cf997d400 0x556cf8b55180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.377+0000 7f49da488640 1 -- 172.21.15.67:0/671152449 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556cf8ae0780 con 0x556cf997d800 2024-03-24T00:49:23.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.377+0000 7f49d9c87640 1 --2- 172.21.15.67:0/671152449 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556cf997dc00 0x556cf8b54680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:49:23.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.378+0000 7f49da488640 1 --2- 172.21.15.67:0/671152449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556cf997d800 0x556cf8b54c00 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x556cf996f5c0 tx=0x556cf8b60cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:23.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.378+0000 7f49d447c640 1 -- 172.21.15.67:0/671152449 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x556cf8ad0820 con 0x556cf997d800 2024-03-24T00:49:23.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.378+0000 7f49d447c640 1 -- 172.21.15.67:0/671152449 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x556cf8ad6e00 con 0x556cf997d800 2024-03-24T00:49:23.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.378+0000 7f49dbcda280 1 -- 172.21.15.67:0/671152449 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x556cf8ae14a0 con 0x556cf997d800 2024-03-24T00:49:23.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.378+0000 7f49d447c640 1 -- 172.21.15.67:0/671152449 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x556cf8ad0680 con 0x556cf997d800 2024-03-24T00:49:23.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.378+0000 7f49dbcda280 1 -- 172.21.15.67:0/671152449 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556cf8ae0960 con 0x556cf997d800 2024-03-24T00:49:23.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.378+0000 7f49dbcda280 4 main: JournalTool: resolving pool 17 2024-03-24T00:49:23.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.379+0000 7f49d447c640 1 -- 172.21.15.67:0/671152449 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x556cf99f4c00 con 0x556cf997d800 2024-03-24T00:49:23.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.379+0000 7f49d447c640 1 --2- 172.21.15.67:0/671152449 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x556cf99edc00 0x556cf8b55700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.379+0000 7f49d447c640 1 -- 172.21.15.67:0/671152449 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6273+0+0 (secure 0 0 0) 0x556cf99eec00 con 0x556cf997d800 2024-03-24T00:49:23.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.379+0000 7f49dbcda280 4 main: JournalTool: creating IoCtx.. 2024-03-24T00:49:23.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.379+0000 7f49dbcda280 4 main: Executing for rank 0 2024-03-24T00:49:23.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.379+0000 7f49dac89640 1 --2- 172.21.15.67:0/671152449 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x556cf99edc00 0x556cf8b55700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.380+0000 7f49dac89640 1 --2- 172.21.15.67:0/671152449 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x556cf99edc00 0x556cf8b55700 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x556cf8b4da70 tx=0x556cf9992de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:23.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.380+0000 7f49dbcda280 1 -- start start 2024-03-24T00:49:23.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.380+0000 7f49dbcda280 1 Processor -- start 2024-03-24T00:49:23.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.380+0000 7f49dbcda280 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556cf8e6c400 0x556cf8b54100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.380+0000 7f49dbcda280 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556cf8e6c800 0x556cf8b53b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.380+0000 7f49dbcda280 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556cf8e6cc00 0x556cf8b53600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.380+0000 7f49dbcda280 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cf98db680 con 0x556cf8e6c800 2024-03-24T00:49:23.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.380+0000 7f49dbcda280 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cf98da600 con 0x556cf8e6c400 2024-03-24T00:49:23.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.380+0000 7f49dbcda280 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x556cf98da780 con 0x556cf8e6cc00 2024-03-24T00:49:23.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.381+0000 7f49da488640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556cf8e6c800 0x556cf8b53b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.381+0000 7f49da488640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556cf8e6c800 0x556cf8b53b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46844/0 (socket says 172.21.15.67:46844) 2024-03-24T00:49:23.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.381+0000 7f49da488640 1 -- 172.21.15.67:0/1155002795 learned_addr learned my addr 172.21.15.67:0/1155002795 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:23.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.381+0000 7f49dac89640 1 --2- 172.21.15.67:0/1155002795 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556cf8e6c400 0x556cf8b54100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.381+0000 7f49d9c87640 1 --2- 172.21.15.67:0/1155002795 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556cf8e6cc00 0x556cf8b53600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.381+0000 7f49da488640 1 -- 172.21.15.67:0/1155002795 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556cf8e6cc00 msgr2=0x556cf8b53600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:23.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.381+0000 7f49da488640 1 --2- 172.21.15.67:0/1155002795 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556cf8e6cc00 0x556cf8b53600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.381+0000 7f49da488640 1 -- 172.21.15.67:0/1155002795 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556cf8e6c400 msgr2=0x556cf8b54100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:23.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.381+0000 7f49da488640 1 --2- 172.21.15.67:0/1155002795 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556cf8e6c400 0x556cf8b54100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.381+0000 7f49da488640 1 -- 172.21.15.67:0/1155002795 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556cf8ae1680 con 0x556cf8e6c800 2024-03-24T00:49:23.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.381+0000 7f49da488640 1 --2- 172.21.15.67:0/1155002795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556cf8e6c800 0x556cf8b53b80 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x556cf9a17620 tx=0x556cf8e258c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2472e3f35410364 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:23.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.381+0000 7f49d1476640 1 -- 172.21.15.67:0/1155002795 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x556cf8da4d00 con 0x556cf8e6c800 2024-03-24T00:49:23.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.381+0000 7f49d1476640 1 -- 172.21.15.67:0/1155002795 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x556cf8ad6c40 con 0x556cf8e6c800 2024-03-24T00:49:23.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.381+0000 7f49d1476640 1 -- 172.21.15.67:0/1155002795 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x556cf8da4ea0 con 0x556cf8e6c800 2024-03-24T00:49:23.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.381+0000 7f49dbcda280 1 -- 172.21.15.67:0/1155002795 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556cf8ae0000 con 0x556cf8e6c800 2024-03-24T00:49:23.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.381+0000 7f49dbcda280 1 -- 172.21.15.67:0/1155002795 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x556cf8ae0780 con 0x556cf8e6c800 2024-03-24T00:49:23.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.381+0000 7f49dbcda280 4 waiting for MDS map... 2024-03-24T00:49:23.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.381+0000 7f49d1476640 1 -- 172.21.15.67:0/1155002795 <== mon.0 v2:172.21.15.67:3300/0 4 ==== osd_map(122..122 src has 1..122) v4 ==== 6273+0+0 (secure 0 0 0) 0x556cf8e7a600 con 0x556cf8e6c800 2024-03-24T00:49:23.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.382+0000 7f49d1476640 1 -- 172.21.15.67:0/1155002795 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 93) v1 ==== 2735+0+0 (secure 0 0 0) 0x556cf8b05500 con 0x556cf8e6c800 2024-03-24T00:49:23.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.382+0000 7f49dbcda280 4 Got MDS map 93 2024-03-24T00:49:23.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.382+0000 7f49dbcda280 1 --2- 172.21.15.67:0/1155002795 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x556cf8e6d000 0x556cf8e92000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.382+0000 7f49dbcda280 1 -- 172.21.15.67:0/1155002795 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:1 17.f 17:f1f69726:::400.00000000:head [writefull 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) v9 -- 0x556cf8e82800 con 0x556cf8e6d000 2024-03-24T00:49:23.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.382+0000 7f49dac89640 1 --2- 172.21.15.67:0/1155002795 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x556cf8e6d000 0x556cf8e92000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.382+0000 7f49dac89640 1 --2- 172.21.15.67:0/1155002795 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x556cf8e6d000 0x556cf8e92000 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=fe4290cd8f5d6b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:23.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.385+0000 7f49dac89640 1 -- 172.21.15.67:0/1155002795 <== osd.1 v2:172.21.15.67:6800/4062510404 1 ==== osd_op_reply(1 400.00000000 [writefull 0~22] v122'2 uv2 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x556cf8ba0000 con 0x556cf8e6d000 2024-03-24T00:49:23.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.385+0000 7f49dbcda280 1 --2- 172.21.15.67:0/1155002795 >> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] conn(0x556cf8e82c00 0x556cf8e92580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.385+0000 7f49dbcda280 1 -- 172.21.15.67:0/1155002795 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:2 17.14 17:292cf221:::200.00000000:head [writefull 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) v9 -- 0x556cf8e83000 con 0x556cf8e82c00 2024-03-24T00:49:23.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.385+0000 7f49d9c87640 1 --2- 172.21.15.67:0/1155002795 >> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] conn(0x556cf8e82c00 0x556cf8e92580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.385+0000 7f49d9c87640 1 --2- 172.21.15.67:0/1155002795 >> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] conn(0x556cf8e82c00 0x556cf8e92580 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=a5e69c93f6cbb625 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:23.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.388+0000 7f49d9c87640 1 -- 172.21.15.67:0/1155002795 <== osd.3 v2:172.21.15.67:6816/3303489481 1 ==== osd_op_reply(2 200.00000000 [writefull 0~90] v122'41 uv41 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x556cf8ba0500 con 0x556cf8e82c00 2024-03-24T00:49:23.398 INFO:teuthology.orchestra.run.smithi067.stdout:writing EResetJournal entry 2024-03-24T00:49:23.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.389+0000 7f49dbcda280 1 -- 172.21.15.67:0/1155002795 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:3 17.14 17:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) v9 -- 0x556cf8e83400 con 0x556cf8e82c00 2024-03-24T00:49:23.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.389+0000 7f49dbcda280 1 --2- 172.21.15.67:0/1155002795 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x556cf8e83800 0x556cf8e92b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.389+0000 7f49dbcda280 1 -- 172.21.15.67:0/1155002795 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:4 17.1b 17:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) v9 -- 0x556cf8e83c00 con 0x556cf8e83800 2024-03-24T00:49:23.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.389+0000 7f49dbcda280 1 --2- 172.21.15.67:0/1155002795 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x556cf8e83000 0x556cf8e93080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.389+0000 7f49dbcda280 1 -- 172.21.15.67:0/1155002795 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:5 17.b 17:d5c7a900:::200.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) v9 -- 0x556cf99ec800 con 0x556cf8e83000 2024-03-24T00:49:23.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.389+0000 7f49dbcda280 1 --2- 172.21.15.67:0/1155002795 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x556cf99ec000 0x556cf8e93600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.389+0000 7f49dbcda280 1 -- 172.21.15.67:0/1155002795 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:6 17.1f 17:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) v9 -- 0x556cf99ec400 con 0x556cf99ec000 2024-03-24T00:49:23.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.389+0000 7f49dbcda280 1 -- 172.21.15.67:0/1155002795 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:7 17.1a 17:5e41ab69:::200.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) v9 -- 0x556cf8eae000 con 0x556cf8e83000 2024-03-24T00:49:23.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.389+0000 7f49dbcda280 1 -- 172.21.15.67:0/1155002795 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:8 17.1e 17:7bf7e7b5:::200.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) v9 -- 0x556cf8eae400 con 0x556cf8e6d000 2024-03-24T00:49:23.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.389+0000 7f49da488640 1 --2- 172.21.15.67:0/1155002795 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x556cf8e83000 0x556cf8e93080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.389+0000 7f49dac89640 1 --2- 172.21.15.67:0/1155002795 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x556cf8e83800 0x556cf8e92b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.389+0000 7f49d9c87640 1 --2- 172.21.15.67:0/1155002795 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x556cf99ec000 0x556cf8e93600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.389+0000 7f49da488640 1 --2- 172.21.15.67:0/1155002795 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x556cf8e83000 0x556cf8e93080 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=82eb080c1d10c94d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:23.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.390+0000 7f49dac89640 1 --2- 172.21.15.67:0/1155002795 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x556cf8e83800 0x556cf8e92b00 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=78a44ae071d04a63 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:23.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.390+0000 7f49d9c87640 1 --2- 172.21.15.67:0/1155002795 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x556cf99ec000 0x556cf8e93600 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=ffb3e1fe76a8677d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:23.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.390+0000 7f49d9c87640 1 -- 172.21.15.67:0/1155002795 <== osd.3 v2:172.21.15.67:6816/3303489481 2 ==== osd_op_reply(3 200.00000001 [delete] v122'42 uv42 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x556cf8ba0500 con 0x556cf8e82c00 2024-03-24T00:49:23.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.390+0000 7f49dac89640 1 -- 172.21.15.67:0/1155002795 <== osd.1 v2:172.21.15.67:6800/4062510404 2 ==== osd_op_reply(8 200.00000006 [delete] v122'3 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x556cf8ba0000 con 0x556cf8e6d000 2024-03-24T00:49:23.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.390+0000 7f49da488640 1 -- 172.21.15.67:0/1155002795 <== osd.0 v2:172.21.15.67:6808/2628130519 1 ==== osd_op_reply(5 200.00000003 [delete] v122'6 uv5 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x556cf8ba0780 con 0x556cf8e83000 2024-03-24T00:49:23.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.392+0000 7f49da488640 1 -- 172.21.15.67:0/1155002795 <== osd.0 v2:172.21.15.67:6808/2628130519 2 ==== osd_op_reply(7 200.00000005 [delete] v122'3 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x556cf8ba0780 con 0x556cf8e83000 2024-03-24T00:49:23.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.392+0000 7f49dac89640 1 -- 172.21.15.67:0/1155002795 <== osd.4 v2:172.21.15.72:6808/477220793 1 ==== osd_op_reply(4 200.00000002 [delete] v122'3 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x556cf8ba0000 con 0x556cf8e83800 2024-03-24T00:49:23.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.392+0000 7f49cfc73640 1 -- 172.21.15.67:0/1155002795 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:9 17.14 17:2e2fa760:::200.00000001:head [write 0~48 in=48b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) v9 -- 0x556cf9887400 con 0x556cf8e82c00 2024-03-24T00:49:23.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.393+0000 7f49d9c87640 1 -- 172.21.15.67:0/1155002795 <== osd.6 v2:172.21.15.72:6816/1725206562 1 ==== osd_op_reply(6 200.00000004 [delete] v122'9 uv8 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x556cf8ba0500 con 0x556cf99ec000 2024-03-24T00:49:23.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.393+0000 7f49d9c87640 1 -- 172.21.15.67:0/1155002795 <== osd.3 v2:172.21.15.67:6816/3303489481 3 ==== osd_op_reply(9 200.00000001 [write 0~48] v122'43 uv43 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x556cf8ba0500 con 0x556cf8e82c00 2024-03-24T00:49:23.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.393+0000 7f49dbcda280 4 Successfully wrote new journal pointer and header for rank 8:0 2024-03-24T00:49:23.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.393+0000 7f49dbcda280 1 -- 172.21.15.67:0/1155002795 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x556cf8e83000 msgr2=0x556cf8e93080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.393+0000 7f49dbcda280 1 --2- 172.21.15.67:0/1155002795 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x556cf8e83000 0x556cf8e93080 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.393+0000 7f49dbcda280 1 -- 172.21.15.67:0/1155002795 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x556cf8e6d000 msgr2=0x556cf8e92000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.393+0000 7f49dbcda280 1 --2- 172.21.15.67:0/1155002795 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x556cf8e6d000 0x556cf8e92000 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.393+0000 7f49dbcda280 1 -- 172.21.15.67:0/1155002795 >> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] conn(0x556cf8e82c00 msgr2=0x556cf8e92580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.393+0000 7f49dbcda280 1 --2- 172.21.15.67:0/1155002795 >> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] conn(0x556cf8e82c00 0x556cf8e92580 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.393+0000 7f49dbcda280 1 -- 172.21.15.67:0/1155002795 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x556cf8e83800 msgr2=0x556cf8e92b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.393+0000 7f49dbcda280 1 --2- 172.21.15.67:0/1155002795 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x556cf8e83800 0x556cf8e92b00 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.393+0000 7f49dbcda280 1 -- 172.21.15.67:0/1155002795 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x556cf99ec000 msgr2=0x556cf8e93600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.393+0000 7f49d9c87640 1 -- 172.21.15.67:0/1155002795 reap_dead start 2024-03-24T00:49:23.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.393+0000 7f49dbcda280 1 --2- 172.21.15.67:0/1155002795 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x556cf99ec000 0x556cf8e93600 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.394+0000 7f49d9c87640 1 -- 172.21.15.67:0/1155002795 reap_dead start 2024-03-24T00:49:23.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.394+0000 7f49dbcda280 1 -- 172.21.15.67:0/1155002795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556cf8e6c800 msgr2=0x556cf8b53b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.394+0000 7f49dbcda280 1 --2- 172.21.15.67:0/1155002795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556cf8e6c800 0x556cf8b53b80 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x556cf9a17620 tx=0x556cf8e258c0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.394+0000 7f49dbcda280 1 -- 172.21.15.67:0/1155002795 shutdown_connections 2024-03-24T00:49:23.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.394+0000 7f49dbcda280 1 --2- 172.21.15.67:0/1155002795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556cf8e6c800 0x556cf8b53b80 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.394+0000 7f49dbcda280 1 -- 172.21.15.67:0/1155002795 >> 172.21.15.67:0/1155002795 conn(0x556cf8e6c000 msgr2=0x556cf8b9b800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:23.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.394+0000 7f49dbcda280 1 -- 172.21.15.67:0/1155002795 shutdown_connections 2024-03-24T00:49:23.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.394+0000 7f49dbcda280 1 -- 172.21.15.67:0/1155002795 wait complete. 2024-03-24T00:49:23.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.394+0000 7f49dbcda280 1 -- 172.21.15.67:0/671152449 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x556cf99edc00 msgr2=0x556cf8b55700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.394+0000 7f49dbcda280 1 --2- 172.21.15.67:0/671152449 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x556cf99edc00 0x556cf8b55700 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x556cf8b4da70 tx=0x556cf9992de0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.394+0000 7f49dbcda280 1 -- 172.21.15.67:0/671152449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556cf997d800 msgr2=0x556cf8b54c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.394+0000 7f49dbcda280 1 --2- 172.21.15.67:0/671152449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556cf997d800 0x556cf8b54c00 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x556cf996f5c0 tx=0x556cf8b60cc0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.394+0000 7f49dbcda280 1 -- 172.21.15.67:0/671152449 shutdown_connections 2024-03-24T00:49:23.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.394+0000 7f49dbcda280 1 --2- 172.21.15.67:0/671152449 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x556cf99edc00 0x556cf8b55700 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.394+0000 7f49dbcda280 1 --2- 172.21.15.67:0/671152449 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556cf997dc00 0x556cf8b54680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.394+0000 7f49dbcda280 1 --2- 172.21.15.67:0/671152449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556cf997d800 0x556cf8b54c00 unknown :-1 s=CLOSED pgs=1833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.394+0000 7f49dbcda280 1 --2- 172.21.15.67:0/671152449 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556cf997d400 0x556cf8b55180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.394+0000 7f49dbcda280 1 -- 172.21.15.67:0/671152449 >> 172.21.15.67:0/671152449 conn(0x556cf997d000 msgr2=0x556cf8b9b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:23.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.394+0000 7f49dbcda280 1 -- 172.21.15.67:0/671152449 shutdown_connections 2024-03-24T00:49:23.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.394+0000 7f49dbcda280 1 -- 172.21.15.67:0/671152449 wait complete. 2024-03-24T00:49:23.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.394+0000 7f49dbcda280 1 -- 172.21.15.67:0/224569962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556cf8aaac00 msgr2=0x556cf8b52580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.394+0000 7f49dbcda280 1 --2- 172.21.15.67:0/224569962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556cf8aaac00 0x556cf8b52580 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x556cf98c1440 tx=0x556cf9842b40 comp rx=0 tx=0).stop 2024-03-24T00:49:23.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.394+0000 7f49dbcda280 1 -- 172.21.15.67:0/224569962 shutdown_connections 2024-03-24T00:49:23.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.394+0000 7f49dbcda280 1 --2- 172.21.15.67:0/224569962 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x556cf8aabc00 0x556cf8b53080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.394+0000 7f49dbcda280 1 --2- 172.21.15.67:0/224569962 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556cf8aab800 0x556cf8b52b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.394+0000 7f49dbcda280 1 --2- 172.21.15.67:0/224569962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556cf8aaac00 0x556cf8b52580 unknown :-1 s=CLOSED pgs=1831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.394+0000 7f49dbcda280 1 -- 172.21.15.67:0/224569962 >> 172.21.15.67:0/224569962 conn(0x556cf8aab000 msgr2=0x556cf8b9a800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:23.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.395+0000 7f49dbcda280 1 -- 172.21.15.67:0/224569962 shutdown_connections 2024-03-24T00:49:23.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.395+0000 7f49dbcda280 1 -- 172.21.15.67:0/224569962 wait complete. 2024-03-24T00:49:23.409 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-journal-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', '--rank', 'cephfs:0', 'journal', 'reset', '--force'] in time 0:00:00.077309, result: writing EResetJournal entry 2024-03-24T00:49:23.410 DEBUG:teuthology.orchestra.run.smithi067:> cephfs-data-scan --debug-mds=20 --debug-ms=1 --debug-objecter=1 scan_extents cephfs_data 2024-03-24T00:49:23.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.483+0000 7ff875c3a740 1 -- start start 2024-03-24T00:49:23.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.483+0000 7ff875c3a740 1 Processor -- start 2024-03-24T00:49:23.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.483+0000 7ff875c3a740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5580d8c6e000 0x5580d7f21080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.483+0000 7ff875c3a740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5580d7e77c00 0x5580d7f21b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.483+0000 7ff875c3a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580d8c6e800 0x5580d7f21600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.483+0000 7ff875c3a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580d8c74480 con 0x5580d8c6e800 2024-03-24T00:49:23.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.483+0000 7ff875c3a740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580d8c74600 con 0x5580d8c6e000 2024-03-24T00:49:23.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.483+0000 7ff875c3a740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5580d8c74780 con 0x5580d7e77c00 2024-03-24T00:49:23.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.483+0000 7ff874557640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580d8c6e800 0x5580d7f21600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.483+0000 7ff874557640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580d8c6e800 0x5580d7f21600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38392/0 (socket says 172.21.15.67:38392) 2024-03-24T00:49:23.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.483+0000 7ff874557640 1 -- 172.21.15.67:0/2611870079 learned_addr learned my addr 172.21.15.67:0/2611870079 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:23.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.484+0000 7ff873d56640 1 --2- 172.21.15.67:0/2611870079 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5580d8c6e000 0x5580d7f21080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.484+0000 7ff873555640 1 --2- 172.21.15.67:0/2611870079 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5580d7e77c00 0x5580d7f21b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.484+0000 7ff874557640 1 -- 172.21.15.67:0/2611870079 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5580d7e77c00 msgr2=0x5580d7f21b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:23.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.484+0000 7ff874557640 1 --2- 172.21.15.67:0/2611870079 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5580d7e77c00 0x5580d7f21b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.484+0000 7ff874557640 1 -- 172.21.15.67:0/2611870079 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5580d8c6e000 msgr2=0x5580d7f21080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:23.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.484+0000 7ff874557640 1 --2- 172.21.15.67:0/2611870079 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5580d8c6e000 0x5580d7f21080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.484+0000 7ff874557640 1 -- 172.21.15.67:0/2611870079 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5580d7ea3680 con 0x5580d8c6e800 2024-03-24T00:49:23.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.484+0000 7ff873d56640 1 --2- 172.21.15.67:0/2611870079 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5580d8c6e000 0x5580d7f21080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:49:23.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.484+0000 7ff874557640 1 --2- 172.21.15.67:0/2611870079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580d8c6e800 0x5580d7f21600 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x5580d8ca5860 tx=0x5580d8c16000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49d67964181e953e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:23.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.484+0000 7ff871551640 1 -- 172.21.15.67:0/2611870079 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5580d7e9cd00 con 0x5580d8c6e800 2024-03-24T00:49:23.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.484+0000 7ff871551640 1 -- 172.21.15.67:0/2611870079 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5580d7ec01c0 con 0x5580d8c6e800 2024-03-24T00:49:23.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.484+0000 7ff871551640 1 -- 172.21.15.67:0/2611870079 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5580d7e9c680 con 0x5580d8c6e800 2024-03-24T00:49:23.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.484+0000 7ff875c3a740 1 -- 172.21.15.67:0/2611870079 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5580d7ea32c0 con 0x5580d8c6e800 2024-03-24T00:49:23.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.484+0000 7ff875c3a740 1 -- 172.21.15.67:0/2611870079 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x5580d7ea2b40 con 0x5580d8c6e800 2024-03-24T00:49:23.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.484+0000 7ff875c3a740 4 waiting for MDS map... 2024-03-24T00:49:23.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.484+0000 7ff871551640 1 -- 172.21.15.67:0/2611870079 <== mon.0 v2:172.21.15.67:3300/0 4 ==== osd_map(122..122 src has 1..122) v4 ==== 6273+0+0 (secure 0 0 0) 0x5580d8d52400 con 0x5580d8c6e800 2024-03-24T00:49:23.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.484+0000 7ff871551640 1 -- 172.21.15.67:0/2611870079 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 93) v1 ==== 2735+0+0 (secure 0 0 0) 0x5580d7ecf180 con 0x5580d8c6e800 2024-03-24T00:49:23.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.484+0000 7ff875c3a740 4 Got MDS map 93 2024-03-24T00:49:23.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.484+0000 7ff875c3a740 4 datascan.main: Using metadata pool output 2024-03-24T00:49:23.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.484+0000 7ff875c3a740 4 datascan.main: connecting to RADOS... 2024-03-24T00:49:23.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.485+0000 7ff875c3a740 1 Processor -- start 2024-03-24T00:49:23.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.485+0000 7ff875c3a740 1 -- start start 2024-03-24T00:49:23.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.485+0000 7ff875c3a740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5580d8d54c00 0x5580d7f20b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.485+0000 7ff875c3a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580d8d55000 0x5580d7f22100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.485+0000 7ff875c3a740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5580d8d55400 0x5580d7f22680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.485+0000 7ff875c3a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580d7ed6180 con 0x5580d8d55000 2024-03-24T00:49:23.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.485+0000 7ff875c3a740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580d8c74f00 con 0x5580d8d54c00 2024-03-24T00:49:23.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.485+0000 7ff875c3a740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5580d8c75080 con 0x5580d8d55400 2024-03-24T00:49:23.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.486+0000 7ff873d56640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580d8d55000 0x5580d7f22100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.486+0000 7ff873d56640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580d8d55000 0x5580d7f22100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38402/0 (socket says 172.21.15.67:38402) 2024-03-24T00:49:23.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.486+0000 7ff873d56640 1 -- 172.21.15.67:0/564151149 learned_addr learned my addr 172.21.15.67:0/564151149 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:23.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.486+0000 7ff873555640 1 --2- 172.21.15.67:0/564151149 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5580d8d55400 0x5580d7f22680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.486+0000 7ff873555640 1 --2- 172.21.15.67:0/564151149 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5580d8d54c00 0x5580d7f20b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.486+0000 7ff873d56640 1 -- 172.21.15.67:0/564151149 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5580d8d55400 msgr2=0x5580d7f22680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:23.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.486+0000 7ff873d56640 1 --2- 172.21.15.67:0/564151149 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5580d8d55400 0x5580d7f22680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.486+0000 7ff873d56640 1 -- 172.21.15.67:0/564151149 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5580d8d54c00 msgr2=0x5580d7f20b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:23.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.486+0000 7ff873d56640 1 --2- 172.21.15.67:0/564151149 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5580d8d54c00 0x5580d7f20b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.486+0000 7ff873d56640 1 -- 172.21.15.67:0/564151149 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5580d7ea23c0 con 0x5580d8d55000 2024-03-24T00:49:23.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.486+0000 7ff873555640 1 --2- 172.21.15.67:0/564151149 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5580d8d54c00 0x5580d7f20b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:49:23.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.486+0000 7ff873d56640 1 --2- 172.21.15.67:0/564151149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580d8d55000 0x5580d7f22100 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x5580d8d68c60 tx=0x5580d8d6d800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfd776b7560b9d3a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:23.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.486+0000 7ff86f54d640 1 -- 172.21.15.67:0/564151149 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5580d7e9cd00 con 0x5580d8d55000 2024-03-24T00:49:23.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.486+0000 7ff86f54d640 1 -- 172.21.15.67:0/564151149 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5580d7ec0c40 con 0x5580d8d55000 2024-03-24T00:49:23.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.486+0000 7ff86f54d640 1 -- 172.21.15.67:0/564151149 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5580d7e9c680 con 0x5580d8d55000 2024-03-24T00:49:23.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.487+0000 7ff875c3a740 1 -- 172.21.15.67:0/564151149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580d8d55000 msgr2=0x5580d7f22100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.487+0000 7ff875c3a740 1 --2- 172.21.15.67:0/564151149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580d8d55000 0x5580d7f22100 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x5580d8d68c60 tx=0x5580d8d6d800 comp rx=0 tx=0).stop 2024-03-24T00:49:23.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.487+0000 7ff875c3a740 1 -- 172.21.15.67:0/564151149 shutdown_connections 2024-03-24T00:49:23.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.487+0000 7ff875c3a740 1 --2- 172.21.15.67:0/564151149 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5580d8d55400 0x5580d7f22680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.487+0000 7ff875c3a740 1 --2- 172.21.15.67:0/564151149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580d8d55000 0x5580d7f22100 unknown :-1 s=CLOSED pgs=1838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.487+0000 7ff875c3a740 1 --2- 172.21.15.67:0/564151149 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5580d8d54c00 0x5580d7f20b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.487+0000 7ff875c3a740 1 -- 172.21.15.67:0/564151149 >> 172.21.15.67:0/564151149 conn(0x5580d8c6e400 msgr2=0x5580d8bff800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:23.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.487+0000 7ff875c3a740 1 -- 172.21.15.67:0/564151149 shutdown_connections 2024-03-24T00:49:23.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.487+0000 7ff875c3a740 1 -- 172.21.15.67:0/564151149 wait complete. 2024-03-24T00:49:23.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.487+0000 7ff875c3a740 1 Processor -- start 2024-03-24T00:49:23.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.487+0000 7ff875c3a740 1 -- start start 2024-03-24T00:49:23.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.488+0000 7ff875c3a740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5580d8d54c00 0x5580d7f20b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.488+0000 7ff875c3a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580d8d55000 0x5580d7f22100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.488+0000 7ff875c3a740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5580d8d55400 0x5580d7f22680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.488+0000 7ff875c3a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580d8c75380 con 0x5580d8d55000 2024-03-24T00:49:23.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.488+0000 7ff875c3a740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580d8c75200 con 0x5580d8d55400 2024-03-24T00:49:23.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.488+0000 7ff875c3a740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5580d8c75500 con 0x5580d8d54c00 2024-03-24T00:49:23.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.488+0000 7ff873d56640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580d8d55000 0x5580d7f22100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.488+0000 7ff873d56640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580d8d55000 0x5580d7f22100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38410/0 (socket says 172.21.15.67:38410) 2024-03-24T00:49:23.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.488+0000 7ff873d56640 1 -- 172.21.15.67:0/1898319739 learned_addr learned my addr 172.21.15.67:0/1898319739 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:23.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.488+0000 7ff873555640 1 --2- 172.21.15.67:0/1898319739 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5580d8d55400 0x5580d7f22680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.488+0000 7ff873555640 1 --2- 172.21.15.67:0/1898319739 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5580d8d54c00 0x5580d7f20b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.488+0000 7ff873d56640 1 -- 172.21.15.67:0/1898319739 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5580d8d54c00 msgr2=0x5580d7f20b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.488+0000 7ff873d56640 1 --2- 172.21.15.67:0/1898319739 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5580d8d54c00 0x5580d7f20b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.488+0000 7ff873d56640 1 -- 172.21.15.67:0/1898319739 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5580d8d55400 msgr2=0x5580d7f22680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.488+0000 7ff873d56640 1 --2- 172.21.15.67:0/1898319739 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5580d8d55400 0x5580d7f22680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.488+0000 7ff873d56640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5580d7ea3860 con 0x5580d8d55000 2024-03-24T00:49:23.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.488+0000 7ff873d56640 1 --2- 172.21.15.67:0/1898319739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580d8d55000 0x5580d7f22100 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x5580d8d69920 tx=0x5580d8c12ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:23.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.488+0000 7ff86dd4a640 1 -- 172.21.15.67:0/1898319739 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5580d7e9c820 con 0x5580d8d55000 2024-03-24T00:49:23.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.488+0000 7ff86dd4a640 1 -- 172.21.15.67:0/1898319739 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5580d7ec0a80 con 0x5580d8d55000 2024-03-24T00:49:23.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.488+0000 7ff875c3a740 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5580d7ea30e0 con 0x5580d8d55000 2024-03-24T00:49:23.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.488+0000 7ff86dd4a640 1 -- 172.21.15.67:0/1898319739 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5580d7e9cd00 con 0x5580d8d55000 2024-03-24T00:49:23.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.489+0000 7ff875c3a740 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5580d7ea3680 con 0x5580d8d55000 2024-03-24T00:49:23.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.489+0000 7ff875c3a740 4 datascan.init: resolving metadata pool 17 2024-03-24T00:49:23.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.490+0000 7ff86dd4a640 1 -- 172.21.15.67:0/1898319739 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x5580d8dc4c00 con 0x5580d8d55000 2024-03-24T00:49:23.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.490+0000 7ff86dd4a640 1 --2- 172.21.15.67:0/1898319739 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x5580d8dc6c00 0x5580d7f23700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.490+0000 7ff86dd4a640 1 -- 172.21.15.67:0/1898319739 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6273+0+0 (secure 0 0 0) 0x5580d8dc2a00 con 0x5580d8d55000 2024-03-24T00:49:23.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.490+0000 7ff875c3a740 4 datascan.init: found metadata pool 'cephfs_metadata' 2024-03-24T00:49:23.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.490+0000 7ff875c3a740 4 datascan.main: data pool 'cephfs_data' has ID 18 2024-03-24T00:49:23.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.490+0000 7ff875c3a740 4 datascan.main: opening data pool 'cephfs_data' 2024-03-24T00:49:23.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.490+0000 7ff875c3a740 1 --2- 172.21.15.67:0/1898319739 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x5580d8d55800 0x5580d7f22c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.490+0000 7ff875c3a740 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:1 18.0 18:00000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8d55c00 con 0x5580d8d55800 2024-03-24T00:49:23.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.490+0000 7ff873555640 1 --2- 172.21.15.67:0/1898319739 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x5580d8dc6c00 0x5580d7f23700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.490+0000 7ff873555640 1 --2- 172.21.15.67:0/1898319739 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x5580d8d55800 0x5580d7f22c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.490+0000 7ff873555640 1 --2- 172.21.15.67:0/1898319739 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x5580d8d55800 0x5580d7f22c00 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:23.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.491+0000 7ff873555640 1 --2- 172.21.15.67:0/1898319739 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x5580d8dc6c00 0x5580d7f23700 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x5580d8ca5500 tx=0x5580d8c168a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:23.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.491+0000 7ff873555640 1 -- 172.21.15.67:0/1898319739 <== osd.0 v2:172.21.15.67:6808/2628130519 1 ==== osd_op_reply(1 [pgnls start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee000 con 0x5580d8d55800 2024-03-24T00:49:23.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.491+0000 7ff86ed4c640 1 --2- 172.21.15.67:0/1898319739 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x5580d8dc7000 0x5580d820a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.491+0000 7ff86ed4c640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:2 18.20 18:04000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8dc7400 con 0x5580d8dc7000 2024-03-24T00:49:23.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.491+0000 7ff874557640 1 --2- 172.21.15.67:0/1898319739 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x5580d8dc7000 0x5580d820a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.492+0000 7ff874557640 1 --2- 172.21.15.67:0/1898319739 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x5580d8dc7000 0x5580d820a000 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:23.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.492+0000 7ff874557640 1 -- 172.21.15.67:0/1898319739 <== osd.5 v2:172.21.15.72:6800/288486206 1 ==== osd_op_reply(2 [pgnls start_epoch 122 out=49b] v120'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee500 con 0x5580d8dc7000 2024-03-24T00:49:23.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.492+0000 7ff86f54d640 1 --2- 172.21.15.67:0/1898319739 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x5580d8dc7c00 0x5580d820a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.492+0000 7ff86f54d640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:3 18.10 18:08000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8216000 con 0x5580d8dc7c00 2024-03-24T00:49:23.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.492+0000 7ff873d56640 1 --2- 172.21.15.67:0/1898319739 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x5580d8dc7c00 0x5580d820a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.492+0000 7ff873d56640 1 --2- 172.21.15.67:0/1898319739 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x5580d8dc7c00 0x5580d820a580 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:23.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.493+0000 7ff873d56640 1 -- 172.21.15.67:0/1898319739 <== osd.6 v2:172.21.15.72:6816/1725206562 1 ==== osd_op_reply(3 [pgnls start_epoch 122 out=49b] v120'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee780 con 0x5580d8dc7c00 2024-03-24T00:49:23.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.493+0000 7ff86ed4c640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:4 18.30 18:0c000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8dc7800 con 0x5580d8d55800 2024-03-24T00:49:23.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.493+0000 7ff873555640 1 -- 172.21.15.67:0/1898319739 <== osd.0 v2:172.21.15.67:6808/2628130519 2 ==== osd_op_reply(4 [pgnls start_epoch 122 out=49b] v120'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee000 con 0x5580d8d55800 2024-03-24T00:49:23.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.493+0000 7ff86f54d640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:5 18.8 18:10000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8216400 con 0x5580d8d55800 2024-03-24T00:49:23.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.493+0000 7ff873555640 1 -- 172.21.15.67:0/1898319739 <== osd.0 v2:172.21.15.67:6808/2628130519 3 ==== osd_op_reply(5 [pgnls start_epoch 122 out=49b] v120'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee000 con 0x5580d8d55800 2024-03-24T00:49:23.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.493+0000 7ff86ed4c640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:6 18.28 18:14000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8216800 con 0x5580d8dc7000 2024-03-24T00:49:23.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.494+0000 7ff874557640 1 -- 172.21.15.67:0/1898319739 <== osd.5 v2:172.21.15.72:6800/288486206 2 ==== osd_op_reply(6 [pgnls start_epoch 122 out=49b] v120'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee500 con 0x5580d8dc7000 2024-03-24T00:49:23.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.494+0000 7ff86f54d640 1 --2- 172.21.15.67:0/1898319739 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x5580d8217400 0x5580d820ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.494+0000 7ff86f54d640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:7 18.18 18:18000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8217800 con 0x5580d8217400 2024-03-24T00:49:23.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.494+0000 7ff873555640 1 --2- 172.21.15.67:0/1898319739 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x5580d8217400 0x5580d820ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.494+0000 7ff873555640 1 --2- 172.21.15.67:0/1898319739 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x5580d8217400 0x5580d820ab00 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:23.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.494+0000 7ff873555640 1 -- 172.21.15.67:0/1898319739 <== osd.1 v2:172.21.15.67:6800/4062510404 1 ==== osd_op_reply(7 [pgnls start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee000 con 0x5580d8217400 2024-03-24T00:49:23.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.494+0000 7ff86ed4c640 1 --2- 172.21.15.67:0/1898319739 >> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] conn(0x5580d8216c00 0x5580d820b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.494+0000 7ff86ed4c640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:8 18.38 18:1c000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8217000 con 0x5580d8216c00 2024-03-24T00:49:23.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.494+0000 7ff874557640 1 --2- 172.21.15.67:0/1898319739 >> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] conn(0x5580d8216c00 0x5580d820b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.495+0000 7ff874557640 1 --2- 172.21.15.67:0/1898319739 >> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] conn(0x5580d8216c00 0x5580d820b600 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:23.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.495+0000 7ff874557640 1 -- 172.21.15.67:0/1898319739 <== osd.3 v2:172.21.15.67:6816/3303489481 1 ==== osd_op_reply(8 [pgnls start_epoch 122 out=49b] v120'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee500 con 0x5580d8216c00 2024-03-24T00:49:23.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.495+0000 7ff86f54d640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:9 18.4 18:20000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8217c00 con 0x5580d8d55800 2024-03-24T00:49:23.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.495+0000 7ff873555640 1 -- 172.21.15.67:0/1898319739 <== osd.0 v2:172.21.15.67:6808/2628130519 4 ==== osd_op_reply(9 [pgnls start_epoch 122 out=49b] v120'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee000 con 0x5580d8d55800 2024-03-24T00:49:23.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.495+0000 7ff86ed4c640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:10 18.24 18:24000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8216400 con 0x5580d8d55800 2024-03-24T00:49:23.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.495+0000 7ff873555640 1 -- 172.21.15.67:0/1898319739 <== osd.0 v2:172.21.15.67:6808/2628130519 5 ==== osd_op_reply(10 [pgnls start_epoch 122 out=49b] v120'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee000 con 0x5580d8d55800 2024-03-24T00:49:23.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.495+0000 7ff86f54d640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:11 18.14 18:28000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8217800 con 0x5580d8dc7c00 2024-03-24T00:49:23.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.496+0000 7ff873d56640 1 -- 172.21.15.67:0/1898319739 <== osd.6 v2:172.21.15.72:6816/1725206562 2 ==== osd_op_reply(11 [pgnls start_epoch 122 out=49b] v120'22 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee780 con 0x5580d8dc7c00 2024-03-24T00:49:23.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.496+0000 7ff86ed4c640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:12 18.34 18:2c000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8dc7800 con 0x5580d8d55800 2024-03-24T00:49:23.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.496+0000 7ff873555640 1 -- 172.21.15.67:0/1898319739 <== osd.0 v2:172.21.15.67:6808/2628130519 6 ==== osd_op_reply(12 [pgnls start_epoch 122 out=49b] v120'14 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee000 con 0x5580d8d55800 2024-03-24T00:49:23.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.496+0000 7ff86f54d640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:13 18.c 18:30000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8228000 con 0x5580d8dc7000 2024-03-24T00:49:23.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.496+0000 7ff874557640 1 -- 172.21.15.67:0/1898319739 <== osd.5 v2:172.21.15.72:6800/288486206 3 ==== osd_op_reply(13 [pgnls start_epoch 122 out=49b] v120'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee500 con 0x5580d8dc7000 2024-03-24T00:49:23.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.496+0000 7ff86ed4c640 1 --2- 172.21.15.67:0/1898319739 >> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] conn(0x5580d8d55c00 0x5580d820bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.496+0000 7ff86ed4c640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] -- osd_op(unknown.0.0:14 18.2c 18:34000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8c6fc00 con 0x5580d8d55c00 2024-03-24T00:49:23.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.497+0000 7ff873d56640 1 --2- 172.21.15.67:0/1898319739 >> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] conn(0x5580d8d55c00 0x5580d820bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.497+0000 7ff873d56640 1 --2- 172.21.15.67:0/1898319739 >> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] conn(0x5580d8d55c00 0x5580d820bb80 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:23.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.497+0000 7ff873d56640 1 -- 172.21.15.67:0/1898319739 <== osd.2 v2:172.21.15.67:6817/1268432561 1 ==== osd_op_reply(14 [pgnls start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee780 con 0x5580d8d55c00 2024-03-24T00:49:23.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.497+0000 7ff86f54d640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:15 18.1c 18:38000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8228400 con 0x5580d8217400 2024-03-24T00:49:23.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.497+0000 7ff873555640 1 -- 172.21.15.67:0/1898319739 <== osd.1 v2:172.21.15.67:6800/4062510404 2 ==== osd_op_reply(15 [pgnls start_epoch 122 out=49b] v120'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee000 con 0x5580d8217400 2024-03-24T00:49:23.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.497+0000 7ff86ed4c640 1 --2- 172.21.15.67:0/1898319739 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x5580d8228000 0x5580d820c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.497+0000 7ff86ed4c640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:16 18.3c 18:3c000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8229400 con 0x5580d8228000 2024-03-24T00:49:23.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.498+0000 7ff873555640 1 --2- 172.21.15.67:0/1898319739 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x5580d8228000 0x5580d820c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.498+0000 7ff873555640 1 --2- 172.21.15.67:0/1898319739 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x5580d8228000 0x5580d820c100 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:23.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.498+0000 7ff873555640 1 -- 172.21.15.67:0/1898319739 <== osd.7 v2:172.21.15.72:6824/2893038746 1 ==== osd_op_reply(16 [pgnls start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee000 con 0x5580d8228000 2024-03-24T00:49:23.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.498+0000 7ff86f54d640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:17 18.2 18:40000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8228800 con 0x5580d8228000 2024-03-24T00:49:23.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.498+0000 7ff873555640 1 -- 172.21.15.67:0/1898319739 <== osd.7 v2:172.21.15.72:6824/2893038746 2 ==== osd_op_reply(17 [pgnls start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee000 con 0x5580d8228000 2024-03-24T00:49:23.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.498+0000 7ff86ed4c640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:18 18.22 18:44000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8216800 con 0x5580d8216c00 2024-03-24T00:49:23.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.499+0000 7ff874557640 1 -- 172.21.15.67:0/1898319739 <== osd.3 v2:172.21.15.67:6816/3303489481 2 ==== osd_op_reply(18 [pgnls start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee500 con 0x5580d8216c00 2024-03-24T00:49:23.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.499+0000 7ff86f54d640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] -- osd_op(unknown.0.0:19 18.12 18:48000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8228c00 con 0x5580d8d55c00 2024-03-24T00:49:23.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.499+0000 7ff873d56640 1 -- 172.21.15.67:0/1898319739 <== osd.2 v2:172.21.15.67:6817/1268432561 2 ==== osd_op_reply(19 [pgnls start_epoch 122 out=81b] v121'22 uv22 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x5580d7fee780 con 0x5580d8d55c00 2024-03-24T00:49:23.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.499+0000 7ff875c3a740 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] -- osd_op(unknown.0.0:20 18.12 18:4bec3886:::10000000002.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8dc7800 con 0x5580d8d55c00 2024-03-24T00:49:23.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.499+0000 7ff873d56640 1 -- 172.21.15.67:0/1898319739 <== osd.2 v2:172.21.15.67:6817/1268432561 3 ==== osd_op_reply(20 10000000002.00000000 [stat out=16b] v0'0 uv22 ondisk = 0) v8 ==== 164+0+16 (crc 0 0 0) 0x5580d7fee780 con 0x5580d8d55c00 2024-03-24T00:49:23.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.499+0000 7ff875c3a740 10 datascan.operator(): handling object 1099511627778.0 2024-03-24T00:49:23.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.499+0000 7ff875c3a740 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] -- osd_op(unknown.0.0:21 18.12 18:4bec3886:::10000000002.00000000:head [call cephfs.accumulate_inode_metadata in=112b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8216000 con 0x5580d8d55c00 2024-03-24T00:49:23.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.501+0000 7ff873d56640 1 -- 172.21.15.67:0/1898319739 <== osd.2 v2:172.21.15.67:6817/1268432561 4 ==== osd_op_reply(21 10000000002.00000000 [call] v122'23 uv23 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x5580d7fee780 con 0x5580d8d55c00 2024-03-24T00:49:23.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.501+0000 7ff875c3a740 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:22 18.32 18:4c000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8217800 con 0x5580d8216c00 2024-03-24T00:49:23.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.501+0000 7ff874557640 1 -- 172.21.15.67:0/1898319739 <== osd.3 v2:172.21.15.67:6816/3303489481 3 ==== osd_op_reply(22 [pgnls start_epoch 122 out=49b] v120'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee500 con 0x5580d8216c00 2024-03-24T00:49:23.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.501+0000 7ff86f54d640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:23 18.a 18:50000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8229000 con 0x5580d8216c00 2024-03-24T00:49:23.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.501+0000 7ff874557640 1 -- 172.21.15.67:0/1898319739 <== osd.3 v2:172.21.15.67:6816/3303489481 4 ==== osd_op_reply(23 [pgnls start_epoch 122 out=49b] v120'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee500 con 0x5580d8216c00 2024-03-24T00:49:23.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.501+0000 7ff86ed4c640 1 --2- 172.21.15.67:0/1898319739 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x5580d8217000 0x5580d820c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.501+0000 7ff86ed4c640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:24 18.2a 18:54000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8dc7400 con 0x5580d8217000 2024-03-24T00:49:23.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.502+0000 7ff874557640 1 --2- 172.21.15.67:0/1898319739 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x5580d8217000 0x5580d820c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.502+0000 7ff874557640 1 --2- 172.21.15.67:0/1898319739 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x5580d8217000 0x5580d820c680 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:23.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.502+0000 7ff874557640 1 -- 172.21.15.67:0/1898319739 <== osd.4 v2:172.21.15.72:6808/477220793 1 ==== osd_op_reply(24 [pgnls start_epoch 122 out=49b] v120'20 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee500 con 0x5580d8217000 2024-03-24T00:49:23.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.502+0000 7ff86f54d640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:25 18.1a 18:58000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8228c00 con 0x5580d8216c00 2024-03-24T00:49:23.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.502+0000 7ff874557640 1 -- 172.21.15.67:0/1898319739 <== osd.3 v2:172.21.15.67:6816/3303489481 5 ==== osd_op_reply(25 [pgnls start_epoch 122 out=49b] v120'14 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee500 con 0x5580d8216c00 2024-03-24T00:49:23.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.503+0000 7ff86ed4c640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:26 18.3a 18:5c000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8229000 con 0x5580d8d55800 2024-03-24T00:49:23.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.503+0000 7ff873555640 1 -- 172.21.15.67:0/1898319739 <== osd.0 v2:172.21.15.67:6808/2628130519 7 ==== osd_op_reply(26 [pgnls start_epoch 122 out=49b] v120'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee000 con 0x5580d8d55800 2024-03-24T00:49:23.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.503+0000 7ff86f54d640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:27 18.6 18:60000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8228400 con 0x5580d8228000 2024-03-24T00:49:23.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.503+0000 7ff873555640 1 -- 172.21.15.67:0/1898319739 <== osd.7 v2:172.21.15.72:6824/2893038746 3 ==== osd_op_reply(27 [pgnls start_epoch 122 out=49b] v120'14 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee000 con 0x5580d8228000 2024-03-24T00:49:23.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.503+0000 7ff86ed4c640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] -- osd_op(unknown.0.0:28 18.26 18:64000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8228c00 con 0x5580d8d55c00 2024-03-24T00:49:23.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.503+0000 7ff873d56640 1 -- 172.21.15.67:0/1898319739 <== osd.2 v2:172.21.15.67:6817/1268432561 5 ==== osd_op_reply(28 [pgnls start_epoch 122 out=49b] v120'30 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee780 con 0x5580d8d55c00 2024-03-24T00:49:23.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.503+0000 7ff86f54d640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:29 18.16 18:68000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8229400 con 0x5580d8216c00 2024-03-24T00:49:23.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.504+0000 7ff874557640 1 -- 172.21.15.67:0/1898319739 <== osd.3 v2:172.21.15.67:6816/3303489481 6 ==== osd_op_reply(29 [pgnls start_epoch 122 out=49b] v120'20 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee500 con 0x5580d8216c00 2024-03-24T00:49:23.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.504+0000 7ff86ed4c640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:30 18.36 18:6c000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8dc7400 con 0x5580d8217400 2024-03-24T00:49:23.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.504+0000 7ff873555640 1 -- 172.21.15.67:0/1898319739 <== osd.1 v2:172.21.15.67:6800/4062510404 3 ==== osd_op_reply(30 [pgnls start_epoch 122 out=81b] v120'33 uv17 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x5580d7fee000 con 0x5580d8217400 2024-03-24T00:49:23.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.504+0000 7ff875c3a740 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:31 18.36 18:6fe38dc2:::100000001f7.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8217c00 con 0x5580d8217400 2024-03-24T00:49:23.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.504+0000 7ff873555640 1 -- 172.21.15.67:0/1898319739 <== osd.1 v2:172.21.15.67:6800/4062510404 4 ==== osd_op_reply(31 100000001f7.00000000 [stat out=16b] v0'0 uv17 ondisk = 0) v8 ==== 164+0+16 (crc 0 0 0) 0x5580d7fee000 con 0x5580d8217400 2024-03-24T00:49:23.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.504+0000 7ff875c3a740 10 datascan.operator(): handling object 1099511628279.0 2024-03-24T00:49:23.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.504+0000 7ff875c3a740 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:32 18.36 18:6fe38dc2:::100000001f7.00000000:head [call cephfs.accumulate_inode_metadata in=112b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8216400 con 0x5580d8217400 2024-03-24T00:49:23.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.505+0000 7ff873555640 1 -- 172.21.15.67:0/1898319739 <== osd.1 v2:172.21.15.67:6800/4062510404 5 ==== osd_op_reply(32 100000001f7.00000000 [call] v122'34 uv34 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x5580d7fee000 con 0x5580d8217400 2024-03-24T00:49:23.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.505+0000 7ff875c3a740 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:33 18.e 18:70000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8c6fc00 con 0x5580d8d55800 2024-03-24T00:49:23.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.506+0000 7ff873555640 1 -- 172.21.15.67:0/1898319739 <== osd.0 v2:172.21.15.67:6808/2628130519 8 ==== osd_op_reply(33 [pgnls start_epoch 122 out=49b] v120'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee000 con 0x5580d8d55800 2024-03-24T00:49:23.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.506+0000 7ff86ed4c640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:34 18.2e 18:74000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8216800 con 0x5580d8216c00 2024-03-24T00:49:23.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.506+0000 7ff874557640 1 -- 172.21.15.67:0/1898319739 <== osd.3 v2:172.21.15.67:6816/3303489481 7 ==== osd_op_reply(34 [pgnls start_epoch 122 out=49b] v120'20 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee500 con 0x5580d8216c00 2024-03-24T00:49:23.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.506+0000 7ff86f54d640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:35 18.1e 18:78000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8228800 con 0x5580d8dc7000 2024-03-24T00:49:23.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.506+0000 7ff874557640 1 -- 172.21.15.67:0/1898319739 <== osd.5 v2:172.21.15.72:6800/288486206 4 ==== osd_op_reply(35 [pgnls start_epoch 122 out=49b] v120'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee500 con 0x5580d8dc7000 2024-03-24T00:49:23.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.506+0000 7ff86ed4c640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:36 18.3e 18:7c000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8217800 con 0x5580d8216c00 2024-03-24T00:49:23.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.506+0000 7ff874557640 1 -- 172.21.15.67:0/1898319739 <== osd.3 v2:172.21.15.67:6816/3303489481 8 ==== osd_op_reply(36 [pgnls start_epoch 122 out=49b] v120'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee500 con 0x5580d8216c00 2024-03-24T00:49:23.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.506+0000 7ff86f54d640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:37 18.1 18:80000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8229800 con 0x5580d8216c00 2024-03-24T00:49:23.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.507+0000 7ff874557640 1 -- 172.21.15.67:0/1898319739 <== osd.3 v2:172.21.15.67:6816/3303489481 9 ==== osd_op_reply(37 [pgnls start_epoch 122 out=49b] v120'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee500 con 0x5580d8216c00 2024-03-24T00:49:23.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.507+0000 7ff86ed4c640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:38 18.21 18:84000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d823a000 con 0x5580d8dc7000 2024-03-24T00:49:23.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.507+0000 7ff874557640 1 -- 172.21.15.67:0/1898319739 <== osd.5 v2:172.21.15.72:6800/288486206 5 ==== osd_op_reply(38 [pgnls start_epoch 122 out=49b] v120'24 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee500 con 0x5580d8dc7000 2024-03-24T00:49:23.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.507+0000 7ff86f54d640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:39 18.11 18:88000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8229c00 con 0x5580d8217400 2024-03-24T00:49:23.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.507+0000 7ff873555640 1 -- 172.21.15.67:0/1898319739 <== osd.1 v2:172.21.15.67:6800/4062510404 6 ==== osd_op_reply(39 [pgnls start_epoch 122 out=49b] v120'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee000 con 0x5580d8217400 2024-03-24T00:49:23.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.507+0000 7ff86ed4c640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:40 18.31 18:8c000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d823a400 con 0x5580d8217000 2024-03-24T00:49:23.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.508+0000 7ff874557640 1 -- 172.21.15.67:0/1898319739 <== osd.4 v2:172.21.15.72:6808/477220793 2 ==== osd_op_reply(40 [pgnls start_epoch 122 out=81b] v120'15 uv8 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x5580d7fee500 con 0x5580d8217000 2024-03-24T00:49:23.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.508+0000 7ff875c3a740 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:41 18.31 18:8de29a03:::10000000004.00000001:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8229000 con 0x5580d8217000 2024-03-24T00:49:23.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.508+0000 7ff874557640 1 -- 172.21.15.67:0/1898319739 <== osd.4 v2:172.21.15.72:6808/477220793 3 ==== osd_op_reply(41 10000000004.00000001 [stat out=16b] v0'0 uv8 ondisk = 0) v8 ==== 164+0+16 (crc 0 0 0) 0x5580d7fee500 con 0x5580d8217000 2024-03-24T00:49:23.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.508+0000 7ff875c3a740 10 datascan.operator(): handling object 1099511627780.1 2024-03-24T00:49:23.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.508+0000 7ff875c3a740 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:42 18.5 18:a367a3e8:::10000000004.00000000:head [call cephfs.accumulate_inode_metadata in=112b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8217c00 con 0x5580d8217400 2024-03-24T00:49:23.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.509+0000 7ff873555640 1 -- 172.21.15.67:0/1898319739 <== osd.1 v2:172.21.15.67:6800/4062510404 7 ==== osd_op_reply(42 10000000004.00000000 [call] v122'14 uv14 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x5580d7fee000 con 0x5580d8217400 2024-03-24T00:49:23.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.509+0000 7ff875c3a740 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:43 18.9 18:90000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8216400 con 0x5580d8216c00 2024-03-24T00:49:23.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.510+0000 7ff874557640 1 -- 172.21.15.67:0/1898319739 <== osd.3 v2:172.21.15.67:6816/3303489481 10 ==== osd_op_reply(43 [pgnls start_epoch 122 out=49b] v120'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee500 con 0x5580d8216c00 2024-03-24T00:49:23.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.510+0000 7ff86ed4c640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:44 18.29 18:94000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d823a800 con 0x5580d8228000 2024-03-24T00:49:23.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.510+0000 7ff873555640 1 -- 172.21.15.67:0/1898319739 <== osd.7 v2:172.21.15.72:6824/2893038746 4 ==== osd_op_reply(44 [pgnls start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee000 con 0x5580d8228000 2024-03-24T00:49:23.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.510+0000 7ff86f54d640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:45 18.19 18:98000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d823a000 con 0x5580d8217000 2024-03-24T00:49:23.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.510+0000 7ff874557640 1 -- 172.21.15.67:0/1898319739 <== osd.4 v2:172.21.15.72:6808/477220793 4 ==== osd_op_reply(45 [pgnls start_epoch 122 out=81b] v121'23 uv23 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x5580d7fee500 con 0x5580d8217000 2024-03-24T00:49:23.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.511+0000 7ff875c3a740 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:46 18.19 18:997b7129:::100000001f8.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8216800 con 0x5580d8217000 2024-03-24T00:49:23.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.511+0000 7ff874557640 1 -- 172.21.15.67:0/1898319739 <== osd.4 v2:172.21.15.72:6808/477220793 5 ==== osd_op_reply(46 100000001f8.00000000 [stat out=16b] v0'0 uv23 ondisk = 0) v8 ==== 164+0+16 (crc 0 0 0) 0x5580d7fee500 con 0x5580d8217000 2024-03-24T00:49:23.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.511+0000 7ff875c3a740 10 datascan.operator(): handling object 1099511628280.0 2024-03-24T00:49:23.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.511+0000 7ff875c3a740 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:47 18.19 18:997b7129:::100000001f8.00000000:head [call cephfs.accumulate_inode_metadata in=112b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8217800 con 0x5580d8217000 2024-03-24T00:49:23.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.512+0000 7ff874557640 1 -- 172.21.15.67:0/1898319739 <== osd.4 v2:172.21.15.72:6808/477220793 6 ==== osd_op_reply(47 100000001f8.00000000 [call] v122'24 uv24 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x5580d7fee500 con 0x5580d8217000 2024-03-24T00:49:23.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.512+0000 7ff875c3a740 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:48 18.39 18:9c000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8c6fc00 con 0x5580d8d55800 2024-03-24T00:49:23.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.512+0000 7ff873555640 1 -- 172.21.15.67:0/1898319739 <== osd.0 v2:172.21.15.67:6808/2628130519 9 ==== osd_op_reply(48 [pgnls start_epoch 122 out=49b] v120'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee000 con 0x5580d8d55800 2024-03-24T00:49:23.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.512+0000 7ff86f54d640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:49 18.5 18:a0000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d823bc00 con 0x5580d8217400 2024-03-24T00:49:23.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.513+0000 7ff873555640 1 -- 172.21.15.67:0/1898319739 <== osd.1 v2:172.21.15.67:6800/4062510404 8 ==== osd_op_reply(49 [pgnls start_epoch 122 out=81b] v122'14 uv14 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x5580d7fee000 con 0x5580d8217400 2024-03-24T00:49:23.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.513+0000 7ff875c3a740 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:50 18.5 18:a367a3e8:::10000000004.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8dc7400 con 0x5580d8217400 2024-03-24T00:49:23.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.513+0000 7ff873555640 1 -- 172.21.15.67:0/1898319739 <== osd.1 v2:172.21.15.67:6800/4062510404 9 ==== osd_op_reply(50 10000000004.00000000 [stat out=16b] v0'0 uv14 ondisk = 0) v8 ==== 164+0+16 (crc 0 0 0) 0x5580d7fee000 con 0x5580d8217400 2024-03-24T00:49:23.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.513+0000 7ff875c3a740 10 datascan.operator(): handling object 1099511627780.0 2024-03-24T00:49:23.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.513+0000 7ff875c3a740 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:51 18.5 18:a367a3e8:::10000000004.00000000:head [call cephfs.accumulate_inode_metadata in=112b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8240000 con 0x5580d8217400 2024-03-24T00:49:23.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.514+0000 7ff873555640 1 -- 172.21.15.67:0/1898319739 <== osd.1 v2:172.21.15.67:6800/4062510404 10 ==== osd_op_reply(51 10000000004.00000000 [call] v122'15 uv15 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x5580d7fee000 con 0x5580d8217400 2024-03-24T00:49:23.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.514+0000 7ff875c3a740 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:52 18.25 18:a4000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8240400 con 0x5580d8dc7c00 2024-03-24T00:49:23.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.514+0000 7ff873d56640 1 -- 172.21.15.67:0/1898319739 <== osd.6 v2:172.21.15.72:6816/1725206562 3 ==== osd_op_reply(52 [pgnls start_epoch 122 out=49b] v120'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee780 con 0x5580d8dc7c00 2024-03-24T00:49:23.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.514+0000 7ff86f54d640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:53 18.15 18:a8000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8228400 con 0x5580d8dc7000 2024-03-24T00:49:23.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.515+0000 7ff874557640 1 -- 172.21.15.67:0/1898319739 <== osd.5 v2:172.21.15.72:6800/288486206 6 ==== osd_op_reply(53 [pgnls start_epoch 122 out=49b] v120'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee500 con 0x5580d8dc7000 2024-03-24T00:49:23.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.515+0000 7ff86ed4c640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:54 18.35 18:ac000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d823ac00 con 0x5580d8217000 2024-03-24T00:49:23.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.515+0000 7ff874557640 1 -- 172.21.15.67:0/1898319739 <== osd.4 v2:172.21.15.72:6808/477220793 7 ==== osd_op_reply(54 [pgnls start_epoch 122 out=49b] v120'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee500 con 0x5580d8217000 2024-03-24T00:49:23.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.515+0000 7ff86f54d640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:55 18.d 18:b0000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8229c00 con 0x5580d8217000 2024-03-24T00:49:23.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.515+0000 7ff874557640 1 -- 172.21.15.67:0/1898319739 <== osd.4 v2:172.21.15.72:6808/477220793 8 ==== osd_op_reply(55 [pgnls start_epoch 122 out=49b] v120'14 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee500 con 0x5580d8217000 2024-03-24T00:49:23.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.515+0000 7ff86ed4c640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:56 18.2d 18:b4000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d823b000 con 0x5580d8dc7c00 2024-03-24T00:49:23.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.516+0000 7ff873d56640 1 -- 172.21.15.67:0/1898319739 <== osd.6 v2:172.21.15.72:6816/1725206562 4 ==== osd_op_reply(56 [pgnls start_epoch 122 out=49b] v120'14 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee780 con 0x5580d8dc7c00 2024-03-24T00:49:23.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.516+0000 7ff86f54d640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:57 18.1d 18:b8000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8229400 con 0x5580d8d55800 2024-03-24T00:49:23.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.516+0000 7ff873555640 1 -- 172.21.15.67:0/1898319739 <== osd.0 v2:172.21.15.67:6808/2628130519 10 ==== osd_op_reply(57 [pgnls start_epoch 122 out=49b] v120'30 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee000 con 0x5580d8d55800 2024-03-24T00:49:23.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.516+0000 7ff86ed4c640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:58 18.3d 18:bc000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d823b400 con 0x5580d8216c00 2024-03-24T00:49:23.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.516+0000 7ff874557640 1 -- 172.21.15.67:0/1898319739 <== osd.3 v2:172.21.15.67:6816/3303489481 11 ==== osd_op_reply(58 [pgnls start_epoch 122 out=49b] v120'14 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee500 con 0x5580d8216c00 2024-03-24T00:49:23.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.516+0000 7ff86f54d640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:59 18.3 18:c0000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8228800 con 0x5580d8217400 2024-03-24T00:49:23.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.516+0000 7ff873555640 1 -- 172.21.15.67:0/1898319739 <== osd.1 v2:172.21.15.67:6800/4062510404 11 ==== osd_op_reply(59 [pgnls start_epoch 122 out=49b] v120'14 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee000 con 0x5580d8217400 2024-03-24T00:49:23.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.516+0000 7ff86ed4c640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] -- osd_op(unknown.0.0:60 18.23 18:c4000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d823b800 con 0x5580d8d55c00 2024-03-24T00:49:23.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.517+0000 7ff873d56640 1 -- 172.21.15.67:0/1898319739 <== osd.2 v2:172.21.15.67:6817/1268432561 6 ==== osd_op_reply(60 [pgnls start_epoch 122 out=49b] v120'14 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee780 con 0x5580d8d55c00 2024-03-24T00:49:23.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.517+0000 7ff86f54d640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:61 18.13 18:c8000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8229800 con 0x5580d8228000 2024-03-24T00:49:23.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.517+0000 7ff873555640 1 -- 172.21.15.67:0/1898319739 <== osd.7 v2:172.21.15.72:6824/2893038746 5 ==== osd_op_reply(61 [pgnls start_epoch 122 out=49b] v120'30 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee000 con 0x5580d8228000 2024-03-24T00:49:23.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.517+0000 7ff86ed4c640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:62 18.33 18:cc000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d823a800 con 0x5580d8217400 2024-03-24T00:49:23.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.518+0000 7ff873555640 1 -- 172.21.15.67:0/1898319739 <== osd.1 v2:172.21.15.67:6800/4062510404 12 ==== osd_op_reply(62 [pgnls start_epoch 122 out=49b] v120'20 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee000 con 0x5580d8217400 2024-03-24T00:49:23.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.518+0000 7ff86f54d640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:63 18.b 18:d0000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8229400 con 0x5580d8dc7000 2024-03-24T00:49:23.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.518+0000 7ff874557640 1 -- 172.21.15.67:0/1898319739 <== osd.5 v2:172.21.15.72:6800/288486206 7 ==== osd_op_reply(63 [pgnls start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee500 con 0x5580d8dc7000 2024-03-24T00:49:23.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.518+0000 7ff86ed4c640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:64 18.2b 18:d4000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d823bc00 con 0x5580d8228000 2024-03-24T00:49:23.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.518+0000 7ff873555640 1 -- 172.21.15.67:0/1898319739 <== osd.7 v2:172.21.15.72:6824/2893038746 6 ==== osd_op_reply(64 [pgnls start_epoch 122 out=49b] v120'22 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee000 con 0x5580d8228000 2024-03-24T00:49:23.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.518+0000 7ff86f54d640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:65 18.1b 18:d8000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8217c00 con 0x5580d8217400 2024-03-24T00:49:23.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.519+0000 7ff873555640 1 -- 172.21.15.67:0/1898319739 <== osd.1 v2:172.21.15.67:6800/4062510404 13 ==== osd_op_reply(65 [pgnls start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee000 con 0x5580d8217400 2024-03-24T00:49:23.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.519+0000 7ff86ed4c640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:66 18.3b 18:dc000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d823b000 con 0x5580d8217000 2024-03-24T00:49:23.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.519+0000 7ff874557640 1 -- 172.21.15.67:0/1898319739 <== osd.4 v2:172.21.15.72:6808/477220793 9 ==== osd_op_reply(66 [pgnls start_epoch 122 out=49b] v120'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee500 con 0x5580d8217000 2024-03-24T00:49:23.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.519+0000 7ff86f54d640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] -- osd_op(unknown.0.0:67 18.7 18:e0000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8216000 con 0x5580d8d55c00 2024-03-24T00:49:23.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.519+0000 7ff873d56640 1 -- 172.21.15.67:0/1898319739 <== osd.2 v2:172.21.15.67:6817/1268432561 7 ==== osd_op_reply(67 [pgnls start_epoch 122 out=49b] v120'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee780 con 0x5580d8d55c00 2024-03-24T00:49:23.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.519+0000 7ff86ed4c640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:68 18.27 18:e4000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8240000 con 0x5580d8dc7000 2024-03-24T00:49:23.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.520+0000 7ff874557640 1 -- 172.21.15.67:0/1898319739 <== osd.5 v2:172.21.15.72:6800/288486206 8 ==== osd_op_reply(68 [pgnls start_epoch 122 out=49b] v120'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee500 con 0x5580d8dc7000 2024-03-24T00:49:23.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.520+0000 7ff86f54d640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:69 18.17 18:e8000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8216400 con 0x5580d8dc7000 2024-03-24T00:49:23.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.520+0000 7ff874557640 1 -- 172.21.15.67:0/1898319739 <== osd.5 v2:172.21.15.72:6800/288486206 9 ==== osd_op_reply(69 [pgnls start_epoch 122 out=49b] v120'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee500 con 0x5580d8dc7000 2024-03-24T00:49:23.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.520+0000 7ff86ed4c640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:70 18.37 18:ec000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8240400 con 0x5580d8228000 2024-03-24T00:49:23.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.520+0000 7ff873555640 1 -- 172.21.15.67:0/1898319739 <== osd.7 v2:172.21.15.72:6824/2893038746 7 ==== osd_op_reply(70 [pgnls start_epoch 122 out=49b] v120'22 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee000 con 0x5580d8228000 2024-03-24T00:49:23.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.520+0000 7ff86f54d640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:71 18.f 18:f0000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8216800 con 0x5580d8dc7000 2024-03-24T00:49:23.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.521+0000 7ff874557640 1 -- 172.21.15.67:0/1898319739 <== osd.5 v2:172.21.15.72:6800/288486206 10 ==== osd_op_reply(71 [pgnls start_epoch 122 out=49b] v120'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee500 con 0x5580d8dc7000 2024-03-24T00:49:23.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.521+0000 7ff86ed4c640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:72 18.2f 18:f4000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8dc7400 con 0x5580d8217400 2024-03-24T00:49:23.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.521+0000 7ff873555640 1 -- 172.21.15.67:0/1898319739 <== osd.1 v2:172.21.15.67:6800/4062510404 14 ==== osd_op_reply(72 [pgnls start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee000 con 0x5580d8217400 2024-03-24T00:49:23.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.521+0000 7ff86f54d640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:73 18.1f 18:f8000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8217800 con 0x5580d8228000 2024-03-24T00:49:23.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.521+0000 7ff873555640 1 -- 172.21.15.67:0/1898319739 <== osd.7 v2:172.21.15.72:6824/2893038746 8 ==== osd_op_reply(73 [pgnls start_epoch 122 out=49b] v120'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee000 con 0x5580d8228000 2024-03-24T00:49:23.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.521+0000 7ff86ed4c640 1 -- 172.21.15.67:0/1898319739 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:74 18.3f 18:fc000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5580d8dc7800 con 0x5580d8217000 2024-03-24T00:49:23.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.522+0000 7ff874557640 1 -- 172.21.15.67:0/1898319739 <== osd.4 v2:172.21.15.72:6808/477220793 10 ==== osd_op_reply(74 [pgnls start_epoch 122 out=49b] v120'17 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5580d7fee500 con 0x5580d8217000 2024-03-24T00:49:23.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.522+0000 7ff875c3a740 1 -- 172.21.15.67:0/1898319739 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x5580d8d55800 msgr2=0x5580d7f22c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.522+0000 7ff875c3a740 1 --2- 172.21.15.67:0/1898319739 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x5580d8d55800 0x5580d7f22c00 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.522+0000 7ff875c3a740 1 -- 172.21.15.67:0/1898319739 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x5580d8217400 msgr2=0x5580d820ab00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.522+0000 7ff875c3a740 1 --2- 172.21.15.67:0/1898319739 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x5580d8217400 0x5580d820ab00 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.522+0000 7ff875c3a740 1 -- 172.21.15.67:0/1898319739 >> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] conn(0x5580d8d55c00 msgr2=0x5580d820bb80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.522+0000 7ff875c3a740 1 --2- 172.21.15.67:0/1898319739 >> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] conn(0x5580d8d55c00 0x5580d820bb80 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.522+0000 7ff875c3a740 1 -- 172.21.15.67:0/1898319739 >> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] conn(0x5580d8216c00 msgr2=0x5580d820b600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.522+0000 7ff875c3a740 1 --2- 172.21.15.67:0/1898319739 >> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] conn(0x5580d8216c00 0x5580d820b600 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.522+0000 7ff875c3a740 1 -- 172.21.15.67:0/1898319739 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x5580d8217000 msgr2=0x5580d820c680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.522+0000 7ff873d56640 1 -- 172.21.15.67:0/1898319739 reap_dead start 2024-03-24T00:49:23.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.522+0000 7ff875c3a740 1 --2- 172.21.15.67:0/1898319739 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x5580d8217000 0x5580d820c680 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.522+0000 7ff875c3a740 1 -- 172.21.15.67:0/1898319739 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x5580d8dc7000 msgr2=0x5580d820a000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.522+0000 7ff875c3a740 1 --2- 172.21.15.67:0/1898319739 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x5580d8dc7000 0x5580d820a000 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.522+0000 7ff875c3a740 1 -- 172.21.15.67:0/1898319739 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x5580d8dc7c00 msgr2=0x5580d820a580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.522+0000 7ff875c3a740 1 --2- 172.21.15.67:0/1898319739 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x5580d8dc7c00 0x5580d820a580 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.522+0000 7ff875c3a740 1 -- 172.21.15.67:0/1898319739 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x5580d8228000 msgr2=0x5580d820c100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.522+0000 7ff875c3a740 1 --2- 172.21.15.67:0/1898319739 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x5580d8228000 0x5580d820c100 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.522+0000 7ff875c3a740 1 -- 172.21.15.67:0/1898319739 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x5580d8dc6c00 msgr2=0x5580d7f23700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.522+0000 7ff873d56640 1 -- 172.21.15.67:0/1898319739 reap_dead start 2024-03-24T00:49:23.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.522+0000 7ff875c3a740 1 --2- 172.21.15.67:0/1898319739 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x5580d8dc6c00 0x5580d7f23700 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x5580d8ca5500 tx=0x5580d8c168a0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.522+0000 7ff875c3a740 1 -- 172.21.15.67:0/1898319739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580d8d55000 msgr2=0x5580d7f22100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.522+0000 7ff875c3a740 1 --2- 172.21.15.67:0/1898319739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580d8d55000 0x5580d7f22100 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x5580d8d69920 tx=0x5580d8c12ba0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.522+0000 7ff875c3a740 1 -- 172.21.15.67:0/1898319739 shutdown_connections 2024-03-24T00:49:23.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.522+0000 7ff875c3a740 1 --2- 172.21.15.67:0/1898319739 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x5580d8dc6c00 0x5580d7f23700 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.522+0000 7ff875c3a740 1 --2- 172.21.15.67:0/1898319739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580d8d55000 0x5580d7f22100 unknown :-1 s=CLOSED pgs=1839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.522+0000 7ff875c3a740 1 -- 172.21.15.67:0/1898319739 >> 172.21.15.67:0/1898319739 conn(0x5580d8c6e400 msgr2=0x5580d8bff800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:23.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.522+0000 7ff875c3a740 1 -- 172.21.15.67:0/1898319739 shutdown_connections 2024-03-24T00:49:23.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.522+0000 7ff875c3a740 1 -- 172.21.15.67:0/1898319739 wait complete. 2024-03-24T00:49:23.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.522+0000 7ff875c3a740 1 -- 172.21.15.67:0/2611870079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580d8c6e800 msgr2=0x5580d7f21600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.522+0000 7ff875c3a740 1 --2- 172.21.15.67:0/2611870079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580d8c6e800 0x5580d7f21600 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x5580d8ca5860 tx=0x5580d8c16000 comp rx=0 tx=0).stop 2024-03-24T00:49:23.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.522+0000 7ff875c3a740 1 -- 172.21.15.67:0/2611870079 shutdown_connections 2024-03-24T00:49:23.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.522+0000 7ff875c3a740 1 --2- 172.21.15.67:0/2611870079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580d8c6e800 0x5580d7f21600 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.522+0000 7ff875c3a740 1 --2- 172.21.15.67:0/2611870079 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5580d7e77c00 0x5580d7f21b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.522+0000 7ff875c3a740 1 --2- 172.21.15.67:0/2611870079 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5580d8c6e000 0x5580d7f21080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.522+0000 7ff875c3a740 1 -- 172.21.15.67:0/2611870079 >> 172.21.15.67:0/2611870079 conn(0x5580d7e77000 msgr2=0x5580d8bff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:23.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.523+0000 7ff875c3a740 1 -- 172.21.15.67:0/2611870079 shutdown_connections 2024-03-24T00:49:23.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.523+0000 7ff875c3a740 1 -- 172.21.15.67:0/2611870079 wait complete. 2024-03-24T00:49:23.535 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-data-scan', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'scan_extents', 'cephfs_data'] in time 0:00:00.125167, result: 2024-03-24T00:49:23.535 DEBUG:teuthology.orchestra.run.smithi067:> cephfs-data-scan --debug-mds=20 --debug-ms=1 --debug-objecter=1 scan_inodes --filter-tag mytag123 cephfs_data 2024-03-24T00:49:23.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.607+0000 7ff1cdc0c740 1 -- start start 2024-03-24T00:49:23.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.607+0000 7ff1cdc0c740 1 Processor -- start 2024-03-24T00:49:23.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.608+0000 7ff1cdc0c740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e2edc52c00 0x55e2edcfc580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.608+0000 7ff1cdc0c740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e2edc53800 0x55e2edcfcb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.608+0000 7ff1cdc0c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e2edc53c00 0x55e2edcfd080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.608+0000 7ff1cdc0c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2edd4c180 con 0x55e2edc53c00 2024-03-24T00:49:23.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.608+0000 7ff1cdc0c740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2edd4c300 con 0x55e2edc53800 2024-03-24T00:49:23.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.608+0000 7ff1cdc0c740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e2edd4c480 con 0x55e2edc52c00 2024-03-24T00:49:23.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.608+0000 7ff1cc0e7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e2edc53c00 0x55e2edcfd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.608+0000 7ff1cc0e7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e2edc53c00 0x55e2edcfd080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38422/0 (socket says 172.21.15.67:38422) 2024-03-24T00:49:23.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.608+0000 7ff1cc0e7640 1 -- 172.21.15.67:0/2878328922 learned_addr learned my addr 172.21.15.67:0/2878328922 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:23.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.608+0000 7ff1cb0e5640 1 --2- 172.21.15.67:0/2878328922 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e2edc53800 0x55e2edcfcb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.608+0000 7ff1cb8e6640 1 --2- 172.21.15.67:0/2878328922 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e2edc52c00 0x55e2edcfc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.608+0000 7ff1cc0e7640 1 -- 172.21.15.67:0/2878328922 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e2edc52c00 msgr2=0x55e2edcfc580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:23.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.608+0000 7ff1cc0e7640 1 --2- 172.21.15.67:0/2878328922 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e2edc52c00 0x55e2edcfc580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.608+0000 7ff1cc0e7640 1 -- 172.21.15.67:0/2878328922 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e2edc53800 msgr2=0x55e2edcfcb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:23.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.608+0000 7ff1cc0e7640 1 --2- 172.21.15.67:0/2878328922 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e2edc53800 0x55e2edcfcb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.608+0000 7ff1cc0e7640 1 -- 172.21.15.67:0/2878328922 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e2edc7f680 con 0x55e2edc53c00 2024-03-24T00:49:23.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.608+0000 7ff1cb8e6640 1 --2- 172.21.15.67:0/2878328922 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e2edc52c00 0x55e2edcfc580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:23.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.609+0000 7ff1cc0e7640 1 --2- 172.21.15.67:0/2878328922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e2edc53c00 0x55e2edcfd080 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x55e2eea59890 tx=0x55e2ee9ee840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e42de7c6f2e390b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:23.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.609+0000 7ff1c90e1640 1 -- 172.21.15.67:0/2878328922 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e2edc789c0 con 0x55e2edc53c00 2024-03-24T00:49:23.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.609+0000 7ff1c90e1640 1 -- 172.21.15.67:0/2878328922 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e2edc9ce00 con 0x55e2edc53c00 2024-03-24T00:49:23.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.609+0000 7ff1c90e1640 1 -- 172.21.15.67:0/2878328922 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e2edc78680 con 0x55e2edc53c00 2024-03-24T00:49:23.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.609+0000 7ff1cdc0c740 1 -- 172.21.15.67:0/2878328922 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e2edc7f860 con 0x55e2edc53c00 2024-03-24T00:49:23.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.609+0000 7ff1cdc0c740 1 -- 172.21.15.67:0/2878328922 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x55e2edc7f2c0 con 0x55e2edc53c00 2024-03-24T00:49:23.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.609+0000 7ff1cdc0c740 4 waiting for MDS map... 2024-03-24T00:49:23.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.609+0000 7ff1c90e1640 1 -- 172.21.15.67:0/2878328922 <== mon.0 v2:172.21.15.67:3300/0 4 ==== osd_map(122..122 src has 1..122) v4 ==== 6273+0+0 (secure 0 0 0) 0x55e2eeb18200 con 0x55e2edc53c00 2024-03-24T00:49:23.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.609+0000 7ff1c90e1640 1 -- 172.21.15.67:0/2878328922 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 93) v1 ==== 2735+0+0 (secure 0 0 0) 0x55e2edcab180 con 0x55e2edc53c00 2024-03-24T00:49:23.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.609+0000 7ff1cdc0c740 4 Got MDS map 93 2024-03-24T00:49:23.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.609+0000 7ff1cdc0c740 10 datascan.parse_kwarg: Applying tag filter: 'mytag123' 2024-03-24T00:49:23.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.609+0000 7ff1cdc0c740 4 datascan.main: Using metadata pool output 2024-03-24T00:49:23.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.609+0000 7ff1cdc0c740 4 datascan.main: connecting to RADOS... 2024-03-24T00:49:23.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.610+0000 7ff1cdc0c740 1 Processor -- start 2024-03-24T00:49:23.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.610+0000 7ff1cdc0c740 1 -- start start 2024-03-24T00:49:23.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.610+0000 7ff1cdc0c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e2eeb6e000 0x55e2edcfd600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.610+0000 7ff1cdc0c740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e2eeb6e400 0x55e2edcfdb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.610+0000 7ff1cdc0c740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e2eeb6e800 0x55e2edcfe100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.610+0000 7ff1cdc0c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2edd4c600 con 0x55e2eeb6e000 2024-03-24T00:49:23.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.610+0000 7ff1cdc0c740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2edd4c780 con 0x55e2eeb6e800 2024-03-24T00:49:23.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.610+0000 7ff1cdc0c740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e2edd4c900 con 0x55e2eeb6e400 2024-03-24T00:49:23.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.610+0000 7ff1cb0e5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e2eeb6e000 0x55e2edcfd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.610+0000 7ff1cb0e5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e2eeb6e000 0x55e2edcfd600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38426/0 (socket says 172.21.15.67:38426) 2024-03-24T00:49:23.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.610+0000 7ff1cb0e5640 1 -- 172.21.15.67:0/385331848 learned_addr learned my addr 172.21.15.67:0/385331848 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:23.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.610+0000 7ff1cb8e6640 1 --2- 172.21.15.67:0/385331848 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e2eeb6e400 0x55e2edcfdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.610+0000 7ff1cb0e5640 1 --2- 172.21.15.67:0/385331848 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e2eeb6e800 0x55e2edcfe100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.611+0000 7ff1cb0e5640 1 -- 172.21.15.67:0/385331848 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e2eeb6e400 msgr2=0x55e2edcfdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:23.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.611+0000 7ff1cb0e5640 1 --2- 172.21.15.67:0/385331848 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e2eeb6e400 0x55e2edcfdb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.611+0000 7ff1cb0e5640 1 -- 172.21.15.67:0/385331848 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e2eeb6e800 msgr2=0x55e2edcfe100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:23.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.611+0000 7ff1cb0e5640 1 --2- 172.21.15.67:0/385331848 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e2eeb6e800 0x55e2edcfe100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.611+0000 7ff1cb0e5640 1 -- 172.21.15.67:0/385331848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e2edc7f4a0 con 0x55e2eeb6e000 2024-03-24T00:49:23.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.611+0000 7ff1cb0e5640 1 --2- 172.21.15.67:0/385331848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e2eeb6e000 0x55e2edcfd600 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x55e2eeb0af60 tx=0x55e2eeb24c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28f9c856a9b8d9d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:23.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.611+0000 7ff1c70dd640 1 -- 172.21.15.67:0/385331848 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e2edc789c0 con 0x55e2eeb6e000 2024-03-24T00:49:23.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.611+0000 7ff1c70dd640 1 -- 172.21.15.67:0/385331848 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e2edc9ca80 con 0x55e2eeb6e000 2024-03-24T00:49:23.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.611+0000 7ff1c70dd640 1 -- 172.21.15.67:0/385331848 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e2edc78680 con 0x55e2eeb6e000 2024-03-24T00:49:23.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.611+0000 7ff1cdc0c740 1 -- 172.21.15.67:0/385331848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e2eeb6e000 msgr2=0x55e2edcfd600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.611+0000 7ff1cdc0c740 1 --2- 172.21.15.67:0/385331848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e2eeb6e000 0x55e2edcfd600 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x55e2eeb0af60 tx=0x55e2eeb24c60 comp rx=0 tx=0).stop 2024-03-24T00:49:23.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.611+0000 7ff1cdc0c740 1 -- 172.21.15.67:0/385331848 shutdown_connections 2024-03-24T00:49:23.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.611+0000 7ff1cdc0c740 1 --2- 172.21.15.67:0/385331848 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e2eeb6e800 0x55e2edcfe100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.611+0000 7ff1cdc0c740 1 --2- 172.21.15.67:0/385331848 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e2eeb6e400 0x55e2edcfdb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.611+0000 7ff1cdc0c740 1 --2- 172.21.15.67:0/385331848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e2eeb6e000 0x55e2edcfd600 unknown :-1 s=CLOSED pgs=1843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.611+0000 7ff1cdc0c740 1 -- 172.21.15.67:0/385331848 >> 172.21.15.67:0/385331848 conn(0x55e2ee9edc00 msgr2=0x55e2edd3b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:23.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.611+0000 7ff1cdc0c740 1 -- 172.21.15.67:0/385331848 shutdown_connections 2024-03-24T00:49:23.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.611+0000 7ff1cdc0c740 1 -- 172.21.15.67:0/385331848 wait complete. 2024-03-24T00:49:23.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.612+0000 7ff1cdc0c740 1 Processor -- start 2024-03-24T00:49:23.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.612+0000 7ff1cdc0c740 1 -- start start 2024-03-24T00:49:23.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.612+0000 7ff1cdc0c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e2eeb6e000 0x55e2edcff180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.612+0000 7ff1cdc0c740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e2eeb6e400 0x55e2edcfec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.612+0000 7ff1cdc0c740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e2eeb6e800 0x55e2edcfe680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.612+0000 7ff1cdc0c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2edd4dc80 con 0x55e2eeb6e000 2024-03-24T00:49:23.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.612+0000 7ff1cdc0c740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2edd4db00 con 0x55e2eeb6e400 2024-03-24T00:49:23.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.612+0000 7ff1cdc0c740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e2edd4d980 con 0x55e2eeb6e800 2024-03-24T00:49:23.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.612+0000 7ff1cb0e5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e2eeb6e000 0x55e2edcff180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.612+0000 7ff1cb0e5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e2eeb6e000 0x55e2edcff180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38436/0 (socket says 172.21.15.67:38436) 2024-03-24T00:49:23.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.613+0000 7ff1cb8e6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e2eeb6e400 0x55e2edcfec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.613+0000 7ff1cb0e5640 1 -- 172.21.15.67:0/724085066 learned_addr learned my addr 172.21.15.67:0/724085066 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:23.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.613+0000 7ff1cb0e5640 1 --2- 172.21.15.67:0/724085066 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e2eeb6e800 0x55e2edcfe680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.613+0000 7ff1cb0e5640 1 -- 172.21.15.67:0/724085066 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e2eeb6e800 msgr2=0x55e2edcfe680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.613+0000 7ff1cb0e5640 1 --2- 172.21.15.67:0/724085066 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e2eeb6e800 0x55e2edcfe680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.613+0000 7ff1cb0e5640 1 -- 172.21.15.67:0/724085066 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e2eeb6e400 msgr2=0x55e2edcfec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.613+0000 7ff1cb0e5640 1 --2- 172.21.15.67:0/724085066 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e2eeb6e400 0x55e2edcfec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.613+0000 7ff1cb0e5640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e2edc7f0e0 con 0x55e2eeb6e000 2024-03-24T00:49:23.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.613+0000 7ff1cb0e5640 1 --2- 172.21.15.67:0/724085066 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e2eeb6e000 0x55e2edcff180 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x55e2eeb0bc20 tx=0x55e2ee9ee960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:23.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.613+0000 7ff1c58da640 1 -- 172.21.15.67:0/724085066 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e2edc784e0 con 0x55e2eeb6e000 2024-03-24T00:49:23.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.613+0000 7ff1c58da640 1 -- 172.21.15.67:0/724085066 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e2edc9d340 con 0x55e2eeb6e000 2024-03-24T00:49:23.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.613+0000 7ff1cdc0c740 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55e2edc7fa40 con 0x55e2eeb6e000 2024-03-24T00:49:23.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.613+0000 7ff1c58da640 1 -- 172.21.15.67:0/724085066 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e2edc789c0 con 0x55e2eeb6e000 2024-03-24T00:49:23.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.613+0000 7ff1cdc0c740 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e2edc7e3c0 con 0x55e2eeb6e000 2024-03-24T00:49:23.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.613+0000 7ff1cdc0c740 4 datascan.init: resolving metadata pool 17 2024-03-24T00:49:23.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.614+0000 7ff1c58da640 1 -- 172.21.15.67:0/724085066 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x55e2eeb12c00 con 0x55e2eeb6e000 2024-03-24T00:49:23.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.615+0000 7ff1c58da640 1 --2- 172.21.15.67:0/724085066 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x55e2edfc6000 0x55e2edcff700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.615+0000 7ff1c58da640 1 -- 172.21.15.67:0/724085066 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6273+0+0 (secure 0 0 0) 0x55e2eeb10a00 con 0x55e2eeb6e000 2024-03-24T00:49:23.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.615+0000 7ff1cdc0c740 4 datascan.init: found metadata pool 'cephfs_metadata' 2024-03-24T00:49:23.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.615+0000 7ff1cdc0c740 4 datascan.main: data pool 'cephfs_data' has ID 18 2024-03-24T00:49:23.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.615+0000 7ff1cdc0c740 4 datascan.main: opening data pool 'cephfs_data' 2024-03-24T00:49:23.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.615+0000 7ff1cdc0c740 1 --2- 172.21.15.67:0/724085066 >> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] conn(0x55e2eeb6ec00 0x55e2edcfe100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.615+0000 7ff1cdc0c740 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:1 17.14 17:287034c4:::1.00000000.inode:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2eeb6f000 con 0x55e2eeb6ec00 2024-03-24T00:49:23.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.615+0000 7ff1cb8e6640 1 --2- 172.21.15.67:0/724085066 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x55e2edfc6000 0x55e2edcff700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.615+0000 7ff1cb0e5640 1 --2- 172.21.15.67:0/724085066 >> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] conn(0x55e2eeb6ec00 0x55e2edcfe100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.615+0000 7ff1cb0e5640 1 --2- 172.21.15.67:0/724085066 >> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] conn(0x55e2eeb6ec00 0x55e2edcfe100 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:23.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.615+0000 7ff1cb8e6640 1 --2- 172.21.15.67:0/724085066 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x55e2edfc6000 0x55e2edcff700 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x55e2eea59b90 tx=0x55e2edfd2000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:23.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.615+0000 7ff1cb0e5640 1 -- 172.21.15.67:0/724085066 <== osd.3 v2:172.21.15.67:6816/3303489481 1 ==== osd_op_reply(1 1.00000000.inode [stat out=16b] v0'0 uv36 ondisk = 0) v8 ==== 160+0+16 (crc 0 0 0) 0x55e2ee9d8000 con 0x55e2eeb6ec00 2024-03-24T00:49:23.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.615+0000 7ff1cdc0c740 1 --2- 172.21.15.67:0/724085066 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x55e2edfc7000 0x55e2edcfdb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.616+0000 7ff1cdc0c740 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:2 17.1f 17:fe07bba1:::100.00000000.inode:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edfc7400 con 0x55e2edfc7000 2024-03-24T00:49:23.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.616+0000 7ff1cc0e7640 1 --2- 172.21.15.67:0/724085066 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x55e2edfc7000 0x55e2edcfdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.616+0000 7ff1cc0e7640 1 --2- 172.21.15.67:0/724085066 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x55e2edfc7000 0x55e2edcfdb80 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:23.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.616+0000 7ff1cc0e7640 1 -- 172.21.15.67:0/724085066 <== osd.6 v2:172.21.15.72:6816/1725206562 1 ==== osd_op_reply(2 100.00000000.inode [stat out=16b] v0'0 uv3 ondisk = 0) v8 ==== 162+0+16 (crc 0 0 0) 0x55e2ee9d8500 con 0x55e2edfc7000 2024-03-24T00:49:23.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.616+0000 7ff1cdc0c740 1 --2- 172.21.15.67:0/724085066 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x55e2edfc7800 0x55e2edcfd600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.616+0000 7ff1cdc0c740 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:3 18.0 18:00000000::::head [pgnls-filter start_epoch 122 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edfc7c00 con 0x55e2edfc7800 2024-03-24T00:49:23.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.617+0000 7ff1cb8e6640 1 --2- 172.21.15.67:0/724085066 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x55e2edfc7800 0x55e2edcfd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.617+0000 7ff1cb8e6640 1 --2- 172.21.15.67:0/724085066 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x55e2edfc7800 0x55e2edcfd600 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:23.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.617+0000 7ff1cb8e6640 1 -- 172.21.15.67:0/724085066 <== osd.0 v2:172.21.15.67:6808/2628130519 1 ==== osd_op_reply(3 [pgnls-filter start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8780 con 0x55e2edfc7800 2024-03-24T00:49:23.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.617+0000 7ff1c68dc640 1 --2- 172.21.15.67:0/724085066 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x55e2edfec000 0x55e2edff2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.617+0000 7ff1c68dc640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:4 18.20 18:04000000::::head [pgnls-filter start_epoch 122 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edfec400 con 0x55e2edfec000 2024-03-24T00:49:23.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.617+0000 7ff1cb0e5640 1 --2- 172.21.15.67:0/724085066 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x55e2edfec000 0x55e2edff2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.618+0000 7ff1cb0e5640 1 --2- 172.21.15.67:0/724085066 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x55e2edfec000 0x55e2edff2000 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:23.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.618+0000 7ff1cb0e5640 1 -- 172.21.15.67:0/724085066 <== osd.5 v2:172.21.15.72:6800/288486206 1 ==== osd_op_reply(4 [pgnls-filter start_epoch 122 out=49b] v120'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8000 con 0x55e2edfec000 2024-03-24T00:49:23.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.618+0000 7ff1c70dd640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:5 18.10 18:08000000::::head [pgnls-filter start_epoch 122 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edfecc00 con 0x55e2edfc7000 2024-03-24T00:49:23.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.619+0000 7ff1cc0e7640 1 -- 172.21.15.67:0/724085066 <== osd.6 v2:172.21.15.72:6816/1725206562 2 ==== osd_op_reply(5 [pgnls-filter start_epoch 122 out=49b] v120'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8500 con 0x55e2edfc7000 2024-03-24T00:49:23.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.619+0000 7ff1c68dc640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:6 18.30 18:0c000000::::head [pgnls-filter start_epoch 122 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edfec800 con 0x55e2edfc7800 2024-03-24T00:49:23.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.619+0000 7ff1cb8e6640 1 -- 172.21.15.67:0/724085066 <== osd.0 v2:172.21.15.67:6808/2628130519 2 ==== osd_op_reply(6 [pgnls-filter start_epoch 122 out=49b] v120'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8780 con 0x55e2edfc7800 2024-03-24T00:49:23.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.619+0000 7ff1c70dd640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:7 18.8 18:10000000::::head [pgnls-filter start_epoch 122 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edfed000 con 0x55e2edfc7800 2024-03-24T00:49:23.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.619+0000 7ff1cb8e6640 1 -- 172.21.15.67:0/724085066 <== osd.0 v2:172.21.15.67:6808/2628130519 3 ==== osd_op_reply(7 [pgnls-filter start_epoch 122 out=49b] v120'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8780 con 0x55e2edfc7800 2024-03-24T00:49:23.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.619+0000 7ff1c68dc640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:8 18.28 18:14000000::::head [pgnls-filter start_epoch 122 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edfec800 con 0x55e2edfec000 2024-03-24T00:49:23.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.619+0000 7ff1cb0e5640 1 -- 172.21.15.67:0/724085066 <== osd.5 v2:172.21.15.72:6800/288486206 2 ==== osd_op_reply(8 [pgnls-filter start_epoch 122 out=49b] v120'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8000 con 0x55e2edfec000 2024-03-24T00:49:23.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.619+0000 7ff1c70dd640 1 --2- 172.21.15.67:0/724085066 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x55e2edfed400 0x55e2edff2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.619+0000 7ff1c70dd640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:9 18.18 18:18000000::::head [pgnls-filter start_epoch 122 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edfed000 con 0x55e2edfed400 2024-03-24T00:49:23.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.619+0000 7ff1cc0e7640 1 --2- 172.21.15.67:0/724085066 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x55e2edfed400 0x55e2edff2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.620+0000 7ff1cc0e7640 1 --2- 172.21.15.67:0/724085066 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x55e2edfed400 0x55e2edff2580 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:23.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.620+0000 7ff1cc0e7640 1 -- 172.21.15.67:0/724085066 <== osd.1 v2:172.21.15.67:6800/4062510404 1 ==== osd_op_reply(9 [pgnls-filter start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8500 con 0x55e2edfed400 2024-03-24T00:49:23.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.620+0000 7ff1c68dc640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:10 18.38 18:1c000000::::head [pgnls-filter start_epoch 122 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edfc6c00 con 0x55e2eeb6ec00 2024-03-24T00:49:23.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.620+0000 7ff1cb0e5640 1 -- 172.21.15.67:0/724085066 <== osd.3 v2:172.21.15.67:6816/3303489481 2 ==== osd_op_reply(10 [pgnls-filter start_epoch 122 out=49b] v120'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8000 con 0x55e2eeb6ec00 2024-03-24T00:49:23.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.620+0000 7ff1c70dd640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:11 18.4 18:20000000::::head [pgnls-filter start_epoch 122 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edfed800 con 0x55e2edfc7800 2024-03-24T00:49:23.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.620+0000 7ff1cb8e6640 1 -- 172.21.15.67:0/724085066 <== osd.0 v2:172.21.15.67:6808/2628130519 4 ==== osd_op_reply(11 [pgnls-filter start_epoch 122 out=49b] v120'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8780 con 0x55e2edfc7800 2024-03-24T00:49:23.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.620+0000 7ff1c68dc640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:12 18.24 18:24000000::::head [pgnls-filter start_epoch 122 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edfc7c00 con 0x55e2edfc7800 2024-03-24T00:49:23.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.621+0000 7ff1cb8e6640 1 -- 172.21.15.67:0/724085066 <== osd.0 v2:172.21.15.67:6808/2628130519 5 ==== osd_op_reply(12 [pgnls-filter start_epoch 122 out=49b] v120'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8780 con 0x55e2edfc7800 2024-03-24T00:49:23.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.621+0000 7ff1c70dd640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:13 18.14 18:28000000::::head [pgnls-filter start_epoch 122 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edfedc00 con 0x55e2edfc7000 2024-03-24T00:49:23.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.621+0000 7ff1cc0e7640 1 -- 172.21.15.67:0/724085066 <== osd.6 v2:172.21.15.72:6816/1725206562 3 ==== osd_op_reply(13 [pgnls-filter start_epoch 122 out=49b] v120'22 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8500 con 0x55e2edfc7000 2024-03-24T00:49:23.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.621+0000 7ff1c68dc640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:14 18.34 18:2c000000::::head [pgnls-filter start_epoch 122 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edfc6c00 con 0x55e2edfc7800 2024-03-24T00:49:23.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.621+0000 7ff1cb8e6640 1 -- 172.21.15.67:0/724085066 <== osd.0 v2:172.21.15.67:6808/2628130519 6 ==== osd_op_reply(14 [pgnls-filter start_epoch 122 out=49b] v120'14 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8780 con 0x55e2edfc7800 2024-03-24T00:49:23.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.621+0000 7ff1c70dd640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:15 18.c 18:30000000::::head [pgnls-filter start_epoch 122 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edffc000 con 0x55e2edfec000 2024-03-24T00:49:23.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.622+0000 7ff1cb0e5640 1 -- 172.21.15.67:0/724085066 <== osd.5 v2:172.21.15.72:6800/288486206 3 ==== osd_op_reply(15 [pgnls-filter start_epoch 122 out=49b] v120'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8000 con 0x55e2edfec000 2024-03-24T00:49:23.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.622+0000 7ff1c68dc640 1 --2- 172.21.15.67:0/724085066 >> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] conn(0x55e2edfec400 0x55e2edff2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.622+0000 7ff1c68dc640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] -- osd_op(unknown.0.0:16 18.2c 18:34000000::::head [pgnls-filter start_epoch 122 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edfec800 con 0x55e2edfec400 2024-03-24T00:49:23.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.622+0000 7ff1cb8e6640 1 --2- 172.21.15.67:0/724085066 >> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] conn(0x55e2edfec400 0x55e2edff2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.622+0000 7ff1cb8e6640 1 --2- 172.21.15.67:0/724085066 >> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] conn(0x55e2edfec400 0x55e2edff2b00 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:23.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.622+0000 7ff1cb8e6640 1 -- 172.21.15.67:0/724085066 <== osd.2 v2:172.21.15.67:6817/1268432561 1 ==== osd_op_reply(16 [pgnls-filter start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8780 con 0x55e2edfec400 2024-03-24T00:49:23.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.622+0000 7ff1c70dd640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:17 18.1c 18:38000000::::head [pgnls-filter start_epoch 122 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edffc400 con 0x55e2edfed400 2024-03-24T00:49:23.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.623+0000 7ff1cc0e7640 1 -- 172.21.15.67:0/724085066 <== osd.1 v2:172.21.15.67:6800/4062510404 2 ==== osd_op_reply(17 [pgnls-filter start_epoch 122 out=49b] v120'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8500 con 0x55e2edfed400 2024-03-24T00:49:23.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.623+0000 7ff1c68dc640 1 --2- 172.21.15.67:0/724085066 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x55e2eeb6f000 0x55e2edff3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.623+0000 7ff1c68dc640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:18 18.3c 18:3c000000::::head [pgnls-filter start_epoch 122 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edffc400 con 0x55e2eeb6f000 2024-03-24T00:49:23.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.623+0000 7ff1cb0e5640 1 --2- 172.21.15.67:0/724085066 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x55e2eeb6f000 0x55e2edff3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.623+0000 7ff1cb0e5640 1 --2- 172.21.15.67:0/724085066 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x55e2eeb6f000 0x55e2edff3080 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:23.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.624+0000 7ff1cb0e5640 1 -- 172.21.15.67:0/724085066 <== osd.7 v2:172.21.15.72:6824/2893038746 1 ==== osd_op_reply(18 [pgnls-filter start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8000 con 0x55e2eeb6f000 2024-03-24T00:49:23.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.624+0000 7ff1c70dd640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:19 18.2 18:40000000::::head [pgnls-filter start_epoch 122 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edffc800 con 0x55e2eeb6f000 2024-03-24T00:49:23.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.624+0000 7ff1cb0e5640 1 -- 172.21.15.67:0/724085066 <== osd.7 v2:172.21.15.72:6824/2893038746 2 ==== osd_op_reply(19 [pgnls-filter start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8000 con 0x55e2eeb6f000 2024-03-24T00:49:23.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.624+0000 7ff1c68dc640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:20 18.22 18:44000000::::head [pgnls-filter start_epoch 122 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edfecc00 con 0x55e2eeb6ec00 2024-03-24T00:49:23.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.624+0000 7ff1cb0e5640 1 -- 172.21.15.67:0/724085066 <== osd.3 v2:172.21.15.67:6816/3303489481 3 ==== osd_op_reply(20 [pgnls-filter start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8000 con 0x55e2eeb6ec00 2024-03-24T00:49:23.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.624+0000 7ff1c70dd640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] -- osd_op(unknown.0.0:21 18.12 18:48000000::::head [pgnls-filter start_epoch 122 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edffcc00 con 0x55e2edfec400 2024-03-24T00:49:23.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.624+0000 7ff1cb8e6640 1 -- 172.21.15.67:0/724085066 <== osd.2 v2:172.21.15.67:6817/1268432561 2 ==== osd_op_reply(21 [pgnls-filter start_epoch 122 out=81b] v122'23 uv23 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x55e2ee9d8780 con 0x55e2edfec400 2024-03-24T00:49:23.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.625+0000 7ff1cdc0c740 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:22 18.0 18:00000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edffcc00 con 0x55e2edfc7800 2024-03-24T00:49:23.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.625+0000 7ff1cb8e6640 1 -- 172.21.15.67:0/724085066 <== osd.0 v2:172.21.15.67:6808/2628130519 7 ==== osd_op_reply(22 [pgnls-filter start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8780 con 0x55e2edfc7800 2024-03-24T00:49:23.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.625+0000 7ff1c70dd640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:23 18.20 18:04000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edffd800 con 0x55e2edfec000 2024-03-24T00:49:23.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.625+0000 7ff1cb0e5640 1 -- 172.21.15.67:0/724085066 <== osd.5 v2:172.21.15.72:6800/288486206 4 ==== osd_op_reply(23 [pgnls-filter start_epoch 122 out=49b] v120'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8000 con 0x55e2edfec000 2024-03-24T00:49:23.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.625+0000 7ff1c68dc640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:24 18.10 18:08000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edfed000 con 0x55e2edfc7000 2024-03-24T00:49:23.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.625+0000 7ff1cc0e7640 1 -- 172.21.15.67:0/724085066 <== osd.6 v2:172.21.15.72:6816/1725206562 4 ==== osd_op_reply(24 [pgnls-filter start_epoch 122 out=49b] v120'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8500 con 0x55e2edfc7000 2024-03-24T00:49:23.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.626+0000 7ff1c70dd640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:25 18.30 18:0c000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edffdc00 con 0x55e2edfc7800 2024-03-24T00:49:23.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.626+0000 7ff1cb8e6640 1 -- 172.21.15.67:0/724085066 <== osd.0 v2:172.21.15.67:6808/2628130519 8 ==== osd_op_reply(25 [pgnls-filter start_epoch 122 out=49b] v120'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8780 con 0x55e2edfc7800 2024-03-24T00:49:23.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.626+0000 7ff1c68dc640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:26 18.8 18:10000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edfedc00 con 0x55e2edfc7800 2024-03-24T00:49:23.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.626+0000 7ff1cb8e6640 1 -- 172.21.15.67:0/724085066 <== osd.0 v2:172.21.15.67:6808/2628130519 9 ==== osd_op_reply(26 [pgnls-filter start_epoch 122 out=49b] v120'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8780 con 0x55e2edfc7800 2024-03-24T00:49:23.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.626+0000 7ff1c70dd640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:27 18.28 18:14000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edffc000 con 0x55e2edfec000 2024-03-24T00:49:23.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.626+0000 7ff1cb0e5640 1 -- 172.21.15.67:0/724085066 <== osd.5 v2:172.21.15.72:6800/288486206 5 ==== osd_op_reply(27 [pgnls-filter start_epoch 122 out=49b] v120'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8000 con 0x55e2edfec000 2024-03-24T00:49:23.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.626+0000 7ff1c68dc640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:28 18.18 18:18000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edfc7400 con 0x55e2edfed400 2024-03-24T00:49:23.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.627+0000 7ff1cc0e7640 1 -- 172.21.15.67:0/724085066 <== osd.1 v2:172.21.15.67:6800/4062510404 3 ==== osd_op_reply(28 [pgnls-filter start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8500 con 0x55e2edfed400 2024-03-24T00:49:23.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.627+0000 7ff1c70dd640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:29 18.38 18:1c000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edffc400 con 0x55e2eeb6ec00 2024-03-24T00:49:23.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.627+0000 7ff1cb0e5640 1 -- 172.21.15.67:0/724085066 <== osd.3 v2:172.21.15.67:6816/3303489481 4 ==== osd_op_reply(29 [pgnls-filter start_epoch 122 out=49b] v120'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8000 con 0x55e2eeb6ec00 2024-03-24T00:49:23.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.627+0000 7ff1c68dc640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:30 18.4 18:20000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2ee002000 con 0x55e2edfc7800 2024-03-24T00:49:23.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.627+0000 7ff1cb8e6640 1 -- 172.21.15.67:0/724085066 <== osd.0 v2:172.21.15.67:6808/2628130519 10 ==== osd_op_reply(30 [pgnls-filter start_epoch 122 out=49b] v120'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8780 con 0x55e2edfc7800 2024-03-24T00:49:23.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.627+0000 7ff1c70dd640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:31 18.24 18:24000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edffc800 con 0x55e2edfc7800 2024-03-24T00:49:23.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.627+0000 7ff1cb8e6640 1 -- 172.21.15.67:0/724085066 <== osd.0 v2:172.21.15.67:6808/2628130519 11 ==== osd_op_reply(31 [pgnls-filter start_epoch 122 out=49b] v120'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8780 con 0x55e2edfc7800 2024-03-24T00:49:23.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.627+0000 7ff1c68dc640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:32 18.14 18:28000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2ee002400 con 0x55e2edfc7000 2024-03-24T00:49:23.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.628+0000 7ff1cc0e7640 1 -- 172.21.15.67:0/724085066 <== osd.6 v2:172.21.15.72:6816/1725206562 5 ==== osd_op_reply(32 [pgnls-filter start_epoch 122 out=49b] v120'22 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8500 con 0x55e2edfc7000 2024-03-24T00:49:23.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.628+0000 7ff1c70dd640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:33 18.34 18:2c000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edffd800 con 0x55e2edfc7800 2024-03-24T00:49:23.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.628+0000 7ff1cb8e6640 1 -- 172.21.15.67:0/724085066 <== osd.0 v2:172.21.15.67:6808/2628130519 12 ==== osd_op_reply(33 [pgnls-filter start_epoch 122 out=49b] v120'14 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8780 con 0x55e2edfc7800 2024-03-24T00:49:23.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.628+0000 7ff1c68dc640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:34 18.c 18:30000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2ee002800 con 0x55e2edfec000 2024-03-24T00:49:23.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.628+0000 7ff1cb0e5640 1 -- 172.21.15.67:0/724085066 <== osd.5 v2:172.21.15.72:6800/288486206 6 ==== osd_op_reply(34 [pgnls-filter start_epoch 122 out=49b] v120'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8000 con 0x55e2edfec000 2024-03-24T00:49:23.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.628+0000 7ff1c70dd640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] -- osd_op(unknown.0.0:35 18.2c 18:34000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edfecc00 con 0x55e2edfec400 2024-03-24T00:49:23.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.628+0000 7ff1cb8e6640 1 -- 172.21.15.67:0/724085066 <== osd.2 v2:172.21.15.67:6817/1268432561 3 ==== osd_op_reply(35 [pgnls-filter start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8780 con 0x55e2edfec400 2024-03-24T00:49:23.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.628+0000 7ff1c68dc640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:36 18.1c 18:38000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2ee002c00 con 0x55e2edfed400 2024-03-24T00:49:23.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.629+0000 7ff1cc0e7640 1 -- 172.21.15.67:0/724085066 <== osd.1 v2:172.21.15.67:6800/4062510404 4 ==== osd_op_reply(36 [pgnls-filter start_epoch 122 out=49b] v120'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8500 con 0x55e2edfed400 2024-03-24T00:49:23.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.629+0000 7ff1c70dd640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:37 18.3c 18:3c000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2ee002000 con 0x55e2eeb6f000 2024-03-24T00:49:23.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.629+0000 7ff1cb0e5640 1 -- 172.21.15.67:0/724085066 <== osd.7 v2:172.21.15.72:6824/2893038746 3 ==== osd_op_reply(37 [pgnls-filter start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8000 con 0x55e2eeb6f000 2024-03-24T00:49:23.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.629+0000 7ff1c68dc640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:38 18.2 18:40000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2ee003000 con 0x55e2eeb6f000 2024-03-24T00:49:23.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.629+0000 7ff1cb0e5640 1 -- 172.21.15.67:0/724085066 <== osd.7 v2:172.21.15.72:6824/2893038746 4 ==== osd_op_reply(38 [pgnls-filter start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8000 con 0x55e2eeb6f000 2024-03-24T00:49:23.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.629+0000 7ff1c70dd640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:39 18.22 18:44000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2ee003800 con 0x55e2eeb6ec00 2024-03-24T00:49:23.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.630+0000 7ff1cb0e5640 1 -- 172.21.15.67:0/724085066 <== osd.3 v2:172.21.15.67:6816/3303489481 5 ==== osd_op_reply(39 [pgnls-filter start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8000 con 0x55e2eeb6ec00 2024-03-24T00:49:23.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.630+0000 7ff1c68dc640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] -- osd_op(unknown.0.0:40 18.12 18:48000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2ee003400 con 0x55e2edfec400 2024-03-24T00:49:23.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.630+0000 7ff1cb8e6640 1 -- 172.21.15.67:0/724085066 <== osd.2 v2:172.21.15.67:6817/1268432561 4 ==== osd_op_reply(40 [pgnls-filter start_epoch 122 out=49b] v122'23 uv23 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8780 con 0x55e2edfec400 2024-03-24T00:49:23.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.630+0000 7ff1c70dd640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:41 18.32 18:4c000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edffcc00 con 0x55e2eeb6ec00 2024-03-24T00:49:23.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.630+0000 7ff1cb0e5640 1 -- 172.21.15.67:0/724085066 <== osd.3 v2:172.21.15.67:6816/3303489481 6 ==== osd_op_reply(41 [pgnls-filter start_epoch 122 out=49b] v120'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8000 con 0x55e2eeb6ec00 2024-03-24T00:49:23.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.630+0000 7ff1c68dc640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:42 18.a 18:50000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2ee002800 con 0x55e2eeb6ec00 2024-03-24T00:49:23.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.630+0000 7ff1cb0e5640 1 -- 172.21.15.67:0/724085066 <== osd.3 v2:172.21.15.67:6816/3303489481 7 ==== osd_op_reply(42 [pgnls-filter start_epoch 122 out=49b] v120'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8000 con 0x55e2eeb6ec00 2024-03-24T00:49:23.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.630+0000 7ff1c70dd640 1 --2- 172.21.15.67:0/724085066 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x55e2edffdc00 0x55e2edff3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.630+0000 7ff1c70dd640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:43 18.2a 18:54000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edffc800 con 0x55e2edffdc00 2024-03-24T00:49:23.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.631+0000 7ff1cc0e7640 1 --2- 172.21.15.67:0/724085066 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x55e2edffdc00 0x55e2edff3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.631+0000 7ff1cc0e7640 1 --2- 172.21.15.67:0/724085066 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x55e2edffdc00 0x55e2edff3600 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:23.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.631+0000 7ff1cc0e7640 1 -- 172.21.15.67:0/724085066 <== osd.4 v2:172.21.15.72:6808/477220793 1 ==== osd_op_reply(43 [pgnls-filter start_epoch 122 out=49b] v120'20 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8500 con 0x55e2edffdc00 2024-03-24T00:49:23.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.631+0000 7ff1c68dc640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:44 18.1a 18:58000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2ee002000 con 0x55e2eeb6ec00 2024-03-24T00:49:23.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.632+0000 7ff1cb0e5640 1 -- 172.21.15.67:0/724085066 <== osd.3 v2:172.21.15.67:6816/3303489481 8 ==== osd_op_reply(44 [pgnls-filter start_epoch 122 out=49b] v120'14 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8000 con 0x55e2eeb6ec00 2024-03-24T00:49:23.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.632+0000 7ff1c70dd640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:45 18.3a 18:5c000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edffd800 con 0x55e2edfc7800 2024-03-24T00:49:23.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.632+0000 7ff1cb8e6640 1 -- 172.21.15.67:0/724085066 <== osd.0 v2:172.21.15.67:6808/2628130519 13 ==== osd_op_reply(45 [pgnls-filter start_epoch 122 out=49b] v120'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8780 con 0x55e2edfc7800 2024-03-24T00:49:23.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.632+0000 7ff1c68dc640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:46 18.6 18:60000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2ee003000 con 0x55e2eeb6f000 2024-03-24T00:49:23.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.632+0000 7ff1cb0e5640 1 -- 172.21.15.67:0/724085066 <== osd.7 v2:172.21.15.72:6824/2893038746 5 ==== osd_op_reply(46 [pgnls-filter start_epoch 122 out=49b] v120'14 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8000 con 0x55e2eeb6f000 2024-03-24T00:49:23.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.632+0000 7ff1c70dd640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] -- osd_op(unknown.0.0:47 18.26 18:64000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edfedc00 con 0x55e2edfec400 2024-03-24T00:49:23.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.633+0000 7ff1cb8e6640 1 -- 172.21.15.67:0/724085066 <== osd.2 v2:172.21.15.67:6817/1268432561 5 ==== osd_op_reply(47 [pgnls-filter start_epoch 122 out=49b] v120'30 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8780 con 0x55e2edfec400 2024-03-24T00:49:23.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.633+0000 7ff1c68dc640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:48 18.16 18:68000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2ee003800 con 0x55e2eeb6ec00 2024-03-24T00:49:23.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.633+0000 7ff1cb0e5640 1 -- 172.21.15.67:0/724085066 <== osd.3 v2:172.21.15.67:6816/3303489481 9 ==== osd_op_reply(48 [pgnls-filter start_epoch 122 out=49b] v120'20 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8000 con 0x55e2eeb6ec00 2024-03-24T00:49:23.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.633+0000 7ff1c70dd640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:49 18.36 18:6c000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2ee008000 con 0x55e2edfed400 2024-03-24T00:49:23.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.633+0000 7ff1cc0e7640 1 -- 172.21.15.67:0/724085066 <== osd.1 v2:172.21.15.67:6800/4062510404 5 ==== osd_op_reply(49 [pgnls-filter start_epoch 122 out=81b] v122'34 uv34 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x55e2ee9d8500 con 0x55e2edfed400 2024-03-24T00:49:23.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.633+0000 7ff1cdc0c740 20 datascan.forall_objects: OSD matched oid 100000001f7.00000000 2024-03-24T00:49:23.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.633+0000 7ff1cdc0c740 10 datascan.operator(): handling object 100000001f7.0 2024-03-24T00:49:23.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.633+0000 7ff1cdc0c740 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:50 18.36 18:6fe38dc2:::100000001f7.00000000:head [getxattr scan_ceiling in=12b,getxattr scan_max_size in=13b,getxattr scan_max_mtime in=14b,getxattr scan_pool_id in=12b,getxattr parent in=6b,getxattr layout in=6b,getxattr symlink in=7b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edffd400 con 0x55e2edfed400 2024-03-24T00:49:23.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.633+0000 7ff1cc0e7640 1 -- 172.21.15.67:0/724085066 <== osd.1 v2:172.21.15.67:6800/4062510404 6 ==== osd_op_reply(50 100000001f7.00000000 [getxattr (22) out=22b,getxattr (8) out=8b,getxattr (8) out=8b,getxattr,getxattr (126) out=126b,getxattr (30) out=30b,getxattr] v0'0 uv34 ondisk = 0) v8 ==== 416+0+194 (crc 0 0 0) 0x55e2ee9d8500 con 0x55e2edfed400 2024-03-24T00:49:23.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.633+0000 7ff1cdc0c740 10 datascan.inject_with_backtrace: inode: 0x0x100000001f7 2024-03-24T00:49:23.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.633+0000 7ff1cdc0c740 10 datascan.inject_with_backtrace: backptr: 0x0x10000000001/bravo 2024-03-24T00:49:23.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.633+0000 7ff1cdc0c740 20 datascan.get_frag_of: dirino=0x10000000001 target_dname=bravo 2024-03-24T00:49:23.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.633+0000 7ff1cdc0c740 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:51 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edfed800 con 0x55e2edfc7800 2024-03-24T00:49:23.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.634+0000 7ff1cb8e6640 1 -- 172.21.15.67:0/724085066 <== osd.0 v2:172.21.15.67:6808/2628130519 14 ==== osd_op_reply(51 10000000001.00000000 [getxattr (95) out=95b] v0'0 uv8 ondisk = 0) v8 ==== 164+0+95 (crc 0 0 0) 0x55e2ee9d8780 con 0x55e2edfc7800 2024-03-24T00:49:23.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.634+0000 7ff1cdc0c740 20 datascan.get_frag_of: got backtrace parent 0x10000000000/flushed 2024-03-24T00:49:23.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.634+0000 7ff1cdc0c740 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:52 17.10 17:0cf6ad0f:::10000000000.00000000:head [omap-get-vals-by-keys in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edfec800 con 0x55e2eeb6ec00 2024-03-24T00:49:23.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.634+0000 7ff1cb0e5640 1 -- 172.21.15.67:0/724085066 <== osd.3 v2:172.21.15.67:6816/3303489481 10 ==== osd_op_reply(52 10000000000.00000000 [omap-get-vals-by-keys out=585b] v0'0 uv12 ondisk = 0) v8 ==== 164+0+585 (crc 0 0 0) 0x55e2ee9d8000 con 0x55e2eeb6ec00 2024-03-24T00:49:23.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.634+0000 7ff1cdc0c740 20 datascan.read_dentry: oid=10000000000.00000000 dname=flushed frag=*, r=0 2024-03-24T00:49:23.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.634+0000 7ff1cdc0c740 20 datascan.get_frag_of: fast path, fragtree is fragtree_t() 2024-03-24T00:49:23.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.634+0000 7ff1cdc0c740 20 datascan.get_frag_of: frag is * 2024-03-24T00:49:23.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.634+0000 7ff1cdc0c740 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:53 17.7 17:e2e07930:::10000000001.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edfc7c00 con 0x55e2edfc7800 2024-03-24T00:49:23.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.634+0000 7ff1cb8e6640 1 -- 172.21.15.67:0/724085066 <== osd.0 v2:172.21.15.67:6808/2628130519 15 ==== osd_op_reply(53 10000000001.00000000 [omap-get-header out=274b] v0'0 uv8 ondisk = 0) v8 ==== 164+0+274 (crc 0 0 0) 0x55e2ee9d8780 con 0x55e2edfc7800 2024-03-24T00:49:23.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.634+0000 7ff1cdc0c740 10 datascan.find_or_create_dirfrag: read_version = 8 2024-03-24T00:49:23.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.634+0000 7ff1cdc0c740 20 datascan.find_or_create_dirfrag: Dirfrag already exists: 0x0x10000000001 * 2024-03-24T00:49:23.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.634+0000 7ff1cdc0c740 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:54 17.7 17:e2e07930:::10000000001.00000000:head [omap-get-vals-by-keys in=18b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edfc6c00 con 0x55e2edfc7800 2024-03-24T00:49:23.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.635+0000 7ff1cb8e6640 1 -- 172.21.15.67:0/724085066 <== osd.0 v2:172.21.15.67:6808/2628130519 16 ==== osd_op_reply(54 10000000001.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv8 ondisk = 0) v8 ==== 164+0+4 (crc 0 0 0) 0x55e2ee9d8780 con 0x55e2edfc7800 2024-03-24T00:49:23.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.635+0000 7ff1cdc0c740 20 datascan.read_dentry: oid=10000000001.00000000 dname=bravo frag=*, r=0 2024-03-24T00:49:23.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.635+0000 7ff1cdc0c740 20 datascan.read_dentry: bravo_head not found in result 2024-03-24T00:49:23.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.635+0000 7ff1cdc0c740 10 datascan.inject_with_backtrace: Linking inode 0x0x100000001f7 at 0x0x10000000001/bravo with size=5 bytes 2024-03-24T00:49:23.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.635+0000 7ff1cdc0c740 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:55 17.7 17:e2e07930:::10000000001.00000000:head [omap-set-vals in=506b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2ee003400 con 0x55e2edfc7800 2024-03-24T00:49:23.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.636+0000 7ff1cb8e6640 1 -- 172.21.15.67:0/724085066 <== osd.0 v2:172.21.15.67:6808/2628130519 17 ==== osd_op_reply(55 10000000001.00000000 [omap-set-vals] v122'9 uv9 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55e2ee9d8780 con 0x55e2edfc7800 2024-03-24T00:49:23.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.636+0000 7ff1cdc0c740 20 datascan.inject_linkage: Injected dentry 0x0x10000000001/bravo pointing to 0x0x100000001f7 2024-03-24T00:49:23.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.636+0000 7ff1cdc0c740 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:56 18.e 18:70000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edfc7c00 con 0x55e2edfc7800 2024-03-24T00:49:23.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.636+0000 7ff1cb8e6640 1 -- 172.21.15.67:0/724085066 <== osd.0 v2:172.21.15.67:6808/2628130519 18 ==== osd_op_reply(56 [pgnls-filter start_epoch 122 out=49b] v120'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8780 con 0x55e2edfc7800 2024-03-24T00:49:23.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.636+0000 7ff1c70dd640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:57 18.2e 18:74000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2ee008400 con 0x55e2eeb6ec00 2024-03-24T00:49:23.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.636+0000 7ff1cb0e5640 1 -- 172.21.15.67:0/724085066 <== osd.3 v2:172.21.15.67:6816/3303489481 11 ==== osd_op_reply(57 [pgnls-filter start_epoch 122 out=49b] v120'20 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8000 con 0x55e2eeb6ec00 2024-03-24T00:49:23.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.636+0000 7ff1c68dc640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:58 18.1e 18:78000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2ee003c00 con 0x55e2edfec000 2024-03-24T00:49:23.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.637+0000 7ff1cb0e5640 1 -- 172.21.15.67:0/724085066 <== osd.5 v2:172.21.15.72:6800/288486206 7 ==== osd_op_reply(58 [pgnls-filter start_epoch 122 out=49b] v120'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8000 con 0x55e2edfec000 2024-03-24T00:49:23.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.637+0000 7ff1c70dd640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:59 18.3e 18:7c000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2ee008800 con 0x55e2eeb6ec00 2024-03-24T00:49:23.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.637+0000 7ff1cb0e5640 1 -- 172.21.15.67:0/724085066 <== osd.3 v2:172.21.15.67:6816/3303489481 12 ==== osd_op_reply(59 [pgnls-filter start_epoch 122 out=49b] v120'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8000 con 0x55e2eeb6ec00 2024-03-24T00:49:23.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.637+0000 7ff1c68dc640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:60 18.1 18:80000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edffc000 con 0x55e2eeb6ec00 2024-03-24T00:49:23.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.637+0000 7ff1cb0e5640 1 -- 172.21.15.67:0/724085066 <== osd.3 v2:172.21.15.67:6816/3303489481 13 ==== osd_op_reply(60 [pgnls-filter start_epoch 122 out=49b] v120'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8000 con 0x55e2eeb6ec00 2024-03-24T00:49:23.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.637+0000 7ff1c70dd640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:61 18.21 18:84000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2ee008c00 con 0x55e2edfec000 2024-03-24T00:49:23.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.638+0000 7ff1cb0e5640 1 -- 172.21.15.67:0/724085066 <== osd.5 v2:172.21.15.72:6800/288486206 8 ==== osd_op_reply(61 [pgnls-filter start_epoch 122 out=49b] v120'24 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8000 con 0x55e2edfec000 2024-03-24T00:49:23.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.638+0000 7ff1c68dc640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:62 18.11 18:88000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edffc400 con 0x55e2edfed400 2024-03-24T00:49:23.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.638+0000 7ff1cc0e7640 1 -- 172.21.15.67:0/724085066 <== osd.1 v2:172.21.15.67:6800/4062510404 7 ==== osd_op_reply(62 [pgnls-filter start_epoch 122 out=49b] v120'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8500 con 0x55e2edfed400 2024-03-24T00:49:23.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.638+0000 7ff1c70dd640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:63 18.31 18:8c000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2ee009000 con 0x55e2edffdc00 2024-03-24T00:49:23.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.638+0000 7ff1cc0e7640 1 -- 172.21.15.67:0/724085066 <== osd.4 v2:172.21.15.72:6808/477220793 2 ==== osd_op_reply(63 [pgnls-filter start_epoch 122 out=49b] v120'15 uv8 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8500 con 0x55e2edffdc00 2024-03-24T00:49:23.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.638+0000 7ff1c68dc640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:64 18.9 18:90000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edffcc00 con 0x55e2eeb6ec00 2024-03-24T00:49:23.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.639+0000 7ff1cb0e5640 1 -- 172.21.15.67:0/724085066 <== osd.3 v2:172.21.15.67:6816/3303489481 14 ==== osd_op_reply(64 [pgnls-filter start_epoch 122 out=49b] v120'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8000 con 0x55e2eeb6ec00 2024-03-24T00:49:23.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.639+0000 7ff1c70dd640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:65 18.29 18:94000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2ee009400 con 0x55e2eeb6f000 2024-03-24T00:49:23.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.639+0000 7ff1cb0e5640 1 -- 172.21.15.67:0/724085066 <== osd.7 v2:172.21.15.72:6824/2893038746 6 ==== osd_op_reply(65 [pgnls-filter start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8000 con 0x55e2eeb6f000 2024-03-24T00:49:23.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.639+0000 7ff1c68dc640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:66 18.19 18:98000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2ee002800 con 0x55e2edffdc00 2024-03-24T00:49:23.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.639+0000 7ff1cc0e7640 1 -- 172.21.15.67:0/724085066 <== osd.4 v2:172.21.15.72:6808/477220793 3 ==== osd_op_reply(66 [pgnls-filter start_epoch 122 out=49b] v122'24 uv24 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8500 con 0x55e2edffdc00 2024-03-24T00:49:23.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.639+0000 7ff1c70dd640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:67 18.39 18:9c000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2ee009800 con 0x55e2edfc7800 2024-03-24T00:49:23.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.640+0000 7ff1cb8e6640 1 -- 172.21.15.67:0/724085066 <== osd.0 v2:172.21.15.67:6808/2628130519 19 ==== osd_op_reply(67 [pgnls-filter start_epoch 122 out=49b] v120'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8780 con 0x55e2edfc7800 2024-03-24T00:49:23.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.640+0000 7ff1c68dc640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:68 18.5 18:a0000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2ee002000 con 0x55e2edfed400 2024-03-24T00:49:23.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.640+0000 7ff1cc0e7640 1 -- 172.21.15.67:0/724085066 <== osd.1 v2:172.21.15.67:6800/4062510404 8 ==== osd_op_reply(68 [pgnls-filter start_epoch 122 out=49b] v122'15 uv15 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8500 con 0x55e2edfed400 2024-03-24T00:49:23.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.640+0000 7ff1c70dd640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:69 18.25 18:a4000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2ee009c00 con 0x55e2edfc7000 2024-03-24T00:49:23.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.640+0000 7ff1cc0e7640 1 -- 172.21.15.67:0/724085066 <== osd.6 v2:172.21.15.72:6816/1725206562 6 ==== osd_op_reply(69 [pgnls-filter start_epoch 122 out=49b] v120'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8500 con 0x55e2edfc7000 2024-03-24T00:49:23.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.640+0000 7ff1c68dc640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:70 18.15 18:a8000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2ee003000 con 0x55e2edfec000 2024-03-24T00:49:23.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.641+0000 7ff1cb0e5640 1 -- 172.21.15.67:0/724085066 <== osd.5 v2:172.21.15.72:6800/288486206 9 ==== osd_op_reply(70 [pgnls-filter start_epoch 122 out=49b] v120'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8000 con 0x55e2edfec000 2024-03-24T00:49:23.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.641+0000 7ff1c70dd640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:71 18.35 18:ac000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2ee009000 con 0x55e2edffdc00 2024-03-24T00:49:23.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.641+0000 7ff1cc0e7640 1 -- 172.21.15.67:0/724085066 <== osd.4 v2:172.21.15.72:6808/477220793 4 ==== osd_op_reply(71 [pgnls-filter start_epoch 122 out=49b] v120'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8500 con 0x55e2edffdc00 2024-03-24T00:49:23.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.641+0000 7ff1c68dc640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:72 18.d 18:b0000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2ee003800 con 0x55e2edffdc00 2024-03-24T00:49:23.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.641+0000 7ff1cc0e7640 1 -- 172.21.15.67:0/724085066 <== osd.4 v2:172.21.15.72:6808/477220793 5 ==== osd_op_reply(72 [pgnls-filter start_epoch 122 out=49b] v120'14 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8500 con 0x55e2edffdc00 2024-03-24T00:49:23.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.641+0000 7ff1c70dd640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:73 18.2d 18:b4000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2ee009c00 con 0x55e2edfc7000 2024-03-24T00:49:23.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.642+0000 7ff1cc0e7640 1 -- 172.21.15.67:0/724085066 <== osd.6 v2:172.21.15.72:6816/1725206562 7 ==== osd_op_reply(73 [pgnls-filter start_epoch 122 out=49b] v120'14 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8500 con 0x55e2edfc7000 2024-03-24T00:49:23.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.642+0000 7ff1c68dc640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:74 18.1d 18:b8000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2ee002400 con 0x55e2edfc7800 2024-03-24T00:49:23.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.642+0000 7ff1cb8e6640 1 -- 172.21.15.67:0/724085066 <== osd.0 v2:172.21.15.67:6808/2628130519 20 ==== osd_op_reply(74 [pgnls-filter start_epoch 122 out=49b] v120'30 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8780 con 0x55e2edfc7800 2024-03-24T00:49:23.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.642+0000 7ff1c70dd640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:75 18.3d 18:bc000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edffd400 con 0x55e2eeb6ec00 2024-03-24T00:49:23.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.643+0000 7ff1cb0e5640 1 -- 172.21.15.67:0/724085066 <== osd.3 v2:172.21.15.67:6816/3303489481 15 ==== osd_op_reply(75 [pgnls-filter start_epoch 122 out=49b] v120'14 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8000 con 0x55e2eeb6ec00 2024-03-24T00:49:23.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.643+0000 7ff1c68dc640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:76 18.3 18:c0000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2ee002c00 con 0x55e2edfed400 2024-03-24T00:49:23.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.643+0000 7ff1cc0e7640 1 -- 172.21.15.67:0/724085066 <== osd.1 v2:172.21.15.67:6800/4062510404 9 ==== osd_op_reply(76 [pgnls-filter start_epoch 122 out=49b] v120'14 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8500 con 0x55e2edfed400 2024-03-24T00:49:23.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.643+0000 7ff1c70dd640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] -- osd_op(unknown.0.0:77 18.23 18:c4000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edffd000 con 0x55e2edfec400 2024-03-24T00:49:23.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.643+0000 7ff1cb8e6640 1 -- 172.21.15.67:0/724085066 <== osd.2 v2:172.21.15.67:6817/1268432561 6 ==== osd_op_reply(77 [pgnls-filter start_epoch 122 out=49b] v120'14 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8780 con 0x55e2edfec400 2024-03-24T00:49:23.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.643+0000 7ff1c68dc640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:78 18.13 18:c8000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edfec800 con 0x55e2eeb6f000 2024-03-24T00:49:23.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.644+0000 7ff1cb0e5640 1 -- 172.21.15.67:0/724085066 <== osd.7 v2:172.21.15.72:6824/2893038746 7 ==== osd_op_reply(78 [pgnls-filter start_epoch 122 out=49b] v120'30 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8000 con 0x55e2eeb6f000 2024-03-24T00:49:23.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.644+0000 7ff1c70dd640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:79 18.33 18:cc000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edffc400 con 0x55e2edfed400 2024-03-24T00:49:23.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.644+0000 7ff1cc0e7640 1 -- 172.21.15.67:0/724085066 <== osd.1 v2:172.21.15.67:6800/4062510404 10 ==== osd_op_reply(79 [pgnls-filter start_epoch 122 out=49b] v120'20 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8500 con 0x55e2edfed400 2024-03-24T00:49:23.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.644+0000 7ff1c68dc640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:80 18.b 18:d0000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2ee008800 con 0x55e2edfec000 2024-03-24T00:49:23.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.644+0000 7ff1cb0e5640 1 -- 172.21.15.67:0/724085066 <== osd.5 v2:172.21.15.72:6800/288486206 10 ==== osd_op_reply(80 [pgnls-filter start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8000 con 0x55e2edfec000 2024-03-24T00:49:23.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.644+0000 7ff1c70dd640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:81 18.2b 18:d4000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edffd800 con 0x55e2eeb6f000 2024-03-24T00:49:23.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.645+0000 7ff1cb0e5640 1 -- 172.21.15.67:0/724085066 <== osd.7 v2:172.21.15.72:6824/2893038746 8 ==== osd_op_reply(81 [pgnls-filter start_epoch 122 out=49b] v120'22 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8000 con 0x55e2eeb6f000 2024-03-24T00:49:23.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.645+0000 7ff1c68dc640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:82 18.1b 18:d8000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2ee008c00 con 0x55e2edfed400 2024-03-24T00:49:23.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.645+0000 7ff1cc0e7640 1 -- 172.21.15.67:0/724085066 <== osd.1 v2:172.21.15.67:6800/4062510404 11 ==== osd_op_reply(82 [pgnls-filter start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8500 con 0x55e2edfed400 2024-03-24T00:49:23.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.645+0000 7ff1c70dd640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:83 18.3b 18:dc000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edffc800 con 0x55e2edffdc00 2024-03-24T00:49:23.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.645+0000 7ff1cc0e7640 1 -- 172.21.15.67:0/724085066 <== osd.4 v2:172.21.15.72:6808/477220793 6 ==== osd_op_reply(83 [pgnls-filter start_epoch 122 out=49b] v120'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8500 con 0x55e2edffdc00 2024-03-24T00:49:23.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.645+0000 7ff1c68dc640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] -- osd_op(unknown.0.0:84 18.7 18:e0000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2ee009400 con 0x55e2edfec400 2024-03-24T00:49:23.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.646+0000 7ff1cb8e6640 1 -- 172.21.15.67:0/724085066 <== osd.2 v2:172.21.15.67:6817/1268432561 7 ==== osd_op_reply(84 [pgnls-filter start_epoch 122 out=49b] v120'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8780 con 0x55e2edfec400 2024-03-24T00:49:23.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.646+0000 7ff1c70dd640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:85 18.27 18:e4000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2ee002800 con 0x55e2edfec000 2024-03-24T00:49:23.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.646+0000 7ff1cb0e5640 1 -- 172.21.15.67:0/724085066 <== osd.5 v2:172.21.15.72:6800/288486206 11 ==== osd_op_reply(85 [pgnls-filter start_epoch 122 out=49b] v120'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8000 con 0x55e2edfec000 2024-03-24T00:49:23.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.646+0000 7ff1c68dc640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:86 18.17 18:e8000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2ee008400 con 0x55e2edfec000 2024-03-24T00:49:23.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.646+0000 7ff1cb0e5640 1 -- 172.21.15.67:0/724085066 <== osd.5 v2:172.21.15.72:6800/288486206 12 ==== osd_op_reply(86 [pgnls-filter start_epoch 122 out=49b] v120'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8000 con 0x55e2edfec000 2024-03-24T00:49:23.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.646+0000 7ff1c70dd640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:87 18.37 18:ec000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2ee002000 con 0x55e2eeb6f000 2024-03-24T00:49:23.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.647+0000 7ff1cb0e5640 1 -- 172.21.15.67:0/724085066 <== osd.7 v2:172.21.15.72:6824/2893038746 9 ==== osd_op_reply(87 [pgnls-filter start_epoch 122 out=49b] v120'22 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8000 con 0x55e2eeb6f000 2024-03-24T00:49:23.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.647+0000 7ff1c68dc640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:88 18.f 18:f0000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edffc000 con 0x55e2edfec000 2024-03-24T00:49:23.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.647+0000 7ff1cb0e5640 1 -- 172.21.15.67:0/724085066 <== osd.5 v2:172.21.15.72:6800/288486206 13 ==== osd_op_reply(88 [pgnls-filter start_epoch 122 out=49b] v120'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8000 con 0x55e2edfec000 2024-03-24T00:49:23.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.647+0000 7ff1c70dd640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:89 18.2f 18:f4000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2ee008000 con 0x55e2edfed400 2024-03-24T00:49:23.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.647+0000 7ff1cc0e7640 1 -- 172.21.15.67:0/724085066 <== osd.1 v2:172.21.15.67:6800/4062510404 12 ==== osd_op_reply(89 [pgnls-filter start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8500 con 0x55e2edfed400 2024-03-24T00:49:23.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.647+0000 7ff1c68dc640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:90 18.1f 18:f8000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2edffcc00 con 0x55e2eeb6f000 2024-03-24T00:49:23.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.647+0000 7ff1cb0e5640 1 -- 172.21.15.67:0/724085066 <== osd.7 v2:172.21.15.72:6824/2893038746 10 ==== osd_op_reply(90 [pgnls-filter start_epoch 122 out=49b] v120'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8000 con 0x55e2eeb6f000 2024-03-24T00:49:23.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.647+0000 7ff1c70dd640 1 -- 172.21.15.67:0/724085066 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:91 18.3f 18:fc000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x55e2ee009000 con 0x55e2edffdc00 2024-03-24T00:49:23.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.648+0000 7ff1cc0e7640 1 -- 172.21.15.67:0/724085066 <== osd.4 v2:172.21.15.72:6808/477220793 7 ==== osd_op_reply(91 [pgnls-filter start_epoch 122 out=49b] v120'17 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55e2ee9d8500 con 0x55e2edffdc00 2024-03-24T00:49:23.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.648+0000 7ff1cdc0c740 1 -- 172.21.15.67:0/724085066 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x55e2edfc7800 msgr2=0x55e2edcfd600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.648+0000 7ff1cdc0c740 1 --2- 172.21.15.67:0/724085066 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x55e2edfc7800 0x55e2edcfd600 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.648+0000 7ff1cdc0c740 1 -- 172.21.15.67:0/724085066 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x55e2edfed400 msgr2=0x55e2edff2580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.648+0000 7ff1cdc0c740 1 --2- 172.21.15.67:0/724085066 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x55e2edfed400 0x55e2edff2580 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.648+0000 7ff1cdc0c740 1 -- 172.21.15.67:0/724085066 >> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] conn(0x55e2edfec400 msgr2=0x55e2edff2b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.648+0000 7ff1cdc0c740 1 --2- 172.21.15.67:0/724085066 >> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] conn(0x55e2edfec400 0x55e2edff2b00 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.648+0000 7ff1cdc0c740 1 -- 172.21.15.67:0/724085066 >> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] conn(0x55e2eeb6ec00 msgr2=0x55e2edcfe100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.648+0000 7ff1cdc0c740 1 --2- 172.21.15.67:0/724085066 >> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] conn(0x55e2eeb6ec00 0x55e2edcfe100 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.648+0000 7ff1cdc0c740 1 -- 172.21.15.67:0/724085066 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x55e2edffdc00 msgr2=0x55e2edff3600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.648+0000 7ff1cdc0c740 1 --2- 172.21.15.67:0/724085066 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x55e2edffdc00 0x55e2edff3600 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.648+0000 7ff1cdc0c740 1 -- 172.21.15.67:0/724085066 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x55e2edfec000 msgr2=0x55e2edff2000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.648+0000 7ff1cdc0c740 1 --2- 172.21.15.67:0/724085066 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x55e2edfec000 0x55e2edff2000 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.648+0000 7ff1cb8e6640 1 -- 172.21.15.67:0/724085066 reap_dead start 2024-03-24T00:49:23.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.648+0000 7ff1cdc0c740 1 -- 172.21.15.67:0/724085066 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x55e2edfc7000 msgr2=0x55e2edcfdb80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.648+0000 7ff1cdc0c740 1 --2- 172.21.15.67:0/724085066 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x55e2edfc7000 0x55e2edcfdb80 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.648+0000 7ff1cdc0c740 1 -- 172.21.15.67:0/724085066 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x55e2eeb6f000 msgr2=0x55e2edff3080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.648+0000 7ff1cdc0c740 1 --2- 172.21.15.67:0/724085066 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x55e2eeb6f000 0x55e2edff3080 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.648+0000 7ff1cb8e6640 1 -- 172.21.15.67:0/724085066 reap_dead start 2024-03-24T00:49:23.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.648+0000 7ff1cdc0c740 1 -- 172.21.15.67:0/724085066 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x55e2edfc6000 msgr2=0x55e2edcff700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.648+0000 7ff1cdc0c740 1 --2- 172.21.15.67:0/724085066 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x55e2edfc6000 0x55e2edcff700 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x55e2eea59b90 tx=0x55e2edfd2000 comp rx=0 tx=0).stop 2024-03-24T00:49:23.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.648+0000 7ff1cdc0c740 1 -- 172.21.15.67:0/724085066 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e2eeb6e000 msgr2=0x55e2edcff180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.648+0000 7ff1cdc0c740 1 --2- 172.21.15.67:0/724085066 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e2eeb6e000 0x55e2edcff180 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x55e2eeb0bc20 tx=0x55e2ee9ee960 comp rx=0 tx=0).stop 2024-03-24T00:49:23.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.648+0000 7ff1cdc0c740 1 -- 172.21.15.67:0/724085066 shutdown_connections 2024-03-24T00:49:23.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.648+0000 7ff1cdc0c740 1 --2- 172.21.15.67:0/724085066 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x55e2edfc6000 0x55e2edcff700 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.648+0000 7ff1cdc0c740 1 --2- 172.21.15.67:0/724085066 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e2eeb6e000 0x55e2edcff180 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.648+0000 7ff1cdc0c740 1 -- 172.21.15.67:0/724085066 >> 172.21.15.67:0/724085066 conn(0x55e2ee9edc00 msgr2=0x55e2edd3b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:23.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.648+0000 7ff1cdc0c740 1 -- 172.21.15.67:0/724085066 shutdown_connections 2024-03-24T00:49:23.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.648+0000 7ff1cdc0c740 1 -- 172.21.15.67:0/724085066 wait complete. 2024-03-24T00:49:23.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.648+0000 7ff1cdc0c740 1 -- 172.21.15.67:0/2878328922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e2edc53c00 msgr2=0x55e2edcfd080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.648+0000 7ff1cdc0c740 1 --2- 172.21.15.67:0/2878328922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e2edc53c00 0x55e2edcfd080 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x55e2eea59890 tx=0x55e2ee9ee840 comp rx=0 tx=0).stop 2024-03-24T00:49:23.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.649+0000 7ff1cdc0c740 1 -- 172.21.15.67:0/2878328922 shutdown_connections 2024-03-24T00:49:23.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.649+0000 7ff1cdc0c740 1 --2- 172.21.15.67:0/2878328922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e2edc53c00 0x55e2edcfd080 unknown :-1 s=CLOSED pgs=1842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.649+0000 7ff1cdc0c740 1 --2- 172.21.15.67:0/2878328922 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e2edc53800 0x55e2edcfcb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.649+0000 7ff1cdc0c740 1 --2- 172.21.15.67:0/2878328922 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x55e2edc52c00 0x55e2edcfc580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.649+0000 7ff1cdc0c740 1 -- 172.21.15.67:0/2878328922 >> 172.21.15.67:0/2878328922 conn(0x55e2edc53000 msgr2=0x55e2edd3a800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:23.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.649+0000 7ff1cdc0c740 1 -- 172.21.15.67:0/2878328922 shutdown_connections 2024-03-24T00:49:23.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.649+0000 7ff1cdc0c740 1 -- 172.21.15.67:0/2878328922 wait complete. 2024-03-24T00:49:23.665 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-data-scan', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'scan_inodes', '--filter-tag', 'mytag123', 'cephfs_data'] in time 0:00:00.130271, result: 2024-03-24T00:49:23.665 DEBUG:teuthology.orchestra.run.smithi067:> cephfs-data-scan --debug-mds=20 --debug-ms=1 --debug-objecter=1 scan_links 2024-03-24T00:49:23.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.735+0000 7f4304231740 1 -- start start 2024-03-24T00:49:23.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.735+0000 7f4304231740 1 Processor -- start 2024-03-24T00:49:23.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.736+0000 7f4304231740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56345257cc00 0x563452627b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.736+0000 7f4304231740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56345257d800 0x563452627600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.736+0000 7f4304231740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56345257dc00 0x563452627080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.736+0000 7f4304231740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x56345336c180 con 0x56345257dc00 2024-03-24T00:49:23.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.736+0000 7f4304231740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634525dc900 con 0x56345257cc00 2024-03-24T00:49:23.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.736+0000 7f4304231740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5634525dc480 con 0x56345257d800 2024-03-24T00:49:23.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.736+0000 7f43022ef640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56345257dc00 0x563452627080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.736+0000 7f43022ef640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56345257dc00 0x563452627080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38446/0 (socket says 172.21.15.67:38446) 2024-03-24T00:49:23.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.736+0000 7f43022ef640 1 -- 172.21.15.67:0/1739904856 learned_addr learned my addr 172.21.15.67:0/1739904856 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:23.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.736+0000 7f4301aee640 1 --2- 172.21.15.67:0/1739904856 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56345257cc00 0x563452627b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.736+0000 7f43012ed640 1 --2- 172.21.15.67:0/1739904856 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56345257d800 0x563452627600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.736+0000 7f43022ef640 1 -- 172.21.15.67:0/1739904856 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56345257d800 msgr2=0x563452627600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:23.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.736+0000 7f43022ef640 1 --2- 172.21.15.67:0/1739904856 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56345257d800 0x563452627600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.736+0000 7f43022ef640 1 -- 172.21.15.67:0/1739904856 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56345257cc00 msgr2=0x563452627b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:23.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.736+0000 7f43022ef640 1 --2- 172.21.15.67:0/1739904856 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56345257cc00 0x563452627b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.736+0000 7f43022ef640 1 -- 172.21.15.67:0/1739904856 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5634525a9680 con 0x56345257dc00 2024-03-24T00:49:23.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.736+0000 7f43012ed640 1 --2- 172.21.15.67:0/1739904856 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56345257d800 0x563452627600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:23.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.736+0000 7f43022ef640 1 --2- 172.21.15.67:0/1739904856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56345257dc00 0x563452627080 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x5634533a91a0 tx=0x56345331a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=966a8a2aff825743 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:23.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.737+0000 7f42ff2e9640 1 -- 172.21.15.67:0/1739904856 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5634525a29c0 con 0x56345257dc00 2024-03-24T00:49:23.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.737+0000 7f42ff2e9640 1 -- 172.21.15.67:0/1739904856 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5634525c61c0 con 0x56345257dc00 2024-03-24T00:49:23.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.737+0000 7f42ff2e9640 1 -- 172.21.15.67:0/1739904856 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5634525a2680 con 0x56345257dc00 2024-03-24T00:49:23.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.737+0000 7f4304231740 1 -- 172.21.15.67:0/1739904856 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5634525a9860 con 0x56345257dc00 2024-03-24T00:49:23.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.737+0000 7f4304231740 1 -- 172.21.15.67:0/1739904856 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x5634525a92c0 con 0x56345257dc00 2024-03-24T00:49:23.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.737+0000 7f4304231740 4 waiting for MDS map... 2024-03-24T00:49:23.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.737+0000 7f42ff2e9640 1 -- 172.21.15.67:0/1739904856 <== mon.0 v2:172.21.15.67:3300/0 4 ==== osd_map(122..122 src has 1..122) v4 ==== 6273+0+0 (secure 0 0 0) 0x563453390600 con 0x56345257dc00 2024-03-24T00:49:23.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.737+0000 7f42ff2e9640 1 -- 172.21.15.67:0/1739904856 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 93) v1 ==== 2735+0+0 (secure 0 0 0) 0x5634525d5180 con 0x56345257dc00 2024-03-24T00:49:23.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.737+0000 7f4304231740 4 Got MDS map 93 2024-03-24T00:49:23.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.737+0000 7f4304231740 4 datascan.main: Using metadata pool output 2024-03-24T00:49:23.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.737+0000 7f4304231740 4 datascan.main: connecting to RADOS... 2024-03-24T00:49:23.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.738+0000 7f4304231740 1 Processor -- start 2024-03-24T00:49:23.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.738+0000 7f4304231740 1 -- start start 2024-03-24T00:49:23.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.738+0000 7f4304231740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5634534ac000 0x563452626b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.738+0000 7f4304231740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5634534ac400 0x563452626580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.738+0000 7f4304231740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5634534ac800 0x563452628100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.738+0000 7f4304231740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634525dc180 con 0x5634534ac800 2024-03-24T00:49:23.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.738+0000 7f4304231740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x56345336cd80 con 0x5634534ac000 2024-03-24T00:49:23.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.738+0000 7f4304231740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x56345336cf00 con 0x5634534ac400 2024-03-24T00:49:23.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.738+0000 7f43012ed640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5634534ac800 0x563452628100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.739+0000 7f43012ed640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5634534ac800 0x563452628100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38456/0 (socket says 172.21.15.67:38456) 2024-03-24T00:49:23.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.739+0000 7f43012ed640 1 -- 172.21.15.67:0/37249474 learned_addr learned my addr 172.21.15.67:0/37249474 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:23.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.739+0000 7f4301aee640 1 --2- 172.21.15.67:0/37249474 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5634534ac400 0x563452626580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.739+0000 7f43012ed640 1 --2- 172.21.15.67:0/37249474 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5634534ac000 0x563452626b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.739+0000 7f43012ed640 1 -- 172.21.15.67:0/37249474 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5634534ac400 msgr2=0x563452626580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:23.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.739+0000 7f43012ed640 1 --2- 172.21.15.67:0/37249474 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5634534ac400 0x563452626580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.739+0000 7f43012ed640 1 -- 172.21.15.67:0/37249474 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5634534ac000 msgr2=0x563452626b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:23.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.739+0000 7f43012ed640 1 --2- 172.21.15.67:0/37249474 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5634534ac000 0x563452626b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.739+0000 7f43012ed640 1 -- 172.21.15.67:0/37249474 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5634525a94a0 con 0x5634534ac800 2024-03-24T00:49:23.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.739+0000 7f43012ed640 1 --2- 172.21.15.67:0/37249474 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5634534ac800 0x563452628100 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x563453454810 tx=0x56345345acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f924903a3a63721b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:23.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.739+0000 7f42fd2e5640 1 -- 172.21.15.67:0/37249474 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5634525a29c0 con 0x5634534ac800 2024-03-24T00:49:23.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.739+0000 7f42fd2e5640 1 -- 172.21.15.67:0/37249474 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5634525c6c40 con 0x5634534ac800 2024-03-24T00:49:23.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.739+0000 7f42fd2e5640 1 -- 172.21.15.67:0/37249474 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5634525a2680 con 0x5634534ac800 2024-03-24T00:49:23.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.740+0000 7f4304231740 1 -- 172.21.15.67:0/37249474 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5634534ac800 msgr2=0x563452628100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.740+0000 7f4304231740 1 --2- 172.21.15.67:0/37249474 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5634534ac800 0x563452628100 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x563453454810 tx=0x56345345acc0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.740+0000 7f4304231740 1 -- 172.21.15.67:0/37249474 shutdown_connections 2024-03-24T00:49:23.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.740+0000 7f4304231740 1 --2- 172.21.15.67:0/37249474 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5634534ac800 0x563452628100 unknown :-1 s=CLOSED pgs=1847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.740+0000 7f4304231740 1 --2- 172.21.15.67:0/37249474 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5634534ac400 0x563452626580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.740+0000 7f4304231740 1 --2- 172.21.15.67:0/37249474 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5634534ac000 0x563452626b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.740+0000 7f4304231740 1 -- 172.21.15.67:0/37249474 >> 172.21.15.67:0/37249474 conn(0x56345336fc00 msgr2=0x563453305000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:23.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.740+0000 7f4304231740 1 -- 172.21.15.67:0/37249474 shutdown_connections 2024-03-24T00:49:23.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.740+0000 7f4304231740 1 -- 172.21.15.67:0/37249474 wait complete. 2024-03-24T00:49:23.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.740+0000 7f4304231740 1 Processor -- start 2024-03-24T00:49:23.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.740+0000 7f4304231740 1 -- start start 2024-03-24T00:49:23.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.741+0000 7f4304231740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5634534ac000 0x563452626b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.741+0000 7f4304231740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5634534ac400 0x563452626580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.741+0000 7f4304231740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5634534ac800 0x563452628100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.741+0000 7f4304231740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x56345336d200 con 0x5634534ac800 2024-03-24T00:49:23.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.741+0000 7f4304231740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x56345336d080 con 0x5634534ac400 2024-03-24T00:49:23.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.741+0000 7f4304231740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x56345336d380 con 0x5634534ac000 2024-03-24T00:49:23.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.741+0000 7f43012ed640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5634534ac800 0x563452628100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.741+0000 7f43012ed640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5634534ac800 0x563452628100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38464/0 (socket says 172.21.15.67:38464) 2024-03-24T00:49:23.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.741+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 learned_addr learned my addr 172.21.15.67:0/3638267307 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:23.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.741+0000 7f4301aee640 1 --2- 172.21.15.67:0/3638267307 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5634534ac400 0x563452626580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.741+0000 7f43012ed640 1 --2- 172.21.15.67:0/3638267307 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5634534ac000 0x563452626b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.741+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5634534ac000 msgr2=0x563452626b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.741+0000 7f43012ed640 1 --2- 172.21.15.67:0/3638267307 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5634534ac000 0x563452626b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.741+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5634534ac400 msgr2=0x563452626580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.741+0000 7f43012ed640 1 --2- 172.21.15.67:0/3638267307 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5634534ac400 0x563452626580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.741+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5634525a90e0 con 0x5634534ac800 2024-03-24T00:49:23.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.741+0000 7f43012ed640 1 --2- 172.21.15.67:0/3638267307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5634534ac800 0x563452628100 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x563453455ad0 tx=0x5634533187e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:23.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.742+0000 7f42fbae2640 1 -- 172.21.15.67:0/3638267307 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5634525a24e0 con 0x5634534ac800 2024-03-24T00:49:23.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.742+0000 7f42fbae2640 1 -- 172.21.15.67:0/3638267307 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5634525c6540 con 0x5634534ac800 2024-03-24T00:49:23.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.742+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5634525a9a40 con 0x5634534ac800 2024-03-24T00:49:23.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.742+0000 7f42fbae2640 1 -- 172.21.15.67:0/3638267307 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5634525a29c0 con 0x5634534ac800 2024-03-24T00:49:23.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.742+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5634525a83c0 con 0x5634534ac800 2024-03-24T00:49:23.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.742+0000 7f4304231740 4 datascan.init: resolving metadata pool 17 2024-03-24T00:49:23.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.743+0000 7f42fbae2640 1 -- 172.21.15.67:0/3638267307 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x5634534c4c00 con 0x5634534ac800 2024-03-24T00:49:23.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.743+0000 7f42fbae2640 1 --2- 172.21.15.67:0/3638267307 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x5634528fa000 0x5634528fc000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.743+0000 7f42fbae2640 1 -- 172.21.15.67:0/3638267307 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6273+0+0 (secure 0 0 0) 0x5634534c2a00 con 0x5634534ac800 2024-03-24T00:49:23.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.743+0000 7f4304231740 4 datascan.init: found metadata pool 'cephfs_metadata' 2024-03-24T00:49:23.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.743+0000 7f4304231740 4 datascan.main: resolving metadata pool 17 2024-03-24T00:49:23.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.743+0000 7f4304231740 1 --2- 172.21.15.67:0/3638267307 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x5634534acc00 0x563452628680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.743+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:1 17.0 17:00000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5634534ad000 con 0x5634534acc00 2024-03-24T00:49:23.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.744+0000 7f4301aee640 1 --2- 172.21.15.67:0/3638267307 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x5634528fa000 0x5634528fc000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.744+0000 7f43012ed640 1 --2- 172.21.15.67:0/3638267307 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x5634534acc00 0x563452628680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.744+0000 7f43012ed640 1 --2- 172.21.15.67:0/3638267307 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x5634534acc00 0x563452628680 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:23.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.744+0000 7f4301aee640 1 --2- 172.21.15.67:0/3638267307 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x5634528fa000 0x5634528fc000 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x56345331ced0 tx=0x56345288cde0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:23.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.744+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 <== osd.7 v2:172.21.15.72:6824/2893038746 1 ==== osd_op_reply(1 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5634526f4000 con 0x5634534acc00 2024-03-24T00:49:23.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.744+0000 7f43012ed640 1 --2- 172.21.15.67:0/3638267307 >> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] conn(0x5634534ad000 0x5634528fc580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.744+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:2 17.10 17:08000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5634528fb000 con 0x5634534ad000 2024-03-24T00:49:23.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.744+0000 7f43022ef640 1 --2- 172.21.15.67:0/3638267307 >> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] conn(0x5634534ad000 0x5634528fc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.745+0000 7f43022ef640 1 --2- 172.21.15.67:0/3638267307 >> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] conn(0x5634534ad000 0x5634528fc580 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:23.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.745+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 <== osd.3 v2:172.21.15.67:6816/3303489481 1 ==== osd_op_reply(2 [pgnls start_epoch 122 out=177b] v121'16 uv16 ondisk = 1) v8 ==== 144+0+177 (crc 0 0 0) 0x5634526f4500 con 0x5634534ad000 2024-03-24T00:49:23.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.745+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:3 17.8 17:10000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5634528fb000 con 0x5634534acc00 2024-03-24T00:49:23.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.745+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 <== osd.7 v2:172.21.15.72:6824/2893038746 2 ==== osd_op_reply(3 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5634526f4000 con 0x5634534acc00 2024-03-24T00:49:23.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.745+0000 7f43012ed640 1 --2- 172.21.15.67:0/3638267307 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x5634528fb000 0x5634528fcb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.745+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:4 17.18 17:18000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5634528fb400 con 0x5634528fb000 2024-03-24T00:49:23.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.745+0000 7f4301aee640 1 --2- 172.21.15.67:0/3638267307 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x5634528fb000 0x5634528fcb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.746+0000 7f4301aee640 1 --2- 172.21.15.67:0/3638267307 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x5634528fb000 0x5634528fcb00 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:23.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.746+0000 7f4301aee640 1 -- 172.21.15.67:0/3638267307 <== osd.0 v2:172.21.15.67:6808/2628130519 1 ==== osd_op_reply(4 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5634526f4780 con 0x5634528fb000 2024-03-24T00:49:23.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.746+0000 7f4301aee640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:5 17.4 17:20000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5634528fb400 con 0x5634528fb000 2024-03-24T00:49:23.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.746+0000 7f4301aee640 1 -- 172.21.15.67:0/3638267307 <== osd.0 v2:172.21.15.67:6808/2628130519 2 ==== osd_op_reply(5 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5634526f4780 con 0x5634528fb000 2024-03-24T00:49:23.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.746+0000 7f4301aee640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:6 17.14 17:28000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5634528fb400 con 0x5634534ad000 2024-03-24T00:49:23.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.746+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 <== osd.3 v2:172.21.15.67:6816/3303489481 2 ==== osd_op_reply(6 [pgnls start_epoch 122 out=149b] v122'43 uv43 ondisk = 1) v8 ==== 144+0+149 (crc 0 0 0) 0x5634526f4500 con 0x5634534ad000 2024-03-24T00:49:23.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.746+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:7 17.c 17:30000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5634528fb400 con 0x5634534acc00 2024-03-24T00:49:23.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.747+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 <== osd.7 v2:172.21.15.72:6824/2893038746 3 ==== osd_op_reply(7 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5634526f4000 con 0x5634534acc00 2024-03-24T00:49:23.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.747+0000 7f43012ed640 1 --2- 172.21.15.67:0/3638267307 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x5634528fb400 0x5634528fd080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.747+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:8 17.1c 17:38000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5634528fb800 con 0x5634528fb400 2024-03-24T00:49:23.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.747+0000 7f43012ed640 1 --2- 172.21.15.67:0/3638267307 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x5634528fb400 0x5634528fd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.747+0000 7f43012ed640 1 --2- 172.21.15.67:0/3638267307 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x5634528fb400 0x5634528fd080 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:23.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.747+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 <== osd.1 v2:172.21.15.67:6800/4062510404 1 ==== osd_op_reply(8 [pgnls start_epoch 122 out=73b] v121'3 uv3 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x5634526f4000 con 0x5634528fb400 2024-03-24T00:49:23.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.747+0000 7f43012ed640 1 --2- 172.21.15.67:0/3638267307 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x5634528fb800 0x5634528fd600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.747+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:9 17.2 17:40000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5634528fbc00 con 0x5634528fb800 2024-03-24T00:49:23.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.748+0000 7f43022ef640 1 --2- 172.21.15.67:0/3638267307 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x5634528fb800 0x5634528fd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.748+0000 7f43022ef640 1 --2- 172.21.15.67:0/3638267307 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x5634528fb800 0x5634528fd600 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:23.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.748+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 <== osd.4 v2:172.21.15.72:6808/477220793 1 ==== osd_op_reply(9 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5634526f4500 con 0x5634528fb800 2024-03-24T00:49:23.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.748+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:10 17.12 17:48000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5634528fbc00 con 0x5634534ad000 2024-03-24T00:49:23.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.749+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 <== osd.3 v2:172.21.15.67:6816/3303489481 3 ==== osd_op_reply(10 [pgnls start_epoch 122 out=81b] v121'3 uv3 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x5634526f4500 con 0x5634534ad000 2024-03-24T00:49:23.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.749+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:11 17.a 17:50000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5634528fbc00 con 0x5634534acc00 2024-03-24T00:49:23.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.749+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 <== osd.7 v2:172.21.15.72:6824/2893038746 4 ==== osd_op_reply(11 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5634526f4000 con 0x5634534acc00 2024-03-24T00:49:23.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.749+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:12 17.1a 17:58000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5634528fbc00 con 0x5634528fb000 2024-03-24T00:49:23.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.749+0000 7f4301aee640 1 -- 172.21.15.67:0/3638267307 <== osd.0 v2:172.21.15.67:6808/2628130519 3 ==== osd_op_reply(12 [pgnls start_epoch 122 out=49b] v122'3 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5634526f4780 con 0x5634528fb000 2024-03-24T00:49:23.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.749+0000 7f4301aee640 1 --2- 172.21.15.67:0/3638267307 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x5634528fbc00 0x5634528fe680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.749+0000 7f4301aee640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:13 17.6 17:60000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x5634528fac00 con 0x5634528fbc00 2024-03-24T00:49:23.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.749+0000 7f4301aee640 1 --2- 172.21.15.67:0/3638267307 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x5634528fbc00 0x5634528fe680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.750+0000 7f4301aee640 1 --2- 172.21.15.67:0/3638267307 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x5634528fbc00 0x5634528fe680 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:23.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.750+0000 7f4301aee640 1 -- 172.21.15.67:0/3638267307 <== osd.6 v2:172.21.15.72:6816/1725206562 1 ==== osd_op_reply(13 [pgnls start_epoch 122 out=77b] v121'9 uv9 ondisk = 1) v8 ==== 144+0+77 (crc 0 0 0) 0x5634526f4780 con 0x5634528fbc00 2024-03-24T00:49:23.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.750+0000 7f4301aee640 1 --2- 172.21.15.67:0/3638267307 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x5634528fac00 0x5634528fec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.750+0000 7f4301aee640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:14 17.16 17:68000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452922000 con 0x5634528fac00 2024-03-24T00:49:23.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.750+0000 7f43012ed640 1 --2- 172.21.15.67:0/3638267307 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x5634528fac00 0x5634528fec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.750+0000 7f43012ed640 1 --2- 172.21.15.67:0/3638267307 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x5634528fac00 0x5634528fec00 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:23.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.751+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 <== osd.5 v2:172.21.15.72:6800/288486206 1 ==== osd_op_reply(14 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5634526f4000 con 0x5634528fac00 2024-03-24T00:49:23.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.751+0000 7f43012ed640 1 --2- 172.21.15.67:0/3638267307 >> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] conn(0x563452922000 0x5634528fdb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:23.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.751+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] -- osd_op(unknown.0.0:15 17.e 17:70000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452922c00 con 0x563452922000 2024-03-24T00:49:23.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.751+0000 7f43022ef640 1 --2- 172.21.15.67:0/3638267307 >> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] conn(0x563452922000 0x5634528fdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:23.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.751+0000 7f43022ef640 1 --2- 172.21.15.67:0/3638267307 >> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] conn(0x563452922000 0x5634528fdb80 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:23.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.751+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 <== osd.2 v2:172.21.15.67:6817/1268432561 1 ==== osd_op_reply(15 [pgnls start_epoch 122 out=73b] v121'3 uv3 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x5634526f4500 con 0x563452922000 2024-03-24T00:49:23.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.751+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:16 17.1e 17:78000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452922c00 con 0x5634528fb400 2024-03-24T00:49:23.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.752+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 <== osd.1 v2:172.21.15.67:6800/4062510404 2 ==== osd_op_reply(16 [pgnls start_epoch 122 out=49b] v122'3 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5634526f4000 con 0x5634528fb400 2024-03-24T00:49:23.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.752+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:17 17.1 17:80000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452922c00 con 0x5634528fb000 2024-03-24T00:49:23.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.752+0000 7f4301aee640 1 -- 172.21.15.67:0/3638267307 <== osd.0 v2:172.21.15.67:6808/2628130519 4 ==== osd_op_reply(17 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5634526f4780 con 0x5634528fb000 2024-03-24T00:49:23.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.752+0000 7f4301aee640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:18 17.11 17:88000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452922c00 con 0x5634534ad000 2024-03-24T00:49:23.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.752+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 <== osd.3 v2:172.21.15.67:6816/3303489481 4 ==== osd_op_reply(18 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5634526f4500 con 0x5634534ad000 2024-03-24T00:49:23.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.752+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:19 17.9 17:90000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452922c00 con 0x5634534ad000 2024-03-24T00:49:23.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.752+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 <== osd.3 v2:172.21.15.67:6816/3303489481 5 ==== osd_op_reply(19 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5634526f4500 con 0x5634534ad000 2024-03-24T00:49:23.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.752+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:20 17.19 17:98000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452922c00 con 0x5634528fac00 2024-03-24T00:49:23.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.753+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 <== osd.5 v2:172.21.15.72:6800/288486206 2 ==== osd_op_reply(20 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5634526f4000 con 0x5634528fac00 2024-03-24T00:49:23.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.753+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:21 17.5 17:a0000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452922c00 con 0x5634528fbc00 2024-03-24T00:49:23.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.753+0000 7f4301aee640 1 -- 172.21.15.67:0/3638267307 <== osd.6 v2:172.21.15.72:6816/1725206562 2 ==== osd_op_reply(21 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5634526f4780 con 0x5634528fbc00 2024-03-24T00:49:23.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.753+0000 7f4301aee640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:22 17.15 17:a8000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452922c00 con 0x5634528fac00 2024-03-24T00:49:23.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.753+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 <== osd.5 v2:172.21.15.72:6800/288486206 3 ==== osd_op_reply(22 [pgnls start_epoch 122 out=97b] v121'4 uv4 ondisk = 1) v8 ==== 144+0+97 (crc 0 0 0) 0x5634526f4000 con 0x5634528fac00 2024-03-24T00:49:23.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.753+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:23 17.d 17:b0000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452922c00 con 0x5634534acc00 2024-03-24T00:49:23.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.754+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 <== osd.7 v2:172.21.15.72:6824/2893038746 5 ==== osd_op_reply(23 [pgnls start_epoch 122 out=74b] v119'1 uv1 ondisk = 1) v8 ==== 144+0+74 (crc 0 0 0) 0x5634526f4000 con 0x5634534acc00 2024-03-24T00:49:23.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.754+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:24 17.1d 17:b8000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452922c00 con 0x5634534acc00 2024-03-24T00:49:23.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.754+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 <== osd.7 v2:172.21.15.72:6824/2893038746 6 ==== osd_op_reply(24 [pgnls start_epoch 122 out=73b] v121'3 uv3 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x5634526f4000 con 0x5634534acc00 2024-03-24T00:49:23.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.754+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:25 17.3 17:c0000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452922c00 con 0x5634528fb800 2024-03-24T00:49:23.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.754+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 <== osd.4 v2:172.21.15.72:6808/477220793 2 ==== osd_op_reply(25 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5634526f4500 con 0x5634528fb800 2024-03-24T00:49:23.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.754+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:26 17.13 17:c8000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452922c00 con 0x5634534ad000 2024-03-24T00:49:23.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.755+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 <== osd.3 v2:172.21.15.67:6816/3303489481 6 ==== osd_op_reply(26 [pgnls start_epoch 122 out=98b] v121'5 uv5 ondisk = 1) v8 ==== 144+0+98 (crc 0 0 0) 0x5634526f4500 con 0x5634534ad000 2024-03-24T00:49:23.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.755+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:27 17.b 17:d0000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452922c00 con 0x5634528fb000 2024-03-24T00:49:23.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.755+0000 7f4301aee640 1 -- 172.21.15.67:0/3638267307 <== osd.0 v2:172.21.15.67:6808/2628130519 5 ==== osd_op_reply(27 [pgnls start_epoch 122 out=76b] v122'6 uv5 ondisk = 1) v8 ==== 144+0+76 (crc 0 0 0) 0x5634526f4780 con 0x5634528fb000 2024-03-24T00:49:23.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.755+0000 7f4301aee640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:28 17.1b 17:d8000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452922c00 con 0x5634528fb800 2024-03-24T00:49:23.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.755+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 <== osd.4 v2:172.21.15.72:6808/477220793 3 ==== osd_op_reply(28 [pgnls start_epoch 122 out=49b] v122'3 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5634526f4500 con 0x5634528fb800 2024-03-24T00:49:23.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.755+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:29 17.7 17:e0000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452922c00 con 0x5634528fb000 2024-03-24T00:49:23.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.756+0000 7f4301aee640 1 -- 172.21.15.67:0/3638267307 <== osd.0 v2:172.21.15.67:6808/2628130519 6 ==== osd_op_reply(29 [pgnls start_epoch 122 out=105b] v122'9 uv9 ondisk = 1) v8 ==== 144+0+105 (crc 0 0 0) 0x5634526f4780 con 0x5634528fb000 2024-03-24T00:49:23.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.756+0000 7f4301aee640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:30 17.17 17:e8000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452922c00 con 0x5634528fbc00 2024-03-24T00:49:23.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.756+0000 7f4301aee640 1 -- 172.21.15.67:0/3638267307 <== osd.6 v2:172.21.15.72:6816/1725206562 3 ==== osd_op_reply(30 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5634526f4780 con 0x5634528fbc00 2024-03-24T00:49:23.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.756+0000 7f4301aee640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:31 17.f 17:f0000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452922c00 con 0x5634528fb400 2024-03-24T00:49:23.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.756+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 <== osd.1 v2:172.21.15.67:6800/4062510404 3 ==== osd_op_reply(31 [pgnls start_epoch 122 out=73b] v122'2 uv2 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x5634526f4000 con 0x5634528fb400 2024-03-24T00:49:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.756+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:32 17.1f 17:f8000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452922c00 con 0x5634528fbc00 2024-03-24T00:49:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.756+0000 7f4301aee640 1 -- 172.21.15.67:0/3638267307 <== osd.6 v2:172.21.15.72:6816/1725206562 4 ==== osd_op_reply(32 [pgnls start_epoch 122 out=101b] v122'9 uv8 ondisk = 1) v8 ==== 144+0+101 (crc 0 0 0) 0x5634526f4780 con 0x5634528fbc00 2024-03-24T00:49:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.756+0000 7f4304231740 10 datascan.scan_links: step 1: handling object 601.00000000 2024-03-24T00:49:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.756+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:33 17.10 17:09401a81:::601.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452928000 con 0x5634534ad000 2024-03-24T00:49:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.757+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 <== osd.3 v2:172.21.15.67:6816/3303489481 7 ==== osd_op_reply(33 601.00000000 [omap-get-vals out=5b] v0'0 uv13 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x5634526f4500 con 0x5634534ad000 2024-03-24T00:49:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.757+0000 7f4304231740 10 datascan.scan_links: step 1: handling object 602.00000000 2024-03-24T00:49:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.757+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:34 17.10 17:0bd6d154:::602.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452928400 con 0x5634534ad000 2024-03-24T00:49:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.757+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 <== osd.3 v2:172.21.15.67:6816/3303489481 8 ==== osd_op_reply(34 602.00000000 [omap-get-vals out=5b] v0'0 uv15 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x5634526f4500 con 0x5634534ad000 2024-03-24T00:49:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.757+0000 7f4304231740 10 datascan.scan_links: step 1: handling object 10000000000.00000000 2024-03-24T00:49:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.757+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:35 17.10 17:0cf6ad0f:::10000000000.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452928800 con 0x5634534ad000 2024-03-24T00:49:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.757+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 <== osd.3 v2:172.21.15.67:6816/3303489481 9 ==== osd_op_reply(35 10000000000.00000000 [omap-get-vals out=1169b] v0'0 uv12 ondisk = 0) v8 ==== 164+0+1169 (crc 0 0 0) 0x5634526f4500 con 0x5634534ad000 2024-03-24T00:49:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.757+0000 7f4304231740 10 datascan.scan_links: step 1: handling object 600.00000000 2024-03-24T00:49:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.757+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:36 17.10 17:0d82a743:::600.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452928c00 con 0x5634534ad000 2024-03-24T00:49:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.758+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 <== osd.3 v2:172.21.15.67:6816/3303489481 10 ==== osd_op_reply(36 600.00000000 [omap-get-vals out=5b] v0'0 uv14 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x5634526f4500 con 0x5634534ad000 2024-03-24T00:49:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.758+0000 7f4304231740 10 datascan.scan_links: step 1: handling object 603.00000000 2024-03-24T00:49:23.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.758+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:37 17.10 17:0d89b25e:::603.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452929000 con 0x5634534ad000 2024-03-24T00:49:23.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.758+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 <== osd.3 v2:172.21.15.67:6816/3303489481 11 ==== osd_op_reply(37 603.00000000 [omap-get-vals out=5b] v0'0 uv16 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x5634526f4500 con 0x5634534ad000 2024-03-24T00:49:23.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.758+0000 7f4304231740 10 datascan.scan_links: step 1: handling object 1.00000000.inode 2024-03-24T00:49:23.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.758+0000 7f4304231740 10 datascan.scan_links: Not a dirfrag: '1.00000000.inode' 2024-03-24T00:49:23.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.758+0000 7f4304231740 10 datascan.scan_links: step 1: handling object 200.00000000 2024-03-24T00:49:23.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.758+0000 7f4304231740 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000000' 2024-03-24T00:49:23.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.758+0000 7f4304231740 10 datascan.scan_links: step 1: handling object 200.00000001 2024-03-24T00:49:23.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.758+0000 7f4304231740 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000001' 2024-03-24T00:49:23.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.758+0000 7f4304231740 10 datascan.scan_links: step 1: handling object 606.00000000 2024-03-24T00:49:23.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.758+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:38 17.14 17:2f55791f:::606.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452929400 con 0x5634534ad000 2024-03-24T00:49:23.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.758+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 <== osd.3 v2:172.21.15.67:6816/3303489481 12 ==== osd_op_reply(38 606.00000000 [omap-get-vals out=5b] v0'0 uv38 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x5634526f4500 con 0x5634534ad000 2024-03-24T00:49:23.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.758+0000 7f4304231740 10 datascan.scan_links: step 1: handling object 607.00000000 2024-03-24T00:49:23.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.758+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:39 17.1c 17:3ed09add:::607.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452928000 con 0x5634528fb400 2024-03-24T00:49:23.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.758+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 <== osd.1 v2:172.21.15.67:6800/4062510404 4 ==== osd_op_reply(39 607.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x5634526f4000 con 0x5634528fb400 2024-03-24T00:49:23.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.758+0000 7f4304231740 10 datascan.scan_links: step 1: handling object 10000000003.00000000 2024-03-24T00:49:23.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.758+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:40 17.12 17:4def72d4:::10000000003.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452928400 con 0x5634534ad000 2024-03-24T00:49:23.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.759+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 <== osd.3 v2:172.21.15.67:6816/3303489481 13 ==== osd_op_reply(40 10000000003.00000000 [omap-get-vals out=584b] v0'0 uv3 ondisk = 0) v8 ==== 164+0+584 (crc 0 0 0) 0x5634526f4500 con 0x5634534ad000 2024-03-24T00:49:23.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.759+0000 7f4304231740 10 datascan.scan_links: step 1: handling object mds0_openfiles.0 2024-03-24T00:49:23.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.759+0000 7f4304231740 10 datascan.scan_links: Not a dirfrag: 'mds0_openfiles.0' 2024-03-24T00:49:23.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.759+0000 7f4304231740 10 datascan.scan_links: step 1: handling object 608.00000000 2024-03-24T00:49:23.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.759+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] -- osd_op(unknown.0.0:41 17.e 17:717a0223:::608.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452928800 con 0x563452922000 2024-03-24T00:49:23.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.759+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 <== osd.2 v2:172.21.15.67:6817/1268432561 2 ==== osd_op_reply(41 608.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x5634526f4500 con 0x563452922000 2024-03-24T00:49:23.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.759+0000 7f4304231740 10 datascan.scan_links: step 1: handling object 604.00000000 2024-03-24T00:49:23.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.759+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:42 17.15 17:a93a17c2:::604.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452928c00 con 0x5634528fac00 2024-03-24T00:49:23.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.759+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 <== osd.5 v2:172.21.15.72:6800/288486206 4 ==== osd_op_reply(42 604.00000000 [omap-get-vals out=5b] v0'0 uv4 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x5634526f4000 con 0x5634528fac00 2024-03-24T00:49:23.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.759+0000 7f4304231740 10 datascan.scan_links: step 1: handling object 500.00000000 2024-03-24T00:49:23.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.759+0000 7f4304231740 10 datascan.scan_links: Not a dirfrag (invalid ino): '500.00000000' 2024-03-24T00:49:23.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.759+0000 7f4304231740 10 datascan.scan_links: step 1: handling object mds_snaptable 2024-03-24T00:49:23.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.759+0000 7f4304231740 10 datascan.scan_links: Not a dirfrag: 'mds_snaptable' 2024-03-24T00:49:23.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.759+0000 7f4304231740 10 datascan.scan_links: step 1: handling object 605.00000000 2024-03-24T00:49:23.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.759+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:43 17.1d 17:b871830b:::605.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452929000 con 0x5634534acc00 2024-03-24T00:49:23.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.760+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 <== osd.7 v2:172.21.15.72:6824/2893038746 7 ==== osd_op_reply(43 605.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x5634526f4000 con 0x5634534acc00 2024-03-24T00:49:23.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.760+0000 7f4304231740 10 datascan.scan_links: step 1: handling object mds0_inotable 2024-03-24T00:49:23.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.760+0000 7f4304231740 10 datascan.scan_links: Not a dirfrag: 'mds0_inotable' 2024-03-24T00:49:23.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.760+0000 7f4304231740 10 datascan.scan_links: step 1: handling object 100.00000000 2024-03-24T00:49:23.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.760+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:44 17.13 17:cd5a64a3:::100.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452929800 con 0x5634534ad000 2024-03-24T00:49:23.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.760+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 <== osd.3 v2:172.21.15.67:6816/3303489481 14 ==== osd_op_reply(44 100.00000000 [omap-get-vals out=5195b] v0'0 uv5 ondisk = 0) v8 ==== 156+0+5195 (crc 0 0 0) 0x5634526f4500 con 0x5634534ad000 2024-03-24T00:49:23.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.760+0000 7f4304231740 10 datascan.scan_links: step 1: handling object mds0_sessionmap 2024-03-24T00:49:23.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.760+0000 7f4304231740 10 datascan.scan_links: Not a dirfrag: 'mds0_sessionmap' 2024-03-24T00:49:23.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.760+0000 7f4304231740 10 datascan.scan_links: step 1: handling object 609.00000000 2024-03-24T00:49:23.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.760+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:45 17.7 17:e0b41d06:::609.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452929c00 con 0x5634528fb000 2024-03-24T00:49:23.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.760+0000 7f4301aee640 1 -- 172.21.15.67:0/3638267307 <== osd.0 v2:172.21.15.67:6808/2628130519 7 ==== osd_op_reply(45 609.00000000 [omap-get-vals out=5b] v0'0 uv7 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x5634526f4780 con 0x5634528fb000 2024-03-24T00:49:23.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.760+0000 7f4304231740 10 datascan.scan_links: step 1: handling object 10000000001.00000000 2024-03-24T00:49:23.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.760+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:46 17.7 17:e2e07930:::10000000001.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452928000 con 0x5634528fb000 2024-03-24T00:49:23.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.761+0000 7f4301aee640 1 -- 172.21.15.67:0/3638267307 <== osd.0 v2:172.21.15.67:6808/2628130519 8 ==== osd_op_reply(46 10000000001.00000000 [omap-get-vals out=1667b] v0'0 uv9 ondisk = 0) v8 ==== 164+0+1667 (crc 0 0 0) 0x5634526f4780 con 0x5634528fb000 2024-03-24T00:49:23.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.761+0000 7f4304231740 20 datascan.scan_links: injected ino detected 2024-03-24T00:49:23.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.761+0000 7f4304231740 10 datascan.scan_links: step 1: handling object 400.00000000 2024-03-24T00:49:23.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.761+0000 7f4304231740 10 datascan.scan_links: Not a dirfrag (invalid ino): '400.00000000' 2024-03-24T00:49:23.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.761+0000 7f4304231740 10 datascan.scan_links: step 1: handling object 100.00000000.inode 2024-03-24T00:49:23.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.761+0000 7f4304231740 10 datascan.scan_links: Not a dirfrag: '100.00000000.inode' 2024-03-24T00:49:23.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.761+0000 7f4304231740 10 datascan.scan_links: step 1: handling object 1.00000000 2024-03-24T00:49:23.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.761+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:47 17.1f 17:ff5b34d6:::1.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452928c00 con 0x5634528fbc00 2024-03-24T00:49:23.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.761+0000 7f4301aee640 1 -- 172.21.15.67:0/3638267307 <== osd.6 v2:172.21.15.72:6816/1725206562 5 ==== osd_op_reply(47 1.00000000 [omap-get-vals out=585b] v0'0 uv8 ondisk = 0) v8 ==== 154+0+585 (crc 0 0 0) 0x5634526f4780 con 0x5634528fbc00 2024-03-24T00:49:23.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.761+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:48 17.0 17:00000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452929000 con 0x5634534acc00 2024-03-24T00:49:23.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.761+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 <== osd.7 v2:172.21.15.72:6824/2893038746 8 ==== osd_op_reply(48 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5634526f4000 con 0x5634534acc00 2024-03-24T00:49:23.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.762+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:49 17.10 17:08000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452929000 con 0x5634534ad000 2024-03-24T00:49:23.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.762+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 <== osd.3 v2:172.21.15.67:6816/3303489481 15 ==== osd_op_reply(49 [pgnls start_epoch 122 out=177b] v121'16 uv16 ondisk = 1) v8 ==== 144+0+177 (crc 0 0 0) 0x5634526f4500 con 0x5634534ad000 2024-03-24T00:49:23.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.762+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:50 17.8 17:10000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452929000 con 0x5634534acc00 2024-03-24T00:49:23.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.762+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 <== osd.7 v2:172.21.15.72:6824/2893038746 9 ==== osd_op_reply(50 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5634526f4000 con 0x5634534acc00 2024-03-24T00:49:23.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.762+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:51 17.18 17:18000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452929000 con 0x5634528fb000 2024-03-24T00:49:23.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.762+0000 7f4301aee640 1 -- 172.21.15.67:0/3638267307 <== osd.0 v2:172.21.15.67:6808/2628130519 9 ==== osd_op_reply(51 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5634526f4780 con 0x5634528fb000 2024-03-24T00:49:23.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.762+0000 7f4301aee640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:52 17.4 17:20000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452929000 con 0x5634528fb000 2024-03-24T00:49:23.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.762+0000 7f4301aee640 1 -- 172.21.15.67:0/3638267307 <== osd.0 v2:172.21.15.67:6808/2628130519 10 ==== osd_op_reply(52 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5634526f4780 con 0x5634528fb000 2024-03-24T00:49:23.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.762+0000 7f4301aee640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:53 17.14 17:28000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452929000 con 0x5634534ad000 2024-03-24T00:49:23.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.763+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 <== osd.3 v2:172.21.15.67:6816/3303489481 16 ==== osd_op_reply(53 [pgnls start_epoch 122 out=149b] v122'43 uv43 ondisk = 1) v8 ==== 144+0+149 (crc 0 0 0) 0x5634526f4500 con 0x5634534ad000 2024-03-24T00:49:23.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.763+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:54 17.c 17:30000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452929000 con 0x5634534acc00 2024-03-24T00:49:23.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.763+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 <== osd.7 v2:172.21.15.72:6824/2893038746 10 ==== osd_op_reply(54 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5634526f4000 con 0x5634534acc00 2024-03-24T00:49:23.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.763+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:55 17.1c 17:38000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452929000 con 0x5634528fb400 2024-03-24T00:49:23.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.763+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 <== osd.1 v2:172.21.15.67:6800/4062510404 5 ==== osd_op_reply(55 [pgnls start_epoch 122 out=73b] v121'3 uv3 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x5634526f4000 con 0x5634528fb400 2024-03-24T00:49:23.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.763+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:56 17.2 17:40000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452929000 con 0x5634528fb800 2024-03-24T00:49:23.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.764+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 <== osd.4 v2:172.21.15.72:6808/477220793 4 ==== osd_op_reply(56 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5634526f4500 con 0x5634528fb800 2024-03-24T00:49:23.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.764+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:57 17.12 17:48000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452929000 con 0x5634534ad000 2024-03-24T00:49:23.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.764+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 <== osd.3 v2:172.21.15.67:6816/3303489481 17 ==== osd_op_reply(57 [pgnls start_epoch 122 out=81b] v121'3 uv3 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x5634526f4500 con 0x5634534ad000 2024-03-24T00:49:23.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.764+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:58 17.a 17:50000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452929000 con 0x5634534acc00 2024-03-24T00:49:23.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.764+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 <== osd.7 v2:172.21.15.72:6824/2893038746 11 ==== osd_op_reply(58 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5634526f4000 con 0x5634534acc00 2024-03-24T00:49:23.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.764+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:59 17.1a 17:58000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452929000 con 0x5634528fb000 2024-03-24T00:49:23.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.765+0000 7f4301aee640 1 -- 172.21.15.67:0/3638267307 <== osd.0 v2:172.21.15.67:6808/2628130519 11 ==== osd_op_reply(59 [pgnls start_epoch 122 out=49b] v122'3 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5634526f4780 con 0x5634528fb000 2024-03-24T00:49:23.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.765+0000 7f4301aee640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:60 17.6 17:60000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452929000 con 0x5634528fbc00 2024-03-24T00:49:23.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.765+0000 7f4301aee640 1 -- 172.21.15.67:0/3638267307 <== osd.6 v2:172.21.15.72:6816/1725206562 6 ==== osd_op_reply(60 [pgnls start_epoch 122 out=77b] v121'9 uv9 ondisk = 1) v8 ==== 144+0+77 (crc 0 0 0) 0x5634526f4780 con 0x5634528fbc00 2024-03-24T00:49:23.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.765+0000 7f4301aee640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:61 17.16 17:68000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452929000 con 0x5634528fac00 2024-03-24T00:49:23.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.765+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 <== osd.5 v2:172.21.15.72:6800/288486206 5 ==== osd_op_reply(61 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5634526f4000 con 0x5634528fac00 2024-03-24T00:49:23.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.765+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] -- osd_op(unknown.0.0:62 17.e 17:70000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452929000 con 0x563452922000 2024-03-24T00:49:23.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.766+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 <== osd.2 v2:172.21.15.67:6817/1268432561 3 ==== osd_op_reply(62 [pgnls start_epoch 122 out=73b] v121'3 uv3 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x5634526f4500 con 0x563452922000 2024-03-24T00:49:23.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.766+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:63 17.1e 17:78000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452929000 con 0x5634528fb400 2024-03-24T00:49:23.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.766+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 <== osd.1 v2:172.21.15.67:6800/4062510404 6 ==== osd_op_reply(63 [pgnls start_epoch 122 out=49b] v122'3 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5634526f4000 con 0x5634528fb400 2024-03-24T00:49:23.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.766+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:64 17.1 17:80000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452929000 con 0x5634528fb000 2024-03-24T00:49:23.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.766+0000 7f4301aee640 1 -- 172.21.15.67:0/3638267307 <== osd.0 v2:172.21.15.67:6808/2628130519 12 ==== osd_op_reply(64 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5634526f4780 con 0x5634528fb000 2024-03-24T00:49:23.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.766+0000 7f4301aee640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:65 17.11 17:88000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452929000 con 0x5634534ad000 2024-03-24T00:49:23.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.766+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 <== osd.3 v2:172.21.15.67:6816/3303489481 18 ==== osd_op_reply(65 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5634526f4500 con 0x5634534ad000 2024-03-24T00:49:23.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.766+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:66 17.9 17:90000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452929000 con 0x5634534ad000 2024-03-24T00:49:23.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.766+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 <== osd.3 v2:172.21.15.67:6816/3303489481 19 ==== osd_op_reply(66 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5634526f4500 con 0x5634534ad000 2024-03-24T00:49:23.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.766+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:67 17.19 17:98000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452929000 con 0x5634528fac00 2024-03-24T00:49:23.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.767+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 <== osd.5 v2:172.21.15.72:6800/288486206 6 ==== osd_op_reply(67 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5634526f4000 con 0x5634528fac00 2024-03-24T00:49:23.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.767+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:68 17.5 17:a0000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452929000 con 0x5634528fbc00 2024-03-24T00:49:23.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.767+0000 7f4301aee640 1 -- 172.21.15.67:0/3638267307 <== osd.6 v2:172.21.15.72:6816/1725206562 7 ==== osd_op_reply(68 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5634526f4780 con 0x5634528fbc00 2024-03-24T00:49:23.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.767+0000 7f4301aee640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:69 17.15 17:a8000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452929000 con 0x5634528fac00 2024-03-24T00:49:23.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.768+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 <== osd.5 v2:172.21.15.72:6800/288486206 7 ==== osd_op_reply(69 [pgnls start_epoch 122 out=97b] v121'4 uv4 ondisk = 1) v8 ==== 144+0+97 (crc 0 0 0) 0x5634526f4000 con 0x5634528fac00 2024-03-24T00:49:23.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.768+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:70 17.d 17:b0000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452929000 con 0x5634534acc00 2024-03-24T00:49:23.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.768+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 <== osd.7 v2:172.21.15.72:6824/2893038746 12 ==== osd_op_reply(70 [pgnls start_epoch 122 out=74b] v119'1 uv1 ondisk = 1) v8 ==== 144+0+74 (crc 0 0 0) 0x5634526f4000 con 0x5634534acc00 2024-03-24T00:49:23.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.768+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:71 17.1d 17:b8000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452929000 con 0x5634534acc00 2024-03-24T00:49:23.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.768+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 <== osd.7 v2:172.21.15.72:6824/2893038746 13 ==== osd_op_reply(71 [pgnls start_epoch 122 out=73b] v121'3 uv3 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x5634526f4000 con 0x5634534acc00 2024-03-24T00:49:23.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.768+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:72 17.3 17:c0000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452929000 con 0x5634528fb800 2024-03-24T00:49:23.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.769+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 <== osd.4 v2:172.21.15.72:6808/477220793 5 ==== osd_op_reply(72 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5634526f4500 con 0x5634528fb800 2024-03-24T00:49:23.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.769+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:73 17.13 17:c8000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452929000 con 0x5634534ad000 2024-03-24T00:49:23.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.769+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 <== osd.3 v2:172.21.15.67:6816/3303489481 20 ==== osd_op_reply(73 [pgnls start_epoch 122 out=98b] v121'5 uv5 ondisk = 1) v8 ==== 144+0+98 (crc 0 0 0) 0x5634526f4500 con 0x5634534ad000 2024-03-24T00:49:23.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.769+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:74 17.b 17:d0000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452929000 con 0x5634528fb000 2024-03-24T00:49:23.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.769+0000 7f4301aee640 1 -- 172.21.15.67:0/3638267307 <== osd.0 v2:172.21.15.67:6808/2628130519 13 ==== osd_op_reply(74 [pgnls start_epoch 122 out=76b] v122'6 uv5 ondisk = 1) v8 ==== 144+0+76 (crc 0 0 0) 0x5634526f4780 con 0x5634528fb000 2024-03-24T00:49:23.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.769+0000 7f4301aee640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- osd_op(unknown.0.0:75 17.1b 17:d8000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452929000 con 0x5634528fb800 2024-03-24T00:49:23.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.769+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 <== osd.4 v2:172.21.15.72:6808/477220793 6 ==== osd_op_reply(75 [pgnls start_epoch 122 out=49b] v122'3 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5634526f4500 con 0x5634528fb800 2024-03-24T00:49:23.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.769+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:76 17.7 17:e0000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452929000 con 0x5634528fb000 2024-03-24T00:49:23.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.770+0000 7f4301aee640 1 -- 172.21.15.67:0/3638267307 <== osd.0 v2:172.21.15.67:6808/2628130519 14 ==== osd_op_reply(76 [pgnls start_epoch 122 out=105b] v122'9 uv9 ondisk = 1) v8 ==== 144+0+105 (crc 0 0 0) 0x5634526f4780 con 0x5634528fb000 2024-03-24T00:49:23.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.770+0000 7f4301aee640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:77 17.17 17:e8000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452929000 con 0x5634528fbc00 2024-03-24T00:49:23.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.770+0000 7f4301aee640 1 -- 172.21.15.67:0/3638267307 <== osd.6 v2:172.21.15.72:6816/1725206562 8 ==== osd_op_reply(77 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5634526f4780 con 0x5634528fbc00 2024-03-24T00:49:23.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.770+0000 7f4301aee640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:78 17.f 17:f0000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452929000 con 0x5634528fb400 2024-03-24T00:49:23.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.770+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 <== osd.1 v2:172.21.15.67:6800/4062510404 7 ==== osd_op_reply(78 [pgnls start_epoch 122 out=73b] v122'2 uv2 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x5634526f4000 con 0x5634528fb400 2024-03-24T00:49:23.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.770+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:79 17.1f 17:f8000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452929000 con 0x5634528fbc00 2024-03-24T00:49:23.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.771+0000 7f4301aee640 1 -- 172.21.15.67:0/3638267307 <== osd.6 v2:172.21.15.72:6816/1725206562 9 ==== osd_op_reply(79 [pgnls start_epoch 122 out=101b] v122'9 uv8 ondisk = 1) v8 ==== 144+0+101 (crc 0 0 0) 0x5634526f4780 con 0x5634528fbc00 2024-03-24T00:49:23.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.771+0000 7f4304231740 10 datascan.scan_links: step 2: handling object 601.00000000 2024-03-24T00:49:23.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.771+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:80 17.10 17:09401a81:::601.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452923c00 con 0x5634534ad000 2024-03-24T00:49:23.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.771+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 <== osd.3 v2:172.21.15.67:6816/3303489481 21 ==== osd_op_reply(80 601.00000000 [omap-get-vals out=5b] v0'0 uv13 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x5634526f4500 con 0x5634534ad000 2024-03-24T00:49:23.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.771+0000 7f4304231740 10 datascan.scan_links: step 2: handling object 602.00000000 2024-03-24T00:49:23.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.771+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:81 17.10 17:0bd6d154:::602.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452923800 con 0x5634534ad000 2024-03-24T00:49:23.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.771+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 <== osd.3 v2:172.21.15.67:6816/3303489481 22 ==== osd_op_reply(81 602.00000000 [omap-get-vals out=5b] v0'0 uv15 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x5634526f4500 con 0x5634534ad000 2024-03-24T00:49:23.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.771+0000 7f4304231740 10 datascan.scan_links: step 2: handling object 10000000000.00000000 2024-03-24T00:49:23.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.771+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:82 17.10 17:0cf6ad0f:::10000000000.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452923400 con 0x5634534ad000 2024-03-24T00:49:23.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.771+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 <== osd.3 v2:172.21.15.67:6816/3303489481 23 ==== osd_op_reply(82 10000000000.00000000 [omap-get-vals out=1169b] v0'0 uv12 ondisk = 0) v8 ==== 164+0+1169 (crc 0 0 0) 0x5634526f4500 con 0x5634534ad000 2024-03-24T00:49:23.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.771+0000 7f4304231740 10 datascan.scan_links: step 2: handling object 600.00000000 2024-03-24T00:49:23.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.771+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:83 17.10 17:0d82a743:::600.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452923000 con 0x5634534ad000 2024-03-24T00:49:23.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.772+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 <== osd.3 v2:172.21.15.67:6816/3303489481 24 ==== osd_op_reply(83 600.00000000 [omap-get-vals out=5b] v0'0 uv14 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x5634526f4500 con 0x5634534ad000 2024-03-24T00:49:23.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.772+0000 7f4304231740 10 datascan.scan_links: step 2: handling object 603.00000000 2024-03-24T00:49:23.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.772+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:84 17.10 17:0d89b25e:::603.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452929400 con 0x5634534ad000 2024-03-24T00:49:23.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.772+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 <== osd.3 v2:172.21.15.67:6816/3303489481 25 ==== osd_op_reply(84 603.00000000 [omap-get-vals out=5b] v0'0 uv16 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x5634526f4500 con 0x5634534ad000 2024-03-24T00:49:23.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.772+0000 7f4304231740 10 datascan.scan_links: step 2: handling object 1.00000000.inode 2024-03-24T00:49:23.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.772+0000 7f4304231740 10 datascan.scan_links: Not a dirfrag: '1.00000000.inode' 2024-03-24T00:49:23.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.772+0000 7f4304231740 10 datascan.scan_links: step 2: handling object 200.00000000 2024-03-24T00:49:23.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.772+0000 7f4304231740 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000000' 2024-03-24T00:49:23.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.772+0000 7f4304231740 10 datascan.scan_links: step 2: handling object 200.00000001 2024-03-24T00:49:23.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.772+0000 7f4304231740 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000001' 2024-03-24T00:49:23.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.772+0000 7f4304231740 10 datascan.scan_links: step 2: handling object 606.00000000 2024-03-24T00:49:23.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.772+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:85 17.14 17:2f55791f:::606.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452928400 con 0x5634534ad000 2024-03-24T00:49:23.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.772+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 <== osd.3 v2:172.21.15.67:6816/3303489481 26 ==== osd_op_reply(85 606.00000000 [omap-get-vals out=5b] v0'0 uv38 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x5634526f4500 con 0x5634534ad000 2024-03-24T00:49:23.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.772+0000 7f4304231740 10 datascan.scan_links: step 2: handling object 607.00000000 2024-03-24T00:49:23.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.772+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- osd_op(unknown.0.0:86 17.1c 17:3ed09add:::607.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452928800 con 0x5634528fb400 2024-03-24T00:49:23.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.773+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 <== osd.1 v2:172.21.15.67:6800/4062510404 8 ==== osd_op_reply(86 607.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x5634526f4000 con 0x5634528fb400 2024-03-24T00:49:23.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.773+0000 7f4304231740 10 datascan.scan_links: step 2: handling object 10000000003.00000000 2024-03-24T00:49:23.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.773+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:87 17.12 17:4def72d4:::10000000003.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452929800 con 0x5634534ad000 2024-03-24T00:49:23.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.773+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 <== osd.3 v2:172.21.15.67:6816/3303489481 27 ==== osd_op_reply(87 10000000003.00000000 [omap-get-vals out=584b] v0'0 uv3 ondisk = 0) v8 ==== 164+0+584 (crc 0 0 0) 0x5634526f4500 con 0x5634534ad000 2024-03-24T00:49:23.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.773+0000 7f4304231740 10 datascan.scan_links: step 2: handling object mds0_openfiles.0 2024-03-24T00:49:23.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.773+0000 7f4304231740 10 datascan.scan_links: Not a dirfrag: 'mds0_openfiles.0' 2024-03-24T00:49:23.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.773+0000 7f4304231740 10 datascan.scan_links: step 2: handling object 608.00000000 2024-03-24T00:49:23.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.773+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] -- osd_op(unknown.0.0:88 17.e 17:717a0223:::608.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452923c00 con 0x563452922000 2024-03-24T00:49:23.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.773+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 <== osd.2 v2:172.21.15.67:6817/1268432561 4 ==== osd_op_reply(88 608.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x5634526f4500 con 0x563452922000 2024-03-24T00:49:23.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.773+0000 7f4304231740 10 datascan.scan_links: step 2: handling object 604.00000000 2024-03-24T00:49:23.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.773+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- osd_op(unknown.0.0:89 17.15 17:a93a17c2:::604.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452923800 con 0x5634528fac00 2024-03-24T00:49:23.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.774+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 <== osd.5 v2:172.21.15.72:6800/288486206 8 ==== osd_op_reply(89 604.00000000 [omap-get-vals out=5b] v0'0 uv4 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x5634526f4000 con 0x5634528fac00 2024-03-24T00:49:23.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.774+0000 7f4304231740 10 datascan.scan_links: step 2: handling object 500.00000000 2024-03-24T00:49:23.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.774+0000 7f4304231740 10 datascan.scan_links: Not a dirfrag (invalid ino): '500.00000000' 2024-03-24T00:49:23.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.774+0000 7f4304231740 10 datascan.scan_links: step 2: handling object mds_snaptable 2024-03-24T00:49:23.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.774+0000 7f4304231740 10 datascan.scan_links: Not a dirfrag: 'mds_snaptable' 2024-03-24T00:49:23.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.774+0000 7f4304231740 10 datascan.scan_links: step 2: handling object 605.00000000 2024-03-24T00:49:23.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.774+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:90 17.1d 17:b871830b:::605.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452922800 con 0x5634534acc00 2024-03-24T00:49:23.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.774+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 <== osd.7 v2:172.21.15.72:6824/2893038746 14 ==== osd_op_reply(90 605.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x5634526f4000 con 0x5634534acc00 2024-03-24T00:49:23.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.774+0000 7f4304231740 10 datascan.scan_links: step 2: handling object mds0_inotable 2024-03-24T00:49:23.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.774+0000 7f4304231740 10 datascan.scan_links: Not a dirfrag: 'mds0_inotable' 2024-03-24T00:49:23.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.774+0000 7f4304231740 10 datascan.scan_links: step 2: handling object 100.00000000 2024-03-24T00:49:23.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.774+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:91 17.13 17:cd5a64a3:::100.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452922400 con 0x5634534ad000 2024-03-24T00:49:23.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.774+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 <== osd.3 v2:172.21.15.67:6816/3303489481 28 ==== osd_op_reply(91 100.00000000 [omap-get-vals out=5195b] v0'0 uv5 ondisk = 0) v8 ==== 156+0+5195 (crc 0 0 0) 0x5634526f4500 con 0x5634534ad000 2024-03-24T00:49:23.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.774+0000 7f4304231740 10 datascan.scan_links: step 2: handling object mds0_sessionmap 2024-03-24T00:49:23.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.774+0000 7f4304231740 10 datascan.scan_links: Not a dirfrag: 'mds0_sessionmap' 2024-03-24T00:49:23.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.774+0000 7f4304231740 10 datascan.scan_links: step 2: handling object 609.00000000 2024-03-24T00:49:23.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.774+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:92 17.7 17:e0b41d06:::609.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452922c00 con 0x5634528fb000 2024-03-24T00:49:23.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.775+0000 7f4301aee640 1 -- 172.21.15.67:0/3638267307 <== osd.0 v2:172.21.15.67:6808/2628130519 15 ==== osd_op_reply(92 609.00000000 [omap-get-vals out=5b] v0'0 uv7 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x5634526f4780 con 0x5634528fb000 2024-03-24T00:49:23.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.775+0000 7f4304231740 10 datascan.scan_links: step 2: handling object 10000000001.00000000 2024-03-24T00:49:23.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.775+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:93 17.7 17:e2e07930:::10000000001.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452929400 con 0x5634528fb000 2024-03-24T00:49:23.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.775+0000 7f4301aee640 1 -- 172.21.15.67:0/3638267307 <== osd.0 v2:172.21.15.67:6808/2628130519 16 ==== osd_op_reply(93 10000000001.00000000 [omap-get-vals out=1667b] v0'0 uv9 ondisk = 0) v8 ==== 164+0+1667 (crc 0 0 0) 0x5634526f4780 con 0x5634528fb000 2024-03-24T00:49:23.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.775+0000 7f4304231740 20 datascan.scan_links: injected ino detected 2024-03-24T00:49:23.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.775+0000 7f4304231740 20 datascan.scan_links: adding 0x100000001f7 for future processing to fix dnfirst 2024-03-24T00:49:23.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.775+0000 7f4304231740 10 datascan.scan_links: step 2: handling object 400.00000000 2024-03-24T00:49:23.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.775+0000 7f4304231740 10 datascan.scan_links: Not a dirfrag (invalid ino): '400.00000000' 2024-03-24T00:49:23.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.775+0000 7f4304231740 10 datascan.scan_links: step 2: handling object 100.00000000.inode 2024-03-24T00:49:23.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.775+0000 7f4304231740 10 datascan.scan_links: Not a dirfrag: '100.00000000.inode' 2024-03-24T00:49:23.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.775+0000 7f4304231740 10 datascan.scan_links: step 2: handling object 1.00000000 2024-03-24T00:49:23.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.775+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- osd_op(unknown.0.0:94 17.1f 17:ff5b34d6:::1.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452928400 con 0x5634528fbc00 2024-03-24T00:49:23.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.775+0000 7f4301aee640 1 -- 172.21.15.67:0/3638267307 <== osd.6 v2:172.21.15.72:6816/1725206562 10 ==== osd_op_reply(94 1.00000000 [omap-get-vals out=585b] v0'0 uv8 ondisk = 0) v8 ==== 154+0+585 (crc 0 0 0) 0x5634526f4780 con 0x5634528fbc00 2024-03-24T00:49:23.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.775+0000 7f4304231740 10 datascan.scan_links: processing 0 dup_primaries, 0 remote_links 2024-03-24T00:49:23.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.775+0000 7f4304231740 10 datascan.scan_links: removing dup dentries from 0 objects 2024-03-24T00:49:23.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.775+0000 7f4304231740 10 datascan.scan_links: processing 0 bad_nlink_inos 2024-03-24T00:49:23.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.775+0000 7f4304231740 10 datascan.scan_links: processing 1 injected_inos 2024-03-24T00:49:23.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.775+0000 7f4304231740 10 datascan.scan_links: handling injected_ino 0x100000001f7 2024-03-24T00:49:23.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.775+0000 7f4304231740 20 datascan.scan_links: fixing linkage (dnfirst) of 0x10000000001:bravo 2024-03-24T00:49:23.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.775+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:95 17.7 17:e2e07930:::10000000001.00000000:head [omap-get-vals-by-keys in=18b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452929800 con 0x5634528fb000 2024-03-24T00:49:23.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.776+0000 7f4301aee640 1 -- 172.21.15.67:0/3638267307 <== osd.0 v2:172.21.15.67:6808/2628130519 17 ==== osd_op_reply(95 10000000001.00000000 [omap-get-vals-by-keys out=506b] v0'0 uv9 ondisk = 0) v8 ==== 164+0+506 (crc 0 0 0) 0x5634526f4780 con 0x5634528fb000 2024-03-24T00:49:23.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.776+0000 7f4304231740 20 datascan.read_dentry: oid=10000000001.00000000 dname=bravo frag=*, r=0 2024-03-24T00:49:23.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.776+0000 7f4304231740 20 datascan.scan_links: first is now 2 2024-03-24T00:49:23.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.776+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- osd_op(unknown.0.0:96 17.7 17:e2e07930:::10000000001.00000000:head [omap-set-vals in=506b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452929c00 con 0x5634528fb000 2024-03-24T00:49:23.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.777+0000 7f4301aee640 1 -- 172.21.15.67:0/3638267307 <== osd.0 v2:172.21.15.67:6808/2628130519 18 ==== osd_op_reply(96 10000000001.00000000 [omap-set-vals] v122'10 uv10 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x5634526f4780 con 0x5634528fb000 2024-03-24T00:49:23.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.777+0000 7f4304231740 20 datascan.inject_linkage: Injected dentry 0x0x10000000001/bravo pointing to 0x0x100000001f7 2024-03-24T00:49:23.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.777+0000 7f4304231740 10 datascan.scan_links: updating inotable 2024-03-24T00:49:23.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.777+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:97 17.13 17:c91d4a1d:::mds0_inotable:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452928000 con 0x5634534ad000 2024-03-24T00:49:23.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.777+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 <== osd.3 v2:172.21.15.67:6816/3303489481 29 ==== osd_op_reply(97 mds0_inotable [read 0~50 out=50b] v0'0 uv4 ondisk = 0) v8 ==== 157+0+50 (crc 0 0 0) 0x5634526f4500 con 0x5634534ad000 2024-03-24T00:49:23.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.777+0000 7f4304231740 10 mds.0.inotable: skip_inos was [0x10000000005~0x1f2,0x100000001f9~0xfffffffe07] 2024-03-24T00:49:23.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.777+0000 7f4304231740 10 mds.0.inotable: skip_inos now [0x100000001f9~0xfffffffe07] 2024-03-24T00:49:23.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.777+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- osd_op(unknown.0.0:98 17.13 17:c91d4a1d:::mds0_inotable:head [writefull 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452923400 con 0x5634534ad000 2024-03-24T00:49:23.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.778+0000 7f43022ef640 1 -- 172.21.15.67:0/3638267307 <== osd.3 v2:172.21.15.67:6816/3303489481 30 ==== osd_op_reply(98 mds0_inotable [writefull 0~34] v122'6 uv6 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x5634526f4500 con 0x5634534ad000 2024-03-24T00:49:23.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.778+0000 7f4304231740 10 datascan.scan_links: updating snaptable 2024-03-24T00:49:23.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.778+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- osd_op(unknown.0.0:99 17.d 17:b50e409b:::mds_snaptable:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v9 -- 0x563452923000 con 0x5634534acc00 2024-03-24T00:49:23.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.779+0000 7f43012ed640 1 -- 172.21.15.67:0/3638267307 <== osd.7 v2:172.21.15.72:6824/2893038746 15 ==== osd_op_reply(99 mds_snaptable [read 0~70 out=70b] v0'0 uv1 ondisk = 0) v8 ==== 157+0+70 (crc 0 0 0) 0x5634526f4000 con 0x5634534acc00 2024-03-24T00:49:23.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.779+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x5634528fb000 msgr2=0x5634528fcb00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.779+0000 7f4304231740 1 --2- 172.21.15.67:0/3638267307 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x5634528fb000 0x5634528fcb00 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.779+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x5634528fb400 msgr2=0x5634528fd080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.779+0000 7f4304231740 1 --2- 172.21.15.67:0/3638267307 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x5634528fb400 0x5634528fd080 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.779+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 >> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] conn(0x563452922000 msgr2=0x5634528fdb80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.779+0000 7f4304231740 1 --2- 172.21.15.67:0/3638267307 >> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] conn(0x563452922000 0x5634528fdb80 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.779+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 >> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] conn(0x5634534ad000 msgr2=0x5634528fc580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.779+0000 7f4304231740 1 --2- 172.21.15.67:0/3638267307 >> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] conn(0x5634534ad000 0x5634528fc580 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.779+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x5634528fb800 msgr2=0x5634528fd600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.779+0000 7f4301aee640 1 -- 172.21.15.67:0/3638267307 reap_dead start 2024-03-24T00:49:23.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.779+0000 7f4304231740 1 --2- 172.21.15.67:0/3638267307 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x5634528fb800 0x5634528fd600 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.779+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x5634528fac00 msgr2=0x5634528fec00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.779+0000 7f4304231740 1 --2- 172.21.15.67:0/3638267307 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x5634528fac00 0x5634528fec00 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.779+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x5634528fbc00 msgr2=0x5634528fe680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.779+0000 7f4304231740 1 --2- 172.21.15.67:0/3638267307 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x5634528fbc00 0x5634528fe680 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.779+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x5634534acc00 msgr2=0x563452628680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.779+0000 7f4304231740 1 --2- 172.21.15.67:0/3638267307 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x5634534acc00 0x563452628680 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.779+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x5634528fa000 msgr2=0x5634528fc000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.779+0000 7f4304231740 1 --2- 172.21.15.67:0/3638267307 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x5634528fa000 0x5634528fc000 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x56345331ced0 tx=0x56345288cde0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.779+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5634534ac800 msgr2=0x563452628100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.779+0000 7f4304231740 1 --2- 172.21.15.67:0/3638267307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5634534ac800 0x563452628100 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x563453455ad0 tx=0x5634533187e0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.779+0000 7f4301aee640 1 -- 172.21.15.67:0/3638267307 reap_dead start 2024-03-24T00:49:23.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.779+0000 7f4301aee640 1 -- 172.21.15.67:0/3638267307 reap_dead start 2024-03-24T00:49:23.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.779+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 shutdown_connections 2024-03-24T00:49:23.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.779+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 >> 172.21.15.67:0/3638267307 conn(0x56345336fc00 msgr2=0x563453305000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:23.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.779+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 shutdown_connections 2024-03-24T00:49:23.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.779+0000 7f4304231740 1 -- 172.21.15.67:0/3638267307 wait complete. 2024-03-24T00:49:23.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.780+0000 7f4304231740 1 -- 172.21.15.67:0/1739904856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56345257dc00 msgr2=0x563452627080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:23.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.780+0000 7f4304231740 1 --2- 172.21.15.67:0/1739904856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56345257dc00 0x563452627080 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x5634533a91a0 tx=0x56345331a2a0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.780+0000 7f4304231740 1 -- 172.21.15.67:0/1739904856 shutdown_connections 2024-03-24T00:49:23.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.780+0000 7f4304231740 1 --2- 172.21.15.67:0/1739904856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56345257dc00 0x563452627080 unknown :-1 s=CLOSED pgs=1846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.780+0000 7f4304231740 1 --2- 172.21.15.67:0/1739904856 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x56345257d800 0x563452627600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.780+0000 7f4304231740 1 --2- 172.21.15.67:0/1739904856 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56345257cc00 0x563452627b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:23.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.780+0000 7f4304231740 1 -- 172.21.15.67:0/1739904856 >> 172.21.15.67:0/1739904856 conn(0x56345257d000 msgr2=0x563453304800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:23.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.780+0000 7f4304231740 1 -- 172.21.15.67:0/1739904856 shutdown_connections 2024-03-24T00:49:23.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:23.780+0000 7f4304231740 1 -- 172.21.15.67:0/1739904856 wait complete. 2024-03-24T00:49:23.802 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-data-scan', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'scan_links'] in time 0:00:00.136422, result: 2024-03-24T00:49:23.803 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-24T00:49:23.826 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-24T00:49:23.874 INFO:teuthology.orchestra.run.smithi067.stdout:[global] 2024-03-24T00:49:23.874 INFO:teuthology.orchestra.run.smithi067.stdout: chdir = "" 2024-03-24T00:49:23.874 INFO:teuthology.orchestra.run.smithi067.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-24T00:49:23.874 INFO:teuthology.orchestra.run.smithi067.stdout: auth supported = cephx 2024-03-24T00:49:23.874 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.874 INFO:teuthology.orchestra.run.smithi067.stdout: filestore xattr use omap = true 2024-03-24T00:49:23.874 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.874 INFO:teuthology.orchestra.run.smithi067.stdout: mon clock drift allowed = 1.000 2024-03-24T00:49:23.875 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.875 INFO:teuthology.orchestra.run.smithi067.stdout: osd crush chooseleaf type = 0 2024-03-24T00:49:23.875 INFO:teuthology.orchestra.run.smithi067.stdout: auth debug = true 2024-03-24T00:49:23.875 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.875 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on old message = true 2024-03-24T00:49:23.875 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on bug = true 2024-03-24T00:49:23.875 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.875 INFO:teuthology.orchestra.run.smithi067.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-24T00:49:23.875 INFO:teuthology.orchestra.run.smithi067.stdout: mon pg warn max object skew = 0 2024-03-24T00:49:23.875 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.875 INFO:teuthology.orchestra.run.smithi067.stdout: # disable pg_autoscaler by default for new pools 2024-03-24T00:49:23.875 INFO:teuthology.orchestra.run.smithi067.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-24T00:49:23.875 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.875 INFO:teuthology.orchestra.run.smithi067.stdout: osd pool default size = 2 2024-03-24T00:49:23.875 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.875 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd allow primary affinity = true 2024-03-24T00:49:23.876 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd allow pg remap = true 2024-03-24T00:49:23.876 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on legacy crush tunables = false 2024-03-24T00:49:23.876 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on crush straw calc version zero = false 2024-03-24T00:49:23.876 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on no sortbitwise = false 2024-03-24T00:49:23.876 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on osd down out interval zero = false 2024-03-24T00:49:23.876 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on too few osds = false 2024-03-24T00:49:23.876 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-24T00:49:23.876 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-24T00:49:23.876 INFO:teuthology.orchestra.run.smithi067.stdout: mon_allow_pool_size_one = true 2024-03-24T00:49:23.876 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.876 INFO:teuthology.orchestra.run.smithi067.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-24T00:49:23.876 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.876 INFO:teuthology.orchestra.run.smithi067.stdout: osd default data pool replay window = 5 2024-03-24T00:49:23.876 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.877 INFO:teuthology.orchestra.run.smithi067.stdout: mon allow pool delete = true 2024-03-24T00:49:23.877 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.877 INFO:teuthology.orchestra.run.smithi067.stdout: mon cluster log file level = debug 2024-03-24T00:49:23.877 INFO:teuthology.orchestra.run.smithi067.stdout: debug asserts on shutdown = true 2024-03-24T00:49:23.877 INFO:teuthology.orchestra.run.smithi067.stdout: mon health detail to clog = false 2024-03-24T00:49:23.877 INFO:teuthology.orchestra.run.smithi067.stdout: mon host = "172.21.15.67,172.21.15.72,[v2:172.21.15.72:3301,v1:172.21.15.72:6790]" 2024-03-24T00:49:23.877 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on skipped message = False 2024-03-24T00:49:23.877 INFO:teuthology.orchestra.run.smithi067.stdout: fsid = de1d0178-808e-49ca-a986-e297e523b976 2024-03-24T00:49:23.877 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.877 INFO:teuthology.orchestra.run.smithi067.stdout:[osd] 2024-03-24T00:49:23.877 INFO:teuthology.orchestra.run.smithi067.stdout: osd journal size = 100 2024-03-24T00:49:23.877 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.877 INFO:teuthology.orchestra.run.smithi067.stdout: osd scrub load threshold = 5.0 2024-03-24T00:49:23.877 INFO:teuthology.orchestra.run.smithi067.stdout: osd scrub max interval = 600 2024-03-24T00:49:23.877 INFO:teuthology.orchestra.run.smithi067.stdout: osd mclock profile = high_recovery_ops 2024-03-24T00:49:23.878 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.878 INFO:teuthology.orchestra.run.smithi067.stdout: osd recover clone overlap = true 2024-03-24T00:49:23.878 INFO:teuthology.orchestra.run.smithi067.stdout: osd recovery max chunk = 1048576 2024-03-24T00:49:23.878 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.878 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug shutdown = true 2024-03-24T00:49:23.878 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug op order = true 2024-03-24T00:49:23.878 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug verify stray on activate = true 2024-03-24T00:49:23.878 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug trim objects = true 2024-03-24T00:49:23.878 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.878 INFO:teuthology.orchestra.run.smithi067.stdout: osd open classes on start = true 2024-03-24T00:49:23.878 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug pg log writeout = true 2024-03-24T00:49:23.878 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.878 INFO:teuthology.orchestra.run.smithi067.stdout: osd deep scrub update digest min age = 30 2024-03-24T00:49:23.878 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.879 INFO:teuthology.orchestra.run.smithi067.stdout: osd map max advance = 10 2024-03-24T00:49:23.879 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.879 INFO:teuthology.orchestra.run.smithi067.stdout: journal zero on create = true 2024-03-24T00:49:23.879 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.879 INFO:teuthology.orchestra.run.smithi067.stdout: filestore ondisk finisher threads = 3 2024-03-24T00:49:23.879 INFO:teuthology.orchestra.run.smithi067.stdout: filestore apply finisher threads = 3 2024-03-24T00:49:23.879 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.879 INFO:teuthology.orchestra.run.smithi067.stdout: bdev debug aio = true 2024-03-24T00:49:23.879 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug misdirected ops = true 2024-03-24T00:49:23.879 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore block size = 96636764160 2024-03-24T00:49:23.879 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore fsck on mount = True 2024-03-24T00:49:23.879 INFO:teuthology.orchestra.run.smithi067.stdout: debug bluefs = 20 2024-03-24T00:49:23.879 INFO:teuthology.orchestra.run.smithi067.stdout: debug bluestore = 20 2024-03-24T00:49:23.879 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-03-24T00:49:23.879 INFO:teuthology.orchestra.run.smithi067.stdout: debug osd = 20 2024-03-24T00:49:23.880 INFO:teuthology.orchestra.run.smithi067.stdout: debug rocksdb = 10 2024-03-24T00:49:23.880 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd backfillfull_ratio = 0.85 2024-03-24T00:49:23.880 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd full ratio = 0.9 2024-03-24T00:49:23.880 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd nearfull ratio = 0.8 2024-03-24T00:49:23.880 INFO:teuthology.orchestra.run.smithi067.stdout: osd failsafe full ratio = 0.95 2024-03-24T00:49:23.880 INFO:teuthology.orchestra.run.smithi067.stdout: osd objectstore = bluestore 2024-03-24T00:49:23.880 INFO:teuthology.orchestra.run.smithi067.stdout: osd op complaint time = 180 2024-03-24T00:49:23.880 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.880 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr] 2024-03-24T00:49:23.880 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-03-24T00:49:23.880 INFO:teuthology.orchestra.run.smithi067.stdout: debug mgr = 20 2024-03-24T00:49:23.880 INFO:teuthology.orchestra.run.smithi067.stdout: debug mon = 20 2024-03-24T00:49:23.880 INFO:teuthology.orchestra.run.smithi067.stdout: debug auth = 20 2024-03-24T00:49:23.880 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min pgs per osd = 4 2024-03-24T00:49:23.881 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min bytes per osd = 10 2024-03-24T00:49:23.881 INFO:teuthology.orchestra.run.smithi067.stdout: mgr/telemetry/nag = false 2024-03-24T00:49:23.881 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.881 INFO:teuthology.orchestra.run.smithi067.stdout:[mon] 2024-03-24T00:49:23.881 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-03-24T00:49:23.881 INFO:teuthology.orchestra.run.smithi067.stdout: debug mon = 20 2024-03-24T00:49:23.881 INFO:teuthology.orchestra.run.smithi067.stdout: debug paxos = 20 2024-03-24T00:49:23.881 INFO:teuthology.orchestra.run.smithi067.stdout: debug auth = 20 2024-03-24T00:49:23.881 INFO:teuthology.orchestra.run.smithi067.stdout: mon data avail warn = 5 2024-03-24T00:49:23.881 INFO:teuthology.orchestra.run.smithi067.stdout: mon mgr mkfs grace = 240 2024-03-24T00:49:23.881 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min pgs per osd = 4 2024-03-24T00:49:23.881 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd reporter subtree level = osd 2024-03-24T00:49:23.881 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd prime pg temp = true 2024-03-24T00:49:23.881 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min bytes per osd = 10 2024-03-24T00:49:23.881 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.882 INFO:teuthology.orchestra.run.smithi067.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-24T00:49:23.882 INFO:teuthology.orchestra.run.smithi067.stdout: auth mon ticket ttl = 660 # 11m 2024-03-24T00:49:23.882 INFO:teuthology.orchestra.run.smithi067.stdout: auth service ticket ttl = 240 # 4m 2024-03-24T00:49:23.882 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.882 INFO:teuthology.orchestra.run.smithi067.stdout: # don't complain about insecure global_id in the test suite 2024-03-24T00:49:23.882 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-24T00:49:23.882 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-24T00:49:23.882 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.882 INFO:teuthology.orchestra.run.smithi067.stdout: # 1m isn't quite enough 2024-03-24T00:49:23.882 INFO:teuthology.orchestra.run.smithi067.stdout: mon_down_mkfs_grace = 2m 2024-03-24T00:49:23.882 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.882 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_filestore_osds = false 2024-03-24T00:49:23.882 INFO:teuthology.orchestra.run.smithi067.stdout: mon op complaint time = 120 2024-03-24T00:49:23.882 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.883 INFO:teuthology.orchestra.run.smithi067.stdout:[client] 2024-03-24T00:49:23.883 INFO:teuthology.orchestra.run.smithi067.stdout: rgw cache enabled = true 2024-03-24T00:49:23.883 INFO:teuthology.orchestra.run.smithi067.stdout: rgw enable ops log = true 2024-03-24T00:49:23.883 INFO:teuthology.orchestra.run.smithi067.stdout: rgw enable usage log = true 2024-03-24T00:49:23.883 INFO:teuthology.orchestra.run.smithi067.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-24T00:49:23.883 INFO:teuthology.orchestra.run.smithi067.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-24T00:49:23.883 INFO:teuthology.orchestra.run.smithi067.stdout: client mount timeout = 600 2024-03-24T00:49:23.883 INFO:teuthology.orchestra.run.smithi067.stdout: debug client = 20 2024-03-24T00:49:23.883 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-03-24T00:49:23.883 INFO:teuthology.orchestra.run.smithi067.stdout: pid file = "" 2024-03-24T00:49:23.883 INFO:teuthology.orchestra.run.smithi067.stdout: rados mon op timeout = 900 2024-03-24T00:49:23.883 INFO:teuthology.orchestra.run.smithi067.stdout: rados osd op timeout = 900 2024-03-24T00:49:23.883 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.a] 2024-03-24T00:49:23.883 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.b] 2024-03-24T00:49:23.883 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.c] 2024-03-24T00:49:23.884 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.a] 2024-03-24T00:49:23.884 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.c] 2024-03-24T00:49:23.884 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.b] 2024-03-24T00:49:23.884 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.d] 2024-03-24T00:49:23.884 INFO:teuthology.orchestra.run.smithi067.stdout:[mds] 2024-03-24T00:49:23.884 INFO:teuthology.orchestra.run.smithi067.stdout: debug mds = 20 2024-03-24T00:49:23.884 INFO:teuthology.orchestra.run.smithi067.stdout: debug mds balancer = 20 2024-03-24T00:49:23.884 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-03-24T00:49:23.884 INFO:teuthology.orchestra.run.smithi067.stdout: mds debug frag = True 2024-03-24T00:49:23.884 INFO:teuthology.orchestra.run.smithi067.stdout: mds debug scatterstat = False 2024-03-24T00:49:23.884 INFO:teuthology.orchestra.run.smithi067.stdout: mds op complaint time = 180 2024-03-24T00:49:23.884 INFO:teuthology.orchestra.run.smithi067.stdout: mds verify scatter = True 2024-03-24T00:49:23.884 INFO:teuthology.orchestra.run.smithi067.stdout: osd op complaint time = 180 2024-03-24T00:49:23.884 INFO:teuthology.orchestra.run.smithi067.stdout: rados mon op timeout = 900 2024-03-24T00:49:23.884 INFO:teuthology.orchestra.run.smithi067.stdout: rados osd op timeout = 900 2024-03-24T00:49:23.885 INFO:teuthology.orchestra.run.smithi072.stdout:[global] 2024-03-24T00:49:23.885 INFO:teuthology.orchestra.run.smithi072.stdout: chdir = "" 2024-03-24T00:49:23.885 INFO:teuthology.orchestra.run.smithi072.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-24T00:49:23.885 INFO:teuthology.orchestra.run.smithi072.stdout: auth supported = cephx 2024-03-24T00:49:23.885 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:23.885 INFO:teuthology.orchestra.run.smithi072.stdout: filestore xattr use omap = true 2024-03-24T00:49:23.885 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:23.885 INFO:teuthology.orchestra.run.smithi072.stdout: mon clock drift allowed = 1.000 2024-03-24T00:49:23.885 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:23.886 INFO:teuthology.orchestra.run.smithi072.stdout: osd crush chooseleaf type = 0 2024-03-24T00:49:23.886 INFO:teuthology.orchestra.run.smithi072.stdout: auth debug = true 2024-03-24T00:49:23.886 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:23.886 INFO:teuthology.orchestra.run.smithi072.stdout: ms die on old message = true 2024-03-24T00:49:23.886 INFO:teuthology.orchestra.run.smithi072.stdout: ms die on bug = true 2024-03-24T00:49:23.886 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:23.886 INFO:teuthology.orchestra.run.smithi072.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-24T00:49:23.886 INFO:teuthology.orchestra.run.smithi072.stdout: mon pg warn max object skew = 0 2024-03-24T00:49:23.886 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:23.886 INFO:teuthology.orchestra.run.smithi072.stdout: # disable pg_autoscaler by default for new pools 2024-03-24T00:49:23.886 INFO:teuthology.orchestra.run.smithi072.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-24T00:49:23.890 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:23.890 INFO:teuthology.orchestra.run.smithi072.stdout: osd pool default size = 2 2024-03-24T00:49:23.890 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:23.890 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd allow primary affinity = true 2024-03-24T00:49:23.890 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd allow pg remap = true 2024-03-24T00:49:23.890 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on legacy crush tunables = false 2024-03-24T00:49:23.890 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on crush straw calc version zero = false 2024-03-24T00:49:23.890 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on no sortbitwise = false 2024-03-24T00:49:23.890 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on osd down out interval zero = false 2024-03-24T00:49:23.890 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on too few osds = false 2024-03-24T00:49:23.891 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-24T00:49:23.891 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-24T00:49:23.891 INFO:teuthology.orchestra.run.smithi072.stdout: mon_allow_pool_size_one = true 2024-03-24T00:49:23.891 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:23.891 INFO:teuthology.orchestra.run.smithi072.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-24T00:49:23.891 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:23.891 INFO:teuthology.orchestra.run.smithi072.stdout: osd default data pool replay window = 5 2024-03-24T00:49:23.891 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:23.891 INFO:teuthology.orchestra.run.smithi072.stdout: mon allow pool delete = true 2024-03-24T00:49:23.891 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:23.891 INFO:teuthology.orchestra.run.smithi072.stdout: mon cluster log file level = debug 2024-03-24T00:49:23.891 INFO:teuthology.orchestra.run.smithi072.stdout: debug asserts on shutdown = true 2024-03-24T00:49:23.891 INFO:teuthology.orchestra.run.smithi072.stdout: mon health detail to clog = false 2024-03-24T00:49:23.891 INFO:teuthology.orchestra.run.smithi072.stdout: mon host = "172.21.15.67,172.21.15.72,[v2:172.21.15.72:3301,v1:172.21.15.72:6790]" 2024-03-24T00:49:23.891 INFO:teuthology.orchestra.run.smithi072.stdout: ms die on skipped message = False 2024-03-24T00:49:23.892 INFO:teuthology.orchestra.run.smithi072.stdout: fsid = de1d0178-808e-49ca-a986-e297e523b976 2024-03-24T00:49:23.892 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:23.892 INFO:teuthology.orchestra.run.smithi072.stdout:[osd] 2024-03-24T00:49:23.892 INFO:teuthology.orchestra.run.smithi072.stdout: osd journal size = 100 2024-03-24T00:49:23.892 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:23.892 INFO:teuthology.orchestra.run.smithi072.stdout: osd scrub load threshold = 5.0 2024-03-24T00:49:23.892 INFO:teuthology.orchestra.run.smithi072.stdout: osd scrub max interval = 600 2024-03-24T00:49:23.892 INFO:teuthology.orchestra.run.smithi072.stdout: osd mclock profile = high_recovery_ops 2024-03-24T00:49:23.892 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:23.892 INFO:teuthology.orchestra.run.smithi072.stdout: osd recover clone overlap = true 2024-03-24T00:49:23.892 INFO:teuthology.orchestra.run.smithi072.stdout: osd recovery max chunk = 1048576 2024-03-24T00:49:23.892 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:23.892 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug shutdown = true 2024-03-24T00:49:23.892 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug op order = true 2024-03-24T00:49:23.892 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug verify stray on activate = true 2024-03-24T00:49:23.893 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug trim objects = true 2024-03-24T00:49:23.893 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:23.893 INFO:teuthology.orchestra.run.smithi072.stdout: osd open classes on start = true 2024-03-24T00:49:23.893 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug pg log writeout = true 2024-03-24T00:49:23.893 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:23.893 INFO:teuthology.orchestra.run.smithi072.stdout: osd deep scrub update digest min age = 30 2024-03-24T00:49:23.893 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:23.893 INFO:teuthology.orchestra.run.smithi072.stdout: osd map max advance = 10 2024-03-24T00:49:23.893 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:23.893 INFO:teuthology.orchestra.run.smithi072.stdout: journal zero on create = true 2024-03-24T00:49:23.893 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:23.893 INFO:teuthology.orchestra.run.smithi072.stdout: filestore ondisk finisher threads = 3 2024-03-24T00:49:23.893 INFO:teuthology.orchestra.run.smithi072.stdout: filestore apply finisher threads = 3 2024-03-24T00:49:23.893 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:23.894 INFO:teuthology.orchestra.run.smithi072.stdout: bdev debug aio = true 2024-03-24T00:49:23.894 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug misdirected ops = true 2024-03-24T00:49:23.894 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore block size = 96636764160 2024-03-24T00:49:23.894 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore fsck on mount = True 2024-03-24T00:49:23.894 INFO:teuthology.orchestra.run.smithi072.stdout: debug bluefs = 20 2024-03-24T00:49:23.894 INFO:teuthology.orchestra.run.smithi072.stdout: debug bluestore = 20 2024-03-24T00:49:23.894 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-24T00:49:23.894 INFO:teuthology.orchestra.run.smithi072.stdout: debug osd = 20 2024-03-24T00:49:23.894 INFO:teuthology.orchestra.run.smithi072.stdout: debug rocksdb = 10 2024-03-24T00:49:23.894 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd backfillfull_ratio = 0.85 2024-03-24T00:49:23.894 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd full ratio = 0.9 2024-03-24T00:49:23.894 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd nearfull ratio = 0.8 2024-03-24T00:49:23.894 INFO:teuthology.orchestra.run.smithi072.stdout: osd failsafe full ratio = 0.95 2024-03-24T00:49:23.894 INFO:teuthology.orchestra.run.smithi072.stdout: osd objectstore = bluestore 2024-03-24T00:49:23.894 INFO:teuthology.orchestra.run.smithi072.stdout: osd op complaint time = 180 2024-03-24T00:49:23.895 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:23.895 INFO:teuthology.orchestra.run.smithi072.stdout:[mgr] 2024-03-24T00:49:23.895 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-24T00:49:23.895 INFO:teuthology.orchestra.run.smithi072.stdout: debug mgr = 20 2024-03-24T00:49:23.895 INFO:teuthology.orchestra.run.smithi072.stdout: debug mon = 20 2024-03-24T00:49:23.895 INFO:teuthology.orchestra.run.smithi072.stdout: debug auth = 20 2024-03-24T00:49:23.895 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min pgs per osd = 4 2024-03-24T00:49:23.895 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min bytes per osd = 10 2024-03-24T00:49:23.895 INFO:teuthology.orchestra.run.smithi072.stdout: mgr/telemetry/nag = false 2024-03-24T00:49:23.895 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:23.895 INFO:teuthology.orchestra.run.smithi072.stdout:[mon] 2024-03-24T00:49:23.895 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-24T00:49:23.895 INFO:teuthology.orchestra.run.smithi072.stdout: debug mon = 20 2024-03-24T00:49:23.895 INFO:teuthology.orchestra.run.smithi072.stdout: debug paxos = 20 2024-03-24T00:49:23.896 INFO:teuthology.orchestra.run.smithi072.stdout: debug auth = 20 2024-03-24T00:49:23.896 INFO:teuthology.orchestra.run.smithi072.stdout: mon data avail warn = 5 2024-03-24T00:49:23.896 INFO:teuthology.orchestra.run.smithi072.stdout: mon mgr mkfs grace = 240 2024-03-24T00:49:23.896 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min pgs per osd = 4 2024-03-24T00:49:23.896 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd reporter subtree level = osd 2024-03-24T00:49:23.896 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd prime pg temp = true 2024-03-24T00:49:23.896 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min bytes per osd = 10 2024-03-24T00:49:23.896 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:23.896 INFO:teuthology.orchestra.run.smithi072.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-24T00:49:23.896 INFO:teuthology.orchestra.run.smithi072.stdout: auth mon ticket ttl = 660 # 11m 2024-03-24T00:49:23.896 INFO:teuthology.orchestra.run.smithi072.stdout: auth service ticket ttl = 240 # 4m 2024-03-24T00:49:23.896 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:23.896 INFO:teuthology.orchestra.run.smithi072.stdout: # don't complain about insecure global_id in the test suite 2024-03-24T00:49:23.896 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-24T00:49:23.896 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-24T00:49:23.897 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:23.897 INFO:teuthology.orchestra.run.smithi072.stdout: # 1m isn't quite enough 2024-03-24T00:49:23.897 INFO:teuthology.orchestra.run.smithi072.stdout: mon_down_mkfs_grace = 2m 2024-03-24T00:49:23.897 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:23.897 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_filestore_osds = false 2024-03-24T00:49:23.897 INFO:teuthology.orchestra.run.smithi072.stdout: mon op complaint time = 120 2024-03-24T00:49:23.897 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:23.897 INFO:teuthology.orchestra.run.smithi072.stdout:[client] 2024-03-24T00:49:23.897 INFO:teuthology.orchestra.run.smithi072.stdout: rgw cache enabled = true 2024-03-24T00:49:23.897 INFO:teuthology.orchestra.run.smithi072.stdout: rgw enable ops log = true 2024-03-24T00:49:23.897 INFO:teuthology.orchestra.run.smithi072.stdout: rgw enable usage log = true 2024-03-24T00:49:23.897 INFO:teuthology.orchestra.run.smithi072.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-24T00:49:23.897 INFO:teuthology.orchestra.run.smithi072.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-24T00:49:23.897 INFO:teuthology.orchestra.run.smithi072.stdout: client mount timeout = 600 2024-03-24T00:49:23.897 INFO:teuthology.orchestra.run.smithi072.stdout: debug client = 20 2024-03-24T00:49:23.897 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-24T00:49:23.898 INFO:teuthology.orchestra.run.smithi072.stdout: pid file = "" 2024-03-24T00:49:23.898 INFO:teuthology.orchestra.run.smithi072.stdout: rados mon op timeout = 900 2024-03-24T00:49:23.898 INFO:teuthology.orchestra.run.smithi072.stdout: rados osd op timeout = 900 2024-03-24T00:49:23.898 INFO:teuthology.orchestra.run.smithi072.stdout:[mon.a] 2024-03-24T00:49:23.898 INFO:teuthology.orchestra.run.smithi072.stdout:[mon.b] 2024-03-24T00:49:23.898 INFO:teuthology.orchestra.run.smithi072.stdout:[mon.c] 2024-03-24T00:49:23.898 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.a] 2024-03-24T00:49:23.898 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.c] 2024-03-24T00:49:23.898 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.b] 2024-03-24T00:49:23.898 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.d] 2024-03-24T00:49:23.898 INFO:teuthology.orchestra.run.smithi072.stdout:[mds] 2024-03-24T00:49:23.898 INFO:teuthology.orchestra.run.smithi072.stdout: debug mds = 20 2024-03-24T00:49:23.898 INFO:teuthology.orchestra.run.smithi072.stdout: debug mds balancer = 20 2024-03-24T00:49:23.898 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-24T00:49:23.898 INFO:teuthology.orchestra.run.smithi072.stdout: mds debug frag = True 2024-03-24T00:49:23.899 INFO:teuthology.orchestra.run.smithi072.stdout: mds debug scatterstat = False 2024-03-24T00:49:23.899 INFO:teuthology.orchestra.run.smithi072.stdout: mds op complaint time = 180 2024-03-24T00:49:23.899 INFO:teuthology.orchestra.run.smithi072.stdout: mds verify scatter = True 2024-03-24T00:49:23.899 INFO:teuthology.orchestra.run.smithi072.stdout: osd op complaint time = 180 2024-03-24T00:49:23.899 INFO:teuthology.orchestra.run.smithi072.stdout: rados mon op timeout = 900 2024-03-24T00:49:23.899 INFO:teuthology.orchestra.run.smithi072.stdout: rados osd op timeout = 900 2024-03-24T00:49:23.901 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-24T00:49:23.943 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-24T00:49:23.991 INFO:teuthology.orchestra.run.smithi067.stdout:[global] 2024-03-24T00:49:23.991 INFO:teuthology.orchestra.run.smithi067.stdout: chdir = "" 2024-03-24T00:49:23.991 INFO:teuthology.orchestra.run.smithi067.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-24T00:49:23.991 INFO:teuthology.orchestra.run.smithi067.stdout: auth supported = cephx 2024-03-24T00:49:23.991 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.991 INFO:teuthology.orchestra.run.smithi067.stdout: filestore xattr use omap = true 2024-03-24T00:49:23.991 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.991 INFO:teuthology.orchestra.run.smithi067.stdout: mon clock drift allowed = 1.000 2024-03-24T00:49:23.992 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.992 INFO:teuthology.orchestra.run.smithi067.stdout: osd crush chooseleaf type = 0 2024-03-24T00:49:23.992 INFO:teuthology.orchestra.run.smithi067.stdout: auth debug = true 2024-03-24T00:49:23.992 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.992 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on old message = true 2024-03-24T00:49:23.992 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on bug = true 2024-03-24T00:49:23.992 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.992 INFO:teuthology.orchestra.run.smithi067.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-24T00:49:23.992 INFO:teuthology.orchestra.run.smithi067.stdout: mon pg warn max object skew = 0 2024-03-24T00:49:23.992 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.992 INFO:teuthology.orchestra.run.smithi067.stdout: # disable pg_autoscaler by default for new pools 2024-03-24T00:49:23.992 INFO:teuthology.orchestra.run.smithi067.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-24T00:49:23.992 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.992 INFO:teuthology.orchestra.run.smithi067.stdout: osd pool default size = 2 2024-03-24T00:49:23.992 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.993 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd allow primary affinity = true 2024-03-24T00:49:23.993 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd allow pg remap = true 2024-03-24T00:49:23.993 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on legacy crush tunables = false 2024-03-24T00:49:23.993 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on crush straw calc version zero = false 2024-03-24T00:49:23.993 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on no sortbitwise = false 2024-03-24T00:49:23.993 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on osd down out interval zero = false 2024-03-24T00:49:23.993 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on too few osds = false 2024-03-24T00:49:23.993 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-24T00:49:23.993 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-24T00:49:23.993 INFO:teuthology.orchestra.run.smithi067.stdout: mon_allow_pool_size_one = true 2024-03-24T00:49:23.993 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.993 INFO:teuthology.orchestra.run.smithi067.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-24T00:49:23.993 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.993 INFO:teuthology.orchestra.run.smithi067.stdout: osd default data pool replay window = 5 2024-03-24T00:49:23.994 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.994 INFO:teuthology.orchestra.run.smithi067.stdout: mon allow pool delete = true 2024-03-24T00:49:23.994 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.994 INFO:teuthology.orchestra.run.smithi067.stdout: mon cluster log file level = debug 2024-03-24T00:49:23.994 INFO:teuthology.orchestra.run.smithi067.stdout: debug asserts on shutdown = true 2024-03-24T00:49:23.994 INFO:teuthology.orchestra.run.smithi067.stdout: mon health detail to clog = false 2024-03-24T00:49:23.994 INFO:teuthology.orchestra.run.smithi067.stdout: mon host = "172.21.15.67,172.21.15.72,[v2:172.21.15.72:3301,v1:172.21.15.72:6790]" 2024-03-24T00:49:23.994 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on skipped message = False 2024-03-24T00:49:23.994 INFO:teuthology.orchestra.run.smithi067.stdout: fsid = de1d0178-808e-49ca-a986-e297e523b976 2024-03-24T00:49:23.994 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.994 INFO:teuthology.orchestra.run.smithi067.stdout:[osd] 2024-03-24T00:49:23.994 INFO:teuthology.orchestra.run.smithi067.stdout: osd journal size = 100 2024-03-24T00:49:23.994 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.994 INFO:teuthology.orchestra.run.smithi067.stdout: osd scrub load threshold = 5.0 2024-03-24T00:49:23.995 INFO:teuthology.orchestra.run.smithi067.stdout: osd scrub max interval = 600 2024-03-24T00:49:23.995 INFO:teuthology.orchestra.run.smithi067.stdout: osd mclock profile = high_recovery_ops 2024-03-24T00:49:23.995 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.995 INFO:teuthology.orchestra.run.smithi067.stdout: osd recover clone overlap = true 2024-03-24T00:49:23.995 INFO:teuthology.orchestra.run.smithi067.stdout: osd recovery max chunk = 1048576 2024-03-24T00:49:23.995 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.995 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug shutdown = true 2024-03-24T00:49:23.995 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug op order = true 2024-03-24T00:49:23.995 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug verify stray on activate = true 2024-03-24T00:49:23.995 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug trim objects = true 2024-03-24T00:49:23.995 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.995 INFO:teuthology.orchestra.run.smithi067.stdout: osd open classes on start = true 2024-03-24T00:49:23.995 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug pg log writeout = true 2024-03-24T00:49:23.995 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.996 INFO:teuthology.orchestra.run.smithi067.stdout: osd deep scrub update digest min age = 30 2024-03-24T00:49:23.996 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.996 INFO:teuthology.orchestra.run.smithi067.stdout: osd map max advance = 10 2024-03-24T00:49:23.996 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.996 INFO:teuthology.orchestra.run.smithi067.stdout: journal zero on create = true 2024-03-24T00:49:23.996 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.996 INFO:teuthology.orchestra.run.smithi067.stdout: filestore ondisk finisher threads = 3 2024-03-24T00:49:23.996 INFO:teuthology.orchestra.run.smithi067.stdout: filestore apply finisher threads = 3 2024-03-24T00:49:23.996 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.996 INFO:teuthology.orchestra.run.smithi067.stdout: bdev debug aio = true 2024-03-24T00:49:23.996 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug misdirected ops = true 2024-03-24T00:49:23.996 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore block size = 96636764160 2024-03-24T00:49:23.996 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore fsck on mount = True 2024-03-24T00:49:23.997 INFO:teuthology.orchestra.run.smithi067.stdout: debug bluefs = 20 2024-03-24T00:49:23.997 INFO:teuthology.orchestra.run.smithi067.stdout: debug bluestore = 20 2024-03-24T00:49:23.997 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-03-24T00:49:23.997 INFO:teuthology.orchestra.run.smithi067.stdout: debug osd = 20 2024-03-24T00:49:23.997 INFO:teuthology.orchestra.run.smithi067.stdout: debug rocksdb = 10 2024-03-24T00:49:23.997 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd backfillfull_ratio = 0.85 2024-03-24T00:49:23.997 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd full ratio = 0.9 2024-03-24T00:49:23.997 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd nearfull ratio = 0.8 2024-03-24T00:49:23.997 INFO:teuthology.orchestra.run.smithi067.stdout: osd failsafe full ratio = 0.95 2024-03-24T00:49:23.997 INFO:teuthology.orchestra.run.smithi067.stdout: osd objectstore = bluestore 2024-03-24T00:49:23.997 INFO:teuthology.orchestra.run.smithi067.stdout: osd op complaint time = 180 2024-03-24T00:49:23.997 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.997 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr] 2024-03-24T00:49:23.997 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-03-24T00:49:23.997 INFO:teuthology.orchestra.run.smithi067.stdout: debug mgr = 20 2024-03-24T00:49:23.997 INFO:teuthology.orchestra.run.smithi067.stdout: debug mon = 20 2024-03-24T00:49:23.998 INFO:teuthology.orchestra.run.smithi067.stdout: debug auth = 20 2024-03-24T00:49:23.998 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min pgs per osd = 4 2024-03-24T00:49:23.998 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min bytes per osd = 10 2024-03-24T00:49:23.998 INFO:teuthology.orchestra.run.smithi067.stdout: mgr/telemetry/nag = false 2024-03-24T00:49:23.998 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.998 INFO:teuthology.orchestra.run.smithi067.stdout:[mon] 2024-03-24T00:49:23.998 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-03-24T00:49:23.998 INFO:teuthology.orchestra.run.smithi067.stdout: debug mon = 20 2024-03-24T00:49:23.998 INFO:teuthology.orchestra.run.smithi067.stdout: debug paxos = 20 2024-03-24T00:49:23.998 INFO:teuthology.orchestra.run.smithi067.stdout: debug auth = 20 2024-03-24T00:49:23.998 INFO:teuthology.orchestra.run.smithi067.stdout: mon data avail warn = 5 2024-03-24T00:49:23.998 INFO:teuthology.orchestra.run.smithi067.stdout: mon mgr mkfs grace = 240 2024-03-24T00:49:23.998 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min pgs per osd = 4 2024-03-24T00:49:23.998 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd reporter subtree level = osd 2024-03-24T00:49:23.999 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd prime pg temp = true 2024-03-24T00:49:23.999 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min bytes per osd = 10 2024-03-24T00:49:23.999 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.999 INFO:teuthology.orchestra.run.smithi067.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-24T00:49:23.999 INFO:teuthology.orchestra.run.smithi067.stdout: auth mon ticket ttl = 660 # 11m 2024-03-24T00:49:23.999 INFO:teuthology.orchestra.run.smithi067.stdout: auth service ticket ttl = 240 # 4m 2024-03-24T00:49:23.999 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.999 INFO:teuthology.orchestra.run.smithi067.stdout: # don't complain about insecure global_id in the test suite 2024-03-24T00:49:23.999 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-24T00:49:23.999 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-24T00:49:23.999 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.999 INFO:teuthology.orchestra.run.smithi067.stdout: # 1m isn't quite enough 2024-03-24T00:49:23.999 INFO:teuthology.orchestra.run.smithi067.stdout: mon_down_mkfs_grace = 2m 2024-03-24T00:49:23.999 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:23.999 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_filestore_osds = false 2024-03-24T00:49:24.000 INFO:teuthology.orchestra.run.smithi067.stdout: mon op complaint time = 120 2024-03-24T00:49:24.000 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:24.000 INFO:teuthology.orchestra.run.smithi067.stdout:[client] 2024-03-24T00:49:24.000 INFO:teuthology.orchestra.run.smithi067.stdout: rgw cache enabled = true 2024-03-24T00:49:24.000 INFO:teuthology.orchestra.run.smithi067.stdout: rgw enable ops log = true 2024-03-24T00:49:24.000 INFO:teuthology.orchestra.run.smithi067.stdout: rgw enable usage log = true 2024-03-24T00:49:24.000 INFO:teuthology.orchestra.run.smithi067.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-24T00:49:24.000 INFO:teuthology.orchestra.run.smithi067.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-24T00:49:24.000 INFO:teuthology.orchestra.run.smithi067.stdout: client mount timeout = 600 2024-03-24T00:49:24.000 INFO:teuthology.orchestra.run.smithi067.stdout: debug client = 20 2024-03-24T00:49:24.000 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-03-24T00:49:24.000 INFO:teuthology.orchestra.run.smithi067.stdout: pid file = "" 2024-03-24T00:49:24.000 INFO:teuthology.orchestra.run.smithi067.stdout: rados mon op timeout = 900 2024-03-24T00:49:24.000 INFO:teuthology.orchestra.run.smithi067.stdout: rados osd op timeout = 900 2024-03-24T00:49:24.000 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.a] 2024-03-24T00:49:24.001 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.b] 2024-03-24T00:49:24.001 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.c] 2024-03-24T00:49:24.001 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.a] 2024-03-24T00:49:24.001 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.c] 2024-03-24T00:49:24.001 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.b] 2024-03-24T00:49:24.001 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.d] 2024-03-24T00:49:24.001 INFO:teuthology.orchestra.run.smithi067.stdout:[mds] 2024-03-24T00:49:24.001 INFO:teuthology.orchestra.run.smithi067.stdout: debug mds = 20 2024-03-24T00:49:24.001 INFO:teuthology.orchestra.run.smithi067.stdout: debug mds balancer = 20 2024-03-24T00:49:24.001 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-03-24T00:49:24.001 INFO:teuthology.orchestra.run.smithi067.stdout: mds debug frag = True 2024-03-24T00:49:24.001 INFO:teuthology.orchestra.run.smithi067.stdout: mds debug scatterstat = True 2024-03-24T00:49:24.001 INFO:teuthology.orchestra.run.smithi067.stdout: mds op complaint time = 180 2024-03-24T00:49:24.001 INFO:teuthology.orchestra.run.smithi067.stdout: mds verify scatter = True 2024-03-24T00:49:24.001 INFO:teuthology.orchestra.run.smithi067.stdout: osd op complaint time = 180 2024-03-24T00:49:24.001 INFO:teuthology.orchestra.run.smithi067.stdout: rados mon op timeout = 900 2024-03-24T00:49:24.002 INFO:teuthology.orchestra.run.smithi067.stdout: rados osd op timeout = 900 2024-03-24T00:49:24.002 INFO:teuthology.orchestra.run.smithi072.stdout:[global] 2024-03-24T00:49:24.002 INFO:teuthology.orchestra.run.smithi072.stdout: chdir = "" 2024-03-24T00:49:24.002 INFO:teuthology.orchestra.run.smithi072.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-24T00:49:24.002 INFO:teuthology.orchestra.run.smithi072.stdout: auth supported = cephx 2024-03-24T00:49:24.002 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:24.002 INFO:teuthology.orchestra.run.smithi072.stdout: filestore xattr use omap = true 2024-03-24T00:49:24.002 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:24.002 INFO:teuthology.orchestra.run.smithi072.stdout: mon clock drift allowed = 1.000 2024-03-24T00:49:24.003 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:24.003 INFO:teuthology.orchestra.run.smithi072.stdout: osd crush chooseleaf type = 0 2024-03-24T00:49:24.003 INFO:teuthology.orchestra.run.smithi072.stdout: auth debug = true 2024-03-24T00:49:24.003 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:24.003 INFO:teuthology.orchestra.run.smithi072.stdout: ms die on old message = true 2024-03-24T00:49:24.003 INFO:teuthology.orchestra.run.smithi072.stdout: ms die on bug = true 2024-03-24T00:49:24.003 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:24.003 INFO:teuthology.orchestra.run.smithi072.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-24T00:49:24.003 INFO:teuthology.orchestra.run.smithi072.stdout: mon pg warn max object skew = 0 2024-03-24T00:49:24.003 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:24.003 INFO:teuthology.orchestra.run.smithi072.stdout: # disable pg_autoscaler by default for new pools 2024-03-24T00:49:24.003 INFO:teuthology.orchestra.run.smithi072.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-24T00:49:24.003 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:24.003 INFO:teuthology.orchestra.run.smithi072.stdout: osd pool default size = 2 2024-03-24T00:49:24.004 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:24.004 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd allow primary affinity = true 2024-03-24T00:49:24.004 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd allow pg remap = true 2024-03-24T00:49:24.004 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on legacy crush tunables = false 2024-03-24T00:49:24.004 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on crush straw calc version zero = false 2024-03-24T00:49:24.004 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on no sortbitwise = false 2024-03-24T00:49:24.004 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on osd down out interval zero = false 2024-03-24T00:49:24.004 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on too few osds = false 2024-03-24T00:49:24.004 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-24T00:49:24.004 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-24T00:49:24.004 INFO:teuthology.orchestra.run.smithi072.stdout: mon_allow_pool_size_one = true 2024-03-24T00:49:24.004 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:24.004 INFO:teuthology.orchestra.run.smithi072.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-24T00:49:24.004 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:24.004 INFO:teuthology.orchestra.run.smithi072.stdout: osd default data pool replay window = 5 2024-03-24T00:49:24.005 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:24.005 INFO:teuthology.orchestra.run.smithi072.stdout: mon allow pool delete = true 2024-03-24T00:49:24.005 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:24.005 INFO:teuthology.orchestra.run.smithi072.stdout: mon cluster log file level = debug 2024-03-24T00:49:24.005 INFO:teuthology.orchestra.run.smithi072.stdout: debug asserts on shutdown = true 2024-03-24T00:49:24.005 INFO:teuthology.orchestra.run.smithi072.stdout: mon health detail to clog = false 2024-03-24T00:49:24.005 INFO:teuthology.orchestra.run.smithi072.stdout: mon host = "172.21.15.67,172.21.15.72,[v2:172.21.15.72:3301,v1:172.21.15.72:6790]" 2024-03-24T00:49:24.005 INFO:teuthology.orchestra.run.smithi072.stdout: ms die on skipped message = False 2024-03-24T00:49:24.005 INFO:teuthology.orchestra.run.smithi072.stdout: fsid = de1d0178-808e-49ca-a986-e297e523b976 2024-03-24T00:49:24.005 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:24.005 INFO:teuthology.orchestra.run.smithi072.stdout:[osd] 2024-03-24T00:49:24.005 INFO:teuthology.orchestra.run.smithi072.stdout: osd journal size = 100 2024-03-24T00:49:24.005 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:24.005 INFO:teuthology.orchestra.run.smithi072.stdout: osd scrub load threshold = 5.0 2024-03-24T00:49:24.006 INFO:teuthology.orchestra.run.smithi072.stdout: osd scrub max interval = 600 2024-03-24T00:49:24.006 INFO:teuthology.orchestra.run.smithi072.stdout: osd mclock profile = high_recovery_ops 2024-03-24T00:49:24.006 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:24.006 INFO:teuthology.orchestra.run.smithi072.stdout: osd recover clone overlap = true 2024-03-24T00:49:24.006 INFO:teuthology.orchestra.run.smithi072.stdout: osd recovery max chunk = 1048576 2024-03-24T00:49:24.006 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:24.006 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug shutdown = true 2024-03-24T00:49:24.006 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug op order = true 2024-03-24T00:49:24.006 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug verify stray on activate = true 2024-03-24T00:49:24.006 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug trim objects = true 2024-03-24T00:49:24.006 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:24.006 INFO:teuthology.orchestra.run.smithi072.stdout: osd open classes on start = true 2024-03-24T00:49:24.006 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug pg log writeout = true 2024-03-24T00:49:24.006 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:24.006 INFO:teuthology.orchestra.run.smithi072.stdout: osd deep scrub update digest min age = 30 2024-03-24T00:49:24.007 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:24.007 INFO:teuthology.orchestra.run.smithi072.stdout: osd map max advance = 10 2024-03-24T00:49:24.007 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:24.007 INFO:teuthology.orchestra.run.smithi072.stdout: journal zero on create = true 2024-03-24T00:49:24.007 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:24.007 INFO:teuthology.orchestra.run.smithi072.stdout: filestore ondisk finisher threads = 3 2024-03-24T00:49:24.007 INFO:teuthology.orchestra.run.smithi072.stdout: filestore apply finisher threads = 3 2024-03-24T00:49:24.007 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:24.007 INFO:teuthology.orchestra.run.smithi072.stdout: bdev debug aio = true 2024-03-24T00:49:24.007 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug misdirected ops = true 2024-03-24T00:49:24.007 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore block size = 96636764160 2024-03-24T00:49:24.007 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore fsck on mount = True 2024-03-24T00:49:24.007 INFO:teuthology.orchestra.run.smithi072.stdout: debug bluefs = 20 2024-03-24T00:49:24.007 INFO:teuthology.orchestra.run.smithi072.stdout: debug bluestore = 20 2024-03-24T00:49:24.007 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-24T00:49:24.008 INFO:teuthology.orchestra.run.smithi072.stdout: debug osd = 20 2024-03-24T00:49:24.008 INFO:teuthology.orchestra.run.smithi072.stdout: debug rocksdb = 10 2024-03-24T00:49:24.008 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd backfillfull_ratio = 0.85 2024-03-24T00:49:24.008 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd full ratio = 0.9 2024-03-24T00:49:24.008 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd nearfull ratio = 0.8 2024-03-24T00:49:24.008 INFO:teuthology.orchestra.run.smithi072.stdout: osd failsafe full ratio = 0.95 2024-03-24T00:49:24.008 INFO:teuthology.orchestra.run.smithi072.stdout: osd objectstore = bluestore 2024-03-24T00:49:24.008 INFO:teuthology.orchestra.run.smithi072.stdout: osd op complaint time = 180 2024-03-24T00:49:24.008 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:24.008 INFO:teuthology.orchestra.run.smithi072.stdout:[mgr] 2024-03-24T00:49:24.008 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-24T00:49:24.008 INFO:teuthology.orchestra.run.smithi072.stdout: debug mgr = 20 2024-03-24T00:49:24.008 INFO:teuthology.orchestra.run.smithi072.stdout: debug mon = 20 2024-03-24T00:49:24.008 INFO:teuthology.orchestra.run.smithi072.stdout: debug auth = 20 2024-03-24T00:49:24.008 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min pgs per osd = 4 2024-03-24T00:49:24.008 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min bytes per osd = 10 2024-03-24T00:49:24.009 INFO:teuthology.orchestra.run.smithi072.stdout: mgr/telemetry/nag = false 2024-03-24T00:49:24.009 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:24.009 INFO:teuthology.orchestra.run.smithi072.stdout:[mon] 2024-03-24T00:49:24.009 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-24T00:49:24.009 INFO:teuthology.orchestra.run.smithi072.stdout: debug mon = 20 2024-03-24T00:49:24.009 INFO:teuthology.orchestra.run.smithi072.stdout: debug paxos = 20 2024-03-24T00:49:24.009 INFO:teuthology.orchestra.run.smithi072.stdout: debug auth = 20 2024-03-24T00:49:24.009 INFO:teuthology.orchestra.run.smithi072.stdout: mon data avail warn = 5 2024-03-24T00:49:24.009 INFO:teuthology.orchestra.run.smithi072.stdout: mon mgr mkfs grace = 240 2024-03-24T00:49:24.009 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min pgs per osd = 4 2024-03-24T00:49:24.009 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd reporter subtree level = osd 2024-03-24T00:49:24.009 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd prime pg temp = true 2024-03-24T00:49:24.009 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min bytes per osd = 10 2024-03-24T00:49:24.009 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:24.009 INFO:teuthology.orchestra.run.smithi072.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-24T00:49:24.010 INFO:teuthology.orchestra.run.smithi072.stdout: auth mon ticket ttl = 660 # 11m 2024-03-24T00:49:24.010 INFO:teuthology.orchestra.run.smithi072.stdout: auth service ticket ttl = 240 # 4m 2024-03-24T00:49:24.010 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:24.010 INFO:teuthology.orchestra.run.smithi072.stdout: # don't complain about insecure global_id in the test suite 2024-03-24T00:49:24.010 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-24T00:49:24.010 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-24T00:49:24.010 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:24.010 INFO:teuthology.orchestra.run.smithi072.stdout: # 1m isn't quite enough 2024-03-24T00:49:24.010 INFO:teuthology.orchestra.run.smithi072.stdout: mon_down_mkfs_grace = 2m 2024-03-24T00:49:24.010 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:24.010 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_filestore_osds = false 2024-03-24T00:49:24.010 INFO:teuthology.orchestra.run.smithi072.stdout: mon op complaint time = 120 2024-03-24T00:49:24.010 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-24T00:49:24.010 INFO:teuthology.orchestra.run.smithi072.stdout:[client] 2024-03-24T00:49:24.010 INFO:teuthology.orchestra.run.smithi072.stdout: rgw cache enabled = true 2024-03-24T00:49:24.011 INFO:teuthology.orchestra.run.smithi072.stdout: rgw enable ops log = true 2024-03-24T00:49:24.011 INFO:teuthology.orchestra.run.smithi072.stdout: rgw enable usage log = true 2024-03-24T00:49:24.011 INFO:teuthology.orchestra.run.smithi072.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-24T00:49:24.011 INFO:teuthology.orchestra.run.smithi072.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-24T00:49:24.011 INFO:teuthology.orchestra.run.smithi072.stdout: client mount timeout = 600 2024-03-24T00:49:24.011 INFO:teuthology.orchestra.run.smithi072.stdout: debug client = 20 2024-03-24T00:49:24.011 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-24T00:49:24.011 INFO:teuthology.orchestra.run.smithi072.stdout: pid file = "" 2024-03-24T00:49:24.011 INFO:teuthology.orchestra.run.smithi072.stdout: rados mon op timeout = 900 2024-03-24T00:49:24.011 INFO:teuthology.orchestra.run.smithi072.stdout: rados osd op timeout = 900 2024-03-24T00:49:24.011 INFO:teuthology.orchestra.run.smithi072.stdout:[mon.a] 2024-03-24T00:49:24.011 INFO:teuthology.orchestra.run.smithi072.stdout:[mon.b] 2024-03-24T00:49:24.011 INFO:teuthology.orchestra.run.smithi072.stdout:[mon.c] 2024-03-24T00:49:24.011 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.a] 2024-03-24T00:49:24.011 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.c] 2024-03-24T00:49:24.012 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.b] 2024-03-24T00:49:24.012 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.d] 2024-03-24T00:49:24.012 INFO:teuthology.orchestra.run.smithi072.stdout:[mds] 2024-03-24T00:49:24.012 INFO:teuthology.orchestra.run.smithi072.stdout: debug mds = 20 2024-03-24T00:49:24.012 INFO:teuthology.orchestra.run.smithi072.stdout: debug mds balancer = 20 2024-03-24T00:49:24.012 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-24T00:49:24.012 INFO:teuthology.orchestra.run.smithi072.stdout: mds debug frag = True 2024-03-24T00:49:24.012 INFO:teuthology.orchestra.run.smithi072.stdout: mds debug scatterstat = True 2024-03-24T00:49:24.012 INFO:teuthology.orchestra.run.smithi072.stdout: mds op complaint time = 180 2024-03-24T00:49:24.012 INFO:teuthology.orchestra.run.smithi072.stdout: mds verify scatter = True 2024-03-24T00:49:24.012 INFO:teuthology.orchestra.run.smithi072.stdout: osd op complaint time = 180 2024-03-24T00:49:24.012 INFO:teuthology.orchestra.run.smithi072.stdout: rados mon op timeout = 900 2024-03-24T00:49:24.012 INFO:teuthology.orchestra.run.smithi072.stdout: rados osd op timeout = 900 2024-03-24T00:49:24.014 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-03-24T00:49:24.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.157+0000 7f8586abf640 1 Processor -- start 2024-03-24T00:49:24.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.157+0000 7f8586abf640 1 -- start start 2024-03-24T00:49:24.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.158+0000 7f8586abf640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f858015dea0 0x7f85800d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:24.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.158+0000 7f8586abf640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f858015e280 0x7f85800d3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:24.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.158+0000 7f8586abf640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f85800b5a60 0x7f85800b5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:24.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.158+0000 7f8586abf640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85800b6410 con 0x7f858015dea0 2024-03-24T00:49:24.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.158+0000 7f8586abf640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85800d2250 con 0x7f85800b5a60 2024-03-24T00:49:24.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.158+0000 7f8586abf640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85800d2530 con 0x7f858015e280 2024-03-24T00:49:24.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.158+0000 7f858485b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f858015dea0 0x7f85800d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:24.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.158+0000 7f858485b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f858015dea0 0x7f85800d33d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38480/0 (socket says 172.21.15.67:38480) 2024-03-24T00:49:24.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.158+0000 7f858485b640 1 -- 172.21.15.67:0/3876754536 learned_addr learned my addr 172.21.15.67:0/3876754536 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:24.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.158+0000 7f857ffff640 1 --2- 172.21.15.67:0/3876754536 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f858015e280 0x7f85800d3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:24.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.158+0000 7f858505c640 1 --2- 172.21.15.67:0/3876754536 >> v2:172.21.15.72:3300/0 conn(0x7f85800b5a60 0x7f85800b5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:24.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.158+0000 7f858485b640 1 -- 172.21.15.67:0/3876754536 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f858015e280 msgr2=0x7f85800d3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:24.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.158+0000 7f858485b640 1 --2- 172.21.15.67:0/3876754536 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f858015e280 0x7f85800d3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:24.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.158+0000 7f858485b640 1 -- 172.21.15.67:0/3876754536 >> v2:172.21.15.72:3300/0 conn(0x7f85800b5a60 msgr2=0x7f85800b5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:24.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.158+0000 7f858485b640 1 --2- 172.21.15.67:0/3876754536 >> v2:172.21.15.72:3300/0 conn(0x7f85800b5a60 0x7f85800b5ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:24.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.158+0000 7f858485b640 1 -- 172.21.15.67:0/3876754536 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85800d2810 con 0x7f858015dea0 2024-03-24T00:49:24.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.159+0000 7f858485b640 1 --2- 172.21.15.67:0/3876754536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f858015dea0 0x7f85800d33d0 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f85800d2460 tx=0x7f857802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bff6844122f29188 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:24.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.159+0000 7f857f7fe640 1 -- 172.21.15.67:0/3876754536 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f857802c9e0 con 0x7f858015dea0 2024-03-24T00:49:24.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.159+0000 7f857f7fe640 1 -- 172.21.15.67:0/3876754536 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f857802cb80 con 0x7f858015dea0 2024-03-24T00:49:24.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.159+0000 7f857f7fe640 1 -- 172.21.15.67:0/3876754536 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85780343f0 con 0x7f858015dea0 2024-03-24T00:49:24.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.160+0000 7f8586abf640 1 -- 172.21.15.67:0/3876754536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f858015dea0 msgr2=0x7f85800d33d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:24.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.160+0000 7f8586abf640 1 --2- 172.21.15.67:0/3876754536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f858015dea0 0x7f85800d33d0 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f85800d2460 tx=0x7f857802be40 comp rx=0 tx=0).stop 2024-03-24T00:49:24.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.160+0000 7f8586abf640 1 -- 172.21.15.67:0/3876754536 shutdown_connections 2024-03-24T00:49:24.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.160+0000 7f8586abf640 1 --2- 172.21.15.67:0/3876754536 >> v2:172.21.15.72:3300/0 conn(0x7f85800b5a60 0x7f85800b5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:24.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.160+0000 7f8586abf640 1 --2- 172.21.15.67:0/3876754536 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f858015e280 0x7f85800d3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:24.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.160+0000 7f8586abf640 1 --2- 172.21.15.67:0/3876754536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f858015dea0 0x7f85800d33d0 unknown :-1 s=CLOSED pgs=1849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:24.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.160+0000 7f8586abf640 1 -- 172.21.15.67:0/3876754536 >> 172.21.15.67:0/3876754536 conn(0x7f8580072da0 msgr2=0x7f85800d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:24.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.160+0000 7f8586abf640 1 -- 172.21.15.67:0/3876754536 shutdown_connections 2024-03-24T00:49:24.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.160+0000 7f8586abf640 1 -- 172.21.15.67:0/3876754536 wait complete. 2024-03-24T00:49:24.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.161+0000 7f8586abf640 1 Processor -- start 2024-03-24T00:49:24.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.161+0000 7f8586abf640 1 -- start start 2024-03-24T00:49:24.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.162+0000 7f8586abf640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f85800b5a60 0x7f85801524d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:24.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.162+0000 7f8586abf640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f858015dea0 0x7f85800aa990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:24.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.162+0000 7f8586abf640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f858015e280 0x7f85800ac310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:24.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.162+0000 7f8586abf640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f858006c7c0 con 0x7f858015e280 2024-03-24T00:49:24.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.162+0000 7f8586abf640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f858006c640 con 0x7f858015dea0 2024-03-24T00:49:24.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.162+0000 7f8586abf640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f858006c940 con 0x7f85800b5a60 2024-03-24T00:49:24.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.162+0000 7f858505c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f858015e280 0x7f85800ac310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:24.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.162+0000 7f858485b640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f85800b5a60 0x7f85801524d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:24.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.162+0000 7f858505c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f858015e280 0x7f85800ac310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38484/0 (socket says 172.21.15.67:38484) 2024-03-24T00:49:24.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.162+0000 7f858505c640 1 -- 172.21.15.67:0/2101356940 learned_addr learned my addr 172.21.15.67:0/2101356940 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:24.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.162+0000 7f857ffff640 1 --2- 172.21.15.67:0/2101356940 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f858015dea0 0x7f85800aa990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:24.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.162+0000 7f858505c640 1 -- 172.21.15.67:0/2101356940 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f85800b5a60 msgr2=0x7f85801524d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:24.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.162+0000 7f858505c640 1 --2- 172.21.15.67:0/2101356940 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f85800b5a60 0x7f85801524d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:24.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.162+0000 7f858505c640 1 -- 172.21.15.67:0/2101356940 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f858015dea0 msgr2=0x7f85800aa990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:24.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.162+0000 7f858505c640 1 --2- 172.21.15.67:0/2101356940 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f858015dea0 0x7f85800aa990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:24.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.162+0000 7f858505c640 1 -- 172.21.15.67:0/2101356940 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85800ab270 con 0x7f858015e280 2024-03-24T00:49:24.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.163+0000 7f858505c640 1 --2- 172.21.15.67:0/2101356940 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f858015e280 0x7f85800ac310 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f857000d580 tx=0x7f857000da40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:24.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.163+0000 7f857d7fa640 1 -- 172.21.15.67:0/2101356940 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8570018020 con 0x7f858015e280 2024-03-24T00:49:24.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.163+0000 7f857d7fa640 1 -- 172.21.15.67:0/2101356940 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8570014620 con 0x7f858015e280 2024-03-24T00:49:24.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.163+0000 7f8586abf640 1 -- 172.21.15.67:0/2101356940 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85800ab550 con 0x7f858015e280 2024-03-24T00:49:24.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.163+0000 7f857d7fa640 1 -- 172.21.15.67:0/2101356940 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85700149f0 con 0x7f858015e280 2024-03-24T00:49:24.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.163+0000 7f8586abf640 1 -- 172.21.15.67:0/2101356940 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85800abb10 con 0x7f858015e280 2024-03-24T00:49:24.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.164+0000 7f8586abf640 1 -- 172.21.15.67:0/2101356940 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f858004f400 con 0x7f858015e280 2024-03-24T00:49:24.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.164+0000 7f857d7fa640 1 -- 172.21.15.67:0/2101356940 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f857001b020 con 0x7f858015e280 2024-03-24T00:49:24.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.165+0000 7f857d7fa640 1 --2- 172.21.15.67:0/2101356940 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f856c06bbd0 0x7f856c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:24.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.165+0000 7f857d7fa640 1 -- 172.21.15.67:0/2101356940 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f857008ad70 con 0x7f858015e280 2024-03-24T00:49:24.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.165+0000 7f858485b640 1 --2- 172.21.15.67:0/2101356940 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f856c06bbd0 0x7f856c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:24.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.165+0000 7f858485b640 1 --2- 172.21.15.67:0/2101356940 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f856c06bbd0 0x7f856c06e090 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f8578035000 tx=0x7f8578030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:24.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.169+0000 7f857d7fa640 1 -- 172.21.15.67:0/2101356940 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8570022030 con 0x7f858015e280 2024-03-24T00:49:24.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:24.394+0000 7f8586abf640 1 -- 172.21.15.67:0/2101356940 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) v1 -- 0x7f85800b9b00 con 0x7f858015e280 2024-03-24T00:49:25.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.322+0000 7f857d7fa640 1 -- 172.21.15.67:0/2101356940 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v95) v1 ==== 162+0+0 (secure 0 0 0) 0x7f8570058fe0 con 0x7f858015e280 2024-03-24T00:49:25.323 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked joinable; MDS may join as newly active. 2024-03-24T00:49:25.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.325+0000 7f8586abf640 1 -- 172.21.15.67:0/2101356940 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f856c06bbd0 msgr2=0x7f856c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:25.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.325+0000 7f8586abf640 1 --2- 172.21.15.67:0/2101356940 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f856c06bbd0 0x7f856c06e090 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f8578035000 tx=0x7f8578030000 comp rx=0 tx=0).stop 2024-03-24T00:49:25.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.325+0000 7f8586abf640 1 -- 172.21.15.67:0/2101356940 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f858015e280 msgr2=0x7f85800ac310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:25.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.325+0000 7f8586abf640 1 --2- 172.21.15.67:0/2101356940 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f858015e280 0x7f85800ac310 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f857000d580 tx=0x7f857000da40 comp rx=0 tx=0).stop 2024-03-24T00:49:25.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.325+0000 7f8586abf640 1 -- 172.21.15.67:0/2101356940 shutdown_connections 2024-03-24T00:49:25.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.325+0000 7f8586abf640 1 --2- 172.21.15.67:0/2101356940 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f856c06bbd0 0x7f856c06e090 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:25.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.325+0000 7f8586abf640 1 --2- 172.21.15.67:0/2101356940 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f858015e280 0x7f85800ac310 unknown :-1 s=CLOSED pgs=1850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:25.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.325+0000 7f8586abf640 1 --2- 172.21.15.67:0/2101356940 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f858015dea0 0x7f85800aa990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:25.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.325+0000 7f8586abf640 1 --2- 172.21.15.67:0/2101356940 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f85800b5a60 0x7f85801524d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:25.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.325+0000 7f8586abf640 1 -- 172.21.15.67:0/2101356940 >> 172.21.15.67:0/2101356940 conn(0x7f8580072da0 msgr2=0x7f85800b6950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:25.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.325+0000 7f8586abf640 1 -- 172.21.15.67:0/2101356940 shutdown_connections 2024-03-24T00:49:25.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.325+0000 7f8586abf640 1 -- 172.21.15.67:0/2101356940 wait complete. 2024-03-24T00:49:25.342 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:49:25.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.448+0000 7f46708a1640 1 Processor -- start 2024-03-24T00:49:25.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.449+0000 7f46708a1640 1 -- start start 2024-03-24T00:49:25.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.449+0000 7f46708a1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f466c15dec0 0x7f466c0ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:25.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.449+0000 7f46708a1640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f466c15e2a0 0x7f466c162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:25.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.449+0000 7f46708a1640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f466c0d07e0 0x7f466c0d0c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:25.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.449+0000 7f46708a1640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f466c0d1190 con 0x7f466c15e2a0 2024-03-24T00:49:25.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.449+0000 7f46708a1640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f466c0cee30 con 0x7f466c0d07e0 2024-03-24T00:49:25.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.449+0000 7f46708a1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f466c0d2120 con 0x7f466c15dec0 2024-03-24T00:49:25.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.450+0000 7f4669d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f466c15e2a0 0x7f466c162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:25.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.450+0000 7f4669d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f466c15e2a0 0x7f466c162630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38494/0 (socket says 172.21.15.67:38494) 2024-03-24T00:49:25.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.450+0000 7f4669d9b640 1 -- 172.21.15.67:0/2582243147 learned_addr learned my addr 172.21.15.67:0/2582243147 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:25.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.450+0000 7f466ad9d640 1 --2- 172.21.15.67:0/2582243147 >> v2:172.21.15.72:3300/0 conn(0x7f466c0d07e0 0x7f466c0d0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:25.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.450+0000 7f466a59c640 1 --2- 172.21.15.67:0/2582243147 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f466c15dec0 0x7f466c0ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:25.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.450+0000 7f4669d9b640 1 -- 172.21.15.67:0/2582243147 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f466c15dec0 msgr2=0x7f466c0ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:25.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.450+0000 7f4669d9b640 1 --2- 172.21.15.67:0/2582243147 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f466c15dec0 0x7f466c0ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:25.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.450+0000 7f4669d9b640 1 -- 172.21.15.67:0/2582243147 >> v2:172.21.15.72:3300/0 conn(0x7f466c0d07e0 msgr2=0x7f466c0d0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:25.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.450+0000 7f4669d9b640 1 --2- 172.21.15.67:0/2582243147 >> v2:172.21.15.72:3300/0 conn(0x7f466c0d07e0 0x7f466c0d0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:25.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.450+0000 7f4669d9b640 1 -- 172.21.15.67:0/2582243147 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f466c0cf640 con 0x7f466c15e2a0 2024-03-24T00:49:25.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.451+0000 7f4669d9b640 1 --2- 172.21.15.67:0/2582243147 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f466c15e2a0 0x7f466c162630 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f466c0d1a50 tx=0x7f465402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdce98cb15ec23ce server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:25.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.451+0000 7f466959a640 1 -- 172.21.15.67:0/2582243147 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f465402e070 con 0x7f466c15e2a0 2024-03-24T00:49:25.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.451+0000 7f466959a640 1 -- 172.21.15.67:0/2582243147 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4654037070 con 0x7f466c15e2a0 2024-03-24T00:49:25.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.451+0000 7f466959a640 1 -- 172.21.15.67:0/2582243147 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4654031a00 con 0x7f466c15e2a0 2024-03-24T00:49:25.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.451+0000 7f46708a1640 1 -- 172.21.15.67:0/2582243147 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f466c15e2a0 msgr2=0x7f466c162630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:25.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.451+0000 7f46708a1640 1 --2- 172.21.15.67:0/2582243147 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f466c15e2a0 0x7f466c162630 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f466c0d1a50 tx=0x7f465402b880 comp rx=0 tx=0).stop 2024-03-24T00:49:25.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.452+0000 7f46708a1640 1 -- 172.21.15.67:0/2582243147 shutdown_connections 2024-03-24T00:49:25.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.452+0000 7f46708a1640 1 --2- 172.21.15.67:0/2582243147 >> v2:172.21.15.72:3300/0 conn(0x7f466c0d07e0 0x7f466c0d0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:25.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.452+0000 7f46708a1640 1 --2- 172.21.15.67:0/2582243147 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f466c15e2a0 0x7f466c162630 unknown :-1 s=CLOSED pgs=1851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:25.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.452+0000 7f46708a1640 1 --2- 172.21.15.67:0/2582243147 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f466c15dec0 0x7f466c0ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:25.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.452+0000 7f46708a1640 1 -- 172.21.15.67:0/2582243147 >> 172.21.15.67:0/2582243147 conn(0x7f466c06c810 msgr2=0x7f466c06cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:25.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.452+0000 7f46708a1640 1 -- 172.21.15.67:0/2582243147 shutdown_connections 2024-03-24T00:49:25.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.452+0000 7f46708a1640 1 -- 172.21.15.67:0/2582243147 wait complete. 2024-03-24T00:49:25.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.453+0000 7f46708a1640 1 Processor -- start 2024-03-24T00:49:25.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.453+0000 7f46708a1640 1 -- start start 2024-03-24T00:49:25.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.454+0000 7f46708a1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f466c0d07e0 0x7f466c160870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:25.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.454+0000 7f46708a1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f466c15dec0 0x7f466c0cc930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:25.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.454+0000 7f46708a1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f466c15e2a0 0x7f466c0cce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:25.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.454+0000 7f46708a1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466c0b5850 con 0x7f466c15dec0 2024-03-24T00:49:25.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.454+0000 7f46708a1640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466c0cf360 con 0x7f466c0d07e0 2024-03-24T00:49:25.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.454+0000 7f46708a1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f466c0b59d0 con 0x7f466c15e2a0 2024-03-24T00:49:25.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.454+0000 7f4669d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f466c15dec0 0x7f466c0cc930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:25.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.454+0000 7f4669d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f466c15dec0 0x7f466c0cc930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38510/0 (socket says 172.21.15.67:38510) 2024-03-24T00:49:25.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.454+0000 7f4669d9b640 1 -- 172.21.15.67:0/1918253460 learned_addr learned my addr 172.21.15.67:0/1918253460 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:25.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.454+0000 7f466a59c640 1 --2- 172.21.15.67:0/1918253460 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f466c0d07e0 0x7f466c160870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:25.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.454+0000 7f466ad9d640 1 --2- 172.21.15.67:0/1918253460 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f466c15e2a0 0x7f466c0cce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:25.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.454+0000 7f4669d9b640 1 -- 172.21.15.67:0/1918253460 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f466c15e2a0 msgr2=0x7f466c0cce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:25.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.454+0000 7f4669d9b640 1 --2- 172.21.15.67:0/1918253460 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f466c15e2a0 0x7f466c0cce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:25.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.454+0000 7f4669d9b640 1 -- 172.21.15.67:0/1918253460 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f466c0d07e0 msgr2=0x7f466c160870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:25.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.454+0000 7f4669d9b640 1 --2- 172.21.15.67:0/1918253460 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f466c0d07e0 0x7f466c160870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:25.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.454+0000 7f4669d9b640 1 -- 172.21.15.67:0/1918253460 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f466c15f1c0 con 0x7f466c15dec0 2024-03-24T00:49:25.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.454+0000 7f466ad9d640 1 --2- 172.21.15.67:0/1918253460 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f466c15e2a0 0x7f466c0cce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:25.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.454+0000 7f4669d9b640 1 --2- 172.21.15.67:0/1918253460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f466c15dec0 0x7f466c0cc930 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f465402b8c0 tx=0x7f46540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:25.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.454+0000 7f4662ffd640 1 -- 172.21.15.67:0/1918253460 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4654032880 con 0x7f466c15dec0 2024-03-24T00:49:25.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.454+0000 7f4662ffd640 1 -- 172.21.15.67:0/1918253460 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4654032a40 con 0x7f466c15dec0 2024-03-24T00:49:25.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.454+0000 7f46708a1640 1 -- 172.21.15.67:0/1918253460 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f466c15f4a0 con 0x7f466c15dec0 2024-03-24T00:49:25.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.454+0000 7f4662ffd640 1 -- 172.21.15.67:0/1918253460 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4654032e30 con 0x7f466c15dec0 2024-03-24T00:49:25.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.454+0000 7f46708a1640 1 -- 172.21.15.67:0/1918253460 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f466c15fa60 con 0x7f466c15dec0 2024-03-24T00:49:25.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.455+0000 7f46708a1640 1 -- 172.21.15.67:0/1918253460 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f466c04f400 con 0x7f466c15dec0 2024-03-24T00:49:25.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.456+0000 7f4662ffd640 1 -- 172.21.15.67:0/1918253460 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f46540453f0 con 0x7f466c15dec0 2024-03-24T00:49:25.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.456+0000 7f4662ffd640 1 --2- 172.21.15.67:0/1918253460 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f464c06bbd0 0x7f464c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:25.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.456+0000 7f4662ffd640 1 -- 172.21.15.67:0/1918253460 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f46540acd60 con 0x7f466c15dec0 2024-03-24T00:49:25.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.457+0000 7f466a59c640 1 --2- 172.21.15.67:0/1918253460 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f464c06bbd0 0x7f464c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:25.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.457+0000 7f466a59c640 1 --2- 172.21.15.67:0/1918253460 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f464c06bbd0 0x7f464c06e090 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f465c0099e0 tx=0x7f465c003cd0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:25.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.461+0000 7f4662ffd640 1 -- 172.21.15.67:0/1918253460 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4654040070 con 0x7f466c15dec0 2024-03-24T00:49:25.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.680+0000 7f46708a1640 1 -- 172.21.15.67:0/1918253460 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f466c0d37e0 con 0x7f466c15dec0 2024-03-24T00:49:25.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.680+0000 7f4662ffd640 1 -- 172.21.15.67:0/1918253460 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+4968 (secure 0 0 0) 0x7f466c0cf640 con 0x7f466c15dec0 2024-03-24T00:49:25.681 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:25.682 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":96,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7298,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/839277571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":839277571},{"type":"v1","addr":"172.21.15.72:6836","nonce":839277571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":8224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2052380377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2052380377},{"type":"v1","addr":"172.21.15.67:6834","nonce":2052380377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":8239,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2196618204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2196618204},{"type":"v1","addr":"172.21.15.72:6837","nonce":2196618204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:48:54.351575+0000","modified":"2024-03-24T00:49:25.326527+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":122,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9666},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9666":{"gid":9666,"name":"a","rank":0,"incarnation":96,"state":"up:replay","state_seq":1,"addr":"172.21.15.67:6835/693801918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":693801918},{"type":"v1","addr":"172.21.15.67:6835","nonce":693801918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":8}]} 2024-03-24T00:49:25.682 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 96 2024-03-24T00:49:25.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.682+0000 7f46708a1640 1 -- 172.21.15.67:0/1918253460 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f464c06bbd0 msgr2=0x7f464c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:25.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.682+0000 7f46708a1640 1 --2- 172.21.15.67:0/1918253460 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f464c06bbd0 0x7f464c06e090 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f465c0099e0 tx=0x7f465c003cd0 comp rx=0 tx=0).stop 2024-03-24T00:49:25.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.682+0000 7f46708a1640 1 -- 172.21.15.67:0/1918253460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f466c15dec0 msgr2=0x7f466c0cc930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:25.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.682+0000 7f46708a1640 1 --2- 172.21.15.67:0/1918253460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f466c15dec0 0x7f466c0cc930 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f465402b8c0 tx=0x7f46540046c0 comp rx=0 tx=0).stop 2024-03-24T00:49:25.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.682+0000 7f46708a1640 1 -- 172.21.15.67:0/1918253460 shutdown_connections 2024-03-24T00:49:25.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.682+0000 7f46708a1640 1 --2- 172.21.15.67:0/1918253460 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f464c06bbd0 0x7f464c06e090 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:25.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.682+0000 7f46708a1640 1 --2- 172.21.15.67:0/1918253460 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f466c15e2a0 0x7f466c0cce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:25.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.682+0000 7f46708a1640 1 --2- 172.21.15.67:0/1918253460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f466c15dec0 0x7f466c0cc930 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:25.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.682+0000 7f46708a1640 1 --2- 172.21.15.67:0/1918253460 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f466c0d07e0 0x7f466c160870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:25.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.682+0000 7f46708a1640 1 -- 172.21.15.67:0/1918253460 >> 172.21.15.67:0/1918253460 conn(0x7f466c06c810 msgr2=0x7f466c0b4e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:25.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.682+0000 7f46708a1640 1 -- 172.21.15.67:0/1918253460 shutdown_connections 2024-03-24T00:49:25.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:25.682+0000 7f46708a1640 1 -- 172.21.15.67:0/1918253460 wait complete. 2024-03-24T00:49:26.096 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 96, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:48:54.351575+0000', 'modified': '2024-03-24T00:49:25.326527+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 122, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9666}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9666': {'gid': 9666, 'name': 'a', 'rank': 0, 'incarnation': 96, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.67:6835/693801918', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6833', 'nonce': 693801918}, {'type': 'v1', 'addr': '172.21.15.67:6835', 'nonce': 693801918}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-24T00:49:26.097 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9666:up:replay 2024-03-24T00:49:27.098 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:49:27.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.211+0000 7f4b36aac640 1 Processor -- start 2024-03-24T00:49:27.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.211+0000 7f4b36aac640 1 -- start start 2024-03-24T00:49:27.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.211+0000 7f4b36aac640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4b3015dec0 0x7f4b300ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:27.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.211+0000 7f4b36aac640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4b3015e2a0 0x7f4b30162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:27.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.211+0000 7f4b36aac640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b300cf640 con 0x7f4b300d07e0 2024-03-24T00:49:27.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.211+0000 7f4b36aac640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b300cee30 con 0x7f4b3015dec0 2024-03-24T00:49:27.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.211+0000 7f4b36aac640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b300d2120 con 0x7f4b3015e2a0 2024-03-24T00:49:27.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.212+0000 7f4b35049640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4b300d07e0 0x7f4b300b5850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50782/0 (socket says 172.21.15.67:50782) 2024-03-24T00:49:27.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.212+0000 7f4b35049640 1 -- 172.21.15.67:0/1963344041 learned_addr learned my addr 172.21.15.67:0/1963344041 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:49:27.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.212+0000 7f4b2ffff640 1 --2- 172.21.15.67:0/1963344041 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4b3015e2a0 0x7f4b30162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:27.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.212+0000 7f4b34848640 1 --2- 172.21.15.67:0/1963344041 >> v2:172.21.15.72:3300/0 conn(0x7f4b3015dec0 0x7f4b300ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:27.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.212+0000 7f4b2f7fe640 1 -- 172.21.15.67:0/1963344041 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2399508685 0 0) 0x7f4b20001110 con 0x7f4b300d07e0 2024-03-24T00:49:27.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.212+0000 7f4b2f7fe640 1 -- 172.21.15.67:0/1963344041 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b18003520 con 0x7f4b300d07e0 2024-03-24T00:49:27.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.212+0000 7f4b2f7fe640 1 -- 172.21.15.67:0/1963344041 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2507372783 0 0) 0x7f4b18003520 con 0x7f4b300d07e0 2024-03-24T00:49:27.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.212+0000 7f4b2ffff640 1 -- 172.21.15.67:0/1963344041 >> v2:172.21.15.72:3300/0 conn(0x7f4b3015dec0 msgr2=0x7f4b300ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:27.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.212+0000 7f4b2ffff640 1 --2- 172.21.15.67:0/1963344041 >> v2:172.21.15.72:3300/0 conn(0x7f4b3015dec0 0x7f4b300ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:27.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.212+0000 7f4b2ffff640 1 -- 172.21.15.67:0/1963344041 >> v1:172.21.15.67:6789/0 conn(0x7f4b300d07e0 legacy=0x7f4b300b5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:27.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.212+0000 7f4b2ffff640 1 -- 172.21.15.67:0/1963344041 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b300cf080 con 0x7f4b3015e2a0 2024-03-24T00:49:27.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.212+0000 7f4b34848640 1 --2- 172.21.15.67:0/1963344041 >> v2:172.21.15.72:3300/0 conn(0x7f4b3015dec0 0x7f4b300ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:49:27.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.213+0000 7f4b2ffff640 1 --2- 172.21.15.67:0/1963344041 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4b3015e2a0 0x7f4b30162630 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4b300cefb0 tx=0x7f4b2402b6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3d57e5a7b623f70 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:27.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.213+0000 7f4b2f7fe640 1 -- 172.21.15.67:0/1963344041 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b24037070 con 0x7f4b3015e2a0 2024-03-24T00:49:27.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.213+0000 7f4b2f7fe640 1 -- 172.21.15.67:0/1963344041 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b24030510 con 0x7f4b3015e2a0 2024-03-24T00:49:27.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.213+0000 7f4b2f7fe640 1 -- 172.21.15.67:0/1963344041 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b20001840 con 0x7f4b3015e2a0 2024-03-24T00:49:27.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.214+0000 7f4b36aac640 1 -- 172.21.15.67:0/1963344041 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4b3015e2a0 msgr2=0x7f4b30162630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:27.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.214+0000 7f4b36aac640 1 --2- 172.21.15.67:0/1963344041 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4b3015e2a0 0x7f4b30162630 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4b300cefb0 tx=0x7f4b2402b6e0 comp rx=0 tx=0).stop 2024-03-24T00:49:27.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.214+0000 7f4b36aac640 1 -- 172.21.15.67:0/1963344041 shutdown_connections 2024-03-24T00:49:27.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.214+0000 7f4b36aac640 1 --2- 172.21.15.67:0/1963344041 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4b3015e2a0 0x7f4b30162630 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:27.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.214+0000 7f4b36aac640 1 --2- 172.21.15.67:0/1963344041 >> v2:172.21.15.72:3300/0 conn(0x7f4b3015dec0 0x7f4b300ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:27.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.214+0000 7f4b36aac640 1 -- 172.21.15.67:0/1963344041 >> 172.21.15.67:0/1963344041 conn(0x7f4b3006c810 msgr2=0x7f4b3006cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:27.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.214+0000 7f4b36aac640 1 -- 172.21.15.67:0/1963344041 shutdown_connections 2024-03-24T00:49:27.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.214+0000 7f4b36aac640 1 -- 172.21.15.67:0/1963344041 wait complete. 2024-03-24T00:49:27.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.215+0000 7f4b36aac640 1 Processor -- start 2024-03-24T00:49:27.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.215+0000 7f4b36aac640 1 -- start start 2024-03-24T00:49:27.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.216+0000 7f4b36aac640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b300d07e0 0x7f4b301f3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:27.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.216+0000 7f4b36aac640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4b3015dec0 0x7f4b30153f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:27.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.216+0000 7f4b36aac640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b3015e2a0 0x7f4b301544d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:27.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.216+0000 7f4b36aac640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b300d3740 con 0x7f4b3015e2a0 2024-03-24T00:49:27.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.216+0000 7f4b36aac640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b300d35c0 con 0x7f4b300d07e0 2024-03-24T00:49:27.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.216+0000 7f4b36aac640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b300d38c0 con 0x7f4b3015dec0 2024-03-24T00:49:27.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.216+0000 7f4b35049640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b3015e2a0 0x7f4b301544d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:27.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.216+0000 7f4b35049640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b3015e2a0 0x7f4b301544d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38512/0 (socket says 172.21.15.67:38512) 2024-03-24T00:49:27.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.216+0000 7f4b35049640 1 -- 172.21.15.67:0/2364441812 learned_addr learned my addr 172.21.15.67:0/2364441812 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:27.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.216+0000 7f4b34848640 1 --2- 172.21.15.67:0/2364441812 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b300d07e0 0x7f4b301f3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:27.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.216+0000 7f4b2ffff640 1 --2- 172.21.15.67:0/2364441812 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4b3015dec0 0x7f4b30153f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:27.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.216+0000 7f4b35049640 1 -- 172.21.15.67:0/2364441812 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4b3015dec0 msgr2=0x7f4b30153f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:27.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.216+0000 7f4b35049640 1 --2- 172.21.15.67:0/2364441812 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4b3015dec0 0x7f4b30153f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:27.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.216+0000 7f4b35049640 1 -- 172.21.15.67:0/2364441812 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b300d07e0 msgr2=0x7f4b301f3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:27.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.216+0000 7f4b35049640 1 --2- 172.21.15.67:0/2364441812 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b300d07e0 0x7f4b301f3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:27.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.216+0000 7f4b35049640 1 -- 172.21.15.67:0/2364441812 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b301f3e70 con 0x7f4b3015e2a0 2024-03-24T00:49:27.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.217+0000 7f4b35049640 1 --2- 172.21.15.67:0/2364441812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b3015e2a0 0x7f4b301544d0 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7f4b2000c780 tx=0x7f4b2000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:27.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.217+0000 7f4b2d7fa640 1 -- 172.21.15.67:0/2364441812 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b20005ad0 con 0x7f4b3015e2a0 2024-03-24T00:49:27.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.217+0000 7f4b2d7fa640 1 -- 172.21.15.67:0/2364441812 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b20005c90 con 0x7f4b3015e2a0 2024-03-24T00:49:27.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.217+0000 7f4b36aac640 1 -- 172.21.15.67:0/2364441812 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b30154b00 con 0x7f4b3015e2a0 2024-03-24T00:49:27.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.217+0000 7f4b2d7fa640 1 -- 172.21.15.67:0/2364441812 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b200072b0 con 0x7f4b3015e2a0 2024-03-24T00:49:27.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.217+0000 7f4b36aac640 1 -- 172.21.15.67:0/2364441812 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b30152f30 con 0x7f4b3015e2a0 2024-03-24T00:49:27.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.218+0000 7f4b36aac640 1 -- 172.21.15.67:0/2364441812 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b3004f400 con 0x7f4b3015e2a0 2024-03-24T00:49:27.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.218+0000 7f4b2d7fa640 1 -- 172.21.15.67:0/2364441812 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4b20007450 con 0x7f4b3015e2a0 2024-03-24T00:49:27.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.219+0000 7f4b2d7fa640 1 --2- 172.21.15.67:0/2364441812 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4b1c06bbd0 0x7f4b1c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:27.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.219+0000 7f4b2d7fa640 1 -- 172.21.15.67:0/2364441812 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f4b2008a950 con 0x7f4b3015e2a0 2024-03-24T00:49:27.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.219+0000 7f4b34848640 1 --2- 172.21.15.67:0/2364441812 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4b1c06bbd0 0x7f4b1c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:27.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.219+0000 7f4b34848640 1 --2- 172.21.15.67:0/2364441812 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4b1c06bbd0 0x7f4b1c06e090 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f4b28009bd0 tx=0x7f4b2800b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:27.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.223+0000 7f4b2d7fa640 1 -- 172.21.15.67:0/2364441812 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4b300cf640 con 0x7f4b3015e2a0 2024-03-24T00:49:27.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.444+0000 7f4b36aac640 1 -- 172.21.15.67:0/2364441812 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b30160460 con 0x7f4b3015e2a0 2024-03-24T00:49:27.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.445+0000 7f4b2d7fa640 1 -- 172.21.15.67:0/2364441812 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+4968 (secure 0 0 0) 0x7f4b20058bb0 con 0x7f4b3015e2a0 2024-03-24T00:49:27.446 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:27.446 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":98,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7298,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/839277571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":839277571},{"type":"v1","addr":"172.21.15.72:6836","nonce":839277571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":8224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2052380377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2052380377},{"type":"v1","addr":"172.21.15.67:6834","nonce":2052380377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":8239,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2196618204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2196618204},{"type":"v1","addr":"172.21.15.72:6837","nonce":2196618204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:48:54.351575+0000","modified":"2024-03-24T00:49:26.340166+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":122,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9666},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9666":{"gid":9666,"name":"a","rank":0,"incarnation":96,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.67:6835/693801918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":693801918},{"type":"v1","addr":"172.21.15.67:6835","nonce":693801918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":8}]} 2024-03-24T00:49:27.446 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 98 2024-03-24T00:49:27.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.447+0000 7f4b36aac640 1 -- 172.21.15.67:0/2364441812 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4b1c06bbd0 msgr2=0x7f4b1c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:27.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.447+0000 7f4b36aac640 1 --2- 172.21.15.67:0/2364441812 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4b1c06bbd0 0x7f4b1c06e090 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f4b28009bd0 tx=0x7f4b2800b040 comp rx=0 tx=0).stop 2024-03-24T00:49:27.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.447+0000 7f4b36aac640 1 -- 172.21.15.67:0/2364441812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b3015e2a0 msgr2=0x7f4b301544d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:27.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.447+0000 7f4b36aac640 1 --2- 172.21.15.67:0/2364441812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b3015e2a0 0x7f4b301544d0 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7f4b2000c780 tx=0x7f4b2000cc40 comp rx=0 tx=0).stop 2024-03-24T00:49:27.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.447+0000 7f4b36aac640 1 -- 172.21.15.67:0/2364441812 shutdown_connections 2024-03-24T00:49:27.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.447+0000 7f4b36aac640 1 --2- 172.21.15.67:0/2364441812 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4b1c06bbd0 0x7f4b1c06e090 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:27.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.447+0000 7f4b36aac640 1 --2- 172.21.15.67:0/2364441812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b3015e2a0 0x7f4b301544d0 unknown :-1 s=CLOSED pgs=1854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:27.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.447+0000 7f4b36aac640 1 --2- 172.21.15.67:0/2364441812 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4b3015dec0 0x7f4b30153f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:27.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.447+0000 7f4b36aac640 1 --2- 172.21.15.67:0/2364441812 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b300d07e0 0x7f4b301f3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:27.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.447+0000 7f4b36aac640 1 -- 172.21.15.67:0/2364441812 >> 172.21.15.67:0/2364441812 conn(0x7f4b3006c810 msgr2=0x7f4b301564d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:27.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.447+0000 7f4b36aac640 1 -- 172.21.15.67:0/2364441812 shutdown_connections 2024-03-24T00:49:27.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:27.447+0000 7f4b36aac640 1 -- 172.21.15.67:0/2364441812 wait complete. 2024-03-24T00:49:27.862 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 98, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:48:54.351575+0000', 'modified': '2024-03-24T00:49:26.340166+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 122, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9666}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9666': {'gid': 9666, 'name': 'a', 'rank': 0, 'incarnation': 96, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.67:6835/693801918', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6833', 'nonce': 693801918}, {'type': 'v1', 'addr': '172.21.15.67:6835', 'nonce': 693801918}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-24T00:49:27.862 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9666:up:rejoin 2024-03-24T00:49:28.863 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:49:28.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.965+0000 7f17779b2640 1 Processor -- start 2024-03-24T00:49:28.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.965+0000 7f17779b2640 1 -- start start 2024-03-24T00:49:28.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.966+0000 7f17779b2640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f177015dec0 0x7f17700ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:28.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.966+0000 7f17779b2640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f17700d0ef0 0x7f1770162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:28.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.966+0000 7f17779b2640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17700cf640 con 0x7f177015e2a0 2024-03-24T00:49:28.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.966+0000 7f17779b2640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17700cee30 con 0x7f177015dec0 2024-03-24T00:49:28.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.966+0000 7f17779b2640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17700d2120 con 0x7f17700d0ef0 2024-03-24T00:49:28.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.967+0000 7f177574e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f177015dec0 0x7f17700ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:28.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.967+0000 7f1774f4d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f177015e2a0 0x7f17700d07e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50790/0 (socket says 172.21.15.67:50790) 2024-03-24T00:49:28.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.967+0000 7f1774f4d640 1 -- 172.21.15.67:0/1309624189 learned_addr learned my addr 172.21.15.67:0/1309624189 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:49:28.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.967+0000 7f1775f4f640 1 --2- 172.21.15.67:0/1309624189 >> v2:172.21.15.72:3300/0 conn(0x7f17700d0ef0 0x7f1770162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:28.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.967+0000 7f177574e640 1 -- 172.21.15.67:0/1309624189 >> v2:172.21.15.72:3300/0 conn(0x7f17700d0ef0 msgr2=0x7f1770162630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:28.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.967+0000 7f177574e640 1 --2- 172.21.15.67:0/1309624189 >> v2:172.21.15.72:3300/0 conn(0x7f17700d0ef0 0x7f1770162630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:28.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.967+0000 7f177574e640 1 -- 172.21.15.67:0/1309624189 >> v1:172.21.15.67:6789/0 conn(0x7f177015e2a0 legacy=0x7f17700d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:28.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.967+0000 7f1767fff640 1 -- 172.21.15.67:0/1309624189 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1418839468 0 0) 0x7f1760001110 con 0x7f177015e2a0 2024-03-24T00:49:28.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.967+0000 7f177574e640 1 -- 172.21.15.67:0/1309624189 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17700cf080 con 0x7f177015dec0 2024-03-24T00:49:28.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.967+0000 7f1775f4f640 1 --2- 172.21.15.67:0/1309624189 >> v2:172.21.15.72:3300/0 conn(0x7f17700d0ef0 0x7f1770162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:28.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.968+0000 7f177574e640 1 --2- 172.21.15.67:0/1309624189 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f177015dec0 0x7f17700ce630 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f17700cae20 tx=0x7f176c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3979db96f8987ce7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:28.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.968+0000 7f1767fff640 1 -- 172.21.15.67:0/1309624189 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f176c0315a0 con 0x7f177015dec0 2024-03-24T00:49:28.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.968+0000 7f1767fff640 1 -- 172.21.15.67:0/1309624189 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f176c037070 con 0x7f177015dec0 2024-03-24T00:49:28.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.968+0000 7f1767fff640 1 -- 172.21.15.67:0/1309624189 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f176c031960 con 0x7f177015dec0 2024-03-24T00:49:28.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.969+0000 7f17779b2640 1 -- 172.21.15.67:0/1309624189 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f177015dec0 msgr2=0x7f17700ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:28.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.969+0000 7f17779b2640 1 --2- 172.21.15.67:0/1309624189 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f177015dec0 0x7f17700ce630 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f17700cae20 tx=0x7f176c02b840 comp rx=0 tx=0).stop 2024-03-24T00:49:28.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.969+0000 7f17779b2640 1 -- 172.21.15.67:0/1309624189 shutdown_connections 2024-03-24T00:49:28.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.969+0000 7f17779b2640 1 --2- 172.21.15.67:0/1309624189 >> v2:172.21.15.72:3300/0 conn(0x7f17700d0ef0 0x7f1770162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:28.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.969+0000 7f17779b2640 1 --2- 172.21.15.67:0/1309624189 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f177015dec0 0x7f17700ce630 unknown :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:28.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.969+0000 7f17779b2640 1 -- 172.21.15.67:0/1309624189 >> 172.21.15.67:0/1309624189 conn(0x7f177006c810 msgr2=0x7f177006cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:28.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.969+0000 7f17779b2640 1 -- 172.21.15.67:0/1309624189 shutdown_connections 2024-03-24T00:49:28.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.969+0000 7f17779b2640 1 -- 172.21.15.67:0/1309624189 wait complete. 2024-03-24T00:49:28.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.970+0000 7f17779b2640 1 Processor -- start 2024-03-24T00:49:28.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.970+0000 7f17779b2640 1 -- start start 2024-03-24T00:49:28.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.971+0000 7f17779b2640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f17700d0ef0 0x7f177014ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:28.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.971+0000 7f17779b2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f177015dec0 0x7f17700b4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:28.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.971+0000 7f17779b2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f177015e2a0 0x7f17700cdaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:28.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.971+0000 7f17779b2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17700d2730 con 0x7f177015dec0 2024-03-24T00:49:28.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.971+0000 7f17779b2640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17700d25b0 con 0x7f177015e2a0 2024-03-24T00:49:28.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.971+0000 7f17779b2640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17700d28b0 con 0x7f17700d0ef0 2024-03-24T00:49:28.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.971+0000 7f1774f4d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f177015dec0 0x7f17700b4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:28.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.971+0000 7f1774f4d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f177015dec0 0x7f17700b4050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38518/0 (socket says 172.21.15.67:38518) 2024-03-24T00:49:28.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.971+0000 7f1774f4d640 1 -- 172.21.15.67:0/2087559291 learned_addr learned my addr 172.21.15.67:0/2087559291 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:28.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.971+0000 7f1775f4f640 1 --2- 172.21.15.67:0/2087559291 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f177015e2a0 0x7f17700cdaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:28.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.971+0000 7f177574e640 1 --2- 172.21.15.67:0/2087559291 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f17700d0ef0 0x7f177014ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:28.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.971+0000 7f1774f4d640 1 -- 172.21.15.67:0/2087559291 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f17700d0ef0 msgr2=0x7f177014ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:28.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.971+0000 7f1774f4d640 1 --2- 172.21.15.67:0/2087559291 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f17700d0ef0 0x7f177014ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:28.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.971+0000 7f1774f4d640 1 -- 172.21.15.67:0/2087559291 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f177015e2a0 msgr2=0x7f17700cdaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:28.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.971+0000 7f1774f4d640 1 --2- 172.21.15.67:0/2087559291 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f177015e2a0 0x7f17700cdaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:28.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.971+0000 7f1774f4d640 1 -- 172.21.15.67:0/2087559291 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17701578e0 con 0x7f177015dec0 2024-03-24T00:49:28.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.971+0000 7f1775f4f640 1 --2- 172.21.15.67:0/2087559291 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f177015e2a0 0x7f17700cdaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:49:28.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.971+0000 7f1774f4d640 1 --2- 172.21.15.67:0/2087559291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f177015dec0 0x7f17700b4050 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f176000c410 tx=0x7f176000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:28.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.972+0000 7f1765ffb640 1 -- 172.21.15.67:0/2087559291 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17600074e0 con 0x7f177015dec0 2024-03-24T00:49:28.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.972+0000 7f1765ffb640 1 -- 172.21.15.67:0/2087559291 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17600047e0 con 0x7f177015dec0 2024-03-24T00:49:28.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.972+0000 7f17779b2640 1 -- 172.21.15.67:0/2087559291 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1770157bc0 con 0x7f177015dec0 2024-03-24T00:49:28.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.972+0000 7f1765ffb640 1 -- 172.21.15.67:0/2087559291 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1760004bd0 con 0x7f177015dec0 2024-03-24T00:49:28.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.972+0000 7f17779b2640 1 -- 172.21.15.67:0/2087559291 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1770158180 con 0x7f177015dec0 2024-03-24T00:49:28.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.973+0000 7f17779b2640 1 -- 172.21.15.67:0/2087559291 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f177004f400 con 0x7f177015dec0 2024-03-24T00:49:28.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.973+0000 7f1765ffb640 1 -- 172.21.15.67:0/2087559291 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1760011070 con 0x7f177015dec0 2024-03-24T00:49:28.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.974+0000 7f1765ffb640 1 --2- 172.21.15.67:0/2087559291 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f175406bbd0 0x7f175406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:28.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.974+0000 7f1765ffb640 1 -- 172.21.15.67:0/2087559291 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f176008a8f0 con 0x7f177015dec0 2024-03-24T00:49:28.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.974+0000 7f177574e640 1 --2- 172.21.15.67:0/2087559291 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f175406bbd0 0x7f175406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:28.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.975+0000 7f177574e640 1 --2- 172.21.15.67:0/2087559291 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f175406bbd0 0x7f175406e090 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f176c036000 tx=0x7f176c03d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:28.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:28.978+0000 7f1765ffb640 1 -- 172.21.15.67:0/2087559291 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f17700cf640 con 0x7f177015dec0 2024-03-24T00:49:29.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.203+0000 7f17779b2640 1 -- 172.21.15.67:0/2087559291 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f17700d1aa0 con 0x7f177015dec0 2024-03-24T00:49:29.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.203+0000 7f1765ffb640 1 -- 172.21.15.67:0/2087559291 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+4975 (secure 0 0 0) 0x7f176000e280 con 0x7f177015dec0 2024-03-24T00:49:29.204 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:29.205 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":99,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7298,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/839277571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":839277571},{"type":"v1","addr":"172.21.15.72:6836","nonce":839277571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":8224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2052380377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2052380377},{"type":"v1","addr":"172.21.15.67:6834","nonce":2052380377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":8239,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2196618204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2196618204},{"type":"v1","addr":"172.21.15.72:6837","nonce":2196618204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:48:54.351575+0000","modified":"2024-03-24T00:49:28.351081+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":122,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9666},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9666":{"gid":9666,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6835/693801918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":693801918},{"type":"v1","addr":"172.21.15.67:6835","nonce":693801918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9666,"qdb_cluster":[9666]},"id":8}]} 2024-03-24T00:49:29.205 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 99 2024-03-24T00:49:29.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.206+0000 7f17779b2640 1 -- 172.21.15.67:0/2087559291 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f175406bbd0 msgr2=0x7f175406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:29.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.206+0000 7f17779b2640 1 --2- 172.21.15.67:0/2087559291 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f175406bbd0 0x7f175406e090 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f176c036000 tx=0x7f176c03d040 comp rx=0 tx=0).stop 2024-03-24T00:49:29.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.206+0000 7f17779b2640 1 -- 172.21.15.67:0/2087559291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f177015dec0 msgr2=0x7f17700b4050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:29.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.206+0000 7f17779b2640 1 --2- 172.21.15.67:0/2087559291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f177015dec0 0x7f17700b4050 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f176000c410 tx=0x7f176000c8d0 comp rx=0 tx=0).stop 2024-03-24T00:49:29.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.206+0000 7f17779b2640 1 -- 172.21.15.67:0/2087559291 shutdown_connections 2024-03-24T00:49:29.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.206+0000 7f17779b2640 1 --2- 172.21.15.67:0/2087559291 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f175406bbd0 0x7f175406e090 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:29.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.206+0000 7f17779b2640 1 --2- 172.21.15.67:0/2087559291 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f177015e2a0 0x7f17700cdaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:29.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.206+0000 7f17779b2640 1 --2- 172.21.15.67:0/2087559291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f177015dec0 0x7f17700b4050 unknown :-1 s=CLOSED pgs=1857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:29.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.206+0000 7f17779b2640 1 --2- 172.21.15.67:0/2087559291 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f17700d0ef0 0x7f177014ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:29.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.206+0000 7f17779b2640 1 -- 172.21.15.67:0/2087559291 >> 172.21.15.67:0/2087559291 conn(0x7f177006c810 msgr2=0x7f17701e7240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:29.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.206+0000 7f17779b2640 1 -- 172.21.15.67:0/2087559291 shutdown_connections 2024-03-24T00:49:29.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.206+0000 7f17779b2640 1 -- 172.21.15.67:0/2087559291 wait complete. 2024-03-24T00:49:29.326 INFO:tasks.ceph.mds.a.smithi067.stderr:2024-03-24T00:49:29.325+0000 7fe4f3a82640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T00:49:29.621 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:48:54.351575+0000', 'modified': '2024-03-24T00:49:28.351081+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 122, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9666}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9666': {'gid': 9666, 'name': 'a', 'rank': 0, 'incarnation': 96, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.67:6835/693801918', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6833', 'nonce': 693801918}, {'type': 'v1', 'addr': '172.21.15.67:6835', 'nonce': 693801918}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9666, 'qdb_cluster': [9666]} 2024-03-24T00:49:29.621 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-24T00:49:29.621 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-24T00:49:29.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.724+0000 7f07453f4640 1 Processor -- start 2024-03-24T00:49:29.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.724+0000 7f07453f4640 1 -- start start 2024-03-24T00:49:29.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.725+0000 7f07453f4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f074016a960 0x7f07400b2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:29.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.725+0000 7f07453f4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f074016ed00 0x7f07400cde80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:29.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.725+0000 7f07453f4640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07400b3300 con 0x7f07400b43a0 2024-03-24T00:49:29.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.725+0000 7f07453f4640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07400b35e0 con 0x7f074016a960 2024-03-24T00:49:29.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.725+0000 7f07453f4640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07400b38c0 con 0x7f074016ed00 2024-03-24T00:49:29.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.725+0000 7f073effd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f074016a960 0x7f07400b2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:29.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.725+0000 7f073effd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f074016a960 0x7f07400b2ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38524/0 (socket says 172.21.15.67:38524) 2024-03-24T00:49:29.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.725+0000 7f073effd640 1 -- 172.21.15.67:0/1484817682 learned_addr learned my addr 172.21.15.67:0/1484817682 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:29.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.725+0000 7f073e7fc640 1 -- 172.21.15.67:0/1484817682 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 846217033 0 0) 0x7f07400b3300 con 0x7f07400b43a0 2024-03-24T00:49:29.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.725+0000 7f073f7fe640 1 --2- 172.21.15.67:0/1484817682 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f074016ed00 0x7f07400cde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:29.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.766+0000 7f073e7fc640 1 -- 172.21.15.67:0/1484817682 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f072c003500 con 0x7f07400b43a0 2024-03-24T00:49:29.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.766+0000 7f073effd640 1 -- 172.21.15.67:0/1484817682 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f074016ed00 msgr2=0x7f07400cde80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:29.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.766+0000 7f073effd640 1 --2- 172.21.15.67:0/1484817682 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f074016ed00 0x7f07400cde80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:29.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.766+0000 7f073effd640 1 -- 172.21.15.67:0/1484817682 >> v1:172.21.15.72:6789/0 conn(0x7f07400b43a0 legacy=0x7f074016e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:29.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.766+0000 7f073effd640 1 -- 172.21.15.67:0/1484817682 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07400b3ba0 con 0x7f074016a960 2024-03-24T00:49:29.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.767+0000 7f073effd640 1 --2- 172.21.15.67:0/1484817682 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f074016a960 0x7f07400b2ad0 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f0730008910 tx=0x7f073002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bda541986258cf4d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:29.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.767+0000 7f073e7fc640 1 -- 172.21.15.67:0/1484817682 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0730008df0 con 0x7f074016a960 2024-03-24T00:49:29.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.767+0000 7f073e7fc640 1 -- 172.21.15.67:0/1484817682 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f073002d040 con 0x7f074016a960 2024-03-24T00:49:29.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.767+0000 7f073e7fc640 1 -- 172.21.15.67:0/1484817682 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f073000b370 con 0x7f074016a960 2024-03-24T00:49:29.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.768+0000 7f07453f4640 1 -- 172.21.15.67:0/1484817682 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f074016a960 msgr2=0x7f07400b2ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:29.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.768+0000 7f07453f4640 1 --2- 172.21.15.67:0/1484817682 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f074016a960 0x7f07400b2ad0 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f0730008910 tx=0x7f073002b900 comp rx=0 tx=0).stop 2024-03-24T00:49:29.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.768+0000 7f07453f4640 1 -- 172.21.15.67:0/1484817682 shutdown_connections 2024-03-24T00:49:29.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.768+0000 7f07453f4640 1 --2- 172.21.15.67:0/1484817682 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f074016ed00 0x7f07400cde80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:29.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.768+0000 7f07453f4640 1 --2- 172.21.15.67:0/1484817682 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f074016a960 0x7f07400b2ad0 unknown :-1 s=CLOSED pgs=1858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:29.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.768+0000 7f07453f4640 1 -- 172.21.15.67:0/1484817682 >> 172.21.15.67:0/1484817682 conn(0x7f074006b670 msgr2=0x7f07400b17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:29.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.768+0000 7f07453f4640 1 -- 172.21.15.67:0/1484817682 shutdown_connections 2024-03-24T00:49:29.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.768+0000 7f07453f4640 1 -- 172.21.15.67:0/1484817682 wait complete. 2024-03-24T00:49:29.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.770+0000 7f07453f4640 1 Processor -- start 2024-03-24T00:49:29.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.770+0000 7f07453f4640 1 -- start start 2024-03-24T00:49:29.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.771+0000 7f07453f4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07400b43a0 0x7f0740200750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:29.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.771+0000 7f07453f4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f074016a960 0x7f074016d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:29.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.771+0000 7f07453f4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f074016ed00 0x7f074016d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:29.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.771+0000 7f07453f4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f074016ad40 con 0x7f074016ed00 2024-03-24T00:49:29.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.771+0000 7f07453f4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07400b3010 con 0x7f07400b43a0 2024-03-24T00:49:29.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.771+0000 7f07453f4640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f074016aec0 con 0x7f074016a960 2024-03-24T00:49:29.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.771+0000 7f073f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f074016ed00 0x7f074016d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:29.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.771+0000 7f073f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f074016ed00 0x7f074016d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38538/0 (socket says 172.21.15.67:38538) 2024-03-24T00:49:29.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.771+0000 7f073f7fe640 1 -- 172.21.15.67:0/3595773585 learned_addr learned my addr 172.21.15.67:0/3595773585 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:29.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.771+0000 7f073effd640 1 --2- 172.21.15.67:0/3595773585 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07400b43a0 0x7f0740200750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:29.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.771+0000 7f07367fc640 1 --2- 172.21.15.67:0/3595773585 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f074016a960 0x7f074016d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:29.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.771+0000 7f073f7fe640 1 -- 172.21.15.67:0/3595773585 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f074016a960 msgr2=0x7f074016d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:29.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.771+0000 7f073f7fe640 1 --2- 172.21.15.67:0/3595773585 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f074016a960 0x7f074016d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:29.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.771+0000 7f073f7fe640 1 -- 172.21.15.67:0/3595773585 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07400b43a0 msgr2=0x7f0740200750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:29.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.771+0000 7f073f7fe640 1 --2- 172.21.15.67:0/3595773585 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07400b43a0 0x7f0740200750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:29.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.771+0000 7f073f7fe640 1 -- 172.21.15.67:0/3595773585 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f074015e880 con 0x7f074016ed00 2024-03-24T00:49:29.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.772+0000 7f073f7fe640 1 --2- 172.21.15.67:0/3595773585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f074016ed00 0x7f074016d660 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f0738008c30 tx=0x7f073800e5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:29.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.772+0000 7f0737fff640 1 -- 172.21.15.67:0/3595773585 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f073800d040 con 0x7f074016ed00 2024-03-24T00:49:29.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.772+0000 7f0737fff640 1 -- 172.21.15.67:0/3595773585 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0738003990 con 0x7f074016ed00 2024-03-24T00:49:29.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.772+0000 7f07453f4640 1 -- 172.21.15.67:0/3595773585 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f074015eb60 con 0x7f074016ed00 2024-03-24T00:49:29.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.772+0000 7f0737fff640 1 -- 172.21.15.67:0/3595773585 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0738003d60 con 0x7f074016ed00 2024-03-24T00:49:29.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.772+0000 7f07453f4640 1 -- 172.21.15.67:0/3595773585 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f074015f120 con 0x7f074016ed00 2024-03-24T00:49:29.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.773+0000 7f0737fff640 1 -- 172.21.15.67:0/3595773585 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0738004030 con 0x7f074016ed00 2024-03-24T00:49:29.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.774+0000 7f0737fff640 1 --2- 172.21.15.67:0/3595773585 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f072006bbd0 0x7f072006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:29.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.774+0000 7f0737fff640 1 -- 172.21.15.67:0/3595773585 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f073808b5e0 con 0x7f074016ed00 2024-03-24T00:49:29.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.774+0000 7f073effd640 1 --2- 172.21.15.67:0/3595773585 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f072006bbd0 0x7f072006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:29.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.775+0000 7f073effd640 1 --2- 172.21.15.67:0/3595773585 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f072006bbd0 0x7f072006e090 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f0730034000 tx=0x7f073000a000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:29.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.776+0000 7f0747fde740 1 Processor -- start 2024-03-24T00:49:29.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.776+0000 7f0747fde740 1 -- start start 2024-03-24T00:49:29.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.777+0000 7f0747fde740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5558eeb13c40 0x5558eeb3a770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:29.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.777+0000 7f0747fde740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5558eeb3acb0 0x5558eeb12c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:29.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.777+0000 7f0747fde740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558eeb14020 0x5558eeb131c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:29.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.777+0000 7f0747fde740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558eea77c20 con 0x5558eeb14020 2024-03-24T00:49:29.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.777+0000 7f0747fde740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558eea20a40 con 0x5558eeb13c40 2024-03-24T00:49:29.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.777+0000 7f0747fde740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5558eea73cc0 con 0x5558eeb3acb0 2024-03-24T00:49:29.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.777+0000 7f073f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558eeb14020 0x5558eeb131c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:29.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.777+0000 7f073f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558eeb14020 0x5558eeb131c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38544/0 (socket says 172.21.15.67:38544) 2024-03-24T00:49:29.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.777+0000 7f073f7fe640 1 -- 172.21.15.67:0/2523200353 learned_addr learned my addr 172.21.15.67:0/2523200353 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:29.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.777+0000 7f073effd640 1 --2- 172.21.15.67:0/2523200353 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5558eeb13c40 0x5558eeb3a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:29.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.777+0000 7f07367fc640 1 --2- 172.21.15.67:0/2523200353 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5558eeb3acb0 0x5558eeb12c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:29.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.778+0000 7f073f7fe640 1 -- 172.21.15.67:0/2523200353 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5558eeb3acb0 msgr2=0x5558eeb12c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:29.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.778+0000 7f073f7fe640 1 --2- 172.21.15.67:0/2523200353 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5558eeb3acb0 0x5558eeb12c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:29.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.778+0000 7f073f7fe640 1 -- 172.21.15.67:0/2523200353 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5558eeb13c40 msgr2=0x5558eeb3a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:29.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.778+0000 7f073f7fe640 1 --2- 172.21.15.67:0/2523200353 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5558eeb13c40 0x5558eeb3a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:29.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.778+0000 7f073f7fe640 1 -- 172.21.15.67:0/2523200353 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5558eea42f90 con 0x5558eeb14020 2024-03-24T00:49:29.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.778+0000 7f073f7fe640 1 --2- 172.21.15.67:0/2523200353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558eeb14020 0x5558eeb131c0 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f0738055bf0 tx=0x7f0738059f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0fc3eac1cfed40a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:29.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.778+0000 7f0734ff9640 1 -- 172.21.15.67:0/2523200353 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0738061ea0 con 0x5558eeb14020 2024-03-24T00:49:29.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.778+0000 7f0734ff9640 1 -- 172.21.15.67:0/2523200353 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0738057a90 con 0x5558eeb14020 2024-03-24T00:49:29.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.778+0000 7f0734ff9640 1 -- 172.21.15.67:0/2523200353 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0738061a10 con 0x5558eeb14020 2024-03-24T00:49:29.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.779+0000 7f0747fde740 1 -- 172.21.15.67:0/2523200353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558eeb14020 msgr2=0x5558eeb131c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:29.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.779+0000 7f0747fde740 1 --2- 172.21.15.67:0/2523200353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558eeb14020 0x5558eeb131c0 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f0738055bf0 tx=0x7f0738059f50 comp rx=0 tx=0).stop 2024-03-24T00:49:29.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.779+0000 7f0747fde740 1 -- 172.21.15.67:0/2523200353 shutdown_connections 2024-03-24T00:49:29.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.779+0000 7f0747fde740 1 --2- 172.21.15.67:0/2523200353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558eeb14020 0x5558eeb131c0 unknown :-1 s=CLOSED pgs=1860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:29.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.779+0000 7f0747fde740 1 --2- 172.21.15.67:0/2523200353 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5558eeb3acb0 0x5558eeb12c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:29.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.779+0000 7f0747fde740 1 --2- 172.21.15.67:0/2523200353 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5558eeb13c40 0x5558eeb3a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:29.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.779+0000 7f0747fde740 1 -- 172.21.15.67:0/2523200353 >> 172.21.15.67:0/2523200353 conn(0x5558eeadaa60 msgr2=0x5558eeb58960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:29.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.779+0000 7f0747fde740 1 -- 172.21.15.67:0/2523200353 shutdown_connections 2024-03-24T00:49:29.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.779+0000 7f0747fde740 1 -- 172.21.15.67:0/2523200353 wait complete. 2024-03-24T00:49:29.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.780+0000 7f0747fde740 1 -- start start 2024-03-24T00:49:29.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.780+0000 7f0747fde740 1 Processor -- start 2024-03-24T00:49:29.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.781+0000 7f0747fde740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558eeb13c40 0x5558eeb3f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:29.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.781+0000 7f0747fde740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5558eeb14020 0x5558eeb35e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:29.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.781+0000 7f0747fde740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5558eeb23970 0x5558eeb23db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:29.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.781+0000 7f0747fde740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558eeb3a2e0 con 0x5558eeb13c40 2024-03-24T00:49:29.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.781+0000 7f0747fde740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558eea33e90 con 0x5558eeb14020 2024-03-24T00:49:29.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.781+0000 7f0747fde740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5558eeb33060 con 0x5558eeb23970 2024-03-24T00:49:29.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.781+0000 7f0747fde740 10 client.0 _ll_register_callbacks cb 0x5558ee78bac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:49:29.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.781+0000 7f073effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558eeb13c40 0x5558eeb3f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:29.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.781+0000 7f073effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558eeb13c40 0x5558eeb3f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38560/0 (socket says 172.21.15.67:38560) 2024-03-24T00:49:29.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.781+0000 7f073effd640 1 -- 172.21.15.67:0/2962513896 learned_addr learned my addr 172.21.15.67:0/2962513896 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:29.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.781+0000 7f07367fc640 1 --2- 172.21.15.67:0/2962513896 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5558eeb14020 0x5558eeb35e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:29.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.781+0000 7f073f7fe640 1 --2- 172.21.15.67:0/2962513896 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5558eeb23970 0x5558eeb23db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:29.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.781+0000 7f073effd640 1 -- 172.21.15.67:0/2962513896 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5558eeb23970 msgr2=0x5558eeb23db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:29.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.781+0000 7f073effd640 1 --2- 172.21.15.67:0/2962513896 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5558eeb23970 0x5558eeb23db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:29.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.781+0000 7f073effd640 1 -- 172.21.15.67:0/2962513896 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5558eeb14020 msgr2=0x5558eeb35e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:29.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.781+0000 7f073effd640 1 --2- 172.21.15.67:0/2962513896 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5558eeb14020 0x5558eeb35e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:29.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.781+0000 7f073effd640 1 -- 172.21.15.67:0/2962513896 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5558eea409d0 con 0x5558eeb13c40 2024-03-24T00:49:29.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.781+0000 7f07367fc640 1 --2- 172.21.15.67:0/2962513896 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5558eeb14020 0x5558eeb35e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:49:29.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.782+0000 7f073effd640 1 --2- 172.21.15.67:0/2962513896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558eeb13c40 0x5558eeb3f220 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f073000bad0 tx=0x7f0730004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59c19e8486170277 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:29.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.782+0000 7f071dffb640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:49:29.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.782+0000 7f071dffb640 1 -- 172.21.15.67:0/2962513896 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f073004b070 con 0x5558eeb13c40 2024-03-24T00:49:29.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.782+0000 7f071dffb640 1 -- 172.21.15.67:0/2962513896 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(122..122 src has 1..122) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f073003b410 con 0x5558eeb13c40 2024-03-24T00:49:29.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.782+0000 7f0747fde740 1 -- 172.21.15.67:0/2962513896 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0738057a90 con 0x5558eeb13c40 2024-03-24T00:49:29.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.782+0000 7f071dffb640 1 -- 172.21.15.67:0/2962513896 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=99) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0730056050 con 0x5558eeb13c40 2024-03-24T00:49:29.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.782+0000 7f0747fde740 10 client.9759 fetch_fsmap learned FSMap version 99 2024-03-24T00:49:29.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.782+0000 7f0747fde740 1 -- 172.21.15.67:0/2962513896 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=99,monmap=2+}) v3 -- 0x5558ee78c2d0 con 0x5558eeb13c40 2024-03-24T00:49:29.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.782+0000 7f071dffb640 1 -- 172.21.15.67:0/2962513896 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0738057a90 con 0x5558eeb13c40 2024-03-24T00:49:29.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.782+0000 7f071dffb640 1 -- 172.21.15.67:0/2962513896 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 99) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f0730059020 con 0x5558eeb13c40 2024-03-24T00:49:29.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.782+0000 7f0747fde740 10 client.9759 fetch_fsmap finished waiting for FSMap version 99 2024-03-24T00:49:29.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.782+0000 7f0747fde740 10 client.9759 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-24T00:49:29.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.782+0000 7f0747fde740 20 client.9759 populate_metadata read hostname 'smithi067' 2024-03-24T00:49:29.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.782+0000 7f0747fde740 1 --2- 172.21.15.67:0/2962513896 >> [v2:172.21.15.67:6833/693801918,v1:172.21.15.67:6835/693801918] conn(0x5558eeb6c140 0x5558eeb6c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:29.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.782+0000 7f0747fde740 4 client.9759 mds_command: new command op to 9666 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:49:29.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.782+0000 7f0747fde740 1 -- 172.21.15.67:0/2962513896 --> [v2:172.21.15.67:6833/693801918,v1:172.21.15.67:6835/693801918] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5558eea85160 con 0x5558eeb6c140 2024-03-24T00:49:29.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.782+0000 7f07367fc640 1 --2- 172.21.15.67:0/2962513896 >> [v2:172.21.15.67:6833/693801918,v1:172.21.15.67:6835/693801918] conn(0x5558eeb6c140 0x5558eeb6c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:29.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.783+0000 7f071dffb640 10 client.9759 ms_handle_connect on v2:172.21.15.67:6833/693801918 2024-03-24T00:49:29.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.783+0000 7f07367fc640 1 --2- 172.21.15.67:0/2962513896 >> [v2:172.21.15.67:6833/693801918,v1:172.21.15.67:6835/693801918] conn(0x5558eeb6c140 0x5558eeb6c520 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30b7f266b6373a66 server_cookie=3dff1bb8f6ee715e in_seq=0 out_seq=0 2024-03-24T00:49:29.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.784+0000 7f071dffb640 1 -- 172.21.15.67:0/2962513896 <== mds.0 v2:172.21.15.67:6833/693801918 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5558eea85160 con 0x5558eeb6c140 2024-03-24T00:49:29.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.784+0000 7f071dffb640 10 client.9759 handle_command_reply: tid=0 2024-03-24T00:49:29.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.784+0000 7f0747fde740 1 client.9759 shutdown 2024-03-24T00:49:29.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.784+0000 7f0747fde740 1 -- 172.21.15.67:0/2962513896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558eeb13c40 msgr2=0x5558eeb3f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:29.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.784+0000 7f0747fde740 1 --2- 172.21.15.67:0/2962513896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558eeb13c40 0x5558eeb3f220 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f073000bad0 tx=0x7f0730004730 comp rx=0 tx=0).stop 2024-03-24T00:49:29.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.784+0000 7f0747fde740 1 -- 172.21.15.67:0/2962513896 shutdown_connections 2024-03-24T00:49:29.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.784+0000 7f0747fde740 1 --2- 172.21.15.67:0/2962513896 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5558eeb23970 0x5558eeb23db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:29.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.784+0000 7f0747fde740 1 --2- 172.21.15.67:0/2962513896 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5558eeb14020 0x5558eeb35e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:29.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.784+0000 7f0747fde740 1 --2- 172.21.15.67:0/2962513896 >> [v2:172.21.15.67:6833/693801918,v1:172.21.15.67:6835/693801918] conn(0x5558eeb6c140 0x5558eeb6c520 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:29.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.784+0000 7f0747fde740 1 --2- 172.21.15.67:0/2962513896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558eeb13c40 0x5558eeb3f220 unknown :-1 s=CLOSED pgs=1861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:29.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.784+0000 7f071dffb640 0 client.9759 ms_handle_reset on v2:172.21.15.67:6833/693801918 2024-03-24T00:49:29.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.784+0000 7f0747fde740 1 -- 172.21.15.67:0/2962513896 >> 172.21.15.67:0/2962513896 conn(0x5558eeadaa60 msgr2=0x5558eeb116d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:29.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.784+0000 7f0747fde740 1 -- 172.21.15.67:0/2962513896 shutdown_connections 2024-03-24T00:49:29.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.784+0000 7f0747fde740 1 -- 172.21.15.67:0/2962513896 wait complete. 2024-03-24T00:49:29.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.784+0000 7f0747fde740 20 client.9759 trim_cache size 0 max 16384 2024-03-24T00:49:29.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.817+0000 7f0747fde740 1 Processor -- start 2024-03-24T00:49:29.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.817+0000 7f0747fde740 1 -- start start 2024-03-24T00:49:29.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.818+0000 7f0747fde740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5558eeb13c40 0x5558eeb68400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:29.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.818+0000 7f0747fde740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5558eeb14020 0x5558eebc1e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:29.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.818+0000 7f0747fde740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558eeb23970 0x5558eeb41190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:29.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.818+0000 7f0747fde740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f074020c760 con 0x5558eeb23970 2024-03-24T00:49:29.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.818+0000 7f0747fde740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0740160850 con 0x5558eeb13c40 2024-03-24T00:49:29.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.818+0000 7f0747fde740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0740160e50 con 0x5558eeb14020 2024-03-24T00:49:29.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.818+0000 7f073f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558eeb23970 0x5558eeb41190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:29.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.818+0000 7f073f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558eeb23970 0x5558eeb41190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38566/0 (socket says 172.21.15.67:38566) 2024-03-24T00:49:29.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.818+0000 7f073f7fe640 1 -- 172.21.15.67:0/1776139256 learned_addr learned my addr 172.21.15.67:0/1776139256 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:29.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.819+0000 7f073f7fe640 1 -- 172.21.15.67:0/1776139256 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5558eeb14020 msgr2=0x5558eebc1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:29.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.819+0000 7f073f7fe640 1 --2- 172.21.15.67:0/1776139256 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5558eeb14020 0x5558eebc1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:29.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.819+0000 7f073f7fe640 1 -- 172.21.15.67:0/1776139256 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5558eeb13c40 msgr2=0x5558eeb68400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:29.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.819+0000 7f073f7fe640 1 --2- 172.21.15.67:0/1776139256 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5558eeb13c40 0x5558eeb68400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:29.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.819+0000 7f073f7fe640 1 -- 172.21.15.67:0/1776139256 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5558ee8566b0 con 0x5558eeb23970 2024-03-24T00:49:29.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.819+0000 7f073f7fe640 1 --2- 172.21.15.67:0/1776139256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558eeb23970 0x5558eeb41190 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f0738059950 tx=0x7f073805b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=add2a62d89f01124 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:29.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.819+0000 7f071ffff640 1 -- 172.21.15.67:0/1776139256 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0738059560 con 0x5558eeb23970 2024-03-24T00:49:29.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.819+0000 7f071ffff640 1 -- 172.21.15.67:0/1776139256 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0738061ea0 con 0x5558eeb23970 2024-03-24T00:49:29.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.819+0000 7f071ffff640 1 -- 172.21.15.67:0/1776139256 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0738055e00 con 0x5558eeb23970 2024-03-24T00:49:29.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.819+0000 7f0747fde740 1 -- 172.21.15.67:0/1776139256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558eeb23970 msgr2=0x5558eeb41190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:29.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.819+0000 7f0747fde740 1 --2- 172.21.15.67:0/1776139256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558eeb23970 0x5558eeb41190 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f0738059950 tx=0x7f073805b000 comp rx=0 tx=0).stop 2024-03-24T00:49:29.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.819+0000 7f0747fde740 1 -- 172.21.15.67:0/1776139256 shutdown_connections 2024-03-24T00:49:29.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.819+0000 7f0747fde740 1 --2- 172.21.15.67:0/1776139256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558eeb23970 0x5558eeb41190 unknown :-1 s=CLOSED pgs=1862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:29.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.819+0000 7f0747fde740 1 --2- 172.21.15.67:0/1776139256 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5558eeb14020 0x5558eebc1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:29.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.819+0000 7f0747fde740 1 --2- 172.21.15.67:0/1776139256 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5558eeb13c40 0x5558eeb68400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:29.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.819+0000 7f0747fde740 1 -- 172.21.15.67:0/1776139256 >> 172.21.15.67:0/1776139256 conn(0x5558eeadaa60 msgr2=0x5558eeb35c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:29.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.820+0000 7f0747fde740 1 -- 172.21.15.67:0/1776139256 shutdown_connections 2024-03-24T00:49:29.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.820+0000 7f0747fde740 1 -- 172.21.15.67:0/1776139256 wait complete. 2024-03-24T00:49:29.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.820+0000 7f0747fde740 1 -- start start 2024-03-24T00:49:29.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.820+0000 7f0747fde740 1 Processor -- start 2024-03-24T00:49:29.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.822+0000 7f0747fde740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5558eeb13c40 0x5558eebbdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:29.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.822+0000 7f0747fde740 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5558eeb14020 0x5558eeb4eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:29.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.822+0000 7f0747fde740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558eeb23970 0x5558eeb3eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:29.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.822+0000 7f0747fde740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558eeb242f0 con 0x5558eeb23970 2024-03-24T00:49:29.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.822+0000 7f0747fde740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558eeb404a0 con 0x5558eeb13c40 2024-03-24T00:49:29.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.822+0000 7f0747fde740 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x5558eeb6a110 con 0x5558eeb14020 2024-03-24T00:49:29.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.822+0000 7f0747fde740 10 client.0 _ll_register_callbacks cb 0x7f072c0036d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:49:29.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.822+0000 7f073f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558eeb23970 0x5558eeb3eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:29.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.822+0000 7f073f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558eeb23970 0x5558eeb3eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38568/0 (socket says 172.21.15.67:38568) 2024-03-24T00:49:29.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.822+0000 7f073f7fe640 1 -- 172.21.15.67:0/3277727148 learned_addr learned my addr 172.21.15.67:0/3277727148 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:29.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.822+0000 7f073effd640 1 --2- 172.21.15.67:0/3277727148 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5558eeb13c40 0x5558eebbdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:29.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.822+0000 7f07367fc640 1 --2- 172.21.15.67:0/3277727148 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5558eeb14020 0x5558eeb4eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:29.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.822+0000 7f073f7fe640 1 -- 172.21.15.67:0/3277727148 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5558eeb14020 msgr2=0x5558eeb4eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:29.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.822+0000 7f073f7fe640 1 --2- 172.21.15.67:0/3277727148 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5558eeb14020 0x5558eeb4eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:29.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.822+0000 7f073f7fe640 1 -- 172.21.15.67:0/3277727148 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5558eeb13c40 msgr2=0x5558eebbdfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:29.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.822+0000 7f073f7fe640 1 --2- 172.21.15.67:0/3277727148 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5558eeb13c40 0x5558eebbdfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:29.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.822+0000 7f073f7fe640 1 -- 172.21.15.67:0/3277727148 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5558eeb15120 con 0x5558eeb23970 2024-03-24T00:49:29.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.822+0000 7f073effd640 1 --2- 172.21.15.67:0/3277727148 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5558eeb13c40 0x5558eebbdfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:29.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.822+0000 7f073f7fe640 1 --2- 172.21.15.67:0/3277727148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558eeb23970 0x5558eeb3eae0 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f07380070e0 tx=0x7f0738004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e020b2af04e6eb9f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:29.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.822+0000 7f071dffb640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-24T00:49:29.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.822+0000 7f071dffb640 1 -- 172.21.15.67:0/3277727148 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0738055e00 con 0x5558eeb23970 2024-03-24T00:49:29.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.823+0000 7f071dffb640 1 -- 172.21.15.67:0/3277727148 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(122..122 src has 1..122) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f0738089440 con 0x5558eeb23970 2024-03-24T00:49:29.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.823+0000 7f0747fde740 1 -- 172.21.15.67:0/3277727148 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0738061ea0 con 0x5558eeb23970 2024-03-24T00:49:29.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.823+0000 7f071dffb640 1 -- 172.21.15.67:0/3277727148 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=99) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0738055fa0 con 0x5558eeb23970 2024-03-24T00:49:29.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.823+0000 7f0747fde740 10 client.9765 fetch_fsmap learned FSMap version 99 2024-03-24T00:49:29.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.823+0000 7f0747fde740 1 -- 172.21.15.67:0/3277727148 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=99,monmap=2+}) v3 -- 0x5558eeb5bf60 con 0x5558eeb23970 2024-03-24T00:49:29.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.823+0000 7f071dffb640 1 -- 172.21.15.67:0/3277727148 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0738061ea0 con 0x5558eeb23970 2024-03-24T00:49:29.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.823+0000 7f071dffb640 1 -- 172.21.15.67:0/3277727148 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 99) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f0738023060 con 0x5558eeb23970 2024-03-24T00:49:29.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.823+0000 7f0747fde740 10 client.9765 fetch_fsmap finished waiting for FSMap version 99 2024-03-24T00:49:29.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.823+0000 7f0747fde740 10 client.9765 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-24T00:49:29.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.823+0000 7f0747fde740 20 client.9765 populate_metadata read hostname 'smithi067' 2024-03-24T00:49:29.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.823+0000 7f0747fde740 1 --2- 172.21.15.67:0/3277727148 >> [v2:172.21.15.67:6833/693801918,v1:172.21.15.67:6835/693801918] conn(0x5558eeb5ba80 0x5558eeb3f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:29.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.823+0000 7f0747fde740 4 client.9765 mds_command: new command op to 9666 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:49:29.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.823+0000 7f0747fde740 1 -- 172.21.15.67:0/3277727148 --> [v2:172.21.15.67:6833/693801918,v1:172.21.15.67:6835/693801918] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5558eea7f4b0 con 0x5558eeb5ba80 2024-03-24T00:49:29.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.823+0000 7f073effd640 1 --2- 172.21.15.67:0/3277727148 >> [v2:172.21.15.67:6833/693801918,v1:172.21.15.67:6835/693801918] conn(0x5558eeb5ba80 0x5558eeb3f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:29.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.823+0000 7f071dffb640 10 client.9765 ms_handle_connect on v2:172.21.15.67:6833/693801918 2024-03-24T00:49:29.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.823+0000 7f073effd640 1 --2- 172.21.15.67:0/3277727148 >> [v2:172.21.15.67:6833/693801918,v1:172.21.15.67:6835/693801918] conn(0x5558eeb5ba80 0x5558eeb3f020 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=719940e94285fcc4 server_cookie=c786977b8c6f595a in_seq=0 out_seq=0 2024-03-24T00:49:29.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.824+0000 7f071dffb640 1 -- 172.21.15.67:0/3277727148 <== mds.0 v2:172.21.15.67:6833/693801918 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x5558eea7f4b0 con 0x5558eeb5ba80 2024-03-24T00:49:29.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.824+0000 7f071dffb640 10 client.9765 handle_command_reply: tid=0 2024-03-24T00:49:29.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.824+0000 7f0747fde740 1 client.9765 shutdown 2024-03-24T00:49:29.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.824+0000 7f0747fde740 1 -- 172.21.15.67:0/3277727148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558eeb23970 msgr2=0x5558eeb3eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:29.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.824+0000 7f0747fde740 1 --2- 172.21.15.67:0/3277727148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558eeb23970 0x5558eeb3eae0 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f07380070e0 tx=0x7f0738004a40 comp rx=0 tx=0).stop 2024-03-24T00:49:29.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.824+0000 7f0747fde740 1 -- 172.21.15.67:0/3277727148 shutdown_connections 2024-03-24T00:49:29.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.824+0000 7f0747fde740 1 --2- 172.21.15.67:0/3277727148 >> [v2:172.21.15.67:6833/693801918,v1:172.21.15.67:6835/693801918] conn(0x5558eeb5ba80 0x5558eeb3f020 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:29.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.824+0000 7f0747fde740 1 --2- 172.21.15.67:0/3277727148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558eeb23970 0x5558eeb3eae0 unknown :-1 s=CLOSED pgs=1863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:29.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.824+0000 7f0747fde740 1 --2- 172.21.15.67:0/3277727148 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x5558eeb14020 0x5558eeb4eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:29.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.824+0000 7f071dffb640 0 client.9765 ms_handle_reset on v2:172.21.15.67:6833/693801918 2024-03-24T00:49:29.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.824+0000 7f0747fde740 1 --2- 172.21.15.67:0/3277727148 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5558eeb13c40 0x5558eebbdfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:29.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.824+0000 7f0747fde740 1 -- 172.21.15.67:0/3277727148 >> 172.21.15.67:0/3277727148 conn(0x5558eeadaa60 msgr2=0x5558eeb34ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:29.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.824+0000 7f0747fde740 1 -- 172.21.15.67:0/3277727148 shutdown_connections 2024-03-24T00:49:29.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.824+0000 7f0747fde740 1 -- 172.21.15.67:0/3277727148 wait complete. 2024-03-24T00:49:29.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.824+0000 7f0747fde740 20 client.9765 trim_cache size 0 max 16384 2024-03-24T00:49:29.834 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:49:29.834 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "de1d0178-808e-49ca-a986-e297e523b976", 2024-03-24T00:49:29.834 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-03-24T00:49:29.834 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 9666, 2024-03-24T00:49:29.834 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-03-24T00:49:29.834 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-03-24T00:49:29.834 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-03-24T00:49:29.834 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 4.4971528110000003, 2024-03-24T00:49:29.834 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 99, 2024-03-24T00:49:29.834 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 122, 2024-03-24T00:49:29.834 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 122, 2024-03-24T00:49:29.834 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 6.4757527890000004 2024-03-24T00:49:29.835 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:49:29.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.824+0000 7f07453f4640 1 -- 172.21.15.67:0/3595773585 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f072006bbd0 msgr2=0x7f072006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:29.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.824+0000 7f07453f4640 1 --2- 172.21.15.67:0/3595773585 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f072006bbd0 0x7f072006e090 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f0730034000 tx=0x7f073000a000 comp rx=0 tx=0).stop 2024-03-24T00:49:29.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.824+0000 7f07453f4640 1 -- 172.21.15.67:0/3595773585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f074016ed00 msgr2=0x7f074016d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:29.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.824+0000 7f07453f4640 1 --2- 172.21.15.67:0/3595773585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f074016ed00 0x7f074016d660 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f0738008c30 tx=0x7f073800e5c0 comp rx=0 tx=0).stop 2024-03-24T00:49:29.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.824+0000 7f07453f4640 1 -- 172.21.15.67:0/3595773585 shutdown_connections 2024-03-24T00:49:29.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.824+0000 7f07453f4640 1 --2- 172.21.15.67:0/3595773585 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f072006bbd0 0x7f072006e090 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:29.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.824+0000 7f07453f4640 1 --2- 172.21.15.67:0/3595773585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f074016ed00 0x7f074016d660 unknown :-1 s=CLOSED pgs=1859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:29.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.825+0000 7f07453f4640 1 --2- 172.21.15.67:0/3595773585 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f074016a960 0x7f074016d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:29.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.825+0000 7f07453f4640 1 --2- 172.21.15.67:0/3595773585 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07400b43a0 0x7f0740200750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:29.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.825+0000 7f07453f4640 1 -- 172.21.15.67:0/3595773585 >> 172.21.15.67:0/3595773585 conn(0x7f074006b670 msgr2=0x7f074006f9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:29.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.825+0000 7f07453f4640 1 -- 172.21.15.67:0/3595773585 shutdown_connections 2024-03-24T00:49:29.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.825+0000 7f07453f4640 1 -- 172.21.15.67:0/3595773585 wait complete. 2024-03-24T00:49:29.839 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T00:49:29.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.940+0000 7f7cc3187640 1 Processor -- start 2024-03-24T00:49:29.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.940+0000 7f7cc3187640 1 -- start start 2024-03-24T00:49:29.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.940+0000 7f7cc3187640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7cbc15bf70 0x7f7cbc0cc690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:29.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.940+0000 7f7cc3187640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7cbc0ce930 0x7f7cbc0b1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:29.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.940+0000 7f7cc3187640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cbc0ccfe0 con 0x7f7cbc0cfc70 2024-03-24T00:49:29.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.940+0000 7f7cc3187640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cbc0cd2c0 con 0x7f7cbc0ce930 2024-03-24T00:49:29.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.940+0000 7f7cc3187640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cbc0cd5a0 con 0x7f7cbc15bf70 2024-03-24T00:49:29.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.941+0000 7f7cbbfff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7cbc0cfc70 0x7f7cbc0ce220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50802/0 (socket says 172.21.15.67:50802) 2024-03-24T00:49:29.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.941+0000 7f7cbbfff640 1 -- 172.21.15.67:0/2116466755 learned_addr learned my addr 172.21.15.67:0/2116466755 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:49:29.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.941+0000 7f7cc0f23640 1 --2- 172.21.15.67:0/2116466755 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7cbc15bf70 0x7f7cbc0cc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:29.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.941+0000 7f7cc1724640 1 --2- 172.21.15.67:0/2116466755 >> v2:172.21.15.72:3300/0 conn(0x7f7cbc0ce930 0x7f7cbc0b1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:29.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.941+0000 7f7cbb7fe640 1 -- 172.21.15.67:0/2116466755 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1677054360 0 0) 0x7f7cbc0ccfe0 con 0x7f7cbc0cfc70 2024-03-24T00:49:29.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.941+0000 7f7cbb7fe640 1 -- 172.21.15.67:0/2116466755 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cac003520 con 0x7f7cbc0cfc70 2024-03-24T00:49:29.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.941+0000 7f7cc0f23640 1 -- 172.21.15.67:0/2116466755 >> v2:172.21.15.72:3300/0 conn(0x7f7cbc0ce930 msgr2=0x7f7cbc0b1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:29.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.941+0000 7f7cc0f23640 1 --2- 172.21.15.67:0/2116466755 >> v2:172.21.15.72:3300/0 conn(0x7f7cbc0ce930 0x7f7cbc0b1cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:29.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.941+0000 7f7cbb7fe640 1 -- 172.21.15.67:0/2116466755 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1056385836 0 0) 0x7f7cac003520 con 0x7f7cbc0cfc70 2024-03-24T00:49:29.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.941+0000 7f7cc0f23640 1 -- 172.21.15.67:0/2116466755 >> v1:172.21.15.67:6789/0 conn(0x7f7cbc0cfc70 legacy=0x7f7cbc0ce220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:29.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.941+0000 7f7cc0f23640 1 -- 172.21.15.67:0/2116466755 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cbc0cd880 con 0x7f7cbc15bf70 2024-03-24T00:49:29.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.941+0000 7f7cc1724640 1 --2- 172.21.15.67:0/2116466755 >> v2:172.21.15.72:3300/0 conn(0x7f7cbc0ce930 0x7f7cbc0b1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:49:29.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.942+0000 7f7cc0f23640 1 --2- 172.21.15.67:0/2116466755 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7cbc15bf70 0x7f7cbc0cc690 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7cb00097a0 tx=0x7f7cb002b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d46c7bbaeac569f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:29.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.942+0000 7f7cbb7fe640 1 -- 172.21.15.67:0/2116466755 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb0037070 con 0x7f7cbc15bf70 2024-03-24T00:49:29.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.942+0000 7f7cbb7fe640 1 -- 172.21.15.67:0/2116466755 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cb0030510 con 0x7f7cbc15bf70 2024-03-24T00:49:29.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.942+0000 7f7cbb7fe640 1 -- 172.21.15.67:0/2116466755 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca8001670 con 0x7f7cbc15bf70 2024-03-24T00:49:29.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.943+0000 7f7cc3187640 1 -- 172.21.15.67:0/2116466755 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7cbc15bf70 msgr2=0x7f7cbc0cc690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:29.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.943+0000 7f7cc3187640 1 --2- 172.21.15.67:0/2116466755 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7cbc15bf70 0x7f7cbc0cc690 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7cb00097a0 tx=0x7f7cb002b790 comp rx=0 tx=0).stop 2024-03-24T00:49:29.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.943+0000 7f7cc3187640 1 -- 172.21.15.67:0/2116466755 shutdown_connections 2024-03-24T00:49:29.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.943+0000 7f7cc3187640 1 --2- 172.21.15.67:0/2116466755 >> v2:172.21.15.72:3300/0 conn(0x7f7cbc0ce930 0x7f7cbc0b1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:29.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.943+0000 7f7cc3187640 1 --2- 172.21.15.67:0/2116466755 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7cbc15bf70 0x7f7cbc0cc690 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:29.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.943+0000 7f7cc3187640 1 -- 172.21.15.67:0/2116466755 >> 172.21.15.67:0/2116466755 conn(0x7f7cbc0d1bf0 msgr2=0x7f7cbc0d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:29.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.943+0000 7f7cc3187640 1 -- 172.21.15.67:0/2116466755 shutdown_connections 2024-03-24T00:49:29.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.943+0000 7f7cc3187640 1 -- 172.21.15.67:0/2116466755 wait complete. 2024-03-24T00:49:29.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.944+0000 7f7cc3187640 1 Processor -- start 2024-03-24T00:49:29.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.944+0000 7f7cc3187640 1 -- start start 2024-03-24T00:49:29.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.945+0000 7f7cc3187640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7cbc0ce930 0x7f7cbc150200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:29.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.945+0000 7f7cc3187640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7cbc0cfc70 0x7f7cbc15d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:29.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.945+0000 7f7cc3187640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cbc15bf70 0x7f7cbc0b69b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:29.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.945+0000 7f7cc3187640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cbc0ccd50 con 0x7f7cbc15bf70 2024-03-24T00:49:29.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.945+0000 7f7cc3187640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cbc0ccbd0 con 0x7f7cbc0cfc70 2024-03-24T00:49:29.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.945+0000 7f7cc3187640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cbc0b2210 con 0x7f7cbc0ce930 2024-03-24T00:49:29.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.945+0000 7f7cc1724640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cbc15bf70 0x7f7cbc0b69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:29.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.945+0000 7f7cc1724640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cbc15bf70 0x7f7cbc0b69b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38582/0 (socket says 172.21.15.67:38582) 2024-03-24T00:49:29.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.945+0000 7f7cc1724640 1 -- 172.21.15.67:0/1607127256 learned_addr learned my addr 172.21.15.67:0/1607127256 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:29.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.945+0000 7f7cbbfff640 1 --2- 172.21.15.67:0/1607127256 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7cbc0cfc70 0x7f7cbc15d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:29.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.945+0000 7f7cc0f23640 1 --2- 172.21.15.67:0/1607127256 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7cbc0ce930 0x7f7cbc150200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:29.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.945+0000 7f7cc1724640 1 -- 172.21.15.67:0/1607127256 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7cbc0ce930 msgr2=0x7f7cbc150200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:29.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.945+0000 7f7cc1724640 1 --2- 172.21.15.67:0/1607127256 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7cbc0ce930 0x7f7cbc150200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:29.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.945+0000 7f7cc1724640 1 -- 172.21.15.67:0/1607127256 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7cbc0cfc70 msgr2=0x7f7cbc15d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:29.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.945+0000 7f7cc1724640 1 --2- 172.21.15.67:0/1607127256 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7cbc0cfc70 0x7f7cbc15d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:29.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.945+0000 7f7cc1724640 1 -- 172.21.15.67:0/1607127256 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cbc15e220 con 0x7f7cbc15bf70 2024-03-24T00:49:29.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.945+0000 7f7cc1724640 1 --2- 172.21.15.67:0/1607127256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cbc15bf70 0x7f7cbc0b69b0 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f7cb4009eb0 tx=0x7f7cb400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:29.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.945+0000 7f7cb97fa640 1 -- 172.21.15.67:0/1607127256 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb4018020 con 0x7f7cbc15bf70 2024-03-24T00:49:29.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.945+0000 7f7cb97fa640 1 -- 172.21.15.67:0/1607127256 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cb4014480 con 0x7f7cbc15bf70 2024-03-24T00:49:29.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.945+0000 7f7cc3187640 1 -- 172.21.15.67:0/1607127256 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cbc15e500 con 0x7f7cbc15bf70 2024-03-24T00:49:29.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.946+0000 7f7cb97fa640 1 -- 172.21.15.67:0/1607127256 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb4014850 con 0x7f7cbc15bf70 2024-03-24T00:49:29.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.946+0000 7f7cc3187640 1 -- 172.21.15.67:0/1607127256 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cbc15eac0 con 0x7f7cbc15bf70 2024-03-24T00:49:29.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.947+0000 7f7cc3187640 1 -- 172.21.15.67:0/1607127256 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cbc04f400 con 0x7f7cbc15bf70 2024-03-24T00:49:29.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.947+0000 7f7cb97fa640 1 -- 172.21.15.67:0/1607127256 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7cb4005d10 con 0x7f7cbc15bf70 2024-03-24T00:49:29.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.947+0000 7f7cb97fa640 1 --2- 172.21.15.67:0/1607127256 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7ca006bbd0 0x7f7ca006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:29.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.947+0000 7f7cb97fa640 1 -- 172.21.15.67:0/1607127256 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f7cb408bae0 con 0x7f7cbc15bf70 2024-03-24T00:49:29.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.948+0000 7f7cc0f23640 1 --2- 172.21.15.67:0/1607127256 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7ca006bbd0 0x7f7ca006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:29.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.948+0000 7f7cc0f23640 1 --2- 172.21.15.67:0/1607127256 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7ca006bbd0 0x7f7ca006e090 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f7cb0034000 tx=0x7f7cb002f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:29.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:29.952+0000 7f7cb97fa640 1 -- 172.21.15.67:0/1607127256 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7cb4022030 con 0x7f7cbc15bf70 2024-03-24T00:49:30.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:30.187+0000 7f7cc3187640 1 -- 172.21.15.67:0/1607127256 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f7cbc0b2e80 con 0x7f7cbc15bf70 2024-03-24T00:49:30.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:30.188+0000 7f7cb97fa640 1 -- 172.21.15.67:0/1607127256 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v99) v1 ==== 53+0+73 (secure 0 0 0) 0x7f7cb4010040 con 0x7f7cbc15bf70 2024-03-24T00:49:30.189 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-24T00:49:30.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:30.189+0000 7f7cc3187640 1 -- 172.21.15.67:0/1607127256 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7ca006bbd0 msgr2=0x7f7ca006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:30.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:30.189+0000 7f7cc3187640 1 --2- 172.21.15.67:0/1607127256 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7ca006bbd0 0x7f7ca006e090 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f7cb0034000 tx=0x7f7cb002f000 comp rx=0 tx=0).stop 2024-03-24T00:49:30.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:30.189+0000 7f7cc3187640 1 -- 172.21.15.67:0/1607127256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cbc15bf70 msgr2=0x7f7cbc0b69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:30.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:30.189+0000 7f7cc3187640 1 --2- 172.21.15.67:0/1607127256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cbc15bf70 0x7f7cbc0b69b0 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f7cb4009eb0 tx=0x7f7cb400c490 comp rx=0 tx=0).stop 2024-03-24T00:49:30.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:30.190+0000 7f7cc3187640 1 -- 172.21.15.67:0/1607127256 shutdown_connections 2024-03-24T00:49:30.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:30.190+0000 7f7cc3187640 1 --2- 172.21.15.67:0/1607127256 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f7ca006bbd0 0x7f7ca006e090 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:30.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:30.190+0000 7f7cc3187640 1 --2- 172.21.15.67:0/1607127256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cbc15bf70 0x7f7cbc0b69b0 unknown :-1 s=CLOSED pgs=1865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:30.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:30.190+0000 7f7cc3187640 1 --2- 172.21.15.67:0/1607127256 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7cbc0cfc70 0x7f7cbc15d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:30.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:30.190+0000 7f7cc3187640 1 --2- 172.21.15.67:0/1607127256 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7cbc0ce930 0x7f7cbc150200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:30.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:30.190+0000 7f7cc3187640 1 -- 172.21.15.67:0/1607127256 >> 172.21.15.67:0/1607127256 conn(0x7f7cbc0d1bf0 msgr2=0x7f7cbc070780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:30.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:30.190+0000 7f7cc3187640 1 -- 172.21.15.67:0/1607127256 shutdown_connections 2024-03-24T00:49:30.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:30.190+0000 7f7cc3187640 1 -- 172.21.15.67:0/1607127256 wait complete. 2024-03-24T00:49:30.201 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-24T00:49:30.201 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T00:49:30.201 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-03-24T00:49:30.201 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-24T00:49:30.201 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T00:49:30.201 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-24T00:49:30.201 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-24T00:49:30.202 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T00:49:30.202 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:49:30.202 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:30.202 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-03-24T00:49:30.216 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T00:49:30.216 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T00:49:30.216 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T00:49:30.217 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:49:30.217 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-03-24T00:49:30.217 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:49:30.217 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:49:30.217 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-03-24T00:49:30.217 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-03-24T00:49:30.217 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:49:30.217 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-03-24T00:49:30.217 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-03-24T00:49:30.217 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:49:30.217 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-03-24T00:49:30.217 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T00:49:30.217 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-03-24T00:49:30.217 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-24T00:49:30.217 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 41943sec preferred_lft 41943sec 2024-03-24T00:49:30.218 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-03-24T00:49:30.218 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:49:30.218 INFO:teuthology.orchestra.run.smithi067.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T00:49:30.218 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 96:cf:26:c8:bf:62 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:49:30.218 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-24T00:49:30.218 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:49:30.218 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::7c6a:5fff:feaa:d41/64 scope link 2024-03-24T00:49:30.218 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:49:30.218 INFO:teuthology.orchestra.run.smithi067.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T00:49:30.218 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ea:22:4b:d5:48:3d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:49:30.218 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::e822:4bff:fed5:483d/64 scope link 2024-03-24T00:49:30.218 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:49:30.218 INFO:teuthology.orchestra.run.smithi067.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T00:49:30.218 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 96:cf:26:c8:bf:62 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:49:30.219 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::94cf:26ff:fec8:bf62/64 scope link 2024-03-24T00:49:30.219 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:49:30.219 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:30.219 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T00:49:30.281 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-03-24T00:49:30.282 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:30.283 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-03-24T00:49:30.337 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-03-24T00:49:30.338 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T00:49:30.338 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-24T00:49:30.338 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-24T00:49:30.338 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T00:49:30.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:30.338 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-03-24T00:49:30.338 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-03-24T00:49:30.338 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-24T00:49:30.338 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-24T00:49:30.339 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-24T00:49:30.339 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-03-24T00:49:30.435 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:30.435 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-03-24T00:49:30.450 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-24T00:49:30.450 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-24T00:49:30.450 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:49:30.450 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:30.451 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-03-24T00:49:30.516 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:49:30.517 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-24T00:49:30.571 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:49:30.572 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:49:30.572 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:49:30.626 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-24T00:49:30.627 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:30.627 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:49:30.714 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:49:30.714 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T00:49:30.714 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-03-24T00:49:30.722 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:49:30.722 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T00:49:30.722 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-03-24T00:49:30.722 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.72:3300,172.21.15.72:3301" 2024-03-24T00:49:30.722 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-24T00:49:30.727 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T00:49:30.727 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T00:49:30.727 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-03-24T00:49:30.727 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.72:3300,172.21.15.72:3301" 2024-03-24T00:49:30.727 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-24T00:49:30.727 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:6789,172.21.15.72:6789,172.21.15.72:6790" 2024-03-24T00:49:30.728 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.67:6789,172.21.15.72:6789,172.21.15.72:6790:/ 2024-03-24T00:49:30.728 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=de1d0178-808e-49ca-a986-e297e523b976" will pass to kernel 2024-03-24T00:49:30.731 INFO:teuthology.orchestra.run.smithi067.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-24T00:49:30.731 INFO:teuthology.orchestra.run.smithi067.stdout: You just mounted a file system that supports labels which does not 2024-03-24T00:49:30.731 INFO:teuthology.orchestra.run.smithi067.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-24T00:49:30.731 INFO:teuthology.orchestra.run.smithi067.stdout: applications will generate AVC messages and not be allowed access to 2024-03-24T00:49:30.731 INFO:teuthology.orchestra.run.smithi067.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-24T00:49:30.731 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-24T00:49:30.731 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:30.731 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:49:30.758 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:30.758 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T00:49:30.813 INFO:teuthology.orchestra.run.smithi067.stdout:client9774 2024-03-24T00:49:30.813 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:49:30.813 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:49:30.868 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:30.869 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec find ./) 2024-03-24T00:49:30.927 INFO:teuthology.orchestra.run.smithi067.stdout:./ 2024-03-24T00:49:30.927 INFO:teuthology.orchestra.run.smithi067.stdout:./parent 2024-03-24T00:49:30.927 INFO:teuthology.orchestra.run.smithi067.stdout:./parent/flushed 2024-03-24T00:49:30.927 INFO:teuthology.orchestra.run.smithi067.stdout:./parent/flushed/charlie 2024-03-24T00:49:30.927 INFO:teuthology.orchestra.run.smithi067.stdout:./parent/flushed/alpha 2024-03-24T00:49:30.927 INFO:teuthology.orchestra.run.smithi067.stdout:./parent/flushed/bravo 2024-03-24T00:49:30.927 INFO:teuthology.orchestra.run.smithi067.stdout:./parent/unflushed 2024-03-24T00:49:30.928 INFO:teuthology.orchestra.run.smithi067.stdout:./parent/unflushed/jfile 2024-03-24T00:49:30.928 DEBUG:teuthology.orchestra.run.smithi067:> stdin-killer -- python3 -c ' 2024-03-24T00:49:30.928 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-03-24T00:49:30.928 DEBUG:teuthology.orchestra.run.smithi067:> import stat 2024-03-24T00:49:30.928 DEBUG:teuthology.orchestra.run.smithi067:> 2024-03-24T00:49:30.928 DEBUG:teuthology.orchestra.run.smithi067:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./").st_ino) 2024-03-24T00:49:30.928 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-03-24T00:49:31.032 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-03-24T00:49:31.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:49:31.040 DEBUG:teuthology.orchestra.run.smithi067:> stdin-killer -- python3 -c ' 2024-03-24T00:49:31.041 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-03-24T00:49:31.041 DEBUG:teuthology.orchestra.run.smithi067:> import stat 2024-03-24T00:49:31.041 DEBUG:teuthology.orchestra.run.smithi067:> 2024-03-24T00:49:31.041 DEBUG:teuthology.orchestra.run.smithi067:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent").st_ino) 2024-03-24T00:49:31.041 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-03-24T00:49:31.146 INFO:teuthology.orchestra.run.smithi067.stdout:1099511627776 2024-03-24T00:49:31.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:49:31.154 DEBUG:teuthology.orchestra.run.smithi067:> stdin-killer -- python3 -c ' 2024-03-24T00:49:31.154 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-03-24T00:49:31.154 DEBUG:teuthology.orchestra.run.smithi067:> import stat 2024-03-24T00:49:31.154 DEBUG:teuthology.orchestra.run.smithi067:> 2024-03-24T00:49:31.154 DEBUG:teuthology.orchestra.run.smithi067:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed").st_ino) 2024-03-24T00:49:31.154 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-03-24T00:49:31.260 INFO:teuthology.orchestra.run.smithi067.stdout:1099511627777 2024-03-24T00:49:31.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:49:31.268 DEBUG:teuthology.orchestra.run.smithi067:> stdin-killer -- python3 -c ' 2024-03-24T00:49:31.269 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-03-24T00:49:31.269 DEBUG:teuthology.orchestra.run.smithi067:> import stat 2024-03-24T00:49:31.269 DEBUG:teuthology.orchestra.run.smithi067:> 2024-03-24T00:49:31.269 DEBUG:teuthology.orchestra.run.smithi067:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed/charlie").st_ino) 2024-03-24T00:49:31.269 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-03-24T00:49:31.374 INFO:teuthology.orchestra.run.smithi067.stdout:1099511628280 2024-03-24T00:49:31.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:49:31.383 DEBUG:teuthology.orchestra.run.smithi067:> stdin-killer -- python3 -c ' 2024-03-24T00:49:31.383 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-03-24T00:49:31.383 DEBUG:teuthology.orchestra.run.smithi067:> import stat 2024-03-24T00:49:31.383 DEBUG:teuthology.orchestra.run.smithi067:> 2024-03-24T00:49:31.383 DEBUG:teuthology.orchestra.run.smithi067:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed/alpha").st_ino) 2024-03-24T00:49:31.383 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-03-24T00:49:31.489 INFO:teuthology.orchestra.run.smithi067.stdout:1099511627778 2024-03-24T00:49:31.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:49:31.497 DEBUG:teuthology.orchestra.run.smithi067:> stdin-killer -- python3 -c ' 2024-03-24T00:49:31.497 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-03-24T00:49:31.497 DEBUG:teuthology.orchestra.run.smithi067:> import stat 2024-03-24T00:49:31.497 DEBUG:teuthology.orchestra.run.smithi067:> 2024-03-24T00:49:31.498 DEBUG:teuthology.orchestra.run.smithi067:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed/bravo").st_ino) 2024-03-24T00:49:31.498 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-03-24T00:49:31.603 INFO:teuthology.orchestra.run.smithi067.stdout:1099511628279 2024-03-24T00:49:31.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:49:31.612 DEBUG:teuthology.orchestra.run.smithi067:> stdin-killer -- python3 -c ' 2024-03-24T00:49:31.612 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-03-24T00:49:31.612 DEBUG:teuthology.orchestra.run.smithi067:> import stat 2024-03-24T00:49:31.612 DEBUG:teuthology.orchestra.run.smithi067:> 2024-03-24T00:49:31.612 DEBUG:teuthology.orchestra.run.smithi067:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/unflushed").st_ino) 2024-03-24T00:49:31.612 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-03-24T00:49:31.723 INFO:teuthology.orchestra.run.smithi067.stdout:1099511627779 2024-03-24T00:49:31.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:49:31.734 DEBUG:teuthology.orchestra.run.smithi067:> stdin-killer -- python3 -c ' 2024-03-24T00:49:31.734 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-03-24T00:49:31.735 DEBUG:teuthology.orchestra.run.smithi067:> import stat 2024-03-24T00:49:31.735 DEBUG:teuthology.orchestra.run.smithi067:> 2024-03-24T00:49:31.735 DEBUG:teuthology.orchestra.run.smithi067:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/unflushed/jfile").st_ino) 2024-03-24T00:49:31.735 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-03-24T00:49:31.838 INFO:teuthology.orchestra.run.smithi067.stdout:1099511627780 2024-03-24T00:49:31.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:49:31.847 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-24T00:49:31.847 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:49:31.928 DEBUG:teuthology.orchestra.run.smithi072:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:49:31.972 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-24T00:49:31.972 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:49:31.972 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:49:31.987 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:49:31.988 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:49:32.042 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:49:32.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.181+0000 7fb1eba80640 1 Processor -- start 2024-03-24T00:49:32.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.181+0000 7fb1eba80640 1 -- start start 2024-03-24T00:49:32.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.182+0000 7fb1eba80640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb1e40b49c0 0x7fb1e415b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:32.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.182+0000 7fb1eba80640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb1e406f100 0x7fb1e4072360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:32.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.182+0000 7fb1eba80640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1e40b4da0 con 0x7fb1e40728a0 2024-03-24T00:49:32.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.182+0000 7fb1eba80640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1e406db30 con 0x7fb1e406f100 2024-03-24T00:49:32.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.182+0000 7fb1eba80640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e406de10 con 0x7fb1e40b49c0 2024-03-24T00:49:32.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.182+0000 7fb1e901b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb1e406f100 0x7fb1e4072360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:32.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.182+0000 7fb1e901b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb1e406f100 0x7fb1e4072360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38594/0 (socket says 172.21.15.67:38594) 2024-03-24T00:49:32.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.182+0000 7fb1e901b640 1 -- 172.21.15.67:0/1878554354 learned_addr learned my addr 172.21.15.67:0/1878554354 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:32.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.183+0000 7fb1e981c640 1 --2- 172.21.15.67:0/1878554354 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb1e40b49c0 0x7fb1e415b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:32.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.183+0000 7fb1e901b640 1 -- 172.21.15.67:0/1878554354 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb1e40b49c0 msgr2=0x7fb1e415b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:32.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.183+0000 7fb1e901b640 1 --2- 172.21.15.67:0/1878554354 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb1e40b49c0 0x7fb1e415b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.183+0000 7fb1e901b640 1 -- 172.21.15.67:0/1878554354 >> v1:172.21.15.72:6789/0 conn(0x7fb1e40728a0 legacy=0x7fb1e415df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:32.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.183+0000 7fb1e901b640 1 -- 172.21.15.67:0/1878554354 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e406e0f0 con 0x7fb1e406f100 2024-03-24T00:49:32.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.183+0000 7fb1e901b640 1 --2- 172.21.15.67:0/1878554354 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1e406f100 0x7fb1e4072360 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7fb1d800bb10 tx=0x7fb1d802e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d1e422386c6e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:32.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.184+0000 7fb1e881a640 1 -- 172.21.15.67:0/1878554354 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d803b070 con 0x7fb1e406f100 2024-03-24T00:49:32.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.184+0000 7fb1e881a640 1 -- 172.21.15.67:0/1878554354 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1d8036440 con 0x7fb1e406f100 2024-03-24T00:49:32.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.184+0000 7fb1e881a640 1 -- 172.21.15.67:0/1878554354 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d8036810 con 0x7fb1e406f100 2024-03-24T00:49:32.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.184+0000 7fb1eba80640 1 -- 172.21.15.67:0/1878554354 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1e406f100 msgr2=0x7fb1e4072360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:32.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.184+0000 7fb1eba80640 1 --2- 172.21.15.67:0/1878554354 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1e406f100 0x7fb1e4072360 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7fb1d800bb10 tx=0x7fb1d802e0c0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.184+0000 7fb1eba80640 1 -- 172.21.15.67:0/1878554354 shutdown_connections 2024-03-24T00:49:32.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.184+0000 7fb1eba80640 1 --2- 172.21.15.67:0/1878554354 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1e406f100 0x7fb1e4072360 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.184+0000 7fb1eba80640 1 --2- 172.21.15.67:0/1878554354 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb1e40b49c0 0x7fb1e415b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.184+0000 7fb1eba80640 1 -- 172.21.15.67:0/1878554354 >> 172.21.15.67:0/1878554354 conn(0x7fb1e40cf9e0 msgr2=0x7fb1e40cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:32.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.184+0000 7fb1eba80640 1 -- 172.21.15.67:0/1878554354 shutdown_connections 2024-03-24T00:49:32.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.184+0000 7fb1eba80640 1 -- 172.21.15.67:0/1878554354 wait complete. 2024-03-24T00:49:32.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.185+0000 7fb1eba80640 1 Processor -- start 2024-03-24T00:49:32.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.185+0000 7fb1eba80640 1 -- start start 2024-03-24T00:49:32.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.186+0000 7fb1eba80640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb1e406f100 0x7fb1e4148530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:32.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.186+0000 7fb1eba80640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1e40728a0 0x7fb1e4149470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:32.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.186+0000 7fb1eba80640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1e40b49c0 0x7fb1e41499b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:32.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.186+0000 7fb1eba80640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e406f560 con 0x7fb1e40728a0 2024-03-24T00:49:32.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.186+0000 7fb1eba80640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e406e6b0 con 0x7fb1e40b49c0 2024-03-24T00:49:32.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.186+0000 7fb1eba80640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e40b3d30 con 0x7fb1e406f100 2024-03-24T00:49:32.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.186+0000 7fb1e901b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1e40728a0 0x7fb1e4149470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:32.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.186+0000 7fb1e901b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1e40728a0 0x7fb1e4149470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38596/0 (socket says 172.21.15.67:38596) 2024-03-24T00:49:32.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.186+0000 7fb1e901b640 1 -- 172.21.15.67:0/722795746 learned_addr learned my addr 172.21.15.67:0/722795746 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:32.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.186+0000 7fb1e981c640 1 --2- 172.21.15.67:0/722795746 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb1e406f100 0x7fb1e4148530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:32.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.186+0000 7fb1ea01d640 1 --2- 172.21.15.67:0/722795746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1e40b49c0 0x7fb1e41499b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:32.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.187+0000 7fb1e901b640 1 -- 172.21.15.67:0/722795746 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb1e406f100 msgr2=0x7fb1e4148530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:32.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.187+0000 7fb1e901b640 1 --2- 172.21.15.67:0/722795746 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb1e406f100 0x7fb1e4148530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.187+0000 7fb1e901b640 1 -- 172.21.15.67:0/722795746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1e40b49c0 msgr2=0x7fb1e41499b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:32.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.187+0000 7fb1e901b640 1 --2- 172.21.15.67:0/722795746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1e40b49c0 0x7fb1e41499b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.187+0000 7fb1e901b640 1 -- 172.21.15.67:0/722795746 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e41f49f0 con 0x7fb1e40728a0 2024-03-24T00:49:32.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.187+0000 7fb1e981c640 1 --2- 172.21.15.67:0/722795746 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb1e406f100 0x7fb1e4148530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:49:32.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.187+0000 7fb1e901b640 1 --2- 172.21.15.67:0/722795746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1e40728a0 0x7fb1e4149470 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7fb1d800b2c0 tx=0x7fb1d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:32.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.187+0000 7fb1e27fc640 1 -- 172.21.15.67:0/722795746 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d803b030 con 0x7fb1e40728a0 2024-03-24T00:49:32.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.187+0000 7fb1e27fc640 1 -- 172.21.15.67:0/722795746 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1d803d050 con 0x7fb1e40728a0 2024-03-24T00:49:32.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.187+0000 7fb1eba80640 1 -- 172.21.15.67:0/722795746 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1e41f4cd0 con 0x7fb1e40728a0 2024-03-24T00:49:32.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.187+0000 7fb1e27fc640 1 -- 172.21.15.67:0/722795746 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d8034870 con 0x7fb1e40728a0 2024-03-24T00:49:32.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.187+0000 7fb1eba80640 1 -- 172.21.15.67:0/722795746 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1e41f5410 con 0x7fb1e40728a0 2024-03-24T00:49:32.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.188+0000 7fb1eba80640 1 -- 172.21.15.67:0/722795746 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1e404f400 con 0x7fb1e40728a0 2024-03-24T00:49:32.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.188+0000 7fb1e27fc640 1 -- 172.21.15.67:0/722795746 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb1d8048060 con 0x7fb1e40728a0 2024-03-24T00:49:32.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.189+0000 7fb1e27fc640 1 --2- 172.21.15.67:0/722795746 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb1d006bbd0 0x7fb1d006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:32.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.189+0000 7fb1e27fc640 1 -- 172.21.15.67:0/722795746 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6273+0+0 (secure 0 0 0) 0x7fb1d80ac260 con 0x7fb1e40728a0 2024-03-24T00:49:32.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.189+0000 7fb1e981c640 1 --2- 172.21.15.67:0/722795746 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb1d006bbd0 0x7fb1d006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:32.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.190+0000 7fb1e981c640 1 --2- 172.21.15.67:0/722795746 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb1d006bbd0 0x7fb1d006e090 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4009fd0 tx=0x7fb1d400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:32.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.193+0000 7fb1e27fc640 1 -- 172.21.15.67:0/722795746 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb1d807ab10 con 0x7fb1e40728a0 2024-03-24T00:49:32.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.379+0000 7fb1eba80640 1 -- 172.21.15.67:0/722795746 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb1e40b3390 con 0x7fb1e40728a0 2024-03-24T00:49:32.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.380+0000 7fb1e27fc640 1 -- 172.21.15.67:0/722795746 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v122) v1 ==== 74+0+17149 (secure 0 0 0) 0x7fb1d807a410 con 0x7fb1e40728a0 2024-03-24T00:49:32.381 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:32.381 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":122,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:49:23.316053+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:48:52.622356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-03-24T00:48:53.311452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6835/3833808901":"2024-03-25T00:49:23.316030+0000","172.21.15.67:6833/3833808901":"2024-03-25T00:49:23.316030+0000","172.21.15.67:6835/312063805":"2024-03-25T00:49:12.456364+0000","172.21.15.67:6833/312063805":"2024-03-25T00:49:12.456364+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:49:32.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.382+0000 7fb1eba80640 1 -- 172.21.15.67:0/722795746 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb1d006bbd0 msgr2=0x7fb1d006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:32.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.382+0000 7fb1eba80640 1 --2- 172.21.15.67:0/722795746 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb1d006bbd0 0x7fb1d006e090 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4009fd0 tx=0x7fb1d400b040 comp rx=0 tx=0).stop 2024-03-24T00:49:32.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.382+0000 7fb1eba80640 1 -- 172.21.15.67:0/722795746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1e40728a0 msgr2=0x7fb1e4149470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:32.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.382+0000 7fb1eba80640 1 --2- 172.21.15.67:0/722795746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1e40728a0 0x7fb1e4149470 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7fb1d800b2c0 tx=0x7fb1d80046c0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.383+0000 7fb1eba80640 1 -- 172.21.15.67:0/722795746 shutdown_connections 2024-03-24T00:49:32.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.383+0000 7fb1eba80640 1 --2- 172.21.15.67:0/722795746 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb1d006bbd0 0x7fb1d006e090 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.383+0000 7fb1eba80640 1 --2- 172.21.15.67:0/722795746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1e40b49c0 0x7fb1e41499b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.383+0000 7fb1eba80640 1 --2- 172.21.15.67:0/722795746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1e40728a0 0x7fb1e4149470 unknown :-1 s=CLOSED pgs=1868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.383+0000 7fb1eba80640 1 --2- 172.21.15.67:0/722795746 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb1e406f100 0x7fb1e4148530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.383+0000 7fb1eba80640 1 -- 172.21.15.67:0/722795746 >> 172.21.15.67:0/722795746 conn(0x7fb1e40cf9e0 msgr2=0x7fb1e4071ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:32.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.383+0000 7fb1eba80640 1 -- 172.21.15.67:0/722795746 shutdown_connections 2024-03-24T00:49:32.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.383+0000 7fb1eba80640 1 -- 172.21.15.67:0/722795746 wait complete. 2024-03-24T00:49:32.394 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:38:39.364950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:48:52.622356+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '119', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:48:53.311452+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '120', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:49:32.395 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:49:32.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.498+0000 7fb0b71c4640 1 Processor -- start 2024-03-24T00:49:32.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.498+0000 7fb0b71c4640 1 -- start start 2024-03-24T00:49:32.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.499+0000 7fb0b71c4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb0b015dea0 0x7fb0b00d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:32.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.499+0000 7fb0b71c4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb0b015e280 0x7fb0b00d3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:32.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.499+0000 7fb0b71c4640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0b00d2250 con 0x7fb0b00b5a60 2024-03-24T00:49:32.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.499+0000 7fb0b71c4640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0b00d2530 con 0x7fb0b015dea0 2024-03-24T00:49:32.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.499+0000 7fb0b71c4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0b00d2810 con 0x7fb0b015e280 2024-03-24T00:49:32.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.499+0000 7fb0b4f60640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb0b015dea0 0x7fb0b00d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:32.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.499+0000 7fb0b4f60640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb0b015dea0 0x7fb0b00d33d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38610/0 (socket says 172.21.15.67:38610) 2024-03-24T00:49:32.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.499+0000 7fb0b4f60640 1 -- 172.21.15.67:0/757266874 learned_addr learned my addr 172.21.15.67:0/757266874 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:32.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.499+0000 7fb0affff640 1 --2- 172.21.15.67:0/757266874 >> v2:172.21.15.72:3300/0 conn(0x7fb0b015e280 0x7fb0b00d3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:32.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.499+0000 7fb0b4f60640 1 -- 172.21.15.67:0/757266874 >> v2:172.21.15.72:3300/0 conn(0x7fb0b015e280 msgr2=0x7fb0b00d3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:32.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.499+0000 7fb0b4f60640 1 --2- 172.21.15.67:0/757266874 >> v2:172.21.15.72:3300/0 conn(0x7fb0b015e280 0x7fb0b00d3910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.499+0000 7fb0b4f60640 1 -- 172.21.15.67:0/757266874 >> v1:172.21.15.72:6789/0 conn(0x7fb0b00b5a60 legacy=0x7fb0b006c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:32.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.499+0000 7fb0b4f60640 1 -- 172.21.15.67:0/757266874 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0b00d2af0 con 0x7fb0b015dea0 2024-03-24T00:49:32.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.500+0000 7fb0b4f60640 1 --2- 172.21.15.67:0/757266874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0b015dea0 0x7fb0b00d33d0 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7fb0b00d04d0 tx=0x7fb0a402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7abb8619efd5d7a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:32.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.500+0000 7fb0af7fe640 1 -- 172.21.15.67:0/757266874 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a402e070 con 0x7fb0b015dea0 2024-03-24T00:49:32.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.500+0000 7fb0af7fe640 1 -- 172.21.15.67:0/757266874 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0a4037070 con 0x7fb0b015dea0 2024-03-24T00:49:32.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.500+0000 7fb0af7fe640 1 -- 172.21.15.67:0/757266874 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a4031a00 con 0x7fb0b015dea0 2024-03-24T00:49:32.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.501+0000 7fb0b71c4640 1 -- 172.21.15.67:0/757266874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0b015dea0 msgr2=0x7fb0b00d33d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:32.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.501+0000 7fb0b71c4640 1 --2- 172.21.15.67:0/757266874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0b015dea0 0x7fb0b00d33d0 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7fb0b00d04d0 tx=0x7fb0a402b880 comp rx=0 tx=0).stop 2024-03-24T00:49:32.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.501+0000 7fb0b71c4640 1 -- 172.21.15.67:0/757266874 shutdown_connections 2024-03-24T00:49:32.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.501+0000 7fb0b71c4640 1 --2- 172.21.15.67:0/757266874 >> v2:172.21.15.72:3300/0 conn(0x7fb0b015e280 0x7fb0b00d3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.501+0000 7fb0b71c4640 1 --2- 172.21.15.67:0/757266874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0b015dea0 0x7fb0b00d33d0 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.501+0000 7fb0b71c4640 1 -- 172.21.15.67:0/757266874 >> 172.21.15.67:0/757266874 conn(0x7fb0b0072da0 msgr2=0x7fb0b00d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:32.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.501+0000 7fb0b71c4640 1 -- 172.21.15.67:0/757266874 shutdown_connections 2024-03-24T00:49:32.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.501+0000 7fb0b71c4640 1 -- 172.21.15.67:0/757266874 wait complete. 2024-03-24T00:49:32.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.502+0000 7fb0b71c4640 1 Processor -- start 2024-03-24T00:49:32.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.502+0000 7fb0b71c4640 1 -- start start 2024-03-24T00:49:32.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.503+0000 7fb0b71c4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb0b00b5a60 0x7fb0b0159540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:32.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.503+0000 7fb0b71c4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0b015dea0 0x7fb0b0157060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:32.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.503+0000 7fb0b71c4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb0b015e280 0x7fb0b01e6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:32.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.503+0000 7fb0b71c4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b00b5eb0 con 0x7fb0b015dea0 2024-03-24T00:49:32.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.503+0000 7fb0b71c4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b00b8310 con 0x7fb0b00b5a60 2024-03-24T00:49:32.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.503+0000 7fb0b71c4640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b00b6030 con 0x7fb0b015e280 2024-03-24T00:49:32.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.503+0000 7fb0affff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0b015dea0 0x7fb0b0157060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:32.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.503+0000 7fb0affff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0b015dea0 0x7fb0b0157060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38616/0 (socket says 172.21.15.67:38616) 2024-03-24T00:49:32.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.503+0000 7fb0affff640 1 -- 172.21.15.67:0/2426573439 learned_addr learned my addr 172.21.15.67:0/2426573439 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:32.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.503+0000 7fb0b4f60640 1 --2- 172.21.15.67:0/2426573439 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb0b00b5a60 0x7fb0b0159540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:32.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.503+0000 7fb0b5761640 1 --2- 172.21.15.67:0/2426573439 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb0b015e280 0x7fb0b01e6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:32.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.503+0000 7fb0affff640 1 -- 172.21.15.67:0/2426573439 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb0b015e280 msgr2=0x7fb0b01e6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:32.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.503+0000 7fb0affff640 1 --2- 172.21.15.67:0/2426573439 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb0b015e280 0x7fb0b01e6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.503+0000 7fb0affff640 1 -- 172.21.15.67:0/2426573439 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb0b00b5a60 msgr2=0x7fb0b0159540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:32.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.503+0000 7fb0affff640 1 --2- 172.21.15.67:0/2426573439 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb0b00b5a60 0x7fb0b0159540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.503+0000 7fb0affff640 1 -- 172.21.15.67:0/2426573439 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0b0158ad0 con 0x7fb0b015dea0 2024-03-24T00:49:32.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.503+0000 7fb0affff640 1 --2- 172.21.15.67:0/2426573439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0b015dea0 0x7fb0b0157060 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7fb09c00c5e0 tx=0x7fb09c00caa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:32.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.504+0000 7fb0ad7fa640 1 -- 172.21.15.67:0/2426573439 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09c015070 con 0x7fb0b015dea0 2024-03-24T00:49:32.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.504+0000 7fb0ad7fa640 1 -- 172.21.15.67:0/2426573439 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb09c004260 con 0x7fb0b015dea0 2024-03-24T00:49:32.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.504+0000 7fb0ad7fa640 1 -- 172.21.15.67:0/2426573439 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09c003610 con 0x7fb0b015dea0 2024-03-24T00:49:32.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.504+0000 7fb0b71c4640 1 -- 172.21.15.67:0/2426573439 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0b015aff0 con 0x7fb0b015dea0 2024-03-24T00:49:32.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.504+0000 7fb0b71c4640 1 -- 172.21.15.67:0/2426573439 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0b0159020 con 0x7fb0b015dea0 2024-03-24T00:49:32.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.505+0000 7fb0b71c4640 1 -- 172.21.15.67:0/2426573439 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0b0157660 con 0x7fb0b015dea0 2024-03-24T00:49:32.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.505+0000 7fb0ad7fa640 1 -- 172.21.15.67:0/2426573439 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb09c0037b0 con 0x7fb0b015dea0 2024-03-24T00:49:32.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.506+0000 7fb0ad7fa640 1 --2- 172.21.15.67:0/2426573439 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb09406bbd0 0x7fb09406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:32.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.506+0000 7fb0ad7fa640 1 -- 172.21.15.67:0/2426573439 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6273+0+0 (secure 0 0 0) 0x7fb09c08ab10 con 0x7fb0b015dea0 2024-03-24T00:49:32.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.506+0000 7fb0b4f60640 1 --2- 172.21.15.67:0/2426573439 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb09406bbd0 0x7fb09406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:32.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.507+0000 7fb0b4f60640 1 --2- 172.21.15.67:0/2426573439 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb09406bbd0 0x7fb09406e090 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fb0a4036000 tx=0x7fb0a403d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:32.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.510+0000 7fb0ad7fa640 1 -- 172.21.15.67:0/2426573439 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb0b0157660 con 0x7fb0b015dea0 2024-03-24T00:49:32.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.763+0000 7fb0b71c4640 1 -- 172.21.15.67:0/2426573439 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb0b004f400 con 0x7fb0b015dea0 2024-03-24T00:49:32.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.763+0000 7fb0ad7fa640 1 -- 172.21.15.67:0/2426573439 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v122) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb09c058cf0 con 0x7fb0b015dea0 2024-03-24T00:49:32.764 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-24T00:49:32.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.766+0000 7fb0b71c4640 1 -- 172.21.15.67:0/2426573439 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb09406bbd0 msgr2=0x7fb09406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:32.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.766+0000 7fb0b71c4640 1 --2- 172.21.15.67:0/2426573439 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb09406bbd0 0x7fb09406e090 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fb0a4036000 tx=0x7fb0a403d040 comp rx=0 tx=0).stop 2024-03-24T00:49:32.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.766+0000 7fb0b71c4640 1 -- 172.21.15.67:0/2426573439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0b015dea0 msgr2=0x7fb0b0157060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:32.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.766+0000 7fb0b71c4640 1 --2- 172.21.15.67:0/2426573439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0b015dea0 0x7fb0b0157060 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7fb09c00c5e0 tx=0x7fb09c00caa0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.766+0000 7fb0b71c4640 1 -- 172.21.15.67:0/2426573439 shutdown_connections 2024-03-24T00:49:32.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.766+0000 7fb0b71c4640 1 --2- 172.21.15.67:0/2426573439 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb09406bbd0 0x7fb09406e090 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.766+0000 7fb0b71c4640 1 --2- 172.21.15.67:0/2426573439 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb0b015e280 0x7fb0b01e6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.766+0000 7fb0b71c4640 1 --2- 172.21.15.67:0/2426573439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0b015dea0 0x7fb0b0157060 unknown :-1 s=CLOSED pgs=1870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.766+0000 7fb0b71c4640 1 --2- 172.21.15.67:0/2426573439 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb0b00b5a60 0x7fb0b0159540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.766+0000 7fb0b71c4640 1 -- 172.21.15.67:0/2426573439 >> 172.21.15.67:0/2426573439 conn(0x7fb0b0072da0 msgr2=0x7fb0b00b42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:32.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.767+0000 7fb0b71c4640 1 -- 172.21.15.67:0/2426573439 shutdown_connections 2024-03-24T00:49:32.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.767+0000 7fb0b71c4640 1 -- 172.21.15.67:0/2426573439 wait complete. 2024-03-24T00:49:32.779 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:49:32.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.886+0000 7ff7d9e27640 1 Processor -- start 2024-03-24T00:49:32.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.886+0000 7ff7d9e27640 1 -- start start 2024-03-24T00:49:32.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.887+0000 7ff7d9e27640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff7d415bc30 0x7ff7d40b1a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:32.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.887+0000 7ff7d9e27640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff7d415c010 0x7ff7d40b1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:32.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.887+0000 7ff7d9e27640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7d40b3ae0 con 0x7ff7d40b2d90 2024-03-24T00:49:32.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.887+0000 7ff7d9e27640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7d40cca60 con 0x7ff7d415bc30 2024-03-24T00:49:32.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.887+0000 7ff7d9e27640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d40b3dc0 con 0x7ff7d415c010 2024-03-24T00:49:32.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.887+0000 7ff7d37fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff7d415bc30 0x7ff7d40b1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:32.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.887+0000 7ff7d3fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff7d40b2d90 0x7ff7d40b3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50810/0 (socket says 172.21.15.67:50810) 2024-03-24T00:49:32.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.887+0000 7ff7d3fff640 1 -- 172.21.15.67:0/3551210269 learned_addr learned my addr 172.21.15.67:0/3551210269 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:49:32.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.887+0000 7ff7d37fe640 1 -- 172.21.15.67:0/3551210269 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff7d415c010 msgr2=0x7ff7d40b1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:32.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.887+0000 7ff7d37fe640 1 --2- 172.21.15.67:0/3551210269 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff7d415c010 0x7ff7d40b1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.887+0000 7ff7d37fe640 1 -- 172.21.15.67:0/3551210269 >> v1:172.21.15.67:6789/0 conn(0x7ff7d40b2d90 legacy=0x7ff7d40b3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:32.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.887+0000 7ff7d37fe640 1 -- 172.21.15.67:0/3551210269 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7d40b40a0 con 0x7ff7d415bc30 2024-03-24T00:49:32.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.888+0000 7ff7d37fe640 1 --2- 172.21.15.67:0/3551210269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7d415bc30 0x7ff7d40b1a00 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7ff7c40097a0 tx=0x7ff7c402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62a496c7bcbd5998 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:32.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.888+0000 7ff7d27fc640 1 -- 172.21.15.67:0/3551210269 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c4035020 con 0x7ff7d415bc30 2024-03-24T00:49:32.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.888+0000 7ff7d27fc640 1 -- 172.21.15.67:0/3551210269 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7c4031830 con 0x7ff7d415bc30 2024-03-24T00:49:32.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.888+0000 7ff7d27fc640 1 -- 172.21.15.67:0/3551210269 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c4031c00 con 0x7ff7d415bc30 2024-03-24T00:49:32.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.889+0000 7ff7d9e27640 1 -- 172.21.15.67:0/3551210269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7d415bc30 msgr2=0x7ff7d40b1a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:32.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.889+0000 7ff7d9e27640 1 --2- 172.21.15.67:0/3551210269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7d415bc30 0x7ff7d40b1a00 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7ff7c40097a0 tx=0x7ff7c402b8d0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.889+0000 7ff7d9e27640 1 -- 172.21.15.67:0/3551210269 shutdown_connections 2024-03-24T00:49:32.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.889+0000 7ff7d9e27640 1 --2- 172.21.15.67:0/3551210269 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff7d415c010 0x7ff7d40b1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.889+0000 7ff7d9e27640 1 --2- 172.21.15.67:0/3551210269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7d415bc30 0x7ff7d40b1a00 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.889+0000 7ff7d9e27640 1 -- 172.21.15.67:0/3551210269 >> 172.21.15.67:0/3551210269 conn(0x7ff7d408b180 msgr2=0x7ff7d408b590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:32.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.889+0000 7ff7d9e27640 1 -- 172.21.15.67:0/3551210269 shutdown_connections 2024-03-24T00:49:32.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.889+0000 7ff7d9e27640 1 -- 172.21.15.67:0/3551210269 wait complete. 2024-03-24T00:49:32.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.890+0000 7ff7d9e27640 1 Processor -- start 2024-03-24T00:49:32.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.890+0000 7ff7d9e27640 1 -- start start 2024-03-24T00:49:32.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.891+0000 7ff7d9e27640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7d40b2d90 0x7ff7d40ae690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:32.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.891+0000 7ff7d9e27640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff7d415bc30 0x7ff7d415d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:32.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.891+0000 7ff7d9e27640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff7d415c010 0x7ff7d40c4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:32.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.891+0000 7ff7d9e27640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d40cd070 con 0x7ff7d40b2d90 2024-03-24T00:49:32.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.891+0000 7ff7d9e27640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d40ccef0 con 0x7ff7d415c010 2024-03-24T00:49:32.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.891+0000 7ff7d9e27640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d40cd1f0 con 0x7ff7d415bc30 2024-03-24T00:49:32.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.891+0000 7ff7d37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7d40b2d90 0x7ff7d40ae690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:32.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.891+0000 7ff7d37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7d40b2d90 0x7ff7d40ae690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38634/0 (socket says 172.21.15.67:38634) 2024-03-24T00:49:32.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.891+0000 7ff7d37fe640 1 -- 172.21.15.67:0/2626887149 learned_addr learned my addr 172.21.15.67:0/2626887149 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:32.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.891+0000 7ff7d37fe640 1 -- 172.21.15.67:0/2626887149 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff7d415bc30 msgr2=0x7ff7d415d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:32.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.891+0000 7ff7d2ffd640 1 --2- 172.21.15.67:0/2626887149 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff7d415bc30 0x7ff7d415d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:32.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.891+0000 7ff7d37fe640 1 --2- 172.21.15.67:0/2626887149 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff7d415bc30 0x7ff7d415d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.891+0000 7ff7d37fe640 1 -- 172.21.15.67:0/2626887149 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff7d415c010 msgr2=0x7ff7d40c4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:32.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.891+0000 7ff7d37fe640 1 --2- 172.21.15.67:0/2626887149 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff7d415c010 0x7ff7d40c4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:32.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.891+0000 7ff7d37fe640 1 -- 172.21.15.67:0/2626887149 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7d406c4c0 con 0x7ff7d40b2d90 2024-03-24T00:49:32.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.891+0000 7ff7d37fe640 1 --2- 172.21.15.67:0/2626887149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7d40b2d90 0x7ff7d40ae690 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7ff7c40098c0 tx=0x7ff7c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:32.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.891+0000 7ff7bbfff640 1 -- 172.21.15.67:0/2626887149 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c4041070 con 0x7ff7d40b2d90 2024-03-24T00:49:32.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.891+0000 7ff7bbfff640 1 -- 172.21.15.67:0/2626887149 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7c403aa30 con 0x7ff7d40b2d90 2024-03-24T00:49:32.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.891+0000 7ff7bbfff640 1 -- 172.21.15.67:0/2626887149 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c4035040 con 0x7ff7d40b2d90 2024-03-24T00:49:32.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.891+0000 7ff7d9e27640 1 -- 172.21.15.67:0/2626887149 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7d406c7a0 con 0x7ff7d40b2d90 2024-03-24T00:49:32.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.892+0000 7ff7d9e27640 1 -- 172.21.15.67:0/2626887149 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7d406cd60 con 0x7ff7d40b2d90 2024-03-24T00:49:32.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.893+0000 7ff7d9e27640 1 -- 172.21.15.67:0/2626887149 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7d404f400 con 0x7ff7d40b2d90 2024-03-24T00:49:32.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.893+0000 7ff7bbfff640 1 -- 172.21.15.67:0/2626887149 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff7c404c060 con 0x7ff7d40b2d90 2024-03-24T00:49:32.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.893+0000 7ff7bbfff640 1 --2- 172.21.15.67:0/2626887149 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff7c006bbd0 0x7ff7c006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:32.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.893+0000 7ff7bbfff640 1 -- 172.21.15.67:0/2626887149 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6273+0+0 (secure 0 0 0) 0x7ff7c40ac710 con 0x7ff7d40b2d90 2024-03-24T00:49:32.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.893+0000 7ff7d2ffd640 1 --2- 172.21.15.67:0/2626887149 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff7c006bbd0 0x7ff7c006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:32.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.894+0000 7ff7d2ffd640 1 --2- 172.21.15.67:0/2626887149 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff7c006bbd0 0x7ff7c006e090 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7ff7d40c4d80 tx=0x7ff7c8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:32.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:32.898+0000 7ff7bbfff640 1 -- 172.21.15.67:0/2626887149 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff7c407b050 con 0x7ff7d40b2d90 2024-03-24T00:49:33.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.087+0000 7ff7d9e27640 1 -- 172.21.15.67:0/2626887149 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff7d40d0920 con 0x7ff7d40b2d90 2024-03-24T00:49:33.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.087+0000 7ff7bbfff640 1 -- 172.21.15.67:0/2626887149 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v122) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff7c40aae20 con 0x7ff7d40b2d90 2024-03-24T00:49:33.088 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-24T00:49:33.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.089+0000 7ff7d9e27640 1 -- 172.21.15.67:0/2626887149 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff7c006bbd0 msgr2=0x7ff7c006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:33.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.089+0000 7ff7d9e27640 1 --2- 172.21.15.67:0/2626887149 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff7c006bbd0 0x7ff7c006e090 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7ff7d40c4d80 tx=0x7ff7c8008040 comp rx=0 tx=0).stop 2024-03-24T00:49:33.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.090+0000 7ff7d9e27640 1 -- 172.21.15.67:0/2626887149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7d40b2d90 msgr2=0x7ff7d40ae690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:33.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.090+0000 7ff7d9e27640 1 --2- 172.21.15.67:0/2626887149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7d40b2d90 0x7ff7d40ae690 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7ff7c40098c0 tx=0x7ff7c40046c0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.090+0000 7ff7d9e27640 1 -- 172.21.15.67:0/2626887149 shutdown_connections 2024-03-24T00:49:33.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.090+0000 7ff7d9e27640 1 --2- 172.21.15.67:0/2626887149 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff7c006bbd0 0x7ff7c006e090 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.090+0000 7ff7d9e27640 1 --2- 172.21.15.67:0/2626887149 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff7d415c010 0x7ff7d40c4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.090+0000 7ff7d9e27640 1 --2- 172.21.15.67:0/2626887149 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff7d415bc30 0x7ff7d415d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.090+0000 7ff7d9e27640 1 --2- 172.21.15.67:0/2626887149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7d40b2d90 0x7ff7d40ae690 unknown :-1 s=CLOSED pgs=1873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.090+0000 7ff7d9e27640 1 -- 172.21.15.67:0/2626887149 >> 172.21.15.67:0/2626887149 conn(0x7ff7d408b180 msgr2=0x7ff7d4070fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:33.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.090+0000 7ff7d9e27640 1 -- 172.21.15.67:0/2626887149 shutdown_connections 2024-03-24T00:49:33.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.090+0000 7ff7d9e27640 1 -- 172.21.15.67:0/2626887149 wait complete. 2024-03-24T00:49:33.101 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:49:33.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.206+0000 7fee7b513640 1 Processor -- start 2024-03-24T00:49:33.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.206+0000 7fee7b513640 1 -- start start 2024-03-24T00:49:33.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.207+0000 7fee7b513640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fee740d3ae0 0x7fee740b5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:33.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.207+0000 7fee7b513640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fee7415e280 0x7fee740b5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:33.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.207+0000 7fee7b513640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee740d2250 con 0x7fee7415dea0 2024-03-24T00:49:33.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.207+0000 7fee7b513640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee740d2530 con 0x7fee740d3ae0 2024-03-24T00:49:33.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.207+0000 7fee7b513640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee740d2810 con 0x7fee7415e280 2024-03-24T00:49:33.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.207+0000 7fee78aae640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fee740d3ae0 0x7fee740b5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:33.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.207+0000 7fee78aae640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fee740d3ae0 0x7fee740b5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38640/0 (socket says 172.21.15.67:38640) 2024-03-24T00:49:33.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.207+0000 7fee78aae640 1 -- 172.21.15.67:0/4278190527 learned_addr learned my addr 172.21.15.67:0/4278190527 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:33.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.207+0000 7fee79ab0640 1 --2- 172.21.15.67:0/4278190527 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fee7415e280 0x7fee740b5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:33.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.208+0000 7fee78aae640 1 -- 172.21.15.67:0/4278190527 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fee7415e280 msgr2=0x7fee740b5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:33.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.208+0000 7fee78aae640 1 --2- 172.21.15.67:0/4278190527 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fee7415e280 0x7fee740b5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.208+0000 7fee78aae640 1 -- 172.21.15.67:0/4278190527 >> v1:172.21.15.72:6789/0 conn(0x7fee7415dea0 legacy=0x7fee740d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:33.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.208+0000 7fee78aae640 1 -- 172.21.15.67:0/4278190527 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee740d2af0 con 0x7fee740d3ae0 2024-03-24T00:49:33.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.208+0000 7fee78aae640 1 --2- 172.21.15.67:0/4278190527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee740d3ae0 0x7fee740b5a60 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7fee640098a0 tx=0x7fee6402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=893ec8d24cebdb06 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:33.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.208+0000 7fee6bfff640 1 -- 172.21.15.67:0/4278190527 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee64035020 con 0x7fee740d3ae0 2024-03-24T00:49:33.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.208+0000 7fee6bfff640 1 -- 172.21.15.67:0/4278190527 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee64031640 con 0x7fee740d3ae0 2024-03-24T00:49:33.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.208+0000 7fee6bfff640 1 -- 172.21.15.67:0/4278190527 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee6402d040 con 0x7fee740d3ae0 2024-03-24T00:49:33.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.209+0000 7fee7b513640 1 -- 172.21.15.67:0/4278190527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee740d3ae0 msgr2=0x7fee740b5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:33.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.209+0000 7fee7b513640 1 --2- 172.21.15.67:0/4278190527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee740d3ae0 0x7fee740b5a60 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7fee640098a0 tx=0x7fee6402b8f0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.209+0000 7fee7b513640 1 -- 172.21.15.67:0/4278190527 shutdown_connections 2024-03-24T00:49:33.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.209+0000 7fee7b513640 1 --2- 172.21.15.67:0/4278190527 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fee7415e280 0x7fee740b5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.209+0000 7fee7b513640 1 --2- 172.21.15.67:0/4278190527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee740d3ae0 0x7fee740b5a60 unknown :-1 s=CLOSED pgs=1874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.209+0000 7fee7b513640 1 -- 172.21.15.67:0/4278190527 >> 172.21.15.67:0/4278190527 conn(0x7fee74072da0 msgr2=0x7fee740d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:33.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.209+0000 7fee7b513640 1 -- 172.21.15.67:0/4278190527 shutdown_connections 2024-03-24T00:49:33.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.209+0000 7fee7b513640 1 -- 172.21.15.67:0/4278190527 wait complete. 2024-03-24T00:49:33.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.211+0000 7fee7b513640 1 Processor -- start 2024-03-24T00:49:33.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.211+0000 7fee7b513640 1 -- start start 2024-03-24T00:49:33.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.212+0000 7fee7b513640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fee740d3ae0 0x7fee740c9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:33.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.212+0000 7fee7b513640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee7415dea0 0x7fee74159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:33.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.212+0000 7fee7b513640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fee7415e280 0x7fee7414ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:33.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.212+0000 7fee7b513640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee740b7480 con 0x7fee7415dea0 2024-03-24T00:49:33.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.212+0000 7fee7b513640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee740b7300 con 0x7fee740d3ae0 2024-03-24T00:49:33.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.212+0000 7fee7b513640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee740b7600 con 0x7fee7415e280 2024-03-24T00:49:33.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.212+0000 7fee78aae640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee7415dea0 0x7fee74159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:33.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.212+0000 7fee78aae640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee7415dea0 0x7fee74159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38648/0 (socket says 172.21.15.67:38648) 2024-03-24T00:49:33.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.212+0000 7fee78aae640 1 -- 172.21.15.67:0/1566580521 learned_addr learned my addr 172.21.15.67:0/1566580521 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:33.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.213+0000 7fee78aae640 1 -- 172.21.15.67:0/1566580521 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fee7415e280 msgr2=0x7fee7414ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:33.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.213+0000 7fee79ab0640 1 --2- 172.21.15.67:0/1566580521 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fee7415e280 0x7fee7414ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:33.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.213+0000 7fee792af640 1 --2- 172.21.15.67:0/1566580521 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fee740d3ae0 0x7fee740c9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:33.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.213+0000 7fee78aae640 1 --2- 172.21.15.67:0/1566580521 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fee7415e280 0x7fee7414ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.213+0000 7fee78aae640 1 -- 172.21.15.67:0/1566580521 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fee740d3ae0 msgr2=0x7fee740c9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:33.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.213+0000 7fee78aae640 1 --2- 172.21.15.67:0/1566580521 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fee740d3ae0 0x7fee740c9cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.213+0000 7fee78aae640 1 -- 172.21.15.67:0/1566580521 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee74158f10 con 0x7fee7415dea0 2024-03-24T00:49:33.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.213+0000 7fee792af640 1 --2- 172.21.15.67:0/1566580521 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fee740d3ae0 0x7fee740c9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:33.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.213+0000 7fee78aae640 1 --2- 172.21.15.67:0/1566580521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee7415dea0 0x7fee74159fb0 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7fee640099c0 tx=0x7fee640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:33.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.213+0000 7fee69ffb640 1 -- 172.21.15.67:0/1566580521 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee64040070 con 0x7fee7415dea0 2024-03-24T00:49:33.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.213+0000 7fee69ffb640 1 -- 172.21.15.67:0/1566580521 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee6403e070 con 0x7fee7415dea0 2024-03-24T00:49:33.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.213+0000 7fee7b513640 1 -- 172.21.15.67:0/1566580521 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee741591f0 con 0x7fee7415dea0 2024-03-24T00:49:33.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.213+0000 7fee69ffb640 1 -- 172.21.15.67:0/1566580521 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee64035040 con 0x7fee7415dea0 2024-03-24T00:49:33.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.213+0000 7fee7b513640 1 -- 172.21.15.67:0/1566580521 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee741597b0 con 0x7fee7415dea0 2024-03-24T00:49:33.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.214+0000 7fee7b513640 1 -- 172.21.15.67:0/1566580521 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee7404f400 con 0x7fee7415dea0 2024-03-24T00:49:33.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.215+0000 7fee69ffb640 1 -- 172.21.15.67:0/1566580521 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fee64038020 con 0x7fee7415dea0 2024-03-24T00:49:33.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.215+0000 7fee69ffb640 1 --2- 172.21.15.67:0/1566580521 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fee5806bbd0 0x7fee5806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:33.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.215+0000 7fee69ffb640 1 -- 172.21.15.67:0/1566580521 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6273+0+0 (secure 0 0 0) 0x7fee640ac600 con 0x7fee7415dea0 2024-03-24T00:49:33.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.215+0000 7fee792af640 1 --2- 172.21.15.67:0/1566580521 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fee5806bbd0 0x7fee5806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:33.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.216+0000 7fee792af640 1 --2- 172.21.15.67:0/1566580521 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fee5806bbd0 0x7fee5806e090 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fee70001680 tx=0x7fee70009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:33.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.220+0000 7fee69ffb640 1 -- 172.21.15.67:0/1566580521 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fee6407aeb0 con 0x7fee7415dea0 2024-03-24T00:49:33.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.401+0000 7fee7b513640 1 -- 172.21.15.67:0/1566580521 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fee740b3270 con 0x7fee7415dea0 2024-03-24T00:49:33.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.402+0000 7fee69ffb640 1 -- 172.21.15.67:0/1566580521 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v122) v1 ==== 100+0+11 (secure 0 0 0) 0x7fee6407a7b0 con 0x7fee7415dea0 2024-03-24T00:49:33.403 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-03-24T00:49:33.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.404+0000 7fee7b513640 1 -- 172.21.15.67:0/1566580521 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fee5806bbd0 msgr2=0x7fee5806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:33.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.404+0000 7fee7b513640 1 --2- 172.21.15.67:0/1566580521 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fee5806bbd0 0x7fee5806e090 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fee70001680 tx=0x7fee70009040 comp rx=0 tx=0).stop 2024-03-24T00:49:33.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.404+0000 7fee7b513640 1 -- 172.21.15.67:0/1566580521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee7415dea0 msgr2=0x7fee74159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:33.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.404+0000 7fee7b513640 1 --2- 172.21.15.67:0/1566580521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee7415dea0 0x7fee74159fb0 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7fee640099c0 tx=0x7fee640046c0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.404+0000 7fee7b513640 1 -- 172.21.15.67:0/1566580521 shutdown_connections 2024-03-24T00:49:33.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.404+0000 7fee7b513640 1 --2- 172.21.15.67:0/1566580521 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fee5806bbd0 0x7fee5806e090 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.404+0000 7fee7b513640 1 --2- 172.21.15.67:0/1566580521 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fee7415e280 0x7fee7414ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.404+0000 7fee7b513640 1 --2- 172.21.15.67:0/1566580521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee7415dea0 0x7fee74159fb0 unknown :-1 s=CLOSED pgs=1875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.404+0000 7fee7b513640 1 --2- 172.21.15.67:0/1566580521 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fee740d3ae0 0x7fee740c9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.404+0000 7fee7b513640 1 -- 172.21.15.67:0/1566580521 >> 172.21.15.67:0/1566580521 conn(0x7fee74072da0 msgr2=0x7fee740b42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:33.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.404+0000 7fee7b513640 1 -- 172.21.15.67:0/1566580521 shutdown_connections 2024-03-24T00:49:33.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.404+0000 7fee7b513640 1 -- 172.21.15.67:0/1566580521 wait complete. 2024-03-24T00:49:33.416 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T00:49:33.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.518+0000 7f674d587640 1 Processor -- start 2024-03-24T00:49:33.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.518+0000 7f674d587640 1 -- start start 2024-03-24T00:49:33.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.519+0000 7f674d587640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f674815c010 0x7f67480d0090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:33.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.519+0000 7f674d587640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f67480d05d0 0x7f67480cc770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:33.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.519+0000 7f674d587640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67480cd430 con 0x7f67480cccb0 2024-03-24T00:49:33.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.519+0000 7f674d587640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67480cfbe0 con 0x7f674815c010 2024-03-24T00:49:33.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.519+0000 7f674d587640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67480b2020 con 0x7f67480d05d0 2024-03-24T00:49:33.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.519+0000 7f6746ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f674815c010 0x7f67480d0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:33.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.519+0000 7f67477fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f67480cccb0 0x7f67480ce200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56462/0 (socket says 172.21.15.67:56462) 2024-03-24T00:49:33.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.519+0000 7f67477fe640 1 -- 172.21.15.67:0/6763297 learned_addr learned my addr 172.21.15.67:0/6763297 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:49:33.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.519+0000 7f673ffff640 1 --2- 172.21.15.67:0/6763297 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f67480d05d0 0x7f67480cc770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:33.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.519+0000 7f67467fc640 1 -- 172.21.15.67:0/6763297 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 275284226 0 0) 0x7f67480cd430 con 0x7f67480cccb0 2024-03-24T00:49:33.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.519+0000 7f67467fc640 1 -- 172.21.15.67:0/6763297 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6734003520 con 0x7f67480cccb0 2024-03-24T00:49:33.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.519+0000 7f6746ffd640 1 -- 172.21.15.67:0/6763297 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f67480d05d0 msgr2=0x7f67480cc770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:33.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.519+0000 7f6746ffd640 1 --2- 172.21.15.67:0/6763297 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f67480d05d0 0x7f67480cc770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.519+0000 7f6746ffd640 1 -- 172.21.15.67:0/6763297 >> v1:172.21.15.67:6789/0 conn(0x7f67480cccb0 legacy=0x7f67480ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:33.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.519+0000 7f6746ffd640 1 -- 172.21.15.67:0/6763297 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67480cd710 con 0x7f674815c010 2024-03-24T00:49:33.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.520+0000 7f6746ffd640 1 --2- 172.21.15.67:0/6763297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f674815c010 0x7f67480d0090 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f67380097a0 tx=0x7f673802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e006e5a3c984c3b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:33.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.520+0000 7f67467fc640 1 -- 172.21.15.67:0/6763297 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6738035020 con 0x7f674815c010 2024-03-24T00:49:33.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.520+0000 7f67467fc640 1 -- 172.21.15.67:0/6763297 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6738031830 con 0x7f674815c010 2024-03-24T00:49:33.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.520+0000 7f67467fc640 1 -- 172.21.15.67:0/6763297 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6738031c00 con 0x7f674815c010 2024-03-24T00:49:33.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.521+0000 7f674d587640 1 -- 172.21.15.67:0/6763297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f674815c010 msgr2=0x7f67480d0090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:33.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.521+0000 7f674d587640 1 --2- 172.21.15.67:0/6763297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f674815c010 0x7f67480d0090 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f67380097a0 tx=0x7f673802b8d0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.521+0000 7f674d587640 1 -- 172.21.15.67:0/6763297 shutdown_connections 2024-03-24T00:49:33.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.521+0000 7f674d587640 1 --2- 172.21.15.67:0/6763297 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f67480d05d0 0x7f67480cc770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.521+0000 7f674d587640 1 --2- 172.21.15.67:0/6763297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f674815c010 0x7f67480d0090 unknown :-1 s=CLOSED pgs=1877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.521+0000 7f674d587640 1 -- 172.21.15.67:0/6763297 >> 172.21.15.67:0/6763297 conn(0x7f674809aea0 msgr2=0x7f674808b180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:33.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.521+0000 7f674d587640 1 -- 172.21.15.67:0/6763297 shutdown_connections 2024-03-24T00:49:33.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.521+0000 7f674d587640 1 -- 172.21.15.67:0/6763297 wait complete. 2024-03-24T00:49:33.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.523+0000 7f674d587640 1 Processor -- start 2024-03-24T00:49:33.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.523+0000 7f674d587640 1 -- start start 2024-03-24T00:49:33.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.524+0000 7f674d587640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f67480cccb0 0x7f6748150b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:33.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.524+0000 7f674d587640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f67480d05d0 0x7f67480c7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:33.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.524+0000 7f674d587640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f674815c010 0x7f67481e6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:33.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.524+0000 7f674d587640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67480d0a10 con 0x7f674815c010 2024-03-24T00:49:33.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.524+0000 7f674d587640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67480b2300 con 0x7f67480d05d0 2024-03-24T00:49:33.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.524+0000 7f674d587640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67480cd140 con 0x7f67480cccb0 2024-03-24T00:49:33.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.524+0000 7f67477fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f674815c010 0x7f67481e6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:33.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.524+0000 7f67477fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f674815c010 0x7f67481e6d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48186/0 (socket says 172.21.15.67:48186) 2024-03-24T00:49:33.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.524+0000 7f67477fe640 1 -- 172.21.15.67:0/3072531758 learned_addr learned my addr 172.21.15.67:0/3072531758 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:33.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.524+0000 7f673ffff640 1 --2- 172.21.15.67:0/3072531758 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f67480d05d0 0x7f67480c7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:33.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.524+0000 7f6746ffd640 1 --2- 172.21.15.67:0/3072531758 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f67480cccb0 0x7f6748150b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:33.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.525+0000 7f67477fe640 1 -- 172.21.15.67:0/3072531758 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f67480cccb0 msgr2=0x7f6748150b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:33.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.525+0000 7f67477fe640 1 --2- 172.21.15.67:0/3072531758 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f67480cccb0 0x7f6748150b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.525+0000 7f67477fe640 1 -- 172.21.15.67:0/3072531758 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f67480d05d0 msgr2=0x7f67480c7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:33.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.525+0000 7f67477fe640 1 --2- 172.21.15.67:0/3072531758 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f67480d05d0 0x7f67480c7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.525+0000 7f67477fe640 1 -- 172.21.15.67:0/3072531758 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67481510c0 con 0x7f674815c010 2024-03-24T00:49:33.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.525+0000 7f67477fe640 1 --2- 172.21.15.67:0/3072531758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f674815c010 0x7f67481e6d50 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f674000cad0 tx=0x7f674000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:33.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.525+0000 7f673f7fe640 1 -- 172.21.15.67:0/3072531758 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6740017070 con 0x7f674815c010 2024-03-24T00:49:33.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.525+0000 7f673f7fe640 1 -- 172.21.15.67:0/3072531758 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f674000ac30 con 0x7f674815c010 2024-03-24T00:49:33.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.525+0000 7f674d587640 1 -- 172.21.15.67:0/3072531758 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67481e5cb0 con 0x7f674815c010 2024-03-24T00:49:33.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.525+0000 7f673f7fe640 1 -- 172.21.15.67:0/3072531758 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6740012400 con 0x7f674815c010 2024-03-24T00:49:33.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.525+0000 7f674d587640 1 -- 172.21.15.67:0/3072531758 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67481e6270 con 0x7f674815c010 2024-03-24T00:49:33.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.526+0000 7f674d587640 1 -- 172.21.15.67:0/3072531758 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f674804f400 con 0x7f674815c010 2024-03-24T00:49:33.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.527+0000 7f673f7fe640 1 -- 172.21.15.67:0/3072531758 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f674001a070 con 0x7f674815c010 2024-03-24T00:49:33.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.527+0000 7f673f7fe640 1 --2- 172.21.15.67:0/3072531758 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f672806bbd0 0x7f672806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:33.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.527+0000 7f673f7fe640 1 -- 172.21.15.67:0/3072531758 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f674008a9a0 con 0x7f674815c010 2024-03-24T00:49:33.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.527+0000 7f6746ffd640 1 --2- 172.21.15.67:0/3072531758 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f672806bbd0 0x7f672806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:33.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.528+0000 7f6746ffd640 1 --2- 172.21.15.67:0/3072531758 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f672806bbd0 0x7f672806e090 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f6738036000 tx=0x7f6738008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:33.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.531+0000 7f673f7fe640 1 -- 172.21.15.67:0/3072531758 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6740058c00 con 0x7f674815c010 2024-03-24T00:49:33.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.726+0000 7f674d587640 1 -- 172.21.15.67:0/3072531758 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f67480b18b0 con 0x7f674815c010 2024-03-24T00:49:33.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.726+0000 7f673f7fe640 1 -- 172.21.15.67:0/3072531758 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 4 entries v122) v1 ==== 80+0+242 (secure 0 0 0) 0x7f674005cca0 con 0x7f674815c010 2024-03-24T00:49:33.728 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6835/3833808901 2024-03-25T00:49:23.316030+0000 2024-03-24T00:49:33.728 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6833/3833808901 2024-03-25T00:49:23.316030+0000 2024-03-24T00:49:33.728 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6835/312063805 2024-03-25T00:49:12.456364+0000 2024-03-24T00:49:33.728 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6833/312063805 2024-03-25T00:49:12.456364+0000 2024-03-24T00:49:33.728 INFO:teuthology.orchestra.run.smithi067.stderr:listed 4 entries 2024-03-24T00:49:33.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.728+0000 7f674d587640 1 -- 172.21.15.67:0/3072531758 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f672806bbd0 msgr2=0x7f672806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:33.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.728+0000 7f674d587640 1 --2- 172.21.15.67:0/3072531758 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f672806bbd0 0x7f672806e090 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f6738036000 tx=0x7f6738008030 comp rx=0 tx=0).stop 2024-03-24T00:49:33.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.728+0000 7f674d587640 1 -- 172.21.15.67:0/3072531758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f674815c010 msgr2=0x7f67481e6d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:33.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.728+0000 7f674d587640 1 --2- 172.21.15.67:0/3072531758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f674815c010 0x7f67481e6d50 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f674000cad0 tx=0x7f674000cf90 comp rx=0 tx=0).stop 2024-03-24T00:49:33.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.728+0000 7f674d587640 1 -- 172.21.15.67:0/3072531758 shutdown_connections 2024-03-24T00:49:33.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.728+0000 7f674d587640 1 --2- 172.21.15.67:0/3072531758 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f672806bbd0 0x7f672806e090 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.728+0000 7f674d587640 1 --2- 172.21.15.67:0/3072531758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f674815c010 0x7f67481e6d50 unknown :-1 s=CLOSED pgs=1878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.728+0000 7f674d587640 1 --2- 172.21.15.67:0/3072531758 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f67480d05d0 0x7f67480c7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.728+0000 7f674d587640 1 --2- 172.21.15.67:0/3072531758 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f67480cccb0 0x7f6748150b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:33.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.728+0000 7f674d587640 1 -- 172.21.15.67:0/3072531758 >> 172.21.15.67:0/3072531758 conn(0x7f674809aea0 msgr2=0x7f67480b3830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:33.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.728+0000 7f674d587640 1 -- 172.21.15.67:0/3072531758 shutdown_connections 2024-03-24T00:49:33.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:33.728+0000 7f674d587640 1 -- 172.21.15.67:0/3072531758 wait complete. 2024-03-24T00:49:33.740 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-24T00:49:33.740 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:33.741 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-24T00:49:33.770 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-03-24T00:49:33.770 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:33.771 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-24T00:49:33.826 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:49:33.826 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:49:33.844 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17590 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248808k,nr_inodes=812202,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:49:33.844 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:49:33.845 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:49:33.880 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056375,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15771 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T00:49:33.880 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:49:33.880 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:49:33.902 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17590 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248808k,nr_inodes=812202,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:49:33.902 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:49:34.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.022+0000 7fd252b59640 1 Processor -- start 2024-03-24T00:49:34.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.022+0000 7fd252b59640 1 -- start start 2024-03-24T00:49:34.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.023+0000 7fd252b59640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd24c15dec0 0x7fd24c0ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:34.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.023+0000 7fd252b59640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd24c0d0ef0 0x7fd24c162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:34.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.023+0000 7fd252b59640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd24c0cf640 con 0x7fd24c15e2a0 2024-03-24T00:49:34.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.023+0000 7fd252b59640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd24c0cee30 con 0x7fd24c15dec0 2024-03-24T00:49:34.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.023+0000 7fd252b59640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd24c0d2120 con 0x7fd24c0d0ef0 2024-03-24T00:49:34.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.023+0000 7fd2508f5640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd24c15dec0 0x7fd24c0ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:34.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.023+0000 7fd2508f5640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd24c15dec0 0x7fd24c0ce630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48194/0 (socket says 172.21.15.67:48194) 2024-03-24T00:49:34.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.023+0000 7fd2508f5640 1 -- 172.21.15.67:0/2518965068 learned_addr learned my addr 172.21.15.67:0/2518965068 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:34.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.023+0000 7fd2508f5640 1 -- 172.21.15.67:0/2518965068 >> v2:172.21.15.72:3300/0 conn(0x7fd24c0d0ef0 msgr2=0x7fd24c162630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:34.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.023+0000 7fd2508f5640 1 --2- 172.21.15.67:0/2518965068 >> v2:172.21.15.72:3300/0 conn(0x7fd24c0d0ef0 0x7fd24c162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.023+0000 7fd2508f5640 1 -- 172.21.15.67:0/2518965068 >> v1:172.21.15.72:6789/0 conn(0x7fd24c15e2a0 legacy=0x7fd24c0d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:34.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.023+0000 7fd2508f5640 1 -- 172.21.15.67:0/2518965068 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd24c0cf080 con 0x7fd24c15dec0 2024-03-24T00:49:34.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.024+0000 7fd2508f5640 1 --2- 172.21.15.67:0/2518965068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd24c15dec0 0x7fd24c0ce630 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7fd24c0cae20 tx=0x7fd23c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0b9be76022b28c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:34.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.024+0000 7fd24b7fe640 1 -- 172.21.15.67:0/2518965068 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23c02e070 con 0x7fd24c15dec0 2024-03-24T00:49:34.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.024+0000 7fd24b7fe640 1 -- 172.21.15.67:0/2518965068 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd23c037070 con 0x7fd24c15dec0 2024-03-24T00:49:34.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.024+0000 7fd24b7fe640 1 -- 172.21.15.67:0/2518965068 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23c031a00 con 0x7fd24c15dec0 2024-03-24T00:49:34.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.024+0000 7fd252b59640 1 -- 172.21.15.67:0/2518965068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd24c15dec0 msgr2=0x7fd24c0ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:34.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.024+0000 7fd252b59640 1 --2- 172.21.15.67:0/2518965068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd24c15dec0 0x7fd24c0ce630 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7fd24c0cae20 tx=0x7fd23c02b880 comp rx=0 tx=0).stop 2024-03-24T00:49:34.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.025+0000 7fd252b59640 1 -- 172.21.15.67:0/2518965068 shutdown_connections 2024-03-24T00:49:34.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.025+0000 7fd252b59640 1 --2- 172.21.15.67:0/2518965068 >> v2:172.21.15.72:3300/0 conn(0x7fd24c0d0ef0 0x7fd24c162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.025+0000 7fd252b59640 1 --2- 172.21.15.67:0/2518965068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd24c15dec0 0x7fd24c0ce630 unknown :-1 s=CLOSED pgs=1879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.025+0000 7fd252b59640 1 -- 172.21.15.67:0/2518965068 >> 172.21.15.67:0/2518965068 conn(0x7fd24c06c810 msgr2=0x7fd24c06cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:34.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.025+0000 7fd252b59640 1 -- 172.21.15.67:0/2518965068 shutdown_connections 2024-03-24T00:49:34.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.025+0000 7fd252b59640 1 -- 172.21.15.67:0/2518965068 wait complete. 2024-03-24T00:49:34.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.026+0000 7fd252b59640 1 Processor -- start 2024-03-24T00:49:34.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.026+0000 7fd252b59640 1 -- start start 2024-03-24T00:49:34.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.027+0000 7fd252b59640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd24c0d0ef0 0x7fd24c0b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:34.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.027+0000 7fd252b59640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd24c15dec0 0x7fd24c0cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:34.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.027+0000 7fd252b59640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd24c15e2a0 0x7fd24c0cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:34.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.027+0000 7fd252b59640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd24c0d2730 con 0x7fd24c0d0ef0 2024-03-24T00:49:34.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.027+0000 7fd252b59640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd24c0d25b0 con 0x7fd24c15e2a0 2024-03-24T00:49:34.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.027+0000 7fd252b59640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd24c0d28b0 con 0x7fd24c15dec0 2024-03-24T00:49:34.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.027+0000 7fd2508f5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd24c0d0ef0 0x7fd24c0b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:34.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.027+0000 7fd2508f5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd24c0d0ef0 0x7fd24c0b61b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48198/0 (socket says 172.21.15.67:48198) 2024-03-24T00:49:34.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.027+0000 7fd2508f5640 1 -- 172.21.15.67:0/3778134999 learned_addr learned my addr 172.21.15.67:0/3778134999 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:34.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.027+0000 7fd2510f6640 1 --2- 172.21.15.67:0/3778134999 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd24c15e2a0 0x7fd24c0cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:34.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.027+0000 7fd24bfff640 1 --2- 172.21.15.67:0/3778134999 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd24c15dec0 0x7fd24c0cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:34.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.027+0000 7fd2508f5640 1 -- 172.21.15.67:0/3778134999 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd24c15dec0 msgr2=0x7fd24c0cb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:34.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.027+0000 7fd2508f5640 1 --2- 172.21.15.67:0/3778134999 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd24c15dec0 0x7fd24c0cb1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.027+0000 7fd2508f5640 1 -- 172.21.15.67:0/3778134999 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd24c15e2a0 msgr2=0x7fd24c0cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:34.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.027+0000 7fd2508f5640 1 --2- 172.21.15.67:0/3778134999 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd24c15e2a0 0x7fd24c0cb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.027+0000 7fd2508f5640 1 -- 172.21.15.67:0/3778134999 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd24c1584e0 con 0x7fd24c0d0ef0 2024-03-24T00:49:34.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.027+0000 7fd2510f6640 1 --2- 172.21.15.67:0/3778134999 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd24c15e2a0 0x7fd24c0cb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:34.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.027+0000 7fd24bfff640 1 --2- 172.21.15.67:0/3778134999 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd24c15dec0 0x7fd24c0cb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:34.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.027+0000 7fd2508f5640 1 --2- 172.21.15.67:0/3778134999 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd24c0d0ef0 0x7fd24c0b61b0 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7fd23c02b8c0 tx=0x7fd23c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:34.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.027+0000 7fd2497fa640 1 -- 172.21.15.67:0/3778134999 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23c032880 con 0x7fd24c0d0ef0 2024-03-24T00:49:34.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.027+0000 7fd2497fa640 1 -- 172.21.15.67:0/3778134999 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd23c032a20 con 0x7fd24c0d0ef0 2024-03-24T00:49:34.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.027+0000 7fd252b59640 1 -- 172.21.15.67:0/3778134999 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd24c1587c0 con 0x7fd24c0d0ef0 2024-03-24T00:49:34.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.028+0000 7fd2497fa640 1 -- 172.21.15.67:0/3778134999 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23c032e10 con 0x7fd24c0d0ef0 2024-03-24T00:49:34.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.028+0000 7fd252b59640 1 -- 172.21.15.67:0/3778134999 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd24c0cd1a0 con 0x7fd24c0d0ef0 2024-03-24T00:49:34.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.028+0000 7fd252b59640 1 -- 172.21.15.67:0/3778134999 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd24c04f400 con 0x7fd24c0d0ef0 2024-03-24T00:49:34.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.029+0000 7fd2497fa640 1 -- 172.21.15.67:0/3778134999 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd23c0453f0 con 0x7fd24c0d0ef0 2024-03-24T00:49:34.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.029+0000 7fd2497fa640 1 --2- 172.21.15.67:0/3778134999 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd23806bbd0 0x7fd23806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:34.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.029+0000 7fd2497fa640 1 -- 172.21.15.67:0/3778134999 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6273+0+0 (secure 0 0 0) 0x7fd23c0acd80 con 0x7fd24c0d0ef0 2024-03-24T00:49:34.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.030+0000 7fd24bfff640 1 --2- 172.21.15.67:0/3778134999 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd23806bbd0 0x7fd23806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:34.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.030+0000 7fd24bfff640 1 --2- 172.21.15.67:0/3778134999 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd23806bbd0 0x7fd23806e090 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fd24c15ee80 tx=0x7fd240008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:34.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.035+0000 7fd2497fa640 1 -- 172.21.15.67:0/3778134999 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd23c040070 con 0x7fd24c0d0ef0 2024-03-24T00:49:34.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.253+0000 7fd252b59640 1 -- 172.21.15.67:0/3778134999 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd24c0d1aa0 con 0x7fd24c0d0ef0 2024-03-24T00:49:34.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.253+0000 7fd2497fa640 1 -- 172.21.15.67:0/3778134999 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+4975 (secure 0 0 0) 0x7fd24c0cf640 con 0x7fd24c0d0ef0 2024-03-24T00:49:34.254 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:34.254 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":99,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7298,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/839277571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":839277571},{"type":"v1","addr":"172.21.15.72:6836","nonce":839277571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":8224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2052380377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2052380377},{"type":"v1","addr":"172.21.15.67:6834","nonce":2052380377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":8239,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2196618204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2196618204},{"type":"v1","addr":"172.21.15.72:6837","nonce":2196618204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:48:54.351575+0000","modified":"2024-03-24T00:49:28.351081+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":122,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9666},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9666":{"gid":9666,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6835/693801918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":693801918},{"type":"v1","addr":"172.21.15.67:6835","nonce":693801918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9666,"qdb_cluster":[9666]},"id":8}]} 2024-03-24T00:49:34.255 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 99 2024-03-24T00:49:34.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.255+0000 7fd252b59640 1 -- 172.21.15.67:0/3778134999 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd23806bbd0 msgr2=0x7fd23806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:34.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.255+0000 7fd252b59640 1 --2- 172.21.15.67:0/3778134999 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd23806bbd0 0x7fd23806e090 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fd24c15ee80 tx=0x7fd240008040 comp rx=0 tx=0).stop 2024-03-24T00:49:34.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.255+0000 7fd252b59640 1 -- 172.21.15.67:0/3778134999 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd24c0d0ef0 msgr2=0x7fd24c0b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:34.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.255+0000 7fd252b59640 1 --2- 172.21.15.67:0/3778134999 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd24c0d0ef0 0x7fd24c0b61b0 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7fd23c02b8c0 tx=0x7fd23c0046c0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.255+0000 7fd252b59640 1 -- 172.21.15.67:0/3778134999 shutdown_connections 2024-03-24T00:49:34.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.255+0000 7fd252b59640 1 --2- 172.21.15.67:0/3778134999 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fd23806bbd0 0x7fd23806e090 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.255+0000 7fd252b59640 1 --2- 172.21.15.67:0/3778134999 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd24c15e2a0 0x7fd24c0cb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.255+0000 7fd252b59640 1 --2- 172.21.15.67:0/3778134999 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd24c15dec0 0x7fd24c0cb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.255+0000 7fd252b59640 1 --2- 172.21.15.67:0/3778134999 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd24c0d0ef0 0x7fd24c0b61b0 unknown :-1 s=CLOSED pgs=1880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.255+0000 7fd252b59640 1 -- 172.21.15.67:0/3778134999 >> 172.21.15.67:0/3778134999 conn(0x7fd24c06c810 msgr2=0x7fd24c0ce320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:34.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.255+0000 7fd252b59640 1 -- 172.21.15.67:0/3778134999 shutdown_connections 2024-03-24T00:49:34.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.255+0000 7fd252b59640 1 -- 172.21.15.67:0/3778134999 wait complete. 2024-03-24T00:49:34.267 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:49:34.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.392+0000 7f2d2ca12640 1 Processor -- start 2024-03-24T00:49:34.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.392+0000 7f2d2ca12640 1 -- start start 2024-03-24T00:49:34.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.393+0000 7f2d2ca12640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2d2815dec0 0x7f2d280ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:34.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.393+0000 7f2d2ca12640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2d2815e2a0 0x7f2d28162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:34.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.393+0000 7f2d2ca12640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d280cf640 con 0x7f2d280d07e0 2024-03-24T00:49:34.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.393+0000 7f2d2ca12640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d280cee30 con 0x7f2d2815e2a0 2024-03-24T00:49:34.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.393+0000 7f2d2ca12640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d280d2120 con 0x7f2d2815dec0 2024-03-24T00:49:34.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.393+0000 7f2d25d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2d2815e2a0 0x7f2d28162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:34.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.393+0000 7f2d25d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2d2815e2a0 0x7f2d28162630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48200/0 (socket says 172.21.15.67:48200) 2024-03-24T00:49:34.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.393+0000 7f2d25d9b640 1 -- 172.21.15.67:0/2937420882 learned_addr learned my addr 172.21.15.67:0/2937420882 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:34.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.393+0000 7f2d2659c640 1 --2- 172.21.15.67:0/2937420882 >> v2:172.21.15.72:3300/0 conn(0x7f2d2815dec0 0x7f2d280ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:34.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.394+0000 7f2d25d9b640 1 -- 172.21.15.67:0/2937420882 >> v2:172.21.15.72:3300/0 conn(0x7f2d2815dec0 msgr2=0x7f2d280ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:34.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.394+0000 7f2d25d9b640 1 --2- 172.21.15.67:0/2937420882 >> v2:172.21.15.72:3300/0 conn(0x7f2d2815dec0 0x7f2d280ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.394+0000 7f2d25d9b640 1 -- 172.21.15.67:0/2937420882 >> v1:172.21.15.72:6789/0 conn(0x7f2d280d07e0 legacy=0x7f2d280d0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:34.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.394+0000 7f2d25d9b640 1 -- 172.21.15.67:0/2937420882 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d280cf080 con 0x7f2d2815e2a0 2024-03-24T00:49:34.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.394+0000 7f2d25d9b640 1 --2- 172.21.15.67:0/2937420882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d2815e2a0 0x7f2d28162630 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f2d280d14a0 tx=0x7f2d1802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3a9e4a51f1ff04c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:34.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.395+0000 7f2d2559a640 1 -- 172.21.15.67:0/2937420882 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d18035020 con 0x7f2d2815e2a0 2024-03-24T00:49:34.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.395+0000 7f2d2559a640 1 -- 172.21.15.67:0/2937420882 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d18031870 con 0x7f2d2815e2a0 2024-03-24T00:49:34.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.395+0000 7f2d2559a640 1 -- 172.21.15.67:0/2937420882 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d18031c40 con 0x7f2d2815e2a0 2024-03-24T00:49:34.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.395+0000 7f2d2ca12640 1 -- 172.21.15.67:0/2937420882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d2815e2a0 msgr2=0x7f2d28162630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:34.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.395+0000 7f2d2ca12640 1 --2- 172.21.15.67:0/2937420882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d2815e2a0 0x7f2d28162630 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f2d280d14a0 tx=0x7f2d1802b910 comp rx=0 tx=0).stop 2024-03-24T00:49:34.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.395+0000 7f2d2ca12640 1 -- 172.21.15.67:0/2937420882 shutdown_connections 2024-03-24T00:49:34.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.395+0000 7f2d2ca12640 1 --2- 172.21.15.67:0/2937420882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d2815e2a0 0x7f2d28162630 unknown :-1 s=CLOSED pgs=1881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.395+0000 7f2d2ca12640 1 --2- 172.21.15.67:0/2937420882 >> v2:172.21.15.72:3300/0 conn(0x7f2d2815dec0 0x7f2d280ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.395+0000 7f2d2ca12640 1 -- 172.21.15.67:0/2937420882 >> 172.21.15.67:0/2937420882 conn(0x7f2d2806c810 msgr2=0x7f2d2806cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:34.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.395+0000 7f2d2ca12640 1 -- 172.21.15.67:0/2937420882 shutdown_connections 2024-03-24T00:49:34.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.395+0000 7f2d2ca12640 1 -- 172.21.15.67:0/2937420882 wait complete. 2024-03-24T00:49:34.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.397+0000 7f2d2ca12640 1 Processor -- start 2024-03-24T00:49:34.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.397+0000 7f2d2ca12640 1 -- start start 2024-03-24T00:49:34.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.397+0000 7f2d2ca12640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2d280d07e0 0x7f2d280b3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:34.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.397+0000 7f2d2ca12640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d2815dec0 0x7f2d28143100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:34.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.397+0000 7f2d2ca12640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d2815e2a0 0x7f2d28143640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:34.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.397+0000 7f2d2ca12640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d280d2730 con 0x7f2d2815dec0 2024-03-24T00:49:34.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.397+0000 7f2d2ca12640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d280d25b0 con 0x7f2d2815e2a0 2024-03-24T00:49:34.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.397+0000 7f2d2ca12640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d280d28b0 con 0x7f2d280d07e0 2024-03-24T00:49:34.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.397+0000 7f2d25d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d2815dec0 0x7f2d28143100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:34.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.397+0000 7f2d25d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d2815dec0 0x7f2d28143100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48216/0 (socket says 172.21.15.67:48216) 2024-03-24T00:49:34.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.397+0000 7f2d25d9b640 1 -- 172.21.15.67:0/3504583462 learned_addr learned my addr 172.21.15.67:0/3504583462 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:34.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.397+0000 7f2d26d9d640 1 --2- 172.21.15.67:0/3504583462 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d2815e2a0 0x7f2d28143640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:34.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.397+0000 7f2d2659c640 1 --2- 172.21.15.67:0/3504583462 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2d280d07e0 0x7f2d280b3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:34.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.398+0000 7f2d25d9b640 1 -- 172.21.15.67:0/3504583462 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2d280d07e0 msgr2=0x7f2d280b3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:34.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.398+0000 7f2d25d9b640 1 --2- 172.21.15.67:0/3504583462 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2d280d07e0 0x7f2d280b3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.398+0000 7f2d25d9b640 1 -- 172.21.15.67:0/3504583462 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d2815e2a0 msgr2=0x7f2d28143640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:34.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.398+0000 7f2d25d9b640 1 --2- 172.21.15.67:0/3504583462 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d2815e2a0 0x7f2d28143640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.398+0000 7f2d25d9b640 1 -- 172.21.15.67:0/3504583462 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d28152a50 con 0x7f2d2815dec0 2024-03-24T00:49:34.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.398+0000 7f2d25d9b640 1 --2- 172.21.15.67:0/3504583462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d2815dec0 0x7f2d28143100 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f2d1802b950 tx=0x7f2d180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:34.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.398+0000 7f2d1effd640 1 -- 172.21.15.67:0/3504583462 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d18040070 con 0x7f2d2815dec0 2024-03-24T00:49:34.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.398+0000 7f2d1effd640 1 -- 172.21.15.67:0/3504583462 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d18030d00 con 0x7f2d2815dec0 2024-03-24T00:49:34.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.398+0000 7f2d2ca12640 1 -- 172.21.15.67:0/3504583462 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d2815a5b0 con 0x7f2d2815dec0 2024-03-24T00:49:34.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.398+0000 7f2d1effd640 1 -- 172.21.15.67:0/3504583462 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d18035040 con 0x7f2d2815dec0 2024-03-24T00:49:34.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.398+0000 7f2d2ca12640 1 -- 172.21.15.67:0/3504583462 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d2815ab70 con 0x7f2d2815dec0 2024-03-24T00:49:34.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.400+0000 7f2d2ca12640 1 -- 172.21.15.67:0/3504583462 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d2804f400 con 0x7f2d2815dec0 2024-03-24T00:49:34.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.400+0000 7f2d1effd640 1 -- 172.21.15.67:0/3504583462 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2d1804b060 con 0x7f2d2815dec0 2024-03-24T00:49:34.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.400+0000 7f2d1effd640 1 --2- 172.21.15.67:0/3504583462 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2d0806bc00 0x7f2d0806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:34.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.400+0000 7f2d1effd640 1 -- 172.21.15.67:0/3504583462 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f2d180ac730 con 0x7f2d2815dec0 2024-03-24T00:49:34.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.400+0000 7f2d2659c640 1 --2- 172.21.15.67:0/3504583462 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2d0806bc00 0x7f2d0806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:34.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.401+0000 7f2d2659c640 1 --2- 172.21.15.67:0/3504583462 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2d0806bc00 0x7f2d0806e0c0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f2d10009b40 tx=0x7f2d10009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:34.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.404+0000 7f2d1effd640 1 -- 172.21.15.67:0/3504583462 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2d28152a50 con 0x7f2d2815dec0 2024-03-24T00:49:34.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.656+0000 7f2d2ca12640 1 -- 172.21.15.67:0/3504583462 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2d280d1aa0 con 0x7f2d2815dec0 2024-03-24T00:49:34.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.657+0000 7f2d1effd640 1 -- 172.21.15.67:0/3504583462 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v122) v1 ==== 74+0+17149 (secure 0 0 0) 0x7f2d1807aa20 con 0x7f2d2815dec0 2024-03-24T00:49:34.657 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:34.657 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":122,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:49:23.316053+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:48:52.622356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-03-24T00:48:53.311452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6835/3833808901":"2024-03-25T00:49:23.316030+0000","172.21.15.67:6833/3833808901":"2024-03-25T00:49:23.316030+0000","172.21.15.67:6835/312063805":"2024-03-25T00:49:12.456364+0000","172.21.15.67:6833/312063805":"2024-03-25T00:49:12.456364+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:49:34.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.659+0000 7f2d2ca12640 1 -- 172.21.15.67:0/3504583462 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2d0806bc00 msgr2=0x7f2d0806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:34.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.659+0000 7f2d2ca12640 1 --2- 172.21.15.67:0/3504583462 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2d0806bc00 0x7f2d0806e0c0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f2d10009b40 tx=0x7f2d10009190 comp rx=0 tx=0).stop 2024-03-24T00:49:34.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.659+0000 7f2d2ca12640 1 -- 172.21.15.67:0/3504583462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d2815dec0 msgr2=0x7f2d28143100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:34.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.659+0000 7f2d2ca12640 1 --2- 172.21.15.67:0/3504583462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d2815dec0 0x7f2d28143100 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f2d1802b950 tx=0x7f2d180046c0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.659+0000 7f2d2ca12640 1 -- 172.21.15.67:0/3504583462 shutdown_connections 2024-03-24T00:49:34.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.659+0000 7f2d2ca12640 1 --2- 172.21.15.67:0/3504583462 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f2d0806bc00 0x7f2d0806e0c0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.659+0000 7f2d2ca12640 1 --2- 172.21.15.67:0/3504583462 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d2815e2a0 0x7f2d28143640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.659+0000 7f2d2ca12640 1 --2- 172.21.15.67:0/3504583462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d2815dec0 0x7f2d28143100 unknown :-1 s=CLOSED pgs=1882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.659+0000 7f2d2ca12640 1 --2- 172.21.15.67:0/3504583462 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f2d280d07e0 0x7f2d280b3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.660+0000 7f2d2ca12640 1 -- 172.21.15.67:0/3504583462 >> 172.21.15.67:0/3504583462 conn(0x7f2d2806c810 msgr2=0x7f2d2815e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:34.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.660+0000 7f2d2ca12640 1 -- 172.21.15.67:0/3504583462 shutdown_connections 2024-03-24T00:49:34.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.660+0000 7f2d2ca12640 1 -- 172.21.15.67:0/3504583462 wait complete. 2024-03-24T00:49:34.674 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:38:39.364950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:48:52.622356+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '119', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:48:53.311452+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '120', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:49:34.674 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:49:34.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.779+0000 7faec44e8640 1 Processor -- start 2024-03-24T00:49:34.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.779+0000 7faec44e8640 1 -- start start 2024-03-24T00:49:34.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.781+0000 7faec44e8640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faebc15e3a0 0x7faebc0b2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:34.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.781+0000 7faec44e8640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7faebc1626d0 0x7faebc0ccaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:34.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.781+0000 7faec44e8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faebc0cd030 0x7faebc0b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:34.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.781+0000 7faec44e8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faebc0afb80 con 0x7faebc0cd030 2024-03-24T00:49:34.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.781+0000 7faec44e8640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faebc15f470 con 0x7faebc1626d0 2024-03-24T00:49:34.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.781+0000 7faec44e8640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faebc15e860 con 0x7faebc15e3a0 2024-03-24T00:49:34.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.781+0000 7faec2a85640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faebc0cd030 0x7faebc0b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:34.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.781+0000 7faec2a85640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faebc0cd030 0x7faebc0b4f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48220/0 (socket says 172.21.15.67:48220) 2024-03-24T00:49:34.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.781+0000 7faec2a85640 1 -- 172.21.15.67:0/2620162268 learned_addr learned my addr 172.21.15.67:0/2620162268 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:34.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.781+0000 7faec2284640 1 --2- 172.21.15.67:0/2620162268 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faebc15e3a0 0x7faebc0b2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:34.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.781+0000 7faec1a83640 1 --2- 172.21.15.67:0/2620162268 >> v2:172.21.15.72:3300/0 conn(0x7faebc1626d0 0x7faebc0ccaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:34.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.781+0000 7faec2a85640 1 -- 172.21.15.67:0/2620162268 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faebc15e3a0 msgr2=0x7faebc0b2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:34.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.781+0000 7faec2a85640 1 --2- 172.21.15.67:0/2620162268 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faebc15e3a0 0x7faebc0b2890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.781+0000 7faec2a85640 1 -- 172.21.15.67:0/2620162268 >> v2:172.21.15.72:3300/0 conn(0x7faebc1626d0 msgr2=0x7faebc0ccaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:34.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.781+0000 7faec2a85640 1 --2- 172.21.15.67:0/2620162268 >> v2:172.21.15.72:3300/0 conn(0x7faebc1626d0 0x7faebc0ccaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.781+0000 7faec2a85640 1 -- 172.21.15.67:0/2620162268 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faebc15eb40 con 0x7faebc0cd030 2024-03-24T00:49:34.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.781+0000 7faec1a83640 1 --2- 172.21.15.67:0/2620162268 >> v2:172.21.15.72:3300/0 conn(0x7faebc1626d0 0x7faebc0ccaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:34.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.782+0000 7faec2a85640 1 --2- 172.21.15.67:0/2620162268 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faebc0cd030 0x7faebc0b4f20 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7faebc0b63e0 tx=0x7faeb402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afe1dc796b1ea939 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:34.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.782+0000 7faec1282640 1 -- 172.21.15.67:0/2620162268 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb402ca70 con 0x7faebc0cd030 2024-03-24T00:49:34.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.782+0000 7faec1282640 1 -- 172.21.15.67:0/2620162268 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeb402cc10 con 0x7faebc0cd030 2024-03-24T00:49:34.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.782+0000 7faec1282640 1 -- 172.21.15.67:0/2620162268 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb40343f0 con 0x7faebc0cd030 2024-03-24T00:49:34.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.783+0000 7faec44e8640 1 -- 172.21.15.67:0/2620162268 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faebc0cd030 msgr2=0x7faebc0b4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:34.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.783+0000 7faec44e8640 1 --2- 172.21.15.67:0/2620162268 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faebc0cd030 0x7faebc0b4f20 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7faebc0b63e0 tx=0x7faeb402bed0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.783+0000 7faec44e8640 1 -- 172.21.15.67:0/2620162268 shutdown_connections 2024-03-24T00:49:34.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.783+0000 7faec44e8640 1 --2- 172.21.15.67:0/2620162268 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faebc0cd030 0x7faebc0b4f20 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.783+0000 7faec44e8640 1 --2- 172.21.15.67:0/2620162268 >> v2:172.21.15.72:3300/0 conn(0x7faebc1626d0 0x7faebc0ccaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.783+0000 7faec44e8640 1 --2- 172.21.15.67:0/2620162268 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faebc15e3a0 0x7faebc0b2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.783+0000 7faec44e8640 1 -- 172.21.15.67:0/2620162268 >> 172.21.15.67:0/2620162268 conn(0x7faebc0b3c40 msgr2=0x7faebc0b1dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:34.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.783+0000 7faec44e8640 1 -- 172.21.15.67:0/2620162268 shutdown_connections 2024-03-24T00:49:34.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.783+0000 7faec44e8640 1 -- 172.21.15.67:0/2620162268 wait complete. 2024-03-24T00:49:34.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.785+0000 7faec44e8640 1 Processor -- start 2024-03-24T00:49:34.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.785+0000 7faec44e8640 1 -- start start 2024-03-24T00:49:34.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.786+0000 7faec44e8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faebc0cd030 0x7faebc0b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:34.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.786+0000 7faec44e8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faebc15e3a0 0x7faebc1f46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:34.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.786+0000 7faec44e8640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faebc1626d0 0x7faebc0ae660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:34.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.786+0000 7faec44e8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faebc162b10 con 0x7faebc15e3a0 2024-03-24T00:49:34.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.786+0000 7faec44e8640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faebc15f100 con 0x7faebc0cd030 2024-03-24T00:49:34.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.786+0000 7faec44e8640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faebc06b040 con 0x7faebc1626d0 2024-03-24T00:49:34.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.786+0000 7faec1a83640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faebc15e3a0 0x7faebc1f46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:34.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.786+0000 7faec1a83640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faebc15e3a0 0x7faebc1f46d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48224/0 (socket says 172.21.15.67:48224) 2024-03-24T00:49:34.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.786+0000 7faec1a83640 1 -- 172.21.15.67:0/4074550300 learned_addr learned my addr 172.21.15.67:0/4074550300 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:34.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.786+0000 7faec2a85640 1 --2- 172.21.15.67:0/4074550300 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faebc1626d0 0x7faebc0ae660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:34.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.786+0000 7faec2284640 1 --2- 172.21.15.67:0/4074550300 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faebc0cd030 0x7faebc0b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:34.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.787+0000 7faec1a83640 1 -- 172.21.15.67:0/4074550300 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faebc1626d0 msgr2=0x7faebc0ae660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:34.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.787+0000 7faec1a83640 1 --2- 172.21.15.67:0/4074550300 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faebc1626d0 0x7faebc0ae660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.787+0000 7faec1a83640 1 -- 172.21.15.67:0/4074550300 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faebc0cd030 msgr2=0x7faebc0b1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:34.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.787+0000 7faec1a83640 1 --2- 172.21.15.67:0/4074550300 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faebc0cd030 0x7faebc0b1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:34.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.787+0000 7faec1a83640 1 -- 172.21.15.67:0/4074550300 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faebc0b23c0 con 0x7faebc15e3a0 2024-03-24T00:49:34.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.787+0000 7faec1a83640 1 --2- 172.21.15.67:0/4074550300 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faebc15e3a0 0x7faebc1f46d0 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7faeac00c820 tx=0x7faeac00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:34.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.787+0000 7faeb2ffd640 1 -- 172.21.15.67:0/4074550300 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeac015070 con 0x7faebc15e3a0 2024-03-24T00:49:34.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.787+0000 7faeb2ffd640 1 -- 172.21.15.67:0/4074550300 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeac007a30 con 0x7faebc15e3a0 2024-03-24T00:49:34.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.787+0000 7faec44e8640 1 -- 172.21.15.67:0/4074550300 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faebc0b0de0 con 0x7faebc15e3a0 2024-03-24T00:49:34.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.787+0000 7faeb2ffd640 1 -- 172.21.15.67:0/4074550300 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeac007de0 con 0x7faebc15e3a0 2024-03-24T00:49:34.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.787+0000 7faec44e8640 1 -- 172.21.15.67:0/4074550300 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faebc0b13a0 con 0x7faebc15e3a0 2024-03-24T00:49:34.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.788+0000 7faec44e8640 1 -- 172.21.15.67:0/4074550300 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faebc04f400 con 0x7faebc15e3a0 2024-03-24T00:49:34.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.788+0000 7faeb2ffd640 1 -- 172.21.15.67:0/4074550300 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7faeac018070 con 0x7faebc15e3a0 2024-03-24T00:49:34.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.789+0000 7faeb2ffd640 1 --2- 172.21.15.67:0/4074550300 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7faea006bbd0 0x7faea006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:34.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.789+0000 7faeb2ffd640 1 -- 172.21.15.67:0/4074550300 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6273+0+0 (secure 0 0 0) 0x7faeac08ab50 con 0x7faebc15e3a0 2024-03-24T00:49:34.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.789+0000 7faec2284640 1 --2- 172.21.15.67:0/4074550300 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7faea006bbd0 0x7faea006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:34.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.790+0000 7faec2284640 1 --2- 172.21.15.67:0/4074550300 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7faea006bbd0 0x7faea006e090 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7faebc1566e0 tx=0x7faeb8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:34.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:34.793+0000 7faeb2ffd640 1 -- 172.21.15.67:0/4074550300 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7faebc0b23c0 con 0x7faebc15e3a0 2024-03-24T00:49:35.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.005+0000 7faec44e8640 1 -- 172.21.15.67:0/4074550300 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7faebc0cda60 con 0x7faebc15e3a0 2024-03-24T00:49:35.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.006+0000 7faeb2ffd640 1 -- 172.21.15.67:0/4074550300 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v122) v1 ==== 93+0+10 (secure 0 0 0) 0x7faeac058db0 con 0x7faebc15e3a0 2024-03-24T00:49:35.006 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-24T00:49:35.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.008+0000 7faec44e8640 1 -- 172.21.15.67:0/4074550300 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7faea006bbd0 msgr2=0x7faea006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:35.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.008+0000 7faec44e8640 1 --2- 172.21.15.67:0/4074550300 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7faea006bbd0 0x7faea006e090 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7faebc1566e0 tx=0x7faeb8008040 comp rx=0 tx=0).stop 2024-03-24T00:49:35.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.008+0000 7faec44e8640 1 -- 172.21.15.67:0/4074550300 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faebc15e3a0 msgr2=0x7faebc1f46d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:35.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.008+0000 7faec44e8640 1 --2- 172.21.15.67:0/4074550300 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faebc15e3a0 0x7faebc1f46d0 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7faeac00c820 tx=0x7faeac00cce0 comp rx=0 tx=0).stop 2024-03-24T00:49:35.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.008+0000 7faec44e8640 1 -- 172.21.15.67:0/4074550300 shutdown_connections 2024-03-24T00:49:35.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.008+0000 7faec44e8640 1 --2- 172.21.15.67:0/4074550300 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7faea006bbd0 0x7faea006e090 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:35.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.008+0000 7faec44e8640 1 --2- 172.21.15.67:0/4074550300 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faebc1626d0 0x7faebc0ae660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:35.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.008+0000 7faec44e8640 1 --2- 172.21.15.67:0/4074550300 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faebc15e3a0 0x7faebc1f46d0 unknown :-1 s=CLOSED pgs=1884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:35.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.008+0000 7faec44e8640 1 --2- 172.21.15.67:0/4074550300 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faebc0cd030 0x7faebc0b1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:35.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.008+0000 7faec44e8640 1 -- 172.21.15.67:0/4074550300 >> 172.21.15.67:0/4074550300 conn(0x7faebc0b3c40 msgr2=0x7faebc0ce7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:35.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.008+0000 7faec44e8640 1 -- 172.21.15.67:0/4074550300 shutdown_connections 2024-03-24T00:49:35.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.008+0000 7faec44e8640 1 -- 172.21.15.67:0/4074550300 wait complete. 2024-03-24T00:49:35.019 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:49:35.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.123+0000 7f99e62a1640 1 Processor -- start 2024-03-24T00:49:35.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.123+0000 7f99e62a1640 1 -- start start 2024-03-24T00:49:35.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.124+0000 7f99e62a1640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f99e015bc30 0x7f99e00b1a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:35.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.124+0000 7f99e62a1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f99e015c010 0x7f99e00b1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:35.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.124+0000 7f99e62a1640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99e00b3ae0 con 0x7f99e00b2d90 2024-03-24T00:49:35.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.124+0000 7f99e62a1640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99e00cca60 con 0x7f99e015bc30 2024-03-24T00:49:35.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.124+0000 7f99e62a1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99e00b3dc0 con 0x7f99e015c010 2024-03-24T00:49:35.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.124+0000 7f99dffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f99e015bc30 0x7f99e00b1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:35.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.124+0000 7f99e483e640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f99e00b2d90 0x7f99e00b3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56470/0 (socket says 172.21.15.67:56470) 2024-03-24T00:49:35.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.124+0000 7f99e483e640 1 -- 172.21.15.67:0/1685180977 learned_addr learned my addr 172.21.15.67:0/1685180977 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:49:35.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.124+0000 7f99df7fe640 1 --2- 172.21.15.67:0/1685180977 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f99e015c010 0x7f99e00b1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:35.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.125+0000 7f99dffff640 1 -- 172.21.15.67:0/1685180977 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f99e015c010 msgr2=0x7f99e00b1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:35.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.125+0000 7f99dffff640 1 --2- 172.21.15.67:0/1685180977 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f99e015c010 0x7f99e00b1f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:35.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.125+0000 7f99dffff640 1 -- 172.21.15.67:0/1685180977 >> v1:172.21.15.67:6789/0 conn(0x7f99e00b2d90 legacy=0x7f99e00b3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:35.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.125+0000 7f99dffff640 1 -- 172.21.15.67:0/1685180977 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99e00b40a0 con 0x7f99e015bc30 2024-03-24T00:49:35.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.125+0000 7f99deffd640 1 -- 172.21.15.67:0/1685180977 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1496340418 0 0) 0x7f99e00b3ae0 con 0x7f99e00b2d90 2024-03-24T00:49:35.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.125+0000 7f99df7fe640 1 --2- 172.21.15.67:0/1685180977 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f99e015c010 0x7f99e00b1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:49:35.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.125+0000 7f99dffff640 1 --2- 172.21.15.67:0/1685180977 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99e015bc30 0x7f99e00b1a00 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f99c80097a0 tx=0x7f99c802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ddf3f41dfe76f5f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:35.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.125+0000 7f99deffd640 1 -- 172.21.15.67:0/1685180977 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99c8035020 con 0x7f99e015bc30 2024-03-24T00:49:35.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.125+0000 7f99deffd640 1 -- 172.21.15.67:0/1685180977 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99c8031810 con 0x7f99e015bc30 2024-03-24T00:49:35.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.125+0000 7f99deffd640 1 -- 172.21.15.67:0/1685180977 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99c8031be0 con 0x7f99e015bc30 2024-03-24T00:49:35.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.126+0000 7f99e62a1640 1 -- 172.21.15.67:0/1685180977 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99e015bc30 msgr2=0x7f99e00b1a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:35.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.126+0000 7f99e62a1640 1 --2- 172.21.15.67:0/1685180977 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99e015bc30 0x7f99e00b1a00 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f99c80097a0 tx=0x7f99c802b8b0 comp rx=0 tx=0).stop 2024-03-24T00:49:35.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.126+0000 7f99e62a1640 1 -- 172.21.15.67:0/1685180977 shutdown_connections 2024-03-24T00:49:35.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.126+0000 7f99e62a1640 1 --2- 172.21.15.67:0/1685180977 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f99e015c010 0x7f99e00b1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:35.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.126+0000 7f99e62a1640 1 --2- 172.21.15.67:0/1685180977 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99e015bc30 0x7f99e00b1a00 unknown :-1 s=CLOSED pgs=1886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:35.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.126+0000 7f99e62a1640 1 -- 172.21.15.67:0/1685180977 >> 172.21.15.67:0/1685180977 conn(0x7f99e008b180 msgr2=0x7f99e008b590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:35.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.126+0000 7f99e62a1640 1 -- 172.21.15.67:0/1685180977 shutdown_connections 2024-03-24T00:49:35.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.126+0000 7f99e62a1640 1 -- 172.21.15.67:0/1685180977 wait complete. 2024-03-24T00:49:35.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.128+0000 7f99e62a1640 1 Processor -- start 2024-03-24T00:49:35.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.128+0000 7f99e62a1640 1 -- start start 2024-03-24T00:49:35.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.129+0000 7f99e62a1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f99e00b2d90 0x7f99e01f0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:35.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.129+0000 7f99e62a1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99e015bc30 0x7f99e01f10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:35.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.129+0000 7f99e62a1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99e015c010 0x7f99e015a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:35.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.129+0000 7f99e62a1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e00cd070 con 0x7f99e015c010 2024-03-24T00:49:35.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.129+0000 7f99e62a1640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e00ccef0 con 0x7f99e015bc30 2024-03-24T00:49:35.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.129+0000 7f99e62a1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99e00cd1f0 con 0x7f99e00b2d90 2024-03-24T00:49:35.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.130+0000 7f99e483e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99e015c010 0x7f99e015a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:35.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.130+0000 7f99e483e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99e015c010 0x7f99e015a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48236/0 (socket says 172.21.15.67:48236) 2024-03-24T00:49:35.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.130+0000 7f99e483e640 1 -- 172.21.15.67:0/703130299 learned_addr learned my addr 172.21.15.67:0/703130299 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:35.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.130+0000 7f99dffff640 1 --2- 172.21.15.67:0/703130299 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f99e00b2d90 0x7f99e01f0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:35.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.130+0000 7f99df7fe640 1 --2- 172.21.15.67:0/703130299 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99e015bc30 0x7f99e01f10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:35.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.130+0000 7f99e483e640 1 -- 172.21.15.67:0/703130299 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f99e00b2d90 msgr2=0x7f99e01f0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:35.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.130+0000 7f99e483e640 1 --2- 172.21.15.67:0/703130299 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f99e00b2d90 0x7f99e01f0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:35.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.130+0000 7f99e483e640 1 -- 172.21.15.67:0/703130299 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99e015bc30 msgr2=0x7f99e01f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:35.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.130+0000 7f99e483e640 1 --2- 172.21.15.67:0/703130299 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99e015bc30 0x7f99e01f10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:35.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.130+0000 7f99e483e640 1 -- 172.21.15.67:0/703130299 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99e006d8c0 con 0x7f99e015c010 2024-03-24T00:49:35.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.130+0000 7f99dffff640 1 --2- 172.21.15.67:0/703130299 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f99e00b2d90 0x7f99e01f0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:35.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.130+0000 7f99df7fe640 1 --2- 172.21.15.67:0/703130299 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99e015bc30 0x7f99e01f10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:35.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.130+0000 7f99e483e640 1 --2- 172.21.15.67:0/703130299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99e015c010 0x7f99e015a8e0 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f99d800c7a0 tx=0x7f99d800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:35.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.130+0000 7f99dcff9640 1 -- 172.21.15.67:0/703130299 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d8018070 con 0x7f99e015c010 2024-03-24T00:49:35.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.130+0000 7f99dcff9640 1 -- 172.21.15.67:0/703130299 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99d800a5e0 con 0x7f99e015c010 2024-03-24T00:49:35.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.130+0000 7f99dcff9640 1 -- 172.21.15.67:0/703130299 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d800aa50 con 0x7f99e015c010 2024-03-24T00:49:35.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.130+0000 7f99e62a1640 1 -- 172.21.15.67:0/703130299 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99e006dba0 con 0x7f99e015c010 2024-03-24T00:49:35.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.130+0000 7f99e62a1640 1 -- 172.21.15.67:0/703130299 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99e00c9d00 con 0x7f99e015c010 2024-03-24T00:49:35.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.131+0000 7f99e62a1640 1 -- 172.21.15.67:0/703130299 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99e00c9f00 con 0x7f99e015c010 2024-03-24T00:49:35.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.132+0000 7f99dcff9640 1 -- 172.21.15.67:0/703130299 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f99d800abf0 con 0x7f99e015c010 2024-03-24T00:49:35.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.132+0000 7f99dcff9640 1 --2- 172.21.15.67:0/703130299 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f99c006bbd0 0x7f99c006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:35.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.132+0000 7f99dcff9640 1 -- 172.21.15.67:0/703130299 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f99d808a350 con 0x7f99e015c010 2024-03-24T00:49:35.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.132+0000 7f99dffff640 1 --2- 172.21.15.67:0/703130299 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f99c006bbd0 0x7f99c006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:35.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.133+0000 7f99dffff640 1 --2- 172.21.15.67:0/703130299 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f99c006bbd0 0x7f99c006e090 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f99c8036000 tx=0x7f99c8008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:35.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.136+0000 7f99dcff9640 1 -- 172.21.15.67:0/703130299 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f99e00c9f00 con 0x7f99e015c010 2024-03-24T00:49:35.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.320+0000 7f99e62a1640 1 -- 172.21.15.67:0/703130299 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f99e00b17e0 con 0x7f99e015c010 2024-03-24T00:49:35.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.320+0000 7f99dcff9640 1 -- 172.21.15.67:0/703130299 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v122) v1 ==== 104+0+11 (secure 0 0 0) 0x7f99d8058c10 con 0x7f99e015c010 2024-03-24T00:49:35.320 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-24T00:49:35.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.322+0000 7f99e62a1640 1 -- 172.21.15.67:0/703130299 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f99c006bbd0 msgr2=0x7f99c006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:35.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.322+0000 7f99e62a1640 1 --2- 172.21.15.67:0/703130299 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f99c006bbd0 0x7f99c006e090 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f99c8036000 tx=0x7f99c8008030 comp rx=0 tx=0).stop 2024-03-24T00:49:35.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.322+0000 7f99e62a1640 1 -- 172.21.15.67:0/703130299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99e015c010 msgr2=0x7f99e015a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:35.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.322+0000 7f99e62a1640 1 --2- 172.21.15.67:0/703130299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99e015c010 0x7f99e015a8e0 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f99d800c7a0 tx=0x7f99d800cc60 comp rx=0 tx=0).stop 2024-03-24T00:49:35.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.322+0000 7f99e62a1640 1 -- 172.21.15.67:0/703130299 shutdown_connections 2024-03-24T00:49:35.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.322+0000 7f99e62a1640 1 --2- 172.21.15.67:0/703130299 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f99c006bbd0 0x7f99c006e090 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:35.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.322+0000 7f99e62a1640 1 --2- 172.21.15.67:0/703130299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99e015c010 0x7f99e015a8e0 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:35.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.322+0000 7f99e62a1640 1 --2- 172.21.15.67:0/703130299 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99e015bc30 0x7f99e01f10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:35.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.322+0000 7f99e62a1640 1 --2- 172.21.15.67:0/703130299 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f99e00b2d90 0x7f99e01f0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:35.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.322+0000 7f99e62a1640 1 -- 172.21.15.67:0/703130299 >> 172.21.15.67:0/703130299 conn(0x7f99e008b180 msgr2=0x7f99e0156950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:35.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.322+0000 7f99e62a1640 1 -- 172.21.15.67:0/703130299 shutdown_connections 2024-03-24T00:49:35.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.322+0000 7f99e62a1640 1 -- 172.21.15.67:0/703130299 wait complete. 2024-03-24T00:49:35.333 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:49:35.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.438+0000 7f4ea8f77640 1 Processor -- start 2024-03-24T00:49:35.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.438+0000 7f4ea8f77640 1 -- start start 2024-03-24T00:49:35.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.439+0000 7f4ea8f77640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4ea4145540 0x7f4ea4160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:35.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.439+0000 7f4ea8f77640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4ea4071af0 0x7f4ea4071f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:35.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.439+0000 7f4ea8f77640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ea406dcf0 con 0x7f4ea415c2a0 2024-03-24T00:49:35.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.439+0000 7f4ea8f77640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ea40724a0 con 0x7f4ea4071af0 2024-03-24T00:49:35.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.439+0000 7f4ea8f77640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea406e620 con 0x7f4ea4145540 2024-03-24T00:49:35.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.439+0000 7f4ea1d9b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4ea415c2a0 0x7f4ea40cef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56474/0 (socket says 172.21.15.67:56474) 2024-03-24T00:49:35.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.439+0000 7f4ea1d9b640 1 -- 172.21.15.67:0/646742455 learned_addr learned my addr 172.21.15.67:0/646742455 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:49:35.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.439+0000 7f4ea2d9d640 1 --2- 172.21.15.67:0/646742455 >> v2:172.21.15.72:3300/0 conn(0x7f4ea4071af0 0x7f4ea4071f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:35.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.439+0000 7f4ea259c640 1 --2- 172.21.15.67:0/646742455 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4ea4145540 0x7f4ea4160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:35.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.439+0000 7f4ea159a640 1 -- 172.21.15.67:0/646742455 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 684621732 0 0) 0x7f4ea406dcf0 con 0x7f4ea415c2a0 2024-03-24T00:49:35.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.439+0000 7f4ea159a640 1 -- 172.21.15.67:0/646742455 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e90003520 con 0x7f4ea415c2a0 2024-03-24T00:49:35.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.439+0000 7f4ea159a640 1 -- 172.21.15.67:0/646742455 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3551655319 0 0) 0x7f4e90003520 con 0x7f4ea415c2a0 2024-03-24T00:49:35.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.439+0000 7f4ea159a640 1 -- 172.21.15.67:0/646742455 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4ea4145540 msgr2=0x7f4ea4160420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:35.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.439+0000 7f4ea159a640 1 --2- 172.21.15.67:0/646742455 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4ea4145540 0x7f4ea4160420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:35.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.439+0000 7f4ea159a640 1 -- 172.21.15.67:0/646742455 >> v2:172.21.15.72:3300/0 conn(0x7f4ea4071af0 msgr2=0x7f4ea4071f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:35.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.439+0000 7f4ea159a640 1 --2- 172.21.15.67:0/646742455 >> v2:172.21.15.72:3300/0 conn(0x7f4ea4071af0 0x7f4ea4071f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:35.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.439+0000 7f4ea159a640 1 -- 172.21.15.67:0/646742455 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ea406dfd0 con 0x7f4ea415c2a0 2024-03-24T00:49:35.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.439+0000 7f4ea159a640 1 -- 172.21.15.67:0/646742455 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f4e8c001670 con 0x7f4ea415c2a0 2024-03-24T00:49:35.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.439+0000 7f4ea259c640 1 --2- 172.21.15.67:0/646742455 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4ea4145540 0x7f4ea4160420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:49:35.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.439+0000 7f4ea159a640 1 -- 172.21.15.67:0/646742455 >> v1:172.21.15.67:6789/0 conn(0x7f4ea415c2a0 legacy=0x7f4ea40cef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:35.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.439+0000 7f4ea159a640 1 --2- 172.21.15.67:0/646742455 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e900039b0 0x7f4e90003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:35.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.439+0000 7f4ea159a640 1 --2- 172.21.15.67:0/646742455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e90004800 0x7f4e90004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:35.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.439+0000 7f4ea159a640 1 --2- 172.21.15.67:0/646742455 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4ea4145540 0x7f4e900051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:35.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.440+0000 7f4ea159a640 1 -- 172.21.15.67:0/646742455 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea40724a0 con 0x7f4e90004800 2024-03-24T00:49:35.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.440+0000 7f4ea159a640 1 -- 172.21.15.67:0/646742455 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea406e620 con 0x7f4e900039b0 2024-03-24T00:49:35.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.440+0000 7f4ea159a640 1 -- 172.21.15.67:0/646742455 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9000b9c0 con 0x7f4ea4145540 2024-03-24T00:49:35.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.440+0000 7f4ea1d9b640 1 --2- 172.21.15.67:0/646742455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e90004800 0x7f4e90004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:35.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.440+0000 7f4ea259c640 1 --2- 172.21.15.67:0/646742455 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e900039b0 0x7f4e90003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:35.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.440+0000 7f4ea2d9d640 1 --2- 172.21.15.67:0/646742455 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4ea4145540 0x7f4e900051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:35.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.440+0000 7f4ea1d9b640 1 -- 172.21.15.67:0/646742455 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4ea4145540 msgr2=0x7f4e900051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:35.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.440+0000 7f4ea1d9b640 1 --2- 172.21.15.67:0/646742455 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4ea4145540 0x7f4e900051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:35.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.440+0000 7f4ea1d9b640 1 -- 172.21.15.67:0/646742455 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e900039b0 msgr2=0x7f4e90003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:35.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.440+0000 7f4ea1d9b640 1 --2- 172.21.15.67:0/646742455 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e900039b0 0x7f4e90003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:35.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.440+0000 7f4ea1d9b640 1 -- 172.21.15.67:0/646742455 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e90003520 con 0x7f4e90004800 2024-03-24T00:49:35.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.441+0000 7f4ea1d9b640 1 --2- 172.21.15.67:0/646742455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e90004800 0x7f4e90004c80 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c007fd0 tx=0x7f4e90005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a207f25c6753d5fc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:35.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.441+0000 7f4ea159a640 1 -- 172.21.15.67:0/646742455 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e8c007d50 con 0x7f4e90004800 2024-03-24T00:49:35.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.441+0000 7f4ea159a640 1 -- 172.21.15.67:0/646742455 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e8c030070 con 0x7f4e90004800 2024-03-24T00:49:35.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.441+0000 7f4ea159a640 1 -- 172.21.15.67:0/646742455 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e8c02cd80 con 0x7f4e90004800 2024-03-24T00:49:35.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.441+0000 7f4ea8f77640 1 -- 172.21.15.67:0/646742455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e90004800 msgr2=0x7f4e90004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:35.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.441+0000 7f4ea8f77640 1 --2- 172.21.15.67:0/646742455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e90004800 0x7f4e90004c80 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c007fd0 tx=0x7f4e90005830 comp rx=0 tx=0).stop 2024-03-24T00:49:35.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.441+0000 7f4ea2d9d640 1 -- 172.21.15.67:0/646742455 reap_dead start 2024-03-24T00:49:35.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.441+0000 7f4ea8f77640 1 -- 172.21.15.67:0/646742455 shutdown_connections 2024-03-24T00:49:35.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.441+0000 7f4ea8f77640 1 -- 172.21.15.67:0/646742455 >> 172.21.15.67:0/646742455 conn(0x7f4ea40cf9e0 msgr2=0x7f4ea40cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:35.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.441+0000 7f4ea8f77640 1 -- 172.21.15.67:0/646742455 shutdown_connections 2024-03-24T00:49:35.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.441+0000 7f4ea8f77640 1 -- 172.21.15.67:0/646742455 wait complete. 2024-03-24T00:49:35.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.443+0000 7f4ea8f77640 1 Processor -- start 2024-03-24T00:49:35.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.443+0000 7f4ea8f77640 1 -- start start 2024-03-24T00:49:35.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.443+0000 7f4ea8f77640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e90004800 0x7f4ea4139c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:35.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.443+0000 7f4ea8f77640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ea415b3a0 0x7f4ea41e7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:35.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.443+0000 7f4ea8f77640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4ea413a4b0 0x7f4ea413a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:35.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.443+0000 7f4ea8f77640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea4160420 con 0x7f4e90004800 2024-03-24T00:49:35.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.443+0000 7f4ea8f77640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea4145920 con 0x7f4ea415b3a0 2024-03-24T00:49:35.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.443+0000 7f4ea8f77640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea41605a0 con 0x7f4ea413a4b0 2024-03-24T00:49:35.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.444+0000 7f4ea259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e90004800 0x7f4ea4139c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:35.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.444+0000 7f4ea259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e90004800 0x7f4ea4139c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48256/0 (socket says 172.21.15.67:48256) 2024-03-24T00:49:35.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.444+0000 7f4ea259c640 1 -- 172.21.15.67:0/1237532850 learned_addr learned my addr 172.21.15.67:0/1237532850 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:35.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.444+0000 7f4ea2d9d640 1 --2- 172.21.15.67:0/1237532850 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4ea413a4b0 0x7f4ea413a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:35.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.444+0000 7f4ea1d9b640 1 --2- 172.21.15.67:0/1237532850 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ea415b3a0 0x7f4ea41e7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:35.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.444+0000 7f4ea259c640 1 -- 172.21.15.67:0/1237532850 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4ea413a4b0 msgr2=0x7f4ea413a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:35.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.444+0000 7f4ea259c640 1 --2- 172.21.15.67:0/1237532850 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4ea413a4b0 0x7f4ea413a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:35.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.444+0000 7f4ea259c640 1 -- 172.21.15.67:0/1237532850 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ea415b3a0 msgr2=0x7f4ea41e7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:35.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.444+0000 7f4ea259c640 1 --2- 172.21.15.67:0/1237532850 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ea415b3a0 0x7f4ea41e7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:35.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.444+0000 7f4ea259c640 1 -- 172.21.15.67:0/1237532850 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ea41e7f90 con 0x7f4e90004800 2024-03-24T00:49:35.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.444+0000 7f4ea1d9b640 1 --2- 172.21.15.67:0/1237532850 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ea415b3a0 0x7f4ea41e7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:49:35.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.444+0000 7f4ea259c640 1 --2- 172.21.15.67:0/1237532850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e90004800 0x7f4ea4139c50 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f4e94009fb0 tx=0x7f4e9400f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:35.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.444+0000 7f4e9affd640 1 -- 172.21.15.67:0/1237532850 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e9400fea0 con 0x7f4e90004800 2024-03-24T00:49:35.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.444+0000 7f4e9affd640 1 -- 172.21.15.67:0/1237532850 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e94003b90 con 0x7f4e90004800 2024-03-24T00:49:35.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.444+0000 7f4e9affd640 1 -- 172.21.15.67:0/1237532850 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e94004020 con 0x7f4e90004800 2024-03-24T00:49:35.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.445+0000 7f4ea8f77640 1 -- 172.21.15.67:0/1237532850 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ea41e79d0 con 0x7f4e90004800 2024-03-24T00:49:35.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.445+0000 7f4ea8f77640 1 -- 172.21.15.67:0/1237532850 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ea4137aa0 con 0x7f4e90004800 2024-03-24T00:49:35.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.446+0000 7f4ea8f77640 1 -- 172.21.15.67:0/1237532850 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ea404f400 con 0x7f4e90004800 2024-03-24T00:49:35.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.446+0000 7f4e9affd640 1 -- 172.21.15.67:0/1237532850 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4e940041c0 con 0x7f4e90004800 2024-03-24T00:49:35.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.446+0000 7f4e9affd640 1 --2- 172.21.15.67:0/1237532850 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4e8406bc20 0x7f4e8406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:35.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.446+0000 7f4e9affd640 1 -- 172.21.15.67:0/1237532850 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f4e9408b880 con 0x7f4e90004800 2024-03-24T00:49:35.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.447+0000 7f4ea1d9b640 1 --2- 172.21.15.67:0/1237532850 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4e8406bc20 0x7f4e8406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:35.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.447+0000 7f4ea1d9b640 1 --2- 172.21.15.67:0/1237532850 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4e8406bc20 0x7f4e8406e0e0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c034000 tx=0x7f4e8c007000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:35.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.451+0000 7f4e9affd640 1 -- 172.21.15.67:0/1237532850 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4e94019030 con 0x7f4e90004800 2024-03-24T00:49:35.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.632+0000 7f4ea8f77640 1 -- 172.21.15.67:0/1237532850 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4ea40d0740 con 0x7f4e90004800 2024-03-24T00:49:35.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.632+0000 7f4e9affd640 1 -- 172.21.15.67:0/1237532850 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v122) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4e94010040 con 0x7f4e90004800 2024-03-24T00:49:35.633 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-03-24T00:49:35.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.635+0000 7f4ea8f77640 1 -- 172.21.15.67:0/1237532850 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4e8406bc20 msgr2=0x7f4e8406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:35.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.635+0000 7f4ea8f77640 1 --2- 172.21.15.67:0/1237532850 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4e8406bc20 0x7f4e8406e0e0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c034000 tx=0x7f4e8c007000 comp rx=0 tx=0).stop 2024-03-24T00:49:35.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.635+0000 7f4ea8f77640 1 -- 172.21.15.67:0/1237532850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e90004800 msgr2=0x7f4ea4139c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:35.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.635+0000 7f4ea8f77640 1 --2- 172.21.15.67:0/1237532850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e90004800 0x7f4ea4139c50 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f4e94009fb0 tx=0x7f4e9400f580 comp rx=0 tx=0).stop 2024-03-24T00:49:35.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.635+0000 7f4ea8f77640 1 -- 172.21.15.67:0/1237532850 shutdown_connections 2024-03-24T00:49:35.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.635+0000 7f4ea8f77640 1 --2- 172.21.15.67:0/1237532850 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4e8406bc20 0x7f4e8406e0e0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:35.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.635+0000 7f4ea8f77640 1 --2- 172.21.15.67:0/1237532850 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4ea413a4b0 0x7f4ea413a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:35.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.635+0000 7f4ea8f77640 1 --2- 172.21.15.67:0/1237532850 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ea415b3a0 0x7f4ea41e7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:35.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.635+0000 7f4ea8f77640 1 --2- 172.21.15.67:0/1237532850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e90004800 0x7f4ea4139c50 unknown :-1 s=CLOSED pgs=1890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:35.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.635+0000 7f4ea8f77640 1 -- 172.21.15.67:0/1237532850 >> 172.21.15.67:0/1237532850 conn(0x7f4ea40cf9e0 msgr2=0x7f4ea40cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:35.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.635+0000 7f4ea8f77640 1 -- 172.21.15.67:0/1237532850 shutdown_connections 2024-03-24T00:49:35.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.635+0000 7f4ea8f77640 1 -- 172.21.15.67:0/1237532850 wait complete. 2024-03-24T00:49:35.645 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:49:35.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.747+0000 7fe8fb859640 1 Processor -- start 2024-03-24T00:49:35.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.747+0000 7fe8fb859640 1 -- start start 2024-03-24T00:49:35.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.748+0000 7fe8fb859640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe8f40b49c0 0x7fe8f415b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:35.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.748+0000 7fe8fb859640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe8f406f100 0x7fe8f4072360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:35.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.748+0000 7fe8fb859640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8f40b4da0 con 0x7fe8f40728a0 2024-03-24T00:49:35.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.748+0000 7fe8fb859640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8f406db30 con 0x7fe8f40b49c0 2024-03-24T00:49:35.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.748+0000 7fe8fb859640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f406de10 con 0x7fe8f406f100 2024-03-24T00:49:35.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.748+0000 7fe8f95f5640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe8f40b49c0 0x7fe8f415b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:35.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.748+0000 7fe8f95f5640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe8f40b49c0 0x7fe8f415b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48260/0 (socket says 172.21.15.67:48260) 2024-03-24T00:49:35.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.748+0000 7fe8f95f5640 1 -- 172.21.15.67:0/4284303792 learned_addr learned my addr 172.21.15.67:0/4284303792 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:35.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.749+0000 7fe8f8df4640 1 --2- 172.21.15.67:0/4284303792 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe8f406f100 0x7fe8f4072360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:35.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.749+0000 7fe8f95f5640 1 -- 172.21.15.67:0/4284303792 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe8f406f100 msgr2=0x7fe8f4072360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:35.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.749+0000 7fe8f95f5640 1 --2- 172.21.15.67:0/4284303792 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe8f406f100 0x7fe8f4072360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:35.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.749+0000 7fe8f95f5640 1 -- 172.21.15.67:0/4284303792 >> v1:172.21.15.72:6789/0 conn(0x7fe8f40728a0 legacy=0x7fe8f415df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:35.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.749+0000 7fe8f95f5640 1 -- 172.21.15.67:0/4284303792 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8f406e0f0 con 0x7fe8f40b49c0 2024-03-24T00:49:35.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.749+0000 7fe8f8df4640 1 --2- 172.21.15.67:0/4284303792 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe8f406f100 0x7fe8f4072360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:35.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.749+0000 7fe8f95f5640 1 --2- 172.21.15.67:0/4284303792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8f40b49c0 0x7fe8f415b430 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7fe8e800aa10 tx=0x7fe8e802cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9ab27a8e0c85134 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:35.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.749+0000 7fe8e7fff640 1 -- 172.21.15.67:0/4284303792 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8e803a070 con 0x7fe8f40b49c0 2024-03-24T00:49:35.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.750+0000 7fe8e7fff640 1 -- 172.21.15.67:0/4284303792 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8e802dae0 con 0x7fe8f40b49c0 2024-03-24T00:49:35.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.750+0000 7fe8e7fff640 1 -- 172.21.15.67:0/4284303792 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8e800b040 con 0x7fe8f40b49c0 2024-03-24T00:49:35.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.750+0000 7fe8fb859640 1 -- 172.21.15.67:0/4284303792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8f40b49c0 msgr2=0x7fe8f415b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:35.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.750+0000 7fe8fb859640 1 --2- 172.21.15.67:0/4284303792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8f40b49c0 0x7fe8f415b430 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7fe8e800aa10 tx=0x7fe8e802cd30 comp rx=0 tx=0).stop 2024-03-24T00:49:35.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.750+0000 7fe8fb859640 1 -- 172.21.15.67:0/4284303792 shutdown_connections 2024-03-24T00:49:35.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.750+0000 7fe8fb859640 1 --2- 172.21.15.67:0/4284303792 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe8f406f100 0x7fe8f4072360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:35.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.750+0000 7fe8fb859640 1 --2- 172.21.15.67:0/4284303792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8f40b49c0 0x7fe8f415b430 unknown :-1 s=CLOSED pgs=1891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:35.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.750+0000 7fe8fb859640 1 -- 172.21.15.67:0/4284303792 >> 172.21.15.67:0/4284303792 conn(0x7fe8f40cf9e0 msgr2=0x7fe8f40cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:35.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.750+0000 7fe8fb859640 1 -- 172.21.15.67:0/4284303792 shutdown_connections 2024-03-24T00:49:35.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.750+0000 7fe8fb859640 1 -- 172.21.15.67:0/4284303792 wait complete. 2024-03-24T00:49:35.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.751+0000 7fe8fb859640 1 Processor -- start 2024-03-24T00:49:35.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.751+0000 7fe8fb859640 1 -- start start 2024-03-24T00:49:35.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.752+0000 7fe8fb859640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8f406f100 0x7fe8f40a8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:35.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.752+0000 7fe8fb859640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe8f40728a0 0x7fe8f415cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:35.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.752+0000 7fe8fb859640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8f40b49c0 0x7fe8f415d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:35.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.752+0000 7fe8fb859640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f406f560 con 0x7fe8f40b49c0 2024-03-24T00:49:35.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.752+0000 7fe8fb859640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f406e6b0 con 0x7fe8f406f100 2024-03-24T00:49:35.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.752+0000 7fe8fb859640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f40b3d30 con 0x7fe8f40728a0 2024-03-24T00:49:35.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.752+0000 7fe8f9df6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8f40b49c0 0x7fe8f415d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:35.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.752+0000 7fe8f9df6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8f40b49c0 0x7fe8f415d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48274/0 (socket says 172.21.15.67:48274) 2024-03-24T00:49:35.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.752+0000 7fe8f9df6640 1 -- 172.21.15.67:0/2816629425 learned_addr learned my addr 172.21.15.67:0/2816629425 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:35.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.752+0000 7fe8f8df4640 1 --2- 172.21.15.67:0/2816629425 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe8f40728a0 0x7fe8f415cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:35.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.752+0000 7fe8f95f5640 1 --2- 172.21.15.67:0/2816629425 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8f406f100 0x7fe8f40a8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:35.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.752+0000 7fe8f9df6640 1 -- 172.21.15.67:0/2816629425 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe8f40728a0 msgr2=0x7fe8f415cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:35.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.752+0000 7fe8f9df6640 1 --2- 172.21.15.67:0/2816629425 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe8f40728a0 0x7fe8f415cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:35.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.752+0000 7fe8f9df6640 1 -- 172.21.15.67:0/2816629425 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8f406f100 msgr2=0x7fe8f40a8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:35.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.752+0000 7fe8f9df6640 1 --2- 172.21.15.67:0/2816629425 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8f406f100 0x7fe8f40a8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:35.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.752+0000 7fe8f9df6640 1 -- 172.21.15.67:0/2816629425 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8f41f52f0 con 0x7fe8f40b49c0 2024-03-24T00:49:35.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.753+0000 7fe8f9df6640 1 --2- 172.21.15.67:0/2816629425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8f40b49c0 0x7fe8f415d0d0 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec00c780 tx=0x7fe8ec00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:35.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.753+0000 7fe8e5ffb640 1 -- 172.21.15.67:0/2816629425 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ec004b60 con 0x7fe8f40b49c0 2024-03-24T00:49:35.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.753+0000 7fe8e5ffb640 1 -- 172.21.15.67:0/2816629425 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8ec004d00 con 0x7fe8f40b49c0 2024-03-24T00:49:35.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.753+0000 7fe8fb859640 1 -- 172.21.15.67:0/2816629425 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8f41f55d0 con 0x7fe8f40b49c0 2024-03-24T00:49:35.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.753+0000 7fe8e5ffb640 1 -- 172.21.15.67:0/2816629425 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ec00aa90 con 0x7fe8f40b49c0 2024-03-24T00:49:35.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.753+0000 7fe8fb859640 1 -- 172.21.15.67:0/2816629425 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8f41f5b90 con 0x7fe8f40b49c0 2024-03-24T00:49:35.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.754+0000 7fe8fb859640 1 -- 172.21.15.67:0/2816629425 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8f4152540 con 0x7fe8f40b49c0 2024-03-24T00:49:35.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.755+0000 7fe8e5ffb640 1 -- 172.21.15.67:0/2816629425 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe8ec00ac30 con 0x7fe8f40b49c0 2024-03-24T00:49:35.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.755+0000 7fe8e5ffb640 1 --2- 172.21.15.67:0/2816629425 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe8d806bbd0 0x7fe8d806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:35.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.755+0000 7fe8e5ffb640 1 -- 172.21.15.67:0/2816629425 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6273+0+0 (secure 0 0 0) 0x7fe8ec08ae40 con 0x7fe8f40b49c0 2024-03-24T00:49:35.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.756+0000 7fe8f95f5640 1 --2- 172.21.15.67:0/2816629425 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe8d806bbd0 0x7fe8d806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:35.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.756+0000 7fe8f95f5640 1 --2- 172.21.15.67:0/2816629425 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe8d806bbd0 0x7fe8d806e090 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fe8e8037000 tx=0x7fe8e8031000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:35.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.760+0000 7fe8e5ffb640 1 -- 172.21.15.67:0/2816629425 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe8ec020070 con 0x7fe8f40b49c0 2024-03-24T00:49:35.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.978+0000 7fe8fb859640 1 -- 172.21.15.67:0/2816629425 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe8f404f400 con 0x7fe8f40b49c0 2024-03-24T00:49:35.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.979+0000 7fe8e5ffb640 1 -- 172.21.15.67:0/2816629425 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+4975 (secure 0 0 0) 0x7fe8ec0590a0 con 0x7fe8f40b49c0 2024-03-24T00:49:35.980 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:35.980 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":99,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7298,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/839277571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":839277571},{"type":"v1","addr":"172.21.15.72:6836","nonce":839277571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":8224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2052380377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2052380377},{"type":"v1","addr":"172.21.15.67:6834","nonce":2052380377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":8239,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2196618204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2196618204},{"type":"v1","addr":"172.21.15.72:6837","nonce":2196618204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:48:54.351575+0000","modified":"2024-03-24T00:49:28.351081+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":122,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9666},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9666":{"gid":9666,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6835/693801918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":693801918},{"type":"v1","addr":"172.21.15.67:6835","nonce":693801918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9666,"qdb_cluster":[9666]},"id":8}]} 2024-03-24T00:49:35.980 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 99 2024-03-24T00:49:35.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.981+0000 7fe8fb859640 1 -- 172.21.15.67:0/2816629425 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe8d806bbd0 msgr2=0x7fe8d806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:35.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.981+0000 7fe8fb859640 1 --2- 172.21.15.67:0/2816629425 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe8d806bbd0 0x7fe8d806e090 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fe8e8037000 tx=0x7fe8e8031000 comp rx=0 tx=0).stop 2024-03-24T00:49:35.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.981+0000 7fe8fb859640 1 -- 172.21.15.67:0/2816629425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8f40b49c0 msgr2=0x7fe8f415d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:35.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.981+0000 7fe8fb859640 1 --2- 172.21.15.67:0/2816629425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8f40b49c0 0x7fe8f415d0d0 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec00c780 tx=0x7fe8ec00cc40 comp rx=0 tx=0).stop 2024-03-24T00:49:35.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.981+0000 7fe8fb859640 1 -- 172.21.15.67:0/2816629425 shutdown_connections 2024-03-24T00:49:35.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.981+0000 7fe8fb859640 1 --2- 172.21.15.67:0/2816629425 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe8d806bbd0 0x7fe8d806e090 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:35.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.981+0000 7fe8fb859640 1 --2- 172.21.15.67:0/2816629425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8f40b49c0 0x7fe8f415d0d0 unknown :-1 s=CLOSED pgs=1892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:35.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.981+0000 7fe8fb859640 1 --2- 172.21.15.67:0/2816629425 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe8f40728a0 0x7fe8f415cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:35.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.981+0000 7fe8fb859640 1 --2- 172.21.15.67:0/2816629425 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8f406f100 0x7fe8f40a8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:35.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.981+0000 7fe8fb859640 1 -- 172.21.15.67:0/2816629425 >> 172.21.15.67:0/2816629425 conn(0x7fe8f40cf9e0 msgr2=0x7fe8f4071ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:35.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.981+0000 7fe8fb859640 1 -- 172.21.15.67:0/2816629425 shutdown_connections 2024-03-24T00:49:35.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:35.981+0000 7fe8fb859640 1 -- 172.21.15.67:0/2816629425 wait complete. 2024-03-24T00:49:35.992 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:49:36.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.093+0000 7f8483cc6640 1 Processor -- start 2024-03-24T00:49:36.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.093+0000 7f8483cc6640 1 -- start start 2024-03-24T00:49:36.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.094+0000 7f8483cc6640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f847c15dec0 0x7f847c0ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:36.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.094+0000 7f8483cc6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f847c0d0ef0 0x7f847c162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:36.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.094+0000 7f8483cc6640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f847c0cf640 con 0x7f847c15e2a0 2024-03-24T00:49:36.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.094+0000 7f8483cc6640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f847c0cee30 con 0x7f847c15dec0 2024-03-24T00:49:36.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.094+0000 7f8483cc6640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f847c0d2120 con 0x7f847c0d0ef0 2024-03-24T00:49:36.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.094+0000 7f8481261640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f847c15e2a0 0x7f847c0d07e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56486/0 (socket says 172.21.15.67:56486) 2024-03-24T00:49:36.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.094+0000 7f8481261640 1 -- 172.21.15.67:0/2832256567 learned_addr learned my addr 172.21.15.67:0/2832256567 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:49:36.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.094+0000 7f8481a62640 1 --2- 172.21.15.67:0/2832256567 >> v2:172.21.15.72:3300/0 conn(0x7f847c15dec0 0x7f847c0ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:36.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.094+0000 7f8482263640 1 --2- 172.21.15.67:0/2832256567 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f847c0d0ef0 0x7f847c162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:36.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.094+0000 7f8480a60640 1 -- 172.21.15.67:0/2832256567 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1274564244 0 0) 0x7f846c001110 con 0x7f847c15e2a0 2024-03-24T00:49:36.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.094+0000 7f8480a60640 1 -- 172.21.15.67:0/2832256567 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8468003520 con 0x7f847c15e2a0 2024-03-24T00:49:36.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.094+0000 7f8480a60640 1 -- 172.21.15.67:0/2832256567 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2061844735 0 0) 0x7f8468003520 con 0x7f847c15e2a0 2024-03-24T00:49:36.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.094+0000 7f8480a60640 1 -- 172.21.15.67:0/2832256567 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f847c0d0ef0 msgr2=0x7f847c162630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:36.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.094+0000 7f8480a60640 1 --2- 172.21.15.67:0/2832256567 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f847c0d0ef0 0x7f847c162630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:36.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.094+0000 7f8480a60640 1 -- 172.21.15.67:0/2832256567 >> v2:172.21.15.72:3300/0 conn(0x7f847c15dec0 msgr2=0x7f847c0ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:36.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.094+0000 7f8480a60640 1 --2- 172.21.15.67:0/2832256567 >> v2:172.21.15.72:3300/0 conn(0x7f847c15dec0 0x7f847c0ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:36.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.094+0000 7f8480a60640 1 -- 172.21.15.67:0/2832256567 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f847c0cf080 con 0x7f847c15e2a0 2024-03-24T00:49:36.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.094+0000 7f8480a60640 1 -- 172.21.15.67:0/2832256567 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f846c001840 con 0x7f847c15e2a0 2024-03-24T00:49:36.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.094+0000 7f8480a60640 1 -- 172.21.15.67:0/2832256567 >> v1:172.21.15.67:6789/0 conn(0x7f847c15e2a0 legacy=0x7f847c0d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:36.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.094+0000 7f8480a60640 1 --2- 172.21.15.67:0/2832256567 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f847c0d0ef0 0x7f84680039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:36.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.094+0000 7f8480a60640 1 --2- 172.21.15.67:0/2832256567 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8468003f30 0x7f8468004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:36.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.094+0000 7f8480a60640 1 --2- 172.21.15.67:0/2832256567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8468004d40 0x7f84680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:36.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.094+0000 7f8480a60640 1 -- 172.21.15.67:0/2832256567 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847c0cee30 con 0x7f8468004d40 2024-03-24T00:49:36.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.094+0000 7f8480a60640 1 -- 172.21.15.67:0/2832256567 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847c0d2120 con 0x7f8468003f30 2024-03-24T00:49:36.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.094+0000 7f8480a60640 1 -- 172.21.15.67:0/2832256567 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f846800ba10 con 0x7f847c0d0ef0 2024-03-24T00:49:36.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.095+0000 7f8482263640 1 --2- 172.21.15.67:0/2832256567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8468004d40 0x7f84680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:36.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.095+0000 7f8481261640 1 --2- 172.21.15.67:0/2832256567 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8468003f30 0x7f8468004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:36.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.095+0000 7f8481a62640 1 --2- 172.21.15.67:0/2832256567 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f847c0d0ef0 0x7f84680039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:36.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.095+0000 7f8482263640 1 -- 172.21.15.67:0/2832256567 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f847c0d0ef0 msgr2=0x7f84680039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:36.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.095+0000 7f8482263640 1 --2- 172.21.15.67:0/2832256567 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f847c0d0ef0 0x7f84680039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:36.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.095+0000 7f8482263640 1 -- 172.21.15.67:0/2832256567 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8468003f30 msgr2=0x7f8468004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:36.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.095+0000 7f8482263640 1 --2- 172.21.15.67:0/2832256567 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8468003f30 0x7f8468004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:36.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.095+0000 7f8482263640 1 -- 172.21.15.67:0/2832256567 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8468003520 con 0x7f8468004d40 2024-03-24T00:49:36.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.096+0000 7f8482263640 1 --2- 172.21.15.67:0/2832256567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8468004d40 0x7f84680051c0 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f8474009fc0 tx=0x7f8468005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dca288a672ec266f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:36.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.096+0000 7f8480a60640 1 -- 172.21.15.67:0/2832256567 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8474039070 con 0x7f8468004d40 2024-03-24T00:49:36.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.096+0000 7f8480a60640 1 -- 172.21.15.67:0/2832256567 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f847402cbd0 con 0x7f8468004d40 2024-03-24T00:49:36.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.096+0000 7f8480a60640 1 -- 172.21.15.67:0/2832256567 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8474009c10 con 0x7f8468004d40 2024-03-24T00:49:36.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.097+0000 7f8483cc6640 1 -- 172.21.15.67:0/2832256567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8468004d40 msgr2=0x7f84680051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:36.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.097+0000 7f8483cc6640 1 --2- 172.21.15.67:0/2832256567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8468004d40 0x7f84680051c0 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f8474009fc0 tx=0x7f8468005830 comp rx=0 tx=0).stop 2024-03-24T00:49:36.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.097+0000 7f8482263640 1 -- 172.21.15.67:0/2832256567 reap_dead start 2024-03-24T00:49:36.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.097+0000 7f8483cc6640 1 -- 172.21.15.67:0/2832256567 shutdown_connections 2024-03-24T00:49:36.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.097+0000 7f8483cc6640 1 -- 172.21.15.67:0/2832256567 >> 172.21.15.67:0/2832256567 conn(0x7f847c06c810 msgr2=0x7f847c06cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:36.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.097+0000 7f8483cc6640 1 -- 172.21.15.67:0/2832256567 shutdown_connections 2024-03-24T00:49:36.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.097+0000 7f8483cc6640 1 -- 172.21.15.67:0/2832256567 wait complete. 2024-03-24T00:49:36.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.098+0000 7f8483cc6640 1 Processor -- start 2024-03-24T00:49:36.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.098+0000 7f8483cc6640 1 -- start start 2024-03-24T00:49:36.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.099+0000 7f8483cc6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f847c14f290 0x7f847c0cd6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:36.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.099+0000 7f8483cc6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f847c0b49b0 0x7f847c15c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:36.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.099+0000 7f8483cc6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f847c0b58c0 0x7f847c15c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:36.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.099+0000 7f8483cc6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847c1627b0 con 0x7f847c0b58c0 2024-03-24T00:49:36.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.099+0000 7f8483cc6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847c162630 con 0x7f847c14f290 2024-03-24T00:49:36.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.099+0000 7f8483cc6640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f847c162930 con 0x7f847c0b49b0 2024-03-24T00:49:36.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.099+0000 7f8482263640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f847c0b58c0 0x7f847c15c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:36.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.099+0000 7f8482263640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f847c0b58c0 0x7f847c15c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48296/0 (socket says 172.21.15.67:48296) 2024-03-24T00:49:36.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.099+0000 7f8482263640 1 -- 172.21.15.67:0/3393491770 learned_addr learned my addr 172.21.15.67:0/3393491770 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:36.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.099+0000 7f8481261640 1 --2- 172.21.15.67:0/3393491770 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f847c0b49b0 0x7f847c15c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:36.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.099+0000 7f8481a62640 1 --2- 172.21.15.67:0/3393491770 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f847c14f290 0x7f847c0cd6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:36.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.099+0000 7f8482263640 1 -- 172.21.15.67:0/3393491770 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f847c0b49b0 msgr2=0x7f847c15c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:36.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.099+0000 7f8482263640 1 --2- 172.21.15.67:0/3393491770 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f847c0b49b0 0x7f847c15c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:36.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.099+0000 7f8482263640 1 -- 172.21.15.67:0/3393491770 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f847c14f290 msgr2=0x7f847c0cd6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:36.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.099+0000 7f8482263640 1 --2- 172.21.15.67:0/3393491770 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f847c14f290 0x7f847c0cd6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:36.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.099+0000 7f8482263640 1 -- 172.21.15.67:0/3393491770 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f847c0b3910 con 0x7f847c0b58c0 2024-03-24T00:49:36.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.099+0000 7f8481a62640 1 --2- 172.21.15.67:0/3393491770 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f847c14f290 0x7f847c0cd6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:49:36.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.099+0000 7f8482263640 1 --2- 172.21.15.67:0/3393491770 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f847c0b58c0 0x7f847c15c690 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f847400e230 tx=0x7f8474009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:36.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.100+0000 7f84727fc640 1 -- 172.21.15.67:0/3393491770 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8474039030 con 0x7f847c0b58c0 2024-03-24T00:49:36.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.100+0000 7f84727fc640 1 -- 172.21.15.67:0/3393491770 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f847403a040 con 0x7f847c0b58c0 2024-03-24T00:49:36.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.100+0000 7f8483cc6640 1 -- 172.21.15.67:0/3393491770 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f847c0b3bf0 con 0x7f847c0b58c0 2024-03-24T00:49:36.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.100+0000 7f84727fc640 1 -- 172.21.15.67:0/3393491770 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8474009c10 con 0x7f847c0b58c0 2024-03-24T00:49:36.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.100+0000 7f8483cc6640 1 -- 172.21.15.67:0/3393491770 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f847c0b41b0 con 0x7f847c0b58c0 2024-03-24T00:49:36.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.101+0000 7f8483cc6640 1 -- 172.21.15.67:0/3393491770 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f847c04f400 con 0x7f847c0b58c0 2024-03-24T00:49:36.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.101+0000 7f84727fc640 1 -- 172.21.15.67:0/3393491770 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8474045020 con 0x7f847c0b58c0 2024-03-24T00:49:36.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.102+0000 7f84727fc640 1 --2- 172.21.15.67:0/3393491770 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f846006bc00 0x7f846006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:36.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.102+0000 7f84727fc640 1 -- 172.21.15.67:0/3393491770 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f84740ad000 con 0x7f847c0b58c0 2024-03-24T00:49:36.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.102+0000 7f8481a62640 1 --2- 172.21.15.67:0/3393491770 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f846006bc00 0x7f846006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:36.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.102+0000 7f8481a62640 1 --2- 172.21.15.67:0/3393491770 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f846006bc00 0x7f846006e0c0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f847c15b4d0 tx=0x7f8478008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:36.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.105+0000 7f84727fc640 1 -- 172.21.15.67:0/3393491770 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f847404c020 con 0x7f847c0b58c0 2024-03-24T00:49:36.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.288+0000 7f8483cc6640 1 -- 172.21.15.67:0/3393491770 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f847c0d1aa0 con 0x7f847c0b58c0 2024-03-24T00:49:36.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.290+0000 7f84727fc640 1 -- 172.21.15.67:0/3393491770 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v122) v1 ==== 74+0+17149 (secure 0 0 0) 0x7f847407bc00 con 0x7f847c0b58c0 2024-03-24T00:49:36.291 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:36.291 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":122,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:49:23.316053+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:48:52.622356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-03-24T00:48:53.311452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6835/3833808901":"2024-03-25T00:49:23.316030+0000","172.21.15.67:6833/3833808901":"2024-03-25T00:49:23.316030+0000","172.21.15.67:6835/312063805":"2024-03-25T00:49:12.456364+0000","172.21.15.67:6833/312063805":"2024-03-25T00:49:12.456364+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:49:36.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.292+0000 7f8483cc6640 1 -- 172.21.15.67:0/3393491770 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f846006bc00 msgr2=0x7f846006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:36.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.292+0000 7f8483cc6640 1 --2- 172.21.15.67:0/3393491770 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f846006bc00 0x7f846006e0c0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f847c15b4d0 tx=0x7f8478008040 comp rx=0 tx=0).stop 2024-03-24T00:49:36.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.292+0000 7f8483cc6640 1 -- 172.21.15.67:0/3393491770 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f847c0b58c0 msgr2=0x7f847c15c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:36.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.292+0000 7f8483cc6640 1 --2- 172.21.15.67:0/3393491770 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f847c0b58c0 0x7f847c15c690 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f847400e230 tx=0x7f8474009750 comp rx=0 tx=0).stop 2024-03-24T00:49:36.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.292+0000 7f8483cc6640 1 -- 172.21.15.67:0/3393491770 shutdown_connections 2024-03-24T00:49:36.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.292+0000 7f8483cc6640 1 --2- 172.21.15.67:0/3393491770 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f846006bc00 0x7f846006e0c0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:36.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.292+0000 7f8483cc6640 1 --2- 172.21.15.67:0/3393491770 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f847c0b58c0 0x7f847c15c690 unknown :-1 s=CLOSED pgs=1895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:36.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.292+0000 7f8483cc6640 1 --2- 172.21.15.67:0/3393491770 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f847c0b49b0 0x7f847c15c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:36.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.292+0000 7f8483cc6640 1 --2- 172.21.15.67:0/3393491770 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f847c14f290 0x7f847c0cd6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:36.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.292+0000 7f8483cc6640 1 -- 172.21.15.67:0/3393491770 >> 172.21.15.67:0/3393491770 conn(0x7f847c06c810 msgr2=0x7f847c15e6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:36.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.292+0000 7f8483cc6640 1 -- 172.21.15.67:0/3393491770 shutdown_connections 2024-03-24T00:49:36.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.292+0000 7f8483cc6640 1 -- 172.21.15.67:0/3393491770 wait complete. 2024-03-24T00:49:36.303 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-24T00:49:36.303 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:49:36.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.404+0000 7f437c187640 1 Processor -- start 2024-03-24T00:49:36.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.404+0000 7f437c187640 1 -- start start 2024-03-24T00:49:36.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.405+0000 7f437c187640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f43740d0ef0 0x7f43740ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:36.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.405+0000 7f437c187640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43740cf640 con 0x7f437415dec0 2024-03-24T00:49:36.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.405+0000 7f437c187640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43740cf080 con 0x7f437415e2a0 2024-03-24T00:49:36.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.405+0000 7f437c187640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43740cee30 con 0x7f43740d0ef0 2024-03-24T00:49:36.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.405+0000 7f4379f23640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f437415dec0 0x7f43740d07e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56490/0 (socket says 172.21.15.67:56490) 2024-03-24T00:49:36.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.405+0000 7f4379f23640 1 -- 172.21.15.67:0/3319900446 learned_addr learned my addr 172.21.15.67:0/3319900446 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:49:36.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.405+0000 7f4378f21640 1 -- 172.21.15.67:0/3319900446 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2994383997 0 0) 0x7f4370001110 con 0x7f437415dec0 2024-03-24T00:49:36.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.405+0000 7f437a724640 1 --2- 172.21.15.67:0/3319900446 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f43740d0ef0 0x7f43740ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:36.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.405+0000 7f4378f21640 1 -- 172.21.15.67:0/3319900446 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4360003520 con 0x7f437415dec0 2024-03-24T00:49:36.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.406+0000 7f4378f21640 1 -- 172.21.15.67:0/3319900446 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3004520374 0 0) 0x7f4360003520 con 0x7f437415dec0 2024-03-24T00:49:36.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.406+0000 7f4378f21640 1 -- 172.21.15.67:0/3319900446 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f43740d0ef0 msgr2=0x7f43740ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:36.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.406+0000 7f4378f21640 1 --2- 172.21.15.67:0/3319900446 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f43740d0ef0 0x7f43740ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:36.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.406+0000 7f4378f21640 1 -- 172.21.15.67:0/3319900446 >> v1:172.21.15.72:6789/0 conn(0x7f437415e2a0 legacy=0x7f43740d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:36.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.406+0000 7f4378f21640 1 -- 172.21.15.67:0/3319900446 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43740cf360 con 0x7f437415dec0 2024-03-24T00:49:36.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.406+0000 7f4378f21640 1 -- 172.21.15.67:0/3319900446 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f4370001840 con 0x7f437415dec0 2024-03-24T00:49:36.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.406+0000 7f437a724640 1 --2- 172.21.15.67:0/3319900446 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f43740d0ef0 0x7f43740ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:49:36.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.406+0000 7f4378f21640 1 -- 172.21.15.67:0/3319900446 >> v1:172.21.15.67:6789/0 conn(0x7f437415dec0 legacy=0x7f43740d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:36.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.406+0000 7f4378f21640 1 --2- 172.21.15.67:0/3319900446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43600039e0 0x7f4360003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:36.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.406+0000 7f4378f21640 1 --2- 172.21.15.67:0/3319900446 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f43740d0ef0 0x7f4360004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:36.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.406+0000 7f4378f21640 1 --2- 172.21.15.67:0/3319900446 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4360004d40 0x7f43600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:36.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.406+0000 7f4378f21640 1 -- 172.21.15.67:0/3319900446 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43740cee30 con 0x7f43600039e0 2024-03-24T00:49:36.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.406+0000 7f4379f23640 1 --2- 172.21.15.67:0/3319900446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43600039e0 0x7f4360003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:36.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.406+0000 7f4378f21640 1 -- 172.21.15.67:0/3319900446 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f436000bbb0 con 0x7f4360004d40 2024-03-24T00:49:36.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.406+0000 7f4378f21640 1 -- 172.21.15.67:0/3319900446 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f436000bdb0 con 0x7f43740d0ef0 2024-03-24T00:49:36.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.406+0000 7f4379722640 1 --2- 172.21.15.67:0/3319900446 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f43740d0ef0 0x7f4360004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:36.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.406+0000 7f437a724640 1 --2- 172.21.15.67:0/3319900446 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4360004d40 0x7f43600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:36.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.406+0000 7f4379f23640 1 -- 172.21.15.67:0/3319900446 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f43740d0ef0 msgr2=0x7f4360004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:36.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.406+0000 7f4379f23640 1 --2- 172.21.15.67:0/3319900446 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f43740d0ef0 0x7f4360004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:36.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.406+0000 7f4379f23640 1 -- 172.21.15.67:0/3319900446 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4360004d40 msgr2=0x7f43600051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:36.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.406+0000 7f4379f23640 1 --2- 172.21.15.67:0/3319900446 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4360004d40 0x7f43600051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:36.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.406+0000 7f4379f23640 1 -- 172.21.15.67:0/3319900446 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4360003520 con 0x7f43600039e0 2024-03-24T00:49:36.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.407+0000 7f4379f23640 1 --2- 172.21.15.67:0/3319900446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43600039e0 0x7f4360003e00 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f437000c4c0 tx=0x7f4360005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eac1ab15f68b794c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:36.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.407+0000 7f4378f21640 1 -- 172.21.15.67:0/3319900446 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f437000c0e0 con 0x7f43600039e0 2024-03-24T00:49:36.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.407+0000 7f4378f21640 1 -- 172.21.15.67:0/3319900446 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4370030070 con 0x7f43600039e0 2024-03-24T00:49:36.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.407+0000 7f4378f21640 1 -- 172.21.15.67:0/3319900446 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43700058d0 con 0x7f43600039e0 2024-03-24T00:49:36.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.408+0000 7f437c187640 1 -- 172.21.15.67:0/3319900446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43600039e0 msgr2=0x7f4360003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:36.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.408+0000 7f437c187640 1 --2- 172.21.15.67:0/3319900446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43600039e0 0x7f4360003e00 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f437000c4c0 tx=0x7f4360005830 comp rx=0 tx=0).stop 2024-03-24T00:49:36.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.408+0000 7f437a724640 1 -- 172.21.15.67:0/3319900446 reap_dead start 2024-03-24T00:49:36.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.408+0000 7f437c187640 1 -- 172.21.15.67:0/3319900446 shutdown_connections 2024-03-24T00:49:36.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.408+0000 7f437c187640 1 -- 172.21.15.67:0/3319900446 >> 172.21.15.67:0/3319900446 conn(0x7f437406c810 msgr2=0x7f437406cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:36.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.408+0000 7f437c187640 1 -- 172.21.15.67:0/3319900446 shutdown_connections 2024-03-24T00:49:36.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.408+0000 7f437c187640 1 -- 172.21.15.67:0/3319900446 wait complete. 2024-03-24T00:49:36.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.409+0000 7f437c187640 1 Processor -- start 2024-03-24T00:49:36.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.409+0000 7f437c187640 1 -- start start 2024-03-24T00:49:36.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.410+0000 7f437c187640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43600039e0 0x7f43740c9b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:36.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.410+0000 7f437c187640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f43740b4c50 0x7f43740b7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:36.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.410+0000 7f437c187640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43740b50d0 0x7f43740b75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:36.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.410+0000 7f437c187640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43740ce9c0 con 0x7f43740b50d0 2024-03-24T00:49:36.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.410+0000 7f437c187640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43740ce840 con 0x7f43600039e0 2024-03-24T00:49:36.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.410+0000 7f437c187640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43740d2120 con 0x7f43740b4c50 2024-03-24T00:49:36.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.410+0000 7f437a724640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43740b50d0 0x7f43740b75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:36.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.410+0000 7f437a724640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43740b50d0 0x7f43740b75d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48314/0 (socket says 172.21.15.67:48314) 2024-03-24T00:49:36.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.410+0000 7f437a724640 1 -- 172.21.15.67:0/1811630026 learned_addr learned my addr 172.21.15.67:0/1811630026 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:36.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.410+0000 7f4379f23640 1 --2- 172.21.15.67:0/1811630026 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43600039e0 0x7f43740c9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:36.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.410+0000 7f4379722640 1 --2- 172.21.15.67:0/1811630026 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f43740b4c50 0x7f43740b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:36.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.410+0000 7f437a724640 1 -- 172.21.15.67:0/1811630026 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f43740b4c50 msgr2=0x7f43740b7090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:36.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.410+0000 7f437a724640 1 --2- 172.21.15.67:0/1811630026 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f43740b4c50 0x7f43740b7090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:36.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.410+0000 7f437a724640 1 -- 172.21.15.67:0/1811630026 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43600039e0 msgr2=0x7f43740c9b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:36.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.410+0000 7f437a724640 1 --2- 172.21.15.67:0/1811630026 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43600039e0 0x7f43740c9b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:36.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.410+0000 7f437a724640 1 -- 172.21.15.67:0/1811630026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43740ca0d0 con 0x7f43740b50d0 2024-03-24T00:49:36.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.410+0000 7f4379f23640 1 --2- 172.21.15.67:0/1811630026 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43600039e0 0x7f43740c9b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:36.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.410+0000 7f437a724640 1 --2- 172.21.15.67:0/1811630026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43740b50d0 0x7f43740b75d0 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7f43740b37e0 tx=0x7f436c010530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:36.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.410+0000 7f436a7fc640 1 -- 172.21.15.67:0/1811630026 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f436c010d20 con 0x7f43740b50d0 2024-03-24T00:49:36.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.410+0000 7f436a7fc640 1 -- 172.21.15.67:0/1811630026 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f436c003990 con 0x7f43740b50d0 2024-03-24T00:49:36.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.410+0000 7f437c187640 1 -- 172.21.15.67:0/1811630026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43740b56a0 con 0x7f43740b50d0 2024-03-24T00:49:36.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.411+0000 7f436a7fc640 1 -- 172.21.15.67:0/1811630026 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f436c003b80 con 0x7f43740b50d0 2024-03-24T00:49:36.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.411+0000 7f437c187640 1 -- 172.21.15.67:0/1811630026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43740b5c60 con 0x7f43740b50d0 2024-03-24T00:49:36.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.412+0000 7f437c187640 1 -- 172.21.15.67:0/1811630026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f437404f400 con 0x7f43740b50d0 2024-03-24T00:49:36.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.412+0000 7f436a7fc640 1 -- 172.21.15.67:0/1811630026 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f436c01e030 con 0x7f43740b50d0 2024-03-24T00:49:36.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.412+0000 7f436a7fc640 1 --2- 172.21.15.67:0/1811630026 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f435806bc00 0x7f435806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:36.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.413+0000 7f436a7fc640 1 -- 172.21.15.67:0/1811630026 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f436c08b2e0 con 0x7f43740b50d0 2024-03-24T00:49:36.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.413+0000 7f4379f23640 1 --2- 172.21.15.67:0/1811630026 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f435806bc00 0x7f435806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:36.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.413+0000 7f4379f23640 1 --2- 172.21.15.67:0/1811630026 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f435806bc00 0x7f435806e0c0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f4370034000 tx=0x7f4370009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:36.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.417+0000 7f436a7fc640 1 -- 172.21.15.67:0/1811630026 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f436c025020 con 0x7f43740b50d0 2024-03-24T00:49:36.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.638+0000 7f437c187640 1 -- 172.21.15.67:0/1811630026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f43740d1aa0 con 0x7f43740b50d0 2024-03-24T00:49:36.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.639+0000 7f436a7fc640 1 -- 172.21.15.67:0/1811630026 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+4975 (secure 0 0 0) 0x7f436c0083a0 con 0x7f43740b50d0 2024-03-24T00:49:36.639 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:36.639 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":99,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7298,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/839277571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":839277571},{"type":"v1","addr":"172.21.15.72:6836","nonce":839277571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":8224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2052380377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2052380377},{"type":"v1","addr":"172.21.15.67:6834","nonce":2052380377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":8239,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2196618204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2196618204},{"type":"v1","addr":"172.21.15.72:6837","nonce":2196618204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:48:54.351575+0000","modified":"2024-03-24T00:49:28.351081+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":122,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9666},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9666":{"gid":9666,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6835/693801918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":693801918},{"type":"v1","addr":"172.21.15.67:6835","nonce":693801918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9666,"qdb_cluster":[9666]},"id":8}]} 2024-03-24T00:49:36.639 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 99 2024-03-24T00:49:36.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.641+0000 7f437c187640 1 -- 172.21.15.67:0/1811630026 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f435806bc00 msgr2=0x7f435806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:36.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.641+0000 7f437c187640 1 --2- 172.21.15.67:0/1811630026 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f435806bc00 0x7f435806e0c0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f4370034000 tx=0x7f4370009030 comp rx=0 tx=0).stop 2024-03-24T00:49:36.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.641+0000 7f437c187640 1 -- 172.21.15.67:0/1811630026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43740b50d0 msgr2=0x7f43740b75d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:36.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.641+0000 7f437c187640 1 --2- 172.21.15.67:0/1811630026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43740b50d0 0x7f43740b75d0 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7f43740b37e0 tx=0x7f436c010530 comp rx=0 tx=0).stop 2024-03-24T00:49:36.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.641+0000 7f437c187640 1 -- 172.21.15.67:0/1811630026 shutdown_connections 2024-03-24T00:49:36.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.641+0000 7f437c187640 1 --2- 172.21.15.67:0/1811630026 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f435806bc00 0x7f435806e0c0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:36.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.641+0000 7f437c187640 1 --2- 172.21.15.67:0/1811630026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43740b50d0 0x7f43740b75d0 unknown :-1 s=CLOSED pgs=1898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:36.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.641+0000 7f437c187640 1 --2- 172.21.15.67:0/1811630026 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f43740b4c50 0x7f43740b7090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:36.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.641+0000 7f437c187640 1 --2- 172.21.15.67:0/1811630026 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43600039e0 0x7f43740c9b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:36.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.641+0000 7f437c187640 1 -- 172.21.15.67:0/1811630026 >> 172.21.15.67:0/1811630026 conn(0x7f437406c810 msgr2=0x7f437415f2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:36.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.641+0000 7f437c187640 1 -- 172.21.15.67:0/1811630026 shutdown_connections 2024-03-24T00:49:36.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.641+0000 7f437c187640 1 -- 172.21.15.67:0/1811630026 wait complete. 2024-03-24T00:49:36.652 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:49:36.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.756+0000 7f08eb709640 1 Processor -- start 2024-03-24T00:49:36.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.756+0000 7f08eb709640 1 -- start start 2024-03-24T00:49:36.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.757+0000 7f08eb709640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f08e4161790 0x7f08e40caad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:36.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.757+0000 7f08eb709640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f08e406b040 0x7f08e40b11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:36.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.757+0000 7f08eb709640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08e4161b70 con 0x7f08e40b1730 2024-03-24T00:49:36.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.757+0000 7f08eb709640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08e40cb010 con 0x7f08e406b040 2024-03-24T00:49:36.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.757+0000 7f08eb709640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08e40d0ea0 con 0x7f08e4161790 2024-03-24T00:49:36.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.757+0000 7f08e8ca4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f08e406b040 0x7f08e40b11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:36.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.757+0000 7f08e8ca4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f08e406b040 0x7f08e40b11f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48322/0 (socket says 172.21.15.67:48322) 2024-03-24T00:49:36.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.758+0000 7f08e8ca4640 1 -- 172.21.15.67:0/3069045089 learned_addr learned my addr 172.21.15.67:0/3069045089 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:36.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.758+0000 7f08e94a5640 1 --2- 172.21.15.67:0/3069045089 >> v2:172.21.15.72:3300/0 conn(0x7f08e4161790 0x7f08e40caad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:36.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.758+0000 7f08e8ca4640 1 -- 172.21.15.67:0/3069045089 >> v2:172.21.15.72:3300/0 conn(0x7f08e4161790 msgr2=0x7f08e40caad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:36.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.758+0000 7f08e8ca4640 1 --2- 172.21.15.67:0/3069045089 >> v2:172.21.15.72:3300/0 conn(0x7f08e4161790 0x7f08e40caad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:36.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.758+0000 7f08e8ca4640 1 -- 172.21.15.67:0/3069045089 >> v1:172.21.15.72:6789/0 conn(0x7f08e40b1730 legacy=0x7f08e40ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:36.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.758+0000 7f08e8ca4640 1 -- 172.21.15.67:0/3069045089 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08e40d1180 con 0x7f08e406b040 2024-03-24T00:49:36.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.758+0000 7f08e94a5640 1 --2- 172.21.15.67:0/3069045089 >> v2:172.21.15.72:3300/0 conn(0x7f08e4161790 0x7f08e40caad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:36.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.759+0000 7f08e8ca4640 1 --2- 172.21.15.67:0/3069045089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08e406b040 0x7f08e40b11f0 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f08e4072620 tx=0x7f08d402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=884e2041e1197070 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:36.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.759+0000 7f08dbfff640 1 -- 172.21.15.67:0/3069045089 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d4035020 con 0x7f08e406b040 2024-03-24T00:49:36.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.759+0000 7f08dbfff640 1 -- 172.21.15.67:0/3069045089 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08d4031870 con 0x7f08e406b040 2024-03-24T00:49:36.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.759+0000 7f08dbfff640 1 -- 172.21.15.67:0/3069045089 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d4031c40 con 0x7f08e406b040 2024-03-24T00:49:36.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.759+0000 7f08eb709640 1 -- 172.21.15.67:0/3069045089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08e406b040 msgr2=0x7f08e40b11f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:36.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.759+0000 7f08eb709640 1 --2- 172.21.15.67:0/3069045089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08e406b040 0x7f08e40b11f0 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f08e4072620 tx=0x7f08d402b910 comp rx=0 tx=0).stop 2024-03-24T00:49:36.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.759+0000 7f08eb709640 1 -- 172.21.15.67:0/3069045089 shutdown_connections 2024-03-24T00:49:36.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.759+0000 7f08eb709640 1 --2- 172.21.15.67:0/3069045089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08e406b040 0x7f08e40b11f0 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:36.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.759+0000 7f08eb709640 1 --2- 172.21.15.67:0/3069045089 >> v2:172.21.15.72:3300/0 conn(0x7f08e4161790 0x7f08e40caad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:36.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.759+0000 7f08eb709640 1 -- 172.21.15.67:0/3069045089 >> 172.21.15.67:0/3069045089 conn(0x7f08e40b5440 msgr2=0x7f08e40b33d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:36.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.759+0000 7f08eb709640 1 -- 172.21.15.67:0/3069045089 shutdown_connections 2024-03-24T00:49:36.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.759+0000 7f08eb709640 1 -- 172.21.15.67:0/3069045089 wait complete. 2024-03-24T00:49:36.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.760+0000 7f08eb709640 1 Processor -- start 2024-03-24T00:49:36.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.760+0000 7f08eb709640 1 -- start start 2024-03-24T00:49:36.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.761+0000 7f08eb709640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f08e406b040 0x7f08e40ae290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:36.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.761+0000 7f08eb709640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08e40b1730 0x7f08e40ae7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:36.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.761+0000 7f08eb709640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f08e4161790 0x7f08e41585e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:36.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.761+0000 7f08eb709640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e40b3cf0 con 0x7f08e40b1730 2024-03-24T00:49:36.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.761+0000 7f08eb709640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e40b09c0 con 0x7f08e406b040 2024-03-24T00:49:36.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.761+0000 7f08eb709640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08e40b3e70 con 0x7f08e4161790 2024-03-24T00:49:36.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.761+0000 7f08e8ca4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08e40b1730 0x7f08e40ae7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:36.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.761+0000 7f08e8ca4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08e40b1730 0x7f08e40ae7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48332/0 (socket says 172.21.15.67:48332) 2024-03-24T00:49:36.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.761+0000 7f08e8ca4640 1 -- 172.21.15.67:0/2133032983 learned_addr learned my addr 172.21.15.67:0/2133032983 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:36.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.761+0000 7f08e94a5640 1 --2- 172.21.15.67:0/2133032983 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f08e406b040 0x7f08e40ae290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:36.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.761+0000 7f08e9ca6640 1 --2- 172.21.15.67:0/2133032983 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f08e4161790 0x7f08e41585e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:36.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.762+0000 7f08e8ca4640 1 -- 172.21.15.67:0/2133032983 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f08e4161790 msgr2=0x7f08e41585e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:36.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.762+0000 7f08e8ca4640 1 --2- 172.21.15.67:0/2133032983 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f08e4161790 0x7f08e41585e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:36.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.762+0000 7f08e8ca4640 1 -- 172.21.15.67:0/2133032983 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f08e406b040 msgr2=0x7f08e40ae290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:36.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.762+0000 7f08e8ca4640 1 --2- 172.21.15.67:0/2133032983 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f08e406b040 0x7f08e40ae290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:36.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.762+0000 7f08e8ca4640 1 -- 172.21.15.67:0/2133032983 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08e415f640 con 0x7f08e40b1730 2024-03-24T00:49:36.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.762+0000 7f08e9ca6640 1 --2- 172.21.15.67:0/2133032983 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f08e4161790 0x7f08e41585e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:36.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.762+0000 7f08e8ca4640 1 --2- 172.21.15.67:0/2133032983 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08e40b1730 0x7f08e40ae7d0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f08d402b950 tx=0x7f08d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:36.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.762+0000 7f08d9ffb640 1 -- 172.21.15.67:0/2133032983 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d4040070 con 0x7f08e40b1730 2024-03-24T00:49:36.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.762+0000 7f08d9ffb640 1 -- 172.21.15.67:0/2133032983 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08d403a420 con 0x7f08e40b1730 2024-03-24T00:49:36.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.762+0000 7f08eb709640 1 -- 172.21.15.67:0/2133032983 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08e415f920 con 0x7f08e40b1730 2024-03-24T00:49:36.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.762+0000 7f08d9ffb640 1 -- 172.21.15.67:0/2133032983 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d4035040 con 0x7f08e40b1730 2024-03-24T00:49:36.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.762+0000 7f08eb709640 1 -- 172.21.15.67:0/2133032983 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08e415fee0 con 0x7f08e40b1730 2024-03-24T00:49:36.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.763+0000 7f08eb709640 1 -- 172.21.15.67:0/2133032983 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08e404f400 con 0x7f08e40b1730 2024-03-24T00:49:36.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.764+0000 7f08d9ffb640 1 -- 172.21.15.67:0/2133032983 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f08d404b060 con 0x7f08e40b1730 2024-03-24T00:49:36.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.765+0000 7f08d9ffb640 1 --2- 172.21.15.67:0/2133032983 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f08c806bbd0 0x7f08c806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:36.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.765+0000 7f08d9ffb640 1 -- 172.21.15.67:0/2133032983 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f08d40acb60 con 0x7f08e40b1730 2024-03-24T00:49:36.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.765+0000 7f08e94a5640 1 --2- 172.21.15.67:0/2133032983 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f08c806bbd0 0x7f08c806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:36.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.765+0000 7f08e94a5640 1 --2- 172.21.15.67:0/2133032983 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f08c806bbd0 0x7f08c806e090 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f08e40b16e0 tx=0x7f08e0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:36.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.769+0000 7f08d9ffb640 1 -- 172.21.15.67:0/2133032983 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f08d407b4a0 con 0x7f08e40b1730 2024-03-24T00:49:36.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.991+0000 7f08eb709640 1 -- 172.21.15.67:0/2133032983 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f08e40cc8d0 con 0x7f08e40b1730 2024-03-24T00:49:36.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.992+0000 7f08d9ffb640 1 -- 172.21.15.67:0/2133032983 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+4975 (secure 0 0 0) 0x7f08d407ada0 con 0x7f08e40b1730 2024-03-24T00:49:36.992 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:36.992 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":99,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7298,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/839277571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":839277571},{"type":"v1","addr":"172.21.15.72:6836","nonce":839277571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":8224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2052380377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2052380377},{"type":"v1","addr":"172.21.15.67:6834","nonce":2052380377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":8239,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2196618204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2196618204},{"type":"v1","addr":"172.21.15.72:6837","nonce":2196618204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:48:54.351575+0000","modified":"2024-03-24T00:49:28.351081+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":122,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9666},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9666":{"gid":9666,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6835/693801918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":693801918},{"type":"v1","addr":"172.21.15.67:6835","nonce":693801918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9666,"qdb_cluster":[9666]},"id":8}]} 2024-03-24T00:49:36.992 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 99 2024-03-24T00:49:36.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.994+0000 7f08eb709640 1 -- 172.21.15.67:0/2133032983 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f08c806bbd0 msgr2=0x7f08c806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:36.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.994+0000 7f08eb709640 1 --2- 172.21.15.67:0/2133032983 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f08c806bbd0 0x7f08c806e090 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f08e40b16e0 tx=0x7f08e0008040 comp rx=0 tx=0).stop 2024-03-24T00:49:36.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.994+0000 7f08eb709640 1 -- 172.21.15.67:0/2133032983 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08e40b1730 msgr2=0x7f08e40ae7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:36.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.994+0000 7f08eb709640 1 --2- 172.21.15.67:0/2133032983 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08e40b1730 0x7f08e40ae7d0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f08d402b950 tx=0x7f08d40046c0 comp rx=0 tx=0).stop 2024-03-24T00:49:36.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.994+0000 7f08eb709640 1 -- 172.21.15.67:0/2133032983 shutdown_connections 2024-03-24T00:49:36.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.994+0000 7f08eb709640 1 --2- 172.21.15.67:0/2133032983 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f08c806bbd0 0x7f08c806e090 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:36.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.994+0000 7f08eb709640 1 --2- 172.21.15.67:0/2133032983 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f08e4161790 0x7f08e41585e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:36.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.994+0000 7f08eb709640 1 --2- 172.21.15.67:0/2133032983 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08e40b1730 0x7f08e40ae7d0 unknown :-1 s=CLOSED pgs=1900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:36.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.994+0000 7f08eb709640 1 --2- 172.21.15.67:0/2133032983 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f08e406b040 0x7f08e40ae290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:36.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.994+0000 7f08eb709640 1 -- 172.21.15.67:0/2133032983 >> 172.21.15.67:0/2133032983 conn(0x7f08e40b5440 msgr2=0x7f08e40b2110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:36.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.994+0000 7f08eb709640 1 -- 172.21.15.67:0/2133032983 shutdown_connections 2024-03-24T00:49:36.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:36.994+0000 7f08eb709640 1 -- 172.21.15.67:0/2133032983 wait complete. 2024-03-24T00:49:37.005 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:49:37.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.106+0000 7f4d93b0a640 1 Processor -- start 2024-03-24T00:49:37.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.106+0000 7f4d93b0a640 1 -- start start 2024-03-24T00:49:37.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.107+0000 7f4d93b0a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4d8c0d0ef0 0x7f4d8c0ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:37.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.107+0000 7f4d93b0a640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d8c0cf640 con 0x7f4d8c15e2a0 2024-03-24T00:49:37.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.107+0000 7f4d93b0a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d8c0cf080 con 0x7f4d8c15dec0 2024-03-24T00:49:37.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.107+0000 7f4d93b0a640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d8c0cee30 con 0x7f4d8c0d0ef0 2024-03-24T00:49:37.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.107+0000 7f4d920a7640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4d8c0d0ef0 0x7f4d8c0ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:37.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.107+0000 7f4d910a5640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4d8c15e2a0 0x7f4d8c0d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56500/0 (socket says 172.21.15.67:56500) 2024-03-24T00:49:37.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.107+0000 7f4d910a5640 1 -- 172.21.15.67:0/3299137246 learned_addr learned my addr 172.21.15.67:0/3299137246 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:49:37.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.107+0000 7f4d920a7640 1 -- 172.21.15.67:0/3299137246 >> v1:172.21.15.72:6789/0 conn(0x7f4d8c15dec0 legacy=0x7f4d8c0d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:37.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.107+0000 7f4d908a4640 1 -- 172.21.15.67:0/3299137246 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2351901481 0 0) 0x7f4d7c001110 con 0x7f4d8c15e2a0 2024-03-24T00:49:37.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.107+0000 7f4d920a7640 1 -- 172.21.15.67:0/3299137246 >> v1:172.21.15.67:6789/0 conn(0x7f4d8c15e2a0 legacy=0x7f4d8c0d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:37.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.107+0000 7f4d920a7640 1 -- 172.21.15.67:0/3299137246 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d8c0cf360 con 0x7f4d8c0d0ef0 2024-03-24T00:49:37.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.108+0000 7f4d920a7640 1 --2- 172.21.15.67:0/3299137246 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d8c0d0ef0 0x7f4d8c0ce630 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f4d8c0cefb0 tx=0x7f4d8402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=588e08d4c794ee01 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:37.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.108+0000 7f4d908a4640 1 -- 172.21.15.67:0/3299137246 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d84039070 con 0x7f4d8c0d0ef0 2024-03-24T00:49:37.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.108+0000 7f4d908a4640 1 -- 172.21.15.67:0/3299137246 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d8402cc50 con 0x7f4d8c0d0ef0 2024-03-24T00:49:37.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.108+0000 7f4d908a4640 1 -- 172.21.15.67:0/3299137246 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d84034440 con 0x7f4d8c0d0ef0 2024-03-24T00:49:37.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.109+0000 7f4d93b0a640 1 -- 172.21.15.67:0/3299137246 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d8c0d0ef0 msgr2=0x7f4d8c0ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:37.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.109+0000 7f4d93b0a640 1 --2- 172.21.15.67:0/3299137246 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d8c0d0ef0 0x7f4d8c0ce630 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f4d8c0cefb0 tx=0x7f4d8402bea0 comp rx=0 tx=0).stop 2024-03-24T00:49:37.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.109+0000 7f4d93b0a640 1 -- 172.21.15.67:0/3299137246 shutdown_connections 2024-03-24T00:49:37.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.109+0000 7f4d93b0a640 1 --2- 172.21.15.67:0/3299137246 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d8c0d0ef0 0x7f4d8c0ce630 unknown :-1 s=CLOSED pgs=1902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:37.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.109+0000 7f4d93b0a640 1 -- 172.21.15.67:0/3299137246 >> 172.21.15.67:0/3299137246 conn(0x7f4d8c06c810 msgr2=0x7f4d8c06cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:37.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.109+0000 7f4d93b0a640 1 -- 172.21.15.67:0/3299137246 shutdown_connections 2024-03-24T00:49:37.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.109+0000 7f4d93b0a640 1 -- 172.21.15.67:0/3299137246 wait complete. 2024-03-24T00:49:37.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.110+0000 7f4d93b0a640 1 Processor -- start 2024-03-24T00:49:37.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.110+0000 7f4d93b0a640 1 -- start start 2024-03-24T00:49:37.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.111+0000 7f4d93b0a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4d8c0d0ef0 0x7f4d8c1e8de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:37.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.111+0000 7f4d93b0a640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4d8c15dec0 0x7f4d8c156f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:37.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.111+0000 7f4d93b0a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d8c15e2a0 0x7f4d8c0b46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:37.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.111+0000 7f4d93b0a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8c162a40 con 0x7f4d8c15e2a0 2024-03-24T00:49:37.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.111+0000 7f4d93b0a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8c1628c0 con 0x7f4d8c0d0ef0 2024-03-24T00:49:37.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.111+0000 7f4d93b0a640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8c0d2120 con 0x7f4d8c15dec0 2024-03-24T00:49:37.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.111+0000 7f4d920a7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d8c15e2a0 0x7f4d8c0b46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:37.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.111+0000 7f4d920a7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d8c15e2a0 0x7f4d8c0b46d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48348/0 (socket says 172.21.15.67:48348) 2024-03-24T00:49:37.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.111+0000 7f4d920a7640 1 -- 172.21.15.67:0/2996565967 learned_addr learned my addr 172.21.15.67:0/2996565967 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:37.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.111+0000 7f4d910a5640 1 --2- 172.21.15.67:0/2996565967 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4d8c15dec0 0x7f4d8c156f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:37.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.111+0000 7f4d918a6640 1 --2- 172.21.15.67:0/2996565967 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4d8c0d0ef0 0x7f4d8c1e8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:37.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.111+0000 7f4d920a7640 1 -- 172.21.15.67:0/2996565967 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4d8c15dec0 msgr2=0x7f4d8c156f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:37.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.111+0000 7f4d920a7640 1 --2- 172.21.15.67:0/2996565967 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4d8c15dec0 0x7f4d8c156f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:37.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.111+0000 7f4d920a7640 1 -- 172.21.15.67:0/2996565967 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4d8c0d0ef0 msgr2=0x7f4d8c1e8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:37.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.111+0000 7f4d920a7640 1 --2- 172.21.15.67:0/2996565967 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4d8c0d0ef0 0x7f4d8c1e8de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:37.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.111+0000 7f4d920a7640 1 -- 172.21.15.67:0/2996565967 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d8c1e7d40 con 0x7f4d8c15e2a0 2024-03-24T00:49:37.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.111+0000 7f4d920a7640 1 --2- 172.21.15.67:0/2996565967 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d8c15e2a0 0x7f4d8c0b46d0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f4d84009e80 tx=0x7f4d84004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:37.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.111+0000 7f4d827fc640 1 -- 172.21.15.67:0/2996565967 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d84039050 con 0x7f4d8c15e2a0 2024-03-24T00:49:37.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.112+0000 7f4d827fc640 1 -- 172.21.15.67:0/2996565967 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d84045070 con 0x7f4d8c15e2a0 2024-03-24T00:49:37.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.112+0000 7f4d93b0a640 1 -- 172.21.15.67:0/2996565967 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d8c1e8020 con 0x7f4d8c15e2a0 2024-03-24T00:49:37.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.112+0000 7f4d827fc640 1 -- 172.21.15.67:0/2996565967 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d8403e610 con 0x7f4d8c15e2a0 2024-03-24T00:49:37.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.112+0000 7f4d93b0a640 1 -- 172.21.15.67:0/2996565967 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d8c1e85e0 con 0x7f4d8c15e2a0 2024-03-24T00:49:37.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.113+0000 7f4d93b0a640 1 -- 172.21.15.67:0/2996565967 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d8c04f400 con 0x7f4d8c15e2a0 2024-03-24T00:49:37.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.113+0000 7f4d827fc640 1 -- 172.21.15.67:0/2996565967 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4d8403e7b0 con 0x7f4d8c15e2a0 2024-03-24T00:49:37.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.114+0000 7f4d827fc640 1 --2- 172.21.15.67:0/2996565967 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4d7006bbd0 0x7f4d7006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:37.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.114+0000 7f4d827fc640 1 -- 172.21.15.67:0/2996565967 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f4d840adad0 con 0x7f4d8c15e2a0 2024-03-24T00:49:37.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.114+0000 7f4d918a6640 1 --2- 172.21.15.67:0/2996565967 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4d7006bbd0 0x7f4d7006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:37.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.114+0000 7f4d918a6640 1 --2- 172.21.15.67:0/2996565967 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4d7006bbd0 0x7f4d7006e090 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f4d880015a0 tx=0x7f4d88009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:37.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.118+0000 7f4d827fc640 1 -- 172.21.15.67:0/2996565967 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4d84032e20 con 0x7f4d8c15e2a0 2024-03-24T00:49:37.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.302+0000 7f4d93b0a640 1 -- 172.21.15.67:0/2996565967 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4d8c0b9790 con 0x7f4d8c15e2a0 2024-03-24T00:49:37.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.303+0000 7f4d827fc640 1 -- 172.21.15.67:0/2996565967 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v122) v1 ==== 74+0+17149 (secure 0 0 0) 0x7f4d8407c530 con 0x7f4d8c15e2a0 2024-03-24T00:49:37.303 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:37.303 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":122,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:49:23.316053+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:48:52.622356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-03-24T00:48:53.311452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6835/3833808901":"2024-03-25T00:49:23.316030+0000","172.21.15.67:6833/3833808901":"2024-03-25T00:49:23.316030+0000","172.21.15.67:6835/312063805":"2024-03-25T00:49:12.456364+0000","172.21.15.67:6833/312063805":"2024-03-25T00:49:12.456364+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:49:37.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.305+0000 7f4d93b0a640 1 -- 172.21.15.67:0/2996565967 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4d7006bbd0 msgr2=0x7f4d7006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:37.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.305+0000 7f4d93b0a640 1 --2- 172.21.15.67:0/2996565967 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4d7006bbd0 0x7f4d7006e090 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f4d880015a0 tx=0x7f4d88009040 comp rx=0 tx=0).stop 2024-03-24T00:49:37.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.305+0000 7f4d93b0a640 1 -- 172.21.15.67:0/2996565967 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d8c15e2a0 msgr2=0x7f4d8c0b46d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:37.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.305+0000 7f4d93b0a640 1 --2- 172.21.15.67:0/2996565967 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d8c15e2a0 0x7f4d8c0b46d0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f4d84009e80 tx=0x7f4d84004a40 comp rx=0 tx=0).stop 2024-03-24T00:49:37.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.305+0000 7f4d93b0a640 1 -- 172.21.15.67:0/2996565967 shutdown_connections 2024-03-24T00:49:37.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.305+0000 7f4d93b0a640 1 --2- 172.21.15.67:0/2996565967 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f4d7006bbd0 0x7f4d7006e090 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:37.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.305+0000 7f4d93b0a640 1 --2- 172.21.15.67:0/2996565967 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d8c15e2a0 0x7f4d8c0b46d0 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:37.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.305+0000 7f4d93b0a640 1 --2- 172.21.15.67:0/2996565967 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4d8c15dec0 0x7f4d8c156f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:37.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.305+0000 7f4d93b0a640 1 --2- 172.21.15.67:0/2996565967 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4d8c0d0ef0 0x7f4d8c1e8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:37.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.305+0000 7f4d93b0a640 1 -- 172.21.15.67:0/2996565967 >> 172.21.15.67:0/2996565967 conn(0x7f4d8c06c810 msgr2=0x7f4d8c0ce420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:37.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.305+0000 7f4d93b0a640 1 -- 172.21.15.67:0/2996565967 shutdown_connections 2024-03-24T00:49:37.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.305+0000 7f4d93b0a640 1 -- 172.21.15.67:0/2996565967 wait complete. 2024-03-24T00:49:37.316 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-24T00:49:37.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.422+0000 7ffac42a1640 1 Processor -- start 2024-03-24T00:49:37.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.422+0000 7ffac42a1640 1 -- start start 2024-03-24T00:49:37.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.423+0000 7ffac42a1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ffabc0d08f0 0x7ffabc166b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:37.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.423+0000 7ffac42a1640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffabc0cce30 con 0x7ffabc099d00 2024-03-24T00:49:37.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.423+0000 7ffac42a1640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffabc0cd110 con 0x7ffabc0cc7b0 2024-03-24T00:49:37.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.423+0000 7ffac42a1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffabc0cda40 con 0x7ffabc0d08f0 2024-03-24T00:49:37.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.423+0000 7ffac203d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ffabc099d00 0x7ffabc15ce90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56510/0 (socket says 172.21.15.67:56510) 2024-03-24T00:49:37.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.423+0000 7ffac203d640 1 -- 172.21.15.67:0/3328737397 learned_addr learned my addr 172.21.15.67:0/3328737397 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:49:37.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.423+0000 7ffac283e640 1 --2- 172.21.15.67:0/3328737397 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ffabc0d08f0 0x7ffabc166b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:37.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.423+0000 7ffac103b640 1 -- 172.21.15.67:0/3328737397 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4117810344 0 0) 0x7ffabc0cce30 con 0x7ffabc099d00 2024-03-24T00:49:37.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.423+0000 7ffac103b640 1 -- 172.21.15.67:0/3328737397 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaa8003520 con 0x7ffabc099d00 2024-03-24T00:49:37.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.423+0000 7ffac103b640 1 -- 172.21.15.67:0/3328737397 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2496334583 0 0) 0x7ffaa8003520 con 0x7ffabc099d00 2024-03-24T00:49:37.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.423+0000 7ffac103b640 1 -- 172.21.15.67:0/3328737397 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ffabc0d08f0 msgr2=0x7ffabc166b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:37.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.423+0000 7ffac103b640 1 --2- 172.21.15.67:0/3328737397 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ffabc0d08f0 0x7ffabc166b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:37.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.423+0000 7ffac103b640 1 -- 172.21.15.67:0/3328737397 >> v1:172.21.15.72:6789/0 conn(0x7ffabc0cc7b0 legacy=0x7ffabc09a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:37.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.423+0000 7ffac103b640 1 -- 172.21.15.67:0/3328737397 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffabc0cd3f0 con 0x7ffabc099d00 2024-03-24T00:49:37.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.424+0000 7ffac103b640 1 -- 172.21.15.67:0/3328737397 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7ffab8001670 con 0x7ffabc099d00 2024-03-24T00:49:37.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.424+0000 7ffac103b640 1 -- 172.21.15.67:0/3328737397 >> v1:172.21.15.67:6789/0 conn(0x7ffabc099d00 legacy=0x7ffabc15ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:37.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.424+0000 7ffac103b640 1 --2- 172.21.15.67:0/3328737397 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ffabc0d08f0 0x7ffaa80039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:37.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.424+0000 7ffac103b640 1 --2- 172.21.15.67:0/3328737397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffaa8003f20 0x7ffaa8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:37.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.424+0000 7ffac103b640 1 --2- 172.21.15.67:0/3328737397 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffaa8004d40 0x7ffaa80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:37.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.424+0000 7ffac103b640 1 -- 172.21.15.67:0/3328737397 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffabc0cda40 con 0x7ffaa8003f20 2024-03-24T00:49:37.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.424+0000 7ffac103b640 1 -- 172.21.15.67:0/3328737397 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa800ba20 con 0x7ffaa8004d40 2024-03-24T00:49:37.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.424+0000 7ffac103b640 1 -- 172.21.15.67:0/3328737397 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa800bc20 con 0x7ffabc0d08f0 2024-03-24T00:49:37.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.424+0000 7ffac183c640 1 --2- 172.21.15.67:0/3328737397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffaa8003f20 0x7ffaa8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:37.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.424+0000 7ffac203d640 1 --2- 172.21.15.67:0/3328737397 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ffabc0d08f0 0x7ffaa80039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:37.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.424+0000 7ffac283e640 1 --2- 172.21.15.67:0/3328737397 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffaa8004d40 0x7ffaa80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:37.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.424+0000 7ffac183c640 1 -- 172.21.15.67:0/3328737397 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ffabc0d08f0 msgr2=0x7ffaa80039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:37.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.424+0000 7ffac183c640 1 --2- 172.21.15.67:0/3328737397 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ffabc0d08f0 0x7ffaa80039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:37.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.424+0000 7ffac183c640 1 -- 172.21.15.67:0/3328737397 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffaa8004d40 msgr2=0x7ffaa80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:37.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.424+0000 7ffac183c640 1 --2- 172.21.15.67:0/3328737397 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffaa8004d40 0x7ffaa80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:37.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.424+0000 7ffac183c640 1 -- 172.21.15.67:0/3328737397 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaa8003520 con 0x7ffaa8003f20 2024-03-24T00:49:37.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.425+0000 7ffac183c640 1 --2- 172.21.15.67:0/3328737397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffaa8003f20 0x7ffaa8004800 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7ffaac001860 tx=0x7ffaa8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75bc891e82abff54 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:37.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.425+0000 7ffac103b640 1 -- 172.21.15.67:0/3328737397 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaac001430 con 0x7ffaa8003f20 2024-03-24T00:49:37.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.425+0000 7ffac103b640 1 -- 172.21.15.67:0/3328737397 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaac032620 con 0x7ffaa8003f20 2024-03-24T00:49:37.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.425+0000 7ffac103b640 1 -- 172.21.15.67:0/3328737397 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaac032a10 con 0x7ffaa8003f20 2024-03-24T00:49:37.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.425+0000 7ffac42a1640 1 -- 172.21.15.67:0/3328737397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffaa8003f20 msgr2=0x7ffaa8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:37.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.425+0000 7ffac42a1640 1 --2- 172.21.15.67:0/3328737397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffaa8003f20 0x7ffaa8004800 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7ffaac001860 tx=0x7ffaa8005830 comp rx=0 tx=0).stop 2024-03-24T00:49:37.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.425+0000 7ffac283e640 1 -- 172.21.15.67:0/3328737397 reap_dead start 2024-03-24T00:49:37.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.426+0000 7ffac42a1640 1 -- 172.21.15.67:0/3328737397 shutdown_connections 2024-03-24T00:49:37.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.426+0000 7ffac42a1640 1 -- 172.21.15.67:0/3328737397 >> 172.21.15.67:0/3328737397 conn(0x7ffabc0b3650 msgr2=0x7ffabc0b3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:37.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.426+0000 7ffac42a1640 1 -- 172.21.15.67:0/3328737397 shutdown_connections 2024-03-24T00:49:37.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.426+0000 7ffac42a1640 1 -- 172.21.15.67:0/3328737397 wait complete. 2024-03-24T00:49:37.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.427+0000 7ffac42a1640 1 Processor -- start 2024-03-24T00:49:37.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.427+0000 7ffac42a1640 1 -- start start 2024-03-24T00:49:37.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.427+0000 7ffac42a1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ffaa8003f20 0x7ffabc15b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:37.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.427+0000 7ffac42a1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffabc06c150 0x7ffabc06d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:37.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.427+0000 7ffac42a1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffabc152b70 0x7ffabc152fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:37.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.427+0000 7ffac42a1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffabc159e40 con 0x7ffabc06c150 2024-03-24T00:49:37.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.427+0000 7ffac42a1640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffabc159cc0 con 0x7ffabc152b70 2024-03-24T00:49:37.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.427+0000 7ffac42a1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffabc164b50 con 0x7ffaa8003f20 2024-03-24T00:49:37.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.428+0000 7ffac183c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffabc06c150 0x7ffabc06d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:37.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.428+0000 7ffac183c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffabc06c150 0x7ffabc06d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48362/0 (socket says 172.21.15.67:48362) 2024-03-24T00:49:37.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.428+0000 7ffac183c640 1 -- 172.21.15.67:0/2353696334 learned_addr learned my addr 172.21.15.67:0/2353696334 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:37.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.428+0000 7ffac183c640 1 -- 172.21.15.67:0/2353696334 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ffaa8003f20 msgr2=0x7ffabc15b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:37.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.428+0000 7ffac183c640 1 --2- 172.21.15.67:0/2353696334 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ffaa8003f20 0x7ffabc15b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:37.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.428+0000 7ffac183c640 1 -- 172.21.15.67:0/2353696334 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffabc152b70 msgr2=0x7ffabc152fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:37.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.428+0000 7ffac183c640 1 --2- 172.21.15.67:0/2353696334 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffabc152b70 0x7ffabc152fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:37.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.428+0000 7ffac183c640 1 -- 172.21.15.67:0/2353696334 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffabc15ad40 con 0x7ffabc06c150 2024-03-24T00:49:37.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.428+0000 7ffac183c640 1 --2- 172.21.15.67:0/2353696334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffabc06c150 0x7ffabc06d4f0 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7ffaac008fd0 tx=0x7ffaac001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:37.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.428+0000 7ffab2ffd640 1 -- 172.21.15.67:0/2353696334 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaac007050 con 0x7ffabc06c150 2024-03-24T00:49:37.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.428+0000 7ffab2ffd640 1 -- 172.21.15.67:0/2353696334 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaac0071f0 con 0x7ffabc06c150 2024-03-24T00:49:37.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.428+0000 7ffac42a1640 1 -- 172.21.15.67:0/2353696334 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffabc06c6a0 con 0x7ffabc06c150 2024-03-24T00:49:37.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.428+0000 7ffab2ffd640 1 -- 172.21.15.67:0/2353696334 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaac0075c0 con 0x7ffabc06c150 2024-03-24T00:49:37.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.428+0000 7ffac42a1640 1 -- 172.21.15.67:0/2353696334 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffabc06cc60 con 0x7ffabc06c150 2024-03-24T00:49:37.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.429+0000 7ffac42a1640 1 -- 172.21.15.67:0/2353696334 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffabc04f400 con 0x7ffabc06c150 2024-03-24T00:49:37.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.430+0000 7ffab2ffd640 1 -- 172.21.15.67:0/2353696334 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ffaac0067d0 con 0x7ffabc06c150 2024-03-24T00:49:37.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.430+0000 7ffab2ffd640 1 --2- 172.21.15.67:0/2353696334 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ffaa006bc00 0x7ffaa006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:37.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.430+0000 7ffab2ffd640 1 -- 172.21.15.67:0/2353696334 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6273+0+0 (secure 0 0 0) 0x7ffaac0ac9b0 con 0x7ffabc06c150 2024-03-24T00:49:37.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.430+0000 7ffac203d640 1 --2- 172.21.15.67:0/2353696334 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ffaa006bc00 0x7ffaa006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:37.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.431+0000 7ffac203d640 1 --2- 172.21.15.67:0/2353696334 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ffaa006bc00 0x7ffaa006e0c0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7ffab80011e0 tx=0x7ffab8001170 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:37.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.435+0000 7ffab2ffd640 1 -- 172.21.15.67:0/2353696334 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ffaac040020 con 0x7ffabc06c150 2024-03-24T00:49:37.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.656+0000 7ffac42a1640 1 -- 172.21.15.67:0/2353696334 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7ffabc159fc0 con 0x7ffabc06c150 2024-03-24T00:49:37.656 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-24T00:49:37.657+0000 7fbfcfa11640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T00:49:37.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.658+0000 7ffab2ffd640 1 -- 172.21.15.67:0/2353696334 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v100) v1 ==== 161+0+0 (secure 0 0 0) 0x7ffaac07b260 con 0x7ffabc06c150 2024-03-24T00:49:37.658 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-24T00:49:37.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.661+0000 7ffac42a1640 1 -- 172.21.15.67:0/2353696334 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ffaa006bc00 msgr2=0x7ffaa006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:37.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.661+0000 7ffac42a1640 1 --2- 172.21.15.67:0/2353696334 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ffaa006bc00 0x7ffaa006e0c0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7ffab80011e0 tx=0x7ffab8001170 comp rx=0 tx=0).stop 2024-03-24T00:49:37.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.661+0000 7ffac42a1640 1 -- 172.21.15.67:0/2353696334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffabc06c150 msgr2=0x7ffabc06d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:37.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.661+0000 7ffac42a1640 1 --2- 172.21.15.67:0/2353696334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffabc06c150 0x7ffabc06d4f0 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7ffaac008fd0 tx=0x7ffaac001230 comp rx=0 tx=0).stop 2024-03-24T00:49:37.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.661+0000 7ffac42a1640 1 -- 172.21.15.67:0/2353696334 shutdown_connections 2024-03-24T00:49:37.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.661+0000 7ffac42a1640 1 --2- 172.21.15.67:0/2353696334 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ffaa006bc00 0x7ffaa006e0c0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:37.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.661+0000 7ffac42a1640 1 --2- 172.21.15.67:0/2353696334 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffabc152b70 0x7ffabc152fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:37.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.661+0000 7ffac42a1640 1 --2- 172.21.15.67:0/2353696334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffabc06c150 0x7ffabc06d4f0 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:37.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.661+0000 7ffac42a1640 1 --2- 172.21.15.67:0/2353696334 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ffaa8003f20 0x7ffabc15b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:37.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.661+0000 7ffac42a1640 1 -- 172.21.15.67:0/2353696334 >> 172.21.15.67:0/2353696334 conn(0x7ffabc0b3650 msgr2=0x7ffabc157100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:37.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.661+0000 7ffac42a1640 1 -- 172.21.15.67:0/2353696334 shutdown_connections 2024-03-24T00:49:37.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.661+0000 7ffac42a1640 1 -- 172.21.15.67:0/2353696334 wait complete. 2024-03-24T00:49:37.662 INFO:tasks.ceph.mds.a.smithi067.stderr: -1195> 2024-03-24T00:49:29.325+0000 7fe4f3a82640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T00:49:37.673 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-24T00:49:37.689 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-03-24T00:49:37.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.777+0000 7f29544dd640 1 Processor -- start 2024-03-24T00:49:37.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.777+0000 7f29544dd640 1 -- start start 2024-03-24T00:49:37.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.778+0000 7f29544dd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f294c0cf120 0x7f294c0b96d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:37.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.778+0000 7f29544dd640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f294c0d23d0 con 0x7f294c15de50 2024-03-24T00:49:37.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.778+0000 7f29544dd640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f294c0d26b0 con 0x7f294c15e230 2024-03-24T00:49:37.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.778+0000 7f29544dd640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f294c0b9c10 con 0x7f294c0cf120 2024-03-24T00:49:37.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.778+0000 7f2952a7a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f294c0cf120 0x7f294c0b96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:37.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.778+0000 7f2952279640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f294c15de50 0x7f294c0cea10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56534/0 (socket says 172.21.15.67:56534) 2024-03-24T00:49:37.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.778+0000 7f2952279640 1 -- 172.21.15.67:0/2293781139 learned_addr learned my addr 172.21.15.67:0/2293781139 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:49:37.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.779+0000 7f2952a7a640 1 -- 172.21.15.67:0/2293781139 >> v1:172.21.15.72:6789/0 conn(0x7f294c15e230 legacy=0x7f294c0b6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:37.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.779+0000 7f2952a7a640 1 -- 172.21.15.67:0/2293781139 >> v1:172.21.15.67:6789/0 conn(0x7f294c15de50 legacy=0x7f294c0cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:37.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.779+0000 7f2952a7a640 1 -- 172.21.15.67:0/2293781139 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f294c0d2990 con 0x7f294c0cf120 2024-03-24T00:49:37.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.779+0000 7f2951277640 1 -- 172.21.15.67:0/2293781139 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3254000757 0 0) 0x7f294c0d23d0 con 0x7f294c15de50 2024-03-24T00:49:37.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.779+0000 7f2952a7a640 1 --2- 172.21.15.67:0/2293781139 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f294c0cf120 0x7f294c0b96d0 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f2944009d90 tx=0x7f294402bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc30d9b8658c2da7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:37.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.779+0000 7f2951277640 1 -- 172.21.15.67:0/2293781139 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2944039070 con 0x7f294c0cf120 2024-03-24T00:49:37.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.780+0000 7f2951277640 1 -- 172.21.15.67:0/2293781139 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f294402cd80 con 0x7f294c0cf120 2024-03-24T00:49:37.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.780+0000 7f2951277640 1 -- 172.21.15.67:0/2293781139 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2944034660 con 0x7f294c0cf120 2024-03-24T00:49:37.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.780+0000 7f29544dd640 1 -- 172.21.15.67:0/2293781139 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f294c0cf120 msgr2=0x7f294c0b96d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:37.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.780+0000 7f29544dd640 1 --2- 172.21.15.67:0/2293781139 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f294c0cf120 0x7f294c0b96d0 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f2944009d90 tx=0x7f294402bfd0 comp rx=0 tx=0).stop 2024-03-24T00:49:37.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.780+0000 7f29544dd640 1 -- 172.21.15.67:0/2293781139 shutdown_connections 2024-03-24T00:49:37.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.780+0000 7f29544dd640 1 --2- 172.21.15.67:0/2293781139 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f294c0cf120 0x7f294c0b96d0 unknown :-1 s=CLOSED pgs=1910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:37.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.780+0000 7f29544dd640 1 -- 172.21.15.67:0/2293781139 >> 172.21.15.67:0/2293781139 conn(0x7f294c072b60 msgr2=0x7f294c072f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:37.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.780+0000 7f29544dd640 1 -- 172.21.15.67:0/2293781139 shutdown_connections 2024-03-24T00:49:37.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.780+0000 7f29544dd640 1 -- 172.21.15.67:0/2293781139 wait complete. 2024-03-24T00:49:37.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.781+0000 7f29544dd640 1 Processor -- start 2024-03-24T00:49:37.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.781+0000 7f29544dd640 1 -- start start 2024-03-24T00:49:37.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.782+0000 7f29544dd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f294c0cf120 0x7f294c0cb950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:37.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.782+0000 7f29544dd640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f294c15de50 0x7f294c14a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:37.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.782+0000 7f29544dd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f294c15e230 0x7f294c0b44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:37.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.782+0000 7f29544dd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f294c0d3930 con 0x7f294c15e230 2024-03-24T00:49:37.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.782+0000 7f29544dd640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f294c0d37b0 con 0x7f294c0cf120 2024-03-24T00:49:37.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.782+0000 7f29544dd640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f294c0d3ab0 con 0x7f294c15de50 2024-03-24T00:49:37.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.782+0000 7f2952a7a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f294c15e230 0x7f294c0b44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:37.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.782+0000 7f2952a7a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f294c15e230 0x7f294c0b44c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48380/0 (socket says 172.21.15.67:48380) 2024-03-24T00:49:37.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.782+0000 7f2952a7a640 1 -- 172.21.15.67:0/3568388466 learned_addr learned my addr 172.21.15.67:0/3568388466 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:37.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.782+0000 7f2952279640 1 --2- 172.21.15.67:0/3568388466 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f294c0cf120 0x7f294c0cb950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:37.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.782+0000 7f2951a78640 1 --2- 172.21.15.67:0/3568388466 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f294c15de50 0x7f294c14a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:37.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.783+0000 7f2952a7a640 1 -- 172.21.15.67:0/3568388466 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f294c15de50 msgr2=0x7f294c14a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:37.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.783+0000 7f2952a7a640 1 --2- 172.21.15.67:0/3568388466 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f294c15de50 0x7f294c14a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:37.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.783+0000 7f2952a7a640 1 -- 172.21.15.67:0/3568388466 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f294c0cf120 msgr2=0x7f294c0cb950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:37.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.783+0000 7f2952a7a640 1 --2- 172.21.15.67:0/3568388466 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f294c0cf120 0x7f294c0cb950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:37.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.783+0000 7f2952a7a640 1 -- 172.21.15.67:0/3568388466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f294c14aa20 con 0x7f294c15e230 2024-03-24T00:49:37.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.783+0000 7f2952a7a640 1 --2- 172.21.15.67:0/3568388466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f294c15e230 0x7f294c0b44c0 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f2944009d90 tx=0x7f2944004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:37.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.783+0000 7f2942ffd640 1 -- 172.21.15.67:0/3568388466 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2944039050 con 0x7f294c15e230 2024-03-24T00:49:37.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.783+0000 7f2942ffd640 1 -- 172.21.15.67:0/3568388466 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f294403a040 con 0x7f294c15e230 2024-03-24T00:49:37.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.783+0000 7f29544dd640 1 -- 172.21.15.67:0/3568388466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f294c1e6910 con 0x7f294c15e230 2024-03-24T00:49:37.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.783+0000 7f2942ffd640 1 -- 172.21.15.67:0/3568388466 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f294403e610 con 0x7f294c15e230 2024-03-24T00:49:37.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.783+0000 7f29544dd640 1 -- 172.21.15.67:0/3568388466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f294c14a630 con 0x7f294c15e230 2024-03-24T00:49:37.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.784+0000 7f29544dd640 1 -- 172.21.15.67:0/3568388466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f294c04f400 con 0x7f294c15e230 2024-03-24T00:49:37.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.784+0000 7f2942ffd640 1 -- 172.21.15.67:0/3568388466 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2944047020 con 0x7f294c15e230 2024-03-24T00:49:37.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.785+0000 7f2942ffd640 1 --2- 172.21.15.67:0/3568388466 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f293006bbd0 0x7f293006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:37.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.785+0000 7f2942ffd640 1 -- 172.21.15.67:0/3568388466 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6359+0+0 (secure 0 0 0) 0x7f29440aced0 con 0x7f294c15e230 2024-03-24T00:49:37.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.785+0000 7f2952279640 1 --2- 172.21.15.67:0/3568388466 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f293006bbd0 0x7f293006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:37.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.785+0000 7f2952279640 1 --2- 172.21.15.67:0/3568388466 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f293006bbd0 0x7f293006e090 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f293c007f70 tx=0x7f293c007f00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:37.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:37.789+0000 7f2942ffd640 1 -- 172.21.15.67:0/3568388466 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f294404e020 con 0x7f294c15e230 2024-03-24T00:49:38.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.012+0000 7f29544dd640 1 -- 172.21.15.67:0/3568388466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f294c0d4880 con 0x7f294c15e230 2024-03-24T00:49:38.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.659+0000 7f2942ffd640 1 -- 172.21.15.67:0/3568388466 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v101) v1 ==== 104+0+0 (secure 0 0 0) 0x7f294407b230 con 0x7f294c15e230 2024-03-24T00:49:38.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.662+0000 7f29544dd640 1 -- 172.21.15.67:0/3568388466 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f293006bbd0 msgr2=0x7f293006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:38.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.662+0000 7f29544dd640 1 --2- 172.21.15.67:0/3568388466 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f293006bbd0 0x7f293006e090 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f293c007f70 tx=0x7f293c007f00 comp rx=0 tx=0).stop 2024-03-24T00:49:38.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.662+0000 7f29544dd640 1 -- 172.21.15.67:0/3568388466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f294c15e230 msgr2=0x7f294c0b44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:38.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.662+0000 7f29544dd640 1 --2- 172.21.15.67:0/3568388466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f294c15e230 0x7f294c0b44c0 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f2944009d90 tx=0x7f2944004a40 comp rx=0 tx=0).stop 2024-03-24T00:49:38.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.662+0000 7f29544dd640 1 -- 172.21.15.67:0/3568388466 shutdown_connections 2024-03-24T00:49:38.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.662+0000 7f29544dd640 1 --2- 172.21.15.67:0/3568388466 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f293006bbd0 0x7f293006e090 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:38.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.662+0000 7f29544dd640 1 --2- 172.21.15.67:0/3568388466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f294c15e230 0x7f294c0b44c0 unknown :-1 s=CLOSED pgs=1911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:38.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.662+0000 7f29544dd640 1 --2- 172.21.15.67:0/3568388466 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f294c15de50 0x7f294c14a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:38.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.662+0000 7f29544dd640 1 --2- 172.21.15.67:0/3568388466 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f294c0cf120 0x7f294c0cb950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:38.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.662+0000 7f29544dd640 1 -- 172.21.15.67:0/3568388466 >> 172.21.15.67:0/3568388466 conn(0x7f294c072b60 msgr2=0x7f294c0ce6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:38.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.662+0000 7f29544dd640 1 -- 172.21.15.67:0/3568388466 shutdown_connections 2024-03-24T00:49:38.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.662+0000 7f29544dd640 1 -- 172.21.15.67:0/3568388466 wait complete. 2024-03-24T00:49:38.675 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-24T00:49:38.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.825+0000 7f52873f4640 1 Processor -- start 2024-03-24T00:49:38.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.825+0000 7f52873f4640 1 -- start start 2024-03-24T00:49:38.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.826+0000 7f52873f4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f528015dfc0 0x7f52801624c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:38.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.826+0000 7f52873f4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f52800d0d50 0x7f52800d2b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:38.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.826+0000 7f52873f4640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5280162a00 con 0x7f52800ced60 2024-03-24T00:49:38.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.826+0000 7f52873f4640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f528015e3a0 con 0x7f528015dfc0 2024-03-24T00:49:38.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.826+0000 7f52873f4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52800cf4d0 con 0x7f52800d0d50 2024-03-24T00:49:38.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.826+0000 7f5285190640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f528015dfc0 0x7f52801624c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:38.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.826+0000 7f528498f640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f52800ced60 0x7f52800d0640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56546/0 (socket says 172.21.15.67:56546) 2024-03-24T00:49:38.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.826+0000 7f528498f640 1 -- 172.21.15.67:0/274814587 learned_addr learned my addr 172.21.15.67:0/274814587 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:49:38.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.826+0000 7f5285991640 1 --2- 172.21.15.67:0/274814587 >> v2:172.21.15.72:3300/0 conn(0x7f52800d0d50 0x7f52800d2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:38.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.827+0000 7f5285190640 1 -- 172.21.15.67:0/274814587 >> v2:172.21.15.72:3300/0 conn(0x7f52800d0d50 msgr2=0x7f52800d2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:38.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.827+0000 7f5285190640 1 --2- 172.21.15.67:0/274814587 >> v2:172.21.15.72:3300/0 conn(0x7f52800d0d50 0x7f52800d2b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:38.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.827+0000 7f5285190640 1 -- 172.21.15.67:0/274814587 >> v1:172.21.15.67:6789/0 conn(0x7f52800ced60 legacy=0x7f52800d0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:38.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.827+0000 7f527ffff640 1 -- 172.21.15.67:0/274814587 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3070265021 0 0) 0x7f5280162a00 con 0x7f52800ced60 2024-03-24T00:49:38.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.827+0000 7f5285190640 1 -- 172.21.15.67:0/274814587 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52800cf1f0 con 0x7f528015dfc0 2024-03-24T00:49:38.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.827+0000 7f5285190640 1 --2- 172.21.15.67:0/274814587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f528015dfc0 0x7f52801624c0 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f52800d0250 tx=0x7f527802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34a8409351aede73 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:38.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.828+0000 7f527ffff640 1 -- 172.21.15.67:0/274814587 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5278035020 con 0x7f528015dfc0 2024-03-24T00:49:38.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.828+0000 7f527ffff640 1 -- 172.21.15.67:0/274814587 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5278031830 con 0x7f528015dfc0 2024-03-24T00:49:38.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.828+0000 7f527ffff640 1 -- 172.21.15.67:0/274814587 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5278031c00 con 0x7f528015dfc0 2024-03-24T00:49:38.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.828+0000 7f52873f4640 1 -- 172.21.15.67:0/274814587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f528015dfc0 msgr2=0x7f52801624c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:38.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.828+0000 7f52873f4640 1 --2- 172.21.15.67:0/274814587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f528015dfc0 0x7f52801624c0 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f52800d0250 tx=0x7f527802b8d0 comp rx=0 tx=0).stop 2024-03-24T00:49:38.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.828+0000 7f52873f4640 1 -- 172.21.15.67:0/274814587 shutdown_connections 2024-03-24T00:49:38.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.828+0000 7f52873f4640 1 --2- 172.21.15.67:0/274814587 >> v2:172.21.15.72:3300/0 conn(0x7f52800d0d50 0x7f52800d2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:38.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.828+0000 7f52873f4640 1 --2- 172.21.15.67:0/274814587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f528015dfc0 0x7f52801624c0 unknown :-1 s=CLOSED pgs=1913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:38.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.828+0000 7f52873f4640 1 -- 172.21.15.67:0/274814587 >> 172.21.15.67:0/274814587 conn(0x7f528006c880 msgr2=0x7f528006cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:38.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.828+0000 7f52873f4640 1 -- 172.21.15.67:0/274814587 shutdown_connections 2024-03-24T00:49:38.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.828+0000 7f52873f4640 1 -- 172.21.15.67:0/274814587 wait complete. 2024-03-24T00:49:38.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.829+0000 7f52873f4640 1 Processor -- start 2024-03-24T00:49:38.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.829+0000 7f52873f4640 1 -- start start 2024-03-24T00:49:38.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.830+0000 7f52873f4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f52800d0d50 0x7f52800c8a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:38.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.830+0000 7f52873f4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f528015dfc0 0x7f528015f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:38.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.830+0000 7f52873f4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52800c8f50 0x7f528015feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:38.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.830+0000 7f52873f4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52800d25e0 con 0x7f52800c8f50 2024-03-24T00:49:38.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.830+0000 7f52873f4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52800d2300 con 0x7f52800d0d50 2024-03-24T00:49:38.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.830+0000 7f52873f4640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52800b6ea0 con 0x7f528015dfc0 2024-03-24T00:49:38.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.830+0000 7f5285991640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52800c8f50 0x7f528015feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:38.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.830+0000 7f5285991640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52800c8f50 0x7f528015feb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48400/0 (socket says 172.21.15.67:48400) 2024-03-24T00:49:38.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.830+0000 7f5285991640 1 -- 172.21.15.67:0/3689923488 learned_addr learned my addr 172.21.15.67:0/3689923488 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:38.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.830+0000 7f528498f640 1 --2- 172.21.15.67:0/3689923488 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f528015dfc0 0x7f528015f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:38.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.830+0000 7f5285190640 1 --2- 172.21.15.67:0/3689923488 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f52800d0d50 0x7f52800c8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:38.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.831+0000 7f5285991640 1 -- 172.21.15.67:0/3689923488 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f528015dfc0 msgr2=0x7f528015f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:38.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.831+0000 7f5285991640 1 --2- 172.21.15.67:0/3689923488 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f528015dfc0 0x7f528015f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:38.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.831+0000 7f5285991640 1 -- 172.21.15.67:0/3689923488 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f52800d0d50 msgr2=0x7f52800c8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:38.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.831+0000 7f5285991640 1 --2- 172.21.15.67:0/3689923488 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f52800d0d50 0x7f52800c8a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:38.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.831+0000 7f5285991640 1 -- 172.21.15.67:0/3689923488 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52800c9420 con 0x7f52800c8f50 2024-03-24T00:49:38.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.831+0000 7f5285991640 1 --2- 172.21.15.67:0/3689923488 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52800c8f50 0x7f528015feb0 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f5270007de0 tx=0x7f527000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:38.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.831+0000 7f527dffb640 1 -- 172.21.15.67:0/3689923488 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5270014620 con 0x7f52800c8f50 2024-03-24T00:49:38.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.831+0000 7f527dffb640 1 -- 172.21.15.67:0/3689923488 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5270019070 con 0x7f52800c8f50 2024-03-24T00:49:38.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.831+0000 7f527dffb640 1 -- 172.21.15.67:0/3689923488 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5270014a20 con 0x7f52800c8f50 2024-03-24T00:49:38.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.831+0000 7f52873f4640 1 -- 172.21.15.67:0/3689923488 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52800c9700 con 0x7f52800c8f50 2024-03-24T00:49:38.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.831+0000 7f52873f4640 1 -- 172.21.15.67:0/3689923488 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52800b38b0 con 0x7f52800c8f50 2024-03-24T00:49:38.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.832+0000 7f52873f4640 1 -- 172.21.15.67:0/3689923488 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f528004f400 con 0x7f52800c8f50 2024-03-24T00:49:38.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.833+0000 7f527dffb640 1 -- 172.21.15.67:0/3689923488 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5270014bc0 con 0x7f52800c8f50 2024-03-24T00:49:38.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.833+0000 7f527dffb640 1 --2- 172.21.15.67:0/3689923488 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f526c06bbd0 0x7f526c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:38.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.833+0000 7f527dffb640 1 -- 172.21.15.67:0/3689923488 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6359+0+0 (secure 0 0 0) 0x7f527008b6b0 con 0x7f52800c8f50 2024-03-24T00:49:38.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.833+0000 7f5285190640 1 --2- 172.21.15.67:0/3689923488 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f526c06bbd0 0x7f526c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:38.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.834+0000 7f5285190640 1 --2- 172.21.15.67:0/3689923488 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f526c06bbd0 0x7f526c06e090 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f5278036000 tx=0x7f5278008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:38.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:38.837+0000 7f527dffb640 1 -- 172.21.15.67:0/3689923488 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5270059f10 con 0x7f52800c8f50 2024-03-24T00:49:39.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.018+0000 7f52873f4640 1 -- 172.21.15.67:0/3689923488 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5280159a30 con 0x7f52800c8f50 2024-03-24T00:49:39.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.665+0000 7f527dffb640 1 -- 172.21.15.67:0/3689923488 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v124) v1 ==== 181+0+0 (secure 0 0 0) 0x7f5270059810 con 0x7f52800c8f50 2024-03-24T00:49:39.665 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' removed 2024-03-24T00:49:39.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.668+0000 7f52873f4640 1 -- 172.21.15.67:0/3689923488 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f526c06bbd0 msgr2=0x7f526c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:39.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.668+0000 7f52873f4640 1 --2- 172.21.15.67:0/3689923488 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f526c06bbd0 0x7f526c06e090 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f5278036000 tx=0x7f5278008030 comp rx=0 tx=0).stop 2024-03-24T00:49:39.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.668+0000 7f52873f4640 1 -- 172.21.15.67:0/3689923488 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52800c8f50 msgr2=0x7f528015feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:39.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.668+0000 7f52873f4640 1 --2- 172.21.15.67:0/3689923488 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52800c8f50 0x7f528015feb0 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f5270007de0 tx=0x7f527000e4b0 comp rx=0 tx=0).stop 2024-03-24T00:49:39.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.670+0000 7f52873f4640 1 -- 172.21.15.67:0/3689923488 shutdown_connections 2024-03-24T00:49:39.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.670+0000 7f52873f4640 1 --2- 172.21.15.67:0/3689923488 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f526c06bbd0 0x7f526c06e090 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:39.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.670+0000 7f52873f4640 1 --2- 172.21.15.67:0/3689923488 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52800c8f50 0x7f528015feb0 unknown :-1 s=CLOSED pgs=1914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:39.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.670+0000 7f52873f4640 1 --2- 172.21.15.67:0/3689923488 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f528015dfc0 0x7f528015f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:39.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.670+0000 7f52873f4640 1 --2- 172.21.15.67:0/3689923488 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f52800d0d50 0x7f52800c8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:39.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.670+0000 7f52873f4640 1 -- 172.21.15.67:0/3689923488 >> 172.21.15.67:0/3689923488 conn(0x7f528006c880 msgr2=0x7f5280161b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:39.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.671+0000 7f52873f4640 1 -- 172.21.15.67:0/3689923488 shutdown_connections 2024-03-24T00:49:39.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.671+0000 7f52873f4640 1 -- 172.21.15.67:0/3689923488 wait complete. 2024-03-24T00:49:39.685 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-24T00:49:39.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.795+0000 7f838f03d640 1 Processor -- start 2024-03-24T00:49:39.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.795+0000 7f838f03d640 1 -- start start 2024-03-24T00:49:39.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.796+0000 7f838f03d640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f838815e1b0 0x7f83880b6230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:39.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.796+0000 7f838f03d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f83880d0e60 0x7f83880d1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:39.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.796+0000 7f838f03d640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8388161d60 con 0x7f83880b2d80 2024-03-24T00:49:39.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.796+0000 7f838f03d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8388162970 con 0x7f83880d0e60 2024-03-24T00:49:39.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.796+0000 7f838f03d640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8388162040 con 0x7f838815e1b0 2024-03-24T00:49:39.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.796+0000 7f838d5da640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f83880d0e60 0x7f83880d1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:39.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.796+0000 7f838d5da640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f83880d0e60 0x7f83880d1fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48404/0 (socket says 172.21.15.67:48404) 2024-03-24T00:49:39.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.796+0000 7f838d5da640 1 -- 172.21.15.67:0/238996868 learned_addr learned my addr 172.21.15.67:0/238996868 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:39.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.796+0000 7f838cdd9640 1 --2- 172.21.15.67:0/238996868 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f838815e1b0 0x7f83880b6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:39.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.796+0000 7f838d5da640 1 -- 172.21.15.67:0/238996868 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f838815e1b0 msgr2=0x7f83880b6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:39.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.796+0000 7f838d5da640 1 --2- 172.21.15.67:0/238996868 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f838815e1b0 0x7f83880b6230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:39.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.796+0000 7f838d5da640 1 -- 172.21.15.67:0/238996868 >> v1:172.21.15.72:6789/0 conn(0x7f83880b2d80 legacy=0x7f83880d0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:39.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.796+0000 7f838d5da640 1 -- 172.21.15.67:0/238996868 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8388162320 con 0x7f83880d0e60 2024-03-24T00:49:39.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.797+0000 7f838d5da640 1 --2- 172.21.15.67:0/238996868 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83880d0e60 0x7f83880d1fd0 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f8370009870 tx=0x7f837002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bc87684bc99c69a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:39.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.797+0000 7f83877fe640 1 -- 172.21.15.67:0/238996868 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83700317c0 con 0x7f83880d0e60 2024-03-24T00:49:39.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.797+0000 7f83877fe640 1 -- 172.21.15.67:0/238996868 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8370037070 con 0x7f83880d0e60 2024-03-24T00:49:39.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.797+0000 7f83877fe640 1 -- 172.21.15.67:0/238996868 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8370031b80 con 0x7f83880d0e60 2024-03-24T00:49:39.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.798+0000 7f838f03d640 1 -- 172.21.15.67:0/238996868 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83880d0e60 msgr2=0x7f83880d1fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:39.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.798+0000 7f838f03d640 1 --2- 172.21.15.67:0/238996868 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83880d0e60 0x7f83880d1fd0 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f8370009870 tx=0x7f837002b860 comp rx=0 tx=0).stop 2024-03-24T00:49:39.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.798+0000 7f838f03d640 1 -- 172.21.15.67:0/238996868 shutdown_connections 2024-03-24T00:49:39.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.798+0000 7f838f03d640 1 --2- 172.21.15.67:0/238996868 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83880d0e60 0x7f83880d1fd0 unknown :-1 s=CLOSED pgs=1915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:39.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.798+0000 7f838f03d640 1 --2- 172.21.15.67:0/238996868 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f838815e1b0 0x7f83880b6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:39.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.798+0000 7f838f03d640 1 -- 172.21.15.67:0/238996868 >> 172.21.15.67:0/238996868 conn(0x7f838806c8f0 msgr2=0x7f838806ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:39.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.798+0000 7f838f03d640 1 -- 172.21.15.67:0/238996868 shutdown_connections 2024-03-24T00:49:39.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.798+0000 7f838f03d640 1 -- 172.21.15.67:0/238996868 wait complete. 2024-03-24T00:49:39.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.799+0000 7f838f03d640 1 Processor -- start 2024-03-24T00:49:39.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.799+0000 7f838f03d640 1 -- start start 2024-03-24T00:49:39.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.800+0000 7f838f03d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83880b2d80 0x7f83881541b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:39.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.800+0000 7f838f03d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83880d0e60 0x7f83880c9df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:39.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.800+0000 7f838f03d640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f838815e1b0 0x7f8388151c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:39.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.800+0000 7f838f03d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f838815e590 con 0x7f83880b2d80 2024-03-24T00:49:39.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.800+0000 7f838f03d640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83880b3570 con 0x7f83880d0e60 2024-03-24T00:49:39.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.800+0000 7f838f03d640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83880d2510 con 0x7f838815e1b0 2024-03-24T00:49:39.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.800+0000 7f838cdd9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83880b2d80 0x7f83881541b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:39.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.800+0000 7f838cdd9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83880b2d80 0x7f83881541b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48408/0 (socket says 172.21.15.67:48408) 2024-03-24T00:49:39.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.800+0000 7f838cdd9640 1 -- 172.21.15.67:0/1868889130 learned_addr learned my addr 172.21.15.67:0/1868889130 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:39.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.800+0000 7f838d5da640 1 --2- 172.21.15.67:0/1868889130 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f838815e1b0 0x7f8388151c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:39.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.800+0000 7f8387fff640 1 --2- 172.21.15.67:0/1868889130 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83880d0e60 0x7f83880c9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:39.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.800+0000 7f838cdd9640 1 -- 172.21.15.67:0/1868889130 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f838815e1b0 msgr2=0x7f8388151c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:39.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.800+0000 7f838cdd9640 1 --2- 172.21.15.67:0/1868889130 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f838815e1b0 0x7f8388151c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:39.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.800+0000 7f838cdd9640 1 -- 172.21.15.67:0/1868889130 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83880d0e60 msgr2=0x7f83880c9df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:39.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.800+0000 7f838cdd9640 1 --2- 172.21.15.67:0/1868889130 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83880d0e60 0x7f83880c9df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:39.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.800+0000 7f838cdd9640 1 -- 172.21.15.67:0/1868889130 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f838815b950 con 0x7f83880b2d80 2024-03-24T00:49:39.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.801+0000 7f838cdd9640 1 --2- 172.21.15.67:0/1868889130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83880b2d80 0x7f83881541b0 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f8380007e70 tx=0x7f838000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:39.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.801+0000 7f83857fa640 1 -- 172.21.15.67:0/1868889130 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8380014620 con 0x7f83880b2d80 2024-03-24T00:49:39.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.801+0000 7f83857fa640 1 -- 172.21.15.67:0/1868889130 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8380019070 con 0x7f83880b2d80 2024-03-24T00:49:39.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.801+0000 7f838f03d640 1 -- 172.21.15.67:0/1868889130 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83881577b0 con 0x7f83880b2d80 2024-03-24T00:49:39.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.801+0000 7f83857fa640 1 -- 172.21.15.67:0/1868889130 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8380014a00 con 0x7f83880b2d80 2024-03-24T00:49:39.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.801+0000 7f838f03d640 1 -- 172.21.15.67:0/1868889130 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8388157d70 con 0x7f83880b2d80 2024-03-24T00:49:39.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.802+0000 7f838f03d640 1 -- 172.21.15.67:0/1868889130 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f838804f400 con 0x7f83880b2d80 2024-03-24T00:49:39.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.802+0000 7f83857fa640 1 -- 172.21.15.67:0/1868889130 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8380014ba0 con 0x7f83880b2d80 2024-03-24T00:49:39.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.803+0000 7f83857fa640 1 --2- 172.21.15.67:0/1868889130 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f836806bbd0 0x7f836806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:39.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.803+0000 7f83857fa640 1 -- 172.21.15.67:0/1868889130 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5921+0+0 (secure 0 0 0) 0x7f838008b250 con 0x7f83880b2d80 2024-03-24T00:49:39.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.803+0000 7f8387fff640 1 --2- 172.21.15.67:0/1868889130 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f836806bbd0 0x7f836806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:39.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.804+0000 7f8387fff640 1 --2- 172.21.15.67:0/1868889130 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f836806bbd0 0x7f836806e090 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f8388156610 tx=0x7f8378007e30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:39.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.807+0000 7f83857fa640 1 -- 172.21.15.67:0/1868889130 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8380059610 con 0x7f83880b2d80 2024-03-24T00:49:39.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:39.986+0000 7f838f03d640 1 -- 172.21.15.67:0/1868889130 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f83880d1b80 con 0x7f83880b2d80 2024-03-24T00:49:40.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.671+0000 7f83857fa640 1 -- 172.21.15.67:0/1868889130 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v125) v1 ==== 169+0+0 (secure 0 0 0) 0x7f838005d6b0 con 0x7f83880b2d80 2024-03-24T00:49:40.673 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' removed 2024-03-24T00:49:40.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.677+0000 7f838f03d640 1 -- 172.21.15.67:0/1868889130 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f836806bbd0 msgr2=0x7f836806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:40.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.677+0000 7f838f03d640 1 --2- 172.21.15.67:0/1868889130 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f836806bbd0 0x7f836806e090 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f8388156610 tx=0x7f8378007e30 comp rx=0 tx=0).stop 2024-03-24T00:49:40.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.677+0000 7f838f03d640 1 -- 172.21.15.67:0/1868889130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83880b2d80 msgr2=0x7f83881541b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:40.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.677+0000 7f838f03d640 1 --2- 172.21.15.67:0/1868889130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83880b2d80 0x7f83881541b0 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f8380007e70 tx=0x7f838000e4b0 comp rx=0 tx=0).stop 2024-03-24T00:49:40.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.678+0000 7f838f03d640 1 -- 172.21.15.67:0/1868889130 shutdown_connections 2024-03-24T00:49:40.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.678+0000 7f838f03d640 1 --2- 172.21.15.67:0/1868889130 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f836806bbd0 0x7f836806e090 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:40.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.678+0000 7f838f03d640 1 --2- 172.21.15.67:0/1868889130 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f838815e1b0 0x7f8388151c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:40.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.678+0000 7f838f03d640 1 --2- 172.21.15.67:0/1868889130 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83880d0e60 0x7f83880c9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:40.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.678+0000 7f838f03d640 1 --2- 172.21.15.67:0/1868889130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83880b2d80 0x7f83881541b0 unknown :-1 s=CLOSED pgs=1916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:40.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.678+0000 7f838f03d640 1 -- 172.21.15.67:0/1868889130 >> 172.21.15.67:0/1868889130 conn(0x7f838806c8f0 msgr2=0x7f83880b3a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:40.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.678+0000 7f838f03d640 1 -- 172.21.15.67:0/1868889130 shutdown_connections 2024-03-24T00:49:40.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.679+0000 7f838f03d640 1 -- 172.21.15.67:0/1868889130 wait complete. 2024-03-24T00:49:40.694 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-24T00:49:40.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.806+0000 7f98bc8b2640 1 Processor -- start 2024-03-24T00:49:40.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.806+0000 7f98bc8b2640 1 -- start start 2024-03-24T00:49:40.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.806+0000 7f98bc8b2640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f98b80d0d50 0x7f98b80ce610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:40.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.807+0000 7f98bc8b2640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98b80cf570 con 0x7f98b815e130 2024-03-24T00:49:40.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.807+0000 7f98bc8b2640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98b80cecf0 con 0x7f98b80d22b0 2024-03-24T00:49:40.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.807+0000 7f98bc8b2640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98b80ceb50 con 0x7f98b80d0d50 2024-03-24T00:49:40.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.807+0000 7f98affff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f98b80d0d50 0x7f98b80ce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:40.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.807+0000 7f98b659c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f98b815e130 0x7f98b80d0640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56560/0 (socket says 172.21.15.67:56560) 2024-03-24T00:49:40.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.807+0000 7f98b659c640 1 -- 172.21.15.67:0/111063890 learned_addr learned my addr 172.21.15.67:0/111063890 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:49:40.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.807+0000 7f98affff640 1 -- 172.21.15.67:0/111063890 >> v1:172.21.15.72:6789/0 conn(0x7f98b80d22b0 legacy=0x7f98b80d2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:40.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.807+0000 7f98affff640 1 -- 172.21.15.67:0/111063890 >> v1:172.21.15.67:6789/0 conn(0x7f98b815e130 legacy=0x7f98b80d0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:40.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.807+0000 7f98affff640 1 -- 172.21.15.67:0/111063890 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b8162a60 con 0x7f98b80d0d50 2024-03-24T00:49:40.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.807+0000 7f98b5d9b640 1 -- 172.21.15.67:0/111063890 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1189427933 0 0) 0x7f98b80cf570 con 0x7f98b815e130 2024-03-24T00:49:40.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.808+0000 7f98affff640 1 --2- 172.21.15.67:0/111063890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98b80d0d50 0x7f98b80ce610 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f98a00097a0 tx=0x7f98a002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6844172dfb8cf07e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:40.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.808+0000 7f98b5d9b640 1 -- 172.21.15.67:0/111063890 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a0035020 con 0x7f98b80d0d50 2024-03-24T00:49:40.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.808+0000 7f98b5d9b640 1 -- 172.21.15.67:0/111063890 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98a0031830 con 0x7f98b80d0d50 2024-03-24T00:49:40.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.808+0000 7f98b5d9b640 1 -- 172.21.15.67:0/111063890 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a0031c00 con 0x7f98b80d0d50 2024-03-24T00:49:40.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.808+0000 7f98bc8b2640 1 -- 172.21.15.67:0/111063890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98b80d0d50 msgr2=0x7f98b80ce610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:40.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.808+0000 7f98bc8b2640 1 --2- 172.21.15.67:0/111063890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98b80d0d50 0x7f98b80ce610 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f98a00097a0 tx=0x7f98a002b8d0 comp rx=0 tx=0).stop 2024-03-24T00:49:40.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.808+0000 7f98bc8b2640 1 -- 172.21.15.67:0/111063890 shutdown_connections 2024-03-24T00:49:40.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.808+0000 7f98bc8b2640 1 --2- 172.21.15.67:0/111063890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98b80d0d50 0x7f98b80ce610 unknown :-1 s=CLOSED pgs=1918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:40.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.808+0000 7f98bc8b2640 1 -- 172.21.15.67:0/111063890 >> 172.21.15.67:0/111063890 conn(0x7f98b806c8a0 msgr2=0x7f98b806ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:40.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.808+0000 7f98bc8b2640 1 -- 172.21.15.67:0/111063890 shutdown_connections 2024-03-24T00:49:40.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.808+0000 7f98bc8b2640 1 -- 172.21.15.67:0/111063890 wait complete. 2024-03-24T00:49:40.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.810+0000 7f98bc8b2640 1 Processor -- start 2024-03-24T00:49:40.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.810+0000 7f98bc8b2640 1 -- start start 2024-03-24T00:49:40.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.810+0000 7f98bc8b2640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f98b80d0d50 0x7f98b814d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:40.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.810+0000 7f98bc8b2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98b80d22b0 0x7f98b80cc950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:40.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.810+0000 7f98bc8b2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98b815e130 0x7f98b80cadf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:40.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.810+0000 7f98bc8b2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b815e510 con 0x7f98b80d22b0 2024-03-24T00:49:40.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.810+0000 7f98bc8b2640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b80cf290 con 0x7f98b815e130 2024-03-24T00:49:40.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.810+0000 7f98bc8b2640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98b80b2e10 con 0x7f98b80d0d50 2024-03-24T00:49:40.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.810+0000 7f98affff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98b80d22b0 0x7f98b80cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:40.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.811+0000 7f98affff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98b80d22b0 0x7f98b80cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48424/0 (socket says 172.21.15.67:48424) 2024-03-24T00:49:40.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.811+0000 7f98affff640 1 -- 172.21.15.67:0/1309228702 learned_addr learned my addr 172.21.15.67:0/1309228702 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:40.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.811+0000 7f98b659c640 1 --2- 172.21.15.67:0/1309228702 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f98b80d0d50 0x7f98b814d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:40.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.811+0000 7f98b6d9d640 1 --2- 172.21.15.67:0/1309228702 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98b815e130 0x7f98b80cadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:40.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.811+0000 7f98affff640 1 -- 172.21.15.67:0/1309228702 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f98b80d0d50 msgr2=0x7f98b814d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:40.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.811+0000 7f98affff640 1 --2- 172.21.15.67:0/1309228702 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f98b80d0d50 0x7f98b814d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:40.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.811+0000 7f98affff640 1 -- 172.21.15.67:0/1309228702 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98b815e130 msgr2=0x7f98b80cadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:40.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.811+0000 7f98affff640 1 --2- 172.21.15.67:0/1309228702 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98b815e130 0x7f98b80cadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:40.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.811+0000 7f98affff640 1 -- 172.21.15.67:0/1309228702 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b80cb8b0 con 0x7f98b80d22b0 2024-03-24T00:49:40.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.811+0000 7f98affff640 1 --2- 172.21.15.67:0/1309228702 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98b80d22b0 0x7f98b80cc950 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f98a002bee0 tx=0x7f98a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:40.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.811+0000 7f98aeffd640 1 -- 172.21.15.67:0/1309228702 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a003ba80 con 0x7f98b80d22b0 2024-03-24T00:49:40.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.811+0000 7f98aeffd640 1 -- 172.21.15.67:0/1309228702 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98a003bc20 con 0x7f98b80d22b0 2024-03-24T00:49:40.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.811+0000 7f98bc8b2640 1 -- 172.21.15.67:0/1309228702 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98b80cbb90 con 0x7f98b80d22b0 2024-03-24T00:49:40.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.811+0000 7f98aeffd640 1 -- 172.21.15.67:0/1309228702 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a00453f0 con 0x7f98b80d22b0 2024-03-24T00:49:40.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.811+0000 7f98bc8b2640 1 -- 172.21.15.67:0/1309228702 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98b80cc150 con 0x7f98b80d22b0 2024-03-24T00:49:40.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.812+0000 7f98bc8b2640 1 -- 172.21.15.67:0/1309228702 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98b804f400 con 0x7f98b80d22b0 2024-03-24T00:49:40.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.813+0000 7f98aeffd640 1 -- 172.21.15.67:0/1309228702 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f98a0038020 con 0x7f98b80d22b0 2024-03-24T00:49:40.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.813+0000 7f98aeffd640 1 --2- 172.21.15.67:0/1309228702 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f989806bbd0 0x7f989806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:40.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.813+0000 7f98aeffd640 1 -- 172.21.15.67:0/1309228702 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f98a00ac770 con 0x7f98b80d22b0 2024-03-24T00:49:40.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.813+0000 7f98b659c640 1 --2- 172.21.15.67:0/1309228702 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f989806bbd0 0x7f989806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:40.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.814+0000 7f98b659c640 1 --2- 172.21.15.67:0/1309228702 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f989806bbd0 0x7f989806e090 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f98a8000f60 tx=0x7f98a8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:40.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:40.818+0000 7f98aeffd640 1 -- 172.21.15.67:0/1309228702 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f98a0040070 con 0x7f98b80d22b0 2024-03-24T00:49:41.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.004+0000 7f98bc8b2640 1 -- 172.21.15.67:0/1309228702 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f98b80d19f0 con 0x7f98b80d22b0 2024-03-24T00:49:41.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.008+0000 7f98aeffd640 1 -- 172.21.15.67:0/1309228702 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v23) v1 ==== 104+0+0 (secure 0 0 0) 0x7f98a007b320 con 0x7f98b80d22b0 2024-03-24T00:49:41.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.010+0000 7f98bc8b2640 1 -- 172.21.15.67:0/1309228702 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f989806bbd0 msgr2=0x7f989806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:41.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.010+0000 7f98bc8b2640 1 --2- 172.21.15.67:0/1309228702 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f989806bbd0 0x7f989806e090 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f98a8000f60 tx=0x7f98a8009040 comp rx=0 tx=0).stop 2024-03-24T00:49:41.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.010+0000 7f98bc8b2640 1 -- 172.21.15.67:0/1309228702 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98b80d22b0 msgr2=0x7f98b80cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:41.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.010+0000 7f98bc8b2640 1 --2- 172.21.15.67:0/1309228702 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98b80d22b0 0x7f98b80cc950 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f98a002bee0 tx=0x7f98a00046c0 comp rx=0 tx=0).stop 2024-03-24T00:49:41.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.010+0000 7f98bc8b2640 1 -- 172.21.15.67:0/1309228702 shutdown_connections 2024-03-24T00:49:41.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.010+0000 7f98bc8b2640 1 --2- 172.21.15.67:0/1309228702 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f989806bbd0 0x7f989806e090 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:41.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.010+0000 7f98bc8b2640 1 --2- 172.21.15.67:0/1309228702 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98b815e130 0x7f98b80cadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:41.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.010+0000 7f98bc8b2640 1 --2- 172.21.15.67:0/1309228702 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98b80d22b0 0x7f98b80cc950 unknown :-1 s=CLOSED pgs=1919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:41.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.010+0000 7f98bc8b2640 1 --2- 172.21.15.67:0/1309228702 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f98b80d0d50 0x7f98b814d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:41.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.010+0000 7f98bc8b2640 1 -- 172.21.15.67:0/1309228702 >> 172.21.15.67:0/1309228702 conn(0x7f98b806c8a0 msgr2=0x7f98b80b3760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:41.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.010+0000 7f98bc8b2640 1 -- 172.21.15.67:0/1309228702 shutdown_connections 2024-03-24T00:49:41.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.010+0000 7f98bc8b2640 1 -- 172.21.15.67:0/1309228702 wait complete. 2024-03-24T00:49:41.020 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_orphan_scan' 2024-03-24T00:49:41.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.122+0000 7f196e8b5640 1 Processor -- start 2024-03-24T00:49:41.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.122+0000 7f196e8b5640 1 -- start start 2024-03-24T00:49:41.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.123+0000 7f196e8b5640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f196815dfb0 0x7f19680cfe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:41.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.123+0000 7f196e8b5640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f196806b750 0x7f19680cb120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:41.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.123+0000 7f196e8b5640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19680d07d0 con 0x7f196815e390 2024-03-24T00:49:41.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.123+0000 7f196e8b5640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19680d0ab0 con 0x7f196815dfb0 2024-03-24T00:49:41.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.123+0000 7f196e8b5640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19680d0d90 con 0x7f196806b750 2024-03-24T00:49:41.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.123+0000 7f19677fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f196815e390 0x7f196806b040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56564/0 (socket says 172.21.15.67:56564) 2024-03-24T00:49:41.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.123+0000 7f19677fe640 1 -- 172.21.15.67:0/681438718 learned_addr learned my addr 172.21.15.67:0/681438718 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:49:41.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.123+0000 7f1966ffd640 1 -- 172.21.15.67:0/681438718 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1914122490 0 0) 0x7f19680d07d0 con 0x7f196815e390 2024-03-24T00:49:41.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.123+0000 7f1966ffd640 1 -- 172.21.15.67:0/681438718 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1954003520 con 0x7f196815e390 2024-03-24T00:49:41.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.123+0000 7f196ce52640 1 --2- 172.21.15.67:0/681438718 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f196806b750 0x7f19680cb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:41.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.123+0000 7f1967fff640 1 --2- 172.21.15.67:0/681438718 >> v2:172.21.15.72:3300/0 conn(0x7f196815dfb0 0x7f19680cfe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:41.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.123+0000 7f1966ffd640 1 -- 172.21.15.67:0/681438718 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3572953124 0 0) 0x7f1954003520 con 0x7f196815e390 2024-03-24T00:49:41.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.123+0000 7f1966ffd640 1 -- 172.21.15.67:0/681438718 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f196806b750 msgr2=0x7f19680cb120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:41.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.123+0000 7f1966ffd640 1 --2- 172.21.15.67:0/681438718 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f196806b750 0x7f19680cb120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:41.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.123+0000 7f1966ffd640 1 -- 172.21.15.67:0/681438718 >> v2:172.21.15.72:3300/0 conn(0x7f196815dfb0 msgr2=0x7f19680cfe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:41.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.123+0000 7f1966ffd640 1 --2- 172.21.15.67:0/681438718 >> v2:172.21.15.72:3300/0 conn(0x7f196815dfb0 0x7f19680cfe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:41.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.123+0000 7f1966ffd640 1 -- 172.21.15.67:0/681438718 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19680d1070 con 0x7f196815e390 2024-03-24T00:49:41.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.124+0000 7f1966ffd640 1 -- 172.21.15.67:0/681438718 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f1958001670 con 0x7f196815e390 2024-03-24T00:49:41.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.124+0000 7f1966ffd640 1 -- 172.21.15.67:0/681438718 >> v1:172.21.15.67:6789/0 conn(0x7f196815e390 legacy=0x7f196806b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:41.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.124+0000 7f1966ffd640 1 --2- 172.21.15.67:0/681438718 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19540039b0 0x7f1954003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:41.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.124+0000 7f1966ffd640 1 --2- 172.21.15.67:0/681438718 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f196806b750 0x7f1954004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:41.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.124+0000 7f1966ffd640 1 --2- 172.21.15.67:0/681438718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1954004d40 0x7f19540051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:41.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.124+0000 7f1966ffd640 1 -- 172.21.15.67:0/681438718 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19680d0ab0 con 0x7f1954004d40 2024-03-24T00:49:41.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.124+0000 7f1966ffd640 1 -- 172.21.15.67:0/681438718 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19680d0d90 con 0x7f19540039b0 2024-03-24T00:49:41.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.124+0000 7f1966ffd640 1 -- 172.21.15.67:0/681438718 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f195400b9c0 con 0x7f196806b750 2024-03-24T00:49:41.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.124+0000 7f196ce52640 1 --2- 172.21.15.67:0/681438718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1954004d40 0x7f19540051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:41.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.124+0000 7f19677fe640 1 --2- 172.21.15.67:0/681438718 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f196806b750 0x7f1954004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:41.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.124+0000 7f1967fff640 1 --2- 172.21.15.67:0/681438718 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19540039b0 0x7f1954003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:41.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.124+0000 7f196ce52640 1 -- 172.21.15.67:0/681438718 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f196806b750 msgr2=0x7f1954004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:41.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.124+0000 7f196ce52640 1 --2- 172.21.15.67:0/681438718 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f196806b750 0x7f1954004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:41.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.124+0000 7f196ce52640 1 -- 172.21.15.67:0/681438718 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19540039b0 msgr2=0x7f1954003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:41.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.124+0000 7f196ce52640 1 --2- 172.21.15.67:0/681438718 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19540039b0 0x7f1954003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:41.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.124+0000 7f196ce52640 1 -- 172.21.15.67:0/681438718 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1954003520 con 0x7f1954004d40 2024-03-24T00:49:41.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.125+0000 7f196ce52640 1 --2- 172.21.15.67:0/681438718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1954004d40 0x7f19540051c0 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f1960009f90 tx=0x7f1954005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9b53853aa38b416 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:41.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.125+0000 7f1966ffd640 1 -- 172.21.15.67:0/681438718 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1960039070 con 0x7f1954004d40 2024-03-24T00:49:41.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.125+0000 7f1966ffd640 1 -- 172.21.15.67:0/681438718 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f196002cb90 con 0x7f1954004d40 2024-03-24T00:49:41.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.125+0000 7f1966ffd640 1 -- 172.21.15.67:0/681438718 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1960009c10 con 0x7f1954004d40 2024-03-24T00:49:41.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.125+0000 7f196e8b5640 1 -- 172.21.15.67:0/681438718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1954004d40 msgr2=0x7f19540051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:41.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.125+0000 7f196e8b5640 1 --2- 172.21.15.67:0/681438718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1954004d40 0x7f19540051c0 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f1960009f90 tx=0x7f1954005830 comp rx=0 tx=0).stop 2024-03-24T00:49:41.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.125+0000 7f196ce52640 1 -- 172.21.15.67:0/681438718 reap_dead start 2024-03-24T00:49:41.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.125+0000 7f196e8b5640 1 -- 172.21.15.67:0/681438718 shutdown_connections 2024-03-24T00:49:41.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.125+0000 7f196e8b5640 1 -- 172.21.15.67:0/681438718 >> 172.21.15.67:0/681438718 conn(0x7f1968099a30 msgr2=0x7f1968099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:41.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.126+0000 7f196e8b5640 1 -- 172.21.15.67:0/681438718 shutdown_connections 2024-03-24T00:49:41.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.126+0000 7f196e8b5640 1 -- 172.21.15.67:0/681438718 wait complete. 2024-03-24T00:49:41.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.127+0000 7f196e8b5640 1 Processor -- start 2024-03-24T00:49:41.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.127+0000 7f196e8b5640 1 -- start start 2024-03-24T00:49:41.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.128+0000 7f196e8b5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1954004d40 0x7f19680b5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:41.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.128+0000 7f196e8b5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f196815f4d0 0x7f19681580b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:41.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.128+0000 7f196e8b5640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f19680cabe0 0x7f19681585f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:41.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.128+0000 7f196e8b5640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19680d03c0 con 0x7f196815f4d0 2024-03-24T00:49:41.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.128+0000 7f196e8b5640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19680d1870 con 0x7f1954004d40 2024-03-24T00:49:41.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.128+0000 7f196e8b5640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19680d0540 con 0x7f19680cabe0 2024-03-24T00:49:41.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.128+0000 7f19677fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f196815f4d0 0x7f19681580b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:41.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.128+0000 7f19677fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f196815f4d0 0x7f19681580b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48448/0 (socket says 172.21.15.67:48448) 2024-03-24T00:49:41.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.128+0000 7f19677fe640 1 -- 172.21.15.67:0/2727545809 learned_addr learned my addr 172.21.15.67:0/2727545809 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:41.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.128+0000 7f1967fff640 1 --2- 172.21.15.67:0/2727545809 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1954004d40 0x7f19680b5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:41.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.128+0000 7f196ce52640 1 --2- 172.21.15.67:0/2727545809 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f19680cabe0 0x7f19681585f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:41.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.128+0000 7f19677fe640 1 -- 172.21.15.67:0/2727545809 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f19680cabe0 msgr2=0x7f19681585f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:41.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.128+0000 7f19677fe640 1 --2- 172.21.15.67:0/2727545809 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f19680cabe0 0x7f19681585f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:41.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.128+0000 7f19677fe640 1 -- 172.21.15.67:0/2727545809 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1954004d40 msgr2=0x7f19680b5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:41.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.128+0000 7f19677fe640 1 --2- 172.21.15.67:0/2727545809 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1954004d40 0x7f19680b5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:41.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.128+0000 7f19677fe640 1 -- 172.21.15.67:0/2727545809 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19680b5a00 con 0x7f196815f4d0 2024-03-24T00:49:41.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.128+0000 7f196ce52640 1 --2- 172.21.15.67:0/2727545809 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f19680cabe0 0x7f19681585f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:49:41.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.128+0000 7f19677fe640 1 --2- 172.21.15.67:0/2727545809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f196815f4d0 0x7f19681580b0 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7f1958007310 tx=0x7f1958007710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:41.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.128+0000 7f1964ff9640 1 -- 172.21.15.67:0/2727545809 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f195800f9c0 con 0x7f196815f4d0 2024-03-24T00:49:41.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.128+0000 7f1964ff9640 1 -- 172.21.15.67:0/2727545809 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1958018070 con 0x7f196815f4d0 2024-03-24T00:49:41.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.128+0000 7f196e8b5640 1 -- 172.21.15.67:0/2727545809 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19680b5ce0 con 0x7f196815f4d0 2024-03-24T00:49:41.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.128+0000 7f1964ff9640 1 -- 172.21.15.67:0/2727545809 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f195800fda0 con 0x7f196815f4d0 2024-03-24T00:49:41.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.128+0000 7f196e8b5640 1 -- 172.21.15.67:0/2727545809 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19680b62a0 con 0x7f196815f4d0 2024-03-24T00:49:41.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.129+0000 7f196e8b5640 1 -- 172.21.15.67:0/2727545809 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f196804f400 con 0x7f196815f4d0 2024-03-24T00:49:41.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.130+0000 7f1964ff9640 1 -- 172.21.15.67:0/2727545809 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f195801d460 con 0x7f196815f4d0 2024-03-24T00:49:41.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.130+0000 7f1964ff9640 1 --2- 172.21.15.67:0/2727545809 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f194806bbd0 0x7f194806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:41.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.130+0000 7f1964ff9640 1 -- 172.21.15.67:0/2727545809 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f195808aaf0 con 0x7f196815f4d0 2024-03-24T00:49:41.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.130+0000 7f1967fff640 1 --2- 172.21.15.67:0/2727545809 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f194806bbd0 0x7f194806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:41.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.131+0000 7f1967fff640 1 --2- 172.21.15.67:0/2727545809 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f194806bbd0 0x7f194806e090 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f1950009bc0 tx=0x7f1950009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:41.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.135+0000 7f1964ff9640 1 -- 172.21.15.67:0/2727545809 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1958059730 con 0x7f196815f4d0 2024-03-24T00:49:41.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.388+0000 7f196e8b5640 1 -- 172.21.15.67:0/2727545809 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_orphan_scan"]} v 0) v1 -- 0x7f19680b6a50 con 0x7f196815f4d0 2024-03-24T00:49:41.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.672+0000 7f1964ff9640 1 -- 172.21.15.67:0/2727545809 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_orphan_scan"]}]=0 v509) v1 ==== 144+0+0 (secure 0 0 0) 0x7f1958059030 con 0x7f196815f4d0 2024-03-24T00:49:41.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.674+0000 7f196e8b5640 1 -- 172.21.15.67:0/2727545809 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f194806bbd0 msgr2=0x7f194806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:41.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.674+0000 7f196e8b5640 1 --2- 172.21.15.67:0/2727545809 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f194806bbd0 0x7f194806e090 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f1950009bc0 tx=0x7f1950009190 comp rx=0 tx=0).stop 2024-03-24T00:49:41.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.674+0000 7f196e8b5640 1 -- 172.21.15.67:0/2727545809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f196815f4d0 msgr2=0x7f19681580b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:41.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.674+0000 7f196e8b5640 1 --2- 172.21.15.67:0/2727545809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f196815f4d0 0x7f19681580b0 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7f1958007310 tx=0x7f1958007710 comp rx=0 tx=0).stop 2024-03-24T00:49:41.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.674+0000 7f196e8b5640 1 -- 172.21.15.67:0/2727545809 shutdown_connections 2024-03-24T00:49:41.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.674+0000 7f196e8b5640 1 --2- 172.21.15.67:0/2727545809 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f194806bbd0 0x7f194806e090 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:41.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.674+0000 7f196e8b5640 1 --2- 172.21.15.67:0/2727545809 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f19680cabe0 0x7f19681585f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:41.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.674+0000 7f196e8b5640 1 --2- 172.21.15.67:0/2727545809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f196815f4d0 0x7f19681580b0 unknown :-1 s=CLOSED pgs=1922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:41.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.674+0000 7f196e8b5640 1 --2- 172.21.15.67:0/2727545809 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1954004d40 0x7f19680b5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:41.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.674+0000 7f196e8b5640 1 -- 172.21.15.67:0/2727545809 >> 172.21.15.67:0/2727545809 conn(0x7f1968099a30 msgr2=0x7f196806b040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:41.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.674+0000 7f196e8b5640 1 -- 172.21.15.67:0/2727545809 shutdown_connections 2024-03-24T00:49:41.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:41.674+0000 7f196e8b5640 1 -- 172.21.15.67:0/2727545809 wait complete. 2024-03-24T00:49:41.686 INFO:tasks.cephfs_test_runner:test_orphan_scan (tasks.cephfs.test_forward_scrub.TestForwardScrub) ... ok 2024-03-24T00:49:41.686 INFO:tasks.cephfs_test_runner: 2024-03-24T00:49:41.686 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-24T00:49:41.686 INFO:tasks.cephfs_test_runner:Ran 7 tests in 606.715s 2024-03-24T00:49:41.686 INFO:tasks.cephfs_test_runner: 2024-03-24T00:49:41.686 INFO:tasks.cephfs_test_runner:OK 2024-03-24T00:49:41.687 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-03-24T00:49:41.697 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-24T00:49:41.709 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-24T00:49:41.709 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:49:41.709 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:49:41.725 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056375,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15771 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T00:49:41.725 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-24T00:49:41.725 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:49:41.780 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056375,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15771 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T00:49:41.781 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:49:41.781 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:49:41.800 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17590 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248808k,nr_inodes=812202,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:49:41.800 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-24T00:49:41.800 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:49:41.858 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17590 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248808k,nr_inodes=812202,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:49:41.858 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:41.858 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-03-24T00:49:41.873 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-24T00:49:41.873 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-24T00:49:41.873 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:41.873 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:49:41.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:41.940 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:49:42.006 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:42.007 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link delete ceph-brx 2024-03-24T00:49:42.113 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:42.113 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-03-24T00:49:42.129 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:42.129 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link delete ceph-brx 2024-03-24T00:49:42.198 INFO:teuthology.orchestra.run.smithi067.stderr:Cannot find device "ceph-brx" 2024-03-24T00:49:42.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:49:42.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:42.199 DEBUG:teuthology.orchestra.run.smithi072:> ip netns list 2024-03-24T00:49:42.218 INFO:teuthology.orchestra.run.smithi072.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-03-24T00:49:42.218 INFO:teuthology.orchestra.run.smithi072.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-24T00:49:42.218 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:42.218 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-24T00:49:42.289 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:42.289 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-24T00:49:42.360 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:42.360 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link delete ceph-brx 2024-03-24T00:49:42.448 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:42.448 DEBUG:teuthology.orchestra.run.smithi072:> ip netns list 2024-03-24T00:49:42.463 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:49:42.463 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link delete ceph-brx 2024-03-24T00:49:42.533 INFO:teuthology.orchestra.run.smithi072.stderr:Cannot find device "ceph-brx" 2024-03-24T00:49:42.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:49:42.535 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-24T00:49:42.545 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-24T00:49:42.545 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:49:42.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.650+0000 7ff574dbc640 1 Processor -- start 2024-03-24T00:49:42.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.650+0000 7ff574dbc640 1 -- start start 2024-03-24T00:49:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.650+0000 7ff574dbc640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff5701445a0 0x7ff5700cee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.650+0000 7ff574dbc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff5700b48d0 0x7ff570072550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.650+0000 7ff574dbc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff57006ec40 0x7ff57006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.650+0000 7ff574dbc640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5700ce990 con 0x7ff5700b48d0 2024-03-24T00:49:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.650+0000 7ff574dbc640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5700b4d30 con 0x7ff57006ec40 2024-03-24T00:49:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.650+0000 7ff574dbc640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5700cddc0 con 0x7ff5701445a0 2024-03-24T00:49:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.651+0000 7ff567fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff5700b48d0 0x7ff570072550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.651+0000 7ff567fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff5700b48d0 0x7ff570072550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48454/0 (socket says 172.21.15.67:48454) 2024-03-24T00:49:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.651+0000 7ff567fff640 1 -- 172.21.15.67:0/909339496 learned_addr learned my addr 172.21.15.67:0/909339496 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.651+0000 7ff56ed9d640 1 --2- 172.21.15.67:0/909339496 >> v2:172.21.15.72:3300/0 conn(0x7ff57006ec40 0x7ff57006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.651+0000 7ff56e59c640 1 --2- 172.21.15.67:0/909339496 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff5701445a0 0x7ff5700cee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.651+0000 7ff567fff640 1 -- 172.21.15.67:0/909339496 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff5701445a0 msgr2=0x7ff5700cee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.651+0000 7ff567fff640 1 --2- 172.21.15.67:0/909339496 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff5701445a0 0x7ff5700cee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:42.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.651+0000 7ff567fff640 1 -- 172.21.15.67:0/909339496 >> v2:172.21.15.72:3300/0 conn(0x7ff57006ec40 msgr2=0x7ff57006f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:42.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.651+0000 7ff567fff640 1 --2- 172.21.15.67:0/909339496 >> v2:172.21.15.72:3300/0 conn(0x7ff57006ec40 0x7ff57006f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:42.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.651+0000 7ff567fff640 1 -- 172.21.15.67:0/909339496 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5700ce060 con 0x7ff5700b48d0 2024-03-24T00:49:42.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.651+0000 7ff56ed9d640 1 --2- 172.21.15.67:0/909339496 >> v2:172.21.15.72:3300/0 conn(0x7ff57006ec40 0x7ff57006f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:49:42.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.652+0000 7ff567fff640 1 --2- 172.21.15.67:0/909339496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5700b48d0 0x7ff570072550 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7ff5700af610 tx=0x7ff55802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88090d73748254f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:42.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.652+0000 7ff56dd9b640 1 -- 172.21.15.67:0/909339496 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff558035020 con 0x7ff5700b48d0 2024-03-24T00:49:42.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.652+0000 7ff56dd9b640 1 -- 172.21.15.67:0/909339496 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff558031870 con 0x7ff5700b48d0 2024-03-24T00:49:42.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.652+0000 7ff56dd9b640 1 -- 172.21.15.67:0/909339496 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff558031c40 con 0x7ff5700b48d0 2024-03-24T00:49:42.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.652+0000 7ff574dbc640 1 -- 172.21.15.67:0/909339496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5700b48d0 msgr2=0x7ff570072550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:42.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.652+0000 7ff574dbc640 1 --2- 172.21.15.67:0/909339496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5700b48d0 0x7ff570072550 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7ff5700af610 tx=0x7ff55802b910 comp rx=0 tx=0).stop 2024-03-24T00:49:42.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.652+0000 7ff574dbc640 1 -- 172.21.15.67:0/909339496 shutdown_connections 2024-03-24T00:49:42.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.652+0000 7ff574dbc640 1 --2- 172.21.15.67:0/909339496 >> v2:172.21.15.72:3300/0 conn(0x7ff57006ec40 0x7ff57006f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:42.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.652+0000 7ff574dbc640 1 --2- 172.21.15.67:0/909339496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5700b48d0 0x7ff570072550 unknown :-1 s=CLOSED pgs=1923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:42.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.652+0000 7ff574dbc640 1 --2- 172.21.15.67:0/909339496 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff5701445a0 0x7ff5700cee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:42.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.652+0000 7ff574dbc640 1 -- 172.21.15.67:0/909339496 >> 172.21.15.67:0/909339496 conn(0x7ff5700cf9e0 msgr2=0x7ff5700cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:42.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.652+0000 7ff574dbc640 1 -- 172.21.15.67:0/909339496 shutdown_connections 2024-03-24T00:49:42.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.652+0000 7ff574dbc640 1 -- 172.21.15.67:0/909339496 wait complete. 2024-03-24T00:49:42.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.654+0000 7ff574dbc640 1 Processor -- start 2024-03-24T00:49:42.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.654+0000 7ff574dbc640 1 -- start start 2024-03-24T00:49:42.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.654+0000 7ff574dbc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff57006ec40 0x7ff5701571d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:42.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.654+0000 7ff574dbc640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff5700b48d0 0x7ff570132a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:42.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.654+0000 7ff574dbc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5701445a0 0x7ff570130660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:42.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.654+0000 7ff574dbc640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5700b2a10 con 0x7ff5701445a0 2024-03-24T00:49:42.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.654+0000 7ff574dbc640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5700ce620 con 0x7ff57006ec40 2024-03-24T00:49:42.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.654+0000 7ff574dbc640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5700b2b90 con 0x7ff5700b48d0 2024-03-24T00:49:42.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.655+0000 7ff56ed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5701445a0 0x7ff570130660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:42.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.655+0000 7ff56ed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5701445a0 0x7ff570130660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48464/0 (socket says 172.21.15.67:48464) 2024-03-24T00:49:42.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.655+0000 7ff56ed9d640 1 -- 172.21.15.67:0/652181995 learned_addr learned my addr 172.21.15.67:0/652181995 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:42.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.655+0000 7ff567fff640 1 --2- 172.21.15.67:0/652181995 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff5700b48d0 0x7ff570132a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:42.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.655+0000 7ff56e59c640 1 --2- 172.21.15.67:0/652181995 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff57006ec40 0x7ff5701571d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:42.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.655+0000 7ff56ed9d640 1 -- 172.21.15.67:0/652181995 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff5700b48d0 msgr2=0x7ff570132a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:42.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.655+0000 7ff56ed9d640 1 --2- 172.21.15.67:0/652181995 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff5700b48d0 0x7ff570132a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:42.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.655+0000 7ff56ed9d640 1 -- 172.21.15.67:0/652181995 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff57006ec40 msgr2=0x7ff5701571d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:42.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.655+0000 7ff56ed9d640 1 --2- 172.21.15.67:0/652181995 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff57006ec40 0x7ff5701571d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:42.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.655+0000 7ff56ed9d640 1 -- 172.21.15.67:0/652181995 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff570133460 con 0x7ff5701445a0 2024-03-24T00:49:42.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.655+0000 7ff56ed9d640 1 --2- 172.21.15.67:0/652181995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5701445a0 0x7ff570130660 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7ff568007e10 tx=0x7ff56800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:42.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.655+0000 7ff566ffd640 1 -- 172.21.15.67:0/652181995 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff56801a070 con 0x7ff5701445a0 2024-03-24T00:49:42.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.655+0000 7ff566ffd640 1 -- 172.21.15.67:0/652181995 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff568003b90 con 0x7ff5701445a0 2024-03-24T00:49:42.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.655+0000 7ff574dbc640 1 -- 172.21.15.67:0/652181995 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff570133740 con 0x7ff5701445a0 2024-03-24T00:49:42.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.655+0000 7ff566ffd640 1 -- 172.21.15.67:0/652181995 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff568004030 con 0x7ff5701445a0 2024-03-24T00:49:42.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.655+0000 7ff574dbc640 1 -- 172.21.15.67:0/652181995 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff570133d00 con 0x7ff5701445a0 2024-03-24T00:49:42.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.656+0000 7ff574dbc640 1 -- 172.21.15.67:0/652181995 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff57004f400 con 0x7ff5701445a0 2024-03-24T00:49:42.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.657+0000 7ff566ffd640 1 -- 172.21.15.67:0/652181995 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff56801d070 con 0x7ff5701445a0 2024-03-24T00:49:42.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.657+0000 7ff566ffd640 1 --2- 172.21.15.67:0/652181995 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff55006bbd0 0x7ff55006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:42.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.657+0000 7ff566ffd640 1 -- 172.21.15.67:0/652181995 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5507+0+0 (secure 0 0 0) 0x7ff56808ac70 con 0x7ff5701445a0 2024-03-24T00:49:42.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.657+0000 7ff56e59c640 1 --2- 172.21.15.67:0/652181995 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff55006bbd0 0x7ff55006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:42.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.658+0000 7ff56e59c640 1 --2- 172.21.15.67:0/652181995 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff55006bbd0 0x7ff55006e090 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7ff560006fd0 tx=0x7ff560008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:42.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.662+0000 7ff566ffd640 1 -- 172.21.15.67:0/652181995 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff568010070 con 0x7ff5701445a0 2024-03-24T00:49:42.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.841+0000 7ff574dbc640 1 -- 172.21.15.67:0/652181995 --> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff570072c20 con 0x7ff55006bbd0 2024-03-24T00:49:42.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.842+0000 7ff566ffd640 1 -- 172.21.15.67:0/652181995 <== mgr.4105 v2:172.21.15.72:6832/2768292274 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13436 (secure 0 0 0) 0x7ff570072c20 con 0x7ff55006bbd0 2024-03-24T00:49:42.842 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:42.842 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-24T00:49:42.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.844+0000 7ff574dbc640 1 -- 172.21.15.67:0/652181995 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff55006bbd0 msgr2=0x7ff55006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:42.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.844+0000 7ff574dbc640 1 --2- 172.21.15.67:0/652181995 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff55006bbd0 0x7ff55006e090 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7ff560006fd0 tx=0x7ff560008040 comp rx=0 tx=0).stop 2024-03-24T00:49:42.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.844+0000 7ff574dbc640 1 -- 172.21.15.67:0/652181995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5701445a0 msgr2=0x7ff570130660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:42.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.844+0000 7ff574dbc640 1 --2- 172.21.15.67:0/652181995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5701445a0 0x7ff570130660 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7ff568007e10 tx=0x7ff56800d4a0 comp rx=0 tx=0).stop 2024-03-24T00:49:42.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.844+0000 7ff574dbc640 1 -- 172.21.15.67:0/652181995 shutdown_connections 2024-03-24T00:49:42.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.844+0000 7ff574dbc640 1 --2- 172.21.15.67:0/652181995 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7ff55006bbd0 0x7ff55006e090 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:42.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.844+0000 7ff574dbc640 1 --2- 172.21.15.67:0/652181995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5701445a0 0x7ff570130660 unknown :-1 s=CLOSED pgs=1924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:42.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.844+0000 7ff574dbc640 1 --2- 172.21.15.67:0/652181995 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff5700b48d0 0x7ff570132a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:42.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.844+0000 7ff574dbc640 1 --2- 172.21.15.67:0/652181995 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff57006ec40 0x7ff5701571d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:42.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.844+0000 7ff574dbc640 1 -- 172.21.15.67:0/652181995 >> 172.21.15.67:0/652181995 conn(0x7ff5700cf9e0 msgr2=0x7ff5700cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:42.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.844+0000 7ff574dbc640 1 -- 172.21.15.67:0/652181995 shutdown_connections 2024-03-24T00:49:42.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.844+0000 7ff574dbc640 1 -- 172.21.15.67:0/652181995 wait complete. 2024-03-24T00:49:42.855 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":449,"stamp":"2024-03-24T00:49:41.480535+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":279040,"kb_used_data":29600,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754695680,"statfs":{"total":773094113280,"available":772808376320,"internally_reserved":0,"allocated":30310400,"data_stored":23715448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8390540,"num_objects":-30,"num_object_clones":0,"num_object_copies":-60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-30,"num_whiteouts":0,"num_read":-258,"num_read_kb":-268,"num_write":-177,"num_write_kb":-8305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-17,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001862"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":364,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-24T00:49:37.893458+0000","last_change":"2024-03-24T00:38:40.953871+0000","last_active":"2024-03-24T00:49:37.893458+0000","last_peered":"2024-03-24T00:49:37.893458+0000","last_clean":"2024-03-24T00:49:37.893458+0000","last_became_active":"2024-03-24T00:38:40.952926+0000","last_became_peered":"2024-03-24T00:38:40.952926+0000","last_unstale":"2024-03-24T00:49:37.893458+0000","last_undegraded":"2024-03-24T00:49:37.893458+0000","last_fullsized":"2024-03-24T00:49:37.893458+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_clean_scrub_stamp":"2024-03-24T00:38:39.722194+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:05.397257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574985,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33276,"kb_used_data":2096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338564,"statfs":{"total":96636764160,"available":96602689536,"internally_reserved":0,"allocated":2146304,"data_stored":1310355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574984,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36776,"kb_used_data":5596,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335064,"statfs":{"total":96636764160,"available":96599105536,"internally_reserved":0,"allocated":5730304,"data_stored":4914090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574984,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36780,"kb_used_data":5600,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335060,"statfs":{"total":96636764160,"available":96599101440,"internally_reserved":0,"allocated":5734400,"data_stored":4913668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574983,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36792,"kb_used_data":5612,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335048,"statfs":{"total":96636764160,"available":96599089152,"internally_reserved":0,"allocated":5746688,"data_stored":4914637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574983,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32692,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339148,"statfs":{"total":96636764160,"available":96603287552,"internally_reserved":0,"allocated":1548288,"data_stored":720355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574984,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32680,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339160,"statfs":{"total":96636764160,"available":96603299840,"internally_reserved":0,"allocated":1536000,"data_stored":719209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574984,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36780,"kb_used_data":5600,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335060,"statfs":{"total":96636764160,"available":96599101440,"internally_reserved":0,"allocated":5734400,"data_stored":4913535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574983,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33264,"kb_used_data":2084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338576,"statfs":{"total":96636764160,"available":96602701824,"internally_reserved":0,"allocated":2134016,"data_stored":1309599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:49:42.855 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:49:42.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.956+0000 7f8bd3d37640 1 Processor -- start 2024-03-24T00:49:42.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.956+0000 7f8bd3d37640 1 -- start start 2024-03-24T00:49:42.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.957+0000 7f8bd3d37640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8bcc0b49c0 0x7f8bcc15b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:42.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.957+0000 7f8bd3d37640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8bcc06f100 0x7f8bcc072360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:42.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.957+0000 7f8bd3d37640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8bcc0728a0 0x7f8bcc071ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:42.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.957+0000 7f8bd3d37640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bcc0b4da0 con 0x7f8bcc06f100 2024-03-24T00:49:42.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.957+0000 7f8bd3d37640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bcc06db30 con 0x7f8bcc0b49c0 2024-03-24T00:49:42.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.957+0000 7f8bd3d37640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bcc06de10 con 0x7f8bcc0728a0 2024-03-24T00:49:42.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.957+0000 7f8bd12d2640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8bcc06f100 0x7f8bcc072360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:42.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.957+0000 7f8bd12d2640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8bcc06f100 0x7f8bcc072360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48470/0 (socket says 172.21.15.67:48470) 2024-03-24T00:49:42.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.957+0000 7f8bd12d2640 1 -- 172.21.15.67:0/3029729322 learned_addr learned my addr 172.21.15.67:0/3029729322 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:42.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.957+0000 7f8bd1ad3640 1 --2- 172.21.15.67:0/3029729322 >> v2:172.21.15.72:3300/0 conn(0x7f8bcc0b49c0 0x7f8bcc15b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:42.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.957+0000 7f8bd22d4640 1 --2- 172.21.15.67:0/3029729322 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8bcc0728a0 0x7f8bcc071ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:42.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.957+0000 7f8bd12d2640 1 -- 172.21.15.67:0/3029729322 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8bcc0728a0 msgr2=0x7f8bcc071ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:42.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.957+0000 7f8bd12d2640 1 --2- 172.21.15.67:0/3029729322 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8bcc0728a0 0x7f8bcc071ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:42.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.957+0000 7f8bd12d2640 1 -- 172.21.15.67:0/3029729322 >> v2:172.21.15.72:3300/0 conn(0x7f8bcc0b49c0 msgr2=0x7f8bcc15b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:42.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.957+0000 7f8bd12d2640 1 --2- 172.21.15.67:0/3029729322 >> v2:172.21.15.72:3300/0 conn(0x7f8bcc0b49c0 0x7f8bcc15b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:42.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.957+0000 7f8bd12d2640 1 -- 172.21.15.67:0/3029729322 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bcc06e0f0 con 0x7f8bcc06f100 2024-03-24T00:49:42.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.957+0000 7f8bd1ad3640 1 --2- 172.21.15.67:0/3029729322 >> v2:172.21.15.72:3300/0 conn(0x7f8bcc0b49c0 0x7f8bcc15b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:42.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.958+0000 7f8bd12d2640 1 --2- 172.21.15.67:0/3029729322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bcc06f100 0x7f8bcc072360 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc072f50 tx=0x7f8bbc02e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58ff456b5a517576 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:42.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.958+0000 7f8bd0ad1640 1 -- 172.21.15.67:0/3029729322 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bbc03b070 con 0x7f8bcc06f100 2024-03-24T00:49:42.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.958+0000 7f8bd0ad1640 1 -- 172.21.15.67:0/3029729322 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bbc02ee00 con 0x7f8bcc06f100 2024-03-24T00:49:42.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.958+0000 7f8bd0ad1640 1 -- 172.21.15.67:0/3029729322 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bbc036660 con 0x7f8bcc06f100 2024-03-24T00:49:42.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.959+0000 7f8bd3d37640 1 -- 172.21.15.67:0/3029729322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bcc06f100 msgr2=0x7f8bcc072360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:42.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.959+0000 7f8bd3d37640 1 --2- 172.21.15.67:0/3029729322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bcc06f100 0x7f8bcc072360 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc072f50 tx=0x7f8bbc02e050 comp rx=0 tx=0).stop 2024-03-24T00:49:42.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.959+0000 7f8bd3d37640 1 -- 172.21.15.67:0/3029729322 shutdown_connections 2024-03-24T00:49:42.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.959+0000 7f8bd3d37640 1 --2- 172.21.15.67:0/3029729322 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8bcc0728a0 0x7f8bcc071ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:42.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.959+0000 7f8bd3d37640 1 --2- 172.21.15.67:0/3029729322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bcc06f100 0x7f8bcc072360 unknown :-1 s=CLOSED pgs=1925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:42.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.959+0000 7f8bd3d37640 1 --2- 172.21.15.67:0/3029729322 >> v2:172.21.15.72:3300/0 conn(0x7f8bcc0b49c0 0x7f8bcc15b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:42.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.959+0000 7f8bd3d37640 1 -- 172.21.15.67:0/3029729322 >> 172.21.15.67:0/3029729322 conn(0x7f8bcc0cf9e0 msgr2=0x7f8bcc0cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:42.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.959+0000 7f8bd3d37640 1 -- 172.21.15.67:0/3029729322 shutdown_connections 2024-03-24T00:49:42.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.959+0000 7f8bd3d37640 1 -- 172.21.15.67:0/3029729322 wait complete. 2024-03-24T00:49:42.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.960+0000 7f8bd3d37640 1 Processor -- start 2024-03-24T00:49:42.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.960+0000 7f8bd3d37640 1 -- start start 2024-03-24T00:49:42.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.961+0000 7f8bd3d37640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8bcc06f100 0x7f8bcc140b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:42.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.961+0000 7f8bd3d37640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bcc0728a0 0x7f8bcc13d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:42.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.961+0000 7f8bd3d37640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8bcc0b49c0 0x7f8bcc149d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:42.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.961+0000 7f8bd3d37640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bcc06f590 con 0x7f8bcc0728a0 2024-03-24T00:49:42.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.961+0000 7f8bd3d37640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bcc06e6b0 con 0x7f8bcc06f100 2024-03-24T00:49:42.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.961+0000 7f8bd3d37640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bcc072010 con 0x7f8bcc0b49c0 2024-03-24T00:49:42.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.961+0000 7f8bd12d2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bcc0728a0 0x7f8bcc13d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:42.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.961+0000 7f8bd12d2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bcc0728a0 0x7f8bcc13d7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48486/0 (socket says 172.21.15.67:48486) 2024-03-24T00:49:42.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.961+0000 7f8bd12d2640 1 -- 172.21.15.67:0/433557680 learned_addr learned my addr 172.21.15.67:0/433557680 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:42.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.961+0000 7f8bd22d4640 1 --2- 172.21.15.67:0/433557680 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8bcc0b49c0 0x7f8bcc149d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:42.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.961+0000 7f8bd1ad3640 1 --2- 172.21.15.67:0/433557680 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8bcc06f100 0x7f8bcc140b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:42.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.961+0000 7f8bd12d2640 1 -- 172.21.15.67:0/433557680 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8bcc0b49c0 msgr2=0x7f8bcc149d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:42.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.961+0000 7f8bd12d2640 1 --2- 172.21.15.67:0/433557680 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8bcc0b49c0 0x7f8bcc149d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:42.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.961+0000 7f8bd12d2640 1 -- 172.21.15.67:0/433557680 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8bcc06f100 msgr2=0x7f8bcc140b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:42.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.961+0000 7f8bd12d2640 1 --2- 172.21.15.67:0/433557680 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8bcc06f100 0x7f8bcc140b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:42.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.961+0000 7f8bd12d2640 1 -- 172.21.15.67:0/433557680 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bcc13e130 con 0x7f8bcc0728a0 2024-03-24T00:49:42.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.961+0000 7f8bd12d2640 1 --2- 172.21.15.67:0/433557680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bcc0728a0 0x7f8bcc13d7e0 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f8bbc02e090 tx=0x7f8bbc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:42.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.962+0000 7f8bc27fc640 1 -- 172.21.15.67:0/433557680 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bbc03b070 con 0x7f8bcc0728a0 2024-03-24T00:49:42.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.962+0000 7f8bc27fc640 1 -- 172.21.15.67:0/433557680 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bbc03f750 con 0x7f8bcc0728a0 2024-03-24T00:49:42.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.962+0000 7f8bd3d37640 1 -- 172.21.15.67:0/433557680 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bcc1f4010 con 0x7f8bcc0728a0 2024-03-24T00:49:42.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.962+0000 7f8bc27fc640 1 -- 172.21.15.67:0/433557680 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bbc03fb40 con 0x7f8bcc0728a0 2024-03-24T00:49:42.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.962+0000 7f8bd3d37640 1 -- 172.21.15.67:0/433557680 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bcc1f4850 con 0x7f8bcc0728a0 2024-03-24T00:49:42.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.963+0000 7f8bd3d37640 1 -- 172.21.15.67:0/433557680 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bcc04f400 con 0x7f8bcc0728a0 2024-03-24T00:49:42.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.963+0000 7f8bc27fc640 1 -- 172.21.15.67:0/433557680 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8bbc048060 con 0x7f8bcc0728a0 2024-03-24T00:49:42.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.963+0000 7f8bc27fc640 1 --2- 172.21.15.67:0/433557680 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8bb006bbd0 0x7f8bb006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:42.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.964+0000 7f8bc27fc640 1 -- 172.21.15.67:0/433557680 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f8bbc0ac110 con 0x7f8bcc0728a0 2024-03-24T00:49:42.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.964+0000 7f8bd1ad3640 1 --2- 172.21.15.67:0/433557680 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8bb006bbd0 0x7f8bb006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:42.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.964+0000 7f8bd1ad3640 1 --2- 172.21.15.67:0/433557680 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8bb006bbd0 0x7f8bb006e090 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f8bc400ab60 tx=0x7f8bc4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:42.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:42.968+0000 7f8bc27fc640 1 -- 172.21.15.67:0/433557680 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8bcc1f4010 con 0x7f8bcc0728a0 2024-03-24T00:49:43.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.146+0000 7f8bd3d37640 1 -- 172.21.15.67:0/433557680 --> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8bcc14bd20 con 0x7f8bb006bbd0 2024-03-24T00:49:43.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.147+0000 7f8bc27fc640 1 -- 172.21.15.67:0/433557680 <== mgr.4105 v2:172.21.15.72:6832/2768292274 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13436 (secure 0 0 0) 0x7f8bcc14bd20 con 0x7f8bb006bbd0 2024-03-24T00:49:43.147 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:43.147 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-24T00:49:43.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.149+0000 7f8bd3d37640 1 -- 172.21.15.67:0/433557680 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8bb006bbd0 msgr2=0x7f8bb006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:43.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.149+0000 7f8bd3d37640 1 --2- 172.21.15.67:0/433557680 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8bb006bbd0 0x7f8bb006e090 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f8bc400ab60 tx=0x7f8bc4008040 comp rx=0 tx=0).stop 2024-03-24T00:49:43.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.149+0000 7f8bd3d37640 1 -- 172.21.15.67:0/433557680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bcc0728a0 msgr2=0x7f8bcc13d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:43.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.149+0000 7f8bd3d37640 1 --2- 172.21.15.67:0/433557680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bcc0728a0 0x7f8bcc13d7e0 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f8bbc02e090 tx=0x7f8bbc004730 comp rx=0 tx=0).stop 2024-03-24T00:49:43.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.149+0000 7f8bd3d37640 1 -- 172.21.15.67:0/433557680 shutdown_connections 2024-03-24T00:49:43.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.149+0000 7f8bd3d37640 1 --2- 172.21.15.67:0/433557680 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f8bb006bbd0 0x7f8bb006e090 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:43.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.149+0000 7f8bd3d37640 1 --2- 172.21.15.67:0/433557680 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8bcc0b49c0 0x7f8bcc149d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:43.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.149+0000 7f8bd3d37640 1 --2- 172.21.15.67:0/433557680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bcc0728a0 0x7f8bcc13d7e0 unknown :-1 s=CLOSED pgs=1926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:43.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.149+0000 7f8bd3d37640 1 --2- 172.21.15.67:0/433557680 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8bcc06f100 0x7f8bcc140b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:43.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.149+0000 7f8bd3d37640 1 -- 172.21.15.67:0/433557680 >> 172.21.15.67:0/433557680 conn(0x7f8bcc0cf9e0 msgr2=0x7f8bcc0ce780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:43.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.149+0000 7f8bd3d37640 1 -- 172.21.15.67:0/433557680 shutdown_connections 2024-03-24T00:49:43.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.149+0000 7f8bd3d37640 1 -- 172.21.15.67:0/433557680 wait complete. 2024-03-24T00:49:43.159 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":449,"stamp":"2024-03-24T00:49:41.480535+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":279040,"kb_used_data":29600,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754695680,"statfs":{"total":773094113280,"available":772808376320,"internally_reserved":0,"allocated":30310400,"data_stored":23715448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8390540,"num_objects":-30,"num_object_clones":0,"num_object_copies":-60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-30,"num_whiteouts":0,"num_read":-258,"num_read_kb":-268,"num_write":-177,"num_write_kb":-8305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-17,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001862"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":364,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-24T00:49:37.893458+0000","last_change":"2024-03-24T00:38:40.953871+0000","last_active":"2024-03-24T00:49:37.893458+0000","last_peered":"2024-03-24T00:49:37.893458+0000","last_clean":"2024-03-24T00:49:37.893458+0000","last_became_active":"2024-03-24T00:38:40.952926+0000","last_became_peered":"2024-03-24T00:38:40.952926+0000","last_unstale":"2024-03-24T00:49:37.893458+0000","last_undegraded":"2024-03-24T00:49:37.893458+0000","last_fullsized":"2024-03-24T00:49:37.893458+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_clean_scrub_stamp":"2024-03-24T00:38:39.722194+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:05.397257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574985,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33276,"kb_used_data":2096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338564,"statfs":{"total":96636764160,"available":96602689536,"internally_reserved":0,"allocated":2146304,"data_stored":1310355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574984,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36776,"kb_used_data":5596,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335064,"statfs":{"total":96636764160,"available":96599105536,"internally_reserved":0,"allocated":5730304,"data_stored":4914090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574984,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36780,"kb_used_data":5600,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335060,"statfs":{"total":96636764160,"available":96599101440,"internally_reserved":0,"allocated":5734400,"data_stored":4913668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574983,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36792,"kb_used_data":5612,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335048,"statfs":{"total":96636764160,"available":96599089152,"internally_reserved":0,"allocated":5746688,"data_stored":4914637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574983,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32692,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339148,"statfs":{"total":96636764160,"available":96603287552,"internally_reserved":0,"allocated":1548288,"data_stored":720355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574984,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32680,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339160,"statfs":{"total":96636764160,"available":96603299840,"internally_reserved":0,"allocated":1536000,"data_stored":719209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574984,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36780,"kb_used_data":5600,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335060,"statfs":{"total":96636764160,"available":96599101440,"internally_reserved":0,"allocated":5734400,"data_stored":4913535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574983,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33264,"kb_used_data":2084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338576,"statfs":{"total":96636764160,"available":96602701824,"internally_reserved":0,"allocated":2134016,"data_stored":1309599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:49:43.160 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-24T00:49:43.160 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:49:43.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.261+0000 7f1eb95cd640 1 Processor -- start 2024-03-24T00:49:43.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.261+0000 7f1eb95cd640 1 -- start start 2024-03-24T00:49:43.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.262+0000 7f1eb95cd640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1eb415e2a0 0x7f1eb40ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:43.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.262+0000 7f1eb95cd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1eb40d0ef0 0x7f1eb4162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:43.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.262+0000 7f1eb95cd640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb40cf640 con 0x7f1eb415dec0 2024-03-24T00:49:43.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.262+0000 7f1eb95cd640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eb40cee30 con 0x7f1eb40d0ef0 2024-03-24T00:49:43.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.262+0000 7f1eb95cd640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb40d2120 con 0x7f1eb415e2a0 2024-03-24T00:49:43.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.262+0000 7f1eb2ffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1eb415dec0 0x7f1eb40d07e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56580/0 (socket says 172.21.15.67:56580) 2024-03-24T00:49:43.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.262+0000 7f1eb2ffd640 1 -- 172.21.15.67:0/3476395568 learned_addr learned my addr 172.21.15.67:0/3476395568 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:49:43.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.263+0000 7f1eb37fe640 1 --2- 172.21.15.67:0/3476395568 >> v2:172.21.15.72:3300/0 conn(0x7f1eb40d0ef0 0x7f1eb4162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:43.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.263+0000 7f1eb27fc640 1 --2- 172.21.15.67:0/3476395568 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1eb415e2a0 0x7f1eb40ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:43.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.263+0000 7f1eb1ffb640 1 -- 172.21.15.67:0/3476395568 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3968658756 0 0) 0x7f1eac001110 con 0x7f1eb415dec0 2024-03-24T00:49:43.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.263+0000 7f1eb1ffb640 1 -- 172.21.15.67:0/3476395568 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ea0003520 con 0x7f1eb415dec0 2024-03-24T00:49:43.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.263+0000 7f1eb37fe640 1 -- 172.21.15.67:0/3476395568 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1eb415e2a0 msgr2=0x7f1eb40ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:43.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.263+0000 7f1eb1ffb640 1 -- 172.21.15.67:0/3476395568 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1103539375 0 0) 0x7f1ea0003520 con 0x7f1eb415dec0 2024-03-24T00:49:43.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.263+0000 7f1eb37fe640 1 --2- 172.21.15.67:0/3476395568 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1eb415e2a0 0x7f1eb40ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:43.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.263+0000 7f1eb37fe640 1 -- 172.21.15.67:0/3476395568 >> v1:172.21.15.67:6789/0 conn(0x7f1eb415dec0 legacy=0x7f1eb40d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:43.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.263+0000 7f1eb37fe640 1 -- 172.21.15.67:0/3476395568 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb40cf080 con 0x7f1eb40d0ef0 2024-03-24T00:49:43.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.263+0000 7f1eb27fc640 1 --2- 172.21.15.67:0/3476395568 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1eb415e2a0 0x7f1eb40ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:49:43.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.264+0000 7f1eb37fe640 1 --2- 172.21.15.67:0/3476395568 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1eb40d0ef0 0x7f1eb4162630 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f1ea4009d90 tx=0x7f1ea402bce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be9df1846c416e84 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:43.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.264+0000 7f1eb1ffb640 1 -- 172.21.15.67:0/3476395568 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea4039070 con 0x7f1eb40d0ef0 2024-03-24T00:49:43.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.264+0000 7f1eb1ffb640 1 -- 172.21.15.67:0/3476395568 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ea402c810 con 0x7f1eb40d0ef0 2024-03-24T00:49:43.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.264+0000 7f1eb1ffb640 1 -- 172.21.15.67:0/3476395568 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eac001840 con 0x7f1eb40d0ef0 2024-03-24T00:49:43.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.264+0000 7f1eb95cd640 1 -- 172.21.15.67:0/3476395568 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1eb40d0ef0 msgr2=0x7f1eb4162630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:43.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.264+0000 7f1eb95cd640 1 --2- 172.21.15.67:0/3476395568 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1eb40d0ef0 0x7f1eb4162630 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f1ea4009d90 tx=0x7f1ea402bce0 comp rx=0 tx=0).stop 2024-03-24T00:49:43.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.264+0000 7f1eb95cd640 1 -- 172.21.15.67:0/3476395568 shutdown_connections 2024-03-24T00:49:43.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.264+0000 7f1eb95cd640 1 --2- 172.21.15.67:0/3476395568 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1eb40d0ef0 0x7f1eb4162630 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:43.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.264+0000 7f1eb95cd640 1 --2- 172.21.15.67:0/3476395568 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1eb415e2a0 0x7f1eb40ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:43.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.264+0000 7f1eb95cd640 1 -- 172.21.15.67:0/3476395568 >> 172.21.15.67:0/3476395568 conn(0x7f1eb406c810 msgr2=0x7f1eb406cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:43.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.264+0000 7f1eb95cd640 1 -- 172.21.15.67:0/3476395568 shutdown_connections 2024-03-24T00:49:43.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.265+0000 7f1eb95cd640 1 -- 172.21.15.67:0/3476395568 wait complete. 2024-03-24T00:49:43.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.266+0000 7f1eb95cd640 1 Processor -- start 2024-03-24T00:49:43.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.266+0000 7f1eb95cd640 1 -- start start 2024-03-24T00:49:43.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.266+0000 7f1eb95cd640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1eb40d0ef0 0x7f1eb41f4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:43.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.266+0000 7f1eb95cd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1eb415dec0 0x7f1eb41f3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:43.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.266+0000 7f1eb95cd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1eb415e2a0 0x7f1eb41e7240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:43.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.266+0000 7f1eb95cd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb40d2730 con 0x7f1eb415dec0 2024-03-24T00:49:43.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.266+0000 7f1eb95cd640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb40d25b0 con 0x7f1eb415e2a0 2024-03-24T00:49:43.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.266+0000 7f1eb95cd640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb40d28b0 con 0x7f1eb40d0ef0 2024-03-24T00:49:43.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.267+0000 7f1eb27fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1eb415dec0 0x7f1eb41f3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:43.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.267+0000 7f1eb27fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1eb415dec0 0x7f1eb41f3870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48498/0 (socket says 172.21.15.67:48498) 2024-03-24T00:49:43.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.267+0000 7f1eb27fc640 1 -- 172.21.15.67:0/2033212366 learned_addr learned my addr 172.21.15.67:0/2033212366 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:43.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.267+0000 7f1eb37fe640 1 --2- 172.21.15.67:0/2033212366 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1eb415e2a0 0x7f1eb41e7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:43.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.267+0000 7f1eb2ffd640 1 --2- 172.21.15.67:0/2033212366 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1eb40d0ef0 0x7f1eb41f4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:43.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.267+0000 7f1eb27fc640 1 -- 172.21.15.67:0/2033212366 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1eb40d0ef0 msgr2=0x7f1eb41f4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:43.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.267+0000 7f1eb27fc640 1 --2- 172.21.15.67:0/2033212366 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1eb40d0ef0 0x7f1eb41f4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:43.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.267+0000 7f1eb27fc640 1 -- 172.21.15.67:0/2033212366 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1eb415e2a0 msgr2=0x7f1eb41e7240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:43.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.267+0000 7f1eb27fc640 1 --2- 172.21.15.67:0/2033212366 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1eb415e2a0 0x7f1eb41e7240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:43.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.267+0000 7f1eb27fc640 1 -- 172.21.15.67:0/2033212366 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb41f7ad0 con 0x7f1eb415dec0 2024-03-24T00:49:43.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.267+0000 7f1eb37fe640 1 --2- 172.21.15.67:0/2033212366 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1eb415e2a0 0x7f1eb41e7240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:49:43.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.267+0000 7f1eb2ffd640 1 --2- 172.21.15.67:0/2033212366 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1eb40d0ef0 0x7f1eb41f4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:49:43.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.267+0000 7f1eb27fc640 1 --2- 172.21.15.67:0/2033212366 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1eb415dec0 0x7f1eb41f3870 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c00e900 tx=0x7f1e9c00edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:43.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.267+0000 7f1eab7fe640 1 -- 172.21.15.67:0/2033212366 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e9c017070 con 0x7f1eb415dec0 2024-03-24T00:49:43.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.267+0000 7f1eab7fe640 1 -- 172.21.15.67:0/2033212366 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e9c00ca00 con 0x7f1eb415dec0 2024-03-24T00:49:43.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.267+0000 7f1eb95cd640 1 -- 172.21.15.67:0/2033212366 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eb41f7db0 con 0x7f1eb415dec0 2024-03-24T00:49:43.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.267+0000 7f1eab7fe640 1 -- 172.21.15.67:0/2033212366 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e9c00cdf0 con 0x7f1eb415dec0 2024-03-24T00:49:43.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.267+0000 7f1eb95cd640 1 -- 172.21.15.67:0/2033212366 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eb41f8370 con 0x7f1eb415dec0 2024-03-24T00:49:43.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.269+0000 7f1eb95cd640 1 -- 172.21.15.67:0/2033212366 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eb40b3970 con 0x7f1eb415dec0 2024-03-24T00:49:43.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.269+0000 7f1eab7fe640 1 -- 172.21.15.67:0/2033212366 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1e9c01a070 con 0x7f1eb415dec0 2024-03-24T00:49:43.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.269+0000 7f1eab7fe640 1 --2- 172.21.15.67:0/2033212366 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1e9406bbd0 0x7f1e9406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:43.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.270+0000 7f1eab7fe640 1 -- 172.21.15.67:0/2033212366 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f1e9c08a7a0 con 0x7f1eb415dec0 2024-03-24T00:49:43.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.270+0000 7f1eb2ffd640 1 --2- 172.21.15.67:0/2033212366 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1e9406bbd0 0x7f1e9406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:43.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.270+0000 7f1eb2ffd640 1 --2- 172.21.15.67:0/2033212366 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1e9406bbd0 0x7f1e9406e090 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f1eac001080 tx=0x7f1eac001010 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:43.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.274+0000 7f1eab7fe640 1 -- 172.21.15.67:0/2033212366 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1e9c058d90 con 0x7f1eb415dec0 2024-03-24T00:49:43.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.451+0000 7f1eb95cd640 1 -- 172.21.15.67:0/2033212366 --> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1eb404f400 con 0x7f1e9406bbd0 2024-03-24T00:49:43.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.452+0000 7f1eab7fe640 1 -- 172.21.15.67:0/2033212366 <== mgr.4105 v2:172.21.15.72:6832/2768292274 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13436 (secure 0 0 0) 0x7f1eb404f400 con 0x7f1e9406bbd0 2024-03-24T00:49:43.452 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:43.452 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-24T00:49:43.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.454+0000 7f1eb95cd640 1 -- 172.21.15.67:0/2033212366 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1e9406bbd0 msgr2=0x7f1e9406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:43.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.454+0000 7f1eb95cd640 1 --2- 172.21.15.67:0/2033212366 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1e9406bbd0 0x7f1e9406e090 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f1eac001080 tx=0x7f1eac001010 comp rx=0 tx=0).stop 2024-03-24T00:49:43.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.454+0000 7f1eb95cd640 1 -- 172.21.15.67:0/2033212366 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1eb415dec0 msgr2=0x7f1eb41f3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:43.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.454+0000 7f1eb95cd640 1 --2- 172.21.15.67:0/2033212366 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1eb415dec0 0x7f1eb41f3870 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c00e900 tx=0x7f1e9c00edc0 comp rx=0 tx=0).stop 2024-03-24T00:49:43.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.454+0000 7f1eb95cd640 1 -- 172.21.15.67:0/2033212366 shutdown_connections 2024-03-24T00:49:43.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.454+0000 7f1eb95cd640 1 --2- 172.21.15.67:0/2033212366 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f1e9406bbd0 0x7f1e9406e090 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:43.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.454+0000 7f1eb95cd640 1 --2- 172.21.15.67:0/2033212366 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1eb415e2a0 0x7f1eb41e7240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:43.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.454+0000 7f1eb95cd640 1 --2- 172.21.15.67:0/2033212366 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1eb415dec0 0x7f1eb41f3870 unknown :-1 s=CLOSED pgs=1928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:43.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.454+0000 7f1eb95cd640 1 --2- 172.21.15.67:0/2033212366 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1eb40d0ef0 0x7f1eb41f4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:43.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.454+0000 7f1eb95cd640 1 -- 172.21.15.67:0/2033212366 >> 172.21.15.67:0/2033212366 conn(0x7f1eb406c810 msgr2=0x7f1eb40d2c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:43.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.454+0000 7f1eb95cd640 1 -- 172.21.15.67:0/2033212366 shutdown_connections 2024-03-24T00:49:43.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.454+0000 7f1eb95cd640 1 -- 172.21.15.67:0/2033212366 wait complete. 2024-03-24T00:49:43.464 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":449,"stamp":"2024-03-24T00:49:41.480535+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":279040,"kb_used_data":29600,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754695680,"statfs":{"total":773094113280,"available":772808376320,"internally_reserved":0,"allocated":30310400,"data_stored":23715448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8390540,"num_objects":-30,"num_object_clones":0,"num_object_copies":-60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-30,"num_whiteouts":0,"num_read":-258,"num_read_kb":-268,"num_write":-177,"num_write_kb":-8305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-17,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001862"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":364,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-24T00:49:37.893458+0000","last_change":"2024-03-24T00:38:40.953871+0000","last_active":"2024-03-24T00:49:37.893458+0000","last_peered":"2024-03-24T00:49:37.893458+0000","last_clean":"2024-03-24T00:49:37.893458+0000","last_became_active":"2024-03-24T00:38:40.952926+0000","last_became_peered":"2024-03-24T00:38:40.952926+0000","last_unstale":"2024-03-24T00:49:37.893458+0000","last_undegraded":"2024-03-24T00:49:37.893458+0000","last_fullsized":"2024-03-24T00:49:37.893458+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_clean_scrub_stamp":"2024-03-24T00:38:39.722194+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:05.397257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574985,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33276,"kb_used_data":2096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338564,"statfs":{"total":96636764160,"available":96602689536,"internally_reserved":0,"allocated":2146304,"data_stored":1310355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574984,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36776,"kb_used_data":5596,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335064,"statfs":{"total":96636764160,"available":96599105536,"internally_reserved":0,"allocated":5730304,"data_stored":4914090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574984,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36780,"kb_used_data":5600,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335060,"statfs":{"total":96636764160,"available":96599101440,"internally_reserved":0,"allocated":5734400,"data_stored":4913668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574983,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36792,"kb_used_data":5612,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335048,"statfs":{"total":96636764160,"available":96599089152,"internally_reserved":0,"allocated":5746688,"data_stored":4914637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574983,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32692,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339148,"statfs":{"total":96636764160,"available":96603287552,"internally_reserved":0,"allocated":1548288,"data_stored":720355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574984,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32680,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339160,"statfs":{"total":96636764160,"available":96603299840,"internally_reserved":0,"allocated":1536000,"data_stored":719209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574984,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36780,"kb_used_data":5600,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335060,"statfs":{"total":96636764160,"available":96599101440,"internally_reserved":0,"allocated":5734400,"data_stored":4913535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574983,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33264,"kb_used_data":2084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338576,"statfs":{"total":96636764160,"available":96602701824,"internally_reserved":0,"allocated":2134016,"data_stored":1309599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:49:43.465 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:49:43.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.566+0000 7f86e8aa1640 1 Processor -- start 2024-03-24T00:49:43.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.566+0000 7f86e8aa1640 1 -- start start 2024-03-24T00:49:43.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.567+0000 7f86e8aa1640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f86e415e2a0 0x7f86e40ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:43.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.567+0000 7f86e8aa1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f86e40d0ef0 0x7f86e4162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:43.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.567+0000 7f86e8aa1640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86e40cf640 con 0x7f86e415dec0 2024-03-24T00:49:43.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.567+0000 7f86e8aa1640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86e40cee30 con 0x7f86e415e2a0 2024-03-24T00:49:43.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.567+0000 7f86e8aa1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86e40d2120 con 0x7f86e40d0ef0 2024-03-24T00:49:43.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.567+0000 7f86e1d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f86e415e2a0 0x7f86e40ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:43.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.567+0000 7f86e259c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f86e415dec0 0x7f86e40d07e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56258/0 (socket says 172.21.15.67:56258) 2024-03-24T00:49:43.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.567+0000 7f86e259c640 1 -- 172.21.15.67:0/283289370 learned_addr learned my addr 172.21.15.67:0/283289370 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:49:43.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.567+0000 7f86e2d9d640 1 --2- 172.21.15.67:0/283289370 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f86e40d0ef0 0x7f86e4162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:43.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.568+0000 7f86e1d9b640 1 -- 172.21.15.67:0/283289370 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f86e40d0ef0 msgr2=0x7f86e4162630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:43.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.568+0000 7f86e1d9b640 1 --2- 172.21.15.67:0/283289370 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f86e40d0ef0 0x7f86e4162630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:43.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.568+0000 7f86e1d9b640 1 -- 172.21.15.67:0/283289370 >> v1:172.21.15.67:6789/0 conn(0x7f86e415dec0 legacy=0x7f86e40d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:43.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.568+0000 7f86e1d9b640 1 -- 172.21.15.67:0/283289370 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86e40cf080 con 0x7f86e415e2a0 2024-03-24T00:49:43.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.568+0000 7f86e1d9b640 1 --2- 172.21.15.67:0/283289370 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86e415e2a0 0x7f86e40ce630 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f86cc0097a0 tx=0x7f86cc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfb162952ebbd189 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:43.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.568+0000 7f86e159a640 1 -- 172.21.15.67:0/283289370 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86cc0315a0 con 0x7f86e415e2a0 2024-03-24T00:49:43.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.568+0000 7f86e159a640 1 -- 172.21.15.67:0/283289370 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86cc037070 con 0x7f86e415e2a0 2024-03-24T00:49:43.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.568+0000 7f86e159a640 1 -- 172.21.15.67:0/283289370 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86cc031960 con 0x7f86e415e2a0 2024-03-24T00:49:43.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.569+0000 7f86e8aa1640 1 -- 172.21.15.67:0/283289370 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86e415e2a0 msgr2=0x7f86e40ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:43.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.569+0000 7f86e8aa1640 1 --2- 172.21.15.67:0/283289370 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86e415e2a0 0x7f86e40ce630 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f86cc0097a0 tx=0x7f86cc02b840 comp rx=0 tx=0).stop 2024-03-24T00:49:43.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.569+0000 7f86e8aa1640 1 -- 172.21.15.67:0/283289370 shutdown_connections 2024-03-24T00:49:43.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.569+0000 7f86e8aa1640 1 --2- 172.21.15.67:0/283289370 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f86e40d0ef0 0x7f86e4162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:43.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.569+0000 7f86e8aa1640 1 --2- 172.21.15.67:0/283289370 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86e415e2a0 0x7f86e40ce630 unknown :-1 s=CLOSED pgs=1930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:43.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.569+0000 7f86e8aa1640 1 -- 172.21.15.67:0/283289370 >> 172.21.15.67:0/283289370 conn(0x7f86e406c810 msgr2=0x7f86e406cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:43.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.569+0000 7f86e8aa1640 1 -- 172.21.15.67:0/283289370 shutdown_connections 2024-03-24T00:49:43.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.569+0000 7f86e8aa1640 1 -- 172.21.15.67:0/283289370 wait complete. 2024-03-24T00:49:43.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.570+0000 7f86e8aa1640 1 Processor -- start 2024-03-24T00:49:43.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.570+0000 7f86e8aa1640 1 -- start start 2024-03-24T00:49:43.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.571+0000 7f86e8aa1640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f86e40d0ef0 0x7f86e4151f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:43.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.571+0000 7f86e8aa1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f86e415dec0 0x7f86e41f43f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:43.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.571+0000 7f86e8aa1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86e415e2a0 0x7f86e4158de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:43.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.571+0000 7f86e8aa1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86e40d2730 con 0x7f86e415e2a0 2024-03-24T00:49:43.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.571+0000 7f86e8aa1640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86e40d25b0 con 0x7f86e415dec0 2024-03-24T00:49:43.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.571+0000 7f86e8aa1640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86e40d28b0 con 0x7f86e40d0ef0 2024-03-24T00:49:43.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.571+0000 7f86e2d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86e415e2a0 0x7f86e4158de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:43.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.571+0000 7f86e2d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86e415e2a0 0x7f86e4158de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43584/0 (socket says 172.21.15.67:43584) 2024-03-24T00:49:43.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.571+0000 7f86e2d9d640 1 -- 172.21.15.67:0/1965487358 learned_addr learned my addr 172.21.15.67:0/1965487358 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:43.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.571+0000 7f86e259c640 1 --2- 172.21.15.67:0/1965487358 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f86e40d0ef0 0x7f86e4151f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:43.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.571+0000 7f86e1d9b640 1 --2- 172.21.15.67:0/1965487358 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f86e415dec0 0x7f86e41f43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:43.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.572+0000 7f86e2d9d640 1 -- 172.21.15.67:0/1965487358 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f86e40d0ef0 msgr2=0x7f86e4151f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:43.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.572+0000 7f86e2d9d640 1 --2- 172.21.15.67:0/1965487358 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f86e40d0ef0 0x7f86e4151f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:43.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.572+0000 7f86e2d9d640 1 -- 172.21.15.67:0/1965487358 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f86e415dec0 msgr2=0x7f86e41f43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:43.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.572+0000 7f86e2d9d640 1 --2- 172.21.15.67:0/1965487358 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f86e415dec0 0x7f86e41f43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:43.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.572+0000 7f86e2d9d640 1 -- 172.21.15.67:0/1965487358 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86e40b51d0 con 0x7f86e415e2a0 2024-03-24T00:49:43.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.572+0000 7f86e2d9d640 1 --2- 172.21.15.67:0/1965487358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86e415e2a0 0x7f86e4158de0 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f86dc00d5d0 tx=0x7f86dc00da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:43.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.572+0000 7f86daffd640 1 -- 172.21.15.67:0/1965487358 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86dc0125f0 con 0x7f86e415e2a0 2024-03-24T00:49:43.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.572+0000 7f86daffd640 1 -- 172.21.15.67:0/1965487358 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86dc019070 con 0x7f86e415e2a0 2024-03-24T00:49:43.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.572+0000 7f86e8aa1640 1 -- 172.21.15.67:0/1965487358 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86e40b54b0 con 0x7f86e415e2a0 2024-03-24T00:49:43.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.572+0000 7f86daffd640 1 -- 172.21.15.67:0/1965487358 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86dc0129d0 con 0x7f86e415e2a0 2024-03-24T00:49:43.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.572+0000 7f86e8aa1640 1 -- 172.21.15.67:0/1965487358 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86e41f4930 con 0x7f86e415e2a0 2024-03-24T00:49:43.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.573+0000 7f86e8aa1640 1 -- 172.21.15.67:0/1965487358 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86e404f400 con 0x7f86e415e2a0 2024-03-24T00:49:43.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.573+0000 7f86daffd640 1 -- 172.21.15.67:0/1965487358 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f86dc012b70 con 0x7f86e415e2a0 2024-03-24T00:49:43.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.574+0000 7f86daffd640 1 --2- 172.21.15.67:0/1965487358 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f86c406bbd0 0x7f86c406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:43.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.574+0000 7f86daffd640 1 -- 172.21.15.67:0/1965487358 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f86dc08ac30 con 0x7f86e415e2a0 2024-03-24T00:49:43.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.574+0000 7f86e259c640 1 --2- 172.21.15.67:0/1965487358 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f86c406bbd0 0x7f86c406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:43.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.574+0000 7f86e259c640 1 --2- 172.21.15.67:0/1965487358 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f86c406bbd0 0x7f86c406e090 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f86d40012e0 tx=0x7f86d4001a30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:43.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.578+0000 7f86daffd640 1 -- 172.21.15.67:0/1965487358 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f86dc059220 con 0x7f86e415e2a0 2024-03-24T00:49:43.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.761+0000 7f86e8aa1640 1 -- 172.21.15.67:0/1965487358 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f86e4161ee0 con 0x7f86e415e2a0 2024-03-24T00:49:43.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.761+0000 7f86daffd640 1 -- 172.21.15.67:0/1965487358 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v125) v1 ==== 74+0+13499 (secure 0 0 0) 0x7f86dc05d2c0 con 0x7f86e415e2a0 2024-03-24T00:49:43.761 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:43.761 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":125,"fsid":"de1d0178-808e-49ca-a986-e297e523b976","created":"2024-03-24T00:38:27.035958+0000","modified":"2024-03-24T00:49:40.665908+0000","last_up_change":"2024-03-24T00:38:38.373399+0000","last_in_change":"2024-03-24T00:38:29.992263+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:38:39.364950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd550ab-622a-4dfb-9caf-c261811d80a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6809","nonce":2628130519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6811","nonce":2628130519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6815","nonce":2628130519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2628130519},{"type":"v1","addr":"172.21.15.67:6813","nonce":2628130519}]},"public_addr":"172.21.15.67:6809/2628130519","cluster_addr":"172.21.15.67:6811/2628130519","heartbeat_back_addr":"172.21.15.67:6815/2628130519","heartbeat_front_addr":"172.21.15.67:6813/2628130519","state":["exists","up"]},{"osd":1,"uuid":"5ec859de-6637-4591-872a-1694d640bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6801","nonce":4062510404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6803","nonce":4062510404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6807","nonce":4062510404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4062510404},{"type":"v1","addr":"172.21.15.67:6805","nonce":4062510404}]},"public_addr":"172.21.15.67:6801/4062510404","cluster_addr":"172.21.15.67:6803/4062510404","heartbeat_back_addr":"172.21.15.67:6807/4062510404","heartbeat_front_addr":"172.21.15.67:6805/4062510404","state":["exists","up"]},{"osd":2,"uuid":"5984a001-8fe2-4555-a6f9-8dbfee8267e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6819","nonce":1268432561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6823","nonce":1268432561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6831","nonce":1268432561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":1268432561},{"type":"v1","addr":"172.21.15.67:6827","nonce":1268432561}]},"public_addr":"172.21.15.67:6819/1268432561","cluster_addr":"172.21.15.67:6823/1268432561","heartbeat_back_addr":"172.21.15.67:6831/1268432561","heartbeat_front_addr":"172.21.15.67:6827/1268432561","state":["exists","up"]},{"osd":3,"uuid":"29a7b5b0-5027-4578-a0c8-06025a48ef5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6818","nonce":3303489481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6822","nonce":3303489481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6830","nonce":3303489481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3303489481},{"type":"v1","addr":"172.21.15.67:6826","nonce":3303489481}]},"public_addr":"172.21.15.67:6818/3303489481","cluster_addr":"172.21.15.67:6822/3303489481","heartbeat_back_addr":"172.21.15.67:6830/3303489481","heartbeat_front_addr":"172.21.15.67:6826/3303489481","state":["exists","up"]},{"osd":4,"uuid":"4d47499f-402f-43e1-a57c-25209f3c763b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6809","nonce":477220793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6811","nonce":477220793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6815","nonce":477220793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":477220793},{"type":"v1","addr":"172.21.15.72:6813","nonce":477220793}]},"public_addr":"172.21.15.72:6809/477220793","cluster_addr":"172.21.15.72:6811/477220793","heartbeat_back_addr":"172.21.15.72:6815/477220793","heartbeat_front_addr":"172.21.15.72:6813/477220793","state":["exists","up"]},{"osd":5,"uuid":"b946d742-1ace-4dd5-a08d-79767c863893","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6801","nonce":288486206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6803","nonce":288486206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6807","nonce":288486206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":288486206},{"type":"v1","addr":"172.21.15.72:6805","nonce":288486206}]},"public_addr":"172.21.15.72:6801/288486206","cluster_addr":"172.21.15.72:6803/288486206","heartbeat_back_addr":"172.21.15.72:6807/288486206","heartbeat_front_addr":"172.21.15.72:6805/288486206","state":["exists","up"]},{"osd":6,"uuid":"93a33f2e-4084-4745-9938-3bb16b67ba37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6817","nonce":1725206562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6819","nonce":1725206562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6823","nonce":1725206562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":1725206562},{"type":"v1","addr":"172.21.15.72:6821","nonce":1725206562}]},"public_addr":"172.21.15.72:6817/1725206562","cluster_addr":"172.21.15.72:6819/1725206562","heartbeat_back_addr":"172.21.15.72:6823/1725206562","heartbeat_front_addr":"172.21.15.72:6821/1725206562","state":["exists","up"]},{"osd":7,"uuid":"7f138bad-98b0-4632-b528-e517933b1d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6825","nonce":2893038746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6827","nonce":2893038746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6831","nonce":2893038746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":2893038746},{"type":"v1","addr":"172.21.15.72:6829","nonce":2893038746}]},"public_addr":"172.21.15.72:6825/2893038746","cluster_addr":"172.21.15.72:6827/2893038746","heartbeat_back_addr":"172.21.15.72:6831/2893038746","heartbeat_front_addr":"172.21.15.72:6829/2893038746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.080224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.069927+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.111584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.157442+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.126086+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.116797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.191391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:38:35.140622+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6835/693801918":"2024-03-25T00:49:37.657184+0000","172.21.15.67:6833/693801918":"2024-03-25T00:49:37.657184+0000","172.21.15.67:6835/3833808901":"2024-03-25T00:49:23.316030+0000","172.21.15.67:6833/3833808901":"2024-03-25T00:49:23.316030+0000","172.21.15.67:6835/312063805":"2024-03-25T00:49:12.456364+0000","172.21.15.67:6833/312063805":"2024-03-25T00:49:12.456364+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:49:43.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.763+0000 7f86e8aa1640 1 -- 172.21.15.67:0/1965487358 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f86c406bbd0 msgr2=0x7f86c406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:43.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.763+0000 7f86e8aa1640 1 --2- 172.21.15.67:0/1965487358 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f86c406bbd0 0x7f86c406e090 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f86d40012e0 tx=0x7f86d4001a30 comp rx=0 tx=0).stop 2024-03-24T00:49:43.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.763+0000 7f86e8aa1640 1 -- 172.21.15.67:0/1965487358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86e415e2a0 msgr2=0x7f86e4158de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:43.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.763+0000 7f86e8aa1640 1 --2- 172.21.15.67:0/1965487358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86e415e2a0 0x7f86e4158de0 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f86dc00d5d0 tx=0x7f86dc00da90 comp rx=0 tx=0).stop 2024-03-24T00:49:43.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.763+0000 7f86e8aa1640 1 -- 172.21.15.67:0/1965487358 shutdown_connections 2024-03-24T00:49:43.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.763+0000 7f86e8aa1640 1 --2- 172.21.15.67:0/1965487358 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f86c406bbd0 0x7f86c406e090 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:43.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.763+0000 7f86e8aa1640 1 --2- 172.21.15.67:0/1965487358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86e415e2a0 0x7f86e4158de0 unknown :-1 s=CLOSED pgs=1931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:43.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.763+0000 7f86e8aa1640 1 --2- 172.21.15.67:0/1965487358 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f86e415dec0 0x7f86e41f43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:43.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.763+0000 7f86e8aa1640 1 --2- 172.21.15.67:0/1965487358 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f86e40d0ef0 0x7f86e4151f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:43.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.763+0000 7f86e8aa1640 1 -- 172.21.15.67:0/1965487358 >> 172.21.15.67:0/1965487358 conn(0x7f86e406c810 msgr2=0x7f86e4160c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:43.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.763+0000 7f86e8aa1640 1 -- 172.21.15.67:0/1965487358 shutdown_connections 2024-03-24T00:49:43.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:43.763+0000 7f86e8aa1640 1 -- 172.21.15.67:0/1965487358 wait complete. 2024-03-24T00:49:44.775 INFO:tasks.ceph:Scrubbing osd.0 2024-03-24T00:49:44.775 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-03-24T00:49:44.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.892+0000 7fe231962640 1 Processor -- start 2024-03-24T00:49:44.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.892+0000 7fe231962640 1 -- start start 2024-03-24T00:49:44.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.893+0000 7fe231962640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe22c0d0e60 0x7fe22c0b6230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:44.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.893+0000 7fe231962640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe22c161d60 con 0x7fe22c15e1b0 2024-03-24T00:49:44.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.893+0000 7fe231962640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe22c162040 con 0x7fe22c0b2d80 2024-03-24T00:49:44.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.893+0000 7fe231962640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe22c162970 con 0x7fe22c0d0e60 2024-03-24T00:49:44.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.893+0000 7fe22a7fc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe22c0d0e60 0x7fe22c0b6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:44.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.893+0000 7fe22affd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe22c15e1b0 0x7fe22c0d0750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56268/0 (socket says 172.21.15.67:56268) 2024-03-24T00:49:44.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.893+0000 7fe22affd640 1 -- 172.21.15.67:0/3750456752 learned_addr learned my addr 172.21.15.67:0/3750456752 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:49:44.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.893+0000 7fe22a7fc640 1 -- 172.21.15.67:0/3750456752 >> v1:172.21.15.72:6789/0 conn(0x7fe22c0b2d80 legacy=0x7fe22c0d1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:44.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.893+0000 7fe22a7fc640 1 -- 172.21.15.67:0/3750456752 >> v1:172.21.15.67:6789/0 conn(0x7fe22c15e1b0 legacy=0x7fe22c0d0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:44.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.893+0000 7fe22a7fc640 1 -- 172.21.15.67:0/3750456752 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe22c162320 con 0x7fe22c0d0e60 2024-03-24T00:49:44.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.893+0000 7fe229ffb640 1 -- 172.21.15.67:0/3750456752 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1311724244 0 0) 0x7fe22c161d60 con 0x7fe22c15e1b0 2024-03-24T00:49:44.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.894+0000 7fe22a7fc640 1 --2- 172.21.15.67:0/3750456752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe22c0d0e60 0x7fe22c0b6230 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7fe2140097a0 tx=0x7fe21402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e97ceb106a6f4d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:44.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.894+0000 7fe229ffb640 1 -- 172.21.15.67:0/3750456752 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe214035020 con 0x7fe22c0d0e60 2024-03-24T00:49:44.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.894+0000 7fe229ffb640 1 -- 172.21.15.67:0/3750456752 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe214031830 con 0x7fe22c0d0e60 2024-03-24T00:49:44.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.894+0000 7fe229ffb640 1 -- 172.21.15.67:0/3750456752 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe214031c00 con 0x7fe22c0d0e60 2024-03-24T00:49:44.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.895+0000 7fe231962640 1 -- 172.21.15.67:0/3750456752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe22c0d0e60 msgr2=0x7fe22c0b6230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:44.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.895+0000 7fe231962640 1 --2- 172.21.15.67:0/3750456752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe22c0d0e60 0x7fe22c0b6230 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7fe2140097a0 tx=0x7fe21402b8d0 comp rx=0 tx=0).stop 2024-03-24T00:49:44.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.895+0000 7fe231962640 1 -- 172.21.15.67:0/3750456752 shutdown_connections 2024-03-24T00:49:44.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.895+0000 7fe231962640 1 --2- 172.21.15.67:0/3750456752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe22c0d0e60 0x7fe22c0b6230 unknown :-1 s=CLOSED pgs=1933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:44.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.895+0000 7fe231962640 1 -- 172.21.15.67:0/3750456752 >> 172.21.15.67:0/3750456752 conn(0x7fe22c06c8f0 msgr2=0x7fe22c06ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:44.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.895+0000 7fe231962640 1 -- 172.21.15.67:0/3750456752 shutdown_connections 2024-03-24T00:49:44.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.895+0000 7fe231962640 1 -- 172.21.15.67:0/3750456752 wait complete. 2024-03-24T00:49:44.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.896+0000 7fe231962640 1 Processor -- start 2024-03-24T00:49:44.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.896+0000 7fe231962640 1 -- start start 2024-03-24T00:49:44.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.897+0000 7fe231962640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe22c0b2d80 0x7fe22c0c7840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:44.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.897+0000 7fe231962640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe22c0d0e60 0x7fe22c0c8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:44.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.897+0000 7fe231962640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe22c15e1b0 0x7fe22c0cc090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:44.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.897+0000 7fe231962640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe22c15e590 con 0x7fe22c0d0e60 2024-03-24T00:49:44.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.897+0000 7fe231962640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe22c0b3570 con 0x7fe22c15e1b0 2024-03-24T00:49:44.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.897+0000 7fe231962640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe22c0d26e0 con 0x7fe22c0b2d80 2024-03-24T00:49:44.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.897+0000 7fe22a7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe22c0d0e60 0x7fe22c0c8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:44.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.897+0000 7fe22a7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe22c0d0e60 0x7fe22c0c8bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43602/0 (socket says 172.21.15.67:43602) 2024-03-24T00:49:44.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.897+0000 7fe22a7fc640 1 -- 172.21.15.67:0/1429769082 learned_addr learned my addr 172.21.15.67:0/1429769082 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:44.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.897+0000 7fe22b7fe640 1 --2- 172.21.15.67:0/1429769082 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe22c15e1b0 0x7fe22c0cc090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:44.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.897+0000 7fe22affd640 1 --2- 172.21.15.67:0/1429769082 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe22c0b2d80 0x7fe22c0c7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:44.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.898+0000 7fe22a7fc640 1 -- 172.21.15.67:0/1429769082 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe22c0b2d80 msgr2=0x7fe22c0c7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:44.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.898+0000 7fe22a7fc640 1 --2- 172.21.15.67:0/1429769082 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe22c0b2d80 0x7fe22c0c7840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:44.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.898+0000 7fe22a7fc640 1 -- 172.21.15.67:0/1429769082 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe22c15e1b0 msgr2=0x7fe22c0cc090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:44.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.898+0000 7fe22a7fc640 1 --2- 172.21.15.67:0/1429769082 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe22c15e1b0 0x7fe22c0cc090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:44.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.898+0000 7fe22a7fc640 1 -- 172.21.15.67:0/1429769082 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe22c1f4250 con 0x7fe22c0d0e60 2024-03-24T00:49:44.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.898+0000 7fe22a7fc640 1 --2- 172.21.15.67:0/1429769082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe22c0d0e60 0x7fe22c0c8bb0 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7fe214000c00 tx=0x7fe2140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:44.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.898+0000 7fe223fff640 1 -- 172.21.15.67:0/1429769082 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe214035050 con 0x7fe22c0d0e60 2024-03-24T00:49:44.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.898+0000 7fe223fff640 1 -- 172.21.15.67:0/1429769082 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe214004230 con 0x7fe22c0d0e60 2024-03-24T00:49:44.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.898+0000 7fe231962640 1 -- 172.21.15.67:0/1429769082 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe22c1f4530 con 0x7fe22c0d0e60 2024-03-24T00:49:44.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.898+0000 7fe223fff640 1 -- 172.21.15.67:0/1429769082 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe21403b740 con 0x7fe22c0d0e60 2024-03-24T00:49:44.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.898+0000 7fe231962640 1 -- 172.21.15.67:0/1429769082 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe22c1f4af0 con 0x7fe22c0d0e60 2024-03-24T00:49:44.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.899+0000 7fe231962640 1 -- 172.21.15.67:0/1429769082 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe22c04f400 con 0x7fe22c0d0e60 2024-03-24T00:49:44.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.900+0000 7fe223fff640 1 -- 172.21.15.67:0/1429769082 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe214038020 con 0x7fe22c0d0e60 2024-03-24T00:49:44.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.900+0000 7fe223fff640 1 --2- 172.21.15.67:0/1429769082 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe20c06bbd0 0x7fe20c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:44.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.900+0000 7fe223fff640 1 -- 172.21.15.67:0/1429769082 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5507+0+0 (secure 0 0 0) 0x7fe2140ac6d0 con 0x7fe22c0d0e60 2024-03-24T00:49:44.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.900+0000 7fe223fff640 1 --2- 172.21.15.67:0/1429769082 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x7fe20c0721b0 0x7fe20c0745f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:44.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.900+0000 7fe223fff640 1 -- 172.21.15.67:0/1429769082 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe214031830 con 0x7fe20c0721b0 2024-03-24T00:49:44.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.900+0000 7fe223fff640 1 -- 172.21.15.67:0/1429769082 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=125) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe2140b3050 con 0x7fe22c0d0e60 2024-03-24T00:49:44.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.900+0000 7fe22affd640 1 --2- 172.21.15.67:0/1429769082 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe20c06bbd0 0x7fe20c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:44.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.900+0000 7fe22b7fe640 1 --2- 172.21.15.67:0/1429769082 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x7fe20c0721b0 0x7fe20c0745f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:44.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.901+0000 7fe22b7fe640 1 --2- 172.21.15.67:0/1429769082 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x7fe20c0721b0 0x7fe20c0745f0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:44.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.901+0000 7fe22affd640 1 --2- 172.21.15.67:0/1429769082 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe20c06bbd0 0x7fe20c06e090 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fe21c0012e0 tx=0x7fe21c004d40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:44.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.901+0000 7fe223fff640 1 -- 172.21.15.67:0/1429769082 <== osd.0 v2:172.21.15.67:6808/2628130519 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe214031830 con 0x7fe20c0721b0 2024-03-24T00:49:44.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.915+0000 7fe231962640 1 -- 172.21.15.67:0/1429769082 --> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fe22c0d1b80 con 0x7fe20c0721b0 2024-03-24T00:49:44.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.915+0000 7fe223fff640 1 -- 172.21.15.67:0/1429769082 <== osd.0 v2:172.21.15.67:6808/2628130519 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fe22c0d1b80 con 0x7fe20c0721b0 2024-03-24T00:49:44.915 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:49:44.915 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-24T00:49:44.915 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:49:44.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.915+0000 7fe231962640 1 -- 172.21.15.67:0/1429769082 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x7fe20c0721b0 msgr2=0x7fe20c0745f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:44.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.915+0000 7fe231962640 1 --2- 172.21.15.67:0/1429769082 >> [v2:172.21.15.67:6808/2628130519,v1:172.21.15.67:6809/2628130519] conn(0x7fe20c0721b0 0x7fe20c0745f0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:44.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.915+0000 7fe231962640 1 -- 172.21.15.67:0/1429769082 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe20c06bbd0 msgr2=0x7fe20c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:44.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.915+0000 7fe231962640 1 --2- 172.21.15.67:0/1429769082 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe20c06bbd0 0x7fe20c06e090 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fe21c0012e0 tx=0x7fe21c004d40 comp rx=0 tx=0).stop 2024-03-24T00:49:44.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.915+0000 7fe231962640 1 -- 172.21.15.67:0/1429769082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe22c0d0e60 msgr2=0x7fe22c0c8bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:44.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.915+0000 7fe231962640 1 --2- 172.21.15.67:0/1429769082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe22c0d0e60 0x7fe22c0c8bb0 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7fe214000c00 tx=0x7fe2140046c0 comp rx=0 tx=0).stop 2024-03-24T00:49:44.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.915+0000 7fe22b7fe640 1 -- 172.21.15.67:0/1429769082 reap_dead start 2024-03-24T00:49:44.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.915+0000 7fe231962640 1 -- 172.21.15.67:0/1429769082 shutdown_connections 2024-03-24T00:49:44.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.915+0000 7fe231962640 1 -- 172.21.15.67:0/1429769082 >> 172.21.15.67:0/1429769082 conn(0x7fe22c06c8f0 msgr2=0x7fe22c0ce020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:44.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.915+0000 7fe231962640 1 -- 172.21.15.67:0/1429769082 shutdown_connections 2024-03-24T00:49:44.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:44.915+0000 7fe231962640 1 -- 172.21.15.67:0/1429769082 wait complete. 2024-03-24T00:49:44.926 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-03-24T00:49:45.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.028+0000 7fcb1e190640 1 Processor -- start 2024-03-24T00:49:45.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.028+0000 7fcb1e190640 1 -- start start 2024-03-24T00:49:45.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.029+0000 7fcb1e190640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcb1815e0f0 0x7fcb180b95b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:45.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.029+0000 7fcb1e190640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcb180b6dc0 0x7fcb180b7210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:45.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.029+0000 7fcb1e190640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb180d2230 con 0x7fcb180b9af0 2024-03-24T00:49:45.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.029+0000 7fcb1e190640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb180b7750 con 0x7fcb1815e0f0 2024-03-24T00:49:45.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.029+0000 7fcb1e190640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb180d2510 con 0x7fcb180b6dc0 2024-03-24T00:49:45.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.029+0000 7fcb177fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcb1815e0f0 0x7fcb180b95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:45.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.029+0000 7fcb177fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcb1815e0f0 0x7fcb180b95b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43604/0 (socket says 172.21.15.67:43604) 2024-03-24T00:49:45.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.029+0000 7fcb177fe640 1 -- 172.21.15.67:0/1567717625 learned_addr learned my addr 172.21.15.67:0/1567717625 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:45.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.029+0000 7fcb17fff640 1 --2- 172.21.15.67:0/1567717625 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcb180b6dc0 0x7fcb180b7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:45.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.030+0000 7fcb177fe640 1 -- 172.21.15.67:0/1567717625 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcb180b6dc0 msgr2=0x7fcb180b7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:45.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.030+0000 7fcb177fe640 1 --2- 172.21.15.67:0/1567717625 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcb180b6dc0 0x7fcb180b7210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:45.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.030+0000 7fcb177fe640 1 -- 172.21.15.67:0/1567717625 >> v1:172.21.15.72:6789/0 conn(0x7fcb180b9af0 legacy=0x7fcb180d32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:45.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.030+0000 7fcb177fe640 1 -- 172.21.15.67:0/1567717625 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb180d27f0 con 0x7fcb1815e0f0 2024-03-24T00:49:45.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.030+0000 7fcb177fe640 1 --2- 172.21.15.67:0/1567717625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb1815e0f0 0x7fcb180b95b0 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7fcb08009e50 tx=0x7fcb0802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b4a03830331ccfb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:45.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.030+0000 7fcb167fc640 1 -- 172.21.15.67:0/1567717625 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb0802cb00 con 0x7fcb1815e0f0 2024-03-24T00:49:45.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.031+0000 7fcb167fc640 1 -- 172.21.15.67:0/1567717625 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb0802cca0 con 0x7fcb1815e0f0 2024-03-24T00:49:45.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.031+0000 7fcb167fc640 1 -- 172.21.15.67:0/1567717625 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb08034610 con 0x7fcb1815e0f0 2024-03-24T00:49:45.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.031+0000 7fcb1e190640 1 -- 172.21.15.67:0/1567717625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb1815e0f0 msgr2=0x7fcb180b95b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:45.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.031+0000 7fcb1e190640 1 --2- 172.21.15.67:0/1567717625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb1815e0f0 0x7fcb180b95b0 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7fcb08009e50 tx=0x7fcb0802bf60 comp rx=0 tx=0).stop 2024-03-24T00:49:45.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.031+0000 7fcb1e190640 1 -- 172.21.15.67:0/1567717625 shutdown_connections 2024-03-24T00:49:45.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.031+0000 7fcb1e190640 1 --2- 172.21.15.67:0/1567717625 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcb180b6dc0 0x7fcb180b7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:45.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.031+0000 7fcb1e190640 1 --2- 172.21.15.67:0/1567717625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb1815e0f0 0x7fcb180b95b0 unknown :-1 s=CLOSED pgs=1935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:45.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.031+0000 7fcb1e190640 1 -- 172.21.15.67:0/1567717625 >> 172.21.15.67:0/1567717625 conn(0x7fcb18072b60 msgr2=0x7fcb18072f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:45.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.031+0000 7fcb1e190640 1 -- 172.21.15.67:0/1567717625 shutdown_connections 2024-03-24T00:49:45.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.031+0000 7fcb1e190640 1 -- 172.21.15.67:0/1567717625 wait complete. 2024-03-24T00:49:45.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.032+0000 7fcb1e190640 1 Processor -- start 2024-03-24T00:49:45.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.032+0000 7fcb1e190640 1 -- start start 2024-03-24T00:49:45.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.033+0000 7fcb1e190640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcb180b6dc0 0x7fcb180aa720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:45.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.033+0000 7fcb1e190640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb180b9af0 0x7fcb180ac3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:45.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.033+0000 7fcb1e190640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb1815e0f0 0x7fcb18158ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:45.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.033+0000 7fcb1e190640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb180b31d0 con 0x7fcb1815e0f0 2024-03-24T00:49:45.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.033+0000 7fcb1e190640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb180d2db0 con 0x7fcb180b9af0 2024-03-24T00:49:45.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.033+0000 7fcb1e190640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb180b3350 con 0x7fcb180b6dc0 2024-03-24T00:49:45.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.033+0000 7fcb17fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb1815e0f0 0x7fcb18158ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:45.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.033+0000 7fcb17fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb1815e0f0 0x7fcb18158ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43616/0 (socket says 172.21.15.67:43616) 2024-03-24T00:49:45.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.033+0000 7fcb17fff640 1 -- 172.21.15.67:0/3945568988 learned_addr learned my addr 172.21.15.67:0/3945568988 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:45.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.033+0000 7fcb177fe640 1 --2- 172.21.15.67:0/3945568988 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcb180b6dc0 0x7fcb180aa720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:45.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.033+0000 7fcb16ffd640 1 --2- 172.21.15.67:0/3945568988 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb180b9af0 0x7fcb180ac3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:45.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.033+0000 7fcb17fff640 1 -- 172.21.15.67:0/3945568988 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcb180b6dc0 msgr2=0x7fcb180aa720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:45.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.034+0000 7fcb17fff640 1 --2- 172.21.15.67:0/3945568988 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcb180b6dc0 0x7fcb180aa720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:45.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.034+0000 7fcb17fff640 1 -- 172.21.15.67:0/3945568988 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb180b9af0 msgr2=0x7fcb180ac3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:45.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.034+0000 7fcb17fff640 1 --2- 172.21.15.67:0/3945568988 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb180b9af0 0x7fcb180ac3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:45.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.034+0000 7fcb17fff640 1 -- 172.21.15.67:0/3945568988 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb180aadb0 con 0x7fcb1815e0f0 2024-03-24T00:49:45.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.034+0000 7fcb17fff640 1 --2- 172.21.15.67:0/3945568988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb1815e0f0 0x7fcb18158ea0 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7fcb1000e930 tx=0x7fcb1000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:45.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.034+0000 7fcafffff640 1 -- 172.21.15.67:0/3945568988 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb10017070 con 0x7fcb1815e0f0 2024-03-24T00:49:45.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.034+0000 7fcafffff640 1 -- 172.21.15.67:0/3945568988 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb1000c980 con 0x7fcb1815e0f0 2024-03-24T00:49:45.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.034+0000 7fcb1e190640 1 -- 172.21.15.67:0/3945568988 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb180ab090 con 0x7fcb1815e0f0 2024-03-24T00:49:45.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.034+0000 7fcafffff640 1 -- 172.21.15.67:0/3945568988 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb1000cd70 con 0x7fcb1815e0f0 2024-03-24T00:49:45.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.034+0000 7fcb1e190640 1 -- 172.21.15.67:0/3945568988 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb180ab650 con 0x7fcb1815e0f0 2024-03-24T00:49:45.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.035+0000 7fcb1e190640 1 -- 172.21.15.67:0/3945568988 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb1804f400 con 0x7fcb1815e0f0 2024-03-24T00:49:45.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.035+0000 7fcafffff640 1 -- 172.21.15.67:0/3945568988 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fcb1001a070 con 0x7fcb1815e0f0 2024-03-24T00:49:45.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.036+0000 7fcafffff640 1 --2- 172.21.15.67:0/3945568988 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcb0406bbd0 0x7fcb0406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:45.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.036+0000 7fcafffff640 1 -- 172.21.15.67:0/3945568988 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5507+0+0 (secure 0 0 0) 0x7fcb1008a910 con 0x7fcb1815e0f0 2024-03-24T00:49:45.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.036+0000 7fcb177fe640 1 --2- 172.21.15.67:0/3945568988 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcb0406bbd0 0x7fcb0406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:45.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.036+0000 7fcb177fe640 1 --2- 172.21.15.67:0/3945568988 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcb0406bbd0 0x7fcb0406e090 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fcb08035000 tx=0x7fcb08030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:45.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.040+0000 7fcafffff640 1 -- 172.21.15.67:0/3945568988 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fcb10058f30 con 0x7fcb1815e0f0 2024-03-24T00:49:45.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.221+0000 7fcb1e190640 1 -- 172.21.15.67:0/3945568988 --> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7fcb180b29b0 con 0x7fcb0406bbd0 2024-03-24T00:49:45.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.222+0000 7fcafffff640 1 -- 172.21.15.67:0/3945568988 <== mgr.4105 v2:172.21.15.72:6832/2768292274 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fcb180b29b0 con 0x7fcb0406bbd0 2024-03-24T00:49:45.221 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 0 to deep-scrub 2024-03-24T00:49:45.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.223+0000 7fcb1e190640 1 -- 172.21.15.67:0/3945568988 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcb0406bbd0 msgr2=0x7fcb0406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:45.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.223+0000 7fcb1e190640 1 --2- 172.21.15.67:0/3945568988 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcb0406bbd0 0x7fcb0406e090 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fcb08035000 tx=0x7fcb08030000 comp rx=0 tx=0).stop 2024-03-24T00:49:45.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.223+0000 7fcb1e190640 1 -- 172.21.15.67:0/3945568988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb1815e0f0 msgr2=0x7fcb18158ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:45.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.223+0000 7fcb1e190640 1 --2- 172.21.15.67:0/3945568988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb1815e0f0 0x7fcb18158ea0 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7fcb1000e930 tx=0x7fcb1000edf0 comp rx=0 tx=0).stop 2024-03-24T00:49:45.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.223+0000 7fcb1e190640 1 -- 172.21.15.67:0/3945568988 shutdown_connections 2024-03-24T00:49:45.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.223+0000 7fcb1e190640 1 --2- 172.21.15.67:0/3945568988 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fcb0406bbd0 0x7fcb0406e090 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:45.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.223+0000 7fcb1e190640 1 --2- 172.21.15.67:0/3945568988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb1815e0f0 0x7fcb18158ea0 unknown :-1 s=CLOSED pgs=1936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:45.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.223+0000 7fcb1e190640 1 --2- 172.21.15.67:0/3945568988 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb180b9af0 0x7fcb180ac3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:45.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.223+0000 7fcb1e190640 1 --2- 172.21.15.67:0/3945568988 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcb180b6dc0 0x7fcb180aa720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:45.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.223+0000 7fcb1e190640 1 -- 172.21.15.67:0/3945568988 >> 172.21.15.67:0/3945568988 conn(0x7fcb18072b60 msgr2=0x7fcb18160570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:45.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.224+0000 7fcb1e190640 1 -- 172.21.15.67:0/3945568988 shutdown_connections 2024-03-24T00:49:45.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.224+0000 7fcb1e190640 1 -- 172.21.15.67:0/3945568988 wait complete. 2024-03-24T00:49:45.234 INFO:tasks.ceph:Scrubbing osd.1 2024-03-24T00:49:45.234 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-03-24T00:49:45.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.339+0000 7f78e5ea7640 1 Processor -- start 2024-03-24T00:49:45.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.339+0000 7f78e5ea7640 1 -- start start 2024-03-24T00:49:45.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.340+0000 7f78e5ea7640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f78e00cd420 0x7f78e0071ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:45.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.340+0000 7f78e5ea7640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f78e0072010 0x7f78e00cee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:45.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.340+0000 7f78e5ea7640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78e015c3c0 con 0x7f78e00cf340 2024-03-24T00:49:45.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.340+0000 7f78e5ea7640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78e015d250 con 0x7f78e0072010 2024-03-24T00:49:45.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.340+0000 7f78e5ea7640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78e00cf7d0 con 0x7f78e00cd420 2024-03-24T00:49:45.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.340+0000 7f78deffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f78e0072010 0x7f78e00cee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:45.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.340+0000 7f78dffff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f78e00cf340 0x7f78e015f9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56270/0 (socket says 172.21.15.67:56270) 2024-03-24T00:49:45.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.340+0000 7f78dffff640 1 -- 172.21.15.67:0/1437741627 learned_addr learned my addr 172.21.15.67:0/1437741627 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:49:45.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.340+0000 7f78df7fe640 1 --2- 172.21.15.67:0/1437741627 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f78e00cd420 0x7f78e0071ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:45.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.341+0000 7f78deffd640 1 -- 172.21.15.67:0/1437741627 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f78e00cd420 msgr2=0x7f78e0071ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:45.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.341+0000 7f78deffd640 1 --2- 172.21.15.67:0/1437741627 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f78e00cd420 0x7f78e0071ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:45.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.341+0000 7f78deffd640 1 -- 172.21.15.67:0/1437741627 >> v1:172.21.15.67:6789/0 conn(0x7f78e00cf340 legacy=0x7f78e015f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:45.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.341+0000 7f78deffd640 1 -- 172.21.15.67:0/1437741627 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78e015c640 con 0x7f78e0072010 2024-03-24T00:49:45.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.341+0000 7f78de7fc640 1 -- 172.21.15.67:0/1437741627 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1757777816 0 0) 0x7f78d8001490 con 0x7f78e00cf340 2024-03-24T00:49:45.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.341+0000 7f78deffd640 1 --2- 172.21.15.67:0/1437741627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78e0072010 0x7f78e00cee00 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f78d40097a0 tx=0x7f78d402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d620f4a142b8c751 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:45.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.341+0000 7f78de7fc640 1 -- 172.21.15.67:0/1437741627 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d4035020 con 0x7f78e0072010 2024-03-24T00:49:45.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.342+0000 7f78de7fc640 1 -- 172.21.15.67:0/1437741627 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78d4031830 con 0x7f78e0072010 2024-03-24T00:49:45.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.342+0000 7f78de7fc640 1 -- 172.21.15.67:0/1437741627 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d4031c00 con 0x7f78e0072010 2024-03-24T00:49:45.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.342+0000 7f78e5ea7640 1 -- 172.21.15.67:0/1437741627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78e0072010 msgr2=0x7f78e00cee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:45.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.342+0000 7f78e5ea7640 1 --2- 172.21.15.67:0/1437741627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78e0072010 0x7f78e00cee00 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f78d40097a0 tx=0x7f78d402b8d0 comp rx=0 tx=0).stop 2024-03-24T00:49:45.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.342+0000 7f78e5ea7640 1 -- 172.21.15.67:0/1437741627 shutdown_connections 2024-03-24T00:49:45.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.342+0000 7f78e5ea7640 1 --2- 172.21.15.67:0/1437741627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78e0072010 0x7f78e00cee00 unknown :-1 s=CLOSED pgs=1938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:45.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.342+0000 7f78e5ea7640 1 --2- 172.21.15.67:0/1437741627 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f78e00cd420 0x7f78e0071ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:45.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.342+0000 7f78e5ea7640 1 -- 172.21.15.67:0/1437741627 >> 172.21.15.67:0/1437741627 conn(0x7f78e00cf9e0 msgr2=0x7f78e00cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:45.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.342+0000 7f78e5ea7640 1 -- 172.21.15.67:0/1437741627 shutdown_connections 2024-03-24T00:49:45.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.342+0000 7f78e5ea7640 1 -- 172.21.15.67:0/1437741627 wait complete. 2024-03-24T00:49:45.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.343+0000 7f78e5ea7640 1 Processor -- start 2024-03-24T00:49:45.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.343+0000 7f78e5ea7640 1 -- start start 2024-03-24T00:49:45.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.344+0000 7f78e5ea7640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f78e0072010 0x7f78e0155330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:45.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.344+0000 7f78e5ea7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78e00cd420 0x7f78e0147360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:45.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.344+0000 7f78e5ea7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78e00cf340 0x7f78e01543e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:45.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.344+0000 7f78e5ea7640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78e015cee0 con 0x7f78e00cf340 2024-03-24T00:49:45.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.344+0000 7f78e5ea7640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78e015cc00 con 0x7f78e00cd420 2024-03-24T00:49:45.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.344+0000 7f78e5ea7640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78e00cd800 con 0x7f78e0072010 2024-03-24T00:49:45.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.344+0000 7f78dffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78e00cf340 0x7f78e01543e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:45.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.344+0000 7f78dffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78e00cf340 0x7f78e01543e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43640/0 (socket says 172.21.15.67:43640) 2024-03-24T00:49:45.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.344+0000 7f78dffff640 1 -- 172.21.15.67:0/2079765871 learned_addr learned my addr 172.21.15.67:0/2079765871 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:45.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.344+0000 7f78df7fe640 1 --2- 172.21.15.67:0/2079765871 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f78e0072010 0x7f78e0155330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:45.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.344+0000 7f78deffd640 1 --2- 172.21.15.67:0/2079765871 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78e00cd420 0x7f78e0147360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:45.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.344+0000 7f78dffff640 1 -- 172.21.15.67:0/2079765871 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f78e0072010 msgr2=0x7f78e0155330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:45.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.344+0000 7f78dffff640 1 --2- 172.21.15.67:0/2079765871 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f78e0072010 0x7f78e0155330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:45.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.344+0000 7f78dffff640 1 -- 172.21.15.67:0/2079765871 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78e00cd420 msgr2=0x7f78e0147360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:45.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.345+0000 7f78dffff640 1 --2- 172.21.15.67:0/2079765871 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78e00cd420 0x7f78e0147360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:45.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.345+0000 7f78dffff640 1 -- 172.21.15.67:0/2079765871 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78e0155870 con 0x7f78e00cf340 2024-03-24T00:49:45.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.345+0000 7f78dffff640 1 --2- 172.21.15.67:0/2079765871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78e00cf340 0x7f78e01543e0 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f78d800c730 tx=0x7f78d800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:45.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.345+0000 7f78c7fff640 1 -- 172.21.15.67:0/2079765871 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d8018070 con 0x7f78e00cf340 2024-03-24T00:49:45.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.345+0000 7f78c7fff640 1 -- 172.21.15.67:0/2079765871 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78d800ce40 con 0x7f78e00cf340 2024-03-24T00:49:45.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.345+0000 7f78c7fff640 1 -- 172.21.15.67:0/2079765871 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d800a860 con 0x7f78e00cf340 2024-03-24T00:49:45.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.345+0000 7f78e5ea7640 1 -- 172.21.15.67:0/2079765871 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78e0155b40 con 0x7f78e00cf340 2024-03-24T00:49:45.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.345+0000 7f78e5ea7640 1 -- 172.21.15.67:0/2079765871 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78e0156100 con 0x7f78e00cf340 2024-03-24T00:49:45.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.346+0000 7f78e5ea7640 1 -- 172.21.15.67:0/2079765871 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f78e004f400 con 0x7f78e00cf340 2024-03-24T00:49:45.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.346+0000 7f78c7fff640 1 -- 172.21.15.67:0/2079765871 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f78d800aa00 con 0x7f78e00cf340 2024-03-24T00:49:45.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.347+0000 7f78c7fff640 1 --2- 172.21.15.67:0/2079765871 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f78cc06bbd0 0x7f78cc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:45.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.347+0000 7f78c7fff640 1 -- 172.21.15.67:0/2079765871 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f78d8089ed0 con 0x7f78e00cf340 2024-03-24T00:49:45.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.347+0000 7f78c7fff640 1 --2- 172.21.15.67:0/2079765871 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x7f78cc0721b0 0x7f78cc0745f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:45.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.347+0000 7f78c7fff640 1 -- 172.21.15.67:0/2079765871 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f78cc074cc0 con 0x7f78cc0721b0 2024-03-24T00:49:45.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.347+0000 7f78c7fff640 1 -- 172.21.15.67:0/2079765871 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=125) v2 ==== 24+0+0 (secure 0 0 0) 0x7f78d808c5d0 con 0x7f78e00cf340 2024-03-24T00:49:45.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.347+0000 7f78df7fe640 1 --2- 172.21.15.67:0/2079765871 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f78cc06bbd0 0x7f78cc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:45.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.347+0000 7f78deffd640 1 --2- 172.21.15.67:0/2079765871 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x7f78cc0721b0 0x7f78cc0745f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:45.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.347+0000 7f78df7fe640 1 --2- 172.21.15.67:0/2079765871 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f78cc06bbd0 0x7f78cc06e090 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f78e00685b0 tx=0x7f78d0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:45.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.347+0000 7f78deffd640 1 --2- 172.21.15.67:0/2079765871 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x7f78cc0721b0 0x7f78cc0745f0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:45.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.348+0000 7f78c7fff640 1 -- 172.21.15.67:0/2079765871 <== osd.1 v2:172.21.15.67:6800/4062510404 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f78cc074cc0 con 0x7f78cc0721b0 2024-03-24T00:49:45.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.361+0000 7f78e5ea7640 1 -- 172.21.15.67:0/2079765871 --> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f78e00d0640 con 0x7f78cc0721b0 2024-03-24T00:49:45.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.361+0000 7f78c7fff640 1 -- 172.21.15.67:0/2079765871 <== osd.1 v2:172.21.15.67:6800/4062510404 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f78e00d0640 con 0x7f78cc0721b0 2024-03-24T00:49:45.361 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:49:45.362 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-24T00:49:45.362 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:49:45.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.362+0000 7f78e5ea7640 1 -- 172.21.15.67:0/2079765871 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x7f78cc0721b0 msgr2=0x7f78cc0745f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:45.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.362+0000 7f78e5ea7640 1 --2- 172.21.15.67:0/2079765871 >> [v2:172.21.15.67:6800/4062510404,v1:172.21.15.67:6801/4062510404] conn(0x7f78cc0721b0 0x7f78cc0745f0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:45.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.362+0000 7f78e5ea7640 1 -- 172.21.15.67:0/2079765871 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f78cc06bbd0 msgr2=0x7f78cc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:45.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.362+0000 7f78e5ea7640 1 --2- 172.21.15.67:0/2079765871 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f78cc06bbd0 0x7f78cc06e090 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f78e00685b0 tx=0x7f78d0009190 comp rx=0 tx=0).stop 2024-03-24T00:49:45.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.362+0000 7f78e5ea7640 1 -- 172.21.15.67:0/2079765871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78e00cf340 msgr2=0x7f78e01543e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:45.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.362+0000 7f78e5ea7640 1 --2- 172.21.15.67:0/2079765871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78e00cf340 0x7f78e01543e0 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f78d800c730 tx=0x7f78d800cbf0 comp rx=0 tx=0).stop 2024-03-24T00:49:45.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.362+0000 7f78dffff640 1 -- 172.21.15.67:0/2079765871 reap_dead start 2024-03-24T00:49:45.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.362+0000 7f78e5ea7640 1 -- 172.21.15.67:0/2079765871 shutdown_connections 2024-03-24T00:49:45.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.362+0000 7f78e5ea7640 1 -- 172.21.15.67:0/2079765871 >> 172.21.15.67:0/2079765871 conn(0x7f78e00cf9e0 msgr2=0x7f78e006b040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:45.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.362+0000 7f78e5ea7640 1 -- 172.21.15.67:0/2079765871 shutdown_connections 2024-03-24T00:49:45.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.362+0000 7f78e5ea7640 1 -- 172.21.15.67:0/2079765871 wait complete. 2024-03-24T00:49:45.372 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-03-24T00:49:45.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.475+0000 7f5c0af10640 1 Processor -- start 2024-03-24T00:49:45.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.475+0000 7f5c0af10640 1 -- start start 2024-03-24T00:49:45.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.476+0000 7f5c0af10640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5c0415af90 0x7f5c040b0730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:45.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.476+0000 7f5c0af10640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5c040cd0a0 0x7f5c040cd500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:45.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.476+0000 7f5c0af10640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5c040cefc0 0x7f5c040cf450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:45.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.476+0000 7f5c0af10640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c0415b370 con 0x7f5c040cefc0 2024-03-24T00:49:45.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.476+0000 7f5c0af10640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c040cb920 con 0x7f5c0415af90 2024-03-24T00:49:45.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.476+0000 7f5c0af10640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c040cdb20 con 0x7f5c040cd0a0 2024-03-24T00:49:45.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.476+0000 7f5c094ad640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5c040cefc0 0x7f5c040cf450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:45.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.476+0000 7f5c094ad640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5c040cefc0 0x7f5c040cf450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43642/0 (socket says 172.21.15.67:43642) 2024-03-24T00:49:45.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.476+0000 7f5c094ad640 1 -- 172.21.15.67:0/11939789 learned_addr learned my addr 172.21.15.67:0/11939789 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:45.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.476+0000 7f5c08cac640 1 --2- 172.21.15.67:0/11939789 >> v2:172.21.15.72:3300/0 conn(0x7f5c0415af90 0x7f5c040b0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:45.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.476+0000 7f5c03fff640 1 --2- 172.21.15.67:0/11939789 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5c040cd0a0 0x7f5c040cd500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:45.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.476+0000 7f5c094ad640 1 -- 172.21.15.67:0/11939789 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5c040cd0a0 msgr2=0x7f5c040cd500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:45.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.476+0000 7f5c094ad640 1 --2- 172.21.15.67:0/11939789 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5c040cd0a0 0x7f5c040cd500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:45.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.476+0000 7f5c094ad640 1 -- 172.21.15.67:0/11939789 >> v2:172.21.15.72:3300/0 conn(0x7f5c0415af90 msgr2=0x7f5c040b0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:45.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.476+0000 7f5c094ad640 1 --2- 172.21.15.67:0/11939789 >> v2:172.21.15.72:3300/0 conn(0x7f5c0415af90 0x7f5c040b0730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:45.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.476+0000 7f5c094ad640 1 -- 172.21.15.67:0/11939789 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c040cde00 con 0x7f5c040cefc0 2024-03-24T00:49:45.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.477+0000 7f5c094ad640 1 --2- 172.21.15.67:0/11939789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c040cefc0 0x7f5c040cf450 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f5c040d0940 tx=0x7f5bfc02e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5471bb059b953d4e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:45.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.477+0000 7f5c037fe640 1 -- 172.21.15.67:0/11939789 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bfc02ea80 con 0x7f5c040cefc0 2024-03-24T00:49:45.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.477+0000 7f5c037fe640 1 -- 172.21.15.67:0/11939789 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bfc02ec20 con 0x7f5c040cefc0 2024-03-24T00:49:45.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.477+0000 7f5c037fe640 1 -- 172.21.15.67:0/11939789 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bfc009600 con 0x7f5c040cefc0 2024-03-24T00:49:45.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.478+0000 7f5c0af10640 1 -- 172.21.15.67:0/11939789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c040cefc0 msgr2=0x7f5c040cf450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:45.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.478+0000 7f5c0af10640 1 --2- 172.21.15.67:0/11939789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c040cefc0 0x7f5c040cf450 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f5c040d0940 tx=0x7f5bfc02e0e0 comp rx=0 tx=0).stop 2024-03-24T00:49:45.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.478+0000 7f5c0af10640 1 -- 172.21.15.67:0/11939789 shutdown_connections 2024-03-24T00:49:45.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.478+0000 7f5c0af10640 1 --2- 172.21.15.67:0/11939789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c040cefc0 0x7f5c040cf450 unknown :-1 s=CLOSED pgs=1940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:45.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.478+0000 7f5c0af10640 1 --2- 172.21.15.67:0/11939789 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5c040cd0a0 0x7f5c040cd500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:45.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.478+0000 7f5c0af10640 1 --2- 172.21.15.67:0/11939789 >> v2:172.21.15.72:3300/0 conn(0x7f5c0415af90 0x7f5c040b0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:45.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.478+0000 7f5c0af10640 1 -- 172.21.15.67:0/11939789 >> 172.21.15.67:0/11939789 conn(0x7f5c04099a30 msgr2=0x7f5c04099e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:45.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.478+0000 7f5c0af10640 1 -- 172.21.15.67:0/11939789 shutdown_connections 2024-03-24T00:49:45.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.478+0000 7f5c0af10640 1 -- 172.21.15.67:0/11939789 wait complete. 2024-03-24T00:49:45.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.479+0000 7f5c0af10640 1 Processor -- start 2024-03-24T00:49:45.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.479+0000 7f5c0af10640 1 -- start start 2024-03-24T00:49:45.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.480+0000 7f5c0af10640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c040cd0a0 0x7f5c040c4510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:45.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.480+0000 7f5c0af10640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c040cefc0 0x7f5c04070120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:45.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.480+0000 7f5c0af10640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5c0415af90 0x7f5c040c6700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:45.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.480+0000 7f5c0af10640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c040cb580 con 0x7f5c040cefc0 2024-03-24T00:49:45.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.480+0000 7f5c0af10640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c040b0c70 con 0x7f5c040cd0a0 2024-03-24T00:49:45.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.480+0000 7f5c0af10640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c040ce9e0 con 0x7f5c0415af90 2024-03-24T00:49:45.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.480+0000 7f5c03fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c040cefc0 0x7f5c04070120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:45.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.480+0000 7f5c03fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c040cefc0 0x7f5c04070120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43654/0 (socket says 172.21.15.67:43654) 2024-03-24T00:49:45.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.480+0000 7f5c03fff640 1 -- 172.21.15.67:0/770260282 learned_addr learned my addr 172.21.15.67:0/770260282 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:45.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.480+0000 7f5c08cac640 1 --2- 172.21.15.67:0/770260282 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c040cd0a0 0x7f5c040c4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:45.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.480+0000 7f5c094ad640 1 --2- 172.21.15.67:0/770260282 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5c0415af90 0x7f5c040c6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:45.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.480+0000 7f5c03fff640 1 -- 172.21.15.67:0/770260282 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5c0415af90 msgr2=0x7f5c040c6700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:45.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.480+0000 7f5c03fff640 1 --2- 172.21.15.67:0/770260282 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5c0415af90 0x7f5c040c6700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:45.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.480+0000 7f5c03fff640 1 -- 172.21.15.67:0/770260282 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c040cd0a0 msgr2=0x7f5c040c4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:45.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.480+0000 7f5c03fff640 1 --2- 172.21.15.67:0/770260282 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c040cd0a0 0x7f5c040c4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:45.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.480+0000 7f5c03fff640 1 -- 172.21.15.67:0/770260282 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c040c4a50 con 0x7f5c040cefc0 2024-03-24T00:49:45.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.480+0000 7f5c03fff640 1 --2- 172.21.15.67:0/770260282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c040cefc0 0x7f5c04070120 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f5bf000e890 tx=0x7f5bf000ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:45.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.480+0000 7f5c017fa640 1 -- 172.21.15.67:0/770260282 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf0017070 con 0x7f5c040cefc0 2024-03-24T00:49:45.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.480+0000 7f5c017fa640 1 -- 172.21.15.67:0/770260282 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bf000c940 con 0x7f5c040cefc0 2024-03-24T00:49:45.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.481+0000 7f5c0af10640 1 -- 172.21.15.67:0/770260282 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c040c4cf0 con 0x7f5c040cefc0 2024-03-24T00:49:45.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.481+0000 7f5c017fa640 1 -- 172.21.15.67:0/770260282 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf000cd10 con 0x7f5c040cefc0 2024-03-24T00:49:45.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.481+0000 7f5c0af10640 1 -- 172.21.15.67:0/770260282 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c04157bf0 con 0x7f5c040cefc0 2024-03-24T00:49:45.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.482+0000 7f5c0af10640 1 -- 172.21.15.67:0/770260282 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c04157a10 con 0x7f5c040cefc0 2024-03-24T00:49:45.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.482+0000 7f5c017fa640 1 -- 172.21.15.67:0/770260282 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5bf001a070 con 0x7f5c040cefc0 2024-03-24T00:49:45.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.483+0000 7f5c017fa640 1 --2- 172.21.15.67:0/770260282 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5be806bbd0 0x7f5be806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:45.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.483+0000 7f5c017fa640 1 -- 172.21.15.67:0/770260282 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f5bf008a980 con 0x7f5c040cefc0 2024-03-24T00:49:45.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.483+0000 7f5c08cac640 1 --2- 172.21.15.67:0/770260282 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5be806bbd0 0x7f5be806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:45.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.483+0000 7f5c08cac640 1 --2- 172.21.15.67:0/770260282 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5be806bbd0 0x7f5be806e090 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f5bf8009aa0 tx=0x7f5bf8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:45.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.487+0000 7f5c017fa640 1 -- 172.21.15.67:0/770260282 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5c04157a10 con 0x7f5c040cefc0 2024-03-24T00:49:45.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.668+0000 7f5c0af10640 1 -- 172.21.15.67:0/770260282 --> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f5c040ceda0 con 0x7f5be806bbd0 2024-03-24T00:49:45.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.668+0000 7f5c017fa640 1 -- 172.21.15.67:0/770260282 <== mgr.4105 v2:172.21.15.72:6832/2768292274 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f5c040ceda0 con 0x7f5be806bbd0 2024-03-24T00:49:45.668 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 1 to deep-scrub 2024-03-24T00:49:45.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.670+0000 7f5c0af10640 1 -- 172.21.15.67:0/770260282 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5be806bbd0 msgr2=0x7f5be806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:45.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.670+0000 7f5c0af10640 1 --2- 172.21.15.67:0/770260282 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5be806bbd0 0x7f5be806e090 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f5bf8009aa0 tx=0x7f5bf8009190 comp rx=0 tx=0).stop 2024-03-24T00:49:45.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.670+0000 7f5c0af10640 1 -- 172.21.15.67:0/770260282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c040cefc0 msgr2=0x7f5c04070120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:45.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.670+0000 7f5c0af10640 1 --2- 172.21.15.67:0/770260282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c040cefc0 0x7f5c04070120 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f5bf000e890 tx=0x7f5bf000ed50 comp rx=0 tx=0).stop 2024-03-24T00:49:45.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.670+0000 7f5c0af10640 1 -- 172.21.15.67:0/770260282 shutdown_connections 2024-03-24T00:49:45.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.670+0000 7f5c0af10640 1 --2- 172.21.15.67:0/770260282 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f5be806bbd0 0x7f5be806e090 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:45.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.670+0000 7f5c0af10640 1 --2- 172.21.15.67:0/770260282 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5c0415af90 0x7f5c040c6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:45.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.671+0000 7f5c0af10640 1 --2- 172.21.15.67:0/770260282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c040cefc0 0x7f5c04070120 unknown :-1 s=CLOSED pgs=1941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:45.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.671+0000 7f5c0af10640 1 --2- 172.21.15.67:0/770260282 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c040cd0a0 0x7f5c040c4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:45.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.671+0000 7f5c0af10640 1 -- 172.21.15.67:0/770260282 >> 172.21.15.67:0/770260282 conn(0x7f5c04099a30 msgr2=0x7f5c0406b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:45.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.671+0000 7f5c0af10640 1 -- 172.21.15.67:0/770260282 shutdown_connections 2024-03-24T00:49:45.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.671+0000 7f5c0af10640 1 -- 172.21.15.67:0/770260282 wait complete. 2024-03-24T00:49:45.681 INFO:tasks.ceph:Scrubbing osd.2 2024-03-24T00:49:45.681 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-03-24T00:49:45.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.786+0000 7fe4b3537640 1 Processor -- start 2024-03-24T00:49:45.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.787+0000 7fe4b3537640 1 -- start start 2024-03-24T00:49:45.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.787+0000 7fe4b3537640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe4ac06cb60 0x7fe4ac09a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:45.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.787+0000 7fe4b3537640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe4ac06cf40 0x7fe4ac0cfe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:45.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.787+0000 7fe4b3537640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4ac0d06d0 con 0x7fe4ac0b2800 2024-03-24T00:49:45.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.787+0000 7fe4b3537640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4ac0d09b0 con 0x7fe4ac06cb60 2024-03-24T00:49:45.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.787+0000 7fe4b3537640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ac0d0c90 con 0x7fe4ac06cf40 2024-03-24T00:49:45.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.788+0000 7fe4b0ad2640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe4ac06cf40 0x7fe4ac0cfe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:45.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.788+0000 7fe4b12d3640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe4ac06cb60 0x7fe4ac09a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:45.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.788+0000 7fe4b0ad2640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe4ac06cf40 0x7fe4ac0cfe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.67:58958/0 (socket says 172.21.15.67:58958) 2024-03-24T00:49:45.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.788+0000 7fe4b1ad4640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fe4ac0b2800 0x7fe4ac0b2c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:39088/0 (socket says 172.21.15.67:39088) 2024-03-24T00:49:45.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.788+0000 7fe4b0ad2640 1 -- 172.21.15.67:0/3601182614 learned_addr learned my addr 172.21.15.67:0/3601182614 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:45.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.788+0000 7fe4b12d3640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe4ac06cb60 0x7fe4ac09a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:59200/0 (socket says 172.21.15.67:59200) 2024-03-24T00:49:45.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.788+0000 7fe4b0ad2640 1 -- 172.21.15.67:0/3601182614 >> v2:172.21.15.72:3300/0 conn(0x7fe4ac06cb60 msgr2=0x7fe4ac09a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:45.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.788+0000 7fe4b0ad2640 1 --2- 172.21.15.67:0/3601182614 >> v2:172.21.15.72:3300/0 conn(0x7fe4ac06cb60 0x7fe4ac09a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:45.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.788+0000 7fe4b0ad2640 1 -- 172.21.15.67:0/3601182614 >> v1:172.21.15.72:6789/0 conn(0x7fe4ac0b2800 legacy=0x7fe4ac0b2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:45.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.788+0000 7fe4b0ad2640 1 -- 172.21.15.67:0/3601182614 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4ac0d0f70 con 0x7fe4ac06cf40 2024-03-24T00:49:45.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.789+0000 7fe4b0ad2640 1 --2- 172.21.15.67:0/3601182614 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe4ac06cf40 0x7fe4ac0cfe10 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe4a0009870 tx=0x7fe4a002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a73fedc98f2b917e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:45.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.789+0000 7fe4abfff640 1 -- 172.21.15.67:0/3601182614 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a0035020 con 0x7fe4ac06cf40 2024-03-24T00:49:45.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.789+0000 7fe4abfff640 1 -- 172.21.15.67:0/3601182614 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4a0031640 con 0x7fe4ac06cf40 2024-03-24T00:49:45.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.789+0000 7fe4abfff640 1 -- 172.21.15.67:0/3601182614 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a002d040 con 0x7fe4ac06cf40 2024-03-24T00:49:45.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.789+0000 7fe4b3537640 1 -- 172.21.15.67:0/3601182614 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe4ac06cf40 msgr2=0x7fe4ac0cfe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:45.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.789+0000 7fe4b3537640 1 --2- 172.21.15.67:0/3601182614 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe4ac06cf40 0x7fe4ac0cfe10 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe4a0009870 tx=0x7fe4a002b8f0 comp rx=0 tx=0).stop 2024-03-24T00:49:45.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.789+0000 7fe4b3537640 1 -- 172.21.15.67:0/3601182614 shutdown_connections 2024-03-24T00:49:45.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.789+0000 7fe4b3537640 1 --2- 172.21.15.67:0/3601182614 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe4ac06cf40 0x7fe4ac0cfe10 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:45.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.789+0000 7fe4b3537640 1 --2- 172.21.15.67:0/3601182614 >> v2:172.21.15.72:3300/0 conn(0x7fe4ac06cb60 0x7fe4ac09a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:45.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.789+0000 7fe4b3537640 1 -- 172.21.15.67:0/3601182614 >> 172.21.15.67:0/3601182614 conn(0x7fe4ac0b36c0 msgr2=0x7fe4ac0b3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:45.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.790+0000 7fe4b3537640 1 -- 172.21.15.67:0/3601182614 shutdown_connections 2024-03-24T00:49:45.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.790+0000 7fe4b3537640 1 -- 172.21.15.67:0/3601182614 wait complete. 2024-03-24T00:49:45.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.791+0000 7fe4b3537640 1 Processor -- start 2024-03-24T00:49:45.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.791+0000 7fe4b3537640 1 -- start start 2024-03-24T00:49:45.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.791+0000 7fe4b3537640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4ac06cb60 0x7fe4ac0c5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:45.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.791+0000 7fe4b3537640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe4ac06cf40 0x7fe4ac157d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:45.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.791+0000 7fe4b3537640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe4ac0b2800 0x7fe4ac0b65c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:45.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.791+0000 7fe4b3537640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ac0cd230 con 0x7fe4ac06cb60 2024-03-24T00:49:45.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.791+0000 7fe4b3537640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ac0d1770 con 0x7fe4ac0b2800 2024-03-24T00:49:45.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.791+0000 7fe4b3537640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ac06d3c0 con 0x7fe4ac06cf40 2024-03-24T00:49:45.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.792+0000 7fe4b12d3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4ac06cb60 0x7fe4ac0c5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:45.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.792+0000 7fe4b12d3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4ac06cb60 0x7fe4ac0c5fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43662/0 (socket says 172.21.15.67:43662) 2024-03-24T00:49:45.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.792+0000 7fe4b12d3640 1 -- 172.21.15.67:0/610545288 learned_addr learned my addr 172.21.15.67:0/610545288 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:45.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.792+0000 7fe4b1ad4640 1 --2- 172.21.15.67:0/610545288 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe4ac0b2800 0x7fe4ac0b65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:45.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.792+0000 7fe4b0ad2640 1 --2- 172.21.15.67:0/610545288 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe4ac06cf40 0x7fe4ac157d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:45.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.792+0000 7fe4b12d3640 1 -- 172.21.15.67:0/610545288 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe4ac06cf40 msgr2=0x7fe4ac157d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:45.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.792+0000 7fe4b12d3640 1 --2- 172.21.15.67:0/610545288 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe4ac06cf40 0x7fe4ac157d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:45.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.792+0000 7fe4b12d3640 1 -- 172.21.15.67:0/610545288 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe4ac0b2800 msgr2=0x7fe4ac0b65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:45.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.792+0000 7fe4b12d3640 1 --2- 172.21.15.67:0/610545288 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe4ac0b2800 0x7fe4ac0b65c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:45.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.792+0000 7fe4b12d3640 1 -- 172.21.15.67:0/610545288 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4ac0c64e0 con 0x7fe4ac06cb60 2024-03-24T00:49:45.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.792+0000 7fe4b12d3640 1 --2- 172.21.15.67:0/610545288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4ac06cb60 0x7fe4ac0c5fa0 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7fe4a400e870 tx=0x7fe4a400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:45.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.792+0000 7fe4a9ffb640 1 -- 172.21.15.67:0/610545288 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a4017070 con 0x7fe4ac06cb60 2024-03-24T00:49:45.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.792+0000 7fe4a9ffb640 1 -- 172.21.15.67:0/610545288 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4a400c940 con 0x7fe4ac06cb60 2024-03-24T00:49:45.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.792+0000 7fe4b3537640 1 -- 172.21.15.67:0/610545288 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4ac0c73e0 con 0x7fe4ac06cb60 2024-03-24T00:49:45.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.792+0000 7fe4a9ffb640 1 -- 172.21.15.67:0/610545288 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a400cd30 con 0x7fe4ac06cb60 2024-03-24T00:49:45.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.792+0000 7fe4b3537640 1 -- 172.21.15.67:0/610545288 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4ac158f20 con 0x7fe4ac06cb60 2024-03-24T00:49:45.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.793+0000 7fe4b3537640 1 -- 172.21.15.67:0/610545288 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe4ac04f400 con 0x7fe4ac06cb60 2024-03-24T00:49:45.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.794+0000 7fe4a9ffb640 1 -- 172.21.15.67:0/610545288 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe4a401a070 con 0x7fe4ac06cb60 2024-03-24T00:49:45.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.794+0000 7fe4a9ffb640 1 --2- 172.21.15.67:0/610545288 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe49806bbd0 0x7fe49806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:45.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.794+0000 7fe4a9ffb640 1 -- 172.21.15.67:0/610545288 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5507+0+0 (secure 0 0 0) 0x7fe4a408a730 con 0x7fe4ac06cb60 2024-03-24T00:49:45.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.794+0000 7fe4a9ffb640 1 --2- 172.21.15.67:0/610545288 >> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] conn(0x7fe4980721b0 0x7fe4980745f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:45.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.794+0000 7fe4a9ffb640 1 -- 172.21.15.67:0/610545288 --> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe4a4004470 con 0x7fe4980721b0 2024-03-24T00:49:45.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.794+0000 7fe4b0ad2640 1 --2- 172.21.15.67:0/610545288 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe49806bbd0 0x7fe49806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:45.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.794+0000 7fe4a9ffb640 1 -- 172.21.15.67:0/610545288 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=125) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe4a4091050 con 0x7fe4ac06cb60 2024-03-24T00:49:45.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.794+0000 7fe4b1ad4640 1 --2- 172.21.15.67:0/610545288 >> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] conn(0x7fe4980721b0 0x7fe4980745f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:45.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.795+0000 7fe4b1ad4640 1 --2- 172.21.15.67:0/610545288 >> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] conn(0x7fe4980721b0 0x7fe4980745f0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:45.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.795+0000 7fe4b0ad2640 1 --2- 172.21.15.67:0/610545288 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe49806bbd0 0x7fe49806e090 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fe4a0036000 tx=0x7fe4a003c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:45.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.795+0000 7fe4a9ffb640 1 -- 172.21.15.67:0/610545288 <== osd.2 v2:172.21.15.67:6817/1268432561 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe4a4004470 con 0x7fe4980721b0 2024-03-24T00:49:45.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.809+0000 7fe4b3537640 1 -- 172.21.15.67:0/610545288 --> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fe4ac159360 con 0x7fe4980721b0 2024-03-24T00:49:45.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.809+0000 7fe4a9ffb640 1 -- 172.21.15.67:0/610545288 <== osd.2 v2:172.21.15.67:6817/1268432561 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fe4ac159360 con 0x7fe4980721b0 2024-03-24T00:49:45.809 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:49:45.809 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-24T00:49:45.809 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:49:45.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.809+0000 7fe4b3537640 1 -- 172.21.15.67:0/610545288 >> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] conn(0x7fe4980721b0 msgr2=0x7fe4980745f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:45.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.809+0000 7fe4b3537640 1 --2- 172.21.15.67:0/610545288 >> [v2:172.21.15.67:6817/1268432561,v1:172.21.15.67:6819/1268432561] conn(0x7fe4980721b0 0x7fe4980745f0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:45.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.809+0000 7fe4b3537640 1 -- 172.21.15.67:0/610545288 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe49806bbd0 msgr2=0x7fe49806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:45.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.809+0000 7fe4b3537640 1 --2- 172.21.15.67:0/610545288 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fe49806bbd0 0x7fe49806e090 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fe4a0036000 tx=0x7fe4a003c040 comp rx=0 tx=0).stop 2024-03-24T00:49:45.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.809+0000 7fe4b3537640 1 -- 172.21.15.67:0/610545288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4ac06cb60 msgr2=0x7fe4ac0c5fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:45.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.809+0000 7fe4b3537640 1 --2- 172.21.15.67:0/610545288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4ac06cb60 0x7fe4ac0c5fa0 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7fe4a400e870 tx=0x7fe4a400ed30 comp rx=0 tx=0).stop 2024-03-24T00:49:45.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.809+0000 7fe4b1ad4640 1 -- 172.21.15.67:0/610545288 reap_dead start 2024-03-24T00:49:45.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.809+0000 7fe4b3537640 1 -- 172.21.15.67:0/610545288 shutdown_connections 2024-03-24T00:49:45.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.809+0000 7fe4b3537640 1 -- 172.21.15.67:0/610545288 >> 172.21.15.67:0/610545288 conn(0x7fe4ac0b36c0 msgr2=0x7fe4ac160540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:45.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.809+0000 7fe4b3537640 1 -- 172.21.15.67:0/610545288 shutdown_connections 2024-03-24T00:49:45.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.809+0000 7fe4b3537640 1 -- 172.21.15.67:0/610545288 wait complete. 2024-03-24T00:49:45.819 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-03-24T00:49:45.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.921+0000 7f077fc6c640 1 Processor -- start 2024-03-24T00:49:45.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.921+0000 7f077fc6c640 1 -- start start 2024-03-24T00:49:45.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.922+0000 7f077fc6c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f077815e0f0 0x7f07780b95b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:45.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.922+0000 7f077fc6c640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f07780b9af0 0x7f07780d32d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:45.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.922+0000 7f077fc6c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f07780d3810 0x7f07780b6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:45.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.922+0000 7f077fc6c640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07780d2230 con 0x7f07780d3810 2024-03-24T00:49:45.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.922+0000 7f077fc6c640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07780d2510 con 0x7f077815e0f0 2024-03-24T00:49:45.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.922+0000 7f077fc6c640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07780d27f0 con 0x7f07780b9af0 2024-03-24T00:49:45.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.922+0000 7f077e209640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f07780d3810 0x7f07780b6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:45.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.922+0000 7f077e209640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f07780d3810 0x7f07780b6dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43674/0 (socket says 172.21.15.67:43674) 2024-03-24T00:49:45.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.922+0000 7f077e209640 1 -- 172.21.15.67:0/1571620047 learned_addr learned my addr 172.21.15.67:0/1571620047 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:45.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.922+0000 7f077da08640 1 --2- 172.21.15.67:0/1571620047 >> v2:172.21.15.72:3300/0 conn(0x7f077815e0f0 0x7f07780b95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:45.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.922+0000 7f077d207640 1 --2- 172.21.15.67:0/1571620047 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f07780b9af0 0x7f07780d32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:45.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.923+0000 7f077e209640 1 -- 172.21.15.67:0/1571620047 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f07780b9af0 msgr2=0x7f07780d32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:45.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.923+0000 7f077e209640 1 --2- 172.21.15.67:0/1571620047 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f07780b9af0 0x7f07780d32d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:45.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.923+0000 7f077e209640 1 -- 172.21.15.67:0/1571620047 >> v2:172.21.15.72:3300/0 conn(0x7f077815e0f0 msgr2=0x7f07780b95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:45.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.923+0000 7f077e209640 1 --2- 172.21.15.67:0/1571620047 >> v2:172.21.15.72:3300/0 conn(0x7f077815e0f0 0x7f07780b95b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:45.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.923+0000 7f077e209640 1 -- 172.21.15.67:0/1571620047 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07780d2ad0 con 0x7f07780d3810 2024-03-24T00:49:45.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.923+0000 7f077e209640 1 --2- 172.21.15.67:0/1571620047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07780d3810 0x7f07780b6dc0 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f077808cc00 tx=0x7f077002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6870c05fe33e7dd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:45.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.923+0000 7f077ca06640 1 -- 172.21.15.67:0/1571620047 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0770035020 con 0x7f07780d3810 2024-03-24T00:49:45.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.924+0000 7f077ca06640 1 -- 172.21.15.67:0/1571620047 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0770031870 con 0x7f07780d3810 2024-03-24T00:49:45.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.924+0000 7f077ca06640 1 -- 172.21.15.67:0/1571620047 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0770031c40 con 0x7f07780d3810 2024-03-24T00:49:45.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.924+0000 7f077fc6c640 1 -- 172.21.15.67:0/1571620047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07780d3810 msgr2=0x7f07780b6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:45.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.924+0000 7f077fc6c640 1 --2- 172.21.15.67:0/1571620047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07780d3810 0x7f07780b6dc0 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f077808cc00 tx=0x7f077002b910 comp rx=0 tx=0).stop 2024-03-24T00:49:45.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.924+0000 7f077fc6c640 1 -- 172.21.15.67:0/1571620047 shutdown_connections 2024-03-24T00:49:45.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.924+0000 7f077fc6c640 1 --2- 172.21.15.67:0/1571620047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07780d3810 0x7f07780b6dc0 unknown :-1 s=CLOSED pgs=1943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:45.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.924+0000 7f077fc6c640 1 --2- 172.21.15.67:0/1571620047 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f07780b9af0 0x7f07780d32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:45.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.924+0000 7f077fc6c640 1 --2- 172.21.15.67:0/1571620047 >> v2:172.21.15.72:3300/0 conn(0x7f077815e0f0 0x7f07780b95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:45.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.924+0000 7f077fc6c640 1 -- 172.21.15.67:0/1571620047 >> 172.21.15.67:0/1571620047 conn(0x7f0778072b60 msgr2=0x7f0778072f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:45.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.924+0000 7f077fc6c640 1 -- 172.21.15.67:0/1571620047 shutdown_connections 2024-03-24T00:49:45.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.924+0000 7f077fc6c640 1 -- 172.21.15.67:0/1571620047 wait complete. 2024-03-24T00:49:45.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.925+0000 7f077fc6c640 1 Processor -- start 2024-03-24T00:49:45.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.925+0000 7f077fc6c640 1 -- start start 2024-03-24T00:49:45.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.926+0000 7f077fc6c640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f07780b9af0 0x7f07781f4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:45.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.926+0000 7f077fc6c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07780d3810 0x7f07780cc690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:45.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.926+0000 7f077fc6c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f077815e0f0 0x7f07780b5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:45.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.926+0000 7f077fc6c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07780b3350 con 0x7f077815e0f0 2024-03-24T00:49:45.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.926+0000 7f077fc6c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07780b31d0 con 0x7f07780d3810 2024-03-24T00:49:45.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.926+0000 7f077fc6c640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07780b7300 con 0x7f07780b9af0 2024-03-24T00:49:45.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.926+0000 7f077e209640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f077815e0f0 0x7f07780b5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:45.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.926+0000 7f077e209640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f077815e0f0 0x7f07780b5cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43686/0 (socket says 172.21.15.67:43686) 2024-03-24T00:49:45.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.926+0000 7f077e209640 1 -- 172.21.15.67:0/2536522569 learned_addr learned my addr 172.21.15.67:0/2536522569 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:45.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.926+0000 7f077d207640 1 --2- 172.21.15.67:0/2536522569 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07780d3810 0x7f07780cc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:45.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.926+0000 7f077da08640 1 --2- 172.21.15.67:0/2536522569 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f07780b9af0 0x7f07781f4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:45.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.926+0000 7f077e209640 1 -- 172.21.15.67:0/2536522569 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f07780b9af0 msgr2=0x7f07781f4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:45.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.926+0000 7f077e209640 1 --2- 172.21.15.67:0/2536522569 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f07780b9af0 0x7f07781f4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:45.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.926+0000 7f077e209640 1 -- 172.21.15.67:0/2536522569 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07780d3810 msgr2=0x7f07780cc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:45.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.926+0000 7f077e209640 1 --2- 172.21.15.67:0/2536522569 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07780d3810 0x7f07780cc690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:45.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.926+0000 7f077e209640 1 -- 172.21.15.67:0/2536522569 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07780cbe10 con 0x7f077815e0f0 2024-03-24T00:49:45.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.927+0000 7f077e209640 1 --2- 172.21.15.67:0/2536522569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f077815e0f0 0x7f07780b5cf0 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f0770009960 tx=0x7f07700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:45.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.927+0000 7f07767fc640 1 -- 172.21.15.67:0/2536522569 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0770040070 con 0x7f077815e0f0 2024-03-24T00:49:45.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.927+0000 7f07767fc640 1 -- 172.21.15.67:0/2536522569 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0770030c20 con 0x7f077815e0f0 2024-03-24T00:49:45.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.927+0000 7f077fc6c640 1 -- 172.21.15.67:0/2536522569 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07780b34d0 con 0x7f077815e0f0 2024-03-24T00:49:45.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.927+0000 7f07767fc640 1 -- 172.21.15.67:0/2536522569 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0770035040 con 0x7f077815e0f0 2024-03-24T00:49:45.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.927+0000 7f077fc6c640 1 -- 172.21.15.67:0/2536522569 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07780b3a90 con 0x7f077815e0f0 2024-03-24T00:49:45.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.928+0000 7f077fc6c640 1 -- 172.21.15.67:0/2536522569 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f077804f400 con 0x7f077815e0f0 2024-03-24T00:49:45.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.928+0000 7f07767fc640 1 -- 172.21.15.67:0/2536522569 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0770038020 con 0x7f077815e0f0 2024-03-24T00:49:45.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.929+0000 7f07767fc640 1 --2- 172.21.15.67:0/2536522569 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f076406bc00 0x7f076406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:45.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.929+0000 7f07767fc640 1 -- 172.21.15.67:0/2536522569 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f07700ac590 con 0x7f077815e0f0 2024-03-24T00:49:45.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.929+0000 7f077da08640 1 --2- 172.21.15.67:0/2536522569 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f076406bc00 0x7f076406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:45.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.929+0000 7f077da08640 1 --2- 172.21.15.67:0/2536522569 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f076406bc00 0x7f076406e0c0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f0768011340 tx=0x7f076800b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:45.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:45.933+0000 7f07767fc640 1 -- 172.21.15.67:0/2536522569 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f077007aaf0 con 0x7f077815e0f0 2024-03-24T00:49:46.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.112+0000 7f077fc6c640 1 -- 172.21.15.67:0/2536522569 --> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f07780b2c90 con 0x7f076406bc00 2024-03-24T00:49:46.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.113+0000 7f07767fc640 1 -- 172.21.15.67:0/2536522569 <== mgr.4105 v2:172.21.15.72:6832/2768292274 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f07780b2c90 con 0x7f076406bc00 2024-03-24T00:49:46.112 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 2 to deep-scrub 2024-03-24T00:49:46.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.114+0000 7f077fc6c640 1 -- 172.21.15.67:0/2536522569 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f076406bc00 msgr2=0x7f076406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:46.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.114+0000 7f077fc6c640 1 --2- 172.21.15.67:0/2536522569 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f076406bc00 0x7f076406e0c0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f0768011340 tx=0x7f076800b040 comp rx=0 tx=0).stop 2024-03-24T00:49:46.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.114+0000 7f077fc6c640 1 -- 172.21.15.67:0/2536522569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f077815e0f0 msgr2=0x7f07780b5cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:46.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.114+0000 7f077fc6c640 1 --2- 172.21.15.67:0/2536522569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f077815e0f0 0x7f07780b5cf0 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f0770009960 tx=0x7f07700046c0 comp rx=0 tx=0).stop 2024-03-24T00:49:46.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.114+0000 7f077fc6c640 1 -- 172.21.15.67:0/2536522569 shutdown_connections 2024-03-24T00:49:46.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.114+0000 7f077fc6c640 1 --2- 172.21.15.67:0/2536522569 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f076406bc00 0x7f076406e0c0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:46.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.114+0000 7f077fc6c640 1 --2- 172.21.15.67:0/2536522569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f077815e0f0 0x7f07780b5cf0 unknown :-1 s=CLOSED pgs=1944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:46.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.114+0000 7f077fc6c640 1 --2- 172.21.15.67:0/2536522569 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07780d3810 0x7f07780cc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:46.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.114+0000 7f077fc6c640 1 --2- 172.21.15.67:0/2536522569 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f07780b9af0 0x7f07781f4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:46.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.114+0000 7f077fc6c640 1 -- 172.21.15.67:0/2536522569 >> 172.21.15.67:0/2536522569 conn(0x7f0778072b60 msgr2=0x7f07780b9100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:46.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.114+0000 7f077fc6c640 1 -- 172.21.15.67:0/2536522569 shutdown_connections 2024-03-24T00:49:46.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.114+0000 7f077fc6c640 1 -- 172.21.15.67:0/2536522569 wait complete. 2024-03-24T00:49:46.125 INFO:tasks.ceph:Scrubbing osd.3 2024-03-24T00:49:46.125 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-03-24T00:49:46.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.230+0000 7f57aea27640 1 Processor -- start 2024-03-24T00:49:46.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.230+0000 7f57aea27640 1 -- start start 2024-03-24T00:49:46.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.231+0000 7f57aea27640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f57a806cb60 0x7f57a809a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:46.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.231+0000 7f57aea27640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57a80d06d0 con 0x7f57a80b2800 2024-03-24T00:49:46.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.231+0000 7f57aea27640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57a80d09b0 con 0x7f57a806cf40 2024-03-24T00:49:46.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.231+0000 7f57aea27640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57a80d0c90 con 0x7f57a806cb60 2024-03-24T00:49:46.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.231+0000 7f57a7fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f57a806cb60 0x7f57a809a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:46.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.231+0000 7f57acfc4640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f57a80b2800 0x7f57a80b2c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56272/0 (socket says 172.21.15.67:56272) 2024-03-24T00:49:46.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.231+0000 7f57acfc4640 1 -- 172.21.15.67:0/4222155750 learned_addr learned my addr 172.21.15.67:0/4222155750 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:49:46.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.232+0000 7f57a7fff640 1 -- 172.21.15.67:0/4222155750 >> v1:172.21.15.72:6789/0 conn(0x7f57a806cf40 legacy=0x7f57a80cfe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:46.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.232+0000 7f57a7fff640 1 -- 172.21.15.67:0/4222155750 >> v1:172.21.15.67:6789/0 conn(0x7f57a80b2800 legacy=0x7f57a80b2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:46.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.232+0000 7f57a7fff640 1 -- 172.21.15.67:0/4222155750 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57a80d0f70 con 0x7f57a806cb60 2024-03-24T00:49:46.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.232+0000 7f57a6ffd640 1 -- 172.21.15.67:0/4222155750 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1589918826 0 0) 0x7f57a80d06d0 con 0x7f57a80b2800 2024-03-24T00:49:46.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.232+0000 7f57a7fff640 1 --2- 172.21.15.67:0/4222155750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57a806cb60 0x7f57a809a780 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7f579c0097a0 tx=0x7f579c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8536bbf204017e24 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:46.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.232+0000 7f57a6ffd640 1 -- 172.21.15.67:0/4222155750 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f579c035020 con 0x7f57a806cb60 2024-03-24T00:49:46.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.233+0000 7f57a6ffd640 1 -- 172.21.15.67:0/4222155750 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f579c031830 con 0x7f57a806cb60 2024-03-24T00:49:46.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.233+0000 7f57a6ffd640 1 -- 172.21.15.67:0/4222155750 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f579c031c00 con 0x7f57a806cb60 2024-03-24T00:49:46.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.233+0000 7f57aea27640 1 -- 172.21.15.67:0/4222155750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57a806cb60 msgr2=0x7f57a809a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:46.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.233+0000 7f57aea27640 1 --2- 172.21.15.67:0/4222155750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57a806cb60 0x7f57a809a780 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7f579c0097a0 tx=0x7f579c02b8d0 comp rx=0 tx=0).stop 2024-03-24T00:49:46.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.233+0000 7f57aea27640 1 -- 172.21.15.67:0/4222155750 shutdown_connections 2024-03-24T00:49:46.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.233+0000 7f57aea27640 1 --2- 172.21.15.67:0/4222155750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57a806cb60 0x7f57a809a780 unknown :-1 s=CLOSED pgs=1946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:46.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.233+0000 7f57aea27640 1 -- 172.21.15.67:0/4222155750 >> 172.21.15.67:0/4222155750 conn(0x7f57a80b36c0 msgr2=0x7f57a80b3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:46.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.233+0000 7f57aea27640 1 -- 172.21.15.67:0/4222155750 shutdown_connections 2024-03-24T00:49:46.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.233+0000 7f57aea27640 1 -- 172.21.15.67:0/4222155750 wait complete. 2024-03-24T00:49:46.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.234+0000 7f57aea27640 1 Processor -- start 2024-03-24T00:49:46.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.234+0000 7f57aea27640 1 -- start start 2024-03-24T00:49:46.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.235+0000 7f57aea27640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57a806cb60 0x7f57a814e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:46.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.235+0000 7f57aea27640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f57a806cf40 0x7f57a80b0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:46.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.235+0000 7f57aea27640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57a80b2800 0x7f57a80b0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:46.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.235+0000 7f57aea27640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a80cd230 con 0x7f57a80b2800 2024-03-24T00:49:46.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.235+0000 7f57aea27640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a80d1770 con 0x7f57a806cb60 2024-03-24T00:49:46.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.235+0000 7f57aea27640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57a806d3c0 con 0x7f57a806cf40 2024-03-24T00:49:46.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.235+0000 7f57acfc4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57a80b2800 0x7f57a80b0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:46.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.235+0000 7f57acfc4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57a80b2800 0x7f57a80b0ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43702/0 (socket says 172.21.15.67:43702) 2024-03-24T00:49:46.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.235+0000 7f57acfc4640 1 -- 172.21.15.67:0/3847259026 learned_addr learned my addr 172.21.15.67:0/3847259026 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:46.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.235+0000 7f57a77fe640 1 --2- 172.21.15.67:0/3847259026 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f57a806cf40 0x7f57a80b0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:46.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.235+0000 7f57acfc4640 1 -- 172.21.15.67:0/3847259026 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f57a806cf40 msgr2=0x7f57a80b0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:46.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.235+0000 7f57acfc4640 1 --2- 172.21.15.67:0/3847259026 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f57a806cf40 0x7f57a80b0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:46.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.235+0000 7f57a7fff640 1 --2- 172.21.15.67:0/3847259026 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57a806cb60 0x7f57a814e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:46.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.235+0000 7f57acfc4640 1 -- 172.21.15.67:0/3847259026 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57a806cb60 msgr2=0x7f57a814e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:46.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.235+0000 7f57acfc4640 1 --2- 172.21.15.67:0/3847259026 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57a806cb60 0x7f57a814e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:46.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.235+0000 7f57acfc4640 1 -- 172.21.15.67:0/3847259026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57a80b1530 con 0x7f57a80b2800 2024-03-24T00:49:46.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.235+0000 7f57a7fff640 1 --2- 172.21.15.67:0/3847259026 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57a806cb60 0x7f57a814e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:46.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.235+0000 7f57a77fe640 1 --2- 172.21.15.67:0/3847259026 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f57a806cf40 0x7f57a80b0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:46.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.236+0000 7f57acfc4640 1 --2- 172.21.15.67:0/3847259026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57a80b2800 0x7f57a80b0ff0 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7f57a000c7d0 tx=0x7f57a000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:46.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.236+0000 7f57a4ff9640 1 -- 172.21.15.67:0/3847259026 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57a0018070 con 0x7f57a80b2800 2024-03-24T00:49:46.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.236+0000 7f57a4ff9640 1 -- 172.21.15.67:0/3847259026 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57a000a640 con 0x7f57a80b2800 2024-03-24T00:49:46.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.236+0000 7f57a4ff9640 1 -- 172.21.15.67:0/3847259026 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57a000aa70 con 0x7f57a80b2800 2024-03-24T00:49:46.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.236+0000 7f57aea27640 1 -- 172.21.15.67:0/3847259026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57a80b1800 con 0x7f57a80b2800 2024-03-24T00:49:46.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.236+0000 7f57aea27640 1 -- 172.21.15.67:0/3847259026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57a80b1dc0 con 0x7f57a80b2800 2024-03-24T00:49:46.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.237+0000 7f57aea27640 1 -- 172.21.15.67:0/3847259026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f57a80b2480 con 0x7f57a80b2800 2024-03-24T00:49:46.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.237+0000 7f57a4ff9640 1 -- 172.21.15.67:0/3847259026 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f57a001b070 con 0x7f57a80b2800 2024-03-24T00:49:46.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.237+0000 7f57a4ff9640 1 --2- 172.21.15.67:0/3847259026 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f579406bbd0 0x7f579406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:46.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.238+0000 7f57a4ff9640 1 -- 172.21.15.67:0/3847259026 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f57a0089ed0 con 0x7f57a80b2800 2024-03-24T00:49:46.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.238+0000 7f57a4ff9640 1 --2- 172.21.15.67:0/3847259026 >> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] conn(0x7f57940721b0 0x7f57940745f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:46.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.238+0000 7f57a7fff640 1 --2- 172.21.15.67:0/3847259026 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f579406bbd0 0x7f579406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:46.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.238+0000 7f57a4ff9640 1 -- 172.21.15.67:0/3847259026 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f57a0017ec0 con 0x7f57940721b0 2024-03-24T00:49:46.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.238+0000 7f57a4ff9640 1 -- 172.21.15.67:0/3847259026 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=125) v2 ==== 24+0+0 (secure 0 0 0) 0x7f57a008c5d0 con 0x7f57a80b2800 2024-03-24T00:49:46.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.238+0000 7f57a77fe640 1 --2- 172.21.15.67:0/3847259026 >> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] conn(0x7f57940721b0 0x7f57940745f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:46.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.238+0000 7f57a77fe640 1 --2- 172.21.15.67:0/3847259026 >> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] conn(0x7f57940721b0 0x7f57940745f0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:46.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.238+0000 7f57a7fff640 1 --2- 172.21.15.67:0/3847259026 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f579406bbd0 0x7f579406e090 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f579c036000 tx=0x7f579c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:46.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.239+0000 7f57a4ff9640 1 -- 172.21.15.67:0/3847259026 <== osd.3 v2:172.21.15.67:6816/3303489481 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f57a0017ec0 con 0x7f57940721b0 2024-03-24T00:49:46.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.252+0000 7f57aea27640 1 -- 172.21.15.67:0/3847259026 --> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f57a8067970 con 0x7f57940721b0 2024-03-24T00:49:46.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.252+0000 7f57a4ff9640 1 -- 172.21.15.67:0/3847259026 <== osd.3 v2:172.21.15.67:6816/3303489481 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f57a8067970 con 0x7f57940721b0 2024-03-24T00:49:46.252 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:49:46.252 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-24T00:49:46.252 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:49:46.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.253+0000 7f57aea27640 1 -- 172.21.15.67:0/3847259026 >> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] conn(0x7f57940721b0 msgr2=0x7f57940745f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:46.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.253+0000 7f57aea27640 1 --2- 172.21.15.67:0/3847259026 >> [v2:172.21.15.67:6816/3303489481,v1:172.21.15.67:6818/3303489481] conn(0x7f57940721b0 0x7f57940745f0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:46.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.253+0000 7f57aea27640 1 -- 172.21.15.67:0/3847259026 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f579406bbd0 msgr2=0x7f579406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:46.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.253+0000 7f57aea27640 1 --2- 172.21.15.67:0/3847259026 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f579406bbd0 0x7f579406e090 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f579c036000 tx=0x7f579c008030 comp rx=0 tx=0).stop 2024-03-24T00:49:46.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.253+0000 7f57aea27640 1 -- 172.21.15.67:0/3847259026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57a80b2800 msgr2=0x7f57a80b0ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:46.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.253+0000 7f57aea27640 1 --2- 172.21.15.67:0/3847259026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57a80b2800 0x7f57a80b0ff0 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7f57a000c7d0 tx=0x7f57a000cc90 comp rx=0 tx=0).stop 2024-03-24T00:49:46.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.253+0000 7f57acfc4640 1 -- 172.21.15.67:0/3847259026 reap_dead start 2024-03-24T00:49:46.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.253+0000 7f57aea27640 1 -- 172.21.15.67:0/3847259026 shutdown_connections 2024-03-24T00:49:46.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.253+0000 7f57aea27640 1 -- 172.21.15.67:0/3847259026 >> 172.21.15.67:0/3847259026 conn(0x7f57a80b36c0 msgr2=0x7f57a815e170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:46.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.253+0000 7f57aea27640 1 -- 172.21.15.67:0/3847259026 shutdown_connections 2024-03-24T00:49:46.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.253+0000 7f57aea27640 1 -- 172.21.15.67:0/3847259026 wait complete. 2024-03-24T00:49:46.263 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-03-24T00:49:46.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.365+0000 7fac5dc8c640 1 Processor -- start 2024-03-24T00:49:46.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.365+0000 7fac5dc8c640 1 -- start start 2024-03-24T00:49:46.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.366+0000 7fac5dc8c640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fac580ceb10 0x7fac5806c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:46.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.366+0000 7fac5dc8c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fac5806d230 0x7fac5815d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:46.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.366+0000 7fac5dc8c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac580d0270 con 0x7fac580d26d0 2024-03-24T00:49:46.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.366+0000 7fac5dc8c640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac580d0e80 con 0x7fac5806d230 2024-03-24T00:49:46.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.366+0000 7fac5dc8c640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac580d0550 con 0x7fac580ceb10 2024-03-24T00:49:46.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.366+0000 7fac57fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fac5806d230 0x7fac5815d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:46.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.366+0000 7fac577fe640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fac580ceb10 0x7fac5806c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:46.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.366+0000 7fac57fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fac5806d230 0x7fac5815d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:59230/0 (socket says 172.21.15.67:59230) 2024-03-24T00:49:46.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.366+0000 7fac57fff640 1 -- 172.21.15.67:0/829020371 learned_addr learned my addr 172.21.15.67:0/829020371 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:46.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.366+0000 7fac56ffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fac580d26d0 0x7fac5806cb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:39106/0 (socket says 172.21.15.67:39106) 2024-03-24T00:49:46.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.367+0000 7fac57fff640 1 -- 172.21.15.67:0/829020371 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fac580ceb10 msgr2=0x7fac5806c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:46.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.367+0000 7fac57fff640 1 --2- 172.21.15.67:0/829020371 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fac580ceb10 0x7fac5806c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:46.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.367+0000 7fac57fff640 1 -- 172.21.15.67:0/829020371 >> v1:172.21.15.72:6789/0 conn(0x7fac580d26d0 legacy=0x7fac5806cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:46.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.367+0000 7fac57fff640 1 -- 172.21.15.67:0/829020371 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac580d0830 con 0x7fac5806d230 2024-03-24T00:49:46.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.367+0000 7fac567fc640 1 -- 172.21.15.67:0/829020371 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 304083303 0 0) 0x7fac580d0270 con 0x7fac580d26d0 2024-03-24T00:49:46.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.367+0000 7fac577fe640 1 --2- 172.21.15.67:0/829020371 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fac580ceb10 0x7fac5806c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:49:46.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.367+0000 7fac57fff640 1 --2- 172.21.15.67:0/829020371 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac5806d230 0x7fac5815d740 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fac50009e50 tx=0x7fac5002bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc669465739aaae0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:46.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.367+0000 7fac567fc640 1 -- 172.21.15.67:0/829020371 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac5002cb00 con 0x7fac5806d230 2024-03-24T00:49:46.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.367+0000 7fac567fc640 1 -- 172.21.15.67:0/829020371 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac5002cca0 con 0x7fac5806d230 2024-03-24T00:49:46.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.368+0000 7fac567fc640 1 -- 172.21.15.67:0/829020371 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac50034610 con 0x7fac5806d230 2024-03-24T00:49:46.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.368+0000 7fac5dc8c640 1 -- 172.21.15.67:0/829020371 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac5806d230 msgr2=0x7fac5815d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:46.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.368+0000 7fac5dc8c640 1 --2- 172.21.15.67:0/829020371 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac5806d230 0x7fac5815d740 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fac50009e50 tx=0x7fac5002bf60 comp rx=0 tx=0).stop 2024-03-24T00:49:46.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.368+0000 7fac5dc8c640 1 -- 172.21.15.67:0/829020371 shutdown_connections 2024-03-24T00:49:46.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.368+0000 7fac5dc8c640 1 --2- 172.21.15.67:0/829020371 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac5806d230 0x7fac5815d740 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:46.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.368+0000 7fac5dc8c640 1 --2- 172.21.15.67:0/829020371 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fac580ceb10 0x7fac5806c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:46.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.368+0000 7fac5dc8c640 1 -- 172.21.15.67:0/829020371 >> 172.21.15.67:0/829020371 conn(0x7fac580d1ad0 msgr2=0x7fac580d1ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:46.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.368+0000 7fac5dc8c640 1 -- 172.21.15.67:0/829020371 shutdown_connections 2024-03-24T00:49:46.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.368+0000 7fac5dc8c640 1 -- 172.21.15.67:0/829020371 wait complete. 2024-03-24T00:49:46.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.369+0000 7fac5dc8c640 1 Processor -- start 2024-03-24T00:49:46.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.369+0000 7fac5dc8c640 1 -- start start 2024-03-24T00:49:46.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.370+0000 7fac5dc8c640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fac5806d230 0x7fac581de660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:46.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.370+0000 7fac5dc8c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac580ceb10 0x7fac580aed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:46.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.370+0000 7fac5dc8c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac580d26d0 0x7fac580b50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:46.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.370+0000 7fac5dc8c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac580cf070 con 0x7fac580d26d0 2024-03-24T00:49:46.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.370+0000 7fac5dc8c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac580ceef0 con 0x7fac580ceb10 2024-03-24T00:49:46.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.370+0000 7fac5dc8c640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac580cfe00 con 0x7fac5806d230 2024-03-24T00:49:46.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.370+0000 7fac57fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac580d26d0 0x7fac580b50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:46.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.370+0000 7fac56ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac580ceb10 0x7fac580aed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:46.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.370+0000 7fac577fe640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fac5806d230 0x7fac581de660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:46.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.370+0000 7fac57fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac580d26d0 0x7fac580b50e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43706/0 (socket says 172.21.15.67:43706) 2024-03-24T00:49:46.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.370+0000 7fac57fff640 1 -- 172.21.15.67:0/3678278274 learned_addr learned my addr 172.21.15.67:0/3678278274 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:46.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.371+0000 7fac577fe640 1 -- 172.21.15.67:0/3678278274 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac580ceb10 msgr2=0x7fac580aed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:46.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.371+0000 7fac577fe640 1 --2- 172.21.15.67:0/3678278274 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac580ceb10 0x7fac580aed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:46.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.371+0000 7fac577fe640 1 -- 172.21.15.67:0/3678278274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac580d26d0 msgr2=0x7fac580b50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:46.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.371+0000 7fac577fe640 1 --2- 172.21.15.67:0/3678278274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac580d26d0 0x7fac580b50e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:46.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.371+0000 7fac577fe640 1 -- 172.21.15.67:0/3678278274 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac581deba0 con 0x7fac5806d230 2024-03-24T00:49:46.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.371+0000 7fac57fff640 1 --2- 172.21.15.67:0/3678278274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac580d26d0 0x7fac580b50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:49:46.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.371+0000 7fac56ffd640 1 --2- 172.21.15.67:0/3678278274 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac580ceb10 0x7fac580aed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:49:46.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.371+0000 7fac577fe640 1 --2- 172.21.15.67:0/3678278274 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fac5806d230 0x7fac581de660 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fac4800e870 tx=0x7fac4800ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:46.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.371+0000 7fac3ffff640 1 -- 172.21.15.67:0/3678278274 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac48017070 con 0x7fac5806d230 2024-03-24T00:49:46.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.371+0000 7fac3ffff640 1 -- 172.21.15.67:0/3678278274 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac4800c940 con 0x7fac5806d230 2024-03-24T00:49:46.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.371+0000 7fac5dc8c640 1 -- 172.21.15.67:0/3678278274 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac580af5c0 con 0x7fac5806d230 2024-03-24T00:49:46.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.371+0000 7fac3ffff640 1 -- 172.21.15.67:0/3678278274 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac4800cd10 con 0x7fac5806d230 2024-03-24T00:49:46.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.371+0000 7fac5dc8c640 1 -- 172.21.15.67:0/3678278274 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac580afb80 con 0x7fac5806d230 2024-03-24T00:49:46.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.372+0000 7fac5dc8c640 1 -- 172.21.15.67:0/3678278274 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac5804f400 con 0x7fac5806d230 2024-03-24T00:49:46.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.373+0000 7fac3ffff640 1 -- 172.21.15.67:0/3678278274 <== mon.2 v2:172.21.15.72:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fac4801a070 con 0x7fac5806d230 2024-03-24T00:49:46.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.373+0000 7fac3ffff640 1 --2- 172.21.15.67:0/3678278274 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fac4406bbd0 0x7fac4406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:46.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.373+0000 7fac3ffff640 1 -- 172.21.15.67:0/3678278274 <== mon.2 v2:172.21.15.72:3301/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5507+0+0 (secure 0 0 0) 0x7fac4808a6b0 con 0x7fac5806d230 2024-03-24T00:49:46.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.373+0000 7fac56ffd640 1 --2- 172.21.15.67:0/3678278274 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fac4406bbd0 0x7fac4406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:46.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.374+0000 7fac56ffd640 1 --2- 172.21.15.67:0/3678278274 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fac4406bbd0 0x7fac4406e090 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fac581e3510 tx=0x7fac4c007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:46.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.378+0000 7fac3ffff640 1 -- 172.21.15.67:0/3678278274 <== mon.2 v2:172.21.15.72:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fac48058ca0 con 0x7fac5806d230 2024-03-24T00:49:46.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.559+0000 7fac5dc8c640 1 -- 172.21.15.67:0/3678278274 --> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7fac580cda30 con 0x7fac4406bbd0 2024-03-24T00:49:46.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.560+0000 7fac3ffff640 1 -- 172.21.15.67:0/3678278274 <== mgr.4105 v2:172.21.15.72:6832/2768292274 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fac4c001cd0 con 0x7fac4406bbd0 2024-03-24T00:49:46.560 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 3 to deep-scrub 2024-03-24T00:49:46.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.562+0000 7fac5dc8c640 1 -- 172.21.15.67:0/3678278274 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fac4406bbd0 msgr2=0x7fac4406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:46.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.562+0000 7fac5dc8c640 1 --2- 172.21.15.67:0/3678278274 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fac4406bbd0 0x7fac4406e090 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fac581e3510 tx=0x7fac4c007d50 comp rx=0 tx=0).stop 2024-03-24T00:49:46.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.562+0000 7fac5dc8c640 1 -- 172.21.15.67:0/3678278274 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fac5806d230 msgr2=0x7fac581de660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:46.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.562+0000 7fac5dc8c640 1 --2- 172.21.15.67:0/3678278274 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fac5806d230 0x7fac581de660 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fac4800e870 tx=0x7fac4800ed30 comp rx=0 tx=0).stop 2024-03-24T00:49:46.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.562+0000 7fac5dc8c640 1 -- 172.21.15.67:0/3678278274 shutdown_connections 2024-03-24T00:49:46.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.562+0000 7fac5dc8c640 1 --2- 172.21.15.67:0/3678278274 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fac4406bbd0 0x7fac4406e090 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:46.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.562+0000 7fac5dc8c640 1 --2- 172.21.15.67:0/3678278274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac580d26d0 0x7fac580b50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:46.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.562+0000 7fac5dc8c640 1 --2- 172.21.15.67:0/3678278274 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac580ceb10 0x7fac580aed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:46.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.562+0000 7fac5dc8c640 1 --2- 172.21.15.67:0/3678278274 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fac5806d230 0x7fac581de660 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:46.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.562+0000 7fac5dc8c640 1 -- 172.21.15.67:0/3678278274 >> 172.21.15.67:0/3678278274 conn(0x7fac580d1ad0 msgr2=0x7fac580cd320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:46.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.562+0000 7fac5dc8c640 1 -- 172.21.15.67:0/3678278274 shutdown_connections 2024-03-24T00:49:46.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.562+0000 7fac5dc8c640 1 -- 172.21.15.67:0/3678278274 wait complete. 2024-03-24T00:49:46.572 INFO:tasks.ceph:Scrubbing osd.4 2024-03-24T00:49:46.572 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-03-24T00:49:46.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.678+0000 7f78c2b22640 1 Processor -- start 2024-03-24T00:49:46.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.678+0000 7f78c2b22640 1 -- start start 2024-03-24T00:49:46.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.679+0000 7f78c2b22640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f78bc0d00b0 0x7f78bc0d0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:46.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.679+0000 7f78c2b22640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f78bc0cd230 0x7f78bc161b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:46.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.679+0000 7f78c2b22640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f78bc0cdb50 0x7f78bc06b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:46.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.679+0000 7f78c2b22640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78bc070170 con 0x7f78bc0d00b0 2024-03-24T00:49:46.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.679+0000 7f78c2b22640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78bc0703c0 con 0x7f78bc0cd230 2024-03-24T00:49:46.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.679+0000 7f78c2b22640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78bc0706a0 con 0x7f78bc0cdb50 2024-03-24T00:49:46.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.679+0000 7f78c08be640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f78bc0d00b0 0x7f78bc0d0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:46.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.679+0000 7f78c08be640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f78bc0d00b0 0x7f78bc0d0490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43718/0 (socket says 172.21.15.67:43718) 2024-03-24T00:49:46.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.679+0000 7f78c08be640 1 -- 172.21.15.67:0/3709185894 learned_addr learned my addr 172.21.15.67:0/3709185894 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:46.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.679+0000 7f78bbfff640 1 --2- 172.21.15.67:0/3709185894 >> v2:172.21.15.72:3300/0 conn(0x7f78bc0cd230 0x7f78bc161b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:46.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.679+0000 7f78c10bf640 1 --2- 172.21.15.67:0/3709185894 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f78bc0cdb50 0x7f78bc06b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:46.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.679+0000 7f78c08be640 1 -- 172.21.15.67:0/3709185894 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f78bc0cdb50 msgr2=0x7f78bc06b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:46.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.679+0000 7f78c08be640 1 --2- 172.21.15.67:0/3709185894 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f78bc0cdb50 0x7f78bc06b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:46.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.679+0000 7f78c08be640 1 -- 172.21.15.67:0/3709185894 >> v2:172.21.15.72:3300/0 conn(0x7f78bc0cd230 msgr2=0x7f78bc161b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:46.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.679+0000 7f78c08be640 1 --2- 172.21.15.67:0/3709185894 >> v2:172.21.15.72:3300/0 conn(0x7f78bc0cd230 0x7f78bc161b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:46.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.679+0000 7f78c08be640 1 -- 172.21.15.67:0/3709185894 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78bc070980 con 0x7f78bc0d00b0 2024-03-24T00:49:46.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.679+0000 7f78bbfff640 1 --2- 172.21.15.67:0/3709185894 >> v2:172.21.15.72:3300/0 conn(0x7f78bc0cd230 0x7f78bc161b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:46.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.679+0000 7f78c10bf640 1 --2- 172.21.15.67:0/3709185894 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f78bc0cdb50 0x7f78bc06b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:46.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.680+0000 7f78c08be640 1 --2- 172.21.15.67:0/3709185894 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78bc0d00b0 0x7f78bc0d0490 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f78bc0caf90 tx=0x7f78b002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbf6a88e0d90310a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:46.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.680+0000 7f78bb7fe640 1 -- 172.21.15.67:0/3709185894 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b002e070 con 0x7f78bc0d00b0 2024-03-24T00:49:46.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.680+0000 7f78bb7fe640 1 -- 172.21.15.67:0/3709185894 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78b0037070 con 0x7f78bc0d00b0 2024-03-24T00:49:46.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.680+0000 7f78bb7fe640 1 -- 172.21.15.67:0/3709185894 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b0031a00 con 0x7f78bc0d00b0 2024-03-24T00:49:46.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.680+0000 7f78c2b22640 1 -- 172.21.15.67:0/3709185894 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78bc0d00b0 msgr2=0x7f78bc0d0490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:46.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.680+0000 7f78c2b22640 1 --2- 172.21.15.67:0/3709185894 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78bc0d00b0 0x7f78bc0d0490 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f78bc0caf90 tx=0x7f78b002b880 comp rx=0 tx=0).stop 2024-03-24T00:49:46.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.680+0000 7f78c2b22640 1 -- 172.21.15.67:0/3709185894 shutdown_connections 2024-03-24T00:49:46.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.680+0000 7f78c2b22640 1 --2- 172.21.15.67:0/3709185894 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f78bc0cdb50 0x7f78bc06b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:46.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.680+0000 7f78c2b22640 1 --2- 172.21.15.67:0/3709185894 >> v2:172.21.15.72:3300/0 conn(0x7f78bc0cd230 0x7f78bc161b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:46.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.680+0000 7f78c2b22640 1 --2- 172.21.15.67:0/3709185894 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78bc0d00b0 0x7f78bc0d0490 unknown :-1 s=CLOSED pgs=1948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:46.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.680+0000 7f78c2b22640 1 -- 172.21.15.67:0/3709185894 >> 172.21.15.67:0/3709185894 conn(0x7f78bc099a30 msgr2=0x7f78bc099e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:46.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.681+0000 7f78c2b22640 1 -- 172.21.15.67:0/3709185894 shutdown_connections 2024-03-24T00:49:46.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.681+0000 7f78c2b22640 1 -- 172.21.15.67:0/3709185894 wait complete. 2024-03-24T00:49:46.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.682+0000 7f78c2b22640 1 Processor -- start 2024-03-24T00:49:46.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.682+0000 7f78c2b22640 1 -- start start 2024-03-24T00:49:46.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.683+0000 7f78c2b22640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78bc0cd230 0x7f78bc0b6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:46.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.683+0000 7f78c2b22640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78bc0cdb50 0x7f78bc0b4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:46.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.683+0000 7f78c2b22640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f78bc0d00b0 0x7f78bc0c9f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:46.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.683+0000 7f78c2b22640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78bc0cdfb0 con 0x7f78bc0cdb50 2024-03-24T00:49:46.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.683+0000 7f78c2b22640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78bc162050 con 0x7f78bc0cd230 2024-03-24T00:49:46.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.683+0000 7f78c2b22640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78bc0ce130 con 0x7f78bc0d00b0 2024-03-24T00:49:46.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.683+0000 7f78bbfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78bc0cdb50 0x7f78bc0b4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:46.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.683+0000 7f78bbfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78bc0cdb50 0x7f78bc0b4050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43720/0 (socket says 172.21.15.67:43720) 2024-03-24T00:49:46.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.683+0000 7f78bbfff640 1 -- 172.21.15.67:0/3113590376 learned_addr learned my addr 172.21.15.67:0/3113590376 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:46.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.683+0000 7f78c08be640 1 --2- 172.21.15.67:0/3113590376 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78bc0cd230 0x7f78bc0b6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:46.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.683+0000 7f78c10bf640 1 --2- 172.21.15.67:0/3113590376 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f78bc0d00b0 0x7f78bc0c9f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:46.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.683+0000 7f78bbfff640 1 -- 172.21.15.67:0/3113590376 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f78bc0d00b0 msgr2=0x7f78bc0c9f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:46.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.683+0000 7f78bbfff640 1 --2- 172.21.15.67:0/3113590376 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f78bc0d00b0 0x7f78bc0c9f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:46.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.683+0000 7f78bbfff640 1 -- 172.21.15.67:0/3113590376 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78bc0cd230 msgr2=0x7f78bc0b6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:46.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.683+0000 7f78bbfff640 1 --2- 172.21.15.67:0/3113590376 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78bc0cd230 0x7f78bc0b6590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:46.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.683+0000 7f78bbfff640 1 -- 172.21.15.67:0/3113590376 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78bc1f3e40 con 0x7f78bc0cdb50 2024-03-24T00:49:46.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.683+0000 7f78bbfff640 1 --2- 172.21.15.67:0/3113590376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78bc0cdb50 0x7f78bc0b4050 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f78a800c7f0 tx=0x7f78a800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:46.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.684+0000 7f78b97fa640 1 -- 172.21.15.67:0/3113590376 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a8015070 con 0x7f78bc0cdb50 2024-03-24T00:49:46.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.684+0000 7f78b97fa640 1 -- 172.21.15.67:0/3113590376 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78a8007820 con 0x7f78bc0cdb50 2024-03-24T00:49:46.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.684+0000 7f78b97fa640 1 -- 172.21.15.67:0/3113590376 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a8007bd0 con 0x7f78bc0cdb50 2024-03-24T00:49:46.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.684+0000 7f78c2b22640 1 -- 172.21.15.67:0/3113590376 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78bc0c5760 con 0x7f78bc0cdb50 2024-03-24T00:49:46.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.684+0000 7f78c2b22640 1 -- 172.21.15.67:0/3113590376 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78bc0c5d20 con 0x7f78bc0cdb50 2024-03-24T00:49:46.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.684+0000 7f78c2b22640 1 -- 172.21.15.67:0/3113590376 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f78bc04f400 con 0x7f78bc0cdb50 2024-03-24T00:49:46.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.685+0000 7f78b97fa640 1 -- 172.21.15.67:0/3113590376 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f78a8018070 con 0x7f78bc0cdb50 2024-03-24T00:49:46.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.685+0000 7f78b97fa640 1 --2- 172.21.15.67:0/3113590376 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f78a006bbd0 0x7f78a006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:46.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.686+0000 7f78b97fa640 1 -- 172.21.15.67:0/3113590376 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f78a808aa40 con 0x7f78bc0cdb50 2024-03-24T00:49:46.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.686+0000 7f78c08be640 1 --2- 172.21.15.67:0/3113590376 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f78a006bbd0 0x7f78a006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:46.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.686+0000 7f78b97fa640 1 --2- 172.21.15.67:0/3113590376 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x7f78a00721b0 0x7f78a00745f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:46.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.686+0000 7f78b97fa640 1 -- 172.21.15.67:0/3113590376 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f78a801ce50 con 0x7f78a00721b0 2024-03-24T00:49:46.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.686+0000 7f78b97fa640 1 -- 172.21.15.67:0/3113590376 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=125) v2 ==== 24+0+0 (secure 0 0 0) 0x7f78a808f2c0 con 0x7f78bc0cdb50 2024-03-24T00:49:46.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.686+0000 7f78c10bf640 1 --2- 172.21.15.67:0/3113590376 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x7f78a00721b0 0x7f78a00745f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:46.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.686+0000 7f78c08be640 1 --2- 172.21.15.67:0/3113590376 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f78a006bbd0 0x7f78a006e090 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f78b0036000 tx=0x7f78b003d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:46.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.686+0000 7f78c10bf640 1 --2- 172.21.15.67:0/3113590376 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x7f78a00721b0 0x7f78a00745f0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:46.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.687+0000 7f78b97fa640 1 -- 172.21.15.67:0/3113590376 <== osd.4 v2:172.21.15.72:6808/477220793 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f78a801ce50 con 0x7f78a00721b0 2024-03-24T00:49:46.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.700+0000 7f78c2b22640 1 -- 172.21.15.67:0/3113590376 --> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f78bc0cd700 con 0x7f78a00721b0 2024-03-24T00:49:46.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.701+0000 7f78b97fa640 1 -- 172.21.15.67:0/3113590376 <== osd.4 v2:172.21.15.72:6808/477220793 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f78bc0cd700 con 0x7f78a00721b0 2024-03-24T00:49:46.700 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:49:46.701 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-24T00:49:46.701 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:49:46.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.701+0000 7f78c2b22640 1 -- 172.21.15.67:0/3113590376 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x7f78a00721b0 msgr2=0x7f78a00745f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:46.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.701+0000 7f78c2b22640 1 --2- 172.21.15.67:0/3113590376 >> [v2:172.21.15.72:6808/477220793,v1:172.21.15.72:6809/477220793] conn(0x7f78a00721b0 0x7f78a00745f0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:46.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.701+0000 7f78c2b22640 1 -- 172.21.15.67:0/3113590376 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f78a006bbd0 msgr2=0x7f78a006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:46.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.701+0000 7f78c2b22640 1 --2- 172.21.15.67:0/3113590376 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f78a006bbd0 0x7f78a006e090 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f78b0036000 tx=0x7f78b003d040 comp rx=0 tx=0).stop 2024-03-24T00:49:46.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.701+0000 7f78c2b22640 1 -- 172.21.15.67:0/3113590376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78bc0cdb50 msgr2=0x7f78bc0b4050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:46.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.701+0000 7f78c2b22640 1 --2- 172.21.15.67:0/3113590376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78bc0cdb50 0x7f78bc0b4050 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f78a800c7f0 tx=0x7f78a800ccb0 comp rx=0 tx=0).stop 2024-03-24T00:49:46.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.701+0000 7f78c10bf640 1 -- 172.21.15.67:0/3113590376 reap_dead start 2024-03-24T00:49:46.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.701+0000 7f78c2b22640 1 -- 172.21.15.67:0/3113590376 shutdown_connections 2024-03-24T00:49:46.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.701+0000 7f78c2b22640 1 -- 172.21.15.67:0/3113590376 >> 172.21.15.67:0/3113590376 conn(0x7f78bc099a30 msgr2=0x7f78bc0b2470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:46.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.701+0000 7f78c2b22640 1 -- 172.21.15.67:0/3113590376 shutdown_connections 2024-03-24T00:49:46.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.701+0000 7f78c2b22640 1 -- 172.21.15.67:0/3113590376 wait complete. 2024-03-24T00:49:46.711 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-03-24T00:49:46.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.813+0000 7fa7887b9640 1 Processor -- start 2024-03-24T00:49:46.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.813+0000 7fa7887b9640 1 -- start start 2024-03-24T00:49:46.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.814+0000 7fa7887b9640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa78015e0f0 0x7fa7800b95b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:46.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.814+0000 7fa7887b9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa7800b6dc0 0x7fa7800b71c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:46.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.814+0000 7fa7887b9640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7800b7700 con 0x7fa7800b9af0 2024-03-24T00:49:46.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.814+0000 7fa7887b9640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7800d2230 con 0x7fa7800b6dc0 2024-03-24T00:49:46.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.814+0000 7fa7887b9640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7800d2510 con 0x7fa78015e0f0 2024-03-24T00:49:46.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.814+0000 7fa786d56640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa7800b6dc0 0x7fa7800b71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:46.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.814+0000 7fa786d56640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa7800b6dc0 0x7fa7800b71c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43734/0 (socket says 172.21.15.67:43734) 2024-03-24T00:49:46.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.814+0000 7fa786d56640 1 -- 172.21.15.67:0/1259984686 learned_addr learned my addr 172.21.15.67:0/1259984686 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:46.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.814+0000 7fa785d54640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa7800b9af0 0x7fa7800d32d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56276/0 (socket says 172.21.15.67:56276) 2024-03-24T00:49:46.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.815+0000 7fa786555640 1 --2- 172.21.15.67:0/1259984686 >> v2:172.21.15.72:3300/0 conn(0x7fa78015e0f0 0x7fa7800b95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:46.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.815+0000 7fa786d56640 1 -- 172.21.15.67:0/1259984686 >> v2:172.21.15.72:3300/0 conn(0x7fa78015e0f0 msgr2=0x7fa7800b95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:46.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.815+0000 7fa786d56640 1 --2- 172.21.15.67:0/1259984686 >> v2:172.21.15.72:3300/0 conn(0x7fa78015e0f0 0x7fa7800b95b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:46.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.815+0000 7fa786d56640 1 -- 172.21.15.67:0/1259984686 >> v1:172.21.15.67:6789/0 conn(0x7fa7800b9af0 legacy=0x7fa7800d32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:46.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.815+0000 7fa786d56640 1 -- 172.21.15.67:0/1259984686 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7800d27f0 con 0x7fa7800b6dc0 2024-03-24T00:49:46.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.815+0000 7fa785553640 1 -- 172.21.15.67:0/1259984686 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1887621402 0 0) 0x7fa7800b7700 con 0x7fa7800b9af0 2024-03-24T00:49:46.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.815+0000 7fa786d56640 1 --2- 172.21.15.67:0/1259984686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7800b6dc0 0x7fa7800b71c0 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7fa78008cc00 tx=0x7fa77002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d5cb3084c6b7681 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:46.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.816+0000 7fa785553640 1 -- 172.21.15.67:0/1259984686 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa77002cb20 con 0x7fa7800b6dc0 2024-03-24T00:49:46.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.816+0000 7fa785553640 1 -- 172.21.15.67:0/1259984686 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa77002ccc0 con 0x7fa7800b6dc0 2024-03-24T00:49:46.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.816+0000 7fa785553640 1 -- 172.21.15.67:0/1259984686 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa770034610 con 0x7fa7800b6dc0 2024-03-24T00:49:46.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.816+0000 7fa7887b9640 1 -- 172.21.15.67:0/1259984686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7800b6dc0 msgr2=0x7fa7800b71c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:46.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.816+0000 7fa7887b9640 1 --2- 172.21.15.67:0/1259984686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7800b6dc0 0x7fa7800b71c0 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7fa78008cc00 tx=0x7fa77002bf80 comp rx=0 tx=0).stop 2024-03-24T00:49:46.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.816+0000 7fa7887b9640 1 -- 172.21.15.67:0/1259984686 shutdown_connections 2024-03-24T00:49:46.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.816+0000 7fa7887b9640 1 --2- 172.21.15.67:0/1259984686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7800b6dc0 0x7fa7800b71c0 unknown :-1 s=CLOSED pgs=1951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:46.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.816+0000 7fa7887b9640 1 --2- 172.21.15.67:0/1259984686 >> v2:172.21.15.72:3300/0 conn(0x7fa78015e0f0 0x7fa7800b95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:46.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.816+0000 7fa7887b9640 1 -- 172.21.15.67:0/1259984686 >> 172.21.15.67:0/1259984686 conn(0x7fa780072b60 msgr2=0x7fa780072f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:46.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.816+0000 7fa7887b9640 1 -- 172.21.15.67:0/1259984686 shutdown_connections 2024-03-24T00:49:46.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.816+0000 7fa7887b9640 1 -- 172.21.15.67:0/1259984686 wait complete. 2024-03-24T00:49:46.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.817+0000 7fa7887b9640 1 Processor -- start 2024-03-24T00:49:46.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.817+0000 7fa7887b9640 1 -- start start 2024-03-24T00:49:46.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.818+0000 7fa7887b9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7800b6dc0 0x7fa7800ae420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:46.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.818+0000 7fa7887b9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa7800b9af0 0x7fa7800ab130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:46.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.818+0000 7fa7887b9640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa78015e0f0 0x7fa7800ac4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:46.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.818+0000 7fa7887b9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7800b31d0 con 0x7fa7800b6dc0 2024-03-24T00:49:46.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.818+0000 7fa7887b9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7800d2db0 con 0x7fa7800b9af0 2024-03-24T00:49:46.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.818+0000 7fa7887b9640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7800b3350 con 0x7fa78015e0f0 2024-03-24T00:49:46.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.818+0000 7fa786555640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7800b6dc0 0x7fa7800ae420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:46.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.818+0000 7fa786555640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7800b6dc0 0x7fa7800ae420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43744/0 (socket says 172.21.15.67:43744) 2024-03-24T00:49:46.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.818+0000 7fa786555640 1 -- 172.21.15.67:0/722844105 learned_addr learned my addr 172.21.15.67:0/722844105 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:46.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.818+0000 7fa786d56640 1 --2- 172.21.15.67:0/722844105 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa78015e0f0 0x7fa7800ac4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:46.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.818+0000 7fa785d54640 1 --2- 172.21.15.67:0/722844105 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa7800b9af0 0x7fa7800ab130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:46.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.818+0000 7fa786555640 1 -- 172.21.15.67:0/722844105 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa78015e0f0 msgr2=0x7fa7800ac4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:46.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.818+0000 7fa786555640 1 --2- 172.21.15.67:0/722844105 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa78015e0f0 0x7fa7800ac4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:46.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.818+0000 7fa786555640 1 -- 172.21.15.67:0/722844105 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa7800b9af0 msgr2=0x7fa7800ab130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:46.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.818+0000 7fa786555640 1 --2- 172.21.15.67:0/722844105 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa7800b9af0 0x7fa7800ab130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:46.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.818+0000 7fa786555640 1 -- 172.21.15.67:0/722844105 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7800ae960 con 0x7fa7800b6dc0 2024-03-24T00:49:46.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.818+0000 7fa785d54640 1 --2- 172.21.15.67:0/722844105 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa7800b9af0 0x7fa7800ab130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:49:46.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.819+0000 7fa786555640 1 --2- 172.21.15.67:0/722844105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7800b6dc0 0x7fa7800ae420 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7fa77800c820 tx=0x7fa77800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:46.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.819+0000 7fa77effd640 1 -- 172.21.15.67:0/722844105 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa778015070 con 0x7fa7800b6dc0 2024-03-24T00:49:46.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.819+0000 7fa77effd640 1 -- 172.21.15.67:0/722844105 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa778004020 con 0x7fa7800b6dc0 2024-03-24T00:49:46.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.819+0000 7fa7887b9640 1 -- 172.21.15.67:0/722844105 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7800ab9f0 con 0x7fa7800b6dc0 2024-03-24T00:49:46.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.819+0000 7fa77effd640 1 -- 172.21.15.67:0/722844105 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7780043f0 con 0x7fa7800b6dc0 2024-03-24T00:49:46.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.819+0000 7fa7887b9640 1 -- 172.21.15.67:0/722844105 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7800cb110 con 0x7fa7800b6dc0 2024-03-24T00:49:46.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.820+0000 7fa7887b9640 1 -- 172.21.15.67:0/722844105 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa78004f400 con 0x7fa7800b6dc0 2024-03-24T00:49:46.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.820+0000 7fa77effd640 1 -- 172.21.15.67:0/722844105 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa778007830 con 0x7fa7800b6dc0 2024-03-24T00:49:46.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.821+0000 7fa77effd640 1 --2- 172.21.15.67:0/722844105 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa76c06bc00 0x7fa76c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:46.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.821+0000 7fa77effd640 1 -- 172.21.15.67:0/722844105 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5507+0+0 (secure 0 0 0) 0x7fa77808a690 con 0x7fa7800b6dc0 2024-03-24T00:49:46.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.821+0000 7fa785d54640 1 --2- 172.21.15.67:0/722844105 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa76c06bc00 0x7fa76c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:46.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.821+0000 7fa785d54640 1 --2- 172.21.15.67:0/722844105 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa76c06bc00 0x7fa76c06e0c0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fa7740079d0 tx=0x7fa774009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:46.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:46.825+0000 7fa77effd640 1 -- 172.21.15.67:0/722844105 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa778058bf0 con 0x7fa7800b6dc0 2024-03-24T00:49:47.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.006+0000 7fa7887b9640 1 -- 172.21.15.67:0/722844105 --> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7fa7800b3a90 con 0x7fa76c06bc00 2024-03-24T00:49:47.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.006+0000 7fa77effd640 1 -- 172.21.15.67:0/722844105 <== mgr.4105 v2:172.21.15.72:6832/2768292274 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fa7800b3a90 con 0x7fa76c06bc00 2024-03-24T00:49:47.006 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 4 to deep-scrub 2024-03-24T00:49:47.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.008+0000 7fa7887b9640 1 -- 172.21.15.67:0/722844105 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa76c06bc00 msgr2=0x7fa76c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:47.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.008+0000 7fa7887b9640 1 --2- 172.21.15.67:0/722844105 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa76c06bc00 0x7fa76c06e0c0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fa7740079d0 tx=0x7fa774009040 comp rx=0 tx=0).stop 2024-03-24T00:49:47.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.008+0000 7fa7887b9640 1 -- 172.21.15.67:0/722844105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7800b6dc0 msgr2=0x7fa7800ae420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:47.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.008+0000 7fa7887b9640 1 --2- 172.21.15.67:0/722844105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7800b6dc0 0x7fa7800ae420 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7fa77800c820 tx=0x7fa77800cce0 comp rx=0 tx=0).stop 2024-03-24T00:49:47.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.008+0000 7fa7887b9640 1 -- 172.21.15.67:0/722844105 shutdown_connections 2024-03-24T00:49:47.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.008+0000 7fa7887b9640 1 --2- 172.21.15.67:0/722844105 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fa76c06bc00 0x7fa76c06e0c0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:47.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.008+0000 7fa7887b9640 1 --2- 172.21.15.67:0/722844105 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa78015e0f0 0x7fa7800ac4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:47.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.008+0000 7fa7887b9640 1 --2- 172.21.15.67:0/722844105 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa7800b9af0 0x7fa7800ab130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:47.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.008+0000 7fa7887b9640 1 --2- 172.21.15.67:0/722844105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7800b6dc0 0x7fa7800ae420 unknown :-1 s=CLOSED pgs=1952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:47.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.008+0000 7fa7887b9640 1 -- 172.21.15.67:0/722844105 >> 172.21.15.67:0/722844105 conn(0x7fa780072b60 msgr2=0x7fa7800d1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:47.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.008+0000 7fa7887b9640 1 -- 172.21.15.67:0/722844105 shutdown_connections 2024-03-24T00:49:47.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.008+0000 7fa7887b9640 1 -- 172.21.15.67:0/722844105 wait complete. 2024-03-24T00:49:47.021 INFO:tasks.ceph:Scrubbing osd.5 2024-03-24T00:49:47.021 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-03-24T00:49:47.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.127+0000 7f20bb7b9640 1 Processor -- start 2024-03-24T00:49:47.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.127+0000 7f20bb7b9640 1 -- start start 2024-03-24T00:49:47.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.128+0000 7f20bb7b9640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f20b406cb60 0x7f20b409a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:47.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.128+0000 7f20bb7b9640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f20b406cf40 0x7f20b40cfe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:47.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.128+0000 7f20bb7b9640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20b40d06d0 con 0x7f20b40b2800 2024-03-24T00:49:47.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.128+0000 7f20bb7b9640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20b40d09b0 con 0x7f20b406cb60 2024-03-24T00:49:47.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.128+0000 7f20bb7b9640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20b40d0c90 con 0x7f20b406cf40 2024-03-24T00:49:47.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.128+0000 7f20b9d56640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f20b40b2800 0x7f20b40b2c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56282/0 (socket says 172.21.15.67:56282) 2024-03-24T00:49:47.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.128+0000 7f20b9d56640 1 -- 172.21.15.67:0/317136142 learned_addr learned my addr 172.21.15.67:0/317136142 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:49:47.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.128+0000 7f20b8d54640 1 --2- 172.21.15.67:0/317136142 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f20b406cf40 0x7f20b40cfe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:47.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.128+0000 7f20abfff640 1 -- 172.21.15.67:0/317136142 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1700639872 0 0) 0x7f20b40d06d0 con 0x7f20b40b2800 2024-03-24T00:49:47.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.128+0000 7f20b9555640 1 --2- 172.21.15.67:0/317136142 >> v2:172.21.15.72:3300/0 conn(0x7f20b406cb60 0x7f20b409a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:47.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.128+0000 7f20abfff640 1 -- 172.21.15.67:0/317136142 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20a0003520 con 0x7f20b40b2800 2024-03-24T00:49:47.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.129+0000 7f20abfff640 1 -- 172.21.15.67:0/317136142 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2052051942 0 0) 0x7f20a0003520 con 0x7f20b40b2800 2024-03-24T00:49:47.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.129+0000 7f20abfff640 1 -- 172.21.15.67:0/317136142 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f20b406cf40 msgr2=0x7f20b40cfe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:47.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.129+0000 7f20abfff640 1 --2- 172.21.15.67:0/317136142 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f20b406cf40 0x7f20b40cfe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:47.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.129+0000 7f20abfff640 1 -- 172.21.15.67:0/317136142 >> v2:172.21.15.72:3300/0 conn(0x7f20b406cb60 msgr2=0x7f20b409a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:47.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.129+0000 7f20abfff640 1 --2- 172.21.15.67:0/317136142 >> v2:172.21.15.72:3300/0 conn(0x7f20b406cb60 0x7f20b409a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:47.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.129+0000 7f20abfff640 1 -- 172.21.15.67:0/317136142 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20b40d0f70 con 0x7f20b40b2800 2024-03-24T00:49:47.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.129+0000 7f20abfff640 1 -- 172.21.15.67:0/317136142 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f20b00019f0 con 0x7f20b40b2800 2024-03-24T00:49:47.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.129+0000 7f20abfff640 1 -- 172.21.15.67:0/317136142 >> v1:172.21.15.67:6789/0 conn(0x7f20b40b2800 legacy=0x7f20b40b2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:47.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.129+0000 7f20abfff640 1 --2- 172.21.15.67:0/317136142 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20a00039b0 0x7f20a0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:47.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.129+0000 7f20abfff640 1 --2- 172.21.15.67:0/317136142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20a0004800 0x7f20a0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:47.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.129+0000 7f20abfff640 1 --2- 172.21.15.67:0/317136142 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f20b406cf40 0x7f20a00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:47.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.129+0000 7f20abfff640 1 -- 172.21.15.67:0/317136142 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20b40d09b0 con 0x7f20a0004800 2024-03-24T00:49:47.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.129+0000 7f20abfff640 1 -- 172.21.15.67:0/317136142 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20b40d0c90 con 0x7f20a00039b0 2024-03-24T00:49:47.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.129+0000 7f20abfff640 1 -- 172.21.15.67:0/317136142 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20a000b9c0 con 0x7f20b406cf40 2024-03-24T00:49:47.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.129+0000 7f20b8d54640 1 --2- 172.21.15.67:0/317136142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20a0004800 0x7f20a0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:47.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.129+0000 7f20b9555640 1 --2- 172.21.15.67:0/317136142 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20a00039b0 0x7f20a0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:47.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.129+0000 7f20b9d56640 1 --2- 172.21.15.67:0/317136142 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f20b406cf40 0x7f20a00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:47.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.129+0000 7f20b8d54640 1 -- 172.21.15.67:0/317136142 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f20b406cf40 msgr2=0x7f20a00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:47.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.129+0000 7f20b8d54640 1 --2- 172.21.15.67:0/317136142 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f20b406cf40 0x7f20a00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:47.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.129+0000 7f20b8d54640 1 -- 172.21.15.67:0/317136142 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20a00039b0 msgr2=0x7f20a0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:47.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.129+0000 7f20b8d54640 1 --2- 172.21.15.67:0/317136142 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20a00039b0 0x7f20a0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:47.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.129+0000 7f20b8d54640 1 -- 172.21.15.67:0/317136142 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a0003520 con 0x7f20a0004800 2024-03-24T00:49:47.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.130+0000 7f20b8d54640 1 --2- 172.21.15.67:0/317136142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20a0004800 0x7f20a0004c80 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7f20a4009af0 tx=0x7f20a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd68a70dcdc2990 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:47.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.130+0000 7f20abfff640 1 -- 172.21.15.67:0/317136142 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a4039070 con 0x7f20a0004800 2024-03-24T00:49:47.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.130+0000 7f20abfff640 1 -- 172.21.15.67:0/317136142 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20a402c6b0 con 0x7f20a0004800 2024-03-24T00:49:47.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.130+0000 7f20abfff640 1 -- 172.21.15.67:0/317136142 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a4009690 con 0x7f20a0004800 2024-03-24T00:49:47.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.130+0000 7f20bb7b9640 1 -- 172.21.15.67:0/317136142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20a0004800 msgr2=0x7f20a0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:47.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.131+0000 7f20bb7b9640 1 --2- 172.21.15.67:0/317136142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20a0004800 0x7f20a0004c80 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7f20a4009af0 tx=0x7f20a0005830 comp rx=0 tx=0).stop 2024-03-24T00:49:47.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.131+0000 7f20b9d56640 1 -- 172.21.15.67:0/317136142 reap_dead start 2024-03-24T00:49:47.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.131+0000 7f20bb7b9640 1 -- 172.21.15.67:0/317136142 shutdown_connections 2024-03-24T00:49:47.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.131+0000 7f20bb7b9640 1 -- 172.21.15.67:0/317136142 >> 172.21.15.67:0/317136142 conn(0x7f20b40b36c0 msgr2=0x7f20b40b3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:47.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.131+0000 7f20bb7b9640 1 -- 172.21.15.67:0/317136142 shutdown_connections 2024-03-24T00:49:47.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.131+0000 7f20bb7b9640 1 -- 172.21.15.67:0/317136142 wait complete. 2024-03-24T00:49:47.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.132+0000 7f20bb7b9640 1 Processor -- start 2024-03-24T00:49:47.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.132+0000 7f20bb7b9640 1 -- start start 2024-03-24T00:49:47.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.132+0000 7f20bb7b9640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f20a0004800 0x7f20b4148eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:47.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.133+0000 7f20bb7b9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20b40b58b0 0x7f20b40b5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:47.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.133+0000 7f20bb7b9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20b4160a00 0x7f20b4160eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:47.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.133+0000 7f20bb7b9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20b40cd230 con 0x7f20b40b58b0 2024-03-24T00:49:47.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.133+0000 7f20bb7b9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20b40d1770 con 0x7f20b4160a00 2024-03-24T00:49:47.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.133+0000 7f20bb7b9640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20b406d3c0 con 0x7f20a0004800 2024-03-24T00:49:47.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.133+0000 7f20b8d54640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20b40b58b0 0x7f20b40b5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:47.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.133+0000 7f20b8d54640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20b40b58b0 0x7f20b40b5d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43756/0 (socket says 172.21.15.67:43756) 2024-03-24T00:49:47.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.133+0000 7f20b8d54640 1 -- 172.21.15.67:0/2051173121 learned_addr learned my addr 172.21.15.67:0/2051173121 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:47.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.133+0000 7f20b9d56640 1 --2- 172.21.15.67:0/2051173121 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20b4160a00 0x7f20b4160eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:47.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.133+0000 7f20b8d54640 1 -- 172.21.15.67:0/2051173121 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f20a0004800 msgr2=0x7f20b4148eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:47.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.133+0000 7f20b9555640 1 --2- 172.21.15.67:0/2051173121 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f20a0004800 0x7f20b4148eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:47.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.133+0000 7f20b8d54640 1 --2- 172.21.15.67:0/2051173121 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f20a0004800 0x7f20b4148eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:47.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.133+0000 7f20b8d54640 1 -- 172.21.15.67:0/2051173121 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20b4160a00 msgr2=0x7f20b4160eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:47.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.133+0000 7f20b8d54640 1 --2- 172.21.15.67:0/2051173121 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20b4160a00 0x7f20b4160eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:47.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.133+0000 7f20b8d54640 1 -- 172.21.15.67:0/2051173121 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20b40b7410 con 0x7f20b40b58b0 2024-03-24T00:49:47.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.133+0000 7f20b9d56640 1 --2- 172.21.15.67:0/2051173121 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20b4160a00 0x7f20b4160eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:49:47.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.133+0000 7f20b8d54640 1 --2- 172.21.15.67:0/2051173121 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20b40b58b0 0x7f20b40b5d30 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7f20a4000c00 tx=0x7f20a4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:47.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.133+0000 7f20a9ffb640 1 -- 172.21.15.67:0/2051173121 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a4042020 con 0x7f20b40b58b0 2024-03-24T00:49:47.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.133+0000 7f20a9ffb640 1 -- 172.21.15.67:0/2051173121 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20a403a040 con 0x7f20b40b58b0 2024-03-24T00:49:47.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.133+0000 7f20bb7b9640 1 -- 172.21.15.67:0/2051173121 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20b40b6370 con 0x7f20b40b58b0 2024-03-24T00:49:47.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.133+0000 7f20a9ffb640 1 -- 172.21.15.67:0/2051173121 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a4039070 con 0x7f20b40b58b0 2024-03-24T00:49:47.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.134+0000 7f20bb7b9640 1 -- 172.21.15.67:0/2051173121 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20b40b6930 con 0x7f20b40b58b0 2024-03-24T00:49:47.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.134+0000 7f20bb7b9640 1 -- 172.21.15.67:0/2051173121 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f20b40b6ff0 con 0x7f20b40b58b0 2024-03-24T00:49:47.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.135+0000 7f20a9ffb640 1 -- 172.21.15.67:0/2051173121 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f20a4045020 con 0x7f20b40b58b0 2024-03-24T00:49:47.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.135+0000 7f20a9ffb640 1 --2- 172.21.15.67:0/2051173121 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f209806bbd0 0x7f209806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:47.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.135+0000 7f20a9ffb640 1 -- 172.21.15.67:0/2051173121 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f20a40abcf0 con 0x7f20b40b58b0 2024-03-24T00:49:47.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.135+0000 7f20a9ffb640 1 --2- 172.21.15.67:0/2051173121 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x7f20980721b0 0x7f20980745f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:47.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.135+0000 7f20a9ffb640 1 -- 172.21.15.67:0/2051173121 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f20a4044e70 con 0x7f20980721b0 2024-03-24T00:49:47.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.135+0000 7f20a9ffb640 1 -- 172.21.15.67:0/2051173121 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=125) v2 ==== 24+0+0 (secure 0 0 0) 0x7f20a40b3050 con 0x7f20b40b58b0 2024-03-24T00:49:47.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.136+0000 7f20b9555640 1 --2- 172.21.15.67:0/2051173121 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f209806bbd0 0x7f209806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:47.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.136+0000 7f20b9d56640 1 --2- 172.21.15.67:0/2051173121 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x7f20980721b0 0x7f20980745f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:47.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.136+0000 7f20b9d56640 1 --2- 172.21.15.67:0/2051173121 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x7f20980721b0 0x7f20980745f0 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:47.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.136+0000 7f20b9555640 1 --2- 172.21.15.67:0/2051173121 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f209806bbd0 0x7f209806e090 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f20b4071850 tx=0x7f20ac008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:47.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.137+0000 7f20a9ffb640 1 -- 172.21.15.67:0/2051173121 <== osd.5 v2:172.21.15.72:6800/288486206 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f20a4044e70 con 0x7f20980721b0 2024-03-24T00:49:47.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.150+0000 7f20bb7b9640 1 -- 172.21.15.67:0/2051173121 --> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f20b404f400 con 0x7f20980721b0 2024-03-24T00:49:47.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.150+0000 7f20a9ffb640 1 -- 172.21.15.67:0/2051173121 <== osd.5 v2:172.21.15.72:6800/288486206 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f20b404f400 con 0x7f20980721b0 2024-03-24T00:49:47.150 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:49:47.150 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-24T00:49:47.150 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:49:47.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.150+0000 7f20bb7b9640 1 -- 172.21.15.67:0/2051173121 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x7f20980721b0 msgr2=0x7f20980745f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:47.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.150+0000 7f20bb7b9640 1 --2- 172.21.15.67:0/2051173121 >> [v2:172.21.15.72:6800/288486206,v1:172.21.15.72:6801/288486206] conn(0x7f20980721b0 0x7f20980745f0 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:47.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.151+0000 7f20bb7b9640 1 -- 172.21.15.67:0/2051173121 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f209806bbd0 msgr2=0x7f209806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:47.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.151+0000 7f20bb7b9640 1 --2- 172.21.15.67:0/2051173121 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f209806bbd0 0x7f209806e090 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f20b4071850 tx=0x7f20ac008040 comp rx=0 tx=0).stop 2024-03-24T00:49:47.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.151+0000 7f20bb7b9640 1 -- 172.21.15.67:0/2051173121 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20b40b58b0 msgr2=0x7f20b40b5d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:47.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.151+0000 7f20bb7b9640 1 --2- 172.21.15.67:0/2051173121 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20b40b58b0 0x7f20b40b5d30 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7f20a4000c00 tx=0x7f20a4004730 comp rx=0 tx=0).stop 2024-03-24T00:49:47.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.151+0000 7f20b9d56640 1 -- 172.21.15.67:0/2051173121 reap_dead start 2024-03-24T00:49:47.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.151+0000 7f20bb7b9640 1 -- 172.21.15.67:0/2051173121 shutdown_connections 2024-03-24T00:49:47.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.151+0000 7f20bb7b9640 1 -- 172.21.15.67:0/2051173121 >> 172.21.15.67:0/2051173121 conn(0x7f20b40b36c0 msgr2=0x7f20b415e0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:47.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.151+0000 7f20bb7b9640 1 -- 172.21.15.67:0/2051173121 shutdown_connections 2024-03-24T00:49:47.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.151+0000 7f20bb7b9640 1 -- 172.21.15.67:0/2051173121 wait complete. 2024-03-24T00:49:47.161 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-03-24T00:49:47.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.263+0000 7f473229a640 1 Processor -- start 2024-03-24T00:49:47.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.263+0000 7f473229a640 1 -- start start 2024-03-24T00:49:47.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.264+0000 7f473229a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f472c06b040 0x7f472c06b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:47.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.264+0000 7f473229a640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f472c0cb0f0 con 0x7f472c0b0210 2024-03-24T00:49:47.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.264+0000 7f473229a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f472c0d0080 con 0x7f472c15e0a0 2024-03-24T00:49:47.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.264+0000 7f473229a640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f472c06b9e0 con 0x7f472c06b040 2024-03-24T00:49:47.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.264+0000 7f4730837640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f472c0b0210 0x7f472c0b0640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56286/0 (socket says 172.21.15.67:56286) 2024-03-24T00:49:47.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.264+0000 7f4730837640 1 -- 172.21.15.67:0/1018982986 learned_addr learned my addr 172.21.15.67:0/1018982986 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:49:47.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.264+0000 7f472b7fe640 1 --2- 172.21.15.67:0/1018982986 >> v2:172.21.15.72:3300/0 conn(0x7f472c06b040 0x7f472c06b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:47.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.264+0000 7f472affd640 1 -- 172.21.15.67:0/1018982986 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 66479693 0 0) 0x7f472c0cb0f0 con 0x7f472c0b0210 2024-03-24T00:49:47.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.264+0000 7f472affd640 1 -- 172.21.15.67:0/1018982986 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4718003520 con 0x7f472c0b0210 2024-03-24T00:49:47.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.264+0000 7f472affd640 1 -- 172.21.15.67:0/1018982986 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1558250498 0 0) 0x7f4718003520 con 0x7f472c0b0210 2024-03-24T00:49:47.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.264+0000 7f472affd640 1 -- 172.21.15.67:0/1018982986 >> v2:172.21.15.72:3300/0 conn(0x7f472c06b040 msgr2=0x7f472c06b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:47.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.264+0000 7f472affd640 1 --2- 172.21.15.67:0/1018982986 >> v2:172.21.15.72:3300/0 conn(0x7f472c06b040 0x7f472c06b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:47.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.264+0000 7f472affd640 1 -- 172.21.15.67:0/1018982986 >> v1:172.21.15.72:6789/0 conn(0x7f472c15e0a0 legacy=0x7f472c0cd6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:47.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.264+0000 7f472affd640 1 -- 172.21.15.67:0/1018982986 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f472c0d0360 con 0x7f472c0b0210 2024-03-24T00:49:47.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.264+0000 7f472affd640 1 -- 172.21.15.67:0/1018982986 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f47240019f0 con 0x7f472c0b0210 2024-03-24T00:49:47.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.264+0000 7f472affd640 1 -- 172.21.15.67:0/1018982986 >> v1:172.21.15.67:6789/0 conn(0x7f472c0b0210 legacy=0x7f472c0b0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:47.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.265+0000 7f472affd640 1 --2- 172.21.15.67:0/1018982986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47180039b0 0x7f4718003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:47.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.265+0000 7f472affd640 1 --2- 172.21.15.67:0/1018982986 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4718004800 0x7f4718004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:47.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.265+0000 7f472affd640 1 --2- 172.21.15.67:0/1018982986 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47180052e0 0x7f471800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:47.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.265+0000 7f472affd640 1 -- 172.21.15.67:0/1018982986 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f471800c180 con 0x7f47180039b0 2024-03-24T00:49:47.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.265+0000 7f472affd640 1 -- 172.21.15.67:0/1018982986 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f471800c330 con 0x7f47180052e0 2024-03-24T00:49:47.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.265+0000 7f472affd640 1 -- 172.21.15.67:0/1018982986 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f471800c530 con 0x7f4718004800 2024-03-24T00:49:47.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.265+0000 7f472bfff640 1 --2- 172.21.15.67:0/1018982986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47180039b0 0x7f4718003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:47.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.265+0000 7f4730837640 1 --2- 172.21.15.67:0/1018982986 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47180052e0 0x7f471800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:47.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.265+0000 7f472b7fe640 1 --2- 172.21.15.67:0/1018982986 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4718004800 0x7f4718004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:47.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.265+0000 7f472bfff640 1 -- 172.21.15.67:0/1018982986 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4718004800 msgr2=0x7f4718004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:47.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.265+0000 7f472bfff640 1 --2- 172.21.15.67:0/1018982986 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4718004800 0x7f4718004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:47.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.265+0000 7f472bfff640 1 -- 172.21.15.67:0/1018982986 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47180052e0 msgr2=0x7f471800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:47.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.265+0000 7f472bfff640 1 --2- 172.21.15.67:0/1018982986 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47180052e0 0x7f471800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:47.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.265+0000 7f472bfff640 1 -- 172.21.15.67:0/1018982986 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4718003520 con 0x7f47180039b0 2024-03-24T00:49:47.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.265+0000 7f4730837640 1 -- 172.21.15.67:0/1018982986 reap_dead start 2024-03-24T00:49:47.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.266+0000 7f472bfff640 1 --2- 172.21.15.67:0/1018982986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47180039b0 0x7f4718003dd0 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f471c001720 tx=0x7f4718005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea0ca2069e93eb97 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:47.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.266+0000 7f472affd640 1 -- 172.21.15.67:0/1018982986 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f471c001370 con 0x7f47180039b0 2024-03-24T00:49:47.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.266+0000 7f472affd640 1 -- 172.21.15.67:0/1018982986 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f471c0052d0 con 0x7f47180039b0 2024-03-24T00:49:47.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.266+0000 7f472affd640 1 -- 172.21.15.67:0/1018982986 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f471c003830 con 0x7f47180039b0 2024-03-24T00:49:47.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.266+0000 7f473229a640 1 -- 172.21.15.67:0/1018982986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47180039b0 msgr2=0x7f4718003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:47.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.266+0000 7f473229a640 1 --2- 172.21.15.67:0/1018982986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47180039b0 0x7f4718003dd0 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f471c001720 tx=0x7f4718005830 comp rx=0 tx=0).stop 2024-03-24T00:49:47.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.266+0000 7f473229a640 1 -- 172.21.15.67:0/1018982986 shutdown_connections 2024-03-24T00:49:47.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.266+0000 7f473229a640 1 --2- 172.21.15.67:0/1018982986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47180039b0 0x7f4718003dd0 unknown :-1 s=CLOSED pgs=1957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:47.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.266+0000 7f473229a640 1 -- 172.21.15.67:0/1018982986 >> 172.21.15.67:0/1018982986 conn(0x7f472c099a30 msgr2=0x7f472c099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:47.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.266+0000 7f473229a640 1 -- 172.21.15.67:0/1018982986 shutdown_connections 2024-03-24T00:49:47.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.266+0000 7f473229a640 1 -- 172.21.15.67:0/1018982986 wait complete. 2024-03-24T00:49:47.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.268+0000 7f473229a640 1 Processor -- start 2024-03-24T00:49:47.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.268+0000 7f473229a640 1 -- start start 2024-03-24T00:49:47.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.268+0000 7f473229a640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f47180039b0 0x7f472c157930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:47.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.268+0000 7f473229a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f472c1601c0 0x7f472c160640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:47.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.268+0000 7f473229a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f472c0aed00 0x7f472c0af180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:47.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.268+0000 7f473229a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f472c0ce350 con 0x7f472c0aed00 2024-03-24T00:49:47.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.268+0000 7f473229a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f472c0afd80 con 0x7f472c1601c0 2024-03-24T00:49:47.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.268+0000 7f473229a640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f472c0ce4d0 con 0x7f47180039b0 2024-03-24T00:49:47.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.269+0000 7f4730837640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f472c0aed00 0x7f472c0af180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:47.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.269+0000 7f4730837640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f472c0aed00 0x7f472c0af180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43768/0 (socket says 172.21.15.67:43768) 2024-03-24T00:49:47.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.269+0000 7f4730837640 1 -- 172.21.15.67:0/1866834439 learned_addr learned my addr 172.21.15.67:0/1866834439 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:47.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.269+0000 7f472b7fe640 1 --2- 172.21.15.67:0/1866834439 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f472c1601c0 0x7f472c160640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:47.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.269+0000 7f472bfff640 1 --2- 172.21.15.67:0/1866834439 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f47180039b0 0x7f472c157930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:47.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.269+0000 7f4730837640 1 -- 172.21.15.67:0/1866834439 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f47180039b0 msgr2=0x7f472c157930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:47.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.269+0000 7f4730837640 1 --2- 172.21.15.67:0/1866834439 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f47180039b0 0x7f472c157930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:47.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.269+0000 7f4730837640 1 -- 172.21.15.67:0/1866834439 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f472c1601c0 msgr2=0x7f472c160640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:47.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.269+0000 7f4730837640 1 --2- 172.21.15.67:0/1866834439 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f472c1601c0 0x7f472c160640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:47.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.269+0000 7f4730837640 1 -- 172.21.15.67:0/1866834439 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f472c0c7c10 con 0x7f472c0aed00 2024-03-24T00:49:47.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.269+0000 7f4730837640 1 --2- 172.21.15.67:0/1866834439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f472c0aed00 0x7f472c0af180 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7f472c0ccfe0 tx=0x7f47240079d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:47.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.269+0000 7f4728ff9640 1 -- 172.21.15.67:0/1866834439 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f472400aaa0 con 0x7f472c0aed00 2024-03-24T00:49:47.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.269+0000 7f4728ff9640 1 -- 172.21.15.67:0/1866834439 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4724018070 con 0x7f472c0aed00 2024-03-24T00:49:47.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.269+0000 7f473229a640 1 -- 172.21.15.67:0/1866834439 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f472c0c7ef0 con 0x7f472c0aed00 2024-03-24T00:49:47.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.269+0000 7f4728ff9640 1 -- 172.21.15.67:0/1866834439 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47240133f0 con 0x7f472c0aed00 2024-03-24T00:49:47.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.269+0000 7f473229a640 1 -- 172.21.15.67:0/1866834439 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f472c0c84b0 con 0x7f472c0aed00 2024-03-24T00:49:47.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.270+0000 7f473229a640 1 -- 172.21.15.67:0/1866834439 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f472c04f400 con 0x7f472c0aed00 2024-03-24T00:49:47.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.271+0000 7f4728ff9640 1 -- 172.21.15.67:0/1866834439 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4724013590 con 0x7f472c0aed00 2024-03-24T00:49:47.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.271+0000 7f4728ff9640 1 --2- 172.21.15.67:0/1866834439 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f470c06bbd0 0x7f470c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:47.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.271+0000 7f4728ff9640 1 -- 172.21.15.67:0/1866834439 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f472408ac30 con 0x7f472c0aed00 2024-03-24T00:49:47.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.271+0000 7f472bfff640 1 --2- 172.21.15.67:0/1866834439 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f470c06bbd0 0x7f470c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:47.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.272+0000 7f472bfff640 1 --2- 172.21.15.67:0/1866834439 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f470c06bbd0 0x7f470c06e090 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f471c033000 tx=0x7f471c009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:47.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.276+0000 7f4728ff9640 1 -- 172.21.15.67:0/1866834439 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4724059870 con 0x7f472c0aed00 2024-03-24T00:49:47.301 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:49:47.303 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:49:47.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.458+0000 7f473229a640 1 -- 172.21.15.67:0/1866834439 --> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f472c15a540 con 0x7f470c06bbd0 2024-03-24T00:49:47.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.458+0000 7f4728ff9640 1 -- 172.21.15.67:0/1866834439 <== mgr.4105 v2:172.21.15.72:6832/2768292274 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f472c15a540 con 0x7f470c06bbd0 2024-03-24T00:49:47.459 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 5 to deep-scrub 2024-03-24T00:49:47.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.460+0000 7f473229a640 1 -- 172.21.15.67:0/1866834439 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f470c06bbd0 msgr2=0x7f470c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:47.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.460+0000 7f473229a640 1 --2- 172.21.15.67:0/1866834439 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f470c06bbd0 0x7f470c06e090 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f471c033000 tx=0x7f471c009030 comp rx=0 tx=0).stop 2024-03-24T00:49:47.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.460+0000 7f473229a640 1 -- 172.21.15.67:0/1866834439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f472c0aed00 msgr2=0x7f472c0af180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:47.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.460+0000 7f473229a640 1 --2- 172.21.15.67:0/1866834439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f472c0aed00 0x7f472c0af180 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7f472c0ccfe0 tx=0x7f47240079d0 comp rx=0 tx=0).stop 2024-03-24T00:49:47.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.460+0000 7f473229a640 1 -- 172.21.15.67:0/1866834439 shutdown_connections 2024-03-24T00:49:47.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.460+0000 7f473229a640 1 --2- 172.21.15.67:0/1866834439 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f470c06bbd0 0x7f470c06e090 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:47.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.460+0000 7f473229a640 1 --2- 172.21.15.67:0/1866834439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f472c0aed00 0x7f472c0af180 unknown :-1 s=CLOSED pgs=1958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:47.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.460+0000 7f473229a640 1 --2- 172.21.15.67:0/1866834439 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f472c1601c0 0x7f472c160640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:47.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.460+0000 7f473229a640 1 --2- 172.21.15.67:0/1866834439 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f47180039b0 0x7f472c157930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:47.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.460+0000 7f473229a640 1 -- 172.21.15.67:0/1866834439 >> 172.21.15.67:0/1866834439 conn(0x7f472c099a30 msgr2=0x7f472c0cc270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:47.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.460+0000 7f473229a640 1 -- 172.21.15.67:0/1866834439 shutdown_connections 2024-03-24T00:49:47.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.460+0000 7f473229a640 1 -- 172.21.15.67:0/1866834439 wait complete. 2024-03-24T00:49:47.471 INFO:tasks.ceph:Scrubbing osd.6 2024-03-24T00:49:47.471 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-03-24T00:49:47.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.577+0000 7f61c9b59640 1 Processor -- start 2024-03-24T00:49:47.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.577+0000 7f61c9b59640 1 -- start start 2024-03-24T00:49:47.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.578+0000 7f61c9b59640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f61c40aeee0 0x7f61c40cef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:47.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.578+0000 7f61c9b59640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f61c4071ad0 0x7f61c40ccc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:47.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.578+0000 7f61c9b59640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61c4072300 con 0x7f61c40cd170 2024-03-24T00:49:47.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.578+0000 7f61c9b59640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61c40cd5e0 con 0x7f61c4071ad0 2024-03-24T00:49:47.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.578+0000 7f61c9b59640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61c406db30 con 0x7f61c40aeee0 2024-03-24T00:49:47.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.578+0000 7f61c2ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f61c4071ad0 0x7f61c40ccc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:47.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.578+0000 7f61c3fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f61c40cd170 0x7f61c415f8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56298/0 (socket says 172.21.15.67:56298) 2024-03-24T00:49:47.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.578+0000 7f61c3fff640 1 -- 172.21.15.67:0/866295041 learned_addr learned my addr 172.21.15.67:0/866295041 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:49:47.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.578+0000 7f61c37fe640 1 --2- 172.21.15.67:0/866295041 >> v2:172.21.15.72:3300/0 conn(0x7f61c40aeee0 0x7f61c40cef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:47.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.578+0000 7f61c2ffd640 1 -- 172.21.15.67:0/866295041 >> v2:172.21.15.72:3300/0 conn(0x7f61c40aeee0 msgr2=0x7f61c40cef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:47.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.578+0000 7f61c2ffd640 1 --2- 172.21.15.67:0/866295041 >> v2:172.21.15.72:3300/0 conn(0x7f61c40aeee0 0x7f61c40cef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:47.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.578+0000 7f61c2ffd640 1 -- 172.21.15.67:0/866295041 >> v1:172.21.15.67:6789/0 conn(0x7f61c40cd170 legacy=0x7f61c415f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:47.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.578+0000 7f61c2ffd640 1 -- 172.21.15.67:0/866295041 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c40725e0 con 0x7f61c4071ad0 2024-03-24T00:49:47.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.578+0000 7f61c27fc640 1 -- 172.21.15.67:0/866295041 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 69454726 0 0) 0x7f61c4072300 con 0x7f61c40cd170 2024-03-24T00:49:47.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.579+0000 7f61c2ffd640 1 --2- 172.21.15.67:0/866295041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61c4071ad0 0x7f61c40ccc30 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7f61c4072f70 tx=0x7f61b802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b7d84400cf64e88 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:47.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.579+0000 7f61c27fc640 1 -- 172.21.15.67:0/866295041 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61b8035020 con 0x7f61c4071ad0 2024-03-24T00:49:47.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.579+0000 7f61c27fc640 1 -- 172.21.15.67:0/866295041 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61b8031830 con 0x7f61c4071ad0 2024-03-24T00:49:47.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.579+0000 7f61c27fc640 1 -- 172.21.15.67:0/866295041 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61b8031c00 con 0x7f61c4071ad0 2024-03-24T00:49:47.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.579+0000 7f61c9b59640 1 -- 172.21.15.67:0/866295041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61c4071ad0 msgr2=0x7f61c40ccc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:47.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.579+0000 7f61c9b59640 1 --2- 172.21.15.67:0/866295041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61c4071ad0 0x7f61c40ccc30 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7f61c4072f70 tx=0x7f61b802b8d0 comp rx=0 tx=0).stop 2024-03-24T00:49:47.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.579+0000 7f61c9b59640 1 -- 172.21.15.67:0/866295041 shutdown_connections 2024-03-24T00:49:47.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.579+0000 7f61c9b59640 1 --2- 172.21.15.67:0/866295041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61c4071ad0 0x7f61c40ccc30 unknown :-1 s=CLOSED pgs=1960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:47.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.579+0000 7f61c9b59640 1 --2- 172.21.15.67:0/866295041 >> v2:172.21.15.72:3300/0 conn(0x7f61c40aeee0 0x7f61c40cef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:47.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.579+0000 7f61c9b59640 1 -- 172.21.15.67:0/866295041 >> 172.21.15.67:0/866295041 conn(0x7f61c40cf9e0 msgr2=0x7f61c40cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:47.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.580+0000 7f61c9b59640 1 -- 172.21.15.67:0/866295041 shutdown_connections 2024-03-24T00:49:47.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.580+0000 7f61c9b59640 1 -- 172.21.15.67:0/866295041 wait complete. 2024-03-24T00:49:47.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.581+0000 7f61c9b59640 1 Processor -- start 2024-03-24T00:49:47.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.581+0000 7f61c9b59640 1 -- start start 2024-03-24T00:49:47.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.582+0000 7f61c9b59640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f61c4071ad0 0x7f61c413a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:47.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.582+0000 7f61c9b59640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f61c40aeee0 0x7f61c4138fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:47.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.582+0000 7f61c9b59640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61c40cd170 0x7f61c413c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:47.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.582+0000 7f61c9b59640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c4071f40 con 0x7f61c40cd170 2024-03-24T00:49:47.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.582+0000 7f61c9b59640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c4072ba0 con 0x7f61c4071ad0 2024-03-24T00:49:47.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.582+0000 7f61c9b59640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61c40720c0 con 0x7f61c40aeee0 2024-03-24T00:49:47.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.582+0000 7f61c3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61c40cd170 0x7f61c413c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:47.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.582+0000 7f61c3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61c40cd170 0x7f61c413c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43778/0 (socket says 172.21.15.67:43778) 2024-03-24T00:49:47.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.582+0000 7f61c3fff640 1 -- 172.21.15.67:0/2130926581 learned_addr learned my addr 172.21.15.67:0/2130926581 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:47.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.582+0000 7f61c37fe640 1 --2- 172.21.15.67:0/2130926581 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f61c4071ad0 0x7f61c413a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:47.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.582+0000 7f61c2ffd640 1 --2- 172.21.15.67:0/2130926581 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f61c40aeee0 0x7f61c4138fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:47.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.582+0000 7f61c3fff640 1 -- 172.21.15.67:0/2130926581 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f61c40aeee0 msgr2=0x7f61c4138fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:47.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.582+0000 7f61c3fff640 1 --2- 172.21.15.67:0/2130926581 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f61c40aeee0 0x7f61c4138fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:47.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.582+0000 7f61c3fff640 1 -- 172.21.15.67:0/2130926581 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f61c4071ad0 msgr2=0x7f61c413a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:47.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.582+0000 7f61c3fff640 1 --2- 172.21.15.67:0/2130926581 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f61c4071ad0 0x7f61c413a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:47.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.582+0000 7f61c3fff640 1 -- 172.21.15.67:0/2130926581 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c415ce10 con 0x7f61c40cd170 2024-03-24T00:49:47.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.582+0000 7f61c3fff640 1 --2- 172.21.15.67:0/2130926581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61c40cd170 0x7f61c413c970 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f61bc00c700 tx=0x7f61bc00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:47.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.582+0000 7f61abfff640 1 -- 172.21.15.67:0/2130926581 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61bc018070 con 0x7f61c40cd170 2024-03-24T00:49:47.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.582+0000 7f61abfff640 1 -- 172.21.15.67:0/2130926581 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61bc00ce10 con 0x7f61c40cd170 2024-03-24T00:49:47.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.582+0000 7f61c9b59640 1 -- 172.21.15.67:0/2130926581 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61c415d0f0 con 0x7f61c40cd170 2024-03-24T00:49:47.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.582+0000 7f61abfff640 1 -- 172.21.15.67:0/2130926581 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61bc00a860 con 0x7f61c40cd170 2024-03-24T00:49:47.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.582+0000 7f61c9b59640 1 -- 172.21.15.67:0/2130926581 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61c415d6b0 con 0x7f61c40cd170 2024-03-24T00:49:47.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.583+0000 7f61c9b59640 1 -- 172.21.15.67:0/2130926581 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f61c415cbc0 con 0x7f61c40cd170 2024-03-24T00:49:47.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.584+0000 7f61abfff640 1 -- 172.21.15.67:0/2130926581 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f61bc01b070 con 0x7f61c40cd170 2024-03-24T00:49:47.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.584+0000 7f61abfff640 1 --2- 172.21.15.67:0/2130926581 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f61b006bbd0 0x7f61b006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:47.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.584+0000 7f61abfff640 1 -- 172.21.15.67:0/2130926581 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f61bc08a4c0 con 0x7f61c40cd170 2024-03-24T00:49:47.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.584+0000 7f61abfff640 1 --2- 172.21.15.67:0/2130926581 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x7f61b00721b0 0x7f61b00745f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:47.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.584+0000 7f61abfff640 1 -- 172.21.15.67:0/2130926581 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f61b0074cc0 con 0x7f61b00721b0 2024-03-24T00:49:47.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.584+0000 7f61c37fe640 1 --2- 172.21.15.67:0/2130926581 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f61b006bbd0 0x7f61b006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:47.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.584+0000 7f61abfff640 1 -- 172.21.15.67:0/2130926581 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=125) v2 ==== 24+0+0 (secure 0 0 0) 0x7f61bc08a950 con 0x7f61c40cd170 2024-03-24T00:49:47.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.585+0000 7f61c2ffd640 1 --2- 172.21.15.67:0/2130926581 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x7f61b00721b0 0x7f61b00745f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:47.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.585+0000 7f61c37fe640 1 --2- 172.21.15.67:0/2130926581 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f61b006bbd0 0x7f61b006e090 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f61b4006fd0 tx=0x7f61b4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:47.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.585+0000 7f61c2ffd640 1 --2- 172.21.15.67:0/2130926581 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x7f61b00721b0 0x7f61b00745f0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:47.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.586+0000 7f61abfff640 1 -- 172.21.15.67:0/2130926581 <== osd.6 v2:172.21.15.72:6816/1725206562 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f61b0074cc0 con 0x7f61b00721b0 2024-03-24T00:49:47.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.599+0000 7f61c9b59640 1 -- 172.21.15.67:0/2130926581 --> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f61c404f400 con 0x7f61b00721b0 2024-03-24T00:49:47.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.600+0000 7f61abfff640 1 -- 172.21.15.67:0/2130926581 <== osd.6 v2:172.21.15.72:6816/1725206562 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f61c404f400 con 0x7f61b00721b0 2024-03-24T00:49:47.600 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:49:47.600 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-24T00:49:47.600 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:49:47.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.600+0000 7f61c9b59640 1 -- 172.21.15.67:0/2130926581 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x7f61b00721b0 msgr2=0x7f61b00745f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:47.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.600+0000 7f61c9b59640 1 --2- 172.21.15.67:0/2130926581 >> [v2:172.21.15.72:6816/1725206562,v1:172.21.15.72:6817/1725206562] conn(0x7f61b00721b0 0x7f61b00745f0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:47.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.600+0000 7f61c9b59640 1 -- 172.21.15.67:0/2130926581 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f61b006bbd0 msgr2=0x7f61b006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:47.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.600+0000 7f61c9b59640 1 --2- 172.21.15.67:0/2130926581 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f61b006bbd0 0x7f61b006e090 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f61b4006fd0 tx=0x7f61b4008040 comp rx=0 tx=0).stop 2024-03-24T00:49:47.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.600+0000 7f61c9b59640 1 -- 172.21.15.67:0/2130926581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61c40cd170 msgr2=0x7f61c413c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:47.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.600+0000 7f61c9b59640 1 --2- 172.21.15.67:0/2130926581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61c40cd170 0x7f61c413c970 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f61bc00c700 tx=0x7f61bc00cbc0 comp rx=0 tx=0).stop 2024-03-24T00:49:47.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.600+0000 7f61c3fff640 1 -- 172.21.15.67:0/2130926581 reap_dead start 2024-03-24T00:49:47.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.600+0000 7f61c9b59640 1 -- 172.21.15.67:0/2130926581 shutdown_connections 2024-03-24T00:49:47.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.600+0000 7f61c9b59640 1 -- 172.21.15.67:0/2130926581 >> 172.21.15.67:0/2130926581 conn(0x7f61c40cf9e0 msgr2=0x7f61c4144470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:47.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.600+0000 7f61c9b59640 1 -- 172.21.15.67:0/2130926581 shutdown_connections 2024-03-24T00:49:47.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.600+0000 7f61c9b59640 1 -- 172.21.15.67:0/2130926581 wait complete. 2024-03-24T00:49:47.610 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-03-24T00:49:47.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.713+0000 7f3c3883f640 1 Processor -- start 2024-03-24T00:49:47.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.713+0000 7f3c3883f640 1 -- start start 2024-03-24T00:49:47.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.714+0000 7f3c3883f640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3c340b1eb0 0x7f3c34099d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:47.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.714+0000 7f3c3883f640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3c3409a240 0x7f3c340cd410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:47.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.714+0000 7f3c3883f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c340b29b0 con 0x7f3c3406de60 2024-03-24T00:49:47.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.714+0000 7f3c3883f640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c340cc9e0 con 0x7f3c340b1eb0 2024-03-24T00:49:47.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.714+0000 7f3c3883f640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c340ccf80 con 0x7f3c3409a240 2024-03-24T00:49:47.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.714+0000 7f3c3259c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3c3406de60 0x7f3c3409ae80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56310/0 (socket says 172.21.15.67:56310) 2024-03-24T00:49:47.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.714+0000 7f3c3259c640 1 -- 172.21.15.67:0/2216008515 learned_addr learned my addr 172.21.15.67:0/2216008515 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:49:47.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.714+0000 7f3c2bfff640 1 --2- 172.21.15.67:0/2216008515 >> v2:172.21.15.72:3300/0 conn(0x7f3c340b1eb0 0x7f3c34099d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:47.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.714+0000 7f3c32d9d640 1 --2- 172.21.15.67:0/2216008515 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3c3409a240 0x7f3c340cd410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:47.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.714+0000 7f3c31d9b640 1 -- 172.21.15.67:0/2216008515 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1879897436 0 0) 0x7f3c340b29b0 con 0x7f3c3406de60 2024-03-24T00:49:47.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.714+0000 7f3c31d9b640 1 -- 172.21.15.67:0/2216008515 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c20003520 con 0x7f3c3406de60 2024-03-24T00:49:47.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.714+0000 7f3c2bfff640 1 -- 172.21.15.67:0/2216008515 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3c3409a240 msgr2=0x7f3c340cd410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:47.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.714+0000 7f3c2bfff640 1 --2- 172.21.15.67:0/2216008515 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3c3409a240 0x7f3c340cd410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:47.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.715+0000 7f3c2bfff640 1 -- 172.21.15.67:0/2216008515 >> v1:172.21.15.67:6789/0 conn(0x7f3c3406de60 legacy=0x7f3c3409ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:47.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.715+0000 7f3c31d9b640 1 -- 172.21.15.67:0/2216008515 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4066130624 0 0) 0x7f3c20003520 con 0x7f3c3406de60 2024-03-24T00:49:47.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.715+0000 7f3c2bfff640 1 -- 172.21.15.67:0/2216008515 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c340b23f0 con 0x7f3c340b1eb0 2024-03-24T00:49:47.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.715+0000 7f3c2bfff640 1 --2- 172.21.15.67:0/2216008515 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3c340b1eb0 0x7f3c34099d00 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c0097a0 tx=0x7f3c1c02b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3592b03f8cd46e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:47.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.716+0000 7f3c31d9b640 1 -- 172.21.15.67:0/2216008515 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c1c037070 con 0x7f3c340b1eb0 2024-03-24T00:49:47.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.716+0000 7f3c31d9b640 1 -- 172.21.15.67:0/2216008515 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c1c030510 con 0x7f3c340b1eb0 2024-03-24T00:49:47.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.716+0000 7f3c31d9b640 1 -- 172.21.15.67:0/2216008515 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c24001670 con 0x7f3c340b1eb0 2024-03-24T00:49:47.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.716+0000 7f3c3883f640 1 -- 172.21.15.67:0/2216008515 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3c340b1eb0 msgr2=0x7f3c34099d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:47.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.716+0000 7f3c3883f640 1 --2- 172.21.15.67:0/2216008515 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3c340b1eb0 0x7f3c34099d00 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c0097a0 tx=0x7f3c1c02b790 comp rx=0 tx=0).stop 2024-03-24T00:49:47.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.716+0000 7f3c3883f640 1 -- 172.21.15.67:0/2216008515 shutdown_connections 2024-03-24T00:49:47.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.716+0000 7f3c3883f640 1 --2- 172.21.15.67:0/2216008515 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3c3409a240 0x7f3c340cd410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:47.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.716+0000 7f3c3883f640 1 --2- 172.21.15.67:0/2216008515 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3c340b1eb0 0x7f3c34099d00 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:47.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.716+0000 7f3c3883f640 1 -- 172.21.15.67:0/2216008515 >> 172.21.15.67:0/2216008515 conn(0x7f3c340b3650 msgr2=0x7f3c340b3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:47.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.716+0000 7f3c3883f640 1 -- 172.21.15.67:0/2216008515 shutdown_connections 2024-03-24T00:49:47.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.716+0000 7f3c3883f640 1 -- 172.21.15.67:0/2216008515 wait complete. 2024-03-24T00:49:47.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.717+0000 7f3c3883f640 1 Processor -- start 2024-03-24T00:49:47.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.717+0000 7f3c3883f640 1 -- start start 2024-03-24T00:49:47.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.718+0000 7f3c3883f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c3406de60 0x7f3c340c70c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:47.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.718+0000 7f3c3883f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3c3409a240 0x7f3c340c8130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:47.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.718+0000 7f3c3883f640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3c340b1eb0 0x7f3c34154f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:47.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.718+0000 7f3c3883f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c3406c150 con 0x7f3c3406de60 2024-03-24T00:49:47.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.718+0000 7f3c3883f640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c340ccc10 con 0x7f3c3409a240 2024-03-24T00:49:47.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.718+0000 7f3c3883f640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c3406c2d0 con 0x7f3c340b1eb0 2024-03-24T00:49:47.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.718+0000 7f3c3259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c3406de60 0x7f3c340c70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:47.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.718+0000 7f3c3259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c3406de60 0x7f3c340c70c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43790/0 (socket says 172.21.15.67:43790) 2024-03-24T00:49:47.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.718+0000 7f3c3259c640 1 -- 172.21.15.67:0/1351695559 learned_addr learned my addr 172.21.15.67:0/1351695559 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:47.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.718+0000 7f3c2bfff640 1 --2- 172.21.15.67:0/1351695559 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3c3409a240 0x7f3c340c8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:47.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.718+0000 7f3c32d9d640 1 --2- 172.21.15.67:0/1351695559 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3c340b1eb0 0x7f3c34154f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:47.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.718+0000 7f3c3259c640 1 -- 172.21.15.67:0/1351695559 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3c340b1eb0 msgr2=0x7f3c34154f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:47.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.718+0000 7f3c3259c640 1 --2- 172.21.15.67:0/1351695559 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3c340b1eb0 0x7f3c34154f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:47.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.719+0000 7f3c3259c640 1 -- 172.21.15.67:0/1351695559 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3c3409a240 msgr2=0x7f3c340c8130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:47.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.719+0000 7f3c3259c640 1 --2- 172.21.15.67:0/1351695559 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3c3409a240 0x7f3c340c8130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:47.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.719+0000 7f3c3259c640 1 -- 172.21.15.67:0/1351695559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c340b4dc0 con 0x7f3c3406de60 2024-03-24T00:49:47.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.719+0000 7f3c3259c640 1 --2- 172.21.15.67:0/1351695559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c3406de60 0x7f3c340c70c0 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f3c240017e0 tx=0x7f3c2400cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:47.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.719+0000 7f3c2affd640 1 -- 172.21.15.67:0/1351695559 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c24005ad0 con 0x7f3c3406de60 2024-03-24T00:49:47.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.719+0000 7f3c2affd640 1 -- 172.21.15.67:0/1351695559 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c24005c90 con 0x7f3c3406de60 2024-03-24T00:49:47.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.719+0000 7f3c3883f640 1 -- 172.21.15.67:0/1351695559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c340b50a0 con 0x7f3c3406de60 2024-03-24T00:49:47.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.719+0000 7f3c2affd640 1 -- 172.21.15.67:0/1351695559 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c240072b0 con 0x7f3c3406de60 2024-03-24T00:49:47.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.719+0000 7f3c3883f640 1 -- 172.21.15.67:0/1351695559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c340b5660 con 0x7f3c3406de60 2024-03-24T00:49:47.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.720+0000 7f3c3883f640 1 -- 172.21.15.67:0/1351695559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c3404f400 con 0x7f3c3406de60 2024-03-24T00:49:47.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.720+0000 7f3c2affd640 1 -- 172.21.15.67:0/1351695559 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3c24007450 con 0x7f3c3406de60 2024-03-24T00:49:47.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.721+0000 7f3c2affd640 1 --2- 172.21.15.67:0/1351695559 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3c1406bc00 0x7f3c1406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:47.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.721+0000 7f3c2affd640 1 -- 172.21.15.67:0/1351695559 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f3c2408a620 con 0x7f3c3406de60 2024-03-24T00:49:47.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.721+0000 7f3c2bfff640 1 --2- 172.21.15.67:0/1351695559 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3c1406bc00 0x7f3c1406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:47.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.721+0000 7f3c2bfff640 1 --2- 172.21.15.67:0/1351695559 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3c1406bc00 0x7f3c1406e0c0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c034000 tx=0x7f3c1c02f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:47.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.725+0000 7f3c2affd640 1 -- 172.21.15.67:0/1351695559 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3c24058b80 con 0x7f3c3406de60 2024-03-24T00:49:47.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.908+0000 7f3c3883f640 1 -- 172.21.15.67:0/1351695559 --> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f3c34070ff0 con 0x7f3c1406bc00 2024-03-24T00:49:47.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.909+0000 7f3c2affd640 1 -- 172.21.15.67:0/1351695559 <== mgr.4105 v2:172.21.15.72:6832/2768292274 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f3c34070ff0 con 0x7f3c1406bc00 2024-03-24T00:49:47.908 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 6 to deep-scrub 2024-03-24T00:49:47.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.910+0000 7f3c3883f640 1 -- 172.21.15.67:0/1351695559 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3c1406bc00 msgr2=0x7f3c1406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:47.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.910+0000 7f3c3883f640 1 --2- 172.21.15.67:0/1351695559 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3c1406bc00 0x7f3c1406e0c0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c034000 tx=0x7f3c1c02f000 comp rx=0 tx=0).stop 2024-03-24T00:49:47.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.910+0000 7f3c3883f640 1 -- 172.21.15.67:0/1351695559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c3406de60 msgr2=0x7f3c340c70c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:47.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.911+0000 7f3c3883f640 1 --2- 172.21.15.67:0/1351695559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c3406de60 0x7f3c340c70c0 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f3c240017e0 tx=0x7f3c2400cbe0 comp rx=0 tx=0).stop 2024-03-24T00:49:47.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.911+0000 7f3c3883f640 1 -- 172.21.15.67:0/1351695559 shutdown_connections 2024-03-24T00:49:47.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.911+0000 7f3c3883f640 1 --2- 172.21.15.67:0/1351695559 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f3c1406bc00 0x7f3c1406e0c0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:47.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.911+0000 7f3c3883f640 1 --2- 172.21.15.67:0/1351695559 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3c340b1eb0 0x7f3c34154f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:47.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.911+0000 7f3c3883f640 1 --2- 172.21.15.67:0/1351695559 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3c3409a240 0x7f3c340c8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:47.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.911+0000 7f3c3883f640 1 --2- 172.21.15.67:0/1351695559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c3406de60 0x7f3c340c70c0 unknown :-1 s=CLOSED pgs=1963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:47.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.911+0000 7f3c3883f640 1 -- 172.21.15.67:0/1351695559 >> 172.21.15.67:0/1351695559 conn(0x7f3c340b3650 msgr2=0x7f3c3406fdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:47.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.911+0000 7f3c3883f640 1 -- 172.21.15.67:0/1351695559 shutdown_connections 2024-03-24T00:49:47.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:47.911+0000 7f3c3883f640 1 -- 172.21.15.67:0/1351695559 wait complete. 2024-03-24T00:49:47.921 INFO:tasks.ceph:Scrubbing osd.7 2024-03-24T00:49:47.921 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-03-24T00:49:48.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.026+0000 7f9b83fc6640 1 Processor -- start 2024-03-24T00:49:48.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.026+0000 7f9b83fc6640 1 -- start start 2024-03-24T00:49:48.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.027+0000 7f9b83fc6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9b7c0ced60 0x7f9b7c0d0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:48.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.027+0000 7f9b83fc6640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f9b7c0d0b80 0x7f9b7c0d2b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:48.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.027+0000 7f9b83fc6640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b7c0cf1f0 con 0x7f9b7c15dfc0 2024-03-24T00:49:48.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.027+0000 7f9b83fc6640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b7c15e3a0 con 0x7f9b7c0d0b80 2024-03-24T00:49:48.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.027+0000 7f9b83fc6640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b7c0cf4d0 con 0x7f9b7c0ced60 2024-03-24T00:49:48.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.027+0000 7f9b81d62640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9b7c15dfc0 0x7f9b7c1624c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56312/0 (socket says 172.21.15.67:56312) 2024-03-24T00:49:48.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.027+0000 7f9b81d62640 1 -- 172.21.15.67:0/3735356115 learned_addr learned my addr 172.21.15.67:0/3735356115 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:49:48.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.028+0000 7f9b82563640 1 --2- 172.21.15.67:0/3735356115 >> v2:172.21.15.72:3300/0 conn(0x7f9b7c0d0b80 0x7f9b7c0d2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:48.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.028+0000 7f9b81561640 1 --2- 172.21.15.67:0/3735356115 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9b7c0ced60 0x7f9b7c0d0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:48.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.028+0000 7f9b80d60640 1 -- 172.21.15.67:0/3735356115 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1521108907 0 0) 0x7f9b7c0cf1f0 con 0x7f9b7c15dfc0 2024-03-24T00:49:48.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.028+0000 7f9b80d60640 1 -- 172.21.15.67:0/3735356115 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b64003520 con 0x7f9b7c15dfc0 2024-03-24T00:49:48.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.028+0000 7f9b80d60640 1 -- 172.21.15.67:0/3735356115 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 239360661 0 0) 0x7f9b64003520 con 0x7f9b7c15dfc0 2024-03-24T00:49:48.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.028+0000 7f9b80d60640 1 -- 172.21.15.67:0/3735356115 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9b7c0ced60 msgr2=0x7f9b7c0d0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:48.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.028+0000 7f9b80d60640 1 --2- 172.21.15.67:0/3735356115 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9b7c0ced60 0x7f9b7c0d0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:48.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.028+0000 7f9b80d60640 1 -- 172.21.15.67:0/3735356115 >> v2:172.21.15.72:3300/0 conn(0x7f9b7c0d0b80 msgr2=0x7f9b7c0d2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:48.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.028+0000 7f9b80d60640 1 --2- 172.21.15.67:0/3735356115 >> v2:172.21.15.72:3300/0 conn(0x7f9b7c0d0b80 0x7f9b7c0d2b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:48.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.028+0000 7f9b80d60640 1 -- 172.21.15.67:0/3735356115 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b7c0d2020 con 0x7f9b7c15dfc0 2024-03-24T00:49:48.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.028+0000 7f9b80d60640 1 -- 172.21.15.67:0/3735356115 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f9b74001670 con 0x7f9b7c15dfc0 2024-03-24T00:49:48.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.028+0000 7f9b82563640 1 --2- 172.21.15.67:0/3735356115 >> v2:172.21.15.72:3300/0 conn(0x7f9b7c0d0b80 0x7f9b7c0d2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:49:48.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.028+0000 7f9b80d60640 1 -- 172.21.15.67:0/3735356115 >> v1:172.21.15.67:6789/0 conn(0x7f9b7c15dfc0 legacy=0x7f9b7c1624c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:48.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.028+0000 7f9b80d60640 1 --2- 172.21.15.67:0/3735356115 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b640039b0 0x7f9b64003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:48.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.028+0000 7f9b80d60640 1 --2- 172.21.15.67:0/3735356115 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b64004800 0x7f9b64004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:48.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.028+0000 7f9b80d60640 1 --2- 172.21.15.67:0/3735356115 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9b640051c0 0x7f9b6400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:48.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.028+0000 7f9b80d60640 1 -- 172.21.15.67:0/3735356115 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b7c15e3a0 con 0x7f9b64004800 2024-03-24T00:49:48.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.028+0000 7f9b80d60640 1 -- 172.21.15.67:0/3735356115 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b7c0cf4d0 con 0x7f9b640039b0 2024-03-24T00:49:48.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.028+0000 7f9b80d60640 1 -- 172.21.15.67:0/3735356115 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b640045d0 con 0x7f9b640051c0 2024-03-24T00:49:48.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.028+0000 7f9b81561640 1 --2- 172.21.15.67:0/3735356115 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b64004800 0x7f9b64004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:48.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.028+0000 7f9b81d62640 1 --2- 172.21.15.67:0/3735356115 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b640039b0 0x7f9b64003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:48.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.028+0000 7f9b82563640 1 --2- 172.21.15.67:0/3735356115 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9b640051c0 0x7f9b6400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:48.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.028+0000 7f9b81561640 1 -- 172.21.15.67:0/3735356115 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9b640051c0 msgr2=0x7f9b6400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:48.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.028+0000 7f9b81561640 1 --2- 172.21.15.67:0/3735356115 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9b640051c0 0x7f9b6400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:48.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.028+0000 7f9b81561640 1 -- 172.21.15.67:0/3735356115 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b640039b0 msgr2=0x7f9b64003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:48.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.028+0000 7f9b81561640 1 --2- 172.21.15.67:0/3735356115 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b640039b0 0x7f9b64003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:48.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.028+0000 7f9b81561640 1 -- 172.21.15.67:0/3735356115 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b64003520 con 0x7f9b64004800 2024-03-24T00:49:48.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.029+0000 7f9b81561640 1 --2- 172.21.15.67:0/3735356115 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b64004800 0x7f9b64004c80 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f9b70009ac0 tx=0x7f9b64005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d11b36cb5ef6673 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:48.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.029+0000 7f9b80d60640 1 -- 172.21.15.67:0/3735356115 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b70039070 con 0x7f9b64004800 2024-03-24T00:49:48.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.029+0000 7f9b80d60640 1 -- 172.21.15.67:0/3735356115 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b7002c6b0 con 0x7f9b64004800 2024-03-24T00:49:48.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.029+0000 7f9b80d60640 1 -- 172.21.15.67:0/3735356115 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b70009690 con 0x7f9b64004800 2024-03-24T00:49:48.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.030+0000 7f9b83fc6640 1 -- 172.21.15.67:0/3735356115 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b64004800 msgr2=0x7f9b64004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:48.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.030+0000 7f9b83fc6640 1 --2- 172.21.15.67:0/3735356115 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b64004800 0x7f9b64004c80 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f9b70009ac0 tx=0x7f9b64005830 comp rx=0 tx=0).stop 2024-03-24T00:49:48.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.030+0000 7f9b82563640 1 -- 172.21.15.67:0/3735356115 reap_dead start 2024-03-24T00:49:48.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.030+0000 7f9b83fc6640 1 -- 172.21.15.67:0/3735356115 shutdown_connections 2024-03-24T00:49:48.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.030+0000 7f9b83fc6640 1 -- 172.21.15.67:0/3735356115 >> 172.21.15.67:0/3735356115 conn(0x7f9b7c06c880 msgr2=0x7f9b7c06cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:48.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.030+0000 7f9b83fc6640 1 -- 172.21.15.67:0/3735356115 shutdown_connections 2024-03-24T00:49:48.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.030+0000 7f9b83fc6640 1 -- 172.21.15.67:0/3735356115 wait complete. 2024-03-24T00:49:48.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.031+0000 7f9b83fc6640 1 Processor -- start 2024-03-24T00:49:48.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.031+0000 7f9b83fc6640 1 -- start start 2024-03-24T00:49:48.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.032+0000 7f9b83fc6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b64004800 0x7f9b7c1f3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:48.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.032+0000 7f9b83fc6640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9b7c0ce2e0 0x7f9b7c0ce760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:48.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.032+0000 7f9b83fc6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b7c0b7540 0x7f9b7c0b79c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:48.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.032+0000 7f9b83fc6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b7c0d0fd0 con 0x7f9b64004800 2024-03-24T00:49:48.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.032+0000 7f9b83fc6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b7c0b31f0 con 0x7f9b7c0b7540 2024-03-24T00:49:48.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.032+0000 7f9b83fc6640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b7c0d25e0 con 0x7f9b7c0ce2e0 2024-03-24T00:49:48.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.032+0000 7f9b81d62640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b64004800 0x7f9b7c1f3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:48.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.032+0000 7f9b81d62640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b64004800 0x7f9b7c1f3590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43812/0 (socket says 172.21.15.67:43812) 2024-03-24T00:49:48.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.032+0000 7f9b81d62640 1 -- 172.21.15.67:0/1070575001 learned_addr learned my addr 172.21.15.67:0/1070575001 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:48.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.032+0000 7f9b81561640 1 --2- 172.21.15.67:0/1070575001 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9b7c0ce2e0 0x7f9b7c0ce760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:48.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.032+0000 7f9b82563640 1 --2- 172.21.15.67:0/1070575001 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b7c0b7540 0x7f9b7c0b79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:48.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.032+0000 7f9b81d62640 1 -- 172.21.15.67:0/1070575001 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9b7c0ce2e0 msgr2=0x7f9b7c0ce760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:48.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.032+0000 7f9b81d62640 1 --2- 172.21.15.67:0/1070575001 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9b7c0ce2e0 0x7f9b7c0ce760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:48.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.032+0000 7f9b81d62640 1 -- 172.21.15.67:0/1070575001 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b7c0b7540 msgr2=0x7f9b7c0b79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:48.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.032+0000 7f9b81d62640 1 --2- 172.21.15.67:0/1070575001 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b7c0b7540 0x7f9b7c0b79c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:48.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.032+0000 7f9b81d62640 1 -- 172.21.15.67:0/1070575001 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b7c1f82f0 con 0x7f9b64004800 2024-03-24T00:49:48.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.032+0000 7f9b81561640 1 --2- 172.21.15.67:0/1070575001 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9b7c0ce2e0 0x7f9b7c0ce760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:49:48.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.032+0000 7f9b81d62640 1 --2- 172.21.15.67:0/1070575001 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b64004800 0x7f9b7c1f3590 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7f9b74007310 tx=0x7f9b740077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:48.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.033+0000 7f9b7a7fc640 1 -- 172.21.15.67:0/1070575001 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b7400f960 con 0x7f9b64004800 2024-03-24T00:49:48.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.033+0000 7f9b7a7fc640 1 -- 172.21.15.67:0/1070575001 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b74018070 con 0x7f9b64004800 2024-03-24T00:49:48.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.033+0000 7f9b83fc6640 1 -- 172.21.15.67:0/1070575001 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b7c1f7530 con 0x7f9b64004800 2024-03-24T00:49:48.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.033+0000 7f9b7a7fc640 1 -- 172.21.15.67:0/1070575001 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b7400fd80 con 0x7f9b64004800 2024-03-24T00:49:48.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.033+0000 7f9b83fc6640 1 -- 172.21.15.67:0/1070575001 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b7c1f7af0 con 0x7f9b64004800 2024-03-24T00:49:48.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.033+0000 7f9b83fc6640 1 -- 172.21.15.67:0/1070575001 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9b7c0b8270 con 0x7f9b64004800 2024-03-24T00:49:48.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.034+0000 7f9b7a7fc640 1 -- 172.21.15.67:0/1070575001 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9b7401d460 con 0x7f9b64004800 2024-03-24T00:49:48.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.034+0000 7f9b7a7fc640 1 --2- 172.21.15.67:0/1070575001 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9b6806bbd0 0x7f9b6806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:48.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.035+0000 7f9b7a7fc640 1 -- 172.21.15.67:0/1070575001 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f9b7408a9f0 con 0x7f9b64004800 2024-03-24T00:49:48.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.035+0000 7f9b81561640 1 --2- 172.21.15.67:0/1070575001 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9b6806bbd0 0x7f9b6806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:48.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.035+0000 7f9b7a7fc640 1 --2- 172.21.15.67:0/1070575001 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x7f9b680721b0 0x7f9b680745f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:48.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.035+0000 7f9b7a7fc640 1 -- 172.21.15.67:0/1070575001 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9b68074cc0 con 0x7f9b680721b0 2024-03-24T00:49:48.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.035+0000 7f9b7a7fc640 1 -- 172.21.15.67:0/1070575001 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=125) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9b7408f2c0 con 0x7f9b64004800 2024-03-24T00:49:48.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.035+0000 7f9b82563640 1 --2- 172.21.15.67:0/1070575001 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x7f9b680721b0 0x7f9b680745f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:48.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.035+0000 7f9b81561640 1 --2- 172.21.15.67:0/1070575001 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9b6806bbd0 0x7f9b6806e090 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f9b70036000 tx=0x7f9b70030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:48.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.035+0000 7f9b82563640 1 --2- 172.21.15.67:0/1070575001 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x7f9b680721b0 0x7f9b680745f0 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:48.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.036+0000 7f9b7a7fc640 1 -- 172.21.15.67:0/1070575001 <== osd.7 v2:172.21.15.72:6824/2893038746 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9b68074cc0 con 0x7f9b680721b0 2024-03-24T00:49:48.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.049+0000 7f9b83fc6640 1 -- 172.21.15.67:0/1070575001 --> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f9b7c04f400 con 0x7f9b680721b0 2024-03-24T00:49:48.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.050+0000 7f9b7a7fc640 1 -- 172.21.15.67:0/1070575001 <== osd.7 v2:172.21.15.72:6824/2893038746 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f9b7c04f400 con 0x7f9b680721b0 2024-03-24T00:49:48.050 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-24T00:49:48.050 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-24T00:49:48.050 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-24T00:49:48.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.050+0000 7f9b83fc6640 1 -- 172.21.15.67:0/1070575001 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x7f9b680721b0 msgr2=0x7f9b680745f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:48.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.050+0000 7f9b83fc6640 1 --2- 172.21.15.67:0/1070575001 >> [v2:172.21.15.72:6824/2893038746,v1:172.21.15.72:6825/2893038746] conn(0x7f9b680721b0 0x7f9b680745f0 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:48.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.050+0000 7f9b83fc6640 1 -- 172.21.15.67:0/1070575001 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9b6806bbd0 msgr2=0x7f9b6806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:48.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.050+0000 7f9b83fc6640 1 --2- 172.21.15.67:0/1070575001 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f9b6806bbd0 0x7f9b6806e090 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f9b70036000 tx=0x7f9b70030000 comp rx=0 tx=0).stop 2024-03-24T00:49:48.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.050+0000 7f9b83fc6640 1 -- 172.21.15.67:0/1070575001 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b64004800 msgr2=0x7f9b7c1f3590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:48.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.050+0000 7f9b83fc6640 1 --2- 172.21.15.67:0/1070575001 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b64004800 0x7f9b7c1f3590 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7f9b74007310 tx=0x7f9b740077d0 comp rx=0 tx=0).stop 2024-03-24T00:49:48.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.050+0000 7f9b82563640 1 -- 172.21.15.67:0/1070575001 reap_dead start 2024-03-24T00:49:48.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.050+0000 7f9b83fc6640 1 -- 172.21.15.67:0/1070575001 shutdown_connections 2024-03-24T00:49:48.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.050+0000 7f9b83fc6640 1 -- 172.21.15.67:0/1070575001 >> 172.21.15.67:0/1070575001 conn(0x7f9b7c06c880 msgr2=0x7f9b7c0c79d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:48.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.050+0000 7f9b83fc6640 1 -- 172.21.15.67:0/1070575001 shutdown_connections 2024-03-24T00:49:48.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.050+0000 7f9b83fc6640 1 -- 172.21.15.67:0/1070575001 wait complete. 2024-03-24T00:49:48.060 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-03-24T00:49:48.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.162+0000 7f856c713640 1 Processor -- start 2024-03-24T00:49:48.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.162+0000 7f856c713640 1 -- start start 2024-03-24T00:49:48.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.163+0000 7f856c713640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f85640b37a0 0x7f8564166d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:48.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.163+0000 7f856c713640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85640cd430 con 0x7f85640d07a0 2024-03-24T00:49:48.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.163+0000 7f856c713640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85640cd710 con 0x7f856415c010 2024-03-24T00:49:48.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.163+0000 7f856c713640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85640cfbe0 con 0x7f85640b37a0 2024-03-24T00:49:48.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.163+0000 7f8569cae640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f85640d07a0 0x7f85640cc770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56316/0 (socket says 172.21.15.67:56316) 2024-03-24T00:49:48.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.163+0000 7f8569cae640 1 -- 172.21.15.67:0/502729068 learned_addr learned my addr 172.21.15.67:0/502729068 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:49:48.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.163+0000 7f856acb0640 1 --2- 172.21.15.67:0/502729068 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f85640b37a0 0x7f8564166d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:48.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.164+0000 7f85694ad640 1 -- 172.21.15.67:0/502729068 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2439435380 0 0) 0x7f85640cd430 con 0x7f85640d07a0 2024-03-24T00:49:48.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.164+0000 7f85694ad640 1 -- 172.21.15.67:0/502729068 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8550003520 con 0x7f85640d07a0 2024-03-24T00:49:48.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.164+0000 7f85694ad640 1 -- 172.21.15.67:0/502729068 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3311574500 0 0) 0x7f8550003520 con 0x7f85640d07a0 2024-03-24T00:49:48.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.164+0000 7f85694ad640 1 -- 172.21.15.67:0/502729068 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f85640b37a0 msgr2=0x7f8564166d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:48.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.164+0000 7f85694ad640 1 --2- 172.21.15.67:0/502729068 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f85640b37a0 0x7f8564166d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:48.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.164+0000 7f85694ad640 1 -- 172.21.15.67:0/502729068 >> v1:172.21.15.72:6789/0 conn(0x7f856415c010 legacy=0x7f85640d0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:48.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.164+0000 7f85694ad640 1 -- 172.21.15.67:0/502729068 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85640cd9f0 con 0x7f85640d07a0 2024-03-24T00:49:48.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.164+0000 7f85694ad640 1 -- 172.21.15.67:0/502729068 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f8554001670 con 0x7f85640d07a0 2024-03-24T00:49:48.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.164+0000 7f85694ad640 1 -- 172.21.15.67:0/502729068 >> v1:172.21.15.67:6789/0 conn(0x7f85640d07a0 legacy=0x7f85640cc770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:48.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.164+0000 7f85694ad640 1 --2- 172.21.15.67:0/502729068 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f85640b37a0 0x7f85500039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:48.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.164+0000 7f85694ad640 1 --2- 172.21.15.67:0/502729068 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8550003f20 0x7f8550004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:48.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.164+0000 7f85694ad640 1 --2- 172.21.15.67:0/502729068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8550004d40 0x7f85500051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:48.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.164+0000 7f85694ad640 1 -- 172.21.15.67:0/502729068 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f855000b9f0 con 0x7f8550004d40 2024-03-24T00:49:48.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.164+0000 7f85694ad640 1 -- 172.21.15.67:0/502729068 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f855000bba0 con 0x7f8550003f20 2024-03-24T00:49:48.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.164+0000 7f85694ad640 1 -- 172.21.15.67:0/502729068 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f855000bda0 con 0x7f85640b37a0 2024-03-24T00:49:48.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.164+0000 7f856acb0640 1 --2- 172.21.15.67:0/502729068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8550004d40 0x7f85500051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:48.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.164+0000 7f8569cae640 1 --2- 172.21.15.67:0/502729068 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8550003f20 0x7f8550004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:48.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.164+0000 7f856a4af640 1 --2- 172.21.15.67:0/502729068 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f85640b37a0 0x7f85500039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:48.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.164+0000 7f856acb0640 1 -- 172.21.15.67:0/502729068 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f85640b37a0 msgr2=0x7f85500039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:48.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.164+0000 7f856acb0640 1 --2- 172.21.15.67:0/502729068 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f85640b37a0 0x7f85500039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:48.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.164+0000 7f856acb0640 1 -- 172.21.15.67:0/502729068 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8550003f20 msgr2=0x7f8550004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:48.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.164+0000 7f856acb0640 1 --2- 172.21.15.67:0/502729068 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8550003f20 0x7f8550004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:48.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.164+0000 7f856acb0640 1 -- 172.21.15.67:0/502729068 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8550003520 con 0x7f8550004d40 2024-03-24T00:49:48.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.165+0000 7f856acb0640 1 --2- 172.21.15.67:0/502729068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8550004d40 0x7f85500051c0 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7f855c009f90 tx=0x7f8550005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=451c191edafb2bcd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:48.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.165+0000 7f85694ad640 1 -- 172.21.15.67:0/502729068 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f855c039070 con 0x7f8550004d40 2024-03-24T00:49:48.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.165+0000 7f85694ad640 1 -- 172.21.15.67:0/502729068 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f855c02cbd0 con 0x7f8550004d40 2024-03-24T00:49:48.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.165+0000 7f85694ad640 1 -- 172.21.15.67:0/502729068 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f855c009c10 con 0x7f8550004d40 2024-03-24T00:49:48.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.166+0000 7f856c713640 1 -- 172.21.15.67:0/502729068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8550004d40 msgr2=0x7f85500051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:48.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.166+0000 7f856c713640 1 --2- 172.21.15.67:0/502729068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8550004d40 0x7f85500051c0 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7f855c009f90 tx=0x7f8550005830 comp rx=0 tx=0).stop 2024-03-24T00:49:48.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.166+0000 7f856acb0640 1 -- 172.21.15.67:0/502729068 reap_dead start 2024-03-24T00:49:48.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.166+0000 7f856c713640 1 -- 172.21.15.67:0/502729068 shutdown_connections 2024-03-24T00:49:48.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.166+0000 7f856c713640 1 -- 172.21.15.67:0/502729068 >> 172.21.15.67:0/502729068 conn(0x7f856409aea0 msgr2=0x7f856408b180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:48.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.166+0000 7f856c713640 1 -- 172.21.15.67:0/502729068 shutdown_connections 2024-03-24T00:49:48.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.166+0000 7f856c713640 1 -- 172.21.15.67:0/502729068 wait complete. 2024-03-24T00:49:48.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.167+0000 7f856c713640 1 Processor -- start 2024-03-24T00:49:48.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.167+0000 7f856c713640 1 -- start start 2024-03-24T00:49:48.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.168+0000 7f856c713640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8550004d40 0x7f8564156fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:48.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.168+0000 7f856c713640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85640c8370 0x7f85640c8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:48.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.168+0000 7f856c713640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85640af8a0 0x7f85640afd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:48.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.168+0000 7f856c713640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85640cd210 con 0x7f85640c8370 2024-03-24T00:49:48.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.168+0000 7f856c713640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85640cd090 con 0x7f85640af8a0 2024-03-24T00:49:48.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.168+0000 7f856c713640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85640b2020 con 0x7f8550004d40 2024-03-24T00:49:48.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.168+0000 7f8569cae640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85640c8370 0x7f85640c8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:48.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.168+0000 7f8569cae640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85640c8370 0x7f85640c8b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43830/0 (socket says 172.21.15.67:43830) 2024-03-24T00:49:48.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.168+0000 7f8569cae640 1 -- 172.21.15.67:0/2719601738 learned_addr learned my addr 172.21.15.67:0/2719601738 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:48.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.168+0000 7f856acb0640 1 --2- 172.21.15.67:0/2719601738 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85640af8a0 0x7f85640afd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:48.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.168+0000 7f856a4af640 1 --2- 172.21.15.67:0/2719601738 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8550004d40 0x7f8564156fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:48.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.168+0000 7f8569cae640 1 -- 172.21.15.67:0/2719601738 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8550004d40 msgr2=0x7f8564156fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:48.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.168+0000 7f8569cae640 1 --2- 172.21.15.67:0/2719601738 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8550004d40 0x7f8564156fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:48.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.168+0000 7f8569cae640 1 -- 172.21.15.67:0/2719601738 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85640af8a0 msgr2=0x7f85640afd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:48.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.168+0000 7f8569cae640 1 --2- 172.21.15.67:0/2719601738 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85640af8a0 0x7f85640afd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:48.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.168+0000 7f8569cae640 1 -- 172.21.15.67:0/2719601738 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85640c94f0 con 0x7f85640c8370 2024-03-24T00:49:48.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.168+0000 7f8569cae640 1 --2- 172.21.15.67:0/2719601738 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85640c8370 0x7f85640c8b80 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f8554000f60 tx=0x7f8554007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:48.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.168+0000 7f855affd640 1 -- 172.21.15.67:0/2719601738 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f855400f9c0 con 0x7f85640c8370 2024-03-24T00:49:48.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.168+0000 7f855affd640 1 -- 172.21.15.67:0/2719601738 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8554018070 con 0x7f85640c8370 2024-03-24T00:49:48.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.168+0000 7f856c713640 1 -- 172.21.15.67:0/2719601738 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85640c97d0 con 0x7f85640c8370 2024-03-24T00:49:48.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.168+0000 7f855affd640 1 -- 172.21.15.67:0/2719601738 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f855400fde0 con 0x7f85640c8370 2024-03-24T00:49:48.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.169+0000 7f856c713640 1 -- 172.21.15.67:0/2719601738 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85640ca000 con 0x7f85640c8370 2024-03-24T00:49:48.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.169+0000 7f856c713640 1 -- 172.21.15.67:0/2719601738 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f856406e4a0 con 0x7f85640c8370 2024-03-24T00:49:48.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.170+0000 7f855affd640 1 -- 172.21.15.67:0/2719601738 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f855401d460 con 0x7f85640c8370 2024-03-24T00:49:48.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.170+0000 7f855affd640 1 --2- 172.21.15.67:0/2719601738 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f854806bc00 0x7f854806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:48.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.170+0000 7f855affd640 1 -- 172.21.15.67:0/2719601738 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f855408ab20 con 0x7f85640c8370 2024-03-24T00:49:48.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.170+0000 7f856a4af640 1 --2- 172.21.15.67:0/2719601738 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f854806bc00 0x7f854806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:48.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.171+0000 7f856a4af640 1 --2- 172.21.15.67:0/2719601738 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f854806bc00 0x7f854806e0c0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f8560003700 tx=0x7f8560009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:48.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.175+0000 7f855affd640 1 -- 172.21.15.67:0/2719601738 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f85540596d0 con 0x7f85640c8370 2024-03-24T00:49:48.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.357+0000 7f856c713640 1 -- 172.21.15.67:0/2719601738 --> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f85640cfe70 con 0x7f854806bc00 2024-03-24T00:49:48.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.357+0000 7f855affd640 1 -- 172.21.15.67:0/2719601738 <== mgr.4105 v2:172.21.15.72:6832/2768292274 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f85640cfe70 con 0x7f854806bc00 2024-03-24T00:49:48.357 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 7 to deep-scrub 2024-03-24T00:49:48.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.359+0000 7f856c713640 1 -- 172.21.15.67:0/2719601738 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f854806bc00 msgr2=0x7f854806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:48.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.359+0000 7f856c713640 1 --2- 172.21.15.67:0/2719601738 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f854806bc00 0x7f854806e0c0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f8560003700 tx=0x7f8560009040 comp rx=0 tx=0).stop 2024-03-24T00:49:48.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.359+0000 7f856c713640 1 -- 172.21.15.67:0/2719601738 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85640c8370 msgr2=0x7f85640c8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:48.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.359+0000 7f856c713640 1 --2- 172.21.15.67:0/2719601738 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85640c8370 0x7f85640c8b80 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f8554000f60 tx=0x7f8554007720 comp rx=0 tx=0).stop 2024-03-24T00:49:48.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.359+0000 7f856c713640 1 -- 172.21.15.67:0/2719601738 shutdown_connections 2024-03-24T00:49:48.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.359+0000 7f856c713640 1 --2- 172.21.15.67:0/2719601738 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f854806bc00 0x7f854806e0c0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:48.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.359+0000 7f856c713640 1 --2- 172.21.15.67:0/2719601738 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85640af8a0 0x7f85640afd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:48.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.359+0000 7f856c713640 1 --2- 172.21.15.67:0/2719601738 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85640c8370 0x7f85640c8b80 unknown :-1 s=CLOSED pgs=1969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:48.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.359+0000 7f856c713640 1 --2- 172.21.15.67:0/2719601738 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8550004d40 0x7f8564156fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:48.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.359+0000 7f856c713640 1 -- 172.21.15.67:0/2719601738 >> 172.21.15.67:0/2719601738 conn(0x7f856409aea0 msgr2=0x7f85640d0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:48.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.359+0000 7f856c713640 1 -- 172.21.15.67:0/2719601738 shutdown_connections 2024-03-24T00:49:48.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.359+0000 7f856c713640 1 -- 172.21.15.67:0/2719601738 wait complete. 2024-03-24T00:49:48.370 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:49:48.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.471+0000 7fb483a6e640 1 Processor -- start 2024-03-24T00:49:48.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.471+0000 7fb483a6e640 1 -- start start 2024-03-24T00:49:48.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.472+0000 7fb483a6e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb47c15dfb0 0x7fb47c0cfe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:48.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.472+0000 7fb483a6e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb47c06b750 0x7fb47c0cb120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:48.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.472+0000 7fb483a6e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb47c0d07d0 con 0x7fb47c15e390 2024-03-24T00:49:48.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.472+0000 7fb483a6e640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb47c0d0ab0 con 0x7fb47c06b750 2024-03-24T00:49:48.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.472+0000 7fb483a6e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb47c0d0d90 con 0x7fb47c15dfb0 2024-03-24T00:49:48.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.472+0000 7fb48200b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb47c06b750 0x7fb47c0cb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:48.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.472+0000 7fb48200b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb47c06b750 0x7fb47c0cb120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43832/0 (socket says 172.21.15.67:43832) 2024-03-24T00:49:48.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.472+0000 7fb48200b640 1 -- 172.21.15.67:0/688062327 learned_addr learned my addr 172.21.15.67:0/688062327 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:48.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.472+0000 7fb48180a640 1 --2- 172.21.15.67:0/688062327 >> v2:172.21.15.72:3300/0 conn(0x7fb47c15dfb0 0x7fb47c0cfe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:48.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.472+0000 7fb48200b640 1 -- 172.21.15.67:0/688062327 >> v2:172.21.15.72:3300/0 conn(0x7fb47c15dfb0 msgr2=0x7fb47c0cfe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:49:48.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.473+0000 7fb48200b640 1 --2- 172.21.15.67:0/688062327 >> v2:172.21.15.72:3300/0 conn(0x7fb47c15dfb0 0x7fb47c0cfe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:48.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.473+0000 7fb48200b640 1 -- 172.21.15.67:0/688062327 >> v1:172.21.15.72:6789/0 conn(0x7fb47c15e390 legacy=0x7fb47c06b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:48.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.473+0000 7fb48200b640 1 -- 172.21.15.67:0/688062327 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb47c0d1070 con 0x7fb47c06b750 2024-03-24T00:49:48.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.473+0000 7fb48180a640 1 --2- 172.21.15.67:0/688062327 >> v2:172.21.15.72:3300/0 conn(0x7fb47c15dfb0 0x7fb47c0cfe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:49:48.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.473+0000 7fb48200b640 1 --2- 172.21.15.67:0/688062327 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb47c06b750 0x7fb47c0cb120 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7fb47c0d1b70 tx=0x7fb47802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d96081a9b842e89b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:48.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.473+0000 7fb480808640 1 -- 172.21.15.67:0/688062327 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb47802cb20 con 0x7fb47c06b750 2024-03-24T00:49:48.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.474+0000 7fb480808640 1 -- 172.21.15.67:0/688062327 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb47802ccc0 con 0x7fb47c06b750 2024-03-24T00:49:48.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.474+0000 7fb480808640 1 -- 172.21.15.67:0/688062327 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb478034610 con 0x7fb47c06b750 2024-03-24T00:49:48.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.474+0000 7fb483a6e640 1 -- 172.21.15.67:0/688062327 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb47c06b750 msgr2=0x7fb47c0cb120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:48.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.474+0000 7fb483a6e640 1 --2- 172.21.15.67:0/688062327 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb47c06b750 0x7fb47c0cb120 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7fb47c0d1b70 tx=0x7fb47802bf80 comp rx=0 tx=0).stop 2024-03-24T00:49:48.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.474+0000 7fb483a6e640 1 -- 172.21.15.67:0/688062327 shutdown_connections 2024-03-24T00:49:48.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.474+0000 7fb483a6e640 1 --2- 172.21.15.67:0/688062327 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb47c06b750 0x7fb47c0cb120 unknown :-1 s=CLOSED pgs=1970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:48.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.474+0000 7fb483a6e640 1 --2- 172.21.15.67:0/688062327 >> v2:172.21.15.72:3300/0 conn(0x7fb47c15dfb0 0x7fb47c0cfe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:48.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.474+0000 7fb483a6e640 1 -- 172.21.15.67:0/688062327 >> 172.21.15.67:0/688062327 conn(0x7fb47c099a30 msgr2=0x7fb47c099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:48.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.474+0000 7fb483a6e640 1 -- 172.21.15.67:0/688062327 shutdown_connections 2024-03-24T00:49:48.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.474+0000 7fb483a6e640 1 -- 172.21.15.67:0/688062327 wait complete. 2024-03-24T00:49:48.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.475+0000 7fb483a6e640 1 Processor -- start 2024-03-24T00:49:48.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.475+0000 7fb483a6e640 1 -- start start 2024-03-24T00:49:48.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.476+0000 7fb483a6e640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb47c06b750 0x7fb47c0c7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:48.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.476+0000 7fb483a6e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb47c15dfb0 0x7fb47c146980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:48.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.476+0000 7fb483a6e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb47c15e390 0x7fb47c1589c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:48.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.476+0000 7fb483a6e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c0d03c0 con 0x7fb47c15dfb0 2024-03-24T00:49:48.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.476+0000 7fb483a6e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c0d1870 con 0x7fb47c15e390 2024-03-24T00:49:48.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.476+0000 7fb483a6e640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c0d0540 con 0x7fb47c06b750 2024-03-24T00:49:48.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.476+0000 7fb481009640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb47c15dfb0 0x7fb47c146980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:48.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.476+0000 7fb481009640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb47c15dfb0 0x7fb47c146980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43844/0 (socket says 172.21.15.67:43844) 2024-03-24T00:49:48.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.476+0000 7fb481009640 1 -- 172.21.15.67:0/3183969661 learned_addr learned my addr 172.21.15.67:0/3183969661 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:49:48.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.476+0000 7fb48180a640 1 --2- 172.21.15.67:0/3183969661 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb47c06b750 0x7fb47c0c7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:48.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.476+0000 7fb48200b640 1 --2- 172.21.15.67:0/3183969661 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb47c15e390 0x7fb47c1589c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:48.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.476+0000 7fb481009640 1 -- 172.21.15.67:0/3183969661 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb47c06b750 msgr2=0x7fb47c0c7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:48.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.476+0000 7fb481009640 1 --2- 172.21.15.67:0/3183969661 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb47c06b750 0x7fb47c0c7ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:48.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.476+0000 7fb481009640 1 -- 172.21.15.67:0/3183969661 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb47c15e390 msgr2=0x7fb47c1589c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:48.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.476+0000 7fb481009640 1 --2- 172.21.15.67:0/3183969661 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb47c15e390 0x7fb47c1589c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:48.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.476+0000 7fb481009640 1 -- 172.21.15.67:0/3183969661 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb47c0c80e0 con 0x7fb47c15dfb0 2024-03-24T00:49:48.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.477+0000 7fb481009640 1 --2- 172.21.15.67:0/3183969661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb47c15dfb0 0x7fb47c146980 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7fb46c00c400 tx=0x7fb46c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:48.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.477+0000 7fb4727fc640 1 -- 172.21.15.67:0/3183969661 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46c0052d0 con 0x7fb47c15dfb0 2024-03-24T00:49:48.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.477+0000 7fb4727fc640 1 -- 172.21.15.67:0/3183969661 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb46c005490 con 0x7fb47c15dfb0 2024-03-24T00:49:48.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.477+0000 7fb483a6e640 1 -- 172.21.15.67:0/3183969661 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb47c144210 con 0x7fb47c15dfb0 2024-03-24T00:49:48.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.477+0000 7fb4727fc640 1 -- 172.21.15.67:0/3183969661 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46c007260 con 0x7fb47c15dfb0 2024-03-24T00:49:48.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.477+0000 7fb483a6e640 1 -- 172.21.15.67:0/3183969661 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb47c1447d0 con 0x7fb47c15dfb0 2024-03-24T00:49:48.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.478+0000 7fb483a6e640 1 -- 172.21.15.67:0/3183969661 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb47c04f400 con 0x7fb47c15dfb0 2024-03-24T00:49:48.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.478+0000 7fb4727fc640 1 -- 172.21.15.67:0/3183969661 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb46c007400 con 0x7fb47c15dfb0 2024-03-24T00:49:48.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.478+0000 7fb4727fc640 1 --2- 172.21.15.67:0/3183969661 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb46006bbd0 0x7fb46006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:48.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.479+0000 7fb4727fc640 1 -- 172.21.15.67:0/3183969661 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5507+0+0 (secure 0 0 0) 0x7fb46c089ef0 con 0x7fb47c15dfb0 2024-03-24T00:49:48.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.479+0000 7fb48180a640 1 --2- 172.21.15.67:0/3183969661 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb46006bbd0 0x7fb46006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:49:48.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.479+0000 7fb48180a640 1 --2- 172.21.15.67:0/3183969661 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb46006bbd0 0x7fb46006e090 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fb474006fd0 tx=0x7fb474008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:49:48.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.483+0000 7fb4727fc640 1 -- 172.21.15.67:0/3183969661 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb46c058aa0 con 0x7fb47c15dfb0 2024-03-24T00:49:48.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.661+0000 7fb483a6e640 1 -- 172.21.15.67:0/3183969661 --> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb47c0b6a50 con 0x7fb46006bbd0 2024-03-24T00:49:48.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.662+0000 7fb4727fc640 1 -- 172.21.15.67:0/3183969661 <== mgr.4105 v2:172.21.15.72:6832/2768292274 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13500 (secure 0 0 0) 0x7fb47c0b6a50 con 0x7fb46006bbd0 2024-03-24T00:49:48.662 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:49:48.662 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-24T00:49:48.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.664+0000 7fb483a6e640 1 -- 172.21.15.67:0/3183969661 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb46006bbd0 msgr2=0x7fb46006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:48.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.664+0000 7fb483a6e640 1 --2- 172.21.15.67:0/3183969661 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb46006bbd0 0x7fb46006e090 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fb474006fd0 tx=0x7fb474008040 comp rx=0 tx=0).stop 2024-03-24T00:49:48.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.664+0000 7fb483a6e640 1 -- 172.21.15.67:0/3183969661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb47c15dfb0 msgr2=0x7fb47c146980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:49:48.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.664+0000 7fb483a6e640 1 --2- 172.21.15.67:0/3183969661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb47c15dfb0 0x7fb47c146980 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7fb46c00c400 tx=0x7fb46c00c8c0 comp rx=0 tx=0).stop 2024-03-24T00:49:48.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.664+0000 7fb483a6e640 1 -- 172.21.15.67:0/3183969661 shutdown_connections 2024-03-24T00:49:48.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.664+0000 7fb483a6e640 1 --2- 172.21.15.67:0/3183969661 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fb46006bbd0 0x7fb46006e090 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:48.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.664+0000 7fb483a6e640 1 --2- 172.21.15.67:0/3183969661 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb47c15e390 0x7fb47c1589c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:48.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.664+0000 7fb483a6e640 1 --2- 172.21.15.67:0/3183969661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb47c15dfb0 0x7fb47c146980 unknown :-1 s=CLOSED pgs=1971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:48.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.664+0000 7fb483a6e640 1 --2- 172.21.15.67:0/3183969661 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb47c06b750 0x7fb47c0c7ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:48.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.664+0000 7fb483a6e640 1 -- 172.21.15.67:0/3183969661 >> 172.21.15.67:0/3183969661 conn(0x7fb47c099a30 msgr2=0x7fb47c06b040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:49:48.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.664+0000 7fb483a6e640 1 -- 172.21.15.67:0/3183969661 shutdown_connections 2024-03-24T00:49:48.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:49:48.664+0000 7fb483a6e640 1 -- 172.21.15.67:0/3183969661 wait complete. 2024-03-24T00:49:48.674 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":452,"stamp":"2024-03-24T00:49:47.481789+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":33,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":262836,"kb_used_data":13384,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754711884,"statfs":{"total":773094113280,"available":772824969216,"internally_reserved":0,"allocated":13705216,"data_stored":7035936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8390540,"num_objects":-30,"num_object_clones":0,"num_object_copies":-60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-30,"num_whiteouts":0,"num_read":-258,"num_read_kb":-268,"num_write":-177,"num_write_kb":-8305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-17,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001877"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":368,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-24T00:49:41.242649+0000","last_change":"2024-03-24T00:38:40.953871+0000","last_active":"2024-03-24T00:49:41.242649+0000","last_peered":"2024-03-24T00:49:41.242649+0000","last_clean":"2024-03-24T00:49:41.242649+0000","last_became_active":"2024-03-24T00:38:40.952926+0000","last_became_peered":"2024-03-24T00:38:40.952926+0000","last_unstale":"2024-03-24T00:49:41.242649+0000","last_undegraded":"2024-03-24T00:49:41.242649+0000","last_fullsized":"2024-03-24T00:49:41.242649+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:39.722194+0000","last_clean_scrub_stamp":"2024-03-24T00:38:39.722194+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:05.397257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574986,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33288,"kb_used_data":2108,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338552,"statfs":{"total":96636764160,"available":96602677248,"internally_reserved":0,"allocated":2158592,"data_stored":1322268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574985,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32708,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339132,"statfs":{"total":96636764160,"available":96603271168,"internally_reserved":0,"allocated":1564672,"data_stored":731900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574985,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32712,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339128,"statfs":{"total":96636764160,"available":96603267072,"internally_reserved":0,"allocated":1564672,"data_stored":731900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574984,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32712,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339128,"statfs":{"total":96636764160,"available":96603267072,"internally_reserved":0,"allocated":1564672,"data_stored":731900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574984,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32708,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339132,"statfs":{"total":96636764160,"available":96603271168,"internally_reserved":0,"allocated":1564672,"data_stored":731900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574985,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32708,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339132,"statfs":{"total":96636764160,"available":96603271168,"internally_reserved":0,"allocated":1564672,"data_stored":731900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574985,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32708,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339132,"statfs":{"total":96636764160,"available":96603271168,"internally_reserved":0,"allocated":1564672,"data_stored":731900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574984,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33292,"kb_used_data":2108,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338548,"statfs":{"total":96636764160,"available":96602673152,"internally_reserved":0,"allocated":2158592,"data_stored":1322268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:49:48.677 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-03-24T00:38:39.722194+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=0, tm_min=38, tm_sec=39, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=0, tm_min=49, tm_sec=43, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T00:49:48.677 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-24T00:50:08.678 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:50:08.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.794+0000 7f56a099d640 1 Processor -- start 2024-03-24T00:50:08.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.794+0000 7f56a099d640 1 -- start start 2024-03-24T00:50:08.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.795+0000 7f56a099d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f569c0b49c0 0x7f569c15b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:08.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.795+0000 7f56a099d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f569c0b4da0 con 0x7f569c072a70 2024-03-24T00:50:08.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.795+0000 7f56a099d640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f569c06db30 con 0x7f569c06f100 2024-03-24T00:50:08.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.795+0000 7f56a099d640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f569c06de10 con 0x7f569c0b49c0 2024-03-24T00:50:08.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.795+0000 7f569ad9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f569c072a70 0x7f569c0b2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55932/0 (socket says 172.21.15.67:55932) 2024-03-24T00:50:08.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.795+0000 7f569ad9d640 1 -- 172.21.15.67:0/3583605924 learned_addr learned my addr 172.21.15.67:0/3583605924 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:50:08.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.795+0000 7f569959a640 1 -- 172.21.15.67:0/3583605924 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1995033248 0 0) 0x7f569c0b4da0 con 0x7f569c072a70 2024-03-24T00:50:08.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.795+0000 7f569959a640 1 -- 172.21.15.67:0/3583605924 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5688003440 con 0x7f569c072a70 2024-03-24T00:50:08.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.795+0000 7f569a59c640 1 --2- 172.21.15.67:0/3583605924 >> v2:172.21.15.72:3300/0 conn(0x7f569c0b49c0 0x7f569c15b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:08.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.795+0000 7f569959a640 1 -- 172.21.15.67:0/3583605924 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 558950760 0 0) 0x7f5688003440 con 0x7f569c072a70 2024-03-24T00:50:08.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.795+0000 7f569959a640 1 -- 172.21.15.67:0/3583605924 >> v2:172.21.15.72:3300/0 conn(0x7f569c0b49c0 msgr2=0x7f569c15b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:50:08.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.795+0000 7f569959a640 1 --2- 172.21.15.67:0/3583605924 >> v2:172.21.15.72:3300/0 conn(0x7f569c0b49c0 0x7f569c15b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:08.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.795+0000 7f569959a640 1 -- 172.21.15.67:0/3583605924 >> v1:172.21.15.72:6789/0 conn(0x7f569c06f100 legacy=0x7f569c072360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:08.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.795+0000 7f569959a640 1 -- 172.21.15.67:0/3583605924 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f569c06e0f0 con 0x7f569c072a70 2024-03-24T00:50:08.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.795+0000 7f569959a640 1 -- 172.21.15.67:0/3583605924 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f568c0019f0 con 0x7f569c072a70 2024-03-24T00:50:08.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.795+0000 7f569959a640 1 -- 172.21.15.67:0/3583605924 >> v1:172.21.15.67:6789/0 conn(0x7f569c072a70 legacy=0x7f569c0b2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:08.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.795+0000 7f569959a640 1 --2- 172.21.15.67:0/3583605924 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5688003aa0 0x7f5688003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:08.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.795+0000 7f569959a640 1 --2- 172.21.15.67:0/3583605924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5688008850 0x7f568800acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:08.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.795+0000 7f569959a640 1 --2- 172.21.15.67:0/3583605924 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f568800b560 0x7f568800da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:08.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.796+0000 7f569959a640 1 -- 172.21.15.67:0/3583605924 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f569c06de10 con 0x7f5688008850 2024-03-24T00:50:08.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.796+0000 7f569959a640 1 -- 172.21.15.67:0/3583605924 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f568800e300 con 0x7f5688003aa0 2024-03-24T00:50:08.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.796+0000 7f569959a640 1 -- 172.21.15.67:0/3583605924 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f568800e500 con 0x7f568800b560 2024-03-24T00:50:08.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.796+0000 7f5699d9b640 1 --2- 172.21.15.67:0/3583605924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5688008850 0x7f568800acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:08.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.796+0000 7f569a59c640 1 --2- 172.21.15.67:0/3583605924 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5688003aa0 0x7f5688003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:08.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.796+0000 7f569ad9d640 1 --2- 172.21.15.67:0/3583605924 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f568800b560 0x7f568800da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:08.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.796+0000 7f5699d9b640 1 -- 172.21.15.67:0/3583605924 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f568800b560 msgr2=0x7f568800da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:50:08.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.796+0000 7f5699d9b640 1 --2- 172.21.15.67:0/3583605924 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f568800b560 0x7f568800da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:08.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.796+0000 7f5699d9b640 1 -- 172.21.15.67:0/3583605924 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5688003aa0 msgr2=0x7f5688003ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:50:08.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.796+0000 7f5699d9b640 1 --2- 172.21.15.67:0/3583605924 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5688003aa0 0x7f5688003ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:08.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.796+0000 7f569ad9d640 1 -- 172.21.15.67:0/3583605924 reap_dead start 2024-03-24T00:50:08.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.796+0000 7f5699d9b640 1 -- 172.21.15.67:0/3583605924 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5688003440 con 0x7f5688008850 2024-03-24T00:50:08.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.797+0000 7f5699d9b640 1 --2- 172.21.15.67:0/3583605924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5688008850 0x7f568800acc0 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f5684001750 tx=0x7f5688003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61e9173914448e27 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:08.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.797+0000 7f569959a640 1 -- 172.21.15.67:0/3583605924 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5684001370 con 0x7f5688008850 2024-03-24T00:50:08.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.797+0000 7f569959a640 1 -- 172.21.15.67:0/3583605924 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56840052d0 con 0x7f5688008850 2024-03-24T00:50:08.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.797+0000 7f569959a640 1 -- 172.21.15.67:0/3583605924 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5684003830 con 0x7f5688008850 2024-03-24T00:50:08.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.797+0000 7f56a099d640 1 -- 172.21.15.67:0/3583605924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5688008850 msgr2=0x7f568800acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:08.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.797+0000 7f56a099d640 1 --2- 172.21.15.67:0/3583605924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5688008850 0x7f568800acc0 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f5684001750 tx=0x7f5688003730 comp rx=0 tx=0).stop 2024-03-24T00:50:08.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.797+0000 7f56a099d640 1 -- 172.21.15.67:0/3583605924 shutdown_connections 2024-03-24T00:50:08.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.797+0000 7f56a099d640 1 --2- 172.21.15.67:0/3583605924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5688008850 0x7f568800acc0 unknown :-1 s=CLOSED pgs=1973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:08.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.797+0000 7f56a099d640 1 -- 172.21.15.67:0/3583605924 >> 172.21.15.67:0/3583605924 conn(0x7f569c0cf9e0 msgr2=0x7f569c0cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:08.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.798+0000 7f56a099d640 1 -- 172.21.15.67:0/3583605924 shutdown_connections 2024-03-24T00:50:08.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.798+0000 7f56a099d640 1 -- 172.21.15.67:0/3583605924 wait complete. 2024-03-24T00:50:08.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.799+0000 7f56a099d640 1 Processor -- start 2024-03-24T00:50:08.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.799+0000 7f56a099d640 1 -- start start 2024-03-24T00:50:08.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.800+0000 7f56a099d640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5688008850 0x7f569c150330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:08.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.800+0000 7f56a099d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f569c0c64a0 0x7f569c0c4980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:08.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.800+0000 7f56a099d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f569c0c4ec0 0x7f569c15cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:08.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.800+0000 7f56a099d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f569c06f590 con 0x7f569c0c64a0 2024-03-24T00:50:08.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.800+0000 7f56a099d640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f569c06e6b0 con 0x7f569c0c4ec0 2024-03-24T00:50:08.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.800+0000 7f56a099d640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f569c0b3d30 con 0x7f5688008850 2024-03-24T00:50:08.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.800+0000 7f5699d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f569c0c64a0 0x7f569c0c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:08.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.800+0000 7f5699d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f569c0c64a0 0x7f569c0c4980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56144/0 (socket says 172.21.15.67:56144) 2024-03-24T00:50:08.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.800+0000 7f5699d9b640 1 -- 172.21.15.67:0/3295984754 learned_addr learned my addr 172.21.15.67:0/3295984754 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:50:08.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.800+0000 7f569a59c640 1 --2- 172.21.15.67:0/3295984754 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5688008850 0x7f569c150330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:08.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.800+0000 7f569ad9d640 1 --2- 172.21.15.67:0/3295984754 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f569c0c4ec0 0x7f569c15cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:08.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.800+0000 7f5699d9b640 1 -- 172.21.15.67:0/3295984754 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5688008850 msgr2=0x7f569c150330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:08.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.800+0000 7f5699d9b640 1 --2- 172.21.15.67:0/3295984754 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5688008850 0x7f569c150330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:08.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.800+0000 7f5699d9b640 1 -- 172.21.15.67:0/3295984754 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f569c0c4ec0 msgr2=0x7f569c15cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:08.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.800+0000 7f5699d9b640 1 --2- 172.21.15.67:0/3295984754 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f569c0c4ec0 0x7f569c15cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:08.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.800+0000 7f5699d9b640 1 -- 172.21.15.67:0/3295984754 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f569c150870 con 0x7f569c0c64a0 2024-03-24T00:50:08.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.800+0000 7f5699d9b640 1 --2- 172.21.15.67:0/3295984754 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f569c0c64a0 0x7f569c0c4980 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f5684005040 tx=0x7f5684001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:08.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.800+0000 7f5692ffd640 1 -- 172.21.15.67:0/3295984754 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5684041070 con 0x7f569c0c64a0 2024-03-24T00:50:08.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.800+0000 7f5692ffd640 1 -- 172.21.15.67:0/3295984754 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5684005480 con 0x7f569c0c64a0 2024-03-24T00:50:08.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.800+0000 7f56a099d640 1 -- 172.21.15.67:0/3295984754 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f569c0c5400 con 0x7f569c0c64a0 2024-03-24T00:50:08.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.801+0000 7f5692ffd640 1 -- 172.21.15.67:0/3295984754 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5684001370 con 0x7f569c0c64a0 2024-03-24T00:50:08.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.801+0000 7f56a099d640 1 -- 172.21.15.67:0/3295984754 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f569c0c59c0 con 0x7f569c0c64a0 2024-03-24T00:50:08.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.802+0000 7f56a099d640 1 -- 172.21.15.67:0/3295984754 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f569c04f400 con 0x7f569c0c64a0 2024-03-24T00:50:08.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.802+0000 7f5692ffd640 1 -- 172.21.15.67:0/3295984754 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5684044070 con 0x7f569c0c64a0 2024-03-24T00:50:08.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.802+0000 7f5692ffd640 1 --2- 172.21.15.67:0/3295984754 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f567c06bc00 0x7f567c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:08.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.803+0000 7f5692ffd640 1 -- 172.21.15.67:0/3295984754 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f56840ac490 con 0x7f569c0c64a0 2024-03-24T00:50:08.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.803+0000 7f569a59c640 1 --2- 172.21.15.67:0/3295984754 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f567c06bc00 0x7f567c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:08.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.803+0000 7f569a59c640 1 --2- 172.21.15.67:0/3295984754 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f567c06bc00 0x7f567c06e0c0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f569c050be0 tx=0x7f5694008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:08.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.807+0000 7f5692ffd640 1 -- 172.21.15.67:0/3295984754 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f568407afb0 con 0x7f569c0c64a0 2024-03-24T00:50:08.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.984+0000 7f56a099d640 1 -- 172.21.15.67:0/3295984754 --> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f569c0b4190 con 0x7f567c06bc00 2024-03-24T00:50:08.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.985+0000 7f5692ffd640 1 -- 172.21.15.67:0/3295984754 <== mgr.4105 v2:172.21.15.72:6832/2768292274 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13379 (secure 0 0 0) 0x7f569c0b4190 con 0x7f567c06bc00 2024-03-24T00:50:08.986 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:50:08.987 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-24T00:50:08.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.987+0000 7f56a099d640 1 -- 172.21.15.67:0/3295984754 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f567c06bc00 msgr2=0x7f567c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:08.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.987+0000 7f56a099d640 1 --2- 172.21.15.67:0/3295984754 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f567c06bc00 0x7f567c06e0c0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f569c050be0 tx=0x7f5694008040 comp rx=0 tx=0).stop 2024-03-24T00:50:08.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.987+0000 7f56a099d640 1 -- 172.21.15.67:0/3295984754 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f569c0c64a0 msgr2=0x7f569c0c4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:08.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.987+0000 7f56a099d640 1 --2- 172.21.15.67:0/3295984754 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f569c0c64a0 0x7f569c0c4980 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f5684005040 tx=0x7f5684001120 comp rx=0 tx=0).stop 2024-03-24T00:50:08.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.987+0000 7f56a099d640 1 -- 172.21.15.67:0/3295984754 shutdown_connections 2024-03-24T00:50:08.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.987+0000 7f56a099d640 1 --2- 172.21.15.67:0/3295984754 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f567c06bc00 0x7f567c06e0c0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:08.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.987+0000 7f56a099d640 1 --2- 172.21.15.67:0/3295984754 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f569c0c4ec0 0x7f569c15cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:08.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.987+0000 7f56a099d640 1 --2- 172.21.15.67:0/3295984754 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f569c0c64a0 0x7f569c0c4980 unknown :-1 s=CLOSED pgs=1974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:08.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.987+0000 7f56a099d640 1 --2- 172.21.15.67:0/3295984754 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5688008850 0x7f569c150330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:08.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.987+0000 7f56a099d640 1 -- 172.21.15.67:0/3295984754 >> 172.21.15.67:0/3295984754 conn(0x7f569c0cf9e0 msgr2=0x7f569c159a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:08.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.987+0000 7f56a099d640 1 -- 172.21.15.67:0/3295984754 shutdown_connections 2024-03-24T00:50:08.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:08.987+0000 7f56a099d640 1 -- 172.21.15.67:0/3295984754 wait complete. 2024-03-24T00:50:08.998 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":462,"stamp":"2024-03-24T00:50:07.485516+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":262824,"kb_used_data":13384,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754711896,"statfs":{"total":773094113280,"available":772824981504,"internally_reserved":0,"allocated":13705216,"data_stored":7035936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002186"},"pg_stats":[{"pgid":"1.0","version":"125'119","reported_seq":379,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-24T00:49:49.106062+0000","last_change":"2024-03-24T00:49:49.106128+0000","last_active":"2024-03-24T00:49:49.106062+0000","last_peered":"2024-03-24T00:49:49.106062+0000","last_clean":"2024-03-24T00:49:49.106062+0000","last_became_active":"2024-03-24T00:38:40.952926+0000","last_became_peered":"2024-03-24T00:38:40.952926+0000","last_unstale":"2024-03-24T00:49:49.106062+0000","last_undegraded":"2024-03-24T00:49:49.106062+0000","last_fullsized":"2024-03-24T00:49:49.106062+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'119","last_scrub_stamp":"2024-03-24T00:49:49.105949+0000","last_deep_scrub":"125'119","last_deep_scrub_stamp":"2024-03-24T00:49:49.105949+0000","last_clean_scrub_stamp":"2024-03-24T00:49:49.105949+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:21:04.603898+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574990,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33284,"kb_used_data":2108,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338556,"statfs":{"total":96636764160,"available":96602681344,"internally_reserved":0,"allocated":2158592,"data_stored":1322268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574989,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32708,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339132,"statfs":{"total":96636764160,"available":96603271168,"internally_reserved":0,"allocated":1564672,"data_stored":731900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574989,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32712,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339128,"statfs":{"total":96636764160,"available":96603267072,"internally_reserved":0,"allocated":1564672,"data_stored":731900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574988,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32712,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339128,"statfs":{"total":96636764160,"available":96603267072,"internally_reserved":0,"allocated":1564672,"data_stored":731900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574988,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32708,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339132,"statfs":{"total":96636764160,"available":96603271168,"internally_reserved":0,"allocated":1564672,"data_stored":731900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574989,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32708,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339132,"statfs":{"total":96636764160,"available":96603271168,"internally_reserved":0,"allocated":1564672,"data_stored":731900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574989,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32708,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339132,"statfs":{"total":96636764160,"available":96603271168,"internally_reserved":0,"allocated":1564672,"data_stored":731900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574988,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33284,"kb_used_data":2108,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338556,"statfs":{"total":96636764160,"available":96602681344,"internally_reserved":0,"allocated":2158592,"data_stored":1322268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:50:08.999 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-03-24T00:50:09.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.100+0000 7f4607190640 1 Processor -- start 2024-03-24T00:50:09.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.100+0000 7f4607190640 1 -- start start 2024-03-24T00:50:09.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.101+0000 7f4607190640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f460015dfd0 0x7f46000cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:09.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.101+0000 7f4607190640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46000cf090 con 0x7f46000b3d30 2024-03-24T00:50:09.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.101+0000 7f4607190640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46000d0320 con 0x7f46000b41e0 2024-03-24T00:50:09.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.101+0000 7f4607190640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46000cc260 con 0x7f460015dfd0 2024-03-24T00:50:09.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.101+0000 7f460572d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f46000b3d30 0x7f46000cc6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55936/0 (socket says 172.21.15.67:55936) 2024-03-24T00:50:09.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.101+0000 7f460572d640 1 -- 172.21.15.67:0/2949891154 learned_addr learned my addr 172.21.15.67:0/2949891154 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-24T00:50:09.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.101+0000 7f4604f2c640 1 --2- 172.21.15.67:0/2949891154 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f460015dfd0 0x7f46000cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:09.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.101+0000 7f45ff7fe640 1 -- 172.21.15.67:0/2949891154 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2039370608 0 0) 0x7f46000cf090 con 0x7f46000b3d30 2024-03-24T00:50:09.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.101+0000 7f45ff7fe640 1 -- 172.21.15.67:0/2949891154 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45ec003520 con 0x7f46000b3d30 2024-03-24T00:50:09.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.101+0000 7f45ff7fe640 1 -- 172.21.15.67:0/2949891154 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1676863737 0 0) 0x7f46000d0320 con 0x7f46000b41e0 2024-03-24T00:50:09.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.101+0000 7f45ff7fe640 1 -- 172.21.15.67:0/2949891154 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46000cf090 con 0x7f46000b41e0 2024-03-24T00:50:09.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.101+0000 7f45ff7fe640 1 -- 172.21.15.67:0/2949891154 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1628132698 0 0) 0x7f45ec003520 con 0x7f46000b3d30 2024-03-24T00:50:09.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.102+0000 7f45ff7fe640 1 -- 172.21.15.67:0/2949891154 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f460015dfd0 msgr2=0x7f46000cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:50:09.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.102+0000 7f45ff7fe640 1 --2- 172.21.15.67:0/2949891154 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f460015dfd0 0x7f46000cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:09.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.102+0000 7f45ff7fe640 1 -- 172.21.15.67:0/2949891154 >> v1:172.21.15.72:6789/0 conn(0x7f46000b41e0 legacy=0x7f46000b3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:09.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.102+0000 7f45ff7fe640 1 -- 172.21.15.67:0/2949891154 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46000d0600 con 0x7f46000b3d30 2024-03-24T00:50:09.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.102+0000 7f45ff7fe640 1 -- 172.21.15.67:0/2949891154 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1463896875 0 0) 0x7f45e8001670 con 0x7f46000b3d30 2024-03-24T00:50:09.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.102+0000 7f4604f2c640 1 --2- 172.21.15.67:0/2949891154 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f460015dfd0 0x7f46000cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:50:09.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.102+0000 7f45ff7fe640 1 -- 172.21.15.67:0/2949891154 >> v1:172.21.15.67:6789/0 conn(0x7f46000b3d30 legacy=0x7f46000cc6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:09.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.102+0000 7f45ff7fe640 1 --2- 172.21.15.67:0/2949891154 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f460015dfd0 0x7f45ec003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:09.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.102+0000 7f45ff7fe640 1 --2- 172.21.15.67:0/2949891154 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f45ec0040b0 0x7f45ec004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:09.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.102+0000 7f45ff7fe640 1 --2- 172.21.15.67:0/2949891154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45ec004d40 0x7f45ec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:09.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.102+0000 7f45ff7fe640 1 -- 172.21.15.67:0/2949891154 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45ec00ba70 con 0x7f45ec004d40 2024-03-24T00:50:09.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.102+0000 7f45ff7fe640 1 -- 172.21.15.67:0/2949891154 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45ec00bbf0 con 0x7f45ec0040b0 2024-03-24T00:50:09.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.102+0000 7f45ff7fe640 1 -- 172.21.15.67:0/2949891154 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45ec00bdf0 con 0x7f460015dfd0 2024-03-24T00:50:09.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.102+0000 7f460572d640 1 --2- 172.21.15.67:0/2949891154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45ec004d40 0x7f45ec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:09.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.102+0000 7f4604f2c640 1 --2- 172.21.15.67:0/2949891154 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f460015dfd0 0x7f45ec003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:09.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.102+0000 7f45fffff640 1 --2- 172.21.15.67:0/2949891154 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f45ec0040b0 0x7f45ec004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:09.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.102+0000 7f460572d640 1 -- 172.21.15.67:0/2949891154 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f460015dfd0 msgr2=0x7f45ec003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:50:09.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.102+0000 7f460572d640 1 --2- 172.21.15.67:0/2949891154 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f460015dfd0 0x7f45ec003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:09.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.102+0000 7f460572d640 1 -- 172.21.15.67:0/2949891154 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f45ec0040b0 msgr2=0x7f45ec004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:50:09.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.102+0000 7f460572d640 1 --2- 172.21.15.67:0/2949891154 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f45ec0040b0 0x7f45ec004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:09.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.102+0000 7f460572d640 1 -- 172.21.15.67:0/2949891154 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45ec003520 con 0x7f45ec004d40 2024-03-24T00:50:09.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.103+0000 7f460572d640 1 --2- 172.21.15.67:0/2949891154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45ec004d40 0x7f45ec0051c0 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f46000d0d40 tx=0x7f45ec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cded1bebdbf789b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:09.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.103+0000 7f45ff7fe640 1 -- 172.21.15.67:0/2949891154 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e8007ca0 con 0x7f45ec004d40 2024-03-24T00:50:09.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.103+0000 7f45ff7fe640 1 -- 172.21.15.67:0/2949891154 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45e8030070 con 0x7f45ec004d40 2024-03-24T00:50:09.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.103+0000 7f45ff7fe640 1 -- 172.21.15.67:0/2949891154 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e8005d10 con 0x7f45ec004d40 2024-03-24T00:50:09.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.103+0000 7f4607190640 1 -- 172.21.15.67:0/2949891154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45ec004d40 msgr2=0x7f45ec0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:09.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.103+0000 7f4607190640 1 --2- 172.21.15.67:0/2949891154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45ec004d40 0x7f45ec0051c0 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f46000d0d40 tx=0x7f45ec005830 comp rx=0 tx=0).stop 2024-03-24T00:50:09.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.103+0000 7f460572d640 1 -- 172.21.15.67:0/2949891154 reap_dead start 2024-03-24T00:50:09.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.104+0000 7f4607190640 1 -- 172.21.15.67:0/2949891154 shutdown_connections 2024-03-24T00:50:09.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.104+0000 7f4607190640 1 -- 172.21.15.67:0/2949891154 >> 172.21.15.67:0/2949891154 conn(0x7f460006fcf0 msgr2=0x7f46000700d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:09.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.104+0000 7f4607190640 1 -- 172.21.15.67:0/2949891154 shutdown_connections 2024-03-24T00:50:09.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.104+0000 7f4607190640 1 -- 172.21.15.67:0/2949891154 wait complete. 2024-03-24T00:50:09.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.105+0000 7f4607190640 1 Processor -- start 2024-03-24T00:50:09.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.105+0000 7f4607190640 1 -- start start 2024-03-24T00:50:09.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.106+0000 7f4607190640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4600150fc0 0x7f46000c9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:09.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.106+0000 7f4607190640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f46000c6350 0x7f46000b6b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:09.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.106+0000 7f4607190640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f46000cb0a0 0x7f46000cb520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:09.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.106+0000 7f4607190640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f460015e3b0 con 0x7f4600150fc0 2024-03-24T00:50:09.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.106+0000 7f4607190640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46000cedb0 con 0x7f46000c6350 2024-03-24T00:50:09.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.106+0000 7f4607190640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46000cf980 con 0x7f46000cb0a0 2024-03-24T00:50:09.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.106+0000 7f4604f2c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4600150fc0 0x7f46000c9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:09.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.106+0000 7f4604f2c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4600150fc0 0x7f46000c9cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56158/0 (socket says 172.21.15.67:56158) 2024-03-24T00:50:09.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.106+0000 7f4604f2c640 1 -- 172.21.15.67:0/1126072884 learned_addr learned my addr 172.21.15.67:0/1126072884 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:50:09.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.106+0000 7f45fffff640 1 --2- 172.21.15.67:0/1126072884 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f46000c6350 0x7f46000b6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:09.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.106+0000 7f460572d640 1 --2- 172.21.15.67:0/1126072884 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f46000cb0a0 0x7f46000cb520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:09.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.106+0000 7f4604f2c640 1 -- 172.21.15.67:0/1126072884 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f46000cb0a0 msgr2=0x7f46000cb520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:09.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.106+0000 7f4604f2c640 1 --2- 172.21.15.67:0/1126072884 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f46000cb0a0 0x7f46000cb520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:09.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.106+0000 7f4604f2c640 1 -- 172.21.15.67:0/1126072884 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f46000c6350 msgr2=0x7f46000b6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:09.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.106+0000 7f4604f2c640 1 --2- 172.21.15.67:0/1126072884 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f46000c6350 0x7f46000b6b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:09.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.106+0000 7f4604f2c640 1 -- 172.21.15.67:0/1126072884 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46000b2220 con 0x7f4600150fc0 2024-03-24T00:50:09.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.106+0000 7f4604f2c640 1 --2- 172.21.15.67:0/1126072884 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4600150fc0 0x7f46000c9cc0 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f45f80119d0 tx=0x7f45f8011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:09.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.106+0000 7f45fd7fa640 1 -- 172.21.15.67:0/1126072884 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45f800fa90 con 0x7f4600150fc0 2024-03-24T00:50:09.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.106+0000 7f45fd7fa640 1 -- 172.21.15.67:0/1126072884 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45f800fc30 con 0x7f4600150fc0 2024-03-24T00:50:09.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.106+0000 7f4607190640 1 -- 172.21.15.67:0/1126072884 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46000ca2a0 con 0x7f4600150fc0 2024-03-24T00:50:09.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.106+0000 7f45fd7fa640 1 -- 172.21.15.67:0/1126072884 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45f8014440 con 0x7f4600150fc0 2024-03-24T00:50:09.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.106+0000 7f4607190640 1 -- 172.21.15.67:0/1126072884 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46000ca860 con 0x7f4600150fc0 2024-03-24T00:50:09.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.107+0000 7f4607190640 1 -- 172.21.15.67:0/1126072884 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f460004f400 con 0x7f4600150fc0 2024-03-24T00:50:09.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.108+0000 7f45fd7fa640 1 -- 172.21.15.67:0/1126072884 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f45f8012040 con 0x7f4600150fc0 2024-03-24T00:50:09.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.108+0000 7f45fd7fa640 1 --2- 172.21.15.67:0/1126072884 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f45e006bbd0 0x7f45e006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:09.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.108+0000 7f45fd7fa640 1 -- 172.21.15.67:0/1126072884 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f45f808af80 con 0x7f4600150fc0 2024-03-24T00:50:09.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.108+0000 7f45fffff640 1 --2- 172.21.15.67:0/1126072884 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f45e006bbd0 0x7f45e006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:09.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.109+0000 7f45fffff640 1 --2- 172.21.15.67:0/1126072884 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f45e006bbd0 0x7f45e006e090 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f46000c55f0 tx=0x7f45f0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:09.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.113+0000 7f45fd7fa640 1 -- 172.21.15.67:0/1126072884 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f46000b2220 con 0x7f4600150fc0 2024-03-24T00:50:09.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.294+0000 7f4607190640 1 -- 172.21.15.67:0/1126072884 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f46000b1710 con 0x7f4600150fc0 2024-03-24T00:50:09.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.298+0000 7f45fd7fa640 1 -- 172.21.15.67:0/1126072884 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v24)=0 v24) v1 ==== 123+0+0 (secure 0 0 0) 0x7f45f8019030 con 0x7f4600150fc0 2024-03-24T00:50:09.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.299+0000 7f45fd7fa640 1 -- 172.21.15.67:0/1126072884 <== mon.0 v2:172.21.15.67:3300/0 8 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f45f8059b30 con 0x7f4600150fc0 2024-03-24T00:50:09.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.300+0000 7f4607190640 1 -- 172.21.15.67:0/1126072884 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f45e006bbd0 msgr2=0x7f45e006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:09.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.300+0000 7f4607190640 1 --2- 172.21.15.67:0/1126072884 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f45e006bbd0 0x7f45e006e090 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f46000c55f0 tx=0x7f45f0009040 comp rx=0 tx=0).stop 2024-03-24T00:50:09.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.300+0000 7f4607190640 1 -- 172.21.15.67:0/1126072884 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4600150fc0 msgr2=0x7f46000c9cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:09.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.300+0000 7f4607190640 1 --2- 172.21.15.67:0/1126072884 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4600150fc0 0x7f46000c9cc0 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f45f80119d0 tx=0x7f45f8011e90 comp rx=0 tx=0).stop 2024-03-24T00:50:09.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.300+0000 7f4607190640 1 -- 172.21.15.67:0/1126072884 shutdown_connections 2024-03-24T00:50:09.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.300+0000 7f4607190640 1 --2- 172.21.15.67:0/1126072884 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7f45e006bbd0 0x7f45e006e090 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:09.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.300+0000 7f4607190640 1 --2- 172.21.15.67:0/1126072884 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f46000cb0a0 0x7f46000cb520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:09.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.300+0000 7f4607190640 1 --2- 172.21.15.67:0/1126072884 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f46000c6350 0x7f46000b6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:09.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.300+0000 7f4607190640 1 --2- 172.21.15.67:0/1126072884 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4600150fc0 0x7f46000c9cc0 unknown :-1 s=CLOSED pgs=1977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:09.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.300+0000 7f4607190640 1 -- 172.21.15.67:0/1126072884 >> 172.21.15.67:0/1126072884 conn(0x7f460006fcf0 msgr2=0x7f460006d6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:09.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.300+0000 7f4607190640 1 -- 172.21.15.67:0/1126072884 shutdown_connections 2024-03-24T00:50:09.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.300+0000 7f4607190640 1 -- 172.21.15.67:0/1126072884 wait complete. 2024-03-24T00:50:09.311 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-24T00:50:09.312 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:50:09.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.412+0000 7fc0b78d3640 1 Processor -- start 2024-03-24T00:50:09.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.412+0000 7fc0b78d3640 1 -- start start 2024-03-24T00:50:09.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.413+0000 7fc0b78d3640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc0b015dec0 0x7fc0b00ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:09.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.413+0000 7fc0b78d3640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc0b00d0ef0 0x7fc0b0162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:09.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.413+0000 7fc0b78d3640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0b00cf640 con 0x7fc0b015e2a0 2024-03-24T00:50:09.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.413+0000 7fc0b78d3640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0b00cee30 con 0x7fc0b015dec0 2024-03-24T00:50:09.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.413+0000 7fc0b78d3640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b00d2120 con 0x7fc0b00d0ef0 2024-03-24T00:50:09.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.414+0000 7fc0b566f640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc0b015dec0 0x7fc0b00ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:09.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.414+0000 7fc0b5e70640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc0b00d0ef0 0x7fc0b0162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:09.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.414+0000 7fc0b566f640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc0b015dec0 0x7fc0b00ce630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.67:45458/0 (socket says 172.21.15.67:45458) 2024-03-24T00:50:09.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.414+0000 7fc0b566f640 1 -- 172.21.15.67:0/3766002356 learned_addr learned my addr 172.21.15.67:0/3766002356 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:50:09.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.414+0000 7fc0b4e6e640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fc0b015e2a0 0x7fc0b00d07e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.67:48150/0 (socket says 172.21.15.67:48150) 2024-03-24T00:50:09.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.414+0000 7fc0b5e70640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc0b00d0ef0 0x7fc0b0162630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.67:47658/0 (socket says 172.21.15.67:47658) 2024-03-24T00:50:09.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.414+0000 7fc0b566f640 1 -- 172.21.15.67:0/3766002356 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc0b00d0ef0 msgr2=0x7fc0b0162630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:50:09.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.414+0000 7fc0b566f640 1 --2- 172.21.15.67:0/3766002356 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc0b00d0ef0 0x7fc0b0162630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:09.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.414+0000 7fc0b566f640 1 -- 172.21.15.67:0/3766002356 >> v1:172.21.15.72:6789/0 conn(0x7fc0b015e2a0 legacy=0x7fc0b00d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:09.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.414+0000 7fc0b566f640 1 -- 172.21.15.67:0/3766002356 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0b00cf080 con 0x7fc0b015dec0 2024-03-24T00:50:09.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.414+0000 7fc0affff640 1 -- 172.21.15.67:0/3766002356 <== mon.1 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4049629454 0 0) 0x7fc0a4001110 con 0x7fc0b015e2a0 2024-03-24T00:50:09.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.415+0000 7fc0b566f640 1 --2- 172.21.15.67:0/3766002356 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc0b015dec0 0x7fc0b00ce630 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fc0a8009870 tx=0x7fc0a802b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c578c445615b65fd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:09.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.415+0000 7fc0affff640 1 -- 172.21.15.67:0/3766002356 <== mon.1 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a8035020 con 0x7fc0b015dec0 2024-03-24T00:50:09.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.415+0000 7fc0affff640 1 -- 172.21.15.67:0/3766002356 <== mon.1 v2:172.21.15.72:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fc0a8031640 con 0x7fc0b015dec0 2024-03-24T00:50:09.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.415+0000 7fc0affff640 1 -- 172.21.15.67:0/3766002356 <== mon.1 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a802d040 con 0x7fc0b015dec0 2024-03-24T00:50:09.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.415+0000 7fc0b78d3640 1 -- 172.21.15.67:0/3766002356 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc0b015dec0 msgr2=0x7fc0b00ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:09.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.415+0000 7fc0b78d3640 1 --2- 172.21.15.67:0/3766002356 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc0b015dec0 0x7fc0b00ce630 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fc0a8009870 tx=0x7fc0a802b8f0 comp rx=0 tx=0).stop 2024-03-24T00:50:09.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.415+0000 7fc0b78d3640 1 -- 172.21.15.67:0/3766002356 shutdown_connections 2024-03-24T00:50:09.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.415+0000 7fc0b78d3640 1 --2- 172.21.15.67:0/3766002356 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc0b00d0ef0 0x7fc0b0162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:09.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.415+0000 7fc0b78d3640 1 --2- 172.21.15.67:0/3766002356 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc0b015dec0 0x7fc0b00ce630 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:09.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.415+0000 7fc0b78d3640 1 -- 172.21.15.67:0/3766002356 >> 172.21.15.67:0/3766002356 conn(0x7fc0b006c810 msgr2=0x7fc0b006cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:09.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.416+0000 7fc0b78d3640 1 -- 172.21.15.67:0/3766002356 shutdown_connections 2024-03-24T00:50:09.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.416+0000 7fc0b78d3640 1 -- 172.21.15.67:0/3766002356 wait complete. 2024-03-24T00:50:09.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.417+0000 7fc0b78d3640 1 Processor -- start 2024-03-24T00:50:09.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.417+0000 7fc0b78d3640 1 -- start start 2024-03-24T00:50:09.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.417+0000 7fc0b78d3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0b00d0ef0 0x7fc0b00cc560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:09.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.417+0000 7fc0b78d3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc0b015dec0 0x7fc0b014c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:09.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.417+0000 7fc0b78d3640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc0b015e2a0 0x7fc0b00cdc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:09.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.417+0000 7fc0b78d3640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b00d2730 con 0x7fc0b00d0ef0 2024-03-24T00:50:09.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.417+0000 7fc0b78d3640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b00d25b0 con 0x7fc0b015dec0 2024-03-24T00:50:09.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.417+0000 7fc0b78d3640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b00d28b0 con 0x7fc0b015e2a0 2024-03-24T00:50:09.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.418+0000 7fc0b566f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0b00d0ef0 0x7fc0b00cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:09.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.418+0000 7fc0b566f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0b00d0ef0 0x7fc0b00cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56170/0 (socket says 172.21.15.67:56170) 2024-03-24T00:50:09.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.418+0000 7fc0b566f640 1 -- 172.21.15.67:0/174855821 learned_addr learned my addr 172.21.15.67:0/174855821 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-24T00:50:09.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.418+0000 7fc0b4e6e640 1 --2- 172.21.15.67:0/174855821 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc0b015dec0 0x7fc0b014c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:09.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.418+0000 7fc0b5e70640 1 --2- 172.21.15.67:0/174855821 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc0b015e2a0 0x7fc0b00cdc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:09.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.418+0000 7fc0b566f640 1 -- 172.21.15.67:0/174855821 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc0b015e2a0 msgr2=0x7fc0b00cdc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:09.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.418+0000 7fc0b566f640 1 --2- 172.21.15.67:0/174855821 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc0b015e2a0 0x7fc0b00cdc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:09.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.418+0000 7fc0b566f640 1 -- 172.21.15.67:0/174855821 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc0b015dec0 msgr2=0x7fc0b014c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:09.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.418+0000 7fc0b566f640 1 --2- 172.21.15.67:0/174855821 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc0b015dec0 0x7fc0b014c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:09.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.418+0000 7fc0b566f640 1 -- 172.21.15.67:0/174855821 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0b00ccbd0 con 0x7fc0b00d0ef0 2024-03-24T00:50:09.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.418+0000 7fc0b566f640 1 --2- 172.21.15.67:0/174855821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0b00d0ef0 0x7fc0b00cc560 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7fc0a802b930 tx=0x7fc0a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:09.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.418+0000 7fc0adffb640 1 -- 172.21.15.67:0/174855821 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a8040070 con 0x7fc0b00d0ef0 2024-03-24T00:50:09.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.418+0000 7fc0adffb640 1 -- 172.21.15.67:0/174855821 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fc0a80399f0 con 0x7fc0b00d0ef0 2024-03-24T00:50:09.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.418+0000 7fc0b78d3640 1 -- 172.21.15.67:0/174855821 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0b00cceb0 con 0x7fc0b00d0ef0 2024-03-24T00:50:09.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.418+0000 7fc0adffb640 1 -- 172.21.15.67:0/174855821 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a8035040 con 0x7fc0b00d0ef0 2024-03-24T00:50:09.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.419+0000 7fc0b78d3640 1 -- 172.21.15.67:0/174855821 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0b00cd470 con 0x7fc0b00d0ef0 2024-03-24T00:50:09.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.420+0000 7fc0b78d3640 1 -- 172.21.15.67:0/174855821 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0b004f400 con 0x7fc0b00d0ef0 2024-03-24T00:50:09.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.420+0000 7fc0adffb640 1 -- 172.21.15.67:0/174855821 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc0a8038020 con 0x7fc0b00d0ef0 2024-03-24T00:50:09.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.420+0000 7fc0adffb640 1 --2- 172.21.15.67:0/174855821 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc09c06bbd0 0x7fc09c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:50:09.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.420+0000 7fc0adffb640 1 -- 172.21.15.67:0/174855821 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5507+0+0 (secure 0 0 0) 0x7fc0a80ac3b0 con 0x7fc0b00d0ef0 2024-03-24T00:50:09.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.421+0000 7fc0b4e6e640 1 --2- 172.21.15.67:0/174855821 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc09c06bbd0 0x7fc09c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:50:09.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.421+0000 7fc0b4e6e640 1 --2- 172.21.15.67:0/174855821 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc09c06bbd0 0x7fc09c06e090 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fc0b01f7420 tx=0x7fc0a4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:50:09.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.425+0000 7fc0adffb640 1 -- 172.21.15.67:0/174855821 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc0a807aff0 con 0x7fc0b00d0ef0 2024-03-24T00:50:09.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.645+0000 7fc0b78d3640 1 -- 172.21.15.67:0/174855821 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc0b00d1aa0 con 0x7fc0b00d0ef0 2024-03-24T00:50:09.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.646+0000 7fc0adffb640 1 -- 172.21.15.67:0/174855821 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3698 (secure 0 0 0) 0x7fc0a807a8f0 con 0x7fc0b00d0ef0 2024-03-24T00:50:09.647 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-24T00:50:09.647 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":101,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7298,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/839277571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":839277571},{"type":"v1","addr":"172.21.15.72:6836","nonce":839277571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":8224,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2052380377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2052380377},{"type":"v1","addr":"172.21.15.67:6834","nonce":2052380377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":8239,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2196618204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2196618204},{"type":"v1","addr":"172.21.15.72:6837","nonce":2196618204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":9888,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1209900873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1209900873},{"type":"v1","addr":"172.21.15.67:6835","nonce":1209900873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101}],"filesystems":[]} 2024-03-24T00:50:09.647 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 101 2024-03-24T00:50:09.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.648+0000 7fc0b78d3640 1 -- 172.21.15.67:0/174855821 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc09c06bbd0 msgr2=0x7fc09c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:09.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.648+0000 7fc0b78d3640 1 --2- 172.21.15.67:0/174855821 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc09c06bbd0 0x7fc09c06e090 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fc0b01f7420 tx=0x7fc0a4009040 comp rx=0 tx=0).stop 2024-03-24T00:50:09.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.648+0000 7fc0b78d3640 1 -- 172.21.15.67:0/174855821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0b00d0ef0 msgr2=0x7fc0b00cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:50:09.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.648+0000 7fc0b78d3640 1 --2- 172.21.15.67:0/174855821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0b00d0ef0 0x7fc0b00cc560 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7fc0a802b930 tx=0x7fc0a80046c0 comp rx=0 tx=0).stop 2024-03-24T00:50:09.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.648+0000 7fc0b78d3640 1 -- 172.21.15.67:0/174855821 shutdown_connections 2024-03-24T00:50:09.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.648+0000 7fc0b78d3640 1 --2- 172.21.15.67:0/174855821 >> [v2:172.21.15.72:6832/2768292274,v1:172.21.15.72:6833/2768292274] conn(0x7fc09c06bbd0 0x7fc09c06e090 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:09.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.648+0000 7fc0b78d3640 1 --2- 172.21.15.67:0/174855821 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc0b015e2a0 0x7fc0b00cdc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:09.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.648+0000 7fc0b78d3640 1 --2- 172.21.15.67:0/174855821 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc0b015dec0 0x7fc0b014c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:09.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.648+0000 7fc0b78d3640 1 --2- 172.21.15.67:0/174855821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0b00d0ef0 0x7fc0b00cc560 unknown :-1 s=CLOSED pgs=1978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:50:09.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.648+0000 7fc0b78d3640 1 -- 172.21.15.67:0/174855821 >> 172.21.15.67:0/174855821 conn(0x7fc0b006c810 msgr2=0x7fc0b00b93f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:50:09.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.648+0000 7fc0b78d3640 1 -- 172.21.15.67:0/174855821 shutdown_connections 2024-03-24T00:50:09.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-24T00:50:09.648+0000 7fc0b78d3640 1 -- 172.21.15.67:0/174855821 wait complete. 2024-03-24T00:50:09.659 DEBUG:tasks.cephfs.filesystem:already dead... 2024-03-24T00:50:09.660 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-24T00:50:09.660 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-24T00:50:09.660 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:50:09.668 INFO:tasks.ceph.mds.a:Stopped 2024-03-24T00:50:09.668 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-24T00:50:09.668 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:50:09.717 INFO:tasks.ceph.mds.c:Stopped 2024-03-24T00:50:09.717 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-24T00:50:09.718 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:50:09.730 INFO:tasks.ceph.mds.b:Stopped 2024-03-24T00:50:09.731 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-24T00:50:09.731 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:50:09.782 INFO:tasks.ceph.mds.d:Stopped 2024-03-24T00:50:09.782 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-24T00:50:09.783 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-24T00:50:09.783 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:50:09.815 INFO:tasks.ceph.osd.0:Stopped 2024-03-24T00:50:09.815 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-24T00:50:09.815 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:50:09.883 INFO:tasks.ceph.osd.1:Stopped 2024-03-24T00:50:09.883 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-24T00:50:09.883 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:50:09.955 INFO:tasks.ceph.osd.2:Stopped 2024-03-24T00:50:09.955 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-24T00:50:09.956 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:50:10.024 INFO:tasks.ceph.osd.3:Stopped 2024-03-24T00:50:10.024 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-24T00:50:10.024 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:50:10.052 INFO:tasks.ceph.osd.4:Stopped 2024-03-24T00:50:10.053 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-24T00:50:10.053 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:50:10.131 INFO:tasks.ceph.osd.5:Stopped 2024-03-24T00:50:10.131 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-24T00:50:10.131 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:50:10.200 INFO:tasks.ceph.osd.6:Stopped 2024-03-24T00:50:10.200 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-24T00:50:10.200 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:50:10.278 INFO:tasks.ceph.osd.7:Stopped 2024-03-24T00:50:10.278 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-24T00:50:10.278 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-24T00:50:10.278 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:50:10.303 INFO:tasks.ceph.mgr.y:Stopped 2024-03-24T00:50:10.303 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-24T00:50:10.303 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:50:10.347 INFO:tasks.ceph.mgr.x:Stopped 2024-03-24T00:50:10.348 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-24T00:50:10.348 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-24T00:50:10.348 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:50:10.358 INFO:tasks.ceph.mon.a:Stopped 2024-03-24T00:50:10.358 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-24T00:50:10.358 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:50:10.400 INFO:tasks.ceph.mon.b:Stopped 2024-03-24T00:50:10.401 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-24T00:50:10.401 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:50:10.453 INFO:tasks.ceph.mon.c:Stopped 2024-03-24T00:50:10.453 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-24T00:50:10.454 DEBUG:teuthology.orchestra.run.smithi067:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'inode wrongly marked free' | egrep -v 'bad backtrace on inode' | egrep -v 'inode table repaired for inode' | egrep -v 'Scrub error on inode' | egrep -v 'Scrub error on dir' | egrep -v 'Metadata damage detected' | head -n 1 2024-03-24T00:50:10.490 INFO:teuthology.orchestra.run.smithi067.stdout:2024-03-24T00:43:31.091435+0000 mds.c (mds.0) 1 : cluster [WRN] Corrupt dentry 'file_to_be_damaged' in dir frag 0x10000000000: Invalid tag char 'd' pos 0: Malformed input [buffer:3](/subdir) 2024-03-24T00:50:10.490 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2024-03-24T00:50:10.490 DEBUG:teuthology.orchestra.run.smithi067:> sudo egrep '\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'inode wrongly marked free' | egrep -v 'bad backtrace on inode' | egrep -v 'inode table repaired for inode' | egrep -v 'Scrub error on inode' | egrep -v 'Scrub error on dir' | egrep -v 'Metadata damage detected' | head -n 1 2024-03-24T00:50:10.563 DEBUG:teuthology.orchestra.run.smithi067:> sudo egrep '\[ERR\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'inode wrongly marked free' | egrep -v 'bad backtrace on inode' | egrep -v 'inode table repaired for inode' | egrep -v 'Scrub error on inode' | egrep -v 'Scrub error on dir' | egrep -v 'Metadata damage detected' | head -n 1 2024-03-24T00:50:10.636 INFO:teuthology.orchestra.run.smithi067.stdout:2024-03-24T00:45:15.653093+0000 mds.c (mds.0) 1 : cluster [ERR] dir 0x10000000000 object missing on disk; some files may be lost (/dir) 2024-03-24T00:50:10.636 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi067.front.sepia.ceph.com 2024-03-24T00:50:10.636 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-24T00:50:10.825 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi067.front.sepia.ceph.com 2024-03-24T00:50:10.826 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-24T00:50:10.909 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi067.front.sepia.ceph.com 2024-03-24T00:50:10.909 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-24T00:50:10.959 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi067.front.sepia.ceph.com 2024-03-24T00:50:10.959 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-24T00:50:11.007 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi072.front.sepia.ceph.com 2024-03-24T00:50:11.007 DEBUG:teuthology.orchestra.run.smithi072:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-24T00:50:11.175 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi072.front.sepia.ceph.com 2024-03-24T00:50:11.175 DEBUG:teuthology.orchestra.run.smithi072:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-24T00:50:11.296 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi072.front.sepia.ceph.com 2024-03-24T00:50:11.297 DEBUG:teuthology.orchestra.run.smithi072:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-24T00:50:11.355 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi072.front.sepia.ceph.com 2024-03-24T00:50:11.355 DEBUG:teuthology.orchestra.run.smithi072:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-24T00:50:11.403 INFO:tasks.ceph:Archiving mon data... 2024-03-24T00:50:11.404 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616442/data/mon.a.tgz 2024-03-24T00:50:11.405 DEBUG:teuthology.orchestra.run.smithi067:> mktemp 2024-03-24T00:50:11.419 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.9UuDBU9NqH 2024-03-24T00:50:11.419 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.9UuDBU9NqH 2024-03-24T00:50:11.571 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.9UuDBU9NqH 2024-03-24T00:50:11.654 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.9UuDBU9NqH is 411KB 2024-03-24T00:50:11.751 DEBUG:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.9UuDBU9NqH 2024-03-24T00:50:11.765 DEBUG:teuthology.misc:Transferring archived files from smithi072:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616442/data/mon.b.tgz 2024-03-24T00:50:11.765 DEBUG:teuthology.orchestra.run.smithi072:> mktemp 2024-03-24T00:50:11.782 INFO:teuthology.orchestra.run.smithi072.stdout:/tmp/tmp.AxXiRJfVoh 2024-03-24T00:50:11.782 DEBUG:teuthology.orchestra.run.smithi072:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.AxXiRJfVoh 2024-03-24T00:50:11.942 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 0666 /tmp/tmp.AxXiRJfVoh 2024-03-24T00:50:12.027 DEBUG:teuthology.orchestra.remote:smithi072:/tmp/tmp.AxXiRJfVoh is 417KB 2024-03-24T00:50:12.122 DEBUG:teuthology.orchestra.run.smithi072:> rm -fr /tmp/tmp.AxXiRJfVoh 2024-03-24T00:50:12.136 DEBUG:teuthology.misc:Transferring archived files from smithi072:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616442/data/mon.c.tgz 2024-03-24T00:50:12.136 DEBUG:teuthology.orchestra.run.smithi072:> mktemp 2024-03-24T00:50:12.190 INFO:teuthology.orchestra.run.smithi072.stdout:/tmp/tmp.NB7wQNZwA4 2024-03-24T00:50:12.190 DEBUG:teuthology.orchestra.run.smithi072:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.NB7wQNZwA4 2024-03-24T00:50:12.334 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 0666 /tmp/tmp.NB7wQNZwA4 2024-03-24T00:50:12.419 DEBUG:teuthology.orchestra.remote:smithi072:/tmp/tmp.NB7wQNZwA4 is 417KB 2024-03-24T00:50:12.515 DEBUG:teuthology.orchestra.run.smithi072:> rm -fr /tmp/tmp.NB7wQNZwA4 2024-03-24T00:50:12.529 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-24T00:50:12.529 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-24T00:50:12.532 DEBUG:teuthology.orchestra.run.smithi072:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-24T00:50:12.616 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-24T00:50:12.651 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-24T00:50:12.651 INFO:tasks.ceph:Archiving crash dumps... 2024-03-24T00:50:12.653 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/lib/ceph/crash to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616442/remote/smithi067/crash 2024-03-24T00:50:12.654 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-24T00:50:12.685 DEBUG:teuthology.misc:Transferring archived files from smithi072:/var/lib/ceph/crash to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616442/remote/smithi072/crash 2024-03-24T00:50:12.686 DEBUG:teuthology.orchestra.run.smithi072:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-24T00:50:12.716 INFO:tasks.ceph:Shutting down logrotate 2024-03-24T00:50:12.717 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:50:12.730 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:50:12.808 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-24T00:50:12.836 DEBUG:teuthology.orchestra.run.smithi072:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-24T00:50:12.862 INFO:tasks.ceph:Compressing logs... 2024-03-24T00:50:12.862 DEBUG:teuthology.orchestra.run.smithi067:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T00:50:12.879 DEBUG:teuthology.orchestra.run.smithi072:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T00:50:12.903 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63576.log 2024-03-24T00:50:12.904 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74378.log 2024-03-24T00:50:12.904 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63894.log 2024-03-24T00:50:12.954 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-24T00:50:12.955 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.3.44421.log 2024-03-24T00:50:12.955 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37582.log 2024-03-24T00:50:12.955 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.3.44421.log: 0.0% -- replaced with /var/log/ceph/ceph-client.3.44421.log.gz 2024-03-24T00:50:12.955 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-24T00:50:12.955 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.37582.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-03-24T00:50:12.955 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37662.log 2024-03-24T00:50:12.955 INFO:teuthology.orchestra.run.smithi072.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.37582.log.gz 2024-03-24T00:50:12.955 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-24T00:50:12.955 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.37662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37622.log 2024-03-24T00:50:12.955 INFO:teuthology.orchestra.run.smithi072.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37662.log.gz 2024-03-24T00:50:12.955 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-24T00:50:12.955 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37422.log 2024-03-24T00:50:12.956 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.37622.log: /var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-24T00:50:12.956 INFO:teuthology.orchestra.run.smithi072.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37622.log.gz 2024-03-24T00:50:12.956 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.37422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37702.log 2024-03-24T00:50:12.956 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37542.log 2024-03-24T00:50:12.956 INFO:teuthology.orchestra.run.smithi072.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.37422.log.gz 2024-03-24T00:50:12.956 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.37702.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37702.log.gz 2024-03-24T00:50:12.956 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.44029.log 2024-03-24T00:50:12.956 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.37542.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-24T00:50:12.956 INFO:teuthology.orchestra.run.smithi072.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37542.log.gz 2024-03-24T00:50:12.956 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.1.44029.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.44029.log.gz 2024-03-24T00:50:12.956 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.44024.log 2024-03-24T00:50:12.956 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-mgr.x.log: 93.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-24T00:50:12.956 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.3.44417.log 2024-03-24T00:50:12.956 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-24T00:50:12.956 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.1.44024.log: /var/log/ceph/ceph-client.3.44417.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.1.44024.log.gz 2024-03-24T00:50:12.956 INFO:teuthology.orchestra.run.smithi072.stderr: -- replaced with /var/log/ceph/ceph-client.3.44417.log.gz 2024-03-24T00:50:12.957 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37462.log 2024-03-24T00:50:12.957 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37502.log 2024-03-24T00:50:12.957 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.37462.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37462.log.gz 2024-03-24T00:50:12.957 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-24T00:50:12.957 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.37502.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37502.log.gz 2024-03-24T00:50:12.957 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-03-24T00:50:12.957 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-mon.b.log: 92.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-24T00:50:12.957 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-24T00:50:12.971 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-mds.d.log: /var/log/ceph/ceph-mon.c.log: 92.0% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-03-24T00:50:13.011 INFO:teuthology.orchestra.run.smithi072.stderr: 93.0% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-03-24T00:50:13.098 INFO:teuthology.orchestra.run.smithi072.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-24T00:50:13.237 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.74378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60425.log 2024-03-24T00:50:13.237 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.63576.log.gz 2024-03-24T00:50:13.237 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.74378.log.gz 2024-03-24T00:50:13.237 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51016.log 2024-03-24T00:50:13.237 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55519.log 2024-03-24T00:50:13.237 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.63894.log.gz 2024-03-24T00:50:13.237 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51016.log: gzip -5 --verbose -- 89.5% /var/log/ceph/ceph-client.admin.61881.log 2024-03-24T00:50:13.237 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.60425.log.gz 2024-03-24T00:50:13.237 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51016.log.gz 2024-03-24T00:50:13.237 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64837.log 2024-03-24T00:50:13.237 INFO:teuthology.orchestra.run.smithi067.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55519.log.gz 2024-03-24T00:50:13.238 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66248.log 2024-03-24T00:50:13.238 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64837.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.61881.log.gz 2024-03-24T00:50:13.238 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77199.log 2024-03-24T00:50:13.238 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.66248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78079.log 2024-03-24T00:50:13.238 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.66248.log.gz 2024-03-24T00:50:13.238 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.77199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51532.log 2024-03-24T00:50:13.238 INFO:teuthology.orchestra.run.smithi067.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.64837.log.gz 2024-03-24T00:50:13.238 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.77199.log.gz 2024-03-24T00:50:13.238 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.78079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43948.log 2024-03-24T00:50:13.238 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51532.log: gzip 88.7% -5 -- replaced with /var/log/ceph/ceph-client.admin.78079.log.gz 2024-03-24T00:50:13.238 INFO:teuthology.orchestra.run.smithi067.stderr: --verbose -- /var/log/ceph/ceph-client.admin.49435.log 2024-03-24T00:50:13.238 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43948.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51532.log.gz 2024-03-24T00:50:13.238 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56080.log 2024-03-24T00:50:13.238 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49435.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43948.log.gz 2024-03-24T00:50:13.238 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69851.log 2024-03-24T00:50:13.239 INFO:teuthology.orchestra.run.smithi067.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49435.log.gz 2024-03-24T00:50:13.239 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56080.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-24T00:50:13.239 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.69851.log: 89.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.56080.log.gz -5 2024-03-24T00:50:13.239 INFO:teuthology.orchestra.run.smithi067.stderr: --verbose -- /var/log/ceph/ceph-client.admin.55134.log 2024-03-24T00:50:13.239 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.69851.log.gz 2024-03-24T00:50:13.239 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79806.log 2024-03-24T00:50:13.239 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74624.log 2024-03-24T00:50:13.239 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3%/var/log/ceph/ceph-client.admin.79806.log: -- replaced with /var/log/ceph/ceph-client.admin.55134.log.gz 2024-03-24T00:50:13.239 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74462.log 2024-03-24T00:50:13.239 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.79806.log.gz 2024-03-24T00:50:13.239 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.74624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80941.log 2024-03-24T00:50:13.239 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.74462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50632.log 2024-03-24T00:50:13.239 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.74624.log.gz 2024-03-24T00:50:13.239 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.80941.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.74462.log.gz 2024-03-24T00:50:13.240 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49687.log 2024-03-24T00:50:13.240 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.80941.log.gz 2024-03-24T00:50:13.240 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.50632.log: /var/log/ceph/ceph-client.admin.84085.log 2024-03-24T00:50:13.240 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50023.log 2024-03-24T00:50:13.240 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50632.log.gz 2024-03-24T00:50:13.240 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49687.log.gz 2024-03-24T00:50:13.240 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.84085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55302.log 2024-03-24T00:50:13.240 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57825.log 2024-03-24T00:50:13.240 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.84085.log.gz 2024-03-24T00:50:13.240 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50023.log.gz 2024-03-24T00:50:13.240 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.55302.log: /var/log/ceph/ceph-client.admin.57311.log 2024-03-24T00:50:13.240 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43860.log 2024-03-24T00:50:13.240 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55302.log.gz 2024-03-24T00:50:13.240 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57825.log.gz 2024-03-24T00:50:13.240 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67689.log 2024-03-24T00:50:13.241 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57311.log.gz 2024-03-24T00:50:13.241 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82782.log 2024-03-24T00:50:13.241 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.67689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.65757.log 2024-03-24T00:50:13.241 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.43860.log.gz 2024-03-24T00:50:13.241 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.67689.log.gz 2024-03-24T00:50:13.241 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.82782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46182.log 2024-03-24T00:50:13.241 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.82782.log.gz 2024-03-24T00:50:13.241 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.65757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46181.log 2024-03-24T00:50:13.241 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.65757.log.gz 2024-03-24T00:50:13.241 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47958.log 2024-03-24T00:50:13.241 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46182.log.gz 2024-03-24T00:50:13.241 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54087.log 2024-03-24T00:50:13.241 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46181.log.gz 2024-03-24T00:50:13.241 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54437.log 2024-03-24T00:50:13.242 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47958.log: /var/log/ceph/ceph-client.admin.54087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54400.log 2024-03-24T00:50:13.242 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5%/var/log/ceph/ceph-client.admin.54437.log: -- replaced with /var/log/ceph/ceph-client.admin.54087.log.gz 2024-03-24T00:50:13.242 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46858.log 2024-03-24T00:50:13.242 INFO:teuthology.orchestra.run.smithi067.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.47958.log.gz 2024-03-24T00:50:13.242 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54400.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.54437.log.gz 2024-03-24T00:50:13.242 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75749.log 2024-03-24T00:50:13.242 INFO:teuthology.orchestra.run.smithi067.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.54400.log.gz 2024-03-24T00:50:13.242 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59949.log 2024-03-24T00:50:13.242 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.75749.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46858.log.gz 2024-03-24T00:50:13.242 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76175.log 2024-03-24T00:50:13.242 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.59949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84355.log 2024-03-24T00:50:13.242 INFO:teuthology.orchestra.run.smithi067.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.75749.log.gz 2024-03-24T00:50:13.242 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.76175.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59949.log.gz 2024-03-24T00:50:13.242 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51700.log 2024-03-24T00:50:13.243 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.84355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51178.log 2024-03-24T00:50:13.243 INFO:teuthology.orchestra.run.smithi067.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.76175.log.gz 2024-03-24T00:50:13.243 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.84355.log.gz 2024-03-24T00:50:13.243 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46525.log 2024-03-24T00:50:13.243 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51700.log.gz 2024-03-24T00:50:13.243 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43904.log 2024-03-24T00:50:13.243 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51178.log: /var/log/ceph/ceph-client.admin.46525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.82110.log 2024-03-24T00:50:13.243 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.51178.log.gz 2024-03-24T00:50:13.243 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43904.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46525.log.gz 2024-03-24T00:50:13.243 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47706.log 2024-03-24T00:50:13.243 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43904.log.gz 2024-03-24T00:50:13.243 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.82110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50716.log 2024-03-24T00:50:13.243 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.82110.log.gz 2024-03-24T00:50:13.243 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50386.log 2024-03-24T00:50:13.244 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4%/var/log/ceph/ceph-client.admin.50716.log: -- replaced with /var/log/ceph/ceph-client.admin.47706.log.gz 2024-03-24T00:50:13.244 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75214.log 2024-03-24T00:50:13.244 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50386.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50716.log.gz 2024-03-24T00:50:13.244 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64676.log 2024-03-24T00:50:13.244 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50386.log.gz 2024-03-24T00:50:13.244 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.75214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45672.log 2024-03-24T00:50:13.244 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.75214.log.gz 2024-03-24T00:50:13.244 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70277.log 2024-03-24T00:50:13.244 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45672.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.64676.log.gz 2024-03-24T00:50:13.244 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73769.log 2024-03-24T00:50:13.244 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45672.log.gz 2024-03-24T00:50:13.244 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.70277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71387.log 2024-03-24T00:50:13.244 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.73769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46857.log 2024-03-24T00:50:13.244 INFO:teuthology.orchestra.run.smithi067.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.70277.log.gz 2024-03-24T00:50:13.244 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.71387.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.73769.log.gz 2024-03-24T00:50:13.245 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57354.log 2024-03-24T00:50:13.245 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43508.log 2024-03-24T00:50:13.245 INFO:teuthology.orchestra.run.smithi067.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.71387.log.gz 2024-03-24T00:50:13.245 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46857.log.gz 2024-03-24T00:50:13.245 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78205.log 2024-03-24T00:50:13.245 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64592.log 2024-03-24T00:50:13.245 INFO:teuthology.orchestra.run.smithi067.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.57354.log.gz 2024-03-24T00:50:13.245 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43508.log.gz 2024-03-24T00:50:13.245 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.78205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69242.log 2024-03-24T00:50:13.245 INFO:teuthology.orchestra.run.smithi067.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.78205.log.gz 2024-03-24T00:50:13.245 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59992.log 2024-03-24T00:50:13.245 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.69242.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.64592.log.gz 2024-03-24T00:50:13.245 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60683.log 2024-03-24T00:50:13.245 INFO:teuthology.orchestra.run.smithi067.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.69242.log.gz 2024-03-24T00:50:13.245 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.59992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62264.log 2024-03-24T00:50:13.245 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60683.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.59992.log.gz 2024-03-24T00:50:13.245 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75851.log 2024-03-24T00:50:13.246 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62264.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.60683.log.gz 2024-03-24T00:50:13.246 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55439.log 2024-03-24T00:50:13.246 INFO:teuthology.orchestra.run.smithi067.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.62264.log.gz 2024-03-24T00:50:13.246 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.75851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67554.log 2024-03-24T00:50:13.246 INFO:teuthology.orchestra.run.smithi067.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.75851.log.gz 2024-03-24T00:50:13.246 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62478.log 2024-03-24T00:50:13.246 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.67554.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55439.log.gz 2024-03-24T00:50:13.246 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55769.log 2024-03-24T00:50:13.246 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.67554.log.gz 2024-03-24T00:50:13.246 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56332.log 2024-03-24T00:50:13.246 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9%/var/log/ceph/ceph-client.admin.55769.log: -- replaced with /var/log/ceph/ceph-client.admin.62478.log.gz 2024-03-24T00:50:13.246 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44202.log 2024-03-24T00:50:13.246 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56332.log: 87.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.55769.log.gz 2024-03-24T00:50:13.246 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.65462.log 2024-03-24T00:50:13.246 INFO:teuthology.orchestra.run.smithi067.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.56332.log.gz 2024-03-24T00:50:13.247 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64087.log 2024-03-24T00:50:13.247 INFO:teuthology.orchestra.run.smithi067.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44202.log.gz 2024-03-24T00:50:13.247 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51430.log 2024-03-24T00:50:13.247 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66071.log 2024-03-24T00:50:13.247 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.64087.log.gz 2024-03-24T00:50:13.247 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51430.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.65462.log.gz 2024-03-24T00:50:13.247 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49729.log 2024-03-24T00:50:13.247 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56206.log 89.3% 2024-03-24T00:50:13.247 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.66071.log: -- replaced with /var/log/ceph/ceph-client.admin.51430.log.gz 2024-03-24T00:50:13.247 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51058.log 2024-03-24T00:50:13.247 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.66071.log.gz 2024-03-24T00:50:13.247 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56206.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49729.log.gz 2024-03-24T00:50:13.247 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49939.log 2024-03-24T00:50:13.247 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51058.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56206.log.gz 2024-03-24T00:50:13.247 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67777.log 2024-03-24T00:50:13.248 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51058.log.gz 2024-03-24T00:50:13.248 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49981.log 2024-03-24T00:50:13.248 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49939.log.gz 2024-03-24T00:50:13.248 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.67777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73727.log 2024-03-24T00:50:13.248 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49981.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.67777.log.gz 2024-03-24T00:50:13.248 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62434.log 2024-03-24T00:50:13.248 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49981.log.gz 2024-03-24T00:50:13.248 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.73727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62829.log 2024-03-24T00:50:13.249 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.73727.log.gz 2024-03-24T00:50:13.249 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54248.log 2024-03-24T00:50:13.249 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62829.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.62434.log.gz 2024-03-24T00:50:13.249 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72888.log 2024-03-24T00:50:13.249 INFO:teuthology.orchestra.run.smithi067.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.62829.log.gz 2024-03-24T00:50:13.249 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47874.log 2024-03-24T00:50:13.249 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54248.log.gz 2024-03-24T00:50:13.249 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.72888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77854.log 2024-03-24T00:50:13.249 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57993.log 2024-03-24T00:50:13.249 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.72888.log.gz 2024-03-24T00:50:13.249 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.77854.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47874.log.gz 2024-03-24T00:50:13.249 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45074.log 2024-03-24T00:50:13.249 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.77854.log.gz 2024-03-24T00:50:13.249 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45032.log 2024-03-24T00:50:13.250 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45074.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.57993.log.gz 2024-03-24T00:50:13.250 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83875.log 2024-03-24T00:50:13.250 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45074.log.gz 2024-03-24T00:50:13.250 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73146.log 2024-03-24T00:50:13.250 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45032.log.gz 2024-03-24T00:50:13.250 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.83875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78607.log 2024-03-24T00:50:13.250 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.73146.log: 89.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.83875.log.gz 2024-03-24T00:50:13.250 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.63141.log 2024-03-24T00:50:13.250 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.78607.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.73146.log.gz 2024-03-24T00:50:13.250 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61400.log 2024-03-24T00:50:13.250 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.78607.log.gz 2024-03-24T00:50:13.250 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75660.log 2024-03-24T00:50:13.250 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60725.log 2024-03-24T00:50:13.251 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.75660.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61400.log.gz 2024-03-24T00:50:13.251 INFO:teuthology.orchestra.run.smithi067.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.63141.log.gz 2024-03-24T00:50:13.251 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52408.log 2024-03-24T00:50:13.251 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82740.log 2024-03-24T00:50:13.251 INFO:teuthology.orchestra.run.smithi067.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.75660.log.gz 2024-03-24T00:50:13.251 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47058.log 2024-03-24T00:50:13.251 INFO:teuthology.orchestra.run.smithi067.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.60725.log.gz 2024-03-24T00:50:13.251 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.82740.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52408.log.gz 2024-03-24T00:50:13.251 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56584.log 2024-03-24T00:50:13.251 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.82740.log.gz 2024-03-24T00:50:13.251 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46185.log 2024-03-24T00:50:13.251 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47058.log.gz 2024-03-24T00:50:13.251 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51220.log 2024-03-24T00:50:13.251 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46185.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56584.log.gz 2024-03-24T00:50:13.252 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81220.log 2024-03-24T00:50:13.252 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51220.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46185.log.gz 2024-03-24T00:50:13.252 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56999.log 2024-03-24T00:50:13.252 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51220.log.gz 2024-03-24T00:50:13.252 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.81220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81103.log 2024-03-24T00:50:13.252 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53961.log 2024-03-24T00:50:13.252 INFO:teuthology.orchestra.run.smithi067.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.81220.log.gz 2024-03-24T00:50:13.252 INFO:teuthology.orchestra.run.smithi067.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.56999.log.gz 2024-03-24T00:50:13.252 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.81103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.79542.log 2024-03-24T00:50:13.252 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.81103.log.gz 2024-03-24T00:50:13.252 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57951.log 2024-03-24T00:50:13.252 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.79542.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53961.log.gz 2024-03-24T00:50:13.252 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.76905.log -- replaced with /var/log/ceph/ceph-client.0.79542.log.gz 2024-03-24T00:50:13.252 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-24T00:50:13.252 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47328.log 2024-03-24T00:50:13.253 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.57951.log.gz 2024-03-24T00:50:13.253 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.76905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54630.log 2024-03-24T00:50:13.253 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.76905.log.gz 2024-03-24T00:50:13.253 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46859.log 2024-03-24T00:50:13.253 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54630.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47328.log.gz 2024-03-24T00:50:13.253 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73272.log 2024-03-24T00:50:13.253 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54630.log.gz 2024-03-24T00:50:13.253 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56392.log 2024-03-24T00:50:13.253 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46859.log.gz 2024-03-24T00:50:13.253 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.73272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63832.log 2024-03-24T00:50:13.253 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56392.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.73272.log.gz 2024-03-24T00:50:13.253 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63015.log 2024-03-24T00:50:13.253 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.56392.log.gz 2024-03-24T00:50:13.253 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53013.log 2024-03-24T00:50:13.254 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.63832.log.gz 2024-03-24T00:50:13.254 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55643.log 2024-03-24T00:50:13.254 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53013.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.63015.log.gz 2024-03-24T00:50:13.254 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49393.log 2024-03-24T00:50:13.254 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53013.log.gz 2024-03-24T00:50:13.254 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81811.log 2024-03-24T00:50:13.254 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.55643.log.gz 2024-03-24T00:50:13.254 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52450.log 2024-03-24T00:50:13.254 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.81811.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49393.log.gz 2024-03-24T00:50:13.254 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62955.log 2024-03-24T00:50:13.254 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.81811.log.gz 2024-03-24T00:50:13.254 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45798.log 2024-03-24T00:50:13.254 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52450.log.gz 2024-03-24T00:50:13.255 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.68454.log 2024-03-24T00:50:13.255 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45798.log: gzip 87.4% -5 -- replaced with /var/log/ceph/ceph-client.admin.62955.log.gz --verbose 2024-03-24T00:50:13.255 INFO:teuthology.orchestra.run.smithi067.stderr: -- /var/log/ceph/ceph-client.admin.83305.log 2024-03-24T00:50:13.255 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45798.log.gz 2024-03-24T00:50:13.255 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.68454.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.0.68454.log.gz 2024-03-24T00:50:13.255 INFO:teuthology.orchestra.run.smithi067.stderr: /var/log/ceph/ceph-client.admin.67987.log 2024-03-24T00:50:13.255 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.83305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.56775.log 2024-03-24T00:50:13.255 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.67987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75375.log 2024-03-24T00:50:13.255 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.83305.log.gz 2024-03-24T00:50:13.255 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.56775.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.67987.log.gz 2024-03-24T00:50:13.255 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.56775.log.gz 2024-03-24T00:50:13.255 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47078.log 2024-03-24T00:50:13.255 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.75375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52845.log 2024-03-24T00:50:13.255 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47078.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.75375.log.gz 2024-03-24T00:50:13.255 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81510.log 2024-03-24T00:50:13.256 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47078.log.gz 2024-03-24T00:50:13.256 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73358.log 2024-03-24T00:50:13.256 INFO:teuthology.orchestra.run.smithi067.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.52845.log.gz 2024-03-24T00:50:13.256 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70459.log 2024-03-24T00:50:13.256 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.81510.log: /var/log/ceph/ceph-client.admin.73358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65161.log 2024-03-24T00:50:13.256 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.70459.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.73358.log.gz 2024-03-24T00:50:13.256 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72323.log 2024-03-24T00:50:13.256 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.70459.log.gz 2024-03-24T00:50:13.256 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84438.log 2024-03-24T00:50:13.256 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.65161.log.gz 2024-03-24T00:50:13.256 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.72323.log: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.81510.log.gz 2024-03-24T00:50:13.256 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50974.log 2024-03-24T00:50:13.256 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.72323.log.gz 2024-03-24T00:50:13.256 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.84438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44032.log 2024-03-24T00:50:13.257 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.84438.log.gz 2024-03-24T00:50:13.257 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80295.log 2024-03-24T00:50:13.257 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44032.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50974.log.gz 2024-03-24T00:50:13.257 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51742.log 2024-03-24T00:50:13.257 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44032.log.gz 2024-03-24T00:50:13.257 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.80295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60791.log 2024-03-24T00:50:13.257 INFO:teuthology.orchestra.run.smithi067.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.80295.log.gz 2024-03-24T00:50:13.257 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84229.log 2024-03-24T00:50:13.257 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60791.log: gzip -5 --verbose 89.3% -- /var/log/ceph/ceph-client.admin.65528.log -- replaced with /var/log/ceph/ceph-client.admin.51742.log.gz 2024-03-24T00:50:13.257 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-24T00:50:13.257 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.60791.log.gz 2024-03-24T00:50:13.257 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.84229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47454.log 2024-03-24T00:50:13.257 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.84229.log.gz 2024-03-24T00:50:13.257 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79355.log 2024-03-24T00:50:13.258 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.65528.log.gz 2024-03-24T00:50:13.258 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66542.log 2024-03-24T00:50:13.258 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.79355.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.47454.log.gz 2024-03-24T00:50:13.258 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55685.log 2024-03-24T00:50:13.258 INFO:teuthology.orchestra.run.smithi067.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.79355.log.gz 2024-03-24T00:50:13.258 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.66542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78331.log 2024-03-24T00:50:13.258 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.66542.log.gz 2024-03-24T00:50:13.258 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72407.log 2024-03-24T00:50:13.258 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.78331.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55685.log.gz 2024-03-24T00:50:13.258 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67735.log 2024-03-24T00:50:13.258 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.78331.log.gz 2024-03-24T00:50:13.258 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.72407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51826.log 2024-03-24T00:50:13.258 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.72407.log.gz 2024-03-24T00:50:13.259 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.67735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50191.log 2024-03-24T00:50:13.259 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.67735.log.gz 2024-03-24T00:50:13.259 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78541.log 2024-03-24T00:50:13.259 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50191.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.51826.log.gz 2024-03-24T00:50:13.259 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49813.log 2024-03-24T00:50:13.259 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.78541.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50191.log.gz 2024-03-24T00:50:13.259 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75172.log 2024-03-24T00:50:13.259 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51346.log 2024-03-24T00:50:13.259 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.75172.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49813.log.gz 2024-03-24T00:50:13.259 INFO:teuthology.orchestra.run.smithi067.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.78541.log.gz 2024-03-24T00:50:13.259 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60970.log 2024-03-24T00:50:13.259 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.75172.log.gz 2024-03-24T00:50:13.259 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51100.log 2024-03-24T00:50:13.259 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51346.log.gz 2024-03-24T00:50:13.259 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56248.log 2024-03-24T00:50:13.260 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51100.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.60970.log.gz 2024-03-24T00:50:13.260 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49222.log 2024-03-24T00:50:13.260 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7%/var/log/ceph/ceph-client.admin.56248.log: -- replaced with /var/log/ceph/ceph-client.admin.51100.log.gz 2024-03-24T00:50:13.260 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79722.log 2024-03-24T00:50:13.260 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.56248.log.gz 2024-03-24T00:50:13.260 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50107.log 2024-03-24T00:50:13.260 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.79722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73685.log 2024-03-24T00:50:13.260 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49222.log.gz 2024-03-24T00:50:13.260 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50107.log: 89.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.79722.log.gz 2024-03-24T00:50:13.260 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.84271.log 2024-03-24T00:50:13.260 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50107.log.gz 2024-03-24T00:50:13.260 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.73685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79211.log 2024-03-24T00:50:13.260 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.84271.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.73685.log.gz 2024-03-24T00:50:13.260 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59522.log 2024-03-24T00:50:13.261 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.79211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81313.log 2024-03-24T00:50:13.261 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.84271.log.gz 2024-03-24T00:50:13.261 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.79211.log.gz 2024-03-24T00:50:13.261 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.59522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78121.log 2024-03-24T00:50:13.261 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.81313.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59522.log.gz 2024-03-24T00:50:13.261 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83076.log 2024-03-24T00:50:13.261 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.78121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57783.log 2024-03-24T00:50:13.261 INFO:teuthology.orchestra.run.smithi067.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.81313.log.gz 2024-03-24T00:50:13.261 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.83076.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.78121.log.gz 2024-03-24T00:50:13.261 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54546.log 2024-03-24T00:50:13.261 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57783.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.83076.log.gz 2024-03-24T00:50:13.261 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55813.log 2024-03-24T00:50:13.261 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57783.log.gz 2024-03-24T00:50:13.261 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58516.log 2024-03-24T00:50:13.262 INFO:teuthology.orchestra.run.smithi067.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.54546.log.gz 2024-03-24T00:50:13.262 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54129.log 2024-03-24T00:50:13.262 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54714.log 2024-03-24T00:50:13.262 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55813.log.gz 2024-03-24T00:50:13.262 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54129.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.58516.log.gz 2024-03-24T00:50:13.262 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67819.log 2024-03-24T00:50:13.262 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54129.log.gz 2024-03-24T00:50:13.262 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45300.log 2024-03-24T00:50:13.262 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.67819.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54714.log.gz 2024-03-24T00:50:13.262 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64133.log 2024-03-24T00:50:13.262 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.67819.log.gz 2024-03-24T00:50:13.262 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45426.log 2024-03-24T00:50:13.262 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3%/var/log/ceph/ceph-client.admin.64133.log: -- replaced with /var/log/ceph/ceph-client.admin.45300.log.gz 2024-03-24T00:50:13.262 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77726.log 2024-03-24T00:50:13.263 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45426.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.64133.log.gz 2024-03-24T00:50:13.263 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50758.log 2024-03-24T00:50:13.263 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45426.log.gz 2024-03-24T00:50:13.263 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.77726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80207.log 2024-03-24T00:50:13.263 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.77726.log.gz 2024-03-24T00:50:13.263 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76003.log 2024-03-24T00:50:13.263 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.80207.log: gzip -5 88.4% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.50758.log.gz /var/log/ceph/ceph-client.admin.69635.log 2024-03-24T00:50:13.263 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-24T00:50:13.263 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.76003.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.80207.log.gz 2024-03-24T00:50:13.263 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49771.log 2024-03-24T00:50:13.263 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.76003.log.gz 2024-03-24T00:50:13.263 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.69635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71053.log 2024-03-24T00:50:13.263 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49045.log 2024-03-24T00:50:13.264 INFO:teuthology.orchestra.run.smithi067.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.69635.log.gz 2024-03-24T00:50:13.264 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.71053.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49771.log.gz 2024-03-24T00:50:13.264 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43589.log 2024-03-24T00:50:13.264 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72551.log 2024-03-24T00:50:13.264 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49045.log: /var/log/ceph/ceph-client.admin.43589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44328.log 2024-03-24T00:50:13.264 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49045.log.gz 2024-03-24T00:50:13.264 INFO:teuthology.orchestra.run.smithi067.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.71053.log.gz 2024-03-24T00:50:13.264 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.72551.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.43589.log.gz 2024-03-24T00:50:13.264 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.64428.log 2024-03-24T00:50:13.264 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.72551.log.gz 2024-03-24T00:50:13.264 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50149.log 2024-03-24T00:50:13.264 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44328.log.gz 2024-03-24T00:50:13.264 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.64428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72281.log 2024-03-24T00:50:13.264 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.64428.log.gz 2024-03-24T00:50:13.264 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77115.log 2024-03-24T00:50:13.265 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.72281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37727.log 2024-03-24T00:50:13.265 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50149.log.gz 2024-03-24T00:50:13.265 INFO:teuthology.orchestra.run.smithi067.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.72281.log.gz 2024-03-24T00:50:13.265 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.77115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52669.log 2024-03-24T00:50:13.265 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.77115.log.gz 2024-03-24T00:50:13.265 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46722.log 2024-03-24T00:50:13.265 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55260.log 2024-03-24T00:50:13.265 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.37727.log.gz 2024-03-24T00:50:13.265 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52669.log.gz 2024-03-24T00:50:13.265 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53139.log 2024-03-24T00:50:13.265 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46722.log.gz 2024-03-24T00:50:13.265 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71539.log 2024-03-24T00:50:13.265 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53139.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55260.log.gz 2024-03-24T00:50:13.265 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67382.log 2024-03-24T00:50:13.266 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.53139.log.gz 2024-03-24T00:50:13.266 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.71539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75961.log 2024-03-24T00:50:13.266 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.67382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54966.log 2024-03-24T00:50:13.266 INFO:teuthology.orchestra.run.smithi067.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.71539.log.gz 2024-03-24T00:50:13.266 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.75961.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.67382.log.gz 2024-03-24T00:50:13.266 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60928.log 2024-03-24T00:50:13.266 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49477.log 2024-03-24T00:50:13.266 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54966.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.75961.log.gz 2024-03-24T00:50:13.266 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67945.log 2024-03-24T00:50:13.266 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54966.log.gz 2024-03-24T00:50:13.266 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49477.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60928.log.gz 2024-03-24T00:50:13.266 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45714.log 2024-03-24T00:50:13.266 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.67945.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49477.log.gz 2024-03-24T00:50:13.266 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52366.log 2024-03-24T00:50:13.266 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.67945.log.gz 2024-03-24T00:50:13.267 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67605.log 2024-03-24T00:50:13.267 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4%/var/log/ceph/ceph-client.admin.52366.log: -- replaced with /var/log/ceph/ceph-client.admin.45714.log.gz 2024-03-24T00:50:13.267 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81146.log 2024-03-24T00:50:13.267 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.67605.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52366.log.gz 2024-03-24T00:50:13.267 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46860.log 2024-03-24T00:50:13.267 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.67605.log.gz 2024-03-24T00:50:13.267 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.81146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76049.log 2024-03-24T00:50:13.267 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49003.log 2024-03-24T00:50:13.267 INFO:teuthology.orchestra.run.smithi067.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.81146.log.gz 2024-03-24T00:50:13.267 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.76049.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46860.log.gz 2024-03-24T00:50:13.267 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61316.log 2024-03-24T00:50:13.267 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.76049.log.gz 2024-03-24T00:50:13.267 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58306.log 2024-03-24T00:50:13.267 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49003.log.gz 2024-03-24T00:50:13.268 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68893.log 2024-03-24T00:50:13.268 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58306.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.61316.log.gz 2024-03-24T00:50:13.268 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66113.log 2024-03-24T00:50:13.268 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58306.log.gz 2024-03-24T00:50:13.268 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.68893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72113.log 2024-03-24T00:50:13.268 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.68893.log.gz 2024-03-24T00:50:13.268 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.66113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64175.log 2024-03-24T00:50:13.268 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.72113.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.66113.log.gz 2024-03-24T00:50:13.268 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84398.log 2024-03-24T00:50:13.268 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.72113.log.gz 2024-03-24T00:50:13.268 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70417.log 2024-03-24T00:50:13.268 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.84398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65269.log 2024-03-24T00:50:13.268 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.70417.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.64175.log.gz 2024-03-24T00:50:13.268 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.84398.log.gz 2024-03-24T00:50:13.268 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60359.log 2024-03-24T00:50:13.268 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51388.log 2024-03-24T00:50:13.268 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.70417.log.gz 2024-03-24T00:50:13.269 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65269.log: /var/log/ceph/ceph-client.admin.60359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61358.log 2024-03-24T00:50:13.269 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70375.log 2024-03-24T00:50:13.269 INFO:teuthology.orchestra.run.smithi067.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.65269.log.gz 2024-03-24T00:50:13.269 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51388.log.gz 2024-03-24T00:50:13.269 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70019.log 2024-03-24T00:50:13.269 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61358.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.60359.log.gz 2024-03-24T00:50:13.269 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.70375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48570.log 2024-03-24T00:50:13.269 INFO:teuthology.orchestra.run.smithi067.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.61358.log.gz 2024-03-24T00:50:13.269 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.70019.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.70375.log.gz 2024-03-24T00:50:13.269 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50890.log 2024-03-24T00:50:13.269 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53307.log 2024-03-24T00:50:13.269 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48570.log.gz 2024-03-24T00:50:13.269 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50890.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.70019.log.gz 2024-03-24T00:50:13.269 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71893.log 2024-03-24T00:50:13.269 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54924.log 2024-03-24T00:50:13.269 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50890.log.gz 2024-03-24T00:50:13.270 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53307.log: /var/log/ceph/ceph-client.admin.71893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83539.log 2024-03-24T00:50:13.270 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53307.log.gz 2024-03-24T00:50:13.270 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54924.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.71893.log.gz 2024-03-24T00:50:13.270 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52803.log 2024-03-24T00:50:13.270 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.83539.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54924.log.gz 2024-03-24T00:50:13.270 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62745.log 2024-03-24T00:50:13.270 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.83539.log.gz 2024-03-24T00:50:13.270 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53457.log 2024-03-24T00:50:13.270 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 88.1% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.52803.log.gz 2024-03-24T00:50:13.270 INFO:teuthology.orchestra.run.smithi067.stderr: /var/log/ceph/ceph-client.admin.60101.log 2024-03-24T00:50:13.270 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62745.log: /var/log/ceph/ceph-client.admin.53457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79289.log 2024-03-24T00:50:13.270 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.62745.log.gz 2024-03-24T00:50:13.270 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53457.log.gz 2024-03-24T00:50:13.271 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63713.log 2024-03-24T00:50:13.271 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.79289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50674.log 2024-03-24T00:50:13.271 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63713.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.60101.log.gz 2024-03-24T00:50:13.271 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74539.log 2024-03-24T00:50:13.271 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.63713.log.gz 2024-03-24T00:50:13.271 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50674.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.79289.log.gz 2024-03-24T00:50:13.271 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47832.log 2024-03-24T00:50:13.271 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50674.log.gz 2024-03-24T00:50:13.271 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.74539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43734.log 2024-03-24T00:50:13.271 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.74539.log.gz 2024-03-24T00:50:13.271 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68029.log 2024-03-24T00:50:13.271 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70235.log 2024-03-24T00:50:13.271 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47832.log.gz 2024-03-24T00:50:13.271 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.68029.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43734.log.gz 2024-03-24T00:50:13.272 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49266.log 2024-03-24T00:50:13.272 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.68029.log.gz 2024-03-24T00:50:13.272 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.70235.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-24T00:50:13.272 INFO:teuthology.orchestra.run.smithi067.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.70235.log.gz 2024-03-24T00:50:13.272 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76863.log 2024-03-24T00:50:13.272 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph.audit.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49266.log.gz 2024-03-24T00:50:13.272 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71605.log 2024-03-24T00:50:13.272 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.76863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76653.log 2024-03-24T00:50:13.272 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.71605.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.76863.log.gz 2024-03-24T00:50:13.272 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68816.log 2024-03-24T00:50:13.272 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.71605.log.gz 2024-03-24T00:50:13.272 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76821.log 2024-03-24T00:50:13.272 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.76653.log: /var/log/ceph/ceph-client.admin.68816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43990.log 2024-03-24T00:50:13.272 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.76653.log.gz 2024-03-24T00:50:13.272 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.76821.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.68816.log.gz 2024-03-24T00:50:13.273 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79848.log 2024-03-24T00:50:13.273 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5/var/log/ceph/ceph-client.admin.43990.log: --verbose -- /var/log/ceph/ceph-client.admin.72365.log 2024-03-24T00:50:13.273 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.76821.log.gz 2024-03-24T00:50:13.273 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.79848.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43990.log.gz 2024-03-24T00:50:13.273 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58857.log 2024-03-24T00:50:13.273 INFO:teuthology.orchestra.run.smithi067.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.79848.log.gz 2024-03-24T00:50:13.273 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46189.log 2024-03-24T00:50:13.273 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.72365.log: /var/log/ceph/ceph-client.admin.58857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51658.log 2024-03-24T00:50:13.273 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.72365.log.gz 2024-03-24T00:50:13.273 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46189.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.58857.log.gz 2024-03-24T00:50:13.273 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48122.log 2024-03-24T00:50:13.273 INFO:teuthology.orchestra.run.smithi067.stderr: 89.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-24T00:50:13.273 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0%/var/log/ceph/ceph-client.admin.51658.log: -- replaced with /var/log/ceph/ceph-client.admin.46189.log.gz 2024-03-24T00:50:13.273 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60617.log 2024-03-24T00:50:13.273 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48122.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51658.log.gz 2024-03-24T00:50:13.273 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67647.log 2024-03-24T00:50:13.274 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48122.log.gz 2024-03-24T00:50:13.274 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73555.log 2024-03-24T00:50:13.274 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.67647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71011.log 2024-03-24T00:50:13.274 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.67647.log.gz 2024-03-24T00:50:13.274 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.73555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71809.log 2024-03-24T00:50:13.274 INFO:teuthology.orchestra.run.smithi067.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.60617.log.gz 2024-03-24T00:50:13.274 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.73555.log.gz -5 2024-03-24T00:50:13.274 INFO:teuthology.orchestra.run.smithi067.stderr: --verbose -- /var/log/ceph/ceph-client.admin.72636.log 2024-03-24T00:50:13.274 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.71011.log: /var/log/ceph/ceph-client.admin.71809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47370.log 2024-03-24T00:50:13.274 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.71011.log.gz 2024-03-24T00:50:13.274 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.71809.log.gz/var/log/ceph/ceph-client.admin.72636.log: 2024-03-24T00:50:13.274 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62703.log 2024-03-24T00:50:13.274 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47370.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.72636.log.gz 2024-03-24T00:50:13.274 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75726.log 2024-03-24T00:50:13.275 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47370.log.gz 2024-03-24T00:50:13.275 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46186.log 2024-03-24T00:50:13.275 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.62703.log.gz 2024-03-24T00:50:13.275 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.75726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80857.log 2024-03-24T00:50:13.275 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46186.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.75726.log.gz 2024-03-24T00:50:13.275 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53265.log 2024-03-24T00:50:13.275 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.80857.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46186.log.gz 2024-03-24T00:50:13.275 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55008.log 2024-03-24T00:50:13.275 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.80857.log.gz 2024-03-24T00:50:13.275 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69033.log 2024-03-24T00:50:13.275 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55008.log: 87.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.53265.log.gz -5 2024-03-24T00:50:13.275 INFO:teuthology.orchestra.run.smithi067.stderr: --verbose -- /var/log/ceph/ceph-client.admin.78289.log 2024-03-24T00:50:13.275 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55008.log.gz 2024-03-24T00:50:13.275 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.69033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65987.log 2024-03-24T00:50:13.276 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.78289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52492.log 2024-03-24T00:50:13.276 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65987.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.78289.log.gz 2024-03-24T00:50:13.276 INFO:teuthology.orchestra.run.smithi067.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.69033.log.gz 2024-03-24T00:50:13.276 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43412.log 2024-03-24T00:50:13.276 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.65987.log.gz 2024-03-24T00:50:13.276 INFO:teuthology.orchestra.run.smithi067.stderr:gzip/var/log/ceph/ceph-client.admin.52492.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.83917.log 2024-03-24T00:50:13.276 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43412.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52492.log.gz 2024-03-24T00:50:13.276 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56946.log 2024-03-24T00:50:13.276 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.83917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83263.log 2024-03-24T00:50:13.276 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43412.log.gz 2024-03-24T00:50:13.276 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56946.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.83917.log.gz 2024-03-24T00:50:13.276 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73961.log 2024-03-24T00:50:13.276 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.83263.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.56946.log.gz 2024-03-24T00:50:13.276 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83136.log 2024-03-24T00:50:13.277 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.73961.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.83263.log.gz 2024-03-24T00:50:13.277 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63626.log 2024-03-24T00:50:13.277 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.73961.log.gz 2024-03-24T00:50:13.277 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.83136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66290.log 2024-03-24T00:50:13.277 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.83136.log.gz 2024-03-24T00:50:13.277 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43649.log 2024-03-24T00:50:13.277 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.66290.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.63626.log.gz 2024-03-24T00:50:13.277 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82866.log 2024-03-24T00:50:13.277 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.66290.log.gz 2024-03-24T00:50:13.277 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51304.log 2024-03-24T00:50:13.277 INFO:teuthology.orchestra.run.smithi067.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43649.log.gz 2024-03-24T00:50:13.277 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.82866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52036.log 2024-03-24T00:50:13.277 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51304.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.82866.log.gz 2024-03-24T00:50:13.277 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59630.log 2024-03-24T00:50:13.277 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51304.log.gz 2024-03-24T00:50:13.278 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71977.log 2024-03-24T00:50:13.278 INFO:teuthology.orchestra.run.smithi067.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52036.log.gz 2024-03-24T00:50:13.278 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.59630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76176.log 2024-03-24T00:50:13.278 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.71977.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.59630.log.gz 2024-03-24T00:50:13.278 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80249.log 2024-03-24T00:50:13.278 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.71977.log.gz 2024-03-24T00:50:13.279 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.76176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58348.log 2024-03-24T00:50:13.279 INFO:teuthology.orchestra.run.smithi067.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.76176.log.gz 2024-03-24T00:50:13.279 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.80249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61671.log 2024-03-24T00:50:13.279 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58348.log: gzip -5 --verbose 88.1% -- /var/log/ceph/ceph-client.admin.57479.log -- replaced with /var/log/ceph/ceph-client.admin.80249.log.gz 2024-03-24T00:50:13.279 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-24T00:50:13.279 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61671.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58348.log.gz 2024-03-24T00:50:13.280 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62661.log 2024-03-24T00:50:13.280 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.61671.log.gz 2024-03-24T00:50:13.280 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66584.log 2024-03-24T00:50:13.280 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62661.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.57479.log.gz 2024-03-24T00:50:13.280 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54363.log 2024-03-24T00:50:13.280 INFO:teuthology.orchestra.run.smithi067.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.62661.log.gz 2024-03-24T00:50:13.280 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.66584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61148.log 2024-03-24T00:50:13.280 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.66584.log.gz 2024-03-24T00:50:13.281 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66416.log 2024-03-24T00:50:13.281 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50470.log 2024-03-24T00:50:13.281 INFO:teuthology.orchestra.run.smithi067.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.54363.log.gz 2024-03-24T00:50:13.281 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.66416.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.61148.log.gz 2024-03-24T00:50:13.281 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83034.log 2024-03-24T00:50:13.281 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50470.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.66416.log.gz 2024-03-24T00:50:13.282 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61628.log 2024-03-24T00:50:13.282 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.83034.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50470.log.gz 2024-03-24T00:50:13.282 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76779.log 2024-03-24T00:50:13.282 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.83034.log.gz 2024-03-24T00:50:13.282 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72491.log 2024-03-24T00:50:13.282 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.76779.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61628.log.gzgzip 2024-03-24T00:50:13.283 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.58815.log 2024-03-24T00:50:13.283 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.76779.log.gz 2024-03-24T00:50:13.283 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.72491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46861.log 2024-03-24T00:50:13.283 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.72491.log.gz 2024-03-24T00:50:13.283 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5/var/log/ceph/ceph-client.admin.58815.log: --verbose -- /var/log/ceph/ceph-client.admin.57741.log 2024-03-24T00:50:13.284 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.80612.log 2024-03-24T00:50:13.284 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.58815.log.gz 2024-03-24T00:50:13.284 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46861.log.gz 2024-03-24T00:50:13.284 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82394.log 2024-03-24T00:50:13.284 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.57741.log.gz 2024-03-24T00:50:13.284 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.80612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77768.log 2024-03-24T00:50:13.285 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.80612.log.gz 2024-03-24T00:50:13.285 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.82394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57074.log 2024-03-24T00:50:13.285 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.77768.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.82394.log.gz 2024-03-24T00:50:13.285 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84480.log 2024-03-24T00:50:13.285 INFO:teuthology.orchestra.run.smithi067.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.77768.log.gz 2024-03-24T00:50:13.285 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52281.log 2024-03-24T00:50:13.286 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.84480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72762.log 2024-03-24T00:50:13.286 INFO:teuthology.orchestra.run.smithi067.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.57074.log.gz 2024-03-24T00:50:13.286 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.84480.log.gz 2024-03-24T00:50:13.286 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50344.log 2024-03-24T00:50:13.286 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52281.log.gz 2024-03-24T00:50:13.286 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.72762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57563.log 2024-03-24T00:50:13.287 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50344.log: gzip -5 --verbose -- 88.4% /var/log/ceph/ceph-client.admin.77031.log 2024-03-24T00:50:13.287 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.72762.log.gz 2024-03-24T00:50:13.287 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50344.log.gz 2024-03-24T00:50:13.287 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66915.log 2024-03-24T00:50:13.287 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57563.log.gz 2024-03-24T00:50:13.287 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.77031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65354.log 2024-03-24T00:50:13.288 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.66915.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.77031.log.gz 2024-03-24T00:50:13.288 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44160.log 2024-03-24T00:50:13.288 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2%/var/log/ceph/ceph-client.admin.65354.log: -- replaced with /var/log/ceph/ceph-client.admin.66915.log.gz 2024-03-24T00:50:13.288 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60886.log 2024-03-24T00:50:13.288 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67340.log 2024-03-24T00:50:13.288 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44160.log.gz 2024-03-24T00:50:13.289 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.74152.log 2024-03-24T00:50:13.289 INFO:teuthology.orchestra.run.smithi067.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.65354.log.gz 2024-03-24T00:50:13.289 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46008.log 2024-03-24T00:50:13.289 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.60886.log.gz 2024-03-24T00:50:13.289 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.67340.log: /var/log/ceph/ceph-client.0.74152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46513.log 2024-03-24T00:50:13.289 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.74152.log.gz 2024-03-24T00:50:13.290 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.67340.log.gz 2024-03-24T00:50:13.290 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49351.log 2024-03-24T00:50:13.290 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46513.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46008.log.gz 2024-03-24T00:50:13.290 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83623.log 2024-03-24T00:50:13.290 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46513.log.gz 2024-03-24T00:50:13.291 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55092.log 2024-03-24T00:50:13.291 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49351.log.gz 2024-03-24T00:50:13.291 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.83623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72197.log 2024-03-24T00:50:13.291 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55092.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.83623.log.gz 2024-03-24T00:50:13.291 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46188.log 2024-03-24T00:50:13.291 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.72197.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55092.log.gz 2024-03-24T00:50:13.292 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.71306.log 2024-03-24T00:50:13.292 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.72197.log.gz 2024-03-24T00:50:13.292 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46190.log 2024-03-24T00:50:13.292 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.71306.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46188.log.gz 2024-03-24T00:50:13.292 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.0.71306.log.gz 2024-03-24T00:50:13.293 INFO:teuthology.orchestra.run.smithi067.stderr: /var/log/ceph/ceph-client.admin.55176.log 2024-03-24T00:50:13.293 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69350.log 2024-03-24T00:50:13.293 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68955.log 2024-03-24T00:50:13.293 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46190.log.gz 2024-03-24T00:50:13.293 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.69350.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55176.log.gz 2024-03-24T00:50:13.293 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83959.log 2024-03-24T00:50:13.294 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.69350.log.gz 2024-03-24T00:50:13.294 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.68955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45924.log 2024-03-24T00:50:13.294 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.83959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.78798.log 2024-03-24T00:50:13.294 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.68955.log.gz 2024-03-24T00:50:13.294 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45924.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.83959.log.gz 2024-03-24T00:50:13.294 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81669.log 2024-03-24T00:50:13.295 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.78798.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45924.log.gz 2024-03-24T00:50:13.295 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66206.log 2024-03-24T00:50:13.295 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.78798.log.gz 2024-03-24T00:50:13.295 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.81669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77385.log 2024-03-24T00:50:13.295 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.81669.log.gz 2024-03-24T00:50:13.295 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5/var/log/ceph/ceph-client.admin.66206.log: --verbose -- /var/log/ceph/ceph-client.admin.73853.log 2024-03-24T00:50:13.296 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.77385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49645.log 2024-03-24T00:50:13.296 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.66206.log.gz 2024-03-24T00:50:13.296 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.77385.log.gz 2024-03-24T00:50:13.296 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.73853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58035.log 2024-03-24T00:50:13.296 INFO:teuthology.orchestra.run.smithi067.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.73853.log.gz 2024-03-24T00:50:13.297 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71453.log 2024-03-24T00:50:13.297 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58035.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49645.log.gz 2024-03-24T00:50:13.297 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75130.log 2024-03-24T00:50:13.297 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.71453.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.58035.log.gz 2024-03-24T00:50:13.297 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65945.log 2024-03-24T00:50:13.297 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.75130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72155.log 2024-03-24T00:50:13.298 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65945.log: 90.7% 88.1% -- replaced with /var/log/ceph/ceph-client.admin.75130.log.gz -- replaced with /var/log/ceph/ceph-client.admin.71453.log.gz 2024-03-24T00:50:13.298 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-24T00:50:13.298 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65203.log 2024-03-24T00:50:13.298 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.65945.log.gz 2024-03-24T00:50:13.298 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.72155.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-24T00:50:13.298 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65203.log: gzip -5 --verbose -- 89.3% /var/log/ceph/ceph-client.admin.61274.log 2024-03-24T00:50:13.299 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.72155.log.gz 2024-03-24T00:50:13.299 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61484.log 2024-03-24T00:50:13.299 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61274.log: 90.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.65203.log.gz -5 2024-03-24T00:50:13.299 INFO:teuthology.orchestra.run.smithi067.stderr: --verbose -- /var/log/ceph/ceph-client.admin.61442.log 2024-03-24T00:50:13.299 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61274.log.gz 2024-03-24T00:50:13.299 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47412.log 2024-03-24T00:50:13.300 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61484.log: /var/log/ceph/ceph-client.admin.61442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84127.log 2024-03-24T00:50:13.300 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61484.log.gz 2024-03-24T00:50:13.300 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47412.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.61442.log.gz 2024-03-24T00:50:13.300 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66029.log 2024-03-24T00:50:13.300 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47412.log.gz 2024-03-24T00:50:13.301 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.84127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79127.log 2024-03-24T00:50:13.301 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.66029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76436.log 2024-03-24T00:50:13.301 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.84127.log.gz 2024-03-24T00:50:13.301 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.79127.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.66029.log.gz 2024-03-24T00:50:13.301 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59336.log 2024-03-24T00:50:13.301 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.79127.log.gz 2024-03-24T00:50:13.302 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66500.log/var/log/ceph/ceph-client.admin.76436.log: 2024-03-24T00:50:13.302 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.59336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67903.log 2024-03-24T00:50:13.302 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59336.log.gz 2024-03-24T00:50:13.302 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.66500.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.76436.log.gz 2024-03-24T00:50:13.302 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45158.log 2024-03-24T00:50:13.302 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.66500.log.gz 2024-03-24T00:50:13.303 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79169.log 2024-03-24T00:50:13.303 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.67903.log: /var/log/ceph/ceph-client.admin.45158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77949.log 2024-03-24T00:50:13.303 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.67903.log.gz 2024-03-24T00:50:13.303 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45158.log.gz 2024-03-24T00:50:13.303 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.79169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61054.log 2024-03-24T00:50:13.303 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.79169.log.gz 2024-03-24T00:50:13.304 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70621.log 2024-03-24T00:50:13.304 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.77949.log: /var/log/ceph/ceph-client.admin.61054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45966.log 2024-03-24T00:50:13.304 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.77949.log.gz 2024-03-24T00:50:13.304 INFO:teuthology.orchestra.run.smithi067.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.61054.log.gz 2024-03-24T00:50:13.304 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.70621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83791.log 2024-03-24T00:50:13.305 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56476.log 2024-03-24T00:50:13.305 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45966.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.70621.log.gz 2024-03-24T00:50:13.305 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.83791.log: 88.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.45966.log.gz 2024-03-24T00:50:13.305 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.0.78794.log 2024-03-24T00:50:13.305 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.83791.log.gz 2024-03-24T00:50:13.305 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58432.log 2024-03-24T00:50:13.306 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56476.log: /var/log/ceph/ceph-client.0.78794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52534.log 2024-03-24T00:50:13.306 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.78794.log.gz 2024-03-24T00:50:13.306 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56476.log.gz 2024-03-24T00:50:13.306 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80983.log 2024-03-24T00:50:13.306 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52534.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.58432.log.gz 2024-03-24T00:50:13.306 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77241.log 2024-03-24T00:50:13.307 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68732.log 2024-03-24T00:50:13.307 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52534.log.gz 2024-03-24T00:50:13.307 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.80983.log: /var/log/ceph/ceph-client.admin.77241.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.80983.log.gz 2024-03-24T00:50:13.307 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64634.log 2024-03-24T00:50:13.307 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.77241.log.gz 2024-03-24T00:50:13.308 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58264.log 2024-03-24T00:50:13.308 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.68732.log: /var/log/ceph/ceph-client.admin.64634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66999.log 2024-03-24T00:50:13.308 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.68732.log.gz 2024-03-24T00:50:13.308 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58264.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.64634.log.gz 2024-03-24T00:50:13.308 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50428.log 2024-03-24T00:50:13.308 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.58264.log.gz 2024-03-24T00:50:13.309 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.66999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51262.log 2024-03-24T00:50:13.309 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47120.log 2024-03-24T00:50:13.309 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3%/var/log/ceph/ceph-client.admin.50428.log: -- replaced with /var/log/ceph/ceph-client.admin.66999.log.gz 2024-03-24T00:50:13.309 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51262.log: 88.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.50428.log.gz -5 2024-03-24T00:50:13.309 INFO:teuthology.orchestra.run.smithi067.stderr: --verbose -- /var/log/ceph/ceph-client.admin.78247.log 2024-03-24T00:50:13.309 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51262.log.gz 2024-03-24T00:50:13.310 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80355.log 2024-03-24T00:50:13.310 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.78247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73314.log 2024-03-24T00:50:13.310 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47120.log.gz 2024-03-24T00:50:13.310 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.80355.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.78247.log.gz 2024-03-24T00:50:13.310 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61965.log 2024-03-24T00:50:13.310 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45342.log 2024-03-24T00:50:13.311 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.73314.log: /var/log/ceph/ceph-client.admin.61965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75489.log 2024-03-24T00:50:13.311 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.73314.log.gz 2024-03-24T00:50:13.311 INFO:teuthology.orchestra.run.smithi067.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.80355.log.gz 2024-03-24T00:50:13.311 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.61965.log.gz 2024-03-24T00:50:13.311 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70965.log 2024-03-24T00:50:13.312 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.75489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55601.log 2024-03-24T00:50:13.312 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45342.log.gz 2024-03-24T00:50:13.312 INFO:teuthology.orchestra.run.smithi067.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.75489.log.gz 2024-03-24T00:50:13.312 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.70965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.65761.log 2024-03-24T00:50:13.312 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.70965.log.gz 2024-03-24T00:50:13.312 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45200.log 2024-03-24T00:50:13.313 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73601.log 2024-03-24T00:50:13.313 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.65761.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55601.log.gz 2024-03-24T00:50:13.313 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.65761.log.gz 2024-03-24T00:50:13.313 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47226.log 2024-03-24T00:50:13.313 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45200.log.gz 2024-03-24T00:50:13.314 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81552.log 2024-03-24T00:50:13.314 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.73601.log: /var/log/ceph/ceph-client.admin.47226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77343.log 2024-03-24T00:50:13.314 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.73601.log.gz 2024-03-24T00:50:13.314 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.81552.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47226.log.gz 2024-03-24T00:50:13.314 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49855.log 2024-03-24T00:50:13.314 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.77343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.74867.log 2024-03-24T00:50:13.315 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49855.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.77343.log.gz 2024-03-24T00:50:13.315 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77684.log 2024-03-24T00:50:13.315 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.74867.log: 87.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.49855.log.gz -5 2024-03-24T00:50:13.315 INFO:teuthology.orchestra.run.smithi067.stderr: --verbose -- /var/log/ceph/ceph-client.admin.62140.log 2024-03-24T00:50:13.315 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.74867.log.gz 2024-03-24T00:50:13.315 INFO:teuthology.orchestra.run.smithi067.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.81552.log.gz 2024-03-24T00:50:13.316 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.77684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55727.log 2024-03-24T00:50:13.316 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62140.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.77684.log.gz 2024-03-24T00:50:13.316 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61190.log 2024-03-24T00:50:13.316 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.62140.log.gz 2024-03-24T00:50:13.316 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62615.log 2024-03-24T00:50:13.317 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55727.log: /var/log/ceph/ceph-client.admin.61190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71647.log 2024-03-24T00:50:13.317 INFO:teuthology.orchestra.run.smithi067.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55727.log.gz 2024-03-24T00:50:13.317 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62615.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61190.log.gz 2024-03-24T00:50:13.317 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68113.log 2024-03-24T00:50:13.317 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58077.log 2024-03-24T00:50:13.317 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.71647.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.62615.log.gz 2024-03-24T00:50:13.318 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.68113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77642.log 2024-03-24T00:50:13.318 INFO:teuthology.orchestra.run.smithi067.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.68113.log.gz 2024-03-24T00:50:13.318 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69785.log 2024-03-24T00:50:13.318 INFO:teuthology.orchestra.run.smithi067.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.71647.log.gz 2024-03-24T00:50:13.318 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74420.log 2024-03-24T00:50:13.319 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.77642.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.58077.log.gz 2024-03-24T00:50:13.319 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.69785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57867.log 2024-03-24T00:50:13.319 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.77642.log.gz 2024-03-24T00:50:13.319 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.74420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64045.log 2024-03-24T00:50:13.319 INFO:teuthology.orchestra.run.smithi067.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.69785.log.gz 2024-03-24T00:50:13.319 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75298.log 2024-03-24T00:50:13.320 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.74420.log.gz 2024-03-24T00:50:13.320 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62787.log 2024-03-24T00:50:13.320 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.57867.log.gz 2024-03-24T00:50:13.320 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.75298.log: 89.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.64045.log.gz 2024-03-24T00:50:13.320 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.76695.log 2024-03-24T00:50:13.320 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.75298.log.gz 2024-03-24T00:50:13.321 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74601.log 2024-03-24T00:50:13.321 INFO:teuthology.orchestra.run.smithi067.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.62787.log.gz 2024-03-24T00:50:13.321 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52929.log 2024-03-24T00:50:13.321 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.76695.log: /var/log/ceph/ceph-client.admin.74601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56434.log 2024-03-24T00:50:13.321 INFO:teuthology.orchestra.run.smithi067.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.76695.log.gz 2024-03-24T00:50:13.322 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52929.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.74601.log.gz 2024-03-24T00:50:13.322 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63207.log 2024-03-24T00:50:13.322 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49173.log 2024-03-24T00:50:13.322 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52929.log.gz 2024-03-24T00:50:13.322 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56434.log: /var/log/ceph/ceph-client.admin.63207.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.56434.log.gz 2024-03-24T00:50:13.322 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76568.log 2024-03-24T00:50:13.323 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.63207.log.gz 2024-03-24T00:50:13.323 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53391.log 2024-03-24T00:50:13.323 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49173.log: /var/log/ceph/ceph-client.admin.76568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51952.log 2024-03-24T00:50:13.323 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49173.log.gz 2024-03-24T00:50:13.323 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53391.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.76568.log.gz 2024-03-24T00:50:13.323 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51910.log 2024-03-24T00:50:13.324 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82950.log 2024-03-24T00:50:13.324 INFO:teuthology.orchestra.run.smithi067.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.53391.log.gz 2024-03-24T00:50:13.324 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51910.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51952.log.gz 2024-03-24T00:50:13.324 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49309.log 2024-03-24T00:50:13.324 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.48441.log 2024-03-24T00:50:13.325 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51910.log.gz 2024-03-24T00:50:13.325 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.82950.log: /var/log/ceph/ceph-client.admin.49309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49519.log 2024-03-24T00:50:13.325 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.82950.log.gz 2024-03-24T00:50:13.325 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.48441.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49309.log.gz 2024-03-24T00:50:13.325 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75809.log 2024-03-24T00:50:13.325 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.48441.log.gz 2024-03-24T00:50:13.325 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63991.log 2024-03-24T00:50:13.326 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.75809.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49519.log.gz 2024-03-24T00:50:13.326 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56122.log 2024-03-24T00:50:13.326 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63670.log 2024-03-24T00:50:13.326 INFO:teuthology.orchestra.run.smithi067.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.75809.log.gz 2024-03-24T00:50:13.326 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.63991.log.gz 2024-03-24T00:50:13.326 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67041.log 2024-03-24T00:50:13.326 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81379.log 2024-03-24T00:50:13.327 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56122.log.gz 2024-03-24T00:50:13.327 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.67041.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.63670.log.gz 2024-03-24T00:50:13.327 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.48870.log 2024-03-24T00:50:13.327 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58474.log 2024-03-24T00:50:13.327 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.81379.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.67041.log.gz 2024-03-24T00:50:13.327 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.2.48870.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.81379.log.gz 2024-03-24T00:50:13.328 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45510.log 2024-03-24T00:50:13.328 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.2.48870.log.gz 2024-03-24T00:50:13.328 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47622.log 2024-03-24T00:50:13.328 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58474.log: /var/log/ceph/ceph-client.admin.45510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47916.log 2024-03-24T00:50:13.328 INFO:teuthology.orchestra.run.smithi067.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.58474.log.gz 2024-03-24T00:50:13.328 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45510.log.gz 2024-03-24T00:50:13.328 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80097.log 2024-03-24T00:50:13.328 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47622.log.gz 2024-03-24T00:50:13.328 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5/var/log/ceph/ceph-client.admin.47916.log: --verbose -- /var/log/ceph/ceph-client.admin.46855.log 2024-03-24T00:50:13.329 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.80097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50800.log 2024-03-24T00:50:13.329 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47916.log.gz 2024-03-24T00:50:13.329 INFO:teuthology.orchestra.run.smithi067.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.80097.log.gz 2024-03-24T00:50:13.329 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.71310.log 2024-03-24T00:50:13.329 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-24T00:50:13.329 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50800.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46855.log.gz 2024-03-24T00:50:13.329 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.71310.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50800.log.gz 2024-03-24T00:50:13.329 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.71310.log.gz 2024-03-24T00:50:13.329 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75907.log 2024-03-24T00:50:13.329 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54798.log 2024-03-24T00:50:13.329 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-osd.3.log: /var/log/ceph/ceph-client.admin.75907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46518.log 2024-03-24T00:50:13.329 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.75907.log.gz 2024-03-24T00:50:13.329 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73811.log 2024-03-24T00:50:13.330 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80153.log 2024-03-24T00:50:13.330 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54798.log.gz 2024-03-24T00:50:13.330 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46518.log: /var/log/ceph/ceph-client.admin.73811.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46518.log.gzgzip 2024-03-24T00:50:13.330 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.66332.log 2024-03-24T00:50:13.330 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50590.log 2024-03-24T00:50:13.330 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.80153.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.73811.log.gz 2024-03-24T00:50:13.331 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.66332.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.80153.log.gz 2024-03-24T00:50:13.331 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72239.log 2024-03-24T00:50:13.331 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.66332.log.gz 2024-03-24T00:50:13.331 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75576.log 2024-03-24T00:50:13.331 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50590.log: /var/log/ceph/ceph-client.admin.72239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54840.log 2024-03-24T00:50:13.331 INFO:teuthology.orchestra.run.smithi067.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50590.log.gz 2024-03-24T00:50:13.332 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.75576.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.72239.log.gz 2024-03-24T00:50:13.332 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57073.log 2024-03-24T00:50:13.332 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.54840.log: /var/log/ceph/ceph-client.admin.62913.log 2024-03-24T00:50:13.332 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57073.log: gzip -5 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54840.log.gz 2024-03-24T00:50:13.332 INFO:teuthology.orchestra.run.smithi067.stderr: --verbose -- 91.0% -- replaced with /var/log/ceph/ceph-client.admin.75576.log.gz 2024-03-24T00:50:13.332 INFO:teuthology.orchestra.run.smithi067.stderr: /var/log/ceph/ceph-client.admin.55857.log 2024-03-24T00:50:13.333 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.53648.log 2024-03-24T00:50:13.333 INFO:teuthology.orchestra.run.smithi067.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.57073.log.gz 2024-03-24T00:50:13.333 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55857.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.62913.log.gz 2024-03-24T00:50:13.333 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77898.log 2024-03-24T00:50:13.333 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.53648.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55857.log.gz 2024-03-24T00:50:13.334 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.74871.log 2024-03-24T00:50:13.334 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.53648.log.gz 2024-03-24T00:50:13.334 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.77898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72449.log 2024-03-24T00:50:13.334 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.74871.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.74871.log.gz 2024-03-24T00:50:13.334 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.77898.log.gz 2024-03-24T00:50:13.334 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77157.log 2024-03-24T00:50:13.335 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82908.log 2024-03-24T00:50:13.335 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.72449.log: gzip -5/var/log/ceph/ceph-client.admin.77157.log: --verbose -- /var/log/ceph/ceph-client.admin.61923.log 2024-03-24T00:50:13.335 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.72449.log.gz 2024-03-24T00:50:13.335 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.82908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64987.log 2024-03-24T00:50:13.335 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.77157.log.gz 2024-03-24T00:50:13.335 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76989.log 2024-03-24T00:50:13.335 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.82908.log.gz 2024-03-24T00:50:13.335 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64987.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.61923.log.gz 2024-03-24T00:50:13.335 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43331.log 2024-03-24T00:50:13.335 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.76989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50512.log 2024-03-24T00:50:13.336 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.76989.log.gz 2024-03-24T00:50:13.336 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43331.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.64987.log.gz 2024-03-24T00:50:13.336 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72720.log 2024-03-24T00:50:13.336 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43331.log.gz 2024-03-24T00:50:13.336 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74336.log 2024-03-24T00:50:13.336 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51784.log 2024-03-24T00:50:13.336 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.72720.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50512.log.gz 2024-03-24T00:50:13.336 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.74336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79085.log 2024-03-24T00:50:13.336 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.72720.log.gz 2024-03-24T00:50:13.337 INFO:teuthology.orchestra.run.smithi067.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.74336.log.gz 2024-03-24T00:50:13.337 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46520.log 2024-03-24T00:50:13.337 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51784.log: /var/log/ceph/ceph-client.admin.79085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64903.log 2024-03-24T00:50:13.337 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51784.log.gz 2024-03-24T00:50:13.337 INFO:teuthology.orchestra.run.smithi067.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.79085.log.gz 2024-03-24T00:50:13.337 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.74148.log 2024-03-24T00:50:13.337 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77810.log 2024-03-24T00:50:13.338 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64903.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46520.log.gz 2024-03-24T00:50:13.338 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.74148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80899.log 2024-03-24T00:50:13.338 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.74148.log.gz 2024-03-24T00:50:13.338 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.77810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58390.log 2024-03-24T00:50:13.338 INFO:teuthology.orchestra.run.smithi067.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.64903.log.gz 2024-03-24T00:50:13.338 INFO:teuthology.orchestra.run.smithi067.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.77810.log.gz 2024-03-24T00:50:13.339 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.80899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81919.log 2024-03-24T00:50:13.339 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.80899.log.gz 2024-03-24T00:50:13.339 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54045.log 2024-03-24T00:50:13.339 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58390.log: /var/log/ceph/ceph-client.admin.81919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68648.log 2024-03-24T00:50:13.339 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.58390.log.gz 2024-03-24T00:50:13.339 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54045.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.81919.log.gz 2024-03-24T00:50:13.340 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68071.log 2024-03-24T00:50:13.340 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54045.log.gz 2024-03-24T00:50:13.340 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.68648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77560.log 2024-03-24T00:50:13.340 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70169.log 2024-03-24T00:50:13.340 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.68648.log.gz 2024-03-24T00:50:13.340 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.68071.log: /var/log/ceph/ceph-client.admin.77560.log: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83221.log 2024-03-24T00:50:13.340 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.68071.log.gz 2024-03-24T00:50:13.341 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.77560.log.gz 2024-03-24T00:50:13.341 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.70169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68263.log 2024-03-24T00:50:13.341 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.83221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66374.log 2024-03-24T00:50:13.341 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.68263.log: 88.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70869.log 2024-03-24T00:50:13.341 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.83221.log.gz 2024-03-24T00:50:13.341 INFO:teuthology.orchestra.run.smithi067.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.70169.log.gz 2024-03-24T00:50:13.341 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.68263.log.gz 2024-03-24T00:50:13.342 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.66374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83581.log 2024-03-24T00:50:13.342 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.66374.log.gz 2024-03-24T00:50:13.342 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.70869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68690.log 2024-03-24T00:50:13.342 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.83581.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.70869.log.gz 2024-03-24T00:50:13.342 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46526.log 2024-03-24T00:50:13.342 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.83581.log.gz 2024-03-24T00:50:13.342 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65420.log 2024-03-24T00:50:13.342 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.68690.log: /var/log/ceph/ceph-client.admin.46526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60251.log 2024-03-24T00:50:13.343 INFO:teuthology.orchestra.run.smithi067.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.68690.log.gz 2024-03-24T00:50:13.343 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59122.log 2024-03-24T00:50:13.343 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46526.log.gz 2024-03-24T00:50:13.343 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.65420.log.gz 2024-03-24T00:50:13.343 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52158.log 2024-03-24T00:50:13.343 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70766.log 2024-03-24T00:50:13.344 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.59122.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.60251.log.gz 2024-03-24T00:50:13.344 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46516.log 2024-03-24T00:50:13.344 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59122.log.gz 2024-03-24T00:50:13.344 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62348.log 2024-03-24T00:50:13.344 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52158.log.gz 2024-03-24T00:50:13.344 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.70766.log: /var/log/ceph/ceph-client.admin.46516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45242.log 2024-03-24T00:50:13.345 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62348.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46516.log.gz 2024-03-24T00:50:13.345 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66729.log 2024-03-24T00:50:13.345 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45242.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.62348.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54289.log 2024-03-24T00:50:13.345 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-24T00:50:13.345 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45242.log.gz 2024-03-24T00:50:13.345 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.66729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65570.log 2024-03-24T00:50:13.345 INFO:teuthology.orchestra.run.smithi067.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.70766.log.gz 2024-03-24T00:50:13.345 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64241.log 2024-03-24T00:50:13.346 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.66729.log.gz 2024-03-24T00:50:13.346 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54289.log: /var/log/ceph/ceph-client.admin.65570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57245.log 2024-03-24T00:50:13.346 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54289.log.gz 2024-03-24T00:50:13.346 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.65570.log.gz 2024-03-24T00:50:13.346 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60167.log 2024-03-24T00:50:13.346 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77283.log 2024-03-24T00:50:13.346 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.64241.log.gz 2024-03-24T00:50:13.347 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51490.log 2024-03-24T00:50:13.347 INFO:teuthology.orchestra.run.smithi067.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.57245.log.gz 2024-03-24T00:50:13.347 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.77283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81183.log 2024-03-24T00:50:13.347 INFO:teuthology.orchestra.run.smithi067.stderr:gzip/var/log/ceph/ceph-client.admin.51490.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.77283.log.gz 2024-03-24T00:50:13.347 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose 90.8% -- /var/log/ceph/ceph-client.admin.59480.log 2024-03-24T00:50:13.347 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.60167.log.gz 2024-03-24T00:50:13.347 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.81183.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51490.log.gz 2024-03-24T00:50:13.348 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59071.log 2024-03-24T00:50:13.348 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52577.log 2024-03-24T00:50:13.348 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.59480.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.81183.log.gz 2024-03-24T00:50:13.348 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.59071.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.59480.log.gz 2024-03-24T00:50:13.348 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63755.log 2024-03-24T00:50:13.348 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59071.log.gz 2024-03-24T00:50:13.348 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67510.log 2024-03-24T00:50:13.349 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52577.log: /var/log/ceph/ceph-client.admin.63755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52716.log 2024-03-24T00:50:13.349 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.52577.log.gz 2024-03-24T00:50:13.349 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.67510.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.63755.log.gz 2024-03-24T00:50:13.349 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55950.log 2024-03-24T00:50:13.349 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66813.log 2024-03-24T00:50:13.349 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.67510.log.gz 2024-03-24T00:50:13.349 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55950.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.52716.log.gz 2024-03-24T00:50:13.350 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44841.log 2024-03-24T00:50:13.350 INFO:teuthology.orchestra.run.smithi067.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.55950.log.gz 2024-03-24T00:50:13.350 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.66813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63917.log 2024-03-24T00:50:13.350 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65836.log 2024-03-24T00:50:13.350 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.66813.log.gz 2024-03-24T00:50:13.350 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51616.log 2024-03-24T00:50:13.350 INFO:teuthology.orchestra.run.smithi067.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.63917.log.gz 2024-03-24T00:50:13.350 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71119.log 2024-03-24T00:50:13.350 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69959.log 2024-03-24T00:50:13.350 INFO:teuthology.orchestra.run.smithi067.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.65836.log.gz 2024-03-24T00:50:13.351 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.71119.log: gzip -5 --verbose 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51616.log.gz 2024-03-24T00:50:13.351 INFO:teuthology.orchestra.run.smithi067.stderr: -- /var/log/ceph/ceph-client.admin.61797.log 2024-03-24T00:50:13.351 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.69959.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.71119.log.gz 2024-03-24T00:50:13.351 INFO:teuthology.orchestra.run.smithi067.stderr: 92.7% -- replaced with /var/log/ceph/ceph-client.admin.44841.log.gz 2024-03-24T00:50:13.351 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45840.log 2024-03-24T00:50:13.351 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.69959.log.gz 2024-03-24T00:50:13.351 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76610.log 2024-03-24T00:50:13.351 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61797.log: /var/log/ceph/ceph-client.admin.45840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46187.log 2024-03-24T00:50:13.351 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.61797.log.gz 2024-03-24T00:50:13.351 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.76610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55908.log 2024-03-24T00:50:13.351 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45840.log.gz 2024-03-24T00:50:13.351 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.76610.log.gz 2024-03-24T00:50:13.351 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71713.log 2024-03-24T00:50:13.351 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46187.log.gz 2024-03-24T00:50:13.351 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52619.log 2024-03-24T00:50:13.352 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55908.log: /var/log/ceph/ceph-client.admin.71713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.59821.log 2024-03-24T00:50:13.352 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.55908.log.gz 2024-03-24T00:50:13.352 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52619.log: 88.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81723.log 2024-03-24T00:50:13.352 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.71713.log.gz 2024-03-24T00:50:13.352 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52619.log.gz 2024-03-24T00:50:13.352 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.59821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67216.log 2024-03-24T00:50:13.352 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.59821.log.gz 2024-03-24T00:50:13.352 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-24T00:50:13.352 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.81723.log: /var/log/ceph/ceph-client.admin.67216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54672.log 2024-03-24T00:50:13.352 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.81723.log.gz 2024-03-24T00:50:13.352 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.67216.log.gz 2024-03-24T00:50:13.352 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.63398.log 2024-03-24T00:50:13.352 INFO:teuthology.orchestra.run.smithi067.stderr:gzip/var/log/ceph/ceph-client.admin.54672.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.56164.log 2024-03-24T00:50:13.352 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.63398.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54672.log.gz 2024-03-24T00:50:13.352 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.63398.log.gz 2024-03-24T00:50:13.353 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84001.log 2024-03-24T00:50:13.353 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59294.log 2024-03-24T00:50:13.353 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56164.log: /var/log/ceph/ceph-client.admin.84001.log: gzip -5 --verbose 89.2% -- replaced with /var/log/ceph/ceph-client.admin.56164.log.gz 2024-03-24T00:50:13.353 INFO:teuthology.orchestra.run.smithi067.stderr: -- /var/log/ceph/ceph.tmp-client.admin.31447.log 2024-03-24T00:50:13.353 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.59294.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.84001.log.gz 2024-03-24T00:50:13.353 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82478.log 2024-03-24T00:50:13.353 INFO:teuthology.orchestra.run.smithi067.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.59294.log.gz 2024-03-24T00:50:13.353 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53097.log 2024-03-24T00:50:13.353 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph.tmp-client.admin.31447.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.31447.log.gz 2024-03-24T00:50:13.353 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.82478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54003.log 2024-03-24T00:50:13.353 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.82478.log.gz 2024-03-24T00:50:13.353 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47748.log 2024-03-24T00:50:13.353 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53097.log.gz 2024-03-24T00:50:13.353 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73895.log 2024-03-24T00:50:13.354 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54003.log.gz 2024-03-24T00:50:13.354 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72804.log 2024-03-24T00:50:13.354 INFO:teuthology.orchestra.run.smithi067.stderr:gzip/var/log/ceph/ceph-client.admin.73895.log: -5 87.9% -- replaced with /var/log/ceph/ceph-client.admin.47748.log.gz 2024-03-24T00:50:13.354 INFO:teuthology.orchestra.run.smithi067.stderr: --verbose -- /var/log/ceph/ceph-client.admin.67298.log 2024-03-24T00:50:13.354 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.72804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83179.log 2024-03-24T00:50:13.354 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.72804.log.gz 2024-03-24T00:50:13.354 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.67298.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.73895.log.gz 2024-03-24T00:50:13.354 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81257.log 2024-03-24T00:50:13.354 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.67298.log.gz 2024-03-24T00:50:13.354 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.83179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62390.log 2024-03-24T00:50:13.354 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.83179.log.gz/var/log/ceph/ceph-client.admin.81257.log: 2024-03-24T00:50:13.354 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.69537.log 2024-03-24T00:50:13.354 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62390.log: gzip -5 --verbose 89.4% -- /var/log/ceph/ceph-client.admin.49130.log -- replaced with /var/log/ceph/ceph-client.admin.81257.log.gz 2024-03-24T00:50:13.354 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-24T00:50:13.355 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54588.log 2024-03-24T00:50:13.355 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.62390.log.gz 2024-03-24T00:50:13.355 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.69537.log: /var/log/ceph/ceph-client.admin.49130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76947.log 2024-03-24T00:50:13.355 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.69537.log.gz 2024-03-24T00:50:13.355 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54588.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49130.log.gz 2024-03-24T00:50:13.355 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58983.log 2024-03-24T00:50:13.355 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.76947.log: gzip -5 --verbose 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54588.log.gz 2024-03-24T00:50:13.355 INFO:teuthology.orchestra.run.smithi067.stderr: -- /var/log/ceph/ceph-client.0.80608.log 2024-03-24T00:50:13.355 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58983.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.76947.log.gz 2024-03-24T00:50:13.355 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70923.log 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58983.log.gz 2024-03-24T00:50:13.355 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-24T00:50:13.355 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.80608.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.80608.log.gz 2024-03-24T00:50:13.355 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61713.log 2024-03-24T00:50:13.355 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.70923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45630.log 2024-03-24T00:50:13.355 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.70923.log.gz 2024-03-24T00:50:13.356 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.59817.log 2024-03-24T00:50:13.356 INFO:teuthology.orchestra.run.smithi067.stderr: 94.4% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-24T00:50:13.356 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45630.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.61713.log.gz 2024-03-24T00:50:13.356 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69154.log 2024-03-24T00:50:13.356 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45630.log.gz 2024-03-24T00:50:13.356 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.59817.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.59817.log.gz 2024-03-24T00:50:13.356 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54756.log 2024-03-24T00:50:13.356 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.69154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70808.log 2024-03-24T00:50:13.356 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.69154.log.gz 2024-03-24T00:50:13.356 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61526.log 2024-03-24T00:50:13.356 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50065.log 2024-03-24T00:50:13.356 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54756.log.gz 2024-03-24T00:50:13.356 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.70808.log: /var/log/ceph/ceph-client.admin.61526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78163.log 2024-03-24T00:50:13.356 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65095.log 2024-03-24T00:50:13.356 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.61526.log.gz 2024-03-24T00:50:13.357 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50065.log.gz 2024-03-24T00:50:13.357 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57605.log 2024-03-24T00:50:13.357 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.78163.log: gzip/var/log/ceph/ceph-client.admin.65095.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.70543.log 2024-03-24T00:50:13.357 INFO:teuthology.orchestra.run.smithi067.stderr: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.70808.log.gz/var/log/ceph/ceph-client.admin.57605.log: 2024-03-24T00:50:13.357 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52887.log 2024-03-24T00:50:13.357 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.78163.log.gz 2024-03-24T00:50:13.357 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.70543.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.57605.log.gz 2024-03-24T00:50:13.357 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66626.log 2024-03-24T00:50:13.357 INFO:teuthology.orchestra.run.smithi067.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.65095.log.gz 2024-03-24T00:50:13.357 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.70543.log.gz 2024-03-24T00:50:13.357 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62573.log 2024-03-24T00:50:13.357 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47496.log 2024-03-24T00:50:13.357 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52887.log.gz 2024-03-24T00:50:13.357 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.66626.log: /var/log/ceph/ceph-client.admin.62573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68774.log 2024-03-24T00:50:13.357 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.66626.log.gz 87.5% -- replaced with /var/log/ceph/ceph-client.admin.62573.log.gz 2024-03-24T00:50:13.358 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-24T00:50:13.358 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81853.log 2024-03-24T00:50:13.358 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.68774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54326.log 2024-03-24T00:50:13.358 INFO:teuthology.orchestra.run.smithi067.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47496.log.gz 2024-03-24T00:50:13.358 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.81853.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.68774.log.gz 2024-03-24T00:50:13.358 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45882.log 2024-03-24T00:50:13.358 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53223.log 2024-03-24T00:50:13.358 INFO:teuthology.orchestra.run.smithi067.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.81853.log.gz 2024-03-24T00:50:13.358 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45882.log: gzip -5 --verbose 89.8% -- replaced with /var/log/ceph/ceph-client.admin.54326.log.gz 2024-03-24T00:50:13.358 INFO:teuthology.orchestra.run.smithi067.stderr: -- /var/log/ceph/ceph-client.admin.58941.log 2024-03-24T00:50:13.358 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53223.log: gzip -5 87.3% -- replaced with /var/log/ceph/ceph-client.admin.45882.log.gz 2024-03-24T00:50:13.358 INFO:teuthology.orchestra.run.smithi067.stderr: --verbose -- /var/log/ceph/ceph-client.admin.84313.log 2024-03-24T00:50:13.358 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53223.log.gz 2024-03-24T00:50:13.358 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72071.log 2024-03-24T00:50:13.359 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.58941.log.gz 2024-03-24T00:50:13.359 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.84313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47119.log 2024-03-24T00:50:13.359 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73188.log 2024-03-24T00:50:13.359 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.72071.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.84313.log.gz 2024-03-24T00:50:13.359 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58733.log 2024-03-24T00:50:13.359 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.72071.log.gz 2024-03-24T00:50:13.359 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.73188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56290.log 2024-03-24T00:50:13.359 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47119.log.gz 2024-03-24T00:50:13.359 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58733.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.73188.log.gz 2024-03-24T00:50:13.359 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47268.log 2024-03-24T00:50:13.359 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.58733.log.gz 2024-03-24T00:50:13.359 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59564.log 2024-03-24T00:50:13.359 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56290.log: /var/log/ceph/ceph-client.admin.47268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.69541.log 2024-03-24T00:50:13.359 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56290.log.gz 2024-03-24T00:50:13.359 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.59564.log: gzip 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47268.log.gz 2024-03-24T00:50:13.360 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.47664.log 2024-03-24T00:50:13.360 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.69541.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.69541.log.gz 2024-03-24T00:50:13.360 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59027.log 2024-03-24T00:50:13.360 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47664.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.59564.log.gz 2024-03-24T00:50:13.360 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67136.log 2024-03-24T00:50:13.360 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47664.log.gz 2024-03-24T00:50:13.360 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.59027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77480.log 2024-03-24T00:50:13.360 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59027.log.gz 2024-03-24T00:50:13.360 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.67136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51574.log 2024-03-24T00:50:13.360 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61839.log 2024-03-24T00:50:13.360 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.77480.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.67136.log.gz 2024-03-24T00:50:13.360 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65053.log 2024-03-24T00:50:13.360 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.77480.log.gz 2024-03-24T00:50:13.360 INFO:teuthology.orchestra.run.smithi067.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51574.log.gz 2024-03-24T00:50:13.360 INFO:teuthology.orchestra.run.smithi067.stderr:gzip/var/log/ceph/ceph-client.admin.61839.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.45116.log 2024-03-24T00:50:13.361 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56038.log 2024-03-24T00:50:13.361 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.61839.log.gz 2024-03-24T00:50:13.361 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.65053.log.gz 2024-03-24T00:50:13.361 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60045.log 2024-03-24T00:50:13.361 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56038.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45116.log.gz 2024-03-24T00:50:13.361 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83665.log 2024-03-24T00:50:13.361 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.56038.log.gz 2024-03-24T00:50:13.361 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61755.log 2024-03-24T00:50:13.361 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60045.log: /var/log/ceph/ceph-client.admin.83665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62871.log 2024-03-24T00:50:13.361 INFO:teuthology.orchestra.run.smithi067.stderr: 90.4%/var/log/ceph/ceph-client.admin.61755.log: -- replaced with /var/log/ceph/ceph-client.admin.60045.log.gz 2024-03-24T00:50:13.361 INFO:teuthology.orchestra.run.smithi067.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.83665.log.gz 2024-03-24T00:50:13.361 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44436.log 2024-03-24T00:50:13.361 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62871.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.61755.log.gz 2024-03-24T00:50:13.361 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69893.log 2024-03-24T00:50:13.361 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.62871.log.gz 2024-03-24T00:50:13.362 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43818.log 2024-03-24T00:50:13.362 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.69893.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44436.log.gz 2024-03-24T00:50:13.362 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-24T00:50:13.362 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59252.log 2024-03-24T00:50:13.362 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43818.log: /var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51994.log 2024-03-24T00:50:13.362 INFO:teuthology.orchestra.run.smithi067.stderr: 91.1% 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43818.log.gz 2024-03-24T00:50:13.362 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.69893.log.gz 2024-03-24T00:50:13.362 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.59252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47538.log 2024-03-24T00:50:13.362 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.59252.log.gz/var/log/ceph/ceph-client.admin.51994.log: 2024-03-24T00:50:13.362 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76413.log 2024-03-24T00:50:13.362 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47538.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51994.log.gz 2024-03-24T00:50:13.362 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47538.log.gz 2024-03-24T00:50:13.362 INFO:teuthology.orchestra.run.smithi067.stderr: -- /var/log/ceph/ceph-client.admin.52323.log 2024-03-24T00:50:13.362 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.76413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.56771.log 2024-03-24T00:50:13.363 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.76413.log.gz/var/log/ceph/ceph-client.admin.52323.log: 2024-03-24T00:50:13.363 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58899.log 2024-03-24T00:50:13.363 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.56771.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52323.log.gz 2024-03-24T00:50:13.363 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.56771.log.gz 2024-03-24T00:50:13.363 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44244.log 2024-03-24T00:50:13.363 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55218.log 2024-03-24T00:50:13.363 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58899.log: /var/log/ceph/ceph-client.admin.44244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56903.log 2024-03-24T00:50:13.363 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.58899.log.gz 2024-03-24T00:50:13.363 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44244.log.gzgzip 2024-03-24T00:50:13.363 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.79890.log 2024-03-24T00:50:13.363 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55218.log: /var/log/ceph/ceph-client.admin.56903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58558.log 2024-03-24T00:50:13.363 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3%/var/log/ceph/ceph-client.admin.79890.log: -- replaced with /var/log/ceph/ceph-client.admin.55218.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.0.79546.log 2024-03-24T00:50:13.363 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% 2024-03-24T00:50:13.363 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.56903.log.gz 2024-03-24T00:50:13.364 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.79890.log.gz 2024-03-24T00:50:13.364 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83497.log 2024-03-24T00:50:13.364 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58558.log: /var/log/ceph/ceph-client.0.79546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73471.log 2024-03-24T00:50:13.364 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.79546.log.gz 2024-03-24T00:50:13.364 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.83497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82992.log 2024-03-24T00:50:13.364 INFO:teuthology.orchestra.run.smithi067.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.58558.log.gz 2024-03-24T00:50:13.364 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 89.0% --verbose -- /var/log/ceph/ceph-client.admin.83749.log 2024-03-24T00:50:13.364 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.83497.log.gz/var/log/ceph/ceph-client.admin.73471.log: 2024-03-24T00:50:13.364 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.82992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.68450.log 2024-03-24T00:50:13.364 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.73471.log.gz 2024-03-24T00:50:13.364 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.82992.log.gz 2024-03-24T00:50:13.364 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68978.log 2024-03-24T00:50:13.364 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.83749.log: /var/log/ceph/ceph-client.0.68450.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/ceph-client.0.68450.log.gz 2024-03-24T00:50:13.364 INFO:teuthology.orchestra.run.smithi067.stderr: -- /var/log/ceph/ceph-client.admin.58119.log 2024-03-24T00:50:13.364 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.83749.log.gz 2024-03-24T00:50:13.364 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.68978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37477.log 2024-03-24T00:50:13.365 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58119.log: gzip -5 90.1% -- replaced with /var/log/ceph/ceph-client.admin.68978.log.gz 2024-03-24T00:50:13.365 INFO:teuthology.orchestra.run.smithi067.stderr: --verbose -- /var/log/ceph/ceph-client.0.48436.log 2024-03-24T00:50:13.365 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58119.log.gz 2024-03-24T00:50:13.365 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68155.log 2024-03-24T00:50:13.365 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5/var/log/ceph/ceph-client.0.48436.log: --verbose 92.0% -- replaced with /var/log/ceph/ceph-client.admin.37477.log.gz 2024-03-24T00:50:13.365 INFO:teuthology.orchestra.run.smithi067.stderr: -- 0.0% -- replaced with /var/log/ceph/ceph-client.0.48436.log.gz 2024-03-24T00:50:13.365 INFO:teuthology.orchestra.run.smithi067.stderr: /var/log/ceph/ceph-client.admin.77991.log 2024-03-24T00:50:13.365 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.68155.log: gzip -5 89.4% --verbose -- /var/log/ceph/ceph-client.admin.82656.log 2024-03-24T00:50:13.365 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.68155.log.gz 2024-03-24T00:50:13.365 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.77991.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-24T00:50:13.365 INFO:teuthology.orchestra.run.smithi067.stderr:gzip 89.0% -- replaced with /var/log/ceph/ceph-client.admin.77991.log.gz 2024-03-24T00:50:13.365 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.82656.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.43692.log 2024-03-24T00:50:13.365 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.82656.log.gz 2024-03-24T00:50:13.365 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70501.log 2024-03-24T00:50:13.365 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62306.log 2024-03-24T00:50:13.366 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43692.log.gz 2024-03-24T00:50:13.366 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.70501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45468.log 2024-03-24T00:50:13.366 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.70501.log.gz/var/log/ceph/ceph-client.admin.62306.log: 2024-03-24T00:50:13.366 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57647.log 2024-03-24T00:50:13.366 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.62306.log.gz 2024-03-24T00:50:13.366 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-24T00:50:13.366 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45468.log: /var/log/ceph/ceph-client.admin.57647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71935.log 2024-03-24T00:50:13.366 INFO:teuthology.orchestra.run.smithi067.stderr: 87.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-24T00:50:13.366 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-osd.2.log: 87.6% 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57647.log.gz 2024-03-24T00:50:13.366 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45468.log.gz 2024-03-24T00:50:13.366 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70085.log 2024-03-24T00:50:13.366 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.71935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46524.log 2024-03-24T00:50:13.366 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.71935.log.gz 2024-03-24T00:50:13.366 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.70085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75256.log 2024-03-24T00:50:13.366 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60533.log 2024-03-24T00:50:13.367 INFO:teuthology.orchestra.run.smithi067.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.70085.log.gz 2024-03-24T00:50:13.367 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.75256.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.46524.log.gz 2024-03-24T00:50:13.367 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50932.log 2024-03-24T00:50:13.367 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54206.log 2024-03-24T00:50:13.367 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.75256.log.gz/var/log/ceph/ceph-client.admin.50932.log: 2024-03-24T00:50:13.367 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37841.log 2024-03-24T00:50:13.367 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50932.log.gz 2024-03-24T00:50:13.367 INFO:teuthology.orchestra.run.smithi067.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.60533.log.gz 2024-03-24T00:50:13.367 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80000.log 2024-03-24T00:50:13.367 INFO:teuthology.orchestra.run.smithi067.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.54206.log.gz 2024-03-24T00:50:13.367 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66458.log 2024-03-24T00:50:13.367 INFO:teuthology.orchestra.run.smithi067.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.37841.log.gz 2024-03-24T00:50:13.367 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.80000.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-03-24T00:50:13.367 INFO:teuthology.orchestra.run.smithi067.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.80000.log.gz 2024-03-24T00:50:13.368 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61012.log 2024-03-24T00:50:13.368 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.66458.log: /var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52239.log 2024-03-24T00:50:13.368 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61012.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.66458.log.gz 2024-03-24T00:50:13.368 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61012.log.gz 2024-03-24T00:50:13.368 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81468.log 2024-03-24T00:50:13.368 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58221.log 2024-03-24T00:50:13.368 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52239.log.gz 2024-03-24T00:50:13.368 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.81468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49087.log 2024-03-24T00:50:13.368 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58221.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.58221.log.gz 2024-03-24T00:50:13.368 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55050.log 2024-03-24T00:50:13.368 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49087.log: 92.2% -- replaced with /var/log/ceph/ceph-client.admin.81468.log.gz 2024-03-24T00:50:13.368 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49087.log.gz 2024-03-24T00:50:13.368 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78457.log 2024-03-24T00:50:13.368 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82436.log 2024-03-24T00:50:13.368 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55050.log.gz 2024-03-24T00:50:13.368 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.78457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74294.log 2024-03-24T00:50:13.369 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6%/var/log/ceph/ceph-client.admin.82436.log: -- replaced with /var/log/ceph/ceph-client.admin.78457.log.gz 2024-03-24T00:50:13.369 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.82436.log.gz 2024-03-24T00:50:13.369 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73230.log 2024-03-24T00:50:13.369 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.74294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44370.log 2024-03-24T00:50:13.369 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.74294.log.gz 2024-03-24T00:50:13.369 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.73230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45588.log 2024-03-24T00:50:13.369 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44370.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.73230.log.gz 2024-03-24T00:50:13.369 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67466.log 91.2% -- replaced with /var/log/ceph/ceph-client.admin.44370.log.gz 2024-03-24T00:50:13.369 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-24T00:50:13.369 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61232.log 2024-03-24T00:50:13.369 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45588.log: /var/log/ceph/ceph-client.admin.67466.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45588.log.gz 2024-03-24T00:50:13.369 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.67466.log.gz 2024-03-24T00:50:13.369 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71851.log 2024-03-24T00:50:13.369 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61232.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.61232.log.gz 2024-03-24T00:50:13.369 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46522.log 2024-03-24T00:50:13.370 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.71851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55344.log 2024-03-24T00:50:13.370 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.71851.log.gz 2024-03-24T00:50:13.370 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54882.log 2024-03-24T00:50:13.370 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46522.log.gz 2024-03-24T00:50:13.370 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55344.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55344.log.gz 2024-03-24T00:50:13.370 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74680.log 2024-03-24T00:50:13.370 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63057.log 2024-03-24T00:50:13.370 INFO:teuthology.orchestra.run.smithi067.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.54882.log.gz 2024-03-24T00:50:13.370 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.74680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83833.log 2024-03-24T00:50:13.370 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63057.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.74680.log.gz 2024-03-24T00:50:13.370 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.63057.log.gz 2024-03-24T00:50:13.370 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69100.log 2024-03-24T00:50:13.371 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.83833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72984.log 2024-03-24T00:50:13.371 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.83833.log.gz 2024-03-24T00:50:13.371 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.69100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46862.log 2024-03-24T00:50:13.371 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.72984.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.69100.log.gz 2024-03-24T00:50:13.371 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.72984.log.gz 2024-03-24T00:50:13.371 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78033.log 2024-03-24T00:50:13.371 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52761.log 2024-03-24T00:50:13.371 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46862.log.gz 2024-03-24T00:50:13.371 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.78033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44076.log 2024-03-24T00:50:13.371 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52761.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.78033.log.gz 2024-03-24T00:50:13.371 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52761.log.gz 2024-03-24T00:50:13.371 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57161.log 2024-03-24T00:50:13.371 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57521.log 2024-03-24T00:50:13.371 INFO:teuthology.orchestra.run.smithi067.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44076.log.gz 2024-03-24T00:50:13.372 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76091.log 2024-03-24T00:50:13.372 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57521.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.57521.log.gz 2024-03-24T00:50:13.372 INFO:teuthology.orchestra.run.smithi067.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.57161.log.gz 2024-03-24T00:50:13.372 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66957.log 2024-03-24T00:50:13.372 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.76091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72594.log 2024-03-24T00:50:13.372 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.66957.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.76091.log.gz 2024-03-24T00:50:13.372 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.66957.log.gz 2024-03-24T00:50:13.372 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73643.log 2024-03-24T00:50:13.372 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.72594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52971.log 2024-03-24T00:50:13.372 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.72594.log.gz 2024-03-24T00:50:13.372 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.73643.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.73643.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67861.log 2024-03-24T00:50:13.372 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-24T00:50:13.372 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52971.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52971.log.gz 2024-03-24T00:50:13.372 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58161.log 2024-03-24T00:50:13.372 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.67861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69284.log 2024-03-24T00:50:13.373 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.67861.log.gz 2024-03-24T00:50:13.373 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.53644.log 2024-03-24T00:50:13.373 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.69284.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.58161.log.gz 2024-03-24T00:50:13.373 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72678.log 2024-03-24T00:50:13.373 INFO:teuthology.orchestra.run.smithi067.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.69284.log.gz 2024-03-24T00:50:13.373 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.53644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76263.log 2024-03-24T00:50:13.373 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.53644.log.gz/var/log/ceph/ceph-client.admin.72678.log: 2024-03-24T00:50:13.373 INFO:teuthology.orchestra.run.smithi067.stderr:gzip 88.4% -- replaced with /var/log/ceph/ceph-client.admin.72678.log.gz 2024-03-24T00:50:13.373 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.55992.log 2024-03-24T00:50:13.373 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44118.log 2024-03-24T00:50:13.373 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.76263.log: /var/log/ceph/ceph-client.admin.55992.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55992.log.gz 2024-03-24T00:50:13.373 INFO:teuthology.orchestra.run.smithi067.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.76263.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45384.log 2024-03-24T00:50:13.373 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-24T00:50:13.374 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44118.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44118.log.gz 2024-03-24T00:50:13.374 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78373.log 2024-03-24T00:50:13.374 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81025.log 2024-03-24T00:50:13.374 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45384.log: /var/log/ceph/ceph-client.admin.78373.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45384.log.gz 2024-03-24T00:50:13.374 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82824.log 2024-03-24T00:50:13.374 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.78373.log.gz 2024-03-24T00:50:13.374 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.81025.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.81025.log.gz 2024-03-24T00:50:13.374 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53055.log 2024-03-24T00:50:13.374 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66771.log 2024-03-24T00:50:13.374 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.82824.log: /var/log/ceph/ceph-client.admin.53055.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.82824.log.gz 2024-03-24T00:50:13.374 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53055.log.gz 2024-03-24T00:50:13.374 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78415.log 2024-03-24T00:50:13.374 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.66771.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.66771.log.gz 2024-03-24T00:50:13.374 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49897.log 2024-03-24T00:50:13.374 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82520.log 2024-03-24T00:50:13.375 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.78415.log: /var/log/ceph/ceph-client.admin.49897.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.78415.log.gz 2024-03-24T00:50:13.375 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49897.log.gz 2024-03-24T00:50:13.375 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.82106.log 2024-03-24T00:50:13.375 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.82520.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.82520.log.gz 2024-03-24T00:50:13.375 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84043.log 2024-03-24T00:50:13.375 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73064.log 2024-03-24T00:50:13.375 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.82106.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.82106.log.gz 2024-03-24T00:50:13.375 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.84043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69196.log 2024-03-24T00:50:13.375 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.84043.log.gz 2024-03-24T00:50:13.375 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.73064.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.73064.log.gz 2024-03-24T00:50:13.375 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47790.log 2024-03-24T00:50:13.375 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58653.log 2024-03-24T00:50:13.375 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.69196.log: /var/log/ceph/ceph-client.admin.47790.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.69196.log.gz 2024-03-24T00:50:13.375 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47790.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53181.log 2024-03-24T00:50:13.375 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-24T00:50:13.375 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58653.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.58653.log.gz 2024-03-24T00:50:13.376 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59206.log 2024-03-24T00:50:13.376 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66686.log 2024-03-24T00:50:13.376 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53181.log: /var/log/ceph/ceph-client.admin.59206.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53181.log.gz 2024-03-24T00:50:13.376 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59206.log.gz 2024-03-24T00:50:13.376 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62530.log 2024-03-24T00:50:13.376 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.66686.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.66686.log.gz 2024-03-24T00:50:13.376 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76737.log 2024-03-24T00:50:13.376 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.64432.log 2024-03-24T00:50:13.376 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62530.log: /var/log/ceph/ceph-client.admin.76737.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.62530.log.gz 2024-03-24T00:50:13.376 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.76737.log.gzgzip 2024-03-24T00:50:13.376 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.45756.log 2024-03-24T00:50:13.376 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.64432.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.64432.log.gz 2024-03-24T00:50:13.376 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78499.log 2024-03-24T00:50:13.376 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81765.log 2024-03-24T00:50:13.376 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45756.log: /var/log/ceph/ceph-client.admin.78499.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.45756.log.gz 2024-03-24T00:50:13.377 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.78499.log.gz 2024-03-24T00:50:13.377 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56518.log 2024-03-24T00:50:13.377 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.81765.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.81765.log.gz 2024-03-24T00:50:13.377 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60467.log 2024-03-24T00:50:13.377 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81402.log 2024-03-24T00:50:13.377 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56518.log: /var/log/ceph/ceph-client.admin.60467.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.56518.log.gz 2024-03-24T00:50:13.377 INFO:teuthology.orchestra.run.smithi067.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.60467.log.gz 2024-03-24T00:50:13.377 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61568.log 2024-03-24T00:50:13.377 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.81402.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.81402.log.gz 2024-03-24T00:50:13.377 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82562.log 2024-03-24T00:50:13.377 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75488.log 2024-03-24T00:50:13.377 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61568.log: /var/log/ceph/ceph-client.admin.82562.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.61568.log.gz 2024-03-24T00:50:13.377 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.82562.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82698.log 2024-03-24T00:50:13.377 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-24T00:50:13.377 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.75488.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.75488.log.gz 2024-03-24T00:50:13.377 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68197.log 2024-03-24T00:50:13.377 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64760.log 2024-03-24T00:50:13.378 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.82698.log: /var/log/ceph/ceph-client.admin.68197.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.82698.log.gz 2024-03-24T00:50:13.378 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59420.log 2024-03-24T00:50:13.378 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64760.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.68197.log.gz 2024-03-24T00:50:13.378 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.64760.log.gz 2024-03-24T00:50:13.378 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72846.log 2024-03-24T00:50:13.378 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47580.log 2024-03-24T00:50:13.378 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.59420.log: /var/log/ceph/ceph-client.admin.72846.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.59420.log.gz 2024-03-24T00:50:13.378 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.72846.log.gz 2024-03-24T00:50:13.378 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59164.log 2024-03-24T00:50:13.378 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80421.log 2024-03-24T00:50:13.378 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.59164.log: /var/log/ceph/ceph-client.admin.47580.log: 88.9% 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47580.log.gz 2024-03-24T00:50:13.378 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.59164.log.gz 2024-03-24T00:50:13.378 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62060.log 2024-03-24T00:50:13.378 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49603.log 2024-03-24T00:50:13.379 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.80421.log: /var/log/ceph/ceph-client.admin.62060.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.80421.log.gz 2024-03-24T00:50:13.379 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.62060.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63099.log 2024-03-24T00:50:13.379 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-24T00:50:13.379 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43776.log 2024-03-24T00:50:13.379 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49603.log: /var/log/ceph/ceph-client.admin.63099.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63099.log.gz 2024-03-24T00:50:13.379 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49603.log.gz 2024-03-24T00:50:13.379 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73402.log 2024-03-24T00:50:13.379 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73513.log 2024-03-24T00:50:13.379 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43776.log.gz 2024-03-24T00:50:13.379 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.73402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77073.log 2024-03-24T00:50:13.379 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.73513.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.73402.log.gz 2024-03-24T00:50:13.379 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.73513.log.gz 2024-03-24T00:50:13.379 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49561.log 2024-03-24T00:50:13.379 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.77073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67424.log 2024-03-24T00:50:13.379 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.77073.log.gz 2024-03-24T00:50:13.379 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84187.log 2024-03-24T00:50:13.380 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.67424.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49561.log.gz 2024-03-24T00:50:13.380 INFO:teuthology.orchestra.run.smithi067.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.67424.log.gz 2024-03-24T00:50:13.380 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50302.log 2024-03-24T00:50:13.380 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.84187.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-03-24T00:50:13.380 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.84187.log.gz 2024-03-24T00:50:13.380 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.63394.log 2024-03-24T00:50:13.380 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50302.log.gz 2024-03-24T00:50:13.380 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53349.log 2024-03-24T00:50:13.380 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.63394.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.63394.log.gz 2024-03-24T00:50:13.380 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79043.log 2024-03-24T00:50:13.380 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53349.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53349.log.gz 2024-03-24T00:50:13.380 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66855.log 2024-03-24T00:50:13.380 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.79043.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.79043.log.gz 2024-03-24T00:50:13.380 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70683.log 2024-03-24T00:50:13.380 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.66855.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.66855.log.gz 2024-03-24T00:50:13.380 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70706.log 2024-03-24T00:50:13.380 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.70683.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.70683.log.gz 2024-03-24T00:50:13.381 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64718.log 2024-03-24T00:50:13.381 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.70706.log: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.70706.log.gz 2024-03-24T00:50:13.381 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83707.log 2024-03-24T00:50:13.381 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64718.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.64718.log.gz 2024-03-24T00:50:13.381 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79764.log 2024-03-24T00:50:13.381 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.83707.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.83707.log.gz 2024-03-24T00:50:13.381 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57909.log 2024-03-24T00:50:13.381 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.79764.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.79764.log.gz 2024-03-24T00:50:13.381 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51868.log 2024-03-24T00:50:13.381 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57909.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.57909.log.gz 2024-03-24T00:50:13.381 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44286.log 2024-03-24T00:50:13.381 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51868.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51868.log.gz 2024-03-24T00:50:13.381 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76347.log 2024-03-24T00:50:13.381 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44286.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44286.log.gz 2024-03-24T00:50:13.381 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62222.log 2024-03-24T00:50:13.382 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.76347.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.76347.log.gz 2024-03-24T00:50:13.382 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.48874.log 2024-03-24T00:50:13.382 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62222.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.62222.log.gz 2024-03-24T00:50:13.382 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59378.log 2024-03-24T00:50:13.382 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.2.48874.log: 0.0% -- replaced with /var/log/ceph/ceph-client.2.48874.log.gz 2024-03-24T00:50:13.382 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52078.log 2024-03-24T00:50:13.382 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.59378.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.59378.log.gz 2024-03-24T00:50:13.382 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60317.log 2024-03-24T00:50:13.382 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52078.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52078.log.gz 2024-03-24T00:50:13.382 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69701.log 2024-03-24T00:50:13.382 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60317.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.60317.log.gz 2024-03-24T00:50:13.382 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.69701.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.69701.log.gz 2024-03-24T00:50:13.383 INFO:teuthology.orchestra.run.smithi072.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-24T00:50:13.383 INFO:teuthology.orchestra.run.smithi067.stderr: 92.1% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-03-24T00:50:13.383 INFO:teuthology.orchestra.run.smithi067.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-03-24T00:50:13.394 INFO:teuthology.orchestra.run.smithi072.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-24T00:50:13.672 INFO:teuthology.orchestra.run.smithi067.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-24T00:50:14.163 INFO:teuthology.orchestra.run.smithi072.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-24T00:50:14.222 INFO:teuthology.orchestra.run.smithi072.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-24T00:50:14.241 INFO:teuthology.orchestra.run.smithi072.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-24T00:50:14.289 INFO:teuthology.orchestra.run.smithi072.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-24T00:50:14.291 INFO:teuthology.orchestra.run.smithi072.stderr: 2024-03-24T00:50:14.291 INFO:teuthology.orchestra.run.smithi072.stderr:real 0m1.375s 2024-03-24T00:50:14.291 INFO:teuthology.orchestra.run.smithi072.stderr:user 0m6.049s 2024-03-24T00:50:14.291 INFO:teuthology.orchestra.run.smithi072.stderr:sys 0m0.230s 2024-03-24T00:50:14.337 INFO:teuthology.orchestra.run.smithi067.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-24T00:50:14.453 INFO:teuthology.orchestra.run.smithi067.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-24T00:50:14.512 INFO:teuthology.orchestra.run.smithi067.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-24T00:50:14.550 INFO:teuthology.orchestra.run.smithi067.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-24T00:50:14.552 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-24T00:50:14.552 INFO:teuthology.orchestra.run.smithi067.stderr:real 0m1.661s 2024-03-24T00:50:14.552 INFO:teuthology.orchestra.run.smithi067.stderr:user 0m7.016s 2024-03-24T00:50:14.552 INFO:teuthology.orchestra.run.smithi067.stderr:sys 0m0.604s 2024-03-24T00:50:14.552 INFO:tasks.ceph:Archiving logs... 2024-03-24T00:50:14.553 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616442/remote/smithi067/log 2024-03-24T00:50:14.554 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-24T00:50:16.156 DEBUG:teuthology.misc:Transferring archived files from smithi072:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616442/remote/smithi072/log 2024-03-24T00:50:16.157 DEBUG:teuthology.orchestra.run.smithi072:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-24T00:50:17.024 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-24T00:50:17.034 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-24T00:50:17.035 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-24T00:50:17.037 DEBUG:teuthology.orchestra.run.smithi072:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-24T00:50:17.127 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-24T00:50:17.137 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-24T00:50:17.137 DEBUG:teuthology.orchestra.run.smithi067:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T00:50:17.140 DEBUG:teuthology.orchestra.run.smithi072:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T00:50:17.154 INFO:teuthology.orchestra.run.smithi067.stderr:bash: line 1: ntpq: command not found 2024-03-24T00:50:17.155 INFO:teuthology.orchestra.run.smithi072.stderr:bash: line 1: ntpq: command not found 2024-03-24T00:50:17.158 INFO:teuthology.orchestra.run.smithi067.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T00:50:17.158 INFO:teuthology.orchestra.run.smithi067.stdout:=============================================================================== 2024-03-24T00:50:17.158 INFO:teuthology.orchestra.run.smithi067.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 56 +112us[ +112us] +/- 101ms 2024-03-24T00:50:17.158 INFO:teuthology.orchestra.run.smithi067.stdout:^* hv02.front.sepia.ceph.com 3 6 377 61 -1207us[-1240us] +/- 77ms 2024-03-24T00:50:17.158 INFO:teuthology.orchestra.run.smithi067.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 59 +974us[ +974us] +/- 68ms 2024-03-24T00:50:17.158 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T00:50:17.159 INFO:teuthology.orchestra.run.smithi072.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T00:50:17.159 INFO:teuthology.orchestra.run.smithi072.stdout:=============================================================================== 2024-03-24T00:50:17.159 INFO:teuthology.orchestra.run.smithi072.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 59 +131us[ +131us] +/- 101ms 2024-03-24T00:50:17.160 INFO:teuthology.orchestra.run.smithi072.stdout:^* hv02.front.sepia.ceph.com 3 6 377 59 -1205us[-1139us] +/- 77ms 2024-03-24T00:50:17.160 INFO:teuthology.orchestra.run.smithi072.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 58 +975us[ +975us] +/- 68ms 2024-03-24T00:50:17.160 INFO:teuthology.orchestra.run.smithi072.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T00:50:17.160 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-24T00:50:17.170 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-24T00:50:17.170 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-24T00:50:17.181 DEBUG:teuthology.orchestra.run.smithi067:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T00:50:17.295 DEBUG:teuthology.orchestra.run.smithi072:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T00:50:17.369 DEBUG:teuthology.orchestra.run.smithi067:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T00:50:17.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:50:17.400 DEBUG:teuthology.orchestra.run.smithi072:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T00:50:17.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:50:17.431 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-24T00:50:17.440 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-24T00:50:17.451 INFO:teuthology.task.internal:Duration was 1295.977798 seconds 2024-03-24T00:50:17.451 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-24T00:50:17.462 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-24T00:50:17.462 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T00:50:17.465 DEBUG:teuthology.orchestra.run.smithi072:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T00:50:17.506 INFO:teuthology.orchestra.run.smithi067.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T00:50:17.515 INFO:teuthology.orchestra.run.smithi072.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T00:50:17.782 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-24T00:50:17.782 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi067.front.sepia.ceph.com 2024-03-24T00:50:17.782 DEBUG:teuthology.orchestra.run.smithi067:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-24T00:50:17.823 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi072.front.sepia.ceph.com 2024-03-24T00:50:17.823 DEBUG:teuthology.orchestra.run.smithi072:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-24T00:50:17.844 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-24T00:50:17.844 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-24T00:50:17.866 DEBUG:teuthology.orchestra.run.smithi072:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-24T00:50:17.913 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-24T00:50:17.913 DEBUG:teuthology.orchestra.run.smithi067:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T00:50:17.935 DEBUG:teuthology.orchestra.run.smithi072:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T00:50:18.067 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-24T00:50:18.078 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-24T00:50:18.078 DEBUG:teuthology.orchestra.run.smithi067:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T00:50:18.105 DEBUG:teuthology.orchestra.run.smithi072:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T00:50:18.131 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-24T00:50:18.142 DEBUG:teuthology.orchestra.run.smithi067:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T00:50:18.148 DEBUG:teuthology.orchestra.run.smithi072:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T00:50:18.170 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = core 2024-03-24T00:50:18.197 INFO:teuthology.orchestra.run.smithi072.stdout:kernel.core_pattern = core 2024-03-24T00:50:18.213 DEBUG:teuthology.orchestra.run.smithi067:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-24T00:50:18.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:50:18.241 DEBUG:teuthology.orchestra.run.smithi072:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-24T00:50:18.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:50:18.267 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-24T00:50:18.279 INFO:teuthology.task.internal:Transferring archived files... 2024-03-24T00:50:18.280 DEBUG:teuthology.misc:Transferring archived files from smithi067:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616442/remote/smithi067 2024-03-24T00:50:18.280 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T00:50:18.319 DEBUG:teuthology.misc:Transferring archived files from smithi072:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616442/remote/smithi072 2024-03-24T00:50:18.319 DEBUG:teuthology.orchestra.run.smithi072:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T00:50:18.354 INFO:teuthology.task.internal:Removing archive directory... 2024-03-24T00:50:18.354 DEBUG:teuthology.orchestra.run.smithi067:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-24T00:50:18.356 DEBUG:teuthology.orchestra.run.smithi072:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-24T00:50:18.404 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-24T00:50:18.416 INFO:teuthology.task.internal:Not uploading archives. 2024-03-24T00:50:18.417 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-24T00:50:18.428 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-24T00:50:18.428 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T00:50:18.430 DEBUG:teuthology.orchestra.run.smithi072:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T00:50:18.442 INFO:teuthology.orchestra.run.smithi067.stdout: 266042 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 00:50 /home/ubuntu/cephtest 2024-03-24T00:50:18.458 INFO:teuthology.orchestra.run.smithi072.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 00:50 /home/ubuntu/cephtest 2024-03-24T00:50:18.460 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-24T00:50:18.471 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-24T00:50:18.496 INFO:teuthology.nuke:Checking targets against current locks 2024-03-24T00:50:18.523 DEBUG:teuthology.nuke:shortname: smithi067 2024-03-24T00:50:18.523 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T00:50:18.547 DEBUG:teuthology.nuke:shortname: smithi072 2024-03-24T00:50:18.547 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T00:50:18.550 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi067.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616442', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 00:17:14.605264', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB8CD0DPVKDGSdsXcNniFZ/nMWBblB6BCthjTFboTuuuZGK72PZsmuWPKA4ddb/TZ5BhSV6teWCVIHkd5gVxOLw='} 2024-03-24T00:50:18.571 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi072.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616442', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 00:17:14.603777', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHKZ2OBRMHPdFxGrk+8dJ5aBUfN0YoKZJ/yHulR5ozeKvvv8ePCbz4UDpFjr7jPvwgWT4STFERjN8cT9GeVy2Ps='} 2024-03-24T00:50:18.607 INFO:teuthology.orchestra.console.smithi067:Power off 2024-03-24T00:50:18.607 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T00:50:18.632 DEBUG:teuthology.orchestra.console.smithi067:power off output: Chassis Power Control: Down/Off 2024-03-24T00:50:18.632 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T00:50:18.648 INFO:teuthology.orchestra.console.smithi072:Power off 2024-03-24T00:50:18.648 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T00:50:18.657 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is on 2024-03-24T00:50:18.670 DEBUG:teuthology.orchestra.console.smithi072:power off output: Chassis Power Control: Down/Off 2024-03-24T00:50:18.671 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T00:50:18.694 DEBUG:teuthology.orchestra.console.smithi072:check power output: Chassis Power is on 2024-03-24T00:50:22.658 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T00:50:22.695 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T00:50:22.770 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is on 2024-03-24T00:50:22.804 DEBUG:teuthology.orchestra.console.smithi072:check power output: Chassis Power is on 2024-03-24T00:50:26.774 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T00:50:26.805 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T00:50:26.885 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is on 2024-03-24T00:50:26.914 DEBUG:teuthology.orchestra.console.smithi072:check power output: Chassis Power is off 2024-03-24T00:50:27.015 INFO:teuthology.orchestra.console.smithi072:Power off completed 2024-03-24T00:50:30.887 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T00:50:30.997 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is off 2024-03-24T00:50:31.098 INFO:teuthology.orchestra.console.smithi067:Power off completed 2024-03-24T00:50:31.203 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/forward-scrub} duration: 1295.9777977466583 failure_reason: '"2024-03-24T00:45:15.653093+0000 mds.c (mds.0) 1 : cluster [ERR] dir 0x10000000000 object missing on disk; some files may be lost (/dir)" in cluster log' flavor: default owner: mchangir success: false 2024-03-24T00:50:31.203 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-24T00:50:31.293 INFO:teuthology.run:FAIL